2024-09-16T01:23:51.581 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T01:23:51.603 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T01:23:51.704 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906477 branch: squid description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/damage} email: ceph-qa@ceph.com first_in_suite: false job_id: '7906477' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 mon warn on pool no app: false osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - bad backtrace - object missing on disk - error reading table object - error reading sessionmap - Error loading MDS rank - missing journal object - Error recovering journal - error decoding table object - failed to read JournalPointer - Corrupt directory entry - Corrupt fnode header - corrupt sessionmap header - Corrupt dentry - Scrub error on inode - Metadata damage detected - MDS_READ_ONLY - force file system read-only - with standby daemon mds - MDS abort because newly corrupt dentry sha1: 8921c8ef947807b717db041d85f0761b7914d5fc ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 8921c8ef947807b717db041d85f0761b7914d5fc kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 subvolume_version: 1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: ad35db06e8924912f01e32b35576297c0f1352b4 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 3393 sha1: 8921c8ef947807b717db041d85f0761b7914d5fc sleep_before_teardown: 0 subset: 294/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ad35db06e8924912f01e32b35576297c0f1352b4 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi088.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJf1XKuP5OBdoKK+dsWzt525t5mpZRRcUek1q/bTQo6bn4Ymh40GssGAU6et8JmZObOPR8vvLFEJgZaS2JlYE2M= smithi154.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGIUUvQMJPOWQntC2askAfxyGFKNXJ75frno6BkG+7XTKf0coPjm/L1oRi78clEDplLj4LZz5B/qYO/4/Ygp37U= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_damage teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_21:24:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T01:23:51.704 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa; will attempt to use it 2024-09-16T01:23:51.705 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks 2024-09-16T01:23:51.705 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T01:23:51.706 INFO:teuthology.task.internal:Checking packages... 2024-09-16T01:23:51.727 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8921c8ef947807b717db041d85f0761b7914d5fc' 2024-09-16T01:23:51.727 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T01:23:51.727 INFO:teuthology.packaging:ref: None 2024-09-16T01:23:51.727 INFO:teuthology.packaging:tag: None 2024-09-16T01:23:51.727 INFO:teuthology.packaging:branch: squid 2024-09-16T01:23:51.727 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T01:23:51.727 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-16T01:23:51.930 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-328.gad35db06 2024-09-16T01:23:51.932 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T01:23:51.941 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T01:23:51.941 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T01:23:51.961 INFO:teuthology.task.internal:Saving configuration 2024-09-16T01:23:51.981 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T01:23:51.987 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T01:23:52.009 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906477', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 01:18:33.989762', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJf1XKuP5OBdoKK+dsWzt525t5mpZRRcUek1q/bTQo6bn4Ymh40GssGAU6et8JmZObOPR8vvLFEJgZaS2JlYE2M='} 2024-09-16T01:23:52.033 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906477', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 01:18:33.987862', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGIUUvQMJPOWQntC2askAfxyGFKNXJ75frno6BkG+7XTKf0coPjm/L1oRi78clEDplLj4LZz5B/qYO/4/Ygp37U='} 2024-09-16T01:23:52.033 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T01:23:52.039 INFO:teuthology.task.internal:roles: ubuntu@smithi088.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-09-16T01:23:52.039 INFO:teuthology.task.internal:roles: ubuntu@smithi154.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-09-16T01:23:52.039 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T01:23:52.102 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3626673be0>, signals=[15]) 2024-09-16T01:23:52.102 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T01:23:52.107 INFO:teuthology.task.internal:Opening connections... 2024-09-16T01:23:52.107 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:23:52.109 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T01:23:52.189 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi154.front.sepia.ceph.com 2024-09-16T01:23:52.190 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T01:23:52.265 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T01:23:52.271 DEBUG:teuthology.orchestra.run.smithi088:> uname -m 2024-09-16T01:23:52.290 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64 2024-09-16T01:23:52.290 DEBUG:teuthology.orchestra.run.smithi088:> cat /etc/os-release 2024-09-16T01:23:52.347 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="CentOS Stream" 2024-09-16T01:23:52.348 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="9" 2024-09-16T01:23:52.348 INFO:teuthology.orchestra.run.smithi088.stdout:ID="centos" 2024-09-16T01:23:52.348 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE="rhel fedora" 2024-09-16T01:23:52.348 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="9" 2024-09-16T01:23:52.348 INFO:teuthology.orchestra.run.smithi088.stdout:PLATFORM_ID="platform:el9" 2024-09-16T01:23:52.348 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T01:23:52.349 INFO:teuthology.orchestra.run.smithi088.stdout:ANSI_COLOR="0;31" 2024-09-16T01:23:52.349 INFO:teuthology.orchestra.run.smithi088.stdout:LOGO="fedora-logo-icon" 2024-09-16T01:23:52.349 INFO:teuthology.orchestra.run.smithi088.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T01:23:52.349 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://centos.org/" 2024-09-16T01:23:52.349 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T01:23:52.349 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T01:23:52.349 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T01:23:52.350 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server 2024-09-16T01:23:52.372 DEBUG:teuthology.orchestra.run.smithi154:> uname -m 2024-09-16T01:23:52.390 INFO:teuthology.orchestra.run.smithi154.stdout:x86_64 2024-09-16T01:23:52.390 DEBUG:teuthology.orchestra.run.smithi154:> cat /etc/os-release 2024-09-16T01:23:52.449 INFO:teuthology.orchestra.run.smithi154.stdout:NAME="CentOS Stream" 2024-09-16T01:23:52.449 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION="9" 2024-09-16T01:23:52.449 INFO:teuthology.orchestra.run.smithi154.stdout:ID="centos" 2024-09-16T01:23:52.449 INFO:teuthology.orchestra.run.smithi154.stdout:ID_LIKE="rhel fedora" 2024-09-16T01:23:52.449 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_ID="9" 2024-09-16T01:23:52.449 INFO:teuthology.orchestra.run.smithi154.stdout:PLATFORM_ID="platform:el9" 2024-09-16T01:23:52.449 INFO:teuthology.orchestra.run.smithi154.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T01:23:52.449 INFO:teuthology.orchestra.run.smithi154.stdout:ANSI_COLOR="0;31" 2024-09-16T01:23:52.449 INFO:teuthology.orchestra.run.smithi154.stdout:LOGO="fedora-logo-icon" 2024-09-16T01:23:52.450 INFO:teuthology.orchestra.run.smithi154.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T01:23:52.450 INFO:teuthology.orchestra.run.smithi154.stdout:HOME_URL="https://centos.org/" 2024-09-16T01:23:52.450 INFO:teuthology.orchestra.run.smithi154.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T01:23:52.450 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T01:23:52.450 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T01:23:52.452 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2024-09-16T01:23:52.471 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T01:23:52.481 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T01:23:52.491 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T01:23:52.492 DEBUG:teuthology.orchestra.run.smithi088:> test '!' -e /home/ubuntu/cephtest 2024-09-16T01:23:52.494 DEBUG:teuthology.orchestra.run.smithi154:> test '!' -e /home/ubuntu/cephtest 2024-09-16T01:23:52.511 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T01:23:52.517 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T01:23:52.517 DEBUG:teuthology.orchestra.run.smithi088:> test -z $(ls -A /var/lib/ceph) 2024-09-16T01:23:52.551 DEBUG:teuthology.orchestra.run.smithi154:> test -z $(ls -A /var/lib/ceph) 2024-09-16T01:23:52.603 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T01:23:52.658 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T01:23:52.672 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-09-16T01:23:52.672 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2024-09-16T01:23:52.673 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2024-09-16T01:23:52.673 DEBUG:teuthology.orchestra.run.smithi088:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T01:23:52.673 DEBUG:teuthology.orchestra.run.smithi154:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T01:23:52.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:23:52.690 DEBUG:teuthology.orchestra.run.smithi154:> uname -r 2024-09-16T01:23:52.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:23:52.691 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-09-16T01:23:52.746 INFO:teuthology.orchestra.run.smithi154.stdout:5.14.0-503.el9.x86_64 2024-09-16T01:23:52.746 INFO:teuthology.task.kernel:Running kernel on smithi154: 5.14.0-503.el9.x86_64 2024-09-16T01:23:52.746 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum install -y kernel 2024-09-16T01:23:52.748 INFO:teuthology.orchestra.run.smithi088.stdout:5.14.0-503.el9.x86_64 2024-09-16T01:23:52.748 INFO:teuthology.task.kernel:Running kernel on smithi088: 5.14.0-503.el9.x86_64 2024-09-16T01:23:52.748 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum install -y kernel 2024-09-16T01:23:55.240 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-09-16T01:23:56.431 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.2 MB 00:01 2024-09-16T01:23:57.182 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - AppStream 24 MB/s | 20 MB 00:00 2024-09-16T01:23:58.349 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - AppStream 25 MB/s | 20 MB 00:00 2024-09-16T01:24:01.457 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-16T01:24:02.841 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - Extras packages 103 kB/s | 19 kB 00:00 2024-09-16T01:24:03.729 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-16T01:24:03.958 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.5 MB 00:01 2024-09-16T01:24:05.442 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 19 kB 00:00 2024-09-16T01:24:06.229 INFO:teuthology.orchestra.run.smithi154.stdout:Extra Packages for Enterprise Linux 34 MB/s | 23 MB 00:00 2024-09-16T01:24:09.176 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-16T01:24:11.182 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T01:24:11.182 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T01:24:11.290 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-09-16T01:24:11.301 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-09-16T01:24:11.301 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-09-16T01:24:11.301 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-09-16T01:24:11.301 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-09-16T01:24:11.301 INFO:teuthology.orchestra.run.smithi088.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T01:24:11.302 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-09-16T01:24:11.302 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T01:24:11.302 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T01:24:11.302 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T01:24:11.302 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:24:11.302 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-09-16T01:24:11.302 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-09-16T01:24:11.302 INFO:teuthology.orchestra.run.smithi088.stdout:Install 4 Packages 2024-09-16T01:24:11.302 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:24:11.303 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 79 M 2024-09-16T01:24:11.303 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 126 M 2024-09-16T01:24:11.303 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-09-16T01:24:11.418 INFO:teuthology.orchestra.run.smithi088.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 238 kB/s | 23 kB 00:00 2024-09-16T01:24:11.743 INFO:teuthology.orchestra.run.smithi154.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-09-16T01:24:11.802 INFO:teuthology.orchestra.run.smithi088.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-09-16T01:24:12.127 INFO:teuthology.orchestra.run.smithi088.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 43 MB/s | 35 MB 00:00 2024-09-16T01:24:12.428 INFO:teuthology.orchestra.run.smithi088.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 28 MB/s | 29 MB 00:01 2024-09-16T01:24:12.428 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-09-16T01:24:12.428 INFO:teuthology.orchestra.run.smithi088.stdout:Total 70 MB/s | 79 MB 00:01 2024-09-16T01:24:13.116 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-09-16T01:24:13.481 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-09-16T01:24:13.481 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-09-16T01:24:13.766 INFO:teuthology.orchestra.run.smithi154.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T01:24:13.767 INFO:teuthology.orchestra.run.smithi154.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T01:24:13.861 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-16T01:24:13.873 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-16T01:24:13.873 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-09-16T01:24:13.873 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-16T01:24:13.873 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-16T01:24:13.873 INFO:teuthology.orchestra.run.smithi154.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T01:24:13.873 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-16T01:24:13.873 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T01:24:13.874 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T01:24:13.874 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T01:24:13.874 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:24:13.874 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-16T01:24:13.874 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-16T01:24:13.874 INFO:teuthology.orchestra.run.smithi154.stdout:Install 4 Packages 2024-09-16T01:24:13.874 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:24:13.875 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 79 M 2024-09-16T01:24:13.875 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 126 M 2024-09-16T01:24:13.875 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-16T01:24:13.986 INFO:teuthology.orchestra.run.smithi154.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 250 kB/s | 23 kB 00:00 2024-09-16T01:24:14.466 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-09-16T01:24:14.466 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-09-16T01:24:16.932 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-09-16T01:24:17.111 INFO:teuthology.orchestra.run.smithi154.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 4.9 MB/s | 16 MB 00:03 2024-09-16T01:24:17.621 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T01:24:17.713 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T01:24:18.320 INFO:teuthology.orchestra.run.smithi154.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 6.6 MB/s | 29 MB 00:04 2024-09-16T01:24:19.295 INFO:teuthology.orchestra.run.smithi154.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 6.4 MB/s | 35 MB 00:05 2024-09-16T01:24:19.295 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-16T01:24:19.295 INFO:teuthology.orchestra.run.smithi154.stdout:Total 15 MB/s | 79 MB 00:05 2024-09-16T01:24:19.974 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-16T01:24:20.325 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-16T01:24:20.325 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-16T01:24:20.505 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T01:24:20.641 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T01:24:21.289 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-16T01:24:21.289 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-16T01:24:23.710 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-16T01:24:24.377 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T01:24:24.463 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T01:24:25.449 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T01:24:25.557 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T01:24:27.274 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T01:24:27.409 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T01:24:30.399 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T01:24:32.225 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T01:24:32.343 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T01:24:37.173 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T01:25:08.958 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T01:25:09.007 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T01:25:09.379 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T01:25:09.379 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T01:25:09.379 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T01:25:09.379 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T01:25:10.187 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T01:25:10.187 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:25:10.187 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-09-16T01:25:10.188 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T01:25:10.188 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T01:25:10.188 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T01:25:10.188 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T01:25:10.188 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:25:10.188 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-09-16T01:25:10.527 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T01:25:10.609 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T01:25:10.610 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T01:25:10.611 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T01:25:10.611 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T01:25:10.611 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T01:25:10.611 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-09-16T01:25:10.611 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi088.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T01:25:10.611 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T01:25:10.669 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T01:25:10.670 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T01:25:10.670 DEBUG:teuthology.orchestra.run.smithi088:> sudo rpm -qi grub2-tools 2024-09-16T01:25:10.701 INFO:teuthology.orchestra.run.smithi088.stdout:Name : grub2-tools 2024-09-16T01:25:10.701 INFO:teuthology.orchestra.run.smithi088.stdout:Epoch : 1 2024-09-16T01:25:10.701 INFO:teuthology.orchestra.run.smithi088.stdout:Version : 2.06 2024-09-16T01:25:10.701 INFO:teuthology.orchestra.run.smithi088.stdout:Release : 82.el9 2024-09-16T01:25:10.702 INFO:teuthology.orchestra.run.smithi088.stdout:Architecture: x86_64 2024-09-16T01:25:10.702 INFO:teuthology.orchestra.run.smithi088.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T01:25:10.702 INFO:teuthology.orchestra.run.smithi088.stdout:Group : Unspecified 2024-09-16T01:25:10.702 INFO:teuthology.orchestra.run.smithi088.stdout:Size : 8274670 2024-09-16T01:25:10.702 INFO:teuthology.orchestra.run.smithi088.stdout:License : GPLv3+ 2024-09-16T01:25:10.702 INFO:teuthology.orchestra.run.smithi088.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T01:25:10.702 INFO:teuthology.orchestra.run.smithi088.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T01:25:10.702 INFO:teuthology.orchestra.run.smithi088.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T01:25:10.702 INFO:teuthology.orchestra.run.smithi088.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T01:25:10.703 INFO:teuthology.orchestra.run.smithi088.stdout:Packager : builder@centos.org 2024-09-16T01:25:10.703 INFO:teuthology.orchestra.run.smithi088.stdout:Vendor : CentOS 2024-09-16T01:25:10.703 INFO:teuthology.orchestra.run.smithi088.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T01:25:10.703 INFO:teuthology.orchestra.run.smithi088.stdout:Summary : Support tools for GRUB. 2024-09-16T01:25:10.703 INFO:teuthology.orchestra.run.smithi088.stdout:Description : 2024-09-16T01:25:10.703 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:25:10.703 INFO:teuthology.orchestra.run.smithi088.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T01:25:10.703 INFO:teuthology.orchestra.run.smithi088.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T01:25:10.703 INFO:teuthology.orchestra.run.smithi088.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T01:25:10.703 INFO:teuthology.orchestra.run.smithi088.stdout:hardware devices. 2024-09-16T01:25:10.703 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:25:10.703 INFO:teuthology.orchestra.run.smithi088.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T01:25:10.704 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T01:25:10.704 INFO:teuthology.task.kernel:Updating grub on smithi088 to boot 5.14.0-505.el9.x86_64 2024-09-16T01:25:10.705 DEBUG:teuthology.orchestra.run.smithi088:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T01:25:11.370 INFO:teuthology.orchestra.run.smithi088.stderr:Generating grub configuration file ... 2024-09-16T01:25:12.793 INFO:teuthology.orchestra.run.smithi088.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T01:25:12.826 INFO:teuthology.orchestra.run.smithi088.stderr:done 2024-09-16T01:25:12.829 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2024-09-16T01:25:12.885 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.QstMXtcPth 2024-09-16T01:25:12.885 DEBUG:teuthology.orchestra.run.smithi088:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.QstMXtcPth 2024-09-16T01:25:12.952 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.QstMXtcPth 2024-09-16T01:25:13.091 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.QstMXtcPth is 6KB 2024-09-16T01:25:13.140 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.QstMXtcPth 2024-09-16T01:25:13.155 DEBUG:teuthology.orchestra.run.smithi088:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T01:25:13.234 INFO:teuthology.orchestra.run.smithi088.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T01:25:13.235 INFO:teuthology.orchestra.run.smithi088.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T01:25:13.235 INFO:teuthology.orchestra.run.smithi088.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T01:25:13.235 INFO:teuthology.orchestra.run.smithi088.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T01:25:13.236 DEBUG:teuthology.orchestra.run.smithi088:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T01:25:13.381 DEBUG:teuthology.orchestra.run.smithi088:> sudo shutdown -r now 2024-09-16T01:25:15.510 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T01:25:15.555 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T01:25:15.915 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T01:25:15.915 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T01:25:15.915 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T01:25:15.915 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T01:25:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T01:25:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:25:16.763 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-16T01:25:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T01:25:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T01:25:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T01:25:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T01:25:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:25:16.763 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-16T01:25:17.118 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T01:25:17.205 INFO:teuthology.orchestra.run.smithi154.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T01:25:17.205 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T01:25:17.205 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T01:25:17.205 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T01:25:17.205 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T01:25:17.205 INFO:teuthology.task.kernel:Installing distro kernel on client.1... 2024-09-16T01:25:17.205 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi154.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T01:25:17.206 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T01:25:17.264 INFO:teuthology.orchestra.run.smithi154.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T01:25:17.264 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T01:25:17.264 DEBUG:teuthology.orchestra.run.smithi154:> sudo rpm -qi grub2-tools 2024-09-16T01:25:17.297 INFO:teuthology.orchestra.run.smithi154.stdout:Name : grub2-tools 2024-09-16T01:25:17.297 INFO:teuthology.orchestra.run.smithi154.stdout:Epoch : 1 2024-09-16T01:25:17.297 INFO:teuthology.orchestra.run.smithi154.stdout:Version : 2.06 2024-09-16T01:25:17.297 INFO:teuthology.orchestra.run.smithi154.stdout:Release : 82.el9 2024-09-16T01:25:17.297 INFO:teuthology.orchestra.run.smithi154.stdout:Architecture: x86_64 2024-09-16T01:25:17.297 INFO:teuthology.orchestra.run.smithi154.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T01:25:17.297 INFO:teuthology.orchestra.run.smithi154.stdout:Group : Unspecified 2024-09-16T01:25:17.297 INFO:teuthology.orchestra.run.smithi154.stdout:Size : 8274670 2024-09-16T01:25:17.297 INFO:teuthology.orchestra.run.smithi154.stdout:License : GPLv3+ 2024-09-16T01:25:17.297 INFO:teuthology.orchestra.run.smithi154.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T01:25:17.297 INFO:teuthology.orchestra.run.smithi154.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T01:25:17.298 INFO:teuthology.orchestra.run.smithi154.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T01:25:17.298 INFO:teuthology.orchestra.run.smithi154.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T01:25:17.298 INFO:teuthology.orchestra.run.smithi154.stdout:Packager : builder@centos.org 2024-09-16T01:25:17.298 INFO:teuthology.orchestra.run.smithi154.stdout:Vendor : CentOS 2024-09-16T01:25:17.298 INFO:teuthology.orchestra.run.smithi154.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T01:25:17.298 INFO:teuthology.orchestra.run.smithi154.stdout:Summary : Support tools for GRUB. 2024-09-16T01:25:17.299 INFO:teuthology.orchestra.run.smithi154.stdout:Description : 2024-09-16T01:25:17.299 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:25:17.299 INFO:teuthology.orchestra.run.smithi154.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T01:25:17.299 INFO:teuthology.orchestra.run.smithi154.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T01:25:17.299 INFO:teuthology.orchestra.run.smithi154.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T01:25:17.299 INFO:teuthology.orchestra.run.smithi154.stdout:hardware devices. 2024-09-16T01:25:17.299 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:25:17.299 INFO:teuthology.orchestra.run.smithi154.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T01:25:17.300 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T01:25:17.300 INFO:teuthology.task.kernel:Updating grub on smithi154 to boot 5.14.0-505.el9.x86_64 2024-09-16T01:25:17.300 DEBUG:teuthology.orchestra.run.smithi154:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T01:25:18.000 INFO:teuthology.orchestra.run.smithi154.stderr:Generating grub configuration file ... 2024-09-16T01:25:19.381 INFO:teuthology.orchestra.run.smithi154.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T01:25:19.400 INFO:teuthology.orchestra.run.smithi154.stderr:done 2024-09-16T01:25:19.402 DEBUG:teuthology.orchestra.run.smithi154:> mktemp 2024-09-16T01:25:19.418 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.UfxJGnITIK 2024-09-16T01:25:19.418 DEBUG:teuthology.orchestra.run.smithi154:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.UfxJGnITIK 2024-09-16T01:25:19.485 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.UfxJGnITIK 2024-09-16T01:25:19.629 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.UfxJGnITIK is 6KB 2024-09-16T01:25:19.650 DEBUG:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.UfxJGnITIK 2024-09-16T01:25:19.665 DEBUG:teuthology.orchestra.run.smithi154:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T01:25:19.728 INFO:teuthology.orchestra.run.smithi154.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T01:25:19.728 INFO:teuthology.orchestra.run.smithi154.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T01:25:19.728 INFO:teuthology.orchestra.run.smithi154.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T01:25:19.728 INFO:teuthology.orchestra.run.smithi154.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T01:25:19.729 DEBUG:teuthology.orchestra.run.smithi154:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T01:25:19.911 DEBUG:teuthology.orchestra.run.smithi154:> sudo shutdown -r now 2024-09-16T01:25:43.407 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-09-16T01:25:43.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-09-16T01:25:43.408 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T01:25:49.921 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-09-16T01:25:49.921 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-16T01:25:49.921 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T01:26:01.790 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-09-16T01:26:08.449 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-09-16T01:26:10.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-09-16T01:26:10.792 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T01:26:13.857 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-09-16T01:26:17.453 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-16T01:26:17.453 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T01:26:20.509 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-09-16T01:26:25.861 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-09-16T01:26:25.861 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T01:26:28.925 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-09-16T01:26:32.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-16T01:26:32.513 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T01:26:43.937 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-09-16T01:26:43.937 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T01:27:15.788 DEBUG:teuthology.orchestra.run.smithi088:> true 2024-09-16T01:27:16.115 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-09-16T01:27:16.116 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-505.el9.x86_64"... 2024-09-16T01:27:16.116 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-09-16T01:27:16.132 INFO:teuthology.orchestra.run.smithi088.stdout:5.14.0-505.el9.x86_64 2024-09-16T01:27:16.132 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T01:27:16.132 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T01:27:16.132 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T01:27:17.134 DEBUG:teuthology.parallel:result is None 2024-09-16T01:27:32.515 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T01:27:47.516 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-16T01:27:47.517 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T01:27:47.872 DEBUG:teuthology.orchestra.run.smithi154:> true 2024-09-16T01:27:48.255 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-16T01:27:48.255 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "5.14.0-505.el9.x86_64"... 2024-09-16T01:27:48.256 DEBUG:teuthology.orchestra.run.smithi154:> uname -r 2024-09-16T01:27:48.272 INFO:teuthology.orchestra.run.smithi154.stdout:5.14.0-505.el9.x86_64 2024-09-16T01:27:48.272 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T01:27:48.272 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T01:27:48.272 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T01:27:49.273 DEBUG:teuthology.parallel:result is None 2024-09-16T01:27:49.274 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T01:27:49.281 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T01:27:49.281 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T01:27:49.284 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T01:27:49.302 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T01:27:49.308 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T01:27:49.317 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T01:27:49.317 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T01:27:49.343 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T01:27:49.662 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T01:27:49.669 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T01:27:49.669 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T01:27:49.672 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T01:27:49.748 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T01:27:49.759 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T01:27:49.774 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T01:27:49.789 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T01:27:49.792 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T01:27:49.799 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T01:27:49.799 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T01:27:49.801 DEBUG:teuthology.orchestra.run.smithi154:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T01:27:49.830 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T01:27:49.838 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T01:27:49.839 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T01:27:49.871 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T01:27:49.887 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T01:27:49.968 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T01:27:50.026 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T01:27:50.110 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T01:27:50.151 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:27:50.151 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T01:27:50.216 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T01:27:50.261 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T01:27:50.315 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T01:27:50.397 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T01:27:50.441 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:27:50.442 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T01:27:50.508 DEBUG:teuthology.orchestra.run.smithi088:> sudo service rsyslog restart 2024-09-16T01:27:50.511 DEBUG:teuthology.orchestra.run.smithi154:> sudo service rsyslog restart 2024-09-16T01:27:50.553 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T01:27:50.594 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T01:27:50.992 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T01:27:51.000 INFO:teuthology.task.internal:Starting timer... 2024-09-16T01:27:51.000 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T01:27:51.010 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T01:27:51.019 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-09-16T01:27:51.019 DEBUG:teuthology.orchestra.run.smithi088:> sudo service auditd rotate 2024-09-16T01:27:51.102 INFO:teuthology.orchestra.run.smithi088.stdout:Rotating logs: 2024-09-16T01:27:51.104 DEBUG:teuthology.orchestra.run.smithi154:> sudo service auditd rotate 2024-09-16T01:27:51.174 INFO:teuthology.orchestra.run.smithi154.stdout:Rotating logs: 2024-09-16T01:27:51.176 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T01:27:51.176 DEBUG:teuthology.orchestra.run.smithi088:> /usr/sbin/getenforce 2024-09-16T01:27:51.201 INFO:teuthology.orchestra.run.smithi088.stdout:Permissive 2024-09-16T01:27:51.201 DEBUG:teuthology.orchestra.run.smithi154:> /usr/sbin/getenforce 2024-09-16T01:27:51.225 INFO:teuthology.orchestra.run.smithi154.stdout:Permissive 2024-09-16T01:27:51.225 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi088.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi154.front.sepia.ceph.com': 'permissive'} 2024-09-16T01:27:51.226 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T01:27:51.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:27:51.268 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T01:27:51.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:27:51.295 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T01:27:51.295 DEBUG:teuthology.orchestra.run.smithi088:> sudo /usr/sbin/setenforce permissive 2024-09-16T01:27:51.348 DEBUG:teuthology.orchestra.run.smithi154:> sudo /usr/sbin/setenforce permissive 2024-09-16T01:27:51.380 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T01:27:51.390 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-16T01:27:51.393 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-16T01:27:51.505 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T01:27:51.515 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T01:27:51.516 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi088.front.sepia.ceph.com,smithi154.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T01:31:34.992 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi088.front.sepia.ceph.com'), Remote(name='ubuntu@smithi154.front.sepia.ceph.com')] 2024-09-16T01:31:34.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-09-16T01:31:34.994 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T01:31:35.075 DEBUG:teuthology.orchestra.run.smithi088:> true 2024-09-16T01:31:35.154 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-09-16T01:31:35.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-16T01:31:35.155 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T01:31:35.237 DEBUG:teuthology.orchestra.run.smithi154:> true 2024-09-16T01:31:35.316 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-16T01:31:35.316 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T01:31:35.326 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T01:31:35.326 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T01:31:35.327 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T01:31:35.330 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T01:31:35.330 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T01:31:35.359 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T01:31:35.374 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T01:31:35.385 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T01:31:35.398 INFO:teuthology.orchestra.run.smithi088.stderr:sudo: ntpd: command not found 2024-09-16T01:31:35.400 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T01:31:35.410 INFO:teuthology.orchestra.run.smithi088.stdout:506 Cannot talk to daemon 2024-09-16T01:31:35.424 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: ntpd: command not found 2024-09-16T01:31:35.424 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T01:31:35.435 INFO:teuthology.orchestra.run.smithi154.stdout:506 Cannot talk to daemon 2024-09-16T01:31:35.438 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T01:31:35.448 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T01:31:35.463 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T01:31:35.506 INFO:teuthology.orchestra.run.smithi088.stderr:bash: line 1: ntpq: command not found 2024-09-16T01:31:35.510 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T01:31:35.510 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2024-09-16T01:31:35.510 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:31:35.511 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:31:35.511 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:31:35.511 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:31:35.519 INFO:teuthology.orchestra.run.smithi154.stderr:bash: line 1: ntpq: command not found 2024-09-16T01:31:35.522 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T01:31:35.523 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2024-09-16T01:31:35.523 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:31:35.523 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:31:35.523 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:31:35.523 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:31:35.524 INFO:teuthology.run_tasks:Running task install... 2024-09-16T01:31:35.532 DEBUG:teuthology.task.install:project ceph 2024-09-16T01:31:35.532 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}} 2024-09-16T01:31:35.532 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'} 2024-09-16T01:31:35.532 INFO:teuthology.task.install:Using flavor: default 2024-09-16T01:31:35.541 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T01:31:35.541 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-09-16T01:31:35.542 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T01:31:35.542 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T01:31:35.544 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-16T01:31:35.544 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T01:31:35.689 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-16T01:31:35.689 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-16T01:31:35.693 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-16T01:31:35.693 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-16T01:31:35.797 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T01:31:35.797 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:31:35.797 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T01:31:35.805 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T01:31:35.805 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:31:35.805 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T01:31:35.823 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-16T01:31:35.823 DEBUG:teuthology.orchestra.run.smithi088:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T01:31:35.831 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-16T01:31:35.831 DEBUG:teuthology.orchestra.run.smithi154:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T01:31:35.888 DEBUG:teuthology.orchestra.run.smithi088:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T01:31:35.896 DEBUG:teuthology.orchestra.run.smithi154:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T01:31:35.964 DEBUG:teuthology.orchestra.run.smithi088:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T01:31:35.972 DEBUG:teuthology.orchestra.run.smithi154:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T01:31:36.028 INFO:teuthology.orchestra.run.smithi088.stdout:check_obsoletes = 1 2024-09-16T01:31:36.029 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum clean all 2024-09-16T01:31:36.036 INFO:teuthology.orchestra.run.smithi154.stdout:check_obsoletes = 1 2024-09-16T01:31:36.037 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum clean all 2024-09-16T01:31:36.262 INFO:teuthology.orchestra.run.smithi088.stdout:45 files removed 2024-09-16T01:31:36.265 INFO:teuthology.orchestra.run.smithi154.stdout:45 files removed 2024-09-16T01:31:36.290 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-16T01:31:36.293 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-16T01:31:36.855 INFO:teuthology.orchestra.run.smithi154.stdout:ceph packages for x86_64 307 kB/s | 83 kB 00:00 2024-09-16T01:31:36.872 INFO:teuthology.orchestra.run.smithi088.stdout:ceph packages for x86_64 279 kB/s | 83 kB 00:00 2024-09-16T01:31:37.139 INFO:teuthology.orchestra.run.smithi088.stdout:ceph noarch packages 53 kB/s | 12 kB 00:00 2024-09-16T01:31:37.139 INFO:teuthology.orchestra.run.smithi154.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-16T01:31:37.398 INFO:teuthology.orchestra.run.smithi088.stdout:ceph source packages 8.5 kB/s | 1.9 kB 00:00 2024-09-16T01:31:37.406 INFO:teuthology.orchestra.run.smithi154.stdout:ceph source packages 8.2 kB/s | 1.9 kB 00:00 2024-09-16T01:31:38.090 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-16T01:31:40.349 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - AppStream 18 MB/s | 20 MB 00:01 2024-09-16T01:31:40.389 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 8.2 MB 00:02 2024-09-16T01:31:43.440 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - AppStream 11 MB/s | 20 MB 00:01 2024-09-16T01:31:45.733 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.5 MB 00:01 2024-09-16T01:31:47.142 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - Extras packages 95 kB/s | 19 kB 00:00 2024-09-16T01:31:48.321 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 21 MB/s | 23 MB 00:01 2024-09-16T01:31:49.682 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.5 MB 00:02 2024-09-16T01:31:51.441 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 19 kB 00:00 2024-09-16T01:31:52.404 INFO:teuthology.orchestra.run.smithi154.stdout:Extra Packages for Enterprise Linux 27 MB/s | 23 MB 00:00 2024-09-16T01:31:53.751 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-09-16T01:31:56.112 INFO:teuthology.orchestra.run.smithi088.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-16T01:31:56.113 INFO:teuthology.orchestra.run.smithi088.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-16T01:31:56.126 INFO:teuthology.orchestra.run.smithi088.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-16T01:31:56.127 INFO:teuthology.orchestra.run.smithi088.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-16T01:31:56.128 INFO:teuthology.orchestra.run.smithi088.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-16T01:31:56.129 INFO:teuthology.orchestra.run.smithi088.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-16T01:31:56.131 INFO:teuthology.orchestra.run.smithi088.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-16T01:31:56.133 INFO:teuthology.orchestra.run.smithi088.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-16T01:31:56.134 INFO:teuthology.orchestra.run.smithi088.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-16T01:31:56.196 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-09-16T01:31:56.217 INFO:teuthology.orchestra.run.smithi088.stdout:====================================================================================== 2024-09-16T01:31:56.217 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-09-16T01:31:56.217 INFO:teuthology.orchestra.run.smithi088.stdout:====================================================================================== 2024-09-16T01:31:56.217 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-09-16T01:31:56.217 INFO:teuthology.orchestra.run.smithi088.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-16T01:31:56.217 INFO:teuthology.orchestra.run.smithi088.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-16T01:31:56.217 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-16T01:31:56.217 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-16T01:31:56.218 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-16T01:31:56.218 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-16T01:31:56.218 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-16T01:31:56.218 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-16T01:31:56.218 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-16T01:31:56.218 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-16T01:31:56.218 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-16T01:31:56.218 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-16T01:31:56.218 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-16T01:31:56.218 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-16T01:31:56.218 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 228 k 2024-09-16T01:31:56.218 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-shell x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 27 k 2024-09-16T01:31:56.219 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-top noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 18 k 2024-09-16T01:31:56.219 INFO:teuthology.orchestra.run.smithi088.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-16T01:31:56.219 INFO:teuthology.orchestra.run.smithi088.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-16T01:31:56.219 INFO:teuthology.orchestra.run.smithi088.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-16T01:31:56.219 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-16T01:31:56.219 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-16T01:31:56.219 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-16T01:31:56.219 INFO:teuthology.orchestra.run.smithi088.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-16T01:31:56.219 INFO:teuthology.orchestra.run.smithi088.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-16T01:31:56.219 INFO:teuthology.orchestra.run.smithi088.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-16T01:31:56.219 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-16T01:31:56.219 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-16T01:31:56.219 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-16T01:31:56.220 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-16T01:31:56.220 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-16T01:31:56.220 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-16T01:31:56.220 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-16T01:31:56.220 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-16T01:31:56.220 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-16T01:31:56.220 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-16T01:31:56.220 INFO:teuthology.orchestra.run.smithi088.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-16T01:31:56.220 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrading: 2024-09-16T01:31:56.220 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-16T01:31:56.220 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-16T01:31:56.221 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-16T01:31:56.221 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-16T01:31:56.221 INFO:teuthology.orchestra.run.smithi088.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-16T01:31:56.221 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-16T01:31:56.221 INFO:teuthology.orchestra.run.smithi088.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-16T01:31:56.221 INFO:teuthology.orchestra.run.smithi088.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-16T01:31:56.221 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-09-16T01:31:56.221 INFO:teuthology.orchestra.run.smithi088.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T01:31:56.221 INFO:teuthology.orchestra.run.smithi088.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-16T01:31:56.221 INFO:teuthology.orchestra.run.smithi088.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T01:31:56.221 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-16T01:31:56.221 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-16T01:31:56.221 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-16T01:31:56.222 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-16T01:31:56.222 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-16T01:31:56.222 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-16T01:31:56.222 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-16T01:31:56.222 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-16T01:31:56.222 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T01:31:56.222 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T01:31:56.222 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T01:31:56.222 INFO:teuthology.orchestra.run.smithi088.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T01:31:56.222 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T01:31:56.222 INFO:teuthology.orchestra.run.smithi088.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T01:31:56.222 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T01:31:56.222 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T01:31:56.223 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-16T01:31:56.223 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T01:31:56.223 INFO:teuthology.orchestra.run.smithi088.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T01:31:56.223 INFO:teuthology.orchestra.run.smithi088.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T01:31:56.223 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T01:31:56.223 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T01:31:56.223 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-16T01:31:56.223 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T01:31:56.223 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-16T01:31:56.223 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T01:31:56.223 INFO:teuthology.orchestra.run.smithi088.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-16T01:31:56.223 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T01:31:56.224 INFO:teuthology.orchestra.run.smithi088.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T01:31:56.224 INFO:teuthology.orchestra.run.smithi088.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T01:31:56.224 INFO:teuthology.orchestra.run.smithi088.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T01:31:56.224 INFO:teuthology.orchestra.run.smithi088.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T01:31:56.224 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T01:31:56.224 INFO:teuthology.orchestra.run.smithi088.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T01:31:56.224 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-16T01:31:56.224 INFO:teuthology.orchestra.run.smithi088.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T01:31:56.224 INFO:teuthology.orchestra.run.smithi088.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T01:31:56.224 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T01:31:56.224 INFO:teuthology.orchestra.run.smithi088.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-16T01:31:56.224 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T01:31:56.225 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T01:31:56.225 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T01:31:56.225 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-16T01:31:56.225 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-16T01:31:56.225 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T01:31:56.225 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T01:31:56.225 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T01:31:56.225 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T01:31:56.225 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-16T01:31:56.226 INFO:teuthology.orchestra.run.smithi088.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-16T01:31:56.226 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T01:31:56.226 INFO:teuthology.orchestra.run.smithi088.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T01:31:56.226 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T01:31:56.226 INFO:teuthology.orchestra.run.smithi088.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T01:31:56.226 INFO:teuthology.orchestra.run.smithi088.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T01:31:56.226 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T01:31:56.226 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T01:31:56.226 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T01:31:56.226 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T01:31:56.226 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T01:31:56.226 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T01:31:56.227 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T01:31:56.227 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T01:31:56.227 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T01:31:56.227 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T01:31:56.227 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T01:31:56.227 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T01:31:56.227 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T01:31:56.227 INFO:teuthology.orchestra.run.smithi088.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T01:31:56.227 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T01:31:56.227 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T01:31:56.227 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T01:31:56.227 INFO:teuthology.orchestra.run.smithi088.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T01:31:56.227 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T01:31:56.228 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T01:31:56.228 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T01:31:56.228 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T01:31:56.228 INFO:teuthology.orchestra.run.smithi088.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T01:31:56.228 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T01:31:56.228 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T01:31:56.228 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T01:31:56.228 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T01:31:56.228 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-16T01:31:56.229 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T01:31:56.229 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T01:31:56.229 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T01:31:56.229 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T01:31:56.229 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T01:31:56.229 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T01:31:56.229 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T01:31:56.229 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T01:31:56.229 INFO:teuthology.orchestra.run.smithi088.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T01:31:56.229 INFO:teuthology.orchestra.run.smithi088.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T01:31:56.229 INFO:teuthology.orchestra.run.smithi088.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T01:31:56.229 INFO:teuthology.orchestra.run.smithi088.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-16T01:31:56.229 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T01:31:56.230 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T01:31:56.230 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T01:31:56.230 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T01:31:56.230 INFO:teuthology.orchestra.run.smithi088.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T01:31:56.230 INFO:teuthology.orchestra.run.smithi088.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T01:31:56.230 INFO:teuthology.orchestra.run.smithi088.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T01:31:56.230 INFO:teuthology.orchestra.run.smithi088.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T01:31:56.230 INFO:teuthology.orchestra.run.smithi088.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T01:31:56.230 INFO:teuthology.orchestra.run.smithi088.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-16T01:31:56.230 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T01:31:56.230 INFO:teuthology.orchestra.run.smithi088.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-16T01:31:56.230 INFO:teuthology.orchestra.run.smithi088.stdout:Installing weak dependencies: 2024-09-16T01:31:56.230 INFO:teuthology.orchestra.run.smithi088.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T01:31:56.230 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:31:56.231 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-09-16T01:31:56.231 INFO:teuthology.orchestra.run.smithi088.stdout:====================================================================================== 2024-09-16T01:31:56.231 INFO:teuthology.orchestra.run.smithi088.stdout:Install 148 Packages 2024-09-16T01:31:56.231 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrade 8 Packages 2024-09-16T01:31:56.231 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:31:56.231 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 226 M 2024-09-16T01:31:56.232 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-09-16T01:31:57.253 INFO:teuthology.orchestra.run.smithi088.stdout:(1/156): ceph-19.1.1-326.g8921c8ef.el9.x86_64.r 24 kB/s | 6.5 kB 00:00 2024-09-16T01:31:57.661 INFO:teuthology.orchestra.run.smithi088.stdout:(2/156): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86 2.1 MB/s | 896 kB 00:00 2024-09-16T01:31:57.745 INFO:teuthology.orchestra.run.smithi088.stdout:(3/156): ceph-immutable-object-cache-19.1.1-326 1.7 MB/s | 145 kB 00:00 2024-09-16T01:31:57.926 INFO:teuthology.orchestra.run.smithi154.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-09-16T01:31:58.278 INFO:teuthology.orchestra.run.smithi088.stdout:(4/156): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_ 4.5 MB/s | 2.4 MB 00:00 2024-09-16T01:31:58.403 INFO:teuthology.orchestra.run.smithi088.stdout:(5/156): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_ 12 MB/s | 1.5 MB 00:00 2024-09-16T01:31:58.745 INFO:teuthology.orchestra.run.smithi088.stdout:(6/156): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_ 14 MB/s | 4.7 MB 00:00 2024-09-16T01:31:58.887 INFO:teuthology.orchestra.run.smithi088.stdout:(7/156): ceph-base-19.1.1-326.g8921c8ef.el9.x86 2.8 MB/s | 5.4 MB 00:01 2024-09-16T01:31:59.787 INFO:teuthology.orchestra.run.smithi088.stdout:(8/156): ceph-osd-19.1.1-326.g8921c8ef.el9.x86_ 16 MB/s | 17 MB 00:01 2024-09-16T01:31:59.913 INFO:teuthology.orchestra.run.smithi088.stdout:(9/156): ceph-radosgw-19.1.1-326.g8921c8ef.el9. 10 MB/s | 11 MB 00:01 2024-09-16T01:31:59.938 INFO:teuthology.orchestra.run.smithi088.stdout:(10/156): ceph-selinux-19.1.1-326.g8921c8ef.el9 168 kB/s | 25 kB 00:00 2024-09-16T01:32:00.155 INFO:teuthology.orchestra.run.smithi088.stdout:(11/156): ceph-common-19.1.1-326.g8921c8ef.el9. 6.7 MB/s | 21 MB 00:03 2024-09-16T01:32:00.180 INFO:teuthology.orchestra.run.smithi088.stdout:(12/156): cephfs-mirror-19.1.1-326.g8921c8ef.el 943 kB/s | 228 kB 00:00 2024-09-16T01:32:00.255 INFO:teuthology.orchestra.run.smithi088.stdout:(13/156): cephfs-shell-19.1.1-326.g8921c8ef.el9 270 kB/s | 27 kB 00:00 2024-09-16T01:32:00.271 INFO:teuthology.orchestra.run.smithi154.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-16T01:32:00.271 INFO:teuthology.orchestra.run.smithi154.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-16T01:32:00.280 INFO:teuthology.orchestra.run.smithi088.stdout:(14/156): libcephfs-devel-19.1.1-326.g8921c8ef. 317 kB/s | 32 kB 00:00 2024-09-16T01:32:00.285 INFO:teuthology.orchestra.run.smithi154.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-16T01:32:00.286 INFO:teuthology.orchestra.run.smithi154.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-16T01:32:00.287 INFO:teuthology.orchestra.run.smithi154.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-16T01:32:00.288 INFO:teuthology.orchestra.run.smithi154.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-16T01:32:00.290 INFO:teuthology.orchestra.run.smithi154.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-16T01:32:00.292 INFO:teuthology.orchestra.run.smithi154.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-16T01:32:00.294 INFO:teuthology.orchestra.run.smithi154.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-16T01:32:00.358 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-16T01:32:00.372 INFO:teuthology.orchestra.run.smithi088.stdout:(15/156): libcephsqlite-19.1.1-326.g8921c8ef.el 1.7 MB/s | 163 kB 00:00 2024-09-16T01:32:00.379 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================== 2024-09-16T01:32:00.379 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-16T01:32:00.379 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================== 2024-09-16T01:32:00.379 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-16T01:32:00.379 INFO:teuthology.orchestra.run.smithi154.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-16T01:32:00.379 INFO:teuthology.orchestra.run.smithi154.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-16T01:32:00.379 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-16T01:32:00.379 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-16T01:32:00.379 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-16T01:32:00.379 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-16T01:32:00.380 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-16T01:32:00.380 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-16T01:32:00.380 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-16T01:32:00.380 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-16T01:32:00.380 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-16T01:32:00.380 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-16T01:32:00.380 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-16T01:32:00.380 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-16T01:32:00.380 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 228 k 2024-09-16T01:32:00.380 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-shell x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 27 k 2024-09-16T01:32:00.380 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-top noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 18 k 2024-09-16T01:32:00.380 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-16T01:32:00.381 INFO:teuthology.orchestra.run.smithi154.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-16T01:32:00.381 INFO:teuthology.orchestra.run.smithi154.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-16T01:32:00.381 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-16T01:32:00.381 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-16T01:32:00.381 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-16T01:32:00.381 INFO:teuthology.orchestra.run.smithi154.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-16T01:32:00.381 INFO:teuthology.orchestra.run.smithi154.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-16T01:32:00.381 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-16T01:32:00.381 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-16T01:32:00.381 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-16T01:32:00.381 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-16T01:32:00.381 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-16T01:32:00.382 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-16T01:32:00.382 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-16T01:32:00.382 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-16T01:32:00.382 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-16T01:32:00.382 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-16T01:32:00.382 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-16T01:32:00.382 INFO:teuthology.orchestra.run.smithi154.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-16T01:32:00.382 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrading: 2024-09-16T01:32:00.382 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-16T01:32:00.382 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-16T01:32:00.382 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-16T01:32:00.382 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-16T01:32:00.382 INFO:teuthology.orchestra.run.smithi154.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-16T01:32:00.383 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-16T01:32:00.383 INFO:teuthology.orchestra.run.smithi154.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-16T01:32:00.383 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-16T01:32:00.383 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-16T01:32:00.383 INFO:teuthology.orchestra.run.smithi154.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T01:32:00.383 INFO:teuthology.orchestra.run.smithi154.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-16T01:32:00.383 INFO:teuthology.orchestra.run.smithi154.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T01:32:00.383 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-16T01:32:00.383 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-16T01:32:00.383 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-16T01:32:00.383 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-16T01:32:00.383 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-16T01:32:00.383 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-16T01:32:00.384 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-16T01:32:00.384 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-16T01:32:00.384 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T01:32:00.384 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T01:32:00.384 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T01:32:00.384 INFO:teuthology.orchestra.run.smithi154.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T01:32:00.384 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T01:32:00.384 INFO:teuthology.orchestra.run.smithi154.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T01:32:00.384 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T01:32:00.384 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T01:32:00.384 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-16T01:32:00.384 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T01:32:00.384 INFO:teuthology.orchestra.run.smithi154.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T01:32:00.385 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T01:32:00.385 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T01:32:00.385 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T01:32:00.385 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-16T01:32:00.385 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T01:32:00.385 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-16T01:32:00.385 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T01:32:00.385 INFO:teuthology.orchestra.run.smithi154.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-16T01:32:00.385 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T01:32:00.385 INFO:teuthology.orchestra.run.smithi154.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T01:32:00.385 INFO:teuthology.orchestra.run.smithi154.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T01:32:00.385 INFO:teuthology.orchestra.run.smithi154.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T01:32:00.385 INFO:teuthology.orchestra.run.smithi154.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T01:32:00.386 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T01:32:00.386 INFO:teuthology.orchestra.run.smithi154.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T01:32:00.386 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-16T01:32:00.386 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T01:32:00.386 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T01:32:00.386 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T01:32:00.386 INFO:teuthology.orchestra.run.smithi154.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-16T01:32:00.386 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T01:32:00.386 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T01:32:00.386 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T01:32:00.386 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-16T01:32:00.386 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-16T01:32:00.386 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T01:32:00.387 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T01:32:00.387 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T01:32:00.387 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T01:32:00.387 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-16T01:32:00.387 INFO:teuthology.orchestra.run.smithi154.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-16T01:32:00.387 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T01:32:00.387 INFO:teuthology.orchestra.run.smithi154.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T01:32:00.387 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T01:32:00.387 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T01:32:00.387 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T01:32:00.387 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T01:32:00.387 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T01:32:00.387 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T01:32:00.387 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T01:32:00.388 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T01:32:00.388 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T01:32:00.388 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T01:32:00.388 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T01:32:00.388 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T01:32:00.388 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T01:32:00.388 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T01:32:00.388 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T01:32:00.388 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T01:32:00.388 INFO:teuthology.orchestra.run.smithi154.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T01:32:00.388 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T01:32:00.388 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T01:32:00.388 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T01:32:00.389 INFO:teuthology.orchestra.run.smithi154.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T01:32:00.389 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T01:32:00.389 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T01:32:00.389 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T01:32:00.389 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T01:32:00.389 INFO:teuthology.orchestra.run.smithi154.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T01:32:00.389 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T01:32:00.389 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T01:32:00.389 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T01:32:00.389 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T01:32:00.389 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-16T01:32:00.389 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T01:32:00.389 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T01:32:00.390 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T01:32:00.390 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T01:32:00.390 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T01:32:00.390 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T01:32:00.390 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T01:32:00.390 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T01:32:00.390 INFO:teuthology.orchestra.run.smithi154.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T01:32:00.390 INFO:teuthology.orchestra.run.smithi154.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T01:32:00.391 INFO:teuthology.orchestra.run.smithi154.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T01:32:00.391 INFO:teuthology.orchestra.run.smithi154.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-16T01:32:00.391 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T01:32:00.391 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T01:32:00.391 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T01:32:00.391 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T01:32:00.391 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T01:32:00.391 INFO:teuthology.orchestra.run.smithi154.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T01:32:00.391 INFO:teuthology.orchestra.run.smithi154.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T01:32:00.391 INFO:teuthology.orchestra.run.smithi154.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T01:32:00.391 INFO:teuthology.orchestra.run.smithi154.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T01:32:00.391 INFO:teuthology.orchestra.run.smithi154.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-16T01:32:00.391 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T01:32:00.391 INFO:teuthology.orchestra.run.smithi154.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-16T01:32:00.392 INFO:teuthology.orchestra.run.smithi154.stdout:Installing weak dependencies: 2024-09-16T01:32:00.392 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T01:32:00.392 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:32:00.392 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-16T01:32:00.392 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================== 2024-09-16T01:32:00.392 INFO:teuthology.orchestra.run.smithi154.stdout:Install 148 Packages 2024-09-16T01:32:00.392 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrade 8 Packages 2024-09-16T01:32:00.392 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:32:00.393 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 226 M 2024-09-16T01:32:00.393 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-16T01:32:00.414 INFO:teuthology.orchestra.run.smithi088.stdout:(16/156): libcephfs2-19.1.1-326.g8921c8ef.el9.x 4.6 MB/s | 751 kB 00:00 2024-09-16T01:32:00.473 INFO:teuthology.orchestra.run.smithi088.stdout:(17/156): librados-devel-19.1.1-326.g8921c8ef.e 1.2 MB/s | 127 kB 00:00 2024-09-16T01:32:00.885 INFO:teuthology.orchestra.run.smithi154.stdout:(1/156): ceph-19.1.1-326.g8921c8ef.el9.x86_64.r 25 kB/s | 6.5 kB 00:00 2024-09-16T01:32:01.223 INFO:teuthology.orchestra.run.smithi088.stdout:(18/156): librgw2-19.1.1-326.g8921c8ef.el9.x86_ 7.1 MB/s | 5.3 MB 00:00 2024-09-16T01:32:01.248 INFO:teuthology.orchestra.run.smithi088.stdout:(19/156): libradosstriper1-19.1.1-326.g8921c8ef 607 kB/s | 506 kB 00:00 2024-09-16T01:32:01.291 INFO:teuthology.orchestra.run.smithi088.stdout:(20/156): python3-ceph-argparse-19.1.1-326.g892 678 kB/s | 45 kB 00:00 2024-09-16T01:32:01.324 INFO:teuthology.orchestra.run.smithi088.stdout:(21/156): python3-ceph-common-19.1.1-326.g8921c 1.8 MB/s | 136 kB 00:00 2024-09-16T01:32:01.374 INFO:teuthology.orchestra.run.smithi088.stdout:(22/156): python3-cephfs-19.1.1-326.g8921c8ef.e 1.9 MB/s | 161 kB 00:00 2024-09-16T01:32:01.400 INFO:teuthology.orchestra.run.smithi088.stdout:(23/156): python3-rados-19.1.1-326.g8921c8ef.el 4.2 MB/s | 320 kB 00:00 2024-09-16T01:32:01.450 INFO:teuthology.orchestra.run.smithi088.stdout:(24/156): python3-rbd-19.1.1-326.g8921c8ef.el9. 3.9 MB/s | 300 kB 00:00 2024-09-16T01:32:01.452 INFO:teuthology.orchestra.run.smithi154.stdout:(2/156): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86 1.5 MB/s | 896 kB 00:00 2024-09-16T01:32:01.484 INFO:teuthology.orchestra.run.smithi088.stdout:(25/156): python3-rgw-19.1.1-326.g8921c8ef.el9. 1.2 MB/s | 99 kB 00:00 2024-09-16T01:32:01.517 INFO:teuthology.orchestra.run.smithi088.stdout:(26/156): rbd-fuse-19.1.1-326.g8921c8ef.el9.x86 1.2 MB/s | 85 kB 00:00 2024-09-16T01:32:01.586 INFO:teuthology.orchestra.run.smithi154.stdout:(3/156): ceph-immutable-object-cache-19.1.1-326 1.1 MB/s | 145 kB 00:00 2024-09-16T01:32:01.601 INFO:teuthology.orchestra.run.smithi088.stdout:(27/156): rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_ 2.0 MB/s | 170 kB 00:00 2024-09-16T01:32:01.685 INFO:teuthology.orchestra.run.smithi088.stdout:(28/156): ceph-grafana-dashboards-19.1.1-326.g8 382 kB/s | 32 kB 00:00 2024-09-16T01:32:01.777 INFO:teuthology.orchestra.run.smithi088.stdout:(29/156): ceph-mgr-cephadm-19.1.1-326.g8921c8ef 1.6 MB/s | 146 kB 00:00 2024-09-16T01:32:01.885 INFO:teuthology.orchestra.run.smithi088.stdout:(30/156): rbd-mirror-19.1.1-326.g8921c8ef.el9.x 7.7 MB/s | 3.1 MB 00:00 2024-09-16T01:32:02.337 INFO:teuthology.orchestra.run.smithi088.stdout:(31/156): ceph-mgr-dashboard-19.1.1-326.g8921c8 6.4 MB/s | 3.6 MB 00:00 2024-09-16T01:32:02.436 INFO:teuthology.orchestra.run.smithi088.stdout:(32/156): ceph-mgr-modules-core-19.1.1-326.g892 2.4 MB/s | 249 kB 00:00 2024-09-16T01:32:02.544 INFO:teuthology.orchestra.run.smithi088.stdout:(33/156): ceph-mgr-rook-19.1.1-326.g8921c8ef.el 455 kB/s | 49 kB 00:00 2024-09-16T01:32:02.619 INFO:teuthology.orchestra.run.smithi154.stdout:(4/156): ceph-base-19.1.1-326.g8921c8ef.el9.x86 2.7 MB/s | 5.4 MB 00:01 2024-09-16T01:32:02.628 INFO:teuthology.orchestra.run.smithi088.stdout:(34/156): ceph-prometheus-alerts-19.1.1-326.g89 198 kB/s | 16 kB 00:00 2024-09-16T01:32:02.661 INFO:teuthology.orchestra.run.smithi154.stdout:(5/156): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_ 2.2 MB/s | 2.4 MB 00:01 2024-09-16T01:32:02.720 INFO:teuthology.orchestra.run.smithi088.stdout:(35/156): ceph-volume-19.1.1-326.g8921c8ef.el9. 3.0 MB/s | 285 kB 00:00 2024-09-16T01:32:02.820 INFO:teuthology.orchestra.run.smithi088.stdout:(36/156): ceph-mgr-diskprediction-local-19.1.1- 7.9 MB/s | 7.4 MB 00:00 2024-09-16T01:32:02.912 INFO:teuthology.orchestra.run.smithi088.stdout:(37/156): cephadm-19.1.1-326.g8921c8ef.el9.noar 3.9 MB/s | 765 kB 00:00 2024-09-16T01:32:02.928 INFO:teuthology.orchestra.run.smithi154.stdout:(6/156): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_ 4.9 MB/s | 1.5 MB 00:00 2024-09-16T01:32:02.937 INFO:teuthology.orchestra.run.smithi088.stdout:(38/156): cephfs-top-19.1.1-326.g8921c8ef.el9.n 156 kB/s | 18 kB 00:00 2024-09-16T01:32:03.071 INFO:teuthology.orchestra.run.smithi088.stdout:(39/156): fuse-2.9.9-16.el9.x86_64.rpm 503 kB/s | 80 kB 00:00 2024-09-16T01:32:03.138 INFO:teuthology.orchestra.run.smithi088.stdout:(40/156): libgfortran-11.5.0-2.el9.x86_64.rpm 3.9 MB/s | 801 kB 00:00 2024-09-16T01:32:03.197 INFO:teuthology.orchestra.run.smithi088.stdout:(41/156): libquadmath-11.5.0-2.el9.x86_64.rpm 1.5 MB/s | 191 kB 00:00 2024-09-16T01:32:03.230 INFO:teuthology.orchestra.run.smithi088.stdout:(42/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.7 MB/s | 253 kB 00:00 2024-09-16T01:32:03.331 INFO:teuthology.orchestra.run.smithi088.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 106 kB 00:00 2024-09-16T01:32:03.373 INFO:teuthology.orchestra.run.smithi088.stdout:(44/156): python3-cryptography-36.0.1-4.el9.x86 7.0 MB/s | 1.2 MB 00:00 2024-09-16T01:32:03.415 INFO:teuthology.orchestra.run.smithi088.stdout:(45/156): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 135 kB 00:00 2024-09-16T01:32:03.456 INFO:teuthology.orchestra.run.smithi088.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-09-16T01:32:03.498 INFO:teuthology.orchestra.run.smithi088.stdout:(47/156): python3-urllib3-1.26.5-6.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-09-16T01:32:03.549 INFO:teuthology.orchestra.run.smithi088.stdout:(48/156): automake-1.16.2-8.el9.noarch.rpm 7.3 MB/s | 689 kB 00:00 2024-09-16T01:32:03.599 INFO:teuthology.orchestra.run.smithi088.stdout:(49/156): bison-3.7.4-5.el9.x86_64.rpm 9.2 MB/s | 943 kB 00:00 2024-09-16T01:32:03.624 INFO:teuthology.orchestra.run.smithi088.stdout:(50/156): boost-program-options-1.75.0-8.el9.x8 1.4 MB/s | 107 kB 00:00 2024-09-16T01:32:03.654 INFO:teuthology.orchestra.run.smithi154.stdout:(7/156): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_ 4.7 MB/s | 4.7 MB 00:00 2024-09-16T01:32:03.666 INFO:teuthology.orchestra.run.smithi088.stdout:(51/156): elfutils-libelf-devel-0.191-4.el9.x86 324 kB/s | 22 kB 00:00 2024-09-16T01:32:03.700 INFO:teuthology.orchestra.run.smithi088.stdout:(52/156): flex-2.6.4-9.el9.x86_64.rpm 4.1 MB/s | 313 kB 00:00 2024-09-16T01:32:03.742 INFO:teuthology.orchestra.run.smithi088.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 433 kB/s | 33 kB 00:00 2024-09-16T01:32:03.817 INFO:teuthology.orchestra.run.smithi088.stdout:(54/156): flexiblas-openblas-openmp-3.0.4-8.el9 243 kB/s | 18 kB 00:00 2024-09-16T01:32:03.884 INFO:teuthology.orchestra.run.smithi088.stdout:(55/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 16 MB/s | 3.0 MB 00:00 2024-09-16T01:32:03.909 INFO:teuthology.orchestra.run.smithi088.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 121 kB/s | 11 kB 00:00 2024-09-16T01:32:04.270 INFO:teuthology.orchestra.run.smithi154.stdout:(8/156): ceph-common-19.1.1-326.g8921c8ef.el9.x 5.8 MB/s | 21 MB 00:03 2024-09-16T01:32:04.335 INFO:teuthology.orchestra.run.smithi088.stdout:(57/156): ceph-test-19.1.1-326.g8921c8ef.el9.x8 11 MB/s | 49 MB 00:04 2024-09-16T01:32:04.360 INFO:teuthology.orchestra.run.smithi088.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 100 kB/s | 45 kB 00:00 2024-09-16T01:32:04.394 INFO:teuthology.orchestra.run.smithi088.stdout:(59/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 315 kB/s | 160 kB 00:00 2024-09-16T01:32:04.452 INFO:teuthology.orchestra.run.smithi088.stdout:(60/156): libtool-2.4.6-46.el9.x86_64.rpm 6.2 MB/s | 581 kB 00:00 2024-09-16T01:32:04.463 INFO:teuthology.orchestra.run.smithi154.stdout:(9/156): ceph-osd-19.1.1-326.g8921c8ef.el9.x86_ 11 MB/s | 17 MB 00:01 2024-09-16T01:32:04.478 INFO:teuthology.orchestra.run.smithi088.stdout:(61/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 292 kB/s | 24 kB 00:00 2024-09-16T01:32:04.479 INFO:teuthology.orchestra.run.smithi154.stdout:(10/156): ceph-selinux-19.1.1-326.g8921c8ef.el9 121 kB/s | 25 kB 00:00 2024-09-16T01:32:04.519 INFO:teuthology.orchestra.run.smithi088.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 711 kB/s | 47 kB 00:00 2024-09-16T01:32:04.553 INFO:teuthology.orchestra.run.smithi088.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-09-16T01:32:04.587 INFO:teuthology.orchestra.run.smithi088.stdout:(64/156): librdkafka-1.6.1-102.el9.x86_64.rpm 2.6 MB/s | 662 kB 00:00 2024-09-16T01:32:04.612 INFO:teuthology.orchestra.run.smithi088.stdout:(65/156): lua-5.4.4-4.el9.x86_64.rpm 2.0 MB/s | 188 kB 00:00 2024-09-16T01:32:04.630 INFO:teuthology.orchestra.run.smithi154.stdout:(11/156): ceph-radosgw-19.1.1-326.g8921c8ef.el9 11 MB/s | 11 MB 00:00 2024-09-16T01:32:04.637 INFO:teuthology.orchestra.run.smithi088.stdout:(66/156): openblas-0.3.26-2.el9.x86_64.rpm 458 kB/s | 38 kB 00:00 2024-09-16T01:32:04.655 INFO:teuthology.orchestra.run.smithi154.stdout:(12/156): cephfs-mirror-19.1.1-326.g8921c8ef.el 1.3 MB/s | 228 kB 00:00 2024-09-16T01:32:04.680 INFO:teuthology.orchestra.run.smithi154.stdout:(13/156): cephfs-shell-19.1.1-326.g8921c8ef.el9 541 kB/s | 27 kB 00:00 2024-09-16T01:32:04.705 INFO:teuthology.orchestra.run.smithi154.stdout:(14/156): libcephfs-devel-19.1.1-326.g8921c8ef. 631 kB/s | 32 kB 00:00 2024-09-16T01:32:04.738 INFO:teuthology.orchestra.run.smithi088.stdout:(67/156): postgresql-13.16-1.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-16T01:32:04.747 INFO:teuthology.orchestra.run.smithi154.stdout:(15/156): libcephfs2-19.1.1-326.g8921c8ef.el9.x 11 MB/s | 751 kB 00:00 2024-09-16T01:32:04.773 INFO:teuthology.orchestra.run.smithi154.stdout:(16/156): libcephsqlite-19.1.1-326.g8921c8ef.el 2.4 MB/s | 163 kB 00:00 2024-09-16T01:32:04.798 INFO:teuthology.orchestra.run.smithi154.stdout:(17/156): librados-devel-19.1.1-326.g8921c8ef.e 2.5 MB/s | 127 kB 00:00 2024-09-16T01:32:04.813 INFO:teuthology.orchestra.run.smithi088.stdout:(68/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 22 MB/s | 4.4 MB 00:00 2024-09-16T01:32:04.823 INFO:teuthology.orchestra.run.smithi154.stdout:(18/156): libradosstriper1-19.1.1-326.g8921c8ef 9.8 MB/s | 506 kB 00:00 2024-09-16T01:32:04.855 INFO:teuthology.orchestra.run.smithi088.stdout:(69/156): postgresql-contrib-13.16-1.el9.x86_64 7.6 MB/s | 910 kB 00:00 2024-09-16T01:32:04.882 INFO:teuthology.orchestra.run.smithi154.stdout:(19/156): python3-ceph-argparse-19.1.1-326.g892 774 kB/s | 45 kB 00:00 2024-09-16T01:32:04.939 INFO:teuthology.orchestra.run.smithi088.stdout:(70/156): openblas-openmp-0.3.26-2.el9.x86_64.r 14 MB/s | 4.9 MB 00:00 2024-09-16T01:32:04.964 INFO:teuthology.orchestra.run.smithi088.stdout:(71/156): postgresql-private-libs-13.16-1.el9.x 915 kB/s | 137 kB 00:00 2024-09-16T01:32:04.965 INFO:teuthology.orchestra.run.smithi154.stdout:(20/156): python3-ceph-common-19.1.1-326.g8921c 1.6 MB/s | 136 kB 00:00 2024-09-16T01:32:05.091 INFO:teuthology.orchestra.run.smithi154.stdout:(21/156): librgw2-19.1.1-326.g8921c8ef.el9.x86_ 18 MB/s | 5.3 MB 00:00 2024-09-16T01:32:05.116 INFO:teuthology.orchestra.run.smithi154.stdout:(22/156): python3-cephfs-19.1.1-326.g8921c8ef.e 1.0 MB/s | 161 kB 00:00 2024-09-16T01:32:05.150 INFO:teuthology.orchestra.run.smithi154.stdout:(23/156): python3-rados-19.1.1-326.g8921c8ef.el 5.4 MB/s | 320 kB 00:00 2024-09-16T01:32:05.183 INFO:teuthology.orchestra.run.smithi154.stdout:(24/156): python3-rbd-19.1.1-326.g8921c8ef.el9. 4.4 MB/s | 300 kB 00:00 2024-09-16T01:32:05.189 INFO:teuthology.orchestra.run.smithi088.stdout:(72/156): postgresql-server-13.16-1.el9.x86_64. 17 MB/s | 5.8 MB 00:00 2024-09-16T01:32:05.209 INFO:teuthology.orchestra.run.smithi154.stdout:(25/156): python3-rgw-19.1.1-326.g8921c8ef.el9. 1.7 MB/s | 99 kB 00:00 2024-09-16T01:32:05.214 INFO:teuthology.orchestra.run.smithi088.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 349 kB/s | 87 kB 00:00 2024-09-16T01:32:05.234 INFO:teuthology.orchestra.run.smithi154.stdout:(26/156): rbd-fuse-19.1.1-326.g8921c8ef.el9.x86 1.7 MB/s | 85 kB 00:00 2024-09-16T01:32:05.256 INFO:teuthology.orchestra.run.smithi088.stdout:(74/156): protobuf-3.14.0-13.el9.x86_64.rpm 3.2 MB/s | 1.0 MB 00:00 2024-09-16T01:32:05.282 INFO:teuthology.orchestra.run.smithi088.stdout:(75/156): python3-colorama-0.4.6-3.el9.noarch.r 915 kB/s | 61 kB 00:00 2024-09-16T01:32:05.284 INFO:teuthology.orchestra.run.smithi154.stdout:(27/156): rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_ 3.3 MB/s | 170 kB 00:00 2024-09-16T01:32:05.357 INFO:teuthology.orchestra.run.smithi088.stdout:(76/156): python3-jinja2-2.11.3-6.el9.noarch.rp 3.2 MB/s | 248 kB 00:00 2024-09-16T01:32:05.359 INFO:teuthology.orchestra.run.smithi154.stdout:(28/156): rbd-mirror-19.1.1-326.g8921c8ef.el9.x 21 MB/s | 3.1 MB 00:00 2024-09-16T01:32:05.385 INFO:teuthology.orchestra.run.smithi154.stdout:(29/156): ceph-grafana-dashboards-19.1.1-326.g8 318 kB/s | 32 kB 00:00 2024-09-16T01:32:05.418 INFO:teuthology.orchestra.run.smithi154.stdout:(30/156): ceph-mgr-cephadm-19.1.1-326.g8921c8ef 2.4 MB/s | 146 kB 00:00 2024-09-16T01:32:05.432 INFO:teuthology.orchestra.run.smithi088.stdout:(77/156): python3-babel-2.9.1-2.el9.noarch.rpm 25 MB/s | 6.0 MB 00:00 2024-09-16T01:32:05.457 INFO:teuthology.orchestra.run.smithi088.stdout:(78/156): python3-jsonpatch-1.21-16.el9.noarch. 256 kB/s | 26 kB 00:00 2024-09-16T01:32:05.483 INFO:teuthology.orchestra.run.smithi088.stdout:(79/156): python3-devel-3.9.19-8.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-09-16T01:32:05.517 INFO:teuthology.orchestra.run.smithi088.stdout:(80/156): python3-jsonpointer-2.0-4.el9.noarch. 228 kB/s | 19 kB 00:00 2024-09-16T01:32:05.542 INFO:teuthology.orchestra.run.smithi088.stdout:(81/156): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-16T01:32:05.567 INFO:teuthology.orchestra.run.smithi088.stdout:(82/156): python3-markupsafe-1.1.1-12.el9.x86_6 413 kB/s | 35 kB 00:00 2024-09-16T01:32:05.609 INFO:teuthology.orchestra.run.smithi088.stdout:(83/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.7 MB/s | 250 kB 00:00 2024-09-16T01:32:05.684 INFO:teuthology.orchestra.run.smithi088.stdout:(84/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 5.1 MB 00:00 2024-09-16T01:32:05.694 INFO:teuthology.orchestra.run.smithi154.stdout:(31/156): ceph-mgr-dashboard-19.1.1-326.g8921c8 12 MB/s | 3.6 MB 00:00 2024-09-16T01:32:05.709 INFO:teuthology.orchestra.run.smithi088.stdout:(85/156): python3-packaging-20.9-5.el9.noarch.r 770 kB/s | 77 kB 00:00 2024-09-16T01:32:05.735 INFO:teuthology.orchestra.run.smithi088.stdout:(86/156): python3-oauthlib-3.1.1-5.el9.noarch.r 1.3 MB/s | 222 kB 00:00 2024-09-16T01:32:05.760 INFO:teuthology.orchestra.run.smithi088.stdout:(87/156): python3-prettytable-0.7.2-27.el9.noar 562 kB/s | 42 kB 00:00 2024-09-16T01:32:05.785 INFO:teuthology.orchestra.run.smithi088.stdout:(88/156): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 269 kB 00:00 2024-09-16T01:32:05.819 INFO:teuthology.orchestra.run.smithi088.stdout:(89/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.8 MB/s | 159 kB 00:00 2024-09-16T01:32:05.836 INFO:teuthology.orchestra.run.smithi154.stdout:(32/156): ceph-mgr-diskprediction-local-19.1.1- 18 MB/s | 7.4 MB 00:00 2024-09-16T01:32:05.844 INFO:teuthology.orchestra.run.smithi088.stdout:(90/156): python3-pyasn1-modules-0.4.8-6.el9.no 3.2 MB/s | 279 kB 00:00 2024-09-16T01:32:05.870 INFO:teuthology.orchestra.run.smithi088.stdout:(91/156): python3-pytz-2021.1-5.el9.noarch.rpm 607 kB/s | 51 kB 00:00 2024-09-16T01:32:05.870 INFO:teuthology.orchestra.run.smithi154.stdout:(33/156): ceph-mgr-modules-core-19.1.1-326.g892 1.4 MB/s | 249 kB 00:00 2024-09-16T01:32:05.894 INFO:teuthology.orchestra.run.smithi154.stdout:(34/156): ceph-mgr-rook-19.1.1-326.g8921c8ef.el 843 kB/s | 49 kB 00:00 2024-09-16T01:32:05.903 INFO:teuthology.orchestra.run.smithi088.stdout:(92/156): python3-requests-oauthlib-1.3.0-12.el 640 kB/s | 54 kB 00:00 2024-09-16T01:32:05.920 INFO:teuthology.orchestra.run.smithi154.stdout:(35/156): ceph-prometheus-alerts-19.1.1-326.g89 328 kB/s | 16 kB 00:00 2024-09-16T01:32:05.937 INFO:teuthology.orchestra.run.smithi088.stdout:(93/156): python3-toml-0.10.2-6.el9.noarch.rpm 622 kB/s | 42 kB 00:00 2024-09-16T01:32:05.954 INFO:teuthology.orchestra.run.smithi154.stdout:(36/156): ceph-volume-19.1.1-326.g8921c8ef.el9. 4.7 MB/s | 285 kB 00:00 2024-09-16T01:32:05.979 INFO:teuthology.orchestra.run.smithi088.stdout:(94/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 587 kB/s | 44 kB 00:00 2024-09-16T01:32:05.987 INFO:teuthology.orchestra.run.smithi154.stdout:(37/156): cephadm-19.1.1-326.g8921c8ef.el9.noar 11 MB/s | 765 kB 00:00 2024-09-16T01:32:06.004 INFO:teuthology.orchestra.run.smithi088.stdout:(95/156): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 221 kB 00:00 2024-09-16T01:32:06.021 INFO:teuthology.orchestra.run.smithi154.stdout:(38/156): cephfs-top-19.1.1-326.g8921c8ef.el9.n 272 kB/s | 18 kB 00:00 2024-09-16T01:32:06.046 INFO:teuthology.orchestra.run.smithi088.stdout:(96/156): qatlib-service-24.02.0-1.el9.x86_64.r 538 kB/s | 36 kB 00:00 2024-09-16T01:32:06.071 INFO:teuthology.orchestra.run.smithi088.stdout:(97/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 708 kB/s | 47 kB 00:00 2024-09-16T01:32:06.130 INFO:teuthology.orchestra.run.smithi088.stdout:(98/156): socat-1.7.4.1-6.el9.x86_64.rpm 3.6 MB/s | 304 kB 00:00 2024-09-16T01:32:06.180 INFO:teuthology.orchestra.run.smithi154.stdout:(39/156): fuse-2.9.9-16.el9.x86_64.rpm 415 kB/s | 80 kB 00:00 2024-09-16T01:32:06.180 INFO:teuthology.orchestra.run.smithi088.stdout:(99/156): uuid-1.6.2-55.el9.x86_64.rpm 533 kB/s | 58 kB 00:00 2024-09-16T01:32:06.205 INFO:teuthology.orchestra.run.smithi088.stdout:(100/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.2 MB/s | 96 kB 00:00 2024-09-16T01:32:06.263 INFO:teuthology.orchestra.run.smithi154.stdout:(40/156): libquadmath-11.5.0-2.el9.x86_64.rpm 2.2 MB/s | 191 kB 00:00 2024-09-16T01:32:06.364 INFO:teuthology.orchestra.run.smithi088.stdout:(101/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-09-16T01:32:06.389 INFO:teuthology.orchestra.run.smithi088.stdout:(102/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 244 kB/s | 45 kB 00:00 2024-09-16T01:32:06.414 INFO:teuthology.orchestra.run.smithi088.stdout:(103/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 272 kB/s | 64 kB 00:00 2024-09-16T01:32:06.481 INFO:teuthology.orchestra.run.smithi088.stdout:(104/156): abseil-cpp-20211102.0-3.el9.x86_64.r 8.1 MB/s | 548 kB 00:00 2024-09-16T01:32:06.515 INFO:teuthology.orchestra.run.smithi088.stdout:(105/156): gperftools-libs-2.9.1-3.el9.x86_64.r 9.1 MB/s | 308 kB 00:00 2024-09-16T01:32:06.549 INFO:teuthology.orchestra.run.smithi088.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 584 kB/s | 19 kB 00:00 2024-09-16T01:32:06.697 INFO:teuthology.orchestra.run.smithi154.stdout:(41/156): ceph-test-19.1.1-326.g8921c8ef.el9.x8 22 MB/s | 49 MB 00:02 2024-09-16T01:32:06.707 INFO:teuthology.orchestra.run.smithi088.stdout:(107/156): libarrow-9.0.0-11.el9.x86_64.rpm 28 MB/s | 4.4 MB 00:00 2024-09-16T01:32:06.722 INFO:teuthology.orchestra.run.smithi154.stdout:(42/156): libgfortran-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 801 kB 00:00 2024-09-16T01:32:06.733 INFO:teuthology.orchestra.run.smithi088.stdout:(108/156): lua-devel-5.4.4-4.el9.x86_64.rpm 61 kB/s | 22 kB 00:00 2024-09-16T01:32:06.747 INFO:teuthology.orchestra.run.smithi154.stdout:(43/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 523 kB/s | 253 kB 00:00 2024-09-16T01:32:06.758 INFO:teuthology.orchestra.run.smithi088.stdout:(109/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 518 kB/s | 26 kB 00:00 2024-09-16T01:32:06.773 INFO:teuthology.orchestra.run.smithi154.stdout:(44/156): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-16T01:32:06.798 INFO:teuthology.orchestra.run.smithi154.stdout:(45/156): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-09-16T01:32:06.800 INFO:teuthology.orchestra.run.smithi088.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 1.9 MB/s | 67 kB 00:00 2024-09-16T01:32:06.825 INFO:teuthology.orchestra.run.smithi088.stdout:(111/156): liboath-2.6.7-2.el9.x86_64.rpm 526 kB/s | 49 kB 00:00 2024-09-16T01:32:06.832 INFO:teuthology.orchestra.run.smithi154.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-09-16T01:32:06.850 INFO:teuthology.orchestra.run.smithi088.stdout:(112/156): luarocks-3.9.2-5.el9.noarch.rpm 3.1 MB/s | 151 kB 00:00 2024-09-16T01:32:06.865 INFO:teuthology.orchestra.run.smithi154.stdout:(47/156): python3-urllib3-1.26.5-6.el9.noarch.r 3.1 MB/s | 215 kB 00:00 2024-09-16T01:32:06.901 INFO:teuthology.orchestra.run.smithi088.stdout:(113/156): python3-asyncssh-2.13.2-5.el9.noarch 11 MB/s | 548 kB 00:00 2024-09-16T01:32:06.926 INFO:teuthology.orchestra.run.smithi088.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-16T01:32:06.951 INFO:teuthology.orchestra.run.smithi088.stdout:(115/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.5 MB/s | 839 kB 00:00 2024-09-16T01:32:06.977 INFO:teuthology.orchestra.run.smithi088.stdout:(116/156): python3-cachetools-4.2.4-1.el9.noarc 641 kB/s | 32 kB 00:00 2024-09-16T01:32:07.002 INFO:teuthology.orchestra.run.smithi088.stdout:(117/156): python3-certifi-2023.05.07-4.el9.noa 280 kB/s | 14 kB 00:00 2024-09-16T01:32:07.027 INFO:teuthology.orchestra.run.smithi088.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-09-16T01:32:07.049 INFO:teuthology.orchestra.run.smithi154.stdout:(48/156): automake-1.16.2-8.el9.noarch.rpm 3.1 MB/s | 689 kB 00:00 2024-09-16T01:32:07.061 INFO:teuthology.orchestra.run.smithi088.stdout:(119/156): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-09-16T01:32:07.086 INFO:teuthology.orchestra.run.smithi088.stdout:(120/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-09-16T01:32:07.091 INFO:teuthology.orchestra.run.smithi154.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 107 kB 00:00 2024-09-16T01:32:07.112 INFO:teuthology.orchestra.run.smithi088.stdout:(121/156): python3-google-auth-2.34.0-1.el9.noa 4.3 MB/s | 220 kB 00:00 2024-09-16T01:32:07.116 INFO:teuthology.orchestra.run.smithi154.stdout:(50/156): bison-3.7.4-5.el9.x86_64.rpm 3.7 MB/s | 943 kB 00:00 2024-09-16T01:32:07.141 INFO:teuthology.orchestra.run.smithi154.stdout:(51/156): elfutils-libelf-devel-0.191-4.el9.x86 433 kB/s | 22 kB 00:00 2024-09-16T01:32:07.145 INFO:teuthology.orchestra.run.smithi088.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 4.3 MB/s | 144 kB 00:00 2024-09-16T01:32:07.170 INFO:teuthology.orchestra.run.smithi088.stdout:(123/156): python3-jaraco-8.2.1-3.el9.noarch.rp 429 kB/s | 11 kB 00:00 2024-09-16T01:32:07.183 INFO:teuthology.orchestra.run.smithi154.stdout:(52/156): python3-cryptography-36.0.1-4.el9.x86 2.5 MB/s | 1.2 MB 00:00 2024-09-16T01:32:07.204 INFO:teuthology.orchestra.run.smithi088.stdout:(124/156): protobuf-compiler-3.14.0-13.el9.x86_ 1.0 MB/s | 863 kB 00:00 2024-09-16T01:32:07.217 INFO:teuthology.orchestra.run.smithi154.stdout:(53/156): flex-2.6.4-9.el9.x86_64.rpm 3.0 MB/s | 313 kB 00:00 2024-09-16T01:32:07.229 INFO:teuthology.orchestra.run.smithi088.stdout:(125/156): python3-jaraco-classes-3.2.1-5.el9.n 302 kB/s | 18 kB 00:00 2024-09-16T01:32:07.242 INFO:teuthology.orchestra.run.smithi154.stdout:(54/156): flexiblas-3.0.4-8.el9.x86_64.rpm 325 kB/s | 33 kB 00:00 2024-09-16T01:32:07.263 INFO:teuthology.orchestra.run.smithi088.stdout:(126/156): python3-jaraco-functools-3.5.0-2.el9 780 kB/s | 19 kB 00:00 2024-09-16T01:32:07.289 INFO:teuthology.orchestra.run.smithi088.stdout:(127/156): python3-jaraco-text-3.2.0-6.el9.noar 849 kB/s | 20 kB 00:00 2024-09-16T01:32:07.292 INFO:teuthology.orchestra.run.smithi154.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 365 kB/s | 18 kB 00:00 2024-09-16T01:32:07.313 INFO:teuthology.orchestra.run.smithi088.stdout:(128/156): python3-jaraco-collections-3.0.0-8.e 212 kB/s | 23 kB 00:00 2024-09-16T01:32:07.318 INFO:teuthology.orchestra.run.smithi154.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 148 kB/s | 11 kB 00:00 2024-09-16T01:32:07.339 INFO:teuthology.orchestra.run.smithi088.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-16T01:32:07.343 INFO:teuthology.orchestra.run.smithi154.stdout:(57/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-09-16T01:32:07.364 INFO:teuthology.orchestra.run.smithi088.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-16T01:32:07.368 INFO:teuthology.orchestra.run.smithi154.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 906 kB/s | 45 kB 00:00 2024-09-16T01:32:07.414 INFO:teuthology.orchestra.run.smithi088.stdout:(131/156): python3-grpcio-1.46.7-10.el9.x86_64. 6.2 MB/s | 2.0 MB 00:00 2024-09-16T01:32:07.435 INFO:teuthology.orchestra.run.smithi154.stdout:(59/156): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 662 kB 00:00 2024-09-16T01:32:07.439 INFO:teuthology.orchestra.run.smithi088.stdout:(132/156): python3-natsort-7.1.1-5.el9.noarch.r 766 kB/s | 58 kB 00:00 2024-09-16T01:32:07.457 INFO:teuthology.orchestra.run.smithi088.stdout:(133/156): python3-portend-3.1.0-2.el9.noarch.r 998 kB/s | 16 kB 00:00 2024-09-16T01:32:07.486 INFO:teuthology.orchestra.run.smithi154.stdout:(60/156): libtool-2.4.6-46.el9.x86_64.rpm 4.8 MB/s | 581 kB 00:00 2024-09-16T01:32:07.490 INFO:teuthology.orchestra.run.smithi088.stdout:(134/156): python3-kubernetes-26.1.0-2.el9.noar 5.1 MB/s | 1.0 MB 00:00 2024-09-16T01:32:07.510 INFO:teuthology.orchestra.run.smithi154.stdout:(61/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 325 kB/s | 24 kB 00:00 2024-09-16T01:32:07.515 INFO:teuthology.orchestra.run.smithi088.stdout:(135/156): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-09-16T01:32:07.536 INFO:teuthology.orchestra.run.smithi154.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 949 kB/s | 47 kB 00:00 2024-09-16T01:32:07.540 INFO:teuthology.orchestra.run.smithi088.stdout:(136/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-09-16T01:32:07.557 INFO:teuthology.orchestra.run.smithi088.stdout:(137/156): python3-pyperclip-1.8.2-2.el9.noarch 675 kB/s | 28 kB 00:00 2024-09-16T01:32:07.561 INFO:teuthology.orchestra.run.smithi154.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-09-16T01:32:07.574 INFO:teuthology.orchestra.run.smithi088.stdout:(138/156): python3-repoze-lru-0.7-16.el9.noarch 525 kB/s | 31 kB 00:00 2024-09-16T01:32:07.586 INFO:teuthology.orchestra.run.smithi154.stdout:(64/156): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-09-16T01:32:07.600 INFO:teuthology.orchestra.run.smithi088.stdout:(139/156): python3-tempora-5.0.0-2.el9.noarch.r 1.4 MB/s | 36 kB 00:00 2024-09-16T01:32:07.612 INFO:teuthology.orchestra.run.smithi154.stdout:(65/156): openblas-0.3.26-2.el9.x86_64.rpm 764 kB/s | 38 kB 00:00 2024-09-16T01:32:07.625 INFO:teuthology.orchestra.run.smithi088.stdout:(140/156): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-09-16T01:32:07.650 INFO:teuthology.orchestra.run.smithi088.stdout:(141/156): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-09-16T01:32:07.676 INFO:teuthology.orchestra.run.smithi088.stdout:(142/156): python3-typing-extensions-4.12.2-2.e 988 kB/s | 74 kB 00:00 2024-09-16T01:32:07.687 INFO:teuthology.orchestra.run.smithi154.stdout:(66/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.9 MB/s | 3.0 MB 00:00 2024-09-16T01:32:07.701 INFO:teuthology.orchestra.run.smithi088.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-16T01:32:07.726 INFO:teuthology.orchestra.run.smithi088.stdout:(144/156): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-16T01:32:07.752 INFO:teuthology.orchestra.run.smithi088.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 397 kB/s | 20 kB 00:00 2024-09-16T01:32:07.785 INFO:teuthology.orchestra.run.smithi088.stdout:(146/156): re2-20211101-20.el9.x86_64.rpm 3.2 MB/s | 191 kB 00:00 2024-09-16T01:32:07.811 INFO:teuthology.orchestra.run.smithi088.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-09-16T01:32:07.829 INFO:teuthology.orchestra.run.smithi154.stdout:(67/156): postgresql-13.16-1.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-09-16T01:32:07.869 INFO:teuthology.orchestra.run.smithi088.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-16T01:32:07.896 INFO:teuthology.orchestra.run.smithi154.stdout:(68/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-09-16T01:32:07.972 INFO:teuthology.orchestra.run.smithi154.stdout:(69/156): openblas-openmp-0.3.26-2.el9.x86_64.r 13 MB/s | 4.9 MB 00:00 2024-09-16T01:32:07.995 INFO:teuthology.orchestra.run.smithi088.stdout:(149/156): NetworkManager-1.49.90-1.el9.x86_64. 19 MB/s | 2.3 MB 00:00 2024-09-16T01:32:08.005 INFO:teuthology.orchestra.run.smithi154.stdout:(70/156): postgresql-contrib-13.16-1.el9.x86_64 5.1 MB/s | 910 kB 00:00 2024-09-16T01:32:08.030 INFO:teuthology.orchestra.run.smithi154.stdout:(71/156): postgresql-private-libs-13.16-1.el9.x 1.0 MB/s | 137 kB 00:00 2024-09-16T01:32:08.081 INFO:teuthology.orchestra.run.smithi154.stdout:(72/156): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-09-16T01:32:08.095 INFO:teuthology.orchestra.run.smithi088.stdout:(150/156): NetworkManager-libnm-1.49.90-1.el9.x 18 MB/s | 1.8 MB 00:00 2024-09-16T01:32:08.123 INFO:teuthology.orchestra.run.smithi154.stdout:(73/156): protobuf-3.14.0-13.el9.x86_64.rpm 8.6 MB/s | 1.0 MB 00:00 2024-09-16T01:32:08.145 INFO:teuthology.orchestra.run.smithi088.stdout:(151/156): librbd1-19.1.1-326.g8921c8ef.el9.x86 9.5 MB/s | 3.2 MB 00:00 2024-09-16T01:32:08.156 INFO:teuthology.orchestra.run.smithi154.stdout:(74/156): python3-colorama-0.4.6-3.el9.noarch.r 1.8 MB/s | 61 kB 00:00 2024-09-16T01:32:08.170 INFO:teuthology.orchestra.run.smithi088.stdout:(152/156): NetworkManager-team-1.49.90-1.el9.x8 542 kB/s | 41 kB 00:00 2024-09-16T01:32:08.206 INFO:teuthology.orchestra.run.smithi154.stdout:(75/156): python3-devel-3.9.19-8.el9.x86_64.rpm 4.8 MB/s | 245 kB 00:00 2024-09-16T01:32:08.229 INFO:teuthology.orchestra.run.smithi088.stdout:(153/156): librados2-19.1.1-326.g8921c8ef.el9.x 7.7 MB/s | 3.4 MB 00:00 2024-09-16T01:32:08.254 INFO:teuthology.orchestra.run.smithi088.stdout:(154/156): NetworkManager-tui-1.49.90-1.el9.x86 2.2 MB/s | 249 kB 00:00 2024-09-16T01:32:08.256 INFO:teuthology.orchestra.run.smithi154.stdout:(76/156): python3-jinja2-2.11.3-6.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-09-16T01:32:08.313 INFO:teuthology.orchestra.run.smithi088.stdout:(155/156): openssl-3.2.2-6.el9.x86_64.rpm 9.5 MB/s | 1.4 MB 00:00 2024-09-16T01:32:08.332 INFO:teuthology.orchestra.run.smithi154.stdout:(77/156): postgresql-server-13.16-1.el9.x86_64. 16 MB/s | 5.8 MB 00:00 2024-09-16T01:32:08.357 INFO:teuthology.orchestra.run.smithi154.stdout:(78/156): python3-jsonpatch-1.21-16.el9.noarch. 256 kB/s | 26 kB 00:00 2024-09-16T01:32:08.384 INFO:teuthology.orchestra.run.smithi154.stdout:(79/156): python3-jsonpointer-2.0-4.el9.noarch. 380 kB/s | 19 kB 00:00 2024-09-16T01:32:08.408 INFO:teuthology.orchestra.run.smithi154.stdout:(80/156): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-09-16T01:32:08.413 INFO:teuthology.orchestra.run.smithi088.stdout:(156/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 13 MB/s | 2.4 MB 00:00 2024-09-16T01:32:08.419 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-09-16T01:32:08.420 INFO:teuthology.orchestra.run.smithi088.stdout:Total 19 MB/s | 226 MB 00:12 2024-09-16T01:32:08.433 INFO:teuthology.orchestra.run.smithi154.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 694 kB/s | 35 kB 00:00 2024-09-16T01:32:08.517 INFO:teuthology.orchestra.run.smithi154.stdout:(82/156): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-09-16T01:32:08.542 INFO:teuthology.orchestra.run.smithi154.stdout:(83/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-09-16T01:32:08.567 INFO:teuthology.orchestra.run.smithi154.stdout:(84/156): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-09-16T01:32:08.593 INFO:teuthology.orchestra.run.smithi154.stdout:(85/156): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-09-16T01:32:08.618 INFO:teuthology.orchestra.run.smithi154.stdout:(86/156): python3-prettytable-0.7.2-27.el9.noar 839 kB/s | 42 kB 00:00 2024-09-16T01:32:08.652 INFO:teuthology.orchestra.run.smithi154.stdout:(87/156): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-09-16T01:32:08.677 INFO:teuthology.orchestra.run.smithi154.stdout:(88/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-09-16T01:32:08.702 INFO:teuthology.orchestra.run.smithi154.stdout:(89/156): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-09-16T01:32:08.727 INFO:teuthology.orchestra.run.smithi154.stdout:(90/156): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-16T01:32:08.752 INFO:teuthology.orchestra.run.smithi154.stdout:(91/156): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-09-16T01:32:08.836 INFO:teuthology.orchestra.run.smithi154.stdout:(92/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-09-16T01:32:08.861 INFO:teuthology.orchestra.run.smithi154.stdout:(93/156): python3-toml-0.10.2-6.el9.noarch.rpm 383 kB/s | 42 kB 00:00 2024-09-16T01:32:08.886 INFO:teuthology.orchestra.run.smithi154.stdout:(94/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 883 kB/s | 44 kB 00:00 2024-09-16T01:32:08.912 INFO:teuthology.orchestra.run.smithi154.stdout:(95/156): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-09-16T01:32:08.937 INFO:teuthology.orchestra.run.smithi154.stdout:(96/156): qatlib-service-24.02.0-1.el9.x86_64.r 710 kB/s | 36 kB 00:00 2024-09-16T01:32:08.963 INFO:teuthology.orchestra.run.smithi154.stdout:(97/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 939 kB/s | 47 kB 00:00 2024-09-16T01:32:08.988 INFO:teuthology.orchestra.run.smithi154.stdout:(98/156): socat-1.7.4.1-6.el9.x86_64.rpm 5.9 MB/s | 304 kB 00:00 2024-09-16T01:32:09.013 INFO:teuthology.orchestra.run.smithi154.stdout:(99/156): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-09-16T01:32:09.038 INFO:teuthology.orchestra.run.smithi154.stdout:(100/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.9 MB/s | 96 kB 00:00 2024-09-16T01:32:09.064 INFO:teuthology.orchestra.run.smithi154.stdout:(101/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-16T01:32:09.089 INFO:teuthology.orchestra.run.smithi154.stdout:(102/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 891 kB/s | 45 kB 00:00 2024-09-16T01:32:09.114 INFO:teuthology.orchestra.run.smithi154.stdout:(103/156): lua-devel-5.4.4-4.el9.x86_64.rpm 444 kB/s | 22 kB 00:00 2024-09-16T01:32:09.156 INFO:teuthology.orchestra.run.smithi154.stdout:(104/156): protobuf-compiler-3.14.0-13.el9.x86_ 13 MB/s | 863 kB 00:00 2024-09-16T01:32:09.207 INFO:teuthology.orchestra.run.smithi154.stdout:(105/156): abseil-cpp-20211102.0-3.el9.x86_64.r 5.8 MB/s | 548 kB 00:00 2024-09-16T01:32:09.232 INFO:teuthology.orchestra.run.smithi154.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 846 kB/s | 19 kB 00:00 2024-09-16T01:32:09.274 INFO:teuthology.orchestra.run.smithi154.stdout:(107/156): gperftools-libs-2.9.1-3.el9.x86_64.r 2.6 MB/s | 308 kB 00:00 2024-09-16T01:32:09.299 INFO:teuthology.orchestra.run.smithi154.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-16T01:32:09.324 INFO:teuthology.orchestra.run.smithi154.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-16T01:32:09.350 INFO:teuthology.orchestra.run.smithi154.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-16T01:32:09.384 INFO:teuthology.orchestra.run.smithi154.stdout:(111/156): luarocks-3.9.2-5.el9.noarch.rpm 4.4 MB/s | 151 kB 00:00 2024-09-16T01:32:09.442 INFO:teuthology.orchestra.run.smithi154.stdout:(112/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 14 MB/s | 839 kB 00:00 2024-09-16T01:32:09.628 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-09-16T01:32:09.634 INFO:teuthology.orchestra.run.smithi154.stdout:(113/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 18 MB/s | 16 MB 00:00 2024-09-16T01:32:09.659 INFO:teuthology.orchestra.run.smithi154.stdout:(114/156): python3-asyncssh-2.13.2-5.el9.noarch 2.5 MB/s | 548 kB 00:00 2024-09-16T01:32:09.693 INFO:teuthology.orchestra.run.smithi154.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 1.2 MB/s | 32 kB 00:00 2024-09-16T01:32:09.718 INFO:teuthology.orchestra.run.smithi154.stdout:(116/156): python3-certifi-2023.05.07-4.el9.noa 606 kB/s | 14 kB 00:00 2024-09-16T01:32:09.743 INFO:teuthology.orchestra.run.smithi154.stdout:(117/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 399 kB/s | 43 kB 00:00 2024-09-16T01:32:09.769 INFO:teuthology.orchestra.run.smithi154.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-09-16T01:32:09.777 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-09-16T01:32:09.778 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-09-16T01:32:09.836 INFO:teuthology.orchestra.run.smithi154.stdout:(119/156): libarrow-9.0.0-11.el9.x86_64.rpm 7.3 MB/s | 4.4 MB 00:00 2024-09-16T01:32:09.861 INFO:teuthology.orchestra.run.smithi154.stdout:(120/156): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-09-16T01:32:09.886 INFO:teuthology.orchestra.run.smithi154.stdout:(121/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.5 MB/s | 303 kB 00:00 2024-09-16T01:32:09.912 INFO:teuthology.orchestra.run.smithi154.stdout:(122/156): python3-google-auth-2.34.0-1.el9.noa 2.8 MB/s | 220 kB 00:00 2024-09-16T01:32:09.937 INFO:teuthology.orchestra.run.smithi154.stdout:(123/156): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-09-16T01:32:09.954 INFO:teuthology.orchestra.run.smithi154.stdout:(124/156): python3-jaraco-8.2.1-3.el9.noarch.rp 253 kB/s | 11 kB 00:00 2024-09-16T01:32:09.980 INFO:teuthology.orchestra.run.smithi154.stdout:(125/156): python3-jaraco-classes-3.2.1-5.el9.n 417 kB/s | 18 kB 00:00 2024-09-16T01:32:10.005 INFO:teuthology.orchestra.run.smithi154.stdout:(126/156): python3-jaraco-collections-3.0.0-8.e 461 kB/s | 23 kB 00:00 2024-09-16T01:32:10.030 INFO:teuthology.orchestra.run.smithi154.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 388 kB/s | 19 kB 00:00 2024-09-16T01:32:10.055 INFO:teuthology.orchestra.run.smithi154.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-09-16T01:32:10.080 INFO:teuthology.orchestra.run.smithi154.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-16T01:32:10.106 INFO:teuthology.orchestra.run.smithi154.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-16T01:32:10.148 INFO:teuthology.orchestra.run.smithi154.stdout:(131/156): python3-kubernetes-26.1.0-2.el9.noar 8.7 MB/s | 1.0 MB 00:00 2024-09-16T01:32:10.173 INFO:teuthology.orchestra.run.smithi154.stdout:(132/156): python3-natsort-7.1.1-5.el9.noarch.r 861 kB/s | 58 kB 00:00 2024-09-16T01:32:10.198 INFO:teuthology.orchestra.run.smithi154.stdout:(133/156): python3-portend-3.1.0-2.el9.noarch.r 657 kB/s | 16 kB 00:00 2024-09-16T01:32:10.223 INFO:teuthology.orchestra.run.smithi154.stdout:(134/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 3.5 MB/s | 90 kB 00:00 2024-09-16T01:32:10.249 INFO:teuthology.orchestra.run.smithi154.stdout:(135/156): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-09-16T01:32:10.274 INFO:teuthology.orchestra.run.smithi154.stdout:(136/156): python3-pyperclip-1.8.2-2.el9.noarch 563 kB/s | 28 kB 00:00 2024-09-16T01:32:10.300 INFO:teuthology.orchestra.run.smithi154.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 611 kB/s | 31 kB 00:00 2024-09-16T01:32:10.325 INFO:teuthology.orchestra.run.smithi154.stdout:(138/156): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-16T01:32:10.384 INFO:teuthology.orchestra.run.smithi154.stdout:(139/156): python3-grpcio-1.46.7-10.el9.x86_64. 3.9 MB/s | 2.0 MB 00:00 2024-09-16T01:32:10.417 INFO:teuthology.orchestra.run.smithi154.stdout:(140/156): python3-routes-2.5.1-5.el9.noarch.rp 1.3 MB/s | 188 kB 00:00 2024-09-16T01:32:10.442 INFO:teuthology.orchestra.run.smithi154.stdout:(141/156): python3-tempora-5.0.0-2.el9.noarch.r 306 kB/s | 36 kB 00:00 2024-09-16T01:32:10.467 INFO:teuthology.orchestra.run.smithi154.stdout:(142/156): python3-typing-extensions-4.12.2-2.e 889 kB/s | 74 kB 00:00 2024-09-16T01:32:10.501 INFO:teuthology.orchestra.run.smithi154.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-09-16T01:32:10.526 INFO:teuthology.orchestra.run.smithi154.stdout:(144/156): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-16T01:32:10.551 INFO:teuthology.orchestra.run.smithi154.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-09-16T01:32:10.577 INFO:teuthology.orchestra.run.smithi154.stdout:(146/156): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-16T01:32:10.602 INFO:teuthology.orchestra.run.smithi154.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-09-16T01:32:10.669 INFO:teuthology.orchestra.run.smithi154.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-16T01:32:10.853 INFO:teuthology.orchestra.run.smithi154.stdout:(149/156): librbd1-19.1.1-326.g8921c8ef.el9.x86 13 MB/s | 3.2 MB 00:00 2024-09-16T01:32:10.911 INFO:teuthology.orchestra.run.smithi154.stdout:(150/156): librados2-19.1.1-326.g8921c8ef.el9.x 10 MB/s | 3.4 MB 00:00 2024-09-16T01:32:11.003 INFO:teuthology.orchestra.run.smithi154.stdout:(151/156): NetworkManager-team-1.49.90-1.el9.x8 444 kB/s | 41 kB 00:00 2024-09-16T01:32:11.137 INFO:teuthology.orchestra.run.smithi154.stdout:(152/156): NetworkManager-tui-1.49.90-1.el9.x86 1.8 MB/s | 249 kB 00:00 2024-09-16T01:32:11.304 INFO:teuthology.orchestra.run.smithi154.stdout:(153/156): NetworkManager-libnm-1.49.90-1.el9.x 4.0 MB/s | 1.8 MB 00:00 2024-09-16T01:32:11.413 INFO:teuthology.orchestra.run.smithi154.stdout:(154/156): NetworkManager-1.49.90-1.el9.x86_64. 3.1 MB/s | 2.3 MB 00:00 2024-09-16T01:32:11.496 INFO:teuthology.orchestra.run.smithi154.stdout:(155/156): openssl-3.2.2-6.el9.x86_64.rpm 3.8 MB/s | 1.4 MB 00:00 2024-09-16T01:32:12.104 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-09-16T01:32:12.105 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-09-16T01:32:12.513 INFO:teuthology.orchestra.run.smithi154.stdout:(156/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 2.0 MB/s | 2.4 MB 00:01 2024-09-16T01:32:12.519 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-16T01:32:12.519 INFO:teuthology.orchestra.run.smithi154.stdout:Total 19 MB/s | 226 MB 00:12 2024-09-16T01:32:13.727 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-16T01:32:13.880 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-16T01:32:13.880 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-16T01:32:16.102 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-16T01:32:16.103 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-16T01:32:18.388 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-09-16T01:32:18.475 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-16T01:32:18.560 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-16T01:32:18.733 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-16T01:32:18.767 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-16T01:32:18.793 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-16T01:32:18.844 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-16T01:32:19.184 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-16T01:32:19.310 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T01:32:19.433 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T01:32:19.461 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T01:32:19.552 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T01:32:19.582 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-16T01:32:19.748 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-16T01:32:19.795 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-16T01:32:19.823 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T01:32:19.927 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T01:32:19.958 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-16T01:32:20.012 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-16T01:32:20.034 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-16T01:32:20.053 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-16T01:32:20.210 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-16T01:32:20.400 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-16T01:32:20.523 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 17/164 2024-09-16T01:32:20.639 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T01:32:20.741 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T01:32:20.989 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T01:32:21.106 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-16T01:32:21.209 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-16T01:32:21.247 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-16T01:32:21.281 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-16T01:32:21.344 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-16T01:32:21.389 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-16T01:32:21.422 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-16T01:32:21.474 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-16T01:32:21.515 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-16T01:32:21.543 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-16T01:32:21.610 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-16T01:32:21.666 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-16T01:32:21.697 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-16T01:32:21.748 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-16T01:32:21.823 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-16T01:32:21.858 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 34/164 2024-09-16T01:32:21.902 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 35/164 2024-09-16T01:32:21.939 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 36/164 2024-09-16T01:32:22.009 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-16T01:32:22.062 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-16T01:32:22.135 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-16T01:32:22.267 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-16T01:32:22.331 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-16T01:32:22.474 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-16T01:32:22.591 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-16T01:32:22.634 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-16T01:32:22.700 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-16T01:32:22.733 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-16T01:32:22.759 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-16T01:32:22.811 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-16T01:32:22.849 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-16T01:32:22.934 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-16T01:32:22.964 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-16T01:32:23.007 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-16T01:32:23.044 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 46/164 2024-09-16T01:32:23.095 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-16T01:32:23.149 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-16T01:32:23.264 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-16T01:32:23.269 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T01:32:23.330 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-16T01:32:23.384 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-16T01:32:23.389 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T01:32:23.428 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T01:32:23.449 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-16T01:32:23.481 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-16T01:32:23.510 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T01:32:23.517 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-16T01:32:23.549 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-16T01:32:23.572 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-16T01:32:23.617 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-16T01:32:23.660 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-16T01:32:23.702 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-16T01:32:23.718 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-16T01:32:23.771 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-16T01:32:23.799 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T01:32:23.836 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-16T01:32:23.887 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-16T01:32:23.894 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T01:32:23.916 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-16T01:32:23.928 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-16T01:32:23.963 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-16T01:32:23.992 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-16T01:32:24.017 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-16T01:32:24.114 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-16T01:32:24.212 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-16T01:32:24.271 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-16T01:32:24.311 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-16T01:32:24.353 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 64/164 2024-09-16T01:32:24.377 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-16T01:32:24.402 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-16T01:32:24.497 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-16T01:32:24.516 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 17/164 2024-09-16T01:32:24.631 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-16T01:32:24.634 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T01:32:24.671 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-16T01:32:24.703 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T01:32:24.705 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-16T01:32:24.744 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-16T01:32:24.780 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-16T01:32:24.811 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-16T01:32:24.841 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-16T01:32:24.876 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-16T01:32:24.909 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-16T01:32:24.951 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-16T01:32:24.960 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-16T01:32:24.994 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-16T01:32:25.069 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-16T01:32:25.081 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-16T01:32:25.124 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-16T01:32:25.155 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-16T01:32:25.169 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-16T01:32:25.206 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-16T01:32:25.240 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-16T01:32:25.256 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-16T01:32:25.306 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-16T01:32:25.315 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-16T01:32:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-16T01:32:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-16T01:32:25.443 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-16T01:32:25.475 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-16T01:32:25.502 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-16T01:32:25.572 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-16T01:32:25.625 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-16T01:32:25.656 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-16T01:32:25.699 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-16T01:32:25.775 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-16T01:32:25.817 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 34/164 2024-09-16T01:32:25.861 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 35/164 2024-09-16T01:32:25.889 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 36/164 2024-09-16T01:32:25.960 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-16T01:32:26.021 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-16T01:32:26.032 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-16T01:32:26.078 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-16T01:32:26.095 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-16T01:32:26.112 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-16T01:32:26.214 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-16T01:32:26.229 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-16T01:32:26.267 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-16T01:32:26.282 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-16T01:32:26.320 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-16T01:32:26.416 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-16T01:32:26.456 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-16T01:32:26.640 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-16T01:32:26.671 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-16T01:32:26.754 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-16T01:32:26.797 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-16T01:32:26.901 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-16T01:32:26.931 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-16T01:32:26.974 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-16T01:32:27.020 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 46/164 2024-09-16T01:32:27.070 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-16T01:32:27.246 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-16T01:32:27.307 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-16T01:32:27.361 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-16T01:32:27.394 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-16T01:32:27.434 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-16T01:32:27.466 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-16T01:32:27.502 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-16T01:32:27.518 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-16T01:32:27.548 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-16T01:32:27.593 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-16T01:32:27.636 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-16T01:32:27.678 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-16T01:32:27.813 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-16T01:32:27.863 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-16T01:32:27.905 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-16T01:32:28.007 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-16T01:32:28.073 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-16T01:32:28.329 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-16T01:32:28.512 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 64/164 2024-09-16T01:32:28.651 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-16T01:32:28.759 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-16T01:32:28.859 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-16T01:32:28.887 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-16T01:32:28.928 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-16T01:32:28.939 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-16T01:32:29.075 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-16T01:32:29.111 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-16T01:32:29.139 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-16T01:32:29.187 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-16T01:32:29.229 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-16T01:32:29.246 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-16T01:32:29.260 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-16T01:32:29.290 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-16T01:32:29.291 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-16T01:32:29.325 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-16T01:32:29.366 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-16T01:32:29.382 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-16T01:32:29.403 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-16T01:32:29.413 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 102/164 2024-09-16T01:32:29.437 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-16T01:32:29.455 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-16T01:32:29.523 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-16T01:32:29.575 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-16T01:32:29.607 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-16T01:32:29.711 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-16T01:32:29.766 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-16T01:32:29.941 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-16T01:32:29.978 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-16T01:32:30.057 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-16T01:32:30.079 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 106/164 2024-09-16T01:32:30.491 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-16T01:32:30.538 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-16T01:32:30.572 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-16T01:32:30.673 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-16T01:32:30.718 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-16T01:32:30.772 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-16T01:32:30.869 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-16T01:32:30.907 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-16T01:32:31.093 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-16T01:32:31.147 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-16T01:32:31.225 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-16T01:32:31.880 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-16T01:32:32.005 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-16T01:32:32.371 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T01:32:32.400 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T01:32:32.952 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T01:32:32.987 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-16T01:32:33.078 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-16T01:32:33.079 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T01:32:33.079 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T01:32:33.079 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:32:33.113 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-16T01:32:33.264 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-16T01:32:33.350 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-16T01:32:33.378 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-16T01:32:33.411 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-16T01:32:33.722 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-16T01:32:33.766 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-16T01:32:33.857 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-16T01:32:33.888 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 102/164 2024-09-16T01:32:33.924 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-16T01:32:34.402 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-16T01:32:34.427 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-16T01:32:34.482 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-16T01:32:34.521 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 106/164 2024-09-16T01:32:36.831 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T01:32:36.860 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T01:32:37.409 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-16T01:32:37.447 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-16T01:32:37.586 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-16T01:32:37.587 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T01:32:37.587 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T01:32:37.587 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:32:37.623 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-16T01:32:44.625 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-16T01:32:44.625 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /sys 2024-09-16T01:32:44.625 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /proc 2024-09-16T01:32:44.625 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /mnt 2024-09-16T01:32:44.625 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /var/tmp 2024-09-16T01:32:44.625 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /home 2024-09-16T01:32:44.626 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /root 2024-09-16T01:32:44.626 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /tmp 2024-09-16T01:32:44.626 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:32:44.798 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-16T01:32:44.814 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-16T01:32:44.814 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:32:44.814 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T01:32:44.814 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:32:46.301 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-16T01:32:46.328 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-16T01:32:46.328 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:32:46.328 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T01:32:46.328 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T01:32:46.328 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T01:32:46.328 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:32:46.551 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-16T01:32:46.575 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-16T01:32:46.575 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:32:46.575 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T01:32:46.575 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T01:32:46.575 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T01:32:46.575 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:32:47.011 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-16T01:32:47.034 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-16T01:32:47.034 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:32:47.034 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T01:32:47.034 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T01:32:47.034 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T01:32:47.034 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:32:47.141 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-16T01:32:47.197 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-16T01:32:47.252 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-16T01:32:47.436 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-16T01:32:47.486 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-16T01:32:47.555 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-16T01:32:47.642 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-16T01:32:47.782 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-16T01:32:47.835 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-16T01:32:47.868 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-16T01:32:47.896 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-16T01:32:48.300 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-16T01:32:48.327 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-16T01:32:48.358 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 126/164 2024-09-16T01:32:48.420 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 127/164 2024-09-16T01:32:48.450 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-16T01:32:49.492 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-16T01:32:49.536 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-16T01:32:49.566 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-16T01:32:49.566 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /sys 2024-09-16T01:32:49.566 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /proc 2024-09-16T01:32:49.566 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /mnt 2024-09-16T01:32:49.566 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /var/tmp 2024-09-16T01:32:49.566 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /home 2024-09-16T01:32:49.566 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /root 2024-09-16T01:32:49.566 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /tmp 2024-09-16T01:32:49.566 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:32:49.743 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-16T01:32:49.761 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-16T01:32:49.761 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:32:49.761 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T01:32:49.761 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:32:50.494 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-16T01:32:50.527 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-16T01:32:50.627 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-16T01:32:50.797 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 131/164 2024-09-16T01:32:50.829 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-16T01:32:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-16T01:32:50.853 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:32:50.853 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T01:32:50.853 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T01:32:50.853 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T01:32:50.853 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:32:50.897 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-16T01:32:50.908 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-16T01:32:51.007 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 134/164 2024-09-16T01:32:51.066 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 135/164 2024-09-16T01:32:51.237 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-16T01:32:51.264 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-16T01:32:51.264 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:32:51.264 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T01:32:51.264 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T01:32:51.264 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T01:32:51.264 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:32:51.487 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-16T01:32:51.510 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-16T01:32:51.510 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:32:51.510 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T01:32:51.510 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T01:32:51.511 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T01:32:51.511 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:32:51.947 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-16T01:32:51.970 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-16T01:32:51.970 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:32:51.970 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T01:32:51.970 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T01:32:51.970 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T01:32:51.970 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:32:51.988 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-16T01:32:52.024 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-16T01:32:52.047 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-16T01:32:52.048 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:32:52.048 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T01:32:52.048 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T01:32:52.048 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T01:32:52.048 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:32:52.100 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-16T01:32:52.101 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-16T01:32:52.123 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-16T01:32:52.123 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:32:52.123 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T01:32:52.123 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:32:52.159 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-16T01:32:52.175 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-16T01:32:52.199 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-16T01:32:52.199 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:32:52.199 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-16T01:32:52.199 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-16T01:32:52.199 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-16T01:32:52.199 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:32:52.214 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-16T01:32:52.401 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-16T01:32:52.455 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-16T01:32:52.501 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-16T01:32:52.524 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-16T01:32:52.525 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:32:52.525 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T01:32:52.525 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T01:32:52.525 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T01:32:52.525 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:32:52.534 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-16T01:32:52.623 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-16T01:32:52.761 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-16T01:32:52.818 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-16T01:32:52.862 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-16T01:32:52.898 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-16T01:32:53.293 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-16T01:32:53.330 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-16T01:32:53.362 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 126/164 2024-09-16T01:32:53.424 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 127/164 2024-09-16T01:32:53.461 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-16T01:32:54.509 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-16T01:32:54.546 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-16T01:32:55.508 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-16T01:32:55.538 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-16T01:32:55.640 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-16T01:32:55.821 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 131/164 2024-09-16T01:32:55.849 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-16T01:32:55.872 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-16T01:32:55.873 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:32:55.873 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T01:32:55.873 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T01:32:55.873 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T01:32:55.873 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:32:55.916 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-16T01:32:55.928 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-16T01:32:56.025 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 134/164 2024-09-16T01:32:56.077 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 135/164 2024-09-16T01:32:57.008 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-16T01:32:57.044 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-16T01:32:57.067 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-16T01:32:57.067 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:32:57.067 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T01:32:57.067 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T01:32:57.067 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T01:32:57.067 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:32:57.111 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-16T01:32:57.112 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 141/164 2024-09-16T01:32:57.134 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-16T01:32:57.134 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:32:57.134 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T01:32:57.134 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:32:57.153 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 142/164 2024-09-16T01:32:57.178 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-16T01:32:57.194 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 143/164 2024-09-16T01:32:57.201 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-16T01:32:57.202 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:32:57.202 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-16T01:32:57.202 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-16T01:32:57.202 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-16T01:32:57.202 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:32:57.226 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-16T01:32:57.253 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-16T01:32:57.458 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 146/164 2024-09-16T01:32:57.500 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-16T01:32:57.504 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-16T01:32:57.527 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-16T01:32:57.527 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T01:32:57.527 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T01:32:57.527 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T01:32:57.527 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T01:32:57.527 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:32:57.530 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 148/164 2024-09-16T01:32:57.560 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 149/164 2024-09-16T01:32:57.606 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 150/164 2024-09-16T01:32:58.246 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T01:32:58.287 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T01:32:58.307 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T01:32:59.136 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 152/164 2024-09-16T01:32:59.225 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-16T01:32:59.282 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-16T01:32:59.382 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-16T01:32:59.419 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-16T01:32:59.444 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-16T01:32:59.471 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-16T01:32:59.471 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-16T01:32:59.555 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-16T01:32:59.555 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-16T01:32:59.639 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-16T01:32:59.663 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-16T01:32:59.669 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T01:32:59.669 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T01:32:59.849 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T01:32:59.849 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-16T01:32:59.933 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-16T01:32:59.998 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T01:33:02.099 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 141/164 2024-09-16T01:33:02.140 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 142/164 2024-09-16T01:33:02.181 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 143/164 2024-09-16T01:33:02.221 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-16T01:33:02.249 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-16T01:33:02.443 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 146/164 2024-09-16T01:33:02.487 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-16T01:33:02.517 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 148/164 2024-09-16T01:33:02.547 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 149/164 2024-09-16T01:33:02.585 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 150/164 2024-09-16T01:33:03.207 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T01:33:03.249 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T01:33:03.269 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-16T01:33:04.106 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 152/164 2024-09-16T01:33:04.204 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-16T01:33:04.254 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-16T01:33:04.344 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-16T01:33:04.373 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-16T01:33:04.398 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-16T01:33:04.425 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-16T01:33:04.425 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-16T01:33:04.492 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-16T01:33:04.492 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-16T01:33:04.560 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-16T01:33:04.584 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-16T01:33:04.589 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T01:33:04.590 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T01:33:04.778 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-16T01:33:04.778 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-16T01:33:04.863 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-16T01:33:04.926 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T01:33:05.713 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T01:33:05.713 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/164 2024-09-16T01:33:05.713 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2/164 2024-09-16T01:33:05.713 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 3/164 2024-09-16T01:33:05.713 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 4/164 2024-09-16T01:33:05.713 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 5/164 2024-09-16T01:33:05.714 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 6/164 2024-09-16T01:33:05.714 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 7/164 2024-09-16T01:33:05.714 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T01:33:05.714 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T01:33:05.714 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-16T01:33:05.714 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 11/164 2024-09-16T01:33:05.714 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 12/164 2024-09-16T01:33:05.714 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T01:33:05.714 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 14/164 2024-09-16T01:33:05.714 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 15/164 2024-09-16T01:33:05.714 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 16/164 2024-09-16T01:33:05.714 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 17/164 2024-09-16T01:33:05.714 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 18/164 2024-09-16T01:33:05.715 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 19/164 2024-09-16T01:33:05.715 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 20/164 2024-09-16T01:33:05.715 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 21/164 2024-09-16T01:33:05.715 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 22/164 2024-09-16T01:33:05.715 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 23/164 2024-09-16T01:33:05.715 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 24/164 2024-09-16T01:33:05.715 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 25/164 2024-09-16T01:33:05.715 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 26/164 2024-09-16T01:33:05.715 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 27/164 2024-09-16T01:33:05.715 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 28/164 2024-09-16T01:33:05.728 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 29/164 2024-09-16T01:33:05.728 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 30/164 2024-09-16T01:33:05.728 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 31/164 2024-09-16T01:33:05.728 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 32/164 2024-09-16T01:33:05.728 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 33/164 2024-09-16T01:33:05.728 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 34/164 2024-09-16T01:33:05.728 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 35/164 2024-09-16T01:33:05.728 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 36/164 2024-09-16T01:33:05.728 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 37/164 2024-09-16T01:33:05.728 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 38/164 2024-09-16T01:33:05.728 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 39/164 2024-09-16T01:33:05.729 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-16T01:33:05.729 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-16T01:33:05.729 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-16T01:33:05.729 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-16T01:33:05.729 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-16T01:33:05.729 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-16T01:33:05.729 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-16T01:33:05.729 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-16T01:33:05.729 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-16T01:33:05.729 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-16T01:33:05.729 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-16T01:33:05.729 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-16T01:33:05.729 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-16T01:33:05.730 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-16T01:33:05.730 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-16T01:33:05.730 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-16T01:33:05.730 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-16T01:33:05.730 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-16T01:33:05.730 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-16T01:33:05.730 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-16T01:33:05.730 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-16T01:33:05.730 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-16T01:33:05.730 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-16T01:33:05.730 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-16T01:33:05.730 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-16T01:33:05.731 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-16T01:33:05.731 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-16T01:33:05.731 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-16T01:33:05.731 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-16T01:33:05.731 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-16T01:33:05.731 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-16T01:33:05.731 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-16T01:33:05.731 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-16T01:33:05.731 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-16T01:33:05.731 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-16T01:33:05.731 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-16T01:33:05.731 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-16T01:33:05.731 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-16T01:33:05.732 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-16T01:33:05.732 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-16T01:33:05.732 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-16T01:33:05.732 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-16T01:33:05.732 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-16T01:33:05.732 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-16T01:33:05.732 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-16T01:33:05.732 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-16T01:33:05.732 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-16T01:33:05.732 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-16T01:33:05.732 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-16T01:33:05.732 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-16T01:33:05.732 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-16T01:33:05.733 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-16T01:33:05.733 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-16T01:33:05.733 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-16T01:33:05.733 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-16T01:33:05.733 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-16T01:33:05.733 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-16T01:33:05.733 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-16T01:33:05.733 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-16T01:33:05.733 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-16T01:33:05.733 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-16T01:33:05.733 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-16T01:33:05.733 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-16T01:33:05.733 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-16T01:33:05.733 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-16T01:33:05.734 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-16T01:33:05.734 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-16T01:33:05.734 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 107/164 2024-09-16T01:33:05.734 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-16T01:33:05.734 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-16T01:33:05.734 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-16T01:33:05.734 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-16T01:33:05.734 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-16T01:33:05.734 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-16T01:33:05.734 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-16T01:33:05.734 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-16T01:33:05.734 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-16T01:33:05.734 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-16T01:33:05.734 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-16T01:33:05.735 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-16T01:33:05.735 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-16T01:33:05.735 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-16T01:33:05.735 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-16T01:33:05.735 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-16T01:33:05.735 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-16T01:33:05.735 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-16T01:33:05.735 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-16T01:33:05.735 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-16T01:33:05.735 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-16T01:33:05.735 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-16T01:33:05.735 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-16T01:33:05.736 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-16T01:33:05.736 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-16T01:33:05.736 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-16T01:33:05.736 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-16T01:33:05.736 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-16T01:33:05.736 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-16T01:33:05.736 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-16T01:33:05.736 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-16T01:33:05.736 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-16T01:33:05.736 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-16T01:33:05.736 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-16T01:33:05.736 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-16T01:33:05.736 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-16T01:33:05.737 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-16T01:33:05.737 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-16T01:33:05.737 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-16T01:33:05.737 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-16T01:33:05.737 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-16T01:33:05.737 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 149/164 2024-09-16T01:33:05.737 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-16T01:33:05.737 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 151/164 2024-09-16T01:33:05.737 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-16T01:33:05.737 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 153/164 2024-09-16T01:33:05.737 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-16T01:33:05.737 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 155/164 2024-09-16T01:33:05.737 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-16T01:33:05.738 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 157/164 2024-09-16T01:33:05.738 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-16T01:33:05.738 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 159/164 2024-09-16T01:33:05.738 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-16T01:33:05.738 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-16T01:33:05.738 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-16T01:33:05.738 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-16T01:33:08.347 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T01:33:08.347 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:08.348 INFO:teuthology.orchestra.run.smithi088.stdout:Upgraded: 2024-09-16T01:33:08.348 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-16T01:33:08.348 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-16T01:33:08.348 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-16T01:33:08.348 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-16T01:33:08.348 INFO:teuthology.orchestra.run.smithi088.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.348 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.348 INFO:teuthology.orchestra.run.smithi088.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-16T01:33:08.348 INFO:teuthology.orchestra.run.smithi088.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-16T01:33:08.348 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-09-16T01:33:08.348 INFO:teuthology.orchestra.run.smithi088.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T01:33:08.348 INFO:teuthology.orchestra.run.smithi088.stdout: automake-1.16.2-8.el9.noarch 2024-09-16T01:33:08.348 INFO:teuthology.orchestra.run.smithi088.stdout: bison-3.7.4-5.el9.x86_64 2024-09-16T01:33:08.349 INFO:teuthology.orchestra.run.smithi088.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T01:33:08.349 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.349 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.349 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.349 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.349 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:33:08.349 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.349 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.349 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.349 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:33:08.349 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:33:08.349 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:33:08.349 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:33:08.349 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:33:08.350 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.350 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.350 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:33:08.350 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.350 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.350 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.350 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:33:08.350 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:33:08.350 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.350 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.350 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:33:08.350 INFO:teuthology.orchestra.run.smithi088.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-16T01:33:08.350 INFO:teuthology.orchestra.run.smithi088.stdout: flex-2.6.4-9.el9.x86_64 2024-09-16T01:33:08.350 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T01:33:08.350 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T01:33:08.351 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T01:33:08.351 INFO:teuthology.orchestra.run.smithi088.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-16T01:33:08.351 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T01:33:08.351 INFO:teuthology.orchestra.run.smithi088.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T01:33:08.351 INFO:teuthology.orchestra.run.smithi088.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-16T01:33:08.351 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T01:33:08.351 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T01:33:08.351 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.351 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.351 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.351 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T01:33:08.351 INFO:teuthology.orchestra.run.smithi088.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T01:33:08.351 INFO:teuthology.orchestra.run.smithi088.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T01:33:08.351 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T01:33:08.352 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T01:33:08.352 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.352 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.352 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T01:33:08.352 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.352 INFO:teuthology.orchestra.run.smithi088.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-16T01:33:08.352 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T01:33:08.352 INFO:teuthology.orchestra.run.smithi088.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-16T01:33:08.352 INFO:teuthology.orchestra.run.smithi088.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-16T01:33:08.352 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T01:33:08.352 INFO:teuthology.orchestra.run.smithi088.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T01:33:08.352 INFO:teuthology.orchestra.run.smithi088.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T01:33:08.352 INFO:teuthology.orchestra.run.smithi088.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T01:33:08.352 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T01:33:08.353 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T01:33:08.353 INFO:teuthology.orchestra.run.smithi088.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-16T01:33:08.353 INFO:teuthology.orchestra.run.smithi088.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T01:33:08.353 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-13.16-1.el9.x86_64 2024-09-16T01:33:08.353 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-16T01:33:08.353 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-16T01:33:08.353 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-16T01:33:08.353 INFO:teuthology.orchestra.run.smithi088.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T01:33:08.353 INFO:teuthology.orchestra.run.smithi088.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T01:33:08.353 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T01:33:08.353 INFO:teuthology.orchestra.run.smithi088.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-16T01:33:08.353 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T01:33:08.353 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T01:33:08.353 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T01:33:08.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T01:33:08.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T01:33:08.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T01:33:08.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T01:33:08.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-16T01:33:08.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-16T01:33:08.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T01:33:08.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T01:33:08.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T01:33:08.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T01:33:08.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T01:33:08.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T01:33:08.354 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T01:33:08.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T01:33:08.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T01:33:08.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T01:33:08.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T01:33:08.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T01:33:08.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T01:33:08.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T01:33:08.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T01:33:08.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T01:33:08.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T01:33:08.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T01:33:08.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T01:33:08.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T01:33:08.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T01:33:08.355 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T01:33:08.356 INFO:teuthology.orchestra.run.smithi088.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T01:33:08.356 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T01:33:08.356 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T01:33:08.356 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T01:33:08.356 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T01:33:08.356 INFO:teuthology.orchestra.run.smithi088.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T01:33:08.356 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T01:33:08.356 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T01:33:08.356 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T01:33:08.356 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T01:33:08.356 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-16T01:33:08.356 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T01:33:08.356 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.356 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.357 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T01:33:08.357 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T01:33:08.357 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T01:33:08.357 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.357 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T01:33:08.357 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T01:33:08.357 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T01:33:08.357 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T01:33:08.357 INFO:teuthology.orchestra.run.smithi088.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T01:33:08.357 INFO:teuthology.orchestra.run.smithi088.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T01:33:08.357 INFO:teuthology.orchestra.run.smithi088.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T01:33:08.357 INFO:teuthology.orchestra.run.smithi088.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-16T01:33:08.357 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T01:33:08.357 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T01:33:08.357 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T01:33:08.358 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T01:33:08.358 INFO:teuthology.orchestra.run.smithi088.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T01:33:08.358 INFO:teuthology.orchestra.run.smithi088.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T01:33:08.358 INFO:teuthology.orchestra.run.smithi088.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T01:33:08.358 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.358 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.358 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:08.358 INFO:teuthology.orchestra.run.smithi088.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T01:33:08.358 INFO:teuthology.orchestra.run.smithi088.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T01:33:08.358 INFO:teuthology.orchestra.run.smithi088.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T01:33:08.358 INFO:teuthology.orchestra.run.smithi088.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-16T01:33:08.358 INFO:teuthology.orchestra.run.smithi088.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-16T01:33:08.358 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T01:33:08.359 INFO:teuthology.orchestra.run.smithi088.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-16T01:33:08.359 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:08.359 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-09-16T01:33:08.778 DEBUG:teuthology.parallel:result is None 2024-09-16T01:33:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T01:33:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/164 2024-09-16T01:33:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2/164 2024-09-16T01:33:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 3/164 2024-09-16T01:33:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 4/164 2024-09-16T01:33:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 5/164 2024-09-16T01:33:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 6/164 2024-09-16T01:33:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 7/164 2024-09-16T01:33:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-16T01:33:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-16T01:33:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-16T01:33:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 11/164 2024-09-16T01:33:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 12/164 2024-09-16T01:33:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-16T01:33:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 14/164 2024-09-16T01:33:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 15/164 2024-09-16T01:33:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 16/164 2024-09-16T01:33:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 17/164 2024-09-16T01:33:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 18/164 2024-09-16T01:33:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 19/164 2024-09-16T01:33:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 20/164 2024-09-16T01:33:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 21/164 2024-09-16T01:33:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 22/164 2024-09-16T01:33:10.792 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 23/164 2024-09-16T01:33:10.792 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 24/164 2024-09-16T01:33:10.792 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 25/164 2024-09-16T01:33:10.792 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 26/164 2024-09-16T01:33:10.792 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 27/164 2024-09-16T01:33:10.792 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 28/164 2024-09-16T01:33:10.792 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 29/164 2024-09-16T01:33:10.792 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 30/164 2024-09-16T01:33:10.804 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 31/164 2024-09-16T01:33:10.804 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 32/164 2024-09-16T01:33:10.804 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 33/164 2024-09-16T01:33:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 34/164 2024-09-16T01:33:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 35/164 2024-09-16T01:33:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 36/164 2024-09-16T01:33:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 37/164 2024-09-16T01:33:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 38/164 2024-09-16T01:33:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 39/164 2024-09-16T01:33:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-16T01:33:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-16T01:33:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-16T01:33:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-16T01:33:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-16T01:33:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-16T01:33:10.806 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-16T01:33:10.806 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-16T01:33:10.806 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-16T01:33:10.806 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-16T01:33:10.806 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-16T01:33:10.806 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-16T01:33:10.806 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-16T01:33:10.806 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-16T01:33:10.806 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-16T01:33:10.806 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-16T01:33:10.806 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-16T01:33:10.806 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-16T01:33:10.806 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-16T01:33:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-16T01:33:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-16T01:33:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-16T01:33:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-16T01:33:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-16T01:33:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-16T01:33:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-16T01:33:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-16T01:33:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-16T01:33:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-16T01:33:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-16T01:33:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-16T01:33:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-16T01:33:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-16T01:33:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-16T01:33:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-16T01:33:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-16T01:33:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-16T01:33:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-16T01:33:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-16T01:33:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-16T01:33:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-16T01:33:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-16T01:33:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-16T01:33:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-16T01:33:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-16T01:33:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-16T01:33:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-16T01:33:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-16T01:33:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-16T01:33:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-16T01:33:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-16T01:33:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-16T01:33:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-16T01:33:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-16T01:33:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-16T01:33:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-16T01:33:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-16T01:33:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-16T01:33:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-16T01:33:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-16T01:33:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-16T01:33:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-16T01:33:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-16T01:33:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-16T01:33:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-16T01:33:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-16T01:33:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-16T01:33:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 107/164 2024-09-16T01:33:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-16T01:33:10.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-16T01:33:10.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-16T01:33:10.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-16T01:33:10.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-16T01:33:10.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-16T01:33:10.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-16T01:33:10.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-16T01:33:10.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-16T01:33:10.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-16T01:33:10.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-16T01:33:10.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-16T01:33:10.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-16T01:33:10.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-16T01:33:10.812 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-16T01:33:10.812 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-16T01:33:10.812 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-16T01:33:10.812 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-16T01:33:10.812 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-16T01:33:10.812 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-16T01:33:10.812 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-16T01:33:10.812 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-16T01:33:10.812 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-16T01:33:10.812 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-16T01:33:10.812 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-16T01:33:10.812 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-16T01:33:10.812 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-16T01:33:10.813 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-16T01:33:10.813 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-16T01:33:10.813 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-16T01:33:10.813 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-16T01:33:10.813 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-16T01:33:10.813 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-16T01:33:10.813 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-16T01:33:10.813 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-16T01:33:10.813 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-16T01:33:10.813 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-16T01:33:10.813 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-16T01:33:10.813 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-16T01:33:10.813 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-16T01:33:10.814 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-16T01:33:10.814 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 149/164 2024-09-16T01:33:10.814 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-16T01:33:10.814 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 151/164 2024-09-16T01:33:10.814 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-16T01:33:10.814 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 153/164 2024-09-16T01:33:10.814 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-16T01:33:10.814 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 155/164 2024-09-16T01:33:10.814 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-16T01:33:10.814 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 157/164 2024-09-16T01:33:10.814 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-16T01:33:10.814 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 159/164 2024-09-16T01:33:10.814 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-16T01:33:10.815 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-16T01:33:10.815 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-16T01:33:10.815 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-16T01:33:13.596 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-16T01:33:13.596 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:13.597 INFO:teuthology.orchestra.run.smithi154.stdout:Upgraded: 2024-09-16T01:33:13.597 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-16T01:33:13.597 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-16T01:33:13.597 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-16T01:33:13.597 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-16T01:33:13.597 INFO:teuthology.orchestra.run.smithi154.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.597 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.597 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-16T01:33:13.597 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-16T01:33:13.597 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-16T01:33:13.597 INFO:teuthology.orchestra.run.smithi154.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T01:33:13.597 INFO:teuthology.orchestra.run.smithi154.stdout: automake-1.16.2-8.el9.noarch 2024-09-16T01:33:13.598 INFO:teuthology.orchestra.run.smithi154.stdout: bison-3.7.4-5.el9.x86_64 2024-09-16T01:33:13.598 INFO:teuthology.orchestra.run.smithi154.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T01:33:13.598 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.598 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.598 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.598 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.598 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:33:13.598 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.598 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.598 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.598 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:33:13.598 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:33:13.599 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:33:13.599 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:33:13.599 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:33:13.599 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.599 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.599 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:33:13.599 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.599 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.599 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.599 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:33:13.599 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:33:13.599 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.600 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.600 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-16T01:33:13.600 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-16T01:33:13.600 INFO:teuthology.orchestra.run.smithi154.stdout: flex-2.6.4-9.el9.x86_64 2024-09-16T01:33:13.600 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T01:33:13.600 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T01:33:13.600 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T01:33:13.600 INFO:teuthology.orchestra.run.smithi154.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-16T01:33:13.600 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T01:33:13.600 INFO:teuthology.orchestra.run.smithi154.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T01:33:13.600 INFO:teuthology.orchestra.run.smithi154.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-16T01:33:13.600 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T01:33:13.600 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T01:33:13.601 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.601 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.601 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.601 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T01:33:13.601 INFO:teuthology.orchestra.run.smithi154.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T01:33:13.601 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T01:33:13.601 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T01:33:13.601 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T01:33:13.601 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.601 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.601 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T01:33:13.601 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.601 INFO:teuthology.orchestra.run.smithi154.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-16T01:33:13.602 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T01:33:13.602 INFO:teuthology.orchestra.run.smithi154.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-16T01:33:13.602 INFO:teuthology.orchestra.run.smithi154.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-16T01:33:13.602 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T01:33:13.602 INFO:teuthology.orchestra.run.smithi154.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T01:33:13.602 INFO:teuthology.orchestra.run.smithi154.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T01:33:13.602 INFO:teuthology.orchestra.run.smithi154.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T01:33:13.602 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T01:33:13.602 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T01:33:13.602 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-16T01:33:13.602 INFO:teuthology.orchestra.run.smithi154.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T01:33:13.602 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-13.16-1.el9.x86_64 2024-09-16T01:33:13.602 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-16T01:33:13.603 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-16T01:33:13.603 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-16T01:33:13.603 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T01:33:13.603 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T01:33:13.603 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T01:33:13.603 INFO:teuthology.orchestra.run.smithi154.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-16T01:33:13.603 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T01:33:13.603 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T01:33:13.603 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T01:33:13.603 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.603 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.603 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.603 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T01:33:13.603 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T01:33:13.604 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T01:33:13.604 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T01:33:13.604 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-16T01:33:13.604 INFO:teuthology.orchestra.run.smithi154.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-16T01:33:13.604 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T01:33:13.604 INFO:teuthology.orchestra.run.smithi154.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T01:33:13.604 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T01:33:13.604 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T01:33:13.604 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T01:33:13.604 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T01:33:13.604 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T01:33:13.604 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T01:33:13.604 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T01:33:13.605 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T01:33:13.605 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T01:33:13.605 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T01:33:13.605 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T01:33:13.605 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T01:33:13.605 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T01:33:13.605 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T01:33:13.605 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T01:33:13.605 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T01:33:13.605 INFO:teuthology.orchestra.run.smithi154.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T01:33:13.605 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T01:33:13.605 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T01:33:13.605 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T01:33:13.606 INFO:teuthology.orchestra.run.smithi154.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T01:33:13.606 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T01:33:13.606 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T01:33:13.606 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T01:33:13.606 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T01:33:13.606 INFO:teuthology.orchestra.run.smithi154.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T01:33:13.606 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T01:33:13.606 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T01:33:13.606 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T01:33:13.606 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T01:33:13.606 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-16T01:33:13.606 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T01:33:13.606 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.606 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.607 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T01:33:13.607 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T01:33:13.607 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T01:33:13.607 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.607 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T01:33:13.607 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T01:33:13.607 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T01:33:13.607 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T01:33:13.607 INFO:teuthology.orchestra.run.smithi154.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T01:33:13.607 INFO:teuthology.orchestra.run.smithi154.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T01:33:13.607 INFO:teuthology.orchestra.run.smithi154.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T01:33:13.607 INFO:teuthology.orchestra.run.smithi154.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-16T01:33:13.608 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T01:33:13.608 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T01:33:13.608 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T01:33:13.608 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T01:33:13.608 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T01:33:13.608 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T01:33:13.608 INFO:teuthology.orchestra.run.smithi154.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T01:33:13.608 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.608 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.608 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-16T01:33:13.608 INFO:teuthology.orchestra.run.smithi154.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T01:33:13.608 INFO:teuthology.orchestra.run.smithi154.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T01:33:13.608 INFO:teuthology.orchestra.run.smithi154.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T01:33:13.609 INFO:teuthology.orchestra.run.smithi154.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-16T01:33:13.609 INFO:teuthology.orchestra.run.smithi154.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-16T01:33:13.609 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T01:33:13.609 INFO:teuthology.orchestra.run.smithi154.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-16T01:33:13.609 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:13.609 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-16T01:33:14.016 DEBUG:teuthology.parallel:result is None 2024-09-16T01:33:14.017 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T01:33:14.162 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T01:33:14.188 INFO:teuthology.orchestra.run.smithi088.stdout:19.1.1-326.g8921c8ef.el9 2024-09-16T01:33:14.189 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-16T01:33:14.189 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-16T01:33:14.191 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T01:33:14.334 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T01:33:14.356 INFO:teuthology.orchestra.run.smithi154.stdout:19.1.1-326.g8921c8ef.el9 2024-09-16T01:33:14.357 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-16T01:33:14.357 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-16T01:33:14.359 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T01:33:14.359 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:33:14.359 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T01:33:14.391 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:33:14.392 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T01:33:14.425 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T01:33:14.425 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:33:14.425 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T01:33:14.461 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T01:33:14.528 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:33:14.528 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T01:33:14.554 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T01:33:14.619 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T01:33:14.620 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:33:14.620 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T01:33:14.646 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T01:33:14.711 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:33:14.711 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T01:33:14.737 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T01:33:14.801 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T01:33:14.802 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:33:14.802 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T01:33:14.828 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T01:33:14.894 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:33:14.894 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T01:33:14.918 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T01:33:14.983 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T01:33:15.053 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T01:33:15.053 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 777 /var/log/ceph 2024-09-16T01:33:15.055 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 777 /var/log/ceph 2024-09-16T01:33:15.079 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T01:33:15.079 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T01:33:15.121 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T01:33:15.146 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T01:33:15.146 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T01:33:15.186 DEBUG:teuthology.orchestra.run.smithi154:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T01:33:15.216 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-09-16T01:33:15.216 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-09-16T01:33:15.217 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-09-16T01:33:15.217 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:33:15.217 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-16T01:33:15.265 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-16T01:33:15.376 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-09-16T01:33:15.400 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:33:15.401 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-16T01:33:15.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-16T01:33:15.527 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-09-16T01:33:15.556 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T01:33:15.556 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'bad backtrace', 'object missing on disk', 'error reading table object', 'error reading sessionmap', 'Error loading MDS rank', 'missing journal object', 'Error recovering journal', 'error decoding table object', 'failed to read JournalPointer', 'Corrupt directory entry', 'Corrupt fnode header', 'corrupt sessionmap header', 'Corrupt dentry', 'Scrub error on inode', 'Metadata damage detected', 'MDS_READ_ONLY', 'force file system read-only', 'with standby daemon mds', 'MDS abort because newly corrupt dentry'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T01:33:15.557 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906477', 'branch': 'squid', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/damage}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7906477', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'bad backtrace', 'object missing on disk', 'error reading table object', 'error reading sessionmap', 'Error loading MDS rank', 'missing journal object', 'Error recovering journal', 'error decoding table object', 'failed to read JournalPointer', 'Corrupt directory entry', 'Corrupt fnode header', 'corrupt sessionmap header', 'Corrupt dentry', 'Scrub error on inode', 'Metadata damage detected', 'MDS_READ_ONLY', 'force file system read-only', 'with standby daemon mds', 'MDS abort because newly corrupt dentry'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 1, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 3393, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'sleep_before_teardown': 0, 'subset': '294/512', 'suite': 'fs', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi088.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJf1XKuP5OBdoKK+dsWzt525t5mpZRRcUek1q/bTQo6bn4Ymh40GssGAU6et8JmZObOPR8vvLFEJgZaS2JlYE2M=', 'smithi154.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGIUUvQMJPOWQntC2askAfxyGFKNXJ75frno6BkG+7XTKf0coPjm/L1oRi78clEDplLj4LZz5B/qYO/4/Ygp37U='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'bad backtrace', 'object missing on disk', 'error reading table object', 'error reading sessionmap', 'Error loading MDS rank', 'missing journal object', 'Error recovering journal', 'error decoding table object', 'failed to read JournalPointer', 'Corrupt directory entry', 'Corrupt fnode header', 'corrupt sessionmap header', 'Corrupt dentry', 'Scrub error on inode', 'Metadata damage detected', 'MDS_READ_ONLY', 'force file system read-only', 'with standby daemon mds', 'MDS abort because newly corrupt dentry'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_damage']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_21:24:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T01:33:15.557 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T01:33:15.559 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T01:33:15.607 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T01:33:15.615 DEBUG:teuthology.orchestra.run.smithi154:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T01:33:15.672 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:33:15.672 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T01:33:15.693 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T01:33:15.693 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_1 2024-09-16T01:33:15.749 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T01:33:15.749 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:33:15.749 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-16T01:33:15.749 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:33:15.749 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:33:15.749 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-09-16 01:33:00.376839123 +0000 2024-09-16T01:33:15.749 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-09-16 01:31:14.727172867 +0000 2024-09-16T01:33:15.749 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-09-16 01:31:14.727172867 +0000 2024-09-16T01:33:15.749 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-09-16 01:31:14.727172867 +0000 2024-09-16T01:33:15.749 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T01:33:15.812 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-09-16T01:33:15.812 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-09-16T01:33:15.812 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000207886 s, 2.5 MB/s 2024-09-16T01:33:15.813 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T01:33:15.870 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_2 2024-09-16T01:33:15.926 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T01:33:15.926 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:33:15.926 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-16T01:33:15.926 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:33:15.927 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:33:15.927 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-09-16 01:33:00.377839104 +0000 2024-09-16T01:33:15.927 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-09-16 01:31:15.053166426 +0000 2024-09-16T01:33:15.927 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-09-16 01:31:15.053166426 +0000 2024-09-16T01:33:15.927 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-09-16 01:31:15.053166426 +0000 2024-09-16T01:33:15.927 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T01:33:15.989 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-09-16T01:33:15.989 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-09-16T01:33:15.989 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000195375 s, 2.6 MB/s 2024-09-16T01:33:15.990 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T01:33:16.046 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_3 2024-09-16T01:33:16.102 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T01:33:16.102 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:33:16.102 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-16T01:33:16.102 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:33:16.102 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:33:16.103 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-09-16 01:33:00.377839104 +0000 2024-09-16T01:33:16.103 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-09-16 01:31:15.382159927 +0000 2024-09-16T01:33:16.103 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-09-16 01:31:15.382159927 +0000 2024-09-16T01:33:16.103 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-09-16 01:31:15.382159927 +0000 2024-09-16T01:33:16.103 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T01:33:16.165 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-09-16T01:33:16.165 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-09-16T01:33:16.165 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000181538 s, 2.8 MB/s 2024-09-16T01:33:16.166 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T01:33:16.223 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_4 2024-09-16T01:33:16.278 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T01:33:16.278 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:33:16.278 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-16T01:33:16.279 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:33:16.279 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:33:16.279 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-09-16 01:33:00.377839104 +0000 2024-09-16T01:33:16.279 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-09-16 01:31:15.716153330 +0000 2024-09-16T01:33:16.279 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-09-16 01:31:15.716153330 +0000 2024-09-16T01:33:16.279 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-09-16 01:31:15.716153330 +0000 2024-09-16T01:33:16.279 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T01:33:16.342 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-09-16T01:33:16.342 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-09-16T01:33:16.342 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000165753 s, 3.1 MB/s 2024-09-16T01:33:16.343 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T01:33:16.399 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T01:33:16.399 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:33:16.399 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T01:33:16.417 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T01:33:16.417 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_1 2024-09-16T01:33:16.475 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T01:33:16.475 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:33:16.475 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-16T01:33:16.475 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:33:16.475 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:33:16.475 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-09-16 01:33:05.314622229 +0000 2024-09-16T01:33:16.475 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-09-16 01:31:14.729491646 +0000 2024-09-16T01:33:16.475 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-09-16 01:31:14.729491646 +0000 2024-09-16T01:33:16.475 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-09-16 01:31:14.729491646 +0000 2024-09-16T01:33:16.476 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T01:33:16.541 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-09-16T01:33:16.542 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-09-16T01:33:16.542 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000233784 s, 2.2 MB/s 2024-09-16T01:33:16.543 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T01:33:16.601 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_2 2024-09-16T01:33:16.659 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T01:33:16.659 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:33:16.659 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-16T01:33:16.659 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:33:16.659 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:33:16.659 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-09-16 01:33:05.315622202 +0000 2024-09-16T01:33:16.660 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-09-16 01:31:15.060482790 +0000 2024-09-16T01:33:16.660 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-09-16 01:31:15.060482790 +0000 2024-09-16T01:33:16.660 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-09-16 01:31:15.060482790 +0000 2024-09-16T01:33:16.660 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T01:33:16.723 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-09-16T01:33:16.723 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-09-16T01:33:16.724 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000194359 s, 2.6 MB/s 2024-09-16T01:33:16.725 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T01:33:16.782 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_3 2024-09-16T01:33:16.838 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T01:33:16.838 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:33:16.839 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-16T01:33:16.839 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:33:16.839 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:33:16.839 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-09-16 01:33:05.315622202 +0000 2024-09-16T01:33:16.839 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-09-16 01:31:15.393473882 +0000 2024-09-16T01:33:16.840 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-09-16 01:31:15.393473882 +0000 2024-09-16T01:33:16.840 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-09-16 01:31:15.393473882 +0000 2024-09-16T01:33:16.840 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T01:33:16.903 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-09-16T01:33:16.903 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-09-16T01:33:16.903 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000161 s, 3.2 MB/s 2024-09-16T01:33:16.904 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T01:33:16.961 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_4 2024-09-16T01:33:17.017 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T01:33:17.017 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T01:33:17.018 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-16T01:33:17.018 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T01:33:17.018 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T01:33:17.018 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-09-16 01:33:05.315622202 +0000 2024-09-16T01:33:17.018 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-09-16 01:31:15.741464571 +0000 2024-09-16T01:33:17.018 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-09-16 01:31:15.741464571 +0000 2024-09-16T01:33:17.018 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-09-16 01:31:15.741464571 +0000 2024-09-16T01:33:17.018 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T01:33:17.081 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-09-16T01:33:17.081 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-09-16T01:33:17.081 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000164827 s, 3.1 MB/s 2024-09-16T01:33:17.082 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T01:33:17.138 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T01:33:17.138 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi088.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi154.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-16T01:33:17.139 INFO:tasks.ceph:Generating config... 2024-09-16T01:33:17.140 INFO:tasks.ceph:[client] client mount timeout = 600 2024-09-16T01:33:17.140 INFO:tasks.ceph:[client] debug client = 20 2024-09-16T01:33:17.140 INFO:tasks.ceph:[client] debug ms = 1 2024-09-16T01:33:17.140 INFO:tasks.ceph:[client] pid file = 2024-09-16T01:33:17.140 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-09-16T01:33:17.140 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-09-16T01:33:17.140 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-09-16T01:33:17.140 INFO:tasks.ceph:[mds] debug mds = 20 2024-09-16T01:33:17.140 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-09-16T01:33:17.140 INFO:tasks.ceph:[mds] debug ms = 1 2024-09-16T01:33:17.141 INFO:tasks.ceph:[mds] mds debug frag = True 2024-09-16T01:33:17.141 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-09-16T01:33:17.141 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-09-16T01:33:17.141 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-09-16T01:33:17.141 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-09-16T01:33:17.141 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-09-16T01:33:17.141 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-09-16T01:33:17.141 INFO:tasks.ceph:[mgr] debug client = 20 2024-09-16T01:33:17.141 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T01:33:17.141 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T01:33:17.141 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T01:33:17.141 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T01:33:17.141 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T01:33:17.141 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-09-16T01:33:17.142 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-09-16T01:33:17.142 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-09-16T01:33:17.142 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-16T01:33:17.142 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-16T01:33:17.142 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-09-16T01:33:17.142 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T01:33:17.142 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T01:33:17.142 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-16T01:33:17.142 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-16T01:33:17.142 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T01:33:17.142 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T01:33:17.142 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-16T01:33:17.142 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T01:33:17.142 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T01:33:17.143 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T01:33:17.143 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T01:33:17.143 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T01:33:17.143 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-09-16T01:33:17.143 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T01:33:17.143 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T01:33:17.194 INFO:teuthology.orchestra.run.smithi088.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T01:33:17.197 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T01:33:17.241 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T01:33:17.306 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.88'), ('mon.b', '172.21.15.154'), ('mon.c', '[v2:172.21.15.154:3301,v1:172.21.15.154:6790]')] 2024-09-16T01:33:17.307 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.88,172.21.15.154,[v2:172.21.15.154:3301,v1:172.21.15.154:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-09-16T01:33:17.308 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:33:17.308 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T01:33:17.363 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.88 --add b 172.21.15.154 --addv c '[v2:172.21.15.154:3301,v1:172.21.15.154:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:33:17.449 INFO:teuthology.orchestra.run.smithi088.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:33:17.450 INFO:teuthology.orchestra.run.smithi088.stdout:monmaptool: generated fsid 5ace2c91-46f9-482b-8a89-731fbdc171bf 2024-09-16T01:33:17.450 INFO:teuthology.orchestra.run.smithi088.stdout:setting min_mon_release = quincy 2024-09-16T01:33:17.450 INFO:teuthology.orchestra.run.smithi088.stdout:epoch 0 2024-09-16T01:33:17.450 INFO:teuthology.orchestra.run.smithi088.stdout:fsid 5ace2c91-46f9-482b-8a89-731fbdc171bf 2024-09-16T01:33:17.450 INFO:teuthology.orchestra.run.smithi088.stdout:last_changed 2024-09-16T01:33:17.449370+0000 2024-09-16T01:33:17.450 INFO:teuthology.orchestra.run.smithi088.stdout:created 2024-09-16T01:33:17.449370+0000 2024-09-16T01:33:17.450 INFO:teuthology.orchestra.run.smithi088.stdout:min_mon_release 17 (quincy) 2024-09-16T01:33:17.450 INFO:teuthology.orchestra.run.smithi088.stdout:election_strategy: 1 2024-09-16T01:33:17.450 INFO:teuthology.orchestra.run.smithi088.stdout:0: [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon.a 2024-09-16T01:33:17.450 INFO:teuthology.orchestra.run.smithi088.stdout:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T01:33:17.450 INFO:teuthology.orchestra.run.smithi088.stdout:2: [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] mon.c 2024-09-16T01:33:17.450 INFO:teuthology.orchestra.run.smithi088.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-16T01:33:17.452 DEBUG:teuthology.orchestra.run.smithi088:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T01:33:17.467 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 5ace2c91-46f9-482b-8a89-731fbdc171bf... 2024-09-16T01:33:17.468 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T01:33:17.509 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T01:33:17.549 INFO:teuthology.orchestra.run.smithi088.stdout:[global] 2024-09-16T01:33:17.549 INFO:teuthology.orchestra.run.smithi088.stdout: chdir = "" 2024-09-16T01:33:17.549 INFO:teuthology.orchestra.run.smithi088.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T01:33:17.549 INFO:teuthology.orchestra.run.smithi088.stdout: auth supported = cephx 2024-09-16T01:33:17.549 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.550 INFO:teuthology.orchestra.run.smithi088.stdout: filestore xattr use omap = true 2024-09-16T01:33:17.550 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.550 INFO:teuthology.orchestra.run.smithi088.stdout: mon clock drift allowed = 1.000 2024-09-16T01:33:17.550 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.550 INFO:teuthology.orchestra.run.smithi088.stdout: osd crush chooseleaf type = 0 2024-09-16T01:33:17.550 INFO:teuthology.orchestra.run.smithi088.stdout: auth debug = true 2024-09-16T01:33:17.550 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.550 INFO:teuthology.orchestra.run.smithi088.stdout: ms die on old message = true 2024-09-16T01:33:17.550 INFO:teuthology.orchestra.run.smithi088.stdout: ms die on bug = true 2024-09-16T01:33:17.550 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.550 INFO:teuthology.orchestra.run.smithi088.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T01:33:17.550 INFO:teuthology.orchestra.run.smithi088.stdout: mon pg warn max object skew = 0 2024-09-16T01:33:17.550 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.551 INFO:teuthology.orchestra.run.smithi088.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T01:33:17.551 INFO:teuthology.orchestra.run.smithi088.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T01:33:17.551 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.551 INFO:teuthology.orchestra.run.smithi088.stdout: osd pool default size = 2 2024-09-16T01:33:17.551 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.551 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd allow primary affinity = true 2024-09-16T01:33:17.551 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd allow pg remap = true 2024-09-16T01:33:17.551 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on legacy crush tunables = false 2024-09-16T01:33:17.551 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on crush straw calc version zero = false 2024-09-16T01:33:17.551 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on no sortbitwise = false 2024-09-16T01:33:17.551 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on osd down out interval zero = false 2024-09-16T01:33:17.552 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on too few osds = false 2024-09-16T01:33:17.552 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T01:33:17.552 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T01:33:17.552 INFO:teuthology.orchestra.run.smithi088.stdout: mon_allow_pool_size_one = true 2024-09-16T01:33:17.552 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.552 INFO:teuthology.orchestra.run.smithi088.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T01:33:17.552 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.552 INFO:teuthology.orchestra.run.smithi088.stdout: osd default data pool replay window = 5 2024-09-16T01:33:17.552 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.552 INFO:teuthology.orchestra.run.smithi088.stdout: mon allow pool delete = true 2024-09-16T01:33:17.552 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.553 INFO:teuthology.orchestra.run.smithi088.stdout: mon cluster log file level = debug 2024-09-16T01:33:17.553 INFO:teuthology.orchestra.run.smithi088.stdout: debug asserts on shutdown = true 2024-09-16T01:33:17.553 INFO:teuthology.orchestra.run.smithi088.stdout: mon health detail to clog = false 2024-09-16T01:33:17.553 INFO:teuthology.orchestra.run.smithi088.stdout: mon host = "172.21.15.88,172.21.15.154,[v2:172.21.15.154:3301,v1:172.21.15.154:6790]" 2024-09-16T01:33:17.553 INFO:teuthology.orchestra.run.smithi088.stdout: ms die on skipped message = False 2024-09-16T01:33:17.553 INFO:teuthology.orchestra.run.smithi088.stdout: fsid = 5ace2c91-46f9-482b-8a89-731fbdc171bf 2024-09-16T01:33:17.553 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.553 INFO:teuthology.orchestra.run.smithi088.stdout:[osd] 2024-09-16T01:33:17.553 INFO:teuthology.orchestra.run.smithi088.stdout: osd journal size = 100 2024-09-16T01:33:17.553 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.553 INFO:teuthology.orchestra.run.smithi088.stdout: osd scrub load threshold = 5.0 2024-09-16T01:33:17.553 INFO:teuthology.orchestra.run.smithi088.stdout: osd scrub max interval = 600 2024-09-16T01:33:17.553 INFO:teuthology.orchestra.run.smithi088.stdout: osd mclock profile = high_recovery_ops 2024-09-16T01:33:17.553 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.553 INFO:teuthology.orchestra.run.smithi088.stdout: osd recover clone overlap = true 2024-09-16T01:33:17.554 INFO:teuthology.orchestra.run.smithi088.stdout: osd recovery max chunk = 1048576 2024-09-16T01:33:17.554 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.554 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug shutdown = true 2024-09-16T01:33:17.554 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug op order = true 2024-09-16T01:33:17.554 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug verify stray on activate = true 2024-09-16T01:33:17.554 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug trim objects = true 2024-09-16T01:33:17.554 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.554 INFO:teuthology.orchestra.run.smithi088.stdout: osd open classes on start = true 2024-09-16T01:33:17.554 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug pg log writeout = true 2024-09-16T01:33:17.554 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.554 INFO:teuthology.orchestra.run.smithi088.stdout: osd deep scrub update digest min age = 30 2024-09-16T01:33:17.554 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.554 INFO:teuthology.orchestra.run.smithi088.stdout: osd map max advance = 10 2024-09-16T01:33:17.554 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.554 INFO:teuthology.orchestra.run.smithi088.stdout: journal zero on create = true 2024-09-16T01:33:17.555 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.555 INFO:teuthology.orchestra.run.smithi088.stdout: filestore ondisk finisher threads = 3 2024-09-16T01:33:17.555 INFO:teuthology.orchestra.run.smithi088.stdout: filestore apply finisher threads = 3 2024-09-16T01:33:17.555 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.555 INFO:teuthology.orchestra.run.smithi088.stdout: bdev debug aio = true 2024-09-16T01:33:17.555 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug misdirected ops = true 2024-09-16T01:33:17.555 INFO:teuthology.orchestra.run.smithi088.stdout: bdev async discard = True 2024-09-16T01:33:17.555 INFO:teuthology.orchestra.run.smithi088.stdout: bdev enable discard = True 2024-09-16T01:33:17.555 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore allocator = bitmap 2024-09-16T01:33:17.555 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore block size = 96636764160 2024-09-16T01:33:17.555 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore fsck on mount = True 2024-09-16T01:33:17.555 INFO:teuthology.orchestra.run.smithi088.stdout: debug bluefs = 1/20 2024-09-16T01:33:17.555 INFO:teuthology.orchestra.run.smithi088.stdout: debug bluestore = 1/20 2024-09-16T01:33:17.556 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2024-09-16T01:33:17.556 INFO:teuthology.orchestra.run.smithi088.stdout: debug osd = 20 2024-09-16T01:33:17.556 INFO:teuthology.orchestra.run.smithi088.stdout: debug rocksdb = 4/10 2024-09-16T01:33:17.556 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T01:33:17.556 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd full ratio = 0.9 2024-09-16T01:33:17.556 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd nearfull ratio = 0.8 2024-09-16T01:33:17.556 INFO:teuthology.orchestra.run.smithi088.stdout: osd failsafe full ratio = 0.95 2024-09-16T01:33:17.556 INFO:teuthology.orchestra.run.smithi088.stdout: osd objectstore = bluestore 2024-09-16T01:33:17.556 INFO:teuthology.orchestra.run.smithi088.stdout: osd op complaint time = 180 2024-09-16T01:33:17.557 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.557 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr] 2024-09-16T01:33:17.557 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2024-09-16T01:33:17.557 INFO:teuthology.orchestra.run.smithi088.stdout: debug mgr = 20 2024-09-16T01:33:17.557 INFO:teuthology.orchestra.run.smithi088.stdout: debug mon = 20 2024-09-16T01:33:17.557 INFO:teuthology.orchestra.run.smithi088.stdout: debug auth = 20 2024-09-16T01:33:17.557 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min pgs per osd = 4 2024-09-16T01:33:17.557 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min bytes per osd = 10 2024-09-16T01:33:17.557 INFO:teuthology.orchestra.run.smithi088.stdout: mgr/telemetry/nag = false 2024-09-16T01:33:17.557 INFO:teuthology.orchestra.run.smithi088.stdout: debug client = 20 2024-09-16T01:33:17.557 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.557 INFO:teuthology.orchestra.run.smithi088.stdout:[mon] 2024-09-16T01:33:17.558 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2024-09-16T01:33:17.558 INFO:teuthology.orchestra.run.smithi088.stdout: debug mon = 20 2024-09-16T01:33:17.558 INFO:teuthology.orchestra.run.smithi088.stdout: debug paxos = 20 2024-09-16T01:33:17.558 INFO:teuthology.orchestra.run.smithi088.stdout: debug auth = 20 2024-09-16T01:33:17.558 INFO:teuthology.orchestra.run.smithi088.stdout: mon data avail warn = 5 2024-09-16T01:33:17.558 INFO:teuthology.orchestra.run.smithi088.stdout: mon mgr mkfs grace = 240 2024-09-16T01:33:17.558 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min pgs per osd = 4 2024-09-16T01:33:17.558 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd reporter subtree level = osd 2024-09-16T01:33:17.558 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd prime pg temp = true 2024-09-16T01:33:17.558 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min bytes per osd = 10 2024-09-16T01:33:17.558 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.558 INFO:teuthology.orchestra.run.smithi088.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T01:33:17.559 INFO:teuthology.orchestra.run.smithi088.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T01:33:17.559 INFO:teuthology.orchestra.run.smithi088.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T01:33:17.559 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.559 INFO:teuthology.orchestra.run.smithi088.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T01:33:17.559 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T01:33:17.559 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T01:33:17.559 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.559 INFO:teuthology.orchestra.run.smithi088.stdout: # 1m isn't quite enough 2024-09-16T01:33:17.559 INFO:teuthology.orchestra.run.smithi088.stdout: mon_down_mkfs_grace = 2m 2024-09-16T01:33:17.559 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.559 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_filestore_osds = false 2024-09-16T01:33:17.559 INFO:teuthology.orchestra.run.smithi088.stdout: mon down mkfs grace = 300 2024-09-16T01:33:17.559 INFO:teuthology.orchestra.run.smithi088.stdout: mon op complaint time = 120 2024-09-16T01:33:17.559 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on pool no app = False 2024-09-16T01:33:17.560 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:33:17.560 INFO:teuthology.orchestra.run.smithi088.stdout:[client] 2024-09-16T01:33:17.560 INFO:teuthology.orchestra.run.smithi088.stdout: rgw cache enabled = true 2024-09-16T01:33:17.560 INFO:teuthology.orchestra.run.smithi088.stdout: rgw enable ops log = true 2024-09-16T01:33:17.560 INFO:teuthology.orchestra.run.smithi088.stdout: rgw enable usage log = true 2024-09-16T01:33:17.560 INFO:teuthology.orchestra.run.smithi088.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T01:33:17.560 INFO:teuthology.orchestra.run.smithi088.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T01:33:17.560 INFO:teuthology.orchestra.run.smithi088.stdout: client mount timeout = 600 2024-09-16T01:33:17.560 INFO:teuthology.orchestra.run.smithi088.stdout: debug client = 20 2024-09-16T01:33:17.560 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2024-09-16T01:33:17.560 INFO:teuthology.orchestra.run.smithi088.stdout: pid file = "" 2024-09-16T01:33:17.561 INFO:teuthology.orchestra.run.smithi088.stdout: rados mon op timeout = 900 2024-09-16T01:33:17.561 INFO:teuthology.orchestra.run.smithi088.stdout: rados osd op timeout = 900 2024-09-16T01:33:17.561 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.a] 2024-09-16T01:33:17.561 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.b] 2024-09-16T01:33:17.561 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.c] 2024-09-16T01:33:17.561 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.a] 2024-09-16T01:33:17.561 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.c] 2024-09-16T01:33:17.561 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.b] 2024-09-16T01:33:17.562 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.d] 2024-09-16T01:33:17.562 INFO:teuthology.orchestra.run.smithi088.stdout:[mds] 2024-09-16T01:33:17.562 INFO:teuthology.orchestra.run.smithi088.stdout: debug mds = 20 2024-09-16T01:33:17.562 INFO:teuthology.orchestra.run.smithi088.stdout: debug mds balancer = 20 2024-09-16T01:33:17.562 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2024-09-16T01:33:17.562 INFO:teuthology.orchestra.run.smithi088.stdout: mds debug frag = True 2024-09-16T01:33:17.562 INFO:teuthology.orchestra.run.smithi088.stdout: mds debug scatterstat = True 2024-09-16T01:33:17.562 INFO:teuthology.orchestra.run.smithi088.stdout: mds op complaint time = 180 2024-09-16T01:33:17.562 INFO:teuthology.orchestra.run.smithi088.stdout: mds verify scatter = True 2024-09-16T01:33:17.562 INFO:teuthology.orchestra.run.smithi088.stdout: osd op complaint time = 180 2024-09-16T01:33:17.562 INFO:teuthology.orchestra.run.smithi088.stdout: rados mon op timeout = 900 2024-09-16T01:33:17.562 INFO:teuthology.orchestra.run.smithi088.stdout: rados osd op timeout = 900 2024-09-16T01:33:17.563 INFO:teuthology.orchestra.run.smithi154.stdout:[global] 2024-09-16T01:33:17.563 INFO:teuthology.orchestra.run.smithi154.stdout: chdir = "" 2024-09-16T01:33:17.563 INFO:teuthology.orchestra.run.smithi154.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T01:33:17.563 INFO:teuthology.orchestra.run.smithi154.stdout: auth supported = cephx 2024-09-16T01:33:17.563 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.563 INFO:teuthology.orchestra.run.smithi154.stdout: filestore xattr use omap = true 2024-09-16T01:33:17.563 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.563 INFO:teuthology.orchestra.run.smithi154.stdout: mon clock drift allowed = 1.000 2024-09-16T01:33:17.563 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.563 INFO:teuthology.orchestra.run.smithi154.stdout: osd crush chooseleaf type = 0 2024-09-16T01:33:17.564 INFO:teuthology.orchestra.run.smithi154.stdout: auth debug = true 2024-09-16T01:33:17.564 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.564 INFO:teuthology.orchestra.run.smithi154.stdout: ms die on old message = true 2024-09-16T01:33:17.564 INFO:teuthology.orchestra.run.smithi154.stdout: ms die on bug = true 2024-09-16T01:33:17.564 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.564 INFO:teuthology.orchestra.run.smithi154.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T01:33:17.564 INFO:teuthology.orchestra.run.smithi154.stdout: mon pg warn max object skew = 0 2024-09-16T01:33:17.564 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.564 INFO:teuthology.orchestra.run.smithi154.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T01:33:17.564 INFO:teuthology.orchestra.run.smithi154.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T01:33:17.564 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.564 INFO:teuthology.orchestra.run.smithi154.stdout: osd pool default size = 2 2024-09-16T01:33:17.564 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.565 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd allow primary affinity = true 2024-09-16T01:33:17.565 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd allow pg remap = true 2024-09-16T01:33:17.565 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on legacy crush tunables = false 2024-09-16T01:33:17.565 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on crush straw calc version zero = false 2024-09-16T01:33:17.565 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on no sortbitwise = false 2024-09-16T01:33:17.565 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on osd down out interval zero = false 2024-09-16T01:33:17.565 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on too few osds = false 2024-09-16T01:33:17.565 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T01:33:17.565 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T01:33:17.565 INFO:teuthology.orchestra.run.smithi154.stdout: mon_allow_pool_size_one = true 2024-09-16T01:33:17.565 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.565 INFO:teuthology.orchestra.run.smithi154.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T01:33:17.565 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.565 INFO:teuthology.orchestra.run.smithi154.stdout: osd default data pool replay window = 5 2024-09-16T01:33:17.566 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.566 INFO:teuthology.orchestra.run.smithi154.stdout: mon allow pool delete = true 2024-09-16T01:33:17.566 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.566 INFO:teuthology.orchestra.run.smithi154.stdout: mon cluster log file level = debug 2024-09-16T01:33:17.566 INFO:teuthology.orchestra.run.smithi154.stdout: debug asserts on shutdown = true 2024-09-16T01:33:17.566 INFO:teuthology.orchestra.run.smithi154.stdout: mon health detail to clog = false 2024-09-16T01:33:17.566 INFO:teuthology.orchestra.run.smithi154.stdout: mon host = "172.21.15.88,172.21.15.154,[v2:172.21.15.154:3301,v1:172.21.15.154:6790]" 2024-09-16T01:33:17.566 INFO:teuthology.orchestra.run.smithi154.stdout: ms die on skipped message = False 2024-09-16T01:33:17.566 INFO:teuthology.orchestra.run.smithi154.stdout: fsid = 5ace2c91-46f9-482b-8a89-731fbdc171bf 2024-09-16T01:33:17.566 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.566 INFO:teuthology.orchestra.run.smithi154.stdout:[osd] 2024-09-16T01:33:17.566 INFO:teuthology.orchestra.run.smithi154.stdout: osd journal size = 100 2024-09-16T01:33:17.567 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.567 INFO:teuthology.orchestra.run.smithi154.stdout: osd scrub load threshold = 5.0 2024-09-16T01:33:17.567 INFO:teuthology.orchestra.run.smithi154.stdout: osd scrub max interval = 600 2024-09-16T01:33:17.567 INFO:teuthology.orchestra.run.smithi154.stdout: osd mclock profile = high_recovery_ops 2024-09-16T01:33:17.567 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.567 INFO:teuthology.orchestra.run.smithi154.stdout: osd recover clone overlap = true 2024-09-16T01:33:17.567 INFO:teuthology.orchestra.run.smithi154.stdout: osd recovery max chunk = 1048576 2024-09-16T01:33:17.567 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.567 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug shutdown = true 2024-09-16T01:33:17.567 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug op order = true 2024-09-16T01:33:17.568 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug verify stray on activate = true 2024-09-16T01:33:17.568 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug trim objects = true 2024-09-16T01:33:17.568 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.568 INFO:teuthology.orchestra.run.smithi154.stdout: osd open classes on start = true 2024-09-16T01:33:17.568 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug pg log writeout = true 2024-09-16T01:33:17.568 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.568 INFO:teuthology.orchestra.run.smithi154.stdout: osd deep scrub update digest min age = 30 2024-09-16T01:33:17.568 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.568 INFO:teuthology.orchestra.run.smithi154.stdout: osd map max advance = 10 2024-09-16T01:33:17.568 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.568 INFO:teuthology.orchestra.run.smithi154.stdout: journal zero on create = true 2024-09-16T01:33:17.568 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.568 INFO:teuthology.orchestra.run.smithi154.stdout: filestore ondisk finisher threads = 3 2024-09-16T01:33:17.569 INFO:teuthology.orchestra.run.smithi154.stdout: filestore apply finisher threads = 3 2024-09-16T01:33:17.569 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.569 INFO:teuthology.orchestra.run.smithi154.stdout: bdev debug aio = true 2024-09-16T01:33:17.569 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug misdirected ops = true 2024-09-16T01:33:17.569 INFO:teuthology.orchestra.run.smithi154.stdout: bdev async discard = True 2024-09-16T01:33:17.569 INFO:teuthology.orchestra.run.smithi154.stdout: bdev enable discard = True 2024-09-16T01:33:17.569 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore allocator = bitmap 2024-09-16T01:33:17.569 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore block size = 96636764160 2024-09-16T01:33:17.569 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore fsck on mount = True 2024-09-16T01:33:17.570 INFO:teuthology.orchestra.run.smithi154.stdout: debug bluefs = 1/20 2024-09-16T01:33:17.570 INFO:teuthology.orchestra.run.smithi154.stdout: debug bluestore = 1/20 2024-09-16T01:33:17.570 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2024-09-16T01:33:17.570 INFO:teuthology.orchestra.run.smithi154.stdout: debug osd = 20 2024-09-16T01:33:17.570 INFO:teuthology.orchestra.run.smithi154.stdout: debug rocksdb = 4/10 2024-09-16T01:33:17.570 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T01:33:17.571 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd full ratio = 0.9 2024-09-16T01:33:17.571 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd nearfull ratio = 0.8 2024-09-16T01:33:17.571 INFO:teuthology.orchestra.run.smithi154.stdout: osd failsafe full ratio = 0.95 2024-09-16T01:33:17.571 INFO:teuthology.orchestra.run.smithi154.stdout: osd objectstore = bluestore 2024-09-16T01:33:17.571 INFO:teuthology.orchestra.run.smithi154.stdout: osd op complaint time = 180 2024-09-16T01:33:17.571 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.572 INFO:teuthology.orchestra.run.smithi154.stdout:[mgr] 2024-09-16T01:33:17.572 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2024-09-16T01:33:17.572 INFO:teuthology.orchestra.run.smithi154.stdout: debug mgr = 20 2024-09-16T01:33:17.572 INFO:teuthology.orchestra.run.smithi154.stdout: debug mon = 20 2024-09-16T01:33:17.572 INFO:teuthology.orchestra.run.smithi154.stdout: debug auth = 20 2024-09-16T01:33:17.572 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min pgs per osd = 4 2024-09-16T01:33:17.572 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min bytes per osd = 10 2024-09-16T01:33:17.573 INFO:teuthology.orchestra.run.smithi154.stdout: mgr/telemetry/nag = false 2024-09-16T01:33:17.573 INFO:teuthology.orchestra.run.smithi154.stdout: debug client = 20 2024-09-16T01:33:17.573 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.573 INFO:teuthology.orchestra.run.smithi154.stdout:[mon] 2024-09-16T01:33:17.573 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2024-09-16T01:33:17.573 INFO:teuthology.orchestra.run.smithi154.stdout: debug mon = 20 2024-09-16T01:33:17.573 INFO:teuthology.orchestra.run.smithi154.stdout: debug paxos = 20 2024-09-16T01:33:17.574 INFO:teuthology.orchestra.run.smithi154.stdout: debug auth = 20 2024-09-16T01:33:17.574 INFO:teuthology.orchestra.run.smithi154.stdout: mon data avail warn = 5 2024-09-16T01:33:17.574 INFO:teuthology.orchestra.run.smithi154.stdout: mon mgr mkfs grace = 240 2024-09-16T01:33:17.574 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min pgs per osd = 4 2024-09-16T01:33:17.574 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd reporter subtree level = osd 2024-09-16T01:33:17.574 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd prime pg temp = true 2024-09-16T01:33:17.574 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min bytes per osd = 10 2024-09-16T01:33:17.574 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.575 INFO:teuthology.orchestra.run.smithi154.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T01:33:17.575 INFO:teuthology.orchestra.run.smithi154.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T01:33:17.575 INFO:teuthology.orchestra.run.smithi154.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T01:33:17.575 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.575 INFO:teuthology.orchestra.run.smithi154.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T01:33:17.575 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T01:33:17.575 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T01:33:17.576 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.576 INFO:teuthology.orchestra.run.smithi154.stdout: # 1m isn't quite enough 2024-09-16T01:33:17.576 INFO:teuthology.orchestra.run.smithi154.stdout: mon_down_mkfs_grace = 2m 2024-09-16T01:33:17.576 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.576 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_filestore_osds = false 2024-09-16T01:33:17.576 INFO:teuthology.orchestra.run.smithi154.stdout: mon down mkfs grace = 300 2024-09-16T01:33:17.577 INFO:teuthology.orchestra.run.smithi154.stdout: mon op complaint time = 120 2024-09-16T01:33:17.577 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on pool no app = False 2024-09-16T01:33:17.577 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T01:33:17.577 INFO:teuthology.orchestra.run.smithi154.stdout:[client] 2024-09-16T01:33:17.577 INFO:teuthology.orchestra.run.smithi154.stdout: rgw cache enabled = true 2024-09-16T01:33:17.577 INFO:teuthology.orchestra.run.smithi154.stdout: rgw enable ops log = true 2024-09-16T01:33:17.577 INFO:teuthology.orchestra.run.smithi154.stdout: rgw enable usage log = true 2024-09-16T01:33:17.578 INFO:teuthology.orchestra.run.smithi154.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T01:33:17.578 INFO:teuthology.orchestra.run.smithi154.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T01:33:17.578 INFO:teuthology.orchestra.run.smithi154.stdout: client mount timeout = 600 2024-09-16T01:33:17.578 INFO:teuthology.orchestra.run.smithi154.stdout: debug client = 20 2024-09-16T01:33:17.578 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2024-09-16T01:33:17.578 INFO:teuthology.orchestra.run.smithi154.stdout: pid file = "" 2024-09-16T01:33:17.578 INFO:teuthology.orchestra.run.smithi154.stdout: rados mon op timeout = 900 2024-09-16T01:33:17.579 INFO:teuthology.orchestra.run.smithi154.stdout: rados osd op timeout = 900 2024-09-16T01:33:17.579 INFO:teuthology.orchestra.run.smithi154.stdout:[mon.a] 2024-09-16T01:33:17.579 INFO:teuthology.orchestra.run.smithi154.stdout:[mon.b] 2024-09-16T01:33:17.579 INFO:teuthology.orchestra.run.smithi154.stdout:[mon.c] 2024-09-16T01:33:17.579 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.a] 2024-09-16T01:33:17.579 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.c] 2024-09-16T01:33:17.579 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.b] 2024-09-16T01:33:17.580 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.d] 2024-09-16T01:33:17.580 INFO:teuthology.orchestra.run.smithi154.stdout:[mds] 2024-09-16T01:33:17.580 INFO:teuthology.orchestra.run.smithi154.stdout: debug mds = 20 2024-09-16T01:33:17.580 INFO:teuthology.orchestra.run.smithi154.stdout: debug mds balancer = 20 2024-09-16T01:33:17.580 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2024-09-16T01:33:17.580 INFO:teuthology.orchestra.run.smithi154.stdout: mds debug frag = True 2024-09-16T01:33:17.580 INFO:teuthology.orchestra.run.smithi154.stdout: mds debug scatterstat = True 2024-09-16T01:33:17.580 INFO:teuthology.orchestra.run.smithi154.stdout: mds op complaint time = 180 2024-09-16T01:33:17.581 INFO:teuthology.orchestra.run.smithi154.stdout: mds verify scatter = True 2024-09-16T01:33:17.581 INFO:teuthology.orchestra.run.smithi154.stdout: osd op complaint time = 180 2024-09-16T01:33:17.581 INFO:teuthology.orchestra.run.smithi154.stdout: rados mon op timeout = 900 2024-09-16T01:33:17.581 INFO:teuthology.orchestra.run.smithi154.stdout: rados osd op timeout = 900 2024-09-16T01:33:17.583 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T01:33:17.583 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T01:33:17.677 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T01:33:17.677 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:33:17.677 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T01:33:17.692 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:33:17.693 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T01:33:17.750 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:33:17.750 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:33:17.751 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T01:33:17.751 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T01:33:17.824 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:33:17.824 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:33:17.880 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi154.front.sepia.ceph.com 2024-09-16T01:33:17.880 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:33:17.880 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T01:33:17.881 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T01:33:17.916 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:33:17.916 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:33:17.973 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T01:33:17.973 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T01:33:17.973 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-16T01:33:18.025 INFO:teuthology.orchestra.run.smithi088.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-16T01:33:18.027 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T01:33:18.111 INFO:teuthology.orchestra.run.smithi154.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T01:33:18.113 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T01:33:18.113 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-09-16T01:33:18.164 INFO:teuthology.orchestra.run.smithi088.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-09-16T01:33:18.167 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-09-16T01:33:18.192 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-09-16T01:33:18.282 INFO:teuthology.orchestra.run.smithi088.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-09-16T01:33:18.285 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-09-16T01:33:18.309 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-09-16T01:33:18.361 INFO:teuthology.orchestra.run.smithi154.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-09-16T01:33:18.363 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-09-16T01:33:18.388 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-09-16T01:33:18.479 INFO:teuthology.orchestra.run.smithi154.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-09-16T01:33:18.481 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-09-16T01:33:18.507 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T01:33:18.507 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T01:33:18.547 INFO:teuthology.orchestra.run.smithi088.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T01:33:18.561 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-09-16T01:33:18.641 INFO:teuthology.orchestra.run.smithi088.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-09-16T01:33:18.654 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-16T01:33:18.694 INFO:teuthology.orchestra.run.smithi154.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-09-16T01:33:18.707 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-09-16T01:33:18.789 INFO:teuthology.orchestra.run.smithi154.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-09-16T01:33:18.801 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T01:33:18.802 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi088.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi154.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-16T01:33:18.802 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T01:33:18.826 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T01:33:18.826 INFO:tasks.ceph:role: osd.0 2024-09-16T01:33:18.826 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:33:18.827 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T01:33:18.898 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T01:33:18.898 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T01:33:18.898 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T01:33:18.898 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T01:33:18.898 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T01:33:18.898 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2024-09-16T01:33:18.898 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T01:33:18.898 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T01:33:18.898 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T01:33:18.899 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T01:33:19.407 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2024-09-16T01:33:19.409 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2024-09-16T01:33:19.409 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T01:33:19.540 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T01:33:19.577 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T01:33:19.644 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T01:33:19.644 INFO:tasks.ceph:role: osd.1 2024-09-16T01:33:19.644 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:33:19.644 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T01:33:19.731 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T01:33:19.732 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T01:33:19.732 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T01:33:19.732 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T01:33:19.732 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T01:33:19.732 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2024-09-16T01:33:19.732 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T01:33:19.732 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T01:33:19.732 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T01:33:19.732 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T01:33:20.179 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2024-09-16T01:33:20.181 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2024-09-16T01:33:20.181 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-16T01:33:20.235 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T01:33:20.264 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T01:33:20.329 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T01:33:20.329 INFO:tasks.ceph:role: osd.2 2024-09-16T01:33:20.330 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:33:20.330 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T01:33:20.402 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T01:33:20.402 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T01:33:20.402 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T01:33:20.402 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T01:33:20.402 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T01:33:20.402 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2024-09-16T01:33:20.402 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T01:33:20.402 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T01:33:20.403 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T01:33:20.403 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T01:33:20.879 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2024-09-16T01:33:20.881 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2024-09-16T01:33:20.882 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-16T01:33:21.018 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T01:33:21.048 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T01:33:21.113 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-16T01:33:21.113 INFO:tasks.ceph:role: osd.3 2024-09-16T01:33:21.113 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:33:21.113 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T01:33:21.184 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T01:33:21.185 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T01:33:21.185 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T01:33:21.185 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T01:33:21.185 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T01:33:21.185 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2024-09-16T01:33:21.185 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T01:33:21.185 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T01:33:21.185 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T01:33:21.185 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T01:33:21.702 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2024-09-16T01:33:21.704 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2024-09-16T01:33:21.704 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-16T01:33:21.844 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T01:33:21.877 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:33:21.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:21.968+0000 7fad2ff29640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T01:33:21.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:21.969+0000 7fad2ff29640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T01:33:21.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:21.969+0000 7fad2ff29640 -1 bdev(0x56100f763800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-16T01:33:21.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:21.969+0000 7fad2ff29640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T01:33:25.629 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T01:33:25.658 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:33:25.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:25.746+0000 7f83617b2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T01:33:25.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:25.747+0000 7f83617b2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T01:33:25.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:25.747+0000 7f83617b2640 -1 bdev(0x562075807800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-16T01:33:25.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:25.747+0000 7f83617b2640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T01:33:28.910 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T01:33:28.938 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:33:29.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:29.025+0000 7f58a0840640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T01:33:29.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:29.025+0000 7f58a0840640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T01:33:29.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:29.025+0000 7f58a0840640 -1 bdev(0x55d6d399d800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-16T01:33:29.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:29.026+0000 7f58a0840640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T01:33:32.931 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T01:33:32.960 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:33:33.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:33.050+0000 7f4e50c0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-16T01:33:33.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:33.051+0000 7f4e50c0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-16T01:33:33.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:33.051+0000 7f4e50c0d640 -1 bdev(0x563f58f6b800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-16T01:33:33.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:33.051+0000 7f4e50c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-16T01:33:36.961 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T01:33:36.991 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-16T01:33:37.022 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T01:33:37.022 INFO:tasks.ceph:role: osd.4 2024-09-16T01:33:37.022 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-16T01:33:37.022 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T01:33:37.099 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T01:33:37.099 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T01:33:37.099 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T01:33:37.099 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T01:33:37.099 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T01:33:37.099 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2024-09-16T01:33:37.099 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T01:33:37.099 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T01:33:37.099 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T01:33:37.099 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T01:33:37.606 INFO:teuthology.orchestra.run.smithi154.stdout:Discarding blocks...Done. 2024-09-16T01:33:37.608 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2024-09-16T01:33:37.608 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-09-16T01:33:37.735 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-16T01:33:37.768 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-16T01:33:37.832 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T01:33:37.833 INFO:tasks.ceph:role: osd.5 2024-09-16T01:33:37.833 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-16T01:33:37.833 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T01:33:37.905 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T01:33:37.906 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T01:33:37.906 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T01:33:37.906 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T01:33:37.906 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T01:33:37.906 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2024-09-16T01:33:37.906 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T01:33:37.906 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T01:33:37.906 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T01:33:37.906 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T01:33:38.353 INFO:teuthology.orchestra.run.smithi154.stdout:Discarding blocks...Done. 2024-09-16T01:33:38.355 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2024-09-16T01:33:38.355 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-16T01:33:38.449 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-16T01:33:38.478 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-16T01:33:38.543 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T01:33:38.544 INFO:tasks.ceph:role: osd.6 2024-09-16T01:33:38.544 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-16T01:33:38.544 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T01:33:38.616 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T01:33:38.616 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T01:33:38.616 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T01:33:38.616 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T01:33:38.616 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T01:33:38.617 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2024-09-16T01:33:38.617 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T01:33:38.617 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T01:33:38.617 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T01:33:38.617 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T01:33:39.103 INFO:teuthology.orchestra.run.smithi154.stdout:Discarding blocks...Done. 2024-09-16T01:33:39.105 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2024-09-16T01:33:39.105 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-09-16T01:33:39.252 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-16T01:33:39.282 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-16T01:33:39.347 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-16T01:33:39.348 INFO:tasks.ceph:role: osd.7 2024-09-16T01:33:39.348 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-16T01:33:39.348 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T01:33:39.420 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T01:33:39.420 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T01:33:39.420 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T01:33:39.420 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T01:33:39.420 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T01:33:39.420 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2024-09-16T01:33:39.420 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T01:33:39.420 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T01:33:39.421 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T01:33:39.421 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T01:33:39.914 INFO:teuthology.orchestra.run.smithi154.stdout:Discarding blocks...Done. 2024-09-16T01:33:39.916 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2024-09-16T01:33:39.916 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-09-16T01:33:40.062 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-16T01:33:40.093 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:33:40.190 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:40.188+0000 7fc8a14dc640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-16T01:33:40.190 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:40.189+0000 7fc8a14dc640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-16T01:33:40.190 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:40.189+0000 7fc8a14dc640 -1 bdev(0x55ccbf1bd800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-16T01:33:40.190 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:40.189+0000 7fc8a14dc640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-16T01:33:43.861 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-16T01:33:43.890 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:33:43.980 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:43.978+0000 7f1673419640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-16T01:33:43.980 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:43.979+0000 7f1673419640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-16T01:33:43.980 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:43.979+0000 7f1673419640 -1 bdev(0x55bf6f347800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-16T01:33:43.980 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:43.979+0000 7f1673419640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-16T01:33:45.558 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:33:45.562 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:33:47.641 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-16T01:33:47.670 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:33:47.761 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:47.760+0000 7fbd2440d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-16T01:33:47.762 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:47.760+0000 7fbd2440d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-16T01:33:47.762 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:47.760+0000 7fbd2440d640 -1 bdev(0x559af624b800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-16T01:33:47.762 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:47.760+0000 7fbd2440d640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-16T01:33:51.657 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-16T01:33:51.687 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:33:51.780 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:51.778+0000 7f1d59558640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-16T01:33:51.780 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:51.778+0000 7f1d59558640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-16T01:33:51.780 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:51.778+0000 7f1d59558640 -1 bdev(0x55ced4b9b800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-16T01:33:51.780 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:51.778+0000 7f1d59558640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-16T01:33:55.444 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-16T01:33:55.475 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T01:33:55.475 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:33:55.475 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-16T01:33:55.505 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:33:55.505 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-09-16T01:33:55.574 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:33:55.574 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-09-16T01:33:55.642 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:33:55.642 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T01:33:55.709 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:33:55.709 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T01:33:55.775 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:33:55.775 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T01:33:55.842 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:33:55.842 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T01:33:55.907 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:33:55.907 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T01:33:55.937 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:33:55.937 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-09-16T01:33:56.005 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:33:56.005 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-09-16T01:33:56.073 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:33:56.073 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-16T01:33:56.140 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:33:56.141 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-16T01:33:56.208 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:33:56.208 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-16T01:33:56.274 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:33:56.274 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-16T01:33:56.340 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:33:56.340 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T01:33:56.359 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:33:56.359 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-09-16T01:33:56.417 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:33:56.417 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-09-16T01:33:56.433 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:33:56.433 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-09-16T01:33:56.489 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T01:33:56.489 DEBUG:teuthology.orchestra.run.smithi088:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T01:33:56.492 DEBUG:teuthology.orchestra.run.smithi154:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T01:33:56.532 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr.y] 2024-09-16T01:33:56.532 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDeiudm/BN5ARAAxbCopbsvxFUZP3drso32qw== 2024-09-16T01:33:56.532 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.a] 2024-09-16T01:33:56.532 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDeiudmy+jJCRAAhnzX/li6D3D07NmThrjXkA== 2024-09-16T01:33:56.533 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.c] 2024-09-16T01:33:56.533 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDeiudmaYHUEBAAzM0qE53LEgHi+hThHJX3Tw== 2024-09-16T01:33:56.533 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.0] 2024-09-16T01:33:56.533 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDhiudmZR7MORAAwDKvphubA+CbzBUWuaA80A== 2024-09-16T01:33:56.533 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.1] 2024-09-16T01:33:56.533 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDliudmNd+SLBAArOUdoZ226c5jlAsKwqERmw== 2024-09-16T01:33:56.533 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.2] 2024-09-16T01:33:56.533 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDpiudm75CUARAAWHtYOg9pE8g+M6IL+8XZ6g== 2024-09-16T01:33:56.533 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.3] 2024-09-16T01:33:56.533 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDtiudm0fgWAxAAf2NOWrpXbOhb3dFXozJagQ== 2024-09-16T01:33:56.533 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr.x] 2024-09-16T01:33:56.534 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDeiudmToGYBhAAKwXS34PtfAVtEtdtHKHlQQ== 2024-09-16T01:33:56.534 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.b] 2024-09-16T01:33:56.534 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDeiudmx7p+FRAApxKUSZoR/nGUZsaYKlrYUg== 2024-09-16T01:33:56.534 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.d] 2024-09-16T01:33:56.534 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDeiudm3TSLHBAAJzlR6H09cc1Pki2FjyoFiA== 2024-09-16T01:33:56.534 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.4] 2024-09-16T01:33:56.534 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQD0iudmdddMCxAAtr2qfj4Nxj3/GHUpxRmtFg== 2024-09-16T01:33:56.534 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.5] 2024-09-16T01:33:56.534 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQD3iudmT5RjOhAAcKwca6BG60gsxYnmy4twGg== 2024-09-16T01:33:56.534 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.6] 2024-09-16T01:33:56.535 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQD7iudmT7daLRAAdfLFijKQku3rJcJ5xcfLsw== 2024-09-16T01:33:56.535 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.7] 2024-09-16T01:33:56.535 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQD/iudmmvtxLhAArYKhQ4zyFHtLgh/+bs5ZhA== 2024-09-16T01:33:56.535 INFO:teuthology.orchestra.run.smithi088.stdout:[client.0] 2024-09-16T01:33:56.535 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDeiudmvPikIBAAyHxNNnwQKygxg09PNQLc/Q== 2024-09-16T01:33:56.535 INFO:teuthology.orchestra.run.smithi088.stdout:[client.2] 2024-09-16T01:33:56.535 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDeiudmB+w9JhAAv63Is5T/cvDbbLOjIkI0wA== 2024-09-16T01:33:56.535 INFO:teuthology.orchestra.run.smithi088.stdout:[client.1] 2024-09-16T01:33:56.535 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDeiudmA69iKRAA8Abwe0Lg2PnZDErgJ1gEbQ== 2024-09-16T01:33:56.535 INFO:teuthology.orchestra.run.smithi088.stdout:[client.3] 2024-09-16T01:33:56.535 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDeiudm4WoFLxAAnYGHGqHyBHHajQ8f3rCjEA== 2024-09-16T01:33:56.553 INFO:teuthology.orchestra.run.smithi154.stdout:[mgr.y] 2024-09-16T01:33:56.553 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDeiudm/BN5ARAAxbCopbsvxFUZP3drso32qw== 2024-09-16T01:33:56.553 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.a] 2024-09-16T01:33:56.553 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDeiudmy+jJCRAAhnzX/li6D3D07NmThrjXkA== 2024-09-16T01:33:56.553 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.c] 2024-09-16T01:33:56.553 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDeiudmaYHUEBAAzM0qE53LEgHi+hThHJX3Tw== 2024-09-16T01:33:56.553 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.0] 2024-09-16T01:33:56.553 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDhiudmZR7MORAAwDKvphubA+CbzBUWuaA80A== 2024-09-16T01:33:56.553 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.1] 2024-09-16T01:33:56.554 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDliudmNd+SLBAArOUdoZ226c5jlAsKwqERmw== 2024-09-16T01:33:56.554 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.2] 2024-09-16T01:33:56.554 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDpiudm75CUARAAWHtYOg9pE8g+M6IL+8XZ6g== 2024-09-16T01:33:56.554 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.3] 2024-09-16T01:33:56.554 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDtiudm0fgWAxAAf2NOWrpXbOhb3dFXozJagQ== 2024-09-16T01:33:56.554 INFO:teuthology.orchestra.run.smithi154.stdout:[mgr.x] 2024-09-16T01:33:56.554 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDeiudmToGYBhAAKwXS34PtfAVtEtdtHKHlQQ== 2024-09-16T01:33:56.554 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.b] 2024-09-16T01:33:56.554 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDeiudmx7p+FRAApxKUSZoR/nGUZsaYKlrYUg== 2024-09-16T01:33:56.554 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.d] 2024-09-16T01:33:56.554 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDeiudm3TSLHBAAJzlR6H09cc1Pki2FjyoFiA== 2024-09-16T01:33:56.555 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.4] 2024-09-16T01:33:56.555 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQD0iudmdddMCxAAtr2qfj4Nxj3/GHUpxRmtFg== 2024-09-16T01:33:56.555 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.5] 2024-09-16T01:33:56.555 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQD3iudmT5RjOhAAcKwca6BG60gsxYnmy4twGg== 2024-09-16T01:33:56.555 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.6] 2024-09-16T01:33:56.555 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQD7iudmT7daLRAAdfLFijKQku3rJcJ5xcfLsw== 2024-09-16T01:33:56.555 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.7] 2024-09-16T01:33:56.555 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQD/iudmmvtxLhAArYKhQ4zyFHtLgh/+bs5ZhA== 2024-09-16T01:33:56.555 INFO:teuthology.orchestra.run.smithi154.stdout:[client.0] 2024-09-16T01:33:56.555 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDeiudmvPikIBAAyHxNNnwQKygxg09PNQLc/Q== 2024-09-16T01:33:56.555 INFO:teuthology.orchestra.run.smithi154.stdout:[client.2] 2024-09-16T01:33:56.556 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDeiudmB+w9JhAAv63Is5T/cvDbbLOjIkI0wA== 2024-09-16T01:33:56.556 INFO:teuthology.orchestra.run.smithi154.stdout:[client.1] 2024-09-16T01:33:56.556 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDeiudmA69iKRAA8Abwe0Lg2PnZDErgJ1gEbQ== 2024-09-16T01:33:56.556 INFO:teuthology.orchestra.run.smithi154.stdout:[client.3] 2024-09-16T01:33:56.556 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDeiudm4WoFLxAAnYGHGqHyBHHajQ8f3rCjEA== 2024-09-16T01:33:56.557 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T01:33:56.579 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T01:33:56.649 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T01:33:56.652 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T01:33:56.705 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T01:33:56.707 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T01:33:56.762 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:33:56.764 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:33:56.815 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:33:56.817 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:33:56.869 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:33:56.871 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:33:56.922 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:33:56.924 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:33:56.974 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T01:33:56.976 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T01:33:57.026 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T01:33:57.029 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T01:33:57.083 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T01:33:57.085 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-16T01:33:57.139 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:33:57.141 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:33:57.192 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:33:57.194 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:33:57.245 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:33:57.247 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:33:57.298 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:33:57.300 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T01:33:57.351 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T01:33:57.353 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T01:33:57.407 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T01:33:57.410 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T01:33:57.463 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T01:33:57.466 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T01:33:57.519 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T01:33:57.521 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T01:33:57.575 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T01:33:57.575 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T01:33:57.600 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T01:33:57.695 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T01:33:57.720 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T01:33:57.745 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T01:33:57.840 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T01:33:57.865 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T01:33:57.930 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T01:33:58.026 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T01:33:58.051 DEBUG:teuthology.orchestra.run.smithi088:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:33:58.053 DEBUG:teuthology.orchestra.run.smithi154:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T01:33:58.106 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T01:33:58.106 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T01:33:58.106 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T01:33:58.109 INFO:tasks.ceph.mon.a:Started 2024-09-16T01:33:58.109 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T01:33:58.109 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T01:33:58.148 INFO:tasks.ceph.mon.b:Started 2024-09-16T01:33:58.148 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T01:33:58.148 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T01:33:58.150 INFO:tasks.ceph.mon.c:Started 2024-09-16T01:33:58.150 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T01:33:58.150 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-16T01:33:58.151 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-16T01:33:58.153 INFO:tasks.ceph.mgr.y:Started 2024-09-16T01:33:58.153 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T01:33:58.153 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T01:33:58.155 INFO:tasks.ceph.mgr.x:Started 2024-09-16T01:33:58.155 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T01:33:58.155 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T01:33:58.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.253+0000 7f613f87b640 1 Processor -- start 2024-09-16T01:33:58.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.253+0000 7f613f87b640 1 -- start start 2024-09-16T01:33:58.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.254+0000 7f613f87b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f613815c260 0x7f61380d1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:58.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.254+0000 7f613f87b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6138072c70 con 0x7f6138072690 2024-09-16T01:33:58.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.254+0000 7f613f87b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61380d1190 con 0x7f61380b3570 2024-09-16T01:33:58.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.254+0000 7f613f87b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f61380d21b0 con 0x7f613815c260 2024-09-16T01:33:58.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.254+0000 7f613de18640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6138072690 0x7f613806ca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58624/0 (socket says 172.21.15.88:58624) 2024-09-16T01:33:58.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.254+0000 7f613de18640 1 -- 172.21.15.88:0/671311518 learned_addr learned my addr 172.21.15.88:0/671311518 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:33:58.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.254+0000 7f613d617640 1 --2- 172.21.15.88:0/671311518 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f613815c260 0x7f61380d1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:58.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.254+0000 7f613d617640 1 -- 172.21.15.88:0/671311518 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f613815c260 msgr2=0x7f61380d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T01:33:58.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.254+0000 7f613d617640 1 -- 172.21.15.88:0/671311518 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f613815c260 msgr2=0x7f61380d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T01:33:58.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.254+0000 7f613d617640 1 --2- 172.21.15.88:0/671311518 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f613815c260 0x7f61380d1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T01:33:58.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.254+0000 7f613d617640 1 --2- 172.21.15.88:0/671311518 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f613815c260 0x7f61380d1c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T01:33:58.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.423+0000 7f612ffff640 1 -- 172.21.15.88:0/671311518 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4004916799 0 0) 0x7f6138072c70 con 0x7f6138072690 2024-09-16T01:33:58.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.423+0000 7f612ffff640 1 -- 172.21.15.88:0/671311518 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6124003520 con 0x7f6138072690 2024-09-16T01:33:58.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.424+0000 7f612ffff640 1 -- 172.21.15.88:0/671311518 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 646687818 0 0) 0x7f6124003520 con 0x7f6138072690 2024-09-16T01:33:58.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.424+0000 7f612ffff640 1 -- 172.21.15.88:0/671311518 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f613815c260 msgr2=0x7f61380d1c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T01:33:58.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.424+0000 7f612ffff640 1 --2- 172.21.15.88:0/671311518 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f613815c260 0x7f61380d1c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:58.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.424+0000 7f612ffff640 1 -- 172.21.15.88:0/671311518 >> v1:172.21.15.154:6789/0 conn(0x7f61380b3570 legacy=0x7f6138071f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:58.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.424+0000 7f612ffff640 1 -- 172.21.15.88:0/671311518 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61380d1470 con 0x7f6138072690 2024-09-16T01:33:58.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.424+0000 7f612ffff640 1 -- 172.21.15.88:0/671311518 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f61300019f0 con 0x7f6138072690 2024-09-16T01:33:58.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.424+0000 7f612ffff640 1 -- 172.21.15.88:0/671311518 >> v1:172.21.15.88:6789/0 conn(0x7f6138072690 legacy=0x7f613806ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:58.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.424+0000 7f612ffff640 1 --2- 172.21.15.88:0/671311518 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f613815c260 0x7f61240039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:58.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.424+0000 7f612ffff640 1 --2- 172.21.15.88:0/671311518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6124004010 0x7f6124004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:58.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.424+0000 7f612ffff640 1 --2- 172.21.15.88:0/671311518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6124004d90 0x7f6124005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:58.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.424+0000 7f612ffff640 1 -- 172.21.15.88:0/671311518 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f61380d21b0 con 0x7f6124004010 2024-09-16T01:33:58.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.424+0000 7f612ffff640 1 -- 172.21.15.88:0/671311518 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f612400bca0 con 0x7f6124004d90 2024-09-16T01:33:58.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.424+0000 7f612ffff640 1 -- 172.21.15.88:0/671311518 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f612400bea0 con 0x7f613815c260 2024-09-16T01:33:58.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.424+0000 7f613d617640 1 --2- 172.21.15.88:0/671311518 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f613815c260 0x7f61240039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.424+0000 7f613ce16640 1 --2- 172.21.15.88:0/671311518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6124004010 0x7f6124004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.424+0000 7f613de18640 1 --2- 172.21.15.88:0/671311518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6124004d90 0x7f6124005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.425+0000 7f613ce16640 1 -- 172.21.15.88:0/671311518 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f613815c260 msgr2=0x7f61240039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:33:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.425+0000 7f613ce16640 1 --2- 172.21.15.88:0/671311518 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f613815c260 0x7f61240039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.425+0000 7f613ce16640 1 -- 172.21.15.88:0/671311518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6124004d90 msgr2=0x7f6124005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:33:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.425+0000 7f613ce16640 1 --2- 172.21.15.88:0/671311518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6124004d90 0x7f6124005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.425+0000 7f613ce16640 1 -- 172.21.15.88:0/671311518 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6124003520 con 0x7f6124004010 2024-09-16T01:33:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.425+0000 7f613ce16640 1 --2- 172.21.15.88:0/671311518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6124004010 0x7f6124004800 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f61380cd570 tx=0x7f6124005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2698907dba0aa7aa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:33:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.425+0000 7f612ffff640 1 -- 172.21.15.88:0/671311518 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6128001370 con 0x7f6124004010 2024-09-16T01:33:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.425+0000 7f612ffff640 1 -- 172.21.15.88:0/671311518 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61280052d0 con 0x7f6124004010 2024-09-16T01:33:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.425+0000 7f612ffff640 1 -- 172.21.15.88:0/671311518 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612802ab90 con 0x7f6124004010 2024-09-16T01:33:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.426+0000 7f613f87b640 1 -- 172.21.15.88:0/671311518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6124004010 msgr2=0x7f6124004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:58.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.426+0000 7f613f87b640 1 --2- 172.21.15.88:0/671311518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6124004010 0x7f6124004800 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f61380cd570 tx=0x7f6124005830 comp rx=0 tx=0).stop 2024-09-16T01:33:58.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.426+0000 7f613de18640 1 -- 172.21.15.88:0/671311518 reap_dead start 2024-09-16T01:33:58.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.426+0000 7f613f87b640 1 -- 172.21.15.88:0/671311518 shutdown_connections 2024-09-16T01:33:58.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.426+0000 7f613f87b640 1 -- 172.21.15.88:0/671311518 >> 172.21.15.88:0/671311518 conn(0x7f61380b6b50 msgr2=0x7f61380b6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:33:58.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.426+0000 7f613f87b640 1 -- 172.21.15.88:0/671311518 shutdown_connections 2024-09-16T01:33:58.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.426+0000 7f613f87b640 1 -- 172.21.15.88:0/671311518 wait complete. 2024-09-16T01:33:58.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.428+0000 7f613f87b640 1 Processor -- start 2024-09-16T01:33:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.428+0000 7f613f87b640 1 -- start start 2024-09-16T01:33:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.429+0000 7f613f87b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6124004010 0x7f6138156830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.429+0000 7f613f87b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6138156d70 0x7f61381f5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.429+0000 7f613f87b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61381f5db0 0x7f6138153dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.429+0000 7f613f87b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f61380d17e0 con 0x7f61381f5db0 2024-09-16T01:33:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.429+0000 7f613f87b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6138072f50 con 0x7f6138156d70 2024-09-16T01:33:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.429+0000 7f613f87b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f61380d1c70 con 0x7f6124004010 2024-09-16T01:33:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.429+0000 7f613d617640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6124004010 0x7f6138156830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.429+0000 7f613ce16640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6138156d70 0x7f61381f5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.429+0000 7f613d617640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6124004010 0x7f6138156830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:45772/0 (socket says 172.21.15.88:45772) 2024-09-16T01:33:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.429+0000 7f613de18640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61381f5db0 0x7f6138153dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.429+0000 7f613d617640 1 -- 172.21.15.88:0/731038240 learned_addr learned my addr 172.21.15.88:0/731038240 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:33:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.429+0000 7f613d617640 1 -- 172.21.15.88:0/731038240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6138156d70 msgr2=0x7f61381f5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.429+0000 7f613d617640 1 --2- 172.21.15.88:0/731038240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6138156d70 0x7f61381f5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.429+0000 7f613d617640 1 -- 172.21.15.88:0/731038240 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61381f5db0 msgr2=0x7f6138153dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.429+0000 7f613d617640 1 --2- 172.21.15.88:0/731038240 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61381f5db0 0x7f6138153dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.429+0000 7f613d617640 1 -- 172.21.15.88:0/731038240 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61381573f0 con 0x7f6124004010 2024-09-16T01:33:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.429+0000 7f613d617640 1 --2- 172.21.15.88:0/731038240 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6124004010 0x7f6138156830 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f61340108a0 tx=0x7f6134010d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:33:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.431+0000 7f612dffb640 1 -- 172.21.15.88:0/731038240 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f613401a020 con 0x7f6124004010 2024-09-16T01:33:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.432+0000 7f612dffb640 1 -- 172.21.15.88:0/731038240 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f613400e9c0 con 0x7f6124004010 2024-09-16T01:33:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.432+0000 7f613f87b640 1 -- 172.21.15.88:0/731038240 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61381576d0 con 0x7f6124004010 2024-09-16T01:33:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.432+0000 7f612dffb640 1 -- 172.21.15.88:0/731038240 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f613400edb0 con 0x7f6124004010 2024-09-16T01:33:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.432+0000 7f613f87b640 1 -- 172.21.15.88:0/731038240 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6138140320 con 0x7f6124004010 2024-09-16T01:33:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.433+0000 7f613f87b640 1 -- 172.21.15.88:0/731038240 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61381f61c0 con 0x7f6124004010 2024-09-16T01:33:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.435+0000 7f612dffb640 1 -- 172.21.15.88:0/731038240 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f613401d020 con 0x7f6124004010 2024-09-16T01:33:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.435+0000 7f612dffb640 1 -- 172.21.15.88:0/731038240 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f61340298a0 con 0x7f6124004010 2024-09-16T01:33:58.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.437+0000 7f612dffb640 1 -- 172.21.15.88:0/731038240 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f61381f61c0 con 0x7f6124004010 2024-09-16T01:33:58.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.501+0000 7f613f87b640 1 -- 172.21.15.88:0/731038240 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f6138157150 con 0x7f6124004010 2024-09-16T01:33:58.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.509+0000 7f612dffb640 1 -- 172.21.15.88:0/731038240 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f6134024020 con 0x7f6124004010 2024-09-16T01:33:58.511 INFO:teuthology.orchestra.run.smithi088.stderr:adjusted tunables profile to default 2024-09-16T01:33:58.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.510+0000 7f613f87b640 1 -- 172.21.15.88:0/731038240 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6124004010 msgr2=0x7f6138156830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:58.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.510+0000 7f613f87b640 1 --2- 172.21.15.88:0/731038240 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6124004010 0x7f6138156830 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f61340108a0 tx=0x7f6134010d60 comp rx=0 tx=0).stop 2024-09-16T01:33:58.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.510+0000 7f613f87b640 1 -- 172.21.15.88:0/731038240 shutdown_connections 2024-09-16T01:33:58.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.510+0000 7f613f87b640 1 --2- 172.21.15.88:0/731038240 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61381f5db0 0x7f6138153dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:58.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.510+0000 7f613f87b640 1 --2- 172.21.15.88:0/731038240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6138156d70 0x7f61381f5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:58.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.510+0000 7f613f87b640 1 --2- 172.21.15.88:0/731038240 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6124004010 0x7f6138156830 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:58.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.510+0000 7f613f87b640 1 -- 172.21.15.88:0/731038240 >> 172.21.15.88:0/731038240 conn(0x7f61380b6b50 msgr2=0x7f61380b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:33:58.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.511+0000 7f613f87b640 1 -- 172.21.15.88:0/731038240 shutdown_connections 2024-09-16T01:33:58.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:33:58.511+0000 7f613f87b640 1 -- 172.21.15.88:0/731038240 wait complete. 2024-09-16T01:33:58.522 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T01:33:58.522 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T01:33:58.523 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:33:58.523 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T01:33:58.525 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T01:33:58.524+0000 7f29693f9040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T01:33:58.530 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:33:58.529+0000 7f1b04f37040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T01:33:58.547 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:33:58.547 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T01:33:58.565 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T01:33:58.564+0000 7f29693f9040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T01:33:58.574 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:33:58.573+0000 7f1b04f37040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T01:33:58.598 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:33:58.598 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T01:33:58.662 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:33:58.662 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T01:33:58.710 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T01:33:58.709+0000 7f29693f9040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T01:33:58.724 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:33:58.723+0000 7f1b04f37040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T01:33:58.727 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:33:58.728 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-16T01:33:58.752 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:33:58.752 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-16T01:33:58.763 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T01:33:58.762+0000 7f29693f9040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T01:33:58.777 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:33:58.776+0000 7f1b04f37040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T01:33:58.787 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:33:58.787 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-16T01:33:58.853 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:33:58.853 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-16T01:33:58.918 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new 29743152-8c11-4c1d-81bd-f6945c3d1401 0 2024-09-16T01:33:58.932 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T01:33:58.931+0000 7f29693f9040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T01:33:58.948 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:33:58.947+0000 7f1b04f37040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T01:33:59.031 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.029+0000 7f60eacd2640 1 Processor -- start 2024-09-16T01:33:59.031 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.029+0000 7f60eacd2640 1 -- start start 2024-09-16T01:33:59.032 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.030+0000 7f60eacd2640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f60e40b3c80 0x7f60e40d1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:59.032 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.030+0000 7f60eacd2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60e4072c70 con 0x7f60e415c260 2024-09-16T01:33:59.032 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.030+0000 7f60eacd2640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60e40d1190 con 0x7f60e4071f80 2024-09-16T01:33:59.032 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.030+0000 7f60eacd2640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f60e40d21b0 con 0x7f60e40b3c80 2024-09-16T01:33:59.032 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.030+0000 7f60e3fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f60e40b3c80 0x7f60e40d1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:59.032 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.030+0000 7f60e926f640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f60e4071f80 0x7f60e40723e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.154:35798/0 (socket says 172.21.15.154:35798) 2024-09-16T01:33:59.032 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.030+0000 7f60e3fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f60e40b3c80 0x7f60e40d1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:51182/0 (socket says 172.21.15.154:51182) 2024-09-16T01:33:59.032 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.030+0000 7f60e926f640 1 -- 172.21.15.154:0/137361996 learned_addr learned my addr 172.21.15.154:0/137361996 (peer_addr_for_me v1:172.21.15.154:0/0) 2024-09-16T01:33:59.032 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.031+0000 7f60e3fff640 1 -- 172.21.15.154:0/137361996 >> v1:172.21.15.154:6789/0 conn(0x7f60e4071f80 legacy=0x7f60e40723e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.032 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.031+0000 7f60e3fff640 1 -- 172.21.15.154:0/137361996 >> v1:172.21.15.88:6789/0 conn(0x7f60e415c260 legacy=0x7f60e40b3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.032 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.031+0000 7f60e3fff640 1 -- 172.21.15.154:0/137361996 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60e40d1470 con 0x7f60e40b3c80 2024-09-16T01:33:59.034 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.031+0000 7f60e3fff640 1 --2- 172.21.15.154:0/137361996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60e40b3c80 0x7f60e40d1c70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f60d00097a0 tx=0x7f60d002a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae2058b4e2c1031f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:33:59.034 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.031+0000 7f60e37fe640 1 -- 172.21.15.154:0/137361996 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d0037070 con 0x7f60e40b3c80 2024-09-16T01:33:59.034 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.031+0000 7f60e37fe640 1 -- 172.21.15.154:0/137361996 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60d0032440 con 0x7f60e40b3c80 2024-09-16T01:33:59.034 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.031+0000 7f60e37fe640 1 -- 172.21.15.154:0/137361996 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d0032810 con 0x7f60e40b3c80 2024-09-16T01:33:59.034 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.032+0000 7f60eacd2640 1 -- 172.21.15.154:0/137361996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60e40b3c80 msgr2=0x7f60e40d1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.034 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.032+0000 7f60eacd2640 1 --2- 172.21.15.154:0/137361996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60e40b3c80 0x7f60e40d1c70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f60d00097a0 tx=0x7f60d002a1a0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.034 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.032+0000 7f60eacd2640 1 -- 172.21.15.154:0/137361996 shutdown_connections 2024-09-16T01:33:59.034 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.032+0000 7f60eacd2640 1 --2- 172.21.15.154:0/137361996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60e40b3c80 0x7f60e40d1c70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.034 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.032+0000 7f60eacd2640 1 -- 172.21.15.154:0/137361996 >> 172.21.15.154:0/137361996 conn(0x7f60e40b6b50 msgr2=0x7f60e40b6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:33:59.034 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.032+0000 7f60eacd2640 1 -- 172.21.15.154:0/137361996 shutdown_connections 2024-09-16T01:33:59.034 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.032+0000 7f60eacd2640 1 -- 172.21.15.154:0/137361996 wait complete. 2024-09-16T01:33:59.035 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.033+0000 7f60eacd2640 1 Processor -- start 2024-09-16T01:33:59.035 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.033+0000 7f60eacd2640 1 -- start start 2024-09-16T01:33:59.037 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.034+0000 7f60eacd2640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e4071f80 0x7f60e415b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:59.037 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.034+0000 7f60eacd2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60e40b3c80 0x7f60e41f2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:59.038 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.034+0000 7f60eacd2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60e415c260 0x7f60e40b0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:59.038 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.034+0000 7f60eacd2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f60e40d17e0 con 0x7f60e40b3c80 2024-09-16T01:33:59.038 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.034+0000 7f60eacd2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f60e4072f50 con 0x7f60e415c260 2024-09-16T01:33:59.038 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.034+0000 7f60eacd2640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f60e406ca40 con 0x7f60e4071f80 2024-09-16T01:33:59.038 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.034+0000 7f60e8a6e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e4071f80 0x7f60e415b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:59.038 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.034+0000 7f60e926f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60e415c260 0x7f60e40b0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:59.038 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.034+0000 7f60e926f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60e415c260 0x7f60e40b0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:51186/0 (socket says 172.21.15.154:51186) 2024-09-16T01:33:59.038 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.034+0000 7f60e926f640 1 -- 172.21.15.154:0/3879792279 learned_addr learned my addr 172.21.15.154:0/3879792279 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-09-16T01:33:59.038 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.034+0000 7f60e8a6e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e4071f80 0x7f60e415b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.154:60998/0 (socket says 172.21.15.154:60998) 2024-09-16T01:33:59.038 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.034+0000 7f60e3fff640 1 --2- 172.21.15.154:0/3879792279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60e40b3c80 0x7f60e41f2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:59.038 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.035+0000 7f60e926f640 1 -- 172.21.15.154:0/3879792279 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e4071f80 msgr2=0x7f60e415b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.038 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.035+0000 7f60e926f640 1 --2- 172.21.15.154:0/3879792279 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e4071f80 0x7f60e415b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.039 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.035+0000 7f60e926f640 1 -- 172.21.15.154:0/3879792279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60e40b3c80 msgr2=0x7f60e41f2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.039 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.035+0000 7f60e926f640 1 --2- 172.21.15.154:0/3879792279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60e40b3c80 0x7f60e41f2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.039 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.035+0000 7f60e926f640 1 -- 172.21.15.154:0/3879792279 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60e41f3410 con 0x7f60e415c260 2024-09-16T01:33:59.039 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.035+0000 7f60e926f640 1 --2- 172.21.15.154:0/3879792279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60e415c260 0x7f60e40b0050 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00cbc0 tx=0x7f60dc00a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:33:59.039 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.035+0000 7f60e17fa640 1 -- 172.21.15.154:0/3879792279 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60dc017070 con 0x7f60e415c260 2024-09-16T01:33:59.039 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.035+0000 7f60e17fa640 1 -- 172.21.15.154:0/3879792279 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60dc00ad40 con 0x7f60e415c260 2024-09-16T01:33:59.039 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.035+0000 7f60eacd2640 1 -- 172.21.15.154:0/3879792279 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60e41f36f0 con 0x7f60e415c260 2024-09-16T01:33:59.039 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.035+0000 7f60e17fa640 1 -- 172.21.15.154:0/3879792279 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60dc012600 con 0x7f60e415c260 2024-09-16T01:33:59.039 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.035+0000 7f60eacd2640 1 -- 172.21.15.154:0/3879792279 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60e40cd2b0 con 0x7f60e415c260 2024-09-16T01:33:59.039 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.035+0000 7f60e17fa640 1 -- 172.21.15.154:0/3879792279 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f60dc01a070 con 0x7f60e415c260 2024-09-16T01:33:59.039 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.035+0000 7f60e17fa640 1 -- 172.21.15.154:0/3879792279 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f60dc01cc60 con 0x7f60e415c260 2024-09-16T01:33:59.039 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.036+0000 7f60eacd2640 1 -- 172.21.15.154:0/3879792279 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60e40d1c70 con 0x7f60e415c260 2024-09-16T01:33:59.040 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.039+0000 7f60e17fa640 1 -- 172.21.15.154:0/3879792279 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f60dc01c440 con 0x7f60e415c260 2024-09-16T01:33:59.057 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T01:33:59.056+0000 7f29693f9040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T01:33:59.060 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:33:59.059+0000 7f1b04f37040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T01:33:59.126 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.124+0000 7f60eacd2640 1 -- 172.21.15.154:0/3879792279 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "29743152-8c11-4c1d-81bd-f6945c3d1401", "id": 0} v 0) -- 0x7f60e4073160 con 0x7f60e415c260 2024-09-16T01:33:59.131 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.130+0000 7f60e17fa640 1 -- 172.21.15.154:0/3879792279 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "29743152-8c11-4c1d-81bd-f6945c3d1401", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f60dc0128d0 con 0x7f60e415c260 2024-09-16T01:33:59.131 INFO:teuthology.orchestra.run.smithi154.stdout:0 2024-09-16T01:33:59.132 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.131+0000 7f60eacd2640 1 -- 172.21.15.154:0/3879792279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60e415c260 msgr2=0x7f60e40b0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.132 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.131+0000 7f60eacd2640 1 --2- 172.21.15.154:0/3879792279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60e415c260 0x7f60e40b0050 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00cbc0 tx=0x7f60dc00a430 comp rx=0 tx=0).stop 2024-09-16T01:33:59.133 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.131+0000 7f60eacd2640 1 -- 172.21.15.154:0/3879792279 shutdown_connections 2024-09-16T01:33:59.133 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.131+0000 7f60eacd2640 1 --2- 172.21.15.154:0/3879792279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60e415c260 0x7f60e40b0050 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.133 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.131+0000 7f60eacd2640 1 --2- 172.21.15.154:0/3879792279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60e40b3c80 0x7f60e41f2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.133 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.131+0000 7f60eacd2640 1 --2- 172.21.15.154:0/3879792279 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e4071f80 0x7f60e415b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.133 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.131+0000 7f60eacd2640 1 -- 172.21.15.154:0/3879792279 >> 172.21.15.154:0/3879792279 conn(0x7f60e40b6b50 msgr2=0x7f60e40cdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:33:59.133 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.131+0000 7f60eacd2640 1 -- 172.21.15.154:0/3879792279 shutdown_connections 2024-09-16T01:33:59.133 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.131+0000 7f60eacd2640 1 -- 172.21.15.154:0/3879792279 wait complete. 2024-09-16T01:33:59.143 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new 191015d7-d294-4241-81e8-509f25cc024d 1 2024-09-16T01:33:59.242 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.240+0000 7f8eea766640 1 Processor -- start 2024-09-16T01:33:59.242 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.240+0000 7f8eea766640 1 -- start start 2024-09-16T01:33:59.243 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.241+0000 7f8eea766640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8ee415c4c0 0x7f8ee40b55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:59.244 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.241+0000 7f8eea766640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8ee40b36a0 0x7f8ee40b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:59.244 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.241+0000 7f8eea766640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ee4072f40 con 0x7f8ee4159000 2024-09-16T01:33:59.244 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.241+0000 7f8eea766640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f8ee40b4050 con 0x7f8ee40b36a0 2024-09-16T01:33:59.244 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.241+0000 7f8eea766640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f8ee40b5130 con 0x7f8ee415c4c0 2024-09-16T01:33:59.244 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.241+0000 7f8ee37fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8ee415c4c0 0x7f8ee40b55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:59.244 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.241+0000 7f8ee37fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8ee415c4c0 0x7f8ee40b55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:51188/0 (socket says 172.21.15.154:51188) 2024-09-16T01:33:59.244 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.241+0000 7f8ee37fe640 1 -- 172.21.15.154:0/361767391 learned_addr learned my addr 172.21.15.154:0/361767391 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-09-16T01:33:59.244 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.242+0000 7f8ee8d03640 1 --2- 172.21.15.154:0/361767391 >> v2:172.21.15.88:3300/0 conn(0x7f8ee40b36a0 0x7f8ee40b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:59.244 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.242+0000 7f8ee37fe640 1 -- 172.21.15.154:0/361767391 >> v2:172.21.15.88:3300/0 conn(0x7f8ee40b36a0 msgr2=0x7f8ee40b3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:33:59.244 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.242+0000 7f8ee37fe640 1 --2- 172.21.15.154:0/361767391 >> v2:172.21.15.88:3300/0 conn(0x7f8ee40b36a0 0x7f8ee40b3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.244 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.242+0000 7f8ee37fe640 1 -- 172.21.15.154:0/361767391 >> v1:172.21.15.88:6789/0 conn(0x7f8ee4159000 legacy=0x7f8ee4072190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.244 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.242+0000 7f8ee37fe640 1 -- 172.21.15.154:0/361767391 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee4072980 con 0x7f8ee415c4c0 2024-09-16T01:33:59.244 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.242+0000 7f8ee37fe640 1 --2- 172.21.15.154:0/361767391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ee415c4c0 0x7f8ee40b55c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40b52b0 tx=0x7f8ecc02c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58a845c63c5a206b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:33:59.245 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.242+0000 7f8ee2ffd640 1 -- 172.21.15.154:0/361767391 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc036020 con 0x7f8ee415c4c0 2024-09-16T01:33:59.245 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.243+0000 7f8ee2ffd640 1 -- 172.21.15.154:0/361767391 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ecc032820 con 0x7f8ee415c4c0 2024-09-16T01:33:59.245 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.243+0000 7f8ee2ffd640 1 -- 172.21.15.154:0/361767391 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc032bf0 con 0x7f8ee415c4c0 2024-09-16T01:33:59.246 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.243+0000 7f8eea766640 1 -- 172.21.15.154:0/361767391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ee415c4c0 msgr2=0x7f8ee40b55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.246 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.243+0000 7f8eea766640 1 --2- 172.21.15.154:0/361767391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ee415c4c0 0x7f8ee40b55c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40b52b0 tx=0x7f8ecc02c920 comp rx=0 tx=0).stop 2024-09-16T01:33:59.246 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.243+0000 7f8eea766640 1 -- 172.21.15.154:0/361767391 shutdown_connections 2024-09-16T01:33:59.246 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.243+0000 7f8eea766640 1 --2- 172.21.15.154:0/361767391 >> v2:172.21.15.88:3300/0 conn(0x7f8ee40b36a0 0x7f8ee40b3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.246 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.243+0000 7f8eea766640 1 --2- 172.21.15.154:0/361767391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ee415c4c0 0x7f8ee40b55c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.246 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.243+0000 7f8eea766640 1 -- 172.21.15.154:0/361767391 >> 172.21.15.154:0/361767391 conn(0x7f8ee40b6b50 msgr2=0x7f8ee40b6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:33:59.246 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.243+0000 7f8eea766640 1 -- 172.21.15.154:0/361767391 shutdown_connections 2024-09-16T01:33:59.246 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.243+0000 7f8eea766640 1 -- 172.21.15.154:0/361767391 wait complete. 2024-09-16T01:33:59.246 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.244+0000 7f8eea766640 1 Processor -- start 2024-09-16T01:33:59.247 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.245+0000 7f8eea766640 1 -- start start 2024-09-16T01:33:59.249 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.245+0000 7f8eea766640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ee40b36a0 0x7f8ee40aa480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:59.249 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.245+0000 7f8eea766640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ee4159000 0x7f8ee415d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:59.249 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.245+0000 7f8eea766640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ee415c4c0 0x7f8ee415d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:59.249 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.245+0000 7f8eea766640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee40b5b00 con 0x7f8ee415c4c0 2024-09-16T01:33:59.249 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.245+0000 7f8eea766640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee40b4dc0 con 0x7f8ee40b36a0 2024-09-16T01:33:59.249 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.245+0000 7f8eea766640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8ee415ca40 con 0x7f8ee4159000 2024-09-16T01:33:59.249 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.245+0000 7f8ee3fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ee40b36a0 0x7f8ee40aa480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:59.249 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.245+0000 7f8ee37fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ee4159000 0x7f8ee415d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:59.249 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.245+0000 7f8ee3fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ee40b36a0 0x7f8ee40aa480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:51190/0 (socket says 172.21.15.154:51190) 2024-09-16T01:33:59.249 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.245+0000 7f8ee3fff640 1 -- 172.21.15.154:0/506298627 learned_addr learned my addr 172.21.15.154:0/506298627 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-09-16T01:33:59.250 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.245+0000 7f8ee8d03640 1 --2- 172.21.15.154:0/506298627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ee415c4c0 0x7f8ee415d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:59.250 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.246+0000 7f8ee3fff640 1 -- 172.21.15.154:0/506298627 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ee4159000 msgr2=0x7f8ee415d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.250 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.246+0000 7f8ee3fff640 1 --2- 172.21.15.154:0/506298627 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ee4159000 0x7f8ee415d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.250 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.246+0000 7f8ee3fff640 1 -- 172.21.15.154:0/506298627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ee415c4c0 msgr2=0x7f8ee415d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.250 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.246+0000 7f8ee3fff640 1 --2- 172.21.15.154:0/506298627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ee415c4c0 0x7f8ee415d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.250 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.246+0000 7f8ee3fff640 1 -- 172.21.15.154:0/506298627 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee41f0ec0 con 0x7f8ee40b36a0 2024-09-16T01:33:59.250 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.246+0000 7f8ee37fe640 1 --2- 172.21.15.154:0/506298627 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ee4159000 0x7f8ee415d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:33:59.250 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.246+0000 7f8ee3fff640 1 --2- 172.21.15.154:0/506298627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ee40b36a0 0x7f8ee40aa480 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8ed400c460 tx=0x7f8ed400c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:33:59.250 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.246+0000 7f8ee0ff9640 1 -- 172.21.15.154:0/506298627 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed40052d0 con 0x7f8ee40b36a0 2024-09-16T01:33:59.250 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.246+0000 7f8ee0ff9640 1 -- 172.21.15.154:0/506298627 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ed4012630 con 0x7f8ee40b36a0 2024-09-16T01:33:59.250 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.246+0000 7f8eea766640 1 -- 172.21.15.154:0/506298627 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ee41f11a0 con 0x7f8ee40b36a0 2024-09-16T01:33:59.250 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.246+0000 7f8ee0ff9640 1 -- 172.21.15.154:0/506298627 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed40054b0 con 0x7f8ee40b36a0 2024-09-16T01:33:59.250 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.246+0000 7f8eea766640 1 -- 172.21.15.154:0/506298627 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ee41f1760 con 0x7f8ee40b36a0 2024-09-16T01:33:59.250 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.246+0000 7f8ee0ff9640 1 -- 172.21.15.154:0/506298627 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8ed4012a20 con 0x7f8ee40b36a0 2024-09-16T01:33:59.251 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.246+0000 7f8ee0ff9640 1 -- 172.21.15.154:0/506298627 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f8ed401b590 con 0x7f8ee40b36a0 2024-09-16T01:33:59.251 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.247+0000 7f8eea766640 1 -- 172.21.15.154:0/506298627 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ee40b3b10 con 0x7f8ee40b36a0 2024-09-16T01:33:59.251 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.249+0000 7f8ee0ff9640 1 -- 172.21.15.154:0/506298627 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f8ed4003610 con 0x7f8ee40b36a0 2024-09-16T01:33:59.318 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.317+0000 7f8eea766640 1 -- 172.21.15.154:0/506298627 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "191015d7-d294-4241-81e8-509f25cc024d", "id": 1} v 0) -- 0x7f8ee40b5db0 con 0x7f8ee40b36a0 2024-09-16T01:33:59.323 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.322+0000 7f8ee0ff9640 1 -- 172.21.15.154:0/506298627 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "191015d7-d294-4241-81e8-509f25cc024d", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f8ed40037f0 con 0x7f8ee40b36a0 2024-09-16T01:33:59.323 INFO:teuthology.orchestra.run.smithi154.stdout:1 2024-09-16T01:33:59.324 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.323+0000 7f8eea766640 1 -- 172.21.15.154:0/506298627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ee40b36a0 msgr2=0x7f8ee40aa480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.324 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.323+0000 7f8eea766640 1 --2- 172.21.15.154:0/506298627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ee40b36a0 0x7f8ee40aa480 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8ed400c460 tx=0x7f8ed400c920 comp rx=0 tx=0).stop 2024-09-16T01:33:59.324 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.323+0000 7f8eea766640 1 -- 172.21.15.154:0/506298627 shutdown_connections 2024-09-16T01:33:59.324 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.323+0000 7f8eea766640 1 --2- 172.21.15.154:0/506298627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ee415c4c0 0x7f8ee415d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.324 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.323+0000 7f8eea766640 1 --2- 172.21.15.154:0/506298627 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ee4159000 0x7f8ee415d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.324 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.323+0000 7f8eea766640 1 --2- 172.21.15.154:0/506298627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ee40b36a0 0x7f8ee40aa480 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.325 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.323+0000 7f8eea766640 1 -- 172.21.15.154:0/506298627 >> 172.21.15.154:0/506298627 conn(0x7f8ee40b6b50 msgr2=0x7f8ee415b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:33:59.325 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.323+0000 7f8eea766640 1 -- 172.21.15.154:0/506298627 shutdown_connections 2024-09-16T01:33:59.325 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.323+0000 7f8eea766640 1 -- 172.21.15.154:0/506298627 wait complete. 2024-09-16T01:33:59.334 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new ba5bc56d-a42c-4ba7-a392-40afa4a9cf04 2 2024-09-16T01:33:59.389 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:33:59.388+0000 7f1b04f37040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T01:33:59.407 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T01:33:59.406+0000 7f29693f9040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T01:33:59.434 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.433+0000 7f3f0e72a640 1 Processor -- start 2024-09-16T01:33:59.434 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.433+0000 7f3f0e72a640 1 -- start start 2024-09-16T01:33:59.435 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.433+0000 7f3f0e72a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f080730d0 0x7f3f0806fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:59.435 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.433+0000 7f3f0e72a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3f0815f780 0x7f3f080734b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:59.436 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.433+0000 7f3f0e72a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f080739f0 con 0x7f3f081622f0 2024-09-16T01:33:59.436 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.434+0000 7f3f0e72a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3f080af8a0 con 0x7f3f0815f780 2024-09-16T01:33:59.436 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.434+0000 7f3f0e72a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3f080717c0 con 0x7f3f080730d0 2024-09-16T01:33:59.436 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.434+0000 7f3f07fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f080730d0 0x7f3f0806fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:59.436 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.434+0000 7f3f077fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3f0815f780 0x7f3f080734b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:59.436 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.434+0000 7f3f07fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f080730d0 0x7f3f0806fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.154:32788/0 (socket says 172.21.15.154:32788) 2024-09-16T01:33:59.436 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.434+0000 7f3f07fff640 1 -- 172.21.15.154:0/844347606 learned_addr learned my addr 172.21.15.154:0/844347606 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-09-16T01:33:59.436 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.434+0000 7f3f077fe640 1 -- 172.21.15.154:0/844347606 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f080730d0 msgr2=0x7f3f0806fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:33:59.436 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.434+0000 7f3f077fe640 1 --2- 172.21.15.154:0/844347606 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f080730d0 0x7f3f0806fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.436 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.434+0000 7f3f077fe640 1 -- 172.21.15.154:0/844347606 >> v1:172.21.15.88:6789/0 conn(0x7f3f081622f0 legacy=0x7f3f08162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.436 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.434+0000 7f3f077fe640 1 -- 172.21.15.154:0/844347606 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f08071aa0 con 0x7f3f0815f780 2024-09-16T01:33:59.436 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.434+0000 7f3f07fff640 1 --2- 172.21.15.154:0/844347606 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f080730d0 0x7f3f0806fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:33:59.437 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:33:59.436+0000 7f1b04f37040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T01:33:59.438 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.435+0000 7f3f077fe640 1 --2- 172.21.15.154:0/844347606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f0815f780 0x7f3f080734b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3efc0097a0 tx=0x7f3efc02a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7fb413b6105ab52 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:33:59.438 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.435+0000 7f3f06ffd640 1 -- 172.21.15.154:0/844347606 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3efc037070 con 0x7f3f0815f780 2024-09-16T01:33:59.438 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.435+0000 7f3f06ffd640 1 -- 172.21.15.154:0/844347606 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3efc032440 con 0x7f3f0815f780 2024-09-16T01:33:59.438 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.435+0000 7f3f06ffd640 1 -- 172.21.15.154:0/844347606 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3efc032810 con 0x7f3f0815f780 2024-09-16T01:33:59.438 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.435+0000 7f3f0e72a640 1 -- 172.21.15.154:0/844347606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f0815f780 msgr2=0x7f3f080734b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.438 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.435+0000 7f3f0e72a640 1 --2- 172.21.15.154:0/844347606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f0815f780 0x7f3f080734b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3efc0097a0 tx=0x7f3efc02a180 comp rx=0 tx=0).stop 2024-09-16T01:33:59.439 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.435+0000 7f3f0e72a640 1 -- 172.21.15.154:0/844347606 shutdown_connections 2024-09-16T01:33:59.439 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.435+0000 7f3f0e72a640 1 --2- 172.21.15.154:0/844347606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f0815f780 0x7f3f080734b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.439 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.435+0000 7f3f0e72a640 1 --2- 172.21.15.154:0/844347606 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f080730d0 0x7f3f0806fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.439 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.435+0000 7f3f0e72a640 1 -- 172.21.15.154:0/844347606 >> 172.21.15.154:0/844347606 conn(0x7f3f080b0aa0 msgr2=0x7f3f080b0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:33:59.439 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.436+0000 7f3f0e72a640 1 -- 172.21.15.154:0/844347606 shutdown_connections 2024-09-16T01:33:59.439 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.436+0000 7f3f0e72a640 1 -- 172.21.15.154:0/844347606 wait complete. 2024-09-16T01:33:59.439 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.437+0000 7f3f0e72a640 1 Processor -- start 2024-09-16T01:33:59.439 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.437+0000 7f3f0e72a640 1 -- start start 2024-09-16T01:33:59.441 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.438+0000 7f3f0e72a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f080730d0 0x7f3f080d01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:59.441 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.438+0000 7f3f0e72a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f0815f780 0x7f3f0814f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:59.441 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.438+0000 7f3f0e72a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f081622f0 0x7f3f080ce1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:59.441 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.438+0000 7f3f0e72a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3f08072340 con 0x7f3f081622f0 2024-09-16T01:33:59.441 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.438+0000 7f3f0e72a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3f08072060 con 0x7f3f0815f780 2024-09-16T01:33:59.442 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.438+0000 7f3f0e72a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3f08159170 con 0x7f3f080730d0 2024-09-16T01:33:59.442 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.438+0000 7f3f07fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f080730d0 0x7f3f080d01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:59.442 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.438+0000 7f3f077fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f0815f780 0x7f3f0814f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:59.442 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.438+0000 7f3f07fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f080730d0 0x7f3f080d01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.154:32800/0 (socket says 172.21.15.154:32800) 2024-09-16T01:33:59.442 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.438+0000 7f3f077fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f0815f780 0x7f3f0814f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:51208/0 (socket says 172.21.15.154:51208) 2024-09-16T01:33:59.442 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.438+0000 7f3f07fff640 1 -- 172.21.15.154:0/783883777 learned_addr learned my addr 172.21.15.154:0/783883777 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-09-16T01:33:59.442 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.438+0000 7f3f0ccc7640 1 --2- 172.21.15.154:0/783883777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f081622f0 0x7f3f080ce1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:59.442 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.438+0000 7f3f077fe640 1 -- 172.21.15.154:0/783883777 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f080730d0 msgr2=0x7f3f080d01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.442 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.438+0000 7f3f077fe640 1 --2- 172.21.15.154:0/783883777 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f080730d0 0x7f3f080d01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.442 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.438+0000 7f3f077fe640 1 -- 172.21.15.154:0/783883777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f081622f0 msgr2=0x7f3f080ce1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.442 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.438+0000 7f3f077fe640 1 --2- 172.21.15.154:0/783883777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f081622f0 0x7f3f080ce1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.442 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.438+0000 7f3f077fe640 1 -- 172.21.15.154:0/783883777 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f0814fe00 con 0x7f3f0815f780 2024-09-16T01:33:59.442 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.438+0000 7f3f07fff640 1 --2- 172.21.15.154:0/783883777 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f080730d0 0x7f3f080d01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:33:59.442 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.438+0000 7f3f077fe640 1 --2- 172.21.15.154:0/783883777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f0815f780 0x7f3f0814f8c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3efc031000 tx=0x7f3efc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:33:59.442 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.438+0000 7f3f04ff9640 1 -- 172.21.15.154:0/783883777 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3efc041030 con 0x7f3f0815f780 2024-09-16T01:33:59.443 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.438+0000 7f3f04ff9640 1 -- 172.21.15.154:0/783883777 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3efc030500 con 0x7f3f0815f780 2024-09-16T01:33:59.443 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.438+0000 7f3f0e72a640 1 -- 172.21.15.154:0/783883777 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f080cf120 con 0x7f3f0815f780 2024-09-16T01:33:59.443 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.438+0000 7f3f04ff9640 1 -- 172.21.15.154:0/783883777 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3efc037040 con 0x7f3f0815f780 2024-09-16T01:33:59.443 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.438+0000 7f3f0e72a640 1 -- 172.21.15.154:0/783883777 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f080cf6e0 con 0x7f3f0815f780 2024-09-16T01:33:59.443 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.439+0000 7f3f04ff9640 1 -- 172.21.15.154:0/783883777 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3efc044070 con 0x7f3f0815f780 2024-09-16T01:33:59.443 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.439+0000 7f3f04ff9640 1 -- 172.21.15.154:0/783883777 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f3efc03dab0 con 0x7f3f0815f780 2024-09-16T01:33:59.443 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.439+0000 7f3f0e72a640 1 -- 172.21.15.154:0/783883777 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f08072860 con 0x7f3f0815f780 2024-09-16T01:33:59.443 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.441+0000 7f3f04ff9640 1 -- 172.21.15.154:0/783883777 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f3efc03dd50 con 0x7f3f0815f780 2024-09-16T01:33:59.466 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T01:33:59.465+0000 7f29693f9040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T01:33:59.512 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.511+0000 7f3f0e72a640 1 -- 172.21.15.154:0/783883777 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ba5bc56d-a42c-4ba7-a392-40afa4a9cf04", "id": 2} v 0) -- 0x7f3f080d0af0 con 0x7f3f0815f780 2024-09-16T01:33:59.518 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.517+0000 7f3f04ff9640 1 -- 172.21.15.154:0/783883777 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ba5bc56d-a42c-4ba7-a392-40afa4a9cf04", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f3efc03d650 con 0x7f3f0815f780 2024-09-16T01:33:59.518 INFO:teuthology.orchestra.run.smithi154.stdout:2 2024-09-16T01:33:59.519 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.518+0000 7f3f0e72a640 1 -- 172.21.15.154:0/783883777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f0815f780 msgr2=0x7f3f0814f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.519 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.518+0000 7f3f0e72a640 1 --2- 172.21.15.154:0/783883777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f0815f780 0x7f3f0814f8c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3efc031000 tx=0x7f3efc0046c0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.519 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.518+0000 7f3f0e72a640 1 -- 172.21.15.154:0/783883777 shutdown_connections 2024-09-16T01:33:59.519 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.518+0000 7f3f0e72a640 1 --2- 172.21.15.154:0/783883777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f081622f0 0x7f3f080ce1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.519 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.518+0000 7f3f0e72a640 1 --2- 172.21.15.154:0/783883777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f0815f780 0x7f3f0814f8c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.519 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.518+0000 7f3f0e72a640 1 --2- 172.21.15.154:0/783883777 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f080730d0 0x7f3f080d01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.520 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.518+0000 7f3f0e72a640 1 -- 172.21.15.154:0/783883777 >> 172.21.15.154:0/783883777 conn(0x7f3f080b0aa0 msgr2=0x7f3f080cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:33:59.520 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.518+0000 7f3f0e72a640 1 -- 172.21.15.154:0/783883777 shutdown_connections 2024-09-16T01:33:59.520 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.518+0000 7f3f0e72a640 1 -- 172.21.15.154:0/783883777 wait complete. 2024-09-16T01:33:59.529 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a 3 2024-09-16T01:33:59.587 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:33:59.586+0000 7f1b04f37040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T01:33:59.624 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T01:33:59.623+0000 7f29693f9040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T01:33:59.627 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.626+0000 7fec8eab7640 1 Processor -- start 2024-09-16T01:33:59.628 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.626+0000 7fec8eab7640 1 -- start start 2024-09-16T01:33:59.629 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.627+0000 7fec8eab7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fec88073160 0x7fec880b6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:59.629 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.627+0000 7fec8eab7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec880d0b20 con 0x7fec880b74a0 2024-09-16T01:33:59.629 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.627+0000 7fec8eab7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec880d0e00 con 0x7fec8815c770 2024-09-16T01:33:59.629 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.627+0000 7fec8eab7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fec880d2310 con 0x7fec88073160 2024-09-16T01:33:59.629 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.627+0000 7fec8c853640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fec8815c770 0x7fec880d0320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.154:35806/0 (socket says 172.21.15.154:35806) 2024-09-16T01:33:59.629 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.627+0000 7fec8c853640 1 -- 172.21.15.154:0/3816834199 learned_addr learned my addr 172.21.15.154:0/3816834199 (peer_addr_for_me v1:172.21.15.154:0/0) 2024-09-16T01:33:59.629 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.627+0000 7fec87fff640 1 --2- 172.21.15.154:0/3816834199 >> v2:172.21.15.88:3300/0 conn(0x7fec88073160 0x7fec880b6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:59.630 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.627+0000 7fec877fe640 1 -- 172.21.15.154:0/3816834199 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 389872906 0 0) 0x7fec880d0e00 con 0x7fec8815c770 2024-09-16T01:33:59.630 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.627+0000 7fec877fe640 1 -- 172.21.15.154:0/3816834199 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec78003520 con 0x7fec8815c770 2024-09-16T01:33:59.630 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.628+0000 7fec87fff640 1 -- 172.21.15.154:0/3816834199 >> v1:172.21.15.154:6789/0 conn(0x7fec8815c770 legacy=0x7fec880d0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.630 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.628+0000 7fec87fff640 1 -- 172.21.15.154:0/3816834199 >> v1:172.21.15.88:6789/0 conn(0x7fec880b74a0 legacy=0x7fec880b5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.630 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.628+0000 7fec877fe640 1 -- 172.21.15.154:0/3816834199 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 666258358 0 0) 0x7fec880d0b20 con 0x7fec880b74a0 2024-09-16T01:33:59.630 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.628+0000 7fec87fff640 1 -- 172.21.15.154:0/3816834199 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec880d10e0 con 0x7fec88073160 2024-09-16T01:33:59.631 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.629+0000 7fec87fff640 1 --2- 172.21.15.154:0/3816834199 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec88073160 0x7fec880b6f60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fec740097a0 tx=0x7fec7402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12cf9c412cf8c2bc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:33:59.631 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.629+0000 7fec877fe640 1 -- 172.21.15.154:0/3816834199 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec74037070 con 0x7fec88073160 2024-09-16T01:33:59.631 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.629+0000 7fec877fe640 1 -- 172.21.15.154:0/3816834199 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec74032440 con 0x7fec88073160 2024-09-16T01:33:59.631 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.629+0000 7fec877fe640 1 -- 172.21.15.154:0/3816834199 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec74032810 con 0x7fec88073160 2024-09-16T01:33:59.631 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.630+0000 7fec8eab7640 1 -- 172.21.15.154:0/3816834199 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec88073160 msgr2=0x7fec880b6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.631 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.630+0000 7fec8eab7640 1 --2- 172.21.15.154:0/3816834199 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec88073160 0x7fec880b6f60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fec740097a0 tx=0x7fec7402a230 comp rx=0 tx=0).stop 2024-09-16T01:33:59.631 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.630+0000 7fec8eab7640 1 -- 172.21.15.154:0/3816834199 shutdown_connections 2024-09-16T01:33:59.632 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.630+0000 7fec8eab7640 1 --2- 172.21.15.154:0/3816834199 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec88073160 0x7fec880b6f60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.632 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.630+0000 7fec8eab7640 1 -- 172.21.15.154:0/3816834199 >> 172.21.15.154:0/3816834199 conn(0x7fec880d24d0 msgr2=0x7fec880d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:33:59.632 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.630+0000 7fec8eab7640 1 -- 172.21.15.154:0/3816834199 shutdown_connections 2024-09-16T01:33:59.632 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.630+0000 7fec8eab7640 1 -- 172.21.15.154:0/3816834199 wait complete. 2024-09-16T01:33:59.632 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.631+0000 7fec8eab7640 1 Processor -- start 2024-09-16T01:33:59.632 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.631+0000 7fec8eab7640 1 -- start start 2024-09-16T01:33:59.634 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.632+0000 7fec8eab7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec88073160 0x7fec881f63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:59.635 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.632+0000 7fec8eab7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fec880b74a0 0x7fec881e4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:59.635 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.632+0000 7fec8eab7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec8815c770 0x7fec88141da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:59.635 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.632+0000 7fec8eab7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fec880b5f30 con 0x7fec88073160 2024-09-16T01:33:59.635 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.632+0000 7fec8eab7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fec880b78d0 con 0x7fec8815c770 2024-09-16T01:33:59.635 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.632+0000 7fec8eab7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fec880b60b0 con 0x7fec880b74a0 2024-09-16T01:33:59.635 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.632+0000 7fec87fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fec880b74a0 0x7fec881e4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:59.635 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.632+0000 7fec8d054640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec8815c770 0x7fec88141da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:59.635 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.632+0000 7fec8d054640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec8815c770 0x7fec88141da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:51214/0 (socket says 172.21.15.154:51214) 2024-09-16T01:33:59.635 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.632+0000 7fec87fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fec880b74a0 0x7fec881e4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.154:32812/0 (socket says 172.21.15.154:32812) 2024-09-16T01:33:59.635 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.632+0000 7fec8d054640 1 -- 172.21.15.154:0/3030484503 learned_addr learned my addr 172.21.15.154:0/3030484503 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-09-16T01:33:59.635 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.632+0000 7fec8c853640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec88073160 0x7fec881f63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:59.636 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.632+0000 7fec8d054640 1 -- 172.21.15.154:0/3030484503 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fec880b74a0 msgr2=0x7fec881e4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.636 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.632+0000 7fec8d054640 1 --2- 172.21.15.154:0/3030484503 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fec880b74a0 0x7fec881e4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.636 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.632+0000 7fec8d054640 1 -- 172.21.15.154:0/3030484503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec88073160 msgr2=0x7fec881f63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.636 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.632+0000 7fec8d054640 1 --2- 172.21.15.154:0/3030484503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec88073160 0x7fec881f63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.636 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.632+0000 7fec8d054640 1 -- 172.21.15.154:0/3030484503 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec88151f20 con 0x7fec8815c770 2024-09-16T01:33:59.636 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.632+0000 7fec8c853640 1 --2- 172.21.15.154:0/3030484503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec88073160 0x7fec881f63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:33:59.636 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.632+0000 7fec87fff640 1 --2- 172.21.15.154:0/3030484503 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fec880b74a0 0x7fec881e4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:33:59.636 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.632+0000 7fec8d054640 1 --2- 172.21.15.154:0/3030484503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec8815c770 0x7fec88141da0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fec8000c760 tx=0x7fec8000cc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:33:59.636 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.633+0000 7fec857fa640 1 -- 172.21.15.154:0/3030484503 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec80003da0 con 0x7fec8815c770 2024-09-16T01:33:59.636 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.633+0000 7fec857fa640 1 -- 172.21.15.154:0/3030484503 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec80005070 con 0x7fec8815c770 2024-09-16T01:33:59.636 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.633+0000 7fec8eab7640 1 -- 172.21.15.154:0/3030484503 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec88154960 con 0x7fec8815c770 2024-09-16T01:33:59.636 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.633+0000 7fec857fa640 1 -- 172.21.15.154:0/3030484503 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec80005460 con 0x7fec8815c770 2024-09-16T01:33:59.636 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.633+0000 7fec8eab7640 1 -- 172.21.15.154:0/3030484503 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec88152470 con 0x7fec8815c770 2024-09-16T01:33:59.636 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.633+0000 7fec857fa640 1 -- 172.21.15.154:0/3030484503 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fec80005600 con 0x7fec8815c770 2024-09-16T01:33:59.636 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.633+0000 7fec857fa640 1 -- 172.21.15.154:0/3030484503 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7fec8001b980 con 0x7fec8815c770 2024-09-16T01:33:59.637 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.634+0000 7fec8eab7640 1 -- 172.21.15.154:0/3030484503 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec880cede0 con 0x7fec8815c770 2024-09-16T01:33:59.637 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.636+0000 7fec857fa640 1 -- 172.21.15.154:0/3030484503 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fec8001b4f0 con 0x7fec8815c770 2024-09-16T01:33:59.706 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.705+0000 7fec8eab7640 1 -- 172.21.15.154:0/3030484503 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a", "id": 3} v 0) -- 0x7fec8815cfd0 con 0x7fec8815c770 2024-09-16T01:33:59.711 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.710+0000 7fec857fa640 1 -- 172.21.15.154:0/3030484503 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7fec8001b770 con 0x7fec8815c770 2024-09-16T01:33:59.711 INFO:teuthology.orchestra.run.smithi154.stdout:3 2024-09-16T01:33:59.713 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.711+0000 7fec8eab7640 1 -- 172.21.15.154:0/3030484503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec8815c770 msgr2=0x7fec88141da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.713 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.711+0000 7fec8eab7640 1 --2- 172.21.15.154:0/3030484503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec8815c770 0x7fec88141da0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fec8000c760 tx=0x7fec8000cc20 comp rx=0 tx=0).stop 2024-09-16T01:33:59.713 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.711+0000 7fec8eab7640 1 -- 172.21.15.154:0/3030484503 shutdown_connections 2024-09-16T01:33:59.713 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.711+0000 7fec8eab7640 1 --2- 172.21.15.154:0/3030484503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec8815c770 0x7fec88141da0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.713 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.711+0000 7fec8eab7640 1 --2- 172.21.15.154:0/3030484503 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fec880b74a0 0x7fec881e4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.713 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.711+0000 7fec8eab7640 1 --2- 172.21.15.154:0/3030484503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec88073160 0x7fec881f63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.713 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.711+0000 7fec8eab7640 1 -- 172.21.15.154:0/3030484503 >> 172.21.15.154:0/3030484503 conn(0x7fec880d24d0 msgr2=0x7fec880b5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:33:59.713 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.711+0000 7fec8eab7640 1 -- 172.21.15.154:0/3030484503 shutdown_connections 2024-09-16T01:33:59.713 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.711+0000 7fec8eab7640 1 -- 172.21.15.154:0/3030484503 wait complete. 2024-09-16T01:33:59.723 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new cb4dded8-72ed-4ac9-8761-39dd4b43692c 4 2024-09-16T01:33:59.822 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.821+0000 7f66100ac640 1 Processor -- start 2024-09-16T01:33:59.822 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.821+0000 7f66100ac640 1 -- start start 2024-09-16T01:33:59.824 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.822+0000 7f66100ac640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f660815c4c0 0x7f66080b55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:59.824 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.822+0000 7f66100ac640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6608072f40 con 0x7f6608159000 2024-09-16T01:33:59.824 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.822+0000 7f66100ac640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6608072980 con 0x7f66080b36a0 2024-09-16T01:33:59.824 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.822+0000 7f66100ac640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f66080b5130 con 0x7f660815c4c0 2024-09-16T01:33:59.824 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.822+0000 7f660d647640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f660815c4c0 0x7f66080b55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:59.824 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.822+0000 7f660e649640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f66080b36a0 0x7f660815ca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.154:35818/0 (socket says 172.21.15.154:35818) 2024-09-16T01:33:59.824 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.822+0000 7f660d647640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f660815c4c0 0x7f66080b55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:51216/0 (socket says 172.21.15.154:51216) 2024-09-16T01:33:59.824 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.822+0000 7f660e649640 1 -- 172.21.15.154:0/4050818923 learned_addr learned my addr 172.21.15.154:0/4050818923 (peer_addr_for_me v1:172.21.15.154:0/0) 2024-09-16T01:33:59.824 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.822+0000 7f660d647640 1 -- 172.21.15.154:0/4050818923 >> v1:172.21.15.154:6789/0 conn(0x7f66080b36a0 legacy=0x7f660815ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.824 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.822+0000 7f660d647640 1 -- 172.21.15.154:0/4050818923 >> v1:172.21.15.88:6789/0 conn(0x7f6608159000 legacy=0x7f6608072190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.825 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.822+0000 7f660ce46640 1 -- 172.21.15.154:0/4050818923 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1858208994 0 0) 0x7f6608072980 con 0x7f66080b36a0 2024-09-16T01:33:59.825 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.822+0000 7f660d647640 1 -- 172.21.15.154:0/4050818923 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6608072c60 con 0x7f660815c4c0 2024-09-16T01:33:59.826 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.823+0000 7f660d647640 1 --2- 172.21.15.154:0/4050818923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f660815c4c0 0x7f66080b55c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f65f80097a0 tx=0x7f65f802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ca11a7f7721a8d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:33:59.826 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.823+0000 7f660ce46640 1 -- 172.21.15.154:0/4050818923 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f8037070 con 0x7f660815c4c0 2024-09-16T01:33:59.826 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.824+0000 7f660ce46640 1 -- 172.21.15.154:0/4050818923 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65f8032440 con 0x7f660815c4c0 2024-09-16T01:33:59.826 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.824+0000 7f660ce46640 1 -- 172.21.15.154:0/4050818923 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f8032810 con 0x7f660815c4c0 2024-09-16T01:33:59.826 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.824+0000 7f66100ac640 1 -- 172.21.15.154:0/4050818923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f660815c4c0 msgr2=0x7f66080b55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.827 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.824+0000 7f66100ac640 1 --2- 172.21.15.154:0/4050818923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f660815c4c0 0x7f66080b55c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f65f80097a0 tx=0x7f65f802a230 comp rx=0 tx=0).stop 2024-09-16T01:33:59.827 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.824+0000 7f66100ac640 1 -- 172.21.15.154:0/4050818923 shutdown_connections 2024-09-16T01:33:59.827 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.824+0000 7f66100ac640 1 --2- 172.21.15.154:0/4050818923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f660815c4c0 0x7f66080b55c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.827 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.824+0000 7f66100ac640 1 -- 172.21.15.154:0/4050818923 >> 172.21.15.154:0/4050818923 conn(0x7f66080b6b50 msgr2=0x7f66080b6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:33:59.827 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.824+0000 7f66100ac640 1 -- 172.21.15.154:0/4050818923 shutdown_connections 2024-09-16T01:33:59.827 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.824+0000 7f66100ac640 1 -- 172.21.15.154:0/4050818923 wait complete. 2024-09-16T01:33:59.830 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.825+0000 7f66100ac640 1 Processor -- start 2024-09-16T01:33:59.830 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.825+0000 7f66100ac640 1 -- start start 2024-09-16T01:33:59.830 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.826+0000 7f66100ac640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66080b36a0 0x7f66080aeff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:59.830 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.826+0000 7f66100ac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6608159000 0x7f6608147ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:59.830 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.826+0000 7f66100ac640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f660815c4c0 0x7f66080b1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:33:59.830 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.826+0000 7f66100ac640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f660815d150 con 0x7f66080b36a0 2024-09-16T01:33:59.830 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.826+0000 7f66100ac640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f66080b5b00 con 0x7f6608159000 2024-09-16T01:33:59.830 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.826+0000 7f66100ac640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f660815d2d0 con 0x7f660815c4c0 2024-09-16T01:33:59.830 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.826+0000 7f660d647640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6608159000 0x7f6608147ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:59.830 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.826+0000 7f660d647640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6608159000 0x7f6608147ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:51222/0 (socket says 172.21.15.154:51222) 2024-09-16T01:33:59.830 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.826+0000 7f660d647640 1 -- 172.21.15.154:0/3006237938 learned_addr learned my addr 172.21.15.154:0/3006237938 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-09-16T01:33:59.830 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.826+0000 7f660e649640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f660815c4c0 0x7f66080b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:59.830 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.826+0000 7f660de48640 1 --2- 172.21.15.154:0/3006237938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66080b36a0 0x7f66080aeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:33:59.831 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.827+0000 7f660d647640 1 -- 172.21.15.154:0/3006237938 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f660815c4c0 msgr2=0x7f66080b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.831 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.827+0000 7f660d647640 1 --2- 172.21.15.154:0/3006237938 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f660815c4c0 0x7f66080b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.831 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.827+0000 7f660d647640 1 -- 172.21.15.154:0/3006237938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66080b36a0 msgr2=0x7f66080aeff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.831 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.827+0000 7f660d647640 1 --2- 172.21.15.154:0/3006237938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66080b36a0 0x7f66080aeff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.831 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.827+0000 7f660d647640 1 -- 172.21.15.154:0/3006237938 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66080af530 con 0x7f6608159000 2024-09-16T01:33:59.831 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.827+0000 7f660d647640 1 --2- 172.21.15.154:0/3006237938 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6608159000 0x7f6608147ef0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f65f8000c00 tx=0x7f65f80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:33:59.831 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.827+0000 7f65fe7fc640 1 -- 172.21.15.154:0/3006237938 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f8037040 con 0x7f6608159000 2024-09-16T01:33:59.831 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.827+0000 7f65fe7fc640 1 -- 172.21.15.154:0/3006237938 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65f8042070 con 0x7f6608159000 2024-09-16T01:33:59.831 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.827+0000 7f65fe7fc640 1 -- 172.21.15.154:0/3006237938 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f8030a70 con 0x7f6608159000 2024-09-16T01:33:59.831 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.827+0000 7f66100ac640 1 -- 172.21.15.154:0/3006237938 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66080adf50 con 0x7f6608159000 2024-09-16T01:33:59.831 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.827+0000 7f66100ac640 1 -- 172.21.15.154:0/3006237938 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66080ae510 con 0x7f6608159000 2024-09-16T01:33:59.831 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.827+0000 7f65fe7fc640 1 -- 172.21.15.154:0/3006237938 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f65f8030c10 con 0x7f6608159000 2024-09-16T01:33:59.831 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.827+0000 7f65fe7fc640 1 -- 172.21.15.154:0/3006237938 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f65f8047c80 con 0x7f6608159000 2024-09-16T01:33:59.831 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.828+0000 7f66100ac640 1 -- 172.21.15.154:0/3006237938 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66080b55c0 con 0x7f6608159000 2024-09-16T01:33:59.832 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.830+0000 7f65fe7fc640 1 -- 172.21.15.154:0/3006237938 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f65f8047460 con 0x7f6608159000 2024-09-16T01:33:59.901 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.900+0000 7f66100ac640 1 -- 172.21.15.154:0/3006237938 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "cb4dded8-72ed-4ac9-8761-39dd4b43692c", "id": 4} v 0) -- 0x7f6608072470 con 0x7f6608159000 2024-09-16T01:33:59.906 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.905+0000 7f65fe7fc640 1 -- 172.21.15.154:0/3006237938 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cb4dded8-72ed-4ac9-8761-39dd4b43692c", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f65f8047640 con 0x7f6608159000 2024-09-16T01:33:59.906 INFO:teuthology.orchestra.run.smithi154.stdout:4 2024-09-16T01:33:59.907 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.906+0000 7f66100ac640 1 -- 172.21.15.154:0/3006237938 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6608159000 msgr2=0x7f6608147ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:33:59.907 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.906+0000 7f66100ac640 1 --2- 172.21.15.154:0/3006237938 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6608159000 0x7f6608147ef0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f65f8000c00 tx=0x7f65f80046c0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.907 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.906+0000 7f66100ac640 1 -- 172.21.15.154:0/3006237938 shutdown_connections 2024-09-16T01:33:59.907 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.906+0000 7f66100ac640 1 --2- 172.21.15.154:0/3006237938 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f660815c4c0 0x7f66080b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.908 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.906+0000 7f66100ac640 1 --2- 172.21.15.154:0/3006237938 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6608159000 0x7f6608147ef0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.908 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.906+0000 7f66100ac640 1 --2- 172.21.15.154:0/3006237938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66080b36a0 0x7f66080aeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:33:59.908 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.906+0000 7f66100ac640 1 -- 172.21.15.154:0/3006237938 >> 172.21.15.154:0/3006237938 conn(0x7f66080b6b50 msgr2=0x7f660815e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:33:59.908 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.906+0000 7f66100ac640 1 -- 172.21.15.154:0/3006237938 shutdown_connections 2024-09-16T01:33:59.908 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:33:59.906+0000 7f66100ac640 1 -- 172.21.15.154:0/3006237938 wait complete. 2024-09-16T01:33:59.917 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new 35e0ad41-838d-43c4-b049-c158fe2b5da7 5 2024-09-16T01:34:00.018 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.017+0000 7fe7d2caf640 1 Processor -- start 2024-09-16T01:34:00.018 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.017+0000 7fe7d2caf640 1 -- start start 2024-09-16T01:34:00.019 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.017+0000 7fe7d2caf640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe7cc0ce390 0x7fe7cc071720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:00.019 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.017+0000 7fe7d2caf640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe7cc0b5830 0x7fe7cc0b5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:00.019 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.017+0000 7fe7d2caf640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7cc0b6170 con 0x7fe7cc15aa00 2024-09-16T01:34:00.020 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.017+0000 7fe7d2caf640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe7cc0cf5b0 con 0x7fe7cc0b5830 2024-09-16T01:34:00.020 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.017+0000 7fe7d2caf640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe7cc0ce9a0 con 0x7fe7cc0ce390 2024-09-16T01:34:00.020 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.018+0000 7fe7cbfff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe7cc0ce390 0x7fe7cc071720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:00.020 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.018+0000 7fe7cbfff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe7cc0ce390 0x7fe7cc071720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:51232/0 (socket says 172.21.15.154:51232) 2024-09-16T01:34:00.020 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.018+0000 7fe7cbfff640 1 -- 172.21.15.154:0/1158373592 learned_addr learned my addr 172.21.15.154:0/1158373592 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-09-16T01:34:00.020 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.018+0000 7fe7d124c640 1 --2- 172.21.15.154:0/1158373592 >> v2:172.21.15.88:3300/0 conn(0x7fe7cc0b5830 0x7fe7cc0b5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:00.020 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.018+0000 7fe7cbfff640 1 -- 172.21.15.154:0/1158373592 >> v2:172.21.15.88:3300/0 conn(0x7fe7cc0b5830 msgr2=0x7fe7cc0b5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:00.020 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.018+0000 7fe7cbfff640 1 --2- 172.21.15.154:0/1158373592 >> v2:172.21.15.88:3300/0 conn(0x7fe7cc0b5830 0x7fe7cc0b5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.020 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.018+0000 7fe7cbfff640 1 -- 172.21.15.154:0/1158373592 >> v1:172.21.15.88:6789/0 conn(0x7fe7cc15aa00 legacy=0x7fe7cc0b4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:00.020 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.018+0000 7fe7cbfff640 1 -- 172.21.15.154:0/1158373592 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7cc0cec80 con 0x7fe7cc0ce390 2024-09-16T01:34:00.022 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.019+0000 7fe7cbfff640 1 --2- 172.21.15.154:0/1158373592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7cc0ce390 0x7fe7cc071720 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0b6fd0 tx=0x7fe7b802c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e41aa3d19197170 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:00.022 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.019+0000 7fe7cb7fe640 1 -- 172.21.15.154:0/1158373592 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b8038070 con 0x7fe7cc0ce390 2024-09-16T01:34:00.022 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.019+0000 7fe7cb7fe640 1 -- 172.21.15.154:0/1158373592 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7b8031710 con 0x7fe7cc0ce390 2024-09-16T01:34:00.022 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.019+0000 7fe7cb7fe640 1 -- 172.21.15.154:0/1158373592 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b8031ae0 con 0x7fe7cc0ce390 2024-09-16T01:34:00.022 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.019+0000 7fe7d2caf640 1 -- 172.21.15.154:0/1158373592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7cc0ce390 msgr2=0x7fe7cc071720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:00.022 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.019+0000 7fe7d2caf640 1 --2- 172.21.15.154:0/1158373592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7cc0ce390 0x7fe7cc071720 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0b6fd0 tx=0x7fe7b802c480 comp rx=0 tx=0).stop 2024-09-16T01:34:00.022 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.019+0000 7fe7d2caf640 1 -- 172.21.15.154:0/1158373592 shutdown_connections 2024-09-16T01:34:00.022 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.019+0000 7fe7d2caf640 1 --2- 172.21.15.154:0/1158373592 >> v2:172.21.15.88:3300/0 conn(0x7fe7cc0b5830 0x7fe7cc0b5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.022 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.019+0000 7fe7d2caf640 1 --2- 172.21.15.154:0/1158373592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7cc0ce390 0x7fe7cc071720 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.022 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.019+0000 7fe7d2caf640 1 -- 172.21.15.154:0/1158373592 >> 172.21.15.154:0/1158373592 conn(0x7fe7cc0b0aa0 msgr2=0x7fe7cc0b0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:00.022 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.019+0000 7fe7d2caf640 1 -- 172.21.15.154:0/1158373592 shutdown_connections 2024-09-16T01:34:00.023 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.019+0000 7fe7d2caf640 1 -- 172.21.15.154:0/1158373592 wait complete. 2024-09-16T01:34:00.023 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.021+0000 7fe7d2caf640 1 Processor -- start 2024-09-16T01:34:00.023 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.021+0000 7fe7d2caf640 1 -- start start 2024-09-16T01:34:00.025 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.021+0000 7fe7d2caf640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7cc0b5830 0x7fe7cc1ef950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:00.025 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.021+0000 7fe7d2caf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7cc0ce390 0x7fe7cc0b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:00.025 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.021+0000 7fe7d2caf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7cc15aa00 0x7fe7cc0b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:00.025 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.022+0000 7fe7d2caf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe7cc15ade0 con 0x7fe7cc0ce390 2024-09-16T01:34:00.025 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.022+0000 7fe7d2caf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe7cc0cf240 con 0x7fe7cc15aa00 2024-09-16T01:34:00.025 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.022+0000 7fe7d2caf640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe7cc15af60 con 0x7fe7cc0b5830 2024-09-16T01:34:00.025 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.022+0000 7fe7d124c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7cc15aa00 0x7fe7cc0b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:00.025 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.022+0000 7fe7d0a4b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7cc0b5830 0x7fe7cc1ef950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:00.025 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.022+0000 7fe7d124c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7cc15aa00 0x7fe7cc0b4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:51238/0 (socket says 172.21.15.154:51238) 2024-09-16T01:34:00.026 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.022+0000 7fe7d124c640 1 -- 172.21.15.154:0/1871098499 learned_addr learned my addr 172.21.15.154:0/1871098499 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-09-16T01:34:00.026 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.022+0000 7fe7cbfff640 1 --2- 172.21.15.154:0/1871098499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7cc0ce390 0x7fe7cc0b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:00.026 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.022+0000 7fe7d124c640 1 -- 172.21.15.154:0/1871098499 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7cc0b5830 msgr2=0x7fe7cc1ef950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:00.026 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.022+0000 7fe7d124c640 1 --2- 172.21.15.154:0/1871098499 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7cc0b5830 0x7fe7cc1ef950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.026 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.022+0000 7fe7d124c640 1 -- 172.21.15.154:0/1871098499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7cc0ce390 msgr2=0x7fe7cc0b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:00.026 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.022+0000 7fe7d124c640 1 --2- 172.21.15.154:0/1871098499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7cc0ce390 0x7fe7cc0b4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.026 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.022+0000 7fe7d124c640 1 -- 172.21.15.154:0/1871098499 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7cc151660 con 0x7fe7cc15aa00 2024-09-16T01:34:00.026 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.022+0000 7fe7d0a4b640 1 --2- 172.21.15.154:0/1871098499 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7cc0b5830 0x7fe7cc1ef950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:34:00.026 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.022+0000 7fe7cbfff640 1 --2- 172.21.15.154:0/1871098499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7cc0ce390 0x7fe7cc0b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:34:00.026 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.022+0000 7fe7d124c640 1 --2- 172.21.15.154:0/1871098499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7cc15aa00 0x7fe7cc0b4ff0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4007e70 tx=0x7fe7c400e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:00.026 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.022+0000 7fe7c97fa640 1 -- 172.21.15.154:0/1871098499 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4014620 con 0x7fe7cc15aa00 2024-09-16T01:34:00.026 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.022+0000 7fe7c97fa640 1 -- 172.21.15.154:0/1871098499 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7c4019070 con 0x7fe7cc15aa00 2024-09-16T01:34:00.027 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.022+0000 7fe7d2caf640 1 -- 172.21.15.154:0/1871098499 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7cc150600 con 0x7fe7cc15aa00 2024-09-16T01:34:00.027 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.022+0000 7fe7c97fa640 1 -- 172.21.15.154:0/1871098499 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4014a20 con 0x7fe7cc15aa00 2024-09-16T01:34:00.027 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.022+0000 7fe7d2caf640 1 -- 172.21.15.154:0/1871098499 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7cc150b80 con 0x7fe7cc15aa00 2024-09-16T01:34:00.027 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.023+0000 7fe7c97fa640 1 -- 172.21.15.154:0/1871098499 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe7c4014bc0 con 0x7fe7cc15aa00 2024-09-16T01:34:00.027 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.023+0000 7fe7c97fa640 1 -- 172.21.15.154:0/1871098499 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fe7c401d940 con 0x7fe7cc15aa00 2024-09-16T01:34:00.027 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.023+0000 7fe7d2caf640 1 -- 172.21.15.154:0/1871098499 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7cc0b5c30 con 0x7fe7cc15aa00 2024-09-16T01:34:00.027 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.025+0000 7fe7c97fa640 1 -- 172.21.15.154:0/1871098499 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fe7c401dc20 con 0x7fe7cc15aa00 2024-09-16T01:34:00.096 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.095+0000 7fe7d2caf640 1 -- 172.21.15.154:0/1871098499 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "35e0ad41-838d-43c4-b049-c158fe2b5da7", "id": 5} v 0) -- 0x7fe7cc0b1dc0 con 0x7fe7cc15aa00 2024-09-16T01:34:00.099 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.098+0000 7fe7c97fa640 1 -- 172.21.15.154:0/1871098499 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "35e0ad41-838d-43c4-b049-c158fe2b5da7", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fe7c401de00 con 0x7fe7cc15aa00 2024-09-16T01:34:00.099 INFO:teuthology.orchestra.run.smithi154.stdout:5 2024-09-16T01:34:00.100 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.099+0000 7fe7d2caf640 1 -- 172.21.15.154:0/1871098499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7cc15aa00 msgr2=0x7fe7cc0b4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:00.100 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.099+0000 7fe7d2caf640 1 --2- 172.21.15.154:0/1871098499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7cc15aa00 0x7fe7cc0b4ff0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4007e70 tx=0x7fe7c400e4b0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.100 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.099+0000 7fe7d2caf640 1 -- 172.21.15.154:0/1871098499 shutdown_connections 2024-09-16T01:34:00.100 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.099+0000 7fe7d2caf640 1 --2- 172.21.15.154:0/1871098499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7cc15aa00 0x7fe7cc0b4ff0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.101 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.099+0000 7fe7d2caf640 1 --2- 172.21.15.154:0/1871098499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7cc0ce390 0x7fe7cc0b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.101 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.099+0000 7fe7d2caf640 1 --2- 172.21.15.154:0/1871098499 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7cc0b5830 0x7fe7cc1ef950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.101 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.099+0000 7fe7d2caf640 1 -- 172.21.15.154:0/1871098499 >> 172.21.15.154:0/1871098499 conn(0x7fe7cc0b0aa0 msgr2=0x7fe7cc06eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:00.101 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.099+0000 7fe7d2caf640 1 -- 172.21.15.154:0/1871098499 shutdown_connections 2024-09-16T01:34:00.101 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.099+0000 7fe7d2caf640 1 -- 172.21.15.154:0/1871098499 wait complete. 2024-09-16T01:34:00.111 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new fdbaae79-3ed1-45bb-a292-2dd48d3aaffd 6 2024-09-16T01:34:00.210 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.208+0000 7faddd2c9640 1 Processor -- start 2024-09-16T01:34:00.210 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.208+0000 7faddd2c9640 1 -- start start 2024-09-16T01:34:00.220 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.209+0000 7faddd2c9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fadd80b3ad0 0x7fadd80b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:00.220 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.209+0000 7faddd2c9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadd81483c0 con 0x7fadd80b36a0 2024-09-16T01:34:00.220 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.209+0000 7faddd2c9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadd80b45f0 con 0x7fadd8147fe0 2024-09-16T01:34:00.221 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.209+0000 7faddd2c9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fadd80d21c0 con 0x7fadd80b3ad0 2024-09-16T01:34:00.221 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.209+0000 7fadd6ffd640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fadd8147fe0 0x7fadd80d1ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.154:35822/0 (socket says 172.21.15.154:35822) 2024-09-16T01:34:00.221 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.209+0000 7fadd6ffd640 1 -- 172.21.15.154:0/3024607949 learned_addr learned my addr 172.21.15.154:0/3024607949 (peer_addr_for_me v1:172.21.15.154:0/0) 2024-09-16T01:34:00.221 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.209+0000 7fadd77fe640 1 --2- 172.21.15.154:0/3024607949 >> v2:172.21.15.88:3300/0 conn(0x7fadd80b3ad0 0x7fadd80b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:00.221 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.209+0000 7fadd5ffb640 1 -- 172.21.15.154:0/3024607949 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 771170970 0 0) 0x7fadd80b45f0 con 0x7fadd8147fe0 2024-09-16T01:34:00.221 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.209+0000 7fadd5ffb640 1 -- 172.21.15.154:0/3024607949 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fadc0003520 con 0x7fadd8147fe0 2024-09-16T01:34:00.222 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.210+0000 7fadd5ffb640 1 -- 172.21.15.154:0/3024607949 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3050390540 0 0) 0x7fadd81483c0 con 0x7fadd80b36a0 2024-09-16T01:34:00.222 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.210+0000 7fadd5ffb640 1 -- 172.21.15.154:0/3024607949 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fadd80b45f0 con 0x7fadd80b36a0 2024-09-16T01:34:00.222 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.210+0000 7fadd5ffb640 1 -- 172.21.15.154:0/3024607949 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 493135392 0 0) 0x7fadc0003520 con 0x7fadd8147fe0 2024-09-16T01:34:00.222 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.210+0000 7fadd77fe640 1 -- 172.21.15.154:0/3024607949 >> v1:172.21.15.154:6789/0 conn(0x7fadd8147fe0 legacy=0x7fadd80d1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:00.222 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.210+0000 7fadd77fe640 1 -- 172.21.15.154:0/3024607949 >> v1:172.21.15.88:6789/0 conn(0x7fadd80b36a0 legacy=0x7fadd80b5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:00.223 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.210+0000 7fadd77fe640 1 -- 172.21.15.154:0/3024607949 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd80b48d0 con 0x7fadd80b3ad0 2024-09-16T01:34:00.223 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.210+0000 7fadd77fe640 1 --2- 172.21.15.154:0/3024607949 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd80b3ad0 0x7fadd80b5da0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fadd0009d90 tx=0x7fadd002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2b0ba989695c4a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:00.223 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.211+0000 7fadd5ffb640 1 -- 172.21.15.154:0/3024607949 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc001670 con 0x7fadd80b3ad0 2024-09-16T01:34:00.223 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.211+0000 7fadd5ffb640 1 -- 172.21.15.154:0/3024607949 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadd0007540 con 0x7fadd80b3ad0 2024-09-16T01:34:00.223 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.211+0000 7fadd5ffb640 1 -- 172.21.15.154:0/3024607949 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd0007910 con 0x7fadd80b3ad0 2024-09-16T01:34:00.223 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.211+0000 7faddd2c9640 1 -- 172.21.15.154:0/3024607949 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd80b3ad0 msgr2=0x7fadd80b5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:00.224 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.211+0000 7faddd2c9640 1 --2- 172.21.15.154:0/3024607949 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd80b3ad0 0x7fadd80b5da0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fadd0009d90 tx=0x7fadd002a610 comp rx=0 tx=0).stop 2024-09-16T01:34:00.224 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.211+0000 7faddd2c9640 1 -- 172.21.15.154:0/3024607949 shutdown_connections 2024-09-16T01:34:00.224 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.211+0000 7faddd2c9640 1 --2- 172.21.15.154:0/3024607949 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd80b3ad0 0x7fadd80b5da0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.224 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.211+0000 7faddd2c9640 1 -- 172.21.15.154:0/3024607949 >> 172.21.15.154:0/3024607949 conn(0x7fadd80b6b50 msgr2=0x7fadd80b6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:00.224 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.211+0000 7faddd2c9640 1 -- 172.21.15.154:0/3024607949 shutdown_connections 2024-09-16T01:34:00.225 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.211+0000 7faddd2c9640 1 -- 172.21.15.154:0/3024607949 wait complete. 2024-09-16T01:34:00.225 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.213+0000 7faddd2c9640 1 Processor -- start 2024-09-16T01:34:00.225 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.213+0000 7faddd2c9640 1 -- start start 2024-09-16T01:34:00.225 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.213+0000 7faddd2c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadd80b36a0 0x7fadd8161620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:00.225 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.213+0000 7faddd2c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd80b3ad0 0x7fadd81f7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:00.225 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.214+0000 7faddd2c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd8147fe0 0x7fadd81f6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:00.226 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.214+0000 7faddd2c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fadd80b5170 con 0x7fadd80b3ad0 2024-09-16T01:34:00.226 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.214+0000 7faddd2c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fadd80b4e90 con 0x7fadd8147fe0 2024-09-16T01:34:00.226 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.214+0000 7faddd2c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fadd80b3f20 con 0x7fadd80b36a0 2024-09-16T01:34:00.226 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.214+0000 7fadd6ffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadd80b36a0 0x7fadd8161620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:00.226 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.214+0000 7fadd77fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd8147fe0 0x7fadd81f6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:00.227 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.214+0000 7fadd6ffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadd80b36a0 0x7fadd8161620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.154:32842/0 (socket says 172.21.15.154:32842) 2024-09-16T01:34:00.227 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.214+0000 7fadd77fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd8147fe0 0x7fadd81f6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:51250/0 (socket says 172.21.15.154:51250) 2024-09-16T01:34:00.227 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.214+0000 7fadd6ffd640 1 -- 172.21.15.154:0/3711106167 learned_addr learned my addr 172.21.15.154:0/3711106167 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-09-16T01:34:00.227 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.214+0000 7fadd67fc640 1 --2- 172.21.15.154:0/3711106167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd80b3ad0 0x7fadd81f7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:00.227 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.214+0000 7fadd67fc640 1 -- 172.21.15.154:0/3711106167 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadd80b36a0 msgr2=0x7fadd8161620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:00.228 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.214+0000 7fadd67fc640 1 --2- 172.21.15.154:0/3711106167 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadd80b36a0 0x7fadd8161620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.228 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.214+0000 7fadd67fc640 1 -- 172.21.15.154:0/3711106167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd8147fe0 msgr2=0x7fadd81f6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:00.228 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.214+0000 7fadd67fc640 1 --2- 172.21.15.154:0/3711106167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd8147fe0 0x7fadd81f6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.228 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.214+0000 7fadd67fc640 1 -- 172.21.15.154:0/3711106167 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd815f290 con 0x7fadd80b3ad0 2024-09-16T01:34:00.228 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.214+0000 7fadd67fc640 1 --2- 172.21.15.154:0/3711106167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd80b3ad0 0x7fadd81f7520 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fadc800c3c0 tx=0x7fadc800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:00.229 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.214+0000 7fadbf7fe640 1 -- 172.21.15.154:0/3711106167 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc8017070 con 0x7fadd80b3ad0 2024-09-16T01:34:00.229 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.214+0000 7fadbf7fe640 1 -- 172.21.15.154:0/3711106167 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadc80052f0 con 0x7fadd80b3ad0 2024-09-16T01:34:00.229 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.214+0000 7fadbf7fe640 1 -- 172.21.15.154:0/3711106167 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc8003830 con 0x7fadd80b3ad0 2024-09-16T01:34:00.229 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.214+0000 7faddd2c9640 1 -- 172.21.15.154:0/3711106167 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadd81f7290 con 0x7fadd80b3ad0 2024-09-16T01:34:00.229 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.215+0000 7faddd2c9640 1 -- 172.21.15.154:0/3711106167 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadd81f7f00 con 0x7fadd80b3ad0 2024-09-16T01:34:00.229 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.215+0000 7fadbf7fe640 1 -- 172.21.15.154:0/3711106167 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fadc801a070 con 0x7fadd80b3ad0 2024-09-16T01:34:00.230 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.215+0000 7fadbf7fe640 1 -- 172.21.15.154:0/3711106167 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fadc8011590 con 0x7fadd80b3ad0 2024-09-16T01:34:00.230 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.215+0000 7faddd2c9640 1 -- 172.21.15.154:0/3711106167 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadd80ae890 con 0x7fadd80b3ad0 2024-09-16T01:34:00.230 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.218+0000 7fadbf7fe640 1 -- 172.21.15.154:0/3711106167 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fadc8021020 con 0x7fadd80b3ad0 2024-09-16T01:34:00.288 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.287+0000 7faddd2c9640 1 -- 172.21.15.154:0/3711106167 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fdbaae79-3ed1-45bb-a292-2dd48d3aaffd", "id": 6} v 0) -- 0x7fadd815eef0 con 0x7fadd80b3ad0 2024-09-16T01:34:00.292 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.291+0000 7fadbf7fe640 1 -- 172.21.15.154:0/3711106167 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fdbaae79-3ed1-45bb-a292-2dd48d3aaffd", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7fadc8011b80 con 0x7fadd80b3ad0 2024-09-16T01:34:00.292 INFO:teuthology.orchestra.run.smithi154.stdout:6 2024-09-16T01:34:00.293 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.292+0000 7faddd2c9640 1 -- 172.21.15.154:0/3711106167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd80b3ad0 msgr2=0x7fadd81f7520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:00.293 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.292+0000 7faddd2c9640 1 --2- 172.21.15.154:0/3711106167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd80b3ad0 0x7fadd81f7520 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fadc800c3c0 tx=0x7fadc800c880 comp rx=0 tx=0).stop 2024-09-16T01:34:00.294 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.292+0000 7faddd2c9640 1 -- 172.21.15.154:0/3711106167 shutdown_connections 2024-09-16T01:34:00.294 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.292+0000 7faddd2c9640 1 --2- 172.21.15.154:0/3711106167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd8147fe0 0x7fadd81f6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.294 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.292+0000 7faddd2c9640 1 --2- 172.21.15.154:0/3711106167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd80b3ad0 0x7fadd81f7520 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.294 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.292+0000 7faddd2c9640 1 --2- 172.21.15.154:0/3711106167 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadd80b36a0 0x7fadd8161620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.294 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.292+0000 7faddd2c9640 1 -- 172.21.15.154:0/3711106167 >> 172.21.15.154:0/3711106167 conn(0x7fadd80b6b50 msgr2=0x7fadd8146d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:00.294 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.292+0000 7faddd2c9640 1 -- 172.21.15.154:0/3711106167 shutdown_connections 2024-09-16T01:34:00.294 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.292+0000 7faddd2c9640 1 -- 172.21.15.154:0/3711106167 wait complete. 2024-09-16T01:34:00.303 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new f6f6b035-c28a-4a9a-bff5-13cf359c2eac 7 2024-09-16T01:34:00.402 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.401+0000 7fe829b66640 1 Processor -- start 2024-09-16T01:34:00.402 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.401+0000 7fe829b66640 1 -- start start 2024-09-16T01:34:00.404 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.402+0000 7fe829b66640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe82415c4c0 0x7fe8240b55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:00.404 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.402+0000 7fe829b66640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8240b36a0 0x7fe8240b3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:00.404 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.402+0000 7fe829b66640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe824072f40 con 0x7fe824159000 2024-09-16T01:34:00.404 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.402+0000 7fe829b66640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe8240b5130 con 0x7fe82415c4c0 2024-09-16T01:34:00.404 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.402+0000 7fe829b66640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe824072980 con 0x7fe8240b36a0 2024-09-16T01:34:00.404 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.402+0000 7fe823fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8240b36a0 0x7fe8240b3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:00.404 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.402+0000 7fe823fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8240b36a0 0x7fe8240b3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.154:32854/0 (socket says 172.21.15.154:32854) 2024-09-16T01:34:00.404 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.402+0000 7fe823fff640 1 -- 172.21.15.154:0/2348025924 learned_addr learned my addr 172.21.15.154:0/2348025924 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-09-16T01:34:00.404 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.402+0000 7fe81affd640 1 --2- 172.21.15.154:0/2348025924 >> v2:172.21.15.88:3300/0 conn(0x7fe82415c4c0 0x7fe8240b55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:00.405 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.402+0000 7fe823fff640 1 -- 172.21.15.154:0/2348025924 >> v2:172.21.15.88:3300/0 conn(0x7fe82415c4c0 msgr2=0x7fe8240b55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:00.405 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.402+0000 7fe823fff640 1 --2- 172.21.15.154:0/2348025924 >> v2:172.21.15.88:3300/0 conn(0x7fe82415c4c0 0x7fe8240b55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.405 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.402+0000 7fe823fff640 1 -- 172.21.15.154:0/2348025924 >> v1:172.21.15.88:6789/0 conn(0x7fe824159000 legacy=0x7fe824072190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:00.405 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.402+0000 7fe823fff640 1 -- 172.21.15.154:0/2348025924 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe824072c60 con 0x7fe8240b36a0 2024-09-16T01:34:00.406 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.403+0000 7fe823fff640 1 --2- 172.21.15.154:0/2348025924 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8240b36a0 0x7fe8240b3b30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe81c009e50 tx=0x7fe81c02a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a208ee9b68c70b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:00.406 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.403+0000 7fe822ffd640 1 -- 172.21.15.154:0/2348025924 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81c036070 con 0x7fe8240b36a0 2024-09-16T01:34:00.406 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.403+0000 7fe822ffd640 1 -- 172.21.15.154:0/2348025924 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe81c031600 con 0x7fe8240b36a0 2024-09-16T01:34:00.406 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.403+0000 7fe822ffd640 1 -- 172.21.15.154:0/2348025924 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81c0319d0 con 0x7fe8240b36a0 2024-09-16T01:34:00.406 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.404+0000 7fe829b66640 1 -- 172.21.15.154:0/2348025924 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8240b36a0 msgr2=0x7fe8240b3b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:00.406 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.404+0000 7fe829b66640 1 --2- 172.21.15.154:0/2348025924 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8240b36a0 0x7fe8240b3b30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe81c009e50 tx=0x7fe81c02a710 comp rx=0 tx=0).stop 2024-09-16T01:34:00.406 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.404+0000 7fe829b66640 1 -- 172.21.15.154:0/2348025924 shutdown_connections 2024-09-16T01:34:00.406 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.404+0000 7fe829b66640 1 --2- 172.21.15.154:0/2348025924 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8240b36a0 0x7fe8240b3b30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.406 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.404+0000 7fe829b66640 1 --2- 172.21.15.154:0/2348025924 >> v2:172.21.15.88:3300/0 conn(0x7fe82415c4c0 0x7fe8240b55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.407 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.404+0000 7fe829b66640 1 -- 172.21.15.154:0/2348025924 >> 172.21.15.154:0/2348025924 conn(0x7fe8240b6b50 msgr2=0x7fe8240b6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:00.407 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.404+0000 7fe829b66640 1 -- 172.21.15.154:0/2348025924 shutdown_connections 2024-09-16T01:34:00.407 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.404+0000 7fe829b66640 1 -- 172.21.15.154:0/2348025924 wait complete. 2024-09-16T01:34:00.409 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.405+0000 7fe829b66640 1 Processor -- start 2024-09-16T01:34:00.409 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.405+0000 7fe829b66640 1 -- start start 2024-09-16T01:34:00.409 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.406+0000 7fe829b66640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8240b36a0 0x7fe8240adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:00.409 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.406+0000 7fe829b66640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe824159000 0x7fe8241490a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:00.409 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.406+0000 7fe829b66640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe82415c4c0 0x7fe82406bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:00.409 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.406+0000 7fe829b66640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe82415ca40 con 0x7fe82415c4c0 2024-09-16T01:34:00.409 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.406+0000 7fe829b66640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe8240b5b00 con 0x7fe824159000 2024-09-16T01:34:00.409 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.406+0000 7fe829b66640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe82415cbc0 con 0x7fe8240b36a0 2024-09-16T01:34:00.410 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.406+0000 7fe81affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe824159000 0x7fe8241490a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:00.410 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.406+0000 7fe8237fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8240b36a0 0x7fe8240adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:00.410 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.406+0000 7fe81affd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe824159000 0x7fe8241490a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:51264/0 (socket says 172.21.15.154:51264) 2024-09-16T01:34:00.410 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.406+0000 7fe81affd640 1 -- 172.21.15.154:0/1105206886 learned_addr learned my addr 172.21.15.154:0/1105206886 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-09-16T01:34:00.410 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.406+0000 7fe823fff640 1 --2- 172.21.15.154:0/1105206886 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe82415c4c0 0x7fe82406bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:00.410 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.407+0000 7fe81affd640 1 -- 172.21.15.154:0/1105206886 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8240b36a0 msgr2=0x7fe8240adda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:00.410 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.407+0000 7fe81affd640 1 --2- 172.21.15.154:0/1105206886 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8240b36a0 0x7fe8240adda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.410 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.407+0000 7fe81affd640 1 -- 172.21.15.154:0/1105206886 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe82415c4c0 msgr2=0x7fe82406bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:00.410 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.407+0000 7fe81affd640 1 --2- 172.21.15.154:0/1105206886 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe82415c4c0 0x7fe82406bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.410 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.407+0000 7fe81affd640 1 -- 172.21.15.154:0/1105206886 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8240ae2e0 con 0x7fe824159000 2024-09-16T01:34:00.410 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.407+0000 7fe823fff640 1 --2- 172.21.15.154:0/1105206886 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe82415c4c0 0x7fe82406bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:34:00.410 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.407+0000 7fe8237fe640 1 --2- 172.21.15.154:0/1105206886 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8240b36a0 0x7fe8240adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:34:00.410 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.407+0000 7fe81affd640 1 --2- 172.21.15.154:0/1105206886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe824159000 0x7fe8241490a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe80c00e870 tx=0x7fe80c00ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:00.411 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.407+0000 7fe820ff9640 1 -- 172.21.15.154:0/1105206886 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80c017070 con 0x7fe824159000 2024-09-16T01:34:00.411 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.407+0000 7fe820ff9640 1 -- 172.21.15.154:0/1105206886 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe80c00c940 con 0x7fe824159000 2024-09-16T01:34:00.411 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.407+0000 7fe829b66640 1 -- 172.21.15.154:0/1105206886 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe824149b80 con 0x7fe824159000 2024-09-16T01:34:00.411 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.407+0000 7fe820ff9640 1 -- 172.21.15.154:0/1105206886 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80c00cd30 con 0x7fe824159000 2024-09-16T01:34:00.411 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.407+0000 7fe829b66640 1 -- 172.21.15.154:0/1105206886 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe82414a140 con 0x7fe824159000 2024-09-16T01:34:00.411 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.407+0000 7fe820ff9640 1 -- 172.21.15.154:0/1105206886 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe80c01a070 con 0x7fe824159000 2024-09-16T01:34:00.411 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.407+0000 7fe820ff9640 1 -- 172.21.15.154:0/1105206886 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7fe80c013ad0 con 0x7fe824159000 2024-09-16T01:34:00.411 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.408+0000 7fe829b66640 1 -- 172.21.15.154:0/1105206886 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8240b55c0 con 0x7fe824159000 2024-09-16T01:34:00.411 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.410+0000 7fe820ff9640 1 -- 172.21.15.154:0/1105206886 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fe8240ae2e0 con 0x7fe824159000 2024-09-16T01:34:00.480 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.479+0000 7fe829b66640 1 -- 172.21.15.154:0/1105206886 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f6f6b035-c28a-4a9a-bff5-13cf359c2eac", "id": 7} v 0) -- 0x7fe8240b0940 con 0x7fe824159000 2024-09-16T01:34:00.488 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.487+0000 7fe820ff9640 1 -- 172.21.15.154:0/1105206886 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f6f6b035-c28a-4a9a-bff5-13cf359c2eac", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7fe80c013470 con 0x7fe824159000 2024-09-16T01:34:00.488 INFO:teuthology.orchestra.run.smithi154.stdout:7 2024-09-16T01:34:00.490 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.488+0000 7fe829b66640 1 -- 172.21.15.154:0/1105206886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe824159000 msgr2=0x7fe8241490a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:00.490 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.488+0000 7fe829b66640 1 --2- 172.21.15.154:0/1105206886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe824159000 0x7fe8241490a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe80c00e870 tx=0x7fe80c00ed30 comp rx=0 tx=0).stop 2024-09-16T01:34:00.490 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.488+0000 7fe829b66640 1 -- 172.21.15.154:0/1105206886 shutdown_connections 2024-09-16T01:34:00.490 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.488+0000 7fe829b66640 1 --2- 172.21.15.154:0/1105206886 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe82415c4c0 0x7fe82406bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.490 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.488+0000 7fe829b66640 1 --2- 172.21.15.154:0/1105206886 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe824159000 0x7fe8241490a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.490 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.488+0000 7fe829b66640 1 --2- 172.21.15.154:0/1105206886 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8240b36a0 0x7fe8240adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.490 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.488+0000 7fe829b66640 1 -- 172.21.15.154:0/1105206886 >> 172.21.15.154:0/1105206886 conn(0x7fe8240b6b50 msgr2=0x7fe8240b6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:00.490 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.488+0000 7fe829b66640 1 -- 172.21.15.154:0/1105206886 shutdown_connections 2024-09-16T01:34:00.490 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:00.488+0000 7fe829b66640 1 -- 172.21.15.154:0/1105206886 wait complete. 2024-09-16T01:34:00.500 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T01:34:00.500 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T01:34:00.502 INFO:tasks.ceph.osd.0:Started 2024-09-16T01:34:00.502 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T01:34:00.502 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T01:34:00.505 INFO:tasks.ceph.osd.1:Started 2024-09-16T01:34:00.505 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T01:34:00.505 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T01:34:00.507 INFO:tasks.ceph.osd.2:Started 2024-09-16T01:34:00.507 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T01:34:00.507 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T01:34:00.509 INFO:tasks.ceph.osd.3:Started 2024-09-16T01:34:00.509 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-16T01:34:00.509 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-16T01:34:00.511 INFO:tasks.ceph.osd.4:Started 2024-09-16T01:34:00.512 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-16T01:34:00.512 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-16T01:34:00.514 INFO:tasks.ceph.osd.5:Started 2024-09-16T01:34:00.514 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-16T01:34:00.514 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-16T01:34:00.516 INFO:tasks.ceph.osd.6:Started 2024-09-16T01:34:00.516 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-16T01:34:00.516 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-16T01:34:00.519 INFO:tasks.ceph.osd.7:Started 2024-09-16T01:34:00.520 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:34:00.566 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:34:00.565+0000 7f1b04f37040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T01:34:00.635 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:34:00.634+0000 7f1b04f37040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T01:34:00.638 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T01:34:00.637+0000 7f29693f9040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T01:34:00.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.642+0000 7efeece8d640 1 Processor -- start 2024-09-16T01:34:00.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.642+0000 7efeece8d640 1 -- start start 2024-09-16T01:34:00.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.643+0000 7efeece8d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efee81450d0 0x7efee80cd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:00.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.643+0000 7efeece8d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efee80cdcb0 0x7efee806e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:00.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.643+0000 7efeece8d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efee815f1d0 con 0x7efee806eb00 2024-09-16T01:34:00.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.643+0000 7efeece8d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7efee806ef50 con 0x7efee81450d0 2024-09-16T01:34:00.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.643+0000 7efeece8d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efee815f4b0 con 0x7efee80cdcb0 2024-09-16T01:34:00.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.643+0000 7efee6d9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7efee806eb00 0x7efee8071fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57450/0 (socket says 172.21.15.88:57450) 2024-09-16T01:34:00.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.643+0000 7efee6d9d640 1 -- 172.21.15.88:0/2330218928 learned_addr learned my addr 172.21.15.88:0/2330218928 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:00.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.643+0000 7efee659c640 1 --2- 172.21.15.88:0/2330218928 >> v2:172.21.15.88:3300/0 conn(0x7efee81450d0 0x7efee80cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:00.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.643+0000 7efee5d9b640 1 --2- 172.21.15.88:0/2330218928 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efee80cdcb0 0x7efee806e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:00.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.643+0000 7efee559a640 1 -- 172.21.15.88:0/2330218928 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 891228384 0 0) 0x7efee815f1d0 con 0x7efee806eb00 2024-09-16T01:34:00.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.643+0000 7efee559a640 1 -- 172.21.15.88:0/2330218928 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efed4003520 con 0x7efee806eb00 2024-09-16T01:34:00.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.644+0000 7efee659c640 1 -- 172.21.15.88:0/2330218928 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efee80cdcb0 msgr2=0x7efee806e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:00.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.644+0000 7efee659c640 1 --2- 172.21.15.88:0/2330218928 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efee80cdcb0 0x7efee806e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.644+0000 7efee659c640 1 -- 172.21.15.88:0/2330218928 >> v1:172.21.15.88:6789/0 conn(0x7efee806eb00 legacy=0x7efee8071fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:00.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.644+0000 7efee659c640 1 -- 172.21.15.88:0/2330218928 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efee815f790 con 0x7efee81450d0 2024-09-16T01:34:00.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.644+0000 7efee5d9b640 1 --2- 172.21.15.88:0/2330218928 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efee80cdcb0 0x7efee806e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:34:00.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.644+0000 7efee659c640 1 --2- 172.21.15.88:0/2330218928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efee81450d0 0x7efee80cd770 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efed80097a0 tx=0x7efed802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a5a2ec0e3fef588 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:00.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.644+0000 7efee559a640 1 -- 172.21.15.88:0/2330218928 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed8037070 con 0x7efee81450d0 2024-09-16T01:34:00.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.645+0000 7efee559a640 1 -- 172.21.15.88:0/2330218928 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efed802ae50 con 0x7efee81450d0 2024-09-16T01:34:00.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.645+0000 7efee559a640 1 -- 172.21.15.88:0/2330218928 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed8032660 con 0x7efee81450d0 2024-09-16T01:34:00.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.645+0000 7efeece8d640 1 -- 172.21.15.88:0/2330218928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efee81450d0 msgr2=0x7efee80cd770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:00.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.645+0000 7efeece8d640 1 --2- 172.21.15.88:0/2330218928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efee81450d0 0x7efee80cd770 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efed80097a0 tx=0x7efed802a100 comp rx=0 tx=0).stop 2024-09-16T01:34:00.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.645+0000 7efeece8d640 1 -- 172.21.15.88:0/2330218928 shutdown_connections 2024-09-16T01:34:00.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.645+0000 7efeece8d640 1 --2- 172.21.15.88:0/2330218928 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efee80cdcb0 0x7efee806e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.645+0000 7efeece8d640 1 --2- 172.21.15.88:0/2330218928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efee81450d0 0x7efee80cd770 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.645+0000 7efeece8d640 1 -- 172.21.15.88:0/2330218928 >> 172.21.15.88:0/2330218928 conn(0x7efee806e1e0 msgr2=0x7efee80b67c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:00.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.645+0000 7efeece8d640 1 -- 172.21.15.88:0/2330218928 shutdown_connections 2024-09-16T01:34:00.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.645+0000 7efeece8d640 1 -- 172.21.15.88:0/2330218928 wait complete. 2024-09-16T01:34:00.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.646+0000 7efeece8d640 1 Processor -- start 2024-09-16T01:34:00.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.647+0000 7efeece8d640 1 -- start start 2024-09-16T01:34:00.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.647+0000 7efeece8d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efee806eb00 0x7efee80ce870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:00.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.647+0000 7efeece8d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efee80cdcb0 0x7efee80cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:00.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.647+0000 7efeece8d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efee81450d0 0x7efee8154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:00.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.647+0000 7efeece8d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7efee80cd3e0 con 0x7efee81450d0 2024-09-16T01:34:00.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.647+0000 7efeece8d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efee815fd50 con 0x7efee80cdcb0 2024-09-16T01:34:00.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.647+0000 7efeece8d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efee80cd560 con 0x7efee806eb00 2024-09-16T01:34:00.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.647+0000 7efee6d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efee81450d0 0x7efee8154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:00.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.647+0000 7efee6d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efee81450d0 0x7efee8154630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39402/0 (socket says 172.21.15.88:39402) 2024-09-16T01:34:00.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.647+0000 7efee6d9d640 1 -- 172.21.15.88:0/2835433724 learned_addr learned my addr 172.21.15.88:0/2835433724 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:00.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.647+0000 7efee659c640 1 --2- 172.21.15.88:0/2835433724 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efee806eb00 0x7efee80ce870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:00.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.648+0000 7efee5d9b640 1 --2- 172.21.15.88:0/2835433724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efee80cdcb0 0x7efee80cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:00.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.648+0000 7efee6d9d640 1 -- 172.21.15.88:0/2835433724 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efee806eb00 msgr2=0x7efee80ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:00.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.648+0000 7efee6d9d640 1 --2- 172.21.15.88:0/2835433724 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efee806eb00 0x7efee80ce870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.648+0000 7efee6d9d640 1 -- 172.21.15.88:0/2835433724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efee80cdcb0 msgr2=0x7efee80cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:00.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.648+0000 7efee6d9d640 1 --2- 172.21.15.88:0/2835433724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efee80cdcb0 0x7efee80cedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.648+0000 7efee6d9d640 1 -- 172.21.15.88:0/2835433724 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efee8155640 con 0x7efee81450d0 2024-09-16T01:34:00.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.648+0000 7efee6d9d640 1 --2- 172.21.15.88:0/2835433724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efee81450d0 0x7efee8154630 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efee000cb70 tx=0x7efee000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:00.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.648+0000 7efedeffd640 1 -- 172.21.15.88:0/2835433724 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee0017070 con 0x7efee81450d0 2024-09-16T01:34:00.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.648+0000 7efedeffd640 1 -- 172.21.15.88:0/2835433724 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efee000ad40 con 0x7efee81450d0 2024-09-16T01:34:00.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.648+0000 7efeece8d640 1 -- 172.21.15.88:0/2835433724 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efee8155920 con 0x7efee81450d0 2024-09-16T01:34:00.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.648+0000 7efedeffd640 1 -- 172.21.15.88:0/2835433724 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee0012620 con 0x7efee81450d0 2024-09-16T01:34:00.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.648+0000 7efeece8d640 1 -- 172.21.15.88:0/2835433724 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efee8155ee0 con 0x7efee81450d0 2024-09-16T01:34:00.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.648+0000 7efedeffd640 1 -- 172.21.15.88:0/2835433724 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7efee001a070 con 0x7efee81450d0 2024-09-16T01:34:00.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.648+0000 7efedeffd640 1 -- 172.21.15.88:0/2835433724 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7efee0007740 con 0x7efee81450d0 2024-09-16T01:34:00.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.649+0000 7efeece8d640 1 -- 172.21.15.88:0/2835433724 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efee806e5c0 con 0x7efee81450d0 2024-09-16T01:34:00.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.651+0000 7efedeffd640 1 -- 172.21.15.88:0/2835433724 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7efee0021070 con 0x7efee81450d0 2024-09-16T01:34:00.692 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T01:34:00.691+0000 7f29693f9040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T01:34:00.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.710+0000 7efeece8d640 1 -- 172.21.15.88:0/2835433724 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efee80632f0 con 0x7efee81450d0 2024-09-16T01:34:00.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.711+0000 7efedeffd640 1 -- 172.21.15.88:0/2835433724 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7efee001c440 con 0x7efee81450d0 2024-09-16T01:34:00.712 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:00.712 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":10,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:34:00.481168+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:34:00.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.712+0000 7efeece8d640 1 -- 172.21.15.88:0/2835433724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efee81450d0 msgr2=0x7efee8154630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:00.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.712+0000 7efeece8d640 1 --2- 172.21.15.88:0/2835433724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efee81450d0 0x7efee8154630 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efee000cb70 tx=0x7efee000a430 comp rx=0 tx=0).stop 2024-09-16T01:34:00.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.712+0000 7efeece8d640 1 -- 172.21.15.88:0/2835433724 shutdown_connections 2024-09-16T01:34:00.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.712+0000 7efeece8d640 1 --2- 172.21.15.88:0/2835433724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efee81450d0 0x7efee8154630 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.712+0000 7efeece8d640 1 --2- 172.21.15.88:0/2835433724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efee80cdcb0 0x7efee80cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.712+0000 7efeece8d640 1 --2- 172.21.15.88:0/2835433724 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efee806eb00 0x7efee80ce870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:00.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.712+0000 7efeece8d640 1 -- 172.21.15.88:0/2835433724 >> 172.21.15.88:0/2835433724 conn(0x7efee806e1e0 msgr2=0x7efee80b3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:00.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.712+0000 7efeece8d640 1 -- 172.21.15.88:0/2835433724 shutdown_connections 2024-09-16T01:34:00.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:00.712+0000 7efeece8d640 1 -- 172.21.15.88:0/2835433724 wait complete. 2024-09-16T01:34:00.724 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T01:34:00.724 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T01:34:01.012 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:34:01.010+0000 7f1b04f37040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T01:34:01.063 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T01:34:01.062+0000 7f29693f9040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T01:34:01.096 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:34:01.095+0000 7f1b04f37040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T01:34:01.148 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T01:34:01.147+0000 7f29693f9040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T01:34:01.206 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:34:01.205+0000 7f1b04f37040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T01:34:01.252 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:34:01.251+0000 7f1b04f37040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T01:34:01.258 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T01:34:01.257+0000 7f29693f9040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T01:34:01.304 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T01:34:01.303+0000 7f29693f9040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T01:34:01.374 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:34:01.364+0000 7f1b04f37040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T01:34:01.416 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T01:34:01.415+0000 7f29693f9040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T01:34:01.419 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:34:01.418+0000 7f1b04f37040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T01:34:01.466 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:34:01.465+0000 7f1b04f37040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T01:34:01.472 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T01:34:01.471+0000 7f29693f9040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T01:34:01.518 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T01:34:01.517+0000 7f29693f9040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T01:34:01.575 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T01:34:01.578 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:34:01.575+0000 7f1b04f37040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T01:34:01.628 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T01:34:01.627+0000 7f29693f9040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T01:34:01.629 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:34:01.628+0000 7f1b04f37040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T01:34:01.665 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.663+0000 7f7d5fee6640 1 Processor -- start 2024-09-16T01:34:01.665 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.664+0000 7f7d5fee6640 1 -- start start 2024-09-16T01:34:01.666 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.664+0000 7f7d5fee6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7d58072290 0x7f7d5806dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:01.667 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.664+0000 7f7d5fee6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d58072670 0x7f7d580ce280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:01.667 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.664+0000 7f7d5fee6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d5806e640 con 0x7f7d580b46c0 2024-09-16T01:34:01.667 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.664+0000 7f7d5fee6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7d5806e920 con 0x7f7d58072290 2024-09-16T01:34:01.667 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.664+0000 7f7d5fee6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7d5806ec00 con 0x7f7d58072670 2024-09-16T01:34:01.667 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.664+0000 7f7d5d481640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7d580b46c0 0x7f7d5815af70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57460/0 (socket says 172.21.15.88:57460) 2024-09-16T01:34:01.667 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.665+0000 7f7d5d481640 1 -- 172.21.15.88:0/40293794 learned_addr learned my addr 172.21.15.88:0/40293794 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:01.667 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.665+0000 7f7d5dc82640 1 --2- 172.21.15.88:0/40293794 >> v2:172.21.15.154:3300/0 conn(0x7f7d58072290 0x7f7d5806dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:01.667 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.665+0000 7f7d5e483640 1 --2- 172.21.15.88:0/40293794 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d58072670 0x7f7d580ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:01.667 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.665+0000 7f7d5cc80640 1 -- 172.21.15.88:0/40293794 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3423998269 0 0) 0x7f7d5806e640 con 0x7f7d580b46c0 2024-09-16T01:34:01.667 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.665+0000 7f7d5cc80640 1 -- 172.21.15.88:0/40293794 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d44005730 con 0x7f7d580b46c0 2024-09-16T01:34:01.667 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.665+0000 7f7d5cc80640 1 -- 172.21.15.88:0/40293794 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2508730508 0 0) 0x7f7d44005730 con 0x7f7d580b46c0 2024-09-16T01:34:01.667 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.665+0000 7f7d5e483640 1 -- 172.21.15.88:0/40293794 >> v2:172.21.15.154:3300/0 conn(0x7f7d58072290 msgr2=0x7f7d5806dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:01.667 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.665+0000 7f7d5e483640 1 --2- 172.21.15.88:0/40293794 >> v2:172.21.15.154:3300/0 conn(0x7f7d58072290 0x7f7d5806dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:01.668 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.665+0000 7f7d5e483640 1 -- 172.21.15.88:0/40293794 >> v1:172.21.15.88:6789/0 conn(0x7f7d580b46c0 legacy=0x7f7d5815af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:01.668 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.665+0000 7f7d5e483640 1 -- 172.21.15.88:0/40293794 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d5806eee0 con 0x7f7d58072670 2024-09-16T01:34:01.668 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.665+0000 7f7d5dc82640 1 --2- 172.21.15.88:0/40293794 >> v2:172.21.15.154:3300/0 conn(0x7f7d58072290 0x7f7d5806dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:34:01.669 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.666+0000 7f7d5e483640 1 --2- 172.21.15.88:0/40293794 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d58072670 0x7f7d580ce280 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7d50009d90 tx=0x7f7d5002a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95463f1e989d3efa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:01.669 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.666+0000 7f7d5cc80640 1 -- 172.21.15.88:0/40293794 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d48001670 con 0x7f7d58072670 2024-09-16T01:34:01.669 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.666+0000 7f7d5cc80640 1 -- 172.21.15.88:0/40293794 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d50007540 con 0x7f7d58072670 2024-09-16T01:34:01.670 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.666+0000 7f7d5cc80640 1 -- 172.21.15.88:0/40293794 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d500078f0 con 0x7f7d58072670 2024-09-16T01:34:01.670 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.667+0000 7f7d5fee6640 1 -- 172.21.15.88:0/40293794 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d58072670 msgr2=0x7f7d580ce280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:01.670 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.667+0000 7f7d5fee6640 1 --2- 172.21.15.88:0/40293794 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d58072670 0x7f7d580ce280 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7d50009d90 tx=0x7f7d5002a5f0 comp rx=0 tx=0).stop 2024-09-16T01:34:01.670 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.667+0000 7f7d5fee6640 1 -- 172.21.15.88:0/40293794 shutdown_connections 2024-09-16T01:34:01.670 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.667+0000 7f7d5fee6640 1 --2- 172.21.15.88:0/40293794 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d58072670 0x7f7d580ce280 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:01.670 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.667+0000 7f7d5fee6640 1 --2- 172.21.15.88:0/40293794 >> v2:172.21.15.154:3300/0 conn(0x7f7d58072290 0x7f7d5806dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:01.670 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.667+0000 7f7d5fee6640 1 -- 172.21.15.88:0/40293794 >> 172.21.15.88:0/40293794 conn(0x7f7d580d15a0 msgr2=0x7f7d580d1980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:01.670 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.667+0000 7f7d5fee6640 1 -- 172.21.15.88:0/40293794 shutdown_connections 2024-09-16T01:34:01.670 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.667+0000 7f7d5fee6640 1 -- 172.21.15.88:0/40293794 wait complete. 2024-09-16T01:34:01.670 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.668+0000 7f7d5fee6640 1 Processor -- start 2024-09-16T01:34:01.670 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.668+0000 7f7d5fee6640 1 -- start start 2024-09-16T01:34:01.672 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.669+0000 7f7d5fee6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d58072290 0x7f7d5815e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:01.673 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.669+0000 7f7d5fee6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d58072670 0x7f7d581e5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:01.673 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.669+0000 7f7d5fee6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d580b46c0 0x7f7d5815ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:01.673 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.669+0000 7f7d5fee6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5806f6e0 con 0x7f7d58072670 2024-09-16T01:34:01.673 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.669+0000 7f7d5fee6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5806e280 con 0x7f7d580b46c0 2024-09-16T01:34:01.673 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.669+0000 7f7d5fee6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7d5815b9c0 con 0x7f7d58072290 2024-09-16T01:34:01.673 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.669+0000 7f7d5d481640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d58072670 0x7f7d581e5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:01.673 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.669+0000 7f7d5dc82640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d58072290 0x7f7d5815e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:01.673 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.669+0000 7f7d5dc82640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d58072290 0x7f7d5815e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:60456/0 (socket says 172.21.15.88:60456) 2024-09-16T01:34:01.673 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.669+0000 7f7d5dc82640 1 -- 172.21.15.88:0/3049005208 learned_addr learned my addr 172.21.15.88:0/3049005208 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:01.673 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.669+0000 7f7d5d481640 1 --2- 172.21.15.88:0/3049005208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d58072670 0x7f7d581e5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39410/0 (socket says 172.21.15.88:39410) 2024-09-16T01:34:01.674 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.669+0000 7f7d5e483640 1 --2- 172.21.15.88:0/3049005208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d580b46c0 0x7f7d5815ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:01.674 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.669+0000 7f7d5dc82640 1 -- 172.21.15.88:0/3049005208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d580b46c0 msgr2=0x7f7d5815ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:01.674 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.669+0000 7f7d5dc82640 1 --2- 172.21.15.88:0/3049005208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d580b46c0 0x7f7d5815ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:01.674 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.669+0000 7f7d5dc82640 1 -- 172.21.15.88:0/3049005208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d58072670 msgr2=0x7f7d581e5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:01.674 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.669+0000 7f7d5dc82640 1 --2- 172.21.15.88:0/3049005208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d58072670 0x7f7d581e5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:01.674 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.669+0000 7f7d5dc82640 1 -- 172.21.15.88:0/3049005208 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d5815bca0 con 0x7f7d58072290 2024-09-16T01:34:01.674 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.669+0000 7f7d5d481640 1 --2- 172.21.15.88:0/3049005208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d58072670 0x7f7d581e5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:34:01.674 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.669+0000 7f7d5dc82640 1 --2- 172.21.15.88:0/3049005208 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d58072290 0x7f7d5815e180 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7d5400e990 tx=0x7f7d5400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:01.674 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.669+0000 7f7d4e7fc640 1 -- 172.21.15.88:0/3049005208 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d54017070 con 0x7f7d58072290 2024-09-16T01:34:01.674 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.669+0000 7f7d4e7fc640 1 -- 172.21.15.88:0/3049005208 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d5400c9e0 con 0x7f7d58072290 2024-09-16T01:34:01.674 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.669+0000 7f7d5fee6640 1 -- 172.21.15.88:0/3049005208 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d5815bf80 con 0x7f7d58072290 2024-09-16T01:34:01.675 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.669+0000 7f7d4e7fc640 1 -- 172.21.15.88:0/3049005208 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5400cdb0 con 0x7f7d58072290 2024-09-16T01:34:01.675 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.669+0000 7f7d5fee6640 1 -- 172.21.15.88:0/3049005208 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d5815c540 con 0x7f7d58072290 2024-09-16T01:34:01.675 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.670+0000 7f7d4e7fc640 1 -- 172.21.15.88:0/3049005208 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7d5401a070 con 0x7f7d58072290 2024-09-16T01:34:01.675 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.670+0000 7f7d4e7fc640 1 -- 172.21.15.88:0/3049005208 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f7d5401cb20 con 0x7f7d58072290 2024-09-16T01:34:01.675 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.670+0000 7f7d5fee6640 1 -- 172.21.15.88:0/3049005208 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d5815b710 con 0x7f7d58072290 2024-09-16T01:34:01.676 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.675+0000 7f7d4e7fc640 1 -- 172.21.15.88:0/3049005208 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f7d5401c450 con 0x7f7d58072290 2024-09-16T01:34:01.681 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T01:34:01.679+0000 7f29693f9040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T01:34:01.685 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:34:01.684+0000 7f1b04f37040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T01:34:01.733 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T01:34:01.732+0000 7f29693f9040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T01:34:01.736 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.735+0000 7f7d5fee6640 1 -- 172.21.15.88:0/3049005208 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7d580b4da0 con 0x7f7d58072290 2024-09-16T01:34:01.737 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.736+0000 7f7d4e7fc640 1 -- 172.21.15.88:0/3049005208 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f7d5401c630 con 0x7f7d58072290 2024-09-16T01:34:01.737 INFO:teuthology.misc.health.smithi088.stdout: 2024-09-16T01:34:01.737 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:34:00.481168+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:34:01.738 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.737+0000 7f7d5fee6640 1 -- 172.21.15.88:0/3049005208 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d58072290 msgr2=0x7f7d5815e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:01.738 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.737+0000 7f7d5fee6640 1 --2- 172.21.15.88:0/3049005208 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d58072290 0x7f7d5815e180 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7d5400e990 tx=0x7f7d5400ee50 comp rx=0 tx=0).stop 2024-09-16T01:34:01.738 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.737+0000 7f7d5fee6640 1 -- 172.21.15.88:0/3049005208 shutdown_connections 2024-09-16T01:34:01.738 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.737+0000 7f7d5fee6640 1 --2- 172.21.15.88:0/3049005208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d580b46c0 0x7f7d5815ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:01.739 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.737+0000 7f7d5fee6640 1 --2- 172.21.15.88:0/3049005208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d58072670 0x7f7d581e5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:01.739 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.737+0000 7f7d5fee6640 1 --2- 172.21.15.88:0/3049005208 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d58072290 0x7f7d5815e180 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:01.739 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.737+0000 7f7d5fee6640 1 -- 172.21.15.88:0/3049005208 >> 172.21.15.88:0/3049005208 conn(0x7f7d580d15a0 msgr2=0x7f7d58159380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:01.739 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.737+0000 7f7d5fee6640 1 -- 172.21.15.88:0/3049005208 shutdown_connections 2024-09-16T01:34:01.739 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:01.737+0000 7f7d5fee6640 1 -- 172.21.15.88:0/3049005208 wait complete. 2024-09-16T01:34:01.747 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-16T01:34:01.756 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:34:01.755+0000 7f1b04f37040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T01:34:01.801 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:34:01.800+0000 7f1b04f37040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T01:34:01.803 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T01:34:01.802+0000 7f29693f9040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T01:34:01.845 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:34:01.843+0000 7f1b04f37040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T01:34:01.846 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T01:34:01.845+0000 7f29693f9040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T01:34:01.889 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T01:34:01.888+0000 7f29693f9040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T01:34:02.001 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-16T01:34:02.000+0000 7ff0c5ea5640 -1 Falling back to public interface 2024-09-16T01:34:02.227 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-09-16T01:34:02.225+0000 7fa498fd4640 -1 Falling back to public interface 2024-09-16T01:34:02.229 INFO:tasks.ceph.osd.5.smithi154.stderr:2024-09-16T01:34:02.228+0000 7f36892ae640 -1 Falling back to public interface 2024-09-16T01:34:02.230 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-09-16T01:34:02.229+0000 7f5ae741c640 -1 Falling back to public interface 2024-09-16T01:34:02.458 INFO:tasks.ceph.osd.0.smithi088.stderr:2024-09-16T01:34:02.457+0000 7f13a8444640 -1 Falling back to public interface 2024-09-16T01:34:02.475 INFO:tasks.ceph.osd.1.smithi088.stderr:2024-09-16T01:34:02.474+0000 7f325cc18640 -1 Falling back to public interface 2024-09-16T01:34:02.488 INFO:tasks.ceph.osd.4.smithi154.stderr:2024-09-16T01:34:02.486+0000 7fe02340f640 -1 Falling back to public interface 2024-09-16T01:34:02.491 INFO:tasks.ceph.osd.7.smithi154.stderr:2024-09-16T01:34:02.490+0000 7f50e681c640 -1 Falling back to public interface 2024-09-16T01:34:02.507 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:34:02.506+0000 7f1b04f37040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T01:34:02.555 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T01:34:02.554+0000 7f29693f9040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T01:34:03.522 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:34:03.520+0000 7f1aa5806640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T01:34:03.522 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-09-16T01:34:03.520+0000 7f1aa5806640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T01:34:06.027 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-09-16T01:34:06.026+0000 7fa498fd4640 -1 osd.3 0 log_to_monitors true 2024-09-16T01:34:06.304 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-16T01:34:06.302+0000 7ff0c5ea5640 -1 osd.6 0 log_to_monitors true 2024-09-16T01:34:06.450 INFO:tasks.ceph.osd.0.smithi088.stderr:2024-09-16T01:34:06.449+0000 7f13a8444640 -1 osd.0 0 log_to_monitors true 2024-09-16T01:34:06.513 INFO:tasks.ceph.osd.4.smithi154.stderr:2024-09-16T01:34:06.512+0000 7fe02340f640 -1 osd.4 0 log_to_monitors true 2024-09-16T01:34:06.811 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-09-16T01:34:06.810+0000 7f5ae741c640 -1 osd.2 0 log_to_monitors true 2024-09-16T01:34:06.817 INFO:tasks.ceph.osd.5.smithi154.stderr:2024-09-16T01:34:06.814+0000 7f36892ae640 -1 osd.5 0 log_to_monitors true 2024-09-16T01:34:06.833 INFO:tasks.ceph.osd.7.smithi154.stderr:2024-09-16T01:34:06.831+0000 7f50e681c640 -1 osd.7 0 log_to_monitors true 2024-09-16T01:34:07.009 INFO:tasks.ceph.osd.1.smithi088.stderr:2024-09-16T01:34:07.008+0000 7f325cc18640 -1 osd.1 0 log_to_monitors true 2024-09-16T01:34:08.554 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T01:34:08.669 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.668+0000 7f71ea0a6640 1 Processor -- start 2024-09-16T01:34:08.669 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.668+0000 7f71ea0a6640 1 -- start start 2024-09-16T01:34:08.671 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.668+0000 7f71ea0a6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e40b60c0 0x7f71e40b64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:08.671 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.668+0000 7f71ea0a6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f71e40d2b60 0x7f71e40d2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:08.671 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.668+0000 7f71ea0a6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71e40b77e0 con 0x7f71e406be50 2024-09-16T01:34:08.671 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.668+0000 7f71ea0a6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f71e40b69e0 con 0x7f71e40d2b60 2024-09-16T01:34:08.671 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.668+0000 7f71ea0a6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f71e40d3890 con 0x7f71e40b60c0 2024-09-16T01:34:08.671 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.669+0000 7f71e3fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f71e406be50 0x7f71e406c300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57470/0 (socket says 172.21.15.88:57470) 2024-09-16T01:34:08.671 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.669+0000 7f71e3fff640 1 -- 172.21.15.88:0/1339473637 learned_addr learned my addr 172.21.15.88:0/1339473637 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:08.672 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.669+0000 7f71e37fe640 1 --2- 172.21.15.88:0/1339473637 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e40b60c0 0x7f71e40b64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:08.672 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.669+0000 7f71e2ffd640 1 --2- 172.21.15.88:0/1339473637 >> v2:172.21.15.154:3300/0 conn(0x7f71e40d2b60 0x7f71e40d2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:08.672 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.669+0000 7f71e27fc640 1 -- 172.21.15.88:0/1339473637 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1144225639 0 0) 0x7f71e40b77e0 con 0x7f71e406be50 2024-09-16T01:34:08.672 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.669+0000 7f71e27fc640 1 -- 172.21.15.88:0/1339473637 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71cc003520 con 0x7f71e406be50 2024-09-16T01:34:08.672 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.669+0000 7f71e2ffd640 1 -- 172.21.15.88:0/1339473637 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e40b60c0 msgr2=0x7f71e40b64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:08.672 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.669+0000 7f71e2ffd640 1 --2- 172.21.15.88:0/1339473637 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e40b60c0 0x7f71e40b64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:08.672 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.669+0000 7f71e2ffd640 1 -- 172.21.15.88:0/1339473637 >> v1:172.21.15.88:6789/0 conn(0x7f71e406be50 legacy=0x7f71e406c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:08.672 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.669+0000 7f71e2ffd640 1 -- 172.21.15.88:0/1339473637 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71e40d3b70 con 0x7f71e40d2b60 2024-09-16T01:34:08.672 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.670+0000 7f71e2ffd640 1 --2- 172.21.15.88:0/1339473637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71e40d2b60 0x7f71e40d2f80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f71d80097a0 tx=0x7f71d802a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ffbf759ae6afdaa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:08.676 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.670+0000 7f71e27fc640 1 -- 172.21.15.88:0/1339473637 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71d8037070 con 0x7f71e40d2b60 2024-09-16T01:34:08.676 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.670+0000 7f71e27fc640 1 -- 172.21.15.88:0/1339473637 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71d8032440 con 0x7f71e40d2b60 2024-09-16T01:34:08.676 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.670+0000 7f71e27fc640 1 -- 172.21.15.88:0/1339473637 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71d80327f0 con 0x7f71e40d2b60 2024-09-16T01:34:08.677 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.671+0000 7f71ea0a6640 1 -- 172.21.15.88:0/1339473637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71e40d2b60 msgr2=0x7f71e40d2f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:08.677 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.671+0000 7f71ea0a6640 1 --2- 172.21.15.88:0/1339473637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71e40d2b60 0x7f71e40d2f80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f71d80097a0 tx=0x7f71d802a210 comp rx=0 tx=0).stop 2024-09-16T01:34:08.677 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.671+0000 7f71ea0a6640 1 -- 172.21.15.88:0/1339473637 shutdown_connections 2024-09-16T01:34:08.677 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.671+0000 7f71ea0a6640 1 --2- 172.21.15.88:0/1339473637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71e40d2b60 0x7f71e40d2f80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:08.677 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.671+0000 7f71ea0a6640 1 --2- 172.21.15.88:0/1339473637 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e40b60c0 0x7f71e40b64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:08.677 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.671+0000 7f71ea0a6640 1 -- 172.21.15.88:0/1339473637 >> 172.21.15.88:0/1339473637 conn(0x7f71e40d10c0 msgr2=0x7f71e40d14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:08.677 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.671+0000 7f71ea0a6640 1 -- 172.21.15.88:0/1339473637 shutdown_connections 2024-09-16T01:34:08.677 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.671+0000 7f71ea0a6640 1 -- 172.21.15.88:0/1339473637 wait complete. 2024-09-16T01:34:08.677 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.672+0000 7f71ea0a6640 1 Processor -- start 2024-09-16T01:34:08.677 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.672+0000 7f71ea0a6640 1 -- start start 2024-09-16T01:34:08.677 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.673+0000 7f71ea0a6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e406be50 0x7f71e41f3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:08.678 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.673+0000 7f71ea0a6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e40b60c0 0x7f71e40ca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:08.678 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.673+0000 7f71ea0a6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71e40d2b60 0x7f71e40cc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:08.678 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.673+0000 7f71ea0a6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f71e40d34c0 con 0x7f71e406be50 2024-09-16T01:34:08.678 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.673+0000 7f71ea0a6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f71e40d4130 con 0x7f71e40d2b60 2024-09-16T01:34:08.678 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.673+0000 7f71ea0a6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f71e40d3640 con 0x7f71e40b60c0 2024-09-16T01:34:08.678 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.673+0000 7f71e37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e406be50 0x7f71e41f3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:08.678 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.673+0000 7f71e37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e406be50 0x7f71e41f3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:39512/0 (socket says 172.21.15.88:39512) 2024-09-16T01:34:08.678 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.673+0000 7f71e37fe640 1 -- 172.21.15.88:0/2795020102 learned_addr learned my addr 172.21.15.88:0/2795020102 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:08.678 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.673+0000 7f71e2ffd640 1 --2- 172.21.15.88:0/2795020102 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e40b60c0 0x7f71e40ca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:08.678 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.673+0000 7f71e3fff640 1 --2- 172.21.15.88:0/2795020102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71e40d2b60 0x7f71e40cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:08.678 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.674+0000 7f71e37fe640 1 -- 172.21.15.88:0/2795020102 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e40b60c0 msgr2=0x7f71e40ca5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:08.678 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.674+0000 7f71e37fe640 1 --2- 172.21.15.88:0/2795020102 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e40b60c0 0x7f71e40ca5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:08.678 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.674+0000 7f71e37fe640 1 -- 172.21.15.88:0/2795020102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71e40d2b60 msgr2=0x7f71e40cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:08.678 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.674+0000 7f71e37fe640 1 --2- 172.21.15.88:0/2795020102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71e40d2b60 0x7f71e40cc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:08.679 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.674+0000 7f71e37fe640 1 -- 172.21.15.88:0/2795020102 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71e41f4e60 con 0x7f71e406be50 2024-09-16T01:34:08.679 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.674+0000 7f71e37fe640 1 --2- 172.21.15.88:0/2795020102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e406be50 0x7f71e41f3f50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f71d400e9b0 tx=0x7f71d400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:08.679 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.674+0000 7f71cbfff640 1 -- 172.21.15.88:0/2795020102 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71d4017070 con 0x7f71e406be50 2024-09-16T01:34:08.679 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.674+0000 7f71cbfff640 1 -- 172.21.15.88:0/2795020102 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71d400c9a0 con 0x7f71e406be50 2024-09-16T01:34:08.679 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.674+0000 7f71cbfff640 1 -- 172.21.15.88:0/2795020102 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71d400cd90 con 0x7f71e406be50 2024-09-16T01:34:08.679 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.674+0000 7f71ea0a6640 1 -- 172.21.15.88:0/2795020102 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71e41f48a0 con 0x7f71e406be50 2024-09-16T01:34:08.679 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.674+0000 7f71ea0a6640 1 -- 172.21.15.88:0/2795020102 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71e40cb840 con 0x7f71e406be50 2024-09-16T01:34:08.679 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.675+0000 7f71ea0a6640 1 -- 172.21.15.88:0/2795020102 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71e415e8e0 con 0x7f71e406be50 2024-09-16T01:34:08.679 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.675+0000 7f71cbfff640 1 -- 172.21.15.88:0/2795020102 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) ==== 90790+0+0 (secure 0 0 0) 0x7f71d401a070 con 0x7f71e406be50 2024-09-16T01:34:08.680 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.676+0000 7f71cbfff640 1 --2- 172.21.15.88:0/2795020102 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f71d006c220 0x7f71d006e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:08.680 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.676+0000 7f71cbfff640 1 -- 172.21.15.88:0/2795020102 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2231+0+0 (secure 0 0 0) 0x7f71d408a430 con 0x7f71e406be50 2024-09-16T01:34:08.680 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.676+0000 7f71e2ffd640 1 --2- 172.21.15.88:0/2795020102 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f71d006c220 0x7f71d006e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:08.680 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.676+0000 7f71e2ffd640 1 --2- 172.21.15.88:0/2795020102 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f71d006c220 0x7f71d006e6e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f71d8034000 tx=0x7f71d802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:08.682 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.681+0000 7f71cbfff640 1 -- 172.21.15.88:0/2795020102 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f71d4059450 con 0x7f71e406be50 2024-09-16T01:34:08.918 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-09-16T01:34:08.917+0000 7fa494740640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T01:34:08.943 INFO:tasks.ceph.osd.0.smithi088.stderr:2024-09-16T01:34:08.942+0000 7f13a3bb0640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T01:34:08.953 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.952+0000 7f71ea0a6640 1 -- 172.21.15.88:0/2795020102 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f71e40d1780 con 0x7f71e406be50 2024-09-16T01:34:08.954 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-09-16T01:34:08.953+0000 7fa48cd72640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:34:08.954 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.953+0000 7f71cbfff640 1 -- 172.21.15.88:0/2795020102 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+7058 (secure 0 0 0) 0x7f71d405d750 con 0x7f71e406be50 2024-09-16T01:34:08.954 INFO:teuthology.misc.health.smithi088.stdout: 2024-09-16T01:34:08.954 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":13,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:34:08.533009+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:34:08.957 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.955+0000 7f71ea0a6640 1 -- 172.21.15.88:0/2795020102 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f71d006c220 msgr2=0x7f71d006e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:08.957 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.955+0000 7f71ea0a6640 1 --2- 172.21.15.88:0/2795020102 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f71d006c220 0x7f71d006e6e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f71d8034000 tx=0x7f71d802e000 comp rx=0 tx=0).stop 2024-09-16T01:34:08.957 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.955+0000 7f71ea0a6640 1 -- 172.21.15.88:0/2795020102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e406be50 msgr2=0x7f71e41f3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:08.957 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.955+0000 7f71ea0a6640 1 --2- 172.21.15.88:0/2795020102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e406be50 0x7f71e41f3f50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f71d400e9b0 tx=0x7f71d400ee70 comp rx=0 tx=0).stop 2024-09-16T01:34:08.958 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.955+0000 7f71ea0a6640 1 -- 172.21.15.88:0/2795020102 shutdown_connections 2024-09-16T01:34:08.958 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.955+0000 7f71ea0a6640 1 --2- 172.21.15.88:0/2795020102 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f71d006c220 0x7f71d006e6e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:08.958 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.955+0000 7f71ea0a6640 1 --2- 172.21.15.88:0/2795020102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71e40d2b60 0x7f71e40cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:08.958 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.955+0000 7f71ea0a6640 1 --2- 172.21.15.88:0/2795020102 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e40b60c0 0x7f71e40ca5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:08.958 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.955+0000 7f71ea0a6640 1 --2- 172.21.15.88:0/2795020102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e406be50 0x7f71e41f3f50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:08.958 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.955+0000 7f71ea0a6640 1 -- 172.21.15.88:0/2795020102 >> 172.21.15.88:0/2795020102 conn(0x7f71e40d10c0 msgr2=0x7f71e415cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:08.958 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.955+0000 7f71ea0a6640 1 -- 172.21.15.88:0/2795020102 shutdown_connections 2024-09-16T01:34:08.958 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:08.955+0000 7f71ea0a6640 1 -- 172.21.15.88:0/2795020102 wait complete. 2024-09-16T01:34:08.966 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-16T01:34:08.967 INFO:tasks.ceph.osd.0.smithi088.stderr:2024-09-16T01:34:08.966+0000 7f139c1e2640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:34:09.049 INFO:tasks.ceph.osd.4.smithi154.stderr:2024-09-16T01:34:09.047+0000 7fe01e5d6640 -1 osd.4 0 waiting for initial osdmap 2024-09-16T01:34:09.058 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-16T01:34:09.056+0000 7ff0c1611640 -1 osd.6 0 waiting for initial osdmap 2024-09-16T01:34:09.108 INFO:tasks.ceph.osd.4.smithi154.stderr:2024-09-16T01:34:09.106+0000 7fe016c08640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:34:09.113 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-09-16T01:34:09.111+0000 7ff0b9c43640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:34:09.750 INFO:tasks.ceph.osd.7.smithi154.stderr:2024-09-16T01:34:09.747+0000 7f50e1a37640 -1 osd.7 0 waiting for initial osdmap 2024-09-16T01:34:09.769 INFO:tasks.ceph.osd.1.smithi088.stderr:2024-09-16T01:34:09.768+0000 7f3257e01640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T01:34:09.771 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-09-16T01:34:09.770+0000 7f5ae2b88640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T01:34:09.775 INFO:tasks.ceph.osd.7.smithi154.stderr:2024-09-16T01:34:09.773+0000 7f50da069640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:34:09.778 INFO:tasks.ceph.osd.1.smithi088.stderr:2024-09-16T01:34:09.777+0000 7f3250433640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:34:09.786 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-09-16T01:34:09.785+0000 7f5adb1ba640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:34:10.351 INFO:tasks.ceph.osd.5.smithi154.stderr:2024-09-16T01:34:10.349+0000 7f3684a1a640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T01:34:10.362 INFO:tasks.ceph.osd.5.smithi154.stderr:2024-09-16T01:34:10.360+0000 7f367d04c640 -1 osd.5 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T01:34:15.636 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:34:15.639 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:34:15.772 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T01:34:15.872 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.871+0000 7f90e32f1640 1 Processor -- start 2024-09-16T01:34:15.873 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.871+0000 7f90e32f1640 1 -- start start 2024-09-16T01:34:15.875 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.872+0000 7f90e32f1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f90dc06e5f0 0x7f90dc169420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:15.875 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.872+0000 7f90e32f1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90dc06eba0 con 0x7f90dc073e10 2024-09-16T01:34:15.875 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.872+0000 7f90e32f1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90dc070b80 con 0x7f90dc158e20 2024-09-16T01:34:15.876 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.872+0000 7f90e32f1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f90dc06cc30 con 0x7f90dc06e5f0 2024-09-16T01:34:15.876 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.872+0000 7f90e088c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f90dc073e10 0x7f90dc06dee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41500/0 (socket says 172.21.15.88:41500) 2024-09-16T01:34:15.876 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.872+0000 7f90e088c640 1 -- 172.21.15.88:0/3040445057 learned_addr learned my addr 172.21.15.88:0/3040445057 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:15.876 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.873+0000 7f90d3fff640 1 -- 172.21.15.88:0/3040445057 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2502277806 0 0) 0x7f90dc06eba0 con 0x7f90dc073e10 2024-09-16T01:34:15.876 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.873+0000 7f90d3fff640 1 -- 172.21.15.88:0/3040445057 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90c8003520 con 0x7f90dc073e10 2024-09-16T01:34:15.876 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.873+0000 7f90e188e640 1 --2- 172.21.15.88:0/3040445057 >> v2:172.21.15.154:3300/0 conn(0x7f90dc06e5f0 0x7f90dc169420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:15.876 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.873+0000 7f90d3fff640 1 -- 172.21.15.88:0/3040445057 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 509259912 0 0) 0x7f90c8003520 con 0x7f90dc073e10 2024-09-16T01:34:15.876 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.873+0000 7f90d3fff640 1 -- 172.21.15.88:0/3040445057 >> v2:172.21.15.154:3300/0 conn(0x7f90dc06e5f0 msgr2=0x7f90dc169420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:15.876 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.873+0000 7f90d3fff640 1 --2- 172.21.15.88:0/3040445057 >> v2:172.21.15.154:3300/0 conn(0x7f90dc06e5f0 0x7f90dc169420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:15.876 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.873+0000 7f90d3fff640 1 -- 172.21.15.88:0/3040445057 >> v1:172.21.15.154:6789/0 conn(0x7f90dc158e20 legacy=0x7f90dc073700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:15.876 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.873+0000 7f90d3fff640 1 -- 172.21.15.88:0/3040445057 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90dc070e60 con 0x7f90dc073e10 2024-09-16T01:34:15.876 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.873+0000 7f90d3fff640 1 -- 172.21.15.88:0/3040445057 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f90cc001670 con 0x7f90dc073e10 2024-09-16T01:34:15.877 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.873+0000 7f90d3fff640 1 -- 172.21.15.88:0/3040445057 >> v1:172.21.15.88:6789/0 conn(0x7f90dc073e10 legacy=0x7f90dc06dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:15.877 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.873+0000 7f90d3fff640 1 --2- 172.21.15.88:0/3040445057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90c8003a20 0x7f90c8003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:15.877 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.873+0000 7f90d3fff640 1 --2- 172.21.15.88:0/3040445057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90c8004800 0x7f90c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:15.877 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.873+0000 7f90d3fff640 1 --2- 172.21.15.88:0/3040445057 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90c80051c0 0x7f90c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:15.877 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.873+0000 7f90d3fff640 1 -- 172.21.15.88:0/3040445057 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f90c800c0c0 con 0x7f90c8004800 2024-09-16T01:34:15.877 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.873+0000 7f90d3fff640 1 -- 172.21.15.88:0/3040445057 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f90c800c270 con 0x7f90c8003a20 2024-09-16T01:34:15.877 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.873+0000 7f90d3fff640 1 -- 172.21.15.88:0/3040445057 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f90c800c470 con 0x7f90c80051c0 2024-09-16T01:34:15.877 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.873+0000 7f90e088c640 1 --2- 172.21.15.88:0/3040445057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90c8004800 0x7f90c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:15.877 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.873+0000 7f90e108d640 1 --2- 172.21.15.88:0/3040445057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90c8003a20 0x7f90c8003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:15.877 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.873+0000 7f90e188e640 1 --2- 172.21.15.88:0/3040445057 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90c80051c0 0x7f90c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:15.877 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.873+0000 7f90e088c640 1 -- 172.21.15.88:0/3040445057 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90c80051c0 msgr2=0x7f90c800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:15.877 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.873+0000 7f90e088c640 1 --2- 172.21.15.88:0/3040445057 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90c80051c0 0x7f90c800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:15.877 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.873+0000 7f90e088c640 1 -- 172.21.15.88:0/3040445057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90c8003a20 msgr2=0x7f90c8003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:15.878 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.873+0000 7f90e088c640 1 --2- 172.21.15.88:0/3040445057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90c8003a20 0x7f90c8003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:15.878 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.873+0000 7f90e088c640 1 -- 172.21.15.88:0/3040445057 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90c8003520 con 0x7f90c8004800 2024-09-16T01:34:15.878 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.873+0000 7f90e188e640 1 -- 172.21.15.88:0/3040445057 reap_dead start 2024-09-16T01:34:15.878 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.874+0000 7f90e088c640 1 --2- 172.21.15.88:0/3040445057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90c8004800 0x7f90c8004c80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f90cc00c210 tx=0x7f90c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d29ea1fd8ec2f44f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:15.878 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.874+0000 7f90d3fff640 1 -- 172.21.15.88:0/3040445057 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90cc007d30 con 0x7f90c8004800 2024-09-16T01:34:15.878 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.874+0000 7f90d3fff640 1 -- 172.21.15.88:0/3040445057 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90cc005ad0 con 0x7f90c8004800 2024-09-16T01:34:15.878 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.874+0000 7f90d3fff640 1 -- 172.21.15.88:0/3040445057 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90cc007070 con 0x7f90c8004800 2024-09-16T01:34:15.881 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.875+0000 7f90e32f1640 1 -- 172.21.15.88:0/3040445057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90c8004800 msgr2=0x7f90c8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:15.882 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.875+0000 7f90e32f1640 1 --2- 172.21.15.88:0/3040445057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90c8004800 0x7f90c8004c80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f90cc00c210 tx=0x7f90c8005830 comp rx=0 tx=0).stop 2024-09-16T01:34:15.882 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.875+0000 7f90e32f1640 1 -- 172.21.15.88:0/3040445057 shutdown_connections 2024-09-16T01:34:15.882 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.875+0000 7f90e32f1640 1 --2- 172.21.15.88:0/3040445057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90c8004800 0x7f90c8004c80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:15.882 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.875+0000 7f90e32f1640 1 -- 172.21.15.88:0/3040445057 >> 172.21.15.88:0/3040445057 conn(0x7f90dc0b4280 msgr2=0x7f90dc0b4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:15.882 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.875+0000 7f90e32f1640 1 -- 172.21.15.88:0/3040445057 shutdown_connections 2024-09-16T01:34:15.882 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.875+0000 7f90e32f1640 1 -- 172.21.15.88:0/3040445057 wait complete. 2024-09-16T01:34:15.882 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.876+0000 7f90e32f1640 1 Processor -- start 2024-09-16T01:34:15.882 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.876+0000 7f90e32f1640 1 -- start start 2024-09-16T01:34:15.882 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.877+0000 7f90e32f1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90c8004800 0x7f90dc1f33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:15.882 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.877+0000 7f90e32f1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90dc1f3900 0x7f90dc15cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:15.882 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.877+0000 7f90e32f1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90dc15d180 0x7f90dc0cea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:15.882 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.877+0000 7f90e32f1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f90dc06dee0 con 0x7f90dc1f3900 2024-09-16T01:34:15.883 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.877+0000 7f90e32f1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f90dc06e060 con 0x7f90c8004800 2024-09-16T01:34:15.883 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.877+0000 7f90e32f1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f90dc06e1e0 con 0x7f90dc15d180 2024-09-16T01:34:15.883 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.877+0000 7f90e088c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90dc1f3900 0x7f90dc15cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:15.883 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.877+0000 7f90e088c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90dc1f3900 0x7f90dc15cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59772/0 (socket says 172.21.15.88:59772) 2024-09-16T01:34:15.883 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.877+0000 7f90e088c640 1 -- 172.21.15.88:0/1334755384 learned_addr learned my addr 172.21.15.88:0/1334755384 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:15.883 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.877+0000 7f90e108d640 1 --2- 172.21.15.88:0/1334755384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90c8004800 0x7f90dc1f33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:15.883 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.877+0000 7f90e188e640 1 --2- 172.21.15.88:0/1334755384 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90dc15d180 0x7f90dc0cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:15.883 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.877+0000 7f90e088c640 1 -- 172.21.15.88:0/1334755384 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90dc15d180 msgr2=0x7f90dc0cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:15.883 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.877+0000 7f90e088c640 1 --2- 172.21.15.88:0/1334755384 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90dc15d180 0x7f90dc0cea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:15.883 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.877+0000 7f90e088c640 1 -- 172.21.15.88:0/1334755384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90c8004800 msgr2=0x7f90dc1f33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:15.883 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.877+0000 7f90e088c640 1 --2- 172.21.15.88:0/1334755384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90c8004800 0x7f90dc1f33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:15.883 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.877+0000 7f90e088c640 1 -- 172.21.15.88:0/1334755384 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90dc0cf7c0 con 0x7f90dc1f3900 2024-09-16T01:34:15.883 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.877+0000 7f90e188e640 1 --2- 172.21.15.88:0/1334755384 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90dc15d180 0x7f90dc0cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:15.884 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.878+0000 7f90e088c640 1 --2- 172.21.15.88:0/1334755384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90dc1f3900 0x7f90dc15cc40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f90cc001970 tx=0x7f90cc001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:15.884 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.878+0000 7f90d1ffb640 1 -- 172.21.15.88:0/1334755384 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90cc041020 con 0x7f90dc1f3900 2024-09-16T01:34:15.884 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.878+0000 7f90d1ffb640 1 -- 172.21.15.88:0/1334755384 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90cc038040 con 0x7f90dc1f3900 2024-09-16T01:34:15.884 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.878+0000 7f90d1ffb640 1 -- 172.21.15.88:0/1334755384 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90cc001670 con 0x7f90dc1f3900 2024-09-16T01:34:15.884 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.878+0000 7f90e32f1640 1 -- 172.21.15.88:0/1334755384 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90dc15a530 con 0x7f90dc1f3900 2024-09-16T01:34:15.884 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.878+0000 7f90e32f1640 1 -- 172.21.15.88:0/1334755384 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90dc15aaf0 con 0x7f90dc1f3900 2024-09-16T01:34:15.884 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.879+0000 7f90e32f1640 1 -- 172.21.15.88:0/1334755384 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90dc15d560 con 0x7f90dc1f3900 2024-09-16T01:34:15.884 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.879+0000 7f90d1ffb640 1 -- 172.21.15.88:0/1334755384 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f90cc044020 con 0x7f90dc1f3900 2024-09-16T01:34:15.884 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.880+0000 7f90d1ffb640 1 --2- 172.21.15.88:0/1334755384 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f90c006c2b0 0x7f90c006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:15.884 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.880+0000 7f90d1ffb640 1 -- 172.21.15.88:0/1334755384 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f90cc0ab360 con 0x7f90dc1f3900 2024-09-16T01:34:15.884 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.880+0000 7f90e108d640 1 --2- 172.21.15.88:0/1334755384 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f90c006c2b0 0x7f90c006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:15.884 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.880+0000 7f90e108d640 1 --2- 172.21.15.88:0/1334755384 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f90c006c2b0 0x7f90c006e770 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f90d8000c00 tx=0x7f90d8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:15.886 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:15.885+0000 7f90d1ffb640 1 -- 172.21.15.88:0/1334755384 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f90cc005aa0 con 0x7f90dc1f3900 2024-09-16T01:34:16.053 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:16.051+0000 7f90e32f1640 1 -- 172.21.15.88:0/1334755384 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f90dc0d3140 con 0x7f90dc1f3900 2024-09-16T01:34:16.053 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:16.052+0000 7f90d1ffb640 1 -- 172.21.15.88:0/1334755384 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12872 (secure 0 0 0) 0x7f90cc079eb0 con 0x7f90dc1f3900 2024-09-16T01:34:16.054 INFO:teuthology.misc.health.smithi088.stdout: 2024-09-16T01:34:16.054 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":17,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:34:12.952736+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:34:16.056 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:16.054+0000 7f90e32f1640 1 -- 172.21.15.88:0/1334755384 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f90c006c2b0 msgr2=0x7f90c006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:16.056 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:16.054+0000 7f90e32f1640 1 --2- 172.21.15.88:0/1334755384 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f90c006c2b0 0x7f90c006e770 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f90d8000c00 tx=0x7f90d8009040 comp rx=0 tx=0).stop 2024-09-16T01:34:16.056 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:16.054+0000 7f90e32f1640 1 -- 172.21.15.88:0/1334755384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90dc1f3900 msgr2=0x7f90dc15cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:16.056 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:16.054+0000 7f90e32f1640 1 --2- 172.21.15.88:0/1334755384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90dc1f3900 0x7f90dc15cc40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f90cc001970 tx=0x7f90cc001a40 comp rx=0 tx=0).stop 2024-09-16T01:34:16.057 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:16.054+0000 7f90e32f1640 1 -- 172.21.15.88:0/1334755384 shutdown_connections 2024-09-16T01:34:16.057 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:16.054+0000 7f90e32f1640 1 --2- 172.21.15.88:0/1334755384 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f90c006c2b0 0x7f90c006e770 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:16.057 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:16.054+0000 7f90e32f1640 1 --2- 172.21.15.88:0/1334755384 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90dc15d180 0x7f90dc0cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:16.057 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:16.054+0000 7f90e32f1640 1 --2- 172.21.15.88:0/1334755384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90dc1f3900 0x7f90dc15cc40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:16.057 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:16.054+0000 7f90e32f1640 1 --2- 172.21.15.88:0/1334755384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90c8004800 0x7f90dc1f33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:16.057 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:16.054+0000 7f90e32f1640 1 -- 172.21.15.88:0/1334755384 >> 172.21.15.88:0/1334755384 conn(0x7f90dc0b4280 msgr2=0x7f90dc0b6c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:16.057 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:16.054+0000 7f90e32f1640 1 -- 172.21.15.88:0/1334755384 shutdown_connections 2024-09-16T01:34:16.057 INFO:teuthology.misc.health.smithi088.stderr:2024-09-16T01:34:16.055+0000 7f90e32f1640 1 -- 172.21.15.88:0/1334755384 wait complete. 2024-09-16T01:34:16.065 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-09-16T01:34:16.065 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T01:34:16.065 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-16T01:34:16.065 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-16T01:34:16.067 INFO:tasks.ceph.mds.a:Started 2024-09-16T01:34:16.068 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-16T01:34:16.068 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-16T01:34:16.070 INFO:tasks.ceph.mds.c:Started 2024-09-16T01:34:16.070 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-16T01:34:16.071 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-16T01:34:16.073 INFO:tasks.ceph.mds.b:Started 2024-09-16T01:34:16.073 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-16T01:34:16.074 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-16T01:34:16.076 INFO:tasks.ceph.mds.d:Started 2024-09-16T01:34:16.076 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-09-16T01:34:16.076 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:34:16.141 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:34:16.141 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:34:16.160 INFO:tasks.ceph.mds.b.smithi154.stdout:starting mds.b at 2024-09-16T01:34:16.160 INFO:tasks.ceph.mds.d.smithi154.stdout:starting mds.d at 2024-09-16T01:34:16.161 INFO:tasks.ceph.mds.d.smithi154.stderr:2024-09-16T01:34:16.160+0000 7f9957e23640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-09-16T01:34:16.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.177+0000 7eff38e7b640 1 Processor -- start 2024-09-16T01:34:16.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.177+0000 7eff38e7b640 1 -- start start 2024-09-16T01:34:16.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.178+0000 7eff38e7b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7eff3415ec10 0x7eff340cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:16.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.178+0000 7eff38e7b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7eff340cf6a0 0x7eff340cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:16.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.178+0000 7eff38e7b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff3406bfd0 0x7eff3406c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:16.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.178+0000 7eff38e7b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7eff340d22a0 con 0x7eff3415ec10 2024-09-16T01:34:16.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.178+0000 7eff38e7b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7eff340d0020 con 0x7eff340cf6a0 2024-09-16T01:34:16.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.178+0000 7eff38e7b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7eff34163330 con 0x7eff3406bfd0 2024-09-16T01:34:16.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.178+0000 7eff3259c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7eff3415ec10 0x7eff340cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:16.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.178+0000 7eff3259c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7eff3415ec10 0x7eff340cefc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59806/0 (socket says 172.21.15.88:59806) 2024-09-16T01:34:16.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.178+0000 7eff3259c640 1 -- 172.21.15.88:0/2059857770 learned_addr learned my addr 172.21.15.88:0/2059857770 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:16.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.178+0000 7eff32d9d640 1 --2- 172.21.15.88:0/2059857770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff3406bfd0 0x7eff3406c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:16.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.178+0000 7eff31d9b640 1 --2- 172.21.15.88:0/2059857770 >> v2:172.21.15.154:3300/0 conn(0x7eff340cf6a0 0x7eff340cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:16.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.179+0000 7eff3259c640 1 -- 172.21.15.88:0/2059857770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff3406bfd0 msgr2=0x7eff3406c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:16.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.179+0000 7eff3259c640 1 --2- 172.21.15.88:0/2059857770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff3406bfd0 0x7eff3406c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:16.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.179+0000 7eff3259c640 1 -- 172.21.15.88:0/2059857770 >> v2:172.21.15.154:3300/0 conn(0x7eff340cf6a0 msgr2=0x7eff340cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:16.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.179+0000 7eff3259c640 1 --2- 172.21.15.88:0/2059857770 >> v2:172.21.15.154:3300/0 conn(0x7eff340cf6a0 0x7eff340cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:16.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.179+0000 7eff3259c640 1 -- 172.21.15.88:0/2059857770 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff340d24f0 con 0x7eff3415ec10 2024-09-16T01:34:16.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.179+0000 7eff32d9d640 1 --2- 172.21.15.88:0/2059857770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff3406bfd0 0x7eff3406c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:34:16.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.179+0000 7eff31d9b640 1 --2- 172.21.15.88:0/2059857770 >> v2:172.21.15.154:3300/0 conn(0x7eff340cf6a0 0x7eff340cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:34:16.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.179+0000 7eff3259c640 1 --2- 172.21.15.88:0/2059857770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff3415ec10 0x7eff340cefc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7eff340ba7c0 tx=0x7eff2402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d89c6094a3f1ff server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:16.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.179+0000 7eff3159a640 1 -- 172.21.15.88:0/2059857770 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff24037070 con 0x7eff3415ec10 2024-09-16T01:34:16.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.179+0000 7eff3159a640 1 -- 172.21.15.88:0/2059857770 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff24032440 con 0x7eff3415ec10 2024-09-16T01:34:16.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.179+0000 7eff3159a640 1 -- 172.21.15.88:0/2059857770 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff24032810 con 0x7eff3415ec10 2024-09-16T01:34:16.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.180+0000 7eff38e7b640 1 -- 172.21.15.88:0/2059857770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff3415ec10 msgr2=0x7eff340cefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:16.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.180+0000 7eff38e7b640 1 --2- 172.21.15.88:0/2059857770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff3415ec10 0x7eff340cefc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7eff340ba7c0 tx=0x7eff2402a140 comp rx=0 tx=0).stop 2024-09-16T01:34:16.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.180+0000 7eff38e7b640 1 -- 172.21.15.88:0/2059857770 shutdown_connections 2024-09-16T01:34:16.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.180+0000 7eff38e7b640 1 --2- 172.21.15.88:0/2059857770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff3406bfd0 0x7eff3406c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:16.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.180+0000 7eff38e7b640 1 --2- 172.21.15.88:0/2059857770 >> v2:172.21.15.154:3300/0 conn(0x7eff340cf6a0 0x7eff340cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:16.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.180+0000 7eff38e7b640 1 --2- 172.21.15.88:0/2059857770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff3415ec10 0x7eff340cefc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:16.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.180+0000 7eff38e7b640 1 -- 172.21.15.88:0/2059857770 >> 172.21.15.88:0/2059857770 conn(0x7eff340d46d0 msgr2=0x7eff340d4ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:16.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.180+0000 7eff38e7b640 1 -- 172.21.15.88:0/2059857770 shutdown_connections 2024-09-16T01:34:16.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.180+0000 7eff38e7b640 1 -- 172.21.15.88:0/2059857770 wait complete. 2024-09-16T01:34:16.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.182+0000 7eff38e7b640 1 Processor -- start 2024-09-16T01:34:16.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.182+0000 7eff38e7b640 1 -- start start 2024-09-16T01:34:16.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.182+0000 7eff38e7b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff3406bfd0 0x7eff341f5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:16.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.182+0000 7eff38e7b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff340cf6a0 0x7eff341f1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:16.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.183+0000 7eff38e7b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff3415ec10 0x7eff340ce280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:16.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.183+0000 7eff38e7b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7eff34162fc0 con 0x7eff340cf6a0 2024-09-16T01:34:16.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.183+0000 7eff38e7b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7eff340d2ab0 con 0x7eff3406bfd0 2024-09-16T01:34:16.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.183+0000 7eff38e7b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7eff340b3700 con 0x7eff3415ec10 2024-09-16T01:34:16.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.183+0000 7eff31d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff340cf6a0 0x7eff341f1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:16.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.183+0000 7eff3259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff3406bfd0 0x7eff341f5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:16.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.183+0000 7eff31d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff340cf6a0 0x7eff341f1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59808/0 (socket says 172.21.15.88:59808) 2024-09-16T01:34:16.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.183+0000 7eff31d9b640 1 -- 172.21.15.88:0/3984700588 learned_addr learned my addr 172.21.15.88:0/3984700588 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:16.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.183+0000 7eff32d9d640 1 --2- 172.21.15.88:0/3984700588 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff3415ec10 0x7eff340ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:16.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.183+0000 7eff3259c640 1 -- 172.21.15.88:0/3984700588 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff3415ec10 msgr2=0x7eff340ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:16.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.183+0000 7eff3259c640 1 --2- 172.21.15.88:0/3984700588 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff3415ec10 0x7eff340ce280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:16.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.183+0000 7eff3259c640 1 -- 172.21.15.88:0/3984700588 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff340cf6a0 msgr2=0x7eff341f1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:16.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.183+0000 7eff3259c640 1 --2- 172.21.15.88:0/3984700588 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff340cf6a0 0x7eff341f1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:16.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.183+0000 7eff3259c640 1 -- 172.21.15.88:0/3984700588 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff341f5550 con 0x7eff3406bfd0 2024-09-16T01:34:16.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.183+0000 7eff32d9d640 1 --2- 172.21.15.88:0/3984700588 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff3415ec10 0x7eff340ce280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:34:16.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.183+0000 7eff31d9b640 1 --2- 172.21.15.88:0/3984700588 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff340cf6a0 0x7eff341f1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:34:16.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.183+0000 7eff3259c640 1 --2- 172.21.15.88:0/3984700588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff3406bfd0 0x7eff341f5010 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7eff24031000 tx=0x7eff240046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:16.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.183+0000 7eff22ffd640 1 -- 172.21.15.88:0/3984700588 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff24041020 con 0x7eff3406bfd0 2024-09-16T01:34:16.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.183+0000 7eff22ffd640 1 -- 172.21.15.88:0/3984700588 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff2403d620 con 0x7eff3406bfd0 2024-09-16T01:34:16.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.183+0000 7eff38e7b640 1 -- 172.21.15.88:0/3984700588 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff340cbd60 con 0x7eff3406bfd0 2024-09-16T01:34:16.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.183+0000 7eff22ffd640 1 -- 172.21.15.88:0/3984700588 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff24037040 con 0x7eff3406bfd0 2024-09-16T01:34:16.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.184+0000 7eff38e7b640 1 -- 172.21.15.88:0/3984700588 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff340cda80 con 0x7eff3406bfd0 2024-09-16T01:34:16.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.184+0000 7eff38e7b640 1 -- 172.21.15.88:0/3984700588 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff340cfae0 con 0x7eff3406bfd0 2024-09-16T01:34:16.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.185+0000 7eff22ffd640 1 -- 172.21.15.88:0/3984700588 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7eff24044020 con 0x7eff3406bfd0 2024-09-16T01:34:16.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.185+0000 7eff22ffd640 1 --2- 172.21.15.88:0/3984700588 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff1406c2b0 0x7eff1406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:16.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.185+0000 7eff22ffd640 1 -- 172.21.15.88:0/3984700588 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7eff240ab580 con 0x7eff3406bfd0 2024-09-16T01:34:16.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.185+0000 7eff31d9b640 1 --2- 172.21.15.88:0/3984700588 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff1406c2b0 0x7eff1406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:16.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.186+0000 7eff31d9b640 1 --2- 172.21.15.88:0/3984700588 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff1406c2b0 0x7eff1406e770 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7eff341f7ae0 tx=0x7eff28009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:16.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.190+0000 7eff22ffd640 1 -- 172.21.15.88:0/3984700588 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7eff2404b030 con 0x7eff3406bfd0 2024-09-16T01:34:16.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.388+0000 7eff38e7b640 1 -- 172.21.15.88:0/3984700588 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff340d5330 con 0x7eff3406bfd0 2024-09-16T01:34:16.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.388+0000 7eff22ffd640 1 -- 172.21.15.88:0/3984700588 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1434 (secure 0 0 0) 0x7eff341f5550 con 0x7eff3406bfd0 2024-09-16T01:34:16.390 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:16.390 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":2,"btime":"2024-09-16T01:34:16:144639+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4293,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/4058543372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4058543372},{"type":"v1","addr":"172.21.15.88:6835","nonce":4058543372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-16T01:34:16.390 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 2 2024-09-16T01:34:16.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.390+0000 7eff38e7b640 1 -- 172.21.15.88:0/3984700588 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff1406c2b0 msgr2=0x7eff1406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:16.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.390+0000 7eff38e7b640 1 --2- 172.21.15.88:0/3984700588 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff1406c2b0 0x7eff1406e770 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7eff341f7ae0 tx=0x7eff28009190 comp rx=0 tx=0).stop 2024-09-16T01:34:16.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.390+0000 7eff38e7b640 1 -- 172.21.15.88:0/3984700588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff3406bfd0 msgr2=0x7eff341f5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:16.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.390+0000 7eff38e7b640 1 --2- 172.21.15.88:0/3984700588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff3406bfd0 0x7eff341f5010 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7eff24031000 tx=0x7eff240046c0 comp rx=0 tx=0).stop 2024-09-16T01:34:16.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.390+0000 7eff38e7b640 1 -- 172.21.15.88:0/3984700588 shutdown_connections 2024-09-16T01:34:16.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.390+0000 7eff38e7b640 1 --2- 172.21.15.88:0/3984700588 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff1406c2b0 0x7eff1406e770 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:16.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.390+0000 7eff38e7b640 1 --2- 172.21.15.88:0/3984700588 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff3415ec10 0x7eff340ce280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:16.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.390+0000 7eff38e7b640 1 --2- 172.21.15.88:0/3984700588 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff340cf6a0 0x7eff341f1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:16.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.390+0000 7eff38e7b640 1 --2- 172.21.15.88:0/3984700588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff3406bfd0 0x7eff341f5010 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:16.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.390+0000 7eff38e7b640 1 -- 172.21.15.88:0/3984700588 >> 172.21.15.88:0/3984700588 conn(0x7eff340d46d0 msgr2=0x7eff340ceee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:16.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.390+0000 7eff38e7b640 1 -- 172.21.15.88:0/3984700588 shutdown_connections 2024-09-16T01:34:16.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:16.390+0000 7eff38e7b640 1 -- 172.21.15.88:0/3984700588 wait complete. 2024-09-16T01:34:18.402 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:34:18.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.511+0000 7f7118b4b640 1 Processor -- start 2024-09-16T01:34:18.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.511+0000 7f7118b4b640 1 -- start start 2024-09-16T01:34:18.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.512+0000 7f7118b4b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f711415c2b0 0x7f71140b6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:18.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.512+0000 7f7118b4b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f711415f200 0x7f711406d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:18.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.512+0000 7f7118b4b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f711415fb10 con 0x7f71140b72f0 2024-09-16T01:34:18.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.512+0000 7f7118b4b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f71140b5900 con 0x7f711415c2b0 2024-09-16T01:34:18.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.512+0000 7f7118b4b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f711415f830 con 0x7f711415f200 2024-09-16T01:34:18.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.512+0000 7f711259c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f711415c2b0 0x7f71140b6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:18.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.512+0000 7f711259c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f711415c2b0 0x7f71140b6db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59816/0 (socket says 172.21.15.88:59816) 2024-09-16T01:34:18.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.512+0000 7f711259c640 1 -- 172.21.15.88:0/625906562 learned_addr learned my addr 172.21.15.88:0/625906562 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:18.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.512+0000 7f7111d9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f71140b72f0 0x7f711415eaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41522/0 (socket says 172.21.15.88:41522) 2024-09-16T01:34:18.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.512+0000 7f7112d9d640 1 --2- 172.21.15.88:0/625906562 >> v2:172.21.15.154:3300/0 conn(0x7f711415f200 0x7f711406d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:18.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.512+0000 7f711259c640 1 -- 172.21.15.88:0/625906562 >> v2:172.21.15.154:3300/0 conn(0x7f711415f200 msgr2=0x7f711406d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:18.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.512+0000 7f711259c640 1 --2- 172.21.15.88:0/625906562 >> v2:172.21.15.154:3300/0 conn(0x7f711415f200 0x7f711406d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:18.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.512+0000 7f711259c640 1 -- 172.21.15.88:0/625906562 >> v1:172.21.15.88:6789/0 conn(0x7f71140b72f0 legacy=0x7f711415eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:18.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.512+0000 7f711259c640 1 -- 172.21.15.88:0/625906562 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f711406ce30 con 0x7f711415c2b0 2024-09-16T01:34:18.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.512+0000 7f711159a640 1 -- 172.21.15.88:0/625906562 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4083880534 0 0) 0x7f711415fb10 con 0x7f71140b72f0 2024-09-16T01:34:18.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.513+0000 7f711259c640 1 --2- 172.21.15.88:0/625906562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f711415c2b0 0x7f71140b6db0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7114072d40 tx=0x7f710402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1512eb2d45d09aec server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:18.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.513+0000 7f711159a640 1 -- 172.21.15.88:0/625906562 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7104037070 con 0x7f711415c2b0 2024-09-16T01:34:18.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.513+0000 7f711159a640 1 -- 172.21.15.88:0/625906562 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7104032440 con 0x7f711415c2b0 2024-09-16T01:34:18.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.513+0000 7f711159a640 1 -- 172.21.15.88:0/625906562 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7104032810 con 0x7f711415c2b0 2024-09-16T01:34:18.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.514+0000 7f7118b4b640 1 -- 172.21.15.88:0/625906562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f711415c2b0 msgr2=0x7f71140b6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:18.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.514+0000 7f7118b4b640 1 --2- 172.21.15.88:0/625906562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f711415c2b0 0x7f71140b6db0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7114072d40 tx=0x7f710402a270 comp rx=0 tx=0).stop 2024-09-16T01:34:18.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.514+0000 7f7118b4b640 1 -- 172.21.15.88:0/625906562 shutdown_connections 2024-09-16T01:34:18.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.514+0000 7f7118b4b640 1 --2- 172.21.15.88:0/625906562 >> v2:172.21.15.154:3300/0 conn(0x7f711415f200 0x7f711406d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:18.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.514+0000 7f7118b4b640 1 --2- 172.21.15.88:0/625906562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f711415c2b0 0x7f71140b6db0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:18.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.514+0000 7f7118b4b640 1 -- 172.21.15.88:0/625906562 >> 172.21.15.88:0/625906562 conn(0x7f71140b4240 msgr2=0x7f71140b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:18.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.514+0000 7f7118b4b640 1 -- 172.21.15.88:0/625906562 shutdown_connections 2024-09-16T01:34:18.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.514+0000 7f7118b4b640 1 -- 172.21.15.88:0/625906562 wait complete. 2024-09-16T01:34:18.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.515+0000 7f7118b4b640 1 Processor -- start 2024-09-16T01:34:18.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.515+0000 7f7118b4b640 1 -- start start 2024-09-16T01:34:18.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.516+0000 7f7118b4b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71140b72f0 0x7f7114159e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:18.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.516+0000 7f7118b4b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f711415c2b0 0x7f711415a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:18.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.516+0000 7f7118b4b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f711415f200 0x7f71140d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:18.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.516+0000 7f7118b4b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f71140b7760 con 0x7f71140b72f0 2024-09-16T01:34:18.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.516+0000 7f7118b4b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f711406d3f0 con 0x7f711415c2b0 2024-09-16T01:34:18.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.516+0000 7f7118b4b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f711406de50 con 0x7f711415f200 2024-09-16T01:34:18.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.516+0000 7f711259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71140b72f0 0x7f7114159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:18.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.516+0000 7f711259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71140b72f0 0x7f7114159e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59832/0 (socket says 172.21.15.88:59832) 2024-09-16T01:34:18.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.516+0000 7f711259c640 1 -- 172.21.15.88:0/1478306747 learned_addr learned my addr 172.21.15.88:0/1478306747 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:18.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.516+0000 7f7112d9d640 1 --2- 172.21.15.88:0/1478306747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f711415f200 0x7f71140d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:18.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.516+0000 7f7111d9b640 1 --2- 172.21.15.88:0/1478306747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f711415c2b0 0x7f711415a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:18.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.517+0000 7f711259c640 1 -- 172.21.15.88:0/1478306747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f711415f200 msgr2=0x7f71140d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:18.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.517+0000 7f711259c640 1 --2- 172.21.15.88:0/1478306747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f711415f200 0x7f71140d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:18.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.517+0000 7f711259c640 1 -- 172.21.15.88:0/1478306747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f711415c2b0 msgr2=0x7f711415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:18.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.517+0000 7f711259c640 1 --2- 172.21.15.88:0/1478306747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f711415c2b0 0x7f711415a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:18.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.517+0000 7f711259c640 1 -- 172.21.15.88:0/1478306747 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71141e8c10 con 0x7f71140b72f0 2024-09-16T01:34:18.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.517+0000 7f711259c640 1 --2- 172.21.15.88:0/1478306747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71140b72f0 0x7f7114159e70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f710402a2b0 tx=0x7f71040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:18.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.517+0000 7f7102ffd640 1 -- 172.21.15.88:0/1478306747 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7104041020 con 0x7f71140b72f0 2024-09-16T01:34:18.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.517+0000 7f7102ffd640 1 -- 172.21.15.88:0/1478306747 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71040306c0 con 0x7f71140b72f0 2024-09-16T01:34:18.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.517+0000 7f7118b4b640 1 -- 172.21.15.88:0/1478306747 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71141e8ef0 con 0x7f71140b72f0 2024-09-16T01:34:18.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.517+0000 7f7102ffd640 1 -- 172.21.15.88:0/1478306747 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7104037040 con 0x7f71140b72f0 2024-09-16T01:34:18.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.517+0000 7f7118b4b640 1 -- 172.21.15.88:0/1478306747 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71141e94b0 con 0x7f71140b72f0 2024-09-16T01:34:18.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.518+0000 7f7118b4b640 1 -- 172.21.15.88:0/1478306747 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71140b6db0 con 0x7f71140b72f0 2024-09-16T01:34:18.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.519+0000 7f7102ffd640 1 -- 172.21.15.88:0/1478306747 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7104044020 con 0x7f71140b72f0 2024-09-16T01:34:18.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.519+0000 7f7102ffd640 1 --2- 172.21.15.88:0/1478306747 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f70f406c2e0 0x7f70f406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:18.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.519+0000 7f7102ffd640 1 -- 172.21.15.88:0/1478306747 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f71040ab900 con 0x7f71140b72f0 2024-09-16T01:34:18.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.519+0000 7f7111d9b640 1 --2- 172.21.15.88:0/1478306747 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f70f406c2e0 0x7f70f406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:18.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.519+0000 7f7111d9b640 1 --2- 172.21.15.88:0/1478306747 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f70f406c2e0 0x7f70f406e7a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f71140b41a0 tx=0x7f7108007b50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:18.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.524+0000 7f7102ffd640 1 -- 172.21.15.88:0/1478306747 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f710404b030 con 0x7f71140b72f0 2024-09-16T01:34:18.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.724+0000 7f7118b4b640 1 -- 172.21.15.88:0/1478306747 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f71140712a0 con 0x7f71140b72f0 2024-09-16T01:34:18.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.724+0000 7f7102ffd640 1 -- 172.21.15.88:0/1478306747 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3912 (secure 0 0 0) 0x7f710407a3c0 con 0x7f71140b72f0 2024-09-16T01:34:18.726 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:18.726 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":3,"btime":"2024-09-16T01:34:17:146097+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/1620784263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1620784263},{"type":"v1","addr":"172.21.15.154:6834","nonce":1620784263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3407296593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":3407296593},{"type":"v1","addr":"172.21.15.154:6835","nonce":3407296593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4293,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/4058543372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4058543372},{"type":"v1","addr":"172.21.15.88:6835","nonce":4058543372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4296,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/3712834803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":3712834803},{"type":"v1","addr":"172.21.15.88:6837","nonce":3712834803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-09-16T01:34:18.726 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 3 2024-09-16T01:34:18.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.726+0000 7f7118b4b640 1 -- 172.21.15.88:0/1478306747 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f70f406c2e0 msgr2=0x7f70f406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:18.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.726+0000 7f7118b4b640 1 --2- 172.21.15.88:0/1478306747 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f70f406c2e0 0x7f70f406e7a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f71140b41a0 tx=0x7f7108007b50 comp rx=0 tx=0).stop 2024-09-16T01:34:18.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.726+0000 7f7118b4b640 1 -- 172.21.15.88:0/1478306747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71140b72f0 msgr2=0x7f7114159e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:18.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.726+0000 7f7118b4b640 1 --2- 172.21.15.88:0/1478306747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71140b72f0 0x7f7114159e70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f710402a2b0 tx=0x7f71040046c0 comp rx=0 tx=0).stop 2024-09-16T01:34:18.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.726+0000 7f7118b4b640 1 -- 172.21.15.88:0/1478306747 shutdown_connections 2024-09-16T01:34:18.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.726+0000 7f7118b4b640 1 --2- 172.21.15.88:0/1478306747 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f70f406c2e0 0x7f70f406e7a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:18.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.726+0000 7f7118b4b640 1 --2- 172.21.15.88:0/1478306747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f711415f200 0x7f71140d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:18.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.726+0000 7f7118b4b640 1 --2- 172.21.15.88:0/1478306747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f711415c2b0 0x7f711415a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:18.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.726+0000 7f7118b4b640 1 --2- 172.21.15.88:0/1478306747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71140b72f0 0x7f7114159e70 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:18.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.726+0000 7f7118b4b640 1 -- 172.21.15.88:0/1478306747 >> 172.21.15.88:0/1478306747 conn(0x7f71140b4240 msgr2=0x7f711415e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:18.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.726+0000 7f7118b4b640 1 -- 172.21.15.88:0/1478306747 shutdown_connections 2024-09-16T01:34:18.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.726+0000 7f7118b4b640 1 -- 172.21.15.88:0/1478306747 wait complete. 2024-09-16T01:34:18.738 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-16T01:34:18.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.835+0000 7fc535f96640 1 Processor -- start 2024-09-16T01:34:18.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.835+0000 7fc535f96640 1 -- start start 2024-09-16T01:34:18.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.836+0000 7fc535f96640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc53015e7a0 0x7fc5300ba240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:18.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.836+0000 7fc535f96640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc5300d3fe0 0x7fc5300d2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:18.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.836+0000 7fc535f96640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc5300cf2f0 0x7fc5300cf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:18.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.836+0000 7fc535f96640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc5300d2cc0 con 0x7fc5300d3fe0 2024-09-16T01:34:18.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.836+0000 7fc535f96640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc5300d4400 con 0x7fc5300cf2f0 2024-09-16T01:34:18.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.836+0000 7fc535f96640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc5300d2f40 con 0x7fc53015e7a0 2024-09-16T01:34:18.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.836+0000 7fc52effd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc5300d3fe0 0x7fc5300d2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:18.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.836+0000 7fc52effd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc5300d3fe0 0x7fc5300d2780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59834/0 (socket says 172.21.15.88:59834) 2024-09-16T01:34:18.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.836+0000 7fc52effd640 1 -- 172.21.15.88:0/621152938 learned_addr learned my addr 172.21.15.88:0/621152938 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:18.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.836+0000 7fc52f7fe640 1 --2- 172.21.15.88:0/621152938 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc53015e7a0 0x7fc5300ba240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:18.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.836+0000 7fc52ffff640 1 --2- 172.21.15.88:0/621152938 >> v2:172.21.15.154:3300/0 conn(0x7fc5300cf2f0 0x7fc5300cf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:18.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.836+0000 7fc52effd640 1 -- 172.21.15.88:0/621152938 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc53015e7a0 msgr2=0x7fc5300ba240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:18.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.836+0000 7fc52effd640 1 --2- 172.21.15.88:0/621152938 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc53015e7a0 0x7fc5300ba240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:18.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.836+0000 7fc52effd640 1 -- 172.21.15.88:0/621152938 >> v2:172.21.15.154:3300/0 conn(0x7fc5300cf2f0 msgr2=0x7fc5300cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:18.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.836+0000 7fc52effd640 1 --2- 172.21.15.88:0/621152938 >> v2:172.21.15.154:3300/0 conn(0x7fc5300cf2f0 0x7fc5300cf7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:18.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.836+0000 7fc52effd640 1 -- 172.21.15.88:0/621152938 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5300d3220 con 0x7fc5300d3fe0 2024-09-16T01:34:18.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.837+0000 7fc52effd640 1 --2- 172.21.15.88:0/621152938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5300d3fe0 0x7fc5300d2780 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc5300b7590 tx=0x7fc52402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=997e7eb6196a2b99 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:18.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.837+0000 7fc52e7fc640 1 -- 172.21.15.88:0/621152938 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc524037070 con 0x7fc5300d3fe0 2024-09-16T01:34:18.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.837+0000 7fc52e7fc640 1 -- 172.21.15.88:0/621152938 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc524032440 con 0x7fc5300d3fe0 2024-09-16T01:34:18.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.837+0000 7fc52e7fc640 1 -- 172.21.15.88:0/621152938 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc524032810 con 0x7fc5300d3fe0 2024-09-16T01:34:18.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.837+0000 7fc535f96640 1 -- 172.21.15.88:0/621152938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5300d3fe0 msgr2=0x7fc5300d2780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:18.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.837+0000 7fc535f96640 1 --2- 172.21.15.88:0/621152938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5300d3fe0 0x7fc5300d2780 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc5300b7590 tx=0x7fc52402a270 comp rx=0 tx=0).stop 2024-09-16T01:34:18.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.837+0000 7fc535f96640 1 -- 172.21.15.88:0/621152938 shutdown_connections 2024-09-16T01:34:18.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.837+0000 7fc535f96640 1 --2- 172.21.15.88:0/621152938 >> v2:172.21.15.154:3300/0 conn(0x7fc5300cf2f0 0x7fc5300cf7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:18.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.837+0000 7fc535f96640 1 --2- 172.21.15.88:0/621152938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5300d3fe0 0x7fc5300d2780 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:18.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.837+0000 7fc535f96640 1 --2- 172.21.15.88:0/621152938 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc53015e7a0 0x7fc5300ba240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:18.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.837+0000 7fc535f96640 1 -- 172.21.15.88:0/621152938 >> 172.21.15.88:0/621152938 conn(0x7fc5300d1550 msgr2=0x7fc5300d1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:18.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.837+0000 7fc535f96640 1 -- 172.21.15.88:0/621152938 shutdown_connections 2024-09-16T01:34:18.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.837+0000 7fc535f96640 1 -- 172.21.15.88:0/621152938 wait complete. 2024-09-16T01:34:18.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.839+0000 7fc535f96640 1 Processor -- start 2024-09-16T01:34:18.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.839+0000 7fc535f96640 1 -- start start 2024-09-16T01:34:18.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.839+0000 7fc535f96640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5300cf2f0 0x7fc5300c9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:18.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.839+0000 7fc535f96640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5300d3fe0 0x7fc5301617d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:18.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.839+0000 7fc535f96640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc53015e7a0 0x7fc5301516f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:18.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.840+0000 7fc535f96640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc5300d3ac0 con 0x7fc5300d3fe0 2024-09-16T01:34:18.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.840+0000 7fc535f96640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc5300d37e0 con 0x7fc5300cf2f0 2024-09-16T01:34:18.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.840+0000 7fc535f96640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc53015eb80 con 0x7fc53015e7a0 2024-09-16T01:34:18.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.840+0000 7fc52effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5300d3fe0 0x7fc5301617d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:18.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.840+0000 7fc52effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5300d3fe0 0x7fc5301617d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59844/0 (socket says 172.21.15.88:59844) 2024-09-16T01:34:18.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.840+0000 7fc52effd640 1 -- 172.21.15.88:0/2826624423 learned_addr learned my addr 172.21.15.88:0/2826624423 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:18.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.840+0000 7fc52f7fe640 1 --2- 172.21.15.88:0/2826624423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5300cf2f0 0x7fc5300c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:18.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.840+0000 7fc52ffff640 1 --2- 172.21.15.88:0/2826624423 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc53015e7a0 0x7fc5301516f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:18.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.840+0000 7fc52effd640 1 -- 172.21.15.88:0/2826624423 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc53015e7a0 msgr2=0x7fc5301516f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:18.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.840+0000 7fc52effd640 1 --2- 172.21.15.88:0/2826624423 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc53015e7a0 0x7fc5301516f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:18.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.840+0000 7fc52effd640 1 -- 172.21.15.88:0/2826624423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5300cf2f0 msgr2=0x7fc5300c9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:18.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.840+0000 7fc52effd640 1 --2- 172.21.15.88:0/2826624423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5300cf2f0 0x7fc5300c9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:18.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.840+0000 7fc52effd640 1 -- 172.21.15.88:0/2826624423 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5300cba60 con 0x7fc5300d3fe0 2024-09-16T01:34:18.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.840+0000 7fc52f7fe640 1 --2- 172.21.15.88:0/2826624423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5300cf2f0 0x7fc5300c9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:18.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.840+0000 7fc52effd640 1 --2- 172.21.15.88:0/2826624423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5300d3fe0 0x7fc5301617d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc524000c00 tx=0x7fc5240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:18.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.840+0000 7fc517fff640 1 -- 172.21.15.88:0/2826624423 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc524041020 con 0x7fc5300d3fe0 2024-09-16T01:34:18.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.840+0000 7fc517fff640 1 -- 172.21.15.88:0/2826624423 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5240306c0 con 0x7fc5300d3fe0 2024-09-16T01:34:18.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.840+0000 7fc535f96640 1 -- 172.21.15.88:0/2826624423 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5300cbd40 con 0x7fc5300d3fe0 2024-09-16T01:34:18.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.840+0000 7fc517fff640 1 -- 172.21.15.88:0/2826624423 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc524037040 con 0x7fc5300d3fe0 2024-09-16T01:34:18.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.840+0000 7fc535f96640 1 -- 172.21.15.88:0/2826624423 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5300cc300 con 0x7fc5300d3fe0 2024-09-16T01:34:18.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.841+0000 7fc535f96640 1 -- 172.21.15.88:0/2826624423 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5300d2780 con 0x7fc5300d3fe0 2024-09-16T01:34:18.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.842+0000 7fc517fff640 1 -- 172.21.15.88:0/2826624423 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc524044020 con 0x7fc5300d3fe0 2024-09-16T01:34:18.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.842+0000 7fc517fff640 1 --2- 172.21.15.88:0/2826624423 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc51c06c2b0 0x7fc51c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:18.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.842+0000 7fc52f7fe640 1 --2- 172.21.15.88:0/2826624423 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc51c06c2b0 0x7fc51c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:18.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.842+0000 7fc517fff640 1 -- 172.21.15.88:0/2826624423 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fc5240ab920 con 0x7fc5300d3fe0 2024-09-16T01:34:18.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.843+0000 7fc52f7fe640 1 --2- 172.21.15.88:0/2826624423 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc51c06c2b0 0x7fc51c06e770 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc5301f7b00 tx=0x7fc520008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:18.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:18.847+0000 7fc517fff640 1 -- 172.21.15.88:0/2826624423 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc52404b030 con 0x7fc5300d3fe0 2024-09-16T01:34:19.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.013+0000 7fc535f96640 1 -- 172.21.15.88:0/2826624423 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fc530160330 con 0x7fc5300d3fe0 2024-09-16T01:34:19.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.013+0000 7fc517fff640 1 -- 172.21.15.88:0/2826624423 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7fc52407a3e0 con 0x7fc5300d3fe0 2024-09-16T01:34:19.015 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:19.015 INFO:teuthology.orchestra.run.smithi088.stdout:[ 2024-09-16T01:34:19.015 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:19.015 INFO:teuthology.orchestra.run.smithi088.stdout: "poolnum": 1, 2024-09-16T01:34:19.015 INFO:teuthology.orchestra.run.smithi088.stdout: "poolname": ".mgr" 2024-09-16T01:34:19.015 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:19.015 INFO:teuthology.orchestra.run.smithi088.stdout:] 2024-09-16T01:34:19.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.015+0000 7fc535f96640 1 -- 172.21.15.88:0/2826624423 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc51c06c2b0 msgr2=0x7fc51c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:19.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.015+0000 7fc535f96640 1 --2- 172.21.15.88:0/2826624423 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc51c06c2b0 0x7fc51c06e770 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc5301f7b00 tx=0x7fc520008040 comp rx=0 tx=0).stop 2024-09-16T01:34:19.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.015+0000 7fc535f96640 1 -- 172.21.15.88:0/2826624423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5300d3fe0 msgr2=0x7fc5301617d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:19.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.015+0000 7fc535f96640 1 --2- 172.21.15.88:0/2826624423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5300d3fe0 0x7fc5301617d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc524000c00 tx=0x7fc5240046c0 comp rx=0 tx=0).stop 2024-09-16T01:34:19.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.015+0000 7fc535f96640 1 -- 172.21.15.88:0/2826624423 shutdown_connections 2024-09-16T01:34:19.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.015+0000 7fc535f96640 1 --2- 172.21.15.88:0/2826624423 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc51c06c2b0 0x7fc51c06e770 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:19.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.015+0000 7fc535f96640 1 --2- 172.21.15.88:0/2826624423 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc53015e7a0 0x7fc5301516f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:19.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.015+0000 7fc535f96640 1 --2- 172.21.15.88:0/2826624423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5300d3fe0 0x7fc5301617d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:19.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.015+0000 7fc535f96640 1 --2- 172.21.15.88:0/2826624423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5300cf2f0 0x7fc5300c9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:19.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.015+0000 7fc535f96640 1 -- 172.21.15.88:0/2826624423 >> 172.21.15.88:0/2826624423 conn(0x7fc5300d1550 msgr2=0x7fc53006c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:19.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.015+0000 7fc535f96640 1 -- 172.21.15.88:0/2826624423 shutdown_connections 2024-09-16T01:34:19.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.015+0000 7fc535f96640 1 -- 172.21.15.88:0/2826624423 wait complete. 2024-09-16T01:34:19.027 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-16T01:34:19.027 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-16T01:34:19.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.127+0000 7f4135a3c640 1 Processor -- start 2024-09-16T01:34:19.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.127+0000 7f4135a3c640 1 -- start start 2024-09-16T01:34:19.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.128+0000 7f4135a3c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f41300ba230 0x7f41300d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:19.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.128+0000 7f4135a3c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f41300cf360 0x7f41300cf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:19.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.128+0000 7f4135a3c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41300d2a20 con 0x7f413015e980 2024-09-16T01:34:19.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.128+0000 7f4135a3c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f41300d27f0 con 0x7f41300cf360 2024-09-16T01:34:19.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.128+0000 7f4135a3c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f41300ba670 con 0x7f41300ba230 2024-09-16T01:34:19.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.128+0000 7f412f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f41300cf360 0x7f41300cf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:19.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.128+0000 7f412f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f41300cf360 0x7f41300cf7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59852/0 (socket says 172.21.15.88:59852) 2024-09-16T01:34:19.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.128+0000 7f412f7fe640 1 -- 172.21.15.88:0/597295660 learned_addr learned my addr 172.21.15.88:0/597295660 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:19.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.128+0000 7f41267fc640 1 --2- 172.21.15.88:0/597295660 >> v2:172.21.15.154:3300/0 conn(0x7f41300ba230 0x7f41300d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:19.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.128+0000 7f412f7fe640 1 -- 172.21.15.88:0/597295660 >> v2:172.21.15.154:3300/0 conn(0x7f41300ba230 msgr2=0x7f41300d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:19.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.128+0000 7f412f7fe640 1 --2- 172.21.15.88:0/597295660 >> v2:172.21.15.154:3300/0 conn(0x7f41300ba230 0x7f41300d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:19.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.128+0000 7f412f7fe640 1 -- 172.21.15.88:0/597295660 >> v1:172.21.15.154:6789/0 conn(0x7f413015e980 legacy=0x7f41300b7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:19.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.128+0000 7f412f7fe640 1 -- 172.21.15.88:0/597295660 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41300d2d00 con 0x7f41300cf360 2024-09-16T01:34:19.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.129+0000 7f412f7fe640 1 --2- 172.21.15.88:0/597295660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41300cf360 0x7f41300cf7f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f413008d9e0 tx=0x7f412802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2009e147d1235954 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:19.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.129+0000 7f412e7fc640 1 -- 172.21.15.88:0/597295660 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4128036070 con 0x7f41300cf360 2024-09-16T01:34:19.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.129+0000 7f412e7fc640 1 -- 172.21.15.88:0/597295660 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4128031600 con 0x7f41300cf360 2024-09-16T01:34:19.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.129+0000 7f412e7fc640 1 -- 172.21.15.88:0/597295660 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41280319d0 con 0x7f41300cf360 2024-09-16T01:34:19.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.129+0000 7f4135a3c640 1 -- 172.21.15.88:0/597295660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41300cf360 msgr2=0x7f41300cf7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:19.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.129+0000 7f4135a3c640 1 --2- 172.21.15.88:0/597295660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41300cf360 0x7f41300cf7f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f413008d9e0 tx=0x7f412802a710 comp rx=0 tx=0).stop 2024-09-16T01:34:19.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.130+0000 7f4135a3c640 1 -- 172.21.15.88:0/597295660 shutdown_connections 2024-09-16T01:34:19.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.130+0000 7f4135a3c640 1 --2- 172.21.15.88:0/597295660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41300cf360 0x7f41300cf7f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:19.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.130+0000 7f4135a3c640 1 --2- 172.21.15.88:0/597295660 >> v2:172.21.15.154:3300/0 conn(0x7f41300ba230 0x7f41300d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:19.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.130+0000 7f4135a3c640 1 -- 172.21.15.88:0/597295660 >> 172.21.15.88:0/597295660 conn(0x7f41300d46e0 msgr2=0x7f41300d4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:19.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.130+0000 7f4135a3c640 1 -- 172.21.15.88:0/597295660 shutdown_connections 2024-09-16T01:34:19.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.130+0000 7f4135a3c640 1 -- 172.21.15.88:0/597295660 wait complete. 2024-09-16T01:34:19.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.131+0000 7f4135a3c640 1 Processor -- start 2024-09-16T01:34:19.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.131+0000 7f4135a3c640 1 -- start start 2024-09-16T01:34:19.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.132+0000 7f4135a3c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41300ba230 0x7f41300b2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:19.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.132+0000 7f4135a3c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41300cf360 0x7f41301e9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:19.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.132+0000 7f4135a3c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f413015e980 0x7f41301ea340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:19.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.132+0000 7f4135a3c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f41300d35a0 con 0x7f413015e980 2024-09-16T01:34:19.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.132+0000 7f4135a3c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f41300d32c0 con 0x7f41300ba230 2024-09-16T01:34:19.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.132+0000 7f4135a3c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f413015ed60 con 0x7f41300cf360 2024-09-16T01:34:19.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.132+0000 7f412f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f413015e980 0x7f41301ea340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:19.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.132+0000 7f412f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f413015e980 0x7f41301ea340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59868/0 (socket says 172.21.15.88:59868) 2024-09-16T01:34:19.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.132+0000 7f412f7fe640 1 -- 172.21.15.88:0/2332946971 learned_addr learned my addr 172.21.15.88:0/2332946971 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:19.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.132+0000 7f412effd640 1 --2- 172.21.15.88:0/2332946971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41300ba230 0x7f41300b2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:19.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.132+0000 7f41267fc640 1 --2- 172.21.15.88:0/2332946971 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41300cf360 0x7f41301e9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:19.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.132+0000 7f412f7fe640 1 -- 172.21.15.88:0/2332946971 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41300cf360 msgr2=0x7f41301e9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:19.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.132+0000 7f412f7fe640 1 --2- 172.21.15.88:0/2332946971 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41300cf360 0x7f41301e9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:19.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.132+0000 7f412f7fe640 1 -- 172.21.15.88:0/2332946971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41300ba230 msgr2=0x7f41300b2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:19.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.132+0000 7f412f7fe640 1 --2- 172.21.15.88:0/2332946971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41300ba230 0x7f41300b2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:19.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.132+0000 7f412f7fe640 1 -- 172.21.15.88:0/2332946971 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41301e8d00 con 0x7f413015e980 2024-09-16T01:34:19.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.132+0000 7f412f7fe640 1 --2- 172.21.15.88:0/2332946971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f413015e980 0x7f41301ea340 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4128009e50 tx=0x7f4128004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:19.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.132+0000 7f412cff9640 1 -- 172.21.15.88:0/2332946971 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4128036050 con 0x7f413015e980 2024-09-16T01:34:19.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.132+0000 7f412cff9640 1 -- 172.21.15.88:0/2332946971 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4128042070 con 0x7f413015e980 2024-09-16T01:34:19.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.132+0000 7f4135a3c640 1 -- 172.21.15.88:0/2332946971 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41300cc800 con 0x7f413015e980 2024-09-16T01:34:19.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.132+0000 7f412cff9640 1 -- 172.21.15.88:0/2332946971 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4128009510 con 0x7f413015e980 2024-09-16T01:34:19.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.132+0000 7f4135a3c640 1 -- 172.21.15.88:0/2332946971 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41300ccdc0 con 0x7f413015e980 2024-09-16T01:34:19.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.133+0000 7f4135a3c640 1 -- 172.21.15.88:0/2332946971 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41300d22b0 con 0x7f413015e980 2024-09-16T01:34:19.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.134+0000 7f412cff9640 1 -- 172.21.15.88:0/2332946971 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4128007bd0 con 0x7f413015e980 2024-09-16T01:34:19.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.134+0000 7f412cff9640 1 --2- 172.21.15.88:0/2332946971 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f411006c2b0 0x7f411006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:19.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.135+0000 7f412effd640 1 --2- 172.21.15.88:0/2332946971 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f411006c2b0 0x7f411006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:19.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.135+0000 7f412cff9640 1 -- 172.21.15.88:0/2332946971 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f41280abd60 con 0x7f413015e980 2024-09-16T01:34:19.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.135+0000 7f412effd640 1 --2- 172.21.15.88:0/2332946971 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f411006c2b0 0x7f411006e770 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4120006fd0 tx=0x7f4120009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:19.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.139+0000 7f412cff9640 1 -- 172.21.15.88:0/2332946971 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f412807a380 con 0x7f413015e980 2024-09-16T01:34:19.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:19.306+0000 7f4135a3c640 1 -- 172.21.15.88:0/2332946971 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f41300b7bc0 con 0x7f413015e980 2024-09-16T01:34:20.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.165+0000 7f412cff9640 1 -- 172.21.15.88:0/2332946971 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7f412807e680 con 0x7f413015e980 2024-09-16T01:34:20.167 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_metadata' created 2024-09-16T01:34:20.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.169+0000 7f4135a3c640 1 -- 172.21.15.88:0/2332946971 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f411006c2b0 msgr2=0x7f411006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:20.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.169+0000 7f4135a3c640 1 --2- 172.21.15.88:0/2332946971 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f411006c2b0 0x7f411006e770 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4120006fd0 tx=0x7f4120009060 comp rx=0 tx=0).stop 2024-09-16T01:34:20.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.169+0000 7f4135a3c640 1 -- 172.21.15.88:0/2332946971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f413015e980 msgr2=0x7f41301ea340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:20.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.169+0000 7f4135a3c640 1 --2- 172.21.15.88:0/2332946971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f413015e980 0x7f41301ea340 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4128009e50 tx=0x7f4128004a40 comp rx=0 tx=0).stop 2024-09-16T01:34:20.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.169+0000 7f4135a3c640 1 -- 172.21.15.88:0/2332946971 shutdown_connections 2024-09-16T01:34:20.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.169+0000 7f4135a3c640 1 --2- 172.21.15.88:0/2332946971 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f411006c2b0 0x7f411006e770 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:20.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.169+0000 7f4135a3c640 1 --2- 172.21.15.88:0/2332946971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f413015e980 0x7f41301ea340 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:20.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.169+0000 7f4135a3c640 1 --2- 172.21.15.88:0/2332946971 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41300cf360 0x7f41301e9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:20.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.169+0000 7f4135a3c640 1 --2- 172.21.15.88:0/2332946971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41300ba230 0x7f41300b2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:20.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.169+0000 7f4135a3c640 1 -- 172.21.15.88:0/2332946971 >> 172.21.15.88:0/2332946971 conn(0x7f41300d46e0 msgr2=0x7f41300b83c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:20.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.170+0000 7f4135a3c640 1 -- 172.21.15.88:0/2332946971 shutdown_connections 2024-09-16T01:34:20.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.170+0000 7f4135a3c640 1 -- 172.21.15.88:0/2332946971 wait complete. 2024-09-16T01:34:20.187 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-16T01:34:20.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.296+0000 7fa2f524c640 1 Processor -- start 2024-09-16T01:34:20.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.296+0000 7fa2f524c640 1 -- start start 2024-09-16T01:34:20.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.297+0000 7fa2f524c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa2f015c520 0x7fa2f00b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:20.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.297+0000 7fa2f524c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2f00cf3d0 con 0x7fa2f006c320 2024-09-16T01:34:20.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.297+0000 7fa2f524c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2f00cf6b0 con 0x7fa2f0070040 2024-09-16T01:34:20.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.297+0000 7fa2f524c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa2f00cffe0 con 0x7fa2f015c520 2024-09-16T01:34:20.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.297+0000 7fa2eed9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa2f015c520 0x7fa2f00b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:20.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.297+0000 7fa2ef59e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa2f006c320 0x7fa2f00d0470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34176/0 (socket says 172.21.15.88:34176) 2024-09-16T01:34:20.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.297+0000 7fa2eed9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa2f015c520 0x7fa2f00b4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58120/0 (socket says 172.21.15.88:58120) 2024-09-16T01:34:20.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.297+0000 7fa2ef59e640 1 -- 172.21.15.88:0/2486879659 learned_addr learned my addr 172.21.15.88:0/2486879659 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:20.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.298+0000 7fa2eed9d640 1 -- 172.21.15.88:0/2486879659 >> v1:172.21.15.154:6789/0 conn(0x7fa2f0070040 legacy=0x7fa2f006bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:20.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.298+0000 7fa2eed9d640 1 -- 172.21.15.88:0/2486879659 >> v1:172.21.15.88:6789/0 conn(0x7fa2f006c320 legacy=0x7fa2f00d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:20.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.298+0000 7fa2edd9b640 1 -- 172.21.15.88:0/2486879659 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1708826615 0 0) 0x7fa2f00cf3d0 con 0x7fa2f006c320 2024-09-16T01:34:20.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.298+0000 7fa2eed9d640 1 -- 172.21.15.88:0/2486879659 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2f00cf990 con 0x7fa2f015c520 2024-09-16T01:34:20.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.298+0000 7fa2eed9d640 1 --2- 172.21.15.88:0/2486879659 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2f015c520 0x7fa2f00b4300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0009d90 tx=0x7fa2e002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b1504ab55b14e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:20.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.299+0000 7fa2edd9b640 1 -- 172.21.15.88:0/2486879659 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e0031600 con 0x7fa2f015c520 2024-09-16T01:34:20.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.299+0000 7fa2edd9b640 1 -- 172.21.15.88:0/2486879659 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2e00317a0 con 0x7fa2f015c520 2024-09-16T01:34:20.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.299+0000 7fa2edd9b640 1 -- 172.21.15.88:0/2486879659 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e0031b70 con 0x7fa2f015c520 2024-09-16T01:34:20.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.299+0000 7fa2f524c640 1 -- 172.21.15.88:0/2486879659 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2f015c520 msgr2=0x7fa2f00b4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:20.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.299+0000 7fa2f524c640 1 --2- 172.21.15.88:0/2486879659 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2f015c520 0x7fa2f00b4300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0009d90 tx=0x7fa2e002a610 comp rx=0 tx=0).stop 2024-09-16T01:34:20.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.299+0000 7fa2f524c640 1 -- 172.21.15.88:0/2486879659 shutdown_connections 2024-09-16T01:34:20.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.299+0000 7fa2f524c640 1 --2- 172.21.15.88:0/2486879659 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2f015c520 0x7fa2f00b4300 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:20.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.299+0000 7fa2f524c640 1 -- 172.21.15.88:0/2486879659 >> 172.21.15.88:0/2486879659 conn(0x7fa2f00d1390 msgr2=0x7fa2f00d17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:20.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.299+0000 7fa2f524c640 1 -- 172.21.15.88:0/2486879659 shutdown_connections 2024-09-16T01:34:20.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.299+0000 7fa2f524c640 1 -- 172.21.15.88:0/2486879659 wait complete. 2024-09-16T01:34:20.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.301+0000 7fa2f524c640 1 Processor -- start 2024-09-16T01:34:20.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.301+0000 7fa2f524c640 1 -- start start 2024-09-16T01:34:20.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.301+0000 7fa2f524c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2f006c320 0x7fa2f015f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:20.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.301+0000 7fa2f524c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa2f0070040 0x7fa2f014e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:20.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.301+0000 7fa2f524c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2f015c520 0x7fa2f015ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:20.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.301+0000 7fa2f524c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f00704a0 con 0x7fa2f015c520 2024-09-16T01:34:20.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.301+0000 7fa2f524c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f00cdb00 con 0x7fa2f006c320 2024-09-16T01:34:20.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.301+0000 7fa2f524c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa2f0070620 con 0x7fa2f0070040 2024-09-16T01:34:20.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.301+0000 7fa2ef59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2f015c520 0x7fa2f015ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:20.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.301+0000 7fa2ef59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2f015c520 0x7fa2f015ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58124/0 (socket says 172.21.15.88:58124) 2024-09-16T01:34:20.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.301+0000 7fa2ef59e640 1 -- 172.21.15.88:0/1475988747 learned_addr learned my addr 172.21.15.88:0/1475988747 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:20.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.302+0000 7fa2ef59e640 1 -- 172.21.15.88:0/1475988747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa2f0070040 msgr2=0x7fa2f014e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:20.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.302+0000 7fa2ef59e640 1 --2- 172.21.15.88:0/1475988747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa2f0070040 0x7fa2f014e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:20.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.302+0000 7fa2ef59e640 1 -- 172.21.15.88:0/1475988747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2f006c320 msgr2=0x7fa2f015f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:20.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.302+0000 7fa2ef59e640 1 --2- 172.21.15.88:0/1475988747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2f006c320 0x7fa2f015f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:20.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.302+0000 7fa2ef59e640 1 -- 172.21.15.88:0/1475988747 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2f01f3210 con 0x7fa2f015c520 2024-09-16T01:34:20.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.302+0000 7fa2ef59e640 1 --2- 172.21.15.88:0/1475988747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2f015c520 0x7fa2f015ad30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa2e800c450 tx=0x7fa2e800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:20.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.302+0000 7fa2e77fe640 1 -- 172.21.15.88:0/1475988747 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e8018070 con 0x7fa2f015c520 2024-09-16T01:34:20.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.302+0000 7fa2e77fe640 1 -- 172.21.15.88:0/1475988747 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2e8007500 con 0x7fa2f015c520 2024-09-16T01:34:20.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.302+0000 7fa2e77fe640 1 -- 172.21.15.88:0/1475988747 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e8004a20 con 0x7fa2f015c520 2024-09-16T01:34:20.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.302+0000 7fa2f524c640 1 -- 172.21.15.88:0/1475988747 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2f01f34b0 con 0x7fa2f015c520 2024-09-16T01:34:20.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.302+0000 7fa2f524c640 1 -- 172.21.15.88:0/1475988747 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2f01f3a70 con 0x7fa2f015c520 2024-09-16T01:34:20.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.303+0000 7fa2f524c640 1 -- 172.21.15.88:0/1475988747 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2f00b4300 con 0x7fa2f015c520 2024-09-16T01:34:20.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.304+0000 7fa2e77fe640 1 -- 172.21.15.88:0/1475988747 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa2e801b070 con 0x7fa2f015c520 2024-09-16T01:34:20.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.304+0000 7fa2e77fe640 1 --2- 172.21.15.88:0/1475988747 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa2d006c2e0 0x7fa2d006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:20.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.304+0000 7fa2e77fe640 1 -- 172.21.15.88:0/1475988747 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7fa2e808a4d0 con 0x7fa2f015c520 2024-09-16T01:34:20.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.304+0000 7fa2eed9d640 1 --2- 172.21.15.88:0/1475988747 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa2d006c2e0 0x7fa2d006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:20.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.304+0000 7fa2eed9d640 1 --2- 172.21.15.88:0/1475988747 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa2d006c2e0 0x7fa2d006e7a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0034000 tx=0x7fa2e002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:20.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.308+0000 7fa2e77fe640 1 -- 172.21.15.88:0/1475988747 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa2e8058e20 con 0x7fa2f015c520 2024-09-16T01:34:20.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:20.475+0000 7fa2f524c640 1 -- 172.21.15.88:0/1475988747 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fa2f00b5a50 con 0x7fa2f015c520 2024-09-16T01:34:21.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.170+0000 7fa2e77fe640 1 -- 172.21.15.88:0/1475988747 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) ==== 170+0+0 (secure 0 0 0) 0x7fa2e8058720 con 0x7fa2f015c520 2024-09-16T01:34:21.171 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_data' created 2024-09-16T01:34:21.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.172+0000 7fa2f524c640 1 -- 172.21.15.88:0/1475988747 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa2d006c2e0 msgr2=0x7fa2d006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:21.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.172+0000 7fa2f524c640 1 --2- 172.21.15.88:0/1475988747 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa2d006c2e0 0x7fa2d006e7a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0034000 tx=0x7fa2e002e000 comp rx=0 tx=0).stop 2024-09-16T01:34:21.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.172+0000 7fa2f524c640 1 -- 172.21.15.88:0/1475988747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2f015c520 msgr2=0x7fa2f015ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:21.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.172+0000 7fa2f524c640 1 --2- 172.21.15.88:0/1475988747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2f015c520 0x7fa2f015ad30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa2e800c450 tx=0x7fa2e800c910 comp rx=0 tx=0).stop 2024-09-16T01:34:21.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.172+0000 7fa2f524c640 1 -- 172.21.15.88:0/1475988747 shutdown_connections 2024-09-16T01:34:21.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.172+0000 7fa2f524c640 1 --2- 172.21.15.88:0/1475988747 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa2d006c2e0 0x7fa2d006e7a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:21.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.172+0000 7fa2f524c640 1 --2- 172.21.15.88:0/1475988747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2f015c520 0x7fa2f015ad30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:21.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.172+0000 7fa2f524c640 1 --2- 172.21.15.88:0/1475988747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa2f0070040 0x7fa2f014e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:21.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.172+0000 7fa2f524c640 1 --2- 172.21.15.88:0/1475988747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2f006c320 0x7fa2f015f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:21.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.172+0000 7fa2f524c640 1 -- 172.21.15.88:0/1475988747 >> 172.21.15.88:0/1475988747 conn(0x7fa2f00d1390 msgr2=0x7fa2f00d0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:21.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.172+0000 7fa2f524c640 1 -- 172.21.15.88:0/1475988747 shutdown_connections 2024-09-16T01:34:21.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.172+0000 7fa2f524c640 1 -- 172.21.15.88:0/1475988747 wait complete. 2024-09-16T01:34:21.194 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-16T01:34:21.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.293+0000 7fcc255e4640 1 Processor -- start 2024-09-16T01:34:21.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.293+0000 7fcc255e4640 1 -- start start 2024-09-16T01:34:21.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.294+0000 7fcc255e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc2015e750 0x7fcc200cf630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:21.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.294+0000 7fcc255e4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc200b7790 con 0x7fcc200cfb70 2024-09-16T01:34:21.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.294+0000 7fcc255e4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc200d3600 con 0x7fcc200b6bb0 2024-09-16T01:34:21.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.294+0000 7fcc255e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcc200d43c0 con 0x7fcc2015e750 2024-09-16T01:34:21.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.294+0000 7fcc1e7fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fcc200cfb70 0x7fcc200b64a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34192/0 (socket says 172.21.15.88:34192) 2024-09-16T01:34:21.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.294+0000 7fcc1e7fc640 1 -- 172.21.15.88:0/446952016 learned_addr learned my addr 172.21.15.88:0/446952016 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:21.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.294+0000 7fcc1dffb640 1 -- 172.21.15.88:0/446952016 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 541965494 0 0) 0x7fcc200b7790 con 0x7fcc200cfb70 2024-09-16T01:34:21.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.294+0000 7fcc1dffb640 1 -- 172.21.15.88:0/446952016 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc08003520 con 0x7fcc200cfb70 2024-09-16T01:34:21.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.294+0000 7fcc1effd640 1 --2- 172.21.15.88:0/446952016 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc2015e750 0x7fcc200cf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:21.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.295+0000 7fcc1dffb640 1 -- 172.21.15.88:0/446952016 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4279210799 0 0) 0x7fcc08003520 con 0x7fcc200cfb70 2024-09-16T01:34:21.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.295+0000 7fcc1dffb640 1 -- 172.21.15.88:0/446952016 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc2015e750 msgr2=0x7fcc200cf630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:21.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.295+0000 7fcc1dffb640 1 --2- 172.21.15.88:0/446952016 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc2015e750 0x7fcc200cf630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:21.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.295+0000 7fcc1dffb640 1 -- 172.21.15.88:0/446952016 >> v1:172.21.15.154:6789/0 conn(0x7fcc200b6bb0 legacy=0x7fcc200b7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:21.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.295+0000 7fcc1dffb640 1 -- 172.21.15.88:0/446952016 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc200d38e0 con 0x7fcc200cfb70 2024-09-16T01:34:21.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.295+0000 7fcc1dffb640 1 -- 172.21.15.88:0/446952016 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fcc14001690 con 0x7fcc200cfb70 2024-09-16T01:34:21.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.295+0000 7fcc1dffb640 1 -- 172.21.15.88:0/446952016 >> v1:172.21.15.88:6789/0 conn(0x7fcc200cfb70 legacy=0x7fcc200b64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:21.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.295+0000 7fcc1dffb640 1 --2- 172.21.15.88:0/446952016 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc080039e0 0x7fcc08003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:21.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.295+0000 7fcc1dffb640 1 --2- 172.21.15.88:0/446952016 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc2015e750 0x7fcc08004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:21.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.295+0000 7fcc1dffb640 1 --2- 172.21.15.88:0/446952016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc08004d40 0x7fcc080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:21.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.295+0000 7fcc1dffb640 1 -- 172.21.15.88:0/446952016 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcc200d43c0 con 0x7fcc080039e0 2024-09-16T01:34:21.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.295+0000 7fcc1dffb640 1 -- 172.21.15.88:0/446952016 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0800baf0 con 0x7fcc08004d40 2024-09-16T01:34:21.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.295+0000 7fcc1dffb640 1 -- 172.21.15.88:0/446952016 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcc0800bcf0 con 0x7fcc2015e750 2024-09-16T01:34:21.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.295+0000 7fcc1effd640 1 --2- 172.21.15.88:0/446952016 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc080039e0 0x7fcc08003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:21.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.295+0000 7fcc1e7fc640 1 --2- 172.21.15.88:0/446952016 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc2015e750 0x7fcc08004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:21.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.295+0000 7fcc1f7fe640 1 --2- 172.21.15.88:0/446952016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc08004d40 0x7fcc080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:21.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.295+0000 7fcc1effd640 1 -- 172.21.15.88:0/446952016 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc2015e750 msgr2=0x7fcc08004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:21.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.295+0000 7fcc1effd640 1 --2- 172.21.15.88:0/446952016 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc2015e750 0x7fcc08004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:21.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.295+0000 7fcc1effd640 1 -- 172.21.15.88:0/446952016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc08004d40 msgr2=0x7fcc080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:21.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.295+0000 7fcc1effd640 1 --2- 172.21.15.88:0/446952016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc08004d40 0x7fcc080051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:21.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.295+0000 7fcc1effd640 1 -- 172.21.15.88:0/446952016 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc08003520 con 0x7fcc080039e0 2024-09-16T01:34:21.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.295+0000 7fcc1f7fe640 1 --2- 172.21.15.88:0/446952016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc08004d40 0x7fcc080051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:21.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.296+0000 7fcc1effd640 1 --2- 172.21.15.88:0/446952016 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc080039e0 0x7fcc08003e00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009ac0 tx=0x7fcc08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2427103a709b97b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:21.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.296+0000 7fcc1dffb640 1 -- 172.21.15.88:0/446952016 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10036070 con 0x7fcc080039e0 2024-09-16T01:34:21.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.296+0000 7fcc1dffb640 1 -- 172.21.15.88:0/446952016 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc10031400 con 0x7fcc080039e0 2024-09-16T01:34:21.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.296+0000 7fcc1dffb640 1 -- 172.21.15.88:0/446952016 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10009690 con 0x7fcc080039e0 2024-09-16T01:34:21.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.296+0000 7fcc255e4640 1 -- 172.21.15.88:0/446952016 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc080039e0 msgr2=0x7fcc08003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:21.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.296+0000 7fcc255e4640 1 --2- 172.21.15.88:0/446952016 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc080039e0 0x7fcc08003e00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009ac0 tx=0x7fcc08005830 comp rx=0 tx=0).stop 2024-09-16T01:34:21.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.296+0000 7fcc1f7fe640 1 -- 172.21.15.88:0/446952016 reap_dead start 2024-09-16T01:34:21.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.296+0000 7fcc255e4640 1 -- 172.21.15.88:0/446952016 shutdown_connections 2024-09-16T01:34:21.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.296+0000 7fcc255e4640 1 -- 172.21.15.88:0/446952016 >> 172.21.15.88:0/446952016 conn(0x7fcc200d1500 msgr2=0x7fcc200d1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:21.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.296+0000 7fcc255e4640 1 -- 172.21.15.88:0/446952016 shutdown_connections 2024-09-16T01:34:21.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.296+0000 7fcc255e4640 1 -- 172.21.15.88:0/446952016 wait complete. 2024-09-16T01:34:21.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.298+0000 7fcc255e4640 1 Processor -- start 2024-09-16T01:34:21.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.298+0000 7fcc255e4640 1 -- start start 2024-09-16T01:34:21.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.298+0000 7fcc255e4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc080039e0 0x7fcc201f3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:21.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.298+0000 7fcc255e4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc20161920 0x7fcc201f7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:21.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.298+0000 7fcc255e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc201f7d70 0x7fcc201f81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:21.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.298+0000 7fcc255e4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcc200b7ac0 con 0x7fcc080039e0 2024-09-16T01:34:21.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.298+0000 7fcc255e4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcc200d33b0 con 0x7fcc20161920 2024-09-16T01:34:21.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.298+0000 7fcc255e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcc2015eb30 con 0x7fcc201f7d70 2024-09-16T01:34:21.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.299+0000 7fcc1effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc080039e0 0x7fcc201f3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:21.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.299+0000 7fcc1effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc080039e0 0x7fcc201f3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58140/0 (socket says 172.21.15.88:58140) 2024-09-16T01:34:21.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.299+0000 7fcc1effd640 1 -- 172.21.15.88:0/3267971292 learned_addr learned my addr 172.21.15.88:0/3267971292 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:21.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.299+0000 7fcc1f7fe640 1 --2- 172.21.15.88:0/3267971292 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc201f7d70 0x7fcc201f81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:21.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.299+0000 7fcc1e7fc640 1 --2- 172.21.15.88:0/3267971292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc20161920 0x7fcc201f7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:21.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.299+0000 7fcc1effd640 1 -- 172.21.15.88:0/3267971292 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc201f7d70 msgr2=0x7fcc201f81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:21.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.299+0000 7fcc1effd640 1 --2- 172.21.15.88:0/3267971292 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc201f7d70 0x7fcc201f81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:21.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.299+0000 7fcc1effd640 1 -- 172.21.15.88:0/3267971292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc20161920 msgr2=0x7fcc201f7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:21.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.299+0000 7fcc1effd640 1 --2- 172.21.15.88:0/3267971292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc20161920 0x7fcc201f7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:21.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.299+0000 7fcc1effd640 1 -- 172.21.15.88:0/3267971292 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc200abb90 con 0x7fcc080039e0 2024-09-16T01:34:21.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.299+0000 7fcc1f7fe640 1 --2- 172.21.15.88:0/3267971292 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc201f7d70 0x7fcc201f81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:21.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.299+0000 7fcc1effd640 1 --2- 172.21.15.88:0/3267971292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc080039e0 0x7fcc201f3ae0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009ac0 tx=0x7fcc10004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:21.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.299+0000 7fcc077fe640 1 -- 172.21.15.88:0/3267971292 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10036050 con 0x7fcc080039e0 2024-09-16T01:34:21.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.299+0000 7fcc077fe640 1 -- 172.21.15.88:0/3267971292 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc10042070 con 0x7fcc080039e0 2024-09-16T01:34:21.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.299+0000 7fcc077fe640 1 -- 172.21.15.88:0/3267971292 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc100044e0 con 0x7fcc080039e0 2024-09-16T01:34:21.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.299+0000 7fcc255e4640 1 -- 172.21.15.88:0/3267971292 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc200abe70 con 0x7fcc080039e0 2024-09-16T01:34:21.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.299+0000 7fcc255e4640 1 -- 172.21.15.88:0/3267971292 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc200ac430 con 0x7fcc080039e0 2024-09-16T01:34:21.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.300+0000 7fcc255e4640 1 -- 172.21.15.88:0/3267971292 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc200cf630 con 0x7fcc080039e0 2024-09-16T01:34:21.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.301+0000 7fcc077fe640 1 -- 172.21.15.88:0/3267971292 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fcc10047460 con 0x7fcc080039e0 2024-09-16T01:34:21.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.301+0000 7fcc077fe640 1 --2- 172.21.15.88:0/3267971292 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcc0c06c2e0 0x7fcc0c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:21.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.301+0000 7fcc1e7fc640 1 --2- 172.21.15.88:0/3267971292 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcc0c06c2e0 0x7fcc0c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:21.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.301+0000 7fcc077fe640 1 -- 172.21.15.88:0/3267971292 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7fcc100abb80 con 0x7fcc080039e0 2024-09-16T01:34:21.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.301+0000 7fcc1e7fc640 1 --2- 172.21.15.88:0/3267971292 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcc0c06c2e0 0x7fcc0c06e7a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcc2015d370 tx=0x7fcc14009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:21.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.305+0000 7fcc077fe640 1 -- 172.21.15.88:0/3267971292 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fcc1004c020 con 0x7fcc080039e0 2024-09-16T01:34:21.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:21.507+0000 7fcc255e4640 1 -- 172.21.15.88:0/3267971292 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fcc200b5af0 con 0x7fcc080039e0 2024-09-16T01:34:22.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.171+0000 7fcc077fe640 1 -- 172.21.15.88:0/3267971292 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7fcc1007a350 con 0x7fcc080039e0 2024-09-16T01:34:22.173 INFO:teuthology.orchestra.run.smithi088.stderr:new fs with metadata pool 2 and data pool 3 2024-09-16T01:34:22.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.175+0000 7fcc255e4640 1 -- 172.21.15.88:0/3267971292 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcc0c06c2e0 msgr2=0x7fcc0c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:22.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.175+0000 7fcc255e4640 1 --2- 172.21.15.88:0/3267971292 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcc0c06c2e0 0x7fcc0c06e7a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcc2015d370 tx=0x7fcc14009040 comp rx=0 tx=0).stop 2024-09-16T01:34:22.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.175+0000 7fcc255e4640 1 -- 172.21.15.88:0/3267971292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc080039e0 msgr2=0x7fcc201f3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:22.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.176+0000 7fcc255e4640 1 --2- 172.21.15.88:0/3267971292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc080039e0 0x7fcc201f3ae0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009ac0 tx=0x7fcc10004730 comp rx=0 tx=0).stop 2024-09-16T01:34:22.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.177+0000 7fcc255e4640 1 -- 172.21.15.88:0/3267971292 shutdown_connections 2024-09-16T01:34:22.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.177+0000 7fcc255e4640 1 --2- 172.21.15.88:0/3267971292 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcc0c06c2e0 0x7fcc0c06e7a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.177+0000 7fcc255e4640 1 --2- 172.21.15.88:0/3267971292 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc201f7d70 0x7fcc201f81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.177+0000 7fcc255e4640 1 --2- 172.21.15.88:0/3267971292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc20161920 0x7fcc201f7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.177+0000 7fcc255e4640 1 --2- 172.21.15.88:0/3267971292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc080039e0 0x7fcc201f3ae0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.177+0000 7fcc255e4640 1 -- 172.21.15.88:0/3267971292 >> 172.21.15.88:0/3267971292 conn(0x7fcc200d1500 msgr2=0x7fcc2015d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:22.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.177+0000 7fcc255e4640 1 -- 172.21.15.88:0/3267971292 shutdown_connections 2024-09-16T01:34:22.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.181+0000 7fcc255e4640 1 -- 172.21.15.88:0/3267971292 wait complete. 2024-09-16T01:34:22.210 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:34:22.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.309+0000 7fe45efb9640 1 Processor -- start 2024-09-16T01:34:22.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.309+0000 7fe45efb9640 1 -- start start 2024-09-16T01:34:22.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.310+0000 7fe45efb9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe45815e7a0 0x7fe4580ba240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:22.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.310+0000 7fe45efb9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4580d3fe0 0x7fe4580d2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:22.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.310+0000 7fe45efb9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4580d2cc0 con 0x7fe4580cf2f0 2024-09-16T01:34:22.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.310+0000 7fe45efb9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe4580d4400 con 0x7fe45815e7a0 2024-09-16T01:34:22.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.310+0000 7fe45efb9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe4580d2f40 con 0x7fe4580d3fe0 2024-09-16T01:34:22.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.310+0000 7fe45cd55640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe45815e7a0 0x7fe4580ba240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:22.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.310+0000 7fe45d556640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe4580cf2f0 0x7fe4580b7720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34204/0 (socket says 172.21.15.88:34204) 2024-09-16T01:34:22.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.310+0000 7fe45d556640 1 -- 172.21.15.88:0/2393692747 learned_addr learned my addr 172.21.15.88:0/2393692747 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:22.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.310+0000 7fe457fff640 1 --2- 172.21.15.88:0/2393692747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4580d3fe0 0x7fe4580d2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:22.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.311+0000 7fe45cd55640 1 -- 172.21.15.88:0/2393692747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4580d3fe0 msgr2=0x7fe4580d2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:22.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.311+0000 7fe45cd55640 1 --2- 172.21.15.88:0/2393692747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4580d3fe0 0x7fe4580d2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.311+0000 7fe45cd55640 1 -- 172.21.15.88:0/2393692747 >> v1:172.21.15.88:6789/0 conn(0x7fe4580cf2f0 legacy=0x7fe4580b7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:22.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.311+0000 7fe45cd55640 1 -- 172.21.15.88:0/2393692747 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4580d3220 con 0x7fe45815e7a0 2024-09-16T01:34:22.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.311+0000 7fe457fff640 1 --2- 172.21.15.88:0/2393692747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4580d3fe0 0x7fe4580d2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:22.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.311+0000 7fe45cd55640 1 --2- 172.21.15.88:0/2393692747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe45815e7a0 0x7fe4580ba240 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe4480097a0 tx=0x7fe44802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b78f60cd6ecb56 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:22.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.311+0000 7fe4577fe640 1 -- 172.21.15.88:0/2393692747 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe448037070 con 0x7fe45815e7a0 2024-09-16T01:34:22.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.311+0000 7fe4577fe640 1 -- 172.21.15.88:0/2393692747 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe448032440 con 0x7fe45815e7a0 2024-09-16T01:34:22.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.311+0000 7fe4577fe640 1 -- 172.21.15.88:0/2393692747 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe448032810 con 0x7fe45815e7a0 2024-09-16T01:34:22.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.312+0000 7fe45efb9640 1 -- 172.21.15.88:0/2393692747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe45815e7a0 msgr2=0x7fe4580ba240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:22.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.312+0000 7fe45efb9640 1 --2- 172.21.15.88:0/2393692747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe45815e7a0 0x7fe4580ba240 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe4480097a0 tx=0x7fe44802a210 comp rx=0 tx=0).stop 2024-09-16T01:34:22.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.312+0000 7fe45efb9640 1 -- 172.21.15.88:0/2393692747 shutdown_connections 2024-09-16T01:34:22.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.312+0000 7fe45efb9640 1 --2- 172.21.15.88:0/2393692747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4580d3fe0 0x7fe4580d2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.312+0000 7fe45efb9640 1 --2- 172.21.15.88:0/2393692747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe45815e7a0 0x7fe4580ba240 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.312+0000 7fe45efb9640 1 -- 172.21.15.88:0/2393692747 >> 172.21.15.88:0/2393692747 conn(0x7fe4580d1550 msgr2=0x7fe4580d1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:22.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.312+0000 7fe45efb9640 1 -- 172.21.15.88:0/2393692747 shutdown_connections 2024-09-16T01:34:22.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.312+0000 7fe45efb9640 1 -- 172.21.15.88:0/2393692747 wait complete. 2024-09-16T01:34:22.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.313+0000 7fe45efb9640 1 Processor -- start 2024-09-16T01:34:22.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.313+0000 7fe45efb9640 1 -- start start 2024-09-16T01:34:22.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.314+0000 7fe45efb9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4580cf2f0 0x7fe458160df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:22.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.314+0000 7fe45efb9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4580d3fe0 0x7fe4580ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:22.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.314+0000 7fe45efb9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe45815e7a0 0x7fe4580cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:22.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.314+0000 7fe45efb9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe4580d3ac0 con 0x7fe45815e7a0 2024-09-16T01:34:22.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.314+0000 7fe45efb9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe4580d37e0 con 0x7fe4580d3fe0 2024-09-16T01:34:22.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.314+0000 7fe45efb9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe45815eb80 con 0x7fe4580cf2f0 2024-09-16T01:34:22.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.314+0000 7fe45d556640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe45815e7a0 0x7fe4580cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:22.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.314+0000 7fe45d556640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe45815e7a0 0x7fe4580cdb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58160/0 (socket says 172.21.15.88:58160) 2024-09-16T01:34:22.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.314+0000 7fe45d556640 1 -- 172.21.15.88:0/2289288850 learned_addr learned my addr 172.21.15.88:0/2289288850 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:22.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.314+0000 7fe457fff640 1 --2- 172.21.15.88:0/2289288850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4580d3fe0 0x7fe4580ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:22.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.314+0000 7fe45cd55640 1 --2- 172.21.15.88:0/2289288850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4580cf2f0 0x7fe458160df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:22.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.314+0000 7fe45d556640 1 -- 172.21.15.88:0/2289288850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4580cf2f0 msgr2=0x7fe458160df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:22.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.314+0000 7fe45d556640 1 --2- 172.21.15.88:0/2289288850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4580cf2f0 0x7fe458160df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.314+0000 7fe45d556640 1 -- 172.21.15.88:0/2289288850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4580d3fe0 msgr2=0x7fe4580ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:22.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.314+0000 7fe45d556640 1 --2- 172.21.15.88:0/2289288850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4580d3fe0 0x7fe4580ca1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.314+0000 7fe45d556640 1 -- 172.21.15.88:0/2289288850 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe45815fd50 con 0x7fe45815e7a0 2024-09-16T01:34:22.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.315+0000 7fe45d556640 1 --2- 172.21.15.88:0/2289288850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe45815e7a0 0x7fe4580cdb20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe45000cbc0 tx=0x7fe45000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:22.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.315+0000 7fe4557fa640 1 -- 172.21.15.88:0/2289288850 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe450017070 con 0x7fe45815e7a0 2024-09-16T01:34:22.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.315+0000 7fe4557fa640 1 -- 172.21.15.88:0/2289288850 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe45000ad40 con 0x7fe45815e7a0 2024-09-16T01:34:22.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.315+0000 7fe45efb9640 1 -- 172.21.15.88:0/2289288850 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe458160030 con 0x7fe45815e7a0 2024-09-16T01:34:22.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.315+0000 7fe4557fa640 1 -- 172.21.15.88:0/2289288850 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe450012600 con 0x7fe45815e7a0 2024-09-16T01:34:22.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.315+0000 7fe45efb9640 1 -- 172.21.15.88:0/2289288850 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4581605f0 con 0x7fe45815e7a0 2024-09-16T01:34:22.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.316+0000 7fe45efb9640 1 -- 172.21.15.88:0/2289288850 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4580d2780 con 0x7fe45815e7a0 2024-09-16T01:34:22.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.316+0000 7fe4557fa640 1 -- 172.21.15.88:0/2289288850 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe45001a070 con 0x7fe45815e7a0 2024-09-16T01:34:22.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.317+0000 7fe4557fa640 1 --2- 172.21.15.88:0/2289288850 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe44406c2b0 0x7fe44406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:22.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.317+0000 7fe4557fa640 1 -- 172.21.15.88:0/2289288850 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe45008b050 con 0x7fe45815e7a0 2024-09-16T01:34:22.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.317+0000 7fe45cd55640 1 --2- 172.21.15.88:0/2289288850 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe44406c2b0 0x7fe44406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:22.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.317+0000 7fe45cd55640 1 --2- 172.21.15.88:0/2289288850 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe44406c2b0 0x7fe44406e770 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe448034000 tx=0x7fe44802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:22.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.321+0000 7fe4557fa640 1 -- 172.21.15.88:0/2289288850 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe4580d2cc0 con 0x7fe45815e7a0 2024-09-16T01:34:22.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.486+0000 7fe45efb9640 1 -- 172.21.15.88:0/2289288850 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe4580b4d00 con 0x7fe45815e7a0 2024-09-16T01:34:22.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.487+0000 7fe4557fa640 1 -- 172.21.15.88:0/2289288850 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16654 (secure 0 0 0) 0x7fe450059200 con 0x7fe45815e7a0 2024-09-16T01:34:22.489 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:22.489 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":20,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:34:21.508175+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:34:19.324549+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:34:20.494051+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:34:22.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.490+0000 7fe45efb9640 1 -- 172.21.15.88:0/2289288850 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe44406c2b0 msgr2=0x7fe44406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:22.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.490+0000 7fe45efb9640 1 --2- 172.21.15.88:0/2289288850 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe44406c2b0 0x7fe44406e770 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe448034000 tx=0x7fe44802e000 comp rx=0 tx=0).stop 2024-09-16T01:34:22.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.490+0000 7fe45efb9640 1 -- 172.21.15.88:0/2289288850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe45815e7a0 msgr2=0x7fe4580cdb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:22.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.490+0000 7fe45efb9640 1 --2- 172.21.15.88:0/2289288850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe45815e7a0 0x7fe4580cdb20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe45000cbc0 tx=0x7fe45000a430 comp rx=0 tx=0).stop 2024-09-16T01:34:22.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.490+0000 7fe45efb9640 1 -- 172.21.15.88:0/2289288850 shutdown_connections 2024-09-16T01:34:22.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.490+0000 7fe45efb9640 1 --2- 172.21.15.88:0/2289288850 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe44406c2b0 0x7fe44406e770 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.490+0000 7fe45efb9640 1 --2- 172.21.15.88:0/2289288850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe45815e7a0 0x7fe4580cdb20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.490+0000 7fe45efb9640 1 --2- 172.21.15.88:0/2289288850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4580d3fe0 0x7fe4580ca1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.490+0000 7fe45efb9640 1 --2- 172.21.15.88:0/2289288850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4580cf2f0 0x7fe458160df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.490+0000 7fe45efb9640 1 -- 172.21.15.88:0/2289288850 >> 172.21.15.88:0/2289288850 conn(0x7fe4580d1550 msgr2=0x7fe458074c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:22.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.490+0000 7fe45efb9640 1 -- 172.21.15.88:0/2289288850 shutdown_connections 2024-09-16T01:34:22.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.490+0000 7fe45efb9640 1 -- 172.21.15.88:0/2289288850 wait complete. 2024-09-16T01:34:22.502 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:34:22.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.598+0000 7fc24ca9b640 1 Processor -- start 2024-09-16T01:34:22.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.598+0000 7fc24ca9b640 1 -- start start 2024-09-16T01:34:22.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.599+0000 7fc24ca9b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc24815c2b0 0x7fc2480b6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:22.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.599+0000 7fc24ca9b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc24815fb10 con 0x7fc2480b72f0 2024-09-16T01:34:22.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.599+0000 7fc24ca9b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2480b5900 con 0x7fc24815f200 2024-09-16T01:34:22.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.599+0000 7fc24ca9b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc24815f830 con 0x7fc24815c2b0 2024-09-16T01:34:22.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.599+0000 7fc245d9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc2480b72f0 0x7fc24815eaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34212/0 (socket says 172.21.15.88:34212) 2024-09-16T01:34:22.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.599+0000 7fc245d9b640 1 -- 172.21.15.88:0/2543467872 learned_addr learned my addr 172.21.15.88:0/2543467872 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:22.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.599+0000 7fc24559a640 1 -- 172.21.15.88:0/2543467872 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2363260305 0 0) 0x7fc24815fb10 con 0x7fc2480b72f0 2024-09-16T01:34:22.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.599+0000 7fc24659c640 1 --2- 172.21.15.88:0/2543467872 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc24815c2b0 0x7fc2480b6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:22.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.599+0000 7fc24559a640 1 -- 172.21.15.88:0/2543467872 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc230003520 con 0x7fc2480b72f0 2024-09-16T01:34:22.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.600+0000 7fc24559a640 1 -- 172.21.15.88:0/2543467872 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2230988497 0 0) 0x7fc230003520 con 0x7fc2480b72f0 2024-09-16T01:34:22.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.600+0000 7fc24559a640 1 -- 172.21.15.88:0/2543467872 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc24815c2b0 msgr2=0x7fc2480b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:22.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.600+0000 7fc24559a640 1 --2- 172.21.15.88:0/2543467872 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc24815c2b0 0x7fc2480b6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.600+0000 7fc24559a640 1 -- 172.21.15.88:0/2543467872 >> v1:172.21.15.154:6789/0 conn(0x7fc24815f200 legacy=0x7fc24806d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:22.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.600+0000 7fc24559a640 1 -- 172.21.15.88:0/2543467872 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc24806ce30 con 0x7fc2480b72f0 2024-09-16T01:34:22.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.600+0000 7fc24559a640 1 -- 172.21.15.88:0/2543467872 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fc23c001670 con 0x7fc2480b72f0 2024-09-16T01:34:22.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.600+0000 7fc24559a640 1 -- 172.21.15.88:0/2543467872 >> v1:172.21.15.88:6789/0 conn(0x7fc2480b72f0 legacy=0x7fc24815eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:22.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.600+0000 7fc24559a640 1 --2- 172.21.15.88:0/2543467872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2300039e0 0x7fc230003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:22.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.600+0000 7fc24559a640 1 --2- 172.21.15.88:0/2543467872 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc230004800 0x7fc230004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:22.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.600+0000 7fc24559a640 1 --2- 172.21.15.88:0/2543467872 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc24815c2b0 0x7fc2300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:22.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.600+0000 7fc24559a640 1 -- 172.21.15.88:0/2543467872 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc24815f830 con 0x7fc230004800 2024-09-16T01:34:22.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.600+0000 7fc24559a640 1 -- 172.21.15.88:0/2543467872 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc23000ba70 con 0x7fc2300039e0 2024-09-16T01:34:22.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.600+0000 7fc24559a640 1 -- 172.21.15.88:0/2543467872 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc23000bc70 con 0x7fc24815c2b0 2024-09-16T01:34:22.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.600+0000 7fc245d9b640 1 --2- 172.21.15.88:0/2543467872 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc230004800 0x7fc230004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:22.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.600+0000 7fc24659c640 1 --2- 172.21.15.88:0/2543467872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2300039e0 0x7fc230003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:22.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.600+0000 7fc246d9d640 1 --2- 172.21.15.88:0/2543467872 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc24815c2b0 0x7fc2300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:22.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.600+0000 7fc245d9b640 1 -- 172.21.15.88:0/2543467872 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc24815c2b0 msgr2=0x7fc2300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:22.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.600+0000 7fc245d9b640 1 --2- 172.21.15.88:0/2543467872 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc24815c2b0 0x7fc2300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.600+0000 7fc245d9b640 1 -- 172.21.15.88:0/2543467872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2300039e0 msgr2=0x7fc230003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:22.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.600+0000 7fc245d9b640 1 --2- 172.21.15.88:0/2543467872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2300039e0 0x7fc230003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.600+0000 7fc245d9b640 1 -- 172.21.15.88:0/2543467872 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc230003520 con 0x7fc230004800 2024-09-16T01:34:22.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.601+0000 7fc245d9b640 1 --2- 172.21.15.88:0/2543467872 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc230004800 0x7fc230004c80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc248071730 tx=0x7fc230005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83ca5be7a5885a25 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:22.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.601+0000 7fc24559a640 1 -- 172.21.15.88:0/2543467872 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c007d10 con 0x7fc230004800 2024-09-16T01:34:22.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.601+0000 7fc24559a640 1 -- 172.21.15.88:0/2543467872 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc23c005ad0 con 0x7fc230004800 2024-09-16T01:34:22.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.601+0000 7fc24559a640 1 -- 172.21.15.88:0/2543467872 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c007070 con 0x7fc230004800 2024-09-16T01:34:22.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.601+0000 7fc24ca9b640 1 -- 172.21.15.88:0/2543467872 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc230004800 msgr2=0x7fc230004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:22.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.601+0000 7fc24ca9b640 1 --2- 172.21.15.88:0/2543467872 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc230004800 0x7fc230004c80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc248071730 tx=0x7fc230005830 comp rx=0 tx=0).stop 2024-09-16T01:34:22.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.602+0000 7fc246d9d640 1 -- 172.21.15.88:0/2543467872 reap_dead start 2024-09-16T01:34:22.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.602+0000 7fc24ca9b640 1 -- 172.21.15.88:0/2543467872 shutdown_connections 2024-09-16T01:34:22.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.602+0000 7fc24ca9b640 1 -- 172.21.15.88:0/2543467872 >> 172.21.15.88:0/2543467872 conn(0x7fc2480b4240 msgr2=0x7fc2480b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:22.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.602+0000 7fc24ca9b640 1 -- 172.21.15.88:0/2543467872 shutdown_connections 2024-09-16T01:34:22.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.602+0000 7fc24ca9b640 1 -- 172.21.15.88:0/2543467872 wait complete. 2024-09-16T01:34:22.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.603+0000 7fc24ca9b640 1 Processor -- start 2024-09-16T01:34:22.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.603+0000 7fc24ca9b640 1 -- start start 2024-09-16T01:34:22.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.604+0000 7fc24ca9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc230004800 0x7fc2481550b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:22.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.604+0000 7fc24ca9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2481e7170 0x7fc248157cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:22.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.604+0000 7fc24ca9b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2481e75f0 0x7fc248158230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:22.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.604+0000 7fc24ca9b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc2480b7730 con 0x7fc230004800 2024-09-16T01:34:22.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.604+0000 7fc24ca9b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc24806d3f0 con 0x7fc2481e7170 2024-09-16T01:34:22.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.604+0000 7fc24ca9b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc2480b6db0 con 0x7fc2481e75f0 2024-09-16T01:34:22.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.604+0000 7fc24659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc230004800 0x7fc2481550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:22.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.604+0000 7fc24659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc230004800 0x7fc2481550b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58176/0 (socket says 172.21.15.88:58176) 2024-09-16T01:34:22.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.604+0000 7fc24659c640 1 -- 172.21.15.88:0/3395591908 learned_addr learned my addr 172.21.15.88:0/3395591908 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:22.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.604+0000 7fc246d9d640 1 --2- 172.21.15.88:0/3395591908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2481e75f0 0x7fc248158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:22.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.604+0000 7fc245d9b640 1 --2- 172.21.15.88:0/3395591908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2481e7170 0x7fc248157cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:22.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.604+0000 7fc24659c640 1 -- 172.21.15.88:0/3395591908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2481e75f0 msgr2=0x7fc248158230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:22.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.604+0000 7fc24659c640 1 --2- 172.21.15.88:0/3395591908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2481e75f0 0x7fc248158230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.604+0000 7fc24659c640 1 -- 172.21.15.88:0/3395591908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2481e7170 msgr2=0x7fc248157cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:22.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.604+0000 7fc24659c640 1 --2- 172.21.15.88:0/3395591908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2481e7170 0x7fc248157cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.604+0000 7fc24659c640 1 -- 172.21.15.88:0/3395591908 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2481e7b50 con 0x7fc230004800 2024-09-16T01:34:22.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.604+0000 7fc246d9d640 1 --2- 172.21.15.88:0/3395591908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2481e75f0 0x7fc248158230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:22.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.604+0000 7fc24659c640 1 --2- 172.21.15.88:0/3395591908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc230004800 0x7fc2481550b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc238011e60 tx=0x7fc23800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:22.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.604+0000 7fc236ffd640 1 -- 172.21.15.88:0/3395591908 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc238015630 con 0x7fc230004800 2024-09-16T01:34:22.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.604+0000 7fc236ffd640 1 -- 172.21.15.88:0/3395591908 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2380157d0 con 0x7fc230004800 2024-09-16T01:34:22.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.604+0000 7fc24ca9b640 1 -- 172.21.15.88:0/3395591908 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2481e7e30 con 0x7fc230004800 2024-09-16T01:34:22.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.604+0000 7fc236ffd640 1 -- 172.21.15.88:0/3395591908 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc238015ba0 con 0x7fc230004800 2024-09-16T01:34:22.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.604+0000 7fc24ca9b640 1 -- 172.21.15.88:0/3395591908 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2481e83f0 con 0x7fc230004800 2024-09-16T01:34:22.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.605+0000 7fc24ca9b640 1 -- 172.21.15.88:0/3395591908 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc24815eaf0 con 0x7fc230004800 2024-09-16T01:34:22.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.606+0000 7fc236ffd640 1 -- 172.21.15.88:0/3395591908 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc23801d400 con 0x7fc230004800 2024-09-16T01:34:22.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.606+0000 7fc236ffd640 1 --2- 172.21.15.88:0/3395591908 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc22806c2e0 0x7fc22806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:22.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.606+0000 7fc236ffd640 1 -- 172.21.15.88:0/3395591908 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc23808c5b0 con 0x7fc230004800 2024-09-16T01:34:22.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.606+0000 7fc245d9b640 1 --2- 172.21.15.88:0/3395591908 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc22806c2e0 0x7fc22806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:22.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.607+0000 7fc245d9b640 1 --2- 172.21.15.88:0/3395591908 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc22806c2e0 0x7fc22806e7a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc2300057e0 tx=0x7fc23c02d000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:22.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.610+0000 7fc236ffd640 1 -- 172.21.15.88:0/3395591908 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc238019030 con 0x7fc230004800 2024-09-16T01:34:22.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.775+0000 7fc24ca9b640 1 -- 172.21.15.88:0/3395591908 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc24806dad0 con 0x7fc230004800 2024-09-16T01:34:22.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.777+0000 7fc236ffd640 1 -- 172.21.15.88:0/3395591908 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16654 (secure 0 0 0) 0x7fc238010040 con 0x7fc230004800 2024-09-16T01:34:22.778 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:22.778 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":20,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:34:21.508175+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:34:19.324549+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:34:20.494051+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:34:22.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.779+0000 7fc24ca9b640 1 -- 172.21.15.88:0/3395591908 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc22806c2e0 msgr2=0x7fc22806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:22.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.779+0000 7fc24ca9b640 1 --2- 172.21.15.88:0/3395591908 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc22806c2e0 0x7fc22806e7a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc2300057e0 tx=0x7fc23c02d000 comp rx=0 tx=0).stop 2024-09-16T01:34:22.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.779+0000 7fc24ca9b640 1 -- 172.21.15.88:0/3395591908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc230004800 msgr2=0x7fc2481550b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:22.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.779+0000 7fc24ca9b640 1 --2- 172.21.15.88:0/3395591908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc230004800 0x7fc2481550b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc238011e60 tx=0x7fc23800f580 comp rx=0 tx=0).stop 2024-09-16T01:34:22.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.779+0000 7fc24ca9b640 1 -- 172.21.15.88:0/3395591908 shutdown_connections 2024-09-16T01:34:22.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.779+0000 7fc24ca9b640 1 --2- 172.21.15.88:0/3395591908 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc22806c2e0 0x7fc22806e7a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.779+0000 7fc24ca9b640 1 --2- 172.21.15.88:0/3395591908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2481e75f0 0x7fc248158230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.779+0000 7fc24ca9b640 1 --2- 172.21.15.88:0/3395591908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2481e7170 0x7fc248157cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.779+0000 7fc24ca9b640 1 --2- 172.21.15.88:0/3395591908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc230004800 0x7fc2481550b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.779+0000 7fc24ca9b640 1 -- 172.21.15.88:0/3395591908 >> 172.21.15.88:0/3395591908 conn(0x7fc2480b4240 msgr2=0x7fc24815fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:22.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.779+0000 7fc24ca9b640 1 -- 172.21.15.88:0/3395591908 shutdown_connections 2024-09-16T01:34:22.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.779+0000 7fc24ca9b640 1 -- 172.21.15.88:0/3395591908 wait complete. 2024-09-16T01:34:22.791 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-16T01:34:22.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.890+0000 7fc8b8fc4640 1 Processor -- start 2024-09-16T01:34:22.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.890+0000 7fc8b8fc4640 1 -- start start 2024-09-16T01:34:22.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.891+0000 7fc8b8fc4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc8b40b6850 0x7fc8b40b7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:22.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.891+0000 7fc8b8fc4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc8b40ba440 0x7fc8b40b80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:22.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.891+0000 7fc8b8fc4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8b415ed20 con 0x7fc8b415e940 2024-09-16T01:34:22.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.891+0000 7fc8b8fc4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc8b40d3ef0 con 0x7fc8b40b6850 2024-09-16T01:34:22.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.891+0000 7fc8b8fc4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc8b40d32e0 con 0x7fc8b40ba440 2024-09-16T01:34:22.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.891+0000 7fc8b1d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc8b40b6850 0x7fc8b40b7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:22.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.891+0000 7fc8b1d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc8b40b6850 0x7fc8b40b7ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58180/0 (socket says 172.21.15.88:58180) 2024-09-16T01:34:22.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.891+0000 7fc8b1d9b640 1 -- 172.21.15.88:0/4062975154 learned_addr learned my addr 172.21.15.88:0/4062975154 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:22.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.891+0000 7fc8b259c640 1 --1- 172.21.15.88:0/4062975154 >> v1:172.21.15.88:6789/0 conn(0x7fc8b415e940 0x7fc8b40b6140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34214/0 (socket says 172.21.15.88:34214) 2024-09-16T01:34:22.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.892+0000 7fc8b1d9b640 1 -- 172.21.15.88:0/4062975154 >> v2:172.21.15.154:3300/0 conn(0x7fc8b40ba440 msgr2=0x7fc8b40b80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:22.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.892+0000 7fc8b1d9b640 1 --2- 172.21.15.88:0/4062975154 >> v2:172.21.15.154:3300/0 conn(0x7fc8b40ba440 0x7fc8b40b80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.892+0000 7fc8b1d9b640 1 -- 172.21.15.88:0/4062975154 >> v1:172.21.15.88:6789/0 conn(0x7fc8b415e940 legacy=0x7fc8b40b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:22.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.892+0000 7fc8b1d9b640 1 -- 172.21.15.88:0/4062975154 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8b40d35c0 con 0x7fc8b40b6850 2024-09-16T01:34:22.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.892+0000 7fc8b1d9b640 1 --2- 172.21.15.88:0/4062975154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b40b6850 0x7fc8b40b7ba0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc8b40d0c90 tx=0x7fc8a802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90041ad83aeea9b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:22.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.892+0000 7fc8b159a640 1 -- 172.21.15.88:0/4062975154 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a8037070 con 0x7fc8b40b6850 2024-09-16T01:34:22.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.892+0000 7fc8b159a640 1 -- 172.21.15.88:0/4062975154 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8a8032440 con 0x7fc8b40b6850 2024-09-16T01:34:22.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.892+0000 7fc8b159a640 1 -- 172.21.15.88:0/4062975154 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a8032810 con 0x7fc8b40b6850 2024-09-16T01:34:22.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.893+0000 7fc8b8fc4640 1 -- 172.21.15.88:0/4062975154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b40b6850 msgr2=0x7fc8b40b7ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:22.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.893+0000 7fc8b8fc4640 1 --2- 172.21.15.88:0/4062975154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b40b6850 0x7fc8b40b7ba0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc8b40d0c90 tx=0x7fc8a802a180 comp rx=0 tx=0).stop 2024-09-16T01:34:22.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.893+0000 7fc8b8fc4640 1 -- 172.21.15.88:0/4062975154 shutdown_connections 2024-09-16T01:34:22.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.893+0000 7fc8b8fc4640 1 --2- 172.21.15.88:0/4062975154 >> v2:172.21.15.154:3300/0 conn(0x7fc8b40ba440 0x7fc8b40b80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.893+0000 7fc8b8fc4640 1 --2- 172.21.15.88:0/4062975154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b40b6850 0x7fc8b40b7ba0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.893+0000 7fc8b8fc4640 1 -- 172.21.15.88:0/4062975154 >> 172.21.15.88:0/4062975154 conn(0x7fc8b40d1140 msgr2=0x7fc8b40d1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:22.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.893+0000 7fc8b8fc4640 1 -- 172.21.15.88:0/4062975154 shutdown_connections 2024-09-16T01:34:22.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.893+0000 7fc8b8fc4640 1 -- 172.21.15.88:0/4062975154 wait complete. 2024-09-16T01:34:22.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.894+0000 7fc8b8fc4640 1 Processor -- start 2024-09-16T01:34:22.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.895+0000 7fc8b8fc4640 1 -- start start 2024-09-16T01:34:22.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.895+0000 7fc8b8fc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8b40b6850 0x7fc8b40adae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:22.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.895+0000 7fc8b8fc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8b40ba440 0x7fc8b4154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:22.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.895+0000 7fc8b8fc4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b415e940 0x7fc8b40c81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:22.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.895+0000 7fc8b8fc4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b40b7510 con 0x7fc8b415e940 2024-09-16T01:34:22.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.895+0000 7fc8b8fc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b40d3b80 con 0x7fc8b40b6850 2024-09-16T01:34:22.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.895+0000 7fc8b8fc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc8b40d2290 con 0x7fc8b40ba440 2024-09-16T01:34:22.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.895+0000 7fc8b2d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b415e940 0x7fc8b40c81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:22.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.895+0000 7fc8b2d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b415e940 0x7fc8b40c81d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58186/0 (socket says 172.21.15.88:58186) 2024-09-16T01:34:22.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.895+0000 7fc8b2d9d640 1 -- 172.21.15.88:0/2189032070 learned_addr learned my addr 172.21.15.88:0/2189032070 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:22.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.896+0000 7fc8b259c640 1 --2- 172.21.15.88:0/2189032070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8b40b6850 0x7fc8b40adae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:22.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.896+0000 7fc8b2d9d640 1 -- 172.21.15.88:0/2189032070 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8b40ba440 msgr2=0x7fc8b4154b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:22.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.896+0000 7fc8b1d9b640 1 --2- 172.21.15.88:0/2189032070 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8b40ba440 0x7fc8b4154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:22.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.896+0000 7fc8b2d9d640 1 --2- 172.21.15.88:0/2189032070 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8b40ba440 0x7fc8b4154b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.896+0000 7fc8b2d9d640 1 -- 172.21.15.88:0/2189032070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8b40b6850 msgr2=0x7fc8b40adae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:22.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.896+0000 7fc8b2d9d640 1 --2- 172.21.15.88:0/2189032070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8b40b6850 0x7fc8b40adae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:22.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.896+0000 7fc8b2d9d640 1 -- 172.21.15.88:0/2189032070 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8b41550a0 con 0x7fc8b415e940 2024-09-16T01:34:22.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.896+0000 7fc8b259c640 1 --2- 172.21.15.88:0/2189032070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8b40b6850 0x7fc8b40adae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:22.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.896+0000 7fc8b1d9b640 1 --2- 172.21.15.88:0/2189032070 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8b40ba440 0x7fc8b4154b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:22.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.896+0000 7fc8b2d9d640 1 --2- 172.21.15.88:0/2189032070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b415e940 0x7fc8b40c81d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac00b9e0 tx=0x7fc8ac00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:22.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.896+0000 7fc8a2ffd640 1 -- 172.21.15.88:0/2189032070 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac019070 con 0x7fc8b415e940 2024-09-16T01:34:22.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.896+0000 7fc8a2ffd640 1 -- 172.21.15.88:0/2189032070 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8ac00cb40 con 0x7fc8b415e940 2024-09-16T01:34:22.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.896+0000 7fc8b8fc4640 1 -- 172.21.15.88:0/2189032070 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8b40c8d40 con 0x7fc8b415e940 2024-09-16T01:34:22.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.896+0000 7fc8a2ffd640 1 -- 172.21.15.88:0/2189032070 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac014440 con 0x7fc8b415e940 2024-09-16T01:34:22.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.896+0000 7fc8b8fc4640 1 -- 172.21.15.88:0/2189032070 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8b40c9fe0 con 0x7fc8b415e940 2024-09-16T01:34:22.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.897+0000 7fc8b8fc4640 1 -- 172.21.15.88:0/2189032070 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8b40c9260 con 0x7fc8b415e940 2024-09-16T01:34:22.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.897+0000 7fc8a2ffd640 1 -- 172.21.15.88:0/2189032070 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc8ac01c070 con 0x7fc8b415e940 2024-09-16T01:34:22.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.898+0000 7fc8a2ffd640 1 --2- 172.21.15.88:0/2189032070 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc89406c2b0 0x7fc89406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:22.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.898+0000 7fc8b259c640 1 --2- 172.21.15.88:0/2189032070 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc89406c2b0 0x7fc89406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:22.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.898+0000 7fc8a2ffd640 1 -- 172.21.15.88:0/2189032070 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc8ac08b070 con 0x7fc8b415e940 2024-09-16T01:34:22.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.898+0000 7fc8b259c640 1 --2- 172.21.15.88:0/2189032070 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc89406c2b0 0x7fc89406e770 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc8b40d1a20 tx=0x7fc8a4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:22.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:22.902+0000 7fc8a2ffd640 1 -- 172.21.15.88:0/2189032070 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc8b41550a0 con 0x7fc8b415e940 2024-09-16T01:34:23.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.100+0000 7fc8b8fc4640 1 -- 172.21.15.88:0/2189032070 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fc8b40b6140 con 0x7fc8b415e940 2024-09-16T01:34:23.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.185+0000 7fc8a2ffd640 1 -- 172.21.15.88:0/2189032070 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) ==== 118+0+0 (secure 0 0 0) 0x7fc8ac059870 con 0x7fc8b415e940 2024-09-16T01:34:23.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.187+0000 7fc8b8fc4640 1 -- 172.21.15.88:0/2189032070 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc89406c2b0 msgr2=0x7fc89406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:23.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.187+0000 7fc8b8fc4640 1 --2- 172.21.15.88:0/2189032070 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc89406c2b0 0x7fc89406e770 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc8b40d1a20 tx=0x7fc8a4009040 comp rx=0 tx=0).stop 2024-09-16T01:34:23.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.187+0000 7fc8b8fc4640 1 -- 172.21.15.88:0/2189032070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b415e940 msgr2=0x7fc8b40c81d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:23.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.187+0000 7fc8b8fc4640 1 --2- 172.21.15.88:0/2189032070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b415e940 0x7fc8b40c81d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac00b9e0 tx=0x7fc8ac00bea0 comp rx=0 tx=0).stop 2024-09-16T01:34:23.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.188+0000 7fc8b8fc4640 1 -- 172.21.15.88:0/2189032070 shutdown_connections 2024-09-16T01:34:23.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.188+0000 7fc8b8fc4640 1 --2- 172.21.15.88:0/2189032070 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc89406c2b0 0x7fc89406e770 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:23.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.188+0000 7fc8b8fc4640 1 --2- 172.21.15.88:0/2189032070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8b415e940 0x7fc8b40c81d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:23.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.188+0000 7fc8b8fc4640 1 --2- 172.21.15.88:0/2189032070 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8b40ba440 0x7fc8b4154b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:23.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.188+0000 7fc8b8fc4640 1 --2- 172.21.15.88:0/2189032070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8b40b6850 0x7fc8b40adae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:23.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.188+0000 7fc8b8fc4640 1 -- 172.21.15.88:0/2189032070 >> 172.21.15.88:0/2189032070 conn(0x7fc8b40d1140 msgr2=0x7fc8b40b8550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:23.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.188+0000 7fc8b8fc4640 1 -- 172.21.15.88:0/2189032070 shutdown_connections 2024-09-16T01:34:23.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.188+0000 7fc8b8fc4640 1 -- 172.21.15.88:0/2189032070 wait complete. 2024-09-16T01:34:23.200 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-16T01:34:23.200 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-16T01:34:23.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.300+0000 7f85cc67b640 1 Processor -- start 2024-09-16T01:34:23.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.300+0000 7f85cc67b640 1 -- start start 2024-09-16T01:34:23.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.301+0000 7f85cc67b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f85c415e6f0 0x7f85c406fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:23.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.301+0000 7f85cc67b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85c40b54d0 0x7f85c415f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:23.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.301+0000 7f85cc67b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85c406e990 con 0x7f85c40b6d40 2024-09-16T01:34:23.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.301+0000 7f85cc67b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f85c406ec70 con 0x7f85c415e6f0 2024-09-16T01:34:23.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.301+0000 7f85cc67b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f85c415ec70 con 0x7f85c40b54d0 2024-09-16T01:34:23.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.301+0000 7f85cac18640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f85c40b6d40 0x7f85c40b71f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34218/0 (socket says 172.21.15.88:34218) 2024-09-16T01:34:23.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.301+0000 7f85cac18640 1 -- 172.21.15.88:0/3485949828 learned_addr learned my addr 172.21.15.88:0/3485949828 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:23.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85ca417640 1 --2- 172.21.15.88:0/3485949828 >> v2:172.21.15.154:3300/0 conn(0x7f85c415e6f0 0x7f85c406fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:23.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85c9c16640 1 --2- 172.21.15.88:0/3485949828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85c40b54d0 0x7f85c415f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:23.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85c9415640 1 -- 172.21.15.88:0/3485949828 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 305375423 0 0) 0x7f85c406e990 con 0x7f85c40b6d40 2024-09-16T01:34:23.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85c9415640 1 -- 172.21.15.88:0/3485949828 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85ac003520 con 0x7f85c40b6d40 2024-09-16T01:34:23.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85c9415640 1 -- 172.21.15.88:0/3485949828 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3069459866 0 0) 0x7f85ac003520 con 0x7f85c40b6d40 2024-09-16T01:34:23.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85c9415640 1 -- 172.21.15.88:0/3485949828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85c40b54d0 msgr2=0x7f85c415f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:23.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85c9415640 1 --2- 172.21.15.88:0/3485949828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85c40b54d0 0x7f85c415f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:23.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85c9415640 1 -- 172.21.15.88:0/3485949828 >> v2:172.21.15.154:3300/0 conn(0x7f85c415e6f0 msgr2=0x7f85c406fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:23.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85c9415640 1 --2- 172.21.15.88:0/3485949828 >> v2:172.21.15.154:3300/0 conn(0x7f85c415e6f0 0x7f85c406fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:23.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85c9415640 1 -- 172.21.15.88:0/3485949828 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85c415ef50 con 0x7f85c40b6d40 2024-09-16T01:34:23.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85c9415640 1 -- 172.21.15.88:0/3485949828 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f85bc001670 con 0x7f85c40b6d40 2024-09-16T01:34:23.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85c9415640 1 -- 172.21.15.88:0/3485949828 >> v1:172.21.15.88:6789/0 conn(0x7f85c40b6d40 legacy=0x7f85c40b71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:23.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85c9415640 1 --2- 172.21.15.88:0/3485949828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85c40b54d0 0x7f85ac0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:23.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85c9415640 1 --2- 172.21.15.88:0/3485949828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85ac003f30 0x7f85ac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:23.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85c9415640 1 --2- 172.21.15.88:0/3485949828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85ac004d40 0x7f85ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:23.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85c9415640 1 -- 172.21.15.88:0/3485949828 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f85c406ec70 con 0x7f85ac003f30 2024-09-16T01:34:23.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85c9415640 1 -- 172.21.15.88:0/3485949828 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f85c415ec70 con 0x7f85ac004d40 2024-09-16T01:34:23.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85c9415640 1 -- 172.21.15.88:0/3485949828 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f85ac00b9c0 con 0x7f85c40b54d0 2024-09-16T01:34:23.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85ca417640 1 --2- 172.21.15.88:0/3485949828 >> v2:172.21.15.154:3300/0 conn(0x7f85c415e6f0 0x7f85c406fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:34:23.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85c9c16640 1 --2- 172.21.15.88:0/3485949828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85ac003f30 0x7f85ac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:23.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85cac18640 1 --2- 172.21.15.88:0/3485949828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85ac004d40 0x7f85ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:23.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85ca417640 1 --2- 172.21.15.88:0/3485949828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85c40b54d0 0x7f85ac0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:23.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85c9c16640 1 -- 172.21.15.88:0/3485949828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85c40b54d0 msgr2=0x7f85ac0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:23.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85c9c16640 1 --2- 172.21.15.88:0/3485949828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85c40b54d0 0x7f85ac0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:23.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85c9c16640 1 -- 172.21.15.88:0/3485949828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85ac004d40 msgr2=0x7f85ac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:23.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85c9c16640 1 --2- 172.21.15.88:0/3485949828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85ac004d40 0x7f85ac0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:23.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85c9c16640 1 -- 172.21.15.88:0/3485949828 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85ac003520 con 0x7f85ac003f30 2024-09-16T01:34:23.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85ca417640 1 --2- 172.21.15.88:0/3485949828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85c40b54d0 0x7f85ac0039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:23.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.302+0000 7f85cac18640 1 --2- 172.21.15.88:0/3485949828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85ac004d40 0x7f85ac0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:23.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.303+0000 7f85c9c16640 1 --2- 172.21.15.88:0/3485949828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85ac003f30 0x7f85ac004800 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f85b8009a90 tx=0x7f85ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efa2e9c522bb29e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:23.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.303+0000 7f85c9415640 1 -- 172.21.15.88:0/3485949828 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b8036070 con 0x7f85ac003f30 2024-09-16T01:34:23.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.303+0000 7f85c9415640 1 -- 172.21.15.88:0/3485949828 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85b8031400 con 0x7f85ac003f30 2024-09-16T01:34:23.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.303+0000 7f85c9415640 1 -- 172.21.15.88:0/3485949828 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b8009690 con 0x7f85ac003f30 2024-09-16T01:34:23.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.304+0000 7f85cc67b640 1 -- 172.21.15.88:0/3485949828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85ac003f30 msgr2=0x7f85ac004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:23.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.304+0000 7f85cc67b640 1 --2- 172.21.15.88:0/3485949828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85ac003f30 0x7f85ac004800 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f85b8009a90 tx=0x7f85ac005830 comp rx=0 tx=0).stop 2024-09-16T01:34:23.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.304+0000 7f85cac18640 1 -- 172.21.15.88:0/3485949828 reap_dead start 2024-09-16T01:34:23.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.304+0000 7f85cc67b640 1 -- 172.21.15.88:0/3485949828 shutdown_connections 2024-09-16T01:34:23.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.304+0000 7f85cc67b640 1 -- 172.21.15.88:0/3485949828 >> 172.21.15.88:0/3485949828 conn(0x7f85c40d2430 msgr2=0x7f85c40d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:23.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.304+0000 7f85cc67b640 1 -- 172.21.15.88:0/3485949828 shutdown_connections 2024-09-16T01:34:23.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.304+0000 7f85cc67b640 1 -- 172.21.15.88:0/3485949828 wait complete. 2024-09-16T01:34:23.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.305+0000 7f85cc67b640 1 Processor -- start 2024-09-16T01:34:23.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.305+0000 7f85cc67b640 1 -- start start 2024-09-16T01:34:23.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.306+0000 7f85cc67b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85ac003f30 0x7f85c414df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:23.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.306+0000 7f85cc67b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85c40d1100 0x7f85c415cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:23.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.306+0000 7f85cc67b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85c415d190 0x7f85c40ce230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:23.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.306+0000 7f85cc67b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f85c406e730 con 0x7f85ac003f30 2024-09-16T01:34:23.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.306+0000 7f85cc67b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f85c406e5b0 con 0x7f85c415d190 2024-09-16T01:34:23.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.306+0000 7f85cc67b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f85c415f750 con 0x7f85c40d1100 2024-09-16T01:34:23.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.306+0000 7f85ca417640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85ac003f30 0x7f85c414df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:23.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.306+0000 7f85ca417640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85ac003f30 0x7f85c414df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58206/0 (socket says 172.21.15.88:58206) 2024-09-16T01:34:23.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.306+0000 7f85ca417640 1 -- 172.21.15.88:0/1974051526 learned_addr learned my addr 172.21.15.88:0/1974051526 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:23.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.306+0000 7f85ca417640 1 -- 172.21.15.88:0/1974051526 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85c40d1100 msgr2=0x7f85c415cc50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:34:23.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.306+0000 7f85ca417640 1 --2- 172.21.15.88:0/1974051526 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85c40d1100 0x7f85c415cc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:23.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.306+0000 7f85ca417640 1 -- 172.21.15.88:0/1974051526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85c415d190 msgr2=0x7f85c40ce230 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:34:23.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.306+0000 7f85ca417640 1 --2- 172.21.15.88:0/1974051526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85c415d190 0x7f85c40ce230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:23.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.306+0000 7f85ca417640 1 -- 172.21.15.88:0/1974051526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85c414e450 con 0x7f85ac003f30 2024-09-16T01:34:23.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.306+0000 7f85ca417640 1 --2- 172.21.15.88:0/1974051526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85ac003f30 0x7f85c414df10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f85b400f460 tx=0x7f85b400f920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:23.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.306+0000 7f85c2ffd640 1 -- 172.21.15.88:0/1974051526 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b40147e0 con 0x7f85ac003f30 2024-09-16T01:34:23.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.306+0000 7f85c2ffd640 1 -- 172.21.15.88:0/1974051526 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85b401b070 con 0x7f85ac003f30 2024-09-16T01:34:23.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.306+0000 7f85cc67b640 1 -- 172.21.15.88:0/1974051526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85c40cba30 con 0x7f85ac003f30 2024-09-16T01:34:23.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.306+0000 7f85c2ffd640 1 -- 172.21.15.88:0/1974051526 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b4014b80 con 0x7f85ac003f30 2024-09-16T01:34:23.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.306+0000 7f85cc67b640 1 -- 172.21.15.88:0/1974051526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85c40cbff0 con 0x7f85ac003f30 2024-09-16T01:34:23.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.307+0000 7f85cc67b640 1 -- 172.21.15.88:0/1974051526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85c406c6c0 con 0x7f85ac003f30 2024-09-16T01:34:23.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.308+0000 7f85c2ffd640 1 -- 172.21.15.88:0/1974051526 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f85b4005d10 con 0x7f85ac003f30 2024-09-16T01:34:23.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.308+0000 7f85c2ffd640 1 --2- 172.21.15.88:0/1974051526 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f85b006c2b0 0x7f85b006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:23.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.308+0000 7f85c2ffd640 1 -- 172.21.15.88:0/1974051526 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f85b408c470 con 0x7f85ac003f30 2024-09-16T01:34:23.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.308+0000 7f85c9c16640 1 --2- 172.21.15.88:0/1974051526 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f85b006c2b0 0x7f85b006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:23.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.309+0000 7f85c9c16640 1 --2- 172.21.15.88:0/1974051526 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f85b006c2b0 0x7f85b006e770 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f85b8033000 tx=0x7f85b8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:23.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.313+0000 7f85c2ffd640 1 -- 172.21.15.88:0/1974051526 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f85b4025020 con 0x7f85ac003f30 2024-09-16T01:34:23.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:23.514+0000 7f85cc67b640 1 -- 172.21.15.88:0/1974051526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f85c415d590 con 0x7f85ac003f30 2024-09-16T01:34:24.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.192+0000 7f85c2ffd640 1 -- 172.21.15.88:0/1974051526 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v6) ==== 122+0+0 (secure 0 0 0) 0x7f85c414e450 con 0x7f85ac003f30 2024-09-16T01:34:24.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.194+0000 7f85cc67b640 1 -- 172.21.15.88:0/1974051526 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f85b006c2b0 msgr2=0x7f85b006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:24.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.194+0000 7f85cc67b640 1 --2- 172.21.15.88:0/1974051526 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f85b006c2b0 0x7f85b006e770 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f85b8033000 tx=0x7f85b8008030 comp rx=0 tx=0).stop 2024-09-16T01:34:24.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.194+0000 7f85cc67b640 1 -- 172.21.15.88:0/1974051526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85ac003f30 msgr2=0x7f85c414df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:24.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.194+0000 7f85cc67b640 1 --2- 172.21.15.88:0/1974051526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85ac003f30 0x7f85c414df10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f85b400f460 tx=0x7f85b400f920 comp rx=0 tx=0).stop 2024-09-16T01:34:24.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.194+0000 7f85cc67b640 1 -- 172.21.15.88:0/1974051526 shutdown_connections 2024-09-16T01:34:24.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.194+0000 7f85cc67b640 1 --2- 172.21.15.88:0/1974051526 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f85b006c2b0 0x7f85b006e770 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.194+0000 7f85cc67b640 1 --2- 172.21.15.88:0/1974051526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85c415d190 0x7f85c40ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.194+0000 7f85cc67b640 1 --2- 172.21.15.88:0/1974051526 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85c40d1100 0x7f85c415cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.194+0000 7f85cc67b640 1 --2- 172.21.15.88:0/1974051526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85ac003f30 0x7f85c414df10 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.194+0000 7f85cc67b640 1 -- 172.21.15.88:0/1974051526 >> 172.21.15.88:0/1974051526 conn(0x7f85c40d2430 msgr2=0x7f85c4161d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:24.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.194+0000 7f85cc67b640 1 -- 172.21.15.88:0/1974051526 shutdown_connections 2024-09-16T01:34:24.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.194+0000 7f85cc67b640 1 -- 172.21.15.88:0/1974051526 wait complete. 2024-09-16T01:34:24.206 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-16T01:34:24.207 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:34:24.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.304+0000 7f002c99f640 1 Processor -- start 2024-09-16T01:34:24.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.304+0000 7f002c99f640 1 -- start start 2024-09-16T01:34:24.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.305+0000 7f002c99f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0028072cb0 0x7f00280b57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:24.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.305+0000 7f002c99f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00280b6270 con 0x7f002815bac0 2024-09-16T01:34:24.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.305+0000 7f002c99f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00280b6550 con 0x7f00280b5d10 2024-09-16T01:34:24.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.305+0000 7f002c99f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0028155d00 con 0x7f0028072cb0 2024-09-16T01:34:24.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.305+0000 7f002659c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f002815bac0 0x7f002815bea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34228/0 (socket says 172.21.15.88:34228) 2024-09-16T01:34:24.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.305+0000 7f002659c640 1 -- 172.21.15.88:0/3608444132 learned_addr learned my addr 172.21.15.88:0/3608444132 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:24.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.305+0000 7f001dd9b640 1 --2- 172.21.15.88:0/3608444132 >> v2:172.21.15.154:3300/0 conn(0x7f0028072cb0 0x7f00280b57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:24.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.305+0000 7f0025d9b640 1 -- 172.21.15.88:0/3608444132 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1040075181 0 0) 0x7f00280b6270 con 0x7f002815bac0 2024-09-16T01:34:24.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.305+0000 7f0025d9b640 1 -- 172.21.15.88:0/3608444132 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0014003520 con 0x7f002815bac0 2024-09-16T01:34:24.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.305+0000 7f0025d9b640 1 -- 172.21.15.88:0/3608444132 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1864202027 0 0) 0x7f0014003520 con 0x7f002815bac0 2024-09-16T01:34:24.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.305+0000 7f0025d9b640 1 -- 172.21.15.88:0/3608444132 >> v2:172.21.15.154:3300/0 conn(0x7f0028072cb0 msgr2=0x7f00280b57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:24.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.305+0000 7f0025d9b640 1 --2- 172.21.15.88:0/3608444132 >> v2:172.21.15.154:3300/0 conn(0x7f0028072cb0 0x7f00280b57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.305+0000 7f0025d9b640 1 -- 172.21.15.88:0/3608444132 >> v1:172.21.15.154:6789/0 conn(0x7f00280b5d10 legacy=0x7f0028164e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:24.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.305+0000 7f0025d9b640 1 -- 172.21.15.88:0/3608444132 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00280b6830 con 0x7f002815bac0 2024-09-16T01:34:24.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.305+0000 7f0025d9b640 1 -- 172.21.15.88:0/3608444132 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f0018001670 con 0x7f002815bac0 2024-09-16T01:34:24.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.305+0000 7f0025d9b640 1 -- 172.21.15.88:0/3608444132 >> v1:172.21.15.88:6789/0 conn(0x7f002815bac0 legacy=0x7f002815bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:24.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.305+0000 7f0025d9b640 1 --2- 172.21.15.88:0/3608444132 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00140039b0 0x7f0014003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:24.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.305+0000 7f0025d9b640 1 --2- 172.21.15.88:0/3608444132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0014004800 0x7f0014004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:24.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.305+0000 7f0025d9b640 1 --2- 172.21.15.88:0/3608444132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00140051c0 0x7f001400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:24.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.305+0000 7f0025d9b640 1 -- 172.21.15.88:0/3608444132 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0028155d00 con 0x7f00140051c0 2024-09-16T01:34:24.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.305+0000 7f0025d9b640 1 -- 172.21.15.88:0/3608444132 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f001400c060 con 0x7f0014004800 2024-09-16T01:34:24.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.305+0000 7f0025d9b640 1 -- 172.21.15.88:0/3608444132 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f001400c260 con 0x7f00140039b0 2024-09-16T01:34:24.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.305+0000 7f0026d9d640 1 --2- 172.21.15.88:0/3608444132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00140051c0 0x7f001400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:24.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.306+0000 7f001dd9b640 1 --2- 172.21.15.88:0/3608444132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0014004800 0x7f0014004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:24.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.306+0000 7f002659c640 1 --2- 172.21.15.88:0/3608444132 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00140039b0 0x7f0014003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:24.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.306+0000 7f0026d9d640 1 -- 172.21.15.88:0/3608444132 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00140039b0 msgr2=0x7f0014003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:24.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.306+0000 7f0026d9d640 1 --2- 172.21.15.88:0/3608444132 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00140039b0 0x7f0014003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.306+0000 7f0026d9d640 1 -- 172.21.15.88:0/3608444132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0014004800 msgr2=0x7f0014004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:24.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.306+0000 7f0026d9d640 1 --2- 172.21.15.88:0/3608444132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0014004800 0x7f0014004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.306+0000 7f0026d9d640 1 -- 172.21.15.88:0/3608444132 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0014003520 con 0x7f00140051c0 2024-09-16T01:34:24.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.306+0000 7f0026d9d640 1 -- 172.21.15.88:0/3608444132 reap_dead start 2024-09-16T01:34:24.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.306+0000 7f0026d9d640 1 --2- 172.21.15.88:0/3608444132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00140051c0 0x7f001400b9c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0020001e20 tx=0x7f0014005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d145881612bfff82 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:24.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.306+0000 7f0025d9b640 1 -- 172.21.15.88:0/3608444132 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020001a70 con 0x7f00140051c0 2024-09-16T01:34:24.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.307+0000 7f0025d9b640 1 -- 172.21.15.88:0/3608444132 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f002002ae40 con 0x7f00140051c0 2024-09-16T01:34:24.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.307+0000 7f0025d9b640 1 -- 172.21.15.88:0/3608444132 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020003bb0 con 0x7f00140051c0 2024-09-16T01:34:24.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.307+0000 7f002c99f640 1 -- 172.21.15.88:0/3608444132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00140051c0 msgr2=0x7f001400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:24.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.307+0000 7f002c99f640 1 --2- 172.21.15.88:0/3608444132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00140051c0 0x7f001400b9c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0020001e20 tx=0x7f0014005830 comp rx=0 tx=0).stop 2024-09-16T01:34:24.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.307+0000 7f002c99f640 1 -- 172.21.15.88:0/3608444132 shutdown_connections 2024-09-16T01:34:24.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.307+0000 7f002c99f640 1 --2- 172.21.15.88:0/3608444132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00140051c0 0x7f001400b9c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.307+0000 7f002c99f640 1 -- 172.21.15.88:0/3608444132 >> 172.21.15.88:0/3608444132 conn(0x7f002806bc10 msgr2=0x7f002806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:24.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.307+0000 7f002c99f640 1 -- 172.21.15.88:0/3608444132 shutdown_connections 2024-09-16T01:34:24.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.307+0000 7f002c99f640 1 -- 172.21.15.88:0/3608444132 wait complete. 2024-09-16T01:34:24.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.308+0000 7f002c99f640 1 Processor -- start 2024-09-16T01:34:24.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.309+0000 7f002c99f640 1 -- start start 2024-09-16T01:34:24.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.309+0000 7f002c99f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00140051c0 0x7f002814eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:24.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.309+0000 7f002c99f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00281f29b0 0x7f002814c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:24.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.309+0000 7f002c99f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f002814cac0 0x7f002815d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:24.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.309+0000 7f002c99f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f00280739d0 con 0x7f002814cac0 2024-09-16T01:34:24.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.309+0000 7f002c99f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0028071810 con 0x7f00281f29b0 2024-09-16T01:34:24.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.309+0000 7f002c99f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f00280b57d0 con 0x7f00140051c0 2024-09-16T01:34:24.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.309+0000 7f0026d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f002814cac0 0x7f002815d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:24.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.309+0000 7f0026d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f002814cac0 0x7f002815d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58220/0 (socket says 172.21.15.88:58220) 2024-09-16T01:34:24.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.309+0000 7f0026d9d640 1 -- 172.21.15.88:0/916081707 learned_addr learned my addr 172.21.15.88:0/916081707 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:24.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.310+0000 7f001dd9b640 1 --2- 172.21.15.88:0/916081707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00281f29b0 0x7f002814c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:24.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.310+0000 7f002659c640 1 --2- 172.21.15.88:0/916081707 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00140051c0 0x7f002814eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:24.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.310+0000 7f0026d9d640 1 -- 172.21.15.88:0/916081707 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00140051c0 msgr2=0x7f002814eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:24.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.310+0000 7f0026d9d640 1 --2- 172.21.15.88:0/916081707 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00140051c0 0x7f002814eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.310+0000 7f0026d9d640 1 -- 172.21.15.88:0/916081707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00281f29b0 msgr2=0x7f002814c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:24.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.310+0000 7f0026d9d640 1 --2- 172.21.15.88:0/916081707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00281f29b0 0x7f002814c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.310+0000 7f0026d9d640 1 -- 172.21.15.88:0/916081707 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00281f5fa0 con 0x7f002814cac0 2024-09-16T01:34:24.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.310+0000 7f0026d9d640 1 --2- 172.21.15.88:0/916081707 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f002814cac0 0x7f002815d6a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0020001f40 tx=0x7f00200014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:24.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.310+0000 7f001f7fe640 1 -- 172.21.15.88:0/916081707 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020007db0 con 0x7f002814cac0 2024-09-16T01:34:24.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.310+0000 7f001f7fe640 1 -- 172.21.15.88:0/916081707 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0020001250 con 0x7f002814cac0 2024-09-16T01:34:24.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.310+0000 7f001f7fe640 1 -- 172.21.15.88:0/916081707 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020005500 con 0x7f002814cac0 2024-09-16T01:34:24.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.310+0000 7f002c99f640 1 -- 172.21.15.88:0/916081707 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00281f2f20 con 0x7f002814cac0 2024-09-16T01:34:24.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.310+0000 7f002c99f640 1 -- 172.21.15.88:0/916081707 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00281f34e0 con 0x7f002814cac0 2024-09-16T01:34:24.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.311+0000 7f002c99f640 1 -- 172.21.15.88:0/916081707 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f002815bea0 con 0x7f002814cac0 2024-09-16T01:34:24.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.312+0000 7f001f7fe640 1 -- 172.21.15.88:0/916081707 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f00200056a0 con 0x7f002814cac0 2024-09-16T01:34:24.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.312+0000 7f001f7fe640 1 --2- 172.21.15.88:0/916081707 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f000806c2e0 0x7f000806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:24.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.312+0000 7f001f7fe640 1 -- 172.21.15.88:0/916081707 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f00200aa830 con 0x7f002814cac0 2024-09-16T01:34:24.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.312+0000 7f002659c640 1 --2- 172.21.15.88:0/916081707 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f000806c2e0 0x7f000806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:24.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.312+0000 7f002659c640 1 --2- 172.21.15.88:0/916081707 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f000806c2e0 0x7f000806e7a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0018001cd0 tx=0x7f0018001c60 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:24.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.316+0000 7f001f7fe640 1 -- 172.21.15.88:0/916081707 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0020043060 con 0x7f002814cac0 2024-09-16T01:34:24.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.513+0000 7f002c99f640 1 -- 172.21.15.88:0/916081707 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f002815b320 con 0x7f002814cac0 2024-09-16T01:34:24.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.514+0000 7f001f7fe640 1 -- 172.21.15.88:0/916081707 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5223 (secure 0 0 0) 0x7f0020039070 con 0x7f002814cac0 2024-09-16T01:34:24.515 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:24.515 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":6,"btime":"2024-09-16T01:34:24:186280+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/1620784263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1620784263},{"type":"v1","addr":"172.21.15.154:6834","nonce":1620784263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3407296593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":3407296593},{"type":"v1","addr":"172.21.15.154:6835","nonce":3407296593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4293,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/4058543372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4058543372},{"type":"v1","addr":"172.21.15.88:6835","nonce":4058543372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:34:21.508480+0000","modified":"2024-09-16T01:34:23.515443+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6837/3712834803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":3712834803},{"type":"v1","addr":"172.21.15.88:6837","nonce":3712834803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4296,"qdb_cluster":[4296]},"id":1}]} 2024-09-16T01:34:24.516 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 6 2024-09-16T01:34:24.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.516+0000 7f002c99f640 1 -- 172.21.15.88:0/916081707 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f000806c2e0 msgr2=0x7f000806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:24.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.516+0000 7f002c99f640 1 --2- 172.21.15.88:0/916081707 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f000806c2e0 0x7f000806e7a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0018001cd0 tx=0x7f0018001c60 comp rx=0 tx=0).stop 2024-09-16T01:34:24.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.516+0000 7f002c99f640 1 -- 172.21.15.88:0/916081707 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f002814cac0 msgr2=0x7f002815d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:24.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.516+0000 7f002c99f640 1 --2- 172.21.15.88:0/916081707 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f002814cac0 0x7f002815d6a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0020001f40 tx=0x7f00200014a0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.516+0000 7f002c99f640 1 -- 172.21.15.88:0/916081707 shutdown_connections 2024-09-16T01:34:24.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.516+0000 7f002c99f640 1 --2- 172.21.15.88:0/916081707 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f000806c2e0 0x7f000806e7a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.516+0000 7f002c99f640 1 --2- 172.21.15.88:0/916081707 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f002814cac0 0x7f002815d6a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.516+0000 7f002c99f640 1 --2- 172.21.15.88:0/916081707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00281f29b0 0x7f002814c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.516+0000 7f002c99f640 1 --2- 172.21.15.88:0/916081707 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00140051c0 0x7f002814eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.516+0000 7f002c99f640 1 -- 172.21.15.88:0/916081707 >> 172.21.15.88:0/916081707 conn(0x7f002806bc10 msgr2=0x7f00280b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:24.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.516+0000 7f002c99f640 1 -- 172.21.15.88:0/916081707 shutdown_connections 2024-09-16T01:34:24.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.516+0000 7f002c99f640 1 -- 172.21.15.88:0/916081707 wait complete. 2024-09-16T01:34:24.528 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:34:24.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.625+0000 7f1daa5e8640 1 Processor -- start 2024-09-16T01:34:24.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.625+0000 7f1daa5e8640 1 -- start start 2024-09-16T01:34:24.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.625+0000 7f1daa5e8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1da415e660 0x7f1da40cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:24.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.625+0000 7f1daa5e8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1da40d1820 0x7f1da40b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:24.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.625+0000 7f1daa5e8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1da415ea40 con 0x7f1da40d3310 2024-09-16T01:34:24.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.625+0000 7f1daa5e8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f1da40d3870 con 0x7f1da415e660 2024-09-16T01:34:24.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.625+0000 7f1daa5e8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1da40d3b50 con 0x7f1da40d1820 2024-09-16T01:34:24.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.626+0000 7f1da37fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1da40d3310 0x7f1da40d1110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34234/0 (socket says 172.21.15.88:34234) 2024-09-16T01:34:24.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.626+0000 7f1da37fe640 1 -- 172.21.15.88:0/3664382737 learned_addr learned my addr 172.21.15.88:0/3664382737 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:24.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.626+0000 7f1da2ffd640 1 -- 172.21.15.88:0/3664382737 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3814954650 0 0) 0x7f1da415ea40 con 0x7f1da40d3310 2024-09-16T01:34:24.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.626+0000 7f1da2ffd640 1 -- 172.21.15.88:0/3664382737 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d8c003520 con 0x7f1da40d3310 2024-09-16T01:34:24.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.626+0000 7f1da8b85640 1 --2- 172.21.15.88:0/3664382737 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1da40d1820 0x7f1da40b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:24.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.626+0000 7f1da3fff640 1 --2- 172.21.15.88:0/3664382737 >> v2:172.21.15.154:3300/0 conn(0x7f1da415e660 0x7f1da40cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:24.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.626+0000 7f1da2ffd640 1 -- 172.21.15.88:0/3664382737 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 396323405 0 0) 0x7f1d8c003520 con 0x7f1da40d3310 2024-09-16T01:34:24.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.626+0000 7f1da2ffd640 1 -- 172.21.15.88:0/3664382737 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1da40d1820 msgr2=0x7f1da40b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:24.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.626+0000 7f1da2ffd640 1 --2- 172.21.15.88:0/3664382737 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1da40d1820 0x7f1da40b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.626+0000 7f1da2ffd640 1 -- 172.21.15.88:0/3664382737 >> v2:172.21.15.154:3300/0 conn(0x7f1da415e660 msgr2=0x7f1da40cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:24.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.626+0000 7f1da2ffd640 1 --2- 172.21.15.88:0/3664382737 >> v2:172.21.15.154:3300/0 conn(0x7f1da415e660 0x7f1da40cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.626+0000 7f1da2ffd640 1 -- 172.21.15.88:0/3664382737 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da40d3e30 con 0x7f1da40d3310 2024-09-16T01:34:24.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.626+0000 7f1da2ffd640 1 -- 172.21.15.88:0/3664382737 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f1d98001670 con 0x7f1da40d3310 2024-09-16T01:34:24.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.626+0000 7f1da3fff640 1 --2- 172.21.15.88:0/3664382737 >> v2:172.21.15.154:3300/0 conn(0x7f1da415e660 0x7f1da40cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:24.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.626+0000 7f1da2ffd640 1 -- 172.21.15.88:0/3664382737 >> v1:172.21.15.88:6789/0 conn(0x7f1da40d3310 legacy=0x7f1da40d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:24.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.626+0000 7f1da2ffd640 1 --2- 172.21.15.88:0/3664382737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d8c0039b0 0x7f1d8c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:24.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.626+0000 7f1da2ffd640 1 --2- 172.21.15.88:0/3664382737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d8c004800 0x7f1d8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:24.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.626+0000 7f1da2ffd640 1 --2- 172.21.15.88:0/3664382737 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1da40d1820 0x7f1d8c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:24.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.626+0000 7f1da2ffd640 1 -- 172.21.15.88:0/3664382737 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1da40d3870 con 0x7f1d8c0039b0 2024-09-16T01:34:24.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.626+0000 7f1da2ffd640 1 -- 172.21.15.88:0/3664382737 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1da40d3b50 con 0x7f1d8c004800 2024-09-16T01:34:24.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.626+0000 7f1da2ffd640 1 -- 172.21.15.88:0/3664382737 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1d8c00ba10 con 0x7f1da40d1820 2024-09-16T01:34:24.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.626+0000 7f1da3fff640 1 --2- 172.21.15.88:0/3664382737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d8c0039b0 0x7f1d8c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:24.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.626+0000 7f1da37fe640 1 --2- 172.21.15.88:0/3664382737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d8c004800 0x7f1d8c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:24.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.626+0000 7f1da8b85640 1 --2- 172.21.15.88:0/3664382737 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1da40d1820 0x7f1d8c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:24.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.627+0000 7f1da3fff640 1 -- 172.21.15.88:0/3664382737 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1da40d1820 msgr2=0x7f1d8c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:24.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.627+0000 7f1da3fff640 1 --2- 172.21.15.88:0/3664382737 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1da40d1820 0x7f1d8c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.627+0000 7f1da3fff640 1 -- 172.21.15.88:0/3664382737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d8c004800 msgr2=0x7f1d8c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:24.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.627+0000 7f1da3fff640 1 --2- 172.21.15.88:0/3664382737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d8c004800 0x7f1d8c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.627+0000 7f1da3fff640 1 -- 172.21.15.88:0/3664382737 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d8c003520 con 0x7f1d8c0039b0 2024-09-16T01:34:24.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.627+0000 7f1da3fff640 1 --2- 172.21.15.88:0/3664382737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d8c0039b0 0x7f1d8c003dd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1d940098a0 tx=0x7f1d8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa83d0c6bdf2a8f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:24.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.627+0000 7f1da2ffd640 1 -- 172.21.15.88:0/3664382737 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9400f040 con 0x7f1d8c0039b0 2024-09-16T01:34:24.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.627+0000 7f1da2ffd640 1 -- 172.21.15.88:0/3664382737 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d9402ad20 con 0x7f1d8c0039b0 2024-09-16T01:34:24.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.627+0000 7f1da2ffd640 1 -- 172.21.15.88:0/3664382737 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d94004250 con 0x7f1d8c0039b0 2024-09-16T01:34:24.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.628+0000 7f1daa5e8640 1 -- 172.21.15.88:0/3664382737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d8c0039b0 msgr2=0x7f1d8c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:24.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.628+0000 7f1daa5e8640 1 --2- 172.21.15.88:0/3664382737 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d8c0039b0 0x7f1d8c003dd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1d940098a0 tx=0x7f1d8c005830 comp rx=0 tx=0).stop 2024-09-16T01:34:24.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.628+0000 7f1da8b85640 1 -- 172.21.15.88:0/3664382737 reap_dead start 2024-09-16T01:34:24.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.628+0000 7f1daa5e8640 1 -- 172.21.15.88:0/3664382737 shutdown_connections 2024-09-16T01:34:24.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.628+0000 7f1daa5e8640 1 -- 172.21.15.88:0/3664382737 >> 172.21.15.88:0/3664382737 conn(0x7f1da40b6410 msgr2=0x7f1da40b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:24.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.628+0000 7f1daa5e8640 1 -- 172.21.15.88:0/3664382737 shutdown_connections 2024-09-16T01:34:24.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.628+0000 7f1daa5e8640 1 -- 172.21.15.88:0/3664382737 wait complete. 2024-09-16T01:34:24.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.629+0000 7f1daa5e8640 1 Processor -- start 2024-09-16T01:34:24.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.629+0000 7f1daa5e8640 1 -- start start 2024-09-16T01:34:24.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.630+0000 7f1daa5e8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d8c0039b0 0x7f1da415ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:24.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.630+0000 7f1daa5e8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1da40cc1e0 0x7f1da41e7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:24.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.630+0000 7f1daa5e8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1da40cc640 0x7f1da41e83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:24.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.630+0000 7f1daa5e8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1da40cfc70 con 0x7f1d8c0039b0 2024-09-16T01:34:24.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.630+0000 7f1daa5e8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1da40d43f0 con 0x7f1da40cc640 2024-09-16T01:34:24.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.630+0000 7f1daa5e8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1da40b7b70 con 0x7f1da40cc1e0 2024-09-16T01:34:24.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.630+0000 7f1da3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d8c0039b0 0x7f1da415ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:24.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.630+0000 7f1da3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d8c0039b0 0x7f1da415ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58238/0 (socket says 172.21.15.88:58238) 2024-09-16T01:34:24.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.630+0000 7f1da3fff640 1 -- 172.21.15.88:0/426841386 learned_addr learned my addr 172.21.15.88:0/426841386 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:24.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.630+0000 7f1da37fe640 1 --2- 172.21.15.88:0/426841386 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1da40cc1e0 0x7f1da41e7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:24.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.630+0000 7f1da3fff640 1 -- 172.21.15.88:0/426841386 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1da40cc1e0 msgr2=0x7f1da41e7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:24.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.630+0000 7f1da3fff640 1 --2- 172.21.15.88:0/426841386 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1da40cc1e0 0x7f1da41e7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.630+0000 7f1da8b85640 1 --2- 172.21.15.88:0/426841386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1da40cc640 0x7f1da41e83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:24.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.630+0000 7f1da3fff640 1 -- 172.21.15.88:0/426841386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1da40cc640 msgr2=0x7f1da41e83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:24.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.630+0000 7f1da3fff640 1 --2- 172.21.15.88:0/426841386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1da40cc640 0x7f1da41e83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.630+0000 7f1da3fff640 1 -- 172.21.15.88:0/426841386 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da40ccb50 con 0x7f1d8c0039b0 2024-09-16T01:34:24.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.630+0000 7f1da8b85640 1 --2- 172.21.15.88:0/426841386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1da40cc640 0x7f1da41e83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:24.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.630+0000 7f1da3fff640 1 --2- 172.21.15.88:0/426841386 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d8c0039b0 0x7f1da415ae90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1d940099c0 tx=0x7f1d94004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:24.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.631+0000 7f1da0ff9640 1 -- 172.21.15.88:0/426841386 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d94041070 con 0x7f1d8c0039b0 2024-09-16T01:34:24.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.631+0000 7f1da0ff9640 1 -- 172.21.15.88:0/426841386 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d9400f000 con 0x7f1d8c0039b0 2024-09-16T01:34:24.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.631+0000 7f1daa5e8640 1 -- 172.21.15.88:0/426841386 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1da40cce30 con 0x7f1d8c0039b0 2024-09-16T01:34:24.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.631+0000 7f1da0ff9640 1 -- 172.21.15.88:0/426841386 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d94003c30 con 0x7f1d8c0039b0 2024-09-16T01:34:24.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.631+0000 7f1daa5e8640 1 -- 172.21.15.88:0/426841386 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1da41f4240 con 0x7f1d8c0039b0 2024-09-16T01:34:24.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.632+0000 7f1daa5e8640 1 -- 172.21.15.88:0/426841386 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1da41f4060 con 0x7f1d8c0039b0 2024-09-16T01:34:24.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.632+0000 7f1da0ff9640 1 -- 172.21.15.88:0/426841386 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1d94044070 con 0x7f1d8c0039b0 2024-09-16T01:34:24.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.633+0000 7f1da0ff9640 1 --2- 172.21.15.88:0/426841386 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1d9006c2b0 0x7f1d9006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:24.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.633+0000 7f1da37fe640 1 --2- 172.21.15.88:0/426841386 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1d9006c2b0 0x7f1d9006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:24.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.633+0000 7f1da0ff9640 1 -- 172.21.15.88:0/426841386 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1d940abac0 con 0x7f1d8c0039b0 2024-09-16T01:34:24.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.633+0000 7f1da37fe640 1 --2- 172.21.15.88:0/426841386 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1d9006c2b0 0x7f1d9006e770 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1d98000fb0 tx=0x7f1d98001df0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:24.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.637+0000 7f1da0ff9640 1 -- 172.21.15.88:0/426841386 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1da41f4060 con 0x7f1d8c0039b0 2024-09-16T01:34:24.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.802+0000 7f1daa5e8640 1 -- 172.21.15.88:0/426841386 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1da40cf730 con 0x7f1d8c0039b0 2024-09-16T01:34:24.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.804+0000 7f1da0ff9640 1 -- 172.21.15.88:0/426841386 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16641 (secure 0 0 0) 0x7f1d9407a1a0 con 0x7f1d8c0039b0 2024-09-16T01:34:24.805 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:24.805 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:34:23.181812+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:34:19.324549+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:34:20.494051+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:34:24.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.805+0000 7f1daa5e8640 1 -- 172.21.15.88:0/426841386 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1d9006c2b0 msgr2=0x7f1d9006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:24.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.805+0000 7f1daa5e8640 1 --2- 172.21.15.88:0/426841386 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1d9006c2b0 0x7f1d9006e770 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1d98000fb0 tx=0x7f1d98001df0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.806+0000 7f1daa5e8640 1 -- 172.21.15.88:0/426841386 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d8c0039b0 msgr2=0x7f1da415ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:24.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.806+0000 7f1daa5e8640 1 --2- 172.21.15.88:0/426841386 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d8c0039b0 0x7f1da415ae90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1d940099c0 tx=0x7f1d94004730 comp rx=0 tx=0).stop 2024-09-16T01:34:24.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.806+0000 7f1daa5e8640 1 -- 172.21.15.88:0/426841386 shutdown_connections 2024-09-16T01:34:24.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.806+0000 7f1daa5e8640 1 --2- 172.21.15.88:0/426841386 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1d9006c2b0 0x7f1d9006e770 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.806+0000 7f1daa5e8640 1 --2- 172.21.15.88:0/426841386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1da40cc640 0x7f1da41e83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.806+0000 7f1daa5e8640 1 --2- 172.21.15.88:0/426841386 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1da40cc1e0 0x7f1da41e7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.806+0000 7f1daa5e8640 1 --2- 172.21.15.88:0/426841386 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d8c0039b0 0x7f1da415ae90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.806+0000 7f1daa5e8640 1 -- 172.21.15.88:0/426841386 >> 172.21.15.88:0/426841386 conn(0x7f1da40b6410 msgr2=0x7f1da406bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:24.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.806+0000 7f1daa5e8640 1 -- 172.21.15.88:0/426841386 shutdown_connections 2024-09-16T01:34:24.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.806+0000 7f1daa5e8640 1 -- 172.21.15.88:0/426841386 wait complete. 2024-09-16T01:34:24.818 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T01:34:24.818 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:34:24.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.914+0000 7fabc92f8640 1 Processor -- start 2024-09-16T01:34:24.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.914+0000 7fabc92f8640 1 -- start start 2024-09-16T01:34:24.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.915+0000 7fabc92f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabc406e920 0x7fabc415f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:24.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.915+0000 7fabc92f8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabc415fb10 con 0x7fabc40b74c0 2024-09-16T01:34:24.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.915+0000 7fabc92f8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabc40b5900 con 0x7fabc415c2b0 2024-09-16T01:34:24.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.915+0000 7fabc92f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fabc415f830 con 0x7fabc406e920 2024-09-16T01:34:24.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.915+0000 7fabba7fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fabc40b74c0 0x7fabc415eaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34250/0 (socket says 172.21.15.88:34250) 2024-09-16T01:34:24.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.915+0000 7fabba7fc640 1 -- 172.21.15.88:0/2028698584 learned_addr learned my addr 172.21.15.88:0/2028698584 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:24.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.915+0000 7fabc37fe640 1 --2- 172.21.15.88:0/2028698584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabc406e920 0x7fabc415f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:24.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.915+0000 7fabc27fc640 1 -- 172.21.15.88:0/2028698584 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3432052148 0 0) 0x7fabc415fb10 con 0x7fabc40b74c0 2024-09-16T01:34:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.915+0000 7fabc27fc640 1 -- 172.21.15.88:0/2028698584 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabb0003520 con 0x7fabc40b74c0 2024-09-16T01:34:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.916+0000 7fabc27fc640 1 -- 172.21.15.88:0/2028698584 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1334918347 0 0) 0x7fabb0003520 con 0x7fabc40b74c0 2024-09-16T01:34:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.916+0000 7fabc27fc640 1 -- 172.21.15.88:0/2028698584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabc406e920 msgr2=0x7fabc415f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.916+0000 7fabc27fc640 1 --2- 172.21.15.88:0/2028698584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabc406e920 0x7fabc415f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.916+0000 7fabc27fc640 1 -- 172.21.15.88:0/2028698584 >> v1:172.21.15.154:6789/0 conn(0x7fabc415c2b0 legacy=0x7fabc40b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.916+0000 7fabc27fc640 1 -- 172.21.15.88:0/2028698584 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc406ce30 con 0x7fabc40b74c0 2024-09-16T01:34:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.916+0000 7fabc27fc640 1 -- 172.21.15.88:0/2028698584 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fabac001670 con 0x7fabc40b74c0 2024-09-16T01:34:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.916+0000 7fabc27fc640 1 -- 172.21.15.88:0/2028698584 >> v1:172.21.15.88:6789/0 conn(0x7fabc40b74c0 legacy=0x7fabc415eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.916+0000 7fabc27fc640 1 --2- 172.21.15.88:0/2028698584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fabb00039e0 0x7fabb0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.916+0000 7fabc27fc640 1 --2- 172.21.15.88:0/2028698584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabb0004800 0x7fabb0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.916+0000 7fabc27fc640 1 --2- 172.21.15.88:0/2028698584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabc406e920 0x7fabb00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.916+0000 7fabc27fc640 1 -- 172.21.15.88:0/2028698584 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fabc415f830 con 0x7fabb00039e0 2024-09-16T01:34:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.916+0000 7fabc27fc640 1 -- 172.21.15.88:0/2028698584 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fabb000ba70 con 0x7fabb0004800 2024-09-16T01:34:24.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.916+0000 7fabc27fc640 1 -- 172.21.15.88:0/2028698584 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fabb000bc70 con 0x7fabc406e920 2024-09-16T01:34:24.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.916+0000 7fabc2ffd640 1 --2- 172.21.15.88:0/2028698584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fabb00039e0 0x7fabb0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:24.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.916+0000 7fabba7fc640 1 --2- 172.21.15.88:0/2028698584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabb0004800 0x7fabb0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:24.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.916+0000 7fabc37fe640 1 --2- 172.21.15.88:0/2028698584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabc406e920 0x7fabb00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:24.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.916+0000 7fabc2ffd640 1 -- 172.21.15.88:0/2028698584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabc406e920 msgr2=0x7fabb00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:24.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.916+0000 7fabc2ffd640 1 --2- 172.21.15.88:0/2028698584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabc406e920 0x7fabb00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.916+0000 7fabc2ffd640 1 -- 172.21.15.88:0/2028698584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabb0004800 msgr2=0x7fabb0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:24.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.916+0000 7fabc2ffd640 1 --2- 172.21.15.88:0/2028698584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabb0004800 0x7fabb0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.916+0000 7fabc2ffd640 1 -- 172.21.15.88:0/2028698584 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb0003520 con 0x7fabb00039e0 2024-09-16T01:34:24.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.916+0000 7fabc37fe640 1 --2- 172.21.15.88:0/2028698584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabc406e920 0x7fabb00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:24.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.917+0000 7fabc2ffd640 1 --2- 172.21.15.88:0/2028698584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fabb00039e0 0x7fabb0003e00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fabb4001720 tx=0x7fabb0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7434b5ab90350584 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:24.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.917+0000 7fabc27fc640 1 -- 172.21.15.88:0/2028698584 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb4001370 con 0x7fabb00039e0 2024-09-16T01:34:24.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.917+0000 7fabc27fc640 1 -- 172.21.15.88:0/2028698584 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabb40052d0 con 0x7fabb00039e0 2024-09-16T01:34:24.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.917+0000 7fabc27fc640 1 -- 172.21.15.88:0/2028698584 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb402ab90 con 0x7fabb00039e0 2024-09-16T01:34:24.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.917+0000 7fabc92f8640 1 -- 172.21.15.88:0/2028698584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fabb00039e0 msgr2=0x7fabb0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:24.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.917+0000 7fabc92f8640 1 --2- 172.21.15.88:0/2028698584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fabb00039e0 0x7fabb0003e00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fabb4001720 tx=0x7fabb0005830 comp rx=0 tx=0).stop 2024-09-16T01:34:24.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.917+0000 7fabc37fe640 1 -- 172.21.15.88:0/2028698584 reap_dead start 2024-09-16T01:34:24.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.917+0000 7fabc92f8640 1 -- 172.21.15.88:0/2028698584 shutdown_connections 2024-09-16T01:34:24.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.917+0000 7fabc92f8640 1 -- 172.21.15.88:0/2028698584 >> 172.21.15.88:0/2028698584 conn(0x7fabc40b4240 msgr2=0x7fabc40b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:24.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.918+0000 7fabc92f8640 1 -- 172.21.15.88:0/2028698584 shutdown_connections 2024-09-16T01:34:24.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.918+0000 7fabc92f8640 1 -- 172.21.15.88:0/2028698584 wait complete. 2024-09-16T01:34:24.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.919+0000 7fabc92f8640 1 Processor -- start 2024-09-16T01:34:24.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.919+0000 7fabc92f8640 1 -- start start 2024-09-16T01:34:24.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.919+0000 7fabc92f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabb00039e0 0x7fabc4156510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:24.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.920+0000 7fabc92f8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fabc40cc4e0 0x7fabc415a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:24.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.920+0000 7fabc92f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabc415abe0 0x7fabc41f43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:24.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.920+0000 7fabc92f8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fabc415f200 con 0x7fabc40cc4e0 2024-09-16T01:34:24.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.920+0000 7fabc92f8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fabc406d3f0 con 0x7fabb00039e0 2024-09-16T01:34:24.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.920+0000 7fabc92f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fabc415f380 con 0x7fabc415abe0 2024-09-16T01:34:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.920+0000 7fabba7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fabc40cc4e0 0x7fabc415a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.920+0000 7fabba7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fabc40cc4e0 0x7fabc415a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58264/0 (socket says 172.21.15.88:58264) 2024-09-16T01:34:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.920+0000 7fabba7fc640 1 -- 172.21.15.88:0/3482044169 learned_addr learned my addr 172.21.15.88:0/3482044169 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.920+0000 7fabc37fe640 1 --2- 172.21.15.88:0/3482044169 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabc415abe0 0x7fabc41f43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.920+0000 7fabba7fc640 1 -- 172.21.15.88:0/3482044169 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabc415abe0 msgr2=0x7fabc41f43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.920+0000 7fabba7fc640 1 --2- 172.21.15.88:0/3482044169 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabc415abe0 0x7fabc41f43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.920+0000 7fabc2ffd640 1 --2- 172.21.15.88:0/3482044169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabb00039e0 0x7fabc4156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.920+0000 7fabba7fc640 1 -- 172.21.15.88:0/3482044169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabb00039e0 msgr2=0x7fabc4156510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.920+0000 7fabba7fc640 1 --2- 172.21.15.88:0/3482044169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabb00039e0 0x7fabc4156510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.920+0000 7fabba7fc640 1 -- 172.21.15.88:0/3482044169 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc4159600 con 0x7fabc40cc4e0 2024-09-16T01:34:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.920+0000 7fabc2ffd640 1 --2- 172.21.15.88:0/3482044169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabb00039e0 0x7fabc4156510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.920+0000 7fabba7fc640 1 --2- 172.21.15.88:0/3482044169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fabc40cc4e0 0x7fabc415a6a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fabac0072c0 tx=0x7fabac007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.920+0000 7fabbbfff640 1 -- 172.21.15.88:0/3482044169 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabac001660 con 0x7fabc40cc4e0 2024-09-16T01:34:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.920+0000 7fabbbfff640 1 -- 172.21.15.88:0/3482044169 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabac018070 con 0x7fabc40cc4e0 2024-09-16T01:34:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.920+0000 7fabbbfff640 1 -- 172.21.15.88:0/3482044169 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabac00fb80 con 0x7fabc40cc4e0 2024-09-16T01:34:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.920+0000 7fabc92f8640 1 -- 172.21.15.88:0/3482044169 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabc41598e0 con 0x7fabc40cc4e0 2024-09-16T01:34:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.920+0000 7fabc92f8640 1 -- 172.21.15.88:0/3482044169 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabc4159ea0 con 0x7fabc40cc4e0 2024-09-16T01:34:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.921+0000 7fabc92f8640 1 -- 172.21.15.88:0/3482044169 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabc4161ed0 con 0x7fabc40cc4e0 2024-09-16T01:34:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.922+0000 7fabbbfff640 1 -- 172.21.15.88:0/3482044169 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fabac00fd20 con 0x7fabc40cc4e0 2024-09-16T01:34:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.922+0000 7fabbbfff640 1 --2- 172.21.15.88:0/3482044169 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faba406c450 0x7faba406e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.922+0000 7fabbbfff640 1 -- 172.21.15.88:0/3482044169 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fabac08b450 con 0x7fabc40cc4e0 2024-09-16T01:34:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.922+0000 7fabc2ffd640 1 --2- 172.21.15.88:0/3482044169 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faba406c450 0x7faba406e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.922+0000 7fabc2ffd640 1 --2- 172.21.15.88:0/3482044169 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faba406c450 0x7faba406e910 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fabb4032000 tx=0x7fabb402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:24.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:24.926+0000 7fabbbfff640 1 -- 172.21.15.88:0/3482044169 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fabac059c50 con 0x7fabc40cc4e0 2024-09-16T01:34:25.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.088+0000 7fabc92f8640 1 -- 172.21.15.88:0/3482044169 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fabc415eaf0 con 0x7faba406c450 2024-09-16T01:34:25.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.094+0000 7fabbbfff640 1 -- 172.21.15.88:0/3482044169 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+241108 (secure 0 0 0) 0x7fabc415eaf0 con 0x7faba406c450 2024-09-16T01:34:25.098 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:25.099 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:34:25.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.097+0000 7fabc92f8640 1 -- 172.21.15.88:0/3482044169 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faba406c450 msgr2=0x7faba406e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:25.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.097+0000 7fabc92f8640 1 --2- 172.21.15.88:0/3482044169 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faba406c450 0x7faba406e910 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fabb4032000 tx=0x7fabb402e000 comp rx=0 tx=0).stop 2024-09-16T01:34:25.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.097+0000 7fabc92f8640 1 -- 172.21.15.88:0/3482044169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fabc40cc4e0 msgr2=0x7fabc415a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:25.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.097+0000 7fabc92f8640 1 --2- 172.21.15.88:0/3482044169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fabc40cc4e0 0x7fabc415a6a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fabac0072c0 tx=0x7fabac007780 comp rx=0 tx=0).stop 2024-09-16T01:34:25.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.097+0000 7fabc92f8640 1 -- 172.21.15.88:0/3482044169 shutdown_connections 2024-09-16T01:34:25.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.097+0000 7fabc92f8640 1 --2- 172.21.15.88:0/3482044169 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faba406c450 0x7faba406e910 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.097+0000 7fabc92f8640 1 --2- 172.21.15.88:0/3482044169 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabc415abe0 0x7fabc41f43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.097+0000 7fabc92f8640 1 --2- 172.21.15.88:0/3482044169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fabc40cc4e0 0x7fabc415a6a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.097+0000 7fabc92f8640 1 --2- 172.21.15.88:0/3482044169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabb00039e0 0x7fabc4156510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.097+0000 7fabc92f8640 1 -- 172.21.15.88:0/3482044169 >> 172.21.15.88:0/3482044169 conn(0x7fabc40b4240 msgr2=0x7fabc415cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:25.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.097+0000 7fabc92f8640 1 -- 172.21.15.88:0/3482044169 shutdown_connections 2024-09-16T01:34:25.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.097+0000 7fabc92f8640 1 -- 172.21.15.88:0/3482044169 wait complete. 2024-09-16T01:34:25.109 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-16T01:34:24.517440+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218888,"kb_used_data":2464,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755832,"statfs":{"total":773094113280,"available":772869971968,"internally_reserved":0,"allocated":2523136,"data_stored":1580914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.350270"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.181767+0000","last_change":"2024-09-16T01:34:21.181767+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.181767+0000","last_undegraded":"2024-09-16T01:34:21.181767+0000","last_fullsized":"2024-09-16T01:34:21.181767+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.179909+0000","last_change":"2024-09-16T01:34:21.179909+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.179909+0000","last_undegraded":"2024-09-16T01:34:21.179909+0000","last_fullsized":"2024-09-16T01:34:21.179909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.179665+0000","last_change":"2024-09-16T01:34:21.179665+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.179665+0000","last_undegraded":"2024-09-16T01:34:21.179665+0000","last_fullsized":"2024-09-16T01:34:21.179665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.178542+0000","last_change":"2024-09-16T01:34:21.178542+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.178542+0000","last_undegraded":"2024-09-16T01:34:21.178542+0000","last_fullsized":"2024-09-16T01:34:21.178542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.179261+0000","last_change":"2024-09-16T01:34:21.179261+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.179261+0000","last_undegraded":"2024-09-16T01:34:21.179261+0000","last_fullsized":"2024-09-16T01:34:21.179261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.179034+0000","last_change":"2024-09-16T01:34:21.179034+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.179034+0000","last_undegraded":"2024-09-16T01:34:21.179034+0000","last_fullsized":"2024-09-16T01:34:21.179034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.178570+0000","last_change":"2024-09-16T01:34:21.178570+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.178570+0000","last_undegraded":"2024-09-16T01:34:21.178570+0000","last_fullsized":"2024-09-16T01:34:21.178570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.174385+0000","last_change":"2024-09-16T01:34:20.174385+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.174385+0000","last_undegraded":"2024-09-16T01:34:20.174385+0000","last_fullsized":"2024-09-16T01:34:20.174385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170457+0000","last_change":"2024-09-16T01:34:20.170457+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170457+0000","last_undegraded":"2024-09-16T01:34:20.170457+0000","last_fullsized":"2024-09-16T01:34:20.170457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.174351+0000","last_change":"2024-09-16T01:34:20.174351+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.174351+0000","last_undegraded":"2024-09-16T01:34:20.174351+0000","last_fullsized":"2024-09-16T01:34:20.174351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T01:34:21.194181+0000","last_change":"2024-09-16T01:34:21.188039+0000","last_active":"2024-09-16T01:34:21.194181+0000","last_peered":"2024-09-16T01:34:21.194181+0000","last_clean":"2024-09-16T01:34:21.194181+0000","last_became_active":"2024-09-16T01:34:21.187743+0000","last_became_peered":"2024-09-16T01:34:21.187743+0000","last_unstale":"2024-09-16T01:34:21.194181+0000","last_undegraded":"2024-09-16T01:34:21.194181+0000","last_fullsized":"2024-09-16T01:34:21.194181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:09:58.843092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.172066+0000","last_change":"2024-09-16T01:34:20.172066+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.172066+0000","last_undegraded":"2024-09-16T01:34:20.172066+0000","last_fullsized":"2024-09-16T01:34:20.172066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.177301+0000","last_change":"2024-09-16T01:34:21.177301+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.177301+0000","last_undegraded":"2024-09-16T01:34:21.177301+0000","last_fullsized":"2024-09-16T01:34:21.177301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T01:34:21.197129+0000","last_change":"2024-09-16T01:34:21.190619+0000","last_active":"2024-09-16T01:34:21.197129+0000","last_peered":"2024-09-16T01:34:21.197129+0000","last_clean":"2024-09-16T01:34:21.197129+0000","last_became_active":"2024-09-16T01:34:21.188943+0000","last_became_peered":"2024-09-16T01:34:21.188943+0000","last_unstale":"2024-09-16T01:34:21.197129+0000","last_undegraded":"2024-09-16T01:34:21.197129+0000","last_fullsized":"2024-09-16T01:34:21.197129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:07:21.563057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.171266+0000","last_change":"2024-09-16T01:34:20.171266+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.171266+0000","last_undegraded":"2024-09-16T01:34:20.171266+0000","last_fullsized":"2024-09-16T01:34:20.171266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170974+0000","last_change":"2024-09-16T01:34:20.170974+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170974+0000","last_undegraded":"2024-09-16T01:34:20.170974+0000","last_fullsized":"2024-09-16T01:34:20.170974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.174386+0000","last_change":"2024-09-16T01:34:20.174386+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.174386+0000","last_undegraded":"2024-09-16T01:34:20.174386+0000","last_fullsized":"2024-09-16T01:34:20.174386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T01:34:21.187991+0000","last_change":"2024-09-16T01:34:21.186203+0000","last_active":"2024-09-16T01:34:21.187991+0000","last_peered":"2024-09-16T01:34:21.187991+0000","last_clean":"2024-09-16T01:34:21.187991+0000","last_became_active":"2024-09-16T01:34:21.185935+0000","last_became_peered":"2024-09-16T01:34:21.185935+0000","last_unstale":"2024-09-16T01:34:21.187991+0000","last_undegraded":"2024-09-16T01:34:21.187991+0000","last_fullsized":"2024-09-16T01:34:21.187991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:56:19.306291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170843+0000","last_change":"2024-09-16T01:34:20.170843+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170843+0000","last_undegraded":"2024-09-16T01:34:20.170843+0000","last_fullsized":"2024-09-16T01:34:20.170843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.179961+0000","last_change":"2024-09-16T01:34:20.179961+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.179961+0000","last_undegraded":"2024-09-16T01:34:20.179961+0000","last_fullsized":"2024-09-16T01:34:20.179961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.172767+0000","last_change":"2024-09-16T01:34:20.172767+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.172767+0000","last_undegraded":"2024-09-16T01:34:20.172767+0000","last_fullsized":"2024-09-16T01:34:20.172767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.179968+0000","last_change":"2024-09-16T01:34:20.179968+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.179968+0000","last_undegraded":"2024-09-16T01:34:20.179968+0000","last_fullsized":"2024-09-16T01:34:20.179968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170553+0000","last_change":"2024-09-16T01:34:20.170553+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170553+0000","last_undegraded":"2024-09-16T01:34:20.170553+0000","last_fullsized":"2024-09-16T01:34:20.170553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.176558+0000","last_change":"2024-09-16T01:34:21.176558+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.176558+0000","last_undegraded":"2024-09-16T01:34:21.176558+0000","last_fullsized":"2024-09-16T01:34:21.176558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.179920+0000","last_change":"2024-09-16T01:34:20.179920+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.179920+0000","last_undegraded":"2024-09-16T01:34:20.179920+0000","last_fullsized":"2024-09-16T01:34:20.179920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T01:34:20.171425+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:34:20.171425+0000","last_peered":"2024-09-16T01:34:20.171425+0000","last_clean":"2024-09-16T01:34:20.171425+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:34:20.171425+0000","last_undegraded":"2024-09-16T01:34:20.171425+0000","last_fullsized":"2024-09-16T01:34:20.171425+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T01:34:21.187731+0000","last_change":"2024-09-16T01:34:21.183119+0000","last_active":"2024-09-16T01:34:21.187731+0000","last_peered":"2024-09-16T01:34:21.187731+0000","last_clean":"2024-09-16T01:34:21.187731+0000","last_became_active":"2024-09-16T01:34:21.182740+0000","last_became_peered":"2024-09-16T01:34:21.182740+0000","last_unstale":"2024-09-16T01:34:21.187731+0000","last_undegraded":"2024-09-16T01:34:21.187731+0000","last_fullsized":"2024-09-16T01:34:21.187731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:49.205242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.169473+0000","last_change":"2024-09-16T01:34:20.169473+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.169473+0000","last_undegraded":"2024-09-16T01:34:20.169473+0000","last_fullsized":"2024-09-16T01:34:20.169473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.179953+0000","last_change":"2024-09-16T01:34:20.179953+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.179953+0000","last_undegraded":"2024-09-16T01:34:20.179953+0000","last_fullsized":"2024-09-16T01:34:20.179953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T01:34:21.197145+0000","last_change":"2024-09-16T01:34:21.190750+0000","last_active":"2024-09-16T01:34:21.197145+0000","last_peered":"2024-09-16T01:34:21.197145+0000","last_clean":"2024-09-16T01:34:21.197145+0000","last_became_active":"2024-09-16T01:34:21.189215+0000","last_became_peered":"2024-09-16T01:34:21.189215+0000","last_unstale":"2024-09-16T01:34:21.197145+0000","last_undegraded":"2024-09-16T01:34:21.197145+0000","last_fullsized":"2024-09-16T01:34:21.197145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:09:25.203416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170569+0000","last_change":"2024-09-16T01:34:20.170569+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170569+0000","last_undegraded":"2024-09-16T01:34:20.170569+0000","last_fullsized":"2024-09-16T01:34:20.170569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.176775+0000","last_change":"2024-09-16T01:34:21.176775+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.176775+0000","last_undegraded":"2024-09-16T01:34:21.176775+0000","last_fullsized":"2024-09-16T01:34:21.176775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.169491+0000","last_change":"2024-09-16T01:34:20.169491+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.169491+0000","last_undegraded":"2024-09-16T01:34:20.169491+0000","last_fullsized":"2024-09-16T01:34:20.169491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170396+0000","last_change":"2024-09-16T01:34:20.170396+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170396+0000","last_undegraded":"2024-09-16T01:34:20.170396+0000","last_fullsized":"2024-09-16T01:34:20.170396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.171160+0000","last_change":"2024-09-16T01:34:20.171160+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.171160+0000","last_undegraded":"2024-09-16T01:34:20.171160+0000","last_fullsized":"2024-09-16T01:34:20.171160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170346+0000","last_change":"2024-09-16T01:34:20.170346+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170346+0000","last_undegraded":"2024-09-16T01:34:20.170346+0000","last_fullsized":"2024-09-16T01:34:20.170346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170547+0000","last_change":"2024-09-16T01:34:20.170547+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170547+0000","last_undegraded":"2024-09-16T01:34:20.170547+0000","last_fullsized":"2024-09-16T01:34:20.170547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170325+0000","last_change":"2024-09-16T01:34:20.170325+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170325+0000","last_undegraded":"2024-09-16T01:34:20.170325+0000","last_fullsized":"2024-09-16T01:34:20.170325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.169501+0000","last_change":"2024-09-16T01:34:20.169501+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.169501+0000","last_undegraded":"2024-09-16T01:34:20.169501+0000","last_fullsized":"2024-09-16T01:34:20.169501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.177698+0000","last_change":"2024-09-16T01:34:21.177698+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.177698+0000","last_undegraded":"2024-09-16T01:34:21.177698+0000","last_fullsized":"2024-09-16T01:34:21.177698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.169483+0000","last_change":"2024-09-16T01:34:20.169483+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.169483+0000","last_undegraded":"2024-09-16T01:34:20.169483+0000","last_fullsized":"2024-09-16T01:34:20.169483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.177680+0000","last_change":"2024-09-16T01:34:21.177680+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.177680+0000","last_undegraded":"2024-09-16T01:34:21.177680+0000","last_fullsized":"2024-09-16T01:34:21.177680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170583+0000","last_change":"2024-09-16T01:34:20.170583+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170583+0000","last_undegraded":"2024-09-16T01:34:20.170583+0000","last_fullsized":"2024-09-16T01:34:20.170583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.174438+0000","last_change":"2024-09-16T01:34:20.174438+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.174438+0000","last_undegraded":"2024-09-16T01:34:20.174438+0000","last_fullsized":"2024-09-16T01:34:20.174438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.174390+0000","last_change":"2024-09-16T01:34:20.174390+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.174390+0000","last_undegraded":"2024-09-16T01:34:20.174390+0000","last_fullsized":"2024-09-16T01:34:20.174390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":1},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":176128,"data_stored":54385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":757760,"data_stored":638936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":48568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":48568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":48568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":48568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":163840,"data_stored":48568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542148,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":770048,"data_stored":644753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:34:25.113 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:34:25.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.247+0000 7f6535731640 1 Processor -- start 2024-09-16T01:34:25.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.247+0000 7f6535731640 1 -- start start 2024-09-16T01:34:25.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.248+0000 7f6535731640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f65301475f0 0x7f65300afd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:25.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.248+0000 7f6535731640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f653006c320 0x7f65300723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:25.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.248+0000 7f6535731640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6530162440 con 0x7f6530147ae0 2024-09-16T01:34:25.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.248+0000 7f6535731640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6530162720 con 0x7f65301475f0 2024-09-16T01:34:25.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.248+0000 7f6535731640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6530162a00 con 0x7f653006c320 2024-09-16T01:34:25.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.248+0000 7f652effd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f65301475f0 0x7f65300afd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:25.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.248+0000 7f652effd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f65301475f0 0x7f65300afd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58274/0 (socket says 172.21.15.88:58274) 2024-09-16T01:34:25.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.248+0000 7f652effd640 1 -- 172.21.15.88:0/4144060501 learned_addr learned my addr 172.21.15.88:0/4144060501 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:25.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.248+0000 7f652effd640 1 -- 172.21.15.88:0/4144060501 >> v2:172.21.15.154:3300/0 conn(0x7f653006c320 msgr2=0x7f65300723d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:34:25.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.248+0000 7f652effd640 1 --2- 172.21.15.88:0/4144060501 >> v2:172.21.15.154:3300/0 conn(0x7f653006c320 0x7f65300723d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.248+0000 7f652effd640 1 -- 172.21.15.88:0/4144060501 >> v1:172.21.15.154:6789/0 conn(0x7f6530147ae0 legacy=0x7f653006bc10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:34:25.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.248+0000 7f652effd640 1 -- 172.21.15.88:0/4144060501 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6530162ce0 con 0x7f65301475f0 2024-09-16T01:34:25.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.249+0000 7f652effd640 1 --2- 172.21.15.88:0/4144060501 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65301475f0 0x7f65300afd40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6530075ae0 tx=0x7f652002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=920d8ecf28019631 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:25.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.249+0000 7f652dffb640 1 -- 172.21.15.88:0/4144060501 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6520037070 con 0x7f65301475f0 2024-09-16T01:34:25.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.249+0000 7f652dffb640 1 -- 172.21.15.88:0/4144060501 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6520032440 con 0x7f65301475f0 2024-09-16T01:34:25.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.249+0000 7f652dffb640 1 -- 172.21.15.88:0/4144060501 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6520032810 con 0x7f65301475f0 2024-09-16T01:34:25.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.250+0000 7f6535731640 1 -- 172.21.15.88:0/4144060501 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65301475f0 msgr2=0x7f65300afd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:25.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.250+0000 7f6535731640 1 --2- 172.21.15.88:0/4144060501 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65301475f0 0x7f65300afd40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6530075ae0 tx=0x7f652002a250 comp rx=0 tx=0).stop 2024-09-16T01:34:25.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.250+0000 7f6535731640 1 -- 172.21.15.88:0/4144060501 shutdown_connections 2024-09-16T01:34:25.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.250+0000 7f6535731640 1 --2- 172.21.15.88:0/4144060501 >> v2:172.21.15.154:3300/0 conn(0x7f653006c320 0x7f65300723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.250+0000 7f6535731640 1 --2- 172.21.15.88:0/4144060501 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f65301475f0 0x7f65300afd40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.250+0000 7f6535731640 1 -- 172.21.15.88:0/4144060501 >> 172.21.15.88:0/4144060501 conn(0x7f653006e170 msgr2=0x7f653006e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:25.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.250+0000 7f6535731640 1 -- 172.21.15.88:0/4144060501 shutdown_connections 2024-09-16T01:34:25.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.250+0000 7f6535731640 1 -- 172.21.15.88:0/4144060501 wait complete. 2024-09-16T01:34:25.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.251+0000 7f6535731640 1 Processor -- start 2024-09-16T01:34:25.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.251+0000 7f6535731640 1 -- start start 2024-09-16T01:34:25.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.252+0000 7f6535731640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f653006c320 0x7f65300b22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:25.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.252+0000 7f6535731640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65301475f0 0x7f65301600d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:25.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.252+0000 7f6535731640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6530147ae0 0x7f6530160610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:25.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.252+0000 7f6535731640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6530072a90 con 0x7f653006c320 2024-09-16T01:34:25.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.252+0000 7f6535731640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6530072910 con 0x7f65301475f0 2024-09-16T01:34:25.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.252+0000 7f6535731640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6530072c10 con 0x7f6530147ae0 2024-09-16T01:34:25.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.252+0000 7f652effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f653006c320 0x7f65300b22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:25.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.252+0000 7f652effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f653006c320 0x7f65300b22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58276/0 (socket says 172.21.15.88:58276) 2024-09-16T01:34:25.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.252+0000 7f652effd640 1 -- 172.21.15.88:0/4217938489 learned_addr learned my addr 172.21.15.88:0/4217938489 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:25.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.252+0000 7f652effd640 1 -- 172.21.15.88:0/4217938489 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6530147ae0 msgr2=0x7f6530160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:25.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.252+0000 7f652effd640 1 --2- 172.21.15.88:0/4217938489 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6530147ae0 0x7f6530160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.252+0000 7f652effd640 1 -- 172.21.15.88:0/4217938489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65301475f0 msgr2=0x7f65301600d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:25.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.252+0000 7f652effd640 1 --2- 172.21.15.88:0/4217938489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65301475f0 0x7f65301600d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.252+0000 7f652effd640 1 -- 172.21.15.88:0/4217938489 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f653015eda0 con 0x7f653006c320 2024-09-16T01:34:25.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.252+0000 7f652effd640 1 --2- 172.21.15.88:0/4217938489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f653006c320 0x7f65300b22a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f65200090b0 tx=0x7f65200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:25.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.253+0000 7f65177fe640 1 -- 172.21.15.88:0/4217938489 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6520041070 con 0x7f653006c320 2024-09-16T01:34:25.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.253+0000 7f65177fe640 1 -- 172.21.15.88:0/4217938489 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f652003c870 con 0x7f653006c320 2024-09-16T01:34:25.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.253+0000 7f6535731640 1 -- 172.21.15.88:0/4217938489 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f653015f0e0 con 0x7f653006c320 2024-09-16T01:34:25.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.253+0000 7f65177fe640 1 -- 172.21.15.88:0/4217938489 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6520037040 con 0x7f653006c320 2024-09-16T01:34:25.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.253+0000 7f6535731640 1 -- 172.21.15.88:0/4217938489 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f653015f920 con 0x7f653006c320 2024-09-16T01:34:25.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.254+0000 7f6535731640 1 -- 172.21.15.88:0/4217938489 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6530076170 con 0x7f653006c320 2024-09-16T01:34:25.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.254+0000 7f65177fe640 1 -- 172.21.15.88:0/4217938489 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6520044070 con 0x7f653006c320 2024-09-16T01:34:25.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.254+0000 7f65177fe640 1 --2- 172.21.15.88:0/4217938489 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f651c06c2b0 0x7f651c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:25.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.255+0000 7f65177fe640 1 -- 172.21.15.88:0/4217938489 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f65200ac140 con 0x7f653006c320 2024-09-16T01:34:25.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.255+0000 7f652e7fc640 1 --2- 172.21.15.88:0/4217938489 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f651c06c2b0 0x7f651c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:25.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.255+0000 7f652e7fc640 1 --2- 172.21.15.88:0/4217938489 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f651c06c2b0 0x7f651c06e770 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f65300b32e0 tx=0x7f6524006ba0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.259+0000 7f65177fe640 1 -- 172.21.15.88:0/4217938489 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6530076170 con 0x7f653006c320 2024-09-16T01:34:25.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.421+0000 7f6535731640 1 -- 172.21.15.88:0/4217938489 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f653015fc60 con 0x7f651c06c2b0 2024-09-16T01:34:25.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.426+0000 7f65177fe640 1 -- 172.21.15.88:0/4217938489 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+241108 (secure 0 0 0) 0x7f653015fc60 con 0x7f651c06c2b0 2024-09-16T01:34:25.429 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:25.430 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:34:25.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.429+0000 7f6535731640 1 -- 172.21.15.88:0/4217938489 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f651c06c2b0 msgr2=0x7f651c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:25.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.429+0000 7f6535731640 1 --2- 172.21.15.88:0/4217938489 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f651c06c2b0 0x7f651c06e770 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f65300b32e0 tx=0x7f6524006ba0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.429+0000 7f6535731640 1 -- 172.21.15.88:0/4217938489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f653006c320 msgr2=0x7f65300b22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:25.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.429+0000 7f6535731640 1 --2- 172.21.15.88:0/4217938489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f653006c320 0x7f65300b22a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f65200090b0 tx=0x7f65200046c0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.429+0000 7f6535731640 1 -- 172.21.15.88:0/4217938489 shutdown_connections 2024-09-16T01:34:25.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.429+0000 7f6535731640 1 --2- 172.21.15.88:0/4217938489 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f651c06c2b0 0x7f651c06e770 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.429+0000 7f6535731640 1 --2- 172.21.15.88:0/4217938489 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6530147ae0 0x7f6530160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.429+0000 7f6535731640 1 --2- 172.21.15.88:0/4217938489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65301475f0 0x7f65301600d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.429+0000 7f6535731640 1 --2- 172.21.15.88:0/4217938489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f653006c320 0x7f65300b22a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.429+0000 7f6535731640 1 -- 172.21.15.88:0/4217938489 >> 172.21.15.88:0/4217938489 conn(0x7f653006e170 msgr2=0x7f65300b0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:25.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.429+0000 7f6535731640 1 -- 172.21.15.88:0/4217938489 shutdown_connections 2024-09-16T01:34:25.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.429+0000 7f6535731640 1 -- 172.21.15.88:0/4217938489 wait complete. 2024-09-16T01:34:25.441 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-16T01:34:24.517440+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218888,"kb_used_data":2464,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755832,"statfs":{"total":773094113280,"available":772869971968,"internally_reserved":0,"allocated":2523136,"data_stored":1580914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.350270"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.181767+0000","last_change":"2024-09-16T01:34:21.181767+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.181767+0000","last_undegraded":"2024-09-16T01:34:21.181767+0000","last_fullsized":"2024-09-16T01:34:21.181767+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.179909+0000","last_change":"2024-09-16T01:34:21.179909+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.179909+0000","last_undegraded":"2024-09-16T01:34:21.179909+0000","last_fullsized":"2024-09-16T01:34:21.179909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.179665+0000","last_change":"2024-09-16T01:34:21.179665+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.179665+0000","last_undegraded":"2024-09-16T01:34:21.179665+0000","last_fullsized":"2024-09-16T01:34:21.179665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.178542+0000","last_change":"2024-09-16T01:34:21.178542+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.178542+0000","last_undegraded":"2024-09-16T01:34:21.178542+0000","last_fullsized":"2024-09-16T01:34:21.178542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.179261+0000","last_change":"2024-09-16T01:34:21.179261+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.179261+0000","last_undegraded":"2024-09-16T01:34:21.179261+0000","last_fullsized":"2024-09-16T01:34:21.179261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.179034+0000","last_change":"2024-09-16T01:34:21.179034+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.179034+0000","last_undegraded":"2024-09-16T01:34:21.179034+0000","last_fullsized":"2024-09-16T01:34:21.179034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.178570+0000","last_change":"2024-09-16T01:34:21.178570+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.178570+0000","last_undegraded":"2024-09-16T01:34:21.178570+0000","last_fullsized":"2024-09-16T01:34:21.178570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.174385+0000","last_change":"2024-09-16T01:34:20.174385+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.174385+0000","last_undegraded":"2024-09-16T01:34:20.174385+0000","last_fullsized":"2024-09-16T01:34:20.174385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170457+0000","last_change":"2024-09-16T01:34:20.170457+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170457+0000","last_undegraded":"2024-09-16T01:34:20.170457+0000","last_fullsized":"2024-09-16T01:34:20.170457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.174351+0000","last_change":"2024-09-16T01:34:20.174351+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.174351+0000","last_undegraded":"2024-09-16T01:34:20.174351+0000","last_fullsized":"2024-09-16T01:34:20.174351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T01:34:21.194181+0000","last_change":"2024-09-16T01:34:21.188039+0000","last_active":"2024-09-16T01:34:21.194181+0000","last_peered":"2024-09-16T01:34:21.194181+0000","last_clean":"2024-09-16T01:34:21.194181+0000","last_became_active":"2024-09-16T01:34:21.187743+0000","last_became_peered":"2024-09-16T01:34:21.187743+0000","last_unstale":"2024-09-16T01:34:21.194181+0000","last_undegraded":"2024-09-16T01:34:21.194181+0000","last_fullsized":"2024-09-16T01:34:21.194181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:09:58.843092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.172066+0000","last_change":"2024-09-16T01:34:20.172066+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.172066+0000","last_undegraded":"2024-09-16T01:34:20.172066+0000","last_fullsized":"2024-09-16T01:34:20.172066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.177301+0000","last_change":"2024-09-16T01:34:21.177301+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.177301+0000","last_undegraded":"2024-09-16T01:34:21.177301+0000","last_fullsized":"2024-09-16T01:34:21.177301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T01:34:21.197129+0000","last_change":"2024-09-16T01:34:21.190619+0000","last_active":"2024-09-16T01:34:21.197129+0000","last_peered":"2024-09-16T01:34:21.197129+0000","last_clean":"2024-09-16T01:34:21.197129+0000","last_became_active":"2024-09-16T01:34:21.188943+0000","last_became_peered":"2024-09-16T01:34:21.188943+0000","last_unstale":"2024-09-16T01:34:21.197129+0000","last_undegraded":"2024-09-16T01:34:21.197129+0000","last_fullsized":"2024-09-16T01:34:21.197129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:07:21.563057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.171266+0000","last_change":"2024-09-16T01:34:20.171266+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.171266+0000","last_undegraded":"2024-09-16T01:34:20.171266+0000","last_fullsized":"2024-09-16T01:34:20.171266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170974+0000","last_change":"2024-09-16T01:34:20.170974+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170974+0000","last_undegraded":"2024-09-16T01:34:20.170974+0000","last_fullsized":"2024-09-16T01:34:20.170974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.174386+0000","last_change":"2024-09-16T01:34:20.174386+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.174386+0000","last_undegraded":"2024-09-16T01:34:20.174386+0000","last_fullsized":"2024-09-16T01:34:20.174386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T01:34:21.187991+0000","last_change":"2024-09-16T01:34:21.186203+0000","last_active":"2024-09-16T01:34:21.187991+0000","last_peered":"2024-09-16T01:34:21.187991+0000","last_clean":"2024-09-16T01:34:21.187991+0000","last_became_active":"2024-09-16T01:34:21.185935+0000","last_became_peered":"2024-09-16T01:34:21.185935+0000","last_unstale":"2024-09-16T01:34:21.187991+0000","last_undegraded":"2024-09-16T01:34:21.187991+0000","last_fullsized":"2024-09-16T01:34:21.187991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:56:19.306291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170843+0000","last_change":"2024-09-16T01:34:20.170843+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170843+0000","last_undegraded":"2024-09-16T01:34:20.170843+0000","last_fullsized":"2024-09-16T01:34:20.170843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.179961+0000","last_change":"2024-09-16T01:34:20.179961+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.179961+0000","last_undegraded":"2024-09-16T01:34:20.179961+0000","last_fullsized":"2024-09-16T01:34:20.179961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.172767+0000","last_change":"2024-09-16T01:34:20.172767+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.172767+0000","last_undegraded":"2024-09-16T01:34:20.172767+0000","last_fullsized":"2024-09-16T01:34:20.172767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.179968+0000","last_change":"2024-09-16T01:34:20.179968+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.179968+0000","last_undegraded":"2024-09-16T01:34:20.179968+0000","last_fullsized":"2024-09-16T01:34:20.179968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170553+0000","last_change":"2024-09-16T01:34:20.170553+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170553+0000","last_undegraded":"2024-09-16T01:34:20.170553+0000","last_fullsized":"2024-09-16T01:34:20.170553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.176558+0000","last_change":"2024-09-16T01:34:21.176558+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.176558+0000","last_undegraded":"2024-09-16T01:34:21.176558+0000","last_fullsized":"2024-09-16T01:34:21.176558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.179920+0000","last_change":"2024-09-16T01:34:20.179920+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.179920+0000","last_undegraded":"2024-09-16T01:34:20.179920+0000","last_fullsized":"2024-09-16T01:34:20.179920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T01:34:20.171425+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:34:20.171425+0000","last_peered":"2024-09-16T01:34:20.171425+0000","last_clean":"2024-09-16T01:34:20.171425+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:34:20.171425+0000","last_undegraded":"2024-09-16T01:34:20.171425+0000","last_fullsized":"2024-09-16T01:34:20.171425+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T01:34:21.187731+0000","last_change":"2024-09-16T01:34:21.183119+0000","last_active":"2024-09-16T01:34:21.187731+0000","last_peered":"2024-09-16T01:34:21.187731+0000","last_clean":"2024-09-16T01:34:21.187731+0000","last_became_active":"2024-09-16T01:34:21.182740+0000","last_became_peered":"2024-09-16T01:34:21.182740+0000","last_unstale":"2024-09-16T01:34:21.187731+0000","last_undegraded":"2024-09-16T01:34:21.187731+0000","last_fullsized":"2024-09-16T01:34:21.187731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:49.205242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.169473+0000","last_change":"2024-09-16T01:34:20.169473+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.169473+0000","last_undegraded":"2024-09-16T01:34:20.169473+0000","last_fullsized":"2024-09-16T01:34:20.169473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.179953+0000","last_change":"2024-09-16T01:34:20.179953+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.179953+0000","last_undegraded":"2024-09-16T01:34:20.179953+0000","last_fullsized":"2024-09-16T01:34:20.179953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T01:34:21.197145+0000","last_change":"2024-09-16T01:34:21.190750+0000","last_active":"2024-09-16T01:34:21.197145+0000","last_peered":"2024-09-16T01:34:21.197145+0000","last_clean":"2024-09-16T01:34:21.197145+0000","last_became_active":"2024-09-16T01:34:21.189215+0000","last_became_peered":"2024-09-16T01:34:21.189215+0000","last_unstale":"2024-09-16T01:34:21.197145+0000","last_undegraded":"2024-09-16T01:34:21.197145+0000","last_fullsized":"2024-09-16T01:34:21.197145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:09:25.203416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170569+0000","last_change":"2024-09-16T01:34:20.170569+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170569+0000","last_undegraded":"2024-09-16T01:34:20.170569+0000","last_fullsized":"2024-09-16T01:34:20.170569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.176775+0000","last_change":"2024-09-16T01:34:21.176775+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.176775+0000","last_undegraded":"2024-09-16T01:34:21.176775+0000","last_fullsized":"2024-09-16T01:34:21.176775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.169491+0000","last_change":"2024-09-16T01:34:20.169491+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.169491+0000","last_undegraded":"2024-09-16T01:34:20.169491+0000","last_fullsized":"2024-09-16T01:34:20.169491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170396+0000","last_change":"2024-09-16T01:34:20.170396+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170396+0000","last_undegraded":"2024-09-16T01:34:20.170396+0000","last_fullsized":"2024-09-16T01:34:20.170396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.171160+0000","last_change":"2024-09-16T01:34:20.171160+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.171160+0000","last_undegraded":"2024-09-16T01:34:20.171160+0000","last_fullsized":"2024-09-16T01:34:20.171160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170346+0000","last_change":"2024-09-16T01:34:20.170346+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170346+0000","last_undegraded":"2024-09-16T01:34:20.170346+0000","last_fullsized":"2024-09-16T01:34:20.170346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170547+0000","last_change":"2024-09-16T01:34:20.170547+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170547+0000","last_undegraded":"2024-09-16T01:34:20.170547+0000","last_fullsized":"2024-09-16T01:34:20.170547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170325+0000","last_change":"2024-09-16T01:34:20.170325+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170325+0000","last_undegraded":"2024-09-16T01:34:20.170325+0000","last_fullsized":"2024-09-16T01:34:20.170325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.169501+0000","last_change":"2024-09-16T01:34:20.169501+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.169501+0000","last_undegraded":"2024-09-16T01:34:20.169501+0000","last_fullsized":"2024-09-16T01:34:20.169501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.177698+0000","last_change":"2024-09-16T01:34:21.177698+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.177698+0000","last_undegraded":"2024-09-16T01:34:21.177698+0000","last_fullsized":"2024-09-16T01:34:21.177698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.169483+0000","last_change":"2024-09-16T01:34:20.169483+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.169483+0000","last_undegraded":"2024-09-16T01:34:20.169483+0000","last_fullsized":"2024-09-16T01:34:20.169483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.177680+0000","last_change":"2024-09-16T01:34:21.177680+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.177680+0000","last_undegraded":"2024-09-16T01:34:21.177680+0000","last_fullsized":"2024-09-16T01:34:21.177680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170583+0000","last_change":"2024-09-16T01:34:20.170583+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170583+0000","last_undegraded":"2024-09-16T01:34:20.170583+0000","last_fullsized":"2024-09-16T01:34:20.170583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.174438+0000","last_change":"2024-09-16T01:34:20.174438+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.174438+0000","last_undegraded":"2024-09-16T01:34:20.174438+0000","last_fullsized":"2024-09-16T01:34:20.174438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.174390+0000","last_change":"2024-09-16T01:34:20.174390+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.174390+0000","last_undegraded":"2024-09-16T01:34:20.174390+0000","last_fullsized":"2024-09-16T01:34:20.174390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":1},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":176128,"data_stored":54385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":757760,"data_stored":638936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":48568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":48568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":48568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":48568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":163840,"data_stored":48568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542148,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":770048,"data_stored":644753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:34:25.445 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:34:25.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.580+0000 7fc9172eb640 1 Processor -- start 2024-09-16T01:34:25.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.580+0000 7fc9172eb640 1 -- start start 2024-09-16T01:34:25.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.581+0000 7fc9172eb640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9100ba240 0x7fc9100d3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:25.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.581+0000 7fc9172eb640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9100ba660 con 0x7fc91015e7a0 2024-09-16T01:34:25.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.581+0000 7fc9172eb640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9100d2f40 con 0x7fc9100cf2f0 2024-09-16T01:34:25.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.581+0000 7fc9172eb640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc9100d3220 con 0x7fc9100ba240 2024-09-16T01:34:25.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.581+0000 7fc915087640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc91015e7a0 0x7fc9100d2780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34264/0 (socket says 172.21.15.88:34264) 2024-09-16T01:34:25.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.581+0000 7fc915087640 1 -- 172.21.15.88:0/653918442 learned_addr learned my addr 172.21.15.88:0/653918442 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:25.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.581+0000 7fc914886640 1 --2- 172.21.15.88:0/653918442 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9100ba240 0x7fc9100d3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:25.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.581+0000 7fc907fff640 1 -- 172.21.15.88:0/653918442 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2250049989 0 0) 0x7fc9100ba660 con 0x7fc91015e7a0 2024-09-16T01:34:25.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.581+0000 7fc907fff640 1 -- 172.21.15.88:0/653918442 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8fc003520 con 0x7fc91015e7a0 2024-09-16T01:34:25.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.581+0000 7fc907fff640 1 -- 172.21.15.88:0/653918442 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 86486008 0 0) 0x7fc8fc003520 con 0x7fc91015e7a0 2024-09-16T01:34:25.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.581+0000 7fc907fff640 1 -- 172.21.15.88:0/653918442 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9100ba240 msgr2=0x7fc9100d3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:25.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.581+0000 7fc907fff640 1 --2- 172.21.15.88:0/653918442 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9100ba240 0x7fc9100d3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.581+0000 7fc907fff640 1 -- 172.21.15.88:0/653918442 >> v1:172.21.15.154:6789/0 conn(0x7fc9100cf2f0 legacy=0x7fc9100b7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:25.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.581+0000 7fc907fff640 1 -- 172.21.15.88:0/653918442 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9100d3500 con 0x7fc91015e7a0 2024-09-16T01:34:25.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.581+0000 7fc907fff640 1 -- 172.21.15.88:0/653918442 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fc90c001670 con 0x7fc91015e7a0 2024-09-16T01:34:25.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.581+0000 7fc907fff640 1 -- 172.21.15.88:0/653918442 >> v1:172.21.15.88:6789/0 conn(0x7fc91015e7a0 legacy=0x7fc9100d2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:25.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.582+0000 7fc907fff640 1 --2- 172.21.15.88:0/653918442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8fc0039e0 0x7fc8fc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:25.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.582+0000 7fc907fff640 1 --2- 172.21.15.88:0/653918442 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9100ba240 0x7fc8fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:25.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.582+0000 7fc907fff640 1 --2- 172.21.15.88:0/653918442 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8fc004d40 0x7fc8fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:25.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.582+0000 7fc907fff640 1 -- 172.21.15.88:0/653918442 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc9100d3220 con 0x7fc8fc004d40 2024-09-16T01:34:25.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.582+0000 7fc907fff640 1 -- 172.21.15.88:0/653918442 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc8fc00b9c0 con 0x7fc8fc0039e0 2024-09-16T01:34:25.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.582+0000 7fc907fff640 1 -- 172.21.15.88:0/653918442 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc8fc00bbc0 con 0x7fc9100ba240 2024-09-16T01:34:25.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.582+0000 7fc915888640 1 --2- 172.21.15.88:0/653918442 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8fc004d40 0x7fc8fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:25.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.582+0000 7fc914886640 1 --2- 172.21.15.88:0/653918442 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9100ba240 0x7fc8fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:25.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.582+0000 7fc915087640 1 --2- 172.21.15.88:0/653918442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8fc0039e0 0x7fc8fc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:25.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.582+0000 7fc915888640 1 -- 172.21.15.88:0/653918442 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9100ba240 msgr2=0x7fc8fc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:25.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.582+0000 7fc915888640 1 --2- 172.21.15.88:0/653918442 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9100ba240 0x7fc8fc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.582+0000 7fc915888640 1 -- 172.21.15.88:0/653918442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8fc0039e0 msgr2=0x7fc8fc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:25.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.582+0000 7fc915888640 1 --2- 172.21.15.88:0/653918442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8fc0039e0 0x7fc8fc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.582+0000 7fc915888640 1 -- 172.21.15.88:0/653918442 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8fc003520 con 0x7fc8fc004d40 2024-09-16T01:34:25.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.583+0000 7fc915888640 1 --2- 172.21.15.88:0/653918442 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8fc004d40 0x7fc8fc0051c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc908001f60 tx=0x7fc8fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8235e1a7360e0061 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:25.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.583+0000 7fc907fff640 1 -- 172.21.15.88:0/653918442 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc908001b30 con 0x7fc8fc004d40 2024-09-16T01:34:25.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.583+0000 7fc907fff640 1 -- 172.21.15.88:0/653918442 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc908031620 con 0x7fc8fc004d40 2024-09-16T01:34:25.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.583+0000 7fc907fff640 1 -- 172.21.15.88:0/653918442 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc908031a10 con 0x7fc8fc004d40 2024-09-16T01:34:25.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.583+0000 7fc9172eb640 1 -- 172.21.15.88:0/653918442 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8fc004d40 msgr2=0x7fc8fc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:25.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.583+0000 7fc9172eb640 1 --2- 172.21.15.88:0/653918442 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8fc004d40 0x7fc8fc0051c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc908001f60 tx=0x7fc8fc005830 comp rx=0 tx=0).stop 2024-09-16T01:34:25.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.583+0000 7fc915888640 1 -- 172.21.15.88:0/653918442 reap_dead start 2024-09-16T01:34:25.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.583+0000 7fc9172eb640 1 -- 172.21.15.88:0/653918442 shutdown_connections 2024-09-16T01:34:25.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.583+0000 7fc9172eb640 1 -- 172.21.15.88:0/653918442 >> 172.21.15.88:0/653918442 conn(0x7fc9100d1550 msgr2=0x7fc9100d1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:25.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.583+0000 7fc9172eb640 1 -- 172.21.15.88:0/653918442 shutdown_connections 2024-09-16T01:34:25.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.583+0000 7fc9172eb640 1 -- 172.21.15.88:0/653918442 wait complete. 2024-09-16T01:34:25.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.584+0000 7fc9172eb640 1 Processor -- start 2024-09-16T01:34:25.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.585+0000 7fc9172eb640 1 -- start start 2024-09-16T01:34:25.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.585+0000 7fc9172eb640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8fc004d40 0x7fc91014e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:25.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.585+0000 7fc9172eb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc910151390 0x7fc9100ccc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:25.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.585+0000 7fc9172eb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9101f7630 0x7fc9100c9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:25.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.585+0000 7fc9172eb640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc91015eb80 con 0x7fc9101f7630 2024-09-16T01:34:25.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.585+0000 7fc9172eb640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc9100d3ac0 con 0x7fc910151390 2024-09-16T01:34:25.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.585+0000 7fc9172eb640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc91015ed00 con 0x7fc8fc004d40 2024-09-16T01:34:25.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.585+0000 7fc915888640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9101f7630 0x7fc9100c9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:25.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.585+0000 7fc915888640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9101f7630 0x7fc9100c9210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58294/0 (socket says 172.21.15.88:58294) 2024-09-16T01:34:25.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.585+0000 7fc915888640 1 -- 172.21.15.88:0/550263167 learned_addr learned my addr 172.21.15.88:0/550263167 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:25.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.585+0000 7fc915087640 1 --2- 172.21.15.88:0/550263167 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8fc004d40 0x7fc91014e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:25.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.585+0000 7fc914886640 1 --2- 172.21.15.88:0/550263167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc910151390 0x7fc9100ccc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:25.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.586+0000 7fc915888640 1 -- 172.21.15.88:0/550263167 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8fc004d40 msgr2=0x7fc91014e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:25.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.586+0000 7fc915888640 1 --2- 172.21.15.88:0/550263167 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8fc004d40 0x7fc91014e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.586+0000 7fc915888640 1 -- 172.21.15.88:0/550263167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc910151390 msgr2=0x7fc9100ccc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:25.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.586+0000 7fc915888640 1 --2- 172.21.15.88:0/550263167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc910151390 0x7fc9100ccc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.586+0000 7fc915888640 1 -- 172.21.15.88:0/550263167 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9101f7ff0 con 0x7fc9101f7630 2024-09-16T01:34:25.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.586+0000 7fc915888640 1 --2- 172.21.15.88:0/550263167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9101f7630 0x7fc9100c9210 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc90802aa00 tx=0x7fc9080015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:25.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.586+0000 7fc905ffb640 1 -- 172.21.15.88:0/550263167 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc908042070 con 0x7fc9101f7630 2024-09-16T01:34:25.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.586+0000 7fc905ffb640 1 -- 172.21.15.88:0/550263167 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9080317d0 con 0x7fc9101f7630 2024-09-16T01:34:25.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.586+0000 7fc9172eb640 1 -- 172.21.15.88:0/550263167 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9101f82d0 con 0x7fc9101f7630 2024-09-16T01:34:25.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.586+0000 7fc905ffb640 1 -- 172.21.15.88:0/550263167 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc908031b80 con 0x7fc9101f7630 2024-09-16T01:34:25.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.586+0000 7fc9172eb640 1 -- 172.21.15.88:0/550263167 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc91014dc50 con 0x7fc9101f7630 2024-09-16T01:34:25.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.587+0000 7fc9172eb640 1 -- 172.21.15.88:0/550263167 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9100d3fe0 con 0x7fc9101f7630 2024-09-16T01:34:25.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.587+0000 7fc905ffb640 1 -- 172.21.15.88:0/550263167 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc908045070 con 0x7fc9101f7630 2024-09-16T01:34:25.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.588+0000 7fc905ffb640 1 --2- 172.21.15.88:0/550263167 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc8f406c2b0 0x7fc8f406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:25.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.588+0000 7fc905ffb640 1 -- 172.21.15.88:0/550263167 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc9080ab490 con 0x7fc9101f7630 2024-09-16T01:34:25.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.588+0000 7fc915087640 1 --2- 172.21.15.88:0/550263167 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc8f406c2b0 0x7fc8f406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:25.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.588+0000 7fc915087640 1 --2- 172.21.15.88:0/550263167 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc8f406c2b0 0x7fc8f406e770 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc90c000c00 tx=0x7fc90c000f80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:25.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.592+0000 7fc905ffb640 1 -- 172.21.15.88:0/550263167 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc908079520 con 0x7fc9101f7630 2024-09-16T01:34:25.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.755+0000 7fc9172eb640 1 -- 172.21.15.88:0/550263167 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc91014de20 con 0x7fc8f406c2b0 2024-09-16T01:34:25.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.760+0000 7fc905ffb640 1 -- 172.21.15.88:0/550263167 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+241108 (secure 0 0 0) 0x7fc91014de20 con 0x7fc8f406c2b0 2024-09-16T01:34:25.764 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:25.765 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:34:25.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.763+0000 7fc9172eb640 1 -- 172.21.15.88:0/550263167 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc8f406c2b0 msgr2=0x7fc8f406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:25.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.763+0000 7fc9172eb640 1 --2- 172.21.15.88:0/550263167 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc8f406c2b0 0x7fc8f406e770 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc90c000c00 tx=0x7fc90c000f80 comp rx=0 tx=0).stop 2024-09-16T01:34:25.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.763+0000 7fc9172eb640 1 -- 172.21.15.88:0/550263167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9101f7630 msgr2=0x7fc9100c9210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:25.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.763+0000 7fc9172eb640 1 --2- 172.21.15.88:0/550263167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9101f7630 0x7fc9100c9210 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc90802aa00 tx=0x7fc9080015b0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.764+0000 7fc9172eb640 1 -- 172.21.15.88:0/550263167 shutdown_connections 2024-09-16T01:34:25.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.764+0000 7fc9172eb640 1 --2- 172.21.15.88:0/550263167 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc8f406c2b0 0x7fc8f406e770 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.764+0000 7fc9172eb640 1 --2- 172.21.15.88:0/550263167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9101f7630 0x7fc9100c9210 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.764+0000 7fc9172eb640 1 --2- 172.21.15.88:0/550263167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc910151390 0x7fc9100ccc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.764+0000 7fc9172eb640 1 --2- 172.21.15.88:0/550263167 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc8fc004d40 0x7fc91014e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.764+0000 7fc9172eb640 1 -- 172.21.15.88:0/550263167 >> 172.21.15.88:0/550263167 conn(0x7fc9100d1550 msgr2=0x7fc9100b7a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:25.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.764+0000 7fc9172eb640 1 -- 172.21.15.88:0/550263167 shutdown_connections 2024-09-16T01:34:25.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.764+0000 7fc9172eb640 1 -- 172.21.15.88:0/550263167 wait complete. 2024-09-16T01:34:25.775 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-16T01:34:24.517440+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218888,"kb_used_data":2464,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755832,"statfs":{"total":773094113280,"available":772869971968,"internally_reserved":0,"allocated":2523136,"data_stored":1580914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.350270"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.181767+0000","last_change":"2024-09-16T01:34:21.181767+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.181767+0000","last_undegraded":"2024-09-16T01:34:21.181767+0000","last_fullsized":"2024-09-16T01:34:21.181767+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.179909+0000","last_change":"2024-09-16T01:34:21.179909+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.179909+0000","last_undegraded":"2024-09-16T01:34:21.179909+0000","last_fullsized":"2024-09-16T01:34:21.179909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.179665+0000","last_change":"2024-09-16T01:34:21.179665+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.179665+0000","last_undegraded":"2024-09-16T01:34:21.179665+0000","last_fullsized":"2024-09-16T01:34:21.179665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.178542+0000","last_change":"2024-09-16T01:34:21.178542+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.178542+0000","last_undegraded":"2024-09-16T01:34:21.178542+0000","last_fullsized":"2024-09-16T01:34:21.178542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.179261+0000","last_change":"2024-09-16T01:34:21.179261+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.179261+0000","last_undegraded":"2024-09-16T01:34:21.179261+0000","last_fullsized":"2024-09-16T01:34:21.179261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.179034+0000","last_change":"2024-09-16T01:34:21.179034+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.179034+0000","last_undegraded":"2024-09-16T01:34:21.179034+0000","last_fullsized":"2024-09-16T01:34:21.179034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.178570+0000","last_change":"2024-09-16T01:34:21.178570+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.178570+0000","last_undegraded":"2024-09-16T01:34:21.178570+0000","last_fullsized":"2024-09-16T01:34:21.178570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.174385+0000","last_change":"2024-09-16T01:34:20.174385+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.174385+0000","last_undegraded":"2024-09-16T01:34:20.174385+0000","last_fullsized":"2024-09-16T01:34:20.174385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170457+0000","last_change":"2024-09-16T01:34:20.170457+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170457+0000","last_undegraded":"2024-09-16T01:34:20.170457+0000","last_fullsized":"2024-09-16T01:34:20.170457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.174351+0000","last_change":"2024-09-16T01:34:20.174351+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.174351+0000","last_undegraded":"2024-09-16T01:34:20.174351+0000","last_fullsized":"2024-09-16T01:34:20.174351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T01:34:21.194181+0000","last_change":"2024-09-16T01:34:21.188039+0000","last_active":"2024-09-16T01:34:21.194181+0000","last_peered":"2024-09-16T01:34:21.194181+0000","last_clean":"2024-09-16T01:34:21.194181+0000","last_became_active":"2024-09-16T01:34:21.187743+0000","last_became_peered":"2024-09-16T01:34:21.187743+0000","last_unstale":"2024-09-16T01:34:21.194181+0000","last_undegraded":"2024-09-16T01:34:21.194181+0000","last_fullsized":"2024-09-16T01:34:21.194181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:09:58.843092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.172066+0000","last_change":"2024-09-16T01:34:20.172066+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.172066+0000","last_undegraded":"2024-09-16T01:34:20.172066+0000","last_fullsized":"2024-09-16T01:34:20.172066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.177301+0000","last_change":"2024-09-16T01:34:21.177301+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.177301+0000","last_undegraded":"2024-09-16T01:34:21.177301+0000","last_fullsized":"2024-09-16T01:34:21.177301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T01:34:21.197129+0000","last_change":"2024-09-16T01:34:21.190619+0000","last_active":"2024-09-16T01:34:21.197129+0000","last_peered":"2024-09-16T01:34:21.197129+0000","last_clean":"2024-09-16T01:34:21.197129+0000","last_became_active":"2024-09-16T01:34:21.188943+0000","last_became_peered":"2024-09-16T01:34:21.188943+0000","last_unstale":"2024-09-16T01:34:21.197129+0000","last_undegraded":"2024-09-16T01:34:21.197129+0000","last_fullsized":"2024-09-16T01:34:21.197129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:07:21.563057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.171266+0000","last_change":"2024-09-16T01:34:20.171266+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.171266+0000","last_undegraded":"2024-09-16T01:34:20.171266+0000","last_fullsized":"2024-09-16T01:34:20.171266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170974+0000","last_change":"2024-09-16T01:34:20.170974+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170974+0000","last_undegraded":"2024-09-16T01:34:20.170974+0000","last_fullsized":"2024-09-16T01:34:20.170974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.174386+0000","last_change":"2024-09-16T01:34:20.174386+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.174386+0000","last_undegraded":"2024-09-16T01:34:20.174386+0000","last_fullsized":"2024-09-16T01:34:20.174386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T01:34:21.187991+0000","last_change":"2024-09-16T01:34:21.186203+0000","last_active":"2024-09-16T01:34:21.187991+0000","last_peered":"2024-09-16T01:34:21.187991+0000","last_clean":"2024-09-16T01:34:21.187991+0000","last_became_active":"2024-09-16T01:34:21.185935+0000","last_became_peered":"2024-09-16T01:34:21.185935+0000","last_unstale":"2024-09-16T01:34:21.187991+0000","last_undegraded":"2024-09-16T01:34:21.187991+0000","last_fullsized":"2024-09-16T01:34:21.187991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:56:19.306291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170843+0000","last_change":"2024-09-16T01:34:20.170843+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170843+0000","last_undegraded":"2024-09-16T01:34:20.170843+0000","last_fullsized":"2024-09-16T01:34:20.170843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.179961+0000","last_change":"2024-09-16T01:34:20.179961+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.179961+0000","last_undegraded":"2024-09-16T01:34:20.179961+0000","last_fullsized":"2024-09-16T01:34:20.179961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.172767+0000","last_change":"2024-09-16T01:34:20.172767+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.172767+0000","last_undegraded":"2024-09-16T01:34:20.172767+0000","last_fullsized":"2024-09-16T01:34:20.172767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.179968+0000","last_change":"2024-09-16T01:34:20.179968+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.179968+0000","last_undegraded":"2024-09-16T01:34:20.179968+0000","last_fullsized":"2024-09-16T01:34:20.179968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170553+0000","last_change":"2024-09-16T01:34:20.170553+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170553+0000","last_undegraded":"2024-09-16T01:34:20.170553+0000","last_fullsized":"2024-09-16T01:34:20.170553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.176558+0000","last_change":"2024-09-16T01:34:21.176558+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.176558+0000","last_undegraded":"2024-09-16T01:34:21.176558+0000","last_fullsized":"2024-09-16T01:34:21.176558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.179920+0000","last_change":"2024-09-16T01:34:20.179920+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.179920+0000","last_undegraded":"2024-09-16T01:34:20.179920+0000","last_fullsized":"2024-09-16T01:34:20.179920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T01:34:20.171425+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:34:20.171425+0000","last_peered":"2024-09-16T01:34:20.171425+0000","last_clean":"2024-09-16T01:34:20.171425+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:34:20.171425+0000","last_undegraded":"2024-09-16T01:34:20.171425+0000","last_fullsized":"2024-09-16T01:34:20.171425+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T01:34:21.187731+0000","last_change":"2024-09-16T01:34:21.183119+0000","last_active":"2024-09-16T01:34:21.187731+0000","last_peered":"2024-09-16T01:34:21.187731+0000","last_clean":"2024-09-16T01:34:21.187731+0000","last_became_active":"2024-09-16T01:34:21.182740+0000","last_became_peered":"2024-09-16T01:34:21.182740+0000","last_unstale":"2024-09-16T01:34:21.187731+0000","last_undegraded":"2024-09-16T01:34:21.187731+0000","last_fullsized":"2024-09-16T01:34:21.187731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:49.205242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.169473+0000","last_change":"2024-09-16T01:34:20.169473+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.169473+0000","last_undegraded":"2024-09-16T01:34:20.169473+0000","last_fullsized":"2024-09-16T01:34:20.169473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.179953+0000","last_change":"2024-09-16T01:34:20.179953+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.179953+0000","last_undegraded":"2024-09-16T01:34:20.179953+0000","last_fullsized":"2024-09-16T01:34:20.179953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T01:34:21.197145+0000","last_change":"2024-09-16T01:34:21.190750+0000","last_active":"2024-09-16T01:34:21.197145+0000","last_peered":"2024-09-16T01:34:21.197145+0000","last_clean":"2024-09-16T01:34:21.197145+0000","last_became_active":"2024-09-16T01:34:21.189215+0000","last_became_peered":"2024-09-16T01:34:21.189215+0000","last_unstale":"2024-09-16T01:34:21.197145+0000","last_undegraded":"2024-09-16T01:34:21.197145+0000","last_fullsized":"2024-09-16T01:34:21.197145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:09:25.203416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170569+0000","last_change":"2024-09-16T01:34:20.170569+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170569+0000","last_undegraded":"2024-09-16T01:34:20.170569+0000","last_fullsized":"2024-09-16T01:34:20.170569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.176775+0000","last_change":"2024-09-16T01:34:21.176775+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.176775+0000","last_undegraded":"2024-09-16T01:34:21.176775+0000","last_fullsized":"2024-09-16T01:34:21.176775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.169491+0000","last_change":"2024-09-16T01:34:20.169491+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.169491+0000","last_undegraded":"2024-09-16T01:34:20.169491+0000","last_fullsized":"2024-09-16T01:34:20.169491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170396+0000","last_change":"2024-09-16T01:34:20.170396+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170396+0000","last_undegraded":"2024-09-16T01:34:20.170396+0000","last_fullsized":"2024-09-16T01:34:20.170396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.171160+0000","last_change":"2024-09-16T01:34:20.171160+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.171160+0000","last_undegraded":"2024-09-16T01:34:20.171160+0000","last_fullsized":"2024-09-16T01:34:20.171160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170346+0000","last_change":"2024-09-16T01:34:20.170346+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170346+0000","last_undegraded":"2024-09-16T01:34:20.170346+0000","last_fullsized":"2024-09-16T01:34:20.170346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170547+0000","last_change":"2024-09-16T01:34:20.170547+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170547+0000","last_undegraded":"2024-09-16T01:34:20.170547+0000","last_fullsized":"2024-09-16T01:34:20.170547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170325+0000","last_change":"2024-09-16T01:34:20.170325+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170325+0000","last_undegraded":"2024-09-16T01:34:20.170325+0000","last_fullsized":"2024-09-16T01:34:20.170325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.169501+0000","last_change":"2024-09-16T01:34:20.169501+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.169501+0000","last_undegraded":"2024-09-16T01:34:20.169501+0000","last_fullsized":"2024-09-16T01:34:20.169501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.177698+0000","last_change":"2024-09-16T01:34:21.177698+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.177698+0000","last_undegraded":"2024-09-16T01:34:21.177698+0000","last_fullsized":"2024-09-16T01:34:21.177698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.169483+0000","last_change":"2024-09-16T01:34:20.169483+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.169483+0000","last_undegraded":"2024-09-16T01:34:20.169483+0000","last_fullsized":"2024-09-16T01:34:20.169483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.177680+0000","last_change":"2024-09-16T01:34:21.177680+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.177680+0000","last_undegraded":"2024-09-16T01:34:21.177680+0000","last_fullsized":"2024-09-16T01:34:21.177680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170583+0000","last_change":"2024-09-16T01:34:20.170583+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170583+0000","last_undegraded":"2024-09-16T01:34:20.170583+0000","last_fullsized":"2024-09-16T01:34:20.170583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.174438+0000","last_change":"2024-09-16T01:34:20.174438+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.174438+0000","last_undegraded":"2024-09-16T01:34:20.174438+0000","last_fullsized":"2024-09-16T01:34:20.174438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.174390+0000","last_change":"2024-09-16T01:34:20.174390+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.174390+0000","last_undegraded":"2024-09-16T01:34:20.174390+0000","last_fullsized":"2024-09-16T01:34:20.174390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":1},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":176128,"data_stored":54385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":757760,"data_stored":638936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":48568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":48568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":48568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":48568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":163840,"data_stored":48568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542148,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":770048,"data_stored":644753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:34:25.779 INFO:tasks.ceph.ceph_manager.ceph:PG 3.3f is not active+clean 2024-09-16T01:34:25.779 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.780 INFO:tasks.ceph.ceph_manager.ceph:PG 3.3e is not active+clean 2024-09-16T01:34:25.780 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.780 INFO:tasks.ceph.ceph_manager.ceph:PG 3.3d is not active+clean 2024-09-16T01:34:25.780 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.780 INFO:tasks.ceph.ceph_manager.ceph:PG 3.3c is not active+clean 2024-09-16T01:34:25.780 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 19, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:21.181767+0000', 'last_change': '2024-09-16T01:34:21.181767+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.181767+0000', 'last_undegraded': '2024-09-16T01:34:21.181767+0000', 'last_fullsized': '2024-09-16T01:34:21.181767+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T01:34:25.780 INFO:tasks.ceph.ceph_manager.ceph:PG 3.3b is not active+clean 2024-09-16T01:34:25.780 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.780 INFO:tasks.ceph.ceph_manager.ceph:PG 3.3a is not active+clean 2024-09-16T01:34:25.781 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 19, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:21.179909+0000', 'last_change': '2024-09-16T01:34:21.179909+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.179909+0000', 'last_undegraded': '2024-09-16T01:34:21.179909+0000', 'last_fullsized': '2024-09-16T01:34:21.179909+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T01:34:25.781 INFO:tasks.ceph.ceph_manager.ceph:PG 3.39 is not active+clean 2024-09-16T01:34:25.781 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.781 INFO:tasks.ceph.ceph_manager.ceph:PG 3.38 is not active+clean 2024-09-16T01:34:25.781 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.781 INFO:tasks.ceph.ceph_manager.ceph:PG 3.37 is not active+clean 2024-09-16T01:34:25.781 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.781 INFO:tasks.ceph.ceph_manager.ceph:PG 3.36 is not active+clean 2024-09-16T01:34:25.781 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.781 INFO:tasks.ceph.ceph_manager.ceph:PG 3.35 is not active+clean 2024-09-16T01:34:25.782 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.782 INFO:tasks.ceph.ceph_manager.ceph:PG 3.34 is not active+clean 2024-09-16T01:34:25.782 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.782 INFO:tasks.ceph.ceph_manager.ceph:PG 3.33 is not active+clean 2024-09-16T01:34:25.782 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.782 INFO:tasks.ceph.ceph_manager.ceph:PG 3.32 is not active+clean 2024-09-16T01:34:25.782 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.782 INFO:tasks.ceph.ceph_manager.ceph:PG 3.31 is not active+clean 2024-09-16T01:34:25.782 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.782 INFO:tasks.ceph.ceph_manager.ceph:PG 3.30 is not active+clean 2024-09-16T01:34:25.782 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 19, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:21.179665+0000', 'last_change': '2024-09-16T01:34:21.179665+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.179665+0000', 'last_undegraded': '2024-09-16T01:34:21.179665+0000', 'last_fullsized': '2024-09-16T01:34:21.179665+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T01:34:25.782 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2f is not active+clean 2024-09-16T01:34:25.783 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 19, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:21.178542+0000', 'last_change': '2024-09-16T01:34:21.178542+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.178542+0000', 'last_undegraded': '2024-09-16T01:34:21.178542+0000', 'last_fullsized': '2024-09-16T01:34:21.178542+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T01:34:25.783 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2e is not active+clean 2024-09-16T01:34:25.783 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.783 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2d is not active+clean 2024-09-16T01:34:25.783 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.783 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2c is not active+clean 2024-09-16T01:34:25.783 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.783 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2b is not active+clean 2024-09-16T01:34:25.783 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.783 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2a is not active+clean 2024-09-16T01:34:25.783 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 19, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:21.179261+0000', 'last_change': '2024-09-16T01:34:21.179261+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.179261+0000', 'last_undegraded': '2024-09-16T01:34:21.179261+0000', 'last_fullsized': '2024-09-16T01:34:21.179261+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T01:34:25.784 INFO:tasks.ceph.ceph_manager.ceph:PG 3.29 is not active+clean 2024-09-16T01:34:25.784 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.784 INFO:tasks.ceph.ceph_manager.ceph:PG 3.28 is not active+clean 2024-09-16T01:34:25.784 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 19, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:21.179034+0000', 'last_change': '2024-09-16T01:34:21.179034+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.179034+0000', 'last_undegraded': '2024-09-16T01:34:21.179034+0000', 'last_fullsized': '2024-09-16T01:34:21.179034+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T01:34:25.784 INFO:tasks.ceph.ceph_manager.ceph:PG 3.27 is not active+clean 2024-09-16T01:34:25.784 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.784 INFO:tasks.ceph.ceph_manager.ceph:PG 3.26 is not active+clean 2024-09-16T01:34:25.784 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.784 INFO:tasks.ceph.ceph_manager.ceph:PG 3.25 is not active+clean 2024-09-16T01:34:25.785 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.785 INFO:tasks.ceph.ceph_manager.ceph:PG 3.24 is not active+clean 2024-09-16T01:34:25.785 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.785 INFO:tasks.ceph.ceph_manager.ceph:PG 3.23 is not active+clean 2024-09-16T01:34:25.785 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.785 INFO:tasks.ceph.ceph_manager.ceph:PG 3.22 is not active+clean 2024-09-16T01:34:25.785 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 19, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:21.178570+0000', 'last_change': '2024-09-16T01:34:21.178570+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.178570+0000', 'last_undegraded': '2024-09-16T01:34:21.178570+0000', 'last_fullsized': '2024-09-16T01:34:21.178570+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T01:34:25.785 INFO:tasks.ceph.ceph_manager.ceph:PG 3.21 is not active+clean 2024-09-16T01:34:25.785 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.785 INFO:tasks.ceph.ceph_manager.ceph:PG 3.20 is not active+clean 2024-09-16T01:34:25.785 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.786 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1a is not active+clean 2024-09-16T01:34:25.786 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.786 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1b is not active+clean 2024-09-16T01:34:25.786 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.174385+0000', 'last_change': '2024-09-16T01:34:20.174385+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.174385+0000', 'last_undegraded': '2024-09-16T01:34:20.174385+0000', 'last_fullsized': '2024-09-16T01:34:20.174385+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T01:34:25.786 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1b is not active+clean 2024-09-16T01:34:25.786 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.786 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1a is not active+clean 2024-09-16T01:34:25.786 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.170457+0000', 'last_change': '2024-09-16T01:34:20.170457+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.170457+0000', 'last_undegraded': '2024-09-16T01:34:20.170457+0000', 'last_fullsized': '2024-09-16T01:34:20.170457+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T01:34:25.786 INFO:tasks.ceph.ceph_manager.ceph:PG 2.19 is not active+clean 2024-09-16T01:34:25.786 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.174351+0000', 'last_change': '2024-09-16T01:34:20.174351+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.174351+0000', 'last_undegraded': '2024-09-16T01:34:20.174351+0000', 'last_fullsized': '2024-09-16T01:34:20.174351+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T01:34:25.786 INFO:tasks.ceph.ceph_manager.ceph:PG 3.18 is not active+clean 2024-09-16T01:34:25.787 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.787 INFO:tasks.ceph.ceph_manager.ceph:PG 3.19 is not active+clean 2024-09-16T01:34:25.787 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.787 INFO:tasks.ceph.ceph_manager.ceph:PG 2.17 is not active+clean 2024-09-16T01:34:25.787 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.172066+0000', 'last_change': '2024-09-16T01:34:20.172066+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.172066+0000', 'last_undegraded': '2024-09-16T01:34:20.172066+0000', 'last_fullsized': '2024-09-16T01:34:20.172066+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T01:34:25.787 INFO:tasks.ceph.ceph_manager.ceph:PG 3.16 is not active+clean 2024-09-16T01:34:25.787 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 19, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:21.177301+0000', 'last_change': '2024-09-16T01:34:21.177301+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.177301+0000', 'last_undegraded': '2024-09-16T01:34:21.177301+0000', 'last_fullsized': '2024-09-16T01:34:21.177301+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T01:34:25.787 INFO:tasks.ceph.ceph_manager.ceph:PG 3.17 is not active+clean 2024-09-16T01:34:25.787 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.787 INFO:tasks.ceph.ceph_manager.ceph:PG 2.15 is not active+clean 2024-09-16T01:34:25.787 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.171266+0000', 'last_change': '2024-09-16T01:34:20.171266+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.171266+0000', 'last_undegraded': '2024-09-16T01:34:20.171266+0000', 'last_fullsized': '2024-09-16T01:34:20.171266+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T01:34:25.788 INFO:tasks.ceph.ceph_manager.ceph:PG 3.14 is not active+clean 2024-09-16T01:34:25.788 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.788 INFO:tasks.ceph.ceph_manager.ceph:PG 2.14 is not active+clean 2024-09-16T01:34:25.788 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.170974+0000', 'last_change': '2024-09-16T01:34:20.170974+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.170974+0000', 'last_undegraded': '2024-09-16T01:34:20.170974+0000', 'last_fullsized': '2024-09-16T01:34:20.170974+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T01:34:25.788 INFO:tasks.ceph.ceph_manager.ceph:PG 3.15 is not active+clean 2024-09-16T01:34:25.788 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.788 INFO:tasks.ceph.ceph_manager.ceph:PG 2.13 is not active+clean 2024-09-16T01:34:25.788 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.174386+0000', 'last_change': '2024-09-16T01:34:20.174386+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.174386+0000', 'last_undegraded': '2024-09-16T01:34:20.174386+0000', 'last_fullsized': '2024-09-16T01:34:20.174386+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T01:34:25.788 INFO:tasks.ceph.ceph_manager.ceph:PG 3.12 is not active+clean 2024-09-16T01:34:25.788 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.788 INFO:tasks.ceph.ceph_manager.ceph:PG 3.13 is not active+clean 2024-09-16T01:34:25.789 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.789 INFO:tasks.ceph.ceph_manager.ceph:PG 2.11 is not active+clean 2024-09-16T01:34:25.789 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.170843+0000', 'last_change': '2024-09-16T01:34:20.170843+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.170843+0000', 'last_undegraded': '2024-09-16T01:34:20.170843+0000', 'last_fullsized': '2024-09-16T01:34:20.170843+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T01:34:25.789 INFO:tasks.ceph.ceph_manager.ceph:PG 3.10 is not active+clean 2024-09-16T01:34:25.789 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.789 INFO:tasks.ceph.ceph_manager.ceph:PG 2.10 is not active+clean 2024-09-16T01:34:25.789 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.179961+0000', 'last_change': '2024-09-16T01:34:20.179961+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.179961+0000', 'last_undegraded': '2024-09-16T01:34:20.179961+0000', 'last_fullsized': '2024-09-16T01:34:20.179961+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T01:34:25.789 INFO:tasks.ceph.ceph_manager.ceph:PG 3.11 is not active+clean 2024-09-16T01:34:25.790 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.790 INFO:tasks.ceph.ceph_manager.ceph:PG 2.f is not active+clean 2024-09-16T01:34:25.790 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.172767+0000', 'last_change': '2024-09-16T01:34:20.172767+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.172767+0000', 'last_undegraded': '2024-09-16T01:34:20.172767+0000', 'last_fullsized': '2024-09-16T01:34:20.172767+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T01:34:25.790 INFO:tasks.ceph.ceph_manager.ceph:PG 3.e is not active+clean 2024-09-16T01:34:25.790 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.790 INFO:tasks.ceph.ceph_manager.ceph:PG 2.e is not active+clean 2024-09-16T01:34:25.790 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.179968+0000', 'last_change': '2024-09-16T01:34:20.179968+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.179968+0000', 'last_undegraded': '2024-09-16T01:34:20.179968+0000', 'last_fullsized': '2024-09-16T01:34:20.179968+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T01:34:25.790 INFO:tasks.ceph.ceph_manager.ceph:PG 3.f is not active+clean 2024-09-16T01:34:25.790 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.790 INFO:tasks.ceph.ceph_manager.ceph:PG 2.d is not active+clean 2024-09-16T01:34:25.791 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.170553+0000', 'last_change': '2024-09-16T01:34:20.170553+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.170553+0000', 'last_undegraded': '2024-09-16T01:34:20.170553+0000', 'last_fullsized': '2024-09-16T01:34:20.170553+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T01:34:25.791 INFO:tasks.ceph.ceph_manager.ceph:PG 3.c is not active+clean 2024-09-16T01:34:25.791 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 19, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:21.176558+0000', 'last_change': '2024-09-16T01:34:21.176558+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.176558+0000', 'last_undegraded': '2024-09-16T01:34:21.176558+0000', 'last_fullsized': '2024-09-16T01:34:21.176558+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T01:34:25.791 INFO:tasks.ceph.ceph_manager.ceph:PG 2.c is not active+clean 2024-09-16T01:34:25.791 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.179920+0000', 'last_change': '2024-09-16T01:34:20.179920+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.179920+0000', 'last_undegraded': '2024-09-16T01:34:20.179920+0000', 'last_fullsized': '2024-09-16T01:34:20.179920+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T01:34:25.791 INFO:tasks.ceph.ceph_manager.ceph:PG 3.d is not active+clean 2024-09-16T01:34:25.791 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.791 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2 is not active+clean 2024-09-16T01:34:25.791 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.791 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean 2024-09-16T01:34:25.791 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.169473+0000', 'last_change': '2024-09-16T01:34:20.169473+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.169473+0000', 'last_undegraded': '2024-09-16T01:34:20.169473+0000', 'last_fullsized': '2024-09-16T01:34:20.169473+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T01:34:25.792 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1 is not active+clean 2024-09-16T01:34:25.792 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.792 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean 2024-09-16T01:34:25.792 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.179953+0000', 'last_change': '2024-09-16T01:34:20.179953+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.179953+0000', 'last_undegraded': '2024-09-16T01:34:20.179953+0000', 'last_fullsized': '2024-09-16T01:34:20.179953+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T01:34:25.792 INFO:tasks.ceph.ceph_manager.ceph:PG 3.0 is not active+clean 2024-09-16T01:34:25.792 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.792 INFO:tasks.ceph.ceph_manager.ceph:PG 3.3 is not active+clean 2024-09-16T01:34:25.792 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.792 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-09-16T01:34:25.792 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.170569+0000', 'last_change': '2024-09-16T01:34:20.170569+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.170569+0000', 'last_undegraded': '2024-09-16T01:34:20.170569+0000', 'last_fullsized': '2024-09-16T01:34:20.170569+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T01:34:25.792 INFO:tasks.ceph.ceph_manager.ceph:PG 3.5 is not active+clean 2024-09-16T01:34:25.793 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 19, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:21.176775+0000', 'last_change': '2024-09-16T01:34:21.176775+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.176775+0000', 'last_undegraded': '2024-09-16T01:34:21.176775+0000', 'last_fullsized': '2024-09-16T01:34:21.176775+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T01:34:25.793 INFO:tasks.ceph.ceph_manager.ceph:PG 2.5 is not active+clean 2024-09-16T01:34:25.793 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.169491+0000', 'last_change': '2024-09-16T01:34:20.169491+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.169491+0000', 'last_undegraded': '2024-09-16T01:34:20.169491+0000', 'last_fullsized': '2024-09-16T01:34:20.169491+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T01:34:25.793 INFO:tasks.ceph.ceph_manager.ceph:PG 3.4 is not active+clean 2024-09-16T01:34:25.793 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.793 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean 2024-09-16T01:34:25.793 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.170396+0000', 'last_change': '2024-09-16T01:34:20.170396+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.170396+0000', 'last_undegraded': '2024-09-16T01:34:20.170396+0000', 'last_fullsized': '2024-09-16T01:34:20.170396+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T01:34:25.793 INFO:tasks.ceph.ceph_manager.ceph:PG 3.7 is not active+clean 2024-09-16T01:34:25.793 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.793 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean 2024-09-16T01:34:25.794 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.171160+0000', 'last_change': '2024-09-16T01:34:20.171160+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.171160+0000', 'last_undegraded': '2024-09-16T01:34:20.171160+0000', 'last_fullsized': '2024-09-16T01:34:20.171160+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T01:34:25.794 INFO:tasks.ceph.ceph_manager.ceph:PG 3.6 is not active+clean 2024-09-16T01:34:25.794 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.794 INFO:tasks.ceph.ceph_manager.ceph:PG 2.8 is not active+clean 2024-09-16T01:34:25.794 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.170346+0000', 'last_change': '2024-09-16T01:34:20.170346+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.170346+0000', 'last_undegraded': '2024-09-16T01:34:20.170346+0000', 'last_fullsized': '2024-09-16T01:34:20.170346+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T01:34:25.794 INFO:tasks.ceph.ceph_manager.ceph:PG 3.9 is not active+clean 2024-09-16T01:34:25.794 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.794 INFO:tasks.ceph.ceph_manager.ceph:PG 2.9 is not active+clean 2024-09-16T01:34:25.794 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.170547+0000', 'last_change': '2024-09-16T01:34:20.170547+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.170547+0000', 'last_undegraded': '2024-09-16T01:34:20.170547+0000', 'last_fullsized': '2024-09-16T01:34:20.170547+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T01:34:25.795 INFO:tasks.ceph.ceph_manager.ceph:PG 3.8 is not active+clean 2024-09-16T01:34:25.795 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.795 INFO:tasks.ceph.ceph_manager.ceph:PG 2.a is not active+clean 2024-09-16T01:34:25.795 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.170325+0000', 'last_change': '2024-09-16T01:34:20.170325+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.170325+0000', 'last_undegraded': '2024-09-16T01:34:20.170325+0000', 'last_fullsized': '2024-09-16T01:34:20.170325+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T01:34:25.795 INFO:tasks.ceph.ceph_manager.ceph:PG 3.b is not active+clean 2024-09-16T01:34:25.795 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.795 INFO:tasks.ceph.ceph_manager.ceph:PG 2.b is not active+clean 2024-09-16T01:34:25.795 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.169501+0000', 'last_change': '2024-09-16T01:34:20.169501+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.169501+0000', 'last_undegraded': '2024-09-16T01:34:20.169501+0000', 'last_fullsized': '2024-09-16T01:34:20.169501+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T01:34:25.795 INFO:tasks.ceph.ceph_manager.ceph:PG 3.a is not active+clean 2024-09-16T01:34:25.796 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.796 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1d is not active+clean 2024-09-16T01:34:25.796 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 19, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:21.177698+0000', 'last_change': '2024-09-16T01:34:21.177698+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.177698+0000', 'last_undegraded': '2024-09-16T01:34:21.177698+0000', 'last_fullsized': '2024-09-16T01:34:21.177698+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T01:34:25.796 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1c is not active+clean 2024-09-16T01:34:25.796 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.169483+0000', 'last_change': '2024-09-16T01:34:20.169483+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.169483+0000', 'last_undegraded': '2024-09-16T01:34:20.169483+0000', 'last_fullsized': '2024-09-16T01:34:20.169483+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T01:34:25.796 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1c is not active+clean 2024-09-16T01:34:25.796 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 19, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:21.177680+0000', 'last_change': '2024-09-16T01:34:21.177680+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.177680+0000', 'last_undegraded': '2024-09-16T01:34:21.177680+0000', 'last_fullsized': '2024-09-16T01:34:21.177680+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-16T01:34:25.796 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1d is not active+clean 2024-09-16T01:34:25.796 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.170583+0000', 'last_change': '2024-09-16T01:34:20.170583+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.170583+0000', 'last_undegraded': '2024-09-16T01:34:20.170583+0000', 'last_fullsized': '2024-09-16T01:34:20.170583+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T01:34:25.796 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1f is not active+clean 2024-09-16T01:34:25.796 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.797 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1e is not active+clean 2024-09-16T01:34:25.797 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.174438+0000', 'last_change': '2024-09-16T01:34:20.174438+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.174438+0000', 'last_undegraded': '2024-09-16T01:34:20.174438+0000', 'last_fullsized': '2024-09-16T01:34:20.174438+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T01:34:25.797 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1e is not active+clean 2024-09-16T01:34:25.797 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-16T01:34:21.167002+0000', 'last_change': '2024-09-16T01:34:21.167002+0000', 'last_active': '2024-09-16T01:34:21.167002+0000', 'last_peered': '2024-09-16T01:34:21.167002+0000', 'last_clean': '2024-09-16T01:34:21.167002+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:21.167002+0000', 'last_undegraded': '2024-09-16T01:34:21.167002+0000', 'last_fullsized': '2024-09-16T01:34:21.167002+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:21.167002+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-16T01:34:25.797 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1f is not active+clean 2024-09-16T01:34:25.797 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 18, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:34:20.174390+0000', 'last_change': '2024-09-16T01:34:20.174390+0000', 'last_active': '2024-09-16T01:34:20.159368+0000', 'last_peered': '2024-09-16T01:34:20.159368+0000', 'last_clean': '2024-09-16T01:34:20.159368+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:34:20.174390+0000', 'last_undegraded': '2024-09-16T01:34:20.174390+0000', 'last_fullsized': '2024-09-16T01:34:20.174390+0000', 'mapping_epoch': 18, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 18, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'last_clean_scrub_stamp': '2024-09-16T01:34:20.159368+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T01:34:25.798 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T01:34:25.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.912+0000 7f2b23303640 1 Processor -- start 2024-09-16T01:34:25.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.912+0000 7f2b23303640 1 -- start start 2024-09-16T01:34:25.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.913+0000 7f2b23303640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2b1c0ba240 0x7f2b1c0d3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:25.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.913+0000 7f2b23303640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b1c0d2f40 con 0x7f2b1c15e7a0 2024-09-16T01:34:25.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.913+0000 7f2b23303640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b1c0d3220 con 0x7f2b1c0cf2f0 2024-09-16T01:34:25.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.913+0000 7f2b23303640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2b1c0ba690 con 0x7f2b1c0ba240 2024-09-16T01:34:25.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.913+0000 7f2b2089e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2b1c0ba240 0x7f2b1c0d3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:25.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.913+0000 7f2b2089e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2b1c0ba240 0x7f2b1c0d3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58310/0 (socket says 172.21.15.88:58310) 2024-09-16T01:34:25.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.913+0000 7f2b2089e640 1 -- 172.21.15.88:0/3160137344 learned_addr learned my addr 172.21.15.88:0/3160137344 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:25.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.913+0000 7f2b2089e640 1 -- 172.21.15.88:0/3160137344 >> v1:172.21.15.154:6789/0 conn(0x7f2b1c0cf2f0 legacy=0x7f2b1c0b7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:25.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.913+0000 7f2b2089e640 1 -- 172.21.15.88:0/3160137344 >> v1:172.21.15.88:6789/0 conn(0x7f2b1c15e7a0 legacy=0x7f2b1c0d2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:25.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.913+0000 7f2b2089e640 1 -- 172.21.15.88:0/3160137344 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b1c0d3500 con 0x7f2b1c0ba240 2024-09-16T01:34:25.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.914+0000 7f2b2089e640 1 --2- 172.21.15.88:0/3160137344 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b1c0ba240 0x7f2b1c0d3fe0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c009870 tx=0x7f2b0c02a3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32ed217210a9b87d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:25.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.914+0000 7f2b13fff640 1 -- 172.21.15.88:0/3160137344 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c037070 con 0x7f2b1c0ba240 2024-09-16T01:34:25.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.914+0000 7f2b13fff640 1 -- 172.21.15.88:0/3160137344 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b0c032640 con 0x7f2b1c0ba240 2024-09-16T01:34:25.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.914+0000 7f2b13fff640 1 -- 172.21.15.88:0/3160137344 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c032a10 con 0x7f2b1c0ba240 2024-09-16T01:34:25.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.915+0000 7f2b23303640 1 -- 172.21.15.88:0/3160137344 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b1c0ba240 msgr2=0x7f2b1c0d3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:25.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.915+0000 7f2b23303640 1 --2- 172.21.15.88:0/3160137344 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b1c0ba240 0x7f2b1c0d3fe0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c009870 tx=0x7f2b0c02a3c0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.915+0000 7f2b23303640 1 -- 172.21.15.88:0/3160137344 shutdown_connections 2024-09-16T01:34:25.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.915+0000 7f2b23303640 1 --2- 172.21.15.88:0/3160137344 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b1c0ba240 0x7f2b1c0d3fe0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.915+0000 7f2b23303640 1 -- 172.21.15.88:0/3160137344 >> 172.21.15.88:0/3160137344 conn(0x7f2b1c0d1550 msgr2=0x7f2b1c0d1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:25.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.915+0000 7f2b23303640 1 -- 172.21.15.88:0/3160137344 shutdown_connections 2024-09-16T01:34:25.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.915+0000 7f2b23303640 1 -- 172.21.15.88:0/3160137344 wait complete. 2024-09-16T01:34:25.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.916+0000 7f2b23303640 1 Processor -- start 2024-09-16T01:34:25.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.916+0000 7f2b23303640 1 -- start start 2024-09-16T01:34:25.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.917+0000 7f2b23303640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b1c0ba240 0x7f2b1c155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:25.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.917+0000 7f2b23303640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b1c0cf2f0 0x7f2b1c15b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:25.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.917+0000 7f2b23303640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b1c15e7a0 0x7f2b1c15d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:25.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.917+0000 7f2b23303640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2b1c15eb80 con 0x7f2b1c15e7a0 2024-09-16T01:34:25.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.917+0000 7f2b23303640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2b1c0d3ac0 con 0x7f2b1c0ba240 2024-09-16T01:34:25.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.917+0000 7f2b23303640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2b1c15ed00 con 0x7f2b1c0cf2f0 2024-09-16T01:34:25.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.917+0000 7f2b218a0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b1c15e7a0 0x7f2b1c15d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:25.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.917+0000 7f2b218a0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b1c15e7a0 0x7f2b1c15d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58318/0 (socket says 172.21.15.88:58318) 2024-09-16T01:34:25.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.917+0000 7f2b218a0640 1 -- 172.21.15.88:0/1560921236 learned_addr learned my addr 172.21.15.88:0/1560921236 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:25.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.917+0000 7f2b2109f640 1 --2- 172.21.15.88:0/1560921236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b1c0ba240 0x7f2b1c155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:25.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.917+0000 7f2b2089e640 1 --2- 172.21.15.88:0/1560921236 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b1c0cf2f0 0x7f2b1c15b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:25.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.917+0000 7f2b218a0640 1 -- 172.21.15.88:0/1560921236 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b1c0cf2f0 msgr2=0x7f2b1c15b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:25.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.917+0000 7f2b218a0640 1 --2- 172.21.15.88:0/1560921236 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b1c0cf2f0 0x7f2b1c15b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.917+0000 7f2b218a0640 1 -- 172.21.15.88:0/1560921236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b1c0ba240 msgr2=0x7f2b1c155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:25.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.917+0000 7f2b218a0640 1 --2- 172.21.15.88:0/1560921236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b1c0ba240 0x7f2b1c155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:25.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.917+0000 7f2b218a0640 1 -- 172.21.15.88:0/1560921236 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b1c161720 con 0x7f2b1c15e7a0 2024-09-16T01:34:25.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.917+0000 7f2b2109f640 1 --2- 172.21.15.88:0/1560921236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b1c0ba240 0x7f2b1c155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:25.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.917+0000 7f2b2089e640 1 --2- 172.21.15.88:0/1560921236 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b1c0cf2f0 0x7f2b1c15b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:25.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.917+0000 7f2b218a0640 1 --2- 172.21.15.88:0/1560921236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b1c15e7a0 0x7f2b1c15d0b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2b1400c7d0 tx=0x7f2b1400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:25.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.917+0000 7f2b11ffb640 1 -- 172.21.15.88:0/1560921236 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b14003b70 con 0x7f2b1c15e7a0 2024-09-16T01:34:25.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.917+0000 7f2b11ffb640 1 -- 172.21.15.88:0/1560921236 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b14003d30 con 0x7f2b1c15e7a0 2024-09-16T01:34:25.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.917+0000 7f2b11ffb640 1 -- 172.21.15.88:0/1560921236 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b1400aa90 con 0x7f2b1c15e7a0 2024-09-16T01:34:25.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.917+0000 7f2b23303640 1 -- 172.21.15.88:0/1560921236 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b1c160e80 con 0x7f2b1c15e7a0 2024-09-16T01:34:25.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.918+0000 7f2b23303640 1 -- 172.21.15.88:0/1560921236 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b1c161440 con 0x7f2b1c15e7a0 2024-09-16T01:34:25.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.918+0000 7f2b23303640 1 -- 172.21.15.88:0/1560921236 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b1c0d3fe0 con 0x7f2b1c15e7a0 2024-09-16T01:34:25.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.919+0000 7f2b11ffb640 1 -- 172.21.15.88:0/1560921236 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2b1400ac30 con 0x7f2b1c15e7a0 2024-09-16T01:34:25.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.919+0000 7f2b11ffb640 1 --2- 172.21.15.88:0/1560921236 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b0006c2b0 0x7f2b0006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:25.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.919+0000 7f2b11ffb640 1 -- 172.21.15.88:0/1560921236 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2b1408b350 con 0x7f2b1c15e7a0 2024-09-16T01:34:25.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.919+0000 7f2b2109f640 1 --2- 172.21.15.88:0/1560921236 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b0006c2b0 0x7f2b0006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:25.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.919+0000 7f2b2109f640 1 --2- 172.21.15.88:0/1560921236 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b0006c2b0 0x7f2b0006e770 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2b18000f60 tx=0x7f2b18009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:25.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:25.923+0000 7f2b11ffb640 1 -- 172.21.15.88:0/1560921236 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2b14059b50 con 0x7f2b1c15e7a0 2024-09-16T01:34:26.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.152+0000 7f2b23303640 1 -- 172.21.15.88:0/1560921236 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f2b1c0b8f20 con 0x7f2b1c15e7a0 2024-09-16T01:34:26.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.152+0000 7f2b11ffb640 1 -- 172.21.15.88:0/1560921236 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1149 (secure 0 0 0) 0x7f2b14059450 con 0x7f2b1c15e7a0 2024-09-16T01:34:26.154 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:26.154 INFO:teuthology.orchestra.run.smithi088.stdout:{"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":27,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":21,"num_osds":8,"num_up_osds":8,"osd_up_since":1726450450,"num_in_osds":8,"osd_in_since":1726450440,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":52},{"state_name":"creating+peering","count":39},{"state_name":"active+clean","count":6}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":224141312,"bytes_avail":772869971968,"bytes_total":773094113280,"unknown_pgs_ratio":0.53608244657516479,"inactive_pgs_ratio":0.40206184983253479},"fsmap":{"epoch":6,"btime":"2024-09-16T01:34:24:186280+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"a","status":"up:active","gid":4296}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-16T01:33:58.413617+0000","services":{}},"progress_events":{}} 2024-09-16T01:34:26.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.155+0000 7f2b23303640 1 -- 172.21.15.88:0/1560921236 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b0006c2b0 msgr2=0x7f2b0006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:26.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.155+0000 7f2b23303640 1 --2- 172.21.15.88:0/1560921236 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b0006c2b0 0x7f2b0006e770 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2b18000f60 tx=0x7f2b18009040 comp rx=0 tx=0).stop 2024-09-16T01:34:26.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.155+0000 7f2b23303640 1 -- 172.21.15.88:0/1560921236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b1c15e7a0 msgr2=0x7f2b1c15d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:26.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.155+0000 7f2b23303640 1 --2- 172.21.15.88:0/1560921236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b1c15e7a0 0x7f2b1c15d0b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2b1400c7d0 tx=0x7f2b1400cc90 comp rx=0 tx=0).stop 2024-09-16T01:34:26.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.155+0000 7f2b23303640 1 -- 172.21.15.88:0/1560921236 shutdown_connections 2024-09-16T01:34:26.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.155+0000 7f2b23303640 1 --2- 172.21.15.88:0/1560921236 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b0006c2b0 0x7f2b0006e770 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:26.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.155+0000 7f2b23303640 1 --2- 172.21.15.88:0/1560921236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b1c15e7a0 0x7f2b1c15d0b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:26.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.155+0000 7f2b23303640 1 --2- 172.21.15.88:0/1560921236 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b1c0cf2f0 0x7f2b1c15b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:26.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.155+0000 7f2b23303640 1 --2- 172.21.15.88:0/1560921236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b1c0ba240 0x7f2b1c155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:26.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.155+0000 7f2b23303640 1 -- 172.21.15.88:0/1560921236 >> 172.21.15.88:0/1560921236 conn(0x7f2b1c0d1550 msgr2=0x7f2b1c0d2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:26.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.155+0000 7f2b23303640 1 -- 172.21.15.88:0/1560921236 shutdown_connections 2024-09-16T01:34:26.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.155+0000 7f2b23303640 1 -- 172.21.15.88:0/1560921236 wait complete. 2024-09-16T01:34:26.166 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-16T01:34:26.167 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:34:26.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.262+0000 7f5095ac9640 1 Processor -- start 2024-09-16T01:34:26.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.263+0000 7f5095ac9640 1 -- start start 2024-09-16T01:34:26.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.263+0000 7f5095ac9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5090073870 0x7f50900b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:26.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.263+0000 7f5095ac9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50900b1960 con 0x7f50900b1bf0 2024-09-16T01:34:26.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.263+0000 7f5095ac9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5090070930 con 0x7f509015c5a0 2024-09-16T01:34:26.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.263+0000 7f5095ac9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5090071520 con 0x7f5090073870 2024-09-16T01:34:26.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.263+0000 7f508ffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f50900b1bf0 0x7f50900b5540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34274/0 (socket says 172.21.15.88:34274) 2024-09-16T01:34:26.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.263+0000 7f508ffff640 1 -- 172.21.15.88:0/1412737936 learned_addr learned my addr 172.21.15.88:0/1412737936 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:26.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f5086ffd640 1 --2- 172.21.15.88:0/1412737936 >> v2:172.21.15.154:3300/0 conn(0x7f5090073870 0x7f50900b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:26.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f508effd640 1 -- 172.21.15.88:0/1412737936 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2140595088 0 0) 0x7f50900b1960 con 0x7f50900b1bf0 2024-09-16T01:34:26.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f508effd640 1 -- 172.21.15.88:0/1412737936 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f507c003520 con 0x7f50900b1bf0 2024-09-16T01:34:26.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f508effd640 1 -- 172.21.15.88:0/1412737936 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2257272895 0 0) 0x7f5090070930 con 0x7f509015c5a0 2024-09-16T01:34:26.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f508effd640 1 -- 172.21.15.88:0/1412737936 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50900b1960 con 0x7f509015c5a0 2024-09-16T01:34:26.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f508effd640 1 -- 172.21.15.88:0/1412737936 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3219993638 0 0) 0x7f507c003520 con 0x7f50900b1bf0 2024-09-16T01:34:26.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f508effd640 1 -- 172.21.15.88:0/1412737936 >> v2:172.21.15.154:3300/0 conn(0x7f5090073870 msgr2=0x7f50900b4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:26.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f508effd640 1 --2- 172.21.15.88:0/1412737936 >> v2:172.21.15.154:3300/0 conn(0x7f5090073870 0x7f50900b4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:26.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f508effd640 1 -- 172.21.15.88:0/1412737936 >> v1:172.21.15.154:6789/0 conn(0x7f509015c5a0 legacy=0x7f5090073160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:26.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f508effd640 1 -- 172.21.15.88:0/1412737936 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5090070bf0 con 0x7f50900b1bf0 2024-09-16T01:34:26.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f508effd640 1 -- 172.21.15.88:0/1412737936 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f50880019f0 con 0x7f50900b1bf0 2024-09-16T01:34:26.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f508effd640 1 -- 172.21.15.88:0/1412737936 >> v1:172.21.15.88:6789/0 conn(0x7f50900b1bf0 legacy=0x7f50900b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:26.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f508effd640 1 --2- 172.21.15.88:0/1412737936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f507c003b90 0x7f507c003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:26.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f508effd640 1 --2- 172.21.15.88:0/1412737936 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507c004800 0x7f507c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:26.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f508effd640 1 --2- 172.21.15.88:0/1412737936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f507c0051c0 0x7f507c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:26.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f508effd640 1 -- 172.21.15.88:0/1412737936 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5090071520 con 0x7f507c003b90 2024-09-16T01:34:26.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f508effd640 1 -- 172.21.15.88:0/1412737936 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f507c00c090 con 0x7f507c0051c0 2024-09-16T01:34:26.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f508effd640 1 -- 172.21.15.88:0/1412737936 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f507c00c290 con 0x7f507c004800 2024-09-16T01:34:26.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f508f7fe640 1 --2- 172.21.15.88:0/1412737936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f507c003b90 0x7f507c003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:26.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f508ffff640 1 --2- 172.21.15.88:0/1412737936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f507c0051c0 0x7f507c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:26.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f5086ffd640 1 --2- 172.21.15.88:0/1412737936 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507c004800 0x7f507c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:26.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f508f7fe640 1 -- 172.21.15.88:0/1412737936 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507c004800 msgr2=0x7f507c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:26.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f508f7fe640 1 --2- 172.21.15.88:0/1412737936 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507c004800 0x7f507c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:26.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f508f7fe640 1 -- 172.21.15.88:0/1412737936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f507c0051c0 msgr2=0x7f507c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:26.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f508f7fe640 1 --2- 172.21.15.88:0/1412737936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f507c0051c0 0x7f507c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:26.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f508f7fe640 1 -- 172.21.15.88:0/1412737936 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f507c003520 con 0x7f507c003b90 2024-09-16T01:34:26.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.264+0000 7f508ffff640 1 -- 172.21.15.88:0/1412737936 reap_dead start 2024-09-16T01:34:26.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.265+0000 7f508f7fe640 1 --2- 172.21.15.88:0/1412737936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f507c003b90 0x7f507c003f90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f50800018d0 tx=0x7f507c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a6513885a38e8df server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:26.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.265+0000 7f508effd640 1 -- 172.21.15.88:0/1412737936 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50800014f0 con 0x7f507c003b90 2024-09-16T01:34:26.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.265+0000 7f508effd640 1 -- 172.21.15.88:0/1412737936 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50800052d0 con 0x7f507c003b90 2024-09-16T01:34:26.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.265+0000 7f508effd640 1 -- 172.21.15.88:0/1412737936 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f508002ac70 con 0x7f507c003b90 2024-09-16T01:34:26.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.266+0000 7f5095ac9640 1 -- 172.21.15.88:0/1412737936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f507c003b90 msgr2=0x7f507c003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:26.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.266+0000 7f5095ac9640 1 --2- 172.21.15.88:0/1412737936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f507c003b90 0x7f507c003f90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f50800018d0 tx=0x7f507c005830 comp rx=0 tx=0).stop 2024-09-16T01:34:26.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.266+0000 7f5095ac9640 1 -- 172.21.15.88:0/1412737936 shutdown_connections 2024-09-16T01:34:26.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.266+0000 7f5095ac9640 1 --2- 172.21.15.88:0/1412737936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f507c003b90 0x7f507c003f90 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:26.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.266+0000 7f5095ac9640 1 -- 172.21.15.88:0/1412737936 >> 172.21.15.88:0/1412737936 conn(0x7f50900b4240 msgr2=0x7f50900b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:26.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.266+0000 7f5095ac9640 1 -- 172.21.15.88:0/1412737936 shutdown_connections 2024-09-16T01:34:26.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.266+0000 7f5095ac9640 1 -- 172.21.15.88:0/1412737936 wait complete. 2024-09-16T01:34:26.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.267+0000 7f5095ac9640 1 Processor -- start 2024-09-16T01:34:26.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.267+0000 7f5095ac9640 1 -- start start 2024-09-16T01:34:26.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.268+0000 7f5095ac9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507c003b90 0x7f50900b0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:26.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.268+0000 7f5095ac9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50901f3170 0x7f50901552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:26.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.268+0000 7f5095ac9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5090154270 0x7f5090155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:26.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.268+0000 7f5095ac9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f50900d0490 con 0x7f5090154270 2024-09-16T01:34:26.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.268+0000 7f5095ac9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f50900711b0 con 0x7f50901f3170 2024-09-16T01:34:26.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.268+0000 7f5095ac9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f50900d0610 con 0x7f507c003b90 2024-09-16T01:34:26.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.268+0000 7f508ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5090154270 0x7f5090155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:26.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.268+0000 7f508ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5090154270 0x7f5090155830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58330/0 (socket says 172.21.15.88:58330) 2024-09-16T01:34:26.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.268+0000 7f508ffff640 1 -- 172.21.15.88:0/2140542321 learned_addr learned my addr 172.21.15.88:0/2140542321 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:26.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.268+0000 7f508f7fe640 1 --2- 172.21.15.88:0/2140542321 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507c003b90 0x7f50900b0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:26.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.268+0000 7f5086ffd640 1 --2- 172.21.15.88:0/2140542321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50901f3170 0x7f50901552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:26.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.268+0000 7f508ffff640 1 -- 172.21.15.88:0/2140542321 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507c003b90 msgr2=0x7f50900b0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:26.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.268+0000 7f508ffff640 1 --2- 172.21.15.88:0/2140542321 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507c003b90 0x7f50900b0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:26.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.268+0000 7f508ffff640 1 -- 172.21.15.88:0/2140542321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50901f3170 msgr2=0x7f50901552f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:26.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.268+0000 7f508ffff640 1 --2- 172.21.15.88:0/2140542321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50901f3170 0x7f50901552f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:26.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.268+0000 7f508ffff640 1 -- 172.21.15.88:0/2140542321 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50901534b0 con 0x7f5090154270 2024-09-16T01:34:26.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.268+0000 7f508ffff640 1 --2- 172.21.15.88:0/2140542321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5090154270 0x7f5090155830 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5090075540 tx=0x7f5088007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:26.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.269+0000 7f508cff9640 1 -- 172.21.15.88:0/2140542321 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f508800aa10 con 0x7f5090154270 2024-09-16T01:34:26.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.269+0000 7f508cff9640 1 -- 172.21.15.88:0/2140542321 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5088019070 con 0x7f5090154270 2024-09-16T01:34:26.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.269+0000 7f5095ac9640 1 -- 172.21.15.88:0/2140542321 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5090153790 con 0x7f5090154270 2024-09-16T01:34:26.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.269+0000 7f508cff9640 1 -- 172.21.15.88:0/2140542321 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f508800ae10 con 0x7f5090154270 2024-09-16T01:34:26.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.269+0000 7f5095ac9640 1 -- 172.21.15.88:0/2140542321 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5090153d50 con 0x7f5090154270 2024-09-16T01:34:26.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.270+0000 7f5095ac9640 1 -- 172.21.15.88:0/2140542321 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5090068cb0 con 0x7f5090154270 2024-09-16T01:34:26.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.270+0000 7f508cff9640 1 -- 172.21.15.88:0/2140542321 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f508801e460 con 0x7f5090154270 2024-09-16T01:34:26.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.271+0000 7f508cff9640 1 --2- 172.21.15.88:0/2140542321 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f507006c2b0 0x7f507006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:26.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.271+0000 7f508cff9640 1 -- 172.21.15.88:0/2140542321 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f508808b6e0 con 0x7f5090154270 2024-09-16T01:34:26.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.271+0000 7f508f7fe640 1 --2- 172.21.15.88:0/2140542321 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f507006c2b0 0x7f507006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:26.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.271+0000 7f508f7fe640 1 --2- 172.21.15.88:0/2140542321 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f507006c2b0 0x7f507006e770 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5080032000 tx=0x7f5080009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:26.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.275+0000 7f508cff9640 1 -- 172.21.15.88:0/2140542321 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5088059ee0 con 0x7f5090154270 2024-09-16T01:34:26.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.438+0000 7f5095ac9640 1 -- 172.21.15.88:0/2140542321 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f50901f3600 con 0x7f507006c2b0 2024-09-16T01:34:26.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.443+0000 7f508cff9640 1 -- 172.21.15.88:0/2140542321 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+241108 (secure 0 0 0) 0x7f50901f3600 con 0x7f507006c2b0 2024-09-16T01:34:26.447 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:26.448 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:34:26.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.446+0000 7f5095ac9640 1 -- 172.21.15.88:0/2140542321 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f507006c2b0 msgr2=0x7f507006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:26.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.446+0000 7f5095ac9640 1 --2- 172.21.15.88:0/2140542321 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f507006c2b0 0x7f507006e770 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5080032000 tx=0x7f5080009030 comp rx=0 tx=0).stop 2024-09-16T01:34:26.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.446+0000 7f5095ac9640 1 -- 172.21.15.88:0/2140542321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5090154270 msgr2=0x7f5090155830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:26.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.446+0000 7f5095ac9640 1 --2- 172.21.15.88:0/2140542321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5090154270 0x7f5090155830 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5090075540 tx=0x7f5088007b10 comp rx=0 tx=0).stop 2024-09-16T01:34:26.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.446+0000 7f5095ac9640 1 -- 172.21.15.88:0/2140542321 shutdown_connections 2024-09-16T01:34:26.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.446+0000 7f5095ac9640 1 --2- 172.21.15.88:0/2140542321 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f507006c2b0 0x7f507006e770 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:26.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.446+0000 7f5095ac9640 1 --2- 172.21.15.88:0/2140542321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5090154270 0x7f5090155830 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:26.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.446+0000 7f5095ac9640 1 --2- 172.21.15.88:0/2140542321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50901f3170 0x7f50901552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:26.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.446+0000 7f5095ac9640 1 --2- 172.21.15.88:0/2140542321 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507c003b90 0x7f50900b0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:26.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.446+0000 7f5095ac9640 1 -- 172.21.15.88:0/2140542321 >> 172.21.15.88:0/2140542321 conn(0x7f50900b4240 msgr2=0x7f50900b3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:26.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.447+0000 7f5095ac9640 1 -- 172.21.15.88:0/2140542321 shutdown_connections 2024-09-16T01:34:26.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:26.447+0000 7f5095ac9640 1 -- 172.21.15.88:0/2140542321 wait complete. 2024-09-16T01:34:26.458 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-16T01:34:24.517440+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":90,"acting":90,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218888,"kb_used_data":2464,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755832,"statfs":{"total":773094113280,"available":772869971968,"internally_reserved":0,"allocated":2523136,"data_stored":1580914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.350270"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.181767+0000","last_change":"2024-09-16T01:34:21.181767+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.181767+0000","last_undegraded":"2024-09-16T01:34:21.181767+0000","last_fullsized":"2024-09-16T01:34:21.181767+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.179909+0000","last_change":"2024-09-16T01:34:21.179909+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.179909+0000","last_undegraded":"2024-09-16T01:34:21.179909+0000","last_fullsized":"2024-09-16T01:34:21.179909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.179665+0000","last_change":"2024-09-16T01:34:21.179665+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.179665+0000","last_undegraded":"2024-09-16T01:34:21.179665+0000","last_fullsized":"2024-09-16T01:34:21.179665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.178542+0000","last_change":"2024-09-16T01:34:21.178542+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.178542+0000","last_undegraded":"2024-09-16T01:34:21.178542+0000","last_fullsized":"2024-09-16T01:34:21.178542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.179261+0000","last_change":"2024-09-16T01:34:21.179261+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.179261+0000","last_undegraded":"2024-09-16T01:34:21.179261+0000","last_fullsized":"2024-09-16T01:34:21.179261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.179034+0000","last_change":"2024-09-16T01:34:21.179034+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.179034+0000","last_undegraded":"2024-09-16T01:34:21.179034+0000","last_fullsized":"2024-09-16T01:34:21.179034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.178570+0000","last_change":"2024-09-16T01:34:21.178570+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.178570+0000","last_undegraded":"2024-09-16T01:34:21.178570+0000","last_fullsized":"2024-09-16T01:34:21.178570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.174385+0000","last_change":"2024-09-16T01:34:20.174385+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.174385+0000","last_undegraded":"2024-09-16T01:34:20.174385+0000","last_fullsized":"2024-09-16T01:34:20.174385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170457+0000","last_change":"2024-09-16T01:34:20.170457+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170457+0000","last_undegraded":"2024-09-16T01:34:20.170457+0000","last_fullsized":"2024-09-16T01:34:20.170457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.174351+0000","last_change":"2024-09-16T01:34:20.174351+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.174351+0000","last_undegraded":"2024-09-16T01:34:20.174351+0000","last_fullsized":"2024-09-16T01:34:20.174351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T01:34:21.194181+0000","last_change":"2024-09-16T01:34:21.188039+0000","last_active":"2024-09-16T01:34:21.194181+0000","last_peered":"2024-09-16T01:34:21.194181+0000","last_clean":"2024-09-16T01:34:21.194181+0000","last_became_active":"2024-09-16T01:34:21.187743+0000","last_became_peered":"2024-09-16T01:34:21.187743+0000","last_unstale":"2024-09-16T01:34:21.194181+0000","last_undegraded":"2024-09-16T01:34:21.194181+0000","last_fullsized":"2024-09-16T01:34:21.194181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:09:58.843092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.172066+0000","last_change":"2024-09-16T01:34:20.172066+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.172066+0000","last_undegraded":"2024-09-16T01:34:20.172066+0000","last_fullsized":"2024-09-16T01:34:20.172066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.177301+0000","last_change":"2024-09-16T01:34:21.177301+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.177301+0000","last_undegraded":"2024-09-16T01:34:21.177301+0000","last_fullsized":"2024-09-16T01:34:21.177301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T01:34:21.197129+0000","last_change":"2024-09-16T01:34:21.190619+0000","last_active":"2024-09-16T01:34:21.197129+0000","last_peered":"2024-09-16T01:34:21.197129+0000","last_clean":"2024-09-16T01:34:21.197129+0000","last_became_active":"2024-09-16T01:34:21.188943+0000","last_became_peered":"2024-09-16T01:34:21.188943+0000","last_unstale":"2024-09-16T01:34:21.197129+0000","last_undegraded":"2024-09-16T01:34:21.197129+0000","last_fullsized":"2024-09-16T01:34:21.197129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:07:21.563057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.171266+0000","last_change":"2024-09-16T01:34:20.171266+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.171266+0000","last_undegraded":"2024-09-16T01:34:20.171266+0000","last_fullsized":"2024-09-16T01:34:20.171266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170974+0000","last_change":"2024-09-16T01:34:20.170974+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170974+0000","last_undegraded":"2024-09-16T01:34:20.170974+0000","last_fullsized":"2024-09-16T01:34:20.170974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.174386+0000","last_change":"2024-09-16T01:34:20.174386+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.174386+0000","last_undegraded":"2024-09-16T01:34:20.174386+0000","last_fullsized":"2024-09-16T01:34:20.174386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T01:34:21.187991+0000","last_change":"2024-09-16T01:34:21.186203+0000","last_active":"2024-09-16T01:34:21.187991+0000","last_peered":"2024-09-16T01:34:21.187991+0000","last_clean":"2024-09-16T01:34:21.187991+0000","last_became_active":"2024-09-16T01:34:21.185935+0000","last_became_peered":"2024-09-16T01:34:21.185935+0000","last_unstale":"2024-09-16T01:34:21.187991+0000","last_undegraded":"2024-09-16T01:34:21.187991+0000","last_fullsized":"2024-09-16T01:34:21.187991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:56:19.306291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170843+0000","last_change":"2024-09-16T01:34:20.170843+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170843+0000","last_undegraded":"2024-09-16T01:34:20.170843+0000","last_fullsized":"2024-09-16T01:34:20.170843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.179961+0000","last_change":"2024-09-16T01:34:20.179961+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.179961+0000","last_undegraded":"2024-09-16T01:34:20.179961+0000","last_fullsized":"2024-09-16T01:34:20.179961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.172767+0000","last_change":"2024-09-16T01:34:20.172767+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.172767+0000","last_undegraded":"2024-09-16T01:34:20.172767+0000","last_fullsized":"2024-09-16T01:34:20.172767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.179968+0000","last_change":"2024-09-16T01:34:20.179968+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.179968+0000","last_undegraded":"2024-09-16T01:34:20.179968+0000","last_fullsized":"2024-09-16T01:34:20.179968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170553+0000","last_change":"2024-09-16T01:34:20.170553+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170553+0000","last_undegraded":"2024-09-16T01:34:20.170553+0000","last_fullsized":"2024-09-16T01:34:20.170553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.176558+0000","last_change":"2024-09-16T01:34:21.176558+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.176558+0000","last_undegraded":"2024-09-16T01:34:21.176558+0000","last_fullsized":"2024-09-16T01:34:21.176558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.179920+0000","last_change":"2024-09-16T01:34:20.179920+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.179920+0000","last_undegraded":"2024-09-16T01:34:20.179920+0000","last_fullsized":"2024-09-16T01:34:20.179920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T01:34:20.171425+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:34:20.171425+0000","last_peered":"2024-09-16T01:34:20.171425+0000","last_clean":"2024-09-16T01:34:20.171425+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:34:20.171425+0000","last_undegraded":"2024-09-16T01:34:20.171425+0000","last_fullsized":"2024-09-16T01:34:20.171425+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T01:34:21.187731+0000","last_change":"2024-09-16T01:34:21.183119+0000","last_active":"2024-09-16T01:34:21.187731+0000","last_peered":"2024-09-16T01:34:21.187731+0000","last_clean":"2024-09-16T01:34:21.187731+0000","last_became_active":"2024-09-16T01:34:21.182740+0000","last_became_peered":"2024-09-16T01:34:21.182740+0000","last_unstale":"2024-09-16T01:34:21.187731+0000","last_undegraded":"2024-09-16T01:34:21.187731+0000","last_fullsized":"2024-09-16T01:34:21.187731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:49.205242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.169473+0000","last_change":"2024-09-16T01:34:20.169473+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.169473+0000","last_undegraded":"2024-09-16T01:34:20.169473+0000","last_fullsized":"2024-09-16T01:34:20.169473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.179953+0000","last_change":"2024-09-16T01:34:20.179953+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.179953+0000","last_undegraded":"2024-09-16T01:34:20.179953+0000","last_fullsized":"2024-09-16T01:34:20.179953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T01:34:21.197145+0000","last_change":"2024-09-16T01:34:21.190750+0000","last_active":"2024-09-16T01:34:21.197145+0000","last_peered":"2024-09-16T01:34:21.197145+0000","last_clean":"2024-09-16T01:34:21.197145+0000","last_became_active":"2024-09-16T01:34:21.189215+0000","last_became_peered":"2024-09-16T01:34:21.189215+0000","last_unstale":"2024-09-16T01:34:21.197145+0000","last_undegraded":"2024-09-16T01:34:21.197145+0000","last_fullsized":"2024-09-16T01:34:21.197145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:09:25.203416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170569+0000","last_change":"2024-09-16T01:34:20.170569+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170569+0000","last_undegraded":"2024-09-16T01:34:20.170569+0000","last_fullsized":"2024-09-16T01:34:20.170569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.176775+0000","last_change":"2024-09-16T01:34:21.176775+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.176775+0000","last_undegraded":"2024-09-16T01:34:21.176775+0000","last_fullsized":"2024-09-16T01:34:21.176775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.169491+0000","last_change":"2024-09-16T01:34:20.169491+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.169491+0000","last_undegraded":"2024-09-16T01:34:20.169491+0000","last_fullsized":"2024-09-16T01:34:20.169491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170396+0000","last_change":"2024-09-16T01:34:20.170396+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170396+0000","last_undegraded":"2024-09-16T01:34:20.170396+0000","last_fullsized":"2024-09-16T01:34:20.170396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.171160+0000","last_change":"2024-09-16T01:34:20.171160+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.171160+0000","last_undegraded":"2024-09-16T01:34:20.171160+0000","last_fullsized":"2024-09-16T01:34:20.171160+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170346+0000","last_change":"2024-09-16T01:34:20.170346+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170346+0000","last_undegraded":"2024-09-16T01:34:20.170346+0000","last_fullsized":"2024-09-16T01:34:20.170346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170547+0000","last_change":"2024-09-16T01:34:20.170547+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170547+0000","last_undegraded":"2024-09-16T01:34:20.170547+0000","last_fullsized":"2024-09-16T01:34:20.170547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170325+0000","last_change":"2024-09-16T01:34:20.170325+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170325+0000","last_undegraded":"2024-09-16T01:34:20.170325+0000","last_fullsized":"2024-09-16T01:34:20.170325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.169501+0000","last_change":"2024-09-16T01:34:20.169501+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.169501+0000","last_undegraded":"2024-09-16T01:34:20.169501+0000","last_fullsized":"2024-09-16T01:34:20.169501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.177698+0000","last_change":"2024-09-16T01:34:21.177698+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.177698+0000","last_undegraded":"2024-09-16T01:34:21.177698+0000","last_fullsized":"2024-09-16T01:34:21.177698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.169483+0000","last_change":"2024-09-16T01:34:20.169483+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.169483+0000","last_undegraded":"2024-09-16T01:34:20.169483+0000","last_fullsized":"2024-09-16T01:34:20.169483+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1,"reported_epoch":19,"state":"creating+peering","last_fresh":"2024-09-16T01:34:21.177680+0000","last_change":"2024-09-16T01:34:21.177680+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.177680+0000","last_undegraded":"2024-09-16T01:34:21.177680+0000","last_fullsized":"2024-09-16T01:34:21.177680+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.170583+0000","last_change":"2024-09-16T01:34:20.170583+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.170583+0000","last_undegraded":"2024-09-16T01:34:20.170583+0000","last_fullsized":"2024-09-16T01:34:20.170583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.174438+0000","last_change":"2024-09-16T01:34:20.174438+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.174438+0000","last_undegraded":"2024-09-16T01:34:20.174438+0000","last_fullsized":"2024-09-16T01:34:20.174438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:34:21.167002+0000","last_change":"2024-09-16T01:34:21.167002+0000","last_active":"2024-09-16T01:34:21.167002+0000","last_peered":"2024-09-16T01:34:21.167002+0000","last_clean":"2024-09-16T01:34:21.167002+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:21.167002+0000","last_undegraded":"2024-09-16T01:34:21.167002+0000","last_fullsized":"2024-09-16T01:34:21.167002+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1,"reported_epoch":18,"state":"creating+peering","last_fresh":"2024-09-16T01:34:20.174390+0000","last_change":"2024-09-16T01:34:20.174390+0000","last_active":"2024-09-16T01:34:20.159368+0000","last_peered":"2024-09-16T01:34:20.159368+0000","last_clean":"2024-09-16T01:34:20.159368+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:34:20.174390+0000","last_undegraded":"2024-09-16T01:34:20.174390+0000","last_fullsized":"2024-09-16T01:34:20.174390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":1},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":176128,"data_stored":54385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27952,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343888,"statfs":{"total":96636764160,"available":96608141312,"internally_reserved":0,"allocated":757760,"data_stored":638936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":48568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":48568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":48568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":48568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":163840,"data_stored":48568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542148,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":770048,"data_stored":644753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:34:27.158 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:34:27.158+0000 7f6b013f6640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:34:29.463 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:34:29.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.573+0000 7f1233ee4640 1 Processor -- start 2024-09-16T01:34:29.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.574+0000 7f1233ee4640 1 -- start start 2024-09-16T01:34:29.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.574+0000 7f1233ee4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f122c06e910 0x7f122c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:29.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.574+0000 7f1233ee4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f122c073b10 0x7f122c0b4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:29.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.574+0000 7f1233ee4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f122c15eb10 con 0x7f122c15c100 2024-09-16T01:34:29.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.574+0000 7f1233ee4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f122c15edd0 con 0x7f122c073b10 2024-09-16T01:34:29.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.574+0000 7f1233ee4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f122c15f0b0 con 0x7f122c06e910 2024-09-16T01:34:29.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.575+0000 7f1231c80640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f122c15c100 0x7f122c06e200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34278/0 (socket says 172.21.15.88:34278) 2024-09-16T01:34:29.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.575+0000 7f1231c80640 1 -- 172.21.15.88:0/3428930269 learned_addr learned my addr 172.21.15.88:0/3428930269 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:29.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.575+0000 7f123147f640 1 --2- 172.21.15.88:0/3428930269 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f122c06e910 0x7f122c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:29.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.575+0000 7f1232481640 1 --2- 172.21.15.88:0/3428930269 >> v2:172.21.15.154:3300/0 conn(0x7f122c073b10 0x7f122c0b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:29.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.575+0000 7f1230c7e640 1 -- 172.21.15.88:0/3428930269 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 222355689 0 0) 0x7f122c15eb10 con 0x7f122c15c100 2024-09-16T01:34:29.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.575+0000 7f1230c7e640 1 -- 172.21.15.88:0/3428930269 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1214003520 con 0x7f122c15c100 2024-09-16T01:34:29.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.575+0000 7f1230c7e640 1 -- 172.21.15.88:0/3428930269 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3327711321 0 0) 0x7f1214003520 con 0x7f122c15c100 2024-09-16T01:34:29.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.575+0000 7f1230c7e640 1 -- 172.21.15.88:0/3428930269 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f122c06e910 msgr2=0x7f122c0735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:29.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.575+0000 7f1230c7e640 1 --2- 172.21.15.88:0/3428930269 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f122c06e910 0x7f122c0735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:29.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.575+0000 7f1230c7e640 1 -- 172.21.15.88:0/3428930269 >> v2:172.21.15.154:3300/0 conn(0x7f122c073b10 msgr2=0x7f122c0b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:29.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.575+0000 7f1230c7e640 1 --2- 172.21.15.88:0/3428930269 >> v2:172.21.15.154:3300/0 conn(0x7f122c073b10 0x7f122c0b4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:29.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.575+0000 7f1230c7e640 1 -- 172.21.15.88:0/3428930269 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122c15f390 con 0x7f122c15c100 2024-09-16T01:34:29.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.575+0000 7f123147f640 1 --2- 172.21.15.88:0/3428930269 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f122c06e910 0x7f122c0735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:34:29.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.575+0000 7f1230c7e640 1 -- 172.21.15.88:0/3428930269 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f121c0019f0 con 0x7f122c15c100 2024-09-16T01:34:29.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.575+0000 7f1230c7e640 1 -- 172.21.15.88:0/3428930269 >> v1:172.21.15.88:6789/0 conn(0x7f122c15c100 legacy=0x7f122c06e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:29.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.575+0000 7f1230c7e640 1 --2- 172.21.15.88:0/3428930269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12140039b0 0x7f1214003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:29.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.575+0000 7f1230c7e640 1 --2- 172.21.15.88:0/3428930269 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f122c06e910 0x7f1214004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:29.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.575+0000 7f1230c7e640 1 --2- 172.21.15.88:0/3428930269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1214004d40 0x7f12140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:29.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.575+0000 7f1230c7e640 1 -- 172.21.15.88:0/3428930269 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f122c15edd0 con 0x7f1214004d40 2024-09-16T01:34:29.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.575+0000 7f1230c7e640 1 -- 172.21.15.88:0/3428930269 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f122c15f0b0 con 0x7f12140039b0 2024-09-16T01:34:29.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.575+0000 7f1230c7e640 1 -- 172.21.15.88:0/3428930269 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f121400b9c0 con 0x7f122c06e910 2024-09-16T01:34:29.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.575+0000 7f1232481640 1 --2- 172.21.15.88:0/3428930269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1214004d40 0x7f12140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:29.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.575+0000 7f1231c80640 1 --2- 172.21.15.88:0/3428930269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12140039b0 0x7f1214003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:29.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.575+0000 7f123147f640 1 --2- 172.21.15.88:0/3428930269 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f122c06e910 0x7f1214004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:29.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.576+0000 7f1232481640 1 -- 172.21.15.88:0/3428930269 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f122c06e910 msgr2=0x7f1214004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:29.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.576+0000 7f1232481640 1 --2- 172.21.15.88:0/3428930269 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f122c06e910 0x7f1214004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:29.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.576+0000 7f1232481640 1 -- 172.21.15.88:0/3428930269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12140039b0 msgr2=0x7f1214003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:29.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.576+0000 7f1232481640 1 --2- 172.21.15.88:0/3428930269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12140039b0 0x7f1214003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:29.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.576+0000 7f1232481640 1 -- 172.21.15.88:0/3428930269 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1214003520 con 0x7f1214004d40 2024-09-16T01:34:29.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.576+0000 7f1232481640 1 --2- 172.21.15.88:0/3428930269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1214004d40 0x7f12140051c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1224009ac0 tx=0x7f1214005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7758172f1355833f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:29.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.576+0000 7f1230c7e640 1 -- 172.21.15.88:0/3428930269 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1224036070 con 0x7f1214004d40 2024-09-16T01:34:29.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.576+0000 7f1230c7e640 1 -- 172.21.15.88:0/3428930269 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1224031400 con 0x7f1214004d40 2024-09-16T01:34:29.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.576+0000 7f1230c7e640 1 -- 172.21.15.88:0/3428930269 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1224009690 con 0x7f1214004d40 2024-09-16T01:34:29.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.577+0000 7f1233ee4640 1 -- 172.21.15.88:0/3428930269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1214004d40 msgr2=0x7f12140051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:29.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.577+0000 7f1233ee4640 1 --2- 172.21.15.88:0/3428930269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1214004d40 0x7f12140051c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1224009ac0 tx=0x7f1214005830 comp rx=0 tx=0).stop 2024-09-16T01:34:29.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.577+0000 7f1232481640 1 -- 172.21.15.88:0/3428930269 reap_dead start 2024-09-16T01:34:29.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.577+0000 7f1233ee4640 1 -- 172.21.15.88:0/3428930269 shutdown_connections 2024-09-16T01:34:29.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.577+0000 7f1233ee4640 1 -- 172.21.15.88:0/3428930269 >> 172.21.15.88:0/3428930269 conn(0x7f122c0b6ec0 msgr2=0x7f122c0b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:29.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.577+0000 7f1233ee4640 1 -- 172.21.15.88:0/3428930269 shutdown_connections 2024-09-16T01:34:29.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.577+0000 7f1233ee4640 1 -- 172.21.15.88:0/3428930269 wait complete. 2024-09-16T01:34:29.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.578+0000 7f1233ee4640 1 Processor -- start 2024-09-16T01:34:29.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.578+0000 7f1233ee4640 1 -- start start 2024-09-16T01:34:29.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.579+0000 7f1233ee4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1214004d40 0x7f122c0cff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:29.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.579+0000 7f1233ee4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f122c0b0940 0x7f122c14ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:29.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.579+0000 7f1233ee4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f122c0af2a0 0x7f122c0b2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:29.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.579+0000 7f1233ee4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f122c073f80 con 0x7f122c0af2a0 2024-09-16T01:34:29.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.579+0000 7f1233ee4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f122c15c4e0 con 0x7f122c0b0940 2024-09-16T01:34:29.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.579+0000 7f1233ee4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f122c074100 con 0x7f1214004d40 2024-09-16T01:34:29.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.579+0000 7f1232481640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f122c0af2a0 0x7f122c0b2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:29.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.579+0000 7f1232481640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f122c0af2a0 0x7f122c0b2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58348/0 (socket says 172.21.15.88:58348) 2024-09-16T01:34:29.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.579+0000 7f1232481640 1 -- 172.21.15.88:0/2353200906 learned_addr learned my addr 172.21.15.88:0/2353200906 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:29.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.579+0000 7f123147f640 1 --2- 172.21.15.88:0/2353200906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f122c0b0940 0x7f122c14ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:29.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.579+0000 7f1231c80640 1 --2- 172.21.15.88:0/2353200906 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1214004d40 0x7f122c0cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:29.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.579+0000 7f1232481640 1 -- 172.21.15.88:0/2353200906 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1214004d40 msgr2=0x7f122c0cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:29.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.579+0000 7f1232481640 1 --2- 172.21.15.88:0/2353200906 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1214004d40 0x7f122c0cff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:29.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.579+0000 7f1232481640 1 -- 172.21.15.88:0/2353200906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f122c0b0940 msgr2=0x7f122c14ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:29.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.579+0000 7f1232481640 1 --2- 172.21.15.88:0/2353200906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f122c0b0940 0x7f122c14ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:29.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.579+0000 7f1232481640 1 -- 172.21.15.88:0/2353200906 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122c0af750 con 0x7f122c0af2a0 2024-09-16T01:34:29.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.580+0000 7f1232481640 1 --2- 172.21.15.88:0/2353200906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f122c0af2a0 0x7f122c0b2370 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1224009ac0 tx=0x7f1224004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:29.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.580+0000 7f122a7fc640 1 -- 172.21.15.88:0/2353200906 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1224036040 con 0x7f122c0af2a0 2024-09-16T01:34:29.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.580+0000 7f122a7fc640 1 -- 172.21.15.88:0/2353200906 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1224041070 con 0x7f122c0af2a0 2024-09-16T01:34:29.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.580+0000 7f1233ee4640 1 -- 172.21.15.88:0/2353200906 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f122c0ceea0 con 0x7f122c0af2a0 2024-09-16T01:34:29.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.580+0000 7f122a7fc640 1 -- 172.21.15.88:0/2353200906 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1224009690 con 0x7f122c0af2a0 2024-09-16T01:34:29.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.580+0000 7f1233ee4640 1 -- 172.21.15.88:0/2353200906 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f122c0cf460 con 0x7f122c0af2a0 2024-09-16T01:34:29.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.581+0000 7f1233ee4640 1 -- 172.21.15.88:0/2353200906 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f122c0735d0 con 0x7f122c0af2a0 2024-09-16T01:34:29.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.581+0000 7f122a7fc640 1 -- 172.21.15.88:0/2353200906 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1224039070 con 0x7f122c0af2a0 2024-09-16T01:34:29.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.582+0000 7f122a7fc640 1 --2- 172.21.15.88:0/2353200906 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f121806c2b0 0x7f121806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:29.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.582+0000 7f1231c80640 1 --2- 172.21.15.88:0/2353200906 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f121806c2b0 0x7f121806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:29.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.582+0000 7f122a7fc640 1 -- 172.21.15.88:0/2353200906 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f12240aab70 con 0x7f122c0af2a0 2024-09-16T01:34:29.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.582+0000 7f1231c80640 1 --2- 172.21.15.88:0/2353200906 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f121806c2b0 0x7f121806e770 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f121c000f80 tx=0x7f121c006040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:29.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.586+0000 7f122a7fc640 1 -- 172.21.15.88:0/2353200906 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f122404a020 con 0x7f122c0af2a0 2024-09-16T01:34:29.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.749+0000 7f1233ee4640 1 -- 172.21.15.88:0/2353200906 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f122c15e9d0 con 0x7f121806c2b0 2024-09-16T01:34:29.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.755+0000 7f122a7fc640 1 -- 172.21.15.88:0/2353200906 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251948 (secure 0 0 0) 0x7f122c15e9d0 con 0x7f121806c2b0 2024-09-16T01:34:29.759 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:29.761 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:34:29.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.758+0000 7f1233ee4640 1 -- 172.21.15.88:0/2353200906 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f121806c2b0 msgr2=0x7f121806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:29.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.758+0000 7f1233ee4640 1 --2- 172.21.15.88:0/2353200906 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f121806c2b0 0x7f121806e770 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f121c000f80 tx=0x7f121c006040 comp rx=0 tx=0).stop 2024-09-16T01:34:29.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.758+0000 7f1233ee4640 1 -- 172.21.15.88:0/2353200906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f122c0af2a0 msgr2=0x7f122c0b2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:29.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.758+0000 7f1233ee4640 1 --2- 172.21.15.88:0/2353200906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f122c0af2a0 0x7f122c0b2370 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1224009ac0 tx=0x7f1224004730 comp rx=0 tx=0).stop 2024-09-16T01:34:29.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.758+0000 7f1233ee4640 1 -- 172.21.15.88:0/2353200906 shutdown_connections 2024-09-16T01:34:29.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.758+0000 7f1233ee4640 1 --2- 172.21.15.88:0/2353200906 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f121806c2b0 0x7f121806e770 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:29.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.758+0000 7f1233ee4640 1 --2- 172.21.15.88:0/2353200906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f122c0af2a0 0x7f122c0b2370 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:29.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.758+0000 7f1233ee4640 1 --2- 172.21.15.88:0/2353200906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f122c0b0940 0x7f122c14ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:29.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.758+0000 7f1233ee4640 1 --2- 172.21.15.88:0/2353200906 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1214004d40 0x7f122c0cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:29.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.758+0000 7f1233ee4640 1 -- 172.21.15.88:0/2353200906 >> 172.21.15.88:0/2353200906 conn(0x7f122c0b6ec0 msgr2=0x7f122c0b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:29.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.758+0000 7f1233ee4640 1 -- 172.21.15.88:0/2353200906 shutdown_connections 2024-09-16T01:34:29.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.758+0000 7f1233ee4640 1 -- 172.21.15.88:0/2353200906 wait complete. 2024-09-16T01:34:29.770 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-16T01:34:28.518332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219148,"kb_used_data":2888,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755572,"statfs":{"total":773094113280,"available":772869705728,"internally_reserved":0,"allocated":2957312,"data_stored":1775436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.346450"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611437+0000","last_change":"2024-09-16T01:34:22.195219+0000","last_active":"2024-09-16T01:34:23.611437+0000","last_peered":"2024-09-16T01:34:23.611437+0000","last_clean":"2024-09-16T01:34:23.611437+0000","last_became_active":"2024-09-16T01:34:22.194926+0000","last_became_peered":"2024-09-16T01:34:22.194926+0000","last_unstale":"2024-09-16T01:34:23.611437+0000","last_undegraded":"2024-09-16T01:34:23.611437+0000","last_fullsized":"2024-09-16T01:34:23.611437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:02.738433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513528+0000","last_change":"2024-09-16T01:34:22.186600+0000","last_active":"2024-09-16T01:34:23.513528+0000","last_peered":"2024-09-16T01:34:23.513528+0000","last_clean":"2024-09-16T01:34:23.513528+0000","last_became_active":"2024-09-16T01:34:22.186325+0000","last_became_peered":"2024-09-16T01:34:22.186325+0000","last_unstale":"2024-09-16T01:34:23.513528+0000","last_undegraded":"2024-09-16T01:34:23.513528+0000","last_fullsized":"2024-09-16T01:34:23.513528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:35:45.862526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609611+0000","last_change":"2024-09-16T01:34:22.187744+0000","last_active":"2024-09-16T01:34:23.609611+0000","last_peered":"2024-09-16T01:34:23.609611+0000","last_clean":"2024-09-16T01:34:23.609611+0000","last_became_active":"2024-09-16T01:34:22.186002+0000","last_became_peered":"2024-09-16T01:34:22.186002+0000","last_unstale":"2024-09-16T01:34:23.609611+0000","last_undegraded":"2024-09-16T01:34:23.609611+0000","last_fullsized":"2024-09-16T01:34:23.609611+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:38:50.126366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610430+0000","last_change":"2024-09-16T01:34:22.194231+0000","last_active":"2024-09-16T01:34:23.610430+0000","last_peered":"2024-09-16T01:34:23.610430+0000","last_clean":"2024-09-16T01:34:23.610430+0000","last_became_active":"2024-09-16T01:34:22.193969+0000","last_became_peered":"2024-09-16T01:34:22.193969+0000","last_unstale":"2024-09-16T01:34:23.610430+0000","last_undegraded":"2024-09-16T01:34:23.610430+0000","last_fullsized":"2024-09-16T01:34:23.610430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:35:45.862526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611192+0000","last_change":"2024-09-16T01:34:22.191565+0000","last_active":"2024-09-16T01:34:23.611192+0000","last_peered":"2024-09-16T01:34:23.611192+0000","last_clean":"2024-09-16T01:34:23.611192+0000","last_became_active":"2024-09-16T01:34:22.191281+0000","last_became_peered":"2024-09-16T01:34:22.191281+0000","last_unstale":"2024-09-16T01:34:23.611192+0000","last_undegraded":"2024-09-16T01:34:23.611192+0000","last_fullsized":"2024-09-16T01:34:23.611192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:49.814244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610462+0000","last_change":"2024-09-16T01:34:22.199917+0000","last_active":"2024-09-16T01:34:23.610462+0000","last_peered":"2024-09-16T01:34:23.610462+0000","last_clean":"2024-09-16T01:34:23.610462+0000","last_became_active":"2024-09-16T01:34:22.198949+0000","last_became_peered":"2024-09-16T01:34:22.198949+0000","last_unstale":"2024-09-16T01:34:23.610462+0000","last_undegraded":"2024-09-16T01:34:23.610462+0000","last_fullsized":"2024-09-16T01:34:23.610462+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:48:44.596067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610965+0000","last_change":"2024-09-16T01:34:22.191158+0000","last_active":"2024-09-16T01:34:23.610965+0000","last_peered":"2024-09-16T01:34:23.610965+0000","last_clean":"2024-09-16T01:34:23.610965+0000","last_became_active":"2024-09-16T01:34:22.190856+0000","last_became_peered":"2024-09-16T01:34:22.190856+0000","last_unstale":"2024-09-16T01:34:23.610965+0000","last_undegraded":"2024-09-16T01:34:23.610965+0000","last_fullsized":"2024-09-16T01:34:23.610965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:05.939767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.187946+0000","last_change":"2024-09-16T01:34:22.200461+0000","last_active":"2024-09-16T01:34:23.187946+0000","last_peered":"2024-09-16T01:34:23.187946+0000","last_clean":"2024-09-16T01:34:23.187946+0000","last_became_active":"2024-09-16T01:34:22.200181+0000","last_became_peered":"2024-09-16T01:34:22.200181+0000","last_unstale":"2024-09-16T01:34:23.187946+0000","last_undegraded":"2024-09-16T01:34:23.187946+0000","last_fullsized":"2024-09-16T01:34:23.187946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:36:03.508489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611582+0000","last_change":"2024-09-16T01:34:22.201897+0000","last_active":"2024-09-16T01:34:23.611582+0000","last_peered":"2024-09-16T01:34:23.611582+0000","last_clean":"2024-09-16T01:34:23.611582+0000","last_became_active":"2024-09-16T01:34:22.201595+0000","last_became_peered":"2024-09-16T01:34:22.201595+0000","last_unstale":"2024-09-16T01:34:23.611582+0000","last_undegraded":"2024-09-16T01:34:23.611582+0000","last_fullsized":"2024-09-16T01:34:23.611582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:07.782481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191787+0000","last_change":"2024-09-16T01:34:22.195243+0000","last_active":"2024-09-16T01:34:23.191787+0000","last_peered":"2024-09-16T01:34:23.191787+0000","last_clean":"2024-09-16T01:34:23.191787+0000","last_became_active":"2024-09-16T01:34:22.194813+0000","last_became_peered":"2024-09-16T01:34:22.194813+0000","last_unstale":"2024-09-16T01:34:23.191787+0000","last_undegraded":"2024-09-16T01:34:23.191787+0000","last_fullsized":"2024-09-16T01:34:23.191787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:11:56.336591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191528+0000","last_change":"2024-09-16T01:34:22.197385+0000","last_active":"2024-09-16T01:34:23.191528+0000","last_peered":"2024-09-16T01:34:23.191528+0000","last_clean":"2024-09-16T01:34:23.191528+0000","last_became_active":"2024-09-16T01:34:22.196747+0000","last_became_peered":"2024-09-16T01:34:22.196747+0000","last_unstale":"2024-09-16T01:34:23.191528+0000","last_undegraded":"2024-09-16T01:34:23.191528+0000","last_fullsized":"2024-09-16T01:34:23.191528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:14:18.368513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.190899+0000","last_change":"2024-09-16T01:34:22.181901+0000","last_active":"2024-09-16T01:34:23.190899+0000","last_peered":"2024-09-16T01:34:23.190899+0000","last_clean":"2024-09-16T01:34:23.190899+0000","last_became_active":"2024-09-16T01:34:22.181627+0000","last_became_peered":"2024-09-16T01:34:22.181627+0000","last_unstale":"2024-09-16T01:34:23.190899+0000","last_undegraded":"2024-09-16T01:34:23.190899+0000","last_fullsized":"2024-09-16T01:34:23.190899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:31:42.191103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188450+0000","last_change":"2024-09-16T01:34:22.192960+0000","last_active":"2024-09-16T01:34:23.188450+0000","last_peered":"2024-09-16T01:34:23.188450+0000","last_clean":"2024-09-16T01:34:23.188450+0000","last_became_active":"2024-09-16T01:34:22.192664+0000","last_became_peered":"2024-09-16T01:34:22.192664+0000","last_unstale":"2024-09-16T01:34:23.188450+0000","last_undegraded":"2024-09-16T01:34:23.188450+0000","last_fullsized":"2024-09-16T01:34:23.188450+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:56:56.990459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609665+0000","last_change":"2024-09-16T01:34:22.192223+0000","last_active":"2024-09-16T01:34:23.609665+0000","last_peered":"2024-09-16T01:34:23.609665+0000","last_clean":"2024-09-16T01:34:23.609665+0000","last_became_active":"2024-09-16T01:34:22.190787+0000","last_became_peered":"2024-09-16T01:34:22.190787+0000","last_unstale":"2024-09-16T01:34:23.609665+0000","last_undegraded":"2024-09-16T01:34:23.609665+0000","last_fullsized":"2024-09-16T01:34:23.609665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:40:27.589986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513177+0000","last_change":"2024-09-16T01:34:22.194793+0000","last_active":"2024-09-16T01:34:23.513177+0000","last_peered":"2024-09-16T01:34:23.513177+0000","last_clean":"2024-09-16T01:34:23.513177+0000","last_became_active":"2024-09-16T01:34:22.194530+0000","last_became_peered":"2024-09-16T01:34:22.194530+0000","last_unstale":"2024-09-16T01:34:23.513177+0000","last_undegraded":"2024-09-16T01:34:23.513177+0000","last_fullsized":"2024-09-16T01:34:23.513177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:10:23.471025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610102+0000","last_change":"2024-09-16T01:34:22.198482+0000","last_active":"2024-09-16T01:34:23.610102+0000","last_peered":"2024-09-16T01:34:23.610102+0000","last_clean":"2024-09-16T01:34:23.610102+0000","last_became_active":"2024-09-16T01:34:22.198216+0000","last_became_peered":"2024-09-16T01:34:22.198216+0000","last_unstale":"2024-09-16T01:34:23.610102+0000","last_undegraded":"2024-09-16T01:34:23.610102+0000","last_fullsized":"2024-09-16T01:34:23.610102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:04.740799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610673+0000","last_change":"2024-09-16T01:34:22.198300+0000","last_active":"2024-09-16T01:34:23.610673+0000","last_peered":"2024-09-16T01:34:23.610673+0000","last_clean":"2024-09-16T01:34:23.610673+0000","last_became_active":"2024-09-16T01:34:22.198024+0000","last_became_peered":"2024-09-16T01:34:22.198024+0000","last_unstale":"2024-09-16T01:34:23.610673+0000","last_undegraded":"2024-09-16T01:34:23.610673+0000","last_fullsized":"2024-09-16T01:34:23.610673+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:49.814244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609788+0000","last_change":"2024-09-16T01:34:22.194454+0000","last_active":"2024-09-16T01:34:23.609788+0000","last_peered":"2024-09-16T01:34:23.609788+0000","last_clean":"2024-09-16T01:34:23.609788+0000","last_became_active":"2024-09-16T01:34:22.193936+0000","last_became_peered":"2024-09-16T01:34:22.193936+0000","last_unstale":"2024-09-16T01:34:23.609788+0000","last_undegraded":"2024-09-16T01:34:23.609788+0000","last_fullsized":"2024-09-16T01:34:23.609788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:42:44.444634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611563+0000","last_change":"2024-09-16T01:34:22.191670+0000","last_active":"2024-09-16T01:34:23.611563+0000","last_peered":"2024-09-16T01:34:23.611563+0000","last_clean":"2024-09-16T01:34:23.611563+0000","last_became_active":"2024-09-16T01:34:22.191341+0000","last_became_peered":"2024-09-16T01:34:22.191341+0000","last_unstale":"2024-09-16T01:34:23.611563+0000","last_undegraded":"2024-09-16T01:34:23.611563+0000","last_fullsized":"2024-09-16T01:34:23.611563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:50:20.263787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191662+0000","last_change":"2024-09-16T01:34:22.193121+0000","last_active":"2024-09-16T01:34:23.191662+0000","last_peered":"2024-09-16T01:34:23.191662+0000","last_clean":"2024-09-16T01:34:23.191662+0000","last_became_active":"2024-09-16T01:34:22.192835+0000","last_became_peered":"2024-09-16T01:34:22.192835+0000","last_unstale":"2024-09-16T01:34:23.191662+0000","last_undegraded":"2024-09-16T01:34:23.191662+0000","last_fullsized":"2024-09-16T01:34:23.191662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:55:12.145787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609601+0000","last_change":"2024-09-16T01:34:22.196878+0000","last_active":"2024-09-16T01:34:23.609601+0000","last_peered":"2024-09-16T01:34:23.609601+0000","last_clean":"2024-09-16T01:34:23.609601+0000","last_became_active":"2024-09-16T01:34:22.196451+0000","last_became_peered":"2024-09-16T01:34:22.196451+0000","last_unstale":"2024-09-16T01:34:23.609601+0000","last_undegraded":"2024-09-16T01:34:23.609601+0000","last_fullsized":"2024-09-16T01:34:23.609601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:42:01.118172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610529+0000","last_change":"2024-09-16T01:34:22.196889+0000","last_active":"2024-09-16T01:34:23.610529+0000","last_peered":"2024-09-16T01:34:23.610529+0000","last_clean":"2024-09-16T01:34:23.610529+0000","last_became_active":"2024-09-16T01:34:22.196583+0000","last_became_peered":"2024-09-16T01:34:22.196583+0000","last_unstale":"2024-09-16T01:34:23.610529+0000","last_undegraded":"2024-09-16T01:34:23.610529+0000","last_fullsized":"2024-09-16T01:34:23.610529+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:05.939767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188009+0000","last_change":"2024-09-16T01:34:22.196090+0000","last_active":"2024-09-16T01:34:23.188009+0000","last_peered":"2024-09-16T01:34:23.188009+0000","last_clean":"2024-09-16T01:34:23.188009+0000","last_became_active":"2024-09-16T01:34:22.195441+0000","last_became_peered":"2024-09-16T01:34:22.195441+0000","last_unstale":"2024-09-16T01:34:23.188009+0000","last_undegraded":"2024-09-16T01:34:23.188009+0000","last_fullsized":"2024-09-16T01:34:23.188009+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:14:18.368513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610192+0000","last_change":"2024-09-16T01:34:22.192211+0000","last_active":"2024-09-16T01:34:23.610192+0000","last_peered":"2024-09-16T01:34:23.610192+0000","last_clean":"2024-09-16T01:34:23.610192+0000","last_became_active":"2024-09-16T01:34:22.191921+0000","last_became_peered":"2024-09-16T01:34:22.191921+0000","last_unstale":"2024-09-16T01:34:23.610192+0000","last_undegraded":"2024-09-16T01:34:23.610192+0000","last_fullsized":"2024-09-16T01:34:23.610192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:37.582768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188086+0000","last_change":"2024-09-16T01:34:22.195710+0000","last_active":"2024-09-16T01:34:23.188086+0000","last_peered":"2024-09-16T01:34:23.188086+0000","last_clean":"2024-09-16T01:34:23.188086+0000","last_became_active":"2024-09-16T01:34:22.195385+0000","last_became_peered":"2024-09-16T01:34:22.195385+0000","last_unstale":"2024-09-16T01:34:23.188086+0000","last_undegraded":"2024-09-16T01:34:23.188086+0000","last_fullsized":"2024-09-16T01:34:23.188086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:32:14.878649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191846+0000","last_change":"2024-09-16T01:34:22.193276+0000","last_active":"2024-09-16T01:34:23.191846+0000","last_peered":"2024-09-16T01:34:23.191846+0000","last_clean":"2024-09-16T01:34:23.191846+0000","last_became_active":"2024-09-16T01:34:22.192973+0000","last_became_peered":"2024-09-16T01:34:22.192973+0000","last_unstale":"2024-09-16T01:34:23.191846+0000","last_undegraded":"2024-09-16T01:34:23.191846+0000","last_fullsized":"2024-09-16T01:34:23.191846+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:16:41.426352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513373+0000","last_change":"2024-09-16T01:34:22.189813+0000","last_active":"2024-09-16T01:34:23.513373+0000","last_peered":"2024-09-16T01:34:23.513373+0000","last_clean":"2024-09-16T01:34:23.513373+0000","last_became_active":"2024-09-16T01:34:22.189535+0000","last_became_peered":"2024-09-16T01:34:22.189535+0000","last_unstale":"2024-09-16T01:34:23.513373+0000","last_undegraded":"2024-09-16T01:34:23.513373+0000","last_fullsized":"2024-09-16T01:34:23.513373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:37.582768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188382+0000","last_change":"2024-09-16T01:34:22.195577+0000","last_active":"2024-09-16T01:34:23.188382+0000","last_peered":"2024-09-16T01:34:23.188382+0000","last_clean":"2024-09-16T01:34:23.188382+0000","last_became_active":"2024-09-16T01:34:22.195271+0000","last_became_peered":"2024-09-16T01:34:22.195271+0000","last_unstale":"2024-09-16T01:34:23.188382+0000","last_undegraded":"2024-09-16T01:34:23.188382+0000","last_fullsized":"2024-09-16T01:34:23.188382+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:13.825111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609899+0000","last_change":"2024-09-16T01:34:22.189995+0000","last_active":"2024-09-16T01:34:23.609899+0000","last_peered":"2024-09-16T01:34:23.609899+0000","last_clean":"2024-09-16T01:34:23.609899+0000","last_became_active":"2024-09-16T01:34:22.187919+0000","last_became_peered":"2024-09-16T01:34:22.187919+0000","last_unstale":"2024-09-16T01:34:23.609899+0000","last_undegraded":"2024-09-16T01:34:23.609899+0000","last_fullsized":"2024-09-16T01:34:23.609899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:14:18.368513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610796+0000","last_change":"2024-09-16T01:34:22.194365+0000","last_active":"2024-09-16T01:34:23.610796+0000","last_peered":"2024-09-16T01:34:23.610796+0000","last_clean":"2024-09-16T01:34:23.610796+0000","last_became_active":"2024-09-16T01:34:22.194094+0000","last_became_peered":"2024-09-16T01:34:22.194094+0000","last_unstale":"2024-09-16T01:34:23.610796+0000","last_undegraded":"2024-09-16T01:34:23.610796+0000","last_fullsized":"2024-09-16T01:34:23.610796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:02.738433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609476+0000","last_change":"2024-09-16T01:34:22.195821+0000","last_active":"2024-09-16T01:34:23.609476+0000","last_peered":"2024-09-16T01:34:23.609476+0000","last_clean":"2024-09-16T01:34:23.609476+0000","last_became_active":"2024-09-16T01:34:22.194968+0000","last_became_peered":"2024-09-16T01:34:22.194968+0000","last_unstale":"2024-09-16T01:34:23.609476+0000","last_undegraded":"2024-09-16T01:34:23.609476+0000","last_fullsized":"2024-09-16T01:34:23.609476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:15:36.858098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191308+0000","last_change":"2024-09-16T01:34:22.189557+0000","last_active":"2024-09-16T01:34:23.191308+0000","last_peered":"2024-09-16T01:34:23.191308+0000","last_clean":"2024-09-16T01:34:23.191308+0000","last_became_active":"2024-09-16T01:34:22.189252+0000","last_became_peered":"2024-09-16T01:34:22.189252+0000","last_unstale":"2024-09-16T01:34:23.191308+0000","last_undegraded":"2024-09-16T01:34:23.191308+0000","last_fullsized":"2024-09-16T01:34:23.191308+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:40:27.589986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.190446+0000","last_change":"2024-09-16T01:34:22.187071+0000","last_active":"2024-09-16T01:34:23.190446+0000","last_peered":"2024-09-16T01:34:23.190446+0000","last_clean":"2024-09-16T01:34:23.190446+0000","last_became_active":"2024-09-16T01:34:22.186773+0000","last_became_peered":"2024-09-16T01:34:22.186773+0000","last_unstale":"2024-09-16T01:34:23.190446+0000","last_undegraded":"2024-09-16T01:34:23.190446+0000","last_fullsized":"2024-09-16T01:34:23.190446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:22:27.895343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188539+0000","last_change":"2024-09-16T01:34:21.187730+0000","last_active":"2024-09-16T01:34:23.188539+0000","last_peered":"2024-09-16T01:34:23.188539+0000","last_clean":"2024-09-16T01:34:23.188539+0000","last_became_active":"2024-09-16T01:34:21.187421+0000","last_became_peered":"2024-09-16T01:34:21.187421+0000","last_unstale":"2024-09-16T01:34:23.188539+0000","last_undegraded":"2024-09-16T01:34:23.188539+0000","last_fullsized":"2024-09-16T01:34:23.188539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:56.560008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609871+0000","last_change":"2024-09-16T01:34:22.197079+0000","last_active":"2024-09-16T01:34:23.609871+0000","last_peered":"2024-09-16T01:34:23.609871+0000","last_clean":"2024-09-16T01:34:23.609871+0000","last_became_active":"2024-09-16T01:34:22.196793+0000","last_became_peered":"2024-09-16T01:34:22.196793+0000","last_unstale":"2024-09-16T01:34:23.609871+0000","last_undegraded":"2024-09-16T01:34:23.609871+0000","last_fullsized":"2024-09-16T01:34:23.609871+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:16:41.426352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191283+0000","last_change":"2024-09-16T01:34:21.188731+0000","last_active":"2024-09-16T01:34:23.191283+0000","last_peered":"2024-09-16T01:34:23.191283+0000","last_clean":"2024-09-16T01:34:23.191283+0000","last_became_active":"2024-09-16T01:34:21.188477+0000","last_became_peered":"2024-09-16T01:34:21.188477+0000","last_unstale":"2024-09-16T01:34:23.191283+0000","last_undegraded":"2024-09-16T01:34:23.191283+0000","last_fullsized":"2024-09-16T01:34:23.191283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:58:12.138025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188059+0000","last_change":"2024-09-16T01:34:21.187868+0000","last_active":"2024-09-16T01:34:23.188059+0000","last_peered":"2024-09-16T01:34:23.188059+0000","last_clean":"2024-09-16T01:34:23.188059+0000","last_became_active":"2024-09-16T01:34:21.187558+0000","last_became_peered":"2024-09-16T01:34:21.187558+0000","last_unstale":"2024-09-16T01:34:23.188059+0000","last_undegraded":"2024-09-16T01:34:23.188059+0000","last_fullsized":"2024-09-16T01:34:23.188059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:58:12.138025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188157+0000","last_change":"2024-09-16T01:34:22.195132+0000","last_active":"2024-09-16T01:34:23.188157+0000","last_peered":"2024-09-16T01:34:23.188157+0000","last_clean":"2024-09-16T01:34:23.188157+0000","last_became_active":"2024-09-16T01:34:22.194861+0000","last_became_peered":"2024-09-16T01:34:22.194861+0000","last_unstale":"2024-09-16T01:34:23.188157+0000","last_undegraded":"2024-09-16T01:34:23.188157+0000","last_fullsized":"2024-09-16T01:34:23.188157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:11:56.336591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610304+0000","last_change":"2024-09-16T01:34:21.188039+0000","last_active":"2024-09-16T01:34:23.610304+0000","last_peered":"2024-09-16T01:34:23.610304+0000","last_clean":"2024-09-16T01:34:23.610304+0000","last_became_active":"2024-09-16T01:34:21.187743+0000","last_became_peered":"2024-09-16T01:34:21.187743+0000","last_unstale":"2024-09-16T01:34:23.610304+0000","last_undegraded":"2024-09-16T01:34:23.610304+0000","last_fullsized":"2024-09-16T01:34:23.610304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:50:53.737531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513172+0000","last_change":"2024-09-16T01:34:22.187569+0000","last_active":"2024-09-16T01:34:23.513172+0000","last_peered":"2024-09-16T01:34:23.513172+0000","last_clean":"2024-09-16T01:34:23.513172+0000","last_became_active":"2024-09-16T01:34:22.187301+0000","last_became_peered":"2024-09-16T01:34:22.187301+0000","last_unstale":"2024-09-16T01:34:23.513172+0000","last_undegraded":"2024-09-16T01:34:23.513172+0000","last_fullsized":"2024-09-16T01:34:23.513172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:46:31.545461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191499+0000","last_change":"2024-09-16T01:34:21.187827+0000","last_active":"2024-09-16T01:34:23.191499+0000","last_peered":"2024-09-16T01:34:23.191499+0000","last_clean":"2024-09-16T01:34:23.191499+0000","last_became_active":"2024-09-16T01:34:21.187542+0000","last_became_peered":"2024-09-16T01:34:21.187542+0000","last_unstale":"2024-09-16T01:34:23.191499+0000","last_undegraded":"2024-09-16T01:34:23.191499+0000","last_fullsized":"2024-09-16T01:34:23.191499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:18:50.444668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610494+0000","last_change":"2024-09-16T01:34:22.197484+0000","last_active":"2024-09-16T01:34:23.610494+0000","last_peered":"2024-09-16T01:34:23.610494+0000","last_clean":"2024-09-16T01:34:23.610494+0000","last_became_active":"2024-09-16T01:34:22.197209+0000","last_became_peered":"2024-09-16T01:34:22.197209+0000","last_unstale":"2024-09-16T01:34:23.610494+0000","last_undegraded":"2024-09-16T01:34:23.610494+0000","last_fullsized":"2024-09-16T01:34:23.610494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:06.384195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610564+0000","last_change":"2024-09-16T01:34:21.190619+0000","last_active":"2024-09-16T01:34:23.610564+0000","last_peered":"2024-09-16T01:34:23.610564+0000","last_clean":"2024-09-16T01:34:23.610564+0000","last_became_active":"2024-09-16T01:34:21.188943+0000","last_became_peered":"2024-09-16T01:34:21.188943+0000","last_unstale":"2024-09-16T01:34:23.610564+0000","last_undegraded":"2024-09-16T01:34:23.610564+0000","last_fullsized":"2024-09-16T01:34:23.610564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:47:09.976612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609914+0000","last_change":"2024-09-16T01:34:22.198543+0000","last_active":"2024-09-16T01:34:23.609914+0000","last_peered":"2024-09-16T01:34:23.609914+0000","last_clean":"2024-09-16T01:34:23.609914+0000","last_became_active":"2024-09-16T01:34:22.198283+0000","last_became_peered":"2024-09-16T01:34:22.198283+0000","last_unstale":"2024-09-16T01:34:23.609914+0000","last_undegraded":"2024-09-16T01:34:23.609914+0000","last_fullsized":"2024-09-16T01:34:23.609914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:32:05.208969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513419+0000","last_change":"2024-09-16T01:34:21.189485+0000","last_active":"2024-09-16T01:34:23.513419+0000","last_peered":"2024-09-16T01:34:23.513419+0000","last_clean":"2024-09-16T01:34:23.513419+0000","last_became_active":"2024-09-16T01:34:21.189201+0000","last_became_peered":"2024-09-16T01:34:21.189201+0000","last_unstale":"2024-09-16T01:34:23.513419+0000","last_undegraded":"2024-09-16T01:34:23.513419+0000","last_fullsized":"2024-09-16T01:34:23.513419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:01:27.300067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.190949+0000","last_change":"2024-09-16T01:34:22.194435+0000","last_active":"2024-09-16T01:34:23.190949+0000","last_peered":"2024-09-16T01:34:23.190949+0000","last_clean":"2024-09-16T01:34:23.190949+0000","last_became_active":"2024-09-16T01:34:22.194058+0000","last_became_peered":"2024-09-16T01:34:22.194058+0000","last_unstale":"2024-09-16T01:34:23.190949+0000","last_undegraded":"2024-09-16T01:34:23.190949+0000","last_fullsized":"2024-09-16T01:34:23.190949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:36:45.083401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191725+0000","last_change":"2024-09-16T01:34:21.185553+0000","last_active":"2024-09-16T01:34:23.191725+0000","last_peered":"2024-09-16T01:34:23.191725+0000","last_clean":"2024-09-16T01:34:23.191725+0000","last_became_active":"2024-09-16T01:34:21.185264+0000","last_became_peered":"2024-09-16T01:34:21.185264+0000","last_unstale":"2024-09-16T01:34:23.191725+0000","last_undegraded":"2024-09-16T01:34:23.191725+0000","last_fullsized":"2024-09-16T01:34:23.191725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:22:26.887709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611437+0000","last_change":"2024-09-16T01:34:22.191352+0000","last_active":"2024-09-16T01:34:23.611437+0000","last_peered":"2024-09-16T01:34:23.611437+0000","last_clean":"2024-09-16T01:34:23.611437+0000","last_became_active":"2024-09-16T01:34:22.191027+0000","last_became_peered":"2024-09-16T01:34:22.191027+0000","last_unstale":"2024-09-16T01:34:23.611437+0000","last_undegraded":"2024-09-16T01:34:23.611437+0000","last_fullsized":"2024-09-16T01:34:23.611437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:18.860118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609920+0000","last_change":"2024-09-16T01:34:21.193072+0000","last_active":"2024-09-16T01:34:23.609920+0000","last_peered":"2024-09-16T01:34:23.609920+0000","last_clean":"2024-09-16T01:34:23.609920+0000","last_became_active":"2024-09-16T01:34:21.192769+0000","last_became_peered":"2024-09-16T01:34:21.192769+0000","last_unstale":"2024-09-16T01:34:23.609920+0000","last_undegraded":"2024-09-16T01:34:23.609920+0000","last_fullsized":"2024-09-16T01:34:23.609920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:00.162566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609665+0000","last_change":"2024-09-16T01:34:22.195668+0000","last_active":"2024-09-16T01:34:23.609665+0000","last_peered":"2024-09-16T01:34:23.609665+0000","last_clean":"2024-09-16T01:34:23.609665+0000","last_became_active":"2024-09-16T01:34:22.195407+0000","last_became_peered":"2024-09-16T01:34:22.195407+0000","last_unstale":"2024-09-16T01:34:23.609665+0000","last_undegraded":"2024-09-16T01:34:23.609665+0000","last_fullsized":"2024-09-16T01:34:23.609665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:36:03.508489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610939+0000","last_change":"2024-09-16T01:34:21.186203+0000","last_active":"2024-09-16T01:34:23.610939+0000","last_peered":"2024-09-16T01:34:23.610939+0000","last_clean":"2024-09-16T01:34:23.610939+0000","last_became_active":"2024-09-16T01:34:21.185935+0000","last_became_peered":"2024-09-16T01:34:21.185935+0000","last_unstale":"2024-09-16T01:34:23.610939+0000","last_undegraded":"2024-09-16T01:34:23.610939+0000","last_fullsized":"2024-09-16T01:34:23.610939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:29.860344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611192+0000","last_change":"2024-09-16T01:34:22.196044+0000","last_active":"2024-09-16T01:34:23.611192+0000","last_peered":"2024-09-16T01:34:23.611192+0000","last_clean":"2024-09-16T01:34:23.611192+0000","last_became_active":"2024-09-16T01:34:22.195783+0000","last_became_peered":"2024-09-16T01:34:22.195783+0000","last_unstale":"2024-09-16T01:34:23.611192+0000","last_undegraded":"2024-09-16T01:34:23.611192+0000","last_fullsized":"2024-09-16T01:34:23.611192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:03.591531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191432+0000","last_change":"2024-09-16T01:34:21.184919+0000","last_active":"2024-09-16T01:34:23.191432+0000","last_peered":"2024-09-16T01:34:23.191432+0000","last_clean":"2024-09-16T01:34:23.191432+0000","last_became_active":"2024-09-16T01:34:21.184542+0000","last_became_peered":"2024-09-16T01:34:21.184542+0000","last_unstale":"2024-09-16T01:34:23.191432+0000","last_undegraded":"2024-09-16T01:34:23.191432+0000","last_fullsized":"2024-09-16T01:34:23.191432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:56.560008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191412+0000","last_change":"2024-09-16T01:34:22.187373+0000","last_active":"2024-09-16T01:34:23.191412+0000","last_peered":"2024-09-16T01:34:23.191412+0000","last_clean":"2024-09-16T01:34:23.191412+0000","last_became_active":"2024-09-16T01:34:22.186468+0000","last_became_peered":"2024-09-16T01:34:22.186468+0000","last_unstale":"2024-09-16T01:34:23.191412+0000","last_undegraded":"2024-09-16T01:34:23.191412+0000","last_fullsized":"2024-09-16T01:34:23.191412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:36:03.508489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609191+0000","last_change":"2024-09-16T01:34:21.189797+0000","last_active":"2024-09-16T01:34:23.609191+0000","last_peered":"2024-09-16T01:34:23.609191+0000","last_clean":"2024-09-16T01:34:23.609191+0000","last_became_active":"2024-09-16T01:34:21.189512+0000","last_became_peered":"2024-09-16T01:34:21.189512+0000","last_unstale":"2024-09-16T01:34:23.609191+0000","last_undegraded":"2024-09-16T01:34:23.609191+0000","last_fullsized":"2024-09-16T01:34:23.609191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:01:27.300067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611069+0000","last_change":"2024-09-16T01:34:22.195318+0000","last_active":"2024-09-16T01:34:23.611069+0000","last_peered":"2024-09-16T01:34:23.611069+0000","last_clean":"2024-09-16T01:34:23.611069+0000","last_became_active":"2024-09-16T01:34:22.195051+0000","last_became_peered":"2024-09-16T01:34:22.195051+0000","last_unstale":"2024-09-16T01:34:23.611069+0000","last_undegraded":"2024-09-16T01:34:23.611069+0000","last_fullsized":"2024-09-16T01:34:23.611069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:06.384195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.190601+0000","last_change":"2024-09-16T01:34:21.191907+0000","last_active":"2024-09-16T01:34:23.190601+0000","last_peered":"2024-09-16T01:34:23.190601+0000","last_clean":"2024-09-16T01:34:23.190601+0000","last_became_active":"2024-09-16T01:34:21.191591+0000","last_became_peered":"2024-09-16T01:34:21.191591+0000","last_unstale":"2024-09-16T01:34:23.190601+0000","last_undegraded":"2024-09-16T01:34:23.190601+0000","last_fullsized":"2024-09-16T01:34:23.190601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:50:28.630517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611504+0000","last_change":"2024-09-16T01:34:22.194591+0000","last_active":"2024-09-16T01:34:23.611504+0000","last_peered":"2024-09-16T01:34:23.611504+0000","last_clean":"2024-09-16T01:34:23.611504+0000","last_became_active":"2024-09-16T01:34:22.194317+0000","last_became_peered":"2024-09-16T01:34:22.194317+0000","last_unstale":"2024-09-16T01:34:23.611504+0000","last_undegraded":"2024-09-16T01:34:23.611504+0000","last_fullsized":"2024-09-16T01:34:23.611504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:45.928615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609751+0000","last_change":"2024-09-16T01:34:21.190706+0000","last_active":"2024-09-16T01:34:23.609751+0000","last_peered":"2024-09-16T01:34:23.609751+0000","last_clean":"2024-09-16T01:34:23.609751+0000","last_became_active":"2024-09-16T01:34:21.189692+0000","last_became_peered":"2024-09-16T01:34:21.189692+0000","last_unstale":"2024-09-16T01:34:23.609751+0000","last_undegraded":"2024-09-16T01:34:23.609751+0000","last_fullsized":"2024-09-16T01:34:23.609751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:11:30.497159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611648+0000","last_change":"2024-09-16T01:34:22.194506+0000","last_active":"2024-09-16T01:34:23.611648+0000","last_peered":"2024-09-16T01:34:23.611648+0000","last_clean":"2024-09-16T01:34:23.611648+0000","last_became_active":"2024-09-16T01:34:22.194223+0000","last_became_peered":"2024-09-16T01:34:22.194223+0000","last_unstale":"2024-09-16T01:34:23.611648+0000","last_undegraded":"2024-09-16T01:34:23.611648+0000","last_fullsized":"2024-09-16T01:34:23.611648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:19.506778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513439+0000","last_change":"2024-09-16T01:34:21.189382+0000","last_active":"2024-09-16T01:34:23.513439+0000","last_peered":"2024-09-16T01:34:23.513439+0000","last_clean":"2024-09-16T01:34:23.513439+0000","last_became_active":"2024-09-16T01:34:21.188897+0000","last_became_peered":"2024-09-16T01:34:21.188897+0000","last_unstale":"2024-09-16T01:34:23.513439+0000","last_undegraded":"2024-09-16T01:34:23.513439+0000","last_fullsized":"2024-09-16T01:34:23.513439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:29.860344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610830+0000","last_change":"2024-09-16T01:34:22.192308+0000","last_active":"2024-09-16T01:34:23.610830+0000","last_peered":"2024-09-16T01:34:23.610830+0000","last_clean":"2024-09-16T01:34:23.610830+0000","last_became_active":"2024-09-16T01:34:22.191748+0000","last_became_peered":"2024-09-16T01:34:22.191748+0000","last_unstale":"2024-09-16T01:34:23.610830+0000","last_undegraded":"2024-09-16T01:34:23.610830+0000","last_fullsized":"2024-09-16T01:34:23.610830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:45.928615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609394+0000","last_change":"2024-09-16T01:34:21.188078+0000","last_active":"2024-09-16T01:34:23.609394+0000","last_peered":"2024-09-16T01:34:23.609394+0000","last_clean":"2024-09-16T01:34:23.609394+0000","last_became_active":"2024-09-16T01:34:21.187822+0000","last_became_peered":"2024-09-16T01:34:21.187822+0000","last_unstale":"2024-09-16T01:34:23.609394+0000","last_undegraded":"2024-09-16T01:34:23.609394+0000","last_fullsized":"2024-09-16T01:34:23.609394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:29.860344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611369+0000","last_change":"2024-09-16T01:34:22.202386+0000","last_active":"2024-09-16T01:34:23.611369+0000","last_peered":"2024-09-16T01:34:23.611369+0000","last_clean":"2024-09-16T01:34:23.611369+0000","last_became_active":"2024-09-16T01:34:22.202089+0000","last_became_peered":"2024-09-16T01:34:22.202089+0000","last_unstale":"2024-09-16T01:34:23.611369+0000","last_undegraded":"2024-09-16T01:34:23.611369+0000","last_fullsized":"2024-09-16T01:34:23.611369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:07.244892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610330+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:34:23.610330+0000","last_peered":"2024-09-16T01:34:23.610330+0000","last_clean":"2024-09-16T01:34:23.610330+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:34:23.610330+0000","last_undegraded":"2024-09-16T01:34:23.610330+0000","last_fullsized":"2024-09-16T01:34:23.610330+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610276+0000","last_change":"2024-09-16T01:34:21.183119+0000","last_active":"2024-09-16T01:34:23.610276+0000","last_peered":"2024-09-16T01:34:23.610276+0000","last_clean":"2024-09-16T01:34:23.610276+0000","last_became_active":"2024-09-16T01:34:21.182740+0000","last_became_peered":"2024-09-16T01:34:21.182740+0000","last_unstale":"2024-09-16T01:34:23.610276+0000","last_undegraded":"2024-09-16T01:34:23.610276+0000","last_fullsized":"2024-09-16T01:34:23.610276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:01:27.300067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188231+0000","last_change":"2024-09-16T01:34:22.202105+0000","last_active":"2024-09-16T01:34:23.188231+0000","last_peered":"2024-09-16T01:34:23.188231+0000","last_clean":"2024-09-16T01:34:23.188231+0000","last_became_active":"2024-09-16T01:34:22.201840+0000","last_became_peered":"2024-09-16T01:34:22.201840+0000","last_unstale":"2024-09-16T01:34:23.188231+0000","last_undegraded":"2024-09-16T01:34:23.188231+0000","last_fullsized":"2024-09-16T01:34:23.188231+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:32:05.208969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610203+0000","last_change":"2024-09-16T01:34:21.190480+0000","last_active":"2024-09-16T01:34:23.610203+0000","last_peered":"2024-09-16T01:34:23.610203+0000","last_clean":"2024-09-16T01:34:23.610203+0000","last_became_active":"2024-09-16T01:34:21.190205+0000","last_became_peered":"2024-09-16T01:34:21.190205+0000","last_unstale":"2024-09-16T01:34:23.610203+0000","last_undegraded":"2024-09-16T01:34:23.610203+0000","last_fullsized":"2024-09-16T01:34:23.610203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:47:09.976612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609791+0000","last_change":"2024-09-16T01:34:22.198514+0000","last_active":"2024-09-16T01:34:23.609791+0000","last_peered":"2024-09-16T01:34:23.609791+0000","last_clean":"2024-09-16T01:34:23.609791+0000","last_became_active":"2024-09-16T01:34:22.197969+0000","last_became_peered":"2024-09-16T01:34:22.197969+0000","last_unstale":"2024-09-16T01:34:23.609791+0000","last_undegraded":"2024-09-16T01:34:23.609791+0000","last_fullsized":"2024-09-16T01:34:23.609791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:32:14.878649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609319+0000","last_change":"2024-09-16T01:34:21.190398+0000","last_active":"2024-09-16T01:34:23.609319+0000","last_peered":"2024-09-16T01:34:23.609319+0000","last_clean":"2024-09-16T01:34:23.609319+0000","last_became_active":"2024-09-16T01:34:21.190141+0000","last_became_peered":"2024-09-16T01:34:21.190141+0000","last_unstale":"2024-09-16T01:34:23.609319+0000","last_undegraded":"2024-09-16T01:34:23.609319+0000","last_fullsized":"2024-09-16T01:34:23.609319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:47:09.976612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513229+0000","last_change":"2024-09-16T01:34:22.187125+0000","last_active":"2024-09-16T01:34:23.513229+0000","last_peered":"2024-09-16T01:34:23.513229+0000","last_clean":"2024-09-16T01:34:23.513229+0000","last_became_active":"2024-09-16T01:34:22.186846+0000","last_became_peered":"2024-09-16T01:34:22.186846+0000","last_unstale":"2024-09-16T01:34:23.513229+0000","last_undegraded":"2024-09-16T01:34:23.513229+0000","last_fullsized":"2024-09-16T01:34:23.513229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:59:47.794894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610948+0000","last_change":"2024-09-16T01:34:21.190750+0000","last_active":"2024-09-16T01:34:23.610948+0000","last_peered":"2024-09-16T01:34:23.610948+0000","last_clean":"2024-09-16T01:34:23.610948+0000","last_became_active":"2024-09-16T01:34:21.189215+0000","last_became_peered":"2024-09-16T01:34:21.189215+0000","last_unstale":"2024-09-16T01:34:23.610948+0000","last_undegraded":"2024-09-16T01:34:23.610948+0000","last_fullsized":"2024-09-16T01:34:23.610948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:11:30.497159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.190850+0000","last_change":"2024-09-16T01:34:22.184566+0000","last_active":"2024-09-16T01:34:23.190850+0000","last_peered":"2024-09-16T01:34:23.190850+0000","last_clean":"2024-09-16T01:34:23.190850+0000","last_became_active":"2024-09-16T01:34:22.184223+0000","last_became_peered":"2024-09-16T01:34:22.184223+0000","last_unstale":"2024-09-16T01:34:23.190850+0000","last_undegraded":"2024-09-16T01:34:23.190850+0000","last_fullsized":"2024-09-16T01:34:23.190850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:33:24.341399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513407+0000","last_change":"2024-09-16T01:34:21.191889+0000","last_active":"2024-09-16T01:34:23.513407+0000","last_peered":"2024-09-16T01:34:23.513407+0000","last_clean":"2024-09-16T01:34:23.513407+0000","last_became_active":"2024-09-16T01:34:21.191594+0000","last_became_peered":"2024-09-16T01:34:21.191594+0000","last_unstale":"2024-09-16T01:34:23.513407+0000","last_undegraded":"2024-09-16T01:34:23.513407+0000","last_fullsized":"2024-09-16T01:34:23.513407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:57.471937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610790+0000","last_change":"2024-09-16T01:34:22.193566+0000","last_active":"2024-09-16T01:34:23.610790+0000","last_peered":"2024-09-16T01:34:23.610790+0000","last_clean":"2024-09-16T01:34:23.610790+0000","last_became_active":"2024-09-16T01:34:22.193305+0000","last_became_peered":"2024-09-16T01:34:22.193305+0000","last_unstale":"2024-09-16T01:34:23.610790+0000","last_undegraded":"2024-09-16T01:34:23.610790+0000","last_fullsized":"2024-09-16T01:34:23.610790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:18.860118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611604+0000","last_change":"2024-09-16T01:34:21.192721+0000","last_active":"2024-09-16T01:34:23.611604+0000","last_peered":"2024-09-16T01:34:23.611604+0000","last_clean":"2024-09-16T01:34:23.611604+0000","last_became_active":"2024-09-16T01:34:21.192423+0000","last_became_peered":"2024-09-16T01:34:21.192423+0000","last_unstale":"2024-09-16T01:34:23.611604+0000","last_undegraded":"2024-09-16T01:34:23.611604+0000","last_fullsized":"2024-09-16T01:34:23.611604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:13.839623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513359+0000","last_change":"2024-09-16T01:34:22.187869+0000","last_active":"2024-09-16T01:34:23.513359+0000","last_peered":"2024-09-16T01:34:23.513359+0000","last_clean":"2024-09-16T01:34:23.513359+0000","last_became_active":"2024-09-16T01:34:22.186507+0000","last_became_peered":"2024-09-16T01:34:22.186507+0000","last_unstale":"2024-09-16T01:34:23.513359+0000","last_undegraded":"2024-09-16T01:34:23.513359+0000","last_fullsized":"2024-09-16T01:34:23.513359+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:04.740799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513369+0000","last_change":"2024-09-16T01:34:21.190145+0000","last_active":"2024-09-16T01:34:23.513369+0000","last_peered":"2024-09-16T01:34:23.513369+0000","last_clean":"2024-09-16T01:34:23.513369+0000","last_became_active":"2024-09-16T01:34:21.189886+0000","last_became_peered":"2024-09-16T01:34:21.189886+0000","last_unstale":"2024-09-16T01:34:23.513369+0000","last_undegraded":"2024-09-16T01:34:23.513369+0000","last_fullsized":"2024-09-16T01:34:23.513369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:47:09.976612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188218+0000","last_change":"2024-09-16T01:34:22.194698+0000","last_active":"2024-09-16T01:34:23.188218+0000","last_peered":"2024-09-16T01:34:23.188218+0000","last_clean":"2024-09-16T01:34:23.188218+0000","last_became_active":"2024-09-16T01:34:22.194443+0000","last_became_peered":"2024-09-16T01:34:22.194443+0000","last_unstale":"2024-09-16T01:34:23.188218+0000","last_undegraded":"2024-09-16T01:34:23.188218+0000","last_fullsized":"2024-09-16T01:34:23.188218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:16:41.426352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191535+0000","last_change":"2024-09-16T01:34:21.189000+0000","last_active":"2024-09-16T01:34:23.191535+0000","last_peered":"2024-09-16T01:34:23.191535+0000","last_clean":"2024-09-16T01:34:23.191535+0000","last_became_active":"2024-09-16T01:34:21.188144+0000","last_became_peered":"2024-09-16T01:34:21.188144+0000","last_unstale":"2024-09-16T01:34:23.191535+0000","last_undegraded":"2024-09-16T01:34:23.191535+0000","last_fullsized":"2024-09-16T01:34:23.191535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:36:22.787904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609862+0000","last_change":"2024-09-16T01:34:22.188803+0000","last_active":"2024-09-16T01:34:23.609862+0000","last_peered":"2024-09-16T01:34:23.609862+0000","last_clean":"2024-09-16T01:34:23.609862+0000","last_became_active":"2024-09-16T01:34:22.188522+0000","last_became_peered":"2024-09-16T01:34:22.188522+0000","last_unstale":"2024-09-16T01:34:23.609862+0000","last_undegraded":"2024-09-16T01:34:23.609862+0000","last_fullsized":"2024-09-16T01:34:23.609862+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:11:56.336591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610066+0000","last_change":"2024-09-16T01:34:21.192126+0000","last_active":"2024-09-16T01:34:23.610066+0000","last_peered":"2024-09-16T01:34:23.610066+0000","last_clean":"2024-09-16T01:34:23.610066+0000","last_became_active":"2024-09-16T01:34:21.191855+0000","last_became_peered":"2024-09-16T01:34:21.191855+0000","last_unstale":"2024-09-16T01:34:23.610066+0000","last_undegraded":"2024-09-16T01:34:23.610066+0000","last_fullsized":"2024-09-16T01:34:23.610066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:57.471937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.190343+0000","last_change":"2024-09-16T01:34:22.186916+0000","last_active":"2024-09-16T01:34:23.190343+0000","last_peered":"2024-09-16T01:34:23.190343+0000","last_clean":"2024-09-16T01:34:23.190343+0000","last_became_active":"2024-09-16T01:34:22.186081+0000","last_became_peered":"2024-09-16T01:34:22.186081+0000","last_unstale":"2024-09-16T01:34:23.190343+0000","last_undegraded":"2024-09-16T01:34:23.190343+0000","last_fullsized":"2024-09-16T01:34:23.190343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:36:23.795538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513225+0000","last_change":"2024-09-16T01:34:21.187955+0000","last_active":"2024-09-16T01:34:23.513225+0000","last_peered":"2024-09-16T01:34:23.513225+0000","last_clean":"2024-09-16T01:34:23.513225+0000","last_became_active":"2024-09-16T01:34:21.187668+0000","last_became_peered":"2024-09-16T01:34:21.187668+0000","last_unstale":"2024-09-16T01:34:23.513225+0000","last_undegraded":"2024-09-16T01:34:23.513225+0000","last_fullsized":"2024-09-16T01:34:23.513225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:11:30.497159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188103+0000","last_change":"2024-09-16T01:34:22.187359+0000","last_active":"2024-09-16T01:34:23.188103+0000","last_peered":"2024-09-16T01:34:23.188103+0000","last_clean":"2024-09-16T01:34:23.188103+0000","last_became_active":"2024-09-16T01:34:22.187088+0000","last_became_peered":"2024-09-16T01:34:22.187088+0000","last_unstale":"2024-09-16T01:34:23.188103+0000","last_undegraded":"2024-09-16T01:34:23.188103+0000","last_fullsized":"2024-09-16T01:34:23.188103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:55:12.145787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513225+0000","last_change":"2024-09-16T01:34:21.184801+0000","last_active":"2024-09-16T01:34:23.513225+0000","last_peered":"2024-09-16T01:34:23.513225+0000","last_clean":"2024-09-16T01:34:23.513225+0000","last_became_active":"2024-09-16T01:34:21.184472+0000","last_became_peered":"2024-09-16T01:34:21.184472+0000","last_unstale":"2024-09-16T01:34:23.513225+0000","last_undegraded":"2024-09-16T01:34:23.513225+0000","last_fullsized":"2024-09-16T01:34:23.513225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:41:15.050218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188586+0000","last_change":"2024-09-16T01:34:22.202201+0000","last_active":"2024-09-16T01:34:23.188586+0000","last_peered":"2024-09-16T01:34:23.188586+0000","last_clean":"2024-09-16T01:34:23.188586+0000","last_became_active":"2024-09-16T01:34:22.201850+0000","last_became_peered":"2024-09-16T01:34:22.201850+0000","last_unstale":"2024-09-16T01:34:23.188586+0000","last_undegraded":"2024-09-16T01:34:23.188586+0000","last_fullsized":"2024-09-16T01:34:23.188586+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:56:15.380999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611287+0000","last_change":"2024-09-16T01:34:21.192632+0000","last_active":"2024-09-16T01:34:23.611287+0000","last_peered":"2024-09-16T01:34:23.611287+0000","last_clean":"2024-09-16T01:34:23.611287+0000","last_became_active":"2024-09-16T01:34:21.192351+0000","last_became_peered":"2024-09-16T01:34:21.192351+0000","last_unstale":"2024-09-16T01:34:23.611287+0000","last_undegraded":"2024-09-16T01:34:23.611287+0000","last_fullsized":"2024-09-16T01:34:23.611287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:15:35.850464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191599+0000","last_change":"2024-09-16T01:34:22.193185+0000","last_active":"2024-09-16T01:34:23.191599+0000","last_peered":"2024-09-16T01:34:23.191599+0000","last_clean":"2024-09-16T01:34:23.191599+0000","last_became_active":"2024-09-16T01:34:22.192869+0000","last_became_peered":"2024-09-16T01:34:22.192869+0000","last_unstale":"2024-09-16T01:34:23.191599+0000","last_undegraded":"2024-09-16T01:34:23.191599+0000","last_fullsized":"2024-09-16T01:34:23.191599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:32:14.878649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610675+0000","last_change":"2024-09-16T01:34:22.200190+0000","last_active":"2024-09-16T01:34:23.610675+0000","last_peered":"2024-09-16T01:34:23.610675+0000","last_clean":"2024-09-16T01:34:23.610675+0000","last_became_active":"2024-09-16T01:34:22.199826+0000","last_became_peered":"2024-09-16T01:34:22.199826+0000","last_unstale":"2024-09-16T01:34:23.610675+0000","last_undegraded":"2024-09-16T01:34:23.610675+0000","last_fullsized":"2024-09-16T01:34:23.610675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:03.591531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611122+0000","last_change":"2024-09-16T01:34:21.188752+0000","last_active":"2024-09-16T01:34:23.611122+0000","last_peered":"2024-09-16T01:34:23.611122+0000","last_clean":"2024-09-16T01:34:23.611122+0000","last_became_active":"2024-09-16T01:34:21.188414+0000","last_became_peered":"2024-09-16T01:34:21.188414+0000","last_unstale":"2024-09-16T01:34:23.611122+0000","last_undegraded":"2024-09-16T01:34:23.611122+0000","last_fullsized":"2024-09-16T01:34:23.611122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:41:15.050218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610726+0000","last_change":"2024-09-16T01:34:22.199939+0000","last_active":"2024-09-16T01:34:23.610726+0000","last_peered":"2024-09-16T01:34:23.610726+0000","last_clean":"2024-09-16T01:34:23.610726+0000","last_became_active":"2024-09-16T01:34:22.199580+0000","last_became_peered":"2024-09-16T01:34:22.199580+0000","last_unstale":"2024-09-16T01:34:23.610726+0000","last_undegraded":"2024-09-16T01:34:23.610726+0000","last_fullsized":"2024-09-16T01:34:23.610726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:07.244892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611491+0000","last_change":"2024-09-16T01:34:21.190410+0000","last_active":"2024-09-16T01:34:23.611491+0000","last_peered":"2024-09-16T01:34:23.611491+0000","last_clean":"2024-09-16T01:34:23.611491+0000","last_became_active":"2024-09-16T01:34:21.190075+0000","last_became_peered":"2024-09-16T01:34:21.190075+0000","last_unstale":"2024-09-16T01:34:23.611491+0000","last_undegraded":"2024-09-16T01:34:23.611491+0000","last_fullsized":"2024-09-16T01:34:23.611491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:42:43.437000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609849+0000","last_change":"2024-09-16T01:34:22.198463+0000","last_active":"2024-09-16T01:34:23.609849+0000","last_peered":"2024-09-16T01:34:23.609849+0000","last_clean":"2024-09-16T01:34:23.609849+0000","last_became_active":"2024-09-16T01:34:22.198214+0000","last_became_peered":"2024-09-16T01:34:22.198214+0000","last_unstale":"2024-09-16T01:34:23.609849+0000","last_undegraded":"2024-09-16T01:34:23.609849+0000","last_fullsized":"2024-09-16T01:34:23.609849+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:55:12.145787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188388+0000","last_change":"2024-09-16T01:34:21.191946+0000","last_active":"2024-09-16T01:34:23.188388+0000","last_peered":"2024-09-16T01:34:23.188388+0000","last_clean":"2024-09-16T01:34:23.188388+0000","last_became_active":"2024-09-16T01:34:21.191675+0000","last_became_peered":"2024-09-16T01:34:21.191675+0000","last_unstale":"2024-09-16T01:34:23.188388+0000","last_undegraded":"2024-09-16T01:34:23.188388+0000","last_fullsized":"2024-09-16T01:34:23.188388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:00.162566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188347+0000","last_change":"2024-09-16T01:34:22.193611+0000","last_active":"2024-09-16T01:34:23.188347+0000","last_peered":"2024-09-16T01:34:23.188347+0000","last_clean":"2024-09-16T01:34:23.188347+0000","last_became_active":"2024-09-16T01:34:22.193334+0000","last_became_peered":"2024-09-16T01:34:22.193334+0000","last_unstale":"2024-09-16T01:34:23.188347+0000","last_undegraded":"2024-09-16T01:34:23.188347+0000","last_fullsized":"2024-09-16T01:34:23.188347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:53:16.220046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609944+0000","last_change":"2024-09-16T01:34:21.192892+0000","last_active":"2024-09-16T01:34:23.609944+0000","last_peered":"2024-09-16T01:34:23.609944+0000","last_clean":"2024-09-16T01:34:23.609944+0000","last_became_active":"2024-09-16T01:34:21.192600+0000","last_became_peered":"2024-09-16T01:34:21.192600+0000","last_unstale":"2024-09-16T01:34:23.609944+0000","last_undegraded":"2024-09-16T01:34:23.609944+0000","last_fullsized":"2024-09-16T01:34:23.609944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:14:16.097098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509447,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":73825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":664685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":75347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":73861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":75347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":212992,"data_stored":73735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542149,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":221184,"data_stored":73895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":819200,"data_stored":664741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:34:29.774 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T01:34:29.774 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T01:34:29.774 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T01:34:29.774 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T01:34:29.774 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T01:34:29.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.907+0000 7ff2b1931640 1 Processor -- start 2024-09-16T01:34:29.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.907+0000 7ff2b1931640 1 -- start start 2024-09-16T01:34:29.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.908+0000 7ff2b1931640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff2ac0b63d0 0x7ff2ac0d39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:29.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.908+0000 7ff2b1931640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2ac0cfc20 con 0x7ff2ac0d1c80 2024-09-16T01:34:29.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.908+0000 7ff2b1931640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2ac0d2ed0 con 0x7ff2ac0cf6b0 2024-09-16T01:34:29.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.908+0000 7ff2b1931640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff2ac0d31b0 con 0x7ff2ac0b63d0 2024-09-16T01:34:29.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.908+0000 7ff2aaffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff2ac0b63d0 0x7ff2ac0d39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:29.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.908+0000 7ff2ab7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff2ac0d1c80 0x7ff2ac1629f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34280/0 (socket says 172.21.15.88:34280) 2024-09-16T01:34:29.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.908+0000 7ff2ab7fe640 1 -- 172.21.15.88:0/3543332862 learned_addr learned my addr 172.21.15.88:0/3543332862 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:29.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.908+0000 7ff2aaffd640 1 -- 172.21.15.88:0/3543332862 >> v1:172.21.15.154:6789/0 conn(0x7ff2ac0cf6b0 legacy=0x7ff2ac0b7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:29.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.908+0000 7ff2aaffd640 1 -- 172.21.15.88:0/3543332862 >> v1:172.21.15.88:6789/0 conn(0x7ff2ac0d1c80 legacy=0x7ff2ac1629f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:29.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.909+0000 7ff2aaffd640 1 -- 172.21.15.88:0/3543332862 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2ac0d3490 con 0x7ff2ac0b63d0 2024-09-16T01:34:29.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.909+0000 7ff2aaffd640 1 --2- 172.21.15.88:0/3543332862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2ac0b63d0 0x7ff2ac0d39b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff29c0097a0 tx=0x7ff29c02a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=711792eb4100b6d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:29.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.909+0000 7ff2aa7fc640 1 -- 172.21.15.88:0/3543332862 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29c037070 con 0x7ff2ac0b63d0 2024-09-16T01:34:29.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.909+0000 7ff2aa7fc640 1 -- 172.21.15.88:0/3543332862 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff29c02acd0 con 0x7ff2ac0b63d0 2024-09-16T01:34:29.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.909+0000 7ff2aa7fc640 1 -- 172.21.15.88:0/3543332862 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29c032660 con 0x7ff2ac0b63d0 2024-09-16T01:34:29.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.910+0000 7ff2b1931640 1 -- 172.21.15.88:0/3543332862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2ac0b63d0 msgr2=0x7ff2ac0d39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:29.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.910+0000 7ff2b1931640 1 --2- 172.21.15.88:0/3543332862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2ac0b63d0 0x7ff2ac0d39b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff29c0097a0 tx=0x7ff29c02a040 comp rx=0 tx=0).stop 2024-09-16T01:34:29.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.910+0000 7ff2b1931640 1 -- 172.21.15.88:0/3543332862 shutdown_connections 2024-09-16T01:34:29.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.910+0000 7ff2b1931640 1 --2- 172.21.15.88:0/3543332862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2ac0b63d0 0x7ff2ac0d39b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:29.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.910+0000 7ff2b1931640 1 -- 172.21.15.88:0/3543332862 >> 172.21.15.88:0/3543332862 conn(0x7ff2ac0d1160 msgr2=0x7ff2ac0d1570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:29.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.910+0000 7ff2b1931640 1 -- 172.21.15.88:0/3543332862 shutdown_connections 2024-09-16T01:34:29.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.910+0000 7ff2b1931640 1 -- 172.21.15.88:0/3543332862 wait complete. 2024-09-16T01:34:29.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.911+0000 7ff2b1931640 1 Processor -- start 2024-09-16T01:34:29.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.911+0000 7ff2b1931640 1 -- start start 2024-09-16T01:34:29.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.912+0000 7ff2b1931640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2ac0b63d0 0x7ff2ac0b2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:29.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.912+0000 7ff2b1931640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2ac0cf6b0 0x7ff2ac0ce650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:29.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.912+0000 7ff2b1931640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2ac0d1c80 0x7ff2ac0b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:29.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.912+0000 7ff2b1931640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff2ac0cb5c0 con 0x7ff2ac0b63d0 2024-09-16T01:34:29.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.912+0000 7ff2b1931640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff2ac0cb440 con 0x7ff2ac0cf6b0 2024-09-16T01:34:29.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.912+0000 7ff2b1931640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff2ac0d3ef0 con 0x7ff2ac0d1c80 2024-09-16T01:34:29.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.912+0000 7ff2aaffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2ac0b63d0 0x7ff2ac0b2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:29.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.912+0000 7ff2aaffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2ac0b63d0 0x7ff2ac0b2790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58378/0 (socket says 172.21.15.88:58378) 2024-09-16T01:34:29.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.912+0000 7ff2aaffd640 1 -- 172.21.15.88:0/3845964183 learned_addr learned my addr 172.21.15.88:0/3845964183 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:29.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.912+0000 7ff2ab7fe640 1 --2- 172.21.15.88:0/3845964183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2ac0d1c80 0x7ff2ac0b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:29.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.912+0000 7ff2a27fc640 1 --2- 172.21.15.88:0/3845964183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2ac0cf6b0 0x7ff2ac0ce650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:29.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.912+0000 7ff2aaffd640 1 -- 172.21.15.88:0/3845964183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2ac0d1c80 msgr2=0x7ff2ac0b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:29.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.912+0000 7ff2aaffd640 1 --2- 172.21.15.88:0/3845964183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2ac0d1c80 0x7ff2ac0b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:29.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.912+0000 7ff2aaffd640 1 -- 172.21.15.88:0/3845964183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2ac0cf6b0 msgr2=0x7ff2ac0ce650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:29.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.912+0000 7ff2aaffd640 1 --2- 172.21.15.88:0/3845964183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2ac0cf6b0 0x7ff2ac0ce650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:29.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.912+0000 7ff2aaffd640 1 -- 172.21.15.88:0/3845964183 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2ac0cbd50 con 0x7ff2ac0b63d0 2024-09-16T01:34:29.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.912+0000 7ff2aaffd640 1 --2- 172.21.15.88:0/3845964183 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2ac0b63d0 0x7ff2ac0b2790 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff29c0098c0 tx=0x7ff29c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:29.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.913+0000 7ff2a8ff9640 1 -- 172.21.15.88:0/3845964183 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29c037040 con 0x7ff2ac0b63d0 2024-09-16T01:34:29.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.913+0000 7ff2a8ff9640 1 -- 172.21.15.88:0/3845964183 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff29c004030 con 0x7ff2ac0b63d0 2024-09-16T01:34:29.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.913+0000 7ff2b1931640 1 -- 172.21.15.88:0/3845964183 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2ac0cc030 con 0x7ff2ac0b63d0 2024-09-16T01:34:29.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.913+0000 7ff2a8ff9640 1 -- 172.21.15.88:0/3845964183 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff29c004420 con 0x7ff2ac0b63d0 2024-09-16T01:34:29.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.913+0000 7ff2b1931640 1 -- 172.21.15.88:0/3845964183 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2ac0cc5f0 con 0x7ff2ac0b63d0 2024-09-16T01:34:29.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.914+0000 7ff2b1931640 1 -- 172.21.15.88:0/3845964183 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2ac0b67b0 con 0x7ff2ac0b63d0 2024-09-16T01:34:29.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.914+0000 7ff2a8ff9640 1 -- 172.21.15.88:0/3845964183 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff29c03a070 con 0x7ff2ac0b63d0 2024-09-16T01:34:29.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.915+0000 7ff2a8ff9640 1 --2- 172.21.15.88:0/3845964183 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff28c06c2b0 0x7ff28c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:29.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.915+0000 7ff2a8ff9640 1 -- 172.21.15.88:0/3845964183 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff29c0abaa0 con 0x7ff2ac0b63d0 2024-09-16T01:34:29.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.915+0000 7ff2a27fc640 1 --2- 172.21.15.88:0/3845964183 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff28c06c2b0 0x7ff28c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:29.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.915+0000 7ff2a27fc640 1 --2- 172.21.15.88:0/3845964183 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff28c06c2b0 0x7ff28c06e770 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac0ba520 tx=0x7ff294009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:29.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:29.919+0000 7ff2a8ff9640 1 -- 172.21.15.88:0/3845964183 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff29c041020 con 0x7ff2ac0b63d0 2024-09-16T01:34:30.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.128+0000 7ff2b1931640 1 -- 172.21.15.88:0/3845964183 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7ff2ac0b67b0 con 0x7ff2ac0b63d0 2024-09-16T01:34:30.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.131+0000 7ff2a8ff9640 1 -- 172.21.15.88:0/3845964183 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+173504 (secure 0 0 0) 0x7ff29c07a2a0 con 0x7ff2ac0b63d0 2024-09-16T01:34:30.134 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:30.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.133+0000 7ff2b1931640 1 -- 172.21.15.88:0/3845964183 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff28c06c2b0 msgr2=0x7ff28c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.133+0000 7ff2b1931640 1 --2- 172.21.15.88:0/3845964183 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff28c06c2b0 0x7ff28c06e770 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac0ba520 tx=0x7ff294009040 comp rx=0 tx=0).stop 2024-09-16T01:34:30.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.133+0000 7ff2b1931640 1 -- 172.21.15.88:0/3845964183 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2ac0b63d0 msgr2=0x7ff2ac0b2790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.133+0000 7ff2b1931640 1 --2- 172.21.15.88:0/3845964183 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2ac0b63d0 0x7ff2ac0b2790 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff29c0098c0 tx=0x7ff29c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.134+0000 7ff2b1931640 1 -- 172.21.15.88:0/3845964183 shutdown_connections 2024-09-16T01:34:30.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.134+0000 7ff2b1931640 1 --2- 172.21.15.88:0/3845964183 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff28c06c2b0 0x7ff28c06e770 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.134+0000 7ff2b1931640 1 --2- 172.21.15.88:0/3845964183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2ac0d1c80 0x7ff2ac0b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.134+0000 7ff2b1931640 1 --2- 172.21.15.88:0/3845964183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2ac0cf6b0 0x7ff2ac0ce650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.134+0000 7ff2b1931640 1 --2- 172.21.15.88:0/3845964183 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2ac0b63d0 0x7ff2ac0b2790 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.134+0000 7ff2b1931640 1 -- 172.21.15.88:0/3845964183 >> 172.21.15.88:0/3845964183 conn(0x7ff2ac0d1160 msgr2=0x7ff2ac160cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:30.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.134+0000 7ff2b1931640 1 -- 172.21.15.88:0/3845964183 shutdown_connections 2024-09-16T01:34:30.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.134+0000 7ff2b1931640 1 -- 172.21.15.88:0/3845964183 wait complete. 2024-09-16T01:34:30.146 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":5,"flags":0,"active_gid":4107,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3938986063},{"type":"v1","addr":"172.21.15.88:6833","nonce":3938986063}]},"active_addr":"172.21.15.88:6833/3938986063","active_change":"2024-09-16T01:34:02.507971+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4106,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":3290217068}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":3084497948}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":2989361737}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":354969964}]}]} 2024-09-16T01:34:30.150 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T01:34:30.150 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T01:34:30.150 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:34:30.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.249+0000 7fde72af8640 1 Processor -- start 2024-09-16T01:34:30.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.249+0000 7fde72af8640 1 -- start start 2024-09-16T01:34:30.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.250+0000 7fde72af8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde6c0b29a0 0x7fde6c15ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.250+0000 7fde72af8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde6c0b6830 con 0x7fde6c15a9c0 2024-09-16T01:34:30.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.250+0000 7fde72af8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde6c0cd0b0 con 0x7fde6c0b5be0 2024-09-16T01:34:30.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.250+0000 7fde72af8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fde6c15f410 con 0x7fde6c0b29a0 2024-09-16T01:34:30.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.250+0000 7fde70894640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fde6c15a9c0 0x7fde6c15e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34284/0 (socket says 172.21.15.88:34284) 2024-09-16T01:34:30.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.250+0000 7fde70894640 1 -- 172.21.15.88:0/763832892 learned_addr learned my addr 172.21.15.88:0/763832892 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:30.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.250+0000 7fde71095640 1 --2- 172.21.15.88:0/763832892 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde6c0b29a0 0x7fde6c15ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.250+0000 7fde6b7fe640 1 -- 172.21.15.88:0/763832892 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3579635339 0 0) 0x7fde6c0b6830 con 0x7fde6c15a9c0 2024-09-16T01:34:30.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.250+0000 7fde6b7fe640 1 -- 172.21.15.88:0/763832892 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde54003520 con 0x7fde6c15a9c0 2024-09-16T01:34:30.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.251+0000 7fde6b7fe640 1 -- 172.21.15.88:0/763832892 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3506348774 0 0) 0x7fde54003520 con 0x7fde6c15a9c0 2024-09-16T01:34:30.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.251+0000 7fde6b7fe640 1 -- 172.21.15.88:0/763832892 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde6c0b29a0 msgr2=0x7fde6c15ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:30.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.251+0000 7fde6b7fe640 1 --2- 172.21.15.88:0/763832892 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde6c0b29a0 0x7fde6c15ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.251+0000 7fde6b7fe640 1 -- 172.21.15.88:0/763832892 >> v1:172.21.15.154:6789/0 conn(0x7fde6c0b5be0 legacy=0x7fde6c0cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.251+0000 7fde6b7fe640 1 -- 172.21.15.88:0/763832892 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde6c0cd390 con 0x7fde6c15a9c0 2024-09-16T01:34:30.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.251+0000 7fde6b7fe640 1 -- 172.21.15.88:0/763832892 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fde5c001670 con 0x7fde6c15a9c0 2024-09-16T01:34:30.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.251+0000 7fde6b7fe640 1 -- 172.21.15.88:0/763832892 >> v1:172.21.15.88:6789/0 conn(0x7fde6c15a9c0 legacy=0x7fde6c15e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.251+0000 7fde6b7fe640 1 --2- 172.21.15.88:0/763832892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde540039e0 0x7fde54003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.251+0000 7fde6b7fe640 1 --2- 172.21.15.88:0/763832892 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde6c0b29a0 0x7fde54004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.251+0000 7fde6b7fe640 1 --2- 172.21.15.88:0/763832892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde54004d40 0x7fde540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.251+0000 7fde6b7fe640 1 -- 172.21.15.88:0/763832892 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fde6c15f410 con 0x7fde54004d40 2024-09-16T01:34:30.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.251+0000 7fde6b7fe640 1 -- 172.21.15.88:0/763832892 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fde5400b9c0 con 0x7fde540039e0 2024-09-16T01:34:30.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.251+0000 7fde6b7fe640 1 -- 172.21.15.88:0/763832892 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fde5400bbc0 con 0x7fde6c0b29a0 2024-09-16T01:34:30.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.251+0000 7fde71095640 1 --2- 172.21.15.88:0/763832892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde54004d40 0x7fde540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.251+0000 7fde70894640 1 --2- 172.21.15.88:0/763832892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde540039e0 0x7fde54003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.251+0000 7fde6bfff640 1 --2- 172.21.15.88:0/763832892 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde6c0b29a0 0x7fde54004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.251+0000 7fde71095640 1 -- 172.21.15.88:0/763832892 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde6c0b29a0 msgr2=0x7fde54004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:30.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.251+0000 7fde71095640 1 --2- 172.21.15.88:0/763832892 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde6c0b29a0 0x7fde54004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.251+0000 7fde71095640 1 -- 172.21.15.88:0/763832892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde540039e0 msgr2=0x7fde54003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:30.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.251+0000 7fde71095640 1 --2- 172.21.15.88:0/763832892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde540039e0 0x7fde54003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.251+0000 7fde71095640 1 -- 172.21.15.88:0/763832892 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde54003520 con 0x7fde54004d40 2024-09-16T01:34:30.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.252+0000 7fde71095640 1 --2- 172.21.15.88:0/763832892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde54004d40 0x7fde540051c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fde6400aac0 tx=0x7fde54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b35a82ab2af6ec server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.252+0000 7fde6b7fe640 1 -- 172.21.15.88:0/763832892 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde64039070 con 0x7fde54004d40 2024-09-16T01:34:30.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.252+0000 7fde6b7fe640 1 -- 172.21.15.88:0/763832892 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde6402c720 con 0x7fde54004d40 2024-09-16T01:34:30.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.252+0000 7fde6b7fe640 1 -- 172.21.15.88:0/763832892 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde6400b040 con 0x7fde54004d40 2024-09-16T01:34:30.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.253+0000 7fde72af8640 1 -- 172.21.15.88:0/763832892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde54004d40 msgr2=0x7fde540051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.253+0000 7fde72af8640 1 --2- 172.21.15.88:0/763832892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde54004d40 0x7fde540051c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fde6400aac0 tx=0x7fde54005830 comp rx=0 tx=0).stop 2024-09-16T01:34:30.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.253+0000 7fde71095640 1 -- 172.21.15.88:0/763832892 reap_dead start 2024-09-16T01:34:30.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.253+0000 7fde72af8640 1 -- 172.21.15.88:0/763832892 shutdown_connections 2024-09-16T01:34:30.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.253+0000 7fde72af8640 1 -- 172.21.15.88:0/763832892 >> 172.21.15.88:0/763832892 conn(0x7fde6c06bc10 msgr2=0x7fde6c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:30.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.253+0000 7fde72af8640 1 -- 172.21.15.88:0/763832892 shutdown_connections 2024-09-16T01:34:30.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.253+0000 7fde72af8640 1 -- 172.21.15.88:0/763832892 wait complete. 2024-09-16T01:34:30.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.254+0000 7fde72af8640 1 Processor -- start 2024-09-16T01:34:30.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.254+0000 7fde72af8640 1 -- start start 2024-09-16T01:34:30.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.255+0000 7fde72af8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde54004d40 0x7fde6c0b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.255+0000 7fde72af8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde6c154f60 0x7fde6c0b1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.255+0000 7fde72af8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde6c0b20f0 0x7fde6c1f7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.255+0000 7fde72af8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fde6c15ada0 con 0x7fde54004d40 2024-09-16T01:34:30.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.255+0000 7fde72af8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fde6c0cd950 con 0x7fde6c0b20f0 2024-09-16T01:34:30.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.255+0000 7fde72af8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fde6c15ed10 con 0x7fde6c154f60 2024-09-16T01:34:30.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.255+0000 7fde70894640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde54004d40 0x7fde6c0b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.255+0000 7fde70894640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde54004d40 0x7fde6c0b02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58396/0 (socket says 172.21.15.88:58396) 2024-09-16T01:34:30.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.255+0000 7fde70894640 1 -- 172.21.15.88:0/2043574165 learned_addr learned my addr 172.21.15.88:0/2043574165 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:30.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.255+0000 7fde6bfff640 1 --2- 172.21.15.88:0/2043574165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde6c154f60 0x7fde6c0b1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.255+0000 7fde71095640 1 --2- 172.21.15.88:0/2043574165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde6c0b20f0 0x7fde6c1f7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.255+0000 7fde70894640 1 -- 172.21.15.88:0/2043574165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde6c154f60 msgr2=0x7fde6c0b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.255+0000 7fde70894640 1 --2- 172.21.15.88:0/2043574165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde6c154f60 0x7fde6c0b1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.255+0000 7fde70894640 1 -- 172.21.15.88:0/2043574165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde6c0b20f0 msgr2=0x7fde6c1f7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.255+0000 7fde70894640 1 --2- 172.21.15.88:0/2043574165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde6c0b20f0 0x7fde6c1f7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.255+0000 7fde70894640 1 -- 172.21.15.88:0/2043574165 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde6c0b0820 con 0x7fde54004d40 2024-09-16T01:34:30.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.255+0000 7fde70894640 1 --2- 172.21.15.88:0/2043574165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde54004d40 0x7fde6c0b02e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fde5c007310 tx=0x7fde5c0077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.255+0000 7fde697fa640 1 -- 172.21.15.88:0/2043574165 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c00fa00 con 0x7fde54004d40 2024-09-16T01:34:30.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.255+0000 7fde697fa640 1 -- 172.21.15.88:0/2043574165 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde5c018070 con 0x7fde54004d40 2024-09-16T01:34:30.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.255+0000 7fde72af8640 1 -- 172.21.15.88:0/2043574165 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde6c0aef60 con 0x7fde54004d40 2024-09-16T01:34:30.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.255+0000 7fde697fa640 1 -- 172.21.15.88:0/2043574165 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c00fe20 con 0x7fde54004d40 2024-09-16T01:34:30.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.255+0000 7fde72af8640 1 -- 172.21.15.88:0/2043574165 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde6c0af520 con 0x7fde54004d40 2024-09-16T01:34:30.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.256+0000 7fde72af8640 1 -- 172.21.15.88:0/2043574165 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde6c15f0a0 con 0x7fde54004d40 2024-09-16T01:34:30.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.257+0000 7fde697fa640 1 -- 172.21.15.88:0/2043574165 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fde5c01d460 con 0x7fde54004d40 2024-09-16T01:34:30.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.257+0000 7fde697fa640 1 --2- 172.21.15.88:0/2043574165 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fde5806c2b0 0x7fde5806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.257+0000 7fde697fa640 1 -- 172.21.15.88:0/2043574165 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fde5c08b3c0 con 0x7fde54004d40 2024-09-16T01:34:30.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.257+0000 7fde6bfff640 1 --2- 172.21.15.88:0/2043574165 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fde5806c2b0 0x7fde5806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.258+0000 7fde6bfff640 1 --2- 172.21.15.88:0/2043574165 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fde5806c2b0 0x7fde5806e770 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fde6c0aee90 tx=0x7fde60009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.262+0000 7fde697fa640 1 -- 172.21.15.88:0/2043574165 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fde5c059570 con 0x7fde54004d40 2024-09-16T01:34:30.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.430+0000 7fde72af8640 1 -- 172.21.15.88:0/2043574165 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fde6c06c3f0 con 0x7fde54004d40 2024-09-16T01:34:30.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.431+0000 7fde697fa640 1 -- 172.21.15.88:0/2043574165 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16641 (secure 0 0 0) 0x7fde5c05d870 con 0x7fde54004d40 2024-09-16T01:34:30.433 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:30.433 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:34:23.181812+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:34:19.324549+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:34:20.494051+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:34:30.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.433+0000 7fde72af8640 1 -- 172.21.15.88:0/2043574165 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fde5806c2b0 msgr2=0x7fde5806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.433+0000 7fde72af8640 1 --2- 172.21.15.88:0/2043574165 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fde5806c2b0 0x7fde5806e770 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fde6c0aee90 tx=0x7fde60009040 comp rx=0 tx=0).stop 2024-09-16T01:34:30.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.433+0000 7fde72af8640 1 -- 172.21.15.88:0/2043574165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde54004d40 msgr2=0x7fde6c0b02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.433+0000 7fde72af8640 1 --2- 172.21.15.88:0/2043574165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde54004d40 0x7fde6c0b02e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fde5c007310 tx=0x7fde5c0077d0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.433+0000 7fde72af8640 1 -- 172.21.15.88:0/2043574165 shutdown_connections 2024-09-16T01:34:30.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.433+0000 7fde72af8640 1 --2- 172.21.15.88:0/2043574165 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fde5806c2b0 0x7fde5806e770 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.433+0000 7fde72af8640 1 --2- 172.21.15.88:0/2043574165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde6c0b20f0 0x7fde6c1f7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.433+0000 7fde72af8640 1 --2- 172.21.15.88:0/2043574165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde6c154f60 0x7fde6c0b1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.433+0000 7fde72af8640 1 --2- 172.21.15.88:0/2043574165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde54004d40 0x7fde6c0b02e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.433+0000 7fde72af8640 1 -- 172.21.15.88:0/2043574165 >> 172.21.15.88:0/2043574165 conn(0x7fde6c06bc10 msgr2=0x7fde6c1560e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:30.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.434+0000 7fde72af8640 1 -- 172.21.15.88:0/2043574165 shutdown_connections 2024-09-16T01:34:30.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.434+0000 7fde72af8640 1 -- 172.21.15.88:0/2043574165 wait complete. 2024-09-16T01:34:30.446 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T01:34:30.446 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:34:30.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.542+0000 7f5b44903640 1 Processor -- start 2024-09-16T01:34:30.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.542+0000 7f5b44903640 1 -- start start 2024-09-16T01:34:30.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.544+0000 7f5b44903640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5b4006e200 0x7f5b4006e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.544+0000 7f5b44903640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5b400d00e0 0x7f5b400d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.544+0000 7f5b44903640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b4006eb60 con 0x7f5b4015c4c0 2024-09-16T01:34:30.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.544+0000 7f5b44903640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f5b400cfc50 con 0x7f5b4006e200 2024-09-16T01:34:30.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.544+0000 7f5b44903640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5b4006cd30 con 0x7f5b400d00e0 2024-09-16T01:34:30.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.544+0000 7f5b3dd9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5b4006e200 0x7f5b4006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.544+0000 7f5b3e59c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5b4015c4c0 0x7f5b4006c620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45144/0 (socket says 172.21.15.88:45144) 2024-09-16T01:34:30.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.544+0000 7f5b3dd9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5b4006e200 0x7f5b4006e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54798/0 (socket says 172.21.15.88:54798) 2024-09-16T01:34:30.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.544+0000 7f5b3e59c640 1 -- 172.21.15.88:0/2195968234 learned_addr learned my addr 172.21.15.88:0/2195968234 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:30.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.544+0000 7f5b3ed9d640 1 --2- 172.21.15.88:0/2195968234 >> v2:172.21.15.154:3300/0 conn(0x7f5b400d00e0 0x7f5b400d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.544+0000 7f5b3dd9b640 1 -- 172.21.15.88:0/2195968234 >> v2:172.21.15.154:3300/0 conn(0x7f5b400d00e0 msgr2=0x7f5b400d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:30.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.544+0000 7f5b3dd9b640 1 --2- 172.21.15.88:0/2195968234 >> v2:172.21.15.154:3300/0 conn(0x7f5b400d00e0 0x7f5b400d04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.544+0000 7f5b3dd9b640 1 -- 172.21.15.88:0/2195968234 >> v1:172.21.15.88:6789/0 conn(0x7f5b4015c4c0 legacy=0x7f5b4006c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.544+0000 7f5b3dd9b640 1 -- 172.21.15.88:0/2195968234 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b4015c8a0 con 0x7f5b4006e200 2024-09-16T01:34:30.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.544+0000 7f5b3d59a640 1 -- 172.21.15.88:0/2195968234 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 675092170 0 0) 0x7f5b4006eb60 con 0x7f5b4015c4c0 2024-09-16T01:34:30.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.545+0000 7f5b3dd9b640 1 --2- 172.21.15.88:0/2195968234 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b4006e200 0x7f5b4006e620 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5b400b0c50 tx=0x7f5b3402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=424bd6147879aa07 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.545+0000 7f5b3d59a640 1 -- 172.21.15.88:0/2195968234 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b34037070 con 0x7f5b4006e200 2024-09-16T01:34:30.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.545+0000 7f5b3d59a640 1 -- 172.21.15.88:0/2195968234 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b34032440 con 0x7f5b4006e200 2024-09-16T01:34:30.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.545+0000 7f5b3d59a640 1 -- 172.21.15.88:0/2195968234 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b34032810 con 0x7f5b4006e200 2024-09-16T01:34:30.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.545+0000 7f5b44903640 1 -- 172.21.15.88:0/2195968234 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b4006e200 msgr2=0x7f5b4006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.545+0000 7f5b44903640 1 --2- 172.21.15.88:0/2195968234 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b4006e200 0x7f5b4006e620 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5b400b0c50 tx=0x7f5b3402a1a0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.545+0000 7f5b44903640 1 -- 172.21.15.88:0/2195968234 shutdown_connections 2024-09-16T01:34:30.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.545+0000 7f5b44903640 1 --2- 172.21.15.88:0/2195968234 >> v2:172.21.15.154:3300/0 conn(0x7f5b400d00e0 0x7f5b400d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.545+0000 7f5b44903640 1 --2- 172.21.15.88:0/2195968234 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b4006e200 0x7f5b4006e620 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.545+0000 7f5b44903640 1 -- 172.21.15.88:0/2195968234 >> 172.21.15.88:0/2195968234 conn(0x7f5b400b6ec0 msgr2=0x7f5b400b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:30.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.546+0000 7f5b44903640 1 -- 172.21.15.88:0/2195968234 shutdown_connections 2024-09-16T01:34:30.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.546+0000 7f5b44903640 1 -- 172.21.15.88:0/2195968234 wait complete. 2024-09-16T01:34:30.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.547+0000 7f5b44903640 1 Processor -- start 2024-09-16T01:34:30.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.547+0000 7f5b44903640 1 -- start start 2024-09-16T01:34:30.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.548+0000 7f5b44903640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b4006e200 0x7f5b40151000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.548+0000 7f5b44903640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b400d00e0 0x7f5b40153370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.548+0000 7f5b44903640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b4015c4c0 0x7f5b401538b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.548+0000 7f5b44903640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5b400cf8e0 con 0x7f5b4015c4c0 2024-09-16T01:34:30.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.548+0000 7f5b44903640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5b400cf600 con 0x7f5b400d00e0 2024-09-16T01:34:30.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.548+0000 7f5b44903640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5b400cf240 con 0x7f5b4006e200 2024-09-16T01:34:30.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.548+0000 7f5b3ed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b4015c4c0 0x7f5b401538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.548+0000 7f5b3ed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b4015c4c0 0x7f5b401538b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54802/0 (socket says 172.21.15.88:54802) 2024-09-16T01:34:30.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.548+0000 7f5b3ed9d640 1 -- 172.21.15.88:0/1486898060 learned_addr learned my addr 172.21.15.88:0/1486898060 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:30.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.548+0000 7f5b3e59c640 1 --2- 172.21.15.88:0/1486898060 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b4006e200 0x7f5b40151000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.548+0000 7f5b3dd9b640 1 --2- 172.21.15.88:0/1486898060 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b400d00e0 0x7f5b40153370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.548+0000 7f5b3ed9d640 1 -- 172.21.15.88:0/1486898060 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b4006e200 msgr2=0x7f5b40151000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.548+0000 7f5b3ed9d640 1 --2- 172.21.15.88:0/1486898060 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b4006e200 0x7f5b40151000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.548+0000 7f5b3ed9d640 1 -- 172.21.15.88:0/1486898060 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b400d00e0 msgr2=0x7f5b40153370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.548+0000 7f5b3ed9d640 1 --2- 172.21.15.88:0/1486898060 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b400d00e0 0x7f5b40153370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.548+0000 7f5b3ed9d640 1 -- 172.21.15.88:0/1486898060 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b40153ef0 con 0x7f5b4015c4c0 2024-09-16T01:34:30.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.548+0000 7f5b3e59c640 1 --2- 172.21.15.88:0/1486898060 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b4006e200 0x7f5b40151000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:30.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.548+0000 7f5b3dd9b640 1 --2- 172.21.15.88:0/1486898060 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b400d00e0 0x7f5b40153370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:30.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.548+0000 7f5b3ed9d640 1 --2- 172.21.15.88:0/1486898060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b4015c4c0 0x7f5b401538b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800cd70 tx=0x7f5b3800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.548+0000 7f5b2effd640 1 -- 172.21.15.88:0/1486898060 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b38003b90 con 0x7f5b4015c4c0 2024-09-16T01:34:30.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.548+0000 7f5b2effd640 1 -- 172.21.15.88:0/1486898060 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b38003d30 con 0x7f5b4015c4c0 2024-09-16T01:34:30.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.548+0000 7f5b2effd640 1 -- 172.21.15.88:0/1486898060 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b38004240 con 0x7f5b4015c4c0 2024-09-16T01:34:30.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.548+0000 7f5b44903640 1 -- 172.21.15.88:0/1486898060 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b401541d0 con 0x7f5b4015c4c0 2024-09-16T01:34:30.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.548+0000 7f5b44903640 1 -- 172.21.15.88:0/1486898060 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b40154790 con 0x7f5b4015c4c0 2024-09-16T01:34:30.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.549+0000 7f5b44903640 1 -- 172.21.15.88:0/1486898060 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b4006da10 con 0x7f5b4015c4c0 2024-09-16T01:34:30.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.550+0000 7f5b2effd640 1 -- 172.21.15.88:0/1486898060 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5b380043e0 con 0x7f5b4015c4c0 2024-09-16T01:34:30.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.550+0000 7f5b2effd640 1 --2- 172.21.15.88:0/1486898060 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5b2006c2b0 0x7f5b2006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.550+0000 7f5b2effd640 1 -- 172.21.15.88:0/1486898060 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5b3808bf80 con 0x7f5b4015c4c0 2024-09-16T01:34:30.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.550+0000 7f5b3e59c640 1 --2- 172.21.15.88:0/1486898060 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5b2006c2b0 0x7f5b2006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.551+0000 7f5b3e59c640 1 --2- 172.21.15.88:0/1486898060 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5b2006c2b0 0x7f5b2006e770 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5b30000f60 tx=0x7f5b30009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.554+0000 7f5b2effd640 1 -- 172.21.15.88:0/1486898060 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5b3801d340 con 0x7f5b4015c4c0 2024-09-16T01:34:30.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.721+0000 7f5b44903640 1 -- 172.21.15.88:0/1486898060 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5b40157390 con 0x7f5b4015c4c0 2024-09-16T01:34:30.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.722+0000 7f5b2effd640 1 -- 172.21.15.88:0/1486898060 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16641 (secure 0 0 0) 0x7f5b3805a6f0 con 0x7f5b4015c4c0 2024-09-16T01:34:30.724 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:30.724 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:34:23.181812+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:34:19.324549+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:34:20.494051+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:34:30.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.724+0000 7f5b44903640 1 -- 172.21.15.88:0/1486898060 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5b2006c2b0 msgr2=0x7f5b2006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.724+0000 7f5b44903640 1 --2- 172.21.15.88:0/1486898060 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5b2006c2b0 0x7f5b2006e770 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5b30000f60 tx=0x7f5b30009040 comp rx=0 tx=0).stop 2024-09-16T01:34:30.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.724+0000 7f5b44903640 1 -- 172.21.15.88:0/1486898060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b4015c4c0 msgr2=0x7f5b401538b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.724+0000 7f5b44903640 1 --2- 172.21.15.88:0/1486898060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b4015c4c0 0x7f5b401538b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800cd70 tx=0x7f5b3800a660 comp rx=0 tx=0).stop 2024-09-16T01:34:30.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.724+0000 7f5b44903640 1 -- 172.21.15.88:0/1486898060 shutdown_connections 2024-09-16T01:34:30.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.724+0000 7f5b44903640 1 --2- 172.21.15.88:0/1486898060 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5b2006c2b0 0x7f5b2006e770 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.724+0000 7f5b44903640 1 --2- 172.21.15.88:0/1486898060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b4015c4c0 0x7f5b401538b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.724+0000 7f5b44903640 1 --2- 172.21.15.88:0/1486898060 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b400d00e0 0x7f5b40153370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.724+0000 7f5b44903640 1 --2- 172.21.15.88:0/1486898060 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b4006e200 0x7f5b40151000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.724+0000 7f5b44903640 1 -- 172.21.15.88:0/1486898060 >> 172.21.15.88:0/1486898060 conn(0x7f5b400b6ec0 msgr2=0x7f5b400b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:30.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.724+0000 7f5b44903640 1 -- 172.21.15.88:0/1486898060 shutdown_connections 2024-09-16T01:34:30.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.724+0000 7f5b44903640 1 -- 172.21.15.88:0/1486898060 wait complete. 2024-09-16T01:34:30.737 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T01:34:30.737 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T01:34:30.737 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T01:34:30.738 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T01:34:30.738 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T01:34:30.738 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T01:34:30.739 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-16T01:34:30.739 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-16T01:34:30.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.905+0000 7fd485caf640 1 Processor -- start 2024-09-16T01:34:30.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.905+0000 7fd485caf640 1 -- start start 2024-09-16T01:34:30.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.906+0000 7fd485caf640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd4801624d0 0x7fd4800b3d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.905+0000 7fd49bae6640 1 Processor -- start 2024-09-16T01:34:30.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.905+0000 7fe7d2c0e640 1 Processor -- start 2024-09-16T01:34:30.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.905+0000 7fe7d2c0e640 1 -- start start 2024-09-16T01:34:30.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.906+0000 7f9a26bdf640 1 Processor -- start 2024-09-16T01:34:30.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.906+0000 7f40cb59e640 1 Processor -- start 2024-09-16T01:34:30.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.906+0000 7f40cb59e640 1 -- start start 2024-09-16T01:34:30.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.906+0000 7fd485caf640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4800ce700 0x7fd4800ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.906+0000 7fd485caf640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4800b42b0 con 0x7fd4801628b0 2024-09-16T01:34:30.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.906+0000 7fd485caf640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd4800cd860 con 0x7fd4801624d0 2024-09-16T01:34:30.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.906+0000 7fd485caf640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd4800cdb40 con 0x7fd4800ce700 2024-09-16T01:34:30.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.906+0000 7fe7d2c0e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe7cc15f770 0x7fe7cc15d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.906+0000 7fe7d2c0e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7cc15fc20 con 0x7fe7cc0b5470 2024-09-16T01:34:30.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.906+0000 7fe7d2c0e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7cc15bb60 con 0x7fe7cc155880 2024-09-16T01:34:30.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.906+0000 7fe7d2c0e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe7cc155c60 con 0x7fe7cc15f770 2024-09-16T01:34:30.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7fd4854ae640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4800ce700 0x7fd4800ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7fd4854ae640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4800ce700 0x7fd4800ccb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:35340/0 (socket says 172.21.15.88:35340) 2024-09-16T01:34:30.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7fd4854ae640 1 -- 172.21.15.88:0/3952540969 learned_addr learned my addr 172.21.15.88:0/3952540969 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:30.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7fd484cad640 1 --2- 172.21.15.88:0/3952540969 >> v2:172.21.15.154:3300/0 conn(0x7fd4801624d0 0x7fd4800b3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7fd4854ae640 1 -- 172.21.15.88:0/3952540969 >> v2:172.21.15.154:3300/0 conn(0x7fd4801624d0 msgr2=0x7fd4800b3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:30.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7fd4854ae640 1 --2- 172.21.15.88:0/3952540969 >> v2:172.21.15.154:3300/0 conn(0x7fd4801624d0 0x7fd4800b3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7fd4854ae640 1 -- 172.21.15.88:0/3952540969 >> v1:172.21.15.154:6789/0 conn(0x7fd4801628b0 legacy=0x7fd4800cdff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:30.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7fd4854ae640 1 -- 172.21.15.88:0/3952540969 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4800b3290 con 0x7fd4800ce700 2024-09-16T01:34:30.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7fd4854ae640 1 --2- 172.21.15.88:0/3952540969 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4800ce700 0x7fd4800ccb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd478009e50 tx=0x7fd47802a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b46875039a20edf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7fa885f9b640 1 Processor -- start 2024-09-16T01:34:30.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7fa885f9b640 1 -- start start 2024-09-16T01:34:30.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7f74016ca640 1 Processor -- start 2024-09-16T01:34:30.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7f74016ca640 1 -- start start 2024-09-16T01:34:30.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7f74016ca640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f73f40e09c0 0x7f73f40556a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7f74016ca640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f73f4035a80 0x7f73f4038230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.905+0000 7fd49bae6640 1 -- start start 2024-09-16T01:34:30.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7fd49bae6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd494162ba0 0x7fd4940b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7fd49bae6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd4940b7530 0x7fd494073440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7fd49a083640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd4940b7530 0x7fd494073440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7fd49a083640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd4940b7530 0x7fd494073440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54824/0 (socket says 172.21.15.88:54824) 2024-09-16T01:34:30.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7fd49a083640 1 -- 172.21.15.88:0/2417140064 learned_addr learned my addr 172.21.15.88:0/2417140064 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:30.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7fd49bae6640 1 -- 172.21.15.88:0/2417140064 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd494167550 con 0x7fd49406cd20 2024-09-16T01:34:30.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7fd49bae6640 1 -- 172.21.15.88:0/2417140064 --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd494167020 con 0x7fd4940b7530 2024-09-16T01:34:30.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7fd49bae6640 1 -- 172.21.15.88:0/2417140064 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd49406e7e0 con 0x7fd494162ba0 2024-09-16T01:34:30.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7fd499882640 1 --2- 172.21.15.88:0/2417140064 >> v2:172.21.15.154:3300/0 conn(0x7fd494162ba0 0x7fd4940b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7fe7cbfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe7cc0b5470 0x7fe7cc15cc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45146/0 (socket says 172.21.15.88:45146) 2024-09-16T01:34:30.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7fe7cbfff640 1 -- 172.21.15.88:0/2719259459 learned_addr learned my addr 172.21.15.88:0/2719259459 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:30.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7fe7d11ab640 1 --2- 172.21.15.88:0/2719259459 >> v2:172.21.15.88:3300/0 conn(0x7fe7cc15f770 0x7fe7cc15d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.906+0000 7f9a26bdf640 1 -- start start 2024-09-16T01:34:30.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7f9a26bdf640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9a201492f0 0x7f9a201666a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7f9a26bdf640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a200b3f40 con 0x7f9a200b4700 2024-09-16T01:34:30.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7f9a26bdf640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a20071200 con 0x7f9a200b60e0 2024-09-16T01:34:30.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7f9a26bdf640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f9a20070fb0 con 0x7f9a201492f0 2024-09-16T01:34:30.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7f9a2517c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9a200b4700 0x7f9a200b4b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45154/0 (socket says 172.21.15.88:45154) 2024-09-16T01:34:30.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7f9a2517c640 1 -- 172.21.15.88:0/1554269364 learned_addr learned my addr 172.21.15.88:0/1554269364 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:30.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7f846abfc640 1 Processor -- start 2024-09-16T01:34:30.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7f846abfc640 1 -- start start 2024-09-16T01:34:30.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7f40cb59e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f40c40ea390 0x7f40c403b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7f40cb59e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40c4053080 0x7f40c40534f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7f40cb59e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f40c4057010 0x7f40c4057460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7f40cb59e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f40c4038020 con 0x7f40c4057010 2024-09-16T01:34:30.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7f40cb59e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f40c4053a30 con 0x7f40c40ea390 2024-09-16T01:34:30.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7f40cb59e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f40c4057c40 con 0x7f40c4053080 2024-09-16T01:34:30.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7f40ca59c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f40c40ea390 0x7f40c403b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7f40ca59c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f40c40ea390 0x7f40c403b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:57634/0 (socket says 172.21.15.88:57634) 2024-09-16T01:34:30.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7f40c9d9b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40c4053080 0x7f40c40534f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7f40ca59c640 1 -- 172.21.15.88:0/3990190498 learned_addr learned my addr 172.21.15.88:0/3990190498 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:30.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7f40cad9d640 1 --2- 172.21.15.88:0/3990190498 >> v2:172.21.15.88:3300/0 conn(0x7f40c4057010 0x7f40c4057460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7f40c9d9b640 1 -- 172.21.15.88:0/3990190498 >> v2:172.21.15.154:3300/0 conn(0x7f40c40ea390 msgr2=0x7f40c403b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:30.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7f40c9d9b640 1 --2- 172.21.15.88:0/3990190498 >> v2:172.21.15.154:3300/0 conn(0x7f40c40ea390 0x7f40c403b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7f40c9d9b640 1 -- 172.21.15.88:0/3990190498 >> v2:172.21.15.88:3300/0 conn(0x7f40c4057010 msgr2=0x7f40c4057460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:30.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7f40c9d9b640 1 --2- 172.21.15.88:0/3990190498 >> v2:172.21.15.88:3300/0 conn(0x7f40c4057010 0x7f40c4057460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7f40c9d9b640 1 -- 172.21.15.88:0/3990190498 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40c40579a0 con 0x7f40c4053080 2024-09-16T01:34:30.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7fd499882640 1 -- 172.21.15.88:0/2417140064 >> v2:172.21.15.88:3300/0 conn(0x7fd4940b7530 msgr2=0x7fd494073440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:30.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7f40cad9d640 1 --2- 172.21.15.88:0/3990190498 >> v2:172.21.15.88:3300/0 conn(0x7f40c4057010 0x7f40c4057460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:30.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7f73fbfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f73f40e09c0 0x7f73f40556a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7f73fbfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f73f40e09c0 0x7f73f40556a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54856/0 (socket says 172.21.15.88:54856) 2024-09-16T01:34:30.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7fd47f7fe640 1 -- 172.21.15.88:0/3952540969 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd478036070 con 0x7fd4800ce700 2024-09-16T01:34:30.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7fd47f7fe640 1 -- 172.21.15.88:0/3952540969 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd478031600 con 0x7fd4800ce700 2024-09-16T01:34:30.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7f73fbfff640 1 -- 172.21.15.88:0/2079411617 learned_addr learned my addr 172.21.15.88:0/2079411617 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:30.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7fd47f7fe640 1 -- 172.21.15.88:0/3952540969 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4780319d0 con 0x7fd4800ce700 2024-09-16T01:34:30.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7f74016ca640 1 --2- 172.21.15.88:0/2079411617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f40354c0 0x7f73f4038770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7f73fb7fe640 1 --2- 172.21.15.88:0/2079411617 >> v2:172.21.15.154:3300/0 conn(0x7f73f4035a80 0x7f73f4038230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7fa885f9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa8780323c0 0x7fa878052350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7fa885f9b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa87801c5c0 con 0x7fa8780dec70 2024-09-16T01:34:30.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7fa885f9b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa87801c830 con 0x7fa878009af0 2024-09-16T01:34:30.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7fa885f9b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa87801cb10 con 0x7fa8780323c0 2024-09-16T01:34:30.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7fe7d11ab640 1 -- 172.21.15.88:0/2719259459 >> v1:172.21.15.154:6789/0 conn(0x7fe7cc155880 legacy=0x7fe7cc0b4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:30.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.907+0000 7fe7cb7fe640 1 -- 172.21.15.88:0/2719259459 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1694286835 0 0) 0x7fe7cc15fc20 con 0x7fe7cc0b5470 2024-09-16T01:34:30.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7fe7d11ab640 1 -- 172.21.15.88:0/2719259459 >> v1:172.21.15.88:6789/0 conn(0x7fe7cc0b5470 legacy=0x7fe7cc15cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7fe7d11ab640 1 -- 172.21.15.88:0/2719259459 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7cc15be40 con 0x7fe7cc15f770 2024-09-16T01:34:30.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7fd498880640 1 -- 172.21.15.88:0/2417140064 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1424166296 0 0) 0x7fd494167550 con 0x7fd49406cd20 2024-09-16T01:34:30.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7fd499882640 1 --2- 172.21.15.88:0/2417140064 >> v2:172.21.15.88:3300/0 conn(0x7fd4940b7530 0x7fd494073440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.908+0000 7fd499882640 1 -- 172.21.15.88:0/2417140064 >> v1:172.21.15.154:6789/0 conn(0x7fd49406cd20 legacy=0x7fd4940b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7fd499882640 1 -- 172.21.15.88:0/2417140064 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd494167270 con 0x7fd494162ba0 2024-09-16T01:34:30.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7fd49a083640 1 --2- 172.21.15.88:0/2417140064 >> v2:172.21.15.88:3300/0 conn(0x7fd4940b7530 0x7fd494073440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:34:30.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7f74016ca640 1 -- 172.21.15.88:0/2079411617 --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f73f4039100 con 0x7f73f40e09c0 2024-09-16T01:34:30.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7f74016ca640 1 -- 172.21.15.88:0/2079411617 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f73f4057400 con 0x7f73f4035a80 2024-09-16T01:34:30.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7f9a2497b640 1 --2- 172.21.15.88:0/1554269364 >> v2:172.21.15.88:3300/0 conn(0x7f9a201492f0 0x7f9a201666a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7f9a177fe640 1 -- 172.21.15.88:0/1554269364 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 987018114 0 0) 0x7f9a200b3f40 con 0x7f9a200b4700 2024-09-16T01:34:30.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7f9a177fe640 1 -- 172.21.15.88:0/1554269364 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a08003520 con 0x7f9a200b4700 2024-09-16T01:34:30.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7f9a177fe640 1 -- 172.21.15.88:0/1554269364 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2110468666 0 0) 0x7f9a20071200 con 0x7f9a200b60e0 2024-09-16T01:34:30.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7f9a177fe640 1 -- 172.21.15.88:0/1554269364 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a08003880 con 0x7f9a200b60e0 2024-09-16T01:34:30.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7f74016ca640 1 -- 172.21.15.88:0/2079411617 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f73f4054ea0 con 0x7f73f40354c0 2024-09-16T01:34:30.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7fa88579a640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fa878009af0 0x7fa878009f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:59716/0 (socket says 172.21.15.88:59716) 2024-09-16T01:34:30.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7fa88579a640 1 -- 172.21.15.88:0/1475243854 learned_addr learned my addr 172.21.15.88:0/1475243854 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:30.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7f846abfc640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f846415e720 0x7f84640d2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7fd485caf640 1 -- 172.21.15.88:0/3952540969 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4800ce700 msgr2=0x7fd4800ccb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7fd485caf640 1 --2- 172.21.15.88:0/3952540969 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4800ce700 0x7fd4800ccb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd478009e50 tx=0x7fd47802a760 comp rx=0 tx=0).stop 2024-09-16T01:34:30.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7fa87ffff640 1 --2- 172.21.15.88:0/1475243854 >> v2:172.21.15.88:3300/0 conn(0x7fa8780323c0 0x7fa878052350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7f7400ec9640 1 --2- 172.21.15.88:0/2079411617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f40354c0 0x7f73f4038770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7f846abfc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f84640d15d0 0x7f84640b7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7f846abfc640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84640d23f0 con 0x7f84640d2f90 2024-09-16T01:34:30.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7f846abfc640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f84640d35c0 con 0x7f84640d15d0 2024-09-16T01:34:30.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7f846abfc640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f84640d38a0 con 0x7f846415e720 2024-09-16T01:34:30.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7f9a177fe640 1 -- 172.21.15.88:0/1554269364 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1078253897 0 0) 0x7f9a08003880 con 0x7f9a200b60e0 2024-09-16T01:34:30.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7f9a2497b640 1 -- 172.21.15.88:0/1554269364 >> v1:172.21.15.154:6789/0 conn(0x7f9a200b60e0 legacy=0x7f9a20166be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7f9a2497b640 1 -- 172.21.15.88:0/1554269364 >> v1:172.21.15.88:6789/0 conn(0x7f9a200b4700 legacy=0x7f9a200b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7f9a2497b640 1 -- 172.21.15.88:0/1554269364 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a20073d90 con 0x7f9a201492f0 2024-09-16T01:34:30.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7fd485caf640 1 -- 172.21.15.88:0/3952540969 shutdown_connections 2024-09-16T01:34:30.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7fd485caf640 1 --2- 172.21.15.88:0/3952540969 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4800ce700 0x7fd4800ccb70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7fd485caf640 1 --2- 172.21.15.88:0/3952540969 >> v2:172.21.15.154:3300/0 conn(0x7fd4801624d0 0x7fd4800b3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7fd485caf640 1 -- 172.21.15.88:0/3952540969 >> 172.21.15.88:0/3952540969 conn(0x7fd48006d0d0 msgr2=0x7fd48006d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:30.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7fe7d11ab640 1 --2- 172.21.15.88:0/2719259459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7cc15f770 0x7fe7cc15d330 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe7c400bfa0 tx=0x7fe7c402a6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae12f7f80665694 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7f846a3fb640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f84640d15d0 0x7f84640b7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7f846a3fb640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f84640d15d0 0x7f84640b7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54870/0 (socket says 172.21.15.88:54870) 2024-09-16T01:34:30.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7f846a3fb640 1 -- 172.21.15.88:0/1106537057 learned_addr learned my addr 172.21.15.88:0/1106537057 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:30.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7f8469bfa640 1 --2- 172.21.15.88:0/1106537057 >> v2:172.21.15.154:3300/0 conn(0x7f846415e720 0x7f84640d2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7f40c9d9b640 1 --2- 172.21.15.88:0/3990190498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40c4053080 0x7f40c40534f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f40c4011310 tx=0x7f40c002a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78a93ab1f9f5d177 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7fd485caf640 1 -- 172.21.15.88:0/3952540969 shutdown_connections 2024-09-16T01:34:30.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7fa87f7fe640 1 -- 172.21.15.88:0/1475243854 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3482454091 0 0) 0x7fa87801c830 con 0x7fa878009af0 2024-09-16T01:34:30.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.909+0000 7fa884f99640 1 --1- 172.21.15.88:0/1475243854 >> v1:172.21.15.88:6789/0 conn(0x7fa8780dec70 0x7fa878006500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45168/0 (socket says 172.21.15.88:45168) 2024-09-16T01:34:30.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7fa87f7fe640 1 -- 172.21.15.88:0/1475243854 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa870003520 con 0x7fa878009af0 2024-09-16T01:34:30.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7fd485caf640 1 -- 172.21.15.88:0/3952540969 wait complete. 2024-09-16T01:34:30.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7fe7cb7fe640 1 -- 172.21.15.88:0/2719259459 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4036070 con 0x7fe7cc15f770 2024-09-16T01:34:30.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7fe7cb7fe640 1 -- 172.21.15.88:0/2719259459 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7c4009540 con 0x7fe7cc15f770 2024-09-16T01:34:30.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7fe7cb7fe640 1 -- 172.21.15.88:0/2719259459 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4036070 con 0x7fe7cc15f770 2024-09-16T01:34:30.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7f40c959a640 1 -- 172.21.15.88:0/3990190498 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c0037070 con 0x7f40c4053080 2024-09-16T01:34:30.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7f40c959a640 1 -- 172.21.15.88:0/3990190498 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40c002ad50 con 0x7f40c4053080 2024-09-16T01:34:30.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7f40c959a640 1 -- 172.21.15.88:0/3990190498 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c0032660 con 0x7f40c4053080 2024-09-16T01:34:30.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7fa87f7fe640 1 -- 172.21.15.88:0/1475243854 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 607547296 0 0) 0x7fa870003520 con 0x7fa878009af0 2024-09-16T01:34:30.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7fa87f7fe640 1 -- 172.21.15.88:0/1475243854 >> v2:172.21.15.88:3300/0 conn(0x7fa8780323c0 msgr2=0x7fa878052350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:30.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7fa87f7fe640 1 --2- 172.21.15.88:0/1475243854 >> v2:172.21.15.88:3300/0 conn(0x7fa8780323c0 0x7fa878052350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7fa87f7fe640 1 -- 172.21.15.88:0/1475243854 >> v1:172.21.15.88:6789/0 conn(0x7fa8780dec70 legacy=0x7fa878006500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7fa87f7fe640 1 -- 172.21.15.88:0/1475243854 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa87801cdf0 con 0x7fa878009af0 2024-09-16T01:34:30.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7f9a2497b640 1 --2- 172.21.15.88:0/1554269364 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a201492f0 0x7f9a201666a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c009720 tx=0x7f9a1c02a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=563ab9c77a0c2937 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7fa87f7fe640 1 -- 172.21.15.88:0/1475243854 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fa8800504d0 con 0x7fa878009af0 2024-09-16T01:34:30.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7fa87f7fe640 1 -- 172.21.15.88:0/1475243854 >> v1:172.21.15.154:6789/0 conn(0x7fa878009af0 legacy=0x7fa878009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7fa87f7fe640 1 --2- 172.21.15.88:0/1475243854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8700039f0 0x7fa870003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7fa87f7fe640 1 --2- 172.21.15.88:0/1475243854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8700043b0 0x7fa870004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7fa87f7fe640 1 --2- 172.21.15.88:0/1475243854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8700050d0 0x7fa8700079a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7fa87f7fe640 1 -- 172.21.15.88:0/1475243854 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa87801cb10 con 0x7fa8700039f0 2024-09-16T01:34:30.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7fa87f7fe640 1 -- 172.21.15.88:0/1475243854 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa870007ff0 con 0x7fa8700050d0 2024-09-16T01:34:30.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7fa87f7fe640 1 -- 172.21.15.88:0/1475243854 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa8700081f0 con 0x7fa8700043b0 2024-09-16T01:34:30.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7f73fbfff640 1 -- 172.21.15.88:0/2079411617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f40354c0 msgr2=0x7f73f4038770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:30.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7f73fbfff640 1 --2- 172.21.15.88:0/2079411617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f40354c0 0x7f73f4038770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7f73fbfff640 1 -- 172.21.15.88:0/2079411617 >> v2:172.21.15.154:3300/0 conn(0x7f73f4035a80 msgr2=0x7f73f4038230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:30.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7f73fbfff640 1 --2- 172.21.15.88:0/2079411617 >> v2:172.21.15.154:3300/0 conn(0x7f73f4035a80 0x7f73f4038230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.910+0000 7f73fbfff640 1 -- 172.21.15.88:0/2079411617 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f40e0da0 con 0x7f73f40e09c0 2024-09-16T01:34:30.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7fd499882640 1 --2- 172.21.15.88:0/2417140064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd494162ba0 0x7fd4940b1b50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd494075ce0 tx=0x7fd49002a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=946a35f3c051d850 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7f8469bfa640 1 -- 172.21.15.88:0/1106537057 >> v2:172.21.15.88:3300/0 conn(0x7f84640d15d0 msgr2=0x7f84640b7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:30.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7f8469bfa640 1 --2- 172.21.15.88:0/1106537057 >> v2:172.21.15.88:3300/0 conn(0x7f84640d15d0 0x7f84640b7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7f8469bfa640 1 -- 172.21.15.88:0/1106537057 >> v1:172.21.15.88:6789/0 conn(0x7f84640d2f90 legacy=0x7f84640d0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:30.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7f8469bfa640 1 -- 172.21.15.88:0/1106537057 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84640d3b80 con 0x7f846415e720 2024-09-16T01:34:30.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7fa88579a640 1 --2- 172.21.15.88:0/1475243854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8700050d0 0x7fa8700079a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7fa87ffff640 1 --2- 172.21.15.88:0/1475243854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8700043b0 0x7fa870004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7f73fbfff640 1 --2- 172.21.15.88:0/2079411617 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73f40e09c0 0x7f73f40556a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f73f40531d0 tx=0x7f73f002a120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8adcf32ab44f63c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7f40cb59e640 1 -- 172.21.15.88:0/3990190498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40c4053080 msgr2=0x7f40c40534f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7f40cb59e640 1 --2- 172.21.15.88:0/3990190498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40c4053080 0x7f40c40534f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f40c4011310 tx=0x7f40c002a000 comp rx=0 tx=0).stop 2024-09-16T01:34:30.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7fd498880640 1 -- 172.21.15.88:0/2417140064 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd490037070 con 0x7fd494162ba0 2024-09-16T01:34:30.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7fa88579a640 1 -- 172.21.15.88:0/1475243854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8700043b0 msgr2=0x7fa870004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:30.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7fd498880640 1 -- 172.21.15.88:0/2417140064 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd490032440 con 0x7fd494162ba0 2024-09-16T01:34:30.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7fa88579a640 1 --2- 172.21.15.88:0/1475243854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8700043b0 0x7fa870004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7fa88579a640 1 -- 172.21.15.88:0/1475243854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8700039f0 msgr2=0x7fa870003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:30.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7fa88579a640 1 --2- 172.21.15.88:0/1475243854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8700039f0 0x7fa870003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7fa88579a640 1 -- 172.21.15.88:0/1475243854 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa87801cdf0 con 0x7fa8700050d0 2024-09-16T01:34:30.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7fe7d2c0e640 1 -- 172.21.15.88:0/2719259459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7cc15f770 msgr2=0x7fe7cc15d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7fe7d2c0e640 1 --2- 172.21.15.88:0/2719259459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7cc15f770 0x7fe7cc15d330 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe7c400bfa0 tx=0x7fe7c402a6b0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7fd498880640 1 -- 172.21.15.88:0/2417140064 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd490032810 con 0x7fd494162ba0 2024-09-16T01:34:30.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7fe7d2c0e640 1 -- 172.21.15.88:0/2719259459 shutdown_connections 2024-09-16T01:34:30.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7fe7d2c0e640 1 --2- 172.21.15.88:0/2719259459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7cc15f770 0x7fe7cc15d330 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7fe7d2c0e640 1 -- 172.21.15.88:0/2719259459 >> 172.21.15.88:0/2719259459 conn(0x7fe7cc0d1390 msgr2=0x7fe7cc0d17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:30.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.911+0000 7f73faffd640 1 -- 172.21.15.88:0/2079411617 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f0004030 con 0x7f73f40e09c0 2024-09-16T01:34:30.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7f73faffd640 1 -- 172.21.15.88:0/2079411617 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73f00041d0 con 0x7f73f40e09c0 2024-09-16T01:34:30.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7f73faffd640 1 -- 172.21.15.88:0/2079411617 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f0003610 con 0x7f73f40e09c0 2024-09-16T01:34:30.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7f9a177fe640 1 -- 172.21.15.88:0/1554269364 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c037070 con 0x7f9a201492f0 2024-09-16T01:34:30.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7f9a177fe640 1 -- 172.21.15.88:0/1554269364 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a1c02abe0 con 0x7f9a201492f0 2024-09-16T01:34:30.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7f9a177fe640 1 -- 172.21.15.88:0/1554269364 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a10001650 con 0x7f9a201492f0 2024-09-16T01:34:30.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7f8469bfa640 1 --2- 172.21.15.88:0/1106537057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f846415e720 0x7f84640d2a50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f84640d1f70 tx=0x7f845402a120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4129ffd7d410843b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7f40cb59e640 1 -- 172.21.15.88:0/3990190498 shutdown_connections 2024-09-16T01:34:30.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7f40cb59e640 1 --2- 172.21.15.88:0/3990190498 >> v2:172.21.15.88:3300/0 conn(0x7f40c4057010 0x7f40c4057460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7fd49bae6640 1 -- 172.21.15.88:0/2417140064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd494162ba0 msgr2=0x7fd4940b1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7fd49bae6640 1 --2- 172.21.15.88:0/2417140064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd494162ba0 0x7fd4940b1b50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd494075ce0 tx=0x7fd49002a1a0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7fe7d2c0e640 1 -- 172.21.15.88:0/2719259459 shutdown_connections 2024-09-16T01:34:30.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7f8468bf8640 1 -- 172.21.15.88:0/1106537057 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8454037070 con 0x7f846415e720 2024-09-16T01:34:30.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7fd485caf640 1 Processor -- start 2024-09-16T01:34:30.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7fa88579a640 1 -- 172.21.15.88:0/1475243854 reap_dead start 2024-09-16T01:34:30.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7fd49bae6640 1 -- 172.21.15.88:0/2417140064 shutdown_connections 2024-09-16T01:34:30.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7fd49bae6640 1 --2- 172.21.15.88:0/2417140064 >> v2:172.21.15.88:3300/0 conn(0x7fd4940b7530 0x7fd494073440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7fd49bae6640 1 --2- 172.21.15.88:0/2417140064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd494162ba0 0x7fd4940b1b50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7fd49bae6640 1 -- 172.21.15.88:0/2417140064 >> 172.21.15.88:0/2417140064 conn(0x7fd4940d24d0 msgr2=0x7fd4940d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:30.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7f8468bf8640 1 -- 172.21.15.88:0/1106537057 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f845402ae30 con 0x7f846415e720 2024-09-16T01:34:30.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7f8468bf8640 1 -- 172.21.15.88:0/1106537057 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8454032640 con 0x7f846415e720 2024-09-16T01:34:30.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7fd485caf640 1 -- start start 2024-09-16T01:34:30.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7f40cb59e640 1 --2- 172.21.15.88:0/3990190498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40c4053080 0x7f40c40534f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7f40cb59e640 1 --2- 172.21.15.88:0/3990190498 >> v2:172.21.15.154:3300/0 conn(0x7f40c40ea390 0x7f40c403b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7f40cb59e640 1 -- 172.21.15.88:0/3990190498 >> 172.21.15.88:0/3990190498 conn(0x7f40c403a520 msgr2=0x7f40c403a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:30.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7f40cb59e640 1 -- 172.21.15.88:0/3990190498 shutdown_connections 2024-09-16T01:34:30.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7f40cb59e640 1 -- 172.21.15.88:0/3990190498 wait complete. 2024-09-16T01:34:30.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7fa88579a640 1 --2- 172.21.15.88:0/1475243854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8700050d0 0x7fa8700079a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa880075d90 tx=0x7fa870005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77e08ae28dfbd02e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7fd49bae6640 1 -- 172.21.15.88:0/2417140064 shutdown_connections 2024-09-16T01:34:30.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.913+0000 7f9a26bdf640 1 -- 172.21.15.88:0/1554269364 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a201492f0 msgr2=0x7f9a201666a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.913+0000 7f74016ca640 1 -- 172.21.15.88:0/2079411617 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73f40e09c0 msgr2=0x7f73f40556a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.913+0000 7f74016ca640 1 --2- 172.21.15.88:0/2079411617 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73f40e09c0 0x7f73f40556a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f73f40531d0 tx=0x7f73f002a120 comp rx=0 tx=0).stop 2024-09-16T01:34:30.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.913+0000 7f9a26bdf640 1 --2- 172.21.15.88:0/1554269364 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a201492f0 0x7f9a201666a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c009720 tx=0x7f9a1c02a0d0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.913+0000 7f9a26bdf640 1 -- 172.21.15.88:0/1554269364 shutdown_connections 2024-09-16T01:34:30.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.913+0000 7f9a26bdf640 1 --2- 172.21.15.88:0/1554269364 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a201492f0 0x7f9a201666a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.913+0000 7f9a26bdf640 1 -- 172.21.15.88:0/1554269364 >> 172.21.15.88:0/1554269364 conn(0x7f9a2006d0f0 msgr2=0x7f9a2006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:30.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.913+0000 7f9a26bdf640 1 -- 172.21.15.88:0/1554269364 shutdown_connections 2024-09-16T01:34:30.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.913+0000 7f9a26bdf640 1 -- 172.21.15.88:0/1554269364 wait complete. 2024-09-16T01:34:30.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7fe7d2c0e640 1 -- 172.21.15.88:0/2719259459 wait complete. 2024-09-16T01:34:30.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.912+0000 7fd49bae6640 1 -- 172.21.15.88:0/2417140064 wait complete. 2024-09-16T01:34:30.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.913+0000 7fa87f7fe640 1 -- 172.21.15.88:0/1475243854 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8800761f0 con 0x7fa8700050d0 2024-09-16T01:34:30.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.913+0000 7fa87f7fe640 1 -- 172.21.15.88:0/1475243854 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8800a78b0 con 0x7fa8700050d0 2024-09-16T01:34:30.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.913+0000 7f846abfc640 1 -- 172.21.15.88:0/1106537057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f846415e720 msgr2=0x7f84640d2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.913+0000 7f846abfc640 1 --2- 172.21.15.88:0/1106537057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f846415e720 0x7f84640d2a50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f84640d1f70 tx=0x7f845402a120 comp rx=0 tx=0).stop 2024-09-16T01:34:30.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.913+0000 7f846abfc640 1 -- 172.21.15.88:0/1106537057 shutdown_connections 2024-09-16T01:34:30.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.913+0000 7f846abfc640 1 --2- 172.21.15.88:0/1106537057 >> v2:172.21.15.88:3300/0 conn(0x7f84640d15d0 0x7f84640b7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.913+0000 7f846abfc640 1 --2- 172.21.15.88:0/1106537057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f846415e720 0x7f84640d2a50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.913+0000 7f846abfc640 1 -- 172.21.15.88:0/1106537057 >> 172.21.15.88:0/1106537057 conn(0x7f8464062630 msgr2=0x7f8464062a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:30.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.913+0000 7f74016ca640 1 -- 172.21.15.88:0/2079411617 shutdown_connections 2024-09-16T01:34:30.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.913+0000 7f74016ca640 1 --2- 172.21.15.88:0/2079411617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f40354c0 0x7f73f4038770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.913+0000 7f74016ca640 1 --2- 172.21.15.88:0/2079411617 >> v2:172.21.15.154:3300/0 conn(0x7f73f4035a80 0x7f73f4038230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.913+0000 7f74016ca640 1 --2- 172.21.15.88:0/2079411617 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73f40e09c0 0x7f73f40556a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.913+0000 7f74016ca640 1 -- 172.21.15.88:0/2079411617 >> 172.21.15.88:0/2079411617 conn(0x7f73f40567b0 msgr2=0x7f73f4056bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:30.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.914+0000 7f74016ca640 1 -- 172.21.15.88:0/2079411617 shutdown_connections 2024-09-16T01:34:30.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.914+0000 7f846abfc640 1 -- 172.21.15.88:0/1106537057 shutdown_connections 2024-09-16T01:34:30.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.914+0000 7f74016ca640 1 -- 172.21.15.88:0/2079411617 wait complete. 2024-09-16T01:34:30.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.914+0000 7fd485caf640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4800ce700 0x7fd4800abb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.914+0000 7fd485caf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4801624d0 0x7fd480150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.914+0000 7fd485caf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4801628b0 0x7fd4800ad350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.914+0000 7fe7d2c0e640 1 Processor -- start 2024-09-16T01:34:30.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.914+0000 7fd485caf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd4800cd0b0 con 0x7fd4801624d0 2024-09-16T01:34:30.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.914+0000 7fd485caf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd4800b3850 con 0x7fd4801628b0 2024-09-16T01:34:30.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.914+0000 7fd485caf640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd4800cd230 con 0x7fd4800ce700 2024-09-16T01:34:30.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.914+0000 7fa87f7fe640 1 -- 172.21.15.88:0/1475243854 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8800761f0 con 0x7fa8700050d0 2024-09-16T01:34:30.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.914+0000 7fd49bae6640 1 Processor -- start 2024-09-16T01:34:30.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.914+0000 7fa885f9b640 1 -- 172.21.15.88:0/1475243854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8700050d0 msgr2=0x7fa8700079a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.914+0000 7fa885f9b640 1 --2- 172.21.15.88:0/1475243854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8700050d0 0x7fa8700079a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa880075d90 tx=0x7fa870005830 comp rx=0 tx=0).stop 2024-09-16T01:34:30.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.914+0000 7fd49bae6640 1 -- start start 2024-09-16T01:34:30.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.914+0000 7fa885f9b640 1 -- 172.21.15.88:0/1475243854 shutdown_connections 2024-09-16T01:34:30.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.914+0000 7f40cb59e640 1 Processor -- start 2024-09-16T01:34:30.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.914+0000 7fa885f9b640 1 --2- 172.21.15.88:0/1475243854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8700050d0 0x7fa8700079a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.914+0000 7fa885f9b640 1 -- 172.21.15.88:0/1475243854 >> 172.21.15.88:0/1475243854 conn(0x7fa878036090 msgr2=0x7fa8780364a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:30.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.914+0000 7f40cb59e640 1 -- start start 2024-09-16T01:34:30.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.914+0000 7f9a26bdf640 1 Processor -- start 2024-09-16T01:34:30.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.914+0000 7f846abfc640 1 -- 172.21.15.88:0/1106537057 wait complete. 2024-09-16T01:34:30.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.915+0000 7fd47ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4801624d0 0x7fd480150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.915+0000 7fd47ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4801624d0 0x7fd480150160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54886/0 (socket says 172.21.15.88:54886) 2024-09-16T01:34:30.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.915+0000 7fd47ffff640 1 -- 172.21.15.88:0/2846427869 learned_addr learned my addr 172.21.15.88:0/2846427869 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:30.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.915+0000 7f9a26bdf640 1 -- start start 2024-09-16T01:34:30.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.915+0000 7fd49bae6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd49406cd20 0x7fd494157150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.915+0000 7fd49bae6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4940b7530 0x7fd49406f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.915+0000 7fd47ffff640 1 -- 172.21.15.88:0/2846427869 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4800ce700 msgr2=0x7fd4800abb60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:34:30.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.915+0000 7fd47ffff640 1 --2- 172.21.15.88:0/2846427869 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4800ce700 0x7fd4800abb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.915+0000 7fd47ffff640 1 -- 172.21.15.88:0/2846427869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4801628b0 msgr2=0x7fd4800ad350 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:34:30.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.915+0000 7fd47ffff640 1 --2- 172.21.15.88:0/2846427869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4801628b0 0x7fd4800ad350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fd47ffff640 1 -- 172.21.15.88:0/2846427869 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd480150fd0 con 0x7fd4801624d0 2024-09-16T01:34:30.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.915+0000 7fd49bae6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd494162ba0 0x7fd49406fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.915+0000 7fd49bae6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd4940d1dc0 con 0x7fd4940b7530 2024-09-16T01:34:30.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.915+0000 7fd49bae6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd49406ed60 con 0x7fd494162ba0 2024-09-16T01:34:30.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.915+0000 7fd49bae6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd4940d1f40 con 0x7fd49406cd20 2024-09-16T01:34:30.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fd499081640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4940b7530 0x7fd49406f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fd499081640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4940b7530 0x7fd49406f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54890/0 (socket says 172.21.15.88:54890) 2024-09-16T01:34:30.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fd499081640 1 -- 172.21.15.88:0/629620761 learned_addr learned my addr 172.21.15.88:0/629620761 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:30.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fd499882640 1 --2- 172.21.15.88:0/629620761 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd49406cd20 0x7fd494157150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fd49a083640 1 --2- 172.21.15.88:0/629620761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd494162ba0 0x7fd49406fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7f74016ca640 1 Processor -- start 2024-09-16T01:34:30.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.914+0000 7fe7d2c0e640 1 -- start start 2024-09-16T01:34:30.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fe7d2c0e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7cc0b5470 0x7fe7cc14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7f40cb59e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40c4053080 0x7f40c404f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fe7d2c0e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7cc155880 0x7fe7cc1f3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7f74016ca640 1 -- start start 2024-09-16T01:34:30.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fd47ffff640 1 --2- 172.21.15.88:0/2846427869 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4801624d0 0x7fd480150160 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd46c00b3f0 tx=0x7fd46c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.915+0000 7f846abfc640 1 Processor -- start 2024-09-16T01:34:30.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7f846abfc640 1 -- start start 2024-09-16T01:34:30.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7f40cb59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40c4057010 0x7f40c417caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fe7d2c0e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7cc15f770 0x7fe7cc1f4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fe7d09aa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7cc0b5470 0x7fe7cc14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fe7d09aa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7cc0b5470 0x7fe7cc14fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54892/0 (socket says 172.21.15.88:54892) 2024-09-16T01:34:30.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fe7d09aa640 1 -- 172.21.15.88:0/2825029594 learned_addr learned my addr 172.21.15.88:0/2825029594 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:30.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7f9a26bdf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a200b4700 0x7f9a201f3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7f9a26bdf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a200b60e0 0x7f9a201ec5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7f9a26bdf640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9a201492f0 0x7f9a201ecb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7f9a26bdf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9a20167470 con 0x7f9a200b60e0 2024-09-16T01:34:30.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7f9a26bdf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9a201672f0 con 0x7f9a200b4700 2024-09-16T01:34:30.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7f9a26bdf640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9a200b52a0 con 0x7f9a201492f0 2024-09-16T01:34:30.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fd499081640 1 -- 172.21.15.88:0/629620761 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd49406cd20 msgr2=0x7fd494157150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fd499081640 1 --2- 172.21.15.88:0/629620761 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd49406cd20 0x7fd494157150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fe7d2c0e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe7cc15c6e0 con 0x7fe7cc0b5470 2024-09-16T01:34:30.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fe7d2c0e640 1 -- 172.21.15.88:0/2825029594 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe7cc15c400 con 0x7fe7cc15f770 2024-09-16T01:34:30.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fe7d2c0e640 1 -- 172.21.15.88:0/2825029594 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe7cc15b770 con 0x7fe7cc155880 2024-09-16T01:34:30.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fd499081640 1 -- 172.21.15.88:0/629620761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd494162ba0 msgr2=0x7fd49406fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fd499081640 1 --2- 172.21.15.88:0/629620761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd494162ba0 0x7fd49406fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fd499081640 1 -- 172.21.15.88:0/629620761 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4941582d0 con 0x7fd4940b7530 2024-09-16T01:34:30.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fd49a083640 1 --2- 172.21.15.88:0/629620761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd494162ba0 0x7fd49406fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:34:30.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fe7cbfff640 1 --2- 172.21.15.88:0/2825029594 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7cc155880 0x7fe7cc1f3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fe7d11ab640 1 --2- 172.21.15.88:0/2825029594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7cc15f770 0x7fe7cc1f4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fd499081640 1 --2- 172.21.15.88:0/629620761 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4940b7530 0x7fd49406f6c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd48400c3f0 tx=0x7fd48400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fd47d7fa640 1 -- 172.21.15.88:0/2846427869 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd46c015020 con 0x7fd4801624d0 2024-09-16T01:34:30.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7fd485caf640 1 -- 172.21.15.88:0/2846427869 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4801512b0 con 0x7fd4801624d0 2024-09-16T01:34:30.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fe7d09aa640 1 -- 172.21.15.88:0/2825029594 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7cc155880 msgr2=0x7fe7cc1f3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fe7d09aa640 1 --2- 172.21.15.88:0/2825029594 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7cc155880 0x7fe7cc1f3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7fe7d09aa640 1 -- 172.21.15.88:0/2825029594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7cc15f770 msgr2=0x7fe7cc1f4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7fe7d09aa640 1 --2- 172.21.15.88:0/2825029594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7cc15f770 0x7fe7cc1f4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7fe7d09aa640 1 -- 172.21.15.88:0/2825029594 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7cc0d0a50 con 0x7fe7cc0b5470 2024-09-16T01:34:30.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7fd485caf640 1 -- 172.21.15.88:0/2846427869 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd480151b50 con 0x7fd4801624d0 2024-09-16T01:34:30.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7fa885f9b640 1 -- 172.21.15.88:0/1475243854 shutdown_connections 2024-09-16T01:34:30.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f40cb59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40c40ea390 0x7f40c4050510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7fd47d7fa640 1 -- 172.21.15.88:0/2846427869 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd46c011830 con 0x7fd4801624d0 2024-09-16T01:34:30.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f40cb59e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f40c4006400 con 0x7f40c4057010 2024-09-16T01:34:30.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7fd48a7fc640 1 -- 172.21.15.88:0/629620761 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd484017070 con 0x7fd4940b7530 2024-09-16T01:34:30.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f846abfc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84640d15d0 0x7f8464203ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f846abfc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84640d2f90 0x7f846406f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f846abfc640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f846415e720 0x7f846406f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f40cb59e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f40c4006120 con 0x7f40c40ea390 2024-09-16T01:34:30.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7fd49bae6640 1 -- 172.21.15.88:0/629620761 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4941585b0 con 0x7fd4940b7530 2024-09-16T01:34:30.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7fe7d09aa640 1 --2- 172.21.15.88:0/2825029594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7cc0b5470 0x7fe7cc14fff0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe7c000c410 tx=0x7fe7c000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f846abfc640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f84640b7fa0 con 0x7f84640d2f90 2024-09-16T01:34:30.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f846abfc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f84640d4140 con 0x7f84640d15d0 2024-09-16T01:34:30.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f846abfc640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f84640b8120 con 0x7f846415e720 2024-09-16T01:34:30.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f40cb59e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f40c40066e0 con 0x7f40c4053080 2024-09-16T01:34:30.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f40c9d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40c4057010 0x7f40c417caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f40c9d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40c4057010 0x7f40c417caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54894/0 (socket says 172.21.15.88:54894) 2024-09-16T01:34:30.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f40c9d9b640 1 -- 172.21.15.88:0/2955451133 learned_addr learned my addr 172.21.15.88:0/2955451133 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:30.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7fd49bae6640 1 -- 172.21.15.88:0/629620761 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd494158b70 con 0x7fd4940b7530 2024-09-16T01:34:30.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.916+0000 7f40ca59c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40c4053080 0x7f40c404f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7fa885f9b640 1 -- 172.21.15.88:0/1475243854 wait complete. 2024-09-16T01:34:30.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7fd48a7fc640 1 -- 172.21.15.88:0/629620761 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4840035f0 con 0x7fd4940b7530 2024-09-16T01:34:30.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7fd48a7fc640 1 -- 172.21.15.88:0/629620761 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4840039e0 con 0x7fd4940b7530 2024-09-16T01:34:30.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f9a2497b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a200b4700 0x7f9a201f3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f9a2497b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a200b4700 0x7f9a201f3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:57708/0 (socket says 172.21.15.88:57708) 2024-09-16T01:34:30.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f9a2497b640 1 -- 172.21.15.88:0/2783725967 learned_addr learned my addr 172.21.15.88:0/2783725967 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:30.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f9a17fff640 1 --2- 172.21.15.88:0/2783725967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a200b60e0 0x7f9a201ec5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f846a3fb640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f846415e720 0x7f846406f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f846a3fb640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f846415e720 0x7f846406f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:35406/0 (socket says 172.21.15.88:35406) 2024-09-16T01:34:30.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f846a3fb640 1 -- 172.21.15.88:0/1400931706 learned_addr learned my addr 172.21.15.88:0/1400931706 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:30.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f84693f9640 1 --2- 172.21.15.88:0/1400931706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84640d2f90 0x7f846406f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f8469bfa640 1 --2- 172.21.15.88:0/1400931706 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84640d15d0 0x7f8464203ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f40cad9d640 1 --2- 172.21.15.88:0/2955451133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40c40ea390 0x7f40c4050510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f40c9d9b640 1 -- 172.21.15.88:0/2955451133 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40c4053080 msgr2=0x7f40c404f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f40c9d9b640 1 --2- 172.21.15.88:0/2955451133 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40c4053080 0x7f40c404f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f40c9d9b640 1 -- 172.21.15.88:0/2955451133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40c40ea390 msgr2=0x7f40c4050510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f40c9d9b640 1 --2- 172.21.15.88:0/2955451133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40c40ea390 0x7f40c4050510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.918+0000 7f40c9d9b640 1 -- 172.21.15.88:0/2955451133 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40c4050d40 con 0x7f40c4057010 2024-09-16T01:34:30.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.918+0000 7f846a3fb640 1 -- 172.21.15.88:0/1400931706 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84640d15d0 msgr2=0x7f8464203ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.918+0000 7f846a3fb640 1 --2- 172.21.15.88:0/1400931706 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84640d15d0 0x7f8464203ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.918+0000 7f846a3fb640 1 -- 172.21.15.88:0/1400931706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84640d2f90 msgr2=0x7f846406f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.918+0000 7f846a3fb640 1 --2- 172.21.15.88:0/1400931706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84640d2f90 0x7f846406f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f9a2517c640 1 --2- 172.21.15.88:0/2783725967 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9a201492f0 0x7f9a201ecb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.918+0000 7f846a3fb640 1 -- 172.21.15.88:0/1400931706 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84641e7fc0 con 0x7f846415e720 2024-09-16T01:34:30.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.918+0000 7f40c9d9b640 1 --2- 172.21.15.88:0/2955451133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40c4057010 0x7f40c417caf0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f40c0009890 tx=0x7f40c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.918+0000 7fd47d7fa640 1 -- 172.21.15.88:0/2846427869 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd46c015020 con 0x7fd4801624d0 2024-09-16T01:34:30.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.918+0000 7f846a3fb640 1 --2- 172.21.15.88:0/1400931706 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f846415e720 0x7f846406f6e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f846000e5f0 tx=0x7f846000eab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f74016ca640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f40354c0 0x7f73f40df5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.918+0000 7f9a2497b640 1 -- 172.21.15.88:0/2783725967 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9a201492f0 msgr2=0x7f9a201ecb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.918+0000 7f9a2497b640 1 --2- 172.21.15.88:0/2783725967 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9a201492f0 0x7f9a201ecb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.918+0000 7fe7c97fa640 1 -- 172.21.15.88:0/2825029594 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c0017070 con 0x7fe7cc0b5470 2024-09-16T01:34:30.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.918+0000 7fe7d2c0e640 1 -- 172.21.15.88:0/2825029594 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7cc1493f0 con 0x7fe7cc0b5470 2024-09-16T01:34:30.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.919+0000 7fe7d2c0e640 1 -- 172.21.15.88:0/2825029594 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7cc1499b0 con 0x7fe7cc0b5470 2024-09-16T01:34:30.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.918+0000 7f9a2497b640 1 -- 172.21.15.88:0/2783725967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a200b60e0 msgr2=0x7f9a201ec5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.918+0000 7f9a2497b640 1 --2- 172.21.15.88:0/2783725967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a200b60e0 0x7f9a201ec5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.918+0000 7f9a2497b640 1 -- 172.21.15.88:0/2783725967 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a201eff40 con 0x7f9a200b4700 2024-09-16T01:34:30.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.919+0000 7f9a2497b640 1 --2- 172.21.15.88:0/2783725967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a200b4700 0x7f9a201f3d70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c02a6c0 tx=0x7f9a1c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f74016ca640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73f4035a80 0x7f73f40dfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f74016ca640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f40e09c0 0x7f73f40d7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f74016ca640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f73f400bf20 con 0x7f73f4035a80 2024-09-16T01:34:30.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f74016ca640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f73f400bda0 con 0x7f73f40e09c0 2024-09-16T01:34:30.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.917+0000 7f74016ca640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f73f400c0a0 con 0x7f73f40354c0 2024-09-16T01:34:30.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.919+0000 7f73fb7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73f4035a80 0x7f73f40dfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.919+0000 7f73fb7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73f4035a80 0x7f73f40dfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54920/0 (socket says 172.21.15.88:54920) 2024-09-16T01:34:30.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.919+0000 7f73fb7fe640 1 -- 172.21.15.88:0/608439694 learned_addr learned my addr 172.21.15.88:0/608439694 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:30.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.919+0000 7f73fbfff640 1 --2- 172.21.15.88:0/608439694 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f40354c0 0x7f73f40df5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.919+0000 7fd485caf640 1 -- 172.21.15.88:0/2846427869 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd4800ccb70 con 0x7fd4801624d0 2024-09-16T01:34:30.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.919+0000 7f845a7fc640 1 -- 172.21.15.88:0/1400931706 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8460017070 con 0x7f846415e720 2024-09-16T01:34:30.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.919+0000 7f845a7fc640 1 -- 172.21.15.88:0/1400931706 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8460004220 con 0x7f846415e720 2024-09-16T01:34:30.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.919+0000 7f845a7fc640 1 -- 172.21.15.88:0/1400931706 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846000f040 con 0x7f846415e720 2024-09-16T01:34:30.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.919+0000 7f7400ec9640 1 --2- 172.21.15.88:0/608439694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f40e09c0 0x7f73f40d7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.919+0000 7f846abfc640 1 -- 172.21.15.88:0/1400931706 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84641e8280 con 0x7f846415e720 2024-09-16T01:34:30.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.919+0000 7f846abfc640 1 -- 172.21.15.88:0/1400931706 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f84641e8840 con 0x7f846415e720 2024-09-16T01:34:30.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.919+0000 7fa885f9b640 1 Processor -- start 2024-09-16T01:34:30.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.919+0000 7fa885f9b640 1 -- start start 2024-09-16T01:34:30.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.919+0000 7fd49bae6640 1 -- 172.21.15.88:0/629620761 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd47c001030 con 0x7fd4940b7530 2024-09-16T01:34:30.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.919+0000 7f73fb7fe640 1 -- 172.21.15.88:0/608439694 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f40354c0 msgr2=0x7f73f40df5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.919+0000 7f40baffd640 1 -- 172.21.15.88:0/2955451133 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c0041020 con 0x7f40c4057010 2024-09-16T01:34:30.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.919+0000 7fd48a7fc640 1 -- 172.21.15.88:0/629620761 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd48401a070 con 0x7fd4940b7530 2024-09-16T01:34:30.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.919+0000 7f73fb7fe640 1 --2- 172.21.15.88:0/608439694 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f40354c0 0x7f73f40df5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.919+0000 7f73fb7fe640 1 -- 172.21.15.88:0/608439694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f40e09c0 msgr2=0x7f73f40d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.919+0000 7f73fb7fe640 1 --2- 172.21.15.88:0/608439694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f40e09c0 0x7f73f40d7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.919+0000 7f73fb7fe640 1 -- 172.21.15.88:0/608439694 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f404fd20 con 0x7f73f4035a80 2024-09-16T01:34:30.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.919+0000 7f9a17fff640 1 --2- 172.21.15.88:0/2783725967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a200b60e0 0x7f9a201ec5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:30.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.920+0000 7fe7d2c0e640 1 -- 172.21.15.88:0/2825029594 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe7cc149bf0 con 0x7fe7cc0b5470 2024-09-16T01:34:30.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.920+0000 7fe7c97fa640 1 -- 172.21.15.88:0/2825029594 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7c00035f0 con 0x7fe7cc0b5470 2024-09-16T01:34:30.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.920+0000 7fe7c97fa640 1 -- 172.21.15.88:0/2825029594 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c0021060 con 0x7fe7cc0b5470 2024-09-16T01:34:30.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.920+0000 7f9a157fa640 1 -- 172.21.15.88:0/2783725967 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c037050 con 0x7f9a200b4700 2024-09-16T01:34:30.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.920+0000 7f9a157fa640 1 -- 172.21.15.88:0/2783725967 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a1c038040 con 0x7f9a200b4700 2024-09-16T01:34:30.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.920+0000 7f40cb59e640 1 -- 172.21.15.88:0/2955451133 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40c404c6c0 con 0x7f40c4057010 2024-09-16T01:34:30.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.920+0000 7f40cb59e640 1 -- 172.21.15.88:0/2955451133 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40c404cc80 con 0x7f40c4057010 2024-09-16T01:34:30.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.920+0000 7f73fb7fe640 1 --2- 172.21.15.88:0/608439694 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73f4035a80 0x7f73f40dfaf0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f73e800d800 tx=0x7f73e800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.920+0000 7f9a26bdf640 1 -- 172.21.15.88:0/2783725967 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a201f0220 con 0x7f9a200b4700 2024-09-16T01:34:30.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.920+0000 7f9a157fa640 1 -- 172.21.15.88:0/2783725967 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c004450 con 0x7f9a200b4700 2024-09-16T01:34:30.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.921+0000 7fa885f9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8700050d0 0x7fa878037de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.921+0000 7fa885f9b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa878173ff0 0x7fa8780359b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.921+0000 7fa885f9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa87804c440 0x7fa878177940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.921+0000 7fd48a7fc640 1 --2- 172.21.15.88:0/629620761 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd47806c2b0 0x7fd47806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.921+0000 7fd48a7fc640 1 -- 172.21.15.88:0/629620761 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd48408a720 con 0x7fd4940b7530 2024-09-16T01:34:30.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.921+0000 7fd48a7fc640 1 --2- 172.21.15.88:0/629620761 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x7fd4780744b0 0x7fd4780768d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.921+0000 7fd48a7fc640 1 -- 172.21.15.88:0/629620761 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd478076f80 con 0x7fd4780744b0 2024-09-16T01:34:30.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.921+0000 7fa885f9b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa87800bbc0 con 0x7fa87804c440 2024-09-16T01:34:30.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.921+0000 7fd499882640 1 --2- 172.21.15.88:0/629620761 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd47806c2b0 0x7fd47806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.922+0000 7f73f8ff9640 1 -- 172.21.15.88:0/608439694 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e800b6c0 con 0x7f73f4035a80 2024-09-16T01:34:30.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.922+0000 7f73f8ff9640 1 -- 172.21.15.88:0/608439694 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73e8019070 con 0x7f73f4035a80 2024-09-16T01:34:30.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.922+0000 7f845a7fc640 1 -- 172.21.15.88:0/1400931706 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8460003610 con 0x7f846415e720 2024-09-16T01:34:30.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.921+0000 7f9a26bdf640 1 -- 172.21.15.88:0/2783725967 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a201f07e0 con 0x7f9a200b4700 2024-09-16T01:34:30.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.921+0000 7f40baffd640 1 -- 172.21.15.88:0/2955451133 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40c002ad50 con 0x7f40c4057010 2024-09-16T01:34:30.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.921+0000 7f40baffd640 1 -- 172.21.15.88:0/2955451133 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c0037040 con 0x7f40c4057010 2024-09-16T01:34:30.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.921+0000 7fd47d7fa640 1 -- 172.21.15.88:0/2846427869 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd46c018020 con 0x7fd4801624d0 2024-09-16T01:34:30.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.922+0000 7fd47d7fa640 1 --2- 172.21.15.88:0/2846427869 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd46406c6b0 0x7fd46406eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.922+0000 7fd47d7fa640 1 -- 172.21.15.88:0/2846427869 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd46c011630 con 0x7fd4801624d0 2024-09-16T01:34:30.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.922+0000 7fd47d7fa640 1 --2- 172.21.15.88:0/2846427869 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x7fd464074850 0x7fd464076c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.922+0000 7fd47d7fa640 1 -- 172.21.15.88:0/2846427869 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd46c022e50 con 0x7fd464074850 2024-09-16T01:34:30.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.922+0000 7fd47d7fa640 1 -- 172.21.15.88:0/2846427869 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd46c08b550 con 0x7fd4801624d0 2024-09-16T01:34:30.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.922+0000 7fd484cad640 1 --2- 172.21.15.88:0/2846427869 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd46406c6b0 0x7fd46406eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.922+0000 7fd49a083640 1 --2- 172.21.15.88:0/629620761 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x7fd4780744b0 0x7fd4780768d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.922+0000 7fd48a7fc640 1 -- 172.21.15.88:0/629620761 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd484091050 con 0x7fd4940b7530 2024-09-16T01:34:30.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.922+0000 7fd499882640 1 --2- 172.21.15.88:0/629620761 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd47806c2b0 0x7fd47806e770 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd490034000 tx=0x7fd49002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.922+0000 7f846abfc640 1 -- 172.21.15.88:0/1400931706 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f84640b7a60 con 0x7f846415e720 2024-09-16T01:34:30.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.922+0000 7f40baffd640 1 -- 172.21.15.88:0/2955451133 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f40c0044020 con 0x7f40c4057010 2024-09-16T01:34:30.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.922+0000 7f40cb59e640 1 -- 172.21.15.88:0/2955451133 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f40c40ea770 con 0x7f40c4057010 2024-09-16T01:34:30.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.923+0000 7fe7c97fa640 1 -- 172.21.15.88:0/2825029594 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe7c001a070 con 0x7fe7cc0b5470 2024-09-16T01:34:30.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.923+0000 7f845a7fc640 1 --2- 172.21.15.88:0/1400931706 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f844806c380 0x7f844806e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.923+0000 7fd484cad640 1 --2- 172.21.15.88:0/2846427869 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd46406c6b0 0x7fd46406eb70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd4740099d0 tx=0x7fd474009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.923+0000 7f9a157fa640 1 -- 172.21.15.88:0/2783725967 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9a1c003610 con 0x7f9a200b4700 2024-09-16T01:34:30.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.923+0000 7fd49a083640 1 --2- 172.21.15.88:0/629620761 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x7fd4780744b0 0x7fd4780768d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.923+0000 7f845a7fc640 1 -- 172.21.15.88:0/1400931706 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f846008ad90 con 0x7f846415e720 2024-09-16T01:34:30.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.923+0000 7f845a7fc640 1 --2- 172.21.15.88:0/1400931706 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x7f8448074520 0x7f8448076940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.923+0000 7f845a7fc640 1 -- 172.21.15.88:0/1400931706 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f846000fe50 con 0x7f8448074520 2024-09-16T01:34:30.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.923+0000 7fe7c97fa640 1 --2- 172.21.15.88:0/2825029594 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe7b006c2e0 0x7fe7b006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.923+0000 7f73f8ff9640 1 -- 172.21.15.88:0/608439694 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e800ba80 con 0x7f73f4035a80 2024-09-16T01:34:30.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.923+0000 7fe7c97fa640 1 -- 172.21.15.88:0/2825029594 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe7c008a770 con 0x7fe7cc0b5470 2024-09-16T01:34:30.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.923+0000 7f74016ca640 1 -- 172.21.15.88:0/608439694 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73f404fa40 con 0x7f73f4035a80 2024-09-16T01:34:30.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.923+0000 7f9a157fa640 1 --2- 172.21.15.88:0/2783725967 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9a0c06c330 0x7f9a0c06e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.923+0000 7fe7c97fa640 1 --2- 172.21.15.88:0/2825029594 >> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] conn(0x7fe7b0074500 0x7fe7b0076920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.923+0000 7fe7c97fa640 1 -- 172.21.15.88:0/2825029594 --> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe7c0007630 con 0x7fe7b0074500 2024-09-16T01:34:30.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.923+0000 7fe7c97fa640 1 -- 172.21.15.88:0/2825029594 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fe7c0091050 con 0x7fe7cc0b5470 2024-09-16T01:34:30.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.923+0000 7f9a157fa640 1 -- 172.21.15.88:0/2783725967 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9a1c0ab760 con 0x7f9a200b4700 2024-09-16T01:34:30.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.923+0000 7f74016ca640 1 -- 172.21.15.88:0/608439694 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73f4039960 con 0x7f73f4035a80 2024-09-16T01:34:30.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.924+0000 7fd48a7fc640 1 -- 172.21.15.88:0/629620761 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd478076f80 con 0x7fd4780744b0 2024-09-16T01:34:30.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.924+0000 7f9a17fff640 1 --2- 172.21.15.88:0/2783725967 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9a0c06c330 0x7f9a0c06e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.924+0000 7f9a26bdf640 1 --2- 172.21.15.88:0/2783725967 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x7f9a201ed040 0x7f9a20162ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.924+0000 7f9a26bdf640 1 -- 172.21.15.88:0/2783725967 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9a201ed420 con 0x7f9a201ed040 2024-09-16T01:34:30.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.924+0000 7f9a2517c640 1 --2- 172.21.15.88:0/2783725967 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x7f9a201ed040 0x7f9a20162ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.924+0000 7f40baffd640 1 --2- 172.21.15.88:0/2955451133 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f40ac06c2b0 0x7f40ac06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.921+0000 7fa884f99640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8700050d0 0x7fa878037de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.921+0000 7fa884f99640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8700050d0 0x7fa878037de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:57746/0 (socket says 172.21.15.88:57746) 2024-09-16T01:34:30.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.921+0000 7fa884f99640 1 -- 172.21.15.88:0/2901457850 learned_addr learned my addr 172.21.15.88:0/2901457850 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:30.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.921+0000 7fa87ffff640 1 --2- 172.21.15.88:0/2901457850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa878173ff0 0x7fa8780359b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.922+0000 7fa88579a640 1 --2- 172.21.15.88:0/2901457850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa87804c440 0x7fa878177940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.924+0000 7f40ca59c640 1 --2- 172.21.15.88:0/2955451133 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f40ac06c2b0 0x7f40ac06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.924+0000 7f40baffd640 1 -- 172.21.15.88:0/2955451133 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f40c00ab560 con 0x7f40c4057010 2024-09-16T01:34:30.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.924+0000 7fa885f9b640 1 -- 172.21.15.88:0/2901457850 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa878032980 con 0x7fa8700050d0 2024-09-16T01:34:30.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.924+0000 7fe7d11ab640 1 --2- 172.21.15.88:0/2825029594 >> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] conn(0x7fe7b0074500 0x7fe7b0076920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.924+0000 7fa885f9b640 1 -- 172.21.15.88:0/2901457850 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa87800bd40 con 0x7fa878173ff0 2024-09-16T01:34:30.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.925+0000 7f9a17fff640 1 --2- 172.21.15.88:0/2783725967 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9a0c06c330 0x7f9a0c06e7f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9a20162380 tx=0x7f9a10009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.925+0000 7f40baffd640 1 --2- 172.21.15.88:0/2955451133 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x7f40ac0744d0 0x7f40ac0768f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.925+0000 7f40baffd640 1 -- 172.21.15.88:0/2955451133 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f40c002be50 con 0x7f40ac0744d0 2024-09-16T01:34:30.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.925+0000 7f40baffd640 1 -- 172.21.15.88:0/2955451133 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f40c00b2050 con 0x7f40c4057010 2024-09-16T01:34:30.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.925+0000 7f74016ca640 1 -- 172.21.15.88:0/608439694 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f73f400a950 con 0x7f73f4035a80 2024-09-16T01:34:30.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.924+0000 7fd4854ae640 1 --2- 172.21.15.88:0/2846427869 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x7fd464074850 0x7fd464076c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.924+0000 7fa884f99640 1 -- 172.21.15.88:0/2901457850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa878173ff0 msgr2=0x7fa8780359b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.924+0000 7fa884f99640 1 --2- 172.21.15.88:0/2901457850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa878173ff0 0x7fa8780359b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.924+0000 7fa884f99640 1 -- 172.21.15.88:0/2901457850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa87804c440 msgr2=0x7fa878177940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:30.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.924+0000 7fa884f99640 1 --2- 172.21.15.88:0/2901457850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa87804c440 0x7fa878177940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:30.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.924+0000 7fa884f99640 1 -- 172.21.15.88:0/2901457850 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8780384e0 con 0x7fa8700050d0 2024-09-16T01:34:30.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.925+0000 7fa884f99640 1 --2- 172.21.15.88:0/2901457850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8700050d0 0x7fa878037de0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa87400c430 tx=0x7fa87400c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.925+0000 7fe7cbfff640 1 --2- 172.21.15.88:0/2825029594 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe7b006c2e0 0x7fe7b006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.925+0000 7fe7d11ab640 1 --2- 172.21.15.88:0/2825029594 >> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] conn(0x7fe7b0074500 0x7fe7b0076920 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.925+0000 7f845a7fc640 1 -- 172.21.15.88:0/1400931706 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f84600551c0 con 0x7f846415e720 2024-09-16T01:34:30.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.925+0000 7f8469bfa640 1 --2- 172.21.15.88:0/1400931706 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f844806c380 0x7f844806e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.925+0000 7fa87d7fa640 1 -- 172.21.15.88:0/2901457850 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa874017070 con 0x7fa8700050d0 2024-09-16T01:34:30.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.925+0000 7fa885f9b640 1 -- 172.21.15.88:0/2901457850 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8780e0450 con 0x7fa8700050d0 2024-09-16T01:34:30.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.925+0000 7fa87d7fa640 1 -- 172.21.15.88:0/2901457850 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8740074e0 con 0x7fa8700050d0 2024-09-16T01:34:30.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.925+0000 7f84693f9640 1 --2- 172.21.15.88:0/1400931706 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x7f8448074520 0x7f8448076940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.925+0000 7f40cad9d640 1 --2- 172.21.15.88:0/2955451133 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x7f40ac0744d0 0x7f40ac0768f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:30.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.925+0000 7fd4854ae640 1 --2- 172.21.15.88:0/2846427869 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x7fd464074850 0x7fd464076c70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.926+0000 7fe7cbfff640 1 --2- 172.21.15.88:0/2825029594 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe7b006c2e0 0x7fe7b006e7a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc070000 tx=0x7fe7b8007b90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.926+0000 7f40ca59c640 1 --2- 172.21.15.88:0/2955451133 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f40ac06c2b0 0x7f40ac06e770 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f40bc009bd0 tx=0x7f40bc009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.926+0000 7f73f8ff9640 1 -- 172.21.15.88:0/608439694 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f73e800bc20 con 0x7f73f4035a80 2024-09-16T01:34:30.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.926+0000 7f84693f9640 1 --2- 172.21.15.88:0/1400931706 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x7f8448074520 0x7f8448076940 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.926+0000 7f8469bfa640 1 --2- 172.21.15.88:0/1400931706 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f844806c380 0x7f844806e840 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8454034000 tx=0x7f845402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.925+0000 7fa87d7fa640 1 -- 172.21.15.88:0/2901457850 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8740059a0 con 0x7fa8700050d0 2024-09-16T01:34:30.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.925+0000 7fa885f9b640 1 -- 172.21.15.88:0/2901457850 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa87804c8d0 con 0x7fa8700050d0 2024-09-16T01:34:30.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.926+0000 7f9a2517c640 1 --2- 172.21.15.88:0/2783725967 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x7f9a201ed040 0x7f9a20162ed0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.926+0000 7f40cad9d640 1 --2- 172.21.15.88:0/2955451133 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x7f40ac0744d0 0x7f40ac0768f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:30.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.926+0000 7fe7c97fa640 1 -- 172.21.15.88:0/2825029594 <== osd.4 v2:172.21.15.154:6816/2520717231 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe7c0007630 con 0x7fe7b0074500 2024-09-16T01:34:30.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.926+0000 7fa885f9b640 1 -- 172.21.15.88:0/2901457850 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa878009f00 con 0x7fa8700050d0 2024-09-16T01:34:30.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.926+0000 7f73f8ff9640 1 --2- 172.21.15.88:0/608439694 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f73e006c380 0x7f73e006e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:30.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.927+0000 7fd47d7fa640 1 -- 172.21.15.88:0/2846427869 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd46c022e50 con 0x7fd464074850 2024-09-16T01:34:30.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.927+0000 7f73f8ff9640 1 -- 172.21.15.88:0/608439694 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f73e808b790 con 0x7f73f4035a80 2024-09-16T01:34:31.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.926+0000 7f9a157fa640 1 -- 172.21.15.88:0/2783725967 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9a201ed420 con 0x7f9a201ed040 2024-09-16T01:34:31.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.927+0000 7f73fbfff640 1 --2- 172.21.15.88:0/608439694 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f73e006c380 0x7f73e006e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.927+0000 7f845a7fc640 1 -- 172.21.15.88:0/1400931706 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f846000fe50 con 0x7f8448074520 2024-09-16T01:34:31.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.927+0000 7f73f8ff9640 1 --2- 172.21.15.88:0/608439694 >> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] conn(0x7f73e00745a0 0x7f73e00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.927+0000 7f73f8ff9640 1 -- 172.21.15.88:0/608439694 --> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f73e8019070 con 0x7f73e00745a0 2024-09-16T01:34:31.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.927+0000 7f73f8ff9640 1 -- 172.21.15.88:0/608439694 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f73e8092050 con 0x7f73f4035a80 2024-09-16T01:34:31.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.927+0000 7f40baffd640 1 -- 172.21.15.88:0/2955451133 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f40c002be50 con 0x7f40ac0744d0 2024-09-16T01:34:31.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.927+0000 7f7400ec9640 1 --2- 172.21.15.88:0/608439694 >> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] conn(0x7f73e00745a0 0x7f73e00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.927+0000 7f73fbfff640 1 --2- 172.21.15.88:0/608439694 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f73e006c380 0x7f73e006e840 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f73f0033000 tx=0x7f73f002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.927+0000 7fa87d7fa640 1 -- 172.21.15.88:0/2901457850 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa87401a070 con 0x7fa8700050d0 2024-09-16T01:34:31.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.928+0000 7fa87d7fa640 1 --2- 172.21.15.88:0/2901457850 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa86406c2b0 0x7fa86406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.928+0000 7f7400ec9640 1 --2- 172.21.15.88:0/608439694 >> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] conn(0x7f73e00745a0 0x7f73e00769c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.928+0000 7fa87d7fa640 1 -- 172.21.15.88:0/2901457850 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa87408a740 con 0x7fa8700050d0 2024-09-16T01:34:31.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.928+0000 7fa87ffff640 1 --2- 172.21.15.88:0/2901457850 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa86406c2b0 0x7fa86406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.928+0000 7fa87d7fa640 1 --2- 172.21.15.88:0/2901457850 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x7fa8640744d0 0x7fa8640768f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.929+0000 7f73f8ff9640 1 -- 172.21.15.88:0/608439694 <== osd.2 v2:172.21.15.88:6808/2918260303 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f73e8019070 con 0x7f73e00745a0 2024-09-16T01:34:31.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.928+0000 7fa87d7fa640 1 -- 172.21.15.88:0/2901457850 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa874015e70 con 0x7fa8640744d0 2024-09-16T01:34:31.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.928+0000 7fa87d7fa640 1 -- 172.21.15.88:0/2901457850 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa874091050 con 0x7fa8700050d0 2024-09-16T01:34:31.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.929+0000 7fa88579a640 1 --2- 172.21.15.88:0/2901457850 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x7fa8640744d0 0x7fa8640768f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.929+0000 7fa88579a640 1 --2- 172.21.15.88:0/2901457850 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x7fa8640744d0 0x7fa8640768f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.930+0000 7fa87ffff640 1 --2- 172.21.15.88:0/2901457850 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa86406c2b0 0x7fa86406e770 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa8781743d0 tx=0x7fa86c019040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.930+0000 7fa87d7fa640 1 -- 172.21.15.88:0/2901457850 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa874015e70 con 0x7fa8640744d0 2024-09-16T01:34:31.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.957+0000 7fd485caf640 1 -- 172.21.15.88:0/2846427869 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd480151d20 con 0x7fd464074850 2024-09-16T01:34:31.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.957+0000 7fe7d2c0e640 1 -- 172.21.15.88:0/2825029594 --> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe7cc0cd8a0 con 0x7fe7b0074500 2024-09-16T01:34:31.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.957+0000 7f846abfc640 1 -- 172.21.15.88:0/1400931706 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f84640ce9b0 con 0x7f8448074520 2024-09-16T01:34:31.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.957+0000 7fd47d7fa640 1 -- 172.21.15.88:0/2846427869 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd480151d20 con 0x7fd464074850 2024-09-16T01:34:31.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.958+0000 7fd49bae6640 1 -- 172.21.15.88:0/629620761 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd47c002dd0 con 0x7fd4780744b0 2024-09-16T01:34:31.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.958+0000 7fe7c97fa640 1 -- 172.21.15.88:0/2825029594 <== osd.4 v2:172.21.15.154:6816/2520717231 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe7cc0cd8a0 con 0x7fe7b0074500 2024-09-16T01:34:31.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.958+0000 7f9a26bdf640 1 -- 172.21.15.88:0/2783725967 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9a200b5660 con 0x7f9a201ed040 2024-09-16T01:34:31.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.958+0000 7fd46affd640 1 -- 172.21.15.88:0/2846427869 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x7fd464074850 msgr2=0x7fd464076c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.958+0000 7fd46affd640 1 --2- 172.21.15.88:0/2846427869 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x7fd464074850 0x7fd464076c70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.958+0000 7f845a7fc640 1 -- 172.21.15.88:0/1400931706 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f84640ce9b0 con 0x7f8448074520 2024-09-16T01:34:31.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.958+0000 7f40cb59e640 1 -- 172.21.15.88:0/2955451133 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f40c404f7e0 con 0x7f40ac0744d0 2024-09-16T01:34:31.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.958+0000 7fd46affd640 1 -- 172.21.15.88:0/2846427869 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd46406c6b0 msgr2=0x7fd46406eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.958+0000 7fd48a7fc640 1 -- 172.21.15.88:0/629620761 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd47c002dd0 con 0x7fd4780744b0 2024-09-16T01:34:31.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.958+0000 7fd46affd640 1 --2- 172.21.15.88:0/2846427869 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd46406c6b0 0x7fd46406eb70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd4740099d0 tx=0x7fd474009190 comp rx=0 tx=0).stop 2024-09-16T01:34:31.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.958+0000 7fd46affd640 1 -- 172.21.15.88:0/2846427869 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4801624d0 msgr2=0x7fd480150160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.958+0000 7fd46affd640 1 --2- 172.21.15.88:0/2846427869 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4801624d0 0x7fd480150160 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd46c00b3f0 tx=0x7fd46c00b8b0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.958+0000 7f9a157fa640 1 -- 172.21.15.88:0/2783725967 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9a200b5660 con 0x7f9a201ed040 2024-09-16T01:34:31.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.958+0000 7fd4854ae640 1 -- 172.21.15.88:0/2846427869 reap_dead start 2024-09-16T01:34:31.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.958+0000 7fe7d2c0e640 1 -- 172.21.15.88:0/2825029594 >> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] conn(0x7fe7b0074500 msgr2=0x7fe7b0076920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.958+0000 7fe7d2c0e640 1 --2- 172.21.15.88:0/2825029594 >> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] conn(0x7fe7b0074500 0x7fe7b0076920 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.958+0000 7fd46affd640 1 -- 172.21.15.88:0/2846427869 shutdown_connections 2024-09-16T01:34:31.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.958+0000 7fd46affd640 1 -- 172.21.15.88:0/2846427869 >> 172.21.15.88:0/2846427869 conn(0x7fd48006d0d0 msgr2=0x7fd480162d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:31.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.958+0000 7fe7d2c0e640 1 -- 172.21.15.88:0/2825029594 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe7b006c2e0 msgr2=0x7fe7b006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.958+0000 7fe7d2c0e640 1 --2- 172.21.15.88:0/2825029594 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe7b006c2e0 0x7fe7b006e7a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc070000 tx=0x7fe7b8007b90 comp rx=0 tx=0).stop 2024-09-16T01:34:31.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7fe7d2c0e640 1 -- 172.21.15.88:0/2825029594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7cc0b5470 msgr2=0x7fe7cc14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7fe7d2c0e640 1 --2- 172.21.15.88:0/2825029594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7cc0b5470 0x7fe7cc14fff0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe7c000c410 tx=0x7fe7c000c8d0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7fe7d11ab640 1 -- 172.21.15.88:0/2825029594 reap_dead start 2024-09-16T01:34:31.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7f846abfc640 1 -- 172.21.15.88:0/1400931706 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x7f8448074520 msgr2=0x7f8448076940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7f846abfc640 1 --2- 172.21.15.88:0/1400931706 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x7f8448074520 0x7f8448076940 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7fd46affd640 1 -- 172.21.15.88:0/2846427869 shutdown_connections 2024-09-16T01:34:31.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7fd49bae6640 1 -- 172.21.15.88:0/629620761 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x7fd4780744b0 msgr2=0x7fd4780768d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7fd49bae6640 1 --2- 172.21.15.88:0/629620761 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x7fd4780744b0 0x7fd4780768d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7fe7d2c0e640 1 -- 172.21.15.88:0/2825029594 shutdown_connections 2024-09-16T01:34:31.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7fe7d2c0e640 1 -- 172.21.15.88:0/2825029594 >> 172.21.15.88:0/2825029594 conn(0x7fe7cc0d1390 msgr2=0x7fe7cc06bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:31.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7fd46affd640 1 -- 172.21.15.88:0/2846427869 wait complete. 2024-09-16T01:34:31.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7f9a26bdf640 1 -- 172.21.15.88:0/2783725967 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x7f9a201ed040 msgr2=0x7f9a20162ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7f9a26bdf640 1 --2- 172.21.15.88:0/2783725967 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x7f9a201ed040 0x7f9a20162ed0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7fe7d2c0e640 1 -- 172.21.15.88:0/2825029594 shutdown_connections 2024-09-16T01:34:31.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7fe7d2c0e640 1 -- 172.21.15.88:0/2825029594 wait complete. 2024-09-16T01:34:31.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7f846abfc640 1 -- 172.21.15.88:0/1400931706 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f844806c380 msgr2=0x7f844806e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7f846abfc640 1 --2- 172.21.15.88:0/1400931706 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f844806c380 0x7f844806e840 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8454034000 tx=0x7f845402e000 comp rx=0 tx=0).stop 2024-09-16T01:34:31.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7f846abfc640 1 -- 172.21.15.88:0/1400931706 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f846415e720 msgr2=0x7f846406f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7f846abfc640 1 --2- 172.21.15.88:0/1400931706 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f846415e720 0x7f846406f6e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f846000e5f0 tx=0x7f846000eab0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7f846a3fb640 1 -- 172.21.15.88:0/1400931706 reap_dead start 2024-09-16T01:34:31.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7f40baffd640 1 -- 172.21.15.88:0/2955451133 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f40c404f7e0 con 0x7f40ac0744d0 2024-09-16T01:34:31.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7f846abfc640 1 -- 172.21.15.88:0/1400931706 shutdown_connections 2024-09-16T01:34:31.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7f846abfc640 1 -- 172.21.15.88:0/1400931706 >> 172.21.15.88:0/1400931706 conn(0x7f8464062630 msgr2=0x7f8464160160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:31.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7f9a26bdf640 1 -- 172.21.15.88:0/2783725967 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9a0c06c330 msgr2=0x7f9a0c06e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7f9a26bdf640 1 --2- 172.21.15.88:0/2783725967 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9a0c06c330 0x7f9a0c06e7f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9a20162380 tx=0x7f9a10009040 comp rx=0 tx=0).stop 2024-09-16T01:34:31.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7f9a26bdf640 1 -- 172.21.15.88:0/2783725967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a200b4700 msgr2=0x7f9a201f3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7f9a26bdf640 1 --2- 172.21.15.88:0/2783725967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a200b4700 0x7f9a201f3d70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c02a6c0 tx=0x7f9a1c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7f9a2517c640 1 -- 172.21.15.88:0/2783725967 reap_dead start 2024-09-16T01:34:31.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7f846abfc640 1 -- 172.21.15.88:0/1400931706 shutdown_connections 2024-09-16T01:34:31.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7f846abfc640 1 -- 172.21.15.88:0/1400931706 wait complete. 2024-09-16T01:34:31.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7fd49bae6640 1 -- 172.21.15.88:0/629620761 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd47806c2b0 msgr2=0x7fd47806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7fd49bae6640 1 --2- 172.21.15.88:0/629620761 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd47806c2b0 0x7fd47806e770 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd490034000 tx=0x7fd49002e000 comp rx=0 tx=0).stop 2024-09-16T01:34:31.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7fd49bae6640 1 -- 172.21.15.88:0/629620761 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4940b7530 msgr2=0x7fd49406f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7fd49bae6640 1 --2- 172.21.15.88:0/629620761 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4940b7530 0x7fd49406f6c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd48400c3f0 tx=0x7fd48400c8b0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.959+0000 7fd49a083640 1 -- 172.21.15.88:0/629620761 reap_dead start 2024-09-16T01:34:31.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.960+0000 7fd49bae6640 1 -- 172.21.15.88:0/629620761 shutdown_connections 2024-09-16T01:34:31.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.960+0000 7fd49bae6640 1 -- 172.21.15.88:0/629620761 >> 172.21.15.88:0/629620761 conn(0x7fd4940d24d0 msgr2=0x7fd4940b2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:31.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.960+0000 7f9a26bdf640 1 -- 172.21.15.88:0/2783725967 shutdown_connections 2024-09-16T01:34:31.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.960+0000 7f9a26bdf640 1 -- 172.21.15.88:0/2783725967 >> 172.21.15.88:0/2783725967 conn(0x7f9a2006d0f0 msgr2=0x7f9a200b3490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:31.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.960+0000 7f9a26bdf640 1 -- 172.21.15.88:0/2783725967 shutdown_connections 2024-09-16T01:34:31.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.960+0000 7f40cb59e640 1 -- 172.21.15.88:0/2955451133 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x7f40ac0744d0 msgr2=0x7f40ac0768f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.960+0000 7f40cb59e640 1 --2- 172.21.15.88:0/2955451133 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x7f40ac0744d0 0x7f40ac0768f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.960+0000 7f9a26bdf640 1 -- 172.21.15.88:0/2783725967 wait complete. 2024-09-16T01:34:31.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.960+0000 7fd49bae6640 1 -- 172.21.15.88:0/629620761 shutdown_connections 2024-09-16T01:34:31.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.960+0000 7f74016ca640 1 -- 172.21.15.88:0/608439694 --> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f73f40108d0 con 0x7f73e00745a0 2024-09-16T01:34:31.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.960+0000 7fd49bae6640 1 -- 172.21.15.88:0/629620761 wait complete. 2024-09-16T01:34:31.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.960+0000 7fa885f9b640 1 -- 172.21.15.88:0/2901457850 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa8780321a0 con 0x7fa8640744d0 2024-09-16T01:34:31.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.960+0000 7f40cb59e640 1 -- 172.21.15.88:0/2955451133 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f40ac06c2b0 msgr2=0x7f40ac06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.960+0000 7f40cb59e640 1 --2- 172.21.15.88:0/2955451133 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f40ac06c2b0 0x7f40ac06e770 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f40bc009bd0 tx=0x7f40bc009240 comp rx=0 tx=0).stop 2024-09-16T01:34:31.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.960+0000 7f40cb59e640 1 -- 172.21.15.88:0/2955451133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40c4057010 msgr2=0x7f40c417caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.960+0000 7f40cb59e640 1 --2- 172.21.15.88:0/2955451133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40c4057010 0x7f40c417caf0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f40c0009890 tx=0x7f40c00046c0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.960+0000 7f40cad9d640 1 -- 172.21.15.88:0/2955451133 reap_dead start 2024-09-16T01:34:31.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.961+0000 7fa87d7fa640 1 -- 172.21.15.88:0/2901457850 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa8780321a0 con 0x7fa8640744d0 2024-09-16T01:34:31.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.961+0000 7f73f8ff9640 1 -- 172.21.15.88:0/608439694 <== osd.2 v2:172.21.15.88:6808/2918260303 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f73f40108d0 con 0x7f73e00745a0 2024-09-16T01:34:31.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.961+0000 7f40cb59e640 1 -- 172.21.15.88:0/2955451133 shutdown_connections 2024-09-16T01:34:31.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.961+0000 7f40cb59e640 1 -- 172.21.15.88:0/2955451133 >> 172.21.15.88:0/2955451133 conn(0x7f40c403a520 msgr2=0x7f40c40eb320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:31.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.961+0000 7f40cb59e640 1 -- 172.21.15.88:0/2955451133 shutdown_connections 2024-09-16T01:34:31.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.961+0000 7f40cb59e640 1 -- 172.21.15.88:0/2955451133 wait complete. 2024-09-16T01:34:31.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.961+0000 7fa885f9b640 1 -- 172.21.15.88:0/2901457850 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x7fa8640744d0 msgr2=0x7fa8640768f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.961+0000 7fa885f9b640 1 --2- 172.21.15.88:0/2901457850 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x7fa8640744d0 0x7fa8640768f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.962+0000 7fa885f9b640 1 -- 172.21.15.88:0/2901457850 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa86406c2b0 msgr2=0x7fa86406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.962+0000 7fa885f9b640 1 --2- 172.21.15.88:0/2901457850 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa86406c2b0 0x7fa86406e770 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa8781743d0 tx=0x7fa86c019040 comp rx=0 tx=0).stop 2024-09-16T01:34:31.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.962+0000 7fa885f9b640 1 -- 172.21.15.88:0/2901457850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8700050d0 msgr2=0x7fa878037de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.962+0000 7fa885f9b640 1 --2- 172.21.15.88:0/2901457850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8700050d0 0x7fa878037de0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa87400c430 tx=0x7fa87400c8f0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.962+0000 7f74016ca640 1 -- 172.21.15.88:0/608439694 >> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] conn(0x7f73e00745a0 msgr2=0x7f73e00769c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.962+0000 7f74016ca640 1 --2- 172.21.15.88:0/608439694 >> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] conn(0x7f73e00745a0 0x7f73e00769c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.962+0000 7fa88579a640 1 -- 172.21.15.88:0/2901457850 reap_dead start 2024-09-16T01:34:31.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.962+0000 7f74016ca640 1 -- 172.21.15.88:0/608439694 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f73e006c380 msgr2=0x7f73e006e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.962+0000 7f74016ca640 1 --2- 172.21.15.88:0/608439694 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f73e006c380 0x7f73e006e840 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f73f0033000 tx=0x7f73f002e000 comp rx=0 tx=0).stop 2024-09-16T01:34:31.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.962+0000 7f74016ca640 1 -- 172.21.15.88:0/608439694 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73f4035a80 msgr2=0x7f73f40dfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.962+0000 7f74016ca640 1 --2- 172.21.15.88:0/608439694 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73f4035a80 0x7f73f40dfaf0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f73e800d800 tx=0x7f73e800dcc0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.962+0000 7f7400ec9640 1 -- 172.21.15.88:0/608439694 reap_dead start 2024-09-16T01:34:31.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.962+0000 7f74016ca640 1 -- 172.21.15.88:0/608439694 shutdown_connections 2024-09-16T01:34:31.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.962+0000 7f74016ca640 1 -- 172.21.15.88:0/608439694 >> 172.21.15.88:0/608439694 conn(0x7f73f40567b0 msgr2=0x7f73f40ddee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:31.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.962+0000 7fa885f9b640 1 -- 172.21.15.88:0/2901457850 shutdown_connections 2024-09-16T01:34:31.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.962+0000 7fa885f9b640 1 -- 172.21.15.88:0/2901457850 >> 172.21.15.88:0/2901457850 conn(0x7fa878036090 msgr2=0x7fa87801d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:31.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.963+0000 7fa885f9b640 1 -- 172.21.15.88:0/2901457850 shutdown_connections 2024-09-16T01:34:31.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.963+0000 7fa885f9b640 1 -- 172.21.15.88:0/2901457850 wait complete. 2024-09-16T01:34:31.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.963+0000 7f74016ca640 1 -- 172.21.15.88:0/608439694 shutdown_connections 2024-09-16T01:34:31.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:30.963+0000 7f74016ca640 1 -- 172.21.15.88:0/608439694 wait complete. 2024-09-16T01:34:31.028 INFO:teuthology.orchestra.run.smithi088.stdout:60129542152 2024-09-16T01:34:31.029 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T01:34:31.030 INFO:teuthology.orchestra.run.smithi088.stdout:60129542151 2024-09-16T01:34:31.030 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T01:34:31.031 INFO:teuthology.orchestra.run.smithi088.stdout:60129542151 2024-09-16T01:34:31.031 INFO:teuthology.orchestra.run.smithi088.stdout:60129542152 2024-09-16T01:34:31.032 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T01:34:31.032 INFO:teuthology.orchestra.run.smithi088.stdout:64424509448 2024-09-16T01:34:31.033 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T01:34:31.033 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T01:34:31.034 INFO:teuthology.orchestra.run.smithi088.stdout:60129542152 2024-09-16T01:34:31.034 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T01:34:31.035 INFO:teuthology.orchestra.run.smithi088.stdout:60129542151 2024-09-16T01:34:31.035 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T01:34:31.036 INFO:teuthology.orchestra.run.smithi088.stdout:60129542152 2024-09-16T01:34:31.037 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T01:34:31.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.235+0000 7fb2eb59e640 1 Processor -- start 2024-09-16T01:34:31.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.235+0000 7fb2eb59e640 1 -- start start 2024-09-16T01:34:31.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.235+0000 7f4c3e3c4640 1 Processor -- start 2024-09-16T01:34:31.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.235+0000 7f4c3e3c4640 1 -- start start 2024-09-16T01:34:31.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.236+0000 7fb2eb59e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb2ec06e350 0x7fb2ec0cfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.236+0000 7fb2eb59e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb2ec0d0210 0x7fb2ec0d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.237+0000 7fb2eb59e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2ec0ccb10 con 0x7fb2ec0b4490 2024-09-16T01:34:31.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.237+0000 7fb2eb59e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb2ec0b3c40 con 0x7fb2ec06e350 2024-09-16T01:34:31.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.237+0000 7fb2eb59e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb2ec0d0bc0 con 0x7fb2ec0d0210 2024-09-16T01:34:31.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.237+0000 7fb2e9d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb2ec06e350 0x7fb2ec0cfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.237+0000 7fb2e9d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb2ec06e350 0x7fb2ec0cfa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54928/0 (socket says 172.21.15.88:54928) 2024-09-16T01:34:31.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.237+0000 7fb2e9d9b640 1 -- 172.21.15.88:0/1923782043 learned_addr learned my addr 172.21.15.88:0/1923782043 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:31.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.237+0000 7f4c3e3c4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4c3806c320 0x7f4c38073350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.237+0000 7f4c3e3c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c38073f30 0x7f4c380b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.237+0000 7f4c3e3c4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c38072f50 con 0x7f4c38162760 2024-09-16T01:34:31.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.237+0000 7f4c3e3c4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f4c380b4430 con 0x7f4c3806c320 2024-09-16T01:34:31.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.237+0000 7f4c3e3c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4c38072c70 con 0x7f4c38073f30 2024-09-16T01:34:31.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.237+0000 7fb2e9d9b640 1 -- 172.21.15.88:0/1923782043 >> v2:172.21.15.154:3300/0 conn(0x7fb2ec0d0210 msgr2=0x7fb2ec0d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:31.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.237+0000 7fb2e9d9b640 1 --2- 172.21.15.88:0/1923782043 >> v2:172.21.15.154:3300/0 conn(0x7fb2ec0d0210 0x7fb2ec0d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.237+0000 7f4c3c961640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c38073f30 0x7f4c380b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.237+0000 7f4c3c961640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c38073f30 0x7f4c380b4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:35458/0 (socket says 172.21.15.88:35458) 2024-09-16T01:34:31.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.237+0000 7f4c3c961640 1 -- 172.21.15.88:0/4231766538 learned_addr learned my addr 172.21.15.88:0/4231766538 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:31.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.237+0000 7f4c377fe640 1 --2- 172.21.15.88:0/4231766538 >> v2:172.21.15.154:3300/0 conn(0x7f4c3806c320 0x7f4c38073350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.237+0000 7fb2e9d9b640 1 -- 172.21.15.88:0/1923782043 >> v1:172.21.15.88:6789/0 conn(0x7fb2ec0b4490 legacy=0x7fb2ec0b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:31.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.237+0000 7fb2e9d9b640 1 -- 172.21.15.88:0/1923782043 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2ec06e9a0 con 0x7fb2ec06e350 2024-09-16T01:34:31.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.238+0000 7f4c36ffd640 1 -- 172.21.15.88:0/4231766538 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 773291644 0 0) 0x7f4c38072f50 con 0x7f4c38162760 2024-09-16T01:34:31.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.238+0000 7f4c36ffd640 1 -- 172.21.15.88:0/4231766538 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c20003520 con 0x7f4c38162760 2024-09-16T01:34:31.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.238+0000 7f4c3c961640 1 -- 172.21.15.88:0/4231766538 >> v2:172.21.15.154:3300/0 conn(0x7f4c3806c320 msgr2=0x7f4c38073350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:31.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.238+0000 7f4c3c961640 1 --2- 172.21.15.88:0/4231766538 >> v2:172.21.15.154:3300/0 conn(0x7f4c3806c320 0x7f4c38073350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.238+0000 7f4c3c961640 1 -- 172.21.15.88:0/4231766538 >> v1:172.21.15.88:6789/0 conn(0x7f4c38162760 legacy=0x7f4c3806bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.238+0000 7f4c3c961640 1 -- 172.21.15.88:0/4231766538 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c380b46d0 con 0x7f4c38073f30 2024-09-16T01:34:31.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.238+0000 7f7b77674640 1 Processor -- start 2024-09-16T01:34:31.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.239+0000 7f7b77674640 1 -- start start 2024-09-16T01:34:31.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.238+0000 7fb2e9d9b640 1 --2- 172.21.15.88:0/1923782043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2ec06e350 0x7fb2ec0cfa50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec0ce520 tx=0x7fb2e402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca1c956d0cbd5850 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.239+0000 7fb2e959a640 1 -- 172.21.15.88:0/1923782043 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e4036070 con 0x7fb2ec06e350 2024-09-16T01:34:31.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.239+0000 7fb2e959a640 1 -- 172.21.15.88:0/1923782043 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2e4031600 con 0x7fb2ec06e350 2024-09-16T01:34:31.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.239+0000 7fb2e959a640 1 -- 172.21.15.88:0/1923782043 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e40319d0 con 0x7fb2ec06e350 2024-09-16T01:34:31.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.239+0000 7ff743ef1640 1 Processor -- start 2024-09-16T01:34:31.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.239+0000 7f4c3c961640 1 --2- 172.21.15.88:0/4231766538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c38073f30 0x7f4c380b4ed0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4c30009e50 tx=0x7f4c3002a630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=286dce62e05afe63 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.239+0000 7f4c36ffd640 1 -- 172.21.15.88:0/4231766538 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c30036070 con 0x7f4c38073f30 2024-09-16T01:34:31.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.239+0000 7f0050e4c640 1 Processor -- start 2024-09-16T01:34:31.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.239+0000 7f4c36ffd640 1 -- 172.21.15.88:0/4231766538 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c30031600 con 0x7f4c38073f30 2024-09-16T01:34:31.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.239+0000 7f4c36ffd640 1 -- 172.21.15.88:0/4231766538 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c300319d0 con 0x7f4c38073f30 2024-09-16T01:34:31.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.240+0000 7f0050e4c640 1 -- start start 2024-09-16T01:34:31.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.240+0000 7f580359e640 1 Processor -- start 2024-09-16T01:34:31.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.240+0000 7f580359e640 1 -- start start 2024-09-16T01:34:31.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.240+0000 7fb2eb59e640 1 -- 172.21.15.88:0/1923782043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2ec06e350 msgr2=0x7fb2ec0cfa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.240+0000 7fb2eb59e640 1 --2- 172.21.15.88:0/1923782043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2ec06e350 0x7fb2ec0cfa50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec0ce520 tx=0x7fb2e402a730 comp rx=0 tx=0).stop 2024-09-16T01:34:31.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.240+0000 7f7b77674640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b7006f0a0 0x7f7b70071990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.240+0000 7f7b77674640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7b700734b0 0x7f7b70073960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.240+0000 7f7b77674640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b700b3990 con 0x7f7b700b3490 2024-09-16T01:34:31.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.240+0000 7f7b77674640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7b700b4510 con 0x7f7b700734b0 2024-09-16T01:34:31.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.240+0000 7f7b77674640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7b700b5df0 con 0x7f7b7006f0a0 2024-09-16T01:34:31.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.240+0000 7fb2eb59e640 1 -- 172.21.15.88:0/1923782043 shutdown_connections 2024-09-16T01:34:31.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.240+0000 7fb2eb59e640 1 --2- 172.21.15.88:0/1923782043 >> v2:172.21.15.154:3300/0 conn(0x7fb2ec0d0210 0x7fb2ec0d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.240+0000 7fb2eb59e640 1 --2- 172.21.15.88:0/1923782043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2ec06e350 0x7fb2ec0cfa50 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.240+0000 7fb2eb59e640 1 -- 172.21.15.88:0/1923782043 >> 172.21.15.88:0/1923782043 conn(0x7fb2ec06d0d0 msgr2=0x7fb2ec06d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:31.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.240+0000 7fb2eb59e640 1 -- 172.21.15.88:0/1923782043 shutdown_connections 2024-09-16T01:34:31.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.240+0000 7fb2eb59e640 1 -- 172.21.15.88:0/1923782043 wait complete. 2024-09-16T01:34:31.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.240+0000 7f7b75410640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b7006f0a0 0x7f7b70071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.240+0000 7f7b75410640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b7006f0a0 0x7f7b70071990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:35470/0 (socket says 172.21.15.88:35470) 2024-09-16T01:34:31.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.240+0000 7f7b75410640 1 -- 172.21.15.88:0/1637225967 learned_addr learned my addr 172.21.15.88:0/1637225967 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:31.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7f7b75410640 1 -- 172.21.15.88:0/1637225967 >> v2:172.21.15.88:3300/0 conn(0x7f7b700734b0 msgr2=0x7f7b70073960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:31.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7f7b75410640 1 --2- 172.21.15.88:0/1637225967 >> v2:172.21.15.88:3300/0 conn(0x7f7b700734b0 0x7f7b70073960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7f7b75410640 1 -- 172.21.15.88:0/1637225967 >> v1:172.21.15.154:6789/0 conn(0x7f7b700b3490 legacy=0x7f7b70143f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:31.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7f7b75410640 1 -- 172.21.15.88:0/1637225967 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b700b3c70 con 0x7f7b7006f0a0 2024-09-16T01:34:31.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.239+0000 7ff743ef1640 1 -- start start 2024-09-16T01:34:31.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7ff743ef1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff73c0cdaf0 0x7ff73c0d02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7f0050e4c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f004c159000 0x7f004c15c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7ff743ef1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff73c072210 0x7ff73c16c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7f0050e4c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f004c072190 0x7f004c0b55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7f0050e4c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f004c0b36a0 0x7f004c0b3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7f0050e4c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f004c0b5130 con 0x7f004c159000 2024-09-16T01:34:31.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7f0050e4c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f004c072f40 con 0x7f004c072190 2024-09-16T01:34:31.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7f0050e4c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f004c072980 con 0x7f004c0b36a0 2024-09-16T01:34:31.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7f4c3e3c4640 1 -- 172.21.15.88:0/4231766538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c38073f30 msgr2=0x7f4c380b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7f4c3e3c4640 1 --2- 172.21.15.88:0/4231766538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c38073f30 0x7f4c380b4ed0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4c30009e50 tx=0x7f4c3002a630 comp rx=0 tx=0).stop 2024-09-16T01:34:31.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7fcc833e0640 1 Processor -- start 2024-09-16T01:34:31.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7f580359e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f57fc053580 0x7f57fc0ea380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7f580359e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57fc006930 con 0x7f57fc04fc60 2024-09-16T01:34:31.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7f580359e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57fc006c10 con 0x7f57fc006470 2024-09-16T01:34:31.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7f580359e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f57fc032770 con 0x7f57fc053580 2024-09-16T01:34:31.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7f004ad9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f004c0b36a0 0x7f004c0b3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7f004ad9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f004c0b36a0 0x7f004c0b3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:35484/0 (socket says 172.21.15.88:35484) 2024-09-16T01:34:31.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7f004ad9d640 1 -- 172.21.15.88:0/867405975 learned_addr learned my addr 172.21.15.88:0/867405975 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:31.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7f0043fff640 1 --2- 172.21.15.88:0/867405975 >> v2:172.21.15.154:3300/0 conn(0x7f004c072190 0x7f004c0b55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.242+0000 7f580259c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f57fc04fc60 0x7f57fc01d630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45192/0 (socket says 172.21.15.88:45192) 2024-09-16T01:34:31.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.242+0000 7f580259c640 1 -- 172.21.15.88:0/1718242664 learned_addr learned my addr 172.21.15.88:0/1718242664 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:31.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.242+0000 7f5802d9d640 1 --2- 172.21.15.88:0/1718242664 >> v2:172.21.15.154:3300/0 conn(0x7f57fc053580 0x7f57fc0ea380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7ff743ef1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff73c062930 con 0x7ff73c071720 2024-09-16T01:34:31.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7ff743ef1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff73c0b3e30 con 0x7ff73c072210 2024-09-16T01:34:31.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7ff743ef1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff73c162ec0 con 0x7ff73c0cdaf0 2024-09-16T01:34:31.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7ff74148c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff73c0cdaf0 0x7ff73c0d02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.242+0000 7f7b75410640 1 --2- 172.21.15.88:0/1637225967 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b7006f0a0 0x7f7b70071990 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7b60009870 tx=0x7f7b6002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94e1900c75ec6d5f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7ff741c8d640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7ff73c071720 0x7ff73c071b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:59724/0 (socket says 172.21.15.88:59724) 2024-09-16T01:34:31.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.242+0000 7f5802d9d640 1 -- 172.21.15.88:0/1718242664 >> v1:172.21.15.154:6789/0 conn(0x7f57fc006470 legacy=0x7f57fc050040 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:34:31.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.242+0000 7f5802d9d640 1 -- 172.21.15.88:0/1718242664 >> v1:172.21.15.88:6789/0 conn(0x7f57fc04fc60 legacy=0x7f57fc01d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.242+0000 7f580159a640 1 -- 172.21.15.88:0/1718242664 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 553929434 0 0) 0x7f57fc006930 con 0x7f57fc04fc60 2024-09-16T01:34:31.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.242+0000 7f5802d9d640 1 -- 172.21.15.88:0/1718242664 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57fc006ef0 con 0x7f57fc053580 2024-09-16T01:34:31.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.242+0000 7fb2eb59e640 1 Processor -- start 2024-09-16T01:34:31.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.242+0000 7f004ad9d640 1 -- 172.21.15.88:0/867405975 >> v2:172.21.15.154:3300/0 conn(0x7f004c072190 msgr2=0x7f004c0b55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:31.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.242+0000 7f004ad9d640 1 --2- 172.21.15.88:0/867405975 >> v2:172.21.15.154:3300/0 conn(0x7f004c072190 0x7f004c0b55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.242+0000 7f004ad9d640 1 -- 172.21.15.88:0/867405975 >> v2:172.21.15.88:3300/0 conn(0x7f004c159000 msgr2=0x7f004c15c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:31.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.242+0000 7f004ad9d640 1 --2- 172.21.15.88:0/867405975 >> v2:172.21.15.88:3300/0 conn(0x7f004c159000 0x7f004c15c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.242+0000 7f004ad9d640 1 -- 172.21.15.88:0/867405975 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f004c072c60 con 0x7f004c0b36a0 2024-09-16T01:34:31.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.242+0000 7fb2eb59e640 1 -- start start 2024-09-16T01:34:31.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7ff741c8d640 1 -- 172.21.15.88:0/2520243490 learned_addr learned my addr 172.21.15.88:0/2520243490 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:31.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.242+0000 7ff74248e640 1 --2- 172.21.15.88:0/2520243490 >> v2:172.21.15.88:3300/0 conn(0x7ff73c072210 0x7ff73c16c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.242+0000 7ff74148c640 1 -- 172.21.15.88:0/2520243490 >> v2:172.21.15.88:3300/0 conn(0x7ff73c072210 msgr2=0x7ff73c16c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:31.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.242+0000 7ff740c8b640 1 -- 172.21.15.88:0/2520243490 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3416093939 0 0) 0x7ff73c062930 con 0x7ff73c071720 2024-09-16T01:34:31.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.242+0000 7ff74148c640 1 --2- 172.21.15.88:0/2520243490 >> v2:172.21.15.88:3300/0 conn(0x7ff73c072210 0x7ff73c16c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.242+0000 7ff74148c640 1 -- 172.21.15.88:0/2520243490 >> v1:172.21.15.154:6789/0 conn(0x7ff73c071720 legacy=0x7ff73c071b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.242+0000 7ff74148c640 1 -- 172.21.15.88:0/2520243490 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff73c0b4080 con 0x7ff73c0cdaf0 2024-09-16T01:34:31.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.243+0000 7ff74248e640 1 --2- 172.21.15.88:0/2520243490 >> v2:172.21.15.88:3300/0 conn(0x7ff73c072210 0x7ff73c16c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:31.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.243+0000 7f4c3e3c4640 1 -- 172.21.15.88:0/4231766538 shutdown_connections 2024-09-16T01:34:31.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.243+0000 7f4c3e3c4640 1 --2- 172.21.15.88:0/4231766538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c38073f30 0x7f4c380b4ed0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.243+0000 7f4c3e3c4640 1 --2- 172.21.15.88:0/4231766538 >> v2:172.21.15.154:3300/0 conn(0x7f4c3806c320 0x7f4c38073350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.243+0000 7f4c3e3c4640 1 -- 172.21.15.88:0/4231766538 >> 172.21.15.88:0/4231766538 conn(0x7f4c3806e0f0 msgr2=0x7f4c3806e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:31.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.243+0000 7ff07768b640 1 Processor -- start 2024-09-16T01:34:31.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.243+0000 7f004ad9d640 1 --2- 172.21.15.88:0/867405975 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f004c0b36a0 0x7f004c0b3b30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f004c06ce10 tx=0x7f004402a6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9950d3ccade8420 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.242+0000 7f7b67fff640 1 -- 172.21.15.88:0/1637225967 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b60037070 con 0x7f7b7006f0a0 2024-09-16T01:34:31.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.242+0000 7f7b67fff640 1 -- 172.21.15.88:0/1637225967 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b60032440 con 0x7f7b7006f0a0 2024-09-16T01:34:31.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.242+0000 7f7b67fff640 1 -- 172.21.15.88:0/1637225967 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b60032810 con 0x7f7b7006f0a0 2024-09-16T01:34:31.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.241+0000 7fcc833e0640 1 -- start start 2024-09-16T01:34:31.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.243+0000 7fcc833e0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcc7c160680 0x7fcc7c06f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.243+0000 7f5802d9d640 1 --2- 172.21.15.88:0/1718242664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57fc053580 0x7f57fc0ea380 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f580402dd50 tx=0x7f5804065a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54ba8542701cea3b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.243+0000 7f4c3e3c4640 1 -- 172.21.15.88:0/4231766538 shutdown_connections 2024-09-16T01:34:31.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.243+0000 7f4c3e3c4640 1 -- 172.21.15.88:0/4231766538 wait complete. 2024-09-16T01:34:31.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.243+0000 7ff07768b640 1 -- start start 2024-09-16T01:34:31.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.243+0000 7fcc823de640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcc7c160680 0x7fcc7c06f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.243+0000 7fcc823de640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcc7c160680 0x7fcc7c06f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54950/0 (socket says 172.21.15.88:54950) 2024-09-16T01:34:31.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.243+0000 7f0049d9b640 1 -- 172.21.15.88:0/867405975 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0044036070 con 0x7f004c0b36a0 2024-09-16T01:34:31.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.243+0000 7f0049d9b640 1 -- 172.21.15.88:0/867405975 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0044031600 con 0x7f004c0b36a0 2024-09-16T01:34:31.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.243+0000 7f0049d9b640 1 -- 172.21.15.88:0/867405975 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0044031a10 con 0x7f004c0b36a0 2024-09-16T01:34:31.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.243+0000 7fb2eb59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2ec06e350 0x7fb2ec0aa480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7fcc82bdf640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fcc7c165200 0x7fcc7c166b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45204/0 (socket says 172.21.15.88:45204) 2024-09-16T01:34:31.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7fb2eb59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2ec0b4490 0x7fb2ec15f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7fb2eb59e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2ec0d0210 0x7fb2ec161db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7fb2eb59e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ec06e780 con 0x7fb2ec0b4490 2024-09-16T01:34:31.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7fb2eb59e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ec06ef60 con 0x7fb2ec06e350 2024-09-16T01:34:31.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7fb2eb59e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb2ec0ab8e0 con 0x7fb2ec0d0210 2024-09-16T01:34:31.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7fb2e9d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2ec0b4490 0x7fb2ec15f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7fb2e9d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2ec0b4490 0x7fb2ec15f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54960/0 (socket says 172.21.15.88:54960) 2024-09-16T01:34:31.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7fb2e9d9b640 1 -- 172.21.15.88:0/1933599293 learned_addr learned my addr 172.21.15.88:0/1933599293 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:31.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7fb2ea59c640 1 --2- 172.21.15.88:0/1933599293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2ec06e350 0x7fb2ec0aa480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.243+0000 7f7b77674640 1 -- 172.21.15.88:0/1637225967 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b7006f0a0 msgr2=0x7f7b70071990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.243+0000 7f7b77674640 1 --2- 172.21.15.88:0/1637225967 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b7006f0a0 0x7f7b70071990 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7b60009870 tx=0x7f7b6002a180 comp rx=0 tx=0).stop 2024-09-16T01:34:31.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7f7b77674640 1 -- 172.21.15.88:0/1637225967 shutdown_connections 2024-09-16T01:34:31.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7f7b77674640 1 --2- 172.21.15.88:0/1637225967 >> v2:172.21.15.88:3300/0 conn(0x7f7b700734b0 0x7f7b70073960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7f7b77674640 1 --2- 172.21.15.88:0/1637225967 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b7006f0a0 0x7f7b70071990 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7f7b77674640 1 -- 172.21.15.88:0/1637225967 >> 172.21.15.88:0/1637225967 conn(0x7f7b7006d0f0 msgr2=0x7f7b7006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:31.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7f7b77674640 1 -- 172.21.15.88:0/1637225967 shutdown_connections 2024-09-16T01:34:31.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7fb2ead9d640 1 --2- 172.21.15.88:0/1933599293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2ec0d0210 0x7fb2ec161db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7fcc833e0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc7c162e80 con 0x7fcc7c165200 2024-09-16T01:34:31.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7fcc81bdd640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fcc7c0cfba0 0x7fcc7c166470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:59748/0 (socket says 172.21.15.88:59748) 2024-09-16T01:34:31.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7fcc833e0640 1 -- 172.21.15.88:0/1246597463 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc7c167290 con 0x7fcc7c0cfba0 2024-09-16T01:34:31.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7fcc81bdd640 1 -- 172.21.15.88:0/1246597463 learned_addr learned my addr 172.21.15.88:0/1246597463 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:31.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7fcc833e0640 1 -- 172.21.15.88:0/1246597463 --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fcc7c163100 con 0x7fcc7c160680 2024-09-16T01:34:31.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7f7b77674640 1 -- 172.21.15.88:0/1637225967 wait complete. 2024-09-16T01:34:31.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7f580159a640 1 -- 172.21.15.88:0/1718242664 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58040b2070 con 0x7f57fc053580 2024-09-16T01:34:31.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7f580159a640 1 -- 172.21.15.88:0/1718242664 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58040ab7d0 con 0x7f57fc053580 2024-09-16T01:34:31.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7f580159a640 1 -- 172.21.15.88:0/1718242664 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58040abc40 con 0x7f57fc053580 2024-09-16T01:34:31.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7fcc813dc640 1 -- 172.21.15.88:0/1246597463 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3019972128 0 0) 0x7fcc74001490 con 0x7fcc7c165200 2024-09-16T01:34:31.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7fcc813dc640 1 -- 172.21.15.88:0/1246597463 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc68003520 con 0x7fcc7c165200 2024-09-16T01:34:31.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7fcc813dc640 1 -- 172.21.15.88:0/1246597463 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1912010983 0 0) 0x7fcc7c167290 con 0x7fcc7c0cfba0 2024-09-16T01:34:31.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7fcc813dc640 1 -- 172.21.15.88:0/1246597463 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc74001490 con 0x7fcc7c0cfba0 2024-09-16T01:34:31.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7fb2e9d9b640 1 -- 172.21.15.88:0/1933599293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2ec0d0210 msgr2=0x7fb2ec161db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7fb2e9d9b640 1 --2- 172.21.15.88:0/1933599293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2ec0d0210 0x7fb2ec161db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7fb2e9d9b640 1 -- 172.21.15.88:0/1933599293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2ec06e350 msgr2=0x7fb2ec0aa480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7fb2e9d9b640 1 --2- 172.21.15.88:0/1933599293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2ec06e350 0x7fb2ec0aa480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7fb2e9d9b640 1 -- 172.21.15.88:0/1933599293 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2ec0ad260 con 0x7fb2ec0b4490 2024-09-16T01:34:31.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7f0050e4c640 1 -- 172.21.15.88:0/867405975 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f004c0b36a0 msgr2=0x7f004c0b3b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7f0050e4c640 1 --2- 172.21.15.88:0/867405975 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f004c0b36a0 0x7f004c0b3b30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f004c06ce10 tx=0x7f004402a6b0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7ff07768b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0700b6890 0x7ff0700b3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7ff07768b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff070162150 0x7ff0700d22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7ff07768b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0700cf2a0 con 0x7ff070161d70 2024-09-16T01:34:31.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7ff07768b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff0700cf760 con 0x7ff070162150 2024-09-16T01:34:31.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.244+0000 7ff07768b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff0700cf9b0 con 0x7ff0700b6890 2024-09-16T01:34:31.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.245+0000 7f0050e4c640 1 -- 172.21.15.88:0/867405975 shutdown_connections 2024-09-16T01:34:31.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.245+0000 7f0050e4c640 1 --2- 172.21.15.88:0/867405975 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f004c0b36a0 0x7f004c0b3b30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.245+0000 7f0050e4c640 1 --2- 172.21.15.88:0/867405975 >> v2:172.21.15.154:3300/0 conn(0x7f004c072190 0x7f004c0b55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.245+0000 7f0050e4c640 1 --2- 172.21.15.88:0/867405975 >> v2:172.21.15.88:3300/0 conn(0x7f004c159000 0x7f004c15c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.245+0000 7f0050e4c640 1 -- 172.21.15.88:0/867405975 >> 172.21.15.88:0/867405975 conn(0x7f004c0b6b50 msgr2=0x7f004c0b6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:31.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.245+0000 7fcc823de640 1 -- 172.21.15.88:0/1246597463 >> v1:172.21.15.154:6789/0 conn(0x7fcc7c0cfba0 legacy=0x7fcc7c166470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.245+0000 7fcc823de640 1 -- 172.21.15.88:0/1246597463 >> v1:172.21.15.88:6789/0 conn(0x7fcc7c165200 legacy=0x7fcc7c166b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.245+0000 7fcc823de640 1 -- 172.21.15.88:0/1246597463 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc7c1633e0 con 0x7fcc7c160680 2024-09-16T01:34:31.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.245+0000 7f0050e4c640 1 -- 172.21.15.88:0/867405975 shutdown_connections 2024-09-16T01:34:31.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.245+0000 7ff075427640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7ff070161d70 0x7ff0700b6180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:59752/0 (socket says 172.21.15.88:59752) 2024-09-16T01:34:31.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.245+0000 7fb2e9d9b640 1 --2- 172.21.15.88:0/1933599293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2ec0b4490 0x7fb2ec15f660 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb2e402ad20 tx=0x7fb2e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.245+0000 7ff075427640 1 -- 172.21.15.88:0/497460723 learned_addr learned my addr 172.21.15.88:0/497460723 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:31.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.245+0000 7ff067fff640 1 -- 172.21.15.88:0/497460723 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1282512564 0 0) 0x7ff0700cf2a0 con 0x7ff070161d70 2024-09-16T01:34:31.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.245+0000 7ff067fff640 1 -- 172.21.15.88:0/497460723 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff05c003520 con 0x7ff070161d70 2024-09-16T01:34:31.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.245+0000 7f580359e640 1 -- 172.21.15.88:0/1718242664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57fc053580 msgr2=0x7f57fc0ea380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.245+0000 7f580359e640 1 --2- 172.21.15.88:0/1718242664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57fc053580 0x7f57fc0ea380 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f580402dd50 tx=0x7f5804065a90 comp rx=0 tx=0).stop 2024-09-16T01:34:31.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.245+0000 7f0050e4c640 1 -- 172.21.15.88:0/867405975 wait complete. 2024-09-16T01:34:31.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.245+0000 7fb2daffd640 1 -- 172.21.15.88:0/1933599293 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e4041030 con 0x7fb2ec0b4490 2024-09-16T01:34:31.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.245+0000 7f4c3e3c4640 1 Processor -- start 2024-09-16T01:34:31.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.245+0000 7fb2eb59e640 1 -- 172.21.15.88:0/1933599293 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2ec0ad540 con 0x7fb2ec0b4490 2024-09-16T01:34:31.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.245+0000 7fb2daffd640 1 -- 172.21.15.88:0/1933599293 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2e4037040 con 0x7fb2ec0b4490 2024-09-16T01:34:31.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.245+0000 7fb2daffd640 1 -- 172.21.15.88:0/1933599293 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e4036070 con 0x7fb2ec0b4490 2024-09-16T01:34:31.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7f4c3e3c4640 1 -- start start 2024-09-16T01:34:31.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7ff074c26640 1 --2- 172.21.15.88:0/497460723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0700b6890 0x7ff0700b3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.245+0000 7fb2eb59e640 1 -- 172.21.15.88:0/1933599293 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2ec15fba0 con 0x7fb2ec0b4490 2024-09-16T01:34:31.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7ff74148c640 1 --2- 172.21.15.88:0/2520243490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff73c0cdaf0 0x7ff73c0d02e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff73c06fcb0 tx=0x7ff72c02a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91cd0d341309ce21 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7fcc823de640 1 --2- 172.21.15.88:0/1246597463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc7c160680 0x7fcc7c06f940 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcc78009a00 tx=0x7fcc7802a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dede218817696fe server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7f7b77674640 1 Processor -- start 2024-09-16T01:34:31.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7ff067fff640 1 -- 172.21.15.88:0/497460723 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 639137317 0 0) 0x7ff05c003520 con 0x7ff070161d70 2024-09-16T01:34:31.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7ff075c28640 1 --2- 172.21.15.88:0/497460723 >> v2:172.21.15.154:3300/0 conn(0x7ff070162150 0x7ff0700d22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7ff067fff640 1 -- 172.21.15.88:0/497460723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0700b6890 msgr2=0x7ff0700b3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:31.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7ff067fff640 1 --2- 172.21.15.88:0/497460723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0700b6890 0x7ff0700b3390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7ff067fff640 1 -- 172.21.15.88:0/497460723 >> v2:172.21.15.154:3300/0 conn(0x7ff070162150 msgr2=0x7ff0700d22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:31.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7ff067fff640 1 --2- 172.21.15.88:0/497460723 >> v2:172.21.15.154:3300/0 conn(0x7ff070162150 0x7ff0700d22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7ff067fff640 1 -- 172.21.15.88:0/497460723 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0700cfc90 con 0x7ff070161d70 2024-09-16T01:34:31.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7ff067fff640 1 -- 172.21.15.88:0/497460723 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7ff0680019f0 con 0x7ff070161d70 2024-09-16T01:34:31.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7ff067fff640 1 -- 172.21.15.88:0/497460723 >> v1:172.21.15.154:6789/0 conn(0x7ff070161d70 legacy=0x7ff0700b6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7ff067fff640 1 --2- 172.21.15.88:0/497460723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff05c0039b0 0x7ff05c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7ff067fff640 1 --2- 172.21.15.88:0/497460723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff05c004800 0x7ff05c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7ff067fff640 1 --2- 172.21.15.88:0/497460723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff05c005210 0x7ff05c0141d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7ff067fff640 1 -- 172.21.15.88:0/497460723 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff0700cf760 con 0x7ff05c005210 2024-09-16T01:34:31.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7ff067fff640 1 -- 172.21.15.88:0/497460723 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff0700cf9b0 con 0x7ff05c004800 2024-09-16T01:34:31.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7ff067fff640 1 -- 172.21.15.88:0/497460723 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff05c0149d0 con 0x7ff05c0039b0 2024-09-16T01:34:31.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7ff075c28640 1 --2- 172.21.15.88:0/497460723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff05c005210 0x7ff05c0141d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7f7b77674640 1 -- start start 2024-09-16T01:34:31.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7f580359e640 1 -- 172.21.15.88:0/1718242664 shutdown_connections 2024-09-16T01:34:31.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7f580359e640 1 --2- 172.21.15.88:0/1718242664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57fc053580 0x7f57fc0ea380 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7f580359e640 1 -- 172.21.15.88:0/1718242664 >> 172.21.15.88:0/1718242664 conn(0x7f57fc036090 msgr2=0x7f57fc0364a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:31.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7f580359e640 1 -- 172.21.15.88:0/1718242664 shutdown_connections 2024-09-16T01:34:31.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7ff740c8b640 1 -- 172.21.15.88:0/2520243490 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72c037070 con 0x7ff73c0cdaf0 2024-09-16T01:34:31.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7ff740c8b640 1 -- 172.21.15.88:0/2520243490 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff72c032440 con 0x7ff73c0cdaf0 2024-09-16T01:34:31.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.246+0000 7ff740c8b640 1 -- 172.21.15.88:0/2520243490 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72c02ae60 con 0x7ff73c0cdaf0 2024-09-16T01:34:31.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.247+0000 7ff075427640 1 --2- 172.21.15.88:0/497460723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff05c0039b0 0x7ff05c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.247+0000 7f0050e4c640 1 Processor -- start 2024-09-16T01:34:31.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.247+0000 7fcc813dc640 1 -- 172.21.15.88:0/1246597463 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc78036070 con 0x7fcc7c160680 2024-09-16T01:34:31.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.247+0000 7fcc813dc640 1 -- 172.21.15.88:0/1246597463 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc78031600 con 0x7fcc7c160680 2024-09-16T01:34:31.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.247+0000 7fcc813dc640 1 -- 172.21.15.88:0/1246597463 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc780319d0 con 0x7fcc7c160680 2024-09-16T01:34:31.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.247+0000 7f4c3e3c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c3806c320 0x7f4c380cfe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.247+0000 7f4c3e3c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c38073f30 0x7f4c3814b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.247+0000 7ff074c26640 1 --2- 172.21.15.88:0/497460723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff05c004800 0x7ff05c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.247+0000 7f4c3e3c4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c38162760 0x7f4c38155440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.247+0000 7f4c377fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c38073f30 0x7f4c3814b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.247+0000 7f580359e640 1 -- 172.21.15.88:0/1718242664 wait complete. 2024-09-16T01:34:31.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.247+0000 7f4c377fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c38073f30 0x7f4c3814b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:35508/0 (socket says 172.21.15.88:35508) 2024-09-16T01:34:31.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.247+0000 7f4c377fe640 1 -- 172.21.15.88:0/2909427476 learned_addr learned my addr 172.21.15.88:0/2909427476 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:31.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.247+0000 7f4c3c961640 1 --2- 172.21.15.88:0/2909427476 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c38162760 0x7f4c38155440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.247+0000 7f7b77674640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b7006f0a0 0x7f7b701f1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.247+0000 7f4c3e3c4640 1 -- 172.21.15.88:0/2909427476 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4c38162b40 con 0x7f4c38162760 2024-09-16T01:34:31.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.247+0000 7ff743ef1640 1 -- 172.21.15.88:0/2520243490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff73c0cdaf0 msgr2=0x7ff73c0d02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.247+0000 7ff743ef1640 1 --2- 172.21.15.88:0/2520243490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff73c0cdaf0 0x7ff73c0d02e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff73c06fcb0 tx=0x7ff72c02a110 comp rx=0 tx=0).stop 2024-09-16T01:34:31.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.247+0000 7f4c3e3c4640 1 -- 172.21.15.88:0/2909427476 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4c38072900 con 0x7f4c3806c320 2024-09-16T01:34:31.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.247+0000 7f4c3e3c4640 1 -- 172.21.15.88:0/2909427476 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4c38162cc0 con 0x7f4c38073f30 2024-09-16T01:34:31.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.247+0000 7ff743ef1640 1 -- 172.21.15.88:0/2520243490 shutdown_connections 2024-09-16T01:34:31.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.247+0000 7ff743ef1640 1 --2- 172.21.15.88:0/2520243490 >> v2:172.21.15.88:3300/0 conn(0x7ff73c072210 0x7ff73c16c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.248+0000 7ff743ef1640 1 --2- 172.21.15.88:0/2520243490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff73c0cdaf0 0x7ff73c0d02e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.248+0000 7ff743ef1640 1 -- 172.21.15.88:0/2520243490 >> 172.21.15.88:0/2520243490 conn(0x7ff73c0b0aa0 msgr2=0x7ff73c0b0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:31.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.248+0000 7ff075c28640 1 -- 172.21.15.88:0/497460723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff05c0039b0 msgr2=0x7ff05c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:31.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.248+0000 7ff743ef1640 1 -- 172.21.15.88:0/2520243490 shutdown_connections 2024-09-16T01:34:31.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.248+0000 7ff075c28640 1 --2- 172.21.15.88:0/497460723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff05c0039b0 0x7ff05c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.248+0000 7ff075c28640 1 -- 172.21.15.88:0/497460723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff05c004800 msgr2=0x7ff05c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:31.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.248+0000 7ff075c28640 1 --2- 172.21.15.88:0/497460723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff05c004800 0x7ff05c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.247+0000 7f0050e4c640 1 -- start start 2024-09-16T01:34:31.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.248+0000 7f0050e4c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f004c072190 0x7f004c142b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.248+0000 7ff075c28640 1 -- 172.21.15.88:0/497460723 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0700cfc90 con 0x7ff05c005210 2024-09-16T01:34:31.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.248+0000 7f4c377fe640 1 -- 172.21.15.88:0/2909427476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c3806c320 msgr2=0x7f4c380cfe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.248+0000 7f4c377fe640 1 --2- 172.21.15.88:0/2909427476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c3806c320 0x7f4c380cfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.248+0000 7f4c377fe640 1 -- 172.21.15.88:0/2909427476 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c38162760 msgr2=0x7f4c38155440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.248+0000 7f7b77674640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b700734b0 0x7f7b70142920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.248+0000 7f7b77674640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b700b3490 0x7f7b701f7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.248+0000 7f7b77674640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7b70073ea0 con 0x7f7b7006f0a0 2024-09-16T01:34:31.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.248+0000 7f7b77674640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7b700b4230 con 0x7f7b700734b0 2024-09-16T01:34:31.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.248+0000 7f7b77674640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7b70071ed0 con 0x7f7b700b3490 2024-09-16T01:34:31.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.248+0000 7f4c377fe640 1 --2- 172.21.15.88:0/2909427476 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c38162760 0x7f4c38155440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.248+0000 7f4c377fe640 1 -- 172.21.15.88:0/2909427476 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c380d0470 con 0x7f4c38073f30 2024-09-16T01:34:31.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7fcc833e0640 1 -- 172.21.15.88:0/1246597463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc7c160680 msgr2=0x7fcc7c06f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7fcc833e0640 1 --2- 172.21.15.88:0/1246597463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc7c160680 0x7fcc7c06f940 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcc78009a00 tx=0x7fcc7802a480 comp rx=0 tx=0).stop 2024-09-16T01:34:31.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7f4c377fe640 1 --2- 172.21.15.88:0/2909427476 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c38073f30 0x7f4c3814b230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c00e870 tx=0x7f4c2c00ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7ff074c26640 1 --2- 172.21.15.88:0/497460723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff05c004800 0x7ff05c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:31.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.248+0000 7ff743ef1640 1 -- 172.21.15.88:0/2520243490 wait complete. 2024-09-16T01:34:31.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7fcc833e0640 1 -- 172.21.15.88:0/1246597463 shutdown_connections 2024-09-16T01:34:31.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7fcc833e0640 1 --2- 172.21.15.88:0/1246597463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc7c160680 0x7fcc7c06f940 secure :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fcc78009a00 tx=0x7fcc7802a480 comp rx=0 tx=0).stop 2024-09-16T01:34:31.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7fcc833e0640 1 -- 172.21.15.88:0/1246597463 >> 172.21.15.88:0/1246597463 conn(0x7fcc7c0b1640 msgr2=0x7fcc7c0b1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:31.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.247+0000 7fb2eb59e640 1 -- 172.21.15.88:0/1933599293 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2ec073a50 con 0x7fb2ec0b4490 2024-09-16T01:34:31.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.248+0000 7fb2daffd640 1 -- 172.21.15.88:0/1933599293 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb2e4044070 con 0x7fb2ec0b4490 2024-09-16T01:34:31.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7fb2daffd640 1 --2- 172.21.15.88:0/1933599293 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb2cc06c2b0 0x7fb2cc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7f0050e4c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f004c0b36a0 0x7f004c143bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7f0050e4c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f004c159000 0x7f004c149e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7f0050e4c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f004c0725b0 con 0x7f004c072190 2024-09-16T01:34:31.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7f0050e4c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f004c0b5b00 con 0x7f004c0b36a0 2024-09-16T01:34:31.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7f0050e4c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f004c072730 con 0x7f004c159000 2024-09-16T01:34:31.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7f004a59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f004c072190 0x7f004c142b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7f004a59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f004c072190 0x7f004c142b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54994/0 (socket says 172.21.15.88:54994) 2024-09-16T01:34:31.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7f004a59c640 1 -- 172.21.15.88:0/2986940918 learned_addr learned my addr 172.21.15.88:0/2986940918 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:31.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7fb2daffd640 1 -- 172.21.15.88:0/1933599293 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb2e40abde0 con 0x7fb2ec0b4490 2024-09-16T01:34:31.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7f4c3c961640 1 --2- 172.21.15.88:0/2909427476 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c38162760 0x7f4c38155440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:34:31.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7f4c34ff9640 1 -- 172.21.15.88:0/2909427476 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c017070 con 0x7f4c38073f30 2024-09-16T01:34:31.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7f4c34ff9640 1 -- 172.21.15.88:0/2909427476 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c2c00c940 con 0x7f4c38073f30 2024-09-16T01:34:31.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7f580359e640 1 Processor -- start 2024-09-16T01:34:31.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7f4c34ff9640 1 -- 172.21.15.88:0/2909427476 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c00cd30 con 0x7f4c38073f30 2024-09-16T01:34:31.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7ff075c28640 1 --2- 172.21.15.88:0/497460723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff05c005210 0x7ff05c0141d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff0700ba7b0 tx=0x7ff05c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bfea0e903f3755f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7f7b74c0f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b700734b0 0x7f7b70142920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7f4c3e3c4640 1 -- 172.21.15.88:0/2909427476 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c380d0750 con 0x7f4c38073f30 2024-09-16T01:34:31.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7f7b75c11640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b700b3490 0x7f7b701f7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7f4c3e3c4640 1 -- 172.21.15.88:0/2909427476 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c38154900 con 0x7f4c38073f30 2024-09-16T01:34:31.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.250+0000 7f580359e640 1 -- start start 2024-09-16T01:34:31.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.250+0000 7f7b75c11640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b700b3490 0x7f7b701f7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:35522/0 (socket says 172.21.15.88:35522) 2024-09-16T01:34:31.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.250+0000 7f7b75c11640 1 -- 172.21.15.88:0/1846311144 learned_addr learned my addr 172.21.15.88:0/1846311144 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:31.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.250+0000 7fb2ea59c640 1 --2- 172.21.15.88:0/1933599293 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb2cc06c2b0 0x7fb2cc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.250+0000 7f7b75410640 1 --2- 172.21.15.88:0/1846311144 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b7006f0a0 0x7f7b701f1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.250+0000 7ff743ef1640 1 Processor -- start 2024-09-16T01:34:31.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.250+0000 7ff743ef1640 1 -- start start 2024-09-16T01:34:31.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7f0043fff640 1 --2- 172.21.15.88:0/2986940918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f004c0b36a0 0x7f004c143bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.249+0000 7f004ad9d640 1 --2- 172.21.15.88:0/2986940918 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f004c159000 0x7f004c149e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.250+0000 7f0043fff640 1 -- 172.21.15.88:0/2986940918 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f004c159000 msgr2=0x7f004c149e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.250+0000 7f0043fff640 1 --2- 172.21.15.88:0/2986940918 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f004c159000 0x7f004c149e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.250+0000 7f0043fff640 1 -- 172.21.15.88:0/2986940918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f004c072190 msgr2=0x7f004c142b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.250+0000 7f0043fff640 1 --2- 172.21.15.88:0/2986940918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f004c072190 0x7f004c142b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.250+0000 7f0043fff640 1 -- 172.21.15.88:0/2986940918 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f004c1446b0 con 0x7f004c0b36a0 2024-09-16T01:34:31.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.250+0000 7f0043fff640 1 --2- 172.21.15.88:0/2986940918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f004c0b36a0 0x7f004c143bf0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f003400e8d0 tx=0x7f003400ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.250+0000 7f0042ffd640 1 -- 172.21.15.88:0/2986940918 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0034024070 con 0x7f004c0b36a0 2024-09-16T01:34:31.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.250+0000 7f7b75410640 1 -- 172.21.15.88:0/1846311144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b700b3490 msgr2=0x7f7b701f7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.250+0000 7f7b75410640 1 --2- 172.21.15.88:0/1846311144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b700b3490 0x7f7b701f7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.250+0000 7f7b75410640 1 -- 172.21.15.88:0/1846311144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b700734b0 msgr2=0x7f7b70142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.250+0000 7f7b75410640 1 --2- 172.21.15.88:0/1846311144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b700734b0 0x7f7b70142920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.250+0000 7f7b75410640 1 -- 172.21.15.88:0/1846311144 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b70141880 con 0x7f7b7006f0a0 2024-09-16T01:34:31.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.250+0000 7f7b74c0f640 1 --2- 172.21.15.88:0/1846311144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b700734b0 0x7f7b70142920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:31.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.250+0000 7f0042ffd640 1 -- 172.21.15.88:0/2986940918 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0034017c60 con 0x7f004c0b36a0 2024-09-16T01:34:31.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.250+0000 7f0050e4c640 1 -- 172.21.15.88:0/2986940918 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f004c144990 con 0x7f004c0b36a0 2024-09-16T01:34:31.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.250+0000 7fb2ea59c640 1 --2- 172.21.15.88:0/1933599293 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb2cc06c2b0 0x7fb2cc06e770 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc004ea0 tx=0x7fb2dc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.250+0000 7f0050e4c640 1 -- 172.21.15.88:0/2986940918 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f004c144f50 con 0x7f004c0b36a0 2024-09-16T01:34:31.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.251+0000 7fcc833e0640 1 -- 172.21.15.88:0/1246597463 shutdown_connections 2024-09-16T01:34:31.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.251+0000 7f0042ffd640 1 -- 172.21.15.88:0/2986940918 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003400c930 con 0x7f004c0b36a0 2024-09-16T01:34:31.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.251+0000 7f7b75410640 1 --2- 172.21.15.88:0/1846311144 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b7006f0a0 0x7f7b701f1a90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7b700b4690 tx=0x7f7b600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.251+0000 7f580359e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57fc006470 0x7f57fc031190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.251+0000 7f580359e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57fc04fc60 0x7f57fc04bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.251+0000 7ff743ef1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff73c071720 0x7ff73c154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.251+0000 7f580359e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57fc053580 0x7f57fc04d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.251+0000 7f4c34ff9640 1 -- 172.21.15.88:0/2909427476 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4c2c01a070 con 0x7f4c38073f30 2024-09-16T01:34:31.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.251+0000 7f580359e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f57fc050750 con 0x7f57fc04fc60 2024-09-16T01:34:31.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.251+0000 7f580359e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f57fc0520e0 con 0x7f57fc006470 2024-09-16T01:34:31.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.251+0000 7f580359e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f57fc0508d0 con 0x7f57fc053580 2024-09-16T01:34:31.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.252+0000 7f5802d9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57fc053580 0x7f57fc04d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.252+0000 7f5802d9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57fc053580 0x7f57fc04d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:35542/0 (socket says 172.21.15.88:35542) 2024-09-16T01:34:31.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.252+0000 7f5802d9d640 1 -- 172.21.15.88:0/970093455 learned_addr learned my addr 172.21.15.88:0/970093455 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:31.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.251+0000 7ff743ef1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff73c072210 0x7ff73c152190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.252+0000 7f5801d9b640 1 --2- 172.21.15.88:0/970093455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57fc04fc60 0x7f57fc04bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.251+0000 7ff743ef1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff73c14f520 0x7ff73c1f7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.251+0000 7ff743ef1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff73c0b5e50 con 0x7ff73c072210 2024-09-16T01:34:31.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.251+0000 7ff743ef1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff73c0b4640 con 0x7ff73c14f520 2024-09-16T01:34:31.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.252+0000 7f580259c640 1 --2- 172.21.15.88:0/970093455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57fc006470 0x7f57fc031190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.251+0000 7ff743ef1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff73c073140 con 0x7ff73c071720 2024-09-16T01:34:31.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.252+0000 7f5801d9b640 1 -- 172.21.15.88:0/970093455 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57fc053580 msgr2=0x7f57fc04d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.252+0000 7f5801d9b640 1 --2- 172.21.15.88:0/970093455 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57fc053580 0x7f57fc04d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.252+0000 7f5801d9b640 1 -- 172.21.15.88:0/970093455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57fc006470 msgr2=0x7f57fc031190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.252+0000 7f5801d9b640 1 --2- 172.21.15.88:0/970093455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57fc006470 0x7f57fc031190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.252+0000 7ff741c8d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff73c071720 0x7ff73c154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.252+0000 7f5801d9b640 1 -- 172.21.15.88:0/970093455 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57fc031be0 con 0x7f57fc04fc60 2024-09-16T01:34:31.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.252+0000 7f4c34ff9640 1 --2- 172.21.15.88:0/2909427476 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c2406c2b0 0x7f4c2406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.252+0000 7f4c34ff9640 1 -- 172.21.15.88:0/2909427476 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4c2c08b710 con 0x7f4c38073f30 2024-09-16T01:34:31.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.252+0000 7f4c37fff640 1 --2- 172.21.15.88:0/2909427476 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c2406c2b0 0x7f4c2406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.251+0000 7fcc833e0640 1 -- 172.21.15.88:0/1246597463 wait complete. 2024-09-16T01:34:31.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.252+0000 7ff741c8d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff73c071720 0x7ff73c154740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:35536/0 (socket says 172.21.15.88:35536) 2024-09-16T01:34:31.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.253+0000 7f4c3e3c4640 1 -- 172.21.15.88:0/2909427476 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c38073350 con 0x7f4c38073f30 2024-09-16T01:34:31.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.253+0000 7f580259c640 1 --2- 172.21.15.88:0/970093455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57fc006470 0x7f57fc031190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:31.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.252+0000 7ff741c8d640 1 -- 172.21.15.88:0/2012847531 learned_addr learned my addr 172.21.15.88:0/2012847531 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:31.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.253+0000 7f0042ffd640 1 -- 172.21.15.88:0/2986940918 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0034027070 con 0x7f004c0b36a0 2024-09-16T01:34:31.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.253+0000 7f0050e4c640 1 -- 172.21.15.88:0/2986940918 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f004c0b55c0 con 0x7f004c0b36a0 2024-09-16T01:34:31.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.253+0000 7f4c37fff640 1 --2- 172.21.15.88:0/2909427476 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c2406c2b0 0x7f4c2406e770 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4c28001600 tx=0x7f4c28007910 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.252+0000 7ff74248e640 1 --2- 172.21.15.88:0/2012847531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff73c14f520 0x7ff73c1f7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.253+0000 7fcc833e0640 1 Processor -- start 2024-09-16T01:34:31.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.253+0000 7f5801d9b640 1 --2- 172.21.15.88:0/970093455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57fc04fc60 0x7f57fc04bd30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f57f800b750 tx=0x7f57f800bc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.253+0000 7fcc833e0640 1 -- start start 2024-09-16T01:34:31.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.252+0000 7ff741c8d640 1 -- 172.21.15.88:0/2012847531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff73c14f520 msgr2=0x7ff73c1f7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.252+0000 7ff741c8d640 1 --2- 172.21.15.88:0/2012847531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff73c14f520 0x7ff73c1f7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.252+0000 7ff741c8d640 1 -- 172.21.15.88:0/2012847531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff73c072210 msgr2=0x7ff73c152190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:34:31.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.253+0000 7ff74148c640 1 --2- 172.21.15.88:0/2012847531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff73c072210 0x7ff73c152190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.253+0000 7ff741c8d640 1 --2- 172.21.15.88:0/2012847531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff73c072210 0x7ff73c152190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.253+0000 7ff741c8d640 1 -- 172.21.15.88:0/2012847531 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff73c1526d0 con 0x7ff73c071720 2024-09-16T01:34:31.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.253+0000 7ff74148c640 1 --2- 172.21.15.88:0/2012847531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff73c072210 0x7ff73c152190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:31.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.253+0000 7f0042ffd640 1 --2- 172.21.15.88:0/2986940918 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f002c06c2e0 0x7f002c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.253+0000 7ff741c8d640 1 --2- 172.21.15.88:0/2012847531 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff73c071720 0x7ff73c154740 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff73400c8c0 tx=0x7ff73400cd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.253+0000 7f004a59c640 1 --2- 172.21.15.88:0/2986940918 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f002c06c2e0 0x7f002c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.253+0000 7f0042ffd640 1 -- 172.21.15.88:0/2986940918 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f00340935f0 con 0x7f004c0b36a0 2024-09-16T01:34:31.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.254+0000 7f004a59c640 1 --2- 172.21.15.88:0/2986940918 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f002c06c2e0 0x7f002c06e7a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f003c0099a0 tx=0x7f003c006f90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.254+0000 7fcc833e0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc7c0cfba0 0x7fcc7c15bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.254+0000 7fcc833e0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc7c165200 0x7fcc7c0aa6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.254+0000 7fcc833e0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc7c06da30 0x7fcc7c1594b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.254+0000 7fcc833e0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c163c80 con 0x7fcc7c165200 2024-09-16T01:34:31.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.254+0000 7fcc833e0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c1639a0 con 0x7fcc7c0cfba0 2024-09-16T01:34:31.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.254+0000 7fcc833e0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcc7c062860 con 0x7fcc7c06da30 2024-09-16T01:34:31.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.254+0000 7fcc81bdd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc7c165200 0x7fcc7c0aa6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.254+0000 7fcc81bdd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc7c165200 0x7fcc7c0aa6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55024/0 (socket says 172.21.15.88:55024) 2024-09-16T01:34:31.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.254+0000 7fcc81bdd640 1 -- 172.21.15.88:0/2981542166 learned_addr learned my addr 172.21.15.88:0/2981542166 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:31.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.254+0000 7fcc823de640 1 --2- 172.21.15.88:0/2981542166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc7c0cfba0 0x7fcc7c15bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.254+0000 7fcc82bdf640 1 --2- 172.21.15.88:0/2981542166 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc7c06da30 0x7fcc7c1594b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.255+0000 7fcc81bdd640 1 -- 172.21.15.88:0/2981542166 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc7c06da30 msgr2=0x7fcc7c1594b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.255+0000 7fcc81bdd640 1 --2- 172.21.15.88:0/2981542166 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc7c06da30 0x7fcc7c1594b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.255+0000 7fcc81bdd640 1 -- 172.21.15.88:0/2981542166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc7c0cfba0 msgr2=0x7fcc7c15bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.255+0000 7fcc81bdd640 1 --2- 172.21.15.88:0/2981542166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc7c0cfba0 0x7fcc7c15bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.255+0000 7fcc81bdd640 1 -- 172.21.15.88:0/2981542166 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc7c15c160 con 0x7fcc7c165200 2024-09-16T01:34:31.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.255+0000 7fcc81bdd640 1 --2- 172.21.15.88:0/2981542166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc7c165200 0x7fcc7c0aa6f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c00c500 tx=0x7fcc6c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.256+0000 7ff067fff640 1 -- 172.21.15.88:0/497460723 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff060037070 con 0x7ff05c005210 2024-09-16T01:34:31.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.256+0000 7f57f2ffd640 1 -- 172.21.15.88:0/970093455 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f8019070 con 0x7f57fc04fc60 2024-09-16T01:34:31.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.256+0000 7f7b65ffb640 1 -- 172.21.15.88:0/1846311144 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b60041070 con 0x7f7b7006f0a0 2024-09-16T01:34:31.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.256+0000 7fb2daffd640 1 -- 172.21.15.88:0/1933599293 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb2ec073a50 con 0x7fb2ec0b4490 2024-09-16T01:34:31.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.256+0000 7f7b77674640 1 -- 172.21.15.88:0/1846311144 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b70141b60 con 0x7f7b7006f0a0 2024-09-16T01:34:31.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.256+0000 7f580359e640 1 -- 172.21.15.88:0/970093455 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57fc0f0dc0 con 0x7f57fc04fc60 2024-09-16T01:34:31.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.256+0000 7fcc72ffd640 1 -- 172.21.15.88:0/2981542166 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c017070 con 0x7fcc7c165200 2024-09-16T01:34:31.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.256+0000 7f580359e640 1 -- 172.21.15.88:0/970093455 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57fc0f1380 con 0x7f57fc04fc60 2024-09-16T01:34:31.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.256+0000 7fcc833e0640 1 -- 172.21.15.88:0/2981542166 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc7c15a3b0 con 0x7fcc7c165200 2024-09-16T01:34:31.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.256+0000 7f57f2ffd640 1 -- 172.21.15.88:0/970093455 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57f800c720 con 0x7f57fc04fc60 2024-09-16T01:34:31.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.257+0000 7f7b77674640 1 -- 172.21.15.88:0/1846311144 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b70142120 con 0x7f7b7006f0a0 2024-09-16T01:34:31.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.256+0000 7fcc833e0640 1 -- 172.21.15.88:0/2981542166 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc7c15ae60 con 0x7fcc7c165200 2024-09-16T01:34:31.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.256+0000 7fcc72ffd640 1 -- 172.21.15.88:0/2981542166 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc6c005ce0 con 0x7fcc7c165200 2024-09-16T01:34:31.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.257+0000 7ff067fff640 1 -- 172.21.15.88:0/497460723 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff06002ab50 con 0x7ff05c005210 2024-09-16T01:34:31.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.258+0000 7ff07768b640 1 -- 172.21.15.88:0/497460723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff05c005210 msgr2=0x7ff05c0141d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.258+0000 7ff07768b640 1 --2- 172.21.15.88:0/497460723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff05c005210 0x7ff05c0141d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff0700ba7b0 tx=0x7ff05c005830 comp rx=0 tx=0).stop 2024-09-16T01:34:31.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.257+0000 7f7b65ffb640 1 -- 172.21.15.88:0/1846311144 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b60032440 con 0x7f7b7006f0a0 2024-09-16T01:34:31.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.257+0000 7f7b65ffb640 1 -- 172.21.15.88:0/1846311144 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b6004b060 con 0x7f7b7006f0a0 2024-09-16T01:34:31.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.258+0000 7f7b77674640 1 -- 172.21.15.88:0/1846311144 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b50005290 con 0x7f7b7006f0a0 2024-09-16T01:34:31.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.257+0000 7f57f2ffd640 1 -- 172.21.15.88:0/970093455 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f800caf0 con 0x7f57fc04fc60 2024-09-16T01:34:31.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.258+0000 7fcc72ffd640 1 -- 172.21.15.88:0/2981542166 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c021060 con 0x7fcc7c165200 2024-09-16T01:34:31.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.258+0000 7fcc833e0640 1 -- 172.21.15.88:0/2981542166 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc64005290 con 0x7fcc7c165200 2024-09-16T01:34:31.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.258+0000 7ff075c28640 1 -- 172.21.15.88:0/497460723 reap_dead start 2024-09-16T01:34:31.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.258+0000 7ff07768b640 1 -- 172.21.15.88:0/497460723 shutdown_connections 2024-09-16T01:34:31.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.258+0000 7ff07768b640 1 -- 172.21.15.88:0/497460723 >> 172.21.15.88:0/497460723 conn(0x7ff07006c500 msgr2=0x7ff07006c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:31.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.258+0000 7ff7327fc640 1 -- 172.21.15.88:0/2012847531 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff734018070 con 0x7ff73c071720 2024-09-16T01:34:31.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.258+0000 7ff7327fc640 1 -- 172.21.15.88:0/2012847531 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff73400a640 con 0x7ff73c071720 2024-09-16T01:34:31.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.259+0000 7f4c34ff9640 1 -- 172.21.15.88:0/2909427476 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4c2c059700 con 0x7f4c38073f30 2024-09-16T01:34:31.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.259+0000 7f580359e640 1 -- 172.21.15.88:0/970093455 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57e8005290 con 0x7f57fc04fc60 2024-09-16T01:34:31.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.258+0000 7ff7327fc640 1 -- 172.21.15.88:0/2012847531 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff734018070 con 0x7ff73c071720 2024-09-16T01:34:31.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.259+0000 7ff743ef1640 1 -- 172.21.15.88:0/2012847531 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff73c152970 con 0x7ff73c071720 2024-09-16T01:34:31.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.259+0000 7ff743ef1640 1 -- 172.21.15.88:0/2012847531 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff73c0ce060 con 0x7ff73c071720 2024-09-16T01:34:31.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.259+0000 7f0042ffd640 1 -- 172.21.15.88:0/2986940918 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0034061690 con 0x7f004c0b36a0 2024-09-16T01:34:31.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.259+0000 7fcc72ffd640 1 -- 172.21.15.88:0/2981542166 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fcc6c01a070 con 0x7fcc7c165200 2024-09-16T01:34:31.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.258+0000 7ff07768b640 1 -- 172.21.15.88:0/497460723 shutdown_connections 2024-09-16T01:34:31.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.260+0000 7fcc72ffd640 1 --2- 172.21.15.88:0/2981542166 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcc6006c4b0 0x7fcc6006e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.260+0000 7fcc72ffd640 1 -- 172.21.15.88:0/2981542166 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fcc6c0070f0 con 0x7fcc7c165200 2024-09-16T01:34:31.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.260+0000 7fcc823de640 1 --2- 172.21.15.88:0/2981542166 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcc6006c4b0 0x7fcc6006e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.261+0000 7ff7327fc640 1 -- 172.21.15.88:0/2012847531 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff73400a830 con 0x7ff73c071720 2024-09-16T01:34:31.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.261+0000 7ff743ef1640 1 -- 172.21.15.88:0/2012847531 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff73c0d02e0 con 0x7ff73c071720 2024-09-16T01:34:31.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.261+0000 7fcc823de640 1 --2- 172.21.15.88:0/2981542166 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcc6006c4b0 0x7fcc6006e970 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcc78033000 tx=0x7fcc78008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.261+0000 7f7b65ffb640 1 -- 172.21.15.88:0/1846311144 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7b60044070 con 0x7f7b7006f0a0 2024-09-16T01:34:31.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.258+0000 7ff07768b640 1 -- 172.21.15.88:0/497460723 wait complete. 2024-09-16T01:34:31.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.262+0000 7f7b65ffb640 1 --2- 172.21.15.88:0/1846311144 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7b5406c2b0 0x7f7b5406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.262+0000 7ff7327fc640 1 --2- 172.21.15.88:0/2012847531 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff72006c4b0 0x7ff72006e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.260+0000 7ff07768b640 1 Processor -- start 2024-09-16T01:34:31.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.262+0000 7ff7327fc640 1 -- 172.21.15.88:0/2012847531 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff73408a590 con 0x7ff73c071720 2024-09-16T01:34:31.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.262+0000 7ff74148c640 1 --2- 172.21.15.88:0/2012847531 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff72006c4b0 0x7ff72006e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.260+0000 7ff07768b640 1 -- start start 2024-09-16T01:34:31.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.262+0000 7f7b74c0f640 1 --2- 172.21.15.88:0/1846311144 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7b5406c2b0 0x7f7b5406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.261+0000 7ff07768b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0700cabb0 0x7ff070145590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.261+0000 7ff07768b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0700caf90 0x7ff070145ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.262+0000 7ff07768b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff07015c010 0x7ff070143560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.262+0000 7ff07768b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff0700d43e0 con 0x7ff07015c010 2024-09-16T01:34:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.262+0000 7ff07768b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff0700d4260 con 0x7ff0700cabb0 2024-09-16T01:34:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.262+0000 7ff07768b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff0700d4560 con 0x7ff0700caf90 2024-09-16T01:34:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.262+0000 7ff074c26640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0700caf90 0x7ff070145ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.262+0000 7ff74148c640 1 --2- 172.21.15.88:0/2012847531 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff72006c4b0 0x7ff72006e970 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff72c034000 tx=0x7ff72c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.262+0000 7ff074c26640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0700caf90 0x7ff070145ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:35556/0 (socket says 172.21.15.88:35556) 2024-09-16T01:34:31.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.263+0000 7f7b74c0f640 1 --2- 172.21.15.88:0/1846311144 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7b5406c2b0 0x7f7b5406e770 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c007fd0 tx=0x7f7b6c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.263+0000 7f7b65ffb640 1 -- 172.21.15.88:0/1846311144 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7b60078520 con 0x7f7b7006f0a0 2024-09-16T01:34:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.262+0000 7ff074c26640 1 -- 172.21.15.88:0/651363286 learned_addr learned my addr 172.21.15.88:0/651363286 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.262+0000 7ff075c28640 1 --2- 172.21.15.88:0/651363286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff07015c010 0x7ff070143560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.262+0000 7ff075427640 1 --2- 172.21.15.88:0/651363286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0700cabb0 0x7ff070145590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.262+0000 7ff074c26640 1 -- 172.21.15.88:0/651363286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0700cabb0 msgr2=0x7ff070145590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.262+0000 7ff074c26640 1 --2- 172.21.15.88:0/651363286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0700cabb0 0x7ff070145590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.262+0000 7ff074c26640 1 -- 172.21.15.88:0/651363286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff07015c010 msgr2=0x7ff070143560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.262+0000 7ff074c26640 1 --2- 172.21.15.88:0/651363286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff07015c010 0x7ff070143560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.262+0000 7ff074c26640 1 -- 172.21.15.88:0/651363286 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff070157c80 con 0x7ff0700caf90 2024-09-16T01:34:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.263+0000 7ff074c26640 1 --2- 172.21.15.88:0/651363286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0700caf90 0x7ff070145ad0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff06c00e7f0 tx=0x7ff06c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.264+0000 7f57f2ffd640 1 -- 172.21.15.88:0/970093455 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f57f801c070 con 0x7f57fc04fc60 2024-09-16T01:34:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.264+0000 7f57f2ffd640 1 --2- 172.21.15.88:0/970093455 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f57e406c2b0 0x7f57e406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.264+0000 7f580259c640 1 --2- 172.21.15.88:0/970093455 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f57e406c2b0 0x7f57e406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.264+0000 7f57f2ffd640 1 -- 172.21.15.88:0/970093455 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f57f808ac60 con 0x7f57fc04fc60 2024-09-16T01:34:31.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.264+0000 7f580259c640 1 --2- 172.21.15.88:0/970093455 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f57e406c2b0 0x7f57e406e770 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f57f4004980 tx=0x7f57f4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.266+0000 7fcc72ffd640 1 -- 172.21.15.88:0/2981542166 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fcc7c15c160 con 0x7fcc7c165200 2024-09-16T01:34:31.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.267+0000 7ff065ffb640 1 -- 172.21.15.88:0/651363286 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff06c019020 con 0x7ff0700caf90 2024-09-16T01:34:31.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.267+0000 7ff065ffb640 1 -- 172.21.15.88:0/651363286 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff06c00a6b0 con 0x7ff0700caf90 2024-09-16T01:34:31.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.267+0000 7ff7327fc640 1 -- 172.21.15.88:0/2012847531 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff7340586c0 con 0x7ff73c071720 2024-09-16T01:34:31.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.267+0000 7ff07768b640 1 -- 172.21.15.88:0/651363286 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0701579a0 con 0x7ff0700caf90 2024-09-16T01:34:31.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.268+0000 7ff07768b640 1 -- 172.21.15.88:0/651363286 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff07015b810 con 0x7ff0700caf90 2024-09-16T01:34:31.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.269+0000 7ff0537fe640 1 -- 172.21.15.88:0/651363286 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff04c005290 con 0x7ff0700caf90 2024-09-16T01:34:31.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.269+0000 7ff065ffb640 1 -- 172.21.15.88:0/651363286 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff06c00aa60 con 0x7ff0700caf90 2024-09-16T01:34:31.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.270+0000 7ff065ffb640 1 -- 172.21.15.88:0/651363286 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff06c01c020 con 0x7ff0700caf90 2024-09-16T01:34:31.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.271+0000 7ff065ffb640 1 --2- 172.21.15.88:0/651363286 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff05406c2b0 0x7ff05406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:31.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.271+0000 7ff065ffb640 1 -- 172.21.15.88:0/651363286 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff06c08bc30 con 0x7ff0700caf90 2024-09-16T01:34:31.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.271+0000 7ff075427640 1 --2- 172.21.15.88:0/651363286 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff05406c2b0 0x7ff05406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:31.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.272+0000 7ff075427640 1 --2- 172.21.15.88:0/651363286 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff05406c2b0 0x7ff05406e770 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff068004bc0 tx=0x7ff0680017b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:31.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.273+0000 7f7b65ffb640 1 -- 172.21.15.88:0/1846311144 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7b6007ab80 con 0x7f7b7006f0a0 2024-09-16T01:34:31.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.275+0000 7ff065ffb640 1 -- 172.21.15.88:0/651363286 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff06c092080 con 0x7ff0700caf90 2024-09-16T01:34:31.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.279+0000 7f57f2ffd640 1 -- 172.21.15.88:0/970093455 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f57f8023020 con 0x7f57fc04fc60 2024-09-16T01:34:31.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.654+0000 7f4c3e3c4640 1 -- 172.21.15.88:0/2909427476 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f4c380b2c90 con 0x7f4c38073f30 2024-09-16T01:34:31.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.655+0000 7f4c34ff9640 1 -- 172.21.15.88:0/2909427476 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4c2c05da00 con 0x7f4c38073f30 2024-09-16T01:34:31.656 INFO:teuthology.orchestra.run.smithi088.stdout:60129542149 2024-09-16T01:34:31.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.656+0000 7fb2eb59e640 1 -- 172.21.15.88:0/1933599293 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fb2ec06f240 con 0x7fb2ec0b4490 2024-09-16T01:34:31.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.657+0000 7fb2daffd640 1 -- 172.21.15.88:0/1933599293 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb2e404b070 con 0x7fb2ec0b4490 2024-09-16T01:34:31.658 INFO:teuthology.orchestra.run.smithi088.stdout:60129542150 2024-09-16T01:34:31.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.658+0000 7f4c1e7fc640 1 -- 172.21.15.88:0/2909427476 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c2406c2b0 msgr2=0x7f4c2406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.658+0000 7f4c1e7fc640 1 --2- 172.21.15.88:0/2909427476 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c2406c2b0 0x7f4c2406e770 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4c28001600 tx=0x7f4c28007910 comp rx=0 tx=0).stop 2024-09-16T01:34:31.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.659+0000 7f4c1e7fc640 1 -- 172.21.15.88:0/2909427476 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c38073f30 msgr2=0x7f4c3814b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.659+0000 7f4c1e7fc640 1 --2- 172.21.15.88:0/2909427476 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c38073f30 0x7f4c3814b230 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c00e870 tx=0x7f4c2c00ed30 comp rx=0 tx=0).stop 2024-09-16T01:34:31.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.659+0000 7f4c1e7fc640 1 -- 172.21.15.88:0/2909427476 shutdown_connections 2024-09-16T01:34:31.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.659+0000 7f4c1e7fc640 1 --2- 172.21.15.88:0/2909427476 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c2406c2b0 0x7f4c2406e770 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.659+0000 7f4c1e7fc640 1 --2- 172.21.15.88:0/2909427476 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c38162760 0x7f4c38155440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.659+0000 7f4c1e7fc640 1 --2- 172.21.15.88:0/2909427476 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c38073f30 0x7f4c3814b230 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.659+0000 7f4c1e7fc640 1 --2- 172.21.15.88:0/2909427476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c3806c320 0x7f4c380cfe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.659+0000 7f4c1e7fc640 1 -- 172.21.15.88:0/2909427476 >> 172.21.15.88:0/2909427476 conn(0x7f4c3806e0f0 msgr2=0x7f4c380b55f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:31.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.660+0000 7f4c1e7fc640 1 -- 172.21.15.88:0/2909427476 shutdown_connections 2024-09-16T01:34:31.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.660+0000 7f4c1e7fc640 1 -- 172.21.15.88:0/2909427476 wait complete. 2024-09-16T01:34:31.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.660+0000 7fb2d8ff9640 1 -- 172.21.15.88:0/1933599293 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb2cc06c2b0 msgr2=0x7fb2cc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.660+0000 7fb2d8ff9640 1 --2- 172.21.15.88:0/1933599293 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb2cc06c2b0 0x7fb2cc06e770 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc004ea0 tx=0x7fb2dc009040 comp rx=0 tx=0).stop 2024-09-16T01:34:31.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.660+0000 7fb2d8ff9640 1 -- 172.21.15.88:0/1933599293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2ec0b4490 msgr2=0x7fb2ec15f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.660+0000 7fb2d8ff9640 1 --2- 172.21.15.88:0/1933599293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2ec0b4490 0x7fb2ec15f660 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb2e402ad20 tx=0x7fb2e4004a40 comp rx=0 tx=0).stop 2024-09-16T01:34:31.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.661+0000 7fcc833e0640 1 -- 172.21.15.88:0/2981542166 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fcc64005580 con 0x7fcc7c165200 2024-09-16T01:34:31.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.661+0000 7ff743ef1640 1 -- 172.21.15.88:0/2012847531 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7ff73c1f7c90 con 0x7ff73c071720 2024-09-16T01:34:31.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.661+0000 7fb2d8ff9640 1 -- 172.21.15.88:0/1933599293 shutdown_connections 2024-09-16T01:34:31.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.661+0000 7fb2d8ff9640 1 --2- 172.21.15.88:0/1933599293 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb2cc06c2b0 0x7fb2cc06e770 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.661+0000 7fb2d8ff9640 1 --2- 172.21.15.88:0/1933599293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb2ec0d0210 0x7fb2ec161db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.661+0000 7fb2d8ff9640 1 --2- 172.21.15.88:0/1933599293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2ec0b4490 0x7fb2ec15f660 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.661+0000 7fb2d8ff9640 1 --2- 172.21.15.88:0/1933599293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2ec06e350 0x7fb2ec0aa480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.661+0000 7fb2d8ff9640 1 -- 172.21.15.88:0/1933599293 >> 172.21.15.88:0/1933599293 conn(0x7fb2ec06d0d0 msgr2=0x7fb2ec0ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:31.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.661+0000 7fb2d8ff9640 1 -- 172.21.15.88:0/1933599293 shutdown_connections 2024-09-16T01:34:31.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.661+0000 7fb2d8ff9640 1 -- 172.21.15.88:0/1933599293 wait complete. 2024-09-16T01:34:31.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.661+0000 7ff7327fc640 1 -- 172.21.15.88:0/2012847531 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff73405c9c0 con 0x7ff73c071720 2024-09-16T01:34:31.665 INFO:teuthology.orchestra.run.smithi088.stdout:60129542149 2024-09-16T01:34:31.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.662+0000 7f0050e4c640 1 -- 172.21.15.88:0/2986940918 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f004c14ee10 con 0x7f004c0b36a0 2024-09-16T01:34:31.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.662+0000 7fcc72ffd640 1 -- 172.21.15.88:0/2981542166 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcc7c15c160 con 0x7fcc7c165200 2024-09-16T01:34:31.666 INFO:teuthology.orchestra.run.smithi088.stdout:60129542150 2024-09-16T01:34:31.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.664+0000 7ff743ef1640 1 -- 172.21.15.88:0/2012847531 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff72006c4b0 msgr2=0x7ff72006e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.664+0000 7ff743ef1640 1 --2- 172.21.15.88:0/2012847531 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff72006c4b0 0x7ff72006e970 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff72c034000 tx=0x7ff72c02e000 comp rx=0 tx=0).stop 2024-09-16T01:34:31.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.664+0000 7ff743ef1640 1 -- 172.21.15.88:0/2012847531 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff73c071720 msgr2=0x7ff73c154740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.664+0000 7ff743ef1640 1 --2- 172.21.15.88:0/2012847531 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff73c071720 0x7ff73c154740 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff73400c8c0 tx=0x7ff73400cd80 comp rx=0 tx=0).stop 2024-09-16T01:34:31.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.664+0000 7ff743ef1640 1 -- 172.21.15.88:0/2012847531 shutdown_connections 2024-09-16T01:34:31.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.664+0000 7ff743ef1640 1 --2- 172.21.15.88:0/2012847531 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff72006c4b0 0x7ff72006e970 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.664+0000 7ff743ef1640 1 --2- 172.21.15.88:0/2012847531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff73c14f520 0x7ff73c1f7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.664+0000 7ff743ef1640 1 --2- 172.21.15.88:0/2012847531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff73c072210 0x7ff73c152190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.664+0000 7ff743ef1640 1 --2- 172.21.15.88:0/2012847531 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff73c071720 0x7ff73c154740 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.664+0000 7ff743ef1640 1 -- 172.21.15.88:0/2012847531 >> 172.21.15.88:0/2012847531 conn(0x7ff73c0b0aa0 msgr2=0x7ff73c15fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:31.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.664+0000 7ff743ef1640 1 -- 172.21.15.88:0/2012847531 shutdown_connections 2024-09-16T01:34:31.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.664+0000 7ff743ef1640 1 -- 172.21.15.88:0/2012847531 wait complete. 2024-09-16T01:34:31.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.664+0000 7f0042ffd640 1 -- 172.21.15.88:0/2986940918 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0034065990 con 0x7f004c0b36a0 2024-09-16T01:34:31.668 INFO:teuthology.orchestra.run.smithi088.stdout:60129542150 2024-09-16T01:34:31.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.666+0000 7fcc70ff9640 1 -- 172.21.15.88:0/2981542166 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcc6006c4b0 msgr2=0x7fcc6006e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.666+0000 7fcc70ff9640 1 --2- 172.21.15.88:0/2981542166 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcc6006c4b0 0x7fcc6006e970 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcc78033000 tx=0x7fcc78008030 comp rx=0 tx=0).stop 2024-09-16T01:34:31.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.666+0000 7fcc70ff9640 1 -- 172.21.15.88:0/2981542166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc7c165200 msgr2=0x7fcc7c0aa6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.666+0000 7fcc70ff9640 1 --2- 172.21.15.88:0/2981542166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc7c165200 0x7fcc7c0aa6f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c00c500 tx=0x7fcc6c00c9c0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.666+0000 7fcc70ff9640 1 -- 172.21.15.88:0/2981542166 shutdown_connections 2024-09-16T01:34:31.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.666+0000 7fcc70ff9640 1 --2- 172.21.15.88:0/2981542166 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcc6006c4b0 0x7fcc6006e970 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.666+0000 7fcc70ff9640 1 --2- 172.21.15.88:0/2981542166 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc7c06da30 0x7fcc7c1594b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.667+0000 7fcc70ff9640 1 --2- 172.21.15.88:0/2981542166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc7c165200 0x7fcc7c0aa6f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.667+0000 7fcc70ff9640 1 --2- 172.21.15.88:0/2981542166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc7c0cfba0 0x7fcc7c15bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.667+0000 7fcc70ff9640 1 -- 172.21.15.88:0/2981542166 >> 172.21.15.88:0/2981542166 conn(0x7fcc7c0b1640 msgr2=0x7fcc7c1656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:31.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.667+0000 7fcc70ff9640 1 -- 172.21.15.88:0/2981542166 shutdown_connections 2024-09-16T01:34:31.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.667+0000 7fcc70ff9640 1 -- 172.21.15.88:0/2981542166 wait complete. 2024-09-16T01:34:31.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.668+0000 7f0050e4c640 1 -- 172.21.15.88:0/2986940918 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f002c06c2e0 msgr2=0x7f002c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.668+0000 7f0050e4c640 1 --2- 172.21.15.88:0/2986940918 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f002c06c2e0 0x7f002c06e7a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f003c0099a0 tx=0x7f003c006f90 comp rx=0 tx=0).stop 2024-09-16T01:34:31.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.668+0000 7f0050e4c640 1 -- 172.21.15.88:0/2986940918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f004c0b36a0 msgr2=0x7f004c143bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.668+0000 7f0050e4c640 1 --2- 172.21.15.88:0/2986940918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f004c0b36a0 0x7f004c143bf0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f003400e8d0 tx=0x7f003400ed90 comp rx=0 tx=0).stop 2024-09-16T01:34:31.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.668+0000 7f0050e4c640 1 -- 172.21.15.88:0/2986940918 shutdown_connections 2024-09-16T01:34:31.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.668+0000 7f0050e4c640 1 --2- 172.21.15.88:0/2986940918 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f002c06c2e0 0x7f002c06e7a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.668+0000 7f0050e4c640 1 --2- 172.21.15.88:0/2986940918 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f004c159000 0x7f004c149e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.668+0000 7f0050e4c640 1 --2- 172.21.15.88:0/2986940918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f004c0b36a0 0x7f004c143bf0 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f003400e8d0 tx=0x7f003400ed90 comp rx=0 tx=0).stop 2024-09-16T01:34:31.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.668+0000 7f0050e4c640 1 --2- 172.21.15.88:0/2986940918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f004c072190 0x7f004c142b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.668+0000 7f0050e4c640 1 -- 172.21.15.88:0/2986940918 >> 172.21.15.88:0/2986940918 conn(0x7f004c0b6b50 msgr2=0x7f004c14e080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:31.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.668+0000 7f0050e4c640 1 -- 172.21.15.88:0/2986940918 shutdown_connections 2024-09-16T01:34:31.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.668+0000 7f0050e4c640 1 -- 172.21.15.88:0/2986940918 wait complete. 2024-09-16T01:34:31.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.670+0000 7f7b77674640 1 -- 172.21.15.88:0/1846311144 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f7b50005850 con 0x7f7b7006f0a0 2024-09-16T01:34:31.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.671+0000 7f7b65ffb640 1 -- 172.21.15.88:0/1846311144 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7b6007a480 con 0x7f7b7006f0a0 2024-09-16T01:34:31.676 INFO:teuthology.orchestra.run.smithi088.stdout:64424509447 2024-09-16T01:34:31.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.675+0000 7f7b77674640 1 -- 172.21.15.88:0/1846311144 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7b5406c2b0 msgr2=0x7f7b5406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.675+0000 7f7b77674640 1 --2- 172.21.15.88:0/1846311144 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7b5406c2b0 0x7f7b5406e770 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c007fd0 tx=0x7f7b6c009040 comp rx=0 tx=0).stop 2024-09-16T01:34:31.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.676+0000 7f7b77674640 1 -- 172.21.15.88:0/1846311144 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b7006f0a0 msgr2=0x7f7b701f1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.676+0000 7f7b77674640 1 --2- 172.21.15.88:0/1846311144 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b7006f0a0 0x7f7b701f1a90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7b700b4690 tx=0x7f7b600046c0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.676+0000 7ff0537fe640 1 -- 172.21.15.88:0/651363286 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7ff04c005850 con 0x7ff0700caf90 2024-09-16T01:34:31.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.676+0000 7f7b77674640 1 -- 172.21.15.88:0/1846311144 shutdown_connections 2024-09-16T01:34:31.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.676+0000 7f7b77674640 1 --2- 172.21.15.88:0/1846311144 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7b5406c2b0 0x7f7b5406e770 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.676+0000 7f7b77674640 1 --2- 172.21.15.88:0/1846311144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b700b3490 0x7f7b701f7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.676+0000 7f7b77674640 1 --2- 172.21.15.88:0/1846311144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b700734b0 0x7f7b70142920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.676+0000 7f7b77674640 1 --2- 172.21.15.88:0/1846311144 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b7006f0a0 0x7f7b701f1a90 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.676+0000 7f7b77674640 1 -- 172.21.15.88:0/1846311144 >> 172.21.15.88:0/1846311144 conn(0x7f7b7006d0f0 msgr2=0x7f7b7015fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:31.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.676+0000 7f7b77674640 1 -- 172.21.15.88:0/1846311144 shutdown_connections 2024-09-16T01:34:31.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.676+0000 7ff065ffb640 1 -- 172.21.15.88:0/651363286 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff06c023020 con 0x7ff0700caf90 2024-09-16T01:34:31.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.677+0000 7f7b77674640 1 -- 172.21.15.88:0/1846311144 wait complete. 2024-09-16T01:34:31.679 INFO:teuthology.orchestra.run.smithi088.stdout:60129542149 2024-09-16T01:34:31.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.677+0000 7f580359e640 1 -- 172.21.15.88:0/970093455 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f57e8005850 con 0x7f57fc04fc60 2024-09-16T01:34:31.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.678+0000 7f57f2ffd640 1 -- 172.21.15.88:0/970093455 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f57f8058d70 con 0x7f57fc04fc60 2024-09-16T01:34:31.681 INFO:teuthology.orchestra.run.smithi088.stdout:60129542150 2024-09-16T01:34:31.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.681+0000 7ff0537fe640 1 -- 172.21.15.88:0/651363286 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff05406c2b0 msgr2=0x7ff05406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.681+0000 7ff0537fe640 1 --2- 172.21.15.88:0/651363286 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff05406c2b0 0x7ff05406e770 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff068004bc0 tx=0x7ff0680017b0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.681+0000 7ff0537fe640 1 -- 172.21.15.88:0/651363286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0700caf90 msgr2=0x7ff070145ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.681+0000 7ff0537fe640 1 --2- 172.21.15.88:0/651363286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0700caf90 0x7ff070145ad0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff06c00e7f0 tx=0x7ff06c00ecb0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.681+0000 7f580359e640 1 -- 172.21.15.88:0/970093455 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f57e406c2b0 msgr2=0x7f57e406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.681+0000 7f580359e640 1 --2- 172.21.15.88:0/970093455 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f57e406c2b0 0x7f57e406e770 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f57f4004980 tx=0x7f57f4008040 comp rx=0 tx=0).stop 2024-09-16T01:34:31.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.681+0000 7f580359e640 1 -- 172.21.15.88:0/970093455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57fc04fc60 msgr2=0x7f57fc04bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:31.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.681+0000 7f580359e640 1 --2- 172.21.15.88:0/970093455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57fc04fc60 0x7f57fc04bd30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f57f800b750 tx=0x7f57f800bc10 comp rx=0 tx=0).stop 2024-09-16T01:34:31.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.682+0000 7f580359e640 1 -- 172.21.15.88:0/970093455 shutdown_connections 2024-09-16T01:34:31.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.682+0000 7f580359e640 1 --2- 172.21.15.88:0/970093455 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f57e406c2b0 0x7f57e406e770 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.682+0000 7f580359e640 1 --2- 172.21.15.88:0/970093455 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57fc053580 0x7f57fc04d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.682+0000 7f580359e640 1 --2- 172.21.15.88:0/970093455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57fc04fc60 0x7f57fc04bd30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.682+0000 7f580359e640 1 --2- 172.21.15.88:0/970093455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57fc006470 0x7f57fc031190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.682+0000 7f580359e640 1 -- 172.21.15.88:0/970093455 >> 172.21.15.88:0/970093455 conn(0x7f57fc036090 msgr2=0x7f57fc0dd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:31.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.682+0000 7ff0537fe640 1 -- 172.21.15.88:0/651363286 shutdown_connections 2024-09-16T01:34:31.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.682+0000 7ff0537fe640 1 --2- 172.21.15.88:0/651363286 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff05406c2b0 0x7ff05406e770 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff068004bc0 tx=0x7ff0680017b0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.682+0000 7ff0537fe640 1 --2- 172.21.15.88:0/651363286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff07015c010 0x7ff070143560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.682+0000 7ff0537fe640 1 --2- 172.21.15.88:0/651363286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0700caf90 0x7ff070145ad0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.682+0000 7ff0537fe640 1 --2- 172.21.15.88:0/651363286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0700cabb0 0x7ff070145590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:31.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.682+0000 7ff0537fe640 1 -- 172.21.15.88:0/651363286 >> 172.21.15.88:0/651363286 conn(0x7ff07006c500 msgr2=0x7ff0700b7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:31.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.682+0000 7f580359e640 1 -- 172.21.15.88:0/970093455 shutdown_connections 2024-09-16T01:34:31.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.682+0000 7f580359e640 1 -- 172.21.15.88:0/970093455 wait complete. 2024-09-16T01:34:31.686 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.0 2024-09-16T01:34:31.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.683+0000 7ff0537fe640 1 -- 172.21.15.88:0/651363286 shutdown_connections 2024-09-16T01:34:31.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:31.683+0000 7ff0537fe640 1 -- 172.21.15.88:0/651363286 wait complete. 2024-09-16T01:34:31.687 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.4 2024-09-16T01:34:31.687 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.1 2024-09-16T01:34:31.690 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.6 2024-09-16T01:34:31.692 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.7 2024-09-16T01:34:31.695 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509447 for osd.5 2024-09-16T01:34:31.698 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.3 2024-09-16T01:34:31.699 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.2 2024-09-16T01:34:32.687 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T01:34:32.688 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T01:34:32.688 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T01:34:32.691 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T01:34:32.692 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T01:34:32.696 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T01:34:32.699 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T01:34:32.699 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T01:34:32.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.831+0000 7ff8e3ceb640 1 Processor -- start 2024-09-16T01:34:32.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.832+0000 7ff8e3ceb640 1 -- start start 2024-09-16T01:34:32.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.832+0000 7f8a52d5f640 1 Processor -- start 2024-09-16T01:34:32.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.833+0000 7ff8e3ceb640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8dc071720 0x7ff8dc15a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.833+0000 7ff8e3ceb640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff8dc062630 0x7ff8dc0b5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.833+0000 7ff8e3ceb640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8dc0ce9a0 con 0x7ff8dc0ce390 2024-09-16T01:34:32.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.833+0000 7ff8e3ceb640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff8dc0ce770 con 0x7ff8dc062630 2024-09-16T01:34:32.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.832+0000 7f8a52d5f640 1 -- start start 2024-09-16T01:34:32.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.833+0000 7f8a52d5f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a4c073160 0x7f8a4c0b6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.833+0000 7f8a52d5f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8a4c0b74a0 0x7f8a4c0b5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.834+0000 7f8a52d5f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a4c0d0b20 con 0x7f8a4c15c770 2024-09-16T01:34:32.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.834+0000 7f8a52d5f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f8a4c0d2310 con 0x7f8a4c0b74a0 2024-09-16T01:34:32.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.834+0000 7f8a50afb640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8a4c15c770 0x7f8a4c0d0320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45210/0 (socket says 172.21.15.88:45210) 2024-09-16T01:34:32.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.834+0000 7f8a50afb640 1 -- 172.21.15.88:0/592151391 learned_addr learned my addr 172.21.15.88:0/592151391 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:32.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.834+0000 7f8a52d5f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8a4c0d0e00 con 0x7f8a4c073160 2024-09-16T01:34:32.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.834+0000 7f8a512fc640 1 --2- 172.21.15.88:0/592151391 >> v2:172.21.15.154:3300/0 conn(0x7f8a4c0b74a0 0x7f8a4c0b5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.834+0000 7f8a4bfff640 1 --2- 172.21.15.88:0/592151391 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a4c073160 0x7f8a4c0b6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.835+0000 7f8a512fc640 1 -- 172.21.15.88:0/592151391 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a4c073160 msgr2=0x7f8a4c0b6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:32.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.835+0000 7f8a512fc640 1 --2- 172.21.15.88:0/592151391 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a4c073160 0x7f8a4c0b6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.835+0000 7f8a512fc640 1 -- 172.21.15.88:0/592151391 >> v1:172.21.15.88:6789/0 conn(0x7f8a4c15c770 legacy=0x7f8a4c0d0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:32.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.835+0000 7f8a512fc640 1 -- 172.21.15.88:0/592151391 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a4c0d10e0 con 0x7f8a4c0b74a0 2024-09-16T01:34:32.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.836+0000 7f3a2405a640 1 Processor -- start 2024-09-16T01:34:32.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.833+0000 7ff8e3ceb640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff8dc0cf5b0 con 0x7ff8dc071720 2024-09-16T01:34:32.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.833+0000 7ff8e1a87640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff8dc0ce390 0x7ff8dc0b4840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45208/0 (socket says 172.21.15.88:45208) 2024-09-16T01:34:32.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.834+0000 7ff8e1a87640 1 -- 172.21.15.88:0/2365541662 learned_addr learned my addr 172.21.15.88:0/2365541662 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:32.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.834+0000 7ff8e2288640 1 --2- 172.21.15.88:0/2365541662 >> v2:172.21.15.88:3300/0 conn(0x7ff8dc062630 0x7ff8dc0b5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.834+0000 7ff8e1286640 1 --2- 172.21.15.88:0/2365541662 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8dc071720 0x7ff8dc15a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.834+0000 7ff8e1286640 1 -- 172.21.15.88:0/2365541662 >> v2:172.21.15.88:3300/0 conn(0x7ff8dc062630 msgr2=0x7ff8dc0b5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:32.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.835+0000 7ff8e0a85640 1 -- 172.21.15.88:0/2365541662 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3157153600 0 0) 0x7ff8dc0ce9a0 con 0x7ff8dc0ce390 2024-09-16T01:34:32.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.835+0000 7ff8e1286640 1 --2- 172.21.15.88:0/2365541662 >> v2:172.21.15.88:3300/0 conn(0x7ff8dc062630 0x7ff8dc0b5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.835+0000 7ff8e1286640 1 -- 172.21.15.88:0/2365541662 >> v1:172.21.15.88:6789/0 conn(0x7ff8dc0ce390 legacy=0x7ff8dc0b4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.835+0000 7ff8e1286640 1 -- 172.21.15.88:0/2365541662 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8dc0cec80 con 0x7ff8dc071720 2024-09-16T01:34:32.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.835+0000 7ff8e2288640 1 --2- 172.21.15.88:0/2365541662 >> v2:172.21.15.88:3300/0 conn(0x7ff8dc062630 0x7ff8dc0b5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:34:32.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.836+0000 7f3a2405a640 1 -- start start 2024-09-16T01:34:32.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.837+0000 7ff8e1286640 1 --2- 172.21.15.88:0/2365541662 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8dc071720 0x7ff8dc15a970 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff8d80097a0 tx=0x7ff8d802a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f1dd38c61e249b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:32.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.836+0000 7f8a512fc640 1 --2- 172.21.15.88:0/592151391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a4c0b74a0 0x7f8a4c0b5820 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8a44009e10 tx=0x7f8a44032ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9eadbc8445b2ba1c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:32.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.837+0000 7f8a4b7fe640 1 -- 172.21.15.88:0/592151391 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a44040070 con 0x7f8a4c0b74a0 2024-09-16T01:34:32.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.837+0000 7f8a4b7fe640 1 -- 172.21.15.88:0/592151391 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a44033bb0 con 0x7f8a4c0b74a0 2024-09-16T01:34:32.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.837+0000 7f8a4b7fe640 1 -- 172.21.15.88:0/592151391 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4403b440 con 0x7f8a4c0b74a0 2024-09-16T01:34:32.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.837+0000 7ff8e0a85640 1 -- 172.21.15.88:0/2365541662 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8037070 con 0x7ff8dc071720 2024-09-16T01:34:32.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.837+0000 7ff8e0a85640 1 -- 172.21.15.88:0/2365541662 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8d8032440 con 0x7ff8dc071720 2024-09-16T01:34:32.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.837+0000 7ff8e0a85640 1 -- 172.21.15.88:0/2365541662 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8032810 con 0x7ff8dc071720 2024-09-16T01:34:32.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.837+0000 7f3a2405a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3a1c161d70 0x7f3a1c0b3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.838+0000 7f3a2405a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3a1c0b6890 0x7f3a1c0d22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.838+0000 7f3a2405a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a1c0cf2a0 con 0x7f3a1c162150 2024-09-16T01:34:32.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.838+0000 7f3a2405a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f3a1c0cf760 con 0x7f3a1c0b6890 2024-09-16T01:34:32.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.838+0000 7f3a2405a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3a1c0cf9b0 con 0x7f3a1c161d70 2024-09-16T01:34:32.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.838+0000 7f8a52d5f640 1 -- 172.21.15.88:0/592151391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a4c0b74a0 msgr2=0x7f8a4c0b5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.838+0000 7f8a52d5f640 1 --2- 172.21.15.88:0/592151391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a4c0b74a0 0x7f8a4c0b5820 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8a44009e10 tx=0x7f8a44032ea0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.838+0000 7f8a52d5f640 1 -- 172.21.15.88:0/592151391 shutdown_connections 2024-09-16T01:34:32.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.838+0000 7f8a52d5f640 1 --2- 172.21.15.88:0/592151391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a4c0b74a0 0x7f8a4c0b5820 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.838+0000 7f8a52d5f640 1 --2- 172.21.15.88:0/592151391 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a4c073160 0x7f8a4c0b6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.838+0000 7f8a52d5f640 1 -- 172.21.15.88:0/592151391 >> 172.21.15.88:0/592151391 conn(0x7f8a4c0d24d0 msgr2=0x7f8a4c0d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:32.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.838+0000 7f8a52d5f640 1 -- 172.21.15.88:0/592151391 shutdown_connections 2024-09-16T01:34:32.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.838+0000 7ff8e3ceb640 1 -- 172.21.15.88:0/2365541662 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8dc071720 msgr2=0x7ff8dc15a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.838+0000 7ff8e3ceb640 1 --2- 172.21.15.88:0/2365541662 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8dc071720 0x7ff8dc15a970 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff8d80097a0 tx=0x7ff8d802a230 comp rx=0 tx=0).stop 2024-09-16T01:34:32.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.838+0000 7f3a225f7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3a1c0b6890 0x7f3a1c0d22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.838+0000 7f3a215f5640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f3a1c162150 0x7f3a1c0b6180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:59762/0 (socket says 172.21.15.88:59762) 2024-09-16T01:34:32.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.839+0000 7f3a215f5640 1 -- 172.21.15.88:0/2541022894 learned_addr learned my addr 172.21.15.88:0/2541022894 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:32.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.838+0000 7f8a52d5f640 1 -- 172.21.15.88:0/592151391 wait complete. 2024-09-16T01:34:32.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.839+0000 7f3a21df6640 1 --2- 172.21.15.88:0/2541022894 >> v2:172.21.15.154:3300/0 conn(0x7f3a1c161d70 0x7f3a1c0b3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.839+0000 7f3a20df4640 1 -- 172.21.15.88:0/2541022894 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2535774543 0 0) 0x7f3a1c0cf2a0 con 0x7f3a1c162150 2024-09-16T01:34:32.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.839+0000 7f3a20df4640 1 -- 172.21.15.88:0/2541022894 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a08003520 con 0x7f3a1c162150 2024-09-16T01:34:32.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.839+0000 7f3a225f7640 1 -- 172.21.15.88:0/2541022894 >> v2:172.21.15.154:3300/0 conn(0x7f3a1c161d70 msgr2=0x7f3a1c0b3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:32.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.839+0000 7f3a225f7640 1 --2- 172.21.15.88:0/2541022894 >> v2:172.21.15.154:3300/0 conn(0x7f3a1c161d70 0x7f3a1c0b3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.839+0000 7f3a20df4640 1 -- 172.21.15.88:0/2541022894 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2384878643 0 0) 0x7f3a08003520 con 0x7f3a1c162150 2024-09-16T01:34:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.840+0000 7f3a225f7640 1 -- 172.21.15.88:0/2541022894 >> v1:172.21.15.154:6789/0 conn(0x7f3a1c162150 legacy=0x7f3a1c0b6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.840+0000 7f3a225f7640 1 -- 172.21.15.88:0/2541022894 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a1c0cfc90 con 0x7f3a1c0b6890 2024-09-16T01:34:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.840+0000 7f3a21df6640 1 --2- 172.21.15.88:0/2541022894 >> v2:172.21.15.154:3300/0 conn(0x7f3a1c161d70 0x7f3a1c0b3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.841+0000 7f8a52d5f640 1 Processor -- start 2024-09-16T01:34:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.840+0000 7ff8e3ceb640 1 -- 172.21.15.88:0/2365541662 shutdown_connections 2024-09-16T01:34:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.841+0000 7ff8e3ceb640 1 --2- 172.21.15.88:0/2365541662 >> v2:172.21.15.88:3300/0 conn(0x7ff8dc062630 0x7ff8dc0b5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.841+0000 7ff8e3ceb640 1 --2- 172.21.15.88:0/2365541662 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8dc071720 0x7ff8dc15a970 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.841+0000 7ff8e3ceb640 1 -- 172.21.15.88:0/2365541662 >> 172.21.15.88:0/2365541662 conn(0x7ff8dc0b0aa0 msgr2=0x7ff8dc0b0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:32.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.841+0000 7f8a52d5f640 1 -- start start 2024-09-16T01:34:32.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.841+0000 7f3a225f7640 1 --2- 172.21.15.88:0/2541022894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a1c0b6890 0x7f3a1c0d22d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c08dbd0 tx=0x7f3a0c02a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24b65b276d5c38e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:32.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.841+0000 7ff8e3ceb640 1 -- 172.21.15.88:0/2365541662 shutdown_connections 2024-09-16T01:34:32.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.841+0000 7ff8e3ceb640 1 -- 172.21.15.88:0/2365541662 wait complete. 2024-09-16T01:34:32.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.842+0000 7f3a20df4640 1 -- 172.21.15.88:0/2541022894 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c037070 con 0x7f3a1c0b6890 2024-09-16T01:34:32.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.842+0000 7f3a20df4640 1 -- 172.21.15.88:0/2541022894 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a0c02abc0 con 0x7f3a1c0b6890 2024-09-16T01:34:32.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.843+0000 7f3a20df4640 1 -- 172.21.15.88:0/2541022894 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c037070 con 0x7f3a1c0b6890 2024-09-16T01:34:32.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.843+0000 7f3a2405a640 1 -- 172.21.15.88:0/2541022894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a1c0b6890 msgr2=0x7f3a1c0d22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.843+0000 7f3a2405a640 1 --2- 172.21.15.88:0/2541022894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a1c0b6890 0x7f3a1c0d22d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c08dbd0 tx=0x7f3a0c02a130 comp rx=0 tx=0).stop 2024-09-16T01:34:32.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.843+0000 7ff8e3ceb640 1 Processor -- start 2024-09-16T01:34:32.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.843+0000 7ff8e3ceb640 1 -- start start 2024-09-16T01:34:32.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.844+0000 7f3a2405a640 1 -- 172.21.15.88:0/2541022894 shutdown_connections 2024-09-16T01:34:32.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.844+0000 7f3a2405a640 1 --2- 172.21.15.88:0/2541022894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a1c0b6890 0x7f3a1c0d22d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.844+0000 7f3a2405a640 1 --2- 172.21.15.88:0/2541022894 >> v2:172.21.15.154:3300/0 conn(0x7f3a1c161d70 0x7f3a1c0b3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.844+0000 7f3a2405a640 1 -- 172.21.15.88:0/2541022894 >> 172.21.15.88:0/2541022894 conn(0x7f3a1c06c500 msgr2=0x7f3a1c06c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:32.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.844+0000 7f8a52d5f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a4c073160 0x7f8a4c14a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.844+0000 7f3a2405a640 1 -- 172.21.15.88:0/2541022894 shutdown_connections 2024-09-16T01:34:32.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.844+0000 7f8a52d5f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a4c0b74a0 0x7f8a4c149f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.844+0000 7f8a52d5f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a4c15c770 0x7f8a4c0cdea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.844+0000 7f8a52d5f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c0b5ee0 con 0x7f8a4c073160 2024-09-16T01:34:32.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.844+0000 7f8a52d5f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c0b5d60 con 0x7f8a4c0b74a0 2024-09-16T01:34:32.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.844+0000 7f8a52d5f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8a4c0b6060 con 0x7f8a4c15c770 2024-09-16T01:34:32.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.844+0000 7f3a2405a640 1 -- 172.21.15.88:0/2541022894 wait complete. 2024-09-16T01:34:32.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.844+0000 7ff8e3ceb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8dc062630 0x7ff8dc15e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.844+0000 7ff8e3ceb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8dc071720 0x7ff8dc15b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.844+0000 7ff8e3ceb640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8dc0ce390 0x7ff8dc0afbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.844+0000 7ff8e3ceb640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff8dc0cf240 con 0x7ff8dc062630 2024-09-16T01:34:32.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.844+0000 7ff8e3ceb640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff8dc071b60 con 0x7ff8dc071720 2024-09-16T01:34:32.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.844+0000 7ff8e3ceb640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff8dc15aeb0 con 0x7ff8dc0ce390 2024-09-16T01:34:32.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.845+0000 7f8a50afb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a4c073160 0x7f8a4c14a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.845+0000 7f8a4bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a4c0b74a0 0x7f8a4c149f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.845+0000 7f8a4bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a4c0b74a0 0x7f8a4c149f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:57908/0 (socket says 172.21.15.88:57908) 2024-09-16T01:34:32.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.845+0000 7f8a4bfff640 1 -- 172.21.15.88:0/109865625 learned_addr learned my addr 172.21.15.88:0/109865625 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:32.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.845+0000 7ff8e1286640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8dc071720 0x7ff8dc15b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.845+0000 7ff8e1286640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8dc071720 0x7ff8dc15b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:57914/0 (socket says 172.21.15.88:57914) 2024-09-16T01:34:32.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.845+0000 7ff8e1286640 1 -- 172.21.15.88:0/2368266193 learned_addr learned my addr 172.21.15.88:0/2368266193 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:32.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.845+0000 7f8a512fc640 1 --2- 172.21.15.88:0/109865625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a4c15c770 0x7f8a4c0cdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.845+0000 7ff8e2288640 1 --2- 172.21.15.88:0/2368266193 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8dc0ce390 0x7ff8dc0afbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.845+0000 7ff8e1a87640 1 --2- 172.21.15.88:0/2368266193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8dc062630 0x7ff8dc15e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.845+0000 7f8a4bfff640 1 -- 172.21.15.88:0/109865625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a4c15c770 msgr2=0x7f8a4c0cdea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.845+0000 7f8a4bfff640 1 --2- 172.21.15.88:0/109865625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a4c15c770 0x7f8a4c0cdea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.846+0000 7ff8e1286640 1 -- 172.21.15.88:0/2368266193 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8dc0ce390 msgr2=0x7ff8dc0afbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.846+0000 7ff8e1286640 1 --2- 172.21.15.88:0/2368266193 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8dc0ce390 0x7ff8dc0afbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.846+0000 7ff8e1286640 1 -- 172.21.15.88:0/2368266193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8dc062630 msgr2=0x7ff8dc15e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.846+0000 7ff8e1286640 1 --2- 172.21.15.88:0/2368266193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8dc062630 0x7ff8dc15e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.846+0000 7ff8e1286640 1 -- 172.21.15.88:0/2368266193 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8dc15e6c0 con 0x7ff8dc071720 2024-09-16T01:34:32.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.846+0000 7ff8e1286640 1 --2- 172.21.15.88:0/2368266193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8dc071720 0x7ff8dc15b170 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8009890 tx=0x7ff8d80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:32.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.845+0000 7f8a4bfff640 1 -- 172.21.15.88:0/109865625 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a4c073160 msgr2=0x7f8a4c14a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.845+0000 7f8a4bfff640 1 --2- 172.21.15.88:0/109865625 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a4c073160 0x7f8a4c14a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.845+0000 7f8a4bfff640 1 -- 172.21.15.88:0/109865625 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a4c0c9990 con 0x7f8a4c0b74a0 2024-09-16T01:34:32.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.846+0000 7f8a50afb640 1 --2- 172.21.15.88:0/109865625 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a4c073160 0x7f8a4c14a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:32.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.846+0000 7f8a4bfff640 1 --2- 172.21.15.88:0/109865625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a4c0b74a0 0x7f8a4c149f00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8a4000c7f0 tx=0x7f8a4000ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:32.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.846+0000 7f8a497fa640 1 -- 172.21.15.88:0/109865625 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a40015070 con 0x7f8a4c0b74a0 2024-09-16T01:34:32.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.847+0000 7f3a2405a640 1 Processor -- start 2024-09-16T01:34:32.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.847+0000 7f3a2405a640 1 -- start start 2024-09-16T01:34:32.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.846+0000 7ff8d27fc640 1 -- 172.21.15.88:0/2368266193 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8037050 con 0x7ff8dc071720 2024-09-16T01:34:32.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.846+0000 7ff8d27fc640 1 -- 172.21.15.88:0/2368266193 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8d8038040 con 0x7ff8dc071720 2024-09-16T01:34:32.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.846+0000 7f8a52d5f640 1 -- 172.21.15.88:0/109865625 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a4c0c9c70 con 0x7f8a4c0b74a0 2024-09-16T01:34:32.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.847+0000 7f8a52d5f640 1 -- 172.21.15.88:0/109865625 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a4c0ca230 con 0x7f8a4c0b74a0 2024-09-16T01:34:32.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.848+0000 7f8a497fa640 1 -- 172.21.15.88:0/109865625 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a40004220 con 0x7f8a4c0b74a0 2024-09-16T01:34:32.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.848+0000 7f3a2405a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a1c0b6890 0x7f3a1c155bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.848+0000 7f3a2405a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a1c161d70 0x7f3a1c156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.847+0000 7ff8e3ceb640 1 -- 172.21.15.88:0/2368266193 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8dc15e9a0 con 0x7ff8dc071720 2024-09-16T01:34:32.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.847+0000 7ff8e3ceb640 1 -- 172.21.15.88:0/2368266193 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8dc08ca10 con 0x7ff8dc071720 2024-09-16T01:34:32.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.849+0000 7f3a215f5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a1c161d70 0x7f3a1c156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.849+0000 7f3a215f5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a1c161d70 0x7f3a1c156d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55082/0 (socket says 172.21.15.88:55082) 2024-09-16T01:34:32.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.849+0000 7f3a215f5640 1 -- 172.21.15.88:0/4230054336 learned_addr learned my addr 172.21.15.88:0/4230054336 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:32.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.849+0000 7f3a21df6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a1c0b6890 0x7f3a1c155bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.849+0000 7f8a497fa640 1 -- 172.21.15.88:0/109865625 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a40007620 con 0x7f8a4c0b74a0 2024-09-16T01:34:32.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.849+0000 7f3a2405a640 1 --2- 172.21.15.88:0/4230054336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a1c162150 0x7f3a1c158430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.849+0000 7ff8bffff640 1 -- 172.21.15.88:0/2368266193 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8dc062b30 con 0x7ff8dc071720 2024-09-16T01:34:32.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.849+0000 7f3a2405a640 1 -- 172.21.15.88:0/4230054336 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c0d43e0 con 0x7f3a1c161d70 2024-09-16T01:34:32.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.849+0000 7f3a2405a640 1 -- 172.21.15.88:0/4230054336 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c0d4260 con 0x7f3a1c162150 2024-09-16T01:34:32.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.849+0000 7ff8d27fc640 1 -- 172.21.15.88:0/2368266193 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8030850 con 0x7ff8dc071720 2024-09-16T01:34:32.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.850+0000 7f3a225f7640 1 --2- 172.21.15.88:0/4230054336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a1c162150 0x7f3a1c158430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.850+0000 7f3a2405a640 1 -- 172.21.15.88:0/4230054336 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3a1c0d4560 con 0x7f3a1c0b6890 2024-09-16T01:34:32.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.850+0000 7f8a52d5f640 1 -- 172.21.15.88:0/109865625 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a2c005290 con 0x7f8a4c0b74a0 2024-09-16T01:34:32.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.850+0000 7f3a215f5640 1 -- 172.21.15.88:0/4230054336 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a1c0b6890 msgr2=0x7f3a1c155bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.850+0000 7f3a215f5640 1 --2- 172.21.15.88:0/4230054336 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a1c0b6890 0x7f3a1c155bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.851+0000 7f3a215f5640 1 -- 172.21.15.88:0/4230054336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a1c162150 msgr2=0x7f3a1c158430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.851+0000 7f8a497fa640 1 -- 172.21.15.88:0/109865625 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8a40007840 con 0x7f8a4c0b74a0 2024-09-16T01:34:32.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.851+0000 7f3a215f5640 1 --2- 172.21.15.88:0/4230054336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a1c162150 0x7f3a1c158430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.851+0000 7f3a215f5640 1 -- 172.21.15.88:0/4230054336 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a1c15b650 con 0x7f3a1c161d70 2024-09-16T01:34:32.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.851+0000 7f3a215f5640 1 --2- 172.21.15.88:0/4230054336 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a1c161d70 0x7f3a1c156d70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3a1400a110 tx=0x7f3a1400a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:32.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.851+0000 7f3a225f7640 1 --2- 172.21.15.88:0/4230054336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a1c162150 0x7f3a1c158430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:32.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.851+0000 7f3a127fc640 1 -- 172.21.15.88:0/4230054336 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a14017070 con 0x7f3a1c161d70 2024-09-16T01:34:32.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.852+0000 7f3a2405a640 1 -- 172.21.15.88:0/4230054336 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a1c1542a0 con 0x7f3a1c161d70 2024-09-16T01:34:32.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.852+0000 7f3a127fc640 1 -- 172.21.15.88:0/4230054336 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a14012400 con 0x7f3a1c161d70 2024-09-16T01:34:32.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.852+0000 7f8a497fa640 1 --2- 172.21.15.88:0/109865625 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8a3806c3b0 0x7f8a3806e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.852+0000 7f3a127fc640 1 -- 172.21.15.88:0/4230054336 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a14017070 con 0x7f3a1c161d70 2024-09-16T01:34:32.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.852+0000 7f8a50afb640 1 --2- 172.21.15.88:0/109865625 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8a3806c3b0 0x7f8a3806e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.852+0000 7ff8d27fc640 1 -- 172.21.15.88:0/2368266193 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff8d8045020 con 0x7ff8dc071720 2024-09-16T01:34:32.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.852+0000 7ff8d27fc640 1 --2- 172.21.15.88:0/2368266193 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff8c006c300 0x7ff8c006e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.852+0000 7ff8d27fc640 1 -- 172.21.15.88:0/2368266193 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff8d80ac300 con 0x7ff8dc071720 2024-09-16T01:34:32.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.853+0000 7ff8e1a87640 1 --2- 172.21.15.88:0/2368266193 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff8c006c300 0x7ff8c006e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.853+0000 7f8a497fa640 1 -- 172.21.15.88:0/109865625 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8a4008b4d0 con 0x7f8a4c0b74a0 2024-09-16T01:34:32.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.853+0000 7ff8e1a87640 1 --2- 172.21.15.88:0/2368266193 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff8c006c300 0x7ff8c006e7c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff8d40012e0 tx=0x7ff8d4005c10 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:32.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.854+0000 7f8a50afb640 1 --2- 172.21.15.88:0/109865625 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8a3806c3b0 0x7f8a3806e870 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c000f60 tx=0x7f8a3c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:32.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.854+0000 7f3a2405a640 1 -- 172.21.15.88:0/4230054336 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a1c15b8f0 con 0x7f3a1c161d70 2024-09-16T01:34:32.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.854+0000 7f3a127fc640 1 -- 172.21.15.88:0/4230054336 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3a1401a070 con 0x7f3a1c161d70 2024-09-16T01:34:32.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.855+0000 7f3a127fc640 1 --2- 172.21.15.88:0/4230054336 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a0006c4b0 0x7f3a0006e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.855+0000 7ff8d27fc640 1 -- 172.21.15.88:0/2368266193 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff8d807aae0 con 0x7ff8dc071720 2024-09-16T01:34:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.856+0000 7f3a127fc640 1 -- 172.21.15.88:0/4230054336 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3a14056fd0 con 0x7f3a1c161d70 2024-09-16T01:34:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.856+0000 7f3a21df6640 1 --2- 172.21.15.88:0/4230054336 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a0006c4b0 0x7f3a0006e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.856+0000 7f3a2405a640 1 -- 172.21.15.88:0/4230054336 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a1c0ced50 con 0x7f3a1c161d70 2024-09-16T01:34:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.859+0000 7f8a497fa640 1 -- 172.21.15.88:0/109865625 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8a40059660 con 0x7f8a4c0b74a0 2024-09-16T01:34:32.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.866+0000 7f3a127fc640 1 -- 172.21.15.88:0/4230054336 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3a14058f30 con 0x7f3a1c161d70 2024-09-16T01:34:32.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.866+0000 7f3a21df6640 1 --2- 172.21.15.88:0/4230054336 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a0006c4b0 0x7f3a0006e970 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3a1800abb0 tx=0x7f3a18008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:32.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.878+0000 7f598f3c4640 1 Processor -- start 2024-09-16T01:34:32.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.879+0000 7f598f3c4640 1 -- start start 2024-09-16T01:34:32.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.880+0000 7f598f3c4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5988162ae0 0x7f5988166ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.880+0000 7f598f3c4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59880d0c00 con 0x7f59880b3570 2024-09-16T01:34:32.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.880+0000 7f598f3c4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59880d0ee0 con 0x7f59880b54d0 2024-09-16T01:34:32.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.880+0000 7f598f3c4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f59880d11c0 con 0x7f5988162ae0 2024-09-16T01:34:32.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.880+0000 7f598d961640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f59880b3570 0x7f598806bc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45224/0 (socket says 172.21.15.88:45224) 2024-09-16T01:34:32.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.880+0000 7f598d961640 1 -- 172.21.15.88:0/3767523268 learned_addr learned my addr 172.21.15.88:0/3767523268 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:32.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.881+0000 7f597ffff640 1 -- 172.21.15.88:0/3767523268 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1883753935 0 0) 0x7f59880d0c00 con 0x7f59880b3570 2024-09-16T01:34:32.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.881+0000 7f597ffff640 1 -- 172.21.15.88:0/3767523268 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5974003520 con 0x7f59880b3570 2024-09-16T01:34:32.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.882+0000 7f597ffff640 1 -- 172.21.15.88:0/3767523268 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2025721195 0 0) 0x7f5974003520 con 0x7f59880b3570 2024-09-16T01:34:32.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.882+0000 7f597ffff640 1 -- 172.21.15.88:0/3767523268 >> v2:172.21.15.88:3300/0 conn(0x7f5988162ae0 msgr2=0x7f5988166ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:32.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.882+0000 7f598d160640 1 --2- 172.21.15.88:0/3767523268 >> v2:172.21.15.88:3300/0 conn(0x7f5988162ae0 0x7f5988166ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.882+0000 7f597ffff640 1 --2- 172.21.15.88:0/3767523268 >> v2:172.21.15.88:3300/0 conn(0x7f5988162ae0 0x7f5988166ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.882+0000 7f597ffff640 1 -- 172.21.15.88:0/3767523268 >> v1:172.21.15.154:6789/0 conn(0x7f59880b54d0 legacy=0x7f59880d1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.882+0000 7f597ffff640 1 -- 172.21.15.88:0/3767523268 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59880d14a0 con 0x7f59880b3570 2024-09-16T01:34:32.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.882+0000 7f597ffff640 1 -- 172.21.15.88:0/3767523268 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f59800019f0 con 0x7f59880b3570 2024-09-16T01:34:32.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.882+0000 7f597ffff640 1 -- 172.21.15.88:0/3767523268 >> v1:172.21.15.88:6789/0 conn(0x7f59880b3570 legacy=0x7f598806bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.883+0000 7f597ffff640 1 --2- 172.21.15.88:0/3767523268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59740039b0 0x7f5974003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.883+0000 7f597ffff640 1 --2- 172.21.15.88:0/3767523268 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5974004800 0x7f5974004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.883+0000 7f597ffff640 1 --2- 172.21.15.88:0/3767523268 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59740052e0 0x7f597400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.883+0000 7f597ffff640 1 -- 172.21.15.88:0/3767523268 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f59880d11c0 con 0x7f59740039b0 2024-09-16T01:34:32.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.883+0000 7f597ffff640 1 -- 172.21.15.88:0/3767523268 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f597400c190 con 0x7f59740052e0 2024-09-16T01:34:32.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.883+0000 7f597ffff640 1 -- 172.21.15.88:0/3767523268 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f597400c390 con 0x7f5974004800 2024-09-16T01:34:32.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.883+0000 7f598d160640 1 --2- 172.21.15.88:0/3767523268 >> v2:172.21.15.88:3300/0 conn(0x7f5988162ae0 0x7f5988166ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:32.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.883+0000 7f598c95f640 1 --2- 172.21.15.88:0/3767523268 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5974004800 0x7f5974004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.883+0000 7f598d160640 1 --2- 172.21.15.88:0/3767523268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59740039b0 0x7f5974003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.883+0000 7f598c95f640 1 -- 172.21.15.88:0/3767523268 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59740052e0 msgr2=0x7f597400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:32.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.883+0000 7f598c95f640 1 --2- 172.21.15.88:0/3767523268 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59740052e0 0x7f597400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.883+0000 7f598c95f640 1 -- 172.21.15.88:0/3767523268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59740039b0 msgr2=0x7f5974003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:32.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.883+0000 7f598c95f640 1 --2- 172.21.15.88:0/3767523268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59740039b0 0x7f5974003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.883+0000 7f598c95f640 1 -- 172.21.15.88:0/3767523268 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5974003520 con 0x7f5974004800 2024-09-16T01:34:32.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.884+0000 7f598d961640 1 -- 172.21.15.88:0/3767523268 reap_dead start 2024-09-16T01:34:32.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.884+0000 7f598d160640 1 --2- 172.21.15.88:0/3767523268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59740039b0 0x7f5974003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:32.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.885+0000 7f598c95f640 1 --2- 172.21.15.88:0/3767523268 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5974004800 0x7f5974004c80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f59840013f0 tx=0x7f5974005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77b1a8b202af9c79 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:32.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.885+0000 7f597ffff640 1 -- 172.21.15.88:0/3767523268 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5984059070 con 0x7f5974004800 2024-09-16T01:34:32.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.885+0000 7f597ffff640 1 -- 172.21.15.88:0/3767523268 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f598404cbf0 con 0x7f5974004800 2024-09-16T01:34:32.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.886+0000 7f598f3c4640 1 -- 172.21.15.88:0/3767523268 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5974004800 msgr2=0x7f5974004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.886+0000 7f598f3c4640 1 --2- 172.21.15.88:0/3767523268 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5974004800 0x7f5974004c80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f59840013f0 tx=0x7f5974005830 comp rx=0 tx=0).stop 2024-09-16T01:34:32.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.886+0000 7f598f3c4640 1 -- 172.21.15.88:0/3767523268 shutdown_connections 2024-09-16T01:34:32.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.886+0000 7f598f3c4640 1 --2- 172.21.15.88:0/3767523268 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5974004800 0x7f5974004c80 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.886+0000 7f598f3c4640 1 -- 172.21.15.88:0/3767523268 >> 172.21.15.88:0/3767523268 conn(0x7f59880b6b50 msgr2=0x7f59880b6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:32.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.886+0000 7f598f3c4640 1 -- 172.21.15.88:0/3767523268 shutdown_connections 2024-09-16T01:34:32.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.886+0000 7f598f3c4640 1 -- 172.21.15.88:0/3767523268 wait complete. 2024-09-16T01:34:32.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.888+0000 7f598f3c4640 1 Processor -- start 2024-09-16T01:34:32.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.888+0000 7f598f3c4640 1 -- start start 2024-09-16T01:34:32.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.889+0000 7f598f3c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5974004800 0x7f598814f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.889+0000 7f598f3c4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59880aae90 0x7f59880aedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.889+0000 7f598f3c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59880ab310 0x7f59880af310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.889+0000 7f598f3c4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5988167250 con 0x7f59880aae90 2024-09-16T01:34:32.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.889+0000 7f598f3c4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f59881670d0 con 0x7f59880ab310 2024-09-16T01:34:32.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.889+0000 7f598f3c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f59881673d0 con 0x7f5974004800 2024-09-16T01:34:32.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.890+0000 7f598c95f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59880aae90 0x7f59880aedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.890+0000 7f598c95f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59880aae90 0x7f59880aedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55102/0 (socket says 172.21.15.88:55102) 2024-09-16T01:34:32.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.890+0000 7f598c95f640 1 -- 172.21.15.88:0/2978386059 learned_addr learned my addr 172.21.15.88:0/2978386059 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:32.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.890+0000 7f598c95f640 1 -- 172.21.15.88:0/2978386059 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5974004800 msgr2=0x7f598814f0d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:34:32.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.890+0000 7f598c95f640 1 --2- 172.21.15.88:0/2978386059 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5974004800 0x7f598814f0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.890+0000 7f598c95f640 1 -- 172.21.15.88:0/2978386059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59880ab310 msgr2=0x7f59880af310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.890+0000 7f598c95f640 1 --2- 172.21.15.88:0/2978386059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59880ab310 0x7f59880af310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.890+0000 7f598c95f640 1 -- 172.21.15.88:0/2978386059 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59880ab8e0 con 0x7f59880aae90 2024-09-16T01:34:32.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.890+0000 7f598c95f640 1 --2- 172.21.15.88:0/2978386059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59880aae90 0x7f59880aedd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5984001920 tx=0x7f5984001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:32.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.891+0000 7f597dffb640 1 -- 172.21.15.88:0/2978386059 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5984059050 con 0x7f59880aae90 2024-09-16T01:34:32.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.892+0000 7f598f3c4640 1 -- 172.21.15.88:0/2978386059 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59880abbc0 con 0x7f59880aae90 2024-09-16T01:34:32.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.892+0000 7f598f3c4640 1 -- 172.21.15.88:0/2978386059 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59880acda0 con 0x7f59880aae90 2024-09-16T01:34:32.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.895+0000 7f597dffb640 1 -- 172.21.15.88:0/2978386059 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5984065070 con 0x7f59880aae90 2024-09-16T01:34:32.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.896+0000 7f597dffb640 1 -- 172.21.15.88:0/2978386059 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5984001430 con 0x7f59880aae90 2024-09-16T01:34:32.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.896+0000 7f597dffb640 1 -- 172.21.15.88:0/2978386059 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5984060450 con 0x7f59880aae90 2024-09-16T01:34:32.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.897+0000 7f597dffb640 1 --2- 172.21.15.88:0/2978386059 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f596c06c3e0 0x7f596c06e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.897+0000 7f598d160640 1 --2- 172.21.15.88:0/2978386059 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f596c06c3e0 0x7f596c06e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.898+0000 7f598d160640 1 --2- 172.21.15.88:0/2978386059 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f596c06c3e0 0x7f596c06e8a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f59880aeb50 tx=0x7f5978008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:32.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.898+0000 7f597dffb640 1 -- 172.21.15.88:0/2978386059 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f59840cdc30 con 0x7f59880aae90 2024-09-16T01:34:32.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.899+0000 7f598f3c4640 1 -- 172.21.15.88:0/2978386059 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5968005290 con 0x7f59880aae90 2024-09-16T01:34:32.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.906+0000 7f1aeb0e4640 1 Processor -- start 2024-09-16T01:34:32.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.906+0000 7f1aeb0e4640 1 -- start start 2024-09-16T01:34:32.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.908+0000 7f1aeb0e4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1ae415a680 0x7f1ae406e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.908+0000 7f1aeb0e4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1ae4073b80 0x7f1ae4164210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.908+0000 7f1aeb0e4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ae4073fc0 con 0x7f1ae406ec20 2024-09-16T01:34:32.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.908+0000 7f1aeb0e4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f1ae4071c20 con 0x7f1ae4073b80 2024-09-16T01:34:32.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.908+0000 7f1aeb0e4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f1ae406f060 con 0x7f1ae415a680 2024-09-16T01:34:32.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.908+0000 7f597dffb640 1 -- 172.21.15.88:0/2978386059 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f598409c380 con 0x7f59880aae90 2024-09-16T01:34:32.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.909+0000 7f1ae3fff640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f1ae406ec20 0x7f1ae4073470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:59766/0 (socket says 172.21.15.88:59766) 2024-09-16T01:34:32.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.909+0000 7f1ae9681640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1ae4073b80 0x7f1ae4164210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.909+0000 7f1ae9681640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1ae4073b80 0x7f1ae4164210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55116/0 (socket says 172.21.15.88:55116) 2024-09-16T01:34:32.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.909+0000 7f1ae3fff640 1 -- 172.21.15.88:0/1277399368 learned_addr learned my addr 172.21.15.88:0/1277399368 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:32.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.909+0000 7f1ae9681640 1 -- 172.21.15.88:0/1277399368 >> v2:172.21.15.154:3300/0 conn(0x7f1ae415a680 msgr2=0x7f1ae406e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:32.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.909+0000 7f1ae9681640 1 --2- 172.21.15.88:0/1277399368 >> v2:172.21.15.154:3300/0 conn(0x7f1ae415a680 0x7f1ae406e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.910+0000 7f1ae9681640 1 -- 172.21.15.88:0/1277399368 >> v1:172.21.15.154:6789/0 conn(0x7f1ae406ec20 legacy=0x7f1ae4073470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:32.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.910+0000 7f1ae9681640 1 -- 172.21.15.88:0/1277399368 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae415aa60 con 0x7f1ae4073b80 2024-09-16T01:34:32.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.911+0000 7f1ae9681640 1 --2- 172.21.15.88:0/1277399368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ae4073b80 0x7f1ae4164210 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1ae40d1380 tx=0x7f1adc02a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a26510512892de2e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:32.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.912+0000 7f1ae37fe640 1 -- 172.21.15.88:0/1277399368 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1adc007740 con 0x7f1ae4073b80 2024-09-16T01:34:32.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.912+0000 7f1ae37fe640 1 -- 172.21.15.88:0/1277399368 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1adc0078e0 con 0x7f1ae4073b80 2024-09-16T01:34:32.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.913+0000 7f1aeb0e4640 1 -- 172.21.15.88:0/1277399368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ae4073b80 msgr2=0x7f1ae4164210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.913+0000 7f1aeb0e4640 1 --2- 172.21.15.88:0/1277399368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ae4073b80 0x7f1ae4164210 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1ae40d1380 tx=0x7f1adc02a640 comp rx=0 tx=0).stop 2024-09-16T01:34:32.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.914+0000 7f1aeb0e4640 1 -- 172.21.15.88:0/1277399368 shutdown_connections 2024-09-16T01:34:32.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.914+0000 7f1aeb0e4640 1 --2- 172.21.15.88:0/1277399368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ae4073b80 0x7f1ae4164210 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.914+0000 7f1aeb0e4640 1 --2- 172.21.15.88:0/1277399368 >> v2:172.21.15.154:3300/0 conn(0x7f1ae415a680 0x7f1ae406e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.914+0000 7f1aeb0e4640 1 -- 172.21.15.88:0/1277399368 >> 172.21.15.88:0/1277399368 conn(0x7f1ae40d03e0 msgr2=0x7f1ae40d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:32.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.914+0000 7f1aeb0e4640 1 -- 172.21.15.88:0/1277399368 shutdown_connections 2024-09-16T01:34:32.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.914+0000 7f1aeb0e4640 1 -- 172.21.15.88:0/1277399368 wait complete. 2024-09-16T01:34:32.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.916+0000 7f1aeb0e4640 1 Processor -- start 2024-09-16T01:34:32.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.916+0000 7f1aeb0e4640 1 -- start start 2024-09-16T01:34:32.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.917+0000 7f1aeb0e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1ae406ec20 0x7f1ae4155270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.917+0000 7f1aeb0e4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ae4073b80 0x7f1ae40ce2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.917+0000 7f1aeb0e4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ae415a680 0x7f1ae40ce820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.917+0000 7f1aeb0e4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae40715d0 con 0x7f1ae4073b80 2024-09-16T01:34:32.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.917+0000 7f1aeb0e4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae4071050 con 0x7f1ae415a680 2024-09-16T01:34:32.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.918+0000 7f1aeb0e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1ae40718b0 con 0x7f1ae406ec20 2024-09-16T01:34:32.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.918+0000 7f1ae8e80640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1ae406ec20 0x7f1ae4155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.918+0000 7f1ae9681640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ae415a680 0x7f1ae40ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.918+0000 7f1ae9681640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ae415a680 0x7f1ae40ce820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:57948/0 (socket says 172.21.15.88:57948) 2024-09-16T01:34:32.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.918+0000 7f1ae9681640 1 -- 172.21.15.88:0/1637576221 learned_addr learned my addr 172.21.15.88:0/1637576221 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:32.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.918+0000 7f1ae3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ae4073b80 0x7f1ae40ce2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.920+0000 7f1ae3fff640 1 -- 172.21.15.88:0/1637576221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1ae406ec20 msgr2=0x7f1ae4155270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.920+0000 7f1ae3fff640 1 --2- 172.21.15.88:0/1637576221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1ae406ec20 0x7f1ae4155270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.920+0000 7f1ae3fff640 1 -- 172.21.15.88:0/1637576221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ae415a680 msgr2=0x7f1ae40ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.920+0000 7f1ae3fff640 1 --2- 172.21.15.88:0/1637576221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ae415a680 0x7f1ae40ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.920+0000 7f1ae3fff640 1 -- 172.21.15.88:0/1637576221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae40ca860 con 0x7f1ae4073b80 2024-09-16T01:34:32.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.920+0000 7f1ae9681640 1 --2- 172.21.15.88:0/1637576221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ae415a680 0x7f1ae40ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:34:32.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.920+0000 7f1ae3fff640 1 --2- 172.21.15.88:0/1637576221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ae4073b80 0x7f1ae40ce2e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1ad000c750 tx=0x7f1ad000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:32.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.921+0000 7f1ae17fa640 1 -- 172.21.15.88:0/1637576221 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad00058d0 con 0x7f1ae4073b80 2024-09-16T01:34:32.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.921+0000 7f1ae17fa640 1 -- 172.21.15.88:0/1637576221 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ad0005a70 con 0x7f1ae4073b80 2024-09-16T01:34:32.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.921+0000 7f1ae17fa640 1 -- 172.21.15.88:0/1637576221 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0005e40 con 0x7f1ae4073b80 2024-09-16T01:34:32.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.921+0000 7f1aeb0e4640 1 -- 172.21.15.88:0/1637576221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ae40cab40 con 0x7f1ae4073b80 2024-09-16T01:34:32.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.921+0000 7f1aeb0e4640 1 -- 172.21.15.88:0/1637576221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ae40cb100 con 0x7f1ae4073b80 2024-09-16T01:34:32.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.924+0000 7f1aeb0e4640 1 -- 172.21.15.88:0/1637576221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ae4069180 con 0x7f1ae4073b80 2024-09-16T01:34:32.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.925+0000 7f1ae17fa640 1 -- 172.21.15.88:0/1637576221 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1ad0007070 con 0x7f1ae4073b80 2024-09-16T01:34:32.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.925+0000 7f1ae17fa640 1 --2- 172.21.15.88:0/1637576221 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ad406c3f0 0x7f1ad406e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.925+0000 7f1ae17fa640 1 -- 172.21.15.88:0/1637576221 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1ad008ae60 con 0x7f1ae4073b80 2024-09-16T01:34:32.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.926+0000 7f1ae8e80640 1 --2- 172.21.15.88:0/1637576221 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ad406c3f0 0x7f1ad406e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.927+0000 7f1ae8e80640 1 --2- 172.21.15.88:0/1637576221 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ad406c3f0 0x7f1ad406e8b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1ad80098e0 tx=0x7f1ad8009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:32.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.932+0000 7f1ae17fa640 1 -- 172.21.15.88:0/1637576221 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1ae4069180 con 0x7f1ae4073b80 2024-09-16T01:34:32.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.951+0000 7fa17c60e640 1 Processor -- start 2024-09-16T01:34:32.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.951+0000 7fa17c60e640 1 -- start start 2024-09-16T01:34:32.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.952+0000 7fa17c60e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa1740c58a0 0x7fa174073ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.952+0000 7fa17c60e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa1740d3170 0x7fa17406cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.952+0000 7fa17c60e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1740d1f70 con 0x7fa1740d2d30 2024-09-16T01:34:32.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.952+0000 7fa17c60e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa1740ce060 con 0x7fa1740d3170 2024-09-16T01:34:32.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.952+0000 7fa17c60e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa1740d28a0 con 0x7fa1740c58a0 2024-09-16T01:34:32.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.952+0000 7fa17abab640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa1740d3170 0x7fa17406cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.952+0000 7fa17abab640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa1740d3170 0x7fa17406cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55128/0 (socket says 172.21.15.88:55128) 2024-09-16T01:34:32.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.952+0000 7fa17abab640 1 -- 172.21.15.88:0/2753098578 learned_addr learned my addr 172.21.15.88:0/2753098578 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:32.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.952+0000 7f7cb481e640 1 Processor -- start 2024-09-16T01:34:32.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.952+0000 7fa17a3aa640 1 --2- 172.21.15.88:0/2753098578 >> v2:172.21.15.154:3300/0 conn(0x7fa1740c58a0 0x7fa174073ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.952+0000 7f7cb481e640 1 -- start start 2024-09-16T01:34:32.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.953+0000 7fa17abab640 1 -- 172.21.15.88:0/2753098578 >> v2:172.21.15.154:3300/0 conn(0x7fa1740c58a0 msgr2=0x7fa174073ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:32.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.953+0000 7fa17abab640 1 --2- 172.21.15.88:0/2753098578 >> v2:172.21.15.154:3300/0 conn(0x7fa1740c58a0 0x7fa174073ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.953+0000 7fa17abab640 1 -- 172.21.15.88:0/2753098578 >> v1:172.21.15.154:6789/0 conn(0x7fa1740d2d30 legacy=0x7fa1740b44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:32.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.953+0000 7fa17abab640 1 -- 172.21.15.88:0/2753098578 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1740d2250 con 0x7fa1740d3170 2024-09-16T01:34:32.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.954+0000 7f3749b86640 1 Processor -- start 2024-09-16T01:34:32.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.954+0000 7f3749b86640 1 -- start start 2024-09-16T01:34:32.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.954+0000 7fa17abab640 1 --2- 172.21.15.88:0/2753098578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1740d3170 0x7fa17406cd00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa174072a90 tx=0x7fa16c02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=561df030b691e019 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:32.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.954+0000 7f7cb481e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7cb006edf0 0x7f7cb015a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.954+0000 7f7cb481e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb0073470 0x7f7cb0073900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.954+0000 7f7cb481e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cb0071050 con 0x7f7cb00d0ed0 2024-09-16T01:34:32.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.954+0000 7f7cb481e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7cb0071c20 con 0x7f7cb006edf0 2024-09-16T01:34:32.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.954+0000 7f7cb481e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7cb00712f0 con 0x7f7cb0073470 2024-09-16T01:34:32.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.954+0000 7f7cadd9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7cb006edf0 0x7f7cb015a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.954+0000 7f7cadd9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7cb006edf0 0x7f7cb015a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55132/0 (socket says 172.21.15.88:55132) 2024-09-16T01:34:32.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.954+0000 7f7cadd9b640 1 -- 172.21.15.88:0/4275765373 learned_addr learned my addr 172.21.15.88:0/4275765373 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:32.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.954+0000 7f7caed9d640 1 --2- 172.21.15.88:0/4275765373 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb0073470 0x7f7cb0073900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.954+0000 7fa1793a8640 1 -- 172.21.15.88:0/2753098578 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16c036070 con 0x7fa1740d3170 2024-09-16T01:34:32.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.955+0000 7fa1793a8640 1 -- 172.21.15.88:0/2753098578 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa16c031600 con 0x7fa1740d3170 2024-09-16T01:34:32.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.955+0000 7f7cadd9b640 1 -- 172.21.15.88:0/4275765373 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb0073470 msgr2=0x7f7cb0073900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:32.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.955+0000 7f7cadd9b640 1 --2- 172.21.15.88:0/4275765373 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb0073470 0x7f7cb0073900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.955+0000 7f7cadd9b640 1 -- 172.21.15.88:0/4275765373 >> v1:172.21.15.154:6789/0 conn(0x7f7cb00d0ed0 legacy=0x7f7cb006e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.955+0000 7f7cadd9b640 1 -- 172.21.15.88:0/4275765373 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cb00715d0 con 0x7f7cb006edf0 2024-09-16T01:34:32.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.955+0000 7f3749b86640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f374415f9f0 0x7f3744062840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.956+0000 7f37437fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f374415f9f0 0x7f3744062840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.956+0000 7fa17c60e640 1 -- 172.21.15.88:0/2753098578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1740d3170 msgr2=0x7fa17406cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.956+0000 7fa17c60e640 1 --2- 172.21.15.88:0/2753098578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1740d3170 0x7fa17406cd00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa174072a90 tx=0x7fa16c02a6f0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.956+0000 7f37437fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f374415f9f0 0x7f3744062840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55146/0 (socket says 172.21.15.88:55146) 2024-09-16T01:34:32.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.956+0000 7f37437fe640 1 -- 172.21.15.88:0/1942414960 learned_addr learned my addr 172.21.15.88:0/1942414960 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:32.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.956+0000 7fa17c60e640 1 -- 172.21.15.88:0/2753098578 shutdown_connections 2024-09-16T01:34:32.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.956+0000 7fa17c60e640 1 --2- 172.21.15.88:0/2753098578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1740d3170 0x7fa17406cd00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.956+0000 7fa17c60e640 1 --2- 172.21.15.88:0/2753098578 >> v2:172.21.15.154:3300/0 conn(0x7fa1740c58a0 0x7fa174073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.956+0000 7fa17c60e640 1 -- 172.21.15.88:0/2753098578 >> 172.21.15.88:0/2753098578 conn(0x7fa174062a20 msgr2=0x7fa174062e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:32.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.956+0000 7f3749b86640 1 --2- 172.21.15.88:0/1942414960 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f37440b6850 0x7f374415cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.956+0000 7f3749b86640 1 -- 172.21.15.88:0/1942414960 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f374415fdd0 con 0x7f37440d0970 2024-09-16T01:34:32.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.956+0000 7f7cadd9b640 1 --2- 172.21.15.88:0/4275765373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb006edf0 0x7f7cb015a680 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0009870 tx=0x7f7ca002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93dbde1b4038a9a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:32.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.956+0000 7f3749b86640 1 -- 172.21.15.88:0/1942414960 --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f37440ce540 con 0x7f374415f9f0 2024-09-16T01:34:32.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.956+0000 7f3749b86640 1 -- 172.21.15.88:0/1942414960 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f37440b0180 con 0x7f37440b6850 2024-09-16T01:34:32.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.956+0000 7f3743fff640 1 --2- 172.21.15.88:0/1942414960 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f37440b6850 0x7f374415cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.956+0000 7f7cad59a640 1 -- 172.21.15.88:0/4275765373 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0037070 con 0x7f7cb006edf0 2024-09-16T01:34:32.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.956+0000 7f7cad59a640 1 -- 172.21.15.88:0/4275765373 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ca0032440 con 0x7f7cb006edf0 2024-09-16T01:34:32.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.956+0000 7f7cad59a640 1 -- 172.21.15.88:0/4275765373 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0032810 con 0x7f7cb006edf0 2024-09-16T01:34:32.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.957+0000 7f3742ffd640 1 -- 172.21.15.88:0/1942414960 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2927507395 0 0) 0x7f374415fdd0 con 0x7f37440d0970 2024-09-16T01:34:32.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.957+0000 7f3742ffd640 1 -- 172.21.15.88:0/1942414960 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3730003520 con 0x7f37440d0970 2024-09-16T01:34:32.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.957+0000 7f37437fe640 1 -- 172.21.15.88:0/1942414960 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f37440b6850 msgr2=0x7f374415cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:32.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.957+0000 7f37437fe640 1 --2- 172.21.15.88:0/1942414960 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f37440b6850 0x7f374415cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.957+0000 7f37437fe640 1 -- 172.21.15.88:0/1942414960 >> v1:172.21.15.88:6789/0 conn(0x7f37440d0970 legacy=0x7f37440b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.957+0000 7f37437fe640 1 -- 172.21.15.88:0/1942414960 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37440b0460 con 0x7f374415f9f0 2024-09-16T01:34:32.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.957+0000 7f7cb481e640 1 -- 172.21.15.88:0/4275765373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb006edf0 msgr2=0x7f7cb015a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.957+0000 7f7cb481e640 1 --2- 172.21.15.88:0/4275765373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb006edf0 0x7f7cb015a680 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0009870 tx=0x7f7ca002a270 comp rx=0 tx=0).stop 2024-09-16T01:34:32.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.958+0000 7fa17c60e640 1 -- 172.21.15.88:0/2753098578 shutdown_connections 2024-09-16T01:34:32.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.958+0000 7fa17c60e640 1 -- 172.21.15.88:0/2753098578 wait complete. 2024-09-16T01:34:32.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.958+0000 7f37437fe640 1 --2- 172.21.15.88:0/1942414960 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f374415f9f0 0x7f3744062840 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f372c009870 tx=0x7f372c02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1af5b13984b6046d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:32.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.959+0000 7f7cb481e640 1 -- 172.21.15.88:0/4275765373 shutdown_connections 2024-09-16T01:34:32.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.959+0000 7f7cb481e640 1 --2- 172.21.15.88:0/4275765373 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb0073470 0x7f7cb0073900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.959+0000 7f7cb481e640 1 --2- 172.21.15.88:0/4275765373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb006edf0 0x7f7cb015a680 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.959+0000 7f3742ffd640 1 -- 172.21.15.88:0/1942414960 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372c037070 con 0x7f374415f9f0 2024-09-16T01:34:32.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.959+0000 7f7cb481e640 1 -- 172.21.15.88:0/4275765373 >> 172.21.15.88:0/4275765373 conn(0x7f7cb00d03e0 msgr2=0x7f7cb00d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:32.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.959+0000 7f3742ffd640 1 -- 172.21.15.88:0/1942414960 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f372c032440 con 0x7f374415f9f0 2024-09-16T01:34:32.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.959+0000 7f7cb481e640 1 -- 172.21.15.88:0/4275765373 shutdown_connections 2024-09-16T01:34:32.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.959+0000 7f3742ffd640 1 -- 172.21.15.88:0/1942414960 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372c032810 con 0x7f374415f9f0 2024-09-16T01:34:32.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.959+0000 7fa17c60e640 1 Processor -- start 2024-09-16T01:34:32.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.960+0000 7f3749b86640 1 -- 172.21.15.88:0/1942414960 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f374415f9f0 msgr2=0x7f3744062840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.960+0000 7f3749b86640 1 --2- 172.21.15.88:0/1942414960 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f374415f9f0 0x7f3744062840 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f372c009870 tx=0x7f372c02a250 comp rx=0 tx=0).stop 2024-09-16T01:34:32.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.960+0000 7f3749b86640 1 -- 172.21.15.88:0/1942414960 shutdown_connections 2024-09-16T01:34:32.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.960+0000 7fa17c60e640 1 -- start start 2024-09-16T01:34:32.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.960+0000 7f3749b86640 1 --2- 172.21.15.88:0/1942414960 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f37440b6850 0x7f374415cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.960+0000 7f3749b86640 1 --2- 172.21.15.88:0/1942414960 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f374415f9f0 0x7f3744062840 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.960+0000 7f3749b86640 1 -- 172.21.15.88:0/1942414960 >> 172.21.15.88:0/1942414960 conn(0x7f37440b0aa0 msgr2=0x7f37440b0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:32.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.959+0000 7f7cb481e640 1 -- 172.21.15.88:0/4275765373 wait complete. 2024-09-16T01:34:32.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.961+0000 7f3749b86640 1 -- 172.21.15.88:0/1942414960 shutdown_connections 2024-09-16T01:34:32.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.961+0000 7f3749b86640 1 -- 172.21.15.88:0/1942414960 wait complete. 2024-09-16T01:34:32.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.961+0000 7f7cb481e640 1 Processor -- start 2024-09-16T01:34:32.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.961+0000 7f7cb481e640 1 -- start start 2024-09-16T01:34:32.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.962+0000 7fa17c60e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1740c58a0 0x7fa1740c7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.962+0000 7fa17a3aa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1740c58a0 0x7fa1740c7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.962+0000 7fa17a3aa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1740c58a0 0x7fa1740c7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55148/0 (socket says 172.21.15.88:55148) 2024-09-16T01:34:32.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.962+0000 7fa17a3aa640 1 -- 172.21.15.88:0/3860234685 learned_addr learned my addr 172.21.15.88:0/3860234685 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:32.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.962+0000 7fa17c60e640 1 --2- 172.21.15.88:0/3860234685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1740d2d30 0x7fa1741520d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.962+0000 7fa17c60e640 1 --2- 172.21.15.88:0/3860234685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1740d3170 0x7fa1741f7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.962+0000 7fa17c60e640 1 -- 172.21.15.88:0/3860234685 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa1740ceb10 con 0x7fa1740c58a0 2024-09-16T01:34:32.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.962+0000 7fa17c60e640 1 -- 172.21.15.88:0/3860234685 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa1740ce990 con 0x7fa1740d3170 2024-09-16T01:34:32.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.962+0000 7fa17c60e640 1 -- 172.21.15.88:0/3860234685 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa1740c5c80 con 0x7fa1740d2d30 2024-09-16T01:34:32.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.963+0000 7fa17abab640 1 --2- 172.21.15.88:0/3860234685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1740d3170 0x7fa1741f7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.963+0000 7fa179ba9640 1 --2- 172.21.15.88:0/3860234685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1740d2d30 0x7fa1741520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.963+0000 7f7cb481e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb006edf0 0x7f7cb014a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.963+0000 7f3749b86640 1 Processor -- start 2024-09-16T01:34:32.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.963+0000 7f3749b86640 1 -- start start 2024-09-16T01:34:32.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.963+0000 7fa179ba9640 1 -- 172.21.15.88:0/3860234685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1740d3170 msgr2=0x7fa1741f7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.963+0000 7fa179ba9640 1 --2- 172.21.15.88:0/3860234685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1740d3170 0x7fa1741f7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.963+0000 7fa179ba9640 1 -- 172.21.15.88:0/3860234685 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1740c58a0 msgr2=0x7fa1740c7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.963+0000 7f7cb481e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cb0073470 0x7f7cb01457d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.963+0000 7fa179ba9640 1 --2- 172.21.15.88:0/3860234685 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1740c58a0 0x7fa1740c7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.963+0000 7fa179ba9640 1 -- 172.21.15.88:0/3860234685 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1740c80d0 con 0x7fa1740d2d30 2024-09-16T01:34:32.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.963+0000 7f7cb481e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb00d0ed0 0x7f7cb0140b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.963+0000 7fa17a3aa640 1 --2- 172.21.15.88:0/3860234685 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1740c58a0 0x7fa1740c7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:34:32.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.963+0000 7fa179ba9640 1 --2- 172.21.15.88:0/3860234685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1740d2d30 0x7fa1741520d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa16400c3e0 tx=0x7fa16400c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:32.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.964+0000 7f7caed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb00d0ed0 0x7f7cb0140b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.964+0000 7fa16affd640 1 -- 172.21.15.88:0/3860234685 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa164011790 con 0x7fa1740d2d30 2024-09-16T01:34:32.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.964+0000 7f7caed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb00d0ed0 0x7f7cb0140b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55164/0 (socket says 172.21.15.88:55164) 2024-09-16T01:34:32.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.964+0000 7f7caed9d640 1 -- 172.21.15.88:0/4213258973 learned_addr learned my addr 172.21.15.88:0/4213258973 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:32.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.964+0000 7f7cadd9b640 1 --2- 172.21.15.88:0/4213258973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cb0073470 0x7f7cb01457d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.964+0000 7f7cae59c640 1 --2- 172.21.15.88:0/4213258973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb006edf0 0x7f7cb014a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.964+0000 7fa17c60e640 1 -- 172.21.15.88:0/3860234685 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1740c83b0 con 0x7fa1740d2d30 2024-09-16T01:34:32.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.964+0000 7fa17c60e640 1 -- 172.21.15.88:0/3860234685 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1740c8970 con 0x7fa1740d2d30 2024-09-16T01:34:32.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.964+0000 7fa16affd640 1 -- 172.21.15.88:0/3860234685 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa164017070 con 0x7fa1740d2d30 2024-09-16T01:34:32.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.964+0000 7fa16affd640 1 -- 172.21.15.88:0/3860234685 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa164011b90 con 0x7fa1740d2d30 2024-09-16T01:34:32.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.964+0000 7f3749b86640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37440b6850 0x7f37440af370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.964+0000 7f7cb481e640 1 -- 172.21.15.88:0/4213258973 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb0073e40 con 0x7f7cb00d0ed0 2024-09-16T01:34:32.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.964+0000 7f3749b86640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37440d0970 0x7f374415e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.964+0000 7f37437fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37440b6850 0x7f37440af370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.964+0000 7f37437fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37440b6850 0x7f37440af370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55172/0 (socket says 172.21.15.88:55172) 2024-09-16T01:34:32.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.964+0000 7f7cb481e640 1 -- 172.21.15.88:0/4213258973 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb00b4840 con 0x7f7cb0073470 2024-09-16T01:34:32.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.964+0000 7f7cb481e640 1 -- 172.21.15.88:0/4213258973 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7cb0073fc0 con 0x7f7cb006edf0 2024-09-16T01:34:32.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.964+0000 7f3749b86640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f374415f9f0 0x7f374415ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.964+0000 7f37437fe640 1 -- 172.21.15.88:0/378047188 learned_addr learned my addr 172.21.15.88:0/378047188 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:32.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.964+0000 7f3749b86640 1 -- 172.21.15.88:0/378047188 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f374415d1c0 con 0x7f37440b6850 2024-09-16T01:34:32.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.964+0000 7f3749b86640 1 -- 172.21.15.88:0/378047188 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f37440ce820 con 0x7f37440d0970 2024-09-16T01:34:32.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.964+0000 7f3749b86640 1 -- 172.21.15.88:0/378047188 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f374415d340 con 0x7f374415f9f0 2024-09-16T01:34:32.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.965+0000 7f37437fe640 1 -- 172.21.15.88:0/378047188 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f374415f9f0 msgr2=0x7f374415ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.965+0000 7f37437fe640 1 --2- 172.21.15.88:0/378047188 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f374415f9f0 0x7f374415ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.965+0000 7f37437fe640 1 -- 172.21.15.88:0/378047188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37440d0970 msgr2=0x7f374415e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.965+0000 7f37437fe640 1 --2- 172.21.15.88:0/378047188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37440d0970 0x7f374415e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.965+0000 7f37437fe640 1 -- 172.21.15.88:0/378047188 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37440af8b0 con 0x7f37440b6850 2024-09-16T01:34:32.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.965+0000 7f7cadd9b640 1 -- 172.21.15.88:0/4213258973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb006edf0 msgr2=0x7f7cb014a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.965+0000 7f7cadd9b640 1 --2- 172.21.15.88:0/4213258973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb006edf0 0x7f7cb014a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.965+0000 7f37437fe640 1 --2- 172.21.15.88:0/378047188 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37440b6850 0x7f37440af370 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f372c000c00 tx=0x7f372c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:32.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.965+0000 7f7cadd9b640 1 -- 172.21.15.88:0/4213258973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb00d0ed0 msgr2=0x7f7cb0140b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:32.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.965+0000 7f3740ff9640 1 -- 172.21.15.88:0/378047188 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372c041020 con 0x7f37440b6850 2024-09-16T01:34:32.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.965+0000 7f7cadd9b640 1 --2- 172.21.15.88:0/4213258973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb00d0ed0 0x7f7cb0140b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:32.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.965+0000 7f7cadd9b640 1 -- 172.21.15.88:0/4213258973 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cb014af60 con 0x7f7cb0073470 2024-09-16T01:34:32.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.965+0000 7f3749b86640 1 -- 172.21.15.88:0/378047188 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37441f4e50 con 0x7f37440b6850 2024-09-16T01:34:32.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.965+0000 7f3749b86640 1 -- 172.21.15.88:0/378047188 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37441f45b0 con 0x7f37440b6850 2024-09-16T01:34:32.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.965+0000 7f3740ff9640 1 -- 172.21.15.88:0/378047188 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f372c03d620 con 0x7f37440b6850 2024-09-16T01:34:32.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.966+0000 7f3740ff9640 1 -- 172.21.15.88:0/378047188 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372c037040 con 0x7f37440b6850 2024-09-16T01:34:32.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.965+0000 7f7caed9d640 1 --2- 172.21.15.88:0/4213258973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb00d0ed0 0x7f7cb0140b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:34:32.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.966+0000 7f7cadd9b640 1 --2- 172.21.15.88:0/4213258973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cb0073470 0x7f7cb01457d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0031000 tx=0x7f7ca00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:32.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.966+0000 7f7ca6ffd640 1 -- 172.21.15.88:0/4213258973 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0041020 con 0x7f7cb0073470 2024-09-16T01:34:32.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.966+0000 7f7cb481e640 1 -- 172.21.15.88:0/4213258973 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cb014b240 con 0x7f7cb0073470 2024-09-16T01:34:32.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.966+0000 7f7cb481e640 1 -- 172.21.15.88:0/4213258973 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cb014b800 con 0x7f7cb0073470 2024-09-16T01:34:32.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.967+0000 7fa16affd640 1 -- 172.21.15.88:0/3860234685 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa16401c460 con 0x7fa1740d2d30 2024-09-16T01:34:32.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.967+0000 7f3739ffb640 1 -- 172.21.15.88:0/378047188 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f374415cc80 con 0x7f37440b6850 2024-09-16T01:34:32.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.967+0000 7fa16affd640 1 --2- 172.21.15.88:0/3860234685 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa15806c2b0 0x7fa15806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.967+0000 7f7ca6ffd640 1 -- 172.21.15.88:0/4213258973 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ca0030700 con 0x7f7cb0073470 2024-09-16T01:34:32.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.968+0000 7f7ca6ffd640 1 -- 172.21.15.88:0/4213258973 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0037040 con 0x7f7cb0073470 2024-09-16T01:34:32.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.968+0000 7fa16affd640 1 -- 172.21.15.88:0/3860234685 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa16408b0f0 con 0x7fa1740d2d30 2024-09-16T01:34:32.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.969+0000 7fa17a3aa640 1 --2- 172.21.15.88:0/3860234685 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa15806c2b0 0x7fa15806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.969+0000 7fa17c60e640 1 -- 172.21.15.88:0/3860234685 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa15c005290 con 0x7fa1740d2d30 2024-09-16T01:34:32.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.969+0000 7f3740ff9640 1 -- 172.21.15.88:0/378047188 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f372c044020 con 0x7f37440b6850 2024-09-16T01:34:32.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.970+0000 7f3740ff9640 1 --2- 172.21.15.88:0/378047188 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f372406c2b0 0x7f372406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.970+0000 7f3740ff9640 1 -- 172.21.15.88:0/378047188 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f372c0abaf0 con 0x7f37440b6850 2024-09-16T01:34:32.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.970+0000 7f373bfff640 1 --2- 172.21.15.88:0/378047188 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f372406c2b0 0x7f372406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.971+0000 7fa17a3aa640 1 --2- 172.21.15.88:0/3860234685 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa15806c2b0 0x7fa15806e770 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa170006fd0 tx=0x7fa170008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:32.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.971+0000 7f373bfff640 1 --2- 172.21.15.88:0/378047188 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f372406c2b0 0x7f372406e770 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3734001560 tx=0x7f3734007c70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:32.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.971+0000 7f7ca6ffd640 1 -- 172.21.15.88:0/4213258973 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7ca0044020 con 0x7f7cb0073470 2024-09-16T01:34:32.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.971+0000 7f7ca4ff9640 1 -- 172.21.15.88:0/4213258973 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c94005290 con 0x7f7cb0073470 2024-09-16T01:34:32.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.972+0000 7f7ca6ffd640 1 --2- 172.21.15.88:0/4213258973 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7c9006c2b0 0x7f7c9006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:32.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.972+0000 7f7cae59c640 1 --2- 172.21.15.88:0/4213258973 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7c9006c2b0 0x7f7c9006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:32.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.972+0000 7f7cae59c640 1 --2- 172.21.15.88:0/4213258973 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7c9006c2b0 0x7f7c9006e770 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7cb01f7de0 tx=0x7f7c98009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:32.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.972+0000 7f7ca6ffd640 1 -- 172.21.15.88:0/4213258973 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7ca00aba80 con 0x7f7cb0073470 2024-09-16T01:34:32.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.974+0000 7f3740ff9640 1 -- 172.21.15.88:0/378047188 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f372c04b030 con 0x7f37440b6850 2024-09-16T01:34:32.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.975+0000 7fa16affd640 1 -- 172.21.15.88:0/3860234685 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa164059170 con 0x7fa1740d2d30 2024-09-16T01:34:32.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:32.976+0000 7f7ca6ffd640 1 -- 172.21.15.88:0/4213258973 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7ca004b030 con 0x7f7cb0073470 2024-09-16T01:34:33.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.247+0000 7f3a2405a640 1 -- 172.21.15.88:0/4230054336 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f3a1c0ba2d0 con 0x7f3a1c161d70 2024-09-16T01:34:33.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.248+0000 7f3a127fc640 1 -- 172.21.15.88:0/4230054336 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3a1405d230 con 0x7f3a1c161d70 2024-09-16T01:34:33.249 INFO:teuthology.orchestra.run.smithi088.stdout:60129542152 2024-09-16T01:34:33.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.251+0000 7f39fffff640 1 -- 172.21.15.88:0/4230054336 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a0006c4b0 msgr2=0x7f3a0006e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.251+0000 7f39fffff640 1 --2- 172.21.15.88:0/4230054336 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a0006c4b0 0x7f3a0006e970 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3a1800abb0 tx=0x7f3a18008040 comp rx=0 tx=0).stop 2024-09-16T01:34:33.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.251+0000 7f39fffff640 1 -- 172.21.15.88:0/4230054336 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a1c161d70 msgr2=0x7f3a1c156d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.251+0000 7f39fffff640 1 --2- 172.21.15.88:0/4230054336 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a1c161d70 0x7f3a1c156d70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3a1400a110 tx=0x7f3a1400a5d0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.252+0000 7f39fffff640 1 -- 172.21.15.88:0/4230054336 shutdown_connections 2024-09-16T01:34:33.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.252+0000 7f39fffff640 1 --2- 172.21.15.88:0/4230054336 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a0006c4b0 0x7f3a0006e970 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.252+0000 7f39fffff640 1 --2- 172.21.15.88:0/4230054336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a1c162150 0x7f3a1c158430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.252+0000 7f39fffff640 1 --2- 172.21.15.88:0/4230054336 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a1c161d70 0x7f3a1c156d70 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.252+0000 7f39fffff640 1 --2- 172.21.15.88:0/4230054336 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a1c0b6890 0x7f3a1c155bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.252+0000 7f39fffff640 1 -- 172.21.15.88:0/4230054336 >> 172.21.15.88:0/4230054336 conn(0x7f3a1c06c500 msgr2=0x7f3a1c0d2ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:33.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.252+0000 7ff8bffff640 1 -- 172.21.15.88:0/2368266193 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7ff8dc06f9c0 con 0x7ff8dc071720 2024-09-16T01:34:33.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.252+0000 7f39fffff640 1 -- 172.21.15.88:0/4230054336 shutdown_connections 2024-09-16T01:34:33.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.252+0000 7f39fffff640 1 -- 172.21.15.88:0/4230054336 wait complete. 2024-09-16T01:34:33.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.253+0000 7ff8d27fc640 1 -- 172.21.15.88:0/2368266193 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff8d80ac0e0 con 0x7ff8dc071720 2024-09-16T01:34:33.255 INFO:teuthology.orchestra.run.smithi088.stdout:60129542151 2024-09-16T01:34:33.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.256+0000 7ff8e3ceb640 1 -- 172.21.15.88:0/2368266193 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff8c006c300 msgr2=0x7ff8c006e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.256+0000 7ff8e3ceb640 1 --2- 172.21.15.88:0/2368266193 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff8c006c300 0x7ff8c006e7c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff8d40012e0 tx=0x7ff8d4005c10 comp rx=0 tx=0).stop 2024-09-16T01:34:33.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.256+0000 7ff8e3ceb640 1 -- 172.21.15.88:0/2368266193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8dc071720 msgr2=0x7ff8dc15b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.257+0000 7ff8e3ceb640 1 --2- 172.21.15.88:0/2368266193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8dc071720 0x7ff8dc15b170 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8009890 tx=0x7ff8d80046c0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.257+0000 7ff8e3ceb640 1 -- 172.21.15.88:0/2368266193 shutdown_connections 2024-09-16T01:34:33.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.257+0000 7ff8e3ceb640 1 --2- 172.21.15.88:0/2368266193 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff8c006c300 0x7ff8c006e7c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.257+0000 7ff8e3ceb640 1 --2- 172.21.15.88:0/2368266193 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8dc0ce390 0x7ff8dc0afbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.257+0000 7ff8e3ceb640 1 --2- 172.21.15.88:0/2368266193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8dc071720 0x7ff8dc15b170 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.257+0000 7ff8e3ceb640 1 --2- 172.21.15.88:0/2368266193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8dc062630 0x7ff8dc15e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.257+0000 7ff8e3ceb640 1 -- 172.21.15.88:0/2368266193 >> 172.21.15.88:0/2368266193 conn(0x7ff8dc0b0aa0 msgr2=0x7ff8dc0b4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:33.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.257+0000 7ff8e3ceb640 1 -- 172.21.15.88:0/2368266193 shutdown_connections 2024-09-16T01:34:33.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.257+0000 7ff8e3ceb640 1 -- 172.21.15.88:0/2368266193 wait complete. 2024-09-16T01:34:33.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.264+0000 7f8a52d5f640 1 -- 172.21.15.88:0/109865625 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f8a2c005580 con 0x7f8a4c0b74a0 2024-09-16T01:34:33.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.268+0000 7f8a497fa640 1 -- 172.21.15.88:0/109865625 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8a4005d960 con 0x7f8a4c0b74a0 2024-09-16T01:34:33.271 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.4 2024-09-16T01:34:33.271 INFO:teuthology.orchestra.run.smithi088.stdout:60129542151 2024-09-16T01:34:33.272 DEBUG:teuthology.parallel:result is None 2024-09-16T01:34:33.273 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.0 2024-09-16T01:34:33.273 DEBUG:teuthology.parallel:result is None 2024-09-16T01:34:33.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.273+0000 7f8a52d5f640 1 -- 172.21.15.88:0/109865625 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8a3806c3b0 msgr2=0x7f8a3806e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.273+0000 7f8a52d5f640 1 --2- 172.21.15.88:0/109865625 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8a3806c3b0 0x7f8a3806e870 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c000f60 tx=0x7f8a3c009040 comp rx=0 tx=0).stop 2024-09-16T01:34:33.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.273+0000 7f8a52d5f640 1 -- 172.21.15.88:0/109865625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a4c0b74a0 msgr2=0x7f8a4c149f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.273+0000 7f8a52d5f640 1 --2- 172.21.15.88:0/109865625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a4c0b74a0 0x7f8a4c149f00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8a4000c7f0 tx=0x7f8a4000ccb0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.273+0000 7f8a52d5f640 1 -- 172.21.15.88:0/109865625 shutdown_connections 2024-09-16T01:34:33.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.273+0000 7f8a52d5f640 1 --2- 172.21.15.88:0/109865625 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8a3806c3b0 0x7f8a3806e870 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.273+0000 7f8a52d5f640 1 --2- 172.21.15.88:0/109865625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a4c15c770 0x7f8a4c0cdea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.273+0000 7f8a52d5f640 1 --2- 172.21.15.88:0/109865625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a4c0b74a0 0x7f8a4c149f00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.273+0000 7f8a52d5f640 1 --2- 172.21.15.88:0/109865625 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a4c073160 0x7f8a4c14a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.273+0000 7f8a52d5f640 1 -- 172.21.15.88:0/109865625 >> 172.21.15.88:0/109865625 conn(0x7f8a4c0d24d0 msgr2=0x7f8a4c073580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:33.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.273+0000 7f8a52d5f640 1 -- 172.21.15.88:0/109865625 shutdown_connections 2024-09-16T01:34:33.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.273+0000 7f8a52d5f640 1 -- 172.21.15.88:0/109865625 wait complete. 2024-09-16T01:34:33.286 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.1 2024-09-16T01:34:33.286 DEBUG:teuthology.parallel:result is None 2024-09-16T01:34:33.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.301+0000 7f598f3c4640 1 -- 172.21.15.88:0/2978386059 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f5968005580 con 0x7f59880aae90 2024-09-16T01:34:33.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.301+0000 7f597dffb640 1 -- 172.21.15.88:0/2978386059 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f598409bc80 con 0x7f59880aae90 2024-09-16T01:34:33.303 INFO:teuthology.orchestra.run.smithi088.stdout:60129542152 2024-09-16T01:34:33.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.305+0000 7f598f3c4640 1 -- 172.21.15.88:0/2978386059 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f596c06c3e0 msgr2=0x7f596c06e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.305+0000 7f598f3c4640 1 --2- 172.21.15.88:0/2978386059 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f596c06c3e0 0x7f596c06e8a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f59880aeb50 tx=0x7f5978008040 comp rx=0 tx=0).stop 2024-09-16T01:34:33.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.305+0000 7f598f3c4640 1 -- 172.21.15.88:0/2978386059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59880aae90 msgr2=0x7f59880aedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.305+0000 7f598f3c4640 1 --2- 172.21.15.88:0/2978386059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59880aae90 0x7f59880aedd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5984001920 tx=0x7f5984001230 comp rx=0 tx=0).stop 2024-09-16T01:34:33.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.305+0000 7f598f3c4640 1 -- 172.21.15.88:0/2978386059 shutdown_connections 2024-09-16T01:34:33.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.305+0000 7f598f3c4640 1 --2- 172.21.15.88:0/2978386059 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f596c06c3e0 0x7f596c06e8a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.305+0000 7f598f3c4640 1 --2- 172.21.15.88:0/2978386059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59880ab310 0x7f59880af310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.305+0000 7f598f3c4640 1 --2- 172.21.15.88:0/2978386059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59880aae90 0x7f59880aedd0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.305+0000 7f598f3c4640 1 --2- 172.21.15.88:0/2978386059 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5974004800 0x7f598814f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.305+0000 7f598f3c4640 1 -- 172.21.15.88:0/2978386059 >> 172.21.15.88:0/2978386059 conn(0x7f59880b6b50 msgr2=0x7f59880b6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:33.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.305+0000 7f598f3c4640 1 -- 172.21.15.88:0/2978386059 shutdown_connections 2024-09-16T01:34:33.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.305+0000 7f598f3c4640 1 -- 172.21.15.88:0/2978386059 wait complete. 2024-09-16T01:34:33.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.311+0000 7fa17c60e640 1 -- 172.21.15.88:0/3860234685 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fa15c005580 con 0x7fa1740d2d30 2024-09-16T01:34:33.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.312+0000 7fa16affd640 1 -- 172.21.15.88:0/3860234685 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa16405d470 con 0x7fa1740d2d30 2024-09-16T01:34:33.313 INFO:teuthology.orchestra.run.smithi088.stdout:60129542151 2024-09-16T01:34:33.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.314+0000 7fa17c60e640 1 -- 172.21.15.88:0/3860234685 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa15806c2b0 msgr2=0x7fa15806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.314+0000 7fa17c60e640 1 --2- 172.21.15.88:0/3860234685 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa15806c2b0 0x7fa15806e770 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa170006fd0 tx=0x7fa170008040 comp rx=0 tx=0).stop 2024-09-16T01:34:33.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.314+0000 7fa17c60e640 1 -- 172.21.15.88:0/3860234685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1740d2d30 msgr2=0x7fa1741520d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.314+0000 7fa17c60e640 1 --2- 172.21.15.88:0/3860234685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1740d2d30 0x7fa1741520d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa16400c3e0 tx=0x7fa16400c8a0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.314+0000 7fa17c60e640 1 -- 172.21.15.88:0/3860234685 shutdown_connections 2024-09-16T01:34:33.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.314+0000 7fa17c60e640 1 --2- 172.21.15.88:0/3860234685 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa15806c2b0 0x7fa15806e770 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.314+0000 7fa17c60e640 1 --2- 172.21.15.88:0/3860234685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1740d3170 0x7fa1741f7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.314+0000 7fa17c60e640 1 --2- 172.21.15.88:0/3860234685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1740d2d30 0x7fa1741520d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.314+0000 7fa17c60e640 1 --2- 172.21.15.88:0/3860234685 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1740c58a0 0x7fa1740c7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.314+0000 7fa17c60e640 1 -- 172.21.15.88:0/3860234685 >> 172.21.15.88:0/3860234685 conn(0x7fa174062a20 msgr2=0x7fa174073660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:33.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.314+0000 7fa17c60e640 1 -- 172.21.15.88:0/3860234685 shutdown_connections 2024-09-16T01:34:33.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.314+0000 7fa17c60e640 1 -- 172.21.15.88:0/3860234685 wait complete. 2024-09-16T01:34:33.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.319+0000 7f1aeb0e4640 1 -- 172.21.15.88:0/1637576221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f1ae40cb300 con 0x7f1ae4073b80 2024-09-16T01:34:33.322 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.6 2024-09-16T01:34:33.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.319+0000 7f1ae17fa640 1 -- 172.21.15.88:0/1637576221 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1ae40cb300 con 0x7f1ae4073b80 2024-09-16T01:34:33.322 INFO:teuthology.orchestra.run.smithi088.stdout:60129542152 2024-09-16T01:34:33.322 DEBUG:teuthology.parallel:result is None 2024-09-16T01:34:33.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.321+0000 7f1aeb0e4640 1 -- 172.21.15.88:0/1637576221 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ad406c3f0 msgr2=0x7f1ad406e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.321+0000 7f1aeb0e4640 1 --2- 172.21.15.88:0/1637576221 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ad406c3f0 0x7f1ad406e8b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1ad80098e0 tx=0x7f1ad8009060 comp rx=0 tx=0).stop 2024-09-16T01:34:33.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.321+0000 7f1aeb0e4640 1 -- 172.21.15.88:0/1637576221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ae4073b80 msgr2=0x7f1ae40ce2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.321+0000 7f1aeb0e4640 1 --2- 172.21.15.88:0/1637576221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ae4073b80 0x7f1ae40ce2e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1ad000c750 tx=0x7f1ad000cc10 comp rx=0 tx=0).stop 2024-09-16T01:34:33.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.322+0000 7f1aeb0e4640 1 -- 172.21.15.88:0/1637576221 shutdown_connections 2024-09-16T01:34:33.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.322+0000 7f1aeb0e4640 1 --2- 172.21.15.88:0/1637576221 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ad406c3f0 0x7f1ad406e8b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.322+0000 7f1aeb0e4640 1 --2- 172.21.15.88:0/1637576221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ae415a680 0x7f1ae40ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.322+0000 7f1aeb0e4640 1 --2- 172.21.15.88:0/1637576221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ae4073b80 0x7f1ae40ce2e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.322+0000 7f1aeb0e4640 1 --2- 172.21.15.88:0/1637576221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1ae406ec20 0x7f1ae4155270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.322+0000 7f1aeb0e4640 1 -- 172.21.15.88:0/1637576221 >> 172.21.15.88:0/1637576221 conn(0x7f1ae40d03e0 msgr2=0x7f1ae40d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:33.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.322+0000 7f1aeb0e4640 1 -- 172.21.15.88:0/1637576221 shutdown_connections 2024-09-16T01:34:33.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.322+0000 7f1aeb0e4640 1 -- 172.21.15.88:0/1637576221 wait complete. 2024-09-16T01:34:33.327 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.3 2024-09-16T01:34:33.328 DEBUG:teuthology.parallel:result is None 2024-09-16T01:34:33.335 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.7 2024-09-16T01:34:33.335 DEBUG:teuthology.parallel:result is None 2024-09-16T01:34:33.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.364+0000 7f7ca4ff9640 1 -- 172.21.15.88:0/4213258973 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f7c94005580 con 0x7f7cb0073470 2024-09-16T01:34:33.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.365+0000 7f3739ffb640 1 -- 172.21.15.88:0/378047188 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f37440b19c0 con 0x7f37440b6850 2024-09-16T01:34:33.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.365+0000 7f7ca6ffd640 1 -- 172.21.15.88:0/4213258973 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7ca007a220 con 0x7f7cb0073470 2024-09-16T01:34:33.367 INFO:teuthology.orchestra.run.smithi088.stdout:60129542152 2024-09-16T01:34:33.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.365+0000 7f3740ff9640 1 -- 172.21.15.88:0/378047188 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f37440af8b0 con 0x7f37440b6850 2024-09-16T01:34:33.368 INFO:teuthology.orchestra.run.smithi088.stdout:64424509449 2024-09-16T01:34:33.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.367+0000 7f3739ffb640 1 -- 172.21.15.88:0/378047188 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f372406c2b0 msgr2=0x7f372406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.367+0000 7f3739ffb640 1 --2- 172.21.15.88:0/378047188 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f372406c2b0 0x7f372406e770 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3734001560 tx=0x7f3734007c70 comp rx=0 tx=0).stop 2024-09-16T01:34:33.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.367+0000 7f3739ffb640 1 -- 172.21.15.88:0/378047188 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37440b6850 msgr2=0x7f37440af370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.367+0000 7f3739ffb640 1 --2- 172.21.15.88:0/378047188 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37440b6850 0x7f37440af370 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f372c000c00 tx=0x7f372c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.367+0000 7f3739ffb640 1 -- 172.21.15.88:0/378047188 shutdown_connections 2024-09-16T01:34:33.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.367+0000 7f3739ffb640 1 --2- 172.21.15.88:0/378047188 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f372406c2b0 0x7f372406e770 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.367+0000 7f3739ffb640 1 --2- 172.21.15.88:0/378047188 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f374415f9f0 0x7f374415ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.367+0000 7f3739ffb640 1 --2- 172.21.15.88:0/378047188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37440d0970 0x7f374415e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.367+0000 7f7ca4ff9640 1 -- 172.21.15.88:0/4213258973 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7c9006c2b0 msgr2=0x7f7c9006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.367+0000 7f7ca4ff9640 1 --2- 172.21.15.88:0/4213258973 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7c9006c2b0 0x7f7c9006e770 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7cb01f7de0 tx=0x7f7c98009040 comp rx=0 tx=0).stop 2024-09-16T01:34:33.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.367+0000 7f3739ffb640 1 --2- 172.21.15.88:0/378047188 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37440b6850 0x7f37440af370 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.367+0000 7f3739ffb640 1 -- 172.21.15.88:0/378047188 >> 172.21.15.88:0/378047188 conn(0x7f37440b0aa0 msgr2=0x7f374406dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:33.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.367+0000 7f7ca4ff9640 1 -- 172.21.15.88:0/4213258973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cb0073470 msgr2=0x7f7cb01457d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.367+0000 7f7ca4ff9640 1 --2- 172.21.15.88:0/4213258973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cb0073470 0x7f7cb01457d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0031000 tx=0x7f7ca00046c0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.367+0000 7f3739ffb640 1 -- 172.21.15.88:0/378047188 shutdown_connections 2024-09-16T01:34:33.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.368+0000 7f3739ffb640 1 -- 172.21.15.88:0/378047188 wait complete. 2024-09-16T01:34:33.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.367+0000 7f7ca4ff9640 1 -- 172.21.15.88:0/4213258973 shutdown_connections 2024-09-16T01:34:33.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.367+0000 7f7ca4ff9640 1 --2- 172.21.15.88:0/4213258973 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7c9006c2b0 0x7f7c9006e770 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.368+0000 7f7ca4ff9640 1 --2- 172.21.15.88:0/4213258973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb00d0ed0 0x7f7cb0140b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.368+0000 7f7ca4ff9640 1 --2- 172.21.15.88:0/4213258973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cb0073470 0x7f7cb01457d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.368+0000 7f7ca4ff9640 1 --2- 172.21.15.88:0/4213258973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb006edf0 0x7f7cb014a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.368+0000 7f7ca4ff9640 1 -- 172.21.15.88:0/4213258973 >> 172.21.15.88:0/4213258973 conn(0x7f7cb00d03e0 msgr2=0x7f7cb00ce900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:33.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.368+0000 7f7ca4ff9640 1 -- 172.21.15.88:0/4213258973 shutdown_connections 2024-09-16T01:34:33.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.368+0000 7f7ca4ff9640 1 -- 172.21.15.88:0/4213258973 wait complete. 2024-09-16T01:34:33.381 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509449 for osd.5 2024-09-16T01:34:33.381 DEBUG:teuthology.parallel:result is None 2024-09-16T01:34:33.382 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.2 2024-09-16T01:34:33.382 DEBUG:teuthology.parallel:result is None 2024-09-16T01:34:33.382 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T01:34:33.382 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:34:33.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.517+0000 7f1bf09f1640 1 Processor -- start 2024-09-16T01:34:33.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.518+0000 7f1bf09f1640 1 -- start start 2024-09-16T01:34:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1bf09f1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bec0b6eb0 0x7f1bec06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1bf09f1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bec0708d0 con 0x7f1bec0b7290 2024-09-16T01:34:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1bf09f1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bec070bb0 con 0x7f1bec0d1b40 2024-09-16T01:34:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1bf09f1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1bec070e90 con 0x7f1bec0b6eb0 2024-09-16T01:34:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1bead9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1bec0b7290 0x7f1bec1628b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45242/0 (socket says 172.21.15.88:45242) 2024-09-16T01:34:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1bead9d640 1 -- 172.21.15.88:0/3233328320 learned_addr learned my addr 172.21.15.88:0/3233328320 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1bea59c640 1 --2- 172.21.15.88:0/3233328320 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bec0b6eb0 0x7f1bec06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:33.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1be959a640 1 -- 172.21.15.88:0/3233328320 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2851171129 0 0) 0x7f1bec0708d0 con 0x7f1bec0b7290 2024-09-16T01:34:33.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1be959a640 1 -- 172.21.15.88:0/3233328320 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bd8003520 con 0x7f1bec0b7290 2024-09-16T01:34:33.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1be959a640 1 -- 172.21.15.88:0/3233328320 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2019008550 0 0) 0x7f1bec070bb0 con 0x7f1bec0d1b40 2024-09-16T01:34:33.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1be959a640 1 -- 172.21.15.88:0/3233328320 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bec0708d0 con 0x7f1bec0d1b40 2024-09-16T01:34:33.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1be959a640 1 -- 172.21.15.88:0/3233328320 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3757770753 0 0) 0x7f1bd8003520 con 0x7f1bec0b7290 2024-09-16T01:34:33.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1be959a640 1 -- 172.21.15.88:0/3233328320 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bec0b6eb0 msgr2=0x7f1bec06fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:33.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1be959a640 1 --2- 172.21.15.88:0/3233328320 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bec0b6eb0 0x7f1bec06fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1be959a640 1 -- 172.21.15.88:0/3233328320 >> v1:172.21.15.154:6789/0 conn(0x7f1bec0d1b40 legacy=0x7f1bec06cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1be959a640 1 -- 172.21.15.88:0/3233328320 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bec071170 con 0x7f1bec0b7290 2024-09-16T01:34:33.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1be959a640 1 -- 172.21.15.88:0/3233328320 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f1be40019f0 con 0x7f1bec0b7290 2024-09-16T01:34:33.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1be959a640 1 -- 172.21.15.88:0/3233328320 >> v1:172.21.15.88:6789/0 conn(0x7f1bec0b7290 legacy=0x7f1bec1628b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1be959a640 1 --2- 172.21.15.88:0/3233328320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bd8003b70 0x7f1bd8003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:33.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1be959a640 1 --2- 172.21.15.88:0/3233328320 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bec0b6eb0 0x7f1bd8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:33.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1be959a640 1 --2- 172.21.15.88:0/3233328320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bd8004d40 0x7f1bd80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:33.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1be959a640 1 -- 172.21.15.88:0/3233328320 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1bec070e90 con 0x7f1bd8003b70 2024-09-16T01:34:33.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1be959a640 1 -- 172.21.15.88:0/3233328320 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd800bac0 con 0x7f1bd8004d40 2024-09-16T01:34:33.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1be959a640 1 -- 172.21.15.88:0/3233328320 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1bd800bcc0 con 0x7f1bec0b6eb0 2024-09-16T01:34:33.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1bea59c640 1 --2- 172.21.15.88:0/3233328320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bd8003b70 0x7f1bd8003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:33.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.519+0000 7f1be9d9b640 1 --2- 172.21.15.88:0/3233328320 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bec0b6eb0 0x7f1bd8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:33.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.520+0000 7f1bead9d640 1 --2- 172.21.15.88:0/3233328320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bd8004d40 0x7f1bd80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:33.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.520+0000 7f1bea59c640 1 -- 172.21.15.88:0/3233328320 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bec0b6eb0 msgr2=0x7f1bd8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:33.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.520+0000 7f1bea59c640 1 --2- 172.21.15.88:0/3233328320 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bec0b6eb0 0x7f1bd8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.520+0000 7f1bea59c640 1 -- 172.21.15.88:0/3233328320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bd8004d40 msgr2=0x7f1bd80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:33.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.520+0000 7f1bea59c640 1 --2- 172.21.15.88:0/3233328320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bd8004d40 0x7f1bd80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.520+0000 7f1bea59c640 1 -- 172.21.15.88:0/3233328320 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd8003520 con 0x7f1bd8003b70 2024-09-16T01:34:33.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.520+0000 7f1bead9d640 1 --2- 172.21.15.88:0/3233328320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bd8004d40 0x7f1bd80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:33.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.520+0000 7f1bea59c640 1 --2- 172.21.15.88:0/3233328320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bd8003b70 0x7f1bd8003f70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4009ac0 tx=0x7f1bd8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50c1094d9f90b255 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:33.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.520+0000 7f1be959a640 1 -- 172.21.15.88:0/3233328320 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd4036070 con 0x7f1bd8003b70 2024-09-16T01:34:33.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.520+0000 7f1be959a640 1 -- 172.21.15.88:0/3233328320 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bd4031400 con 0x7f1bd8003b70 2024-09-16T01:34:33.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.520+0000 7f1be959a640 1 -- 172.21.15.88:0/3233328320 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd4009690 con 0x7f1bd8003b70 2024-09-16T01:34:33.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.521+0000 7f1bf09f1640 1 -- 172.21.15.88:0/3233328320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bd8003b70 msgr2=0x7f1bd8003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.521+0000 7f1bf09f1640 1 --2- 172.21.15.88:0/3233328320 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bd8003b70 0x7f1bd8003f70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4009ac0 tx=0x7f1bd8005830 comp rx=0 tx=0).stop 2024-09-16T01:34:33.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.521+0000 7f1bead9d640 1 -- 172.21.15.88:0/3233328320 reap_dead start 2024-09-16T01:34:33.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.521+0000 7f1bf09f1640 1 -- 172.21.15.88:0/3233328320 shutdown_connections 2024-09-16T01:34:33.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.521+0000 7f1bf09f1640 1 -- 172.21.15.88:0/3233328320 >> 172.21.15.88:0/3233328320 conn(0x7f1bec0d2450 msgr2=0x7f1bec0d2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:33.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.521+0000 7f1bf09f1640 1 -- 172.21.15.88:0/3233328320 shutdown_connections 2024-09-16T01:34:33.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.521+0000 7f1bf09f1640 1 -- 172.21.15.88:0/3233328320 wait complete. 2024-09-16T01:34:33.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.522+0000 7f1bf09f1640 1 Processor -- start 2024-09-16T01:34:33.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.522+0000 7f1bf09f1640 1 -- start start 2024-09-16T01:34:33.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.523+0000 7f1bf09f1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bd8003b70 0x7f1bec155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:33.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.523+0000 7f1bf09f1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bec154690 0x7f1bec152bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:33.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.523+0000 7f1bf09f1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bec156200 0x7f1bec1566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:33.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.523+0000 7f1bf09f1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1bec071af0 con 0x7f1bec156200 2024-09-16T01:34:33.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.523+0000 7f1bf09f1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1bec071970 con 0x7f1bd8003b70 2024-09-16T01:34:33.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.523+0000 7f1bf09f1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1bec06fd00 con 0x7f1bec154690 2024-09-16T01:34:33.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.523+0000 7f1bead9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bec156200 0x7f1bec1566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:33.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.523+0000 7f1bead9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bec156200 0x7f1bec1566b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55196/0 (socket says 172.21.15.88:55196) 2024-09-16T01:34:33.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.523+0000 7f1bead9d640 1 -- 172.21.15.88:0/4093619977 learned_addr learned my addr 172.21.15.88:0/4093619977 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:33.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.523+0000 7f1be9d9b640 1 --2- 172.21.15.88:0/4093619977 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bec154690 0x7f1bec152bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:33.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.524+0000 7f1bead9d640 1 -- 172.21.15.88:0/4093619977 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bec154690 msgr2=0x7f1bec152bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.524+0000 7f1bea59c640 1 --2- 172.21.15.88:0/4093619977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bd8003b70 0x7f1bec155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:33.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.524+0000 7f1bead9d640 1 --2- 172.21.15.88:0/4093619977 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bec154690 0x7f1bec152bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.524+0000 7f1bead9d640 1 -- 172.21.15.88:0/4093619977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bd8003b70 msgr2=0x7f1bec155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.524+0000 7f1bead9d640 1 --2- 172.21.15.88:0/4093619977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bd8003b70 0x7f1bec155960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.524+0000 7f1bead9d640 1 -- 172.21.15.88:0/4093619977 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bec154b90 con 0x7f1bec156200 2024-09-16T01:34:33.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.524+0000 7f1bead9d640 1 --2- 172.21.15.88:0/4093619977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bec156200 0x7f1bec1566b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1be4001780 tx=0x7f1be4007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:33.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.524+0000 7f1be2ffd640 1 -- 172.21.15.88:0/4093619977 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be400eb00 con 0x7f1bec156200 2024-09-16T01:34:33.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.524+0000 7f1be2ffd640 1 -- 172.21.15.88:0/4093619977 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1be4018070 con 0x7f1bec156200 2024-09-16T01:34:33.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.524+0000 7f1bf09f1640 1 -- 172.21.15.88:0/4093619977 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bec154e70 con 0x7f1bec156200 2024-09-16T01:34:33.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.524+0000 7f1be2ffd640 1 -- 172.21.15.88:0/4093619977 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be4013420 con 0x7f1bec156200 2024-09-16T01:34:33.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.524+0000 7f1bf09f1640 1 -- 172.21.15.88:0/4093619977 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bec15e5a0 con 0x7f1bec156200 2024-09-16T01:34:33.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.525+0000 7f1bf09f1640 1 -- 172.21.15.88:0/4093619977 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bec0700d0 con 0x7f1bec156200 2024-09-16T01:34:33.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.525+0000 7f1be2ffd640 1 -- 172.21.15.88:0/4093619977 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1be40135c0 con 0x7f1bec156200 2024-09-16T01:34:33.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.526+0000 7f1be2ffd640 1 --2- 172.21.15.88:0/4093619977 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1bcc06c2e0 0x7f1bcc06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:33.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.526+0000 7f1bea59c640 1 --2- 172.21.15.88:0/4093619977 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1bcc06c2e0 0x7f1bcc06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:33.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.526+0000 7f1be2ffd640 1 -- 172.21.15.88:0/4093619977 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1be408b600 con 0x7f1bec156200 2024-09-16T01:34:33.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.526+0000 7f1bea59c640 1 --2- 172.21.15.88:0/4093619977 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1bcc06c2e0 0x7f1bcc06e7a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4033000 tx=0x7f1bd4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:33.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.530+0000 7f1be2ffd640 1 -- 172.21.15.88:0/4093619977 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1be4059d70 con 0x7f1bec156200 2024-09-16T01:34:33.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.691+0000 7f1bf09f1640 1 -- 172.21.15.88:0/4093619977 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1bec0b19b0 con 0x7f1bcc06c2e0 2024-09-16T01:34:33.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.697+0000 7f1be2ffd640 1 -- 172.21.15.88:0/4093619977 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251837 (secure 0 0 0) 0x7f1bec0b19b0 con 0x7f1bcc06c2e0 2024-09-16T01:34:33.702 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:33.703 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:34:33.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.700+0000 7f1bf09f1640 1 -- 172.21.15.88:0/4093619977 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1bcc06c2e0 msgr2=0x7f1bcc06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.700+0000 7f1bf09f1640 1 --2- 172.21.15.88:0/4093619977 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1bcc06c2e0 0x7f1bcc06e7a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4033000 tx=0x7f1bd4008030 comp rx=0 tx=0).stop 2024-09-16T01:34:33.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.700+0000 7f1bf09f1640 1 -- 172.21.15.88:0/4093619977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bec156200 msgr2=0x7f1bec1566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.700+0000 7f1bf09f1640 1 --2- 172.21.15.88:0/4093619977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bec156200 0x7f1bec1566b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1be4001780 tx=0x7f1be4007bc0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.700+0000 7f1bf09f1640 1 -- 172.21.15.88:0/4093619977 shutdown_connections 2024-09-16T01:34:33.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.700+0000 7f1bf09f1640 1 --2- 172.21.15.88:0/4093619977 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1bcc06c2e0 0x7f1bcc06e7a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.700+0000 7f1bf09f1640 1 --2- 172.21.15.88:0/4093619977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bec156200 0x7f1bec1566b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.700+0000 7f1bf09f1640 1 --2- 172.21.15.88:0/4093619977 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bec154690 0x7f1bec152bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.700+0000 7f1bf09f1640 1 --2- 172.21.15.88:0/4093619977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bd8003b70 0x7f1bec155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.700+0000 7f1bf09f1640 1 -- 172.21.15.88:0/4093619977 >> 172.21.15.88:0/4093619977 conn(0x7f1bec0d2450 msgr2=0x7f1bec0b3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:33.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.700+0000 7f1bf09f1640 1 -- 172.21.15.88:0/4093619977 shutdown_connections 2024-09-16T01:34:33.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.700+0000 7f1bf09f1640 1 -- 172.21.15.88:0/4093619977 wait complete. 2024-09-16T01:34:33.713 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-09-16T01:34:32.519087+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219148,"kb_used_data":2888,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755572,"statfs":{"total":773094113280,"available":772869705728,"internally_reserved":0,"allocated":2957312,"data_stored":1775436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001920"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611437+0000","last_change":"2024-09-16T01:34:22.195219+0000","last_active":"2024-09-16T01:34:23.611437+0000","last_peered":"2024-09-16T01:34:23.611437+0000","last_clean":"2024-09-16T01:34:23.611437+0000","last_became_active":"2024-09-16T01:34:22.194926+0000","last_became_peered":"2024-09-16T01:34:22.194926+0000","last_unstale":"2024-09-16T01:34:23.611437+0000","last_undegraded":"2024-09-16T01:34:23.611437+0000","last_fullsized":"2024-09-16T01:34:23.611437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:02.738433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513528+0000","last_change":"2024-09-16T01:34:22.186600+0000","last_active":"2024-09-16T01:34:23.513528+0000","last_peered":"2024-09-16T01:34:23.513528+0000","last_clean":"2024-09-16T01:34:23.513528+0000","last_became_active":"2024-09-16T01:34:22.186325+0000","last_became_peered":"2024-09-16T01:34:22.186325+0000","last_unstale":"2024-09-16T01:34:23.513528+0000","last_undegraded":"2024-09-16T01:34:23.513528+0000","last_fullsized":"2024-09-16T01:34:23.513528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:35:45.862526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609611+0000","last_change":"2024-09-16T01:34:22.187744+0000","last_active":"2024-09-16T01:34:23.609611+0000","last_peered":"2024-09-16T01:34:23.609611+0000","last_clean":"2024-09-16T01:34:23.609611+0000","last_became_active":"2024-09-16T01:34:22.186002+0000","last_became_peered":"2024-09-16T01:34:22.186002+0000","last_unstale":"2024-09-16T01:34:23.609611+0000","last_undegraded":"2024-09-16T01:34:23.609611+0000","last_fullsized":"2024-09-16T01:34:23.609611+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:38:50.126366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610430+0000","last_change":"2024-09-16T01:34:22.194231+0000","last_active":"2024-09-16T01:34:23.610430+0000","last_peered":"2024-09-16T01:34:23.610430+0000","last_clean":"2024-09-16T01:34:23.610430+0000","last_became_active":"2024-09-16T01:34:22.193969+0000","last_became_peered":"2024-09-16T01:34:22.193969+0000","last_unstale":"2024-09-16T01:34:23.610430+0000","last_undegraded":"2024-09-16T01:34:23.610430+0000","last_fullsized":"2024-09-16T01:34:23.610430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:35:45.862526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611192+0000","last_change":"2024-09-16T01:34:22.191565+0000","last_active":"2024-09-16T01:34:23.611192+0000","last_peered":"2024-09-16T01:34:23.611192+0000","last_clean":"2024-09-16T01:34:23.611192+0000","last_became_active":"2024-09-16T01:34:22.191281+0000","last_became_peered":"2024-09-16T01:34:22.191281+0000","last_unstale":"2024-09-16T01:34:23.611192+0000","last_undegraded":"2024-09-16T01:34:23.611192+0000","last_fullsized":"2024-09-16T01:34:23.611192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:49.814244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610462+0000","last_change":"2024-09-16T01:34:22.199917+0000","last_active":"2024-09-16T01:34:23.610462+0000","last_peered":"2024-09-16T01:34:23.610462+0000","last_clean":"2024-09-16T01:34:23.610462+0000","last_became_active":"2024-09-16T01:34:22.198949+0000","last_became_peered":"2024-09-16T01:34:22.198949+0000","last_unstale":"2024-09-16T01:34:23.610462+0000","last_undegraded":"2024-09-16T01:34:23.610462+0000","last_fullsized":"2024-09-16T01:34:23.610462+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:48:44.596067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610965+0000","last_change":"2024-09-16T01:34:22.191158+0000","last_active":"2024-09-16T01:34:23.610965+0000","last_peered":"2024-09-16T01:34:23.610965+0000","last_clean":"2024-09-16T01:34:23.610965+0000","last_became_active":"2024-09-16T01:34:22.190856+0000","last_became_peered":"2024-09-16T01:34:22.190856+0000","last_unstale":"2024-09-16T01:34:23.610965+0000","last_undegraded":"2024-09-16T01:34:23.610965+0000","last_fullsized":"2024-09-16T01:34:23.610965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:05.939767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.187946+0000","last_change":"2024-09-16T01:34:22.200461+0000","last_active":"2024-09-16T01:34:23.187946+0000","last_peered":"2024-09-16T01:34:23.187946+0000","last_clean":"2024-09-16T01:34:23.187946+0000","last_became_active":"2024-09-16T01:34:22.200181+0000","last_became_peered":"2024-09-16T01:34:22.200181+0000","last_unstale":"2024-09-16T01:34:23.187946+0000","last_undegraded":"2024-09-16T01:34:23.187946+0000","last_fullsized":"2024-09-16T01:34:23.187946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:36:03.508489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611582+0000","last_change":"2024-09-16T01:34:22.201897+0000","last_active":"2024-09-16T01:34:23.611582+0000","last_peered":"2024-09-16T01:34:23.611582+0000","last_clean":"2024-09-16T01:34:23.611582+0000","last_became_active":"2024-09-16T01:34:22.201595+0000","last_became_peered":"2024-09-16T01:34:22.201595+0000","last_unstale":"2024-09-16T01:34:23.611582+0000","last_undegraded":"2024-09-16T01:34:23.611582+0000","last_fullsized":"2024-09-16T01:34:23.611582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:07.782481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191787+0000","last_change":"2024-09-16T01:34:22.195243+0000","last_active":"2024-09-16T01:34:23.191787+0000","last_peered":"2024-09-16T01:34:23.191787+0000","last_clean":"2024-09-16T01:34:23.191787+0000","last_became_active":"2024-09-16T01:34:22.194813+0000","last_became_peered":"2024-09-16T01:34:22.194813+0000","last_unstale":"2024-09-16T01:34:23.191787+0000","last_undegraded":"2024-09-16T01:34:23.191787+0000","last_fullsized":"2024-09-16T01:34:23.191787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:11:56.336591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191528+0000","last_change":"2024-09-16T01:34:22.197385+0000","last_active":"2024-09-16T01:34:23.191528+0000","last_peered":"2024-09-16T01:34:23.191528+0000","last_clean":"2024-09-16T01:34:23.191528+0000","last_became_active":"2024-09-16T01:34:22.196747+0000","last_became_peered":"2024-09-16T01:34:22.196747+0000","last_unstale":"2024-09-16T01:34:23.191528+0000","last_undegraded":"2024-09-16T01:34:23.191528+0000","last_fullsized":"2024-09-16T01:34:23.191528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:14:18.368513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.190899+0000","last_change":"2024-09-16T01:34:22.181901+0000","last_active":"2024-09-16T01:34:23.190899+0000","last_peered":"2024-09-16T01:34:23.190899+0000","last_clean":"2024-09-16T01:34:23.190899+0000","last_became_active":"2024-09-16T01:34:22.181627+0000","last_became_peered":"2024-09-16T01:34:22.181627+0000","last_unstale":"2024-09-16T01:34:23.190899+0000","last_undegraded":"2024-09-16T01:34:23.190899+0000","last_fullsized":"2024-09-16T01:34:23.190899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:31:42.191103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188450+0000","last_change":"2024-09-16T01:34:22.192960+0000","last_active":"2024-09-16T01:34:23.188450+0000","last_peered":"2024-09-16T01:34:23.188450+0000","last_clean":"2024-09-16T01:34:23.188450+0000","last_became_active":"2024-09-16T01:34:22.192664+0000","last_became_peered":"2024-09-16T01:34:22.192664+0000","last_unstale":"2024-09-16T01:34:23.188450+0000","last_undegraded":"2024-09-16T01:34:23.188450+0000","last_fullsized":"2024-09-16T01:34:23.188450+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:56:56.990459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609665+0000","last_change":"2024-09-16T01:34:22.192223+0000","last_active":"2024-09-16T01:34:23.609665+0000","last_peered":"2024-09-16T01:34:23.609665+0000","last_clean":"2024-09-16T01:34:23.609665+0000","last_became_active":"2024-09-16T01:34:22.190787+0000","last_became_peered":"2024-09-16T01:34:22.190787+0000","last_unstale":"2024-09-16T01:34:23.609665+0000","last_undegraded":"2024-09-16T01:34:23.609665+0000","last_fullsized":"2024-09-16T01:34:23.609665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:40:27.589986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513177+0000","last_change":"2024-09-16T01:34:22.194793+0000","last_active":"2024-09-16T01:34:23.513177+0000","last_peered":"2024-09-16T01:34:23.513177+0000","last_clean":"2024-09-16T01:34:23.513177+0000","last_became_active":"2024-09-16T01:34:22.194530+0000","last_became_peered":"2024-09-16T01:34:22.194530+0000","last_unstale":"2024-09-16T01:34:23.513177+0000","last_undegraded":"2024-09-16T01:34:23.513177+0000","last_fullsized":"2024-09-16T01:34:23.513177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:10:23.471025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610102+0000","last_change":"2024-09-16T01:34:22.198482+0000","last_active":"2024-09-16T01:34:23.610102+0000","last_peered":"2024-09-16T01:34:23.610102+0000","last_clean":"2024-09-16T01:34:23.610102+0000","last_became_active":"2024-09-16T01:34:22.198216+0000","last_became_peered":"2024-09-16T01:34:22.198216+0000","last_unstale":"2024-09-16T01:34:23.610102+0000","last_undegraded":"2024-09-16T01:34:23.610102+0000","last_fullsized":"2024-09-16T01:34:23.610102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:04.740799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610673+0000","last_change":"2024-09-16T01:34:22.198300+0000","last_active":"2024-09-16T01:34:23.610673+0000","last_peered":"2024-09-16T01:34:23.610673+0000","last_clean":"2024-09-16T01:34:23.610673+0000","last_became_active":"2024-09-16T01:34:22.198024+0000","last_became_peered":"2024-09-16T01:34:22.198024+0000","last_unstale":"2024-09-16T01:34:23.610673+0000","last_undegraded":"2024-09-16T01:34:23.610673+0000","last_fullsized":"2024-09-16T01:34:23.610673+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:49.814244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609788+0000","last_change":"2024-09-16T01:34:22.194454+0000","last_active":"2024-09-16T01:34:23.609788+0000","last_peered":"2024-09-16T01:34:23.609788+0000","last_clean":"2024-09-16T01:34:23.609788+0000","last_became_active":"2024-09-16T01:34:22.193936+0000","last_became_peered":"2024-09-16T01:34:22.193936+0000","last_unstale":"2024-09-16T01:34:23.609788+0000","last_undegraded":"2024-09-16T01:34:23.609788+0000","last_fullsized":"2024-09-16T01:34:23.609788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:42:44.444634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611563+0000","last_change":"2024-09-16T01:34:22.191670+0000","last_active":"2024-09-16T01:34:23.611563+0000","last_peered":"2024-09-16T01:34:23.611563+0000","last_clean":"2024-09-16T01:34:23.611563+0000","last_became_active":"2024-09-16T01:34:22.191341+0000","last_became_peered":"2024-09-16T01:34:22.191341+0000","last_unstale":"2024-09-16T01:34:23.611563+0000","last_undegraded":"2024-09-16T01:34:23.611563+0000","last_fullsized":"2024-09-16T01:34:23.611563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:50:20.263787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191662+0000","last_change":"2024-09-16T01:34:22.193121+0000","last_active":"2024-09-16T01:34:23.191662+0000","last_peered":"2024-09-16T01:34:23.191662+0000","last_clean":"2024-09-16T01:34:23.191662+0000","last_became_active":"2024-09-16T01:34:22.192835+0000","last_became_peered":"2024-09-16T01:34:22.192835+0000","last_unstale":"2024-09-16T01:34:23.191662+0000","last_undegraded":"2024-09-16T01:34:23.191662+0000","last_fullsized":"2024-09-16T01:34:23.191662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:55:12.145787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609601+0000","last_change":"2024-09-16T01:34:22.196878+0000","last_active":"2024-09-16T01:34:23.609601+0000","last_peered":"2024-09-16T01:34:23.609601+0000","last_clean":"2024-09-16T01:34:23.609601+0000","last_became_active":"2024-09-16T01:34:22.196451+0000","last_became_peered":"2024-09-16T01:34:22.196451+0000","last_unstale":"2024-09-16T01:34:23.609601+0000","last_undegraded":"2024-09-16T01:34:23.609601+0000","last_fullsized":"2024-09-16T01:34:23.609601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:42:01.118172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610529+0000","last_change":"2024-09-16T01:34:22.196889+0000","last_active":"2024-09-16T01:34:23.610529+0000","last_peered":"2024-09-16T01:34:23.610529+0000","last_clean":"2024-09-16T01:34:23.610529+0000","last_became_active":"2024-09-16T01:34:22.196583+0000","last_became_peered":"2024-09-16T01:34:22.196583+0000","last_unstale":"2024-09-16T01:34:23.610529+0000","last_undegraded":"2024-09-16T01:34:23.610529+0000","last_fullsized":"2024-09-16T01:34:23.610529+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:05.939767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188009+0000","last_change":"2024-09-16T01:34:22.196090+0000","last_active":"2024-09-16T01:34:23.188009+0000","last_peered":"2024-09-16T01:34:23.188009+0000","last_clean":"2024-09-16T01:34:23.188009+0000","last_became_active":"2024-09-16T01:34:22.195441+0000","last_became_peered":"2024-09-16T01:34:22.195441+0000","last_unstale":"2024-09-16T01:34:23.188009+0000","last_undegraded":"2024-09-16T01:34:23.188009+0000","last_fullsized":"2024-09-16T01:34:23.188009+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:14:18.368513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610192+0000","last_change":"2024-09-16T01:34:22.192211+0000","last_active":"2024-09-16T01:34:23.610192+0000","last_peered":"2024-09-16T01:34:23.610192+0000","last_clean":"2024-09-16T01:34:23.610192+0000","last_became_active":"2024-09-16T01:34:22.191921+0000","last_became_peered":"2024-09-16T01:34:22.191921+0000","last_unstale":"2024-09-16T01:34:23.610192+0000","last_undegraded":"2024-09-16T01:34:23.610192+0000","last_fullsized":"2024-09-16T01:34:23.610192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:37.582768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188086+0000","last_change":"2024-09-16T01:34:22.195710+0000","last_active":"2024-09-16T01:34:23.188086+0000","last_peered":"2024-09-16T01:34:23.188086+0000","last_clean":"2024-09-16T01:34:23.188086+0000","last_became_active":"2024-09-16T01:34:22.195385+0000","last_became_peered":"2024-09-16T01:34:22.195385+0000","last_unstale":"2024-09-16T01:34:23.188086+0000","last_undegraded":"2024-09-16T01:34:23.188086+0000","last_fullsized":"2024-09-16T01:34:23.188086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:32:14.878649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191846+0000","last_change":"2024-09-16T01:34:22.193276+0000","last_active":"2024-09-16T01:34:23.191846+0000","last_peered":"2024-09-16T01:34:23.191846+0000","last_clean":"2024-09-16T01:34:23.191846+0000","last_became_active":"2024-09-16T01:34:22.192973+0000","last_became_peered":"2024-09-16T01:34:22.192973+0000","last_unstale":"2024-09-16T01:34:23.191846+0000","last_undegraded":"2024-09-16T01:34:23.191846+0000","last_fullsized":"2024-09-16T01:34:23.191846+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:16:41.426352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513373+0000","last_change":"2024-09-16T01:34:22.189813+0000","last_active":"2024-09-16T01:34:23.513373+0000","last_peered":"2024-09-16T01:34:23.513373+0000","last_clean":"2024-09-16T01:34:23.513373+0000","last_became_active":"2024-09-16T01:34:22.189535+0000","last_became_peered":"2024-09-16T01:34:22.189535+0000","last_unstale":"2024-09-16T01:34:23.513373+0000","last_undegraded":"2024-09-16T01:34:23.513373+0000","last_fullsized":"2024-09-16T01:34:23.513373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:37.582768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188382+0000","last_change":"2024-09-16T01:34:22.195577+0000","last_active":"2024-09-16T01:34:23.188382+0000","last_peered":"2024-09-16T01:34:23.188382+0000","last_clean":"2024-09-16T01:34:23.188382+0000","last_became_active":"2024-09-16T01:34:22.195271+0000","last_became_peered":"2024-09-16T01:34:22.195271+0000","last_unstale":"2024-09-16T01:34:23.188382+0000","last_undegraded":"2024-09-16T01:34:23.188382+0000","last_fullsized":"2024-09-16T01:34:23.188382+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:13.825111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609899+0000","last_change":"2024-09-16T01:34:22.189995+0000","last_active":"2024-09-16T01:34:23.609899+0000","last_peered":"2024-09-16T01:34:23.609899+0000","last_clean":"2024-09-16T01:34:23.609899+0000","last_became_active":"2024-09-16T01:34:22.187919+0000","last_became_peered":"2024-09-16T01:34:22.187919+0000","last_unstale":"2024-09-16T01:34:23.609899+0000","last_undegraded":"2024-09-16T01:34:23.609899+0000","last_fullsized":"2024-09-16T01:34:23.609899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:14:18.368513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610796+0000","last_change":"2024-09-16T01:34:22.194365+0000","last_active":"2024-09-16T01:34:23.610796+0000","last_peered":"2024-09-16T01:34:23.610796+0000","last_clean":"2024-09-16T01:34:23.610796+0000","last_became_active":"2024-09-16T01:34:22.194094+0000","last_became_peered":"2024-09-16T01:34:22.194094+0000","last_unstale":"2024-09-16T01:34:23.610796+0000","last_undegraded":"2024-09-16T01:34:23.610796+0000","last_fullsized":"2024-09-16T01:34:23.610796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:02.738433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609476+0000","last_change":"2024-09-16T01:34:22.195821+0000","last_active":"2024-09-16T01:34:23.609476+0000","last_peered":"2024-09-16T01:34:23.609476+0000","last_clean":"2024-09-16T01:34:23.609476+0000","last_became_active":"2024-09-16T01:34:22.194968+0000","last_became_peered":"2024-09-16T01:34:22.194968+0000","last_unstale":"2024-09-16T01:34:23.609476+0000","last_undegraded":"2024-09-16T01:34:23.609476+0000","last_fullsized":"2024-09-16T01:34:23.609476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:15:36.858098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191308+0000","last_change":"2024-09-16T01:34:22.189557+0000","last_active":"2024-09-16T01:34:23.191308+0000","last_peered":"2024-09-16T01:34:23.191308+0000","last_clean":"2024-09-16T01:34:23.191308+0000","last_became_active":"2024-09-16T01:34:22.189252+0000","last_became_peered":"2024-09-16T01:34:22.189252+0000","last_unstale":"2024-09-16T01:34:23.191308+0000","last_undegraded":"2024-09-16T01:34:23.191308+0000","last_fullsized":"2024-09-16T01:34:23.191308+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:40:27.589986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.190446+0000","last_change":"2024-09-16T01:34:22.187071+0000","last_active":"2024-09-16T01:34:23.190446+0000","last_peered":"2024-09-16T01:34:23.190446+0000","last_clean":"2024-09-16T01:34:23.190446+0000","last_became_active":"2024-09-16T01:34:22.186773+0000","last_became_peered":"2024-09-16T01:34:22.186773+0000","last_unstale":"2024-09-16T01:34:23.190446+0000","last_undegraded":"2024-09-16T01:34:23.190446+0000","last_fullsized":"2024-09-16T01:34:23.190446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:22:27.895343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188539+0000","last_change":"2024-09-16T01:34:21.187730+0000","last_active":"2024-09-16T01:34:23.188539+0000","last_peered":"2024-09-16T01:34:23.188539+0000","last_clean":"2024-09-16T01:34:23.188539+0000","last_became_active":"2024-09-16T01:34:21.187421+0000","last_became_peered":"2024-09-16T01:34:21.187421+0000","last_unstale":"2024-09-16T01:34:23.188539+0000","last_undegraded":"2024-09-16T01:34:23.188539+0000","last_fullsized":"2024-09-16T01:34:23.188539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:56.560008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609871+0000","last_change":"2024-09-16T01:34:22.197079+0000","last_active":"2024-09-16T01:34:23.609871+0000","last_peered":"2024-09-16T01:34:23.609871+0000","last_clean":"2024-09-16T01:34:23.609871+0000","last_became_active":"2024-09-16T01:34:22.196793+0000","last_became_peered":"2024-09-16T01:34:22.196793+0000","last_unstale":"2024-09-16T01:34:23.609871+0000","last_undegraded":"2024-09-16T01:34:23.609871+0000","last_fullsized":"2024-09-16T01:34:23.609871+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:16:41.426352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191283+0000","last_change":"2024-09-16T01:34:21.188731+0000","last_active":"2024-09-16T01:34:23.191283+0000","last_peered":"2024-09-16T01:34:23.191283+0000","last_clean":"2024-09-16T01:34:23.191283+0000","last_became_active":"2024-09-16T01:34:21.188477+0000","last_became_peered":"2024-09-16T01:34:21.188477+0000","last_unstale":"2024-09-16T01:34:23.191283+0000","last_undegraded":"2024-09-16T01:34:23.191283+0000","last_fullsized":"2024-09-16T01:34:23.191283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:58:12.138025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188059+0000","last_change":"2024-09-16T01:34:21.187868+0000","last_active":"2024-09-16T01:34:23.188059+0000","last_peered":"2024-09-16T01:34:23.188059+0000","last_clean":"2024-09-16T01:34:23.188059+0000","last_became_active":"2024-09-16T01:34:21.187558+0000","last_became_peered":"2024-09-16T01:34:21.187558+0000","last_unstale":"2024-09-16T01:34:23.188059+0000","last_undegraded":"2024-09-16T01:34:23.188059+0000","last_fullsized":"2024-09-16T01:34:23.188059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:58:12.138025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188157+0000","last_change":"2024-09-16T01:34:22.195132+0000","last_active":"2024-09-16T01:34:23.188157+0000","last_peered":"2024-09-16T01:34:23.188157+0000","last_clean":"2024-09-16T01:34:23.188157+0000","last_became_active":"2024-09-16T01:34:22.194861+0000","last_became_peered":"2024-09-16T01:34:22.194861+0000","last_unstale":"2024-09-16T01:34:23.188157+0000","last_undegraded":"2024-09-16T01:34:23.188157+0000","last_fullsized":"2024-09-16T01:34:23.188157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:11:56.336591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610304+0000","last_change":"2024-09-16T01:34:21.188039+0000","last_active":"2024-09-16T01:34:23.610304+0000","last_peered":"2024-09-16T01:34:23.610304+0000","last_clean":"2024-09-16T01:34:23.610304+0000","last_became_active":"2024-09-16T01:34:21.187743+0000","last_became_peered":"2024-09-16T01:34:21.187743+0000","last_unstale":"2024-09-16T01:34:23.610304+0000","last_undegraded":"2024-09-16T01:34:23.610304+0000","last_fullsized":"2024-09-16T01:34:23.610304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:50:53.737531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513172+0000","last_change":"2024-09-16T01:34:22.187569+0000","last_active":"2024-09-16T01:34:23.513172+0000","last_peered":"2024-09-16T01:34:23.513172+0000","last_clean":"2024-09-16T01:34:23.513172+0000","last_became_active":"2024-09-16T01:34:22.187301+0000","last_became_peered":"2024-09-16T01:34:22.187301+0000","last_unstale":"2024-09-16T01:34:23.513172+0000","last_undegraded":"2024-09-16T01:34:23.513172+0000","last_fullsized":"2024-09-16T01:34:23.513172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:46:31.545461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191499+0000","last_change":"2024-09-16T01:34:21.187827+0000","last_active":"2024-09-16T01:34:23.191499+0000","last_peered":"2024-09-16T01:34:23.191499+0000","last_clean":"2024-09-16T01:34:23.191499+0000","last_became_active":"2024-09-16T01:34:21.187542+0000","last_became_peered":"2024-09-16T01:34:21.187542+0000","last_unstale":"2024-09-16T01:34:23.191499+0000","last_undegraded":"2024-09-16T01:34:23.191499+0000","last_fullsized":"2024-09-16T01:34:23.191499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:18:50.444668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610494+0000","last_change":"2024-09-16T01:34:22.197484+0000","last_active":"2024-09-16T01:34:23.610494+0000","last_peered":"2024-09-16T01:34:23.610494+0000","last_clean":"2024-09-16T01:34:23.610494+0000","last_became_active":"2024-09-16T01:34:22.197209+0000","last_became_peered":"2024-09-16T01:34:22.197209+0000","last_unstale":"2024-09-16T01:34:23.610494+0000","last_undegraded":"2024-09-16T01:34:23.610494+0000","last_fullsized":"2024-09-16T01:34:23.610494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:06.384195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610564+0000","last_change":"2024-09-16T01:34:21.190619+0000","last_active":"2024-09-16T01:34:23.610564+0000","last_peered":"2024-09-16T01:34:23.610564+0000","last_clean":"2024-09-16T01:34:23.610564+0000","last_became_active":"2024-09-16T01:34:21.188943+0000","last_became_peered":"2024-09-16T01:34:21.188943+0000","last_unstale":"2024-09-16T01:34:23.610564+0000","last_undegraded":"2024-09-16T01:34:23.610564+0000","last_fullsized":"2024-09-16T01:34:23.610564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:47:09.976612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609914+0000","last_change":"2024-09-16T01:34:22.198543+0000","last_active":"2024-09-16T01:34:23.609914+0000","last_peered":"2024-09-16T01:34:23.609914+0000","last_clean":"2024-09-16T01:34:23.609914+0000","last_became_active":"2024-09-16T01:34:22.198283+0000","last_became_peered":"2024-09-16T01:34:22.198283+0000","last_unstale":"2024-09-16T01:34:23.609914+0000","last_undegraded":"2024-09-16T01:34:23.609914+0000","last_fullsized":"2024-09-16T01:34:23.609914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:32:05.208969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513419+0000","last_change":"2024-09-16T01:34:21.189485+0000","last_active":"2024-09-16T01:34:23.513419+0000","last_peered":"2024-09-16T01:34:23.513419+0000","last_clean":"2024-09-16T01:34:23.513419+0000","last_became_active":"2024-09-16T01:34:21.189201+0000","last_became_peered":"2024-09-16T01:34:21.189201+0000","last_unstale":"2024-09-16T01:34:23.513419+0000","last_undegraded":"2024-09-16T01:34:23.513419+0000","last_fullsized":"2024-09-16T01:34:23.513419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:01:27.300067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.190949+0000","last_change":"2024-09-16T01:34:22.194435+0000","last_active":"2024-09-16T01:34:23.190949+0000","last_peered":"2024-09-16T01:34:23.190949+0000","last_clean":"2024-09-16T01:34:23.190949+0000","last_became_active":"2024-09-16T01:34:22.194058+0000","last_became_peered":"2024-09-16T01:34:22.194058+0000","last_unstale":"2024-09-16T01:34:23.190949+0000","last_undegraded":"2024-09-16T01:34:23.190949+0000","last_fullsized":"2024-09-16T01:34:23.190949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:36:45.083401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191725+0000","last_change":"2024-09-16T01:34:21.185553+0000","last_active":"2024-09-16T01:34:23.191725+0000","last_peered":"2024-09-16T01:34:23.191725+0000","last_clean":"2024-09-16T01:34:23.191725+0000","last_became_active":"2024-09-16T01:34:21.185264+0000","last_became_peered":"2024-09-16T01:34:21.185264+0000","last_unstale":"2024-09-16T01:34:23.191725+0000","last_undegraded":"2024-09-16T01:34:23.191725+0000","last_fullsized":"2024-09-16T01:34:23.191725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:22:26.887709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611437+0000","last_change":"2024-09-16T01:34:22.191352+0000","last_active":"2024-09-16T01:34:23.611437+0000","last_peered":"2024-09-16T01:34:23.611437+0000","last_clean":"2024-09-16T01:34:23.611437+0000","last_became_active":"2024-09-16T01:34:22.191027+0000","last_became_peered":"2024-09-16T01:34:22.191027+0000","last_unstale":"2024-09-16T01:34:23.611437+0000","last_undegraded":"2024-09-16T01:34:23.611437+0000","last_fullsized":"2024-09-16T01:34:23.611437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:18.860118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609920+0000","last_change":"2024-09-16T01:34:21.193072+0000","last_active":"2024-09-16T01:34:23.609920+0000","last_peered":"2024-09-16T01:34:23.609920+0000","last_clean":"2024-09-16T01:34:23.609920+0000","last_became_active":"2024-09-16T01:34:21.192769+0000","last_became_peered":"2024-09-16T01:34:21.192769+0000","last_unstale":"2024-09-16T01:34:23.609920+0000","last_undegraded":"2024-09-16T01:34:23.609920+0000","last_fullsized":"2024-09-16T01:34:23.609920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:00.162566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609665+0000","last_change":"2024-09-16T01:34:22.195668+0000","last_active":"2024-09-16T01:34:23.609665+0000","last_peered":"2024-09-16T01:34:23.609665+0000","last_clean":"2024-09-16T01:34:23.609665+0000","last_became_active":"2024-09-16T01:34:22.195407+0000","last_became_peered":"2024-09-16T01:34:22.195407+0000","last_unstale":"2024-09-16T01:34:23.609665+0000","last_undegraded":"2024-09-16T01:34:23.609665+0000","last_fullsized":"2024-09-16T01:34:23.609665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:36:03.508489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610939+0000","last_change":"2024-09-16T01:34:21.186203+0000","last_active":"2024-09-16T01:34:23.610939+0000","last_peered":"2024-09-16T01:34:23.610939+0000","last_clean":"2024-09-16T01:34:23.610939+0000","last_became_active":"2024-09-16T01:34:21.185935+0000","last_became_peered":"2024-09-16T01:34:21.185935+0000","last_unstale":"2024-09-16T01:34:23.610939+0000","last_undegraded":"2024-09-16T01:34:23.610939+0000","last_fullsized":"2024-09-16T01:34:23.610939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:29.860344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611192+0000","last_change":"2024-09-16T01:34:22.196044+0000","last_active":"2024-09-16T01:34:23.611192+0000","last_peered":"2024-09-16T01:34:23.611192+0000","last_clean":"2024-09-16T01:34:23.611192+0000","last_became_active":"2024-09-16T01:34:22.195783+0000","last_became_peered":"2024-09-16T01:34:22.195783+0000","last_unstale":"2024-09-16T01:34:23.611192+0000","last_undegraded":"2024-09-16T01:34:23.611192+0000","last_fullsized":"2024-09-16T01:34:23.611192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:03.591531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191432+0000","last_change":"2024-09-16T01:34:21.184919+0000","last_active":"2024-09-16T01:34:23.191432+0000","last_peered":"2024-09-16T01:34:23.191432+0000","last_clean":"2024-09-16T01:34:23.191432+0000","last_became_active":"2024-09-16T01:34:21.184542+0000","last_became_peered":"2024-09-16T01:34:21.184542+0000","last_unstale":"2024-09-16T01:34:23.191432+0000","last_undegraded":"2024-09-16T01:34:23.191432+0000","last_fullsized":"2024-09-16T01:34:23.191432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:56.560008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191412+0000","last_change":"2024-09-16T01:34:22.187373+0000","last_active":"2024-09-16T01:34:23.191412+0000","last_peered":"2024-09-16T01:34:23.191412+0000","last_clean":"2024-09-16T01:34:23.191412+0000","last_became_active":"2024-09-16T01:34:22.186468+0000","last_became_peered":"2024-09-16T01:34:22.186468+0000","last_unstale":"2024-09-16T01:34:23.191412+0000","last_undegraded":"2024-09-16T01:34:23.191412+0000","last_fullsized":"2024-09-16T01:34:23.191412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:36:03.508489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609191+0000","last_change":"2024-09-16T01:34:21.189797+0000","last_active":"2024-09-16T01:34:23.609191+0000","last_peered":"2024-09-16T01:34:23.609191+0000","last_clean":"2024-09-16T01:34:23.609191+0000","last_became_active":"2024-09-16T01:34:21.189512+0000","last_became_peered":"2024-09-16T01:34:21.189512+0000","last_unstale":"2024-09-16T01:34:23.609191+0000","last_undegraded":"2024-09-16T01:34:23.609191+0000","last_fullsized":"2024-09-16T01:34:23.609191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:01:27.300067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611069+0000","last_change":"2024-09-16T01:34:22.195318+0000","last_active":"2024-09-16T01:34:23.611069+0000","last_peered":"2024-09-16T01:34:23.611069+0000","last_clean":"2024-09-16T01:34:23.611069+0000","last_became_active":"2024-09-16T01:34:22.195051+0000","last_became_peered":"2024-09-16T01:34:22.195051+0000","last_unstale":"2024-09-16T01:34:23.611069+0000","last_undegraded":"2024-09-16T01:34:23.611069+0000","last_fullsized":"2024-09-16T01:34:23.611069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:06.384195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.190601+0000","last_change":"2024-09-16T01:34:21.191907+0000","last_active":"2024-09-16T01:34:23.190601+0000","last_peered":"2024-09-16T01:34:23.190601+0000","last_clean":"2024-09-16T01:34:23.190601+0000","last_became_active":"2024-09-16T01:34:21.191591+0000","last_became_peered":"2024-09-16T01:34:21.191591+0000","last_unstale":"2024-09-16T01:34:23.190601+0000","last_undegraded":"2024-09-16T01:34:23.190601+0000","last_fullsized":"2024-09-16T01:34:23.190601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:50:28.630517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611504+0000","last_change":"2024-09-16T01:34:22.194591+0000","last_active":"2024-09-16T01:34:23.611504+0000","last_peered":"2024-09-16T01:34:23.611504+0000","last_clean":"2024-09-16T01:34:23.611504+0000","last_became_active":"2024-09-16T01:34:22.194317+0000","last_became_peered":"2024-09-16T01:34:22.194317+0000","last_unstale":"2024-09-16T01:34:23.611504+0000","last_undegraded":"2024-09-16T01:34:23.611504+0000","last_fullsized":"2024-09-16T01:34:23.611504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:45.928615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609751+0000","last_change":"2024-09-16T01:34:21.190706+0000","last_active":"2024-09-16T01:34:23.609751+0000","last_peered":"2024-09-16T01:34:23.609751+0000","last_clean":"2024-09-16T01:34:23.609751+0000","last_became_active":"2024-09-16T01:34:21.189692+0000","last_became_peered":"2024-09-16T01:34:21.189692+0000","last_unstale":"2024-09-16T01:34:23.609751+0000","last_undegraded":"2024-09-16T01:34:23.609751+0000","last_fullsized":"2024-09-16T01:34:23.609751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:11:30.497159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611648+0000","last_change":"2024-09-16T01:34:22.194506+0000","last_active":"2024-09-16T01:34:23.611648+0000","last_peered":"2024-09-16T01:34:23.611648+0000","last_clean":"2024-09-16T01:34:23.611648+0000","last_became_active":"2024-09-16T01:34:22.194223+0000","last_became_peered":"2024-09-16T01:34:22.194223+0000","last_unstale":"2024-09-16T01:34:23.611648+0000","last_undegraded":"2024-09-16T01:34:23.611648+0000","last_fullsized":"2024-09-16T01:34:23.611648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:19.506778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513439+0000","last_change":"2024-09-16T01:34:21.189382+0000","last_active":"2024-09-16T01:34:23.513439+0000","last_peered":"2024-09-16T01:34:23.513439+0000","last_clean":"2024-09-16T01:34:23.513439+0000","last_became_active":"2024-09-16T01:34:21.188897+0000","last_became_peered":"2024-09-16T01:34:21.188897+0000","last_unstale":"2024-09-16T01:34:23.513439+0000","last_undegraded":"2024-09-16T01:34:23.513439+0000","last_fullsized":"2024-09-16T01:34:23.513439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:29.860344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610830+0000","last_change":"2024-09-16T01:34:22.192308+0000","last_active":"2024-09-16T01:34:23.610830+0000","last_peered":"2024-09-16T01:34:23.610830+0000","last_clean":"2024-09-16T01:34:23.610830+0000","last_became_active":"2024-09-16T01:34:22.191748+0000","last_became_peered":"2024-09-16T01:34:22.191748+0000","last_unstale":"2024-09-16T01:34:23.610830+0000","last_undegraded":"2024-09-16T01:34:23.610830+0000","last_fullsized":"2024-09-16T01:34:23.610830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:45.928615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609394+0000","last_change":"2024-09-16T01:34:21.188078+0000","last_active":"2024-09-16T01:34:23.609394+0000","last_peered":"2024-09-16T01:34:23.609394+0000","last_clean":"2024-09-16T01:34:23.609394+0000","last_became_active":"2024-09-16T01:34:21.187822+0000","last_became_peered":"2024-09-16T01:34:21.187822+0000","last_unstale":"2024-09-16T01:34:23.609394+0000","last_undegraded":"2024-09-16T01:34:23.609394+0000","last_fullsized":"2024-09-16T01:34:23.609394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:29.860344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611369+0000","last_change":"2024-09-16T01:34:22.202386+0000","last_active":"2024-09-16T01:34:23.611369+0000","last_peered":"2024-09-16T01:34:23.611369+0000","last_clean":"2024-09-16T01:34:23.611369+0000","last_became_active":"2024-09-16T01:34:22.202089+0000","last_became_peered":"2024-09-16T01:34:22.202089+0000","last_unstale":"2024-09-16T01:34:23.611369+0000","last_undegraded":"2024-09-16T01:34:23.611369+0000","last_fullsized":"2024-09-16T01:34:23.611369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:07.244892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610330+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:34:23.610330+0000","last_peered":"2024-09-16T01:34:23.610330+0000","last_clean":"2024-09-16T01:34:23.610330+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:34:23.610330+0000","last_undegraded":"2024-09-16T01:34:23.610330+0000","last_fullsized":"2024-09-16T01:34:23.610330+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610276+0000","last_change":"2024-09-16T01:34:21.183119+0000","last_active":"2024-09-16T01:34:23.610276+0000","last_peered":"2024-09-16T01:34:23.610276+0000","last_clean":"2024-09-16T01:34:23.610276+0000","last_became_active":"2024-09-16T01:34:21.182740+0000","last_became_peered":"2024-09-16T01:34:21.182740+0000","last_unstale":"2024-09-16T01:34:23.610276+0000","last_undegraded":"2024-09-16T01:34:23.610276+0000","last_fullsized":"2024-09-16T01:34:23.610276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:01:27.300067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188231+0000","last_change":"2024-09-16T01:34:22.202105+0000","last_active":"2024-09-16T01:34:23.188231+0000","last_peered":"2024-09-16T01:34:23.188231+0000","last_clean":"2024-09-16T01:34:23.188231+0000","last_became_active":"2024-09-16T01:34:22.201840+0000","last_became_peered":"2024-09-16T01:34:22.201840+0000","last_unstale":"2024-09-16T01:34:23.188231+0000","last_undegraded":"2024-09-16T01:34:23.188231+0000","last_fullsized":"2024-09-16T01:34:23.188231+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:32:05.208969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610203+0000","last_change":"2024-09-16T01:34:21.190480+0000","last_active":"2024-09-16T01:34:23.610203+0000","last_peered":"2024-09-16T01:34:23.610203+0000","last_clean":"2024-09-16T01:34:23.610203+0000","last_became_active":"2024-09-16T01:34:21.190205+0000","last_became_peered":"2024-09-16T01:34:21.190205+0000","last_unstale":"2024-09-16T01:34:23.610203+0000","last_undegraded":"2024-09-16T01:34:23.610203+0000","last_fullsized":"2024-09-16T01:34:23.610203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:47:09.976612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609791+0000","last_change":"2024-09-16T01:34:22.198514+0000","last_active":"2024-09-16T01:34:23.609791+0000","last_peered":"2024-09-16T01:34:23.609791+0000","last_clean":"2024-09-16T01:34:23.609791+0000","last_became_active":"2024-09-16T01:34:22.197969+0000","last_became_peered":"2024-09-16T01:34:22.197969+0000","last_unstale":"2024-09-16T01:34:23.609791+0000","last_undegraded":"2024-09-16T01:34:23.609791+0000","last_fullsized":"2024-09-16T01:34:23.609791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:32:14.878649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609319+0000","last_change":"2024-09-16T01:34:21.190398+0000","last_active":"2024-09-16T01:34:23.609319+0000","last_peered":"2024-09-16T01:34:23.609319+0000","last_clean":"2024-09-16T01:34:23.609319+0000","last_became_active":"2024-09-16T01:34:21.190141+0000","last_became_peered":"2024-09-16T01:34:21.190141+0000","last_unstale":"2024-09-16T01:34:23.609319+0000","last_undegraded":"2024-09-16T01:34:23.609319+0000","last_fullsized":"2024-09-16T01:34:23.609319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:47:09.976612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513229+0000","last_change":"2024-09-16T01:34:22.187125+0000","last_active":"2024-09-16T01:34:23.513229+0000","last_peered":"2024-09-16T01:34:23.513229+0000","last_clean":"2024-09-16T01:34:23.513229+0000","last_became_active":"2024-09-16T01:34:22.186846+0000","last_became_peered":"2024-09-16T01:34:22.186846+0000","last_unstale":"2024-09-16T01:34:23.513229+0000","last_undegraded":"2024-09-16T01:34:23.513229+0000","last_fullsized":"2024-09-16T01:34:23.513229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:59:47.794894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610948+0000","last_change":"2024-09-16T01:34:21.190750+0000","last_active":"2024-09-16T01:34:23.610948+0000","last_peered":"2024-09-16T01:34:23.610948+0000","last_clean":"2024-09-16T01:34:23.610948+0000","last_became_active":"2024-09-16T01:34:21.189215+0000","last_became_peered":"2024-09-16T01:34:21.189215+0000","last_unstale":"2024-09-16T01:34:23.610948+0000","last_undegraded":"2024-09-16T01:34:23.610948+0000","last_fullsized":"2024-09-16T01:34:23.610948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:11:30.497159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.190850+0000","last_change":"2024-09-16T01:34:22.184566+0000","last_active":"2024-09-16T01:34:23.190850+0000","last_peered":"2024-09-16T01:34:23.190850+0000","last_clean":"2024-09-16T01:34:23.190850+0000","last_became_active":"2024-09-16T01:34:22.184223+0000","last_became_peered":"2024-09-16T01:34:22.184223+0000","last_unstale":"2024-09-16T01:34:23.190850+0000","last_undegraded":"2024-09-16T01:34:23.190850+0000","last_fullsized":"2024-09-16T01:34:23.190850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:33:24.341399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513407+0000","last_change":"2024-09-16T01:34:21.191889+0000","last_active":"2024-09-16T01:34:23.513407+0000","last_peered":"2024-09-16T01:34:23.513407+0000","last_clean":"2024-09-16T01:34:23.513407+0000","last_became_active":"2024-09-16T01:34:21.191594+0000","last_became_peered":"2024-09-16T01:34:21.191594+0000","last_unstale":"2024-09-16T01:34:23.513407+0000","last_undegraded":"2024-09-16T01:34:23.513407+0000","last_fullsized":"2024-09-16T01:34:23.513407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:57.471937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610790+0000","last_change":"2024-09-16T01:34:22.193566+0000","last_active":"2024-09-16T01:34:23.610790+0000","last_peered":"2024-09-16T01:34:23.610790+0000","last_clean":"2024-09-16T01:34:23.610790+0000","last_became_active":"2024-09-16T01:34:22.193305+0000","last_became_peered":"2024-09-16T01:34:22.193305+0000","last_unstale":"2024-09-16T01:34:23.610790+0000","last_undegraded":"2024-09-16T01:34:23.610790+0000","last_fullsized":"2024-09-16T01:34:23.610790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:18.860118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611604+0000","last_change":"2024-09-16T01:34:21.192721+0000","last_active":"2024-09-16T01:34:23.611604+0000","last_peered":"2024-09-16T01:34:23.611604+0000","last_clean":"2024-09-16T01:34:23.611604+0000","last_became_active":"2024-09-16T01:34:21.192423+0000","last_became_peered":"2024-09-16T01:34:21.192423+0000","last_unstale":"2024-09-16T01:34:23.611604+0000","last_undegraded":"2024-09-16T01:34:23.611604+0000","last_fullsized":"2024-09-16T01:34:23.611604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:13.839623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513359+0000","last_change":"2024-09-16T01:34:22.187869+0000","last_active":"2024-09-16T01:34:23.513359+0000","last_peered":"2024-09-16T01:34:23.513359+0000","last_clean":"2024-09-16T01:34:23.513359+0000","last_became_active":"2024-09-16T01:34:22.186507+0000","last_became_peered":"2024-09-16T01:34:22.186507+0000","last_unstale":"2024-09-16T01:34:23.513359+0000","last_undegraded":"2024-09-16T01:34:23.513359+0000","last_fullsized":"2024-09-16T01:34:23.513359+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:04.740799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513369+0000","last_change":"2024-09-16T01:34:21.190145+0000","last_active":"2024-09-16T01:34:23.513369+0000","last_peered":"2024-09-16T01:34:23.513369+0000","last_clean":"2024-09-16T01:34:23.513369+0000","last_became_active":"2024-09-16T01:34:21.189886+0000","last_became_peered":"2024-09-16T01:34:21.189886+0000","last_unstale":"2024-09-16T01:34:23.513369+0000","last_undegraded":"2024-09-16T01:34:23.513369+0000","last_fullsized":"2024-09-16T01:34:23.513369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:47:09.976612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188218+0000","last_change":"2024-09-16T01:34:22.194698+0000","last_active":"2024-09-16T01:34:23.188218+0000","last_peered":"2024-09-16T01:34:23.188218+0000","last_clean":"2024-09-16T01:34:23.188218+0000","last_became_active":"2024-09-16T01:34:22.194443+0000","last_became_peered":"2024-09-16T01:34:22.194443+0000","last_unstale":"2024-09-16T01:34:23.188218+0000","last_undegraded":"2024-09-16T01:34:23.188218+0000","last_fullsized":"2024-09-16T01:34:23.188218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:16:41.426352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191535+0000","last_change":"2024-09-16T01:34:21.189000+0000","last_active":"2024-09-16T01:34:23.191535+0000","last_peered":"2024-09-16T01:34:23.191535+0000","last_clean":"2024-09-16T01:34:23.191535+0000","last_became_active":"2024-09-16T01:34:21.188144+0000","last_became_peered":"2024-09-16T01:34:21.188144+0000","last_unstale":"2024-09-16T01:34:23.191535+0000","last_undegraded":"2024-09-16T01:34:23.191535+0000","last_fullsized":"2024-09-16T01:34:23.191535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:36:22.787904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609862+0000","last_change":"2024-09-16T01:34:22.188803+0000","last_active":"2024-09-16T01:34:23.609862+0000","last_peered":"2024-09-16T01:34:23.609862+0000","last_clean":"2024-09-16T01:34:23.609862+0000","last_became_active":"2024-09-16T01:34:22.188522+0000","last_became_peered":"2024-09-16T01:34:22.188522+0000","last_unstale":"2024-09-16T01:34:23.609862+0000","last_undegraded":"2024-09-16T01:34:23.609862+0000","last_fullsized":"2024-09-16T01:34:23.609862+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:11:56.336591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610066+0000","last_change":"2024-09-16T01:34:21.192126+0000","last_active":"2024-09-16T01:34:23.610066+0000","last_peered":"2024-09-16T01:34:23.610066+0000","last_clean":"2024-09-16T01:34:23.610066+0000","last_became_active":"2024-09-16T01:34:21.191855+0000","last_became_peered":"2024-09-16T01:34:21.191855+0000","last_unstale":"2024-09-16T01:34:23.610066+0000","last_undegraded":"2024-09-16T01:34:23.610066+0000","last_fullsized":"2024-09-16T01:34:23.610066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:57.471937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.190343+0000","last_change":"2024-09-16T01:34:22.186916+0000","last_active":"2024-09-16T01:34:23.190343+0000","last_peered":"2024-09-16T01:34:23.190343+0000","last_clean":"2024-09-16T01:34:23.190343+0000","last_became_active":"2024-09-16T01:34:22.186081+0000","last_became_peered":"2024-09-16T01:34:22.186081+0000","last_unstale":"2024-09-16T01:34:23.190343+0000","last_undegraded":"2024-09-16T01:34:23.190343+0000","last_fullsized":"2024-09-16T01:34:23.190343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:36:23.795538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513225+0000","last_change":"2024-09-16T01:34:21.187955+0000","last_active":"2024-09-16T01:34:23.513225+0000","last_peered":"2024-09-16T01:34:23.513225+0000","last_clean":"2024-09-16T01:34:23.513225+0000","last_became_active":"2024-09-16T01:34:21.187668+0000","last_became_peered":"2024-09-16T01:34:21.187668+0000","last_unstale":"2024-09-16T01:34:23.513225+0000","last_undegraded":"2024-09-16T01:34:23.513225+0000","last_fullsized":"2024-09-16T01:34:23.513225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:11:30.497159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188103+0000","last_change":"2024-09-16T01:34:22.187359+0000","last_active":"2024-09-16T01:34:23.188103+0000","last_peered":"2024-09-16T01:34:23.188103+0000","last_clean":"2024-09-16T01:34:23.188103+0000","last_became_active":"2024-09-16T01:34:22.187088+0000","last_became_peered":"2024-09-16T01:34:22.187088+0000","last_unstale":"2024-09-16T01:34:23.188103+0000","last_undegraded":"2024-09-16T01:34:23.188103+0000","last_fullsized":"2024-09-16T01:34:23.188103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:55:12.145787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513225+0000","last_change":"2024-09-16T01:34:21.184801+0000","last_active":"2024-09-16T01:34:23.513225+0000","last_peered":"2024-09-16T01:34:23.513225+0000","last_clean":"2024-09-16T01:34:23.513225+0000","last_became_active":"2024-09-16T01:34:21.184472+0000","last_became_peered":"2024-09-16T01:34:21.184472+0000","last_unstale":"2024-09-16T01:34:23.513225+0000","last_undegraded":"2024-09-16T01:34:23.513225+0000","last_fullsized":"2024-09-16T01:34:23.513225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:41:15.050218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188586+0000","last_change":"2024-09-16T01:34:22.202201+0000","last_active":"2024-09-16T01:34:23.188586+0000","last_peered":"2024-09-16T01:34:23.188586+0000","last_clean":"2024-09-16T01:34:23.188586+0000","last_became_active":"2024-09-16T01:34:22.201850+0000","last_became_peered":"2024-09-16T01:34:22.201850+0000","last_unstale":"2024-09-16T01:34:23.188586+0000","last_undegraded":"2024-09-16T01:34:23.188586+0000","last_fullsized":"2024-09-16T01:34:23.188586+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:56:15.380999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611287+0000","last_change":"2024-09-16T01:34:21.192632+0000","last_active":"2024-09-16T01:34:23.611287+0000","last_peered":"2024-09-16T01:34:23.611287+0000","last_clean":"2024-09-16T01:34:23.611287+0000","last_became_active":"2024-09-16T01:34:21.192351+0000","last_became_peered":"2024-09-16T01:34:21.192351+0000","last_unstale":"2024-09-16T01:34:23.611287+0000","last_undegraded":"2024-09-16T01:34:23.611287+0000","last_fullsized":"2024-09-16T01:34:23.611287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:15:35.850464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191599+0000","last_change":"2024-09-16T01:34:22.193185+0000","last_active":"2024-09-16T01:34:23.191599+0000","last_peered":"2024-09-16T01:34:23.191599+0000","last_clean":"2024-09-16T01:34:23.191599+0000","last_became_active":"2024-09-16T01:34:22.192869+0000","last_became_peered":"2024-09-16T01:34:22.192869+0000","last_unstale":"2024-09-16T01:34:23.191599+0000","last_undegraded":"2024-09-16T01:34:23.191599+0000","last_fullsized":"2024-09-16T01:34:23.191599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:32:14.878649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610675+0000","last_change":"2024-09-16T01:34:22.200190+0000","last_active":"2024-09-16T01:34:23.610675+0000","last_peered":"2024-09-16T01:34:23.610675+0000","last_clean":"2024-09-16T01:34:23.610675+0000","last_became_active":"2024-09-16T01:34:22.199826+0000","last_became_peered":"2024-09-16T01:34:22.199826+0000","last_unstale":"2024-09-16T01:34:23.610675+0000","last_undegraded":"2024-09-16T01:34:23.610675+0000","last_fullsized":"2024-09-16T01:34:23.610675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:03.591531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611122+0000","last_change":"2024-09-16T01:34:21.188752+0000","last_active":"2024-09-16T01:34:23.611122+0000","last_peered":"2024-09-16T01:34:23.611122+0000","last_clean":"2024-09-16T01:34:23.611122+0000","last_became_active":"2024-09-16T01:34:21.188414+0000","last_became_peered":"2024-09-16T01:34:21.188414+0000","last_unstale":"2024-09-16T01:34:23.611122+0000","last_undegraded":"2024-09-16T01:34:23.611122+0000","last_fullsized":"2024-09-16T01:34:23.611122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:41:15.050218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610726+0000","last_change":"2024-09-16T01:34:22.199939+0000","last_active":"2024-09-16T01:34:23.610726+0000","last_peered":"2024-09-16T01:34:23.610726+0000","last_clean":"2024-09-16T01:34:23.610726+0000","last_became_active":"2024-09-16T01:34:22.199580+0000","last_became_peered":"2024-09-16T01:34:22.199580+0000","last_unstale":"2024-09-16T01:34:23.610726+0000","last_undegraded":"2024-09-16T01:34:23.610726+0000","last_fullsized":"2024-09-16T01:34:23.610726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:07.244892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611491+0000","last_change":"2024-09-16T01:34:21.190410+0000","last_active":"2024-09-16T01:34:23.611491+0000","last_peered":"2024-09-16T01:34:23.611491+0000","last_clean":"2024-09-16T01:34:23.611491+0000","last_became_active":"2024-09-16T01:34:21.190075+0000","last_became_peered":"2024-09-16T01:34:21.190075+0000","last_unstale":"2024-09-16T01:34:23.611491+0000","last_undegraded":"2024-09-16T01:34:23.611491+0000","last_fullsized":"2024-09-16T01:34:23.611491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:42:43.437000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609849+0000","last_change":"2024-09-16T01:34:22.198463+0000","last_active":"2024-09-16T01:34:23.609849+0000","last_peered":"2024-09-16T01:34:23.609849+0000","last_clean":"2024-09-16T01:34:23.609849+0000","last_became_active":"2024-09-16T01:34:22.198214+0000","last_became_peered":"2024-09-16T01:34:22.198214+0000","last_unstale":"2024-09-16T01:34:23.609849+0000","last_undegraded":"2024-09-16T01:34:23.609849+0000","last_fullsized":"2024-09-16T01:34:23.609849+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:55:12.145787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188388+0000","last_change":"2024-09-16T01:34:21.191946+0000","last_active":"2024-09-16T01:34:23.188388+0000","last_peered":"2024-09-16T01:34:23.188388+0000","last_clean":"2024-09-16T01:34:23.188388+0000","last_became_active":"2024-09-16T01:34:21.191675+0000","last_became_peered":"2024-09-16T01:34:21.191675+0000","last_unstale":"2024-09-16T01:34:23.188388+0000","last_undegraded":"2024-09-16T01:34:23.188388+0000","last_fullsized":"2024-09-16T01:34:23.188388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:00.162566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188347+0000","last_change":"2024-09-16T01:34:22.193611+0000","last_active":"2024-09-16T01:34:23.188347+0000","last_peered":"2024-09-16T01:34:23.188347+0000","last_clean":"2024-09-16T01:34:23.188347+0000","last_became_active":"2024-09-16T01:34:22.193334+0000","last_became_peered":"2024-09-16T01:34:22.193334+0000","last_unstale":"2024-09-16T01:34:23.188347+0000","last_undegraded":"2024-09-16T01:34:23.188347+0000","last_fullsized":"2024-09-16T01:34:23.188347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:53:16.220046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609944+0000","last_change":"2024-09-16T01:34:21.192892+0000","last_active":"2024-09-16T01:34:23.609944+0000","last_peered":"2024-09-16T01:34:23.609944+0000","last_clean":"2024-09-16T01:34:23.609944+0000","last_became_active":"2024-09-16T01:34:21.192600+0000","last_became_peered":"2024-09-16T01:34:21.192600+0000","last_unstale":"2024-09-16T01:34:23.609944+0000","last_undegraded":"2024-09-16T01:34:23.609944+0000","last_fullsized":"2024-09-16T01:34:23.609944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:14:16.097098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509449,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":73825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542152,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":664685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":75347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":73861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542151,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":75347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542152,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":212992,"data_stored":73735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542151,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":221184,"data_stored":73895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":819200,"data_stored":664741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:34:33.716 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:34:33.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.848+0000 7f3c4bbfd640 1 Processor -- start 2024-09-16T01:34:33.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.848+0000 7f3c4bbfd640 1 -- start start 2024-09-16T01:34:33.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.849+0000 7f3c4bbfd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c4415c550 0x7f3c440715e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:33.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.849+0000 7f3c4bbfd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3c440b6e10 0x7f3c440b5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:33.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.849+0000 7f3c4bbfd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c440d0270 con 0x7f3c44073160 2024-09-16T01:34:33.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.849+0000 7f3c4bbfd640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f3c4415c930 con 0x7f3c440b6e10 2024-09-16T01:34:33.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.849+0000 7f3c4bbfd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3c440d04f0 con 0x7f3c4415c550 2024-09-16T01:34:33.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.849+0000 7f3c49198640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3c440b6e10 0x7f3c440b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:33.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.849+0000 7f3c4a19a640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3c44073160 0x7f3c440d0cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45248/0 (socket says 172.21.15.88:45248) 2024-09-16T01:34:33.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.849+0000 7f3c49198640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3c440b6e10 0x7f3c440b5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55212/0 (socket says 172.21.15.88:55212) 2024-09-16T01:34:33.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.849+0000 7f3c4a19a640 1 -- 172.21.15.88:0/2597981965 learned_addr learned my addr 172.21.15.88:0/2597981965 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:33.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.849+0000 7f3c49999640 1 --2- 172.21.15.88:0/2597981965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c4415c550 0x7f3c440715e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:33.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.850+0000 7f3c49198640 1 -- 172.21.15.88:0/2597981965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c4415c550 msgr2=0x7f3c440715e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:33.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.850+0000 7f3c49198640 1 --2- 172.21.15.88:0/2597981965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c4415c550 0x7f3c440715e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.850+0000 7f3c49198640 1 -- 172.21.15.88:0/2597981965 >> v1:172.21.15.88:6789/0 conn(0x7f3c44073160 legacy=0x7f3c440d0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.850+0000 7f3c49198640 1 -- 172.21.15.88:0/2597981965 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c440b7580 con 0x7f3c440b6e10 2024-09-16T01:34:33.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.850+0000 7f3c48997640 1 -- 172.21.15.88:0/2597981965 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1185165068 0 0) 0x7f3c3c001110 con 0x7f3c44073160 2024-09-16T01:34:33.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.850+0000 7f3c49198640 1 --2- 172.21.15.88:0/2597981965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c440b6e10 0x7f3c440b5540 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3c380097a0 tx=0x7f3c3802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2075c73326fb5256 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:33.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.850+0000 7f3c48997640 1 -- 172.21.15.88:0/2597981965 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c38037070 con 0x7f3c440b6e10 2024-09-16T01:34:33.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.850+0000 7f3c48997640 1 -- 172.21.15.88:0/2597981965 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c38032440 con 0x7f3c440b6e10 2024-09-16T01:34:33.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.850+0000 7f3c48997640 1 -- 172.21.15.88:0/2597981965 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c38032810 con 0x7f3c440b6e10 2024-09-16T01:34:33.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.851+0000 7f3c4bbfd640 1 -- 172.21.15.88:0/2597981965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c440b6e10 msgr2=0x7f3c440b5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.851+0000 7f3c4bbfd640 1 --2- 172.21.15.88:0/2597981965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c440b6e10 0x7f3c440b5540 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3c380097a0 tx=0x7f3c3802a180 comp rx=0 tx=0).stop 2024-09-16T01:34:33.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.851+0000 7f3c4bbfd640 1 -- 172.21.15.88:0/2597981965 shutdown_connections 2024-09-16T01:34:33.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.851+0000 7f3c4bbfd640 1 --2- 172.21.15.88:0/2597981965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c440b6e10 0x7f3c440b5540 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.851+0000 7f3c4bbfd640 1 --2- 172.21.15.88:0/2597981965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c4415c550 0x7f3c440715e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.851+0000 7f3c4bbfd640 1 -- 172.21.15.88:0/2597981965 >> 172.21.15.88:0/2597981965 conn(0x7f3c440b4240 msgr2=0x7f3c440b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:33.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.851+0000 7f3c4bbfd640 1 -- 172.21.15.88:0/2597981965 shutdown_connections 2024-09-16T01:34:33.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.851+0000 7f3c4bbfd640 1 -- 172.21.15.88:0/2597981965 wait complete. 2024-09-16T01:34:33.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.852+0000 7f3c4bbfd640 1 Processor -- start 2024-09-16T01:34:33.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.852+0000 7f3c4bbfd640 1 -- start start 2024-09-16T01:34:33.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.853+0000 7f3c4bbfd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c44073160 0x7f3c4414b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:33.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.853+0000 7f3c4bbfd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c440b6e10 0x7f3c44140ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:33.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.853+0000 7f3c4bbfd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c4415c550 0x7f3c44141010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:33.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.853+0000 7f3c4bbfd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3c440b7250 con 0x7f3c44073160 2024-09-16T01:34:33.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.853+0000 7f3c4bbfd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3c440d07b0 con 0x7f3c440b6e10 2024-09-16T01:34:33.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.853+0000 7f3c4bbfd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3c440b5a80 con 0x7f3c4415c550 2024-09-16T01:34:33.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.853+0000 7f3c49999640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c44073160 0x7f3c4414b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:33.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.853+0000 7f3c49999640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c44073160 0x7f3c4414b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55218/0 (socket says 172.21.15.88:55218) 2024-09-16T01:34:33.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.853+0000 7f3c49999640 1 -- 172.21.15.88:0/305449549 learned_addr learned my addr 172.21.15.88:0/305449549 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:33.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.853+0000 7f3c49198640 1 --2- 172.21.15.88:0/305449549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c440b6e10 0x7f3c44140ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:33.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.853+0000 7f3c4a19a640 1 --2- 172.21.15.88:0/305449549 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c4415c550 0x7f3c44141010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:33.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.853+0000 7f3c49999640 1 -- 172.21.15.88:0/305449549 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c4415c550 msgr2=0x7f3c44141010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.853+0000 7f3c49999640 1 --2- 172.21.15.88:0/305449549 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c4415c550 0x7f3c44141010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.853+0000 7f3c49999640 1 -- 172.21.15.88:0/305449549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c440b6e10 msgr2=0x7f3c44140ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:33.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.853+0000 7f3c49999640 1 --2- 172.21.15.88:0/305449549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c440b6e10 0x7f3c44140ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:33.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.853+0000 7f3c49999640 1 -- 172.21.15.88:0/305449549 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c441f2740 con 0x7f3c44073160 2024-09-16T01:34:33.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.853+0000 7f3c49198640 1 --2- 172.21.15.88:0/305449549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c440b6e10 0x7f3c44140ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:33.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.853+0000 7f3c4a19a640 1 --2- 172.21.15.88:0/305449549 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c4415c550 0x7f3c44141010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:33.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.853+0000 7f3c49999640 1 --2- 172.21.15.88:0/305449549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c44073160 0x7f3c4414b2f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3c34007e40 tx=0x7f3c3400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:33.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.854+0000 7f3c427fc640 1 -- 172.21.15.88:0/305449549 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c34014620 con 0x7f3c44073160 2024-09-16T01:34:33.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.854+0000 7f3c427fc640 1 -- 172.21.15.88:0/305449549 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c34019070 con 0x7f3c44073160 2024-09-16T01:34:33.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.854+0000 7f3c4bbfd640 1 -- 172.21.15.88:0/305449549 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c441f2a20 con 0x7f3c44073160 2024-09-16T01:34:33.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.854+0000 7f3c427fc640 1 -- 172.21.15.88:0/305449549 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c34014a00 con 0x7f3c44073160 2024-09-16T01:34:33.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.854+0000 7f3c4bbfd640 1 -- 172.21.15.88:0/305449549 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c441f2fe0 con 0x7f3c44073160 2024-09-16T01:34:33.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.855+0000 7f3c4bbfd640 1 -- 172.21.15.88:0/305449549 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c440b5540 con 0x7f3c44073160 2024-09-16T01:34:33.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.855+0000 7f3c427fc640 1 -- 172.21.15.88:0/305449549 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3c34014ba0 con 0x7f3c44073160 2024-09-16T01:34:33.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.855+0000 7f3c427fc640 1 --2- 172.21.15.88:0/305449549 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3c3006c2b0 0x7f3c3006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:33.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.856+0000 7f3c427fc640 1 -- 172.21.15.88:0/305449549 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3c3408bc00 con 0x7f3c44073160 2024-09-16T01:34:33.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.856+0000 7f3c49198640 1 --2- 172.21.15.88:0/305449549 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3c3006c2b0 0x7f3c3006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:33.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.856+0000 7f3c49198640 1 --2- 172.21.15.88:0/305449549 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3c3006c2b0 0x7f3c3006e770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3c38034000 tx=0x7f3c3802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:33.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:33.860+0000 7f3c427fc640 1 -- 172.21.15.88:0/305449549 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3c34092080 con 0x7f3c44073160 2024-09-16T01:34:34.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.024+0000 7f3c4bbfd640 1 -- 172.21.15.88:0/305449549 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3c44159b90 con 0x7f3c3006c2b0 2024-09-16T01:34:34.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.030+0000 7f3c427fc640 1 -- 172.21.15.88:0/305449549 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251837 (secure 0 0 0) 0x7f3c44159b90 con 0x7f3c3006c2b0 2024-09-16T01:34:34.035 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:34.036 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:34:34.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.033+0000 7f3c4bbfd640 1 -- 172.21.15.88:0/305449549 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3c3006c2b0 msgr2=0x7f3c3006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:34.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.033+0000 7f3c4bbfd640 1 --2- 172.21.15.88:0/305449549 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3c3006c2b0 0x7f3c3006e770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3c38034000 tx=0x7f3c3802e000 comp rx=0 tx=0).stop 2024-09-16T01:34:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.033+0000 7f3c4bbfd640 1 -- 172.21.15.88:0/305449549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c44073160 msgr2=0x7f3c4414b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.033+0000 7f3c4bbfd640 1 --2- 172.21.15.88:0/305449549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c44073160 0x7f3c4414b2f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3c34007e40 tx=0x7f3c3400e4b0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.033+0000 7f3c4bbfd640 1 -- 172.21.15.88:0/305449549 shutdown_connections 2024-09-16T01:34:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.033+0000 7f3c4bbfd640 1 --2- 172.21.15.88:0/305449549 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3c3006c2b0 0x7f3c3006e770 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.033+0000 7f3c4bbfd640 1 --2- 172.21.15.88:0/305449549 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c4415c550 0x7f3c44141010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.033+0000 7f3c4bbfd640 1 --2- 172.21.15.88:0/305449549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c440b6e10 0x7f3c44140ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.033+0000 7f3c4bbfd640 1 --2- 172.21.15.88:0/305449549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c44073160 0x7f3c4414b2f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.033+0000 7f3c4bbfd640 1 -- 172.21.15.88:0/305449549 >> 172.21.15.88:0/305449549 conn(0x7f3c440b4240 msgr2=0x7f3c440d0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.033+0000 7f3c4bbfd640 1 -- 172.21.15.88:0/305449549 shutdown_connections 2024-09-16T01:34:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.033+0000 7f3c4bbfd640 1 -- 172.21.15.88:0/305449549 wait complete. 2024-09-16T01:34:34.047 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-09-16T01:34:32.519087+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219148,"kb_used_data":2888,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755572,"statfs":{"total":773094113280,"available":772869705728,"internally_reserved":0,"allocated":2957312,"data_stored":1775436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001920"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611437+0000","last_change":"2024-09-16T01:34:22.195219+0000","last_active":"2024-09-16T01:34:23.611437+0000","last_peered":"2024-09-16T01:34:23.611437+0000","last_clean":"2024-09-16T01:34:23.611437+0000","last_became_active":"2024-09-16T01:34:22.194926+0000","last_became_peered":"2024-09-16T01:34:22.194926+0000","last_unstale":"2024-09-16T01:34:23.611437+0000","last_undegraded":"2024-09-16T01:34:23.611437+0000","last_fullsized":"2024-09-16T01:34:23.611437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:02.738433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513528+0000","last_change":"2024-09-16T01:34:22.186600+0000","last_active":"2024-09-16T01:34:23.513528+0000","last_peered":"2024-09-16T01:34:23.513528+0000","last_clean":"2024-09-16T01:34:23.513528+0000","last_became_active":"2024-09-16T01:34:22.186325+0000","last_became_peered":"2024-09-16T01:34:22.186325+0000","last_unstale":"2024-09-16T01:34:23.513528+0000","last_undegraded":"2024-09-16T01:34:23.513528+0000","last_fullsized":"2024-09-16T01:34:23.513528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:35:45.862526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609611+0000","last_change":"2024-09-16T01:34:22.187744+0000","last_active":"2024-09-16T01:34:23.609611+0000","last_peered":"2024-09-16T01:34:23.609611+0000","last_clean":"2024-09-16T01:34:23.609611+0000","last_became_active":"2024-09-16T01:34:22.186002+0000","last_became_peered":"2024-09-16T01:34:22.186002+0000","last_unstale":"2024-09-16T01:34:23.609611+0000","last_undegraded":"2024-09-16T01:34:23.609611+0000","last_fullsized":"2024-09-16T01:34:23.609611+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:38:50.126366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610430+0000","last_change":"2024-09-16T01:34:22.194231+0000","last_active":"2024-09-16T01:34:23.610430+0000","last_peered":"2024-09-16T01:34:23.610430+0000","last_clean":"2024-09-16T01:34:23.610430+0000","last_became_active":"2024-09-16T01:34:22.193969+0000","last_became_peered":"2024-09-16T01:34:22.193969+0000","last_unstale":"2024-09-16T01:34:23.610430+0000","last_undegraded":"2024-09-16T01:34:23.610430+0000","last_fullsized":"2024-09-16T01:34:23.610430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:35:45.862526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611192+0000","last_change":"2024-09-16T01:34:22.191565+0000","last_active":"2024-09-16T01:34:23.611192+0000","last_peered":"2024-09-16T01:34:23.611192+0000","last_clean":"2024-09-16T01:34:23.611192+0000","last_became_active":"2024-09-16T01:34:22.191281+0000","last_became_peered":"2024-09-16T01:34:22.191281+0000","last_unstale":"2024-09-16T01:34:23.611192+0000","last_undegraded":"2024-09-16T01:34:23.611192+0000","last_fullsized":"2024-09-16T01:34:23.611192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:49.814244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610462+0000","last_change":"2024-09-16T01:34:22.199917+0000","last_active":"2024-09-16T01:34:23.610462+0000","last_peered":"2024-09-16T01:34:23.610462+0000","last_clean":"2024-09-16T01:34:23.610462+0000","last_became_active":"2024-09-16T01:34:22.198949+0000","last_became_peered":"2024-09-16T01:34:22.198949+0000","last_unstale":"2024-09-16T01:34:23.610462+0000","last_undegraded":"2024-09-16T01:34:23.610462+0000","last_fullsized":"2024-09-16T01:34:23.610462+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:48:44.596067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610965+0000","last_change":"2024-09-16T01:34:22.191158+0000","last_active":"2024-09-16T01:34:23.610965+0000","last_peered":"2024-09-16T01:34:23.610965+0000","last_clean":"2024-09-16T01:34:23.610965+0000","last_became_active":"2024-09-16T01:34:22.190856+0000","last_became_peered":"2024-09-16T01:34:22.190856+0000","last_unstale":"2024-09-16T01:34:23.610965+0000","last_undegraded":"2024-09-16T01:34:23.610965+0000","last_fullsized":"2024-09-16T01:34:23.610965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:05.939767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.187946+0000","last_change":"2024-09-16T01:34:22.200461+0000","last_active":"2024-09-16T01:34:23.187946+0000","last_peered":"2024-09-16T01:34:23.187946+0000","last_clean":"2024-09-16T01:34:23.187946+0000","last_became_active":"2024-09-16T01:34:22.200181+0000","last_became_peered":"2024-09-16T01:34:22.200181+0000","last_unstale":"2024-09-16T01:34:23.187946+0000","last_undegraded":"2024-09-16T01:34:23.187946+0000","last_fullsized":"2024-09-16T01:34:23.187946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:36:03.508489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611582+0000","last_change":"2024-09-16T01:34:22.201897+0000","last_active":"2024-09-16T01:34:23.611582+0000","last_peered":"2024-09-16T01:34:23.611582+0000","last_clean":"2024-09-16T01:34:23.611582+0000","last_became_active":"2024-09-16T01:34:22.201595+0000","last_became_peered":"2024-09-16T01:34:22.201595+0000","last_unstale":"2024-09-16T01:34:23.611582+0000","last_undegraded":"2024-09-16T01:34:23.611582+0000","last_fullsized":"2024-09-16T01:34:23.611582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:07.782481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191787+0000","last_change":"2024-09-16T01:34:22.195243+0000","last_active":"2024-09-16T01:34:23.191787+0000","last_peered":"2024-09-16T01:34:23.191787+0000","last_clean":"2024-09-16T01:34:23.191787+0000","last_became_active":"2024-09-16T01:34:22.194813+0000","last_became_peered":"2024-09-16T01:34:22.194813+0000","last_unstale":"2024-09-16T01:34:23.191787+0000","last_undegraded":"2024-09-16T01:34:23.191787+0000","last_fullsized":"2024-09-16T01:34:23.191787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:11:56.336591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191528+0000","last_change":"2024-09-16T01:34:22.197385+0000","last_active":"2024-09-16T01:34:23.191528+0000","last_peered":"2024-09-16T01:34:23.191528+0000","last_clean":"2024-09-16T01:34:23.191528+0000","last_became_active":"2024-09-16T01:34:22.196747+0000","last_became_peered":"2024-09-16T01:34:22.196747+0000","last_unstale":"2024-09-16T01:34:23.191528+0000","last_undegraded":"2024-09-16T01:34:23.191528+0000","last_fullsized":"2024-09-16T01:34:23.191528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:14:18.368513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.190899+0000","last_change":"2024-09-16T01:34:22.181901+0000","last_active":"2024-09-16T01:34:23.190899+0000","last_peered":"2024-09-16T01:34:23.190899+0000","last_clean":"2024-09-16T01:34:23.190899+0000","last_became_active":"2024-09-16T01:34:22.181627+0000","last_became_peered":"2024-09-16T01:34:22.181627+0000","last_unstale":"2024-09-16T01:34:23.190899+0000","last_undegraded":"2024-09-16T01:34:23.190899+0000","last_fullsized":"2024-09-16T01:34:23.190899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:31:42.191103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188450+0000","last_change":"2024-09-16T01:34:22.192960+0000","last_active":"2024-09-16T01:34:23.188450+0000","last_peered":"2024-09-16T01:34:23.188450+0000","last_clean":"2024-09-16T01:34:23.188450+0000","last_became_active":"2024-09-16T01:34:22.192664+0000","last_became_peered":"2024-09-16T01:34:22.192664+0000","last_unstale":"2024-09-16T01:34:23.188450+0000","last_undegraded":"2024-09-16T01:34:23.188450+0000","last_fullsized":"2024-09-16T01:34:23.188450+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:56:56.990459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609665+0000","last_change":"2024-09-16T01:34:22.192223+0000","last_active":"2024-09-16T01:34:23.609665+0000","last_peered":"2024-09-16T01:34:23.609665+0000","last_clean":"2024-09-16T01:34:23.609665+0000","last_became_active":"2024-09-16T01:34:22.190787+0000","last_became_peered":"2024-09-16T01:34:22.190787+0000","last_unstale":"2024-09-16T01:34:23.609665+0000","last_undegraded":"2024-09-16T01:34:23.609665+0000","last_fullsized":"2024-09-16T01:34:23.609665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:40:27.589986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513177+0000","last_change":"2024-09-16T01:34:22.194793+0000","last_active":"2024-09-16T01:34:23.513177+0000","last_peered":"2024-09-16T01:34:23.513177+0000","last_clean":"2024-09-16T01:34:23.513177+0000","last_became_active":"2024-09-16T01:34:22.194530+0000","last_became_peered":"2024-09-16T01:34:22.194530+0000","last_unstale":"2024-09-16T01:34:23.513177+0000","last_undegraded":"2024-09-16T01:34:23.513177+0000","last_fullsized":"2024-09-16T01:34:23.513177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:10:23.471025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610102+0000","last_change":"2024-09-16T01:34:22.198482+0000","last_active":"2024-09-16T01:34:23.610102+0000","last_peered":"2024-09-16T01:34:23.610102+0000","last_clean":"2024-09-16T01:34:23.610102+0000","last_became_active":"2024-09-16T01:34:22.198216+0000","last_became_peered":"2024-09-16T01:34:22.198216+0000","last_unstale":"2024-09-16T01:34:23.610102+0000","last_undegraded":"2024-09-16T01:34:23.610102+0000","last_fullsized":"2024-09-16T01:34:23.610102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:04.740799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610673+0000","last_change":"2024-09-16T01:34:22.198300+0000","last_active":"2024-09-16T01:34:23.610673+0000","last_peered":"2024-09-16T01:34:23.610673+0000","last_clean":"2024-09-16T01:34:23.610673+0000","last_became_active":"2024-09-16T01:34:22.198024+0000","last_became_peered":"2024-09-16T01:34:22.198024+0000","last_unstale":"2024-09-16T01:34:23.610673+0000","last_undegraded":"2024-09-16T01:34:23.610673+0000","last_fullsized":"2024-09-16T01:34:23.610673+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:49.814244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609788+0000","last_change":"2024-09-16T01:34:22.194454+0000","last_active":"2024-09-16T01:34:23.609788+0000","last_peered":"2024-09-16T01:34:23.609788+0000","last_clean":"2024-09-16T01:34:23.609788+0000","last_became_active":"2024-09-16T01:34:22.193936+0000","last_became_peered":"2024-09-16T01:34:22.193936+0000","last_unstale":"2024-09-16T01:34:23.609788+0000","last_undegraded":"2024-09-16T01:34:23.609788+0000","last_fullsized":"2024-09-16T01:34:23.609788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:42:44.444634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611563+0000","last_change":"2024-09-16T01:34:22.191670+0000","last_active":"2024-09-16T01:34:23.611563+0000","last_peered":"2024-09-16T01:34:23.611563+0000","last_clean":"2024-09-16T01:34:23.611563+0000","last_became_active":"2024-09-16T01:34:22.191341+0000","last_became_peered":"2024-09-16T01:34:22.191341+0000","last_unstale":"2024-09-16T01:34:23.611563+0000","last_undegraded":"2024-09-16T01:34:23.611563+0000","last_fullsized":"2024-09-16T01:34:23.611563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:50:20.263787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191662+0000","last_change":"2024-09-16T01:34:22.193121+0000","last_active":"2024-09-16T01:34:23.191662+0000","last_peered":"2024-09-16T01:34:23.191662+0000","last_clean":"2024-09-16T01:34:23.191662+0000","last_became_active":"2024-09-16T01:34:22.192835+0000","last_became_peered":"2024-09-16T01:34:22.192835+0000","last_unstale":"2024-09-16T01:34:23.191662+0000","last_undegraded":"2024-09-16T01:34:23.191662+0000","last_fullsized":"2024-09-16T01:34:23.191662+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:55:12.145787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609601+0000","last_change":"2024-09-16T01:34:22.196878+0000","last_active":"2024-09-16T01:34:23.609601+0000","last_peered":"2024-09-16T01:34:23.609601+0000","last_clean":"2024-09-16T01:34:23.609601+0000","last_became_active":"2024-09-16T01:34:22.196451+0000","last_became_peered":"2024-09-16T01:34:22.196451+0000","last_unstale":"2024-09-16T01:34:23.609601+0000","last_undegraded":"2024-09-16T01:34:23.609601+0000","last_fullsized":"2024-09-16T01:34:23.609601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:42:01.118172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610529+0000","last_change":"2024-09-16T01:34:22.196889+0000","last_active":"2024-09-16T01:34:23.610529+0000","last_peered":"2024-09-16T01:34:23.610529+0000","last_clean":"2024-09-16T01:34:23.610529+0000","last_became_active":"2024-09-16T01:34:22.196583+0000","last_became_peered":"2024-09-16T01:34:22.196583+0000","last_unstale":"2024-09-16T01:34:23.610529+0000","last_undegraded":"2024-09-16T01:34:23.610529+0000","last_fullsized":"2024-09-16T01:34:23.610529+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:05.939767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188009+0000","last_change":"2024-09-16T01:34:22.196090+0000","last_active":"2024-09-16T01:34:23.188009+0000","last_peered":"2024-09-16T01:34:23.188009+0000","last_clean":"2024-09-16T01:34:23.188009+0000","last_became_active":"2024-09-16T01:34:22.195441+0000","last_became_peered":"2024-09-16T01:34:22.195441+0000","last_unstale":"2024-09-16T01:34:23.188009+0000","last_undegraded":"2024-09-16T01:34:23.188009+0000","last_fullsized":"2024-09-16T01:34:23.188009+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:14:18.368513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610192+0000","last_change":"2024-09-16T01:34:22.192211+0000","last_active":"2024-09-16T01:34:23.610192+0000","last_peered":"2024-09-16T01:34:23.610192+0000","last_clean":"2024-09-16T01:34:23.610192+0000","last_became_active":"2024-09-16T01:34:22.191921+0000","last_became_peered":"2024-09-16T01:34:22.191921+0000","last_unstale":"2024-09-16T01:34:23.610192+0000","last_undegraded":"2024-09-16T01:34:23.610192+0000","last_fullsized":"2024-09-16T01:34:23.610192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:37.582768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188086+0000","last_change":"2024-09-16T01:34:22.195710+0000","last_active":"2024-09-16T01:34:23.188086+0000","last_peered":"2024-09-16T01:34:23.188086+0000","last_clean":"2024-09-16T01:34:23.188086+0000","last_became_active":"2024-09-16T01:34:22.195385+0000","last_became_peered":"2024-09-16T01:34:22.195385+0000","last_unstale":"2024-09-16T01:34:23.188086+0000","last_undegraded":"2024-09-16T01:34:23.188086+0000","last_fullsized":"2024-09-16T01:34:23.188086+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:32:14.878649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191846+0000","last_change":"2024-09-16T01:34:22.193276+0000","last_active":"2024-09-16T01:34:23.191846+0000","last_peered":"2024-09-16T01:34:23.191846+0000","last_clean":"2024-09-16T01:34:23.191846+0000","last_became_active":"2024-09-16T01:34:22.192973+0000","last_became_peered":"2024-09-16T01:34:22.192973+0000","last_unstale":"2024-09-16T01:34:23.191846+0000","last_undegraded":"2024-09-16T01:34:23.191846+0000","last_fullsized":"2024-09-16T01:34:23.191846+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:16:41.426352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513373+0000","last_change":"2024-09-16T01:34:22.189813+0000","last_active":"2024-09-16T01:34:23.513373+0000","last_peered":"2024-09-16T01:34:23.513373+0000","last_clean":"2024-09-16T01:34:23.513373+0000","last_became_active":"2024-09-16T01:34:22.189535+0000","last_became_peered":"2024-09-16T01:34:22.189535+0000","last_unstale":"2024-09-16T01:34:23.513373+0000","last_undegraded":"2024-09-16T01:34:23.513373+0000","last_fullsized":"2024-09-16T01:34:23.513373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:37.582768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188382+0000","last_change":"2024-09-16T01:34:22.195577+0000","last_active":"2024-09-16T01:34:23.188382+0000","last_peered":"2024-09-16T01:34:23.188382+0000","last_clean":"2024-09-16T01:34:23.188382+0000","last_became_active":"2024-09-16T01:34:22.195271+0000","last_became_peered":"2024-09-16T01:34:22.195271+0000","last_unstale":"2024-09-16T01:34:23.188382+0000","last_undegraded":"2024-09-16T01:34:23.188382+0000","last_fullsized":"2024-09-16T01:34:23.188382+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:13.825111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609899+0000","last_change":"2024-09-16T01:34:22.189995+0000","last_active":"2024-09-16T01:34:23.609899+0000","last_peered":"2024-09-16T01:34:23.609899+0000","last_clean":"2024-09-16T01:34:23.609899+0000","last_became_active":"2024-09-16T01:34:22.187919+0000","last_became_peered":"2024-09-16T01:34:22.187919+0000","last_unstale":"2024-09-16T01:34:23.609899+0000","last_undegraded":"2024-09-16T01:34:23.609899+0000","last_fullsized":"2024-09-16T01:34:23.609899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:14:18.368513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610796+0000","last_change":"2024-09-16T01:34:22.194365+0000","last_active":"2024-09-16T01:34:23.610796+0000","last_peered":"2024-09-16T01:34:23.610796+0000","last_clean":"2024-09-16T01:34:23.610796+0000","last_became_active":"2024-09-16T01:34:22.194094+0000","last_became_peered":"2024-09-16T01:34:22.194094+0000","last_unstale":"2024-09-16T01:34:23.610796+0000","last_undegraded":"2024-09-16T01:34:23.610796+0000","last_fullsized":"2024-09-16T01:34:23.610796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:02.738433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609476+0000","last_change":"2024-09-16T01:34:22.195821+0000","last_active":"2024-09-16T01:34:23.609476+0000","last_peered":"2024-09-16T01:34:23.609476+0000","last_clean":"2024-09-16T01:34:23.609476+0000","last_became_active":"2024-09-16T01:34:22.194968+0000","last_became_peered":"2024-09-16T01:34:22.194968+0000","last_unstale":"2024-09-16T01:34:23.609476+0000","last_undegraded":"2024-09-16T01:34:23.609476+0000","last_fullsized":"2024-09-16T01:34:23.609476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:15:36.858098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191308+0000","last_change":"2024-09-16T01:34:22.189557+0000","last_active":"2024-09-16T01:34:23.191308+0000","last_peered":"2024-09-16T01:34:23.191308+0000","last_clean":"2024-09-16T01:34:23.191308+0000","last_became_active":"2024-09-16T01:34:22.189252+0000","last_became_peered":"2024-09-16T01:34:22.189252+0000","last_unstale":"2024-09-16T01:34:23.191308+0000","last_undegraded":"2024-09-16T01:34:23.191308+0000","last_fullsized":"2024-09-16T01:34:23.191308+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:40:27.589986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.190446+0000","last_change":"2024-09-16T01:34:22.187071+0000","last_active":"2024-09-16T01:34:23.190446+0000","last_peered":"2024-09-16T01:34:23.190446+0000","last_clean":"2024-09-16T01:34:23.190446+0000","last_became_active":"2024-09-16T01:34:22.186773+0000","last_became_peered":"2024-09-16T01:34:22.186773+0000","last_unstale":"2024-09-16T01:34:23.190446+0000","last_undegraded":"2024-09-16T01:34:23.190446+0000","last_fullsized":"2024-09-16T01:34:23.190446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:22:27.895343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188539+0000","last_change":"2024-09-16T01:34:21.187730+0000","last_active":"2024-09-16T01:34:23.188539+0000","last_peered":"2024-09-16T01:34:23.188539+0000","last_clean":"2024-09-16T01:34:23.188539+0000","last_became_active":"2024-09-16T01:34:21.187421+0000","last_became_peered":"2024-09-16T01:34:21.187421+0000","last_unstale":"2024-09-16T01:34:23.188539+0000","last_undegraded":"2024-09-16T01:34:23.188539+0000","last_fullsized":"2024-09-16T01:34:23.188539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:56.560008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609871+0000","last_change":"2024-09-16T01:34:22.197079+0000","last_active":"2024-09-16T01:34:23.609871+0000","last_peered":"2024-09-16T01:34:23.609871+0000","last_clean":"2024-09-16T01:34:23.609871+0000","last_became_active":"2024-09-16T01:34:22.196793+0000","last_became_peered":"2024-09-16T01:34:22.196793+0000","last_unstale":"2024-09-16T01:34:23.609871+0000","last_undegraded":"2024-09-16T01:34:23.609871+0000","last_fullsized":"2024-09-16T01:34:23.609871+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:16:41.426352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191283+0000","last_change":"2024-09-16T01:34:21.188731+0000","last_active":"2024-09-16T01:34:23.191283+0000","last_peered":"2024-09-16T01:34:23.191283+0000","last_clean":"2024-09-16T01:34:23.191283+0000","last_became_active":"2024-09-16T01:34:21.188477+0000","last_became_peered":"2024-09-16T01:34:21.188477+0000","last_unstale":"2024-09-16T01:34:23.191283+0000","last_undegraded":"2024-09-16T01:34:23.191283+0000","last_fullsized":"2024-09-16T01:34:23.191283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:58:12.138025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188059+0000","last_change":"2024-09-16T01:34:21.187868+0000","last_active":"2024-09-16T01:34:23.188059+0000","last_peered":"2024-09-16T01:34:23.188059+0000","last_clean":"2024-09-16T01:34:23.188059+0000","last_became_active":"2024-09-16T01:34:21.187558+0000","last_became_peered":"2024-09-16T01:34:21.187558+0000","last_unstale":"2024-09-16T01:34:23.188059+0000","last_undegraded":"2024-09-16T01:34:23.188059+0000","last_fullsized":"2024-09-16T01:34:23.188059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:58:12.138025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188157+0000","last_change":"2024-09-16T01:34:22.195132+0000","last_active":"2024-09-16T01:34:23.188157+0000","last_peered":"2024-09-16T01:34:23.188157+0000","last_clean":"2024-09-16T01:34:23.188157+0000","last_became_active":"2024-09-16T01:34:22.194861+0000","last_became_peered":"2024-09-16T01:34:22.194861+0000","last_unstale":"2024-09-16T01:34:23.188157+0000","last_undegraded":"2024-09-16T01:34:23.188157+0000","last_fullsized":"2024-09-16T01:34:23.188157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:11:56.336591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610304+0000","last_change":"2024-09-16T01:34:21.188039+0000","last_active":"2024-09-16T01:34:23.610304+0000","last_peered":"2024-09-16T01:34:23.610304+0000","last_clean":"2024-09-16T01:34:23.610304+0000","last_became_active":"2024-09-16T01:34:21.187743+0000","last_became_peered":"2024-09-16T01:34:21.187743+0000","last_unstale":"2024-09-16T01:34:23.610304+0000","last_undegraded":"2024-09-16T01:34:23.610304+0000","last_fullsized":"2024-09-16T01:34:23.610304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:50:53.737531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513172+0000","last_change":"2024-09-16T01:34:22.187569+0000","last_active":"2024-09-16T01:34:23.513172+0000","last_peered":"2024-09-16T01:34:23.513172+0000","last_clean":"2024-09-16T01:34:23.513172+0000","last_became_active":"2024-09-16T01:34:22.187301+0000","last_became_peered":"2024-09-16T01:34:22.187301+0000","last_unstale":"2024-09-16T01:34:23.513172+0000","last_undegraded":"2024-09-16T01:34:23.513172+0000","last_fullsized":"2024-09-16T01:34:23.513172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:46:31.545461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191499+0000","last_change":"2024-09-16T01:34:21.187827+0000","last_active":"2024-09-16T01:34:23.191499+0000","last_peered":"2024-09-16T01:34:23.191499+0000","last_clean":"2024-09-16T01:34:23.191499+0000","last_became_active":"2024-09-16T01:34:21.187542+0000","last_became_peered":"2024-09-16T01:34:21.187542+0000","last_unstale":"2024-09-16T01:34:23.191499+0000","last_undegraded":"2024-09-16T01:34:23.191499+0000","last_fullsized":"2024-09-16T01:34:23.191499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:18:50.444668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610494+0000","last_change":"2024-09-16T01:34:22.197484+0000","last_active":"2024-09-16T01:34:23.610494+0000","last_peered":"2024-09-16T01:34:23.610494+0000","last_clean":"2024-09-16T01:34:23.610494+0000","last_became_active":"2024-09-16T01:34:22.197209+0000","last_became_peered":"2024-09-16T01:34:22.197209+0000","last_unstale":"2024-09-16T01:34:23.610494+0000","last_undegraded":"2024-09-16T01:34:23.610494+0000","last_fullsized":"2024-09-16T01:34:23.610494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:06.384195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610564+0000","last_change":"2024-09-16T01:34:21.190619+0000","last_active":"2024-09-16T01:34:23.610564+0000","last_peered":"2024-09-16T01:34:23.610564+0000","last_clean":"2024-09-16T01:34:23.610564+0000","last_became_active":"2024-09-16T01:34:21.188943+0000","last_became_peered":"2024-09-16T01:34:21.188943+0000","last_unstale":"2024-09-16T01:34:23.610564+0000","last_undegraded":"2024-09-16T01:34:23.610564+0000","last_fullsized":"2024-09-16T01:34:23.610564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:47:09.976612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609914+0000","last_change":"2024-09-16T01:34:22.198543+0000","last_active":"2024-09-16T01:34:23.609914+0000","last_peered":"2024-09-16T01:34:23.609914+0000","last_clean":"2024-09-16T01:34:23.609914+0000","last_became_active":"2024-09-16T01:34:22.198283+0000","last_became_peered":"2024-09-16T01:34:22.198283+0000","last_unstale":"2024-09-16T01:34:23.609914+0000","last_undegraded":"2024-09-16T01:34:23.609914+0000","last_fullsized":"2024-09-16T01:34:23.609914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:32:05.208969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513419+0000","last_change":"2024-09-16T01:34:21.189485+0000","last_active":"2024-09-16T01:34:23.513419+0000","last_peered":"2024-09-16T01:34:23.513419+0000","last_clean":"2024-09-16T01:34:23.513419+0000","last_became_active":"2024-09-16T01:34:21.189201+0000","last_became_peered":"2024-09-16T01:34:21.189201+0000","last_unstale":"2024-09-16T01:34:23.513419+0000","last_undegraded":"2024-09-16T01:34:23.513419+0000","last_fullsized":"2024-09-16T01:34:23.513419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:01:27.300067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.190949+0000","last_change":"2024-09-16T01:34:22.194435+0000","last_active":"2024-09-16T01:34:23.190949+0000","last_peered":"2024-09-16T01:34:23.190949+0000","last_clean":"2024-09-16T01:34:23.190949+0000","last_became_active":"2024-09-16T01:34:22.194058+0000","last_became_peered":"2024-09-16T01:34:22.194058+0000","last_unstale":"2024-09-16T01:34:23.190949+0000","last_undegraded":"2024-09-16T01:34:23.190949+0000","last_fullsized":"2024-09-16T01:34:23.190949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:36:45.083401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191725+0000","last_change":"2024-09-16T01:34:21.185553+0000","last_active":"2024-09-16T01:34:23.191725+0000","last_peered":"2024-09-16T01:34:23.191725+0000","last_clean":"2024-09-16T01:34:23.191725+0000","last_became_active":"2024-09-16T01:34:21.185264+0000","last_became_peered":"2024-09-16T01:34:21.185264+0000","last_unstale":"2024-09-16T01:34:23.191725+0000","last_undegraded":"2024-09-16T01:34:23.191725+0000","last_fullsized":"2024-09-16T01:34:23.191725+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:22:26.887709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611437+0000","last_change":"2024-09-16T01:34:22.191352+0000","last_active":"2024-09-16T01:34:23.611437+0000","last_peered":"2024-09-16T01:34:23.611437+0000","last_clean":"2024-09-16T01:34:23.611437+0000","last_became_active":"2024-09-16T01:34:22.191027+0000","last_became_peered":"2024-09-16T01:34:22.191027+0000","last_unstale":"2024-09-16T01:34:23.611437+0000","last_undegraded":"2024-09-16T01:34:23.611437+0000","last_fullsized":"2024-09-16T01:34:23.611437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:18.860118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609920+0000","last_change":"2024-09-16T01:34:21.193072+0000","last_active":"2024-09-16T01:34:23.609920+0000","last_peered":"2024-09-16T01:34:23.609920+0000","last_clean":"2024-09-16T01:34:23.609920+0000","last_became_active":"2024-09-16T01:34:21.192769+0000","last_became_peered":"2024-09-16T01:34:21.192769+0000","last_unstale":"2024-09-16T01:34:23.609920+0000","last_undegraded":"2024-09-16T01:34:23.609920+0000","last_fullsized":"2024-09-16T01:34:23.609920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:00.162566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609665+0000","last_change":"2024-09-16T01:34:22.195668+0000","last_active":"2024-09-16T01:34:23.609665+0000","last_peered":"2024-09-16T01:34:23.609665+0000","last_clean":"2024-09-16T01:34:23.609665+0000","last_became_active":"2024-09-16T01:34:22.195407+0000","last_became_peered":"2024-09-16T01:34:22.195407+0000","last_unstale":"2024-09-16T01:34:23.609665+0000","last_undegraded":"2024-09-16T01:34:23.609665+0000","last_fullsized":"2024-09-16T01:34:23.609665+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:36:03.508489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610939+0000","last_change":"2024-09-16T01:34:21.186203+0000","last_active":"2024-09-16T01:34:23.610939+0000","last_peered":"2024-09-16T01:34:23.610939+0000","last_clean":"2024-09-16T01:34:23.610939+0000","last_became_active":"2024-09-16T01:34:21.185935+0000","last_became_peered":"2024-09-16T01:34:21.185935+0000","last_unstale":"2024-09-16T01:34:23.610939+0000","last_undegraded":"2024-09-16T01:34:23.610939+0000","last_fullsized":"2024-09-16T01:34:23.610939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:29.860344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611192+0000","last_change":"2024-09-16T01:34:22.196044+0000","last_active":"2024-09-16T01:34:23.611192+0000","last_peered":"2024-09-16T01:34:23.611192+0000","last_clean":"2024-09-16T01:34:23.611192+0000","last_became_active":"2024-09-16T01:34:22.195783+0000","last_became_peered":"2024-09-16T01:34:22.195783+0000","last_unstale":"2024-09-16T01:34:23.611192+0000","last_undegraded":"2024-09-16T01:34:23.611192+0000","last_fullsized":"2024-09-16T01:34:23.611192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:03.591531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191432+0000","last_change":"2024-09-16T01:34:21.184919+0000","last_active":"2024-09-16T01:34:23.191432+0000","last_peered":"2024-09-16T01:34:23.191432+0000","last_clean":"2024-09-16T01:34:23.191432+0000","last_became_active":"2024-09-16T01:34:21.184542+0000","last_became_peered":"2024-09-16T01:34:21.184542+0000","last_unstale":"2024-09-16T01:34:23.191432+0000","last_undegraded":"2024-09-16T01:34:23.191432+0000","last_fullsized":"2024-09-16T01:34:23.191432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:56.560008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191412+0000","last_change":"2024-09-16T01:34:22.187373+0000","last_active":"2024-09-16T01:34:23.191412+0000","last_peered":"2024-09-16T01:34:23.191412+0000","last_clean":"2024-09-16T01:34:23.191412+0000","last_became_active":"2024-09-16T01:34:22.186468+0000","last_became_peered":"2024-09-16T01:34:22.186468+0000","last_unstale":"2024-09-16T01:34:23.191412+0000","last_undegraded":"2024-09-16T01:34:23.191412+0000","last_fullsized":"2024-09-16T01:34:23.191412+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:36:03.508489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609191+0000","last_change":"2024-09-16T01:34:21.189797+0000","last_active":"2024-09-16T01:34:23.609191+0000","last_peered":"2024-09-16T01:34:23.609191+0000","last_clean":"2024-09-16T01:34:23.609191+0000","last_became_active":"2024-09-16T01:34:21.189512+0000","last_became_peered":"2024-09-16T01:34:21.189512+0000","last_unstale":"2024-09-16T01:34:23.609191+0000","last_undegraded":"2024-09-16T01:34:23.609191+0000","last_fullsized":"2024-09-16T01:34:23.609191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:01:27.300067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611069+0000","last_change":"2024-09-16T01:34:22.195318+0000","last_active":"2024-09-16T01:34:23.611069+0000","last_peered":"2024-09-16T01:34:23.611069+0000","last_clean":"2024-09-16T01:34:23.611069+0000","last_became_active":"2024-09-16T01:34:22.195051+0000","last_became_peered":"2024-09-16T01:34:22.195051+0000","last_unstale":"2024-09-16T01:34:23.611069+0000","last_undegraded":"2024-09-16T01:34:23.611069+0000","last_fullsized":"2024-09-16T01:34:23.611069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:06.384195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.190601+0000","last_change":"2024-09-16T01:34:21.191907+0000","last_active":"2024-09-16T01:34:23.190601+0000","last_peered":"2024-09-16T01:34:23.190601+0000","last_clean":"2024-09-16T01:34:23.190601+0000","last_became_active":"2024-09-16T01:34:21.191591+0000","last_became_peered":"2024-09-16T01:34:21.191591+0000","last_unstale":"2024-09-16T01:34:23.190601+0000","last_undegraded":"2024-09-16T01:34:23.190601+0000","last_fullsized":"2024-09-16T01:34:23.190601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:50:28.630517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611504+0000","last_change":"2024-09-16T01:34:22.194591+0000","last_active":"2024-09-16T01:34:23.611504+0000","last_peered":"2024-09-16T01:34:23.611504+0000","last_clean":"2024-09-16T01:34:23.611504+0000","last_became_active":"2024-09-16T01:34:22.194317+0000","last_became_peered":"2024-09-16T01:34:22.194317+0000","last_unstale":"2024-09-16T01:34:23.611504+0000","last_undegraded":"2024-09-16T01:34:23.611504+0000","last_fullsized":"2024-09-16T01:34:23.611504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:45.928615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609751+0000","last_change":"2024-09-16T01:34:21.190706+0000","last_active":"2024-09-16T01:34:23.609751+0000","last_peered":"2024-09-16T01:34:23.609751+0000","last_clean":"2024-09-16T01:34:23.609751+0000","last_became_active":"2024-09-16T01:34:21.189692+0000","last_became_peered":"2024-09-16T01:34:21.189692+0000","last_unstale":"2024-09-16T01:34:23.609751+0000","last_undegraded":"2024-09-16T01:34:23.609751+0000","last_fullsized":"2024-09-16T01:34:23.609751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:11:30.497159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611648+0000","last_change":"2024-09-16T01:34:22.194506+0000","last_active":"2024-09-16T01:34:23.611648+0000","last_peered":"2024-09-16T01:34:23.611648+0000","last_clean":"2024-09-16T01:34:23.611648+0000","last_became_active":"2024-09-16T01:34:22.194223+0000","last_became_peered":"2024-09-16T01:34:22.194223+0000","last_unstale":"2024-09-16T01:34:23.611648+0000","last_undegraded":"2024-09-16T01:34:23.611648+0000","last_fullsized":"2024-09-16T01:34:23.611648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:19.506778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513439+0000","last_change":"2024-09-16T01:34:21.189382+0000","last_active":"2024-09-16T01:34:23.513439+0000","last_peered":"2024-09-16T01:34:23.513439+0000","last_clean":"2024-09-16T01:34:23.513439+0000","last_became_active":"2024-09-16T01:34:21.188897+0000","last_became_peered":"2024-09-16T01:34:21.188897+0000","last_unstale":"2024-09-16T01:34:23.513439+0000","last_undegraded":"2024-09-16T01:34:23.513439+0000","last_fullsized":"2024-09-16T01:34:23.513439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:29.860344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610830+0000","last_change":"2024-09-16T01:34:22.192308+0000","last_active":"2024-09-16T01:34:23.610830+0000","last_peered":"2024-09-16T01:34:23.610830+0000","last_clean":"2024-09-16T01:34:23.610830+0000","last_became_active":"2024-09-16T01:34:22.191748+0000","last_became_peered":"2024-09-16T01:34:22.191748+0000","last_unstale":"2024-09-16T01:34:23.610830+0000","last_undegraded":"2024-09-16T01:34:23.610830+0000","last_fullsized":"2024-09-16T01:34:23.610830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:45.928615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609394+0000","last_change":"2024-09-16T01:34:21.188078+0000","last_active":"2024-09-16T01:34:23.609394+0000","last_peered":"2024-09-16T01:34:23.609394+0000","last_clean":"2024-09-16T01:34:23.609394+0000","last_became_active":"2024-09-16T01:34:21.187822+0000","last_became_peered":"2024-09-16T01:34:21.187822+0000","last_unstale":"2024-09-16T01:34:23.609394+0000","last_undegraded":"2024-09-16T01:34:23.609394+0000","last_fullsized":"2024-09-16T01:34:23.609394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:29.860344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611369+0000","last_change":"2024-09-16T01:34:22.202386+0000","last_active":"2024-09-16T01:34:23.611369+0000","last_peered":"2024-09-16T01:34:23.611369+0000","last_clean":"2024-09-16T01:34:23.611369+0000","last_became_active":"2024-09-16T01:34:22.202089+0000","last_became_peered":"2024-09-16T01:34:22.202089+0000","last_unstale":"2024-09-16T01:34:23.611369+0000","last_undegraded":"2024-09-16T01:34:23.611369+0000","last_fullsized":"2024-09-16T01:34:23.611369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:07.244892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610330+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:34:23.610330+0000","last_peered":"2024-09-16T01:34:23.610330+0000","last_clean":"2024-09-16T01:34:23.610330+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:34:23.610330+0000","last_undegraded":"2024-09-16T01:34:23.610330+0000","last_fullsized":"2024-09-16T01:34:23.610330+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610276+0000","last_change":"2024-09-16T01:34:21.183119+0000","last_active":"2024-09-16T01:34:23.610276+0000","last_peered":"2024-09-16T01:34:23.610276+0000","last_clean":"2024-09-16T01:34:23.610276+0000","last_became_active":"2024-09-16T01:34:21.182740+0000","last_became_peered":"2024-09-16T01:34:21.182740+0000","last_unstale":"2024-09-16T01:34:23.610276+0000","last_undegraded":"2024-09-16T01:34:23.610276+0000","last_fullsized":"2024-09-16T01:34:23.610276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:01:27.300067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188231+0000","last_change":"2024-09-16T01:34:22.202105+0000","last_active":"2024-09-16T01:34:23.188231+0000","last_peered":"2024-09-16T01:34:23.188231+0000","last_clean":"2024-09-16T01:34:23.188231+0000","last_became_active":"2024-09-16T01:34:22.201840+0000","last_became_peered":"2024-09-16T01:34:22.201840+0000","last_unstale":"2024-09-16T01:34:23.188231+0000","last_undegraded":"2024-09-16T01:34:23.188231+0000","last_fullsized":"2024-09-16T01:34:23.188231+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:32:05.208969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610203+0000","last_change":"2024-09-16T01:34:21.190480+0000","last_active":"2024-09-16T01:34:23.610203+0000","last_peered":"2024-09-16T01:34:23.610203+0000","last_clean":"2024-09-16T01:34:23.610203+0000","last_became_active":"2024-09-16T01:34:21.190205+0000","last_became_peered":"2024-09-16T01:34:21.190205+0000","last_unstale":"2024-09-16T01:34:23.610203+0000","last_undegraded":"2024-09-16T01:34:23.610203+0000","last_fullsized":"2024-09-16T01:34:23.610203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:47:09.976612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609791+0000","last_change":"2024-09-16T01:34:22.198514+0000","last_active":"2024-09-16T01:34:23.609791+0000","last_peered":"2024-09-16T01:34:23.609791+0000","last_clean":"2024-09-16T01:34:23.609791+0000","last_became_active":"2024-09-16T01:34:22.197969+0000","last_became_peered":"2024-09-16T01:34:22.197969+0000","last_unstale":"2024-09-16T01:34:23.609791+0000","last_undegraded":"2024-09-16T01:34:23.609791+0000","last_fullsized":"2024-09-16T01:34:23.609791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:32:14.878649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609319+0000","last_change":"2024-09-16T01:34:21.190398+0000","last_active":"2024-09-16T01:34:23.609319+0000","last_peered":"2024-09-16T01:34:23.609319+0000","last_clean":"2024-09-16T01:34:23.609319+0000","last_became_active":"2024-09-16T01:34:21.190141+0000","last_became_peered":"2024-09-16T01:34:21.190141+0000","last_unstale":"2024-09-16T01:34:23.609319+0000","last_undegraded":"2024-09-16T01:34:23.609319+0000","last_fullsized":"2024-09-16T01:34:23.609319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:47:09.976612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513229+0000","last_change":"2024-09-16T01:34:22.187125+0000","last_active":"2024-09-16T01:34:23.513229+0000","last_peered":"2024-09-16T01:34:23.513229+0000","last_clean":"2024-09-16T01:34:23.513229+0000","last_became_active":"2024-09-16T01:34:22.186846+0000","last_became_peered":"2024-09-16T01:34:22.186846+0000","last_unstale":"2024-09-16T01:34:23.513229+0000","last_undegraded":"2024-09-16T01:34:23.513229+0000","last_fullsized":"2024-09-16T01:34:23.513229+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:59:47.794894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610948+0000","last_change":"2024-09-16T01:34:21.190750+0000","last_active":"2024-09-16T01:34:23.610948+0000","last_peered":"2024-09-16T01:34:23.610948+0000","last_clean":"2024-09-16T01:34:23.610948+0000","last_became_active":"2024-09-16T01:34:21.189215+0000","last_became_peered":"2024-09-16T01:34:21.189215+0000","last_unstale":"2024-09-16T01:34:23.610948+0000","last_undegraded":"2024-09-16T01:34:23.610948+0000","last_fullsized":"2024-09-16T01:34:23.610948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:11:30.497159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.190850+0000","last_change":"2024-09-16T01:34:22.184566+0000","last_active":"2024-09-16T01:34:23.190850+0000","last_peered":"2024-09-16T01:34:23.190850+0000","last_clean":"2024-09-16T01:34:23.190850+0000","last_became_active":"2024-09-16T01:34:22.184223+0000","last_became_peered":"2024-09-16T01:34:22.184223+0000","last_unstale":"2024-09-16T01:34:23.190850+0000","last_undegraded":"2024-09-16T01:34:23.190850+0000","last_fullsized":"2024-09-16T01:34:23.190850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:33:24.341399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513407+0000","last_change":"2024-09-16T01:34:21.191889+0000","last_active":"2024-09-16T01:34:23.513407+0000","last_peered":"2024-09-16T01:34:23.513407+0000","last_clean":"2024-09-16T01:34:23.513407+0000","last_became_active":"2024-09-16T01:34:21.191594+0000","last_became_peered":"2024-09-16T01:34:21.191594+0000","last_unstale":"2024-09-16T01:34:23.513407+0000","last_undegraded":"2024-09-16T01:34:23.513407+0000","last_fullsized":"2024-09-16T01:34:23.513407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:57.471937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610790+0000","last_change":"2024-09-16T01:34:22.193566+0000","last_active":"2024-09-16T01:34:23.610790+0000","last_peered":"2024-09-16T01:34:23.610790+0000","last_clean":"2024-09-16T01:34:23.610790+0000","last_became_active":"2024-09-16T01:34:22.193305+0000","last_became_peered":"2024-09-16T01:34:22.193305+0000","last_unstale":"2024-09-16T01:34:23.610790+0000","last_undegraded":"2024-09-16T01:34:23.610790+0000","last_fullsized":"2024-09-16T01:34:23.610790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:18.860118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611604+0000","last_change":"2024-09-16T01:34:21.192721+0000","last_active":"2024-09-16T01:34:23.611604+0000","last_peered":"2024-09-16T01:34:23.611604+0000","last_clean":"2024-09-16T01:34:23.611604+0000","last_became_active":"2024-09-16T01:34:21.192423+0000","last_became_peered":"2024-09-16T01:34:21.192423+0000","last_unstale":"2024-09-16T01:34:23.611604+0000","last_undegraded":"2024-09-16T01:34:23.611604+0000","last_fullsized":"2024-09-16T01:34:23.611604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:13.839623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513359+0000","last_change":"2024-09-16T01:34:22.187869+0000","last_active":"2024-09-16T01:34:23.513359+0000","last_peered":"2024-09-16T01:34:23.513359+0000","last_clean":"2024-09-16T01:34:23.513359+0000","last_became_active":"2024-09-16T01:34:22.186507+0000","last_became_peered":"2024-09-16T01:34:22.186507+0000","last_unstale":"2024-09-16T01:34:23.513359+0000","last_undegraded":"2024-09-16T01:34:23.513359+0000","last_fullsized":"2024-09-16T01:34:23.513359+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:04.740799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513369+0000","last_change":"2024-09-16T01:34:21.190145+0000","last_active":"2024-09-16T01:34:23.513369+0000","last_peered":"2024-09-16T01:34:23.513369+0000","last_clean":"2024-09-16T01:34:23.513369+0000","last_became_active":"2024-09-16T01:34:21.189886+0000","last_became_peered":"2024-09-16T01:34:21.189886+0000","last_unstale":"2024-09-16T01:34:23.513369+0000","last_undegraded":"2024-09-16T01:34:23.513369+0000","last_fullsized":"2024-09-16T01:34:23.513369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:47:09.976612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188218+0000","last_change":"2024-09-16T01:34:22.194698+0000","last_active":"2024-09-16T01:34:23.188218+0000","last_peered":"2024-09-16T01:34:23.188218+0000","last_clean":"2024-09-16T01:34:23.188218+0000","last_became_active":"2024-09-16T01:34:22.194443+0000","last_became_peered":"2024-09-16T01:34:22.194443+0000","last_unstale":"2024-09-16T01:34:23.188218+0000","last_undegraded":"2024-09-16T01:34:23.188218+0000","last_fullsized":"2024-09-16T01:34:23.188218+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:16:41.426352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191535+0000","last_change":"2024-09-16T01:34:21.189000+0000","last_active":"2024-09-16T01:34:23.191535+0000","last_peered":"2024-09-16T01:34:23.191535+0000","last_clean":"2024-09-16T01:34:23.191535+0000","last_became_active":"2024-09-16T01:34:21.188144+0000","last_became_peered":"2024-09-16T01:34:21.188144+0000","last_unstale":"2024-09-16T01:34:23.191535+0000","last_undegraded":"2024-09-16T01:34:23.191535+0000","last_fullsized":"2024-09-16T01:34:23.191535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:36:22.787904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609862+0000","last_change":"2024-09-16T01:34:22.188803+0000","last_active":"2024-09-16T01:34:23.609862+0000","last_peered":"2024-09-16T01:34:23.609862+0000","last_clean":"2024-09-16T01:34:23.609862+0000","last_became_active":"2024-09-16T01:34:22.188522+0000","last_became_peered":"2024-09-16T01:34:22.188522+0000","last_unstale":"2024-09-16T01:34:23.609862+0000","last_undegraded":"2024-09-16T01:34:23.609862+0000","last_fullsized":"2024-09-16T01:34:23.609862+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:11:56.336591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610066+0000","last_change":"2024-09-16T01:34:21.192126+0000","last_active":"2024-09-16T01:34:23.610066+0000","last_peered":"2024-09-16T01:34:23.610066+0000","last_clean":"2024-09-16T01:34:23.610066+0000","last_became_active":"2024-09-16T01:34:21.191855+0000","last_became_peered":"2024-09-16T01:34:21.191855+0000","last_unstale":"2024-09-16T01:34:23.610066+0000","last_undegraded":"2024-09-16T01:34:23.610066+0000","last_fullsized":"2024-09-16T01:34:23.610066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:57.471937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.190343+0000","last_change":"2024-09-16T01:34:22.186916+0000","last_active":"2024-09-16T01:34:23.190343+0000","last_peered":"2024-09-16T01:34:23.190343+0000","last_clean":"2024-09-16T01:34:23.190343+0000","last_became_active":"2024-09-16T01:34:22.186081+0000","last_became_peered":"2024-09-16T01:34:22.186081+0000","last_unstale":"2024-09-16T01:34:23.190343+0000","last_undegraded":"2024-09-16T01:34:23.190343+0000","last_fullsized":"2024-09-16T01:34:23.190343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:36:23.795538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513225+0000","last_change":"2024-09-16T01:34:21.187955+0000","last_active":"2024-09-16T01:34:23.513225+0000","last_peered":"2024-09-16T01:34:23.513225+0000","last_clean":"2024-09-16T01:34:23.513225+0000","last_became_active":"2024-09-16T01:34:21.187668+0000","last_became_peered":"2024-09-16T01:34:21.187668+0000","last_unstale":"2024-09-16T01:34:23.513225+0000","last_undegraded":"2024-09-16T01:34:23.513225+0000","last_fullsized":"2024-09-16T01:34:23.513225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:11:30.497159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188103+0000","last_change":"2024-09-16T01:34:22.187359+0000","last_active":"2024-09-16T01:34:23.188103+0000","last_peered":"2024-09-16T01:34:23.188103+0000","last_clean":"2024-09-16T01:34:23.188103+0000","last_became_active":"2024-09-16T01:34:22.187088+0000","last_became_peered":"2024-09-16T01:34:22.187088+0000","last_unstale":"2024-09-16T01:34:23.188103+0000","last_undegraded":"2024-09-16T01:34:23.188103+0000","last_fullsized":"2024-09-16T01:34:23.188103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:55:12.145787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.513225+0000","last_change":"2024-09-16T01:34:21.184801+0000","last_active":"2024-09-16T01:34:23.513225+0000","last_peered":"2024-09-16T01:34:23.513225+0000","last_clean":"2024-09-16T01:34:23.513225+0000","last_became_active":"2024-09-16T01:34:21.184472+0000","last_became_peered":"2024-09-16T01:34:21.184472+0000","last_unstale":"2024-09-16T01:34:23.513225+0000","last_undegraded":"2024-09-16T01:34:23.513225+0000","last_fullsized":"2024-09-16T01:34:23.513225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:41:15.050218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188586+0000","last_change":"2024-09-16T01:34:22.202201+0000","last_active":"2024-09-16T01:34:23.188586+0000","last_peered":"2024-09-16T01:34:23.188586+0000","last_clean":"2024-09-16T01:34:23.188586+0000","last_became_active":"2024-09-16T01:34:22.201850+0000","last_became_peered":"2024-09-16T01:34:22.201850+0000","last_unstale":"2024-09-16T01:34:23.188586+0000","last_undegraded":"2024-09-16T01:34:23.188586+0000","last_fullsized":"2024-09-16T01:34:23.188586+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:56:15.380999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611287+0000","last_change":"2024-09-16T01:34:21.192632+0000","last_active":"2024-09-16T01:34:23.611287+0000","last_peered":"2024-09-16T01:34:23.611287+0000","last_clean":"2024-09-16T01:34:23.611287+0000","last_became_active":"2024-09-16T01:34:21.192351+0000","last_became_peered":"2024-09-16T01:34:21.192351+0000","last_unstale":"2024-09-16T01:34:23.611287+0000","last_undegraded":"2024-09-16T01:34:23.611287+0000","last_fullsized":"2024-09-16T01:34:23.611287+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:15:35.850464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.191599+0000","last_change":"2024-09-16T01:34:22.193185+0000","last_active":"2024-09-16T01:34:23.191599+0000","last_peered":"2024-09-16T01:34:23.191599+0000","last_clean":"2024-09-16T01:34:23.191599+0000","last_became_active":"2024-09-16T01:34:22.192869+0000","last_became_peered":"2024-09-16T01:34:22.192869+0000","last_unstale":"2024-09-16T01:34:23.191599+0000","last_undegraded":"2024-09-16T01:34:23.191599+0000","last_fullsized":"2024-09-16T01:34:23.191599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:32:14.878649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610675+0000","last_change":"2024-09-16T01:34:22.200190+0000","last_active":"2024-09-16T01:34:23.610675+0000","last_peered":"2024-09-16T01:34:23.610675+0000","last_clean":"2024-09-16T01:34:23.610675+0000","last_became_active":"2024-09-16T01:34:22.199826+0000","last_became_peered":"2024-09-16T01:34:22.199826+0000","last_unstale":"2024-09-16T01:34:23.610675+0000","last_undegraded":"2024-09-16T01:34:23.610675+0000","last_fullsized":"2024-09-16T01:34:23.610675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:03.591531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611122+0000","last_change":"2024-09-16T01:34:21.188752+0000","last_active":"2024-09-16T01:34:23.611122+0000","last_peered":"2024-09-16T01:34:23.611122+0000","last_clean":"2024-09-16T01:34:23.611122+0000","last_became_active":"2024-09-16T01:34:21.188414+0000","last_became_peered":"2024-09-16T01:34:21.188414+0000","last_unstale":"2024-09-16T01:34:23.611122+0000","last_undegraded":"2024-09-16T01:34:23.611122+0000","last_fullsized":"2024-09-16T01:34:23.611122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:41:15.050218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.610726+0000","last_change":"2024-09-16T01:34:22.199939+0000","last_active":"2024-09-16T01:34:23.610726+0000","last_peered":"2024-09-16T01:34:23.610726+0000","last_clean":"2024-09-16T01:34:23.610726+0000","last_became_active":"2024-09-16T01:34:22.199580+0000","last_became_peered":"2024-09-16T01:34:22.199580+0000","last_unstale":"2024-09-16T01:34:23.610726+0000","last_undegraded":"2024-09-16T01:34:23.610726+0000","last_fullsized":"2024-09-16T01:34:23.610726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:07.244892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.611491+0000","last_change":"2024-09-16T01:34:21.190410+0000","last_active":"2024-09-16T01:34:23.611491+0000","last_peered":"2024-09-16T01:34:23.611491+0000","last_clean":"2024-09-16T01:34:23.611491+0000","last_became_active":"2024-09-16T01:34:21.190075+0000","last_became_peered":"2024-09-16T01:34:21.190075+0000","last_unstale":"2024-09-16T01:34:23.611491+0000","last_undegraded":"2024-09-16T01:34:23.611491+0000","last_fullsized":"2024-09-16T01:34:23.611491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:42:43.437000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609849+0000","last_change":"2024-09-16T01:34:22.198463+0000","last_active":"2024-09-16T01:34:23.609849+0000","last_peered":"2024-09-16T01:34:23.609849+0000","last_clean":"2024-09-16T01:34:23.609849+0000","last_became_active":"2024-09-16T01:34:22.198214+0000","last_became_peered":"2024-09-16T01:34:22.198214+0000","last_unstale":"2024-09-16T01:34:23.609849+0000","last_undegraded":"2024-09-16T01:34:23.609849+0000","last_fullsized":"2024-09-16T01:34:23.609849+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:55:12.145787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188388+0000","last_change":"2024-09-16T01:34:21.191946+0000","last_active":"2024-09-16T01:34:23.188388+0000","last_peered":"2024-09-16T01:34:23.188388+0000","last_clean":"2024-09-16T01:34:23.188388+0000","last_became_active":"2024-09-16T01:34:21.191675+0000","last_became_peered":"2024-09-16T01:34:21.191675+0000","last_unstale":"2024-09-16T01:34:23.188388+0000","last_undegraded":"2024-09-16T01:34:23.188388+0000","last_fullsized":"2024-09-16T01:34:23.188388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:00.162566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.188347+0000","last_change":"2024-09-16T01:34:22.193611+0000","last_active":"2024-09-16T01:34:23.188347+0000","last_peered":"2024-09-16T01:34:23.188347+0000","last_clean":"2024-09-16T01:34:23.188347+0000","last_became_active":"2024-09-16T01:34:22.193334+0000","last_became_peered":"2024-09-16T01:34:22.193334+0000","last_unstale":"2024-09-16T01:34:23.188347+0000","last_undegraded":"2024-09-16T01:34:23.188347+0000","last_fullsized":"2024-09-16T01:34:23.188347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:21.167002+0000","last_clean_scrub_stamp":"2024-09-16T01:34:21.167002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:53:16.220046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-16T01:34:23.609944+0000","last_change":"2024-09-16T01:34:21.192892+0000","last_active":"2024-09-16T01:34:23.609944+0000","last_peered":"2024-09-16T01:34:23.609944+0000","last_clean":"2024-09-16T01:34:23.609944+0000","last_became_active":"2024-09-16T01:34:21.192600+0000","last_became_peered":"2024-09-16T01:34:21.192600+0000","last_unstale":"2024-09-16T01:34:23.609944+0000","last_undegraded":"2024-09-16T01:34:23.609944+0000","last_fullsized":"2024-09-16T01:34:23.609944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:20.159368+0000","last_clean_scrub_stamp":"2024-09-16T01:34:20.159368+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:14:16.097098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509449,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":73825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542152,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":664685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":75347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":73861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542151,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":75347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542152,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":212992,"data_stored":73735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542151,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":221184,"data_stored":73895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":819200,"data_stored":664741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:34:34.050 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T01:34:34.051 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T01:34:34.051 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T01:34:34.051 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T01:34:34.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.182+0000 7f961248d640 1 Processor -- start 2024-09-16T01:34:34.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.182+0000 7f961248d640 1 -- start start 2024-09-16T01:34:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.183+0000 7f961248d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f960c0b74c0 0x7f960c15eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.183+0000 7f961248d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f960c15f030 0x7f960c06d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.183+0000 7f961248d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f960c15fb10 con 0x7f960c15c2b0 2024-09-16T01:34:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.183+0000 7f961248d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f960c0b5900 con 0x7f960c0b74c0 2024-09-16T01:34:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.183+0000 7f961248d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f960c15f830 con 0x7f960c15f030 2024-09-16T01:34:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.183+0000 7f9603fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f960c0b74c0 0x7f960c15eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.183+0000 7f9603fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f960c0b74c0 0x7f960c15eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55230/0 (socket says 172.21.15.88:55230) 2024-09-16T01:34:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.183+0000 7f9603fff640 1 -- 172.21.15.88:0/1982954882 learned_addr learned my addr 172.21.15.88:0/1982954882 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.184+0000 7f9610a2a640 1 --2- 172.21.15.88:0/1982954882 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f960c15f030 0x7f960c06d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.184+0000 7f9603fff640 1 -- 172.21.15.88:0/1982954882 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f960c15f030 msgr2=0x7f960c06d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.184+0000 7f9603fff640 1 --2- 172.21.15.88:0/1982954882 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f960c15f030 0x7f960c06d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.184+0000 7f9603fff640 1 -- 172.21.15.88:0/1982954882 >> v1:172.21.15.154:6789/0 conn(0x7f960c15c2b0 legacy=0x7f960c0b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:34.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.184+0000 7f9603fff640 1 -- 172.21.15.88:0/1982954882 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f960c06ce30 con 0x7f960c0b74c0 2024-09-16T01:34:34.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.184+0000 7f9610a2a640 1 --2- 172.21.15.88:0/1982954882 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f960c15f030 0x7f960c06d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:34.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.184+0000 7f9603fff640 1 --2- 172.21.15.88:0/1982954882 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f960c0b74c0 0x7f960c15eaf0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f95f4009870 tx=0x7f95f402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e68a9eb6134cc139 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:34.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.185+0000 7f960b7fe640 1 -- 172.21.15.88:0/1982954882 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f4037070 con 0x7f960c0b74c0 2024-09-16T01:34:34.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.185+0000 7f960b7fe640 1 -- 172.21.15.88:0/1982954882 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95f4032440 con 0x7f960c0b74c0 2024-09-16T01:34:34.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.185+0000 7f960b7fe640 1 -- 172.21.15.88:0/1982954882 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f4032810 con 0x7f960c0b74c0 2024-09-16T01:34:34.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.185+0000 7f961248d640 1 -- 172.21.15.88:0/1982954882 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f960c0b74c0 msgr2=0x7f960c15eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:34.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.185+0000 7f961248d640 1 --2- 172.21.15.88:0/1982954882 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f960c0b74c0 0x7f960c15eaf0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f95f4009870 tx=0x7f95f402a270 comp rx=0 tx=0).stop 2024-09-16T01:34:34.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.185+0000 7f961248d640 1 -- 172.21.15.88:0/1982954882 shutdown_connections 2024-09-16T01:34:34.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.185+0000 7f961248d640 1 --2- 172.21.15.88:0/1982954882 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f960c15f030 0x7f960c06d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.185+0000 7f961248d640 1 --2- 172.21.15.88:0/1982954882 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f960c0b74c0 0x7f960c15eaf0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.185+0000 7f961248d640 1 -- 172.21.15.88:0/1982954882 >> 172.21.15.88:0/1982954882 conn(0x7f960c0b4240 msgr2=0x7f960c0b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:34.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.185+0000 7f961248d640 1 -- 172.21.15.88:0/1982954882 shutdown_connections 2024-09-16T01:34:34.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.185+0000 7f961248d640 1 -- 172.21.15.88:0/1982954882 wait complete. 2024-09-16T01:34:34.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.186+0000 7f961248d640 1 Processor -- start 2024-09-16T01:34:34.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.186+0000 7f961248d640 1 -- start start 2024-09-16T01:34:34.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.187+0000 7f961248d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f960c0b74c0 0x7f960c157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:34.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.187+0000 7f961248d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f960c15c2b0 0x7f960c1e6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:34.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.187+0000 7f961248d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f960c15f030 0x7f960c145d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:34.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.187+0000 7f961248d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f960c15f4c0 con 0x7f960c15f030 2024-09-16T01:34:34.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.187+0000 7f961248d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f960c06d3f0 con 0x7f960c15c2b0 2024-09-16T01:34:34.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.187+0000 7f961248d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f960c06de50 con 0x7f960c0b74c0 2024-09-16T01:34:34.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.187+0000 7f9610a2a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f960c15f030 0x7f960c145d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:34.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.187+0000 7f9610a2a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f960c15f030 0x7f960c145d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55246/0 (socket says 172.21.15.88:55246) 2024-09-16T01:34:34.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.187+0000 7f9610a2a640 1 -- 172.21.15.88:0/2052502321 learned_addr learned my addr 172.21.15.88:0/2052502321 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:34.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.187+0000 7f960bfff640 1 --2- 172.21.15.88:0/2052502321 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f960c0b74c0 0x7f960c157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:34.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.187+0000 7f9603fff640 1 --2- 172.21.15.88:0/2052502321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f960c15c2b0 0x7f960c1e6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:34.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.187+0000 7f9610a2a640 1 -- 172.21.15.88:0/2052502321 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f960c0b74c0 msgr2=0x7f960c157290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.187+0000 7f9610a2a640 1 --2- 172.21.15.88:0/2052502321 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f960c0b74c0 0x7f960c157290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.187+0000 7f9610a2a640 1 -- 172.21.15.88:0/2052502321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f960c15c2b0 msgr2=0x7f960c1e6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.187+0000 7f9610a2a640 1 --2- 172.21.15.88:0/2052502321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f960c15c2b0 0x7f960c1e6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.187+0000 7f9610a2a640 1 -- 172.21.15.88:0/2052502321 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f960c143dc0 con 0x7f960c15f030 2024-09-16T01:34:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.187+0000 7f9603fff640 1 --2- 172.21.15.88:0/2052502321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f960c15c2b0 0x7f960c1e6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.187+0000 7f960bfff640 1 --2- 172.21.15.88:0/2052502321 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f960c0b74c0 0x7f960c157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.188+0000 7f9610a2a640 1 --2- 172.21.15.88:0/2052502321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f960c15f030 0x7f960c145d20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9604007ea0 tx=0x7f9604007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.188+0000 7f96097fa640 1 -- 172.21.15.88:0/2052502321 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f960401a070 con 0x7f960c15f030 2024-09-16T01:34:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.188+0000 7f96097fa640 1 -- 172.21.15.88:0/2052502321 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9604003bb0 con 0x7f960c15f030 2024-09-16T01:34:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.188+0000 7f961248d640 1 -- 172.21.15.88:0/2052502321 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f960c1440a0 con 0x7f960c15f030 2024-09-16T01:34:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.188+0000 7f96097fa640 1 -- 172.21.15.88:0/2052502321 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f960400de70 con 0x7f960c15f030 2024-09-16T01:34:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.188+0000 7f961248d640 1 -- 172.21.15.88:0/2052502321 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f960c144660 con 0x7f960c15f030 2024-09-16T01:34:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.189+0000 7f961248d640 1 -- 172.21.15.88:0/2052502321 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f960c15eaf0 con 0x7f960c15f030 2024-09-16T01:34:34.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.189+0000 7f96097fa640 1 -- 172.21.15.88:0/2052502321 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f960401d070 con 0x7f960c15f030 2024-09-16T01:34:34.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.190+0000 7f96097fa640 1 --2- 172.21.15.88:0/2052502321 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f95ec06c2b0 0x7f95ec06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:34.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.190+0000 7f96097fa640 1 -- 172.21.15.88:0/2052502321 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f960408b630 con 0x7f960c15f030 2024-09-16T01:34:34.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.190+0000 7f960bfff640 1 --2- 172.21.15.88:0/2052502321 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f95ec06c2b0 0x7f95ec06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:34.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.190+0000 7f960bfff640 1 --2- 172.21.15.88:0/2052502321 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f95ec06c2b0 0x7f95ec06e770 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f95fc001700 tx=0x7f95fc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:34.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.194+0000 7f96097fa640 1 -- 172.21.15.88:0/2052502321 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9604010070 con 0x7f960c15f030 2024-09-16T01:34:34.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.397+0000 7f961248d640 1 -- 172.21.15.88:0/2052502321 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f960c0705e0 con 0x7f960c15f030 2024-09-16T01:34:34.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.397+0000 7f96097fa640 1 -- 172.21.15.88:0/2052502321 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f9604059e30 con 0x7f960c15f030 2024-09-16T01:34:34.399 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:34.400 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T01:34:34.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.399+0000 7f961248d640 1 -- 172.21.15.88:0/2052502321 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f95ec06c2b0 msgr2=0x7f95ec06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:34.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.399+0000 7f961248d640 1 --2- 172.21.15.88:0/2052502321 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f95ec06c2b0 0x7f95ec06e770 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f95fc001700 tx=0x7f95fc009040 comp rx=0 tx=0).stop 2024-09-16T01:34:34.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.399+0000 7f961248d640 1 -- 172.21.15.88:0/2052502321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f960c15f030 msgr2=0x7f960c145d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:34.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.399+0000 7f961248d640 1 --2- 172.21.15.88:0/2052502321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f960c15f030 0x7f960c145d20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9604007ea0 tx=0x7f9604007f90 comp rx=0 tx=0).stop 2024-09-16T01:34:34.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.399+0000 7f961248d640 1 -- 172.21.15.88:0/2052502321 shutdown_connections 2024-09-16T01:34:34.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.399+0000 7f961248d640 1 --2- 172.21.15.88:0/2052502321 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f95ec06c2b0 0x7f95ec06e770 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.399+0000 7f961248d640 1 --2- 172.21.15.88:0/2052502321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f960c15f030 0x7f960c145d20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.399+0000 7f961248d640 1 --2- 172.21.15.88:0/2052502321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f960c15c2b0 0x7f960c1e6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.399+0000 7f961248d640 1 --2- 172.21.15.88:0/2052502321 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f960c0b74c0 0x7f960c157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.399+0000 7f961248d640 1 -- 172.21.15.88:0/2052502321 >> 172.21.15.88:0/2052502321 conn(0x7f960c0b4240 msgr2=0x7f960c15fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:34.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.399+0000 7f961248d640 1 -- 172.21.15.88:0/2052502321 shutdown_connections 2024-09-16T01:34:34.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.399+0000 7f961248d640 1 -- 172.21.15.88:0/2052502321 wait complete. 2024-09-16T01:34:34.412 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T01:34:34.412 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:34:34.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.508+0000 7f5817e9b640 1 Processor -- start 2024-09-16T01:34:34.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.508+0000 7f5817e9b640 1 -- start start 2024-09-16T01:34:34.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.509+0000 7f5817e9b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58100cf6b0 0x7f58101629f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:34.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.509+0000 7f5817e9b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58100cfc20 con 0x7f58100b63d0 2024-09-16T01:34:34.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.509+0000 7f5817e9b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58100d2ed0 con 0x7f5810162f30 2024-09-16T01:34:34.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.509+0000 7f5817e9b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f58100d40c0 con 0x7f58100cf6b0 2024-09-16T01:34:34.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.509+0000 7f5815c37640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f58100b63d0 0x7f58100d39b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45250/0 (socket says 172.21.15.88:45250) 2024-09-16T01:34:34.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.509+0000 7f5815c37640 1 -- 172.21.15.88:0/3749160959 learned_addr learned my addr 172.21.15.88:0/3749160959 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:34.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.509+0000 7f5815436640 1 --2- 172.21.15.88:0/3749160959 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58100cf6b0 0x7f58101629f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:34.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.509+0000 7f5814c35640 1 -- 172.21.15.88:0/3749160959 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3259183822 0 0) 0x7f58100cfc20 con 0x7f58100b63d0 2024-09-16T01:34:34.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.509+0000 7f5814c35640 1 -- 172.21.15.88:0/3749160959 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57fc003520 con 0x7f58100b63d0 2024-09-16T01:34:34.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.509+0000 7f5814c35640 1 -- 172.21.15.88:0/3749160959 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3288183029 0 0) 0x7f57fc003520 con 0x7f58100b63d0 2024-09-16T01:34:34.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.509+0000 7f5814c35640 1 -- 172.21.15.88:0/3749160959 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58100cf6b0 msgr2=0x7f58101629f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:34.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.509+0000 7f5814c35640 1 --2- 172.21.15.88:0/3749160959 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58100cf6b0 0x7f58101629f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.509+0000 7f5814c35640 1 -- 172.21.15.88:0/3749160959 >> v1:172.21.15.154:6789/0 conn(0x7f5810162f30 legacy=0x7f58100b7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:34.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.509+0000 7f5814c35640 1 -- 172.21.15.88:0/3749160959 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58100d31b0 con 0x7f58100b63d0 2024-09-16T01:34:34.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.509+0000 7f5814c35640 1 -- 172.21.15.88:0/3749160959 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f580c001670 con 0x7f58100b63d0 2024-09-16T01:34:34.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.509+0000 7f5814c35640 1 -- 172.21.15.88:0/3749160959 >> v1:172.21.15.88:6789/0 conn(0x7f58100b63d0 legacy=0x7f58100d39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:34.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.509+0000 7f5814c35640 1 --2- 172.21.15.88:0/3749160959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57fc0039e0 0x7f57fc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:34.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.509+0000 7f5814c35640 1 --2- 172.21.15.88:0/3749160959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57fc004390 0x7f57fc004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:34.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.509+0000 7f5814c35640 1 --2- 172.21.15.88:0/3749160959 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58100cf6b0 0x7f57fc0050f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:34.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.509+0000 7f5814c35640 1 -- 172.21.15.88:0/3749160959 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f58100d40c0 con 0x7f57fc0039e0 2024-09-16T01:34:34.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.509+0000 7f5814c35640 1 -- 172.21.15.88:0/3749160959 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f57fc00ba20 con 0x7f57fc004390 2024-09-16T01:34:34.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.509+0000 7f5814c35640 1 -- 172.21.15.88:0/3749160959 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f57fc00bc20 con 0x7f58100cf6b0 2024-09-16T01:34:34.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.509+0000 7f5815c37640 1 --2- 172.21.15.88:0/3749160959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57fc0039e0 0x7f57fc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:34.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.510+0000 7f5815436640 1 --2- 172.21.15.88:0/3749160959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57fc004390 0x7f57fc004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:34.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.510+0000 7f5816438640 1 --2- 172.21.15.88:0/3749160959 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58100cf6b0 0x7f57fc0050f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:34.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.510+0000 7f5815c37640 1 -- 172.21.15.88:0/3749160959 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58100cf6b0 msgr2=0x7f57fc0050f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:34.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.510+0000 7f5815c37640 1 --2- 172.21.15.88:0/3749160959 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58100cf6b0 0x7f57fc0050f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.510+0000 7f5815c37640 1 -- 172.21.15.88:0/3749160959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57fc004390 msgr2=0x7f57fc004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:34.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.510+0000 7f5815c37640 1 --2- 172.21.15.88:0/3749160959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57fc004390 0x7f57fc004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.510+0000 7f5815c37640 1 -- 172.21.15.88:0/3749160959 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57fc003520 con 0x7f57fc0039e0 2024-09-16T01:34:34.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.510+0000 7f5816438640 1 --2- 172.21.15.88:0/3749160959 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58100cf6b0 0x7f57fc0050f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:34.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.510+0000 7f5815c37640 1 --2- 172.21.15.88:0/3749160959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57fc0039e0 0x7f57fc003dc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f580c007fd0 tx=0x7f57fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c8a8abfa52244f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:34.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.510+0000 7f5814c35640 1 -- 172.21.15.88:0/3749160959 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580c007d50 con 0x7f57fc0039e0 2024-09-16T01:34:34.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.511+0000 7f5814c35640 1 -- 172.21.15.88:0/3749160959 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f580c0058d0 con 0x7f57fc0039e0 2024-09-16T01:34:34.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.511+0000 7f5814c35640 1 -- 172.21.15.88:0/3749160959 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580c005ca0 con 0x7f57fc0039e0 2024-09-16T01:34:34.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.511+0000 7f5817e9b640 1 -- 172.21.15.88:0/3749160959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57fc0039e0 msgr2=0x7f57fc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:34.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.511+0000 7f5817e9b640 1 --2- 172.21.15.88:0/3749160959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57fc0039e0 0x7f57fc003dc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f580c007fd0 tx=0x7f57fc005830 comp rx=0 tx=0).stop 2024-09-16T01:34:34.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.511+0000 7f5816438640 1 -- 172.21.15.88:0/3749160959 reap_dead start 2024-09-16T01:34:34.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.511+0000 7f5817e9b640 1 -- 172.21.15.88:0/3749160959 shutdown_connections 2024-09-16T01:34:34.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.511+0000 7f5817e9b640 1 -- 172.21.15.88:0/3749160959 >> 172.21.15.88:0/3749160959 conn(0x7f58100d1160 msgr2=0x7f58100d1570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:34.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.511+0000 7f5817e9b640 1 -- 172.21.15.88:0/3749160959 shutdown_connections 2024-09-16T01:34:34.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.511+0000 7f5817e9b640 1 -- 172.21.15.88:0/3749160959 wait complete. 2024-09-16T01:34:34.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.512+0000 7f5817e9b640 1 Processor -- start 2024-09-16T01:34:34.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.512+0000 7f5817e9b640 1 -- start start 2024-09-16T01:34:34.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.513+0000 7f5817e9b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57fc0039e0 0x7f58100cde10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:34.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.513+0000 7f5817e9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f581015b4a0 0x7f58100cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:34.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.513+0000 7f5817e9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f581014fa00 0x7f58100ca790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:34.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.513+0000 7f5817e9b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f58100cb5c0 con 0x7f581015b4a0 2024-09-16T01:34:34.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.513+0000 7f5817e9b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f58100cb440 con 0x7f581014fa00 2024-09-16T01:34:34.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.513+0000 7f5817e9b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f58100d1c80 con 0x7f57fc0039e0 2024-09-16T01:34:34.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.513+0000 7f5815436640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f581015b4a0 0x7f58100cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:34.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.513+0000 7f5815436640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f581015b4a0 0x7f58100cc1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55264/0 (socket says 172.21.15.88:55264) 2024-09-16T01:34:34.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.513+0000 7f5815436640 1 -- 172.21.15.88:0/2259485500 learned_addr learned my addr 172.21.15.88:0/2259485500 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:34.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.513+0000 7f5816438640 1 --2- 172.21.15.88:0/2259485500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f581014fa00 0x7f58100ca790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:34.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.513+0000 7f5815436640 1 -- 172.21.15.88:0/2259485500 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57fc0039e0 msgr2=0x7f58100cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:34.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.513+0000 7f5815c37640 1 --2- 172.21.15.88:0/2259485500 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57fc0039e0 0x7f58100cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:34.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.513+0000 7f5815436640 1 --2- 172.21.15.88:0/2259485500 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57fc0039e0 0x7f58100cde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.513+0000 7f5815436640 1 -- 172.21.15.88:0/2259485500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f581014fa00 msgr2=0x7f58100ca790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:34.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.513+0000 7f5815436640 1 --2- 172.21.15.88:0/2259485500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f581014fa00 0x7f58100ca790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.513+0000 7f5815436640 1 -- 172.21.15.88:0/2259485500 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58100cc9f0 con 0x7f581015b4a0 2024-09-16T01:34:34.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.513+0000 7f5816438640 1 --2- 172.21.15.88:0/2259485500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f581014fa00 0x7f58100ca790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:34.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.513+0000 7f5815c37640 1 --2- 172.21.15.88:0/2259485500 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57fc0039e0 0x7f58100cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:34.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.514+0000 7f5815436640 1 --2- 172.21.15.88:0/2259485500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f581015b4a0 0x7f58100cc1c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f58040119a0 tx=0x7f5804011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:34.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.514+0000 7f58027fc640 1 -- 172.21.15.88:0/2259485500 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580400faf0 con 0x7f581015b4a0 2024-09-16T01:34:34.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.514+0000 7f58027fc640 1 -- 172.21.15.88:0/2259485500 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f580400fc90 con 0x7f581015b4a0 2024-09-16T01:34:34.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.514+0000 7f58027fc640 1 -- 172.21.15.88:0/2259485500 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5804014440 con 0x7f581015b4a0 2024-09-16T01:34:34.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.514+0000 7f5817e9b640 1 -- 172.21.15.88:0/2259485500 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58100c8420 con 0x7f581015b4a0 2024-09-16T01:34:34.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.514+0000 7f5817e9b640 1 -- 172.21.15.88:0/2259485500 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58100cc700 con 0x7f581015b4a0 2024-09-16T01:34:34.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.515+0000 7f5817e9b640 1 -- 172.21.15.88:0/2259485500 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58100b67b0 con 0x7f581015b4a0 2024-09-16T01:34:34.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.515+0000 7f58027fc640 1 -- 172.21.15.88:0/2259485500 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5804012040 con 0x7f581015b4a0 2024-09-16T01:34:34.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.516+0000 7f58027fc640 1 --2- 172.21.15.88:0/2259485500 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f57f406c2b0 0x7f57f406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:34.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.516+0000 7f5815c37640 1 --2- 172.21.15.88:0/2259485500 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f57f406c2b0 0x7f57f406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:34.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.516+0000 7f58027fc640 1 -- 172.21.15.88:0/2259485500 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f580408b9b0 con 0x7f581015b4a0 2024-09-16T01:34:34.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.516+0000 7f5815c37640 1 --2- 172.21.15.88:0/2259485500 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f57f406c2b0 0x7f57f406e770 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f580c033000 tx=0x7f580c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:34.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.520+0000 7f58027fc640 1 -- 172.21.15.88:0/2259485500 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5804019030 con 0x7f581015b4a0 2024-09-16T01:34:34.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.721+0000 7f5817e9b640 1 -- 172.21.15.88:0/2259485500 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f58100cf280 con 0x7f581015b4a0 2024-09-16T01:34:34.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.721+0000 7f58027fc640 1 -- 172.21.15.88:0/2259485500 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5223 (secure 0 0 0) 0x7f5804059b60 con 0x7f581015b4a0 2024-09-16T01:34:34.724 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:34.724 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":6,"btime":"2024-09-16T01:34:24:186280+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/1620784263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1620784263},{"type":"v1","addr":"172.21.15.154:6834","nonce":1620784263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3407296593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":3407296593},{"type":"v1","addr":"172.21.15.154:6835","nonce":3407296593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4293,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/4058543372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4058543372},{"type":"v1","addr":"172.21.15.88:6835","nonce":4058543372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:34:21.508480+0000","modified":"2024-09-16T01:34:23.515443+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6837/3712834803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":3712834803},{"type":"v1","addr":"172.21.15.88:6837","nonce":3712834803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4296,"qdb_cluster":[4296]},"id":1}]} 2024-09-16T01:34:34.724 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 6 2024-09-16T01:34:34.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.723+0000 7f5817e9b640 1 -- 172.21.15.88:0/2259485500 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f57f406c2b0 msgr2=0x7f57f406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:34.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.723+0000 7f5817e9b640 1 --2- 172.21.15.88:0/2259485500 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f57f406c2b0 0x7f57f406e770 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f580c033000 tx=0x7f580c02e000 comp rx=0 tx=0).stop 2024-09-16T01:34:34.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.723+0000 7f5817e9b640 1 -- 172.21.15.88:0/2259485500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f581015b4a0 msgr2=0x7f58100cc1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:34.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.723+0000 7f5817e9b640 1 --2- 172.21.15.88:0/2259485500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f581015b4a0 0x7f58100cc1c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f58040119a0 tx=0x7f5804011e60 comp rx=0 tx=0).stop 2024-09-16T01:34:34.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.723+0000 7f5817e9b640 1 -- 172.21.15.88:0/2259485500 shutdown_connections 2024-09-16T01:34:34.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.723+0000 7f5817e9b640 1 --2- 172.21.15.88:0/2259485500 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f57f406c2b0 0x7f57f406e770 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.723+0000 7f5817e9b640 1 --2- 172.21.15.88:0/2259485500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f581014fa00 0x7f58100ca790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.723+0000 7f5817e9b640 1 --2- 172.21.15.88:0/2259485500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f581015b4a0 0x7f58100cc1c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.723+0000 7f5817e9b640 1 --2- 172.21.15.88:0/2259485500 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57fc0039e0 0x7f58100cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.723+0000 7f5817e9b640 1 -- 172.21.15.88:0/2259485500 >> 172.21.15.88:0/2259485500 conn(0x7f58100d1160 msgr2=0x7f58101617b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:34.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.723+0000 7f5817e9b640 1 -- 172.21.15.88:0/2259485500 shutdown_connections 2024-09-16T01:34:34.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.723+0000 7f5817e9b640 1 -- 172.21.15.88:0/2259485500 wait complete. 2024-09-16T01:34:34.736 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:34:34.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.832+0000 7f0052d6d640 1 Processor -- start 2024-09-16T01:34:34.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.832+0000 7f0052d6d640 1 -- start start 2024-09-16T01:34:34.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.833+0000 7f0052d6d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f004c0d1b40 0x7f004c1618a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:34.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.833+0000 7f0052d6d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f004c0708d0 con 0x7f004c0b7290 2024-09-16T01:34:34.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.833+0000 7f0052d6d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f004c070bb0 con 0x7f004c0b6eb0 2024-09-16T01:34:34.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.833+0000 7f0052d6d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f004c070e90 con 0x7f004c0d1b40 2024-09-16T01:34:34.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.833+0000 7f005130a640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f004c0b7290 0x7f004c161de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45260/0 (socket says 172.21.15.88:45260) 2024-09-16T01:34:34.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.833+0000 7f005130a640 1 -- 172.21.15.88:0/931748470 learned_addr learned my addr 172.21.15.88:0/931748470 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:34.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.833+0000 7f004bfff640 1 --2- 172.21.15.88:0/931748470 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f004c0d1b40 0x7f004c1618a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:34.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.833+0000 7f004b7fe640 1 -- 172.21.15.88:0/931748470 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2925844743 0 0) 0x7f004c0708d0 con 0x7f004c0b7290 2024-09-16T01:34:34.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.833+0000 7f004b7fe640 1 -- 172.21.15.88:0/931748470 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0034003520 con 0x7f004c0b7290 2024-09-16T01:34:34.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.834+0000 7f004b7fe640 1 -- 172.21.15.88:0/931748470 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1111327627 0 0) 0x7f004c070bb0 con 0x7f004c0b6eb0 2024-09-16T01:34:34.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.834+0000 7f004bfff640 1 -- 172.21.15.88:0/931748470 >> v1:172.21.15.154:6789/0 conn(0x7f004c0b6eb0 legacy=0x7f004c06fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:34.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.834+0000 7f004bfff640 1 -- 172.21.15.88:0/931748470 >> v1:172.21.15.88:6789/0 conn(0x7f004c0b7290 legacy=0x7f004c161de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:34.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.834+0000 7f004bfff640 1 -- 172.21.15.88:0/931748470 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f004c071170 con 0x7f004c0d1b40 2024-09-16T01:34:34.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.834+0000 7f004bfff640 1 --2- 172.21.15.88:0/931748470 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f004c0d1b40 0x7f004c1618a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f00400097d0 tx=0x7f004002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ed16bbe52ad6ae5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:34.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.834+0000 7f004b7fe640 1 -- 172.21.15.88:0/931748470 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0040037070 con 0x7f004c0d1b40 2024-09-16T01:34:34.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.835+0000 7f004b7fe640 1 -- 172.21.15.88:0/931748470 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0040032440 con 0x7f004c0d1b40 2024-09-16T01:34:34.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.835+0000 7f004b7fe640 1 -- 172.21.15.88:0/931748470 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0040032810 con 0x7f004c0d1b40 2024-09-16T01:34:34.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.835+0000 7f0052d6d640 1 -- 172.21.15.88:0/931748470 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f004c0d1b40 msgr2=0x7f004c1618a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:34.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.835+0000 7f0052d6d640 1 --2- 172.21.15.88:0/931748470 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f004c0d1b40 0x7f004c1618a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f00400097d0 tx=0x7f004002a180 comp rx=0 tx=0).stop 2024-09-16T01:34:34.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.835+0000 7f0052d6d640 1 -- 172.21.15.88:0/931748470 shutdown_connections 2024-09-16T01:34:34.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.835+0000 7f0052d6d640 1 --2- 172.21.15.88:0/931748470 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f004c0d1b40 0x7f004c1618a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.835+0000 7f0052d6d640 1 -- 172.21.15.88:0/931748470 >> 172.21.15.88:0/931748470 conn(0x7f004c0d2450 msgr2=0x7f004c0d2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:34.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.835+0000 7f0052d6d640 1 -- 172.21.15.88:0/931748470 shutdown_connections 2024-09-16T01:34:34.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.835+0000 7f0052d6d640 1 -- 172.21.15.88:0/931748470 wait complete. 2024-09-16T01:34:34.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.836+0000 7f0052d6d640 1 Processor -- start 2024-09-16T01:34:34.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.836+0000 7f0052d6d640 1 -- start start 2024-09-16T01:34:34.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.837+0000 7f0052d6d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f004c0b6eb0 0x7f004c145e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:34.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.837+0000 7f0052d6d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f004c0b7290 0x7f004c155a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:34.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.837+0000 7f0052d6d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f004c0d1b40 0x7f004c155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:34.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.837+0000 7f0052d6d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f004c071af0 con 0x7f004c0b6eb0 2024-09-16T01:34:34.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.837+0000 7f0052d6d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f004c071970 con 0x7f004c0b7290 2024-09-16T01:34:34.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.837+0000 7f0052d6d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f004c1624f0 con 0x7f004c0d1b40 2024-09-16T01:34:34.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.837+0000 7f0050b09640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f004c0b6eb0 0x7f004c145e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:34.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.837+0000 7f0050b09640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f004c0b6eb0 0x7f004c145e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55278/0 (socket says 172.21.15.88:55278) 2024-09-16T01:34:34.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.837+0000 7f0050b09640 1 -- 172.21.15.88:0/2412641896 learned_addr learned my addr 172.21.15.88:0/2412641896 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:34.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.837+0000 7f005130a640 1 --2- 172.21.15.88:0/2412641896 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f004c0d1b40 0x7f004c155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:34.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.837+0000 7f004bfff640 1 --2- 172.21.15.88:0/2412641896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f004c0b7290 0x7f004c155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:34.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.837+0000 7f0050b09640 1 -- 172.21.15.88:0/2412641896 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f004c0d1b40 msgr2=0x7f004c155fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:34.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.837+0000 7f0050b09640 1 --2- 172.21.15.88:0/2412641896 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f004c0d1b40 0x7f004c155fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.838+0000 7f0050b09640 1 -- 172.21.15.88:0/2412641896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f004c0b7290 msgr2=0x7f004c155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:34.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.838+0000 7f0050b09640 1 --2- 172.21.15.88:0/2412641896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f004c0b7290 0x7f004c155a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:34.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.838+0000 7f0050b09640 1 -- 172.21.15.88:0/2412641896 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f004c157ca0 con 0x7f004c0b6eb0 2024-09-16T01:34:34.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.838+0000 7f0050b09640 1 --2- 172.21.15.88:0/2412641896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f004c0b6eb0 0x7f004c145e40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f003c00c4e0 tx=0x7f003c00c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:34.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.838+0000 7f00497fa640 1 -- 172.21.15.88:0/2412641896 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c017070 con 0x7f004c0b6eb0 2024-09-16T01:34:34.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.838+0000 7f00497fa640 1 -- 172.21.15.88:0/2412641896 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f003c0035f0 con 0x7f004c0b6eb0 2024-09-16T01:34:34.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.838+0000 7f0052d6d640 1 -- 172.21.15.88:0/2412641896 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f004c157f80 con 0x7f004c0b6eb0 2024-09-16T01:34:34.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.838+0000 7f00497fa640 1 -- 172.21.15.88:0/2412641896 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c0039e0 con 0x7f004c0b6eb0 2024-09-16T01:34:34.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.838+0000 7f0052d6d640 1 -- 172.21.15.88:0/2412641896 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f004c158540 con 0x7f004c0b6eb0 2024-09-16T01:34:34.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.839+0000 7f0052d6d640 1 -- 172.21.15.88:0/2412641896 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f004c1628e0 con 0x7f004c0b6eb0 2024-09-16T01:34:34.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.839+0000 7f00497fa640 1 -- 172.21.15.88:0/2412641896 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f003c01a070 con 0x7f004c0b6eb0 2024-09-16T01:34:34.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.840+0000 7f00497fa640 1 --2- 172.21.15.88:0/2412641896 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f003806c2b0 0x7f003806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:34.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.840+0000 7f004bfff640 1 --2- 172.21.15.88:0/2412641896 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f003806c2b0 0x7f003806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:34.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.840+0000 7f00497fa640 1 -- 172.21.15.88:0/2412641896 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f003c08a680 con 0x7f004c0b6eb0 2024-09-16T01:34:34.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.840+0000 7f004bfff640 1 --2- 172.21.15.88:0/2412641896 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f003806c2b0 0x7f003806e770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0040034000 tx=0x7f004002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:34.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:34.844+0000 7f00497fa640 1 -- 172.21.15.88:0/2412641896 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f003c058830 con 0x7f004c0b6eb0 2024-09-16T01:34:35.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.042+0000 7f0052d6d640 1 -- 172.21.15.88:0/2412641896 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f004c0b19b0 con 0x7f004c0b6eb0 2024-09-16T01:34:35.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.042+0000 7f00497fa640 1 -- 172.21.15.88:0/2412641896 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5223 (secure 0 0 0) 0x7f003c05cb30 con 0x7f004c0b6eb0 2024-09-16T01:34:35.045 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:35.045 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":6,"btime":"2024-09-16T01:34:24:186280+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/1620784263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1620784263},{"type":"v1","addr":"172.21.15.154:6834","nonce":1620784263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3407296593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":3407296593},{"type":"v1","addr":"172.21.15.154:6835","nonce":3407296593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4293,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/4058543372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4058543372},{"type":"v1","addr":"172.21.15.88:6835","nonce":4058543372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:34:21.508480+0000","modified":"2024-09-16T01:34:23.515443+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6837/3712834803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":3712834803},{"type":"v1","addr":"172.21.15.88:6837","nonce":3712834803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4296,"qdb_cluster":[4296]},"id":1}]} 2024-09-16T01:34:35.045 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 6 2024-09-16T01:34:35.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.045+0000 7f0052d6d640 1 -- 172.21.15.88:0/2412641896 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f003806c2b0 msgr2=0x7f003806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:35.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.045+0000 7f0052d6d640 1 --2- 172.21.15.88:0/2412641896 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f003806c2b0 0x7f003806e770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0040034000 tx=0x7f004002e000 comp rx=0 tx=0).stop 2024-09-16T01:34:35.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.045+0000 7f0052d6d640 1 -- 172.21.15.88:0/2412641896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f004c0b6eb0 msgr2=0x7f004c145e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:35.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.045+0000 7f0052d6d640 1 --2- 172.21.15.88:0/2412641896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f004c0b6eb0 0x7f004c145e40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f003c00c4e0 tx=0x7f003c00c9a0 comp rx=0 tx=0).stop 2024-09-16T01:34:35.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.045+0000 7f0052d6d640 1 -- 172.21.15.88:0/2412641896 shutdown_connections 2024-09-16T01:34:35.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.045+0000 7f0052d6d640 1 --2- 172.21.15.88:0/2412641896 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f003806c2b0 0x7f003806e770 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:35.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.045+0000 7f0052d6d640 1 --2- 172.21.15.88:0/2412641896 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f004c0d1b40 0x7f004c155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:35.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.045+0000 7f0052d6d640 1 --2- 172.21.15.88:0/2412641896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f004c0b7290 0x7f004c155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:35.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.045+0000 7f0052d6d640 1 --2- 172.21.15.88:0/2412641896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f004c0b6eb0 0x7f004c145e40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:35.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.045+0000 7f0052d6d640 1 -- 172.21.15.88:0/2412641896 >> 172.21.15.88:0/2412641896 conn(0x7f004c0d2450 msgr2=0x7f004c06fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:35.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.045+0000 7f0052d6d640 1 -- 172.21.15.88:0/2412641896 shutdown_connections 2024-09-16T01:34:35.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.045+0000 7f0052d6d640 1 -- 172.21.15.88:0/2412641896 wait complete. 2024-09-16T01:34:35.058 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:34:35.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.153+0000 7f71d62f1640 1 Processor -- start 2024-09-16T01:34:35.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.153+0000 7f71d62f1640 1 -- start start 2024-09-16T01:34:35.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.154+0000 7f71d62f1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71d00b63d0 0x7f71d00d39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:35.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.154+0000 7f71d62f1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f71d00d1c80 0x7f71d00b7f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:35.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.154+0000 7f71d62f1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71d00cfc20 con 0x7f71d00cf6b0 2024-09-16T01:34:35.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.154+0000 7f71d62f1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f71d00d2ed0 con 0x7f71d00d1c80 2024-09-16T01:34:35.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.154+0000 7f71d62f1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f71d00d31b0 con 0x7f71d00b63d0 2024-09-16T01:34:35.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.154+0000 7f71d488e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f71d00d1c80 0x7f71d00b7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:35.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.154+0000 7f71cf7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f71d00cf6b0 0x7f71d00b7880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45262/0 (socket says 172.21.15.88:45262) 2024-09-16T01:34:35.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.154+0000 7f71d488e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f71d00d1c80 0x7f71d00b7f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55288/0 (socket says 172.21.15.88:55288) 2024-09-16T01:34:35.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.154+0000 7f71cf7fe640 1 -- 172.21.15.88:0/2413092028 learned_addr learned my addr 172.21.15.88:0/2413092028 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:35.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.154+0000 7f71cffff640 1 --2- 172.21.15.88:0/2413092028 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71d00b63d0 0x7f71d00d39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:35.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.154+0000 7f71d488e640 1 -- 172.21.15.88:0/2413092028 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71d00b63d0 msgr2=0x7f71d00d39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:35.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.154+0000 7f71d488e640 1 --2- 172.21.15.88:0/2413092028 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71d00b63d0 0x7f71d00d39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:35.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.154+0000 7f71d488e640 1 -- 172.21.15.88:0/2413092028 >> v1:172.21.15.88:6789/0 conn(0x7f71d00cf6b0 legacy=0x7f71d00b7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:35.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.154+0000 7f71d488e640 1 -- 172.21.15.88:0/2413092028 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71d00d3490 con 0x7f71d00d1c80 2024-09-16T01:34:35.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.154+0000 7f71ceffd640 1 -- 172.21.15.88:0/2413092028 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3177095590 0 0) 0x7f71d00cfc20 con 0x7f71d00cf6b0 2024-09-16T01:34:35.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.155+0000 7f71d488e640 1 --2- 172.21.15.88:0/2413092028 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71d00d1c80 0x7f71d00b7f90 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f71c8009d90 tx=0x7f71c802a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b17849d25a8ed194 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:35.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.155+0000 7f71ceffd640 1 -- 172.21.15.88:0/2413092028 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c8007740 con 0x7f71d00d1c80 2024-09-16T01:34:35.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.155+0000 7f71ceffd640 1 -- 172.21.15.88:0/2413092028 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71c80078e0 con 0x7f71d00d1c80 2024-09-16T01:34:35.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.155+0000 7f71ceffd640 1 -- 172.21.15.88:0/2413092028 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c8007cb0 con 0x7f71d00d1c80 2024-09-16T01:34:35.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.156+0000 7f71d62f1640 1 -- 172.21.15.88:0/2413092028 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71d00d1c80 msgr2=0x7f71d00b7f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:35.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.156+0000 7f71d62f1640 1 --2- 172.21.15.88:0/2413092028 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71d00d1c80 0x7f71d00b7f90 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f71c8009d90 tx=0x7f71c802a530 comp rx=0 tx=0).stop 2024-09-16T01:34:35.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.156+0000 7f71d62f1640 1 -- 172.21.15.88:0/2413092028 shutdown_connections 2024-09-16T01:34:35.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.156+0000 7f71d62f1640 1 --2- 172.21.15.88:0/2413092028 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71d00d1c80 0x7f71d00b7f90 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:35.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.156+0000 7f71d62f1640 1 --2- 172.21.15.88:0/2413092028 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71d00b63d0 0x7f71d00d39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:35.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.156+0000 7f71d62f1640 1 -- 172.21.15.88:0/2413092028 >> 172.21.15.88:0/2413092028 conn(0x7f71d00d1160 msgr2=0x7f71d00d1570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:35.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.156+0000 7f71d62f1640 1 -- 172.21.15.88:0/2413092028 shutdown_connections 2024-09-16T01:34:35.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.156+0000 7f71d62f1640 1 -- 172.21.15.88:0/2413092028 wait complete. 2024-09-16T01:34:35.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.157+0000 7f71d62f1640 1 Processor -- start 2024-09-16T01:34:35.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.157+0000 7f71d62f1640 1 -- start start 2024-09-16T01:34:35.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.158+0000 7f71d62f1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71d00b63d0 0x7f71d0157480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:35.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.158+0000 7f71d62f1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71d00cf6b0 0x7f71d015b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:35.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.158+0000 7f71d62f1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71d00d1c80 0x7f71d01f4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:35.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.158+0000 7f71d62f1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f71d00d4070 con 0x7f71d00cf6b0 2024-09-16T01:34:35.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.158+0000 7f71d62f1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f71d00d3ef0 con 0x7f71d00b63d0 2024-09-16T01:34:35.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.158+0000 7f71d62f1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f71d00b67b0 con 0x7f71d00d1c80 2024-09-16T01:34:35.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.158+0000 7f71cf7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71d00cf6b0 0x7f71d015b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:35.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.158+0000 7f71cf7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71d00cf6b0 0x7f71d015b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55298/0 (socket says 172.21.15.88:55298) 2024-09-16T01:34:35.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.158+0000 7f71cf7fe640 1 -- 172.21.15.88:0/3848281042 learned_addr learned my addr 172.21.15.88:0/3848281042 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:35.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.158+0000 7f71cffff640 1 --2- 172.21.15.88:0/3848281042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71d00b63d0 0x7f71d0157480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:35.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.158+0000 7f71d488e640 1 --2- 172.21.15.88:0/3848281042 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71d00d1c80 0x7f71d01f4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:35.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.158+0000 7f71cf7fe640 1 -- 172.21.15.88:0/3848281042 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71d00d1c80 msgr2=0x7f71d01f4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:35.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.158+0000 7f71cf7fe640 1 --2- 172.21.15.88:0/3848281042 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71d00d1c80 0x7f71d01f4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:35.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.158+0000 7f71cf7fe640 1 -- 172.21.15.88:0/3848281042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71d00b63d0 msgr2=0x7f71d0157480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:35.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.158+0000 7f71cf7fe640 1 --2- 172.21.15.88:0/3848281042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71d00b63d0 0x7f71d0157480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:35.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.158+0000 7f71cf7fe640 1 -- 172.21.15.88:0/3848281042 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71d015c4e0 con 0x7f71d00cf6b0 2024-09-16T01:34:35.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.158+0000 7f71d488e640 1 --2- 172.21.15.88:0/3848281042 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71d00d1c80 0x7f71d01f4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:35.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.158+0000 7f71cf7fe640 1 --2- 172.21.15.88:0/3848281042 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71d00cf6b0 0x7f71d015b970 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f71c400c4d0 tx=0x7f71c400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:35.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.158+0000 7f71ccff9640 1 -- 172.21.15.88:0/3848281042 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c4018070 con 0x7f71d00cf6b0 2024-09-16T01:34:35.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.158+0000 7f71ccff9640 1 -- 172.21.15.88:0/3848281042 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71c4007500 con 0x7f71d00cf6b0 2024-09-16T01:34:35.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.158+0000 7f71d62f1640 1 -- 172.21.15.88:0/3848281042 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71d015c7c0 con 0x7f71d00cf6b0 2024-09-16T01:34:35.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.158+0000 7f71ccff9640 1 -- 172.21.15.88:0/3848281042 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c4004a20 con 0x7f71d00cf6b0 2024-09-16T01:34:35.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.159+0000 7f71d62f1640 1 -- 172.21.15.88:0/3848281042 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71d01f4740 con 0x7f71d00cf6b0 2024-09-16T01:34:35.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.159+0000 7f71d62f1640 1 -- 172.21.15.88:0/3848281042 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71d00d39b0 con 0x7f71d00cf6b0 2024-09-16T01:34:35.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.160+0000 7f71ccff9640 1 -- 172.21.15.88:0/3848281042 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f71c401b070 con 0x7f71d00cf6b0 2024-09-16T01:34:35.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.160+0000 7f71ccff9640 1 --2- 172.21.15.88:0/3848281042 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f71bc06c2b0 0x7f71bc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:35.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.160+0000 7f71cffff640 1 --2- 172.21.15.88:0/3848281042 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f71bc06c2b0 0x7f71bc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:35.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.161+0000 7f71ccff9640 1 -- 172.21.15.88:0/3848281042 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f71c408a670 con 0x7f71d00cf6b0 2024-09-16T01:34:35.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.161+0000 7f71cffff640 1 --2- 172.21.15.88:0/3848281042 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f71bc06c2b0 0x7f71bc06e770 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f71c0006fd0 tx=0x7f71c0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:35.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.165+0000 7f71ccff9640 1 -- 172.21.15.88:0/3848281042 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f71c4058820 con 0x7f71d00cf6b0 2024-09-16T01:34:35.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.335+0000 7f71d62f1640 1 -- 172.21.15.88:0/3848281042 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f71d00cf280 con 0x7f71d00cf6b0 2024-09-16T01:34:35.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.336+0000 7f71ccff9640 1 -- 172.21.15.88:0/3848281042 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16641 (secure 0 0 0) 0x7f71c405cb20 con 0x7f71d00cf6b0 2024-09-16T01:34:35.338 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:35.338 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:34:23.181812+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:34:19.324549+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:34:20.494051+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:34:35.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.338+0000 7f71d62f1640 1 -- 172.21.15.88:0/3848281042 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f71bc06c2b0 msgr2=0x7f71bc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:35.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.338+0000 7f71d62f1640 1 --2- 172.21.15.88:0/3848281042 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f71bc06c2b0 0x7f71bc06e770 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f71c0006fd0 tx=0x7f71c0008040 comp rx=0 tx=0).stop 2024-09-16T01:34:35.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.338+0000 7f71d62f1640 1 -- 172.21.15.88:0/3848281042 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71d00cf6b0 msgr2=0x7f71d015b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:35.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.338+0000 7f71d62f1640 1 --2- 172.21.15.88:0/3848281042 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71d00cf6b0 0x7f71d015b970 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f71c400c4d0 tx=0x7f71c400c990 comp rx=0 tx=0).stop 2024-09-16T01:34:35.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.338+0000 7f71d62f1640 1 -- 172.21.15.88:0/3848281042 shutdown_connections 2024-09-16T01:34:35.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.338+0000 7f71d62f1640 1 --2- 172.21.15.88:0/3848281042 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f71bc06c2b0 0x7f71bc06e770 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:35.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.338+0000 7f71d62f1640 1 --2- 172.21.15.88:0/3848281042 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71d00d1c80 0x7f71d01f4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:35.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.338+0000 7f71d62f1640 1 --2- 172.21.15.88:0/3848281042 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71d00cf6b0 0x7f71d015b970 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:35.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.338+0000 7f71d62f1640 1 --2- 172.21.15.88:0/3848281042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71d00b63d0 0x7f71d0157480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:35.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.338+0000 7f71d62f1640 1 -- 172.21.15.88:0/3848281042 >> 172.21.15.88:0/3848281042 conn(0x7f71d00d1160 msgr2=0x7f71d01617b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:35.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.338+0000 7f71d62f1640 1 -- 172.21.15.88:0/3848281042 shutdown_connections 2024-09-16T01:34:35.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.338+0000 7f71d62f1640 1 -- 172.21.15.88:0/3848281042 wait complete. 2024-09-16T01:34:35.351 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:34:35.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.447+0000 7f4711b8d640 1 Processor -- start 2024-09-16T01:34:35.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.448+0000 7f4711b8d640 1 -- start start 2024-09-16T01:34:35.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.448+0000 7f4711b8d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f470c0d3310 0x7f470c0d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:35.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.448+0000 7f4711b8d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f470c15ea40 con 0x7f470c0d1650 2024-09-16T01:34:35.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.448+0000 7f4711b8d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f470c0d3870 con 0x7f470c15e660 2024-09-16T01:34:35.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.448+0000 7f4711b8d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f470c0d1ac0 con 0x7f470c0d3310 2024-09-16T01:34:35.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.448+0000 7f470affd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f470c0d3310 0x7f470c0d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:35.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.449+0000 7f470affd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f470c0d3310 0x7f470c0d1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55304/0 (socket says 172.21.15.88:55304) 2024-09-16T01:34:35.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.449+0000 7f470affd640 1 -- 172.21.15.88:0/1512745398 learned_addr learned my addr 172.21.15.88:0/1512745398 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:35.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.449+0000 7f470affd640 1 -- 172.21.15.88:0/1512745398 >> v1:172.21.15.154:6789/0 conn(0x7f470c15e660 legacy=0x7f470c0cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:35.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.449+0000 7f470affd640 1 -- 172.21.15.88:0/1512745398 >> v1:172.21.15.88:6789/0 conn(0x7f470c0d1650 legacy=0x7f470c0b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:35.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.449+0000 7f470affd640 1 -- 172.21.15.88:0/1512745398 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f470c0d3b50 con 0x7f470c0d3310 2024-09-16T01:34:35.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.449+0000 7f470a7fc640 1 -- 172.21.15.88:0/1512745398 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2822679703 0 0) 0x7f470c15ea40 con 0x7f470c0d1650 2024-09-16T01:34:35.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.449+0000 7f470affd640 1 --2- 172.21.15.88:0/1512745398 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f470c0d3310 0x7f470c0d1110 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f46f4009e50 tx=0x7f46f402a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3cefb51e4726155 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:35.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.450+0000 7f470a7fc640 1 -- 172.21.15.88:0/1512745398 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f4036070 con 0x7f470c0d3310 2024-09-16T01:34:35.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.450+0000 7f470a7fc640 1 -- 172.21.15.88:0/1512745398 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46f4031600 con 0x7f470c0d3310 2024-09-16T01:34:35.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.450+0000 7f470a7fc640 1 -- 172.21.15.88:0/1512745398 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f40319d0 con 0x7f470c0d3310 2024-09-16T01:34:35.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.450+0000 7f4711b8d640 1 -- 172.21.15.88:0/1512745398 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f470c0d3310 msgr2=0x7f470c0d1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:35.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.450+0000 7f4711b8d640 1 --2- 172.21.15.88:0/1512745398 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f470c0d3310 0x7f470c0d1110 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f46f4009e50 tx=0x7f46f402a760 comp rx=0 tx=0).stop 2024-09-16T01:34:35.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.450+0000 7f4711b8d640 1 -- 172.21.15.88:0/1512745398 shutdown_connections 2024-09-16T01:34:35.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.450+0000 7f4711b8d640 1 --2- 172.21.15.88:0/1512745398 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f470c0d3310 0x7f470c0d1110 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:35.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.450+0000 7f4711b8d640 1 -- 172.21.15.88:0/1512745398 >> 172.21.15.88:0/1512745398 conn(0x7f470c0b6410 msgr2=0x7f470c0b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:35.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.450+0000 7f4711b8d640 1 -- 172.21.15.88:0/1512745398 shutdown_connections 2024-09-16T01:34:35.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.450+0000 7f4711b8d640 1 -- 172.21.15.88:0/1512745398 wait complete. 2024-09-16T01:34:35.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.451+0000 7f4711b8d640 1 Processor -- start 2024-09-16T01:34:35.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.452+0000 7f4711b8d640 1 -- start start 2024-09-16T01:34:35.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.452+0000 7f4711b8d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f470c0d1650 0x7f470c158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:35.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.452+0000 7f4711b8d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f470c0d3310 0x7f470c1e8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:35.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.452+0000 7f4711b8d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f470c15e660 0x7f470c160c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:35.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.452+0000 7f4711b8d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f470c0d43f0 con 0x7f470c15e660 2024-09-16T01:34:35.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.452+0000 7f4711b8d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f470c0d4110 con 0x7f470c0d3310 2024-09-16T01:34:35.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.452+0000 7f4711b8d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f470c0b8280 con 0x7f470c0d1650 2024-09-16T01:34:35.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.452+0000 7f470bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f470c15e660 0x7f470c160c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:35.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.452+0000 7f470bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f470c15e660 0x7f470c160c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55308/0 (socket says 172.21.15.88:55308) 2024-09-16T01:34:35.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.452+0000 7f470bfff640 1 -- 172.21.15.88:0/737946537 learned_addr learned my addr 172.21.15.88:0/737946537 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:35.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.453+0000 7f470affd640 1 --2- 172.21.15.88:0/737946537 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f470c0d3310 0x7f470c1e8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:35.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.453+0000 7f470b7fe640 1 --2- 172.21.15.88:0/737946537 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f470c0d1650 0x7f470c158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:35.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.453+0000 7f470bfff640 1 -- 172.21.15.88:0/737946537 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f470c0d1650 msgr2=0x7f470c158770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:35.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.453+0000 7f470bfff640 1 --2- 172.21.15.88:0/737946537 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f470c0d1650 0x7f470c158770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:35.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.453+0000 7f470bfff640 1 -- 172.21.15.88:0/737946537 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f470c0d3310 msgr2=0x7f470c1e8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:35.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.453+0000 7f470bfff640 1 --2- 172.21.15.88:0/737946537 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f470c0d3310 0x7f470c1e8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:35.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.453+0000 7f470bfff640 1 -- 172.21.15.88:0/737946537 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f470c1e9400 con 0x7f470c15e660 2024-09-16T01:34:35.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.453+0000 7f470bfff640 1 --2- 172.21.15.88:0/737946537 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f470c15e660 0x7f470c160c10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f470400c4b0 tx=0x7f470400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:35.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.453+0000 7f4703fff640 1 -- 172.21.15.88:0/737946537 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4704005480 con 0x7f470c15e660 2024-09-16T01:34:35.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.453+0000 7f4703fff640 1 -- 172.21.15.88:0/737946537 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47040035f0 con 0x7f470c15e660 2024-09-16T01:34:35.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.453+0000 7f4703fff640 1 -- 172.21.15.88:0/737946537 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47040039c0 con 0x7f470c15e660 2024-09-16T01:34:35.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.453+0000 7f4711b8d640 1 -- 172.21.15.88:0/737946537 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f470c0cd0c0 con 0x7f470c15e660 2024-09-16T01:34:35.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.453+0000 7f4711b8d640 1 -- 172.21.15.88:0/737946537 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f470c158ee0 con 0x7f470c15e660 2024-09-16T01:34:35.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.454+0000 7f4711b8d640 1 -- 172.21.15.88:0/737946537 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f470c0d1110 con 0x7f470c15e660 2024-09-16T01:34:35.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.454+0000 7f4703fff640 1 -- 172.21.15.88:0/737946537 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4704003b60 con 0x7f470c15e660 2024-09-16T01:34:35.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.455+0000 7f4703fff640 1 --2- 172.21.15.88:0/737946537 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f46ec06c450 0x7f46ec06e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:35.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.455+0000 7f470b7fe640 1 --2- 172.21.15.88:0/737946537 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f46ec06c450 0x7f46ec06e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:35.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.455+0000 7f4703fff640 1 -- 172.21.15.88:0/737946537 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f470408afe0 con 0x7f470c15e660 2024-09-16T01:34:35.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.455+0000 7f470b7fe640 1 --2- 172.21.15.88:0/737946537 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f46ec06c450 0x7f46ec06e910 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f46fc00a730 tx=0x7f46fc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:35.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.459+0000 7f4703fff640 1 -- 172.21.15.88:0/737946537 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f47040597e0 con 0x7f470c15e660 2024-09-16T01:34:35.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.659+0000 7f4711b8d640 1 -- 172.21.15.88:0/737946537 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f470c0b6890 con 0x7f470c15e660 2024-09-16T01:34:35.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.659+0000 7f4703fff640 1 -- 172.21.15.88:0/737946537 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5223 (secure 0 0 0) 0x7f47040590e0 con 0x7f470c15e660 2024-09-16T01:34:35.662 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:35.663 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":6,"btime":"2024-09-16T01:34:24:186280+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/1620784263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1620784263},{"type":"v1","addr":"172.21.15.154:6834","nonce":1620784263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3407296593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":3407296593},{"type":"v1","addr":"172.21.15.154:6835","nonce":3407296593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4293,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/4058543372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4058543372},{"type":"v1","addr":"172.21.15.88:6835","nonce":4058543372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:34:21.508480+0000","modified":"2024-09-16T01:34:23.515443+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6837/3712834803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":3712834803},{"type":"v1","addr":"172.21.15.88:6837","nonce":3712834803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4296,"qdb_cluster":[4296]},"id":1}]} 2024-09-16T01:34:35.663 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 6 2024-09-16T01:34:35.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.661+0000 7f4711b8d640 1 -- 172.21.15.88:0/737946537 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f46ec06c450 msgr2=0x7f46ec06e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:35.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.661+0000 7f4711b8d640 1 --2- 172.21.15.88:0/737946537 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f46ec06c450 0x7f46ec06e910 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f46fc00a730 tx=0x7f46fc008040 comp rx=0 tx=0).stop 2024-09-16T01:34:35.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.661+0000 7f4711b8d640 1 -- 172.21.15.88:0/737946537 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f470c15e660 msgr2=0x7f470c160c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:35.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.661+0000 7f4711b8d640 1 --2- 172.21.15.88:0/737946537 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f470c15e660 0x7f470c160c10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f470400c4b0 tx=0x7f470400c970 comp rx=0 tx=0).stop 2024-09-16T01:34:35.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.661+0000 7f4711b8d640 1 -- 172.21.15.88:0/737946537 shutdown_connections 2024-09-16T01:34:35.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.661+0000 7f4711b8d640 1 --2- 172.21.15.88:0/737946537 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f46ec06c450 0x7f46ec06e910 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:35.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.661+0000 7f4711b8d640 1 --2- 172.21.15.88:0/737946537 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f470c15e660 0x7f470c160c10 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:35.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.661+0000 7f4711b8d640 1 --2- 172.21.15.88:0/737946537 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f470c0d3310 0x7f470c1e8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:35.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.661+0000 7f4711b8d640 1 --2- 172.21.15.88:0/737946537 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f470c0d1650 0x7f470c158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:35.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.661+0000 7f4711b8d640 1 -- 172.21.15.88:0/737946537 >> 172.21.15.88:0/737946537 conn(0x7f470c0b6410 msgr2=0x7f470c15bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:35.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.662+0000 7f4711b8d640 1 -- 172.21.15.88:0/737946537 shutdown_connections 2024-09-16T01:34:35.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:35.662+0000 7f4711b8d640 1 -- 172.21.15.88:0/737946537 wait complete. 2024-09-16T01:34:36.077 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:34:21.508480+0000', 'modified': '2024-09-16T01:34:23.515443+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4296}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4296': {'gid': 4296, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.88:6837/3712834803', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6836', 'nonce': 3712834803}, {'type': 'v1', 'addr': '172.21.15.88:6837', 'nonce': 3712834803}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4296, 'qdb_cluster': [4296]} 2024-09-16T01:34:36.078 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T01:34:36.078 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-16T01:34:36.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.176+0000 7fc9ef0d0640 1 Processor -- start 2024-09-16T01:34:36.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.176+0000 7fc9ef0d0640 1 -- start start 2024-09-16T01:34:36.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.177+0000 7fc9ef0d0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc9e806e5b0 0x7fc9e8073b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.177+0000 7fc9ef0d0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e806eae0 con 0x7fc9e815f7f0 2024-09-16T01:34:36.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.177+0000 7fc9ef0d0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e815ed10 con 0x7fc9e80b6d50 2024-09-16T01:34:36.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.177+0000 7fc9ef0d0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc9e815eff0 con 0x7fc9e806e5b0 2024-09-16T01:34:36.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.177+0000 7fc9ed66d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc9e815f7f0 0x7fc9e806cd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45284/0 (socket says 172.21.15.88:45284) 2024-09-16T01:34:36.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.177+0000 7fc9ed66d640 1 -- 172.21.15.88:0/3612220872 learned_addr learned my addr 172.21.15.88:0/3612220872 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:36.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.177+0000 7fc9dffff640 1 --2- 172.21.15.88:0/3612220872 >> v2:172.21.15.154:3300/0 conn(0x7fc9e806e5b0 0x7fc9e8073b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.177+0000 7fc9df7fe640 1 -- 172.21.15.88:0/3612220872 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3873303629 0 0) 0x7fc9e806eae0 con 0x7fc9e815f7f0 2024-09-16T01:34:36.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.177+0000 7fc9df7fe640 1 -- 172.21.15.88:0/3612220872 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9d0003520 con 0x7fc9e815f7f0 2024-09-16T01:34:36.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.177+0000 7fc9df7fe640 1 -- 172.21.15.88:0/3612220872 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2391062100 0 0) 0x7fc9d0003520 con 0x7fc9e815f7f0 2024-09-16T01:34:36.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.177+0000 7fc9df7fe640 1 -- 172.21.15.88:0/3612220872 >> v2:172.21.15.154:3300/0 conn(0x7fc9e806e5b0 msgr2=0x7fc9e8073b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:36.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.177+0000 7fc9df7fe640 1 --2- 172.21.15.88:0/3612220872 >> v2:172.21.15.154:3300/0 conn(0x7fc9e806e5b0 0x7fc9e8073b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.177+0000 7fc9df7fe640 1 -- 172.21.15.88:0/3612220872 >> v1:172.21.15.154:6789/0 conn(0x7fc9e80b6d50 legacy=0x7fc9e80b7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:36.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.177+0000 7fc9df7fe640 1 -- 172.21.15.88:0/3612220872 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e815f2d0 con 0x7fc9e815f7f0 2024-09-16T01:34:36.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.177+0000 7fc9df7fe640 1 -- 172.21.15.88:0/3612220872 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fc9e00019f0 con 0x7fc9e815f7f0 2024-09-16T01:34:36.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.177+0000 7fc9dffff640 1 --2- 172.21.15.88:0/3612220872 >> v2:172.21.15.154:3300/0 conn(0x7fc9e806e5b0 0x7fc9e8073b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:34:36.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.177+0000 7fc9df7fe640 1 -- 172.21.15.88:0/3612220872 >> v1:172.21.15.88:6789/0 conn(0x7fc9e815f7f0 legacy=0x7fc9e806cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:36.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.177+0000 7fc9df7fe640 1 --2- 172.21.15.88:0/3612220872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9d00039b0 0x7fc9d0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.177+0000 7fc9df7fe640 1 --2- 172.21.15.88:0/3612220872 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9d0004800 0x7fc9d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.178+0000 7fc9df7fe640 1 --2- 172.21.15.88:0/3612220872 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9d00051c0 0x7fc9d000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.178+0000 7fc9df7fe640 1 -- 172.21.15.88:0/3612220872 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e815eff0 con 0x7fc9d00051c0 2024-09-16T01:34:36.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.178+0000 7fc9df7fe640 1 -- 172.21.15.88:0/3612220872 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d000c060 con 0x7fc9d00039b0 2024-09-16T01:34:36.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.178+0000 7fc9df7fe640 1 -- 172.21.15.88:0/3612220872 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc9d000c260 con 0x7fc9d0004800 2024-09-16T01:34:36.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.178+0000 7fc9ed66d640 1 --2- 172.21.15.88:0/3612220872 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9d00051c0 0x7fc9d000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.178+0000 7fc9dffff640 1 --2- 172.21.15.88:0/3612220872 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9d0004800 0x7fc9d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.178+0000 7fc9ece6c640 1 --2- 172.21.15.88:0/3612220872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9d00039b0 0x7fc9d0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.178+0000 7fc9ed66d640 1 -- 172.21.15.88:0/3612220872 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9d0004800 msgr2=0x7fc9d0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:36.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.178+0000 7fc9ed66d640 1 --2- 172.21.15.88:0/3612220872 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9d0004800 0x7fc9d0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.178+0000 7fc9ed66d640 1 -- 172.21.15.88:0/3612220872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9d00039b0 msgr2=0x7fc9d0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:36.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.178+0000 7fc9ed66d640 1 --2- 172.21.15.88:0/3612220872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9d00039b0 0x7fc9d0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.178+0000 7fc9ed66d640 1 -- 172.21.15.88:0/3612220872 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d0003520 con 0x7fc9d00051c0 2024-09-16T01:34:36.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.178+0000 7fc9ed66d640 1 -- 172.21.15.88:0/3612220872 reap_dead start 2024-09-16T01:34:36.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.179+0000 7fc9ed66d640 1 --2- 172.21.15.88:0/3612220872 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9d00051c0 0x7fc9d000b9c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000a850 tx=0x7fc9d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d272c2f38f0dcda server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:36.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.179+0000 7fc9df7fe640 1 -- 172.21.15.88:0/3612220872 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e000a4a0 con 0x7fc9d00051c0 2024-09-16T01:34:36.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.179+0000 7fc9df7fe640 1 -- 172.21.15.88:0/3612220872 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9e002f070 con 0x7fc9d00051c0 2024-09-16T01:34:36.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.179+0000 7fc9df7fe640 1 -- 172.21.15.88:0/3612220872 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e002bce0 con 0x7fc9d00051c0 2024-09-16T01:34:36.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.179+0000 7fc9ef0d0640 1 -- 172.21.15.88:0/3612220872 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9d00051c0 msgr2=0x7fc9d000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:36.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.179+0000 7fc9ef0d0640 1 --2- 172.21.15.88:0/3612220872 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9d00051c0 0x7fc9d000b9c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000a850 tx=0x7fc9d0005830 comp rx=0 tx=0).stop 2024-09-16T01:34:36.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.179+0000 7fc9ef0d0640 1 -- 172.21.15.88:0/3612220872 shutdown_connections 2024-09-16T01:34:36.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.179+0000 7fc9ef0d0640 1 --2- 172.21.15.88:0/3612220872 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9d00051c0 0x7fc9d000b9c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.179+0000 7fc9ef0d0640 1 -- 172.21.15.88:0/3612220872 >> 172.21.15.88:0/3612220872 conn(0x7fc9e80d24d0 msgr2=0x7fc9e80d28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:36.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.179+0000 7fc9ef0d0640 1 -- 172.21.15.88:0/3612220872 shutdown_connections 2024-09-16T01:34:36.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.179+0000 7fc9ef0d0640 1 -- 172.21.15.88:0/3612220872 wait complete. 2024-09-16T01:34:36.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.180+0000 7fc9ef0d0640 1 Processor -- start 2024-09-16T01:34:36.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.180+0000 7fc9ef0d0640 1 -- start start 2024-09-16T01:34:36.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.181+0000 7fc9ef0d0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9d00051c0 0x7fc9e814e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.181+0000 7fc9ef0d0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9e80cc910 0x7fc9e80ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.181+0000 7fc9ef0d0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9e80aa480 0x7fc9e80aa900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.181+0000 7fc9ef0d0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e815e8b0 con 0x7fc9d00051c0 2024-09-16T01:34:36.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.181+0000 7fc9ef0d0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e806edc0 con 0x7fc9e80cc910 2024-09-16T01:34:36.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.181+0000 7fc9ef0d0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc9e815ea30 con 0x7fc9e80aa480 2024-09-16T01:34:36.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.181+0000 7fc9ece6c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9d00051c0 0x7fc9e814e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.181+0000 7fc9ece6c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9d00051c0 0x7fc9e814e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55312/0 (socket says 172.21.15.88:55312) 2024-09-16T01:34:36.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.181+0000 7fc9ece6c640 1 -- 172.21.15.88:0/643633525 learned_addr learned my addr 172.21.15.88:0/643633525 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:36.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.182+0000 7fc9ed66d640 1 --2- 172.21.15.88:0/643633525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9e80aa480 0x7fc9e80aa900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.182+0000 7fc9dffff640 1 --2- 172.21.15.88:0/643633525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9e80cc910 0x7fc9e80ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.182+0000 7fc9ece6c640 1 -- 172.21.15.88:0/643633525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9e80aa480 msgr2=0x7fc9e80aa900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:36.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.182+0000 7fc9ece6c640 1 --2- 172.21.15.88:0/643633525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9e80aa480 0x7fc9e80aa900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.182+0000 7fc9ece6c640 1 -- 172.21.15.88:0/643633525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9e80cc910 msgr2=0x7fc9e80ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:36.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.182+0000 7fc9ece6c640 1 --2- 172.21.15.88:0/643633525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9e80cc910 0x7fc9e80ccd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.182+0000 7fc9ece6c640 1 -- 172.21.15.88:0/643633525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e80cd3d0 con 0x7fc9d00051c0 2024-09-16T01:34:36.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.182+0000 7fc9ece6c640 1 --2- 172.21.15.88:0/643633525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9d00051c0 0x7fc9e814e4e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc9e400f650 tx=0x7fc9e400fb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:36.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.182+0000 7fc9dd7fa640 1 -- 172.21.15.88:0/643633525 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e40050d0 con 0x7fc9d00051c0 2024-09-16T01:34:36.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.182+0000 7fc9dd7fa640 1 -- 172.21.15.88:0/643633525 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9e4018070 con 0x7fc9d00051c0 2024-09-16T01:34:36.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.182+0000 7fc9ef0d0640 1 -- 172.21.15.88:0/643633525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9e80cd6b0 con 0x7fc9d00051c0 2024-09-16T01:34:36.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.182+0000 7fc9dd7fa640 1 -- 172.21.15.88:0/643633525 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e4005490 con 0x7fc9d00051c0 2024-09-16T01:34:36.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.182+0000 7fc9ef0d0640 1 -- 172.21.15.88:0/643633525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9e80cdc70 con 0x7fc9d00051c0 2024-09-16T01:34:36.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.183+0000 7fc9dd7fa640 1 -- 172.21.15.88:0/643633525 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc9e400d6d0 con 0x7fc9d00051c0 2024-09-16T01:34:36.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.184+0000 7fc9dd7fa640 1 --2- 172.21.15.88:0/643633525 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc9d406c2b0 0x7fc9d406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.184+0000 7fc9dd7fa640 1 -- 172.21.15.88:0/643633525 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc9e408b1b0 con 0x7fc9d00051c0 2024-09-16T01:34:36.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.184+0000 7fc9dffff640 1 --2- 172.21.15.88:0/643633525 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc9d406c2b0 0x7fc9d406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.184+0000 7fc9dffff640 1 --2- 172.21.15.88:0/643633525 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc9d406c2b0 0x7fc9d406e770 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8160ea0 tx=0x7fc9d8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:36.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.185+0000 7fc9f1e6b740 1 Processor -- start 2024-09-16T01:34:36.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.185+0000 7fc9f1e6b740 1 -- start start 2024-09-16T01:34:36.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.185+0000 7fc9f1e6b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df022c0 0x56248df47a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.185+0000 7fc9f1e6b740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56248deffa80 0x56248df026a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.185+0000 7fc9f1e6b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56248df03a50 0x56248defc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.185+0000 7fc9f1e6b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56248de64520 con 0x56248df022c0 2024-09-16T01:34:36.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.185+0000 7fc9f1e6b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56248de0c3e0 con 0x56248df03a50 2024-09-16T01:34:36.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.185+0000 7fc9f1e6b740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56248de604e0 con 0x56248deffa80 2024-09-16T01:34:36.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.185+0000 7fc9ece6c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df022c0 0x56248df47a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.185+0000 7fc9ece6c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df022c0 0x56248df47a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55326/0 (socket says 172.21.15.88:55326) 2024-09-16T01:34:36.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.186+0000 7fc9ece6c640 1 -- 172.21.15.88:0/2041355132 learned_addr learned my addr 172.21.15.88:0/2041355132 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:36.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.186+0000 7fc9dffff640 1 --2- 172.21.15.88:0/2041355132 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56248deffa80 0x56248df026a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.186+0000 7fc9ed66d640 1 --2- 172.21.15.88:0/2041355132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56248df03a50 0x56248defc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.186+0000 7fc9ece6c640 1 -- 172.21.15.88:0/2041355132 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56248deffa80 msgr2=0x56248df026a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:36.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.186+0000 7fc9ece6c640 1 --2- 172.21.15.88:0/2041355132 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56248deffa80 0x56248df026a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.186+0000 7fc9ece6c640 1 -- 172.21.15.88:0/2041355132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56248df03a50 msgr2=0x56248defc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:36.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.186+0000 7fc9ece6c640 1 --2- 172.21.15.88:0/2041355132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56248df03a50 0x56248defc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.186+0000 7fc9ece6c640 1 -- 172.21.15.88:0/2041355132 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56248de272f0 con 0x56248df022c0 2024-09-16T01:34:36.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.186+0000 7fc9dffff640 1 --2- 172.21.15.88:0/2041355132 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56248deffa80 0x56248df026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:36.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.186+0000 7fc9ed66d640 1 --2- 172.21.15.88:0/2041355132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56248df03a50 0x56248defc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:36.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.186+0000 7fc9ece6c640 1 --2- 172.21.15.88:0/2041355132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df022c0 0x56248df47a60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4000fb0 tx=0x7fc9e4059920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=971ca80458454559 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:36.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.186+0000 7fc9ce7fc640 1 -- 172.21.15.88:0/2041355132 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e405d5d0 con 0x56248df022c0 2024-09-16T01:34:36.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.186+0000 7fc9ce7fc640 1 -- 172.21.15.88:0/2041355132 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9e4061e00 con 0x56248df022c0 2024-09-16T01:34:36.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.186+0000 7fc9ce7fc640 1 -- 172.21.15.88:0/2041355132 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e4057520 con 0x56248df022c0 2024-09-16T01:34:36.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.187+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/2041355132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df022c0 msgr2=0x56248df47a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:36.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.187+0000 7fc9f1e6b740 1 --2- 172.21.15.88:0/2041355132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df022c0 0x56248df47a60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4000fb0 tx=0x7fc9e4059920 comp rx=0 tx=0).stop 2024-09-16T01:34:36.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.187+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/2041355132 shutdown_connections 2024-09-16T01:34:36.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.187+0000 7fc9f1e6b740 1 --2- 172.21.15.88:0/2041355132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56248df03a50 0x56248defc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.187+0000 7fc9f1e6b740 1 --2- 172.21.15.88:0/2041355132 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56248deffa80 0x56248df026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.187+0000 7fc9f1e6b740 1 --2- 172.21.15.88:0/2041355132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df022c0 0x56248df47a60 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.187+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/2041355132 >> 172.21.15.88:0/2041355132 conn(0x56248defea80 msgr2=0x56248df46f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:36.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.187+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/2041355132 shutdown_connections 2024-09-16T01:34:36.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.187+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/2041355132 wait complete. 2024-09-16T01:34:36.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.187+0000 7fc9f1e6b740 1 -- start start 2024-09-16T01:34:36.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.187+0000 7fc9f1e6b740 1 Processor -- start 2024-09-16T01:34:36.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.188+0000 7fc9f1e6b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56248df022c0 0x56248df210c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.188+0000 7fc9f1e6b740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56248df03a50 0x56248df29ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.188+0000 7fc9f1e6b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df28360 0x56248df287e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.188+0000 7fc9f1e6b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56248defdf20 con 0x56248df28360 2024-09-16T01:34:36.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.188+0000 7fc9f1e6b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56248dded450 con 0x56248df022c0 2024-09-16T01:34:36.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.188+0000 7fc9f1e6b740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56248df48290 con 0x56248df03a50 2024-09-16T01:34:36.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.188+0000 7fc9f1e6b740 10 client.0 _ll_register_callbacks cb 0x56248db76ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:34:36.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.188+0000 7fc9ed66d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df28360 0x56248df287e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.188+0000 7fc9ed66d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df28360 0x56248df287e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55336/0 (socket says 172.21.15.88:55336) 2024-09-16T01:34:36.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.188+0000 7fc9ed66d640 1 -- 172.21.15.88:0/1124043154 learned_addr learned my addr 172.21.15.88:0/1124043154 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:36.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.188+0000 7fc9dffff640 1 --2- 172.21.15.88:0/1124043154 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56248df03a50 0x56248df29ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.188+0000 7fc9ece6c640 1 --2- 172.21.15.88:0/1124043154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56248df022c0 0x56248df210c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.188+0000 7fc9ed66d640 1 -- 172.21.15.88:0/1124043154 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56248df03a50 msgr2=0x56248df29ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:36.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.188+0000 7fc9ed66d640 1 --2- 172.21.15.88:0/1124043154 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56248df03a50 0x56248df29ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.188+0000 7fc9ed66d640 1 -- 172.21.15.88:0/1124043154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56248df022c0 msgr2=0x56248df210c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:36.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.189+0000 7fc9ed66d640 1 --2- 172.21.15.88:0/1124043154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56248df022c0 0x56248df210c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.189+0000 7fc9ed66d640 1 -- 172.21.15.88:0/1124043154 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56248db772d0 con 0x56248df28360 2024-09-16T01:34:36.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.189+0000 7fc9ece6c640 1 --2- 172.21.15.88:0/1124043154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56248df022c0 0x56248df210c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:36.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.189+0000 7fc9ed66d640 1 --2- 172.21.15.88:0/1124043154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df28360 0x56248df287e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000a6b0 tx=0x7fc9e002d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ed06077c99b1503 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:36.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.189+0000 7fc9c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:34:36.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.189+0000 7fc9c77fe640 1 -- 172.21.15.88:0/1124043154 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e002b880 con 0x56248df28360 2024-09-16T01:34:36.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.189+0000 7fc9c77fe640 1 -- 172.21.15.88:0/1124043154 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc9e0041ab0 con 0x56248df28360 2024-09-16T01:34:36.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.189+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/1124043154 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9e4061e00 con 0x56248df28360 2024-09-16T01:34:36.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.189+0000 7fc9c77fe640 1 -- 172.21.15.88:0/1124043154 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) ==== 24+0+0 (secure 0 0 0) 0x7fc9e0041cb0 con 0x56248df28360 2024-09-16T01:34:36.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.189+0000 7fc9f1e6b740 10 client.4656 fetch_fsmap learned FSMap version 6 2024-09-16T01:34:36.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.189+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/1124043154 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) -- 0x56248dc416b0 con 0x56248df28360 2024-09-16T01:34:36.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.189+0000 7fc9c77fe640 1 -- 172.21.15.88:0/1124043154 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9e4061e00 con 0x56248df28360 2024-09-16T01:34:36.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.189+0000 7fc9c77fe640 1 -- 172.21.15.88:0/1124043154 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 6) ==== 3502+0+0 (secure 0 0 0) 0x7fc9e002ba20 con 0x56248df28360 2024-09-16T01:34:36.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.189+0000 7fc9f1e6b740 10 client.4656 fetch_fsmap finished waiting for FSMap version 6 2024-09-16T01:34:36.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.189+0000 7fc9f1e6b740 10 client.4656 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:34:36.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.189+0000 7fc9f1e6b740 20 client.4656 populate_metadata read hostname 'smithi088' 2024-09-16T01:34:36.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.189+0000 7fc9f1e6b740 1 --2- 172.21.15.88:0/1124043154 >> [v2:172.21.15.88:6836/3712834803,v1:172.21.15.88:6837/3712834803] conn(0x56248defb7d0 0x56248defbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.189+0000 7fc9f1e6b740 4 client.4656 mds_command: new command op to 4296 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:34:36.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.189+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/1124043154 --> [v2:172.21.15.88:6836/3712834803,v1:172.21.15.88:6837/3712834803] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56248de71c50 con 0x56248defb7d0 2024-09-16T01:34:36.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.189+0000 7fc9ece6c640 1 --2- 172.21.15.88:0/1124043154 >> [v2:172.21.15.88:6836/3712834803,v1:172.21.15.88:6837/3712834803] conn(0x56248defb7d0 0x56248defbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.190+0000 7fc9c77fe640 10 client.4656 ms_handle_connect on v2:172.21.15.88:6836/3712834803 2024-09-16T01:34:36.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.190+0000 7fc9ece6c640 1 --2- 172.21.15.88:0/1124043154 >> [v2:172.21.15.88:6836/3712834803,v1:172.21.15.88:6837/3712834803] conn(0x56248defb7d0 0x56248defbbb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=281623d46abcc558 server_cookie=a2b90caabafb9ce3 in_seq=0 out_seq=0 2024-09-16T01:34:36.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.190+0000 7fc9c77fe640 1 -- 172.21.15.88:0/1124043154 <== mds.0 v2:172.21.15.88:6836/3712834803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56248de71c50 con 0x56248defb7d0 2024-09-16T01:34:36.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.190+0000 7fc9c77fe640 10 client.4656 handle_command_reply: tid=0 2024-09-16T01:34:36.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.190+0000 7fc9f1e6b740 1 client.4656 shutdown 2024-09-16T01:34:36.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.191+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/1124043154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df28360 msgr2=0x56248df287e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:36.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.191+0000 7fc9f1e6b740 1 --2- 172.21.15.88:0/1124043154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df28360 0x56248df287e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000a6b0 tx=0x7fc9e002d070 comp rx=0 tx=0).stop 2024-09-16T01:34:36.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.191+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/1124043154 shutdown_connections 2024-09-16T01:34:36.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.191+0000 7fc9f1e6b740 1 --2- 172.21.15.88:0/1124043154 >> [v2:172.21.15.88:6836/3712834803,v1:172.21.15.88:6837/3712834803] conn(0x56248defb7d0 0x56248defbbb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.191+0000 7fc9f1e6b740 1 --2- 172.21.15.88:0/1124043154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df28360 0x56248df287e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.191+0000 7fc9f1e6b740 1 --2- 172.21.15.88:0/1124043154 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56248df03a50 0x56248df29ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.191+0000 7fc9f1e6b740 1 --2- 172.21.15.88:0/1124043154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56248df022c0 0x56248df210c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.191+0000 7fc9c77fe640 0 client.4656 ms_handle_reset on v2:172.21.15.88:6836/3712834803 2024-09-16T01:34:36.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.191+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/1124043154 >> 172.21.15.88:0/1124043154 conn(0x56248deffa80 msgr2=0x56248df45f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:36.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.191+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/1124043154 shutdown_connections 2024-09-16T01:34:36.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.191+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/1124043154 wait complete. 2024-09-16T01:34:36.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.191+0000 7fc9f1e6b740 20 client.4656 trim_cache size 0 max 16384 2024-09-16T01:34:36.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.208+0000 7fc9f1e6b740 1 Processor -- start 2024-09-16T01:34:36.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.208+0000 7fc9f1e6b740 1 -- start start 2024-09-16T01:34:36.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.209+0000 7fc9f1e6b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df022c0 0x56248df231f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.209+0000 7fc9f1e6b740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56248df03a50 0x56248df440a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.209+0000 7fc9f1e6b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56248df28360 0x56248df445e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.209+0000 7fc9f1e6b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e8200360 con 0x56248df022c0 2024-09-16T01:34:36.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.209+0000 7fc9f1e6b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e80d1ba0 con 0x56248df28360 2024-09-16T01:34:36.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.209+0000 7fc9f1e6b740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc9e814c6e0 con 0x56248df03a50 2024-09-16T01:34:36.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.209+0000 7fc9ece6c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df022c0 0x56248df231f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.209+0000 7fc9ece6c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df022c0 0x56248df231f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55338/0 (socket says 172.21.15.88:55338) 2024-09-16T01:34:36.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.209+0000 7fc9ece6c640 1 -- 172.21.15.88:0/923203576 learned_addr learned my addr 172.21.15.88:0/923203576 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:36.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.209+0000 7fc9ed66d640 1 --2- 172.21.15.88:0/923203576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56248df28360 0x56248df445e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.209+0000 7fc9dffff640 1 --2- 172.21.15.88:0/923203576 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56248df03a50 0x56248df440a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.210+0000 7fc9ece6c640 1 -- 172.21.15.88:0/923203576 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56248df03a50 msgr2=0x56248df440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:36.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.210+0000 7fc9ece6c640 1 --2- 172.21.15.88:0/923203576 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56248df03a50 0x56248df440a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.210+0000 7fc9ece6c640 1 -- 172.21.15.88:0/923203576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56248df28360 msgr2=0x56248df445e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:36.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.210+0000 7fc9ece6c640 1 --2- 172.21.15.88:0/923203576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56248df28360 0x56248df445e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.210+0000 7fc9ece6c640 1 -- 172.21.15.88:0/923203576 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56248df4b370 con 0x56248df022c0 2024-09-16T01:34:36.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.210+0000 7fc9ece6c640 1 --2- 172.21.15.88:0/923203576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df022c0 0x56248df231f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4059160 tx=0x7fc9e408b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6597cb2d8adab41b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:36.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.210+0000 7fc9cdffb640 1 -- 172.21.15.88:0/923203576 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e4061e00 con 0x56248df022c0 2024-09-16T01:34:36.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.210+0000 7fc9cdffb640 1 -- 172.21.15.88:0/923203576 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9e4012e30 con 0x56248df022c0 2024-09-16T01:34:36.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.210+0000 7fc9cdffb640 1 -- 172.21.15.88:0/923203576 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e40892a0 con 0x56248df022c0 2024-09-16T01:34:36.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.210+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/923203576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df022c0 msgr2=0x56248df231f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:36.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.210+0000 7fc9f1e6b740 1 --2- 172.21.15.88:0/923203576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df022c0 0x56248df231f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4059160 tx=0x7fc9e408b000 comp rx=0 tx=0).stop 2024-09-16T01:34:36.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.210+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/923203576 shutdown_connections 2024-09-16T01:34:36.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.210+0000 7fc9f1e6b740 1 --2- 172.21.15.88:0/923203576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56248df28360 0x56248df445e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.210+0000 7fc9f1e6b740 1 --2- 172.21.15.88:0/923203576 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56248df03a50 0x56248df440a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.210+0000 7fc9f1e6b740 1 --2- 172.21.15.88:0/923203576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df022c0 0x56248df231f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.210+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/923203576 >> 172.21.15.88:0/923203576 conn(0x56248deffa80 msgr2=0x56248defd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:36.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.210+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/923203576 shutdown_connections 2024-09-16T01:34:36.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.210+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/923203576 wait complete. 2024-09-16T01:34:36.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.211+0000 7fc9f1e6b740 1 -- start start 2024-09-16T01:34:36.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.211+0000 7fc9f1e6b740 1 Processor -- start 2024-09-16T01:34:36.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.212+0000 7fc9f1e6b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56248df022c0 0x56248dfbf560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.212+0000 7fc9f1e6b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df03a50 0x56248df5c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.212+0000 7fc9f1e6b740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56248df28360 0x56248df5cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.212+0000 7fc9f1e6b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56248df216a0 con 0x56248df03a50 2024-09-16T01:34:36.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.212+0000 7fc9f1e6b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56248df57f10 con 0x56248df022c0 2024-09-16T01:34:36.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.212+0000 7fc9f1e6b740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56248df2a1e0 con 0x56248df28360 2024-09-16T01:34:36.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.212+0000 7fc9f1e6b740 10 client.0 _ll_register_callbacks cb 0x7fc9d0004700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:34:36.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.212+0000 7fc9dffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df03a50 0x56248df5c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.212+0000 7fc9dffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df03a50 0x56248df5c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55352/0 (socket says 172.21.15.88:55352) 2024-09-16T01:34:36.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.212+0000 7fc9dffff640 1 -- 172.21.15.88:0/4260341709 learned_addr learned my addr 172.21.15.88:0/4260341709 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:36.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.212+0000 7fc9ece6c640 1 --2- 172.21.15.88:0/4260341709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56248df022c0 0x56248dfbf560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.212+0000 7fc9ed66d640 1 --2- 172.21.15.88:0/4260341709 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56248df28360 0x56248df5cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.212+0000 7fc9dffff640 1 -- 172.21.15.88:0/4260341709 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56248df28360 msgr2=0x56248df5cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:36.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.212+0000 7fc9dffff640 1 --2- 172.21.15.88:0/4260341709 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56248df28360 0x56248df5cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.212+0000 7fc9dffff640 1 -- 172.21.15.88:0/4260341709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56248df022c0 msgr2=0x56248dfbf560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:36.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.212+0000 7fc9dffff640 1 --2- 172.21.15.88:0/4260341709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56248df022c0 0x56248dfbf560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.212+0000 7fc9dffff640 1 -- 172.21.15.88:0/4260341709 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56248dfc6670 con 0x56248df03a50 2024-09-16T01:34:36.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.212+0000 7fc9ece6c640 1 --2- 172.21.15.88:0/4260341709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56248df022c0 0x56248dfbf560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:36.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.212+0000 7fc9dffff640 1 --2- 172.21.15.88:0/4260341709 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df03a50 0x56248df5c5f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc9d80249b0 tx=0x7fc9d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f265b69a1df12d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:36.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.212+0000 7fc9c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:34:36.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.212+0000 7fc9c77fe640 1 -- 172.21.15.88:0/4260341709 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d8016d40 con 0x56248df03a50 2024-09-16T01:34:36.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.213+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/4260341709 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9e4012e30 con 0x56248df03a50 2024-09-16T01:34:36.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.213+0000 7fc9c77fe640 1 -- 172.21.15.88:0/4260341709 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc9d802eaf0 con 0x56248df03a50 2024-09-16T01:34:36.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.213+0000 7fc9c77fe640 1 -- 172.21.15.88:0/4260341709 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) ==== 24+0+0 (secure 0 0 0) 0x7fc9d802ecf0 con 0x56248df03a50 2024-09-16T01:34:36.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.213+0000 7fc9f1e6b740 10 client.4662 fetch_fsmap learned FSMap version 6 2024-09-16T01:34:36.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.213+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/4260341709 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) -- 0x56248dfc7bf0 con 0x56248df03a50 2024-09-16T01:34:36.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.213+0000 7fc9c77fe640 1 -- 172.21.15.88:0/4260341709 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9e4012e30 con 0x56248df03a50 2024-09-16T01:34:36.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.213+0000 7fc9c77fe640 1 -- 172.21.15.88:0/4260341709 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 6) ==== 3502+0+0 (secure 0 0 0) 0x7fc9d8035050 con 0x56248df03a50 2024-09-16T01:34:36.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.213+0000 7fc9f1e6b740 10 client.4662 fetch_fsmap finished waiting for FSMap version 6 2024-09-16T01:34:36.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.213+0000 7fc9f1e6b740 10 client.4662 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:34:36.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.213+0000 7fc9f1e6b740 20 client.4662 populate_metadata read hostname 'smithi088' 2024-09-16T01:34:36.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.213+0000 7fc9f1e6b740 1 --2- 172.21.15.88:0/4260341709 >> [v2:172.21.15.88:6836/3712834803,v1:172.21.15.88:6837/3712834803] conn(0x56248dfc7670 0x56248df45f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.213+0000 7fc9f1e6b740 4 client.4662 mds_command: new command op to 4296 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:34:36.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.213+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/4260341709 --> [v2:172.21.15.88:6836/3712834803,v1:172.21.15.88:6837/3712834803] -- command(tid 0: {"prefix": "status"}) -- 0x56248de6bfa0 con 0x56248dfc7670 2024-09-16T01:34:36.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.213+0000 7fc9ece6c640 1 --2- 172.21.15.88:0/4260341709 >> [v2:172.21.15.88:6836/3712834803,v1:172.21.15.88:6837/3712834803] conn(0x56248dfc7670 0x56248df45f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.213+0000 7fc9c77fe640 10 client.4662 ms_handle_connect on v2:172.21.15.88:6836/3712834803 2024-09-16T01:34:36.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.213+0000 7fc9ece6c640 1 --2- 172.21.15.88:0/4260341709 >> [v2:172.21.15.88:6836/3712834803,v1:172.21.15.88:6837/3712834803] conn(0x56248dfc7670 0x56248df45f20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6431c4266dfaf67 server_cookie=c82130f4d5a8a1f2 in_seq=0 out_seq=0 2024-09-16T01:34:36.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.213+0000 7fc9c77fe640 1 -- 172.21.15.88:0/4260341709 <== mds.0 v2:172.21.15.88:6836/3712834803 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x56248de6bfa0 con 0x56248dfc7670 2024-09-16T01:34:36.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.213+0000 7fc9c77fe640 10 client.4662 handle_command_reply: tid=0 2024-09-16T01:34:36.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.214+0000 7fc9f1e6b740 1 client.4662 shutdown 2024-09-16T01:34:36.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.214+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/4260341709 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df03a50 msgr2=0x56248df5c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:36.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.214+0000 7fc9f1e6b740 1 --2- 172.21.15.88:0/4260341709 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df03a50 0x56248df5c5f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc9d80249b0 tx=0x7fc9d8004730 comp rx=0 tx=0).stop 2024-09-16T01:34:36.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.214+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/4260341709 shutdown_connections 2024-09-16T01:34:36.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.214+0000 7fc9f1e6b740 1 --2- 172.21.15.88:0/4260341709 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56248df28360 0x56248df5cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.214+0000 7fc9f1e6b740 1 --2- 172.21.15.88:0/4260341709 >> [v2:172.21.15.88:6836/3712834803,v1:172.21.15.88:6837/3712834803] conn(0x56248dfc7670 0x56248df45f20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.214+0000 7fc9f1e6b740 1 --2- 172.21.15.88:0/4260341709 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56248df03a50 0x56248df5c5f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.214+0000 7fc9f1e6b740 1 --2- 172.21.15.88:0/4260341709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56248df022c0 0x56248dfbf560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.214+0000 7fc9c77fe640 0 client.4662 ms_handle_reset on v2:172.21.15.88:6836/3712834803 2024-09-16T01:34:36.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.214+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/4260341709 >> 172.21.15.88:0/4260341709 conn(0x56248deffa80 msgr2=0x56248df20850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:36.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.214+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/4260341709 shutdown_connections 2024-09-16T01:34:36.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.214+0000 7fc9f1e6b740 1 -- 172.21.15.88:0/4260341709 wait complete. 2024-09-16T01:34:36.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.214+0000 7fc9f1e6b740 20 client.4662 trim_cache size 0 max 16384 2024-09-16T01:34:36.223 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:34:36.223 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "5ace2c91-46f9-482b-8a89-731fbdc171bf", 2024-09-16T01:34:36.223 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2024-09-16T01:34:36.223 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 4296, 2024-09-16T01:34:36.223 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2024-09-16T01:34:36.224 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2024-09-16T01:34:36.224 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2024-09-16T01:34:36.224 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 14.042511047, 2024-09-16T01:34:36.224 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 6, 2024-09-16T01:34:36.224 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 21, 2024-09-16T01:34:36.224 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 21, 2024-09-16T01:34:36.224 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 20.072306786999999 2024-09-16T01:34:36.224 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:34:36.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.214+0000 7fc9ef0d0640 1 -- 172.21.15.88:0/643633525 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc9d406c2b0 msgr2=0x7fc9d406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:36.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.214+0000 7fc9ef0d0640 1 --2- 172.21.15.88:0/643633525 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc9d406c2b0 0x7fc9d406e770 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8160ea0 tx=0x7fc9d8008040 comp rx=0 tx=0).stop 2024-09-16T01:34:36.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.214+0000 7fc9ef0d0640 1 -- 172.21.15.88:0/643633525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9d00051c0 msgr2=0x7fc9e814e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:36.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.214+0000 7fc9ef0d0640 1 --2- 172.21.15.88:0/643633525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9d00051c0 0x7fc9e814e4e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc9e400f650 tx=0x7fc9e400fb10 comp rx=0 tx=0).stop 2024-09-16T01:34:36.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.214+0000 7fc9ef0d0640 1 -- 172.21.15.88:0/643633525 shutdown_connections 2024-09-16T01:34:36.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.214+0000 7fc9ef0d0640 1 --2- 172.21.15.88:0/643633525 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc9d406c2b0 0x7fc9d406e770 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.214+0000 7fc9ef0d0640 1 --2- 172.21.15.88:0/643633525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9e80aa480 0x7fc9e80aa900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.214+0000 7fc9ef0d0640 1 --2- 172.21.15.88:0/643633525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9e80cc910 0x7fc9e80ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.214+0000 7fc9ef0d0640 1 --2- 172.21.15.88:0/643633525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9d00051c0 0x7fc9e814e4e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.214+0000 7fc9ef0d0640 1 -- 172.21.15.88:0/643633525 >> 172.21.15.88:0/643633525 conn(0x7fc9e80d24d0 msgr2=0x7fc9e815ad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:36.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.214+0000 7fc9ef0d0640 1 -- 172.21.15.88:0/643633525 shutdown_connections 2024-09-16T01:34:36.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.215+0000 7fc9ef0d0640 1 -- 172.21.15.88:0/643633525 wait complete. 2024-09-16T01:34:36.227 INFO:teuthology.run_tasks:Running task sequential... 2024-09-16T01:34:36.237 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-09-16T01:34:36.237 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-16T01:34:36.237 INFO:teuthology.task.print:Enabling mgr modules 2024-09-16T01:34:36.237 INFO:teuthology.run_tasks:Running task kclient... 2024-09-16T01:34:36.272 INFO:tasks.kclient:Mounting kernel clients... 2024-09-16T01:34:36.272 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-09-16T01:34:36.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:36.273 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:34:36.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:36.289 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link delete ceph-brx 2024-09-16T01:34:36.355 INFO:teuthology.orchestra.run.smithi088.stderr:Cannot find device "ceph-brx" 2024-09-16T01:34:36.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:34:36.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:36.357 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:34:36.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:36.412 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link delete ceph-brx 2024-09-16T01:34:36.478 INFO:teuthology.orchestra.run.smithi088.stderr:Cannot find device "ceph-brx" 2024-09-16T01:34:36.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:34:36.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:36.480 DEBUG:teuthology.orchestra.run.smithi154:> ip netns list 2024-09-16T01:34:36.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:36.499 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip link delete ceph-brx 2024-09-16T01:34:36.572 INFO:teuthology.orchestra.run.smithi154.stderr:Cannot find device "ceph-brx" 2024-09-16T01:34:36.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:34:36.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:36.574 DEBUG:teuthology.orchestra.run.smithi154:> ip netns list 2024-09-16T01:34:36.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:36.632 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip link delete ceph-brx 2024-09-16T01:34:36.700 INFO:teuthology.orchestra.run.smithi154.stderr:Cannot find device "ceph-brx" 2024-09-16T01:34:36.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:34:36.702 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-09-16T01:34:36.702 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-16T01:34:36.702 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T01:34:36.703 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:34:36.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.799+0000 7fde9ef91640 1 Processor -- start 2024-09-16T01:34:36.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.799+0000 7fde9ef91640 1 -- start start 2024-09-16T01:34:36.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.799+0000 7fde9ef91640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde9815e660 0x7fde980cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.799+0000 7fde9ef91640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fde980d3310 0x7fde980d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.799+0000 7fde9ef91640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fde980d1650 0x7fde9806bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.799+0000 7fde9ef91640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fde9815ea40 con 0x7fde980d3310 2024-09-16T01:34:36.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.799+0000 7fde9ef91640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fde980d3870 con 0x7fde980d1650 2024-09-16T01:34:36.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.799+0000 7fde9ef91640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fde980d3b50 con 0x7fde9815e660 2024-09-16T01:34:36.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.800+0000 7fde97fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fde980d3310 0x7fde980d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.800+0000 7fde97fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fde980d3310 0x7fde980d1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55366/0 (socket says 172.21.15.88:55366) 2024-09-16T01:34:36.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.800+0000 7fde97fff640 1 -- 172.21.15.88:0/2597825267 learned_addr learned my addr 172.21.15.88:0/2597825267 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:36.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.800+0000 7fde9cd2d640 1 --2- 172.21.15.88:0/2597825267 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde9815e660 0x7fde980cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.800+0000 7fde9d52e640 1 --2- 172.21.15.88:0/2597825267 >> v2:172.21.15.154:3300/0 conn(0x7fde980d1650 0x7fde9806bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.800+0000 7fde97fff640 1 -- 172.21.15.88:0/2597825267 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde9815e660 msgr2=0x7fde980cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:36.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.800+0000 7fde97fff640 1 --2- 172.21.15.88:0/2597825267 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde9815e660 0x7fde980cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.800+0000 7fde97fff640 1 -- 172.21.15.88:0/2597825267 >> v2:172.21.15.154:3300/0 conn(0x7fde980d1650 msgr2=0x7fde9806bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:36.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.800+0000 7fde97fff640 1 --2- 172.21.15.88:0/2597825267 >> v2:172.21.15.154:3300/0 conn(0x7fde980d1650 0x7fde9806bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.800+0000 7fde97fff640 1 -- 172.21.15.88:0/2597825267 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde980d3e30 con 0x7fde980d3310 2024-09-16T01:34:36.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.800+0000 7fde97fff640 1 --2- 172.21.15.88:0/2597825267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde980d3310 0x7fde980d1110 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fde980cb760 tx=0x7fde8402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a407e68d7560aff server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:36.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.801+0000 7fde977fe640 1 -- 172.21.15.88:0/2597825267 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84037070 con 0x7fde980d3310 2024-09-16T01:34:36.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.801+0000 7fde977fe640 1 -- 172.21.15.88:0/2597825267 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde84032440 con 0x7fde980d3310 2024-09-16T01:34:36.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.801+0000 7fde977fe640 1 -- 172.21.15.88:0/2597825267 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84032810 con 0x7fde980d3310 2024-09-16T01:34:36.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.801+0000 7fde9ef91640 1 -- 172.21.15.88:0/2597825267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde980d3310 msgr2=0x7fde980d1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:36.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.801+0000 7fde9ef91640 1 --2- 172.21.15.88:0/2597825267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde980d3310 0x7fde980d1110 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fde980cb760 tx=0x7fde8402a1e0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.801+0000 7fde9ef91640 1 -- 172.21.15.88:0/2597825267 shutdown_connections 2024-09-16T01:34:36.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.801+0000 7fde9ef91640 1 --2- 172.21.15.88:0/2597825267 >> v2:172.21.15.154:3300/0 conn(0x7fde980d1650 0x7fde9806bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.801+0000 7fde9ef91640 1 --2- 172.21.15.88:0/2597825267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde980d3310 0x7fde980d1110 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.801+0000 7fde9ef91640 1 --2- 172.21.15.88:0/2597825267 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde9815e660 0x7fde980cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.801+0000 7fde9ef91640 1 -- 172.21.15.88:0/2597825267 >> 172.21.15.88:0/2597825267 conn(0x7fde980b6410 msgr2=0x7fde980b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:36.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.801+0000 7fde9ef91640 1 -- 172.21.15.88:0/2597825267 shutdown_connections 2024-09-16T01:34:36.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.801+0000 7fde9ef91640 1 -- 172.21.15.88:0/2597825267 wait complete. 2024-09-16T01:34:36.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.802+0000 7fde9ef91640 1 Processor -- start 2024-09-16T01:34:36.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.802+0000 7fde9ef91640 1 -- start start 2024-09-16T01:34:36.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.803+0000 7fde9ef91640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde980d1650 0x7fde981e7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.803+0000 7fde9ef91640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde980d3310 0x7fde98146ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.803+0000 7fde9ef91640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde9815e660 0x7fde981473e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.803+0000 7fde9ef91640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fde980cfc70 con 0x7fde980d3310 2024-09-16T01:34:36.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.803+0000 7fde9ef91640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fde980d43f0 con 0x7fde980d1650 2024-09-16T01:34:36.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.803+0000 7fde9ef91640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fde9806c510 con 0x7fde9815e660 2024-09-16T01:34:36.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.803+0000 7fde97fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde980d3310 0x7fde98146ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.803+0000 7fde97fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde980d3310 0x7fde98146ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55370/0 (socket says 172.21.15.88:55370) 2024-09-16T01:34:36.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.803+0000 7fde97fff640 1 -- 172.21.15.88:0/1216065611 learned_addr learned my addr 172.21.15.88:0/1216065611 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:36.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.803+0000 7fde9cd2d640 1 --2- 172.21.15.88:0/1216065611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde980d1650 0x7fde981e7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.803+0000 7fde9d52e640 1 --2- 172.21.15.88:0/1216065611 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde9815e660 0x7fde981473e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.804+0000 7fde97fff640 1 -- 172.21.15.88:0/1216065611 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde9815e660 msgr2=0x7fde981473e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:36.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.804+0000 7fde97fff640 1 --2- 172.21.15.88:0/1216065611 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde9815e660 0x7fde981473e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.804+0000 7fde97fff640 1 -- 172.21.15.88:0/1216065611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde980d1650 msgr2=0x7fde981e7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:36.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.804+0000 7fde97fff640 1 --2- 172.21.15.88:0/1216065611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde980d1650 0x7fde981e7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:36.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.804+0000 7fde97fff640 1 -- 172.21.15.88:0/1216065611 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde98145e00 con 0x7fde980d3310 2024-09-16T01:34:36.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.804+0000 7fde97fff640 1 --2- 172.21.15.88:0/1216065611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde980d3310 0x7fde98146ea0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fde8402a220 tx=0x7fde840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:36.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.804+0000 7fde957fa640 1 -- 172.21.15.88:0/1216065611 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84041020 con 0x7fde980d3310 2024-09-16T01:34:36.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.804+0000 7fde957fa640 1 -- 172.21.15.88:0/1216065611 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde840306c0 con 0x7fde980d3310 2024-09-16T01:34:36.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.804+0000 7fde9ef91640 1 -- 172.21.15.88:0/1216065611 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde981460e0 con 0x7fde980d3310 2024-09-16T01:34:36.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.804+0000 7fde957fa640 1 -- 172.21.15.88:0/1216065611 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84037040 con 0x7fde980d3310 2024-09-16T01:34:36.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.804+0000 7fde9ef91640 1 -- 172.21.15.88:0/1216065611 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde981466a0 con 0x7fde980d3310 2024-09-16T01:34:36.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.805+0000 7fde9ef91640 1 -- 172.21.15.88:0/1216065611 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde9815dec0 con 0x7fde980d3310 2024-09-16T01:34:36.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.805+0000 7fde957fa640 1 -- 172.21.15.88:0/1216065611 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fde84044020 con 0x7fde980d3310 2024-09-16T01:34:36.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.806+0000 7fde957fa640 1 --2- 172.21.15.88:0/1216065611 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fde7c06c2e0 0x7fde7c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:36.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.806+0000 7fde957fa640 1 -- 172.21.15.88:0/1216065611 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fde840abb40 con 0x7fde980d3310 2024-09-16T01:34:36.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.806+0000 7fde9cd2d640 1 --2- 172.21.15.88:0/1216065611 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fde7c06c2e0 0x7fde7c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:36.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.806+0000 7fde9cd2d640 1 --2- 172.21.15.88:0/1216065611 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fde7c06c2e0 0x7fde7c06e7a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fde8c0079a0 tx=0x7fde8c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:36.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:36.810+0000 7fde957fa640 1 -- 172.21.15.88:0/1216065611 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fde9815dec0 con 0x7fde980d3310 2024-09-16T01:34:37.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:37.008+0000 7fde9ef91640 1 -- 172.21.15.88:0/1216065611 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fde980d1110 con 0x7fde980d3310 2024-09-16T01:34:37.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:37.008+0000 7fde957fa640 1 -- 172.21.15.88:0/1216065611 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) ==== 53+0+73 (secure 0 0 0) 0x7fde8404b030 con 0x7fde980d3310 2024-09-16T01:34:37.011 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:34:37.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:37.010+0000 7fde9ef91640 1 -- 172.21.15.88:0/1216065611 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fde7c06c2e0 msgr2=0x7fde7c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:37.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:37.010+0000 7fde9ef91640 1 --2- 172.21.15.88:0/1216065611 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fde7c06c2e0 0x7fde7c06e7a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fde8c0079a0 tx=0x7fde8c008040 comp rx=0 tx=0).stop 2024-09-16T01:34:37.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:37.010+0000 7fde9ef91640 1 -- 172.21.15.88:0/1216065611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde980d3310 msgr2=0x7fde98146ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:37.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:37.010+0000 7fde9ef91640 1 --2- 172.21.15.88:0/1216065611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde980d3310 0x7fde98146ea0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fde8402a220 tx=0x7fde840046c0 comp rx=0 tx=0).stop 2024-09-16T01:34:37.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:37.010+0000 7fde9ef91640 1 -- 172.21.15.88:0/1216065611 shutdown_connections 2024-09-16T01:34:37.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:37.010+0000 7fde9ef91640 1 --2- 172.21.15.88:0/1216065611 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fde7c06c2e0 0x7fde7c06e7a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:37.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:37.010+0000 7fde9ef91640 1 --2- 172.21.15.88:0/1216065611 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde9815e660 0x7fde981473e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:37.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:37.010+0000 7fde9ef91640 1 --2- 172.21.15.88:0/1216065611 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde980d3310 0x7fde98146ea0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:37.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:37.010+0000 7fde9ef91640 1 --2- 172.21.15.88:0/1216065611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde980d1650 0x7fde981e7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:37.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:37.010+0000 7fde9ef91640 1 -- 172.21.15.88:0/1216065611 >> 172.21.15.88:0/1216065611 conn(0x7fde980b6410 msgr2=0x7fde9815bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:37.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:37.010+0000 7fde9ef91640 1 -- 172.21.15.88:0/1216065611 shutdown_connections 2024-09-16T01:34:37.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:37.010+0000 7fde9ef91640 1 -- 172.21.15.88:0/1216065611 wait complete. 2024-09-16T01:34:37.024 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T01:34:37.024 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:34:37.024 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2024-09-16T01:34:37.024 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T01:34:37.024 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T01:34:37.024 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T01:34:37.024 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T01:34:37.024 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:34:37.024 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:34:37.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:37.025 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2024-09-16T01:34:37.040 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:34:37.040 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:34:37.041 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:34:37.041 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:37.041 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-09-16T01:34:37.041 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:37.041 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:34:37.041 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:34:37.041 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2024-09-16T01:34:37.041 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:34:37.041 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:34:37.041 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2024-09-16T01:34:37.041 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:34:37.041 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2024-09-16T01:34:37.042 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:34:37.042 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2024-09-16T01:34:37.042 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:34:37.042 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42747sec preferred_lft 42747sec 2024-09-16T01:34:37.042 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link noprefixroute 2024-09-16T01:34:37.042 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:37.042 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-16T01:34:37.043 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:34:37.043 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:34:37.043 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link add name ceph-brx type bridge 2024-09-16T01:34:37.043 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip addr flush dev ceph-brx 2024-09-16T01:34:37.043 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set ceph-brx up 2024-09-16T01:34:37.043 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-16T01:34:37.043 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:34:37.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:34:37.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:37.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:37.258 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:34:37.284 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-09-16T01:34:37.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:37.286 DEBUG:teuthology.orchestra.run.smithi088:> route 2024-09-16T01:34:37.370 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2024-09-16T01:34:37.370 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:34:37.370 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:34:37.371 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:34:37.371 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:34:37.371 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:34:37.371 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:34:37.371 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:34:37.371 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:34:37.371 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:34:37.371 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:34:37.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:34:37.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:37.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:37.612 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:34:37.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:37.628 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list-id 2024-09-16T01:34:37.685 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:34:37.685 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:34:37.685 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:34:37.685 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-16T01:34:37.685 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:34:37.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:34:37.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:37.804 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-16T01:34:37.804 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:34:37.804 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:34:37.804 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-16T01:34:37.804 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-16T01:34:37.804 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-16T01:34:37.804 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-16T01:34:37.804 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-16T01:34:37.804 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:34:37.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:34:37.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:37.929 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:34:37.929 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:34:37.929 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2024-09-16T01:34:37.929 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set dev brx.0 master ceph-brx 2024-09-16T01:34:37.929 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:34:37.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:34:38.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:38.006 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:34:38.007 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T01:34:38.022 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:34:38.022 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:34:38.022 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:34:38.077 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-16T01:34:38.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:38.077 DEBUG:teuthology.orchestra.run.smithi088:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:34:38.167 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:34:38.167 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:34:38.167 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:34:38.289 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:34:38.289 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:34:38.289 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:34:38.289 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:34:38.289 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:34:38.299 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:34:38.299 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:34:38.299 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:34:38.299 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:34:38.299 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:34:38.299 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790" 2024-09-16T01:34:38.300 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2024-09-16T01:34:38.300 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5ace2c91-46f9-482b-8a89-731fbdc171bf" will pass to kernel 2024-09-16T01:34:38.300 INFO:teuthology.orchestra.run.smithi088.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-16T01:34:38.300 INFO:teuthology.orchestra.run.smithi088.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:34:38.300 INFO:teuthology.orchestra.run.smithi088.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:34:38.300 INFO:teuthology.orchestra.run.smithi088.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:34:38.301 INFO:teuthology.orchestra.run.smithi088.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:34:38.302 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:34:38.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:38.302 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:34:38.327 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:34:38.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:34:38.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:38.430 INFO:teuthology.orchestra.run.smithi088.stdout:client4457 2024-09-16T01:34:38.431 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:34:38.431 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:34:38.446 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:34:38.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:34:38.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:38.549 INFO:teuthology.orchestra.run.smithi088.stdout:client4457 2024-09-16T01:34:38.549 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:34:38.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:34:38.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:38.609 INFO:teuthology.orchestra.run.smithi088.stdout:client4457 2024-09-16T01:34:38.609 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/5ace2c91-46f9-482b-8a89-731fbdc171bf.client4457/status') 2024-09-16T01:34:38.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:34:38.670 INFO:teuthology.orchestra.run.smithi088.stdout:instance: client.4457 (3)172.21.15.88:0/4019802755 2024-09-16T01:34:38.670 INFO:teuthology.orchestra.run.smithi088.stdout:blocklisted: false 2024-09-16T01:34:38.670 INFO:teuthology.orchestra.run.smithi088.stderr:0+1 records in 2024-09-16T01:34:38.670 INFO:teuthology.orchestra.run.smithi088.stderr:0+1 records out 2024-09-16T01:34:38.670 INFO:teuthology.orchestra.run.smithi088.stderr:70 bytes copied, 2.2167e-05 s, 3.2 MB/s 2024-09-16T01:34:38.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:38.677 INFO:tasks.kclient:client.2 config is {'syntax': 'v1'} 2024-09-16T01:34:38.677 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-16T01:34:38.677 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-09-16T01:34:38.677 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:34:38.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.774+0000 7f71ec680640 1 Processor -- start 2024-09-16T01:34:38.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.774+0000 7f71ec680640 1 -- start start 2024-09-16T01:34:38.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.775+0000 7f71ec680640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e4155800 0x7f71e40b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:38.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.775+0000 7f71ec680640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f71e415c330 0x7f71e40b4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:38.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.775+0000 7f71ec680640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71e415c770 con 0x7f71e40cf570 2024-09-16T01:34:38.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.775+0000 7f71ec680640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f71e40b4e60 con 0x7f71e415c330 2024-09-16T01:34:38.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.775+0000 7f71ec680640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f71e40b50a0 con 0x7f71e4155800 2024-09-16T01:34:38.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.775+0000 7f71e9c1b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f71e415c330 0x7f71e40b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:38.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.775+0000 7f71eac1d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f71e40cf570 0x7f71e415c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:45300/0 (socket says 172.21.15.88:45300) 2024-09-16T01:34:38.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.775+0000 7f71eac1d640 1 -- 172.21.15.88:0/2091414219 learned_addr learned my addr 172.21.15.88:0/2091414219 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:38.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.776+0000 7f71e9c1b640 1 -- 172.21.15.88:0/2091414219 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e4155800 msgr2=0x7f71e40b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:38.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.776+0000 7f71ea41c640 1 --2- 172.21.15.88:0/2091414219 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e4155800 0x7f71e40b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:38.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.776+0000 7f71e9c1b640 1 --2- 172.21.15.88:0/2091414219 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e4155800 0x7f71e40b6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:38.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.776+0000 7f71e9c1b640 1 -- 172.21.15.88:0/2091414219 >> v1:172.21.15.88:6789/0 conn(0x7f71e40cf570 legacy=0x7f71e415c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:38.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.776+0000 7f71e941a640 1 -- 172.21.15.88:0/2091414219 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3907705988 0 0) 0x7f71dc001110 con 0x7f71e40cf570 2024-09-16T01:34:38.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.776+0000 7f71e9c1b640 1 -- 172.21.15.88:0/2091414219 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71e40b5380 con 0x7f71e415c330 2024-09-16T01:34:38.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.776+0000 7f71e9c1b640 1 --2- 172.21.15.88:0/2091414219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e415c330 0x7f71e40b4920 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f71d80097a0 tx=0x7f71d802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b999975f3f018d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:38.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.776+0000 7f71e941a640 1 -- 172.21.15.88:0/2091414219 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71d8037070 con 0x7f71e415c330 2024-09-16T01:34:38.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.776+0000 7f71e941a640 1 -- 172.21.15.88:0/2091414219 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71d8032440 con 0x7f71e415c330 2024-09-16T01:34:38.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.776+0000 7f71e941a640 1 -- 172.21.15.88:0/2091414219 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71d8032810 con 0x7f71e415c330 2024-09-16T01:34:38.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.777+0000 7f71ec680640 1 -- 172.21.15.88:0/2091414219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e415c330 msgr2=0x7f71e40b4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:38.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.777+0000 7f71ec680640 1 --2- 172.21.15.88:0/2091414219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e415c330 0x7f71e40b4920 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f71d80097a0 tx=0x7f71d802a180 comp rx=0 tx=0).stop 2024-09-16T01:34:38.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.777+0000 7f71ec680640 1 -- 172.21.15.88:0/2091414219 shutdown_connections 2024-09-16T01:34:38.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.777+0000 7f71ec680640 1 --2- 172.21.15.88:0/2091414219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e415c330 0x7f71e40b4920 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:38.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.777+0000 7f71ec680640 1 --2- 172.21.15.88:0/2091414219 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e4155800 0x7f71e40b6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:38.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.777+0000 7f71ec680640 1 -- 172.21.15.88:0/2091414219 >> 172.21.15.88:0/2091414219 conn(0x7f71e406bc10 msgr2=0x7f71e406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:38.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.777+0000 7f71ec680640 1 -- 172.21.15.88:0/2091414219 shutdown_connections 2024-09-16T01:34:38.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.777+0000 7f71ec680640 1 -- 172.21.15.88:0/2091414219 wait complete. 2024-09-16T01:34:38.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.778+0000 7f71ec680640 1 Processor -- start 2024-09-16T01:34:38.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.778+0000 7f71ec680640 1 -- start start 2024-09-16T01:34:38.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.779+0000 7f71ec680640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e40cf570 0x7f71e41e4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:38.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.779+0000 7f71ec680640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71e4155800 0x7f71e414f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:38.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.779+0000 7f71ec680640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e415c330 0x7f71e41e66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:38.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.779+0000 7f71ec680640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f71e40b5c20 con 0x7f71e415c330 2024-09-16T01:34:38.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.779+0000 7f71ec680640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f71e40b5940 con 0x7f71e4155800 2024-09-16T01:34:38.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.779+0000 7f71ec680640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f71e4155be0 con 0x7f71e40cf570 2024-09-16T01:34:38.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.779+0000 7f71eac1d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e415c330 0x7f71e41e66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:38.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.779+0000 7f71eac1d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e415c330 0x7f71e41e66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55378/0 (socket says 172.21.15.88:55378) 2024-09-16T01:34:38.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.779+0000 7f71eac1d640 1 -- 172.21.15.88:0/3224514406 learned_addr learned my addr 172.21.15.88:0/3224514406 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:38.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.779+0000 7f71e9c1b640 1 --2- 172.21.15.88:0/3224514406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71e4155800 0x7f71e414f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:38.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.779+0000 7f71ea41c640 1 --2- 172.21.15.88:0/3224514406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e40cf570 0x7f71e41e4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:38.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.779+0000 7f71eac1d640 1 -- 172.21.15.88:0/3224514406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e40cf570 msgr2=0x7f71e41e4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:38.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.779+0000 7f71eac1d640 1 --2- 172.21.15.88:0/3224514406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e40cf570 0x7f71e41e4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:38.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.779+0000 7f71eac1d640 1 -- 172.21.15.88:0/3224514406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71e4155800 msgr2=0x7f71e414f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:38.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.779+0000 7f71eac1d640 1 --2- 172.21.15.88:0/3224514406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71e4155800 0x7f71e414f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:38.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.779+0000 7f71eac1d640 1 -- 172.21.15.88:0/3224514406 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71e41e5bf0 con 0x7f71e415c330 2024-09-16T01:34:38.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.779+0000 7f71ea41c640 1 --2- 172.21.15.88:0/3224514406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e40cf570 0x7f71e41e4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:38.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.780+0000 7f71eac1d640 1 --2- 172.21.15.88:0/3224514406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e415c330 0x7f71e41e66f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f71dc00c410 tx=0x7f71dc001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:38.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.780+0000 7f71e2ffd640 1 -- 172.21.15.88:0/3224514406 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71dc0074e0 con 0x7f71e415c330 2024-09-16T01:34:38.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.780+0000 7f71e2ffd640 1 -- 172.21.15.88:0/3224514406 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71dc0076a0 con 0x7f71e415c330 2024-09-16T01:34:38.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.780+0000 7f71e2ffd640 1 -- 172.21.15.88:0/3224514406 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71dc004a20 con 0x7f71e415c330 2024-09-16T01:34:38.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.780+0000 7f71ec680640 1 -- 172.21.15.88:0/3224514406 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71e41e5350 con 0x7f71e415c330 2024-09-16T01:34:38.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.780+0000 7f71ec680640 1 -- 172.21.15.88:0/3224514406 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71e41e5910 con 0x7f71e415c330 2024-09-16T01:34:38.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.781+0000 7f71ec680640 1 -- 172.21.15.88:0/3224514406 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71e40d0450 con 0x7f71e415c330 2024-09-16T01:34:38.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.781+0000 7f71e2ffd640 1 -- 172.21.15.88:0/3224514406 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f71dc011070 con 0x7f71e415c330 2024-09-16T01:34:38.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.782+0000 7f71e2ffd640 1 --2- 172.21.15.88:0/3224514406 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f71d006c2e0 0x7f71d006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:38.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.782+0000 7f71e2ffd640 1 -- 172.21.15.88:0/3224514406 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f71dc08b050 con 0x7f71e415c330 2024-09-16T01:34:38.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.782+0000 7f71ea41c640 1 --2- 172.21.15.88:0/3224514406 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f71d006c2e0 0x7f71d006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:38.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.782+0000 7f71ea41c640 1 --2- 172.21.15.88:0/3224514406 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f71d006c2e0 0x7f71d006e7a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f71d4007f50 tx=0x7f71d40074a0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:38.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.786+0000 7f71e2ffd640 1 -- 172.21.15.88:0/3224514406 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f71e40d0450 con 0x7f71e415c330 2024-09-16T01:34:38.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.987+0000 7f71ec680640 1 -- 172.21.15.88:0/3224514406 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f71e40b6140 con 0x7f71e415c330 2024-09-16T01:34:38.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.987+0000 7f71e2ffd640 1 -- 172.21.15.88:0/3224514406 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) ==== 53+0+73 (secure 0 0 0) 0x7f71e415c770 con 0x7f71e415c330 2024-09-16T01:34:38.990 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:34:38.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.989+0000 7f71ec680640 1 -- 172.21.15.88:0/3224514406 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f71d006c2e0 msgr2=0x7f71d006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:38.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.989+0000 7f71ec680640 1 --2- 172.21.15.88:0/3224514406 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f71d006c2e0 0x7f71d006e7a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f71d4007f50 tx=0x7f71d40074a0 comp rx=0 tx=0).stop 2024-09-16T01:34:38.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.989+0000 7f71ec680640 1 -- 172.21.15.88:0/3224514406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e415c330 msgr2=0x7f71e41e66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:38.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.989+0000 7f71ec680640 1 --2- 172.21.15.88:0/3224514406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e415c330 0x7f71e41e66f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f71dc00c410 tx=0x7f71dc001e70 comp rx=0 tx=0).stop 2024-09-16T01:34:38.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.989+0000 7f71ec680640 1 -- 172.21.15.88:0/3224514406 shutdown_connections 2024-09-16T01:34:38.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.989+0000 7f71ec680640 1 --2- 172.21.15.88:0/3224514406 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f71d006c2e0 0x7f71d006e7a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:38.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.989+0000 7f71ec680640 1 --2- 172.21.15.88:0/3224514406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e415c330 0x7f71e41e66f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:38.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.989+0000 7f71ec680640 1 --2- 172.21.15.88:0/3224514406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71e4155800 0x7f71e414f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:38.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.989+0000 7f71ec680640 1 --2- 172.21.15.88:0/3224514406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e40cf570 0x7f71e41e4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:38.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.989+0000 7f71ec680640 1 -- 172.21.15.88:0/3224514406 >> 172.21.15.88:0/3224514406 conn(0x7f71e406bc10 msgr2=0x7f71e4071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:38.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.989+0000 7f71ec680640 1 -- 172.21.15.88:0/3224514406 shutdown_connections 2024-09-16T01:34:38.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:38.989+0000 7f71ec680640 1 -- 172.21.15.88:0/3224514406 wait complete. 2024-09-16T01:34:39.002 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T01:34:39.003 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:34:39.003 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2024-09-16T01:34:39.003 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-09-16T01:34:39.003 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-09-16T01:34:39.003 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T01:34:39.003 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T01:34:39.003 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:34:39.003 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-09-16T01:34:39.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:39.003 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2024-09-16T01:34:39.019 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:34:39.019 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:34:39.019 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:34:39.020 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:39.020 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-09-16T01:34:39.020 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:39.020 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:34:39.020 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:34:39.020 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2024-09-16T01:34:39.020 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:34:39.020 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:34:39.020 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2024-09-16T01:34:39.020 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:34:39.020 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2024-09-16T01:34:39.020 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:34:39.021 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2024-09-16T01:34:39.021 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:34:39.021 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42745sec preferred_lft 42745sec 2024-09-16T01:34:39.021 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link noprefixroute 2024-09-16T01:34:39.021 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:39.021 INFO:teuthology.orchestra.run.smithi088.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:34:39.021 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 12:6f:29:be:95:48 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:34:39.021 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T01:34:39.021 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:39.021 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::e486:6eff:fe81:a1c/64 scope link 2024-09-16T01:34:39.021 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:39.022 INFO:teuthology.orchestra.run.smithi088.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:34:39.022 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 12:6f:29:be:95:48 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:34:39.022 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::106f:29ff:febe:9548/64 scope link tentative 2024-09-16T01:34:39.022 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:39.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:39.022 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:34:39.084 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-09-16T01:34:39.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:39.085 DEBUG:teuthology.orchestra.run.smithi088:> route 2024-09-16T01:34:39.141 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2024-09-16T01:34:39.141 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:34:39.141 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:34:39.141 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:34:39.141 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:34:39.142 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:34:39.142 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:34:39.142 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:34:39.142 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:34:39.142 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:34:39.142 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:34:39.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:34:39.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:39.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:39.267 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:34:39.282 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T01:34:39.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:39.283 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list-id 2024-09-16T01:34:39.338 INFO:teuthology.orchestra.run.smithi088.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-09-16T01:34:39.339 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:34:39.339 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:34:39.339 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T01:34:39.339 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-09-16T01:34:39.339 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:34:39.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:34:39.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:39.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:39.455 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-09-16T01:34:39.479 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:34:39.480 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:34:39.480 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:34:39.480 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:39.480 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-09-16T01:34:39.480 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:39.480 INFO:teuthology.orchestra.run.smithi088.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:34:39.480 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 7e:5b:46:94:a3:81 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-16T01:34:39.480 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-16T01:34:39.480 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:39.480 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::7c5b:46ff:fe94:a381/64 scope link 2024-09-16T01:34:39.480 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:39.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:39.482 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-09-16T01:34:39.546 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:34:39.546 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:34:39.547 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:34:39.547 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:39.547 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-09-16T01:34:39.547 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:39.547 INFO:teuthology.orchestra.run.smithi088.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:34:39.547 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 7e:5b:46:94:a3:81 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-16T01:34:39.547 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-16T01:34:39.547 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:39.547 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::7c5b:46ff:fe94:a381/64 scope link 2024-09-16T01:34:39.547 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:39.548 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-09-16T01:34:39.549 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:34:39.549 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:34:39.549 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-09-16T01:34:39.549 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-09-16T01:34:39.549 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-09-16T01:34:39.549 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-09-16T01:34:39.549 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-09-16T01:34:39.549 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:34:39.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:34:39.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:39.700 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:34:39.700 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:34:39.700 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.1 up 2024-09-16T01:34:39.700 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set dev brx.1 master ceph-brx 2024-09-16T01:34:39.700 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:34:39.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:34:39.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:39.776 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:34:39.776 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-09-16T01:34:39.791 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-09-16T01:34:39.791 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:34:39.791 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-09-16T01:34:39.846 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-16T01:34:39.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:39.846 DEBUG:teuthology.orchestra.run.smithi088:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-09-16T01:34:39.929 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-09-16T01:34:39.929 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-09-16T01:34:39.929 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:34:39.938 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-09-16T01:34:39.938 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-09-16T01:34:39.938 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:34:39.938 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:34:39.939 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:34:39.944 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-09-16T01:34:39.944 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-09-16T01:34:39.944 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:34:39.944 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:34:39.944 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:34:39.944 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790" 2024-09-16T01:34:39.945 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2024-09-16T01:34:39.945 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs,key=2,fsid=5ace2c91-46f9-482b-8a89-731fbdc171bf" will pass to kernel 2024-09-16T01:34:39.945 INFO:teuthology.orchestra.run.smithi088.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-09-16T01:34:39.945 INFO:teuthology.orchestra.run.smithi088.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:34:39.945 INFO:teuthology.orchestra.run.smithi088.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:34:39.945 INFO:teuthology.orchestra.run.smithi088.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:34:39.945 INFO:teuthology.orchestra.run.smithi088.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:34:39.948 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:34:39.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:39.949 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-09-16T01:34:40.014 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:34:40.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:34:40.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:40.110 INFO:teuthology.orchestra.run.smithi088.stdout:client4522 2024-09-16T01:34:40.110 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:34:40.110 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:34:40.126 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:34:40.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:34:40.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:40.221 INFO:teuthology.orchestra.run.smithi088.stdout:client4522 2024-09-16T01:34:40.222 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:34:40.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:34:40.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:40.277 INFO:teuthology.orchestra.run.smithi088.stdout:client4522 2024-09-16T01:34:40.277 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/5ace2c91-46f9-482b-8a89-731fbdc171bf.client4522/status') 2024-09-16T01:34:40.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:34:40.334 INFO:teuthology.orchestra.run.smithi088.stdout:instance: client.4522 (3)172.21.15.88:0/2984444657 2024-09-16T01:34:40.335 INFO:teuthology.orchestra.run.smithi088.stdout:blocklisted: false 2024-09-16T01:34:40.335 INFO:teuthology.orchestra.run.smithi088.stderr:0+1 records in 2024-09-16T01:34:40.335 INFO:teuthology.orchestra.run.smithi088.stderr:0+1 records out 2024-09-16T01:34:40.335 INFO:teuthology.orchestra.run.smithi088.stderr:70 bytes copied, 4.0597e-05 s, 1.7 MB/s 2024-09-16T01:34:40.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:40.342 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-09-16T01:34:40.342 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-16T01:34:40.342 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-16T01:34:40.342 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:34:40.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.438+0000 7f30216e4640 1 Processor -- start 2024-09-16T01:34:40.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.438+0000 7f30216e4640 1 -- start start 2024-09-16T01:34:40.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.439+0000 7f30216e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f301c0d3310 0x7f301c0d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:40.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.439+0000 7f30216e4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f301c15ea40 con 0x7f301c0d1650 2024-09-16T01:34:40.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.439+0000 7f30216e4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f301c0d3870 con 0x7f301c15e660 2024-09-16T01:34:40.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.439+0000 7f30216e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f301c0d1ac0 con 0x7f301c0d3310 2024-09-16T01:34:40.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.439+0000 7f301b7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f301c0d1650 0x7f301c0b7b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54028/0 (socket says 172.21.15.88:54028) 2024-09-16T01:34:40.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.439+0000 7f301b7fe640 1 -- 172.21.15.88:0/384445189 learned_addr learned my addr 172.21.15.88:0/384445189 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:40.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.439+0000 7f3019ffb640 1 -- 172.21.15.88:0/384445189 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1025920249 0 0) 0x7f301c15ea40 con 0x7f301c0d1650 2024-09-16T01:34:40.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.439+0000 7f301a7fc640 1 --2- 172.21.15.88:0/384445189 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f301c0d3310 0x7f301c0d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:40.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.439+0000 7f3019ffb640 1 -- 172.21.15.88:0/384445189 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3004003520 con 0x7f301c0d1650 2024-09-16T01:34:40.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.440+0000 7f3019ffb640 1 -- 172.21.15.88:0/384445189 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4192196602 0 0) 0x7f3004003520 con 0x7f301c0d1650 2024-09-16T01:34:40.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.440+0000 7f3019ffb640 1 -- 172.21.15.88:0/384445189 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f301c0d3310 msgr2=0x7f301c0d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:40.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.440+0000 7f3019ffb640 1 --2- 172.21.15.88:0/384445189 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f301c0d3310 0x7f301c0d1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:40.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.440+0000 7f3019ffb640 1 -- 172.21.15.88:0/384445189 >> v1:172.21.15.154:6789/0 conn(0x7f301c15e660 legacy=0x7f301c0cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:40.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.440+0000 7f3019ffb640 1 -- 172.21.15.88:0/384445189 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f301c0d3b50 con 0x7f301c0d1650 2024-09-16T01:34:40.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.440+0000 7f3019ffb640 1 -- 172.21.15.88:0/384445189 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f30140019f0 con 0x7f301c0d1650 2024-09-16T01:34:40.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.440+0000 7f3019ffb640 1 -- 172.21.15.88:0/384445189 >> v1:172.21.15.88:6789/0 conn(0x7f301c0d1650 legacy=0x7f301c0b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:40.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.440+0000 7f3019ffb640 1 --2- 172.21.15.88:0/384445189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30040039e0 0x7f3004003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:40.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.440+0000 7f3019ffb640 1 --2- 172.21.15.88:0/384445189 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f301c0d3310 0x7f3004004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:40.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.440+0000 7f3019ffb640 1 --2- 172.21.15.88:0/384445189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3004004d40 0x7f30040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:40.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.440+0000 7f3019ffb640 1 -- 172.21.15.88:0/384445189 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f301c0d1ac0 con 0x7f3004004d40 2024-09-16T01:34:40.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.440+0000 7f3019ffb640 1 -- 172.21.15.88:0/384445189 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f300400b9f0 con 0x7f30040039e0 2024-09-16T01:34:40.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.440+0000 7f3019ffb640 1 -- 172.21.15.88:0/384445189 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f300400bbf0 con 0x7f301c0d3310 2024-09-16T01:34:40.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.440+0000 7f301b7fe640 1 --2- 172.21.15.88:0/384445189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3004004d40 0x7f30040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:40.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.440+0000 7f301affd640 1 --2- 172.21.15.88:0/384445189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30040039e0 0x7f3004003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:40.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.440+0000 7f301a7fc640 1 --2- 172.21.15.88:0/384445189 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f301c0d3310 0x7f3004004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:40.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.440+0000 7f301b7fe640 1 -- 172.21.15.88:0/384445189 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f301c0d3310 msgr2=0x7f3004004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:40.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.440+0000 7f301b7fe640 1 --2- 172.21.15.88:0/384445189 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f301c0d3310 0x7f3004004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:40.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.440+0000 7f301b7fe640 1 -- 172.21.15.88:0/384445189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30040039e0 msgr2=0x7f3004003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:40.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.440+0000 7f301b7fe640 1 --2- 172.21.15.88:0/384445189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30040039e0 0x7f3004003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:40.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.440+0000 7f301b7fe640 1 -- 172.21.15.88:0/384445189 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3004003520 con 0x7f3004004d40 2024-09-16T01:34:40.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.440+0000 7f301a7fc640 1 --2- 172.21.15.88:0/384445189 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f301c0d3310 0x7f3004004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:40.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.441+0000 7f301b7fe640 1 --2- 172.21.15.88:0/384445189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3004004d40 0x7f30040051c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f301400a860 tx=0x7f3004005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63ee1867b954c70 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:40.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.441+0000 7f3019ffb640 1 -- 172.21.15.88:0/384445189 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f301400a480 con 0x7f3004004d40 2024-09-16T01:34:40.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.441+0000 7f3019ffb640 1 -- 172.21.15.88:0/384445189 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f301402f070 con 0x7f3004004d40 2024-09-16T01:34:40.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.441+0000 7f3019ffb640 1 -- 172.21.15.88:0/384445189 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f301402bcb0 con 0x7f3004004d40 2024-09-16T01:34:40.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.442+0000 7f30216e4640 1 -- 172.21.15.88:0/384445189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3004004d40 msgr2=0x7f30040051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:40.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.442+0000 7f30216e4640 1 --2- 172.21.15.88:0/384445189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3004004d40 0x7f30040051c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f301400a860 tx=0x7f3004005830 comp rx=0 tx=0).stop 2024-09-16T01:34:40.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.442+0000 7f301b7fe640 1 -- 172.21.15.88:0/384445189 reap_dead start 2024-09-16T01:34:40.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.442+0000 7f30216e4640 1 -- 172.21.15.88:0/384445189 shutdown_connections 2024-09-16T01:34:40.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.442+0000 7f30216e4640 1 -- 172.21.15.88:0/384445189 >> 172.21.15.88:0/384445189 conn(0x7f301c0b6410 msgr2=0x7f301c0b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:40.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.442+0000 7f30216e4640 1 -- 172.21.15.88:0/384445189 shutdown_connections 2024-09-16T01:34:40.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.442+0000 7f30216e4640 1 -- 172.21.15.88:0/384445189 wait complete. 2024-09-16T01:34:40.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.443+0000 7f30216e4640 1 Processor -- start 2024-09-16T01:34:40.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.443+0000 7f30216e4640 1 -- start start 2024-09-16T01:34:40.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.444+0000 7f30216e4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3004004d40 0x7f301c1f3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:40.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.444+0000 7f30216e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f301c155cc0 0x7f301c149640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:40.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.444+0000 7f30216e4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f301c15a560 0x7f301c15a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:40.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.444+0000 7f30216e4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f301c0d43f0 con 0x7f301c15a560 2024-09-16T01:34:40.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.444+0000 7f30216e4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f301c0d4110 con 0x7f3004004d40 2024-09-16T01:34:40.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.444+0000 7f30216e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f301c0d1110 con 0x7f301c155cc0 2024-09-16T01:34:40.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.444+0000 7f301b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f301c15a560 0x7f301c15a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:40.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.444+0000 7f301b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f301c15a560 0x7f301c15a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54760/0 (socket says 172.21.15.88:54760) 2024-09-16T01:34:40.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.444+0000 7f301b7fe640 1 -- 172.21.15.88:0/1763487681 learned_addr learned my addr 172.21.15.88:0/1763487681 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:40.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.444+0000 7f301affd640 1 --2- 172.21.15.88:0/1763487681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3004004d40 0x7f301c1f3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:40.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.444+0000 7f301a7fc640 1 --2- 172.21.15.88:0/1763487681 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f301c155cc0 0x7f301c149640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:40.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.444+0000 7f301b7fe640 1 -- 172.21.15.88:0/1763487681 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f301c155cc0 msgr2=0x7f301c149640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:40.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.444+0000 7f301b7fe640 1 --2- 172.21.15.88:0/1763487681 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f301c155cc0 0x7f301c149640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:40.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.444+0000 7f301b7fe640 1 -- 172.21.15.88:0/1763487681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3004004d40 msgr2=0x7f301c1f3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:40.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.444+0000 7f301b7fe640 1 --2- 172.21.15.88:0/1763487681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3004004d40 0x7f301c1f3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:40.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.444+0000 7f301b7fe640 1 -- 172.21.15.88:0/1763487681 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f301c1f3f70 con 0x7f301c15a560 2024-09-16T01:34:40.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.444+0000 7f301affd640 1 --2- 172.21.15.88:0/1763487681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3004004d40 0x7f301c1f3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:40.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.444+0000 7f301b7fe640 1 --2- 172.21.15.88:0/1763487681 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f301c15a560 0x7f301c15a9e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f301400a980 tx=0x7f301c0d3a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:40.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.444+0000 7f30037fe640 1 -- 172.21.15.88:0/1763487681 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f301400a480 con 0x7f301c15a560 2024-09-16T01:34:40.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.444+0000 7f30037fe640 1 -- 172.21.15.88:0/1763487681 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3014044070 con 0x7f301c15a560 2024-09-16T01:34:40.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.444+0000 7f30216e4640 1 -- 172.21.15.88:0/1763487681 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f301c1613e0 con 0x7f301c15a560 2024-09-16T01:34:40.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.444+0000 7f30037fe640 1 -- 172.21.15.88:0/1763487681 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3014001fd0 con 0x7f301c15a560 2024-09-16T01:34:40.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.444+0000 7f30216e4640 1 -- 172.21.15.88:0/1763487681 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f301c1619a0 con 0x7f301c15a560 2024-09-16T01:34:40.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.445+0000 7f30216e4640 1 -- 172.21.15.88:0/1763487681 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f301c0cf730 con 0x7f301c15a560 2024-09-16T01:34:40.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.446+0000 7f30037fe640 1 -- 172.21.15.88:0/1763487681 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3014034670 con 0x7f301c15a560 2024-09-16T01:34:40.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.446+0000 7f30037fe640 1 --2- 172.21.15.88:0/1763487681 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f300806c2b0 0x7f300806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:40.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.446+0000 7f30037fe640 1 -- 172.21.15.88:0/1763487681 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f30140abad0 con 0x7f301c15a560 2024-09-16T01:34:40.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.446+0000 7f301affd640 1 --2- 172.21.15.88:0/1763487681 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f300806c2b0 0x7f300806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:40.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.447+0000 7f301affd640 1 --2- 172.21.15.88:0/1763487681 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f300806c2b0 0x7f300806e770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f300c003700 tx=0x7f300c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:40.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.450+0000 7f30037fe640 1 -- 172.21.15.88:0/1763487681 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f301407a1b0 con 0x7f301c15a560 2024-09-16T01:34:40.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.654+0000 7f30216e4640 1 -- 172.21.15.88:0/1763487681 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f301c0cf730 con 0x7f301c15a560 2024-09-16T01:34:40.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.654+0000 7f30037fe640 1 -- 172.21.15.88:0/1763487681 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) ==== 53+0+73 (secure 0 0 0) 0x7f3014079ab0 con 0x7f301c15a560 2024-09-16T01:34:40.656 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:34:40.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.656+0000 7f30216e4640 1 -- 172.21.15.88:0/1763487681 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f300806c2b0 msgr2=0x7f300806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:40.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.656+0000 7f30216e4640 1 --2- 172.21.15.88:0/1763487681 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f300806c2b0 0x7f300806e770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f300c003700 tx=0x7f300c009040 comp rx=0 tx=0).stop 2024-09-16T01:34:40.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.656+0000 7f30216e4640 1 -- 172.21.15.88:0/1763487681 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f301c15a560 msgr2=0x7f301c15a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:40.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.656+0000 7f30216e4640 1 --2- 172.21.15.88:0/1763487681 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f301c15a560 0x7f301c15a9e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f301400a980 tx=0x7f301c0d3a40 comp rx=0 tx=0).stop 2024-09-16T01:34:40.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.656+0000 7f30216e4640 1 -- 172.21.15.88:0/1763487681 shutdown_connections 2024-09-16T01:34:40.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.656+0000 7f30216e4640 1 --2- 172.21.15.88:0/1763487681 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f300806c2b0 0x7f300806e770 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:40.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.656+0000 7f30216e4640 1 --2- 172.21.15.88:0/1763487681 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f301c15a560 0x7f301c15a9e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:40.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.656+0000 7f30216e4640 1 --2- 172.21.15.88:0/1763487681 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f301c155cc0 0x7f301c149640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:40.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.656+0000 7f30216e4640 1 --2- 172.21.15.88:0/1763487681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3004004d40 0x7f301c1f3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:40.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.656+0000 7f30216e4640 1 -- 172.21.15.88:0/1763487681 >> 172.21.15.88:0/1763487681 conn(0x7f301c0b6410 msgr2=0x7f301c06bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:40.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.656+0000 7f30216e4640 1 -- 172.21.15.88:0/1763487681 shutdown_connections 2024-09-16T01:34:40.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:40.656+0000 7f30216e4640 1 -- 172.21.15.88:0/1763487681 wait complete. 2024-09-16T01:34:40.669 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T01:34:40.669 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:34:40.669 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi154.front.sepia.ceph.com 2024-09-16T01:34:40.669 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-16T01:34:40.669 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-16T01:34:40.669 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T01:34:40.669 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T01:34:40.669 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:34:40.669 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-16T01:34:40.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:40.670 DEBUG:teuthology.orchestra.run.smithi154:> ip addr 2024-09-16T01:34:40.689 INFO:teuthology.orchestra.run.smithi154.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:34:40.689 INFO:teuthology.orchestra.run.smithi154.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:34:40.689 INFO:teuthology.orchestra.run.smithi154.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:34:40.689 INFO:teuthology.orchestra.run.smithi154.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:40.689 INFO:teuthology.orchestra.run.smithi154.stdout: inet6 ::1/128 scope host 2024-09-16T01:34:40.690 INFO:teuthology.orchestra.run.smithi154.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:40.690 INFO:teuthology.orchestra.run.smithi154.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:34:40.690 INFO:teuthology.orchestra.run.smithi154.stdout: link/ether 0c:c4:7a:d9:1a:54 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:34:40.690 INFO:teuthology.orchestra.run.smithi154.stdout: altname enp7s0f0 2024-09-16T01:34:40.690 INFO:teuthology.orchestra.run.smithi154.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:34:40.690 INFO:teuthology.orchestra.run.smithi154.stdout: link/ether 0c:c4:7a:d9:1a:55 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:34:40.690 INFO:teuthology.orchestra.run.smithi154.stdout: altname enp7s0f1 2024-09-16T01:34:40.690 INFO:teuthology.orchestra.run.smithi154.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:34:40.690 INFO:teuthology.orchestra.run.smithi154.stdout: link/ether 0c:c4:7a:88:77:60 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:34:40.690 INFO:teuthology.orchestra.run.smithi154.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:34:40.690 INFO:teuthology.orchestra.run.smithi154.stdout: link/ether 0c:c4:7a:88:77:61 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:34:40.690 INFO:teuthology.orchestra.run.smithi154.stdout: inet 172.21.15.154/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:34:40.691 INFO:teuthology.orchestra.run.smithi154.stdout: valid_lft 42746sec preferred_lft 42746sec 2024-09-16T01:34:40.691 INFO:teuthology.orchestra.run.smithi154.stdout: inet6 fe80::ec4:7aff:fe88:7761/64 scope link noprefixroute 2024-09-16T01:34:40.691 INFO:teuthology.orchestra.run.smithi154.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:40.691 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-16T01:34:40.691 DEBUG:teuthology.orchestra.run.smithi154:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:34:40.691 DEBUG:teuthology.orchestra.run.smithi154:> set -e 2024-09-16T01:34:40.691 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip link add name ceph-brx type bridge 2024-09-16T01:34:40.691 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip addr flush dev ceph-brx 2024-09-16T01:34:40.691 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip link set ceph-brx up 2024-09-16T01:34:40.691 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-16T01:34:40.692 DEBUG:teuthology.orchestra.run.smithi154:> ') 2024-09-16T01:34:40.785 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:34:40.948 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:40.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:40.955 DEBUG:teuthology.orchestra.run.smithi154:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:34:40.981 INFO:teuthology.orchestra.run.smithi154.stdout:1 2024-09-16T01:34:40.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:40.983 DEBUG:teuthology.orchestra.run.smithi154:> route 2024-09-16T01:34:41.044 INFO:teuthology.orchestra.run.smithi154.stdout:Kernel IP routing table 2024-09-16T01:34:41.044 INFO:teuthology.orchestra.run.smithi154.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:34:41.044 INFO:teuthology.orchestra.run.smithi154.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:34:41.044 INFO:teuthology.orchestra.run.smithi154.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:34:41.044 INFO:teuthology.orchestra.run.smithi154.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:34:41.045 DEBUG:teuthology.orchestra.run.smithi154:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:34:41.045 DEBUG:teuthology.orchestra.run.smithi154:> set -e 2024-09-16T01:34:41.045 DEBUG:teuthology.orchestra.run.smithi154:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:34:41.045 DEBUG:teuthology.orchestra.run.smithi154:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:34:41.045 DEBUG:teuthology.orchestra.run.smithi154:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:34:41.045 DEBUG:teuthology.orchestra.run.smithi154:> ') 2024-09-16T01:34:41.132 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:34:41.356 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:41.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:41.363 DEBUG:teuthology.orchestra.run.smithi154:> ip netns list 2024-09-16T01:34:41.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:41.378 DEBUG:teuthology.orchestra.run.smithi154:> ip netns list-id 2024-09-16T01:34:41.433 DEBUG:teuthology.orchestra.run.smithi154:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:34:41.433 DEBUG:teuthology.orchestra.run.smithi154:> set -e 2024-09-16T01:34:41.433 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T01:34:41.433 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-09-16T01:34:41.433 DEBUG:teuthology.orchestra.run.smithi154:> ') 2024-09-16T01:34:41.521 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:34:41.544 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:41.551 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-09-16T01:34:41.551 DEBUG:teuthology.orchestra.run.smithi154:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:34:41.551 DEBUG:teuthology.orchestra.run.smithi154:> set -e 2024-09-16T01:34:41.551 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-09-16T01:34:41.551 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-16T01:34:41.551 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-09-16T01:34:41.551 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-09-16T01:34:41.551 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-09-16T01:34:41.551 DEBUG:teuthology.orchestra.run.smithi154:> ') 2024-09-16T01:34:41.599 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:34:41.668 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:41.674 DEBUG:teuthology.orchestra.run.smithi154:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:34:41.674 DEBUG:teuthology.orchestra.run.smithi154:> set -e 2024-09-16T01:34:41.674 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip link set brx.0 up 2024-09-16T01:34:41.674 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip link set dev brx.0 master ceph-brx 2024-09-16T01:34:41.674 DEBUG:teuthology.orchestra.run.smithi154:> ') 2024-09-16T01:34:41.722 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:34:41.744 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:41.750 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:34:41.751 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-16T01:34:41.766 INFO:teuthology.orchestra.run.smithi154.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-16T01:34:41.766 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:34:41.767 DEBUG:teuthology.orchestra.run.smithi154:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-16T01:34:41.821 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-16T01:34:41.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:41.822 DEBUG:teuthology.orchestra.run.smithi154:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-16T01:34:41.911 INFO:teuthology.orchestra.run.smithi154.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-16T01:34:41.911 INFO:teuthology.orchestra.run.smithi154.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-16T01:34:41.912 INFO:teuthology.orchestra.run.smithi154.stdout:invalid new device string format 2024-09-16T01:34:42.034 INFO:teuthology.orchestra.run.smithi154.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-16T01:34:42.034 INFO:teuthology.orchestra.run.smithi154.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-16T01:34:42.034 INFO:teuthology.orchestra.run.smithi154.stdout:invalid new device string format 2024-09-16T01:34:42.034 INFO:teuthology.orchestra.run.smithi154.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:34:42.034 INFO:teuthology.orchestra.run.smithi154.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:34:42.039 INFO:teuthology.orchestra.run.smithi154.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-16T01:34:42.039 INFO:teuthology.orchestra.run.smithi154.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-16T01:34:42.039 INFO:teuthology.orchestra.run.smithi154.stdout:invalid new device string format 2024-09-16T01:34:42.039 INFO:teuthology.orchestra.run.smithi154.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:34:42.039 INFO:teuthology.orchestra.run.smithi154.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:34:42.039 INFO:teuthology.orchestra.run.smithi154.stdout:mount.ceph: resolved to: "172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790" 2024-09-16T01:34:42.040 INFO:teuthology.orchestra.run.smithi154.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2024-09-16T01:34:42.040 INFO:teuthology.orchestra.run.smithi154.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=5ace2c91-46f9-482b-8a89-731fbdc171bf" will pass to kernel 2024-09-16T01:34:42.040 INFO:teuthology.orchestra.run.smithi154.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-16T01:34:42.040 INFO:teuthology.orchestra.run.smithi154.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:34:42.040 INFO:teuthology.orchestra.run.smithi154.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:34:42.040 INFO:teuthology.orchestra.run.smithi154.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:34:42.040 INFO:teuthology.orchestra.run.smithi154.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:34:42.042 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:34:42.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:42.042 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-16T01:34:42.071 DEBUG:teuthology.orchestra.run.smithi154:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:34:42.161 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:42 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:34:42.163 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:42.168 INFO:teuthology.orchestra.run.smithi154.stdout:client4531 2024-09-16T01:34:42.168 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:34:42.169 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:34:42.184 DEBUG:teuthology.orchestra.run.smithi154:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:34:42.273 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:42 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:34:42.274 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:42.280 INFO:teuthology.orchestra.run.smithi154.stdout:client4531 2024-09-16T01:34:42.281 DEBUG:teuthology.orchestra.run.smithi154:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:34:42.329 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:42 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:34:42.331 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:42.337 INFO:teuthology.orchestra.run.smithi154.stdout:client4531 2024-09-16T01:34:42.337 DEBUG:teuthology.orchestra.run.smithi154:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/5ace2c91-46f9-482b-8a89-731fbdc171bf.client4531/status') 2024-09-16T01:34:42.386 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:34:42.396 INFO:teuthology.orchestra.run.smithi154.stdout:instance: client.4531 (3)192.168.0.1:0/2041292033 2024-09-16T01:34:42.396 INFO:teuthology.orchestra.run.smithi154.stdout:blocklisted: false 2024-09-16T01:34:42.396 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records in 2024-09-16T01:34:42.396 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records out 2024-09-16T01:34:42.396 INFO:teuthology.orchestra.run.smithi154.stderr:69 bytes copied, 2.3219e-05 s, 3.0 MB/s 2024-09-16T01:34:42.397 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:42.403 INFO:tasks.kclient:client.3 config is {'syntax': 'v1'} 2024-09-16T01:34:42.403 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-16T01:34:42.403 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-09-16T01:34:42.403 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:34:42.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.513+0000 7fa1c63fd640 1 Processor -- start 2024-09-16T01:34:42.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.513+0000 7fa1c63fd640 1 -- start start 2024-09-16T01:34:42.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.515+0000 7fa1c63fd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa1c015c170 0x7fa1c015c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:42.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.515+0000 7fa1c63fd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1c00cf6c0 0x7fa1c00cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:42.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.515+0000 7fa1c63fd640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1c00b49c0 con 0x7fa1c00b5dd0 2024-09-16T01:34:42.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.515+0000 7fa1c63fd640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa1c015ca90 con 0x7fa1c015c170 2024-09-16T01:34:42.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.515+0000 7fa1c63fd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa1c00b4cf0 con 0x7fa1c00cf6c0 2024-09-16T01:34:42.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.515+0000 7fa1bf7fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1c00cf6c0 0x7fa1c00cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:42.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.515+0000 7fa1bffff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa1c015c170 0x7fa1c015c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:42.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.515+0000 7fa1c499a640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fa1c00b5dd0 0x7fa1c015ffb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:34528/0 (socket says 172.21.15.88:34528) 2024-09-16T01:34:42.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.515+0000 7fa1bf7fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1c00cf6c0 0x7fa1c00cfb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:37772/0 (socket says 172.21.15.88:37772) 2024-09-16T01:34:42.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.515+0000 7fa1c499a640 1 -- 172.21.15.88:0/3710615601 learned_addr learned my addr 172.21.15.88:0/3710615601 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:42.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.515+0000 7fa1bffff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa1c015c170 0x7fa1c015c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:54532/0 (socket says 172.21.15.88:54532) 2024-09-16T01:34:42.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.515+0000 7fa1bf7fe640 1 -- 172.21.15.88:0/3710615601 >> v2:172.21.15.154:3300/0 conn(0x7fa1c015c170 msgr2=0x7fa1c015c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:42.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.515+0000 7fa1bf7fe640 1 --2- 172.21.15.88:0/3710615601 >> v2:172.21.15.154:3300/0 conn(0x7fa1c015c170 0x7fa1c015c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:42.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.515+0000 7fa1bf7fe640 1 -- 172.21.15.88:0/3710615601 >> v1:172.21.15.154:6789/0 conn(0x7fa1c00b5dd0 legacy=0x7fa1c015ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:42.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.515+0000 7fa1bf7fe640 1 -- 172.21.15.88:0/3710615601 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1c00b4fd0 con 0x7fa1c00cf6c0 2024-09-16T01:34:42.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.515+0000 7fa1beffd640 1 -- 172.21.15.88:0/3710615601 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1258052733 0 0) 0x7fa1c00b49c0 con 0x7fa1c00b5dd0 2024-09-16T01:34:42.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.515+0000 7fa1bffff640 1 --2- 172.21.15.88:0/3710615601 >> v2:172.21.15.154:3300/0 conn(0x7fa1c015c170 0x7fa1c015c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:34:42.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.515+0000 7fa1bf7fe640 1 --2- 172.21.15.88:0/3710615601 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1c00cf6c0 0x7fa1c00cfb00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa1b40097a0 tx=0x7fa1b402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69c7c7194ea30cf9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:42.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.515+0000 7fa1beffd640 1 -- 172.21.15.88:0/3710615601 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b4037070 con 0x7fa1c00cf6c0 2024-09-16T01:34:42.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.515+0000 7fa1beffd640 1 -- 172.21.15.88:0/3710615601 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1b4032440 con 0x7fa1c00cf6c0 2024-09-16T01:34:42.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.515+0000 7fa1beffd640 1 -- 172.21.15.88:0/3710615601 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b4032810 con 0x7fa1c00cf6c0 2024-09-16T01:34:42.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.516+0000 7fa1c63fd640 1 -- 172.21.15.88:0/3710615601 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1c00cf6c0 msgr2=0x7fa1c00cfb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:42.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.516+0000 7fa1c63fd640 1 --2- 172.21.15.88:0/3710615601 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1c00cf6c0 0x7fa1c00cfb00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa1b40097a0 tx=0x7fa1b402a180 comp rx=0 tx=0).stop 2024-09-16T01:34:42.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.516+0000 7fa1c63fd640 1 -- 172.21.15.88:0/3710615601 shutdown_connections 2024-09-16T01:34:42.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.516+0000 7fa1c63fd640 1 --2- 172.21.15.88:0/3710615601 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1c00cf6c0 0x7fa1c00cfb00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:42.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.516+0000 7fa1c63fd640 1 --2- 172.21.15.88:0/3710615601 >> v2:172.21.15.154:3300/0 conn(0x7fa1c015c170 0x7fa1c015c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:42.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.516+0000 7fa1c63fd640 1 -- 172.21.15.88:0/3710615601 >> 172.21.15.88:0/3710615601 conn(0x7fa1c006bc10 msgr2=0x7fa1c006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:42.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.516+0000 7fa1c63fd640 1 -- 172.21.15.88:0/3710615601 shutdown_connections 2024-09-16T01:34:42.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.516+0000 7fa1c63fd640 1 -- 172.21.15.88:0/3710615601 wait complete. 2024-09-16T01:34:42.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.517+0000 7fa1c63fd640 1 Processor -- start 2024-09-16T01:34:42.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.517+0000 7fa1c63fd640 1 -- start start 2024-09-16T01:34:42.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.518+0000 7fa1c63fd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1c00b5dd0 0x7fa1c01f3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:42.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.518+0000 7fa1c63fd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1c00cf6c0 0x7fa1c01ef590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:42.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.518+0000 7fa1c63fd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1c015c170 0x7fa1c01561e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:42.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.518+0000 7fa1c63fd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c00b5870 con 0x7fa1c015c170 2024-09-16T01:34:42.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.518+0000 7fa1c63fd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c00b5590 con 0x7fa1c00b5dd0 2024-09-16T01:34:42.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.518+0000 7fa1c63fd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa1c00b5b50 con 0x7fa1c00cf6c0 2024-09-16T01:34:42.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.518+0000 7fa1c499a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1c015c170 0x7fa1c01561e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:42.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.518+0000 7fa1c499a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1c015c170 0x7fa1c01561e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54764/0 (socket says 172.21.15.88:54764) 2024-09-16T01:34:42.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.518+0000 7fa1c499a640 1 -- 172.21.15.88:0/862090197 learned_addr learned my addr 172.21.15.88:0/862090197 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:42.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.518+0000 7fa1bf7fe640 1 --2- 172.21.15.88:0/862090197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1c00cf6c0 0x7fa1c01ef590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:42.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.518+0000 7fa1bffff640 1 --2- 172.21.15.88:0/862090197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1c00b5dd0 0x7fa1c01f3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:42.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.518+0000 7fa1c499a640 1 -- 172.21.15.88:0/862090197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1c00cf6c0 msgr2=0x7fa1c01ef590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:42.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.518+0000 7fa1c499a640 1 --2- 172.21.15.88:0/862090197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1c00cf6c0 0x7fa1c01ef590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:42.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.518+0000 7fa1c499a640 1 -- 172.21.15.88:0/862090197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1c00b5dd0 msgr2=0x7fa1c01f3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:42.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.518+0000 7fa1c499a640 1 --2- 172.21.15.88:0/862090197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1c00b5dd0 0x7fa1c01f3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:42.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.518+0000 7fa1c499a640 1 -- 172.21.15.88:0/862090197 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1c01efad0 con 0x7fa1c015c170 2024-09-16T01:34:42.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.519+0000 7fa1c499a640 1 --2- 172.21.15.88:0/862090197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1c015c170 0x7fa1c01561e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa1b800c800 tx=0x7fa1b800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:42.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.519+0000 7fa1bcff9640 1 -- 172.21.15.88:0/862090197 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b8018070 con 0x7fa1c015c170 2024-09-16T01:34:42.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.519+0000 7fa1bcff9640 1 -- 172.21.15.88:0/862090197 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1b8004d80 con 0x7fa1c015c170 2024-09-16T01:34:42.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.519+0000 7fa1c63fd640 1 -- 172.21.15.88:0/862090197 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1c0156ae0 con 0x7fa1c015c170 2024-09-16T01:34:42.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.519+0000 7fa1bcff9640 1 -- 172.21.15.88:0/862090197 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b800a880 con 0x7fa1c015c170 2024-09-16T01:34:42.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.519+0000 7fa1c63fd640 1 -- 172.21.15.88:0/862090197 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1c01570a0 con 0x7fa1c015c170 2024-09-16T01:34:42.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.520+0000 7fa1c63fd640 1 -- 172.21.15.88:0/862090197 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1c01522f0 con 0x7fa1c015c170 2024-09-16T01:34:42.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.521+0000 7fa1bcff9640 1 -- 172.21.15.88:0/862090197 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa1b801b070 con 0x7fa1c015c170 2024-09-16T01:34:42.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.521+0000 7fa1bcff9640 1 --2- 172.21.15.88:0/862090197 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa1ac06c2e0 0x7fa1ac06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:42.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.521+0000 7fa1bcff9640 1 -- 172.21.15.88:0/862090197 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa1b808a8b0 con 0x7fa1c015c170 2024-09-16T01:34:42.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.521+0000 7fa1bffff640 1 --2- 172.21.15.88:0/862090197 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa1ac06c2e0 0x7fa1ac06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:42.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.522+0000 7fa1bffff640 1 --2- 172.21.15.88:0/862090197 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa1ac06c2e0 0x7fa1ac06e7a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0009560 tx=0x7fa1b0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:42.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.525+0000 7fa1bcff9640 1 -- 172.21.15.88:0/862090197 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa1c01522f0 con 0x7fa1c015c170 2024-09-16T01:34:42.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.727+0000 7fa1c63fd640 1 -- 172.21.15.88:0/862090197 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa1c015c550 con 0x7fa1c015c170 2024-09-16T01:34:42.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.728+0000 7fa1bcff9640 1 -- 172.21.15.88:0/862090197 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) ==== 53+0+73 (secure 0 0 0) 0x7fa1b8058a90 con 0x7fa1c015c170 2024-09-16T01:34:42.729 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:34:42.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.730+0000 7fa1c63fd640 1 -- 172.21.15.88:0/862090197 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa1ac06c2e0 msgr2=0x7fa1ac06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:42.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.730+0000 7fa1c63fd640 1 --2- 172.21.15.88:0/862090197 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa1ac06c2e0 0x7fa1ac06e7a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0009560 tx=0x7fa1b0009190 comp rx=0 tx=0).stop 2024-09-16T01:34:42.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.730+0000 7fa1c63fd640 1 -- 172.21.15.88:0/862090197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1c015c170 msgr2=0x7fa1c01561e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:42.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.730+0000 7fa1c63fd640 1 --2- 172.21.15.88:0/862090197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1c015c170 0x7fa1c01561e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa1b800c800 tx=0x7fa1b800ccc0 comp rx=0 tx=0).stop 2024-09-16T01:34:42.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.730+0000 7fa1c63fd640 1 -- 172.21.15.88:0/862090197 shutdown_connections 2024-09-16T01:34:42.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.730+0000 7fa1c63fd640 1 --2- 172.21.15.88:0/862090197 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa1ac06c2e0 0x7fa1ac06e7a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:42.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.730+0000 7fa1c63fd640 1 --2- 172.21.15.88:0/862090197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1c015c170 0x7fa1c01561e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:42.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.730+0000 7fa1c63fd640 1 --2- 172.21.15.88:0/862090197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1c00cf6c0 0x7fa1c01ef590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:42.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.730+0000 7fa1c63fd640 1 --2- 172.21.15.88:0/862090197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1c00b5dd0 0x7fa1c01f3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:42.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.730+0000 7fa1c63fd640 1 -- 172.21.15.88:0/862090197 >> 172.21.15.88:0/862090197 conn(0x7fa1c006bc10 msgr2=0x7fa1c01515a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:42.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.730+0000 7fa1c63fd640 1 -- 172.21.15.88:0/862090197 shutdown_connections 2024-09-16T01:34:42.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:42.730+0000 7fa1c63fd640 1 -- 172.21.15.88:0/862090197 wait complete. 2024-09-16T01:34:42.742 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-16T01:34:42.742 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:34:42.742 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi154.front.sepia.ceph.com 2024-09-16T01:34:42.742 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-09-16T01:34:42.742 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-09-16T01:34:42.742 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-16T01:34:42.742 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-16T01:34:42.742 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:34:42.742 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-09-16T01:34:42.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:42.742 DEBUG:teuthology.orchestra.run.smithi154:> ip addr 2024-09-16T01:34:42.760 INFO:teuthology.orchestra.run.smithi154.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:34:42.760 INFO:teuthology.orchestra.run.smithi154.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:34:42.760 INFO:teuthology.orchestra.run.smithi154.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:34:42.760 INFO:teuthology.orchestra.run.smithi154.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:42.760 INFO:teuthology.orchestra.run.smithi154.stdout: inet6 ::1/128 scope host 2024-09-16T01:34:42.760 INFO:teuthology.orchestra.run.smithi154.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:42.760 INFO:teuthology.orchestra.run.smithi154.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:34:42.760 INFO:teuthology.orchestra.run.smithi154.stdout: link/ether 0c:c4:7a:d9:1a:54 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:34:42.760 INFO:teuthology.orchestra.run.smithi154.stdout: altname enp7s0f0 2024-09-16T01:34:42.760 INFO:teuthology.orchestra.run.smithi154.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:34:42.761 INFO:teuthology.orchestra.run.smithi154.stdout: link/ether 0c:c4:7a:d9:1a:55 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:34:42.761 INFO:teuthology.orchestra.run.smithi154.stdout: altname enp7s0f1 2024-09-16T01:34:42.761 INFO:teuthology.orchestra.run.smithi154.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:34:42.761 INFO:teuthology.orchestra.run.smithi154.stdout: link/ether 0c:c4:7a:88:77:60 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:34:42.761 INFO:teuthology.orchestra.run.smithi154.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:34:42.761 INFO:teuthology.orchestra.run.smithi154.stdout: link/ether 0c:c4:7a:88:77:61 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:34:42.761 INFO:teuthology.orchestra.run.smithi154.stdout: inet 172.21.15.154/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:34:42.761 INFO:teuthology.orchestra.run.smithi154.stdout: valid_lft 42744sec preferred_lft 42744sec 2024-09-16T01:34:42.761 INFO:teuthology.orchestra.run.smithi154.stdout: inet6 fe80::ec4:7aff:fe88:7761/64 scope link noprefixroute 2024-09-16T01:34:42.761 INFO:teuthology.orchestra.run.smithi154.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:42.761 INFO:teuthology.orchestra.run.smithi154.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:34:42.761 INFO:teuthology.orchestra.run.smithi154.stdout: link/ether da:7c:1d:79:3b:16 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:34:42.761 INFO:teuthology.orchestra.run.smithi154.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T01:34:42.761 INFO:teuthology.orchestra.run.smithi154.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:42.762 INFO:teuthology.orchestra.run.smithi154.stdout: inet6 fe80::a01a:32ff:feb3:c9d9/64 scope link 2024-09-16T01:34:42.762 INFO:teuthology.orchestra.run.smithi154.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:42.762 INFO:teuthology.orchestra.run.smithi154.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:34:42.762 INFO:teuthology.orchestra.run.smithi154.stdout: link/ether da:7c:1d:79:3b:16 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T01:34:42.762 INFO:teuthology.orchestra.run.smithi154.stdout: inet6 fe80::d87c:1dff:fe79:3b16/64 scope link tentative 2024-09-16T01:34:42.762 INFO:teuthology.orchestra.run.smithi154.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:42.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:42.762 DEBUG:teuthology.orchestra.run.smithi154:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:34:42.824 INFO:teuthology.orchestra.run.smithi154.stdout:1 2024-09-16T01:34:42.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:42.826 DEBUG:teuthology.orchestra.run.smithi154:> route 2024-09-16T01:34:42.882 INFO:teuthology.orchestra.run.smithi154.stdout:Kernel IP routing table 2024-09-16T01:34:42.883 INFO:teuthology.orchestra.run.smithi154.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:34:42.883 INFO:teuthology.orchestra.run.smithi154.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:34:42.883 INFO:teuthology.orchestra.run.smithi154.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:34:42.883 INFO:teuthology.orchestra.run.smithi154.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:34:42.883 DEBUG:teuthology.orchestra.run.smithi154:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:34:42.883 DEBUG:teuthology.orchestra.run.smithi154:> set -e 2024-09-16T01:34:42.883 DEBUG:teuthology.orchestra.run.smithi154:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:34:42.883 DEBUG:teuthology.orchestra.run.smithi154:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:34:42.883 DEBUG:teuthology.orchestra.run.smithi154:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:34:42.883 DEBUG:teuthology.orchestra.run.smithi154:> ') 2024-09-16T01:34:42.971 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:34:43.003 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:43.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:43.009 DEBUG:teuthology.orchestra.run.smithi154:> ip netns list 2024-09-16T01:34:43.025 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-16T01:34:43.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:43.025 DEBUG:teuthology.orchestra.run.smithi154:> ip netns list-id 2024-09-16T01:34:43.081 INFO:teuthology.orchestra.run.smithi154.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-09-16T01:34:43.081 DEBUG:teuthology.orchestra.run.smithi154:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:34:43.081 DEBUG:teuthology.orchestra.run.smithi154:> set -e 2024-09-16T01:34:43.082 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T01:34:43.082 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-09-16T01:34:43.082 DEBUG:teuthology.orchestra.run.smithi154:> ') 2024-09-16T01:34:43.171 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:34:43.194 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:43.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:43.200 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-09-16T01:34:43.224 INFO:teuthology.orchestra.run.smithi154.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:34:43.225 INFO:teuthology.orchestra.run.smithi154.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:34:43.225 INFO:teuthology.orchestra.run.smithi154.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:34:43.225 INFO:teuthology.orchestra.run.smithi154.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:43.225 INFO:teuthology.orchestra.run.smithi154.stdout: inet6 ::1/128 scope host 2024-09-16T01:34:43.225 INFO:teuthology.orchestra.run.smithi154.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:43.225 INFO:teuthology.orchestra.run.smithi154.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:34:43.225 INFO:teuthology.orchestra.run.smithi154.stdout: link/ether be:8a:1a:4b:5f:24 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-16T01:34:43.225 INFO:teuthology.orchestra.run.smithi154.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-16T01:34:43.225 INFO:teuthology.orchestra.run.smithi154.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:43.225 INFO:teuthology.orchestra.run.smithi154.stdout: inet6 fe80::bc8a:1aff:fe4b:5f24/64 scope link tentative 2024-09-16T01:34:43.225 INFO:teuthology.orchestra.run.smithi154.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:43.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:43.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-09-16T01:34:43.292 INFO:teuthology.orchestra.run.smithi154.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:34:43.293 INFO:teuthology.orchestra.run.smithi154.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:34:43.293 INFO:teuthology.orchestra.run.smithi154.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:34:43.293 INFO:teuthology.orchestra.run.smithi154.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:43.293 INFO:teuthology.orchestra.run.smithi154.stdout: inet6 ::1/128 scope host 2024-09-16T01:34:43.293 INFO:teuthology.orchestra.run.smithi154.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:43.293 INFO:teuthology.orchestra.run.smithi154.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:34:43.293 INFO:teuthology.orchestra.run.smithi154.stdout: link/ether be:8a:1a:4b:5f:24 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-16T01:34:43.293 INFO:teuthology.orchestra.run.smithi154.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-16T01:34:43.294 INFO:teuthology.orchestra.run.smithi154.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:43.294 INFO:teuthology.orchestra.run.smithi154.stdout: inet6 fe80::bc8a:1aff:fe4b:5f24/64 scope link tentative 2024-09-16T01:34:43.294 INFO:teuthology.orchestra.run.smithi154.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:34:43.295 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-09-16T01:34:43.295 DEBUG:teuthology.orchestra.run.smithi154:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:34:43.295 DEBUG:teuthology.orchestra.run.smithi154:> set -e 2024-09-16T01:34:43.296 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-09-16T01:34:43.296 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-09-16T01:34:43.296 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-09-16T01:34:43.296 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-09-16T01:34:43.296 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-09-16T01:34:43.296 DEBUG:teuthology.orchestra.run.smithi154:> ') 2024-09-16T01:34:43.384 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:34:43.447 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:43.453 DEBUG:teuthology.orchestra.run.smithi154:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:34:43.453 DEBUG:teuthology.orchestra.run.smithi154:> set -e 2024-09-16T01:34:43.453 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip link set brx.1 up 2024-09-16T01:34:43.453 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip link set dev brx.1 master ceph-brx 2024-09-16T01:34:43.454 DEBUG:teuthology.orchestra.run.smithi154:> ') 2024-09-16T01:34:43.502 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:34:43.525 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:43.532 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:34:43.532 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-09-16T01:34:43.548 INFO:teuthology.orchestra.run.smithi154.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-09-16T01:34:43.548 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:34:43.548 DEBUG:teuthology.orchestra.run.smithi154:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-09-16T01:34:43.603 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-16T01:34:43.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:43.603 DEBUG:teuthology.orchestra.run.smithi154:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-09-16T01:34:43.689 INFO:teuthology.orchestra.run.smithi154.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-09-16T01:34:43.689 INFO:teuthology.orchestra.run.smithi154.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-09-16T01:34:43.689 INFO:teuthology.orchestra.run.smithi154.stdout:invalid new device string format 2024-09-16T01:34:43.698 INFO:teuthology.orchestra.run.smithi154.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-09-16T01:34:43.698 INFO:teuthology.orchestra.run.smithi154.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-09-16T01:34:43.698 INFO:teuthology.orchestra.run.smithi154.stdout:invalid new device string format 2024-09-16T01:34:43.698 INFO:teuthology.orchestra.run.smithi154.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:34:43.698 INFO:teuthology.orchestra.run.smithi154.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:34:43.704 INFO:teuthology.orchestra.run.smithi154.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-09-16T01:34:43.704 INFO:teuthology.orchestra.run.smithi154.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-09-16T01:34:43.704 INFO:teuthology.orchestra.run.smithi154.stdout:invalid new device string format 2024-09-16T01:34:43.704 INFO:teuthology.orchestra.run.smithi154.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:34:43.704 INFO:teuthology.orchestra.run.smithi154.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:34:43.704 INFO:teuthology.orchestra.run.smithi154.stdout:mount.ceph: resolved to: "172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790" 2024-09-16T01:34:43.704 INFO:teuthology.orchestra.run.smithi154.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2024-09-16T01:34:43.704 INFO:teuthology.orchestra.run.smithi154.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs,key=3,fsid=5ace2c91-46f9-482b-8a89-731fbdc171bf" will pass to kernel 2024-09-16T01:34:43.705 INFO:teuthology.orchestra.run.smithi154.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-09-16T01:34:43.705 INFO:teuthology.orchestra.run.smithi154.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:34:43.705 INFO:teuthology.orchestra.run.smithi154.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:34:43.705 INFO:teuthology.orchestra.run.smithi154.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:34:43.705 INFO:teuthology.orchestra.run.smithi154.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:34:43.707 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:34:43.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:43.707 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-09-16T01:34:43.732 DEBUG:teuthology.orchestra.run.smithi154:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:34:43.821 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:34:43.823 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:43.829 INFO:teuthology.orchestra.run.smithi154.stdout:client4543 2024-09-16T01:34:43.830 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:34:43.830 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:34:43.846 DEBUG:teuthology.orchestra.run.smithi154:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:34:43.934 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:34:43.936 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:43.942 INFO:teuthology.orchestra.run.smithi154.stdout:client4543 2024-09-16T01:34:43.942 DEBUG:teuthology.orchestra.run.smithi154:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:34:43.990 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:34:43.992 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:43.998 INFO:teuthology.orchestra.run.smithi154.stdout:client4543 2024-09-16T01:34:43.998 DEBUG:teuthology.orchestra.run.smithi154:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/5ace2c91-46f9-482b-8a89-731fbdc171bf.client4543/status') 2024-09-16T01:34:44.046 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:34:44.056 INFO:teuthology.orchestra.run.smithi154.stdout:instance: client.4543 (3)192.168.0.2:0/4046239304 2024-09-16T01:34:44.056 INFO:teuthology.orchestra.run.smithi154.stdout:blocklisted: false 2024-09-16T01:34:44.057 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records in 2024-09-16T01:34:44.057 INFO:teuthology.orchestra.run.smithi154.stderr:0+1 records out 2024-09-16T01:34:44.057 INFO:teuthology.orchestra.run.smithi154.stderr:69 bytes copied, 3.1625e-05 s, 2.2 MB/s 2024-09-16T01:34:44.057 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T01:34:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:34:44.064 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-09-16T01:34:44.096 INFO:tasks.cephfs_test_runner:Starting test: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage) 2024-09-16T01:34:44.096 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:34:44.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.206+0000 7fe3aa0ba640 1 Processor -- start 2024-09-16T01:34:44.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.206+0000 7fe3aa0ba640 1 -- start start 2024-09-16T01:34:44.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.207+0000 7fe3aa0ba640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3a40cfb90 0x7fe3a40d1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:44.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.207+0000 7fe3aa0ba640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe3a40d2090 0x7fe3a406f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:44.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.207+0000 7fe3aa0ba640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3a40d1870 con 0x7fe3a406f880 2024-09-16T01:34:44.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.207+0000 7fe3aa0ba640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe3a415d810 con 0x7fe3a40d2090 2024-09-16T01:34:44.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.207+0000 7fe3aa0ba640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe3a415cc00 con 0x7fe3a40cfb90 2024-09-16T01:34:44.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.207+0000 7fe39bfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe3a40d2090 0x7fe3a406f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:44.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.207+0000 7fe39bfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe3a40d2090 0x7fe3a406f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54766/0 (socket says 172.21.15.88:54766) 2024-09-16T01:34:44.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.207+0000 7fe39bfff640 1 -- 172.21.15.88:0/3654815446 learned_addr learned my addr 172.21.15.88:0/3654815446 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:44.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.207+0000 7fe3a37fe640 1 --2- 172.21.15.88:0/3654815446 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3a40cfb90 0x7fe3a40d1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:44.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.207+0000 7fe39bfff640 1 -- 172.21.15.88:0/3654815446 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3a40cfb90 msgr2=0x7fe3a40d1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:44.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.207+0000 7fe39bfff640 1 --2- 172.21.15.88:0/3654815446 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3a40cfb90 0x7fe3a40d1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.207+0000 7fe39bfff640 1 -- 172.21.15.88:0/3654815446 >> v1:172.21.15.154:6789/0 conn(0x7fe3a406f880 legacy=0x7fe3a415ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:44.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.207+0000 7fe39bfff640 1 -- 172.21.15.88:0/3654815446 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3a415cee0 con 0x7fe3a40d2090 2024-09-16T01:34:44.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.208+0000 7fe39bfff640 1 --2- 172.21.15.88:0/3654815446 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3a40d2090 0x7fe3a406f340 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe38c009870 tx=0x7fe38c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7630a0ed0c698cfc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:44.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.208+0000 7fe3a2ffd640 1 -- 172.21.15.88:0/3654815446 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe38c037070 con 0x7fe3a40d2090 2024-09-16T01:34:44.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.208+0000 7fe3a2ffd640 1 -- 172.21.15.88:0/3654815446 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe38c032440 con 0x7fe3a40d2090 2024-09-16T01:34:44.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.208+0000 7fe3a2ffd640 1 -- 172.21.15.88:0/3654815446 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe38c032810 con 0x7fe3a40d2090 2024-09-16T01:34:44.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.209+0000 7fe3aa0ba640 1 -- 172.21.15.88:0/3654815446 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3a40d2090 msgr2=0x7fe3a406f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:44.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.209+0000 7fe3aa0ba640 1 --2- 172.21.15.88:0/3654815446 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3a40d2090 0x7fe3a406f340 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe38c009870 tx=0x7fe38c02a270 comp rx=0 tx=0).stop 2024-09-16T01:34:44.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.209+0000 7fe3aa0ba640 1 -- 172.21.15.88:0/3654815446 shutdown_connections 2024-09-16T01:34:44.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.209+0000 7fe3aa0ba640 1 --2- 172.21.15.88:0/3654815446 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3a40d2090 0x7fe3a406f340 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.209+0000 7fe3aa0ba640 1 --2- 172.21.15.88:0/3654815446 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3a40cfb90 0x7fe3a40d1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.209+0000 7fe3aa0ba640 1 -- 172.21.15.88:0/3654815446 >> 172.21.15.88:0/3654815446 conn(0x7fe3a4062630 msgr2=0x7fe3a4062a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:44.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.209+0000 7fe3aa0ba640 1 -- 172.21.15.88:0/3654815446 shutdown_connections 2024-09-16T01:34:44.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.209+0000 7fe3aa0ba640 1 -- 172.21.15.88:0/3654815446 wait complete. 2024-09-16T01:34:44.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.210+0000 7fe3aa0ba640 1 Processor -- start 2024-09-16T01:34:44.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.210+0000 7fe3aa0ba640 1 -- start start 2024-09-16T01:34:44.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.211+0000 7fe3aa0ba640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3a406f880 0x7fe3a40cdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:44.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.211+0000 7fe3aa0ba640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3a40cfb90 0x7fe3a41540b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:44.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.211+0000 7fe3aa0ba640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3a40d2090 0x7fe3a40b2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:44.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.211+0000 7fe3aa0ba640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a40cff70 con 0x7fe3a40d2090 2024-09-16T01:34:44.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.211+0000 7fe3aa0ba640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a415d4a0 con 0x7fe3a406f880 2024-09-16T01:34:44.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.211+0000 7fe3aa0ba640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe3a40b55b0 con 0x7fe3a40cfb90 2024-09-16T01:34:44.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.211+0000 7fe3a3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3a40d2090 0x7fe3a40b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:44.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.211+0000 7fe3a3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3a40d2090 0x7fe3a40b2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54768/0 (socket says 172.21.15.88:54768) 2024-09-16T01:34:44.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.211+0000 7fe3a3fff640 1 -- 172.21.15.88:0/749422001 learned_addr learned my addr 172.21.15.88:0/749422001 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:44.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.211+0000 7fe39bfff640 1 --2- 172.21.15.88:0/749422001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3a40cfb90 0x7fe3a41540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:44.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.211+0000 7fe3a37fe640 1 --2- 172.21.15.88:0/749422001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3a406f880 0x7fe3a40cdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:44.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.211+0000 7fe3a3fff640 1 -- 172.21.15.88:0/749422001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3a40cfb90 msgr2=0x7fe3a41540b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:44.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.211+0000 7fe3a3fff640 1 --2- 172.21.15.88:0/749422001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3a40cfb90 0x7fe3a41540b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.211+0000 7fe3a3fff640 1 -- 172.21.15.88:0/749422001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3a406f880 msgr2=0x7fe3a40cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:44.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.211+0000 7fe3a3fff640 1 --2- 172.21.15.88:0/749422001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3a406f880 0x7fe3a40cdbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.211+0000 7fe3a3fff640 1 -- 172.21.15.88:0/749422001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3a4154a90 con 0x7fe3a40d2090 2024-09-16T01:34:44.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.211+0000 7fe3a3fff640 1 --2- 172.21.15.88:0/749422001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3a40d2090 0x7fe3a40b2700 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00c7a0 tx=0x7fe39c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:44.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.211+0000 7fe3a0ff9640 1 -- 172.21.15.88:0/749422001 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39c003b70 con 0x7fe3a40d2090 2024-09-16T01:34:44.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.211+0000 7fe3a0ff9640 1 -- 172.21.15.88:0/749422001 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe39c003d30 con 0x7fe3a40d2090 2024-09-16T01:34:44.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.211+0000 7fe3aa0ba640 1 -- 172.21.15.88:0/749422001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3a4154d70 con 0x7fe3a40d2090 2024-09-16T01:34:44.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.211+0000 7fe3a0ff9640 1 -- 172.21.15.88:0/749422001 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39c00aa90 con 0x7fe3a40d2090 2024-09-16T01:34:44.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.212+0000 7fe3aa0ba640 1 -- 172.21.15.88:0/749422001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3a4155330 con 0x7fe3a40d2090 2024-09-16T01:34:44.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.212+0000 7fe3aa0ba640 1 -- 172.21.15.88:0/749422001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3a40d1b50 con 0x7fe3a40d2090 2024-09-16T01:34:44.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.213+0000 7fe3a0ff9640 1 -- 172.21.15.88:0/749422001 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe39c00ac30 con 0x7fe3a40d2090 2024-09-16T01:34:44.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.213+0000 7fe3a0ff9640 1 --2- 172.21.15.88:0/749422001 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe38406c2b0 0x7fe38406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:44.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.214+0000 7fe3a37fe640 1 --2- 172.21.15.88:0/749422001 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe38406c2b0 0x7fe38406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:44.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.214+0000 7fe3a0ff9640 1 -- 172.21.15.88:0/749422001 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe39c08b260 con 0x7fe3a40d2090 2024-09-16T01:34:44.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.214+0000 7fe3a37fe640 1 --2- 172.21.15.88:0/749422001 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe38406c2b0 0x7fe38406e770 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe394009b40 tx=0x7fe394009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:44.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.218+0000 7fe3a0ff9640 1 -- 172.21.15.88:0/749422001 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe39c059380 con 0x7fe3a40d2090 2024-09-16T01:34:44.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.386+0000 7fe3aa0ba640 1 -- 172.21.15.88:0/749422001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe3a415c430 con 0x7fe3a40d2090 2024-09-16T01:34:44.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.387+0000 7fe3a0ff9640 1 -- 172.21.15.88:0/749422001 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16641 (secure 0 0 0) 0x7fe39c05d680 con 0x7fe3a40d2090 2024-09-16T01:34:44.389 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:44.389 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:34:23.181812+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:34:19.324549+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:34:20.494051+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:34:44.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.389+0000 7fe3aa0ba640 1 -- 172.21.15.88:0/749422001 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe38406c2b0 msgr2=0x7fe38406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:44.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.389+0000 7fe3aa0ba640 1 --2- 172.21.15.88:0/749422001 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe38406c2b0 0x7fe38406e770 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe394009b40 tx=0x7fe394009190 comp rx=0 tx=0).stop 2024-09-16T01:34:44.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.389+0000 7fe3aa0ba640 1 -- 172.21.15.88:0/749422001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3a40d2090 msgr2=0x7fe3a40b2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:44.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.389+0000 7fe3aa0ba640 1 --2- 172.21.15.88:0/749422001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3a40d2090 0x7fe3a40b2700 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00c7a0 tx=0x7fe39c00cc60 comp rx=0 tx=0).stop 2024-09-16T01:34:44.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.389+0000 7fe3aa0ba640 1 -- 172.21.15.88:0/749422001 shutdown_connections 2024-09-16T01:34:44.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.389+0000 7fe3aa0ba640 1 --2- 172.21.15.88:0/749422001 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe38406c2b0 0x7fe38406e770 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.389+0000 7fe3aa0ba640 1 --2- 172.21.15.88:0/749422001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3a40d2090 0x7fe3a40b2700 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.389+0000 7fe3aa0ba640 1 --2- 172.21.15.88:0/749422001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3a40cfb90 0x7fe3a41540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.389+0000 7fe3aa0ba640 1 --2- 172.21.15.88:0/749422001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3a406f880 0x7fe3a40cdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.389+0000 7fe3aa0ba640 1 -- 172.21.15.88:0/749422001 >> 172.21.15.88:0/749422001 conn(0x7fe3a4062630 msgr2=0x7fe3a4155af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:44.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.389+0000 7fe3aa0ba640 1 -- 172.21.15.88:0/749422001 shutdown_connections 2024-09-16T01:34:44.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.389+0000 7fe3aa0ba640 1 -- 172.21.15.88:0/749422001 wait complete. 2024-09-16T01:34:44.401 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T01:34:10.547178+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-16T01:34:19.324549+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-16T01:34:20.494051+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T01:34:44.402 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T01:34:44.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.501+0000 7fa15bf6f640 1 Processor -- start 2024-09-16T01:34:44.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.502+0000 7fa15bf6f640 1 -- start start 2024-09-16T01:34:44.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.502+0000 7fa15bf6f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa15415e7a0 0x7fa15406caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:44.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.502+0000 7fa15bf6f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa1540cf1f0 0x7fa1540b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:44.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.502+0000 7fa15bf6f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1540cf860 con 0x7fa1540b6720 2024-09-16T01:34:44.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.502+0000 7fa15bf6f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa1540d2ca0 con 0x7fa1540cf1f0 2024-09-16T01:34:44.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.502+0000 7fa15bf6f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa1540cfb40 con 0x7fa15415e7a0 2024-09-16T01:34:44.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.503+0000 7fa15950a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa1540cf1f0 0x7fa1540b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:44.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.503+0000 7fa15950a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa1540cf1f0 0x7fa1540b61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54778/0 (socket says 172.21.15.88:54778) 2024-09-16T01:34:44.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.503+0000 7fa15950a640 1 -- 172.21.15.88:0/3955260185 learned_addr learned my addr 172.21.15.88:0/3955260185 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:44.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.503+0000 7fa159d0b640 1 --2- 172.21.15.88:0/3955260185 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa15415e7a0 0x7fa15406caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:44.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.503+0000 7fa15950a640 1 -- 172.21.15.88:0/3955260185 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa15415e7a0 msgr2=0x7fa15406caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:44.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.503+0000 7fa15950a640 1 --2- 172.21.15.88:0/3955260185 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa15415e7a0 0x7fa15406caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.503+0000 7fa15950a640 1 -- 172.21.15.88:0/3955260185 >> v1:172.21.15.154:6789/0 conn(0x7fa1540b6720 legacy=0x7fa1541623c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:44.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.503+0000 7fa15950a640 1 -- 172.21.15.88:0/3955260185 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1540cfe20 con 0x7fa1540cf1f0 2024-09-16T01:34:44.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.504+0000 7fa15950a640 1 --2- 172.21.15.88:0/3955260185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1540cf1f0 0x7fa1540b61e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa150009870 tx=0x7fa15002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80ea5661adf32c76 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:44.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.504+0000 7fa158d09640 1 -- 172.21.15.88:0/3955260185 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa150037070 con 0x7fa1540cf1f0 2024-09-16T01:34:44.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.504+0000 7fa158d09640 1 -- 172.21.15.88:0/3955260185 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa150032440 con 0x7fa1540cf1f0 2024-09-16T01:34:44.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.504+0000 7fa158d09640 1 -- 172.21.15.88:0/3955260185 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa150032810 con 0x7fa1540cf1f0 2024-09-16T01:34:44.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.504+0000 7fa15bf6f640 1 -- 172.21.15.88:0/3955260185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1540cf1f0 msgr2=0x7fa1540b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:44.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.504+0000 7fa15bf6f640 1 --2- 172.21.15.88:0/3955260185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1540cf1f0 0x7fa1540b61e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa150009870 tx=0x7fa15002a1c0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.504+0000 7fa15bf6f640 1 -- 172.21.15.88:0/3955260185 shutdown_connections 2024-09-16T01:34:44.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.504+0000 7fa15bf6f640 1 --2- 172.21.15.88:0/3955260185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1540cf1f0 0x7fa1540b61e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.504+0000 7fa15bf6f640 1 --2- 172.21.15.88:0/3955260185 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa15415e7a0 0x7fa15406caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.504+0000 7fa15bf6f640 1 -- 172.21.15.88:0/3955260185 >> 172.21.15.88:0/3955260185 conn(0x7fa1540d11e0 msgr2=0x7fa1540d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:44.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.504+0000 7fa15bf6f640 1 -- 172.21.15.88:0/3955260185 shutdown_connections 2024-09-16T01:34:44.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.504+0000 7fa15bf6f640 1 -- 172.21.15.88:0/3955260185 wait complete. 2024-09-16T01:34:44.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.506+0000 7fa15bf6f640 1 Processor -- start 2024-09-16T01:34:44.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.506+0000 7fa15bf6f640 1 -- start start 2024-09-16T01:34:44.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.506+0000 7fa15bf6f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1540b6720 0x7fa1540ca790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:44.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.506+0000 7fa15bf6f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1540cf1f0 0x7fa1541594a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:44.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.506+0000 7fa15bf6f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa15415e7a0 0x7fa1540c8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:44.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.506+0000 7fa15bf6f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa15406cfe0 con 0x7fa1540cf1f0 2024-09-16T01:34:44.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.506+0000 7fa15bf6f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa1540b7a00 con 0x7fa15415e7a0 2024-09-16T01:34:44.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.506+0000 7fa15bf6f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa1540d4420 con 0x7fa1540b6720 2024-09-16T01:34:44.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.506+0000 7fa15950a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1540cf1f0 0x7fa1541594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:44.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.506+0000 7fa15950a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1540cf1f0 0x7fa1541594a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54784/0 (socket says 172.21.15.88:54784) 2024-09-16T01:34:44.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.506+0000 7fa15950a640 1 -- 172.21.15.88:0/301620917 learned_addr learned my addr 172.21.15.88:0/301620917 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:44.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.506+0000 7fa159d0b640 1 --2- 172.21.15.88:0/301620917 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1540b6720 0x7fa1540ca790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:44.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.506+0000 7fa15a50c640 1 --2- 172.21.15.88:0/301620917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa15415e7a0 0x7fa1540c8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:44.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.507+0000 7fa15950a640 1 -- 172.21.15.88:0/301620917 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1540b6720 msgr2=0x7fa1540ca790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:44.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.507+0000 7fa15950a640 1 --2- 172.21.15.88:0/301620917 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1540b6720 0x7fa1540ca790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.507+0000 7fa15950a640 1 -- 172.21.15.88:0/301620917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa15415e7a0 msgr2=0x7fa1540c8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:44.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.507+0000 7fa15950a640 1 --2- 172.21.15.88:0/301620917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa15415e7a0 0x7fa1540c8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.507+0000 7fa15950a640 1 -- 172.21.15.88:0/301620917 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1541465f0 con 0x7fa1540cf1f0 2024-09-16T01:34:44.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.507+0000 7fa15950a640 1 --2- 172.21.15.88:0/301620917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1540cf1f0 0x7fa1541594a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa150009870 tx=0x7fa1500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:44.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.507+0000 7fa1467fc640 1 -- 172.21.15.88:0/301620917 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa150041070 con 0x7fa1540cf1f0 2024-09-16T01:34:44.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.507+0000 7fa1467fc640 1 -- 172.21.15.88:0/301620917 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa15003c890 con 0x7fa1540cf1f0 2024-09-16T01:34:44.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.507+0000 7fa1467fc640 1 -- 172.21.15.88:0/301620917 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa150037040 con 0x7fa1540cf1f0 2024-09-16T01:34:44.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.507+0000 7fa15bf6f640 1 -- 172.21.15.88:0/301620917 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1540cae20 con 0x7fa1540cf1f0 2024-09-16T01:34:44.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.507+0000 7fa15bf6f640 1 -- 172.21.15.88:0/301620917 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1541468b0 con 0x7fa1540cf1f0 2024-09-16T01:34:44.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.508+0000 7fa15bf6f640 1 -- 172.21.15.88:0/301620917 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa15415eb80 con 0x7fa1540cf1f0 2024-09-16T01:34:44.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.508+0000 7fa1467fc640 1 -- 172.21.15.88:0/301620917 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa150044070 con 0x7fa1540cf1f0 2024-09-16T01:34:44.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.509+0000 7fa1467fc640 1 --2- 172.21.15.88:0/301620917 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa13806c2b0 0x7fa13806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:44.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.509+0000 7fa1467fc640 1 -- 172.21.15.88:0/301620917 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa1500abed0 con 0x7fa1540cf1f0 2024-09-16T01:34:44.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.509+0000 7fa159d0b640 1 --2- 172.21.15.88:0/301620917 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa13806c2b0 0x7fa13806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:44.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.509+0000 7fa159d0b640 1 --2- 172.21.15.88:0/301620917 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa13806c2b0 0x7fa13806e770 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa1480079a0 tx=0x7fa148008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:44.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.513+0000 7fa1467fc640 1 -- 172.21.15.88:0/301620917 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa15007a1a0 con 0x7fa1540cf1f0 2024-09-16T01:34:44.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.679+0000 7fa15bf6f640 1 -- 172.21.15.88:0/301620917 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa1540d2a80 con 0x7fa1540cf1f0 2024-09-16T01:34:44.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.679+0000 7fa1467fc640 1 -- 172.21.15.88:0/301620917 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fa15007e4a0 con 0x7fa1540cf1f0 2024-09-16T01:34:44.681 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2024-09-16T01:34:44.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.681+0000 7fa15bf6f640 1 -- 172.21.15.88:0/301620917 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa13806c2b0 msgr2=0x7fa13806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:44.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.681+0000 7fa15bf6f640 1 --2- 172.21.15.88:0/301620917 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa13806c2b0 0x7fa13806e770 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa1480079a0 tx=0x7fa148008040 comp rx=0 tx=0).stop 2024-09-16T01:34:44.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.681+0000 7fa15bf6f640 1 -- 172.21.15.88:0/301620917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1540cf1f0 msgr2=0x7fa1541594a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:44.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.681+0000 7fa15bf6f640 1 --2- 172.21.15.88:0/301620917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1540cf1f0 0x7fa1541594a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa150009870 tx=0x7fa1500046c0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.681+0000 7fa15bf6f640 1 -- 172.21.15.88:0/301620917 shutdown_connections 2024-09-16T01:34:44.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.681+0000 7fa15bf6f640 1 --2- 172.21.15.88:0/301620917 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa13806c2b0 0x7fa13806e770 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.681+0000 7fa15bf6f640 1 --2- 172.21.15.88:0/301620917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa15415e7a0 0x7fa1540c8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.681+0000 7fa15bf6f640 1 --2- 172.21.15.88:0/301620917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1540cf1f0 0x7fa1541594a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.681+0000 7fa15bf6f640 1 --2- 172.21.15.88:0/301620917 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1540b6720 0x7fa1540ca790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.681+0000 7fa15bf6f640 1 -- 172.21.15.88:0/301620917 >> 172.21.15.88:0/301620917 conn(0x7fa1540d11e0 msgr2=0x7fa15415b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:44.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.682+0000 7fa15bf6f640 1 -- 172.21.15.88:0/301620917 shutdown_connections 2024-09-16T01:34:44.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.682+0000 7fa15bf6f640 1 -- 172.21.15.88:0/301620917 wait complete. 2024-09-16T01:34:44.694 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-16T01:34:44.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.793+0000 7f589f04c640 1 Processor -- start 2024-09-16T01:34:44.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.794+0000 7f589f04c640 1 -- start start 2024-09-16T01:34:44.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.794+0000 7f589f04c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5898155880 0x7f589815f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:44.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.794+0000 7f589f04c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58980b4d60 0x7f58980b51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:44.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.794+0000 7f589f04c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f589806f630 0x7f589806fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:44.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.794+0000 7f589f04c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f5898155c60 con 0x7f5898155880 2024-09-16T01:34:44.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.794+0000 7f589f04c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f589815fcb0 con 0x7f589806f630 2024-09-16T01:34:44.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.794+0000 7f589f04c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f589815bb60 con 0x7f58980b4d60 2024-09-16T01:34:44.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.794+0000 7f589cde8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5898155880 0x7f589815f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:44.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.795+0000 7f589cde8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5898155880 0x7f589815f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54794/0 (socket says 172.21.15.88:54794) 2024-09-16T01:34:44.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.795+0000 7f589cde8640 1 -- 172.21.15.88:0/757352720 learned_addr learned my addr 172.21.15.88:0/757352720 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:44.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.795+0000 7f5897fff640 1 --2- 172.21.15.88:0/757352720 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58980b4d60 0x7f58980b51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:44.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.795+0000 7f589d5e9640 1 --2- 172.21.15.88:0/757352720 >> v2:172.21.15.154:3300/0 conn(0x7f589806f630 0x7f589806fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:44.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.795+0000 7f589cde8640 1 -- 172.21.15.88:0/757352720 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58980b4d60 msgr2=0x7f58980b51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:44.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.795+0000 7f589cde8640 1 --2- 172.21.15.88:0/757352720 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58980b4d60 0x7f58980b51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.795+0000 7f589cde8640 1 -- 172.21.15.88:0/757352720 >> v2:172.21.15.154:3300/0 conn(0x7f589806f630 msgr2=0x7f589806fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:44.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.795+0000 7f589cde8640 1 --2- 172.21.15.88:0/757352720 >> v2:172.21.15.154:3300/0 conn(0x7f589806f630 0x7f589806fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.795+0000 7f589cde8640 1 -- 172.21.15.88:0/757352720 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f589815be40 con 0x7f5898155880 2024-09-16T01:34:44.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.795+0000 7f589cde8640 1 --2- 172.21.15.88:0/757352720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5898155880 0x7f589815f770 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f58980cd520 tx=0x7f588802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81db9eae6bb1390a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:44.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.796+0000 7f58977fe640 1 -- 172.21.15.88:0/757352720 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5888037070 con 0x7f5898155880 2024-09-16T01:34:44.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.796+0000 7f58977fe640 1 -- 172.21.15.88:0/757352720 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5888032440 con 0x7f5898155880 2024-09-16T01:34:44.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.796+0000 7f58977fe640 1 -- 172.21.15.88:0/757352720 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5888032810 con 0x7f5898155880 2024-09-16T01:34:44.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.796+0000 7f589f04c640 1 -- 172.21.15.88:0/757352720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5898155880 msgr2=0x7f589815f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:44.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.796+0000 7f589f04c640 1 --2- 172.21.15.88:0/757352720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5898155880 0x7f589815f770 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f58980cd520 tx=0x7f588802a1e0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.796+0000 7f589f04c640 1 -- 172.21.15.88:0/757352720 shutdown_connections 2024-09-16T01:34:44.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.796+0000 7f589f04c640 1 --2- 172.21.15.88:0/757352720 >> v2:172.21.15.154:3300/0 conn(0x7f589806f630 0x7f589806fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.796+0000 7f589f04c640 1 --2- 172.21.15.88:0/757352720 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58980b4d60 0x7f58980b51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.796+0000 7f589f04c640 1 --2- 172.21.15.88:0/757352720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5898155880 0x7f589815f770 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.796+0000 7f589f04c640 1 -- 172.21.15.88:0/757352720 >> 172.21.15.88:0/757352720 conn(0x7f58980d1390 msgr2=0x7f58980d17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:44.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.796+0000 7f589f04c640 1 -- 172.21.15.88:0/757352720 shutdown_connections 2024-09-16T01:34:44.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.796+0000 7f589f04c640 1 -- 172.21.15.88:0/757352720 wait complete. 2024-09-16T01:34:44.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.797+0000 7f589f04c640 1 Processor -- start 2024-09-16T01:34:44.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.797+0000 7f589f04c640 1 -- start start 2024-09-16T01:34:44.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.798+0000 7f589f04c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f589806f630 0x7f589814ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:44.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.798+0000 7f589f04c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58980b4d60 0x7f5898156550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:44.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.798+0000 7f589f04c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5898155880 0x7f589814b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:44.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.798+0000 7f589f04c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f589815c6e0 con 0x7f589806f630 2024-09-16T01:34:44.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.798+0000 7f589f04c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f589815c400 con 0x7f5898155880 2024-09-16T01:34:44.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.798+0000 7f589f04c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f589815b770 con 0x7f58980b4d60 2024-09-16T01:34:44.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.798+0000 7f589cde8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f589806f630 0x7f589814ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:44.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.798+0000 7f589cde8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f589806f630 0x7f589814ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54800/0 (socket says 172.21.15.88:54800) 2024-09-16T01:34:44.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.798+0000 7f589cde8640 1 -- 172.21.15.88:0/2261986941 learned_addr learned my addr 172.21.15.88:0/2261986941 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:44.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.798+0000 7f5897fff640 1 --2- 172.21.15.88:0/2261986941 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58980b4d60 0x7f5898156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:44.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.799+0000 7f589d5e9640 1 --2- 172.21.15.88:0/2261986941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5898155880 0x7f589814b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:44.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.799+0000 7f589cde8640 1 -- 172.21.15.88:0/2261986941 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58980b4d60 msgr2=0x7f5898156550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:44.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.799+0000 7f589cde8640 1 --2- 172.21.15.88:0/2261986941 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58980b4d60 0x7f5898156550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.799+0000 7f589cde8640 1 -- 172.21.15.88:0/2261986941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5898155880 msgr2=0x7f589814b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:44.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.799+0000 7f589cde8640 1 --2- 172.21.15.88:0/2261986941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5898155880 0x7f589814b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.799+0000 7f589cde8640 1 -- 172.21.15.88:0/2261986941 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5898156f10 con 0x7f589806f630 2024-09-16T01:34:44.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.799+0000 7f5897fff640 1 --2- 172.21.15.88:0/2261986941 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58980b4d60 0x7f5898156550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:44.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.799+0000 7f589d5e9640 1 --2- 172.21.15.88:0/2261986941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5898155880 0x7f589814b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:44.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.799+0000 7f589cde8640 1 --2- 172.21.15.88:0/2261986941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f589806f630 0x7f589814ad20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f588802a220 tx=0x7f58880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:44.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.799+0000 7f58957fa640 1 -- 172.21.15.88:0/2261986941 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5888041020 con 0x7f589806f630 2024-09-16T01:34:44.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.799+0000 7f58957fa640 1 -- 172.21.15.88:0/2261986941 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58880306c0 con 0x7f589806f630 2024-09-16T01:34:44.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.799+0000 7f589f04c640 1 -- 172.21.15.88:0/2261986941 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58980cfa70 con 0x7f589806f630 2024-09-16T01:34:44.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.799+0000 7f58957fa640 1 -- 172.21.15.88:0/2261986941 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5888037040 con 0x7f589806f630 2024-09-16T01:34:44.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.799+0000 7f589f04c640 1 -- 172.21.15.88:0/2261986941 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58980cf310 con 0x7f589806f630 2024-09-16T01:34:44.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.800+0000 7f589f04c640 1 -- 172.21.15.88:0/2261986941 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58980b0d80 con 0x7f589806f630 2024-09-16T01:34:44.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.801+0000 7f58957fa640 1 -- 172.21.15.88:0/2261986941 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5888044020 con 0x7f589806f630 2024-09-16T01:34:44.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.801+0000 7f58957fa640 1 --2- 172.21.15.88:0/2261986941 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f588406c2b0 0x7f588406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:44.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.801+0000 7f58957fa640 1 -- 172.21.15.88:0/2261986941 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f58880abc00 con 0x7f589806f630 2024-09-16T01:34:44.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.801+0000 7f5897fff640 1 --2- 172.21.15.88:0/2261986941 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f588406c2b0 0x7f588406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:44.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.801+0000 7f5897fff640 1 --2- 172.21.15.88:0/2261986941 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f588406c2b0 0x7f588406e770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5898154800 tx=0x7f588c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:44.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.805+0000 7f58957fa640 1 -- 172.21.15.88:0/2261986941 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f58880b2080 con 0x7f589806f630 2024-09-16T01:34:44.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.969+0000 7f589f04c640 1 -- 172.21.15.88:0/2261986941 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f58980cd8a0 con 0x7f589806f630 2024-09-16T01:34:44.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.969+0000 7f58957fa640 1 -- 172.21.15.88:0/2261986941 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f588804b030 con 0x7f589806f630 2024-09-16T01:34:44.971 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2024-09-16T01:34:44.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.971+0000 7f589f04c640 1 -- 172.21.15.88:0/2261986941 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f588406c2b0 msgr2=0x7f588406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:44.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.971+0000 7f589f04c640 1 --2- 172.21.15.88:0/2261986941 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f588406c2b0 0x7f588406e770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5898154800 tx=0x7f588c008040 comp rx=0 tx=0).stop 2024-09-16T01:34:44.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.971+0000 7f589f04c640 1 -- 172.21.15.88:0/2261986941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f589806f630 msgr2=0x7f589814ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:44.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.971+0000 7f589f04c640 1 --2- 172.21.15.88:0/2261986941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f589806f630 0x7f589814ad20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f588802a220 tx=0x7f58880046c0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.971+0000 7f589f04c640 1 -- 172.21.15.88:0/2261986941 shutdown_connections 2024-09-16T01:34:44.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.971+0000 7f589f04c640 1 --2- 172.21.15.88:0/2261986941 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f588406c2b0 0x7f588406e770 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.971+0000 7f589f04c640 1 --2- 172.21.15.88:0/2261986941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5898155880 0x7f589814b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.971+0000 7f589f04c640 1 --2- 172.21.15.88:0/2261986941 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58980b4d60 0x7f5898156550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.971+0000 7f589f04c640 1 --2- 172.21.15.88:0/2261986941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f589806f630 0x7f589814ad20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:44.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.971+0000 7f589f04c640 1 -- 172.21.15.88:0/2261986941 >> 172.21.15.88:0/2261986941 conn(0x7f58980d1390 msgr2=0x7f5898154970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:44.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.971+0000 7f589f04c640 1 -- 172.21.15.88:0/2261986941 shutdown_connections 2024-09-16T01:34:44.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:44.971+0000 7f589f04c640 1 -- 172.21.15.88:0/2261986941 wait complete. 2024-09-16T01:34:44.983 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-16T01:34:45.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.082+0000 7f16bdf66640 1 Processor -- start 2024-09-16T01:34:45.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.082+0000 7f16bdf66640 1 -- start start 2024-09-16T01:34:45.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.083+0000 7f16bdf66640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f16b815e7a0 0x7f16b806caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:45.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.083+0000 7f16bdf66640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16b80cf1f0 0x7f16b80b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:45.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.083+0000 7f16bdf66640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16b80cf860 con 0x7f16b80b6720 2024-09-16T01:34:45.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.083+0000 7f16bdf66640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f16b80d2ca0 con 0x7f16b815e7a0 2024-09-16T01:34:45.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.083+0000 7f16bdf66640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f16b80cfb40 con 0x7f16b80cf1f0 2024-09-16T01:34:45.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.084+0000 7f16aeffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16b80cf1f0 0x7f16b80b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:45.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.084+0000 7f16b77fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f16b815e7a0 0x7f16b806caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:45.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.084+0000 7f16b77fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f16b815e7a0 0x7f16b806caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:54596/0 (socket says 172.21.15.88:54596) 2024-09-16T01:34:45.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.084+0000 7f16b77fe640 1 -- 172.21.15.88:0/3396779023 learned_addr learned my addr 172.21.15.88:0/3396779023 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:45.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.084+0000 7f16b77fe640 1 -- 172.21.15.88:0/3396779023 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16b80cf1f0 msgr2=0x7f16b80b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:45.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.084+0000 7f16b77fe640 1 --2- 172.21.15.88:0/3396779023 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16b80cf1f0 0x7f16b80b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:45.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.084+0000 7f16b77fe640 1 -- 172.21.15.88:0/3396779023 >> v1:172.21.15.154:6789/0 conn(0x7f16b80b6720 legacy=0x7f16b81623c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:45.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.084+0000 7f16b77fe640 1 -- 172.21.15.88:0/3396779023 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b80cfe20 con 0x7f16b815e7a0 2024-09-16T01:34:45.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.084+0000 7f16b6ffd640 1 -- 172.21.15.88:0/3396779023 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3865982996 0 0) 0x7f16b80cf860 con 0x7f16b80b6720 2024-09-16T01:34:45.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.085+0000 7f16b77fe640 1 --2- 172.21.15.88:0/3396779023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16b815e7a0 0x7f16b806caa0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f16a8009870 tx=0x7f16a802a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b039585e48d8a91 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:45.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.085+0000 7f16b6ffd640 1 -- 172.21.15.88:0/3396779023 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a8037070 con 0x7f16b815e7a0 2024-09-16T01:34:45.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.085+0000 7f16b6ffd640 1 -- 172.21.15.88:0/3396779023 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16a8032440 con 0x7f16b815e7a0 2024-09-16T01:34:45.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.085+0000 7f16b6ffd640 1 -- 172.21.15.88:0/3396779023 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a8032810 con 0x7f16b815e7a0 2024-09-16T01:34:45.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.085+0000 7f16bdf66640 1 -- 172.21.15.88:0/3396779023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16b815e7a0 msgr2=0x7f16b806caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:45.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.085+0000 7f16bdf66640 1 --2- 172.21.15.88:0/3396779023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16b815e7a0 0x7f16b806caa0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f16a8009870 tx=0x7f16a802a1c0 comp rx=0 tx=0).stop 2024-09-16T01:34:45.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.085+0000 7f16bdf66640 1 -- 172.21.15.88:0/3396779023 shutdown_connections 2024-09-16T01:34:45.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.085+0000 7f16bdf66640 1 --2- 172.21.15.88:0/3396779023 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16b80cf1f0 0x7f16b80b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:45.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.085+0000 7f16bdf66640 1 --2- 172.21.15.88:0/3396779023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16b815e7a0 0x7f16b806caa0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:45.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.085+0000 7f16bdf66640 1 -- 172.21.15.88:0/3396779023 >> 172.21.15.88:0/3396779023 conn(0x7f16b80d11e0 msgr2=0x7f16b80d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:45.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.085+0000 7f16bdf66640 1 -- 172.21.15.88:0/3396779023 shutdown_connections 2024-09-16T01:34:45.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.085+0000 7f16bdf66640 1 -- 172.21.15.88:0/3396779023 wait complete. 2024-09-16T01:34:45.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.087+0000 7f16bdf66640 1 Processor -- start 2024-09-16T01:34:45.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.087+0000 7f16bdf66640 1 -- start start 2024-09-16T01:34:45.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.087+0000 7f16bdf66640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16b80b6720 0x7f16b8153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:45.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.087+0000 7f16bdf66640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16b80cf1f0 0x7f16b81f8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:45.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.087+0000 7f16bdf66640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16b815e7a0 0x7f16b80ce3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:45.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.087+0000 7f16bdf66640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f16b806cfe0 con 0x7f16b80b6720 2024-09-16T01:34:45.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.088+0000 7f16bdf66640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f16b80b7a00 con 0x7f16b815e7a0 2024-09-16T01:34:45.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.088+0000 7f16bdf66640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f16b80d4420 con 0x7f16b80cf1f0 2024-09-16T01:34:45.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.088+0000 7f16b77fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16b80b6720 0x7f16b8153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:45.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.088+0000 7f16b77fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16b80b6720 0x7f16b8153120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54812/0 (socket says 172.21.15.88:54812) 2024-09-16T01:34:45.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.088+0000 7f16b77fe640 1 -- 172.21.15.88:0/1691329425 learned_addr learned my addr 172.21.15.88:0/1691329425 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:45.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.088+0000 7f16aeffd640 1 --2- 172.21.15.88:0/1691329425 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16b80cf1f0 0x7f16b81f8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:45.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.088+0000 7f16b7fff640 1 --2- 172.21.15.88:0/1691329425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16b815e7a0 0x7f16b80ce3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:45.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.088+0000 7f16b77fe640 1 -- 172.21.15.88:0/1691329425 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16b80cf1f0 msgr2=0x7f16b81f8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:45.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.088+0000 7f16b77fe640 1 --2- 172.21.15.88:0/1691329425 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16b80cf1f0 0x7f16b81f8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:45.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.088+0000 7f16b77fe640 1 -- 172.21.15.88:0/1691329425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16b815e7a0 msgr2=0x7f16b80ce3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:45.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.088+0000 7f16b77fe640 1 --2- 172.21.15.88:0/1691329425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16b815e7a0 0x7f16b80ce3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:45.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.088+0000 7f16b77fe640 1 -- 172.21.15.88:0/1691329425 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b81505d0 con 0x7f16b80b6720 2024-09-16T01:34:45.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.088+0000 7f16aeffd640 1 --2- 172.21.15.88:0/1691329425 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16b80cf1f0 0x7f16b81f8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:45.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.088+0000 7f16b7fff640 1 --2- 172.21.15.88:0/1691329425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16b815e7a0 0x7f16b80ce3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:45.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.088+0000 7f16b77fe640 1 --2- 172.21.15.88:0/1691329425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16b80b6720 0x7f16b8153120 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f16a8009870 tx=0x7f16a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:45.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.088+0000 7f16b4ff9640 1 -- 172.21.15.88:0/1691329425 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a8037040 con 0x7f16b80b6720 2024-09-16T01:34:45.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.088+0000 7f16b4ff9640 1 -- 172.21.15.88:0/1691329425 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16a8004230 con 0x7f16b80b6720 2024-09-16T01:34:45.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.088+0000 7f16bdf66640 1 -- 172.21.15.88:0/1691329425 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16b8153800 con 0x7f16b80b6720 2024-09-16T01:34:45.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.088+0000 7f16b4ff9640 1 -- 172.21.15.88:0/1691329425 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a803c670 con 0x7f16b80b6720 2024-09-16T01:34:45.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.088+0000 7f16bdf66640 1 -- 172.21.15.88:0/1691329425 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16b8150b70 con 0x7f16b80b6720 2024-09-16T01:34:45.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.089+0000 7f16bdf66640 1 -- 172.21.15.88:0/1691329425 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16b815eb80 con 0x7f16b80b6720 2024-09-16T01:34:45.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.090+0000 7f16b4ff9640 1 -- 172.21.15.88:0/1691329425 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f16a803a070 con 0x7f16b80b6720 2024-09-16T01:34:45.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.090+0000 7f16b4ff9640 1 --2- 172.21.15.88:0/1691329425 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f169806c2e0 0x7f169806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:45.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.090+0000 7f16b4ff9640 1 -- 172.21.15.88:0/1691329425 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f16a80abbf0 con 0x7f16b80b6720 2024-09-16T01:34:45.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.090+0000 7f16aeffd640 1 --2- 172.21.15.88:0/1691329425 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f169806c2e0 0x7f169806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:45.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.091+0000 7f16aeffd640 1 --2- 172.21.15.88:0/1691329425 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f169806c2e0 0x7f169806e7a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f16b81f7ea0 tx=0x7f16a0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:45.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.094+0000 7f16b4ff9640 1 -- 172.21.15.88:0/1691329425 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f16a8041020 con 0x7f16b80b6720 2024-09-16T01:34:45.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.262+0000 7f16bdf66640 1 -- 172.21.15.88:0/1691329425 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f16b80cefd0 con 0x7f16b80b6720 2024-09-16T01:34:45.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.263+0000 7f16b4ff9640 1 -- 172.21.15.88:0/1691329425 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f16a807a360 con 0x7f16b80b6720 2024-09-16T01:34:45.264 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 64 2024-09-16T01:34:45.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.265+0000 7f16bdf66640 1 -- 172.21.15.88:0/1691329425 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f169806c2e0 msgr2=0x7f169806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:45.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.265+0000 7f16bdf66640 1 --2- 172.21.15.88:0/1691329425 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f169806c2e0 0x7f169806e7a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f16b81f7ea0 tx=0x7f16a0008040 comp rx=0 tx=0).stop 2024-09-16T01:34:45.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.265+0000 7f16bdf66640 1 -- 172.21.15.88:0/1691329425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16b80b6720 msgr2=0x7f16b8153120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:45.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.265+0000 7f16bdf66640 1 --2- 172.21.15.88:0/1691329425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16b80b6720 0x7f16b8153120 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f16a8009870 tx=0x7f16a80046c0 comp rx=0 tx=0).stop 2024-09-16T01:34:45.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.265+0000 7f16bdf66640 1 -- 172.21.15.88:0/1691329425 shutdown_connections 2024-09-16T01:34:45.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.265+0000 7f16bdf66640 1 --2- 172.21.15.88:0/1691329425 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f169806c2e0 0x7f169806e7a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:45.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.265+0000 7f16bdf66640 1 --2- 172.21.15.88:0/1691329425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16b815e7a0 0x7f16b80ce3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:45.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.265+0000 7f16bdf66640 1 --2- 172.21.15.88:0/1691329425 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16b80cf1f0 0x7f16b81f8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:45.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.265+0000 7f16bdf66640 1 --2- 172.21.15.88:0/1691329425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16b80b6720 0x7f16b8153120 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:45.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.265+0000 7f16bdf66640 1 -- 172.21.15.88:0/1691329425 >> 172.21.15.88:0/1691329425 conn(0x7f16b80d11e0 msgr2=0x7f16b815cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:45.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.265+0000 7f16bdf66640 1 -- 172.21.15.88:0/1691329425 shutdown_connections 2024-09-16T01:34:45.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.265+0000 7f16bdf66640 1 -- 172.21.15.88:0/1691329425 wait complete. 2024-09-16T01:34:45.277 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_damage.TestDamage.test_damaged_dentry' 2024-09-16T01:34:45.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.374+0000 7fe839eac640 1 Processor -- start 2024-09-16T01:34:45.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.374+0000 7fe839eac640 1 -- start start 2024-09-16T01:34:45.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.375+0000 7fe839eac640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe834155800 0x7fe8340b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:45.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.375+0000 7fe839eac640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe8340cfc80 0x7fe8340b4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:45.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.375+0000 7fe839eac640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8340b50a0 con 0x7fe83415c330 2024-09-16T01:34:45.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.375+0000 7fe839eac640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe8340b4e60 con 0x7fe834155800 2024-09-16T01:34:45.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.375+0000 7fe839eac640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe83415c7a0 con 0x7fe8340cfc80 2024-09-16T01:34:45.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.375+0000 7fe8337fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe834155800 0x7fe8340b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:45.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.375+0000 7fe8337fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe834155800 0x7fe8340b6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54820/0 (socket says 172.21.15.88:54820) 2024-09-16T01:34:45.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.375+0000 7fe8337fe640 1 -- 172.21.15.88:0/1603962561 learned_addr learned my addr 172.21.15.88:0/1603962561 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:45.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.375+0000 7fe832ffd640 1 --1- 172.21.15.88:0/1603962561 >> v1:172.21.15.88:6789/0 conn(0x7fe83415c330 0x7fe8340cf570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54040/0 (socket says 172.21.15.88:54040) 2024-09-16T01:34:45.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.375+0000 7fe833fff640 1 --2- 172.21.15.88:0/1603962561 >> v2:172.21.15.154:3300/0 conn(0x7fe8340cfc80 0x7fe8340b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:45.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.376+0000 7fe8337fe640 1 -- 172.21.15.88:0/1603962561 >> v2:172.21.15.154:3300/0 conn(0x7fe8340cfc80 msgr2=0x7fe8340b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:45.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.376+0000 7fe8337fe640 1 --2- 172.21.15.88:0/1603962561 >> v2:172.21.15.154:3300/0 conn(0x7fe8340cfc80 0x7fe8340b4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:45.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.376+0000 7fe8337fe640 1 -- 172.21.15.88:0/1603962561 >> v1:172.21.15.88:6789/0 conn(0x7fe83415c330 legacy=0x7fe8340cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:45.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.376+0000 7fe8337fe640 1 -- 172.21.15.88:0/1603962561 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8340b5380 con 0x7fe834155800 2024-09-16T01:34:45.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.376+0000 7fe8337fe640 1 --2- 172.21.15.88:0/1603962561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe834155800 0x7fe8340b6140 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe83415b7b0 tx=0x7fe82402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3fe7d0a218ffee4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:45.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.376+0000 7fe8327fc640 1 -- 172.21.15.88:0/1603962561 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe824037070 con 0x7fe834155800 2024-09-16T01:34:45.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.376+0000 7fe8327fc640 1 -- 172.21.15.88:0/1603962561 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe824032440 con 0x7fe834155800 2024-09-16T01:34:45.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.376+0000 7fe8327fc640 1 -- 172.21.15.88:0/1603962561 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe824032810 con 0x7fe834155800 2024-09-16T01:34:45.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.377+0000 7fe839eac640 1 -- 172.21.15.88:0/1603962561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe834155800 msgr2=0x7fe8340b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:45.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.377+0000 7fe839eac640 1 --2- 172.21.15.88:0/1603962561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe834155800 0x7fe8340b6140 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe83415b7b0 tx=0x7fe82402a270 comp rx=0 tx=0).stop 2024-09-16T01:34:45.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.377+0000 7fe839eac640 1 -- 172.21.15.88:0/1603962561 shutdown_connections 2024-09-16T01:34:45.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.377+0000 7fe839eac640 1 --2- 172.21.15.88:0/1603962561 >> v2:172.21.15.154:3300/0 conn(0x7fe8340cfc80 0x7fe8340b4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:45.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.377+0000 7fe839eac640 1 --2- 172.21.15.88:0/1603962561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe834155800 0x7fe8340b6140 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:45.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.377+0000 7fe839eac640 1 -- 172.21.15.88:0/1603962561 >> 172.21.15.88:0/1603962561 conn(0x7fe83406bc10 msgr2=0x7fe83406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:45.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.377+0000 7fe839eac640 1 -- 172.21.15.88:0/1603962561 shutdown_connections 2024-09-16T01:34:45.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.377+0000 7fe839eac640 1 -- 172.21.15.88:0/1603962561 wait complete. 2024-09-16T01:34:45.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.378+0000 7fe839eac640 1 Processor -- start 2024-09-16T01:34:45.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.378+0000 7fe839eac640 1 -- start start 2024-09-16T01:34:45.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.379+0000 7fe839eac640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8340cfc80 0x7fe834145fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:45.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.379+0000 7fe839eac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe834155800 0x7fe83415d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:45.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.379+0000 7fe839eac640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe83415c330 0x7fe83414e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:45.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.379+0000 7fe839eac640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe8340b5c20 con 0x7fe83415c330 2024-09-16T01:34:45.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.379+0000 7fe839eac640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe8340b5940 con 0x7fe834155800 2024-09-16T01:34:45.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.379+0000 7fe839eac640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe834155be0 con 0x7fe8340cfc80 2024-09-16T01:34:45.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.379+0000 7fe833fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe83415c330 0x7fe83414e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:45.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.379+0000 7fe833fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe83415c330 0x7fe83414e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54828/0 (socket says 172.21.15.88:54828) 2024-09-16T01:34:45.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.379+0000 7fe833fff640 1 -- 172.21.15.88:0/2966785379 learned_addr learned my addr 172.21.15.88:0/2966785379 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:45.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.379+0000 7fe832ffd640 1 --2- 172.21.15.88:0/2966785379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe834155800 0x7fe83415d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:45.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.379+0000 7fe8337fe640 1 --2- 172.21.15.88:0/2966785379 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8340cfc80 0x7fe834145fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:45.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.379+0000 7fe833fff640 1 -- 172.21.15.88:0/2966785379 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8340cfc80 msgr2=0x7fe834145fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:45.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.379+0000 7fe833fff640 1 --2- 172.21.15.88:0/2966785379 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8340cfc80 0x7fe834145fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:45.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.380+0000 7fe833fff640 1 -- 172.21.15.88:0/2966785379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe834155800 msgr2=0x7fe83415d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:45.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.380+0000 7fe833fff640 1 --2- 172.21.15.88:0/2966785379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe834155800 0x7fe83415d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:45.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.380+0000 7fe833fff640 1 -- 172.21.15.88:0/2966785379 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe83415a150 con 0x7fe83415c330 2024-09-16T01:34:45.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.380+0000 7fe832ffd640 1 --2- 172.21.15.88:0/2966785379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe834155800 0x7fe83415d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:45.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.380+0000 7fe833fff640 1 --2- 172.21.15.88:0/2966785379 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe83415c330 0x7fe83414e440 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe82c00e860 tx=0x7fe82c00ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:45.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.380+0000 7fe81bfff640 1 -- 172.21.15.88:0/2966785379 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82c017070 con 0x7fe83415c330 2024-09-16T01:34:45.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.380+0000 7fe81bfff640 1 -- 172.21.15.88:0/2966785379 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe82c00c6d0 con 0x7fe83415c330 2024-09-16T01:34:45.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.380+0000 7fe839eac640 1 -- 172.21.15.88:0/2966785379 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe83415a3f0 con 0x7fe83415c330 2024-09-16T01:34:45.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.380+0000 7fe81bfff640 1 -- 172.21.15.88:0/2966785379 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82c00cac0 con 0x7fe83415c330 2024-09-16T01:34:45.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.380+0000 7fe839eac640 1 -- 172.21.15.88:0/2966785379 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe83415a9b0 con 0x7fe83415c330 2024-09-16T01:34:45.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.381+0000 7fe839eac640 1 -- 172.21.15.88:0/2966785379 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8340b46f0 con 0x7fe83415c330 2024-09-16T01:34:45.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.381+0000 7fe81bfff640 1 -- 172.21.15.88:0/2966785379 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe82c01a070 con 0x7fe83415c330 2024-09-16T01:34:45.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.382+0000 7fe81bfff640 1 --2- 172.21.15.88:0/2966785379 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe82006c2b0 0x7fe82006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:45.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.382+0000 7fe81bfff640 1 -- 172.21.15.88:0/2966785379 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe82c08b3c0 con 0x7fe83415c330 2024-09-16T01:34:45.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.382+0000 7fe8337fe640 1 --2- 172.21.15.88:0/2966785379 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe82006c2b0 0x7fe82006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:45.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.382+0000 7fe8337fe640 1 --2- 172.21.15.88:0/2966785379 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe82006c2b0 0x7fe82006e770 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe824034000 tx=0x7fe82402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:45.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.386+0000 7fe81bfff640 1 -- 172.21.15.88:0/2966785379 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe8340b46f0 con 0x7fe83415c330 2024-09-16T01:34:45.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:45.619+0000 7fe839eac640 1 -- 172.21.15.88:0/2966785379 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_damaged_dentry"]} v 0) -- 0x7fe8340b63c0 con 0x7fe83415c330 2024-09-16T01:34:45.795 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:34:45.798 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:34:46.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.258+0000 7fe81bfff640 1 -- 172.21.15.88:0/2966785379 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_damaged_dentry"]}]=0 v46) ==== 137+0+0 (secure 0 0 0) 0x7fe82c059bc0 con 0x7fe83415c330 2024-09-16T01:34:46.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.260+0000 7fe839eac640 1 -- 172.21.15.88:0/2966785379 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe82006c2b0 msgr2=0x7fe82006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:46.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.260+0000 7fe839eac640 1 --2- 172.21.15.88:0/2966785379 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe82006c2b0 0x7fe82006e770 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe824034000 tx=0x7fe82402e000 comp rx=0 tx=0).stop 2024-09-16T01:34:46.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.260+0000 7fe839eac640 1 -- 172.21.15.88:0/2966785379 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe83415c330 msgr2=0x7fe83414e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:46.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.260+0000 7fe839eac640 1 --2- 172.21.15.88:0/2966785379 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe83415c330 0x7fe83414e440 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe82c00e860 tx=0x7fe82c00ed20 comp rx=0 tx=0).stop 2024-09-16T01:34:46.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.260+0000 7fe839eac640 1 -- 172.21.15.88:0/2966785379 shutdown_connections 2024-09-16T01:34:46.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.260+0000 7fe839eac640 1 --2- 172.21.15.88:0/2966785379 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe82006c2b0 0x7fe82006e770 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:46.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.260+0000 7fe839eac640 1 --2- 172.21.15.88:0/2966785379 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe83415c330 0x7fe83414e440 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:46.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.260+0000 7fe839eac640 1 --2- 172.21.15.88:0/2966785379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe834155800 0x7fe83415d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:46.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.260+0000 7fe839eac640 1 --2- 172.21.15.88:0/2966785379 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8340cfc80 0x7fe834145fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:46.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.260+0000 7fe839eac640 1 -- 172.21.15.88:0/2966785379 >> 172.21.15.88:0/2966785379 conn(0x7fe83406bc10 msgr2=0x7fe8340ce520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:46.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.260+0000 7fe839eac640 1 -- 172.21.15.88:0/2966785379 shutdown_connections 2024-09-16T01:34:46.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.260+0000 7fe839eac640 1 -- 172.21.15.88:0/2966785379 wait complete. 2024-09-16T01:34:46.273 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T01:34:46.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.373+0000 7fd5be7df640 1 Processor -- start 2024-09-16T01:34:46.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.374+0000 7fd5be7df640 1 -- start start 2024-09-16T01:34:46.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.374+0000 7fd5be7df640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd5b815c2a0 0x7fd5b80735d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:46.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.374+0000 7fd5be7df640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd5b806fe20 0x7fd5b806bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:46.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.374+0000 7fd5be7df640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5b8074120 con 0x7fd5b8073b10 2024-09-16T01:34:46.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.374+0000 7fd5be7df640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd5b815c680 con 0x7fd5b806fe20 2024-09-16T01:34:46.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.374+0000 7fd5be7df640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd5b815ef00 con 0x7fd5b815c2a0 2024-09-16T01:34:46.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.375+0000 7fd5bcd7c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd5b806fe20 0x7fd5b806bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:46.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.375+0000 7fd5bcd7c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd5b806fe20 0x7fd5b806bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54842/0 (socket says 172.21.15.88:54842) 2024-09-16T01:34:46.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.375+0000 7fd5bcd7c640 1 -- 172.21.15.88:0/3351590600 learned_addr learned my addr 172.21.15.88:0/3351590600 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:46.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.375+0000 7fd5bcd7c640 1 -- 172.21.15.88:0/3351590600 >> v2:172.21.15.154:3300/0 conn(0x7fd5b815c2a0 msgr2=0x7fd5b80735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:46.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.375+0000 7fd5bcd7c640 1 --2- 172.21.15.88:0/3351590600 >> v2:172.21.15.154:3300/0 conn(0x7fd5b815c2a0 0x7fd5b80735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:46.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.375+0000 7fd5bcd7c640 1 -- 172.21.15.88:0/3351590600 >> v1:172.21.15.154:6789/0 conn(0x7fd5b8073b10 legacy=0x7fd5b806f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:46.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.375+0000 7fd5bcd7c640 1 -- 172.21.15.88:0/3351590600 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b815f1e0 con 0x7fd5b806fe20 2024-09-16T01:34:46.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.376+0000 7fd5bcd7c640 1 --2- 172.21.15.88:0/3351590600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5b806fe20 0x7fd5b806bc10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd5b80b4030 tx=0x7fd5b002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40b61f4be6a9f74c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:46.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.376+0000 7fd5b6ffd640 1 -- 172.21.15.88:0/3351590600 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b0036070 con 0x7fd5b806fe20 2024-09-16T01:34:46.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.376+0000 7fd5b6ffd640 1 -- 172.21.15.88:0/3351590600 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5b0031600 con 0x7fd5b806fe20 2024-09-16T01:34:46.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.376+0000 7fd5b6ffd640 1 -- 172.21.15.88:0/3351590600 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b00319d0 con 0x7fd5b806fe20 2024-09-16T01:34:46.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.376+0000 7fd5be7df640 1 -- 172.21.15.88:0/3351590600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5b806fe20 msgr2=0x7fd5b806bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:46.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.376+0000 7fd5be7df640 1 --2- 172.21.15.88:0/3351590600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5b806fe20 0x7fd5b806bc10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd5b80b4030 tx=0x7fd5b002a6f0 comp rx=0 tx=0).stop 2024-09-16T01:34:46.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.376+0000 7fd5be7df640 1 -- 172.21.15.88:0/3351590600 shutdown_connections 2024-09-16T01:34:46.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.376+0000 7fd5be7df640 1 --2- 172.21.15.88:0/3351590600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5b806fe20 0x7fd5b806bc10 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:46.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.376+0000 7fd5be7df640 1 --2- 172.21.15.88:0/3351590600 >> v2:172.21.15.154:3300/0 conn(0x7fd5b815c2a0 0x7fd5b80735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:46.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.376+0000 7fd5be7df640 1 -- 172.21.15.88:0/3351590600 >> 172.21.15.88:0/3351590600 conn(0x7fd5b80b3070 msgr2=0x7fd5b80b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:46.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.376+0000 7fd5be7df640 1 -- 172.21.15.88:0/3351590600 shutdown_connections 2024-09-16T01:34:46.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.376+0000 7fd5be7df640 1 -- 172.21.15.88:0/3351590600 wait complete. 2024-09-16T01:34:46.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.377+0000 7fd5be7df640 1 Processor -- start 2024-09-16T01:34:46.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.378+0000 7fd5be7df640 1 -- start start 2024-09-16T01:34:46.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.378+0000 7fd5be7df640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5b806fe20 0x7fd5b8158850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:46.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.378+0000 7fd5be7df640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5b8073b10 0x7fd5b8146c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:46.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.378+0000 7fd5be7df640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5b815c2a0 0x7fd5b815d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:46.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.378+0000 7fd5be7df640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b80b5f20 con 0x7fd5b8073b10 2024-09-16T01:34:46.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.378+0000 7fd5be7df640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b815ebf0 con 0x7fd5b806fe20 2024-09-16T01:34:46.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.378+0000 7fd5be7df640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd5b80b60a0 con 0x7fd5b815c2a0 2024-09-16T01:34:46.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.378+0000 7fd5b77fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5b8073b10 0x7fd5b8146c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:46.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.378+0000 7fd5b77fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5b8073b10 0x7fd5b8146c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54858/0 (socket says 172.21.15.88:54858) 2024-09-16T01:34:46.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.378+0000 7fd5b77fe640 1 -- 172.21.15.88:0/287767440 learned_addr learned my addr 172.21.15.88:0/287767440 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:46.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.379+0000 7fd5b7fff640 1 --2- 172.21.15.88:0/287767440 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5b806fe20 0x7fd5b8158850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:46.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.379+0000 7fd5bcd7c640 1 --2- 172.21.15.88:0/287767440 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5b815c2a0 0x7fd5b815d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:46.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.379+0000 7fd5b77fe640 1 -- 172.21.15.88:0/287767440 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5b815c2a0 msgr2=0x7fd5b815d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:46.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.379+0000 7fd5b77fe640 1 --2- 172.21.15.88:0/287767440 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5b815c2a0 0x7fd5b815d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:46.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.379+0000 7fd5b77fe640 1 -- 172.21.15.88:0/287767440 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5b806fe20 msgr2=0x7fd5b8158850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:46.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.379+0000 7fd5b77fe640 1 --2- 172.21.15.88:0/287767440 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5b806fe20 0x7fd5b8158850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:46.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.379+0000 7fd5b77fe640 1 -- 172.21.15.88:0/287767440 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b8158d90 con 0x7fd5b8073b10 2024-09-16T01:34:46.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.379+0000 7fd5bcd7c640 1 --2- 172.21.15.88:0/287767440 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5b815c2a0 0x7fd5b815d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:46.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.379+0000 7fd5b77fe640 1 --2- 172.21.15.88:0/287767440 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5b8073b10 0x7fd5b8146c90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac00c410 tx=0x7fd5ac00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:46.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.379+0000 7fd5b4ff9640 1 -- 172.21.15.88:0/287767440 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac017070 con 0x7fd5b8073b10 2024-09-16T01:34:46.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.379+0000 7fd5b4ff9640 1 -- 172.21.15.88:0/287767440 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5ac0035f0 con 0x7fd5b8073b10 2024-09-16T01:34:46.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.379+0000 7fd5be7df640 1 -- 172.21.15.88:0/287767440 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5b81579a0 con 0x7fd5b8073b10 2024-09-16T01:34:46.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.379+0000 7fd5b4ff9640 1 -- 172.21.15.88:0/287767440 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac003a20 con 0x7fd5b8073b10 2024-09-16T01:34:46.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.379+0000 7fd5be7df640 1 -- 172.21.15.88:0/287767440 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5b815fd20 con 0x7fd5b8073b10 2024-09-16T01:34:46.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.380+0000 7fd5be7df640 1 -- 172.21.15.88:0/287767440 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5b815f9e0 con 0x7fd5b8073b10 2024-09-16T01:34:46.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.381+0000 7fd5b4ff9640 1 -- 172.21.15.88:0/287767440 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd5ac01a070 con 0x7fd5b8073b10 2024-09-16T01:34:46.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.381+0000 7fd5b4ff9640 1 --2- 172.21.15.88:0/287767440 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd5a406c2b0 0x7fd5a406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:46.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.381+0000 7fd5b7fff640 1 --2- 172.21.15.88:0/287767440 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd5a406c2b0 0x7fd5a406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:46.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.381+0000 7fd5b4ff9640 1 -- 172.21.15.88:0/287767440 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd5ac08a7a0 con 0x7fd5b8073b10 2024-09-16T01:34:46.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.382+0000 7fd5b7fff640 1 --2- 172.21.15.88:0/287767440 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd5a406c2b0 0x7fd5a406e770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd5a80098e0 tx=0x7fd5a8009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:46.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.386+0000 7fd5b4ff9640 1 -- 172.21.15.88:0/287767440 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd5ac0010c0 con 0x7fd5b8073b10 2024-09-16T01:34:46.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.552+0000 7fd5be7df640 1 -- 172.21.15.88:0/287767440 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fd5b815fef0 con 0x7fd5b8073b10 2024-09-16T01:34:46.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.555+0000 7fd5b4ff9640 1 -- 172.21.15.88:0/287767440 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7fd5ac058950 con 0x7fd5b8073b10 2024-09-16T01:34:46.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.557+0000 7fd5be7df640 1 -- 172.21.15.88:0/287767440 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd5a406c2b0 msgr2=0x7fd5a406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:46.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.557+0000 7fd5be7df640 1 --2- 172.21.15.88:0/287767440 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd5a406c2b0 0x7fd5a406e770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd5a80098e0 tx=0x7fd5a8009060 comp rx=0 tx=0).stop 2024-09-16T01:34:46.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.557+0000 7fd5be7df640 1 -- 172.21.15.88:0/287767440 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5b8073b10 msgr2=0x7fd5b8146c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:46.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.557+0000 7fd5be7df640 1 --2- 172.21.15.88:0/287767440 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5b8073b10 0x7fd5b8146c90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac00c410 tx=0x7fd5ac00c8d0 comp rx=0 tx=0).stop 2024-09-16T01:34:46.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.557+0000 7fd5be7df640 1 -- 172.21.15.88:0/287767440 shutdown_connections 2024-09-16T01:34:46.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.557+0000 7fd5be7df640 1 --2- 172.21.15.88:0/287767440 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd5a406c2b0 0x7fd5a406e770 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:46.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.557+0000 7fd5be7df640 1 --2- 172.21.15.88:0/287767440 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5b815c2a0 0x7fd5b815d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:46.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.557+0000 7fd5be7df640 1 --2- 172.21.15.88:0/287767440 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5b8073b10 0x7fd5b8146c90 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:46.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.557+0000 7fd5be7df640 1 --2- 172.21.15.88:0/287767440 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5b806fe20 0x7fd5b8158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:46.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.557+0000 7fd5be7df640 1 -- 172.21.15.88:0/287767440 >> 172.21.15.88:0/287767440 conn(0x7fd5b80b3070 msgr2=0x7fd5b80b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:46.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.557+0000 7fd5be7df640 1 -- 172.21.15.88:0/287767440 shutdown_connections 2024-09-16T01:34:46.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.557+0000 7fd5be7df640 1 -- 172.21.15.88:0/287767440 wait complete. 2024-09-16T01:34:46.570 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T01:34:46.570 DEBUG:teuthology.orchestra.run.smithi088:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T01:34:46.619 DEBUG:teuthology.orchestra.run.smithi154:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T01:34:46.670 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T01:34:46.671 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:34:46.671 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:34:46.688 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:34:46.688 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:34:46.745 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:34:46.745 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:34:46.802 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:34:46.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.938+0000 7f6e5fef8640 1 Processor -- start 2024-09-16T01:34:46.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.939+0000 7f6e5fef8640 1 -- start start 2024-09-16T01:34:46.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.939+0000 7f6e5fef8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6e580b68f0 0x7f6e581696f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:46.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.939+0000 7f6e5fef8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e580cf860 con 0x7f6e580cf1f0 2024-09-16T01:34:46.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.939+0000 7f6e5fef8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e580cfb40 con 0x7f6e5815e7a0 2024-09-16T01:34:46.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.939+0000 7f6e5fef8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6e580d2ca0 con 0x7f6e580b68f0 2024-09-16T01:34:46.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.939+0000 7f6e5e495640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6e580b68f0 0x7f6e581696f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:46.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.939+0000 7f6e5e495640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6e580b68f0 0x7f6e581696f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54870/0 (socket says 172.21.15.88:54870) 2024-09-16T01:34:46.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.939+0000 7f6e5d493640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6e580cf1f0 0x7f6e580b61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54054/0 (socket says 172.21.15.88:54054) 2024-09-16T01:34:46.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.939+0000 7f6e5e495640 1 -- 172.21.15.88:0/2315339945 learned_addr learned my addr 172.21.15.88:0/2315339945 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:46.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.940+0000 7f6e5e495640 1 -- 172.21.15.88:0/2315339945 >> v1:172.21.15.154:6789/0 conn(0x7f6e5815e7a0 legacy=0x7f6e5806caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:46.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.940+0000 7f6e5e495640 1 -- 172.21.15.88:0/2315339945 >> v1:172.21.15.88:6789/0 conn(0x7f6e580cf1f0 legacy=0x7f6e580b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:46.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.940+0000 7f6e5e495640 1 -- 172.21.15.88:0/2315339945 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e580cfe20 con 0x7f6e580b68f0 2024-09-16T01:34:46.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.940+0000 7f6e5e495640 1 --2- 172.21.15.88:0/2315339945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e580b68f0 0x7f6e581696f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6e50009e50 tx=0x7f6e5002a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fabcdf6020eac6ba server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:46.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.941+0000 7f6e5cc92640 1 -- 172.21.15.88:0/2315339945 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e50036070 con 0x7f6e580b68f0 2024-09-16T01:34:46.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.941+0000 7f6e5cc92640 1 -- 172.21.15.88:0/2315339945 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e50031600 con 0x7f6e580b68f0 2024-09-16T01:34:46.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.941+0000 7f6e5cc92640 1 -- 172.21.15.88:0/2315339945 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e500319d0 con 0x7f6e580b68f0 2024-09-16T01:34:46.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.941+0000 7f6e5fef8640 1 -- 172.21.15.88:0/2315339945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e580b68f0 msgr2=0x7f6e581696f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:46.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.941+0000 7f6e5fef8640 1 --2- 172.21.15.88:0/2315339945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e580b68f0 0x7f6e581696f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6e50009e50 tx=0x7f6e5002a6d0 comp rx=0 tx=0).stop 2024-09-16T01:34:46.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.941+0000 7f6e5fef8640 1 -- 172.21.15.88:0/2315339945 shutdown_connections 2024-09-16T01:34:46.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.941+0000 7f6e5fef8640 1 --2- 172.21.15.88:0/2315339945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e580b68f0 0x7f6e581696f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:46.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.941+0000 7f6e5fef8640 1 -- 172.21.15.88:0/2315339945 >> 172.21.15.88:0/2315339945 conn(0x7f6e580d11e0 msgr2=0x7f6e580d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:46.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.941+0000 7f6e5fef8640 1 -- 172.21.15.88:0/2315339945 shutdown_connections 2024-09-16T01:34:46.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.941+0000 7f6e5fef8640 1 -- 172.21.15.88:0/2315339945 wait complete. 2024-09-16T01:34:46.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.942+0000 7f6e5fef8640 1 Processor -- start 2024-09-16T01:34:46.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.942+0000 7f6e5fef8640 1 -- start start 2024-09-16T01:34:46.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.943+0000 7f6e5fef8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e580b68f0 0x7f6e5815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:46.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.943+0000 7f6e5fef8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e580cf1f0 0x7f6e5814fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:46.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.943+0000 7f6e5fef8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e5815e7a0 0x7f6e58161220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:46.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.943+0000 7f6e5fef8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6e580d4420 con 0x7f6e5815e7a0 2024-09-16T01:34:46.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.943+0000 7f6e5fef8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6e580b7a00 con 0x7f6e580b68f0 2024-09-16T01:34:46.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.943+0000 7f6e5fef8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6e5815f2d0 con 0x7f6e580cf1f0 2024-09-16T01:34:46.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.943+0000 7f6e5e495640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e5815e7a0 0x7f6e58161220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:46.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.943+0000 7f6e5e495640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e5815e7a0 0x7f6e58161220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54884/0 (socket says 172.21.15.88:54884) 2024-09-16T01:34:46.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.943+0000 7f6e5e495640 1 -- 172.21.15.88:0/378354983 learned_addr learned my addr 172.21.15.88:0/378354983 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:46.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.943+0000 7f6e5d493640 1 --2- 172.21.15.88:0/378354983 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e580cf1f0 0x7f6e5814fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:46.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.943+0000 7f6e5dc94640 1 --2- 172.21.15.88:0/378354983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e580b68f0 0x7f6e5815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:46.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.943+0000 7f6e5e495640 1 -- 172.21.15.88:0/378354983 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e580cf1f0 msgr2=0x7f6e5814fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:46.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.944+0000 7f6e5e495640 1 --2- 172.21.15.88:0/378354983 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e580cf1f0 0x7f6e5814fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:46.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.944+0000 7f6e5e495640 1 -- 172.21.15.88:0/378354983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e580b68f0 msgr2=0x7f6e5815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:46.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.944+0000 7f6e5e495640 1 --2- 172.21.15.88:0/378354983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e580b68f0 0x7f6e5815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:46.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.944+0000 7f6e5e495640 1 -- 172.21.15.88:0/378354983 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e5814ffa0 con 0x7f6e5815e7a0 2024-09-16T01:34:46.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.944+0000 7f6e5d493640 1 --2- 172.21.15.88:0/378354983 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e580cf1f0 0x7f6e5814fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:46.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.944+0000 7f6e5dc94640 1 --2- 172.21.15.88:0/378354983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e580b68f0 0x7f6e5815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:46.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.944+0000 7f6e5e495640 1 --2- 172.21.15.88:0/378354983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e5815e7a0 0x7f6e58161220 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6e50009e50 tx=0x7f6e50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:46.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.944+0000 7f6e4e7fc640 1 -- 172.21.15.88:0/378354983 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e50041020 con 0x7f6e5815e7a0 2024-09-16T01:34:46.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.944+0000 7f6e4e7fc640 1 -- 172.21.15.88:0/378354983 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e50007780 con 0x7f6e5815e7a0 2024-09-16T01:34:46.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.944+0000 7f6e4e7fc640 1 -- 172.21.15.88:0/378354983 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e50036040 con 0x7f6e5815e7a0 2024-09-16T01:34:46.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.944+0000 7f6e5fef8640 1 -- 172.21.15.88:0/378354983 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e580b4550 con 0x7f6e5815e7a0 2024-09-16T01:34:46.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.944+0000 7f6e5fef8640 1 -- 172.21.15.88:0/378354983 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e580b4b10 con 0x7f6e5815e7a0 2024-09-16T01:34:46.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.945+0000 7f6e5fef8640 1 -- 172.21.15.88:0/378354983 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e5815eb80 con 0x7f6e5815e7a0 2024-09-16T01:34:46.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.945+0000 7f6e4e7fc640 1 -- 172.21.15.88:0/378354983 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6e50044020 con 0x7f6e5815e7a0 2024-09-16T01:34:46.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.946+0000 7f6e4e7fc640 1 --2- 172.21.15.88:0/378354983 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6e3c06c2e0 0x7f6e3c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:46.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.946+0000 7f6e4e7fc640 1 -- 172.21.15.88:0/378354983 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6e500ac150 con 0x7f6e5815e7a0 2024-09-16T01:34:46.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.946+0000 7f6e5dc94640 1 --2- 172.21.15.88:0/378354983 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6e3c06c2e0 0x7f6e3c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:46.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.946+0000 7f6e5dc94640 1 --2- 172.21.15.88:0/378354983 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6e3c06c2e0 0x7f6e3c06e7a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6e5400a810 tx=0x7f6e54008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:46.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:46.950+0000 7f6e4e7fc640 1 -- 172.21.15.88:0/378354983 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6e5004b030 con 0x7f6e5815e7a0 2024-09-16T01:34:47.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.126+0000 7f6e5fef8640 1 -- 172.21.15.88:0/378354983 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6e580d2a80 con 0x7f6e5815e7a0 2024-09-16T01:34:47.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.127+0000 7f6e4e7fc640 1 -- 172.21.15.88:0/378354983 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f6e5007a950 con 0x7f6e5815e7a0 2024-09-16T01:34:47.129 INFO:teuthology.orchestra.run.smithi088.stderr:listed 0 entries 2024-09-16T01:34:47.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.129+0000 7f6e5fef8640 1 -- 172.21.15.88:0/378354983 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6e3c06c2e0 msgr2=0x7f6e3c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:47.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.129+0000 7f6e5fef8640 1 --2- 172.21.15.88:0/378354983 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6e3c06c2e0 0x7f6e3c06e7a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6e5400a810 tx=0x7f6e54008040 comp rx=0 tx=0).stop 2024-09-16T01:34:47.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.129+0000 7f6e5fef8640 1 -- 172.21.15.88:0/378354983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e5815e7a0 msgr2=0x7f6e58161220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:47.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.129+0000 7f6e5fef8640 1 --2- 172.21.15.88:0/378354983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e5815e7a0 0x7f6e58161220 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6e50009e50 tx=0x7f6e50004a40 comp rx=0 tx=0).stop 2024-09-16T01:34:47.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.129+0000 7f6e5fef8640 1 -- 172.21.15.88:0/378354983 shutdown_connections 2024-09-16T01:34:47.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.129+0000 7f6e5fef8640 1 --2- 172.21.15.88:0/378354983 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6e3c06c2e0 0x7f6e3c06e7a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:47.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.129+0000 7f6e5fef8640 1 --2- 172.21.15.88:0/378354983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e5815e7a0 0x7f6e58161220 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:47.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.129+0000 7f6e5fef8640 1 --2- 172.21.15.88:0/378354983 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e580cf1f0 0x7f6e5814fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:47.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.129+0000 7f6e5fef8640 1 --2- 172.21.15.88:0/378354983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e580b68f0 0x7f6e5815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:47.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.129+0000 7f6e5fef8640 1 -- 172.21.15.88:0/378354983 >> 172.21.15.88:0/378354983 conn(0x7f6e580d11e0 msgr2=0x7f6e58159bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:47.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.129+0000 7f6e5fef8640 1 -- 172.21.15.88:0/378354983 shutdown_connections 2024-09-16T01:34:47.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.129+0000 7f6e5fef8640 1 -- 172.21.15.88:0/378354983 wait complete. 2024-09-16T01:34:47.141 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-16T01:34:47.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:47.142 DEBUG:teuthology.orchestra.run.smithi088:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-09-16T01:34:47.168 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:34:47.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:47.169 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:34:47.235 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:34:47.235 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:34:47.256 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:34:47.257 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:34:47.317 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:34:47.317 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:34:47.375 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:34:47.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.473+0000 7efefb8af640 1 Processor -- start 2024-09-16T01:34:47.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.473+0000 7efefb8af640 1 -- start start 2024-09-16T01:34:47.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.474+0000 7efefb8af640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efef415c710 0x7efef4073970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:47.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.474+0000 7efefb8af640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efef40cdf50 con 0x7efef40b53f0 2024-09-16T01:34:47.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.474+0000 7efefb8af640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efef40ce230 con 0x7efef4073eb0 2024-09-16T01:34:47.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.474+0000 7efefb8af640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7efef40ce510 con 0x7efef415c710 2024-09-16T01:34:47.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.474+0000 7efef9e4c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7efef40b53f0 0x7efef40ceff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54068/0 (socket says 172.21.15.88:54068) 2024-09-16T01:34:47.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef9e4c640 1 -- 172.21.15.88:0/633111153 learned_addr learned my addr 172.21.15.88:0/633111153 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:47.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef964b640 1 --2- 172.21.15.88:0/633111153 >> v2:172.21.15.154:3300/0 conn(0x7efef415c710 0x7efef4073970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:47.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef3fff640 1 -- 172.21.15.88:0/633111153 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 135774937 0 0) 0x7efef40cdf50 con 0x7efef40b53f0 2024-09-16T01:34:47.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef3fff640 1 -- 172.21.15.88:0/633111153 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efedc003520 con 0x7efef40b53f0 2024-09-16T01:34:47.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef3fff640 1 -- 172.21.15.88:0/633111153 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2693656027 0 0) 0x7efef40ce230 con 0x7efef4073eb0 2024-09-16T01:34:47.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef3fff640 1 -- 172.21.15.88:0/633111153 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efef40cdf50 con 0x7efef4073eb0 2024-09-16T01:34:47.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef3fff640 1 -- 172.21.15.88:0/633111153 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3871406154 0 0) 0x7efedc003520 con 0x7efef40b53f0 2024-09-16T01:34:47.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef3fff640 1 -- 172.21.15.88:0/633111153 >> v2:172.21.15.154:3300/0 conn(0x7efef415c710 msgr2=0x7efef4073970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:47.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef3fff640 1 --2- 172.21.15.88:0/633111153 >> v2:172.21.15.154:3300/0 conn(0x7efef415c710 0x7efef4073970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:47.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef3fff640 1 -- 172.21.15.88:0/633111153 >> v1:172.21.15.154:6789/0 conn(0x7efef4073eb0 legacy=0x7efef40d2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:47.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef3fff640 1 -- 172.21.15.88:0/633111153 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef40ce7f0 con 0x7efef40b53f0 2024-09-16T01:34:47.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef3fff640 1 -- 172.21.15.88:0/633111153 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7efeec001670 con 0x7efef40b53f0 2024-09-16T01:34:47.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef3fff640 1 -- 172.21.15.88:0/633111153 >> v1:172.21.15.88:6789/0 conn(0x7efef40b53f0 legacy=0x7efef40ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:47.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef3fff640 1 --2- 172.21.15.88:0/633111153 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efedc003b40 0x7efedc003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:47.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef3fff640 1 --2- 172.21.15.88:0/633111153 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efedc004800 0x7efedc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:47.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef3fff640 1 --2- 172.21.15.88:0/633111153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efedc0051c0 0x7efedc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:47.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef3fff640 1 -- 172.21.15.88:0/633111153 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7efef40ce510 con 0x7efedc003b40 2024-09-16T01:34:47.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef3fff640 1 -- 172.21.15.88:0/633111153 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efedc00c0f0 con 0x7efedc0051c0 2024-09-16T01:34:47.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef3fff640 1 -- 172.21.15.88:0/633111153 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efedc00c2f0 con 0x7efedc004800 2024-09-16T01:34:47.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef964b640 1 --2- 172.21.15.88:0/633111153 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efedc003b40 0x7efedc003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:47.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef8e4a640 1 --2- 172.21.15.88:0/633111153 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efedc004800 0x7efedc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:47.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef9e4c640 1 --2- 172.21.15.88:0/633111153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efedc0051c0 0x7efedc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:47.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef964b640 1 -- 172.21.15.88:0/633111153 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efedc004800 msgr2=0x7efedc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:47.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef964b640 1 --2- 172.21.15.88:0/633111153 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efedc004800 0x7efedc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:47.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef964b640 1 -- 172.21.15.88:0/633111153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efedc0051c0 msgr2=0x7efedc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:47.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef964b640 1 --2- 172.21.15.88:0/633111153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efedc0051c0 0x7efedc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:47.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef964b640 1 -- 172.21.15.88:0/633111153 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efedc003520 con 0x7efedc003b40 2024-09-16T01:34:47.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef9e4c640 1 --2- 172.21.15.88:0/633111153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efedc0051c0 0x7efedc00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:47.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.475+0000 7efef9e4c640 1 -- 172.21.15.88:0/633111153 reap_dead start 2024-09-16T01:34:47.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.476+0000 7efef964b640 1 --2- 172.21.15.88:0/633111153 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efedc003b40 0x7efedc003f40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7efee4009fc0 tx=0x7efedc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=737f88aa25ce3ddb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:47.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.476+0000 7efef3fff640 1 -- 172.21.15.88:0/633111153 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee4009c10 con 0x7efedc003b40 2024-09-16T01:34:47.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.476+0000 7efef3fff640 1 -- 172.21.15.88:0/633111153 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efee4036070 con 0x7efedc003b40 2024-09-16T01:34:47.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.476+0000 7efef3fff640 1 -- 172.21.15.88:0/633111153 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee4031880 con 0x7efedc003b40 2024-09-16T01:34:47.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.477+0000 7efefb8af640 1 -- 172.21.15.88:0/633111153 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efedc003b40 msgr2=0x7efedc003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:47.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.477+0000 7efefb8af640 1 --2- 172.21.15.88:0/633111153 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efedc003b40 0x7efedc003f40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7efee4009fc0 tx=0x7efedc005830 comp rx=0 tx=0).stop 2024-09-16T01:34:47.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.477+0000 7efefb8af640 1 -- 172.21.15.88:0/633111153 shutdown_connections 2024-09-16T01:34:47.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.477+0000 7efefb8af640 1 --2- 172.21.15.88:0/633111153 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efedc003b40 0x7efedc003f40 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:47.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.477+0000 7efefb8af640 1 -- 172.21.15.88:0/633111153 >> 172.21.15.88:0/633111153 conn(0x7efef4062a20 msgr2=0x7efef4062e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:47.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.477+0000 7efefb8af640 1 -- 172.21.15.88:0/633111153 shutdown_connections 2024-09-16T01:34:47.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.477+0000 7efefb8af640 1 -- 172.21.15.88:0/633111153 wait complete. 2024-09-16T01:34:47.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.478+0000 7efefb8af640 1 Processor -- start 2024-09-16T01:34:47.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.478+0000 7efefb8af640 1 -- start start 2024-09-16T01:34:47.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.479+0000 7efefb8af640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efedc003b40 0x7efef406e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:47.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.479+0000 7efefb8af640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efef415cb80 0x7efef415d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:47.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.479+0000 7efefb8af640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efef415eb70 0x7efef415eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:47.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.479+0000 7efefb8af640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7efef4073970 con 0x7efef415cb80 2024-09-16T01:34:47.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.479+0000 7efefb8af640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efef40cdc00 con 0x7efedc003b40 2024-09-16T01:34:47.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.479+0000 7efefb8af640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efef4073af0 con 0x7efef415eb70 2024-09-16T01:34:47.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.479+0000 7efef8e4a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efef415cb80 0x7efef415d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:47.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.479+0000 7efef8e4a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efef415cb80 0x7efef415d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54898/0 (socket says 172.21.15.88:54898) 2024-09-16T01:34:47.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.479+0000 7efef8e4a640 1 -- 172.21.15.88:0/4230129122 learned_addr learned my addr 172.21.15.88:0/4230129122 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:47.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.479+0000 7efef9e4c640 1 --2- 172.21.15.88:0/4230129122 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efef415eb70 0x7efef415eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:47.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.479+0000 7efef964b640 1 --2- 172.21.15.88:0/4230129122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efedc003b40 0x7efef406e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:47.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.479+0000 7efef8e4a640 1 -- 172.21.15.88:0/4230129122 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efef415eb70 msgr2=0x7efef415eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:47.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.479+0000 7efef8e4a640 1 --2- 172.21.15.88:0/4230129122 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efef415eb70 0x7efef415eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:47.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.479+0000 7efef8e4a640 1 -- 172.21.15.88:0/4230129122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efedc003b40 msgr2=0x7efef406e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:47.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.479+0000 7efef8e4a640 1 --2- 172.21.15.88:0/4230129122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efedc003b40 0x7efef406e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:47.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.479+0000 7efef8e4a640 1 -- 172.21.15.88:0/4230129122 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef415f880 con 0x7efef415cb80 2024-09-16T01:34:47.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.479+0000 7efef8e4a640 1 --2- 172.21.15.88:0/4230129122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efef415cb80 0x7efef415d000 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7efee800f7a0 tx=0x7efee800fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:47.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.480+0000 7efef1ffb640 1 -- 172.21.15.88:0/4230129122 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee800fe50 con 0x7efef415cb80 2024-09-16T01:34:47.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.480+0000 7efef1ffb640 1 -- 172.21.15.88:0/4230129122 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efee8018070 con 0x7efef415cb80 2024-09-16T01:34:47.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.480+0000 7efefb8af640 1 -- 172.21.15.88:0/4230129122 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efef40c95f0 con 0x7efef415cb80 2024-09-16T01:34:47.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.480+0000 7efef1ffb640 1 -- 172.21.15.88:0/4230129122 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee800d6d0 con 0x7efef415cb80 2024-09-16T01:34:47.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.480+0000 7efefb8af640 1 -- 172.21.15.88:0/4230129122 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efef40c9bb0 con 0x7efef415cb80 2024-09-16T01:34:47.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.481+0000 7efefb8af640 1 -- 172.21.15.88:0/4230129122 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efef40d2110 con 0x7efef415cb80 2024-09-16T01:34:47.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.481+0000 7efef1ffb640 1 -- 172.21.15.88:0/4230129122 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7efee800d870 con 0x7efef415cb80 2024-09-16T01:34:47.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.482+0000 7efef1ffb640 1 --2- 172.21.15.88:0/4230129122 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efee006c2b0 0x7efee006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:47.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.482+0000 7efef1ffb640 1 -- 172.21.15.88:0/4230129122 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7efee808b270 con 0x7efef415cb80 2024-09-16T01:34:47.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.482+0000 7efef964b640 1 --2- 172.21.15.88:0/4230129122 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efee006c2b0 0x7efee006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:47.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.482+0000 7efef964b640 1 --2- 172.21.15.88:0/4230129122 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efee006c2b0 0x7efee006e770 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efee4035000 tx=0x7efee402f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:47.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.486+0000 7efef1ffb640 1 -- 172.21.15.88:0/4230129122 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7efee80599e0 con 0x7efef415cb80 2024-09-16T01:34:47.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.665+0000 7efefb8af640 1 -- 172.21.15.88:0/4230129122 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7efef40b7810 con 0x7efef415cb80 2024-09-16T01:34:47.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.665+0000 7efef1ffb640 1 -- 172.21.15.88:0/4230129122 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7efee80592e0 con 0x7efef415cb80 2024-09-16T01:34:47.667 INFO:teuthology.orchestra.run.smithi088.stderr:listed 0 entries 2024-09-16T01:34:47.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.667+0000 7efefb8af640 1 -- 172.21.15.88:0/4230129122 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efee006c2b0 msgr2=0x7efee006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:47.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.667+0000 7efefb8af640 1 --2- 172.21.15.88:0/4230129122 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efee006c2b0 0x7efee006e770 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efee4035000 tx=0x7efee402f000 comp rx=0 tx=0).stop 2024-09-16T01:34:47.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.667+0000 7efefb8af640 1 -- 172.21.15.88:0/4230129122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efef415cb80 msgr2=0x7efef415d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:47.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.667+0000 7efefb8af640 1 --2- 172.21.15.88:0/4230129122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efef415cb80 0x7efef415d000 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7efee800f7a0 tx=0x7efee800fc60 comp rx=0 tx=0).stop 2024-09-16T01:34:47.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.667+0000 7efefb8af640 1 -- 172.21.15.88:0/4230129122 shutdown_connections 2024-09-16T01:34:47.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.667+0000 7efefb8af640 1 --2- 172.21.15.88:0/4230129122 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efee006c2b0 0x7efee006e770 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:47.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.667+0000 7efefb8af640 1 --2- 172.21.15.88:0/4230129122 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efef415eb70 0x7efef415eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:47.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.667+0000 7efefb8af640 1 --2- 172.21.15.88:0/4230129122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efef415cb80 0x7efef415d000 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:47.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.667+0000 7efefb8af640 1 --2- 172.21.15.88:0/4230129122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efedc003b40 0x7efef406e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:47.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.668+0000 7efefb8af640 1 -- 172.21.15.88:0/4230129122 >> 172.21.15.88:0/4230129122 conn(0x7efef4062a20 msgr2=0x7efef40b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:47.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.668+0000 7efefb8af640 1 -- 172.21.15.88:0/4230129122 shutdown_connections 2024-09-16T01:34:47.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:47.668+0000 7efefb8af640 1 -- 172.21.15.88:0/4230129122 wait complete. 2024-09-16T01:34:47.680 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-16T01:34:47.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:47.680 DEBUG:teuthology.orchestra.run.smithi154:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-09-16T01:34:47.712 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi154.front.sepia.ceph.com 2024-09-16T01:34:47.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:47.713 DEBUG:teuthology.orchestra.run.smithi154:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-16T01:34:47.777 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:34:47.778 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:34:47.794 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:34:47.794 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:34:47.850 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:34:47.850 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:34:47.906 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:34:48.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.047+0000 7f9ede3a6640 1 Processor -- start 2024-09-16T01:34:48.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.047+0000 7f9ede3a6640 1 -- start start 2024-09-16T01:34:48.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.048+0000 7f9ede3a6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ed80cf1f0 0x7f9ed80b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:48.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.048+0000 7f9ede3a6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9ed80b6720 0x7f9ed80d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:48.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.048+0000 7f9ede3a6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ed80cf860 con 0x7f9ed815e7a0 2024-09-16T01:34:48.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.048+0000 7f9ede3a6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9ed80d2ca0 con 0x7f9ed80b6720 2024-09-16T01:34:48.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.048+0000 7f9ede3a6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9ed80cfb40 con 0x7f9ed80cf1f0 2024-09-16T01:34:48.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.048+0000 7f9ed77fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ed80cf1f0 0x7f9ed80b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:48.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.048+0000 7f9edc943640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9ed80b6720 0x7f9ed80d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:48.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.048+0000 7f9ed77fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ed80cf1f0 0x7f9ed80b61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:37938/0 (socket says 172.21.15.88:37938) 2024-09-16T01:34:48.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.048+0000 7f9ed77fe640 1 -- 172.21.15.88:0/4145134689 learned_addr learned my addr 172.21.15.88:0/4145134689 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:48.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.048+0000 7f9edc943640 1 -- 172.21.15.88:0/4145134689 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ed80cf1f0 msgr2=0x7f9ed80b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:48.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.048+0000 7f9edc943640 1 --2- 172.21.15.88:0/4145134689 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ed80cf1f0 0x7f9ed80b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.049+0000 7f9edc943640 1 -- 172.21.15.88:0/4145134689 >> v1:172.21.15.154:6789/0 conn(0x7f9ed815e7a0 legacy=0x7f9ed806caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:48.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.049+0000 7f9edc943640 1 -- 172.21.15.88:0/4145134689 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ed80cfe20 con 0x7f9ed80b6720 2024-09-16T01:34:48.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.049+0000 7f9ed77fe640 1 --2- 172.21.15.88:0/4145134689 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ed80cf1f0 0x7f9ed80b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:34:48.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.049+0000 7f9edc943640 1 --2- 172.21.15.88:0/4145134689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ed80b6720 0x7f9ed80d3410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0009d90 tx=0x7f9ed002a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db189ae6ec646d00 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:48.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.049+0000 7f9ed6ffd640 1 -- 172.21.15.88:0/4145134689 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0031600 con 0x7f9ed80b6720 2024-09-16T01:34:48.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.050+0000 7f9ed6ffd640 1 -- 172.21.15.88:0/4145134689 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ed00317a0 con 0x7f9ed80b6720 2024-09-16T01:34:48.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.050+0000 7f9ed6ffd640 1 -- 172.21.15.88:0/4145134689 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0031b70 con 0x7f9ed80b6720 2024-09-16T01:34:48.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.050+0000 7f9ede3a6640 1 -- 172.21.15.88:0/4145134689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ed80b6720 msgr2=0x7f9ed80d3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:48.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.050+0000 7f9ede3a6640 1 --2- 172.21.15.88:0/4145134689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ed80b6720 0x7f9ed80d3410 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0009d90 tx=0x7f9ed002a5d0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.050+0000 7f9ede3a6640 1 -- 172.21.15.88:0/4145134689 shutdown_connections 2024-09-16T01:34:48.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.050+0000 7f9ede3a6640 1 --2- 172.21.15.88:0/4145134689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ed80b6720 0x7f9ed80d3410 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.050+0000 7f9ede3a6640 1 --2- 172.21.15.88:0/4145134689 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ed80cf1f0 0x7f9ed80b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.050+0000 7f9ede3a6640 1 -- 172.21.15.88:0/4145134689 >> 172.21.15.88:0/4145134689 conn(0x7f9ed80d11e0 msgr2=0x7f9ed80d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:48.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.050+0000 7f9ede3a6640 1 -- 172.21.15.88:0/4145134689 shutdown_connections 2024-09-16T01:34:48.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.050+0000 7f9ede3a6640 1 -- 172.21.15.88:0/4145134689 wait complete. 2024-09-16T01:34:48.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.051+0000 7f9ede3a6640 1 Processor -- start 2024-09-16T01:34:48.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.052+0000 7f9ede3a6640 1 -- start start 2024-09-16T01:34:48.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.052+0000 7f9ede3a6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ed80b6720 0x7f9ed814fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:48.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.052+0000 7f9ede3a6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ed80cf1f0 0x7f9ed8161060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:48.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.052+0000 7f9ede3a6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ed815e7a0 0x7f9ed80cdd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:48.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.052+0000 7f9ede3a6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed815eb80 con 0x7f9ed815e7a0 2024-09-16T01:34:48.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.052+0000 7f9ede3a6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed80b7a00 con 0x7f9ed80cf1f0 2024-09-16T01:34:48.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.052+0000 7f9ede3a6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9ed815ed00 con 0x7f9ed80b6720 2024-09-16T01:34:48.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.052+0000 7f9edc943640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ed815e7a0 0x7f9ed80cdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:48.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.052+0000 7f9edc943640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ed815e7a0 0x7f9ed80cdd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54900/0 (socket says 172.21.15.88:54900) 2024-09-16T01:34:48.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.052+0000 7f9edc943640 1 -- 172.21.15.88:0/4202277373 learned_addr learned my addr 172.21.15.88:0/4202277373 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:48.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.052+0000 7f9ed7fff640 1 --2- 172.21.15.88:0/4202277373 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ed80b6720 0x7f9ed814fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:48.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.052+0000 7f9ed77fe640 1 --2- 172.21.15.88:0/4202277373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ed80cf1f0 0x7f9ed8161060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:48.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.053+0000 7f9edc943640 1 -- 172.21.15.88:0/4202277373 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ed80b6720 msgr2=0x7f9ed814fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:48.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.053+0000 7f9edc943640 1 --2- 172.21.15.88:0/4202277373 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ed80b6720 0x7f9ed814fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.053+0000 7f9edc943640 1 -- 172.21.15.88:0/4202277373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ed80cf1f0 msgr2=0x7f9ed8161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:48.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.053+0000 7f9edc943640 1 --2- 172.21.15.88:0/4202277373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ed80cf1f0 0x7f9ed8161060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.053+0000 7f9edc943640 1 -- 172.21.15.88:0/4202277373 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ed80ce7d0 con 0x7f9ed815e7a0 2024-09-16T01:34:48.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.053+0000 7f9edc943640 1 --2- 172.21.15.88:0/4202277373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ed815e7a0 0x7f9ed80cdd50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0009eb0 tx=0x7f9ed0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:48.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.053+0000 7f9ed4ff9640 1 -- 172.21.15.88:0/4202277373 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0040070 con 0x7f9ed815e7a0 2024-09-16T01:34:48.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.053+0000 7f9ed4ff9640 1 -- 172.21.15.88:0/4202277373 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ed0030b60 con 0x7f9ed815e7a0 2024-09-16T01:34:48.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.053+0000 7f9ede3a6640 1 -- 172.21.15.88:0/4202277373 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ed8160580 con 0x7f9ed815e7a0 2024-09-16T01:34:48.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.053+0000 7f9ed4ff9640 1 -- 172.21.15.88:0/4202277373 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0009450 con 0x7f9ed815e7a0 2024-09-16T01:34:48.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.053+0000 7f9ede3a6640 1 -- 172.21.15.88:0/4202277373 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ed8150280 con 0x7f9ed815e7a0 2024-09-16T01:34:48.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.054+0000 7f9ede3a6640 1 -- 172.21.15.88:0/4202277373 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ed80b61e0 con 0x7f9ed815e7a0 2024-09-16T01:34:48.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.055+0000 7f9ed4ff9640 1 -- 172.21.15.88:0/4202277373 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9ed0039040 con 0x7f9ed815e7a0 2024-09-16T01:34:48.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.055+0000 7f9ed4ff9640 1 --2- 172.21.15.88:0/4202277373 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9ec406c2e0 0x7f9ec406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:48.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.055+0000 7f9ed4ff9640 1 -- 172.21.15.88:0/4202277373 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9ed00aca60 con 0x7f9ed815e7a0 2024-09-16T01:34:48.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.055+0000 7f9ed7fff640 1 --2- 172.21.15.88:0/4202277373 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9ec406c2e0 0x7f9ec406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:48.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.055+0000 7f9ed7fff640 1 --2- 172.21.15.88:0/4202277373 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9ec406c2e0 0x7f9ec406e7a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9ed815fcb0 tx=0x7f9ec8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:48.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.059+0000 7f9ed4ff9640 1 -- 172.21.15.88:0/4202277373 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9ed007ad30 con 0x7f9ed815e7a0 2024-09-16T01:34:48.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.237+0000 7f9ede3a6640 1 -- 172.21.15.88:0/4202277373 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f9ed80d2a80 con 0x7f9ed815e7a0 2024-09-16T01:34:48.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.237+0000 7f9ed4ff9640 1 -- 172.21.15.88:0/4202277373 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f9ed007f030 con 0x7f9ed815e7a0 2024-09-16T01:34:48.239 INFO:teuthology.orchestra.run.smithi088.stderr:listed 0 entries 2024-09-16T01:34:48.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.240+0000 7f9ede3a6640 1 -- 172.21.15.88:0/4202277373 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9ec406c2e0 msgr2=0x7f9ec406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:48.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.240+0000 7f9ede3a6640 1 --2- 172.21.15.88:0/4202277373 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9ec406c2e0 0x7f9ec406e7a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9ed815fcb0 tx=0x7f9ec8009040 comp rx=0 tx=0).stop 2024-09-16T01:34:48.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.240+0000 7f9ede3a6640 1 -- 172.21.15.88:0/4202277373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ed815e7a0 msgr2=0x7f9ed80cdd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:48.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.240+0000 7f9ede3a6640 1 --2- 172.21.15.88:0/4202277373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ed815e7a0 0x7f9ed80cdd50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0009eb0 tx=0x7f9ed0004a40 comp rx=0 tx=0).stop 2024-09-16T01:34:48.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.240+0000 7f9ede3a6640 1 -- 172.21.15.88:0/4202277373 shutdown_connections 2024-09-16T01:34:48.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.240+0000 7f9ede3a6640 1 --2- 172.21.15.88:0/4202277373 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9ec406c2e0 0x7f9ec406e7a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.240+0000 7f9ede3a6640 1 --2- 172.21.15.88:0/4202277373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ed815e7a0 0x7f9ed80cdd50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.240+0000 7f9ede3a6640 1 --2- 172.21.15.88:0/4202277373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ed80cf1f0 0x7f9ed8161060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.240+0000 7f9ede3a6640 1 --2- 172.21.15.88:0/4202277373 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ed80b6720 0x7f9ed814fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.240+0000 7f9ede3a6640 1 -- 172.21.15.88:0/4202277373 >> 172.21.15.88:0/4202277373 conn(0x7f9ed80d11e0 msgr2=0x7f9ed815d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:48.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.240+0000 7f9ede3a6640 1 -- 172.21.15.88:0/4202277373 shutdown_connections 2024-09-16T01:34:48.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.240+0000 7f9ede3a6640 1 -- 172.21.15.88:0/4202277373 wait complete. 2024-09-16T01:34:48.252 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-09-16T01:34:48.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:48.252 DEBUG:teuthology.orchestra.run.smithi088:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-09-16T01:34:48.278 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:34:48.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:48.278 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-09-16T01:34:48.333 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:34:48.334 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:34:48.353 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:34:48.353 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:34:48.413 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:34:48.413 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:34:48.471 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:34:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.570+0000 7f08f42ba640 1 Processor -- start 2024-09-16T01:34:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.571+0000 7f08f42ba640 1 -- start start 2024-09-16T01:34:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.571+0000 7f08f42ba640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08ec0b68f0 0x7f08ec1696f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.571+0000 7f08f42ba640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08ec0cf860 con 0x7f08ec0cf1f0 2024-09-16T01:34:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.571+0000 7f08f42ba640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08ec0cfb40 con 0x7f08ec15e7a0 2024-09-16T01:34:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.571+0000 7f08f42ba640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f08ec0d2ca0 con 0x7f08ec0b68f0 2024-09-16T01:34:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.572+0000 7f08f1855640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f08ec0cf1f0 0x7f08ec0b61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54082/0 (socket says 172.21.15.88:54082) 2024-09-16T01:34:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.572+0000 7f08f1855640 1 -- 172.21.15.88:0/1412416033 learned_addr learned my addr 172.21.15.88:0/1412416033 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.572+0000 7f08f2857640 1 --2- 172.21.15.88:0/1412416033 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08ec0b68f0 0x7f08ec1696f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.572+0000 7f08f1054640 1 -- 172.21.15.88:0/1412416033 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 436047659 0 0) 0x7f08ec0cf860 con 0x7f08ec0cf1f0 2024-09-16T01:34:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.572+0000 7f08f1054640 1 -- 172.21.15.88:0/1412416033 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08d8003520 con 0x7f08ec0cf1f0 2024-09-16T01:34:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.572+0000 7f08f1054640 1 -- 172.21.15.88:0/1412416033 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1529863794 0 0) 0x7f08d8003520 con 0x7f08ec0cf1f0 2024-09-16T01:34:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.572+0000 7f08f1054640 1 -- 172.21.15.88:0/1412416033 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08ec0b68f0 msgr2=0x7f08ec1696f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.572+0000 7f08f1054640 1 --2- 172.21.15.88:0/1412416033 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08ec0b68f0 0x7f08ec1696f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.572+0000 7f08f1054640 1 -- 172.21.15.88:0/1412416033 >> v1:172.21.15.154:6789/0 conn(0x7f08ec15e7a0 legacy=0x7f08ec06caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.572+0000 7f08f1054640 1 -- 172.21.15.88:0/1412416033 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08ec0cfe20 con 0x7f08ec0cf1f0 2024-09-16T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.572+0000 7f08f1054640 1 -- 172.21.15.88:0/1412416033 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f08dc001670 con 0x7f08ec0cf1f0 2024-09-16T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.572+0000 7f08f1054640 1 -- 172.21.15.88:0/1412416033 >> v1:172.21.15.88:6789/0 conn(0x7f08ec0cf1f0 legacy=0x7f08ec0b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.572+0000 7f08f1054640 1 --2- 172.21.15.88:0/1412416033 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08ec0b68f0 0x7f08d80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.572+0000 7f08f1054640 1 --2- 172.21.15.88:0/1412416033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08d8003f20 0x7f08d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.572+0000 7f08f1054640 1 --2- 172.21.15.88:0/1412416033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08d8004d40 0x7f08d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.572+0000 7f08f1054640 1 -- 172.21.15.88:0/1412416033 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec0d2ca0 con 0x7f08d8003f20 2024-09-16T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.572+0000 7f08f1054640 1 -- 172.21.15.88:0/1412416033 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f08d800bad0 con 0x7f08d8004d40 2024-09-16T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.572+0000 7f08f1054640 1 -- 172.21.15.88:0/1412416033 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f08d800bcd0 con 0x7f08ec0b68f0 2024-09-16T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.572+0000 7f08f1855640 1 --2- 172.21.15.88:0/1412416033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08d8003f20 0x7f08d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.572+0000 7f08f2056640 1 --2- 172.21.15.88:0/1412416033 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08ec0b68f0 0x7f08d80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.572+0000 7f08f2857640 1 --2- 172.21.15.88:0/1412416033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08d8004d40 0x7f08d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.573+0000 7f08f1855640 1 -- 172.21.15.88:0/1412416033 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08ec0b68f0 msgr2=0x7f08d80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.573+0000 7f08f1855640 1 --2- 172.21.15.88:0/1412416033 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08ec0b68f0 0x7f08d80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.573+0000 7f08f1855640 1 -- 172.21.15.88:0/1412416033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08d8004d40 msgr2=0x7f08d80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.573+0000 7f08f1855640 1 --2- 172.21.15.88:0/1412416033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08d8004d40 0x7f08d80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.573+0000 7f08f1855640 1 -- 172.21.15.88:0/1412416033 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08d8003520 con 0x7f08d8003f20 2024-09-16T01:34:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.573+0000 7f08f1855640 1 --2- 172.21.15.88:0/1412416033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08d8003f20 0x7f08d8004800 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00c290 tx=0x7f08d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4b18632531c007 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.573+0000 7f08f1054640 1 -- 172.21.15.88:0/1412416033 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc007d50 con 0x7f08d8003f20 2024-09-16T01:34:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.573+0000 7f08f1054640 1 -- 172.21.15.88:0/1412416033 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08dc005af0 con 0x7f08d8003f20 2024-09-16T01:34:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.573+0000 7f08f1054640 1 -- 172.21.15.88:0/1412416033 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08dc007070 con 0x7f08d8003f20 2024-09-16T01:34:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.574+0000 7f08f42ba640 1 -- 172.21.15.88:0/1412416033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08d8003f20 msgr2=0x7f08d8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.574+0000 7f08f42ba640 1 --2- 172.21.15.88:0/1412416033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08d8003f20 0x7f08d8004800 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00c290 tx=0x7f08d8005830 comp rx=0 tx=0).stop 2024-09-16T01:34:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.574+0000 7f08f2857640 1 -- 172.21.15.88:0/1412416033 reap_dead start 2024-09-16T01:34:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.574+0000 7f08f42ba640 1 -- 172.21.15.88:0/1412416033 shutdown_connections 2024-09-16T01:34:48.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.574+0000 7f08f42ba640 1 -- 172.21.15.88:0/1412416033 >> 172.21.15.88:0/1412416033 conn(0x7f08ec0d11e0 msgr2=0x7f08ec0d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:48.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.574+0000 7f08f42ba640 1 -- 172.21.15.88:0/1412416033 shutdown_connections 2024-09-16T01:34:48.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.574+0000 7f08f42ba640 1 -- 172.21.15.88:0/1412416033 wait complete. 2024-09-16T01:34:48.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.575+0000 7f08f42ba640 1 Processor -- start 2024-09-16T01:34:48.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.575+0000 7f08f42ba640 1 -- start start 2024-09-16T01:34:48.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.576+0000 7f08f42ba640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08d8003f20 0x7f08ec0cc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:48.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.576+0000 7f08f42ba640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08ec14f980 0x7f08ec0b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:48.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.576+0000 7f08f42ba640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08ec14fe00 0x7f08ec0b80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:48.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.576+0000 7f08f42ba640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec0d4420 con 0x7f08ec14fe00 2024-09-16T01:34:48.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.576+0000 7f08f42ba640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec0b7a00 con 0x7f08d8003f20 2024-09-16T01:34:48.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.576+0000 7f08f42ba640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f08ec0b9b70 con 0x7f08ec14f980 2024-09-16T01:34:48.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.576+0000 7f08f2857640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08ec14fe00 0x7f08ec0b80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:48.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.576+0000 7f08f2857640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08ec14fe00 0x7f08ec0b80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54922/0 (socket says 172.21.15.88:54922) 2024-09-16T01:34:48.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.576+0000 7f08f2857640 1 -- 172.21.15.88:0/3004669845 learned_addr learned my addr 172.21.15.88:0/3004669845 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:48.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.576+0000 7f08f1855640 1 --2- 172.21.15.88:0/3004669845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08ec14f980 0x7f08ec0b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:48.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.576+0000 7f08f2056640 1 --2- 172.21.15.88:0/3004669845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08d8003f20 0x7f08ec0cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:48.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.576+0000 7f08f2857640 1 -- 172.21.15.88:0/3004669845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08ec14f980 msgr2=0x7f08ec0b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:48.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.576+0000 7f08f2857640 1 --2- 172.21.15.88:0/3004669845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08ec14f980 0x7f08ec0b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.576+0000 7f08f2857640 1 -- 172.21.15.88:0/3004669845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08d8003f20 msgr2=0x7f08ec0cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:48.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.576+0000 7f08f2857640 1 --2- 172.21.15.88:0/3004669845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08d8003f20 0x7f08ec0cc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.576+0000 7f08f2857640 1 -- 172.21.15.88:0/3004669845 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08ec0ce670 con 0x7f08ec14fe00 2024-09-16T01:34:48.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.576+0000 7f08f2056640 1 --2- 172.21.15.88:0/3004669845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08d8003f20 0x7f08ec0cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:48.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.577+0000 7f08f2857640 1 --2- 172.21.15.88:0/3004669845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08ec14fe00 0x7f08ec0b80c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f08ec08d950 tx=0x7f08e4011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:48.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.577+0000 7f08e2ffd640 1 -- 172.21.15.88:0/3004669845 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e4004240 con 0x7f08ec14fe00 2024-09-16T01:34:48.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.577+0000 7f08e2ffd640 1 -- 172.21.15.88:0/3004669845 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08e401b070 con 0x7f08ec14fe00 2024-09-16T01:34:48.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.577+0000 7f08f42ba640 1 -- 172.21.15.88:0/3004669845 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08ec0cb9d0 con 0x7f08ec14fe00 2024-09-16T01:34:48.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.577+0000 7f08e2ffd640 1 -- 172.21.15.88:0/3004669845 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e400f7a0 con 0x7f08ec14fe00 2024-09-16T01:34:48.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.577+0000 7f08f42ba640 1 -- 172.21.15.88:0/3004669845 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08ec0cbf90 con 0x7f08ec14fe00 2024-09-16T01:34:48.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.578+0000 7f08f42ba640 1 -- 172.21.15.88:0/3004669845 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08ec15eb80 con 0x7f08ec14fe00 2024-09-16T01:34:48.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.578+0000 7f08e2ffd640 1 -- 172.21.15.88:0/3004669845 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f08e4012040 con 0x7f08ec14fe00 2024-09-16T01:34:48.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.579+0000 7f08e2ffd640 1 --2- 172.21.15.88:0/3004669845 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f08d006c2e0 0x7f08d006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:48.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.579+0000 7f08e2ffd640 1 -- 172.21.15.88:0/3004669845 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f08e408b1e0 con 0x7f08ec14fe00 2024-09-16T01:34:48.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.579+0000 7f08f2056640 1 --2- 172.21.15.88:0/3004669845 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f08d006c2e0 0x7f08d006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:48.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.579+0000 7f08f2056640 1 --2- 172.21.15.88:0/3004669845 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f08d006c2e0 0x7f08d006e7a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f08ec0b97b0 tx=0x7f08e8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:48.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.583+0000 7f08e2ffd640 1 -- 172.21.15.88:0/3004669845 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f08ec0ce670 con 0x7f08ec14fe00 2024-09-16T01:34:48.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.766+0000 7f08f42ba640 1 -- 172.21.15.88:0/3004669845 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f08ec0d2a80 con 0x7f08ec14fe00 2024-09-16T01:34:48.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.766+0000 7f08e2ffd640 1 -- 172.21.15.88:0/3004669845 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f08e4059950 con 0x7f08ec14fe00 2024-09-16T01:34:48.768 INFO:teuthology.orchestra.run.smithi088.stderr:listed 0 entries 2024-09-16T01:34:48.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.768+0000 7f08f42ba640 1 -- 172.21.15.88:0/3004669845 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f08d006c2e0 msgr2=0x7f08d006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:48.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.768+0000 7f08f42ba640 1 --2- 172.21.15.88:0/3004669845 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f08d006c2e0 0x7f08d006e7a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f08ec0b97b0 tx=0x7f08e8009040 comp rx=0 tx=0).stop 2024-09-16T01:34:48.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.768+0000 7f08f42ba640 1 -- 172.21.15.88:0/3004669845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08ec14fe00 msgr2=0x7f08ec0b80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:48.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.768+0000 7f08f42ba640 1 --2- 172.21.15.88:0/3004669845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08ec14fe00 0x7f08ec0b80c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f08ec08d950 tx=0x7f08e4011cf0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.768+0000 7f08f42ba640 1 -- 172.21.15.88:0/3004669845 shutdown_connections 2024-09-16T01:34:48.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.768+0000 7f08f42ba640 1 --2- 172.21.15.88:0/3004669845 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f08d006c2e0 0x7f08d006e7a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.768+0000 7f08f42ba640 1 --2- 172.21.15.88:0/3004669845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08ec14fe00 0x7f08ec0b80c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.768+0000 7f08f42ba640 1 --2- 172.21.15.88:0/3004669845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08ec14f980 0x7f08ec0b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.768+0000 7f08f42ba640 1 --2- 172.21.15.88:0/3004669845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08d8003f20 0x7f08ec0cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.768+0000 7f08f42ba640 1 -- 172.21.15.88:0/3004669845 >> 172.21.15.88:0/3004669845 conn(0x7f08ec0d11e0 msgr2=0x7f08ec15cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:48.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.768+0000 7f08f42ba640 1 -- 172.21.15.88:0/3004669845 shutdown_connections 2024-09-16T01:34:48.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.768+0000 7f08f42ba640 1 -- 172.21.15.88:0/3004669845 wait complete. 2024-09-16T01:34:48.780 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-09-16T01:34:48.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:48.781 DEBUG:teuthology.orchestra.run.smithi154:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-09-16T01:34:48.812 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi154.front.sepia.ceph.com 2024-09-16T01:34:48.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:34:48.812 DEBUG:teuthology.orchestra.run.smithi154:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-09-16T01:34:48.870 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-16T01:34:48.871 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:34:48.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.968+0000 7eff21a53640 1 Processor -- start 2024-09-16T01:34:48.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.968+0000 7eff21a53640 1 -- start start 2024-09-16T01:34:48.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.968+0000 7eff21a53640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7eff1c0d3310 0x7eff1c0d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:48.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.968+0000 7eff21a53640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff1c0d1650 0x7eff1c0b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:48.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.968+0000 7eff21a53640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff1c15ea40 con 0x7eff1c15e660 2024-09-16T01:34:48.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.968+0000 7eff21a53640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7eff1c0d1a90 con 0x7eff1c0d3310 2024-09-16T01:34:48.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.968+0000 7eff21a53640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7eff1c0d3870 con 0x7eff1c0d1650 2024-09-16T01:34:48.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.969+0000 7eff1affd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7eff1c15e660 0x7eff1c0cf730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54096/0 (socket says 172.21.15.88:54096) 2024-09-16T01:34:48.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.969+0000 7eff1affd640 1 -- 172.21.15.88:0/1030890213 learned_addr learned my addr 172.21.15.88:0/1030890213 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:48.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.969+0000 7eff1a7fc640 1 --2- 172.21.15.88:0/1030890213 >> v2:172.21.15.154:3300/0 conn(0x7eff1c0d3310 0x7eff1c0d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:48.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.969+0000 7eff1b7fe640 1 --2- 172.21.15.88:0/1030890213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff1c0d1650 0x7eff1c0b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:48.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.969+0000 7eff19ffb640 1 -- 172.21.15.88:0/1030890213 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3857524872 0 0) 0x7eff1c15ea40 con 0x7eff1c15e660 2024-09-16T01:34:48.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.969+0000 7eff19ffb640 1 -- 172.21.15.88:0/1030890213 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff04003520 con 0x7eff1c15e660 2024-09-16T01:34:48.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.969+0000 7eff19ffb640 1 -- 172.21.15.88:0/1030890213 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1140316104 0 0) 0x7eff04003520 con 0x7eff1c15e660 2024-09-16T01:34:48.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.969+0000 7eff19ffb640 1 -- 172.21.15.88:0/1030890213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff1c0d1650 msgr2=0x7eff1c0b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:48.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.969+0000 7eff19ffb640 1 --2- 172.21.15.88:0/1030890213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff1c0d1650 0x7eff1c0b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.969+0000 7eff19ffb640 1 -- 172.21.15.88:0/1030890213 >> v2:172.21.15.154:3300/0 conn(0x7eff1c0d3310 msgr2=0x7eff1c0d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:48.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.969+0000 7eff19ffb640 1 --2- 172.21.15.88:0/1030890213 >> v2:172.21.15.154:3300/0 conn(0x7eff1c0d3310 0x7eff1c0d1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.969+0000 7eff19ffb640 1 -- 172.21.15.88:0/1030890213 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff1c0d3b50 con 0x7eff1c15e660 2024-09-16T01:34:48.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.969+0000 7eff19ffb640 1 -- 172.21.15.88:0/1030890213 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7eff0c001670 con 0x7eff1c15e660 2024-09-16T01:34:48.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.969+0000 7eff19ffb640 1 -- 172.21.15.88:0/1030890213 >> v1:172.21.15.88:6789/0 conn(0x7eff1c15e660 legacy=0x7eff1c0cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:48.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.969+0000 7eff19ffb640 1 --2- 172.21.15.88:0/1030890213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff040039b0 0x7eff04003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:48.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.969+0000 7eff19ffb640 1 --2- 172.21.15.88:0/1030890213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff04004800 0x7eff04004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:48.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.969+0000 7eff19ffb640 1 --2- 172.21.15.88:0/1030890213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff1c0d1650 0x7eff040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:48.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.969+0000 7eff19ffb640 1 -- 172.21.15.88:0/1030890213 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c0d1a90 con 0x7eff04004800 2024-09-16T01:34:48.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.969+0000 7eff19ffb640 1 -- 172.21.15.88:0/1030890213 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c0d3870 con 0x7eff040039b0 2024-09-16T01:34:48.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.969+0000 7eff19ffb640 1 -- 172.21.15.88:0/1030890213 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7eff0400b9c0 con 0x7eff1c0d1650 2024-09-16T01:34:48.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.969+0000 7eff1a7fc640 1 --2- 172.21.15.88:0/1030890213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff04004800 0x7eff04004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:48.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.969+0000 7eff1affd640 1 --2- 172.21.15.88:0/1030890213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff040039b0 0x7eff04003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:48.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.969+0000 7eff1b7fe640 1 --2- 172.21.15.88:0/1030890213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff1c0d1650 0x7eff040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:48.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.970+0000 7eff1a7fc640 1 -- 172.21.15.88:0/1030890213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff1c0d1650 msgr2=0x7eff040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:48.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.970+0000 7eff1a7fc640 1 --2- 172.21.15.88:0/1030890213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff1c0d1650 0x7eff040051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.970+0000 7eff1a7fc640 1 -- 172.21.15.88:0/1030890213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff040039b0 msgr2=0x7eff04003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:48.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.970+0000 7eff1a7fc640 1 --2- 172.21.15.88:0/1030890213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff040039b0 0x7eff04003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.970+0000 7eff1a7fc640 1 -- 172.21.15.88:0/1030890213 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff04003520 con 0x7eff04004800 2024-09-16T01:34:48.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.970+0000 7eff1affd640 1 --2- 172.21.15.88:0/1030890213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff040039b0 0x7eff04003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:48.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.970+0000 7eff1b7fe640 1 --2- 172.21.15.88:0/1030890213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff1c0d1650 0x7eff040051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:48.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.970+0000 7eff1a7fc640 1 --2- 172.21.15.88:0/1030890213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff04004800 0x7eff04004c80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7eff10009ac0 tx=0x7eff04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3b16e42bcb9f479 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:48.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.970+0000 7eff19ffb640 1 -- 172.21.15.88:0/1030890213 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff10036070 con 0x7eff04004800 2024-09-16T01:34:48.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.970+0000 7eff19ffb640 1 -- 172.21.15.88:0/1030890213 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff10031400 con 0x7eff04004800 2024-09-16T01:34:48.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.970+0000 7eff19ffb640 1 -- 172.21.15.88:0/1030890213 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff10009690 con 0x7eff04004800 2024-09-16T01:34:48.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.971+0000 7eff21a53640 1 -- 172.21.15.88:0/1030890213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff04004800 msgr2=0x7eff04004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:48.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.971+0000 7eff21a53640 1 --2- 172.21.15.88:0/1030890213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff04004800 0x7eff04004c80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7eff10009ac0 tx=0x7eff04005830 comp rx=0 tx=0).stop 2024-09-16T01:34:48.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.971+0000 7eff1b7fe640 1 -- 172.21.15.88:0/1030890213 reap_dead start 2024-09-16T01:34:48.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.971+0000 7eff21a53640 1 -- 172.21.15.88:0/1030890213 shutdown_connections 2024-09-16T01:34:48.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.971+0000 7eff21a53640 1 -- 172.21.15.88:0/1030890213 >> 172.21.15.88:0/1030890213 conn(0x7eff1c0b6410 msgr2=0x7eff1c0b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:48.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.971+0000 7eff21a53640 1 -- 172.21.15.88:0/1030890213 shutdown_connections 2024-09-16T01:34:48.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.971+0000 7eff21a53640 1 -- 172.21.15.88:0/1030890213 wait complete. 2024-09-16T01:34:48.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.972+0000 7eff21a53640 1 Processor -- start 2024-09-16T01:34:48.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.972+0000 7eff21a53640 1 -- start start 2024-09-16T01:34:48.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.973+0000 7eff21a53640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff04004800 0x7eff1c150d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:48.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.973+0000 7eff21a53640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff1c147780 0x7eff1c1578d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:48.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.973+0000 7eff21a53640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff1c151860 0x7eff1c151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:48.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.973+0000 7eff21a53640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c0d43f0 con 0x7eff1c151860 2024-09-16T01:34:48.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.973+0000 7eff21a53640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c0d4110 con 0x7eff1c147780 2024-09-16T01:34:48.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.973+0000 7eff21a53640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7eff1c0b7b70 con 0x7eff04004800 2024-09-16T01:34:48.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.973+0000 7eff1b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff1c151860 0x7eff1c151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:48.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.973+0000 7eff1b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff1c151860 0x7eff1c151ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54952/0 (socket says 172.21.15.88:54952) 2024-09-16T01:34:48.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.973+0000 7eff1b7fe640 1 -- 172.21.15.88:0/4263867896 learned_addr learned my addr 172.21.15.88:0/4263867896 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:48.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.973+0000 7eff1a7fc640 1 --2- 172.21.15.88:0/4263867896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff1c147780 0x7eff1c1578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:48.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.973+0000 7eff1affd640 1 --2- 172.21.15.88:0/4263867896 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff04004800 0x7eff1c150d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:48.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.974+0000 7eff1b7fe640 1 -- 172.21.15.88:0/4263867896 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff04004800 msgr2=0x7eff1c150d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:48.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.974+0000 7eff1b7fe640 1 --2- 172.21.15.88:0/4263867896 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff04004800 0x7eff1c150d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.974+0000 7eff1b7fe640 1 -- 172.21.15.88:0/4263867896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff1c147780 msgr2=0x7eff1c1578d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:48.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.974+0000 7eff1b7fe640 1 --2- 172.21.15.88:0/4263867896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff1c147780 0x7eff1c1578d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:48.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.974+0000 7eff1b7fe640 1 -- 172.21.15.88:0/4263867896 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff1c157e10 con 0x7eff1c151860 2024-09-16T01:34:48.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.974+0000 7eff1b7fe640 1 --2- 172.21.15.88:0/4263867896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff1c151860 0x7eff1c151ce0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7eff14009fd0 tx=0x7eff14010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:48.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.974+0000 7eff03fff640 1 -- 172.21.15.88:0/4263867896 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff1401a030 con 0x7eff1c151860 2024-09-16T01:34:48.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.974+0000 7eff03fff640 1 -- 172.21.15.88:0/4263867896 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff14010e30 con 0x7eff1c151860 2024-09-16T01:34:48.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.974+0000 7eff21a53640 1 -- 172.21.15.88:0/4263867896 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff1c1580b0 con 0x7eff1c151860 2024-09-16T01:34:48.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.974+0000 7eff03fff640 1 -- 172.21.15.88:0/4263867896 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff14003bd0 con 0x7eff1c151860 2024-09-16T01:34:48.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.974+0000 7eff21a53640 1 -- 172.21.15.88:0/4263867896 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff1c158670 con 0x7eff1c151860 2024-09-16T01:34:48.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.975+0000 7eff21a53640 1 -- 172.21.15.88:0/4263867896 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff1c147b60 con 0x7eff1c151860 2024-09-16T01:34:48.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.975+0000 7eff03fff640 1 -- 172.21.15.88:0/4263867896 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7eff14005d10 con 0x7eff1c151860 2024-09-16T01:34:48.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.976+0000 7eff03fff640 1 --2- 172.21.15.88:0/4263867896 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff0806c2b0 0x7eff0806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:48.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.976+0000 7eff03fff640 1 -- 172.21.15.88:0/4263867896 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7eff1408bc80 con 0x7eff1c151860 2024-09-16T01:34:48.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.976+0000 7eff1affd640 1 --2- 172.21.15.88:0/4263867896 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff0806c2b0 0x7eff0806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:48.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.976+0000 7eff1affd640 1 --2- 172.21.15.88:0/4263867896 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff0806c2b0 0x7eff0806e770 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7eff0c000f60 tx=0x7eff0c007040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:48.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:48.980+0000 7eff03fff640 1 -- 172.21.15.88:0/4263867896 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7eff14024070 con 0x7eff1c151860 2024-09-16T01:34:49.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.178+0000 7eff21a53640 1 -- 172.21.15.88:0/4263867896 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff1c0d1110 con 0x7eff1c151860 2024-09-16T01:34:49.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.179+0000 7eff03fff640 1 -- 172.21.15.88:0/4263867896 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5223 (secure 0 0 0) 0x7eff14059e30 con 0x7eff1c151860 2024-09-16T01:34:49.180 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:49.180 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":6,"btime":"2024-09-16T01:34:24:186280+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/1620784263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1620784263},{"type":"v1","addr":"172.21.15.154:6834","nonce":1620784263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3407296593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":3407296593},{"type":"v1","addr":"172.21.15.154:6835","nonce":3407296593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4293,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/4058543372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4058543372},{"type":"v1","addr":"172.21.15.88:6835","nonce":4058543372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:34:21.508480+0000","modified":"2024-09-16T01:34:23.515443+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6837/3712834803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":3712834803},{"type":"v1","addr":"172.21.15.88:6837","nonce":3712834803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4296,"qdb_cluster":[4296]},"id":1}]} 2024-09-16T01:34:49.181 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 6 2024-09-16T01:34:49.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.181+0000 7eff21a53640 1 -- 172.21.15.88:0/4263867896 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff0806c2b0 msgr2=0x7eff0806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:49.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.181+0000 7eff21a53640 1 --2- 172.21.15.88:0/4263867896 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff0806c2b0 0x7eff0806e770 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7eff0c000f60 tx=0x7eff0c007040 comp rx=0 tx=0).stop 2024-09-16T01:34:49.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.181+0000 7eff21a53640 1 -- 172.21.15.88:0/4263867896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff1c151860 msgr2=0x7eff1c151ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:49.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.181+0000 7eff21a53640 1 --2- 172.21.15.88:0/4263867896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff1c151860 0x7eff1c151ce0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7eff14009fd0 tx=0x7eff14010590 comp rx=0 tx=0).stop 2024-09-16T01:34:49.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.181+0000 7eff21a53640 1 -- 172.21.15.88:0/4263867896 shutdown_connections 2024-09-16T01:34:49.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.181+0000 7eff21a53640 1 --2- 172.21.15.88:0/4263867896 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff0806c2b0 0x7eff0806e770 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.181+0000 7eff21a53640 1 --2- 172.21.15.88:0/4263867896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff1c151860 0x7eff1c151ce0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.181+0000 7eff21a53640 1 --2- 172.21.15.88:0/4263867896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff1c147780 0x7eff1c1578d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.181+0000 7eff21a53640 1 --2- 172.21.15.88:0/4263867896 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff04004800 0x7eff1c150d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.181+0000 7eff21a53640 1 -- 172.21.15.88:0/4263867896 >> 172.21.15.88:0/4263867896 conn(0x7eff1c0b6410 msgr2=0x7eff1c15d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:49.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.181+0000 7eff21a53640 1 -- 172.21.15.88:0/4263867896 shutdown_connections 2024-09-16T01:34:49.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.181+0000 7eff21a53640 1 -- 172.21.15.88:0/4263867896 wait complete. 2024-09-16T01:34:49.193 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:34:49.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.291+0000 7fefb1ed2640 1 Processor -- start 2024-09-16T01:34:49.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.291+0000 7fefb1ed2640 1 -- start start 2024-09-16T01:34:49.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.291+0000 7fefb1ed2640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefac0728e0 0x7fefac15ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:49.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.291+0000 7fefb1ed2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefac15fcc0 con 0x7fefac06e5c0 2024-09-16T01:34:49.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.291+0000 7fefb1ed2640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefac15f700 con 0x7fefac15f260 2024-09-16T01:34:49.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.291+0000 7fefb1ed2640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fefac160380 con 0x7fefac0728e0 2024-09-16T01:34:49.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.292+0000 7fefab7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fefac06e5c0 0x7fefac06e9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54106/0 (socket says 172.21.15.88:54106) 2024-09-16T01:34:49.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.292+0000 7fefab7fe640 1 -- 172.21.15.88:0/4248820043 learned_addr learned my addr 172.21.15.88:0/4248820043 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:49.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.292+0000 7fefaa7fc640 1 -- 172.21.15.88:0/4248820043 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 734667528 0 0) 0x7fefac15fcc0 con 0x7fefac06e5c0 2024-09-16T01:34:49.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.292+0000 7fefaa7fc640 1 -- 172.21.15.88:0/4248820043 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef94003520 con 0x7fefac06e5c0 2024-09-16T01:34:49.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.292+0000 7fefaaffd640 1 --2- 172.21.15.88:0/4248820043 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefac0728e0 0x7fefac15ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:49.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.292+0000 7fefaa7fc640 1 -- 172.21.15.88:0/4248820043 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3175282660 0 0) 0x7fef94003520 con 0x7fefac06e5c0 2024-09-16T01:34:49.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.292+0000 7fefaa7fc640 1 -- 172.21.15.88:0/4248820043 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefac0728e0 msgr2=0x7fefac15ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:49.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.292+0000 7fefaa7fc640 1 --2- 172.21.15.88:0/4248820043 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefac0728e0 0x7fefac15ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.292+0000 7fefaa7fc640 1 -- 172.21.15.88:0/4248820043 >> v1:172.21.15.154:6789/0 conn(0x7fefac15f260 legacy=0x7fefac160810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:49.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.292+0000 7fefaa7fc640 1 -- 172.21.15.88:0/4248820043 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefac15f9e0 con 0x7fefac06e5c0 2024-09-16T01:34:49.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.292+0000 7fefaaffd640 1 --2- 172.21.15.88:0/4248820043 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefac0728e0 0x7fefac15ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:49.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.292+0000 7fefaa7fc640 1 -- 172.21.15.88:0/4248820043 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fef9c001670 con 0x7fefac06e5c0 2024-09-16T01:34:49.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.292+0000 7fefaa7fc640 1 -- 172.21.15.88:0/4248820043 >> v1:172.21.15.88:6789/0 conn(0x7fefac06e5c0 legacy=0x7fefac06e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:49.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.292+0000 7fefaa7fc640 1 --2- 172.21.15.88:0/4248820043 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefac0728e0 0x7fef940039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:49.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.292+0000 7fefaa7fc640 1 --2- 172.21.15.88:0/4248820043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef94003fd0 0x7fef94004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:49.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.292+0000 7fefaa7fc640 1 --2- 172.21.15.88:0/4248820043 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef94004d40 0x7fef940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:49.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.292+0000 7fefaa7fc640 1 -- 172.21.15.88:0/4248820043 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fefac160380 con 0x7fef94003fd0 2024-09-16T01:34:49.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.292+0000 7fefaa7fc640 1 -- 172.21.15.88:0/4248820043 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fef9400bbb0 con 0x7fef94004d40 2024-09-16T01:34:49.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.292+0000 7fefaa7fc640 1 -- 172.21.15.88:0/4248820043 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fef9400bdb0 con 0x7fefac0728e0 2024-09-16T01:34:49.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.292+0000 7fefaaffd640 1 --2- 172.21.15.88:0/4248820043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef94003fd0 0x7fef94004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:49.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.292+0000 7fefab7fe640 1 --2- 172.21.15.88:0/4248820043 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefac0728e0 0x7fef940039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:49.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.292+0000 7fefaaffd640 1 -- 172.21.15.88:0/4248820043 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefac0728e0 msgr2=0x7fef940039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:49.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.292+0000 7fefaaffd640 1 --2- 172.21.15.88:0/4248820043 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefac0728e0 0x7fef940039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.293+0000 7fefaaffd640 1 -- 172.21.15.88:0/4248820043 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef94004d40 msgr2=0x7fef940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:49.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.293+0000 7fefaaffd640 1 --2- 172.21.15.88:0/4248820043 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef94004d40 0x7fef940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.293+0000 7fefaaffd640 1 -- 172.21.15.88:0/4248820043 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef94003520 con 0x7fef94003fd0 2024-09-16T01:34:49.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.293+0000 7fefaaffd640 1 --2- 172.21.15.88:0/4248820043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef94003fd0 0x7fef94004800 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fefac0743b0 tx=0x7fef94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f039e053120334 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:49.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.293+0000 7fefaa7fc640 1 -- 172.21.15.88:0/4248820043 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa0037070 con 0x7fef94003fd0 2024-09-16T01:34:49.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.293+0000 7fefaa7fc640 1 -- 172.21.15.88:0/4248820043 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefa002ac30 con 0x7fef94003fd0 2024-09-16T01:34:49.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.293+0000 7fefaa7fc640 1 -- 172.21.15.88:0/4248820043 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefa000f040 con 0x7fef94003fd0 2024-09-16T01:34:49.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.294+0000 7fefb1ed2640 1 -- 172.21.15.88:0/4248820043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef94003fd0 msgr2=0x7fef94004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:49.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.294+0000 7fefb1ed2640 1 --2- 172.21.15.88:0/4248820043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef94003fd0 0x7fef94004800 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fefac0743b0 tx=0x7fef94005830 comp rx=0 tx=0).stop 2024-09-16T01:34:49.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.294+0000 7fefabfff640 1 -- 172.21.15.88:0/4248820043 reap_dead start 2024-09-16T01:34:49.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.294+0000 7fefb1ed2640 1 -- 172.21.15.88:0/4248820043 shutdown_connections 2024-09-16T01:34:49.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.294+0000 7fefb1ed2640 1 -- 172.21.15.88:0/4248820043 >> 172.21.15.88:0/4248820043 conn(0x7fefac06e1e0 msgr2=0x7fefac0b67c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:49.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.294+0000 7fefb1ed2640 1 -- 172.21.15.88:0/4248820043 shutdown_connections 2024-09-16T01:34:49.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.294+0000 7fefb1ed2640 1 -- 172.21.15.88:0/4248820043 wait complete. 2024-09-16T01:34:49.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.295+0000 7fefb1ed2640 1 Processor -- start 2024-09-16T01:34:49.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.295+0000 7fefb1ed2640 1 -- start start 2024-09-16T01:34:49.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.296+0000 7fefb1ed2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef94003fd0 0x7fefac151850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:49.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.296+0000 7fefb1ed2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefac150330 0x7fefac153230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:49.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.296+0000 7fefb1ed2640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefac150c60 0x7fefac153770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:49.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.296+0000 7fefb1ed2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fefac15ed20 con 0x7fef94003fd0 2024-09-16T01:34:49.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.296+0000 7fefb1ed2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fefac15c670 con 0x7fefac150330 2024-09-16T01:34:49.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.296+0000 7fefb1ed2640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fefac15eea0 con 0x7fefac150c60 2024-09-16T01:34:49.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.296+0000 7fefab7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef94003fd0 0x7fefac151850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:49.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.296+0000 7fefab7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef94003fd0 0x7fefac151850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54964/0 (socket says 172.21.15.88:54964) 2024-09-16T01:34:49.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.296+0000 7fefab7fe640 1 -- 172.21.15.88:0/2637263227 learned_addr learned my addr 172.21.15.88:0/2637263227 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:49.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.296+0000 7fefabfff640 1 --2- 172.21.15.88:0/2637263227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefac150c60 0x7fefac153770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:49.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.296+0000 7fefaaffd640 1 --2- 172.21.15.88:0/2637263227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefac150330 0x7fefac153230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:49.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.296+0000 7fefab7fe640 1 -- 172.21.15.88:0/2637263227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefac150c60 msgr2=0x7fefac153770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:49.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.296+0000 7fefab7fe640 1 --2- 172.21.15.88:0/2637263227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefac150c60 0x7fefac153770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.296+0000 7fefab7fe640 1 -- 172.21.15.88:0/2637263227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefac150330 msgr2=0x7fefac153230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:49.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.296+0000 7fefab7fe640 1 --2- 172.21.15.88:0/2637263227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefac150330 0x7fefac153230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.296+0000 7fefab7fe640 1 -- 172.21.15.88:0/2637263227 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefac0cbd20 con 0x7fef94003fd0 2024-09-16T01:34:49.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.296+0000 7fefab7fe640 1 --2- 172.21.15.88:0/2637263227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef94003fd0 0x7fefac151850 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fef9c0072d0 tx=0x7fef9c007790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:49.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.297+0000 7fef93fff640 1 -- 172.21.15.88:0/2637263227 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c01a070 con 0x7fef94003fd0 2024-09-16T01:34:49.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.297+0000 7fef93fff640 1 -- 172.21.15.88:0/2637263227 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef9c00f9f0 con 0x7fef94003fd0 2024-09-16T01:34:49.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.297+0000 7fefb1ed2640 1 -- 172.21.15.88:0/2637263227 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefac151d90 con 0x7fef94003fd0 2024-09-16T01:34:49.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.297+0000 7fef93fff640 1 -- 172.21.15.88:0/2637263227 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c00fde0 con 0x7fef94003fd0 2024-09-16T01:34:49.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.297+0000 7fefb1ed2640 1 -- 172.21.15.88:0/2637263227 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefac156dc0 con 0x7fef94003fd0 2024-09-16T01:34:49.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.298+0000 7fefb1ed2640 1 -- 172.21.15.88:0/2637263227 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefac06e9a0 con 0x7fef94003fd0 2024-09-16T01:34:49.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.298+0000 7fef93fff640 1 -- 172.21.15.88:0/2637263227 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fef9c01d070 con 0x7fef94003fd0 2024-09-16T01:34:49.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.298+0000 7fef93fff640 1 --2- 172.21.15.88:0/2637263227 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fef9806c2b0 0x7fef9806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:49.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.299+0000 7fef93fff640 1 -- 172.21.15.88:0/2637263227 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fef9c08ade0 con 0x7fef94003fd0 2024-09-16T01:34:49.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.299+0000 7fefaaffd640 1 --2- 172.21.15.88:0/2637263227 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fef9806c2b0 0x7fef9806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:49.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.299+0000 7fefaaffd640 1 --2- 172.21.15.88:0/2637263227 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fef9806c2b0 0x7fef9806e770 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fefa0034000 tx=0x7fefa002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:49.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.303+0000 7fef93fff640 1 -- 172.21.15.88:0/2637263227 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fef9c0595e0 con 0x7fef94003fd0 2024-09-16T01:34:49.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.504+0000 7fefb1ed2640 1 -- 172.21.15.88:0/2637263227 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fefac161980 con 0x7fef94003fd0 2024-09-16T01:34:49.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.504+0000 7fef93fff640 1 -- 172.21.15.88:0/2637263227 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5223 (secure 0 0 0) 0x7fef9c058ee0 con 0x7fef94003fd0 2024-09-16T01:34:49.506 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:49.506 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":6,"btime":"2024-09-16T01:34:24:186280+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/1620784263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1620784263},{"type":"v1","addr":"172.21.15.154:6834","nonce":1620784263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3407296593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":3407296593},{"type":"v1","addr":"172.21.15.154:6835","nonce":3407296593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4293,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/4058543372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4058543372},{"type":"v1","addr":"172.21.15.88:6835","nonce":4058543372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:34:21.508480+0000","modified":"2024-09-16T01:34:23.515443+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6837/3712834803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":3712834803},{"type":"v1","addr":"172.21.15.88:6837","nonce":3712834803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4296,"qdb_cluster":[4296]},"id":1}]} 2024-09-16T01:34:49.506 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 6 2024-09-16T01:34:49.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.506+0000 7fefb1ed2640 1 -- 172.21.15.88:0/2637263227 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fef9806c2b0 msgr2=0x7fef9806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:49.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.506+0000 7fefb1ed2640 1 --2- 172.21.15.88:0/2637263227 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fef9806c2b0 0x7fef9806e770 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fefa0034000 tx=0x7fefa002e000 comp rx=0 tx=0).stop 2024-09-16T01:34:49.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.507+0000 7fefb1ed2640 1 -- 172.21.15.88:0/2637263227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef94003fd0 msgr2=0x7fefac151850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:49.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.507+0000 7fefb1ed2640 1 --2- 172.21.15.88:0/2637263227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef94003fd0 0x7fefac151850 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fef9c0072d0 tx=0x7fef9c007790 comp rx=0 tx=0).stop 2024-09-16T01:34:49.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.507+0000 7fefb1ed2640 1 -- 172.21.15.88:0/2637263227 shutdown_connections 2024-09-16T01:34:49.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.507+0000 7fefb1ed2640 1 --2- 172.21.15.88:0/2637263227 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fef9806c2b0 0x7fef9806e770 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.507+0000 7fefb1ed2640 1 --2- 172.21.15.88:0/2637263227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefac150c60 0x7fefac153770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.507+0000 7fefb1ed2640 1 --2- 172.21.15.88:0/2637263227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefac150330 0x7fefac153230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.507+0000 7fefb1ed2640 1 --2- 172.21.15.88:0/2637263227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef94003fd0 0x7fefac151850 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.507+0000 7fefb1ed2640 1 -- 172.21.15.88:0/2637263227 >> 172.21.15.88:0/2637263227 conn(0x7fefac06e1e0 msgr2=0x7fefac15da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:49.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.507+0000 7fefb1ed2640 1 -- 172.21.15.88:0/2637263227 shutdown_connections 2024-09-16T01:34:49.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.507+0000 7fefb1ed2640 1 -- 172.21.15.88:0/2637263227 wait complete. 2024-09-16T01:34:49.519 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:34:49.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.615+0000 7f602ee45640 1 Processor -- start 2024-09-16T01:34:49.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.615+0000 7f602ee45640 1 -- start start 2024-09-16T01:34:49.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.616+0000 7f602ee45640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f602806bc10 0x7f602806bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:49.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.616+0000 7f602ee45640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f602815c710 0x7f6028072050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:49.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.616+0000 7f602ee45640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f602806c530 con 0x7f6028072590 2024-09-16T01:34:49.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.616+0000 7f602ee45640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6028072a30 con 0x7f602815c710 2024-09-16T01:34:49.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.616+0000 7f602ee45640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6028072d10 con 0x7f602806bc10 2024-09-16T01:34:49.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.616+0000 7f601ffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f602815c710 0x7f6028072050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:49.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.616+0000 7f602d3e2640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6028072590 0x7f60280d01e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54120/0 (socket says 172.21.15.88:54120) 2024-09-16T01:34:49.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.616+0000 7f602d3e2640 1 -- 172.21.15.88:0/1217469783 learned_addr learned my addr 172.21.15.88:0/1217469783 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:49.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.616+0000 7f601ffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f602815c710 0x7f6028072050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54972/0 (socket says 172.21.15.88:54972) 2024-09-16T01:34:49.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.616+0000 7f602cbe1640 1 --2- 172.21.15.88:0/1217469783 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f602806bc10 0x7f602806bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:49.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.616+0000 7f601ffff640 1 -- 172.21.15.88:0/1217469783 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f602806bc10 msgr2=0x7f602806bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:49.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.616+0000 7f601ffff640 1 --2- 172.21.15.88:0/1217469783 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f602806bc10 0x7f602806bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.616+0000 7f601ffff640 1 -- 172.21.15.88:0/1217469783 >> v1:172.21.15.88:6789/0 conn(0x7f6028072590 legacy=0x7f60280d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:49.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.616+0000 7f601f7fe640 1 -- 172.21.15.88:0/1217469783 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3544648782 0 0) 0x7f602806c530 con 0x7f6028072590 2024-09-16T01:34:49.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.616+0000 7f601ffff640 1 -- 172.21.15.88:0/1217469783 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6028072ff0 con 0x7f602815c710 2024-09-16T01:34:49.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.617+0000 7f601ffff640 1 --2- 172.21.15.88:0/1217469783 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f602815c710 0x7f6028072050 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f60180097a0 tx=0x7f601802e590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc61dc0d07e7112e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:49.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.617+0000 7f601f7fe640 1 -- 172.21.15.88:0/1217469783 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601803a070 con 0x7f602815c710 2024-09-16T01:34:49.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.617+0000 7f601f7fe640 1 -- 172.21.15.88:0/1217469783 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6018033710 con 0x7f602815c710 2024-09-16T01:34:49.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.617+0000 7f601f7fe640 1 -- 172.21.15.88:0/1217469783 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6018033ae0 con 0x7f602815c710 2024-09-16T01:34:49.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.618+0000 7f602ee45640 1 -- 172.21.15.88:0/1217469783 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f602815c710 msgr2=0x7f6028072050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:49.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.618+0000 7f602ee45640 1 --2- 172.21.15.88:0/1217469783 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f602815c710 0x7f6028072050 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f60180097a0 tx=0x7f601802e590 comp rx=0 tx=0).stop 2024-09-16T01:34:49.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.618+0000 7f602ee45640 1 -- 172.21.15.88:0/1217469783 shutdown_connections 2024-09-16T01:34:49.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.618+0000 7f602ee45640 1 --2- 172.21.15.88:0/1217469783 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f602815c710 0x7f6028072050 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.618+0000 7f602ee45640 1 --2- 172.21.15.88:0/1217469783 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f602806bc10 0x7f602806bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.618+0000 7f602ee45640 1 -- 172.21.15.88:0/1217469783 >> 172.21.15.88:0/1217469783 conn(0x7f60280d14d0 msgr2=0x7f60280d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:49.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.618+0000 7f602ee45640 1 -- 172.21.15.88:0/1217469783 shutdown_connections 2024-09-16T01:34:49.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.618+0000 7f602ee45640 1 -- 172.21.15.88:0/1217469783 wait complete. 2024-09-16T01:34:49.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.619+0000 7f602ee45640 1 Processor -- start 2024-09-16T01:34:49.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.619+0000 7f602ee45640 1 -- start start 2024-09-16T01:34:49.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.620+0000 7f602ee45640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f602806bc10 0x7f60280c7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:49.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.620+0000 7f602ee45640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6028072590 0x7f602815e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:49.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.620+0000 7f602ee45640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f602815c710 0x7f60280cf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:49.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.620+0000 7f602ee45640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f602806f850 con 0x7f602806bc10 2024-09-16T01:34:49.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.620+0000 7f602ee45640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f60280735b0 con 0x7f6028072590 2024-09-16T01:34:49.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.620+0000 7f602ee45640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f602806f9d0 con 0x7f602815c710 2024-09-16T01:34:49.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.620+0000 7f602cbe1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f602806bc10 0x7f60280c7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:49.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.620+0000 7f602cbe1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f602806bc10 0x7f60280c7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54980/0 (socket says 172.21.15.88:54980) 2024-09-16T01:34:49.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.620+0000 7f602cbe1640 1 -- 172.21.15.88:0/2389720362 learned_addr learned my addr 172.21.15.88:0/2389720362 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:49.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.620+0000 7f602d3e2640 1 --2- 172.21.15.88:0/2389720362 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f602815c710 0x7f60280cf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:49.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.620+0000 7f601ffff640 1 --2- 172.21.15.88:0/2389720362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6028072590 0x7f602815e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:49.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.620+0000 7f602cbe1640 1 -- 172.21.15.88:0/2389720362 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f602815c710 msgr2=0x7f60280cf5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:49.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.620+0000 7f602cbe1640 1 --2- 172.21.15.88:0/2389720362 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f602815c710 0x7f60280cf5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.620+0000 7f602cbe1640 1 -- 172.21.15.88:0/2389720362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6028072590 msgr2=0x7f602815e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:49.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.620+0000 7f602cbe1640 1 --2- 172.21.15.88:0/2389720362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6028072590 0x7f602815e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.620+0000 7f602cbe1640 1 -- 172.21.15.88:0/2389720362 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f602815e5f0 con 0x7f602806bc10 2024-09-16T01:34:49.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.620+0000 7f602cbe1640 1 --2- 172.21.15.88:0/2389720362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f602806bc10 0x7f60280c7680 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f602400d800 tx=0x7f602400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:49.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.621+0000 7f601d7fa640 1 -- 172.21.15.88:0/2389720362 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f602400b6c0 con 0x7f602806bc10 2024-09-16T01:34:49.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.621+0000 7f601d7fa640 1 -- 172.21.15.88:0/2389720362 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6024019070 con 0x7f602806bc10 2024-09-16T01:34:49.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.621+0000 7f601d7fa640 1 -- 172.21.15.88:0/2389720362 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f602400ba80 con 0x7f602806bc10 2024-09-16T01:34:49.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.621+0000 7f602ee45640 1 -- 172.21.15.88:0/2389720362 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60281f02e0 con 0x7f602806bc10 2024-09-16T01:34:49.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.621+0000 7f602ee45640 1 -- 172.21.15.88:0/2389720362 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60281f0880 con 0x7f602806bc10 2024-09-16T01:34:49.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.622+0000 7f602ee45640 1 -- 172.21.15.88:0/2389720362 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6028072030 con 0x7f602806bc10 2024-09-16T01:34:49.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.622+0000 7f601d7fa640 1 -- 172.21.15.88:0/2389720362 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f602400bc20 con 0x7f602806bc10 2024-09-16T01:34:49.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.622+0000 7f601d7fa640 1 --2- 172.21.15.88:0/2389720362 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f601406c2b0 0x7f601406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:49.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.623+0000 7f601d7fa640 1 -- 172.21.15.88:0/2389720362 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f602408b870 con 0x7f602806bc10 2024-09-16T01:34:49.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.623+0000 7f601ffff640 1 --2- 172.21.15.88:0/2389720362 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f601406c2b0 0x7f601406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:49.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.623+0000 7f601ffff640 1 --2- 172.21.15.88:0/2389720362 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f601406c2b0 0x7f601406e770 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6018037000 tx=0x7f6018008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:49.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.627+0000 7f601d7fa640 1 -- 172.21.15.88:0/2389720362 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f602815e5f0 con 0x7f602806bc10 2024-09-16T01:34:49.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.792+0000 7f602ee45640 1 -- 172.21.15.88:0/2389720362 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6028157c60 con 0x7f602806bc10 2024-09-16T01:34:49.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.794+0000 7f601d7fa640 1 -- 172.21.15.88:0/2389720362 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16641 (secure 0 0 0) 0x7f602405a070 con 0x7f602806bc10 2024-09-16T01:34:49.795 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:49.795 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:34:23.181812+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:34:19.324549+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:34:20.494051+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:34:49.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.796+0000 7f602ee45640 1 -- 172.21.15.88:0/2389720362 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f601406c2b0 msgr2=0x7f601406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:49.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.796+0000 7f602ee45640 1 --2- 172.21.15.88:0/2389720362 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f601406c2b0 0x7f601406e770 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6018037000 tx=0x7f6018008030 comp rx=0 tx=0).stop 2024-09-16T01:34:49.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.796+0000 7f602ee45640 1 -- 172.21.15.88:0/2389720362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f602806bc10 msgr2=0x7f60280c7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:49.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.796+0000 7f602ee45640 1 --2- 172.21.15.88:0/2389720362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f602806bc10 0x7f60280c7680 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f602400d800 tx=0x7f602400dcc0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.796+0000 7f602ee45640 1 -- 172.21.15.88:0/2389720362 shutdown_connections 2024-09-16T01:34:49.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.796+0000 7f602ee45640 1 --2- 172.21.15.88:0/2389720362 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f601406c2b0 0x7f601406e770 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.796+0000 7f602ee45640 1 --2- 172.21.15.88:0/2389720362 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f602815c710 0x7f60280cf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.796+0000 7f602ee45640 1 --2- 172.21.15.88:0/2389720362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6028072590 0x7f602815e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.796+0000 7f602ee45640 1 --2- 172.21.15.88:0/2389720362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f602806bc10 0x7f60280c7680 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.796+0000 7f602ee45640 1 -- 172.21.15.88:0/2389720362 >> 172.21.15.88:0/2389720362 conn(0x7f60280d14d0 msgr2=0x7f60280b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:49.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.796+0000 7f602ee45640 1 -- 172.21.15.88:0/2389720362 shutdown_connections 2024-09-16T01:34:49.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.796+0000 7f602ee45640 1 -- 172.21.15.88:0/2389720362 wait complete. 2024-09-16T01:34:49.809 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-16T01:34:49.809 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:34:49.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.905+0000 7f30bf26e640 1 Processor -- start 2024-09-16T01:34:49.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.905+0000 7f30bf26e640 1 -- start start 2024-09-16T01:34:49.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.906+0000 7f30bf26e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f30b80d0fe0 0x7f30b806cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:49.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.906+0000 7f30bf26e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30b806d260 0x7f30b80b5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:49.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.906+0000 7f30bf26e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30b815f130 con 0x7f30b815b460 2024-09-16T01:34:49.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.906+0000 7f30bf26e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f30b815eee0 con 0x7f30b80d0fe0 2024-09-16T01:34:49.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.906+0000 7f30bf26e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f30b815f410 con 0x7f30b806d260 2024-09-16T01:34:49.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.907+0000 7f30bd80b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30b806d260 0x7f30b80b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:49.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.907+0000 7f30bc809640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f30b80d0fe0 0x7f30b806cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:49.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.907+0000 7f30bd80b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30b806d260 0x7f30b80b5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:38050/0 (socket says 172.21.15.88:38050) 2024-09-16T01:34:49.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.907+0000 7f30bd80b640 1 -- 172.21.15.88:0/4225207180 learned_addr learned my addr 172.21.15.88:0/4225207180 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:49.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.907+0000 7f30bd80b640 1 -- 172.21.15.88:0/4225207180 >> v2:172.21.15.154:3300/0 conn(0x7f30b80d0fe0 msgr2=0x7f30b806cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:49.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.907+0000 7f30bd80b640 1 --2- 172.21.15.88:0/4225207180 >> v2:172.21.15.154:3300/0 conn(0x7f30b80d0fe0 0x7f30b806cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.907+0000 7f30bd80b640 1 -- 172.21.15.88:0/4225207180 >> v1:172.21.15.154:6789/0 conn(0x7f30b815b460 legacy=0x7f30b80b44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:49.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.907+0000 7f30bd80b640 1 -- 172.21.15.88:0/4225207180 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30b815f6f0 con 0x7f30b806d260 2024-09-16T01:34:49.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.908+0000 7f30bd80b640 1 --2- 172.21.15.88:0/4225207180 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30b806d260 0x7f30b80b5080 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f30b000d150 tx=0x7f30b002c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e402d1a3da1affc7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:49.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.908+0000 7f30affff640 1 -- 172.21.15.88:0/4225207180 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b0009600 con 0x7f30b806d260 2024-09-16T01:34:49.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.908+0000 7f30affff640 1 -- 172.21.15.88:0/4225207180 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30b0037070 con 0x7f30b806d260 2024-09-16T01:34:49.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.908+0000 7f30affff640 1 -- 172.21.15.88:0/4225207180 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b00099c0 con 0x7f30b806d260 2024-09-16T01:34:49.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.908+0000 7f30bf26e640 1 -- 172.21.15.88:0/4225207180 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30b806d260 msgr2=0x7f30b80b5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:49.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.908+0000 7f30bf26e640 1 --2- 172.21.15.88:0/4225207180 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30b806d260 0x7f30b80b5080 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f30b000d150 tx=0x7f30b002c930 comp rx=0 tx=0).stop 2024-09-16T01:34:49.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.908+0000 7f30bf26e640 1 -- 172.21.15.88:0/4225207180 shutdown_connections 2024-09-16T01:34:49.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.908+0000 7f30bf26e640 1 --2- 172.21.15.88:0/4225207180 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30b806d260 0x7f30b80b5080 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.908+0000 7f30bf26e640 1 --2- 172.21.15.88:0/4225207180 >> v2:172.21.15.154:3300/0 conn(0x7f30b80d0fe0 0x7f30b806cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.908+0000 7f30bf26e640 1 -- 172.21.15.88:0/4225207180 >> 172.21.15.88:0/4225207180 conn(0x7f30b80b6ec0 msgr2=0x7f30b80b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:49.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.909+0000 7f30bf26e640 1 -- 172.21.15.88:0/4225207180 shutdown_connections 2024-09-16T01:34:49.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.909+0000 7f30bf26e640 1 -- 172.21.15.88:0/4225207180 wait complete. 2024-09-16T01:34:49.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.910+0000 7f30bf26e640 1 Processor -- start 2024-09-16T01:34:49.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.910+0000 7f30bf26e640 1 -- start start 2024-09-16T01:34:49.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.910+0000 7f30bf26e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30b806d260 0x7f30b814ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:49.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.910+0000 7f30bf26e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30b80d0fe0 0x7f30b815e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:49.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.910+0000 7f30bf26e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30b815b460 0x7f30b81509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:49.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.910+0000 7f30bf26e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f30b80b5740 con 0x7f30b806d260 2024-09-16T01:34:49.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.910+0000 7f30bf26e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f30b80b55c0 con 0x7f30b80d0fe0 2024-09-16T01:34:49.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.910+0000 7f30bf26e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f30b80b58c0 con 0x7f30b815b460 2024-09-16T01:34:49.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.911+0000 7f30bd00a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30b806d260 0x7f30b814ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:49.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.911+0000 7f30bd00a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30b806d260 0x7f30b814ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54996/0 (socket says 172.21.15.88:54996) 2024-09-16T01:34:49.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.911+0000 7f30bd00a640 1 -- 172.21.15.88:0/3258600033 learned_addr learned my addr 172.21.15.88:0/3258600033 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:49.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.911+0000 7f30bd80b640 1 --2- 172.21.15.88:0/3258600033 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30b815b460 0x7f30b81509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:49.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.911+0000 7f30bc809640 1 --2- 172.21.15.88:0/3258600033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30b80d0fe0 0x7f30b815e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:49.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.911+0000 7f30bd00a640 1 -- 172.21.15.88:0/3258600033 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30b815b460 msgr2=0x7f30b81509e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:49.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.911+0000 7f30bd00a640 1 --2- 172.21.15.88:0/3258600033 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30b815b460 0x7f30b81509e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.911+0000 7f30bd00a640 1 -- 172.21.15.88:0/3258600033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30b80d0fe0 msgr2=0x7f30b815e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:49.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.911+0000 7f30bd00a640 1 --2- 172.21.15.88:0/3258600033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30b80d0fe0 0x7f30b815e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:49.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.911+0000 7f30bd00a640 1 -- 172.21.15.88:0/3258600033 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30b815cfe0 con 0x7f30b806d260 2024-09-16T01:34:49.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.911+0000 7f30bd00a640 1 --2- 172.21.15.88:0/3258600033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30b806d260 0x7f30b814ef30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f30b400c3c0 tx=0x7f30b400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:49.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.911+0000 7f30adffb640 1 -- 172.21.15.88:0/3258600033 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b40052d0 con 0x7f30b806d260 2024-09-16T01:34:49.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.911+0000 7f30adffb640 1 -- 172.21.15.88:0/3258600033 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30b4005490 con 0x7f30b806d260 2024-09-16T01:34:49.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.911+0000 7f30adffb640 1 -- 172.21.15.88:0/3258600033 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b4007260 con 0x7f30b806d260 2024-09-16T01:34:49.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.911+0000 7f30bf26e640 1 -- 172.21.15.88:0/3258600033 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30b815d2c0 con 0x7f30b806d260 2024-09-16T01:34:49.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.911+0000 7f30bf26e640 1 -- 172.21.15.88:0/3258600033 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30b815d880 con 0x7f30b806d260 2024-09-16T01:34:49.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.913+0000 7f30bf26e640 1 -- 172.21.15.88:0/3258600033 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30b80b5080 con 0x7f30b806d260 2024-09-16T01:34:49.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.913+0000 7f30adffb640 1 -- 172.21.15.88:0/3258600033 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f30b4007400 con 0x7f30b806d260 2024-09-16T01:34:49.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.913+0000 7f30adffb640 1 --2- 172.21.15.88:0/3258600033 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f309c06c2b0 0x7f309c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:49.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.913+0000 7f30adffb640 1 -- 172.21.15.88:0/3258600033 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f30b408a860 con 0x7f30b806d260 2024-09-16T01:34:49.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.913+0000 7f30bc809640 1 --2- 172.21.15.88:0/3258600033 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f309c06c2b0 0x7f309c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:49.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.914+0000 7f30bc809640 1 --2- 172.21.15.88:0/3258600033 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f309c06c2b0 0x7f309c06e770 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f30b81592e0 tx=0x7f30a8009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:49.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:49.917+0000 7f30adffb640 1 -- 172.21.15.88:0/3258600033 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f30b4058980 con 0x7f30b806d260 2024-09-16T01:34:50.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.113+0000 7f30bf26e640 1 -- 172.21.15.88:0/3258600033 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30b80b4c20 con 0x7f30b806d260 2024-09-16T01:34:50.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.114+0000 7f30adffb640 1 -- 172.21.15.88:0/3258600033 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5223 (secure 0 0 0) 0x7f30b408a070 con 0x7f30b806d260 2024-09-16T01:34:50.116 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:50.116 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":6,"btime":"2024-09-16T01:34:24:186280+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/1620784263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1620784263},{"type":"v1","addr":"172.21.15.154:6834","nonce":1620784263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3407296593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":3407296593},{"type":"v1","addr":"172.21.15.154:6835","nonce":3407296593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4293,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/4058543372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4058543372},{"type":"v1","addr":"172.21.15.88:6835","nonce":4058543372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:34:21.508480+0000","modified":"2024-09-16T01:34:23.515443+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6837/3712834803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":3712834803},{"type":"v1","addr":"172.21.15.88:6837","nonce":3712834803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4296,"qdb_cluster":[4296]},"id":1}]} 2024-09-16T01:34:50.116 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 6 2024-09-16T01:34:50.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.116+0000 7f30bf26e640 1 -- 172.21.15.88:0/3258600033 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f309c06c2b0 msgr2=0x7f309c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:50.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.116+0000 7f30bf26e640 1 --2- 172.21.15.88:0/3258600033 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f309c06c2b0 0x7f309c06e770 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f30b81592e0 tx=0x7f30a8009190 comp rx=0 tx=0).stop 2024-09-16T01:34:50.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.116+0000 7f30bf26e640 1 -- 172.21.15.88:0/3258600033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30b806d260 msgr2=0x7f30b814ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:50.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.116+0000 7f30bf26e640 1 --2- 172.21.15.88:0/3258600033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30b806d260 0x7f30b814ef30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f30b400c3c0 tx=0x7f30b400c880 comp rx=0 tx=0).stop 2024-09-16T01:34:50.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.116+0000 7f30bf26e640 1 -- 172.21.15.88:0/3258600033 shutdown_connections 2024-09-16T01:34:50.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.116+0000 7f30bf26e640 1 --2- 172.21.15.88:0/3258600033 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f309c06c2b0 0x7f309c06e770 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.116+0000 7f30bf26e640 1 --2- 172.21.15.88:0/3258600033 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f30b815b460 0x7f30b81509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.116+0000 7f30bf26e640 1 --2- 172.21.15.88:0/3258600033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30b80d0fe0 0x7f30b815e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.116+0000 7f30bf26e640 1 --2- 172.21.15.88:0/3258600033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30b806d260 0x7f30b814ef30 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.116+0000 7f30bf26e640 1 -- 172.21.15.88:0/3258600033 >> 172.21.15.88:0/3258600033 conn(0x7f30b80b6ec0 msgr2=0x7f30b8072760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:50.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.116+0000 7f30bf26e640 1 -- 172.21.15.88:0/3258600033 shutdown_connections 2024-09-16T01:34:50.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.116+0000 7f30bf26e640 1 -- 172.21.15.88:0/3258600033 wait complete. 2024-09-16T01:34:50.128 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:34:50.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.224+0000 7fb017b8d640 1 Processor -- start 2024-09-16T01:34:50.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.224+0000 7fb017b8d640 1 -- start start 2024-09-16T01:34:50.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.225+0000 7fb017b8d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0100b4920 0x7fb01015b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:50.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.225+0000 7fb017b8d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb0100cd780 0x7fb0100cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:50.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.225+0000 7fb017b8d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0100ce4f0 con 0x7fb0100b4d00 2024-09-16T01:34:50.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.225+0000 7fb017b8d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb0100cf100 con 0x7fb0100cd780 2024-09-16T01:34:50.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.225+0000 7fb017b8d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb0100ce7d0 con 0x7fb0100b4920 2024-09-16T01:34:50.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.225+0000 7fb01612a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb0100cd780 0x7fb0100cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:50.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.225+0000 7fb015128640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb0100b4d00 0x7fb0100b5140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54122/0 (socket says 172.21.15.88:54122) 2024-09-16T01:34:50.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.225+0000 7fb01612a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb0100cd780 0x7fb0100cdbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55004/0 (socket says 172.21.15.88:55004) 2024-09-16T01:34:50.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.225+0000 7fb015128640 1 -- 172.21.15.88:0/383917458 learned_addr learned my addr 172.21.15.88:0/383917458 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:50.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.225+0000 7fb015929640 1 --2- 172.21.15.88:0/383917458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0100b4920 0x7fb01015b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:50.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.225+0000 7fb01612a640 1 -- 172.21.15.88:0/383917458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0100b4920 msgr2=0x7fb01015b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:50.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.225+0000 7fb01612a640 1 --2- 172.21.15.88:0/383917458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0100b4920 0x7fb01015b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.225+0000 7fb01612a640 1 -- 172.21.15.88:0/383917458 >> v1:172.21.15.88:6789/0 conn(0x7fb0100b4d00 legacy=0x7fb0100b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:50.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.225+0000 7fb01612a640 1 -- 172.21.15.88:0/383917458 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0100ceab0 con 0x7fb0100cd780 2024-09-16T01:34:50.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.225+0000 7fb014927640 1 -- 172.21.15.88:0/383917458 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3855641459 0 0) 0x7fb0100ce4f0 con 0x7fb0100b4d00 2024-09-16T01:34:50.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.226+0000 7fb01612a640 1 --2- 172.21.15.88:0/383917458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0100cd780 0x7fb0100cdbf0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb008009d90 tx=0x7fb00802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf16fea520f76f9b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:50.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.226+0000 7fb014927640 1 -- 172.21.15.88:0/383917458 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb008007740 con 0x7fb0100cd780 2024-09-16T01:34:50.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.226+0000 7fb014927640 1 -- 172.21.15.88:0/383917458 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0080078e0 con 0x7fb0100cd780 2024-09-16T01:34:50.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.226+0000 7fb014927640 1 -- 172.21.15.88:0/383917458 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb008007cb0 con 0x7fb0100cd780 2024-09-16T01:34:50.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.227+0000 7fb017b8d640 1 -- 172.21.15.88:0/383917458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0100cd780 msgr2=0x7fb0100cdbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:50.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.227+0000 7fb017b8d640 1 --2- 172.21.15.88:0/383917458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0100cd780 0x7fb0100cdbf0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb008009d90 tx=0x7fb00802a660 comp rx=0 tx=0).stop 2024-09-16T01:34:50.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.227+0000 7fb017b8d640 1 -- 172.21.15.88:0/383917458 shutdown_connections 2024-09-16T01:34:50.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.227+0000 7fb017b8d640 1 --2- 172.21.15.88:0/383917458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0100cd780 0x7fb0100cdbf0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.227+0000 7fb017b8d640 1 --2- 172.21.15.88:0/383917458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0100b4920 0x7fb01015b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.227+0000 7fb017b8d640 1 -- 172.21.15.88:0/383917458 >> 172.21.15.88:0/383917458 conn(0x7fb01006bc10 msgr2=0x7fb01006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:50.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.227+0000 7fb017b8d640 1 -- 172.21.15.88:0/383917458 shutdown_connections 2024-09-16T01:34:50.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.227+0000 7fb017b8d640 1 -- 172.21.15.88:0/383917458 wait complete. 2024-09-16T01:34:50.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.228+0000 7fb017b8d640 1 Processor -- start 2024-09-16T01:34:50.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.228+0000 7fb017b8d640 1 -- start start 2024-09-16T01:34:50.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.228+0000 7fb017b8d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0100b4920 0x7fb0100cc300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:50.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.228+0000 7fb017b8d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0100b4d00 0x7fb010134610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:50.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.229+0000 7fb017b8d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0100cd780 0x7fb01014f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:50.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.229+0000 7fb017b8d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb0100ce130 con 0x7fb0100b4920 2024-09-16T01:34:50.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.229+0000 7fb017b8d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb01015bd10 con 0x7fb0100cd780 2024-09-16T01:34:50.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.229+0000 7fb017b8d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb0100ce2b0 con 0x7fb0100b4d00 2024-09-16T01:34:50.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.229+0000 7fb015929640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0100b4920 0x7fb0100cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:50.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.229+0000 7fb015929640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0100b4920 0x7fb0100cc300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55020/0 (socket says 172.21.15.88:55020) 2024-09-16T01:34:50.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.229+0000 7fb015929640 1 -- 172.21.15.88:0/1606189442 learned_addr learned my addr 172.21.15.88:0/1606189442 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:50.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.229+0000 7fb015128640 1 --2- 172.21.15.88:0/1606189442 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0100b4d00 0x7fb010134610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:50.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.229+0000 7fb01612a640 1 --2- 172.21.15.88:0/1606189442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0100cd780 0x7fb01014f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:50.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.229+0000 7fb015929640 1 -- 172.21.15.88:0/1606189442 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0100b4d00 msgr2=0x7fb010134610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:50.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.229+0000 7fb015929640 1 --2- 172.21.15.88:0/1606189442 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0100b4d00 0x7fb010134610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.229+0000 7fb015929640 1 -- 172.21.15.88:0/1606189442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0100cd780 msgr2=0x7fb01014f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:50.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.229+0000 7fb015929640 1 --2- 172.21.15.88:0/1606189442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0100cd780 0x7fb01014f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.229+0000 7fb015929640 1 -- 172.21.15.88:0/1606189442 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb010134f80 con 0x7fb0100b4920 2024-09-16T01:34:50.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.229+0000 7fb01612a640 1 --2- 172.21.15.88:0/1606189442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0100cd780 0x7fb01014f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:50.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.229+0000 7fb015929640 1 --2- 172.21.15.88:0/1606189442 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0100b4920 0x7fb0100cc300 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb00c00e9b0 tx=0x7fb00c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:50.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.229+0000 7fb0067fc640 1 -- 172.21.15.88:0/1606189442 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c017070 con 0x7fb0100b4920 2024-09-16T01:34:50.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.229+0000 7fb0067fc640 1 -- 172.21.15.88:0/1606189442 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb00c00c9a0 con 0x7fb0100b4920 2024-09-16T01:34:50.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.229+0000 7fb017b8d640 1 -- 172.21.15.88:0/1606189442 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb010135260 con 0x7fb0100b4920 2024-09-16T01:34:50.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.229+0000 7fb0067fc640 1 -- 172.21.15.88:0/1606189442 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c00cd90 con 0x7fb0100b4920 2024-09-16T01:34:50.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.229+0000 7fb017b8d640 1 -- 172.21.15.88:0/1606189442 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0100cc840 con 0x7fb0100b4920 2024-09-16T01:34:50.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.230+0000 7fb017b8d640 1 -- 172.21.15.88:0/1606189442 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0100b2730 con 0x7fb0100b4920 2024-09-16T01:34:50.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.231+0000 7fb0067fc640 1 -- 172.21.15.88:0/1606189442 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb00c01a070 con 0x7fb0100b4920 2024-09-16T01:34:50.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.231+0000 7fb0067fc640 1 --2- 172.21.15.88:0/1606189442 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faff406c2b0 0x7faff406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:50.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.231+0000 7fb0067fc640 1 -- 172.21.15.88:0/1606189442 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb00c08b720 con 0x7fb0100b4920 2024-09-16T01:34:50.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.231+0000 7fb015128640 1 --2- 172.21.15.88:0/1606189442 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faff406c2b0 0x7faff406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:50.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.231+0000 7fb015128640 1 --2- 172.21.15.88:0/1606189442 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faff406c2b0 0x7faff406e770 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb01015b060 tx=0x7fb000007b50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:50.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.235+0000 7fb0067fc640 1 -- 172.21.15.88:0/1606189442 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb0100b2730 con 0x7fb0100b4920 2024-09-16T01:34:50.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.435+0000 7fb017b8d640 1 -- 172.21.15.88:0/1606189442 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb010072b80 con 0x7fb0100b4920 2024-09-16T01:34:50.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.435+0000 7fb0067fc640 1 -- 172.21.15.88:0/1606189442 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5223 (secure 0 0 0) 0x7fb00c059f20 con 0x7fb0100b4920 2024-09-16T01:34:50.437 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:50.437 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":6,"btime":"2024-09-16T01:34:24:186280+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/1620784263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":1620784263},{"type":"v1","addr":"172.21.15.154:6834","nonce":1620784263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/3407296593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":3407296593},{"type":"v1","addr":"172.21.15.154:6835","nonce":3407296593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4293,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/4058543372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4058543372},{"type":"v1","addr":"172.21.15.88:6835","nonce":4058543372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:34:21.508480+0000","modified":"2024-09-16T01:34:23.515443+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4296":{"gid":4296,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6837/3712834803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":3712834803},{"type":"v1","addr":"172.21.15.88:6837","nonce":3712834803}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4296,"qdb_cluster":[4296]},"id":1}]} 2024-09-16T01:34:50.437 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 6 2024-09-16T01:34:50.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.437+0000 7fb017b8d640 1 -- 172.21.15.88:0/1606189442 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faff406c2b0 msgr2=0x7faff406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:50.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.437+0000 7fb017b8d640 1 --2- 172.21.15.88:0/1606189442 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faff406c2b0 0x7faff406e770 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb01015b060 tx=0x7fb000007b50 comp rx=0 tx=0).stop 2024-09-16T01:34:50.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.437+0000 7fb017b8d640 1 -- 172.21.15.88:0/1606189442 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0100b4920 msgr2=0x7fb0100cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:50.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.437+0000 7fb017b8d640 1 --2- 172.21.15.88:0/1606189442 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0100b4920 0x7fb0100cc300 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb00c00e9b0 tx=0x7fb00c00ee70 comp rx=0 tx=0).stop 2024-09-16T01:34:50.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.437+0000 7fb017b8d640 1 -- 172.21.15.88:0/1606189442 shutdown_connections 2024-09-16T01:34:50.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.437+0000 7fb017b8d640 1 --2- 172.21.15.88:0/1606189442 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faff406c2b0 0x7faff406e770 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.437+0000 7fb017b8d640 1 --2- 172.21.15.88:0/1606189442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0100cd780 0x7fb01014f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.437+0000 7fb017b8d640 1 --2- 172.21.15.88:0/1606189442 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0100b4d00 0x7fb010134610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.437+0000 7fb017b8d640 1 --2- 172.21.15.88:0/1606189442 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0100b4920 0x7fb0100cc300 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.437+0000 7fb017b8d640 1 -- 172.21.15.88:0/1606189442 >> 172.21.15.88:0/1606189442 conn(0x7fb01006bc10 msgr2=0x7fb010072000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:50.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.438+0000 7fb017b8d640 1 -- 172.21.15.88:0/1606189442 shutdown_connections 2024-09-16T01:34:50.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.438+0000 7fb017b8d640 1 -- 172.21.15.88:0/1606189442 wait complete. 2024-09-16T01:34:50.450 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:34:50.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.548+0000 7f7dbfeb7640 1 Processor -- start 2024-09-16T01:34:50.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.548+0000 7f7dbfeb7640 1 -- start start 2024-09-16T01:34:50.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.549+0000 7f7dbfeb7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7db80b6eb0 0x7f7db806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:50.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.549+0000 7f7dbfeb7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7db80d1b40 0x7f7db806cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:50.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.549+0000 7f7dbfeb7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7db80b7290 0x7f7db806d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:50.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.549+0000 7f7dbfeb7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7db80708d0 con 0x7f7db80b7290 2024-09-16T01:34:50.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.549+0000 7f7dbfeb7640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7db8070bb0 con 0x7f7db80d1b40 2024-09-16T01:34:50.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.549+0000 7f7dbfeb7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7db8070e90 con 0x7f7db80b6eb0 2024-09-16T01:34:50.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.549+0000 7f7dbe454640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7db80b7290 0x7f7db806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:50.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.549+0000 7f7dbe454640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7db80b7290 0x7f7db806d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55368/0 (socket says 172.21.15.88:55368) 2024-09-16T01:34:50.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.549+0000 7f7dbe454640 1 -- 172.21.15.88:0/266796640 learned_addr learned my addr 172.21.15.88:0/266796640 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:50.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.549+0000 7f7dbdc53640 1 --2- 172.21.15.88:0/266796640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7db80b6eb0 0x7f7db806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:50.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.549+0000 7f7dbe454640 1 -- 172.21.15.88:0/266796640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7db80b6eb0 msgr2=0x7f7db806fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:50.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.549+0000 7f7dbd452640 1 --2- 172.21.15.88:0/266796640 >> v2:172.21.15.154:3300/0 conn(0x7f7db80d1b40 0x7f7db806cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:50.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.549+0000 7f7dbe454640 1 --2- 172.21.15.88:0/266796640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7db80b6eb0 0x7f7db806fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.549+0000 7f7dbe454640 1 -- 172.21.15.88:0/266796640 >> v2:172.21.15.154:3300/0 conn(0x7f7db80d1b40 msgr2=0x7f7db806cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:50.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.549+0000 7f7dbe454640 1 --2- 172.21.15.88:0/266796640 >> v2:172.21.15.154:3300/0 conn(0x7f7db80d1b40 0x7f7db806cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.549+0000 7f7dbe454640 1 -- 172.21.15.88:0/266796640 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7db8071170 con 0x7f7db80b7290 2024-09-16T01:34:50.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.549+0000 7f7dbdc53640 1 --2- 172.21.15.88:0/266796640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7db80b6eb0 0x7f7db806fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:50.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.549+0000 7f7dbd452640 1 --2- 172.21.15.88:0/266796640 >> v2:172.21.15.154:3300/0 conn(0x7f7db80d1b40 0x7f7db806cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:50.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.550+0000 7f7dbe454640 1 --2- 172.21.15.88:0/266796640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7db80b7290 0x7f7db806d260 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7db8070dc0 tx=0x7f7db002a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1e968c8be1a9d52 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:50.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.550+0000 7f7dbcc51640 1 -- 172.21.15.88:0/266796640 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db0036070 con 0x7f7db80b7290 2024-09-16T01:34:50.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.550+0000 7f7dbcc51640 1 -- 172.21.15.88:0/266796640 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7db0031600 con 0x7f7db80b7290 2024-09-16T01:34:50.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.550+0000 7f7dbcc51640 1 -- 172.21.15.88:0/266796640 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db00319d0 con 0x7f7db80b7290 2024-09-16T01:34:50.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.551+0000 7f7dbfeb7640 1 -- 172.21.15.88:0/266796640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7db80b7290 msgr2=0x7f7db806d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:50.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.551+0000 7f7dbfeb7640 1 --2- 172.21.15.88:0/266796640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7db80b7290 0x7f7db806d260 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7db8070dc0 tx=0x7f7db002a5f0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.551+0000 7f7dbfeb7640 1 -- 172.21.15.88:0/266796640 shutdown_connections 2024-09-16T01:34:50.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.551+0000 7f7dbfeb7640 1 --2- 172.21.15.88:0/266796640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7db80b7290 0x7f7db806d260 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.551+0000 7f7dbfeb7640 1 --2- 172.21.15.88:0/266796640 >> v2:172.21.15.154:3300/0 conn(0x7f7db80d1b40 0x7f7db806cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.551+0000 7f7dbfeb7640 1 --2- 172.21.15.88:0/266796640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7db80b6eb0 0x7f7db806fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.551+0000 7f7dbfeb7640 1 -- 172.21.15.88:0/266796640 >> 172.21.15.88:0/266796640 conn(0x7f7db80d2450 msgr2=0x7f7db80d2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:50.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.551+0000 7f7dbfeb7640 1 -- 172.21.15.88:0/266796640 shutdown_connections 2024-09-16T01:34:50.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.551+0000 7f7dbfeb7640 1 -- 172.21.15.88:0/266796640 wait complete. 2024-09-16T01:34:50.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.552+0000 7f7dbfeb7640 1 Processor -- start 2024-09-16T01:34:50.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.552+0000 7f7dbfeb7640 1 -- start start 2024-09-16T01:34:50.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.553+0000 7f7dbfeb7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7db80b6eb0 0x7f7db8156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:50.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.553+0000 7f7dbfeb7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7db80b7290 0x7f7db81f7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:50.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.553+0000 7f7dbfeb7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7db80d1b40 0x7f7db81f8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:50.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.553+0000 7f7dbfeb7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7db8071970 con 0x7f7db80d1b40 2024-09-16T01:34:50.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.553+0000 7f7dbfeb7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7db8070240 con 0x7f7db80b7290 2024-09-16T01:34:50.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.553+0000 7f7dbfeb7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7db8071af0 con 0x7f7db80b6eb0 2024-09-16T01:34:50.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.553+0000 7f7dbe454640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7db80d1b40 0x7f7db81f8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:50.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.553+0000 7f7dbe454640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7db80d1b40 0x7f7db81f8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55372/0 (socket says 172.21.15.88:55372) 2024-09-16T01:34:50.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.553+0000 7f7dbe454640 1 -- 172.21.15.88:0/751030182 learned_addr learned my addr 172.21.15.88:0/751030182 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:50.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.553+0000 7f7dbdc53640 1 --2- 172.21.15.88:0/751030182 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7db80b6eb0 0x7f7db8156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:50.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.553+0000 7f7dbd452640 1 --2- 172.21.15.88:0/751030182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7db80b7290 0x7f7db81f7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:50.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.553+0000 7f7dbe454640 1 -- 172.21.15.88:0/751030182 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7db80b6eb0 msgr2=0x7f7db8156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:50.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.553+0000 7f7dbe454640 1 --2- 172.21.15.88:0/751030182 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7db80b6eb0 0x7f7db8156900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.553+0000 7f7dbe454640 1 -- 172.21.15.88:0/751030182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7db80b7290 msgr2=0x7f7db81f7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:50.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.553+0000 7f7dbe454640 1 --2- 172.21.15.88:0/751030182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7db80b7290 0x7f7db81f7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.553+0000 7f7dbe454640 1 -- 172.21.15.88:0/751030182 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7db815a010 con 0x7f7db80d1b40 2024-09-16T01:34:50.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.553+0000 7f7dbe454640 1 --2- 172.21.15.88:0/751030182 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7db80d1b40 0x7f7db81f8230 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7db0009f10 tx=0x7f7db0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:50.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.553+0000 7f7dae7fc640 1 -- 172.21.15.88:0/751030182 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db0036050 con 0x7f7db80d1b40 2024-09-16T01:34:50.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.553+0000 7f7dae7fc640 1 -- 172.21.15.88:0/751030182 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7db0042070 con 0x7f7db80d1b40 2024-09-16T01:34:50.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.553+0000 7f7dbfeb7640 1 -- 172.21.15.88:0/751030182 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7db8156e40 con 0x7f7db80d1b40 2024-09-16T01:34:50.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.553+0000 7f7dae7fc640 1 -- 172.21.15.88:0/751030182 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db0009510 con 0x7f7db80d1b40 2024-09-16T01:34:50.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.554+0000 7f7dbfeb7640 1 -- 172.21.15.88:0/751030182 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7db8159250 con 0x7f7db80d1b40 2024-09-16T01:34:50.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.555+0000 7f7dbfeb7640 1 -- 172.21.15.88:0/751030182 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7db806fd00 con 0x7f7db80d1b40 2024-09-16T01:34:50.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.555+0000 7f7dae7fc640 1 -- 172.21.15.88:0/751030182 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7db0007bd0 con 0x7f7db80d1b40 2024-09-16T01:34:50.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.555+0000 7f7dae7fc640 1 --2- 172.21.15.88:0/751030182 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7d9c06c2b0 0x7f7d9c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:50.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.555+0000 7f7dae7fc640 1 -- 172.21.15.88:0/751030182 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7db00ac290 con 0x7f7db80d1b40 2024-09-16T01:34:50.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.555+0000 7f7dbdc53640 1 --2- 172.21.15.88:0/751030182 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7d9c06c2b0 0x7f7d9c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:50.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.556+0000 7f7dbdc53640 1 --2- 172.21.15.88:0/751030182 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7d9c06c2b0 0x7f7d9c06e770 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7db4006fd0 tx=0x7f7db4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:50.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.560+0000 7f7dae7fc640 1 -- 172.21.15.88:0/751030182 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7db007a3b0 con 0x7f7db80d1b40 2024-09-16T01:34:50.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.726+0000 7f7dbfeb7640 1 -- 172.21.15.88:0/751030182 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7db80b19b0 con 0x7f7db80d1b40 2024-09-16T01:34:50.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.728+0000 7f7dae7fc640 1 -- 172.21.15.88:0/751030182 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16641 (secure 0 0 0) 0x7f7db007e6b0 con 0x7f7db80d1b40 2024-09-16T01:34:50.729 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:50.729 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:34:23.181812+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:34:19.324549+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-16T01:34:20.494051+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:34:50.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.730+0000 7f7dbfeb7640 1 -- 172.21.15.88:0/751030182 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7d9c06c2b0 msgr2=0x7f7d9c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:50.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.730+0000 7f7dbfeb7640 1 --2- 172.21.15.88:0/751030182 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7d9c06c2b0 0x7f7d9c06e770 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7db4006fd0 tx=0x7f7db4008040 comp rx=0 tx=0).stop 2024-09-16T01:34:50.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.730+0000 7f7dbfeb7640 1 -- 172.21.15.88:0/751030182 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7db80d1b40 msgr2=0x7f7db81f8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:50.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.730+0000 7f7dbfeb7640 1 --2- 172.21.15.88:0/751030182 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7db80d1b40 0x7f7db81f8230 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7db0009f10 tx=0x7f7db0004a40 comp rx=0 tx=0).stop 2024-09-16T01:34:50.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.730+0000 7f7dbfeb7640 1 -- 172.21.15.88:0/751030182 shutdown_connections 2024-09-16T01:34:50.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.730+0000 7f7dbfeb7640 1 --2- 172.21.15.88:0/751030182 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7d9c06c2b0 0x7f7d9c06e770 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.730+0000 7f7dbfeb7640 1 --2- 172.21.15.88:0/751030182 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7db80d1b40 0x7f7db81f8230 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.730+0000 7f7dbfeb7640 1 --2- 172.21.15.88:0/751030182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7db80b7290 0x7f7db81f7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.730+0000 7f7dbfeb7640 1 --2- 172.21.15.88:0/751030182 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7db80b6eb0 0x7f7db8156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.730+0000 7f7dbfeb7640 1 -- 172.21.15.88:0/751030182 >> 172.21.15.88:0/751030182 conn(0x7f7db80d2450 msgr2=0x7f7db815ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:50.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.730+0000 7f7dbfeb7640 1 -- 172.21.15.88:0/751030182 shutdown_connections 2024-09-16T01:34:50.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.730+0000 7f7dbfeb7640 1 -- 172.21.15.88:0/751030182 wait complete. 2024-09-16T01:34:50.742 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:34:50.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.840+0000 7f0e5131f640 1 Processor -- start 2024-09-16T01:34:50.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.840+0000 7f0e5131f640 1 -- start start 2024-09-16T01:34:50.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.841+0000 7f0e5131f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0e4c0cf1f0 0x7f0e4c0b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:50.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.841+0000 7f0e5131f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0e4c0b6720 0x7f0e4c0d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:50.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.841+0000 7f0e5131f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e4c0cf860 con 0x7f0e4c15e7a0 2024-09-16T01:34:50.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.841+0000 7f0e5131f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f0e4c0b6b90 con 0x7f0e4c0cf1f0 2024-09-16T01:34:50.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.841+0000 7f0e5131f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0e4c0d2ca0 con 0x7f0e4c0b6720 2024-09-16T01:34:50.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.841+0000 7f0e427fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0e4c0cf1f0 0x7f0e4c0b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:50.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.841+0000 7f0e427fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0e4c0cf1f0 0x7f0e4c0b61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55386/0 (socket says 172.21.15.88:55386) 2024-09-16T01:34:50.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.841+0000 7f0e427fc640 1 -- 172.21.15.88:0/2196982865 learned_addr learned my addr 172.21.15.88:0/2196982865 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:50.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.841+0000 7f0e4b7fe640 1 --2- 172.21.15.88:0/2196982865 >> v2:172.21.15.154:3300/0 conn(0x7f0e4c0b6720 0x7f0e4c0d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:50.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.841+0000 7f0e427fc640 1 -- 172.21.15.88:0/2196982865 >> v2:172.21.15.154:3300/0 conn(0x7f0e4c0b6720 msgr2=0x7f0e4c0d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:50.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.841+0000 7f0e427fc640 1 --2- 172.21.15.88:0/2196982865 >> v2:172.21.15.154:3300/0 conn(0x7f0e4c0b6720 0x7f0e4c0d3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.841+0000 7f0e427fc640 1 -- 172.21.15.88:0/2196982865 >> v1:172.21.15.154:6789/0 conn(0x7f0e4c15e7a0 legacy=0x7f0e4c06caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:50.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.841+0000 7f0e427fc640 1 -- 172.21.15.88:0/2196982865 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e4c0cfb40 con 0x7f0e4c0cf1f0 2024-09-16T01:34:50.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.841+0000 7f0e4b7fe640 1 --2- 172.21.15.88:0/2196982865 >> v2:172.21.15.154:3300/0 conn(0x7f0e4c0b6720 0x7f0e4c0d3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:50.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.842+0000 7f0e427fc640 1 --2- 172.21.15.88:0/2196982865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e4c0cf1f0 0x7f0e4c0b61e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c08d930 tx=0x7f0e3402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18e16f9f8d493dc7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:50.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.842+0000 7f0e4a7fc640 1 -- 172.21.15.88:0/2196982865 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e34037070 con 0x7f0e4c0cf1f0 2024-09-16T01:34:50.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.842+0000 7f0e4a7fc640 1 -- 172.21.15.88:0/2196982865 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e34032440 con 0x7f0e4c0cf1f0 2024-09-16T01:34:50.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.842+0000 7f0e4a7fc640 1 -- 172.21.15.88:0/2196982865 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e34032810 con 0x7f0e4c0cf1f0 2024-09-16T01:34:50.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.842+0000 7f0e5131f640 1 -- 172.21.15.88:0/2196982865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e4c0cf1f0 msgr2=0x7f0e4c0b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:50.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.843+0000 7f0e5131f640 1 --2- 172.21.15.88:0/2196982865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e4c0cf1f0 0x7f0e4c0b61e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c08d930 tx=0x7f0e3402a1e0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.843+0000 7f0e5131f640 1 -- 172.21.15.88:0/2196982865 shutdown_connections 2024-09-16T01:34:50.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.843+0000 7f0e5131f640 1 --2- 172.21.15.88:0/2196982865 >> v2:172.21.15.154:3300/0 conn(0x7f0e4c0b6720 0x7f0e4c0d3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.843+0000 7f0e5131f640 1 --2- 172.21.15.88:0/2196982865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e4c0cf1f0 0x7f0e4c0b61e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.843+0000 7f0e5131f640 1 -- 172.21.15.88:0/2196982865 >> 172.21.15.88:0/2196982865 conn(0x7f0e4c0d11e0 msgr2=0x7f0e4c0d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:50.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.843+0000 7f0e5131f640 1 -- 172.21.15.88:0/2196982865 shutdown_connections 2024-09-16T01:34:50.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.843+0000 7f0e5131f640 1 -- 172.21.15.88:0/2196982865 wait complete. 2024-09-16T01:34:50.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.844+0000 7f0e5131f640 1 Processor -- start 2024-09-16T01:34:50.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.844+0000 7f0e5131f640 1 -- start start 2024-09-16T01:34:50.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.845+0000 7f0e5131f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e4c0b6720 0x7f0e4c157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:50.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.845+0000 7f0e5131f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e4c0cf1f0 0x7f0e4c0d4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:50.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.845+0000 7f0e5131f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e4c15e7a0 0x7f0e4c0d48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:50.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.845+0000 7f0e5131f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0e4c0b7a00 con 0x7f0e4c0b6720 2024-09-16T01:34:50.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.845+0000 7f0e5131f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0e4c0d2ef0 con 0x7f0e4c0cf1f0 2024-09-16T01:34:50.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.845+0000 7f0e5131f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0e4c15eb80 con 0x7f0e4c15e7a0 2024-09-16T01:34:50.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.845+0000 7f0e4affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e4c0b6720 0x7f0e4c157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:50.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.845+0000 7f0e4affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e4c0b6720 0x7f0e4c157d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55388/0 (socket says 172.21.15.88:55388) 2024-09-16T01:34:50.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.845+0000 7f0e4affd640 1 -- 172.21.15.88:0/2788456467 learned_addr learned my addr 172.21.15.88:0/2788456467 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:50.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.845+0000 7f0e4b7fe640 1 --2- 172.21.15.88:0/2788456467 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e4c15e7a0 0x7f0e4c0d48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:50.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.845+0000 7f0e427fc640 1 --2- 172.21.15.88:0/2788456467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e4c0cf1f0 0x7f0e4c0d4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:50.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.845+0000 7f0e4affd640 1 -- 172.21.15.88:0/2788456467 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e4c15e7a0 msgr2=0x7f0e4c0d48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:50.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.845+0000 7f0e4affd640 1 --2- 172.21.15.88:0/2788456467 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e4c15e7a0 0x7f0e4c0d48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.845+0000 7f0e4affd640 1 -- 172.21.15.88:0/2788456467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e4c0cf1f0 msgr2=0x7f0e4c0d4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:50.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.845+0000 7f0e4affd640 1 --2- 172.21.15.88:0/2788456467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e4c0cf1f0 0x7f0e4c0d4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:50.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.845+0000 7f0e4affd640 1 -- 172.21.15.88:0/2788456467 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e4c158270 con 0x7f0e4c0b6720 2024-09-16T01:34:50.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.845+0000 7f0e4affd640 1 --2- 172.21.15.88:0/2788456467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e4c0b6720 0x7f0e4c157d30 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c00c430 tx=0x7f0e3c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:50.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.845+0000 7f0e43fff640 1 -- 172.21.15.88:0/2788456467 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c0052d0 con 0x7f0e4c0b6720 2024-09-16T01:34:50.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.845+0000 7f0e43fff640 1 -- 172.21.15.88:0/2788456467 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e3c005490 con 0x7f0e4c0b6720 2024-09-16T01:34:50.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.845+0000 7f0e43fff640 1 -- 172.21.15.88:0/2788456467 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c007260 con 0x7f0e4c0b6720 2024-09-16T01:34:50.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.845+0000 7f0e5131f640 1 -- 172.21.15.88:0/2788456467 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e4c15efc0 con 0x7f0e4c0b6720 2024-09-16T01:34:50.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.846+0000 7f0e5131f640 1 -- 172.21.15.88:0/2788456467 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e4c15f540 con 0x7f0e4c0b6720 2024-09-16T01:34:50.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.846+0000 7f0e5131f640 1 -- 172.21.15.88:0/2788456467 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e4c0b61e0 con 0x7f0e4c0b6720 2024-09-16T01:34:50.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.847+0000 7f0e43fff640 1 -- 172.21.15.88:0/2788456467 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0e3c007400 con 0x7f0e4c0b6720 2024-09-16T01:34:50.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.847+0000 7f0e43fff640 1 --2- 172.21.15.88:0/2788456467 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0e2c06c2b0 0x7f0e2c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:50.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.847+0000 7f0e43fff640 1 -- 172.21.15.88:0/2788456467 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0e3c08a8a0 con 0x7f0e4c0b6720 2024-09-16T01:34:50.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.847+0000 7f0e427fc640 1 --2- 172.21.15.88:0/2788456467 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0e2c06c2b0 0x7f0e2c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:50.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.848+0000 7f0e427fc640 1 --2- 172.21.15.88:0/2788456467 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0e2c06c2b0 0x7f0e2c06e770 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0e34034000 tx=0x7f0e3402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:50.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:50.852+0000 7f0e43fff640 1 -- 172.21.15.88:0/2788456467 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0e4c158270 con 0x7f0e4c0b6720 2024-09-16T01:34:51.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.050+0000 7f0e5131f640 1 -- 172.21.15.88:0/2788456467 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f0e4c0d2a80 con 0x7f0e4c0b6720 2024-09-16T01:34:51.052 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:34:51.051+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:34:51.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.052+0000 7f0e43fff640 1 -- 172.21.15.88:0/2788456467 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v7) ==== 161+0+0 (secure 0 0 0) 0x7f0e3c0589c0 con 0x7f0e4c0b6720 2024-09-16T01:34:51.053 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:34:51.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.055+0000 7f0e5131f640 1 -- 172.21.15.88:0/2788456467 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0e2c06c2b0 msgr2=0x7f0e2c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:51.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.055+0000 7f0e5131f640 1 --2- 172.21.15.88:0/2788456467 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0e2c06c2b0 0x7f0e2c06e770 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0e34034000 tx=0x7f0e3402e000 comp rx=0 tx=0).stop 2024-09-16T01:34:51.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.055+0000 7f0e5131f640 1 -- 172.21.15.88:0/2788456467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e4c0b6720 msgr2=0x7f0e4c157d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:51.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.055+0000 7f0e5131f640 1 --2- 172.21.15.88:0/2788456467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e4c0b6720 0x7f0e4c157d30 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c00c430 tx=0x7f0e3c00c8f0 comp rx=0 tx=0).stop 2024-09-16T01:34:51.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.055+0000 7f0e5131f640 1 -- 172.21.15.88:0/2788456467 shutdown_connections 2024-09-16T01:34:51.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.055+0000 7f0e5131f640 1 --2- 172.21.15.88:0/2788456467 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0e2c06c2b0 0x7f0e2c06e770 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:51.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.055+0000 7f0e5131f640 1 --2- 172.21.15.88:0/2788456467 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e4c15e7a0 0x7f0e4c0d48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:51.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.055+0000 7f0e5131f640 1 --2- 172.21.15.88:0/2788456467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e4c0cf1f0 0x7f0e4c0d4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:51.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.055+0000 7f0e5131f640 1 --2- 172.21.15.88:0/2788456467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e4c0b6720 0x7f0e4c157d30 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:51.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.055+0000 7f0e5131f640 1 -- 172.21.15.88:0/2788456467 >> 172.21.15.88:0/2788456467 conn(0x7f0e4c0d11e0 msgr2=0x7f0e4c158810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:51.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.055+0000 7f0e5131f640 1 -- 172.21.15.88:0/2788456467 shutdown_connections 2024-09-16T01:34:51.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.055+0000 7f0e5131f640 1 -- 172.21.15.88:0/2788456467 wait complete. 2024-09-16T01:34:51.059 INFO:tasks.ceph.mds.a.smithi088.stderr: -6570> 2024-09-16T01:34:27.158+0000 7f6b013f6640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:34:51.068 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-16T01:34:51.089 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:34:51.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.168+0000 7f27319c4640 1 Processor -- start 2024-09-16T01:34:51.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.168+0000 7f27319c4640 1 -- start start 2024-09-16T01:34:51.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.169+0000 7f27319c4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f272c15e7a0 0x7f272c06caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:51.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.169+0000 7f27319c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f272c0cf1f0 0x7f272c0b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:51.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.169+0000 7f27319c4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f272c0cf860 con 0x7f272c0b6720 2024-09-16T01:34:51.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.169+0000 7f27319c4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f272c0d2ca0 con 0x7f272c15e7a0 2024-09-16T01:34:51.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.169+0000 7f27319c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f272c0cfb40 con 0x7f272c0cf1f0 2024-09-16T01:34:51.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.169+0000 7f272affd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f272c15e7a0 0x7f272c06caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:51.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.169+0000 7f272b7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f272c0b6720 0x7f272c1623c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33606/0 (socket says 172.21.15.88:33606) 2024-09-16T01:34:51.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.169+0000 7f272affd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f272c15e7a0 0x7f272c06caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55406/0 (socket says 172.21.15.88:55406) 2024-09-16T01:34:51.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.169+0000 7f272b7fe640 1 -- 172.21.15.88:0/1620793994 learned_addr learned my addr 172.21.15.88:0/1620793994 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:51.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.170+0000 7f272a7fc640 1 --2- 172.21.15.88:0/1620793994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f272c0cf1f0 0x7f272c0b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:51.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.170+0000 7f272affd640 1 -- 172.21.15.88:0/1620793994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f272c0cf1f0 msgr2=0x7f272c0b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:51.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.170+0000 7f272affd640 1 --2- 172.21.15.88:0/1620793994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f272c0cf1f0 0x7f272c0b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:51.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.170+0000 7f272affd640 1 -- 172.21.15.88:0/1620793994 >> v1:172.21.15.88:6789/0 conn(0x7f272c0b6720 legacy=0x7f272c1623c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:51.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.170+0000 7f272affd640 1 -- 172.21.15.88:0/1620793994 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f272c0cfe20 con 0x7f272c15e7a0 2024-09-16T01:34:51.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.170+0000 7f2729ffb640 1 -- 172.21.15.88:0/1620793994 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2152435467 0 0) 0x7f272c0cf860 con 0x7f272c0b6720 2024-09-16T01:34:51.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.170+0000 7f272affd640 1 --2- 172.21.15.88:0/1620793994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f272c15e7a0 0x7f272c06caa0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f271c0097a0 tx=0x7f271c02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f157c8b52f186d03 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:51.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.171+0000 7f2729ffb640 1 -- 172.21.15.88:0/1620793994 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f271c037070 con 0x7f272c15e7a0 2024-09-16T01:34:51.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.171+0000 7f2729ffb640 1 -- 172.21.15.88:0/1620793994 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f271c02ae50 con 0x7f272c15e7a0 2024-09-16T01:34:51.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.171+0000 7f2729ffb640 1 -- 172.21.15.88:0/1620793994 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f271c032660 con 0x7f272c15e7a0 2024-09-16T01:34:51.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.171+0000 7f27319c4640 1 -- 172.21.15.88:0/1620793994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f272c15e7a0 msgr2=0x7f272c06caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:51.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.171+0000 7f27319c4640 1 --2- 172.21.15.88:0/1620793994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f272c15e7a0 0x7f272c06caa0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f271c0097a0 tx=0x7f271c02a100 comp rx=0 tx=0).stop 2024-09-16T01:34:51.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.171+0000 7f27319c4640 1 -- 172.21.15.88:0/1620793994 shutdown_connections 2024-09-16T01:34:51.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.171+0000 7f27319c4640 1 --2- 172.21.15.88:0/1620793994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f272c0cf1f0 0x7f272c0b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:51.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.171+0000 7f27319c4640 1 --2- 172.21.15.88:0/1620793994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f272c15e7a0 0x7f272c06caa0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:51.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.171+0000 7f27319c4640 1 -- 172.21.15.88:0/1620793994 >> 172.21.15.88:0/1620793994 conn(0x7f272c0d11e0 msgr2=0x7f272c0d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:51.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.171+0000 7f27319c4640 1 -- 172.21.15.88:0/1620793994 shutdown_connections 2024-09-16T01:34:51.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.171+0000 7f27319c4640 1 -- 172.21.15.88:0/1620793994 wait complete. 2024-09-16T01:34:51.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.172+0000 7f27319c4640 1 Processor -- start 2024-09-16T01:34:51.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.173+0000 7f27319c4640 1 -- start start 2024-09-16T01:34:51.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.173+0000 7f27319c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f272c0b6720 0x7f272c14fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:51.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.173+0000 7f27319c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f272c0cf1f0 0x7f272c0caa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:51.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.173+0000 7f27319c4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f272c15e7a0 0x7f272c0b51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:51.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.173+0000 7f27319c4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f272c06cfe0 con 0x7f272c15e7a0 2024-09-16T01:34:51.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.173+0000 7f27319c4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f272c0b7a00 con 0x7f272c0b6720 2024-09-16T01:34:51.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.173+0000 7f27319c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f272c0d4420 con 0x7f272c0cf1f0 2024-09-16T01:34:51.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.173+0000 7f272b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f272c15e7a0 0x7f272c0b51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:51.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.173+0000 7f272b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f272c15e7a0 0x7f272c0b51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55420/0 (socket says 172.21.15.88:55420) 2024-09-16T01:34:51.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.173+0000 7f272b7fe640 1 -- 172.21.15.88:0/1533442741 learned_addr learned my addr 172.21.15.88:0/1533442741 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:51.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.174+0000 7f272b7fe640 1 -- 172.21.15.88:0/1533442741 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f272c0cf1f0 msgr2=0x7f272c0caa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:51.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.174+0000 7f272b7fe640 1 --2- 172.21.15.88:0/1533442741 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f272c0cf1f0 0x7f272c0caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:51.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.174+0000 7f272b7fe640 1 -- 172.21.15.88:0/1533442741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f272c0b6720 msgr2=0x7f272c14fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:51.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.174+0000 7f272b7fe640 1 --2- 172.21.15.88:0/1533442741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f272c0b6720 0x7f272c14fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:51.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.174+0000 7f272b7fe640 1 -- 172.21.15.88:0/1533442741 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f272c0c9ca0 con 0x7f272c15e7a0 2024-09-16T01:34:51.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.174+0000 7f272b7fe640 1 --2- 172.21.15.88:0/1533442741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f272c15e7a0 0x7f272c0b51f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f272400c7d0 tx=0x7f272400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:51.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.174+0000 7f2713fff640 1 -- 172.21.15.88:0/1533442741 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2724018070 con 0x7f272c15e7a0 2024-09-16T01:34:51.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.174+0000 7f2713fff640 1 -- 172.21.15.88:0/1533442741 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f272400a640 con 0x7f272c15e7a0 2024-09-16T01:34:51.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.174+0000 7f27319c4640 1 -- 172.21.15.88:0/1533442741 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f272c0c9f80 con 0x7f272c15e7a0 2024-09-16T01:34:51.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.174+0000 7f2713fff640 1 -- 172.21.15.88:0/1533442741 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f272400aa70 con 0x7f272c15e7a0 2024-09-16T01:34:51.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.174+0000 7f27319c4640 1 -- 172.21.15.88:0/1533442741 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f272c0ca540 con 0x7f272c15e7a0 2024-09-16T01:34:51.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.175+0000 7f27319c4640 1 -- 172.21.15.88:0/1533442741 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f272c15eb80 con 0x7f272c15e7a0 2024-09-16T01:34:51.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.175+0000 7f2713fff640 1 -- 172.21.15.88:0/1533442741 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f272401b070 con 0x7f272c15e7a0 2024-09-16T01:34:51.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.176+0000 7f2713fff640 1 --2- 172.21.15.88:0/1533442741 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f271806c2e0 0x7f271806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:51.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.176+0000 7f272affd640 1 --2- 172.21.15.88:0/1533442741 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f271806c2e0 0x7f271806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:51.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.176+0000 7f2713fff640 1 -- 172.21.15.88:0/1533442741 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5795+0+0 (secure 0 0 0) 0x7f272408a9f0 con 0x7f272c15e7a0 2024-09-16T01:34:51.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.176+0000 7f272affd640 1 --2- 172.21.15.88:0/1533442741 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f271806c2e0 0x7f271806e7a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f271c034000 tx=0x7f271c02c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:51.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.180+0000 7f2713fff640 1 -- 172.21.15.88:0/1533442741 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2724059190 con 0x7f272c15e7a0 2024-09-16T01:34:51.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:51.379+0000 7f27319c4640 1 -- 172.21.15.88:0/1533442741 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f272c0d2a80 con 0x7f272c15e7a0 2024-09-16T01:34:52.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.054+0000 7f2713fff640 1 -- 172.21.15.88:0/1533442741 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v8) ==== 104+0+0 (secure 0 0 0) 0x7f2724058a90 con 0x7f272c15e7a0 2024-09-16T01:34:52.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.056+0000 7f27319c4640 1 -- 172.21.15.88:0/1533442741 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f271806c2e0 msgr2=0x7f271806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:52.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.056+0000 7f27319c4640 1 --2- 172.21.15.88:0/1533442741 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f271806c2e0 0x7f271806e7a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f271c034000 tx=0x7f271c02c000 comp rx=0 tx=0).stop 2024-09-16T01:34:52.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.056+0000 7f27319c4640 1 -- 172.21.15.88:0/1533442741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f272c15e7a0 msgr2=0x7f272c0b51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:52.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.056+0000 7f27319c4640 1 --2- 172.21.15.88:0/1533442741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f272c15e7a0 0x7f272c0b51f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f272400c7d0 tx=0x7f272400cc90 comp rx=0 tx=0).stop 2024-09-16T01:34:52.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.056+0000 7f27319c4640 1 -- 172.21.15.88:0/1533442741 shutdown_connections 2024-09-16T01:34:52.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.056+0000 7f27319c4640 1 --2- 172.21.15.88:0/1533442741 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f271806c2e0 0x7f271806e7a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:52.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.056+0000 7f27319c4640 1 --2- 172.21.15.88:0/1533442741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f272c15e7a0 0x7f272c0b51f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:52.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.056+0000 7f27319c4640 1 --2- 172.21.15.88:0/1533442741 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f272c0cf1f0 0x7f272c0caa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:52.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.056+0000 7f27319c4640 1 --2- 172.21.15.88:0/1533442741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f272c0b6720 0x7f272c14fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:52.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.056+0000 7f27319c4640 1 -- 172.21.15.88:0/1533442741 >> 172.21.15.88:0/1533442741 conn(0x7f272c0d11e0 msgr2=0x7f272c15aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:52.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.056+0000 7f27319c4640 1 -- 172.21.15.88:0/1533442741 shutdown_connections 2024-09-16T01:34:52.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.056+0000 7f27319c4640 1 -- 172.21.15.88:0/1533442741 wait complete. 2024-09-16T01:34:52.070 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-16T01:34:52.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.215+0000 7f80e03a6640 1 Processor -- start 2024-09-16T01:34:52.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.216+0000 7f80e03a6640 1 -- start start 2024-09-16T01:34:52.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.216+0000 7f80e03a6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80d815e570 0x7f80d80b5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:52.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.217+0000 7f80e03a6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f80d806fb10 0x7f80d815e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:52.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.217+0000 7f80e03a6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80d815f020 con 0x7f80d80714a0 2024-09-16T01:34:52.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.217+0000 7f80e03a6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f80d815fc30 con 0x7f80d806fb10 2024-09-16T01:34:52.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.217+0000 7f80e03a6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f80d815f300 con 0x7f80d815e570 2024-09-16T01:34:52.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.217+0000 7f80dd941640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f80d80714a0 0x7f80d806f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33616/0 (socket says 172.21.15.88:33616) 2024-09-16T01:34:52.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.217+0000 7f80dd941640 1 -- 172.21.15.88:0/3731294600 learned_addr learned my addr 172.21.15.88:0/3731294600 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:52.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.217+0000 7f80de142640 1 --2- 172.21.15.88:0/3731294600 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80d815e570 0x7f80d80b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:52.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.217+0000 7f80de943640 1 --2- 172.21.15.88:0/3731294600 >> v2:172.21.15.154:3300/0 conn(0x7f80d806fb10 0x7f80d815e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:52.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.217+0000 7f80dd140640 1 -- 172.21.15.88:0/3731294600 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3148260388 0 0) 0x7f80d815f020 con 0x7f80d80714a0 2024-09-16T01:34:52.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.217+0000 7f80dd140640 1 -- 172.21.15.88:0/3731294600 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80c4003520 con 0x7f80d80714a0 2024-09-16T01:34:52.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.217+0000 7f80dd140640 1 -- 172.21.15.88:0/3731294600 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 386808829 0 0) 0x7f80c4003520 con 0x7f80d80714a0 2024-09-16T01:34:52.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.217+0000 7f80dd140640 1 -- 172.21.15.88:0/3731294600 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80d815e570 msgr2=0x7f80d80b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:52.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.217+0000 7f80dd140640 1 --2- 172.21.15.88:0/3731294600 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80d815e570 0x7f80d80b5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:52.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.217+0000 7f80dd140640 1 -- 172.21.15.88:0/3731294600 >> v2:172.21.15.154:3300/0 conn(0x7f80d806fb10 msgr2=0x7f80d815e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:52.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.217+0000 7f80dd140640 1 --2- 172.21.15.88:0/3731294600 >> v2:172.21.15.154:3300/0 conn(0x7f80d806fb10 0x7f80d815e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:52.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.217+0000 7f80dd140640 1 -- 172.21.15.88:0/3731294600 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80d815f5e0 con 0x7f80d80714a0 2024-09-16T01:34:52.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.217+0000 7f80dd140640 1 -- 172.21.15.88:0/3731294600 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f80c8001670 con 0x7f80d80714a0 2024-09-16T01:34:52.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.217+0000 7f80dd140640 1 -- 172.21.15.88:0/3731294600 >> v1:172.21.15.88:6789/0 conn(0x7f80d80714a0 legacy=0x7f80d806f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:52.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.217+0000 7f80dd140640 1 --2- 172.21.15.88:0/3731294600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80c40039b0 0x7f80c4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:52.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.217+0000 7f80dd140640 1 --2- 172.21.15.88:0/3731294600 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80d815e570 0x7f80c4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:52.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.217+0000 7f80dd140640 1 --2- 172.21.15.88:0/3731294600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80c4004d40 0x7f80c40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:52.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.217+0000 7f80dd140640 1 -- 172.21.15.88:0/3731294600 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f80d815fc30 con 0x7f80c4004d40 2024-09-16T01:34:52.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.217+0000 7f80dd140640 1 -- 172.21.15.88:0/3731294600 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f80d815f300 con 0x7f80c40039b0 2024-09-16T01:34:52.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.217+0000 7f80dd140640 1 -- 172.21.15.88:0/3731294600 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f80c400b9c0 con 0x7f80d815e570 2024-09-16T01:34:52.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.217+0000 7f80de943640 1 --2- 172.21.15.88:0/3731294600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80c4004d40 0x7f80c40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:52.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.218+0000 7f80de142640 1 --2- 172.21.15.88:0/3731294600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80c40039b0 0x7f80c4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:52.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.218+0000 7f80dd941640 1 --2- 172.21.15.88:0/3731294600 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80d815e570 0x7f80c4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:52.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.218+0000 7f80de943640 1 -- 172.21.15.88:0/3731294600 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80d815e570 msgr2=0x7f80c4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:52.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.218+0000 7f80de943640 1 --2- 172.21.15.88:0/3731294600 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80d815e570 0x7f80c4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:52.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.218+0000 7f80de943640 1 -- 172.21.15.88:0/3731294600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80c40039b0 msgr2=0x7f80c4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:52.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.218+0000 7f80de943640 1 --2- 172.21.15.88:0/3731294600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80c40039b0 0x7f80c4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:52.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.218+0000 7f80de943640 1 -- 172.21.15.88:0/3731294600 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80c4003520 con 0x7f80c4004d40 2024-09-16T01:34:52.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.218+0000 7f80de142640 1 --2- 172.21.15.88:0/3731294600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80c40039b0 0x7f80c4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:52.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.218+0000 7f80de943640 1 --2- 172.21.15.88:0/3731294600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80c4004d40 0x7f80c40051c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f80d0009fc0 tx=0x7f80c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=485a755434313718 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:52.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.219+0000 7f80dd140640 1 -- 172.21.15.88:0/3731294600 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d0009c10 con 0x7f80c4004d40 2024-09-16T01:34:52.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.219+0000 7f80dd140640 1 -- 172.21.15.88:0/3731294600 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80d0036070 con 0x7f80c4004d40 2024-09-16T01:34:52.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.219+0000 7f80dd140640 1 -- 172.21.15.88:0/3731294600 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d002d070 con 0x7f80c4004d40 2024-09-16T01:34:52.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.219+0000 7f80e03a6640 1 -- 172.21.15.88:0/3731294600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80c4004d40 msgr2=0x7f80c40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:52.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.219+0000 7f80e03a6640 1 --2- 172.21.15.88:0/3731294600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80c4004d40 0x7f80c40051c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f80d0009fc0 tx=0x7f80c4005830 comp rx=0 tx=0).stop 2024-09-16T01:34:52.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.219+0000 7f80de943640 1 -- 172.21.15.88:0/3731294600 reap_dead start 2024-09-16T01:34:52.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.219+0000 7f80e03a6640 1 -- 172.21.15.88:0/3731294600 shutdown_connections 2024-09-16T01:34:52.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.219+0000 7f80e03a6640 1 -- 172.21.15.88:0/3731294600 >> 172.21.15.88:0/3731294600 conn(0x7f80d80b4240 msgr2=0x7f80d80b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:52.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.219+0000 7f80e03a6640 1 -- 172.21.15.88:0/3731294600 shutdown_connections 2024-09-16T01:34:52.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.219+0000 7f80e03a6640 1 -- 172.21.15.88:0/3731294600 wait complete. 2024-09-16T01:34:52.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.220+0000 7f80e03a6640 1 Processor -- start 2024-09-16T01:34:52.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.220+0000 7f80e03a6640 1 -- start start 2024-09-16T01:34:52.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.221+0000 7f80e03a6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80c4004d40 0x7f80d80cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:52.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.221+0000 7f80e03a6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80d80d1610 0x7f80d8155050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:52.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.221+0000 7f80e03a6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80d80cf2f0 0x7f80d80cf770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:52.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.221+0000 7f80e03a6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f80d80b56c0 con 0x7f80d80d1610 2024-09-16T01:34:52.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.221+0000 7f80e03a6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f80d80b5540 con 0x7f80c4004d40 2024-09-16T01:34:52.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.221+0000 7f80e03a6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f80d80b5840 con 0x7f80d80cf2f0 2024-09-16T01:34:52.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.221+0000 7f80dd941640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80d80d1610 0x7f80d8155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:52.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.221+0000 7f80dd941640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80d80d1610 0x7f80d8155050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55442/0 (socket says 172.21.15.88:55442) 2024-09-16T01:34:52.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.221+0000 7f80dd941640 1 -- 172.21.15.88:0/270430656 learned_addr learned my addr 172.21.15.88:0/270430656 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:52.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.221+0000 7f80de142640 1 --2- 172.21.15.88:0/270430656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80c4004d40 0x7f80d80cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:52.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.221+0000 7f80de943640 1 --2- 172.21.15.88:0/270430656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80d80cf2f0 0x7f80d80cf770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:52.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.222+0000 7f80dd941640 1 -- 172.21.15.88:0/270430656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80d80cf2f0 msgr2=0x7f80d80cf770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:52.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.222+0000 7f80dd941640 1 --2- 172.21.15.88:0/270430656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80d80cf2f0 0x7f80d80cf770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:52.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.222+0000 7f80dd941640 1 -- 172.21.15.88:0/270430656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80c4004d40 msgr2=0x7f80d80cb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:52.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.222+0000 7f80dd941640 1 --2- 172.21.15.88:0/270430656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80c4004d40 0x7f80d80cb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:52.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.222+0000 7f80dd941640 1 -- 172.21.15.88:0/270430656 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80d80cbec0 con 0x7f80d80d1610 2024-09-16T01:34:52.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.222+0000 7f80dd941640 1 --2- 172.21.15.88:0/270430656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80d80d1610 0x7f80d8155050 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f80c8001200 tx=0x7f80c800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:52.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.222+0000 7f80ceffd640 1 -- 172.21.15.88:0/270430656 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c8019070 con 0x7f80d80d1610 2024-09-16T01:34:52.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.222+0000 7f80ceffd640 1 -- 172.21.15.88:0/270430656 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80c8005240 con 0x7f80d80d1610 2024-09-16T01:34:52.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.222+0000 7f80ceffd640 1 -- 172.21.15.88:0/270430656 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c8005630 con 0x7f80d80d1610 2024-09-16T01:34:52.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.222+0000 7f80e03a6640 1 -- 172.21.15.88:0/270430656 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80d80cc1a0 con 0x7f80d80d1610 2024-09-16T01:34:52.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.222+0000 7f80e03a6640 1 -- 172.21.15.88:0/270430656 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80d80cc760 con 0x7f80d80d1610 2024-09-16T01:34:52.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.223+0000 7f80e03a6640 1 -- 172.21.15.88:0/270430656 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80d815e950 con 0x7f80d80d1610 2024-09-16T01:34:52.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.223+0000 7f80ceffd640 1 -- 172.21.15.88:0/270430656 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f80c801c070 con 0x7f80d80d1610 2024-09-16T01:34:52.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.224+0000 7f80ceffd640 1 --2- 172.21.15.88:0/270430656 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f80bc06c2b0 0x7f80bc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:52.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.224+0000 7f80ceffd640 1 -- 172.21.15.88:0/270430656 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5795+0+0 (secure 0 0 0) 0x7f80c808ae00 con 0x7f80d80d1610 2024-09-16T01:34:52.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.224+0000 7f80de142640 1 --2- 172.21.15.88:0/270430656 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f80bc06c2b0 0x7f80bc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:52.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.224+0000 7f80de142640 1 --2- 172.21.15.88:0/270430656 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f80bc06c2b0 0x7f80bc06e770 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f80d4009990 tx=0x7f80d4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:52.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.228+0000 7f80ceffd640 1 -- 172.21.15.88:0/270430656 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f80c80595e0 con 0x7f80d80d1610 2024-09-16T01:34:52.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:52.395+0000 7f80e03a6640 1 -- 172.21.15.88:0/270430656 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f80d806eb60 con 0x7f80d80d1610 2024-09-16T01:34:53.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.281+0000 7f80ceffd640 1 -- 172.21.15.88:0/270430656 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v23) ==== 181+0+0 (secure 0 0 0) 0x7f80c8058ee0 con 0x7f80d80d1610 2024-09-16T01:34:53.283 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_metadata' removed 2024-09-16T01:34:53.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.284+0000 7f80e03a6640 1 -- 172.21.15.88:0/270430656 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f80bc06c2b0 msgr2=0x7f80bc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:53.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.284+0000 7f80e03a6640 1 --2- 172.21.15.88:0/270430656 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f80bc06c2b0 0x7f80bc06e770 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f80d4009990 tx=0x7f80d4008040 comp rx=0 tx=0).stop 2024-09-16T01:34:53.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.284+0000 7f80e03a6640 1 -- 172.21.15.88:0/270430656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80d80d1610 msgr2=0x7f80d8155050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:53.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.284+0000 7f80e03a6640 1 --2- 172.21.15.88:0/270430656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80d80d1610 0x7f80d8155050 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f80c8001200 tx=0x7f80c800ed30 comp rx=0 tx=0).stop 2024-09-16T01:34:53.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.285+0000 7f80e03a6640 1 -- 172.21.15.88:0/270430656 shutdown_connections 2024-09-16T01:34:53.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.285+0000 7f80e03a6640 1 --2- 172.21.15.88:0/270430656 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f80bc06c2b0 0x7f80bc06e770 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:53.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.285+0000 7f80e03a6640 1 --2- 172.21.15.88:0/270430656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80d80cf2f0 0x7f80d80cf770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:53.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.285+0000 7f80e03a6640 1 --2- 172.21.15.88:0/270430656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80d80d1610 0x7f80d8155050 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:53.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.285+0000 7f80e03a6640 1 --2- 172.21.15.88:0/270430656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80c4004d40 0x7f80d80cb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:53.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.285+0000 7f80e03a6640 1 -- 172.21.15.88:0/270430656 >> 172.21.15.88:0/270430656 conn(0x7f80d80b4240 msgr2=0x7f80d806dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:53.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.285+0000 7f80e03a6640 1 -- 172.21.15.88:0/270430656 shutdown_connections 2024-09-16T01:34:53.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.285+0000 7f80e03a6640 1 -- 172.21.15.88:0/270430656 wait complete. 2024-09-16T01:34:53.298 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-16T01:34:53.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.402+0000 7fed4aa5c640 1 Processor -- start 2024-09-16T01:34:53.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.402+0000 7fed4aa5c640 1 -- start start 2024-09-16T01:34:53.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.403+0000 7fed4aa5c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fed440ceda0 0x7fed44073490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:53.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.403+0000 7fed4aa5c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fed440cf180 0x7fed440cf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:53.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.403+0000 7fed4aa5c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed4406dc10 0x7fed4406e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:53.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.403+0000 7fed4aa5c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fed440722d0 con 0x7fed440cf180 2024-09-16T01:34:53.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.403+0000 7fed4aa5c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fed44072590 con 0x7fed440ceda0 2024-09-16T01:34:53.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.403+0000 7fed4aa5c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fed44072870 con 0x7fed4406dc10 2024-09-16T01:34:53.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.403+0000 7fed437fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fed440cf180 0x7fed440cf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:53.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.403+0000 7fed437fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fed440cf180 0x7fed440cf5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55448/0 (socket says 172.21.15.88:55448) 2024-09-16T01:34:53.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.403+0000 7fed437fe640 1 -- 172.21.15.88:0/3578775122 learned_addr learned my addr 172.21.15.88:0/3578775122 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:53.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.403+0000 7fed437fe640 1 -- 172.21.15.88:0/3578775122 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed4406dc10 msgr2=0x7fed4406e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:53.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.403+0000 7fed437fe640 1 --2- 172.21.15.88:0/3578775122 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed4406dc10 0x7fed4406e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:53.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.403+0000 7fed437fe640 1 -- 172.21.15.88:0/3578775122 >> v2:172.21.15.154:3300/0 conn(0x7fed440ceda0 msgr2=0x7fed44073490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:53.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.403+0000 7fed437fe640 1 --2- 172.21.15.88:0/3578775122 >> v2:172.21.15.154:3300/0 conn(0x7fed440ceda0 0x7fed44073490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:53.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.403+0000 7fed437fe640 1 -- 172.21.15.88:0/3578775122 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed44072b50 con 0x7fed440cf180 2024-09-16T01:34:53.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.404+0000 7fed437fe640 1 --2- 172.21.15.88:0/3578775122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed440cf180 0x7fed440cf5c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fed440b3b10 tx=0x7fed3802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e6ec90f3d1f274 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:53.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.404+0000 7fed42ffd640 1 -- 172.21.15.88:0/3578775122 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed38037070 con 0x7fed440cf180 2024-09-16T01:34:53.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.404+0000 7fed42ffd640 1 -- 172.21.15.88:0/3578775122 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed38032440 con 0x7fed440cf180 2024-09-16T01:34:53.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.404+0000 7fed42ffd640 1 -- 172.21.15.88:0/3578775122 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed38032810 con 0x7fed440cf180 2024-09-16T01:34:53.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.405+0000 7fed4aa5c640 1 -- 172.21.15.88:0/3578775122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed440cf180 msgr2=0x7fed440cf5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:53.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.405+0000 7fed4aa5c640 1 --2- 172.21.15.88:0/3578775122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed440cf180 0x7fed440cf5c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fed440b3b10 tx=0x7fed3802a1e0 comp rx=0 tx=0).stop 2024-09-16T01:34:53.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.405+0000 7fed4aa5c640 1 -- 172.21.15.88:0/3578775122 shutdown_connections 2024-09-16T01:34:53.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.405+0000 7fed4aa5c640 1 --2- 172.21.15.88:0/3578775122 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed4406dc10 0x7fed4406e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:53.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.405+0000 7fed4aa5c640 1 --2- 172.21.15.88:0/3578775122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed440cf180 0x7fed440cf5c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:53.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.405+0000 7fed4aa5c640 1 --2- 172.21.15.88:0/3578775122 >> v2:172.21.15.154:3300/0 conn(0x7fed440ceda0 0x7fed44073490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:53.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.405+0000 7fed4aa5c640 1 -- 172.21.15.88:0/3578775122 >> 172.21.15.88:0/3578775122 conn(0x7fed4406cb50 msgr2=0x7fed4406cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:53.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.405+0000 7fed4aa5c640 1 -- 172.21.15.88:0/3578775122 shutdown_connections 2024-09-16T01:34:53.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.405+0000 7fed4aa5c640 1 -- 172.21.15.88:0/3578775122 wait complete. 2024-09-16T01:34:53.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.406+0000 7fed4aa5c640 1 Processor -- start 2024-09-16T01:34:53.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.406+0000 7fed4aa5c640 1 -- start start 2024-09-16T01:34:53.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.407+0000 7fed4aa5c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed4406dc10 0x7fed4414c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:53.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.407+0000 7fed4aa5c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed440ceda0 0x7fed441e41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:53.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.407+0000 7fed4aa5c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed440cf180 0x7fed441e3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:53.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.407+0000 7fed4aa5c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fed440739d0 con 0x7fed440cf180 2024-09-16T01:34:53.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.407+0000 7fed4aa5c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fed440d0510 con 0x7fed4406dc10 2024-09-16T01:34:53.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.407+0000 7fed4aa5c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fed44073b50 con 0x7fed440ceda0 2024-09-16T01:34:53.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.407+0000 7fed48ff9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed440cf180 0x7fed441e3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:53.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.407+0000 7fed48ff9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed440cf180 0x7fed441e3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55458/0 (socket says 172.21.15.88:55458) 2024-09-16T01:34:53.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.407+0000 7fed48ff9640 1 -- 172.21.15.88:0/1312474804 learned_addr learned my addr 172.21.15.88:0/1312474804 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:53.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.407+0000 7fed437fe640 1 --2- 172.21.15.88:0/1312474804 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed440ceda0 0x7fed441e41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:53.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.407+0000 7fed43fff640 1 --2- 172.21.15.88:0/1312474804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed4406dc10 0x7fed4414c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:53.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.407+0000 7fed48ff9640 1 -- 172.21.15.88:0/1312474804 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed440ceda0 msgr2=0x7fed441e41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:53.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.407+0000 7fed48ff9640 1 --2- 172.21.15.88:0/1312474804 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed440ceda0 0x7fed441e41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:53.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.407+0000 7fed48ff9640 1 -- 172.21.15.88:0/1312474804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed4406dc10 msgr2=0x7fed4414c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:53.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.407+0000 7fed48ff9640 1 --2- 172.21.15.88:0/1312474804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed4406dc10 0x7fed4414c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:53.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.407+0000 7fed48ff9640 1 -- 172.21.15.88:0/1312474804 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed441e5960 con 0x7fed440cf180 2024-09-16T01:34:53.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.407+0000 7fed43fff640 1 --2- 172.21.15.88:0/1312474804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed4406dc10 0x7fed4414c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:53.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.407+0000 7fed48ff9640 1 --2- 172.21.15.88:0/1312474804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed440cf180 0x7fed441e3080 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fed3c00b970 tx=0x7fed3c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:53.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.408+0000 7fed40ff9640 1 -- 172.21.15.88:0/1312474804 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c019070 con 0x7fed440cf180 2024-09-16T01:34:53.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.408+0000 7fed40ff9640 1 -- 172.21.15.88:0/1312474804 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed3c00cb60 con 0x7fed440cf180 2024-09-16T01:34:53.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.408+0000 7fed4aa5c640 1 -- 172.21.15.88:0/1312474804 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed441e48c0 con 0x7fed440cf180 2024-09-16T01:34:53.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.408+0000 7fed40ff9640 1 -- 172.21.15.88:0/1312474804 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c014440 con 0x7fed440cf180 2024-09-16T01:34:53.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.408+0000 7fed4aa5c640 1 -- 172.21.15.88:0/1312474804 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed441e4e80 con 0x7fed440cf180 2024-09-16T01:34:53.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.409+0000 7fed4aa5c640 1 -- 172.21.15.88:0/1312474804 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed44073490 con 0x7fed440cf180 2024-09-16T01:34:53.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.409+0000 7fed40ff9640 1 -- 172.21.15.88:0/1312474804 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fed3c01c070 con 0x7fed440cf180 2024-09-16T01:34:53.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.410+0000 7fed40ff9640 1 --2- 172.21.15.88:0/1312474804 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fed3006c2b0 0x7fed3006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:53.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.410+0000 7fed43fff640 1 --2- 172.21.15.88:0/1312474804 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fed3006c2b0 0x7fed3006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:53.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.410+0000 7fed40ff9640 1 -- 172.21.15.88:0/1312474804 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5357+0+0 (secure 0 0 0) 0x7fed3c08af80 con 0x7fed440cf180 2024-09-16T01:34:53.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.410+0000 7fed43fff640 1 --2- 172.21.15.88:0/1312474804 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fed3006c2b0 0x7fed3006e770 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fed4415a830 tx=0x7fed34008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:53.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.414+0000 7fed40ff9640 1 -- 172.21.15.88:0/1312474804 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fed3c059200 con 0x7fed440cf180 2024-09-16T01:34:53.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:53.581+0000 7fed4aa5c640 1 -- 172.21.15.88:0/1312474804 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fed440ce100 con 0x7fed440cf180 2024-09-16T01:34:54.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.283+0000 7fed40ff9640 1 -- 172.21.15.88:0/1312474804 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v24) ==== 169+0+0 (secure 0 0 0) 0x7fed3c05d500 con 0x7fed440cf180 2024-09-16T01:34:54.284 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_data' removed 2024-09-16T01:34:54.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.286+0000 7fed4aa5c640 1 -- 172.21.15.88:0/1312474804 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fed3006c2b0 msgr2=0x7fed3006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:54.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.286+0000 7fed4aa5c640 1 --2- 172.21.15.88:0/1312474804 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fed3006c2b0 0x7fed3006e770 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fed4415a830 tx=0x7fed34008040 comp rx=0 tx=0).stop 2024-09-16T01:34:54.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.286+0000 7fed4aa5c640 1 -- 172.21.15.88:0/1312474804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed440cf180 msgr2=0x7fed441e3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:54.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.286+0000 7fed4aa5c640 1 --2- 172.21.15.88:0/1312474804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed440cf180 0x7fed441e3080 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fed3c00b970 tx=0x7fed3c00be30 comp rx=0 tx=0).stop 2024-09-16T01:34:54.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.286+0000 7fed4aa5c640 1 -- 172.21.15.88:0/1312474804 shutdown_connections 2024-09-16T01:34:54.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.287+0000 7fed4aa5c640 1 --2- 172.21.15.88:0/1312474804 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fed3006c2b0 0x7fed3006e770 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:54.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.287+0000 7fed4aa5c640 1 --2- 172.21.15.88:0/1312474804 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed440cf180 0x7fed441e3080 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:54.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.287+0000 7fed4aa5c640 1 --2- 172.21.15.88:0/1312474804 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed440ceda0 0x7fed441e41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:54.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.287+0000 7fed4aa5c640 1 --2- 172.21.15.88:0/1312474804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed4406dc10 0x7fed4414c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:54.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.287+0000 7fed4aa5c640 1 -- 172.21.15.88:0/1312474804 >> 172.21.15.88:0/1312474804 conn(0x7fed4406cb50 msgr2=0x7fed440b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:54.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.287+0000 7fed4aa5c640 1 -- 172.21.15.88:0/1312474804 shutdown_connections 2024-09-16T01:34:54.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.287+0000 7fed4aa5c640 1 -- 172.21.15.88:0/1312474804 wait complete. 2024-09-16T01:34:54.301 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-16T01:34:54.301 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-16T01:34:54.301 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-16T01:34:54.301 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:34:54.302 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-16T01:34:54.302 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-16T01:34:54.302 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-16T01:34:54.302 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:34:54.302 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-16T01:34:54.302 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-16T01:34:54.302 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-16T01:34:54.303 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:34:54.303 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-16T01:34:54.303 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-16T01:34:54.303 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-16T01:34:54.303 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:34:54.311 INFO:tasks.ceph.mds.a:Stopped 2024-09-16T01:34:54.311 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-16T01:34:54.312 INFO:tasks.ceph.mds.c:Stopped 2024-09-16T01:34:54.312 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-16T01:34:54.313 INFO:tasks.ceph.mds.b:Stopped 2024-09-16T01:34:54.313 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-16T01:34:54.314 INFO:tasks.ceph.mds.d:Stopped 2024-09-16T01:34:54.314 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-16T01:34:54.353 INFO:tasks.ceph.mds.a:Started 2024-09-16T01:34:54.354 DEBUG:teuthology.parallel:result is None 2024-09-16T01:34:54.355 INFO:tasks.ceph.mds.c:Started 2024-09-16T01:34:54.355 DEBUG:teuthology.parallel:result is None 2024-09-16T01:34:54.356 INFO:tasks.ceph.mds.b:Started 2024-09-16T01:34:54.356 DEBUG:teuthology.parallel:result is None 2024-09-16T01:34:54.357 INFO:tasks.ceph.mds.d:Started 2024-09-16T01:34:54.357 DEBUG:teuthology.parallel:result is None 2024-09-16T01:34:54.357 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-16T01:34:54.429 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:34:54.431 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:34:54.438 INFO:tasks.ceph.mds.b.smithi154.stdout:starting mds.b at 2024-09-16T01:34:54.438 INFO:tasks.ceph.mds.d.smithi154.stdout:starting mds.d at 2024-09-16T01:34:54.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.461+0000 7f5fad8f8640 1 Processor -- start 2024-09-16T01:34:54.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.461+0000 7f5fad8f8640 1 -- start start 2024-09-16T01:34:54.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.462+0000 7f5fad8f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5fa8155880 0x7f5fa815f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:54.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.462+0000 7f5fad8f8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5fa80b4d60 0x7f5fa80b51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:54.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.462+0000 7f5fad8f8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fa815bb60 con 0x7f5fa806f630 2024-09-16T01:34:54.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.462+0000 7f5fad8f8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f5fa8155c60 con 0x7f5fa80b4d60 2024-09-16T01:34:54.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.462+0000 7f5fad8f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5fa80b5700 con 0x7f5fa8155880 2024-09-16T01:34:54.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.462+0000 7f5fa67fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5fa80b4d60 0x7f5fa80b51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:54.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.462+0000 7f5fa67fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5fa80b4d60 0x7f5fa80b51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55492/0 (socket says 172.21.15.88:55492) 2024-09-16T01:34:54.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.462+0000 7f5fa67fc640 1 -- 172.21.15.88:0/563213167 learned_addr learned my addr 172.21.15.88:0/563213167 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:54.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.462+0000 7f5fa6ffd640 1 --2- 172.21.15.88:0/563213167 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5fa8155880 0x7f5fa815f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:54.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.462+0000 7f5fa67fc640 1 -- 172.21.15.88:0/563213167 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5fa8155880 msgr2=0x7f5fa815f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:54.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.462+0000 7f5fa67fc640 1 --2- 172.21.15.88:0/563213167 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5fa8155880 0x7f5fa815f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:54.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.462+0000 7f5fa67fc640 1 -- 172.21.15.88:0/563213167 >> v1:172.21.15.154:6789/0 conn(0x7f5fa806f630 legacy=0x7f5fa806fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:54.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.462+0000 7f5fa67fc640 1 -- 172.21.15.88:0/563213167 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa815be40 con 0x7f5fa80b4d60 2024-09-16T01:34:54.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.463+0000 7f5fa67fc640 1 --2- 172.21.15.88:0/563213167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fa80b4d60 0x7f5fa80b51c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c009870 tx=0x7f5f9c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7564b11675bb74aa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:54.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.463+0000 7f5fa5ffb640 1 -- 172.21.15.88:0/563213167 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c037070 con 0x7f5fa80b4d60 2024-09-16T01:34:54.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.463+0000 7f5fa5ffb640 1 -- 172.21.15.88:0/563213167 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f9c032440 con 0x7f5fa80b4d60 2024-09-16T01:34:54.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.463+0000 7f5fa5ffb640 1 -- 172.21.15.88:0/563213167 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c032810 con 0x7f5fa80b4d60 2024-09-16T01:34:54.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.463+0000 7f5fad8f8640 1 -- 172.21.15.88:0/563213167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fa80b4d60 msgr2=0x7f5fa80b51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:54.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.463+0000 7f5fad8f8640 1 --2- 172.21.15.88:0/563213167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fa80b4d60 0x7f5fa80b51c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c009870 tx=0x7f5f9c02a270 comp rx=0 tx=0).stop 2024-09-16T01:34:54.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.464+0000 7f5fad8f8640 1 -- 172.21.15.88:0/563213167 shutdown_connections 2024-09-16T01:34:54.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.464+0000 7f5fad8f8640 1 --2- 172.21.15.88:0/563213167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fa80b4d60 0x7f5fa80b51c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:54.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.464+0000 7f5fad8f8640 1 --2- 172.21.15.88:0/563213167 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5fa8155880 0x7f5fa815f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:54.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.464+0000 7f5fad8f8640 1 -- 172.21.15.88:0/563213167 >> 172.21.15.88:0/563213167 conn(0x7f5fa80d1390 msgr2=0x7f5fa80d17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:54.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.464+0000 7f5fad8f8640 1 -- 172.21.15.88:0/563213167 shutdown_connections 2024-09-16T01:34:54.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.464+0000 7f5fad8f8640 1 -- 172.21.15.88:0/563213167 wait complete. 2024-09-16T01:34:54.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.465+0000 7f5fad8f8640 1 Processor -- start 2024-09-16T01:34:54.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.465+0000 7f5fad8f8640 1 -- start start 2024-09-16T01:34:54.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.466+0000 7f5fad8f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5fa806f630 0x7f5fa8147e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:54.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.466+0000 7f5fad8f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5fa80b4d60 0x7f5fa80cea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:54.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.466+0000 7f5fad8f8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fa8155880 0x7f5fa8157fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:54.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.466+0000 7f5fad8f8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa815c400 con 0x7f5fa8155880 2024-09-16T01:34:54.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.466+0000 7f5fad8f8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa815fcb0 con 0x7f5fa806f630 2024-09-16T01:34:54.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.466+0000 7f5fad8f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5fa815c6e0 con 0x7f5fa80b4d60 2024-09-16T01:34:54.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.466+0000 7f5fa77fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fa8155880 0x7f5fa8157fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:54.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.466+0000 7f5fa77fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fa8155880 0x7f5fa8157fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55498/0 (socket says 172.21.15.88:55498) 2024-09-16T01:34:54.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.466+0000 7f5fa77fe640 1 -- 172.21.15.88:0/1487456425 learned_addr learned my addr 172.21.15.88:0/1487456425 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:54.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.466+0000 7f5fa6ffd640 1 --2- 172.21.15.88:0/1487456425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5fa806f630 0x7f5fa8147e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:54.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.466+0000 7f5fa67fc640 1 --2- 172.21.15.88:0/1487456425 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5fa80b4d60 0x7f5fa80cea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:54.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.466+0000 7f5fa77fe640 1 -- 172.21.15.88:0/1487456425 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5fa80b4d60 msgr2=0x7f5fa80cea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:54.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.466+0000 7f5fa77fe640 1 --2- 172.21.15.88:0/1487456425 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5fa80b4d60 0x7f5fa80cea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:54.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.466+0000 7f5fa77fe640 1 -- 172.21.15.88:0/1487456425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5fa806f630 msgr2=0x7f5fa8147e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:54.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.466+0000 7f5fa77fe640 1 --2- 172.21.15.88:0/1487456425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5fa806f630 0x7f5fa8147e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:54.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.466+0000 7f5fa77fe640 1 -- 172.21.15.88:0/1487456425 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa80cf9e0 con 0x7f5fa8155880 2024-09-16T01:34:54.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.466+0000 7f5fa77fe640 1 --2- 172.21.15.88:0/1487456425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fa8155880 0x7f5fa8157fc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5fa000c7a0 tx=0x7f5fa000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:54.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.466+0000 7f5f8ffff640 1 -- 172.21.15.88:0/1487456425 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0003b70 con 0x7f5fa8155880 2024-09-16T01:34:54.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.466+0000 7f5f8ffff640 1 -- 172.21.15.88:0/1487456425 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fa0003d30 con 0x7f5fa8155880 2024-09-16T01:34:54.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.466+0000 7f5fad8f8640 1 -- 172.21.15.88:0/1487456425 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fa81e5f90 con 0x7f5fa8155880 2024-09-16T01:34:54.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.466+0000 7f5f8ffff640 1 -- 172.21.15.88:0/1487456425 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa000aa90 con 0x7f5fa8155880 2024-09-16T01:34:54.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.466+0000 7f5fad8f8640 1 -- 172.21.15.88:0/1487456425 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fa80cfc80 con 0x7f5fa8155880 2024-09-16T01:34:54.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.468+0000 7f5fad8f8640 1 -- 172.21.15.88:0/1487456425 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fa815b770 con 0x7f5fa8155880 2024-09-16T01:34:54.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.468+0000 7f5f8ffff640 1 -- 172.21.15.88:0/1487456425 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5fa000ac30 con 0x7f5fa8155880 2024-09-16T01:34:54.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.468+0000 7f5f8ffff640 1 --2- 172.21.15.88:0/1487456425 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5f9406c2b0 0x7f5f9406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:54.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.468+0000 7f5f8ffff640 1 -- 172.21.15.88:0/1487456425 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 4943+0+0 (secure 0 0 0) 0x7f5fa008af40 con 0x7f5fa8155880 2024-09-16T01:34:54.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.468+0000 7f5fa6ffd640 1 --2- 172.21.15.88:0/1487456425 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5f9406c2b0 0x7f5f9406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:54.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.469+0000 7f5fa6ffd640 1 --2- 172.21.15.88:0/1487456425 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5f9406c2b0 0x7f5f9406e770 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5f980079a0 tx=0x7f5f98008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:54.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.473+0000 7f5f8ffff640 1 -- 172.21.15.88:0/1487456425 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5fa0059360 con 0x7f5fa8155880 2024-09-16T01:34:54.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:54.650+0000 7f5fad8f8640 1 -- 172.21.15.88:0/1487456425 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f5fa80cfe50 con 0x7f5fa8155880 2024-09-16T01:34:55.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.308+0000 7f5f8ffff640 1 -- 172.21.15.88:0/1487456425 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v26) ==== 97+0+0 (secure 0 0 0) 0x7f5fa005d660 con 0x7f5fa8155880 2024-09-16T01:34:55.310 INFO:teuthology.orchestra.run.smithi088.stderr: removed all blocklist entries 2024-09-16T01:34:55.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.312+0000 7f5fad8f8640 1 -- 172.21.15.88:0/1487456425 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5f9406c2b0 msgr2=0x7f5f9406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:55.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.312+0000 7f5fad8f8640 1 --2- 172.21.15.88:0/1487456425 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5f9406c2b0 0x7f5f9406e770 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5f980079a0 tx=0x7f5f98008040 comp rx=0 tx=0).stop 2024-09-16T01:34:55.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.312+0000 7f5fad8f8640 1 -- 172.21.15.88:0/1487456425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fa8155880 msgr2=0x7f5fa8157fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:55.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.312+0000 7f5fad8f8640 1 --2- 172.21.15.88:0/1487456425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fa8155880 0x7f5fa8157fc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5fa000c7a0 tx=0x7f5fa000cc60 comp rx=0 tx=0).stop 2024-09-16T01:34:55.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.312+0000 7f5fad8f8640 1 -- 172.21.15.88:0/1487456425 shutdown_connections 2024-09-16T01:34:55.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.312+0000 7f5fad8f8640 1 --2- 172.21.15.88:0/1487456425 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5f9406c2b0 0x7f5f9406e770 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:55.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.312+0000 7f5fad8f8640 1 --2- 172.21.15.88:0/1487456425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fa8155880 0x7f5fa8157fc0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:55.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.312+0000 7f5fad8f8640 1 --2- 172.21.15.88:0/1487456425 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5fa80b4d60 0x7f5fa80cea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:55.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.312+0000 7f5fad8f8640 1 --2- 172.21.15.88:0/1487456425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5fa806f630 0x7f5fa8147e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:55.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.312+0000 7f5fad8f8640 1 -- 172.21.15.88:0/1487456425 >> 172.21.15.88:0/1487456425 conn(0x7f5fa80d1390 msgr2=0x7f5fa80b1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:55.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.312+0000 7f5fad8f8640 1 -- 172.21.15.88:0/1487456425 shutdown_connections 2024-09-16T01:34:55.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.312+0000 7f5fad8f8640 1 -- 172.21.15.88:0/1487456425 wait complete. 2024-09-16T01:34:55.327 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-16T01:34:55.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.430+0000 7f6cfccaf640 1 Processor -- start 2024-09-16T01:34:55.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.430+0000 7f6cfccaf640 1 -- start start 2024-09-16T01:34:55.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.431+0000 7f6cfccaf640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6cf806f340 0x7f6cf806f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:55.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.431+0000 7f6cfccaf640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cf80d1870 con 0x7f6cf80cfb90 2024-09-16T01:34:55.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.431+0000 7f6cfccaf640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cf815cc00 con 0x7f6cf8073350 2024-09-16T01:34:55.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.431+0000 7f6cfccaf640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6cf815d810 con 0x7f6cf806f340 2024-09-16T01:34:55.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.431+0000 7f6cedd9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6cf806f340 0x7f6cf806f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:55.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.431+0000 7f6cf659c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6cf80cfb90 0x7f6cf80d1b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33632/0 (socket says 172.21.15.88:33632) 2024-09-16T01:34:55.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.431+0000 7f6cf659c640 1 -- 172.21.15.88:0/2396999758 learned_addr learned my addr 172.21.15.88:0/2396999758 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:55.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.432+0000 7f6cedd9b640 1 -- 172.21.15.88:0/2396999758 >> v1:172.21.15.154:6789/0 conn(0x7f6cf8073350 legacy=0x7f6cf80737c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:55.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.432+0000 7f6cedd9b640 1 -- 172.21.15.88:0/2396999758 >> v1:172.21.15.88:6789/0 conn(0x7f6cf80cfb90 legacy=0x7f6cf80d1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:55.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.432+0000 7f6cf5d9b640 1 -- 172.21.15.88:0/2396999758 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2987503172 0 0) 0x7f6cf80d1870 con 0x7f6cf80cfb90 2024-09-16T01:34:55.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.432+0000 7f6cedd9b640 1 -- 172.21.15.88:0/2396999758 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf815cee0 con 0x7f6cf806f340 2024-09-16T01:34:55.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.432+0000 7f6cedd9b640 1 --2- 172.21.15.88:0/2396999758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf806f340 0x7f6cf806f780 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00097a0 tx=0x7f6ce002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee6bbf198df31091 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:55.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.433+0000 7f6cf5d9b640 1 -- 172.21.15.88:0/2396999758 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0037070 con 0x7f6cf806f340 2024-09-16T01:34:55.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.433+0000 7f6cf5d9b640 1 -- 172.21.15.88:0/2396999758 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ce0032440 con 0x7f6cf806f340 2024-09-16T01:34:55.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.433+0000 7f6cf5d9b640 1 -- 172.21.15.88:0/2396999758 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0032810 con 0x7f6cf806f340 2024-09-16T01:34:55.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.433+0000 7f6cfccaf640 1 -- 172.21.15.88:0/2396999758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf806f340 msgr2=0x7f6cf806f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:55.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.433+0000 7f6cfccaf640 1 --2- 172.21.15.88:0/2396999758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf806f340 0x7f6cf806f780 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00097a0 tx=0x7f6ce002a230 comp rx=0 tx=0).stop 2024-09-16T01:34:55.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.433+0000 7f6cfccaf640 1 -- 172.21.15.88:0/2396999758 shutdown_connections 2024-09-16T01:34:55.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.433+0000 7f6cfccaf640 1 --2- 172.21.15.88:0/2396999758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf806f340 0x7f6cf806f780 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:55.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.433+0000 7f6cfccaf640 1 -- 172.21.15.88:0/2396999758 >> 172.21.15.88:0/2396999758 conn(0x7f6cf8062630 msgr2=0x7f6cf8062a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:55.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.433+0000 7f6cfccaf640 1 -- 172.21.15.88:0/2396999758 shutdown_connections 2024-09-16T01:34:55.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.433+0000 7f6cfccaf640 1 -- 172.21.15.88:0/2396999758 wait complete. 2024-09-16T01:34:55.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.435+0000 7f6cfccaf640 1 Processor -- start 2024-09-16T01:34:55.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.435+0000 7f6cfccaf640 1 -- start start 2024-09-16T01:34:55.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.435+0000 7f6cfccaf640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6cf806f340 0x7f6cf81517e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:55.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.435+0000 7f6cfccaf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf8073350 0x7f6cf8153ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:55.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.435+0000 7f6cfccaf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf80cfb90 0x7f6cf8154400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:55.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.435+0000 7f6cfccaf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf815d4a0 con 0x7f6cf8073350 2024-09-16T01:34:55.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.435+0000 7f6cfccaf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf806fcc0 con 0x7f6cf80cfb90 2024-09-16T01:34:55.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.435+0000 7f6cfccaf640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6cf80d2260 con 0x7f6cf806f340 2024-09-16T01:34:55.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.435+0000 7f6cedd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf8073350 0x7f6cf8153ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:55.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.435+0000 7f6cedd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf8073350 0x7f6cf8153ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55504/0 (socket says 172.21.15.88:55504) 2024-09-16T01:34:55.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.435+0000 7f6cedd9b640 1 -- 172.21.15.88:0/897492498 learned_addr learned my addr 172.21.15.88:0/897492498 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:55.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.436+0000 7f6cf659c640 1 --2- 172.21.15.88:0/897492498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6cf806f340 0x7f6cf81517e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:55.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.436+0000 7f6cf6d9d640 1 --2- 172.21.15.88:0/897492498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf80cfb90 0x7f6cf8154400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:55.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.436+0000 7f6cedd9b640 1 -- 172.21.15.88:0/897492498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6cf806f340 msgr2=0x7f6cf81517e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:55.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.436+0000 7f6cedd9b640 1 --2- 172.21.15.88:0/897492498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6cf806f340 0x7f6cf81517e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:55.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.436+0000 7f6cedd9b640 1 -- 172.21.15.88:0/897492498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf80cfb90 msgr2=0x7f6cf8154400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:55.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.436+0000 7f6cedd9b640 1 --2- 172.21.15.88:0/897492498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf80cfb90 0x7f6cf8154400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:55.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.436+0000 7f6cedd9b640 1 -- 172.21.15.88:0/897492498 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf81522a0 con 0x7f6cf8073350 2024-09-16T01:34:55.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.436+0000 7f6cedd9b640 1 --2- 172.21.15.88:0/897492498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf8073350 0x7f6cf8153ec0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00097a0 tx=0x7f6ce00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:55.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.436+0000 7f6cef7fe640 1 -- 172.21.15.88:0/897492498 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0037050 con 0x7f6cf8073350 2024-09-16T01:34:55.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.436+0000 7f6cef7fe640 1 -- 172.21.15.88:0/897492498 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ce0038040 con 0x7f6cf8073350 2024-09-16T01:34:55.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.436+0000 7f6cef7fe640 1 -- 172.21.15.88:0/897492498 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0004490 con 0x7f6cf8073350 2024-09-16T01:34:55.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.436+0000 7f6cfccaf640 1 -- 172.21.15.88:0/897492498 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cf8152580 con 0x7f6cf8073350 2024-09-16T01:34:55.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.436+0000 7f6cfccaf640 1 -- 172.21.15.88:0/897492498 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cf81574d0 con 0x7f6cf8073350 2024-09-16T01:34:55.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.437+0000 7f6cfccaf640 1 -- 172.21.15.88:0/897492498 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cf80cff70 con 0x7f6cf8073350 2024-09-16T01:34:55.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.438+0000 7f6cef7fe640 1 -- 172.21.15.88:0/897492498 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6ce0030900 con 0x7f6cf8073350 2024-09-16T01:34:55.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.438+0000 7f6cef7fe640 1 --2- 172.21.15.88:0/897492498 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6cd806c2b0 0x7f6cd806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:55.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.438+0000 7f6cf659c640 1 --2- 172.21.15.88:0/897492498 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6cd806c2b0 0x7f6cd806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:55.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.438+0000 7f6cef7fe640 1 -- 172.21.15.88:0/897492498 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7f6ce00abbe0 con 0x7f6cf8073350 2024-09-16T01:34:55.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.438+0000 7f6cf659c640 1 --2- 172.21.15.88:0/897492498 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6cd806c2b0 0x7f6cd806e770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8000f60 tx=0x7f6ce8007e40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:55.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.442+0000 7f6cef7fe640 1 -- 172.21.15.88:0/897492498 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6ce007a200 con 0x7f6cf8073350 2024-09-16T01:34:55.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.678+0000 7f6cfccaf640 1 -- 172.21.15.88:0/897492498 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f6cf80b6b50 con 0x7f6cf8073350 2024-09-16T01:34:55.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.678+0000 7f6cef7fe640 1 -- 172.21.15.88:0/897492498 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) ==== 80+0+6645 (secure 0 0 0) 0x7f6ce007e500 con 0x7f6cf8073350 2024-09-16T01:34:55.680 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:55.680 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:34:55.680 INFO:teuthology.orchestra.run.smithi088.stdout: "auth_dump": [ 2024-09-16T01:34:55.680 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:55.680 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mds.a", 2024-09-16T01:34:55.680 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmy+jJCRAAhnzX/li6D3D07NmThrjXkA==", 2024-09-16T01:34:55.680 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:34:55.680 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:34:55.680 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:34:55.680 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:34:55.680 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:34:55.681 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.681 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:34:55.681 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:55.681 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mds.b", 2024-09-16T01:34:55.681 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmx7p+FRAApxKUSZoR/nGUZsaYKlrYUg==", 2024-09-16T01:34:55.681 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:34:55.681 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:34:55.681 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:34:55.681 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:34:55.681 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:34:55.681 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.681 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:34:55.681 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:55.682 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mds.c", 2024-09-16T01:34:55.682 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmaYHUEBAAzM0qE53LEgHi+hThHJX3Tw==", 2024-09-16T01:34:55.682 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:34:55.682 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:34:55.682 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:34:55.682 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:34:55.682 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:34:55.682 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.682 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:34:55.682 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:55.682 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mds.d", 2024-09-16T01:34:55.682 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudm3TSLHBAAJzlR6H09cc1Pki2FjyoFiA==", 2024-09-16T01:34:55.682 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:34:55.682 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:34:55.683 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:34:55.683 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:34:55.683 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:34:55.683 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.683 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:34:55.683 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:55.683 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.0", 2024-09-16T01:34:55.683 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDhiudmZR7MORAAwDKvphubA+CbzBUWuaA80A==", 2024-09-16T01:34:55.683 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:34:55.683 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:34:55.683 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:34:55.683 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:34:55.683 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.684 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:34:55.684 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:55.684 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.1", 2024-09-16T01:34:55.684 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDliudmNd+SLBAArOUdoZ226c5jlAsKwqERmw==", 2024-09-16T01:34:55.684 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:34:55.684 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:34:55.684 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:34:55.684 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:34:55.684 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.684 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:34:55.684 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:55.684 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.2", 2024-09-16T01:34:55.684 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDpiudm75CUARAAWHtYOg9pE8g+M6IL+8XZ6g==", 2024-09-16T01:34:55.685 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:34:55.685 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:34:55.685 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:34:55.685 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:34:55.685 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.685 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:34:55.685 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:55.685 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.3", 2024-09-16T01:34:55.685 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDtiudm0fgWAxAAf2NOWrpXbOhb3dFXozJagQ==", 2024-09-16T01:34:55.685 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:34:55.685 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:34:55.685 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:34:55.685 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:34:55.686 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.686 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:34:55.686 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:55.686 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.4", 2024-09-16T01:34:55.686 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQD0iudmdddMCxAAtr2qfj4Nxj3/GHUpxRmtFg==", 2024-09-16T01:34:55.686 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:34:55.686 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:34:55.686 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:34:55.686 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:34:55.686 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.686 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:34:55.686 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:55.686 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.5", 2024-09-16T01:34:55.686 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQD3iudmT5RjOhAAcKwca6BG60gsxYnmy4twGg==", 2024-09-16T01:34:55.687 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:34:55.687 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:34:55.687 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:34:55.687 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:34:55.687 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.687 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:34:55.687 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:55.687 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.6", 2024-09-16T01:34:55.687 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQD7iudmT7daLRAAdfLFijKQku3rJcJ5xcfLsw==", 2024-09-16T01:34:55.687 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:34:55.687 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:34:55.687 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:34:55.688 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:34:55.688 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.688 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:34:55.688 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:55.688 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.7", 2024-09-16T01:34:55.688 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQD/iudmmvtxLhAArYKhQ4zyFHtLgh/+bs5ZhA==", 2024-09-16T01:34:55.688 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:34:55.688 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:34:55.688 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:34:55.688 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:34:55.688 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.688 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:34:55.688 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:55.688 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.0", 2024-09-16T01:34:55.689 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmvPikIBAAyHxNNnwQKygxg09PNQLc/Q==", 2024-09-16T01:34:55.689 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:34:55.689 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:34:55.689 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow r", 2024-09-16T01:34:55.689 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow rw", 2024-09-16T01:34:55.689 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow rwx" 2024-09-16T01:34:55.689 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.689 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:34:55.689 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:55.689 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.1", 2024-09-16T01:34:55.689 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmA69iKRAA8Abwe0Lg2PnZDErgJ1gEbQ==", 2024-09-16T01:34:55.689 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:34:55.690 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:34:55.690 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow r", 2024-09-16T01:34:55.690 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow rw", 2024-09-16T01:34:55.690 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow rwx" 2024-09-16T01:34:55.690 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.690 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:34:55.690 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:55.690 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.2", 2024-09-16T01:34:55.690 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmB+w9JhAAv63Is5T/cvDbbLOjIkI0wA==", 2024-09-16T01:34:55.690 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:34:55.690 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:34:55.690 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow r", 2024-09-16T01:34:55.690 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow rw", 2024-09-16T01:34:55.691 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow rwx" 2024-09-16T01:34:55.691 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.691 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:34:55.691 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:55.691 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.3", 2024-09-16T01:34:55.691 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudm4WoFLxAAnYGHGqHyBHHajQ8f3rCjEA==", 2024-09-16T01:34:55.691 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:34:55.691 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:34:55.691 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow r", 2024-09-16T01:34:55.691 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow rw", 2024-09-16T01:34:55.691 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow rwx" 2024-09-16T01:34:55.691 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.691 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:34:55.691 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:55.692 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.admin", 2024-09-16T01:34:55.692 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDdiudmhcE0KBAANauQKyUY01/KlWMrN2x/5Q==", 2024-09-16T01:34:55.692 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:34:55.692 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow *", 2024-09-16T01:34:55.692 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:34:55.692 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:34:55.692 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:34:55.692 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.692 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:34:55.692 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:55.692 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.bootstrap-mds", 2024-09-16T01:34:55.692 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQAGi+dmv/udGBAAlCezDzrakbV79CcCtd+ung==", 2024-09-16T01:34:55.692 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:34:55.693 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile bootstrap-mds" 2024-09-16T01:34:55.693 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.693 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:34:55.693 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:55.693 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.bootstrap-mgr", 2024-09-16T01:34:55.693 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQAGi+dmWQOeGBAA519Zn4zvQ34Q7u+X9yU+2w==", 2024-09-16T01:34:55.693 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:34:55.693 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile bootstrap-mgr" 2024-09-16T01:34:55.693 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.693 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:34:55.693 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:55.693 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.bootstrap-osd", 2024-09-16T01:34:55.693 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQAGi+dmKQqeGBAAqTcuuh3m19D231omCJh0lg==", 2024-09-16T01:34:55.694 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:34:55.694 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile bootstrap-osd" 2024-09-16T01:34:55.694 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.694 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:34:55.694 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:55.694 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.bootstrap-rbd", 2024-09-16T01:34:55.694 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQAGi+dmiBCeGBAABbLRJpR0KebLCFKzK8QzHw==", 2024-09-16T01:34:55.694 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:34:55.694 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile bootstrap-rbd" 2024-09-16T01:34:55.694 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.694 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:34:55.694 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:55.694 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-09-16T01:34:55.694 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQAGi+dm0BeeGBAA8gkzh0rgA0er7O6yhgjOhA==", 2024-09-16T01:34:55.695 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:34:55.695 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-09-16T01:34:55.695 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.695 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:34:55.695 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:55.695 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.bootstrap-rgw", 2024-09-16T01:34:55.695 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQAGi+dmrx6eGBAAauQ7Bg1U7n9vYZ6RwNRUdw==", 2024-09-16T01:34:55.695 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:34:55.695 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile bootstrap-rgw" 2024-09-16T01:34:55.695 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.695 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:34:55.695 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:55.695 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mgr.x", 2024-09-16T01:34:55.696 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmToGYBhAAKwXS34PtfAVtEtdtHKHlQQ==", 2024-09-16T01:34:55.696 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:34:55.696 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow *", 2024-09-16T01:34:55.696 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile mgr", 2024-09-16T01:34:55.696 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:34:55.696 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.696 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:34:55.696 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:55.696 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mgr.y", 2024-09-16T01:34:55.696 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudm/BN5ARAAxbCopbsvxFUZP3drso32qw==", 2024-09-16T01:34:55.696 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:34:55.696 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow *", 2024-09-16T01:34:55.696 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile mgr", 2024-09-16T01:34:55.697 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:34:55.697 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.697 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:55.697 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-09-16T01:34:55.697 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:34:55.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.681+0000 7f6cfccaf640 1 -- 172.21.15.88:0/897492498 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6cd806c2b0 msgr2=0x7f6cd806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:55.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.681+0000 7f6cfccaf640 1 --2- 172.21.15.88:0/897492498 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6cd806c2b0 0x7f6cd806e770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8000f60 tx=0x7f6ce8007e40 comp rx=0 tx=0).stop 2024-09-16T01:34:55.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.681+0000 7f6cfccaf640 1 -- 172.21.15.88:0/897492498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf8073350 msgr2=0x7f6cf8153ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:55.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.681+0000 7f6cfccaf640 1 --2- 172.21.15.88:0/897492498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf8073350 0x7f6cf8153ec0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00097a0 tx=0x7f6ce00046c0 comp rx=0 tx=0).stop 2024-09-16T01:34:55.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.681+0000 7f6cfccaf640 1 -- 172.21.15.88:0/897492498 shutdown_connections 2024-09-16T01:34:55.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.681+0000 7f6cfccaf640 1 --2- 172.21.15.88:0/897492498 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6cd806c2b0 0x7f6cd806e770 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:55.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.681+0000 7f6cfccaf640 1 --2- 172.21.15.88:0/897492498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf80cfb90 0x7f6cf8154400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:55.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.681+0000 7f6cfccaf640 1 --2- 172.21.15.88:0/897492498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6cf8073350 0x7f6cf8153ec0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:55.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.681+0000 7f6cfccaf640 1 --2- 172.21.15.88:0/897492498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6cf806f340 0x7f6cf81517e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:55.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.681+0000 7f6cfccaf640 1 -- 172.21.15.88:0/897492498 >> 172.21.15.88:0/897492498 conn(0x7f6cf8062630 msgr2=0x7f6cf815dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:55.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.681+0000 7f6cfccaf640 1 -- 172.21.15.88:0/897492498 shutdown_connections 2024-09-16T01:34:55.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.681+0000 7f6cfccaf640 1 -- 172.21.15.88:0/897492498 wait complete. 2024-09-16T01:34:55.699 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-09-16T01:34:55.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.798+0000 7efc5b19f640 1 Processor -- start 2024-09-16T01:34:55.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.799+0000 7efc5b19f640 1 -- start start 2024-09-16T01:34:55.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.799+0000 7efc5b19f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc5415e7a0 0x7efc5406caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:55.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.799+0000 7efc5b19f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc540cf860 con 0x7efc540cf1f0 2024-09-16T01:34:55.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.799+0000 7efc5b19f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc540cfb40 con 0x7efc540b68f0 2024-09-16T01:34:55.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.799+0000 7efc5b19f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efc540d2ca0 con 0x7efc5415e7a0 2024-09-16T01:34:55.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.799+0000 7efc53fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7efc540cf1f0 0x7efc540b61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33644/0 (socket says 172.21.15.88:33644) 2024-09-16T01:34:55.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.800+0000 7efc53fff640 1 -- 172.21.15.88:0/3718897618 learned_addr learned my addr 172.21.15.88:0/3718897618 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:55.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.800+0000 7efc58f3b640 1 --2- 172.21.15.88:0/3718897618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc5415e7a0 0x7efc5406caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:55.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.800+0000 7efc537fe640 1 -- 172.21.15.88:0/3718897618 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 819175751 0 0) 0x7efc540cf860 con 0x7efc540cf1f0 2024-09-16T01:34:55.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.800+0000 7efc537fe640 1 -- 172.21.15.88:0/3718897618 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc44003520 con 0x7efc540cf1f0 2024-09-16T01:34:55.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.800+0000 7efc537fe640 1 -- 172.21.15.88:0/3718897618 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4242473435 0 0) 0x7efc540cfb40 con 0x7efc540b68f0 2024-09-16T01:34:55.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.800+0000 7efc537fe640 1 -- 172.21.15.88:0/3718897618 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc540cf860 con 0x7efc540b68f0 2024-09-16T01:34:55.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.800+0000 7efc537fe640 1 -- 172.21.15.88:0/3718897618 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1793873305 0 0) 0x7efc44003520 con 0x7efc540cf1f0 2024-09-16T01:34:55.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.800+0000 7efc537fe640 1 -- 172.21.15.88:0/3718897618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc5415e7a0 msgr2=0x7efc5406caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:55.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.800+0000 7efc537fe640 1 --2- 172.21.15.88:0/3718897618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc5415e7a0 0x7efc5406caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:55.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.800+0000 7efc537fe640 1 -- 172.21.15.88:0/3718897618 >> v1:172.21.15.154:6789/0 conn(0x7efc540b68f0 legacy=0x7efc540d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:55.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.800+0000 7efc537fe640 1 -- 172.21.15.88:0/3718897618 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc540cfe20 con 0x7efc540cf1f0 2024-09-16T01:34:55.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.800+0000 7efc537fe640 1 -- 172.21.15.88:0/3718897618 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7efc40001670 con 0x7efc540cf1f0 2024-09-16T01:34:55.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.800+0000 7efc58f3b640 1 --2- 172.21.15.88:0/3718897618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc5415e7a0 0x7efc5406caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:34:55.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.800+0000 7efc537fe640 1 -- 172.21.15.88:0/3718897618 >> v1:172.21.15.88:6789/0 conn(0x7efc540cf1f0 legacy=0x7efc540b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:55.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.800+0000 7efc537fe640 1 --2- 172.21.15.88:0/3718897618 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc44003b70 0x7efc44003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:55.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.800+0000 7efc537fe640 1 --2- 172.21.15.88:0/3718897618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc5415e7a0 0x7efc44004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:55.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.800+0000 7efc537fe640 1 --2- 172.21.15.88:0/3718897618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc44004d40 0x7efc440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:55.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.800+0000 7efc537fe640 1 -- 172.21.15.88:0/3718897618 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7efc540d2ca0 con 0x7efc44003b70 2024-09-16T01:34:55.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.800+0000 7efc537fe640 1 -- 172.21.15.88:0/3718897618 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efc4400baa0 con 0x7efc44004d40 2024-09-16T01:34:55.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.800+0000 7efc537fe640 1 -- 172.21.15.88:0/3718897618 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efc4400bca0 con 0x7efc5415e7a0 2024-09-16T01:34:55.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.800+0000 7efc58f3b640 1 --2- 172.21.15.88:0/3718897618 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc44003b70 0x7efc44003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:55.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.800+0000 7efc53fff640 1 --2- 172.21.15.88:0/3718897618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc5415e7a0 0x7efc44004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:55.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.800+0000 7efc5973c640 1 --2- 172.21.15.88:0/3718897618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc44004d40 0x7efc440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:55.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.800+0000 7efc58f3b640 1 -- 172.21.15.88:0/3718897618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc5415e7a0 msgr2=0x7efc44004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:55.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.801+0000 7efc58f3b640 1 --2- 172.21.15.88:0/3718897618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc5415e7a0 0x7efc44004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:55.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.801+0000 7efc58f3b640 1 -- 172.21.15.88:0/3718897618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc44004d40 msgr2=0x7efc440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:55.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.801+0000 7efc58f3b640 1 --2- 172.21.15.88:0/3718897618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc44004d40 0x7efc440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:55.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.801+0000 7efc58f3b640 1 -- 172.21.15.88:0/3718897618 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc44003520 con 0x7efc44003b70 2024-09-16T01:34:55.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.801+0000 7efc53fff640 1 --2- 172.21.15.88:0/3718897618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc5415e7a0 0x7efc44004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:55.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.801+0000 7efc58f3b640 1 --2- 172.21.15.88:0/3718897618 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc44003b70 0x7efc44003f70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7efc48009c90 tx=0x7efc44005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38bb9c9de760dd2a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:55.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.801+0000 7efc537fe640 1 -- 172.21.15.88:0/3718897618 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc48036070 con 0x7efc44003b70 2024-09-16T01:34:55.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.801+0000 7efc537fe640 1 -- 172.21.15.88:0/3718897618 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc48031400 con 0x7efc44003b70 2024-09-16T01:34:55.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.801+0000 7efc537fe640 1 -- 172.21.15.88:0/3718897618 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc48009860 con 0x7efc44003b70 2024-09-16T01:34:55.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.802+0000 7efc5b19f640 1 -- 172.21.15.88:0/3718897618 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc44003b70 msgr2=0x7efc44003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:55.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.802+0000 7efc5b19f640 1 --2- 172.21.15.88:0/3718897618 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc44003b70 0x7efc44003f70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7efc48009c90 tx=0x7efc44005830 comp rx=0 tx=0).stop 2024-09-16T01:34:55.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.802+0000 7efc5973c640 1 -- 172.21.15.88:0/3718897618 reap_dead start 2024-09-16T01:34:55.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.802+0000 7efc5b19f640 1 -- 172.21.15.88:0/3718897618 shutdown_connections 2024-09-16T01:34:55.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.802+0000 7efc5b19f640 1 -- 172.21.15.88:0/3718897618 >> 172.21.15.88:0/3718897618 conn(0x7efc540d11e0 msgr2=0x7efc540d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:55.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.802+0000 7efc5b19f640 1 -- 172.21.15.88:0/3718897618 shutdown_connections 2024-09-16T01:34:55.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.802+0000 7efc5b19f640 1 -- 172.21.15.88:0/3718897618 wait complete. 2024-09-16T01:34:55.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.803+0000 7efc5b19f640 1 Processor -- start 2024-09-16T01:34:55.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.803+0000 7efc5b19f640 1 -- start start 2024-09-16T01:34:55.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.804+0000 7efc5b19f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc44003b70 0x7efc540cd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:55.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.804+0000 7efc5b19f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc54157ba0 0x7efc541618c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:55.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.804+0000 7efc5b19f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc540b87b0 0x7efc540b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:55.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.804+0000 7efc5b19f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7efc5415eb80 con 0x7efc44003b70 2024-09-16T01:34:55.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.804+0000 7efc5b19f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efc540b7a00 con 0x7efc54157ba0 2024-09-16T01:34:55.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.804+0000 7efc5b19f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efc5415ed00 con 0x7efc540b87b0 2024-09-16T01:34:55.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.804+0000 7efc58f3b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc44003b70 0x7efc540cd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:55.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.804+0000 7efc58f3b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc44003b70 0x7efc540cd5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55522/0 (socket says 172.21.15.88:55522) 2024-09-16T01:34:55.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.804+0000 7efc58f3b640 1 -- 172.21.15.88:0/3944520120 learned_addr learned my addr 172.21.15.88:0/3944520120 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:55.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.804+0000 7efc5973c640 1 --2- 172.21.15.88:0/3944520120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc540b87b0 0x7efc540b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:55.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.804+0000 7efc53fff640 1 --2- 172.21.15.88:0/3944520120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc54157ba0 0x7efc541618c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:55.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.804+0000 7efc58f3b640 1 -- 172.21.15.88:0/3944520120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc540b87b0 msgr2=0x7efc540b8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:55.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.804+0000 7efc58f3b640 1 --2- 172.21.15.88:0/3944520120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc540b87b0 0x7efc540b8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:55.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.804+0000 7efc58f3b640 1 -- 172.21.15.88:0/3944520120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc54157ba0 msgr2=0x7efc541618c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:55.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.804+0000 7efc58f3b640 1 --2- 172.21.15.88:0/3944520120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc54157ba0 0x7efc541618c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:55.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.804+0000 7efc58f3b640 1 -- 172.21.15.88:0/3944520120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc540cc240 con 0x7efc44003b70 2024-09-16T01:34:55.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.804+0000 7efc5973c640 1 --2- 172.21.15.88:0/3944520120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc540b87b0 0x7efc540b8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:55.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.805+0000 7efc58f3b640 1 --2- 172.21.15.88:0/3944520120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc44003b70 0x7efc540cd5c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7efc48000c00 tx=0x7efc48004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:55.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.805+0000 7efc517fa640 1 -- 172.21.15.88:0/3944520120 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc48040070 con 0x7efc44003b70 2024-09-16T01:34:55.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.805+0000 7efc517fa640 1 -- 172.21.15.88:0/3944520120 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc480093e0 con 0x7efc44003b70 2024-09-16T01:34:55.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.805+0000 7efc517fa640 1 -- 172.21.15.88:0/3944520120 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc48036040 con 0x7efc44003b70 2024-09-16T01:34:55.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.805+0000 7efc5b19f640 1 -- 172.21.15.88:0/3944520120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc540cc520 con 0x7efc44003b70 2024-09-16T01:34:55.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.805+0000 7efc5b19f640 1 -- 172.21.15.88:0/3944520120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc540ccae0 con 0x7efc44003b70 2024-09-16T01:34:55.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.806+0000 7efc5b19f640 1 -- 172.21.15.88:0/3944520120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc540b61e0 con 0x7efc44003b70 2024-09-16T01:34:55.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.806+0000 7efc517fa640 1 -- 172.21.15.88:0/3944520120 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7efc48043070 con 0x7efc44003b70 2024-09-16T01:34:55.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.807+0000 7efc517fa640 1 --2- 172.21.15.88:0/3944520120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efc3806c2e0 0x7efc3806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:55.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.807+0000 7efc517fa640 1 -- 172.21.15.88:0/3944520120 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7efc480ab770 con 0x7efc44003b70 2024-09-16T01:34:55.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.807+0000 7efc53fff640 1 --2- 172.21.15.88:0/3944520120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efc3806c2e0 0x7efc3806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:55.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.807+0000 7efc53fff640 1 --2- 172.21.15.88:0/3944520120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efc3806c2e0 0x7efc3806e7a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7efc540d2110 tx=0x7efc40009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:55.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:55.815+0000 7efc517fa640 1 -- 172.21.15.88:0/3944520120 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7efc48079b50 con 0x7efc44003b70 2024-09-16T01:34:56.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.055+0000 7efc5b19f640 1 -- 172.21.15.88:0/3944520120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7efc540d33a0 con 0x7efc44003b70 2024-09-16T01:34:56.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.057+0000 7efc517fa640 1 -- 172.21.15.88:0/3944520120 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7efc4807de50 con 0x7efc44003b70 2024-09-16T01:34:56.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.059+0000 7efc5b19f640 1 -- 172.21.15.88:0/3944520120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efc3806c2e0 msgr2=0x7efc3806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:56.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.059+0000 7efc5b19f640 1 --2- 172.21.15.88:0/3944520120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efc3806c2e0 0x7efc3806e7a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7efc540d2110 tx=0x7efc40009040 comp rx=0 tx=0).stop 2024-09-16T01:34:56.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.059+0000 7efc5b19f640 1 -- 172.21.15.88:0/3944520120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc44003b70 msgr2=0x7efc540cd5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:56.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.059+0000 7efc5b19f640 1 --2- 172.21.15.88:0/3944520120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc44003b70 0x7efc540cd5c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7efc48000c00 tx=0x7efc48004730 comp rx=0 tx=0).stop 2024-09-16T01:34:56.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.059+0000 7efc5b19f640 1 -- 172.21.15.88:0/3944520120 shutdown_connections 2024-09-16T01:34:56.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.059+0000 7efc5b19f640 1 --2- 172.21.15.88:0/3944520120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efc3806c2e0 0x7efc3806e7a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.059+0000 7efc5b19f640 1 --2- 172.21.15.88:0/3944520120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc540b87b0 0x7efc540b8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.059+0000 7efc5b19f640 1 --2- 172.21.15.88:0/3944520120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc54157ba0 0x7efc541618c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.059+0000 7efc5b19f640 1 --2- 172.21.15.88:0/3944520120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc44003b70 0x7efc540cd5c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.059+0000 7efc5b19f640 1 -- 172.21.15.88:0/3944520120 >> 172.21.15.88:0/3944520120 conn(0x7efc540d11e0 msgr2=0x7efc54159e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:56.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.059+0000 7efc5b19f640 1 -- 172.21.15.88:0/3944520120 shutdown_connections 2024-09-16T01:34:56.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.059+0000 7efc5b19f640 1 -- 172.21.15.88:0/3944520120 wait complete. 2024-09-16T01:34:56.072 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-09-16T01:34:56.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.173+0000 7f9f028dd640 1 Processor -- start 2024-09-16T01:34:56.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.173+0000 7f9f028dd640 1 -- start start 2024-09-16T01:34:56.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.173+0000 7f9f028dd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9efc15a050 0x7f9efc15d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:56.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.173+0000 7f9f028dd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9efc0731c0 0x7f9efc073600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:56.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.174+0000 7f9f028dd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9efc15a430 con 0x7f9efc15ffd0 2024-09-16T01:34:56.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.174+0000 7f9f028dd640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f9efc073b40 con 0x7f9efc15a050 2024-09-16T01:34:56.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.174+0000 7f9f028dd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9efc15d9a0 con 0x7f9efc0731c0 2024-09-16T01:34:56.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.174+0000 7f9efbfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9efc15a050 0x7f9efc15d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:56.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.174+0000 7f9f00e7a640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9efc15ffd0 0x7f9efc160460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33654/0 (socket says 172.21.15.88:33654) 2024-09-16T01:34:56.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.174+0000 7f9f00e7a640 1 -- 172.21.15.88:0/3350142760 learned_addr learned my addr 172.21.15.88:0/3350142760 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:56.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.174+0000 7f9efb7fe640 1 --2- 172.21.15.88:0/3350142760 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9efc0731c0 0x7f9efc073600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:56.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.174+0000 7f9efbfff640 1 -- 172.21.15.88:0/3350142760 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9efc0731c0 msgr2=0x7f9efc073600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:56.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.174+0000 7f9efbfff640 1 --2- 172.21.15.88:0/3350142760 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9efc0731c0 0x7f9efc073600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.174+0000 7f9efbfff640 1 -- 172.21.15.88:0/3350142760 >> v1:172.21.15.88:6789/0 conn(0x7f9efc15ffd0 legacy=0x7f9efc160460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:56.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.174+0000 7f9efaffd640 1 -- 172.21.15.88:0/3350142760 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1006491557 0 0) 0x7f9efc15a430 con 0x7f9efc15ffd0 2024-09-16T01:34:56.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.174+0000 7f9efbfff640 1 -- 172.21.15.88:0/3350142760 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9efc0cd0a0 con 0x7f9efc15a050 2024-09-16T01:34:56.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.174+0000 7f9efb7fe640 1 --2- 172.21.15.88:0/3350142760 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9efc0731c0 0x7f9efc073600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:56.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.175+0000 7f9efbfff640 1 --2- 172.21.15.88:0/3350142760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9efc15a050 0x7f9efc15d460 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9eec0097a0 tx=0x7f9eec02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6488f8d927253b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:56.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.175+0000 7f9efaffd640 1 -- 172.21.15.88:0/3350142760 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eec037070 con 0x7f9efc15a050 2024-09-16T01:34:56.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.175+0000 7f9efaffd640 1 -- 172.21.15.88:0/3350142760 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9eec02ae50 con 0x7f9efc15a050 2024-09-16T01:34:56.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.175+0000 7f9efaffd640 1 -- 172.21.15.88:0/3350142760 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eec032660 con 0x7f9efc15a050 2024-09-16T01:34:56.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.175+0000 7f9f028dd640 1 -- 172.21.15.88:0/3350142760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9efc15a050 msgr2=0x7f9efc15d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:56.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.175+0000 7f9f028dd640 1 --2- 172.21.15.88:0/3350142760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9efc15a050 0x7f9efc15d460 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9eec0097a0 tx=0x7f9eec02a100 comp rx=0 tx=0).stop 2024-09-16T01:34:56.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.176+0000 7f9f028dd640 1 -- 172.21.15.88:0/3350142760 shutdown_connections 2024-09-16T01:34:56.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.176+0000 7f9f028dd640 1 --2- 172.21.15.88:0/3350142760 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9efc0731c0 0x7f9efc073600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.176+0000 7f9f028dd640 1 --2- 172.21.15.88:0/3350142760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9efc15a050 0x7f9efc15d460 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.176+0000 7f9f028dd640 1 -- 172.21.15.88:0/3350142760 >> 172.21.15.88:0/3350142760 conn(0x7f9efc06d0f0 msgr2=0x7f9efc06d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:56.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.176+0000 7f9f028dd640 1 -- 172.21.15.88:0/3350142760 shutdown_connections 2024-09-16T01:34:56.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.176+0000 7f9f028dd640 1 -- 172.21.15.88:0/3350142760 wait complete. 2024-09-16T01:34:56.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.177+0000 7f9f028dd640 1 Processor -- start 2024-09-16T01:34:56.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.177+0000 7f9f028dd640 1 -- start start 2024-09-16T01:34:56.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.178+0000 7f9f028dd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9efc0731c0 0x7f9efc070fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:56.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.178+0000 7f9f028dd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9efc15a050 0x7f9efc1f3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:56.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.178+0000 7f9f028dd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9efc15ffd0 0x7f9efc0aa5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:56.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.178+0000 7f9f028dd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9efc0cd940 con 0x7f9efc0731c0 2024-09-16T01:34:56.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.178+0000 7f9f028dd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9efc0cd660 con 0x7f9efc15a050 2024-09-16T01:34:56.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.178+0000 7f9f028dd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9efc0cdc20 con 0x7f9efc15ffd0 2024-09-16T01:34:56.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.178+0000 7f9efbfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9efc0731c0 0x7f9efc070fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:56.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.178+0000 7f9efbfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9efc0731c0 0x7f9efc070fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55538/0 (socket says 172.21.15.88:55538) 2024-09-16T01:34:56.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.178+0000 7f9efbfff640 1 -- 172.21.15.88:0/2316333826 learned_addr learned my addr 172.21.15.88:0/2316333826 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:56.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.178+0000 7f9f00e7a640 1 --2- 172.21.15.88:0/2316333826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9efc15ffd0 0x7f9efc0aa5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:56.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.178+0000 7f9efbfff640 1 -- 172.21.15.88:0/2316333826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9efc15ffd0 msgr2=0x7f9efc0aa5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:56.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.178+0000 7f9efbfff640 1 --2- 172.21.15.88:0/2316333826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9efc15ffd0 0x7f9efc0aa5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.178+0000 7f9efbfff640 1 -- 172.21.15.88:0/2316333826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9efc15a050 msgr2=0x7f9efc1f3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:56.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.178+0000 7f9efbfff640 1 --2- 172.21.15.88:0/2316333826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9efc15a050 0x7f9efc1f3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.178+0000 7f9efbfff640 1 -- 172.21.15.88:0/2316333826 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9efc072710 con 0x7f9efc0731c0 2024-09-16T01:34:56.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.178+0000 7f9efbfff640 1 --2- 172.21.15.88:0/2316333826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9efc0731c0 0x7f9efc070fb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9eec035040 tx=0x7f9eec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:56.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.178+0000 7f9ef8ff9640 1 -- 172.21.15.88:0/2316333826 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eec041070 con 0x7f9efc0731c0 2024-09-16T01:34:56.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.178+0000 7f9ef8ff9640 1 -- 172.21.15.88:0/2316333826 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9eec03c670 con 0x7f9efc0731c0 2024-09-16T01:34:56.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.178+0000 7f9ef8ff9640 1 -- 172.21.15.88:0/2316333826 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eec037040 con 0x7f9efc0731c0 2024-09-16T01:34:56.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.178+0000 7f9f028dd640 1 -- 172.21.15.88:0/2316333826 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9efc0aab10 con 0x7f9efc0731c0 2024-09-16T01:34:56.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.179+0000 7f9f028dd640 1 -- 172.21.15.88:0/2316333826 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9efc071950 con 0x7f9efc0731c0 2024-09-16T01:34:56.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.179+0000 7f9f028dd640 1 -- 172.21.15.88:0/2316333826 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9efc0cc580 con 0x7f9efc0731c0 2024-09-16T01:34:56.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.180+0000 7f9ef8ff9640 1 -- 172.21.15.88:0/2316333826 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9eec03a070 con 0x7f9efc0731c0 2024-09-16T01:34:56.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.181+0000 7f9ef8ff9640 1 --2- 172.21.15.88:0/2316333826 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9ee806c2b0 0x7f9ee806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:56.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.181+0000 7f9efb7fe640 1 --2- 172.21.15.88:0/2316333826 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9ee806c2b0 0x7f9ee806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:56.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.181+0000 7f9ef8ff9640 1 -- 172.21.15.88:0/2316333826 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7f9eec0ab440 con 0x7f9efc0731c0 2024-09-16T01:34:56.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.181+0000 7f9efb7fe640 1 --2- 172.21.15.88:0/2316333826 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9ee806c2b0 0x7f9ee806e770 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0006fd0 tx=0x7f9ef0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:56.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.185+0000 7f9ef8ff9640 1 -- 172.21.15.88:0/2316333826 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9efc0cc580 con 0x7f9efc0731c0 2024-09-16T01:34:56.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.442+0000 7f9f028dd640 1 -- 172.21.15.88:0/2316333826 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7f9efc0b36e0 con 0x7f9efc0731c0 2024-09-16T01:34:56.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.443+0000 7f9ef8ff9640 1 -- 172.21.15.88:0/2316333826 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7f9efc0aab10 con 0x7f9efc0731c0 2024-09-16T01:34:56.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.446+0000 7f9f028dd640 1 -- 172.21.15.88:0/2316333826 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9ee806c2b0 msgr2=0x7f9ee806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:56.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.446+0000 7f9f028dd640 1 --2- 172.21.15.88:0/2316333826 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9ee806c2b0 0x7f9ee806e770 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0006fd0 tx=0x7f9ef0008040 comp rx=0 tx=0).stop 2024-09-16T01:34:56.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.446+0000 7f9f028dd640 1 -- 172.21.15.88:0/2316333826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9efc0731c0 msgr2=0x7f9efc070fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:56.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.446+0000 7f9f028dd640 1 --2- 172.21.15.88:0/2316333826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9efc0731c0 0x7f9efc070fb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9eec035040 tx=0x7f9eec0046c0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.446+0000 7f9f028dd640 1 -- 172.21.15.88:0/2316333826 shutdown_connections 2024-09-16T01:34:56.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.446+0000 7f9f028dd640 1 --2- 172.21.15.88:0/2316333826 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9ee806c2b0 0x7f9ee806e770 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.446+0000 7f9f028dd640 1 --2- 172.21.15.88:0/2316333826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9efc15ffd0 0x7f9efc0aa5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.446+0000 7f9f028dd640 1 --2- 172.21.15.88:0/2316333826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9efc15a050 0x7f9efc1f3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.446+0000 7f9f028dd640 1 --2- 172.21.15.88:0/2316333826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9efc0731c0 0x7f9efc070fb0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.446+0000 7f9f028dd640 1 -- 172.21.15.88:0/2316333826 >> 172.21.15.88:0/2316333826 conn(0x7f9efc06d0f0 msgr2=0x7f9efc15a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:56.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.446+0000 7f9f028dd640 1 -- 172.21.15.88:0/2316333826 shutdown_connections 2024-09-16T01:34:56.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.446+0000 7f9f028dd640 1 -- 172.21.15.88:0/2316333826 wait complete. 2024-09-16T01:34:56.459 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-09-16T01:34:56.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.557+0000 7efd2b482640 1 Processor -- start 2024-09-16T01:34:56.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.557+0000 7efd2b482640 1 -- start start 2024-09-16T01:34:56.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.558+0000 7efd2b482640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efd24073160 0x7efd240b6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:56.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.558+0000 7efd2b482640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd240b74a0 0x7efd240b5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:56.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.558+0000 7efd2b482640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd240d0b20 con 0x7efd2415c770 2024-09-16T01:34:56.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.558+0000 7efd2b482640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7efd240b7910 con 0x7efd24073160 2024-09-16T01:34:56.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.558+0000 7efd2b482640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efd240d2310 con 0x7efd240b74a0 2024-09-16T01:34:56.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.558+0000 7efd2921e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7efd2415c770 0x7efd240d0320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33662/0 (socket says 172.21.15.88:33662) 2024-09-16T01:34:56.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.558+0000 7efd2921e640 1 -- 172.21.15.88:0/2117144078 learned_addr learned my addr 172.21.15.88:0/2117144078 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:56.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.559+0000 7efd29a1f640 1 --2- 172.21.15.88:0/2117144078 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd240b74a0 0x7efd240b5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:56.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.559+0000 7efd28a1d640 1 --2- 172.21.15.88:0/2117144078 >> v2:172.21.15.154:3300/0 conn(0x7efd24073160 0x7efd240b6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:56.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.559+0000 7efd17fff640 1 -- 172.21.15.88:0/2117144078 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2935290847 0 0) 0x7efd240d0b20 con 0x7efd2415c770 2024-09-16T01:34:56.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.559+0000 7efd17fff640 1 -- 172.21.15.88:0/2117144078 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd10003520 con 0x7efd2415c770 2024-09-16T01:34:56.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.559+0000 7efd29a1f640 1 -- 172.21.15.88:0/2117144078 >> v2:172.21.15.154:3300/0 conn(0x7efd24073160 msgr2=0x7efd240b6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:56.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.559+0000 7efd29a1f640 1 --2- 172.21.15.88:0/2117144078 >> v2:172.21.15.154:3300/0 conn(0x7efd24073160 0x7efd240b6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.559+0000 7efd17fff640 1 -- 172.21.15.88:0/2117144078 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 379420299 0 0) 0x7efd10003520 con 0x7efd2415c770 2024-09-16T01:34:56.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.559+0000 7efd29a1f640 1 -- 172.21.15.88:0/2117144078 >> v1:172.21.15.88:6789/0 conn(0x7efd2415c770 legacy=0x7efd240d0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:56.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.559+0000 7efd29a1f640 1 -- 172.21.15.88:0/2117144078 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd240d0e00 con 0x7efd240b74a0 2024-09-16T01:34:56.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.560+0000 7efd29a1f640 1 --2- 172.21.15.88:0/2117144078 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd240b74a0 0x7efd240b5820 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efd1c009d90 tx=0x7efd1c02a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=497275c84d5cb5bf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:56.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.560+0000 7efd17fff640 1 -- 172.21.15.88:0/2117144078 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd18001670 con 0x7efd240b74a0 2024-09-16T01:34:56.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.560+0000 7efd17fff640 1 -- 172.21.15.88:0/2117144078 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd1c007540 con 0x7efd240b74a0 2024-09-16T01:34:56.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.560+0000 7efd17fff640 1 -- 172.21.15.88:0/2117144078 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1c007910 con 0x7efd240b74a0 2024-09-16T01:34:56.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.560+0000 7efd2b482640 1 -- 172.21.15.88:0/2117144078 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd240b74a0 msgr2=0x7efd240b5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:56.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.560+0000 7efd2b482640 1 --2- 172.21.15.88:0/2117144078 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd240b74a0 0x7efd240b5820 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efd1c009d90 tx=0x7efd1c02a5f0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.560+0000 7efd2b482640 1 -- 172.21.15.88:0/2117144078 shutdown_connections 2024-09-16T01:34:56.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.560+0000 7efd2b482640 1 --2- 172.21.15.88:0/2117144078 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd240b74a0 0x7efd240b5820 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.560+0000 7efd2b482640 1 --2- 172.21.15.88:0/2117144078 >> v2:172.21.15.154:3300/0 conn(0x7efd24073160 0x7efd240b6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.561+0000 7efd2b482640 1 -- 172.21.15.88:0/2117144078 >> 172.21.15.88:0/2117144078 conn(0x7efd240d24d0 msgr2=0x7efd240d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:56.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.561+0000 7efd2b482640 1 -- 172.21.15.88:0/2117144078 shutdown_connections 2024-09-16T01:34:56.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.561+0000 7efd2b482640 1 -- 172.21.15.88:0/2117144078 wait complete. 2024-09-16T01:34:56.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.562+0000 7efd2b482640 1 Processor -- start 2024-09-16T01:34:56.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.562+0000 7efd2b482640 1 -- start start 2024-09-16T01:34:56.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.562+0000 7efd2b482640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd24073160 0x7efd241e6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:56.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.562+0000 7efd2b482640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd240b74a0 0x7efd24145ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:56.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.562+0000 7efd2b482640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd2415c770 0x7efd24154ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:56.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.562+0000 7efd2b482640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7efd240b5d60 con 0x7efd24073160 2024-09-16T01:34:56.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.563+0000 7efd2b482640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efd240d13c0 con 0x7efd240b74a0 2024-09-16T01:34:56.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.563+0000 7efd2b482640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efd240b5ee0 con 0x7efd2415c770 2024-09-16T01:34:56.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.563+0000 7efd2921e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd24073160 0x7efd241e6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:56.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.563+0000 7efd2921e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd24073160 0x7efd241e6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55540/0 (socket says 172.21.15.88:55540) 2024-09-16T01:34:56.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.563+0000 7efd2921e640 1 -- 172.21.15.88:0/2985106772 learned_addr learned my addr 172.21.15.88:0/2985106772 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:56.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.563+0000 7efd29a1f640 1 --2- 172.21.15.88:0/2985106772 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd2415c770 0x7efd24154ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:56.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.563+0000 7efd28a1d640 1 --2- 172.21.15.88:0/2985106772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd240b74a0 0x7efd24145ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:56.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.563+0000 7efd2921e640 1 -- 172.21.15.88:0/2985106772 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd2415c770 msgr2=0x7efd24154ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:56.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.563+0000 7efd2921e640 1 --2- 172.21.15.88:0/2985106772 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd2415c770 0x7efd24154ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.563+0000 7efd2921e640 1 -- 172.21.15.88:0/2985106772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd240b74a0 msgr2=0x7efd24145ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:56.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.563+0000 7efd2921e640 1 --2- 172.21.15.88:0/2985106772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd240b74a0 0x7efd24145ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.563+0000 7efd2921e640 1 -- 172.21.15.88:0/2985106772 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd241e5970 con 0x7efd24073160 2024-09-16T01:34:56.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.563+0000 7efd2921e640 1 --2- 172.21.15.88:0/2985106772 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd24073160 0x7efd241e6a10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7efd1800c700 tx=0x7efd1800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:56.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.563+0000 7efd15ffb640 1 -- 172.21.15.88:0/2985106772 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd18017070 con 0x7efd24073160 2024-09-16T01:34:56.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.563+0000 7efd15ffb640 1 -- 172.21.15.88:0/2985106772 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd18005ad0 con 0x7efd24073160 2024-09-16T01:34:56.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.563+0000 7efd2b482640 1 -- 172.21.15.88:0/2985106772 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd241e5c50 con 0x7efd24073160 2024-09-16T01:34:56.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.563+0000 7efd15ffb640 1 -- 172.21.15.88:0/2985106772 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd18007070 con 0x7efd24073160 2024-09-16T01:34:56.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.564+0000 7efd2b482640 1 -- 172.21.15.88:0/2985106772 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd241e6210 con 0x7efd24073160 2024-09-16T01:34:56.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.565+0000 7efd15ffb640 1 -- 172.21.15.88:0/2985106772 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7efd1801a070 con 0x7efd24073160 2024-09-16T01:34:56.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.565+0000 7efd2b482640 1 -- 172.21.15.88:0/2985106772 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd241596d0 con 0x7efd24073160 2024-09-16T01:34:56.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.565+0000 7efd15ffb640 1 --2- 172.21.15.88:0/2985106772 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efd0806c2b0 0x7efd0806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:56.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.565+0000 7efd15ffb640 1 -- 172.21.15.88:0/2985106772 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7efd1808a440 con 0x7efd24073160 2024-09-16T01:34:56.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.565+0000 7efd28a1d640 1 --2- 172.21.15.88:0/2985106772 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efd0806c2b0 0x7efd0806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:56.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.566+0000 7efd28a1d640 1 --2- 172.21.15.88:0/2985106772 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efd0806c2b0 0x7efd0806e770 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efd241e8670 tx=0x7efd20009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:56.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.570+0000 7efd15ffb640 1 -- 172.21.15.88:0/2985106772 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7efd18058940 con 0x7efd24073160 2024-09-16T01:34:56.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.876+0000 7efd2b482640 1 -- 172.21.15.88:0/2985106772 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7efd241e8ad0 con 0x7efd24073160 2024-09-16T01:34:56.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.878+0000 7efd15ffb640 1 -- 172.21.15.88:0/2985106772 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7efd1805cc40 con 0x7efd24073160 2024-09-16T01:34:56.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.881+0000 7efd2b482640 1 -- 172.21.15.88:0/2985106772 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efd0806c2b0 msgr2=0x7efd0806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:56.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.881+0000 7efd2b482640 1 --2- 172.21.15.88:0/2985106772 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efd0806c2b0 0x7efd0806e770 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efd241e8670 tx=0x7efd20009190 comp rx=0 tx=0).stop 2024-09-16T01:34:56.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.881+0000 7efd2b482640 1 -- 172.21.15.88:0/2985106772 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd24073160 msgr2=0x7efd241e6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:56.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.881+0000 7efd2b482640 1 --2- 172.21.15.88:0/2985106772 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd24073160 0x7efd241e6a10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7efd1800c700 tx=0x7efd1800cbc0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.881+0000 7efd2b482640 1 -- 172.21.15.88:0/2985106772 shutdown_connections 2024-09-16T01:34:56.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.881+0000 7efd2b482640 1 --2- 172.21.15.88:0/2985106772 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efd0806c2b0 0x7efd0806e770 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.881+0000 7efd2b482640 1 --2- 172.21.15.88:0/2985106772 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd2415c770 0x7efd24154ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.881+0000 7efd2b482640 1 --2- 172.21.15.88:0/2985106772 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd240b74a0 0x7efd24145ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.881+0000 7efd2b482640 1 --2- 172.21.15.88:0/2985106772 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd24073160 0x7efd241e6a10 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.881+0000 7efd2b482640 1 -- 172.21.15.88:0/2985106772 >> 172.21.15.88:0/2985106772 conn(0x7efd240d24d0 msgr2=0x7efd24073640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:56.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.881+0000 7efd2b482640 1 -- 172.21.15.88:0/2985106772 shutdown_connections 2024-09-16T01:34:56.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.881+0000 7efd2b482640 1 -- 172.21.15.88:0/2985106772 wait complete. 2024-09-16T01:34:56.893 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-09-16T01:34:56.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.991+0000 7f6be79d6640 1 Processor -- start 2024-09-16T01:34:56.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.991+0000 7f6be79d6640 1 -- start start 2024-09-16T01:34:56.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.992+0000 7f6be79d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6be015c2a0 0x7f6be00735d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:56.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.992+0000 7f6be79d6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6be006fe20 0x7f6be006bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:56.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.992+0000 7f6be79d6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6be0074120 con 0x7f6be0073b10 2024-09-16T01:34:56.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.992+0000 7f6be79d6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6be015c680 con 0x7f6be006fe20 2024-09-16T01:34:56.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.992+0000 7f6be79d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6be015ef00 con 0x7f6be015c2a0 2024-09-16T01:34:56.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.992+0000 7f6be5772640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6be015c2a0 0x7f6be00735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:56.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.992+0000 7f6be5f73640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6be006fe20 0x7f6be006bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:56.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.992+0000 7f6be5772640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6be015c2a0 0x7f6be00735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:45868/0 (socket says 172.21.15.88:45868) 2024-09-16T01:34:56.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.992+0000 7f6be5772640 1 -- 172.21.15.88:0/3234694221 learned_addr learned my addr 172.21.15.88:0/3234694221 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:56.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.992+0000 7f6be5772640 1 -- 172.21.15.88:0/3234694221 >> v2:172.21.15.154:3300/0 conn(0x7f6be006fe20 msgr2=0x7f6be006bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:56.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.992+0000 7f6be5772640 1 --2- 172.21.15.88:0/3234694221 >> v2:172.21.15.154:3300/0 conn(0x7f6be006fe20 0x7f6be006bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.992+0000 7f6be5772640 1 -- 172.21.15.88:0/3234694221 >> v1:172.21.15.154:6789/0 conn(0x7f6be0073b10 legacy=0x7f6be006f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:56.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.992+0000 7f6be5772640 1 -- 172.21.15.88:0/3234694221 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6be015f1e0 con 0x7f6be015c2a0 2024-09-16T01:34:56.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.993+0000 7f6be5772640 1 --2- 172.21.15.88:0/3234694221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6be015c2a0 0x7f6be00735d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc009870 tx=0x7f6bdc02a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4228329db186847c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:56.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.993+0000 7f6bd3fff640 1 -- 172.21.15.88:0/3234694221 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc037070 con 0x7f6be015c2a0 2024-09-16T01:34:56.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.993+0000 7f6bd3fff640 1 -- 172.21.15.88:0/3234694221 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bdc032440 con 0x7f6be015c2a0 2024-09-16T01:34:56.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.993+0000 7f6bd3fff640 1 -- 172.21.15.88:0/3234694221 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc032810 con 0x7f6be015c2a0 2024-09-16T01:34:56.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.994+0000 7f6be79d6640 1 -- 172.21.15.88:0/3234694221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6be015c2a0 msgr2=0x7f6be00735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:56.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.994+0000 7f6be79d6640 1 --2- 172.21.15.88:0/3234694221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6be015c2a0 0x7f6be00735d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc009870 tx=0x7f6bdc02a270 comp rx=0 tx=0).stop 2024-09-16T01:34:56.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.994+0000 7f6be79d6640 1 -- 172.21.15.88:0/3234694221 shutdown_connections 2024-09-16T01:34:56.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.994+0000 7f6be79d6640 1 --2- 172.21.15.88:0/3234694221 >> v2:172.21.15.154:3300/0 conn(0x7f6be006fe20 0x7f6be006bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.994+0000 7f6be79d6640 1 --2- 172.21.15.88:0/3234694221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6be015c2a0 0x7f6be00735d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:56.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.994+0000 7f6be79d6640 1 -- 172.21.15.88:0/3234694221 >> 172.21.15.88:0/3234694221 conn(0x7f6be00b3070 msgr2=0x7f6be00b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:56.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.994+0000 7f6be79d6640 1 -- 172.21.15.88:0/3234694221 shutdown_connections 2024-09-16T01:34:56.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.994+0000 7f6be79d6640 1 -- 172.21.15.88:0/3234694221 wait complete. 2024-09-16T01:34:56.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.995+0000 7f6be79d6640 1 Processor -- start 2024-09-16T01:34:56.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.995+0000 7f6be79d6640 1 -- start start 2024-09-16T01:34:56.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.996+0000 7f6be79d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6be006fe20 0x7f6be01531b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:56.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.996+0000 7f6be79d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6be0073b10 0x7f6be0149c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:56.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.996+0000 7f6be79d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6be015c2a0 0x7f6be01581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:56.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.996+0000 7f6be79d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6be00b5f20 con 0x7f6be015c2a0 2024-09-16T01:34:56.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.996+0000 7f6be79d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6be015ebf0 con 0x7f6be006fe20 2024-09-16T01:34:56.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.996+0000 7f6be79d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6be00b60a0 con 0x7f6be0073b10 2024-09-16T01:34:56.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.996+0000 7f6be5f73640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6be015c2a0 0x7f6be01581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:56.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.996+0000 7f6be5f73640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6be015c2a0 0x7f6be01581b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55548/0 (socket says 172.21.15.88:55548) 2024-09-16T01:34:56.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.996+0000 7f6be5f73640 1 -- 172.21.15.88:0/4144094331 learned_addr learned my addr 172.21.15.88:0/4144094331 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:57.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.996+0000 7f6be4f71640 1 --2- 172.21.15.88:0/4144094331 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6be0073b10 0x7f6be0149c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:57.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.996+0000 7f6be5772640 1 --2- 172.21.15.88:0/4144094331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6be006fe20 0x7f6be01531b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:57.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.996+0000 7f6be5f73640 1 -- 172.21.15.88:0/4144094331 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6be0073b10 msgr2=0x7f6be0149c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:57.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.996+0000 7f6be5f73640 1 --2- 172.21.15.88:0/4144094331 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6be0073b10 0x7f6be0149c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.996+0000 7f6be5f73640 1 -- 172.21.15.88:0/4144094331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6be006fe20 msgr2=0x7f6be01531b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:57.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.996+0000 7f6be5f73640 1 --2- 172.21.15.88:0/4144094331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6be006fe20 0x7f6be01531b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.996+0000 7f6be5f73640 1 -- 172.21.15.88:0/4144094331 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6be0160830 con 0x7f6be015c2a0 2024-09-16T01:34:57.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.996+0000 7f6be5f73640 1 --2- 172.21.15.88:0/4144094331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6be015c2a0 0x7f6be01581b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6bd800efc0 tx=0x7f6bd800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:57.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.997+0000 7f6bd1ffb640 1 -- 172.21.15.88:0/4144094331 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8018020 con 0x7f6be015c2a0 2024-09-16T01:34:57.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.997+0000 7f6bd1ffb640 1 -- 172.21.15.88:0/4144094331 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bd8014480 con 0x7f6be015c2a0 2024-09-16T01:34:57.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.997+0000 7f6be79d6640 1 -- 172.21.15.88:0/4144094331 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6be0161800 con 0x7f6be015c2a0 2024-09-16T01:34:57.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.997+0000 7f6bd1ffb640 1 -- 172.21.15.88:0/4144094331 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8014870 con 0x7f6be015c2a0 2024-09-16T01:34:57.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.997+0000 7f6be79d6640 1 -- 172.21.15.88:0/4144094331 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6be014a190 con 0x7f6be015c2a0 2024-09-16T01:34:57.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.998+0000 7f6be79d6640 1 -- 172.21.15.88:0/4144094331 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6be015fa70 con 0x7f6be015c2a0 2024-09-16T01:34:57.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.998+0000 7f6bd1ffb640 1 -- 172.21.15.88:0/4144094331 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6bd8005d10 con 0x7f6be015c2a0 2024-09-16T01:34:57.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.999+0000 7f6bd1ffb640 1 --2- 172.21.15.88:0/4144094331 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6bc406c2b0 0x7f6bc406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:57.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.999+0000 7f6bd1ffb640 1 -- 172.21.15.88:0/4144094331 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7f6bd808bff0 con 0x7f6be015c2a0 2024-09-16T01:34:57.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.999+0000 7f6be5772640 1 --2- 172.21.15.88:0/4144094331 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6bc406c2b0 0x7f6bc406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:57.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:56.999+0000 7f6be5772640 1 --2- 172.21.15.88:0/4144094331 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6bc406c2b0 0x7f6bc406e770 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc034000 tx=0x7f6bdc02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:57.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.003+0000 7f6bd1ffb640 1 -- 172.21.15.88:0/4144094331 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6bd8022030 con 0x7f6be015c2a0 2024-09-16T01:34:57.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.230+0000 7f6be79d6640 1 -- 172.21.15.88:0/4144094331 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f6be00b1010 con 0x7f6be015c2a0 2024-09-16T01:34:57.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.232+0000 7f6bd1ffb640 1 -- 172.21.15.88:0/4144094331 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7f6bd8010040 con 0x7f6be015c2a0 2024-09-16T01:34:57.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.235+0000 7f6be79d6640 1 -- 172.21.15.88:0/4144094331 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6bc406c2b0 msgr2=0x7f6bc406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:57.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.235+0000 7f6be79d6640 1 --2- 172.21.15.88:0/4144094331 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6bc406c2b0 0x7f6bc406e770 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc034000 tx=0x7f6bdc02e000 comp rx=0 tx=0).stop 2024-09-16T01:34:57.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.235+0000 7f6be79d6640 1 -- 172.21.15.88:0/4144094331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6be015c2a0 msgr2=0x7f6be01581b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:57.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.235+0000 7f6be79d6640 1 --2- 172.21.15.88:0/4144094331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6be015c2a0 0x7f6be01581b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6bd800efc0 tx=0x7f6bd800c490 comp rx=0 tx=0).stop 2024-09-16T01:34:57.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.235+0000 7f6be79d6640 1 -- 172.21.15.88:0/4144094331 shutdown_connections 2024-09-16T01:34:57.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.235+0000 7f6be79d6640 1 --2- 172.21.15.88:0/4144094331 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6bc406c2b0 0x7f6bc406e770 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.235+0000 7f6be79d6640 1 --2- 172.21.15.88:0/4144094331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6be015c2a0 0x7f6be01581b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.235+0000 7f6be79d6640 1 --2- 172.21.15.88:0/4144094331 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6be0073b10 0x7f6be0149c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.235+0000 7f6be79d6640 1 --2- 172.21.15.88:0/4144094331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6be006fe20 0x7f6be01531b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.235+0000 7f6be79d6640 1 -- 172.21.15.88:0/4144094331 >> 172.21.15.88:0/4144094331 conn(0x7f6be00b3070 msgr2=0x7f6be00b1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:57.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.235+0000 7f6be79d6640 1 -- 172.21.15.88:0/4144094331 shutdown_connections 2024-09-16T01:34:57.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.235+0000 7f6be79d6640 1 -- 172.21.15.88:0/4144094331 wait complete. 2024-09-16T01:34:57.247 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-09-16T01:34:57.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.345+0000 7f560549b640 1 Processor -- start 2024-09-16T01:34:57.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.345+0000 7f560549b640 1 -- start start 2024-09-16T01:34:57.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.346+0000 7f560549b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56000b68f0 0x7f56001696f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:57.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.346+0000 7f560549b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56000cf860 con 0x7f56000cf1f0 2024-09-16T01:34:57.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.346+0000 7f560549b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56000cfb40 con 0x7f560015e7a0 2024-09-16T01:34:57.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.346+0000 7f560549b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f56000d2ca0 con 0x7f56000b68f0 2024-09-16T01:34:57.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.346+0000 7f55f7fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f56000cf1f0 0x7f56000b61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33676/0 (socket says 172.21.15.88:33676) 2024-09-16T01:34:57.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.346+0000 7f55f7fff640 1 -- 172.21.15.88:0/327138231 learned_addr learned my addr 172.21.15.88:0/327138231 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:57.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.346+0000 7f55ff7fe640 1 --2- 172.21.15.88:0/327138231 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56000b68f0 0x7f56001696f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:57.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.346+0000 7f55fe7fc640 1 -- 172.21.15.88:0/327138231 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 400762448 0 0) 0x7f56000cf860 con 0x7f56000cf1f0 2024-09-16T01:34:57.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.346+0000 7f55fe7fc640 1 -- 172.21.15.88:0/327138231 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55ec003520 con 0x7f56000cf1f0 2024-09-16T01:34:57.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.346+0000 7f55fe7fc640 1 -- 172.21.15.88:0/327138231 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2477759983 0 0) 0x7f55ec003520 con 0x7f56000cf1f0 2024-09-16T01:34:57.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.346+0000 7f55fe7fc640 1 -- 172.21.15.88:0/327138231 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56000b68f0 msgr2=0x7f56001696f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:57.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.346+0000 7f55fe7fc640 1 --2- 172.21.15.88:0/327138231 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56000b68f0 0x7f56001696f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.346+0000 7f55fe7fc640 1 -- 172.21.15.88:0/327138231 >> v1:172.21.15.154:6789/0 conn(0x7f560015e7a0 legacy=0x7f560006caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:57.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.346+0000 7f55fe7fc640 1 -- 172.21.15.88:0/327138231 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56000cfe20 con 0x7f56000cf1f0 2024-09-16T01:34:57.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.346+0000 7f55fe7fc640 1 -- 172.21.15.88:0/327138231 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f55e8001670 con 0x7f56000cf1f0 2024-09-16T01:34:57.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.347+0000 7f55fe7fc640 1 -- 172.21.15.88:0/327138231 >> v1:172.21.15.88:6789/0 conn(0x7f56000cf1f0 legacy=0x7f56000b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:57.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.347+0000 7f55fe7fc640 1 --2- 172.21.15.88:0/327138231 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56000b68f0 0x7f55ec0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:57.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.347+0000 7f55fe7fc640 1 --2- 172.21.15.88:0/327138231 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55ec003f60 0x7f55ec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:57.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.347+0000 7f55fe7fc640 1 --2- 172.21.15.88:0/327138231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55ec004d40 0x7f55ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:57.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.347+0000 7f55fe7fc640 1 -- 172.21.15.88:0/327138231 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f56000d2ca0 con 0x7f55ec003f60 2024-09-16T01:34:57.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.347+0000 7f55fe7fc640 1 -- 172.21.15.88:0/327138231 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f55ec00bb50 con 0x7f55ec004d40 2024-09-16T01:34:57.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.347+0000 7f55fe7fc640 1 -- 172.21.15.88:0/327138231 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f55ec00bd50 con 0x7f56000b68f0 2024-09-16T01:34:57.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.347+0000 7f55f7fff640 1 --2- 172.21.15.88:0/327138231 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55ec003f60 0x7f55ec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:57.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.347+0000 7f55feffd640 1 --2- 172.21.15.88:0/327138231 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56000b68f0 0x7f55ec0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:57.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.347+0000 7f55ff7fe640 1 --2- 172.21.15.88:0/327138231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55ec004d40 0x7f55ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:57.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.347+0000 7f55f7fff640 1 -- 172.21.15.88:0/327138231 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56000b68f0 msgr2=0x7f55ec0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:57.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.347+0000 7f55f7fff640 1 --2- 172.21.15.88:0/327138231 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56000b68f0 0x7f55ec0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.347+0000 7f55f7fff640 1 -- 172.21.15.88:0/327138231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55ec004d40 msgr2=0x7f55ec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:57.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.347+0000 7f55f7fff640 1 --2- 172.21.15.88:0/327138231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55ec004d40 0x7f55ec0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.347+0000 7f55f7fff640 1 -- 172.21.15.88:0/327138231 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55ec003520 con 0x7f55ec003f60 2024-09-16T01:34:57.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.347+0000 7f55feffd640 1 --2- 172.21.15.88:0/327138231 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56000b68f0 0x7f55ec0039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:57.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.348+0000 7f55f7fff640 1 --2- 172.21.15.88:0/327138231 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55ec003f60 0x7f55ec004800 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f55e800c210 tx=0x7f55ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=107722603907624a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:57.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.348+0000 7f55fe7fc640 1 -- 172.21.15.88:0/327138231 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e8007d30 con 0x7f55ec003f60 2024-09-16T01:34:57.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.348+0000 7f55fe7fc640 1 -- 172.21.15.88:0/327138231 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55e8005ad0 con 0x7f55ec003f60 2024-09-16T01:34:57.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.348+0000 7f55fe7fc640 1 -- 172.21.15.88:0/327138231 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e8007070 con 0x7f55ec003f60 2024-09-16T01:34:57.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.348+0000 7f560549b640 1 -- 172.21.15.88:0/327138231 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55ec003f60 msgr2=0x7f55ec004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:57.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.348+0000 7f560549b640 1 --2- 172.21.15.88:0/327138231 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55ec003f60 0x7f55ec004800 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f55e800c210 tx=0x7f55ec005830 comp rx=0 tx=0).stop 2024-09-16T01:34:57.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.348+0000 7f55ff7fe640 1 -- 172.21.15.88:0/327138231 reap_dead start 2024-09-16T01:34:57.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.348+0000 7f560549b640 1 -- 172.21.15.88:0/327138231 shutdown_connections 2024-09-16T01:34:57.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.348+0000 7f560549b640 1 -- 172.21.15.88:0/327138231 >> 172.21.15.88:0/327138231 conn(0x7f56000d11e0 msgr2=0x7f56000d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:57.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.348+0000 7f560549b640 1 -- 172.21.15.88:0/327138231 shutdown_connections 2024-09-16T01:34:57.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.348+0000 7f560549b640 1 -- 172.21.15.88:0/327138231 wait complete. 2024-09-16T01:34:57.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.350+0000 7f560549b640 1 Processor -- start 2024-09-16T01:34:57.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.350+0000 7f560549b640 1 -- start start 2024-09-16T01:34:57.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.350+0000 7f560549b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55ec003f60 0x7f5600158510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:57.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.350+0000 7f560549b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56000ce510 0x7f56000ce990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:57.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.350+0000 7f560549b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56000cbd40 0x7f56000cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:57.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.350+0000 7f560549b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f56000d4420 con 0x7f56000cbd40 2024-09-16T01:34:57.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.350+0000 7f560549b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f56000b7a00 con 0x7f55ec003f60 2024-09-16T01:34:57.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.350+0000 7f560549b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f56000cc700 con 0x7f56000ce510 2024-09-16T01:34:57.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.350+0000 7f55ff7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56000cbd40 0x7f56000cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:57.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.350+0000 7f55ff7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56000cbd40 0x7f56000cc1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55564/0 (socket says 172.21.15.88:55564) 2024-09-16T01:34:57.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.350+0000 7f55ff7fe640 1 -- 172.21.15.88:0/3514615354 learned_addr learned my addr 172.21.15.88:0/3514615354 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:57.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.350+0000 7f55f7fff640 1 --2- 172.21.15.88:0/3514615354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56000ce510 0x7f56000ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:57.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.351+0000 7f55feffd640 1 --2- 172.21.15.88:0/3514615354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55ec003f60 0x7f5600158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:57.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.351+0000 7f55ff7fe640 1 -- 172.21.15.88:0/3514615354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56000ce510 msgr2=0x7f56000ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:57.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.351+0000 7f55ff7fe640 1 --2- 172.21.15.88:0/3514615354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56000ce510 0x7f56000ce990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.351+0000 7f55ff7fe640 1 -- 172.21.15.88:0/3514615354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55ec003f60 msgr2=0x7f5600158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:57.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.351+0000 7f55ff7fe640 1 --2- 172.21.15.88:0/3514615354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55ec003f60 0x7f5600158510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.351+0000 7f55ff7fe640 1 -- 172.21.15.88:0/3514615354 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56001590d0 con 0x7f56000cbd40 2024-09-16T01:34:57.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.351+0000 7f55feffd640 1 --2- 172.21.15.88:0/3514615354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55ec003f60 0x7f5600158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:57.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.351+0000 7f55ff7fe640 1 --2- 172.21.15.88:0/3514615354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56000cbd40 0x7f56000cc1c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f560008d530 tx=0x7f55f800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:57.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.351+0000 7f55f77fe640 1 -- 172.21.15.88:0/3514615354 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f800fde0 con 0x7f56000cbd40 2024-09-16T01:34:57.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.351+0000 7f55f77fe640 1 -- 172.21.15.88:0/3514615354 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55f801d070 con 0x7f56000cbd40 2024-09-16T01:34:57.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.351+0000 7f55f77fe640 1 -- 172.21.15.88:0/3514615354 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f8015670 con 0x7f56000cbd40 2024-09-16T01:34:57.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.351+0000 7f560549b640 1 -- 172.21.15.88:0/3514615354 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56000cc920 con 0x7f56000cbd40 2024-09-16T01:34:57.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.351+0000 7f560549b640 1 -- 172.21.15.88:0/3514615354 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56000ccee0 con 0x7f56000cbd40 2024-09-16T01:34:57.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.352+0000 7f560549b640 1 -- 172.21.15.88:0/3514615354 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f560015eb80 con 0x7f56000cbd40 2024-09-16T01:34:57.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.352+0000 7f55f77fe640 1 -- 172.21.15.88:0/3514615354 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f55f8015810 con 0x7f56000cbd40 2024-09-16T01:34:57.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.353+0000 7f55f77fe640 1 --2- 172.21.15.88:0/3514615354 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f55e006c2e0 0x7f55e006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:57.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.353+0000 7f55f77fe640 1 -- 172.21.15.88:0/3514615354 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7f55f808b930 con 0x7f56000cbd40 2024-09-16T01:34:57.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.353+0000 7f55feffd640 1 --2- 172.21.15.88:0/3514615354 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f55e006c2e0 0x7f55e006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:57.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.353+0000 7f55feffd640 1 --2- 172.21.15.88:0/3514615354 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f55e006c2e0 0x7f55e006e7a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f560015b750 tx=0x7f55f0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:57.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.357+0000 7f55f77fe640 1 -- 172.21.15.88:0/3514615354 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f55f8010040 con 0x7f56000cbd40 2024-09-16T01:34:57.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.591+0000 7f560549b640 1 -- 172.21.15.88:0/3514615354 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7f56000d2a80 con 0x7f56000cbd40 2024-09-16T01:34:57.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.592+0000 7f55f77fe640 1 -- 172.21.15.88:0/3514615354 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) ==== 97+0+0 (secure 0 0 0) 0x7f55f805a3f0 con 0x7f56000cbd40 2024-09-16T01:34:57.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.595+0000 7f560549b640 1 -- 172.21.15.88:0/3514615354 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f55e006c2e0 msgr2=0x7f55e006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:57.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.595+0000 7f560549b640 1 --2- 172.21.15.88:0/3514615354 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f55e006c2e0 0x7f55e006e7a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f560015b750 tx=0x7f55f0009040 comp rx=0 tx=0).stop 2024-09-16T01:34:57.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.595+0000 7f560549b640 1 -- 172.21.15.88:0/3514615354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56000cbd40 msgr2=0x7f56000cc1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:57.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.595+0000 7f560549b640 1 --2- 172.21.15.88:0/3514615354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56000cbd40 0x7f56000cc1c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f560008d530 tx=0x7f55f800f580 comp rx=0 tx=0).stop 2024-09-16T01:34:57.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.595+0000 7f560549b640 1 -- 172.21.15.88:0/3514615354 shutdown_connections 2024-09-16T01:34:57.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.595+0000 7f560549b640 1 --2- 172.21.15.88:0/3514615354 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f55e006c2e0 0x7f55e006e7a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.595+0000 7f560549b640 1 --2- 172.21.15.88:0/3514615354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56000cbd40 0x7f56000cc1c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.595+0000 7f560549b640 1 --2- 172.21.15.88:0/3514615354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56000ce510 0x7f56000ce990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.595+0000 7f560549b640 1 --2- 172.21.15.88:0/3514615354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55ec003f60 0x7f5600158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.595+0000 7f560549b640 1 -- 172.21.15.88:0/3514615354 >> 172.21.15.88:0/3514615354 conn(0x7f56000d11e0 msgr2=0x7f56000b8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:57.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.595+0000 7f560549b640 1 -- 172.21.15.88:0/3514615354 shutdown_connections 2024-09-16T01:34:57.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.595+0000 7f560549b640 1 -- 172.21.15.88:0/3514615354 wait complete. 2024-09-16T01:34:57.606 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-09-16T01:34:57.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.705+0000 7f2875274640 1 Processor -- start 2024-09-16T01:34:57.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.705+0000 7f2875274640 1 -- start start 2024-09-16T01:34:57.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.706+0000 7f2875274640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f28700cf1f0 0x7f28700b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:57.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.706+0000 7f2875274640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28700cf860 con 0x7f287015e7a0 2024-09-16T01:34:57.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.706+0000 7f2875274640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28700cfb40 con 0x7f28700b6720 2024-09-16T01:34:57.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.706+0000 7f2875274640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f28700b6b90 con 0x7f28700cf1f0 2024-09-16T01:34:57.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.706+0000 7f286effd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f287015e7a0 0x7f287006caa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33692/0 (socket says 172.21.15.88:33692) 2024-09-16T01:34:57.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.706+0000 7f286effd640 1 -- 172.21.15.88:0/2749258564 learned_addr learned my addr 172.21.15.88:0/2749258564 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:57.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.706+0000 7f286dffb640 1 -- 172.21.15.88:0/2749258564 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1038338155 0 0) 0x7f28700cf860 con 0x7f287015e7a0 2024-09-16T01:34:57.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.706+0000 7f286dffb640 1 -- 172.21.15.88:0/2749258564 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2858003520 con 0x7f287015e7a0 2024-09-16T01:34:57.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.706+0000 7f286e7fc640 1 --2- 172.21.15.88:0/2749258564 >> v2:172.21.15.154:3300/0 conn(0x7f28700cf1f0 0x7f28700b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:57.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.706+0000 7f286dffb640 1 -- 172.21.15.88:0/2749258564 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 234082741 0 0) 0x7f2858003520 con 0x7f287015e7a0 2024-09-16T01:34:57.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.706+0000 7f286dffb640 1 -- 172.21.15.88:0/2749258564 >> v2:172.21.15.154:3300/0 conn(0x7f28700cf1f0 msgr2=0x7f28700b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:57.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.706+0000 7f286dffb640 1 --2- 172.21.15.88:0/2749258564 >> v2:172.21.15.154:3300/0 conn(0x7f28700cf1f0 0x7f28700b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.706+0000 7f286dffb640 1 -- 172.21.15.88:0/2749258564 >> v1:172.21.15.154:6789/0 conn(0x7f28700b6720 legacy=0x7f28700d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:57.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.706+0000 7f286dffb640 1 -- 172.21.15.88:0/2749258564 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28700cfe20 con 0x7f287015e7a0 2024-09-16T01:34:57.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.706+0000 7f286dffb640 1 -- 172.21.15.88:0/2749258564 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f2860001670 con 0x7f287015e7a0 2024-09-16T01:34:57.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.706+0000 7f286dffb640 1 -- 172.21.15.88:0/2749258564 >> v1:172.21.15.88:6789/0 conn(0x7f287015e7a0 legacy=0x7f287006caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:57.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.707+0000 7f286dffb640 1 --2- 172.21.15.88:0/2749258564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28580039b0 0x7f2858003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:57.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.707+0000 7f286dffb640 1 --2- 172.21.15.88:0/2749258564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2858004800 0x7f2858004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:57.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.707+0000 7f286dffb640 1 --2- 172.21.15.88:0/2749258564 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28580052e0 0x7f285800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:57.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.707+0000 7f286dffb640 1 -- 172.21.15.88:0/2749258564 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f28700b6b90 con 0x7f28580039b0 2024-09-16T01:34:57.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.707+0000 7f286dffb640 1 -- 172.21.15.88:0/2749258564 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f285800c190 con 0x7f2858004800 2024-09-16T01:34:57.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.707+0000 7f286dffb640 1 -- 172.21.15.88:0/2749258564 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f285800c390 con 0x7f28580052e0 2024-09-16T01:34:57.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.707+0000 7f286effd640 1 --2- 172.21.15.88:0/2749258564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28580039b0 0x7f2858003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:57.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.707+0000 7f286e7fc640 1 --2- 172.21.15.88:0/2749258564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2858004800 0x7f2858004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:57.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.707+0000 7f286f7fe640 1 --2- 172.21.15.88:0/2749258564 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28580052e0 0x7f285800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:57.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.707+0000 7f286effd640 1 -- 172.21.15.88:0/2749258564 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28580052e0 msgr2=0x7f285800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:57.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.707+0000 7f286effd640 1 --2- 172.21.15.88:0/2749258564 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28580052e0 0x7f285800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.707+0000 7f286effd640 1 -- 172.21.15.88:0/2749258564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2858004800 msgr2=0x7f2858004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:57.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.707+0000 7f286effd640 1 --2- 172.21.15.88:0/2749258564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2858004800 0x7f2858004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.707+0000 7f286effd640 1 -- 172.21.15.88:0/2749258564 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2858003520 con 0x7f28580039b0 2024-09-16T01:34:57.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.707+0000 7f286f7fe640 1 -- 172.21.15.88:0/2749258564 reap_dead start 2024-09-16T01:34:57.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.707+0000 7f286e7fc640 1 --2- 172.21.15.88:0/2749258564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2858004800 0x7f2858004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:57.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.708+0000 7f286effd640 1 --2- 172.21.15.88:0/2749258564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28580039b0 0x7f2858003db0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2860007fd0 tx=0x7f2858005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff78d84eafa2cc06 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:57.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.708+0000 7f286dffb640 1 -- 172.21.15.88:0/2749258564 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2860007d50 con 0x7f28580039b0 2024-09-16T01:34:57.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.708+0000 7f286dffb640 1 -- 172.21.15.88:0/2749258564 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28600058d0 con 0x7f28580039b0 2024-09-16T01:34:57.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.708+0000 7f286dffb640 1 -- 172.21.15.88:0/2749258564 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2860005d20 con 0x7f28580039b0 2024-09-16T01:34:57.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.708+0000 7f2875274640 1 -- 172.21.15.88:0/2749258564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28580039b0 msgr2=0x7f2858003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:57.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.708+0000 7f2875274640 1 --2- 172.21.15.88:0/2749258564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28580039b0 0x7f2858003db0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2860007fd0 tx=0x7f2858005830 comp rx=0 tx=0).stop 2024-09-16T01:34:57.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.708+0000 7f2875274640 1 -- 172.21.15.88:0/2749258564 shutdown_connections 2024-09-16T01:34:57.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.708+0000 7f2875274640 1 --2- 172.21.15.88:0/2749258564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28580039b0 0x7f2858003db0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.708+0000 7f2875274640 1 -- 172.21.15.88:0/2749258564 >> 172.21.15.88:0/2749258564 conn(0x7f28700d11e0 msgr2=0x7f28700d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:57.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.708+0000 7f2875274640 1 -- 172.21.15.88:0/2749258564 shutdown_connections 2024-09-16T01:34:57.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.708+0000 7f2875274640 1 -- 172.21.15.88:0/2749258564 wait complete. 2024-09-16T01:34:57.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.710+0000 7f2875274640 1 Processor -- start 2024-09-16T01:34:57.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.710+0000 7f2875274640 1 -- start start 2024-09-16T01:34:57.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.710+0000 7f2875274640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28580039b0 0x7f287014fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:57.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.710+0000 7f2875274640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28700c9b60 0x7f28700c9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:57.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.710+0000 7f2875274640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28700ca0b0 0x7f2870158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:57.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.710+0000 7f2875274640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f28700b7a00 con 0x7f28580039b0 2024-09-16T01:34:57.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.710+0000 7f2875274640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f28700d2ca0 con 0x7f28700ca0b0 2024-09-16T01:34:57.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.710+0000 7f2875274640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f287015eb80 con 0x7f28700c9b60 2024-09-16T01:34:57.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.710+0000 7f286effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28580039b0 0x7f287014fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:57.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.710+0000 7f286effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28580039b0 0x7f287014fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55592/0 (socket says 172.21.15.88:55592) 2024-09-16T01:34:57.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.710+0000 7f286effd640 1 -- 172.21.15.88:0/2606160001 learned_addr learned my addr 172.21.15.88:0/2606160001 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:57.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.711+0000 7f286effd640 1 -- 172.21.15.88:0/2606160001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28700c9b60 msgr2=0x7f28700c9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:57.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.711+0000 7f286effd640 1 --2- 172.21.15.88:0/2606160001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28700c9b60 0x7f28700c9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.711+0000 7f286effd640 1 -- 172.21.15.88:0/2606160001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28700ca0b0 msgr2=0x7f2870158810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:57.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.711+0000 7f286effd640 1 --2- 172.21.15.88:0/2606160001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28700ca0b0 0x7f2870158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.711+0000 7f286effd640 1 -- 172.21.15.88:0/2606160001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28700c82f0 con 0x7f28580039b0 2024-09-16T01:34:57.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.711+0000 7f286effd640 1 --2- 172.21.15.88:0/2606160001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28580039b0 0x7f287014fbb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f286000c380 tx=0x7f2860001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:57.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.711+0000 7f28577fe640 1 -- 172.21.15.88:0/2606160001 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2860041020 con 0x7f28580039b0 2024-09-16T01:34:57.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.711+0000 7f28577fe640 1 -- 172.21.15.88:0/2606160001 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2860007270 con 0x7f28580039b0 2024-09-16T01:34:57.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.711+0000 7f28577fe640 1 -- 172.21.15.88:0/2606160001 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2860038040 con 0x7f28580039b0 2024-09-16T01:34:57.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.711+0000 7f2875274640 1 -- 172.21.15.88:0/2606160001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28700c85d0 con 0x7f28580039b0 2024-09-16T01:34:57.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.711+0000 7f2875274640 1 -- 172.21.15.88:0/2606160001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28700c8b90 con 0x7f28580039b0 2024-09-16T01:34:57.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.712+0000 7f2875274640 1 -- 172.21.15.88:0/2606160001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28700b61e0 con 0x7f28580039b0 2024-09-16T01:34:57.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.712+0000 7f28577fe640 1 -- 172.21.15.88:0/2606160001 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2860044020 con 0x7f28580039b0 2024-09-16T01:34:57.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.713+0000 7f28577fe640 1 --2- 172.21.15.88:0/2606160001 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f285c06c2e0 0x7f285c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:57.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.713+0000 7f286e7fc640 1 --2- 172.21.15.88:0/2606160001 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f285c06c2e0 0x7f285c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:57.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.713+0000 7f28577fe640 1 -- 172.21.15.88:0/2606160001 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7f28600aafc0 con 0x7f28580039b0 2024-09-16T01:34:57.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.713+0000 7f286e7fc640 1 --2- 172.21.15.88:0/2606160001 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f285c06c2e0 0x7f285c06e7a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2864006d40 tx=0x7f2864006bd0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:57.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.717+0000 7f28577fe640 1 -- 172.21.15.88:0/2606160001 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f286004b030 con 0x7f28580039b0 2024-09-16T01:34:57.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.946+0000 7f2875274640 1 -- 172.21.15.88:0/2606160001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f28700d1940 con 0x7f28580039b0 2024-09-16T01:34:57.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.949+0000 7f28577fe640 1 -- 172.21.15.88:0/2606160001 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7f2860079b10 con 0x7f28580039b0 2024-09-16T01:34:57.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.951+0000 7f2875274640 1 -- 172.21.15.88:0/2606160001 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f285c06c2e0 msgr2=0x7f285c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:57.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.951+0000 7f2875274640 1 --2- 172.21.15.88:0/2606160001 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f285c06c2e0 0x7f285c06e7a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2864006d40 tx=0x7f2864006bd0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.951+0000 7f2875274640 1 -- 172.21.15.88:0/2606160001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28580039b0 msgr2=0x7f287014fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:57.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.951+0000 7f2875274640 1 --2- 172.21.15.88:0/2606160001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28580039b0 0x7f287014fbb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f286000c380 tx=0x7f2860001a40 comp rx=0 tx=0).stop 2024-09-16T01:34:57.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.951+0000 7f2875274640 1 -- 172.21.15.88:0/2606160001 shutdown_connections 2024-09-16T01:34:57.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.951+0000 7f2875274640 1 --2- 172.21.15.88:0/2606160001 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f285c06c2e0 0x7f285c06e7a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.951+0000 7f2875274640 1 --2- 172.21.15.88:0/2606160001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28700ca0b0 0x7f2870158810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.951+0000 7f2875274640 1 --2- 172.21.15.88:0/2606160001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28700c9b60 0x7f28700c9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.951+0000 7f2875274640 1 --2- 172.21.15.88:0/2606160001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28580039b0 0x7f287014fbb0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:57.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.951+0000 7f2875274640 1 -- 172.21.15.88:0/2606160001 >> 172.21.15.88:0/2606160001 conn(0x7f28700d11e0 msgr2=0x7f287015c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:57.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.951+0000 7f2875274640 1 -- 172.21.15.88:0/2606160001 shutdown_connections 2024-09-16T01:34:57.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:57.951+0000 7f2875274640 1 -- 172.21.15.88:0/2606160001 wait complete. 2024-09-16T01:34:57.963 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-16T01:34:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.062+0000 7fc7da842640 1 Processor -- start 2024-09-16T01:34:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.062+0000 7fc7da842640 1 -- start start 2024-09-16T01:34:58.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.063+0000 7fc7da842640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc7d415e950 0x7fc7d406ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:58.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.063+0000 7fc7da842640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7d4072050 0x7fc7d406d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:58.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.063+0000 7fc7da842640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7d406faf0 con 0x7fc7d40735b0 2024-09-16T01:34:58.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.063+0000 7fc7da842640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc7d406fdd0 con 0x7fc7d415e950 2024-09-16T01:34:58.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.063+0000 7fc7da842640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc7d40700b0 con 0x7fc7d4072050 2024-09-16T01:34:58.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.063+0000 7fc7d3fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc7d415e950 0x7fc7d406ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:58.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.063+0000 7fc7d37fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc7d40735b0 0x7fc7d406bc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33706/0 (socket says 172.21.15.88:33706) 2024-09-16T01:34:58.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.063+0000 7fc7d37fe640 1 -- 172.21.15.88:0/1697052706 learned_addr learned my addr 172.21.15.88:0/1697052706 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:58.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.064+0000 7fc7d3fff640 1 -- 172.21.15.88:0/1697052706 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7d4072050 msgr2=0x7fc7d406d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:58.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.064+0000 7fc7d8ddf640 1 --2- 172.21.15.88:0/1697052706 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7d4072050 0x7fc7d406d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:58.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.064+0000 7fc7d3fff640 1 --2- 172.21.15.88:0/1697052706 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7d4072050 0x7fc7d406d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:58.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.064+0000 7fc7d3fff640 1 -- 172.21.15.88:0/1697052706 >> v1:172.21.15.88:6789/0 conn(0x7fc7d40735b0 legacy=0x7fc7d406bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:58.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.064+0000 7fc7d2ffd640 1 -- 172.21.15.88:0/1697052706 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2855967532 0 0) 0x7fc7d406faf0 con 0x7fc7d40735b0 2024-09-16T01:34:58.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.064+0000 7fc7d3fff640 1 -- 172.21.15.88:0/1697052706 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7d4070390 con 0x7fc7d415e950 2024-09-16T01:34:58.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.064+0000 7fc7d8ddf640 1 --2- 172.21.15.88:0/1697052706 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7d4072050 0x7fc7d406d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:58.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.064+0000 7fc7d3fff640 1 --2- 172.21.15.88:0/1697052706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7d415e950 0x7fc7d406ce30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc7c40097a0 tx=0x7fc7c402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7cfe63e64ee497c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:58.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.064+0000 7fc7d2ffd640 1 -- 172.21.15.88:0/1697052706 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c4037070 con 0x7fc7d415e950 2024-09-16T01:34:58.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.064+0000 7fc7d2ffd640 1 -- 172.21.15.88:0/1697052706 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7c4032440 con 0x7fc7d415e950 2024-09-16T01:34:58.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.064+0000 7fc7d2ffd640 1 -- 172.21.15.88:0/1697052706 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c4032810 con 0x7fc7d415e950 2024-09-16T01:34:58.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.065+0000 7fc7da842640 1 -- 172.21.15.88:0/1697052706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7d415e950 msgr2=0x7fc7d406ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:58.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.065+0000 7fc7da842640 1 --2- 172.21.15.88:0/1697052706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7d415e950 0x7fc7d406ce30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc7c40097a0 tx=0x7fc7c402a180 comp rx=0 tx=0).stop 2024-09-16T01:34:58.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.065+0000 7fc7da842640 1 -- 172.21.15.88:0/1697052706 shutdown_connections 2024-09-16T01:34:58.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.065+0000 7fc7da842640 1 --2- 172.21.15.88:0/1697052706 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7d4072050 0x7fc7d406d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:58.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.065+0000 7fc7da842640 1 --2- 172.21.15.88:0/1697052706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7d415e950 0x7fc7d406ce30 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:58.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.065+0000 7fc7da842640 1 -- 172.21.15.88:0/1697052706 >> 172.21.15.88:0/1697052706 conn(0x7fc7d40d14d0 msgr2=0x7fc7d40d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:58.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.065+0000 7fc7da842640 1 -- 172.21.15.88:0/1697052706 shutdown_connections 2024-09-16T01:34:58.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.065+0000 7fc7da842640 1 -- 172.21.15.88:0/1697052706 wait complete. 2024-09-16T01:34:58.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.066+0000 7fc7da842640 1 Processor -- start 2024-09-16T01:34:58.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.066+0000 7fc7da842640 1 -- start start 2024-09-16T01:34:58.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.067+0000 7fc7da842640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7d4072050 0x7fc7d414b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:58.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.067+0000 7fc7da842640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7d40735b0 0x7fc7d4159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:58.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.067+0000 7fc7da842640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7d415e950 0x7fc7d415a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:58.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.067+0000 7fc7da842640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d40d0230 con 0x7fc7d415e950 2024-09-16T01:34:58.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.067+0000 7fc7da842640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d40d00b0 con 0x7fc7d40735b0 2024-09-16T01:34:58.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.067+0000 7fc7da842640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc7d40d03b0 con 0x7fc7d4072050 2024-09-16T01:34:58.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.067+0000 7fc7d8ddf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7d415e950 0x7fc7d415a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:58.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.067+0000 7fc7d8ddf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7d415e950 0x7fc7d415a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55622/0 (socket says 172.21.15.88:55622) 2024-09-16T01:34:58.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.067+0000 7fc7d8ddf640 1 -- 172.21.15.88:0/259004764 learned_addr learned my addr 172.21.15.88:0/259004764 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:58.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.067+0000 7fc7d3fff640 1 --2- 172.21.15.88:0/259004764 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7d4072050 0x7fc7d414b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:58.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.067+0000 7fc7d37fe640 1 --2- 172.21.15.88:0/259004764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7d40735b0 0x7fc7d4159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:58.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.067+0000 7fc7d8ddf640 1 -- 172.21.15.88:0/259004764 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7d4072050 msgr2=0x7fc7d414b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:58.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.067+0000 7fc7d8ddf640 1 --2- 172.21.15.88:0/259004764 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7d4072050 0x7fc7d414b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:58.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.067+0000 7fc7d8ddf640 1 -- 172.21.15.88:0/259004764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7d40735b0 msgr2=0x7fc7d4159f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:58.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.067+0000 7fc7d8ddf640 1 --2- 172.21.15.88:0/259004764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7d40735b0 0x7fc7d4159f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:58.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.068+0000 7fc7d8ddf640 1 -- 172.21.15.88:0/259004764 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7d4157c20 con 0x7fc7d415e950 2024-09-16T01:34:58.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.068+0000 7fc7d37fe640 1 --2- 172.21.15.88:0/259004764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7d40735b0 0x7fc7d4159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:58.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.068+0000 7fc7d8ddf640 1 --2- 172.21.15.88:0/259004764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7d415e950 0x7fc7d415a470 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc00cd50 tx=0x7fc7cc00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:58.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.068+0000 7fc7d0ff9640 1 -- 172.21.15.88:0/259004764 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc00f680 con 0x7fc7d415e950 2024-09-16T01:34:58.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.068+0000 7fc7d0ff9640 1 -- 172.21.15.88:0/259004764 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7cc00f840 con 0x7fc7d415e950 2024-09-16T01:34:58.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.068+0000 7fc7d0ff9640 1 -- 172.21.15.88:0/259004764 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc00fc30 con 0x7fc7d415e950 2024-09-16T01:34:58.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.068+0000 7fc7da842640 1 -- 172.21.15.88:0/259004764 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7d41558f0 con 0x7fc7d415e950 2024-09-16T01:34:58.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.068+0000 7fc7da842640 1 -- 172.21.15.88:0/259004764 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7d4155eb0 con 0x7fc7d415e950 2024-09-16T01:34:58.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.069+0000 7fc7da842640 1 -- 172.21.15.88:0/259004764 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7d41560b0 con 0x7fc7d415e950 2024-09-16T01:34:58.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.069+0000 7fc7d0ff9640 1 -- 172.21.15.88:0/259004764 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc7cc0213f0 con 0x7fc7d415e950 2024-09-16T01:34:58.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.070+0000 7fc7d0ff9640 1 --2- 172.21.15.88:0/259004764 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc7c006c2b0 0x7fc7c006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:58.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.070+0000 7fc7d0ff9640 1 -- 172.21.15.88:0/259004764 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7fc7cc08be80 con 0x7fc7d415e950 2024-09-16T01:34:58.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.070+0000 7fc7d3fff640 1 --2- 172.21.15.88:0/259004764 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc7c006c2b0 0x7fc7c006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:58.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.070+0000 7fc7d3fff640 1 --2- 172.21.15.88:0/259004764 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc7c006c2b0 0x7fc7c006e770 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4034000 tx=0x7fc7c402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:58.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.074+0000 7fc7d0ff9640 1 -- 172.21.15.88:0/259004764 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc7d41560b0 con 0x7fc7d415e950 2024-09-16T01:34:58.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.240+0000 7fc7da842640 1 -- 172.21.15.88:0/259004764 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fc7d40b2710 con 0x7fc7d415e950 2024-09-16T01:34:58.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.241+0000 7fc7d0ff9640 1 -- 172.21.15.88:0/259004764 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v28) ==== 84+0+65 (secure 0 0 0) 0x7fc7cc014030 con 0x7fc7d415e950 2024-09-16T01:34:58.243 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:34:58.243 INFO:teuthology.orchestra.run.smithi088.stdout:[ 2024-09-16T01:34:58.243 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:34:58.243 INFO:teuthology.orchestra.run.smithi088.stdout: "poolnum": 1, 2024-09-16T01:34:58.243 INFO:teuthology.orchestra.run.smithi088.stdout: "poolname": ".mgr" 2024-09-16T01:34:58.243 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:34:58.243 INFO:teuthology.orchestra.run.smithi088.stdout:] 2024-09-16T01:34:58.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.243+0000 7fc7da842640 1 -- 172.21.15.88:0/259004764 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc7c006c2b0 msgr2=0x7fc7c006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:58.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.243+0000 7fc7da842640 1 --2- 172.21.15.88:0/259004764 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc7c006c2b0 0x7fc7c006e770 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4034000 tx=0x7fc7c402e000 comp rx=0 tx=0).stop 2024-09-16T01:34:58.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.243+0000 7fc7da842640 1 -- 172.21.15.88:0/259004764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7d415e950 msgr2=0x7fc7d415a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:58.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.243+0000 7fc7da842640 1 --2- 172.21.15.88:0/259004764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7d415e950 0x7fc7d415a470 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc00cd50 tx=0x7fc7cc00a650 comp rx=0 tx=0).stop 2024-09-16T01:34:58.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.243+0000 7fc7da842640 1 -- 172.21.15.88:0/259004764 shutdown_connections 2024-09-16T01:34:58.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.243+0000 7fc7da842640 1 --2- 172.21.15.88:0/259004764 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc7c006c2b0 0x7fc7c006e770 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:58.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.243+0000 7fc7da842640 1 --2- 172.21.15.88:0/259004764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7d415e950 0x7fc7d415a470 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:58.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.243+0000 7fc7da842640 1 --2- 172.21.15.88:0/259004764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7d40735b0 0x7fc7d4159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:58.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.243+0000 7fc7da842640 1 --2- 172.21.15.88:0/259004764 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc7d4072050 0x7fc7d414b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:58.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.243+0000 7fc7da842640 1 -- 172.21.15.88:0/259004764 >> 172.21.15.88:0/259004764 conn(0x7fc7d40d14d0 msgr2=0x7fc7d40b5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:58.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.243+0000 7fc7da842640 1 -- 172.21.15.88:0/259004764 shutdown_connections 2024-09-16T01:34:58.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.243+0000 7fc7da842640 1 -- 172.21.15.88:0/259004764 wait complete. 2024-09-16T01:34:58.255 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-16T01:34:58.255 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-16T01:34:58.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.356+0000 7f200f3d8640 1 Processor -- start 2024-09-16T01:34:58.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.356+0000 7f200f3d8640 1 -- start start 2024-09-16T01:34:58.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.357+0000 7f200f3d8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20080b64a0 0x7f20080d07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:58.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.357+0000 7f200f3d8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20080d0e60 con 0x7f200815e7c0 2024-09-16T01:34:58.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.357+0000 7f200f3d8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20080d1140 con 0x7f200806bc10 2024-09-16T01:34:58.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.357+0000 7f200f3d8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f20080b49b0 con 0x7f20080b64a0 2024-09-16T01:34:58.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.357+0000 7f200d174640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f200815e7c0 0x7f20080d00d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33720/0 (socket says 172.21.15.88:33720) 2024-09-16T01:34:58.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.357+0000 7f200d174640 1 -- 172.21.15.88:0/1026652789 learned_addr learned my addr 172.21.15.88:0/1026652789 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:58.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.358+0000 7f2007fff640 1 -- 172.21.15.88:0/1026652789 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3288746674 0 0) 0x7f20080d0e60 con 0x7f200815e7c0 2024-09-16T01:34:58.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.358+0000 7f200c973640 1 --2- 172.21.15.88:0/1026652789 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20080b64a0 0x7f20080d07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:58.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.358+0000 7f2007fff640 1 -- 172.21.15.88:0/1026652789 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ff0003520 con 0x7f200815e7c0 2024-09-16T01:34:58.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.358+0000 7f2007fff640 1 -- 172.21.15.88:0/1026652789 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2213402631 0 0) 0x7f1ff0003520 con 0x7f200815e7c0 2024-09-16T01:34:58.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.358+0000 7f2007fff640 1 -- 172.21.15.88:0/1026652789 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20080b64a0 msgr2=0x7f20080d07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:58.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.358+0000 7f2007fff640 1 --2- 172.21.15.88:0/1026652789 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20080b64a0 0x7f20080d07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:58.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.358+0000 7f2007fff640 1 -- 172.21.15.88:0/1026652789 >> v1:172.21.15.154:6789/0 conn(0x7f200806bc10 legacy=0x7f200806c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:58.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.358+0000 7f2007fff640 1 -- 172.21.15.88:0/1026652789 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20080b4c90 con 0x7f200815e7c0 2024-09-16T01:34:58.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.358+0000 7f2007fff640 1 -- 172.21.15.88:0/1026652789 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f1ff80019f0 con 0x7f200815e7c0 2024-09-16T01:34:58.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.358+0000 7f2007fff640 1 -- 172.21.15.88:0/1026652789 >> v1:172.21.15.88:6789/0 conn(0x7f200815e7c0 legacy=0x7f20080d00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:58.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.358+0000 7f2007fff640 1 --2- 172.21.15.88:0/1026652789 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20080b64a0 0x7f1ff00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:58.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.358+0000 7f2007fff640 1 --2- 172.21.15.88:0/1026652789 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff0003f60 0x7f1ff0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:58.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.358+0000 7f2007fff640 1 --2- 172.21.15.88:0/1026652789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ff0004d40 0x7f1ff00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:58.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.358+0000 7f2007fff640 1 -- 172.21.15.88:0/1026652789 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f20080b49b0 con 0x7f1ff0003f60 2024-09-16T01:34:58.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.358+0000 7f2007fff640 1 -- 172.21.15.88:0/1026652789 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1ff000baa0 con 0x7f1ff0004d40 2024-09-16T01:34:58.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.358+0000 7f2007fff640 1 -- 172.21.15.88:0/1026652789 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1ff000bca0 con 0x7f20080b64a0 2024-09-16T01:34:58.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.358+0000 7f200c973640 1 --2- 172.21.15.88:0/1026652789 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff0003f60 0x7f1ff0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:58.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.358+0000 7f200d174640 1 --2- 172.21.15.88:0/1026652789 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20080b64a0 0x7f1ff00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:58.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.358+0000 7f200d975640 1 --2- 172.21.15.88:0/1026652789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ff0004d40 0x7f1ff00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:58.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.358+0000 7f200c973640 1 -- 172.21.15.88:0/1026652789 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20080b64a0 msgr2=0x7f1ff00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:58.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.358+0000 7f200c973640 1 --2- 172.21.15.88:0/1026652789 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20080b64a0 0x7f1ff00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:58.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.358+0000 7f200c973640 1 -- 172.21.15.88:0/1026652789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ff0004d40 msgr2=0x7f1ff00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:58.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.358+0000 7f200c973640 1 --2- 172.21.15.88:0/1026652789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ff0004d40 0x7f1ff00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:58.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.358+0000 7f200c973640 1 -- 172.21.15.88:0/1026652789 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ff0003520 con 0x7f1ff0003f60 2024-09-16T01:34:58.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.359+0000 7f200c973640 1 --2- 172.21.15.88:0/1026652789 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff0003f60 0x7f1ff0004800 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc009ac0 tx=0x7f1ff0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c1e5c4c0a03ed3d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:58.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.359+0000 7f2007fff640 1 -- 172.21.15.88:0/1026652789 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc036070 con 0x7f1ff0003f60 2024-09-16T01:34:58.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.359+0000 7f2007fff640 1 -- 172.21.15.88:0/1026652789 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ffc031400 con 0x7f1ff0003f60 2024-09-16T01:34:58.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.359+0000 7f2007fff640 1 -- 172.21.15.88:0/1026652789 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc009690 con 0x7f1ff0003f60 2024-09-16T01:34:58.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.360+0000 7f200f3d8640 1 -- 172.21.15.88:0/1026652789 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff0003f60 msgr2=0x7f1ff0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:58.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.360+0000 7f200f3d8640 1 --2- 172.21.15.88:0/1026652789 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff0003f60 0x7f1ff0004800 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc009ac0 tx=0x7f1ff0005830 comp rx=0 tx=0).stop 2024-09-16T01:34:58.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.360+0000 7f200d975640 1 -- 172.21.15.88:0/1026652789 reap_dead start 2024-09-16T01:34:58.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.360+0000 7f200f3d8640 1 -- 172.21.15.88:0/1026652789 shutdown_connections 2024-09-16T01:34:58.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.360+0000 7f200f3d8640 1 -- 172.21.15.88:0/1026652789 >> 172.21.15.88:0/1026652789 conn(0x7f20080d14d0 msgr2=0x7f20080d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:58.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.360+0000 7f200f3d8640 1 -- 172.21.15.88:0/1026652789 shutdown_connections 2024-09-16T01:34:58.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.360+0000 7f200f3d8640 1 -- 172.21.15.88:0/1026652789 wait complete. 2024-09-16T01:34:58.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.361+0000 7f200f3d8640 1 Processor -- start 2024-09-16T01:34:58.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.361+0000 7f200f3d8640 1 -- start start 2024-09-16T01:34:58.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.362+0000 7f200f3d8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff0003f60 0x7f200815f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:58.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.362+0000 7f200f3d8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f200814d580 0x7f2008161880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:58.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.362+0000 7f200f3d8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f200806c080 0x7f200806c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:58.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.362+0000 7f200f3d8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f200815ed20 con 0x7f1ff0003f60 2024-09-16T01:34:58.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.362+0000 7f200f3d8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f200815eba0 con 0x7f200806c080 2024-09-16T01:34:58.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.362+0000 7f200f3d8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f20080d07e0 con 0x7f200814d580 2024-09-16T01:34:58.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.362+0000 7f200d174640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff0003f60 0x7f200815f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:58.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.362+0000 7f200d174640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff0003f60 0x7f200815f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55646/0 (socket says 172.21.15.88:55646) 2024-09-16T01:34:58.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.362+0000 7f200d174640 1 -- 172.21.15.88:0/3991573477 learned_addr learned my addr 172.21.15.88:0/3991573477 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:58.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.362+0000 7f200c973640 1 --2- 172.21.15.88:0/3991573477 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f200814d580 0x7f2008161880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:58.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.362+0000 7f200d975640 1 --2- 172.21.15.88:0/3991573477 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f200806c080 0x7f200806c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:58.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.362+0000 7f200d174640 1 -- 172.21.15.88:0/3991573477 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f200814d580 msgr2=0x7f2008161880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:58.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.362+0000 7f200d174640 1 --2- 172.21.15.88:0/3991573477 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f200814d580 0x7f2008161880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:58.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.362+0000 7f200d174640 1 -- 172.21.15.88:0/3991573477 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f200806c080 msgr2=0x7f200806c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:58.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.362+0000 7f200d174640 1 --2- 172.21.15.88:0/3991573477 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f200806c080 0x7f200806c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:58.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.362+0000 7f200d174640 1 -- 172.21.15.88:0/3991573477 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f200814da00 con 0x7f1ff0003f60 2024-09-16T01:34:58.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.362+0000 7f200d975640 1 --2- 172.21.15.88:0/3991573477 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f200806c080 0x7f200806c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:34:58.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.362+0000 7f200d174640 1 --2- 172.21.15.88:0/3991573477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff0003f60 0x7f200815f8b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1ff80079a0 tx=0x7f1ff800a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:58.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.362+0000 7f2005ffb640 1 -- 172.21.15.88:0/3991573477 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff80133f0 con 0x7f1ff0003f60 2024-09-16T01:34:58.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.363+0000 7f2005ffb640 1 -- 172.21.15.88:0/3991573477 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ff8018070 con 0x7f1ff0003f60 2024-09-16T01:34:58.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.363+0000 7f200f3d8640 1 -- 172.21.15.88:0/3991573477 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2008161dc0 con 0x7f1ff0003f60 2024-09-16T01:34:58.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.363+0000 7f2005ffb640 1 -- 172.21.15.88:0/3991573477 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff80137d0 con 0x7f1ff0003f60 2024-09-16T01:34:58.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.363+0000 7f200f3d8640 1 -- 172.21.15.88:0/3991573477 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2008160220 con 0x7f1ff0003f60 2024-09-16T01:34:58.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.364+0000 7f200f3d8640 1 -- 172.21.15.88:0/3991573477 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f200806ca40 con 0x7f1ff0003f60 2024-09-16T01:34:58.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.364+0000 7f2005ffb640 1 -- 172.21.15.88:0/3991573477 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1ff8013b80 con 0x7f1ff0003f60 2024-09-16T01:34:58.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.365+0000 7f2005ffb640 1 --2- 172.21.15.88:0/3991573477 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ff406c2e0 0x7f1ff406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:58.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.365+0000 7f2005ffb640 1 -- 172.21.15.88:0/3991573477 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 4857+0+0 (secure 0 0 0) 0x7f1ff808bb40 con 0x7f1ff0003f60 2024-09-16T01:34:58.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.365+0000 7f200c973640 1 --2- 172.21.15.88:0/3991573477 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ff406c2e0 0x7f1ff406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:58.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.365+0000 7f200c973640 1 --2- 172.21.15.88:0/3991573477 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ff406c2e0 0x7f1ff406e7a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc033000 tx=0x7f1ffc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:58.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.369+0000 7f2005ffb640 1 -- 172.21.15.88:0/3991573477 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1ff805a600 con 0x7f1ff0003f60 2024-09-16T01:34:58.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:58.536+0000 7f200f3d8640 1 -- 172.21.15.88:0/3991573477 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f20080d00d0 con 0x7f1ff0003f60 2024-09-16T01:34:59.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.325+0000 7f2005ffb640 1 -- 172.21.15.88:0/3991573477 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v29) ==== 138+0+0 (secure 0 0 0) 0x7f1ff8059f00 con 0x7f1ff0003f60 2024-09-16T01:34:59.326 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_metadata' created 2024-09-16T01:34:59.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.328+0000 7f200f3d8640 1 -- 172.21.15.88:0/3991573477 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ff406c2e0 msgr2=0x7f1ff406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:59.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.328+0000 7f200f3d8640 1 --2- 172.21.15.88:0/3991573477 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ff406c2e0 0x7f1ff406e7a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc033000 tx=0x7f1ffc008030 comp rx=0 tx=0).stop 2024-09-16T01:34:59.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.328+0000 7f200f3d8640 1 -- 172.21.15.88:0/3991573477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff0003f60 msgr2=0x7f200815f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:59.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.328+0000 7f200f3d8640 1 --2- 172.21.15.88:0/3991573477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff0003f60 0x7f200815f8b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1ff80079a0 tx=0x7f1ff800a730 comp rx=0 tx=0).stop 2024-09-16T01:34:59.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.328+0000 7f200f3d8640 1 -- 172.21.15.88:0/3991573477 shutdown_connections 2024-09-16T01:34:59.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.328+0000 7f200f3d8640 1 --2- 172.21.15.88:0/3991573477 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ff406c2e0 0x7f1ff406e7a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:59.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.328+0000 7f200f3d8640 1 --2- 172.21.15.88:0/3991573477 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f200806c080 0x7f200806c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:59.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.328+0000 7f200f3d8640 1 --2- 172.21.15.88:0/3991573477 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f200814d580 0x7f2008161880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:59.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.328+0000 7f200f3d8640 1 --2- 172.21.15.88:0/3991573477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff0003f60 0x7f200815f8b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:59.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.328+0000 7f200f3d8640 1 -- 172.21.15.88:0/3991573477 >> 172.21.15.88:0/3991573477 conn(0x7f20080d14d0 msgr2=0x7f2008072b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:59.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.328+0000 7f200f3d8640 1 -- 172.21.15.88:0/3991573477 shutdown_connections 2024-09-16T01:34:59.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.328+0000 7f200f3d8640 1 -- 172.21.15.88:0/3991573477 wait complete. 2024-09-16T01:34:59.347 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-16T01:34:59.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.457+0000 7f2abeaa6640 1 Processor -- start 2024-09-16T01:34:59.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.457+0000 7f2abeaa6640 1 -- start start 2024-09-16T01:34:59.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.458+0000 7f2abeaa6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2ab80cf1f0 0x7f2ab80b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:59.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.458+0000 7f2abeaa6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ab80cf860 con 0x7f2ab815e7a0 2024-09-16T01:34:59.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.458+0000 7f2abeaa6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ab80cfb40 con 0x7f2ab80b6720 2024-09-16T01:34:59.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.458+0000 7f2abeaa6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2ab80b6b90 con 0x7f2ab80cf1f0 2024-09-16T01:34:59.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.458+0000 7f2abc842640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f2ab815e7a0 0x7f2ab806caa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33724/0 (socket says 172.21.15.88:33724) 2024-09-16T01:34:59.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.458+0000 7f2abc842640 1 -- 172.21.15.88:0/1376529559 learned_addr learned my addr 172.21.15.88:0/1376529559 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:34:59.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.458+0000 7f2aaf7fe640 1 -- 172.21.15.88:0/1376529559 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2370171675 0 0) 0x7f2ab80cf860 con 0x7f2ab815e7a0 2024-09-16T01:34:59.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.458+0000 7f2aaf7fe640 1 -- 172.21.15.88:0/1376529559 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2aa0003520 con 0x7f2ab815e7a0 2024-09-16T01:34:59.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.458+0000 7f2aaffff640 1 --2- 172.21.15.88:0/1376529559 >> v2:172.21.15.154:3300/0 conn(0x7f2ab80cf1f0 0x7f2ab80b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:59.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.459+0000 7f2aaf7fe640 1 -- 172.21.15.88:0/1376529559 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2337981410 0 0) 0x7f2aa0003520 con 0x7f2ab815e7a0 2024-09-16T01:34:59.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.459+0000 7f2aaf7fe640 1 -- 172.21.15.88:0/1376529559 >> v2:172.21.15.154:3300/0 conn(0x7f2ab80cf1f0 msgr2=0x7f2ab80b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:59.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.459+0000 7f2aaf7fe640 1 --2- 172.21.15.88:0/1376529559 >> v2:172.21.15.154:3300/0 conn(0x7f2ab80cf1f0 0x7f2ab80b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:59.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.459+0000 7f2aaf7fe640 1 -- 172.21.15.88:0/1376529559 >> v1:172.21.15.154:6789/0 conn(0x7f2ab80b6720 legacy=0x7f2ab80d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:59.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.459+0000 7f2aaf7fe640 1 -- 172.21.15.88:0/1376529559 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab80cfe20 con 0x7f2ab815e7a0 2024-09-16T01:34:59.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.459+0000 7f2aaf7fe640 1 -- 172.21.15.88:0/1376529559 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f2ab4001670 con 0x7f2ab815e7a0 2024-09-16T01:34:59.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.459+0000 7f2aaf7fe640 1 -- 172.21.15.88:0/1376529559 >> v1:172.21.15.88:6789/0 conn(0x7f2ab815e7a0 legacy=0x7f2ab806caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:59.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.459+0000 7f2aaf7fe640 1 --2- 172.21.15.88:0/1376529559 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2aa00039b0 0x7f2aa0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:59.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.459+0000 7f2aaf7fe640 1 --2- 172.21.15.88:0/1376529559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2aa0004800 0x7f2aa0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:59.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.459+0000 7f2aaf7fe640 1 --2- 172.21.15.88:0/1376529559 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2aa00051c0 0x7f2aa000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:59.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.459+0000 7f2aaf7fe640 1 -- 172.21.15.88:0/1376529559 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab80b6b90 con 0x7f2aa00039b0 2024-09-16T01:34:59.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.459+0000 7f2aaf7fe640 1 -- 172.21.15.88:0/1376529559 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2aa000c160 con 0x7f2aa0004800 2024-09-16T01:34:59.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.459+0000 7f2aaf7fe640 1 -- 172.21.15.88:0/1376529559 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2aa000c360 con 0x7f2aa00051c0 2024-09-16T01:34:59.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.459+0000 7f2abc842640 1 --2- 172.21.15.88:0/1376529559 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2aa00039b0 0x7f2aa0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:59.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.459+0000 7f2aaffff640 1 --2- 172.21.15.88:0/1376529559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2aa0004800 0x7f2aa0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:59.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.459+0000 7f2abd043640 1 --2- 172.21.15.88:0/1376529559 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2aa00051c0 0x7f2aa000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:59.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.459+0000 7f2abc842640 1 -- 172.21.15.88:0/1376529559 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2aa00051c0 msgr2=0x7f2aa000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:59.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.459+0000 7f2abc842640 1 --2- 172.21.15.88:0/1376529559 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2aa00051c0 0x7f2aa000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:59.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.459+0000 7f2abc842640 1 -- 172.21.15.88:0/1376529559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2aa0004800 msgr2=0x7f2aa0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:34:59.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.459+0000 7f2abc842640 1 --2- 172.21.15.88:0/1376529559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2aa0004800 0x7f2aa0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:59.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.459+0000 7f2abd043640 1 -- 172.21.15.88:0/1376529559 reap_dead start 2024-09-16T01:34:59.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.459+0000 7f2abc842640 1 -- 172.21.15.88:0/1376529559 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aa0003520 con 0x7f2aa00039b0 2024-09-16T01:34:59.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.460+0000 7f2abc842640 1 --2- 172.21.15.88:0/1376529559 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2aa00039b0 0x7f2aa0003dd0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2ab400c210 tx=0x7f2aa0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91bbf571cef0a9fc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.460+0000 7f2aaf7fe640 1 -- 172.21.15.88:0/1376529559 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab4007d30 con 0x7f2aa00039b0 2024-09-16T01:34:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.460+0000 7f2aaf7fe640 1 -- 172.21.15.88:0/1376529559 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ab4005ad0 con 0x7f2aa00039b0 2024-09-16T01:34:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.460+0000 7f2aaf7fe640 1 -- 172.21.15.88:0/1376529559 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab4007070 con 0x7f2aa00039b0 2024-09-16T01:34:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.460+0000 7f2abeaa6640 1 -- 172.21.15.88:0/1376529559 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2aa00039b0 msgr2=0x7f2aa0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.460+0000 7f2abeaa6640 1 --2- 172.21.15.88:0/1376529559 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2aa00039b0 0x7f2aa0003dd0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2ab400c210 tx=0x7f2aa0005830 comp rx=0 tx=0).stop 2024-09-16T01:34:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.460+0000 7f2abeaa6640 1 -- 172.21.15.88:0/1376529559 shutdown_connections 2024-09-16T01:34:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.460+0000 7f2abeaa6640 1 --2- 172.21.15.88:0/1376529559 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2aa00039b0 0x7f2aa0003dd0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.460+0000 7f2abeaa6640 1 -- 172.21.15.88:0/1376529559 >> 172.21.15.88:0/1376529559 conn(0x7f2ab80d11e0 msgr2=0x7f2ab80d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:34:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.461+0000 7f2abeaa6640 1 -- 172.21.15.88:0/1376529559 shutdown_connections 2024-09-16T01:34:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.461+0000 7f2abeaa6640 1 -- 172.21.15.88:0/1376529559 wait complete. 2024-09-16T01:34:59.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.462+0000 7f2abeaa6640 1 Processor -- start 2024-09-16T01:34:59.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.462+0000 7f2abeaa6640 1 -- start start 2024-09-16T01:34:59.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.463+0000 7f2abeaa6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2aa00039b0 0x7f2ab815d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:59.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.463+0000 7f2abeaa6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ab8156ca0 0x7f2ab80b54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:59.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.463+0000 7f2abeaa6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ab80cb830 0x7f2ab80b8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:59.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.463+0000 7f2abeaa6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab80b7a00 con 0x7f2aa00039b0 2024-09-16T01:34:59.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.463+0000 7f2abeaa6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab80d2ca0 con 0x7f2ab80cb830 2024-09-16T01:34:59.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.463+0000 7f2abeaa6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2ab815eb80 con 0x7f2ab8156ca0 2024-09-16T01:34:59.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.463+0000 7f2abc842640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2aa00039b0 0x7f2ab815d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.463+0000 7f2abc842640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2aa00039b0 0x7f2ab815d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55670/0 (socket says 172.21.15.88:55670) 2024-09-16T01:34:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.463+0000 7f2abc842640 1 -- 172.21.15.88:0/1504925217 learned_addr learned my addr 172.21.15.88:0/1504925217 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:34:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.463+0000 7f2abd043640 1 --2- 172.21.15.88:0/1504925217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ab80cb830 0x7f2ab80b8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.463+0000 7f2aaffff640 1 --2- 172.21.15.88:0/1504925217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ab8156ca0 0x7f2ab80b54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.463+0000 7f2abc842640 1 -- 172.21.15.88:0/1504925217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ab8156ca0 msgr2=0x7f2ab80b54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.463+0000 7f2abc842640 1 --2- 172.21.15.88:0/1504925217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ab8156ca0 0x7f2ab80b54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.463+0000 7f2abc842640 1 -- 172.21.15.88:0/1504925217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ab80cb830 msgr2=0x7f2ab80b8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:34:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.463+0000 7f2abc842640 1 --2- 172.21.15.88:0/1504925217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ab80cb830 0x7f2ab80b8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:34:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.463+0000 7f2abc842640 1 -- 172.21.15.88:0/1504925217 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab80b7f20 con 0x7f2aa00039b0 2024-09-16T01:34:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.463+0000 7f2abd043640 1 --2- 172.21.15.88:0/1504925217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ab80cb830 0x7f2ab80b8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.463+0000 7f2aaffff640 1 --2- 172.21.15.88:0/1504925217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ab8156ca0 0x7f2ab80b54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:34:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.463+0000 7f2abc842640 1 --2- 172.21.15.88:0/1504925217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2aa00039b0 0x7f2ab815d3f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2ab400c210 tx=0x7f2ab4001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.463+0000 7f2aad7fa640 1 -- 172.21.15.88:0/1504925217 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab4041070 con 0x7f2aa00039b0 2024-09-16T01:34:59.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.463+0000 7f2aad7fa640 1 -- 172.21.15.88:0/1504925217 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ab4000ed0 con 0x7f2aa00039b0 2024-09-16T01:34:59.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.463+0000 7f2abeaa6640 1 -- 172.21.15.88:0/1504925217 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ab80b8200 con 0x7f2aa00039b0 2024-09-16T01:34:59.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.463+0000 7f2aad7fa640 1 -- 172.21.15.88:0/1504925217 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab40072d0 con 0x7f2aa00039b0 2024-09-16T01:34:59.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.463+0000 7f2abeaa6640 1 -- 172.21.15.88:0/1504925217 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ab80b87c0 con 0x7f2aa00039b0 2024-09-16T01:34:59.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.464+0000 7f2abeaa6640 1 -- 172.21.15.88:0/1504925217 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ab80b61e0 con 0x7f2aa00039b0 2024-09-16T01:34:59.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.465+0000 7f2aad7fa640 1 -- 172.21.15.88:0/1504925217 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2ab4044070 con 0x7f2aa00039b0 2024-09-16T01:34:59.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.465+0000 7f2aad7fa640 1 --2- 172.21.15.88:0/1504925217 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2aa406c2b0 0x7f2aa406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:34:59.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.465+0000 7f2aad7fa640 1 -- 172.21.15.88:0/1504925217 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 5227+0+0 (secure 0 0 0) 0x7f2ab40ab6a0 con 0x7f2aa00039b0 2024-09-16T01:34:59.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.465+0000 7f2aaffff640 1 --2- 172.21.15.88:0/1504925217 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2aa406c2b0 0x7f2aa406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:34:59.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.466+0000 7f2aaffff640 1 --2- 172.21.15.88:0/1504925217 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2aa406c2b0 0x7f2aa406e770 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8006d40 tx=0x7f2aa8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:34:59.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.470+0000 7f2aad7fa640 1 -- 172.21.15.88:0/1504925217 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2ab404b070 con 0x7f2aa00039b0 2024-09-16T01:34:59.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:34:59.637+0000 7f2abeaa6640 1 -- 172.21.15.88:0/1504925217 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f2ab80b4d50 con 0x7f2aa00039b0 2024-09-16T01:35:00.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.330+0000 7f2aad7fa640 1 -- 172.21.15.88:0/1504925217 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v30) ==== 170+0+0 (secure 0 0 0) 0x7f2ab4004e20 con 0x7f2aa00039b0 2024-09-16T01:35:00.331 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_data' created 2024-09-16T01:35:00.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.333+0000 7f2abeaa6640 1 -- 172.21.15.88:0/1504925217 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2aa406c2b0 msgr2=0x7f2aa406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:00.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.333+0000 7f2abeaa6640 1 --2- 172.21.15.88:0/1504925217 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2aa406c2b0 0x7f2aa406e770 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8006d40 tx=0x7f2aa8008040 comp rx=0 tx=0).stop 2024-09-16T01:35:00.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.333+0000 7f2abeaa6640 1 -- 172.21.15.88:0/1504925217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2aa00039b0 msgr2=0x7f2ab815d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:00.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.333+0000 7f2abeaa6640 1 --2- 172.21.15.88:0/1504925217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2aa00039b0 0x7f2ab815d3f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2ab400c210 tx=0x7f2ab4001a40 comp rx=0 tx=0).stop 2024-09-16T01:35:00.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.333+0000 7f2abeaa6640 1 -- 172.21.15.88:0/1504925217 shutdown_connections 2024-09-16T01:35:00.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.333+0000 7f2abeaa6640 1 --2- 172.21.15.88:0/1504925217 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2aa406c2b0 0x7f2aa406e770 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:00.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.333+0000 7f2abeaa6640 1 --2- 172.21.15.88:0/1504925217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ab80cb830 0x7f2ab80b8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:00.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.333+0000 7f2abeaa6640 1 --2- 172.21.15.88:0/1504925217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ab8156ca0 0x7f2ab80b54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:00.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.333+0000 7f2abeaa6640 1 --2- 172.21.15.88:0/1504925217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2aa00039b0 0x7f2ab815d3f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:00.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.333+0000 7f2abeaa6640 1 -- 172.21.15.88:0/1504925217 >> 172.21.15.88:0/1504925217 conn(0x7f2ab80d11e0 msgr2=0x7f2ab815abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:00.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.333+0000 7f2abeaa6640 1 -- 172.21.15.88:0/1504925217 shutdown_connections 2024-09-16T01:35:00.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.333+0000 7f2abeaa6640 1 -- 172.21.15.88:0/1504925217 wait complete. 2024-09-16T01:35:00.357 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-16T01:35:00.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.461+0000 7fbca0f11640 1 Processor -- start 2024-09-16T01:35:00.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.461+0000 7fbca0f11640 1 -- start start 2024-09-16T01:35:00.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.462+0000 7fbca0f11640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbc9c0b6290 0x7fbc9c071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.462+0000 7fbca0f11640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc9c072530 0x7fbc9c15bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.462+0000 7fbca0f11640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc9c0ccff0 con 0x7fbc9c15b480 2024-09-16T01:35:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.462+0000 7fbca0f11640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbc9c0cd2d0 con 0x7fbc9c0b6290 2024-09-16T01:35:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.462+0000 7fbca0f11640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbc9c0cd5b0 con 0x7fbc9c072530 2024-09-16T01:35:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.462+0000 7fbc9ad9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc9c072530 0x7fbc9c15bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.462+0000 7fbc99d9b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbc9c0b6290 0x7fbc9c071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.462+0000 7fbc9ad9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc9c072530 0x7fbc9c15bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:44406/0 (socket says 172.21.15.88:44406) 2024-09-16T01:35:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.462+0000 7fbc9ad9d640 1 -- 172.21.15.88:0/1887082915 learned_addr learned my addr 172.21.15.88:0/1887082915 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.462+0000 7fbc99d9b640 1 --2- 172.21.15.88:0/1887082915 >> v2:172.21.15.154:3300/0 conn(0x7fbc9c0b6290 0x7fbc9c071ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:52652/0 (socket says 172.21.15.88:52652) 2024-09-16T01:35:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.462+0000 7fbc9a59c640 1 --1- 172.21.15.88:0/1887082915 >> v1:172.21.15.154:6789/0 conn(0x7fbc9c15b480 0x7fbc9c0b5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:45152/0 (socket says 172.21.15.88:45152) 2024-09-16T01:35:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.462+0000 7fbc9ad9d640 1 -- 172.21.15.88:0/1887082915 >> v2:172.21.15.154:3300/0 conn(0x7fbc9c0b6290 msgr2=0x7fbc9c071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:00.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.462+0000 7fbc9ad9d640 1 --2- 172.21.15.88:0/1887082915 >> v2:172.21.15.154:3300/0 conn(0x7fbc9c0b6290 0x7fbc9c071ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:00.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.462+0000 7fbc9ad9d640 1 -- 172.21.15.88:0/1887082915 >> v1:172.21.15.154:6789/0 conn(0x7fbc9c15b480 legacy=0x7fbc9c0b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:00.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.462+0000 7fbc9959a640 1 -- 172.21.15.88:0/1887082915 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1817660420 0 0) 0x7fbc9c0ccff0 con 0x7fbc9c15b480 2024-09-16T01:35:00.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.462+0000 7fbc9ad9d640 1 -- 172.21.15.88:0/1887082915 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc9c0cd890 con 0x7fbc9c072530 2024-09-16T01:35:00.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.462+0000 7fbc99d9b640 1 --2- 172.21.15.88:0/1887082915 >> v2:172.21.15.154:3300/0 conn(0x7fbc9c0b6290 0x7fbc9c071ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:35:00.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.463+0000 7fbc9ad9d640 1 --2- 172.21.15.88:0/1887082915 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc9c072530 0x7fbc9c15bc60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbc94009e50 tx=0x7fbc9402a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb7804109ac8ad67 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:00.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.463+0000 7fbc9959a640 1 -- 172.21.15.88:0/1887082915 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc94036070 con 0x7fbc9c072530 2024-09-16T01:35:00.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.463+0000 7fbc9959a640 1 -- 172.21.15.88:0/1887082915 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc94031600 con 0x7fbc9c072530 2024-09-16T01:35:00.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.463+0000 7fbc9959a640 1 -- 172.21.15.88:0/1887082915 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc940319d0 con 0x7fbc9c072530 2024-09-16T01:35:00.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.464+0000 7fbca0f11640 1 -- 172.21.15.88:0/1887082915 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc9c072530 msgr2=0x7fbc9c15bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:00.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.464+0000 7fbca0f11640 1 --2- 172.21.15.88:0/1887082915 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc9c072530 0x7fbc9c15bc60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbc94009e50 tx=0x7fbc9402a680 comp rx=0 tx=0).stop 2024-09-16T01:35:00.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.464+0000 7fbca0f11640 1 -- 172.21.15.88:0/1887082915 shutdown_connections 2024-09-16T01:35:00.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.464+0000 7fbca0f11640 1 --2- 172.21.15.88:0/1887082915 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc9c072530 0x7fbc9c15bc60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:00.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.464+0000 7fbca0f11640 1 --2- 172.21.15.88:0/1887082915 >> v2:172.21.15.154:3300/0 conn(0x7fbc9c0b6290 0x7fbc9c071ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:00.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.464+0000 7fbca0f11640 1 -- 172.21.15.88:0/1887082915 >> 172.21.15.88:0/1887082915 conn(0x7fbc9c06d0f0 msgr2=0x7fbc9c06d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:00.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.464+0000 7fbca0f11640 1 -- 172.21.15.88:0/1887082915 shutdown_connections 2024-09-16T01:35:00.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.464+0000 7fbca0f11640 1 -- 172.21.15.88:0/1887082915 wait complete. 2024-09-16T01:35:00.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.465+0000 7fbca0f11640 1 Processor -- start 2024-09-16T01:35:00.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.465+0000 7fbca0f11640 1 -- start start 2024-09-16T01:35:00.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.466+0000 7fbca0f11640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc9c072530 0x7fbc9c141b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:00.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.466+0000 7fbca0f11640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc9c15b480 0x7fbc9c151150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:00.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.466+0000 7fbca0f11640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc9c1470d0 0x7fbc9c158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:00.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.466+0000 7fbca0f11640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9c15b9e0 con 0x7fbc9c072530 2024-09-16T01:35:00.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.466+0000 7fbca0f11640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9c15b860 con 0x7fbc9c15b480 2024-09-16T01:35:00.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.466+0000 7fbca0f11640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbc9c15c1a0 con 0x7fbc9c1470d0 2024-09-16T01:35:00.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.466+0000 7fbc9a59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc9c072530 0x7fbc9c141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:00.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.466+0000 7fbc9a59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc9c072530 0x7fbc9c141b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36344/0 (socket says 172.21.15.88:36344) 2024-09-16T01:35:00.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.466+0000 7fbc9a59c640 1 -- 172.21.15.88:0/3095091485 learned_addr learned my addr 172.21.15.88:0/3095091485 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:00.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.466+0000 7fbc9ad9d640 1 --2- 172.21.15.88:0/3095091485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc9c1470d0 0x7fbc9c158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:00.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.466+0000 7fbc99d9b640 1 --2- 172.21.15.88:0/3095091485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc9c15b480 0x7fbc9c151150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:00.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.466+0000 7fbc9a59c640 1 -- 172.21.15.88:0/3095091485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc9c1470d0 msgr2=0x7fbc9c158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:00.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.466+0000 7fbc9a59c640 1 --2- 172.21.15.88:0/3095091485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc9c1470d0 0x7fbc9c158ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:00.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.466+0000 7fbc9a59c640 1 -- 172.21.15.88:0/3095091485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc9c15b480 msgr2=0x7fbc9c151150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:00.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.466+0000 7fbc9a59c640 1 --2- 172.21.15.88:0/3095091485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc9c15b480 0x7fbc9c151150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:00.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.466+0000 7fbc9a59c640 1 -- 172.21.15.88:0/3095091485 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc9c149d60 con 0x7fbc9c072530 2024-09-16T01:35:00.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.466+0000 7fbc9a59c640 1 --2- 172.21.15.88:0/3095091485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc9c072530 0x7fbc9c141b80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbc8400c460 tx=0x7fbc8400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:00.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.467+0000 7fbc92ffd640 1 -- 172.21.15.88:0/3095091485 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc84018070 con 0x7fbc9c072530 2024-09-16T01:35:00.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.467+0000 7fbc92ffd640 1 -- 172.21.15.88:0/3095091485 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc84007500 con 0x7fbc9c072530 2024-09-16T01:35:00.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.467+0000 7fbca0f11640 1 -- 172.21.15.88:0/3095091485 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc9c14a040 con 0x7fbc9c072530 2024-09-16T01:35:00.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.467+0000 7fbc92ffd640 1 -- 172.21.15.88:0/3095091485 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc84004a20 con 0x7fbc9c072530 2024-09-16T01:35:00.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.467+0000 7fbca0f11640 1 -- 172.21.15.88:0/3095091485 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc9c146920 con 0x7fbc9c072530 2024-09-16T01:35:00.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.468+0000 7fbca0f11640 1 -- 172.21.15.88:0/3095091485 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc9c15bc60 con 0x7fbc9c072530 2024-09-16T01:35:00.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.468+0000 7fbc92ffd640 1 -- 172.21.15.88:0/3095091485 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fbc84004bc0 con 0x7fbc9c072530 2024-09-16T01:35:00.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.469+0000 7fbc92ffd640 1 --2- 172.21.15.88:0/3095091485 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc7c06c2b0 0x7fbc7c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:00.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.469+0000 7fbc92ffd640 1 -- 172.21.15.88:0/3095091485 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5609+0+0 (secure 0 0 0) 0x7fbc8408aa30 con 0x7fbc9c072530 2024-09-16T01:35:00.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.469+0000 7fbc99d9b640 1 --2- 172.21.15.88:0/3095091485 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc7c06c2b0 0x7fbc7c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:00.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.469+0000 7fbc99d9b640 1 --2- 172.21.15.88:0/3095091485 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc7c06c2b0 0x7fbc7c06e770 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c009560 tx=0x7fbc8c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:00.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.473+0000 7fbc92ffd640 1 -- 172.21.15.88:0/3095091485 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fbc84021080 con 0x7fbc9c072530 2024-09-16T01:35:00.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:00.673+0000 7fbca0f11640 1 -- 172.21.15.88:0/3095091485 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fbc9c06d5e0 con 0x7fbc9c072530 2024-09-16T01:35:01.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.345+0000 7fbc92ffd640 1 -- 172.21.15.88:0/3095091485 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v11) ==== 172+0+0 (secure 0 0 0) 0x7fbc84059200 con 0x7fbc9c072530 2024-09-16T01:35:01.349 INFO:teuthology.orchestra.run.smithi088.stderr:new fs with metadata pool 4 and data pool 5 2024-09-16T01:35:01.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.353+0000 7fbca0f11640 1 -- 172.21.15.88:0/3095091485 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc7c06c2b0 msgr2=0x7fbc7c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:01.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.353+0000 7fbca0f11640 1 --2- 172.21.15.88:0/3095091485 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc7c06c2b0 0x7fbc7c06e770 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c009560 tx=0x7fbc8c008040 comp rx=0 tx=0).stop 2024-09-16T01:35:01.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.353+0000 7fbca0f11640 1 -- 172.21.15.88:0/3095091485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc9c072530 msgr2=0x7fbc9c141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:01.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.353+0000 7fbca0f11640 1 --2- 172.21.15.88:0/3095091485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc9c072530 0x7fbc9c141b80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbc8400c460 tx=0x7fbc8400c920 comp rx=0 tx=0).stop 2024-09-16T01:35:01.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.354+0000 7fbca0f11640 1 -- 172.21.15.88:0/3095091485 shutdown_connections 2024-09-16T01:35:01.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.354+0000 7fbca0f11640 1 --2- 172.21.15.88:0/3095091485 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc7c06c2b0 0x7fbc7c06e770 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:01.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.354+0000 7fbca0f11640 1 --2- 172.21.15.88:0/3095091485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc9c1470d0 0x7fbc9c158ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:01.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.354+0000 7fbca0f11640 1 --2- 172.21.15.88:0/3095091485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc9c15b480 0x7fbc9c151150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:01.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.354+0000 7fbca0f11640 1 --2- 172.21.15.88:0/3095091485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc9c072530 0x7fbc9c141b80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:01.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.354+0000 7fbca0f11640 1 -- 172.21.15.88:0/3095091485 >> 172.21.15.88:0/3095091485 conn(0x7fbc9c06d0f0 msgr2=0x7fbc9c15cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:01.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.354+0000 7fbca0f11640 1 -- 172.21.15.88:0/3095091485 shutdown_connections 2024-09-16T01:35:01.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.354+0000 7fbca0f11640 1 -- 172.21.15.88:0/3095091485 wait complete. 2024-09-16T01:35:01.370 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:35:01.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.468+0000 7f03e5caf640 1 Processor -- start 2024-09-16T01:35:01.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.468+0000 7f03e5caf640 1 -- start start 2024-09-16T01:35:01.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.469+0000 7f03e5caf640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f03e015a9c0 0x7f03e015e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:01.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.469+0000 7f03e5caf640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03e00b5be0 0x7f03e015eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:01.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.469+0000 7f03e5caf640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03e00b6830 con 0x7f03e00b29a0 2024-09-16T01:35:01.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.469+0000 7f03e5caf640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f03e015f410 con 0x7f03e015a9c0 2024-09-16T01:35:01.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.469+0000 7f03e5caf640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f03e00cd0b0 con 0x7f03e00b5be0 2024-09-16T01:35:01.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.469+0000 7f03df7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f03e015a9c0 0x7f03e015e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:01.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.469+0000 7f03df7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f03e015a9c0 0x7f03e015e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36352/0 (socket says 172.21.15.88:36352) 2024-09-16T01:35:01.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.469+0000 7f03df7fe640 1 -- 172.21.15.88:0/3410452592 learned_addr learned my addr 172.21.15.88:0/3410452592 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:01.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.469+0000 7f03deffd640 1 --2- 172.21.15.88:0/3410452592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03e00b5be0 0x7f03e015eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:01.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.469+0000 7f03df7fe640 1 -- 172.21.15.88:0/3410452592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03e00b5be0 msgr2=0x7f03e015eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:01.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.469+0000 7f03df7fe640 1 --2- 172.21.15.88:0/3410452592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03e00b5be0 0x7f03e015eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:01.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.469+0000 7f03df7fe640 1 -- 172.21.15.88:0/3410452592 >> v1:172.21.15.154:6789/0 conn(0x7f03e00b29a0 legacy=0x7f03e00b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:01.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.469+0000 7f03df7fe640 1 -- 172.21.15.88:0/3410452592 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e00cd390 con 0x7f03e015a9c0 2024-09-16T01:35:01.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.469+0000 7f03deffd640 1 --2- 172.21.15.88:0/3410452592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03e00b5be0 0x7f03e015eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:01.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.470+0000 7f03df7fe640 1 --2- 172.21.15.88:0/3410452592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03e015a9c0 0x7f03e015e600 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f03d0009870 tx=0x7f03d002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42246802bfc0a2aa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:01.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.470+0000 7f03de7fc640 1 -- 172.21.15.88:0/3410452592 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d0037070 con 0x7f03e015a9c0 2024-09-16T01:35:01.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.470+0000 7f03de7fc640 1 -- 172.21.15.88:0/3410452592 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03d0032440 con 0x7f03e015a9c0 2024-09-16T01:35:01.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.470+0000 7f03de7fc640 1 -- 172.21.15.88:0/3410452592 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d0032810 con 0x7f03e015a9c0 2024-09-16T01:35:01.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.471+0000 7f03e5caf640 1 -- 172.21.15.88:0/3410452592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03e015a9c0 msgr2=0x7f03e015e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:01.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.471+0000 7f03e5caf640 1 --2- 172.21.15.88:0/3410452592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03e015a9c0 0x7f03e015e600 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f03d0009870 tx=0x7f03d002a270 comp rx=0 tx=0).stop 2024-09-16T01:35:01.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.471+0000 7f03e5caf640 1 -- 172.21.15.88:0/3410452592 shutdown_connections 2024-09-16T01:35:01.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.471+0000 7f03e5caf640 1 --2- 172.21.15.88:0/3410452592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03e00b5be0 0x7f03e015eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:01.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.471+0000 7f03e5caf640 1 --2- 172.21.15.88:0/3410452592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03e015a9c0 0x7f03e015e600 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:01.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.471+0000 7f03e5caf640 1 -- 172.21.15.88:0/3410452592 >> 172.21.15.88:0/3410452592 conn(0x7f03e006bc10 msgr2=0x7f03e006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:01.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.471+0000 7f03e5caf640 1 -- 172.21.15.88:0/3410452592 shutdown_connections 2024-09-16T01:35:01.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.471+0000 7f03e5caf640 1 -- 172.21.15.88:0/3410452592 wait complete. 2024-09-16T01:35:01.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.472+0000 7f03e5caf640 1 Processor -- start 2024-09-16T01:35:01.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.472+0000 7f03e5caf640 1 -- start start 2024-09-16T01:35:01.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.473+0000 7f03e5caf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03e00b29a0 0x7f03e01f80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:01.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.473+0000 7f03e5caf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03e00b5be0 0x7f03e01357f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:01.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.473+0000 7f03e5caf640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03e015a9c0 0x7f03e014edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:01.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.473+0000 7f03e5caf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f03e015ada0 con 0x7f03e00b29a0 2024-09-16T01:35:01.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.473+0000 7f03e5caf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f03e00cd950 con 0x7f03e00b5be0 2024-09-16T01:35:01.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.473+0000 7f03e5caf640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f03e015f080 con 0x7f03e015a9c0 2024-09-16T01:35:01.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.473+0000 7f03df7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03e00b29a0 0x7f03e01f80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:01.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.473+0000 7f03df7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03e00b29a0 0x7f03e01f80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36356/0 (socket says 172.21.15.88:36356) 2024-09-16T01:35:01.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.473+0000 7f03df7fe640 1 -- 172.21.15.88:0/2261285206 learned_addr learned my addr 172.21.15.88:0/2261285206 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:01.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.473+0000 7f03deffd640 1 --2- 172.21.15.88:0/2261285206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03e00b5be0 0x7f03e01357f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:01.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.473+0000 7f03dffff640 1 --2- 172.21.15.88:0/2261285206 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03e015a9c0 0x7f03e014edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:01.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.473+0000 7f03df7fe640 1 -- 172.21.15.88:0/2261285206 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03e015a9c0 msgr2=0x7f03e014edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:01.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.473+0000 7f03df7fe640 1 --2- 172.21.15.88:0/2261285206 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03e015a9c0 0x7f03e014edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:01.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.473+0000 7f03df7fe640 1 -- 172.21.15.88:0/2261285206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03e00b5be0 msgr2=0x7f03e01357f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:01.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.473+0000 7f03df7fe640 1 --2- 172.21.15.88:0/2261285206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03e00b5be0 0x7f03e01357f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:01.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.473+0000 7f03df7fe640 1 -- 172.21.15.88:0/2261285206 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03e01f85e0 con 0x7f03e00b29a0 2024-09-16T01:35:01.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.473+0000 7f03deffd640 1 --2- 172.21.15.88:0/2261285206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03e00b5be0 0x7f03e01357f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:01.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.473+0000 7f03df7fe640 1 --2- 172.21.15.88:0/2261285206 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03e00b29a0 0x7f03e01f80a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f03d0000c00 tx=0x7f03d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:01.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.473+0000 7f03c7fff640 1 -- 172.21.15.88:0/2261285206 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d0041020 con 0x7f03e00b29a0 2024-09-16T01:35:01.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.473+0000 7f03c7fff640 1 -- 172.21.15.88:0/2261285206 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03d00306c0 con 0x7f03e00b29a0 2024-09-16T01:35:01.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.474+0000 7f03c7fff640 1 -- 172.21.15.88:0/2261285206 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03d0037040 con 0x7f03e00b29a0 2024-09-16T01:35:01.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.474+0000 7f03e5caf640 1 -- 172.21.15.88:0/2261285206 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03e0134750 con 0x7f03e00b29a0 2024-09-16T01:35:01.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.474+0000 7f03e5caf640 1 -- 172.21.15.88:0/2261285206 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03e0134d10 con 0x7f03e00b29a0 2024-09-16T01:35:01.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.474+0000 7f03e5caf640 1 -- 172.21.15.88:0/2261285206 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03e006de40 con 0x7f03e00b29a0 2024-09-16T01:35:01.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.475+0000 7f03c7fff640 1 -- 172.21.15.88:0/2261285206 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f03d0044020 con 0x7f03e00b29a0 2024-09-16T01:35:01.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.475+0000 7f03c7fff640 1 --2- 172.21.15.88:0/2261285206 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f03cc06c2b0 0x7f03cc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:01.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.476+0000 7f03c7fff640 1 -- 172.21.15.88:0/2261285206 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5709+0+0 (secure 0 0 0) 0x7f03d00abcb0 con 0x7f03e00b29a0 2024-09-16T01:35:01.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.476+0000 7f03deffd640 1 --2- 172.21.15.88:0/2261285206 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f03cc06c2b0 0x7f03cc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:01.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.476+0000 7f03deffd640 1 --2- 172.21.15.88:0/2261285206 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f03cc06c2b0 0x7f03cc06e770 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f03e014b7e0 tx=0x7f03d4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:01.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.480+0000 7f03c7fff640 1 -- 172.21.15.88:0/2261285206 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f03d004b030 con 0x7f03e00b29a0 2024-09-16T01:35:01.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.647+0000 7f03e5caf640 1 -- 172.21.15.88:0/2261285206 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f03e00633d0 con 0x7f03e00b29a0 2024-09-16T01:35:01.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.649+0000 7f03c7fff640 1 -- 172.21.15.88:0/2261285206 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) ==== 74+0+16718 (secure 0 0 0) 0x7f03d007a420 con 0x7f03e00b29a0 2024-09-16T01:35:01.650 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:35:01.650 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":31,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:35:00.674346+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:34:58.555164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-16T01:34:59.655760+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:35:01.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.650+0000 7f03e5caf640 1 -- 172.21.15.88:0/2261285206 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f03cc06c2b0 msgr2=0x7f03cc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:01.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.650+0000 7f03e5caf640 1 --2- 172.21.15.88:0/2261285206 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f03cc06c2b0 0x7f03cc06e770 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f03e014b7e0 tx=0x7f03d4008040 comp rx=0 tx=0).stop 2024-09-16T01:35:01.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.650+0000 7f03e5caf640 1 -- 172.21.15.88:0/2261285206 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03e00b29a0 msgr2=0x7f03e01f80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:01.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.650+0000 7f03e5caf640 1 --2- 172.21.15.88:0/2261285206 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03e00b29a0 0x7f03e01f80a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f03d0000c00 tx=0x7f03d00046c0 comp rx=0 tx=0).stop 2024-09-16T01:35:01.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.650+0000 7f03e5caf640 1 -- 172.21.15.88:0/2261285206 shutdown_connections 2024-09-16T01:35:01.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.650+0000 7f03e5caf640 1 --2- 172.21.15.88:0/2261285206 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f03cc06c2b0 0x7f03cc06e770 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:01.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.650+0000 7f03e5caf640 1 --2- 172.21.15.88:0/2261285206 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03e015a9c0 0x7f03e014edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:01.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.650+0000 7f03e5caf640 1 --2- 172.21.15.88:0/2261285206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03e00b5be0 0x7f03e01357f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:01.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.650+0000 7f03e5caf640 1 --2- 172.21.15.88:0/2261285206 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03e00b29a0 0x7f03e01f80a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:01.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.650+0000 7f03e5caf640 1 -- 172.21.15.88:0/2261285206 >> 172.21.15.88:0/2261285206 conn(0x7f03e006bc10 msgr2=0x7f03e0071d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:01.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.651+0000 7f03e5caf640 1 -- 172.21.15.88:0/2261285206 shutdown_connections 2024-09-16T01:35:01.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.651+0000 7f03e5caf640 1 -- 172.21.15.88:0/2261285206 wait complete. 2024-09-16T01:35:01.663 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:35:01.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.759+0000 7f6c2f09b640 1 Processor -- start 2024-09-16T01:35:01.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.759+0000 7f6c2f09b640 1 -- start start 2024-09-16T01:35:01.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.760+0000 7f6c2f09b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c280b74c0 0x7f6c2815eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.760+0000 7f6c2f09b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6c2815f030 0x7f6c2806d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.760+0000 7f6c2f09b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c2815fb10 con 0x7f6c2815c2b0 2024-09-16T01:35:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.760+0000 7f6c2f09b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6c280b5900 con 0x7f6c2815f030 2024-09-16T01:35:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.760+0000 7f6c2f09b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6c2815f830 con 0x7f6c280b74c0 2024-09-16T01:35:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.760+0000 7f6c1ffff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c280b74c0 0x7f6c2815eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.760+0000 7f6c2d638640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6c2815f030 0x7f6c2806d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.760+0000 7f6c1ffff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c280b74c0 0x7f6c2815eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:44448/0 (socket says 172.21.15.88:44448) 2024-09-16T01:35:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.760+0000 7f6c1ffff640 1 -- 172.21.15.88:0/700646289 learned_addr learned my addr 172.21.15.88:0/700646289 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:01.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.761+0000 7f6c1f7fe640 1 -- 172.21.15.88:0/700646289 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 831392909 0 0) 0x7f6c2815fb10 con 0x7f6c2815c2b0 2024-09-16T01:35:01.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.761+0000 7f6c1f7fe640 1 -- 172.21.15.88:0/700646289 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c10003520 con 0x7f6c2815c2b0 2024-09-16T01:35:01.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.761+0000 7f6c2d638640 1 -- 172.21.15.88:0/700646289 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c280b74c0 msgr2=0x7f6c2815eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:01.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.761+0000 7f6c2d638640 1 --2- 172.21.15.88:0/700646289 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c280b74c0 0x7f6c2815eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:01.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.761+0000 7f6c2d638640 1 -- 172.21.15.88:0/700646289 >> v1:172.21.15.154:6789/0 conn(0x7f6c2815c2b0 legacy=0x7f6c280b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:01.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.761+0000 7f6c2d638640 1 -- 172.21.15.88:0/700646289 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c2806ce30 con 0x7f6c2815f030 2024-09-16T01:35:01.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.761+0000 7f6c1ffff640 1 --2- 172.21.15.88:0/700646289 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c280b74c0 0x7f6c2815eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:35:01.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.761+0000 7f6c2d638640 1 --2- 172.21.15.88:0/700646289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c2815f030 0x7f6c2806d910 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6c20009d90 tx=0x7f6c2002a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da2f701a58176774 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.762+0000 7f6c1f7fe640 1 -- 172.21.15.88:0/700646289 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c20007740 con 0x7f6c2815f030 2024-09-16T01:35:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.762+0000 7f6c1f7fe640 1 -- 172.21.15.88:0/700646289 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c200078e0 con 0x7f6c2815f030 2024-09-16T01:35:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.762+0000 7f6c1f7fe640 1 -- 172.21.15.88:0/700646289 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c20007c90 con 0x7f6c2815f030 2024-09-16T01:35:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.762+0000 7f6c2f09b640 1 -- 172.21.15.88:0/700646289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c2815f030 msgr2=0x7f6c2806d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.762+0000 7f6c2f09b640 1 --2- 172.21.15.88:0/700646289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c2815f030 0x7f6c2806d910 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6c20009d90 tx=0x7f6c2002a660 comp rx=0 tx=0).stop 2024-09-16T01:35:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.762+0000 7f6c2f09b640 1 -- 172.21.15.88:0/700646289 shutdown_connections 2024-09-16T01:35:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.762+0000 7f6c2f09b640 1 --2- 172.21.15.88:0/700646289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c2815f030 0x7f6c2806d910 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.762+0000 7f6c2f09b640 1 --2- 172.21.15.88:0/700646289 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c280b74c0 0x7f6c2815eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.762+0000 7f6c2f09b640 1 -- 172.21.15.88:0/700646289 >> 172.21.15.88:0/700646289 conn(0x7f6c280b4240 msgr2=0x7f6c280b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.762+0000 7f6c2f09b640 1 -- 172.21.15.88:0/700646289 shutdown_connections 2024-09-16T01:35:01.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.762+0000 7f6c2f09b640 1 -- 172.21.15.88:0/700646289 wait complete. 2024-09-16T01:35:01.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.763+0000 7f6c2f09b640 1 Processor -- start 2024-09-16T01:35:01.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.763+0000 7f6c2f09b640 1 -- start start 2024-09-16T01:35:01.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.764+0000 7f6c2f09b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c280b74c0 0x7f6c28158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:01.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.764+0000 7f6c2f09b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c2815c2b0 0x7f6c28160230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:01.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.764+0000 7f6c2f09b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c2815f030 0x7f6c28160770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:01.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.764+0000 7f6c2f09b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2815f4c0 con 0x7f6c2815c2b0 2024-09-16T01:35:01.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.764+0000 7f6c2f09b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2806d3f0 con 0x7f6c280b74c0 2024-09-16T01:35:01.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.764+0000 7f6c2f09b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6c2806de50 con 0x7f6c2815f030 2024-09-16T01:35:01.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.764+0000 7f6c1ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c2815c2b0 0x7f6c28160230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:01.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.764+0000 7f6c1ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c2815c2b0 0x7f6c28160230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36372/0 (socket says 172.21.15.88:36372) 2024-09-16T01:35:01.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.764+0000 7f6c1ffff640 1 -- 172.21.15.88:0/3429563397 learned_addr learned my addr 172.21.15.88:0/3429563397 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:01.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.765+0000 7f6c2d638640 1 --2- 172.21.15.88:0/3429563397 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c2815f030 0x7f6c28160770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:01.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.765+0000 7f6c1ffff640 1 -- 172.21.15.88:0/3429563397 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c2815f030 msgr2=0x7f6c28160770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:01.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.765+0000 7f6c1ffff640 1 --2- 172.21.15.88:0/3429563397 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c2815f030 0x7f6c28160770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:01.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.765+0000 7f6c1ffff640 1 -- 172.21.15.88:0/3429563397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c280b74c0 msgr2=0x7f6c28158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:01.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.765+0000 7f6c2ce37640 1 --2- 172.21.15.88:0/3429563397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c280b74c0 0x7f6c28158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:01.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.765+0000 7f6c1ffff640 1 --2- 172.21.15.88:0/3429563397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c280b74c0 0x7f6c28158dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:01.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.765+0000 7f6c1ffff640 1 -- 172.21.15.88:0/3429563397 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c281e7020 con 0x7f6c2815c2b0 2024-09-16T01:35:01.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.765+0000 7f6c1ffff640 1 --2- 172.21.15.88:0/3429563397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c2815c2b0 0x7f6c28160230 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6c1800e9d0 tx=0x7f6c1800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:01.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.765+0000 7f6c1d7fa640 1 -- 172.21.15.88:0/3429563397 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c18017070 con 0x7f6c2815c2b0 2024-09-16T01:35:01.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.765+0000 7f6c1d7fa640 1 -- 172.21.15.88:0/3429563397 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c1800ca60 con 0x7f6c2815c2b0 2024-09-16T01:35:01.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.765+0000 7f6c2f09b640 1 -- 172.21.15.88:0/3429563397 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c281e7300 con 0x7f6c2815c2b0 2024-09-16T01:35:01.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.765+0000 7f6c1d7fa640 1 -- 172.21.15.88:0/3429563397 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c1800ce10 con 0x7f6c2815c2b0 2024-09-16T01:35:01.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.765+0000 7f6c2f09b640 1 -- 172.21.15.88:0/3429563397 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c28161040 con 0x7f6c2815c2b0 2024-09-16T01:35:01.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.766+0000 7f6c2f09b640 1 -- 172.21.15.88:0/3429563397 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c281612d0 con 0x7f6c2815c2b0 2024-09-16T01:35:01.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.767+0000 7f6c1d7fa640 1 -- 172.21.15.88:0/3429563397 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6c1801a070 con 0x7f6c2815c2b0 2024-09-16T01:35:01.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.767+0000 7f6c1d7fa640 1 --2- 172.21.15.88:0/3429563397 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6c1406c2b0 0x7f6c1406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:01.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.767+0000 7f6c1d7fa640 1 -- 172.21.15.88:0/3429563397 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5709+0+0 (secure 0 0 0) 0x7f6c1808b710 con 0x7f6c2815c2b0 2024-09-16T01:35:01.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.767+0000 7f6c2ce37640 1 --2- 172.21.15.88:0/3429563397 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6c1406c2b0 0x7f6c1406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:01.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.767+0000 7f6c2ce37640 1 --2- 172.21.15.88:0/3429563397 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6c1406c2b0 0x7f6c1406e770 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6c24007fd0 tx=0x7f6c24009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:01.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.771+0000 7f6c1d7fa640 1 -- 172.21.15.88:0/3429563397 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6c18059f10 con 0x7f6c2815c2b0 2024-09-16T01:35:01.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.941+0000 7f6c2f09b640 1 -- 172.21.15.88:0/3429563397 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6c28068e10 con 0x7f6c2815c2b0 2024-09-16T01:35:01.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.942+0000 7f6c1d7fa640 1 -- 172.21.15.88:0/3429563397 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) ==== 74+0+16718 (secure 0 0 0) 0x7f6c28068e10 con 0x7f6c2815c2b0 2024-09-16T01:35:01.944 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:35:01.944 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":31,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:35:00.674346+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:34:58.555164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-16T01:34:59.655760+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:35:01.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.944+0000 7f6c2f09b640 1 -- 172.21.15.88:0/3429563397 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6c1406c2b0 msgr2=0x7f6c1406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:01.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.944+0000 7f6c2f09b640 1 --2- 172.21.15.88:0/3429563397 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6c1406c2b0 0x7f6c1406e770 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6c24007fd0 tx=0x7f6c24009040 comp rx=0 tx=0).stop 2024-09-16T01:35:01.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.944+0000 7f6c2f09b640 1 -- 172.21.15.88:0/3429563397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c2815c2b0 msgr2=0x7f6c28160230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:01.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.944+0000 7f6c2f09b640 1 --2- 172.21.15.88:0/3429563397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c2815c2b0 0x7f6c28160230 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6c1800e9d0 tx=0x7f6c1800ee90 comp rx=0 tx=0).stop 2024-09-16T01:35:01.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.944+0000 7f6c2f09b640 1 -- 172.21.15.88:0/3429563397 shutdown_connections 2024-09-16T01:35:01.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.944+0000 7f6c2f09b640 1 --2- 172.21.15.88:0/3429563397 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6c1406c2b0 0x7f6c1406e770 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:01.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.944+0000 7f6c2f09b640 1 --2- 172.21.15.88:0/3429563397 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c2815f030 0x7f6c28160770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:01.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.944+0000 7f6c2f09b640 1 --2- 172.21.15.88:0/3429563397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c2815c2b0 0x7f6c28160230 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:01.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.944+0000 7f6c2f09b640 1 --2- 172.21.15.88:0/3429563397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c280b74c0 0x7f6c28158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:01.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.944+0000 7f6c2f09b640 1 -- 172.21.15.88:0/3429563397 >> 172.21.15.88:0/3429563397 conn(0x7f6c280b4240 msgr2=0x7f6c280b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:01.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.944+0000 7f6c2f09b640 1 -- 172.21.15.88:0/3429563397 shutdown_connections 2024-09-16T01:35:01.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:01.944+0000 7f6c2f09b640 1 -- 172.21.15.88:0/3429563397 wait complete. 2024-09-16T01:35:01.957 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-16T01:35:02.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.057+0000 7f714296d640 1 Processor -- start 2024-09-16T01:35:02.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.057+0000 7f714296d640 1 -- start start 2024-09-16T01:35:02.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.058+0000 7f714296d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f713c0716c0 0x7f713c0731f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:02.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.058+0000 7f714296d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f713c073730 0x7f713c15eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:02.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.058+0000 7f714296d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f713c073f60 con 0x7f713c15e7c0 2024-09-16T01:35:02.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.058+0000 7f714296d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f713c0b4b60 con 0x7f713c073730 2024-09-16T01:35:02.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.058+0000 7f714296d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f713c0b4e40 con 0x7f713c0716c0 2024-09-16T01:35:02.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.058+0000 7f7140f0a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f713c073730 0x7f713c15eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:02.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.059+0000 7f713bfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f713c15e7c0 0x7f713c070fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35954/0 (socket says 172.21.15.88:35954) 2024-09-16T01:35:02.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.059+0000 7f7140f0a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f713c073730 0x7f713c15eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36382/0 (socket says 172.21.15.88:36382) 2024-09-16T01:35:02.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.059+0000 7f713bfff640 1 -- 172.21.15.88:0/353537968 learned_addr learned my addr 172.21.15.88:0/353537968 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:02.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.059+0000 7f7140f0a640 1 -- 172.21.15.88:0/353537968 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f713c0716c0 msgr2=0x7f713c0731f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:02.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.059+0000 7f7133fff640 1 --2- 172.21.15.88:0/353537968 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f713c0716c0 0x7f713c0731f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:02.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.059+0000 7f7140f0a640 1 --2- 172.21.15.88:0/353537968 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f713c0716c0 0x7f713c0731f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:02.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.059+0000 7f7140f0a640 1 -- 172.21.15.88:0/353537968 >> v1:172.21.15.88:6789/0 conn(0x7f713c15e7c0 legacy=0x7f713c070fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:02.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.059+0000 7f713b7fe640 1 -- 172.21.15.88:0/353537968 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4182340835 0 0) 0x7f713c073f60 con 0x7f713c15e7c0 2024-09-16T01:35:02.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.059+0000 7f7140f0a640 1 -- 172.21.15.88:0/353537968 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f713c0b5120 con 0x7f713c073730 2024-09-16T01:35:02.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.059+0000 7f7133fff640 1 --2- 172.21.15.88:0/353537968 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f713c0716c0 0x7f713c0731f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:02.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.059+0000 7f7140f0a640 1 --2- 172.21.15.88:0/353537968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f713c073730 0x7f713c15eba0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7134009d90 tx=0x7f713402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeb5e9f1174eb310 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:02.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.060+0000 7f713b7fe640 1 -- 172.21.15.88:0/353537968 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7134007740 con 0x7f713c073730 2024-09-16T01:35:02.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.060+0000 7f713b7fe640 1 -- 172.21.15.88:0/353537968 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71340078e0 con 0x7f713c073730 2024-09-16T01:35:02.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.060+0000 7f713b7fe640 1 -- 172.21.15.88:0/353537968 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7134007cb0 con 0x7f713c073730 2024-09-16T01:35:02.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.060+0000 7f714296d640 1 -- 172.21.15.88:0/353537968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f713c073730 msgr2=0x7f713c15eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:02.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.060+0000 7f714296d640 1 --2- 172.21.15.88:0/353537968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f713c073730 0x7f713c15eba0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7134009d90 tx=0x7f713402a660 comp rx=0 tx=0).stop 2024-09-16T01:35:02.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.060+0000 7f714296d640 1 -- 172.21.15.88:0/353537968 shutdown_connections 2024-09-16T01:35:02.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.060+0000 7f714296d640 1 --2- 172.21.15.88:0/353537968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f713c073730 0x7f713c15eba0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:02.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.060+0000 7f714296d640 1 --2- 172.21.15.88:0/353537968 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f713c0716c0 0x7f713c0731f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:02.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.060+0000 7f714296d640 1 -- 172.21.15.88:0/353537968 >> 172.21.15.88:0/353537968 conn(0x7f713c06d0f0 msgr2=0x7f713c06d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:02.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.060+0000 7f714296d640 1 -- 172.21.15.88:0/353537968 shutdown_connections 2024-09-16T01:35:02.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.060+0000 7f714296d640 1 -- 172.21.15.88:0/353537968 wait complete. 2024-09-16T01:35:02.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.061+0000 7f714296d640 1 Processor -- start 2024-09-16T01:35:02.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.062+0000 7f714296d640 1 -- start start 2024-09-16T01:35:02.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.062+0000 7f714296d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f713c0716c0 0x7f713c159c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:02.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.062+0000 7f714296d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f713c073730 0x7f713c1521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:02.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.062+0000 7f714296d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f713c15e7c0 0x7f713c158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:02.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.062+0000 7f714296d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f713c15ff50 con 0x7f713c15e7c0 2024-09-16T01:35:02.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.062+0000 7f714296d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f713c15fdd0 con 0x7f713c0716c0 2024-09-16T01:35:02.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.062+0000 7f714296d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f713c15f0e0 con 0x7f713c073730 2024-09-16T01:35:02.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.062+0000 7f7140f0a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f713c15e7c0 0x7f713c158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:02.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.062+0000 7f7140f0a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f713c15e7c0 0x7f713c158020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36394/0 (socket says 172.21.15.88:36394) 2024-09-16T01:35:02.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.062+0000 7f7140f0a640 1 -- 172.21.15.88:0/898441548 learned_addr learned my addr 172.21.15.88:0/898441548 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:02.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.063+0000 7f7140f0a640 1 -- 172.21.15.88:0/898441548 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f713c073730 msgr2=0x7f713c1521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:02.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.063+0000 7f7133fff640 1 --2- 172.21.15.88:0/898441548 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f713c073730 0x7f713c1521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:02.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.063+0000 7f713bfff640 1 --2- 172.21.15.88:0/898441548 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f713c0716c0 0x7f713c159c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:02.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.063+0000 7f7140f0a640 1 --2- 172.21.15.88:0/898441548 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f713c073730 0x7f713c1521b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:02.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.063+0000 7f7140f0a640 1 -- 172.21.15.88:0/898441548 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f713c0716c0 msgr2=0x7f713c159c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:02.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.063+0000 7f7140f0a640 1 --2- 172.21.15.88:0/898441548 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f713c0716c0 0x7f713c159c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:02.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.063+0000 7f7140f0a640 1 -- 172.21.15.88:0/898441548 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f713c1f4280 con 0x7f713c15e7c0 2024-09-16T01:35:02.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.063+0000 7f713bfff640 1 --2- 172.21.15.88:0/898441548 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f713c0716c0 0x7f713c159c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:02.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.063+0000 7f7133fff640 1 --2- 172.21.15.88:0/898441548 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f713c073730 0x7f713c1521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:02.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.063+0000 7f7140f0a640 1 --2- 172.21.15.88:0/898441548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f713c15e7c0 0x7f713c158020 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f713402ac50 tx=0x7f7134004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:02.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.063+0000 7f71397fa640 1 -- 172.21.15.88:0/898441548 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7134040070 con 0x7f713c15e7c0 2024-09-16T01:35:02.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.063+0000 7f71397fa640 1 -- 172.21.15.88:0/898441548 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7134031c90 con 0x7f713c15e7c0 2024-09-16T01:35:02.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.063+0000 7f71397fa640 1 -- 172.21.15.88:0/898441548 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7134045680 con 0x7f713c15e7c0 2024-09-16T01:35:02.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.063+0000 7f714296d640 1 -- 172.21.15.88:0/898441548 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f713c1f4560 con 0x7f713c15e7c0 2024-09-16T01:35:02.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.063+0000 7f714296d640 1 -- 172.21.15.88:0/898441548 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f713c1f4b20 con 0x7f713c15e7c0 2024-09-16T01:35:02.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.064+0000 7f714296d640 1 -- 172.21.15.88:0/898441548 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f713c0731f0 con 0x7f713c15e7c0 2024-09-16T01:35:02.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.065+0000 7f71397fa640 1 -- 172.21.15.88:0/898441548 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7134039040 con 0x7f713c15e7c0 2024-09-16T01:35:02.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.065+0000 7f71397fa640 1 --2- 172.21.15.88:0/898441548 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f711c06c2b0 0x7f711c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:02.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.065+0000 7f71397fa640 1 -- 172.21.15.88:0/898441548 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5709+0+0 (secure 0 0 0) 0x7f71340acdf0 con 0x7f713c15e7c0 2024-09-16T01:35:02.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.065+0000 7f713bfff640 1 --2- 172.21.15.88:0/898441548 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f711c06c2b0 0x7f711c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:02.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.065+0000 7f713bfff640 1 --2- 172.21.15.88:0/898441548 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f711c06c2b0 0x7f711c06e770 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f712c000f60 tx=0x7f712c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:02.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.069+0000 7f71397fa640 1 -- 172.21.15.88:0/898441548 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7134043360 con 0x7f713c15e7c0 2024-09-16T01:35:02.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.270+0000 7f714296d640 1 -- 172.21.15.88:0/898441548 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f713c0b59e0 con 0x7f713c15e7c0 2024-09-16T01:35:02.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.351+0000 7f71397fa640 1 -- 172.21.15.88:0/898441548 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v12) ==== 118+0+0 (secure 0 0 0) 0x7f713407b620 con 0x7f713c15e7c0 2024-09-16T01:35:02.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.354+0000 7f714296d640 1 -- 172.21.15.88:0/898441548 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f711c06c2b0 msgr2=0x7f711c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:02.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.354+0000 7f714296d640 1 --2- 172.21.15.88:0/898441548 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f711c06c2b0 0x7f711c06e770 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f712c000f60 tx=0x7f712c009040 comp rx=0 tx=0).stop 2024-09-16T01:35:02.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.354+0000 7f714296d640 1 -- 172.21.15.88:0/898441548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f713c15e7c0 msgr2=0x7f713c158020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:02.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.354+0000 7f714296d640 1 --2- 172.21.15.88:0/898441548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f713c15e7c0 0x7f713c158020 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f713402ac50 tx=0x7f7134004a40 comp rx=0 tx=0).stop 2024-09-16T01:35:02.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.354+0000 7f714296d640 1 -- 172.21.15.88:0/898441548 shutdown_connections 2024-09-16T01:35:02.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.354+0000 7f714296d640 1 --2- 172.21.15.88:0/898441548 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f711c06c2b0 0x7f711c06e770 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:02.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.354+0000 7f714296d640 1 --2- 172.21.15.88:0/898441548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f713c15e7c0 0x7f713c158020 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:02.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.354+0000 7f714296d640 1 --2- 172.21.15.88:0/898441548 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f713c073730 0x7f713c1521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:02.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.354+0000 7f714296d640 1 --2- 172.21.15.88:0/898441548 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f713c0716c0 0x7f713c159c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:02.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.354+0000 7f714296d640 1 -- 172.21.15.88:0/898441548 >> 172.21.15.88:0/898441548 conn(0x7f713c06d0f0 msgr2=0x7f713c154d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:02.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.354+0000 7f714296d640 1 -- 172.21.15.88:0/898441548 shutdown_connections 2024-09-16T01:35:02.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.354+0000 7f714296d640 1 -- 172.21.15.88:0/898441548 wait complete. 2024-09-16T01:35:02.366 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-16T01:35:02.366 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-16T01:35:02.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.467+0000 7f10ba978640 1 Processor -- start 2024-09-16T01:35:02.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.467+0000 7f10ba978640 1 -- start start 2024-09-16T01:35:02.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.468+0000 7f10ba978640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10b4158a00 0x7f10b4071f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:02.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.468+0000 7f10ba978640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f10b4072470 0x7f10b40b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:02.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.468+0000 7f10ba978640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10b40728f0 con 0x7f10b4158de0 2024-09-16T01:35:02.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.468+0000 7f10ba978640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f10b4072bd0 con 0x7f10b4072470 2024-09-16T01:35:02.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.468+0000 7f10ba978640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f10b4072eb0 con 0x7f10b4158a00 2024-09-16T01:35:02.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.468+0000 7f10b37fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f10b4072470 0x7f10b40b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:02.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.468+0000 7f10b8f15640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f10b4158de0 0x7f10b4162110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35962/0 (socket says 172.21.15.88:35962) 2024-09-16T01:35:02.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.468+0000 7f10b8f15640 1 -- 172.21.15.88:0/823765139 learned_addr learned my addr 172.21.15.88:0/823765139 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:02.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.468+0000 7f10b3fff640 1 --2- 172.21.15.88:0/823765139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10b4158a00 0x7f10b4071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:02.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.468+0000 7f10b37fe640 1 -- 172.21.15.88:0/823765139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10b4158a00 msgr2=0x7f10b4071f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:02.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.468+0000 7f10b37fe640 1 --2- 172.21.15.88:0/823765139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10b4158a00 0x7f10b4071f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:02.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.468+0000 7f10b2ffd640 1 -- 172.21.15.88:0/823765139 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 42987739 0 0) 0x7f10b40728f0 con 0x7f10b4158de0 2024-09-16T01:35:02.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.468+0000 7f10b37fe640 1 -- 172.21.15.88:0/823765139 >> v1:172.21.15.88:6789/0 conn(0x7f10b4158de0 legacy=0x7f10b4162110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:02.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.468+0000 7f10b37fe640 1 -- 172.21.15.88:0/823765139 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10b4073190 con 0x7f10b4072470 2024-09-16T01:35:02.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.468+0000 7f10b3fff640 1 --2- 172.21.15.88:0/823765139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10b4158a00 0x7f10b4071f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:02.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.469+0000 7f10b37fe640 1 --2- 172.21.15.88:0/823765139 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10b4072470 0x7f10b40b4e60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f10a80097a0 tx=0x7f10a802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99373dd20e9cbd68 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:02.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.469+0000 7f10b2ffd640 1 -- 172.21.15.88:0/823765139 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a8037070 con 0x7f10b4072470 2024-09-16T01:35:02.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.469+0000 7f10b2ffd640 1 -- 172.21.15.88:0/823765139 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10a8032440 con 0x7f10b4072470 2024-09-16T01:35:02.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.469+0000 7f10b2ffd640 1 -- 172.21.15.88:0/823765139 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a8032810 con 0x7f10b4072470 2024-09-16T01:35:02.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.470+0000 7f10ba978640 1 -- 172.21.15.88:0/823765139 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10b4072470 msgr2=0x7f10b40b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:02.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.470+0000 7f10ba978640 1 --2- 172.21.15.88:0/823765139 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10b4072470 0x7f10b40b4e60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f10a80097a0 tx=0x7f10a802a180 comp rx=0 tx=0).stop 2024-09-16T01:35:02.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.470+0000 7f10ba978640 1 -- 172.21.15.88:0/823765139 shutdown_connections 2024-09-16T01:35:02.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.470+0000 7f10ba978640 1 --2- 172.21.15.88:0/823765139 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10b4072470 0x7f10b40b4e60 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:02.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.470+0000 7f10ba978640 1 --2- 172.21.15.88:0/823765139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10b4158a00 0x7f10b4071f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:02.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.470+0000 7f10ba978640 1 -- 172.21.15.88:0/823765139 >> 172.21.15.88:0/823765139 conn(0x7f10b406cb50 msgr2=0x7f10b406cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:02.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.470+0000 7f10ba978640 1 -- 172.21.15.88:0/823765139 shutdown_connections 2024-09-16T01:35:02.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.470+0000 7f10ba978640 1 -- 172.21.15.88:0/823765139 wait complete. 2024-09-16T01:35:02.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.471+0000 7f10ba978640 1 Processor -- start 2024-09-16T01:35:02.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.471+0000 7f10ba978640 1 -- start start 2024-09-16T01:35:02.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.472+0000 7f10ba978640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10b4072470 0x7f10b414b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:02.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.472+0000 7f10ba978640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10b4158a00 0x7f10b4150810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:02.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.472+0000 7f10ba978640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10b4158de0 0x7f10b4154a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:02.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.472+0000 7f10ba978640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f10b40b5ca0 con 0x7f10b4158a00 2024-09-16T01:35:02.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.472+0000 7f10ba978640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f10b40b5b20 con 0x7f10b4158de0 2024-09-16T01:35:02.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.472+0000 7f10ba978640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f10b406faf0 con 0x7f10b4072470 2024-09-16T01:35:02.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.472+0000 7f10b37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10b4158a00 0x7f10b4150810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:02.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.472+0000 7f10b37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10b4158a00 0x7f10b4150810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36412/0 (socket says 172.21.15.88:36412) 2024-09-16T01:35:02.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.472+0000 7f10b37fe640 1 -- 172.21.15.88:0/3626362914 learned_addr learned my addr 172.21.15.88:0/3626362914 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:02.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.472+0000 7f10b3fff640 1 --2- 172.21.15.88:0/3626362914 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10b4072470 0x7f10b414b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:02.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.472+0000 7f10b8f15640 1 --2- 172.21.15.88:0/3626362914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10b4158de0 0x7f10b4154a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:02.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.472+0000 7f10b37fe640 1 -- 172.21.15.88:0/3626362914 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10b4072470 msgr2=0x7f10b414b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:02.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.472+0000 7f10b37fe640 1 --2- 172.21.15.88:0/3626362914 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10b4072470 0x7f10b414b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:02.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.472+0000 7f10b37fe640 1 -- 172.21.15.88:0/3626362914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10b4158de0 msgr2=0x7f10b4154a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:02.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.472+0000 7f10b37fe640 1 --2- 172.21.15.88:0/3626362914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10b4158de0 0x7f10b4154a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:02.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.472+0000 7f10b37fe640 1 -- 172.21.15.88:0/3626362914 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10b4150ef0 con 0x7f10b4158a00 2024-09-16T01:35:02.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.472+0000 7f10b3fff640 1 --2- 172.21.15.88:0/3626362914 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10b4072470 0x7f10b414b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:02.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.472+0000 7f10b8f15640 1 --2- 172.21.15.88:0/3626362914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10b4158de0 0x7f10b4154a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:02.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.472+0000 7f10b37fe640 1 --2- 172.21.15.88:0/3626362914 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10b4158a00 0x7f10b4150810 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f10a8000c00 tx=0x7f10a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:02.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.473+0000 7f10b0ff9640 1 -- 172.21.15.88:0/3626362914 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a8037040 con 0x7f10b4158a00 2024-09-16T01:35:02.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.473+0000 7f10b0ff9640 1 -- 172.21.15.88:0/3626362914 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10a8042070 con 0x7f10b4158a00 2024-09-16T01:35:02.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.473+0000 7f10ba978640 1 -- 172.21.15.88:0/3626362914 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10b41511d0 con 0x7f10b4158a00 2024-09-16T01:35:02.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.473+0000 7f10b0ff9640 1 -- 172.21.15.88:0/3626362914 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a8030a70 con 0x7f10b4158a00 2024-09-16T01:35:02.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.473+0000 7f10ba978640 1 -- 172.21.15.88:0/3626362914 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10b4151790 con 0x7f10b4158a00 2024-09-16T01:35:02.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.474+0000 7f10ba978640 1 -- 172.21.15.88:0/3626362914 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10b4071f30 con 0x7f10b4158a00 2024-09-16T01:35:02.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.474+0000 7f10b0ff9640 1 -- 172.21.15.88:0/3626362914 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f10a8030c10 con 0x7f10b4158a00 2024-09-16T01:35:02.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.474+0000 7f10b0ff9640 1 --2- 172.21.15.88:0/3626362914 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f10a006c2b0 0x7f10a006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:02.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.475+0000 7f10b0ff9640 1 -- 172.21.15.88:0/3626362914 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f10a80abf00 con 0x7f10b4158a00 2024-09-16T01:35:02.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.475+0000 7f10b3fff640 1 --2- 172.21.15.88:0/3626362914 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f10a006c2b0 0x7f10a006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:02.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.475+0000 7f10b3fff640 1 --2- 172.21.15.88:0/3626362914 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f10a006c2b0 0x7f10a006e770 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f10a400aa30 tx=0x7f10a4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:02.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.479+0000 7f10b0ff9640 1 -- 172.21.15.88:0/3626362914 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f10a8045360 con 0x7f10b4158a00 2024-09-16T01:35:02.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:02.680+0000 7f10ba978640 1 -- 172.21.15.88:0/3626362914 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f10b4075620 con 0x7f10b4158a00 2024-09-16T01:35:03.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.356+0000 7f10b0ff9640 1 -- 172.21.15.88:0/3626362914 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v13) ==== 122+0+0 (secure 0 0 0) 0x7f10a807a670 con 0x7f10b4158a00 2024-09-16T01:35:03.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.359+0000 7f10ba978640 1 -- 172.21.15.88:0/3626362914 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f10a006c2b0 msgr2=0x7f10a006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:03.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.359+0000 7f10ba978640 1 --2- 172.21.15.88:0/3626362914 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f10a006c2b0 0x7f10a006e770 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f10a400aa30 tx=0x7f10a4008040 comp rx=0 tx=0).stop 2024-09-16T01:35:03.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.359+0000 7f10ba978640 1 -- 172.21.15.88:0/3626362914 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10b4158a00 msgr2=0x7f10b4150810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:03.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.359+0000 7f10ba978640 1 --2- 172.21.15.88:0/3626362914 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10b4158a00 0x7f10b4150810 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f10a8000c00 tx=0x7f10a80046c0 comp rx=0 tx=0).stop 2024-09-16T01:35:03.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.359+0000 7f10ba978640 1 -- 172.21.15.88:0/3626362914 shutdown_connections 2024-09-16T01:35:03.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.359+0000 7f10ba978640 1 --2- 172.21.15.88:0/3626362914 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f10a006c2b0 0x7f10a006e770 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:03.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.359+0000 7f10ba978640 1 --2- 172.21.15.88:0/3626362914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10b4158de0 0x7f10b4154a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:03.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.359+0000 7f10ba978640 1 --2- 172.21.15.88:0/3626362914 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10b4158a00 0x7f10b4150810 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:03.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.359+0000 7f10ba978640 1 --2- 172.21.15.88:0/3626362914 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10b4072470 0x7f10b414b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:03.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.359+0000 7f10ba978640 1 -- 172.21.15.88:0/3626362914 >> 172.21.15.88:0/3626362914 conn(0x7f10b406cb50 msgr2=0x7f10b40b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:03.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.359+0000 7f10ba978640 1 -- 172.21.15.88:0/3626362914 shutdown_connections 2024-09-16T01:35:03.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.359+0000 7f10ba978640 1 -- 172.21.15.88:0/3626362914 wait complete. 2024-09-16T01:35:03.372 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-16T01:35:03.373 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:35:03.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.472+0000 7f72d44d7640 1 Processor -- start 2024-09-16T01:35:03.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.472+0000 7f72d44d7640 1 -- start start 2024-09-16T01:35:03.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.473+0000 7f72d44d7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f72cc15e660 0x7f72cc0cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:03.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.473+0000 7f72d44d7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72cc0d3310 0x7f72cc0d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:03.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.473+0000 7f72d44d7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72cc15ea40 con 0x7f72cc0d1650 2024-09-16T01:35:03.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.473+0000 7f72d44d7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f72cc0d1ac0 con 0x7f72cc15e660 2024-09-16T01:35:03.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.473+0000 7f72d44d7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f72cc0d3870 con 0x7f72cc0d3310 2024-09-16T01:35:03.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.473+0000 7f72d2273640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f72cc15e660 0x7f72cc0cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:03.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.473+0000 7f72d2a74640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f72cc0d1650 0x7f72cc06bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35966/0 (socket says 172.21.15.88:35966) 2024-09-16T01:35:03.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.473+0000 7f72d2a74640 1 -- 172.21.15.88:0/168853032 learned_addr learned my addr 172.21.15.88:0/168853032 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:03.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.474+0000 7f72d2273640 1 -- 172.21.15.88:0/168853032 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72cc0d3310 msgr2=0x7f72cc0d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:03.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.474+0000 7f72d2273640 1 --2- 172.21.15.88:0/168853032 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72cc0d3310 0x7f72cc0d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:03.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.474+0000 7f72d2273640 1 -- 172.21.15.88:0/168853032 >> v1:172.21.15.88:6789/0 conn(0x7f72cc0d1650 legacy=0x7f72cc06bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:03.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.474+0000 7f72d2273640 1 -- 172.21.15.88:0/168853032 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72cc0d3b50 con 0x7f72cc15e660 2024-09-16T01:35:03.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.474+0000 7f72d1271640 1 -- 172.21.15.88:0/168853032 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4036265098 0 0) 0x7f72cc15ea40 con 0x7f72cc0d1650 2024-09-16T01:35:03.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.474+0000 7f72d2273640 1 --2- 172.21.15.88:0/168853032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72cc15e660 0x7f72cc0cf730 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f72bc0097a0 tx=0x7f72bc02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e2e0f1b0b0c3388 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:03.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.475+0000 7f72d1271640 1 -- 172.21.15.88:0/168853032 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc037070 con 0x7f72cc15e660 2024-09-16T01:35:03.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.475+0000 7f72d1271640 1 -- 172.21.15.88:0/168853032 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72bc032440 con 0x7f72cc15e660 2024-09-16T01:35:03.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.475+0000 7f72d1271640 1 -- 172.21.15.88:0/168853032 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc032810 con 0x7f72cc15e660 2024-09-16T01:35:03.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.475+0000 7f72d44d7640 1 -- 172.21.15.88:0/168853032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72cc15e660 msgr2=0x7f72cc0cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:03.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.475+0000 7f72d44d7640 1 --2- 172.21.15.88:0/168853032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72cc15e660 0x7f72cc0cf730 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f72bc0097a0 tx=0x7f72bc02a180 comp rx=0 tx=0).stop 2024-09-16T01:35:03.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.475+0000 7f72d44d7640 1 -- 172.21.15.88:0/168853032 shutdown_connections 2024-09-16T01:35:03.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.475+0000 7f72d44d7640 1 --2- 172.21.15.88:0/168853032 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72cc0d3310 0x7f72cc0d1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:03.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.475+0000 7f72d44d7640 1 --2- 172.21.15.88:0/168853032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72cc15e660 0x7f72cc0cf730 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:03.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.475+0000 7f72d44d7640 1 -- 172.21.15.88:0/168853032 >> 172.21.15.88:0/168853032 conn(0x7f72cc0b6410 msgr2=0x7f72cc0b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:03.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.475+0000 7f72d44d7640 1 -- 172.21.15.88:0/168853032 shutdown_connections 2024-09-16T01:35:03.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.475+0000 7f72d44d7640 1 -- 172.21.15.88:0/168853032 wait complete. 2024-09-16T01:35:03.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.476+0000 7f72d44d7640 1 Processor -- start 2024-09-16T01:35:03.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.476+0000 7f72d44d7640 1 -- start start 2024-09-16T01:35:03.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.477+0000 7f72d44d7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72cc0d1650 0x7f72cc158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:03.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.477+0000 7f72d44d7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72cc0d3310 0x7f72cc155d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:03.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.477+0000 7f72d44d7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72cc15e660 0x7f72cc144b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:03.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.477+0000 7f72d44d7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc0d43f0 con 0x7f72cc15e660 2024-09-16T01:35:03.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.477+0000 7f72d44d7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc0d4110 con 0x7f72cc0d3310 2024-09-16T01:35:03.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.477+0000 7f72d44d7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f72cc0cfc70 con 0x7f72cc0d1650 2024-09-16T01:35:03.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.477+0000 7f72d2a74640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72cc15e660 0x7f72cc144b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:03.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.477+0000 7f72d2a74640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72cc15e660 0x7f72cc144b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36434/0 (socket says 172.21.15.88:36434) 2024-09-16T01:35:03.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.477+0000 7f72d2a74640 1 -- 172.21.15.88:0/2393019036 learned_addr learned my addr 172.21.15.88:0/2393019036 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:03.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.477+0000 7f72d2273640 1 --2- 172.21.15.88:0/2393019036 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72cc0d1650 0x7f72cc158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:03.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.477+0000 7f72d1a72640 1 --2- 172.21.15.88:0/2393019036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72cc0d3310 0x7f72cc155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:03.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.478+0000 7f72d2a74640 1 -- 172.21.15.88:0/2393019036 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72cc0d1650 msgr2=0x7f72cc158ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:03.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.478+0000 7f72d2a74640 1 --2- 172.21.15.88:0/2393019036 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72cc0d1650 0x7f72cc158ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:03.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.478+0000 7f72d2a74640 1 -- 172.21.15.88:0/2393019036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72cc0d3310 msgr2=0x7f72cc155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:03.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.478+0000 7f72d2a74640 1 --2- 172.21.15.88:0/2393019036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72cc0d3310 0x7f72cc155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:03.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.478+0000 7f72d2a74640 1 -- 172.21.15.88:0/2393019036 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72cc145880 con 0x7f72cc15e660 2024-09-16T01:35:03.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.478+0000 7f72d2a74640 1 --2- 172.21.15.88:0/2393019036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72cc15e660 0x7f72cc144b50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f72c400c7a0 tx=0x7f72c400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:03.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.478+0000 7f72caffd640 1 -- 172.21.15.88:0/2393019036 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c4018070 con 0x7f72cc15e660 2024-09-16T01:35:03.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.478+0000 7f72caffd640 1 -- 172.21.15.88:0/2393019036 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72c400a640 con 0x7f72cc15e660 2024-09-16T01:35:03.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.478+0000 7f72d44d7640 1 -- 172.21.15.88:0/2393019036 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72cc157cf0 con 0x7f72cc15e660 2024-09-16T01:35:03.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.478+0000 7f72caffd640 1 -- 172.21.15.88:0/2393019036 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c400aa70 con 0x7f72cc15e660 2024-09-16T01:35:03.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.478+0000 7f72d44d7640 1 -- 172.21.15.88:0/2393019036 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72cc1582b0 con 0x7f72cc15e660 2024-09-16T01:35:03.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.479+0000 7f72d44d7640 1 -- 172.21.15.88:0/2393019036 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72cc0d1110 con 0x7f72cc15e660 2024-09-16T01:35:03.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.479+0000 7f72caffd640 1 -- 172.21.15.88:0/2393019036 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f72c401b070 con 0x7f72cc15e660 2024-09-16T01:35:03.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.480+0000 7f72caffd640 1 --2- 172.21.15.88:0/2393019036 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f72b806c2b0 0x7f72b806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:03.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.480+0000 7f72caffd640 1 -- 172.21.15.88:0/2393019036 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f72c408aa20 con 0x7f72cc15e660 2024-09-16T01:35:03.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.480+0000 7f72d2273640 1 --2- 172.21.15.88:0/2393019036 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f72b806c2b0 0x7f72b806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:03.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.480+0000 7f72d2273640 1 --2- 172.21.15.88:0/2393019036 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f72b806c2b0 0x7f72b806e770 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f72bc034000 tx=0x7f72bc02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:03.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.484+0000 7f72caffd640 1 -- 172.21.15.88:0/2393019036 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f72c4059190 con 0x7f72cc15e660 2024-09-16T01:35:03.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.684+0000 7f72d44d7640 1 -- 172.21.15.88:0/2393019036 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f72cc15c8a0 con 0x7f72cc15e660 2024-09-16T01:35:03.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.684+0000 7f72caffd640 1 -- 172.21.15.88:0/2393019036 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+5229 (secure 0 0 0) 0x7f72c4058a90 con 0x7f72cc15e660 2024-09-16T01:35:03.686 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:35:03.686 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":13,"btime":"2024-09-16T01:35:03:353018+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1005331674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1005331674},{"type":"v1","addr":"172.21.15.88:6837","nonce":1005331674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/3427247662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3427247662},{"type":"v1","addr":"172.21.15.154:6834","nonce":3427247662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/4059835337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":4059835337},{"type":"v1","addr":"172.21.15.154:6835","nonce":4059835337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:35:00.674725+0000","modified":"2024-09-16T01:35:02.681832+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4830":{"gid":4830,"name":"c","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6835/4230443772","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4230443772},{"type":"v1","addr":"172.21.15.88:6835","nonce":4230443772}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4830,"qdb_cluster":[4830]},"id":2}]} 2024-09-16T01:35:03.686 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 13 2024-09-16T01:35:03.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.686+0000 7f72d44d7640 1 -- 172.21.15.88:0/2393019036 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f72b806c2b0 msgr2=0x7f72b806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:03.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.686+0000 7f72d44d7640 1 --2- 172.21.15.88:0/2393019036 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f72b806c2b0 0x7f72b806e770 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f72bc034000 tx=0x7f72bc02e000 comp rx=0 tx=0).stop 2024-09-16T01:35:03.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.686+0000 7f72d44d7640 1 -- 172.21.15.88:0/2393019036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72cc15e660 msgr2=0x7f72cc144b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:03.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.686+0000 7f72d44d7640 1 --2- 172.21.15.88:0/2393019036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72cc15e660 0x7f72cc144b50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f72c400c7a0 tx=0x7f72c400cc60 comp rx=0 tx=0).stop 2024-09-16T01:35:03.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.686+0000 7f72d44d7640 1 -- 172.21.15.88:0/2393019036 shutdown_connections 2024-09-16T01:35:03.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.686+0000 7f72d44d7640 1 --2- 172.21.15.88:0/2393019036 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f72b806c2b0 0x7f72b806e770 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:03.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.686+0000 7f72d44d7640 1 --2- 172.21.15.88:0/2393019036 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72cc15e660 0x7f72cc144b50 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:03.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.686+0000 7f72d44d7640 1 --2- 172.21.15.88:0/2393019036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72cc0d3310 0x7f72cc155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:03.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.686+0000 7f72d44d7640 1 --2- 172.21.15.88:0/2393019036 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72cc0d1650 0x7f72cc158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:03.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.686+0000 7f72d44d7640 1 -- 172.21.15.88:0/2393019036 >> 172.21.15.88:0/2393019036 conn(0x7f72cc0b6410 msgr2=0x7f72cc15b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:03.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.687+0000 7f72d44d7640 1 -- 172.21.15.88:0/2393019036 shutdown_connections 2024-09-16T01:35:03.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.687+0000 7f72d44d7640 1 -- 172.21.15.88:0/2393019036 wait complete. 2024-09-16T01:35:03.699 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:35:03.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.795+0000 7f083dc8c640 1 Processor -- start 2024-09-16T01:35:03.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.795+0000 7f083dc8c640 1 -- start start 2024-09-16T01:35:03.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.796+0000 7f083dc8c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08380b5be0 0x7f083815ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:03.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.796+0000 7f083dc8c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f08380b29a0 0x7f08380cde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:03.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.796+0000 7f083dc8c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08380b6830 con 0x7f083815a9c0 2024-09-16T01:35:03.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.796+0000 7f083dc8c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f083815f410 con 0x7f08380b29a0 2024-09-16T01:35:03.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.796+0000 7f083dc8c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f08380cd0b0 con 0x7f08380b5be0 2024-09-16T01:35:03.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.796+0000 7f0837fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f08380b29a0 0x7f08380cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:03.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.796+0000 7f0836ffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08380b5be0 0x7f083815ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:03.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.796+0000 7f0836ffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08380b5be0 0x7f083815ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:44508/0 (socket says 172.21.15.88:44508) 2024-09-16T01:35:03.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.796+0000 7f0837fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f08380b29a0 0x7f08380cde70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:52706/0 (socket says 172.21.15.88:52706) 2024-09-16T01:35:03.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.796+0000 7f0836ffd640 1 -- 172.21.15.88:0/378604727 learned_addr learned my addr 172.21.15.88:0/378604727 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:03.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.797+0000 7f0836ffd640 1 -- 172.21.15.88:0/378604727 >> v2:172.21.15.154:3300/0 conn(0x7f08380b29a0 msgr2=0x7f08380cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:03.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.797+0000 7f0836ffd640 1 --2- 172.21.15.88:0/378604727 >> v2:172.21.15.154:3300/0 conn(0x7f08380b29a0 0x7f08380cde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:03.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.797+0000 7f0836ffd640 1 -- 172.21.15.88:0/378604727 >> v1:172.21.15.154:6789/0 conn(0x7f083815a9c0 legacy=0x7f083815e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:03.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.797+0000 7f0836ffd640 1 -- 172.21.15.88:0/378604727 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08380cd390 con 0x7f08380b5be0 2024-09-16T01:35:03.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.797+0000 7f0837fff640 1 --2- 172.21.15.88:0/378604727 >> v2:172.21.15.154:3300/0 conn(0x7f08380b29a0 0x7f08380cde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:35:03.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.797+0000 7f0836ffd640 1 --2- 172.21.15.88:0/378604727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08380b5be0 0x7f083815ed10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f082c009870 tx=0x7f082c02a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a2a710ac81fb7dc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:03.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.798+0000 7f08367fc640 1 -- 172.21.15.88:0/378604727 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082c037070 con 0x7f08380b5be0 2024-09-16T01:35:03.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.798+0000 7f08367fc640 1 -- 172.21.15.88:0/378604727 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f082c032440 con 0x7f08380b5be0 2024-09-16T01:35:03.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.798+0000 7f08367fc640 1 -- 172.21.15.88:0/378604727 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f082c032810 con 0x7f08380b5be0 2024-09-16T01:35:03.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.798+0000 7f083dc8c640 1 -- 172.21.15.88:0/378604727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08380b5be0 msgr2=0x7f083815ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:03.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.798+0000 7f083dc8c640 1 --2- 172.21.15.88:0/378604727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08380b5be0 0x7f083815ed10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f082c009870 tx=0x7f082c02a270 comp rx=0 tx=0).stop 2024-09-16T01:35:03.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.798+0000 7f083dc8c640 1 -- 172.21.15.88:0/378604727 shutdown_connections 2024-09-16T01:35:03.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.798+0000 7f083dc8c640 1 --2- 172.21.15.88:0/378604727 >> v2:172.21.15.154:3300/0 conn(0x7f08380b29a0 0x7f08380cde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:03.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.798+0000 7f083dc8c640 1 --2- 172.21.15.88:0/378604727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08380b5be0 0x7f083815ed10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:03.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.798+0000 7f083dc8c640 1 -- 172.21.15.88:0/378604727 >> 172.21.15.88:0/378604727 conn(0x7f083806bc10 msgr2=0x7f083806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:03.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.798+0000 7f083dc8c640 1 -- 172.21.15.88:0/378604727 shutdown_connections 2024-09-16T01:35:03.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.798+0000 7f083dc8c640 1 -- 172.21.15.88:0/378604727 wait complete. 2024-09-16T01:35:03.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.799+0000 7f083dc8c640 1 Processor -- start 2024-09-16T01:35:03.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.800+0000 7f083dc8c640 1 -- start start 2024-09-16T01:35:03.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.800+0000 7f083dc8c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08380b29a0 0x7f0838149310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:03.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.800+0000 7f083dc8c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08380b5be0 0x7f08381486a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:03.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.800+0000 7f083dc8c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f083815a9c0 0x7f0838156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:03.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.800+0000 7f083dc8c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f083815ada0 con 0x7f08380b29a0 2024-09-16T01:35:03.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.800+0000 7f083dc8c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f08380cd950 con 0x7f083815a9c0 2024-09-16T01:35:03.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.800+0000 7f083dc8c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f083806de40 con 0x7f08380b5be0 2024-09-16T01:35:03.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.800+0000 7f08377fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08380b29a0 0x7f0838149310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:03.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.801+0000 7f08377fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08380b29a0 0x7f0838149310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36448/0 (socket says 172.21.15.88:36448) 2024-09-16T01:35:03.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.801+0000 7f08377fe640 1 -- 172.21.15.88:0/2091202998 learned_addr learned my addr 172.21.15.88:0/2091202998 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:03.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.801+0000 7f0836ffd640 1 --2- 172.21.15.88:0/2091202998 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08380b5be0 0x7f08381486a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:03.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.801+0000 7f0837fff640 1 --2- 172.21.15.88:0/2091202998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f083815a9c0 0x7f0838156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:03.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.801+0000 7f08377fe640 1 -- 172.21.15.88:0/2091202998 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08380b5be0 msgr2=0x7f08381486a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:03.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.801+0000 7f08377fe640 1 --2- 172.21.15.88:0/2091202998 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08380b5be0 0x7f08381486a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:03.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.801+0000 7f08377fe640 1 -- 172.21.15.88:0/2091202998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f083815a9c0 msgr2=0x7f0838156020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:03.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.801+0000 7f08377fe640 1 --2- 172.21.15.88:0/2091202998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f083815a9c0 0x7f0838156020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:03.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.801+0000 7f08377fe640 1 -- 172.21.15.88:0/2091202998 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0838148be0 con 0x7f08380b29a0 2024-09-16T01:35:03.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.801+0000 7f0837fff640 1 --2- 172.21.15.88:0/2091202998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f083815a9c0 0x7f0838156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:03.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.801+0000 7f08377fe640 1 --2- 172.21.15.88:0/2091202998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08380b29a0 0x7f0838149310 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f082800c730 tx=0x7f082800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:03.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.801+0000 7f081ffff640 1 -- 172.21.15.88:0/2091202998 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0828016020 con 0x7f08380b29a0 2024-09-16T01:35:03.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.801+0000 7f081ffff640 1 -- 172.21.15.88:0/2091202998 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0828006d50 con 0x7f08380b29a0 2024-09-16T01:35:03.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.801+0000 7f083dc8c640 1 -- 172.21.15.88:0/2091202998 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0838153790 con 0x7f08380b29a0 2024-09-16T01:35:03.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.801+0000 7f081ffff640 1 -- 172.21.15.88:0/2091202998 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0828003bb0 con 0x7f08380b29a0 2024-09-16T01:35:03.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.801+0000 7f083dc8c640 1 -- 172.21.15.88:0/2091202998 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f083815afc0 con 0x7f08380b29a0 2024-09-16T01:35:03.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.802+0000 7f083dc8c640 1 -- 172.21.15.88:0/2091202998 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f083806e1f0 con 0x7f08380b29a0 2024-09-16T01:35:03.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.803+0000 7f081ffff640 1 -- 172.21.15.88:0/2091202998 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f082800a850 con 0x7f08380b29a0 2024-09-16T01:35:03.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.803+0000 7f081ffff640 1 --2- 172.21.15.88:0/2091202998 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f082406c2b0 0x7f082406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:03.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.803+0000 7f081ffff640 1 -- 172.21.15.88:0/2091202998 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f082808af80 con 0x7f08380b29a0 2024-09-16T01:35:03.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.803+0000 7f0836ffd640 1 --2- 172.21.15.88:0/2091202998 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f082406c2b0 0x7f082406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:03.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.804+0000 7f0836ffd640 1 --2- 172.21.15.88:0/2091202998 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f082406c2b0 0x7f082406e770 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f082c034000 tx=0x7f082c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:03.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.807+0000 7f081ffff640 1 -- 172.21.15.88:0/2091202998 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0828020020 con 0x7f08380b29a0 2024-09-16T01:35:03.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.975+0000 7f083dc8c640 1 -- 172.21.15.88:0/2091202998 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0838072a20 con 0x7f08380b29a0 2024-09-16T01:35:03.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.977+0000 7f081ffff640 1 -- 172.21.15.88:0/2091202998 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) ==== 74+0+16705 (secure 0 0 0) 0x7f0828059780 con 0x7f08380b29a0 2024-09-16T01:35:03.978 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:35:03.978 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":32,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:35:02.342361+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:34:58.555164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-16T01:34:59.655760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:35:03.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.978+0000 7f083dc8c640 1 -- 172.21.15.88:0/2091202998 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f082406c2b0 msgr2=0x7f082406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:03.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.978+0000 7f083dc8c640 1 --2- 172.21.15.88:0/2091202998 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f082406c2b0 0x7f082406e770 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f082c034000 tx=0x7f082c02e000 comp rx=0 tx=0).stop 2024-09-16T01:35:03.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.978+0000 7f083dc8c640 1 -- 172.21.15.88:0/2091202998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08380b29a0 msgr2=0x7f0838149310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:03.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.978+0000 7f083dc8c640 1 --2- 172.21.15.88:0/2091202998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08380b29a0 0x7f0838149310 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f082800c730 tx=0x7f082800cbf0 comp rx=0 tx=0).stop 2024-09-16T01:35:03.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.978+0000 7f083dc8c640 1 -- 172.21.15.88:0/2091202998 shutdown_connections 2024-09-16T01:35:03.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.978+0000 7f083dc8c640 1 --2- 172.21.15.88:0/2091202998 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f082406c2b0 0x7f082406e770 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:03.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.978+0000 7f083dc8c640 1 --2- 172.21.15.88:0/2091202998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f083815a9c0 0x7f0838156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:03.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.978+0000 7f083dc8c640 1 --2- 172.21.15.88:0/2091202998 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08380b5be0 0x7f08381486a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:03.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.979+0000 7f083dc8c640 1 --2- 172.21.15.88:0/2091202998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08380b29a0 0x7f0838149310 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:03.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.979+0000 7f083dc8c640 1 -- 172.21.15.88:0/2091202998 >> 172.21.15.88:0/2091202998 conn(0x7f083806bc10 msgr2=0x7f08380cdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:03.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.979+0000 7f083dc8c640 1 -- 172.21.15.88:0/2091202998 shutdown_connections 2024-09-16T01:35:03.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:03.979+0000 7f083dc8c640 1 -- 172.21.15.88:0/2091202998 wait complete. 2024-09-16T01:35:03.991 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T01:35:03.991 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:35:04.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.088+0000 7fae1aa65640 1 Processor -- start 2024-09-16T01:35:04.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.088+0000 7fae1aa65640 1 -- start start 2024-09-16T01:35:04.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.088+0000 7fae1aa65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae140b6140 0x7fae1415c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:04.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.089+0000 7fae1aa65640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae140b50a0 con 0x7fae140cfc80 2024-09-16T01:35:04.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.089+0000 7fae1aa65640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae140b5380 con 0x7fae14155800 2024-09-16T01:35:04.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.089+0000 7fae1aa65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fae140b6550 con 0x7fae140b6140 2024-09-16T01:35:04.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.089+0000 7fae13fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fae140cfc80 0x7fae1415c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35974/0 (socket says 172.21.15.88:35974) 2024-09-16T01:35:04.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.089+0000 7fae13fff640 1 -- 172.21.15.88:0/3924512526 learned_addr learned my addr 172.21.15.88:0/3924512526 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:04.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.089+0000 7fae19002640 1 --2- 172.21.15.88:0/3924512526 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae140b6140 0x7fae1415c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:04.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.089+0000 7fae137fe640 1 -- 172.21.15.88:0/3924512526 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 458048301 0 0) 0x7fae140b50a0 con 0x7fae140cfc80 2024-09-16T01:35:04.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.089+0000 7fae137fe640 1 -- 172.21.15.88:0/3924512526 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae04003520 con 0x7fae140cfc80 2024-09-16T01:35:04.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.089+0000 7fae137fe640 1 -- 172.21.15.88:0/3924512526 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2392949528 0 0) 0x7fae140b5380 con 0x7fae14155800 2024-09-16T01:35:04.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.089+0000 7fae137fe640 1 -- 172.21.15.88:0/3924512526 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae140b50a0 con 0x7fae14155800 2024-09-16T01:35:04.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.089+0000 7fae19002640 1 -- 172.21.15.88:0/3924512526 >> v1:172.21.15.154:6789/0 conn(0x7fae14155800 legacy=0x7fae140cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:04.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.089+0000 7fae19002640 1 -- 172.21.15.88:0/3924512526 >> v1:172.21.15.88:6789/0 conn(0x7fae140cfc80 legacy=0x7fae1415c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:04.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.089+0000 7fae19002640 1 -- 172.21.15.88:0/3924512526 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae140b5660 con 0x7fae140b6140 2024-09-16T01:35:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.090+0000 7fae19002640 1 --2- 172.21.15.88:0/3924512526 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae140b6140 0x7fae1415c330 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fae0c009dc0 tx=0x7fae0c02a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4477d1b31b20290 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.090+0000 7fae137fe640 1 -- 172.21.15.88:0/3924512526 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0c007740 con 0x7fae140b6140 2024-09-16T01:35:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.090+0000 7fae137fe640 1 -- 172.21.15.88:0/3924512526 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae0c0078e0 con 0x7fae140b6140 2024-09-16T01:35:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.090+0000 7fae137fe640 1 -- 172.21.15.88:0/3924512526 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae0c007cb0 con 0x7fae140b6140 2024-09-16T01:35:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.091+0000 7fae1aa65640 1 -- 172.21.15.88:0/3924512526 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae140b6140 msgr2=0x7fae1415c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.091+0000 7fae1aa65640 1 --2- 172.21.15.88:0/3924512526 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae140b6140 0x7fae1415c330 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fae0c009dc0 tx=0x7fae0c02a6d0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.091+0000 7fae1aa65640 1 -- 172.21.15.88:0/3924512526 shutdown_connections 2024-09-16T01:35:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.091+0000 7fae1aa65640 1 --2- 172.21.15.88:0/3924512526 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae140b6140 0x7fae1415c330 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.091+0000 7fae1aa65640 1 -- 172.21.15.88:0/3924512526 >> 172.21.15.88:0/3924512526 conn(0x7fae1406bc10 msgr2=0x7fae1406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.091+0000 7fae1aa65640 1 -- 172.21.15.88:0/3924512526 shutdown_connections 2024-09-16T01:35:04.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.091+0000 7fae1aa65640 1 -- 172.21.15.88:0/3924512526 wait complete. 2024-09-16T01:35:04.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.092+0000 7fae1aa65640 1 Processor -- start 2024-09-16T01:35:04.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.092+0000 7fae1aa65640 1 -- start start 2024-09-16T01:35:04.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.093+0000 7fae1aa65640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae140b6140 0x7fae14155d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:04.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.093+0000 7fae1aa65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae140cfc80 0x7fae141f4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:04.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.093+0000 7fae1aa65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae14155800 0x7fae14151450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:04.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.093+0000 7fae1aa65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fae14155be0 con 0x7fae140cfc80 2024-09-16T01:35:04.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.093+0000 7fae1aa65640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fae140b5c20 con 0x7fae140b6140 2024-09-16T01:35:04.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.093+0000 7fae1aa65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fae140b4920 con 0x7fae14155800 2024-09-16T01:35:04.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.093+0000 7fae13fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae140cfc80 0x7fae141f4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:04.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.093+0000 7fae13fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae140cfc80 0x7fae141f4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36454/0 (socket says 172.21.15.88:36454) 2024-09-16T01:35:04.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.093+0000 7fae13fff640 1 -- 172.21.15.88:0/3386557151 learned_addr learned my addr 172.21.15.88:0/3386557151 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:04.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.093+0000 7fae19002640 1 --2- 172.21.15.88:0/3386557151 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae14155800 0x7fae14151450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:04.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.093+0000 7fae18801640 1 --2- 172.21.15.88:0/3386557151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae140b6140 0x7fae14155d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:04.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.093+0000 7fae13fff640 1 -- 172.21.15.88:0/3386557151 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae14155800 msgr2=0x7fae14151450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:04.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.093+0000 7fae13fff640 1 --2- 172.21.15.88:0/3386557151 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae14155800 0x7fae14151450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.093+0000 7fae13fff640 1 -- 172.21.15.88:0/3386557151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae140b6140 msgr2=0x7fae14155d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:04.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.093+0000 7fae13fff640 1 --2- 172.21.15.88:0/3386557151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae140b6140 0x7fae14155d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.093+0000 7fae13fff640 1 -- 172.21.15.88:0/3386557151 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae141563f0 con 0x7fae140cfc80 2024-09-16T01:35:04.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.094+0000 7fae13fff640 1 --2- 172.21.15.88:0/3386557151 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae140cfc80 0x7fae141f4cc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fae0000c4d0 tx=0x7fae0000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:04.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.094+0000 7fae117fa640 1 -- 172.21.15.88:0/3386557151 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae00017070 con 0x7fae140cfc80 2024-09-16T01:35:04.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.094+0000 7fae117fa640 1 -- 172.21.15.88:0/3386557151 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae00005ce0 con 0x7fae140cfc80 2024-09-16T01:35:04.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.094+0000 7fae1aa65640 1 -- 172.21.15.88:0/3386557151 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae141566d0 con 0x7fae140cfc80 2024-09-16T01:35:04.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.094+0000 7fae117fa640 1 -- 172.21.15.88:0/3386557151 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae00007270 con 0x7fae140cfc80 2024-09-16T01:35:04.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.094+0000 7fae1aa65640 1 -- 172.21.15.88:0/3386557151 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae14156c90 con 0x7fae140cfc80 2024-09-16T01:35:04.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.095+0000 7fae1aa65640 1 -- 172.21.15.88:0/3386557151 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae140b4d00 con 0x7fae140cfc80 2024-09-16T01:35:04.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.095+0000 7fae117fa640 1 -- 172.21.15.88:0/3386557151 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fae0001a070 con 0x7fae140cfc80 2024-09-16T01:35:04.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.096+0000 7fae117fa640 1 --2- 172.21.15.88:0/3386557151 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fadf806c2b0 0x7fadf806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:04.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.096+0000 7fae117fa640 1 -- 172.21.15.88:0/3386557151 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fae0008a680 con 0x7fae140cfc80 2024-09-16T01:35:04.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.096+0000 7fae18801640 1 --2- 172.21.15.88:0/3386557151 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fadf806c2b0 0x7fadf806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:04.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.096+0000 7fae18801640 1 --2- 172.21.15.88:0/3386557151 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fadf806c2b0 0x7fadf806e770 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fae08007fd0 tx=0x7fae08009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:04.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.100+0000 7fae117fa640 1 -- 172.21.15.88:0/3386557151 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fae000587b0 con 0x7fae140cfc80 2024-09-16T01:35:04.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.263+0000 7fae1aa65640 1 -- 172.21.15.88:0/3386557151 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fae14151990 con 0x7fadf806c2b0 2024-09-16T01:35:04.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.269+0000 7fae117fa640 1 -- 172.21.15.88:0/3386557151 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+247449 (secure 0 0 0) 0x7fae14151990 con 0x7fadf806c2b0 2024-09-16T01:35:04.273 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:35:04.274 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:35:04.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.272+0000 7fae1aa65640 1 -- 172.21.15.88:0/3386557151 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fadf806c2b0 msgr2=0x7fadf806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:04.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.272+0000 7fae1aa65640 1 --2- 172.21.15.88:0/3386557151 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fadf806c2b0 0x7fadf806e770 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fae08007fd0 tx=0x7fae08009040 comp rx=0 tx=0).stop 2024-09-16T01:35:04.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.272+0000 7fae1aa65640 1 -- 172.21.15.88:0/3386557151 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae140cfc80 msgr2=0x7fae141f4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:04.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.272+0000 7fae1aa65640 1 --2- 172.21.15.88:0/3386557151 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae140cfc80 0x7fae141f4cc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fae0000c4d0 tx=0x7fae0000c990 comp rx=0 tx=0).stop 2024-09-16T01:35:04.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.273+0000 7fae1aa65640 1 -- 172.21.15.88:0/3386557151 shutdown_connections 2024-09-16T01:35:04.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.273+0000 7fae1aa65640 1 --2- 172.21.15.88:0/3386557151 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fadf806c2b0 0x7fadf806e770 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.273+0000 7fae1aa65640 1 --2- 172.21.15.88:0/3386557151 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae14155800 0x7fae14151450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.273+0000 7fae1aa65640 1 --2- 172.21.15.88:0/3386557151 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae140cfc80 0x7fae141f4cc0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.273+0000 7fae1aa65640 1 --2- 172.21.15.88:0/3386557151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae140b6140 0x7fae14155d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.273+0000 7fae1aa65640 1 -- 172.21.15.88:0/3386557151 >> 172.21.15.88:0/3386557151 conn(0x7fae1406bc10 msgr2=0x7fae14071570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:04.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.273+0000 7fae1aa65640 1 -- 172.21.15.88:0/3386557151 shutdown_connections 2024-09-16T01:35:04.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.273+0000 7fae1aa65640 1 -- 172.21.15.88:0/3386557151 wait complete. 2024-09-16T01:35:04.285 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":51,"stamp":"2024-09-16T01:35:02.524194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590458,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":163,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":28,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219912,"kb_used_data":3680,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754808,"statfs":{"total":773094113280,"available":772868923392,"internally_reserved":0,"allocated":3768320,"data_stored":2133950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000516"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337649+0000","last_change":"2024-09-16T01:35:00.337649+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337649+0000","last_undegraded":"2024-09-16T01:35:00.337649+0000","last_fullsized":"2024-09-16T01:35:00.337649+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339909+0000","last_change":"2024-09-16T01:35:00.339909+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339909+0000","last_undegraded":"2024-09-16T01:35:00.339909+0000","last_fullsized":"2024-09-16T01:35:00.339909+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.357429+0000","last_change":"2024-09-16T01:35:01.353507+0000","last_active":"2024-09-16T01:35:01.357429+0000","last_peered":"2024-09-16T01:35:01.357429+0000","last_clean":"2024-09-16T01:35:01.357429+0000","last_became_active":"2024-09-16T01:35:01.352327+0000","last_became_peered":"2024-09-16T01:35:01.352327+0000","last_unstale":"2024-09-16T01:35:01.357429+0000","last_undegraded":"2024-09-16T01:35:01.357429+0000","last_fullsized":"2024-09-16T01:35:01.357429+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:55.644944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.340201+0000","last_change":"2024-09-16T01:35:00.340201+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.340201+0000","last_undegraded":"2024-09-16T01:35:00.340201+0000","last_fullsized":"2024-09-16T01:35:00.340201+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.341590+0000","last_change":"2024-09-16T01:35:00.341590+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.341590+0000","last_undegraded":"2024-09-16T01:35:00.341590+0000","last_fullsized":"2024-09-16T01:35:00.341590+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338751+0000","last_change":"2024-09-16T01:35:00.338751+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338751+0000","last_undegraded":"2024-09-16T01:35:00.338751+0000","last_fullsized":"2024-09-16T01:35:00.338751+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.345760+0000","last_change":"2024-09-16T01:35:00.345760+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.345760+0000","last_undegraded":"2024-09-16T01:35:00.345760+0000","last_fullsized":"2024-09-16T01:35:00.345760+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.347744+0000","last_change":"2024-09-16T01:35:00.347744+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.347744+0000","last_undegraded":"2024-09-16T01:35:00.347744+0000","last_fullsized":"2024-09-16T01:35:00.347744+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337574+0000","last_change":"2024-09-16T01:35:00.337574+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337574+0000","last_undegraded":"2024-09-16T01:35:00.337574+0000","last_fullsized":"2024-09-16T01:35:00.337574+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339626+0000","last_change":"2024-09-16T01:35:00.339626+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339626+0000","last_undegraded":"2024-09-16T01:35:00.339626+0000","last_fullsized":"2024-09-16T01:35:00.339626+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.344537+0000","last_change":"2024-09-16T01:35:00.344537+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.344537+0000","last_undegraded":"2024-09-16T01:35:00.344537+0000","last_fullsized":"2024-09-16T01:35:00.344537+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.354548+0000","last_change":"2024-09-16T01:35:01.349955+0000","last_active":"2024-09-16T01:35:01.354548+0000","last_peered":"2024-09-16T01:35:01.354548+0000","last_clean":"2024-09-16T01:35:01.354548+0000","last_became_active":"2024-09-16T01:35:01.349689+0000","last_became_peered":"2024-09-16T01:35:01.349689+0000","last_unstale":"2024-09-16T01:35:01.354548+0000","last_undegraded":"2024-09-16T01:35:01.354548+0000","last_fullsized":"2024-09-16T01:35:01.354548+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:02:10.872199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.341163+0000","last_change":"2024-09-16T01:35:00.341163+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.341163+0000","last_undegraded":"2024-09-16T01:35:00.341163+0000","last_fullsized":"2024-09-16T01:35:00.341163+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337577+0000","last_change":"2024-09-16T01:35:00.337577+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337577+0000","last_undegraded":"2024-09-16T01:35:00.337577+0000","last_fullsized":"2024-09-16T01:35:00.337577+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.353724+0000","last_change":"2024-09-16T01:35:01.344630+0000","last_active":"2024-09-16T01:35:01.353724+0000","last_peered":"2024-09-16T01:35:01.353724+0000","last_clean":"2024-09-16T01:35:01.353724+0000","last_became_active":"2024-09-16T01:35:01.344340+0000","last_became_peered":"2024-09-16T01:35:01.344340+0000","last_unstale":"2024-09-16T01:35:01.353724+0000","last_undegraded":"2024-09-16T01:35:01.353724+0000","last_fullsized":"2024-09-16T01:35:01.353724+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:06:54.456413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.342533+0000","last_change":"2024-09-16T01:35:00.342533+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.342533+0000","last_undegraded":"2024-09-16T01:35:00.342533+0000","last_fullsized":"2024-09-16T01:35:00.342533+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.343962+0000","last_change":"2024-09-16T01:35:00.343962+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.343962+0000","last_undegraded":"2024-09-16T01:35:00.343962+0000","last_fullsized":"2024-09-16T01:35:00.343962+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339533+0000","last_change":"2024-09-16T01:35:00.339533+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339533+0000","last_undegraded":"2024-09-16T01:35:00.339533+0000","last_fullsized":"2024-09-16T01:35:00.339533+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337206+0000","last_change":"2024-09-16T01:35:00.337206+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337206+0000","last_undegraded":"2024-09-16T01:35:00.337206+0000","last_fullsized":"2024-09-16T01:35:00.337206+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.340512+0000","last_change":"2024-09-16T01:35:00.340512+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.340512+0000","last_undegraded":"2024-09-16T01:35:00.340512+0000","last_fullsized":"2024-09-16T01:35:00.340512+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339909+0000","last_change":"2024-09-16T01:35:00.339909+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339909+0000","last_undegraded":"2024-09-16T01:35:00.339909+0000","last_fullsized":"2024-09-16T01:35:00.339909+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338369+0000","last_change":"2024-09-16T01:35:00.338369+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338369+0000","last_undegraded":"2024-09-16T01:35:00.338369+0000","last_fullsized":"2024-09-16T01:35:00.338369+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.335686+0000","last_change":"2024-09-16T01:35:00.335686+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.335686+0000","last_undegraded":"2024-09-16T01:35:00.335686+0000","last_fullsized":"2024-09-16T01:35:00.335686+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.353084+0000","last_change":"2024-09-16T01:35:01.344447+0000","last_active":"2024-09-16T01:35:01.353084+0000","last_peered":"2024-09-16T01:35:01.353084+0000","last_clean":"2024-09-16T01:35:01.353084+0000","last_became_active":"2024-09-16T01:35:01.344135+0000","last_became_peered":"2024-09-16T01:35:01.344135+0000","last_unstale":"2024-09-16T01:35:01.353084+0000","last_undegraded":"2024-09-16T01:35:01.353084+0000","last_fullsized":"2024-09-16T01:35:01.353084+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:44:42.157718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338640+0000","last_change":"2024-09-16T01:35:00.338640+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338640+0000","last_undegraded":"2024-09-16T01:35:00.338640+0000","last_fullsized":"2024-09-16T01:35:00.338640+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339123+0000","last_change":"2024-09-16T01:35:00.339123+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339123+0000","last_undegraded":"2024-09-16T01:35:00.339123+0000","last_fullsized":"2024-09-16T01:35:00.339123+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337376+0000","last_change":"2024-09-16T01:35:00.337376+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337376+0000","last_undegraded":"2024-09-16T01:35:00.337376+0000","last_fullsized":"2024-09-16T01:35:00.337376+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.352090+0000","last_change":"2024-09-16T01:35:01.343532+0000","last_active":"2024-09-16T01:35:01.352090+0000","last_peered":"2024-09-16T01:35:01.352090+0000","last_clean":"2024-09-16T01:35:01.352090+0000","last_became_active":"2024-09-16T01:35:01.342495+0000","last_became_peered":"2024-09-16T01:35:01.342495+0000","last_unstale":"2024-09-16T01:35:01.352090+0000","last_undegraded":"2024-09-16T01:35:01.352090+0000","last_fullsized":"2024-09-16T01:35:01.352090+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:32:33.652857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339456+0000","last_change":"2024-09-16T01:35:00.339456+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339456+0000","last_undegraded":"2024-09-16T01:35:00.339456+0000","last_fullsized":"2024-09-16T01:35:00.339456+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.343049+0000","last_change":"2024-09-16T01:35:00.343049+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.343049+0000","last_undegraded":"2024-09-16T01:35:00.343049+0000","last_fullsized":"2024-09-16T01:35:00.343049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338353+0000","last_change":"2024-09-16T01:35:00.338353+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338353+0000","last_undegraded":"2024-09-16T01:35:00.338353+0000","last_fullsized":"2024-09-16T01:35:00.338353+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.347252+0000","last_change":"2024-09-16T01:35:00.347252+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.347252+0000","last_undegraded":"2024-09-16T01:35:00.347252+0000","last_fullsized":"2024-09-16T01:35:00.347252+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.335647+0000","last_change":"2024-09-16T01:35:00.335647+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.335647+0000","last_undegraded":"2024-09-16T01:35:00.335647+0000","last_fullsized":"2024-09-16T01:35:00.335647+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.358487+0000","last_change":"2024-09-16T01:35:00.355732+0000","last_active":"2024-09-16T01:35:00.358487+0000","last_peered":"2024-09-16T01:35:00.358487+0000","last_clean":"2024-09-16T01:35:00.358487+0000","last_became_active":"2024-09-16T01:35:00.355431+0000","last_became_peered":"2024-09-16T01:35:00.355431+0000","last_unstale":"2024-09-16T01:35:00.358487+0000","last_undegraded":"2024-09-16T01:35:00.358487+0000","last_fullsized":"2024-09-16T01:35:00.358487+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:48:54.665338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337125+0000","last_change":"2024-09-16T01:35:00.337125+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337125+0000","last_undegraded":"2024-09-16T01:35:00.337125+0000","last_fullsized":"2024-09-16T01:35:00.337125+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352461+0000","last_change":"2024-09-16T01:35:00.348929+0000","last_active":"2024-09-16T01:35:00.352461+0000","last_peered":"2024-09-16T01:35:00.352461+0000","last_clean":"2024-09-16T01:35:00.352461+0000","last_became_active":"2024-09-16T01:35:00.348651+0000","last_became_peered":"2024-09-16T01:35:00.348651+0000","last_unstale":"2024-09-16T01:35:00.352461+0000","last_undegraded":"2024-09-16T01:35:00.352461+0000","last_fullsized":"2024-09-16T01:35:00.352461+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:52:02.746213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339423+0000","last_change":"2024-09-16T01:35:00.339423+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339423+0000","last_undegraded":"2024-09-16T01:35:00.339423+0000","last_fullsized":"2024-09-16T01:35:00.339423+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.346035+0000","last_change":"2024-09-16T01:35:00.344907+0000","last_active":"2024-09-16T01:35:00.346035+0000","last_peered":"2024-09-16T01:35:00.346035+0000","last_clean":"2024-09-16T01:35:00.346035+0000","last_became_active":"2024-09-16T01:35:00.344634+0000","last_became_peered":"2024-09-16T01:35:00.344634+0000","last_unstale":"2024-09-16T01:35:00.346035+0000","last_undegraded":"2024-09-16T01:35:00.346035+0000","last_fullsized":"2024-09-16T01:35:00.346035+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:00.732201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.336922+0000","last_change":"2024-09-16T01:35:00.336922+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.336922+0000","last_undegraded":"2024-09-16T01:35:00.336922+0000","last_fullsized":"2024-09-16T01:35:00.336922+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352455+0000","last_change":"2024-09-16T01:35:00.348009+0000","last_active":"2024-09-16T01:35:00.352455+0000","last_peered":"2024-09-16T01:35:00.352455+0000","last_clean":"2024-09-16T01:35:00.352455+0000","last_became_active":"2024-09-16T01:35:00.347736+0000","last_became_peered":"2024-09-16T01:35:00.347736+0000","last_unstale":"2024-09-16T01:35:00.352455+0000","last_undegraded":"2024-09-16T01:35:00.352455+0000","last_fullsized":"2024-09-16T01:35:00.352455+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:44.087710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.342570+0000","last_change":"2024-09-16T01:35:00.342570+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.342570+0000","last_undegraded":"2024-09-16T01:35:00.342570+0000","last_fullsized":"2024-09-16T01:35:00.342570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353110+0000","last_change":"2024-09-16T01:35:00.348379+0000","last_active":"2024-09-16T01:35:00.353110+0000","last_peered":"2024-09-16T01:35:00.353110+0000","last_clean":"2024-09-16T01:35:00.353110+0000","last_became_active":"2024-09-16T01:35:00.348008+0000","last_became_peered":"2024-09-16T01:35:00.348008+0000","last_unstale":"2024-09-16T01:35:00.353110+0000","last_undegraded":"2024-09-16T01:35:00.353110+0000","last_fullsized":"2024-09-16T01:35:00.353110+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:56:53.528942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.354400+0000","last_change":"2024-09-16T01:35:01.346165+0000","last_active":"2024-09-16T01:35:01.354400+0000","last_peered":"2024-09-16T01:35:01.354400+0000","last_clean":"2024-09-16T01:35:01.354400+0000","last_became_active":"2024-09-16T01:35:01.344479+0000","last_became_peered":"2024-09-16T01:35:01.344479+0000","last_unstale":"2024-09-16T01:35:01.354400+0000","last_undegraded":"2024-09-16T01:35:01.354400+0000","last_fullsized":"2024-09-16T01:35:01.354400+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:58:00.280748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353081+0000","last_change":"2024-09-16T01:35:00.347395+0000","last_active":"2024-09-16T01:35:00.353081+0000","last_peered":"2024-09-16T01:35:00.353081+0000","last_clean":"2024-09-16T01:35:00.353081+0000","last_became_active":"2024-09-16T01:35:00.346752+0000","last_became_peered":"2024-09-16T01:35:00.346752+0000","last_unstale":"2024-09-16T01:35:00.353081+0000","last_undegraded":"2024-09-16T01:35:00.353081+0000","last_fullsized":"2024-09-16T01:35:00.353081+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:51.973054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338296+0000","last_change":"2024-09-16T01:35:00.338296+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338296+0000","last_undegraded":"2024-09-16T01:35:00.338296+0000","last_fullsized":"2024-09-16T01:35:00.338296+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352488+0000","last_change":"2024-09-16T01:35:00.348232+0000","last_active":"2024-09-16T01:35:00.352488+0000","last_peered":"2024-09-16T01:35:00.352488+0000","last_clean":"2024-09-16T01:35:00.352488+0000","last_became_active":"2024-09-16T01:35:00.347976+0000","last_became_peered":"2024-09-16T01:35:00.347976+0000","last_unstale":"2024-09-16T01:35:00.352488+0000","last_undegraded":"2024-09-16T01:35:00.352488+0000","last_fullsized":"2024-09-16T01:35:00.352488+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:49:23.770293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.336753+0000","last_change":"2024-09-16T01:35:00.336753+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.336753+0000","last_undegraded":"2024-09-16T01:35:00.336753+0000","last_fullsized":"2024-09-16T01:35:00.336753+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.347485+0000","last_change":"2024-09-16T01:35:00.342334+0000","last_active":"2024-09-16T01:35:00.347485+0000","last_peered":"2024-09-16T01:35:00.347485+0000","last_clean":"2024-09-16T01:35:00.347485+0000","last_became_active":"2024-09-16T01:35:00.342056+0000","last_became_peered":"2024-09-16T01:35:00.342056+0000","last_unstale":"2024-09-16T01:35:00.347485+0000","last_undegraded":"2024-09-16T01:35:00.347485+0000","last_fullsized":"2024-09-16T01:35:00.347485+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:32:43.356912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.355290+0000","last_change":"2024-09-16T01:35:01.349973+0000","last_active":"2024-09-16T01:35:01.355290+0000","last_peered":"2024-09-16T01:35:01.355290+0000","last_clean":"2024-09-16T01:35:01.355290+0000","last_became_active":"2024-09-16T01:35:01.349733+0000","last_became_peered":"2024-09-16T01:35:01.349733+0000","last_unstale":"2024-09-16T01:35:01.355290+0000","last_undegraded":"2024-09-16T01:35:01.355290+0000","last_fullsized":"2024-09-16T01:35:01.355290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:59:24.976252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352519+0000","last_change":"2024-09-16T01:35:00.349540+0000","last_active":"2024-09-16T01:35:00.352519+0000","last_peered":"2024-09-16T01:35:00.352519+0000","last_clean":"2024-09-16T01:35:00.352519+0000","last_became_active":"2024-09-16T01:35:00.349296+0000","last_became_peered":"2024-09-16T01:35:00.349296+0000","last_unstale":"2024-09-16T01:35:00.352519+0000","last_undegraded":"2024-09-16T01:35:00.352519+0000","last_fullsized":"2024-09-16T01:35:00.352519+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:53:24.648600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338365+0000","last_change":"2024-09-16T01:35:00.338365+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338365+0000","last_undegraded":"2024-09-16T01:35:00.338365+0000","last_fullsized":"2024-09-16T01:35:00.338365+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.349142+0000","last_change":"2024-09-16T01:35:00.346787+0000","last_active":"2024-09-16T01:35:00.349142+0000","last_peered":"2024-09-16T01:35:00.349142+0000","last_clean":"2024-09-16T01:35:00.349142+0000","last_became_active":"2024-09-16T01:35:00.346527+0000","last_became_peered":"2024-09-16T01:35:00.346527+0000","last_unstale":"2024-09-16T01:35:00.349142+0000","last_undegraded":"2024-09-16T01:35:00.349142+0000","last_fullsized":"2024-09-16T01:35:00.349142+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:48:54.665338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338003+0000","last_change":"2024-09-16T01:35:00.338003+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338003+0000","last_undegraded":"2024-09-16T01:35:00.338003+0000","last_fullsized":"2024-09-16T01:35:00.338003+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"31'3","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.362725+0000","last_change":"2024-09-16T01:35:00.352778+0000","last_active":"2024-09-16T01:35:01.362725+0000","last_peered":"2024-09-16T01:35:01.362725+0000","last_clean":"2024-09-16T01:35:01.362725+0000","last_became_active":"2024-09-16T01:35:00.352534+0000","last_became_peered":"2024-09-16T01:35:00.352534+0000","last_unstale":"2024-09-16T01:35:01.362725+0000","last_undegraded":"2024-09-16T01:35:01.362725+0000","last_fullsized":"2024-09-16T01:35:01.362725+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:32:30.768011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.357410+0000","last_change":"2024-09-16T01:35:01.353288+0000","last_active":"2024-09-16T01:35:01.357410+0000","last_peered":"2024-09-16T01:35:01.357410+0000","last_clean":"2024-09-16T01:35:01.357410+0000","last_became_active":"2024-09-16T01:35:01.352428+0000","last_became_peered":"2024-09-16T01:35:01.352428+0000","last_unstale":"2024-09-16T01:35:01.357410+0000","last_undegraded":"2024-09-16T01:35:01.357410+0000","last_fullsized":"2024-09-16T01:35:01.357410+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:48:49.160580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352983+0000","last_change":"2024-09-16T01:35:00.350506+0000","last_active":"2024-09-16T01:35:00.352983+0000","last_peered":"2024-09-16T01:35:00.352983+0000","last_clean":"2024-09-16T01:35:00.352983+0000","last_became_active":"2024-09-16T01:35:00.350215+0000","last_became_peered":"2024-09-16T01:35:00.350215+0000","last_unstale":"2024-09-16T01:35:00.352983+0000","last_undegraded":"2024-09-16T01:35:00.352983+0000","last_fullsized":"2024-09-16T01:35:00.352983+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:43:39.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339530+0000","last_change":"2024-09-16T01:35:00.339530+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339530+0000","last_undegraded":"2024-09-16T01:35:00.339530+0000","last_fullsized":"2024-09-16T01:35:00.339530+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.348029+0000","last_change":"2024-09-16T01:35:00.342977+0000","last_active":"2024-09-16T01:35:00.348029+0000","last_peered":"2024-09-16T01:35:00.348029+0000","last_clean":"2024-09-16T01:35:00.348029+0000","last_became_active":"2024-09-16T01:35:00.342699+0000","last_became_peered":"2024-09-16T01:35:00.342699+0000","last_unstale":"2024-09-16T01:35:00.348029+0000","last_undegraded":"2024-09-16T01:35:00.348029+0000","last_fullsized":"2024-09-16T01:35:00.348029+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:53:54.367989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337148+0000","last_change":"2024-09-16T01:35:00.337148+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337148+0000","last_undegraded":"2024-09-16T01:35:00.337148+0000","last_fullsized":"2024-09-16T01:35:00.337148+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.348456+0000","last_change":"2024-09-16T01:35:00.343581+0000","last_active":"2024-09-16T01:35:00.348456+0000","last_peered":"2024-09-16T01:35:00.348456+0000","last_clean":"2024-09-16T01:35:00.348456+0000","last_became_active":"2024-09-16T01:35:00.343294+0000","last_became_peered":"2024-09-16T01:35:00.343294+0000","last_unstale":"2024-09-16T01:35:00.348456+0000","last_undegraded":"2024-09-16T01:35:00.348456+0000","last_fullsized":"2024-09-16T01:35:00.348456+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:57.654721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.346877+0000","last_change":"2024-09-16T01:35:00.346877+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.346877+0000","last_undegraded":"2024-09-16T01:35:00.346877+0000","last_fullsized":"2024-09-16T01:35:00.346877+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353028+0000","last_change":"2024-09-16T01:35:00.351166+0000","last_active":"2024-09-16T01:35:00.353028+0000","last_peered":"2024-09-16T01:35:00.353028+0000","last_clean":"2024-09-16T01:35:00.353028+0000","last_became_active":"2024-09-16T01:35:00.350881+0000","last_became_peered":"2024-09-16T01:35:00.350881+0000","last_unstale":"2024-09-16T01:35:00.353028+0000","last_undegraded":"2024-09-16T01:35:00.353028+0000","last_fullsized":"2024-09-16T01:35:00.353028+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:54:05.742892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337056+0000","last_change":"2024-09-16T01:35:00.337056+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337056+0000","last_undegraded":"2024-09-16T01:35:00.337056+0000","last_fullsized":"2024-09-16T01:35:00.337056+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.350509+0000","last_change":"2024-09-16T01:35:00.345939+0000","last_active":"2024-09-16T01:35:00.350509+0000","last_peered":"2024-09-16T01:35:00.350509+0000","last_clean":"2024-09-16T01:35:00.350509+0000","last_became_active":"2024-09-16T01:35:00.345653+0000","last_became_peered":"2024-09-16T01:35:00.345653+0000","last_unstale":"2024-09-16T01:35:00.350509+0000","last_undegraded":"2024-09-16T01:35:00.350509+0000","last_fullsized":"2024-09-16T01:35:00.350509+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:57.839735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339311+0000","last_change":"2024-09-16T01:35:00.339311+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339311+0000","last_undegraded":"2024-09-16T01:35:00.339311+0000","last_fullsized":"2024-09-16T01:35:00.339311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352801+0000","last_change":"2024-09-16T01:35:00.350284+0000","last_active":"2024-09-16T01:35:00.352801+0000","last_peered":"2024-09-16T01:35:00.352801+0000","last_clean":"2024-09-16T01:35:00.352801+0000","last_became_active":"2024-09-16T01:35:00.349516+0000","last_became_peered":"2024-09-16T01:35:00.349516+0000","last_unstale":"2024-09-16T01:35:00.352801+0000","last_undegraded":"2024-09-16T01:35:00.352801+0000","last_fullsized":"2024-09-16T01:35:00.352801+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:22:55.631379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.354497+0000","last_change":"2024-09-16T01:35:01.347677+0000","last_active":"2024-09-16T01:35:01.354497+0000","last_peered":"2024-09-16T01:35:01.354497+0000","last_clean":"2024-09-16T01:35:01.354497+0000","last_became_active":"2024-09-16T01:35:01.345167+0000","last_became_peered":"2024-09-16T01:35:01.345167+0000","last_unstale":"2024-09-16T01:35:01.354497+0000","last_undegraded":"2024-09-16T01:35:01.354497+0000","last_fullsized":"2024-09-16T01:35:01.354497+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:34:25.731515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.347434+0000","last_change":"2024-09-16T01:35:00.342181+0000","last_active":"2024-09-16T01:35:00.347434+0000","last_peered":"2024-09-16T01:35:00.347434+0000","last_clean":"2024-09-16T01:35:00.347434+0000","last_became_active":"2024-09-16T01:35:00.341878+0000","last_became_peered":"2024-09-16T01:35:00.341878+0000","last_unstale":"2024-09-16T01:35:00.347434+0000","last_undegraded":"2024-09-16T01:35:00.347434+0000","last_fullsized":"2024-09-16T01:35:00.347434+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:17:19.574295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339159+0000","last_change":"2024-09-16T01:35:00.339159+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339159+0000","last_undegraded":"2024-09-16T01:35:00.339159+0000","last_fullsized":"2024-09-16T01:35:00.339159+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353150+0000","last_change":"2024-09-16T01:35:00.349566+0000","last_active":"2024-09-16T01:35:00.353150+0000","last_peered":"2024-09-16T01:35:00.353150+0000","last_clean":"2024-09-16T01:35:00.353150+0000","last_became_active":"2024-09-16T01:35:00.349243+0000","last_became_peered":"2024-09-16T01:35:00.349243+0000","last_unstale":"2024-09-16T01:35:00.353150+0000","last_undegraded":"2024-09-16T01:35:00.353150+0000","last_fullsized":"2024-09-16T01:35:00.353150+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:43:05.059186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.341010+0000","last_change":"2024-09-16T01:35:00.341010+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.341010+0000","last_undegraded":"2024-09-16T01:35:00.341010+0000","last_fullsized":"2024-09-16T01:35:00.341010+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353118+0000","last_change":"2024-09-16T01:35:00.348167+0000","last_active":"2024-09-16T01:35:00.353118+0000","last_peered":"2024-09-16T01:35:00.353118+0000","last_clean":"2024-09-16T01:35:00.353118+0000","last_became_active":"2024-09-16T01:35:00.347714+0000","last_became_peered":"2024-09-16T01:35:00.347714+0000","last_unstale":"2024-09-16T01:35:00.353118+0000","last_undegraded":"2024-09-16T01:35:00.353118+0000","last_fullsized":"2024-09-16T01:35:00.353118+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:29:35.905712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.342418+0000","last_change":"2024-09-16T01:35:00.342418+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.342418+0000","last_undegraded":"2024-09-16T01:35:00.342418+0000","last_fullsized":"2024-09-16T01:35:00.342418+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.346227+0000","last_change":"2024-09-16T01:35:00.345340+0000","last_active":"2024-09-16T01:35:00.346227+0000","last_peered":"2024-09-16T01:35:00.346227+0000","last_clean":"2024-09-16T01:35:00.346227+0000","last_became_active":"2024-09-16T01:35:00.344987+0000","last_became_peered":"2024-09-16T01:35:00.344987+0000","last_unstale":"2024-09-16T01:35:00.346227+0000","last_undegraded":"2024-09-16T01:35:00.346227+0000","last_fullsized":"2024-09-16T01:35:00.346227+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:02:54.003881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.340705+0000","last_change":"2024-09-16T01:35:00.340705+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.340705+0000","last_undegraded":"2024-09-16T01:35:00.340705+0000","last_fullsized":"2024-09-16T01:35:00.340705+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.354908+0000","last_change":"2024-09-16T01:35:00.349814+0000","last_active":"2024-09-16T01:35:00.354908+0000","last_peered":"2024-09-16T01:35:00.354908+0000","last_clean":"2024-09-16T01:35:00.354908+0000","last_became_active":"2024-09-16T01:35:00.349543+0000","last_became_peered":"2024-09-16T01:35:00.349543+0000","last_unstale":"2024-09-16T01:35:00.354908+0000","last_undegraded":"2024-09-16T01:35:00.354908+0000","last_fullsized":"2024-09-16T01:35:00.354908+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:02:54.003881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.342349+0000","last_change":"2024-09-16T01:35:00.342349+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.342349+0000","last_undegraded":"2024-09-16T01:35:00.342349+0000","last_fullsized":"2024-09-16T01:35:00.342349+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353146+0000","last_change":"2024-09-16T01:35:00.348335+0000","last_active":"2024-09-16T01:35:00.353146+0000","last_peered":"2024-09-16T01:35:00.353146+0000","last_clean":"2024-09-16T01:35:00.353146+0000","last_became_active":"2024-09-16T01:35:00.347099+0000","last_became_peered":"2024-09-16T01:35:00.347099+0000","last_unstale":"2024-09-16T01:35:00.353146+0000","last_undegraded":"2024-09-16T01:35:00.353146+0000","last_fullsized":"2024-09-16T01:35:00.353146+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:57:35.138402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.346143+0000","last_change":"2024-09-16T01:35:00.346143+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.346143+0000","last_undegraded":"2024-09-16T01:35:00.346143+0000","last_fullsized":"2024-09-16T01:35:00.346143+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.337675+0000","last_change":"2024-09-16T01:35:00.352839+0000","last_active":"2024-09-16T01:35:01.337675+0000","last_peered":"2024-09-16T01:35:01.337675+0000","last_clean":"2024-09-16T01:35:01.337675+0000","last_became_active":"2024-09-16T01:35:00.352568+0000","last_became_peered":"2024-09-16T01:35:00.352568+0000","last_unstale":"2024-09-16T01:35:01.337675+0000","last_undegraded":"2024-09-16T01:35:01.337675+0000","last_fullsized":"2024-09-16T01:35:01.337675+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:30:51.141162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.342110+0000","last_change":"2024-09-16T01:35:00.342110+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.342110+0000","last_undegraded":"2024-09-16T01:35:00.342110+0000","last_fullsized":"2024-09-16T01:35:00.342110+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.357904+0000","last_change":"2024-09-16T01:35:00.355682+0000","last_active":"2024-09-16T01:35:00.357904+0000","last_peered":"2024-09-16T01:35:00.357904+0000","last_clean":"2024-09-16T01:35:00.357904+0000","last_became_active":"2024-09-16T01:35:00.355347+0000","last_became_peered":"2024-09-16T01:35:00.355347+0000","last_unstale":"2024-09-16T01:35:00.357904+0000","last_undegraded":"2024-09-16T01:35:00.357904+0000","last_fullsized":"2024-09-16T01:35:00.357904+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:07.992492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.359762+0000","last_change":"2024-09-16T01:35:01.355640+0000","last_active":"2024-09-16T01:35:01.359762+0000","last_peered":"2024-09-16T01:35:01.359762+0000","last_clean":"2024-09-16T01:35:01.359762+0000","last_became_active":"2024-09-16T01:35:01.355364+0000","last_became_peered":"2024-09-16T01:35:01.355364+0000","last_unstale":"2024-09-16T01:35:01.359762+0000","last_undegraded":"2024-09-16T01:35:01.359762+0000","last_fullsized":"2024-09-16T01:35:01.359762+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:26:10.193445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.346212+0000","last_change":"2024-09-16T01:35:00.343346+0000","last_active":"2024-09-16T01:35:00.346212+0000","last_peered":"2024-09-16T01:35:00.346212+0000","last_clean":"2024-09-16T01:35:00.346212+0000","last_became_active":"2024-09-16T01:35:00.343049+0000","last_became_peered":"2024-09-16T01:35:00.343049+0000","last_unstale":"2024-09-16T01:35:00.346212+0000","last_undegraded":"2024-09-16T01:35:00.346212+0000","last_fullsized":"2024-09-16T01:35:00.346212+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:00.117677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.335782+0000","last_change":"2024-09-16T01:35:00.335782+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.335782+0000","last_undegraded":"2024-09-16T01:35:00.335782+0000","last_fullsized":"2024-09-16T01:35:00.335782+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352715+0000","last_change":"2024-09-16T01:35:00.349236+0000","last_active":"2024-09-16T01:35:00.352715+0000","last_peered":"2024-09-16T01:35:00.352715+0000","last_clean":"2024-09-16T01:35:00.352715+0000","last_became_active":"2024-09-16T01:35:00.348947+0000","last_became_peered":"2024-09-16T01:35:00.348947+0000","last_unstale":"2024-09-16T01:35:00.352715+0000","last_undegraded":"2024-09-16T01:35:00.352715+0000","last_fullsized":"2024-09-16T01:35:00.352715+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:18:05.552429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.336999+0000","last_change":"2024-09-16T01:35:00.336999+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.336999+0000","last_undegraded":"2024-09-16T01:35:00.336999+0000","last_fullsized":"2024-09-16T01:35:00.336999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.357770+0000","last_change":"2024-09-16T01:35:00.355723+0000","last_active":"2024-09-16T01:35:00.357770+0000","last_peered":"2024-09-16T01:35:00.357770+0000","last_clean":"2024-09-16T01:35:00.357770+0000","last_became_active":"2024-09-16T01:35:00.355425+0000","last_became_peered":"2024-09-16T01:35:00.355425+0000","last_unstale":"2024-09-16T01:35:00.357770+0000","last_undegraded":"2024-09-16T01:35:00.357770+0000","last_fullsized":"2024-09-16T01:35:00.357770+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:00.732201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.335633+0000","last_change":"2024-09-16T01:35:00.335633+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.335633+0000","last_undegraded":"2024-09-16T01:35:00.335633+0000","last_fullsized":"2024-09-16T01:35:00.335633+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.357916+0000","last_change":"2024-09-16T01:35:00.355687+0000","last_active":"2024-09-16T01:35:00.357916+0000","last_peered":"2024-09-16T01:35:00.357916+0000","last_clean":"2024-09-16T01:35:00.357916+0000","last_became_active":"2024-09-16T01:35:00.355357+0000","last_became_peered":"2024-09-16T01:35:00.355357+0000","last_unstale":"2024-09-16T01:35:00.357916+0000","last_undegraded":"2024-09-16T01:35:00.357916+0000","last_fullsized":"2024-09-16T01:35:00.357916+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:30:55.369673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337712+0000","last_change":"2024-09-16T01:35:00.337712+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337712+0000","last_undegraded":"2024-09-16T01:35:00.337712+0000","last_fullsized":"2024-09-16T01:35:00.337712+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353170+0000","last_change":"2024-09-16T01:35:00.351123+0000","last_active":"2024-09-16T01:35:00.353170+0000","last_peered":"2024-09-16T01:35:00.353170+0000","last_clean":"2024-09-16T01:35:00.353170+0000","last_became_active":"2024-09-16T01:35:00.350803+0000","last_became_peered":"2024-09-16T01:35:00.350803+0000","last_unstale":"2024-09-16T01:35:00.353170+0000","last_undegraded":"2024-09-16T01:35:00.353170+0000","last_fullsized":"2024-09-16T01:35:00.353170+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:51:21.665512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339066+0000","last_change":"2024-09-16T01:35:00.339066+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339066+0000","last_undegraded":"2024-09-16T01:35:00.339066+0000","last_fullsized":"2024-09-16T01:35:00.339066+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353039+0000","last_change":"2024-09-16T01:35:00.351535+0000","last_active":"2024-09-16T01:35:00.353039+0000","last_peered":"2024-09-16T01:35:00.353039+0000","last_clean":"2024-09-16T01:35:00.353039+0000","last_became_active":"2024-09-16T01:35:00.351219+0000","last_became_peered":"2024-09-16T01:35:00.351219+0000","last_unstale":"2024-09-16T01:35:00.353039+0000","last_undegraded":"2024-09-16T01:35:00.353039+0000","last_fullsized":"2024-09-16T01:35:00.353039+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:04.599770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.342993+0000","last_change":"2024-09-16T01:35:00.342993+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.342993+0000","last_undegraded":"2024-09-16T01:35:00.342993+0000","last_fullsized":"2024-09-16T01:35:00.342993+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352480+0000","last_change":"2024-09-16T01:35:00.344871+0000","last_active":"2024-09-16T01:35:00.352480+0000","last_peered":"2024-09-16T01:35:00.352480+0000","last_clean":"2024-09-16T01:35:00.352480+0000","last_became_active":"2024-09-16T01:35:00.344582+0000","last_became_peered":"2024-09-16T01:35:00.344582+0000","last_unstale":"2024-09-16T01:35:00.352480+0000","last_undegraded":"2024-09-16T01:35:00.352480+0000","last_fullsized":"2024-09-16T01:35:00.352480+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:00.117677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":185,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.348767+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:35:00.348767+0000","last_peered":"2024-09-16T01:35:00.348767+0000","last_clean":"2024-09-16T01:35:00.348767+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:35:00.348767+0000","last_undegraded":"2024-09-16T01:35:00.348767+0000","last_fullsized":"2024-09-16T01:35:00.348767+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509455,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":331776,"data_stored":123852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542158,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27920,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343920,"statfs":{"total":96636764160,"available":96608174080,"internally_reserved":0,"allocated":913408,"data_stored":707953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542158,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":117585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542158,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":117585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542157,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":117585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542158,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":117585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542157,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":117585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542157,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":925696,"data_stored":714220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:35:04.289 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:35:04.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.422+0000 7fec0fcac640 1 Processor -- start 2024-09-16T01:35:04.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.422+0000 7fec0fcac640 1 -- start start 2024-09-16T01:35:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.423+0000 7fec0fcac640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fec0806d540 0x7fec080cf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.423+0000 7fec0fcac640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fec080735b0 0x7fec080cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.423+0000 7fec0fcac640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec0806faf0 con 0x7fec0815e950 2024-09-16T01:35:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.423+0000 7fec0fcac640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fec0806fdd0 con 0x7fec0806d540 2024-09-16T01:35:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.423+0000 7fec0fcac640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fec080700b0 con 0x7fec080735b0 2024-09-16T01:35:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.423+0000 7fec0da48640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fec0815e950 0x7fec0806ce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35976/0 (socket says 172.21.15.88:35976) 2024-09-16T01:35:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.423+0000 7fec0da48640 1 -- 172.21.15.88:0/306157381 learned_addr learned my addr 172.21.15.88:0/306157381 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.423+0000 7fec0d247640 1 --2- 172.21.15.88:0/306157381 >> v2:172.21.15.88:3300/0 conn(0x7fec0806d540 0x7fec080cf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.423+0000 7fec0e249640 1 --2- 172.21.15.88:0/306157381 >> v2:172.21.15.154:3300/0 conn(0x7fec080735b0 0x7fec080cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.423+0000 7fec0d247640 1 -- 172.21.15.88:0/306157381 >> v2:172.21.15.154:3300/0 conn(0x7fec080735b0 msgr2=0x7fec080cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.423+0000 7fec0d247640 1 --2- 172.21.15.88:0/306157381 >> v2:172.21.15.154:3300/0 conn(0x7fec080735b0 0x7fec080cf5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.423+0000 7fec0d247640 1 -- 172.21.15.88:0/306157381 >> v1:172.21.15.88:6789/0 conn(0x7fec0815e950 legacy=0x7fec0806ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.423+0000 7fec0d247640 1 -- 172.21.15.88:0/306157381 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec08070390 con 0x7fec0806d540 2024-09-16T01:35:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.423+0000 7fec0ca46640 1 -- 172.21.15.88:0/306157381 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1792333106 0 0) 0x7fec0806faf0 con 0x7fec0815e950 2024-09-16T01:35:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.423+0000 7fec0e249640 1 --2- 172.21.15.88:0/306157381 >> v2:172.21.15.154:3300/0 conn(0x7fec080735b0 0x7fec080cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:04.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.424+0000 7fec0d247640 1 --2- 172.21.15.88:0/306157381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec0806d540 0x7fec080cf0a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fec08071c80 tx=0x7febf802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d2679a91d63543d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:04.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.424+0000 7fec0ca46640 1 -- 172.21.15.88:0/306157381 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf8037070 con 0x7fec0806d540 2024-09-16T01:35:04.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.424+0000 7fec0ca46640 1 -- 172.21.15.88:0/306157381 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febf8032440 con 0x7fec0806d540 2024-09-16T01:35:04.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.424+0000 7fec0ca46640 1 -- 172.21.15.88:0/306157381 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf8032810 con 0x7fec0806d540 2024-09-16T01:35:04.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.425+0000 7fec0fcac640 1 -- 172.21.15.88:0/306157381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec0806d540 msgr2=0x7fec080cf0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:04.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.425+0000 7fec0fcac640 1 --2- 172.21.15.88:0/306157381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec0806d540 0x7fec080cf0a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fec08071c80 tx=0x7febf802a1a0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.425+0000 7fec0fcac640 1 -- 172.21.15.88:0/306157381 shutdown_connections 2024-09-16T01:35:04.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.425+0000 7fec0fcac640 1 --2- 172.21.15.88:0/306157381 >> v2:172.21.15.154:3300/0 conn(0x7fec080735b0 0x7fec080cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.425+0000 7fec0fcac640 1 --2- 172.21.15.88:0/306157381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec0806d540 0x7fec080cf0a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.425+0000 7fec0fcac640 1 -- 172.21.15.88:0/306157381 >> 172.21.15.88:0/306157381 conn(0x7fec080d14d0 msgr2=0x7fec080d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:04.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.425+0000 7fec0fcac640 1 -- 172.21.15.88:0/306157381 shutdown_connections 2024-09-16T01:35:04.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.425+0000 7fec0fcac640 1 -- 172.21.15.88:0/306157381 wait complete. 2024-09-16T01:35:04.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.426+0000 7fec0fcac640 1 Processor -- start 2024-09-16T01:35:04.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.426+0000 7fec0fcac640 1 -- start start 2024-09-16T01:35:04.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.427+0000 7fec0fcac640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec0806d540 0x7fec08146d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:04.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.427+0000 7fec0fcac640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fec080735b0 0x7fec08143a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:04.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.427+0000 7fec0fcac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec0815e950 0x7fec08160f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:04.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.427+0000 7fec0fcac640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fec080721d0 con 0x7fec0806d540 2024-09-16T01:35:04.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.427+0000 7fec0fcac640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fec08072050 con 0x7fec0815e950 2024-09-16T01:35:04.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.427+0000 7fec0fcac640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fec08072350 con 0x7fec080735b0 2024-09-16T01:35:04.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.427+0000 7fec0da48640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec0806d540 0x7fec08146d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:04.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.427+0000 7fec0da48640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec0806d540 0x7fec08146d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36474/0 (socket says 172.21.15.88:36474) 2024-09-16T01:35:04.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.427+0000 7fec0da48640 1 -- 172.21.15.88:0/1481813873 learned_addr learned my addr 172.21.15.88:0/1481813873 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:04.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.427+0000 7fec0da48640 1 -- 172.21.15.88:0/1481813873 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fec080735b0 msgr2=0x7fec08143a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:04.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.427+0000 7fec0e249640 1 --2- 172.21.15.88:0/1481813873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec0815e950 0x7fec08160f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:04.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.427+0000 7fec0da48640 1 --2- 172.21.15.88:0/1481813873 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fec080735b0 0x7fec08143a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.427+0000 7fec0da48640 1 -- 172.21.15.88:0/1481813873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec0815e950 msgr2=0x7fec08160f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:04.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.427+0000 7fec0da48640 1 --2- 172.21.15.88:0/1481813873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec0815e950 0x7fec08160f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.427+0000 7fec0da48640 1 -- 172.21.15.88:0/1481813873 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec08147280 con 0x7fec0806d540 2024-09-16T01:35:04.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.427+0000 7fec0da48640 1 --2- 172.21.15.88:0/1481813873 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec0806d540 0x7fec08146d40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fec0400c4d0 tx=0x7fec0400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:04.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.427+0000 7febfe7fc640 1 -- 172.21.15.88:0/1481813873 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec04018070 con 0x7fec0806d540 2024-09-16T01:35:04.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.427+0000 7febfe7fc640 1 -- 172.21.15.88:0/1481813873 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec040074e0 con 0x7fec0806d540 2024-09-16T01:35:04.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.427+0000 7fec0fcac640 1 -- 172.21.15.88:0/1481813873 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec08157c20 con 0x7fec0806d540 2024-09-16T01:35:04.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.427+0000 7febfe7fc640 1 -- 172.21.15.88:0/1481813873 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec04004a20 con 0x7fec0806d540 2024-09-16T01:35:04.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.427+0000 7fec0fcac640 1 -- 172.21.15.88:0/1481813873 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec081f7b60 con 0x7fec0806d540 2024-09-16T01:35:04.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.428+0000 7fec0fcac640 1 -- 172.21.15.88:0/1481813873 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec0806ce30 con 0x7fec0806d540 2024-09-16T01:35:04.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.429+0000 7febfe7fc640 1 -- 172.21.15.88:0/1481813873 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fec0401b070 con 0x7fec0806d540 2024-09-16T01:35:04.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.429+0000 7febfe7fc640 1 --2- 172.21.15.88:0/1481813873 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7febec06c2b0 0x7febec06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:04.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.430+0000 7fec0d247640 1 --2- 172.21.15.88:0/1481813873 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7febec06c2b0 0x7febec06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:04.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.430+0000 7febfe7fc640 1 -- 172.21.15.88:0/1481813873 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fec0408a6f0 con 0x7fec0806d540 2024-09-16T01:35:04.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.430+0000 7fec0d247640 1 --2- 172.21.15.88:0/1481813873 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7febec06c2b0 0x7febec06e770 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7febf8034000 tx=0x7febf802c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:04.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.434+0000 7febfe7fc640 1 -- 172.21.15.88:0/1481813873 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fec08147280 con 0x7fec0806d540 2024-09-16T01:35:04.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.600+0000 7fec0fcac640 1 -- 172.21.15.88:0/1481813873 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fec080b37e0 con 0x7febec06c2b0 2024-09-16T01:35:04.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.606+0000 7febfe7fc640 1 -- 172.21.15.88:0/1481813873 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+247449 (secure 0 0 0) 0x7fec080b37e0 con 0x7febec06c2b0 2024-09-16T01:35:04.610 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:35:04.611 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:35:04.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.609+0000 7fec0fcac640 1 -- 172.21.15.88:0/1481813873 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7febec06c2b0 msgr2=0x7febec06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:04.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.609+0000 7fec0fcac640 1 --2- 172.21.15.88:0/1481813873 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7febec06c2b0 0x7febec06e770 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7febf8034000 tx=0x7febf802c000 comp rx=0 tx=0).stop 2024-09-16T01:35:04.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.609+0000 7fec0fcac640 1 -- 172.21.15.88:0/1481813873 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec0806d540 msgr2=0x7fec08146d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:04.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.609+0000 7fec0fcac640 1 --2- 172.21.15.88:0/1481813873 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec0806d540 0x7fec08146d40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fec0400c4d0 tx=0x7fec0400c990 comp rx=0 tx=0).stop 2024-09-16T01:35:04.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.609+0000 7fec0fcac640 1 -- 172.21.15.88:0/1481813873 shutdown_connections 2024-09-16T01:35:04.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.609+0000 7fec0fcac640 1 --2- 172.21.15.88:0/1481813873 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7febec06c2b0 0x7febec06e770 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.609+0000 7fec0fcac640 1 --2- 172.21.15.88:0/1481813873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec0815e950 0x7fec08160f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.609+0000 7fec0fcac640 1 --2- 172.21.15.88:0/1481813873 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fec080735b0 0x7fec08143a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.609+0000 7fec0fcac640 1 --2- 172.21.15.88:0/1481813873 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec0806d540 0x7fec08146d40 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.609+0000 7fec0fcac640 1 -- 172.21.15.88:0/1481813873 >> 172.21.15.88:0/1481813873 conn(0x7fec080d14d0 msgr2=0x7fec080d0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:04.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.609+0000 7fec0fcac640 1 -- 172.21.15.88:0/1481813873 shutdown_connections 2024-09-16T01:35:04.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.609+0000 7fec0fcac640 1 -- 172.21.15.88:0/1481813873 wait complete. 2024-09-16T01:35:04.622 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-09-16T01:35:04.524493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590458,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":163,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":28,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219912,"kb_used_data":3680,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754808,"statfs":{"total":773094113280,"available":772868923392,"internally_reserved":0,"allocated":3768320,"data_stored":2133950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.198029"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337649+0000","last_change":"2024-09-16T01:35:00.337649+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337649+0000","last_undegraded":"2024-09-16T01:35:00.337649+0000","last_fullsized":"2024-09-16T01:35:00.337649+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339909+0000","last_change":"2024-09-16T01:35:00.339909+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339909+0000","last_undegraded":"2024-09-16T01:35:00.339909+0000","last_fullsized":"2024-09-16T01:35:00.339909+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.357429+0000","last_change":"2024-09-16T01:35:01.353507+0000","last_active":"2024-09-16T01:35:01.357429+0000","last_peered":"2024-09-16T01:35:01.357429+0000","last_clean":"2024-09-16T01:35:01.357429+0000","last_became_active":"2024-09-16T01:35:01.352327+0000","last_became_peered":"2024-09-16T01:35:01.352327+0000","last_unstale":"2024-09-16T01:35:01.357429+0000","last_undegraded":"2024-09-16T01:35:01.357429+0000","last_fullsized":"2024-09-16T01:35:01.357429+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:55.644944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.340201+0000","last_change":"2024-09-16T01:35:00.340201+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.340201+0000","last_undegraded":"2024-09-16T01:35:00.340201+0000","last_fullsized":"2024-09-16T01:35:00.340201+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.341590+0000","last_change":"2024-09-16T01:35:00.341590+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.341590+0000","last_undegraded":"2024-09-16T01:35:00.341590+0000","last_fullsized":"2024-09-16T01:35:00.341590+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338751+0000","last_change":"2024-09-16T01:35:00.338751+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338751+0000","last_undegraded":"2024-09-16T01:35:00.338751+0000","last_fullsized":"2024-09-16T01:35:00.338751+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.345760+0000","last_change":"2024-09-16T01:35:00.345760+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.345760+0000","last_undegraded":"2024-09-16T01:35:00.345760+0000","last_fullsized":"2024-09-16T01:35:00.345760+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.347744+0000","last_change":"2024-09-16T01:35:00.347744+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.347744+0000","last_undegraded":"2024-09-16T01:35:00.347744+0000","last_fullsized":"2024-09-16T01:35:00.347744+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337574+0000","last_change":"2024-09-16T01:35:00.337574+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337574+0000","last_undegraded":"2024-09-16T01:35:00.337574+0000","last_fullsized":"2024-09-16T01:35:00.337574+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339626+0000","last_change":"2024-09-16T01:35:00.339626+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339626+0000","last_undegraded":"2024-09-16T01:35:00.339626+0000","last_fullsized":"2024-09-16T01:35:00.339626+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.344537+0000","last_change":"2024-09-16T01:35:00.344537+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.344537+0000","last_undegraded":"2024-09-16T01:35:00.344537+0000","last_fullsized":"2024-09-16T01:35:00.344537+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.354548+0000","last_change":"2024-09-16T01:35:01.349955+0000","last_active":"2024-09-16T01:35:01.354548+0000","last_peered":"2024-09-16T01:35:01.354548+0000","last_clean":"2024-09-16T01:35:01.354548+0000","last_became_active":"2024-09-16T01:35:01.349689+0000","last_became_peered":"2024-09-16T01:35:01.349689+0000","last_unstale":"2024-09-16T01:35:01.354548+0000","last_undegraded":"2024-09-16T01:35:01.354548+0000","last_fullsized":"2024-09-16T01:35:01.354548+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:02:10.872199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.341163+0000","last_change":"2024-09-16T01:35:00.341163+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.341163+0000","last_undegraded":"2024-09-16T01:35:00.341163+0000","last_fullsized":"2024-09-16T01:35:00.341163+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337577+0000","last_change":"2024-09-16T01:35:00.337577+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337577+0000","last_undegraded":"2024-09-16T01:35:00.337577+0000","last_fullsized":"2024-09-16T01:35:00.337577+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.353724+0000","last_change":"2024-09-16T01:35:01.344630+0000","last_active":"2024-09-16T01:35:01.353724+0000","last_peered":"2024-09-16T01:35:01.353724+0000","last_clean":"2024-09-16T01:35:01.353724+0000","last_became_active":"2024-09-16T01:35:01.344340+0000","last_became_peered":"2024-09-16T01:35:01.344340+0000","last_unstale":"2024-09-16T01:35:01.353724+0000","last_undegraded":"2024-09-16T01:35:01.353724+0000","last_fullsized":"2024-09-16T01:35:01.353724+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:06:54.456413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.342533+0000","last_change":"2024-09-16T01:35:00.342533+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.342533+0000","last_undegraded":"2024-09-16T01:35:00.342533+0000","last_fullsized":"2024-09-16T01:35:00.342533+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.343962+0000","last_change":"2024-09-16T01:35:00.343962+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.343962+0000","last_undegraded":"2024-09-16T01:35:00.343962+0000","last_fullsized":"2024-09-16T01:35:00.343962+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339533+0000","last_change":"2024-09-16T01:35:00.339533+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339533+0000","last_undegraded":"2024-09-16T01:35:00.339533+0000","last_fullsized":"2024-09-16T01:35:00.339533+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337206+0000","last_change":"2024-09-16T01:35:00.337206+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337206+0000","last_undegraded":"2024-09-16T01:35:00.337206+0000","last_fullsized":"2024-09-16T01:35:00.337206+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.340512+0000","last_change":"2024-09-16T01:35:00.340512+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.340512+0000","last_undegraded":"2024-09-16T01:35:00.340512+0000","last_fullsized":"2024-09-16T01:35:00.340512+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339909+0000","last_change":"2024-09-16T01:35:00.339909+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339909+0000","last_undegraded":"2024-09-16T01:35:00.339909+0000","last_fullsized":"2024-09-16T01:35:00.339909+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338369+0000","last_change":"2024-09-16T01:35:00.338369+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338369+0000","last_undegraded":"2024-09-16T01:35:00.338369+0000","last_fullsized":"2024-09-16T01:35:00.338369+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.335686+0000","last_change":"2024-09-16T01:35:00.335686+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.335686+0000","last_undegraded":"2024-09-16T01:35:00.335686+0000","last_fullsized":"2024-09-16T01:35:00.335686+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.353084+0000","last_change":"2024-09-16T01:35:01.344447+0000","last_active":"2024-09-16T01:35:01.353084+0000","last_peered":"2024-09-16T01:35:01.353084+0000","last_clean":"2024-09-16T01:35:01.353084+0000","last_became_active":"2024-09-16T01:35:01.344135+0000","last_became_peered":"2024-09-16T01:35:01.344135+0000","last_unstale":"2024-09-16T01:35:01.353084+0000","last_undegraded":"2024-09-16T01:35:01.353084+0000","last_fullsized":"2024-09-16T01:35:01.353084+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:44:42.157718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338640+0000","last_change":"2024-09-16T01:35:00.338640+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338640+0000","last_undegraded":"2024-09-16T01:35:00.338640+0000","last_fullsized":"2024-09-16T01:35:00.338640+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339123+0000","last_change":"2024-09-16T01:35:00.339123+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339123+0000","last_undegraded":"2024-09-16T01:35:00.339123+0000","last_fullsized":"2024-09-16T01:35:00.339123+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337376+0000","last_change":"2024-09-16T01:35:00.337376+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337376+0000","last_undegraded":"2024-09-16T01:35:00.337376+0000","last_fullsized":"2024-09-16T01:35:00.337376+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.352090+0000","last_change":"2024-09-16T01:35:01.343532+0000","last_active":"2024-09-16T01:35:01.352090+0000","last_peered":"2024-09-16T01:35:01.352090+0000","last_clean":"2024-09-16T01:35:01.352090+0000","last_became_active":"2024-09-16T01:35:01.342495+0000","last_became_peered":"2024-09-16T01:35:01.342495+0000","last_unstale":"2024-09-16T01:35:01.352090+0000","last_undegraded":"2024-09-16T01:35:01.352090+0000","last_fullsized":"2024-09-16T01:35:01.352090+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:32:33.652857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339456+0000","last_change":"2024-09-16T01:35:00.339456+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339456+0000","last_undegraded":"2024-09-16T01:35:00.339456+0000","last_fullsized":"2024-09-16T01:35:00.339456+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.343049+0000","last_change":"2024-09-16T01:35:00.343049+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.343049+0000","last_undegraded":"2024-09-16T01:35:00.343049+0000","last_fullsized":"2024-09-16T01:35:00.343049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338353+0000","last_change":"2024-09-16T01:35:00.338353+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338353+0000","last_undegraded":"2024-09-16T01:35:00.338353+0000","last_fullsized":"2024-09-16T01:35:00.338353+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.347252+0000","last_change":"2024-09-16T01:35:00.347252+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.347252+0000","last_undegraded":"2024-09-16T01:35:00.347252+0000","last_fullsized":"2024-09-16T01:35:00.347252+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.335647+0000","last_change":"2024-09-16T01:35:00.335647+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.335647+0000","last_undegraded":"2024-09-16T01:35:00.335647+0000","last_fullsized":"2024-09-16T01:35:00.335647+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.358487+0000","last_change":"2024-09-16T01:35:00.355732+0000","last_active":"2024-09-16T01:35:00.358487+0000","last_peered":"2024-09-16T01:35:00.358487+0000","last_clean":"2024-09-16T01:35:00.358487+0000","last_became_active":"2024-09-16T01:35:00.355431+0000","last_became_peered":"2024-09-16T01:35:00.355431+0000","last_unstale":"2024-09-16T01:35:00.358487+0000","last_undegraded":"2024-09-16T01:35:00.358487+0000","last_fullsized":"2024-09-16T01:35:00.358487+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:48:54.665338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337125+0000","last_change":"2024-09-16T01:35:00.337125+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337125+0000","last_undegraded":"2024-09-16T01:35:00.337125+0000","last_fullsized":"2024-09-16T01:35:00.337125+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352461+0000","last_change":"2024-09-16T01:35:00.348929+0000","last_active":"2024-09-16T01:35:00.352461+0000","last_peered":"2024-09-16T01:35:00.352461+0000","last_clean":"2024-09-16T01:35:00.352461+0000","last_became_active":"2024-09-16T01:35:00.348651+0000","last_became_peered":"2024-09-16T01:35:00.348651+0000","last_unstale":"2024-09-16T01:35:00.352461+0000","last_undegraded":"2024-09-16T01:35:00.352461+0000","last_fullsized":"2024-09-16T01:35:00.352461+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:52:02.746213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339423+0000","last_change":"2024-09-16T01:35:00.339423+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339423+0000","last_undegraded":"2024-09-16T01:35:00.339423+0000","last_fullsized":"2024-09-16T01:35:00.339423+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.346035+0000","last_change":"2024-09-16T01:35:00.344907+0000","last_active":"2024-09-16T01:35:00.346035+0000","last_peered":"2024-09-16T01:35:00.346035+0000","last_clean":"2024-09-16T01:35:00.346035+0000","last_became_active":"2024-09-16T01:35:00.344634+0000","last_became_peered":"2024-09-16T01:35:00.344634+0000","last_unstale":"2024-09-16T01:35:00.346035+0000","last_undegraded":"2024-09-16T01:35:00.346035+0000","last_fullsized":"2024-09-16T01:35:00.346035+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:00.732201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.336922+0000","last_change":"2024-09-16T01:35:00.336922+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.336922+0000","last_undegraded":"2024-09-16T01:35:00.336922+0000","last_fullsized":"2024-09-16T01:35:00.336922+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352455+0000","last_change":"2024-09-16T01:35:00.348009+0000","last_active":"2024-09-16T01:35:00.352455+0000","last_peered":"2024-09-16T01:35:00.352455+0000","last_clean":"2024-09-16T01:35:00.352455+0000","last_became_active":"2024-09-16T01:35:00.347736+0000","last_became_peered":"2024-09-16T01:35:00.347736+0000","last_unstale":"2024-09-16T01:35:00.352455+0000","last_undegraded":"2024-09-16T01:35:00.352455+0000","last_fullsized":"2024-09-16T01:35:00.352455+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:44.087710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.342570+0000","last_change":"2024-09-16T01:35:00.342570+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.342570+0000","last_undegraded":"2024-09-16T01:35:00.342570+0000","last_fullsized":"2024-09-16T01:35:00.342570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353110+0000","last_change":"2024-09-16T01:35:00.348379+0000","last_active":"2024-09-16T01:35:00.353110+0000","last_peered":"2024-09-16T01:35:00.353110+0000","last_clean":"2024-09-16T01:35:00.353110+0000","last_became_active":"2024-09-16T01:35:00.348008+0000","last_became_peered":"2024-09-16T01:35:00.348008+0000","last_unstale":"2024-09-16T01:35:00.353110+0000","last_undegraded":"2024-09-16T01:35:00.353110+0000","last_fullsized":"2024-09-16T01:35:00.353110+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:56:53.528942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.354400+0000","last_change":"2024-09-16T01:35:01.346165+0000","last_active":"2024-09-16T01:35:01.354400+0000","last_peered":"2024-09-16T01:35:01.354400+0000","last_clean":"2024-09-16T01:35:01.354400+0000","last_became_active":"2024-09-16T01:35:01.344479+0000","last_became_peered":"2024-09-16T01:35:01.344479+0000","last_unstale":"2024-09-16T01:35:01.354400+0000","last_undegraded":"2024-09-16T01:35:01.354400+0000","last_fullsized":"2024-09-16T01:35:01.354400+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:58:00.280748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353081+0000","last_change":"2024-09-16T01:35:00.347395+0000","last_active":"2024-09-16T01:35:00.353081+0000","last_peered":"2024-09-16T01:35:00.353081+0000","last_clean":"2024-09-16T01:35:00.353081+0000","last_became_active":"2024-09-16T01:35:00.346752+0000","last_became_peered":"2024-09-16T01:35:00.346752+0000","last_unstale":"2024-09-16T01:35:00.353081+0000","last_undegraded":"2024-09-16T01:35:00.353081+0000","last_fullsized":"2024-09-16T01:35:00.353081+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:51.973054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338296+0000","last_change":"2024-09-16T01:35:00.338296+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338296+0000","last_undegraded":"2024-09-16T01:35:00.338296+0000","last_fullsized":"2024-09-16T01:35:00.338296+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352488+0000","last_change":"2024-09-16T01:35:00.348232+0000","last_active":"2024-09-16T01:35:00.352488+0000","last_peered":"2024-09-16T01:35:00.352488+0000","last_clean":"2024-09-16T01:35:00.352488+0000","last_became_active":"2024-09-16T01:35:00.347976+0000","last_became_peered":"2024-09-16T01:35:00.347976+0000","last_unstale":"2024-09-16T01:35:00.352488+0000","last_undegraded":"2024-09-16T01:35:00.352488+0000","last_fullsized":"2024-09-16T01:35:00.352488+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:49:23.770293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.336753+0000","last_change":"2024-09-16T01:35:00.336753+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.336753+0000","last_undegraded":"2024-09-16T01:35:00.336753+0000","last_fullsized":"2024-09-16T01:35:00.336753+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.347485+0000","last_change":"2024-09-16T01:35:00.342334+0000","last_active":"2024-09-16T01:35:00.347485+0000","last_peered":"2024-09-16T01:35:00.347485+0000","last_clean":"2024-09-16T01:35:00.347485+0000","last_became_active":"2024-09-16T01:35:00.342056+0000","last_became_peered":"2024-09-16T01:35:00.342056+0000","last_unstale":"2024-09-16T01:35:00.347485+0000","last_undegraded":"2024-09-16T01:35:00.347485+0000","last_fullsized":"2024-09-16T01:35:00.347485+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:32:43.356912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.355290+0000","last_change":"2024-09-16T01:35:01.349973+0000","last_active":"2024-09-16T01:35:01.355290+0000","last_peered":"2024-09-16T01:35:01.355290+0000","last_clean":"2024-09-16T01:35:01.355290+0000","last_became_active":"2024-09-16T01:35:01.349733+0000","last_became_peered":"2024-09-16T01:35:01.349733+0000","last_unstale":"2024-09-16T01:35:01.355290+0000","last_undegraded":"2024-09-16T01:35:01.355290+0000","last_fullsized":"2024-09-16T01:35:01.355290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:59:24.976252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352519+0000","last_change":"2024-09-16T01:35:00.349540+0000","last_active":"2024-09-16T01:35:00.352519+0000","last_peered":"2024-09-16T01:35:00.352519+0000","last_clean":"2024-09-16T01:35:00.352519+0000","last_became_active":"2024-09-16T01:35:00.349296+0000","last_became_peered":"2024-09-16T01:35:00.349296+0000","last_unstale":"2024-09-16T01:35:00.352519+0000","last_undegraded":"2024-09-16T01:35:00.352519+0000","last_fullsized":"2024-09-16T01:35:00.352519+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:53:24.648600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338365+0000","last_change":"2024-09-16T01:35:00.338365+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338365+0000","last_undegraded":"2024-09-16T01:35:00.338365+0000","last_fullsized":"2024-09-16T01:35:00.338365+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.349142+0000","last_change":"2024-09-16T01:35:00.346787+0000","last_active":"2024-09-16T01:35:00.349142+0000","last_peered":"2024-09-16T01:35:00.349142+0000","last_clean":"2024-09-16T01:35:00.349142+0000","last_became_active":"2024-09-16T01:35:00.346527+0000","last_became_peered":"2024-09-16T01:35:00.346527+0000","last_unstale":"2024-09-16T01:35:00.349142+0000","last_undegraded":"2024-09-16T01:35:00.349142+0000","last_fullsized":"2024-09-16T01:35:00.349142+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:48:54.665338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338003+0000","last_change":"2024-09-16T01:35:00.338003+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338003+0000","last_undegraded":"2024-09-16T01:35:00.338003+0000","last_fullsized":"2024-09-16T01:35:00.338003+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"31'3","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.362725+0000","last_change":"2024-09-16T01:35:00.352778+0000","last_active":"2024-09-16T01:35:01.362725+0000","last_peered":"2024-09-16T01:35:01.362725+0000","last_clean":"2024-09-16T01:35:01.362725+0000","last_became_active":"2024-09-16T01:35:00.352534+0000","last_became_peered":"2024-09-16T01:35:00.352534+0000","last_unstale":"2024-09-16T01:35:01.362725+0000","last_undegraded":"2024-09-16T01:35:01.362725+0000","last_fullsized":"2024-09-16T01:35:01.362725+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:32:30.768011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.357410+0000","last_change":"2024-09-16T01:35:01.353288+0000","last_active":"2024-09-16T01:35:01.357410+0000","last_peered":"2024-09-16T01:35:01.357410+0000","last_clean":"2024-09-16T01:35:01.357410+0000","last_became_active":"2024-09-16T01:35:01.352428+0000","last_became_peered":"2024-09-16T01:35:01.352428+0000","last_unstale":"2024-09-16T01:35:01.357410+0000","last_undegraded":"2024-09-16T01:35:01.357410+0000","last_fullsized":"2024-09-16T01:35:01.357410+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:48:49.160580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352983+0000","last_change":"2024-09-16T01:35:00.350506+0000","last_active":"2024-09-16T01:35:00.352983+0000","last_peered":"2024-09-16T01:35:00.352983+0000","last_clean":"2024-09-16T01:35:00.352983+0000","last_became_active":"2024-09-16T01:35:00.350215+0000","last_became_peered":"2024-09-16T01:35:00.350215+0000","last_unstale":"2024-09-16T01:35:00.352983+0000","last_undegraded":"2024-09-16T01:35:00.352983+0000","last_fullsized":"2024-09-16T01:35:00.352983+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:43:39.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339530+0000","last_change":"2024-09-16T01:35:00.339530+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339530+0000","last_undegraded":"2024-09-16T01:35:00.339530+0000","last_fullsized":"2024-09-16T01:35:00.339530+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.348029+0000","last_change":"2024-09-16T01:35:00.342977+0000","last_active":"2024-09-16T01:35:00.348029+0000","last_peered":"2024-09-16T01:35:00.348029+0000","last_clean":"2024-09-16T01:35:00.348029+0000","last_became_active":"2024-09-16T01:35:00.342699+0000","last_became_peered":"2024-09-16T01:35:00.342699+0000","last_unstale":"2024-09-16T01:35:00.348029+0000","last_undegraded":"2024-09-16T01:35:00.348029+0000","last_fullsized":"2024-09-16T01:35:00.348029+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:53:54.367989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337148+0000","last_change":"2024-09-16T01:35:00.337148+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337148+0000","last_undegraded":"2024-09-16T01:35:00.337148+0000","last_fullsized":"2024-09-16T01:35:00.337148+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.348456+0000","last_change":"2024-09-16T01:35:00.343581+0000","last_active":"2024-09-16T01:35:00.348456+0000","last_peered":"2024-09-16T01:35:00.348456+0000","last_clean":"2024-09-16T01:35:00.348456+0000","last_became_active":"2024-09-16T01:35:00.343294+0000","last_became_peered":"2024-09-16T01:35:00.343294+0000","last_unstale":"2024-09-16T01:35:00.348456+0000","last_undegraded":"2024-09-16T01:35:00.348456+0000","last_fullsized":"2024-09-16T01:35:00.348456+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:57.654721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.346877+0000","last_change":"2024-09-16T01:35:00.346877+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.346877+0000","last_undegraded":"2024-09-16T01:35:00.346877+0000","last_fullsized":"2024-09-16T01:35:00.346877+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353028+0000","last_change":"2024-09-16T01:35:00.351166+0000","last_active":"2024-09-16T01:35:00.353028+0000","last_peered":"2024-09-16T01:35:00.353028+0000","last_clean":"2024-09-16T01:35:00.353028+0000","last_became_active":"2024-09-16T01:35:00.350881+0000","last_became_peered":"2024-09-16T01:35:00.350881+0000","last_unstale":"2024-09-16T01:35:00.353028+0000","last_undegraded":"2024-09-16T01:35:00.353028+0000","last_fullsized":"2024-09-16T01:35:00.353028+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:54:05.742892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337056+0000","last_change":"2024-09-16T01:35:00.337056+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337056+0000","last_undegraded":"2024-09-16T01:35:00.337056+0000","last_fullsized":"2024-09-16T01:35:00.337056+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.350509+0000","last_change":"2024-09-16T01:35:00.345939+0000","last_active":"2024-09-16T01:35:00.350509+0000","last_peered":"2024-09-16T01:35:00.350509+0000","last_clean":"2024-09-16T01:35:00.350509+0000","last_became_active":"2024-09-16T01:35:00.345653+0000","last_became_peered":"2024-09-16T01:35:00.345653+0000","last_unstale":"2024-09-16T01:35:00.350509+0000","last_undegraded":"2024-09-16T01:35:00.350509+0000","last_fullsized":"2024-09-16T01:35:00.350509+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:57.839735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339311+0000","last_change":"2024-09-16T01:35:00.339311+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339311+0000","last_undegraded":"2024-09-16T01:35:00.339311+0000","last_fullsized":"2024-09-16T01:35:00.339311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352801+0000","last_change":"2024-09-16T01:35:00.350284+0000","last_active":"2024-09-16T01:35:00.352801+0000","last_peered":"2024-09-16T01:35:00.352801+0000","last_clean":"2024-09-16T01:35:00.352801+0000","last_became_active":"2024-09-16T01:35:00.349516+0000","last_became_peered":"2024-09-16T01:35:00.349516+0000","last_unstale":"2024-09-16T01:35:00.352801+0000","last_undegraded":"2024-09-16T01:35:00.352801+0000","last_fullsized":"2024-09-16T01:35:00.352801+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:22:55.631379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.354497+0000","last_change":"2024-09-16T01:35:01.347677+0000","last_active":"2024-09-16T01:35:01.354497+0000","last_peered":"2024-09-16T01:35:01.354497+0000","last_clean":"2024-09-16T01:35:01.354497+0000","last_became_active":"2024-09-16T01:35:01.345167+0000","last_became_peered":"2024-09-16T01:35:01.345167+0000","last_unstale":"2024-09-16T01:35:01.354497+0000","last_undegraded":"2024-09-16T01:35:01.354497+0000","last_fullsized":"2024-09-16T01:35:01.354497+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:34:25.731515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.347434+0000","last_change":"2024-09-16T01:35:00.342181+0000","last_active":"2024-09-16T01:35:00.347434+0000","last_peered":"2024-09-16T01:35:00.347434+0000","last_clean":"2024-09-16T01:35:00.347434+0000","last_became_active":"2024-09-16T01:35:00.341878+0000","last_became_peered":"2024-09-16T01:35:00.341878+0000","last_unstale":"2024-09-16T01:35:00.347434+0000","last_undegraded":"2024-09-16T01:35:00.347434+0000","last_fullsized":"2024-09-16T01:35:00.347434+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:17:19.574295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339159+0000","last_change":"2024-09-16T01:35:00.339159+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339159+0000","last_undegraded":"2024-09-16T01:35:00.339159+0000","last_fullsized":"2024-09-16T01:35:00.339159+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353150+0000","last_change":"2024-09-16T01:35:00.349566+0000","last_active":"2024-09-16T01:35:00.353150+0000","last_peered":"2024-09-16T01:35:00.353150+0000","last_clean":"2024-09-16T01:35:00.353150+0000","last_became_active":"2024-09-16T01:35:00.349243+0000","last_became_peered":"2024-09-16T01:35:00.349243+0000","last_unstale":"2024-09-16T01:35:00.353150+0000","last_undegraded":"2024-09-16T01:35:00.353150+0000","last_fullsized":"2024-09-16T01:35:00.353150+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:43:05.059186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.341010+0000","last_change":"2024-09-16T01:35:00.341010+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.341010+0000","last_undegraded":"2024-09-16T01:35:00.341010+0000","last_fullsized":"2024-09-16T01:35:00.341010+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353118+0000","last_change":"2024-09-16T01:35:00.348167+0000","last_active":"2024-09-16T01:35:00.353118+0000","last_peered":"2024-09-16T01:35:00.353118+0000","last_clean":"2024-09-16T01:35:00.353118+0000","last_became_active":"2024-09-16T01:35:00.347714+0000","last_became_peered":"2024-09-16T01:35:00.347714+0000","last_unstale":"2024-09-16T01:35:00.353118+0000","last_undegraded":"2024-09-16T01:35:00.353118+0000","last_fullsized":"2024-09-16T01:35:00.353118+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:29:35.905712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.342418+0000","last_change":"2024-09-16T01:35:00.342418+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.342418+0000","last_undegraded":"2024-09-16T01:35:00.342418+0000","last_fullsized":"2024-09-16T01:35:00.342418+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.346227+0000","last_change":"2024-09-16T01:35:00.345340+0000","last_active":"2024-09-16T01:35:00.346227+0000","last_peered":"2024-09-16T01:35:00.346227+0000","last_clean":"2024-09-16T01:35:00.346227+0000","last_became_active":"2024-09-16T01:35:00.344987+0000","last_became_peered":"2024-09-16T01:35:00.344987+0000","last_unstale":"2024-09-16T01:35:00.346227+0000","last_undegraded":"2024-09-16T01:35:00.346227+0000","last_fullsized":"2024-09-16T01:35:00.346227+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:02:54.003881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.340705+0000","last_change":"2024-09-16T01:35:00.340705+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.340705+0000","last_undegraded":"2024-09-16T01:35:00.340705+0000","last_fullsized":"2024-09-16T01:35:00.340705+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.354908+0000","last_change":"2024-09-16T01:35:00.349814+0000","last_active":"2024-09-16T01:35:00.354908+0000","last_peered":"2024-09-16T01:35:00.354908+0000","last_clean":"2024-09-16T01:35:00.354908+0000","last_became_active":"2024-09-16T01:35:00.349543+0000","last_became_peered":"2024-09-16T01:35:00.349543+0000","last_unstale":"2024-09-16T01:35:00.354908+0000","last_undegraded":"2024-09-16T01:35:00.354908+0000","last_fullsized":"2024-09-16T01:35:00.354908+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:02:54.003881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.342349+0000","last_change":"2024-09-16T01:35:00.342349+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.342349+0000","last_undegraded":"2024-09-16T01:35:00.342349+0000","last_fullsized":"2024-09-16T01:35:00.342349+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353146+0000","last_change":"2024-09-16T01:35:00.348335+0000","last_active":"2024-09-16T01:35:00.353146+0000","last_peered":"2024-09-16T01:35:00.353146+0000","last_clean":"2024-09-16T01:35:00.353146+0000","last_became_active":"2024-09-16T01:35:00.347099+0000","last_became_peered":"2024-09-16T01:35:00.347099+0000","last_unstale":"2024-09-16T01:35:00.353146+0000","last_undegraded":"2024-09-16T01:35:00.353146+0000","last_fullsized":"2024-09-16T01:35:00.353146+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:57:35.138402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.346143+0000","last_change":"2024-09-16T01:35:00.346143+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.346143+0000","last_undegraded":"2024-09-16T01:35:00.346143+0000","last_fullsized":"2024-09-16T01:35:00.346143+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.337675+0000","last_change":"2024-09-16T01:35:00.352839+0000","last_active":"2024-09-16T01:35:01.337675+0000","last_peered":"2024-09-16T01:35:01.337675+0000","last_clean":"2024-09-16T01:35:01.337675+0000","last_became_active":"2024-09-16T01:35:00.352568+0000","last_became_peered":"2024-09-16T01:35:00.352568+0000","last_unstale":"2024-09-16T01:35:01.337675+0000","last_undegraded":"2024-09-16T01:35:01.337675+0000","last_fullsized":"2024-09-16T01:35:01.337675+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:30:51.141162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.342110+0000","last_change":"2024-09-16T01:35:00.342110+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.342110+0000","last_undegraded":"2024-09-16T01:35:00.342110+0000","last_fullsized":"2024-09-16T01:35:00.342110+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.357904+0000","last_change":"2024-09-16T01:35:00.355682+0000","last_active":"2024-09-16T01:35:00.357904+0000","last_peered":"2024-09-16T01:35:00.357904+0000","last_clean":"2024-09-16T01:35:00.357904+0000","last_became_active":"2024-09-16T01:35:00.355347+0000","last_became_peered":"2024-09-16T01:35:00.355347+0000","last_unstale":"2024-09-16T01:35:00.357904+0000","last_undegraded":"2024-09-16T01:35:00.357904+0000","last_fullsized":"2024-09-16T01:35:00.357904+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:07.992492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.359762+0000","last_change":"2024-09-16T01:35:01.355640+0000","last_active":"2024-09-16T01:35:01.359762+0000","last_peered":"2024-09-16T01:35:01.359762+0000","last_clean":"2024-09-16T01:35:01.359762+0000","last_became_active":"2024-09-16T01:35:01.355364+0000","last_became_peered":"2024-09-16T01:35:01.355364+0000","last_unstale":"2024-09-16T01:35:01.359762+0000","last_undegraded":"2024-09-16T01:35:01.359762+0000","last_fullsized":"2024-09-16T01:35:01.359762+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:26:10.193445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.346212+0000","last_change":"2024-09-16T01:35:00.343346+0000","last_active":"2024-09-16T01:35:00.346212+0000","last_peered":"2024-09-16T01:35:00.346212+0000","last_clean":"2024-09-16T01:35:00.346212+0000","last_became_active":"2024-09-16T01:35:00.343049+0000","last_became_peered":"2024-09-16T01:35:00.343049+0000","last_unstale":"2024-09-16T01:35:00.346212+0000","last_undegraded":"2024-09-16T01:35:00.346212+0000","last_fullsized":"2024-09-16T01:35:00.346212+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:00.117677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.335782+0000","last_change":"2024-09-16T01:35:00.335782+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.335782+0000","last_undegraded":"2024-09-16T01:35:00.335782+0000","last_fullsized":"2024-09-16T01:35:00.335782+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352715+0000","last_change":"2024-09-16T01:35:00.349236+0000","last_active":"2024-09-16T01:35:00.352715+0000","last_peered":"2024-09-16T01:35:00.352715+0000","last_clean":"2024-09-16T01:35:00.352715+0000","last_became_active":"2024-09-16T01:35:00.348947+0000","last_became_peered":"2024-09-16T01:35:00.348947+0000","last_unstale":"2024-09-16T01:35:00.352715+0000","last_undegraded":"2024-09-16T01:35:00.352715+0000","last_fullsized":"2024-09-16T01:35:00.352715+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:18:05.552429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.336999+0000","last_change":"2024-09-16T01:35:00.336999+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.336999+0000","last_undegraded":"2024-09-16T01:35:00.336999+0000","last_fullsized":"2024-09-16T01:35:00.336999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.357770+0000","last_change":"2024-09-16T01:35:00.355723+0000","last_active":"2024-09-16T01:35:00.357770+0000","last_peered":"2024-09-16T01:35:00.357770+0000","last_clean":"2024-09-16T01:35:00.357770+0000","last_became_active":"2024-09-16T01:35:00.355425+0000","last_became_peered":"2024-09-16T01:35:00.355425+0000","last_unstale":"2024-09-16T01:35:00.357770+0000","last_undegraded":"2024-09-16T01:35:00.357770+0000","last_fullsized":"2024-09-16T01:35:00.357770+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:00.732201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.335633+0000","last_change":"2024-09-16T01:35:00.335633+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.335633+0000","last_undegraded":"2024-09-16T01:35:00.335633+0000","last_fullsized":"2024-09-16T01:35:00.335633+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.357916+0000","last_change":"2024-09-16T01:35:00.355687+0000","last_active":"2024-09-16T01:35:00.357916+0000","last_peered":"2024-09-16T01:35:00.357916+0000","last_clean":"2024-09-16T01:35:00.357916+0000","last_became_active":"2024-09-16T01:35:00.355357+0000","last_became_peered":"2024-09-16T01:35:00.355357+0000","last_unstale":"2024-09-16T01:35:00.357916+0000","last_undegraded":"2024-09-16T01:35:00.357916+0000","last_fullsized":"2024-09-16T01:35:00.357916+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:30:55.369673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337712+0000","last_change":"2024-09-16T01:35:00.337712+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337712+0000","last_undegraded":"2024-09-16T01:35:00.337712+0000","last_fullsized":"2024-09-16T01:35:00.337712+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353170+0000","last_change":"2024-09-16T01:35:00.351123+0000","last_active":"2024-09-16T01:35:00.353170+0000","last_peered":"2024-09-16T01:35:00.353170+0000","last_clean":"2024-09-16T01:35:00.353170+0000","last_became_active":"2024-09-16T01:35:00.350803+0000","last_became_peered":"2024-09-16T01:35:00.350803+0000","last_unstale":"2024-09-16T01:35:00.353170+0000","last_undegraded":"2024-09-16T01:35:00.353170+0000","last_fullsized":"2024-09-16T01:35:00.353170+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:51:21.665512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339066+0000","last_change":"2024-09-16T01:35:00.339066+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339066+0000","last_undegraded":"2024-09-16T01:35:00.339066+0000","last_fullsized":"2024-09-16T01:35:00.339066+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353039+0000","last_change":"2024-09-16T01:35:00.351535+0000","last_active":"2024-09-16T01:35:00.353039+0000","last_peered":"2024-09-16T01:35:00.353039+0000","last_clean":"2024-09-16T01:35:00.353039+0000","last_became_active":"2024-09-16T01:35:00.351219+0000","last_became_peered":"2024-09-16T01:35:00.351219+0000","last_unstale":"2024-09-16T01:35:00.353039+0000","last_undegraded":"2024-09-16T01:35:00.353039+0000","last_fullsized":"2024-09-16T01:35:00.353039+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:04.599770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.342993+0000","last_change":"2024-09-16T01:35:00.342993+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.342993+0000","last_undegraded":"2024-09-16T01:35:00.342993+0000","last_fullsized":"2024-09-16T01:35:00.342993+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352480+0000","last_change":"2024-09-16T01:35:00.344871+0000","last_active":"2024-09-16T01:35:00.352480+0000","last_peered":"2024-09-16T01:35:00.352480+0000","last_clean":"2024-09-16T01:35:00.352480+0000","last_became_active":"2024-09-16T01:35:00.344582+0000","last_became_peered":"2024-09-16T01:35:00.344582+0000","last_unstale":"2024-09-16T01:35:00.352480+0000","last_undegraded":"2024-09-16T01:35:00.352480+0000","last_fullsized":"2024-09-16T01:35:00.352480+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:00.117677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":185,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.348767+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:35:00.348767+0000","last_peered":"2024-09-16T01:35:00.348767+0000","last_clean":"2024-09-16T01:35:00.348767+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:35:00.348767+0000","last_undegraded":"2024-09-16T01:35:00.348767+0000","last_fullsized":"2024-09-16T01:35:00.348767+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509455,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":331776,"data_stored":123852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542158,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27920,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343920,"statfs":{"total":96636764160,"available":96608174080,"internally_reserved":0,"allocated":913408,"data_stored":707953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542158,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":117585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542158,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":117585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542157,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":117585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542158,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":117585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542157,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":117585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542157,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":925696,"data_stored":714220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:35:04.625 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:35:04.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.760+0000 7f21156ac640 1 Processor -- start 2024-09-16T01:35:04.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.760+0000 7f21156ac640 1 -- start start 2024-09-16T01:35:04.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.761+0000 7f21156ac640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21100d3310 0x7f21100d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:04.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.761+0000 7f21156ac640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f21100d1650 0x7f21100b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:04.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.761+0000 7f21156ac640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f211015ea40 con 0x7f211015e660 2024-09-16T01:35:04.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.761+0000 7f21156ac640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f21100d1ac0 con 0x7f21100d1650 2024-09-16T01:35:04.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.761+0000 7f21156ac640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f21100d3870 con 0x7f21100d3310 2024-09-16T01:35:04.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.761+0000 7f210effd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f211015e660 0x7f21100cf730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35980/0 (socket says 172.21.15.88:35980) 2024-09-16T01:35:04.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.761+0000 7f210effd640 1 -- 172.21.15.88:0/1046984475 learned_addr learned my addr 172.21.15.88:0/1046984475 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:04.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.761+0000 7f210dffb640 1 -- 172.21.15.88:0/1046984475 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1179999294 0 0) 0x7f211015ea40 con 0x7f211015e660 2024-09-16T01:35:04.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.761+0000 7f210dffb640 1 -- 172.21.15.88:0/1046984475 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20f8003520 con 0x7f211015e660 2024-09-16T01:35:04.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.762+0000 7f210e7fc640 1 --2- 172.21.15.88:0/1046984475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21100d3310 0x7f21100d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:04.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.762+0000 7f210f7fe640 1 --2- 172.21.15.88:0/1046984475 >> v2:172.21.15.154:3300/0 conn(0x7f21100d1650 0x7f21100b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:04.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.762+0000 7f210dffb640 1 -- 172.21.15.88:0/1046984475 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 5924642 0 0) 0x7f20f8003520 con 0x7f211015e660 2024-09-16T01:35:04.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.762+0000 7f210dffb640 1 -- 172.21.15.88:0/1046984475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21100d3310 msgr2=0x7f21100d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:04.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.762+0000 7f210dffb640 1 --2- 172.21.15.88:0/1046984475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21100d3310 0x7f21100d1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.762+0000 7f210dffb640 1 -- 172.21.15.88:0/1046984475 >> v2:172.21.15.154:3300/0 conn(0x7f21100d1650 msgr2=0x7f21100b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:04.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.762+0000 7f210dffb640 1 --2- 172.21.15.88:0/1046984475 >> v2:172.21.15.154:3300/0 conn(0x7f21100d1650 0x7f21100b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.762+0000 7f210dffb640 1 -- 172.21.15.88:0/1046984475 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21100d3b50 con 0x7f211015e660 2024-09-16T01:35:04.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.762+0000 7f210dffb640 1 -- 172.21.15.88:0/1046984475 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f21000019f0 con 0x7f211015e660 2024-09-16T01:35:04.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.762+0000 7f210e7fc640 1 --2- 172.21.15.88:0/1046984475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21100d3310 0x7f21100d1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:04.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.762+0000 7f210dffb640 1 -- 172.21.15.88:0/1046984475 >> v1:172.21.15.88:6789/0 conn(0x7f211015e660 legacy=0x7f21100cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:04.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.762+0000 7f210dffb640 1 --2- 172.21.15.88:0/1046984475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20f80039b0 0x7f20f8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:04.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.762+0000 7f210dffb640 1 --2- 172.21.15.88:0/1046984475 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f8004800 0x7f20f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:04.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.762+0000 7f210dffb640 1 --2- 172.21.15.88:0/1046984475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21100d3310 0x7f20f80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:04.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.762+0000 7f210dffb640 1 -- 172.21.15.88:0/1046984475 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f21100d1ac0 con 0x7f20f8004800 2024-09-16T01:35:04.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.762+0000 7f210dffb640 1 -- 172.21.15.88:0/1046984475 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f21100d3870 con 0x7f20f80039b0 2024-09-16T01:35:04.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.762+0000 7f210dffb640 1 -- 172.21.15.88:0/1046984475 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f20f800ba70 con 0x7f21100d3310 2024-09-16T01:35:04.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.762+0000 7f210e7fc640 1 --2- 172.21.15.88:0/1046984475 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f8004800 0x7f20f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:04.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.762+0000 7f210effd640 1 --2- 172.21.15.88:0/1046984475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20f80039b0 0x7f20f8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:04.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.762+0000 7f210f7fe640 1 --2- 172.21.15.88:0/1046984475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21100d3310 0x7f20f80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:04.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.762+0000 7f210e7fc640 1 -- 172.21.15.88:0/1046984475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21100d3310 msgr2=0x7f20f80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:04.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.762+0000 7f210e7fc640 1 --2- 172.21.15.88:0/1046984475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21100d3310 0x7f20f80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.762+0000 7f210e7fc640 1 -- 172.21.15.88:0/1046984475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20f80039b0 msgr2=0x7f20f8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:04.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.762+0000 7f210e7fc640 1 --2- 172.21.15.88:0/1046984475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20f80039b0 0x7f20f8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.762+0000 7f210e7fc640 1 -- 172.21.15.88:0/1046984475 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20f8003520 con 0x7f20f8004800 2024-09-16T01:35:04.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.763+0000 7f210e7fc640 1 --2- 172.21.15.88:0/1046984475 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f8004800 0x7f20f8004c80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f21040098a0 tx=0x7f20f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2d8ba1539e4f54 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:04.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.763+0000 7f210dffb640 1 -- 172.21.15.88:0/1046984475 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f210400f040 con 0x7f20f8004800 2024-09-16T01:35:04.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.763+0000 7f210dffb640 1 -- 172.21.15.88:0/1046984475 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f210402ace0 con 0x7f20f8004800 2024-09-16T01:35:04.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.763+0000 7f210dffb640 1 -- 172.21.15.88:0/1046984475 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2104004250 con 0x7f20f8004800 2024-09-16T01:35:04.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.763+0000 7f21156ac640 1 -- 172.21.15.88:0/1046984475 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f8004800 msgr2=0x7f20f8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:04.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.763+0000 7f21156ac640 1 --2- 172.21.15.88:0/1046984475 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f8004800 0x7f20f8004c80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f21040098a0 tx=0x7f20f8005830 comp rx=0 tx=0).stop 2024-09-16T01:35:04.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.763+0000 7f210f7fe640 1 -- 172.21.15.88:0/1046984475 reap_dead start 2024-09-16T01:35:04.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.764+0000 7f21156ac640 1 -- 172.21.15.88:0/1046984475 shutdown_connections 2024-09-16T01:35:04.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.764+0000 7f21156ac640 1 -- 172.21.15.88:0/1046984475 >> 172.21.15.88:0/1046984475 conn(0x7f21100b6410 msgr2=0x7f21100b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:04.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.764+0000 7f21156ac640 1 -- 172.21.15.88:0/1046984475 shutdown_connections 2024-09-16T01:35:04.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.764+0000 7f21156ac640 1 -- 172.21.15.88:0/1046984475 wait complete. 2024-09-16T01:35:04.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.765+0000 7f21156ac640 1 Processor -- start 2024-09-16T01:35:04.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.765+0000 7f21156ac640 1 -- start start 2024-09-16T01:35:04.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.766+0000 7f21156ac640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f8004800 0x7f21100ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:04.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.766+0000 7f21156ac640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21101571b0 0x7f2110151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:04.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.766+0000 7f21156ac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21100ce9b0 0x7f21100cedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:04.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.766+0000 7f21156ac640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f21100d43f0 con 0x7f20f8004800 2024-09-16T01:35:04.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.766+0000 7f21156ac640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f21100d4110 con 0x7f21100ce9b0 2024-09-16T01:35:04.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.766+0000 7f21156ac640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f21100d1110 con 0x7f21101571b0 2024-09-16T01:35:04.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.766+0000 7f210effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f8004800 0x7f21100ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:04.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.766+0000 7f210effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f8004800 0x7f21100ce1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36494/0 (socket says 172.21.15.88:36494) 2024-09-16T01:35:04.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.766+0000 7f210effd640 1 -- 172.21.15.88:0/1774708655 learned_addr learned my addr 172.21.15.88:0/1774708655 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:04.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.766+0000 7f210e7fc640 1 --2- 172.21.15.88:0/1774708655 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21101571b0 0x7f2110151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:04.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.766+0000 7f210f7fe640 1 --2- 172.21.15.88:0/1774708655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21100ce9b0 0x7f21100cedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:04.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.766+0000 7f210effd640 1 -- 172.21.15.88:0/1774708655 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21101571b0 msgr2=0x7f2110151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:04.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.766+0000 7f210effd640 1 --2- 172.21.15.88:0/1774708655 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21101571b0 0x7f2110151a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.766+0000 7f210effd640 1 -- 172.21.15.88:0/1774708655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21100ce9b0 msgr2=0x7f21100cedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:04.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.766+0000 7f210effd640 1 --2- 172.21.15.88:0/1774708655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21100ce9b0 0x7f21100cedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.766+0000 7f210effd640 1 -- 172.21.15.88:0/1774708655 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21101579b0 con 0x7f20f8004800 2024-09-16T01:35:04.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.766+0000 7f210f7fe640 1 --2- 172.21.15.88:0/1774708655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21100ce9b0 0x7f21100cedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:04.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.766+0000 7f210effd640 1 --2- 172.21.15.88:0/1774708655 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f8004800 0x7f21100ce1c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2100007a20 tx=0x7f210000a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:04.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.766+0000 7f20f77fe640 1 -- 172.21.15.88:0/1774708655 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2100001a30 con 0x7f20f8004800 2024-09-16T01:35:04.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.766+0000 7f20f77fe640 1 -- 172.21.15.88:0/1774708655 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2100018070 con 0x7f20f8004800 2024-09-16T01:35:04.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.766+0000 7f21156ac640 1 -- 172.21.15.88:0/1774708655 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2110157c90 con 0x7f20f8004800 2024-09-16T01:35:04.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.766+0000 7f20f77fe640 1 -- 172.21.15.88:0/1774708655 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2100013630 con 0x7f20f8004800 2024-09-16T01:35:04.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.766+0000 7f21156ac640 1 -- 172.21.15.88:0/1774708655 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2110158250 con 0x7f20f8004800 2024-09-16T01:35:04.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.767+0000 7f21156ac640 1 -- 172.21.15.88:0/1774708655 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21100cf730 con 0x7f20f8004800 2024-09-16T01:35:04.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.768+0000 7f20f77fe640 1 -- 172.21.15.88:0/1774708655 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f21000137d0 con 0x7f20f8004800 2024-09-16T01:35:04.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.768+0000 7f20f77fe640 1 --2- 172.21.15.88:0/1774708655 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f20fc06c2b0 0x7f20fc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:04.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.768+0000 7f20f77fe640 1 -- 172.21.15.88:0/1774708655 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f210008bc00 con 0x7f20f8004800 2024-09-16T01:35:04.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.768+0000 7f210e7fc640 1 --2- 172.21.15.88:0/1774708655 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f20fc06c2b0 0x7f20fc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:04.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.769+0000 7f210e7fc640 1 --2- 172.21.15.88:0/1774708655 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f20fc06c2b0 0x7f20fc06e770 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2104033000 tx=0x7f210402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:04.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.773+0000 7f20f77fe640 1 -- 172.21.15.88:0/1774708655 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f210005a370 con 0x7f20f8004800 2024-09-16T01:35:04.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.936+0000 7f21156ac640 1 -- 172.21.15.88:0/1774708655 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f21100cf330 con 0x7f20fc06c2b0 2024-09-16T01:35:04.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.941+0000 7f20f77fe640 1 -- 172.21.15.88:0/1774708655 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+247449 (secure 0 0 0) 0x7f21100cf330 con 0x7f20fc06c2b0 2024-09-16T01:35:04.945 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:35:04.946 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:35:04.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.944+0000 7f21156ac640 1 -- 172.21.15.88:0/1774708655 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f20fc06c2b0 msgr2=0x7f20fc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:04.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.944+0000 7f21156ac640 1 --2- 172.21.15.88:0/1774708655 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f20fc06c2b0 0x7f20fc06e770 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2104033000 tx=0x7f210402e000 comp rx=0 tx=0).stop 2024-09-16T01:35:04.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.944+0000 7f21156ac640 1 -- 172.21.15.88:0/1774708655 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f8004800 msgr2=0x7f21100ce1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:04.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.944+0000 7f21156ac640 1 --2- 172.21.15.88:0/1774708655 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f8004800 0x7f21100ce1c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2100007a20 tx=0x7f210000a730 comp rx=0 tx=0).stop 2024-09-16T01:35:04.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.944+0000 7f21156ac640 1 -- 172.21.15.88:0/1774708655 shutdown_connections 2024-09-16T01:35:04.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.944+0000 7f21156ac640 1 --2- 172.21.15.88:0/1774708655 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f20fc06c2b0 0x7f20fc06e770 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.944+0000 7f21156ac640 1 --2- 172.21.15.88:0/1774708655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21100ce9b0 0x7f21100cedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.944+0000 7f21156ac640 1 --2- 172.21.15.88:0/1774708655 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f21101571b0 0x7f2110151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.944+0000 7f21156ac640 1 --2- 172.21.15.88:0/1774708655 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f8004800 0x7f21100ce1c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:04.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.944+0000 7f21156ac640 1 -- 172.21.15.88:0/1774708655 >> 172.21.15.88:0/1774708655 conn(0x7f21100b6410 msgr2=0x7f21100b5820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:04.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.944+0000 7f21156ac640 1 -- 172.21.15.88:0/1774708655 shutdown_connections 2024-09-16T01:35:04.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:04.944+0000 7f21156ac640 1 -- 172.21.15.88:0/1774708655 wait complete. 2024-09-16T01:35:04.956 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-09-16T01:35:04.524493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590458,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":163,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":28,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219912,"kb_used_data":3680,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754808,"statfs":{"total":773094113280,"available":772868923392,"internally_reserved":0,"allocated":3768320,"data_stored":2133950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.198029"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337649+0000","last_change":"2024-09-16T01:35:00.337649+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337649+0000","last_undegraded":"2024-09-16T01:35:00.337649+0000","last_fullsized":"2024-09-16T01:35:00.337649+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339909+0000","last_change":"2024-09-16T01:35:00.339909+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339909+0000","last_undegraded":"2024-09-16T01:35:00.339909+0000","last_fullsized":"2024-09-16T01:35:00.339909+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.357429+0000","last_change":"2024-09-16T01:35:01.353507+0000","last_active":"2024-09-16T01:35:01.357429+0000","last_peered":"2024-09-16T01:35:01.357429+0000","last_clean":"2024-09-16T01:35:01.357429+0000","last_became_active":"2024-09-16T01:35:01.352327+0000","last_became_peered":"2024-09-16T01:35:01.352327+0000","last_unstale":"2024-09-16T01:35:01.357429+0000","last_undegraded":"2024-09-16T01:35:01.357429+0000","last_fullsized":"2024-09-16T01:35:01.357429+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:55.644944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.340201+0000","last_change":"2024-09-16T01:35:00.340201+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.340201+0000","last_undegraded":"2024-09-16T01:35:00.340201+0000","last_fullsized":"2024-09-16T01:35:00.340201+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.341590+0000","last_change":"2024-09-16T01:35:00.341590+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.341590+0000","last_undegraded":"2024-09-16T01:35:00.341590+0000","last_fullsized":"2024-09-16T01:35:00.341590+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338751+0000","last_change":"2024-09-16T01:35:00.338751+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338751+0000","last_undegraded":"2024-09-16T01:35:00.338751+0000","last_fullsized":"2024-09-16T01:35:00.338751+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.345760+0000","last_change":"2024-09-16T01:35:00.345760+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.345760+0000","last_undegraded":"2024-09-16T01:35:00.345760+0000","last_fullsized":"2024-09-16T01:35:00.345760+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.347744+0000","last_change":"2024-09-16T01:35:00.347744+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.347744+0000","last_undegraded":"2024-09-16T01:35:00.347744+0000","last_fullsized":"2024-09-16T01:35:00.347744+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337574+0000","last_change":"2024-09-16T01:35:00.337574+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337574+0000","last_undegraded":"2024-09-16T01:35:00.337574+0000","last_fullsized":"2024-09-16T01:35:00.337574+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339626+0000","last_change":"2024-09-16T01:35:00.339626+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339626+0000","last_undegraded":"2024-09-16T01:35:00.339626+0000","last_fullsized":"2024-09-16T01:35:00.339626+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.344537+0000","last_change":"2024-09-16T01:35:00.344537+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.344537+0000","last_undegraded":"2024-09-16T01:35:00.344537+0000","last_fullsized":"2024-09-16T01:35:00.344537+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.354548+0000","last_change":"2024-09-16T01:35:01.349955+0000","last_active":"2024-09-16T01:35:01.354548+0000","last_peered":"2024-09-16T01:35:01.354548+0000","last_clean":"2024-09-16T01:35:01.354548+0000","last_became_active":"2024-09-16T01:35:01.349689+0000","last_became_peered":"2024-09-16T01:35:01.349689+0000","last_unstale":"2024-09-16T01:35:01.354548+0000","last_undegraded":"2024-09-16T01:35:01.354548+0000","last_fullsized":"2024-09-16T01:35:01.354548+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:02:10.872199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.341163+0000","last_change":"2024-09-16T01:35:00.341163+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.341163+0000","last_undegraded":"2024-09-16T01:35:00.341163+0000","last_fullsized":"2024-09-16T01:35:00.341163+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337577+0000","last_change":"2024-09-16T01:35:00.337577+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337577+0000","last_undegraded":"2024-09-16T01:35:00.337577+0000","last_fullsized":"2024-09-16T01:35:00.337577+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.353724+0000","last_change":"2024-09-16T01:35:01.344630+0000","last_active":"2024-09-16T01:35:01.353724+0000","last_peered":"2024-09-16T01:35:01.353724+0000","last_clean":"2024-09-16T01:35:01.353724+0000","last_became_active":"2024-09-16T01:35:01.344340+0000","last_became_peered":"2024-09-16T01:35:01.344340+0000","last_unstale":"2024-09-16T01:35:01.353724+0000","last_undegraded":"2024-09-16T01:35:01.353724+0000","last_fullsized":"2024-09-16T01:35:01.353724+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:06:54.456413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.342533+0000","last_change":"2024-09-16T01:35:00.342533+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.342533+0000","last_undegraded":"2024-09-16T01:35:00.342533+0000","last_fullsized":"2024-09-16T01:35:00.342533+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.343962+0000","last_change":"2024-09-16T01:35:00.343962+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.343962+0000","last_undegraded":"2024-09-16T01:35:00.343962+0000","last_fullsized":"2024-09-16T01:35:00.343962+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339533+0000","last_change":"2024-09-16T01:35:00.339533+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339533+0000","last_undegraded":"2024-09-16T01:35:00.339533+0000","last_fullsized":"2024-09-16T01:35:00.339533+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337206+0000","last_change":"2024-09-16T01:35:00.337206+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337206+0000","last_undegraded":"2024-09-16T01:35:00.337206+0000","last_fullsized":"2024-09-16T01:35:00.337206+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.340512+0000","last_change":"2024-09-16T01:35:00.340512+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.340512+0000","last_undegraded":"2024-09-16T01:35:00.340512+0000","last_fullsized":"2024-09-16T01:35:00.340512+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339909+0000","last_change":"2024-09-16T01:35:00.339909+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339909+0000","last_undegraded":"2024-09-16T01:35:00.339909+0000","last_fullsized":"2024-09-16T01:35:00.339909+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338369+0000","last_change":"2024-09-16T01:35:00.338369+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338369+0000","last_undegraded":"2024-09-16T01:35:00.338369+0000","last_fullsized":"2024-09-16T01:35:00.338369+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.335686+0000","last_change":"2024-09-16T01:35:00.335686+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.335686+0000","last_undegraded":"2024-09-16T01:35:00.335686+0000","last_fullsized":"2024-09-16T01:35:00.335686+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.353084+0000","last_change":"2024-09-16T01:35:01.344447+0000","last_active":"2024-09-16T01:35:01.353084+0000","last_peered":"2024-09-16T01:35:01.353084+0000","last_clean":"2024-09-16T01:35:01.353084+0000","last_became_active":"2024-09-16T01:35:01.344135+0000","last_became_peered":"2024-09-16T01:35:01.344135+0000","last_unstale":"2024-09-16T01:35:01.353084+0000","last_undegraded":"2024-09-16T01:35:01.353084+0000","last_fullsized":"2024-09-16T01:35:01.353084+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:44:42.157718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338640+0000","last_change":"2024-09-16T01:35:00.338640+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338640+0000","last_undegraded":"2024-09-16T01:35:00.338640+0000","last_fullsized":"2024-09-16T01:35:00.338640+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339123+0000","last_change":"2024-09-16T01:35:00.339123+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339123+0000","last_undegraded":"2024-09-16T01:35:00.339123+0000","last_fullsized":"2024-09-16T01:35:00.339123+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337376+0000","last_change":"2024-09-16T01:35:00.337376+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337376+0000","last_undegraded":"2024-09-16T01:35:00.337376+0000","last_fullsized":"2024-09-16T01:35:00.337376+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.352090+0000","last_change":"2024-09-16T01:35:01.343532+0000","last_active":"2024-09-16T01:35:01.352090+0000","last_peered":"2024-09-16T01:35:01.352090+0000","last_clean":"2024-09-16T01:35:01.352090+0000","last_became_active":"2024-09-16T01:35:01.342495+0000","last_became_peered":"2024-09-16T01:35:01.342495+0000","last_unstale":"2024-09-16T01:35:01.352090+0000","last_undegraded":"2024-09-16T01:35:01.352090+0000","last_fullsized":"2024-09-16T01:35:01.352090+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:32:33.652857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339456+0000","last_change":"2024-09-16T01:35:00.339456+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339456+0000","last_undegraded":"2024-09-16T01:35:00.339456+0000","last_fullsized":"2024-09-16T01:35:00.339456+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.343049+0000","last_change":"2024-09-16T01:35:00.343049+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.343049+0000","last_undegraded":"2024-09-16T01:35:00.343049+0000","last_fullsized":"2024-09-16T01:35:00.343049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338353+0000","last_change":"2024-09-16T01:35:00.338353+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338353+0000","last_undegraded":"2024-09-16T01:35:00.338353+0000","last_fullsized":"2024-09-16T01:35:00.338353+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.347252+0000","last_change":"2024-09-16T01:35:00.347252+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.347252+0000","last_undegraded":"2024-09-16T01:35:00.347252+0000","last_fullsized":"2024-09-16T01:35:00.347252+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.335647+0000","last_change":"2024-09-16T01:35:00.335647+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.335647+0000","last_undegraded":"2024-09-16T01:35:00.335647+0000","last_fullsized":"2024-09-16T01:35:00.335647+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.358487+0000","last_change":"2024-09-16T01:35:00.355732+0000","last_active":"2024-09-16T01:35:00.358487+0000","last_peered":"2024-09-16T01:35:00.358487+0000","last_clean":"2024-09-16T01:35:00.358487+0000","last_became_active":"2024-09-16T01:35:00.355431+0000","last_became_peered":"2024-09-16T01:35:00.355431+0000","last_unstale":"2024-09-16T01:35:00.358487+0000","last_undegraded":"2024-09-16T01:35:00.358487+0000","last_fullsized":"2024-09-16T01:35:00.358487+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:48:54.665338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337125+0000","last_change":"2024-09-16T01:35:00.337125+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337125+0000","last_undegraded":"2024-09-16T01:35:00.337125+0000","last_fullsized":"2024-09-16T01:35:00.337125+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352461+0000","last_change":"2024-09-16T01:35:00.348929+0000","last_active":"2024-09-16T01:35:00.352461+0000","last_peered":"2024-09-16T01:35:00.352461+0000","last_clean":"2024-09-16T01:35:00.352461+0000","last_became_active":"2024-09-16T01:35:00.348651+0000","last_became_peered":"2024-09-16T01:35:00.348651+0000","last_unstale":"2024-09-16T01:35:00.352461+0000","last_undegraded":"2024-09-16T01:35:00.352461+0000","last_fullsized":"2024-09-16T01:35:00.352461+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:52:02.746213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339423+0000","last_change":"2024-09-16T01:35:00.339423+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339423+0000","last_undegraded":"2024-09-16T01:35:00.339423+0000","last_fullsized":"2024-09-16T01:35:00.339423+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.346035+0000","last_change":"2024-09-16T01:35:00.344907+0000","last_active":"2024-09-16T01:35:00.346035+0000","last_peered":"2024-09-16T01:35:00.346035+0000","last_clean":"2024-09-16T01:35:00.346035+0000","last_became_active":"2024-09-16T01:35:00.344634+0000","last_became_peered":"2024-09-16T01:35:00.344634+0000","last_unstale":"2024-09-16T01:35:00.346035+0000","last_undegraded":"2024-09-16T01:35:00.346035+0000","last_fullsized":"2024-09-16T01:35:00.346035+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:00.732201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.336922+0000","last_change":"2024-09-16T01:35:00.336922+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.336922+0000","last_undegraded":"2024-09-16T01:35:00.336922+0000","last_fullsized":"2024-09-16T01:35:00.336922+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352455+0000","last_change":"2024-09-16T01:35:00.348009+0000","last_active":"2024-09-16T01:35:00.352455+0000","last_peered":"2024-09-16T01:35:00.352455+0000","last_clean":"2024-09-16T01:35:00.352455+0000","last_became_active":"2024-09-16T01:35:00.347736+0000","last_became_peered":"2024-09-16T01:35:00.347736+0000","last_unstale":"2024-09-16T01:35:00.352455+0000","last_undegraded":"2024-09-16T01:35:00.352455+0000","last_fullsized":"2024-09-16T01:35:00.352455+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:44.087710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.342570+0000","last_change":"2024-09-16T01:35:00.342570+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.342570+0000","last_undegraded":"2024-09-16T01:35:00.342570+0000","last_fullsized":"2024-09-16T01:35:00.342570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353110+0000","last_change":"2024-09-16T01:35:00.348379+0000","last_active":"2024-09-16T01:35:00.353110+0000","last_peered":"2024-09-16T01:35:00.353110+0000","last_clean":"2024-09-16T01:35:00.353110+0000","last_became_active":"2024-09-16T01:35:00.348008+0000","last_became_peered":"2024-09-16T01:35:00.348008+0000","last_unstale":"2024-09-16T01:35:00.353110+0000","last_undegraded":"2024-09-16T01:35:00.353110+0000","last_fullsized":"2024-09-16T01:35:00.353110+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:56:53.528942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.354400+0000","last_change":"2024-09-16T01:35:01.346165+0000","last_active":"2024-09-16T01:35:01.354400+0000","last_peered":"2024-09-16T01:35:01.354400+0000","last_clean":"2024-09-16T01:35:01.354400+0000","last_became_active":"2024-09-16T01:35:01.344479+0000","last_became_peered":"2024-09-16T01:35:01.344479+0000","last_unstale":"2024-09-16T01:35:01.354400+0000","last_undegraded":"2024-09-16T01:35:01.354400+0000","last_fullsized":"2024-09-16T01:35:01.354400+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:58:00.280748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353081+0000","last_change":"2024-09-16T01:35:00.347395+0000","last_active":"2024-09-16T01:35:00.353081+0000","last_peered":"2024-09-16T01:35:00.353081+0000","last_clean":"2024-09-16T01:35:00.353081+0000","last_became_active":"2024-09-16T01:35:00.346752+0000","last_became_peered":"2024-09-16T01:35:00.346752+0000","last_unstale":"2024-09-16T01:35:00.353081+0000","last_undegraded":"2024-09-16T01:35:00.353081+0000","last_fullsized":"2024-09-16T01:35:00.353081+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:51.973054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338296+0000","last_change":"2024-09-16T01:35:00.338296+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338296+0000","last_undegraded":"2024-09-16T01:35:00.338296+0000","last_fullsized":"2024-09-16T01:35:00.338296+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352488+0000","last_change":"2024-09-16T01:35:00.348232+0000","last_active":"2024-09-16T01:35:00.352488+0000","last_peered":"2024-09-16T01:35:00.352488+0000","last_clean":"2024-09-16T01:35:00.352488+0000","last_became_active":"2024-09-16T01:35:00.347976+0000","last_became_peered":"2024-09-16T01:35:00.347976+0000","last_unstale":"2024-09-16T01:35:00.352488+0000","last_undegraded":"2024-09-16T01:35:00.352488+0000","last_fullsized":"2024-09-16T01:35:00.352488+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:49:23.770293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.336753+0000","last_change":"2024-09-16T01:35:00.336753+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.336753+0000","last_undegraded":"2024-09-16T01:35:00.336753+0000","last_fullsized":"2024-09-16T01:35:00.336753+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.347485+0000","last_change":"2024-09-16T01:35:00.342334+0000","last_active":"2024-09-16T01:35:00.347485+0000","last_peered":"2024-09-16T01:35:00.347485+0000","last_clean":"2024-09-16T01:35:00.347485+0000","last_became_active":"2024-09-16T01:35:00.342056+0000","last_became_peered":"2024-09-16T01:35:00.342056+0000","last_unstale":"2024-09-16T01:35:00.347485+0000","last_undegraded":"2024-09-16T01:35:00.347485+0000","last_fullsized":"2024-09-16T01:35:00.347485+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:32:43.356912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.355290+0000","last_change":"2024-09-16T01:35:01.349973+0000","last_active":"2024-09-16T01:35:01.355290+0000","last_peered":"2024-09-16T01:35:01.355290+0000","last_clean":"2024-09-16T01:35:01.355290+0000","last_became_active":"2024-09-16T01:35:01.349733+0000","last_became_peered":"2024-09-16T01:35:01.349733+0000","last_unstale":"2024-09-16T01:35:01.355290+0000","last_undegraded":"2024-09-16T01:35:01.355290+0000","last_fullsized":"2024-09-16T01:35:01.355290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:59:24.976252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352519+0000","last_change":"2024-09-16T01:35:00.349540+0000","last_active":"2024-09-16T01:35:00.352519+0000","last_peered":"2024-09-16T01:35:00.352519+0000","last_clean":"2024-09-16T01:35:00.352519+0000","last_became_active":"2024-09-16T01:35:00.349296+0000","last_became_peered":"2024-09-16T01:35:00.349296+0000","last_unstale":"2024-09-16T01:35:00.352519+0000","last_undegraded":"2024-09-16T01:35:00.352519+0000","last_fullsized":"2024-09-16T01:35:00.352519+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:53:24.648600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338365+0000","last_change":"2024-09-16T01:35:00.338365+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338365+0000","last_undegraded":"2024-09-16T01:35:00.338365+0000","last_fullsized":"2024-09-16T01:35:00.338365+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.349142+0000","last_change":"2024-09-16T01:35:00.346787+0000","last_active":"2024-09-16T01:35:00.349142+0000","last_peered":"2024-09-16T01:35:00.349142+0000","last_clean":"2024-09-16T01:35:00.349142+0000","last_became_active":"2024-09-16T01:35:00.346527+0000","last_became_peered":"2024-09-16T01:35:00.346527+0000","last_unstale":"2024-09-16T01:35:00.349142+0000","last_undegraded":"2024-09-16T01:35:00.349142+0000","last_fullsized":"2024-09-16T01:35:00.349142+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:48:54.665338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338003+0000","last_change":"2024-09-16T01:35:00.338003+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338003+0000","last_undegraded":"2024-09-16T01:35:00.338003+0000","last_fullsized":"2024-09-16T01:35:00.338003+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"31'3","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.362725+0000","last_change":"2024-09-16T01:35:00.352778+0000","last_active":"2024-09-16T01:35:01.362725+0000","last_peered":"2024-09-16T01:35:01.362725+0000","last_clean":"2024-09-16T01:35:01.362725+0000","last_became_active":"2024-09-16T01:35:00.352534+0000","last_became_peered":"2024-09-16T01:35:00.352534+0000","last_unstale":"2024-09-16T01:35:01.362725+0000","last_undegraded":"2024-09-16T01:35:01.362725+0000","last_fullsized":"2024-09-16T01:35:01.362725+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:32:30.768011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.357410+0000","last_change":"2024-09-16T01:35:01.353288+0000","last_active":"2024-09-16T01:35:01.357410+0000","last_peered":"2024-09-16T01:35:01.357410+0000","last_clean":"2024-09-16T01:35:01.357410+0000","last_became_active":"2024-09-16T01:35:01.352428+0000","last_became_peered":"2024-09-16T01:35:01.352428+0000","last_unstale":"2024-09-16T01:35:01.357410+0000","last_undegraded":"2024-09-16T01:35:01.357410+0000","last_fullsized":"2024-09-16T01:35:01.357410+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:48:49.160580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352983+0000","last_change":"2024-09-16T01:35:00.350506+0000","last_active":"2024-09-16T01:35:00.352983+0000","last_peered":"2024-09-16T01:35:00.352983+0000","last_clean":"2024-09-16T01:35:00.352983+0000","last_became_active":"2024-09-16T01:35:00.350215+0000","last_became_peered":"2024-09-16T01:35:00.350215+0000","last_unstale":"2024-09-16T01:35:00.352983+0000","last_undegraded":"2024-09-16T01:35:00.352983+0000","last_fullsized":"2024-09-16T01:35:00.352983+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:43:39.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339530+0000","last_change":"2024-09-16T01:35:00.339530+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339530+0000","last_undegraded":"2024-09-16T01:35:00.339530+0000","last_fullsized":"2024-09-16T01:35:00.339530+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.348029+0000","last_change":"2024-09-16T01:35:00.342977+0000","last_active":"2024-09-16T01:35:00.348029+0000","last_peered":"2024-09-16T01:35:00.348029+0000","last_clean":"2024-09-16T01:35:00.348029+0000","last_became_active":"2024-09-16T01:35:00.342699+0000","last_became_peered":"2024-09-16T01:35:00.342699+0000","last_unstale":"2024-09-16T01:35:00.348029+0000","last_undegraded":"2024-09-16T01:35:00.348029+0000","last_fullsized":"2024-09-16T01:35:00.348029+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:53:54.367989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337148+0000","last_change":"2024-09-16T01:35:00.337148+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337148+0000","last_undegraded":"2024-09-16T01:35:00.337148+0000","last_fullsized":"2024-09-16T01:35:00.337148+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.348456+0000","last_change":"2024-09-16T01:35:00.343581+0000","last_active":"2024-09-16T01:35:00.348456+0000","last_peered":"2024-09-16T01:35:00.348456+0000","last_clean":"2024-09-16T01:35:00.348456+0000","last_became_active":"2024-09-16T01:35:00.343294+0000","last_became_peered":"2024-09-16T01:35:00.343294+0000","last_unstale":"2024-09-16T01:35:00.348456+0000","last_undegraded":"2024-09-16T01:35:00.348456+0000","last_fullsized":"2024-09-16T01:35:00.348456+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:57.654721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.346877+0000","last_change":"2024-09-16T01:35:00.346877+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.346877+0000","last_undegraded":"2024-09-16T01:35:00.346877+0000","last_fullsized":"2024-09-16T01:35:00.346877+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353028+0000","last_change":"2024-09-16T01:35:00.351166+0000","last_active":"2024-09-16T01:35:00.353028+0000","last_peered":"2024-09-16T01:35:00.353028+0000","last_clean":"2024-09-16T01:35:00.353028+0000","last_became_active":"2024-09-16T01:35:00.350881+0000","last_became_peered":"2024-09-16T01:35:00.350881+0000","last_unstale":"2024-09-16T01:35:00.353028+0000","last_undegraded":"2024-09-16T01:35:00.353028+0000","last_fullsized":"2024-09-16T01:35:00.353028+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:54:05.742892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337056+0000","last_change":"2024-09-16T01:35:00.337056+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337056+0000","last_undegraded":"2024-09-16T01:35:00.337056+0000","last_fullsized":"2024-09-16T01:35:00.337056+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.350509+0000","last_change":"2024-09-16T01:35:00.345939+0000","last_active":"2024-09-16T01:35:00.350509+0000","last_peered":"2024-09-16T01:35:00.350509+0000","last_clean":"2024-09-16T01:35:00.350509+0000","last_became_active":"2024-09-16T01:35:00.345653+0000","last_became_peered":"2024-09-16T01:35:00.345653+0000","last_unstale":"2024-09-16T01:35:00.350509+0000","last_undegraded":"2024-09-16T01:35:00.350509+0000","last_fullsized":"2024-09-16T01:35:00.350509+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:57.839735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339311+0000","last_change":"2024-09-16T01:35:00.339311+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339311+0000","last_undegraded":"2024-09-16T01:35:00.339311+0000","last_fullsized":"2024-09-16T01:35:00.339311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352801+0000","last_change":"2024-09-16T01:35:00.350284+0000","last_active":"2024-09-16T01:35:00.352801+0000","last_peered":"2024-09-16T01:35:00.352801+0000","last_clean":"2024-09-16T01:35:00.352801+0000","last_became_active":"2024-09-16T01:35:00.349516+0000","last_became_peered":"2024-09-16T01:35:00.349516+0000","last_unstale":"2024-09-16T01:35:00.352801+0000","last_undegraded":"2024-09-16T01:35:00.352801+0000","last_fullsized":"2024-09-16T01:35:00.352801+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:22:55.631379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.354497+0000","last_change":"2024-09-16T01:35:01.347677+0000","last_active":"2024-09-16T01:35:01.354497+0000","last_peered":"2024-09-16T01:35:01.354497+0000","last_clean":"2024-09-16T01:35:01.354497+0000","last_became_active":"2024-09-16T01:35:01.345167+0000","last_became_peered":"2024-09-16T01:35:01.345167+0000","last_unstale":"2024-09-16T01:35:01.354497+0000","last_undegraded":"2024-09-16T01:35:01.354497+0000","last_fullsized":"2024-09-16T01:35:01.354497+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:34:25.731515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.347434+0000","last_change":"2024-09-16T01:35:00.342181+0000","last_active":"2024-09-16T01:35:00.347434+0000","last_peered":"2024-09-16T01:35:00.347434+0000","last_clean":"2024-09-16T01:35:00.347434+0000","last_became_active":"2024-09-16T01:35:00.341878+0000","last_became_peered":"2024-09-16T01:35:00.341878+0000","last_unstale":"2024-09-16T01:35:00.347434+0000","last_undegraded":"2024-09-16T01:35:00.347434+0000","last_fullsized":"2024-09-16T01:35:00.347434+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:17:19.574295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339159+0000","last_change":"2024-09-16T01:35:00.339159+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339159+0000","last_undegraded":"2024-09-16T01:35:00.339159+0000","last_fullsized":"2024-09-16T01:35:00.339159+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353150+0000","last_change":"2024-09-16T01:35:00.349566+0000","last_active":"2024-09-16T01:35:00.353150+0000","last_peered":"2024-09-16T01:35:00.353150+0000","last_clean":"2024-09-16T01:35:00.353150+0000","last_became_active":"2024-09-16T01:35:00.349243+0000","last_became_peered":"2024-09-16T01:35:00.349243+0000","last_unstale":"2024-09-16T01:35:00.353150+0000","last_undegraded":"2024-09-16T01:35:00.353150+0000","last_fullsized":"2024-09-16T01:35:00.353150+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:43:05.059186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.341010+0000","last_change":"2024-09-16T01:35:00.341010+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.341010+0000","last_undegraded":"2024-09-16T01:35:00.341010+0000","last_fullsized":"2024-09-16T01:35:00.341010+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353118+0000","last_change":"2024-09-16T01:35:00.348167+0000","last_active":"2024-09-16T01:35:00.353118+0000","last_peered":"2024-09-16T01:35:00.353118+0000","last_clean":"2024-09-16T01:35:00.353118+0000","last_became_active":"2024-09-16T01:35:00.347714+0000","last_became_peered":"2024-09-16T01:35:00.347714+0000","last_unstale":"2024-09-16T01:35:00.353118+0000","last_undegraded":"2024-09-16T01:35:00.353118+0000","last_fullsized":"2024-09-16T01:35:00.353118+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:29:35.905712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.342418+0000","last_change":"2024-09-16T01:35:00.342418+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.342418+0000","last_undegraded":"2024-09-16T01:35:00.342418+0000","last_fullsized":"2024-09-16T01:35:00.342418+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.346227+0000","last_change":"2024-09-16T01:35:00.345340+0000","last_active":"2024-09-16T01:35:00.346227+0000","last_peered":"2024-09-16T01:35:00.346227+0000","last_clean":"2024-09-16T01:35:00.346227+0000","last_became_active":"2024-09-16T01:35:00.344987+0000","last_became_peered":"2024-09-16T01:35:00.344987+0000","last_unstale":"2024-09-16T01:35:00.346227+0000","last_undegraded":"2024-09-16T01:35:00.346227+0000","last_fullsized":"2024-09-16T01:35:00.346227+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:02:54.003881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.340705+0000","last_change":"2024-09-16T01:35:00.340705+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.340705+0000","last_undegraded":"2024-09-16T01:35:00.340705+0000","last_fullsized":"2024-09-16T01:35:00.340705+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.354908+0000","last_change":"2024-09-16T01:35:00.349814+0000","last_active":"2024-09-16T01:35:00.354908+0000","last_peered":"2024-09-16T01:35:00.354908+0000","last_clean":"2024-09-16T01:35:00.354908+0000","last_became_active":"2024-09-16T01:35:00.349543+0000","last_became_peered":"2024-09-16T01:35:00.349543+0000","last_unstale":"2024-09-16T01:35:00.354908+0000","last_undegraded":"2024-09-16T01:35:00.354908+0000","last_fullsized":"2024-09-16T01:35:00.354908+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:02:54.003881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.342349+0000","last_change":"2024-09-16T01:35:00.342349+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.342349+0000","last_undegraded":"2024-09-16T01:35:00.342349+0000","last_fullsized":"2024-09-16T01:35:00.342349+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353146+0000","last_change":"2024-09-16T01:35:00.348335+0000","last_active":"2024-09-16T01:35:00.353146+0000","last_peered":"2024-09-16T01:35:00.353146+0000","last_clean":"2024-09-16T01:35:00.353146+0000","last_became_active":"2024-09-16T01:35:00.347099+0000","last_became_peered":"2024-09-16T01:35:00.347099+0000","last_unstale":"2024-09-16T01:35:00.353146+0000","last_undegraded":"2024-09-16T01:35:00.353146+0000","last_fullsized":"2024-09-16T01:35:00.353146+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:57:35.138402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.346143+0000","last_change":"2024-09-16T01:35:00.346143+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.346143+0000","last_undegraded":"2024-09-16T01:35:00.346143+0000","last_fullsized":"2024-09-16T01:35:00.346143+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.337675+0000","last_change":"2024-09-16T01:35:00.352839+0000","last_active":"2024-09-16T01:35:01.337675+0000","last_peered":"2024-09-16T01:35:01.337675+0000","last_clean":"2024-09-16T01:35:01.337675+0000","last_became_active":"2024-09-16T01:35:00.352568+0000","last_became_peered":"2024-09-16T01:35:00.352568+0000","last_unstale":"2024-09-16T01:35:01.337675+0000","last_undegraded":"2024-09-16T01:35:01.337675+0000","last_fullsized":"2024-09-16T01:35:01.337675+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:30:51.141162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.342110+0000","last_change":"2024-09-16T01:35:00.342110+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.342110+0000","last_undegraded":"2024-09-16T01:35:00.342110+0000","last_fullsized":"2024-09-16T01:35:00.342110+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.357904+0000","last_change":"2024-09-16T01:35:00.355682+0000","last_active":"2024-09-16T01:35:00.357904+0000","last_peered":"2024-09-16T01:35:00.357904+0000","last_clean":"2024-09-16T01:35:00.357904+0000","last_became_active":"2024-09-16T01:35:00.355347+0000","last_became_peered":"2024-09-16T01:35:00.355347+0000","last_unstale":"2024-09-16T01:35:00.357904+0000","last_undegraded":"2024-09-16T01:35:00.357904+0000","last_fullsized":"2024-09-16T01:35:00.357904+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:07.992492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.359762+0000","last_change":"2024-09-16T01:35:01.355640+0000","last_active":"2024-09-16T01:35:01.359762+0000","last_peered":"2024-09-16T01:35:01.359762+0000","last_clean":"2024-09-16T01:35:01.359762+0000","last_became_active":"2024-09-16T01:35:01.355364+0000","last_became_peered":"2024-09-16T01:35:01.355364+0000","last_unstale":"2024-09-16T01:35:01.359762+0000","last_undegraded":"2024-09-16T01:35:01.359762+0000","last_fullsized":"2024-09-16T01:35:01.359762+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:26:10.193445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.346212+0000","last_change":"2024-09-16T01:35:00.343346+0000","last_active":"2024-09-16T01:35:00.346212+0000","last_peered":"2024-09-16T01:35:00.346212+0000","last_clean":"2024-09-16T01:35:00.346212+0000","last_became_active":"2024-09-16T01:35:00.343049+0000","last_became_peered":"2024-09-16T01:35:00.343049+0000","last_unstale":"2024-09-16T01:35:00.346212+0000","last_undegraded":"2024-09-16T01:35:00.346212+0000","last_fullsized":"2024-09-16T01:35:00.346212+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:00.117677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.335782+0000","last_change":"2024-09-16T01:35:00.335782+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.335782+0000","last_undegraded":"2024-09-16T01:35:00.335782+0000","last_fullsized":"2024-09-16T01:35:00.335782+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352715+0000","last_change":"2024-09-16T01:35:00.349236+0000","last_active":"2024-09-16T01:35:00.352715+0000","last_peered":"2024-09-16T01:35:00.352715+0000","last_clean":"2024-09-16T01:35:00.352715+0000","last_became_active":"2024-09-16T01:35:00.348947+0000","last_became_peered":"2024-09-16T01:35:00.348947+0000","last_unstale":"2024-09-16T01:35:00.352715+0000","last_undegraded":"2024-09-16T01:35:00.352715+0000","last_fullsized":"2024-09-16T01:35:00.352715+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:18:05.552429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.336999+0000","last_change":"2024-09-16T01:35:00.336999+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.336999+0000","last_undegraded":"2024-09-16T01:35:00.336999+0000","last_fullsized":"2024-09-16T01:35:00.336999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.357770+0000","last_change":"2024-09-16T01:35:00.355723+0000","last_active":"2024-09-16T01:35:00.357770+0000","last_peered":"2024-09-16T01:35:00.357770+0000","last_clean":"2024-09-16T01:35:00.357770+0000","last_became_active":"2024-09-16T01:35:00.355425+0000","last_became_peered":"2024-09-16T01:35:00.355425+0000","last_unstale":"2024-09-16T01:35:00.357770+0000","last_undegraded":"2024-09-16T01:35:00.357770+0000","last_fullsized":"2024-09-16T01:35:00.357770+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:00.732201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.335633+0000","last_change":"2024-09-16T01:35:00.335633+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.335633+0000","last_undegraded":"2024-09-16T01:35:00.335633+0000","last_fullsized":"2024-09-16T01:35:00.335633+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.357916+0000","last_change":"2024-09-16T01:35:00.355687+0000","last_active":"2024-09-16T01:35:00.357916+0000","last_peered":"2024-09-16T01:35:00.357916+0000","last_clean":"2024-09-16T01:35:00.357916+0000","last_became_active":"2024-09-16T01:35:00.355357+0000","last_became_peered":"2024-09-16T01:35:00.355357+0000","last_unstale":"2024-09-16T01:35:00.357916+0000","last_undegraded":"2024-09-16T01:35:00.357916+0000","last_fullsized":"2024-09-16T01:35:00.357916+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:30:55.369673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337712+0000","last_change":"2024-09-16T01:35:00.337712+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337712+0000","last_undegraded":"2024-09-16T01:35:00.337712+0000","last_fullsized":"2024-09-16T01:35:00.337712+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353170+0000","last_change":"2024-09-16T01:35:00.351123+0000","last_active":"2024-09-16T01:35:00.353170+0000","last_peered":"2024-09-16T01:35:00.353170+0000","last_clean":"2024-09-16T01:35:00.353170+0000","last_became_active":"2024-09-16T01:35:00.350803+0000","last_became_peered":"2024-09-16T01:35:00.350803+0000","last_unstale":"2024-09-16T01:35:00.353170+0000","last_undegraded":"2024-09-16T01:35:00.353170+0000","last_fullsized":"2024-09-16T01:35:00.353170+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:51:21.665512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339066+0000","last_change":"2024-09-16T01:35:00.339066+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339066+0000","last_undegraded":"2024-09-16T01:35:00.339066+0000","last_fullsized":"2024-09-16T01:35:00.339066+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353039+0000","last_change":"2024-09-16T01:35:00.351535+0000","last_active":"2024-09-16T01:35:00.353039+0000","last_peered":"2024-09-16T01:35:00.353039+0000","last_clean":"2024-09-16T01:35:00.353039+0000","last_became_active":"2024-09-16T01:35:00.351219+0000","last_became_peered":"2024-09-16T01:35:00.351219+0000","last_unstale":"2024-09-16T01:35:00.353039+0000","last_undegraded":"2024-09-16T01:35:00.353039+0000","last_fullsized":"2024-09-16T01:35:00.353039+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:04.599770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.342993+0000","last_change":"2024-09-16T01:35:00.342993+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.342993+0000","last_undegraded":"2024-09-16T01:35:00.342993+0000","last_fullsized":"2024-09-16T01:35:00.342993+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352480+0000","last_change":"2024-09-16T01:35:00.344871+0000","last_active":"2024-09-16T01:35:00.352480+0000","last_peered":"2024-09-16T01:35:00.352480+0000","last_clean":"2024-09-16T01:35:00.352480+0000","last_became_active":"2024-09-16T01:35:00.344582+0000","last_became_peered":"2024-09-16T01:35:00.344582+0000","last_unstale":"2024-09-16T01:35:00.352480+0000","last_undegraded":"2024-09-16T01:35:00.352480+0000","last_fullsized":"2024-09-16T01:35:00.352480+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:00.117677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":185,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.348767+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:35:00.348767+0000","last_peered":"2024-09-16T01:35:00.348767+0000","last_clean":"2024-09-16T01:35:00.348767+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:35:00.348767+0000","last_undegraded":"2024-09-16T01:35:00.348767+0000","last_fullsized":"2024-09-16T01:35:00.348767+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509455,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":331776,"data_stored":123852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542158,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27920,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343920,"statfs":{"total":96636764160,"available":96608174080,"internally_reserved":0,"allocated":913408,"data_stored":707953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542158,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":117585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542158,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":117585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542157,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":117585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542158,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":117585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542157,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":117585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542157,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":925696,"data_stored":714220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:35:04.960 INFO:tasks.ceph.ceph_manager.ceph:PG 5.3f is not active+clean 2024-09-16T01:35:04.960 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.337649+0000', 'last_change': '2024-09-16T01:35:00.337649+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.337649+0000', 'last_undegraded': '2024-09-16T01:35:00.337649+0000', 'last_fullsized': '2024-09-16T01:35:00.337649+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T01:35:04.960 INFO:tasks.ceph.ceph_manager.ceph:PG 5.3e is not active+clean 2024-09-16T01:35:04.960 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.339909+0000', 'last_change': '2024-09-16T01:35:00.339909+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.339909+0000', 'last_undegraded': '2024-09-16T01:35:00.339909+0000', 'last_fullsized': '2024-09-16T01:35:00.339909+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T01:35:04.960 INFO:tasks.ceph.ceph_manager.ceph:PG 5.3c is not active+clean 2024-09-16T01:35:04.960 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.340201+0000', 'last_change': '2024-09-16T01:35:00.340201+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.340201+0000', 'last_undegraded': '2024-09-16T01:35:00.340201+0000', 'last_fullsized': '2024-09-16T01:35:00.340201+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T01:35:04.960 INFO:tasks.ceph.ceph_manager.ceph:PG 5.3b is not active+clean 2024-09-16T01:35:04.961 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.341590+0000', 'last_change': '2024-09-16T01:35:00.341590+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.341590+0000', 'last_undegraded': '2024-09-16T01:35:00.341590+0000', 'last_fullsized': '2024-09-16T01:35:00.341590+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T01:35:04.961 INFO:tasks.ceph.ceph_manager.ceph:PG 5.3a is not active+clean 2024-09-16T01:35:04.961 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.338751+0000', 'last_change': '2024-09-16T01:35:00.338751+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.338751+0000', 'last_undegraded': '2024-09-16T01:35:00.338751+0000', 'last_fullsized': '2024-09-16T01:35:00.338751+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T01:35:04.961 INFO:tasks.ceph.ceph_manager.ceph:PG 5.39 is not active+clean 2024-09-16T01:35:04.961 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.345760+0000', 'last_change': '2024-09-16T01:35:00.345760+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.345760+0000', 'last_undegraded': '2024-09-16T01:35:00.345760+0000', 'last_fullsized': '2024-09-16T01:35:00.345760+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T01:35:04.961 INFO:tasks.ceph.ceph_manager.ceph:PG 5.38 is not active+clean 2024-09-16T01:35:04.961 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.347744+0000', 'last_change': '2024-09-16T01:35:00.347744+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.347744+0000', 'last_undegraded': '2024-09-16T01:35:00.347744+0000', 'last_fullsized': '2024-09-16T01:35:00.347744+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T01:35:04.961 INFO:tasks.ceph.ceph_manager.ceph:PG 5.37 is not active+clean 2024-09-16T01:35:04.961 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.337574+0000', 'last_change': '2024-09-16T01:35:00.337574+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.337574+0000', 'last_undegraded': '2024-09-16T01:35:00.337574+0000', 'last_fullsized': '2024-09-16T01:35:00.337574+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T01:35:04.961 INFO:tasks.ceph.ceph_manager.ceph:PG 5.36 is not active+clean 2024-09-16T01:35:04.961 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.339626+0000', 'last_change': '2024-09-16T01:35:00.339626+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.339626+0000', 'last_undegraded': '2024-09-16T01:35:00.339626+0000', 'last_fullsized': '2024-09-16T01:35:00.339626+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T01:35:04.962 INFO:tasks.ceph.ceph_manager.ceph:PG 5.35 is not active+clean 2024-09-16T01:35:04.962 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.344537+0000', 'last_change': '2024-09-16T01:35:00.344537+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.344537+0000', 'last_undegraded': '2024-09-16T01:35:00.344537+0000', 'last_fullsized': '2024-09-16T01:35:00.344537+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T01:35:04.962 INFO:tasks.ceph.ceph_manager.ceph:PG 5.33 is not active+clean 2024-09-16T01:35:04.962 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.341163+0000', 'last_change': '2024-09-16T01:35:00.341163+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.341163+0000', 'last_undegraded': '2024-09-16T01:35:00.341163+0000', 'last_fullsized': '2024-09-16T01:35:00.341163+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T01:35:04.962 INFO:tasks.ceph.ceph_manager.ceph:PG 5.32 is not active+clean 2024-09-16T01:35:04.962 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.337577+0000', 'last_change': '2024-09-16T01:35:00.337577+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.337577+0000', 'last_undegraded': '2024-09-16T01:35:00.337577+0000', 'last_fullsized': '2024-09-16T01:35:00.337577+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T01:35:04.962 INFO:tasks.ceph.ceph_manager.ceph:PG 5.30 is not active+clean 2024-09-16T01:35:04.962 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.342533+0000', 'last_change': '2024-09-16T01:35:00.342533+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.342533+0000', 'last_undegraded': '2024-09-16T01:35:00.342533+0000', 'last_fullsized': '2024-09-16T01:35:00.342533+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T01:35:04.962 INFO:tasks.ceph.ceph_manager.ceph:PG 5.2f is not active+clean 2024-09-16T01:35:04.962 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.343962+0000', 'last_change': '2024-09-16T01:35:00.343962+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.343962+0000', 'last_undegraded': '2024-09-16T01:35:00.343962+0000', 'last_fullsized': '2024-09-16T01:35:00.343962+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T01:35:04.963 INFO:tasks.ceph.ceph_manager.ceph:PG 5.2e is not active+clean 2024-09-16T01:35:04.963 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.339533+0000', 'last_change': '2024-09-16T01:35:00.339533+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.339533+0000', 'last_undegraded': '2024-09-16T01:35:00.339533+0000', 'last_fullsized': '2024-09-16T01:35:00.339533+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T01:35:04.963 INFO:tasks.ceph.ceph_manager.ceph:PG 5.2d is not active+clean 2024-09-16T01:35:04.963 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.337206+0000', 'last_change': '2024-09-16T01:35:00.337206+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.337206+0000', 'last_undegraded': '2024-09-16T01:35:00.337206+0000', 'last_fullsized': '2024-09-16T01:35:00.337206+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T01:35:04.963 INFO:tasks.ceph.ceph_manager.ceph:PG 5.2c is not active+clean 2024-09-16T01:35:04.963 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.340512+0000', 'last_change': '2024-09-16T01:35:00.340512+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.340512+0000', 'last_undegraded': '2024-09-16T01:35:00.340512+0000', 'last_fullsized': '2024-09-16T01:35:00.340512+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T01:35:04.963 INFO:tasks.ceph.ceph_manager.ceph:PG 5.2b is not active+clean 2024-09-16T01:35:04.963 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.339909+0000', 'last_change': '2024-09-16T01:35:00.339909+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.339909+0000', 'last_undegraded': '2024-09-16T01:35:00.339909+0000', 'last_fullsized': '2024-09-16T01:35:00.339909+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T01:35:04.963 INFO:tasks.ceph.ceph_manager.ceph:PG 5.2a is not active+clean 2024-09-16T01:35:04.963 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.338369+0000', 'last_change': '2024-09-16T01:35:00.338369+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.338369+0000', 'last_undegraded': '2024-09-16T01:35:00.338369+0000', 'last_fullsized': '2024-09-16T01:35:00.338369+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T01:35:04.964 INFO:tasks.ceph.ceph_manager.ceph:PG 5.29 is not active+clean 2024-09-16T01:35:04.964 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.335686+0000', 'last_change': '2024-09-16T01:35:00.335686+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.335686+0000', 'last_undegraded': '2024-09-16T01:35:00.335686+0000', 'last_fullsized': '2024-09-16T01:35:00.335686+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T01:35:04.964 INFO:tasks.ceph.ceph_manager.ceph:PG 5.27 is not active+clean 2024-09-16T01:35:04.964 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.338640+0000', 'last_change': '2024-09-16T01:35:00.338640+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.338640+0000', 'last_undegraded': '2024-09-16T01:35:00.338640+0000', 'last_fullsized': '2024-09-16T01:35:00.338640+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T01:35:04.964 INFO:tasks.ceph.ceph_manager.ceph:PG 5.26 is not active+clean 2024-09-16T01:35:04.964 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.339123+0000', 'last_change': '2024-09-16T01:35:00.339123+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.339123+0000', 'last_undegraded': '2024-09-16T01:35:00.339123+0000', 'last_fullsized': '2024-09-16T01:35:00.339123+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T01:35:04.964 INFO:tasks.ceph.ceph_manager.ceph:PG 5.25 is not active+clean 2024-09-16T01:35:04.964 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.337376+0000', 'last_change': '2024-09-16T01:35:00.337376+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.337376+0000', 'last_undegraded': '2024-09-16T01:35:00.337376+0000', 'last_fullsized': '2024-09-16T01:35:00.337376+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T01:35:04.964 INFO:tasks.ceph.ceph_manager.ceph:PG 5.23 is not active+clean 2024-09-16T01:35:04.964 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.339456+0000', 'last_change': '2024-09-16T01:35:00.339456+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.339456+0000', 'last_undegraded': '2024-09-16T01:35:00.339456+0000', 'last_fullsized': '2024-09-16T01:35:00.339456+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T01:35:04.965 INFO:tasks.ceph.ceph_manager.ceph:PG 5.22 is not active+clean 2024-09-16T01:35:04.965 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.343049+0000', 'last_change': '2024-09-16T01:35:00.343049+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.343049+0000', 'last_undegraded': '2024-09-16T01:35:00.343049+0000', 'last_fullsized': '2024-09-16T01:35:00.343049+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T01:35:04.965 INFO:tasks.ceph.ceph_manager.ceph:PG 5.21 is not active+clean 2024-09-16T01:35:04.965 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.338353+0000', 'last_change': '2024-09-16T01:35:00.338353+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.338353+0000', 'last_undegraded': '2024-09-16T01:35:00.338353+0000', 'last_fullsized': '2024-09-16T01:35:00.338353+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T01:35:04.965 INFO:tasks.ceph.ceph_manager.ceph:PG 5.20 is not active+clean 2024-09-16T01:35:04.965 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.347252+0000', 'last_change': '2024-09-16T01:35:00.347252+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.347252+0000', 'last_undegraded': '2024-09-16T01:35:00.347252+0000', 'last_fullsized': '2024-09-16T01:35:00.347252+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T01:35:04.965 INFO:tasks.ceph.ceph_manager.ceph:PG 5.1e is not active+clean 2024-09-16T01:35:04.965 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.335647+0000', 'last_change': '2024-09-16T01:35:00.335647+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.335647+0000', 'last_undegraded': '2024-09-16T01:35:00.335647+0000', 'last_fullsized': '2024-09-16T01:35:00.335647+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T01:35:04.965 INFO:tasks.ceph.ceph_manager.ceph:PG 5.1f is not active+clean 2024-09-16T01:35:04.965 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.337125+0000', 'last_change': '2024-09-16T01:35:00.337125+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.337125+0000', 'last_undegraded': '2024-09-16T01:35:00.337125+0000', 'last_fullsized': '2024-09-16T01:35:00.337125+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T01:35:04.965 INFO:tasks.ceph.ceph_manager.ceph:PG 5.1c is not active+clean 2024-09-16T01:35:04.966 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.339423+0000', 'last_change': '2024-09-16T01:35:00.339423+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.339423+0000', 'last_undegraded': '2024-09-16T01:35:00.339423+0000', 'last_fullsized': '2024-09-16T01:35:00.339423+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T01:35:04.966 INFO:tasks.ceph.ceph_manager.ceph:PG 5.1d is not active+clean 2024-09-16T01:35:04.966 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.336922+0000', 'last_change': '2024-09-16T01:35:00.336922+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.336922+0000', 'last_undegraded': '2024-09-16T01:35:00.336922+0000', 'last_fullsized': '2024-09-16T01:35:00.336922+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T01:35:04.966 INFO:tasks.ceph.ceph_manager.ceph:PG 5.1a is not active+clean 2024-09-16T01:35:04.966 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.342570+0000', 'last_change': '2024-09-16T01:35:00.342570+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.342570+0000', 'last_undegraded': '2024-09-16T01:35:00.342570+0000', 'last_fullsized': '2024-09-16T01:35:00.342570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T01:35:04.966 INFO:tasks.ceph.ceph_manager.ceph:PG 5.18 is not active+clean 2024-09-16T01:35:04.966 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.338296+0000', 'last_change': '2024-09-16T01:35:00.338296+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.338296+0000', 'last_undegraded': '2024-09-16T01:35:00.338296+0000', 'last_fullsized': '2024-09-16T01:35:00.338296+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T01:35:04.966 INFO:tasks.ceph.ceph_manager.ceph:PG 5.19 is not active+clean 2024-09-16T01:35:04.966 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.336753+0000', 'last_change': '2024-09-16T01:35:00.336753+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.336753+0000', 'last_undegraded': '2024-09-16T01:35:00.336753+0000', 'last_fullsized': '2024-09-16T01:35:00.336753+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T01:35:04.967 INFO:tasks.ceph.ceph_manager.ceph:PG 5.17 is not active+clean 2024-09-16T01:35:04.967 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.338365+0000', 'last_change': '2024-09-16T01:35:00.338365+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.338365+0000', 'last_undegraded': '2024-09-16T01:35:00.338365+0000', 'last_fullsized': '2024-09-16T01:35:00.338365+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T01:35:04.967 INFO:tasks.ceph.ceph_manager.ceph:PG 5.14 is not active+clean 2024-09-16T01:35:04.967 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.338003+0000', 'last_change': '2024-09-16T01:35:00.338003+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.338003+0000', 'last_undegraded': '2024-09-16T01:35:00.338003+0000', 'last_fullsized': '2024-09-16T01:35:00.338003+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T01:35:04.967 INFO:tasks.ceph.ceph_manager.ceph:PG 5.12 is not active+clean 2024-09-16T01:35:04.967 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.339530+0000', 'last_change': '2024-09-16T01:35:00.339530+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.339530+0000', 'last_undegraded': '2024-09-16T01:35:00.339530+0000', 'last_fullsized': '2024-09-16T01:35:00.339530+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T01:35:04.967 INFO:tasks.ceph.ceph_manager.ceph:PG 5.13 is not active+clean 2024-09-16T01:35:04.967 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.337148+0000', 'last_change': '2024-09-16T01:35:00.337148+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.337148+0000', 'last_undegraded': '2024-09-16T01:35:00.337148+0000', 'last_fullsized': '2024-09-16T01:35:00.337148+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T01:35:04.967 INFO:tasks.ceph.ceph_manager.ceph:PG 5.10 is not active+clean 2024-09-16T01:35:04.967 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.346877+0000', 'last_change': '2024-09-16T01:35:00.346877+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.346877+0000', 'last_undegraded': '2024-09-16T01:35:00.346877+0000', 'last_fullsized': '2024-09-16T01:35:00.346877+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T01:35:04.967 INFO:tasks.ceph.ceph_manager.ceph:PG 5.11 is not active+clean 2024-09-16T01:35:04.967 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.337056+0000', 'last_change': '2024-09-16T01:35:00.337056+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.337056+0000', 'last_undegraded': '2024-09-16T01:35:00.337056+0000', 'last_fullsized': '2024-09-16T01:35:00.337056+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T01:35:04.968 INFO:tasks.ceph.ceph_manager.ceph:PG 5.e is not active+clean 2024-09-16T01:35:04.968 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.339311+0000', 'last_change': '2024-09-16T01:35:00.339311+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.339311+0000', 'last_undegraded': '2024-09-16T01:35:00.339311+0000', 'last_fullsized': '2024-09-16T01:35:00.339311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T01:35:04.968 INFO:tasks.ceph.ceph_manager.ceph:PG 5.c is not active+clean 2024-09-16T01:35:04.968 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.339159+0000', 'last_change': '2024-09-16T01:35:00.339159+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.339159+0000', 'last_undegraded': '2024-09-16T01:35:00.339159+0000', 'last_fullsized': '2024-09-16T01:35:00.339159+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T01:35:04.968 INFO:tasks.ceph.ceph_manager.ceph:PG 5.d is not active+clean 2024-09-16T01:35:04.968 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.341010+0000', 'last_change': '2024-09-16T01:35:00.341010+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.341010+0000', 'last_undegraded': '2024-09-16T01:35:00.341010+0000', 'last_fullsized': '2024-09-16T01:35:00.341010+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T01:35:04.968 INFO:tasks.ceph.ceph_manager.ceph:PG 5.a is not active+clean 2024-09-16T01:35:04.968 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.342418+0000', 'last_change': '2024-09-16T01:35:00.342418+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.342418+0000', 'last_undegraded': '2024-09-16T01:35:00.342418+0000', 'last_fullsized': '2024-09-16T01:35:00.342418+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T01:35:04.968 INFO:tasks.ceph.ceph_manager.ceph:PG 5.b is not active+clean 2024-09-16T01:35:04.968 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.340705+0000', 'last_change': '2024-09-16T01:35:00.340705+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.340705+0000', 'last_undegraded': '2024-09-16T01:35:00.340705+0000', 'last_fullsized': '2024-09-16T01:35:00.340705+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T01:35:04.969 INFO:tasks.ceph.ceph_manager.ceph:PG 5.8 is not active+clean 2024-09-16T01:35:04.969 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.342349+0000', 'last_change': '2024-09-16T01:35:00.342349+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.342349+0000', 'last_undegraded': '2024-09-16T01:35:00.342349+0000', 'last_fullsized': '2024-09-16T01:35:00.342349+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T01:35:04.969 INFO:tasks.ceph.ceph_manager.ceph:PG 5.9 is not active+clean 2024-09-16T01:35:04.969 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.346143+0000', 'last_change': '2024-09-16T01:35:00.346143+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.346143+0000', 'last_undegraded': '2024-09-16T01:35:00.346143+0000', 'last_fullsized': '2024-09-16T01:35:00.346143+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T01:35:04.969 INFO:tasks.ceph.ceph_manager.ceph:PG 5.6 is not active+clean 2024-09-16T01:35:04.969 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.342110+0000', 'last_change': '2024-09-16T01:35:00.342110+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.342110+0000', 'last_undegraded': '2024-09-16T01:35:00.342110+0000', 'last_fullsized': '2024-09-16T01:35:00.342110+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T01:35:04.969 INFO:tasks.ceph.ceph_manager.ceph:PG 5.5 is not active+clean 2024-09-16T01:35:04.969 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.335782+0000', 'last_change': '2024-09-16T01:35:00.335782+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.335782+0000', 'last_undegraded': '2024-09-16T01:35:00.335782+0000', 'last_fullsized': '2024-09-16T01:35:00.335782+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T01:35:04.969 INFO:tasks.ceph.ceph_manager.ceph:PG 5.2 is not active+clean 2024-09-16T01:35:04.969 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.336999+0000', 'last_change': '2024-09-16T01:35:00.336999+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.336999+0000', 'last_undegraded': '2024-09-16T01:35:00.336999+0000', 'last_fullsized': '2024-09-16T01:35:00.336999+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T01:35:04.969 INFO:tasks.ceph.ceph_manager.ceph:PG 5.3 is not active+clean 2024-09-16T01:35:04.970 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.335633+0000', 'last_change': '2024-09-16T01:35:00.335633+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.335633+0000', 'last_undegraded': '2024-09-16T01:35:00.335633+0000', 'last_fullsized': '2024-09-16T01:35:00.335633+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T01:35:04.970 INFO:tasks.ceph.ceph_manager.ceph:PG 5.0 is not active+clean 2024-09-16T01:35:04.970 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.337712+0000', 'last_change': '2024-09-16T01:35:00.337712+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.337712+0000', 'last_undegraded': '2024-09-16T01:35:00.337712+0000', 'last_fullsized': '2024-09-16T01:35:00.337712+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T01:35:04.970 INFO:tasks.ceph.ceph_manager.ceph:PG 5.1 is not active+clean 2024-09-16T01:35:04.970 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.339066+0000', 'last_change': '2024-09-16T01:35:00.339066+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.339066+0000', 'last_undegraded': '2024-09-16T01:35:00.339066+0000', 'last_fullsized': '2024-09-16T01:35:00.339066+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T01:35:04.970 INFO:tasks.ceph.ceph_manager.ceph:PG 5.4 is not active+clean 2024-09-16T01:35:04.970 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:35:00.342993+0000', 'last_change': '2024-09-16T01:35:00.342993+0000', 'last_active': '2024-09-16T01:35:00.326194+0000', 'last_peered': '2024-09-16T01:35:00.326194+0000', 'last_clean': '2024-09-16T01:35:00.326194+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:35:00.342993+0000', 'last_undegraded': '2024-09-16T01:35:00.342993+0000', 'last_fullsized': '2024-09-16T01:35:00.342993+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'last_clean_scrub_stamp': '2024-09-16T01:35:00.326194+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T01:35:04.971 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T01:35:05.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.094+0000 7f073c35f640 1 Processor -- start 2024-09-16T01:35:05.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.094+0000 7f073c35f640 1 -- start start 2024-09-16T01:35:05.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f073c35f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f073406bc10 0x7f073406bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:05.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f073c35f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f073415cdb0 0x7f073415d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:05.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f073c35f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f073406c530 con 0x7f073415c710 2024-09-16T01:35:05.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f073c35f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0734072760 con 0x7f073415cdb0 2024-09-16T01:35:05.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f073c35f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0734072a30 con 0x7f073406bc10 2024-09-16T01:35:05.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f07398fa640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f073415c710 0x7f0734072050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35986/0 (socket says 172.21.15.88:35986) 2024-09-16T01:35:05.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f07398fa640 1 -- 172.21.15.88:0/1639286264 learned_addr learned my addr 172.21.15.88:0/1639286264 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:05.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f07390f9640 1 -- 172.21.15.88:0/1639286264 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2047867086 0 0) 0x7f073406c530 con 0x7f073415c710 2024-09-16T01:35:05.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f073a0fb640 1 --2- 172.21.15.88:0/1639286264 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f073406bc10 0x7f073406bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:05.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f07390f9640 1 -- 172.21.15.88:0/1639286264 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f071c003520 con 0x7f073415c710 2024-09-16T01:35:05.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f073a8fc640 1 --2- 172.21.15.88:0/1639286264 >> v2:172.21.15.154:3300/0 conn(0x7f073415cdb0 0x7f073415d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f07390f9640 1 -- 172.21.15.88:0/1639286264 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1037273144 0 0) 0x7f071c003520 con 0x7f073415c710 2024-09-16T01:35:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f07390f9640 1 -- 172.21.15.88:0/1639286264 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f073406bc10 msgr2=0x7f073406bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f07390f9640 1 --2- 172.21.15.88:0/1639286264 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f073406bc10 0x7f073406bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f07390f9640 1 -- 172.21.15.88:0/1639286264 >> v2:172.21.15.154:3300/0 conn(0x7f073415cdb0 msgr2=0x7f073415d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f07390f9640 1 --2- 172.21.15.88:0/1639286264 >> v2:172.21.15.154:3300/0 conn(0x7f073415cdb0 0x7f073415d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f07390f9640 1 -- 172.21.15.88:0/1639286264 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0734072d10 con 0x7f073415c710 2024-09-16T01:35:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f07390f9640 1 -- 172.21.15.88:0/1639286264 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f0728001670 con 0x7f073415c710 2024-09-16T01:35:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f073a8fc640 1 --2- 172.21.15.88:0/1639286264 >> v2:172.21.15.154:3300/0 conn(0x7f073415cdb0 0x7f073415d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f07390f9640 1 -- 172.21.15.88:0/1639286264 >> v1:172.21.15.88:6789/0 conn(0x7f073415c710 legacy=0x7f0734072050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f07390f9640 1 --2- 172.21.15.88:0/1639286264 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f071c0039b0 0x7f071c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f07390f9640 1 --2- 172.21.15.88:0/1639286264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f071c004800 0x7f071c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f07390f9640 1 --2- 172.21.15.88:0/1639286264 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f073406bc10 0x7f071c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:05.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f07390f9640 1 -- 172.21.15.88:0/1639286264 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0734072760 con 0x7f071c0039b0 2024-09-16T01:35:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f07390f9640 1 -- 172.21.15.88:0/1639286264 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0734072a30 con 0x7f071c004800 2024-09-16T01:35:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f07390f9640 1 -- 172.21.15.88:0/1639286264 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f071c00b9c0 con 0x7f073406bc10 2024-09-16T01:35:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.095+0000 7f073a0fb640 1 --2- 172.21.15.88:0/1639286264 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f071c0039b0 0x7f071c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.096+0000 7f07398fa640 1 --2- 172.21.15.88:0/1639286264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f071c004800 0x7f071c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.096+0000 7f073a8fc640 1 --2- 172.21.15.88:0/1639286264 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f073406bc10 0x7f071c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.096+0000 7f073a0fb640 1 -- 172.21.15.88:0/1639286264 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f073406bc10 msgr2=0x7f071c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.096+0000 7f073a0fb640 1 --2- 172.21.15.88:0/1639286264 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f073406bc10 0x7f071c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.096+0000 7f073a0fb640 1 -- 172.21.15.88:0/1639286264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f071c004800 msgr2=0x7f071c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.096+0000 7f073a0fb640 1 --2- 172.21.15.88:0/1639286264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f071c004800 0x7f071c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.096+0000 7f073a0fb640 1 -- 172.21.15.88:0/1639286264 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f071c003520 con 0x7f071c0039b0 2024-09-16T01:35:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.096+0000 7f073a0fb640 1 --2- 172.21.15.88:0/1639286264 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f071c0039b0 0x7f071c003dd0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f072c009ac0 tx=0x7f071c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4289260d6a5bf9f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:05.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.097+0000 7f07390f9640 1 -- 172.21.15.88:0/1639286264 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072c036070 con 0x7f071c0039b0 2024-09-16T01:35:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.097+0000 7f07390f9640 1 -- 172.21.15.88:0/1639286264 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f072c031400 con 0x7f071c0039b0 2024-09-16T01:35:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.097+0000 7f07390f9640 1 -- 172.21.15.88:0/1639286264 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072c009690 con 0x7f071c0039b0 2024-09-16T01:35:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.097+0000 7f073c35f640 1 -- 172.21.15.88:0/1639286264 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f071c0039b0 msgr2=0x7f071c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.097+0000 7f073c35f640 1 --2- 172.21.15.88:0/1639286264 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f071c0039b0 0x7f071c003dd0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f072c009ac0 tx=0x7f071c005830 comp rx=0 tx=0).stop 2024-09-16T01:35:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.097+0000 7f073a8fc640 1 -- 172.21.15.88:0/1639286264 reap_dead start 2024-09-16T01:35:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.097+0000 7f073c35f640 1 -- 172.21.15.88:0/1639286264 shutdown_connections 2024-09-16T01:35:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.097+0000 7f073c35f640 1 -- 172.21.15.88:0/1639286264 >> 172.21.15.88:0/1639286264 conn(0x7f07340d14d0 msgr2=0x7f07340d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.097+0000 7f073c35f640 1 -- 172.21.15.88:0/1639286264 shutdown_connections 2024-09-16T01:35:05.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.097+0000 7f073c35f640 1 -- 172.21.15.88:0/1639286264 wait complete. 2024-09-16T01:35:05.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.098+0000 7f073c35f640 1 Processor -- start 2024-09-16T01:35:05.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.098+0000 7f073c35f640 1 -- start start 2024-09-16T01:35:05.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.099+0000 7f073c35f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f071c0039b0 0x7f0734152da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:05.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.099+0000 7f073c35f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0734153a50 0x7f0734145fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:05.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.099+0000 7f073c35f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0734155160 0x7f0734155610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:05.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.099+0000 7f073c35f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f07340735b0 con 0x7f071c0039b0 2024-09-16T01:35:05.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.099+0000 7f073c35f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f07340732d0 con 0x7f0734155160 2024-09-16T01:35:05.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.099+0000 7f073c35f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f073406f850 con 0x7f0734153a50 2024-09-16T01:35:05.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.099+0000 7f073a0fb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f071c0039b0 0x7f0734152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:05.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.099+0000 7f073a0fb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f071c0039b0 0x7f0734152da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36516/0 (socket says 172.21.15.88:36516) 2024-09-16T01:35:05.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.099+0000 7f073a0fb640 1 -- 172.21.15.88:0/3155996202 learned_addr learned my addr 172.21.15.88:0/3155996202 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:05.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.099+0000 7f07398fa640 1 --2- 172.21.15.88:0/3155996202 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0734153a50 0x7f0734145fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:05.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.099+0000 7f073a8fc640 1 --2- 172.21.15.88:0/3155996202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0734155160 0x7f0734155610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:05.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.099+0000 7f073a0fb640 1 -- 172.21.15.88:0/3155996202 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0734153a50 msgr2=0x7f0734145fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:05.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.099+0000 7f073a0fb640 1 --2- 172.21.15.88:0/3155996202 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0734153a50 0x7f0734145fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:05.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.099+0000 7f073a0fb640 1 -- 172.21.15.88:0/3155996202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0734155160 msgr2=0x7f0734155610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:05.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.099+0000 7f073a0fb640 1 --2- 172.21.15.88:0/3155996202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0734155160 0x7f0734155610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:05.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.099+0000 7f073a0fb640 1 -- 172.21.15.88:0/3155996202 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07341540c0 con 0x7f071c0039b0 2024-09-16T01:35:05.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.100+0000 7f073a0fb640 1 --2- 172.21.15.88:0/3155996202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f071c0039b0 0x7f0734152da0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f072c009be0 tx=0x7f072c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:05.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.100+0000 7f0732ffd640 1 -- 172.21.15.88:0/3155996202 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072c040020 con 0x7f071c0039b0 2024-09-16T01:35:05.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.100+0000 7f0732ffd640 1 -- 172.21.15.88:0/3155996202 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f072c003b60 con 0x7f071c0039b0 2024-09-16T01:35:05.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.100+0000 7f0732ffd640 1 -- 172.21.15.88:0/3155996202 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072c036040 con 0x7f071c0039b0 2024-09-16T01:35:05.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.100+0000 7f073c35f640 1 -- 172.21.15.88:0/3155996202 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07341543a0 con 0x7f071c0039b0 2024-09-16T01:35:05.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.100+0000 7f073c35f640 1 -- 172.21.15.88:0/3155996202 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0734154960 con 0x7f071c0039b0 2024-09-16T01:35:05.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.101+0000 7f073c35f640 1 -- 172.21.15.88:0/3155996202 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f073406bff0 con 0x7f071c0039b0 2024-09-16T01:35:05.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.101+0000 7f0732ffd640 1 -- 172.21.15.88:0/3155996202 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f072c043020 con 0x7f071c0039b0 2024-09-16T01:35:05.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.102+0000 7f0732ffd640 1 --2- 172.21.15.88:0/3155996202 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f072006c2b0 0x7f072006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:05.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.102+0000 7f0732ffd640 1 -- 172.21.15.88:0/3155996202 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f072c0ab4d0 con 0x7f071c0039b0 2024-09-16T01:35:05.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.102+0000 7f07398fa640 1 --2- 172.21.15.88:0/3155996202 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f072006c2b0 0x7f072006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:05.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.102+0000 7f07398fa640 1 --2- 172.21.15.88:0/3155996202 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f072006c2b0 0x7f072006e770 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0728007fd0 tx=0x7f0728009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:05.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.106+0000 7f0732ffd640 1 -- 172.21.15.88:0/3155996202 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f072c04a070 con 0x7f071c0039b0 2024-09-16T01:35:05.313 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:35:05.312+0000 7f176e854640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:35:05.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.335+0000 7f073c35f640 1 -- 172.21.15.88:0/3155996202 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f07340b3f70 con 0x7f071c0039b0 2024-09-16T01:35:05.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.335+0000 7f0732ffd640 1 -- 172.21.15.88:0/3155996202 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1138 (secure 0 0 0) 0x7f072c0a9e20 con 0x7f071c0039b0 2024-09-16T01:35:05.337 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:35:05.337 INFO:teuthology.orchestra.run.smithi088.stdout:{"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":66,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":32,"num_osds":8,"num_up_osds":8,"osd_up_since":1726450450,"num_in_osds":8,"osd_in_since":1726450440,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":54},{"state_name":"active+clean","count":43}],"num_pgs":97,"num_pools":3,"num_objects":4,"data_bytes":590458,"bytes_used":225189888,"bytes_avail":772868923392,"bytes_total":773094113280,"inactive_pgs_ratio":0.55670100450515747,"write_bytes_sec":255,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":13,"btime":"2024-09-16T01:35:03:353018+0000","id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"c","status":"up:active","gid":4830}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-16T01:33:58.413617+0000","services":{}},"progress_events":{}} 2024-09-16T01:35:05.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.337+0000 7f073c35f640 1 -- 172.21.15.88:0/3155996202 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f072006c2b0 msgr2=0x7f072006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:05.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.337+0000 7f073c35f640 1 --2- 172.21.15.88:0/3155996202 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f072006c2b0 0x7f072006e770 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0728007fd0 tx=0x7f0728009040 comp rx=0 tx=0).stop 2024-09-16T01:35:05.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.337+0000 7f073c35f640 1 -- 172.21.15.88:0/3155996202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f071c0039b0 msgr2=0x7f0734152da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:05.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.337+0000 7f073c35f640 1 --2- 172.21.15.88:0/3155996202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f071c0039b0 0x7f0734152da0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f072c009be0 tx=0x7f072c004730 comp rx=0 tx=0).stop 2024-09-16T01:35:05.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.337+0000 7f073c35f640 1 -- 172.21.15.88:0/3155996202 shutdown_connections 2024-09-16T01:35:05.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.337+0000 7f073c35f640 1 --2- 172.21.15.88:0/3155996202 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f072006c2b0 0x7f072006e770 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:05.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.337+0000 7f073c35f640 1 --2- 172.21.15.88:0/3155996202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0734155160 0x7f0734155610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:05.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.337+0000 7f073c35f640 1 --2- 172.21.15.88:0/3155996202 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0734153a50 0x7f0734145fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:05.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.337+0000 7f073c35f640 1 --2- 172.21.15.88:0/3155996202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f071c0039b0 0x7f0734152da0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:05.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.337+0000 7f073c35f640 1 -- 172.21.15.88:0/3155996202 >> 172.21.15.88:0/3155996202 conn(0x7f07340d14d0 msgr2=0x7f0734072050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:05.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.337+0000 7f073c35f640 1 -- 172.21.15.88:0/3155996202 shutdown_connections 2024-09-16T01:35:05.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.337+0000 7f073c35f640 1 -- 172.21.15.88:0/3155996202 wait complete. 2024-09-16T01:35:05.349 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-16T01:35:05.349 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:35:05.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.446+0000 7fdfd2713640 1 Processor -- start 2024-09-16T01:35:05.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.446+0000 7fdfd2713640 1 -- start start 2024-09-16T01:35:05.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.447+0000 7fdfd2713640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdfcc15c170 0x7fdfcc15c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:05.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.447+0000 7fdfd2713640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdfcc0cf6c0 0x7fdfcc0cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:05.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.447+0000 7fdfd2713640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdfcc0b5dd0 0x7fdfcc0b6280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:05.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.447+0000 7fdfd2713640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fdfcc15ca90 con 0x7fdfcc0cf6c0 2024-09-16T01:35:05.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.447+0000 7fdfd2713640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fdfcc0b4cf0 con 0x7fdfcc15c170 2024-09-16T01:35:05.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.447+0000 7fdfd2713640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdfcc0b4fd0 con 0x7fdfcc0b5dd0 2024-09-16T01:35:05.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.447+0000 7fdfcb7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdfcc0cf6c0 0x7fdfcc0cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:05.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.447+0000 7fdfcb7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdfcc0cf6c0 0x7fdfcc0cfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36530/0 (socket says 172.21.15.88:36530) 2024-09-16T01:35:05.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.447+0000 7fdfcb7fe640 1 -- 172.21.15.88:0/1526654308 learned_addr learned my addr 172.21.15.88:0/1526654308 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:05.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.448+0000 7fdfcb7fe640 1 -- 172.21.15.88:0/1526654308 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdfcc0b5dd0 msgr2=0x7fdfcc0b6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:05.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.448+0000 7fdfd0cb0640 1 --2- 172.21.15.88:0/1526654308 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdfcc0b5dd0 0x7fdfcc0b6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:05.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.448+0000 7fdfcb7fe640 1 --2- 172.21.15.88:0/1526654308 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdfcc0b5dd0 0x7fdfcc0b6280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:05.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.448+0000 7fdfcbfff640 1 --2- 172.21.15.88:0/1526654308 >> v2:172.21.15.154:3300/0 conn(0x7fdfcc15c170 0x7fdfcc15c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:05.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.448+0000 7fdfcb7fe640 1 -- 172.21.15.88:0/1526654308 >> v2:172.21.15.154:3300/0 conn(0x7fdfcc15c170 msgr2=0x7fdfcc15c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:05.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.448+0000 7fdfcb7fe640 1 --2- 172.21.15.88:0/1526654308 >> v2:172.21.15.154:3300/0 conn(0x7fdfcc15c170 0x7fdfcc15c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:05.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.448+0000 7fdfcb7fe640 1 -- 172.21.15.88:0/1526654308 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfcc0b52b0 con 0x7fdfcc0cf6c0 2024-09-16T01:35:05.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.448+0000 7fdfcb7fe640 1 --2- 172.21.15.88:0/1526654308 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfcc0cf6c0 0x7fdfcc0cfb30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0009870 tx=0x7fdfc002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c00f2889464a22b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:05.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.449+0000 7fdfcaffd640 1 -- 172.21.15.88:0/1526654308 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc0037070 con 0x7fdfcc0cf6c0 2024-09-16T01:35:05.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.449+0000 7fdfcaffd640 1 -- 172.21.15.88:0/1526654308 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfc0032440 con 0x7fdfcc0cf6c0 2024-09-16T01:35:05.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.449+0000 7fdfcaffd640 1 -- 172.21.15.88:0/1526654308 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc0032810 con 0x7fdfcc0cf6c0 2024-09-16T01:35:05.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.449+0000 7fdfd2713640 1 -- 172.21.15.88:0/1526654308 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfcc0cf6c0 msgr2=0x7fdfcc0cfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:05.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.449+0000 7fdfd2713640 1 --2- 172.21.15.88:0/1526654308 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfcc0cf6c0 0x7fdfcc0cfb30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0009870 tx=0x7fdfc002a210 comp rx=0 tx=0).stop 2024-09-16T01:35:05.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.449+0000 7fdfd2713640 1 -- 172.21.15.88:0/1526654308 shutdown_connections 2024-09-16T01:35:05.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.449+0000 7fdfd2713640 1 --2- 172.21.15.88:0/1526654308 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdfcc0b5dd0 0x7fdfcc0b6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:05.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.449+0000 7fdfd2713640 1 --2- 172.21.15.88:0/1526654308 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfcc0cf6c0 0x7fdfcc0cfb30 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:05.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.449+0000 7fdfd2713640 1 --2- 172.21.15.88:0/1526654308 >> v2:172.21.15.154:3300/0 conn(0x7fdfcc15c170 0x7fdfcc15c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:05.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.449+0000 7fdfd2713640 1 -- 172.21.15.88:0/1526654308 >> 172.21.15.88:0/1526654308 conn(0x7fdfcc06bc10 msgr2=0x7fdfcc06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:05.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.449+0000 7fdfd2713640 1 -- 172.21.15.88:0/1526654308 shutdown_connections 2024-09-16T01:35:05.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.449+0000 7fdfd2713640 1 -- 172.21.15.88:0/1526654308 wait complete. 2024-09-16T01:35:05.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.450+0000 7fdfd2713640 1 Processor -- start 2024-09-16T01:35:05.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.451+0000 7fdfd2713640 1 -- start start 2024-09-16T01:35:05.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.451+0000 7fdfd2713640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdfcc0b5dd0 0x7fdfcc14b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:05.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.451+0000 7fdfd2713640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdfcc0cf6c0 0x7fdfcc157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:05.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.451+0000 7fdfd2713640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfcc15c170 0x7fdfcc15a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:05.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.451+0000 7fdfd2713640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdfcc0b5b50 con 0x7fdfcc15c170 2024-09-16T01:35:05.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.451+0000 7fdfd2713640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdfcc0b5870 con 0x7fdfcc0b5dd0 2024-09-16T01:35:05.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.451+0000 7fdfd2713640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdfcc15ffb0 con 0x7fdfcc0cf6c0 2024-09-16T01:35:05.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.451+0000 7fdfd0cb0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfcc15c170 0x7fdfcc15a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:05.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.451+0000 7fdfd0cb0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfcc15c170 0x7fdfcc15a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36532/0 (socket says 172.21.15.88:36532) 2024-09-16T01:35:05.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.451+0000 7fdfd0cb0640 1 -- 172.21.15.88:0/21387929 learned_addr learned my addr 172.21.15.88:0/21387929 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:05.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.451+0000 7fdfcb7fe640 1 --2- 172.21.15.88:0/21387929 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdfcc0cf6c0 0x7fdfcc157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:05.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.451+0000 7fdfcbfff640 1 --2- 172.21.15.88:0/21387929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdfcc0b5dd0 0x7fdfcc14b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:05.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.452+0000 7fdfd0cb0640 1 -- 172.21.15.88:0/21387929 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdfcc0cf6c0 msgr2=0x7fdfcc157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:05.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.452+0000 7fdfd0cb0640 1 --2- 172.21.15.88:0/21387929 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdfcc0cf6c0 0x7fdfcc157ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:05.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.452+0000 7fdfd0cb0640 1 -- 172.21.15.88:0/21387929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdfcc0b5dd0 msgr2=0x7fdfcc14b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:05.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.452+0000 7fdfd0cb0640 1 --2- 172.21.15.88:0/21387929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdfcc0b5dd0 0x7fdfcc14b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:05.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.452+0000 7fdfd0cb0640 1 -- 172.21.15.88:0/21387929 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfcc1561e0 con 0x7fdfcc15c170 2024-09-16T01:35:05.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.452+0000 7fdfd0cb0640 1 --2- 172.21.15.88:0/21387929 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfcc15c170 0x7fdfcc15a8e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400cd50 tx=0x7fdfc400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:05.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.452+0000 7fdfc8ff9640 1 -- 172.21.15.88:0/21387929 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4003b90 con 0x7fdfcc15c170 2024-09-16T01:35:05.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.452+0000 7fdfc8ff9640 1 -- 172.21.15.88:0/21387929 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfc4003d30 con 0x7fdfcc15c170 2024-09-16T01:35:05.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.452+0000 7fdfc8ff9640 1 -- 172.21.15.88:0/21387929 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4004240 con 0x7fdfcc15c170 2024-09-16T01:35:05.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.452+0000 7fdfd2713640 1 -- 172.21.15.88:0/21387929 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfcc1564c0 con 0x7fdfcc15c170 2024-09-16T01:35:05.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.452+0000 7fdfd2713640 1 -- 172.21.15.88:0/21387929 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfcc156a80 con 0x7fdfcc15c170 2024-09-16T01:35:05.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.453+0000 7fdfd2713640 1 -- 172.21.15.88:0/21387929 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfcc15c550 con 0x7fdfcc15c170 2024-09-16T01:35:05.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.453+0000 7fdfc8ff9640 1 -- 172.21.15.88:0/21387929 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fdfc40043e0 con 0x7fdfcc15c170 2024-09-16T01:35:05.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.454+0000 7fdfc8ff9640 1 --2- 172.21.15.88:0/21387929 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdfb806c2e0 0x7fdfb806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:05.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.454+0000 7fdfcbfff640 1 --2- 172.21.15.88:0/21387929 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdfb806c2e0 0x7fdfb806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:05.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.454+0000 7fdfc8ff9640 1 -- 172.21.15.88:0/21387929 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fdfc408bce0 con 0x7fdfcc15c170 2024-09-16T01:35:05.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.454+0000 7fdfcbfff640 1 --2- 172.21.15.88:0/21387929 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdfb806c2e0 0x7fdfb806e7a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc00aa20 tx=0x7fdfbc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:05.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.458+0000 7fdfc8ff9640 1 -- 172.21.15.88:0/21387929 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fdfc4059e00 con 0x7fdfcc15c170 2024-09-16T01:35:05.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.619+0000 7fdfd2713640 1 -- 172.21.15.88:0/21387929 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdfcc0cf1c0 con 0x7fdfb806c2e0 2024-09-16T01:35:05.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.625+0000 7fdfc8ff9640 1 -- 172.21.15.88:0/21387929 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+247449 (secure 0 0 0) 0x7fdfcc0cf1c0 con 0x7fdfb806c2e0 2024-09-16T01:35:05.629 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:35:05.630 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:35:05.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.628+0000 7fdfd2713640 1 -- 172.21.15.88:0/21387929 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdfb806c2e0 msgr2=0x7fdfb806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:05.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.628+0000 7fdfd2713640 1 --2- 172.21.15.88:0/21387929 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdfb806c2e0 0x7fdfb806e7a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc00aa20 tx=0x7fdfbc008040 comp rx=0 tx=0).stop 2024-09-16T01:35:05.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.628+0000 7fdfd2713640 1 -- 172.21.15.88:0/21387929 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfcc15c170 msgr2=0x7fdfcc15a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:05.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.628+0000 7fdfd2713640 1 --2- 172.21.15.88:0/21387929 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfcc15c170 0x7fdfcc15a8e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400cd50 tx=0x7fdfc400a660 comp rx=0 tx=0).stop 2024-09-16T01:35:05.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.628+0000 7fdfd2713640 1 -- 172.21.15.88:0/21387929 shutdown_connections 2024-09-16T01:35:05.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.628+0000 7fdfd2713640 1 --2- 172.21.15.88:0/21387929 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdfb806c2e0 0x7fdfb806e7a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:05.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.628+0000 7fdfd2713640 1 --2- 172.21.15.88:0/21387929 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdfcc15c170 0x7fdfcc15a8e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:05.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.628+0000 7fdfd2713640 1 --2- 172.21.15.88:0/21387929 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdfcc0cf6c0 0x7fdfcc157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:05.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.628+0000 7fdfd2713640 1 --2- 172.21.15.88:0/21387929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdfcc0b5dd0 0x7fdfcc14b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:05.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.628+0000 7fdfd2713640 1 -- 172.21.15.88:0/21387929 >> 172.21.15.88:0/21387929 conn(0x7fdfcc06bc10 msgr2=0x7fdfcc0b3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:05.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.628+0000 7fdfd2713640 1 -- 172.21.15.88:0/21387929 shutdown_connections 2024-09-16T01:35:05.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:05.628+0000 7fdfd2713640 1 -- 172.21.15.88:0/21387929 wait complete. 2024-09-16T01:35:05.640 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-09-16T01:35:04.524493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590458,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":163,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":28,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219912,"kb_used_data":3680,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754808,"statfs":{"total":773094113280,"available":772868923392,"internally_reserved":0,"allocated":3768320,"data_stored":2133950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.198029"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337649+0000","last_change":"2024-09-16T01:35:00.337649+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337649+0000","last_undegraded":"2024-09-16T01:35:00.337649+0000","last_fullsized":"2024-09-16T01:35:00.337649+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339909+0000","last_change":"2024-09-16T01:35:00.339909+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339909+0000","last_undegraded":"2024-09-16T01:35:00.339909+0000","last_fullsized":"2024-09-16T01:35:00.339909+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.357429+0000","last_change":"2024-09-16T01:35:01.353507+0000","last_active":"2024-09-16T01:35:01.357429+0000","last_peered":"2024-09-16T01:35:01.357429+0000","last_clean":"2024-09-16T01:35:01.357429+0000","last_became_active":"2024-09-16T01:35:01.352327+0000","last_became_peered":"2024-09-16T01:35:01.352327+0000","last_unstale":"2024-09-16T01:35:01.357429+0000","last_undegraded":"2024-09-16T01:35:01.357429+0000","last_fullsized":"2024-09-16T01:35:01.357429+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:55.644944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.340201+0000","last_change":"2024-09-16T01:35:00.340201+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.340201+0000","last_undegraded":"2024-09-16T01:35:00.340201+0000","last_fullsized":"2024-09-16T01:35:00.340201+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.341590+0000","last_change":"2024-09-16T01:35:00.341590+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.341590+0000","last_undegraded":"2024-09-16T01:35:00.341590+0000","last_fullsized":"2024-09-16T01:35:00.341590+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338751+0000","last_change":"2024-09-16T01:35:00.338751+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338751+0000","last_undegraded":"2024-09-16T01:35:00.338751+0000","last_fullsized":"2024-09-16T01:35:00.338751+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.345760+0000","last_change":"2024-09-16T01:35:00.345760+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.345760+0000","last_undegraded":"2024-09-16T01:35:00.345760+0000","last_fullsized":"2024-09-16T01:35:00.345760+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.347744+0000","last_change":"2024-09-16T01:35:00.347744+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.347744+0000","last_undegraded":"2024-09-16T01:35:00.347744+0000","last_fullsized":"2024-09-16T01:35:00.347744+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337574+0000","last_change":"2024-09-16T01:35:00.337574+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337574+0000","last_undegraded":"2024-09-16T01:35:00.337574+0000","last_fullsized":"2024-09-16T01:35:00.337574+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339626+0000","last_change":"2024-09-16T01:35:00.339626+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339626+0000","last_undegraded":"2024-09-16T01:35:00.339626+0000","last_fullsized":"2024-09-16T01:35:00.339626+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.344537+0000","last_change":"2024-09-16T01:35:00.344537+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.344537+0000","last_undegraded":"2024-09-16T01:35:00.344537+0000","last_fullsized":"2024-09-16T01:35:00.344537+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.354548+0000","last_change":"2024-09-16T01:35:01.349955+0000","last_active":"2024-09-16T01:35:01.354548+0000","last_peered":"2024-09-16T01:35:01.354548+0000","last_clean":"2024-09-16T01:35:01.354548+0000","last_became_active":"2024-09-16T01:35:01.349689+0000","last_became_peered":"2024-09-16T01:35:01.349689+0000","last_unstale":"2024-09-16T01:35:01.354548+0000","last_undegraded":"2024-09-16T01:35:01.354548+0000","last_fullsized":"2024-09-16T01:35:01.354548+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:02:10.872199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.341163+0000","last_change":"2024-09-16T01:35:00.341163+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.341163+0000","last_undegraded":"2024-09-16T01:35:00.341163+0000","last_fullsized":"2024-09-16T01:35:00.341163+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337577+0000","last_change":"2024-09-16T01:35:00.337577+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337577+0000","last_undegraded":"2024-09-16T01:35:00.337577+0000","last_fullsized":"2024-09-16T01:35:00.337577+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.353724+0000","last_change":"2024-09-16T01:35:01.344630+0000","last_active":"2024-09-16T01:35:01.353724+0000","last_peered":"2024-09-16T01:35:01.353724+0000","last_clean":"2024-09-16T01:35:01.353724+0000","last_became_active":"2024-09-16T01:35:01.344340+0000","last_became_peered":"2024-09-16T01:35:01.344340+0000","last_unstale":"2024-09-16T01:35:01.353724+0000","last_undegraded":"2024-09-16T01:35:01.353724+0000","last_fullsized":"2024-09-16T01:35:01.353724+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:06:54.456413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.342533+0000","last_change":"2024-09-16T01:35:00.342533+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.342533+0000","last_undegraded":"2024-09-16T01:35:00.342533+0000","last_fullsized":"2024-09-16T01:35:00.342533+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.343962+0000","last_change":"2024-09-16T01:35:00.343962+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.343962+0000","last_undegraded":"2024-09-16T01:35:00.343962+0000","last_fullsized":"2024-09-16T01:35:00.343962+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339533+0000","last_change":"2024-09-16T01:35:00.339533+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339533+0000","last_undegraded":"2024-09-16T01:35:00.339533+0000","last_fullsized":"2024-09-16T01:35:00.339533+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337206+0000","last_change":"2024-09-16T01:35:00.337206+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337206+0000","last_undegraded":"2024-09-16T01:35:00.337206+0000","last_fullsized":"2024-09-16T01:35:00.337206+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.340512+0000","last_change":"2024-09-16T01:35:00.340512+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.340512+0000","last_undegraded":"2024-09-16T01:35:00.340512+0000","last_fullsized":"2024-09-16T01:35:00.340512+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339909+0000","last_change":"2024-09-16T01:35:00.339909+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339909+0000","last_undegraded":"2024-09-16T01:35:00.339909+0000","last_fullsized":"2024-09-16T01:35:00.339909+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338369+0000","last_change":"2024-09-16T01:35:00.338369+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338369+0000","last_undegraded":"2024-09-16T01:35:00.338369+0000","last_fullsized":"2024-09-16T01:35:00.338369+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.335686+0000","last_change":"2024-09-16T01:35:00.335686+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.335686+0000","last_undegraded":"2024-09-16T01:35:00.335686+0000","last_fullsized":"2024-09-16T01:35:00.335686+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.353084+0000","last_change":"2024-09-16T01:35:01.344447+0000","last_active":"2024-09-16T01:35:01.353084+0000","last_peered":"2024-09-16T01:35:01.353084+0000","last_clean":"2024-09-16T01:35:01.353084+0000","last_became_active":"2024-09-16T01:35:01.344135+0000","last_became_peered":"2024-09-16T01:35:01.344135+0000","last_unstale":"2024-09-16T01:35:01.353084+0000","last_undegraded":"2024-09-16T01:35:01.353084+0000","last_fullsized":"2024-09-16T01:35:01.353084+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:44:42.157718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338640+0000","last_change":"2024-09-16T01:35:00.338640+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338640+0000","last_undegraded":"2024-09-16T01:35:00.338640+0000","last_fullsized":"2024-09-16T01:35:00.338640+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339123+0000","last_change":"2024-09-16T01:35:00.339123+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339123+0000","last_undegraded":"2024-09-16T01:35:00.339123+0000","last_fullsized":"2024-09-16T01:35:00.339123+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337376+0000","last_change":"2024-09-16T01:35:00.337376+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337376+0000","last_undegraded":"2024-09-16T01:35:00.337376+0000","last_fullsized":"2024-09-16T01:35:00.337376+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.352090+0000","last_change":"2024-09-16T01:35:01.343532+0000","last_active":"2024-09-16T01:35:01.352090+0000","last_peered":"2024-09-16T01:35:01.352090+0000","last_clean":"2024-09-16T01:35:01.352090+0000","last_became_active":"2024-09-16T01:35:01.342495+0000","last_became_peered":"2024-09-16T01:35:01.342495+0000","last_unstale":"2024-09-16T01:35:01.352090+0000","last_undegraded":"2024-09-16T01:35:01.352090+0000","last_fullsized":"2024-09-16T01:35:01.352090+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:32:33.652857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339456+0000","last_change":"2024-09-16T01:35:00.339456+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339456+0000","last_undegraded":"2024-09-16T01:35:00.339456+0000","last_fullsized":"2024-09-16T01:35:00.339456+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.343049+0000","last_change":"2024-09-16T01:35:00.343049+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.343049+0000","last_undegraded":"2024-09-16T01:35:00.343049+0000","last_fullsized":"2024-09-16T01:35:00.343049+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338353+0000","last_change":"2024-09-16T01:35:00.338353+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338353+0000","last_undegraded":"2024-09-16T01:35:00.338353+0000","last_fullsized":"2024-09-16T01:35:00.338353+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.347252+0000","last_change":"2024-09-16T01:35:00.347252+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.347252+0000","last_undegraded":"2024-09-16T01:35:00.347252+0000","last_fullsized":"2024-09-16T01:35:00.347252+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.335647+0000","last_change":"2024-09-16T01:35:00.335647+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.335647+0000","last_undegraded":"2024-09-16T01:35:00.335647+0000","last_fullsized":"2024-09-16T01:35:00.335647+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.358487+0000","last_change":"2024-09-16T01:35:00.355732+0000","last_active":"2024-09-16T01:35:00.358487+0000","last_peered":"2024-09-16T01:35:00.358487+0000","last_clean":"2024-09-16T01:35:00.358487+0000","last_became_active":"2024-09-16T01:35:00.355431+0000","last_became_peered":"2024-09-16T01:35:00.355431+0000","last_unstale":"2024-09-16T01:35:00.358487+0000","last_undegraded":"2024-09-16T01:35:00.358487+0000","last_fullsized":"2024-09-16T01:35:00.358487+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:48:54.665338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337125+0000","last_change":"2024-09-16T01:35:00.337125+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337125+0000","last_undegraded":"2024-09-16T01:35:00.337125+0000","last_fullsized":"2024-09-16T01:35:00.337125+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352461+0000","last_change":"2024-09-16T01:35:00.348929+0000","last_active":"2024-09-16T01:35:00.352461+0000","last_peered":"2024-09-16T01:35:00.352461+0000","last_clean":"2024-09-16T01:35:00.352461+0000","last_became_active":"2024-09-16T01:35:00.348651+0000","last_became_peered":"2024-09-16T01:35:00.348651+0000","last_unstale":"2024-09-16T01:35:00.352461+0000","last_undegraded":"2024-09-16T01:35:00.352461+0000","last_fullsized":"2024-09-16T01:35:00.352461+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:52:02.746213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339423+0000","last_change":"2024-09-16T01:35:00.339423+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339423+0000","last_undegraded":"2024-09-16T01:35:00.339423+0000","last_fullsized":"2024-09-16T01:35:00.339423+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.346035+0000","last_change":"2024-09-16T01:35:00.344907+0000","last_active":"2024-09-16T01:35:00.346035+0000","last_peered":"2024-09-16T01:35:00.346035+0000","last_clean":"2024-09-16T01:35:00.346035+0000","last_became_active":"2024-09-16T01:35:00.344634+0000","last_became_peered":"2024-09-16T01:35:00.344634+0000","last_unstale":"2024-09-16T01:35:00.346035+0000","last_undegraded":"2024-09-16T01:35:00.346035+0000","last_fullsized":"2024-09-16T01:35:00.346035+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:00.732201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.336922+0000","last_change":"2024-09-16T01:35:00.336922+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.336922+0000","last_undegraded":"2024-09-16T01:35:00.336922+0000","last_fullsized":"2024-09-16T01:35:00.336922+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352455+0000","last_change":"2024-09-16T01:35:00.348009+0000","last_active":"2024-09-16T01:35:00.352455+0000","last_peered":"2024-09-16T01:35:00.352455+0000","last_clean":"2024-09-16T01:35:00.352455+0000","last_became_active":"2024-09-16T01:35:00.347736+0000","last_became_peered":"2024-09-16T01:35:00.347736+0000","last_unstale":"2024-09-16T01:35:00.352455+0000","last_undegraded":"2024-09-16T01:35:00.352455+0000","last_fullsized":"2024-09-16T01:35:00.352455+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:44.087710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.342570+0000","last_change":"2024-09-16T01:35:00.342570+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.342570+0000","last_undegraded":"2024-09-16T01:35:00.342570+0000","last_fullsized":"2024-09-16T01:35:00.342570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353110+0000","last_change":"2024-09-16T01:35:00.348379+0000","last_active":"2024-09-16T01:35:00.353110+0000","last_peered":"2024-09-16T01:35:00.353110+0000","last_clean":"2024-09-16T01:35:00.353110+0000","last_became_active":"2024-09-16T01:35:00.348008+0000","last_became_peered":"2024-09-16T01:35:00.348008+0000","last_unstale":"2024-09-16T01:35:00.353110+0000","last_undegraded":"2024-09-16T01:35:00.353110+0000","last_fullsized":"2024-09-16T01:35:00.353110+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:56:53.528942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.354400+0000","last_change":"2024-09-16T01:35:01.346165+0000","last_active":"2024-09-16T01:35:01.354400+0000","last_peered":"2024-09-16T01:35:01.354400+0000","last_clean":"2024-09-16T01:35:01.354400+0000","last_became_active":"2024-09-16T01:35:01.344479+0000","last_became_peered":"2024-09-16T01:35:01.344479+0000","last_unstale":"2024-09-16T01:35:01.354400+0000","last_undegraded":"2024-09-16T01:35:01.354400+0000","last_fullsized":"2024-09-16T01:35:01.354400+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:58:00.280748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353081+0000","last_change":"2024-09-16T01:35:00.347395+0000","last_active":"2024-09-16T01:35:00.353081+0000","last_peered":"2024-09-16T01:35:00.353081+0000","last_clean":"2024-09-16T01:35:00.353081+0000","last_became_active":"2024-09-16T01:35:00.346752+0000","last_became_peered":"2024-09-16T01:35:00.346752+0000","last_unstale":"2024-09-16T01:35:00.353081+0000","last_undegraded":"2024-09-16T01:35:00.353081+0000","last_fullsized":"2024-09-16T01:35:00.353081+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:51.973054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338296+0000","last_change":"2024-09-16T01:35:00.338296+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338296+0000","last_undegraded":"2024-09-16T01:35:00.338296+0000","last_fullsized":"2024-09-16T01:35:00.338296+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352488+0000","last_change":"2024-09-16T01:35:00.348232+0000","last_active":"2024-09-16T01:35:00.352488+0000","last_peered":"2024-09-16T01:35:00.352488+0000","last_clean":"2024-09-16T01:35:00.352488+0000","last_became_active":"2024-09-16T01:35:00.347976+0000","last_became_peered":"2024-09-16T01:35:00.347976+0000","last_unstale":"2024-09-16T01:35:00.352488+0000","last_undegraded":"2024-09-16T01:35:00.352488+0000","last_fullsized":"2024-09-16T01:35:00.352488+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:49:23.770293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.336753+0000","last_change":"2024-09-16T01:35:00.336753+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.336753+0000","last_undegraded":"2024-09-16T01:35:00.336753+0000","last_fullsized":"2024-09-16T01:35:00.336753+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.347485+0000","last_change":"2024-09-16T01:35:00.342334+0000","last_active":"2024-09-16T01:35:00.347485+0000","last_peered":"2024-09-16T01:35:00.347485+0000","last_clean":"2024-09-16T01:35:00.347485+0000","last_became_active":"2024-09-16T01:35:00.342056+0000","last_became_peered":"2024-09-16T01:35:00.342056+0000","last_unstale":"2024-09-16T01:35:00.347485+0000","last_undegraded":"2024-09-16T01:35:00.347485+0000","last_fullsized":"2024-09-16T01:35:00.347485+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:32:43.356912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.355290+0000","last_change":"2024-09-16T01:35:01.349973+0000","last_active":"2024-09-16T01:35:01.355290+0000","last_peered":"2024-09-16T01:35:01.355290+0000","last_clean":"2024-09-16T01:35:01.355290+0000","last_became_active":"2024-09-16T01:35:01.349733+0000","last_became_peered":"2024-09-16T01:35:01.349733+0000","last_unstale":"2024-09-16T01:35:01.355290+0000","last_undegraded":"2024-09-16T01:35:01.355290+0000","last_fullsized":"2024-09-16T01:35:01.355290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:59:24.976252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352519+0000","last_change":"2024-09-16T01:35:00.349540+0000","last_active":"2024-09-16T01:35:00.352519+0000","last_peered":"2024-09-16T01:35:00.352519+0000","last_clean":"2024-09-16T01:35:00.352519+0000","last_became_active":"2024-09-16T01:35:00.349296+0000","last_became_peered":"2024-09-16T01:35:00.349296+0000","last_unstale":"2024-09-16T01:35:00.352519+0000","last_undegraded":"2024-09-16T01:35:00.352519+0000","last_fullsized":"2024-09-16T01:35:00.352519+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:53:24.648600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338365+0000","last_change":"2024-09-16T01:35:00.338365+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338365+0000","last_undegraded":"2024-09-16T01:35:00.338365+0000","last_fullsized":"2024-09-16T01:35:00.338365+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.349142+0000","last_change":"2024-09-16T01:35:00.346787+0000","last_active":"2024-09-16T01:35:00.349142+0000","last_peered":"2024-09-16T01:35:00.349142+0000","last_clean":"2024-09-16T01:35:00.349142+0000","last_became_active":"2024-09-16T01:35:00.346527+0000","last_became_peered":"2024-09-16T01:35:00.346527+0000","last_unstale":"2024-09-16T01:35:00.349142+0000","last_undegraded":"2024-09-16T01:35:00.349142+0000","last_fullsized":"2024-09-16T01:35:00.349142+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:48:54.665338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.338003+0000","last_change":"2024-09-16T01:35:00.338003+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.338003+0000","last_undegraded":"2024-09-16T01:35:00.338003+0000","last_fullsized":"2024-09-16T01:35:00.338003+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"31'3","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.362725+0000","last_change":"2024-09-16T01:35:00.352778+0000","last_active":"2024-09-16T01:35:01.362725+0000","last_peered":"2024-09-16T01:35:01.362725+0000","last_clean":"2024-09-16T01:35:01.362725+0000","last_became_active":"2024-09-16T01:35:00.352534+0000","last_became_peered":"2024-09-16T01:35:00.352534+0000","last_unstale":"2024-09-16T01:35:01.362725+0000","last_undegraded":"2024-09-16T01:35:01.362725+0000","last_fullsized":"2024-09-16T01:35:01.362725+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:32:30.768011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.357410+0000","last_change":"2024-09-16T01:35:01.353288+0000","last_active":"2024-09-16T01:35:01.357410+0000","last_peered":"2024-09-16T01:35:01.357410+0000","last_clean":"2024-09-16T01:35:01.357410+0000","last_became_active":"2024-09-16T01:35:01.352428+0000","last_became_peered":"2024-09-16T01:35:01.352428+0000","last_unstale":"2024-09-16T01:35:01.357410+0000","last_undegraded":"2024-09-16T01:35:01.357410+0000","last_fullsized":"2024-09-16T01:35:01.357410+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:48:49.160580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352983+0000","last_change":"2024-09-16T01:35:00.350506+0000","last_active":"2024-09-16T01:35:00.352983+0000","last_peered":"2024-09-16T01:35:00.352983+0000","last_clean":"2024-09-16T01:35:00.352983+0000","last_became_active":"2024-09-16T01:35:00.350215+0000","last_became_peered":"2024-09-16T01:35:00.350215+0000","last_unstale":"2024-09-16T01:35:00.352983+0000","last_undegraded":"2024-09-16T01:35:00.352983+0000","last_fullsized":"2024-09-16T01:35:00.352983+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:43:39.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339530+0000","last_change":"2024-09-16T01:35:00.339530+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339530+0000","last_undegraded":"2024-09-16T01:35:00.339530+0000","last_fullsized":"2024-09-16T01:35:00.339530+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.348029+0000","last_change":"2024-09-16T01:35:00.342977+0000","last_active":"2024-09-16T01:35:00.348029+0000","last_peered":"2024-09-16T01:35:00.348029+0000","last_clean":"2024-09-16T01:35:00.348029+0000","last_became_active":"2024-09-16T01:35:00.342699+0000","last_became_peered":"2024-09-16T01:35:00.342699+0000","last_unstale":"2024-09-16T01:35:00.348029+0000","last_undegraded":"2024-09-16T01:35:00.348029+0000","last_fullsized":"2024-09-16T01:35:00.348029+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:53:54.367989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337148+0000","last_change":"2024-09-16T01:35:00.337148+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337148+0000","last_undegraded":"2024-09-16T01:35:00.337148+0000","last_fullsized":"2024-09-16T01:35:00.337148+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.348456+0000","last_change":"2024-09-16T01:35:00.343581+0000","last_active":"2024-09-16T01:35:00.348456+0000","last_peered":"2024-09-16T01:35:00.348456+0000","last_clean":"2024-09-16T01:35:00.348456+0000","last_became_active":"2024-09-16T01:35:00.343294+0000","last_became_peered":"2024-09-16T01:35:00.343294+0000","last_unstale":"2024-09-16T01:35:00.348456+0000","last_undegraded":"2024-09-16T01:35:00.348456+0000","last_fullsized":"2024-09-16T01:35:00.348456+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:57.654721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.346877+0000","last_change":"2024-09-16T01:35:00.346877+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.346877+0000","last_undegraded":"2024-09-16T01:35:00.346877+0000","last_fullsized":"2024-09-16T01:35:00.346877+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353028+0000","last_change":"2024-09-16T01:35:00.351166+0000","last_active":"2024-09-16T01:35:00.353028+0000","last_peered":"2024-09-16T01:35:00.353028+0000","last_clean":"2024-09-16T01:35:00.353028+0000","last_became_active":"2024-09-16T01:35:00.350881+0000","last_became_peered":"2024-09-16T01:35:00.350881+0000","last_unstale":"2024-09-16T01:35:00.353028+0000","last_undegraded":"2024-09-16T01:35:00.353028+0000","last_fullsized":"2024-09-16T01:35:00.353028+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:54:05.742892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337056+0000","last_change":"2024-09-16T01:35:00.337056+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337056+0000","last_undegraded":"2024-09-16T01:35:00.337056+0000","last_fullsized":"2024-09-16T01:35:00.337056+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.350509+0000","last_change":"2024-09-16T01:35:00.345939+0000","last_active":"2024-09-16T01:35:00.350509+0000","last_peered":"2024-09-16T01:35:00.350509+0000","last_clean":"2024-09-16T01:35:00.350509+0000","last_became_active":"2024-09-16T01:35:00.345653+0000","last_became_peered":"2024-09-16T01:35:00.345653+0000","last_unstale":"2024-09-16T01:35:00.350509+0000","last_undegraded":"2024-09-16T01:35:00.350509+0000","last_fullsized":"2024-09-16T01:35:00.350509+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:57.839735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339311+0000","last_change":"2024-09-16T01:35:00.339311+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339311+0000","last_undegraded":"2024-09-16T01:35:00.339311+0000","last_fullsized":"2024-09-16T01:35:00.339311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352801+0000","last_change":"2024-09-16T01:35:00.350284+0000","last_active":"2024-09-16T01:35:00.352801+0000","last_peered":"2024-09-16T01:35:00.352801+0000","last_clean":"2024-09-16T01:35:00.352801+0000","last_became_active":"2024-09-16T01:35:00.349516+0000","last_became_peered":"2024-09-16T01:35:00.349516+0000","last_unstale":"2024-09-16T01:35:00.352801+0000","last_undegraded":"2024-09-16T01:35:00.352801+0000","last_fullsized":"2024-09-16T01:35:00.352801+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:22:55.631379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.354497+0000","last_change":"2024-09-16T01:35:01.347677+0000","last_active":"2024-09-16T01:35:01.354497+0000","last_peered":"2024-09-16T01:35:01.354497+0000","last_clean":"2024-09-16T01:35:01.354497+0000","last_became_active":"2024-09-16T01:35:01.345167+0000","last_became_peered":"2024-09-16T01:35:01.345167+0000","last_unstale":"2024-09-16T01:35:01.354497+0000","last_undegraded":"2024-09-16T01:35:01.354497+0000","last_fullsized":"2024-09-16T01:35:01.354497+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:34:25.731515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.347434+0000","last_change":"2024-09-16T01:35:00.342181+0000","last_active":"2024-09-16T01:35:00.347434+0000","last_peered":"2024-09-16T01:35:00.347434+0000","last_clean":"2024-09-16T01:35:00.347434+0000","last_became_active":"2024-09-16T01:35:00.341878+0000","last_became_peered":"2024-09-16T01:35:00.341878+0000","last_unstale":"2024-09-16T01:35:00.347434+0000","last_undegraded":"2024-09-16T01:35:00.347434+0000","last_fullsized":"2024-09-16T01:35:00.347434+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:17:19.574295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339159+0000","last_change":"2024-09-16T01:35:00.339159+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339159+0000","last_undegraded":"2024-09-16T01:35:00.339159+0000","last_fullsized":"2024-09-16T01:35:00.339159+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353150+0000","last_change":"2024-09-16T01:35:00.349566+0000","last_active":"2024-09-16T01:35:00.353150+0000","last_peered":"2024-09-16T01:35:00.353150+0000","last_clean":"2024-09-16T01:35:00.353150+0000","last_became_active":"2024-09-16T01:35:00.349243+0000","last_became_peered":"2024-09-16T01:35:00.349243+0000","last_unstale":"2024-09-16T01:35:00.353150+0000","last_undegraded":"2024-09-16T01:35:00.353150+0000","last_fullsized":"2024-09-16T01:35:00.353150+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:43:05.059186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.341010+0000","last_change":"2024-09-16T01:35:00.341010+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.341010+0000","last_undegraded":"2024-09-16T01:35:00.341010+0000","last_fullsized":"2024-09-16T01:35:00.341010+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353118+0000","last_change":"2024-09-16T01:35:00.348167+0000","last_active":"2024-09-16T01:35:00.353118+0000","last_peered":"2024-09-16T01:35:00.353118+0000","last_clean":"2024-09-16T01:35:00.353118+0000","last_became_active":"2024-09-16T01:35:00.347714+0000","last_became_peered":"2024-09-16T01:35:00.347714+0000","last_unstale":"2024-09-16T01:35:00.353118+0000","last_undegraded":"2024-09-16T01:35:00.353118+0000","last_fullsized":"2024-09-16T01:35:00.353118+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:29:35.905712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.342418+0000","last_change":"2024-09-16T01:35:00.342418+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.342418+0000","last_undegraded":"2024-09-16T01:35:00.342418+0000","last_fullsized":"2024-09-16T01:35:00.342418+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.346227+0000","last_change":"2024-09-16T01:35:00.345340+0000","last_active":"2024-09-16T01:35:00.346227+0000","last_peered":"2024-09-16T01:35:00.346227+0000","last_clean":"2024-09-16T01:35:00.346227+0000","last_became_active":"2024-09-16T01:35:00.344987+0000","last_became_peered":"2024-09-16T01:35:00.344987+0000","last_unstale":"2024-09-16T01:35:00.346227+0000","last_undegraded":"2024-09-16T01:35:00.346227+0000","last_fullsized":"2024-09-16T01:35:00.346227+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:02:54.003881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.340705+0000","last_change":"2024-09-16T01:35:00.340705+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.340705+0000","last_undegraded":"2024-09-16T01:35:00.340705+0000","last_fullsized":"2024-09-16T01:35:00.340705+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.354908+0000","last_change":"2024-09-16T01:35:00.349814+0000","last_active":"2024-09-16T01:35:00.354908+0000","last_peered":"2024-09-16T01:35:00.354908+0000","last_clean":"2024-09-16T01:35:00.354908+0000","last_became_active":"2024-09-16T01:35:00.349543+0000","last_became_peered":"2024-09-16T01:35:00.349543+0000","last_unstale":"2024-09-16T01:35:00.354908+0000","last_undegraded":"2024-09-16T01:35:00.354908+0000","last_fullsized":"2024-09-16T01:35:00.354908+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:02:54.003881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.342349+0000","last_change":"2024-09-16T01:35:00.342349+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.342349+0000","last_undegraded":"2024-09-16T01:35:00.342349+0000","last_fullsized":"2024-09-16T01:35:00.342349+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353146+0000","last_change":"2024-09-16T01:35:00.348335+0000","last_active":"2024-09-16T01:35:00.353146+0000","last_peered":"2024-09-16T01:35:00.353146+0000","last_clean":"2024-09-16T01:35:00.353146+0000","last_became_active":"2024-09-16T01:35:00.347099+0000","last_became_peered":"2024-09-16T01:35:00.347099+0000","last_unstale":"2024-09-16T01:35:00.353146+0000","last_undegraded":"2024-09-16T01:35:00.353146+0000","last_fullsized":"2024-09-16T01:35:00.353146+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:57:35.138402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.346143+0000","last_change":"2024-09-16T01:35:00.346143+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.346143+0000","last_undegraded":"2024-09-16T01:35:00.346143+0000","last_fullsized":"2024-09-16T01:35:00.346143+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.337675+0000","last_change":"2024-09-16T01:35:00.352839+0000","last_active":"2024-09-16T01:35:01.337675+0000","last_peered":"2024-09-16T01:35:01.337675+0000","last_clean":"2024-09-16T01:35:01.337675+0000","last_became_active":"2024-09-16T01:35:00.352568+0000","last_became_peered":"2024-09-16T01:35:00.352568+0000","last_unstale":"2024-09-16T01:35:01.337675+0000","last_undegraded":"2024-09-16T01:35:01.337675+0000","last_fullsized":"2024-09-16T01:35:01.337675+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:30:51.141162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.342110+0000","last_change":"2024-09-16T01:35:00.342110+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.342110+0000","last_undegraded":"2024-09-16T01:35:00.342110+0000","last_fullsized":"2024-09-16T01:35:00.342110+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.357904+0000","last_change":"2024-09-16T01:35:00.355682+0000","last_active":"2024-09-16T01:35:00.357904+0000","last_peered":"2024-09-16T01:35:00.357904+0000","last_clean":"2024-09-16T01:35:00.357904+0000","last_became_active":"2024-09-16T01:35:00.355347+0000","last_became_peered":"2024-09-16T01:35:00.355347+0000","last_unstale":"2024-09-16T01:35:00.357904+0000","last_undegraded":"2024-09-16T01:35:00.357904+0000","last_fullsized":"2024-09-16T01:35:00.357904+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:07.992492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-16T01:35:01.359762+0000","last_change":"2024-09-16T01:35:01.355640+0000","last_active":"2024-09-16T01:35:01.359762+0000","last_peered":"2024-09-16T01:35:01.359762+0000","last_clean":"2024-09-16T01:35:01.359762+0000","last_became_active":"2024-09-16T01:35:01.355364+0000","last_became_peered":"2024-09-16T01:35:01.355364+0000","last_unstale":"2024-09-16T01:35:01.359762+0000","last_undegraded":"2024-09-16T01:35:01.359762+0000","last_fullsized":"2024-09-16T01:35:01.359762+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:26:10.193445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.346212+0000","last_change":"2024-09-16T01:35:00.343346+0000","last_active":"2024-09-16T01:35:00.346212+0000","last_peered":"2024-09-16T01:35:00.346212+0000","last_clean":"2024-09-16T01:35:00.346212+0000","last_became_active":"2024-09-16T01:35:00.343049+0000","last_became_peered":"2024-09-16T01:35:00.343049+0000","last_unstale":"2024-09-16T01:35:00.346212+0000","last_undegraded":"2024-09-16T01:35:00.346212+0000","last_fullsized":"2024-09-16T01:35:00.346212+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:00.117677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.335782+0000","last_change":"2024-09-16T01:35:00.335782+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.335782+0000","last_undegraded":"2024-09-16T01:35:00.335782+0000","last_fullsized":"2024-09-16T01:35:00.335782+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352715+0000","last_change":"2024-09-16T01:35:00.349236+0000","last_active":"2024-09-16T01:35:00.352715+0000","last_peered":"2024-09-16T01:35:00.352715+0000","last_clean":"2024-09-16T01:35:00.352715+0000","last_became_active":"2024-09-16T01:35:00.348947+0000","last_became_peered":"2024-09-16T01:35:00.348947+0000","last_unstale":"2024-09-16T01:35:00.352715+0000","last_undegraded":"2024-09-16T01:35:00.352715+0000","last_fullsized":"2024-09-16T01:35:00.352715+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:18:05.552429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.336999+0000","last_change":"2024-09-16T01:35:00.336999+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.336999+0000","last_undegraded":"2024-09-16T01:35:00.336999+0000","last_fullsized":"2024-09-16T01:35:00.336999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.357770+0000","last_change":"2024-09-16T01:35:00.355723+0000","last_active":"2024-09-16T01:35:00.357770+0000","last_peered":"2024-09-16T01:35:00.357770+0000","last_clean":"2024-09-16T01:35:00.357770+0000","last_became_active":"2024-09-16T01:35:00.355425+0000","last_became_peered":"2024-09-16T01:35:00.355425+0000","last_unstale":"2024-09-16T01:35:00.357770+0000","last_undegraded":"2024-09-16T01:35:00.357770+0000","last_fullsized":"2024-09-16T01:35:00.357770+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:00.732201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.335633+0000","last_change":"2024-09-16T01:35:00.335633+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.335633+0000","last_undegraded":"2024-09-16T01:35:00.335633+0000","last_fullsized":"2024-09-16T01:35:00.335633+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.357916+0000","last_change":"2024-09-16T01:35:00.355687+0000","last_active":"2024-09-16T01:35:00.357916+0000","last_peered":"2024-09-16T01:35:00.357916+0000","last_clean":"2024-09-16T01:35:00.357916+0000","last_became_active":"2024-09-16T01:35:00.355357+0000","last_became_peered":"2024-09-16T01:35:00.355357+0000","last_unstale":"2024-09-16T01:35:00.357916+0000","last_undegraded":"2024-09-16T01:35:00.357916+0000","last_fullsized":"2024-09-16T01:35:00.357916+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:30:55.369673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.337712+0000","last_change":"2024-09-16T01:35:00.337712+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.337712+0000","last_undegraded":"2024-09-16T01:35:00.337712+0000","last_fullsized":"2024-09-16T01:35:00.337712+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353170+0000","last_change":"2024-09-16T01:35:00.351123+0000","last_active":"2024-09-16T01:35:00.353170+0000","last_peered":"2024-09-16T01:35:00.353170+0000","last_clean":"2024-09-16T01:35:00.353170+0000","last_became_active":"2024-09-16T01:35:00.350803+0000","last_became_peered":"2024-09-16T01:35:00.350803+0000","last_unstale":"2024-09-16T01:35:00.353170+0000","last_undegraded":"2024-09-16T01:35:00.353170+0000","last_fullsized":"2024-09-16T01:35:00.353170+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:51:21.665512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.339066+0000","last_change":"2024-09-16T01:35:00.339066+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.339066+0000","last_undegraded":"2024-09-16T01:35:00.339066+0000","last_fullsized":"2024-09-16T01:35:00.339066+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.353039+0000","last_change":"2024-09-16T01:35:00.351535+0000","last_active":"2024-09-16T01:35:00.353039+0000","last_peered":"2024-09-16T01:35:00.353039+0000","last_clean":"2024-09-16T01:35:00.353039+0000","last_became_active":"2024-09-16T01:35:00.351219+0000","last_became_peered":"2024-09-16T01:35:00.351219+0000","last_unstale":"2024-09-16T01:35:00.353039+0000","last_undegraded":"2024-09-16T01:35:00.353039+0000","last_fullsized":"2024-09-16T01:35:00.353039+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:04.599770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-09-16T01:35:00.342993+0000","last_change":"2024-09-16T01:35:00.342993+0000","last_active":"2024-09-16T01:35:00.326194+0000","last_peered":"2024-09-16T01:35:00.326194+0000","last_clean":"2024-09-16T01:35:00.326194+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:35:00.342993+0000","last_undegraded":"2024-09-16T01:35:00.342993+0000","last_fullsized":"2024-09-16T01:35:00.342993+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.352480+0000","last_change":"2024-09-16T01:35:00.344871+0000","last_active":"2024-09-16T01:35:00.352480+0000","last_peered":"2024-09-16T01:35:00.352480+0000","last_clean":"2024-09-16T01:35:00.352480+0000","last_became_active":"2024-09-16T01:35:00.344582+0000","last_became_peered":"2024-09-16T01:35:00.344582+0000","last_unstale":"2024-09-16T01:35:00.352480+0000","last_undegraded":"2024-09-16T01:35:00.352480+0000","last_fullsized":"2024-09-16T01:35:00.352480+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:00.117677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":185,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T01:35:00.348767+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:35:00.348767+0000","last_peered":"2024-09-16T01:35:00.348767+0000","last_clean":"2024-09-16T01:35:00.348767+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:35:00.348767+0000","last_undegraded":"2024-09-16T01:35:00.348767+0000","last_fullsized":"2024-09-16T01:35:00.348767+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509455,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":331776,"data_stored":123852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542158,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27920,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343920,"statfs":{"total":96636764160,"available":96608174080,"internally_reserved":0,"allocated":913408,"data_stored":707953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542158,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":117585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542158,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":117585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542157,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":117585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542158,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":117585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542157,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":319488,"data_stored":117585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542157,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":925696,"data_stored":714220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:35:08.645 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:35:08.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.755+0000 7efd58677640 1 Processor -- start 2024-09-16T01:35:08.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.755+0000 7efd58677640 1 -- start start 2024-09-16T01:35:08.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.756+0000 7efd58677640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd500b01f0 0x7efd501620e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:08.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.756+0000 7efd58677640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efd50073180 0x7efd500735b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:08.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.756+0000 7efd58677640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd5015e920 con 0x7efd5015e540 2024-09-16T01:35:08.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.756+0000 7efd58677640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7efd50163010 con 0x7efd50073180 2024-09-16T01:35:08.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.756+0000 7efd58677640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efd50073af0 con 0x7efd500b01f0 2024-09-16T01:35:08.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.756+0000 7efd56c14640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efd50073180 0x7efd500735b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:08.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.756+0000 7efd56c14640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efd50073180 0x7efd500735b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36534/0 (socket says 172.21.15.88:36534) 2024-09-16T01:35:08.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.756+0000 7efd56c14640 1 -- 172.21.15.88:0/4228048766 learned_addr learned my addr 172.21.15.88:0/4228048766 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:08.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.756+0000 7efd56413640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7efd5015e540 0x7efd500b52c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35988/0 (socket says 172.21.15.88:35988) 2024-09-16T01:35:08.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.756+0000 7efd55c12640 1 --2- 172.21.15.88:0/4228048766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd500b01f0 0x7efd501620e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:08.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.756+0000 7efd56c14640 1 -- 172.21.15.88:0/4228048766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd500b01f0 msgr2=0x7efd501620e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:08.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.756+0000 7efd56c14640 1 --2- 172.21.15.88:0/4228048766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd500b01f0 0x7efd501620e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:08.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.756+0000 7efd56c14640 1 -- 172.21.15.88:0/4228048766 >> v1:172.21.15.88:6789/0 conn(0x7efd5015e540 legacy=0x7efd500b52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:08.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.756+0000 7efd56c14640 1 -- 172.21.15.88:0/4228048766 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd50162a50 con 0x7efd50073180 2024-09-16T01:35:08.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.756+0000 7efd55411640 1 -- 172.21.15.88:0/4228048766 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2159193664 0 0) 0x7efd5015e920 con 0x7efd5015e540 2024-09-16T01:35:08.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.757+0000 7efd56c14640 1 --2- 172.21.15.88:0/4228048766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd50073180 0x7efd500735b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7efd48009e50 tx=0x7efd4802a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=589ced36eac11864 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:08.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.757+0000 7efd55411640 1 -- 172.21.15.88:0/4228048766 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd48036070 con 0x7efd50073180 2024-09-16T01:35:08.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.757+0000 7efd55411640 1 -- 172.21.15.88:0/4228048766 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd48031600 con 0x7efd50073180 2024-09-16T01:35:08.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.757+0000 7efd55411640 1 -- 172.21.15.88:0/4228048766 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd480319d0 con 0x7efd50073180 2024-09-16T01:35:08.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.758+0000 7efd58677640 1 -- 172.21.15.88:0/4228048766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd50073180 msgr2=0x7efd500735b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:08.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.758+0000 7efd58677640 1 --2- 172.21.15.88:0/4228048766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd50073180 0x7efd500735b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7efd48009e50 tx=0x7efd4802a5f0 comp rx=0 tx=0).stop 2024-09-16T01:35:08.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.758+0000 7efd58677640 1 -- 172.21.15.88:0/4228048766 shutdown_connections 2024-09-16T01:35:08.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.758+0000 7efd58677640 1 --2- 172.21.15.88:0/4228048766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd50073180 0x7efd500735b0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:08.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.758+0000 7efd58677640 1 --2- 172.21.15.88:0/4228048766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd500b01f0 0x7efd501620e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:08.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.758+0000 7efd58677640 1 -- 172.21.15.88:0/4228048766 >> 172.21.15.88:0/4228048766 conn(0x7efd5006bc10 msgr2=0x7efd5006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:08.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.758+0000 7efd58677640 1 -- 172.21.15.88:0/4228048766 shutdown_connections 2024-09-16T01:35:08.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.758+0000 7efd58677640 1 -- 172.21.15.88:0/4228048766 wait complete. 2024-09-16T01:35:08.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.759+0000 7efd58677640 1 Processor -- start 2024-09-16T01:35:08.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.759+0000 7efd58677640 1 -- start start 2024-09-16T01:35:08.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.760+0000 7efd58677640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd50073180 0x7efd501f84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:08.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.760+0000 7efd58677640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd500b01f0 0x7efd5014a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:08.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.760+0000 7efd58677640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd5015e540 0x7efd5006bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:08.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.760+0000 7efd58677640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7efd500aeca0 con 0x7efd50073180 2024-09-16T01:35:08.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.760+0000 7efd58677640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efd500ae9c0 con 0x7efd500b01f0 2024-09-16T01:35:08.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.760+0000 7efd58677640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efd50162620 con 0x7efd5015e540 2024-09-16T01:35:08.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.760+0000 7efd56413640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd50073180 0x7efd501f84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:08.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.760+0000 7efd56413640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd50073180 0x7efd501f84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36538/0 (socket says 172.21.15.88:36538) 2024-09-16T01:35:08.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.760+0000 7efd56413640 1 -- 172.21.15.88:0/1767221597 learned_addr learned my addr 172.21.15.88:0/1767221597 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:08.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.760+0000 7efd56c14640 1 --2- 172.21.15.88:0/1767221597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd5015e540 0x7efd5006bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:08.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.760+0000 7efd55c12640 1 --2- 172.21.15.88:0/1767221597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd500b01f0 0x7efd5014a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:08.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.760+0000 7efd56413640 1 -- 172.21.15.88:0/1767221597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd5015e540 msgr2=0x7efd5006bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:08.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.760+0000 7efd56413640 1 --2- 172.21.15.88:0/1767221597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd5015e540 0x7efd5006bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:08.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.760+0000 7efd56413640 1 -- 172.21.15.88:0/1767221597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd500b01f0 msgr2=0x7efd5014a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:08.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.760+0000 7efd56413640 1 --2- 172.21.15.88:0/1767221597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd500b01f0 0x7efd5014a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:08.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.760+0000 7efd56413640 1 -- 172.21.15.88:0/1767221597 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd500cf650 con 0x7efd50073180 2024-09-16T01:35:08.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.760+0000 7efd56413640 1 --2- 172.21.15.88:0/1767221597 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd50073180 0x7efd501f84a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7efd4c00c500 tx=0x7efd4c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:08.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.761+0000 7efd46ffd640 1 -- 172.21.15.88:0/1767221597 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd4c018070 con 0x7efd50073180 2024-09-16T01:35:08.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.761+0000 7efd46ffd640 1 -- 172.21.15.88:0/1767221597 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd4c007580 con 0x7efd50073180 2024-09-16T01:35:08.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.761+0000 7efd58677640 1 -- 172.21.15.88:0/1767221597 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd500cf930 con 0x7efd50073180 2024-09-16T01:35:08.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.761+0000 7efd46ffd640 1 -- 172.21.15.88:0/1767221597 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd4c004a20 con 0x7efd50073180 2024-09-16T01:35:08.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.761+0000 7efd58677640 1 -- 172.21.15.88:0/1767221597 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd5014a980 con 0x7efd50073180 2024-09-16T01:35:08.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.762+0000 7efd58677640 1 -- 172.21.15.88:0/1767221597 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd500735b0 con 0x7efd50073180 2024-09-16T01:35:08.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.762+0000 7efd46ffd640 1 -- 172.21.15.88:0/1767221597 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7efd4c01b070 con 0x7efd50073180 2024-09-16T01:35:08.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.763+0000 7efd46ffd640 1 --2- 172.21.15.88:0/1767221597 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efd3406c2e0 0x7efd3406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:08.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.763+0000 7efd55c12640 1 --2- 172.21.15.88:0/1767221597 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efd3406c2e0 0x7efd3406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:08.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.763+0000 7efd46ffd640 1 -- 172.21.15.88:0/1767221597 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7efd4c08a6a0 con 0x7efd50073180 2024-09-16T01:35:08.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.763+0000 7efd55c12640 1 --2- 172.21.15.88:0/1767221597 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efd3406c2e0 0x7efd3406e7a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7efd40006fd0 tx=0x7efd40008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:08.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.767+0000 7efd46ffd640 1 -- 172.21.15.88:0/1767221597 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7efd4c058ea0 con 0x7efd50073180 2024-09-16T01:35:08.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.931+0000 7efd58677640 1 -- 172.21.15.88:0/1767221597 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efd500b1ac0 con 0x7efd3406c2e0 2024-09-16T01:35:08.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.937+0000 7efd46ffd640 1 -- 172.21.15.88:0/1767221597 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251951 (secure 0 0 0) 0x7efd500b1ac0 con 0x7efd3406c2e0 2024-09-16T01:35:08.941 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:35:08.942 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:35:08.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.940+0000 7efd58677640 1 -- 172.21.15.88:0/1767221597 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efd3406c2e0 msgr2=0x7efd3406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:08.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.940+0000 7efd58677640 1 --2- 172.21.15.88:0/1767221597 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efd3406c2e0 0x7efd3406e7a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7efd40006fd0 tx=0x7efd40008040 comp rx=0 tx=0).stop 2024-09-16T01:35:08.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.940+0000 7efd58677640 1 -- 172.21.15.88:0/1767221597 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd50073180 msgr2=0x7efd501f84a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:08.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.940+0000 7efd58677640 1 --2- 172.21.15.88:0/1767221597 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd50073180 0x7efd501f84a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7efd4c00c500 tx=0x7efd4c00c9c0 comp rx=0 tx=0).stop 2024-09-16T01:35:08.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.940+0000 7efd58677640 1 -- 172.21.15.88:0/1767221597 shutdown_connections 2024-09-16T01:35:08.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.940+0000 7efd58677640 1 --2- 172.21.15.88:0/1767221597 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efd3406c2e0 0x7efd3406e7a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:08.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.940+0000 7efd58677640 1 --2- 172.21.15.88:0/1767221597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd5015e540 0x7efd5006bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:08.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.940+0000 7efd58677640 1 --2- 172.21.15.88:0/1767221597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd500b01f0 0x7efd5014a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:08.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.940+0000 7efd58677640 1 --2- 172.21.15.88:0/1767221597 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd50073180 0x7efd501f84a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:08.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.940+0000 7efd58677640 1 -- 172.21.15.88:0/1767221597 >> 172.21.15.88:0/1767221597 conn(0x7efd5006bc10 msgr2=0x7efd50072a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:08.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.940+0000 7efd58677640 1 -- 172.21.15.88:0/1767221597 shutdown_connections 2024-09-16T01:35:08.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:08.940+0000 7efd58677640 1 -- 172.21.15.88:0/1767221597 wait complete. 2024-09-16T01:35:08.952 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2024-09-16T01:35:08.525342+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220260,"kb_used_data":4008,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754460,"statfs":{"total":773094113280,"available":772868567040,"internally_reserved":0,"allocated":4104192,"data_stored":2280972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001430"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.349859+0000","last_change":"2024-09-16T01:35:01.345676+0000","last_active":"2024-09-16T01:35:02.349859+0000","last_peered":"2024-09-16T01:35:02.349859+0000","last_clean":"2024-09-16T01:35:02.349859+0000","last_became_active":"2024-09-16T01:35:01.345355+0000","last_became_peered":"2024-09-16T01:35:01.345355+0000","last_unstale":"2024-09-16T01:35:02.349859+0000","last_undegraded":"2024-09-16T01:35:02.349859+0000","last_fullsized":"2024-09-16T01:35:02.349859+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:06.583186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.216108+0000","last_change":"2024-09-16T01:35:01.344948+0000","last_active":"2024-09-16T01:35:03.216108+0000","last_peered":"2024-09-16T01:35:03.216108+0000","last_clean":"2024-09-16T01:35:03.216108+0000","last_became_active":"2024-09-16T01:35:01.344671+0000","last_became_peered":"2024-09-16T01:35:01.344671+0000","last_unstale":"2024-09-16T01:35:03.216108+0000","last_undegraded":"2024-09-16T01:35:03.216108+0000","last_fullsized":"2024-09-16T01:35:03.216108+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:42.054339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.350527+0000","last_change":"2024-09-16T01:35:01.353507+0000","last_active":"2024-09-16T01:35:02.350527+0000","last_peered":"2024-09-16T01:35:02.350527+0000","last_clean":"2024-09-16T01:35:02.350527+0000","last_became_active":"2024-09-16T01:35:01.352327+0000","last_became_peered":"2024-09-16T01:35:01.352327+0000","last_unstale":"2024-09-16T01:35:02.350527+0000","last_undegraded":"2024-09-16T01:35:02.350527+0000","last_fullsized":"2024-09-16T01:35:02.350527+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:17:02.572816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.518504+0000","last_change":"2024-09-16T01:35:01.352941+0000","last_active":"2024-09-16T01:35:02.518504+0000","last_peered":"2024-09-16T01:35:02.518504+0000","last_clean":"2024-09-16T01:35:02.518504+0000","last_became_active":"2024-09-16T01:35:01.352667+0000","last_became_peered":"2024-09-16T01:35:01.352667+0000","last_unstale":"2024-09-16T01:35:02.518504+0000","last_undegraded":"2024-09-16T01:35:02.518504+0000","last_fullsized":"2024-09-16T01:35:02.518504+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:26:10.193445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215381+0000","last_change":"2024-09-16T01:35:01.346569+0000","last_active":"2024-09-16T01:35:03.215381+0000","last_peered":"2024-09-16T01:35:03.215381+0000","last_clean":"2024-09-16T01:35:03.215381+0000","last_became_active":"2024-09-16T01:35:01.346316+0000","last_became_peered":"2024-09-16T01:35:01.346316+0000","last_unstale":"2024-09-16T01:35:03.215381+0000","last_undegraded":"2024-09-16T01:35:03.215381+0000","last_fullsized":"2024-09-16T01:35:03.215381+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:18:06.563678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.216087+0000","last_change":"2024-09-16T01:35:01.344131+0000","last_active":"2024-09-16T01:35:03.216087+0000","last_peered":"2024-09-16T01:35:03.216087+0000","last_clean":"2024-09-16T01:35:03.216087+0000","last_became_active":"2024-09-16T01:35:01.343874+0000","last_became_peered":"2024-09-16T01:35:01.343874+0000","last_unstale":"2024-09-16T01:35:03.216087+0000","last_undegraded":"2024-09-16T01:35:03.216087+0000","last_fullsized":"2024-09-16T01:35:03.216087+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:15:17.598991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215604+0000","last_change":"2024-09-16T01:35:01.353567+0000","last_active":"2024-09-16T01:35:03.215604+0000","last_peered":"2024-09-16T01:35:03.215604+0000","last_clean":"2024-09-16T01:35:03.215604+0000","last_became_active":"2024-09-16T01:35:01.353291+0000","last_became_peered":"2024-09-16T01:35:01.353291+0000","last_unstale":"2024-09-16T01:35:03.215604+0000","last_undegraded":"2024-09-16T01:35:03.215604+0000","last_fullsized":"2024-09-16T01:35:03.215604+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:21.032947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215410+0000","last_change":"2024-09-16T01:35:01.351136+0000","last_active":"2024-09-16T01:35:03.215410+0000","last_peered":"2024-09-16T01:35:03.215410+0000","last_clean":"2024-09-16T01:35:03.215410+0000","last_became_active":"2024-09-16T01:35:01.350532+0000","last_became_peered":"2024-09-16T01:35:01.350532+0000","last_unstale":"2024-09-16T01:35:03.215410+0000","last_undegraded":"2024-09-16T01:35:03.215410+0000","last_fullsized":"2024-09-16T01:35:03.215410+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:32:38.069472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.350515+0000","last_change":"2024-09-16T01:35:01.353966+0000","last_active":"2024-09-16T01:35:02.350515+0000","last_peered":"2024-09-16T01:35:02.350515+0000","last_clean":"2024-09-16T01:35:02.350515+0000","last_became_active":"2024-09-16T01:35:01.353732+0000","last_became_peered":"2024-09-16T01:35:01.353732+0000","last_unstale":"2024-09-16T01:35:02.350515+0000","last_undegraded":"2024-09-16T01:35:02.350515+0000","last_fullsized":"2024-09-16T01:35:02.350515+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:24:36.805818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.518457+0000","last_change":"2024-09-16T01:35:01.359872+0000","last_active":"2024-09-16T01:35:02.518457+0000","last_peered":"2024-09-16T01:35:02.518457+0000","last_clean":"2024-09-16T01:35:02.518457+0000","last_became_active":"2024-09-16T01:35:01.359604+0000","last_became_peered":"2024-09-16T01:35:01.359604+0000","last_unstale":"2024-09-16T01:35:02.518457+0000","last_undegraded":"2024-09-16T01:35:02.518457+0000","last_fullsized":"2024-09-16T01:35:02.518457+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:55.644944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215835+0000","last_change":"2024-09-16T01:35:01.355296+0000","last_active":"2024-09-16T01:35:03.215835+0000","last_peered":"2024-09-16T01:35:03.215835+0000","last_clean":"2024-09-16T01:35:03.215835+0000","last_became_active":"2024-09-16T01:35:01.355021+0000","last_became_peered":"2024-09-16T01:35:01.355021+0000","last_unstale":"2024-09-16T01:35:03.215835+0000","last_undegraded":"2024-09-16T01:35:03.215835+0000","last_fullsized":"2024-09-16T01:35:03.215835+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:03:20.835145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.348917+0000","last_change":"2024-09-16T01:35:01.349955+0000","last_active":"2024-09-16T01:35:02.348917+0000","last_peered":"2024-09-16T01:35:02.348917+0000","last_clean":"2024-09-16T01:35:02.348917+0000","last_became_active":"2024-09-16T01:35:01.349689+0000","last_became_peered":"2024-09-16T01:35:01.349689+0000","last_unstale":"2024-09-16T01:35:02.348917+0000","last_undegraded":"2024-09-16T01:35:02.348917+0000","last_fullsized":"2024-09-16T01:35:02.348917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:59:21.032947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215440+0000","last_change":"2024-09-16T01:35:01.353891+0000","last_active":"2024-09-16T01:35:03.215440+0000","last_peered":"2024-09-16T01:35:03.215440+0000","last_clean":"2024-09-16T01:35:03.215440+0000","last_became_active":"2024-09-16T01:35:01.353524+0000","last_became_peered":"2024-09-16T01:35:01.353524+0000","last_unstale":"2024-09-16T01:35:03.215440+0000","last_undegraded":"2024-09-16T01:35:03.215440+0000","last_fullsized":"2024-09-16T01:35:03.215440+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:31:33.955197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.216122+0000","last_change":"2024-09-16T01:35:01.343730+0000","last_active":"2024-09-16T01:35:03.216122+0000","last_peered":"2024-09-16T01:35:03.216122+0000","last_clean":"2024-09-16T01:35:03.216122+0000","last_became_active":"2024-09-16T01:35:01.341950+0000","last_became_peered":"2024-09-16T01:35:01.341950+0000","last_unstale":"2024-09-16T01:35:03.216122+0000","last_undegraded":"2024-09-16T01:35:03.216122+0000","last_fullsized":"2024-09-16T01:35:03.216122+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:43:07.764828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.348534+0000","last_change":"2024-09-16T01:35:01.344630+0000","last_active":"2024-09-16T01:35:02.348534+0000","last_peered":"2024-09-16T01:35:02.348534+0000","last_clean":"2024-09-16T01:35:02.348534+0000","last_became_active":"2024-09-16T01:35:01.344340+0000","last_became_peered":"2024-09-16T01:35:01.344340+0000","last_unstale":"2024-09-16T01:35:02.348534+0000","last_undegraded":"2024-09-16T01:35:02.348534+0000","last_fullsized":"2024-09-16T01:35:02.348534+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:54:56.271334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215881+0000","last_change":"2024-09-16T01:35:01.352597+0000","last_active":"2024-09-16T01:35:03.215881+0000","last_peered":"2024-09-16T01:35:03.215881+0000","last_clean":"2024-09-16T01:35:03.215881+0000","last_became_active":"2024-09-16T01:35:01.350882+0000","last_became_peered":"2024-09-16T01:35:01.350882+0000","last_unstale":"2024-09-16T01:35:03.215881+0000","last_undegraded":"2024-09-16T01:35:03.215881+0000","last_fullsized":"2024-09-16T01:35:03.215881+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:28.050249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215864+0000","last_change":"2024-09-16T01:35:01.349417+0000","last_active":"2024-09-16T01:35:03.215864+0000","last_peered":"2024-09-16T01:35:03.215864+0000","last_clean":"2024-09-16T01:35:03.215864+0000","last_became_active":"2024-09-16T01:35:01.349173+0000","last_became_peered":"2024-09-16T01:35:01.349173+0000","last_unstale":"2024-09-16T01:35:03.215864+0000","last_undegraded":"2024-09-16T01:35:03.215864+0000","last_fullsized":"2024-09-16T01:35:03.215864+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:17:02.572816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.216113+0000","last_change":"2024-09-16T01:35:01.348976+0000","last_active":"2024-09-16T01:35:03.216113+0000","last_peered":"2024-09-16T01:35:03.216113+0000","last_clean":"2024-09-16T01:35:03.216113+0000","last_became_active":"2024-09-16T01:35:01.348392+0000","last_became_peered":"2024-09-16T01:35:01.348392+0000","last_unstale":"2024-09-16T01:35:03.216113+0000","last_undegraded":"2024-09-16T01:35:03.216113+0000","last_fullsized":"2024-09-16T01:35:03.216113+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:13:00.399130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215791+0000","last_change":"2024-09-16T01:35:01.352341+0000","last_active":"2024-09-16T01:35:03.215791+0000","last_peered":"2024-09-16T01:35:03.215791+0000","last_clean":"2024-09-16T01:35:03.215791+0000","last_became_active":"2024-09-16T01:35:01.352083+0000","last_became_peered":"2024-09-16T01:35:01.352083+0000","last_unstale":"2024-09-16T01:35:03.215791+0000","last_undegraded":"2024-09-16T01:35:03.215791+0000","last_fullsized":"2024-09-16T01:35:03.215791+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:46:06.523968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215436+0000","last_change":"2024-09-16T01:35:01.343155+0000","last_active":"2024-09-16T01:35:03.215436+0000","last_peered":"2024-09-16T01:35:03.215436+0000","last_clean":"2024-09-16T01:35:03.215436+0000","last_became_active":"2024-09-16T01:35:01.342951+0000","last_became_peered":"2024-09-16T01:35:01.342951+0000","last_unstale":"2024-09-16T01:35:03.215436+0000","last_undegraded":"2024-09-16T01:35:03.215436+0000","last_fullsized":"2024-09-16T01:35:03.215436+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:30:56.380922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.350206+0000","last_change":"2024-09-16T01:35:01.344058+0000","last_active":"2024-09-16T01:35:02.350206+0000","last_peered":"2024-09-16T01:35:02.350206+0000","last_clean":"2024-09-16T01:35:02.350206+0000","last_became_active":"2024-09-16T01:35:01.343752+0000","last_became_peered":"2024-09-16T01:35:01.343752+0000","last_unstale":"2024-09-16T01:35:02.350206+0000","last_undegraded":"2024-09-16T01:35:02.350206+0000","last_fullsized":"2024-09-16T01:35:02.350206+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:40:27.148082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.349133+0000","last_change":"2024-09-16T01:35:01.353308+0000","last_active":"2024-09-16T01:35:02.349133+0000","last_peered":"2024-09-16T01:35:02.349133+0000","last_clean":"2024-09-16T01:35:02.349133+0000","last_became_active":"2024-09-16T01:35:01.352905+0000","last_became_peered":"2024-09-16T01:35:01.352905+0000","last_unstale":"2024-09-16T01:35:02.349133+0000","last_undegraded":"2024-09-16T01:35:02.349133+0000","last_fullsized":"2024-09-16T01:35:02.349133+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:58:06.888752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.759420+0000","last_change":"2024-09-16T01:35:01.347034+0000","last_active":"2024-09-16T01:35:02.759420+0000","last_peered":"2024-09-16T01:35:02.759420+0000","last_clean":"2024-09-16T01:35:02.759420+0000","last_became_active":"2024-09-16T01:35:01.346748+0000","last_became_peered":"2024-09-16T01:35:01.346748+0000","last_unstale":"2024-09-16T01:35:02.759420+0000","last_undegraded":"2024-09-16T01:35:02.759420+0000","last_fullsized":"2024-09-16T01:35:02.759420+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:58:06.888752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.348402+0000","last_change":"2024-09-16T01:35:01.344447+0000","last_active":"2024-09-16T01:35:02.348402+0000","last_peered":"2024-09-16T01:35:02.348402+0000","last_clean":"2024-09-16T01:35:02.348402+0000","last_became_active":"2024-09-16T01:35:01.344135+0000","last_became_peered":"2024-09-16T01:35:01.344135+0000","last_unstale":"2024-09-16T01:35:02.348402+0000","last_undegraded":"2024-09-16T01:35:02.348402+0000","last_fullsized":"2024-09-16T01:35:02.348402+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:32:38.069472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.518639+0000","last_change":"2024-09-16T01:35:01.352216+0000","last_active":"2024-09-16T01:35:02.518639+0000","last_peered":"2024-09-16T01:35:02.518639+0000","last_clean":"2024-09-16T01:35:02.518639+0000","last_became_active":"2024-09-16T01:35:01.351898+0000","last_became_peered":"2024-09-16T01:35:01.351898+0000","last_unstale":"2024-09-16T01:35:02.518639+0000","last_undegraded":"2024-09-16T01:35:02.518639+0000","last_fullsized":"2024-09-16T01:35:02.518639+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:32:38.069472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.216234+0000","last_change":"2024-09-16T01:35:01.349233+0000","last_active":"2024-09-16T01:35:03.216234+0000","last_peered":"2024-09-16T01:35:03.216234+0000","last_clean":"2024-09-16T01:35:03.216234+0000","last_became_active":"2024-09-16T01:35:01.348717+0000","last_became_peered":"2024-09-16T01:35:01.348717+0000","last_unstale":"2024-09-16T01:35:03.216234+0000","last_undegraded":"2024-09-16T01:35:03.216234+0000","last_fullsized":"2024-09-16T01:35:03.216234+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:01:33.098504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.518684+0000","last_change":"2024-09-16T01:35:01.352906+0000","last_active":"2024-09-16T01:35:02.518684+0000","last_peered":"2024-09-16T01:35:02.518684+0000","last_clean":"2024-09-16T01:35:02.518684+0000","last_became_active":"2024-09-16T01:35:01.352637+0000","last_became_peered":"2024-09-16T01:35:01.352637+0000","last_unstale":"2024-09-16T01:35:02.518684+0000","last_undegraded":"2024-09-16T01:35:02.518684+0000","last_fullsized":"2024-09-16T01:35:02.518684+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:15.500897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.350147+0000","last_change":"2024-09-16T01:35:01.343532+0000","last_active":"2024-09-16T01:35:02.350147+0000","last_peered":"2024-09-16T01:35:02.350147+0000","last_clean":"2024-09-16T01:35:02.350147+0000","last_became_active":"2024-09-16T01:35:01.342495+0000","last_became_peered":"2024-09-16T01:35:01.342495+0000","last_unstale":"2024-09-16T01:35:02.350147+0000","last_undegraded":"2024-09-16T01:35:02.350147+0000","last_fullsized":"2024-09-16T01:35:02.350147+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:29:06.256382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.350396+0000","last_change":"2024-09-16T01:35:01.350805+0000","last_active":"2024-09-16T01:35:02.350396+0000","last_peered":"2024-09-16T01:35:02.350396+0000","last_clean":"2024-09-16T01:35:02.350396+0000","last_became_active":"2024-09-16T01:35:01.350384+0000","last_became_peered":"2024-09-16T01:35:01.350384+0000","last_unstale":"2024-09-16T01:35:02.350396+0000","last_undegraded":"2024-09-16T01:35:02.350396+0000","last_fullsized":"2024-09-16T01:35:02.350396+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:21:22.973238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215571+0000","last_change":"2024-09-16T01:35:01.348776+0000","last_active":"2024-09-16T01:35:03.215571+0000","last_peered":"2024-09-16T01:35:03.215571+0000","last_clean":"2024-09-16T01:35:03.215571+0000","last_became_active":"2024-09-16T01:35:01.348500+0000","last_became_peered":"2024-09-16T01:35:01.348500+0000","last_unstale":"2024-09-16T01:35:03.215571+0000","last_undegraded":"2024-09-16T01:35:03.215571+0000","last_fullsized":"2024-09-16T01:35:03.215571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:19.640903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.349411+0000","last_change":"2024-09-16T01:35:01.350071+0000","last_active":"2024-09-16T01:35:02.349411+0000","last_peered":"2024-09-16T01:35:02.349411+0000","last_clean":"2024-09-16T01:35:02.349411+0000","last_became_active":"2024-09-16T01:35:01.349645+0000","last_became_peered":"2024-09-16T01:35:01.349645+0000","last_unstale":"2024-09-16T01:35:02.349411+0000","last_undegraded":"2024-09-16T01:35:02.349411+0000","last_fullsized":"2024-09-16T01:35:02.349411+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:48:09.381675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215563+0000","last_change":"2024-09-16T01:35:01.352141+0000","last_active":"2024-09-16T01:35:03.215563+0000","last_peered":"2024-09-16T01:35:03.215563+0000","last_clean":"2024-09-16T01:35:03.215563+0000","last_became_active":"2024-09-16T01:35:01.351430+0000","last_became_peered":"2024-09-16T01:35:01.351430+0000","last_unstale":"2024-09-16T01:35:03.215563+0000","last_undegraded":"2024-09-16T01:35:03.215563+0000","last_fullsized":"2024-09-16T01:35:03.215563+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:15.500897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.759579+0000","last_change":"2024-09-16T01:35:01.349127+0000","last_active":"2024-09-16T01:35:02.759579+0000","last_peered":"2024-09-16T01:35:02.759579+0000","last_clean":"2024-09-16T01:35:02.759579+0000","last_became_active":"2024-09-16T01:35:01.348873+0000","last_became_peered":"2024-09-16T01:35:01.348873+0000","last_unstale":"2024-09-16T01:35:02.759579+0000","last_undegraded":"2024-09-16T01:35:02.759579+0000","last_fullsized":"2024-09-16T01:35:02.759579+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:40:27.148082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"31'4","reported_seq":18,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.350097+0000","last_change":"2024-09-16T01:35:00.355732+0000","last_active":"2024-09-16T01:35:02.350097+0000","last_peered":"2024-09-16T01:35:02.350097+0000","last_clean":"2024-09-16T01:35:02.350097+0000","last_became_active":"2024-09-16T01:35:00.355431+0000","last_became_peered":"2024-09-16T01:35:00.355431+0000","last_unstale":"2024-09-16T01:35:02.350097+0000","last_undegraded":"2024-09-16T01:35:02.350097+0000","last_fullsized":"2024-09-16T01:35:02.350097+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:57.839735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.350279+0000","last_change":"2024-09-16T01:35:01.351197+0000","last_active":"2024-09-16T01:35:02.350279+0000","last_peered":"2024-09-16T01:35:02.350279+0000","last_clean":"2024-09-16T01:35:02.350279+0000","last_became_active":"2024-09-16T01:35:01.350899+0000","last_became_peered":"2024-09-16T01:35:01.350899+0000","last_unstale":"2024-09-16T01:35:02.350279+0000","last_undegraded":"2024-09-16T01:35:02.350279+0000","last_fullsized":"2024-09-16T01:35:02.350279+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:58:06.888752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"31'1","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.759554+0000","last_change":"2024-09-16T01:35:00.348929+0000","last_active":"2024-09-16T01:35:02.759554+0000","last_peered":"2024-09-16T01:35:02.759554+0000","last_clean":"2024-09-16T01:35:02.759554+0000","last_became_active":"2024-09-16T01:35:00.348651+0000","last_became_peered":"2024-09-16T01:35:00.348651+0000","last_unstale":"2024-09-16T01:35:02.759554+0000","last_undegraded":"2024-09-16T01:35:02.759554+0000","last_fullsized":"2024-09-16T01:35:02.759554+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:57.839735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.350727+0000","last_change":"2024-09-16T01:35:01.351890+0000","last_active":"2024-09-16T01:35:02.350727+0000","last_peered":"2024-09-16T01:35:02.350727+0000","last_clean":"2024-09-16T01:35:02.350727+0000","last_became_active":"2024-09-16T01:35:01.351554+0000","last_became_peered":"2024-09-16T01:35:01.351554+0000","last_unstale":"2024-09-16T01:35:02.350727+0000","last_undegraded":"2024-09-16T01:35:02.350727+0000","last_fullsized":"2024-09-16T01:35:02.350727+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:11:19.806171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"31'2","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215788+0000","last_change":"2024-09-16T01:35:00.344907+0000","last_active":"2024-09-16T01:35:03.215788+0000","last_peered":"2024-09-16T01:35:03.215788+0000","last_clean":"2024-09-16T01:35:03.215788+0000","last_became_active":"2024-09-16T01:35:00.344634+0000","last_became_peered":"2024-09-16T01:35:00.344634+0000","last_unstale":"2024-09-16T01:35:03.215788+0000","last_undegraded":"2024-09-16T01:35:03.215788+0000","last_fullsized":"2024-09-16T01:35:03.215788+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:02:46.941258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.518670+0000","last_change":"2024-09-16T01:35:01.353770+0000","last_active":"2024-09-16T01:35:02.518670+0000","last_peered":"2024-09-16T01:35:02.518670+0000","last_clean":"2024-09-16T01:35:02.518670+0000","last_became_active":"2024-09-16T01:35:01.353539+0000","last_became_peered":"2024-09-16T01:35:01.353539+0000","last_unstale":"2024-09-16T01:35:02.518670+0000","last_undegraded":"2024-09-16T01:35:02.518670+0000","last_fullsized":"2024-09-16T01:35:02.518670+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:54:56.271334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"31'2","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215523+0000","last_change":"2024-09-16T01:35:00.348009+0000","last_active":"2024-09-16T01:35:03.215523+0000","last_peered":"2024-09-16T01:35:03.215523+0000","last_clean":"2024-09-16T01:35:03.215523+0000","last_became_active":"2024-09-16T01:35:00.347736+0000","last_became_peered":"2024-09-16T01:35:00.347736+0000","last_unstale":"2024-09-16T01:35:03.215523+0000","last_undegraded":"2024-09-16T01:35:03.215523+0000","last_fullsized":"2024-09-16T01:35:03.215523+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:44.532138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215796+0000","last_change":"2024-09-16T01:35:01.352589+0000","last_active":"2024-09-16T01:35:03.215796+0000","last_peered":"2024-09-16T01:35:03.215796+0000","last_clean":"2024-09-16T01:35:03.215796+0000","last_became_active":"2024-09-16T01:35:01.352102+0000","last_became_peered":"2024-09-16T01:35:01.352102+0000","last_unstale":"2024-09-16T01:35:03.215796+0000","last_undegraded":"2024-09-16T01:35:03.215796+0000","last_fullsized":"2024-09-16T01:35:03.215796+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:20:19.372703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"31'1","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.350323+0000","last_change":"2024-09-16T01:35:00.348379+0000","last_active":"2024-09-16T01:35:02.350323+0000","last_peered":"2024-09-16T01:35:02.350323+0000","last_clean":"2024-09-16T01:35:02.350323+0000","last_became_active":"2024-09-16T01:35:00.348008+0000","last_became_peered":"2024-09-16T01:35:00.348008+0000","last_unstale":"2024-09-16T01:35:02.350323+0000","last_undegraded":"2024-09-16T01:35:02.350323+0000","last_fullsized":"2024-09-16T01:35:02.350323+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:00.732201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.348573+0000","last_change":"2024-09-16T01:35:01.346165+0000","last_active":"2024-09-16T01:35:02.348573+0000","last_peered":"2024-09-16T01:35:02.348573+0000","last_clean":"2024-09-16T01:35:02.348573+0000","last_became_active":"2024-09-16T01:35:01.344479+0000","last_became_peered":"2024-09-16T01:35:01.344479+0000","last_unstale":"2024-09-16T01:35:02.348573+0000","last_undegraded":"2024-09-16T01:35:02.348573+0000","last_fullsized":"2024-09-16T01:35:02.348573+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:15.500897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"31'1","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.349525+0000","last_change":"2024-09-16T01:35:00.347395+0000","last_active":"2024-09-16T01:35:02.349525+0000","last_peered":"2024-09-16T01:35:02.349525+0000","last_clean":"2024-09-16T01:35:02.349525+0000","last_became_active":"2024-09-16T01:35:00.346752+0000","last_became_peered":"2024-09-16T01:35:00.346752+0000","last_unstale":"2024-09-16T01:35:02.349525+0000","last_undegraded":"2024-09-16T01:35:02.349525+0000","last_fullsized":"2024-09-16T01:35:02.349525+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:48:54.665338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.349065+0000","last_change":"2024-09-16T01:35:01.347375+0000","last_active":"2024-09-16T01:35:02.349065+0000","last_peered":"2024-09-16T01:35:02.349065+0000","last_clean":"2024-09-16T01:35:02.349065+0000","last_became_active":"2024-09-16T01:35:01.346119+0000","last_became_peered":"2024-09-16T01:35:01.346119+0000","last_unstale":"2024-09-16T01:35:02.349065+0000","last_undegraded":"2024-09-16T01:35:02.349065+0000","last_fullsized":"2024-09-16T01:35:02.349065+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:06.583186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215745+0000","last_change":"2024-09-16T01:35:00.348232+0000","last_active":"2024-09-16T01:35:03.215745+0000","last_peered":"2024-09-16T01:35:03.215745+0000","last_clean":"2024-09-16T01:35:03.215745+0000","last_became_active":"2024-09-16T01:35:00.347976+0000","last_became_peered":"2024-09-16T01:35:00.347976+0000","last_unstale":"2024-09-16T01:35:03.215745+0000","last_undegraded":"2024-09-16T01:35:03.215745+0000","last_fullsized":"2024-09-16T01:35:03.215745+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:05.571937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.518205+0000","last_change":"2024-09-16T01:35:01.353520+0000","last_active":"2024-09-16T01:35:02.518205+0000","last_peered":"2024-09-16T01:35:02.518205+0000","last_clean":"2024-09-16T01:35:02.518205+0000","last_became_active":"2024-09-16T01:35:01.353291+0000","last_became_peered":"2024-09-16T01:35:01.353291+0000","last_unstale":"2024-09-16T01:35:02.518205+0000","last_undegraded":"2024-09-16T01:35:02.518205+0000","last_fullsized":"2024-09-16T01:35:02.518205+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:59:24.976252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.349177+0000","last_change":"2024-09-16T01:35:00.342334+0000","last_active":"2024-09-16T01:35:02.349177+0000","last_peered":"2024-09-16T01:35:02.349177+0000","last_clean":"2024-09-16T01:35:02.349177+0000","last_became_active":"2024-09-16T01:35:00.342056+0000","last_became_peered":"2024-09-16T01:35:00.342056+0000","last_unstale":"2024-09-16T01:35:02.349177+0000","last_undegraded":"2024-09-16T01:35:02.349177+0000","last_fullsized":"2024-09-16T01:35:02.349177+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:31:17.603221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.348749+0000","last_change":"2024-09-16T01:35:01.349973+0000","last_active":"2024-09-16T01:35:02.348749+0000","last_peered":"2024-09-16T01:35:02.348749+0000","last_clean":"2024-09-16T01:35:02.348749+0000","last_became_active":"2024-09-16T01:35:01.349733+0000","last_became_peered":"2024-09-16T01:35:01.349733+0000","last_unstale":"2024-09-16T01:35:02.348749+0000","last_undegraded":"2024-09-16T01:35:02.348749+0000","last_fullsized":"2024-09-16T01:35:02.348749+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:13:19.640903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215959+0000","last_change":"2024-09-16T01:35:00.349540+0000","last_active":"2024-09-16T01:35:03.215959+0000","last_peered":"2024-09-16T01:35:03.215959+0000","last_clean":"2024-09-16T01:35:03.215959+0000","last_became_active":"2024-09-16T01:35:00.349296+0000","last_became_peered":"2024-09-16T01:35:00.349296+0000","last_unstale":"2024-09-16T01:35:03.215959+0000","last_undegraded":"2024-09-16T01:35:03.215959+0000","last_fullsized":"2024-09-16T01:35:03.215959+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:58:05.877503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.216007+0000","last_change":"2024-09-16T01:35:01.358119+0000","last_active":"2024-09-16T01:35:03.216007+0000","last_peered":"2024-09-16T01:35:03.216007+0000","last_clean":"2024-09-16T01:35:03.216007+0000","last_became_active":"2024-09-16T01:35:01.357844+0000","last_became_peered":"2024-09-16T01:35:01.357844+0000","last_unstale":"2024-09-16T01:35:03.216007+0000","last_undegraded":"2024-09-16T01:35:03.216007+0000","last_fullsized":"2024-09-16T01:35:03.216007+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:27:06.347572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.759658+0000","last_change":"2024-09-16T01:35:00.346787+0000","last_active":"2024-09-16T01:35:02.759658+0000","last_peered":"2024-09-16T01:35:02.759658+0000","last_clean":"2024-09-16T01:35:02.759658+0000","last_became_active":"2024-09-16T01:35:00.346527+0000","last_became_peered":"2024-09-16T01:35:00.346527+0000","last_unstale":"2024-09-16T01:35:02.759658+0000","last_undegraded":"2024-09-16T01:35:02.759658+0000","last_fullsized":"2024-09-16T01:35:02.759658+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:49:23.770293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.216059+0000","last_change":"2024-09-16T01:35:01.348188+0000","last_active":"2024-09-16T01:35:03.216059+0000","last_peered":"2024-09-16T01:35:03.216059+0000","last_clean":"2024-09-16T01:35:03.216059+0000","last_became_active":"2024-09-16T01:35:01.347928+0000","last_became_peered":"2024-09-16T01:35:01.347928+0000","last_unstale":"2024-09-16T01:35:03.216059+0000","last_undegraded":"2024-09-16T01:35:03.216059+0000","last_fullsized":"2024-09-16T01:35:03.216059+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:00:01.874360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"31'3","reported_seq":17,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.350509+0000","last_change":"2024-09-16T01:35:00.352778+0000","last_active":"2024-09-16T01:35:02.350509+0000","last_peered":"2024-09-16T01:35:02.350509+0000","last_clean":"2024-09-16T01:35:02.350509+0000","last_became_active":"2024-09-16T01:35:00.352534+0000","last_became_peered":"2024-09-16T01:35:00.352534+0000","last_unstale":"2024-09-16T01:35:02.350509+0000","last_undegraded":"2024-09-16T01:35:02.350509+0000","last_fullsized":"2024-09-16T01:35:02.350509+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:32:30.768011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.350333+0000","last_change":"2024-09-16T01:35:01.353288+0000","last_active":"2024-09-16T01:35:02.350333+0000","last_peered":"2024-09-16T01:35:02.350333+0000","last_clean":"2024-09-16T01:35:02.350333+0000","last_became_active":"2024-09-16T01:35:01.352428+0000","last_became_peered":"2024-09-16T01:35:01.352428+0000","last_unstale":"2024-09-16T01:35:02.350333+0000","last_undegraded":"2024-09-16T01:35:02.350333+0000","last_fullsized":"2024-09-16T01:35:02.350333+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:03:20.835145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"31'6","reported_seq":20,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.216137+0000","last_change":"2024-09-16T01:35:00.350506+0000","last_active":"2024-09-16T01:35:03.216137+0000","last_peered":"2024-09-16T01:35:03.216137+0000","last_clean":"2024-09-16T01:35:03.216137+0000","last_became_active":"2024-09-16T01:35:00.350215+0000","last_became_peered":"2024-09-16T01:35:00.350215+0000","last_unstale":"2024-09-16T01:35:03.216137+0000","last_undegraded":"2024-09-16T01:35:03.216137+0000","last_fullsized":"2024-09-16T01:35:03.216137+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:23:37.920022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.518242+0000","last_change":"2024-09-16T01:35:01.354083+0000","last_active":"2024-09-16T01:35:02.518242+0000","last_peered":"2024-09-16T01:35:02.518242+0000","last_clean":"2024-09-16T01:35:02.518242+0000","last_became_active":"2024-09-16T01:35:01.353851+0000","last_became_peered":"2024-09-16T01:35:01.353851+0000","last_unstale":"2024-09-16T01:35:02.518242+0000","last_undegraded":"2024-09-16T01:35:02.518242+0000","last_fullsized":"2024-09-16T01:35:02.518242+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:48:49.160580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"31'2","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.350444+0000","last_change":"2024-09-16T01:35:00.342977+0000","last_active":"2024-09-16T01:35:02.350444+0000","last_peered":"2024-09-16T01:35:02.350444+0000","last_clean":"2024-09-16T01:35:02.350444+0000","last_became_active":"2024-09-16T01:35:00.342699+0000","last_became_peered":"2024-09-16T01:35:00.342699+0000","last_unstale":"2024-09-16T01:35:02.350444+0000","last_undegraded":"2024-09-16T01:35:02.350444+0000","last_fullsized":"2024-09-16T01:35:02.350444+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:57.839735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215941+0000","last_change":"2024-09-16T01:35:01.346497+0000","last_active":"2024-09-16T01:35:03.215941+0000","last_peered":"2024-09-16T01:35:03.215941+0000","last_clean":"2024-09-16T01:35:03.215941+0000","last_became_active":"2024-09-16T01:35:01.346236+0000","last_became_peered":"2024-09-16T01:35:01.346236+0000","last_unstale":"2024-09-16T01:35:03.215941+0000","last_undegraded":"2024-09-16T01:35:03.215941+0000","last_fullsized":"2024-09-16T01:35:03.215941+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:58:14.167743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.518204+0000","last_change":"2024-09-16T01:35:00.343581+0000","last_active":"2024-09-16T01:35:02.518204+0000","last_peered":"2024-09-16T01:35:02.518204+0000","last_clean":"2024-09-16T01:35:02.518204+0000","last_became_active":"2024-09-16T01:35:00.343294+0000","last_became_peered":"2024-09-16T01:35:00.343294+0000","last_unstale":"2024-09-16T01:35:02.518204+0000","last_undegraded":"2024-09-16T01:35:02.518204+0000","last_fullsized":"2024-09-16T01:35:02.518204+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:39:56.221559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215663+0000","last_change":"2024-09-16T01:35:01.352550+0000","last_active":"2024-09-16T01:35:03.215663+0000","last_peered":"2024-09-16T01:35:03.215663+0000","last_clean":"2024-09-16T01:35:03.215663+0000","last_became_active":"2024-09-16T01:35:01.351880+0000","last_became_peered":"2024-09-16T01:35:01.351880+0000","last_unstale":"2024-09-16T01:35:03.215663+0000","last_undegraded":"2024-09-16T01:35:03.215663+0000","last_fullsized":"2024-09-16T01:35:03.215663+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:20:14.597682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215683+0000","last_change":"2024-09-16T01:35:00.351166+0000","last_active":"2024-09-16T01:35:03.215683+0000","last_peered":"2024-09-16T01:35:03.215683+0000","last_clean":"2024-09-16T01:35:03.215683+0000","last_became_active":"2024-09-16T01:35:00.350881+0000","last_became_peered":"2024-09-16T01:35:00.350881+0000","last_unstale":"2024-09-16T01:35:03.215683+0000","last_undegraded":"2024-09-16T01:35:03.215683+0000","last_fullsized":"2024-09-16T01:35:03.215683+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:05:40.741525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.349569+0000","last_change":"2024-09-16T01:35:01.342355+0000","last_active":"2024-09-16T01:35:02.349569+0000","last_peered":"2024-09-16T01:35:02.349569+0000","last_clean":"2024-09-16T01:35:02.349569+0000","last_became_active":"2024-09-16T01:35:01.342085+0000","last_became_peered":"2024-09-16T01:35:01.342085+0000","last_unstale":"2024-09-16T01:35:02.349569+0000","last_undegraded":"2024-09-16T01:35:02.349569+0000","last_fullsized":"2024-09-16T01:35:02.349569+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:13.177105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"31'8","reported_seq":22,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215603+0000","last_change":"2024-09-16T01:35:00.345939+0000","last_active":"2024-09-16T01:35:03.215603+0000","last_peered":"2024-09-16T01:35:03.215603+0000","last_clean":"2024-09-16T01:35:03.215603+0000","last_became_active":"2024-09-16T01:35:00.345653+0000","last_became_peered":"2024-09-16T01:35:00.345653+0000","last_unstale":"2024-09-16T01:35:03.215603+0000","last_undegraded":"2024-09-16T01:35:03.215603+0000","last_fullsized":"2024-09-16T01:35:03.215603+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:12.165856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.349286+0000","last_change":"2024-09-16T01:35:01.343053+0000","last_active":"2024-09-16T01:35:02.349286+0000","last_peered":"2024-09-16T01:35:02.349286+0000","last_clean":"2024-09-16T01:35:02.349286+0000","last_became_active":"2024-09-16T01:35:01.342780+0000","last_became_peered":"2024-09-16T01:35:01.342780+0000","last_unstale":"2024-09-16T01:35:02.349286+0000","last_undegraded":"2024-09-16T01:35:02.349286+0000","last_fullsized":"2024-09-16T01:35:02.349286+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:23:38.931271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"31'1","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.518702+0000","last_change":"2024-09-16T01:35:00.350284+0000","last_active":"2024-09-16T01:35:02.518702+0000","last_peered":"2024-09-16T01:35:02.518702+0000","last_clean":"2024-09-16T01:35:02.518702+0000","last_became_active":"2024-09-16T01:35:00.349516+0000","last_became_peered":"2024-09-16T01:35:00.349516+0000","last_unstale":"2024-09-16T01:35:02.518702+0000","last_undegraded":"2024-09-16T01:35:02.518702+0000","last_fullsized":"2024-09-16T01:35:02.518702+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:31:32.943948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.350097+0000","last_change":"2024-09-16T01:35:01.347677+0000","last_active":"2024-09-16T01:35:02.350097+0000","last_peered":"2024-09-16T01:35:02.350097+0000","last_clean":"2024-09-16T01:35:02.350097+0000","last_became_active":"2024-09-16T01:35:01.345167+0000","last_became_peered":"2024-09-16T01:35:01.345167+0000","last_unstale":"2024-09-16T01:35:02.350097+0000","last_undegraded":"2024-09-16T01:35:02.350097+0000","last_fullsized":"2024-09-16T01:35:02.350097+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:20:14.597682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"31'2","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.349390+0000","last_change":"2024-09-16T01:35:00.342181+0000","last_active":"2024-09-16T01:35:02.349390+0000","last_peered":"2024-09-16T01:35:02.349390+0000","last_clean":"2024-09-16T01:35:02.349390+0000","last_became_active":"2024-09-16T01:35:00.341878+0000","last_became_peered":"2024-09-16T01:35:00.341878+0000","last_unstale":"2024-09-16T01:35:02.349390+0000","last_undegraded":"2024-09-16T01:35:02.349390+0000","last_fullsized":"2024-09-16T01:35:02.349390+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:02:54.003881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.518568+0000","last_change":"2024-09-16T01:35:01.352199+0000","last_active":"2024-09-16T01:35:02.518568+0000","last_peered":"2024-09-16T01:35:02.518568+0000","last_clean":"2024-09-16T01:35:02.518568+0000","last_became_active":"2024-09-16T01:35:01.351715+0000","last_became_peered":"2024-09-16T01:35:01.351715+0000","last_unstale":"2024-09-16T01:35:02.518568+0000","last_undegraded":"2024-09-16T01:35:02.518568+0000","last_fullsized":"2024-09-16T01:35:02.518568+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:21:17.807801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"31'1","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.349505+0000","last_change":"2024-09-16T01:35:00.349566+0000","last_active":"2024-09-16T01:35:02.349505+0000","last_peered":"2024-09-16T01:35:02.349505+0000","last_clean":"2024-09-16T01:35:02.349505+0000","last_became_active":"2024-09-16T01:35:00.349243+0000","last_became_peered":"2024-09-16T01:35:00.349243+0000","last_unstale":"2024-09-16T01:35:02.349505+0000","last_undegraded":"2024-09-16T01:35:02.349505+0000","last_fullsized":"2024-09-16T01:35:02.349505+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:26:08.139041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215400+0000","last_change":"2024-09-16T01:35:01.349744+0000","last_active":"2024-09-16T01:35:03.215400+0000","last_peered":"2024-09-16T01:35:03.215400+0000","last_clean":"2024-09-16T01:35:03.215400+0000","last_became_active":"2024-09-16T01:35:01.349501+0000","last_became_peered":"2024-09-16T01:35:01.349501+0000","last_unstale":"2024-09-16T01:35:03.215400+0000","last_undegraded":"2024-09-16T01:35:03.215400+0000","last_fullsized":"2024-09-16T01:35:03.215400+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:22:56.642629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.350676+0000","last_change":"2024-09-16T01:35:00.348167+0000","last_active":"2024-09-16T01:35:02.350676+0000","last_peered":"2024-09-16T01:35:02.350676+0000","last_clean":"2024-09-16T01:35:02.350676+0000","last_became_active":"2024-09-16T01:35:00.347714+0000","last_became_peered":"2024-09-16T01:35:00.347714+0000","last_unstale":"2024-09-16T01:35:02.350676+0000","last_undegraded":"2024-09-16T01:35:02.350676+0000","last_fullsized":"2024-09-16T01:35:02.350676+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:52:02.746213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215557+0000","last_change":"2024-09-16T01:35:01.346233+0000","last_active":"2024-09-16T01:35:03.215557+0000","last_peered":"2024-09-16T01:35:03.215557+0000","last_clean":"2024-09-16T01:35:03.215557+0000","last_became_active":"2024-09-16T01:35:01.345907+0000","last_became_peered":"2024-09-16T01:35:01.345907+0000","last_unstale":"2024-09-16T01:35:03.215557+0000","last_undegraded":"2024-09-16T01:35:03.215557+0000","last_fullsized":"2024-09-16T01:35:03.215557+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:12:12.072018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"31'2","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.759589+0000","last_change":"2024-09-16T01:35:00.345340+0000","last_active":"2024-09-16T01:35:02.759589+0000","last_peered":"2024-09-16T01:35:02.759589+0000","last_clean":"2024-09-16T01:35:02.759589+0000","last_became_active":"2024-09-16T01:35:00.344987+0000","last_became_peered":"2024-09-16T01:35:00.344987+0000","last_unstale":"2024-09-16T01:35:02.759589+0000","last_undegraded":"2024-09-16T01:35:02.759589+0000","last_fullsized":"2024-09-16T01:35:02.759589+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:43:39.411082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215548+0000","last_change":"2024-09-16T01:35:01.342112+0000","last_active":"2024-09-16T01:35:03.215548+0000","last_peered":"2024-09-16T01:35:03.215548+0000","last_clean":"2024-09-16T01:35:03.215548+0000","last_became_active":"2024-09-16T01:35:01.341821+0000","last_became_peered":"2024-09-16T01:35:01.341821+0000","last_unstale":"2024-09-16T01:35:03.215548+0000","last_undegraded":"2024-09-16T01:35:03.215548+0000","last_fullsized":"2024-09-16T01:35:03.215548+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:03.894611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.349886+0000","last_change":"2024-09-16T01:35:00.349814+0000","last_active":"2024-09-16T01:35:02.349886+0000","last_peered":"2024-09-16T01:35:02.349886+0000","last_clean":"2024-09-16T01:35:02.349886+0000","last_became_active":"2024-09-16T01:35:00.349543+0000","last_became_peered":"2024-09-16T01:35:00.349543+0000","last_unstale":"2024-09-16T01:35:02.349886+0000","last_undegraded":"2024-09-16T01:35:02.349886+0000","last_fullsized":"2024-09-16T01:35:02.349886+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:00.732201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215264+0000","last_change":"2024-09-16T01:35:01.342758+0000","last_active":"2024-09-16T01:35:03.215264+0000","last_peered":"2024-09-16T01:35:03.215264+0000","last_clean":"2024-09-16T01:35:03.215264+0000","last_became_active":"2024-09-16T01:35:01.342499+0000","last_became_peered":"2024-09-16T01:35:01.342499+0000","last_unstale":"2024-09-16T01:35:03.215264+0000","last_undegraded":"2024-09-16T01:35:03.215264+0000","last_fullsized":"2024-09-16T01:35:03.215264+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:58.665970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.349885+0000","last_change":"2024-09-16T01:35:00.348335+0000","last_active":"2024-09-16T01:35:02.349885+0000","last_peered":"2024-09-16T01:35:02.349885+0000","last_clean":"2024-09-16T01:35:02.349885+0000","last_became_active":"2024-09-16T01:35:00.347099+0000","last_became_peered":"2024-09-16T01:35:00.347099+0000","last_unstale":"2024-09-16T01:35:02.349885+0000","last_undegraded":"2024-09-16T01:35:02.349885+0000","last_fullsized":"2024-09-16T01:35:02.349885+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:04:24.380059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215454+0000","last_change":"2024-09-16T01:35:01.346350+0000","last_active":"2024-09-16T01:35:03.215454+0000","last_peered":"2024-09-16T01:35:03.215454+0000","last_clean":"2024-09-16T01:35:03.215454+0000","last_became_active":"2024-09-16T01:35:01.346089+0000","last_became_peered":"2024-09-16T01:35:01.346089+0000","last_unstale":"2024-09-16T01:35:03.215454+0000","last_undegraded":"2024-09-16T01:35:03.215454+0000","last_fullsized":"2024-09-16T01:35:03.215454+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:54:56.271334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.348351+0000","last_change":"2024-09-16T01:35:00.352839+0000","last_active":"2024-09-16T01:35:02.348351+0000","last_peered":"2024-09-16T01:35:02.348351+0000","last_clean":"2024-09-16T01:35:02.348351+0000","last_became_active":"2024-09-16T01:35:00.352568+0000","last_became_peered":"2024-09-16T01:35:00.352568+0000","last_unstale":"2024-09-16T01:35:02.348351+0000","last_undegraded":"2024-09-16T01:35:02.348351+0000","last_fullsized":"2024-09-16T01:35:02.348351+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:30:51.141162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215390+0000","last_change":"2024-09-16T01:35:01.343350+0000","last_active":"2024-09-16T01:35:03.215390+0000","last_peered":"2024-09-16T01:35:03.215390+0000","last_clean":"2024-09-16T01:35:03.215390+0000","last_became_active":"2024-09-16T01:35:01.343079+0000","last_became_peered":"2024-09-16T01:35:01.343079+0000","last_unstale":"2024-09-16T01:35:03.215390+0000","last_undegraded":"2024-09-16T01:35:03.215390+0000","last_fullsized":"2024-09-16T01:35:03.215390+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:09.003741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"31'2","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.518749+0000","last_change":"2024-09-16T01:35:00.355682+0000","last_active":"2024-09-16T01:35:02.518749+0000","last_peered":"2024-09-16T01:35:02.518749+0000","last_clean":"2024-09-16T01:35:02.518749+0000","last_became_active":"2024-09-16T01:35:00.355347+0000","last_became_peered":"2024-09-16T01:35:00.355347+0000","last_unstale":"2024-09-16T01:35:02.518749+0000","last_undegraded":"2024-09-16T01:35:02.518749+0000","last_fullsized":"2024-09-16T01:35:02.518749+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:02.883362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.350266+0000","last_change":"2024-09-16T01:35:01.355640+0000","last_active":"2024-09-16T01:35:02.350266+0000","last_peered":"2024-09-16T01:35:02.350266+0000","last_clean":"2024-09-16T01:35:02.350266+0000","last_became_active":"2024-09-16T01:35:01.355364+0000","last_became_peered":"2024-09-16T01:35:01.355364+0000","last_unstale":"2024-09-16T01:35:02.350266+0000","last_undegraded":"2024-09-16T01:35:02.350266+0000","last_fullsized":"2024-09-16T01:35:02.350266+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:20:19.372703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"32'1","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.759690+0000","last_change":"2024-09-16T01:35:00.343346+0000","last_active":"2024-09-16T01:35:02.759690+0000","last_peered":"2024-09-16T01:35:02.759690+0000","last_clean":"2024-09-16T01:35:02.759690+0000","last_became_active":"2024-09-16T01:35:00.343049+0000","last_became_peered":"2024-09-16T01:35:00.343049+0000","last_unstale":"2024-09-16T01:35:02.759690+0000","last_undegraded":"2024-09-16T01:35:02.759690+0000","last_fullsized":"2024-09-16T01:35:02.759690+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:53:24.648600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.759462+0000","last_change":"2024-09-16T01:35:01.352593+0000","last_active":"2024-09-16T01:35:02.759462+0000","last_peered":"2024-09-16T01:35:02.759462+0000","last_clean":"2024-09-16T01:35:02.759462+0000","last_became_active":"2024-09-16T01:35:01.352292+0000","last_became_peered":"2024-09-16T01:35:01.352292+0000","last_unstale":"2024-09-16T01:35:02.759462+0000","last_undegraded":"2024-09-16T01:35:02.759462+0000","last_fullsized":"2024-09-16T01:35:02.759462+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:23:38.931271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.518602+0000","last_change":"2024-09-16T01:35:00.349236+0000","last_active":"2024-09-16T01:35:02.518602+0000","last_peered":"2024-09-16T01:35:02.518602+0000","last_clean":"2024-09-16T01:35:02.518602+0000","last_became_active":"2024-09-16T01:35:00.348947+0000","last_became_peered":"2024-09-16T01:35:00.348947+0000","last_unstale":"2024-09-16T01:35:02.518602+0000","last_undegraded":"2024-09-16T01:35:02.518602+0000","last_fullsized":"2024-09-16T01:35:02.518602+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:12:11.060769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.349760+0000","last_change":"2024-09-16T01:35:01.347200+0000","last_active":"2024-09-16T01:35:02.349760+0000","last_peered":"2024-09-16T01:35:02.349760+0000","last_clean":"2024-09-16T01:35:02.349760+0000","last_became_active":"2024-09-16T01:35:01.346942+0000","last_became_peered":"2024-09-16T01:35:01.346942+0000","last_unstale":"2024-09-16T01:35:02.349760+0000","last_undegraded":"2024-09-16T01:35:02.349760+0000","last_fullsized":"2024-09-16T01:35:02.349760+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:02:47.952507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.759657+0000","last_change":"2024-09-16T01:35:00.355723+0000","last_active":"2024-09-16T01:35:02.759657+0000","last_peered":"2024-09-16T01:35:02.759657+0000","last_clean":"2024-09-16T01:35:02.759657+0000","last_became_active":"2024-09-16T01:35:00.355425+0000","last_became_peered":"2024-09-16T01:35:00.355425+0000","last_unstale":"2024-09-16T01:35:02.759657+0000","last_undegraded":"2024-09-16T01:35:02.759657+0000","last_fullsized":"2024-09-16T01:35:02.759657+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:54:05.742892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.759555+0000","last_change":"2024-09-16T01:35:01.347469+0000","last_active":"2024-09-16T01:35:02.759555+0000","last_peered":"2024-09-16T01:35:02.759555+0000","last_clean":"2024-09-16T01:35:02.759555+0000","last_became_active":"2024-09-16T01:35:01.347121+0000","last_became_peered":"2024-09-16T01:35:01.347121+0000","last_unstale":"2024-09-16T01:35:02.759555+0000","last_undegraded":"2024-09-16T01:35:02.759555+0000","last_fullsized":"2024-09-16T01:35:02.759555+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:05:41.752774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.518299+0000","last_change":"2024-09-16T01:35:00.355687+0000","last_active":"2024-09-16T01:35:02.518299+0000","last_peered":"2024-09-16T01:35:02.518299+0000","last_clean":"2024-09-16T01:35:02.518299+0000","last_became_active":"2024-09-16T01:35:00.355357+0000","last_became_peered":"2024-09-16T01:35:00.355357+0000","last_unstale":"2024-09-16T01:35:02.518299+0000","last_undegraded":"2024-09-16T01:35:02.518299+0000","last_fullsized":"2024-09-16T01:35:02.518299+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:45:56.563617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215438+0000","last_change":"2024-09-16T01:35:01.348668+0000","last_active":"2024-09-16T01:35:03.215438+0000","last_peered":"2024-09-16T01:35:03.215438+0000","last_clean":"2024-09-16T01:35:03.215438+0000","last_became_active":"2024-09-16T01:35:01.348444+0000","last_became_peered":"2024-09-16T01:35:01.348444+0000","last_unstale":"2024-09-16T01:35:03.215438+0000","last_undegraded":"2024-09-16T01:35:03.215438+0000","last_fullsized":"2024-09-16T01:35:03.215438+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:59:11.658637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.350262+0000","last_change":"2024-09-16T01:35:00.351123+0000","last_active":"2024-09-16T01:35:02.350262+0000","last_peered":"2024-09-16T01:35:02.350262+0000","last_clean":"2024-09-16T01:35:02.350262+0000","last_became_active":"2024-09-16T01:35:00.350803+0000","last_became_peered":"2024-09-16T01:35:00.350803+0000","last_unstale":"2024-09-16T01:35:02.350262+0000","last_undegraded":"2024-09-16T01:35:02.350262+0000","last_fullsized":"2024-09-16T01:35:02.350262+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:00.117677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.349780+0000","last_change":"2024-09-16T01:35:01.348863+0000","last_active":"2024-09-16T01:35:02.349780+0000","last_peered":"2024-09-16T01:35:02.349780+0000","last_clean":"2024-09-16T01:35:02.349780+0000","last_became_active":"2024-09-16T01:35:01.348461+0000","last_became_peered":"2024-09-16T01:35:01.348461+0000","last_unstale":"2024-09-16T01:35:02.349780+0000","last_undegraded":"2024-09-16T01:35:02.349780+0000","last_fullsized":"2024-09-16T01:35:02.349780+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:05:41.752774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215563+0000","last_change":"2024-09-16T01:35:00.351535+0000","last_active":"2024-09-16T01:35:03.215563+0000","last_peered":"2024-09-16T01:35:03.215563+0000","last_clean":"2024-09-16T01:35:03.215563+0000","last_became_active":"2024-09-16T01:35:00.351219+0000","last_became_peered":"2024-09-16T01:35:00.351219+0000","last_unstale":"2024-09-16T01:35:03.215563+0000","last_undegraded":"2024-09-16T01:35:03.215563+0000","last_fullsized":"2024-09-16T01:35:03.215563+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:48:08.370426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215786+0000","last_change":"2024-09-16T01:35:01.353261+0000","last_active":"2024-09-16T01:35:03.215786+0000","last_peered":"2024-09-16T01:35:03.215786+0000","last_clean":"2024-09-16T01:35:03.215786+0000","last_became_active":"2024-09-16T01:35:01.352844+0000","last_became_peered":"2024-09-16T01:35:01.352844+0000","last_unstale":"2024-09-16T01:35:03.215786+0000","last_undegraded":"2024-09-16T01:35:03.215786+0000","last_fullsized":"2024-09-16T01:35:03.215786+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:35:00.326194+0000","last_clean_scrub_stamp":"2024-09-16T01:35:00.326194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:29:06.256382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:02.349314+0000","last_change":"2024-09-16T01:35:00.344871+0000","last_active":"2024-09-16T01:35:02.349314+0000","last_peered":"2024-09-16T01:35:02.349314+0000","last_clean":"2024-09-16T01:35:02.349314+0000","last_became_active":"2024-09-16T01:35:00.344582+0000","last_became_peered":"2024-09-16T01:35:00.344582+0000","last_unstale":"2024-09-16T01:35:02.349314+0000","last_undegraded":"2024-09-16T01:35:02.349314+0000","last_fullsized":"2024-09-16T01:35:02.349314+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:59.314945+0000","last_clean_scrub_stamp":"2024-09-16T01:34:59.314945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:52:02.746213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":189,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-16T01:35:03.215922+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:35:03.215922+0000","last_peered":"2024-09-16T01:35:03.215922+0000","last_clean":"2024-09-16T01:35:03.215922+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:35:03.215922+0000","last_undegraded":"2024-09-16T01:35:03.215922+0000","last_fullsized":"2024-09-16T01:35:03.215922+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509456,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":364544,"data_stored":137599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542159,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":954368,"data_stored":727385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542159,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":364544,"data_stored":137543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542159,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":364544,"data_stored":137031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542158,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":372736,"data_stored":138561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542159,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":360448,"data_stored":136997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542158,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":372736,"data_stored":138561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542158,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27956,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343884,"statfs":{"total":96636764160,"available":96608137216,"internally_reserved":0,"allocated":950272,"data_stored":727295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:35:08.956 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T01:35:08.956 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-16T01:35:09.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.097+0000 7fb9df82e640 1 Processor -- start 2024-09-16T01:35:09.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.098+0000 7fb9df82e640 1 -- start start 2024-09-16T01:35:09.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.098+0000 7fb9df82e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb9d815e7a0 0x7fb9d806caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:09.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.098+0000 7fb9df82e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb9d80cf1f0 0x7fb9d80b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:09.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.099+0000 7fb9df82e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9d80cf860 con 0x7fb9d80b6720 2024-09-16T01:35:09.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.099+0000 7fb9df82e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb9d80b6b90 con 0x7fb9d80cf1f0 2024-09-16T01:35:09.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.099+0000 7fb9df82e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb9d80d2ca0 con 0x7fb9d815e7a0 2024-09-16T01:35:09.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.099+0000 7fb9dcdc9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb9d80cf1f0 0x7fb9d80b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:09.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.099+0000 7fb9dddcb640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb9d80b6720 0x7fb9d81623c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35990/0 (socket says 172.21.15.88:35990) 2024-09-16T01:35:09.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.099+0000 7fb9dddcb640 1 -- 172.21.15.88:0/3058486543 learned_addr learned my addr 172.21.15.88:0/3058486543 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:09.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.099+0000 7fb9dd5ca640 1 --2- 172.21.15.88:0/3058486543 >> v2:172.21.15.154:3300/0 conn(0x7fb9d815e7a0 0x7fb9d806caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:09.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.099+0000 7fb9dcdc9640 1 -- 172.21.15.88:0/3058486543 >> v2:172.21.15.154:3300/0 conn(0x7fb9d815e7a0 msgr2=0x7fb9d806caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:09.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.099+0000 7fb9dcdc9640 1 --2- 172.21.15.88:0/3058486543 >> v2:172.21.15.154:3300/0 conn(0x7fb9d815e7a0 0x7fb9d806caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:09.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.099+0000 7fb9dcdc9640 1 -- 172.21.15.88:0/3058486543 >> v1:172.21.15.88:6789/0 conn(0x7fb9d80b6720 legacy=0x7fb9d81623c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:09.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.099+0000 7fb9dcdc9640 1 -- 172.21.15.88:0/3058486543 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9d80cfb40 con 0x7fb9d80cf1f0 2024-09-16T01:35:09.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.099+0000 7fb9cffff640 1 -- 172.21.15.88:0/3058486543 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3312610092 0 0) 0x7fb9d80cf860 con 0x7fb9d80b6720 2024-09-16T01:35:09.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.100+0000 7fb9dcdc9640 1 --2- 172.21.15.88:0/3058486543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9d80cf1f0 0x7fb9d80b61e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb9d80d5820 tx=0x7fb9c802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5256de1041078a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:09.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.100+0000 7fb9cffff640 1 -- 172.21.15.88:0/3058486543 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8037070 con 0x7fb9d80cf1f0 2024-09-16T01:35:09.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.100+0000 7fb9cffff640 1 -- 172.21.15.88:0/3058486543 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9c8032440 con 0x7fb9d80cf1f0 2024-09-16T01:35:09.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.100+0000 7fb9cffff640 1 -- 172.21.15.88:0/3058486543 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c802ae60 con 0x7fb9d80cf1f0 2024-09-16T01:35:09.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.100+0000 7fb9df82e640 1 -- 172.21.15.88:0/3058486543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9d80cf1f0 msgr2=0x7fb9d80b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:09.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.100+0000 7fb9df82e640 1 --2- 172.21.15.88:0/3058486543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9d80cf1f0 0x7fb9d80b61e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb9d80d5820 tx=0x7fb9c802a110 comp rx=0 tx=0).stop 2024-09-16T01:35:09.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.100+0000 7fb9df82e640 1 -- 172.21.15.88:0/3058486543 shutdown_connections 2024-09-16T01:35:09.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.100+0000 7fb9df82e640 1 --2- 172.21.15.88:0/3058486543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9d80cf1f0 0x7fb9d80b61e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:09.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.100+0000 7fb9df82e640 1 --2- 172.21.15.88:0/3058486543 >> v2:172.21.15.154:3300/0 conn(0x7fb9d815e7a0 0x7fb9d806caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:09.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.100+0000 7fb9df82e640 1 -- 172.21.15.88:0/3058486543 >> 172.21.15.88:0/3058486543 conn(0x7fb9d80d11e0 msgr2=0x7fb9d80d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:09.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.101+0000 7fb9df82e640 1 -- 172.21.15.88:0/3058486543 shutdown_connections 2024-09-16T01:35:09.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.101+0000 7fb9df82e640 1 -- 172.21.15.88:0/3058486543 wait complete. 2024-09-16T01:35:09.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.102+0000 7fb9df82e640 1 Processor -- start 2024-09-16T01:35:09.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.102+0000 7fb9df82e640 1 -- start start 2024-09-16T01:35:09.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.102+0000 7fb9df82e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9d80b6720 0x7fb9d8161190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:09.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.102+0000 7fb9df82e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9d80cf1f0 0x7fb9d80c8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:09.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.102+0000 7fb9df82e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9d815e7a0 0x7fb9d80d4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:09.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.102+0000 7fb9df82e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d80b7a00 con 0x7fb9d80cf1f0 2024-09-16T01:35:09.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.102+0000 7fb9df82e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d80d2ef0 con 0x7fb9d80b6720 2024-09-16T01:35:09.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.102+0000 7fb9df82e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb9d806cfe0 con 0x7fb9d815e7a0 2024-09-16T01:35:09.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.102+0000 7fb9dcdc9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9d80cf1f0 0x7fb9d80c8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:09.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.103+0000 7fb9dcdc9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9d80cf1f0 0x7fb9d80c8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36562/0 (socket says 172.21.15.88:36562) 2024-09-16T01:35:09.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.103+0000 7fb9dcdc9640 1 -- 172.21.15.88:0/608367319 learned_addr learned my addr 172.21.15.88:0/608367319 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:09.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.103+0000 7fb9dd5ca640 1 --2- 172.21.15.88:0/608367319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9d80b6720 0x7fb9d8161190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:09.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.103+0000 7fb9dddcb640 1 --2- 172.21.15.88:0/608367319 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9d815e7a0 0x7fb9d80d4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:09.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.103+0000 7fb9dcdc9640 1 -- 172.21.15.88:0/608367319 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9d815e7a0 msgr2=0x7fb9d80d4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:09.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.103+0000 7fb9dcdc9640 1 --2- 172.21.15.88:0/608367319 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9d815e7a0 0x7fb9d80d4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:09.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.103+0000 7fb9dcdc9640 1 -- 172.21.15.88:0/608367319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9d80b6720 msgr2=0x7fb9d8161190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:09.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.103+0000 7fb9dcdc9640 1 --2- 172.21.15.88:0/608367319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9d80b6720 0x7fb9d8161190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:09.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.103+0000 7fb9dcdc9640 1 -- 172.21.15.88:0/608367319 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9d81616d0 con 0x7fb9d80cf1f0 2024-09-16T01:35:09.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.103+0000 7fb9dd5ca640 1 --2- 172.21.15.88:0/608367319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9d80b6720 0x7fb9d8161190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:09.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.103+0000 7fb9dcdc9640 1 --2- 172.21.15.88:0/608367319 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9d80cf1f0 0x7fb9d80c8290 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8000c00 tx=0x7fb9c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:09.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.103+0000 7fb9cdffb640 1 -- 172.21.15.88:0/608367319 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8037040 con 0x7fb9d80cf1f0 2024-09-16T01:35:09.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.103+0000 7fb9cdffb640 1 -- 172.21.15.88:0/608367319 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9c8042070 con 0x7fb9d80cf1f0 2024-09-16T01:35:09.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.103+0000 7fb9df82e640 1 -- 172.21.15.88:0/608367319 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9d80c8ae0 con 0x7fb9d80cf1f0 2024-09-16T01:35:09.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.103+0000 7fb9cdffb640 1 -- 172.21.15.88:0/608367319 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8030bd0 con 0x7fb9d80cf1f0 2024-09-16T01:35:09.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.103+0000 7fb9df82e640 1 -- 172.21.15.88:0/608367319 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9d80c90a0 con 0x7fb9d80cf1f0 2024-09-16T01:35:09.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.104+0000 7fb9df82e640 1 -- 172.21.15.88:0/608367319 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9d815eb80 con 0x7fb9d80cf1f0 2024-09-16T01:35:09.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.105+0000 7fb9cdffb640 1 -- 172.21.15.88:0/608367319 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb9c8047460 con 0x7fb9d80cf1f0 2024-09-16T01:35:09.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.105+0000 7fb9cdffb640 1 --2- 172.21.15.88:0/608367319 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb9bc06c2b0 0x7fb9bc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:09.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.105+0000 7fb9cdffb640 1 -- 172.21.15.88:0/608367319 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fb9c80ac020 con 0x7fb9d80cf1f0 2024-09-16T01:35:09.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.105+0000 7fb9dd5ca640 1 --2- 172.21.15.88:0/608367319 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb9bc06c2b0 0x7fb9bc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:09.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.106+0000 7fb9dd5ca640 1 --2- 172.21.15.88:0/608367319 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb9bc06c2b0 0x7fb9bc06e770 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4006fd0 tx=0x7fb9d4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:09.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.109+0000 7fb9cdffb640 1 -- 172.21.15.88:0/608367319 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb9c8032e20 con 0x7fb9d80cf1f0 2024-09-16T01:35:09.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.338+0000 7fb9df82e640 1 -- 172.21.15.88:0/608367319 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fb9d80c8030 con 0x7fb9d80cf1f0 2024-09-16T01:35:09.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.341+0000 7fb9cdffb640 1 -- 172.21.15.88:0/608367319 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) ==== 190+0+147 (secure 0 0 0) 0x7fb9c807a790 con 0x7fb9d80cf1f0 2024-09-16T01:35:09.342 INFO:teuthology.orchestra.run.smithi088.stdout:[client.0] 2024-09-16T01:35:09.342 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDeiudmvPikIBAAyHxNNnwQKygxg09PNQLc/Q== 2024-09-16T01:35:09.342 INFO:teuthology.orchestra.run.smithi088.stdout: caps mds = "allow" 2024-09-16T01:35:09.342 INFO:teuthology.orchestra.run.smithi088.stdout: caps mon = "allow r" 2024-09-16T01:35:09.343 INFO:teuthology.orchestra.run.smithi088.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-16T01:35:09.343 INFO:teuthology.orchestra.run.smithi088.stderr:updated caps for client.0 2024-09-16T01:35:09.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.343+0000 7fb9df82e640 1 -- 172.21.15.88:0/608367319 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb9bc06c2b0 msgr2=0x7fb9bc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:09.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.343+0000 7fb9df82e640 1 --2- 172.21.15.88:0/608367319 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb9bc06c2b0 0x7fb9bc06e770 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4006fd0 tx=0x7fb9d4008040 comp rx=0 tx=0).stop 2024-09-16T01:35:09.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.343+0000 7fb9df82e640 1 -- 172.21.15.88:0/608367319 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9d80cf1f0 msgr2=0x7fb9d80c8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:09.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.343+0000 7fb9df82e640 1 --2- 172.21.15.88:0/608367319 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9d80cf1f0 0x7fb9d80c8290 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8000c00 tx=0x7fb9c80046c0 comp rx=0 tx=0).stop 2024-09-16T01:35:09.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.343+0000 7fb9df82e640 1 -- 172.21.15.88:0/608367319 shutdown_connections 2024-09-16T01:35:09.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.343+0000 7fb9df82e640 1 --2- 172.21.15.88:0/608367319 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb9bc06c2b0 0x7fb9bc06e770 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:09.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.343+0000 7fb9df82e640 1 --2- 172.21.15.88:0/608367319 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9d815e7a0 0x7fb9d80d4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:09.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.343+0000 7fb9df82e640 1 --2- 172.21.15.88:0/608367319 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9d80cf1f0 0x7fb9d80c8290 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:09.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.343+0000 7fb9df82e640 1 --2- 172.21.15.88:0/608367319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9d80b6720 0x7fb9d8161190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:09.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.343+0000 7fb9df82e640 1 -- 172.21.15.88:0/608367319 >> 172.21.15.88:0/608367319 conn(0x7fb9d80d11e0 msgr2=0x7fb9d8159cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:09.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.343+0000 7fb9df82e640 1 -- 172.21.15.88:0/608367319 shutdown_connections 2024-09-16T01:35:09.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.343+0000 7fb9df82e640 1 -- 172.21.15.88:0/608367319 wait complete. 2024-09-16T01:35:09.359 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:35:09.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.457+0000 7f59ecee4640 1 Processor -- start 2024-09-16T01:35:09.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.457+0000 7f59ecee4640 1 -- start start 2024-09-16T01:35:09.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.458+0000 7f59ecee4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f59e806d540 0x7f59e80cf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:09.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.458+0000 7f59ecee4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f59e80735b0 0x7f59e80cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:09.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.458+0000 7f59ecee4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59e806faf0 con 0x7f59e815e950 2024-09-16T01:35:09.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.458+0000 7f59ecee4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f59e806fdd0 con 0x7f59e80735b0 2024-09-16T01:35:09.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.458+0000 7f59ecee4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f59e80700b0 con 0x7f59e806d540 2024-09-16T01:35:09.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.458+0000 7f59e6d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f59e80735b0 0x7f59e80cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:09.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.458+0000 7f59e6d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f59e80735b0 0x7f59e80cf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36578/0 (socket says 172.21.15.88:36578) 2024-09-16T01:35:09.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.458+0000 7f59e6d9d640 1 -- 172.21.15.88:0/723436236 learned_addr learned my addr 172.21.15.88:0/723436236 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:09.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.458+0000 7f59ddd9b640 1 --2- 172.21.15.88:0/723436236 >> v2:172.21.15.154:3300/0 conn(0x7f59e806d540 0x7f59e80cf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:09.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.458+0000 7f59e6d9d640 1 -- 172.21.15.88:0/723436236 >> v2:172.21.15.154:3300/0 conn(0x7f59e806d540 msgr2=0x7f59e80cf0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:09.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.458+0000 7f59e6d9d640 1 --2- 172.21.15.88:0/723436236 >> v2:172.21.15.154:3300/0 conn(0x7f59e806d540 0x7f59e80cf0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:09.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.458+0000 7f59e6d9d640 1 -- 172.21.15.88:0/723436236 >> v1:172.21.15.154:6789/0 conn(0x7f59e815e950 legacy=0x7f59e806ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:09.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.458+0000 7f59e6d9d640 1 -- 172.21.15.88:0/723436236 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e8070390 con 0x7f59e80735b0 2024-09-16T01:35:09.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.458+0000 7f59ddd9b640 1 --2- 172.21.15.88:0/723436236 >> v2:172.21.15.154:3300/0 conn(0x7f59e806d540 0x7f59e80cf0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:09.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.459+0000 7f59e6d9d640 1 --2- 172.21.15.88:0/723436236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59e80735b0 0x7f59e80cf5e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f59e80743c0 tx=0x7f59e002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71211bab24bc3cb7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:09.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.459+0000 7f59e5d9b640 1 -- 172.21.15.88:0/723436236 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e0036070 con 0x7f59e80735b0 2024-09-16T01:35:09.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.459+0000 7f59e5d9b640 1 -- 172.21.15.88:0/723436236 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59e0031600 con 0x7f59e80735b0 2024-09-16T01:35:09.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.459+0000 7f59e5d9b640 1 -- 172.21.15.88:0/723436236 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e00319d0 con 0x7f59e80735b0 2024-09-16T01:35:09.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.459+0000 7f59ecee4640 1 -- 172.21.15.88:0/723436236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59e80735b0 msgr2=0x7f59e80cf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:09.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.459+0000 7f59ecee4640 1 --2- 172.21.15.88:0/723436236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59e80735b0 0x7f59e80cf5e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f59e80743c0 tx=0x7f59e002a730 comp rx=0 tx=0).stop 2024-09-16T01:35:09.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.459+0000 7f59ecee4640 1 -- 172.21.15.88:0/723436236 shutdown_connections 2024-09-16T01:35:09.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.459+0000 7f59ecee4640 1 --2- 172.21.15.88:0/723436236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59e80735b0 0x7f59e80cf5e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:09.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.459+0000 7f59ecee4640 1 --2- 172.21.15.88:0/723436236 >> v2:172.21.15.154:3300/0 conn(0x7f59e806d540 0x7f59e80cf0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:09.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.459+0000 7f59ecee4640 1 -- 172.21.15.88:0/723436236 >> 172.21.15.88:0/723436236 conn(0x7f59e80d14d0 msgr2=0x7f59e80d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:09.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.459+0000 7f59ecee4640 1 -- 172.21.15.88:0/723436236 shutdown_connections 2024-09-16T01:35:09.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.460+0000 7f59ecee4640 1 -- 172.21.15.88:0/723436236 wait complete. 2024-09-16T01:35:09.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.461+0000 7f59ecee4640 1 Processor -- start 2024-09-16T01:35:09.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.461+0000 7f59ecee4640 1 -- start start 2024-09-16T01:35:09.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.462+0000 7f59ecee4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59e806d540 0x7f59e81617c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:09.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.462+0000 7f59ecee4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59e80735b0 0x7f59e814da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:09.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.462+0000 7f59ecee4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59e815e950 0x7f59e81e93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:09.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.462+0000 7f59ecee4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f59e80721d0 con 0x7f59e80735b0 2024-09-16T01:35:09.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.462+0000 7f59ecee4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f59e8072050 con 0x7f59e806d540 2024-09-16T01:35:09.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.462+0000 7f59ecee4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f59e8072350 con 0x7f59e815e950 2024-09-16T01:35:09.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.462+0000 7f59ddd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59e80735b0 0x7f59e814da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:09.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.462+0000 7f59ddd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59e80735b0 0x7f59e814da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36580/0 (socket says 172.21.15.88:36580) 2024-09-16T01:35:09.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.462+0000 7f59ddd9b640 1 -- 172.21.15.88:0/933590126 learned_addr learned my addr 172.21.15.88:0/933590126 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:09.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.462+0000 7f59e659c640 1 --2- 172.21.15.88:0/933590126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59e806d540 0x7f59e81617c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:09.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.462+0000 7f59e6d9d640 1 --2- 172.21.15.88:0/933590126 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59e815e950 0x7f59e81e93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:09.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.462+0000 7f59ddd9b640 1 -- 172.21.15.88:0/933590126 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59e815e950 msgr2=0x7f59e81e93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:09.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.462+0000 7f59ddd9b640 1 --2- 172.21.15.88:0/933590126 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59e815e950 0x7f59e81e93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:09.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.462+0000 7f59ddd9b640 1 -- 172.21.15.88:0/933590126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59e806d540 msgr2=0x7f59e81617c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:09.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.462+0000 7f59ddd9b640 1 --2- 172.21.15.88:0/933590126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59e806d540 0x7f59e81617c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:09.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.462+0000 7f59ddd9b640 1 -- 172.21.15.88:0/933590126 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e815fab0 con 0x7f59e80735b0 2024-09-16T01:35:09.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.462+0000 7f59e659c640 1 --2- 172.21.15.88:0/933590126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59e806d540 0x7f59e81617c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:09.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.462+0000 7f59ddd9b640 1 --2- 172.21.15.88:0/933590126 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59e80735b0 0x7f59e814da60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f59d000d7d0 tx=0x7f59d000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:09.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.462+0000 7f59df7fe640 1 -- 172.21.15.88:0/933590126 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d0004030 con 0x7f59e80735b0 2024-09-16T01:35:09.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.462+0000 7f59df7fe640 1 -- 172.21.15.88:0/933590126 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59d00041d0 con 0x7f59e80735b0 2024-09-16T01:35:09.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.462+0000 7f59ecee4640 1 -- 172.21.15.88:0/933590126 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59e815fd90 con 0x7f59e80735b0 2024-09-16T01:35:09.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.462+0000 7f59df7fe640 1 -- 172.21.15.88:0/933590126 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d00096a0 con 0x7f59e80735b0 2024-09-16T01:35:09.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.462+0000 7f59ecee4640 1 -- 172.21.15.88:0/933590126 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59e8160350 con 0x7f59e80735b0 2024-09-16T01:35:09.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.464+0000 7f59ecee4640 1 -- 172.21.15.88:0/933590126 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59e806ce30 con 0x7f59e80735b0 2024-09-16T01:35:09.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.464+0000 7f59df7fe640 1 -- 172.21.15.88:0/933590126 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f59d0009840 con 0x7f59e80735b0 2024-09-16T01:35:09.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.464+0000 7f59df7fe640 1 --2- 172.21.15.88:0/933590126 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f59c806c2b0 0x7f59c806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:09.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.464+0000 7f59df7fe640 1 -- 172.21.15.88:0/933590126 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f59d008b6e0 con 0x7f59e80735b0 2024-09-16T01:35:09.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.464+0000 7f59e659c640 1 --2- 172.21.15.88:0/933590126 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f59c806c2b0 0x7f59c806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:09.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.465+0000 7f59e659c640 1 --2- 172.21.15.88:0/933590126 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f59c806c2b0 0x7f59c806e770 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f59d8001680 tx=0x7f59d8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:09.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.469+0000 7f59df7fe640 1 -- 172.21.15.88:0/933590126 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f59d0059ee0 con 0x7f59e80735b0 2024-09-16T01:35:09.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.667+0000 7f59ecee4640 1 -- 172.21.15.88:0/933590126 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59e815e0c0 con 0x7f59e80735b0 2024-09-16T01:35:09.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.667+0000 7f59df7fe640 1 -- 172.21.15.88:0/933590126 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+5229 (secure 0 0 0) 0x7f59d00597e0 con 0x7f59e80735b0 2024-09-16T01:35:09.669 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:35:09.670 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":13,"btime":"2024-09-16T01:35:03:353018+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1005331674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1005331674},{"type":"v1","addr":"172.21.15.88:6837","nonce":1005331674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/3427247662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3427247662},{"type":"v1","addr":"172.21.15.154:6834","nonce":3427247662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/4059835337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":4059835337},{"type":"v1","addr":"172.21.15.154:6835","nonce":4059835337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:35:00.674725+0000","modified":"2024-09-16T01:35:02.681832+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4830":{"gid":4830,"name":"c","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6835/4230443772","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4230443772},{"type":"v1","addr":"172.21.15.88:6835","nonce":4230443772}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4830,"qdb_cluster":[4830]},"id":2}]} 2024-09-16T01:35:09.671 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 13 2024-09-16T01:35:09.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.669+0000 7f59ecee4640 1 -- 172.21.15.88:0/933590126 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f59c806c2b0 msgr2=0x7f59c806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:09.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.669+0000 7f59ecee4640 1 --2- 172.21.15.88:0/933590126 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f59c806c2b0 0x7f59c806e770 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f59d8001680 tx=0x7f59d8009040 comp rx=0 tx=0).stop 2024-09-16T01:35:09.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.669+0000 7f59ecee4640 1 -- 172.21.15.88:0/933590126 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59e80735b0 msgr2=0x7f59e814da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:09.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.669+0000 7f59ecee4640 1 --2- 172.21.15.88:0/933590126 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59e80735b0 0x7f59e814da60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f59d000d7d0 tx=0x7f59d000dc90 comp rx=0 tx=0).stop 2024-09-16T01:35:09.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.669+0000 7f59ecee4640 1 -- 172.21.15.88:0/933590126 shutdown_connections 2024-09-16T01:35:09.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.669+0000 7f59ecee4640 1 --2- 172.21.15.88:0/933590126 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f59c806c2b0 0x7f59c806e770 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:09.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.669+0000 7f59ecee4640 1 --2- 172.21.15.88:0/933590126 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59e815e950 0x7f59e81e93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:09.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.669+0000 7f59ecee4640 1 --2- 172.21.15.88:0/933590126 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59e80735b0 0x7f59e814da60 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:09.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.669+0000 7f59ecee4640 1 --2- 172.21.15.88:0/933590126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59e806d540 0x7f59e81617c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:09.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.669+0000 7f59ecee4640 1 -- 172.21.15.88:0/933590126 >> 172.21.15.88:0/933590126 conn(0x7f59e80d14d0 msgr2=0x7f59e80b5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:09.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.669+0000 7f59ecee4640 1 -- 172.21.15.88:0/933590126 shutdown_connections 2024-09-16T01:35:09.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:09.669+0000 7f59ecee4640 1 -- 172.21.15.88:0/933590126 wait complete. 2024-09-16T01:35:10.084 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 13, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:35:00.674725+0000', 'modified': '2024-09-16T01:35:02.681832+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4830}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4830': {'gid': 4830, 'name': 'c', 'rank': 0, 'incarnation': 11, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.88:6835/4230443772', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 4230443772}, {'type': 'v1', 'addr': '172.21.15.88:6835', 'nonce': 4230443772}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4830, 'qdb_cluster': [4830]} 2024-09-16T01:35:10.085 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T01:35:10.085 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-16T01:35:10.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.185+0000 7fcb4c91f640 1 Processor -- start 2024-09-16T01:35:10.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.185+0000 7fcb4c91f640 1 -- start start 2024-09-16T01:35:10.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.186+0000 7fcb4c91f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcb4815b480 0x7fcb480b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.186+0000 7fcb4c91f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcb48072700 0x7fcb480b3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.186+0000 7fcb4c91f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb480ccff0 con 0x7fcb480b60c0 2024-09-16T01:35:10.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.186+0000 7fcb4c91f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fcb480cd2d0 con 0x7fcb4815b480 2024-09-16T01:35:10.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.186+0000 7fcb4c91f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fcb480cd5b0 con 0x7fcb48072700 2024-09-16T01:35:10.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.186+0000 7fcb4659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcb4815b480 0x7fcb480b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.186+0000 7fcb3dd9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fcb480b60c0 0x7fcb48071ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35996/0 (socket says 172.21.15.88:35996) 2024-09-16T01:35:10.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.186+0000 7fcb3dd9b640 1 -- 172.21.15.88:0/1815808234 learned_addr learned my addr 172.21.15.88:0/1815808234 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:10.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.186+0000 7fcb46d9d640 1 --2- 172.21.15.88:0/1815808234 >> v2:172.21.15.154:3300/0 conn(0x7fcb48072700 0x7fcb480b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.186+0000 7fcb4659c640 1 -- 172.21.15.88:0/1815808234 >> v2:172.21.15.154:3300/0 conn(0x7fcb48072700 msgr2=0x7fcb480b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:10.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.186+0000 7fcb4659c640 1 --2- 172.21.15.88:0/1815808234 >> v2:172.21.15.154:3300/0 conn(0x7fcb48072700 0x7fcb480b3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.186+0000 7fcb4659c640 1 -- 172.21.15.88:0/1815808234 >> v1:172.21.15.88:6789/0 conn(0x7fcb480b60c0 legacy=0x7fcb48071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:10.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.186+0000 7fcb45d9b640 1 -- 172.21.15.88:0/1815808234 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4277394683 0 0) 0x7fcb480ccff0 con 0x7fcb480b60c0 2024-09-16T01:35:10.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.186+0000 7fcb4659c640 1 -- 172.21.15.88:0/1815808234 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb480cd890 con 0x7fcb4815b480 2024-09-16T01:35:10.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.187+0000 7fcb4659c640 1 --2- 172.21.15.88:0/1815808234 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb4815b480 0x7fcb480b5b80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fcb480cdf80 tx=0x7fcb3802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=723aea9eab18aabb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:10.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.187+0000 7fcb45d9b640 1 -- 172.21.15.88:0/1815808234 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb38037070 con 0x7fcb4815b480 2024-09-16T01:35:10.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.187+0000 7fcb45d9b640 1 -- 172.21.15.88:0/1815808234 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb38032440 con 0x7fcb4815b480 2024-09-16T01:35:10.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.187+0000 7fcb45d9b640 1 -- 172.21.15.88:0/1815808234 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb3802ae60 con 0x7fcb4815b480 2024-09-16T01:35:10.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.188+0000 7fcb4c91f640 1 -- 172.21.15.88:0/1815808234 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb4815b480 msgr2=0x7fcb480b5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:10.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.188+0000 7fcb4c91f640 1 --2- 172.21.15.88:0/1815808234 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb4815b480 0x7fcb480b5b80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fcb480cdf80 tx=0x7fcb3802a110 comp rx=0 tx=0).stop 2024-09-16T01:35:10.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.188+0000 7fcb4c91f640 1 -- 172.21.15.88:0/1815808234 shutdown_connections 2024-09-16T01:35:10.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.188+0000 7fcb4c91f640 1 --2- 172.21.15.88:0/1815808234 >> v2:172.21.15.154:3300/0 conn(0x7fcb48072700 0x7fcb480b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.188+0000 7fcb4c91f640 1 --2- 172.21.15.88:0/1815808234 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb4815b480 0x7fcb480b5b80 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.188+0000 7fcb4c91f640 1 -- 172.21.15.88:0/1815808234 >> 172.21.15.88:0/1815808234 conn(0x7fcb4806d0f0 msgr2=0x7fcb4806d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:10.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.188+0000 7fcb4c91f640 1 -- 172.21.15.88:0/1815808234 shutdown_connections 2024-09-16T01:35:10.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.188+0000 7fcb4c91f640 1 -- 172.21.15.88:0/1815808234 wait complete. 2024-09-16T01:35:10.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.189+0000 7fcb4c91f640 1 Processor -- start 2024-09-16T01:35:10.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.189+0000 7fcb4c91f640 1 -- start start 2024-09-16T01:35:10.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.190+0000 7fcb4c91f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb48072700 0x7fcb481506c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.190+0000 7fcb4c91f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb480b60c0 0x7fcb480cf740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.190+0000 7fcb4c91f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb4815b480 0x7fcb48156840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.190+0000 7fcb4c91f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4815b9e0 con 0x7fcb480b60c0 2024-09-16T01:35:10.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.190+0000 7fcb4c91f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4815b860 con 0x7fcb48072700 2024-09-16T01:35:10.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.190+0000 7fcb4c91f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcb480b3ad0 con 0x7fcb4815b480 2024-09-16T01:35:10.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.190+0000 7fcb3dd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb480b60c0 0x7fcb480cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.190+0000 7fcb3dd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb480b60c0 0x7fcb480cf740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36596/0 (socket says 172.21.15.88:36596) 2024-09-16T01:35:10.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.190+0000 7fcb3dd9b640 1 -- 172.21.15.88:0/300456736 learned_addr learned my addr 172.21.15.88:0/300456736 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:10.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.190+0000 7fcb4659c640 1 --2- 172.21.15.88:0/300456736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb48072700 0x7fcb481506c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.190+0000 7fcb46d9d640 1 --2- 172.21.15.88:0/300456736 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb4815b480 0x7fcb48156840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.190+0000 7fcb3dd9b640 1 -- 172.21.15.88:0/300456736 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb4815b480 msgr2=0x7fcb48156840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:10.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.190+0000 7fcb3dd9b640 1 --2- 172.21.15.88:0/300456736 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb4815b480 0x7fcb48156840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.190+0000 7fcb3dd9b640 1 -- 172.21.15.88:0/300456736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb48072700 msgr2=0x7fcb481506c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:10.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.190+0000 7fcb3dd9b640 1 --2- 172.21.15.88:0/300456736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb48072700 0x7fcb481506c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.190+0000 7fcb3dd9b640 1 -- 172.21.15.88:0/300456736 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb48150c00 con 0x7fcb480b60c0 2024-09-16T01:35:10.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.190+0000 7fcb4659c640 1 --2- 172.21.15.88:0/300456736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb48072700 0x7fcb481506c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:10.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.191+0000 7fcb3dd9b640 1 --2- 172.21.15.88:0/300456736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb480b60c0 0x7fcb480cf740 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fcb3000c4a0 tx=0x7fcb3000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:10.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.191+0000 7fcb3f7fe640 1 -- 172.21.15.88:0/300456736 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb30018070 con 0x7fcb480b60c0 2024-09-16T01:35:10.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.191+0000 7fcb3f7fe640 1 -- 172.21.15.88:0/300456736 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb300074e0 con 0x7fcb480b60c0 2024-09-16T01:35:10.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.191+0000 7fcb4c91f640 1 -- 172.21.15.88:0/300456736 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb480d0d70 con 0x7fcb480b60c0 2024-09-16T01:35:10.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.191+0000 7fcb3f7fe640 1 -- 172.21.15.88:0/300456736 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb30004a20 con 0x7fcb480b60c0 2024-09-16T01:35:10.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.191+0000 7fcb4c91f640 1 -- 172.21.15.88:0/300456736 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb480cef40 con 0x7fcb480b60c0 2024-09-16T01:35:10.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.193+0000 7fcb3f7fe640 1 -- 172.21.15.88:0/300456736 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fcb3001b070 con 0x7fcb480b60c0 2024-09-16T01:35:10.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.193+0000 7fcb3f7fe640 1 --2- 172.21.15.88:0/300456736 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcb2806c2b0 0x7fcb2806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.193+0000 7fcb3f7fe640 1 -- 172.21.15.88:0/300456736 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fcb3008a790 con 0x7fcb480b60c0 2024-09-16T01:35:10.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.193+0000 7fcb4659c640 1 --2- 172.21.15.88:0/300456736 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcb2806c2b0 0x7fcb2806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.194+0000 7fcb4659c640 1 --2- 172.21.15.88:0/300456736 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcb2806c2b0 0x7fcb2806e770 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcb38034000 tx=0x7fcb3802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:10.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.194+0000 7fcb4f6b4740 1 Processor -- start 2024-09-16T01:35:10.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.194+0000 7fcb4f6b4740 1 -- start start 2024-09-16T01:35:10.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.195+0000 7fcb4f6b4740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55590f26c2c0 0x55590f2b1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.195+0000 7fcb4f6b4740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f269a80 0x55590f26c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.195+0000 7fcb4f6b4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55590f26da50 0x55590f2668b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.195+0000 7fcb4f6b4740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55590f1ce520 con 0x55590f269a80 2024-09-16T01:35:10.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.195+0000 7fcb4f6b4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55590f1763e0 con 0x55590f26da50 2024-09-16T01:35:10.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.195+0000 7fcb4f6b4740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55590f1ca4e0 con 0x55590f26c2c0 2024-09-16T01:35:10.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.195+0000 7fcb3dd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f269a80 0x55590f26c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.195+0000 7fcb3dd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f269a80 0x55590f26c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36598/0 (socket says 172.21.15.88:36598) 2024-09-16T01:35:10.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.195+0000 7fcb3dd9b640 1 -- 172.21.15.88:0/2798044960 learned_addr learned my addr 172.21.15.88:0/2798044960 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:10.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.195+0000 7fcb4659c640 1 --2- 172.21.15.88:0/2798044960 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55590f26c2c0 0x55590f2b1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.195+0000 7fcb46d9d640 1 --2- 172.21.15.88:0/2798044960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55590f26da50 0x55590f2668b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.195+0000 7fcb3dd9b640 1 -- 172.21.15.88:0/2798044960 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55590f26c2c0 msgr2=0x55590f2b1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:10.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.195+0000 7fcb3dd9b640 1 --2- 172.21.15.88:0/2798044960 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55590f26c2c0 0x55590f2b1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.195+0000 7fcb3dd9b640 1 -- 172.21.15.88:0/2798044960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55590f26da50 msgr2=0x55590f2668b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.195+0000 7fcb3dd9b640 1 --2- 172.21.15.88:0/2798044960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55590f26da50 0x55590f2668b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.195+0000 7fcb3dd9b640 1 -- 172.21.15.88:0/2798044960 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55590f1912f0 con 0x55590f269a80 2024-09-16T01:35:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.195+0000 7fcb46d9d640 1 --2- 172.21.15.88:0/2798044960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55590f26da50 0x55590f2668b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.195+0000 7fcb3dd9b640 1 --2- 172.21.15.88:0/2798044960 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f269a80 0x55590f26c6a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fcb30088c30 tx=0x7fcb3000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b64713f98fc73ac8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.195+0000 7fcb27fff640 1 -- 172.21.15.88:0/2798044960 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb3005f070 con 0x55590f269a80 2024-09-16T01:35:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.195+0000 7fcb27fff640 1 -- 172.21.15.88:0/2798044960 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb30056b00 con 0x55590f269a80 2024-09-16T01:35:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.195+0000 7fcb27fff640 1 -- 172.21.15.88:0/2798044960 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb30004bc0 con 0x55590f269a80 2024-09-16T01:35:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.196+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/2798044960 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f269a80 msgr2=0x55590f26c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.196+0000 7fcb4f6b4740 1 --2- 172.21.15.88:0/2798044960 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f269a80 0x55590f26c6a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fcb30088c30 tx=0x7fcb3000cf90 comp rx=0 tx=0).stop 2024-09-16T01:35:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.196+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/2798044960 shutdown_connections 2024-09-16T01:35:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.196+0000 7fcb4f6b4740 1 --2- 172.21.15.88:0/2798044960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55590f26da50 0x55590f2668b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.196+0000 7fcb4f6b4740 1 --2- 172.21.15.88:0/2798044960 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f269a80 0x55590f26c6a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.196+0000 7fcb4f6b4740 1 --2- 172.21.15.88:0/2798044960 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55590f26c2c0 0x55590f2b1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.196+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/2798044960 >> 172.21.15.88:0/2798044960 conn(0x55590f268a80 msgr2=0x55590f2b0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:10.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.196+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/2798044960 shutdown_connections 2024-09-16T01:35:10.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.196+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/2798044960 wait complete. 2024-09-16T01:35:10.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.197+0000 7fcb4f6b4740 1 -- start start 2024-09-16T01:35:10.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.197+0000 7fcb4f6b4740 1 Processor -- start 2024-09-16T01:35:10.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.198+0000 7fcb4f6b4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55590f26c2c0 0x55590f295960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.198+0000 7fcb4f6b4740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f26da50 0x55590f2b71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.198+0000 7fcb4f6b4740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55590f292830 0x55590f297cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.198+0000 7fcb4f6b4740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55590f267f20 con 0x55590f26da50 2024-09-16T01:35:10.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.198+0000 7fcb4f6b4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55590f157450 con 0x55590f26c2c0 2024-09-16T01:35:10.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.198+0000 7fcb4f6b4740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55590f2b2290 con 0x55590f292830 2024-09-16T01:35:10.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.198+0000 7fcb4f6b4740 10 client.0 _ll_register_callbacks cb 0x55590eee0ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:35:10.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.198+0000 7fcb3dd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f26da50 0x55590f2b71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.198+0000 7fcb3dd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f26da50 0x55590f2b71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36612/0 (socket says 172.21.15.88:36612) 2024-09-16T01:35:10.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.198+0000 7fcb3dd9b640 1 -- 172.21.15.88:0/3508567205 learned_addr learned my addr 172.21.15.88:0/3508567205 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:10.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.198+0000 7fcb4659c640 1 --2- 172.21.15.88:0/3508567205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55590f26c2c0 0x55590f295960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.198+0000 7fcb46d9d640 1 --2- 172.21.15.88:0/3508567205 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55590f292830 0x55590f297cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.198+0000 7fcb3dd9b640 1 -- 172.21.15.88:0/3508567205 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55590f292830 msgr2=0x55590f297cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:10.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.198+0000 7fcb3dd9b640 1 --2- 172.21.15.88:0/3508567205 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55590f292830 0x55590f297cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.198+0000 7fcb3dd9b640 1 -- 172.21.15.88:0/3508567205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55590f26c2c0 msgr2=0x55590f295960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:10.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.198+0000 7fcb3dd9b640 1 --2- 172.21.15.88:0/3508567205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55590f26c2c0 0x55590f295960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.198+0000 7fcb3dd9b640 1 -- 172.21.15.88:0/3508567205 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55590eee12d0 con 0x55590f26da50 2024-09-16T01:35:10.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.198+0000 7fcb3dd9b640 1 --2- 172.21.15.88:0/3508567205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f26da50 0x55590f2b71f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fcb30058500 tx=0x7fcb30007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c867cc480b8d42de server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:10.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.198+0000 7fcb257fa640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:35:10.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.198+0000 7fcb257fa640 1 -- 172.21.15.88:0/3508567205 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb30004bc0 con 0x55590f26da50 2024-09-16T01:35:10.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.198+0000 7fcb257fa640 1 -- 172.21.15.88:0/3508567205 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fcb30004d60 con 0x55590f26da50 2024-09-16T01:35:10.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.198+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/3508567205 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb30056b00 con 0x55590f26da50 2024-09-16T01:35:10.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.199+0000 7fcb257fa640 1 -- 172.21.15.88:0/3508567205 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) ==== 24+0+0 (secure 0 0 0) 0x7fcb3007c050 con 0x55590f26da50 2024-09-16T01:35:10.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.199+0000 7fcb4f6b4740 10 client.5028 fetch_fsmap learned FSMap version 13 2024-09-16T01:35:10.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.199+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/3508567205 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) -- 0x55590efab6b0 con 0x55590f26da50 2024-09-16T01:35:10.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.199+0000 7fcb257fa640 1 -- 172.21.15.88:0/3508567205 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb30056b00 con 0x55590f26da50 2024-09-16T01:35:10.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.199+0000 7fcb257fa640 1 -- 172.21.15.88:0/3508567205 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 13) ==== 3502+0+0 (secure 0 0 0) 0x7fcb3007f020 con 0x55590f26da50 2024-09-16T01:35:10.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.199+0000 7fcb4f6b4740 10 client.5028 fetch_fsmap finished waiting for FSMap version 13 2024-09-16T01:35:10.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.199+0000 7fcb4f6b4740 10 client.5028 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T01:35:10.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.199+0000 7fcb4f6b4740 20 client.5028 populate_metadata read hostname 'smithi088' 2024-09-16T01:35:10.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.199+0000 7fcb4f6b4740 1 --2- 172.21.15.88:0/3508567205 >> [v2:172.21.15.88:6834/4230443772,v1:172.21.15.88:6835/4230443772] conn(0x55590f2c1020 0x55590f2c1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.199+0000 7fcb4f6b4740 4 client.5028 mds_command: new command op to 4830 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:35:10.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.199+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/3508567205 --> [v2:172.21.15.88:6834/4230443772,v1:172.21.15.88:6835/4230443772] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55590f1dbc50 con 0x55590f2c1020 2024-09-16T01:35:10.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.199+0000 7fcb4659c640 1 --2- 172.21.15.88:0/3508567205 >> [v2:172.21.15.88:6834/4230443772,v1:172.21.15.88:6835/4230443772] conn(0x55590f2c1020 0x55590f2c1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.199+0000 7fcb257fa640 10 client.5028 ms_handle_connect on v2:172.21.15.88:6834/4230443772 2024-09-16T01:35:10.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.199+0000 7fcb4659c640 1 --2- 172.21.15.88:0/3508567205 >> [v2:172.21.15.88:6834/4230443772,v1:172.21.15.88:6835/4230443772] conn(0x55590f2c1020 0x55590f2c1400 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98fc28e9c1d5fa1c server_cookie=29df6f2d707b19e7 in_seq=0 out_seq=0 2024-09-16T01:35:10.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.200+0000 7fcb257fa640 1 -- 172.21.15.88:0/3508567205 <== mds.0 v2:172.21.15.88:6834/4230443772 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55590f1dbc50 con 0x55590f2c1020 2024-09-16T01:35:10.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.200+0000 7fcb257fa640 10 client.5028 handle_command_reply: tid=0 2024-09-16T01:35:10.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.200+0000 7fcb4f6b4740 1 client.5028 shutdown 2024-09-16T01:35:10.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.200+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/3508567205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f26da50 msgr2=0x55590f2b71f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:10.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.200+0000 7fcb4f6b4740 1 --2- 172.21.15.88:0/3508567205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f26da50 0x55590f2b71f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fcb30058500 tx=0x7fcb30007f90 comp rx=0 tx=0).stop 2024-09-16T01:35:10.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.200+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/3508567205 shutdown_connections 2024-09-16T01:35:10.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.200+0000 7fcb4f6b4740 1 --2- 172.21.15.88:0/3508567205 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55590f292830 0x55590f297cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.200+0000 7fcb4f6b4740 1 --2- 172.21.15.88:0/3508567205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f26da50 0x55590f2b71f0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.200+0000 7fcb4f6b4740 1 --2- 172.21.15.88:0/3508567205 >> [v2:172.21.15.88:6834/4230443772,v1:172.21.15.88:6835/4230443772] conn(0x55590f2c1020 0x55590f2c1400 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.200+0000 7fcb4f6b4740 1 --2- 172.21.15.88:0/3508567205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55590f26c2c0 0x55590f295960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.200+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/3508567205 >> 172.21.15.88:0/3508567205 conn(0x55590f269a80 msgr2=0x55590f2b2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:10.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.200+0000 7fcb257fa640 0 client.5028 ms_handle_reset on v2:172.21.15.88:6834/4230443772 2024-09-16T01:35:10.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.200+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/3508567205 shutdown_connections 2024-09-16T01:35:10.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.200+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/3508567205 wait complete. 2024-09-16T01:35:10.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.200+0000 7fcb4f6b4740 20 client.5028 trim_cache size 0 max 16384 2024-09-16T01:35:10.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.218+0000 7fcb4f6b4740 1 Processor -- start 2024-09-16T01:35:10.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.218+0000 7fcb4f6b4740 1 -- start start 2024-09-16T01:35:10.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.219+0000 7fcb4f6b4740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55590f26c2c0 0x55590f2af0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.219+0000 7fcb4f6b4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55590f26da50 0x55590f2b0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.219+0000 7fcb4f6b4740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f292830 0x55590f2b1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.219+0000 7fcb4f6b4740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcb481fd9e0 con 0x55590f292830 2024-09-16T01:35:10.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.219+0000 7fcb4f6b4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcb481594d0 con 0x55590f26da50 2024-09-16T01:35:10.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.219+0000 7fcb4f6b4740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcb48159b20 con 0x55590f26c2c0 2024-09-16T01:35:10.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.219+0000 7fcb46d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f292830 0x55590f2b1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.219+0000 7fcb46d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f292830 0x55590f2b1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36618/0 (socket says 172.21.15.88:36618) 2024-09-16T01:35:10.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.219+0000 7fcb46d9d640 1 -- 172.21.15.88:0/2335850056 learned_addr learned my addr 172.21.15.88:0/2335850056 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:10.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.219+0000 7fcb3dd9b640 1 --2- 172.21.15.88:0/2335850056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55590f26da50 0x55590f2b0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.219+0000 7fcb4659c640 1 --2- 172.21.15.88:0/2335850056 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55590f26c2c0 0x55590f2af0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.219+0000 7fcb46d9d640 1 -- 172.21.15.88:0/2335850056 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55590f26c2c0 msgr2=0x55590f2af0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:10.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.219+0000 7fcb46d9d640 1 --2- 172.21.15.88:0/2335850056 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55590f26c2c0 0x55590f2af0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.219+0000 7fcb46d9d640 1 -- 172.21.15.88:0/2335850056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55590f26da50 msgr2=0x55590f2b0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:10.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.219+0000 7fcb46d9d640 1 --2- 172.21.15.88:0/2335850056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55590f26da50 0x55590f2b0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.219+0000 7fcb46d9d640 1 -- 172.21.15.88:0/2335850056 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55590f26ca10 con 0x55590f292830 2024-09-16T01:35:10.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.219+0000 7fcb46d9d640 1 --2- 172.21.15.88:0/2335850056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f292830 0x55590f2b1340 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fcb40009f20 tx=0x7fcb400126d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ba67aa58ee37889 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:10.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.219+0000 7fcb277fe640 1 -- 172.21.15.88:0/2335850056 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4001a070 con 0x55590f292830 2024-09-16T01:35:10.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.219+0000 7fcb277fe640 1 -- 172.21.15.88:0/2335850056 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb40016630 con 0x55590f292830 2024-09-16T01:35:10.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.219+0000 7fcb277fe640 1 -- 172.21.15.88:0/2335850056 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb40016a20 con 0x55590f292830 2024-09-16T01:35:10.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.220+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/2335850056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f292830 msgr2=0x55590f2b1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:10.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.220+0000 7fcb4f6b4740 1 --2- 172.21.15.88:0/2335850056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f292830 0x55590f2b1340 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fcb40009f20 tx=0x7fcb400126d0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.220+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/2335850056 shutdown_connections 2024-09-16T01:35:10.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.220+0000 7fcb4f6b4740 1 --2- 172.21.15.88:0/2335850056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f292830 0x55590f2b1340 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.220+0000 7fcb4f6b4740 1 --2- 172.21.15.88:0/2335850056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55590f26da50 0x55590f2b0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.220+0000 7fcb4f6b4740 1 --2- 172.21.15.88:0/2335850056 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55590f26c2c0 0x55590f2af0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.220+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/2335850056 >> 172.21.15.88:0/2335850056 conn(0x55590f269a80 msgr2=0x55590f2adb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:10.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.220+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/2335850056 shutdown_connections 2024-09-16T01:35:10.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.220+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/2335850056 wait complete. 2024-09-16T01:35:10.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.220+0000 7fcb4f6b4740 1 -- start start 2024-09-16T01:35:10.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.221+0000 7fcb4f6b4740 1 Processor -- start 2024-09-16T01:35:10.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.221+0000 7fcb4f6b4740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55590f26c2c0 0x55590f2c0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.221+0000 7fcb4f6b4740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f26da50 0x55590f2d44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.221+0000 7fcb4f6b4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55590f292830 0x55590f2b10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.221+0000 7fcb4f6b4740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55590f267f20 con 0x55590f26da50 2024-09-16T01:35:10.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.221+0000 7fcb4f6b4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55590f157450 con 0x55590f292830 2024-09-16T01:35:10.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.222+0000 7fcb4f6b4740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55590f293cf0 con 0x55590f26c2c0 2024-09-16T01:35:10.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.222+0000 7fcb4f6b4740 10 client.0 _ll_register_callbacks cb 0x7fcb30058f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:35:10.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.222+0000 7fcb3dd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f26da50 0x55590f2d44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.222+0000 7fcb3dd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f26da50 0x55590f2d44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36628/0 (socket says 172.21.15.88:36628) 2024-09-16T01:35:10.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.222+0000 7fcb3dd9b640 1 -- 172.21.15.88:0/304489162 learned_addr learned my addr 172.21.15.88:0/304489162 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:10.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.222+0000 7fcb4659c640 1 --2- 172.21.15.88:0/304489162 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55590f26c2c0 0x55590f2c0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.222+0000 7fcb46d9d640 1 --2- 172.21.15.88:0/304489162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55590f292830 0x55590f2b10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.222+0000 7fcb3dd9b640 1 -- 172.21.15.88:0/304489162 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55590f26c2c0 msgr2=0x55590f2c0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:10.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.222+0000 7fcb3dd9b640 1 --2- 172.21.15.88:0/304489162 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55590f26c2c0 0x55590f2c0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.222+0000 7fcb3dd9b640 1 -- 172.21.15.88:0/304489162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55590f292830 msgr2=0x55590f2b10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:10.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.222+0000 7fcb3dd9b640 1 --2- 172.21.15.88:0/304489162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55590f292830 0x55590f2b10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.222+0000 7fcb3dd9b640 1 -- 172.21.15.88:0/304489162 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55590f2d4fc0 con 0x55590f26da50 2024-09-16T01:35:10.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.222+0000 7fcb3dd9b640 1 --2- 172.21.15.88:0/304489162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f26da50 0x55590f2d44e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fcb3000c470 tx=0x7fcb30006040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6c09e3a24a8d789 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:10.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.222+0000 7fcb257fa640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:35:10.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.222+0000 7fcb257fa640 1 -- 172.21.15.88:0/304489162 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb300585d0 con 0x55590f26da50 2024-09-16T01:35:10.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.222+0000 7fcb257fa640 1 -- 172.21.15.88:0/304489162 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fcb30059b30 con 0x55590f26da50 2024-09-16T01:35:10.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.222+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/304489162 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb40016630 con 0x55590f26da50 2024-09-16T01:35:10.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.222+0000 7fcb257fa640 1 -- 172.21.15.88:0/304489162 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) ==== 24+0+0 (secure 0 0 0) 0x7fcb30059d30 con 0x55590f26da50 2024-09-16T01:35:10.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.222+0000 7fcb4f6b4740 10 client.5034 fetch_fsmap learned FSMap version 13 2024-09-16T01:35:10.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.222+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/304489162 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) -- 0x55590f2d4fc0 con 0x55590f26da50 2024-09-16T01:35:10.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.222+0000 7fcb257fa640 1 -- 172.21.15.88:0/304489162 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb40016630 con 0x55590f26da50 2024-09-16T01:35:10.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.222+0000 7fcb257fa640 1 -- 172.21.15.88:0/304489162 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 13) ==== 3502+0+0 (secure 0 0 0) 0x7fcb30062ce0 con 0x55590f26da50 2024-09-16T01:35:10.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.222+0000 7fcb4f6b4740 10 client.5034 fetch_fsmap finished waiting for FSMap version 13 2024-09-16T01:35:10.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.223+0000 7fcb4f6b4740 10 client.5034 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T01:35:10.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.223+0000 7fcb4f6b4740 20 client.5034 populate_metadata read hostname 'smithi088' 2024-09-16T01:35:10.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.223+0000 7fcb4f6b4740 1 --2- 172.21.15.88:0/304489162 >> [v2:172.21.15.88:6834/4230443772,v1:172.21.15.88:6835/4230443772] conn(0x55590f332cb0 0x55590f2b5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.223+0000 7fcb4f6b4740 4 client.5034 mds_command: new command op to 4830 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:35:10.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.223+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/304489162 --> [v2:172.21.15.88:6834/4230443772,v1:172.21.15.88:6835/4230443772] -- command(tid 0: {"prefix": "status"}) -- 0x55590f1d5fa0 con 0x55590f332cb0 2024-09-16T01:35:10.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.223+0000 7fcb4659c640 1 --2- 172.21.15.88:0/304489162 >> [v2:172.21.15.88:6834/4230443772,v1:172.21.15.88:6835/4230443772] conn(0x55590f332cb0 0x55590f2b5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.223+0000 7fcb257fa640 10 client.5034 ms_handle_connect on v2:172.21.15.88:6834/4230443772 2024-09-16T01:35:10.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.223+0000 7fcb4659c640 1 --2- 172.21.15.88:0/304489162 >> [v2:172.21.15.88:6834/4230443772,v1:172.21.15.88:6835/4230443772] conn(0x55590f332cb0 0x55590f2b5520 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e9d691c664a58cb server_cookie=e6f9a5c2840030c8 in_seq=0 out_seq=0 2024-09-16T01:35:10.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.223+0000 7fcb257fa640 1 -- 172.21.15.88:0/304489162 <== mds.0 v2:172.21.15.88:6834/4230443772 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55590f1d5fa0 con 0x55590f332cb0 2024-09-16T01:35:10.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.223+0000 7fcb257fa640 10 client.5034 handle_command_reply: tid=0 2024-09-16T01:35:10.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.223+0000 7fcb4f6b4740 1 client.5034 shutdown 2024-09-16T01:35:10.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.223+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/304489162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f26da50 msgr2=0x55590f2d44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:10.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.223+0000 7fcb4f6b4740 1 --2- 172.21.15.88:0/304489162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f26da50 0x55590f2d44e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fcb3000c470 tx=0x7fcb30006040 comp rx=0 tx=0).stop 2024-09-16T01:35:10.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.223+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/304489162 shutdown_connections 2024-09-16T01:35:10.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.223+0000 7fcb4f6b4740 1 --2- 172.21.15.88:0/304489162 >> [v2:172.21.15.88:6834/4230443772,v1:172.21.15.88:6835/4230443772] conn(0x55590f332cb0 0x55590f2b5520 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.223+0000 7fcb4f6b4740 1 --2- 172.21.15.88:0/304489162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55590f292830 0x55590f2b10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.223+0000 7fcb4f6b4740 1 --2- 172.21.15.88:0/304489162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55590f26da50 0x55590f2d44e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.223+0000 7fcb257fa640 0 client.5034 ms_handle_reset on v2:172.21.15.88:6834/4230443772 2024-09-16T01:35:10.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.223+0000 7fcb4f6b4740 1 --2- 172.21.15.88:0/304489162 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55590f26c2c0 0x55590f2c0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.223+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/304489162 >> 172.21.15.88:0/304489162 conn(0x55590f269a80 msgr2=0x55590f298c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:10.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.223+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/304489162 shutdown_connections 2024-09-16T01:35:10.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.223+0000 7fcb4f6b4740 1 -- 172.21.15.88:0/304489162 wait complete. 2024-09-16T01:35:10.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.224+0000 7fcb4f6b4740 20 client.5034 trim_cache size 0 max 16384 2024-09-16T01:35:10.232 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:35:10.233 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "5ace2c91-46f9-482b-8a89-731fbdc171bf", 2024-09-16T01:35:10.233 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2024-09-16T01:35:10.233 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 4830, 2024-09-16T01:35:10.233 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2024-09-16T01:35:10.233 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2024-09-16T01:35:10.233 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2024-09-16T01:35:10.233 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 8.8812898350000005, 2024-09-16T01:35:10.233 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 13, 2024-09-16T01:35:10.233 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 32, 2024-09-16T01:35:10.233 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 32, 2024-09-16T01:35:10.233 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 15.793073521 2024-09-16T01:35:10.233 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:35:10.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.224+0000 7fcb4c91f640 1 -- 172.21.15.88:0/300456736 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcb2806c2b0 msgr2=0x7fcb2806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:10.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.224+0000 7fcb4c91f640 1 --2- 172.21.15.88:0/300456736 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcb2806c2b0 0x7fcb2806e770 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcb38034000 tx=0x7fcb3802e000 comp rx=0 tx=0).stop 2024-09-16T01:35:10.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.224+0000 7fcb4c91f640 1 -- 172.21.15.88:0/300456736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb480b60c0 msgr2=0x7fcb480cf740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:10.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.224+0000 7fcb4c91f640 1 --2- 172.21.15.88:0/300456736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb480b60c0 0x7fcb480cf740 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fcb3000c4a0 tx=0x7fcb3000c960 comp rx=0 tx=0).stop 2024-09-16T01:35:10.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.224+0000 7fcb4c91f640 1 -- 172.21.15.88:0/300456736 shutdown_connections 2024-09-16T01:35:10.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.224+0000 7fcb4c91f640 1 --2- 172.21.15.88:0/300456736 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcb2806c2b0 0x7fcb2806e770 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.224+0000 7fcb4c91f640 1 --2- 172.21.15.88:0/300456736 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb4815b480 0x7fcb48156840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.224+0000 7fcb4c91f640 1 --2- 172.21.15.88:0/300456736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb480b60c0 0x7fcb480cf740 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.224+0000 7fcb4c91f640 1 --2- 172.21.15.88:0/300456736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb48072700 0x7fcb481506c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.224+0000 7fcb4c91f640 1 -- 172.21.15.88:0/300456736 >> 172.21.15.88:0/300456736 conn(0x7fcb4806d0f0 msgr2=0x7fcb480b3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:10.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.224+0000 7fcb4c91f640 1 -- 172.21.15.88:0/300456736 shutdown_connections 2024-09-16T01:35:10.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.224+0000 7fcb4c91f640 1 -- 172.21.15.88:0/300456736 wait complete. 2024-09-16T01:35:10.236 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:35:10.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.333+0000 7fc170a4c640 1 Processor -- start 2024-09-16T01:35:10.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.333+0000 7fc170a4c640 1 -- start start 2024-09-16T01:35:10.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.334+0000 7fc170a4c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc16c0b74c0 0x7fc16c15eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.334+0000 7fc170a4c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc16c15fb10 con 0x7fc16c15f030 2024-09-16T01:35:10.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.334+0000 7fc170a4c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc16c0b5900 con 0x7fc16c15c2b0 2024-09-16T01:35:10.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.334+0000 7fc170a4c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc16c15f830 con 0x7fc16c0b74c0 2024-09-16T01:35:10.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.334+0000 7fc16ad9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc16c15f030 0x7fc16c06d910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54496/0 (socket says 172.21.15.88:54496) 2024-09-16T01:35:10.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.334+0000 7fc16ad9d640 1 -- 172.21.15.88:0/3599218827 learned_addr learned my addr 172.21.15.88:0/3599218827 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:10.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.334+0000 7fc163fff640 1 --2- 172.21.15.88:0/3599218827 >> v2:172.21.15.154:3300/0 conn(0x7fc16c0b74c0 0x7fc16c15eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.334+0000 7fc169d9b640 1 -- 172.21.15.88:0/3599218827 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1641527105 0 0) 0x7fc16c15fb10 con 0x7fc16c15f030 2024-09-16T01:35:10.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.334+0000 7fc169d9b640 1 -- 172.21.15.88:0/3599218827 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc158003520 con 0x7fc16c15f030 2024-09-16T01:35:10.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.334+0000 7fc169d9b640 1 -- 172.21.15.88:0/3599218827 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3244685826 0 0) 0x7fc158003520 con 0x7fc16c15f030 2024-09-16T01:35:10.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.334+0000 7fc169d9b640 1 -- 172.21.15.88:0/3599218827 >> v2:172.21.15.154:3300/0 conn(0x7fc16c0b74c0 msgr2=0x7fc16c15eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:10.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.334+0000 7fc169d9b640 1 --2- 172.21.15.88:0/3599218827 >> v2:172.21.15.154:3300/0 conn(0x7fc16c0b74c0 0x7fc16c15eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.334+0000 7fc169d9b640 1 -- 172.21.15.88:0/3599218827 >> v1:172.21.15.154:6789/0 conn(0x7fc16c15c2b0 legacy=0x7fc16c0b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:10.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.334+0000 7fc169d9b640 1 -- 172.21.15.88:0/3599218827 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc16c06ce30 con 0x7fc16c15f030 2024-09-16T01:35:10.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.334+0000 7fc169d9b640 1 -- 172.21.15.88:0/3599218827 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fc1640019f0 con 0x7fc16c15f030 2024-09-16T01:35:10.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.334+0000 7fc169d9b640 1 -- 172.21.15.88:0/3599218827 >> v1:172.21.15.88:6789/0 conn(0x7fc16c15f030 legacy=0x7fc16c06d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:10.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.334+0000 7fc169d9b640 1 --2- 172.21.15.88:0/3599218827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc158003a00 0x7fc158003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.334+0000 7fc169d9b640 1 --2- 172.21.15.88:0/3599218827 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc158004800 0x7fc158004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.334+0000 7fc169d9b640 1 --2- 172.21.15.88:0/3599218827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1580051c0 0x7fc15800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.334+0000 7fc169d9b640 1 -- 172.21.15.88:0/3599218827 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc16c15f830 con 0x7fc158003a00 2024-09-16T01:35:10.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.335+0000 7fc169d9b640 1 -- 172.21.15.88:0/3599218827 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc15800c0c0 con 0x7fc1580051c0 2024-09-16T01:35:10.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.335+0000 7fc169d9b640 1 -- 172.21.15.88:0/3599218827 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc15800c2c0 con 0x7fc158004800 2024-09-16T01:35:10.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.335+0000 7fc16a59c640 1 --2- 172.21.15.88:0/3599218827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc158003a00 0x7fc158003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.335+0000 7fc163fff640 1 --2- 172.21.15.88:0/3599218827 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc158004800 0x7fc158004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.335+0000 7fc16ad9d640 1 --2- 172.21.15.88:0/3599218827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1580051c0 0x7fc15800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.335+0000 7fc16a59c640 1 -- 172.21.15.88:0/3599218827 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc158004800 msgr2=0x7fc158004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:10.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.335+0000 7fc16a59c640 1 --2- 172.21.15.88:0/3599218827 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc158004800 0x7fc158004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.335+0000 7fc16a59c640 1 -- 172.21.15.88:0/3599218827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1580051c0 msgr2=0x7fc15800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:10.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.335+0000 7fc16a59c640 1 --2- 172.21.15.88:0/3599218827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1580051c0 0x7fc15800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.335+0000 7fc16a59c640 1 -- 172.21.15.88:0/3599218827 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc158003520 con 0x7fc158003a00 2024-09-16T01:35:10.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.335+0000 7fc16ad9d640 1 --2- 172.21.15.88:0/3599218827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1580051c0 0x7fc15800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:10.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.335+0000 7fc16ad9d640 1 -- 172.21.15.88:0/3599218827 reap_dead start 2024-09-16T01:35:10.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.335+0000 7fc16a59c640 1 --2- 172.21.15.88:0/3599218827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc158003a00 0x7fc158003e20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc15c001750 tx=0x7fc158005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd1d8292045be780 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:10.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.336+0000 7fc169d9b640 1 -- 172.21.15.88:0/3599218827 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc15c001370 con 0x7fc158003a00 2024-09-16T01:35:10.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.336+0000 7fc169d9b640 1 -- 172.21.15.88:0/3599218827 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc15c0052d0 con 0x7fc158003a00 2024-09-16T01:35:10.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.336+0000 7fc169d9b640 1 -- 172.21.15.88:0/3599218827 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc15c02abc0 con 0x7fc158003a00 2024-09-16T01:35:10.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.336+0000 7fc170a4c640 1 -- 172.21.15.88:0/3599218827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc158003a00 msgr2=0x7fc158003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:10.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.336+0000 7fc170a4c640 1 --2- 172.21.15.88:0/3599218827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc158003a00 0x7fc158003e20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc15c001750 tx=0x7fc158005830 comp rx=0 tx=0).stop 2024-09-16T01:35:10.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.336+0000 7fc170a4c640 1 -- 172.21.15.88:0/3599218827 shutdown_connections 2024-09-16T01:35:10.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.336+0000 7fc170a4c640 1 --2- 172.21.15.88:0/3599218827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc158003a00 0x7fc158003e20 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.336+0000 7fc170a4c640 1 -- 172.21.15.88:0/3599218827 >> 172.21.15.88:0/3599218827 conn(0x7fc16c0b4240 msgr2=0x7fc16c0b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:10.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.336+0000 7fc170a4c640 1 -- 172.21.15.88:0/3599218827 shutdown_connections 2024-09-16T01:35:10.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.336+0000 7fc170a4c640 1 -- 172.21.15.88:0/3599218827 wait complete. 2024-09-16T01:35:10.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.338+0000 7fc170a4c640 1 Processor -- start 2024-09-16T01:35:10.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.338+0000 7fc170a4c640 1 -- start start 2024-09-16T01:35:10.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.338+0000 7fc170a4c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc158003a00 0x7fc16c1f49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.338+0000 7fc170a4c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc16c0d2660 0x7fc16c157ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.338+0000 7fc170a4c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc16c0d2ae0 0x7fc16c0cecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.338+0000 7fc170a4c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc16c15f480 con 0x7fc16c0d2660 2024-09-16T01:35:10.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.338+0000 7fc170a4c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc16c06d3f0 con 0x7fc16c0d2ae0 2024-09-16T01:35:10.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.338+0000 7fc170a4c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc16c156e50 con 0x7fc158003a00 2024-09-16T01:35:10.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.338+0000 7fc163fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc16c0d2660 0x7fc16c157ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.338+0000 7fc163fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc16c0d2660 0x7fc16c157ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33548/0 (socket says 172.21.15.88:33548) 2024-09-16T01:35:10.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.338+0000 7fc163fff640 1 -- 172.21.15.88:0/196891460 learned_addr learned my addr 172.21.15.88:0/196891460 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:10.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.339+0000 7fc16a59c640 1 --2- 172.21.15.88:0/196891460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc158003a00 0x7fc16c1f49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.339+0000 7fc16ad9d640 1 --2- 172.21.15.88:0/196891460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc16c0d2ae0 0x7fc16c0cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.339+0000 7fc163fff640 1 -- 172.21.15.88:0/196891460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc158003a00 msgr2=0x7fc16c1f49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:10.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.339+0000 7fc163fff640 1 --2- 172.21.15.88:0/196891460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc158003a00 0x7fc16c1f49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.339+0000 7fc163fff640 1 -- 172.21.15.88:0/196891460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc16c0d2ae0 msgr2=0x7fc16c0cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:10.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.339+0000 7fc163fff640 1 --2- 172.21.15.88:0/196891460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc16c0d2ae0 0x7fc16c0cecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.339+0000 7fc163fff640 1 -- 172.21.15.88:0/196891460 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc16c157130 con 0x7fc16c0d2660 2024-09-16T01:35:10.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.339+0000 7fc163fff640 1 --2- 172.21.15.88:0/196891460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc16c0d2660 0x7fc16c157ed0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc154011a70 tx=0x7fc154011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:10.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.339+0000 7fc162ffd640 1 -- 172.21.15.88:0/196891460 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc15400fb50 con 0x7fc16c0d2660 2024-09-16T01:35:10.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.339+0000 7fc162ffd640 1 -- 172.21.15.88:0/196891460 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc15400fcf0 con 0x7fc16c0d2660 2024-09-16T01:35:10.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.339+0000 7fc170a4c640 1 -- 172.21.15.88:0/196891460 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc16c157410 con 0x7fc16c0d2660 2024-09-16T01:35:10.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.339+0000 7fc162ffd640 1 -- 172.21.15.88:0/196891460 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc15400fea0 con 0x7fc16c0d2660 2024-09-16T01:35:10.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.339+0000 7fc170a4c640 1 -- 172.21.15.88:0/196891460 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc16c1579d0 con 0x7fc16c0d2660 2024-09-16T01:35:10.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.340+0000 7fc170a4c640 1 -- 172.21.15.88:0/196891460 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc16c15eaf0 con 0x7fc16c0d2660 2024-09-16T01:35:10.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.341+0000 7fc162ffd640 1 -- 172.21.15.88:0/196891460 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc154012040 con 0x7fc16c0d2660 2024-09-16T01:35:10.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.341+0000 7fc162ffd640 1 --2- 172.21.15.88:0/196891460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc14c06c2b0 0x7fc14c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:10.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.341+0000 7fc162ffd640 1 -- 172.21.15.88:0/196891460 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fc15408bb20 con 0x7fc16c0d2660 2024-09-16T01:35:10.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.341+0000 7fc16a59c640 1 --2- 172.21.15.88:0/196891460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc14c06c2b0 0x7fc14c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:10.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.341+0000 7fc16a59c640 1 --2- 172.21.15.88:0/196891460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc14c06c2b0 0x7fc14c06e770 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc15c032000 tx=0x7fc15c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:10.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.345+0000 7fc162ffd640 1 -- 172.21.15.88:0/196891460 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc154019030 con 0x7fc16c0d2660 2024-09-16T01:35:10.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.547+0000 7fc170a4c640 1 -- 172.21.15.88:0/196891460 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc16c0b46c0 con 0x7fc16c0d2660 2024-09-16T01:35:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.547+0000 7fc162ffd640 1 -- 172.21.15.88:0/196891460 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v13) ==== 53+0+73 (secure 0 0 0) 0x7fc15405a290 con 0x7fc16c0d2660 2024-09-16T01:35:10.549 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:35:10.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.549+0000 7fc170a4c640 1 -- 172.21.15.88:0/196891460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc14c06c2b0 msgr2=0x7fc14c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:10.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.549+0000 7fc170a4c640 1 --2- 172.21.15.88:0/196891460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc14c06c2b0 0x7fc14c06e770 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc15c032000 tx=0x7fc15c02e000 comp rx=0 tx=0).stop 2024-09-16T01:35:10.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.549+0000 7fc170a4c640 1 -- 172.21.15.88:0/196891460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc16c0d2660 msgr2=0x7fc16c157ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:10.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.549+0000 7fc170a4c640 1 --2- 172.21.15.88:0/196891460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc16c0d2660 0x7fc16c157ed0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc154011a70 tx=0x7fc154011f30 comp rx=0 tx=0).stop 2024-09-16T01:35:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.549+0000 7fc170a4c640 1 -- 172.21.15.88:0/196891460 shutdown_connections 2024-09-16T01:35:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.549+0000 7fc170a4c640 1 --2- 172.21.15.88:0/196891460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc14c06c2b0 0x7fc14c06e770 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.549+0000 7fc170a4c640 1 --2- 172.21.15.88:0/196891460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc16c0d2ae0 0x7fc16c0cecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.549+0000 7fc170a4c640 1 --2- 172.21.15.88:0/196891460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc16c0d2660 0x7fc16c157ed0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.549+0000 7fc170a4c640 1 --2- 172.21.15.88:0/196891460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc158003a00 0x7fc16c1f49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.549+0000 7fc170a4c640 1 -- 172.21.15.88:0/196891460 >> 172.21.15.88:0/196891460 conn(0x7fc16c0b4240 msgr2=0x7fc16c0cd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.549+0000 7fc170a4c640 1 -- 172.21.15.88:0/196891460 shutdown_connections 2024-09-16T01:35:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10.549+0000 7fc170a4c640 1 -- 172.21.15.88:0/196891460 wait complete. 2024-09-16T01:35:10.561 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-16T01:35:10.561 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:35:10.562 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2024-09-16T01:35:10.562 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T01:35:10.562 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T01:35:10.562 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-16T01:35:10.562 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-16T01:35:10.562 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:35:10.562 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:35:10.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:10.562 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2024-09-16T01:35:10.578 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:35:10.578 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:35:10.578 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:35:10.578 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:35:10.579 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-09-16T01:35:10.579 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:35:10.579 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:35:10.579 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:35:10.579 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2024-09-16T01:35:10.579 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:35:10.579 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:35:10.579 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2024-09-16T01:35:10.579 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:35:10.579 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2024-09-16T01:35:10.579 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:35:10.579 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2024-09-16T01:35:10.579 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:35:10.579 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42713sec preferred_lft 42713sec 2024-09-16T01:35:10.580 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link noprefixroute 2024-09-16T01:35:10.580 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:35:10.580 INFO:teuthology.orchestra.run.smithi088.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:35:10.580 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff 2024-09-16T01:35:10.580 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T01:35:10.580 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:35:10.580 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::e486:6eff:fe81:a1c/64 scope link 2024-09-16T01:35:10.580 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:35:10.580 INFO:teuthology.orchestra.run.smithi088.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:35:10.580 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 12:6f:29:be:95:48 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:35:10.580 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::106f:29ff:febe:9548/64 scope link 2024-09-16T01:35:10.580 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:35:10.580 INFO:teuthology.orchestra.run.smithi088.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:35:10.581 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T01:35:10.581 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::c63:b4ff:feb3:ad9e/64 scope link 2024-09-16T01:35:10.581 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:35:10.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:10.581 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:35:10.642 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-09-16T01:35:10.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:10.644 DEBUG:teuthology.orchestra.run.smithi088:> route 2024-09-16T01:35:10.700 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2024-09-16T01:35:10.700 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:35:10.700 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:35:10.700 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:35:10.700 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:35:10.701 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:35:10.701 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:35:10.701 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:35:10.701 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:35:10.701 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:35:10.701 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:35:10.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:35:10.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:35:10.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:10.826 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:35:10.841 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T01:35:10.841 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T01:35:10.841 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:35:10.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:10.842 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2024-09-16T01:35:10.906 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:35:10.906 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T01:35:10.961 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:35:10.962 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:35:10.962 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:35:11.017 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-16T01:35:11.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:11.017 DEBUG:teuthology.orchestra.run.smithi088:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:35:11.099 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:35:11.100 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:35:11.100 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:35:11.109 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:35:11.109 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:35:11.109 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:35:11.109 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:35:11.109 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:35:11.114 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:35:11.114 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:35:11.114 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:35:11.114 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:35:11.114 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:35:11.114 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790" 2024-09-16T01:35:11.114 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2024-09-16T01:35:11.114 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5ace2c91-46f9-482b-8a89-731fbdc171bf" will pass to kernel 2024-09-16T01:35:11.115 INFO:teuthology.orchestra.run.smithi088.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-16T01:35:11.115 INFO:teuthology.orchestra.run.smithi088.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:35:11.115 INFO:teuthology.orchestra.run.smithi088.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:35:11.115 INFO:teuthology.orchestra.run.smithi088.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:35:11.115 INFO:teuthology.orchestra.run.smithi088.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:35:11.117 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:35:11.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:11.117 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:35:11.183 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:35:11.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:35:11.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:35:11.285 INFO:teuthology.orchestra.run.smithi088.stdout:client4801 2024-09-16T01:35:11.285 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:35:11.285 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:35:11.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:11.302 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir/) 2024-09-16T01:35:11.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:11.362 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch subdir/file_undamaged) 2024-09-16T01:35:11.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:11.419 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch subdir/file_to_be_damaged) 2024-09-16T01:35:11.476 DEBUG:teuthology.orchestra.run.smithi088:> stdin-killer -- python3 -c ' 2024-09-16T01:35:11.476 DEBUG:teuthology.orchestra.run.smithi088:> import os 2024-09-16T01:35:11.476 DEBUG:teuthology.orchestra.run.smithi088:> import stat 2024-09-16T01:35:11.476 DEBUG:teuthology.orchestra.run.smithi088:> 2024-09-16T01:35:11.476 DEBUG:teuthology.orchestra.run.smithi088:> print(os.stat("/home/ubuntu/cephtest/mnt.0/subdir").st_ino) 2024-09-16T01:35:11.476 DEBUG:teuthology.orchestra.run.smithi088:> ' 2024-09-16T01:35:11.580 INFO:teuthology.orchestra.run.smithi088.stdout:1099511627776 2024-09-16T01:35:11.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:35:11.589 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:35:11.589 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:35:11.645 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:35:11.645 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:35:11.700 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:35:11.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.838+0000 7f86a9e74640 1 Processor -- start 2024-09-16T01:35:11.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.838+0000 7f86a9e74640 1 -- start start 2024-09-16T01:35:11.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.839+0000 7f86a9e74640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f86a40731c0 0x7f86a4073600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:11.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.839+0000 7f86a9e74640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f86a415de80 0x7f86a415e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:11.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.839+0000 7f86a9e74640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86a415a430 con 0x7f86a415a050 2024-09-16T01:35:11.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.839+0000 7f86a9e74640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f86a4073b40 con 0x7f86a40731c0 2024-09-16T01:35:11.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.839+0000 7f86a9e74640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f86a40cd0a0 con 0x7f86a415de80 2024-09-16T01:35:11.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.839+0000 7f86a2ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f86a40731c0 0x7f86a4073600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:11.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.839+0000 7f86a2ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f86a40731c0 0x7f86a4073600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33562/0 (socket says 172.21.15.88:33562) 2024-09-16T01:35:11.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.839+0000 7f86a2ffd640 1 -- 172.21.15.88:0/1224107610 learned_addr learned my addr 172.21.15.88:0/1224107610 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:11.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.839+0000 7f86a2ffd640 1 -- 172.21.15.88:0/1224107610 >> v2:172.21.15.154:3300/0 conn(0x7f86a415de80 msgr2=0x7f86a415e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:11.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.839+0000 7f86a2ffd640 1 --2- 172.21.15.88:0/1224107610 >> v2:172.21.15.154:3300/0 conn(0x7f86a415de80 0x7f86a415e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:11.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.839+0000 7f86a2ffd640 1 -- 172.21.15.88:0/1224107610 >> v1:172.21.15.154:6789/0 conn(0x7f86a415a050 legacy=0x7f86a415d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:11.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.839+0000 7f86a2ffd640 1 -- 172.21.15.88:0/1224107610 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a40cd380 con 0x7f86a40731c0 2024-09-16T01:35:11.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.840+0000 7f86a2ffd640 1 --2- 172.21.15.88:0/1224107610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86a40731c0 0x7f86a4073600 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f86a40c6040 tx=0x7f869802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69b59abffcb0b02d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:11.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.840+0000 7f86a27fc640 1 -- 172.21.15.88:0/1224107610 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8698037070 con 0x7f86a40731c0 2024-09-16T01:35:11.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.840+0000 7f86a27fc640 1 -- 172.21.15.88:0/1224107610 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8698032440 con 0x7f86a40731c0 2024-09-16T01:35:11.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.840+0000 7f86a27fc640 1 -- 172.21.15.88:0/1224107610 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8698032810 con 0x7f86a40731c0 2024-09-16T01:35:11.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.841+0000 7f86a9e74640 1 -- 172.21.15.88:0/1224107610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86a40731c0 msgr2=0x7f86a4073600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:11.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.841+0000 7f86a9e74640 1 --2- 172.21.15.88:0/1224107610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86a40731c0 0x7f86a4073600 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f86a40c6040 tx=0x7f869802a1a0 comp rx=0 tx=0).stop 2024-09-16T01:35:11.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.841+0000 7f86a9e74640 1 -- 172.21.15.88:0/1224107610 shutdown_connections 2024-09-16T01:35:11.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.841+0000 7f86a9e74640 1 --2- 172.21.15.88:0/1224107610 >> v2:172.21.15.154:3300/0 conn(0x7f86a415de80 0x7f86a415e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:11.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.841+0000 7f86a9e74640 1 --2- 172.21.15.88:0/1224107610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86a40731c0 0x7f86a4073600 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:11.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.841+0000 7f86a9e74640 1 -- 172.21.15.88:0/1224107610 >> 172.21.15.88:0/1224107610 conn(0x7f86a406d0f0 msgr2=0x7f86a406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:11.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.841+0000 7f86a9e74640 1 -- 172.21.15.88:0/1224107610 shutdown_connections 2024-09-16T01:35:11.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.841+0000 7f86a9e74640 1 -- 172.21.15.88:0/1224107610 wait complete. 2024-09-16T01:35:11.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.842+0000 7f86a9e74640 1 Processor -- start 2024-09-16T01:35:11.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.842+0000 7f86a9e74640 1 -- start start 2024-09-16T01:35:11.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.843+0000 7f86a9e74640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86a40731c0 0x7f86a4147fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:11.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.843+0000 7f86a9e74640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86a415a050 0x7f86a40cbc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:11.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.843+0000 7f86a9e74640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86a415de80 0x7f86a414fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:11.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.843+0000 7f86a9e74640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f86a40cdc20 con 0x7f86a40731c0 2024-09-16T01:35:11.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.843+0000 7f86a9e74640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f86a40cd940 con 0x7f86a415a050 2024-09-16T01:35:11.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.843+0000 7f86a9e74640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f86a40b36e0 con 0x7f86a415de80 2024-09-16T01:35:11.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.843+0000 7f86a37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86a40731c0 0x7f86a4147fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:11.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.843+0000 7f86a37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86a40731c0 0x7f86a4147fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33570/0 (socket says 172.21.15.88:33570) 2024-09-16T01:35:11.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.843+0000 7f86a37fe640 1 -- 172.21.15.88:0/595745191 learned_addr learned my addr 172.21.15.88:0/595745191 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:11.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.843+0000 7f86a37fe640 1 -- 172.21.15.88:0/595745191 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86a415de80 msgr2=0x7f86a414fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:11.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.843+0000 7f86a37fe640 1 --2- 172.21.15.88:0/595745191 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86a415de80 0x7f86a414fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:11.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.843+0000 7f86a37fe640 1 -- 172.21.15.88:0/595745191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86a415a050 msgr2=0x7f86a40cbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:11.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.843+0000 7f86a2ffd640 1 --2- 172.21.15.88:0/595745191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86a415a050 0x7f86a40cbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:11.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.843+0000 7f86a37fe640 1 --2- 172.21.15.88:0/595745191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86a415a050 0x7f86a40cbc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:11.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.844+0000 7f86a37fe640 1 -- 172.21.15.88:0/595745191 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a4153400 con 0x7f86a40731c0 2024-09-16T01:35:11.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.844+0000 7f86a37fe640 1 --2- 172.21.15.88:0/595745191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86a40731c0 0x7f86a4147fa0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f869400c7d0 tx=0x7f869400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:11.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.844+0000 7f868bfff640 1 -- 172.21.15.88:0/595745191 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8694017070 con 0x7f86a40731c0 2024-09-16T01:35:11.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.844+0000 7f868bfff640 1 -- 172.21.15.88:0/595745191 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8694003da0 con 0x7f86a40731c0 2024-09-16T01:35:11.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.844+0000 7f86a9e74640 1 -- 172.21.15.88:0/595745191 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86a41536e0 con 0x7f86a40731c0 2024-09-16T01:35:11.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.844+0000 7f868bfff640 1 -- 172.21.15.88:0/595745191 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86940052b0 con 0x7f86a40731c0 2024-09-16T01:35:11.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.844+0000 7f86a9e74640 1 -- 172.21.15.88:0/595745191 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86a4071130 con 0x7f86a40731c0 2024-09-16T01:35:11.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.845+0000 7f86a9e74640 1 -- 172.21.15.88:0/595745191 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86a40ccd00 con 0x7f86a40731c0 2024-09-16T01:35:11.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.845+0000 7f868bfff640 1 -- 172.21.15.88:0/595745191 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f869401a070 con 0x7f86a40731c0 2024-09-16T01:35:11.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.846+0000 7f868bfff640 1 --2- 172.21.15.88:0/595745191 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f869006c2b0 0x7f869006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:11.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.846+0000 7f868bfff640 1 -- 172.21.15.88:0/595745191 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f869408ab60 con 0x7f86a40731c0 2024-09-16T01:35:11.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.846+0000 7f86a2ffd640 1 --2- 172.21.15.88:0/595745191 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f869006c2b0 0x7f869006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:11.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.846+0000 7f86a2ffd640 1 --2- 172.21.15.88:0/595745191 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f869006c2b0 0x7f869006e770 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8698034000 tx=0x7f869802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:11.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:11.850+0000 7f868bfff640 1 -- 172.21.15.88:0/595745191 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f86940592d0 con 0x7f86a40731c0 2024-09-16T01:35:12.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.026+0000 7f86a9e74640 1 -- 172.21.15.88:0/595745191 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f86a415f180 con 0x7f86a40731c0 2024-09-16T01:35:12.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.026+0000 7f868bfff640 1 -- 172.21.15.88:0/595745191 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v32) ==== 80+0+0 (secure 0 0 0) 0x7f8694058bd0 con 0x7f86a40731c0 2024-09-16T01:35:12.027 INFO:teuthology.orchestra.run.smithi088.stderr:listed 0 entries 2024-09-16T01:35:12.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.028+0000 7f86a9e74640 1 -- 172.21.15.88:0/595745191 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f869006c2b0 msgr2=0x7f869006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:12.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.028+0000 7f86a9e74640 1 --2- 172.21.15.88:0/595745191 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f869006c2b0 0x7f869006e770 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8698034000 tx=0x7f869802e000 comp rx=0 tx=0).stop 2024-09-16T01:35:12.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.028+0000 7f86a9e74640 1 -- 172.21.15.88:0/595745191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86a40731c0 msgr2=0x7f86a4147fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:12.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.028+0000 7f86a9e74640 1 --2- 172.21.15.88:0/595745191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86a40731c0 0x7f86a4147fa0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f869400c7d0 tx=0x7f869400cc90 comp rx=0 tx=0).stop 2024-09-16T01:35:12.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.028+0000 7f86a9e74640 1 -- 172.21.15.88:0/595745191 shutdown_connections 2024-09-16T01:35:12.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.028+0000 7f86a9e74640 1 --2- 172.21.15.88:0/595745191 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f869006c2b0 0x7f869006e770 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.028+0000 7f86a9e74640 1 --2- 172.21.15.88:0/595745191 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86a415de80 0x7f86a414fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.028+0000 7f86a9e74640 1 --2- 172.21.15.88:0/595745191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86a415a050 0x7f86a40cbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.028+0000 7f86a9e74640 1 --2- 172.21.15.88:0/595745191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86a40731c0 0x7f86a4147fa0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.028+0000 7f86a9e74640 1 -- 172.21.15.88:0/595745191 >> 172.21.15.88:0/595745191 conn(0x7f86a406d0f0 msgr2=0x7f86a415d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:12.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.028+0000 7f86a9e74640 1 -- 172.21.15.88:0/595745191 shutdown_connections 2024-09-16T01:35:12.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.028+0000 7f86a9e74640 1 -- 172.21.15.88:0/595745191 wait complete. 2024-09-16T01:35:12.040 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-16T01:35:12.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:12.040 DEBUG:teuthology.orchestra.run.smithi088:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-16T01:35:12.082 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:35:12.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:12.083 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:35:12.139 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:35:12.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.275+0000 7f598b58d640 1 Processor -- start 2024-09-16T01:35:12.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.275+0000 7f598b58d640 1 -- start start 2024-09-16T01:35:12.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.276+0000 7f598b58d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f598415c170 0x7f598415c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.276+0000 7f598b58d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f59840cf6c0 0x7f59840cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.276+0000 7f598b58d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59840b5dd0 0x7f59840b6280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.276+0000 7f598b58d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f598415ca90 con 0x7f598415c170 2024-09-16T01:35:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.276+0000 7f598b58d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f59840b4cf0 con 0x7f59840cf6c0 2024-09-16T01:35:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.276+0000 7f598b58d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f59840b4fd0 con 0x7f59840b5dd0 2024-09-16T01:35:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.276+0000 7f5989329640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f598415c170 0x7f598415c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.276+0000 7f5989329640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f598415c170 0x7f598415c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33578/0 (socket says 172.21.15.88:33578) 2024-09-16T01:35:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.276+0000 7f5989329640 1 -- 172.21.15.88:0/3289525020 learned_addr learned my addr 172.21.15.88:0/3289525020 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.276+0000 7f5989329640 1 -- 172.21.15.88:0/3289525020 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59840b5dd0 msgr2=0x7f59840b6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.276+0000 7f5989b2a640 1 --2- 172.21.15.88:0/3289525020 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59840b5dd0 0x7f59840b6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.276+0000 7f5988b28640 1 --2- 172.21.15.88:0/3289525020 >> v2:172.21.15.154:3300/0 conn(0x7f59840cf6c0 0x7f59840cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.277+0000 7f5989329640 1 --2- 172.21.15.88:0/3289525020 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59840b5dd0 0x7f59840b6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.277+0000 7f5989329640 1 -- 172.21.15.88:0/3289525020 >> v2:172.21.15.154:3300/0 conn(0x7f59840cf6c0 msgr2=0x7f59840cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:12.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.277+0000 7f5989329640 1 --2- 172.21.15.88:0/3289525020 >> v2:172.21.15.154:3300/0 conn(0x7f59840cf6c0 0x7f59840cfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.277+0000 7f5989329640 1 -- 172.21.15.88:0/3289525020 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59840b52b0 con 0x7f598415c170 2024-09-16T01:35:12.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.277+0000 7f5989b2a640 1 --2- 172.21.15.88:0/3289525020 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59840b5dd0 0x7f59840b6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:12.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.277+0000 7f5989329640 1 --2- 172.21.15.88:0/3289525020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f598415c170 0x7f598415c550 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5980009e50 tx=0x7f598002a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19ddf77e0fcf609b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:12.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.277+0000 7f597bfff640 1 -- 172.21.15.88:0/3289525020 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5980036070 con 0x7f598415c170 2024-09-16T01:35:12.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.277+0000 7f597bfff640 1 -- 172.21.15.88:0/3289525020 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5980031600 con 0x7f598415c170 2024-09-16T01:35:12.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.277+0000 7f597bfff640 1 -- 172.21.15.88:0/3289525020 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59800319d0 con 0x7f598415c170 2024-09-16T01:35:12.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.278+0000 7f598b58d640 1 -- 172.21.15.88:0/3289525020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f598415c170 msgr2=0x7f598415c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:12.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.278+0000 7f598b58d640 1 --2- 172.21.15.88:0/3289525020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f598415c170 0x7f598415c550 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5980009e50 tx=0x7f598002a6c0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.278+0000 7f598b58d640 1 -- 172.21.15.88:0/3289525020 shutdown_connections 2024-09-16T01:35:12.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.278+0000 7f598b58d640 1 --2- 172.21.15.88:0/3289525020 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59840b5dd0 0x7f59840b6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.278+0000 7f598b58d640 1 --2- 172.21.15.88:0/3289525020 >> v2:172.21.15.154:3300/0 conn(0x7f59840cf6c0 0x7f59840cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.278+0000 7f598b58d640 1 --2- 172.21.15.88:0/3289525020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f598415c170 0x7f598415c550 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.278+0000 7f598b58d640 1 -- 172.21.15.88:0/3289525020 >> 172.21.15.88:0/3289525020 conn(0x7f598406bc10 msgr2=0x7f598406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:12.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.278+0000 7f598b58d640 1 -- 172.21.15.88:0/3289525020 shutdown_connections 2024-09-16T01:35:12.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.278+0000 7f598b58d640 1 -- 172.21.15.88:0/3289525020 wait complete. 2024-09-16T01:35:12.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.279+0000 7f598b58d640 1 Processor -- start 2024-09-16T01:35:12.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.279+0000 7f598b58d640 1 -- start start 2024-09-16T01:35:12.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.280+0000 7f598b58d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59840b5dd0 0x7f59841502d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:12.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.280+0000 7f598b58d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59840cf6c0 0x7f5984150860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:12.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.280+0000 7f598b58d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f598415c170 0x7f59840b1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:12.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.280+0000 7f598b58d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f59840b5b50 con 0x7f59840cf6c0 2024-09-16T01:35:12.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.280+0000 7f598b58d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f59840b5870 con 0x7f59840b5dd0 2024-09-16T01:35:12.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.280+0000 7f598b58d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f598415ffb0 con 0x7f598415c170 2024-09-16T01:35:12.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.280+0000 7f5988b28640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59840cf6c0 0x7f5984150860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:12.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.280+0000 7f5988b28640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59840cf6c0 0x7f5984150860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33588/0 (socket says 172.21.15.88:33588) 2024-09-16T01:35:12.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.280+0000 7f5988b28640 1 -- 172.21.15.88:0/679648221 learned_addr learned my addr 172.21.15.88:0/679648221 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:12.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.280+0000 7f5989b2a640 1 --2- 172.21.15.88:0/679648221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f598415c170 0x7f59840b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:12.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.280+0000 7f5989329640 1 --2- 172.21.15.88:0/679648221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59840b5dd0 0x7f59841502d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:12.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.280+0000 7f5988b28640 1 -- 172.21.15.88:0/679648221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f598415c170 msgr2=0x7f59840b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:12.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.280+0000 7f5988b28640 1 --2- 172.21.15.88:0/679648221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f598415c170 0x7f59840b1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.280+0000 7f5988b28640 1 -- 172.21.15.88:0/679648221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59840b5dd0 msgr2=0x7f59841502d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:12.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.280+0000 7f5988b28640 1 --2- 172.21.15.88:0/679648221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59840b5dd0 0x7f59841502d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.280+0000 7f5988b28640 1 -- 172.21.15.88:0/679648221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59840afe80 con 0x7f59840cf6c0 2024-09-16T01:35:12.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.280+0000 7f5989b2a640 1 --2- 172.21.15.88:0/679648221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f598415c170 0x7f59840b1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:12.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.281+0000 7f5988b28640 1 --2- 172.21.15.88:0/679648221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59840cf6c0 0x7f5984150860 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f597400c820 tx=0x7f597400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:12.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.281+0000 7f5979ffb640 1 -- 172.21.15.88:0/679648221 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5974015070 con 0x7f59840cf6c0 2024-09-16T01:35:12.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.281+0000 7f5979ffb640 1 -- 172.21.15.88:0/679648221 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5974004240 con 0x7f59840cf6c0 2024-09-16T01:35:12.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.281+0000 7f598b58d640 1 -- 172.21.15.88:0/679648221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59840b0120 con 0x7f59840cf6c0 2024-09-16T01:35:12.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.281+0000 7f5979ffb640 1 -- 172.21.15.88:0/679648221 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f597400ce10 con 0x7f59840cf6c0 2024-09-16T01:35:12.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.281+0000 7f598b58d640 1 -- 172.21.15.88:0/679648221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59840b06e0 con 0x7f59840cf6c0 2024-09-16T01:35:12.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.282+0000 7f598b58d640 1 -- 172.21.15.88:0/679648221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f598415c550 con 0x7f59840cf6c0 2024-09-16T01:35:12.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.282+0000 7f5979ffb640 1 -- 172.21.15.88:0/679648221 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5974007830 con 0x7f59840cf6c0 2024-09-16T01:35:12.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.283+0000 7f5979ffb640 1 --2- 172.21.15.88:0/679648221 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f596806c2b0 0x7f596806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:12.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.283+0000 7f5989329640 1 --2- 172.21.15.88:0/679648221 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f596806c2b0 0x7f596806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:12.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.283+0000 7f5979ffb640 1 -- 172.21.15.88:0/679648221 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f597408b290 con 0x7f59840cf6c0 2024-09-16T01:35:12.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.283+0000 7f5989329640 1 --2- 172.21.15.88:0/679648221 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f596806c2b0 0x7f596806e770 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5980033000 tx=0x7f5980009000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:12.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.287+0000 7f5979ffb640 1 -- 172.21.15.88:0/679648221 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5974059a90 con 0x7f59840cf6c0 2024-09-16T01:35:12.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.487+0000 7f598b58d640 1 -- 172.21.15.88:0/679648221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59840b4810 con 0x7f59840cf6c0 2024-09-16T01:35:12.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.487+0000 7f5979ffb640 1 -- 172.21.15.88:0/679648221 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+5229 (secure 0 0 0) 0x7f5974059390 con 0x7f59840cf6c0 2024-09-16T01:35:12.489 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:35:12.489 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":13,"btime":"2024-09-16T01:35:03:353018+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1005331674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1005331674},{"type":"v1","addr":"172.21.15.88:6837","nonce":1005331674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/3427247662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3427247662},{"type":"v1","addr":"172.21.15.154:6834","nonce":3427247662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/4059835337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":4059835337},{"type":"v1","addr":"172.21.15.154:6835","nonce":4059835337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:35:00.674725+0000","modified":"2024-09-16T01:35:02.681832+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4830},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4830":{"gid":4830,"name":"c","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6835/4230443772","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4230443772},{"type":"v1","addr":"172.21.15.88:6835","nonce":4230443772}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4830,"qdb_cluster":[4830]},"id":2}]} 2024-09-16T01:35:12.489 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 13 2024-09-16T01:35:12.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.489+0000 7f598b58d640 1 -- 172.21.15.88:0/679648221 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f596806c2b0 msgr2=0x7f596806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:12.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.489+0000 7f598b58d640 1 --2- 172.21.15.88:0/679648221 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f596806c2b0 0x7f596806e770 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5980033000 tx=0x7f5980009000 comp rx=0 tx=0).stop 2024-09-16T01:35:12.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.489+0000 7f598b58d640 1 -- 172.21.15.88:0/679648221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59840cf6c0 msgr2=0x7f5984150860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:12.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.489+0000 7f598b58d640 1 --2- 172.21.15.88:0/679648221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59840cf6c0 0x7f5984150860 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f597400c820 tx=0x7f597400cce0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.489+0000 7f598b58d640 1 -- 172.21.15.88:0/679648221 shutdown_connections 2024-09-16T01:35:12.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.489+0000 7f598b58d640 1 --2- 172.21.15.88:0/679648221 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f596806c2b0 0x7f596806e770 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.489+0000 7f598b58d640 1 --2- 172.21.15.88:0/679648221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f598415c170 0x7f59840b1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.489+0000 7f598b58d640 1 --2- 172.21.15.88:0/679648221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59840cf6c0 0x7f5984150860 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.489+0000 7f598b58d640 1 --2- 172.21.15.88:0/679648221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59840b5dd0 0x7f59841502d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.489+0000 7f598b58d640 1 -- 172.21.15.88:0/679648221 >> 172.21.15.88:0/679648221 conn(0x7f598406bc10 msgr2=0x7f598415f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:12.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.489+0000 7f598b58d640 1 -- 172.21.15.88:0/679648221 shutdown_connections 2024-09-16T01:35:12.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.489+0000 7f598b58d640 1 -- 172.21.15.88:0/679648221 wait complete. 2024-09-16T01:35:12.502 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-09-16T01:35:12.625 INFO:teuthology.orchestra.run.smithi088.stdout:{"message":"","return_code":0} 2024-09-16T01:35:12.625 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-16T01:35:12.626 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:35:12.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.724+0000 7fbead1cf640 1 Processor -- start 2024-09-16T01:35:12.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.724+0000 7fbead1cf640 1 -- start start 2024-09-16T01:35:12.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.725+0000 7fbead1cf640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbea815e6d0 0x7fbea8073c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:12.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.725+0000 7fbead1cf640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbea80cf520 0x7fbea80d1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:12.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.725+0000 7fbead1cf640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbea806dba0 con 0x7fbea806d6c0 2024-09-16T01:35:12.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.725+0000 7fbead1cf640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fbea80d1070 con 0x7fbea815e6d0 2024-09-16T01:35:12.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.725+0000 7fbead1cf640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbea80d1350 con 0x7fbea80cf520 2024-09-16T01:35:12.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.725+0000 7fbea6d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbea815e6d0 0x7fbea8073c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:12.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.725+0000 7fbea6d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbea815e6d0 0x7fbea8073c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33602/0 (socket says 172.21.15.88:33602) 2024-09-16T01:35:12.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.725+0000 7fbea6d9d640 1 -- 172.21.15.88:0/3880438567 learned_addr learned my addr 172.21.15.88:0/3880438567 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:12.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.725+0000 7fbea6d9d640 1 -- 172.21.15.88:0/3880438567 >> v2:172.21.15.154:3300/0 conn(0x7fbea80cf520 msgr2=0x7fbea80d1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:12.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.725+0000 7fbea6d9d640 1 --2- 172.21.15.88:0/3880438567 >> v2:172.21.15.154:3300/0 conn(0x7fbea80cf520 0x7fbea80d1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.725+0000 7fbea6d9d640 1 -- 172.21.15.88:0/3880438567 >> v1:172.21.15.154:6789/0 conn(0x7fbea806d6c0 legacy=0x7fbea80cee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:12.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.725+0000 7fbea6d9d640 1 -- 172.21.15.88:0/3880438567 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbea815eab0 con 0x7fbea815e6d0 2024-09-16T01:35:12.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.726+0000 7fbea6d9d640 1 --2- 172.21.15.88:0/3880438567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea815e6d0 0x7fbea8073c00 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbea80d1630 tx=0x7fbe9802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c892f3912d3890d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:12.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.726+0000 7fbea5d9b640 1 -- 172.21.15.88:0/3880438567 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe98036070 con 0x7fbea815e6d0 2024-09-16T01:35:12.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.726+0000 7fbea5d9b640 1 -- 172.21.15.88:0/3880438567 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe98031600 con 0x7fbea815e6d0 2024-09-16T01:35:12.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.726+0000 7fbea5d9b640 1 -- 172.21.15.88:0/3880438567 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe980319d0 con 0x7fbea815e6d0 2024-09-16T01:35:12.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.727+0000 7fbead1cf640 1 -- 172.21.15.88:0/3880438567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea815e6d0 msgr2=0x7fbea8073c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:12.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.727+0000 7fbead1cf640 1 --2- 172.21.15.88:0/3880438567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea815e6d0 0x7fbea8073c00 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fbea80d1630 tx=0x7fbe9802a660 comp rx=0 tx=0).stop 2024-09-16T01:35:12.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.727+0000 7fbead1cf640 1 -- 172.21.15.88:0/3880438567 shutdown_connections 2024-09-16T01:35:12.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.727+0000 7fbead1cf640 1 --2- 172.21.15.88:0/3880438567 >> v2:172.21.15.154:3300/0 conn(0x7fbea80cf520 0x7fbea80d1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.727+0000 7fbead1cf640 1 --2- 172.21.15.88:0/3880438567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea815e6d0 0x7fbea8073c00 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.727+0000 7fbead1cf640 1 -- 172.21.15.88:0/3880438567 >> 172.21.15.88:0/3880438567 conn(0x7fbea80b7d70 msgr2=0x7fbea8062630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:12.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.727+0000 7fbead1cf640 1 -- 172.21.15.88:0/3880438567 shutdown_connections 2024-09-16T01:35:12.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.727+0000 7fbead1cf640 1 -- 172.21.15.88:0/3880438567 wait complete. 2024-09-16T01:35:12.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.728+0000 7fbead1cf640 1 Processor -- start 2024-09-16T01:35:12.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.728+0000 7fbead1cf640 1 -- start start 2024-09-16T01:35:12.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.729+0000 7fbead1cf640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbea806d6c0 0x7fbea8156430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:12.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.729+0000 7fbead1cf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbea80cf520 0x7fbea815ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:12.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.729+0000 7fbead1cf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea815e6d0 0x7fbea80b6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:12.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.729+0000 7fbead1cf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbea80d2410 con 0x7fbea815e6d0 2024-09-16T01:35:12.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.729+0000 7fbead1cf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbea80d2290 con 0x7fbea80cf520 2024-09-16T01:35:12.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.729+0000 7fbead1cf640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbea80d2590 con 0x7fbea806d6c0 2024-09-16T01:35:12.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.729+0000 7fbea759e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea815e6d0 0x7fbea80b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:12.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.729+0000 7fbea759e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea815e6d0 0x7fbea80b6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33618/0 (socket says 172.21.15.88:33618) 2024-09-16T01:35:12.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.729+0000 7fbea759e640 1 -- 172.21.15.88:0/3406731286 learned_addr learned my addr 172.21.15.88:0/3406731286 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:12.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.729+0000 7fbea6d9d640 1 --2- 172.21.15.88:0/3406731286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbea806d6c0 0x7fbea8156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:12.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.729+0000 7fbea659c640 1 --2- 172.21.15.88:0/3406731286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbea80cf520 0x7fbea815ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:12.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.730+0000 7fbea759e640 1 -- 172.21.15.88:0/3406731286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbea806d6c0 msgr2=0x7fbea8156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:12.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.730+0000 7fbea759e640 1 --2- 172.21.15.88:0/3406731286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbea806d6c0 0x7fbea8156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.730+0000 7fbea759e640 1 -- 172.21.15.88:0/3406731286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbea80cf520 msgr2=0x7fbea815ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:12.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.730+0000 7fbea759e640 1 --2- 172.21.15.88:0/3406731286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbea80cf520 0x7fbea815ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.730+0000 7fbea759e640 1 -- 172.21.15.88:0/3406731286 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbea8156970 con 0x7fbea815e6d0 2024-09-16T01:35:12.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.730+0000 7fbea659c640 1 --2- 172.21.15.88:0/3406731286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbea80cf520 0x7fbea815ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:12.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.730+0000 7fbea759e640 1 --2- 172.21.15.88:0/3406731286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea815e6d0 0x7fbea80b6200 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fbea000b450 tx=0x7fbea000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:12.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.730+0000 7fbe8f7fe640 1 -- 172.21.15.88:0/3406731286 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea0015020 con 0x7fbea815e6d0 2024-09-16T01:35:12.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.730+0000 7fbe8f7fe640 1 -- 172.21.15.88:0/3406731286 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbea0011890 con 0x7fbea815e6d0 2024-09-16T01:35:12.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.730+0000 7fbead1cf640 1 -- 172.21.15.88:0/3406731286 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbea815f6c0 con 0x7fbea815e6d0 2024-09-16T01:35:12.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.730+0000 7fbe8f7fe640 1 -- 172.21.15.88:0/3406731286 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea0011c60 con 0x7fbea815e6d0 2024-09-16T01:35:12.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.730+0000 7fbead1cf640 1 -- 172.21.15.88:0/3406731286 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbea815fc80 con 0x7fbea815e6d0 2024-09-16T01:35:12.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.731+0000 7fbead1cf640 1 -- 172.21.15.88:0/3406731286 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbea806c740 con 0x7fbea815e6d0 2024-09-16T01:35:12.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.731+0000 7fbe8f7fe640 1 -- 172.21.15.88:0/3406731286 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fbea0018020 con 0x7fbea815e6d0 2024-09-16T01:35:12.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.732+0000 7fbe8f7fe640 1 --2- 172.21.15.88:0/3406731286 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbe9406c2e0 0x7fbe9406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:12.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.732+0000 7fbea6d9d640 1 --2- 172.21.15.88:0/3406731286 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbe9406c2e0 0x7fbe9406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:12.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.732+0000 7fbe8f7fe640 1 -- 172.21.15.88:0/3406731286 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fbea008b410 con 0x7fbea815e6d0 2024-09-16T01:35:12.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.732+0000 7fbea6d9d640 1 --2- 172.21.15.88:0/3406731286 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbe9406c2e0 0x7fbe9406e7a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbe98033000 tx=0x7fbe98009000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:12.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.736+0000 7fbe8f7fe640 1 -- 172.21.15.88:0/3406731286 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fbea001f020 con 0x7fbea815e6d0 2024-09-16T01:35:12.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.936+0000 7fbead1cf640 1 -- 172.21.15.88:0/3406731286 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fbea806e490 con 0x7fbea815e6d0 2024-09-16T01:35:12.938 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:35:12.937+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:35:12.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.938+0000 7fbe8f7fe640 1 -- 172.21.15.88:0/3406731286 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v14) ==== 161+0+0 (secure 0 0 0) 0x7fbea000d040 con 0x7fbea815e6d0 2024-09-16T01:35:12.939 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:35:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.940+0000 7fbead1cf640 1 -- 172.21.15.88:0/3406731286 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbe9406c2e0 msgr2=0x7fbe9406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.940+0000 7fbead1cf640 1 --2- 172.21.15.88:0/3406731286 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbe9406c2e0 0x7fbe9406e7a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbe98033000 tx=0x7fbe98009000 comp rx=0 tx=0).stop 2024-09-16T01:35:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.940+0000 7fbead1cf640 1 -- 172.21.15.88:0/3406731286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea815e6d0 msgr2=0x7fbea80b6200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.940+0000 7fbead1cf640 1 --2- 172.21.15.88:0/3406731286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea815e6d0 0x7fbea80b6200 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fbea000b450 tx=0x7fbea000b910 comp rx=0 tx=0).stop 2024-09-16T01:35:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.940+0000 7fbead1cf640 1 -- 172.21.15.88:0/3406731286 shutdown_connections 2024-09-16T01:35:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.940+0000 7fbead1cf640 1 --2- 172.21.15.88:0/3406731286 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbe9406c2e0 0x7fbe9406e7a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.940+0000 7fbead1cf640 1 --2- 172.21.15.88:0/3406731286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea815e6d0 0x7fbea80b6200 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.940+0000 7fbead1cf640 1 --2- 172.21.15.88:0/3406731286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbea80cf520 0x7fbea815ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.940+0000 7fbead1cf640 1 --2- 172.21.15.88:0/3406731286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbea806d6c0 0x7fbea8156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.940+0000 7fbead1cf640 1 -- 172.21.15.88:0/3406731286 >> 172.21.15.88:0/3406731286 conn(0x7fbea80b7d70 msgr2=0x7fbea80b5460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.940+0000 7fbead1cf640 1 -- 172.21.15.88:0/3406731286 shutdown_connections 2024-09-16T01:35:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.940+0000 7fbead1cf640 1 -- 172.21.15.88:0/3406731286 wait complete. 2024-09-16T01:35:12.944 INFO:tasks.ceph.mds.c.smithi088.stderr: -4591> 2024-09-16T01:35:05.312+0000 7f176e854640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:35:12.953 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 10000000000.00000000 file_to_be_damaged_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-09-16T01:35:12.975 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:35:12.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.997+0000 7f56fe80df40 1 Processor -- start 2024-09-16T01:35:12.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.998+0000 7f56fe80df40 1 -- start start 2024-09-16T01:35:13.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.998+0000 7f56fe80df40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559e3103e480 0x559e310db1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:13.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.998+0000 7f56fe80df40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e3105be60 0x559e310e2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:13.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.998+0000 7f56fe80df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559e310e7f00 0x559e310e2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:13.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.998+0000 7f56fe80df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x559e3105c270 con 0x559e310e7f00 2024-09-16T01:35:13.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.998+0000 7f56fe80df40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559e30f64510 con 0x559e3105be60 2024-09-16T01:35:13.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.998+0000 7f56fe80df40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x559e310e8970 con 0x559e3103e480 2024-09-16T01:35:13.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.998+0000 7f56fbe96640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559e310e7f00 0x559e310e2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:13.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.998+0000 7f56fbe96640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559e310e7f00 0x559e310e2ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33668/0 (socket says 172.21.15.88:33668) 2024-09-16T01:35:13.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.998+0000 7f56fbe96640 1 -- 172.21.15.88:0/2782667163 learned_addr learned my addr 172.21.15.88:0/2782667163 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:13.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.998+0000 7f56fb695640 1 --2- 172.21.15.88:0/2782667163 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559e3103e480 0x559e310db1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:13.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.998+0000 7f56f2e94640 1 --2- 172.21.15.88:0/2782667163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e3105be60 0x559e310e2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:13.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.999+0000 7f56fbe96640 1 -- 172.21.15.88:0/2782667163 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559e3103e480 msgr2=0x559e310db1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:13.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.999+0000 7f56fbe96640 1 --2- 172.21.15.88:0/2782667163 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559e3103e480 0x559e310db1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:13.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.999+0000 7f56fbe96640 1 -- 172.21.15.88:0/2782667163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e3105be60 msgr2=0x559e310e2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:13.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.999+0000 7f56fbe96640 1 --2- 172.21.15.88:0/2782667163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e3105be60 0x559e310e2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:13.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.999+0000 7f56fbe96640 1 -- 172.21.15.88:0/2782667163 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e310e3c90 con 0x559e310e7f00 2024-09-16T01:35:13.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.999+0000 7f56fbe96640 1 --2- 172.21.15.88:0/2782667163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559e310e7f00 0x559e310e2ad0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f56f4001660 tx=0x7f56f4001b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b570835585bd7e84 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:13.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.999+0000 7f56fa693640 1 -- 172.21.15.88:0/2782667163 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f4003650 con 0x559e310e7f00 2024-09-16T01:35:13.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.999+0000 7f56fa693640 1 -- 172.21.15.88:0/2782667163 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56f4003830 con 0x559e310e7f00 2024-09-16T01:35:13.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.999+0000 7f56fa693640 1 -- 172.21.15.88:0/2782667163 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f4003ca0 con 0x559e310e7f00 2024-09-16T01:35:13.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.999+0000 7f56fe80df40 1 -- 172.21.15.88:0/2782667163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559e310e7f00 msgr2=0x559e310e2ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:13.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.999+0000 7f56fe80df40 1 --2- 172.21.15.88:0/2782667163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559e310e7f00 0x559e310e2ad0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f56f4001660 tx=0x7f56f4001b20 comp rx=0 tx=0).stop 2024-09-16T01:35:13.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.999+0000 7f56fe80df40 1 -- 172.21.15.88:0/2782667163 shutdown_connections 2024-09-16T01:35:13.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.999+0000 7f56fe80df40 1 --2- 172.21.15.88:0/2782667163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559e310e7f00 0x559e310e2ad0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:13.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.999+0000 7f56fe80df40 1 --2- 172.21.15.88:0/2782667163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e3105be60 0x559e310e2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:13.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.999+0000 7f56fe80df40 1 --2- 172.21.15.88:0/2782667163 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559e3103e480 0x559e310db1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:13.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.999+0000 7f56fe80df40 1 -- 172.21.15.88:0/2782667163 >> 172.21.15.88:0/2782667163 conn(0x559e3103d960 msgr2=0x559e31041720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:13.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.999+0000 7f56fe80df40 1 -- 172.21.15.88:0/2782667163 shutdown_connections 2024-09-16T01:35:13.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:12.999+0000 7f56fe80df40 1 -- 172.21.15.88:0/2782667163 wait complete. 2024-09-16T01:35:13.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.000+0000 7f56fe80df40 1 Processor -- start 2024-09-16T01:35:13.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.000+0000 7f56fe80df40 1 -- start start 2024-09-16T01:35:13.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.001+0000 7f56fe80df40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e3103e480 0x559e310d6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:13.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.001+0000 7f56fe80df40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559e3105be60 0x559e310138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:13.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.001+0000 7f56fe80df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559e310e7f00 0x559e31012c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:13.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.001+0000 7f56fe80df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x559e31015600 con 0x559e310e7f00 2024-09-16T01:35:13.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.001+0000 7f56fe80df40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559e310e7690 con 0x559e3103e480 2024-09-16T01:35:13.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.001+0000 7f56fe80df40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x559e310158e0 con 0x559e3105be60 2024-09-16T01:35:13.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.001+0000 7f56fbe96640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559e310e7f00 0x559e31012c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:13.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.001+0000 7f56fbe96640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559e310e7f00 0x559e31012c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33682/0 (socket says 172.21.15.88:33682) 2024-09-16T01:35:13.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.001+0000 7f56fbe96640 1 -- 172.21.15.88:0/510527208 learned_addr learned my addr 172.21.15.88:0/510527208 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:13.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.001+0000 7f56f2e94640 1 --2- 172.21.15.88:0/510527208 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559e3105be60 0x559e310138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:13.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.001+0000 7f56fb695640 1 --2- 172.21.15.88:0/510527208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e3103e480 0x559e310d6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:13.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.001+0000 7f56fbe96640 1 -- 172.21.15.88:0/510527208 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559e3105be60 msgr2=0x559e310138b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:13.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.001+0000 7f56fbe96640 1 --2- 172.21.15.88:0/510527208 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559e3105be60 0x559e310138b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:13.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.001+0000 7f56fbe96640 1 -- 172.21.15.88:0/510527208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e3103e480 msgr2=0x559e310d6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:13.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.001+0000 7f56fbe96640 1 --2- 172.21.15.88:0/510527208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559e3103e480 0x559e310d6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:13.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.001+0000 7f56fbe96640 1 -- 172.21.15.88:0/510527208 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e310e7350 con 0x559e310e7f00 2024-09-16T01:35:13.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.001+0000 7f56fbe96640 1 --2- 172.21.15.88:0/510527208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559e310e7f00 0x559e31012c30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x559e310e99d0 tx=0x7f56f4001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:13.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.001+0000 7f56f9691640 1 -- 172.21.15.88:0/510527208 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f4013620 con 0x559e310e7f00 2024-09-16T01:35:13.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.001+0000 7f56f9691640 1 -- 172.21.15.88:0/510527208 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56f400a040 con 0x559e310e7f00 2024-09-16T01:35:13.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.001+0000 7f56fe80df40 1 -- 172.21.15.88:0/510527208 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559e310d9090 con 0x559e310e7f00 2024-09-16T01:35:13.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.001+0000 7f56f9691640 1 -- 172.21.15.88:0/510527208 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f4013a00 con 0x559e310e7f00 2024-09-16T01:35:13.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.002+0000 7f56fe80df40 1 -- 172.21.15.88:0/510527208 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e310d9650 con 0x559e310e7f00 2024-09-16T01:35:13.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.003+0000 7f56f9691640 1 -- 172.21.15.88:0/510527208 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f56f4026060 con 0x559e310e7f00 2024-09-16T01:35:13.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.003+0000 7f56f9691640 1 --2- 172.21.15.88:0/510527208 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f56dc06ac90 0x7f56dc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:13.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.003+0000 7f56f9691640 1 -- 172.21.15.88:0/510527208 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f56f408b580 con 0x559e310e7f00 2024-09-16T01:35:13.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.003+0000 7f56fb695640 1 --2- 172.21.15.88:0/510527208 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f56dc06ac90 0x7f56dc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:13.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.003+0000 7f56fe80df40 1 --2- 172.21.15.88:0/510527208 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x559e31171870 0x559e310da360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:13.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.003+0000 7f56fe80df40 1 -- 172.21.15.88:0/510527208 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33) -- 0x559e310e6ee0 con 0x559e31171870 2024-09-16T01:35:13.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.004+0000 7f56f2e94640 1 --2- 172.21.15.88:0/510527208 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x559e31171870 0x559e310da360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:13.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.004+0000 7f56fb695640 1 --2- 172.21.15.88:0/510527208 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f56dc06ac90 0x7f56dc06d150 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f56ec057000 tx=0x7f56ec052000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:13.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.004+0000 7f56f2e94640 1 --2- 172.21.15.88:0/510527208 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x559e31171870 0x559e310da360 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:13.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.007+0000 7f56f2e94640 1 -- 172.21.15.88:0/510527208 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-vals] v33'11 uv11 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f56e4009970 con 0x559e31171870 2024-09-16T01:35:13.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.007+0000 7f56fe80df40 1 -- 172.21.15.88:0/510527208 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x559e31171870 msgr2=0x559e310da360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:13.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.007+0000 7f56fe80df40 1 --2- 172.21.15.88:0/510527208 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x559e31171870 0x559e310da360 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:13.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.007+0000 7f56fe80df40 1 -- 172.21.15.88:0/510527208 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f56dc06ac90 msgr2=0x7f56dc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:13.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.007+0000 7f56fe80df40 1 --2- 172.21.15.88:0/510527208 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f56dc06ac90 0x7f56dc06d150 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f56ec057000 tx=0x7f56ec052000 comp rx=0 tx=0).stop 2024-09-16T01:35:13.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.007+0000 7f56fe80df40 1 -- 172.21.15.88:0/510527208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559e310e7f00 msgr2=0x559e31012c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:13.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.007+0000 7f56fe80df40 1 --2- 172.21.15.88:0/510527208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559e310e7f00 0x559e31012c30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x559e310e99d0 tx=0x7f56f4001250 comp rx=0 tx=0).stop 2024-09-16T01:35:13.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.007+0000 7f56fbe96640 1 -- 172.21.15.88:0/510527208 reap_dead start 2024-09-16T01:35:13.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.007+0000 7f56fe80df40 1 -- 172.21.15.88:0/510527208 shutdown_connections 2024-09-16T01:35:13.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.007+0000 7f56fe80df40 1 -- 172.21.15.88:0/510527208 >> 172.21.15.88:0/510527208 conn(0x559e3103d960 msgr2=0x559e31014430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:13.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.007+0000 7f56fe80df40 1 -- 172.21.15.88:0/510527208 shutdown_connections 2024-09-16T01:35:13.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.007+0000 7f56fe80df40 1 -- 172.21.15.88:0/510527208 wait complete. 2024-09-16T01:35:13.012 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:35:13.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.153+0000 7f4ace80e640 1 Processor -- start 2024-09-16T01:35:13.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.154+0000 7f4ace80e640 1 -- start start 2024-09-16T01:35:13.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.154+0000 7f4ace80e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4ac815e7a0 0x7f4ac806caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:13.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.154+0000 7f4ace80e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ac80cf1f0 0x7f4ac80b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:13.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.154+0000 7f4ace80e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ac80cf860 con 0x7f4ac80b6720 2024-09-16T01:35:13.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.154+0000 7f4ace80e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f4ac80d2ca0 con 0x7f4ac815e7a0 2024-09-16T01:35:13.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.154+0000 7f4ace80e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4ac80cfb40 con 0x7f4ac80cf1f0 2024-09-16T01:35:13.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.155+0000 7f4ac7fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4ac815e7a0 0x7f4ac806caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:13.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.155+0000 7f4ac77fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ac80cf1f0 0x7f4ac80b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:13.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.155+0000 7f4ac7fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4ac815e7a0 0x7f4ac806caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:46738/0 (socket says 172.21.15.88:46738) 2024-09-16T01:35:13.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.155+0000 7f4ac77fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ac80cf1f0 0x7f4ac80b61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:47914/0 (socket says 172.21.15.88:47914) 2024-09-16T01:35:13.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.155+0000 7f4accdab640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f4ac80b6720 0x7f4ac81623c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:57584/0 (socket says 172.21.15.88:57584) 2024-09-16T01:35:13.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.155+0000 7f4ac7fff640 1 -- 172.21.15.88:0/2979870246 learned_addr learned my addr 172.21.15.88:0/2979870246 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:13.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.155+0000 7f4ac7fff640 1 -- 172.21.15.88:0/2979870246 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ac80cf1f0 msgr2=0x7f4ac80b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:13.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.155+0000 7f4ac7fff640 1 --2- 172.21.15.88:0/2979870246 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ac80cf1f0 0x7f4ac80b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:13.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.155+0000 7f4ac7fff640 1 -- 172.21.15.88:0/2979870246 >> v1:172.21.15.154:6789/0 conn(0x7f4ac80b6720 legacy=0x7f4ac81623c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:13.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.155+0000 7f4ac7fff640 1 -- 172.21.15.88:0/2979870246 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ac80cfe20 con 0x7f4ac815e7a0 2024-09-16T01:35:13.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.156+0000 7f4ac7fff640 1 --2- 172.21.15.88:0/2979870246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ac815e7a0 0x7f4ac806caa0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8009870 tx=0x7f4ab802a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7f5929e7706daea server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:13.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.156+0000 7f4ac6ffd640 1 -- 172.21.15.88:0/2979870246 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab8037070 con 0x7f4ac815e7a0 2024-09-16T01:35:13.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.156+0000 7f4ac6ffd640 1 -- 172.21.15.88:0/2979870246 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ab8032440 con 0x7f4ac815e7a0 2024-09-16T01:35:13.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.156+0000 7f4ac6ffd640 1 -- 172.21.15.88:0/2979870246 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab8032810 con 0x7f4ac815e7a0 2024-09-16T01:35:13.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.157+0000 7f4ace80e640 1 -- 172.21.15.88:0/2979870246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ac815e7a0 msgr2=0x7f4ac806caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:13.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.157+0000 7f4ace80e640 1 --2- 172.21.15.88:0/2979870246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ac815e7a0 0x7f4ac806caa0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8009870 tx=0x7f4ab802a1c0 comp rx=0 tx=0).stop 2024-09-16T01:35:13.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.157+0000 7f4ace80e640 1 -- 172.21.15.88:0/2979870246 shutdown_connections 2024-09-16T01:35:13.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.157+0000 7f4ace80e640 1 --2- 172.21.15.88:0/2979870246 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ac80cf1f0 0x7f4ac80b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:13.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.157+0000 7f4ace80e640 1 --2- 172.21.15.88:0/2979870246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ac815e7a0 0x7f4ac806caa0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:13.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.157+0000 7f4ace80e640 1 -- 172.21.15.88:0/2979870246 >> 172.21.15.88:0/2979870246 conn(0x7f4ac80d11e0 msgr2=0x7f4ac80d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:13.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.157+0000 7f4ace80e640 1 -- 172.21.15.88:0/2979870246 shutdown_connections 2024-09-16T01:35:13.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.157+0000 7f4ace80e640 1 -- 172.21.15.88:0/2979870246 wait complete. 2024-09-16T01:35:13.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.158+0000 7f4ace80e640 1 Processor -- start 2024-09-16T01:35:13.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.158+0000 7f4ace80e640 1 -- start start 2024-09-16T01:35:13.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.159+0000 7f4ace80e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ac80b6720 0x7f4ac815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:13.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.159+0000 7f4ace80e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ac80cf1f0 0x7f4ac80caa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:13.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.159+0000 7f4ace80e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ac815e7a0 0x7f4ac8160990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:13.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.159+0000 7f4ace80e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac806cfe0 con 0x7f4ac815e7a0 2024-09-16T01:35:13.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.159+0000 7f4ace80e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac80b7a00 con 0x7f4ac80cf1f0 2024-09-16T01:35:13.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.159+0000 7f4ace80e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4ac80d4420 con 0x7f4ac80b6720 2024-09-16T01:35:13.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.159+0000 7f4accdab640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ac815e7a0 0x7f4ac8160990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:13.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.159+0000 7f4accdab640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ac815e7a0 0x7f4ac8160990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33688/0 (socket says 172.21.15.88:33688) 2024-09-16T01:35:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.159+0000 7f4accdab640 1 -- 172.21.15.88:0/1754520514 learned_addr learned my addr 172.21.15.88:0/1754520514 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.159+0000 7f4ac77fe640 1 --2- 172.21.15.88:0/1754520514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ac80cf1f0 0x7f4ac80caa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.159+0000 7f4ac7fff640 1 --2- 172.21.15.88:0/1754520514 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ac80b6720 0x7f4ac815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.159+0000 7f4accdab640 1 -- 172.21.15.88:0/1754520514 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ac80b6720 msgr2=0x7f4ac815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.159+0000 7f4accdab640 1 --2- 172.21.15.88:0/1754520514 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ac80b6720 0x7f4ac815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.159+0000 7f4accdab640 1 -- 172.21.15.88:0/1754520514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ac80cf1f0 msgr2=0x7f4ac80caa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.159+0000 7f4accdab640 1 --2- 172.21.15.88:0/1754520514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ac80cf1f0 0x7f4ac80caa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.159+0000 7f4accdab640 1 -- 172.21.15.88:0/1754520514 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ac8158100 con 0x7f4ac815e7a0 2024-09-16T01:35:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.159+0000 7f4accdab640 1 --2- 172.21.15.88:0/1754520514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ac815e7a0 0x7f4ac8160990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4ac000c7b0 tx=0x7f4ac000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.159+0000 7f4ac4ff9640 1 -- 172.21.15.88:0/1754520514 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac0017070 con 0x7f4ac815e7a0 2024-09-16T01:35:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.159+0000 7f4ac4ff9640 1 -- 172.21.15.88:0/1754520514 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ac000a640 con 0x7f4ac815e7a0 2024-09-16T01:35:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.159+0000 7f4ac4ff9640 1 -- 172.21.15.88:0/1754520514 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac000a9f0 con 0x7f4ac815e7a0 2024-09-16T01:35:13.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.159+0000 7f4ace80e640 1 -- 172.21.15.88:0/1754520514 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ac815a110 con 0x7f4ac815e7a0 2024-09-16T01:35:13.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.159+0000 7f4ace80e640 1 -- 172.21.15.88:0/1754520514 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ac815a3d0 con 0x7f4ac815e7a0 2024-09-16T01:35:13.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.160+0000 7f4ace80e640 1 -- 172.21.15.88:0/1754520514 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ac815eb80 con 0x7f4ac815e7a0 2024-09-16T01:35:13.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.161+0000 7f4ac4ff9640 1 -- 172.21.15.88:0/1754520514 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4ac001a070 con 0x7f4ac815e7a0 2024-09-16T01:35:13.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.161+0000 7f4ac4ff9640 1 --2- 172.21.15.88:0/1754520514 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4ab406c2b0 0x7f4ab406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:13.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.161+0000 7f4ac4ff9640 1 -- 172.21.15.88:0/1754520514 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f4ac008a980 con 0x7f4ac815e7a0 2024-09-16T01:35:13.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.161+0000 7f4ac7fff640 1 --2- 172.21.15.88:0/1754520514 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4ab406c2b0 0x7f4ab406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:13.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.162+0000 7f4ac7fff640 1 --2- 172.21.15.88:0/1754520514 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4ab406c2b0 0x7f4ab406e770 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8034000 tx=0x7f4ab802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:13.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.166+0000 7f4ac4ff9640 1 -- 172.21.15.88:0/1754520514 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4ac8158100 con 0x7f4ac815e7a0 2024-09-16T01:35:13.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.364+0000 7f4ace80e640 1 -- 172.21.15.88:0/1754520514 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f4ac80d2a80 con 0x7f4ac815e7a0 2024-09-16T01:35:13.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.939+0000 7f4ac4ff9640 1 -- 172.21.15.88:0/1754520514 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v15) ==== 162+0+0 (secure 0 0 0) 0x7f4ac0058a40 con 0x7f4ac815e7a0 2024-09-16T01:35:13.941 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:35:13.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.942+0000 7f4ace80e640 1 -- 172.21.15.88:0/1754520514 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4ab406c2b0 msgr2=0x7f4ab406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:13.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.942+0000 7f4ace80e640 1 --2- 172.21.15.88:0/1754520514 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4ab406c2b0 0x7f4ab406e770 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8034000 tx=0x7f4ab802e000 comp rx=0 tx=0).stop 2024-09-16T01:35:13.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.942+0000 7f4ace80e640 1 -- 172.21.15.88:0/1754520514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ac815e7a0 msgr2=0x7f4ac8160990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:13.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.942+0000 7f4ace80e640 1 --2- 172.21.15.88:0/1754520514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ac815e7a0 0x7f4ac8160990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4ac000c7b0 tx=0x7f4ac000cc70 comp rx=0 tx=0).stop 2024-09-16T01:35:13.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.942+0000 7f4ace80e640 1 -- 172.21.15.88:0/1754520514 shutdown_connections 2024-09-16T01:35:13.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.942+0000 7f4ace80e640 1 --2- 172.21.15.88:0/1754520514 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4ab406c2b0 0x7f4ab406e770 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:13.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.942+0000 7f4ace80e640 1 --2- 172.21.15.88:0/1754520514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ac815e7a0 0x7f4ac8160990 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:13.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.942+0000 7f4ace80e640 1 --2- 172.21.15.88:0/1754520514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ac80cf1f0 0x7f4ac80caa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:13.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.942+0000 7f4ace80e640 1 --2- 172.21.15.88:0/1754520514 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ac80b6720 0x7f4ac815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:13.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.942+0000 7f4ace80e640 1 -- 172.21.15.88:0/1754520514 >> 172.21.15.88:0/1754520514 conn(0x7f4ac80d11e0 msgr2=0x7f4ac81628b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:13.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.942+0000 7f4ace80e640 1 -- 172.21.15.88:0/1754520514 shutdown_connections 2024-09-16T01:35:13.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:13.942+0000 7f4ace80e640 1 -- 172.21.15.88:0/1754520514 wait complete. 2024-09-16T01:35:13.956 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:35:14.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.054+0000 7f077edcf640 1 Processor -- start 2024-09-16T01:35:14.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.054+0000 7f077edcf640 1 -- start start 2024-09-16T01:35:14.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.055+0000 7f077edcf640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f07780d1b40 0x7f07781618a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:14.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.055+0000 7f077edcf640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f07780b7290 0x7f0778161de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:14.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.055+0000 7f077edcf640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07780708d0 con 0x7f07780b6eb0 2024-09-16T01:35:14.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.055+0000 7f077edcf640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f0778070bb0 con 0x7f07780b7290 2024-09-16T01:35:14.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.055+0000 7f077edcf640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0778070e90 con 0x7f07780d1b40 2024-09-16T01:35:14.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.055+0000 7f077d36c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f07780b7290 0x7f0778161de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:14.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.055+0000 7f077cb6b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f07780b6eb0 0x7f077806fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54530/0 (socket says 172.21.15.88:54530) 2024-09-16T01:35:14.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.055+0000 7f077cb6b640 1 -- 172.21.15.88:0/2666107987 learned_addr learned my addr 172.21.15.88:0/2666107987 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:14.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.055+0000 7f077d36c640 1 -- 172.21.15.88:0/2666107987 >> v2:172.21.15.154:3300/0 conn(0x7f07780d1b40 msgr2=0x7f07781618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:14.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.055+0000 7f077d36c640 1 --2- 172.21.15.88:0/2666107987 >> v2:172.21.15.154:3300/0 conn(0x7f07780d1b40 0x7f07781618a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:14.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.055+0000 7f077d36c640 1 -- 172.21.15.88:0/2666107987 >> v1:172.21.15.88:6789/0 conn(0x7f07780b6eb0 legacy=0x7f077806fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:14.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.055+0000 7f077d36c640 1 -- 172.21.15.88:0/2666107987 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0778071170 con 0x7f07780b7290 2024-09-16T01:35:14.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.055+0000 7f076f7fe640 1 -- 172.21.15.88:0/2666107987 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2958614073 0 0) 0x7f07780708d0 con 0x7f07780b6eb0 2024-09-16T01:35:14.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.056+0000 7f077d36c640 1 --2- 172.21.15.88:0/2666107987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07780b7290 0x7f0778161de0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0778072d60 tx=0x7f077002a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af269f8464cdaaa7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:14.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.056+0000 7f076f7fe640 1 -- 172.21.15.88:0/2666107987 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0770007740 con 0x7f07780b7290 2024-09-16T01:35:14.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.056+0000 7f076f7fe640 1 -- 172.21.15.88:0/2666107987 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07700078e0 con 0x7f07780b7290 2024-09-16T01:35:14.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.056+0000 7f076f7fe640 1 -- 172.21.15.88:0/2666107987 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0770007cb0 con 0x7f07780b7290 2024-09-16T01:35:14.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.056+0000 7f077edcf640 1 -- 172.21.15.88:0/2666107987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07780b7290 msgr2=0x7f0778161de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:14.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.056+0000 7f077edcf640 1 --2- 172.21.15.88:0/2666107987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07780b7290 0x7f0778161de0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0778072d60 tx=0x7f077002a560 comp rx=0 tx=0).stop 2024-09-16T01:35:14.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.057+0000 7f077edcf640 1 -- 172.21.15.88:0/2666107987 shutdown_connections 2024-09-16T01:35:14.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.057+0000 7f077edcf640 1 --2- 172.21.15.88:0/2666107987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07780b7290 0x7f0778161de0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:14.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.057+0000 7f077edcf640 1 --2- 172.21.15.88:0/2666107987 >> v2:172.21.15.154:3300/0 conn(0x7f07780d1b40 0x7f07781618a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:14.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.057+0000 7f077edcf640 1 -- 172.21.15.88:0/2666107987 >> 172.21.15.88:0/2666107987 conn(0x7f07780d2450 msgr2=0x7f07780d2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:14.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.057+0000 7f077edcf640 1 -- 172.21.15.88:0/2666107987 shutdown_connections 2024-09-16T01:35:14.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.057+0000 7f077edcf640 1 -- 172.21.15.88:0/2666107987 wait complete. 2024-09-16T01:35:14.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.058+0000 7f077edcf640 1 Processor -- start 2024-09-16T01:35:14.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.058+0000 7f077edcf640 1 -- start start 2024-09-16T01:35:14.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.059+0000 7f077edcf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07780b6eb0 0x7f0778149070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:14.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.059+0000 7f077edcf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07780b7290 0x7f0778159780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:14.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.059+0000 7f077edcf640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07780d1b40 0x7f077815efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:14.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.059+0000 7f077edcf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0778071af0 con 0x7f07780b7290 2024-09-16T01:35:14.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.059+0000 7f077edcf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0778071970 con 0x7f07780b6eb0 2024-09-16T01:35:14.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.059+0000 7f077edcf640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0778162320 con 0x7f07780d1b40 2024-09-16T01:35:14.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.059+0000 7f076ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07780b7290 0x7f0778159780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:14.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.059+0000 7f076ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07780b7290 0x7f0778159780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33712/0 (socket says 172.21.15.88:33712) 2024-09-16T01:35:14.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.059+0000 7f076ffff640 1 -- 172.21.15.88:0/980589551 learned_addr learned my addr 172.21.15.88:0/980589551 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.059+0000 7f077cb6b640 1 --2- 172.21.15.88:0/980589551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07780b6eb0 0x7f0778149070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.059+0000 7f077d36c640 1 --2- 172.21.15.88:0/980589551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07780d1b40 0x7f077815efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.059+0000 7f076ffff640 1 -- 172.21.15.88:0/980589551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07780d1b40 msgr2=0x7f077815efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.059+0000 7f076ffff640 1 --2- 172.21.15.88:0/980589551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07780d1b40 0x7f077815efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.059+0000 7f076ffff640 1 -- 172.21.15.88:0/980589551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07780b6eb0 msgr2=0x7f0778149070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.059+0000 7f076ffff640 1 --2- 172.21.15.88:0/980589551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07780b6eb0 0x7f0778149070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.059+0000 7f076ffff640 1 -- 172.21.15.88:0/980589551 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07781533f0 con 0x7f07780b7290 2024-09-16T01:35:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.059+0000 7f076ffff640 1 --2- 172.21.15.88:0/980589551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07780b7290 0x7f0778159780 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f076800b410 tx=0x7f076800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.059+0000 7f076d7fa640 1 -- 172.21.15.88:0/980589551 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0768015020 con 0x7f07780b7290 2024-09-16T01:35:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.059+0000 7f076d7fa640 1 -- 172.21.15.88:0/980589551 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0768011850 con 0x7f07780b7290 2024-09-16T01:35:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.059+0000 7f077edcf640 1 -- 172.21.15.88:0/980589551 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f077815df00 con 0x7f07780b7290 2024-09-16T01:35:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.059+0000 7f076d7fa640 1 -- 172.21.15.88:0/980589551 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0768011c40 con 0x7f07780b7290 2024-09-16T01:35:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.059+0000 7f077edcf640 1 -- 172.21.15.88:0/980589551 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f077815e4c0 con 0x7f07780b7290 2024-09-16T01:35:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.060+0000 7f077edcf640 1 -- 172.21.15.88:0/980589551 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f077806fd00 con 0x7f07780b7290 2024-09-16T01:35:14.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.061+0000 7f076d7fa640 1 -- 172.21.15.88:0/980589551 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0768018020 con 0x7f07780b7290 2024-09-16T01:35:14.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.061+0000 7f076d7fa640 1 --2- 172.21.15.88:0/980589551 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f076406c2b0 0x7f076406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:14.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.061+0000 7f077cb6b640 1 --2- 172.21.15.88:0/980589551 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f076406c2b0 0x7f076406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:14.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.061+0000 7f076d7fa640 1 -- 172.21.15.88:0/980589551 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f076808b3a0 con 0x7f07780b7290 2024-09-16T01:35:14.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.062+0000 7f077cb6b640 1 --2- 172.21.15.88:0/980589551 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f076406c2b0 0x7f076406e770 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0774000f60 tx=0x7f0774007b50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:14.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.066+0000 7f076d7fa640 1 -- 172.21.15.88:0/980589551 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f076801f020 con 0x7f07780b7290 2024-09-16T01:35:14.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.264+0000 7f077edcf640 1 -- 172.21.15.88:0/980589551 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f07780b19b0 con 0x7f07780b7290 2024-09-16T01:35:14.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.264+0000 7f076d7fa640 1 -- 172.21.15.88:0/980589551 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5223 (secure 0 0 0) 0x7f07680172c0 con 0x7f07780b7290 2024-09-16T01:35:14.266 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:35:14.266 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":16,"btime":"2024-09-16T01:35:13:942496+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1005331674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1005331674},{"type":"v1","addr":"172.21.15.88:6837","nonce":1005331674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/3427247662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3427247662},{"type":"v1","addr":"172.21.15.154:6834","nonce":3427247662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/4059835337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":4059835337},{"type":"v1","addr":"172.21.15.154:6835","nonce":4059835337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:35:00.674725+0000","modified":"2024-09-16T01:35:13.942488+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"c","rank":0,"incarnation":16,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6835/2870334334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870334334},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870334334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-16T01:35:14.266 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 16 2024-09-16T01:35:14.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.266+0000 7f077edcf640 1 -- 172.21.15.88:0/980589551 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f076406c2b0 msgr2=0x7f076406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:14.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.266+0000 7f077edcf640 1 --2- 172.21.15.88:0/980589551 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f076406c2b0 0x7f076406e770 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0774000f60 tx=0x7f0774007b50 comp rx=0 tx=0).stop 2024-09-16T01:35:14.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.266+0000 7f077edcf640 1 -- 172.21.15.88:0/980589551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07780b7290 msgr2=0x7f0778159780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:14.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.266+0000 7f077edcf640 1 --2- 172.21.15.88:0/980589551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07780b7290 0x7f0778159780 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f076800b410 tx=0x7f076800b8d0 comp rx=0 tx=0).stop 2024-09-16T01:35:14.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.266+0000 7f077edcf640 1 -- 172.21.15.88:0/980589551 shutdown_connections 2024-09-16T01:35:14.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.266+0000 7f077edcf640 1 --2- 172.21.15.88:0/980589551 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f076406c2b0 0x7f076406e770 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:14.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.266+0000 7f077edcf640 1 --2- 172.21.15.88:0/980589551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07780d1b40 0x7f077815efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:14.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.266+0000 7f077edcf640 1 --2- 172.21.15.88:0/980589551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07780b7290 0x7f0778159780 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:14.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.266+0000 7f077edcf640 1 --2- 172.21.15.88:0/980589551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07780b6eb0 0x7f0778149070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:14.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.266+0000 7f077edcf640 1 -- 172.21.15.88:0/980589551 >> 172.21.15.88:0/980589551 conn(0x7f07780d2450 msgr2=0x7f077815d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:14.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.267+0000 7f077edcf640 1 -- 172.21.15.88:0/980589551 shutdown_connections 2024-09-16T01:35:14.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:14.267+0000 7f077edcf640 1 -- 172.21.15.88:0/980589551 wait complete. 2024-09-16T01:35:14.682 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:35:00.674725+0000', 'modified': '2024-09-16T01:35:13.942488+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5067}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5067': {'gid': 5067, 'name': 'c', 'rank': 0, 'incarnation': 16, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.88:6835/2870334334', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 2870334334}, {'type': 'v1', 'addr': '172.21.15.88:6835', 'nonce': 2870334334}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-16T01:35:14.682 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5067:up:replay 2024-09-16T01:35:15.682 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:35:15.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.791+0000 7f03ddd4b640 1 Processor -- start 2024-09-16T01:35:15.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.791+0000 7f03ddd4b640 1 -- start start 2024-09-16T01:35:15.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.792+0000 7f03ddd4b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f03d8072050 0x7f03d806cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:15.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.792+0000 7f03ddd4b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03d806d670 con 0x7f03d815b540 2024-09-16T01:35:15.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.792+0000 7f03ddd4b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03d8073760 con 0x7f03d806d260 2024-09-16T01:35:15.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.792+0000 7f03ddd4b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f03d80726c0 con 0x7f03d8072050 2024-09-16T01:35:15.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.792+0000 7f03d77fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f03d815b540 0x7f03d80d0d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54536/0 (socket says 172.21.15.88:54536) 2024-09-16T01:35:15.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.792+0000 7f03d77fe640 1 -- 172.21.15.88:0/3567962768 learned_addr learned my addr 172.21.15.88:0/3567962768 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:15.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d67fc640 1 -- 172.21.15.88:0/3567962768 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1782352046 0 0) 0x7f03c8001490 con 0x7f03d815b540 2024-09-16T01:35:15.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d67fc640 1 -- 172.21.15.88:0/3567962768 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03c0003520 con 0x7f03d815b540 2024-09-16T01:35:15.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d6ffd640 1 --2- 172.21.15.88:0/3567962768 >> v2:172.21.15.154:3300/0 conn(0x7f03d8072050 0x7f03d806cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:15.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d67fc640 1 -- 172.21.15.88:0/3567962768 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 49691875 0 0) 0x7f03c0003520 con 0x7f03d815b540 2024-09-16T01:35:15.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d67fc640 1 -- 172.21.15.88:0/3567962768 >> v2:172.21.15.154:3300/0 conn(0x7f03d8072050 msgr2=0x7f03d806cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:15.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d67fc640 1 --2- 172.21.15.88:0/3567962768 >> v2:172.21.15.154:3300/0 conn(0x7f03d8072050 0x7f03d806cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:15.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d67fc640 1 -- 172.21.15.88:0/3567962768 >> v1:172.21.15.154:6789/0 conn(0x7f03d806d260 legacy=0x7f03d815b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:15.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d67fc640 1 -- 172.21.15.88:0/3567962768 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03d80729a0 con 0x7f03d815b540 2024-09-16T01:35:15.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d67fc640 1 -- 172.21.15.88:0/3567962768 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f03c8001bc0 con 0x7f03d815b540 2024-09-16T01:35:15.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d67fc640 1 -- 172.21.15.88:0/3567962768 >> v1:172.21.15.88:6789/0 conn(0x7f03d815b540 legacy=0x7f03d80d0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:15.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d67fc640 1 --2- 172.21.15.88:0/3567962768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03c00039b0 0x7f03c0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:15.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d67fc640 1 --2- 172.21.15.88:0/3567962768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03c0004800 0x7f03c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:15.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d67fc640 1 --2- 172.21.15.88:0/3567962768 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03c00051c0 0x7f03c000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:15.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d67fc640 1 -- 172.21.15.88:0/3567962768 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f03d80726c0 con 0x7f03c0004800 2024-09-16T01:35:15.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d67fc640 1 -- 172.21.15.88:0/3567962768 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f03c000c110 con 0x7f03c00039b0 2024-09-16T01:35:15.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d67fc640 1 -- 172.21.15.88:0/3567962768 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f03c000c310 con 0x7f03c00051c0 2024-09-16T01:35:15.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d6ffd640 1 --2- 172.21.15.88:0/3567962768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03c0004800 0x7f03c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:15.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d77fe640 1 --2- 172.21.15.88:0/3567962768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03c00039b0 0x7f03c0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:15.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d7fff640 1 --2- 172.21.15.88:0/3567962768 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03c00051c0 0x7f03c000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:15.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d6ffd640 1 -- 172.21.15.88:0/3567962768 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03c00051c0 msgr2=0x7f03c000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:15.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d6ffd640 1 --2- 172.21.15.88:0/3567962768 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03c00051c0 0x7f03c000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:15.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d6ffd640 1 -- 172.21.15.88:0/3567962768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03c00039b0 msgr2=0x7f03c0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:15.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d6ffd640 1 --2- 172.21.15.88:0/3567962768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03c00039b0 0x7f03c0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:15.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d6ffd640 1 -- 172.21.15.88:0/3567962768 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03c0003520 con 0x7f03c0004800 2024-09-16T01:35:15.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d7fff640 1 -- 172.21.15.88:0/3567962768 reap_dead start 2024-09-16T01:35:15.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.793+0000 7f03d77fe640 1 --2- 172.21.15.88:0/3567962768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03c00039b0 0x7f03c0003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:15.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.794+0000 7f03d6ffd640 1 --2- 172.21.15.88:0/3567962768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03c0004800 0x7f03c0004c80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f03cc00ba30 tx=0x7f03c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4a76f632669463c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:15.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.794+0000 7f03d67fc640 1 -- 172.21.15.88:0/3567962768 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03cc030070 con 0x7f03c0004800 2024-09-16T01:35:15.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.794+0000 7f03d67fc640 1 -- 172.21.15.88:0/3567962768 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03cc02ade0 con 0x7f03c0004800 2024-09-16T01:35:15.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.794+0000 7f03d67fc640 1 -- 172.21.15.88:0/3567962768 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03cc00b600 con 0x7f03c0004800 2024-09-16T01:35:15.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.795+0000 7f03ddd4b640 1 -- 172.21.15.88:0/3567962768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03c0004800 msgr2=0x7f03c0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:15.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.795+0000 7f03ddd4b640 1 --2- 172.21.15.88:0/3567962768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03c0004800 0x7f03c0004c80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f03cc00ba30 tx=0x7f03c0005830 comp rx=0 tx=0).stop 2024-09-16T01:35:15.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.795+0000 7f03ddd4b640 1 -- 172.21.15.88:0/3567962768 shutdown_connections 2024-09-16T01:35:15.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.795+0000 7f03ddd4b640 1 --2- 172.21.15.88:0/3567962768 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03c0004800 0x7f03c0004c80 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:15.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.795+0000 7f03ddd4b640 1 -- 172.21.15.88:0/3567962768 >> 172.21.15.88:0/3567962768 conn(0x7f03d80d14d0 msgr2=0x7f03d80d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:15.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.795+0000 7f03ddd4b640 1 -- 172.21.15.88:0/3567962768 shutdown_connections 2024-09-16T01:35:15.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.795+0000 7f03ddd4b640 1 -- 172.21.15.88:0/3567962768 wait complete. 2024-09-16T01:35:15.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.796+0000 7f03ddd4b640 1 Processor -- start 2024-09-16T01:35:15.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.796+0000 7f03ddd4b640 1 -- start start 2024-09-16T01:35:15.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.797+0000 7f03ddd4b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03c0004800 0x7f03d8153350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:15.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.797+0000 7f03ddd4b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03d81539d0 0x7f03d8150190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:15.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.797+0000 7f03ddd4b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03d814c740 0x7f03d8155730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:15.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.797+0000 7f03ddd4b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f03d8073240 con 0x7f03d81539d0 2024-09-16T01:35:15.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.797+0000 7f03ddd4b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f03d8072f60 con 0x7f03d814c740 2024-09-16T01:35:15.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.797+0000 7f03ddd4b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f03d806cd20 con 0x7f03c0004800 2024-09-16T01:35:15.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.797+0000 7f03d6ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03d81539d0 0x7f03d8150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:15.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.797+0000 7f03d6ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03d81539d0 0x7f03d8150190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33734/0 (socket says 172.21.15.88:33734) 2024-09-16T01:35:15.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.797+0000 7f03d6ffd640 1 -- 172.21.15.88:0/3517782506 learned_addr learned my addr 172.21.15.88:0/3517782506 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:15.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.797+0000 7f03d77fe640 1 --2- 172.21.15.88:0/3517782506 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03c0004800 0x7f03d8153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:15.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.797+0000 7f03d7fff640 1 --2- 172.21.15.88:0/3517782506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03d814c740 0x7f03d8155730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:15.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.797+0000 7f03d6ffd640 1 -- 172.21.15.88:0/3517782506 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03c0004800 msgr2=0x7f03d8153350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:15.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.797+0000 7f03d6ffd640 1 --2- 172.21.15.88:0/3517782506 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03c0004800 0x7f03d8153350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:15.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.797+0000 7f03d6ffd640 1 -- 172.21.15.88:0/3517782506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03d814c740 msgr2=0x7f03d8155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:15.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.797+0000 7f03d6ffd640 1 --2- 172.21.15.88:0/3517782506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03d814c740 0x7f03d8155730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:15.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.797+0000 7f03d6ffd640 1 -- 172.21.15.88:0/3517782506 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03d814f410 con 0x7f03d81539d0 2024-09-16T01:35:15.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.798+0000 7f03d6ffd640 1 --2- 172.21.15.88:0/3517782506 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03d81539d0 0x7f03d8150190 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f03cc00bb50 tx=0x7f03cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:15.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.798+0000 7f03bffff640 1 -- 172.21.15.88:0/3517782506 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03cc043030 con 0x7f03d81539d0 2024-09-16T01:35:15.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.798+0000 7f03bffff640 1 -- 172.21.15.88:0/3517782506 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03cc02abe0 con 0x7f03d81539d0 2024-09-16T01:35:15.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.798+0000 7f03ddd4b640 1 -- 172.21.15.88:0/3517782506 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03d81541b0 con 0x7f03d81539d0 2024-09-16T01:35:15.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.798+0000 7f03bffff640 1 -- 172.21.15.88:0/3517782506 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03cc00b600 con 0x7f03d81539d0 2024-09-16T01:35:15.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.798+0000 7f03ddd4b640 1 -- 172.21.15.88:0/3517782506 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03d814f990 con 0x7f03d81539d0 2024-09-16T01:35:15.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.799+0000 7f03ddd4b640 1 -- 172.21.15.88:0/3517782506 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03d814fe70 con 0x7f03d81539d0 2024-09-16T01:35:15.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.799+0000 7f03bffff640 1 -- 172.21.15.88:0/3517782506 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f03cc046070 con 0x7f03d81539d0 2024-09-16T01:35:15.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.800+0000 7f03bffff640 1 --2- 172.21.15.88:0/3517782506 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f03c406c2b0 0x7f03c406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:15.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.800+0000 7f03d77fe640 1 --2- 172.21.15.88:0/3517782506 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f03c406c2b0 0x7f03c406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:15.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.800+0000 7f03bffff640 1 -- 172.21.15.88:0/3517782506 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f03cc0abfb0 con 0x7f03d81539d0 2024-09-16T01:35:15.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.800+0000 7f03d77fe640 1 --2- 172.21.15.88:0/3517782506 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f03c406c2b0 0x7f03c406e770 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f03c8002230 tx=0x7f03c8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:15.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:15.804+0000 7f03bffff640 1 -- 172.21.15.88:0/3517782506 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f03d814fe70 con 0x7f03d81539d0 2024-09-16T01:35:15.944 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:35:15.947 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:35:16.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:16.007+0000 7f03ddd4b640 1 -- 172.21.15.88:0/3517782506 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03d806fe20 con 0x7f03d81539d0 2024-09-16T01:35:16.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:16.007+0000 7f03bffff640 1 -- 172.21.15.88:0/3517782506 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5223 (secure 0 0 0) 0x7f03cc07a630 con 0x7f03d81539d0 2024-09-16T01:35:16.009 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:35:16.009 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":18,"btime":"2024-09-16T01:35:15:945607+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1005331674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1005331674},{"type":"v1","addr":"172.21.15.88:6837","nonce":1005331674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/3427247662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3427247662},{"type":"v1","addr":"172.21.15.154:6834","nonce":3427247662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/4059835337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":4059835337},{"type":"v1","addr":"172.21.15.154:6835","nonce":4059835337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:35:00.674725+0000","modified":"2024-09-16T01:35:14.947412+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"c","rank":0,"incarnation":16,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.88:6835/2870334334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870334334},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870334334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-16T01:35:16.009 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 18 2024-09-16T01:35:16.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:16.009+0000 7f03ddd4b640 1 -- 172.21.15.88:0/3517782506 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f03c406c2b0 msgr2=0x7f03c406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:16.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:16.009+0000 7f03ddd4b640 1 --2- 172.21.15.88:0/3517782506 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f03c406c2b0 0x7f03c406e770 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f03c8002230 tx=0x7f03c8009040 comp rx=0 tx=0).stop 2024-09-16T01:35:16.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:16.009+0000 7f03ddd4b640 1 -- 172.21.15.88:0/3517782506 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03d81539d0 msgr2=0x7f03d8150190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:16.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:16.009+0000 7f03ddd4b640 1 --2- 172.21.15.88:0/3517782506 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03d81539d0 0x7f03d8150190 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f03cc00bb50 tx=0x7f03cc0046c0 comp rx=0 tx=0).stop 2024-09-16T01:35:16.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:16.009+0000 7f03ddd4b640 1 -- 172.21.15.88:0/3517782506 shutdown_connections 2024-09-16T01:35:16.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:16.009+0000 7f03ddd4b640 1 --2- 172.21.15.88:0/3517782506 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f03c406c2b0 0x7f03c406e770 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:16.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:16.009+0000 7f03ddd4b640 1 --2- 172.21.15.88:0/3517782506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03d814c740 0x7f03d8155730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:16.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:16.009+0000 7f03ddd4b640 1 --2- 172.21.15.88:0/3517782506 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03d81539d0 0x7f03d8150190 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:16.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:16.009+0000 7f03ddd4b640 1 --2- 172.21.15.88:0/3517782506 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03c0004800 0x7f03d8153350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:16.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:16.010+0000 7f03ddd4b640 1 -- 172.21.15.88:0/3517782506 >> 172.21.15.88:0/3517782506 conn(0x7f03d80d14d0 msgr2=0x7f03d80b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:16.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:16.010+0000 7f03ddd4b640 1 -- 172.21.15.88:0/3517782506 shutdown_connections 2024-09-16T01:35:16.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:16.010+0000 7f03ddd4b640 1 -- 172.21.15.88:0/3517782506 wait complete. 2024-09-16T01:35:16.425 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:35:00.674725+0000', 'modified': '2024-09-16T01:35:14.947412+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5067}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5067': {'gid': 5067, 'name': 'c', 'rank': 0, 'incarnation': 16, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.88:6835/2870334334', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 2870334334}, {'type': 'v1', 'addr': '172.21.15.88:6835', 'nonce': 2870334334}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-16T01:35:16.426 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5067:up:rejoin 2024-09-16T01:35:17.427 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:35:17.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.536+0000 7f7418ff6640 1 Processor -- start 2024-09-16T01:35:17.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.536+0000 7f7418ff6640 1 -- start start 2024-09-16T01:35:17.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.537+0000 7f7418ff6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f741415c2b0 0x7f74140b6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:17.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.537+0000 7f7418ff6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f741415f200 0x7f741406d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:17.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.537+0000 7f7418ff6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f741415fb10 con 0x7f74140b72f0 2024-09-16T01:35:17.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.537+0000 7f7418ff6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f74140b5900 con 0x7f741415c2b0 2024-09-16T01:35:17.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.537+0000 7f7418ff6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f741415f830 con 0x7f741415f200 2024-09-16T01:35:17.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.538+0000 7f7412d9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f741415f200 0x7f741406d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:17.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.538+0000 7f741259c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f741415c2b0 0x7f74140b6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:17.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.538+0000 7f7412d9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f741415f200 0x7f741406d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:47960/0 (socket says 172.21.15.88:47960) 2024-09-16T01:35:17.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.538+0000 7f7411d9b640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f74140b72f0 0x7f741415eaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:57594/0 (socket says 172.21.15.88:57594) 2024-09-16T01:35:17.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.538+0000 7f7412d9d640 1 -- 172.21.15.88:0/1602507062 learned_addr learned my addr 172.21.15.88:0/1602507062 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:17.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.538+0000 7f741259c640 1 -- 172.21.15.88:0/1602507062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f741415f200 msgr2=0x7f741406d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:17.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.538+0000 7f741259c640 1 --2- 172.21.15.88:0/1602507062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f741415f200 0x7f741406d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:17.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.538+0000 7f741259c640 1 -- 172.21.15.88:0/1602507062 >> v1:172.21.15.154:6789/0 conn(0x7f74140b72f0 legacy=0x7f741415eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:17.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.538+0000 7f741259c640 1 -- 172.21.15.88:0/1602507062 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f741406ce30 con 0x7f741415c2b0 2024-09-16T01:35:17.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.538+0000 7f741159a640 1 -- 172.21.15.88:0/1602507062 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1540662880 0 0) 0x7f741415fb10 con 0x7f74140b72f0 2024-09-16T01:35:17.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.539+0000 7f741259c640 1 --2- 172.21.15.88:0/1602507062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f741415c2b0 0x7f74140b6db0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f74040097a0 tx=0x7f740402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8dae5f022f07d973 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:17.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.539+0000 7f741159a640 1 -- 172.21.15.88:0/1602507062 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7404037070 con 0x7f741415c2b0 2024-09-16T01:35:17.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.539+0000 7f741159a640 1 -- 172.21.15.88:0/1602507062 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7404032440 con 0x7f741415c2b0 2024-09-16T01:35:17.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.539+0000 7f741159a640 1 -- 172.21.15.88:0/1602507062 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7404032810 con 0x7f741415c2b0 2024-09-16T01:35:17.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.539+0000 7f7418ff6640 1 -- 172.21.15.88:0/1602507062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f741415c2b0 msgr2=0x7f74140b6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:17.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.539+0000 7f7418ff6640 1 --2- 172.21.15.88:0/1602507062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f741415c2b0 0x7f74140b6db0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f74040097a0 tx=0x7f740402a180 comp rx=0 tx=0).stop 2024-09-16T01:35:17.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.540+0000 7f7418ff6640 1 -- 172.21.15.88:0/1602507062 shutdown_connections 2024-09-16T01:35:17.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.540+0000 7f7418ff6640 1 --2- 172.21.15.88:0/1602507062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f741415f200 0x7f741406d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:17.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.540+0000 7f7418ff6640 1 --2- 172.21.15.88:0/1602507062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f741415c2b0 0x7f74140b6db0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:17.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.540+0000 7f7418ff6640 1 -- 172.21.15.88:0/1602507062 >> 172.21.15.88:0/1602507062 conn(0x7f74140b4240 msgr2=0x7f74140b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:17.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.540+0000 7f7418ff6640 1 -- 172.21.15.88:0/1602507062 shutdown_connections 2024-09-16T01:35:17.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.540+0000 7f7418ff6640 1 -- 172.21.15.88:0/1602507062 wait complete. 2024-09-16T01:35:17.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.541+0000 7f7418ff6640 1 Processor -- start 2024-09-16T01:35:17.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.541+0000 7f7418ff6640 1 -- start start 2024-09-16T01:35:17.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.542+0000 7f7418ff6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74140b72f0 0x7f7414153840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:17.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.542+0000 7f7418ff6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f741415c2b0 0x7f7414161420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:17.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.542+0000 7f7418ff6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f741415f200 0x7f7414159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:17.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.542+0000 7f7418ff6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f74140b7760 con 0x7f74140b72f0 2024-09-16T01:35:17.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.542+0000 7f7418ff6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f741406d3f0 con 0x7f741415f200 2024-09-16T01:35:17.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.542+0000 7f7418ff6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f741406de50 con 0x7f741415c2b0 2024-09-16T01:35:17.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.542+0000 7f741259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74140b72f0 0x7f7414153840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:17.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.542+0000 7f741259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74140b72f0 0x7f7414153840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33742/0 (socket says 172.21.15.88:33742) 2024-09-16T01:35:17.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.542+0000 7f741259c640 1 -- 172.21.15.88:0/1316598244 learned_addr learned my addr 172.21.15.88:0/1316598244 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:17.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.542+0000 7f7411d9b640 1 --2- 172.21.15.88:0/1316598244 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f741415c2b0 0x7f7414161420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:17.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.542+0000 7f7412d9d640 1 --2- 172.21.15.88:0/1316598244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f741415f200 0x7f7414159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:17.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.542+0000 7f741259c640 1 -- 172.21.15.88:0/1316598244 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f741415c2b0 msgr2=0x7f7414161420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:17.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.542+0000 7f741259c640 1 --2- 172.21.15.88:0/1316598244 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f741415c2b0 0x7f7414161420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:17.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.542+0000 7f741259c640 1 -- 172.21.15.88:0/1316598244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f741415f200 msgr2=0x7f7414159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:17.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.542+0000 7f741259c640 1 --2- 172.21.15.88:0/1316598244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f741415f200 0x7f7414159970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:17.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.542+0000 7f741259c640 1 -- 172.21.15.88:0/1316598244 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74141f4440 con 0x7f74140b72f0 2024-09-16T01:35:17.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.543+0000 7f741259c640 1 --2- 172.21.15.88:0/1316598244 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74140b72f0 0x7f7414153840 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f74040097a0 tx=0x7f74040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:17.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.543+0000 7f7402ffd640 1 -- 172.21.15.88:0/1316598244 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7404037050 con 0x7f74140b72f0 2024-09-16T01:35:17.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.543+0000 7f7402ffd640 1 -- 172.21.15.88:0/1316598244 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7404038040 con 0x7f74140b72f0 2024-09-16T01:35:17.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.543+0000 7f7418ff6640 1 -- 172.21.15.88:0/1316598244 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74141f4720 con 0x7f74140b72f0 2024-09-16T01:35:17.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.543+0000 7f7402ffd640 1 -- 172.21.15.88:0/1316598244 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7404004490 con 0x7f74140b72f0 2024-09-16T01:35:17.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.543+0000 7f7418ff6640 1 -- 172.21.15.88:0/1316598244 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74141f4ce0 con 0x7f74140b72f0 2024-09-16T01:35:17.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.544+0000 7f7418ff6640 1 -- 172.21.15.88:0/1316598244 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74140b6db0 con 0x7f74140b72f0 2024-09-16T01:35:17.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.544+0000 7f7402ffd640 1 -- 172.21.15.88:0/1316598244 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f740403c6d0 con 0x7f74140b72f0 2024-09-16T01:35:17.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.545+0000 7f7402ffd640 1 --2- 172.21.15.88:0/1316598244 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f73f406c2b0 0x7f73f406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:17.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.545+0000 7f7402ffd640 1 -- 172.21.15.88:0/1316598244 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f74040abd40 con 0x7f74140b72f0 2024-09-16T01:35:17.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.545+0000 7f7411d9b640 1 --2- 172.21.15.88:0/1316598244 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f73f406c2b0 0x7f73f406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:17.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.545+0000 7f7411d9b640 1 --2- 172.21.15.88:0/1316598244 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f73f406c2b0 0x7f73f406e770 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f74140b4720 tx=0x7f7408007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:17.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.549+0000 7f7402ffd640 1 -- 172.21.15.88:0/1316598244 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7404079e90 con 0x7f74140b72f0 2024-09-16T01:35:17.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.749+0000 7f7418ff6640 1 -- 172.21.15.88:0/1316598244 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f741406dc30 con 0x7f74140b72f0 2024-09-16T01:35:17.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.750+0000 7f7402ffd640 1 -- 172.21.15.88:0/1316598244 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5230 (secure 0 0 0) 0x7f740407e190 con 0x7f74140b72f0 2024-09-16T01:35:17.752 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:35:17.752 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":19,"btime":"2024-09-16T01:35:16:950413+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1005331674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1005331674},{"type":"v1","addr":"172.21.15.88:6837","nonce":1005331674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/3427247662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3427247662},{"type":"v1","addr":"172.21.15.154:6834","nonce":3427247662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/4059835337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":4059835337},{"type":"v1","addr":"172.21.15.154:6835","nonce":4059835337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:35:00.674725+0000","modified":"2024-09-16T01:35:16.950408+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"c","rank":0,"incarnation":16,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6835/2870334334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870334334},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870334334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5067,"qdb_cluster":[5067]},"id":2}]} 2024-09-16T01:35:17.752 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 19 2024-09-16T01:35:17.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.752+0000 7f7418ff6640 1 -- 172.21.15.88:0/1316598244 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f73f406c2b0 msgr2=0x7f73f406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:17.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.752+0000 7f7418ff6640 1 --2- 172.21.15.88:0/1316598244 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f73f406c2b0 0x7f73f406e770 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f74140b4720 tx=0x7f7408007d50 comp rx=0 tx=0).stop 2024-09-16T01:35:17.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.752+0000 7f7418ff6640 1 -- 172.21.15.88:0/1316598244 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74140b72f0 msgr2=0x7f7414153840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:17.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.752+0000 7f7418ff6640 1 --2- 172.21.15.88:0/1316598244 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74140b72f0 0x7f7414153840 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f74040097a0 tx=0x7f74040046c0 comp rx=0 tx=0).stop 2024-09-16T01:35:17.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.752+0000 7f7418ff6640 1 -- 172.21.15.88:0/1316598244 shutdown_connections 2024-09-16T01:35:17.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.752+0000 7f7418ff6640 1 --2- 172.21.15.88:0/1316598244 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f73f406c2b0 0x7f73f406e770 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:17.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.752+0000 7f7418ff6640 1 --2- 172.21.15.88:0/1316598244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f741415f200 0x7f7414159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:17.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.752+0000 7f7418ff6640 1 --2- 172.21.15.88:0/1316598244 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f741415c2b0 0x7f7414161420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:17.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.752+0000 7f7418ff6640 1 --2- 172.21.15.88:0/1316598244 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74140b72f0 0x7f7414153840 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:17.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.752+0000 7f7418ff6640 1 -- 172.21.15.88:0/1316598244 >> 172.21.15.88:0/1316598244 conn(0x7f74140b4240 msgr2=0x7f74140cd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:17.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.752+0000 7f7418ff6640 1 -- 172.21.15.88:0/1316598244 shutdown_connections 2024-09-16T01:35:17.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:17.752+0000 7f7418ff6640 1 -- 172.21.15.88:0/1316598244 wait complete. 2024-09-16T01:35:18.168 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 19, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:35:00.674725+0000', 'modified': '2024-09-16T01:35:16.950408+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 33, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5067}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5067': {'gid': 5067, 'name': 'c', 'rank': 0, 'incarnation': 16, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.88:6835/2870334334', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 2870334334}, {'type': 'v1', 'addr': '172.21.15.88:6835', 'nonce': 2870334334}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5067, 'qdb_cluster': [5067]} 2024-09-16T01:35:18.168 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T01:35:18.169 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-16T01:35:18.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.269+0000 7fc5c414b640 1 Processor -- start 2024-09-16T01:35:18.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.269+0000 7fc5c414b640 1 -- start start 2024-09-16T01:35:18.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.269+0000 7fc5c414b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc5bc15c290 0x7fc5bc06d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:18.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.269+0000 7fc5c414b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5bc06fe20 0x7fc5bc0cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:18.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.270+0000 7fc5c414b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5bc06cc20 con 0x7fc5bc15d680 2024-09-16T01:35:18.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.270+0000 7fc5c414b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc5bc06bc10 con 0x7fc5bc15c290 2024-09-16T01:35:18.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.270+0000 7fc5c414b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc5bc06be60 con 0x7fc5bc06fe20 2024-09-16T01:35:18.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.270+0000 7fc5c1ee7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc5bc15c290 0x7fc5bc06d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:18.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.270+0000 7fc5c1ee7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc5bc15c290 0x7fc5bc06d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33754/0 (socket says 172.21.15.88:33754) 2024-09-16T01:35:18.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.270+0000 7fc5c1ee7640 1 -- 172.21.15.88:0/3925228775 learned_addr learned my addr 172.21.15.88:0/3925228775 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:18.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.270+0000 7fc5c1ee7640 1 -- 172.21.15.88:0/3925228775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5bc06fe20 msgr2=0x7fc5bc0cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:18.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.270+0000 7fc5c1ee7640 1 --2- 172.21.15.88:0/3925228775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5bc06fe20 0x7fc5bc0cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.270+0000 7fc5c1ee7640 1 -- 172.21.15.88:0/3925228775 >> v1:172.21.15.154:6789/0 conn(0x7fc5bc15d680 legacy=0x7fc5bc06f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:18.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.270+0000 7fc5c1ee7640 1 -- 172.21.15.88:0/3925228775 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5bc06c140 con 0x7fc5bc15c290 2024-09-16T01:35:18.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.271+0000 7fc5c1ee7640 1 --2- 172.21.15.88:0/3925228775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5bc15c290 0x7fc5bc06d500 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4009870 tx=0x7fc5b402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de6c544eba78407a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:18.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.271+0000 7fc5c0ee5640 1 -- 172.21.15.88:0/3925228775 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4037070 con 0x7fc5bc15c290 2024-09-16T01:35:18.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.271+0000 7fc5c0ee5640 1 -- 172.21.15.88:0/3925228775 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5b4032440 con 0x7fc5bc15c290 2024-09-16T01:35:18.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.271+0000 7fc5c0ee5640 1 -- 172.21.15.88:0/3925228775 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4032810 con 0x7fc5bc15c290 2024-09-16T01:35:18.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.271+0000 7fc5c414b640 1 -- 172.21.15.88:0/3925228775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5bc15c290 msgr2=0x7fc5bc06d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:18.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.271+0000 7fc5c414b640 1 --2- 172.21.15.88:0/3925228775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5bc15c290 0x7fc5bc06d500 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4009870 tx=0x7fc5b402a210 comp rx=0 tx=0).stop 2024-09-16T01:35:18.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.271+0000 7fc5c414b640 1 -- 172.21.15.88:0/3925228775 shutdown_connections 2024-09-16T01:35:18.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.271+0000 7fc5c414b640 1 --2- 172.21.15.88:0/3925228775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5bc06fe20 0x7fc5bc0cf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.271+0000 7fc5c414b640 1 --2- 172.21.15.88:0/3925228775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5bc15c290 0x7fc5bc06d500 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.271+0000 7fc5c414b640 1 -- 172.21.15.88:0/3925228775 >> 172.21.15.88:0/3925228775 conn(0x7fc5bc0b3070 msgr2=0x7fc5bc0b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:18.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.271+0000 7fc5c414b640 1 -- 172.21.15.88:0/3925228775 shutdown_connections 2024-09-16T01:35:18.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.271+0000 7fc5c414b640 1 -- 172.21.15.88:0/3925228775 wait complete. 2024-09-16T01:35:18.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.273+0000 7fc5c414b640 1 Processor -- start 2024-09-16T01:35:18.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.273+0000 7fc5c414b640 1 -- start start 2024-09-16T01:35:18.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.273+0000 7fc5c414b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5bc06fe20 0x7fc5bc14bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:18.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.273+0000 7fc5c414b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5bc15c290 0x7fc5bc14ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:18.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.273+0000 7fc5c414b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5bc15d680 0x7fc5bc14d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:18.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.273+0000 7fc5c414b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc5bc06da40 con 0x7fc5bc15c290 2024-09-16T01:35:18.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.273+0000 7fc5c414b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc5bc06c700 con 0x7fc5bc15d680 2024-09-16T01:35:18.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.273+0000 7fc5c414b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc5bc06dbc0 con 0x7fc5bc06fe20 2024-09-16T01:35:18.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.273+0000 7fc5c16e6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5bc15c290 0x7fc5bc14ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:18.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.273+0000 7fc5c16e6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5bc15c290 0x7fc5bc14ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33766/0 (socket says 172.21.15.88:33766) 2024-09-16T01:35:18.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.273+0000 7fc5c16e6640 1 -- 172.21.15.88:0/4056619716 learned_addr learned my addr 172.21.15.88:0/4056619716 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:18.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.274+0000 7fc5c26e8640 1 --2- 172.21.15.88:0/4056619716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5bc15d680 0x7fc5bc14d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:18.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.274+0000 7fc5c16e6640 1 -- 172.21.15.88:0/4056619716 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5bc06fe20 msgr2=0x7fc5bc14bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:18.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.274+0000 7fc5c1ee7640 1 --2- 172.21.15.88:0/4056619716 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5bc06fe20 0x7fc5bc14bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:18.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.274+0000 7fc5c16e6640 1 --2- 172.21.15.88:0/4056619716 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5bc06fe20 0x7fc5bc14bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.274+0000 7fc5c16e6640 1 -- 172.21.15.88:0/4056619716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5bc15d680 msgr2=0x7fc5bc14d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:18.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.274+0000 7fc5c16e6640 1 --2- 172.21.15.88:0/4056619716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5bc15d680 0x7fc5bc14d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.274+0000 7fc5c16e6640 1 -- 172.21.15.88:0/4056619716 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5bc1519d0 con 0x7fc5bc15c290 2024-09-16T01:35:18.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.274+0000 7fc5c26e8640 1 --2- 172.21.15.88:0/4056619716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5bc15d680 0x7fc5bc14d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:18.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.274+0000 7fc5c1ee7640 1 --2- 172.21.15.88:0/4056619716 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5bc06fe20 0x7fc5bc14bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:18.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.274+0000 7fc5c16e6640 1 --2- 172.21.15.88:0/4056619716 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5bc15c290 0x7fc5bc14ce80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000ba50 tx=0x7fc5b000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:18.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.274+0000 7fc5ba7fc640 1 -- 172.21.15.88:0/4056619716 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0019070 con 0x7fc5bc15c290 2024-09-16T01:35:18.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.274+0000 7fc5ba7fc640 1 -- 172.21.15.88:0/4056619716 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5b000cc10 con 0x7fc5bc15c290 2024-09-16T01:35:18.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.274+0000 7fc5ba7fc640 1 -- 172.21.15.88:0/4056619716 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0014440 con 0x7fc5bc15c290 2024-09-16T01:35:18.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.274+0000 7fc5c414b640 1 -- 172.21.15.88:0/4056619716 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5bc151cb0 con 0x7fc5bc15c290 2024-09-16T01:35:18.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.274+0000 7fc5c414b640 1 -- 172.21.15.88:0/4056619716 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5bc14dca0 con 0x7fc5bc15c290 2024-09-16T01:35:18.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.275+0000 7fc5ba7fc640 1 -- 172.21.15.88:0/4056619716 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc5b001c070 con 0x7fc5bc15c290 2024-09-16T01:35:18.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.276+0000 7fc5ba7fc640 1 --2- 172.21.15.88:0/4056619716 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc5a806c2b0 0x7fc5a806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:18.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.276+0000 7fc5ba7fc640 1 -- 172.21.15.88:0/4056619716 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fc5b008b280 con 0x7fc5bc15c290 2024-09-16T01:35:18.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.276+0000 7fc5c1ee7640 1 --2- 172.21.15.88:0/4056619716 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc5a806c2b0 0x7fc5a806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:18.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.276+0000 7fc5c1ee7640 1 --2- 172.21.15.88:0/4056619716 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc5a806c2b0 0x7fc5a806e770 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4034000 tx=0x7fc5b402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:18.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.277+0000 7fc5c6fc0740 1 Processor -- start 2024-09-16T01:35:18.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.277+0000 7fc5c6fc0740 1 -- start start 2024-09-16T01:35:18.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.278+0000 7fc5c6fc0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b02318c2c0 0x55b0231d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:18.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.278+0000 7fc5c6fc0740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b023189a80 0x55b02318c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:18.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.278+0000 7fc5c6fc0740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b02318da50 0x55b0231868b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:18.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.278+0000 7fc5c6fc0740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b0230ee520 con 0x55b02318da50 2024-09-16T01:35:18.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.278+0000 7fc5c6fc0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b0230963e0 con 0x55b02318c2c0 2024-09-16T01:35:18.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.278+0000 7fc5c6fc0740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55b0230ea4e0 con 0x55b023189a80 2024-09-16T01:35:18.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.278+0000 7fc5c26e8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b02318da50 0x55b0231868b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:18.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.278+0000 7fc5c26e8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b02318da50 0x55b0231868b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33772/0 (socket says 172.21.15.88:33772) 2024-09-16T01:35:18.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.278+0000 7fc5c26e8640 1 -- 172.21.15.88:0/1561821884 learned_addr learned my addr 172.21.15.88:0/1561821884 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:18.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.278+0000 7fc5c1ee7640 1 --2- 172.21.15.88:0/1561821884 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b02318c2c0 0x55b0231d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:18.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.278+0000 7fc5c16e6640 1 --2- 172.21.15.88:0/1561821884 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b023189a80 0x55b02318c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:18.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.278+0000 7fc5c26e8640 1 -- 172.21.15.88:0/1561821884 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b023189a80 msgr2=0x55b02318c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:18.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.278+0000 7fc5c26e8640 1 --2- 172.21.15.88:0/1561821884 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b023189a80 0x55b02318c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.278+0000 7fc5c26e8640 1 -- 172.21.15.88:0/1561821884 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b02318c2c0 msgr2=0x55b0231d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:18.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.278+0000 7fc5c26e8640 1 --2- 172.21.15.88:0/1561821884 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b02318c2c0 0x55b0231d1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.278+0000 7fc5c26e8640 1 -- 172.21.15.88:0/1561821884 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0230b12f0 con 0x55b02318da50 2024-09-16T01:35:18.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.278+0000 7fc5c1ee7640 1 --2- 172.21.15.88:0/1561821884 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b02318c2c0 0x55b0231d1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:18.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.278+0000 7fc5c16e6640 1 --2- 172.21.15.88:0/1561821884 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b023189a80 0x55b02318c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:18.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.278+0000 7fc5c26e8640 1 --2- 172.21.15.88:0/1561821884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b02318da50 0x55b0231868b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac00a9f0 tx=0x7fc5ac00aeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e17d5472d074ba07 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:18.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.278+0000 7fc5a37fe640 1 -- 172.21.15.88:0/1561821884 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac015070 con 0x55b02318da50 2024-09-16T01:35:18.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.278+0000 7fc5a37fe640 1 -- 172.21.15.88:0/1561821884 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5ac00cc60 con 0x55b02318da50 2024-09-16T01:35:18.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.279+0000 7fc5a37fe640 1 -- 172.21.15.88:0/1561821884 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac0113f0 con 0x55b02318da50 2024-09-16T01:35:18.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.279+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/1561821884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b02318da50 msgr2=0x55b0231868b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:18.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.279+0000 7fc5c6fc0740 1 --2- 172.21.15.88:0/1561821884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b02318da50 0x55b0231868b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac00a9f0 tx=0x7fc5ac00aeb0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.279+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/1561821884 shutdown_connections 2024-09-16T01:35:18.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.279+0000 7fc5c6fc0740 1 --2- 172.21.15.88:0/1561821884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b02318da50 0x55b0231868b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.279+0000 7fc5c6fc0740 1 --2- 172.21.15.88:0/1561821884 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b023189a80 0x55b02318c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.279+0000 7fc5c6fc0740 1 --2- 172.21.15.88:0/1561821884 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b02318c2c0 0x55b0231d1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.279+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/1561821884 >> 172.21.15.88:0/1561821884 conn(0x55b023188a80 msgr2=0x55b0231d0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:18.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.279+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/1561821884 shutdown_connections 2024-09-16T01:35:18.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.279+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/1561821884 wait complete. 2024-09-16T01:35:18.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.280+0000 7fc5c6fc0740 1 -- start start 2024-09-16T01:35:18.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.280+0000 7fc5c6fc0740 1 Processor -- start 2024-09-16T01:35:18.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.281+0000 7fc5c6fc0740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b02318c2c0 0x55b0231d4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:18.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.281+0000 7fc5c6fc0740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b02318da50 0x55b0231872e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:18.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.281+0000 7fc5c6fc0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0231d4b70 0x55b0231a7720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:18.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.281+0000 7fc5c6fc0740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b023187f20 con 0x55b02318c2c0 2024-09-16T01:35:18.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.281+0000 7fc5c6fc0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b023077450 con 0x55b0231d4b70 2024-09-16T01:35:18.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.281+0000 7fc5c6fc0740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55b0231d2290 con 0x55b02318da50 2024-09-16T01:35:18.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.281+0000 7fc5c6fc0740 10 client.0 _ll_register_callbacks cb 0x55b022e00ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:35:18.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.281+0000 7fc5c1ee7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b02318c2c0 0x55b0231d4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:18.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.281+0000 7fc5c1ee7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b02318c2c0 0x55b0231d4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33778/0 (socket says 172.21.15.88:33778) 2024-09-16T01:35:18.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.281+0000 7fc5c1ee7640 1 -- 172.21.15.88:0/3445117489 learned_addr learned my addr 172.21.15.88:0/3445117489 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:18.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.281+0000 7fc5c16e6640 1 --2- 172.21.15.88:0/3445117489 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b02318da50 0x55b0231872e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:18.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.281+0000 7fc5c26e8640 1 --2- 172.21.15.88:0/3445117489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0231d4b70 0x55b0231a7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:18.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.281+0000 7fc5c1ee7640 1 -- 172.21.15.88:0/3445117489 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b02318da50 msgr2=0x55b0231872e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:18.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.281+0000 7fc5c1ee7640 1 --2- 172.21.15.88:0/3445117489 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b02318da50 0x55b0231872e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.281+0000 7fc5c1ee7640 1 -- 172.21.15.88:0/3445117489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0231d4b70 msgr2=0x55b0231a7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:18.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.281+0000 7fc5c1ee7640 1 --2- 172.21.15.88:0/3445117489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0231d4b70 0x55b0231a7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.281+0000 7fc5c1ee7640 1 -- 172.21.15.88:0/3445117489 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b022e012d0 con 0x55b02318c2c0 2024-09-16T01:35:18.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.281+0000 7fc5c1ee7640 1 --2- 172.21.15.88:0/3445117489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b02318c2c0 0x55b0231d4630 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4035040 tx=0x7fc5b402c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f1d054fd81ce149 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:18.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.281+0000 7fc5a0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:35:18.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.281+0000 7fc5a0ff9640 1 -- 172.21.15.88:0/3445117489 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b404e060 con 0x55b02318c2c0 2024-09-16T01:35:18.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.281+0000 7fc5a0ff9640 1 -- 172.21.15.88:0/3445117489 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fc5b403edc0 con 0x55b02318c2c0 2024-09-16T01:35:18.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.281+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/3445117489 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5ac00cc60 con 0x55b02318c2c0 2024-09-16T01:35:18.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.282+0000 7fc5a0ff9640 1 -- 172.21.15.88:0/3445117489 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fc5b4054050 con 0x55b02318c2c0 2024-09-16T01:35:18.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.282+0000 7fc5c6fc0740 10 client.5112 fetch_fsmap learned FSMap version 19 2024-09-16T01:35:18.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.282+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/3445117489 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x55b022ecb6b0 con 0x55b02318c2c0 2024-09-16T01:35:18.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.282+0000 7fc5a0ff9640 1 -- 172.21.15.88:0/3445117489 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5ac00cc60 con 0x55b02318c2c0 2024-09-16T01:35:18.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.282+0000 7fc5a0ff9640 1 -- 172.21.15.88:0/3445117489 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 19) ==== 3502+0+0 (secure 0 0 0) 0x7fc5b4057020 con 0x55b02318c2c0 2024-09-16T01:35:18.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.282+0000 7fc5c6fc0740 10 client.5112 fetch_fsmap finished waiting for FSMap version 19 2024-09-16T01:35:18.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.282+0000 7fc5c6fc0740 10 client.5112 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T01:35:18.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.282+0000 7fc5c6fc0740 20 client.5112 populate_metadata read hostname 'smithi088' 2024-09-16T01:35:18.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.282+0000 7fc5c6fc0740 1 --2- 172.21.15.88:0/3445117489 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x55b023187820 0x55b0231e4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:18.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.282+0000 7fc5c6fc0740 4 client.5112 mds_command: new command op to 5067 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:35:18.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.282+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/3445117489 --> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0230fbc50 con 0x55b023187820 2024-09-16T01:35:18.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.282+0000 7fc5c16e6640 1 --2- 172.21.15.88:0/3445117489 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x55b023187820 0x55b0231e4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:18.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.282+0000 7fc5a0ff9640 10 client.5112 ms_handle_connect on v2:172.21.15.88:6834/2870334334 2024-09-16T01:35:18.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.282+0000 7fc5c16e6640 1 --2- 172.21.15.88:0/3445117489 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x55b023187820 0x55b0231e4930 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44c478f96ab3b01c server_cookie=87dde1015048e432 in_seq=0 out_seq=0 2024-09-16T01:35:18.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.283+0000 7fc5a0ff9640 1 -- 172.21.15.88:0/3445117489 <== mds.0 v2:172.21.15.88:6834/2870334334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b0230fbc50 con 0x55b023187820 2024-09-16T01:35:18.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.283+0000 7fc5a0ff9640 10 client.5112 handle_command_reply: tid=0 2024-09-16T01:35:18.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.283+0000 7fc5c6fc0740 1 client.5112 shutdown 2024-09-16T01:35:18.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.283+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/3445117489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b02318c2c0 msgr2=0x55b0231d4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:18.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.283+0000 7fc5c6fc0740 1 --2- 172.21.15.88:0/3445117489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b02318c2c0 0x55b0231d4630 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4035040 tx=0x7fc5b402c070 comp rx=0 tx=0).stop 2024-09-16T01:35:18.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.283+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/3445117489 shutdown_connections 2024-09-16T01:35:18.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.283+0000 7fc5c6fc0740 1 --2- 172.21.15.88:0/3445117489 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x55b023187820 0x55b0231e4930 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.283+0000 7fc5c6fc0740 1 --2- 172.21.15.88:0/3445117489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b0231d4b70 0x55b0231a7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.283+0000 7fc5c6fc0740 1 --2- 172.21.15.88:0/3445117489 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b02318da50 0x55b0231872e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.283+0000 7fc5c6fc0740 1 --2- 172.21.15.88:0/3445117489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b02318c2c0 0x55b0231d4630 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.283+0000 7fc5a0ff9640 0 client.5112 ms_handle_reset on v2:172.21.15.88:6834/2870334334 2024-09-16T01:35:18.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.283+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/3445117489 >> 172.21.15.88:0/3445117489 conn(0x55b023189a80 msgr2=0x55b0231d04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:18.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.283+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/3445117489 shutdown_connections 2024-09-16T01:35:18.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.283+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/3445117489 wait complete. 2024-09-16T01:35:18.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.283+0000 7fc5c6fc0740 20 client.5112 trim_cache size 0 max 16384 2024-09-16T01:35:18.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.301+0000 7fc5c6fc0740 1 Processor -- start 2024-09-16T01:35:18.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.301+0000 7fc5c6fc0740 1 -- start start 2024-09-16T01:35:18.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.302+0000 7fc5c6fc0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b02318c2c0 0x55b0231e5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:18.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.302+0000 7fc5c6fc0740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b02318da50 0x55b0231cfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:18.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.302+0000 7fc5c6fc0740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b0231d4b70 0x55b0231e1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:18.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.302+0000 7fc5c6fc0740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc5bc1fdd30 con 0x55b02318da50 2024-09-16T01:35:18.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.302+0000 7fc5c6fc0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc5bc1f6710 con 0x55b02318c2c0 2024-09-16T01:35:18.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.302+0000 7fc5c6fc0740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc5bc153350 con 0x55b0231d4b70 2024-09-16T01:35:18.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.302+0000 7fc5c16e6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b02318da50 0x55b0231cfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:18.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.302+0000 7fc5c16e6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b02318da50 0x55b0231cfcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33780/0 (socket says 172.21.15.88:33780) 2024-09-16T01:35:18.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.302+0000 7fc5c16e6640 1 -- 172.21.15.88:0/2674408829 learned_addr learned my addr 172.21.15.88:0/2674408829 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:18.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.302+0000 7fc5c26e8640 1 --2- 172.21.15.88:0/2674408829 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b0231d4b70 0x55b0231e1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:18.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.302+0000 7fc5c1ee7640 1 --2- 172.21.15.88:0/2674408829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b02318c2c0 0x55b0231e5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:18.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.302+0000 7fc5c16e6640 1 -- 172.21.15.88:0/2674408829 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b0231d4b70 msgr2=0x55b0231e1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:18.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.302+0000 7fc5c16e6640 1 --2- 172.21.15.88:0/2674408829 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b0231d4b70 0x55b0231e1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.302+0000 7fc5c16e6640 1 -- 172.21.15.88:0/2674408829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b02318c2c0 msgr2=0x55b0231e5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:18.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.302+0000 7fc5c16e6640 1 --2- 172.21.15.88:0/2674408829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b02318c2c0 0x55b0231e5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.302+0000 7fc5c16e6640 1 -- 172.21.15.88:0/2674408829 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0231b2fd0 con 0x55b02318da50 2024-09-16T01:35:18.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.302+0000 7fc5c1ee7640 1 --2- 172.21.15.88:0/2674408829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b02318c2c0 0x55b0231e5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:18.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.302+0000 7fc5c26e8640 1 --2- 172.21.15.88:0/2674408829 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b0231d4b70 0x55b0231e1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:18.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.302+0000 7fc5c16e6640 1 --2- 172.21.15.88:0/2674408829 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b02318da50 0x55b0231cfcb0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0009aa0 tx=0x7fc5b000e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6174eb4b80355289 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:18.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.302+0000 7fc5a2ffd640 1 -- 172.21.15.88:0/2674408829 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0057340 con 0x55b02318da50 2024-09-16T01:35:18.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.302+0000 7fc5a2ffd640 1 -- 172.21.15.88:0/2674408829 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5b00575a0 con 0x55b02318da50 2024-09-16T01:35:18.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.302+0000 7fc5a2ffd640 1 -- 172.21.15.88:0/2674408829 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b00619f0 con 0x55b02318da50 2024-09-16T01:35:18.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.303+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/2674408829 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b02318da50 msgr2=0x55b0231cfcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:18.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.303+0000 7fc5c6fc0740 1 --2- 172.21.15.88:0/2674408829 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b02318da50 0x55b0231cfcb0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0009aa0 tx=0x7fc5b000e040 comp rx=0 tx=0).stop 2024-09-16T01:35:18.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.303+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/2674408829 shutdown_connections 2024-09-16T01:35:18.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.303+0000 7fc5c6fc0740 1 --2- 172.21.15.88:0/2674408829 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b0231d4b70 0x55b0231e1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.303+0000 7fc5c6fc0740 1 --2- 172.21.15.88:0/2674408829 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b02318da50 0x55b0231cfcb0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.303+0000 7fc5c6fc0740 1 --2- 172.21.15.88:0/2674408829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b02318c2c0 0x55b0231e5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.303+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/2674408829 >> 172.21.15.88:0/2674408829 conn(0x55b023189a80 msgr2=0x55b0231a7380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:18.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.303+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/2674408829 shutdown_connections 2024-09-16T01:35:18.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.303+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/2674408829 wait complete. 2024-09-16T01:35:18.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.303+0000 7fc5c6fc0740 1 -- start start 2024-09-16T01:35:18.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.304+0000 7fc5c6fc0740 1 Processor -- start 2024-09-16T01:35:18.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.304+0000 7fc5c6fc0740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b02318c2c0 0x55b023187220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.304+0000 7fc5c6fc0740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b0231d4b70 0x55b0231d0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.304+0000 7fc5c6fc0740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b0231d1470 0x55b0232351f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.304+0000 7fc5c6fc0740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b0231b5960 con 0x55b0231d4b70 2024-09-16T01:35:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.304+0000 7fc5c6fc0740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b0231b5ba0 con 0x55b02318c2c0 2024-09-16T01:35:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.304+0000 7fc5c6fc0740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55b0231abdb0 con 0x55b0231d1470 2024-09-16T01:35:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.305+0000 7fc5c6fc0740 10 client.0 _ll_register_callbacks cb 0x7fc5b4035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:35:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.305+0000 7fc5c16e6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b0231d4b70 0x55b0231d0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.305+0000 7fc5c16e6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b0231d4b70 0x55b0231d0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33784/0 (socket says 172.21.15.88:33784) 2024-09-16T01:35:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.305+0000 7fc5c16e6640 1 -- 172.21.15.88:0/873115061 learned_addr learned my addr 172.21.15.88:0/873115061 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.305+0000 7fc5c26e8640 1 --2- 172.21.15.88:0/873115061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b0231d1470 0x55b0232351f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.305+0000 7fc5c1ee7640 1 --2- 172.21.15.88:0/873115061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b02318c2c0 0x55b023187220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.305+0000 7fc5c16e6640 1 -- 172.21.15.88:0/873115061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b0231d1470 msgr2=0x55b0232351f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.305+0000 7fc5c16e6640 1 --2- 172.21.15.88:0/873115061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b0231d1470 0x55b0232351f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.305+0000 7fc5c16e6640 1 -- 172.21.15.88:0/873115061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b02318c2c0 msgr2=0x55b023187220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.305+0000 7fc5c16e6640 1 --2- 172.21.15.88:0/873115061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b02318c2c0 0x55b023187220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.305+0000 7fc5c16e6640 1 -- 172.21.15.88:0/873115061 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0231e4f60 con 0x55b0231d4b70 2024-09-16T01:35:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.305+0000 7fc5c1ee7640 1 --2- 172.21.15.88:0/873115061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b02318c2c0 0x55b023187220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.305+0000 7fc5c16e6640 1 --2- 172.21.15.88:0/873115061 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b0231d4b70 0x55b0231d0f30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0009aa0 tx=0x7fc5b4008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e30c76b902d67a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.305+0000 7fc5a0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:35:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.305+0000 7fc5a0ff9640 1 -- 172.21.15.88:0/873115061 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0071040 con 0x55b0231d4b70 2024-09-16T01:35:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.305+0000 7fc5a0ff9640 1 -- 172.21.15.88:0/873115061 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fc5b0055910 con 0x55b0231d4b70 2024-09-16T01:35:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.305+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/873115061 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5b00575a0 con 0x55b0231d4b70 2024-09-16T01:35:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.305+0000 7fc5a0ff9640 1 -- 172.21.15.88:0/873115061 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fc5b00619f0 con 0x55b0231d4b70 2024-09-16T01:35:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.305+0000 7fc5c6fc0740 10 client.5118 fetch_fsmap learned FSMap version 19 2024-09-16T01:35:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.305+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/873115061 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x55b023253b30 con 0x55b0231d4b70 2024-09-16T01:35:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.305+0000 7fc5a0ff9640 1 -- 172.21.15.88:0/873115061 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5b00575a0 con 0x55b0231d4b70 2024-09-16T01:35:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.305+0000 7fc5a0ff9640 1 -- 172.21.15.88:0/873115061 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 19) ==== 3502+0+0 (secure 0 0 0) 0x7fc5b005d1b0 con 0x55b0231d4b70 2024-09-16T01:35:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.305+0000 7fc5c6fc0740 10 client.5118 fetch_fsmap finished waiting for FSMap version 19 2024-09-16T01:35:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.305+0000 7fc5c6fc0740 10 client.5118 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T01:35:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.305+0000 7fc5c6fc0740 20 client.5118 populate_metadata read hostname 'smithi088' 2024-09-16T01:35:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.306+0000 7fc5c6fc0740 1 --2- 172.21.15.88:0/873115061 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x55b0232501f0 0x55b0231aa260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.306+0000 7fc5c6fc0740 4 client.5118 mds_command: new command op to 5067 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:35:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.306+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/873115061 --> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] -- command(tid 0: {"prefix": "status"}) -- 0x55b0230f5fa0 con 0x55b0232501f0 2024-09-16T01:35:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.306+0000 7fc5c1ee7640 1 --2- 172.21.15.88:0/873115061 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x55b0232501f0 0x55b0231aa260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.306+0000 7fc5a0ff9640 10 client.5118 ms_handle_connect on v2:172.21.15.88:6834/2870334334 2024-09-16T01:35:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.306+0000 7fc5c1ee7640 1 --2- 172.21.15.88:0/873115061 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x55b0232501f0 0x55b0231aa260 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e48376afd3bfd9c3 server_cookie=af23a23f5c5625b2 in_seq=0 out_seq=0 2024-09-16T01:35:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.306+0000 7fc5a0ff9640 1 -- 172.21.15.88:0/873115061 <== mds.0 v2:172.21.15.88:6834/2870334334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x55b0230f5fa0 con 0x55b0232501f0 2024-09-16T01:35:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.306+0000 7fc5a0ff9640 10 client.5118 handle_command_reply: tid=0 2024-09-16T01:35:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.306+0000 7fc5c6fc0740 1 client.5118 shutdown 2024-09-16T01:35:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.306+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/873115061 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b0231d4b70 msgr2=0x55b0231d0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.306+0000 7fc5c6fc0740 1 --2- 172.21.15.88:0/873115061 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b0231d4b70 0x55b0231d0f30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0009aa0 tx=0x7fc5b4008030 comp rx=0 tx=0).stop 2024-09-16T01:35:18.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.306+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/873115061 shutdown_connections 2024-09-16T01:35:18.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.306+0000 7fc5c6fc0740 1 --2- 172.21.15.88:0/873115061 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x55b0232501f0 0x55b0231aa260 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.306+0000 7fc5c6fc0740 1 --2- 172.21.15.88:0/873115061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b0231d1470 0x55b0232351f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.306+0000 7fc5c6fc0740 1 --2- 172.21.15.88:0/873115061 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b0231d4b70 0x55b0231d0f30 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.306+0000 7fc5a0ff9640 0 client.5118 ms_handle_reset on v2:172.21.15.88:6834/2870334334 2024-09-16T01:35:18.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.306+0000 7fc5c6fc0740 1 --2- 172.21.15.88:0/873115061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b02318c2c0 0x55b023187220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.306+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/873115061 >> 172.21.15.88:0/873115061 conn(0x55b023189a80 msgr2=0x55b0231b4bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:18.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.306+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/873115061 shutdown_connections 2024-09-16T01:35:18.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.306+0000 7fc5c6fc0740 1 -- 172.21.15.88:0/873115061 wait complete. 2024-09-16T01:35:18.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.307+0000 7fc5c6fc0740 20 client.5118 trim_cache size 0 max 16384 2024-09-16T01:35:18.315 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:35:18.316 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "5ace2c91-46f9-482b-8a89-731fbdc171bf", 2024-09-16T01:35:18.316 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2024-09-16T01:35:18.316 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 5067, 2024-09-16T01:35:18.316 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2024-09-16T01:35:18.316 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2024-09-16T01:35:18.316 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2024-09-16T01:35:18.316 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 4.361040086, 2024-09-16T01:35:18.316 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 19, 2024-09-16T01:35:18.316 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 33, 2024-09-16T01:35:18.316 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 33, 2024-09-16T01:35:18.317 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 5.3304987019999999 2024-09-16T01:35:18.317 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:35:18.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.307+0000 7fc5c414b640 1 -- 172.21.15.88:0/4056619716 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc5a806c2b0 msgr2=0x7fc5a806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:18.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.307+0000 7fc5c414b640 1 --2- 172.21.15.88:0/4056619716 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc5a806c2b0 0x7fc5a806e770 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4034000 tx=0x7fc5b402e000 comp rx=0 tx=0).stop 2024-09-16T01:35:18.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.307+0000 7fc5c414b640 1 -- 172.21.15.88:0/4056619716 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5bc15c290 msgr2=0x7fc5bc14ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:18.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.307+0000 7fc5c414b640 1 --2- 172.21.15.88:0/4056619716 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5bc15c290 0x7fc5bc14ce80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000ba50 tx=0x7fc5b000bf10 comp rx=0 tx=0).stop 2024-09-16T01:35:18.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.307+0000 7fc5c414b640 1 -- 172.21.15.88:0/4056619716 shutdown_connections 2024-09-16T01:35:18.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.307+0000 7fc5c414b640 1 --2- 172.21.15.88:0/4056619716 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc5a806c2b0 0x7fc5a806e770 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.307+0000 7fc5c414b640 1 --2- 172.21.15.88:0/4056619716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5bc15d680 0x7fc5bc14d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.307+0000 7fc5c414b640 1 --2- 172.21.15.88:0/4056619716 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5bc15c290 0x7fc5bc14ce80 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.307+0000 7fc5c414b640 1 --2- 172.21.15.88:0/4056619716 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5bc06fe20 0x7fc5bc14bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.307+0000 7fc5c414b640 1 -- 172.21.15.88:0/4056619716 >> 172.21.15.88:0/4056619716 conn(0x7fc5bc0b3070 msgr2=0x7fc5bc1579f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:18.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.307+0000 7fc5c414b640 1 -- 172.21.15.88:0/4056619716 shutdown_connections 2024-09-16T01:35:18.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.307+0000 7fc5c414b640 1 -- 172.21.15.88:0/4056619716 wait complete. 2024-09-16T01:35:18.319 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:35:18.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.416+0000 7f078a8e4640 1 Processor -- start 2024-09-16T01:35:18.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.416+0000 7f078a8e4640 1 -- start start 2024-09-16T01:35:18.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.417+0000 7f078a8e4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f078415c2b0 0x7f07840b6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:18.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.417+0000 7f078a8e4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f078415fb10 con 0x7f078415f200 2024-09-16T01:35:18.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.417+0000 7f078a8e4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07840b5900 con 0x7f07840b72f0 2024-09-16T01:35:18.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.417+0000 7f078a8e4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f078415f830 con 0x7f078415c2b0 2024-09-16T01:35:18.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.417+0000 7f0788e81640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f078415f200 0x7f078406d910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54542/0 (socket says 172.21.15.88:54542) 2024-09-16T01:35:18.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.417+0000 7f0788e81640 1 -- 172.21.15.88:0/2674032278 learned_addr learned my addr 172.21.15.88:0/2674032278 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:18.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.417+0000 7f0783fff640 1 --2- 172.21.15.88:0/2674032278 >> v2:172.21.15.154:3300/0 conn(0x7f078415c2b0 0x7f07840b6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:18.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.417+0000 7f0782ffd640 1 -- 172.21.15.88:0/2674032278 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1225102876 0 0) 0x7f078415fb10 con 0x7f078415f200 2024-09-16T01:35:18.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.417+0000 7f0782ffd640 1 -- 172.21.15.88:0/2674032278 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f076c003520 con 0x7f078415f200 2024-09-16T01:35:18.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.417+0000 7f0783fff640 1 -- 172.21.15.88:0/2674032278 >> v1:172.21.15.154:6789/0 conn(0x7f07840b72f0 legacy=0x7f078415eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:18.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.417+0000 7f0783fff640 1 -- 172.21.15.88:0/2674032278 >> v1:172.21.15.88:6789/0 conn(0x7f078415f200 legacy=0x7f078406d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:18.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.417+0000 7f0782ffd640 1 -- 172.21.15.88:0/2674032278 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2183037906 0 0) 0x7f076c003520 con 0x7f078415f200 2024-09-16T01:35:18.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.417+0000 7f0783fff640 1 -- 172.21.15.88:0/2674032278 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078406ce30 con 0x7f078415c2b0 2024-09-16T01:35:18.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.418+0000 7f0783fff640 1 --2- 172.21.15.88:0/2674032278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f078415c2b0 0x7f07840b6db0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f07740097a0 tx=0x7f077402a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7b8209233e6d96e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:18.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.418+0000 7f0782ffd640 1 -- 172.21.15.88:0/2674032278 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0774037070 con 0x7f078415c2b0 2024-09-16T01:35:18.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.419+0000 7f0782ffd640 1 -- 172.21.15.88:0/2674032278 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f077402ac50 con 0x7f078415c2b0 2024-09-16T01:35:18.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.419+0000 7f0782ffd640 1 -- 172.21.15.88:0/2674032278 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077c0019f0 con 0x7f078415c2b0 2024-09-16T01:35:18.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.419+0000 7f078a8e4640 1 -- 172.21.15.88:0/2674032278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f078415c2b0 msgr2=0x7f07840b6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:18.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.419+0000 7f078a8e4640 1 --2- 172.21.15.88:0/2674032278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f078415c2b0 0x7f07840b6db0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f07740097a0 tx=0x7f077402a1c0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.419+0000 7f078a8e4640 1 -- 172.21.15.88:0/2674032278 shutdown_connections 2024-09-16T01:35:18.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.419+0000 7f078a8e4640 1 --2- 172.21.15.88:0/2674032278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f078415c2b0 0x7f07840b6db0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.419+0000 7f078a8e4640 1 -- 172.21.15.88:0/2674032278 >> 172.21.15.88:0/2674032278 conn(0x7f07840b4240 msgr2=0x7f07840b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:18.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.419+0000 7f078a8e4640 1 -- 172.21.15.88:0/2674032278 shutdown_connections 2024-09-16T01:35:18.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.419+0000 7f078a8e4640 1 -- 172.21.15.88:0/2674032278 wait complete. 2024-09-16T01:35:18.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.420+0000 7f078a8e4640 1 Processor -- start 2024-09-16T01:35:18.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.420+0000 7f078a8e4640 1 -- start start 2024-09-16T01:35:18.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.421+0000 7f078a8e4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07840b72f0 0x7f07840d0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:18.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.421+0000 7f078a8e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f078415c2b0 0x7f078414ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:18.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.421+0000 7f078a8e4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f078415f200 0x7f07841f4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:18.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.421+0000 7f078a8e4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f07840b7760 con 0x7f078415f200 2024-09-16T01:35:18.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.421+0000 7f078a8e4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f078406d3f0 con 0x7f07840b72f0 2024-09-16T01:35:18.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.421+0000 7f078a8e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f078406e020 con 0x7f078415c2b0 2024-09-16T01:35:18.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.421+0000 7f0788e81640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f078415f200 0x7f07841f4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:18.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.421+0000 7f0788e81640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f078415f200 0x7f07841f4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33788/0 (socket says 172.21.15.88:33788) 2024-09-16T01:35:18.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.421+0000 7f0788e81640 1 -- 172.21.15.88:0/2379031432 learned_addr learned my addr 172.21.15.88:0/2379031432 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:18.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.422+0000 7f0783fff640 1 --2- 172.21.15.88:0/2379031432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07840b72f0 0x7f07840d0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:18.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.422+0000 7f07837fe640 1 --2- 172.21.15.88:0/2379031432 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f078415c2b0 0x7f078414ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:18.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.422+0000 7f0788e81640 1 -- 172.21.15.88:0/2379031432 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f078415c2b0 msgr2=0x7f078414ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:18.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.422+0000 7f0788e81640 1 --2- 172.21.15.88:0/2379031432 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f078415c2b0 0x7f078414ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.422+0000 7f0788e81640 1 -- 172.21.15.88:0/2379031432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07840b72f0 msgr2=0x7f07840d0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:18.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.422+0000 7f0788e81640 1 --2- 172.21.15.88:0/2379031432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07840b72f0 0x7f07840d0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.422+0000 7f0788e81640 1 -- 172.21.15.88:0/2379031432 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07840d0da0 con 0x7f078415f200 2024-09-16T01:35:18.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.422+0000 7f0788e81640 1 --2- 172.21.15.88:0/2379031432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f078415f200 0x7f07841f4250 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f077c00cc00 tx=0x7f077c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:18.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.422+0000 7f0780ff9640 1 -- 172.21.15.88:0/2379031432 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077c017070 con 0x7f078415f200 2024-09-16T01:35:18.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.422+0000 7f0780ff9640 1 -- 172.21.15.88:0/2379031432 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f077c00ce10 con 0x7f078415f200 2024-09-16T01:35:18.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.422+0000 7f078a8e4640 1 -- 172.21.15.88:0/2379031432 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f078414d3b0 con 0x7f078415f200 2024-09-16T01:35:18.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.422+0000 7f0780ff9640 1 -- 172.21.15.88:0/2379031432 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077c012600 con 0x7f078415f200 2024-09-16T01:35:18.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.422+0000 7f078a8e4640 1 -- 172.21.15.88:0/2379031432 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07840d1320 con 0x7f078415f200 2024-09-16T01:35:18.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.423+0000 7f078a8e4640 1 -- 172.21.15.88:0/2379031432 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0784068e10 con 0x7f078415f200 2024-09-16T01:35:18.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.423+0000 7f0780ff9640 1 -- 172.21.15.88:0/2379031432 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f077c01a070 con 0x7f078415f200 2024-09-16T01:35:18.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.424+0000 7f0780ff9640 1 --2- 172.21.15.88:0/2379031432 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f077006c2b0 0x7f077006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:18.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.424+0000 7f0780ff9640 1 -- 172.21.15.88:0/2379031432 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f077c08b2a0 con 0x7f078415f200 2024-09-16T01:35:18.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.424+0000 7f0783fff640 1 --2- 172.21.15.88:0/2379031432 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f077006c2b0 0x7f077006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:18.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.424+0000 7f0783fff640 1 --2- 172.21.15.88:0/2379031432 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f077006c2b0 0x7f077006e770 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0774034000 tx=0x7f077402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:18.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.428+0000 7f0780ff9640 1 -- 172.21.15.88:0/2379031432 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0784068e10 con 0x7f078415f200 2024-09-16T01:35:18.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.630+0000 7f078a8e4640 1 -- 172.21.15.88:0/2379031432 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f07840b6db0 con 0x7f078415f200 2024-09-16T01:35:18.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.631+0000 7f0780ff9640 1 -- 172.21.15.88:0/2379031432 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) ==== 53+0+73 (secure 0 0 0) 0x7f077c0592e0 con 0x7f078415f200 2024-09-16T01:35:18.632 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:35:18.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.633+0000 7f078a8e4640 1 -- 172.21.15.88:0/2379031432 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f077006c2b0 msgr2=0x7f077006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:18.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.633+0000 7f078a8e4640 1 --2- 172.21.15.88:0/2379031432 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f077006c2b0 0x7f077006e770 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0774034000 tx=0x7f077402e000 comp rx=0 tx=0).stop 2024-09-16T01:35:18.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.633+0000 7f078a8e4640 1 -- 172.21.15.88:0/2379031432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f078415f200 msgr2=0x7f07841f4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:18.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.633+0000 7f078a8e4640 1 --2- 172.21.15.88:0/2379031432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f078415f200 0x7f07841f4250 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f077c00cc00 tx=0x7f077c00a430 comp rx=0 tx=0).stop 2024-09-16T01:35:18.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.633+0000 7f078a8e4640 1 -- 172.21.15.88:0/2379031432 shutdown_connections 2024-09-16T01:35:18.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.633+0000 7f078a8e4640 1 --2- 172.21.15.88:0/2379031432 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f077006c2b0 0x7f077006e770 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.633+0000 7f078a8e4640 1 --2- 172.21.15.88:0/2379031432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f078415f200 0x7f07841f4250 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.633+0000 7f078a8e4640 1 --2- 172.21.15.88:0/2379031432 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f078415c2b0 0x7f078414ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.633+0000 7f078a8e4640 1 --2- 172.21.15.88:0/2379031432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07840b72f0 0x7f07840d0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:18.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.633+0000 7f078a8e4640 1 -- 172.21.15.88:0/2379031432 >> 172.21.15.88:0/2379031432 conn(0x7f07840b4240 msgr2=0x7f078406e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:18.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.633+0000 7f078a8e4640 1 -- 172.21.15.88:0/2379031432 shutdown_connections 2024-09-16T01:35:18.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18.633+0000 7f078a8e4640 1 -- 172.21.15.88:0/2379031432 wait complete. 2024-09-16T01:35:18.645 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-16T01:35:18.645 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:35:18.646 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2024-09-16T01:35:18.646 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T01:35:18.646 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T01:35:18.646 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-16T01:35:18.646 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-16T01:35:18.646 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:35:18.646 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:35:18.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:18.646 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2024-09-16T01:35:18.662 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:35:18.662 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:35:18.662 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:35:18.662 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:35:18.663 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-09-16T01:35:18.663 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:35:18.663 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:35:18.663 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:35:18.663 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2024-09-16T01:35:18.663 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:35:18.663 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:35:18.663 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2024-09-16T01:35:18.663 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:35:18.663 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2024-09-16T01:35:18.663 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:35:18.663 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2024-09-16T01:35:18.663 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:35:18.663 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42705sec preferred_lft 42705sec 2024-09-16T01:35:18.664 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link noprefixroute 2024-09-16T01:35:18.664 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:35:18.664 INFO:teuthology.orchestra.run.smithi088.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:35:18.664 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff 2024-09-16T01:35:18.664 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T01:35:18.664 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:35:18.664 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::e486:6eff:fe81:a1c/64 scope link 2024-09-16T01:35:18.664 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:35:18.664 INFO:teuthology.orchestra.run.smithi088.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:35:18.664 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 12:6f:29:be:95:48 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:35:18.664 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::106f:29ff:febe:9548/64 scope link 2024-09-16T01:35:18.664 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:35:18.664 INFO:teuthology.orchestra.run.smithi088.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:35:18.665 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T01:35:18.665 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::c63:b4ff:feb3:ad9e/64 scope link 2024-09-16T01:35:18.665 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:35:18.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:18.665 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:35:18.726 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-09-16T01:35:18.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:18.728 DEBUG:teuthology.orchestra.run.smithi088:> route 2024-09-16T01:35:18.785 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2024-09-16T01:35:18.785 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:35:18.785 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:35:18.785 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:35:18.785 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:35:18.785 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:35:18.785 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:35:18.785 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:35:18.785 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:35:18.785 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:35:18.786 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:35:18.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:35:18.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:35:18.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:18.911 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:35:18.926 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T01:35:18.926 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T01:35:18.927 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:35:18.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:18.927 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2024-09-16T01:35:18.948 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:35:18.946+0000 7f22afb75640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:35:18.971 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:35:18.972 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T01:35:19.027 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:35:19.027 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:35:19.027 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:35:19.082 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-16T01:35:19.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:19.082 DEBUG:teuthology.orchestra.run.smithi088:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:35:19.165 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:35:19.165 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:35:19.165 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:35:19.173 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:35:19.173 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:35:19.173 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:35:19.173 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:35:19.174 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:35:19.178 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:35:19.178 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:35:19.178 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:35:19.178 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:35:19.178 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:35:19.178 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790" 2024-09-16T01:35:19.179 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2024-09-16T01:35:19.179 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5ace2c91-46f9-482b-8a89-731fbdc171bf" will pass to kernel 2024-09-16T01:35:19.179 INFO:teuthology.orchestra.run.smithi088.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-16T01:35:19.179 INFO:teuthology.orchestra.run.smithi088.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:35:19.179 INFO:teuthology.orchestra.run.smithi088.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:35:19.179 INFO:teuthology.orchestra.run.smithi088.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:35:19.179 INFO:teuthology.orchestra.run.smithi088.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:35:19.181 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:35:19.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:19.181 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:35:19.246 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:35:19.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:35:19.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:35:19.342 INFO:teuthology.orchestra.run.smithi088.stdout:client5127 2024-09-16T01:35:19.343 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:35:19.343 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:35:19.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:19.358 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls subdir/) 2024-09-16T01:35:19.415 INFO:teuthology.orchestra.run.smithi088.stdout:file_undamaged 2024-09-16T01:35:19.415 DEBUG:teuthology.orchestra.run.smithi088:> stdin-killer -- python3 -c ' 2024-09-16T01:35:19.415 DEBUG:teuthology.orchestra.run.smithi088:> import os 2024-09-16T01:35:19.415 DEBUG:teuthology.orchestra.run.smithi088:> import stat 2024-09-16T01:35:19.415 DEBUG:teuthology.orchestra.run.smithi088:> import json 2024-09-16T01:35:19.416 DEBUG:teuthology.orchestra.run.smithi088:> import sys 2024-09-16T01:35:19.416 DEBUG:teuthology.orchestra.run.smithi088:> 2024-09-16T01:35:19.416 DEBUG:teuthology.orchestra.run.smithi088:> try: 2024-09-16T01:35:19.416 DEBUG:teuthology.orchestra.run.smithi088:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/file_to_be_damaged'"'"') 2024-09-16T01:35:19.416 DEBUG:teuthology.orchestra.run.smithi088:> except OSError as e: 2024-09-16T01:35:19.416 DEBUG:teuthology.orchestra.run.smithi088:> sys.exit(e.errno) 2024-09-16T01:35:19.416 DEBUG:teuthology.orchestra.run.smithi088:> 2024-09-16T01:35:19.416 DEBUG:teuthology.orchestra.run.smithi088:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-09-16T01:35:19.416 DEBUG:teuthology.orchestra.run.smithi088:> print(json.dumps( 2024-09-16T01:35:19.416 DEBUG:teuthology.orchestra.run.smithi088:> dict([(a, getattr(s, a)) for a in attrs]), 2024-09-16T01:35:19.416 DEBUG:teuthology.orchestra.run.smithi088:> indent=2)) 2024-09-16T01:35:19.416 DEBUG:teuthology.orchestra.run.smithi088:> ' 2024-09-16T01:35:19.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19 stdin-killer INFO: command exited with status 2: exiting normally with same code! 2024-09-16T01:35:19.530 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-16T01:35:19.530 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:35:19.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.668+0000 7f31e28af640 1 Processor -- start 2024-09-16T01:35:19.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.668+0000 7f31e28af640 1 -- start start 2024-09-16T01:35:19.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.668+0000 7f31e28af640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f31dc0d3310 0x7f31dc0d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:19.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.668+0000 7f31e28af640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31dc0d1650 0x7f31dc0b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:19.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.668+0000 7f31e28af640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31dc15ea40 con 0x7f31dc15e660 2024-09-16T01:35:19.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.668+0000 7f31e28af640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f31dc0d1a90 con 0x7f31dc0d3310 2024-09-16T01:35:19.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.668+0000 7f31e28af640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f31dc0d3870 con 0x7f31dc0d1650 2024-09-16T01:35:19.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.669+0000 7f31d3fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f31dc0d3310 0x7f31dc0d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:19.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.669+0000 7f31dbfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f31dc15e660 0x7f31dc0cf730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54558/0 (socket says 172.21.15.88:54558) 2024-09-16T01:35:19.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.669+0000 7f31d3fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f31dc0d3310 0x7f31dc0d1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33798/0 (socket says 172.21.15.88:33798) 2024-09-16T01:35:19.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.669+0000 7f31dbfff640 1 -- 172.21.15.88:0/3972447971 learned_addr learned my addr 172.21.15.88:0/3972447971 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:19.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.669+0000 7f31e0e4c640 1 --2- 172.21.15.88:0/3972447971 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31dc0d1650 0x7f31dc0b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:19.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.669+0000 7f31d3fff640 1 -- 172.21.15.88:0/3972447971 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31dc0d1650 msgr2=0x7f31dc0b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:19.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.669+0000 7f31d3fff640 1 --2- 172.21.15.88:0/3972447971 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31dc0d1650 0x7f31dc0b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:19.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.669+0000 7f31d3fff640 1 -- 172.21.15.88:0/3972447971 >> v1:172.21.15.88:6789/0 conn(0x7f31dc15e660 legacy=0x7f31dc0cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:19.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.669+0000 7f31d3fff640 1 -- 172.21.15.88:0/3972447971 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31dc0d3b50 con 0x7f31dc0d3310 2024-09-16T01:35:19.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.669+0000 7f31db7fe640 1 -- 172.21.15.88:0/3972447971 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2054905481 0 0) 0x7f31dc15ea40 con 0x7f31dc15e660 2024-09-16T01:35:19.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.670+0000 7f31d3fff640 1 --2- 172.21.15.88:0/3972447971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31dc0d3310 0x7f31dc0d1110 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f31c40097a0 tx=0x7f31c402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ce3bab02277f76 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:19.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.670+0000 7f31db7fe640 1 -- 172.21.15.88:0/3972447971 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c4037070 con 0x7f31dc0d3310 2024-09-16T01:35:19.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.670+0000 7f31db7fe640 1 -- 172.21.15.88:0/3972447971 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31c4032440 con 0x7f31dc0d3310 2024-09-16T01:35:19.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.670+0000 7f31db7fe640 1 -- 172.21.15.88:0/3972447971 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c4032810 con 0x7f31dc0d3310 2024-09-16T01:35:19.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.670+0000 7f31e28af640 1 -- 172.21.15.88:0/3972447971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31dc0d3310 msgr2=0x7f31dc0d1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:19.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.670+0000 7f31e28af640 1 --2- 172.21.15.88:0/3972447971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31dc0d3310 0x7f31dc0d1110 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f31c40097a0 tx=0x7f31c402a180 comp rx=0 tx=0).stop 2024-09-16T01:35:19.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.670+0000 7f31e28af640 1 -- 172.21.15.88:0/3972447971 shutdown_connections 2024-09-16T01:35:19.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.670+0000 7f31e28af640 1 --2- 172.21.15.88:0/3972447971 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31dc0d1650 0x7f31dc0b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:19.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.670+0000 7f31e28af640 1 --2- 172.21.15.88:0/3972447971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31dc0d3310 0x7f31dc0d1110 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:19.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.670+0000 7f31e28af640 1 -- 172.21.15.88:0/3972447971 >> 172.21.15.88:0/3972447971 conn(0x7f31dc0b6410 msgr2=0x7f31dc0b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:19.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.670+0000 7f31e28af640 1 -- 172.21.15.88:0/3972447971 shutdown_connections 2024-09-16T01:35:19.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.671+0000 7f31e28af640 1 -- 172.21.15.88:0/3972447971 wait complete. 2024-09-16T01:35:19.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.672+0000 7f31e28af640 1 Processor -- start 2024-09-16T01:35:19.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.672+0000 7f31e28af640 1 -- start start 2024-09-16T01:35:19.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.672+0000 7f31e28af640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31dc0d1650 0x7f31dc14fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:19.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.672+0000 7f31e28af640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31dc0d3310 0x7f31dc0ab090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:19.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.672+0000 7f31e28af640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31dc15e660 0x7f31dc0ab5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:19.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.672+0000 7f31e28af640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f31dc0d43f0 con 0x7f31dc0d1650 2024-09-16T01:35:19.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.672+0000 7f31e28af640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f31dc0d4110 con 0x7f31dc15e660 2024-09-16T01:35:19.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.672+0000 7f31e28af640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f31dc0b80b0 con 0x7f31dc0d3310 2024-09-16T01:35:19.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.672+0000 7f31dbfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31dc0d1650 0x7f31dc14fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:19.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.672+0000 7f31dbfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31dc0d1650 0x7f31dc14fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33808/0 (socket says 172.21.15.88:33808) 2024-09-16T01:35:19.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.672+0000 7f31dbfff640 1 -- 172.21.15.88:0/1639915249 learned_addr learned my addr 172.21.15.88:0/1639915249 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:19.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.673+0000 7f31d3fff640 1 --2- 172.21.15.88:0/1639915249 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31dc0d3310 0x7f31dc0ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:19.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.673+0000 7f31e0e4c640 1 --2- 172.21.15.88:0/1639915249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31dc15e660 0x7f31dc0ab5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:19.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.673+0000 7f31dbfff640 1 -- 172.21.15.88:0/1639915249 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31dc0d3310 msgr2=0x7f31dc0ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:19.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.673+0000 7f31dbfff640 1 --2- 172.21.15.88:0/1639915249 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31dc0d3310 0x7f31dc0ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:19.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.673+0000 7f31dbfff640 1 -- 172.21.15.88:0/1639915249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31dc15e660 msgr2=0x7f31dc0ab5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:19.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.673+0000 7f31dbfff640 1 --2- 172.21.15.88:0/1639915249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31dc15e660 0x7f31dc0ab5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:19.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.673+0000 7f31dbfff640 1 -- 172.21.15.88:0/1639915249 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31dc0cd7d0 con 0x7f31dc0d1650 2024-09-16T01:35:19.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.673+0000 7f31dbfff640 1 --2- 172.21.15.88:0/1639915249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31dc0d1650 0x7f31dc14fac0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00c460 tx=0x7f31cc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:19.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.673+0000 7f31d97fa640 1 -- 172.21.15.88:0/1639915249 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc018070 con 0x7f31dc0d1650 2024-09-16T01:35:19.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.673+0000 7f31d97fa640 1 -- 172.21.15.88:0/1639915249 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31cc007500 con 0x7f31dc0d1650 2024-09-16T01:35:19.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.673+0000 7f31d97fa640 1 -- 172.21.15.88:0/1639915249 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc004a20 con 0x7f31dc0d1650 2024-09-16T01:35:19.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.673+0000 7f31e28af640 1 -- 172.21.15.88:0/1639915249 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31dc0cdab0 con 0x7f31dc0d1650 2024-09-16T01:35:19.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.673+0000 7f31e28af640 1 -- 172.21.15.88:0/1639915249 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31dc0ce070 con 0x7f31dc0d1650 2024-09-16T01:35:19.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.674+0000 7f31e28af640 1 -- 172.21.15.88:0/1639915249 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31dc0b7b70 con 0x7f31dc0d1650 2024-09-16T01:35:19.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.675+0000 7f31d97fa640 1 -- 172.21.15.88:0/1639915249 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f31cc01b070 con 0x7f31dc0d1650 2024-09-16T01:35:19.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.675+0000 7f31d97fa640 1 --2- 172.21.15.88:0/1639915249 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f31bc06c2b0 0x7f31bc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:19.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.675+0000 7f31d97fa640 1 -- 172.21.15.88:0/1639915249 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f31cc08a6b0 con 0x7f31dc0d1650 2024-09-16T01:35:19.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.675+0000 7f31d3fff640 1 --2- 172.21.15.88:0/1639915249 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f31bc06c2b0 0x7f31bc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:19.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.676+0000 7f31d3fff640 1 --2- 172.21.15.88:0/1639915249 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f31bc06c2b0 0x7f31bc06e770 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f31c4034000 tx=0x7f31c402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:19.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.680+0000 7f31d97fa640 1 -- 172.21.15.88:0/1639915249 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f31cc058e50 con 0x7f31dc0d1650 2024-09-16T01:35:19.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.880+0000 7f31e28af640 1 -- 172.21.15.88:0/1639915249 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31dc15dba0 con 0x7f31dc0d1650 2024-09-16T01:35:19.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.881+0000 7f31d97fa640 1 -- 172.21.15.88:0/1639915249 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5230 (secure 0 0 0) 0x7f31cc058750 con 0x7f31dc0d1650 2024-09-16T01:35:19.883 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:35:19.883 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":19,"btime":"2024-09-16T01:35:16:950413+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1005331674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1005331674},{"type":"v1","addr":"172.21.15.88:6837","nonce":1005331674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/3427247662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3427247662},{"type":"v1","addr":"172.21.15.154:6834","nonce":3427247662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/4059835337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":4059835337},{"type":"v1","addr":"172.21.15.154:6835","nonce":4059835337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:35:00.674725+0000","modified":"2024-09-16T01:35:16.950408+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"c","rank":0,"incarnation":16,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6835/2870334334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870334334},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870334334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5067,"qdb_cluster":[5067]},"id":2}]} 2024-09-16T01:35:19.883 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 19 2024-09-16T01:35:19.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.883+0000 7f31e28af640 1 -- 172.21.15.88:0/1639915249 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f31bc06c2b0 msgr2=0x7f31bc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:19.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.883+0000 7f31e28af640 1 --2- 172.21.15.88:0/1639915249 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f31bc06c2b0 0x7f31bc06e770 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f31c4034000 tx=0x7f31c402e000 comp rx=0 tx=0).stop 2024-09-16T01:35:19.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.883+0000 7f31e28af640 1 -- 172.21.15.88:0/1639915249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31dc0d1650 msgr2=0x7f31dc14fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:19.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.883+0000 7f31e28af640 1 --2- 172.21.15.88:0/1639915249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31dc0d1650 0x7f31dc14fac0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00c460 tx=0x7f31cc00c920 comp rx=0 tx=0).stop 2024-09-16T01:35:19.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.883+0000 7f31e28af640 1 -- 172.21.15.88:0/1639915249 shutdown_connections 2024-09-16T01:35:19.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.883+0000 7f31e28af640 1 --2- 172.21.15.88:0/1639915249 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f31bc06c2b0 0x7f31bc06e770 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:19.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.883+0000 7f31e28af640 1 --2- 172.21.15.88:0/1639915249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31dc15e660 0x7f31dc0ab5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:19.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.883+0000 7f31e28af640 1 --2- 172.21.15.88:0/1639915249 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31dc0d3310 0x7f31dc0ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:19.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.883+0000 7f31e28af640 1 --2- 172.21.15.88:0/1639915249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31dc0d1650 0x7f31dc14fac0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:19.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.883+0000 7f31e28af640 1 -- 172.21.15.88:0/1639915249 >> 172.21.15.88:0/1639915249 conn(0x7f31dc0b6410 msgr2=0x7f31dc15bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:19.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.883+0000 7f31e28af640 1 -- 172.21.15.88:0/1639915249 shutdown_connections 2024-09-16T01:35:19.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.883+0000 7f31e28af640 1 -- 172.21.15.88:0/1639915249 wait complete. 2024-09-16T01:35:19.896 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage ls --format=json-pretty 2024-09-16T01:35:19.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.996+0000 7efc0dcba640 1 Processor -- start 2024-09-16T01:35:19.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.996+0000 7efc0dcba640 1 -- start start 2024-09-16T01:35:20.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.996+0000 7efc0dcba640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efc0815c290 0x7efc0806d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.996+0000 7efc0dcba640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc0815d680 0x7efc0806f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.996+0000 7efc0dcba640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc0806cc20 con 0x7efc0806fc50 2024-09-16T01:35:20.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.996+0000 7efc0dcba640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7efc080700c0 con 0x7efc0815c290 2024-09-16T01:35:20.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.996+0000 7efc0dcba640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efc0806bc10 con 0x7efc0815d680 2024-09-16T01:35:20.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.997+0000 7efc077fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efc0815c290 0x7efc0806d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.997+0000 7efc077fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efc0815c290 0x7efc0806d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33810/0 (socket says 172.21.15.88:33810) 2024-09-16T01:35:20.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.997+0000 7efc077fe640 1 -- 172.21.15.88:0/3565241606 learned_addr learned my addr 172.21.15.88:0/3565241606 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:20.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.997+0000 7efc06ffd640 1 --2- 172.21.15.88:0/3565241606 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc0815d680 0x7efc0806f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.997+0000 7efc077fe640 1 -- 172.21.15.88:0/3565241606 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc0815d680 msgr2=0x7efc0806f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:20.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.997+0000 7efc077fe640 1 --2- 172.21.15.88:0/3565241606 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc0815d680 0x7efc0806f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.997+0000 7efc077fe640 1 -- 172.21.15.88:0/3565241606 >> v1:172.21.15.154:6789/0 conn(0x7efc0806fc50 legacy=0x7efc0815ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:20.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.997+0000 7efc077fe640 1 -- 172.21.15.88:0/3565241606 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc0806be60 con 0x7efc0815c290 2024-09-16T01:35:20.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.998+0000 7efc077fe640 1 --2- 172.21.15.88:0/3565241606 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc0815c290 0x7efc0806d500 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7efbf8009870 tx=0x7efbf802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7d9be8571721c74 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:20.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.998+0000 7efc067fc640 1 -- 172.21.15.88:0/3565241606 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf8037070 con 0x7efc0815c290 2024-09-16T01:35:20.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.998+0000 7efc067fc640 1 -- 172.21.15.88:0/3565241606 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbf8032440 con 0x7efc0815c290 2024-09-16T01:35:20.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.998+0000 7efc067fc640 1 -- 172.21.15.88:0/3565241606 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf8032810 con 0x7efc0815c290 2024-09-16T01:35:20.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.998+0000 7efc0dcba640 1 -- 172.21.15.88:0/3565241606 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc0815c290 msgr2=0x7efc0806d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:20.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.998+0000 7efc0dcba640 1 --2- 172.21.15.88:0/3565241606 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc0815c290 0x7efc0806d500 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7efbf8009870 tx=0x7efbf802a270 comp rx=0 tx=0).stop 2024-09-16T01:35:20.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.998+0000 7efc0dcba640 1 -- 172.21.15.88:0/3565241606 shutdown_connections 2024-09-16T01:35:20.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.998+0000 7efc0dcba640 1 --2- 172.21.15.88:0/3565241606 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc0815d680 0x7efc0806f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.998+0000 7efc0dcba640 1 --2- 172.21.15.88:0/3565241606 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc0815c290 0x7efc0806d500 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.998+0000 7efc0dcba640 1 -- 172.21.15.88:0/3565241606 >> 172.21.15.88:0/3565241606 conn(0x7efc080b3070 msgr2=0x7efc080b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:20.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.998+0000 7efc0dcba640 1 -- 172.21.15.88:0/3565241606 shutdown_connections 2024-09-16T01:35:20.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:19.998+0000 7efc0dcba640 1 -- 172.21.15.88:0/3565241606 wait complete. 2024-09-16T01:35:20.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.000+0000 7efc0dcba640 1 Processor -- start 2024-09-16T01:35:20.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.000+0000 7efc0dcba640 1 -- start start 2024-09-16T01:35:20.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.000+0000 7efc0dcba640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc0806fc50 0x7efc081599b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.000+0000 7efc0dcba640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc0815c290 0x7efc0814d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.000+0000 7efc0dcba640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc0815d680 0x7efc08073100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.000+0000 7efc0dcba640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7efc0806c700 con 0x7efc0815d680 2024-09-16T01:35:20.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.000+0000 7efc0dcba640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efc0806c420 con 0x7efc0815c290 2024-09-16T01:35:20.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.000+0000 7efc0dcba640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efc0806da40 con 0x7efc0806fc50 2024-09-16T01:35:20.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.001+0000 7efc07fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc0815d680 0x7efc08073100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.001+0000 7efc077fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc0806fc50 0x7efc081599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.001+0000 7efc07fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc0815d680 0x7efc08073100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33816/0 (socket says 172.21.15.88:33816) 2024-09-16T01:35:20.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.001+0000 7efc07fff640 1 -- 172.21.15.88:0/151801069 learned_addr learned my addr 172.21.15.88:0/151801069 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:20.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.001+0000 7efc06ffd640 1 --2- 172.21.15.88:0/151801069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc0815c290 0x7efc0814d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.001+0000 7efc07fff640 1 -- 172.21.15.88:0/151801069 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc0806fc50 msgr2=0x7efc081599b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:20.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.001+0000 7efc07fff640 1 --2- 172.21.15.88:0/151801069 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc0806fc50 0x7efc081599b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.001+0000 7efc07fff640 1 -- 172.21.15.88:0/151801069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc0815c290 msgr2=0x7efc0814d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:20.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.001+0000 7efc07fff640 1 --2- 172.21.15.88:0/151801069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc0815c290 0x7efc0814d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.001+0000 7efc07fff640 1 -- 172.21.15.88:0/151801069 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc0814e050 con 0x7efc0815d680 2024-09-16T01:35:20.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.001+0000 7efc07fff640 1 --2- 172.21.15.88:0/151801069 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc0815d680 0x7efc08073100 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7efc0000c3c0 tx=0x7efc0000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:20.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.001+0000 7efbeffff640 1 -- 172.21.15.88:0/151801069 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc00017070 con 0x7efc0815d680 2024-09-16T01:35:20.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.001+0000 7efbeffff640 1 -- 172.21.15.88:0/151801069 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc000052d0 con 0x7efc0815d680 2024-09-16T01:35:20.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.001+0000 7efc0dcba640 1 -- 172.21.15.88:0/151801069 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc0815a070 con 0x7efc0815d680 2024-09-16T01:35:20.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.001+0000 7efbeffff640 1 -- 172.21.15.88:0/151801069 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc00003810 con 0x7efc0815d680 2024-09-16T01:35:20.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.001+0000 7efc0dcba640 1 -- 172.21.15.88:0/151801069 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc0815a630 con 0x7efc0815d680 2024-09-16T01:35:20.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.003+0000 7efbeffff640 1 -- 172.21.15.88:0/151801069 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7efc0001a070 con 0x7efc0815d680 2024-09-16T01:35:20.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.003+0000 7efbeffff640 1 --2- 172.21.15.88:0/151801069 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efbf406c2b0 0x7efbf406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.003+0000 7efbeffff640 1 -- 172.21.15.88:0/151801069 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7efc0008a770 con 0x7efc0815d680 2024-09-16T01:35:20.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.003+0000 7efc077fe640 1 --2- 172.21.15.88:0/151801069 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efbf406c2b0 0x7efbf406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.004+0000 7efc077fe640 1 --2- 172.21.15.88:0/151801069 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efbf406c2b0 0x7efbf406e770 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efbf8034000 tx=0x7efbf802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:20.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.004+0000 7efc10a57740 1 Processor -- start 2024-09-16T01:35:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.004+0000 7efc10a57740 1 -- start start 2024-09-16T01:35:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.005+0000 7efc10a57740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564353211f00 0x5643532122e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.005+0000 7efc10a57740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56435320f6c0 0x564353211250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.005+0000 7efc10a57740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56435320fb20 0x5643531f1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.005+0000 7efc10a57740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56435312e930 con 0x56435320fb20 2024-09-16T01:35:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.005+0000 7efc10a57740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5643530d6570 con 0x564353211f00 2024-09-16T01:35:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.005+0000 7efc10a57740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56435312a8f0 con 0x56435320f6c0 2024-09-16T01:35:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.005+0000 7efc07fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56435320fb20 0x5643531f1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.005+0000 7efc07fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56435320fb20 0x5643531f1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33824/0 (socket says 172.21.15.88:33824) 2024-09-16T01:35:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.005+0000 7efc07fff640 1 -- 172.21.15.88:0/3516408243 learned_addr learned my addr 172.21.15.88:0/3516408243 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.005+0000 7efc077fe640 1 --2- 172.21.15.88:0/3516408243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564353211f00 0x5643532122e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.005+0000 7efc06ffd640 1 --2- 172.21.15.88:0/3516408243 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56435320f6c0 0x564353211250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.005+0000 7efc07fff640 1 -- 172.21.15.88:0/3516408243 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56435320f6c0 msgr2=0x564353211250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.005+0000 7efc07fff640 1 --2- 172.21.15.88:0/3516408243 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56435320f6c0 0x564353211250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.005+0000 7efc07fff640 1 -- 172.21.15.88:0/3516408243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564353211f00 msgr2=0x5643532122e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.005+0000 7efc07fff640 1 --2- 172.21.15.88:0/3516408243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564353211f00 0x5643532122e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.005+0000 7efc07fff640 1 -- 172.21.15.88:0/3516408243 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643530eb200 con 0x56435320fb20 2024-09-16T01:35:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.005+0000 7efc06ffd640 1 --2- 172.21.15.88:0/3516408243 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56435320f6c0 0x564353211250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.005+0000 7efc077fe640 1 --2- 172.21.15.88:0/3516408243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564353211f00 0x5643532122e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.005+0000 7efc07fff640 1 --2- 172.21.15.88:0/3516408243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56435320fb20 0x5643531f1e80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7efc0000c390 tx=0x7efc000055c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f243d10ebbd83e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.006+0000 7efbed7fa640 1 -- 172.21.15.88:0/3516408243 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc00061360 con 0x56435320fb20 2024-09-16T01:35:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.006+0000 7efbed7fa640 1 -- 172.21.15.88:0/3516408243 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc00056aa0 con 0x56435320fb20 2024-09-16T01:35:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.006+0000 7efbed7fa640 1 -- 172.21.15.88:0/3516408243 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc00057bf0 con 0x56435320fb20 2024-09-16T01:35:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.006+0000 7efc10a57740 1 -- 172.21.15.88:0/3516408243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56435320fb20 msgr2=0x5643531f1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.006+0000 7efc10a57740 1 --2- 172.21.15.88:0/3516408243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56435320fb20 0x5643531f1e80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7efc0000c390 tx=0x7efc000055c0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.006+0000 7efc10a57740 1 -- 172.21.15.88:0/3516408243 shutdown_connections 2024-09-16T01:35:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.006+0000 7efc10a57740 1 --2- 172.21.15.88:0/3516408243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56435320fb20 0x5643531f1e80 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.006+0000 7efc10a57740 1 --2- 172.21.15.88:0/3516408243 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56435320f6c0 0x564353211250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.006+0000 7efc10a57740 1 --2- 172.21.15.88:0/3516408243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564353211f00 0x5643532122e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.006+0000 7efc10a57740 1 -- 172.21.15.88:0/3516408243 >> 172.21.15.88:0/3516408243 conn(0x5643531c7df0 msgr2=0x56435320ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:20.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.006+0000 7efc10a57740 1 -- 172.21.15.88:0/3516408243 shutdown_connections 2024-09-16T01:35:20.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.006+0000 7efc10a57740 1 -- 172.21.15.88:0/3516408243 wait complete. 2024-09-16T01:35:20.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.007+0000 7efc10a57740 1 -- start start 2024-09-16T01:35:20.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.007+0000 7efc10a57740 1 Processor -- start 2024-09-16T01:35:20.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.008+0000 7efc10a57740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56435320fb20 0x5643531ef7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.008+0000 7efc10a57740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564353211f00 0x564353208710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.008+0000 7efc10a57740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564353208c50 0x564353206e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.008+0000 7efc10a57740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56435320e620 con 0x564353208c50 2024-09-16T01:35:20.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.008+0000 7efc10a57740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5643530b76a0 con 0x56435320fb20 2024-09-16T01:35:20.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.008+0000 7efc10a57740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56435320e900 con 0x564353211f00 2024-09-16T01:35:20.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.008+0000 7efc10a57740 10 client.0 _ll_register_callbacks cb 0x5643530ef300 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:35:20.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.008+0000 7efc07fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564353208c50 0x564353206e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.008+0000 7efc07fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564353208c50 0x564353206e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33834/0 (socket says 172.21.15.88:33834) 2024-09-16T01:35:20.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.008+0000 7efc07fff640 1 -- 172.21.15.88:0/235422512 learned_addr learned my addr 172.21.15.88:0/235422512 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:20.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.008+0000 7efc06ffd640 1 --2- 172.21.15.88:0/235422512 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564353211f00 0x564353208710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.008+0000 7efc077fe640 1 --2- 172.21.15.88:0/235422512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56435320fb20 0x5643531ef7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.008+0000 7efc07fff640 1 -- 172.21.15.88:0/235422512 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564353211f00 msgr2=0x564353208710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:20.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.008+0000 7efc07fff640 1 --2- 172.21.15.88:0/235422512 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564353211f00 0x564353208710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.008+0000 7efc07fff640 1 -- 172.21.15.88:0/235422512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56435320fb20 msgr2=0x5643531ef7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:20.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.008+0000 7efc07fff640 1 --2- 172.21.15.88:0/235422512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56435320fb20 0x5643531ef7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.008+0000 7efc07fff640 1 -- 172.21.15.88:0/235422512 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564352f0d7c0 con 0x564353208c50 2024-09-16T01:35:20.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.008+0000 7efc077fe640 1 --2- 172.21.15.88:0/235422512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56435320fb20 0x5643531ef7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:20.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.008+0000 7efc07fff640 1 --2- 172.21.15.88:0/235422512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564353208c50 0x564353206e10 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7efc000584a0 tx=0x7efc00000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8de3dc231d85e834 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:20.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.008+0000 7efbe67fc640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:35:20.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.008+0000 7efbe67fc640 1 -- 172.21.15.88:0/235422512 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc00057df0 con 0x564353208c50 2024-09-16T01:35:20.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.008+0000 7efbe67fc640 1 -- 172.21.15.88:0/235422512 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7efc000544f0 con 0x564353208c50 2024-09-16T01:35:20.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.008+0000 7efc10a57740 1 -- 172.21.15.88:0/235422512 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc00056aa0 con 0x564353208c50 2024-09-16T01:35:20.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.009+0000 7efbe67fc640 1 -- 172.21.15.88:0/235422512 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7efc000546f0 con 0x564353208c50 2024-09-16T01:35:20.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.009+0000 7efc10a57740 10 client.5148 fetch_fsmap learned FSMap version 19 2024-09-16T01:35:20.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.009+0000 7efc10a57740 1 -- 172.21.15.88:0/235422512 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x564352f6df50 con 0x564353208c50 2024-09-16T01:35:20.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.009+0000 7efbe67fc640 1 -- 172.21.15.88:0/235422512 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc00056aa0 con 0x564353208c50 2024-09-16T01:35:20.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.009+0000 7efbe67fc640 1 -- 172.21.15.88:0/235422512 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 19) ==== 3502+0+0 (secure 0 0 0) 0x7efc00077b20 con 0x564353208c50 2024-09-16T01:35:20.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.009+0000 7efc10a57740 10 client.5148 fetch_fsmap finished waiting for FSMap version 19 2024-09-16T01:35:20.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.009+0000 7efc10a57740 10 client.5148 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T01:35:20.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.009+0000 7efc10a57740 20 client.5148 populate_metadata read hostname 'smithi088' 2024-09-16T01:35:20.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.009+0000 7efc10a57740 1 --2- 172.21.15.88:0/235422512 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x5643532244a0 0x5643531f3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.009+0000 7efc10a57740 4 client.5148 mds_command: new command op to 5067 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:35:20.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.009+0000 7efc10a57740 1 -- 172.21.15.88:0/235422512 --> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56435313c060 con 0x5643532244a0 2024-09-16T01:35:20.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.009+0000 7efc077fe640 1 --2- 172.21.15.88:0/235422512 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x5643532244a0 0x5643531f3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.009+0000 7efbe67fc640 10 client.5148 ms_handle_connect on v2:172.21.15.88:6834/2870334334 2024-09-16T01:35:20.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.009+0000 7efc077fe640 1 --2- 172.21.15.88:0/235422512 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x5643532244a0 0x5643531f3960 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85b0069738438152 server_cookie=5acf2caa43c1582 in_seq=0 out_seq=0 2024-09-16T01:35:20.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.010+0000 7efbe67fc640 1 -- 172.21.15.88:0/235422512 <== mds.0 v2:172.21.15.88:6834/2870334334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56435313c060 con 0x5643532244a0 2024-09-16T01:35:20.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.010+0000 7efbe67fc640 10 client.5148 handle_command_reply: tid=0 2024-09-16T01:35:20.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.010+0000 7efc10a57740 1 client.5148 shutdown 2024-09-16T01:35:20.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.010+0000 7efc10a57740 1 -- 172.21.15.88:0/235422512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564353208c50 msgr2=0x564353206e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:20.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.010+0000 7efc10a57740 1 --2- 172.21.15.88:0/235422512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564353208c50 0x564353206e10 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7efc000584a0 tx=0x7efc00000ed0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.010+0000 7efc10a57740 1 -- 172.21.15.88:0/235422512 shutdown_connections 2024-09-16T01:35:20.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.010+0000 7efc10a57740 1 --2- 172.21.15.88:0/235422512 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x5643532244a0 0x5643531f3960 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.010+0000 7efc10a57740 1 --2- 172.21.15.88:0/235422512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564353208c50 0x564353206e10 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.010+0000 7efbe67fc640 0 client.5148 ms_handle_reset on v2:172.21.15.88:6834/2870334334 2024-09-16T01:35:20.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.010+0000 7efc10a57740 1 --2- 172.21.15.88:0/235422512 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564353211f00 0x564353208710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.010+0000 7efc10a57740 1 --2- 172.21.15.88:0/235422512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56435320fb20 0x5643531ef7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.010+0000 7efc10a57740 1 -- 172.21.15.88:0/235422512 >> 172.21.15.88:0/235422512 conn(0x56435320f6c0 msgr2=0x5643531f2470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:20.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.010+0000 7efc10a57740 1 -- 172.21.15.88:0/235422512 shutdown_connections 2024-09-16T01:35:20.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.010+0000 7efc10a57740 1 -- 172.21.15.88:0/235422512 wait complete. 2024-09-16T01:35:20.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.010+0000 7efc10a57740 20 client.5148 trim_cache size 0 max 16384 2024-09-16T01:35:20.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.023+0000 7efc10a57740 1 Processor -- start 2024-09-16T01:35:20.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.023+0000 7efc10a57740 1 -- start start 2024-09-16T01:35:20.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.024+0000 7efc10a57740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564353208c50 0x564353207ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.024+0000 7efc10a57740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56435320fb20 0x564353205150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.024+0000 7efc10a57740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564353211f00 0x564353205690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.024+0000 7efc10a57740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7efc081fdb90 con 0x564353208c50 2024-09-16T01:35:20.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.024+0000 7efc10a57740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efc080b2c40 con 0x56435320fb20 2024-09-16T01:35:20.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.024+0000 7efc10a57740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efc081523c0 con 0x564353211f00 2024-09-16T01:35:20.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.024+0000 7efc077fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564353208c50 0x564353207ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.024+0000 7efc077fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564353208c50 0x564353207ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33842/0 (socket says 172.21.15.88:33842) 2024-09-16T01:35:20.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.024+0000 7efc077fe640 1 -- 172.21.15.88:0/2740444166 learned_addr learned my addr 172.21.15.88:0/2740444166 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:20.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.024+0000 7efc077fe640 1 -- 172.21.15.88:0/2740444166 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564353211f00 msgr2=0x564353205690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:20.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.024+0000 7efc07fff640 1 --2- 172.21.15.88:0/2740444166 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564353211f00 0x564353205690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.025+0000 7efc06ffd640 1 --2- 172.21.15.88:0/2740444166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56435320fb20 0x564353205150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.025+0000 7efc077fe640 1 --2- 172.21.15.88:0/2740444166 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564353211f00 0x564353205690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.025+0000 7efc077fe640 1 -- 172.21.15.88:0/2740444166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56435320fb20 msgr2=0x564353205150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:20.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.025+0000 7efc077fe640 1 --2- 172.21.15.88:0/2740444166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56435320fb20 0x564353205150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.025+0000 7efc077fe640 1 -- 172.21.15.88:0/2740444166 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564353207180 con 0x564353208c50 2024-09-16T01:35:20.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.025+0000 7efc07fff640 1 --2- 172.21.15.88:0/2740444166 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564353211f00 0x564353205690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:20.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.025+0000 7efc06ffd640 1 --2- 172.21.15.88:0/2740444166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56435320fb20 0x564353205150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:20.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.025+0000 7efc077fe640 1 --2- 172.21.15.88:0/2740444166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564353208c50 0x564353207ff0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7efbf8009870 tx=0x7efbf802c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f7b5a9ff909dd42 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:20.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.025+0000 7efbecff9640 1 -- 172.21.15.88:0/2740444166 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf8058070 con 0x564353208c50 2024-09-16T01:35:20.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.025+0000 7efbecff9640 1 -- 172.21.15.88:0/2740444166 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbf8032d40 con 0x564353208c50 2024-09-16T01:35:20.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.025+0000 7efbecff9640 1 -- 172.21.15.88:0/2740444166 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf8037040 con 0x564353208c50 2024-09-16T01:35:20.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.025+0000 7efc10a57740 1 -- 172.21.15.88:0/2740444166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564353208c50 msgr2=0x564353207ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:20.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.025+0000 7efc10a57740 1 --2- 172.21.15.88:0/2740444166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564353208c50 0x564353207ff0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7efbf8009870 tx=0x7efbf802c000 comp rx=0 tx=0).stop 2024-09-16T01:35:20.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.026+0000 7efc10a57740 1 -- 172.21.15.88:0/2740444166 shutdown_connections 2024-09-16T01:35:20.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.026+0000 7efc10a57740 1 --2- 172.21.15.88:0/2740444166 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564353211f00 0x564353205690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.026+0000 7efc10a57740 1 --2- 172.21.15.88:0/2740444166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56435320fb20 0x564353205150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.026+0000 7efc10a57740 1 --2- 172.21.15.88:0/2740444166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564353208c50 0x564353207ff0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.026+0000 7efc10a57740 1 -- 172.21.15.88:0/2740444166 >> 172.21.15.88:0/2740444166 conn(0x56435320f6c0 msgr2=0x564353206310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:20.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.026+0000 7efc10a57740 1 -- 172.21.15.88:0/2740444166 shutdown_connections 2024-09-16T01:35:20.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.026+0000 7efc10a57740 1 -- 172.21.15.88:0/2740444166 wait complete. 2024-09-16T01:35:20.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.026+0000 7efc10a57740 1 -- start start 2024-09-16T01:35:20.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.026+0000 7efc10a57740 1 Processor -- start 2024-09-16T01:35:20.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.027+0000 7efc10a57740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564353208c50 0x564353207ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.027+0000 7efc10a57740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56435320fb20 0x564353208530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.027+0000 7efc10a57740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564353211f00 0x56435328d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.027+0000 7efc10a57740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56435320e620 con 0x564353211f00 2024-09-16T01:35:20.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.027+0000 7efc10a57740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56435320e900 con 0x564353208c50 2024-09-16T01:35:20.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.027+0000 7efc10a57740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5643530b76a0 con 0x56435320fb20 2024-09-16T01:35:20.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.027+0000 7efc10a57740 10 client.0 _ll_register_callbacks cb 0x7efc0006f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:35:20.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.027+0000 7efc07fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564353211f00 0x56435328d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.027+0000 7efc07fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564353211f00 0x56435328d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33848/0 (socket says 172.21.15.88:33848) 2024-09-16T01:35:20.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.027+0000 7efc07fff640 1 -- 172.21.15.88:0/3117879759 learned_addr learned my addr 172.21.15.88:0/3117879759 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:20.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.027+0000 7efc077fe640 1 --2- 172.21.15.88:0/3117879759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564353208c50 0x564353207ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.027+0000 7efc06ffd640 1 --2- 172.21.15.88:0/3117879759 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56435320fb20 0x564353208530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.027+0000 7efc07fff640 1 -- 172.21.15.88:0/3117879759 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56435320fb20 msgr2=0x564353208530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:20.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.027+0000 7efc07fff640 1 --2- 172.21.15.88:0/3117879759 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56435320fb20 0x564353208530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.027+0000 7efc07fff640 1 -- 172.21.15.88:0/3117879759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564353208c50 msgr2=0x564353207ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:20.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.027+0000 7efc07fff640 1 --2- 172.21.15.88:0/3117879759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564353208c50 0x564353207ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.027+0000 7efc07fff640 1 -- 172.21.15.88:0/3117879759 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564352e010b0 con 0x564353211f00 2024-09-16T01:35:20.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.028+0000 7efc07fff640 1 --2- 172.21.15.88:0/3117879759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564353211f00 0x56435328d900 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7efc00000fb0 tx=0x7efc0001e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48954556dda29f8d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:20.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.028+0000 7efbe67fc640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:35:20.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.028+0000 7efbe67fc640 1 -- 172.21.15.88:0/3117879759 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc00088b80 con 0x564353211f00 2024-09-16T01:35:20.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.028+0000 7efbe67fc640 1 -- 172.21.15.88:0/3117879759 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7efc00057660 con 0x564353211f00 2024-09-16T01:35:20.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.028+0000 7efc10a57740 1 -- 172.21.15.88:0/3117879759 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efbf8032d40 con 0x564353211f00 2024-09-16T01:35:20.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.028+0000 7efbe67fc640 1 -- 172.21.15.88:0/3117879759 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7efc00057860 con 0x564353211f00 2024-09-16T01:35:20.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.028+0000 7efc10a57740 10 client.5154 fetch_fsmap learned FSMap version 19 2024-09-16T01:35:20.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.028+0000 7efc10a57740 1 -- 172.21.15.88:0/3117879759 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x564352e010b0 con 0x564353211f00 2024-09-16T01:35:20.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.028+0000 7efbe67fc640 1 -- 172.21.15.88:0/3117879759 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbf8032d40 con 0x564353211f00 2024-09-16T01:35:20.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.028+0000 7efbe67fc640 1 -- 172.21.15.88:0/3117879759 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 19) ==== 3502+0+0 (secure 0 0 0) 0x7efc0005dc30 con 0x564353211f00 2024-09-16T01:35:20.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.028+0000 7efc10a57740 10 client.5154 fetch_fsmap finished waiting for FSMap version 19 2024-09-16T01:35:20.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.028+0000 7efc10a57740 10 client.5154 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T01:35:20.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.028+0000 7efc10a57740 20 client.5154 populate_metadata read hostname 'smithi088' 2024-09-16T01:35:20.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.028+0000 7efc10a57740 1 --2- 172.21.15.88:0/3117879759 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x564353213920 0x564353213d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.028+0000 7efc10a57740 4 client.5154 mds_command: new command op to 5067 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-09-16T01:35:20.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.028+0000 7efc10a57740 1 -- 172.21.15.88:0/3117879759 --> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x5643531363b0 con 0x564353213920 2024-09-16T01:35:20.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.028+0000 7efc077fe640 1 --2- 172.21.15.88:0/3117879759 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x564353213920 0x564353213d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.028+0000 7efbe67fc640 10 client.5154 ms_handle_connect on v2:172.21.15.88:6834/2870334334 2024-09-16T01:35:20.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.029+0000 7efc077fe640 1 --2- 172.21.15.88:0/3117879759 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x564353213920 0x564353213d00 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e545ff6e17f4dd1c server_cookie=f6210d110f414884 in_seq=0 out_seq=0 2024-09-16T01:35:20.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.029+0000 7efbe67fc640 1 -- 172.21.15.88:0/3117879759 <== mds.0 v2:172.21.15.88:6834/2870334334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+237 (crc 0 0 0) 0x5643531363b0 con 0x564353213920 2024-09-16T01:35:20.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.029+0000 7efbe67fc640 10 client.5154 handle_command_reply: tid=0 2024-09-16T01:35:20.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.029+0000 7efc10a57740 1 client.5154 shutdown 2024-09-16T01:35:20.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.029+0000 7efc10a57740 1 -- 172.21.15.88:0/3117879759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564353211f00 msgr2=0x56435328d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:20.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.029+0000 7efc10a57740 1 --2- 172.21.15.88:0/3117879759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564353211f00 0x56435328d900 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7efc00000fb0 tx=0x7efc0001e000 comp rx=0 tx=0).stop 2024-09-16T01:35:20.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.029+0000 7efc10a57740 1 -- 172.21.15.88:0/3117879759 shutdown_connections 2024-09-16T01:35:20.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.029+0000 7efc10a57740 1 --2- 172.21.15.88:0/3117879759 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x564353213920 0x564353213d00 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.029+0000 7efc10a57740 1 --2- 172.21.15.88:0/3117879759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564353211f00 0x56435328d900 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.029+0000 7efc10a57740 1 --2- 172.21.15.88:0/3117879759 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56435320fb20 0x564353208530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.029+0000 7efbe67fc640 0 client.5154 ms_handle_reset on v2:172.21.15.88:6834/2870334334 2024-09-16T01:35:20.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.029+0000 7efc10a57740 1 --2- 172.21.15.88:0/3117879759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564353208c50 0x564353207ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.029+0000 7efc10a57740 1 -- 172.21.15.88:0/3117879759 >> 172.21.15.88:0/3117879759 conn(0x56435320f6c0 msgr2=0x564353209410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:20.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.029+0000 7efc10a57740 1 -- 172.21.15.88:0/3117879759 shutdown_connections 2024-09-16T01:35:20.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.029+0000 7efc10a57740 1 -- 172.21.15.88:0/3117879759 wait complete. 2024-09-16T01:35:20.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.029+0000 7efc10a57740 20 client.5154 trim_cache size 0 max 16384 2024-09-16T01:35:20.039 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:35:20.039 INFO:teuthology.orchestra.run.smithi088.stdout:[ 2024-09-16T01:35:20.039 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:35:20.039 INFO:teuthology.orchestra.run.smithi088.stdout: "damage_type": "dentry", 2024-09-16T01:35:20.039 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 3136708976, 2024-09-16T01:35:20.039 INFO:teuthology.orchestra.run.smithi088.stdout: "ino": 1099511627776, 2024-09-16T01:35:20.039 INFO:teuthology.orchestra.run.smithi088.stdout: "frag": "*", 2024-09-16T01:35:20.039 INFO:teuthology.orchestra.run.smithi088.stdout: "dname": "file_to_be_damaged", 2024-09-16T01:35:20.039 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_id": "head", 2024-09-16T01:35:20.039 INFO:teuthology.orchestra.run.smithi088.stdout: "path": "/subdir/file_to_be_damaged" 2024-09-16T01:35:20.039 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:35:20.039 INFO:teuthology.orchestra.run.smithi088.stdout:] 2024-09-16T01:35:20.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.029+0000 7efc0dcba640 1 -- 172.21.15.88:0/151801069 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efbf406c2b0 msgr2=0x7efbf406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:20.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.029+0000 7efc0dcba640 1 --2- 172.21.15.88:0/151801069 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efbf406c2b0 0x7efbf406e770 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efbf8034000 tx=0x7efbf802e000 comp rx=0 tx=0).stop 2024-09-16T01:35:20.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.029+0000 7efc0dcba640 1 -- 172.21.15.88:0/151801069 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc0815d680 msgr2=0x7efc08073100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:20.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.029+0000 7efc0dcba640 1 --2- 172.21.15.88:0/151801069 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc0815d680 0x7efc08073100 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7efc0000c3c0 tx=0x7efc0000c880 comp rx=0 tx=0).stop 2024-09-16T01:35:20.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.030+0000 7efc0dcba640 1 -- 172.21.15.88:0/151801069 shutdown_connections 2024-09-16T01:35:20.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.030+0000 7efc0dcba640 1 --2- 172.21.15.88:0/151801069 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efbf406c2b0 0x7efbf406e770 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.030+0000 7efc0dcba640 1 --2- 172.21.15.88:0/151801069 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc0815d680 0x7efc08073100 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.030+0000 7efc0dcba640 1 --2- 172.21.15.88:0/151801069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc0815c290 0x7efc0814d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.030+0000 7efc0dcba640 1 --2- 172.21.15.88:0/151801069 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efc0806fc50 0x7efc081599b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.030+0000 7efc0dcba640 1 -- 172.21.15.88:0/151801069 >> 172.21.15.88:0/151801069 conn(0x7efc080b3070 msgr2=0x7efc0815e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:20.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.030+0000 7efc0dcba640 1 -- 172.21.15.88:0/151801069 shutdown_connections 2024-09-16T01:35:20.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.030+0000 7efc0dcba640 1 -- 172.21.15.88:0/151801069 wait complete. 2024-09-16T01:35:20.042 DEBUG:teuthology.orchestra.run.smithi088:> stdin-killer -- python3 -c ' 2024-09-16T01:35:20.043 DEBUG:teuthology.orchestra.run.smithi088:> import sys 2024-09-16T01:35:20.043 DEBUG:teuthology.orchestra.run.smithi088:> import errno 2024-09-16T01:35:20.043 DEBUG:teuthology.orchestra.run.smithi088:> 2024-09-16T01:35:20.043 DEBUG:teuthology.orchestra.run.smithi088:> try: 2024-09-16T01:35:20.043 DEBUG:teuthology.orchestra.run.smithi088:> f = open("/home/ubuntu/cephtest/mnt.0/subdir/file_to_be_damaged", "w") 2024-09-16T01:35:20.043 DEBUG:teuthology.orchestra.run.smithi088:> f.close() 2024-09-16T01:35:20.043 DEBUG:teuthology.orchestra.run.smithi088:> except IOError as e: 2024-09-16T01:35:20.043 DEBUG:teuthology.orchestra.run.smithi088:> sys.exit(errno.EIO) 2024-09-16T01:35:20.043 DEBUG:teuthology.orchestra.run.smithi088:> ' 2024-09-16T01:35:20.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20 stdin-killer INFO: command exited with status 5: exiting normally with same code! 2024-09-16T01:35:20.116 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-09-16T01:35:20.116 DEBUG:teuthology.orchestra.run.smithi088:> stdin-killer -- python3 -c ' 2024-09-16T01:35:20.116 DEBUG:teuthology.orchestra.run.smithi088:> import os 2024-09-16T01:35:20.116 DEBUG:teuthology.orchestra.run.smithi088:> import stat 2024-09-16T01:35:20.117 DEBUG:teuthology.orchestra.run.smithi088:> import json 2024-09-16T01:35:20.117 DEBUG:teuthology.orchestra.run.smithi088:> import sys 2024-09-16T01:35:20.117 DEBUG:teuthology.orchestra.run.smithi088:> 2024-09-16T01:35:20.117 DEBUG:teuthology.orchestra.run.smithi088:> try: 2024-09-16T01:35:20.117 DEBUG:teuthology.orchestra.run.smithi088:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/file_to_be_damaged'"'"') 2024-09-16T01:35:20.117 DEBUG:teuthology.orchestra.run.smithi088:> except OSError as e: 2024-09-16T01:35:20.117 DEBUG:teuthology.orchestra.run.smithi088:> sys.exit(e.errno) 2024-09-16T01:35:20.117 DEBUG:teuthology.orchestra.run.smithi088:> 2024-09-16T01:35:20.117 DEBUG:teuthology.orchestra.run.smithi088:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-09-16T01:35:20.117 DEBUG:teuthology.orchestra.run.smithi088:> print(json.dumps( 2024-09-16T01:35:20.117 DEBUG:teuthology.orchestra.run.smithi088:> dict([(a, getattr(s, a)) for a in attrs]), 2024-09-16T01:35:20.117 DEBUG:teuthology.orchestra.run.smithi088:> indent=2)) 2024-09-16T01:35:20.117 DEBUG:teuthology.orchestra.run.smithi088:> ' 2024-09-16T01:35:20.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20 stdin-killer INFO: command exited with status 5: exiting normally with same code! 2024-09-16T01:35:20.230 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-09-16T01:35:20.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:20.231 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.files ./subdir) 2024-09-16T01:35:20.287 INFO:teuthology.orchestra.run.smithi088.stdout:2 2024-09-16T01:35:20.287 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:35:20.288 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:35:20.342 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:35:20.342 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:35:20.399 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:35:20.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.537+0000 7fd96506e640 1 Processor -- start 2024-09-16T01:35:20.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.537+0000 7fd96506e640 1 -- start start 2024-09-16T01:35:20.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.538+0000 7fd96506e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd96015c2a0 0x7fd9600735d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.538+0000 7fd96506e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd960074120 con 0x7fd96006fe20 2024-09-16T01:35:20.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.538+0000 7fd96506e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd96015ef00 con 0x7fd960073b10 2024-09-16T01:35:20.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.538+0000 7fd96506e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd96015c680 con 0x7fd96015c2a0 2024-09-16T01:35:20.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.538+0000 7fd95f59e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd96006fe20 0x7fd96006bc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55908/0 (socket says 172.21.15.88:55908) 2024-09-16T01:35:20.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.538+0000 7fd95f59e640 1 -- 172.21.15.88:0/3733903217 learned_addr learned my addr 172.21.15.88:0/3733903217 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:20.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.538+0000 7fd95e59c640 1 -- 172.21.15.88:0/3733903217 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2042588195 0 0) 0x7fd960074120 con 0x7fd96006fe20 2024-09-16T01:35:20.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.538+0000 7fd95e59c640 1 -- 172.21.15.88:0/3733903217 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd94c003520 con 0x7fd96006fe20 2024-09-16T01:35:20.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.538+0000 7fd95ed9d640 1 --2- 172.21.15.88:0/3733903217 >> v2:172.21.15.154:3300/0 conn(0x7fd96015c2a0 0x7fd9600735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.538+0000 7fd95e59c640 1 -- 172.21.15.88:0/3733903217 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 721181264 0 0) 0x7fd94c003520 con 0x7fd96006fe20 2024-09-16T01:35:20.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.538+0000 7fd95e59c640 1 -- 172.21.15.88:0/3733903217 >> v2:172.21.15.154:3300/0 conn(0x7fd96015c2a0 msgr2=0x7fd9600735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:20.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.538+0000 7fd95e59c640 1 --2- 172.21.15.88:0/3733903217 >> v2:172.21.15.154:3300/0 conn(0x7fd96015c2a0 0x7fd9600735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.538+0000 7fd95e59c640 1 -- 172.21.15.88:0/3733903217 >> v1:172.21.15.154:6789/0 conn(0x7fd960073b10 legacy=0x7fd96006f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:20.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.538+0000 7fd95e59c640 1 -- 172.21.15.88:0/3733903217 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd96015f1e0 con 0x7fd96006fe20 2024-09-16T01:35:20.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.538+0000 7fd95e59c640 1 -- 172.21.15.88:0/3733903217 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fd9580019f0 con 0x7fd96006fe20 2024-09-16T01:35:20.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.538+0000 7fd95e59c640 1 -- 172.21.15.88:0/3733903217 >> v1:172.21.15.88:6789/0 conn(0x7fd96006fe20 legacy=0x7fd96006bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:20.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.538+0000 7fd95e59c640 1 --2- 172.21.15.88:0/3733903217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd94c0039b0 0x7fd94c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.538+0000 7fd95e59c640 1 --2- 172.21.15.88:0/3733903217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd94c004800 0x7fd94c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.539+0000 7fd95e59c640 1 --2- 172.21.15.88:0/3733903217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd94c0051c0 0x7fd94c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.539+0000 7fd95e59c640 1 -- 172.21.15.88:0/3733903217 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd94c00c0c0 con 0x7fd94c0051c0 2024-09-16T01:35:20.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.539+0000 7fd95e59c640 1 -- 172.21.15.88:0/3733903217 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd94c00c270 con 0x7fd94c0039b0 2024-09-16T01:35:20.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.539+0000 7fd95e59c640 1 -- 172.21.15.88:0/3733903217 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd94c00c470 con 0x7fd94c004800 2024-09-16T01:35:20.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.539+0000 7fd95f59e640 1 --2- 172.21.15.88:0/3733903217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd94c0051c0 0x7fd94c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.539+0000 7fd95ed9d640 1 --2- 172.21.15.88:0/3733903217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd94c0039b0 0x7fd94c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.539+0000 7fd95659c640 1 --2- 172.21.15.88:0/3733903217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd94c004800 0x7fd94c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.539+0000 7fd95f59e640 1 -- 172.21.15.88:0/3733903217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd94c004800 msgr2=0x7fd94c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:20.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.539+0000 7fd95f59e640 1 --2- 172.21.15.88:0/3733903217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd94c004800 0x7fd94c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.539+0000 7fd95f59e640 1 -- 172.21.15.88:0/3733903217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd94c0039b0 msgr2=0x7fd94c003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:20.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.539+0000 7fd95f59e640 1 --2- 172.21.15.88:0/3733903217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd94c0039b0 0x7fd94c003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.539+0000 7fd95f59e640 1 -- 172.21.15.88:0/3733903217 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd94c003520 con 0x7fd94c0051c0 2024-09-16T01:35:20.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.539+0000 7fd95659c640 1 --2- 172.21.15.88:0/3733903217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd94c004800 0x7fd94c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:20.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.539+0000 7fd95f59e640 1 -- 172.21.15.88:0/3733903217 reap_dead start 2024-09-16T01:35:20.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.540+0000 7fd95f59e640 1 --2- 172.21.15.88:0/3733903217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd94c0051c0 0x7fd94c00b9c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd960162140 tx=0x7fd94c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3250d720697487ce server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:20.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.540+0000 7fd95e59c640 1 -- 172.21.15.88:0/3733903217 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95800a4a0 con 0x7fd94c0051c0 2024-09-16T01:35:20.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.540+0000 7fd95e59c640 1 -- 172.21.15.88:0/3733903217 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd95802f070 con 0x7fd94c0051c0 2024-09-16T01:35:20.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.540+0000 7fd95e59c640 1 -- 172.21.15.88:0/3733903217 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95802bc50 con 0x7fd94c0051c0 2024-09-16T01:35:20.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.540+0000 7fd96506e640 1 -- 172.21.15.88:0/3733903217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd94c0051c0 msgr2=0x7fd94c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:20.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.540+0000 7fd96506e640 1 --2- 172.21.15.88:0/3733903217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd94c0051c0 0x7fd94c00b9c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd960162140 tx=0x7fd94c005830 comp rx=0 tx=0).stop 2024-09-16T01:35:20.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.540+0000 7fd96506e640 1 -- 172.21.15.88:0/3733903217 shutdown_connections 2024-09-16T01:35:20.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.540+0000 7fd96506e640 1 --2- 172.21.15.88:0/3733903217 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd94c0051c0 0x7fd94c00b9c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.540+0000 7fd96506e640 1 -- 172.21.15.88:0/3733903217 >> 172.21.15.88:0/3733903217 conn(0x7fd9600b3070 msgr2=0x7fd9600b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:20.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.540+0000 7fd96506e640 1 -- 172.21.15.88:0/3733903217 shutdown_connections 2024-09-16T01:35:20.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.540+0000 7fd96506e640 1 -- 172.21.15.88:0/3733903217 wait complete. 2024-09-16T01:35:20.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.542+0000 7fd96506e640 1 Processor -- start 2024-09-16T01:35:20.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.542+0000 7fd96506e640 1 -- start start 2024-09-16T01:35:20.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.543+0000 7fd96506e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd94c0051c0 0x7fd960161050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.543+0000 7fd96506e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd960159e60 0x7fd96014efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.543+0000 7fd96506e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9600ceab0 0x7fd96015a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.543+0000 7fd96506e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd9600b5f20 con 0x7fd960159e60 2024-09-16T01:35:20.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.543+0000 7fd96506e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd96015ebf0 con 0x7fd94c0051c0 2024-09-16T01:35:20.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.543+0000 7fd96506e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd9600b60a0 con 0x7fd9600ceab0 2024-09-16T01:35:20.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.543+0000 7fd95659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd960159e60 0x7fd96014efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.543+0000 7fd95659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd960159e60 0x7fd96014efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47828/0 (socket says 172.21.15.88:47828) 2024-09-16T01:35:20.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.543+0000 7fd95659c640 1 -- 172.21.15.88:0/662089256 learned_addr learned my addr 172.21.15.88:0/662089256 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:20.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.543+0000 7fd95ed9d640 1 --2- 172.21.15.88:0/662089256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd94c0051c0 0x7fd960161050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.543+0000 7fd95f59e640 1 --2- 172.21.15.88:0/662089256 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9600ceab0 0x7fd96015a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.543+0000 7fd95659c640 1 -- 172.21.15.88:0/662089256 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9600ceab0 msgr2=0x7fd96015a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:20.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.543+0000 7fd95659c640 1 --2- 172.21.15.88:0/662089256 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9600ceab0 0x7fd96015a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.543+0000 7fd95659c640 1 -- 172.21.15.88:0/662089256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd94c0051c0 msgr2=0x7fd960161050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:20.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.543+0000 7fd95659c640 1 --2- 172.21.15.88:0/662089256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd94c0051c0 0x7fd960161050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.543+0000 7fd95659c640 1 -- 172.21.15.88:0/662089256 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd96015a670 con 0x7fd960159e60 2024-09-16T01:35:20.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.543+0000 7fd95659c640 1 --2- 172.21.15.88:0/662089256 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd960159e60 0x7fd96014efd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd94800e490 tx=0x7fd94800e950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:20.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.543+0000 7fd957fff640 1 -- 172.21.15.88:0/662089256 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd948007020 con 0x7fd960159e60 2024-09-16T01:35:20.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.543+0000 7fd957fff640 1 -- 172.21.15.88:0/662089256 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9480071c0 con 0x7fd960159e60 2024-09-16T01:35:20.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.543+0000 7fd96506e640 1 -- 172.21.15.88:0/662089256 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd96006c110 con 0x7fd960159e60 2024-09-16T01:35:20.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.544+0000 7fd957fff640 1 -- 172.21.15.88:0/662089256 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9480073b0 con 0x7fd960159e60 2024-09-16T01:35:20.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.544+0000 7fd96506e640 1 -- 172.21.15.88:0/662089256 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd96006c6d0 con 0x7fd960159e60 2024-09-16T01:35:20.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.544+0000 7fd96506e640 1 -- 172.21.15.88:0/662089256 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd96015f9e0 con 0x7fd960159e60 2024-09-16T01:35:20.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.545+0000 7fd957fff640 1 -- 172.21.15.88:0/662089256 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd94801b020 con 0x7fd960159e60 2024-09-16T01:35:20.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.545+0000 7fd957fff640 1 --2- 172.21.15.88:0/662089256 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd94006c2e0 0x7fd94006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.545+0000 7fd95ed9d640 1 --2- 172.21.15.88:0/662089256 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd94006c2e0 0x7fd94006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.545+0000 7fd957fff640 1 -- 172.21.15.88:0/662089256 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fd94808acb0 con 0x7fd960159e60 2024-09-16T01:35:20.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.546+0000 7fd95ed9d640 1 --2- 172.21.15.88:0/662089256 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd94006c2e0 0x7fd94006e7a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd950006fd0 tx=0x7fd950008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:20.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.550+0000 7fd957fff640 1 -- 172.21.15.88:0/662089256 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd948058e00 con 0x7fd960159e60 2024-09-16T01:35:20.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.726+0000 7fd96506e640 1 -- 172.21.15.88:0/662089256 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd9600b5790 con 0x7fd960159e60 2024-09-16T01:35:20.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.726+0000 7fd957fff640 1 -- 172.21.15.88:0/662089256 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v33) ==== 80+0+122 (secure 0 0 0) 0x7fd94805d100 con 0x7fd960159e60 2024-09-16T01:35:20.728 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/4230443772 2024-09-17T01:35:12.937737+0000 2024-09-16T01:35:20.728 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/4230443772 2024-09-17T01:35:12.937737+0000 2024-09-16T01:35:20.728 INFO:teuthology.orchestra.run.smithi088.stderr:listed 2 entries 2024-09-16T01:35:20.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.728+0000 7fd96506e640 1 -- 172.21.15.88:0/662089256 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd94006c2e0 msgr2=0x7fd94006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:20.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.728+0000 7fd96506e640 1 --2- 172.21.15.88:0/662089256 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd94006c2e0 0x7fd94006e7a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd950006fd0 tx=0x7fd950008040 comp rx=0 tx=0).stop 2024-09-16T01:35:20.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.728+0000 7fd96506e640 1 -- 172.21.15.88:0/662089256 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd960159e60 msgr2=0x7fd96014efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:20.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.728+0000 7fd96506e640 1 --2- 172.21.15.88:0/662089256 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd960159e60 0x7fd96014efd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd94800e490 tx=0x7fd94800e950 comp rx=0 tx=0).stop 2024-09-16T01:35:20.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.728+0000 7fd96506e640 1 -- 172.21.15.88:0/662089256 shutdown_connections 2024-09-16T01:35:20.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.728+0000 7fd96506e640 1 --2- 172.21.15.88:0/662089256 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd94006c2e0 0x7fd94006e7a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.728+0000 7fd96506e640 1 --2- 172.21.15.88:0/662089256 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9600ceab0 0x7fd96015a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.728+0000 7fd96506e640 1 --2- 172.21.15.88:0/662089256 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd960159e60 0x7fd96014efd0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.728+0000 7fd96506e640 1 --2- 172.21.15.88:0/662089256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd94c0051c0 0x7fd960161050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.728+0000 7fd96506e640 1 -- 172.21.15.88:0/662089256 >> 172.21.15.88:0/662089256 conn(0x7fd9600b3070 msgr2=0x7fd9600b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:20.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.728+0000 7fd96506e640 1 -- 172.21.15.88:0/662089256 shutdown_connections 2024-09-16T01:35:20.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.728+0000 7fd96506e640 1 -- 172.21.15.88:0/662089256 wait complete. 2024-09-16T01:35:20.741 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-16T01:35:20.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:20.741 DEBUG:teuthology.orchestra.run.smithi088:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-16T01:35:20.768 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:35:20.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:20.768 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:35:20.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:20.824 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.2:0 scrub start /subdir repair 2024-09-16T01:35:20.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.966+0000 7f28c5d98640 1 Processor -- start 2024-09-16T01:35:20.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.966+0000 7f28c5d98640 1 -- start start 2024-09-16T01:35:20.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.967+0000 7f28c5d98640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28c015e570 0x7f28c00b5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.967+0000 7f28c5d98640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f28c00714a0 0x7f28c015e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.967+0000 7f28c5d98640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28c015f020 con 0x7f28c006f400 2024-09-16T01:35:20.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.967+0000 7f28c5d98640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f28c015fc30 con 0x7f28c00714a0 2024-09-16T01:35:20.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.967+0000 7f28c5d98640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f28c015f300 con 0x7f28c015e570 2024-09-16T01:35:20.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.967+0000 7f28bf7fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28c015e570 0x7f28c00b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.967+0000 7f28beffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f28c00714a0 0x7f28c015e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.967+0000 7f28beffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f28c00714a0 0x7f28c015e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:36986/0 (socket says 172.21.15.88:36986) 2024-09-16T01:35:20.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.967+0000 7f28beffd640 1 -- 172.21.15.88:0/1570365845 learned_addr learned my addr 172.21.15.88:0/1570365845 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:20.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.967+0000 7f28beffd640 1 -- 172.21.15.88:0/1570365845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28c015e570 msgr2=0x7f28c00b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:20.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.967+0000 7f28beffd640 1 --2- 172.21.15.88:0/1570365845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28c015e570 0x7f28c00b5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.967+0000 7f28beffd640 1 -- 172.21.15.88:0/1570365845 >> v1:172.21.15.154:6789/0 conn(0x7f28c006f400 legacy=0x7f28c0073160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:20.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.967+0000 7f28beffd640 1 -- 172.21.15.88:0/1570365845 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c015f5e0 con 0x7f28c00714a0 2024-09-16T01:35:20.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.967+0000 7f28bf7fe640 1 --2- 172.21.15.88:0/1570365845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28c015e570 0x7f28c00b5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:35:20.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.968+0000 7f28beffd640 1 --2- 172.21.15.88:0/1570365845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28c00714a0 0x7f28c015e950 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f28b4009870 tx=0x7f28b402a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a3d3b76cb222680 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:20.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.968+0000 7f28be7fc640 1 -- 172.21.15.88:0/1570365845 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b4037070 con 0x7f28c00714a0 2024-09-16T01:35:20.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.968+0000 7f28be7fc640 1 -- 172.21.15.88:0/1570365845 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28b4032440 con 0x7f28c00714a0 2024-09-16T01:35:20.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.968+0000 7f28be7fc640 1 -- 172.21.15.88:0/1570365845 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b4032810 con 0x7f28c00714a0 2024-09-16T01:35:20.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.969+0000 7f28c5d98640 1 -- 172.21.15.88:0/1570365845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28c00714a0 msgr2=0x7f28c015e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:20.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.969+0000 7f28c5d98640 1 --2- 172.21.15.88:0/1570365845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28c00714a0 0x7f28c015e950 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f28b4009870 tx=0x7f28b402a270 comp rx=0 tx=0).stop 2024-09-16T01:35:20.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.969+0000 7f28c5d98640 1 -- 172.21.15.88:0/1570365845 shutdown_connections 2024-09-16T01:35:20.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.969+0000 7f28c5d98640 1 --2- 172.21.15.88:0/1570365845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28c00714a0 0x7f28c015e950 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.969+0000 7f28c5d98640 1 --2- 172.21.15.88:0/1570365845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28c015e570 0x7f28c00b5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.969+0000 7f28c5d98640 1 -- 172.21.15.88:0/1570365845 >> 172.21.15.88:0/1570365845 conn(0x7f28c00b4240 msgr2=0x7f28c00b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:20.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.969+0000 7f28c5d98640 1 -- 172.21.15.88:0/1570365845 shutdown_connections 2024-09-16T01:35:20.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.969+0000 7f28c5d98640 1 -- 172.21.15.88:0/1570365845 wait complete. 2024-09-16T01:35:20.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.970+0000 7f28c5d98640 1 Processor -- start 2024-09-16T01:35:20.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.970+0000 7f28c5d98640 1 -- start start 2024-09-16T01:35:20.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.971+0000 7f28c5d98640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28c006f400 0x7f28c0158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.971+0000 7f28c5d98640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28c00714a0 0x7f28c00ccdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.971+0000 7f28c5d98640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28c015e570 0x7f28c014bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.971+0000 7f28c5d98640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f28c006e0f0 con 0x7f28c00714a0 2024-09-16T01:35:20.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.971+0000 7f28c5d98640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f28c006df70 con 0x7f28c015e570 2024-09-16T01:35:20.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.971+0000 7f28c5d98640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f28c006e270 con 0x7f28c006f400 2024-09-16T01:35:20.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.971+0000 7f28beffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28c00714a0 0x7f28c00ccdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.971+0000 7f28beffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28c00714a0 0x7f28c00ccdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47832/0 (socket says 172.21.15.88:47832) 2024-09-16T01:35:20.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.971+0000 7f28beffd640 1 -- 172.21.15.88:0/3692055654 learned_addr learned my addr 172.21.15.88:0/3692055654 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:20.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.971+0000 7f28bf7fe640 1 --2- 172.21.15.88:0/3692055654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28c006f400 0x7f28c0158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.971+0000 7f28bffff640 1 --2- 172.21.15.88:0/3692055654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28c015e570 0x7f28c014bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.971+0000 7f28beffd640 1 -- 172.21.15.88:0/3692055654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28c006f400 msgr2=0x7f28c0158a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:20.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.971+0000 7f28beffd640 1 --2- 172.21.15.88:0/3692055654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28c006f400 0x7f28c0158a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.971+0000 7f28beffd640 1 -- 172.21.15.88:0/3692055654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28c015e570 msgr2=0x7f28c014bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:20.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.971+0000 7f28beffd640 1 --2- 172.21.15.88:0/3692055654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28c015e570 0x7f28c014bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.971+0000 7f28beffd640 1 -- 172.21.15.88:0/3692055654 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c00cd300 con 0x7f28c00714a0 2024-09-16T01:35:20.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.971+0000 7f28bf7fe640 1 --2- 172.21.15.88:0/3692055654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28c006f400 0x7f28c0158a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:20.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.971+0000 7f28bffff640 1 --2- 172.21.15.88:0/3692055654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28c015e570 0x7f28c014bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:20.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.971+0000 7f28beffd640 1 --2- 172.21.15.88:0/3692055654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28c00714a0 0x7f28c00ccdc0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f28b4000c00 tx=0x7f28b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:20.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.971+0000 7f28a7fff640 1 -- 172.21.15.88:0/3692055654 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b4037040 con 0x7f28c00714a0 2024-09-16T01:35:20.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.972+0000 7f28a7fff640 1 -- 172.21.15.88:0/3692055654 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28b4042070 con 0x7f28c00714a0 2024-09-16T01:35:20.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.972+0000 7f28a7fff640 1 -- 172.21.15.88:0/3692055654 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b4030880 con 0x7f28c00714a0 2024-09-16T01:35:20.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.972+0000 7f28c5d98640 1 -- 172.21.15.88:0/3692055654 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28c014c490 con 0x7f28c00714a0 2024-09-16T01:35:20.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.972+0000 7f28c5d98640 1 -- 172.21.15.88:0/3692055654 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28c00cd860 con 0x7f28c00714a0 2024-09-16T01:35:20.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.973+0000 7f28a7fff640 1 -- 172.21.15.88:0/3692055654 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f28b4030a20 con 0x7f28c00714a0 2024-09-16T01:35:20.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.973+0000 7f28a7fff640 1 --2- 172.21.15.88:0/3692055654 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f28ac06c2b0 0x7f28ac06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.974+0000 7f28a7fff640 1 -- 172.21.15.88:0/3692055654 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f28b40abd20 con 0x7f28c00714a0 2024-09-16T01:35:20.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.974+0000 7f28bf7fe640 1 --2- 172.21.15.88:0/3692055654 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f28ac06c2b0 0x7f28ac06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.974+0000 7f28bf7fe640 1 --2- 172.21.15.88:0/3692055654 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f28ac06c2b0 0x7f28ac06e770 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f28b0009ac0 tx=0x7f28b0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:20.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.974+0000 7f28c8c11740 1 Processor -- start 2024-09-16T01:35:20.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.974+0000 7f28c8c11740 1 -- start start 2024-09-16T01:35:20.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.975+0000 7f28c8c11740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624a79a8500 0x5624a79a88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.975+0000 7f28c8c11740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5624a79a5cc0 0x5624a79a7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.975+0000 7f28c8c11740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a79a6120 0x5624a7988480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.975+0000 7f28c8c11740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5624a78c4ac0 con 0x5624a79a6120 2024-09-16T01:35:20.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.975+0000 7f28c8c11740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5624a786c7b0 con 0x5624a79a8500 2024-09-16T01:35:20.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.975+0000 7f28c8c11740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5624a78c0a80 con 0x5624a79a5cc0 2024-09-16T01:35:20.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.975+0000 7f28bffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a79a6120 0x5624a7988480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.975+0000 7f28bffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a79a6120 0x5624a7988480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47836/0 (socket says 172.21.15.88:47836) 2024-09-16T01:35:20.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.976+0000 7f28bffff640 1 -- 172.21.15.88:0/1396720790 learned_addr learned my addr 172.21.15.88:0/1396720790 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:20.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.976+0000 7f28beffd640 1 --2- 172.21.15.88:0/1396720790 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5624a79a5cc0 0x5624a79a7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.976+0000 7f28bf7fe640 1 --2- 172.21.15.88:0/1396720790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624a79a8500 0x5624a79a88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.976+0000 7f28bffff640 1 -- 172.21.15.88:0/1396720790 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5624a79a5cc0 msgr2=0x5624a79a7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:20.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.976+0000 7f28bffff640 1 --2- 172.21.15.88:0/1396720790 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5624a79a5cc0 0x5624a79a7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.976+0000 7f28bffff640 1 -- 172.21.15.88:0/1396720790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624a79a8500 msgr2=0x5624a79a88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:20.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.976+0000 7f28bffff640 1 --2- 172.21.15.88:0/1396720790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624a79a8500 0x5624a79a88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.976+0000 7f28bffff640 1 -- 172.21.15.88:0/1396720790 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624a76a3a20 con 0x5624a79a6120 2024-09-16T01:35:20.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.976+0000 7f28bffff640 1 --2- 172.21.15.88:0/1396720790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a79a6120 0x5624a7988480 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f28b8001d90 tx=0x7f28b800d150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d6dfb03c93dfaff server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:20.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.976+0000 7f28a57fa640 1 -- 172.21.15.88:0/1396720790 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b8016020 con 0x5624a79a6120 2024-09-16T01:35:20.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.976+0000 7f28a57fa640 1 -- 172.21.15.88:0/1396720790 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28b800dd90 con 0x5624a79a6120 2024-09-16T01:35:20.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.976+0000 7f28a57fa640 1 -- 172.21.15.88:0/1396720790 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b80125f0 con 0x5624a79a6120 2024-09-16T01:35:20.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.977+0000 7f28c8c11740 1 -- 172.21.15.88:0/1396720790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a79a6120 msgr2=0x5624a7988480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:20.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.977+0000 7f28c8c11740 1 --2- 172.21.15.88:0/1396720790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a79a6120 0x5624a7988480 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f28b8001d90 tx=0x7f28b800d150 comp rx=0 tx=0).stop 2024-09-16T01:35:20.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.977+0000 7f28c8c11740 1 -- 172.21.15.88:0/1396720790 shutdown_connections 2024-09-16T01:35:20.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.977+0000 7f28c8c11740 1 --2- 172.21.15.88:0/1396720790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a79a6120 0x5624a7988480 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.977+0000 7f28c8c11740 1 --2- 172.21.15.88:0/1396720790 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5624a79a5cc0 0x5624a79a7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.977+0000 7f28c8c11740 1 --2- 172.21.15.88:0/1396720790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624a79a8500 0x5624a79a88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.977+0000 7f28c8c11740 1 -- 172.21.15.88:0/1396720790 >> 172.21.15.88:0/1396720790 conn(0x5624a795e160 msgr2=0x5624a79a6570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:20.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.977+0000 7f28c8c11740 1 -- 172.21.15.88:0/1396720790 shutdown_connections 2024-09-16T01:35:20.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.977+0000 7f28c8c11740 1 -- 172.21.15.88:0/1396720790 wait complete. 2024-09-16T01:35:20.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.977+0000 7f28c8c11740 1 -- start start 2024-09-16T01:35:20.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.977+0000 7f28c8c11740 1 Processor -- start 2024-09-16T01:35:20.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.978+0000 7f28c8c11740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a79a6120 0x5624a799f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.978+0000 7f28c8c11740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624a79a8500 0x5624a79860e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.978+0000 7f28c8c11740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5624a7986620 0x5624a7989550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.978+0000 7f28c8c11740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5624a79a4c20 con 0x5624a79a6120 2024-09-16T01:35:20.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.978+0000 7f28c8c11740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5624a784d7b0 con 0x5624a79a8500 2024-09-16T01:35:20.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.978+0000 7f28c8c11740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5624a79a4f00 con 0x5624a7986620 2024-09-16T01:35:20.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.978+0000 7f28c8c11740 10 client.0 _ll_register_callbacks cb 0x5624a75d6e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:35:20.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.978+0000 7f28bf7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a79a6120 0x5624a799f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.978+0000 7f28bf7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a79a6120 0x5624a799f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47838/0 (socket says 172.21.15.88:47838) 2024-09-16T01:35:20.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.978+0000 7f28bf7fe640 1 -- 172.21.15.88:0/2218867511 learned_addr learned my addr 172.21.15.88:0/2218867511 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:20.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.979+0000 7f28bffff640 1 --2- 172.21.15.88:0/2218867511 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5624a7986620 0x5624a7989550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.979+0000 7f28beffd640 1 --2- 172.21.15.88:0/2218867511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624a79a8500 0x5624a79860e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.979+0000 7f28bf7fe640 1 -- 172.21.15.88:0/2218867511 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5624a7986620 msgr2=0x5624a7989550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:20.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.979+0000 7f28bf7fe640 1 --2- 172.21.15.88:0/2218867511 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5624a7986620 0x5624a7989550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.979+0000 7f28bf7fe640 1 -- 172.21.15.88:0/2218867511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624a79a8500 msgr2=0x5624a79860e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:20.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.979+0000 7f28bf7fe640 1 --2- 172.21.15.88:0/2218867511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624a79a8500 0x5624a79860e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.979+0000 7f28bf7fe640 1 -- 172.21.15.88:0/2218867511 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624a7597310 con 0x5624a79a6120 2024-09-16T01:35:20.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.979+0000 7f28bffff640 1 --2- 172.21.15.88:0/2218867511 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5624a7986620 0x5624a7989550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:20.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.979+0000 7f28beffd640 1 --2- 172.21.15.88:0/2218867511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624a79a8500 0x5624a79860e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:20.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.979+0000 7f28bf7fe640 1 --2- 172.21.15.88:0/2218867511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a79a6120 0x5624a799f680 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f28b00157e0 tx=0x7f28b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36080dbdbd5d710 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:20.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.979+0000 7f289e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:35:20.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.979+0000 7f289e7fc640 1 -- 172.21.15.88:0/2218867511 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b000f040 con 0x5624a79a6120 2024-09-16T01:35:20.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.979+0000 7f289e7fc640 1 -- 172.21.15.88:0/2218867511 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f28b0016ba0 con 0x5624a79a6120 2024-09-16T01:35:20.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.979+0000 7f28c8c11740 1 -- 172.21.15.88:0/2218867511 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28b800dd90 con 0x5624a79a6120 2024-09-16T01:35:20.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.979+0000 7f289e7fc640 1 -- 172.21.15.88:0/2218867511 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f28b002f050 con 0x5624a79a6120 2024-09-16T01:35:20.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.979+0000 7f28c8c11740 10 client.5169 fetch_fsmap learned FSMap version 19 2024-09-16T01:35:20.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.979+0000 7f28c8c11740 1 -- 172.21.15.88:0/2218867511 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x5624a77041b0 con 0x5624a79a6120 2024-09-16T01:35:20.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.979+0000 7f289e7fc640 1 -- 172.21.15.88:0/2218867511 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28b800dd90 con 0x5624a79a6120 2024-09-16T01:35:20.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.979+0000 7f289e7fc640 1 -- 172.21.15.88:0/2218867511 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 19) ==== 3502+0+0 (secure 0 0 0) 0x7f28b0032020 con 0x5624a79a6120 2024-09-16T01:35:20.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.979+0000 7f28c8c11740 10 client.5169 fetch_fsmap finished waiting for FSMap version 19 2024-09-16T01:35:20.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.979+0000 7f28c8c11740 10 client.5169 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-09-16T01:35:20.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.979+0000 7f28c8c11740 20 client.5169 populate_metadata read hostname 'smithi088' 2024-09-16T01:35:20.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.979+0000 7f28c8c11740 1 --2- 172.21.15.88:0/2218867511 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x5624a79b7000 0x5624a798b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:20.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.979+0000 7f28c8c11740 4 client.5169 mds_command: new command op to 5067 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:35:20.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.979+0000 7f28c8c11740 1 -- 172.21.15.88:0/2218867511 --> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624a78d21f0 con 0x5624a79b7000 2024-09-16T01:35:20.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.980+0000 7f28beffd640 1 --2- 172.21.15.88:0/2218867511 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x5624a79b7000 0x5624a798b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:20.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.980+0000 7f28beffd640 1 --2- 172.21.15.88:0/2218867511 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x5624a79b7000 0x5624a798b8f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78f71e916c171c2 server_cookie=f8d0705695c08ddc in_seq=0 out_seq=0 2024-09-16T01:35:20.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.980+0000 7f289e7fc640 10 client.5169 ms_handle_connect on v2:172.21.15.88:6834/2870334334 2024-09-16T01:35:20.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.980+0000 7f289e7fc640 1 -- 172.21.15.88:0/2218867511 <== mds.0 v2:172.21.15.88:6834/2870334334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5624a78d21f0 con 0x5624a79b7000 2024-09-16T01:35:20.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.980+0000 7f289e7fc640 10 client.5169 handle_command_reply: tid=0 2024-09-16T01:35:20.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.980+0000 7f28c8c11740 1 client.5169 shutdown 2024-09-16T01:35:20.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.980+0000 7f28c8c11740 1 -- 172.21.15.88:0/2218867511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a79a6120 msgr2=0x5624a799f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:20.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.980+0000 7f28c8c11740 1 --2- 172.21.15.88:0/2218867511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a79a6120 0x5624a799f680 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f28b00157e0 tx=0x7f28b0004730 comp rx=0 tx=0).stop 2024-09-16T01:35:20.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.980+0000 7f28c8c11740 1 -- 172.21.15.88:0/2218867511 shutdown_connections 2024-09-16T01:35:20.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.980+0000 7f28c8c11740 1 --2- 172.21.15.88:0/2218867511 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x5624a79b7000 0x5624a798b8f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.980+0000 7f28c8c11740 1 --2- 172.21.15.88:0/2218867511 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5624a7986620 0x5624a7989550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.981+0000 7f28c8c11740 1 --2- 172.21.15.88:0/2218867511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624a79a8500 0x5624a79860e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.981+0000 7f289e7fc640 0 client.5169 ms_handle_reset on v2:172.21.15.88:6834/2870334334 2024-09-16T01:35:20.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.981+0000 7f28c8c11740 1 --2- 172.21.15.88:0/2218867511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a79a6120 0x5624a799f680 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:20.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.981+0000 7f28c8c11740 1 -- 172.21.15.88:0/2218867511 >> 172.21.15.88:0/2218867511 conn(0x5624a79a5cc0 msgr2=0x5624a7987690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:20.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.981+0000 7f28c8c11740 1 -- 172.21.15.88:0/2218867511 shutdown_connections 2024-09-16T01:35:20.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.981+0000 7f28c8c11740 1 -- 172.21.15.88:0/2218867511 wait complete. 2024-09-16T01:35:20.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.981+0000 7f28c8c11740 20 client.5169 trim_cache size 0 max 16384 2024-09-16T01:35:20.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.998+0000 7f28c8c11740 1 Processor -- start 2024-09-16T01:35:20.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.998+0000 7f28c8c11740 1 -- start start 2024-09-16T01:35:21.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.999+0000 7f28c8c11740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a7986620 0x5624a798c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.999+0000 7f28c8c11740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624a79a6120 0x5624a79cac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.999+0000 7f28c8c11740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5624a79a8500 0x5624a7985890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.999+0000 7f28c8c11740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f28c0200da0 con 0x5624a7986620 2024-09-16T01:35:21.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.999+0000 7f28c8c11740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f28c01ffdd0 con 0x5624a79a6120 2024-09-16T01:35:21.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.999+0000 7f28c8c11740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f28c01f4b80 con 0x5624a79a8500 2024-09-16T01:35:21.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.999+0000 7f28bf7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a7986620 0x5624a798c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:21.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.999+0000 7f28bf7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a7986620 0x5624a798c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47852/0 (socket says 172.21.15.88:47852) 2024-09-16T01:35:21.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.999+0000 7f28bf7fe640 1 -- 172.21.15.88:0/2723028980 learned_addr learned my addr 172.21.15.88:0/2723028980 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:21.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.999+0000 7f28bffff640 1 --2- 172.21.15.88:0/2723028980 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5624a79a8500 0x5624a7985890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:21.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.999+0000 7f28beffd640 1 --2- 172.21.15.88:0/2723028980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624a79a6120 0x5624a79cac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:21.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.999+0000 7f28bf7fe640 1 -- 172.21.15.88:0/2723028980 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5624a79a8500 msgr2=0x5624a7985890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:21.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.999+0000 7f28bf7fe640 1 --2- 172.21.15.88:0/2723028980 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5624a79a8500 0x5624a7985890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.999+0000 7f28bf7fe640 1 -- 172.21.15.88:0/2723028980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624a79a6120 msgr2=0x5624a79cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:21.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.999+0000 7f28bf7fe640 1 --2- 172.21.15.88:0/2723028980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624a79a6120 0x5624a79cac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.999+0000 7f28bf7fe640 1 -- 172.21.15.88:0/2723028980 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624a75703d0 con 0x5624a7986620 2024-09-16T01:35:21.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.999+0000 7f28bffff640 1 --2- 172.21.15.88:0/2723028980 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5624a79a8500 0x5624a7985890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:21.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.999+0000 7f28bf7fe640 1 --2- 172.21.15.88:0/2723028980 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a7986620 0x5624a798c0e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f28b00094b0 tx=0x7f28b0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67054e587ea1a100 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:21.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.999+0000 7f28a4ff9640 1 -- 172.21.15.88:0/2723028980 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b0037070 con 0x5624a7986620 2024-09-16T01:35:21.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.999+0000 7f28a4ff9640 1 -- 172.21.15.88:0/2723028980 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28b0027070 con 0x5624a7986620 2024-09-16T01:35:21.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:20.999+0000 7f28a4ff9640 1 -- 172.21.15.88:0/2723028980 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b0016a10 con 0x5624a7986620 2024-09-16T01:35:21.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.000+0000 7f28c8c11740 1 -- 172.21.15.88:0/2723028980 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a7986620 msgr2=0x5624a798c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:21.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.000+0000 7f28c8c11740 1 --2- 172.21.15.88:0/2723028980 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a7986620 0x5624a798c0e0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f28b00094b0 tx=0x7f28b0017000 comp rx=0 tx=0).stop 2024-09-16T01:35:21.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.000+0000 7f28c8c11740 1 -- 172.21.15.88:0/2723028980 shutdown_connections 2024-09-16T01:35:21.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.000+0000 7f28c8c11740 1 --2- 172.21.15.88:0/2723028980 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5624a79a8500 0x5624a7985890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.000+0000 7f28c8c11740 1 --2- 172.21.15.88:0/2723028980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624a79a6120 0x5624a79cac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.000+0000 7f28c8c11740 1 --2- 172.21.15.88:0/2723028980 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a7986620 0x5624a798c0e0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.000+0000 7f28c8c11740 1 -- 172.21.15.88:0/2723028980 >> 172.21.15.88:0/2723028980 conn(0x5624a79a5cc0 msgr2=0x5624a79a8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:21.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.000+0000 7f28c8c11740 1 -- 172.21.15.88:0/2723028980 shutdown_connections 2024-09-16T01:35:21.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.000+0000 7f28c8c11740 1 -- 172.21.15.88:0/2723028980 wait complete. 2024-09-16T01:35:21.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.001+0000 7f28c8c11740 1 -- start start 2024-09-16T01:35:21.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.001+0000 7f28c8c11740 1 Processor -- start 2024-09-16T01:35:21.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.001+0000 7f28c8c11740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5624a7986620 0x5624a79a3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.001+0000 7f28c8c11740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624a79a6120 0x5624a7a23900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.001+0000 7f28c8c11740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a79a8500 0x5624a798a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.001+0000 7f28c8c11740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5624a7989a00 con 0x5624a79a8500 2024-09-16T01:35:21.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.001+0000 7f28c8c11740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f28c01ff850 con 0x5624a79a6120 2024-09-16T01:35:21.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.001+0000 7f28c8c11740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5624a79a0120 con 0x5624a7986620 2024-09-16T01:35:21.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.001+0000 7f28c8c11740 10 client.0 _ll_register_callbacks cb 0x7f28b0009770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:35:21.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.002+0000 7f28bffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a79a8500 0x5624a798a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:21.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.002+0000 7f28bffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a79a8500 0x5624a798a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47866/0 (socket says 172.21.15.88:47866) 2024-09-16T01:35:21.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.002+0000 7f28bffff640 1 -- 172.21.15.88:0/527595416 learned_addr learned my addr 172.21.15.88:0/527595416 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:21.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.002+0000 7f28bf7fe640 1 --2- 172.21.15.88:0/527595416 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5624a7986620 0x5624a79a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:21.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.002+0000 7f28beffd640 1 --2- 172.21.15.88:0/527595416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624a79a6120 0x5624a7a23900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:21.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.002+0000 7f28bffff640 1 -- 172.21.15.88:0/527595416 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5624a7986620 msgr2=0x5624a79a3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:21.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.002+0000 7f28bffff640 1 --2- 172.21.15.88:0/527595416 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5624a7986620 0x5624a79a3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.002+0000 7f28bffff640 1 -- 172.21.15.88:0/527595416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624a79a6120 msgr2=0x5624a7a23900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:21.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.002+0000 7f28bffff640 1 --2- 172.21.15.88:0/527595416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624a79a6120 0x5624a7a23900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.002+0000 7f28bffff640 1 -- 172.21.15.88:0/527595416 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624a7a23e40 con 0x5624a79a8500 2024-09-16T01:35:21.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.002+0000 7f28beffd640 1 --2- 172.21.15.88:0/527595416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624a79a6120 0x5624a7a23900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:21.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.002+0000 7f28bffff640 1 --2- 172.21.15.88:0/527595416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a79a8500 0x5624a798a5a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f28b800d650 tx=0x7f28b8001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c3a92dd821827e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:21.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.002+0000 7f289e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:35:21.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.002+0000 7f289e7fc640 1 -- 172.21.15.88:0/527595416 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b8025060 con 0x5624a79a8500 2024-09-16T01:35:21.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.002+0000 7f289e7fc640 1 -- 172.21.15.88:0/527595416 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f28b80052c0 con 0x5624a79a8500 2024-09-16T01:35:21.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.002+0000 7f28c8c11740 1 -- 172.21.15.88:0/527595416 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28b0027070 con 0x5624a79a8500 2024-09-16T01:35:21.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.002+0000 7f289e7fc640 1 -- 172.21.15.88:0/527595416 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f28b802b050 con 0x5624a79a8500 2024-09-16T01:35:21.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.002+0000 7f28c8c11740 10 client.5175 fetch_fsmap learned FSMap version 19 2024-09-16T01:35:21.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.002+0000 7f28c8c11740 1 -- 172.21.15.88:0/527595416 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x5624a79bc3a0 con 0x5624a79a8500 2024-09-16T01:35:21.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.002+0000 7f289e7fc640 1 -- 172.21.15.88:0/527595416 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28b0027070 con 0x5624a79a8500 2024-09-16T01:35:21.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.002+0000 7f289e7fc640 1 -- 172.21.15.88:0/527595416 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 19) ==== 3502+0+0 (secure 0 0 0) 0x7f28b802e020 con 0x5624a79a8500 2024-09-16T01:35:21.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.003+0000 7f28c8c11740 10 client.5175 fetch_fsmap finished waiting for FSMap version 19 2024-09-16T01:35:21.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.003+0000 7f28c8c11740 10 client.5175 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-09-16T01:35:21.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.003+0000 7f28c8c11740 20 client.5175 populate_metadata read hostname 'smithi088' 2024-09-16T01:35:21.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.003+0000 7f28c8c11740 1 --2- 172.21.15.88:0/527595416 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x5624a798aae0 0x5624a799d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.003+0000 7f28c8c11740 4 client.5175 mds_command: new command op to 5067 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/subdir", "scrubops": ["repair"]}] 2024-09-16T01:35:21.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.003+0000 7f28c8c11740 1 -- 172.21.15.88:0/527595416 --> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] -- command(tid 0: {"prefix": "scrub start", "path": "/subdir", "scrubops": ["repair"]}) -- 0x5624a78cc540 con 0x5624a798aae0 2024-09-16T01:35:21.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.003+0000 7f28bf7fe640 1 --2- 172.21.15.88:0/527595416 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x5624a798aae0 0x5624a799d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:21.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.003+0000 7f289e7fc640 10 client.5175 ms_handle_connect on v2:172.21.15.88:6834/2870334334 2024-09-16T01:35:21.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.003+0000 7f28bf7fe640 1 --2- 172.21.15.88:0/527595416 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x5624a798aae0 0x5624a799d3d0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5095e94e01f0115 server_cookie=1f26c409d788fcb in_seq=0 out_seq=0 2024-09-16T01:35:21.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.003+0000 7f289e7fc640 1 -- 172.21.15.88:0/527595416 <== mds.0 v2:172.21.15.88:6834/2870334334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5624a78cc540 con 0x5624a798aae0 2024-09-16T01:35:21.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.003+0000 7f289e7fc640 10 client.5175 handle_command_reply: tid=0 2024-09-16T01:35:21.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.003+0000 7f28c8c11740 1 client.5175 shutdown 2024-09-16T01:35:21.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.004+0000 7f28c8c11740 1 -- 172.21.15.88:0/527595416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a79a8500 msgr2=0x5624a798a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:21.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.004+0000 7f28c8c11740 1 --2- 172.21.15.88:0/527595416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a79a8500 0x5624a798a5a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f28b800d650 tx=0x7f28b8001250 comp rx=0 tx=0).stop 2024-09-16T01:35:21.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.004+0000 7f28c8c11740 1 -- 172.21.15.88:0/527595416 shutdown_connections 2024-09-16T01:35:21.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.004+0000 7f28c8c11740 1 --2- 172.21.15.88:0/527595416 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x5624a798aae0 0x5624a799d3d0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.004+0000 7f28c8c11740 1 --2- 172.21.15.88:0/527595416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5624a79a8500 0x5624a798a5a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.004+0000 7f28c8c11740 1 --2- 172.21.15.88:0/527595416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5624a79a6120 0x5624a7a23900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.004+0000 7f28c8c11740 1 --2- 172.21.15.88:0/527595416 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5624a7986620 0x5624a79a3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.004+0000 7f28c8c11740 1 -- 172.21.15.88:0/527595416 >> 172.21.15.88:0/527595416 conn(0x5624a79a5cc0 msgr2=0x5624a7957240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:21.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.004+0000 7f289e7fc640 0 client.5175 ms_handle_reset on v2:172.21.15.88:6834/2870334334 2024-09-16T01:35:21.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.004+0000 7f28c8c11740 1 -- 172.21.15.88:0/527595416 shutdown_connections 2024-09-16T01:35:21.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.004+0000 7f28c8c11740 1 -- 172.21.15.88:0/527595416 wait complete. 2024-09-16T01:35:21.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.004+0000 7f28c8c11740 20 client.5175 trim_cache size 0 max 16384 2024-09-16T01:35:21.012 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:35:21.012 INFO:teuthology.orchestra.run.smithi088.stdout: "return_code": 0, 2024-09-16T01:35:21.012 INFO:teuthology.orchestra.run.smithi088.stdout: "scrub_tag": "9823f7b0-585e-4120-a79a-63d2185e414f", 2024-09-16T01:35:21.012 INFO:teuthology.orchestra.run.smithi088.stdout: "mode": "asynchronous" 2024-09-16T01:35:21.012 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:35:21.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.004+0000 7f28c5d98640 1 -- 172.21.15.88:0/3692055654 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f28ac06c2b0 msgr2=0x7f28ac06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:21.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.004+0000 7f28c5d98640 1 --2- 172.21.15.88:0/3692055654 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f28ac06c2b0 0x7f28ac06e770 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f28b0009ac0 tx=0x7f28b0008040 comp rx=0 tx=0).stop 2024-09-16T01:35:21.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.004+0000 7f28c5d98640 1 -- 172.21.15.88:0/3692055654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28c00714a0 msgr2=0x7f28c00ccdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:21.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.004+0000 7f28c5d98640 1 --2- 172.21.15.88:0/3692055654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28c00714a0 0x7f28c00ccdc0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f28b4000c00 tx=0x7f28b40046c0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.004+0000 7f28c5d98640 1 -- 172.21.15.88:0/3692055654 shutdown_connections 2024-09-16T01:35:21.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.004+0000 7f28c5d98640 1 --2- 172.21.15.88:0/3692055654 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f28ac06c2b0 0x7f28ac06e770 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.004+0000 7f28c5d98640 1 --2- 172.21.15.88:0/3692055654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28c015e570 0x7f28c014bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.004+0000 7f28c5d98640 1 --2- 172.21.15.88:0/3692055654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f28c00714a0 0x7f28c00ccdc0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.004+0000 7f28c5d98640 1 --2- 172.21.15.88:0/3692055654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f28c006f400 0x7f28c0158a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.004+0000 7f28c5d98640 1 -- 172.21.15.88:0/3692055654 >> 172.21.15.88:0/3692055654 conn(0x7f28c00b4240 msgr2=0x7f28c00b6dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:21.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.005+0000 7f28c5d98640 1 -- 172.21.15.88:0/3692055654 shutdown_connections 2024-09-16T01:35:21.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.005+0000 7f28c5d98640 1 -- 172.21.15.88:0/3692055654 wait complete. 2024-09-16T01:35:21.017 INFO:tasks.cephfs.test_damage:{ "return_code": 0, "scrub_tag": "9823f7b0-585e-4120-a79a-63d2185e414f", "mode": "asynchronous" } 2024-09-16T01:35:21.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:21.017 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.2:0 scrub status 2024-09-16T01:35:21.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.116+0000 7f7435cb7640 1 Processor -- start 2024-09-16T01:35:21.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.116+0000 7f7435cb7640 1 -- start start 2024-09-16T01:35:21.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.117+0000 7f7435cb7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f743015e760 0x7f7430073d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.117+0000 7f7435cb7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f743006fda0 0x7f74300b5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.117+0000 7f7435cb7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7430074260 con 0x7f74300b59a0 2024-09-16T01:35:21.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.117+0000 7f7435cb7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f74300d1540 con 0x7f743015e760 2024-09-16T01:35:21.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.117+0000 7f7435cb7640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f743006d810 con 0x7f743006fda0 2024-09-16T01:35:21.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.117+0000 7f742f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f743015e760 0x7f7430073d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:21.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.117+0000 7f742f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f743015e760 0x7f7430073d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47876/0 (socket says 172.21.15.88:47876) 2024-09-16T01:35:21.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.117+0000 7f742f7fe640 1 -- 172.21.15.88:0/3095721952 learned_addr learned my addr 172.21.15.88:0/3095721952 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:21.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.118+0000 7f742f7fe640 1 -- 172.21.15.88:0/3095721952 >> v2:172.21.15.154:3300/0 conn(0x7f743006fda0 msgr2=0x7f74300b5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:21.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.118+0000 7f742effd640 1 --2- 172.21.15.88:0/3095721952 >> v2:172.21.15.154:3300/0 conn(0x7f743006fda0 0x7f74300b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:21.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.118+0000 7f742f7fe640 1 --2- 172.21.15.88:0/3095721952 >> v2:172.21.15.154:3300/0 conn(0x7f743006fda0 0x7f74300b5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.118+0000 7f742f7fe640 1 -- 172.21.15.88:0/3095721952 >> v1:172.21.15.154:6789/0 conn(0x7f74300b59a0 legacy=0x7f74300d1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:21.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.118+0000 7f742f7fe640 1 -- 172.21.15.88:0/3095721952 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f743006daf0 con 0x7f743015e760 2024-09-16T01:35:21.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.118+0000 7f742f7fe640 1 --2- 172.21.15.88:0/3095721952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f743015e760 0x7f7430073d20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f743006d160 tx=0x7f742002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88654263be4072b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:21.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.118+0000 7f742e7fc640 1 -- 172.21.15.88:0/3095721952 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7420037070 con 0x7f743015e760 2024-09-16T01:35:21.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.119+0000 7f742e7fc640 1 -- 172.21.15.88:0/3095721952 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7420032440 con 0x7f743015e760 2024-09-16T01:35:21.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.119+0000 7f742e7fc640 1 -- 172.21.15.88:0/3095721952 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7420032810 con 0x7f743015e760 2024-09-16T01:35:21.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.119+0000 7f7435cb7640 1 -- 172.21.15.88:0/3095721952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f743015e760 msgr2=0x7f7430073d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:21.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.119+0000 7f7435cb7640 1 --2- 172.21.15.88:0/3095721952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f743015e760 0x7f7430073d20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f743006d160 tx=0x7f742002a230 comp rx=0 tx=0).stop 2024-09-16T01:35:21.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.119+0000 7f7435cb7640 1 -- 172.21.15.88:0/3095721952 shutdown_connections 2024-09-16T01:35:21.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.119+0000 7f7435cb7640 1 --2- 172.21.15.88:0/3095721952 >> v2:172.21.15.154:3300/0 conn(0x7f743006fda0 0x7f74300b5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.119+0000 7f7435cb7640 1 --2- 172.21.15.88:0/3095721952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f743015e760 0x7f7430073d20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.119+0000 7f7435cb7640 1 -- 172.21.15.88:0/3095721952 >> 172.21.15.88:0/3095721952 conn(0x7f74300b7d70 msgr2=0x7f7430062630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:21.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.119+0000 7f7435cb7640 1 -- 172.21.15.88:0/3095721952 shutdown_connections 2024-09-16T01:35:21.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.119+0000 7f7435cb7640 1 -- 172.21.15.88:0/3095721952 wait complete. 2024-09-16T01:35:21.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.120+0000 7f7435cb7640 1 Processor -- start 2024-09-16T01:35:21.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.120+0000 7f7435cb7640 1 -- start start 2024-09-16T01:35:21.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.121+0000 7f7435cb7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f743006fda0 0x7f74301f3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.121+0000 7f7435cb7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74300b59a0 0x7f74301f77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.121+0000 7f7435cb7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f743015e760 0x7f74301f7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.121+0000 7f7435cb7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f74300b5e30 con 0x7f743006fda0 2024-09-16T01:35:21.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.121+0000 7f7435cb7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f743006ddd0 con 0x7f74300b59a0 2024-09-16T01:35:21.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.121+0000 7f7435cb7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f74300b5fb0 con 0x7f743015e760 2024-09-16T01:35:21.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.121+0000 7f742f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f743006fda0 0x7f74301f3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:21.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.121+0000 7f742f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f743006fda0 0x7f74301f3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47886/0 (socket says 172.21.15.88:47886) 2024-09-16T01:35:21.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.121+0000 7f742f7fe640 1 -- 172.21.15.88:0/4128225103 learned_addr learned my addr 172.21.15.88:0/4128225103 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:21.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.121+0000 7f742ffff640 1 --2- 172.21.15.88:0/4128225103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f743015e760 0x7f74301f7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:21.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.121+0000 7f742effd640 1 --2- 172.21.15.88:0/4128225103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74300b59a0 0x7f74301f77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:21.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.121+0000 7f742f7fe640 1 -- 172.21.15.88:0/4128225103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f743015e760 msgr2=0x7f74301f7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:21.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.121+0000 7f742f7fe640 1 --2- 172.21.15.88:0/4128225103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f743015e760 0x7f74301f7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.121+0000 7f742f7fe640 1 -- 172.21.15.88:0/4128225103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74300b59a0 msgr2=0x7f74301f77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:21.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.121+0000 7f742f7fe640 1 --2- 172.21.15.88:0/4128225103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74300b59a0 0x7f74301f77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.122+0000 7f742f7fe640 1 -- 172.21.15.88:0/4128225103 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74301603a0 con 0x7f743006fda0 2024-09-16T01:35:21.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.122+0000 7f742f7fe640 1 --2- 172.21.15.88:0/4128225103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f743006fda0 0x7f74301f3a20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7420000c00 tx=0x7f74200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:21.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.122+0000 7f7417fff640 1 -- 172.21.15.88:0/4128225103 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7420041020 con 0x7f743006fda0 2024-09-16T01:35:21.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.122+0000 7f7417fff640 1 -- 172.21.15.88:0/4128225103 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7420030970 con 0x7f743006fda0 2024-09-16T01:35:21.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.122+0000 7f7417fff640 1 -- 172.21.15.88:0/4128225103 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7420037040 con 0x7f743006fda0 2024-09-16T01:35:21.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.122+0000 7f7435cb7640 1 -- 172.21.15.88:0/4128225103 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7430160680 con 0x7f743006fda0 2024-09-16T01:35:21.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.122+0000 7f7435cb7640 1 -- 172.21.15.88:0/4128225103 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7430155e20 con 0x7f743006fda0 2024-09-16T01:35:21.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.123+0000 7f7417fff640 1 -- 172.21.15.88:0/4128225103 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7420044020 con 0x7f743006fda0 2024-09-16T01:35:21.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.124+0000 7f7417fff640 1 --2- 172.21.15.88:0/4128225103 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f741c06c2b0 0x7f741c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.124+0000 7f7417fff640 1 -- 172.21.15.88:0/4128225103 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f74200abf60 con 0x7f743006fda0 2024-09-16T01:35:21.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.124+0000 7f742effd640 1 --2- 172.21.15.88:0/4128225103 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f741c06c2b0 0x7f741c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:21.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.124+0000 7f742effd640 1 --2- 172.21.15.88:0/4128225103 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f741c06c2b0 0x7f741c06e770 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f74300b58d0 tx=0x7f7424008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:21.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.125+0000 7f7438a4f740 1 Processor -- start 2024-09-16T01:35:21.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.125+0000 7f7438a4f740 1 -- start start 2024-09-16T01:35:21.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.126+0000 7f7438a4f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56401b346920 0x56401b38c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.126+0000 7f7438a4f740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56401b3440e0 0x56401b346d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.126+0000 7f7438a4f740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b3480b0 0x56401b3411a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.126+0000 7f7438a4f740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56401b2a96f0 con 0x56401b3480b0 2024-09-16T01:35:21.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.126+0000 7f7438a4f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56401b251330 con 0x56401b346920 2024-09-16T01:35:21.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.126+0000 7f7438a4f740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56401b2a56b0 con 0x56401b3440e0 2024-09-16T01:35:21.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.126+0000 7f742ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b3480b0 0x56401b3411a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:21.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.126+0000 7f742ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b3480b0 0x56401b3411a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47892/0 (socket says 172.21.15.88:47892) 2024-09-16T01:35:21.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.126+0000 7f742ffff640 1 -- 172.21.15.88:0/265085664 learned_addr learned my addr 172.21.15.88:0/265085664 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:21.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.126+0000 7f742f7fe640 1 --2- 172.21.15.88:0/265085664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56401b346920 0x56401b38c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:21.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.126+0000 7f742effd640 1 --2- 172.21.15.88:0/265085664 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56401b3440e0 0x56401b346d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:21.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.126+0000 7f742ffff640 1 -- 172.21.15.88:0/265085664 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56401b3440e0 msgr2=0x56401b346d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:21.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.126+0000 7f742ffff640 1 --2- 172.21.15.88:0/265085664 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56401b3440e0 0x56401b346d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.126+0000 7f742ffff640 1 -- 172.21.15.88:0/265085664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56401b346920 msgr2=0x56401b38c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:21.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.126+0000 7f742ffff640 1 --2- 172.21.15.88:0/265085664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56401b346920 0x56401b38c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.126+0000 7f742ffff640 1 -- 172.21.15.88:0/265085664 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56401b2770d0 con 0x56401b3480b0 2024-09-16T01:35:21.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.126+0000 7f742ffff640 1 --2- 172.21.15.88:0/265085664 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b3480b0 0x56401b3411a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f7428001cf0 tx=0x7f74280020f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dabe80c5b6f9fc1a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:21.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.126+0000 7f74157fa640 1 -- 172.21.15.88:0/265085664 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7428019070 con 0x56401b3480b0 2024-09-16T01:35:21.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.126+0000 7f74157fa640 1 -- 172.21.15.88:0/265085664 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7428003b80 con 0x56401b3480b0 2024-09-16T01:35:21.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.126+0000 7f74157fa640 1 -- 172.21.15.88:0/265085664 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f742800e860 con 0x56401b3480b0 2024-09-16T01:35:21.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.127+0000 7f7438a4f740 1 -- 172.21.15.88:0/265085664 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b3480b0 msgr2=0x56401b3411a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:21.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.127+0000 7f7438a4f740 1 --2- 172.21.15.88:0/265085664 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b3480b0 0x56401b3411a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f7428001cf0 tx=0x7f74280020f0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.127+0000 7f7438a4f740 1 -- 172.21.15.88:0/265085664 shutdown_connections 2024-09-16T01:35:21.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.127+0000 7f7438a4f740 1 --2- 172.21.15.88:0/265085664 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b3480b0 0x56401b3411a0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.127+0000 7f7438a4f740 1 --2- 172.21.15.88:0/265085664 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56401b3440e0 0x56401b346d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.127+0000 7f7438a4f740 1 --2- 172.21.15.88:0/265085664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56401b346920 0x56401b38c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.127+0000 7f7438a4f740 1 -- 172.21.15.88:0/265085664 >> 172.21.15.88:0/265085664 conn(0x56401b343370 msgr2=0x56401b38b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:21.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.127+0000 7f7438a4f740 1 -- 172.21.15.88:0/265085664 shutdown_connections 2024-09-16T01:35:21.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.127+0000 7f7438a4f740 1 -- 172.21.15.88:0/265085664 wait complete. 2024-09-16T01:35:21.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.128+0000 7f7438a4f740 1 -- start start 2024-09-16T01:35:21.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.128+0000 7f7438a4f740 1 Processor -- start 2024-09-16T01:35:21.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.128+0000 7f7438a4f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56401b346920 0x56401b36e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.128+0000 7f7438a4f740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b3480b0 0x56401b366f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.129+0000 7f7438a4f740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56401b36fcf0 0x56401b369bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.129+0000 7f7438a4f740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56401b342810 con 0x56401b3480b0 2024-09-16T01:35:21.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.129+0000 7f7438a4f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56401b2324f0 con 0x56401b346920 2024-09-16T01:35:21.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.129+0000 7f7438a4f740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56401b38c8f0 con 0x56401b36fcf0 2024-09-16T01:35:21.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.129+0000 7f7438a4f740 10 client.0 _ll_register_callbacks cb 0x56401b253b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:35:21.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.129+0000 7f742effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b3480b0 0x56401b366f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:21.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.129+0000 7f742effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b3480b0 0x56401b366f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47894/0 (socket says 172.21.15.88:47894) 2024-09-16T01:35:21.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.129+0000 7f742effd640 1 -- 172.21.15.88:0/3629778011 learned_addr learned my addr 172.21.15.88:0/3629778011 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:21.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.129+0000 7f742f7fe640 1 --2- 172.21.15.88:0/3629778011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56401b346920 0x56401b36e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:21.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.129+0000 7f742ffff640 1 --2- 172.21.15.88:0/3629778011 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56401b36fcf0 0x56401b369bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:21.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.129+0000 7f742effd640 1 -- 172.21.15.88:0/3629778011 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56401b36fcf0 msgr2=0x56401b369bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:21.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.129+0000 7f742effd640 1 --2- 172.21.15.88:0/3629778011 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56401b36fcf0 0x56401b369bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.129+0000 7f742effd640 1 -- 172.21.15.88:0/3629778011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56401b346920 msgr2=0x56401b36e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:21.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.129+0000 7f742effd640 1 --2- 172.21.15.88:0/3629778011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56401b346920 0x56401b36e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.129+0000 7f742effd640 1 -- 172.21.15.88:0/3629778011 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56401b0867d0 con 0x56401b3480b0 2024-09-16T01:35:21.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.129+0000 7f742effd640 1 --2- 172.21.15.88:0/3629778011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b3480b0 0x56401b366f50 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7424013700 tx=0x7f74240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=425e172ad50ebc75 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:21.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.129+0000 7f740e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:35:21.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.129+0000 7f740e7fc640 1 -- 172.21.15.88:0/3629778011 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7424024070 con 0x56401b3480b0 2024-09-16T01:35:21.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.129+0000 7f740e7fc640 1 -- 172.21.15.88:0/3629778011 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f7424014850 con 0x56401b3480b0 2024-09-16T01:35:21.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.129+0000 7f7438a4f740 1 -- 172.21.15.88:0/3629778011 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7428003b80 con 0x56401b3480b0 2024-09-16T01:35:21.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.129+0000 7f740e7fc640 1 -- 172.21.15.88:0/3629778011 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f742402f050 con 0x56401b3480b0 2024-09-16T01:35:21.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.129+0000 7f7438a4f740 10 client.5187 fetch_fsmap learned FSMap version 19 2024-09-16T01:35:21.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.129+0000 7f7438a4f740 1 -- 172.21.15.88:0/3629778011 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x56401afbc3f0 con 0x56401b3480b0 2024-09-16T01:35:21.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.130+0000 7f740e7fc640 1 -- 172.21.15.88:0/3629778011 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7428003b80 con 0x56401b3480b0 2024-09-16T01:35:21.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.130+0000 7f740e7fc640 1 -- 172.21.15.88:0/3629778011 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 19) ==== 3502+0+0 (secure 0 0 0) 0x7f7424032020 con 0x56401b3480b0 2024-09-16T01:35:21.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.130+0000 7f7438a4f740 10 client.5187 fetch_fsmap finished waiting for FSMap version 19 2024-09-16T01:35:21.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.130+0000 7f7438a4f740 10 client.5187 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-09-16T01:35:21.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.130+0000 7f7438a4f740 20 client.5187 populate_metadata read hostname 'smithi088' 2024-09-16T01:35:21.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.130+0000 7f7438a4f740 1 --2- 172.21.15.88:0/3629778011 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x56401b36a0f0 0x56401b367d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.130+0000 7f7438a4f740 4 client.5187 mds_command: new command op to 5067 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:35:21.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.130+0000 7f7438a4f740 1 -- 172.21.15.88:0/3629778011 --> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56401b2b6e20 con 0x56401b36a0f0 2024-09-16T01:35:21.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.130+0000 7f742f7fe640 1 --2- 172.21.15.88:0/3629778011 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x56401b36a0f0 0x56401b367d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:21.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.130+0000 7f740e7fc640 10 client.5187 ms_handle_connect on v2:172.21.15.88:6834/2870334334 2024-09-16T01:35:21.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.130+0000 7f742f7fe640 1 --2- 172.21.15.88:0/3629778011 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x56401b36a0f0 0x56401b367d20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed5de5b829b554cd server_cookie=4b290b7f3ac0a367 in_seq=0 out_seq=0 2024-09-16T01:35:21.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.131+0000 7f740e7fc640 1 -- 172.21.15.88:0/3629778011 <== mds.0 v2:172.21.15.88:6834/2870334334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56401b2b6e20 con 0x56401b36a0f0 2024-09-16T01:35:21.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.131+0000 7f740e7fc640 10 client.5187 handle_command_reply: tid=0 2024-09-16T01:35:21.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.131+0000 7f7438a4f740 1 client.5187 shutdown 2024-09-16T01:35:21.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.131+0000 7f7438a4f740 1 -- 172.21.15.88:0/3629778011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b3480b0 msgr2=0x56401b366f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:21.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.131+0000 7f7438a4f740 1 --2- 172.21.15.88:0/3629778011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b3480b0 0x56401b366f50 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7424013700 tx=0x7f74240046c0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.131+0000 7f7438a4f740 1 -- 172.21.15.88:0/3629778011 shutdown_connections 2024-09-16T01:35:21.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.131+0000 7f7438a4f740 1 --2- 172.21.15.88:0/3629778011 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x56401b36a0f0 0x56401b367d20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.131+0000 7f7438a4f740 1 --2- 172.21.15.88:0/3629778011 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56401b36fcf0 0x56401b369bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.131+0000 7f7438a4f740 1 --2- 172.21.15.88:0/3629778011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b3480b0 0x56401b366f50 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.131+0000 7f740e7fc640 0 client.5187 ms_handle_reset on v2:172.21.15.88:6834/2870334334 2024-09-16T01:35:21.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.131+0000 7f7438a4f740 1 --2- 172.21.15.88:0/3629778011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56401b346920 0x56401b36e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.131+0000 7f7438a4f740 1 -- 172.21.15.88:0/3629778011 >> 172.21.15.88:0/3629778011 conn(0x56401b3440e0 msgr2=0x56401b38aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:21.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.131+0000 7f7438a4f740 1 -- 172.21.15.88:0/3629778011 shutdown_connections 2024-09-16T01:35:21.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.131+0000 7f7438a4f740 1 -- 172.21.15.88:0/3629778011 wait complete. 2024-09-16T01:35:21.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.131+0000 7f7438a4f740 20 client.5187 trim_cache size 0 max 16384 2024-09-16T01:35:21.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.148+0000 7f7438a4f740 1 Processor -- start 2024-09-16T01:35:21.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.148+0000 7f7438a4f740 1 -- start start 2024-09-16T01:35:21.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.149+0000 7f7438a4f740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56401b346920 0x56401b389800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.149+0000 7f7438a4f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56401b3480b0 0x56401b3af380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.149+0000 7f7438a4f740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b36fcf0 0x56401b3af8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.149+0000 7f7438a4f740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7430200b70 con 0x56401b36fcf0 2024-09-16T01:35:21.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.149+0000 7f7438a4f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f74301ffba0 con 0x56401b3480b0 2024-09-16T01:35:21.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.149+0000 7f7438a4f740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f743015f6c0 con 0x56401b346920 2024-09-16T01:35:21.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.149+0000 7f742ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b36fcf0 0x56401b3af8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:21.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.149+0000 7f742ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b36fcf0 0x56401b3af8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47908/0 (socket says 172.21.15.88:47908) 2024-09-16T01:35:21.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.149+0000 7f742ffff640 1 -- 172.21.15.88:0/352234625 learned_addr learned my addr 172.21.15.88:0/352234625 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:21.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.149+0000 7f742ffff640 1 -- 172.21.15.88:0/352234625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56401b346920 msgr2=0x56401b389800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:21.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.149+0000 7f742ffff640 1 --2- 172.21.15.88:0/352234625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56401b346920 0x56401b389800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.149+0000 7f742ffff640 1 -- 172.21.15.88:0/352234625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56401b3480b0 msgr2=0x56401b3af380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:21.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.149+0000 7f742ffff640 1 --2- 172.21.15.88:0/352234625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56401b3480b0 0x56401b3af380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.149+0000 7f742ffff640 1 -- 172.21.15.88:0/352234625 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56401af7c0f0 con 0x56401b36fcf0 2024-09-16T01:35:21.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.149+0000 7f742ffff640 1 --2- 172.21.15.88:0/352234625 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b36fcf0 0x56401b3af8c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7428013000 tx=0x7f742800c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdea8b52eb1b9595 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:21.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.150+0000 7f7414ff9640 1 -- 172.21.15.88:0/352234625 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7428024070 con 0x56401b36fcf0 2024-09-16T01:35:21.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.150+0000 7f7414ff9640 1 -- 172.21.15.88:0/352234625 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f742801a040 con 0x56401b36fcf0 2024-09-16T01:35:21.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.150+0000 7f7414ff9640 1 -- 172.21.15.88:0/352234625 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7428019070 con 0x56401b36fcf0 2024-09-16T01:35:21.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.150+0000 7f7438a4f740 1 -- 172.21.15.88:0/352234625 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b36fcf0 msgr2=0x56401b3af8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:21.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.150+0000 7f7438a4f740 1 --2- 172.21.15.88:0/352234625 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b36fcf0 0x56401b3af8c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7428013000 tx=0x7f742800c000 comp rx=0 tx=0).stop 2024-09-16T01:35:21.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.150+0000 7f7438a4f740 1 -- 172.21.15.88:0/352234625 shutdown_connections 2024-09-16T01:35:21.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.150+0000 7f7438a4f740 1 --2- 172.21.15.88:0/352234625 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b36fcf0 0x56401b3af8c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.150+0000 7f7438a4f740 1 --2- 172.21.15.88:0/352234625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56401b3480b0 0x56401b3af380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.150+0000 7f7438a4f740 1 --2- 172.21.15.88:0/352234625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56401b346920 0x56401b389800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.150+0000 7f7438a4f740 1 -- 172.21.15.88:0/352234625 >> 172.21.15.88:0/352234625 conn(0x56401b3440e0 msgr2=0x56401b3716f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:21.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.150+0000 7f7438a4f740 1 -- 172.21.15.88:0/352234625 shutdown_connections 2024-09-16T01:35:21.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.150+0000 7f7438a4f740 1 -- 172.21.15.88:0/352234625 wait complete. 2024-09-16T01:35:21.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.151+0000 7f7438a4f740 1 -- start start 2024-09-16T01:35:21.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.151+0000 7f7438a4f740 1 Processor -- start 2024-09-16T01:35:21.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f7438a4f740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56401b346920 0x56401b40a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f7438a4f740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b3480b0 0x56401b409150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f7438a4f740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56401b36fcf0 0x56401b38aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f7438a4f740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f743015fd10 con 0x56401b3480b0 2024-09-16T01:35:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f7438a4f740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56401b3afe00 con 0x56401b346920 2024-09-16T01:35:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f7438a4f740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56401b40a950 con 0x56401b36fcf0 2024-09-16T01:35:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f7438a4f740 10 client.0 _ll_register_callbacks cb 0x7f742400a6f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:35:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f742effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b3480b0 0x56401b409150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f742effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b3480b0 0x56401b409150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47910/0 (socket says 172.21.15.88:47910) 2024-09-16T01:35:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f742effd640 1 -- 172.21.15.88:0/1179814986 learned_addr learned my addr 172.21.15.88:0/1179814986 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f742ffff640 1 --2- 172.21.15.88:0/1179814986 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56401b36fcf0 0x56401b38aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f742effd640 1 -- 172.21.15.88:0/1179814986 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56401b36fcf0 msgr2=0x56401b38aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f742f7fe640 1 --2- 172.21.15.88:0/1179814986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56401b346920 0x56401b40a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f742effd640 1 --2- 172.21.15.88:0/1179814986 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56401b36fcf0 0x56401b38aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f742effd640 1 -- 172.21.15.88:0/1179814986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56401b346920 msgr2=0x56401b40a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f742effd640 1 --2- 172.21.15.88:0/1179814986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56401b346920 0x56401b40a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f742effd640 1 -- 172.21.15.88:0/1179814986 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56401b341530 con 0x56401b3480b0 2024-09-16T01:35:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f742f7fe640 1 --2- 172.21.15.88:0/1179814986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56401b346920 0x56401b40a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f742effd640 1 --2- 172.21.15.88:0/1179814986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b3480b0 0x56401b409150 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f7424014df0 tx=0x7f7424013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c30807b67c47f6cc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f740e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:35:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f740e7fc640 1 -- 172.21.15.88:0/1179814986 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7424037070 con 0x56401b3480b0 2024-09-16T01:35:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f740e7fc640 1 -- 172.21.15.88:0/1179814986 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f7424029650 con 0x56401b3480b0 2024-09-16T01:35:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f7438a4f740 1 -- 172.21.15.88:0/1179814986 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f742801a040 con 0x56401b3480b0 2024-09-16T01:35:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f740e7fc640 1 -- 172.21.15.88:0/1179814986 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f7424042050 con 0x56401b3480b0 2024-09-16T01:35:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f7438a4f740 10 client.5193 fetch_fsmap learned FSMap version 19 2024-09-16T01:35:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.152+0000 7f7438a4f740 1 -- 172.21.15.88:0/1179814986 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x56401b341100 con 0x56401b3480b0 2024-09-16T01:35:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.153+0000 7f740e7fc640 1 -- 172.21.15.88:0/1179814986 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f742801a040 con 0x56401b3480b0 2024-09-16T01:35:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.153+0000 7f740e7fc640 1 -- 172.21.15.88:0/1179814986 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 19) ==== 3502+0+0 (secure 0 0 0) 0x7f7424045020 con 0x56401b3480b0 2024-09-16T01:35:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.153+0000 7f7438a4f740 10 client.5193 fetch_fsmap finished waiting for FSMap version 19 2024-09-16T01:35:21.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.153+0000 7f7438a4f740 10 client.5193 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-09-16T01:35:21.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.153+0000 7f7438a4f740 20 client.5193 populate_metadata read hostname 'smithi088' 2024-09-16T01:35:21.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.153+0000 7f7438a4f740 1 --2- 172.21.15.88:0/1179814986 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x56401b33c7e0 0x56401b3400c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:21.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.153+0000 7f7438a4f740 4 client.5193 mds_command: new command op to 5067 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T01:35:21.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.153+0000 7f7438a4f740 1 -- 172.21.15.88:0/1179814986 --> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56401b2b1170 con 0x56401b33c7e0 2024-09-16T01:35:21.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.153+0000 7f742f7fe640 1 --2- 172.21.15.88:0/1179814986 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x56401b33c7e0 0x56401b3400c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:21.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.153+0000 7f740e7fc640 10 client.5193 ms_handle_connect on v2:172.21.15.88:6834/2870334334 2024-09-16T01:35:21.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.153+0000 7f742f7fe640 1 --2- 172.21.15.88:0/1179814986 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x56401b33c7e0 0x56401b3400c0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6a4b502943f62c2 server_cookie=1b870f2691cb7284 in_seq=0 out_seq=0 2024-09-16T01:35:21.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.153+0000 7f740e7fc640 1 -- 172.21.15.88:0/1179814986 <== mds.0 v2:172.21.15.88:6834/2870334334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+256 (crc 0 0 0) 0x56401b2b1170 con 0x56401b33c7e0 2024-09-16T01:35:21.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.153+0000 7f740e7fc640 10 client.5193 handle_command_reply: tid=0 2024-09-16T01:35:21.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.153+0000 7f7438a4f740 1 client.5193 shutdown 2024-09-16T01:35:21.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.153+0000 7f7438a4f740 1 -- 172.21.15.88:0/1179814986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b3480b0 msgr2=0x56401b409150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:21.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.153+0000 7f7438a4f740 1 --2- 172.21.15.88:0/1179814986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b3480b0 0x56401b409150 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f7424014df0 tx=0x7f7424013000 comp rx=0 tx=0).stop 2024-09-16T01:35:21.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.153+0000 7f7438a4f740 1 -- 172.21.15.88:0/1179814986 shutdown_connections 2024-09-16T01:35:21.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.153+0000 7f7438a4f740 1 --2- 172.21.15.88:0/1179814986 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x56401b33c7e0 0x56401b3400c0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.153+0000 7f7438a4f740 1 --2- 172.21.15.88:0/1179814986 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56401b36fcf0 0x56401b38aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.153+0000 7f7438a4f740 1 --2- 172.21.15.88:0/1179814986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56401b3480b0 0x56401b409150 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.153+0000 7f740e7fc640 0 client.5193 ms_handle_reset on v2:172.21.15.88:6834/2870334334 2024-09-16T01:35:21.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.153+0000 7f7438a4f740 1 --2- 172.21.15.88:0/1179814986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56401b346920 0x56401b40a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.153+0000 7f7438a4f740 1 -- 172.21.15.88:0/1179814986 >> 172.21.15.88:0/1179814986 conn(0x56401b3440e0 msgr2=0x56401b38b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:21.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.153+0000 7f7438a4f740 1 -- 172.21.15.88:0/1179814986 shutdown_connections 2024-09-16T01:35:21.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.154+0000 7f7438a4f740 1 -- 172.21.15.88:0/1179814986 wait complete. 2024-09-16T01:35:21.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.154+0000 7f7438a4f740 20 client.5193 trim_cache size 0 max 16384 2024-09-16T01:35:21.163 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:35:21.163 INFO:teuthology.orchestra.run.smithi088.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-16T01:35:21.163 INFO:teuthology.orchestra.run.smithi088.stdout: "scrubs": { 2024-09-16T01:35:21.163 INFO:teuthology.orchestra.run.smithi088.stdout: "9823f7b0-585e-4120-a79a-63d2185e414f": { 2024-09-16T01:35:21.163 INFO:teuthology.orchestra.run.smithi088.stdout: "path": "/subdir", 2024-09-16T01:35:21.163 INFO:teuthology.orchestra.run.smithi088.stdout: "tag": "9823f7b0-585e-4120-a79a-63d2185e414f", 2024-09-16T01:35:21.163 INFO:teuthology.orchestra.run.smithi088.stdout: "options": "repair" 2024-09-16T01:35:21.163 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:35:21.163 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:35:21.164 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:35:21.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.154+0000 7f7435cb7640 1 -- 172.21.15.88:0/4128225103 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f741c06c2b0 msgr2=0x7f741c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:21.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.154+0000 7f7435cb7640 1 --2- 172.21.15.88:0/4128225103 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f741c06c2b0 0x7f741c06e770 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f74300b58d0 tx=0x7f7424008040 comp rx=0 tx=0).stop 2024-09-16T01:35:21.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.154+0000 7f7435cb7640 1 -- 172.21.15.88:0/4128225103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f743006fda0 msgr2=0x7f74301f3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:21.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.154+0000 7f7435cb7640 1 --2- 172.21.15.88:0/4128225103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f743006fda0 0x7f74301f3a20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7420000c00 tx=0x7f74200046c0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.154+0000 7f7435cb7640 1 -- 172.21.15.88:0/4128225103 shutdown_connections 2024-09-16T01:35:21.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.154+0000 7f7435cb7640 1 --2- 172.21.15.88:0/4128225103 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f741c06c2b0 0x7f741c06e770 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.154+0000 7f7435cb7640 1 --2- 172.21.15.88:0/4128225103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f743015e760 0x7f74301f7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.154+0000 7f7435cb7640 1 --2- 172.21.15.88:0/4128225103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74300b59a0 0x7f74301f77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.154+0000 7f7435cb7640 1 --2- 172.21.15.88:0/4128225103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f743006fda0 0x7f74301f3a20 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:21.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.154+0000 7f7435cb7640 1 -- 172.21.15.88:0/4128225103 >> 172.21.15.88:0/4128225103 conn(0x7f74300b7d70 msgr2=0x7f74300b6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:21.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.154+0000 7f7435cb7640 1 -- 172.21.15.88:0/4128225103 shutdown_connections 2024-09-16T01:35:21.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:21.154+0000 7f7435cb7640 1 -- 172.21.15.88:0/4128225103 wait complete. 2024-09-16T01:35:21.166 INFO:tasks.cephfs.filesystem:scrub status for tag:9823f7b0-585e-4120-a79a-63d2185e414f - {'path': '/subdir', 'tag': '9823f7b0-585e-4120-a79a-63d2185e414f', 'options': 'repair'} 2024-09-16T01:35:46.318 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:35:46.321 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:35:51.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:51.167 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.2:0 scrub status 2024-09-16T01:35:51.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.278+0000 7fd12179f640 1 Processor -- start 2024-09-16T01:35:51.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.278+0000 7fd12179f640 1 -- start start 2024-09-16T01:35:51.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.278+0000 7fd12179f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd11c15e7a0 0x7fd11c06caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.278+0000 7fd12179f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd11c0b68f0 0x7fd11c0d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.278+0000 7fd12179f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd11c0cf860 con 0x7fd11c0cf1f0 2024-09-16T01:35:51.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.278+0000 7fd12179f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd11c0d2ca0 con 0x7fd11c15e7a0 2024-09-16T01:35:51.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.278+0000 7fd12179f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd11c0cfb40 con 0x7fd11c0b68f0 2024-09-16T01:35:51.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.279+0000 7fd11affd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd11c15e7a0 0x7fd11c06caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:51.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.279+0000 7fd11a7fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd11c0cf1f0 0x7fd11c0b61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:48778/0 (socket says 172.21.15.88:48778) 2024-09-16T01:35:51.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.279+0000 7fd11a7fc640 1 -- 172.21.15.88:0/267539671 learned_addr learned my addr 172.21.15.88:0/267539671 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:51.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.279+0000 7fd11affd640 1 -- 172.21.15.88:0/267539671 >> v2:172.21.15.154:3300/0 conn(0x7fd11c0b68f0 msgr2=0x7fd11c0d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:51.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.279+0000 7fd11affd640 1 --2- 172.21.15.88:0/267539671 >> v2:172.21.15.154:3300/0 conn(0x7fd11c0b68f0 0x7fd11c0d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.279+0000 7fd11affd640 1 -- 172.21.15.88:0/267539671 >> v1:172.21.15.88:6789/0 conn(0x7fd11c0cf1f0 legacy=0x7fd11c0b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:51.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.279+0000 7fd11affd640 1 -- 172.21.15.88:0/267539671 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd11c0cfe20 con 0x7fd11c15e7a0 2024-09-16T01:35:51.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.279+0000 7fd119ffb640 1 -- 172.21.15.88:0/267539671 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 883933109 0 0) 0x7fd11c0cf860 con 0x7fd11c0cf1f0 2024-09-16T01:35:51.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.280+0000 7fd11affd640 1 --2- 172.21.15.88:0/267539671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd11c15e7a0 0x7fd11c06caa0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd11c08d930 tx=0x7fd11402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5703563fdc1e223 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:51.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.280+0000 7fd119ffb640 1 -- 172.21.15.88:0/267539671 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd114007740 con 0x7fd11c15e7a0 2024-09-16T01:35:51.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.280+0000 7fd119ffb640 1 -- 172.21.15.88:0/267539671 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1140078e0 con 0x7fd11c15e7a0 2024-09-16T01:35:51.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.280+0000 7fd119ffb640 1 -- 172.21.15.88:0/267539671 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd114007cb0 con 0x7fd11c15e7a0 2024-09-16T01:35:51.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.280+0000 7fd12179f640 1 -- 172.21.15.88:0/267539671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd11c15e7a0 msgr2=0x7fd11c06caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:51.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.280+0000 7fd12179f640 1 --2- 172.21.15.88:0/267539671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd11c15e7a0 0x7fd11c06caa0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd11c08d930 tx=0x7fd11402a610 comp rx=0 tx=0).stop 2024-09-16T01:35:51.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.280+0000 7fd12179f640 1 -- 172.21.15.88:0/267539671 shutdown_connections 2024-09-16T01:35:51.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.280+0000 7fd12179f640 1 --2- 172.21.15.88:0/267539671 >> v2:172.21.15.154:3300/0 conn(0x7fd11c0b68f0 0x7fd11c0d3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.280+0000 7fd12179f640 1 --2- 172.21.15.88:0/267539671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd11c15e7a0 0x7fd11c06caa0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.280+0000 7fd12179f640 1 -- 172.21.15.88:0/267539671 >> 172.21.15.88:0/267539671 conn(0x7fd11c0d11e0 msgr2=0x7fd11c0d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:51.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.280+0000 7fd12179f640 1 -- 172.21.15.88:0/267539671 shutdown_connections 2024-09-16T01:35:51.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.280+0000 7fd12179f640 1 -- 172.21.15.88:0/267539671 wait complete. 2024-09-16T01:35:51.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.282+0000 7fd12179f640 1 Processor -- start 2024-09-16T01:35:51.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.282+0000 7fd12179f640 1 -- start start 2024-09-16T01:35:51.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.283+0000 7fd12179f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd11c0b68f0 0x7fd11c1594d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.283+0000 7fd12179f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd11c0cf1f0 0x7fd11c0cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.283+0000 7fd12179f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd11c15e7a0 0x7fd11c15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.283+0000 7fd12179f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd11c06cfe0 con 0x7fd11c15e7a0 2024-09-16T01:35:51.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.283+0000 7fd12179f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd11c0b7a00 con 0x7fd11c0b68f0 2024-09-16T01:35:51.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.283+0000 7fd12179f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd11c15eb80 con 0x7fd11c0cf1f0 2024-09-16T01:35:51.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.283+0000 7fd11b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd11c15e7a0 0x7fd11c15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:51.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.283+0000 7fd11b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd11c15e7a0 0x7fd11c15ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40358/0 (socket says 172.21.15.88:40358) 2024-09-16T01:35:51.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.283+0000 7fd11b7fe640 1 -- 172.21.15.88:0/3329250066 learned_addr learned my addr 172.21.15.88:0/3329250066 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:51.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.283+0000 7fd11a7fc640 1 --2- 172.21.15.88:0/3329250066 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd11c0cf1f0 0x7fd11c0cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:51.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.283+0000 7fd11affd640 1 --2- 172.21.15.88:0/3329250066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd11c0b68f0 0x7fd11c1594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:51.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.283+0000 7fd11b7fe640 1 -- 172.21.15.88:0/3329250066 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd11c0cf1f0 msgr2=0x7fd11c0cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:51.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.283+0000 7fd11b7fe640 1 --2- 172.21.15.88:0/3329250066 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd11c0cf1f0 0x7fd11c0cacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.283+0000 7fd11b7fe640 1 -- 172.21.15.88:0/3329250066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd11c0b68f0 msgr2=0x7fd11c1594d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:51.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.283+0000 7fd11b7fe640 1 --2- 172.21.15.88:0/3329250066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd11c0b68f0 0x7fd11c1594d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.283+0000 7fd11b7fe640 1 -- 172.21.15.88:0/3329250066 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd11c0c9c50 con 0x7fd11c15e7a0 2024-09-16T01:35:51.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.283+0000 7fd11a7fc640 1 --2- 172.21.15.88:0/3329250066 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd11c0cf1f0 0x7fd11c0cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:51.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.283+0000 7fd11affd640 1 --2- 172.21.15.88:0/3329250066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd11c0b68f0 0x7fd11c1594d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:51.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.283+0000 7fd11b7fe640 1 --2- 172.21.15.88:0/3329250066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd11c15e7a0 0x7fd11c15ef90 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd10c00b3c0 tx=0x7fd10c00b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:51.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.283+0000 7fd1137fe640 1 -- 172.21.15.88:0/3329250066 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd10c011830 con 0x7fd11c15e7a0 2024-09-16T01:35:51.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.283+0000 7fd1137fe640 1 -- 172.21.15.88:0/3329250066 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd10c016070 con 0x7fd11c15e7a0 2024-09-16T01:35:51.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.283+0000 7fd12179f640 1 -- 172.21.15.88:0/3329250066 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd11c0c9f30 con 0x7fd11c15e7a0 2024-09-16T01:35:51.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.283+0000 7fd1137fe640 1 -- 172.21.15.88:0/3329250066 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd10c011c30 con 0x7fd11c15e7a0 2024-09-16T01:35:51.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.283+0000 7fd12179f640 1 -- 172.21.15.88:0/3329250066 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd11c0ca4f0 con 0x7fd11c15e7a0 2024-09-16T01:35:51.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.286+0000 7fd1137fe640 1 -- 172.21.15.88:0/3329250066 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd10c01b460 con 0x7fd11c15e7a0 2024-09-16T01:35:51.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.286+0000 7fd1137fe640 1 --2- 172.21.15.88:0/3329250066 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd0fc06c2b0 0x7fd0fc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.286+0000 7fd1137fe640 1 -- 172.21.15.88:0/3329250066 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fd10c08b0d0 con 0x7fd11c15e7a0 2024-09-16T01:35:51.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.286+0000 7fd11affd640 1 --2- 172.21.15.88:0/3329250066 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd0fc06c2b0 0x7fd0fc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:51.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.286+0000 7fd11affd640 1 --2- 172.21.15.88:0/3329250066 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd0fc06c2b0 0x7fd0fc06e770 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd114034000 tx=0x7fd11402f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:51.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.287+0000 7fd12461c740 1 Processor -- start 2024-09-16T01:35:51.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.287+0000 7fd12461c740 1 -- start start 2024-09-16T01:35:51.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.288+0000 7fd12461c740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b5244f920 0x563b524950c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.288+0000 7fd12461c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b5244d0e0 0x563b5244fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.288+0000 7fd12461c740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563b524510b0 0x563b5244a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.288+0000 7fd12461c740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x563b523b26f0 con 0x563b5244f920 2024-09-16T01:35:51.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.288+0000 7fd12461c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563b5235a330 con 0x563b5244d0e0 2024-09-16T01:35:51.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.288+0000 7fd12461c740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x563b523ae6b0 con 0x563b524510b0 2024-09-16T01:35:51.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.288+0000 7fd11affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b5244f920 0x563b524950c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:51.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.288+0000 7fd11affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b5244f920 0x563b524950c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40370/0 (socket says 172.21.15.88:40370) 2024-09-16T01:35:51.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.288+0000 7fd11affd640 1 -- 172.21.15.88:0/1040551969 learned_addr learned my addr 172.21.15.88:0/1040551969 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:51.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.288+0000 7fd11b7fe640 1 --2- 172.21.15.88:0/1040551969 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563b524510b0 0x563b5244a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:51.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.288+0000 7fd11a7fc640 1 --2- 172.21.15.88:0/1040551969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b5244d0e0 0x563b5244fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:51.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.288+0000 7fd11affd640 1 -- 172.21.15.88:0/1040551969 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563b524510b0 msgr2=0x563b5244a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:51.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.288+0000 7fd11affd640 1 --2- 172.21.15.88:0/1040551969 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563b524510b0 0x563b5244a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.288+0000 7fd11affd640 1 -- 172.21.15.88:0/1040551969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b5244d0e0 msgr2=0x563b5244fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:51.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.288+0000 7fd11affd640 1 --2- 172.21.15.88:0/1040551969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b5244d0e0 0x563b5244fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.288+0000 7fd11affd640 1 -- 172.21.15.88:0/1040551969 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b523800d0 con 0x563b5244f920 2024-09-16T01:35:51.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.288+0000 7fd11a7fc640 1 --2- 172.21.15.88:0/1040551969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b5244d0e0 0x563b5244fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:51.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.288+0000 7fd11affd640 1 --2- 172.21.15.88:0/1040551969 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b5244f920 0x563b524950c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd114007c80 tx=0x7fd11402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3466b233b319c61 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:51.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.288+0000 7fd110ff9640 1 -- 172.21.15.88:0/1040551969 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11404d070 con 0x563b5244f920 2024-09-16T01:35:51.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.288+0000 7fd110ff9640 1 -- 172.21.15.88:0/1040551969 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd11403d9b0 con 0x563b5244f920 2024-09-16T01:35:51.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.288+0000 7fd110ff9640 1 -- 172.21.15.88:0/1040551969 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11403dd60 con 0x563b5244f920 2024-09-16T01:35:51.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.289+0000 7fd12461c740 1 -- 172.21.15.88:0/1040551969 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b5244f920 msgr2=0x563b524950c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:51.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.289+0000 7fd12461c740 1 --2- 172.21.15.88:0/1040551969 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b5244f920 0x563b524950c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd114007c80 tx=0x7fd11402e000 comp rx=0 tx=0).stop 2024-09-16T01:35:51.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.289+0000 7fd12461c740 1 -- 172.21.15.88:0/1040551969 shutdown_connections 2024-09-16T01:35:51.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.289+0000 7fd12461c740 1 --2- 172.21.15.88:0/1040551969 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563b524510b0 0x563b5244a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.289+0000 7fd12461c740 1 --2- 172.21.15.88:0/1040551969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b5244d0e0 0x563b5244fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.289+0000 7fd12461c740 1 --2- 172.21.15.88:0/1040551969 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b5244f920 0x563b524950c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.289+0000 7fd12461c740 1 -- 172.21.15.88:0/1040551969 >> 172.21.15.88:0/1040551969 conn(0x563b5244c370 msgr2=0x563b52494590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:51.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.289+0000 7fd12461c740 1 -- 172.21.15.88:0/1040551969 shutdown_connections 2024-09-16T01:35:51.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.289+0000 7fd12461c740 1 -- 172.21.15.88:0/1040551969 wait complete. 2024-09-16T01:35:51.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.290+0000 7fd12461c740 1 -- start start 2024-09-16T01:35:51.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.290+0000 7fd12461c740 1 Processor -- start 2024-09-16T01:35:51.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.291+0000 7fd12461c740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563b5244f920 0x563b524a4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.291+0000 7fd12461c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b524510b0 0x563b5244acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.291+0000 7fd12461c740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b524758c0 0x563b52475d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.291+0000 7fd12461c740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x563b5244b810 con 0x563b524758c0 2024-09-16T01:35:51.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.291+0000 7fd12461c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563b5233b4f0 con 0x563b524510b0 2024-09-16T01:35:51.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.291+0000 7fd12461c740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x563b524958f0 con 0x563b5244f920 2024-09-16T01:35:51.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.291+0000 7fd12461c740 10 client.0 _ll_register_callbacks cb 0x563b5235cb50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:35:51.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.291+0000 7fd11b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b524758c0 0x563b52475d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:51.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.291+0000 7fd11b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b524758c0 0x563b52475d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40384/0 (socket says 172.21.15.88:40384) 2024-09-16T01:35:51.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.291+0000 7fd11b7fe640 1 -- 172.21.15.88:0/2761042457 learned_addr learned my addr 172.21.15.88:0/2761042457 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:51.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.291+0000 7fd11affd640 1 --2- 172.21.15.88:0/2761042457 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563b5244f920 0x563b524a4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:51.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.291+0000 7fd11a7fc640 1 --2- 172.21.15.88:0/2761042457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b524510b0 0x563b5244acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:51.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.291+0000 7fd11b7fe640 1 -- 172.21.15.88:0/2761042457 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563b5244f920 msgr2=0x563b524a4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:51.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.291+0000 7fd11b7fe640 1 --2- 172.21.15.88:0/2761042457 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563b5244f920 0x563b524a4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.291+0000 7fd11b7fe640 1 -- 172.21.15.88:0/2761042457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b524510b0 msgr2=0x563b5244acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:51.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.291+0000 7fd11b7fe640 1 --2- 172.21.15.88:0/2761042457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b524510b0 0x563b5244acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.291+0000 7fd11b7fe640 1 -- 172.21.15.88:0/2761042457 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b5218f7d0 con 0x563b524758c0 2024-09-16T01:35:51.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.291+0000 7fd11b7fe640 1 --2- 172.21.15.88:0/2761042457 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b524758c0 0x563b52475d40 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd10c0555d0 tx=0x7fd10c05d5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acefeb86c2e2a6f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:51.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.291+0000 7fd0f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:35:51.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.291+0000 7fd0f9ffb640 1 -- 172.21.15.88:0/2761042457 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd10c0711d0 con 0x563b524758c0 2024-09-16T01:35:51.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.291+0000 7fd0f9ffb640 1 -- 172.21.15.88:0/2761042457 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fd10c0618c0 con 0x563b524758c0 2024-09-16T01:35:51.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.291+0000 7fd12461c740 1 -- 172.21.15.88:0/2761042457 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd11403d9b0 con 0x563b524758c0 2024-09-16T01:35:51.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.292+0000 7fd0f9ffb640 1 -- 172.21.15.88:0/2761042457 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fd10c089050 con 0x563b524758c0 2024-09-16T01:35:51.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.292+0000 7fd12461c740 10 client.5208 fetch_fsmap learned FSMap version 19 2024-09-16T01:35:51.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.292+0000 7fd12461c740 1 -- 172.21.15.88:0/2761042457 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x563b520c53f0 con 0x563b524758c0 2024-09-16T01:35:51.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.292+0000 7fd0f9ffb640 1 -- 172.21.15.88:0/2761042457 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd11403d9b0 con 0x563b524758c0 2024-09-16T01:35:51.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.292+0000 7fd0f9ffb640 1 -- 172.21.15.88:0/2761042457 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 19) ==== 3502+0+0 (secure 0 0 0) 0x7fd10c0585c0 con 0x563b524758c0 2024-09-16T01:35:51.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.292+0000 7fd12461c740 10 client.5208 fetch_fsmap finished waiting for FSMap version 19 2024-09-16T01:35:51.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.292+0000 7fd12461c740 10 client.5208 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-09-16T01:35:51.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.292+0000 7fd12461c740 20 client.5208 populate_metadata read hostname 'smithi088' 2024-09-16T01:35:51.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.292+0000 7fd12461c740 1 --2- 172.21.15.88:0/2761042457 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x563b524a8640 0x563b524a8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.292+0000 7fd12461c740 4 client.5208 mds_command: new command op to 5067 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:35:51.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.292+0000 7fd12461c740 1 -- 172.21.15.88:0/2761042457 --> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b523bfe20 con 0x563b524a8640 2024-09-16T01:35:51.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.292+0000 7fd11affd640 1 --2- 172.21.15.88:0/2761042457 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x563b524a8640 0x563b524a8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:51.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.292+0000 7fd0f9ffb640 10 client.5208 ms_handle_connect on v2:172.21.15.88:6834/2870334334 2024-09-16T01:35:51.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.292+0000 7fd11affd640 1 --2- 172.21.15.88:0/2761042457 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x563b524a8640 0x563b524a8a20 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca273c6283413bd2 server_cookie=9d57d7ab6ec38703 in_seq=0 out_seq=0 2024-09-16T01:35:51.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.293+0000 7fd0f9ffb640 1 -- 172.21.15.88:0/2761042457 <== mds.0 v2:172.21.15.88:6834/2870334334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563b523bfe20 con 0x563b524a8640 2024-09-16T01:35:51.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.293+0000 7fd0f9ffb640 10 client.5208 handle_command_reply: tid=0 2024-09-16T01:35:51.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.293+0000 7fd12461c740 1 client.5208 shutdown 2024-09-16T01:35:51.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.293+0000 7fd12461c740 1 -- 172.21.15.88:0/2761042457 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b524758c0 msgr2=0x563b52475d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:51.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.293+0000 7fd12461c740 1 --2- 172.21.15.88:0/2761042457 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b524758c0 0x563b52475d40 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd10c0555d0 tx=0x7fd10c05d5e0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.293+0000 7fd12461c740 1 -- 172.21.15.88:0/2761042457 shutdown_connections 2024-09-16T01:35:51.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.293+0000 7fd12461c740 1 --2- 172.21.15.88:0/2761042457 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x563b524a8640 0x563b524a8a20 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.293+0000 7fd12461c740 1 --2- 172.21.15.88:0/2761042457 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b524758c0 0x563b52475d40 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.293+0000 7fd12461c740 1 --2- 172.21.15.88:0/2761042457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b524510b0 0x563b5244acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.293+0000 7fd12461c740 1 --2- 172.21.15.88:0/2761042457 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563b5244f920 0x563b524a4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.293+0000 7fd0f9ffb640 0 client.5208 ms_handle_reset on v2:172.21.15.88:6834/2870334334 2024-09-16T01:35:51.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.293+0000 7fd12461c740 1 -- 172.21.15.88:0/2761042457 >> 172.21.15.88:0/2761042457 conn(0x563b5244d0e0 msgr2=0x563b52493ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:51.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.293+0000 7fd12461c740 1 -- 172.21.15.88:0/2761042457 shutdown_connections 2024-09-16T01:35:51.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.293+0000 7fd12461c740 1 -- 172.21.15.88:0/2761042457 wait complete. 2024-09-16T01:35:51.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.293+0000 7fd12461c740 20 client.5208 trim_cache size 0 max 16384 2024-09-16T01:35:51.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.310+0000 7fd12461c740 1 Processor -- start 2024-09-16T01:35:51.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.310+0000 7fd12461c740 1 -- start start 2024-09-16T01:35:51.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.311+0000 7fd12461c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b5244f920 0x563b524b8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.311+0000 7fd12461c740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b524510b0 0x563b5244ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.311+0000 7fd12461c740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563b524758c0 0x563b524780d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.311+0000 7fd12461c740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd11c200ac0 con 0x563b524510b0 2024-09-16T01:35:51.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.311+0000 7fd12461c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd11c1ffb90 con 0x563b5244f920 2024-09-16T01:35:51.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.311+0000 7fd12461c740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd11c0c9000 con 0x563b524758c0 2024-09-16T01:35:51.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.311+0000 7fd11a7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b524510b0 0x563b5244ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:51.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.311+0000 7fd11a7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b524510b0 0x563b5244ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40396/0 (socket says 172.21.15.88:40396) 2024-09-16T01:35:51.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.311+0000 7fd11a7fc640 1 -- 172.21.15.88:0/56723462 learned_addr learned my addr 172.21.15.88:0/56723462 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:51.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.311+0000 7fd11a7fc640 1 -- 172.21.15.88:0/56723462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563b524758c0 msgr2=0x563b524780d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:51.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.311+0000 7fd11b7fe640 1 --2- 172.21.15.88:0/56723462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563b524758c0 0x563b524780d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:51.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.311+0000 7fd11affd640 1 --2- 172.21.15.88:0/56723462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b5244f920 0x563b524b8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:51.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.311+0000 7fd11a7fc640 1 --2- 172.21.15.88:0/56723462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563b524758c0 0x563b524780d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.311+0000 7fd11a7fc640 1 -- 172.21.15.88:0/56723462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b5244f920 msgr2=0x563b524b8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:51.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.311+0000 7fd11a7fc640 1 --2- 172.21.15.88:0/56723462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b5244f920 0x563b524b8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.311+0000 7fd11a7fc640 1 -- 172.21.15.88:0/56723462 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b520850f0 con 0x563b524510b0 2024-09-16T01:35:51.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.311+0000 7fd11affd640 1 --2- 172.21.15.88:0/56723462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b5244f920 0x563b524b8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:51.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.312+0000 7fd11a7fc640 1 --2- 172.21.15.88:0/56723462 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b524510b0 0x563b5244ec70 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fd104007540 tx=0x7fd104005bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11f4949dd8c023be server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:51.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.312+0000 7fd0fbfff640 1 -- 172.21.15.88:0/56723462 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd10401a020 con 0x563b524510b0 2024-09-16T01:35:51.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.312+0000 7fd0fbfff640 1 -- 172.21.15.88:0/56723462 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd104005cf0 con 0x563b524510b0 2024-09-16T01:35:51.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.312+0000 7fd0fbfff640 1 -- 172.21.15.88:0/56723462 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd104001a30 con 0x563b524510b0 2024-09-16T01:35:51.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.312+0000 7fd12461c740 1 -- 172.21.15.88:0/56723462 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b524510b0 msgr2=0x563b5244ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:51.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.312+0000 7fd12461c740 1 --2- 172.21.15.88:0/56723462 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b524510b0 0x563b5244ec70 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fd104007540 tx=0x7fd104005bc0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.312+0000 7fd12461c740 1 -- 172.21.15.88:0/56723462 shutdown_connections 2024-09-16T01:35:51.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.312+0000 7fd12461c740 1 --2- 172.21.15.88:0/56723462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563b524758c0 0x563b524780d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.312+0000 7fd12461c740 1 --2- 172.21.15.88:0/56723462 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b524510b0 0x563b5244ec70 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.312+0000 7fd12461c740 1 --2- 172.21.15.88:0/56723462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b5244f920 0x563b524b8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.312+0000 7fd12461c740 1 -- 172.21.15.88:0/56723462 >> 172.21.15.88:0/56723462 conn(0x563b5244d0e0 msgr2=0x563b5246ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:51.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.312+0000 7fd12461c740 1 -- 172.21.15.88:0/56723462 shutdown_connections 2024-09-16T01:35:51.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.312+0000 7fd12461c740 1 -- 172.21.15.88:0/56723462 wait complete. 2024-09-16T01:35:51.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.313+0000 7fd12461c740 1 -- start start 2024-09-16T01:35:51.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.313+0000 7fd12461c740 1 Processor -- start 2024-09-16T01:35:51.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd12461c740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b5244f920 0x563b52491ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd12461c740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563b524510b0 0x563b52517210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd12461c740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b524758c0 0x563b52474ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd12461c740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x563b52476e70 con 0x563b5244f920 2024-09-16T01:35:51.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd12461c740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd11c0c9650 con 0x563b524758c0 2024-09-16T01:35:51.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd12461c740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x563b524a3df0 con 0x563b524510b0 2024-09-16T01:35:51.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd12461c740 10 client.0 _ll_register_callbacks cb 0x7fd10c058b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:35:51.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd11affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b5244f920 0x563b52491ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:51.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd11affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b5244f920 0x563b52491ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40398/0 (socket says 172.21.15.88:40398) 2024-09-16T01:35:51.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd11affd640 1 -- 172.21.15.88:0/2725837685 learned_addr learned my addr 172.21.15.88:0/2725837685 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:51.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd11a7fc640 1 --2- 172.21.15.88:0/2725837685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563b524510b0 0x563b52517210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:51.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd11b7fe640 1 --2- 172.21.15.88:0/2725837685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b524758c0 0x563b52474ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:51.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd11affd640 1 -- 172.21.15.88:0/2725837685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563b524510b0 msgr2=0x563b52517210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:51.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd11affd640 1 --2- 172.21.15.88:0/2725837685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563b524510b0 0x563b52517210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd11affd640 1 -- 172.21.15.88:0/2725837685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b524758c0 msgr2=0x563b52474ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:51.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd11affd640 1 --2- 172.21.15.88:0/2725837685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b524758c0 0x563b52474ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd11affd640 1 -- 172.21.15.88:0/2725837685 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b52475210 con 0x563b5244f920 2024-09-16T01:35:51.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd11a7fc640 1 --2- 172.21.15.88:0/2725837685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563b524510b0 0x563b52517210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:51.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd11affd640 1 --2- 172.21.15.88:0/2725837685 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b5244f920 0x563b52491ae0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd114009bd0 tx=0x7fd114004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0c7bc6a6f2a0adf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:51.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd0f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:35:51.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd0f9ffb640 1 -- 172.21.15.88:0/2725837685 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd114065060 con 0x563b5244f920 2024-09-16T01:35:51.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd0f9ffb640 1 -- 172.21.15.88:0/2725837685 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fd11403dbb0 con 0x563b5244f920 2024-09-16T01:35:51.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd12461c740 1 -- 172.21.15.88:0/2725837685 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd104005cf0 con 0x563b5244f920 2024-09-16T01:35:51.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd0f9ffb640 1 -- 172.21.15.88:0/2725837685 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fd11406b050 con 0x563b5244f920 2024-09-16T01:35:51.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd12461c740 10 client.5214 fetch_fsmap learned FSMap version 19 2024-09-16T01:35:51.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.314+0000 7fd12461c740 1 -- 172.21.15.88:0/2725837685 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x563b52513d00 con 0x563b5244f920 2024-09-16T01:35:51.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.315+0000 7fd0f9ffb640 1 -- 172.21.15.88:0/2725837685 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd104005cf0 con 0x563b5244f920 2024-09-16T01:35:51.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.315+0000 7fd0f9ffb640 1 -- 172.21.15.88:0/2725837685 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 19) ==== 3502+0+0 (secure 0 0 0) 0x7fd11406e020 con 0x563b5244f920 2024-09-16T01:35:51.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.315+0000 7fd12461c740 10 client.5214 fetch_fsmap finished waiting for FSMap version 19 2024-09-16T01:35:51.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.315+0000 7fd12461c740 10 client.5214 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-09-16T01:35:51.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.315+0000 7fd12461c740 20 client.5214 populate_metadata read hostname 'smithi088' 2024-09-16T01:35:51.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.315+0000 7fd12461c740 1 --2- 172.21.15.88:0/2725837685 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x563b52509cc0 0x563b52513190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.315+0000 7fd12461c740 4 client.5214 mds_command: new command op to 5067 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-16T01:35:51.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.315+0000 7fd12461c740 1 -- 172.21.15.88:0/2725837685 --> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563b523ba170 con 0x563b52509cc0 2024-09-16T01:35:51.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.315+0000 7fd11a7fc640 1 --2- 172.21.15.88:0/2725837685 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x563b52509cc0 0x563b52513190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:51.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.315+0000 7fd0f9ffb640 10 client.5214 ms_handle_connect on v2:172.21.15.88:6834/2870334334 2024-09-16T01:35:51.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.315+0000 7fd11a7fc640 1 --2- 172.21.15.88:0/2725837685 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x563b52509cc0 0x563b52513190 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca582e499d305dca server_cookie=695c112711659e97 in_seq=0 out_seq=0 2024-09-16T01:35:51.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.315+0000 7fd0f9ffb640 1 -- 172.21.15.88:0/2725837685 <== mds.0 v2:172.21.15.88:6834/2870334334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563b523ba170 con 0x563b52509cc0 2024-09-16T01:35:51.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.315+0000 7fd0f9ffb640 10 client.5214 handle_command_reply: tid=0 2024-09-16T01:35:51.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.315+0000 7fd12461c740 1 client.5214 shutdown 2024-09-16T01:35:51.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.315+0000 7fd12461c740 1 -- 172.21.15.88:0/2725837685 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b5244f920 msgr2=0x563b52491ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:51.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.315+0000 7fd12461c740 1 --2- 172.21.15.88:0/2725837685 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b5244f920 0x563b52491ae0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd114009bd0 tx=0x7fd114004ab0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.315+0000 7fd12461c740 1 -- 172.21.15.88:0/2725837685 shutdown_connections 2024-09-16T01:35:51.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.315+0000 7fd12461c740 1 --2- 172.21.15.88:0/2725837685 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x563b52509cc0 0x563b52513190 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.315+0000 7fd12461c740 1 --2- 172.21.15.88:0/2725837685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b524758c0 0x563b52474ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.315+0000 7fd12461c740 1 --2- 172.21.15.88:0/2725837685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563b524510b0 0x563b52517210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.315+0000 7fd0f9ffb640 0 client.5214 ms_handle_reset on v2:172.21.15.88:6834/2870334334 2024-09-16T01:35:51.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.315+0000 7fd12461c740 1 --2- 172.21.15.88:0/2725837685 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563b5244f920 0x563b52491ae0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.315+0000 7fd12461c740 1 -- 172.21.15.88:0/2725837685 >> 172.21.15.88:0/2725837685 conn(0x563b5244d0e0 msgr2=0x563b52493d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:51.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.316+0000 7fd12461c740 1 -- 172.21.15.88:0/2725837685 shutdown_connections 2024-09-16T01:35:51.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.316+0000 7fd12461c740 1 -- 172.21.15.88:0/2725837685 wait complete. 2024-09-16T01:35:51.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.316+0000 7fd12461c740 20 client.5214 trim_cache size 0 max 16384 2024-09-16T01:35:51.324 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:35:51.324 INFO:teuthology.orchestra.run.smithi088.stdout: "status": "no active scrubs running", 2024-09-16T01:35:51.324 INFO:teuthology.orchestra.run.smithi088.stdout: "scrubs": {} 2024-09-16T01:35:51.324 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:35:51.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.316+0000 7fd12179f640 1 -- 172.21.15.88:0/3329250066 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd0fc06c2b0 msgr2=0x7fd0fc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:51.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.316+0000 7fd12179f640 1 --2- 172.21.15.88:0/3329250066 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd0fc06c2b0 0x7fd0fc06e770 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd114034000 tx=0x7fd11402f000 comp rx=0 tx=0).stop 2024-09-16T01:35:51.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.316+0000 7fd12179f640 1 -- 172.21.15.88:0/3329250066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd11c15e7a0 msgr2=0x7fd11c15ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:51.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.316+0000 7fd12179f640 1 --2- 172.21.15.88:0/3329250066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd11c15e7a0 0x7fd11c15ef90 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd10c00b3c0 tx=0x7fd10c00b880 comp rx=0 tx=0).stop 2024-09-16T01:35:51.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.316+0000 7fd12179f640 1 -- 172.21.15.88:0/3329250066 shutdown_connections 2024-09-16T01:35:51.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.316+0000 7fd12179f640 1 --2- 172.21.15.88:0/3329250066 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd0fc06c2b0 0x7fd0fc06e770 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.316+0000 7fd12179f640 1 --2- 172.21.15.88:0/3329250066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd11c15e7a0 0x7fd11c15ef90 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.316+0000 7fd12179f640 1 --2- 172.21.15.88:0/3329250066 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd11c0cf1f0 0x7fd11c0cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.316+0000 7fd12179f640 1 --2- 172.21.15.88:0/3329250066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd11c0b68f0 0x7fd11c1594d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.316+0000 7fd12179f640 1 -- 172.21.15.88:0/3329250066 >> 172.21.15.88:0/3329250066 conn(0x7fd11c0d11e0 msgr2=0x7fd11c15da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:51.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.316+0000 7fd12179f640 1 -- 172.21.15.88:0/3329250066 shutdown_connections 2024-09-16T01:35:51.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.316+0000 7fd12179f640 1 -- 172.21.15.88:0/3329250066 wait complete. 2024-09-16T01:35:51.329 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-16T01:35:51.329 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:35:51.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.426+0000 7efd98eed640 1 Processor -- start 2024-09-16T01:35:51.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.426+0000 7efd98eed640 1 -- start start 2024-09-16T01:35:51.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.427+0000 7efd98eed640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efd94155800 0x7efd940b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.427+0000 7efd98eed640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efd9415c330 0x7efd940b4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.427+0000 7efd98eed640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd940b50a0 con 0x7efd940cf570 2024-09-16T01:35:51.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.427+0000 7efd98eed640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7efd940b4e60 con 0x7efd94155800 2024-09-16T01:35:51.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.427+0000 7efd98eed640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7efd9415c7a0 con 0x7efd9415c330 2024-09-16T01:35:51.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.427+0000 7efd9259c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efd94155800 0x7efd940b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:51.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.427+0000 7efd92d9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7efd940cf570 0x7efd940cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:48780/0 (socket says 172.21.15.88:48780) 2024-09-16T01:35:51.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.427+0000 7efd92d9d640 1 -- 172.21.15.88:0/560740241 learned_addr learned my addr 172.21.15.88:0/560740241 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:51.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.427+0000 7efd9259c640 1 -- 172.21.15.88:0/560740241 >> v2:172.21.15.154:3300/0 conn(0x7efd9415c330 msgr2=0x7efd940b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:51.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.427+0000 7efd9259c640 1 --2- 172.21.15.88:0/560740241 >> v2:172.21.15.154:3300/0 conn(0x7efd9415c330 0x7efd940b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.427+0000 7efd9259c640 1 -- 172.21.15.88:0/560740241 >> v1:172.21.15.88:6789/0 conn(0x7efd940cf570 legacy=0x7efd940cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:51.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.427+0000 7efd9159a640 1 -- 172.21.15.88:0/560740241 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3062877550 0 0) 0x7efd940b50a0 con 0x7efd940cf570 2024-09-16T01:35:51.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.427+0000 7efd9259c640 1 -- 172.21.15.88:0/560740241 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd940b5380 con 0x7efd94155800 2024-09-16T01:35:51.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.428+0000 7efd9259c640 1 --2- 172.21.15.88:0/560740241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd94155800 0x7efd940b6140 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7efd9406e210 tx=0x7efd8402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb388ed78f745e06 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:51.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.428+0000 7efd9159a640 1 -- 172.21.15.88:0/560740241 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd84037070 con 0x7efd94155800 2024-09-16T01:35:51.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.428+0000 7efd9159a640 1 -- 172.21.15.88:0/560740241 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd84032440 con 0x7efd94155800 2024-09-16T01:35:51.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.428+0000 7efd9159a640 1 -- 172.21.15.88:0/560740241 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd84032810 con 0x7efd94155800 2024-09-16T01:35:51.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.429+0000 7efd98eed640 1 -- 172.21.15.88:0/560740241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd94155800 msgr2=0x7efd940b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:51.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.429+0000 7efd98eed640 1 --2- 172.21.15.88:0/560740241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd94155800 0x7efd940b6140 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7efd9406e210 tx=0x7efd8402a1a0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.429+0000 7efd98eed640 1 -- 172.21.15.88:0/560740241 shutdown_connections 2024-09-16T01:35:51.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.429+0000 7efd98eed640 1 --2- 172.21.15.88:0/560740241 >> v2:172.21.15.154:3300/0 conn(0x7efd9415c330 0x7efd940b4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.429+0000 7efd98eed640 1 --2- 172.21.15.88:0/560740241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd94155800 0x7efd940b6140 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.429+0000 7efd98eed640 1 -- 172.21.15.88:0/560740241 >> 172.21.15.88:0/560740241 conn(0x7efd9406bc10 msgr2=0x7efd9406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:51.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.429+0000 7efd98eed640 1 -- 172.21.15.88:0/560740241 shutdown_connections 2024-09-16T01:35:51.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.429+0000 7efd98eed640 1 -- 172.21.15.88:0/560740241 wait complete. 2024-09-16T01:35:51.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.430+0000 7efd98eed640 1 Processor -- start 2024-09-16T01:35:51.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.430+0000 7efd98eed640 1 -- start start 2024-09-16T01:35:51.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.431+0000 7efd98eed640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd940cf570 0x7efd94144e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.431+0000 7efd98eed640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd94155800 0x7efd941533f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.431+0000 7efd98eed640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd9415c330 0x7efd94151970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.431+0000 7efd98eed640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7efd940b5c20 con 0x7efd9415c330 2024-09-16T01:35:51.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.431+0000 7efd98eed640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efd940b5940 con 0x7efd940cf570 2024-09-16T01:35:51.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.431+0000 7efd98eed640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efd94155be0 con 0x7efd94155800 2024-09-16T01:35:51.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.431+0000 7efd92d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd9415c330 0x7efd94151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:51.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.431+0000 7efd92d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd9415c330 0x7efd94151970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40422/0 (socket says 172.21.15.88:40422) 2024-09-16T01:35:51.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.431+0000 7efd92d9d640 1 -- 172.21.15.88:0/3630318904 learned_addr learned my addr 172.21.15.88:0/3630318904 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:51.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.431+0000 7efd9259c640 1 --2- 172.21.15.88:0/3630318904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd940cf570 0x7efd94144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:51.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.431+0000 7efd91d9b640 1 --2- 172.21.15.88:0/3630318904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd94155800 0x7efd941533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:51.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.431+0000 7efd92d9d640 1 -- 172.21.15.88:0/3630318904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd94155800 msgr2=0x7efd941533f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:51.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.431+0000 7efd92d9d640 1 --2- 172.21.15.88:0/3630318904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd94155800 0x7efd941533f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.431+0000 7efd92d9d640 1 -- 172.21.15.88:0/3630318904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd940cf570 msgr2=0x7efd94144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:51.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.431+0000 7efd92d9d640 1 --2- 172.21.15.88:0/3630318904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd940cf570 0x7efd94144e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.431+0000 7efd92d9d640 1 -- 172.21.15.88:0/3630318904 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd94152350 con 0x7efd9415c330 2024-09-16T01:35:51.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.431+0000 7efd92d9d640 1 --2- 172.21.15.88:0/3630318904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd9415c330 0x7efd94151970 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7efd8c00c800 tx=0x7efd8c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:51.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.432+0000 7efd82ffd640 1 -- 172.21.15.88:0/3630318904 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd8c018070 con 0x7efd9415c330 2024-09-16T01:35:51.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.432+0000 7efd82ffd640 1 -- 172.21.15.88:0/3630318904 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd8c004d80 con 0x7efd9415c330 2024-09-16T01:35:51.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.432+0000 7efd98eed640 1 -- 172.21.15.88:0/3630318904 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd94152630 con 0x7efd9415c330 2024-09-16T01:35:51.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.432+0000 7efd82ffd640 1 -- 172.21.15.88:0/3630318904 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd8c00a820 con 0x7efd9415c330 2024-09-16T01:35:51.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.432+0000 7efd98eed640 1 -- 172.21.15.88:0/3630318904 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd94152ed0 con 0x7efd9415c330 2024-09-16T01:35:51.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.433+0000 7efd98eed640 1 -- 172.21.15.88:0/3630318904 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd940ceee0 con 0x7efd9415c330 2024-09-16T01:35:51.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.433+0000 7efd82ffd640 1 -- 172.21.15.88:0/3630318904 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7efd8c01b070 con 0x7efd9415c330 2024-09-16T01:35:51.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.434+0000 7efd82ffd640 1 --2- 172.21.15.88:0/3630318904 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efd7406c2e0 0x7efd7406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:51.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.434+0000 7efd9259c640 1 --2- 172.21.15.88:0/3630318904 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efd7406c2e0 0x7efd7406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:51.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.434+0000 7efd82ffd640 1 -- 172.21.15.88:0/3630318904 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7efd8c08a990 con 0x7efd9415c330 2024-09-16T01:35:51.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.434+0000 7efd9259c640 1 --2- 172.21.15.88:0/3630318904 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efd7406c2e0 0x7efd7406e7a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7efd84034000 tx=0x7efd8402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:51.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.438+0000 7efd82ffd640 1 -- 172.21.15.88:0/3630318904 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7efd940ceee0 con 0x7efd9415c330 2024-09-16T01:35:51.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.640+0000 7efd98eed640 1 -- 172.21.15.88:0/3630318904 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7efd94152800 con 0x7efd9415c330 2024-09-16T01:35:51.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.641+0000 7efd82ffd640 1 -- 172.21.15.88:0/3630318904 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) ==== 53+0+73 (secure 0 0 0) 0x7efd8c0591f0 con 0x7efd9415c330 2024-09-16T01:35:51.642 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:35:51.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.643+0000 7efd98eed640 1 -- 172.21.15.88:0/3630318904 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efd7406c2e0 msgr2=0x7efd7406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:51.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.643+0000 7efd98eed640 1 --2- 172.21.15.88:0/3630318904 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efd7406c2e0 0x7efd7406e7a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7efd84034000 tx=0x7efd8402e000 comp rx=0 tx=0).stop 2024-09-16T01:35:51.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.643+0000 7efd98eed640 1 -- 172.21.15.88:0/3630318904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd9415c330 msgr2=0x7efd94151970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:51.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.643+0000 7efd98eed640 1 --2- 172.21.15.88:0/3630318904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd9415c330 0x7efd94151970 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7efd8c00c800 tx=0x7efd8c00ccc0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.643+0000 7efd98eed640 1 -- 172.21.15.88:0/3630318904 shutdown_connections 2024-09-16T01:35:51.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.643+0000 7efd98eed640 1 --2- 172.21.15.88:0/3630318904 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efd7406c2e0 0x7efd7406e7a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.643+0000 7efd98eed640 1 --2- 172.21.15.88:0/3630318904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd9415c330 0x7efd94151970 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.643+0000 7efd98eed640 1 --2- 172.21.15.88:0/3630318904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd94155800 0x7efd941533f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.643+0000 7efd98eed640 1 --2- 172.21.15.88:0/3630318904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd940cf570 0x7efd94144e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:51.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.643+0000 7efd98eed640 1 -- 172.21.15.88:0/3630318904 >> 172.21.15.88:0/3630318904 conn(0x7efd9406bc10 msgr2=0x7efd9406ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:51.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.643+0000 7efd98eed640 1 -- 172.21.15.88:0/3630318904 shutdown_connections 2024-09-16T01:35:51.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51.643+0000 7efd98eed640 1 -- 172.21.15.88:0/3630318904 wait complete. 2024-09-16T01:35:51.655 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-16T01:35:51.655 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:35:51.655 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2024-09-16T01:35:51.655 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T01:35:51.655 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T01:35:51.655 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-16T01:35:51.655 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-16T01:35:51.655 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:35:51.655 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:35:51.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:51.656 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2024-09-16T01:35:51.671 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:35:51.671 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:35:51.671 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:35:51.671 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:35:51.671 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-09-16T01:35:51.672 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:35:51.672 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:35:51.672 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:35:51.672 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2024-09-16T01:35:51.672 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:35:51.672 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:35:51.672 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2024-09-16T01:35:51.672 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:35:51.672 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2024-09-16T01:35:51.672 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:35:51.672 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2024-09-16T01:35:51.672 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:35:51.672 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42672sec preferred_lft 42672sec 2024-09-16T01:35:51.673 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link noprefixroute 2024-09-16T01:35:51.673 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:35:51.673 INFO:teuthology.orchestra.run.smithi088.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:35:51.673 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff 2024-09-16T01:35:51.673 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T01:35:51.673 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:35:51.673 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::e486:6eff:fe81:a1c/64 scope link 2024-09-16T01:35:51.673 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:35:51.673 INFO:teuthology.orchestra.run.smithi088.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:35:51.673 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 12:6f:29:be:95:48 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:35:51.673 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::106f:29ff:febe:9548/64 scope link 2024-09-16T01:35:51.673 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:35:51.673 INFO:teuthology.orchestra.run.smithi088.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:35:51.674 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T01:35:51.674 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::c63:b4ff:feb3:ad9e/64 scope link 2024-09-16T01:35:51.674 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:35:51.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:51.674 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:35:51.734 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-09-16T01:35:51.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:51.736 DEBUG:teuthology.orchestra.run.smithi088:> route 2024-09-16T01:35:51.792 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2024-09-16T01:35:51.792 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:35:51.792 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:35:51.792 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:35:51.792 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:35:51.793 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:35:51.793 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:35:51.793 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:35:51.793 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:35:51.793 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:35:51.793 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:35:51.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:35:51.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:35:51.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:51.918 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:35:51.933 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T01:35:51.933 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T01:35:51.933 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:35:51.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:51.933 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2024-09-16T01:35:51.998 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:35:51.998 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T01:35:52.054 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:35:52.054 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:35:52.055 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:35:52.110 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-16T01:35:52.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:52.111 DEBUG:teuthology.orchestra.run.smithi088:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:35:52.194 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:35:52.194 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:35:52.194 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:35:52.202 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:35:52.203 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:35:52.203 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:35:52.203 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:35:52.203 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:35:52.208 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:35:52.209 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:35:52.209 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:35:52.209 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:35:52.209 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:35:52.209 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790" 2024-09-16T01:35:52.209 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2024-09-16T01:35:52.209 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5ace2c91-46f9-482b-8a89-731fbdc171bf" will pass to kernel 2024-09-16T01:35:52.209 INFO:teuthology.orchestra.run.smithi088.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-16T01:35:52.210 INFO:teuthology.orchestra.run.smithi088.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:35:52.210 INFO:teuthology.orchestra.run.smithi088.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:35:52.210 INFO:teuthology.orchestra.run.smithi088.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:35:52.210 INFO:teuthology.orchestra.run.smithi088.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:35:52.211 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:35:52.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:52.211 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:35:52.276 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:35:52.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:35:52.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:35:52.372 INFO:teuthology.orchestra.run.smithi088.stdout:client4784 2024-09-16T01:35:52.372 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:35:52.372 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:35:52.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:52.388 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.files ./subdir) 2024-09-16T01:35:52.444 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-09-16T01:35:52.444 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 10000000000.00000000 file_to_be_damaged_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-09-16T01:35:52.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.527+0000 7f218fb1cf40 1 Processor -- start 2024-09-16T01:35:52.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.528+0000 7f218fb1cf40 1 -- start start 2024-09-16T01:35:52.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.528+0000 7f218fb1cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5609dd807b40 0x5609dd846f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.528+0000 7f218fb1cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5609dd8dc080 0x5609dd84fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.528+0000 7f218fb1cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5609dd8dc9a0 0x5609dd8d3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.528+0000 7f218fb1cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5609dd8518e0 con 0x5609dd807b40 2024-09-16T01:35:52.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.528+0000 7f218fb1cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5609dd759510 con 0x5609dd8dc9a0 2024-09-16T01:35:52.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.528+0000 7f218fb1cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5609dd851b80 con 0x5609dd8dc080 2024-09-16T01:35:52.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.528+0000 7f218cbad640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5609dd807b40 0x5609dd846f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.528+0000 7f218cbad640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5609dd807b40 0x5609dd846f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40448/0 (socket says 172.21.15.88:40448) 2024-09-16T01:35:52.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.528+0000 7f218cbad640 1 -- 172.21.15.88:0/4076819640 learned_addr learned my addr 172.21.15.88:0/4076819640 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:52.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.528+0000 7f2187fff640 1 --2- 172.21.15.88:0/4076819640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5609dd8dc080 0x5609dd84fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.528+0000 7f218d3ae640 1 --2- 172.21.15.88:0/4076819640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5609dd8dc9a0 0x5609dd8d3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.529+0000 7f218cbad640 1 -- 172.21.15.88:0/4076819640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5609dd8dc080 msgr2=0x5609dd84fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:52.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.529+0000 7f218cbad640 1 --2- 172.21.15.88:0/4076819640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5609dd8dc080 0x5609dd84fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.529+0000 7f218cbad640 1 -- 172.21.15.88:0/4076819640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5609dd8dc9a0 msgr2=0x5609dd8d3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:52.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.529+0000 7f218cbad640 1 --2- 172.21.15.88:0/4076819640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5609dd8dc9a0 0x5609dd8d3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.529+0000 7f218cbad640 1 -- 172.21.15.88:0/4076819640 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609dd806b00 con 0x5609dd807b40 2024-09-16T01:35:52.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.529+0000 7f218cbad640 1 --2- 172.21.15.88:0/4076819640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5609dd807b40 0x5609dd846f10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f218000a7a0 tx=0x7f218000a890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0b6a388ca7dc72e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:52.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.529+0000 7f2186ffd640 1 -- 172.21.15.88:0/4076819640 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2180019070 con 0x5609dd807b40 2024-09-16T01:35:52.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.529+0000 7f2186ffd640 1 -- 172.21.15.88:0/4076819640 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2180007600 con 0x5609dd807b40 2024-09-16T01:35:52.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.529+0000 7f2186ffd640 1 -- 172.21.15.88:0/4076819640 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21800079b0 con 0x5609dd807b40 2024-09-16T01:35:52.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.529+0000 7f218fb1cf40 1 -- 172.21.15.88:0/4076819640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5609dd807b40 msgr2=0x5609dd846f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:52.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.529+0000 7f218fb1cf40 1 --2- 172.21.15.88:0/4076819640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5609dd807b40 0x5609dd846f10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f218000a7a0 tx=0x7f218000a890 comp rx=0 tx=0).stop 2024-09-16T01:35:52.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.529+0000 7f218fb1cf40 1 -- 172.21.15.88:0/4076819640 shutdown_connections 2024-09-16T01:35:52.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.529+0000 7f218fb1cf40 1 --2- 172.21.15.88:0/4076819640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5609dd8dc9a0 0x5609dd8d3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.529+0000 7f218fb1cf40 1 --2- 172.21.15.88:0/4076819640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5609dd8dc080 0x5609dd84fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.529+0000 7f218fb1cf40 1 --2- 172.21.15.88:0/4076819640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5609dd807b40 0x5609dd846f10 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.529+0000 7f218fb1cf40 1 -- 172.21.15.88:0/4076819640 >> 172.21.15.88:0/4076819640 conn(0x5609dd832960 msgr2=0x5609dd8de3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:52.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.529+0000 7f218fb1cf40 1 -- 172.21.15.88:0/4076819640 shutdown_connections 2024-09-16T01:35:52.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.529+0000 7f218fb1cf40 1 -- 172.21.15.88:0/4076819640 wait complete. 2024-09-16T01:35:52.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.530+0000 7f218fb1cf40 1 Processor -- start 2024-09-16T01:35:52.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.530+0000 7f218fb1cf40 1 -- start start 2024-09-16T01:35:52.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.531+0000 7f218fb1cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5609dd807b40 0x5609dd8deb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.531+0000 7f218fb1cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5609dd8dc080 0x5609dd8ca520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.531+0000 7f218fb1cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5609dd8dc9a0 0x5609dd8d2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.531+0000 7f218fb1cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5609dd809630 con 0x5609dd8dc080 2024-09-16T01:35:52.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.531+0000 7f218fb1cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5609dd84e340 con 0x5609dd807b40 2024-09-16T01:35:52.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.531+0000 7f218fb1cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5609dd84e060 con 0x5609dd8dc9a0 2024-09-16T01:35:52.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.531+0000 7f2187fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5609dd8dc080 0x5609dd8ca520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.531+0000 7f2187fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5609dd8dc080 0x5609dd8ca520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40456/0 (socket says 172.21.15.88:40456) 2024-09-16T01:35:52.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.531+0000 7f2187fff640 1 -- 172.21.15.88:0/2746974200 learned_addr learned my addr 172.21.15.88:0/2746974200 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:52.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.531+0000 7f218cbad640 1 --2- 172.21.15.88:0/2746974200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5609dd807b40 0x5609dd8deb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.531+0000 7f218d3ae640 1 --2- 172.21.15.88:0/2746974200 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5609dd8dc9a0 0x5609dd8d2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.531+0000 7f2187fff640 1 -- 172.21.15.88:0/2746974200 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5609dd8dc9a0 msgr2=0x5609dd8d2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:52.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.531+0000 7f2187fff640 1 --2- 172.21.15.88:0/2746974200 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5609dd8dc9a0 0x5609dd8d2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.531+0000 7f2187fff640 1 -- 172.21.15.88:0/2746974200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5609dd807b40 msgr2=0x5609dd8deb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:52.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.531+0000 7f2187fff640 1 --2- 172.21.15.88:0/2746974200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5609dd807b40 0x5609dd8deb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.531+0000 7f2187fff640 1 -- 172.21.15.88:0/2746974200 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609dd8d6820 con 0x5609dd8dc080 2024-09-16T01:35:52.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.531+0000 7f2187fff640 1 --2- 172.21.15.88:0/2746974200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5609dd8dc080 0x5609dd8ca520 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2178055040 tx=0x7f2178002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:52.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.531+0000 7f2185ffb640 1 -- 172.21.15.88:0/2746974200 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2178056040 con 0x5609dd8dc080 2024-09-16T01:35:52.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.531+0000 7f2185ffb640 1 -- 172.21.15.88:0/2746974200 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2178052bb0 con 0x5609dd8dc080 2024-09-16T01:35:52.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.531+0000 7f218fb1cf40 1 -- 172.21.15.88:0/2746974200 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5609dd807790 con 0x5609dd8dc080 2024-09-16T01:35:52.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.531+0000 7f2185ffb640 1 -- 172.21.15.88:0/2746974200 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2178066440 con 0x5609dd8dc080 2024-09-16T01:35:52.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.532+0000 7f218fb1cf40 1 -- 172.21.15.88:0/2746974200 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609dd8d6d70 con 0x5609dd8dc080 2024-09-16T01:35:52.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.533+0000 7f2185ffb640 1 -- 172.21.15.88:0/2746974200 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2178059020 con 0x5609dd8dc080 2024-09-16T01:35:52.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.533+0000 7f2185ffb640 1 --2- 172.21.15.88:0/2746974200 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f217006ac90 0x7f217006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.533+0000 7f2185ffb640 1 -- 172.21.15.88:0/2746974200 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f21780ce400 con 0x5609dd8dc080 2024-09-16T01:35:52.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.533+0000 7f218cbad640 1 --2- 172.21.15.88:0/2746974200 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f217006ac90 0x7f217006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.533+0000 7f218fb1cf40 1 --2- 172.21.15.88:0/2746974200 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5609dd8d7fc0 0x5609dd8d8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.533+0000 7f218fb1cf40 1 -- 172.21.15.88:0/2746974200 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33) -- 0x5609dd8d8980 con 0x5609dd8d7fc0 2024-09-16T01:35:52.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.534+0000 7f218d3ae640 1 --2- 172.21.15.88:0/2746974200 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5609dd8d7fc0 0x5609dd8d8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.534+0000 7f218cbad640 1 --2- 172.21.15.88:0/2746974200 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f217006ac90 0x7f217006d150 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f218000a740 tx=0x7f218000c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:52.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.534+0000 7f218d3ae640 1 --2- 172.21.15.88:0/2746974200 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5609dd8d7fc0 0x5609dd8d8440 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:52.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.535+0000 7f218d3ae640 1 -- 172.21.15.88:0/2746974200 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-vals] v33'12 uv12 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f2188010660 con 0x5609dd8d7fc0 2024-09-16T01:35:52.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.535+0000 7f218fb1cf40 1 -- 172.21.15.88:0/2746974200 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5609dd8d7fc0 msgr2=0x5609dd8d8440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:52.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.535+0000 7f218fb1cf40 1 --2- 172.21.15.88:0/2746974200 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5609dd8d7fc0 0x5609dd8d8440 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.535+0000 7f218fb1cf40 1 -- 172.21.15.88:0/2746974200 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f217006ac90 msgr2=0x7f217006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:52.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.535+0000 7f218fb1cf40 1 --2- 172.21.15.88:0/2746974200 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f217006ac90 0x7f217006d150 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f218000a740 tx=0x7f218000c000 comp rx=0 tx=0).stop 2024-09-16T01:35:52.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.535+0000 7f218fb1cf40 1 -- 172.21.15.88:0/2746974200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5609dd8dc080 msgr2=0x5609dd8ca520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:52.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.535+0000 7f218fb1cf40 1 --2- 172.21.15.88:0/2746974200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5609dd8dc080 0x5609dd8ca520 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2178055040 tx=0x7f2178002fb0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.535+0000 7f218d3ae640 1 -- 172.21.15.88:0/2746974200 reap_dead start 2024-09-16T01:35:52.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.535+0000 7f218fb1cf40 1 -- 172.21.15.88:0/2746974200 shutdown_connections 2024-09-16T01:35:52.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.535+0000 7f218fb1cf40 1 -- 172.21.15.88:0/2746974200 >> 172.21.15.88:0/2746974200 conn(0x5609dd832960 msgr2=0x5609dd806120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:52.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.535+0000 7f218fb1cf40 1 -- 172.21.15.88:0/2746974200 shutdown_connections 2024-09-16T01:35:52.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.535+0000 7f218fb1cf40 1 -- 172.21.15.88:0/2746974200 wait complete. 2024-09-16T01:35:52.542 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:35:52.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.638+0000 7f8dd3318640 1 Processor -- start 2024-09-16T01:35:52.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.639+0000 7f8dd3318640 1 -- start start 2024-09-16T01:35:52.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.639+0000 7f8dd3318640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dcc15e560 0x7f8dcc06e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.639+0000 7f8dd3318640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8dcc073130 0x7f8dcc0735c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.639+0000 7f8dd3318640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dcc0b5220 con 0x7f8dcc072220 2024-09-16T01:35:52.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.639+0000 7f8dd3318640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f8dcc0aefa0 con 0x7f8dcc073130 2024-09-16T01:35:52.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.639+0000 7f8dd3318640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8dcc0b54c0 con 0x7f8dcc15e560 2024-09-16T01:35:52.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.640+0000 7f8dd08b3640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8dcc072220 0x7f8dcc0ae890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:48790/0 (socket says 172.21.15.88:48790) 2024-09-16T01:35:52.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.640+0000 7f8dd08b3640 1 -- 172.21.15.88:0/2189204295 learned_addr learned my addr 172.21.15.88:0/2189204295 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:52.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.640+0000 7f8dd10b4640 1 --2- 172.21.15.88:0/2189204295 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dcc15e560 0x7f8dcc06e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.640+0000 7f8dd18b5640 1 --2- 172.21.15.88:0/2189204295 >> v2:172.21.15.154:3300/0 conn(0x7f8dcc073130 0x7f8dcc0735c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.640+0000 7f8dcbfff640 1 -- 172.21.15.88:0/2189204295 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2038883892 0 0) 0x7f8dcc0b5220 con 0x7f8dcc072220 2024-09-16T01:35:52.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.640+0000 7f8dcbfff640 1 -- 172.21.15.88:0/2189204295 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8db4003520 con 0x7f8dcc072220 2024-09-16T01:35:52.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.640+0000 7f8dcbfff640 1 -- 172.21.15.88:0/2189204295 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2823045241 0 0) 0x7f8db4003520 con 0x7f8dcc072220 2024-09-16T01:35:52.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.640+0000 7f8dcbfff640 1 -- 172.21.15.88:0/2189204295 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dcc15e560 msgr2=0x7f8dcc06e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:52.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.640+0000 7f8dcbfff640 1 --2- 172.21.15.88:0/2189204295 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dcc15e560 0x7f8dcc06e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.640+0000 7f8dcbfff640 1 -- 172.21.15.88:0/2189204295 >> v2:172.21.15.154:3300/0 conn(0x7f8dcc073130 msgr2=0x7f8dcc0735c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:52.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.640+0000 7f8dcbfff640 1 --2- 172.21.15.88:0/2189204295 >> v2:172.21.15.154:3300/0 conn(0x7f8dcc073130 0x7f8dcc0735c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.640+0000 7f8dcbfff640 1 -- 172.21.15.88:0/2189204295 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dcc0b57a0 con 0x7f8dcc072220 2024-09-16T01:35:52.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.640+0000 7f8dcbfff640 1 -- 172.21.15.88:0/2189204295 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f8dc0001690 con 0x7f8dcc072220 2024-09-16T01:35:52.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.640+0000 7f8dcbfff640 1 -- 172.21.15.88:0/2189204295 >> v1:172.21.15.88:6789/0 conn(0x7f8dcc072220 legacy=0x7f8dcc0ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:52.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.640+0000 7f8dcbfff640 1 --2- 172.21.15.88:0/2189204295 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dcc15e560 0x7f8db40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.640+0000 7f8dcbfff640 1 --2- 172.21.15.88:0/2189204295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8db4003f30 0x7f8db4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.640+0000 7f8dcbfff640 1 --2- 172.21.15.88:0/2189204295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8db4004d40 0x7f8db40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.640+0000 7f8dcbfff640 1 -- 172.21.15.88:0/2189204295 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8dcc0aefa0 con 0x7f8db4003f30 2024-09-16T01:35:52.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.640+0000 7f8dcbfff640 1 -- 172.21.15.88:0/2189204295 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8dcc0b54c0 con 0x7f8db4004d40 2024-09-16T01:35:52.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.640+0000 7f8dcbfff640 1 -- 172.21.15.88:0/2189204295 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8db400b9c0 con 0x7f8dcc15e560 2024-09-16T01:35:52.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.640+0000 7f8dd08b3640 1 --2- 172.21.15.88:0/2189204295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8db4003f30 0x7f8db4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.640+0000 7f8dd10b4640 1 --2- 172.21.15.88:0/2189204295 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dcc15e560 0x7f8db40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.640+0000 7f8dd18b5640 1 --2- 172.21.15.88:0/2189204295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8db4004d40 0x7f8db40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.641+0000 7f8dd08b3640 1 -- 172.21.15.88:0/2189204295 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dcc15e560 msgr2=0x7f8db40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:52.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.641+0000 7f8dd08b3640 1 --2- 172.21.15.88:0/2189204295 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dcc15e560 0x7f8db40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.641+0000 7f8dd08b3640 1 -- 172.21.15.88:0/2189204295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8db4004d40 msgr2=0x7f8db40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:52.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.641+0000 7f8dd08b3640 1 --2- 172.21.15.88:0/2189204295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8db4004d40 0x7f8db40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.641+0000 7f8dd08b3640 1 -- 172.21.15.88:0/2189204295 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db4003520 con 0x7f8db4003f30 2024-09-16T01:35:52.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.641+0000 7f8dd18b5640 1 --2- 172.21.15.88:0/2189204295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8db4004d40 0x7f8db40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:52.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.641+0000 7f8dd08b3640 1 --2- 172.21.15.88:0/2189204295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8db4003f30 0x7f8db4004800 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc0b53f0 tx=0x7f8db4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709035edd54a1c81 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:52.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.641+0000 7f8dcbfff640 1 -- 172.21.15.88:0/2189204295 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc0007d70 con 0x7f8db4003f30 2024-09-16T01:35:52.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.641+0000 7f8dcbfff640 1 -- 172.21.15.88:0/2189204295 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dc00058d0 con 0x7f8db4003f30 2024-09-16T01:35:52.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.641+0000 7f8dcbfff640 1 -- 172.21.15.88:0/2189204295 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc0005ca0 con 0x7f8db4003f30 2024-09-16T01:35:52.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.642+0000 7f8dd3318640 1 -- 172.21.15.88:0/2189204295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8db4003f30 msgr2=0x7f8db4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:52.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.642+0000 7f8dd3318640 1 --2- 172.21.15.88:0/2189204295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8db4003f30 0x7f8db4004800 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc0b53f0 tx=0x7f8db4005830 comp rx=0 tx=0).stop 2024-09-16T01:35:52.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.642+0000 7f8dd18b5640 1 -- 172.21.15.88:0/2189204295 reap_dead start 2024-09-16T01:35:52.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.642+0000 7f8dd3318640 1 -- 172.21.15.88:0/2189204295 shutdown_connections 2024-09-16T01:35:52.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.642+0000 7f8dd3318640 1 -- 172.21.15.88:0/2189204295 >> 172.21.15.88:0/2189204295 conn(0x7f8dcc06bc10 msgr2=0x7f8dcc06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:52.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.642+0000 7f8dd3318640 1 -- 172.21.15.88:0/2189204295 shutdown_connections 2024-09-16T01:35:52.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.642+0000 7f8dd3318640 1 -- 172.21.15.88:0/2189204295 wait complete. 2024-09-16T01:35:52.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.643+0000 7f8dd3318640 1 Processor -- start 2024-09-16T01:35:52.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.643+0000 7f8dd3318640 1 -- start start 2024-09-16T01:35:52.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.644+0000 7f8dd3318640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8db4003f30 0x7f8dcc15c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.644+0000 7f8dd3318640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dcc15d4b0 0x7f8dcc1f4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.644+0000 7f8dd3318640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dcc15d930 0x7f8dcc1f4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.644+0000 7f8dd3318640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8dcc072660 con 0x7f8db4003f30 2024-09-16T01:35:52.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.644+0000 7f8dd3318640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8dcc0b5d60 con 0x7f8dcc15d4b0 2024-09-16T01:35:52.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.644+0000 7f8dd3318640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8dcc15e940 con 0x7f8dcc15d930 2024-09-16T01:35:52.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.644+0000 7f8dd10b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8db4003f30 0x7f8dcc15c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.644+0000 7f8dd10b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8db4003f30 0x7f8dcc15c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40476/0 (socket says 172.21.15.88:40476) 2024-09-16T01:35:52.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.644+0000 7f8dd10b4640 1 -- 172.21.15.88:0/2449406187 learned_addr learned my addr 172.21.15.88:0/2449406187 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:52.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.644+0000 7f8dd08b3640 1 --2- 172.21.15.88:0/2449406187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dcc15d4b0 0x7f8dcc1f4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.644+0000 7f8dd18b5640 1 --2- 172.21.15.88:0/2449406187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dcc15d930 0x7f8dcc1f4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.644+0000 7f8dd10b4640 1 -- 172.21.15.88:0/2449406187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dcc15d930 msgr2=0x7f8dcc1f4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:52.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.644+0000 7f8dd10b4640 1 --2- 172.21.15.88:0/2449406187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dcc15d930 0x7f8dcc1f4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.644+0000 7f8dd10b4640 1 -- 172.21.15.88:0/2449406187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dcc15d4b0 msgr2=0x7f8dcc1f4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:52.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.644+0000 7f8dd10b4640 1 --2- 172.21.15.88:0/2449406187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dcc15d4b0 0x7f8dcc1f4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.644+0000 7f8dd10b4640 1 -- 172.21.15.88:0/2449406187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dcc160800 con 0x7f8db4003f30 2024-09-16T01:35:52.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.644+0000 7f8dd10b4640 1 --2- 172.21.15.88:0/2449406187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8db4003f30 0x7f8dcc15c650 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc011e60 tx=0x7f8dbc00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:52.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.645+0000 7f8dc9ffb640 1 -- 172.21.15.88:0/2449406187 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc015630 con 0x7f8db4003f30 2024-09-16T01:35:52.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.645+0000 7f8dc9ffb640 1 -- 172.21.15.88:0/2449406187 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dbc0157d0 con 0x7f8db4003f30 2024-09-16T01:35:52.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.645+0000 7f8dd3318640 1 -- 172.21.15.88:0/2449406187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dcc160aa0 con 0x7f8db4003f30 2024-09-16T01:35:52.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.645+0000 7f8dc9ffb640 1 -- 172.21.15.88:0/2449406187 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc015ba0 con 0x7f8db4003f30 2024-09-16T01:35:52.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.645+0000 7f8dd3318640 1 -- 172.21.15.88:0/2449406187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dcc161060 con 0x7f8db4003f30 2024-09-16T01:35:52.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.646+0000 7f8dd3318640 1 -- 172.21.15.88:0/2449406187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dcc06e780 con 0x7f8db4003f30 2024-09-16T01:35:52.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.646+0000 7f8dc9ffb640 1 -- 172.21.15.88:0/2449406187 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8dbc01d400 con 0x7f8db4003f30 2024-09-16T01:35:52.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.647+0000 7f8dc9ffb640 1 --2- 172.21.15.88:0/2449406187 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8db806c2b0 0x7f8db806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.647+0000 7f8dc9ffb640 1 -- 172.21.15.88:0/2449406187 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f8dbc08c5b0 con 0x7f8db4003f30 2024-09-16T01:35:52.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.647+0000 7f8dd08b3640 1 --2- 172.21.15.88:0/2449406187 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8db806c2b0 0x7f8db806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.647+0000 7f8dd08b3640 1 --2- 172.21.15.88:0/2449406187 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8db806c2b0 0x7f8db806e770 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0033000 tx=0x7f8dc002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:52.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.651+0000 7f8dc9ffb640 1 -- 172.21.15.88:0/2449406187 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8dbc019030 con 0x7f8db4003f30 2024-09-16T01:35:52.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.847+0000 7f8dd3318640 1 -- 172.21.15.88:0/2449406187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8dcc15a0b0 con 0x7f8db4003f30 2024-09-16T01:35:52.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.848+0000 7f8dc9ffb640 1 -- 172.21.15.88:0/2449406187 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5230 (secure 0 0 0) 0x7f8dbc010040 con 0x7f8db4003f30 2024-09-16T01:35:52.849 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:35:52.849 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":19,"btime":"2024-09-16T01:35:16:950413+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1005331674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1005331674},{"type":"v1","addr":"172.21.15.88:6837","nonce":1005331674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/3427247662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3427247662},{"type":"v1","addr":"172.21.15.154:6834","nonce":3427247662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/4059835337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":4059835337},{"type":"v1","addr":"172.21.15.154:6835","nonce":4059835337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:35:00.674725+0000","modified":"2024-09-16T01:35:16.950408+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"c","rank":0,"incarnation":16,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6835/2870334334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870334334},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870334334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5067,"qdb_cluster":[5067]},"id":2}]} 2024-09-16T01:35:52.850 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 19 2024-09-16T01:35:52.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.850+0000 7f8dd3318640 1 -- 172.21.15.88:0/2449406187 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8db806c2b0 msgr2=0x7f8db806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:52.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.850+0000 7f8dd3318640 1 --2- 172.21.15.88:0/2449406187 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8db806c2b0 0x7f8db806e770 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0033000 tx=0x7f8dc002e000 comp rx=0 tx=0).stop 2024-09-16T01:35:52.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.850+0000 7f8dd3318640 1 -- 172.21.15.88:0/2449406187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8db4003f30 msgr2=0x7f8dcc15c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:52.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.850+0000 7f8dd3318640 1 --2- 172.21.15.88:0/2449406187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8db4003f30 0x7f8dcc15c650 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc011e60 tx=0x7f8dbc00f580 comp rx=0 tx=0).stop 2024-09-16T01:35:52.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.850+0000 7f8dd3318640 1 -- 172.21.15.88:0/2449406187 shutdown_connections 2024-09-16T01:35:52.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.850+0000 7f8dd3318640 1 --2- 172.21.15.88:0/2449406187 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8db806c2b0 0x7f8db806e770 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.850+0000 7f8dd3318640 1 --2- 172.21.15.88:0/2449406187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dcc15d930 0x7f8dcc1f4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.850+0000 7f8dd3318640 1 --2- 172.21.15.88:0/2449406187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dcc15d4b0 0x7f8dcc1f4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.850+0000 7f8dd3318640 1 --2- 172.21.15.88:0/2449406187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8db4003f30 0x7f8dcc15c650 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.850+0000 7f8dd3318640 1 -- 172.21.15.88:0/2449406187 >> 172.21.15.88:0/2449406187 conn(0x7f8dcc06bc10 msgr2=0x7f8dcc0735c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:52.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.850+0000 7f8dd3318640 1 -- 172.21.15.88:0/2449406187 shutdown_connections 2024-09-16T01:35:52.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.850+0000 7f8dd3318640 1 -- 172.21.15.88:0/2449406187 wait complete. 2024-09-16T01:35:52.862 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage rm 3136708976 2024-09-16T01:35:52.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.961+0000 7f4db01ed640 1 Processor -- start 2024-09-16T01:35:52.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.961+0000 7f4db01ed640 1 -- start start 2024-09-16T01:35:52.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.962+0000 7f4db01ed640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4da80b64a0 0x7f4da80b6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.962+0000 7f4db01ed640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4da80b7f50 0x7f4da80b83c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.962+0000 7f4db01ed640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4da80b7790 con 0x7f4da815e750 2024-09-16T01:35:52.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.962+0000 7f4db01ed640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f4da80b6e50 con 0x7f4da80b64a0 2024-09-16T01:35:52.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.962+0000 7f4db01ed640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4da80d43c0 con 0x7f4da80b7f50 2024-09-16T01:35:52.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.962+0000 7f4dad788640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4da80b64a0 0x7f4da80b6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.962+0000 7f4dadf89640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4da815e750 0x7f4da80cf630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:48798/0 (socket says 172.21.15.88:48798) 2024-09-16T01:35:52.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.962+0000 7f4dadf89640 1 -- 172.21.15.88:0/3069895807 learned_addr learned my addr 172.21.15.88:0/3069895807 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:52.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.962+0000 7f4dae78a640 1 --2- 172.21.15.88:0/3069895807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4da80b7f50 0x7f4da80b83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.962+0000 7f4dad788640 1 -- 172.21.15.88:0/3069895807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4da80b7f50 msgr2=0x7f4da80b83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:52.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.962+0000 7f4dad788640 1 --2- 172.21.15.88:0/3069895807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4da80b7f50 0x7f4da80b83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.962+0000 7f4dad788640 1 -- 172.21.15.88:0/3069895807 >> v1:172.21.15.88:6789/0 conn(0x7f4da815e750 legacy=0x7f4da80cf630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:52.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.962+0000 7f4dad788640 1 -- 172.21.15.88:0/3069895807 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4da80d3600 con 0x7f4da80b64a0 2024-09-16T01:35:52.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.962+0000 7f4dacf87640 1 -- 172.21.15.88:0/3069895807 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3648615922 0 0) 0x7f4da80b7790 con 0x7f4da815e750 2024-09-16T01:35:52.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.962+0000 7f4dae78a640 1 --2- 172.21.15.88:0/3069895807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4da80b7f50 0x7f4da80b83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:52.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.963+0000 7f4dad788640 1 --2- 172.21.15.88:0/3069895807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4da80b64a0 0x7f4da80b6910 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4da80d4540 tx=0x7f4d9802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=497acec1154dd8c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:52.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.963+0000 7f4dacf87640 1 -- 172.21.15.88:0/3069895807 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d98037070 con 0x7f4da80b64a0 2024-09-16T01:35:52.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.963+0000 7f4dacf87640 1 -- 172.21.15.88:0/3069895807 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d98032440 con 0x7f4da80b64a0 2024-09-16T01:35:52.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.963+0000 7f4dacf87640 1 -- 172.21.15.88:0/3069895807 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d98032810 con 0x7f4da80b64a0 2024-09-16T01:35:52.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.964+0000 7f4db01ed640 1 -- 172.21.15.88:0/3069895807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4da80b64a0 msgr2=0x7f4da80b6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:52.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.964+0000 7f4db01ed640 1 --2- 172.21.15.88:0/3069895807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4da80b64a0 0x7f4da80b6910 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4da80d4540 tx=0x7f4d9802a180 comp rx=0 tx=0).stop 2024-09-16T01:35:52.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.964+0000 7f4db01ed640 1 -- 172.21.15.88:0/3069895807 shutdown_connections 2024-09-16T01:35:52.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.964+0000 7f4db01ed640 1 --2- 172.21.15.88:0/3069895807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4da80b7f50 0x7f4da80b83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.964+0000 7f4db01ed640 1 --2- 172.21.15.88:0/3069895807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4da80b64a0 0x7f4da80b6910 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.964+0000 7f4db01ed640 1 -- 172.21.15.88:0/3069895807 >> 172.21.15.88:0/3069895807 conn(0x7f4da80d1500 msgr2=0x7f4da80d1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:52.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.964+0000 7f4db01ed640 1 -- 172.21.15.88:0/3069895807 shutdown_connections 2024-09-16T01:35:52.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.964+0000 7f4db01ed640 1 -- 172.21.15.88:0/3069895807 wait complete. 2024-09-16T01:35:52.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.965+0000 7f4db01ed640 1 Processor -- start 2024-09-16T01:35:52.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.965+0000 7f4db01ed640 1 -- start start 2024-09-16T01:35:52.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.966+0000 7f4db01ed640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4da80b64a0 0x7f4da8156410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.966+0000 7f4db01ed640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4da80b7f50 0x7f4da81f49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.966+0000 7f4db01ed640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4da815e750 0x7f4da8157690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.966+0000 7f4db01ed640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4da80b7ac0 con 0x7f4da815e750 2024-09-16T01:35:52.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.966+0000 7f4db01ed640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4da80d33b0 con 0x7f4da80b7f50 2024-09-16T01:35:52.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.966+0000 7f4db01ed640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4da80d3bc0 con 0x7f4da80b64a0 2024-09-16T01:35:52.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.966+0000 7f4dae78a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4da815e750 0x7f4da8157690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.966+0000 7f4dae78a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4da815e750 0x7f4da8157690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40490/0 (socket says 172.21.15.88:40490) 2024-09-16T01:35:52.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.966+0000 7f4dae78a640 1 -- 172.21.15.88:0/2237158275 learned_addr learned my addr 172.21.15.88:0/2237158275 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:52.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.966+0000 7f4dad788640 1 --2- 172.21.15.88:0/2237158275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4da80b7f50 0x7f4da81f49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.966+0000 7f4dae78a640 1 -- 172.21.15.88:0/2237158275 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4da80b64a0 msgr2=0x7f4da8156410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:52.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.966+0000 7f4dae78a640 1 --2- 172.21.15.88:0/2237158275 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4da80b64a0 0x7f4da8156410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.966+0000 7f4dae78a640 1 -- 172.21.15.88:0/2237158275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4da80b7f50 msgr2=0x7f4da81f49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:52.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.966+0000 7f4dae78a640 1 --2- 172.21.15.88:0/2237158275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4da80b7f50 0x7f4da81f49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.966+0000 7f4dae78a640 1 -- 172.21.15.88:0/2237158275 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4da81e3e60 con 0x7f4da815e750 2024-09-16T01:35:52.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.966+0000 7f4dad788640 1 --2- 172.21.15.88:0/2237158275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4da80b7f50 0x7f4da81f49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:52.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.966+0000 7f4dae78a640 1 --2- 172.21.15.88:0/2237158275 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4da815e750 0x7f4da8157690 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4da000cf20 tx=0x7f4da000a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:52.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.967+0000 7f4d9e7fc640 1 -- 172.21.15.88:0/2237158275 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da001a070 con 0x7f4da815e750 2024-09-16T01:35:52.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.967+0000 7f4d9e7fc640 1 -- 172.21.15.88:0/2237158275 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4da000d840 con 0x7f4da815e750 2024-09-16T01:35:52.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.967+0000 7f4db01ed640 1 -- 172.21.15.88:0/2237158275 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4da81e4140 con 0x7f4da815e750 2024-09-16T01:35:52.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.967+0000 7f4d9e7fc640 1 -- 172.21.15.88:0/2237158275 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da000dc30 con 0x7f4da815e750 2024-09-16T01:35:52.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.967+0000 7f4db01ed640 1 -- 172.21.15.88:0/2237158275 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4da81f4140 con 0x7f4da815e750 2024-09-16T01:35:52.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.968+0000 7f4d9e7fc640 1 -- 172.21.15.88:0/2237158275 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4da001d070 con 0x7f4da815e750 2024-09-16T01:35:52.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.969+0000 7f4d9e7fc640 1 --2- 172.21.15.88:0/2237158275 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d8c06c2e0 0x7f4d8c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.969+0000 7f4dadf89640 1 --2- 172.21.15.88:0/2237158275 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d8c06c2e0 0x7f4d8c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.969+0000 7f4d9e7fc640 1 -- 172.21.15.88:0/2237158275 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f4da008bcd0 con 0x7f4da815e750 2024-09-16T01:35:52.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.969+0000 7f4dadf89640 1 --2- 172.21.15.88:0/2237158275 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d8c06c2e0 0x7f4d8c06e7a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4da4005880 tx=0x7f4da4007a00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:52.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.970+0000 7f4db305d740 1 Processor -- start 2024-09-16T01:35:52.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.970+0000 7f4db305d740 1 -- start start 2024-09-16T01:35:52.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.970+0000 7f4db305d740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55eace0a9ea0 0x55eace0aa280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.970+0000 7f4db305d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eace0a7660 0x55eace0a91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.971+0000 7f4db305d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a7ac0 0x55eace089e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.971+0000 7f4db305d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55eacdfc68d0 con 0x55eace0a7ac0 2024-09-16T01:35:52.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.971+0000 7f4db305d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55eacdf6e510 con 0x55eace0a7660 2024-09-16T01:35:52.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.971+0000 7f4db305d740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55eacdfc2890 con 0x55eace0a9ea0 2024-09-16T01:35:52.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.971+0000 7f4dae78a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a7ac0 0x55eace089e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.971+0000 7f4dae78a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a7ac0 0x55eace089e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40492/0 (socket says 172.21.15.88:40492) 2024-09-16T01:35:52.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.971+0000 7f4dae78a640 1 -- 172.21.15.88:0/2136913949 learned_addr learned my addr 172.21.15.88:0/2136913949 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:52.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.971+0000 7f4dadf89640 1 --2- 172.21.15.88:0/2136913949 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55eace0a9ea0 0x55eace0aa280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.971+0000 7f4dad788640 1 --2- 172.21.15.88:0/2136913949 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eace0a7660 0x55eace0a91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.971+0000 7f4dae78a640 1 -- 172.21.15.88:0/2136913949 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55eace0a9ea0 msgr2=0x55eace0aa280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:52.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.971+0000 7f4dae78a640 1 --2- 172.21.15.88:0/2136913949 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55eace0a9ea0 0x55eace0aa280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.971+0000 7f4dae78a640 1 -- 172.21.15.88:0/2136913949 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eace0a7660 msgr2=0x55eace0a91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:52.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.971+0000 7f4dae78a640 1 --2- 172.21.15.88:0/2136913949 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eace0a7660 0x55eace0a91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.971+0000 7f4dae78a640 1 -- 172.21.15.88:0/2136913949 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eacdf831a0 con 0x55eace0a7ac0 2024-09-16T01:35:52.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.971+0000 7f4dad788640 1 --2- 172.21.15.88:0/2136913949 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eace0a7660 0x55eace0a91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:52.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.971+0000 7f4dae78a640 1 --2- 172.21.15.88:0/2136913949 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a7ac0 0x55eace089e20 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4da0062450 tx=0x7f4da0059da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2279f84fe1531c24 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:52.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.971+0000 7f4d8b7fe640 1 -- 172.21.15.88:0/2136913949 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da005d070 con 0x55eace0a7ac0 2024-09-16T01:35:52.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.971+0000 7f4d8b7fe640 1 -- 172.21.15.88:0/2136913949 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4da000ddd0 con 0x55eace0a7ac0 2024-09-16T01:35:52.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.971+0000 7f4d8b7fe640 1 -- 172.21.15.88:0/2136913949 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da00580a0 con 0x55eace0a7ac0 2024-09-16T01:35:52.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.972+0000 7f4db305d740 1 -- 172.21.15.88:0/2136913949 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a7ac0 msgr2=0x55eace089e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:52.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.972+0000 7f4db305d740 1 --2- 172.21.15.88:0/2136913949 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a7ac0 0x55eace089e20 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4da0062450 tx=0x7f4da0059da0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.972+0000 7f4db305d740 1 -- 172.21.15.88:0/2136913949 shutdown_connections 2024-09-16T01:35:52.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.972+0000 7f4db305d740 1 --2- 172.21.15.88:0/2136913949 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a7ac0 0x55eace089e20 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.972+0000 7f4db305d740 1 --2- 172.21.15.88:0/2136913949 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eace0a7660 0x55eace0a91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.972+0000 7f4db305d740 1 --2- 172.21.15.88:0/2136913949 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55eace0a9ea0 0x55eace0aa280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.972+0000 7f4db305d740 1 -- 172.21.15.88:0/2136913949 >> 172.21.15.88:0/2136913949 conn(0x55eace05fd90 msgr2=0x55eace0a7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:52.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.972+0000 7f4db305d740 1 -- 172.21.15.88:0/2136913949 shutdown_connections 2024-09-16T01:35:52.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.972+0000 7f4db305d740 1 -- 172.21.15.88:0/2136913949 wait complete. 2024-09-16T01:35:52.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.973+0000 7f4db305d740 1 -- start start 2024-09-16T01:35:52.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.973+0000 7f4db305d740 1 Processor -- start 2024-09-16T01:35:52.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.974+0000 7f4db305d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eace0a7ac0 0x55eace0a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.974+0000 7f4db305d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a9ea0 0x55eace0b8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.974+0000 7f4db305d740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55eace0a1660 0x55eace0a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.974+0000 7f4db305d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55eace0a65c0 con 0x55eace0a9ea0 2024-09-16T01:35:52.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.974+0000 7f4db305d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55eacdf4f640 con 0x55eace0a7ac0 2024-09-16T01:35:52.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.974+0000 7f4db305d740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55eace0a68a0 con 0x55eace0a1660 2024-09-16T01:35:52.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.974+0000 7f4db305d740 10 client.0 _ll_register_callbacks cb 0x55eacdf872a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:35:52.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.974+0000 7f4dad788640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a9ea0 0x55eace0b8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.974+0000 7f4dad788640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a9ea0 0x55eace0b8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40500/0 (socket says 172.21.15.88:40500) 2024-09-16T01:35:52.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.974+0000 7f4dad788640 1 -- 172.21.15.88:0/818518945 learned_addr learned my addr 172.21.15.88:0/818518945 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:52.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.974+0000 7f4dadf89640 1 --2- 172.21.15.88:0/818518945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eace0a7ac0 0x55eace0a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.974+0000 7f4dae78a640 1 --2- 172.21.15.88:0/818518945 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55eace0a1660 0x55eace0a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.974+0000 7f4dad788640 1 -- 172.21.15.88:0/818518945 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55eace0a1660 msgr2=0x55eace0a3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:52.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.974+0000 7f4dad788640 1 --2- 172.21.15.88:0/818518945 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55eace0a1660 0x55eace0a3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.974+0000 7f4dad788640 1 -- 172.21.15.88:0/818518945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eace0a7ac0 msgr2=0x55eace0a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:52.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.974+0000 7f4dad788640 1 --2- 172.21.15.88:0/818518945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eace0a7ac0 0x55eace0a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.974+0000 7f4dad788640 1 -- 172.21.15.88:0/818518945 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eacdda5710 con 0x55eace0a9ea0 2024-09-16T01:35:52.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.974+0000 7f4dad788640 1 --2- 172.21.15.88:0/818518945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a9ea0 0x55eace0b8cd0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4d98009890 tx=0x7f4d98004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8ac62e718739664 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:52.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.974+0000 7f4d88ff9640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:35:52.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.974+0000 7f4d88ff9640 1 -- 172.21.15.88:0/818518945 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d98037040 con 0x55eace0a9ea0 2024-09-16T01:35:52.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.974+0000 7f4d88ff9640 1 -- 172.21.15.88:0/818518945 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f4d9803c640 con 0x55eace0a9ea0 2024-09-16T01:35:52.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.974+0000 7f4db305d740 1 -- 172.21.15.88:0/818518945 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4da000ddd0 con 0x55eace0a9ea0 2024-09-16T01:35:52.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.974+0000 7f4d88ff9640 1 -- 172.21.15.88:0/818518945 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f4d9804c050 con 0x55eace0a9ea0 2024-09-16T01:35:52.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.975+0000 7f4db305d740 10 client.5253 fetch_fsmap learned FSMap version 19 2024-09-16T01:35:52.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.975+0000 7f4db305d740 1 -- 172.21.15.88:0/818518945 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x55eacde05ea0 con 0x55eace0a9ea0 2024-09-16T01:35:52.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.975+0000 7f4d88ff9640 1 -- 172.21.15.88:0/818518945 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4da000ddd0 con 0x55eace0a9ea0 2024-09-16T01:35:52.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.975+0000 7f4d88ff9640 1 -- 172.21.15.88:0/818518945 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 19) ==== 3502+0+0 (secure 0 0 0) 0x7f4d9804f020 con 0x55eace0a9ea0 2024-09-16T01:35:52.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.975+0000 7f4db305d740 10 client.5253 fetch_fsmap finished waiting for FSMap version 19 2024-09-16T01:35:52.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.975+0000 7f4db305d740 10 client.5253 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T01:35:52.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.975+0000 7f4db305d740 20 client.5253 populate_metadata read hostname 'smithi088' 2024-09-16T01:35:52.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.975+0000 7f4db305d740 1 --2- 172.21.15.88:0/818518945 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x55eace0a4db0 0x55eace0a5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.975+0000 7f4db305d740 4 client.5253 mds_command: new command op to 5067 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:35:52.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.975+0000 7f4db305d740 1 -- 172.21.15.88:0/818518945 --> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eacdfd4000 con 0x55eace0a4db0 2024-09-16T01:35:52.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.975+0000 7f4dadf89640 1 --2- 172.21.15.88:0/818518945 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x55eace0a4db0 0x55eace0a5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.975+0000 7f4d88ff9640 10 client.5253 ms_handle_connect on v2:172.21.15.88:6834/2870334334 2024-09-16T01:35:52.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.975+0000 7f4dadf89640 1 --2- 172.21.15.88:0/818518945 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x55eace0a4db0 0x55eace0a5190 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bde71216233439c server_cookie=dbdeaf7c822ee182 in_seq=0 out_seq=0 2024-09-16T01:35:52.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.976+0000 7f4d88ff9640 1 -- 172.21.15.88:0/818518945 <== mds.0 v2:172.21.15.88:6834/2870334334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55eacdfd4000 con 0x55eace0a4db0 2024-09-16T01:35:52.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.976+0000 7f4d88ff9640 10 client.5253 handle_command_reply: tid=0 2024-09-16T01:35:52.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.976+0000 7f4db305d740 1 client.5253 shutdown 2024-09-16T01:35:52.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.976+0000 7f4db305d740 1 -- 172.21.15.88:0/818518945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a9ea0 msgr2=0x55eace0b8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:52.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.976+0000 7f4db305d740 1 --2- 172.21.15.88:0/818518945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a9ea0 0x55eace0b8cd0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4d98009890 tx=0x7f4d98004730 comp rx=0 tx=0).stop 2024-09-16T01:35:52.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.976+0000 7f4db305d740 1 -- 172.21.15.88:0/818518945 shutdown_connections 2024-09-16T01:35:52.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.976+0000 7f4db305d740 1 --2- 172.21.15.88:0/818518945 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x55eace0a4db0 0x55eace0a5190 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.976+0000 7f4db305d740 1 --2- 172.21.15.88:0/818518945 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55eace0a1660 0x55eace0a3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.976+0000 7f4d88ff9640 0 client.5253 ms_handle_reset on v2:172.21.15.88:6834/2870334334 2024-09-16T01:35:52.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.976+0000 7f4db305d740 1 --2- 172.21.15.88:0/818518945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a9ea0 0x55eace0b8cd0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.976+0000 7f4db305d740 1 --2- 172.21.15.88:0/818518945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eace0a7ac0 0x55eace0a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.976+0000 7f4db305d740 1 -- 172.21.15.88:0/818518945 >> 172.21.15.88:0/818518945 conn(0x55eace0a7660 msgr2=0x55eace0ab470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:52.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.976+0000 7f4db305d740 1 -- 172.21.15.88:0/818518945 shutdown_connections 2024-09-16T01:35:52.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.976+0000 7f4db305d740 1 -- 172.21.15.88:0/818518945 wait complete. 2024-09-16T01:35:52.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.976+0000 7f4db305d740 20 client.5253 trim_cache size 0 max 16384 2024-09-16T01:35:52.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.989+0000 7f4db305d740 1 Processor -- start 2024-09-16T01:35:52.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.989+0000 7f4db305d740 1 -- start start 2024-09-16T01:35:52.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.990+0000 7f4db305d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a1660 0x55eace08cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.990+0000 7f4db305d740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55eace0a7ac0 0x55eace0895f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.990+0000 7f4db305d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eace0a9ea0 0x55eace0a5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:52.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.990+0000 7f4db305d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4da81ffc30 con 0x55eace0a1660 2024-09-16T01:35:52.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.990+0000 7f4db305d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4da81f84f0 con 0x55eace0a9ea0 2024-09-16T01:35:52.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.990+0000 7f4db305d740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4da81f59b0 con 0x55eace0a7ac0 2024-09-16T01:35:52.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.990+0000 7f4dadf89640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a1660 0x55eace08cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:52.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.990+0000 7f4dadf89640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a1660 0x55eace08cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40514/0 (socket says 172.21.15.88:40514) 2024-09-16T01:35:52.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.990+0000 7f4dadf89640 1 -- 172.21.15.88:0/722831178 learned_addr learned my addr 172.21.15.88:0/722831178 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:52.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.991+0000 7f4dadf89640 1 -- 172.21.15.88:0/722831178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55eace0a7ac0 msgr2=0x55eace0895f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:52.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.991+0000 7f4dadf89640 1 --2- 172.21.15.88:0/722831178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55eace0a7ac0 0x55eace0895f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.991+0000 7f4dadf89640 1 -- 172.21.15.88:0/722831178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eace0a9ea0 msgr2=0x55eace0a5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:52.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.991+0000 7f4dadf89640 1 --2- 172.21.15.88:0/722831178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eace0a9ea0 0x55eace0a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.991+0000 7f4dadf89640 1 -- 172.21.15.88:0/722831178 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eacdc99000 con 0x55eace0a1660 2024-09-16T01:35:52.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.991+0000 7f4dadf89640 1 --2- 172.21.15.88:0/722831178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a1660 0x55eace08cda0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x55eace0a22f0 tx=0x7f4da4010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=697ebdca59c4ba7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:52.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.991+0000 7f4d8affd640 1 -- 172.21.15.88:0/722831178 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da4014630 con 0x55eace0a1660 2024-09-16T01:35:52.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.991+0000 7f4d8affd640 1 -- 172.21.15.88:0/722831178 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4da40147d0 con 0x55eace0a1660 2024-09-16T01:35:52.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.991+0000 7f4d8affd640 1 -- 172.21.15.88:0/722831178 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da4014b80 con 0x55eace0a1660 2024-09-16T01:35:52.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.991+0000 7f4db305d740 1 -- 172.21.15.88:0/722831178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a1660 msgr2=0x55eace08cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:52.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.991+0000 7f4db305d740 1 --2- 172.21.15.88:0/722831178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a1660 0x55eace08cda0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x55eace0a22f0 tx=0x7f4da4010000 comp rx=0 tx=0).stop 2024-09-16T01:35:52.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.991+0000 7f4db305d740 1 -- 172.21.15.88:0/722831178 shutdown_connections 2024-09-16T01:35:52.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.991+0000 7f4db305d740 1 --2- 172.21.15.88:0/722831178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eace0a9ea0 0x55eace0a5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.991+0000 7f4db305d740 1 --2- 172.21.15.88:0/722831178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55eace0a7ac0 0x55eace0895f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.991+0000 7f4db305d740 1 --2- 172.21.15.88:0/722831178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a1660 0x55eace08cda0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:52.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.991+0000 7f4db305d740 1 -- 172.21.15.88:0/722831178 >> 172.21.15.88:0/722831178 conn(0x55eace0a7660 msgr2=0x55eace0b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:52.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.991+0000 7f4db305d740 1 -- 172.21.15.88:0/722831178 shutdown_connections 2024-09-16T01:35:52.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.991+0000 7f4db305d740 1 -- 172.21.15.88:0/722831178 wait complete. 2024-09-16T01:35:52.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.992+0000 7f4db305d740 1 -- start start 2024-09-16T01:35:52.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.992+0000 7f4db305d740 1 Processor -- start 2024-09-16T01:35:52.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.993+0000 7f4db305d740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a1660 0x55eace0a5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:53.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.993+0000 7f4db305d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eace0a7ac0 0x55eace0b8180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:53.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.993+0000 7f4db305d740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55eace0a9ea0 0x55eace0cc120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:53.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.993+0000 7f4db305d740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55eace0a1ae0 con 0x55eace0a1660 2024-09-16T01:35:53.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.993+0000 7f4db305d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55eace0a4b40 con 0x55eace0a7ac0 2024-09-16T01:35:53.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.993+0000 7f4db305d740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55eace0883a0 con 0x55eace0a9ea0 2024-09-16T01:35:53.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.993+0000 7f4db305d740 10 client.0 _ll_register_callbacks cb 0x7f4d98035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:35:53.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.993+0000 7f4dadf89640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a1660 0x55eace0a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:53.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.993+0000 7f4dadf89640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a1660 0x55eace0a5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40524/0 (socket says 172.21.15.88:40524) 2024-09-16T01:35:53.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.993+0000 7f4dadf89640 1 -- 172.21.15.88:0/2198032869 learned_addr learned my addr 172.21.15.88:0/2198032869 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:53.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.993+0000 7f4dae78a640 1 --2- 172.21.15.88:0/2198032869 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55eace0a9ea0 0x55eace0cc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:53.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.993+0000 7f4dad788640 1 --2- 172.21.15.88:0/2198032869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eace0a7ac0 0x55eace0b8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:53.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.993+0000 7f4dadf89640 1 -- 172.21.15.88:0/2198032869 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55eace0a9ea0 msgr2=0x55eace0cc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:53.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.993+0000 7f4dadf89640 1 --2- 172.21.15.88:0/2198032869 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55eace0a9ea0 0x55eace0cc120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.993+0000 7f4dadf89640 1 -- 172.21.15.88:0/2198032869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eace0a7ac0 msgr2=0x55eace0b8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:53.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.993+0000 7f4dadf89640 1 --2- 172.21.15.88:0/2198032869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eace0a7ac0 0x55eace0b8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.993+0000 7f4dadf89640 1 -- 172.21.15.88:0/2198032869 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eace1200b0 con 0x55eace0a1660 2024-09-16T01:35:53.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.994+0000 7f4dadf89640 1 --2- 172.21.15.88:0/2198032869 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a1660 0x55eace0a5380 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x55eace088e40 tx=0x7f4da4001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a2a0ca0d49fcf18 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:53.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.994+0000 7f4d88ff9640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:35:53.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.994+0000 7f4d88ff9640 1 -- 172.21.15.88:0/2198032869 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da4014d90 con 0x55eace0a1660 2024-09-16T01:35:53.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.994+0000 7f4d88ff9640 1 -- 172.21.15.88:0/2198032869 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f4da402d410 con 0x55eace0a1660 2024-09-16T01:35:53.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.994+0000 7f4db305d740 1 -- 172.21.15.88:0/2198032869 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4da40147d0 con 0x55eace0a1660 2024-09-16T01:35:53.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.994+0000 7f4d88ff9640 1 -- 172.21.15.88:0/2198032869 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f4da403e050 con 0x55eace0a1660 2024-09-16T01:35:53.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.994+0000 7f4db305d740 10 client.5259 fetch_fsmap learned FSMap version 19 2024-09-16T01:35:53.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.994+0000 7f4db305d740 1 -- 172.21.15.88:0/2198032869 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x55eace08b2c0 con 0x55eace0a1660 2024-09-16T01:35:53.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.994+0000 7f4d88ff9640 1 -- 172.21.15.88:0/2198032869 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4da40147d0 con 0x55eace0a1660 2024-09-16T01:35:53.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.994+0000 7f4d88ff9640 1 -- 172.21.15.88:0/2198032869 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 19) ==== 3502+0+0 (secure 0 0 0) 0x7f4da4041020 con 0x55eace0a1660 2024-09-16T01:35:53.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.994+0000 7f4db305d740 10 client.5259 fetch_fsmap finished waiting for FSMap version 19 2024-09-16T01:35:53.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.994+0000 7f4db305d740 10 client.5259 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T01:35:53.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.994+0000 7f4db305d740 20 client.5259 populate_metadata read hostname 'smithi088' 2024-09-16T01:35:53.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.994+0000 7f4db305d740 1 --2- 172.21.15.88:0/2198032869 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x55eace0a0d00 0x55eace0a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:53.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.994+0000 7f4db305d740 4 client.5259 mds_command: new command op to 5067 tid=0 multi_id=0 [{"prefix": "damage rm", "damage_id": 3136708976}] 2024-09-16T01:35:53.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.994+0000 7f4db305d740 1 -- 172.21.15.88:0/2198032869 --> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] -- command(tid 0: {"prefix": "damage rm", "damage_id": 3136708976}) -- 0x55eacdfce350 con 0x55eace0a0d00 2024-09-16T01:35:53.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.994+0000 7f4dad788640 1 --2- 172.21.15.88:0/2198032869 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x55eace0a0d00 0x55eace0a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:53.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.994+0000 7f4d88ff9640 10 client.5259 ms_handle_connect on v2:172.21.15.88:6834/2870334334 2024-09-16T01:35:53.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.995+0000 7f4dad788640 1 --2- 172.21.15.88:0/2198032869 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x55eace0a0d00 0x55eace0a10e0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f619ad6cc72eb715 server_cookie=a962cba984c9566f in_seq=0 out_seq=0 2024-09-16T01:35:53.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.995+0000 7f4d88ff9640 1 -- 172.21.15.88:0/2198032869 <== mds.0 v2:172.21.15.88:6834/2870334334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+0 (crc 0 0 0) 0x55eacdfce350 con 0x55eace0a0d00 2024-09-16T01:35:53.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.995+0000 7f4d88ff9640 10 client.5259 handle_command_reply: tid=0 2024-09-16T01:35:53.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.995+0000 7f4db305d740 1 client.5259 shutdown 2024-09-16T01:35:53.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.995+0000 7f4db305d740 1 -- 172.21.15.88:0/2198032869 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a1660 msgr2=0x55eace0a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:53.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.995+0000 7f4db305d740 1 --2- 172.21.15.88:0/2198032869 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a1660 0x55eace0a5380 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x55eace088e40 tx=0x7f4da4001080 comp rx=0 tx=0).stop 2024-09-16T01:35:53.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.995+0000 7f4db305d740 1 -- 172.21.15.88:0/2198032869 shutdown_connections 2024-09-16T01:35:53.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.995+0000 7f4db305d740 1 --2- 172.21.15.88:0/2198032869 >> [v2:172.21.15.88:6834/2870334334,v1:172.21.15.88:6835/2870334334] conn(0x55eace0a0d00 0x55eace0a10e0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.995+0000 7f4db305d740 1 --2- 172.21.15.88:0/2198032869 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55eace0a9ea0 0x55eace0cc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.995+0000 7f4db305d740 1 --2- 172.21.15.88:0/2198032869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eace0a7ac0 0x55eace0b8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.995+0000 7f4db305d740 1 --2- 172.21.15.88:0/2198032869 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55eace0a1660 0x55eace0a5380 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.995+0000 7f4d88ff9640 0 client.5259 ms_handle_reset on v2:172.21.15.88:6834/2870334334 2024-09-16T01:35:53.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.995+0000 7f4db305d740 1 -- 172.21.15.88:0/2198032869 >> 172.21.15.88:0/2198032869 conn(0x55eace0a7660 msgr2=0x55eace08a930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:53.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.995+0000 7f4db305d740 1 -- 172.21.15.88:0/2198032869 shutdown_connections 2024-09-16T01:35:53.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.995+0000 7f4db305d740 1 -- 172.21.15.88:0/2198032869 wait complete. 2024-09-16T01:35:53.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.995+0000 7f4db305d740 20 client.5259 trim_cache size 0 max 16384 2024-09-16T01:35:53.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.996+0000 7f4db01ed640 1 -- 172.21.15.88:0/2237158275 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d8c06c2e0 msgr2=0x7f4d8c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:53.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.996+0000 7f4db01ed640 1 --2- 172.21.15.88:0/2237158275 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d8c06c2e0 0x7f4d8c06e7a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4da4005880 tx=0x7f4da4007a00 comp rx=0 tx=0).stop 2024-09-16T01:35:53.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.996+0000 7f4db01ed640 1 -- 172.21.15.88:0/2237158275 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4da815e750 msgr2=0x7f4da8157690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:53.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.996+0000 7f4db01ed640 1 --2- 172.21.15.88:0/2237158275 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4da815e750 0x7f4da8157690 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4da000cf20 tx=0x7f4da000a7d0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.996+0000 7f4db01ed640 1 -- 172.21.15.88:0/2237158275 shutdown_connections 2024-09-16T01:35:53.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.996+0000 7f4db01ed640 1 --2- 172.21.15.88:0/2237158275 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d8c06c2e0 0x7f4d8c06e7a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.996+0000 7f4db01ed640 1 --2- 172.21.15.88:0/2237158275 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4da815e750 0x7f4da8157690 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.996+0000 7f4db01ed640 1 --2- 172.21.15.88:0/2237158275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4da80b7f50 0x7f4da81f49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.996+0000 7f4db01ed640 1 --2- 172.21.15.88:0/2237158275 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4da80b64a0 0x7f4da8156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.996+0000 7f4db01ed640 1 -- 172.21.15.88:0/2237158275 >> 172.21.15.88:0/2237158275 conn(0x7f4da80d1500 msgr2=0x7f4da80b83c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:53.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.996+0000 7f4db01ed640 1 -- 172.21.15.88:0/2237158275 shutdown_connections 2024-09-16T01:35:53.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:52.996+0000 7f4db01ed640 1 -- 172.21.15.88:0/2237158275 wait complete. 2024-09-16T01:35:53.008 DEBUG:teuthology.orchestra.run.smithi088:> stdin-killer -- python3 -c ' 2024-09-16T01:35:53.008 DEBUG:teuthology.orchestra.run.smithi088:> import sys 2024-09-16T01:35:53.008 DEBUG:teuthology.orchestra.run.smithi088:> import errno 2024-09-16T01:35:53.008 DEBUG:teuthology.orchestra.run.smithi088:> 2024-09-16T01:35:53.008 DEBUG:teuthology.orchestra.run.smithi088:> try: 2024-09-16T01:35:53.008 DEBUG:teuthology.orchestra.run.smithi088:> f = open("/home/ubuntu/cephtest/mnt.0/subdir/file_to_be_damaged", "w") 2024-09-16T01:35:53.008 DEBUG:teuthology.orchestra.run.smithi088:> f.close() 2024-09-16T01:35:53.008 DEBUG:teuthology.orchestra.run.smithi088:> except IOError as e: 2024-09-16T01:35:53.008 DEBUG:teuthology.orchestra.run.smithi088:> sys.exit(errno.EIO) 2024-09-16T01:35:53.009 DEBUG:teuthology.orchestra.run.smithi088:> ' 2024-09-16T01:35:53.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:35:53.084 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T01:35:53.084 DEBUG:teuthology.orchestra.run.smithi088:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T01:35:53.174 DEBUG:teuthology.orchestra.run.smithi154:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T01:35:53.226 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T01:35:53.226 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T01:35:53.226 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:35:53.226 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:35:53.242 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:35:53.242 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:35:53.300 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:35:53.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.438+0000 7f7780b66640 1 Processor -- start 2024-09-16T01:35:53.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.438+0000 7f7780b66640 1 -- start start 2024-09-16T01:35:53.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.439+0000 7f7780b66640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f777c15a050 0x7f777c15d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:53.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.439+0000 7f7780b66640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f777c0731c0 0x7f777c073600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:53.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.439+0000 7f7780b66640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f777c15a430 con 0x7f777c15ffd0 2024-09-16T01:35:53.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.439+0000 7f7780b66640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f777c073b40 con 0x7f777c0731c0 2024-09-16T01:35:53.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.439+0000 7f7780b66640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f777c15d9a0 con 0x7f777c15a050 2024-09-16T01:35:53.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.439+0000 7f7779d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f777c0731c0 0x7f777c073600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:53.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.439+0000 7f777ad9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f777c15ffd0 0x7f777c160440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:48814/0 (socket says 172.21.15.88:48814) 2024-09-16T01:35:53.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.439+0000 7f777ad9d640 1 -- 172.21.15.88:0/903996109 learned_addr learned my addr 172.21.15.88:0/903996109 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:53.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.439+0000 7f777a59c640 1 --2- 172.21.15.88:0/903996109 >> v2:172.21.15.154:3300/0 conn(0x7f777c15a050 0x7f777c15d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:53.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.439+0000 7f7779d9b640 1 -- 172.21.15.88:0/903996109 >> v2:172.21.15.154:3300/0 conn(0x7f777c15a050 msgr2=0x7f777c15d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:53.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.439+0000 7f7779d9b640 1 --2- 172.21.15.88:0/903996109 >> v2:172.21.15.154:3300/0 conn(0x7f777c15a050 0x7f777c15d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.439+0000 7f7779d9b640 1 -- 172.21.15.88:0/903996109 >> v1:172.21.15.88:6789/0 conn(0x7f777c15ffd0 legacy=0x7f777c160440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:53.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.439+0000 7f7779d9b640 1 -- 172.21.15.88:0/903996109 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f777c0cd0a0 con 0x7f777c0731c0 2024-09-16T01:35:53.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.439+0000 7f777959a640 1 -- 172.21.15.88:0/903996109 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3368197317 0 0) 0x7f777c15a430 con 0x7f777c15ffd0 2024-09-16T01:35:53.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.439+0000 7f777a59c640 1 --2- 172.21.15.88:0/903996109 >> v2:172.21.15.154:3300/0 conn(0x7f777c15a050 0x7f777c15d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:53.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.440+0000 7f7779d9b640 1 --2- 172.21.15.88:0/903996109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f777c0731c0 0x7f777c073600 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f777c069470 tx=0x7f777002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da8bc833b3ddebb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:53.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.440+0000 7f777959a640 1 -- 172.21.15.88:0/903996109 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7770037070 con 0x7f777c0731c0 2024-09-16T01:35:53.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.440+0000 7f777959a640 1 -- 172.21.15.88:0/903996109 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7770032440 con 0x7f777c0731c0 2024-09-16T01:35:53.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.440+0000 7f777959a640 1 -- 172.21.15.88:0/903996109 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f777002ae60 con 0x7f777c0731c0 2024-09-16T01:35:53.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.440+0000 7f7780b66640 1 -- 172.21.15.88:0/903996109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f777c0731c0 msgr2=0x7f777c073600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:53.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.440+0000 7f7780b66640 1 --2- 172.21.15.88:0/903996109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f777c0731c0 0x7f777c073600 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f777c069470 tx=0x7f777002a110 comp rx=0 tx=0).stop 2024-09-16T01:35:53.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.441+0000 7f7780b66640 1 -- 172.21.15.88:0/903996109 shutdown_connections 2024-09-16T01:35:53.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.441+0000 7f7780b66640 1 --2- 172.21.15.88:0/903996109 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f777c0731c0 0x7f777c073600 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.441+0000 7f7780b66640 1 --2- 172.21.15.88:0/903996109 >> v2:172.21.15.154:3300/0 conn(0x7f777c15a050 0x7f777c15d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.441+0000 7f7780b66640 1 -- 172.21.15.88:0/903996109 >> 172.21.15.88:0/903996109 conn(0x7f777c06d0f0 msgr2=0x7f777c06d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:53.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.441+0000 7f7780b66640 1 -- 172.21.15.88:0/903996109 shutdown_connections 2024-09-16T01:35:53.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.441+0000 7f7780b66640 1 -- 172.21.15.88:0/903996109 wait complete. 2024-09-16T01:35:53.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.442+0000 7f7780b66640 1 Processor -- start 2024-09-16T01:35:53.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.442+0000 7f7780b66640 1 -- start start 2024-09-16T01:35:53.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.443+0000 7f7780b66640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f777c0731c0 0x7f777c151970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:53.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.443+0000 7f7780b66640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f777c15a050 0x7f777c1547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:53.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.443+0000 7f7780b66640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f777c15ffd0 0x7f777c0cc070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:53.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.443+0000 7f7780b66640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f777c0cd940 con 0x7f777c0731c0 2024-09-16T01:35:53.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.443+0000 7f7780b66640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f777c0cd660 con 0x7f777c15ffd0 2024-09-16T01:35:53.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.443+0000 7f7780b66640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f777c0cdc20 con 0x7f777c15a050 2024-09-16T01:35:53.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.443+0000 7f777a59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f777c0731c0 0x7f777c151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:53.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.443+0000 7f777a59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f777c0731c0 0x7f777c151970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40550/0 (socket says 172.21.15.88:40550) 2024-09-16T01:35:53.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.443+0000 7f777a59c640 1 -- 172.21.15.88:0/1343308721 learned_addr learned my addr 172.21.15.88:0/1343308721 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:53.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.443+0000 7f7779d9b640 1 --2- 172.21.15.88:0/1343308721 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f777c15a050 0x7f777c1547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:53.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.443+0000 7f777ad9d640 1 --2- 172.21.15.88:0/1343308721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f777c15ffd0 0x7f777c0cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:53.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.443+0000 7f777a59c640 1 -- 172.21.15.88:0/1343308721 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f777c15a050 msgr2=0x7f777c1547a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:53.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.443+0000 7f777a59c640 1 --2- 172.21.15.88:0/1343308721 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f777c15a050 0x7f777c1547a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.443+0000 7f777a59c640 1 -- 172.21.15.88:0/1343308721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f777c15ffd0 msgr2=0x7f777c0cc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:53.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.443+0000 7f777a59c640 1 --2- 172.21.15.88:0/1343308721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f777c15ffd0 0x7f777c0cc070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.443+0000 7f777a59c640 1 -- 172.21.15.88:0/1343308721 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f777c154ce0 con 0x7f777c0731c0 2024-09-16T01:35:53.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.443+0000 7f777a59c640 1 --2- 172.21.15.88:0/1343308721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f777c0731c0 0x7f777c151970 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f776c00c820 tx=0x7f776c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:53.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.443+0000 7f776affd640 1 -- 172.21.15.88:0/1343308721 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c015070 con 0x7f777c0731c0 2024-09-16T01:35:53.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.443+0000 7f776affd640 1 -- 172.21.15.88:0/1343308721 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f776c007a30 con 0x7f777c0731c0 2024-09-16T01:35:53.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.443+0000 7f7780b66640 1 -- 172.21.15.88:0/1343308721 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f777c151fc0 con 0x7f777c0731c0 2024-09-16T01:35:53.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.443+0000 7f776affd640 1 -- 172.21.15.88:0/1343308721 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c007e00 con 0x7f777c0731c0 2024-09-16T01:35:53.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.443+0000 7f7780b66640 1 -- 172.21.15.88:0/1343308721 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f777c152860 con 0x7f777c0731c0 2024-09-16T01:35:53.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.444+0000 7f7780b66640 1 -- 172.21.15.88:0/1343308721 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f777c0b36e0 con 0x7f777c0731c0 2024-09-16T01:35:53.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.445+0000 7f776affd640 1 -- 172.21.15.88:0/1343308721 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f776c018070 con 0x7f777c0731c0 2024-09-16T01:35:53.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.445+0000 7f776affd640 1 --2- 172.21.15.88:0/1343308721 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f775c06c2b0 0x7f775c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:53.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.445+0000 7f776affd640 1 -- 172.21.15.88:0/1343308721 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f776c08b370 con 0x7f777c0731c0 2024-09-16T01:35:53.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.445+0000 7f7779d9b640 1 --2- 172.21.15.88:0/1343308721 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f775c06c2b0 0x7f775c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:53.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.446+0000 7f7779d9b640 1 --2- 172.21.15.88:0/1343308721 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f775c06c2b0 0x7f775c06e770 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7770034000 tx=0x7f777002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:53.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.450+0000 7f776affd640 1 -- 172.21.15.88:0/1343308721 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f776c059b10 con 0x7f777c0731c0 2024-09-16T01:35:53.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.626+0000 7f7780b66640 1 -- 172.21.15.88:0/1343308721 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f777c0d0fe0 con 0x7f777c0731c0 2024-09-16T01:35:53.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.627+0000 7f776affd640 1 -- 172.21.15.88:0/1343308721 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v33) ==== 80+0+122 (secure 0 0 0) 0x7f776c059410 con 0x7f777c0731c0 2024-09-16T01:35:53.628 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/4230443772 2024-09-17T01:35:12.937737+0000 2024-09-16T01:35:53.629 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/4230443772 2024-09-17T01:35:12.937737+0000 2024-09-16T01:35:53.629 INFO:teuthology.orchestra.run.smithi088.stderr:listed 2 entries 2024-09-16T01:35:53.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.629+0000 7f7780b66640 1 -- 172.21.15.88:0/1343308721 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f775c06c2b0 msgr2=0x7f775c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:53.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.629+0000 7f7780b66640 1 --2- 172.21.15.88:0/1343308721 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f775c06c2b0 0x7f775c06e770 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7770034000 tx=0x7f777002e000 comp rx=0 tx=0).stop 2024-09-16T01:35:53.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.629+0000 7f7780b66640 1 -- 172.21.15.88:0/1343308721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f777c0731c0 msgr2=0x7f777c151970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:53.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.629+0000 7f7780b66640 1 --2- 172.21.15.88:0/1343308721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f777c0731c0 0x7f777c151970 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f776c00c820 tx=0x7f776c00cc20 comp rx=0 tx=0).stop 2024-09-16T01:35:53.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.629+0000 7f7780b66640 1 -- 172.21.15.88:0/1343308721 shutdown_connections 2024-09-16T01:35:53.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.629+0000 7f7780b66640 1 --2- 172.21.15.88:0/1343308721 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f775c06c2b0 0x7f775c06e770 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.629+0000 7f7780b66640 1 --2- 172.21.15.88:0/1343308721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f777c15ffd0 0x7f777c0cc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.629+0000 7f7780b66640 1 --2- 172.21.15.88:0/1343308721 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f777c15a050 0x7f777c1547a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.629+0000 7f7780b66640 1 --2- 172.21.15.88:0/1343308721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f777c0731c0 0x7f777c151970 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.629+0000 7f7780b66640 1 -- 172.21.15.88:0/1343308721 >> 172.21.15.88:0/1343308721 conn(0x7f777c06d0f0 msgr2=0x7f777c0d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:53.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.629+0000 7f7780b66640 1 -- 172.21.15.88:0/1343308721 shutdown_connections 2024-09-16T01:35:53.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.629+0000 7f7780b66640 1 -- 172.21.15.88:0/1343308721 wait complete. 2024-09-16T01:35:53.641 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-16T01:35:53.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:53.641 DEBUG:teuthology.orchestra.run.smithi088:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-16T01:35:53.668 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:35:53.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:35:53.669 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:35:53.724 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T01:35:53.725 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:35:53.725 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:35:53.744 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056026,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1347 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T01:35:53.744 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T01:35:53.744 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:35:53.744 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:35:53.780 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:35:53.780 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T01:35:53.780 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:35:53.780 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:35:53.802 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056026,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1347 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T01:35:53.802 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:35:53.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.916+0000 7f3064352640 1 Processor -- start 2024-09-16T01:35:53.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.916+0000 7f3064352640 1 -- start start 2024-09-16T01:35:53.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.917+0000 7f3064352640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f305c15e950 0x7f305c06ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:53.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.917+0000 7f3064352640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f305c0735b0 0x7f305c06d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:53.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.917+0000 7f3064352640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f305c072050 0x7f305c06bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:53.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.917+0000 7f3064352640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f305c06faf0 con 0x7f305c072050 2024-09-16T01:35:53.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.917+0000 7f3064352640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f305c06fdd0 con 0x7f305c0735b0 2024-09-16T01:35:53.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.917+0000 7f3064352640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f305c0700b0 con 0x7f305c15e950 2024-09-16T01:35:53.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.917+0000 7f30628ef640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f305c072050 0x7f305c06bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:53.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.917+0000 7f30628ef640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f305c072050 0x7f305c06bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40562/0 (socket says 172.21.15.88:40562) 2024-09-16T01:35:53.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.917+0000 7f30628ef640 1 -- 172.21.15.88:0/357767904 learned_addr learned my addr 172.21.15.88:0/357767904 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:53.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.917+0000 7f30628ef640 1 -- 172.21.15.88:0/357767904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f305c15e950 msgr2=0x7f305c06ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:53.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.917+0000 7f30628ef640 1 --2- 172.21.15.88:0/357767904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f305c15e950 0x7f305c06ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.917+0000 7f30628ef640 1 -- 172.21.15.88:0/357767904 >> v2:172.21.15.154:3300/0 conn(0x7f305c0735b0 msgr2=0x7f305c06d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:53.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.917+0000 7f30628ef640 1 --2- 172.21.15.88:0/357767904 >> v2:172.21.15.154:3300/0 conn(0x7f305c0735b0 0x7f305c06d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.917+0000 7f30628ef640 1 -- 172.21.15.88:0/357767904 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f305c070390 con 0x7f305c072050 2024-09-16T01:35:53.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.918+0000 7f30628ef640 1 --2- 172.21.15.88:0/357767904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f305c072050 0x7f305c06bc10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f305c06f220 tx=0x7f305402a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c17a33297585227 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:53.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.918+0000 7f30610ec640 1 -- 172.21.15.88:0/357767904 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3054036070 con 0x7f305c072050 2024-09-16T01:35:53.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.918+0000 7f30610ec640 1 -- 172.21.15.88:0/357767904 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3054031600 con 0x7f305c072050 2024-09-16T01:35:53.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.918+0000 7f30610ec640 1 -- 172.21.15.88:0/357767904 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30540319d0 con 0x7f305c072050 2024-09-16T01:35:53.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.918+0000 7f3064352640 1 -- 172.21.15.88:0/357767904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f305c072050 msgr2=0x7f305c06bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:53.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.918+0000 7f3064352640 1 --2- 172.21.15.88:0/357767904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f305c072050 0x7f305c06bc10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f305c06f220 tx=0x7f305402a6c0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.919+0000 7f3064352640 1 -- 172.21.15.88:0/357767904 shutdown_connections 2024-09-16T01:35:53.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.919+0000 7f3064352640 1 --2- 172.21.15.88:0/357767904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f305c072050 0x7f305c06bc10 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.919+0000 7f3064352640 1 --2- 172.21.15.88:0/357767904 >> v2:172.21.15.154:3300/0 conn(0x7f305c0735b0 0x7f305c06d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.919+0000 7f3064352640 1 --2- 172.21.15.88:0/357767904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f305c15e950 0x7f305c06ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.919+0000 7f3064352640 1 -- 172.21.15.88:0/357767904 >> 172.21.15.88:0/357767904 conn(0x7f305c0d14d0 msgr2=0x7f305c0d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:53.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.919+0000 7f3064352640 1 -- 172.21.15.88:0/357767904 shutdown_connections 2024-09-16T01:35:53.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.919+0000 7f3064352640 1 -- 172.21.15.88:0/357767904 wait complete. 2024-09-16T01:35:53.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.920+0000 7f3064352640 1 Processor -- start 2024-09-16T01:35:53.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.920+0000 7f3064352640 1 -- start start 2024-09-16T01:35:53.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.920+0000 7f3064352640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f305c072050 0x7f305c161910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:53.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.920+0000 7f3064352640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f305c0735b0 0x7f305c15f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:53.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.920+0000 7f3064352640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f305c15e950 0x7f305c147fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:53.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.920+0000 7f3064352640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f305c06c2d0 con 0x7f305c15e950 2024-09-16T01:35:53.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.920+0000 7f3064352640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f305c06c150 con 0x7f305c0735b0 2024-09-16T01:35:53.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.920+0000 7f3064352640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f305c06c450 con 0x7f305c072050 2024-09-16T01:35:53.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.921+0000 7f30628ef640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f305c15e950 0x7f305c147fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:53.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.921+0000 7f30628ef640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f305c15e950 0x7f305c147fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40568/0 (socket says 172.21.15.88:40568) 2024-09-16T01:35:53.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.921+0000 7f30628ef640 1 -- 172.21.15.88:0/1802446987 learned_addr learned my addr 172.21.15.88:0/1802446987 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:53.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.921+0000 7f30620ee640 1 --2- 172.21.15.88:0/1802446987 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f305c072050 0x7f305c161910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:53.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.921+0000 7f30618ed640 1 --2- 172.21.15.88:0/1802446987 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f305c0735b0 0x7f305c15f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:53.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.921+0000 7f30628ef640 1 -- 172.21.15.88:0/1802446987 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f305c072050 msgr2=0x7f305c161910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:53.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.921+0000 7f30628ef640 1 --2- 172.21.15.88:0/1802446987 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f305c072050 0x7f305c161910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.921+0000 7f30628ef640 1 -- 172.21.15.88:0/1802446987 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f305c0735b0 msgr2=0x7f305c15f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:53.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.921+0000 7f30628ef640 1 --2- 172.21.15.88:0/1802446987 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f305c0735b0 0x7f305c15f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:53.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.921+0000 7f30628ef640 1 -- 172.21.15.88:0/1802446987 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f305c15fe70 con 0x7f305c15e950 2024-09-16T01:35:53.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.921+0000 7f30628ef640 1 --2- 172.21.15.88:0/1802446987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f305c15e950 0x7f305c147fe0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3054034040 tx=0x7f3054004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:53.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.921+0000 7f3052ffd640 1 -- 172.21.15.88:0/1802446987 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3054041020 con 0x7f305c15e950 2024-09-16T01:35:53.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.921+0000 7f3052ffd640 1 -- 172.21.15.88:0/1802446987 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30540077a0 con 0x7f305c15e950 2024-09-16T01:35:53.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.921+0000 7f3064352640 1 -- 172.21.15.88:0/1802446987 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f305c160150 con 0x7f305c15e950 2024-09-16T01:35:53.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.921+0000 7f3052ffd640 1 -- 172.21.15.88:0/1802446987 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3054036040 con 0x7f305c15e950 2024-09-16T01:35:53.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.922+0000 7f3064352640 1 -- 172.21.15.88:0/1802446987 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f305c158500 con 0x7f305c15e950 2024-09-16T01:35:53.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.922+0000 7f3064352640 1 -- 172.21.15.88:0/1802446987 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f305c160670 con 0x7f305c15e950 2024-09-16T01:35:53.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.923+0000 7f3052ffd640 1 -- 172.21.15.88:0/1802446987 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3054044020 con 0x7f305c15e950 2024-09-16T01:35:53.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.923+0000 7f3052ffd640 1 --2- 172.21.15.88:0/1802446987 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f304006c2b0 0x7f304006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:53.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.923+0000 7f3052ffd640 1 -- 172.21.15.88:0/1802446987 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f30540ac250 con 0x7f305c15e950 2024-09-16T01:35:53.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.923+0000 7f30620ee640 1 --2- 172.21.15.88:0/1802446987 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f304006c2b0 0x7f304006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:53.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.924+0000 7f30620ee640 1 --2- 172.21.15.88:0/1802446987 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f304006c2b0 0x7f304006e770 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f30580098e0 tx=0x7f3058009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:53.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:53.927+0000 7f3052ffd640 1 -- 172.21.15.88:0/1802446987 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f305404b020 con 0x7f305c15e950 2024-09-16T01:35:54.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.133+0000 7f3064352640 1 -- 172.21.15.88:0/1802446987 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f305c06c8b0 con 0x7f305c15e950 2024-09-16T01:35:54.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.134+0000 7f3052ffd640 1 -- 172.21.15.88:0/1802446987 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5230 (secure 0 0 0) 0x7f305407a3a0 con 0x7f305c15e950 2024-09-16T01:35:54.135 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:35:54.135 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":19,"btime":"2024-09-16T01:35:16:950413+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1005331674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1005331674},{"type":"v1","addr":"172.21.15.88:6837","nonce":1005331674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/3427247662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3427247662},{"type":"v1","addr":"172.21.15.154:6834","nonce":3427247662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/4059835337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":4059835337},{"type":"v1","addr":"172.21.15.154:6835","nonce":4059835337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:35:00.674725+0000","modified":"2024-09-16T01:35:16.950408+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"c","rank":0,"incarnation":16,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6835/2870334334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870334334},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870334334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5067,"qdb_cluster":[5067]},"id":2}]} 2024-09-16T01:35:54.135 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 19 2024-09-16T01:35:54.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.136+0000 7f3064352640 1 -- 172.21.15.88:0/1802446987 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f304006c2b0 msgr2=0x7f304006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:54.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.136+0000 7f3064352640 1 --2- 172.21.15.88:0/1802446987 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f304006c2b0 0x7f304006e770 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f30580098e0 tx=0x7f3058009060 comp rx=0 tx=0).stop 2024-09-16T01:35:54.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.136+0000 7f3064352640 1 -- 172.21.15.88:0/1802446987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f305c15e950 msgr2=0x7f305c147fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:54.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.136+0000 7f3064352640 1 --2- 172.21.15.88:0/1802446987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f305c15e950 0x7f305c147fe0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3054034040 tx=0x7f3054004a40 comp rx=0 tx=0).stop 2024-09-16T01:35:54.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.136+0000 7f3064352640 1 -- 172.21.15.88:0/1802446987 shutdown_connections 2024-09-16T01:35:54.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.136+0000 7f3064352640 1 --2- 172.21.15.88:0/1802446987 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f304006c2b0 0x7f304006e770 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.136+0000 7f3064352640 1 --2- 172.21.15.88:0/1802446987 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f305c15e950 0x7f305c147fe0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.136+0000 7f3064352640 1 --2- 172.21.15.88:0/1802446987 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f305c0735b0 0x7f305c15f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.136+0000 7f3064352640 1 --2- 172.21.15.88:0/1802446987 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f305c072050 0x7f305c161910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.136+0000 7f3064352640 1 -- 172.21.15.88:0/1802446987 >> 172.21.15.88:0/1802446987 conn(0x7f305c0d14d0 msgr2=0x7f305c0cf0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:54.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.136+0000 7f3064352640 1 -- 172.21.15.88:0/1802446987 shutdown_connections 2024-09-16T01:35:54.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.136+0000 7f3064352640 1 -- 172.21.15.88:0/1802446987 wait complete. 2024-09-16T01:35:54.148 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:35:54.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.244+0000 7f3aa6d11640 1 Processor -- start 2024-09-16T01:35:54.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.245+0000 7f3aa6d11640 1 -- start start 2024-09-16T01:35:54.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.245+0000 7f3aa6d11640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3aa0073ce0 0x7f3aa0169260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:54.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.245+0000 7f3aa6d11640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3aa015eb10 con 0x7f3aa015c100 2024-09-16T01:35:54.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.245+0000 7f3aa6d11640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3aa015edd0 con 0x7f3aa006e910 2024-09-16T01:35:54.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.245+0000 7f3aa6d11640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3aa0074170 con 0x7f3aa0073ce0 2024-09-16T01:35:54.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.246+0000 7f3aa4aad640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3aa015c100 0x7f3aa006e200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:48822/0 (socket says 172.21.15.88:48822) 2024-09-16T01:35:54.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.246+0000 7f3aa4aad640 1 -- 172.21.15.88:0/2588307736 learned_addr learned my addr 172.21.15.88:0/2588307736 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:54.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.246+0000 7f3a9f7fe640 1 -- 172.21.15.88:0/2588307736 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3485591342 0 0) 0x7f3aa015eb10 con 0x7f3aa015c100 2024-09-16T01:35:54.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.246+0000 7f3a9f7fe640 1 -- 172.21.15.88:0/2588307736 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a8c003520 con 0x7f3aa015c100 2024-09-16T01:35:54.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.246+0000 7f3aa52ae640 1 --2- 172.21.15.88:0/2588307736 >> v2:172.21.15.154:3300/0 conn(0x7f3aa0073ce0 0x7f3aa0169260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:54.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.246+0000 7f3a9f7fe640 1 -- 172.21.15.88:0/2588307736 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 989135811 0 0) 0x7f3a8c003520 con 0x7f3aa015c100 2024-09-16T01:35:54.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.246+0000 7f3a9f7fe640 1 -- 172.21.15.88:0/2588307736 >> v2:172.21.15.154:3300/0 conn(0x7f3aa0073ce0 msgr2=0x7f3aa0169260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:54.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.246+0000 7f3a9f7fe640 1 --2- 172.21.15.88:0/2588307736 >> v2:172.21.15.154:3300/0 conn(0x7f3aa0073ce0 0x7f3aa0169260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.246+0000 7f3a9f7fe640 1 -- 172.21.15.88:0/2588307736 >> v1:172.21.15.154:6789/0 conn(0x7f3aa006e910 legacy=0x7f3aa00735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:54.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.246+0000 7f3a9f7fe640 1 -- 172.21.15.88:0/2588307736 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aa015f0b0 con 0x7f3aa015c100 2024-09-16T01:35:54.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.246+0000 7f3a9f7fe640 1 -- 172.21.15.88:0/2588307736 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f3a98001670 con 0x7f3aa015c100 2024-09-16T01:35:54.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.246+0000 7f3a9f7fe640 1 -- 172.21.15.88:0/2588307736 >> v1:172.21.15.88:6789/0 conn(0x7f3aa015c100 legacy=0x7f3aa006e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:54.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.246+0000 7f3a9f7fe640 1 --2- 172.21.15.88:0/2588307736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a8c0039b0 0x7f3a8c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:54.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.246+0000 7f3a9f7fe640 1 --2- 172.21.15.88:0/2588307736 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a8c004800 0x7f3a8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:54.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.246+0000 7f3a9f7fe640 1 --2- 172.21.15.88:0/2588307736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a8c0051c0 0x7f3a8c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:54.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.246+0000 7f3a9f7fe640 1 -- 172.21.15.88:0/2588307736 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa0074170 con 0x7f3a8c0051c0 2024-09-16T01:35:54.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.246+0000 7f3a9f7fe640 1 -- 172.21.15.88:0/2588307736 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3a8c00c060 con 0x7f3a8c0039b0 2024-09-16T01:35:54.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.246+0000 7f3a9f7fe640 1 -- 172.21.15.88:0/2588307736 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3a8c00c260 con 0x7f3a8c004800 2024-09-16T01:35:54.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.246+0000 7f3aa52ae640 1 --2- 172.21.15.88:0/2588307736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a8c0051c0 0x7f3a8c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:54.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.246+0000 7f3aa4aad640 1 --2- 172.21.15.88:0/2588307736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a8c0039b0 0x7f3a8c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:54.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.246+0000 7f3a9ffff640 1 --2- 172.21.15.88:0/2588307736 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a8c004800 0x7f3a8c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:54.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.247+0000 7f3aa52ae640 1 -- 172.21.15.88:0/2588307736 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a8c004800 msgr2=0x7f3a8c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:54.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.247+0000 7f3aa52ae640 1 --2- 172.21.15.88:0/2588307736 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a8c004800 0x7f3a8c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.247+0000 7f3aa52ae640 1 -- 172.21.15.88:0/2588307736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a8c0039b0 msgr2=0x7f3a8c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:54.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.247+0000 7f3aa52ae640 1 --2- 172.21.15.88:0/2588307736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a8c0039b0 0x7f3a8c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.247+0000 7f3aa52ae640 1 -- 172.21.15.88:0/2588307736 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a8c003520 con 0x7f3a8c0051c0 2024-09-16T01:35:54.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.247+0000 7f3a9ffff640 1 --2- 172.21.15.88:0/2588307736 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a8c004800 0x7f3a8c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:54.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.247+0000 7f3aa52ae640 1 -- 172.21.15.88:0/2588307736 reap_dead start 2024-09-16T01:35:54.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.247+0000 7f3aa52ae640 1 --2- 172.21.15.88:0/2588307736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a8c0051c0 0x7f3a8c00b9c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3a88009ac0 tx=0x7f3a8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=901fe20779e38c0c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:54.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.247+0000 7f3a9f7fe640 1 -- 172.21.15.88:0/2588307736 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a88009690 con 0x7f3a8c0051c0 2024-09-16T01:35:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.247+0000 7f3a9f7fe640 1 -- 172.21.15.88:0/2588307736 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a88004020 con 0x7f3a8c0051c0 2024-09-16T01:35:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.247+0000 7f3a9f7fe640 1 -- 172.21.15.88:0/2588307736 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a88004410 con 0x7f3a8c0051c0 2024-09-16T01:35:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.248+0000 7f3aa6d11640 1 -- 172.21.15.88:0/2588307736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a8c0051c0 msgr2=0x7f3a8c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.248+0000 7f3aa6d11640 1 --2- 172.21.15.88:0/2588307736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a8c0051c0 0x7f3a8c00b9c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3a88009ac0 tx=0x7f3a8c005830 comp rx=0 tx=0).stop 2024-09-16T01:35:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.248+0000 7f3aa6d11640 1 -- 172.21.15.88:0/2588307736 shutdown_connections 2024-09-16T01:35:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.248+0000 7f3aa6d11640 1 --2- 172.21.15.88:0/2588307736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a8c0051c0 0x7f3a8c00b9c0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.248+0000 7f3aa6d11640 1 -- 172.21.15.88:0/2588307736 >> 172.21.15.88:0/2588307736 conn(0x7f3aa00b6ec0 msgr2=0x7f3aa00b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.248+0000 7f3aa6d11640 1 -- 172.21.15.88:0/2588307736 shutdown_connections 2024-09-16T01:35:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.248+0000 7f3aa6d11640 1 -- 172.21.15.88:0/2588307736 wait complete. 2024-09-16T01:35:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.249+0000 7f3aa6d11640 1 Processor -- start 2024-09-16T01:35:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.249+0000 7f3aa6d11640 1 -- start start 2024-09-16T01:35:54.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.250+0000 7f3aa6d11640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a8c0051c0 0x7f3aa0144650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:54.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.250+0000 7f3aa6d11640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3aa0162df0 0x7f3aa015fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:54.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.250+0000 7f3aa6d11640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3aa01601f0 0x7f3aa00b54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:54.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.250+0000 7f3aa6d11640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa015c4e0 con 0x7f3a8c0051c0 2024-09-16T01:35:54.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.250+0000 7f3aa6d11640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa015f670 con 0x7f3aa01601f0 2024-09-16T01:35:54.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.250+0000 7f3aa6d11640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3aa00735d0 con 0x7f3aa0162df0 2024-09-16T01:35:54.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.250+0000 7f3aa4aad640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a8c0051c0 0x7f3aa0144650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:54.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.250+0000 7f3aa4aad640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a8c0051c0 0x7f3aa0144650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40586/0 (socket says 172.21.15.88:40586) 2024-09-16T01:35:54.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.250+0000 7f3aa4aad640 1 -- 172.21.15.88:0/3196569063 learned_addr learned my addr 172.21.15.88:0/3196569063 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:54.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.250+0000 7f3a9ffff640 1 --2- 172.21.15.88:0/3196569063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3aa0162df0 0x7f3aa015fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:54.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.250+0000 7f3aa52ae640 1 --2- 172.21.15.88:0/3196569063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3aa01601f0 0x7f3aa00b54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:54.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.250+0000 7f3aa4aad640 1 -- 172.21.15.88:0/3196569063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3aa0162df0 msgr2=0x7f3aa015fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:54.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.250+0000 7f3aa4aad640 1 --2- 172.21.15.88:0/3196569063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3aa0162df0 0x7f3aa015fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.250+0000 7f3aa4aad640 1 -- 172.21.15.88:0/3196569063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3aa01601f0 msgr2=0x7f3aa00b54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:54.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.250+0000 7f3aa4aad640 1 --2- 172.21.15.88:0/3196569063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3aa01601f0 0x7f3aa00b54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.250+0000 7f3aa4aad640 1 -- 172.21.15.88:0/3196569063 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aa0161d50 con 0x7f3a8c0051c0 2024-09-16T01:35:54.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.251+0000 7f3aa4aad640 1 --2- 172.21.15.88:0/3196569063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a8c0051c0 0x7f3aa0144650 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3a98004ea0 tx=0x7f3a98004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:54.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.251+0000 7f3a9d7fa640 1 -- 172.21.15.88:0/3196569063 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9800f6e0 con 0x7f3a8c0051c0 2024-09-16T01:35:54.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.251+0000 7f3a9d7fa640 1 -- 172.21.15.88:0/3196569063 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a98018070 con 0x7f3a8c0051c0 2024-09-16T01:35:54.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.251+0000 7f3aa6d11640 1 -- 172.21.15.88:0/3196569063 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3aa0162030 con 0x7f3a8c0051c0 2024-09-16T01:35:54.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.251+0000 7f3a9d7fa640 1 -- 172.21.15.88:0/3196569063 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9800fae0 con 0x7f3a8c0051c0 2024-09-16T01:35:54.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.251+0000 7f3aa6d11640 1 -- 172.21.15.88:0/3196569063 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3aa01625f0 con 0x7f3a8c0051c0 2024-09-16T01:35:54.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.252+0000 7f3aa6d11640 1 -- 172.21.15.88:0/3196569063 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3aa0073960 con 0x7f3a8c0051c0 2024-09-16T01:35:54.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.252+0000 7f3a9d7fa640 1 -- 172.21.15.88:0/3196569063 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3a9800fc80 con 0x7f3a8c0051c0 2024-09-16T01:35:54.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.253+0000 7f3a9d7fa640 1 --2- 172.21.15.88:0/3196569063 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a8006c2e0 0x7f3a8006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:54.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.253+0000 7f3a9d7fa640 1 -- 172.21.15.88:0/3196569063 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f3a9808b400 con 0x7f3a8c0051c0 2024-09-16T01:35:54.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.253+0000 7f3a9ffff640 1 --2- 172.21.15.88:0/3196569063 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a8006c2e0 0x7f3a8006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:54.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.253+0000 7f3a9ffff640 1 --2- 172.21.15.88:0/3196569063 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a8006c2e0 0x7f3a8006e7a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3a90007fd0 tx=0x7f3a90009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:54.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.257+0000 7f3a9d7fa640 1 -- 172.21.15.88:0/3196569063 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3a98059b10 con 0x7f3a8c0051c0 2024-09-16T01:35:54.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.455+0000 7f3aa6d11640 1 -- 172.21.15.88:0/3196569063 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3aa006d220 con 0x7f3a8c0051c0 2024-09-16T01:35:54.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.455+0000 7f3a9d7fa640 1 -- 172.21.15.88:0/3196569063 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5230 (secure 0 0 0) 0x7f3a98059410 con 0x7f3a8c0051c0 2024-09-16T01:35:54.457 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:35:54.457 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":19,"btime":"2024-09-16T01:35:16:950413+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1005331674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1005331674},{"type":"v1","addr":"172.21.15.88:6837","nonce":1005331674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/3427247662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3427247662},{"type":"v1","addr":"172.21.15.154:6834","nonce":3427247662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/4059835337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":4059835337},{"type":"v1","addr":"172.21.15.154:6835","nonce":4059835337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:35:00.674725+0000","modified":"2024-09-16T01:35:16.950408+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"c","rank":0,"incarnation":16,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6835/2870334334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870334334},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870334334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5067,"qdb_cluster":[5067]},"id":2}]} 2024-09-16T01:35:54.457 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 19 2024-09-16T01:35:54.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.457+0000 7f3aa6d11640 1 -- 172.21.15.88:0/3196569063 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a8006c2e0 msgr2=0x7f3a8006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:54.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.457+0000 7f3aa6d11640 1 --2- 172.21.15.88:0/3196569063 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a8006c2e0 0x7f3a8006e7a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3a90007fd0 tx=0x7f3a90009040 comp rx=0 tx=0).stop 2024-09-16T01:35:54.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.457+0000 7f3aa6d11640 1 -- 172.21.15.88:0/3196569063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a8c0051c0 msgr2=0x7f3aa0144650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:54.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.457+0000 7f3aa6d11640 1 --2- 172.21.15.88:0/3196569063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a8c0051c0 0x7f3aa0144650 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3a98004ea0 tx=0x7f3a98004f90 comp rx=0 tx=0).stop 2024-09-16T01:35:54.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.457+0000 7f3aa6d11640 1 -- 172.21.15.88:0/3196569063 shutdown_connections 2024-09-16T01:35:54.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.457+0000 7f3aa6d11640 1 --2- 172.21.15.88:0/3196569063 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a8006c2e0 0x7f3a8006e7a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.457+0000 7f3aa6d11640 1 --2- 172.21.15.88:0/3196569063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3aa01601f0 0x7f3aa00b54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.457+0000 7f3aa6d11640 1 --2- 172.21.15.88:0/3196569063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3aa0162df0 0x7f3aa015fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.457+0000 7f3aa6d11640 1 --2- 172.21.15.88:0/3196569063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a8c0051c0 0x7f3aa0144650 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.457+0000 7f3aa6d11640 1 -- 172.21.15.88:0/3196569063 >> 172.21.15.88:0/3196569063 conn(0x7f3aa00b6ec0 msgr2=0x7f3aa015e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:54.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.457+0000 7f3aa6d11640 1 -- 172.21.15.88:0/3196569063 shutdown_connections 2024-09-16T01:35:54.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.457+0000 7f3aa6d11640 1 -- 172.21.15.88:0/3196569063 wait complete. 2024-09-16T01:35:54.469 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:35:54.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.567+0000 7feec2cba640 1 Processor -- start 2024-09-16T01:35:54.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.567+0000 7feec2cba640 1 -- start start 2024-09-16T01:35:54.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.568+0000 7feec2cba640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feebc0b4920 0x7feebc15b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:54.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.568+0000 7feec2cba640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7feebc0cd780 0x7feebc0cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:54.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.568+0000 7feec2cba640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feebc0ce4f0 con 0x7feebc0b4d00 2024-09-16T01:35:54.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.568+0000 7feec2cba640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7feebc0cf100 con 0x7feebc0cd780 2024-09-16T01:35:54.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.568+0000 7feec2cba640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7feebc0ce7d0 con 0x7feebc0b4920 2024-09-16T01:35:54.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.568+0000 7feec1257640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7feebc0cd780 0x7feebc0cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:54.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.568+0000 7feec1257640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7feebc0cd780 0x7feebc0cdbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40600/0 (socket says 172.21.15.88:40600) 2024-09-16T01:35:54.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.568+0000 7feec1257640 1 -- 172.21.15.88:0/481224931 learned_addr learned my addr 172.21.15.88:0/481224931 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:54.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.568+0000 7feec0a56640 1 --2- 172.21.15.88:0/481224931 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feebc0b4920 0x7feebc15b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:54.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.568+0000 7feec1257640 1 -- 172.21.15.88:0/481224931 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feebc0b4920 msgr2=0x7feebc15b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:54.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.568+0000 7feec1257640 1 --2- 172.21.15.88:0/481224931 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feebc0b4920 0x7feebc15b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.568+0000 7feec1257640 1 -- 172.21.15.88:0/481224931 >> v1:172.21.15.154:6789/0 conn(0x7feebc0b4d00 legacy=0x7feebc0b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:54.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.568+0000 7feec1257640 1 -- 172.21.15.88:0/481224931 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feebc0ceab0 con 0x7feebc0cd780 2024-09-16T01:35:54.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.569+0000 7feec1257640 1 --2- 172.21.15.88:0/481224931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feebc0cd780 0x7feebc0cdbf0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7feeb400d150 tx=0x7feeb402c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2deb2bb09d80f4f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:54.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.569+0000 7feebb7fe640 1 -- 172.21.15.88:0/481224931 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb400b2e0 con 0x7feebc0cd780 2024-09-16T01:35:54.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.569+0000 7feebb7fe640 1 -- 172.21.15.88:0/481224931 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feeb4009600 con 0x7feebc0cd780 2024-09-16T01:35:54.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.569+0000 7feebb7fe640 1 -- 172.21.15.88:0/481224931 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb400b480 con 0x7feebc0cd780 2024-09-16T01:35:54.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.570+0000 7feec2cba640 1 -- 172.21.15.88:0/481224931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feebc0cd780 msgr2=0x7feebc0cdbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:54.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.570+0000 7feec2cba640 1 --2- 172.21.15.88:0/481224931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feebc0cd780 0x7feebc0cdbf0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7feeb400d150 tx=0x7feeb402c8e0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.570+0000 7feec2cba640 1 -- 172.21.15.88:0/481224931 shutdown_connections 2024-09-16T01:35:54.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.570+0000 7feec2cba640 1 --2- 172.21.15.88:0/481224931 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feebc0cd780 0x7feebc0cdbf0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.570+0000 7feec2cba640 1 --2- 172.21.15.88:0/481224931 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feebc0b4920 0x7feebc15b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.570+0000 7feec2cba640 1 -- 172.21.15.88:0/481224931 >> 172.21.15.88:0/481224931 conn(0x7feebc06bc10 msgr2=0x7feebc06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:54.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.570+0000 7feec2cba640 1 -- 172.21.15.88:0/481224931 shutdown_connections 2024-09-16T01:35:54.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.570+0000 7feec2cba640 1 -- 172.21.15.88:0/481224931 wait complete. 2024-09-16T01:35:54.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.571+0000 7feec2cba640 1 Processor -- start 2024-09-16T01:35:54.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.571+0000 7feec2cba640 1 -- start start 2024-09-16T01:35:54.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.572+0000 7feec2cba640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feebc0b4920 0x7feebc1e9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:54.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.572+0000 7feec2cba640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feebc0b4d00 0x7feebc1e3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:54.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.572+0000 7feec2cba640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feebc0cd780 0x7feebc1e4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:54.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.572+0000 7feec2cba640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7feebc0ce130 con 0x7feebc0cd780 2024-09-16T01:35:54.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.572+0000 7feec2cba640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7feebc15bd10 con 0x7feebc0b4d00 2024-09-16T01:35:54.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.572+0000 7feec2cba640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7feebc0ce2b0 con 0x7feebc0b4920 2024-09-16T01:35:54.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.572+0000 7feec1257640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feebc0cd780 0x7feebc1e4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:54.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.572+0000 7feec1257640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feebc0cd780 0x7feebc1e4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40614/0 (socket says 172.21.15.88:40614) 2024-09-16T01:35:54.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.572+0000 7feec1257640 1 -- 172.21.15.88:0/1138090141 learned_addr learned my addr 172.21.15.88:0/1138090141 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:54.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.572+0000 7feec0a56640 1 --2- 172.21.15.88:0/1138090141 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feebc0b4920 0x7feebc1e9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:54.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.572+0000 7feebbfff640 1 --2- 172.21.15.88:0/1138090141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feebc0b4d00 0x7feebc1e3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:54.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.572+0000 7feec1257640 1 -- 172.21.15.88:0/1138090141 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feebc0b4920 msgr2=0x7feebc1e9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:54.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.572+0000 7feec1257640 1 --2- 172.21.15.88:0/1138090141 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feebc0b4920 0x7feebc1e9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.572+0000 7feec1257640 1 -- 172.21.15.88:0/1138090141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feebc0b4d00 msgr2=0x7feebc1e3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:54.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.572+0000 7feec1257640 1 --2- 172.21.15.88:0/1138090141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feebc0b4d00 0x7feebc1e3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.572+0000 7feec1257640 1 -- 172.21.15.88:0/1138090141 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feebc156520 con 0x7feebc0cd780 2024-09-16T01:35:54.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.572+0000 7feebbfff640 1 --2- 172.21.15.88:0/1138090141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feebc0b4d00 0x7feebc1e3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:54.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.572+0000 7feec1257640 1 --2- 172.21.15.88:0/1138090141 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feebc0cd780 0x7feebc1e4000 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7feeb400d270 tx=0x7feeb4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:54.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.572+0000 7feeb97fa640 1 -- 172.21.15.88:0/1138090141 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb4040070 con 0x7feebc0cd780 2024-09-16T01:35:54.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.573+0000 7feeb97fa640 1 -- 172.21.15.88:0/1138090141 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feeb4009370 con 0x7feebc0cd780 2024-09-16T01:35:54.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.573+0000 7feec2cba640 1 -- 172.21.15.88:0/1138090141 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feebc156800 con 0x7feebc0cd780 2024-09-16T01:35:54.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.573+0000 7feeb97fa640 1 -- 172.21.15.88:0/1138090141 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb400b720 con 0x7feebc0cd780 2024-09-16T01:35:54.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.573+0000 7feec2cba640 1 -- 172.21.15.88:0/1138090141 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feebc156dc0 con 0x7feebc0cd780 2024-09-16T01:35:54.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.574+0000 7feec2cba640 1 -- 172.21.15.88:0/1138090141 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feebc15b7d0 con 0x7feebc0cd780 2024-09-16T01:35:54.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.574+0000 7feeb97fa640 1 -- 172.21.15.88:0/1138090141 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7feeb4043070 con 0x7feebc0cd780 2024-09-16T01:35:54.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.575+0000 7feeb97fa640 1 --2- 172.21.15.88:0/1138090141 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7feea006c2b0 0x7feea006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:54.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.575+0000 7feec0a56640 1 --2- 172.21.15.88:0/1138090141 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7feea006c2b0 0x7feea006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:54.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.575+0000 7feeb97fa640 1 -- 172.21.15.88:0/1138090141 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7feeb4030020 con 0x7feebc0cd780 2024-09-16T01:35:54.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.575+0000 7feec0a56640 1 --2- 172.21.15.88:0/1138090141 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7feea006c2b0 0x7feea006e770 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7feeb0006fd0 tx=0x7feeb0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:54.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.579+0000 7feeb97fa640 1 -- 172.21.15.88:0/1138090141 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7feeb404a030 con 0x7feebc0cd780 2024-09-16T01:35:54.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.745+0000 7feec2cba640 1 -- 172.21.15.88:0/1138090141 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feebc0cb850 con 0x7feebc0cd780 2024-09-16T01:35:54.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.746+0000 7feeb97fa640 1 -- 172.21.15.88:0/1138090141 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16834 (secure 0 0 0) 0x7feeb407a9a0 con 0x7feebc0cd780 2024-09-16T01:35:54.748 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:35:54.748 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":33,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:35:12.937760+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:34:58.555164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-16T01:34:59.655760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:6835/4230443772":"2024-09-17T01:35:12.937737+0000","172.21.15.88:6834/4230443772":"2024-09-17T01:35:12.937737+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:35:54.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.748+0000 7feec2cba640 1 -- 172.21.15.88:0/1138090141 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7feea006c2b0 msgr2=0x7feea006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:54.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.748+0000 7feec2cba640 1 --2- 172.21.15.88:0/1138090141 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7feea006c2b0 0x7feea006e770 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7feeb0006fd0 tx=0x7feeb0008040 comp rx=0 tx=0).stop 2024-09-16T01:35:54.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.748+0000 7feec2cba640 1 -- 172.21.15.88:0/1138090141 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feebc0cd780 msgr2=0x7feebc1e4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:54.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.748+0000 7feec2cba640 1 --2- 172.21.15.88:0/1138090141 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feebc0cd780 0x7feebc1e4000 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7feeb400d270 tx=0x7feeb4004a40 comp rx=0 tx=0).stop 2024-09-16T01:35:54.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.748+0000 7feec2cba640 1 -- 172.21.15.88:0/1138090141 shutdown_connections 2024-09-16T01:35:54.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.748+0000 7feec2cba640 1 --2- 172.21.15.88:0/1138090141 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7feea006c2b0 0x7feea006e770 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.748+0000 7feec2cba640 1 --2- 172.21.15.88:0/1138090141 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feebc0cd780 0x7feebc1e4000 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.748+0000 7feec2cba640 1 --2- 172.21.15.88:0/1138090141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feebc0b4d00 0x7feebc1e3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.748+0000 7feec2cba640 1 --2- 172.21.15.88:0/1138090141 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feebc0b4920 0x7feebc1e9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.748+0000 7feec2cba640 1 -- 172.21.15.88:0/1138090141 >> 172.21.15.88:0/1138090141 conn(0x7feebc06bc10 msgr2=0x7feebc0b58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:54.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.748+0000 7feec2cba640 1 -- 172.21.15.88:0/1138090141 shutdown_connections 2024-09-16T01:35:54.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.748+0000 7feec2cba640 1 -- 172.21.15.88:0/1138090141 wait complete. 2024-09-16T01:35:54.761 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-16T01:35:54.761 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:35:54.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.858+0000 7fdcba884640 1 Processor -- start 2024-09-16T01:35:54.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.858+0000 7fdcba884640 1 -- start start 2024-09-16T01:35:54.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.859+0000 7fdcba884640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdcb415c170 0x7fdcb415c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:54.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.859+0000 7fdcba884640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdcb40b64e0 0x7fdcb40cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:54.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.859+0000 7fdcba884640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcb40b4cf0 con 0x7fdcb40cf6c0 2024-09-16T01:35:54.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.859+0000 7fdcba884640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fdcb415ca90 con 0x7fdcb40b64e0 2024-09-16T01:35:54.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.859+0000 7fdcba884640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdcb40b49c0 con 0x7fdcb415c170 2024-09-16T01:35:54.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.859+0000 7fdcb37fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fdcb40cf6c0 0x7fdcb40b5dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:48824/0 (socket says 172.21.15.88:48824) 2024-09-16T01:35:54.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.859+0000 7fdcb37fe640 1 -- 172.21.15.88:0/217397655 learned_addr learned my addr 172.21.15.88:0/217397655 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:54.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.859+0000 7fdcb3fff640 1 --2- 172.21.15.88:0/217397655 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdcb415c170 0x7fdcb415c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:54.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.859+0000 7fdcb8e21640 1 --2- 172.21.15.88:0/217397655 >> v2:172.21.15.154:3300/0 conn(0x7fdcb40b64e0 0x7fdcb40cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:54.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.859+0000 7fdcb2ffd640 1 -- 172.21.15.88:0/217397655 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4100293387 0 0) 0x7fdcb40b4cf0 con 0x7fdcb40cf6c0 2024-09-16T01:35:54.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.859+0000 7fdcb2ffd640 1 -- 172.21.15.88:0/217397655 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc9c003520 con 0x7fdcb40cf6c0 2024-09-16T01:35:54.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.860+0000 7fdcb2ffd640 1 -- 172.21.15.88:0/217397655 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1619684582 0 0) 0x7fdc9c003520 con 0x7fdcb40cf6c0 2024-09-16T01:35:54.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.860+0000 7fdcb2ffd640 1 -- 172.21.15.88:0/217397655 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdcb415c170 msgr2=0x7fdcb415c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:54.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.860+0000 7fdcb2ffd640 1 --2- 172.21.15.88:0/217397655 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdcb415c170 0x7fdcb415c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.860+0000 7fdcb2ffd640 1 -- 172.21.15.88:0/217397655 >> v2:172.21.15.154:3300/0 conn(0x7fdcb40b64e0 msgr2=0x7fdcb40cfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:54.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.860+0000 7fdcb2ffd640 1 --2- 172.21.15.88:0/217397655 >> v2:172.21.15.154:3300/0 conn(0x7fdcb40b64e0 0x7fdcb40cfb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.860+0000 7fdcb2ffd640 1 -- 172.21.15.88:0/217397655 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb40b4fd0 con 0x7fdcb40cf6c0 2024-09-16T01:35:54.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.860+0000 7fdcb2ffd640 1 -- 172.21.15.88:0/217397655 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fdca8001670 con 0x7fdcb40cf6c0 2024-09-16T01:35:54.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.860+0000 7fdcb2ffd640 1 -- 172.21.15.88:0/217397655 >> v1:172.21.15.88:6789/0 conn(0x7fdcb40cf6c0 legacy=0x7fdcb40b5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:54.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.860+0000 7fdcb2ffd640 1 --2- 172.21.15.88:0/217397655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc9c0039b0 0x7fdc9c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:54.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.860+0000 7fdcb2ffd640 1 --2- 172.21.15.88:0/217397655 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdcb415c170 0x7fdc9c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:54.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.860+0000 7fdcb2ffd640 1 --2- 172.21.15.88:0/217397655 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc9c004d40 0x7fdc9c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:54.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.860+0000 7fdcb2ffd640 1 -- 172.21.15.88:0/217397655 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb415ca90 con 0x7fdc9c004d40 2024-09-16T01:35:54.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.860+0000 7fdcb2ffd640 1 -- 172.21.15.88:0/217397655 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb40b49c0 con 0x7fdc9c0039b0 2024-09-16T01:35:54.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.860+0000 7fdcb2ffd640 1 -- 172.21.15.88:0/217397655 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdc9c00b9c0 con 0x7fdcb415c170 2024-09-16T01:35:54.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.860+0000 7fdcb8e21640 1 --2- 172.21.15.88:0/217397655 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc9c004d40 0x7fdc9c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:54.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.860+0000 7fdcb37fe640 1 --2- 172.21.15.88:0/217397655 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdcb415c170 0x7fdc9c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:54.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.860+0000 7fdcb3fff640 1 --2- 172.21.15.88:0/217397655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc9c0039b0 0x7fdc9c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:54.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.860+0000 7fdcb8e21640 1 -- 172.21.15.88:0/217397655 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdcb415c170 msgr2=0x7fdc9c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:54.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.860+0000 7fdcb8e21640 1 --2- 172.21.15.88:0/217397655 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdcb415c170 0x7fdc9c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.860+0000 7fdcb8e21640 1 -- 172.21.15.88:0/217397655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc9c0039b0 msgr2=0x7fdc9c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:54.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.860+0000 7fdcb8e21640 1 --2- 172.21.15.88:0/217397655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc9c0039b0 0x7fdc9c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.860+0000 7fdcb8e21640 1 -- 172.21.15.88:0/217397655 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc9c003520 con 0x7fdc9c004d40 2024-09-16T01:35:54.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.861+0000 7fdcb8e21640 1 --2- 172.21.15.88:0/217397655 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc9c004d40 0x7fdc9c0051c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fdcac009fc0 tx=0x7fdc9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61b4e88f514e12cd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:54.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.861+0000 7fdcb2ffd640 1 -- 172.21.15.88:0/217397655 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcac009c10 con 0x7fdc9c004d40 2024-09-16T01:35:54.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.861+0000 7fdcb2ffd640 1 -- 172.21.15.88:0/217397655 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcac036070 con 0x7fdc9c004d40 2024-09-16T01:35:54.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.861+0000 7fdcb2ffd640 1 -- 172.21.15.88:0/217397655 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcac02d070 con 0x7fdc9c004d40 2024-09-16T01:35:54.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.861+0000 7fdcba884640 1 -- 172.21.15.88:0/217397655 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc9c004d40 msgr2=0x7fdc9c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:54.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.861+0000 7fdcba884640 1 --2- 172.21.15.88:0/217397655 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc9c004d40 0x7fdc9c0051c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fdcac009fc0 tx=0x7fdc9c005830 comp rx=0 tx=0).stop 2024-09-16T01:35:54.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.861+0000 7fdcb8e21640 1 -- 172.21.15.88:0/217397655 reap_dead start 2024-09-16T01:35:54.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.861+0000 7fdcba884640 1 -- 172.21.15.88:0/217397655 shutdown_connections 2024-09-16T01:35:54.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.861+0000 7fdcba884640 1 -- 172.21.15.88:0/217397655 >> 172.21.15.88:0/217397655 conn(0x7fdcb406bc10 msgr2=0x7fdcb406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:54.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.861+0000 7fdcba884640 1 -- 172.21.15.88:0/217397655 shutdown_connections 2024-09-16T01:35:54.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.862+0000 7fdcba884640 1 -- 172.21.15.88:0/217397655 wait complete. 2024-09-16T01:35:54.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.863+0000 7fdcba884640 1 Processor -- start 2024-09-16T01:35:54.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.863+0000 7fdcba884640 1 -- start start 2024-09-16T01:35:54.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.864+0000 7fdcba884640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdcb413dad0 0x7fdcb41f64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:54.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.864+0000 7fdcba884640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdcb4140310 0x7fdcb4155d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:54.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.864+0000 7fdcba884640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdcb41562d0 0x7fdcb41509f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:54.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.864+0000 7fdcba884640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb40b5870 con 0x7fdcb413dad0 2024-09-16T01:35:54.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.864+0000 7fdcba884640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb40b5590 con 0x7fdcb41562d0 2024-09-16T01:35:54.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.864+0000 7fdcba884640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdcb40b5b50 con 0x7fdcb4140310 2024-09-16T01:35:54.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.864+0000 7fdcb3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdcb413dad0 0x7fdcb41f64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:54.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.864+0000 7fdcb3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdcb413dad0 0x7fdcb41f64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40626/0 (socket says 172.21.15.88:40626) 2024-09-16T01:35:54.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.864+0000 7fdcb3fff640 1 -- 172.21.15.88:0/881322286 learned_addr learned my addr 172.21.15.88:0/881322286 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:54.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.864+0000 7fdcb37fe640 1 --2- 172.21.15.88:0/881322286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdcb4140310 0x7fdcb4155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:54.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.864+0000 7fdcb8e21640 1 --2- 172.21.15.88:0/881322286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdcb41562d0 0x7fdcb41509f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:54.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.864+0000 7fdcb3fff640 1 -- 172.21.15.88:0/881322286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdcb4140310 msgr2=0x7fdcb4155d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:54.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.864+0000 7fdcb3fff640 1 --2- 172.21.15.88:0/881322286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdcb4140310 0x7fdcb4155d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.864+0000 7fdcb3fff640 1 -- 172.21.15.88:0/881322286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdcb41562d0 msgr2=0x7fdcb41509f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:54.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.864+0000 7fdcb3fff640 1 --2- 172.21.15.88:0/881322286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdcb41562d0 0x7fdcb41509f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:54.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.864+0000 7fdcb3fff640 1 -- 172.21.15.88:0/881322286 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb414e650 con 0x7fdcb413dad0 2024-09-16T01:35:54.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.864+0000 7fdcb37fe640 1 --2- 172.21.15.88:0/881322286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdcb4140310 0x7fdcb4155d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:54.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.864+0000 7fdcb8e21640 1 --2- 172.21.15.88:0/881322286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdcb41562d0 0x7fdcb41509f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:54.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.864+0000 7fdcb3fff640 1 --2- 172.21.15.88:0/881322286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdcb413dad0 0x7fdcb41f64f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fdca40118e0 tx=0x7fdca4011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:54.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.864+0000 7fdcb0ff9640 1 -- 172.21.15.88:0/881322286 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca400fa90 con 0x7fdcb413dad0 2024-09-16T01:35:54.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.864+0000 7fdcb0ff9640 1 -- 172.21.15.88:0/881322286 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdca400fc30 con 0x7fdcb413dad0 2024-09-16T01:35:54.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.864+0000 7fdcba884640 1 -- 172.21.15.88:0/881322286 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcb41407f0 con 0x7fdcb413dad0 2024-09-16T01:35:54.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.864+0000 7fdcb0ff9640 1 -- 172.21.15.88:0/881322286 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca4014440 con 0x7fdcb413dad0 2024-09-16T01:35:54.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.864+0000 7fdcba884640 1 -- 172.21.15.88:0/881322286 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcb4140db0 con 0x7fdcb413dad0 2024-09-16T01:35:54.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.865+0000 7fdcba884640 1 -- 172.21.15.88:0/881322286 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcb41566e0 con 0x7fdcb413dad0 2024-09-16T01:35:54.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.866+0000 7fdcb0ff9640 1 -- 172.21.15.88:0/881322286 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fdca4012040 con 0x7fdcb413dad0 2024-09-16T01:35:54.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.866+0000 7fdcb0ff9640 1 --2- 172.21.15.88:0/881322286 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdca006c2e0 0x7fdca006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:54.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.866+0000 7fdcb37fe640 1 --2- 172.21.15.88:0/881322286 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdca006c2e0 0x7fdca006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:54.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.866+0000 7fdcb0ff9640 1 -- 172.21.15.88:0/881322286 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7fdca408baf0 con 0x7fdcb413dad0 2024-09-16T01:35:54.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.867+0000 7fdcb37fe640 1 --2- 172.21.15.88:0/881322286 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdca006c2e0 0x7fdca006e7a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdca8004840 tx=0x7fdca8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:54.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:54.870+0000 7fdcb0ff9640 1 -- 172.21.15.88:0/881322286 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fdcb41566e0 con 0x7fdcb413dad0 2024-09-16T01:35:55.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.070+0000 7fdcba884640 1 -- 172.21.15.88:0/881322286 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdcb415c550 con 0x7fdcb413dad0 2024-09-16T01:35:55.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.071+0000 7fdcb0ff9640 1 -- 172.21.15.88:0/881322286 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5230 (secure 0 0 0) 0x7fdca4019030 con 0x7fdcb413dad0 2024-09-16T01:35:55.073 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:35:55.073 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":19,"btime":"2024-09-16T01:35:16:950413+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1005331674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1005331674},{"type":"v1","addr":"172.21.15.88:6837","nonce":1005331674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/3427247662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3427247662},{"type":"v1","addr":"172.21.15.154:6834","nonce":3427247662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/4059835337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":4059835337},{"type":"v1","addr":"172.21.15.154:6835","nonce":4059835337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:35:00.674725+0000","modified":"2024-09-16T01:35:16.950408+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"c","rank":0,"incarnation":16,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6835/2870334334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870334334},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870334334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5067,"qdb_cluster":[5067]},"id":2}]} 2024-09-16T01:35:55.073 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 19 2024-09-16T01:35:55.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.073+0000 7fdcba884640 1 -- 172.21.15.88:0/881322286 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdca006c2e0 msgr2=0x7fdca006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:55.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.073+0000 7fdcba884640 1 --2- 172.21.15.88:0/881322286 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdca006c2e0 0x7fdca006e7a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdca8004840 tx=0x7fdca8009040 comp rx=0 tx=0).stop 2024-09-16T01:35:55.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.073+0000 7fdcba884640 1 -- 172.21.15.88:0/881322286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdcb413dad0 msgr2=0x7fdcb41f64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:55.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.073+0000 7fdcba884640 1 --2- 172.21.15.88:0/881322286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdcb413dad0 0x7fdcb41f64f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fdca40118e0 tx=0x7fdca4011da0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.073+0000 7fdcba884640 1 -- 172.21.15.88:0/881322286 shutdown_connections 2024-09-16T01:35:55.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.073+0000 7fdcba884640 1 --2- 172.21.15.88:0/881322286 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdca006c2e0 0x7fdca006e7a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.073+0000 7fdcba884640 1 --2- 172.21.15.88:0/881322286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdcb41562d0 0x7fdcb41509f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.073+0000 7fdcba884640 1 --2- 172.21.15.88:0/881322286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdcb4140310 0x7fdcb4155d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.073+0000 7fdcba884640 1 --2- 172.21.15.88:0/881322286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdcb413dad0 0x7fdcb41f64f0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.073+0000 7fdcba884640 1 -- 172.21.15.88:0/881322286 >> 172.21.15.88:0/881322286 conn(0x7fdcb406bc10 msgr2=0x7fdcb4072650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:55.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.073+0000 7fdcba884640 1 -- 172.21.15.88:0/881322286 shutdown_connections 2024-09-16T01:35:55.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.073+0000 7fdcba884640 1 -- 172.21.15.88:0/881322286 wait complete. 2024-09-16T01:35:55.085 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:35:55.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.182+0000 7f03b0113640 1 Processor -- start 2024-09-16T01:35:55.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.182+0000 7f03b0113640 1 -- start start 2024-09-16T01:35:55.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.183+0000 7f03b0113640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f03a806e800 0x7f03a80ae890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:55.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.183+0000 7f03b0113640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f03a8072d50 0x7f03a80b57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:55.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.183+0000 7f03b0113640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03a80b52c0 con 0x7f03a815e540 2024-09-16T01:35:55.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.183+0000 7f03b0113640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f03a80aedd0 con 0x7f03a806e800 2024-09-16T01:35:55.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.183+0000 7f03b0113640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f03a80af000 con 0x7f03a8072d50 2024-09-16T01:35:55.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.183+0000 7f03adeaf640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f03a815e540 0x7f03a806e0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:48834/0 (socket says 172.21.15.88:48834) 2024-09-16T01:35:55.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.183+0000 7f03adeaf640 1 -- 172.21.15.88:0/3547222698 learned_addr learned my addr 172.21.15.88:0/3547222698 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:55.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.183+0000 7f03ad6ae640 1 --2- 172.21.15.88:0/3547222698 >> v2:172.21.15.88:3300/0 conn(0x7f03a806e800 0x7f03a80ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:55.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.183+0000 7f03ae6b0640 1 --2- 172.21.15.88:0/3547222698 >> v2:172.21.15.154:3300/0 conn(0x7f03a8072d50 0x7f03a80b57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:55.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.183+0000 7f03acead640 1 -- 172.21.15.88:0/3547222698 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 436157882 0 0) 0x7f03a80b52c0 con 0x7f03a815e540 2024-09-16T01:35:55.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.183+0000 7f03acead640 1 -- 172.21.15.88:0/3547222698 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0390003520 con 0x7f03a815e540 2024-09-16T01:35:55.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.183+0000 7f03ad6ae640 1 -- 172.21.15.88:0/3547222698 >> v2:172.21.15.154:3300/0 conn(0x7f03a8072d50 msgr2=0x7f03a80b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:55.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.183+0000 7f03ad6ae640 1 --2- 172.21.15.88:0/3547222698 >> v2:172.21.15.154:3300/0 conn(0x7f03a8072d50 0x7f03a80b57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.183+0000 7f03ad6ae640 1 -- 172.21.15.88:0/3547222698 >> v1:172.21.15.88:6789/0 conn(0x7f03a815e540 legacy=0x7f03a806e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:55.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.183+0000 7f03ad6ae640 1 -- 172.21.15.88:0/3547222698 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03a80af2e0 con 0x7f03a806e800 2024-09-16T01:35:55.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.184+0000 7f03ad6ae640 1 --2- 172.21.15.88:0/3547222698 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03a806e800 0x7f03a80ae890 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f03a806f1c0 tx=0x7f039c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3967b0cea7b04c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:55.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.184+0000 7f03acead640 1 -- 172.21.15.88:0/3547222698 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c037070 con 0x7f03a806e800 2024-09-16T01:35:55.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.184+0000 7f03acead640 1 -- 172.21.15.88:0/3547222698 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f039c032440 con 0x7f03a806e800 2024-09-16T01:35:55.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.184+0000 7f03acead640 1 -- 172.21.15.88:0/3547222698 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c02ae60 con 0x7f03a806e800 2024-09-16T01:35:55.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.185+0000 7f03b0113640 1 -- 172.21.15.88:0/3547222698 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03a806e800 msgr2=0x7f03a80ae890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:55.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.185+0000 7f03b0113640 1 --2- 172.21.15.88:0/3547222698 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03a806e800 0x7f03a80ae890 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f03a806f1c0 tx=0x7f039c02a110 comp rx=0 tx=0).stop 2024-09-16T01:35:55.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.185+0000 7f03b0113640 1 -- 172.21.15.88:0/3547222698 shutdown_connections 2024-09-16T01:35:55.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.185+0000 7f03b0113640 1 --2- 172.21.15.88:0/3547222698 >> v2:172.21.15.154:3300/0 conn(0x7f03a8072d50 0x7f03a80b57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.185+0000 7f03b0113640 1 --2- 172.21.15.88:0/3547222698 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03a806e800 0x7f03a80ae890 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.185+0000 7f03b0113640 1 -- 172.21.15.88:0/3547222698 >> 172.21.15.88:0/3547222698 conn(0x7f03a806bc10 msgr2=0x7f03a806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:55.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.185+0000 7f03b0113640 1 -- 172.21.15.88:0/3547222698 shutdown_connections 2024-09-16T01:35:55.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.185+0000 7f03b0113640 1 -- 172.21.15.88:0/3547222698 wait complete. 2024-09-16T01:35:55.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.186+0000 7f03b0113640 1 Processor -- start 2024-09-16T01:35:55.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.186+0000 7f03b0113640 1 -- start start 2024-09-16T01:35:55.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.187+0000 7f03b0113640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03a806e800 0x7f03a8153620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:55.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.187+0000 7f03b0113640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03a8072d50 0x7f03a8161860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:55.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.187+0000 7f03b0113640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03a815e540 0x7f03a80cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:55.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.187+0000 7f03b0113640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f03a815e920 con 0x7f03a806e800 2024-09-16T01:35:55.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.187+0000 7f03b0113640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f03a80af8a0 con 0x7f03a8072d50 2024-09-16T01:35:55.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.187+0000 7f03b0113640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f03a80b5ce0 con 0x7f03a815e540 2024-09-16T01:35:55.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.187+0000 7f03adeaf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03a806e800 0x7f03a8153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:55.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.187+0000 7f03adeaf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03a806e800 0x7f03a8153620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40656/0 (socket says 172.21.15.88:40656) 2024-09-16T01:35:55.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.187+0000 7f03adeaf640 1 -- 172.21.15.88:0/1234765460 learned_addr learned my addr 172.21.15.88:0/1234765460 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:55.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.187+0000 7f03ad6ae640 1 --2- 172.21.15.88:0/1234765460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03a8072d50 0x7f03a8161860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:55.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.187+0000 7f03ae6b0640 1 --2- 172.21.15.88:0/1234765460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03a815e540 0x7f03a80cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:55.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.187+0000 7f03adeaf640 1 -- 172.21.15.88:0/1234765460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03a815e540 msgr2=0x7f03a80cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:55.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.187+0000 7f03adeaf640 1 --2- 172.21.15.88:0/1234765460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03a815e540 0x7f03a80cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.187+0000 7f03adeaf640 1 -- 172.21.15.88:0/1234765460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03a8072d50 msgr2=0x7f03a8161860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:55.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.187+0000 7f03adeaf640 1 --2- 172.21.15.88:0/1234765460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03a8072d50 0x7f03a8161860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.187+0000 7f03adeaf640 1 -- 172.21.15.88:0/1234765460 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03a80ce4a0 con 0x7f03a806e800 2024-09-16T01:35:55.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.187+0000 7f03adeaf640 1 --2- 172.21.15.88:0/1234765460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03a806e800 0x7f03a8153620 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f039800caa0 tx=0x7f039800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:55.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.187+0000 7f03a67fc640 1 -- 172.21.15.88:0/1234765460 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0398017070 con 0x7f03a806e800 2024-09-16T01:35:55.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.187+0000 7f03a67fc640 1 -- 172.21.15.88:0/1234765460 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f039800ac00 con 0x7f03a806e800 2024-09-16T01:35:55.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.187+0000 7f03b0113640 1 -- 172.21.15.88:0/1234765460 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03a815ecf0 con 0x7f03a806e800 2024-09-16T01:35:55.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.187+0000 7f03a67fc640 1 -- 172.21.15.88:0/1234765460 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0398012400 con 0x7f03a806e800 2024-09-16T01:35:55.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.188+0000 7f03b0113640 1 -- 172.21.15.88:0/1234765460 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03a815f2b0 con 0x7f03a806e800 2024-09-16T01:35:55.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.188+0000 7f03b0113640 1 -- 172.21.15.88:0/1234765460 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03a80ae890 con 0x7f03a806e800 2024-09-16T01:35:55.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.189+0000 7f03a67fc640 1 -- 172.21.15.88:0/1234765460 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f039801a070 con 0x7f03a806e800 2024-09-16T01:35:55.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.189+0000 7f03a67fc640 1 --2- 172.21.15.88:0/1234765460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f039406c2b0 0x7f039406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:55.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.189+0000 7f03a67fc640 1 -- 172.21.15.88:0/1234765460 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f039808b210 con 0x7f03a806e800 2024-09-16T01:35:55.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.189+0000 7f03ad6ae640 1 --2- 172.21.15.88:0/1234765460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f039406c2b0 0x7f039406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:55.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.190+0000 7f03ad6ae640 1 --2- 172.21.15.88:0/1234765460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f039406c2b0 0x7f039406e770 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f039c034000 tx=0x7f039c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:55.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.194+0000 7f03a67fc640 1 -- 172.21.15.88:0/1234765460 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f03980599b0 con 0x7f03a806e800 2024-09-16T01:35:55.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.391+0000 7f03b0113640 1 -- 172.21.15.88:0/1234765460 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03a80b21d0 con 0x7f03a806e800 2024-09-16T01:35:55.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.392+0000 7f03a67fc640 1 -- 172.21.15.88:0/1234765460 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5230 (secure 0 0 0) 0x7f03980592b0 con 0x7f03a806e800 2024-09-16T01:35:55.393 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:35:55.394 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":19,"btime":"2024-09-16T01:35:16:950413+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1005331674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1005331674},{"type":"v1","addr":"172.21.15.88:6837","nonce":1005331674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/3427247662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3427247662},{"type":"v1","addr":"172.21.15.154:6834","nonce":3427247662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/4059835337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":4059835337},{"type":"v1","addr":"172.21.15.154:6835","nonce":4059835337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:35:00.674725+0000","modified":"2024-09-16T01:35:16.950408+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"c","rank":0,"incarnation":16,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6835/2870334334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870334334},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870334334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5067,"qdb_cluster":[5067]},"id":2}]} 2024-09-16T01:35:55.394 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 19 2024-09-16T01:35:55.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.394+0000 7f03b0113640 1 -- 172.21.15.88:0/1234765460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f039406c2b0 msgr2=0x7f039406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:55.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.394+0000 7f03b0113640 1 --2- 172.21.15.88:0/1234765460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f039406c2b0 0x7f039406e770 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f039c034000 tx=0x7f039c02e000 comp rx=0 tx=0).stop 2024-09-16T01:35:55.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.394+0000 7f03b0113640 1 -- 172.21.15.88:0/1234765460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03a806e800 msgr2=0x7f03a8153620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:55.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.394+0000 7f03b0113640 1 --2- 172.21.15.88:0/1234765460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03a806e800 0x7f03a8153620 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f039800caa0 tx=0x7f039800cf60 comp rx=0 tx=0).stop 2024-09-16T01:35:55.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.394+0000 7f03b0113640 1 -- 172.21.15.88:0/1234765460 shutdown_connections 2024-09-16T01:35:55.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.394+0000 7f03b0113640 1 --2- 172.21.15.88:0/1234765460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f039406c2b0 0x7f039406e770 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.394+0000 7f03b0113640 1 --2- 172.21.15.88:0/1234765460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03a815e540 0x7f03a80cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.394+0000 7f03b0113640 1 --2- 172.21.15.88:0/1234765460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03a8072d50 0x7f03a8161860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.394+0000 7f03b0113640 1 --2- 172.21.15.88:0/1234765460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03a806e800 0x7f03a8153620 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.394+0000 7f03b0113640 1 -- 172.21.15.88:0/1234765460 >> 172.21.15.88:0/1234765460 conn(0x7f03a806bc10 msgr2=0x7f03a80b6040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:55.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.394+0000 7f03b0113640 1 -- 172.21.15.88:0/1234765460 shutdown_connections 2024-09-16T01:35:55.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.394+0000 7f03b0113640 1 -- 172.21.15.88:0/1234765460 wait complete. 2024-09-16T01:35:55.406 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:35:55.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.503+0000 7f6b85c23640 1 Processor -- start 2024-09-16T01:35:55.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.503+0000 7f6b85c23640 1 -- start start 2024-09-16T01:35:55.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.504+0000 7f6b85c23640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6b8015b460 0x7f6b800b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:55.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.504+0000 7f6b85c23640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6b800d0fe0 0x7f6b8006cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:55.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.504+0000 7f6b85c23640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b8006d260 0x7f6b8006e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:55.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.504+0000 7f6b85c23640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6b8015eee0 con 0x7f6b8015b460 2024-09-16T01:35:55.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.504+0000 7f6b85c23640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6b800b49f0 con 0x7f6b800d0fe0 2024-09-16T01:35:55.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.504+0000 7f6b85c23640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6b8015f130 con 0x7f6b8006d260 2024-09-16T01:35:55.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.504+0000 7f6b7f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6b8015b460 0x7f6b800b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:55.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.504+0000 7f6b7f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6b8015b460 0x7f6b800b44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40664/0 (socket says 172.21.15.88:40664) 2024-09-16T01:35:55.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.504+0000 7f6b7f7fe640 1 -- 172.21.15.88:0/1710132115 learned_addr learned my addr 172.21.15.88:0/1710132115 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:55.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.504+0000 7f6b7ffff640 1 --2- 172.21.15.88:0/1710132115 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b8006d260 0x7f6b8006e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:55.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.504+0000 7f6b7effd640 1 --2- 172.21.15.88:0/1710132115 >> v2:172.21.15.154:3300/0 conn(0x7f6b800d0fe0 0x7f6b8006cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:55.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.504+0000 7f6b7f7fe640 1 -- 172.21.15.88:0/1710132115 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b8006d260 msgr2=0x7f6b8006e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:55.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.504+0000 7f6b7f7fe640 1 --2- 172.21.15.88:0/1710132115 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b8006d260 0x7f6b8006e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.504+0000 7f6b7f7fe640 1 -- 172.21.15.88:0/1710132115 >> v2:172.21.15.154:3300/0 conn(0x7f6b800d0fe0 msgr2=0x7f6b8006cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:55.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.504+0000 7f6b7f7fe640 1 --2- 172.21.15.88:0/1710132115 >> v2:172.21.15.154:3300/0 conn(0x7f6b800d0fe0 0x7f6b8006cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.504+0000 7f6b7f7fe640 1 -- 172.21.15.88:0/1710132115 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b8015f410 con 0x7f6b8015b460 2024-09-16T01:35:55.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.505+0000 7f6b7f7fe640 1 --2- 172.21.15.88:0/1710132115 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b8015b460 0x7f6b800b44b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6b800b7610 tx=0x7f6b7002c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4839cbcaf3ff1a5d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:55.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.505+0000 7f6b7e7fc640 1 -- 172.21.15.88:0/1710132115 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b70032790 con 0x7f6b8015b460 2024-09-16T01:35:55.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.505+0000 7f6b7e7fc640 1 -- 172.21.15.88:0/1710132115 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b70038070 con 0x7f6b8015b460 2024-09-16T01:35:55.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.505+0000 7f6b7e7fc640 1 -- 172.21.15.88:0/1710132115 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b70032b50 con 0x7f6b8015b460 2024-09-16T01:35:55.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.505+0000 7f6b85c23640 1 -- 172.21.15.88:0/1710132115 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b8015b460 msgr2=0x7f6b800b44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:55.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.505+0000 7f6b85c23640 1 --2- 172.21.15.88:0/1710132115 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b8015b460 0x7f6b800b44b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6b800b7610 tx=0x7f6b7002c890 comp rx=0 tx=0).stop 2024-09-16T01:35:55.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.505+0000 7f6b85c23640 1 -- 172.21.15.88:0/1710132115 shutdown_connections 2024-09-16T01:35:55.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.505+0000 7f6b85c23640 1 --2- 172.21.15.88:0/1710132115 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b8006d260 0x7f6b8006e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.505+0000 7f6b85c23640 1 --2- 172.21.15.88:0/1710132115 >> v2:172.21.15.154:3300/0 conn(0x7f6b800d0fe0 0x7f6b8006cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.505+0000 7f6b85c23640 1 --2- 172.21.15.88:0/1710132115 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b8015b460 0x7f6b800b44b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.505+0000 7f6b85c23640 1 -- 172.21.15.88:0/1710132115 >> 172.21.15.88:0/1710132115 conn(0x7f6b800b6ec0 msgr2=0x7f6b800b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:55.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.506+0000 7f6b85c23640 1 -- 172.21.15.88:0/1710132115 shutdown_connections 2024-09-16T01:35:55.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.506+0000 7f6b85c23640 1 -- 172.21.15.88:0/1710132115 wait complete. 2024-09-16T01:35:55.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.507+0000 7f6b85c23640 1 Processor -- start 2024-09-16T01:35:55.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.507+0000 7f6b85c23640 1 -- start start 2024-09-16T01:35:55.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.508+0000 7f6b85c23640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b8006d260 0x7f6b800c6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:55.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.508+0000 7f6b85c23640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b800d0fe0 0x7f6b800c62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:55.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.508+0000 7f6b85c23640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b8015b460 0x7f6b80157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:55.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.508+0000 7f6b85c23640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8006e740 con 0x7f6b8015b460 2024-09-16T01:35:55.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.508+0000 7f6b85c23640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8015f9d0 con 0x7f6b800d0fe0 2024-09-16T01:35:55.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.508+0000 7f6b85c23640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6b8006e8c0 con 0x7f6b8006d260 2024-09-16T01:35:55.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.508+0000 7f6b7ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b8015b460 0x7f6b80157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:55.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.508+0000 7f6b7ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b8015b460 0x7f6b80157870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40674/0 (socket says 172.21.15.88:40674) 2024-09-16T01:35:55.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.508+0000 7f6b7ffff640 1 -- 172.21.15.88:0/2613523425 learned_addr learned my addr 172.21.15.88:0/2613523425 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:55.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.508+0000 7f6b7f7fe640 1 --2- 172.21.15.88:0/2613523425 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b8006d260 0x7f6b800c6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:55.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.508+0000 7f6b7effd640 1 --2- 172.21.15.88:0/2613523425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b800d0fe0 0x7f6b800c62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:55.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.508+0000 7f6b7ffff640 1 -- 172.21.15.88:0/2613523425 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b8006d260 msgr2=0x7f6b800c6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:55.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.508+0000 7f6b7ffff640 1 --2- 172.21.15.88:0/2613523425 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b8006d260 0x7f6b800c6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.508+0000 7f6b7ffff640 1 -- 172.21.15.88:0/2613523425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b800d0fe0 msgr2=0x7f6b800c62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:55.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.508+0000 7f6b7ffff640 1 --2- 172.21.15.88:0/2613523425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b800d0fe0 0x7f6b800c62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.508+0000 7f6b7ffff640 1 -- 172.21.15.88:0/2613523425 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b8015cb60 con 0x7f6b8015b460 2024-09-16T01:35:55.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.508+0000 7f6b7ffff640 1 --2- 172.21.15.88:0/2613523425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b8015b460 0x7f6b80157870 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f6b78007e60 tx=0x7f6b7800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:55.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.508+0000 7f6b67fff640 1 -- 172.21.15.88:0/2613523425 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b78014620 con 0x7f6b8015b460 2024-09-16T01:35:55.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.508+0000 7f6b67fff640 1 -- 172.21.15.88:0/2613523425 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b78019070 con 0x7f6b8015b460 2024-09-16T01:35:55.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.508+0000 7f6b67fff640 1 -- 172.21.15.88:0/2613523425 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b78014a20 con 0x7f6b8015b460 2024-09-16T01:35:55.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.508+0000 7f6b85c23640 1 -- 172.21.15.88:0/2613523425 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b8015ce40 con 0x7f6b8015b460 2024-09-16T01:35:55.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.508+0000 7f6b85c23640 1 -- 172.21.15.88:0/2613523425 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b8015d400 con 0x7f6b8015b460 2024-09-16T01:35:55.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.509+0000 7f6b85c23640 1 -- 172.21.15.88:0/2613523425 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b800b4c20 con 0x7f6b8015b460 2024-09-16T01:35:55.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.510+0000 7f6b67fff640 1 -- 172.21.15.88:0/2613523425 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6b78014bc0 con 0x7f6b8015b460 2024-09-16T01:35:55.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.510+0000 7f6b67fff640 1 --2- 172.21.15.88:0/2613523425 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6b6c06c2e0 0x7f6b6c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:55.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.510+0000 7f6b67fff640 1 -- 172.21.15.88:0/2613523425 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f6b7808bba0 con 0x7f6b8015b460 2024-09-16T01:35:55.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.510+0000 7f6b7f7fe640 1 --2- 172.21.15.88:0/2613523425 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6b6c06c2e0 0x7f6b6c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:55.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.511+0000 7f6b7f7fe640 1 --2- 172.21.15.88:0/2613523425 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6b6c06c2e0 0x7f6b6c06e7a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6b70037000 tx=0x7f6b70032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:55.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.514+0000 7f6b67fff640 1 -- 172.21.15.88:0/2613523425 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6b78059c60 con 0x7f6b8015b460 2024-09-16T01:35:55.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.682+0000 7f6b85c23640 1 -- 172.21.15.88:0/2613523425 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6b800b44f0 con 0x7f6b8015b460 2024-09-16T01:35:55.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.683+0000 7f6b67fff640 1 -- 172.21.15.88:0/2613523425 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16834 (secure 0 0 0) 0x7f6b7805df60 con 0x7f6b8015b460 2024-09-16T01:35:55.685 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:35:55.685 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":33,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:35:12.937760+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:34:58.555164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-16T01:34:59.655760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:6835/4230443772":"2024-09-17T01:35:12.937737+0000","172.21.15.88:6834/4230443772":"2024-09-17T01:35:12.937737+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:35:55.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.685+0000 7f6b85c23640 1 -- 172.21.15.88:0/2613523425 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6b6c06c2e0 msgr2=0x7f6b6c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:55.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.685+0000 7f6b85c23640 1 --2- 172.21.15.88:0/2613523425 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6b6c06c2e0 0x7f6b6c06e7a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6b70037000 tx=0x7f6b70032000 comp rx=0 tx=0).stop 2024-09-16T01:35:55.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.685+0000 7f6b85c23640 1 -- 172.21.15.88:0/2613523425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b8015b460 msgr2=0x7f6b80157870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:55.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.685+0000 7f6b85c23640 1 --2- 172.21.15.88:0/2613523425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b8015b460 0x7f6b80157870 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f6b78007e60 tx=0x7f6b7800e4b0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.686+0000 7f6b85c23640 1 -- 172.21.15.88:0/2613523425 shutdown_connections 2024-09-16T01:35:55.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.686+0000 7f6b85c23640 1 --2- 172.21.15.88:0/2613523425 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6b6c06c2e0 0x7f6b6c06e7a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.686+0000 7f6b85c23640 1 --2- 172.21.15.88:0/2613523425 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b8015b460 0x7f6b80157870 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.686+0000 7f6b85c23640 1 --2- 172.21.15.88:0/2613523425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b800d0fe0 0x7f6b800c62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.686+0000 7f6b85c23640 1 --2- 172.21.15.88:0/2613523425 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b8006d260 0x7f6b800c6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.686+0000 7f6b85c23640 1 -- 172.21.15.88:0/2613523425 >> 172.21.15.88:0/2613523425 conn(0x7f6b800b6ec0 msgr2=0x7f6b8015a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:55.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.686+0000 7f6b85c23640 1 -- 172.21.15.88:0/2613523425 shutdown_connections 2024-09-16T01:35:55.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.686+0000 7f6b85c23640 1 -- 172.21.15.88:0/2613523425 wait complete. 2024-09-16T01:35:55.698 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:35:55.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.796+0000 7f0958853640 1 Processor -- start 2024-09-16T01:35:55.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.796+0000 7f0958853640 1 -- start start 2024-09-16T01:35:55.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.797+0000 7f0958853640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f09540b6290 0x7f0954071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:55.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.797+0000 7f0958853640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0954072530 0x7f095415bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:55.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.797+0000 7f0958853640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09540ccff0 con 0x7f095415b480 2024-09-16T01:35:55.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.797+0000 7f0958853640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f09540cd2d0 con 0x7f09540b6290 2024-09-16T01:35:55.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.797+0000 7f0958853640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f09540cd5b0 con 0x7f0954072530 2024-09-16T01:35:55.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.797+0000 7f095259c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f095415b480 0x7f09540b5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:48838/0 (socket says 172.21.15.88:48838) 2024-09-16T01:35:55.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.797+0000 7f095259c640 1 -- 172.21.15.88:0/2811167525 learned_addr learned my addr 172.21.15.88:0/2811167525 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:55.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f0951d9b640 1 --2- 172.21.15.88:0/2811167525 >> v2:172.21.15.154:3300/0 conn(0x7f09540b6290 0x7f0954071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:55.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f0952d9d640 1 --2- 172.21.15.88:0/2811167525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0954072530 0x7f095415bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:55.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f095159a640 1 -- 172.21.15.88:0/2811167525 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3616769217 0 0) 0x7f09540ccff0 con 0x7f095415b480 2024-09-16T01:35:55.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f095159a640 1 -- 172.21.15.88:0/2811167525 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0940003520 con 0x7f095415b480 2024-09-16T01:35:55.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f095159a640 1 -- 172.21.15.88:0/2811167525 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2250182770 0 0) 0x7f0940003520 con 0x7f095415b480 2024-09-16T01:35:55.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f095159a640 1 -- 172.21.15.88:0/2811167525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0954072530 msgr2=0x7f095415bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:55.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f095159a640 1 --2- 172.21.15.88:0/2811167525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0954072530 0x7f095415bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f095159a640 1 -- 172.21.15.88:0/2811167525 >> v2:172.21.15.154:3300/0 conn(0x7f09540b6290 msgr2=0x7f0954071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:55.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f095159a640 1 --2- 172.21.15.88:0/2811167525 >> v2:172.21.15.154:3300/0 conn(0x7f09540b6290 0x7f0954071ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f095159a640 1 -- 172.21.15.88:0/2811167525 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09540cd890 con 0x7f095415b480 2024-09-16T01:35:55.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f095159a640 1 -- 172.21.15.88:0/2811167525 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f09440019f0 con 0x7f095415b480 2024-09-16T01:35:55.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f095159a640 1 -- 172.21.15.88:0/2811167525 >> v1:172.21.15.88:6789/0 conn(0x7f095415b480 legacy=0x7f09540b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:55.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f095159a640 1 --2- 172.21.15.88:0/2811167525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09400039b0 0x7f0940003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:55.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f095159a640 1 --2- 172.21.15.88:0/2811167525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0940004800 0x7f0940004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:55.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f095159a640 1 --2- 172.21.15.88:0/2811167525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09400051c0 0x7f094000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:55.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f095159a640 1 -- 172.21.15.88:0/2811167525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f09540cd2d0 con 0x7f09400039b0 2024-09-16T01:35:55.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f095159a640 1 -- 172.21.15.88:0/2811167525 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f09540cd5b0 con 0x7f0940004800 2024-09-16T01:35:55.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f095159a640 1 -- 172.21.15.88:0/2811167525 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f094000bf30 con 0x7f09400051c0 2024-09-16T01:35:55.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f0952d9d640 1 --2- 172.21.15.88:0/2811167525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0954072530 0x7f095415bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:35:55.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f095259c640 1 --2- 172.21.15.88:0/2811167525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09400039b0 0x7f0940003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:55.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f0951d9b640 1 --2- 172.21.15.88:0/2811167525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0940004800 0x7f0940004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:55.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f0952d9d640 1 --2- 172.21.15.88:0/2811167525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09400051c0 0x7f094000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:55.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f095259c640 1 -- 172.21.15.88:0/2811167525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09400051c0 msgr2=0x7f094000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:55.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f095259c640 1 --2- 172.21.15.88:0/2811167525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09400051c0 0x7f094000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f095259c640 1 -- 172.21.15.88:0/2811167525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0940004800 msgr2=0x7f0940004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:55.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f095259c640 1 --2- 172.21.15.88:0/2811167525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0940004800 0x7f0940004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f095259c640 1 -- 172.21.15.88:0/2811167525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0940003520 con 0x7f09400039b0 2024-09-16T01:35:55.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.798+0000 7f0952d9d640 1 --2- 172.21.15.88:0/2811167525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09400051c0 0x7f094000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:35:55.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.799+0000 7f095259c640 1 --2- 172.21.15.88:0/2811167525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09400039b0 0x7f0940003dd0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f094400a920 tx=0x7f0940005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c5cd1f2b24467fd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:55.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.799+0000 7f095159a640 1 -- 172.21.15.88:0/2811167525 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f094400a4c0 con 0x7f09400039b0 2024-09-16T01:35:55.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.799+0000 7f095159a640 1 -- 172.21.15.88:0/2811167525 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f094402f070 con 0x7f09400039b0 2024-09-16T01:35:55.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.799+0000 7f095159a640 1 -- 172.21.15.88:0/2811167525 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f094402bd00 con 0x7f09400039b0 2024-09-16T01:35:55.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.800+0000 7f0958853640 1 -- 172.21.15.88:0/2811167525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09400039b0 msgr2=0x7f0940003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:55.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.800+0000 7f0958853640 1 --2- 172.21.15.88:0/2811167525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09400039b0 0x7f0940003dd0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f094400a920 tx=0x7f0940005830 comp rx=0 tx=0).stop 2024-09-16T01:35:55.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.800+0000 7f0952d9d640 1 -- 172.21.15.88:0/2811167525 reap_dead start 2024-09-16T01:35:55.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.800+0000 7f0958853640 1 -- 172.21.15.88:0/2811167525 shutdown_connections 2024-09-16T01:35:55.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.800+0000 7f0958853640 1 -- 172.21.15.88:0/2811167525 >> 172.21.15.88:0/2811167525 conn(0x7f095406d0f0 msgr2=0x7f095406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:55.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.800+0000 7f0958853640 1 -- 172.21.15.88:0/2811167525 shutdown_connections 2024-09-16T01:35:55.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.800+0000 7f0958853640 1 -- 172.21.15.88:0/2811167525 wait complete. 2024-09-16T01:35:55.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.801+0000 7f0958853640 1 Processor -- start 2024-09-16T01:35:55.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.801+0000 7f0958853640 1 -- start start 2024-09-16T01:35:55.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.802+0000 7f0958853640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09400039b0 0x7f09540d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:55.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.802+0000 7f0958853640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f095415c250 0x7f0954151140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:55.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.802+0000 7f0958853640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0954146e80 0x7f0954147300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:55.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.802+0000 7f0958853640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f095415b9e0 con 0x7f095415c250 2024-09-16T01:35:55.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.802+0000 7f0958853640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f095415b860 con 0x7f09400039b0 2024-09-16T01:35:55.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.802+0000 7f0958853640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f09540b3590 con 0x7f0954146e80 2024-09-16T01:35:55.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.802+0000 7f0951d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f095415c250 0x7f0954151140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:55.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.802+0000 7f0951d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f095415c250 0x7f0954151140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40690/0 (socket says 172.21.15.88:40690) 2024-09-16T01:35:55.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.802+0000 7f0951d9b640 1 -- 172.21.15.88:0/295835841 learned_addr learned my addr 172.21.15.88:0/295835841 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:55.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.802+0000 7f0952d9d640 1 --2- 172.21.15.88:0/295835841 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0954146e80 0x7f0954147300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:55.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.802+0000 7f095259c640 1 --2- 172.21.15.88:0/295835841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09400039b0 0x7f09540d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:55.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.802+0000 7f0951d9b640 1 -- 172.21.15.88:0/295835841 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0954146e80 msgr2=0x7f0954147300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:55.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.802+0000 7f0951d9b640 1 --2- 172.21.15.88:0/295835841 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0954146e80 0x7f0954147300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.802+0000 7f0951d9b640 1 -- 172.21.15.88:0/295835841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09400039b0 msgr2=0x7f09540d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:55.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.802+0000 7f0951d9b640 1 --2- 172.21.15.88:0/295835841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09400039b0 0x7f09540d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:55.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.802+0000 7f0951d9b640 1 -- 172.21.15.88:0/295835841 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09540cfab0 con 0x7f095415c250 2024-09-16T01:35:55.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.802+0000 7f0951d9b640 1 --2- 172.21.15.88:0/295835841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f095415c250 0x7f0954151140 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f093c0115d0 tx=0x7f093c011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:55.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.803+0000 7f094affd640 1 -- 172.21.15.88:0/295835841 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093c015870 con 0x7f095415c250 2024-09-16T01:35:55.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.803+0000 7f094affd640 1 -- 172.21.15.88:0/295835841 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f093c015a10 con 0x7f095415c250 2024-09-16T01:35:55.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.803+0000 7f0958853640 1 -- 172.21.15.88:0/295835841 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09540cfd90 con 0x7f095415c250 2024-09-16T01:35:55.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.803+0000 7f094affd640 1 -- 172.21.15.88:0/295835841 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093c015e00 con 0x7f095415c250 2024-09-16T01:35:55.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.803+0000 7f0958853640 1 -- 172.21.15.88:0/295835841 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09540d0350 con 0x7f095415c250 2024-09-16T01:35:55.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.803+0000 7f0958853640 1 -- 172.21.15.88:0/295835841 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0954071ff0 con 0x7f095415c250 2024-09-16T01:35:55.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.804+0000 7f094affd640 1 -- 172.21.15.88:0/295835841 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f093c00f480 con 0x7f095415c250 2024-09-16T01:35:55.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.804+0000 7f094affd640 1 --2- 172.21.15.88:0/295835841 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f093406c2b0 0x7f093406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:55.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.804+0000 7f094affd640 1 -- 172.21.15.88:0/295835841 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5795+0+0 (secure 0 0 0) 0x7f093c08bc60 con 0x7f095415c250 2024-09-16T01:35:55.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.804+0000 7f095259c640 1 --2- 172.21.15.88:0/295835841 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f093406c2b0 0x7f093406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:55.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.805+0000 7f095259c640 1 --2- 172.21.15.88:0/295835841 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f093406c2b0 0x7f093406e770 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0944033000 tx=0x7f0944007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:55.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:55.809+0000 7f094affd640 1 -- 172.21.15.88:0/295835841 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f093c019030 con 0x7f095415c250 2024-09-16T01:35:56.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.008+0000 7f0958853640 1 -- 172.21.15.88:0/295835841 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f09540b4640 con 0x7f095415c250 2024-09-16T01:35:56.010 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:35:56.009+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:35:56.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.010+0000 7f094affd640 1 -- 172.21.15.88:0/295835841 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v20) ==== 161+0+0 (secure 0 0 0) 0x7f093c05a400 con 0x7f095415c250 2024-09-16T01:35:56.012 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:35:56.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.013+0000 7f0958853640 1 -- 172.21.15.88:0/295835841 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f093406c2b0 msgr2=0x7f093406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:56.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.013+0000 7f0958853640 1 --2- 172.21.15.88:0/295835841 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f093406c2b0 0x7f093406e770 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0944033000 tx=0x7f0944007000 comp rx=0 tx=0).stop 2024-09-16T01:35:56.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.013+0000 7f0958853640 1 -- 172.21.15.88:0/295835841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f095415c250 msgr2=0x7f0954151140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:56.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.013+0000 7f0958853640 1 --2- 172.21.15.88:0/295835841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f095415c250 0x7f0954151140 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f093c0115d0 tx=0x7f093c011a90 comp rx=0 tx=0).stop 2024-09-16T01:35:56.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.014+0000 7f0958853640 1 -- 172.21.15.88:0/295835841 shutdown_connections 2024-09-16T01:35:56.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.014+0000 7f0958853640 1 --2- 172.21.15.88:0/295835841 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f093406c2b0 0x7f093406e770 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:56.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.014+0000 7f0958853640 1 --2- 172.21.15.88:0/295835841 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0954146e80 0x7f0954147300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:56.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.014+0000 7f0958853640 1 --2- 172.21.15.88:0/295835841 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f095415c250 0x7f0954151140 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:56.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.014+0000 7f0958853640 1 --2- 172.21.15.88:0/295835841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09400039b0 0x7f09540d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:56.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.014+0000 7f0958853640 1 -- 172.21.15.88:0/295835841 >> 172.21.15.88:0/295835841 conn(0x7f095406d0f0 msgr2=0x7f09540b4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:56.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.014+0000 7f0958853640 1 -- 172.21.15.88:0/295835841 shutdown_connections 2024-09-16T01:35:56.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.014+0000 7f0958853640 1 -- 172.21.15.88:0/295835841 wait complete. 2024-09-16T01:35:56.017 INFO:tasks.ceph.mds.c.smithi088.stderr: -3303> 2024-09-16T01:35:18.946+0000 7f22afb75640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:35:56.026 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-16T01:35:56.047 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:35:56.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.126+0000 7f67560af640 1 Processor -- start 2024-09-16T01:35:56.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.126+0000 7f67560af640 1 -- start start 2024-09-16T01:35:56.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.127+0000 7f67560af640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f67500b54d0 0x7f675006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:56.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.127+0000 7f67560af640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f675006e990 con 0x7f67500b7450 2024-09-16T01:35:56.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.127+0000 7f67560af640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f675006ec70 con 0x7f675015e6f0 2024-09-16T01:35:56.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.127+0000 7f67560af640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f675015ec70 con 0x7f67500b54d0 2024-09-16T01:35:56.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.127+0000 7f674effd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f67500b54d0 0x7f675006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:56.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.127+0000 7f674ffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f67500b7450 0x7f6750070830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:48864/0 (socket says 172.21.15.88:48864) 2024-09-16T01:35:56.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.127+0000 7f674ffff640 1 -- 172.21.15.88:0/1549586697 learned_addr learned my addr 172.21.15.88:0/1549586697 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:56.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.127+0000 7f674effd640 1 -- 172.21.15.88:0/1549586697 >> v1:172.21.15.154:6789/0 conn(0x7f675015e6f0 legacy=0x7f67500b6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:56.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.127+0000 7f674effd640 1 -- 172.21.15.88:0/1549586697 >> v1:172.21.15.88:6789/0 conn(0x7f67500b7450 legacy=0x7f6750070830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:56.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.127+0000 7f674effd640 1 -- 172.21.15.88:0/1549586697 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f675015ef50 con 0x7f67500b54d0 2024-09-16T01:35:56.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.128+0000 7f674effd640 1 --2- 172.21.15.88:0/1549586697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67500b54d0 0x7f675006fd00 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f67440097a0 tx=0x7f674402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac58afecb3ab9dd6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:56.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.128+0000 7f674e7fc640 1 -- 172.21.15.88:0/1549586697 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6744037070 con 0x7f67500b54d0 2024-09-16T01:35:56.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.128+0000 7f674e7fc640 1 -- 172.21.15.88:0/1549586697 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6744032440 con 0x7f67500b54d0 2024-09-16T01:35:56.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.128+0000 7f674e7fc640 1 -- 172.21.15.88:0/1549586697 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6744032810 con 0x7f67500b54d0 2024-09-16T01:35:56.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.128+0000 7f67560af640 1 -- 172.21.15.88:0/1549586697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67500b54d0 msgr2=0x7f675006fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:56.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.128+0000 7f67560af640 1 --2- 172.21.15.88:0/1549586697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67500b54d0 0x7f675006fd00 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f67440097a0 tx=0x7f674402a230 comp rx=0 tx=0).stop 2024-09-16T01:35:56.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.128+0000 7f67560af640 1 -- 172.21.15.88:0/1549586697 shutdown_connections 2024-09-16T01:35:56.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.128+0000 7f67560af640 1 --2- 172.21.15.88:0/1549586697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67500b54d0 0x7f675006fd00 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:56.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.128+0000 7f67560af640 1 -- 172.21.15.88:0/1549586697 >> 172.21.15.88:0/1549586697 conn(0x7f67500d2430 msgr2=0x7f67500d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:56.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.129+0000 7f67560af640 1 -- 172.21.15.88:0/1549586697 shutdown_connections 2024-09-16T01:35:56.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.129+0000 7f67560af640 1 -- 172.21.15.88:0/1549586697 wait complete. 2024-09-16T01:35:56.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.130+0000 7f67560af640 1 Processor -- start 2024-09-16T01:35:56.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.130+0000 7f67560af640 1 -- start start 2024-09-16T01:35:56.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.130+0000 7f67560af640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67500b54d0 0x7f6750151160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:56.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.130+0000 7f67560af640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67500b7450 0x7f6750158760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:56.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.130+0000 7f67560af640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f675015e6f0 0x7f6750158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:56.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.130+0000 7f67560af640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f675006e730 con 0x7f675015e6f0 2024-09-16T01:35:56.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.130+0000 7f67560af640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f675006e5b0 con 0x7f67500b7450 2024-09-16T01:35:56.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.130+0000 7f67560af640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f675015f750 con 0x7f67500b54d0 2024-09-16T01:35:56.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.131+0000 7f674ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f675015e6f0 0x7f6750158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:56.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.131+0000 7f674ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f675015e6f0 0x7f6750158ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40710/0 (socket says 172.21.15.88:40710) 2024-09-16T01:35:56.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.131+0000 7f674ffff640 1 -- 172.21.15.88:0/2106664360 learned_addr learned my addr 172.21.15.88:0/2106664360 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:56.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.131+0000 7f674effd640 1 --2- 172.21.15.88:0/2106664360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67500b7450 0x7f6750158760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:56.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.131+0000 7f674f7fe640 1 --2- 172.21.15.88:0/2106664360 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67500b54d0 0x7f6750151160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:56.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.131+0000 7f674ffff640 1 -- 172.21.15.88:0/2106664360 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67500b54d0 msgr2=0x7f6750151160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:56.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.131+0000 7f674ffff640 1 --2- 172.21.15.88:0/2106664360 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67500b54d0 0x7f6750151160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:56.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.131+0000 7f674ffff640 1 -- 172.21.15.88:0/2106664360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67500b7450 msgr2=0x7f6750158760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:56.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.131+0000 7f674ffff640 1 --2- 172.21.15.88:0/2106664360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67500b7450 0x7f6750158760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:56.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.131+0000 7f674ffff640 1 -- 172.21.15.88:0/2106664360 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f675015b6e0 con 0x7f675015e6f0 2024-09-16T01:35:56.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.131+0000 7f674ffff640 1 --2- 172.21.15.88:0/2106664360 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f675015e6f0 0x7f6750158ca0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f674800cbc0 tx=0x7f674800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:56.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.131+0000 7f6737fff640 1 -- 172.21.15.88:0/2106664360 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6748017070 con 0x7f675015e6f0 2024-09-16T01:35:56.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.131+0000 7f6737fff640 1 -- 172.21.15.88:0/2106664360 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f674800ad40 con 0x7f675015e6f0 2024-09-16T01:35:56.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.131+0000 7f67560af640 1 -- 172.21.15.88:0/2106664360 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67501516a0 con 0x7f675015e6f0 2024-09-16T01:35:56.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.131+0000 7f6737fff640 1 -- 172.21.15.88:0/2106664360 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6748012600 con 0x7f675015e6f0 2024-09-16T01:35:56.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.131+0000 7f67560af640 1 -- 172.21.15.88:0/2106664360 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67501e8e50 con 0x7f675015e6f0 2024-09-16T01:35:56.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.132+0000 7f67560af640 1 -- 172.21.15.88:0/2106664360 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f675006c6c0 con 0x7f675015e6f0 2024-09-16T01:35:56.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.133+0000 7f6737fff640 1 -- 172.21.15.88:0/2106664360 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f674801a070 con 0x7f675015e6f0 2024-09-16T01:35:56.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.133+0000 7f6737fff640 1 --2- 172.21.15.88:0/2106664360 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f673c06c2e0 0x7f673c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:56.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.133+0000 7f6737fff640 1 -- 172.21.15.88:0/2106664360 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5881+0+0 (secure 0 0 0) 0x7f674808b0b0 con 0x7f675015e6f0 2024-09-16T01:35:56.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.133+0000 7f674f7fe640 1 --2- 172.21.15.88:0/2106664360 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f673c06c2e0 0x7f673c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:56.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.133+0000 7f674f7fe640 1 --2- 172.21.15.88:0/2106664360 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f673c06c2e0 0x7f673c06e7a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f67501f7bd0 tx=0x7f6740009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:56.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.137+0000 7f6737fff640 1 -- 172.21.15.88:0/2106664360 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f67480591b0 con 0x7f675015e6f0 2024-09-16T01:35:56.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:56.336+0000 7f67560af640 1 -- 172.21.15.88:0/2106664360 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f675015fae0 con 0x7f675015e6f0 2024-09-16T01:35:57.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.026+0000 7f6737fff640 1 -- 172.21.15.88:0/2106664360 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v21) ==== 104+0+0 (secure 0 0 0) 0x7f674805d4b0 con 0x7f675015e6f0 2024-09-16T01:35:57.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.029+0000 7f67560af640 1 -- 172.21.15.88:0/2106664360 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f673c06c2e0 msgr2=0x7f673c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:57.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.029+0000 7f67560af640 1 --2- 172.21.15.88:0/2106664360 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f673c06c2e0 0x7f673c06e7a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f67501f7bd0 tx=0x7f6740009040 comp rx=0 tx=0).stop 2024-09-16T01:35:57.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.029+0000 7f67560af640 1 -- 172.21.15.88:0/2106664360 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f675015e6f0 msgr2=0x7f6750158ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:57.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.029+0000 7f67560af640 1 --2- 172.21.15.88:0/2106664360 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f675015e6f0 0x7f6750158ca0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f674800cbc0 tx=0x7f674800a430 comp rx=0 tx=0).stop 2024-09-16T01:35:57.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.029+0000 7f67560af640 1 -- 172.21.15.88:0/2106664360 shutdown_connections 2024-09-16T01:35:57.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.029+0000 7f67560af640 1 --2- 172.21.15.88:0/2106664360 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f673c06c2e0 0x7f673c06e7a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:57.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.029+0000 7f67560af640 1 --2- 172.21.15.88:0/2106664360 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f675015e6f0 0x7f6750158ca0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:57.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.029+0000 7f67560af640 1 --2- 172.21.15.88:0/2106664360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67500b7450 0x7f6750158760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:57.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.029+0000 7f67560af640 1 --2- 172.21.15.88:0/2106664360 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67500b54d0 0x7f6750151160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:57.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.029+0000 7f67560af640 1 -- 172.21.15.88:0/2106664360 >> 172.21.15.88:0/2106664360 conn(0x7f67500d2430 msgr2=0x7f67500d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:57.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.030+0000 7f67560af640 1 -- 172.21.15.88:0/2106664360 shutdown_connections 2024-09-16T01:35:57.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.030+0000 7f67560af640 1 -- 172.21.15.88:0/2106664360 wait complete. 2024-09-16T01:35:57.045 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-16T01:35:57.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.190+0000 7f6c35ca6640 1 Processor -- start 2024-09-16T01:35:57.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.190+0000 7f6c35ca6640 1 -- start start 2024-09-16T01:35:57.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.191+0000 7f6c35ca6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6c3006f710 0x7f6c3006fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:57.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.191+0000 7f6c35ca6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c30074120 con 0x7f6c3006bc10 2024-09-16T01:35:57.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.191+0000 7f6c35ca6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c3015ef00 con 0x7f6c3015c2a0 2024-09-16T01:35:57.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.191+0000 7f6c35ca6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6c3015c680 con 0x7f6c3006f710 2024-09-16T01:35:57.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.191+0000 7f6c2effd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6c3006f710 0x7f6c3006fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:57.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.191+0000 7f6c2ffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6c3006bc10 0x7f6c3006c080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:48866/0 (socket says 172.21.15.88:48866) 2024-09-16T01:35:57.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.191+0000 7f6c2ffff640 1 -- 172.21.15.88:0/1650617166 learned_addr learned my addr 172.21.15.88:0/1650617166 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:35:57.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.192+0000 7f6c2effd640 1 -- 172.21.15.88:0/1650617166 >> v1:172.21.15.154:6789/0 conn(0x7f6c3015c2a0 legacy=0x7f6c300735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:57.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.192+0000 7f6c2effd640 1 -- 172.21.15.88:0/1650617166 >> v1:172.21.15.88:6789/0 conn(0x7f6c3006bc10 legacy=0x7f6c3006c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:57.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.192+0000 7f6c2effd640 1 -- 172.21.15.88:0/1650617166 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c3015f1e0 con 0x7f6c3006f710 2024-09-16T01:35:57.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.192+0000 7f6c2e7fc640 1 -- 172.21.15.88:0/1650617166 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1787421056 0 0) 0x7f6c30074120 con 0x7f6c3006bc10 2024-09-16T01:35:57.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.192+0000 7f6c2effd640 1 --2- 172.21.15.88:0/1650617166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c3006f710 0x7f6c3006fba0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6c240097a0 tx=0x7f6c2402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81906ed35c78526f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:57.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.193+0000 7f6c2e7fc640 1 -- 172.21.15.88:0/1650617166 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c24037070 con 0x7f6c3006f710 2024-09-16T01:35:57.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.193+0000 7f6c2e7fc640 1 -- 172.21.15.88:0/1650617166 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c24032440 con 0x7f6c3006f710 2024-09-16T01:35:57.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.193+0000 7f6c2e7fc640 1 -- 172.21.15.88:0/1650617166 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c24032810 con 0x7f6c3006f710 2024-09-16T01:35:57.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.193+0000 7f6c35ca6640 1 -- 172.21.15.88:0/1650617166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c3006f710 msgr2=0x7f6c3006fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:57.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.193+0000 7f6c35ca6640 1 --2- 172.21.15.88:0/1650617166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c3006f710 0x7f6c3006fba0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6c240097a0 tx=0x7f6c2402a1e0 comp rx=0 tx=0).stop 2024-09-16T01:35:57.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.193+0000 7f6c35ca6640 1 -- 172.21.15.88:0/1650617166 shutdown_connections 2024-09-16T01:35:57.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.193+0000 7f6c35ca6640 1 --2- 172.21.15.88:0/1650617166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c3006f710 0x7f6c3006fba0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:57.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.193+0000 7f6c35ca6640 1 -- 172.21.15.88:0/1650617166 >> 172.21.15.88:0/1650617166 conn(0x7f6c300b3070 msgr2=0x7f6c300b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:57.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.193+0000 7f6c35ca6640 1 -- 172.21.15.88:0/1650617166 shutdown_connections 2024-09-16T01:35:57.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.193+0000 7f6c35ca6640 1 -- 172.21.15.88:0/1650617166 wait complete. 2024-09-16T01:35:57.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.194+0000 7f6c35ca6640 1 Processor -- start 2024-09-16T01:35:57.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.195+0000 7f6c35ca6640 1 -- start start 2024-09-16T01:35:57.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.195+0000 7f6c35ca6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c3006bc10 0x7f6c3015e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:57.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.195+0000 7f6c35ca6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c3006f710 0x7f6c301584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:57.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.195+0000 7f6c35ca6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3015c2a0 0x7f6c301f7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:57.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.195+0000 7f6c35ca6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6c300b5f20 con 0x7f6c3006bc10 2024-09-16T01:35:57.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.195+0000 7f6c35ca6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3015ebf0 con 0x7f6c3015c2a0 2024-09-16T01:35:57.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.195+0000 7f6c35ca6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6c300b60a0 con 0x7f6c3006f710 2024-09-16T01:35:57.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.195+0000 7f6c2f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c3006bc10 0x7f6c3015e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:57.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.195+0000 7f6c2f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c3006bc10 0x7f6c3015e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40728/0 (socket says 172.21.15.88:40728) 2024-09-16T01:35:57.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.195+0000 7f6c2f7fe640 1 -- 172.21.15.88:0/5262865 learned_addr learned my addr 172.21.15.88:0/5262865 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:57.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.196+0000 7f6c2f7fe640 1 -- 172.21.15.88:0/5262865 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c3006f710 msgr2=0x7f6c301584a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:57.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.196+0000 7f6c2f7fe640 1 --2- 172.21.15.88:0/5262865 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c3006f710 0x7f6c301584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:57.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.196+0000 7f6c2f7fe640 1 -- 172.21.15.88:0/5262865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3015c2a0 msgr2=0x7f6c301f7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:57.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.196+0000 7f6c2f7fe640 1 --2- 172.21.15.88:0/5262865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3015c2a0 0x7f6c301f7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:57.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.196+0000 7f6c2f7fe640 1 -- 172.21.15.88:0/5262865 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c30160850 con 0x7f6c3006bc10 2024-09-16T01:35:57.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.196+0000 7f6c2f7fe640 1 --2- 172.21.15.88:0/5262865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c3006bc10 0x7f6c3015e630 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6c2000b3f0 tx=0x7f6c2000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:57.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.196+0000 7f6c17fff640 1 -- 172.21.15.88:0/5262865 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c20015020 con 0x7f6c3006bc10 2024-09-16T01:35:57.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.196+0000 7f6c17fff640 1 -- 172.21.15.88:0/5262865 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c20011830 con 0x7f6c3006bc10 2024-09-16T01:35:57.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.196+0000 7f6c35ca6640 1 -- 172.21.15.88:0/5262865 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c30160b30 con 0x7f6c3006bc10 2024-09-16T01:35:57.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.196+0000 7f6c17fff640 1 -- 172.21.15.88:0/5262865 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c20011c20 con 0x7f6c3006bc10 2024-09-16T01:35:57.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.196+0000 7f6c35ca6640 1 -- 172.21.15.88:0/5262865 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c30161c40 con 0x7f6c3006bc10 2024-09-16T01:35:57.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.197+0000 7f6c35ca6640 1 -- 172.21.15.88:0/5262865 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c3015f9e0 con 0x7f6c3006bc10 2024-09-16T01:35:57.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.197+0000 7f6c17fff640 1 -- 172.21.15.88:0/5262865 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6c20018020 con 0x7f6c3006bc10 2024-09-16T01:35:57.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.198+0000 7f6c17fff640 1 --2- 172.21.15.88:0/5262865 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6c1c06c2b0 0x7f6c1c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:57.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.198+0000 7f6c2effd640 1 --2- 172.21.15.88:0/5262865 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6c1c06c2b0 0x7f6c1c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:57.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.198+0000 7f6c17fff640 1 -- 172.21.15.88:0/5262865 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5881+0+0 (secure 0 0 0) 0x7f6c2008b1a0 con 0x7f6c3006bc10 2024-09-16T01:35:57.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.198+0000 7f6c2effd640 1 --2- 172.21.15.88:0/5262865 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6c1c06c2b0 0x7f6c1c06e770 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6c24034000 tx=0x7f6c2402c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:57.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.202+0000 7f6c17fff640 1 -- 172.21.15.88:0/5262865 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6c2001f030 con 0x7f6c3006bc10 2024-09-16T01:35:57.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:57.373+0000 7f6c35ca6640 1 -- 172.21.15.88:0/5262865 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6c300d2250 con 0x7f6c3006bc10 2024-09-16T01:35:58.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.027+0000 7f6c17fff640 1 -- 172.21.15.88:0/5262865 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v35) ==== 181+0+0 (secure 0 0 0) 0x7f6c200172c0 con 0x7f6c3006bc10 2024-09-16T01:35:58.029 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_metadata' removed 2024-09-16T01:35:58.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.031+0000 7f6c35ca6640 1 -- 172.21.15.88:0/5262865 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6c1c06c2b0 msgr2=0x7f6c1c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:58.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.031+0000 7f6c35ca6640 1 --2- 172.21.15.88:0/5262865 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6c1c06c2b0 0x7f6c1c06e770 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6c24034000 tx=0x7f6c2402c000 comp rx=0 tx=0).stop 2024-09-16T01:35:58.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.031+0000 7f6c35ca6640 1 -- 172.21.15.88:0/5262865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c3006bc10 msgr2=0x7f6c3015e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:58.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.031+0000 7f6c35ca6640 1 --2- 172.21.15.88:0/5262865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c3006bc10 0x7f6c3015e630 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6c2000b3f0 tx=0x7f6c2000b8b0 comp rx=0 tx=0).stop 2024-09-16T01:35:58.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.031+0000 7f6c35ca6640 1 -- 172.21.15.88:0/5262865 shutdown_connections 2024-09-16T01:35:58.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.031+0000 7f6c35ca6640 1 --2- 172.21.15.88:0/5262865 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6c1c06c2b0 0x7f6c1c06e770 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:58.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.031+0000 7f6c35ca6640 1 --2- 172.21.15.88:0/5262865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3015c2a0 0x7f6c301f7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:58.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.031+0000 7f6c35ca6640 1 --2- 172.21.15.88:0/5262865 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c3006f710 0x7f6c301584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:58.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.031+0000 7f6c35ca6640 1 --2- 172.21.15.88:0/5262865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c3006bc10 0x7f6c3015e630 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:58.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.031+0000 7f6c35ca6640 1 -- 172.21.15.88:0/5262865 >> 172.21.15.88:0/5262865 conn(0x7f6c300b3070 msgr2=0x7f6c300af6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:58.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.032+0000 7f6c35ca6640 1 -- 172.21.15.88:0/5262865 shutdown_connections 2024-09-16T01:35:58.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.032+0000 7f6c35ca6640 1 -- 172.21.15.88:0/5262865 wait complete. 2024-09-16T01:35:58.047 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-16T01:35:58.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.150+0000 7fdc53359640 1 Processor -- start 2024-09-16T01:35:58.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.151+0000 7fdc53359640 1 -- start start 2024-09-16T01:35:58.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.151+0000 7fdc53359640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdc4c15e570 0x7fdc4c0b5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:58.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.151+0000 7fdc53359640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdc4c0714a0 0x7fdc4c15e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:58.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.151+0000 7fdc53359640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc4c06f400 0x7fdc4c06f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:58.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.151+0000 7fdc53359640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fdc4c06fdb0 con 0x7fdc4c15e570 2024-09-16T01:35:58.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.151+0000 7fdc53359640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fdc4c15fc30 con 0x7fdc4c0714a0 2024-09-16T01:35:58.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.151+0000 7fdc53359640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdc4c15f020 con 0x7fdc4c06f400 2024-09-16T01:35:58.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.152+0000 7fdc510f5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdc4c15e570 0x7fdc4c0b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:58.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.152+0000 7fdc510f5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdc4c15e570 0x7fdc4c0b5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40730/0 (socket says 172.21.15.88:40730) 2024-09-16T01:35:58.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.152+0000 7fdc510f5640 1 -- 172.21.15.88:0/1644602615 learned_addr learned my addr 172.21.15.88:0/1644602615 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:58.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.152+0000 7fdc508f4640 1 --2- 172.21.15.88:0/1644602615 >> v2:172.21.15.154:3300/0 conn(0x7fdc4c0714a0 0x7fdc4c15e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:58.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.152+0000 7fdc518f6640 1 --2- 172.21.15.88:0/1644602615 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc4c06f400 0x7fdc4c06f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:58.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.152+0000 7fdc510f5640 1 -- 172.21.15.88:0/1644602615 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc4c06f400 msgr2=0x7fdc4c06f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:58.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.152+0000 7fdc510f5640 1 --2- 172.21.15.88:0/1644602615 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc4c06f400 0x7fdc4c06f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:58.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.152+0000 7fdc510f5640 1 -- 172.21.15.88:0/1644602615 >> v2:172.21.15.154:3300/0 conn(0x7fdc4c0714a0 msgr2=0x7fdc4c15e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:58.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.152+0000 7fdc510f5640 1 --2- 172.21.15.88:0/1644602615 >> v2:172.21.15.154:3300/0 conn(0x7fdc4c0714a0 0x7fdc4c15e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:58.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.152+0000 7fdc510f5640 1 -- 172.21.15.88:0/1644602615 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc4c15f300 con 0x7fdc4c15e570 2024-09-16T01:35:58.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.152+0000 7fdc510f5640 1 --2- 172.21.15.88:0/1644602615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc4c15e570 0x7fdc4c0b5540 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c0b6b30 tx=0x7fdc4802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23dda2124800b74a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:58.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.153+0000 7fdc43fff640 1 -- 172.21.15.88:0/1644602615 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc48037070 con 0x7fdc4c15e570 2024-09-16T01:35:58.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.153+0000 7fdc43fff640 1 -- 172.21.15.88:0/1644602615 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc48032440 con 0x7fdc4c15e570 2024-09-16T01:35:58.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.153+0000 7fdc43fff640 1 -- 172.21.15.88:0/1644602615 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc48032810 con 0x7fdc4c15e570 2024-09-16T01:35:58.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.153+0000 7fdc53359640 1 -- 172.21.15.88:0/1644602615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc4c15e570 msgr2=0x7fdc4c0b5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:58.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.153+0000 7fdc53359640 1 --2- 172.21.15.88:0/1644602615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc4c15e570 0x7fdc4c0b5540 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c0b6b30 tx=0x7fdc4802a140 comp rx=0 tx=0).stop 2024-09-16T01:35:58.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.153+0000 7fdc53359640 1 -- 172.21.15.88:0/1644602615 shutdown_connections 2024-09-16T01:35:58.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.153+0000 7fdc53359640 1 --2- 172.21.15.88:0/1644602615 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc4c06f400 0x7fdc4c06f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:58.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.153+0000 7fdc53359640 1 --2- 172.21.15.88:0/1644602615 >> v2:172.21.15.154:3300/0 conn(0x7fdc4c0714a0 0x7fdc4c15e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:58.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.153+0000 7fdc53359640 1 --2- 172.21.15.88:0/1644602615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc4c15e570 0x7fdc4c0b5540 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:58.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.153+0000 7fdc53359640 1 -- 172.21.15.88:0/1644602615 >> 172.21.15.88:0/1644602615 conn(0x7fdc4c0b4240 msgr2=0x7fdc4c0b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:58.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.153+0000 7fdc53359640 1 -- 172.21.15.88:0/1644602615 shutdown_connections 2024-09-16T01:35:58.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.153+0000 7fdc53359640 1 -- 172.21.15.88:0/1644602615 wait complete. 2024-09-16T01:35:58.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.154+0000 7fdc53359640 1 Processor -- start 2024-09-16T01:35:58.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.155+0000 7fdc53359640 1 -- start start 2024-09-16T01:35:58.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.155+0000 7fdc53359640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc4c06f400 0x7fdc4c0cfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:58.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.155+0000 7fdc53359640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc4c0714a0 0x7fdc4c1f4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:58.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.155+0000 7fdc53359640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc4c15e570 0x7fdc4c150040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:58.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.155+0000 7fdc53359640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdc4c073160 con 0x7fdc4c0714a0 2024-09-16T01:35:58.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.155+0000 7fdc53359640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdc4c15f8c0 con 0x7fdc4c06f400 2024-09-16T01:35:58.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.155+0000 7fdc53359640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdc4c0732e0 con 0x7fdc4c15e570 2024-09-16T01:35:58.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.155+0000 7fdc508f4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc4c0714a0 0x7fdc4c1f4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:58.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.155+0000 7fdc508f4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc4c0714a0 0x7fdc4c1f4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40740/0 (socket says 172.21.15.88:40740) 2024-09-16T01:35:58.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.155+0000 7fdc508f4640 1 -- 172.21.15.88:0/598414674 learned_addr learned my addr 172.21.15.88:0/598414674 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:58.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.155+0000 7fdc510f5640 1 --2- 172.21.15.88:0/598414674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc4c06f400 0x7fdc4c0cfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:58.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.155+0000 7fdc518f6640 1 --2- 172.21.15.88:0/598414674 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc4c15e570 0x7fdc4c150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:58.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.156+0000 7fdc508f4640 1 -- 172.21.15.88:0/598414674 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc4c15e570 msgr2=0x7fdc4c150040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:58.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.156+0000 7fdc508f4640 1 --2- 172.21.15.88:0/598414674 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc4c15e570 0x7fdc4c150040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:58.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.156+0000 7fdc508f4640 1 -- 172.21.15.88:0/598414674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc4c06f400 msgr2=0x7fdc4c0cfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:58.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.156+0000 7fdc508f4640 1 --2- 172.21.15.88:0/598414674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc4c06f400 0x7fdc4c0cfa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:58.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.156+0000 7fdc508f4640 1 -- 172.21.15.88:0/598414674 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc4c1f8760 con 0x7fdc4c0714a0 2024-09-16T01:35:58.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.156+0000 7fdc510f5640 1 --2- 172.21.15.88:0/598414674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc4c06f400 0x7fdc4c0cfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:35:58.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.156+0000 7fdc508f4640 1 --2- 172.21.15.88:0/598414674 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc4c0714a0 0x7fdc4c1f4cb0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c00d830 tx=0x7fdc3c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:58.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.156+0000 7fdc41ffb640 1 -- 172.21.15.88:0/598414674 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c00b6c0 con 0x7fdc4c0714a0 2024-09-16T01:35:58.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.156+0000 7fdc41ffb640 1 -- 172.21.15.88:0/598414674 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc3c019070 con 0x7fdc4c0714a0 2024-09-16T01:35:58.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.156+0000 7fdc53359640 1 -- 172.21.15.88:0/598414674 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc4c1f7b50 con 0x7fdc4c0714a0 2024-09-16T01:35:58.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.156+0000 7fdc41ffb640 1 -- 172.21.15.88:0/598414674 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c00ba80 con 0x7fdc4c0714a0 2024-09-16T01:35:58.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.156+0000 7fdc53359640 1 -- 172.21.15.88:0/598414674 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc4c1f8110 con 0x7fdc4c0714a0 2024-09-16T01:35:58.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.157+0000 7fdc53359640 1 -- 172.21.15.88:0/598414674 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc4c0708a0 con 0x7fdc4c0714a0 2024-09-16T01:35:58.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.158+0000 7fdc41ffb640 1 -- 172.21.15.88:0/598414674 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fdc3c01e460 con 0x7fdc4c0714a0 2024-09-16T01:35:58.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.158+0000 7fdc41ffb640 1 --2- 172.21.15.88:0/598414674 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdc3006c2b0 0x7fdc3006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:58.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.158+0000 7fdc510f5640 1 --2- 172.21.15.88:0/598414674 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdc3006c2b0 0x7fdc3006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:58.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.158+0000 7fdc41ffb640 1 -- 172.21.15.88:0/598414674 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5443+0+0 (secure 0 0 0) 0x7fdc3c08b950 con 0x7fdc4c0714a0 2024-09-16T01:35:58.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.159+0000 7fdc510f5640 1 --2- 172.21.15.88:0/598414674 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdc3006c2b0 0x7fdc3006e770 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdc48034000 tx=0x7fdc4802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:58.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.163+0000 7fdc41ffb640 1 -- 172.21.15.88:0/598414674 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fdc4c0708a0 con 0x7fdc4c0714a0 2024-09-16T01:35:58.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:58.328+0000 7fdc53359640 1 -- 172.21.15.88:0/598414674 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fdc4c06f870 con 0x7fdc4c0714a0 2024-09-16T01:35:59.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.038+0000 7fdc41ffb640 1 -- 172.21.15.88:0/598414674 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v36) ==== 169+0+0 (secure 0 0 0) 0x7fdc3c05a1c0 con 0x7fdc4c0714a0 2024-09-16T01:35:59.039 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_data' removed 2024-09-16T01:35:59.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.040+0000 7fdc53359640 1 -- 172.21.15.88:0/598414674 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdc3006c2b0 msgr2=0x7fdc3006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:59.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.040+0000 7fdc53359640 1 --2- 172.21.15.88:0/598414674 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdc3006c2b0 0x7fdc3006e770 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdc48034000 tx=0x7fdc4802e000 comp rx=0 tx=0).stop 2024-09-16T01:35:59.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.040+0000 7fdc53359640 1 -- 172.21.15.88:0/598414674 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc4c0714a0 msgr2=0x7fdc4c1f4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:59.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.040+0000 7fdc53359640 1 --2- 172.21.15.88:0/598414674 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc4c0714a0 0x7fdc4c1f4cb0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c00d830 tx=0x7fdc3c00dcf0 comp rx=0 tx=0).stop 2024-09-16T01:35:59.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.040+0000 7fdc53359640 1 -- 172.21.15.88:0/598414674 shutdown_connections 2024-09-16T01:35:59.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.040+0000 7fdc53359640 1 --2- 172.21.15.88:0/598414674 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdc3006c2b0 0x7fdc3006e770 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:59.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.040+0000 7fdc53359640 1 --2- 172.21.15.88:0/598414674 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdc4c15e570 0x7fdc4c150040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:59.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.040+0000 7fdc53359640 1 --2- 172.21.15.88:0/598414674 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdc4c0714a0 0x7fdc4c1f4cb0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:59.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.040+0000 7fdc53359640 1 --2- 172.21.15.88:0/598414674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdc4c06f400 0x7fdc4c0cfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:59.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.040+0000 7fdc53359640 1 -- 172.21.15.88:0/598414674 >> 172.21.15.88:0/598414674 conn(0x7fdc4c0b4240 msgr2=0x7fdc4c0d2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:59.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.040+0000 7fdc53359640 1 -- 172.21.15.88:0/598414674 shutdown_connections 2024-09-16T01:35:59.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.040+0000 7fdc53359640 1 -- 172.21.15.88:0/598414674 wait complete. 2024-09-16T01:35:59.055 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-16T01:35:59.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.154+0000 7f61130f1640 1 Processor -- start 2024-09-16T01:35:59.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.154+0000 7f61130f1640 1 -- start start 2024-09-16T01:35:59.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.155+0000 7f61130f1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f610c15e6f0 0x7f610c06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:59.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.155+0000 7f61130f1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f610c0b54d0 0x7f610c15f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:59.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.155+0000 7f61130f1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f610c06e990 con 0x7f610c0b6d40 2024-09-16T01:35:59.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.155+0000 7f61130f1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f610c06ec70 con 0x7f610c0b54d0 2024-09-16T01:35:59.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.155+0000 7f61130f1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f610c15ec70 con 0x7f610c15e6f0 2024-09-16T01:35:59.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.155+0000 7f6103fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f610c0b54d0 0x7f610c15f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:59.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.155+0000 7f6103fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f610c0b54d0 0x7f610c15f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40756/0 (socket says 172.21.15.88:40756) 2024-09-16T01:35:59.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.155+0000 7f6103fff640 1 -- 172.21.15.88:0/2093075245 learned_addr learned my addr 172.21.15.88:0/2093075245 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:59.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.155+0000 7f6110e8d640 1 --2- 172.21.15.88:0/2093075245 >> v2:172.21.15.154:3300/0 conn(0x7f610c15e6f0 0x7f610c06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:59.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.155+0000 7f6103fff640 1 -- 172.21.15.88:0/2093075245 >> v2:172.21.15.154:3300/0 conn(0x7f610c15e6f0 msgr2=0x7f610c06fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:59.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.155+0000 7f6103fff640 1 --2- 172.21.15.88:0/2093075245 >> v2:172.21.15.154:3300/0 conn(0x7f610c15e6f0 0x7f610c06fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:59.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.155+0000 7f6103fff640 1 -- 172.21.15.88:0/2093075245 >> v1:172.21.15.154:6789/0 conn(0x7f610c0b6d40 legacy=0x7f610c0b71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:59.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.155+0000 7f6103fff640 1 -- 172.21.15.88:0/2093075245 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f610c15ef50 con 0x7f610c0b54d0 2024-09-16T01:35:59.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.156+0000 7f6103fff640 1 --2- 172.21.15.88:0/2093075245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f610c0b54d0 0x7f610c15f750 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f610c069920 tx=0x7f60fc02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44a35e71f08eeff5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:59.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.156+0000 7f61037fe640 1 -- 172.21.15.88:0/2093075245 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60fc037070 con 0x7f610c0b54d0 2024-09-16T01:35:59.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.156+0000 7f61037fe640 1 -- 172.21.15.88:0/2093075245 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60fc032440 con 0x7f610c0b54d0 2024-09-16T01:35:59.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.156+0000 7f61037fe640 1 -- 172.21.15.88:0/2093075245 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60fc032810 con 0x7f610c0b54d0 2024-09-16T01:35:59.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.157+0000 7f61130f1640 1 -- 172.21.15.88:0/2093075245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f610c0b54d0 msgr2=0x7f610c15f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:59.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.157+0000 7f61130f1640 1 --2- 172.21.15.88:0/2093075245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f610c0b54d0 0x7f610c15f750 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f610c069920 tx=0x7f60fc02a230 comp rx=0 tx=0).stop 2024-09-16T01:35:59.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.157+0000 7f61130f1640 1 -- 172.21.15.88:0/2093075245 shutdown_connections 2024-09-16T01:35:59.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.157+0000 7f61130f1640 1 --2- 172.21.15.88:0/2093075245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f610c0b54d0 0x7f610c15f750 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:59.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.157+0000 7f61130f1640 1 --2- 172.21.15.88:0/2093075245 >> v2:172.21.15.154:3300/0 conn(0x7f610c15e6f0 0x7f610c06fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:59.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.157+0000 7f61130f1640 1 -- 172.21.15.88:0/2093075245 >> 172.21.15.88:0/2093075245 conn(0x7f610c0d2430 msgr2=0x7f610c0d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:59.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.157+0000 7f61130f1640 1 -- 172.21.15.88:0/2093075245 shutdown_connections 2024-09-16T01:35:59.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.157+0000 7f61130f1640 1 -- 172.21.15.88:0/2093075245 wait complete. 2024-09-16T01:35:59.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.158+0000 7f61130f1640 1 Processor -- start 2024-09-16T01:35:59.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.158+0000 7f61130f1640 1 -- start start 2024-09-16T01:35:59.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.159+0000 7f61130f1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f610c0b54d0 0x7f610c15b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:59.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.159+0000 7f61130f1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f610c0b6d40 0x7f610c148d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:59.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.159+0000 7f61130f1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f610c15e6f0 0x7f610c1ea160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:59.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.159+0000 7f61130f1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f610c06e730 con 0x7f610c0b54d0 2024-09-16T01:35:59.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.159+0000 7f61130f1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f610c06e5b0 con 0x7f610c15e6f0 2024-09-16T01:35:59.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.159+0000 7f61130f1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f610c070830 con 0x7f610c0b6d40 2024-09-16T01:35:59.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.159+0000 7f6110e8d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f610c0b54d0 0x7f610c15b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:59.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.159+0000 7f6110e8d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f610c0b54d0 0x7f610c15b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40760/0 (socket says 172.21.15.88:40760) 2024-09-16T01:35:59.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.159+0000 7f6110e8d640 1 -- 172.21.15.88:0/2932308329 learned_addr learned my addr 172.21.15.88:0/2932308329 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:59.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.159+0000 7f6103fff640 1 --2- 172.21.15.88:0/2932308329 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f610c0b6d40 0x7f610c148d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:59.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.159+0000 7f6110e8d640 1 -- 172.21.15.88:0/2932308329 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f610c0b6d40 msgr2=0x7f610c148d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:59.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.159+0000 7f6110e8d640 1 --2- 172.21.15.88:0/2932308329 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f610c0b6d40 0x7f610c148d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:59.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.159+0000 7f611168e640 1 --2- 172.21.15.88:0/2932308329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f610c15e6f0 0x7f610c1ea160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:59.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.159+0000 7f6110e8d640 1 -- 172.21.15.88:0/2932308329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f610c15e6f0 msgr2=0x7f610c1ea160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:59.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.159+0000 7f6110e8d640 1 --2- 172.21.15.88:0/2932308329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f610c15e6f0 0x7f610c1ea160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:59.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.159+0000 7f6110e8d640 1 -- 172.21.15.88:0/2932308329 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f610c1596e0 con 0x7f610c0b54d0 2024-09-16T01:35:59.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.159+0000 7f6110e8d640 1 --2- 172.21.15.88:0/2932308329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f610c0b54d0 0x7f610c15b130 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f610800d5f0 tx=0x7f610800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:59.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.160+0000 7f61017fa640 1 -- 172.21.15.88:0/2932308329 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61080125f0 con 0x7f610c0b54d0 2024-09-16T01:35:59.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.160+0000 7f61017fa640 1 -- 172.21.15.88:0/2932308329 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6108019070 con 0x7f610c0b54d0 2024-09-16T01:35:59.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.160+0000 7f61017fa640 1 -- 172.21.15.88:0/2932308329 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61080129d0 con 0x7f610c0b54d0 2024-09-16T01:35:59.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.160+0000 7f61130f1640 1 -- 172.21.15.88:0/2932308329 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f610c15a370 con 0x7f610c0b54d0 2024-09-16T01:35:59.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.160+0000 7f61130f1640 1 -- 172.21.15.88:0/2932308329 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f610c15ac10 con 0x7f610c0b54d0 2024-09-16T01:35:59.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.161+0000 7f61130f1640 1 -- 172.21.15.88:0/2932308329 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f610c15a540 con 0x7f610c0b54d0 2024-09-16T01:35:59.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.161+0000 7f61017fa640 1 -- 172.21.15.88:0/2932308329 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6108012b70 con 0x7f610c0b54d0 2024-09-16T01:35:59.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.161+0000 7f61017fa640 1 --2- 172.21.15.88:0/2932308329 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f60f806c2e0 0x7f60f806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:59.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.161+0000 7f61017fa640 1 -- 172.21.15.88:0/2932308329 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5029+0+0 (secure 0 0 0) 0x7f610808b5f0 con 0x7f610c0b54d0 2024-09-16T01:35:59.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.161+0000 7f6103fff640 1 --2- 172.21.15.88:0/2932308329 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f60f806c2e0 0x7f60f806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:59.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.162+0000 7f6103fff640 1 --2- 172.21.15.88:0/2932308329 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f60f806c2e0 0x7f60f806e7a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f60fc034000 tx=0x7f60fc02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:59.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.166+0000 7f61017fa640 1 -- 172.21.15.88:0/2932308329 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6108059f70 con 0x7f610c0b54d0 2024-09-16T01:35:59.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.334+0000 7f61130f1640 1 -- 172.21.15.88:0/2932308329 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f610c06c6c0 con 0x7f610c0b54d0 2024-09-16T01:35:59.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.337+0000 7f61017fa640 1 -- 172.21.15.88:0/2932308329 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7f6108059870 con 0x7f610c0b54d0 2024-09-16T01:35:59.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.339+0000 7f61130f1640 1 -- 172.21.15.88:0/2932308329 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f60f806c2e0 msgr2=0x7f60f806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:59.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.339+0000 7f61130f1640 1 --2- 172.21.15.88:0/2932308329 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f60f806c2e0 0x7f60f806e7a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f60fc034000 tx=0x7f60fc02e000 comp rx=0 tx=0).stop 2024-09-16T01:35:59.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.339+0000 7f61130f1640 1 -- 172.21.15.88:0/2932308329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f610c0b54d0 msgr2=0x7f610c15b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:59.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.339+0000 7f61130f1640 1 --2- 172.21.15.88:0/2932308329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f610c0b54d0 0x7f610c15b130 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f610800d5f0 tx=0x7f610800dab0 comp rx=0 tx=0).stop 2024-09-16T01:35:59.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.339+0000 7f61130f1640 1 -- 172.21.15.88:0/2932308329 shutdown_connections 2024-09-16T01:35:59.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.339+0000 7f61130f1640 1 --2- 172.21.15.88:0/2932308329 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f60f806c2e0 0x7f60f806e7a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:59.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.339+0000 7f61130f1640 1 --2- 172.21.15.88:0/2932308329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f610c15e6f0 0x7f610c1ea160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:59.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.339+0000 7f61130f1640 1 --2- 172.21.15.88:0/2932308329 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f610c0b6d40 0x7f610c148d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:59.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.339+0000 7f61130f1640 1 --2- 172.21.15.88:0/2932308329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f610c0b54d0 0x7f610c15b130 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:59.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.339+0000 7f61130f1640 1 -- 172.21.15.88:0/2932308329 >> 172.21.15.88:0/2932308329 conn(0x7f610c0d2430 msgr2=0x7f610c157640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:59.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.339+0000 7f61130f1640 1 -- 172.21.15.88:0/2932308329 shutdown_connections 2024-09-16T01:35:59.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.339+0000 7f61130f1640 1 -- 172.21.15.88:0/2932308329 wait complete. 2024-09-16T01:35:59.351 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_damage.TestDamage.test_damaged_dentry' 2024-09-16T01:35:59.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.449+0000 7f2a86c5e640 1 Processor -- start 2024-09-16T01:35:59.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.449+0000 7f2a86c5e640 1 -- start start 2024-09-16T01:35:59.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.450+0000 7f2a86c5e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a8015a9c0 0x7f2a8015e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:59.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.450+0000 7f2a86c5e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2a800b29a0 0x7f2a800cde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:59.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.450+0000 7f2a86c5e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a800b6830 con 0x7f2a800b5be0 2024-09-16T01:35:59.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.450+0000 7f2a86c5e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2a8015f410 con 0x7f2a800b29a0 2024-09-16T01:35:59.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.450+0000 7f2a86c5e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2a800cd0b0 con 0x7f2a8015a9c0 2024-09-16T01:35:59.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.450+0000 7f2a851fb640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2a800b29a0 0x7f2a800cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:59.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.450+0000 7f2a851fb640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2a800b29a0 0x7f2a800cde70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40776/0 (socket says 172.21.15.88:40776) 2024-09-16T01:35:59.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.450+0000 7f2a851fb640 1 -- 172.21.15.88:0/3225154897 learned_addr learned my addr 172.21.15.88:0/3225154897 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:59.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.450+0000 7f2a849fa640 1 --2- 172.21.15.88:0/3225154897 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a8015a9c0 0x7f2a8015e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:59.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.450+0000 7f2a851fb640 1 -- 172.21.15.88:0/3225154897 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a8015a9c0 msgr2=0x7f2a8015e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:59.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.450+0000 7f2a851fb640 1 --2- 172.21.15.88:0/3225154897 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a8015a9c0 0x7f2a8015e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:59.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.450+0000 7f2a851fb640 1 -- 172.21.15.88:0/3225154897 >> v1:172.21.15.154:6789/0 conn(0x7f2a800b5be0 legacy=0x7f2a8015eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:35:59.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.450+0000 7f2a851fb640 1 -- 172.21.15.88:0/3225154897 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a800cd390 con 0x7f2a800b29a0 2024-09-16T01:35:59.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.451+0000 7f2a851fb640 1 --2- 172.21.15.88:0/3225154897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a800b29a0 0x7f2a800cde70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2a78009e50 tx=0x7f2a7802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4aad266030ec02d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:59.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.451+0000 7f2a7f7fe640 1 -- 172.21.15.88:0/3225154897 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a78036070 con 0x7f2a800b29a0 2024-09-16T01:35:59.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.451+0000 7f2a7f7fe640 1 -- 172.21.15.88:0/3225154897 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a78031600 con 0x7f2a800b29a0 2024-09-16T01:35:59.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.451+0000 7f2a7f7fe640 1 -- 172.21.15.88:0/3225154897 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a780319d0 con 0x7f2a800b29a0 2024-09-16T01:35:59.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.452+0000 7f2a86c5e640 1 -- 172.21.15.88:0/3225154897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a800b29a0 msgr2=0x7f2a800cde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:59.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.452+0000 7f2a86c5e640 1 --2- 172.21.15.88:0/3225154897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a800b29a0 0x7f2a800cde70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2a78009e50 tx=0x7f2a7802a6c0 comp rx=0 tx=0).stop 2024-09-16T01:35:59.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.452+0000 7f2a86c5e640 1 -- 172.21.15.88:0/3225154897 shutdown_connections 2024-09-16T01:35:59.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.452+0000 7f2a86c5e640 1 --2- 172.21.15.88:0/3225154897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a800b29a0 0x7f2a800cde70 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:59.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.452+0000 7f2a86c5e640 1 --2- 172.21.15.88:0/3225154897 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a8015a9c0 0x7f2a8015e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:59.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.452+0000 7f2a86c5e640 1 -- 172.21.15.88:0/3225154897 >> 172.21.15.88:0/3225154897 conn(0x7f2a8006bc10 msgr2=0x7f2a8006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:35:59.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.452+0000 7f2a86c5e640 1 -- 172.21.15.88:0/3225154897 shutdown_connections 2024-09-16T01:35:59.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.452+0000 7f2a86c5e640 1 -- 172.21.15.88:0/3225154897 wait complete. 2024-09-16T01:35:59.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.453+0000 7f2a86c5e640 1 Processor -- start 2024-09-16T01:35:59.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.453+0000 7f2a86c5e640 1 -- start start 2024-09-16T01:35:59.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.454+0000 7f2a86c5e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a800b29a0 0x7f2a80155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:59.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.454+0000 7f2a86c5e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a800b5be0 0x7f2a801501f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:59.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.454+0000 7f2a86c5e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a8015a9c0 0x7f2a80150730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:59.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.454+0000 7f2a86c5e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8015ada0 con 0x7f2a8015a9c0 2024-09-16T01:35:59.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.454+0000 7f2a86c5e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2a800cd950 con 0x7f2a800b29a0 2024-09-16T01:35:59.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.454+0000 7f2a86c5e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2a8006de40 con 0x7f2a800b5be0 2024-09-16T01:35:59.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.454+0000 7f2a851fb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a8015a9c0 0x7f2a80150730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:59.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.454+0000 7f2a851fb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a8015a9c0 0x7f2a80150730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40790/0 (socket says 172.21.15.88:40790) 2024-09-16T01:35:59.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.454+0000 7f2a851fb640 1 -- 172.21.15.88:0/1642643881 learned_addr learned my addr 172.21.15.88:0/1642643881 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:35:59.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.454+0000 7f2a851fb640 1 -- 172.21.15.88:0/1642643881 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a800b5be0 msgr2=0x7f2a801501f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:59.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.454+0000 7f2a851fb640 1 --2- 172.21.15.88:0/1642643881 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a800b5be0 0x7f2a801501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:59.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.454+0000 7f2a851fb640 1 -- 172.21.15.88:0/1642643881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a800b29a0 msgr2=0x7f2a80155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:35:59.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.454+0000 7f2a851fb640 1 --2- 172.21.15.88:0/1642643881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a800b29a0 0x7f2a80155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:35:59.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.454+0000 7f2a851fb640 1 -- 172.21.15.88:0/1642643881 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a80152a20 con 0x7f2a8015a9c0 2024-09-16T01:35:59.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.455+0000 7f2a851fb640 1 --2- 172.21.15.88:0/1642643881 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a8015a9c0 0x7f2a80150730 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2a78009f70 tx=0x7f2a78004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:59.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.455+0000 7f2a7d7fa640 1 -- 172.21.15.88:0/1642643881 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7803f020 con 0x7f2a8015a9c0 2024-09-16T01:35:59.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.455+0000 7f2a7d7fa640 1 -- 172.21.15.88:0/1642643881 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a78037040 con 0x7f2a8015a9c0 2024-09-16T01:35:59.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.455+0000 7f2a86c5e640 1 -- 172.21.15.88:0/1642643881 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a80152d00 con 0x7f2a8015a9c0 2024-09-16T01:35:59.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.455+0000 7f2a7d7fa640 1 -- 172.21.15.88:0/1642643881 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a78036070 con 0x7f2a8015a9c0 2024-09-16T01:35:59.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.455+0000 7f2a86c5e640 1 -- 172.21.15.88:0/1642643881 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a801535a0 con 0x7f2a8015a9c0 2024-09-16T01:35:59.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.456+0000 7f2a86c5e640 1 -- 172.21.15.88:0/1642643881 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a8006e1f0 con 0x7f2a8015a9c0 2024-09-16T01:35:59.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.456+0000 7f2a7d7fa640 1 -- 172.21.15.88:0/1642643881 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2a78042020 con 0x7f2a8015a9c0 2024-09-16T01:35:59.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.457+0000 7f2a7d7fa640 1 --2- 172.21.15.88:0/1642643881 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2a6c06c2b0 0x7f2a6c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:35:59.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.457+0000 7f2a849fa640 1 --2- 172.21.15.88:0/1642643881 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2a6c06c2b0 0x7f2a6c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:35:59.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.457+0000 7f2a7d7fa640 1 -- 172.21.15.88:0/1642643881 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5029+0+0 (secure 0 0 0) 0x7f2a780ab6b0 con 0x7f2a8015a9c0 2024-09-16T01:35:59.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.457+0000 7f2a849fa640 1 --2- 172.21.15.88:0/1642643881 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2a6c06c2b0 0x7f2a6c06e770 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2a7000a810 tx=0x7f2a70008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:35:59.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.461+0000 7f2a7d7fa640 1 -- 172.21.15.88:0/1642643881 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2a78049030 con 0x7f2a8015a9c0 2024-09-16T01:35:59.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:35:59.690+0000 7f2a86c5e640 1 -- 172.21.15.88:0/1642643881 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_damaged_dentry"]} v 0) -- 0x7f2a800b4810 con 0x7f2a8015a9c0 2024-09-16T01:36:00.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.039+0000 7f2a7d7fa640 1 -- 172.21.15.88:0/1642643881 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_damaged_dentry"]}]=0 v105) ==== 134+0+0 (secure 0 0 0) 0x7f2a7807a0c0 con 0x7f2a8015a9c0 2024-09-16T01:36:00.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.042+0000 7f2a86c5e640 1 -- 172.21.15.88:0/1642643881 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2a6c06c2b0 msgr2=0x7f2a6c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:00.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.042+0000 7f2a86c5e640 1 --2- 172.21.15.88:0/1642643881 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2a6c06c2b0 0x7f2a6c06e770 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2a7000a810 tx=0x7f2a70008040 comp rx=0 tx=0).stop 2024-09-16T01:36:00.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.042+0000 7f2a86c5e640 1 -- 172.21.15.88:0/1642643881 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a8015a9c0 msgr2=0x7f2a80150730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:00.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.042+0000 7f2a86c5e640 1 --2- 172.21.15.88:0/1642643881 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a8015a9c0 0x7f2a80150730 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2a78009f70 tx=0x7f2a78004a40 comp rx=0 tx=0).stop 2024-09-16T01:36:00.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.042+0000 7f2a86c5e640 1 -- 172.21.15.88:0/1642643881 shutdown_connections 2024-09-16T01:36:00.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.042+0000 7f2a86c5e640 1 --2- 172.21.15.88:0/1642643881 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2a6c06c2b0 0x7f2a6c06e770 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.042+0000 7f2a86c5e640 1 --2- 172.21.15.88:0/1642643881 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a8015a9c0 0x7f2a80150730 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.042+0000 7f2a86c5e640 1 --2- 172.21.15.88:0/1642643881 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a800b5be0 0x7f2a801501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.042+0000 7f2a86c5e640 1 --2- 172.21.15.88:0/1642643881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a800b29a0 0x7f2a80155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.042+0000 7f2a86c5e640 1 -- 172.21.15.88:0/1642643881 >> 172.21.15.88:0/1642643881 conn(0x7f2a8006bc10 msgr2=0x7f2a8006bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:00.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.043+0000 7f2a86c5e640 1 -- 172.21.15.88:0/1642643881 shutdown_connections 2024-09-16T01:36:00.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.043+0000 7f2a86c5e640 1 -- 172.21.15.88:0/1642643881 wait complete. 2024-09-16T01:36:00.055 INFO:tasks.cephfs_test_runner:test_damaged_dentry (tasks.cephfs.test_damage.TestDamage) ... ok 2024-09-16T01:36:00.056 INFO:tasks.cephfs_test_runner:Starting test: test_dentry_first_existing (tasks.cephfs.test_damage.TestDamage) That the MDS won't abort when the dentry is already known to be damaged. 2024-09-16T01:36:00.056 INFO:tasks.cephfs_test_runner:test_dentry_first_existing (tasks.cephfs.test_damage.TestDamage) 2024-09-16T01:36:00.056 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:36:00.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.154+0000 7ff4ad8a6640 1 Processor -- start 2024-09-16T01:36:00.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.154+0000 7ff4ad8a6640 1 -- start start 2024-09-16T01:36:00.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.155+0000 7ff4ad8a6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff4a815e660 0x7ff4a80cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:00.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.155+0000 7ff4ad8a6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4a815ea40 con 0x7ff4a80d1820 2024-09-16T01:36:00.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.155+0000 7ff4ad8a6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4a80d3870 con 0x7ff4a80d3310 2024-09-16T01:36:00.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.155+0000 7ff4ad8a6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff4a80d3b50 con 0x7ff4a815e660 2024-09-16T01:36:00.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.155+0000 7ff4a6ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff4a815e660 0x7ff4a80cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:00.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.155+0000 7ff4a77fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff4a80d1820 0x7ff4a80b7b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:48868/0 (socket says 172.21.15.88:48868) 2024-09-16T01:36:00.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.155+0000 7ff4a77fe640 1 -- 172.21.15.88:0/3996239562 learned_addr learned my addr 172.21.15.88:0/3996239562 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:00.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.156+0000 7ff4a6ffd640 1 -- 172.21.15.88:0/3996239562 >> v1:172.21.15.154:6789/0 conn(0x7ff4a80d3310 legacy=0x7ff4a80d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:00.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.156+0000 7ff4a6ffd640 1 -- 172.21.15.88:0/3996239562 >> v1:172.21.15.88:6789/0 conn(0x7ff4a80d1820 legacy=0x7ff4a80b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:00.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.156+0000 7ff4a67fc640 1 -- 172.21.15.88:0/3996239562 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1178290629 0 0) 0x7ff4a815ea40 con 0x7ff4a80d1820 2024-09-16T01:36:00.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.156+0000 7ff4a6ffd640 1 -- 172.21.15.88:0/3996239562 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a80d3e30 con 0x7ff4a815e660 2024-09-16T01:36:00.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.156+0000 7ff4a6ffd640 1 --2- 172.21.15.88:0/3996239562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4a815e660 0x7ff4a80cf730 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff4980097a0 tx=0x7ff49802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d789f09ee404588a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:00.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.156+0000 7ff4a67fc640 1 -- 172.21.15.88:0/3996239562 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff498037070 con 0x7ff4a815e660 2024-09-16T01:36:00.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.157+0000 7ff4a67fc640 1 -- 172.21.15.88:0/3996239562 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff498032440 con 0x7ff4a815e660 2024-09-16T01:36:00.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.157+0000 7ff4a67fc640 1 -- 172.21.15.88:0/3996239562 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff498032810 con 0x7ff4a815e660 2024-09-16T01:36:00.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.157+0000 7ff4ad8a6640 1 -- 172.21.15.88:0/3996239562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4a815e660 msgr2=0x7ff4a80cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:00.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.157+0000 7ff4ad8a6640 1 --2- 172.21.15.88:0/3996239562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4a815e660 0x7ff4a80cf730 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff4980097a0 tx=0x7ff49802a1e0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.157+0000 7ff4ad8a6640 1 -- 172.21.15.88:0/3996239562 shutdown_connections 2024-09-16T01:36:00.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.157+0000 7ff4ad8a6640 1 --2- 172.21.15.88:0/3996239562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4a815e660 0x7ff4a80cf730 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.157+0000 7ff4ad8a6640 1 -- 172.21.15.88:0/3996239562 >> 172.21.15.88:0/3996239562 conn(0x7ff4a80b6410 msgr2=0x7ff4a80b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:00.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.157+0000 7ff4ad8a6640 1 -- 172.21.15.88:0/3996239562 shutdown_connections 2024-09-16T01:36:00.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.157+0000 7ff4ad8a6640 1 -- 172.21.15.88:0/3996239562 wait complete. 2024-09-16T01:36:00.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.158+0000 7ff4ad8a6640 1 Processor -- start 2024-09-16T01:36:00.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.158+0000 7ff4ad8a6640 1 -- start start 2024-09-16T01:36:00.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.159+0000 7ff4ad8a6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4a80d1820 0x7ff4a81f7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:00.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.159+0000 7ff4ad8a6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4a80d3310 0x7ff4a8155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:00.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.159+0000 7ff4ad8a6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4a815e660 0x7ff4a8157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:00.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.159+0000 7ff4ad8a6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a80cfc70 con 0x7ff4a815e660 2024-09-16T01:36:00.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.159+0000 7ff4ad8a6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a80d43f0 con 0x7ff4a80d3310 2024-09-16T01:36:00.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.159+0000 7ff4ad8a6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff4a80b8280 con 0x7ff4a80d1820 2024-09-16T01:36:00.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.159+0000 7ff4a77fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4a815e660 0x7ff4a8157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:00.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.159+0000 7ff4a77fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4a815e660 0x7ff4a8157870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40814/0 (socket says 172.21.15.88:40814) 2024-09-16T01:36:00.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.159+0000 7ff4a77fe640 1 -- 172.21.15.88:0/3846416326 learned_addr learned my addr 172.21.15.88:0/3846416326 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:00.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.159+0000 7ff4a6ffd640 1 --2- 172.21.15.88:0/3846416326 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4a80d1820 0x7ff4a81f7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:00.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.160+0000 7ff4a77fe640 1 -- 172.21.15.88:0/3846416326 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4a80d1820 msgr2=0x7ff4a81f7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:00.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.160+0000 7ff4a77fe640 1 --2- 172.21.15.88:0/3846416326 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4a80d1820 0x7ff4a81f7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.160+0000 7ff4a77fe640 1 -- 172.21.15.88:0/3846416326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4a80d3310 msgr2=0x7ff4a8155bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:00.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.160+0000 7ff49e7fc640 1 --2- 172.21.15.88:0/3846416326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4a80d3310 0x7ff4a8155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:00.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.160+0000 7ff4a77fe640 1 --2- 172.21.15.88:0/3846416326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4a80d3310 0x7ff4a8155bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.160+0000 7ff4a77fe640 1 -- 172.21.15.88:0/3846416326 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a81584d0 con 0x7ff4a815e660 2024-09-16T01:36:00.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.160+0000 7ff4a77fe640 1 --2- 172.21.15.88:0/3846416326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4a815e660 0x7ff4a8157870 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff4a000c7a0 tx=0x7ff4a000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:00.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.160+0000 7ff4a4ff9640 1 -- 172.21.15.88:0/3846416326 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a0018070 con 0x7ff4a815e660 2024-09-16T01:36:00.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.160+0000 7ff4a4ff9640 1 -- 172.21.15.88:0/3846416326 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4a000a640 con 0x7ff4a815e660 2024-09-16T01:36:00.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.160+0000 7ff4ad8a6640 1 -- 172.21.15.88:0/3846416326 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4a8158770 con 0x7ff4a815e660 2024-09-16T01:36:00.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.160+0000 7ff4a4ff9640 1 -- 172.21.15.88:0/3846416326 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a000aa70 con 0x7ff4a815e660 2024-09-16T01:36:00.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.160+0000 7ff4ad8a6640 1 -- 172.21.15.88:0/3846416326 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4a8158d30 con 0x7ff4a815e660 2024-09-16T01:36:00.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.161+0000 7ff4ad8a6640 1 -- 172.21.15.88:0/3846416326 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4a8158f00 con 0x7ff4a815e660 2024-09-16T01:36:00.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.161+0000 7ff4a4ff9640 1 -- 172.21.15.88:0/3846416326 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff4a001b070 con 0x7ff4a815e660 2024-09-16T01:36:00.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.162+0000 7ff4a4ff9640 1 --2- 172.21.15.88:0/3846416326 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff48806c2b0 0x7ff48806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:00.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.162+0000 7ff4a4ff9640 1 -- 172.21.15.88:0/3846416326 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5029+0+0 (secure 0 0 0) 0x7ff4a008a7f0 con 0x7ff4a815e660 2024-09-16T01:36:00.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.162+0000 7ff4a6ffd640 1 --2- 172.21.15.88:0/3846416326 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff48806c2b0 0x7ff48806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:00.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.162+0000 7ff4a6ffd640 1 --2- 172.21.15.88:0/3846416326 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff48806c2b0 0x7ff48806e770 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff498034000 tx=0x7ff49802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:00.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.166+0000 7ff4a4ff9640 1 -- 172.21.15.88:0/3846416326 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff4a0059170 con 0x7ff4a815e660 2024-09-16T01:36:00.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.333+0000 7ff4ad8a6640 1 -- 172.21.15.88:0/3846416326 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff4a80cf730 con 0x7ff4a815e660 2024-09-16T01:36:00.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.334+0000 7ff4a4ff9640 1 -- 172.21.15.88:0/3846416326 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) ==== 74+0+13138 (secure 0 0 0) 0x7ff4a0058a70 con 0x7ff4a815e660 2024-09-16T01:36:00.335 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:00.335 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":36,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:35:59.033837+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:6835/2870334334":"2024-09-17T01:35:56.009971+0000","172.21.15.88:6834/2870334334":"2024-09-17T01:35:56.009971+0000","172.21.15.88:6835/4230443772":"2024-09-17T01:35:12.937737+0000","172.21.15.88:6834/4230443772":"2024-09-17T01:35:12.937737+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:36:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.336+0000 7ff4ad8a6640 1 -- 172.21.15.88:0/3846416326 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff48806c2b0 msgr2=0x7ff48806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.336+0000 7ff4ad8a6640 1 --2- 172.21.15.88:0/3846416326 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff48806c2b0 0x7ff48806e770 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff498034000 tx=0x7ff49802e000 comp rx=0 tx=0).stop 2024-09-16T01:36:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.336+0000 7ff4ad8a6640 1 -- 172.21.15.88:0/3846416326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4a815e660 msgr2=0x7ff4a8157870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.336+0000 7ff4ad8a6640 1 --2- 172.21.15.88:0/3846416326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4a815e660 0x7ff4a8157870 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff4a000c7a0 tx=0x7ff4a000cc60 comp rx=0 tx=0).stop 2024-09-16T01:36:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.336+0000 7ff4ad8a6640 1 -- 172.21.15.88:0/3846416326 shutdown_connections 2024-09-16T01:36:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.336+0000 7ff4ad8a6640 1 --2- 172.21.15.88:0/3846416326 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff48806c2b0 0x7ff48806e770 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.336+0000 7ff4ad8a6640 1 --2- 172.21.15.88:0/3846416326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4a815e660 0x7ff4a8157870 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.336+0000 7ff4ad8a6640 1 --2- 172.21.15.88:0/3846416326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4a80d3310 0x7ff4a8155bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.336+0000 7ff4ad8a6640 1 --2- 172.21.15.88:0/3846416326 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4a80d1820 0x7ff4a81f7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.336+0000 7ff4ad8a6640 1 -- 172.21.15.88:0/3846416326 >> 172.21.15.88:0/3846416326 conn(0x7ff4a80b6410 msgr2=0x7ff4a80b5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.336+0000 7ff4ad8a6640 1 -- 172.21.15.88:0/3846416326 shutdown_connections 2024-09-16T01:36:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.336+0000 7ff4ad8a6640 1 -- 172.21.15.88:0/3846416326 wait complete. 2024-09-16T01:36:00.349 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T01:34:10.547178+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T01:36:00.349 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T01:36:00.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.449+0000 7f5fd2b86640 1 Processor -- start 2024-09-16T01:36:00.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.449+0000 7f5fd2b86640 1 -- start start 2024-09-16T01:36:00.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.450+0000 7f5fd2b86640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5fcc0d1c20 0x7f5fcc0cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:00.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.450+0000 7f5fd2b86640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fcc0d4530 con 0x7f5fcc0cf590 2024-09-16T01:36:00.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.450+0000 7f5fd2b86640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fcc0d3490 con 0x7f5fcc0d1130 2024-09-16T01:36:00.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.450+0000 7f5fd2b86640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f5fcc0d40a0 con 0x7f5fcc0d1c20 2024-09-16T01:36:00.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.450+0000 7f5fcbfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5fcc0d1c20 0x7f5fcc0cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:00.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.450+0000 7f5fd1123640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5fcc0cf590 0x7f5fcc0cfa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59302/0 (socket says 172.21.15.88:59302) 2024-09-16T01:36:00.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.450+0000 7f5fd1123640 1 -- 172.21.15.88:0/892483096 learned_addr learned my addr 172.21.15.88:0/892483096 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:00.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.451+0000 7f5fcb7fe640 1 -- 172.21.15.88:0/892483096 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4078560661 0 0) 0x7f5fcc0d4530 con 0x7f5fcc0cf590 2024-09-16T01:36:00.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.451+0000 7f5fcb7fe640 1 -- 172.21.15.88:0/892483096 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fb4003520 con 0x7f5fcc0cf590 2024-09-16T01:36:00.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.451+0000 7f5fcbfff640 1 -- 172.21.15.88:0/892483096 >> v1:172.21.15.154:6789/0 conn(0x7f5fcc0d1130 legacy=0x7f5fcc0d1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:00.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.451+0000 7f5fcbfff640 1 -- 172.21.15.88:0/892483096 >> v1:172.21.15.88:6789/0 conn(0x7f5fcc0cf590 legacy=0x7f5fcc0cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:00.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.451+0000 7f5fcbfff640 1 -- 172.21.15.88:0/892483096 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fcc0d3770 con 0x7f5fcc0d1c20 2024-09-16T01:36:00.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.451+0000 7f5fcbfff640 1 --2- 172.21.15.88:0/892483096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fcc0d1c20 0x7f5fcc0cf050 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f5fc00097a0 tx=0x7f5fc002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9063749ca19f66f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:00.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.452+0000 7f5fcb7fe640 1 -- 172.21.15.88:0/892483096 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc0037070 con 0x7f5fcc0d1c20 2024-09-16T01:36:00.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.452+0000 7f5fcb7fe640 1 -- 172.21.15.88:0/892483096 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fc0032440 con 0x7f5fcc0d1c20 2024-09-16T01:36:00.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.452+0000 7f5fcb7fe640 1 -- 172.21.15.88:0/892483096 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc0032810 con 0x7f5fcc0d1c20 2024-09-16T01:36:00.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.452+0000 7f5fd2b86640 1 -- 172.21.15.88:0/892483096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fcc0d1c20 msgr2=0x7f5fcc0cf050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:00.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.452+0000 7f5fd2b86640 1 --2- 172.21.15.88:0/892483096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fcc0d1c20 0x7f5fcc0cf050 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f5fc00097a0 tx=0x7f5fc002a1e0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.452+0000 7f5fd2b86640 1 -- 172.21.15.88:0/892483096 shutdown_connections 2024-09-16T01:36:00.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.452+0000 7f5fd2b86640 1 --2- 172.21.15.88:0/892483096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fcc0d1c20 0x7f5fcc0cf050 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.452+0000 7f5fd2b86640 1 -- 172.21.15.88:0/892483096 >> 172.21.15.88:0/892483096 conn(0x7f5fcc06c020 msgr2=0x7f5fcc06c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:00.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.452+0000 7f5fd2b86640 1 -- 172.21.15.88:0/892483096 shutdown_connections 2024-09-16T01:36:00.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.452+0000 7f5fd2b86640 1 -- 172.21.15.88:0/892483096 wait complete. 2024-09-16T01:36:00.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.453+0000 7f5fd2b86640 1 Processor -- start 2024-09-16T01:36:00.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.453+0000 7f5fd2b86640 1 -- start start 2024-09-16T01:36:00.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.454+0000 7f5fd2b86640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5fcc0cf590 0x7f5fcc14a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:00.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.454+0000 7f5fd2b86640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fcc0d1130 0x7f5fcc15cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:00.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.454+0000 7f5fd2b86640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5fcc0d1c20 0x7f5fcc15d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:00.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.454+0000 7f5fd2b86640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5fcc0d0130 con 0x7f5fcc0d1130 2024-09-16T01:36:00.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.454+0000 7f5fd2b86640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5fcc0d3d30 con 0x7f5fcc0cf590 2024-09-16T01:36:00.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.454+0000 7f5fd2b86640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5fcc0d02b0 con 0x7f5fcc0d1c20 2024-09-16T01:36:00.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.454+0000 7f5fcbfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fcc0d1130 0x7f5fcc15cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:00.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.454+0000 7f5fcbfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fcc0d1130 0x7f5fcc15cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50806/0 (socket says 172.21.15.88:50806) 2024-09-16T01:36:00.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.454+0000 7f5fcbfff640 1 -- 172.21.15.88:0/3186504059 learned_addr learned my addr 172.21.15.88:0/3186504059 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:00.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.455+0000 7f5fcbfff640 1 -- 172.21.15.88:0/3186504059 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5fcc0d1c20 msgr2=0x7f5fcc15d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:00.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.455+0000 7f5fcbfff640 1 --2- 172.21.15.88:0/3186504059 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5fcc0d1c20 0x7f5fcc15d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.455+0000 7f5fcbfff640 1 -- 172.21.15.88:0/3186504059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5fcc0cf590 msgr2=0x7f5fcc14a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:00.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.455+0000 7f5fcbfff640 1 --2- 172.21.15.88:0/3186504059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5fcc0cf590 0x7f5fcc14a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.455+0000 7f5fcbfff640 1 -- 172.21.15.88:0/3186504059 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fcc14ab10 con 0x7f5fcc0d1130 2024-09-16T01:36:00.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.455+0000 7f5fcbfff640 1 --2- 172.21.15.88:0/3186504059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fcc0d1130 0x7f5fcc15cf50 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0000c00 tx=0x7f5fc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:00.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.455+0000 7f5fc97fa640 1 -- 172.21.15.88:0/3186504059 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc0037040 con 0x7f5fcc0d1130 2024-09-16T01:36:00.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.455+0000 7f5fc97fa640 1 -- 172.21.15.88:0/3186504059 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fc0042070 con 0x7f5fcc0d1130 2024-09-16T01:36:00.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.455+0000 7f5fd2b86640 1 -- 172.21.15.88:0/3186504059 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fcc14adf0 con 0x7f5fcc0d1130 2024-09-16T01:36:00.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.455+0000 7f5fc97fa640 1 -- 172.21.15.88:0/3186504059 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc0030940 con 0x7f5fcc0d1130 2024-09-16T01:36:00.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.455+0000 7f5fd2b86640 1 -- 172.21.15.88:0/3186504059 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fcc14b3b0 con 0x7f5fcc0d1130 2024-09-16T01:36:00.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.456+0000 7f5fd2b86640 1 -- 172.21.15.88:0/3186504059 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fcc0d3070 con 0x7f5fcc0d1130 2024-09-16T01:36:00.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.456+0000 7f5fc97fa640 1 -- 172.21.15.88:0/3186504059 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5fc0030ae0 con 0x7f5fcc0d1130 2024-09-16T01:36:00.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.457+0000 7f5fc97fa640 1 --2- 172.21.15.88:0/3186504059 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5fb806c2e0 0x7f5fb806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:00.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.457+0000 7f5fc97fa640 1 -- 172.21.15.88:0/3186504059 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5029+0+0 (secure 0 0 0) 0x7f5fc00aba10 con 0x7f5fcc0d1130 2024-09-16T01:36:00.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.457+0000 7f5fd0922640 1 --2- 172.21.15.88:0/3186504059 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5fb806c2e0 0x7f5fb806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:00.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.457+0000 7f5fd0922640 1 --2- 172.21.15.88:0/3186504059 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5fb806c2e0 0x7f5fb806e7a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc0043c0 tx=0x7f5fbc007d30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:00.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.461+0000 7f5fc97fa640 1 -- 172.21.15.88:0/3186504059 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5fc0045360 con 0x7f5fcc0d1130 2024-09-16T01:36:00.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.629+0000 7f5fd2b86640 1 -- 172.21.15.88:0/3186504059 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5fcc15bd90 con 0x7f5fcc0d1130 2024-09-16T01:36:00.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.630+0000 7f5fc97fa640 1 -- 172.21.15.88:0/3186504059 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) ==== 93+0+10 (secure 0 0 0) 0x7f5fc007a4b0 con 0x7f5fcc0d1130 2024-09-16T01:36:00.632 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2024-09-16T01:36:00.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.632+0000 7f5fd2b86640 1 -- 172.21.15.88:0/3186504059 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5fb806c2e0 msgr2=0x7f5fb806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:00.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.632+0000 7f5fd2b86640 1 --2- 172.21.15.88:0/3186504059 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5fb806c2e0 0x7f5fb806e7a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc0043c0 tx=0x7f5fbc007d30 comp rx=0 tx=0).stop 2024-09-16T01:36:00.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.632+0000 7f5fd2b86640 1 -- 172.21.15.88:0/3186504059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fcc0d1130 msgr2=0x7f5fcc15cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:00.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.632+0000 7f5fd2b86640 1 --2- 172.21.15.88:0/3186504059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fcc0d1130 0x7f5fcc15cf50 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0000c00 tx=0x7f5fc00046c0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.632+0000 7f5fd2b86640 1 -- 172.21.15.88:0/3186504059 shutdown_connections 2024-09-16T01:36:00.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.632+0000 7f5fd2b86640 1 --2- 172.21.15.88:0/3186504059 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5fb806c2e0 0x7f5fb806e7a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.632+0000 7f5fd2b86640 1 --2- 172.21.15.88:0/3186504059 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5fcc0d1c20 0x7f5fcc15d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.632+0000 7f5fd2b86640 1 --2- 172.21.15.88:0/3186504059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fcc0d1130 0x7f5fcc15cf50 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.632+0000 7f5fd2b86640 1 --2- 172.21.15.88:0/3186504059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5fcc0cf590 0x7f5fcc14a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.632+0000 7f5fd2b86640 1 -- 172.21.15.88:0/3186504059 >> 172.21.15.88:0/3186504059 conn(0x7f5fcc06c020 msgr2=0x7f5fcc0b5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:00.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.632+0000 7f5fd2b86640 1 -- 172.21.15.88:0/3186504059 shutdown_connections 2024-09-16T01:36:00.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.632+0000 7f5fd2b86640 1 -- 172.21.15.88:0/3186504059 wait complete. 2024-09-16T01:36:00.644 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_existing' 2024-09-16T01:36:00.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.742+0000 7fc88d2e4640 1 Processor -- start 2024-09-16T01:36:00.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.742+0000 7fc88d2e4640 1 -- start start 2024-09-16T01:36:00.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.742+0000 7fc88d2e4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc88815e660 0x7fc8880cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:00.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.742+0000 7fc88d2e4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc8880d1820 0x7fc8880b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:00.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.742+0000 7fc88d2e4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc88815ea40 con 0x7fc8880d3310 2024-09-16T01:36:00.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.742+0000 7fc88d2e4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc8880d3870 con 0x7fc8880d1820 2024-09-16T01:36:00.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.742+0000 7fc88d2e4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc8880d3b50 con 0x7fc88815e660 2024-09-16T01:36:00.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.742+0000 7fc8877fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc8880d1820 0x7fc8880b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:00.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.743+0000 7fc8877fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc8880d1820 0x7fc8880b7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50812/0 (socket says 172.21.15.88:50812) 2024-09-16T01:36:00.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.743+0000 7fc8877fe640 1 -- 172.21.15.88:0/3286739034 learned_addr learned my addr 172.21.15.88:0/3286739034 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:00.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.743+0000 7fc886ffd640 1 --2- 172.21.15.88:0/3286739034 >> v2:172.21.15.154:3300/0 conn(0x7fc88815e660 0x7fc8880cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:00.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.743+0000 7fc8877fe640 1 -- 172.21.15.88:0/3286739034 >> v2:172.21.15.154:3300/0 conn(0x7fc88815e660 msgr2=0x7fc8880cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:00.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.743+0000 7fc8877fe640 1 --2- 172.21.15.88:0/3286739034 >> v2:172.21.15.154:3300/0 conn(0x7fc88815e660 0x7fc8880cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.743+0000 7fc8877fe640 1 -- 172.21.15.88:0/3286739034 >> v1:172.21.15.154:6789/0 conn(0x7fc8880d3310 legacy=0x7fc8880d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:00.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.743+0000 7fc8877fe640 1 -- 172.21.15.88:0/3286739034 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8880d3e30 con 0x7fc8880d1820 2024-09-16T01:36:00.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.743+0000 7fc8877fe640 1 --2- 172.21.15.88:0/3286739034 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8880d1820 0x7fc8880b7b70 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc8880d10e0 tx=0x7fc88002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=665eb8c40fef65e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:00.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.744+0000 7fc885ffb640 1 -- 172.21.15.88:0/3286739034 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc880037070 con 0x7fc8880d1820 2024-09-16T01:36:00.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.744+0000 7fc885ffb640 1 -- 172.21.15.88:0/3286739034 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc880032440 con 0x7fc8880d1820 2024-09-16T01:36:00.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.744+0000 7fc885ffb640 1 -- 172.21.15.88:0/3286739034 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc880032810 con 0x7fc8880d1820 2024-09-16T01:36:00.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.744+0000 7fc88d2e4640 1 -- 172.21.15.88:0/3286739034 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8880d1820 msgr2=0x7fc8880b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:00.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.744+0000 7fc88d2e4640 1 --2- 172.21.15.88:0/3286739034 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8880d1820 0x7fc8880b7b70 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc8880d10e0 tx=0x7fc88002a230 comp rx=0 tx=0).stop 2024-09-16T01:36:00.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.744+0000 7fc88d2e4640 1 -- 172.21.15.88:0/3286739034 shutdown_connections 2024-09-16T01:36:00.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.744+0000 7fc88d2e4640 1 --2- 172.21.15.88:0/3286739034 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8880d1820 0x7fc8880b7b70 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.744+0000 7fc88d2e4640 1 --2- 172.21.15.88:0/3286739034 >> v2:172.21.15.154:3300/0 conn(0x7fc88815e660 0x7fc8880cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.744+0000 7fc88d2e4640 1 -- 172.21.15.88:0/3286739034 >> 172.21.15.88:0/3286739034 conn(0x7fc8880b6410 msgr2=0x7fc8880b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:00.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.744+0000 7fc88d2e4640 1 -- 172.21.15.88:0/3286739034 shutdown_connections 2024-09-16T01:36:00.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.744+0000 7fc88d2e4640 1 -- 172.21.15.88:0/3286739034 wait complete. 2024-09-16T01:36:00.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.745+0000 7fc88d2e4640 1 Processor -- start 2024-09-16T01:36:00.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.746+0000 7fc88d2e4640 1 -- start start 2024-09-16T01:36:00.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.746+0000 7fc88d2e4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8880d1820 0x7fc888153a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:00.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.746+0000 7fc88d2e4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8880d3310 0x7fc8880cac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:00.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.746+0000 7fc88d2e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc88815e660 0x7fc88815ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:00.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.746+0000 7fc88d2e4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc8880cfc70 con 0x7fc8880d3310 2024-09-16T01:36:00.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.746+0000 7fc88d2e4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc8880d43f0 con 0x7fc8880d1820 2024-09-16T01:36:00.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.746+0000 7fc88d2e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc8880b80b0 con 0x7fc88815e660 2024-09-16T01:36:00.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.746+0000 7fc8867fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8880d3310 0x7fc8880cac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:00.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.746+0000 7fc8867fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8880d3310 0x7fc8880cac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50822/0 (socket says 172.21.15.88:50822) 2024-09-16T01:36:00.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.746+0000 7fc8867fc640 1 -- 172.21.15.88:0/1094252578 learned_addr learned my addr 172.21.15.88:0/1094252578 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:00.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.747+0000 7fc8867fc640 1 -- 172.21.15.88:0/1094252578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc88815e660 msgr2=0x7fc88815ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:00.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.747+0000 7fc8867fc640 1 --2- 172.21.15.88:0/1094252578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc88815e660 0x7fc88815ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.747+0000 7fc8867fc640 1 -- 172.21.15.88:0/1094252578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8880d1820 msgr2=0x7fc888153a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:00.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.747+0000 7fc8867fc640 1 --2- 172.21.15.88:0/1094252578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8880d1820 0x7fc888153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:00.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.747+0000 7fc8867fc640 1 -- 172.21.15.88:0/1094252578 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc888153f80 con 0x7fc8880d3310 2024-09-16T01:36:00.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.747+0000 7fc8867fc640 1 --2- 172.21.15.88:0/1094252578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8880d3310 0x7fc8880cac40 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fc87000c440 tx=0x7fc87000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:00.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.747+0000 7fc87f7fe640 1 -- 172.21.15.88:0/1094252578 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc870017070 con 0x7fc8880d3310 2024-09-16T01:36:00.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.747+0000 7fc87f7fe640 1 -- 172.21.15.88:0/1094252578 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8700035f0 con 0x7fc8880d3310 2024-09-16T01:36:00.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.747+0000 7fc87f7fe640 1 -- 172.21.15.88:0/1094252578 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8700039e0 con 0x7fc8880d3310 2024-09-16T01:36:00.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.747+0000 7fc88d2e4640 1 -- 172.21.15.88:0/1094252578 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8880c91b0 con 0x7fc8880d3310 2024-09-16T01:36:00.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.747+0000 7fc88d2e4640 1 -- 172.21.15.88:0/1094252578 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8880ba410 con 0x7fc8880d3310 2024-09-16T01:36:00.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.748+0000 7fc88d2e4640 1 -- 172.21.15.88:0/1094252578 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8880b7b70 con 0x7fc8880d3310 2024-09-16T01:36:00.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.748+0000 7fc87f7fe640 1 -- 172.21.15.88:0/1094252578 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc87001a070 con 0x7fc8880d3310 2024-09-16T01:36:00.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.749+0000 7fc87f7fe640 1 --2- 172.21.15.88:0/1094252578 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc86806c2b0 0x7fc86806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:00.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.749+0000 7fc87f7fe640 1 -- 172.21.15.88:0/1094252578 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5029+0+0 (secure 0 0 0) 0x7fc87008a530 con 0x7fc8880d3310 2024-09-16T01:36:00.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.749+0000 7fc886ffd640 1 --2- 172.21.15.88:0/1094252578 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc86806c2b0 0x7fc86806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:00.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.749+0000 7fc886ffd640 1 --2- 172.21.15.88:0/1094252578 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc86806c2b0 0x7fc86806e770 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc8881f77d0 tx=0x7fc8780072f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:00.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.753+0000 7fc87f7fe640 1 -- 172.21.15.88:0/1094252578 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc870058f40 con 0x7fc8880d3310 2024-09-16T01:36:00.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:00.981+0000 7fc88d2e4640 1 -- 172.21.15.88:0/1094252578 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_existing"]} v 0) -- 0x7fc888159860 con 0x7fc8880d3310 2024-09-16T01:36:01.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.043+0000 7fc87f7fe640 1 -- 172.21.15.88:0/1094252578 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_existing"]}]=0 v106) ==== 144+0+0 (secure 0 0 0) 0x7fc8700010c0 con 0x7fc8880d3310 2024-09-16T01:36:01.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.046+0000 7fc88d2e4640 1 -- 172.21.15.88:0/1094252578 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc86806c2b0 msgr2=0x7fc86806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:01.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.046+0000 7fc88d2e4640 1 --2- 172.21.15.88:0/1094252578 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc86806c2b0 0x7fc86806e770 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc8881f77d0 tx=0x7fc8780072f0 comp rx=0 tx=0).stop 2024-09-16T01:36:01.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.046+0000 7fc88d2e4640 1 -- 172.21.15.88:0/1094252578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8880d3310 msgr2=0x7fc8880cac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:01.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.046+0000 7fc88d2e4640 1 --2- 172.21.15.88:0/1094252578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8880d3310 0x7fc8880cac40 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fc87000c440 tx=0x7fc87000c900 comp rx=0 tx=0).stop 2024-09-16T01:36:01.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.046+0000 7fc88d2e4640 1 -- 172.21.15.88:0/1094252578 shutdown_connections 2024-09-16T01:36:01.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.046+0000 7fc88d2e4640 1 --2- 172.21.15.88:0/1094252578 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc86806c2b0 0x7fc86806e770 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:01.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.046+0000 7fc88d2e4640 1 --2- 172.21.15.88:0/1094252578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc88815e660 0x7fc88815ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:01.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.046+0000 7fc88d2e4640 1 --2- 172.21.15.88:0/1094252578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc8880d3310 0x7fc8880cac40 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:01.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.046+0000 7fc88d2e4640 1 --2- 172.21.15.88:0/1094252578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8880d1820 0x7fc888153a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:01.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.046+0000 7fc88d2e4640 1 -- 172.21.15.88:0/1094252578 >> 172.21.15.88:0/1094252578 conn(0x7fc8880b6410 msgr2=0x7fc8880b67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:01.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.046+0000 7fc88d2e4640 1 -- 172.21.15.88:0/1094252578 shutdown_connections 2024-09-16T01:36:01.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.046+0000 7fc88d2e4640 1 -- 172.21.15.88:0/1094252578 wait complete. 2024-09-16T01:36:01.058 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T01:36:01.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.157+0000 7f135c6e6640 1 Processor -- start 2024-09-16T01:36:01.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.157+0000 7f135c6e6640 1 -- start start 2024-09-16T01:36:01.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.158+0000 7f135c6e6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f13540b6290 0x7f1354071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:01.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.158+0000 7f135c6e6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1354072530 0x7f135415bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:01.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.158+0000 7f135c6e6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13540ccff0 con 0x7f135415b480 2024-09-16T01:36:01.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.158+0000 7f135c6e6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f13540cd2d0 con 0x7f13540b6290 2024-09-16T01:36:01.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.158+0000 7f135c6e6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f13540cd5b0 con 0x7f1354072530 2024-09-16T01:36:01.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.158+0000 7f1359c81640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f13540b6290 0x7f1354071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:01.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.158+0000 7f135a482640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f135415b480 0x7f13540b5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59318/0 (socket says 172.21.15.88:59318) 2024-09-16T01:36:01.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.158+0000 7f135a482640 1 -- 172.21.15.88:0/165849752 learned_addr learned my addr 172.21.15.88:0/165849752 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:01.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.158+0000 7f1359c81640 1 -- 172.21.15.88:0/165849752 >> v2:172.21.15.154:3300/0 conn(0x7f1354072530 msgr2=0x7f135415bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:01.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.158+0000 7f1359c81640 1 --2- 172.21.15.88:0/165849752 >> v2:172.21.15.154:3300/0 conn(0x7f1354072530 0x7f135415bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:01.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.158+0000 7f1359c81640 1 -- 172.21.15.88:0/165849752 >> v1:172.21.15.88:6789/0 conn(0x7f135415b480 legacy=0x7f13540b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:01.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.158+0000 7f1359480640 1 -- 172.21.15.88:0/165849752 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1564288925 0 0) 0x7f13540ccff0 con 0x7f135415b480 2024-09-16T01:36:01.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.158+0000 7f1359c81640 1 -- 172.21.15.88:0/165849752 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13540cd890 con 0x7f13540b6290 2024-09-16T01:36:01.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.159+0000 7f1359c81640 1 --2- 172.21.15.88:0/165849752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13540b6290 0x7f1354071ff0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f13540733f0 tx=0x7f134802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3a543547f3d7ffb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:01.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.159+0000 7f1359480640 1 -- 172.21.15.88:0/165849752 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1348037070 con 0x7f13540b6290 2024-09-16T01:36:01.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.159+0000 7f1359480640 1 -- 172.21.15.88:0/165849752 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1348032440 con 0x7f13540b6290 2024-09-16T01:36:01.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.159+0000 7f1359480640 1 -- 172.21.15.88:0/165849752 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134802ae60 con 0x7f13540b6290 2024-09-16T01:36:01.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.160+0000 7f135c6e6640 1 -- 172.21.15.88:0/165849752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13540b6290 msgr2=0x7f1354071ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:01.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.160+0000 7f135c6e6640 1 --2- 172.21.15.88:0/165849752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13540b6290 0x7f1354071ff0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f13540733f0 tx=0x7f134802a110 comp rx=0 tx=0).stop 2024-09-16T01:36:01.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.160+0000 7f135c6e6640 1 -- 172.21.15.88:0/165849752 shutdown_connections 2024-09-16T01:36:01.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.160+0000 7f135c6e6640 1 --2- 172.21.15.88:0/165849752 >> v2:172.21.15.154:3300/0 conn(0x7f1354072530 0x7f135415bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:01.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.160+0000 7f135c6e6640 1 --2- 172.21.15.88:0/165849752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13540b6290 0x7f1354071ff0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:01.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.160+0000 7f135c6e6640 1 -- 172.21.15.88:0/165849752 >> 172.21.15.88:0/165849752 conn(0x7f135406d0f0 msgr2=0x7f135406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:01.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.160+0000 7f135c6e6640 1 -- 172.21.15.88:0/165849752 shutdown_connections 2024-09-16T01:36:01.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.160+0000 7f135c6e6640 1 -- 172.21.15.88:0/165849752 wait complete. 2024-09-16T01:36:01.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.161+0000 7f135c6e6640 1 Processor -- start 2024-09-16T01:36:01.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.161+0000 7f135c6e6640 1 -- start start 2024-09-16T01:36:01.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.162+0000 7f135c6e6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1354072530 0x7f13540d0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:01.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.162+0000 7f135c6e6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f135415b480 0x7f13540ce9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:01.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.162+0000 7f135c6e6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1354136f20 0x7f135414e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:01.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.162+0000 7f135c6e6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f135415b9e0 con 0x7f1354072530 2024-09-16T01:36:01.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.162+0000 7f135c6e6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f135415b860 con 0x7f1354136f20 2024-09-16T01:36:01.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.162+0000 7f135c6e6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f135415c1a0 con 0x7f135415b480 2024-09-16T01:36:01.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.162+0000 7f135a482640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1354072530 0x7f13540d0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:01.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.162+0000 7f135a482640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1354072530 0x7f13540d0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50830/0 (socket says 172.21.15.88:50830) 2024-09-16T01:36:01.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.162+0000 7f135a482640 1 -- 172.21.15.88:0/1064031890 learned_addr learned my addr 172.21.15.88:0/1064031890 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:01.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.162+0000 7f1359c81640 1 --2- 172.21.15.88:0/1064031890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f135415b480 0x7f13540ce9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:01.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.162+0000 7f135a482640 1 -- 172.21.15.88:0/1064031890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f135415b480 msgr2=0x7f13540ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:01.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.162+0000 7f135a482640 1 --2- 172.21.15.88:0/1064031890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f135415b480 0x7f13540ce9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:01.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.162+0000 7f135ac83640 1 --2- 172.21.15.88:0/1064031890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1354136f20 0x7f135414e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:01.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.162+0000 7f135a482640 1 -- 172.21.15.88:0/1064031890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1354136f20 msgr2=0x7f135414e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:01.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.162+0000 7f135a482640 1 --2- 172.21.15.88:0/1064031890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1354136f20 0x7f135414e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:01.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.162+0000 7f135a482640 1 -- 172.21.15.88:0/1064031890 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13540cef10 con 0x7f1354072530 2024-09-16T01:36:01.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.162+0000 7f1359c81640 1 --2- 172.21.15.88:0/1064031890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f135415b480 0x7f13540ce9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:01.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.162+0000 7f135ac83640 1 --2- 172.21.15.88:0/1064031890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1354136f20 0x7f135414e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:01.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.162+0000 7f135a482640 1 --2- 172.21.15.88:0/1064031890 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1354072530 0x7f13540d0c00 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f134c00c460 tx=0x7f134c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:01.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.162+0000 7f1352ffd640 1 -- 172.21.15.88:0/1064031890 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134c018070 con 0x7f1354072530 2024-09-16T01:36:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.162+0000 7f1352ffd640 1 -- 172.21.15.88:0/1064031890 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f134c007500 con 0x7f1354072530 2024-09-16T01:36:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.162+0000 7f135c6e6640 1 -- 172.21.15.88:0/1064031890 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13540cf190 con 0x7f1354072530 2024-09-16T01:36:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.162+0000 7f1352ffd640 1 -- 172.21.15.88:0/1064031890 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134c004a20 con 0x7f1354072530 2024-09-16T01:36:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.163+0000 7f135c6e6640 1 -- 172.21.15.88:0/1064031890 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f135414dd20 con 0x7f1354072530 2024-09-16T01:36:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.163+0000 7f135c6e6640 1 -- 172.21.15.88:0/1064031890 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13540cf6b0 con 0x7f1354072530 2024-09-16T01:36:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.164+0000 7f1352ffd640 1 -- 172.21.15.88:0/1064031890 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f134c01b070 con 0x7f1354072530 2024-09-16T01:36:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.165+0000 7f1352ffd640 1 --2- 172.21.15.88:0/1064031890 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f134006c2e0 0x7f134006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.165+0000 7f1359c81640 1 --2- 172.21.15.88:0/1064031890 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f134006c2e0 0x7f134006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.165+0000 7f1352ffd640 1 -- 172.21.15.88:0/1064031890 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5029+0+0 (secure 0 0 0) 0x7f134c08a3d0 con 0x7f1354072530 2024-09-16T01:36:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.165+0000 7f1359c81640 1 --2- 172.21.15.88:0/1064031890 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f134006c2e0 0x7f134006e7a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1348034000 tx=0x7f134802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:01.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.169+0000 7f1352ffd640 1 -- 172.21.15.88:0/1064031890 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f13540cf6b0 con 0x7f1354072530 2024-09-16T01:36:01.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.335+0000 7f135c6e6640 1 -- 172.21.15.88:0/1064031890 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f135415bc60 con 0x7f1354072530 2024-09-16T01:36:01.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.338+0000 7f1352ffd640 1 -- 172.21.15.88:0/1064031890 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) ==== 122+0+0 (secure 0 0 0) 0x7f13540cef10 con 0x7f1354072530 2024-09-16T01:36:01.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.340+0000 7f135c6e6640 1 -- 172.21.15.88:0/1064031890 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f134006c2e0 msgr2=0x7f134006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:01.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.340+0000 7f135c6e6640 1 --2- 172.21.15.88:0/1064031890 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f134006c2e0 0x7f134006e7a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1348034000 tx=0x7f134802e000 comp rx=0 tx=0).stop 2024-09-16T01:36:01.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.340+0000 7f135c6e6640 1 -- 172.21.15.88:0/1064031890 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1354072530 msgr2=0x7f13540d0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:01.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.340+0000 7f135c6e6640 1 --2- 172.21.15.88:0/1064031890 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1354072530 0x7f13540d0c00 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f134c00c460 tx=0x7f134c00c920 comp rx=0 tx=0).stop 2024-09-16T01:36:01.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.340+0000 7f135c6e6640 1 -- 172.21.15.88:0/1064031890 shutdown_connections 2024-09-16T01:36:01.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.340+0000 7f135c6e6640 1 --2- 172.21.15.88:0/1064031890 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f134006c2e0 0x7f134006e7a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.340+0000 7f135c6e6640 1 --2- 172.21.15.88:0/1064031890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1354136f20 0x7f135414e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.340+0000 7f135c6e6640 1 --2- 172.21.15.88:0/1064031890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f135415b480 0x7f13540ce9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.340+0000 7f135c6e6640 1 --2- 172.21.15.88:0/1064031890 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1354072530 0x7f13540d0c00 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.341+0000 7f135c6e6640 1 -- 172.21.15.88:0/1064031890 >> 172.21.15.88:0/1064031890 conn(0x7f135406d0f0 msgr2=0x7f135415d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.341+0000 7f135c6e6640 1 -- 172.21.15.88:0/1064031890 shutdown_connections 2024-09-16T01:36:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.341+0000 7f135c6e6640 1 -- 172.21.15.88:0/1064031890 wait complete. 2024-09-16T01:36:01.353 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T01:36:01.353 DEBUG:teuthology.orchestra.run.smithi088:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T01:36:01.391 DEBUG:teuthology.orchestra.run.smithi154:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T01:36:01.435 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T01:36:01.435 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:36:01.435 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:36:01.451 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:36:01.452 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:36:01.452 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:36:01.494 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056026,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1347 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T01:36:01.494 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:36:01.494 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:36:01.509 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:36:01.509 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:36:01.509 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:36:01.552 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056026,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1347 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T01:36:01.552 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-16T01:36:01.552 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:36:01.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.650+0000 7f4093add640 1 Processor -- start 2024-09-16T01:36:01.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.650+0000 7f4093add640 1 -- start start 2024-09-16T01:36:01.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.651+0000 7f4093add640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f408c15c170 0x7f408c15c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:01.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.651+0000 7f4093add640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f408c0cf6c0 0x7f408c0cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:01.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.651+0000 7f4093add640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f408c0b49c0 con 0x7f408c0b5dd0 2024-09-16T01:36:01.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.651+0000 7f4093add640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f408c15ca90 con 0x7f408c15c170 2024-09-16T01:36:01.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.651+0000 7f4093add640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f408c0b4cf0 con 0x7f408c0cf6c0 2024-09-16T01:36:01.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.651+0000 7f4091879640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f408c15c170 0x7f408c15c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:01.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.651+0000 7f4091078640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f408c0cf6c0 0x7f408c0cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:01.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.651+0000 7f409207a640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f408c0b5dd0 0x7f408c15ffb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:58602/0 (socket says 172.21.15.88:58602) 2024-09-16T01:36:01.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.651+0000 7f4091879640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f408c15c170 0x7f408c15c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:39792/0 (socket says 172.21.15.88:39792) 2024-09-16T01:36:01.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.651+0000 7f409207a640 1 -- 172.21.15.88:0/4108827705 learned_addr learned my addr 172.21.15.88:0/4108827705 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:01.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.651+0000 7f4091078640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f408c0cf6c0 0x7f408c0cfb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:47398/0 (socket says 172.21.15.88:47398) 2024-09-16T01:36:01.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.651+0000 7f4091078640 1 -- 172.21.15.88:0/4108827705 >> v2:172.21.15.154:3300/0 conn(0x7f408c15c170 msgr2=0x7f408c15c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:01.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.651+0000 7f4091078640 1 --2- 172.21.15.88:0/4108827705 >> v2:172.21.15.154:3300/0 conn(0x7f408c15c170 0x7f408c15c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:01.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.651+0000 7f4091078640 1 -- 172.21.15.88:0/4108827705 >> v1:172.21.15.154:6789/0 conn(0x7f408c0b5dd0 legacy=0x7f408c15ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:01.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.652+0000 7f4091078640 1 -- 172.21.15.88:0/4108827705 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f408c0b4fd0 con 0x7f408c0cf6c0 2024-09-16T01:36:01.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.652+0000 7f4091879640 1 --2- 172.21.15.88:0/4108827705 >> v2:172.21.15.154:3300/0 conn(0x7f408c15c170 0x7f408c15c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:36:01.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.652+0000 7f4091078640 1 --2- 172.21.15.88:0/4108827705 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f408c0cf6c0 0x7f408c0cfb00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f407c0097a0 tx=0x7f407c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0169cfa58e10642 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:01.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.652+0000 7f4090877640 1 -- 172.21.15.88:0/4108827705 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407c037070 con 0x7f408c0cf6c0 2024-09-16T01:36:01.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.652+0000 7f4090877640 1 -- 172.21.15.88:0/4108827705 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f407c032440 con 0x7f408c0cf6c0 2024-09-16T01:36:01.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.652+0000 7f4090877640 1 -- 172.21.15.88:0/4108827705 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407c032810 con 0x7f408c0cf6c0 2024-09-16T01:36:01.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.653+0000 7f4093add640 1 -- 172.21.15.88:0/4108827705 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f408c0cf6c0 msgr2=0x7f408c0cfb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:01.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.653+0000 7f4093add640 1 --2- 172.21.15.88:0/4108827705 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f408c0cf6c0 0x7f408c0cfb00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f407c0097a0 tx=0x7f407c02a180 comp rx=0 tx=0).stop 2024-09-16T01:36:01.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.653+0000 7f4093add640 1 -- 172.21.15.88:0/4108827705 shutdown_connections 2024-09-16T01:36:01.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.653+0000 7f4093add640 1 --2- 172.21.15.88:0/4108827705 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f408c0cf6c0 0x7f408c0cfb00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:01.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.653+0000 7f4093add640 1 --2- 172.21.15.88:0/4108827705 >> v2:172.21.15.154:3300/0 conn(0x7f408c15c170 0x7f408c15c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:01.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.653+0000 7f4093add640 1 -- 172.21.15.88:0/4108827705 >> 172.21.15.88:0/4108827705 conn(0x7f408c06bc10 msgr2=0x7f408c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:01.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.653+0000 7f4093add640 1 -- 172.21.15.88:0/4108827705 shutdown_connections 2024-09-16T01:36:01.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.653+0000 7f4093add640 1 -- 172.21.15.88:0/4108827705 wait complete. 2024-09-16T01:36:01.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.654+0000 7f4093add640 1 Processor -- start 2024-09-16T01:36:01.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.654+0000 7f4093add640 1 -- start start 2024-09-16T01:36:01.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.655+0000 7f4093add640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f408c0b5dd0 0x7f408c150d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:01.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.655+0000 7f4093add640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f408c0cf6c0 0x7f408c14a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:01.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.655+0000 7f4093add640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f408c15c170 0x7f408c154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:01.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.655+0000 7f4093add640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f408c0b5870 con 0x7f408c0b5dd0 2024-09-16T01:36:01.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.655+0000 7f4093add640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f408c0b5590 con 0x7f408c15c170 2024-09-16T01:36:01.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.655+0000 7f4093add640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f408c0b5b50 con 0x7f408c0cf6c0 2024-09-16T01:36:01.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.655+0000 7f4091879640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f408c0b5dd0 0x7f408c150d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:01.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.655+0000 7f4091879640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f408c0b5dd0 0x7f408c150d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50832/0 (socket says 172.21.15.88:50832) 2024-09-16T01:36:01.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.655+0000 7f4091879640 1 -- 172.21.15.88:0/1698055661 learned_addr learned my addr 172.21.15.88:0/1698055661 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:01.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.655+0000 7f4091078640 1 --2- 172.21.15.88:0/1698055661 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f408c0cf6c0 0x7f408c14a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:01.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.655+0000 7f409207a640 1 --2- 172.21.15.88:0/1698055661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f408c15c170 0x7f408c154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:01.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.655+0000 7f4091879640 1 -- 172.21.15.88:0/1698055661 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f408c0cf6c0 msgr2=0x7f408c14a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:01.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.655+0000 7f4091879640 1 --2- 172.21.15.88:0/1698055661 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f408c0cf6c0 0x7f408c14a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:01.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.655+0000 7f4091879640 1 -- 172.21.15.88:0/1698055661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f408c15c170 msgr2=0x7f408c154b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:01.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.655+0000 7f4091879640 1 --2- 172.21.15.88:0/1698055661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f408c15c170 0x7f408c154b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:01.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.655+0000 7f4091879640 1 -- 172.21.15.88:0/1698055661 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f408c153da0 con 0x7f408c0b5dd0 2024-09-16T01:36:01.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.655+0000 7f409207a640 1 --2- 172.21.15.88:0/1698055661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f408c15c170 0x7f408c154b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:01.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.655+0000 7f4091078640 1 --2- 172.21.15.88:0/1698055661 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f408c0cf6c0 0x7f408c14a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:01.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.656+0000 7f4091879640 1 --2- 172.21.15.88:0/1698055661 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f408c0b5dd0 0x7f408c150d60 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f408800e960 tx=0x7f408800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:01.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.656+0000 7f40827fc640 1 -- 172.21.15.88:0/1698055661 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4088017070 con 0x7f408c0b5dd0 2024-09-16T01:36:01.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.656+0000 7f40827fc640 1 -- 172.21.15.88:0/1698055661 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f408800ca00 con 0x7f408c0b5dd0 2024-09-16T01:36:01.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.656+0000 7f4093add640 1 -- 172.21.15.88:0/1698055661 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f408c154080 con 0x7f408c0b5dd0 2024-09-16T01:36:01.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.656+0000 7f40827fc640 1 -- 172.21.15.88:0/1698055661 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408800cdd0 con 0x7f408c0b5dd0 2024-09-16T01:36:01.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.656+0000 7f4093add640 1 -- 172.21.15.88:0/1698055661 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f408c14b4e0 con 0x7f408c0b5dd0 2024-09-16T01:36:01.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.657+0000 7f4093add640 1 -- 172.21.15.88:0/1698055661 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f408c0cfb00 con 0x7f408c0b5dd0 2024-09-16T01:36:01.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.657+0000 7f40827fc640 1 -- 172.21.15.88:0/1698055661 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f408801a070 con 0x7f408c0b5dd0 2024-09-16T01:36:01.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.658+0000 7f40827fc640 1 --2- 172.21.15.88:0/1698055661 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f407006c2b0 0x7f407006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:01.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.658+0000 7f40827fc640 1 -- 172.21.15.88:0/1698055661 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5029+0+0 (secure 0 0 0) 0x7f408808b410 con 0x7f408c0b5dd0 2024-09-16T01:36:01.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.658+0000 7f4091078640 1 --2- 172.21.15.88:0/1698055661 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f407006c2b0 0x7f407006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:01.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.658+0000 7f4091078640 1 --2- 172.21.15.88:0/1698055661 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f407006c2b0 0x7f407006e770 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f407c034000 tx=0x7f407c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:01.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.662+0000 7f40827fc640 1 -- 172.21.15.88:0/1698055661 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4088059e30 con 0x7f408c0b5dd0 2024-09-16T01:36:01.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.861+0000 7f4093add640 1 -- 172.21.15.88:0/1698055661 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f408c152260 con 0x7f408c0b5dd0 2024-09-16T01:36:01.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.862+0000 7f40827fc640 1 -- 172.21.15.88:0/1698055661 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+3917 (secure 0 0 0) 0x7f4088059730 con 0x7f408c0b5dd0 2024-09-16T01:36:01.863 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:01.863 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":21,"btime":"2024-09-16T01:35:57:021381+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1005331674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1005331674},{"type":"v1","addr":"172.21.15.88:6837","nonce":1005331674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/3427247662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3427247662},{"type":"v1","addr":"172.21.15.154:6834","nonce":3427247662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":4663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/4059835337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":4059835337},{"type":"v1","addr":"172.21.15.154:6835","nonce":4059835337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5319,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3203861427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3203861427},{"type":"v1","addr":"172.21.15.88:6835","nonce":3203861427}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[]} 2024-09-16T01:36:01.864 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 21 2024-09-16T01:36:01.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.863+0000 7f4093add640 1 -- 172.21.15.88:0/1698055661 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f407006c2b0 msgr2=0x7f407006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:01.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.863+0000 7f4093add640 1 --2- 172.21.15.88:0/1698055661 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f407006c2b0 0x7f407006e770 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f407c034000 tx=0x7f407c02e000 comp rx=0 tx=0).stop 2024-09-16T01:36:01.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.863+0000 7f4093add640 1 -- 172.21.15.88:0/1698055661 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f408c0b5dd0 msgr2=0x7f408c150d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:01.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.863+0000 7f4093add640 1 --2- 172.21.15.88:0/1698055661 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f408c0b5dd0 0x7f408c150d60 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f408800e960 tx=0x7f408800ee20 comp rx=0 tx=0).stop 2024-09-16T01:36:01.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.864+0000 7f4093add640 1 -- 172.21.15.88:0/1698055661 shutdown_connections 2024-09-16T01:36:01.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.864+0000 7f4093add640 1 --2- 172.21.15.88:0/1698055661 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f407006c2b0 0x7f407006e770 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:01.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.864+0000 7f4093add640 1 --2- 172.21.15.88:0/1698055661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f408c15c170 0x7f408c154b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:01.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.864+0000 7f4093add640 1 --2- 172.21.15.88:0/1698055661 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f408c0cf6c0 0x7f408c14a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:01.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.864+0000 7f4093add640 1 --2- 172.21.15.88:0/1698055661 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f408c0b5dd0 0x7f408c150d60 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:01.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.864+0000 7f4093add640 1 -- 172.21.15.88:0/1698055661 >> 172.21.15.88:0/1698055661 conn(0x7f408c06bc10 msgr2=0x7f408c06d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:01.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.864+0000 7f4093add640 1 -- 172.21.15.88:0/1698055661 shutdown_connections 2024-09-16T01:36:01.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:01.864+0000 7f4093add640 1 -- 172.21.15.88:0/1698055661 wait complete. 2024-09-16T01:36:01.877 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-16T01:36:01.878 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-16T01:36:01.878 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-16T01:36:01.878 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:36:01.878 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-16T01:36:01.878 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-16T01:36:01.878 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-16T01:36:01.879 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:36:01.879 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-16T01:36:01.879 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-16T01:36:01.879 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-16T01:36:01.879 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:36:01.879 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-16T01:36:01.879 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-16T01:36:01.880 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-16T01:36:01.880 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:36:01.889 INFO:tasks.ceph.mds.c:Stopped 2024-09-16T01:36:01.890 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-16T01:36:01.890 INFO:tasks.ceph.mds.a:Stopped 2024-09-16T01:36:01.891 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-16T01:36:01.892 INFO:tasks.ceph.mds.b:Stopped 2024-09-16T01:36:01.892 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-16T01:36:01.892 INFO:tasks.ceph.mds.d:Stopped 2024-09-16T01:36:01.893 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-16T01:36:01.932 INFO:tasks.ceph.mds.c:Started 2024-09-16T01:36:01.933 DEBUG:teuthology.parallel:result is None 2024-09-16T01:36:01.934 INFO:tasks.ceph.mds.a:Started 2024-09-16T01:36:01.934 DEBUG:teuthology.parallel:result is None 2024-09-16T01:36:01.935 INFO:tasks.ceph.mds.b:Started 2024-09-16T01:36:01.935 DEBUG:teuthology.parallel:result is None 2024-09-16T01:36:01.936 INFO:tasks.ceph.mds.d:Started 2024-09-16T01:36:01.936 DEBUG:teuthology.parallel:result is None 2024-09-16T01:36:01.936 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-16T01:36:02.008 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:36:02.009 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:36:02.018 INFO:tasks.ceph.mds.b.smithi154.stdout:starting mds.b at 2024-09-16T01:36:02.018 INFO:tasks.ceph.mds.d.smithi154.stdout:starting mds.d at 2024-09-16T01:36:02.019 INFO:tasks.ceph.mds.d.smithi154.stderr:2024-09-16T01:36:02.017+0000 7f441adcd640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-09-16T01:36:02.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.039+0000 7fc1d313b640 1 Processor -- start 2024-09-16T01:36:02.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.039+0000 7fc1d313b640 1 -- start start 2024-09-16T01:36:02.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.040+0000 7fc1d313b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1cc15b480 0x7fc1cc0b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:02.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.040+0000 7fc1d313b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc1cc072700 0x7fc1cc15bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:02.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.040+0000 7fc1d313b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1cc0ccff0 con 0x7fc1cc0b60c0 2024-09-16T01:36:02.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.040+0000 7fc1d313b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc1cc0b6500 con 0x7fc1cc072700 2024-09-16T01:36:02.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.040+0000 7fc1d313b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc1cc0cd2d0 con 0x7fc1cc15b480 2024-09-16T01:36:02.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.040+0000 7fc1d0ed7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1cc15b480 0x7fc1cc0b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:02.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.040+0000 7fc1d16d8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc1cc072700 0x7fc1cc15bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:02.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.040+0000 7fc1cbfff640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fc1cc0b60c0 0x7fc1cc071ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:58612/0 (socket says 172.21.15.88:58612) 2024-09-16T01:36:02.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.040+0000 7fc1d0ed7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1cc15b480 0x7fc1cc0b5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:47440/0 (socket says 172.21.15.88:47440) 2024-09-16T01:36:02.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.040+0000 7fc1cbfff640 1 -- 172.21.15.88:0/429956137 learned_addr learned my addr 172.21.15.88:0/429956137 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:02.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.040+0000 7fc1d0ed7640 1 -- 172.21.15.88:0/429956137 >> v2:172.21.15.154:3300/0 conn(0x7fc1cc072700 msgr2=0x7fc1cc15bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:02.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.040+0000 7fc1d0ed7640 1 --2- 172.21.15.88:0/429956137 >> v2:172.21.15.154:3300/0 conn(0x7fc1cc072700 0x7fc1cc15bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:02.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.040+0000 7fc1d0ed7640 1 -- 172.21.15.88:0/429956137 >> v1:172.21.15.154:6789/0 conn(0x7fc1cc0b60c0 legacy=0x7fc1cc071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:02.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.040+0000 7fc1d0ed7640 1 -- 172.21.15.88:0/429956137 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1cc0cd5b0 con 0x7fc1cc15b480 2024-09-16T01:36:02.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.041+0000 7fc1d0ed7640 1 --2- 172.21.15.88:0/429956137 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1cc15b480 0x7fc1cc0b5b80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc0097a0 tx=0x7fc1bc02a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=640576d735660dfd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:02.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.041+0000 7fc1cb7fe640 1 -- 172.21.15.88:0/429956137 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc037070 con 0x7fc1cc15b480 2024-09-16T01:36:02.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.041+0000 7fc1cb7fe640 1 -- 172.21.15.88:0/429956137 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1bc02ae50 con 0x7fc1cc15b480 2024-09-16T01:36:02.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.041+0000 7fc1cb7fe640 1 -- 172.21.15.88:0/429956137 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc032660 con 0x7fc1cc15b480 2024-09-16T01:36:02.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.042+0000 7fc1d313b640 1 -- 172.21.15.88:0/429956137 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1cc15b480 msgr2=0x7fc1cc0b5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:02.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.042+0000 7fc1d313b640 1 --2- 172.21.15.88:0/429956137 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1cc15b480 0x7fc1cc0b5b80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc0097a0 tx=0x7fc1bc02a100 comp rx=0 tx=0).stop 2024-09-16T01:36:02.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.042+0000 7fc1d313b640 1 -- 172.21.15.88:0/429956137 shutdown_connections 2024-09-16T01:36:02.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.042+0000 7fc1d313b640 1 --2- 172.21.15.88:0/429956137 >> v2:172.21.15.154:3300/0 conn(0x7fc1cc072700 0x7fc1cc15bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:02.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.042+0000 7fc1d313b640 1 --2- 172.21.15.88:0/429956137 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1cc15b480 0x7fc1cc0b5b80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:02.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.042+0000 7fc1d313b640 1 -- 172.21.15.88:0/429956137 >> 172.21.15.88:0/429956137 conn(0x7fc1cc06d0f0 msgr2=0x7fc1cc06d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:02.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.042+0000 7fc1d313b640 1 -- 172.21.15.88:0/429956137 shutdown_connections 2024-09-16T01:36:02.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.042+0000 7fc1d313b640 1 -- 172.21.15.88:0/429956137 wait complete. 2024-09-16T01:36:02.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.043+0000 7fc1d313b640 1 Processor -- start 2024-09-16T01:36:02.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.043+0000 7fc1d313b640 1 -- start start 2024-09-16T01:36:02.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.044+0000 7fc1d313b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1cc072700 0x7fc1cc146660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:02.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.044+0000 7fc1d313b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1cc0b60c0 0x7fc1cc14a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:02.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.044+0000 7fc1d313b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1cc15b480 0x7fc1cc0ce500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:02.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.044+0000 7fc1d313b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc15b860 con 0x7fc1cc15b480 2024-09-16T01:36:02.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.044+0000 7fc1d313b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc0cdb70 con 0x7fc1cc072700 2024-09-16T01:36:02.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.044+0000 7fc1d313b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc1cc15b9e0 con 0x7fc1cc0b60c0 2024-09-16T01:36:02.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.044+0000 7fc1d0ed7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1cc072700 0x7fc1cc146660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:02.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.044+0000 7fc1d0ed7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1cc072700 0x7fc1cc146660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:39852/0 (socket says 172.21.15.88:39852) 2024-09-16T01:36:02.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.044+0000 7fc1d0ed7640 1 -- 172.21.15.88:0/3833539460 learned_addr learned my addr 172.21.15.88:0/3833539460 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:02.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.045+0000 7fc1cbfff640 1 --2- 172.21.15.88:0/3833539460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1cc0b60c0 0x7fc1cc14a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:02.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.045+0000 7fc1d16d8640 1 --2- 172.21.15.88:0/3833539460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1cc15b480 0x7fc1cc0ce500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:02.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.045+0000 7fc1d0ed7640 1 -- 172.21.15.88:0/3833539460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1cc0b60c0 msgr2=0x7fc1cc14a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:02.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.045+0000 7fc1d0ed7640 1 --2- 172.21.15.88:0/3833539460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1cc0b60c0 0x7fc1cc14a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:02.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.045+0000 7fc1d0ed7640 1 -- 172.21.15.88:0/3833539460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1cc15b480 msgr2=0x7fc1cc0ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:02.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.045+0000 7fc1d0ed7640 1 --2- 172.21.15.88:0/3833539460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1cc15b480 0x7fc1cc0ce500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:02.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.045+0000 7fc1d0ed7640 1 -- 172.21.15.88:0/3833539460 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1cc14bd90 con 0x7fc1cc072700 2024-09-16T01:36:02.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.045+0000 7fc1d16d8640 1 --2- 172.21.15.88:0/3833539460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1cc15b480 0x7fc1cc0ce500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:36:02.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.045+0000 7fc1d0ed7640 1 --2- 172.21.15.88:0/3833539460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1cc072700 0x7fc1cc146660 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc000c00 tx=0x7fc1bc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:02.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.049+0000 7fc1c97fa640 1 -- 172.21.15.88:0/3833539460 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc037040 con 0x7fc1cc072700 2024-09-16T01:36:02.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.049+0000 7fc1c97fa640 1 -- 172.21.15.88:0/3833539460 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1bc039600 con 0x7fc1cc072700 2024-09-16T01:36:02.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.049+0000 7fc1d313b640 1 -- 172.21.15.88:0/3833539460 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1cc1522f0 con 0x7fc1cc072700 2024-09-16T01:36:02.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.049+0000 7fc1c97fa640 1 -- 172.21.15.88:0/3833539460 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc0399d0 con 0x7fc1cc072700 2024-09-16T01:36:02.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.049+0000 7fc1d313b640 1 -- 172.21.15.88:0/3833539460 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1cc14c030 con 0x7fc1cc072700 2024-09-16T01:36:02.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.050+0000 7fc1d313b640 1 -- 172.21.15.88:0/3833539460 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1cc0b5b80 con 0x7fc1cc072700 2024-09-16T01:36:02.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.051+0000 7fc1c97fa640 1 -- 172.21.15.88:0/3833539460 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc1bc04a060 con 0x7fc1cc072700 2024-09-16T01:36:02.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.051+0000 7fc1c97fa640 1 --2- 172.21.15.88:0/3833539460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc1b806c2b0 0x7fc1b806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:02.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.051+0000 7fc1cbfff640 1 --2- 172.21.15.88:0/3833539460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc1b806c2b0 0x7fc1b806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:02.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.051+0000 7fc1cbfff640 1 --2- 172.21.15.88:0/3833539460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc1b806c2b0 0x7fc1b806e770 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc14fbf0 tx=0x7fc1c0007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:02.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.052+0000 7fc1c97fa640 1 -- 172.21.15.88:0/3833539460 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5029+0+0 (secure 0 0 0) 0x7fc1bc079290 con 0x7fc1cc072700 2024-09-16T01:36:02.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.057+0000 7fc1c97fa640 1 -- 172.21.15.88:0/3833539460 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc1bc03e070 con 0x7fc1cc072700 2024-09-16T01:36:02.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:02.234+0000 7fc1d313b640 1 -- 172.21.15.88:0/3833539460 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fc1cc15a9f0 con 0x7fc1cc072700 2024-09-16T01:36:03.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.067+0000 7fc1c97fa640 1 -- 172.21.15.88:0/3833539460 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v41) ==== 97+0+0 (secure 0 0 0) 0x7fc1bc07b1f0 con 0x7fc1cc072700 2024-09-16T01:36:03.068 INFO:teuthology.orchestra.run.smithi088.stderr: removed all blocklist entries 2024-09-16T01:36:03.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.069+0000 7fc1d313b640 1 -- 172.21.15.88:0/3833539460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc1b806c2b0 msgr2=0x7fc1b806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:03.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.069+0000 7fc1d313b640 1 --2- 172.21.15.88:0/3833539460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc1b806c2b0 0x7fc1b806e770 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc14fbf0 tx=0x7fc1c0007e80 comp rx=0 tx=0).stop 2024-09-16T01:36:03.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.069+0000 7fc1d313b640 1 -- 172.21.15.88:0/3833539460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1cc072700 msgr2=0x7fc1cc146660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:03.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.069+0000 7fc1d313b640 1 --2- 172.21.15.88:0/3833539460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1cc072700 0x7fc1cc146660 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc000c00 tx=0x7fc1bc0046c0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.069+0000 7fc1d313b640 1 -- 172.21.15.88:0/3833539460 shutdown_connections 2024-09-16T01:36:03.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.069+0000 7fc1d313b640 1 --2- 172.21.15.88:0/3833539460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc1b806c2b0 0x7fc1b806e770 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.069+0000 7fc1d313b640 1 --2- 172.21.15.88:0/3833539460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1cc15b480 0x7fc1cc0ce500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.069+0000 7fc1d313b640 1 --2- 172.21.15.88:0/3833539460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1cc0b60c0 0x7fc1cc14a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.069+0000 7fc1d313b640 1 --2- 172.21.15.88:0/3833539460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1cc072700 0x7fc1cc146660 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.069+0000 7fc1d313b640 1 -- 172.21.15.88:0/3833539460 >> 172.21.15.88:0/3833539460 conn(0x7fc1cc06d0f0 msgr2=0x7fc1cc0b2730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:03.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.069+0000 7fc1d313b640 1 -- 172.21.15.88:0/3833539460 shutdown_connections 2024-09-16T01:36:03.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.069+0000 7fc1d313b640 1 -- 172.21.15.88:0/3833539460 wait complete. 2024-09-16T01:36:03.084 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-16T01:36:03.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.187+0000 7fe83022c640 1 Processor -- start 2024-09-16T01:36:03.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.187+0000 7fe83022c640 1 -- start start 2024-09-16T01:36:03.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.188+0000 7fe83022c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe82815c2b0 0x7fe8280b6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:03.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.188+0000 7fe83022c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe82815f200 0x7fe82806d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:03.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.188+0000 7fe83022c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe82815fb10 con 0x7fe8280b72f0 2024-09-16T01:36:03.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.188+0000 7fe83022c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe8280b5900 con 0x7fe82815f200 2024-09-16T01:36:03.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.188+0000 7fe83022c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe82815f830 con 0x7fe82815c2b0 2024-09-16T01:36:03.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.188+0000 7fe82e7c9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe82815f200 0x7fe82806d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:03.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.188+0000 7fe82e7c9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe82815f200 0x7fe82806d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50896/0 (socket says 172.21.15.88:50896) 2024-09-16T01:36:03.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.188+0000 7fe82e7c9640 1 -- 172.21.15.88:0/883578577 learned_addr learned my addr 172.21.15.88:0/883578577 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:03.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.188+0000 7fe82dfc8640 1 --2- 172.21.15.88:0/883578577 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe82815c2b0 0x7fe8280b6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:03.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.188+0000 7fe82e7c9640 1 -- 172.21.15.88:0/883578577 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe82815c2b0 msgr2=0x7fe8280b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:03.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.188+0000 7fe82e7c9640 1 --2- 172.21.15.88:0/883578577 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe82815c2b0 0x7fe8280b6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.189+0000 7fe82e7c9640 1 -- 172.21.15.88:0/883578577 >> v1:172.21.15.154:6789/0 conn(0x7fe8280b72f0 legacy=0x7fe82815eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:03.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.189+0000 7fe82e7c9640 1 -- 172.21.15.88:0/883578577 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe82806ce30 con 0x7fe82815f200 2024-09-16T01:36:03.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.189+0000 7fe82e7c9640 1 --2- 172.21.15.88:0/883578577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe82815f200 0x7fe82806d910 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe820009870 tx=0x7fe82002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daf5e8b9212fa336 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:03.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.189+0000 7fe82cfc6640 1 -- 172.21.15.88:0/883578577 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe820037070 con 0x7fe82815f200 2024-09-16T01:36:03.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.189+0000 7fe82cfc6640 1 -- 172.21.15.88:0/883578577 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe820032440 con 0x7fe82815f200 2024-09-16T01:36:03.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.189+0000 7fe82cfc6640 1 -- 172.21.15.88:0/883578577 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe820032810 con 0x7fe82815f200 2024-09-16T01:36:03.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.190+0000 7fe83022c640 1 -- 172.21.15.88:0/883578577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe82815f200 msgr2=0x7fe82806d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:03.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.190+0000 7fe83022c640 1 --2- 172.21.15.88:0/883578577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe82815f200 0x7fe82806d910 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe820009870 tx=0x7fe82002a270 comp rx=0 tx=0).stop 2024-09-16T01:36:03.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.190+0000 7fe83022c640 1 -- 172.21.15.88:0/883578577 shutdown_connections 2024-09-16T01:36:03.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.190+0000 7fe83022c640 1 --2- 172.21.15.88:0/883578577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe82815f200 0x7fe82806d910 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.190+0000 7fe83022c640 1 --2- 172.21.15.88:0/883578577 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe82815c2b0 0x7fe8280b6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.190+0000 7fe83022c640 1 -- 172.21.15.88:0/883578577 >> 172.21.15.88:0/883578577 conn(0x7fe8280b4240 msgr2=0x7fe8280b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:03.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.190+0000 7fe83022c640 1 -- 172.21.15.88:0/883578577 shutdown_connections 2024-09-16T01:36:03.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.190+0000 7fe83022c640 1 -- 172.21.15.88:0/883578577 wait complete. 2024-09-16T01:36:03.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.191+0000 7fe83022c640 1 Processor -- start 2024-09-16T01:36:03.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.191+0000 7fe83022c640 1 -- start start 2024-09-16T01:36:03.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.192+0000 7fe83022c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8280b72f0 0x7fe8280cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:03.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.192+0000 7fe83022c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe82815c2b0 0x7fe8281f5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:03.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.192+0000 7fe83022c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe82815f200 0x7fe82814b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:03.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.192+0000 7fe83022c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe8280b7730 con 0x7fe8280b72f0 2024-09-16T01:36:03.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.192+0000 7fe83022c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe82806d3f0 con 0x7fe82815c2b0 2024-09-16T01:36:03.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.192+0000 7fe83022c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe82806de50 con 0x7fe82815f200 2024-09-16T01:36:03.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.192+0000 7fe82dfc8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8280b72f0 0x7fe8280cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:03.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.192+0000 7fe82dfc8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8280b72f0 0x7fe8280cb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50900/0 (socket says 172.21.15.88:50900) 2024-09-16T01:36:03.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.192+0000 7fe82dfc8640 1 -- 172.21.15.88:0/1247016620 learned_addr learned my addr 172.21.15.88:0/1247016620 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:03.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.192+0000 7fe82d7c7640 1 --2- 172.21.15.88:0/1247016620 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe82815c2b0 0x7fe8281f5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:03.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.192+0000 7fe82e7c9640 1 --2- 172.21.15.88:0/1247016620 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe82815f200 0x7fe82814b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:03.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.192+0000 7fe82dfc8640 1 -- 172.21.15.88:0/1247016620 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe82815f200 msgr2=0x7fe82814b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:03.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.192+0000 7fe82dfc8640 1 --2- 172.21.15.88:0/1247016620 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe82815f200 0x7fe82814b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.192+0000 7fe82dfc8640 1 -- 172.21.15.88:0/1247016620 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe82815c2b0 msgr2=0x7fe8281f5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:03.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.192+0000 7fe82dfc8640 1 --2- 172.21.15.88:0/1247016620 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe82815c2b0 0x7fe8281f5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.192+0000 7fe82dfc8640 1 -- 172.21.15.88:0/1247016620 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8281f5540 con 0x7fe8280b72f0 2024-09-16T01:36:03.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.192+0000 7fe82dfc8640 1 --2- 172.21.15.88:0/1247016620 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8280b72f0 0x7fe8280cb8c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fe824007ec0 tx=0x7fe82400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:03.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.193+0000 7fe81e7fc640 1 -- 172.21.15.88:0/1247016620 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe824014620 con 0x7fe8280b72f0 2024-09-16T01:36:03.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.193+0000 7fe81e7fc640 1 -- 172.21.15.88:0/1247016620 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe824019070 con 0x7fe8280b72f0 2024-09-16T01:36:03.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.193+0000 7fe81e7fc640 1 -- 172.21.15.88:0/1247016620 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe824014a20 con 0x7fe8280b72f0 2024-09-16T01:36:03.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.193+0000 7fe83022c640 1 -- 172.21.15.88:0/1247016620 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8280cbea0 con 0x7fe8280b72f0 2024-09-16T01:36:03.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.193+0000 7fe83022c640 1 -- 172.21.15.88:0/1247016620 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8280cc460 con 0x7fe8280b72f0 2024-09-16T01:36:03.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.194+0000 7fe83022c640 1 -- 172.21.15.88:0/1247016620 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8280b6db0 con 0x7fe8280b72f0 2024-09-16T01:36:03.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.194+0000 7fe81e7fc640 1 -- 172.21.15.88:0/1247016620 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe824014bc0 con 0x7fe8280b72f0 2024-09-16T01:36:03.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.195+0000 7fe81e7fc640 1 --2- 172.21.15.88:0/1247016620 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe80c06c2b0 0x7fe80c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:03.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.195+0000 7fe81e7fc640 1 -- 172.21.15.88:0/1247016620 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4857+0+0 (secure 0 0 0) 0x7fe82408b940 con 0x7fe8280b72f0 2024-09-16T01:36:03.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.195+0000 7fe82d7c7640 1 --2- 172.21.15.88:0/1247016620 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe80c06c2b0 0x7fe80c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:03.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.195+0000 7fe82d7c7640 1 --2- 172.21.15.88:0/1247016620 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe80c06c2b0 0x7fe80c06e770 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe82815a0b0 tx=0x7fe818009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:03.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.199+0000 7fe81e7fc640 1 -- 172.21.15.88:0/1247016620 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe8281f5540 con 0x7fe8280b72f0 2024-09-16T01:36:03.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.428+0000 7fe83022c640 1 -- 172.21.15.88:0/1247016620 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fe828161cf0 con 0x7fe8280b72f0 2024-09-16T01:36:03.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.428+0000 7fe81e7fc640 1 -- 172.21.15.88:0/1247016620 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) ==== 80+0+5304 (secure 0 0 0) 0x7fe82405a490 con 0x7fe8280b72f0 2024-09-16T01:36:03.430 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:03.430 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:36:03.430 INFO:teuthology.orchestra.run.smithi088.stdout: "auth_dump": [ 2024-09-16T01:36:03.430 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:03.430 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mds.a", 2024-09-16T01:36:03.430 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmy+jJCRAAhnzX/li6D3D07NmThrjXkA==", 2024-09-16T01:36:03.430 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:03.431 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:36:03.431 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:36:03.431 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:36:03.431 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:03.431 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:03.431 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:03.431 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:03.431 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mds.b", 2024-09-16T01:36:03.432 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmx7p+FRAApxKUSZoR/nGUZsaYKlrYUg==", 2024-09-16T01:36:03.432 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:03.432 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:36:03.432 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:36:03.432 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:36:03.432 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:03.432 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:03.432 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:03.432 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:03.432 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mds.c", 2024-09-16T01:36:03.432 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmaYHUEBAAzM0qE53LEgHi+hThHJX3Tw==", 2024-09-16T01:36:03.432 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:03.432 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:36:03.433 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:36:03.433 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:36:03.433 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:03.433 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:03.433 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:03.433 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:03.433 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mds.d", 2024-09-16T01:36:03.433 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudm3TSLHBAAJzlR6H09cc1Pki2FjyoFiA==", 2024-09-16T01:36:03.433 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:03.433 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:36:03.433 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:36:03.433 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:36:03.434 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:03.434 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:03.434 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:03.434 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:03.434 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.0", 2024-09-16T01:36:03.434 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDhiudmZR7MORAAwDKvphubA+CbzBUWuaA80A==", 2024-09-16T01:36:03.434 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:03.434 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:36:03.434 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:36:03.434 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:03.434 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:03.434 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:03.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:03.435 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.1", 2024-09-16T01:36:03.435 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDliudmNd+SLBAArOUdoZ226c5jlAsKwqERmw==", 2024-09-16T01:36:03.435 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:03.435 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:36:03.435 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:36:03.435 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:03.435 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:03.435 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:03.435 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:03.435 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.2", 2024-09-16T01:36:03.435 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDpiudm75CUARAAWHtYOg9pE8g+M6IL+8XZ6g==", 2024-09-16T01:36:03.435 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:03.436 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:36:03.436 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:36:03.436 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:03.436 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:03.436 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:03.436 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:03.436 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.3", 2024-09-16T01:36:03.436 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDtiudm0fgWAxAAf2NOWrpXbOhb3dFXozJagQ==", 2024-09-16T01:36:03.436 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:03.436 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:36:03.436 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:36:03.436 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:03.436 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:03.437 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:03.437 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:03.437 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.4", 2024-09-16T01:36:03.437 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQD0iudmdddMCxAAtr2qfj4Nxj3/GHUpxRmtFg==", 2024-09-16T01:36:03.437 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:03.437 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:36:03.437 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:36:03.437 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:03.437 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:03.437 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:03.437 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:03.437 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.5", 2024-09-16T01:36:03.437 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQD3iudmT5RjOhAAcKwca6BG60gsxYnmy4twGg==", 2024-09-16T01:36:03.438 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:03.438 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:36:03.438 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:36:03.438 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:03.438 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:03.438 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:03.438 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:03.438 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.6", 2024-09-16T01:36:03.438 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQD7iudmT7daLRAAdfLFijKQku3rJcJ5xcfLsw==", 2024-09-16T01:36:03.438 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:03.438 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:36:03.438 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:36:03.438 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:03.439 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:03.439 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:03.439 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:03.439 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.7", 2024-09-16T01:36:03.439 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQD/iudmmvtxLhAArYKhQ4zyFHtLgh/+bs5ZhA==", 2024-09-16T01:36:03.439 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:03.439 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:36:03.439 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:36:03.439 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:03.439 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:03.439 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:03.439 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:03.440 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.0", 2024-09-16T01:36:03.440 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmvPikIBAAyHxNNnwQKygxg09PNQLc/Q==", 2024-09-16T01:36:03.440 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:03.440 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:36:03.440 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow r", 2024-09-16T01:36:03.440 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-16T01:36:03.440 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:03.440 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:03.440 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:03.440 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.1", 2024-09-16T01:36:03.440 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmA69iKRAA8Abwe0Lg2PnZDErgJ1gEbQ==", 2024-09-16T01:36:03.441 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:03.441 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:36:03.441 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow r", 2024-09-16T01:36:03.441 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow rw", 2024-09-16T01:36:03.441 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow rwx" 2024-09-16T01:36:03.441 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:03.441 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:03.441 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:03.441 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.2", 2024-09-16T01:36:03.441 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmB+w9JhAAv63Is5T/cvDbbLOjIkI0wA==", 2024-09-16T01:36:03.441 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:03.441 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:36:03.441 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow r", 2024-09-16T01:36:03.442 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow rw", 2024-09-16T01:36:03.442 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow rwx" 2024-09-16T01:36:03.442 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:03.442 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:03.442 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:03.442 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.3", 2024-09-16T01:36:03.442 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudm4WoFLxAAnYGHGqHyBHHajQ8f3rCjEA==", 2024-09-16T01:36:03.442 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:03.442 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:36:03.442 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow r", 2024-09-16T01:36:03.442 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow rw", 2024-09-16T01:36:03.442 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow rwx" 2024-09-16T01:36:03.442 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:03.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:03.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:03.443 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.admin", 2024-09-16T01:36:03.443 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDdiudmhcE0KBAANauQKyUY01/KlWMrN2x/5Q==", 2024-09-16T01:36:03.443 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:03.443 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow *", 2024-09-16T01:36:03.443 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:36:03.443 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:36:03.443 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:03.443 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:03.443 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:03.443 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:03.443 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mgr.x", 2024-09-16T01:36:03.444 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmToGYBhAAKwXS34PtfAVtEtdtHKHlQQ==", 2024-09-16T01:36:03.444 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:03.444 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow *", 2024-09-16T01:36:03.444 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile mgr", 2024-09-16T01:36:03.444 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:03.444 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:03.444 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:03.444 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:03.444 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mgr.y", 2024-09-16T01:36:03.444 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudm/BN5ARAAxbCopbsvxFUZP3drso32qw==", 2024-09-16T01:36:03.444 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:03.444 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow *", 2024-09-16T01:36:03.445 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile mgr", 2024-09-16T01:36:03.445 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:03.445 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:03.445 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:03.445 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-09-16T01:36:03.445 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:36:03.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.430+0000 7fe83022c640 1 -- 172.21.15.88:0/1247016620 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe80c06c2b0 msgr2=0x7fe80c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:03.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.430+0000 7fe83022c640 1 --2- 172.21.15.88:0/1247016620 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe80c06c2b0 0x7fe80c06e770 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe82815a0b0 tx=0x7fe818009040 comp rx=0 tx=0).stop 2024-09-16T01:36:03.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.430+0000 7fe83022c640 1 -- 172.21.15.88:0/1247016620 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8280b72f0 msgr2=0x7fe8280cb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:03.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.430+0000 7fe83022c640 1 --2- 172.21.15.88:0/1247016620 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8280b72f0 0x7fe8280cb8c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fe824007ec0 tx=0x7fe82400e4b0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.430+0000 7fe83022c640 1 -- 172.21.15.88:0/1247016620 shutdown_connections 2024-09-16T01:36:03.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.430+0000 7fe83022c640 1 --2- 172.21.15.88:0/1247016620 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe80c06c2b0 0x7fe80c06e770 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.430+0000 7fe83022c640 1 --2- 172.21.15.88:0/1247016620 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe82815f200 0x7fe82814b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.430+0000 7fe83022c640 1 --2- 172.21.15.88:0/1247016620 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe82815c2b0 0x7fe8281f5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.430+0000 7fe83022c640 1 --2- 172.21.15.88:0/1247016620 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8280b72f0 0x7fe8280cb8c0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.430+0000 7fe83022c640 1 -- 172.21.15.88:0/1247016620 >> 172.21.15.88:0/1247016620 conn(0x7fe8280b4240 msgr2=0x7fe8280b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:03.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.430+0000 7fe83022c640 1 -- 172.21.15.88:0/1247016620 shutdown_connections 2024-09-16T01:36:03.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.430+0000 7fe83022c640 1 -- 172.21.15.88:0/1247016620 wait complete. 2024-09-16T01:36:03.448 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-16T01:36:03.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.545+0000 7f4c3a942640 1 Processor -- start 2024-09-16T01:36:03.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.545+0000 7f4c3a942640 1 -- start start 2024-09-16T01:36:03.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.546+0000 7f4c3a942640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c3415c2b0 0x7f4c340b6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:03.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.546+0000 7f4c3a942640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4c3415f200 0x7f4c3406d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:03.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.546+0000 7f4c3a942640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c3415fb10 con 0x7f4c340b72f0 2024-09-16T01:36:03.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.546+0000 7f4c3a942640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f4c340b5900 con 0x7f4c3415f200 2024-09-16T01:36:03.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.546+0000 7f4c3a942640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4c3415f830 con 0x7f4c3415c2b0 2024-09-16T01:36:03.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.546+0000 7f4c2bfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4c340b72f0 0x7f4c3415eaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59332/0 (socket says 172.21.15.88:59332) 2024-09-16T01:36:03.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.546+0000 7f4c2bfff640 1 -- 172.21.15.88:0/3996098449 learned_addr learned my addr 172.21.15.88:0/3996098449 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:03.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c337fe640 1 -- 172.21.15.88:0/3996098449 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1046765115 0 0) 0x7f4c3415fb10 con 0x7f4c340b72f0 2024-09-16T01:36:03.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c337fe640 1 -- 172.21.15.88:0/3996098449 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c20003520 con 0x7f4c340b72f0 2024-09-16T01:36:03.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c38edf640 1 --2- 172.21.15.88:0/3996098449 >> v2:172.21.15.154:3300/0 conn(0x7f4c3415f200 0x7f4c3406d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:03.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c33fff640 1 --2- 172.21.15.88:0/3996098449 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c3415c2b0 0x7f4c340b6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:03.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c337fe640 1 -- 172.21.15.88:0/3996098449 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 19416766 0 0) 0x7f4c20003520 con 0x7f4c340b72f0 2024-09-16T01:36:03.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c337fe640 1 -- 172.21.15.88:0/3996098449 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c3415c2b0 msgr2=0x7f4c340b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:03.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c337fe640 1 --2- 172.21.15.88:0/3996098449 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c3415c2b0 0x7f4c340b6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c337fe640 1 -- 172.21.15.88:0/3996098449 >> v2:172.21.15.154:3300/0 conn(0x7f4c3415f200 msgr2=0x7f4c3406d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:03.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c337fe640 1 --2- 172.21.15.88:0/3996098449 >> v2:172.21.15.154:3300/0 conn(0x7f4c3415f200 0x7f4c3406d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c337fe640 1 -- 172.21.15.88:0/3996098449 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c3406ce30 con 0x7f4c340b72f0 2024-09-16T01:36:03.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c337fe640 1 -- 172.21.15.88:0/3996098449 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f4c1c001670 con 0x7f4c340b72f0 2024-09-16T01:36:03.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c337fe640 1 -- 172.21.15.88:0/3996098449 >> v1:172.21.15.88:6789/0 conn(0x7f4c340b72f0 legacy=0x7f4c3415eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:03.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c337fe640 1 --2- 172.21.15.88:0/3996098449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c200039b0 0x7f4c20003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:03.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c337fe640 1 --2- 172.21.15.88:0/3996098449 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c20004800 0x7f4c20004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:03.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c337fe640 1 --2- 172.21.15.88:0/3996098449 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c3415c2b0 0x7f4c200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:03.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c337fe640 1 -- 172.21.15.88:0/3996098449 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4c340b5900 con 0x7f4c20004800 2024-09-16T01:36:03.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c337fe640 1 -- 172.21.15.88:0/3996098449 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4c3415f830 con 0x7f4c200039b0 2024-09-16T01:36:03.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c337fe640 1 -- 172.21.15.88:0/3996098449 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4c2000b9c0 con 0x7f4c3415c2b0 2024-09-16T01:36:03.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c2bfff640 1 --2- 172.21.15.88:0/3996098449 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c20004800 0x7f4c20004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:03.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c33fff640 1 --2- 172.21.15.88:0/3996098449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c200039b0 0x7f4c20003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:03.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c38edf640 1 --2- 172.21.15.88:0/3996098449 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c3415c2b0 0x7f4c200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:03.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c2bfff640 1 -- 172.21.15.88:0/3996098449 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c3415c2b0 msgr2=0x7f4c200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:03.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c2bfff640 1 --2- 172.21.15.88:0/3996098449 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c3415c2b0 0x7f4c200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c2bfff640 1 -- 172.21.15.88:0/3996098449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c200039b0 msgr2=0x7f4c20003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:03.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c2bfff640 1 --2- 172.21.15.88:0/3996098449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c200039b0 0x7f4c20003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.547+0000 7f4c2bfff640 1 -- 172.21.15.88:0/3996098449 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c20003520 con 0x7f4c20004800 2024-09-16T01:36:03.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.548+0000 7f4c2bfff640 1 --2- 172.21.15.88:0/3996098449 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c20004800 0x7f4c20004c80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4c34071730 tx=0x7f4c20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7a7218a1c7c1976 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:03.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.548+0000 7f4c337fe640 1 -- 172.21.15.88:0/3996098449 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c007c80 con 0x7f4c20004800 2024-09-16T01:36:03.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.548+0000 7f4c337fe640 1 -- 172.21.15.88:0/3996098449 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c1c005af0 con 0x7f4c20004800 2024-09-16T01:36:03.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.548+0000 7f4c337fe640 1 -- 172.21.15.88:0/3996098449 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c007070 con 0x7f4c20004800 2024-09-16T01:36:03.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.549+0000 7f4c3a942640 1 -- 172.21.15.88:0/3996098449 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c20004800 msgr2=0x7f4c20004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:03.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.549+0000 7f4c3a942640 1 --2- 172.21.15.88:0/3996098449 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c20004800 0x7f4c20004c80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4c34071730 tx=0x7f4c20005830 comp rx=0 tx=0).stop 2024-09-16T01:36:03.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.549+0000 7f4c38edf640 1 -- 172.21.15.88:0/3996098449 reap_dead start 2024-09-16T01:36:03.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.549+0000 7f4c3a942640 1 -- 172.21.15.88:0/3996098449 shutdown_connections 2024-09-16T01:36:03.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.549+0000 7f4c3a942640 1 -- 172.21.15.88:0/3996098449 >> 172.21.15.88:0/3996098449 conn(0x7f4c340b4240 msgr2=0x7f4c340b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:03.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.549+0000 7f4c3a942640 1 -- 172.21.15.88:0/3996098449 shutdown_connections 2024-09-16T01:36:03.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.549+0000 7f4c3a942640 1 -- 172.21.15.88:0/3996098449 wait complete. 2024-09-16T01:36:03.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.550+0000 7f4c3a942640 1 Processor -- start 2024-09-16T01:36:03.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.550+0000 7f4c3a942640 1 -- start start 2024-09-16T01:36:03.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.551+0000 7f4c3a942640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c20004800 0x7f4c34149fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:03.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.551+0000 7f4c3a942640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c34146ff0 0x7f4c34147470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:03.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.551+0000 7f4c3a942640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c341e7390 0x7f4c341e7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:03.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.551+0000 7f4c3a942640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4c340b7730 con 0x7f4c34146ff0 2024-09-16T01:36:03.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.551+0000 7f4c3a942640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4c3406d3f0 con 0x7f4c20004800 2024-09-16T01:36:03.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.551+0000 7f4c3a942640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4c340b6db0 con 0x7f4c341e7390 2024-09-16T01:36:03.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.551+0000 7f4c2bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c34146ff0 0x7f4c34147470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:03.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.551+0000 7f4c2bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c34146ff0 0x7f4c34147470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50932/0 (socket says 172.21.15.88:50932) 2024-09-16T01:36:03.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.551+0000 7f4c2bfff640 1 -- 172.21.15.88:0/2126078390 learned_addr learned my addr 172.21.15.88:0/2126078390 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:03.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.551+0000 7f4c33fff640 1 --2- 172.21.15.88:0/2126078390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c20004800 0x7f4c34149fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:03.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.551+0000 7f4c38edf640 1 --2- 172.21.15.88:0/2126078390 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c341e7390 0x7f4c341e7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:03.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.551+0000 7f4c2bfff640 1 -- 172.21.15.88:0/2126078390 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c341e7390 msgr2=0x7f4c341e7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:03.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.551+0000 7f4c2bfff640 1 --2- 172.21.15.88:0/2126078390 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c341e7390 0x7f4c341e7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.551+0000 7f4c2bfff640 1 -- 172.21.15.88:0/2126078390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c20004800 msgr2=0x7f4c34149fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:03.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.551+0000 7f4c2bfff640 1 --2- 172.21.15.88:0/2126078390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c20004800 0x7f4c34149fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.551+0000 7f4c2bfff640 1 -- 172.21.15.88:0/2126078390 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c341479b0 con 0x7f4c34146ff0 2024-09-16T01:36:03.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.551+0000 7f4c2bfff640 1 --2- 172.21.15.88:0/2126078390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c34146ff0 0x7f4c34147470 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c00c250 tx=0x7f4c3406cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:03.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.551+0000 7f4c317fa640 1 -- 172.21.15.88:0/2126078390 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c041030 con 0x7f4c34146ff0 2024-09-16T01:36:03.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.551+0000 7f4c317fa640 1 -- 172.21.15.88:0/2126078390 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c1c038040 con 0x7f4c34146ff0 2024-09-16T01:36:03.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.551+0000 7f4c317fa640 1 -- 172.21.15.88:0/2126078390 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c001670 con 0x7f4c34146ff0 2024-09-16T01:36:03.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.551+0000 7f4c3a942640 1 -- 172.21.15.88:0/2126078390 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c340d2270 con 0x7f4c34146ff0 2024-09-16T01:36:03.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.552+0000 7f4c3a942640 1 -- 172.21.15.88:0/2126078390 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c340d2830 con 0x7f4c34146ff0 2024-09-16T01:36:03.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.552+0000 7f4c3a942640 1 -- 172.21.15.88:0/2126078390 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c3415eaf0 con 0x7f4c34146ff0 2024-09-16T01:36:03.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.553+0000 7f4c317fa640 1 -- 172.21.15.88:0/2126078390 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4c1c044070 con 0x7f4c34146ff0 2024-09-16T01:36:03.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.553+0000 7f4c317fa640 1 --2- 172.21.15.88:0/2126078390 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c1406c2e0 0x7f4c1406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:03.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.553+0000 7f4c317fa640 1 -- 172.21.15.88:0/2126078390 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4857+0+0 (secure 0 0 0) 0x7f4c1c0aab10 con 0x7f4c34146ff0 2024-09-16T01:36:03.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.553+0000 7f4c33fff640 1 --2- 172.21.15.88:0/2126078390 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c1406c2e0 0x7f4c1406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:03.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.554+0000 7f4c33fff640 1 --2- 172.21.15.88:0/2126078390 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c1406c2e0 0x7f4c1406e7a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4c24009a60 tx=0x7f4c24009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:03.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.558+0000 7f4c317fa640 1 -- 172.21.15.88:0/2126078390 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4c1c04b020 con 0x7f4c34146ff0 2024-09-16T01:36:03.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.724+0000 7f4c3a942640 1 -- 172.21.15.88:0/2126078390 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f4c3406ea60 con 0x7f4c34146ff0 2024-09-16T01:36:03.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.725+0000 7f4c317fa640 1 -- 172.21.15.88:0/2126078390 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v41) ==== 84+0+65 (secure 0 0 0) 0x7f4c1c079540 con 0x7f4c34146ff0 2024-09-16T01:36:03.726 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:03.726 INFO:teuthology.orchestra.run.smithi088.stdout:[ 2024-09-16T01:36:03.727 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:03.727 INFO:teuthology.orchestra.run.smithi088.stdout: "poolnum": 1, 2024-09-16T01:36:03.727 INFO:teuthology.orchestra.run.smithi088.stdout: "poolname": ".mgr" 2024-09-16T01:36:03.727 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:03.727 INFO:teuthology.orchestra.run.smithi088.stdout:] 2024-09-16T01:36:03.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.727+0000 7f4c3a942640 1 -- 172.21.15.88:0/2126078390 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c1406c2e0 msgr2=0x7f4c1406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:03.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.727+0000 7f4c3a942640 1 --2- 172.21.15.88:0/2126078390 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c1406c2e0 0x7f4c1406e7a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4c24009a60 tx=0x7f4c24009190 comp rx=0 tx=0).stop 2024-09-16T01:36:03.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.727+0000 7f4c3a942640 1 -- 172.21.15.88:0/2126078390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c34146ff0 msgr2=0x7f4c34147470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:03.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.727+0000 7f4c3a942640 1 --2- 172.21.15.88:0/2126078390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c34146ff0 0x7f4c34147470 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c00c250 tx=0x7f4c3406cd20 comp rx=0 tx=0).stop 2024-09-16T01:36:03.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.727+0000 7f4c3a942640 1 -- 172.21.15.88:0/2126078390 shutdown_connections 2024-09-16T01:36:03.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.727+0000 7f4c3a942640 1 --2- 172.21.15.88:0/2126078390 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c1406c2e0 0x7f4c1406e7a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.727+0000 7f4c3a942640 1 --2- 172.21.15.88:0/2126078390 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c341e7390 0x7f4c341e7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.727+0000 7f4c3a942640 1 --2- 172.21.15.88:0/2126078390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c34146ff0 0x7f4c34147470 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.727+0000 7f4c3a942640 1 --2- 172.21.15.88:0/2126078390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c20004800 0x7f4c34149fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.727+0000 7f4c3a942640 1 -- 172.21.15.88:0/2126078390 >> 172.21.15.88:0/2126078390 conn(0x7f4c340b4240 msgr2=0x7f4c340cd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:03.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.727+0000 7f4c3a942640 1 -- 172.21.15.88:0/2126078390 shutdown_connections 2024-09-16T01:36:03.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.727+0000 7f4c3a942640 1 -- 172.21.15.88:0/2126078390 wait complete. 2024-09-16T01:36:03.739 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-16T01:36:03.739 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-16T01:36:03.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.839+0000 7f3ab8df1640 1 Processor -- start 2024-09-16T01:36:03.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.839+0000 7f3ab8df1640 1 -- start start 2024-09-16T01:36:03.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.840+0000 7f3ab8df1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ab40b3920 0x7f3ab4162b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:03.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.840+0000 7f3ab8df1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3ab40d29a0 0x7f3ab40d2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:03.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.840+0000 7f3ab8df1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ab415ed70 con 0x7f3ab415e990 2024-09-16T01:36:03.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.840+0000 7f3ab8df1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f3ab40cff20 con 0x7f3ab40d29a0 2024-09-16T01:36:03.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.840+0000 7f3ab8df1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3ab40b3d60 con 0x7f3ab40b3920 2024-09-16T01:36:03.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.841+0000 7f3ab2d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3ab40d29a0 0x7f3ab40d2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:03.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.841+0000 7f3ab259c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3ab415e990 0x7f3ab4162450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59334/0 (socket says 172.21.15.88:59334) 2024-09-16T01:36:03.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.841+0000 7f3ab2d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3ab40d29a0 0x7f3ab40d2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50936/0 (socket says 172.21.15.88:50936) 2024-09-16T01:36:03.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.841+0000 7f3ab259c640 1 -- 172.21.15.88:0/2287212007 learned_addr learned my addr 172.21.15.88:0/2287212007 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:03.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.841+0000 7f3ab1d9b640 1 --2- 172.21.15.88:0/2287212007 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ab40b3920 0x7f3ab4162b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:03.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.841+0000 7f3ab2d9d640 1 -- 172.21.15.88:0/2287212007 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ab40b3920 msgr2=0x7f3ab4162b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:03.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.841+0000 7f3ab2d9d640 1 --2- 172.21.15.88:0/2287212007 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ab40b3920 0x7f3ab4162b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.841+0000 7f3ab2d9d640 1 -- 172.21.15.88:0/2287212007 >> v1:172.21.15.88:6789/0 conn(0x7f3ab415e990 legacy=0x7f3ab4162450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:03.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.841+0000 7f3ab2d9d640 1 -- 172.21.15.88:0/2287212007 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ab40d1fa0 con 0x7f3ab40d29a0 2024-09-16T01:36:03.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.841+0000 7f3ab159a640 1 -- 172.21.15.88:0/2287212007 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 844426710 0 0) 0x7f3aa4001110 con 0x7f3ab415e990 2024-09-16T01:36:03.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.841+0000 7f3ab1d9b640 1 --2- 172.21.15.88:0/2287212007 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ab40b3920 0x7f3ab4162b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:03.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.842+0000 7f3ab2d9d640 1 --2- 172.21.15.88:0/2287212007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ab40d29a0 0x7f3ab40d2e50 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3aac009d90 tx=0x7f3aac02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cafe531a520db439 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:03.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.842+0000 7f3ab159a640 1 -- 172.21.15.88:0/2287212007 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aac031600 con 0x7f3ab40d29a0 2024-09-16T01:36:03.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.842+0000 7f3ab159a640 1 -- 172.21.15.88:0/2287212007 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3aac0317a0 con 0x7f3ab40d29a0 2024-09-16T01:36:03.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.842+0000 7f3ab159a640 1 -- 172.21.15.88:0/2287212007 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aac031b70 con 0x7f3ab40d29a0 2024-09-16T01:36:03.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.842+0000 7f3ab8df1640 1 -- 172.21.15.88:0/2287212007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ab40d29a0 msgr2=0x7f3ab40d2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:03.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.842+0000 7f3ab8df1640 1 --2- 172.21.15.88:0/2287212007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ab40d29a0 0x7f3ab40d2e50 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3aac009d90 tx=0x7f3aac02a5d0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.842+0000 7f3ab8df1640 1 -- 172.21.15.88:0/2287212007 shutdown_connections 2024-09-16T01:36:03.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.842+0000 7f3ab8df1640 1 --2- 172.21.15.88:0/2287212007 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ab40d29a0 0x7f3ab40d2e50 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.842+0000 7f3ab8df1640 1 --2- 172.21.15.88:0/2287212007 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ab40b3920 0x7f3ab4162b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.842+0000 7f3ab8df1640 1 -- 172.21.15.88:0/2287212007 >> 172.21.15.88:0/2287212007 conn(0x7f3ab40d1200 msgr2=0x7f3ab40d1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:03.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.842+0000 7f3ab8df1640 1 -- 172.21.15.88:0/2287212007 shutdown_connections 2024-09-16T01:36:03.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.842+0000 7f3ab8df1640 1 -- 172.21.15.88:0/2287212007 wait complete. 2024-09-16T01:36:03.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.844+0000 7f3ab8df1640 1 Processor -- start 2024-09-16T01:36:03.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.844+0000 7f3ab8df1640 1 -- start start 2024-09-16T01:36:03.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.844+0000 7f3ab8df1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ab40b3920 0x7f3ab40b30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:03.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.844+0000 7f3ab8df1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ab40d29a0 0x7f3ab40cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:03.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.844+0000 7f3ab8df1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ab415e990 0x7f3ab40ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:03.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.844+0000 7f3ab8df1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab41630a0 con 0x7f3ab415e990 2024-09-16T01:36:03.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.844+0000 7f3ab8df1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab40cfc40 con 0x7f3ab40d29a0 2024-09-16T01:36:03.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.844+0000 7f3ab8df1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3ab4163220 con 0x7f3ab40b3920 2024-09-16T01:36:03.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.845+0000 7f3ab2d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ab415e990 0x7f3ab40ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:03.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.845+0000 7f3ab2d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ab415e990 0x7f3ab40ccde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50944/0 (socket says 172.21.15.88:50944) 2024-09-16T01:36:03.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.845+0000 7f3ab2d9d640 1 -- 172.21.15.88:0/2551954333 learned_addr learned my addr 172.21.15.88:0/2551954333 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:03.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.845+0000 7f3ab259c640 1 --2- 172.21.15.88:0/2551954333 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ab40b3920 0x7f3ab40b30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:03.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.845+0000 7f3ab1d9b640 1 --2- 172.21.15.88:0/2551954333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ab40d29a0 0x7f3ab40cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:03.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.845+0000 7f3ab2d9d640 1 -- 172.21.15.88:0/2551954333 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ab40b3920 msgr2=0x7f3ab40b30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:03.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.845+0000 7f3ab2d9d640 1 --2- 172.21.15.88:0/2551954333 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ab40b3920 0x7f3ab40b30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.845+0000 7f3ab2d9d640 1 -- 172.21.15.88:0/2551954333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ab40d29a0 msgr2=0x7f3ab40cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:03.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.845+0000 7f3ab2d9d640 1 --2- 172.21.15.88:0/2551954333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ab40d29a0 0x7f3ab40cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:03.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.845+0000 7f3ab2d9d640 1 -- 172.21.15.88:0/2551954333 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ab40b3620 con 0x7f3ab415e990 2024-09-16T01:36:03.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.845+0000 7f3ab1d9b640 1 --2- 172.21.15.88:0/2551954333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ab40d29a0 0x7f3ab40cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:03.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.845+0000 7f3ab2d9d640 1 --2- 172.21.15.88:0/2551954333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ab415e990 0x7f3ab40ccde0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3aac009e80 tx=0x7f3aac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:03.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.845+0000 7f3aa2ffd640 1 -- 172.21.15.88:0/2551954333 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aac03f070 con 0x7f3ab415e990 2024-09-16T01:36:03.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.845+0000 7f3aa2ffd640 1 -- 172.21.15.88:0/2551954333 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3aac030b90 con 0x7f3ab415e990 2024-09-16T01:36:03.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.845+0000 7f3ab8df1640 1 -- 172.21.15.88:0/2551954333 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ab40cb5c0 con 0x7f3ab415e990 2024-09-16T01:36:03.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.845+0000 7f3aa2ffd640 1 -- 172.21.15.88:0/2551954333 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aac009450 con 0x7f3ab415e990 2024-09-16T01:36:03.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.845+0000 7f3ab8df1640 1 -- 172.21.15.88:0/2551954333 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ab414dde0 con 0x7f3ab415e990 2024-09-16T01:36:03.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.846+0000 7f3ab8df1640 1 -- 172.21.15.88:0/2551954333 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ab4162450 con 0x7f3ab415e990 2024-09-16T01:36:03.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.847+0000 7f3aa2ffd640 1 -- 172.21.15.88:0/2551954333 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3aac03b410 con 0x7f3ab415e990 2024-09-16T01:36:03.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.847+0000 7f3aa2ffd640 1 --2- 172.21.15.88:0/2551954333 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a9406c2b0 0x7f3a9406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:03.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.847+0000 7f3aa2ffd640 1 -- 172.21.15.88:0/2551954333 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4857+0+0 (secure 0 0 0) 0x7f3aac0ac700 con 0x7f3ab415e990 2024-09-16T01:36:03.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.847+0000 7f3ab259c640 1 --2- 172.21.15.88:0/2551954333 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a9406c2b0 0x7f3a9406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:03.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.848+0000 7f3ab259c640 1 --2- 172.21.15.88:0/2551954333 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a9406c2b0 0x7f3a9406e770 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4000f60 tx=0x7f3aa4007c00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:03.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:03.851+0000 7f3aa2ffd640 1 -- 172.21.15.88:0/2551954333 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3aac07b130 con 0x7f3ab415e990 2024-09-16T01:36:04.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.016+0000 7f3ab8df1640 1 -- 172.21.15.88:0/2551954333 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f3ab41605b0 con 0x7f3ab415e990 2024-09-16T01:36:04.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.341+0000 7f3aa2ffd640 1 -- 172.21.15.88:0/2551954333 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v42) ==== 138+0+0 (secure 0 0 0) 0x7f3aac07aa30 con 0x7f3ab415e990 2024-09-16T01:36:04.342 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_metadata' created 2024-09-16T01:36:04.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.344+0000 7f3ab8df1640 1 -- 172.21.15.88:0/2551954333 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a9406c2b0 msgr2=0x7f3a9406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:04.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.344+0000 7f3ab8df1640 1 --2- 172.21.15.88:0/2551954333 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a9406c2b0 0x7f3a9406e770 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4000f60 tx=0x7f3aa4007c00 comp rx=0 tx=0).stop 2024-09-16T01:36:04.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.344+0000 7f3ab8df1640 1 -- 172.21.15.88:0/2551954333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ab415e990 msgr2=0x7f3ab40ccde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:04.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.344+0000 7f3ab8df1640 1 --2- 172.21.15.88:0/2551954333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ab415e990 0x7f3ab40ccde0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3aac009e80 tx=0x7f3aac004a40 comp rx=0 tx=0).stop 2024-09-16T01:36:04.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.344+0000 7f3ab8df1640 1 -- 172.21.15.88:0/2551954333 shutdown_connections 2024-09-16T01:36:04.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.344+0000 7f3ab8df1640 1 --2- 172.21.15.88:0/2551954333 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a9406c2b0 0x7f3a9406e770 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:04.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.344+0000 7f3ab8df1640 1 --2- 172.21.15.88:0/2551954333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ab415e990 0x7f3ab40ccde0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:04.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.344+0000 7f3ab8df1640 1 --2- 172.21.15.88:0/2551954333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ab40d29a0 0x7f3ab40cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:04.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.344+0000 7f3ab8df1640 1 --2- 172.21.15.88:0/2551954333 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ab40b3920 0x7f3ab40b30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:04.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.344+0000 7f3ab8df1640 1 -- 172.21.15.88:0/2551954333 >> 172.21.15.88:0/2551954333 conn(0x7f3ab40d1200 msgr2=0x7f3ab415fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:04.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.344+0000 7f3ab8df1640 1 -- 172.21.15.88:0/2551954333 shutdown_connections 2024-09-16T01:36:04.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.344+0000 7f3ab8df1640 1 -- 172.21.15.88:0/2551954333 wait complete. 2024-09-16T01:36:04.359 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-16T01:36:04.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.463+0000 7f2d56d28640 1 Processor -- start 2024-09-16T01:36:04.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.463+0000 7f2d56d28640 1 -- start start 2024-09-16T01:36:04.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.464+0000 7f2d56d28640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2d500b54d0 0x7f2d5006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:04.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.464+0000 7f2d56d28640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d5006e990 con 0x7f2d500b7450 2024-09-16T01:36:04.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.464+0000 7f2d56d28640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d5006ec70 con 0x7f2d5015e6f0 2024-09-16T01:36:04.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.464+0000 7f2d56d28640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2d5015ec70 con 0x7f2d500b54d0 2024-09-16T01:36:04.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.464+0000 7f2d552c5640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f2d500b7450 0x7f2d50070830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59350/0 (socket says 172.21.15.88:59350) 2024-09-16T01:36:04.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.464+0000 7f2d552c5640 1 -- 172.21.15.88:0/4264542748 learned_addr learned my addr 172.21.15.88:0/4264542748 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:04.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.464+0000 7f2d4ffff640 1 --2- 172.21.15.88:0/4264542748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2d500b54d0 0x7f2d5006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:04.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.464+0000 7f2d4f7fe640 1 -- 172.21.15.88:0/4264542748 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1149664115 0 0) 0x7f2d5006e990 con 0x7f2d500b7450 2024-09-16T01:36:04.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.464+0000 7f2d4f7fe640 1 -- 172.21.15.88:0/4264542748 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d3c003520 con 0x7f2d500b7450 2024-09-16T01:36:04.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.465+0000 7f2d4f7fe640 1 -- 172.21.15.88:0/4264542748 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2569670445 0 0) 0x7f2d3c003520 con 0x7f2d500b7450 2024-09-16T01:36:04.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.465+0000 7f2d4f7fe640 1 -- 172.21.15.88:0/4264542748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2d500b54d0 msgr2=0x7f2d5006fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:04.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.465+0000 7f2d4f7fe640 1 --2- 172.21.15.88:0/4264542748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2d500b54d0 0x7f2d5006fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:04.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.465+0000 7f2d4f7fe640 1 -- 172.21.15.88:0/4264542748 >> v1:172.21.15.154:6789/0 conn(0x7f2d5015e6f0 legacy=0x7f2d500b6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:04.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.465+0000 7f2d4f7fe640 1 -- 172.21.15.88:0/4264542748 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d5015ef50 con 0x7f2d500b7450 2024-09-16T01:36:04.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.465+0000 7f2d4f7fe640 1 -- 172.21.15.88:0/4264542748 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f2d480019f0 con 0x7f2d500b7450 2024-09-16T01:36:04.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.465+0000 7f2d4f7fe640 1 -- 172.21.15.88:0/4264542748 >> v1:172.21.15.88:6789/0 conn(0x7f2d500b7450 legacy=0x7f2d50070830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:04.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.465+0000 7f2d4f7fe640 1 --2- 172.21.15.88:0/4264542748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2d500b54d0 0x7f2d3c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:04.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.465+0000 7f2d4f7fe640 1 --2- 172.21.15.88:0/4264542748 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d3c003f20 0x7f2d3c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:04.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.465+0000 7f2d4f7fe640 1 --2- 172.21.15.88:0/4264542748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d3c004d40 0x7f2d3c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:04.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.465+0000 7f2d4f7fe640 1 -- 172.21.15.88:0/4264542748 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5015ec70 con 0x7f2d3c003f20 2024-09-16T01:36:04.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.465+0000 7f2d4f7fe640 1 -- 172.21.15.88:0/4264542748 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2d3c00ba20 con 0x7f2d3c004d40 2024-09-16T01:36:04.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.465+0000 7f2d4f7fe640 1 -- 172.21.15.88:0/4264542748 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2d3c00bc20 con 0x7f2d500b54d0 2024-09-16T01:36:04.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.465+0000 7f2d4ffff640 1 --2- 172.21.15.88:0/4264542748 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d3c003f20 0x7f2d3c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:04.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.465+0000 7f2d54ac4640 1 --2- 172.21.15.88:0/4264542748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2d500b54d0 0x7f2d3c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:04.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.465+0000 7f2d552c5640 1 --2- 172.21.15.88:0/4264542748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d3c004d40 0x7f2d3c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:04.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.465+0000 7f2d4ffff640 1 -- 172.21.15.88:0/4264542748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2d500b54d0 msgr2=0x7f2d3c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:04.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.465+0000 7f2d4ffff640 1 --2- 172.21.15.88:0/4264542748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2d500b54d0 0x7f2d3c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:04.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.465+0000 7f2d4ffff640 1 -- 172.21.15.88:0/4264542748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d3c004d40 msgr2=0x7f2d3c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:04.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.465+0000 7f2d4ffff640 1 --2- 172.21.15.88:0/4264542748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d3c004d40 0x7f2d3c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:04.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.465+0000 7f2d4ffff640 1 -- 172.21.15.88:0/4264542748 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d3c003520 con 0x7f2d3c003f20 2024-09-16T01:36:04.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.465+0000 7f2d552c5640 1 --2- 172.21.15.88:0/4264542748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d3c004d40 0x7f2d3c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:04.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.466+0000 7f2d4ffff640 1 --2- 172.21.15.88:0/4264542748 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d3c003f20 0x7f2d3c004800 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f2d38009a90 tx=0x7f2d3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bc966ccf7d3f298 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:04.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.466+0000 7f2d4f7fe640 1 -- 172.21.15.88:0/4264542748 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d38036070 con 0x7f2d3c003f20 2024-09-16T01:36:04.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.466+0000 7f2d4f7fe640 1 -- 172.21.15.88:0/4264542748 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d38031400 con 0x7f2d3c003f20 2024-09-16T01:36:04.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.466+0000 7f2d4f7fe640 1 -- 172.21.15.88:0/4264542748 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d38009690 con 0x7f2d3c003f20 2024-09-16T01:36:04.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.467+0000 7f2d56d28640 1 -- 172.21.15.88:0/4264542748 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d3c003f20 msgr2=0x7f2d3c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:04.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.467+0000 7f2d56d28640 1 --2- 172.21.15.88:0/4264542748 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d3c003f20 0x7f2d3c004800 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f2d38009a90 tx=0x7f2d3c005830 comp rx=0 tx=0).stop 2024-09-16T01:36:04.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.467+0000 7f2d552c5640 1 -- 172.21.15.88:0/4264542748 reap_dead start 2024-09-16T01:36:04.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.467+0000 7f2d56d28640 1 -- 172.21.15.88:0/4264542748 shutdown_connections 2024-09-16T01:36:04.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.467+0000 7f2d56d28640 1 -- 172.21.15.88:0/4264542748 >> 172.21.15.88:0/4264542748 conn(0x7f2d500d2430 msgr2=0x7f2d500d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:04.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.467+0000 7f2d56d28640 1 -- 172.21.15.88:0/4264542748 shutdown_connections 2024-09-16T01:36:04.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.467+0000 7f2d56d28640 1 -- 172.21.15.88:0/4264542748 wait complete. 2024-09-16T01:36:04.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.468+0000 7f2d56d28640 1 Processor -- start 2024-09-16T01:36:04.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.468+0000 7f2d56d28640 1 -- start start 2024-09-16T01:36:04.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.469+0000 7f2d56d28640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d3c003f20 0x7f2d501f5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:04.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.469+0000 7f2d56d28640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d5015a020 0x7f2d500ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:04.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.469+0000 7f2d56d28640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2d500cf1e0 0x7f2d500d0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:04.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.469+0000 7f2d56d28640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5006e730 con 0x7f2d5015a020 2024-09-16T01:36:04.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.469+0000 7f2d56d28640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5006e5b0 con 0x7f2d3c003f20 2024-09-16T01:36:04.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.469+0000 7f2d56d28640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2d5006fd00 con 0x7f2d500cf1e0 2024-09-16T01:36:04.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.469+0000 7f2d4ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d5015a020 0x7f2d500ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:04.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.469+0000 7f2d4ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d5015a020 0x7f2d500ceca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50958/0 (socket says 172.21.15.88:50958) 2024-09-16T01:36:04.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.469+0000 7f2d4ffff640 1 -- 172.21.15.88:0/1145839493 learned_addr learned my addr 172.21.15.88:0/1145839493 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:04.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.469+0000 7f2d54ac4640 1 --2- 172.21.15.88:0/1145839493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d3c003f20 0x7f2d501f5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:04.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.469+0000 7f2d552c5640 1 --2- 172.21.15.88:0/1145839493 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2d500cf1e0 0x7f2d500d0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:04.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.469+0000 7f2d4ffff640 1 -- 172.21.15.88:0/1145839493 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2d500cf1e0 msgr2=0x7f2d500d0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:04.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.469+0000 7f2d4ffff640 1 --2- 172.21.15.88:0/1145839493 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2d500cf1e0 0x7f2d500d0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:04.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.469+0000 7f2d4ffff640 1 -- 172.21.15.88:0/1145839493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d3c003f20 msgr2=0x7f2d501f5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:04.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.469+0000 7f2d4ffff640 1 --2- 172.21.15.88:0/1145839493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d3c003f20 0x7f2d501f5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:04.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.469+0000 7f2d4ffff640 1 -- 172.21.15.88:0/1145839493 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d500cb9f0 con 0x7f2d5015a020 2024-09-16T01:36:04.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.469+0000 7f2d552c5640 1 --2- 172.21.15.88:0/1145839493 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2d500cf1e0 0x7f2d500d0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:04.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.469+0000 7f2d4ffff640 1 --2- 172.21.15.88:0/1145839493 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d5015a020 0x7f2d500ceca0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f2d38009a90 tx=0x7f2d38004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:04.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.469+0000 7f2d4d7fa640 1 -- 172.21.15.88:0/1145839493 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d38040020 con 0x7f2d5015a020 2024-09-16T01:36:04.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.469+0000 7f2d4d7fa640 1 -- 172.21.15.88:0/1145839493 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d3803b660 con 0x7f2d5015a020 2024-09-16T01:36:04.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.470+0000 7f2d56d28640 1 -- 172.21.15.88:0/1145839493 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d500cbcd0 con 0x7f2d5015a020 2024-09-16T01:36:04.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.470+0000 7f2d4d7fa640 1 -- 172.21.15.88:0/1145839493 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d38036040 con 0x7f2d5015a020 2024-09-16T01:36:04.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.470+0000 7f2d56d28640 1 -- 172.21.15.88:0/1145839493 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d500cc290 con 0x7f2d5015a020 2024-09-16T01:36:04.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.470+0000 7f2d56d28640 1 -- 172.21.15.88:0/1145839493 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d5006c6c0 con 0x7f2d5015a020 2024-09-16T01:36:04.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.471+0000 7f2d4d7fa640 1 -- 172.21.15.88:0/1145839493 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2d38043020 con 0x7f2d5015a020 2024-09-16T01:36:04.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.472+0000 7f2d4d7fa640 1 --2- 172.21.15.88:0/1145839493 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2d3006c2e0 0x7f2d3006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:04.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.472+0000 7f2d54ac4640 1 --2- 172.21.15.88:0/1145839493 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2d3006c2e0 0x7f2d3006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:04.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.472+0000 7f2d4d7fa640 1 -- 172.21.15.88:0/1145839493 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5227+0+0 (secure 0 0 0) 0x7f2d380aba60 con 0x7f2d5015a020 2024-09-16T01:36:04.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.472+0000 7f2d54ac4640 1 --2- 172.21.15.88:0/1145839493 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2d3006c2e0 0x7f2d3006e7a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2d400019c0 tx=0x7f2d40009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:04.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.476+0000 7f2d4d7fa640 1 -- 172.21.15.88:0/1145839493 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2d3804a030 con 0x7f2d5015a020 2024-09-16T01:36:04.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:04.644+0000 7f2d56d28640 1 -- 172.21.15.88:0/1145839493 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f2d501f5670 con 0x7f2d5015a020 2024-09-16T01:36:05.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.346+0000 7f2d4d7fa640 1 -- 172.21.15.88:0/1145839493 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v43) ==== 170+0+0 (secure 0 0 0) 0x7f2d3807a3b0 con 0x7f2d5015a020 2024-09-16T01:36:05.348 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_data' created 2024-09-16T01:36:05.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.349+0000 7f2d56d28640 1 -- 172.21.15.88:0/1145839493 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2d3006c2e0 msgr2=0x7f2d3006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:05.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.349+0000 7f2d56d28640 1 --2- 172.21.15.88:0/1145839493 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2d3006c2e0 0x7f2d3006e7a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2d400019c0 tx=0x7f2d40009040 comp rx=0 tx=0).stop 2024-09-16T01:36:05.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.349+0000 7f2d56d28640 1 -- 172.21.15.88:0/1145839493 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d5015a020 msgr2=0x7f2d500ceca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:05.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.349+0000 7f2d56d28640 1 --2- 172.21.15.88:0/1145839493 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d5015a020 0x7f2d500ceca0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f2d38009a90 tx=0x7f2d38004730 comp rx=0 tx=0).stop 2024-09-16T01:36:05.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.349+0000 7f2d56d28640 1 -- 172.21.15.88:0/1145839493 shutdown_connections 2024-09-16T01:36:05.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.349+0000 7f2d56d28640 1 --2- 172.21.15.88:0/1145839493 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2d3006c2e0 0x7f2d3006e7a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:05.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.349+0000 7f2d56d28640 1 --2- 172.21.15.88:0/1145839493 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2d500cf1e0 0x7f2d500d0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:05.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.349+0000 7f2d56d28640 1 --2- 172.21.15.88:0/1145839493 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d5015a020 0x7f2d500ceca0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:05.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.349+0000 7f2d56d28640 1 --2- 172.21.15.88:0/1145839493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d3c003f20 0x7f2d501f5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:05.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.349+0000 7f2d56d28640 1 -- 172.21.15.88:0/1145839493 >> 172.21.15.88:0/1145839493 conn(0x7f2d500d2430 msgr2=0x7f2d5015ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:05.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.349+0000 7f2d56d28640 1 -- 172.21.15.88:0/1145839493 shutdown_connections 2024-09-16T01:36:05.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.349+0000 7f2d56d28640 1 -- 172.21.15.88:0/1145839493 wait complete. 2024-09-16T01:36:05.374 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-16T01:36:05.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.476+0000 7f507a978640 1 Processor -- start 2024-09-16T01:36:05.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.476+0000 7f507a978640 1 -- start start 2024-09-16T01:36:05.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.477+0000 7f507a978640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f50740b64a0 0x7f50740b6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:05.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.477+0000 7f507a978640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50740b7790 con 0x7f507415e750 2024-09-16T01:36:05.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.477+0000 7f507a978640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50740d3600 con 0x7f50740b7f50 2024-09-16T01:36:05.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.477+0000 7f507a978640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f50740b6e50 con 0x7f50740b64a0 2024-09-16T01:36:05.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.477+0000 7f5073fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f507415e750 0x7f50740cf630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59366/0 (socket says 172.21.15.88:59366) 2024-09-16T01:36:05.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.477+0000 7f5073fff640 1 -- 172.21.15.88:0/2590956548 learned_addr learned my addr 172.21.15.88:0/2590956548 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:05.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.477+0000 7f50737fe640 1 --2- 172.21.15.88:0/2590956548 >> v2:172.21.15.88:3300/0 conn(0x7f50740b64a0 0x7f50740b6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:05.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.478+0000 7f50737fe640 1 -- 172.21.15.88:0/2590956548 >> v1:172.21.15.154:6789/0 conn(0x7f50740b7f50 legacy=0x7f50740b8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:05.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.478+0000 7f50737fe640 1 -- 172.21.15.88:0/2590956548 >> v1:172.21.15.88:6789/0 conn(0x7f507415e750 legacy=0x7f50740cf630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:05.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.478+0000 7f5072ffd640 1 -- 172.21.15.88:0/2590956548 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2297750501 0 0) 0x7f50740b7790 con 0x7f507415e750 2024-09-16T01:36:05.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.478+0000 7f50737fe640 1 -- 172.21.15.88:0/2590956548 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50740d38e0 con 0x7f50740b64a0 2024-09-16T01:36:05.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.478+0000 7f50737fe640 1 --2- 172.21.15.88:0/2590956548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50740b64a0 0x7f50740b6910 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f50680097a0 tx=0x7f506802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a20b01ec79eee4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:05.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.479+0000 7f5072ffd640 1 -- 172.21.15.88:0/2590956548 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5068037070 con 0x7f50740b64a0 2024-09-16T01:36:05.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.479+0000 7f5072ffd640 1 -- 172.21.15.88:0/2590956548 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5068032440 con 0x7f50740b64a0 2024-09-16T01:36:05.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.479+0000 7f5072ffd640 1 -- 172.21.15.88:0/2590956548 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5068032810 con 0x7f50740b64a0 2024-09-16T01:36:05.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.479+0000 7f507a978640 1 -- 172.21.15.88:0/2590956548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50740b64a0 msgr2=0x7f50740b6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:05.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.479+0000 7f507a978640 1 --2- 172.21.15.88:0/2590956548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50740b64a0 0x7f50740b6910 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f50680097a0 tx=0x7f506802a1e0 comp rx=0 tx=0).stop 2024-09-16T01:36:05.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.479+0000 7f507a978640 1 -- 172.21.15.88:0/2590956548 shutdown_connections 2024-09-16T01:36:05.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.479+0000 7f507a978640 1 --2- 172.21.15.88:0/2590956548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50740b64a0 0x7f50740b6910 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:05.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.479+0000 7f507a978640 1 -- 172.21.15.88:0/2590956548 >> 172.21.15.88:0/2590956548 conn(0x7f50740d1500 msgr2=0x7f50740d1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:05.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.479+0000 7f507a978640 1 -- 172.21.15.88:0/2590956548 shutdown_connections 2024-09-16T01:36:05.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.479+0000 7f507a978640 1 -- 172.21.15.88:0/2590956548 wait complete. 2024-09-16T01:36:05.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.480+0000 7f507a978640 1 Processor -- start 2024-09-16T01:36:05.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.481+0000 7f507a978640 1 -- start start 2024-09-16T01:36:05.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.481+0000 7f507a978640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50740b64a0 0x7f50740cab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:05.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.481+0000 7f507a978640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50740b7f50 0x7f50740cd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:05.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.481+0000 7f507a978640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507415e750 0x7f50740cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:05.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.481+0000 7f507a978640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f50740d33b0 con 0x7f50740b7f50 2024-09-16T01:36:05.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.481+0000 7f507a978640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f50740d43c0 con 0x7f50740b64a0 2024-09-16T01:36:05.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.481+0000 7f507a978640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f50740b7ac0 con 0x7f507415e750 2024-09-16T01:36:05.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.481+0000 7f50737fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50740b7f50 0x7f50740cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:05.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.481+0000 7f50737fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50740b7f50 0x7f50740cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50972/0 (socket says 172.21.15.88:50972) 2024-09-16T01:36:05.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.481+0000 7f50737fe640 1 -- 172.21.15.88:0/4151542295 learned_addr learned my addr 172.21.15.88:0/4151542295 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:05.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.482+0000 7f5073fff640 1 --2- 172.21.15.88:0/4151542295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50740b64a0 0x7f50740cab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:05.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.482+0000 7f5078f15640 1 --2- 172.21.15.88:0/4151542295 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507415e750 0x7f50740cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:05.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.482+0000 7f50737fe640 1 -- 172.21.15.88:0/4151542295 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507415e750 msgr2=0x7f50740cd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:05.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.482+0000 7f50737fe640 1 --2- 172.21.15.88:0/4151542295 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507415e750 0x7f50740cd9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:05.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.482+0000 7f50737fe640 1 -- 172.21.15.88:0/4151542295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50740b64a0 msgr2=0x7f50740cab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:05.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.482+0000 7f50737fe640 1 --2- 172.21.15.88:0/4151542295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50740b64a0 0x7f50740cab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:05.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.482+0000 7f50737fe640 1 -- 172.21.15.88:0/4151542295 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5074159090 con 0x7f50740b7f50 2024-09-16T01:36:05.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.482+0000 7f5078f15640 1 --2- 172.21.15.88:0/4151542295 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507415e750 0x7f50740cd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:05.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.482+0000 7f50737fe640 1 --2- 172.21.15.88:0/4151542295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50740b7f50 0x7f50740cd480 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f50680097a0 tx=0x7f50680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:05.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.482+0000 7f5070ff9640 1 -- 172.21.15.88:0/4151542295 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5068037040 con 0x7f50740b7f50 2024-09-16T01:36:05.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.482+0000 7f5070ff9640 1 -- 172.21.15.88:0/4151542295 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5068004250 con 0x7f50740b7f50 2024-09-16T01:36:05.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.482+0000 7f507a978640 1 -- 172.21.15.88:0/4151542295 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5074159370 con 0x7f50740b7f50 2024-09-16T01:36:05.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.482+0000 7f5070ff9640 1 -- 172.21.15.88:0/4151542295 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5068032e20 con 0x7f50740b7f50 2024-09-16T01:36:05.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.482+0000 7f507a978640 1 -- 172.21.15.88:0/4151542295 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5074159930 con 0x7f50740b7f50 2024-09-16T01:36:05.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.483+0000 7f507a978640 1 -- 172.21.15.88:0/4151542295 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50740d0bc0 con 0x7f50740b7f50 2024-09-16T01:36:05.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.484+0000 7f5070ff9640 1 -- 172.21.15.88:0/4151542295 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5068030770 con 0x7f50740b7f50 2024-09-16T01:36:05.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.484+0000 7f5070ff9640 1 --2- 172.21.15.88:0/4151542295 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f506006c2b0 0x7f506006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:05.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.484+0000 7f5073fff640 1 --2- 172.21.15.88:0/4151542295 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f506006c2b0 0x7f506006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:05.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.484+0000 7f5070ff9640 1 -- 172.21.15.88:0/4151542295 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5609+0+0 (secure 0 0 0) 0x7f50680ac660 con 0x7f50740b7f50 2024-09-16T01:36:05.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.485+0000 7f5073fff640 1 --2- 172.21.15.88:0/4151542295 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f506006c2b0 0x7f506006e770 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5064000f60 tx=0x7f5064007c00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:05.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.489+0000 7f5070ff9640 1 -- 172.21.15.88:0/4151542295 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f50740d0bc0 con 0x7f50740b7f50 2024-09-16T01:36:05.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:05.690+0000 7f507a978640 1 -- 172.21.15.88:0/4151542295 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f50740b6910 con 0x7f50740b7f50 2024-09-16T01:36:06.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.348+0000 7f5070ff9640 1 -- 172.21.15.88:0/4151542295 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v23) ==== 172+0+0 (secure 0 0 0) 0x7f506807fe20 con 0x7f50740b7f50 2024-09-16T01:36:06.349 INFO:teuthology.orchestra.run.smithi088.stderr:new fs with metadata pool 6 and data pool 7 2024-09-16T01:36:06.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.352+0000 7f507a978640 1 -- 172.21.15.88:0/4151542295 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f506006c2b0 msgr2=0x7f506006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:06.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.352+0000 7f507a978640 1 --2- 172.21.15.88:0/4151542295 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f506006c2b0 0x7f506006e770 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5064000f60 tx=0x7f5064007c00 comp rx=0 tx=0).stop 2024-09-16T01:36:06.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.352+0000 7f507a978640 1 -- 172.21.15.88:0/4151542295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50740b7f50 msgr2=0x7f50740cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:06.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.352+0000 7f507a978640 1 --2- 172.21.15.88:0/4151542295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50740b7f50 0x7f50740cd480 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f50680097a0 tx=0x7f50680046c0 comp rx=0 tx=0).stop 2024-09-16T01:36:06.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.352+0000 7f507a978640 1 -- 172.21.15.88:0/4151542295 shutdown_connections 2024-09-16T01:36:06.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.352+0000 7f507a978640 1 --2- 172.21.15.88:0/4151542295 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f506006c2b0 0x7f506006e770 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:06.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.352+0000 7f507a978640 1 --2- 172.21.15.88:0/4151542295 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507415e750 0x7f50740cd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:06.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.352+0000 7f507a978640 1 --2- 172.21.15.88:0/4151542295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50740b7f50 0x7f50740cd480 secure :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f50680097a0 tx=0x7f50680046c0 comp rx=0 tx=0).stop 2024-09-16T01:36:06.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.352+0000 7f507a978640 1 --2- 172.21.15.88:0/4151542295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50740b64a0 0x7f50740cab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:06.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.352+0000 7f507a978640 1 -- 172.21.15.88:0/4151542295 >> 172.21.15.88:0/4151542295 conn(0x7f50740d1500 msgr2=0x7f50740cec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:06.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.355+0000 7f507a978640 1 -- 172.21.15.88:0/4151542295 shutdown_connections 2024-09-16T01:36:06.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.357+0000 7f507a978640 1 -- 172.21.15.88:0/4151542295 wait complete. 2024-09-16T01:36:06.382 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:36:06.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.479+0000 7f3affe1e640 1 Processor -- start 2024-09-16T01:36:06.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.479+0000 7f3affe1e640 1 -- start start 2024-09-16T01:36:06.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.480+0000 7f3affe1e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3af8155800 0x7f3af80b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:06.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.480+0000 7f3affe1e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3af815c330 0x7f3af80b4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:06.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.480+0000 7f3affe1e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3af80b50a0 con 0x7f3af80cf570 2024-09-16T01:36:06.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.480+0000 7f3affe1e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3af80b4e60 con 0x7f3af8155800 2024-09-16T01:36:06.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.480+0000 7f3affe1e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3af815c7a0 con 0x7f3af815c330 2024-09-16T01:36:06.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.480+0000 7f3afd3b9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3af815c330 0x7f3af80b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:06.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.480+0000 7f3afdbba640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3af8155800 0x7f3af80b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:06.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.480+0000 7f3afe3bb640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f3af80cf570 0x7f3af815c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:58660/0 (socket says 172.21.15.88:58660) 2024-09-16T01:36:06.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.480+0000 7f3afe3bb640 1 -- 172.21.15.88:0/3172546051 learned_addr learned my addr 172.21.15.88:0/3172546051 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:06.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.481+0000 7f3afd3b9640 1 -- 172.21.15.88:0/3172546051 >> v2:172.21.15.154:3300/0 conn(0x7f3af8155800 msgr2=0x7f3af80b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:06.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.481+0000 7f3afd3b9640 1 --2- 172.21.15.88:0/3172546051 >> v2:172.21.15.154:3300/0 conn(0x7f3af8155800 0x7f3af80b6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:06.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.481+0000 7f3afd3b9640 1 -- 172.21.15.88:0/3172546051 >> v1:172.21.15.154:6789/0 conn(0x7f3af80cf570 legacy=0x7f3af815c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:06.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.481+0000 7f3afd3b9640 1 -- 172.21.15.88:0/3172546051 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3af80b5380 con 0x7f3af815c330 2024-09-16T01:36:06.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.481+0000 7f3afdbba640 1 --2- 172.21.15.88:0/3172546051 >> v2:172.21.15.154:3300/0 conn(0x7f3af8155800 0x7f3af80b6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:36:06.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.481+0000 7f3afd3b9640 1 --2- 172.21.15.88:0/3172546051 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3af815c330 0x7f3af80b4920 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3ae80097a0 tx=0x7f3ae802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18399e117c8b000f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:06.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.482+0000 7f3afcbb8640 1 -- 172.21.15.88:0/3172546051 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae8037070 con 0x7f3af815c330 2024-09-16T01:36:06.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.482+0000 7f3afcbb8640 1 -- 172.21.15.88:0/3172546051 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ae8032440 con 0x7f3af815c330 2024-09-16T01:36:06.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.482+0000 7f3afcbb8640 1 -- 172.21.15.88:0/3172546051 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae8032810 con 0x7f3af815c330 2024-09-16T01:36:06.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.482+0000 7f3affe1e640 1 -- 172.21.15.88:0/3172546051 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3af815c330 msgr2=0x7f3af80b4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:06.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.482+0000 7f3affe1e640 1 --2- 172.21.15.88:0/3172546051 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3af815c330 0x7f3af80b4920 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3ae80097a0 tx=0x7f3ae802a180 comp rx=0 tx=0).stop 2024-09-16T01:36:06.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.482+0000 7f3affe1e640 1 -- 172.21.15.88:0/3172546051 shutdown_connections 2024-09-16T01:36:06.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.482+0000 7f3affe1e640 1 --2- 172.21.15.88:0/3172546051 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3af815c330 0x7f3af80b4920 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:06.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.482+0000 7f3affe1e640 1 --2- 172.21.15.88:0/3172546051 >> v2:172.21.15.154:3300/0 conn(0x7f3af8155800 0x7f3af80b6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:06.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.482+0000 7f3affe1e640 1 -- 172.21.15.88:0/3172546051 >> 172.21.15.88:0/3172546051 conn(0x7f3af806bc10 msgr2=0x7f3af806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:06.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.482+0000 7f3affe1e640 1 -- 172.21.15.88:0/3172546051 shutdown_connections 2024-09-16T01:36:06.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.482+0000 7f3affe1e640 1 -- 172.21.15.88:0/3172546051 wait complete. 2024-09-16T01:36:06.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.483+0000 7f3affe1e640 1 Processor -- start 2024-09-16T01:36:06.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.483+0000 7f3affe1e640 1 -- start start 2024-09-16T01:36:06.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.484+0000 7f3affe1e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3af80cf570 0x7f3af8143ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:06.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.484+0000 7f3affe1e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3af8155800 0x7f3af814bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:06.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.484+0000 7f3affe1e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3af815c330 0x7f3af814c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:06.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.484+0000 7f3affe1e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3af80b5c20 con 0x7f3af8155800 2024-09-16T01:36:06.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.484+0000 7f3affe1e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3af80b5940 con 0x7f3af80cf570 2024-09-16T01:36:06.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.484+0000 7f3affe1e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3af8155be0 con 0x7f3af815c330 2024-09-16T01:36:06.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.484+0000 7f3afd3b9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3af8155800 0x7f3af814bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:06.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.484+0000 7f3afd3b9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3af8155800 0x7f3af814bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50980/0 (socket says 172.21.15.88:50980) 2024-09-16T01:36:06.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.484+0000 7f3afd3b9640 1 -- 172.21.15.88:0/1145091278 learned_addr learned my addr 172.21.15.88:0/1145091278 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:06.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.484+0000 7f3afdbba640 1 --2- 172.21.15.88:0/1145091278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3af80cf570 0x7f3af8143ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:06.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.484+0000 7f3afe3bb640 1 --2- 172.21.15.88:0/1145091278 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3af815c330 0x7f3af814c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:06.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.484+0000 7f3afd3b9640 1 -- 172.21.15.88:0/1145091278 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3af815c330 msgr2=0x7f3af814c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:06.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.484+0000 7f3afd3b9640 1 --2- 172.21.15.88:0/1145091278 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3af815c330 0x7f3af814c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:06.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.484+0000 7f3afd3b9640 1 -- 172.21.15.88:0/1145091278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3af80cf570 msgr2=0x7f3af8143ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:06.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.484+0000 7f3afd3b9640 1 --2- 172.21.15.88:0/1145091278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3af80cf570 0x7f3af8143ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:06.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.484+0000 7f3afd3b9640 1 -- 172.21.15.88:0/1145091278 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3af815a820 con 0x7f3af8155800 2024-09-16T01:36:06.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.485+0000 7f3afd3b9640 1 --2- 172.21.15.88:0/1145091278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3af8155800 0x7f3af814bdb0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f3ae802a1c0 tx=0x7f3ae80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:06.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.485+0000 7f3aee7fc640 1 -- 172.21.15.88:0/1145091278 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae8037040 con 0x7f3af8155800 2024-09-16T01:36:06.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.485+0000 7f3aee7fc640 1 -- 172.21.15.88:0/1145091278 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ae8042070 con 0x7f3af8155800 2024-09-16T01:36:06.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.485+0000 7f3affe1e640 1 -- 172.21.15.88:0/1145091278 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3af815ab00 con 0x7f3af8155800 2024-09-16T01:36:06.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.485+0000 7f3aee7fc640 1 -- 172.21.15.88:0/1145091278 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae8030920 con 0x7f3af8155800 2024-09-16T01:36:06.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.485+0000 7f3affe1e640 1 -- 172.21.15.88:0/1145091278 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3af815b0c0 con 0x7f3af8155800 2024-09-16T01:36:06.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.486+0000 7f3affe1e640 1 -- 172.21.15.88:0/1145091278 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3af80b6140 con 0x7f3af8155800 2024-09-16T01:36:06.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.486+0000 7f3aee7fc640 1 -- 172.21.15.88:0/1145091278 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3ae8030ac0 con 0x7f3af8155800 2024-09-16T01:36:06.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.487+0000 7f3aee7fc640 1 --2- 172.21.15.88:0/1145091278 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3adc06c2b0 0x7f3adc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:06.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.487+0000 7f3aee7fc640 1 -- 172.21.15.88:0/1145091278 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f3ae80abe40 con 0x7f3af8155800 2024-09-16T01:36:06.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.487+0000 7f3afdbba640 1 --2- 172.21.15.88:0/1145091278 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3adc06c2b0 0x7f3adc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:06.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.487+0000 7f3afdbba640 1 --2- 172.21.15.88:0/1145091278 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3adc06c2b0 0x7f3adc06e770 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3af815ec80 tx=0x7f3af4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:06.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.491+0000 7f3aee7fc640 1 -- 172.21.15.88:0/1145091278 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3ae807a650 con 0x7f3af8155800 2024-09-16T01:36:06.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.659+0000 7f3affe1e640 1 -- 172.21.15.88:0/1145091278 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3af8152aa0 con 0x7f3af8155800 2024-09-16T01:36:06.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.660+0000 7f3aee7fc640 1 -- 172.21.15.88:0/1145091278 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+16662 (secure 0 0 0) 0x7f3ae8097130 con 0x7f3af8155800 2024-09-16T01:36:06.661 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:06.662 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":44,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:36:05.691238+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:36:04.035328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-16T01:36:04.663046+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:36:06.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.662+0000 7f3affe1e640 1 -- 172.21.15.88:0/1145091278 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3adc06c2b0 msgr2=0x7f3adc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:06.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.662+0000 7f3affe1e640 1 --2- 172.21.15.88:0/1145091278 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3adc06c2b0 0x7f3adc06e770 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3af815ec80 tx=0x7f3af4008040 comp rx=0 tx=0).stop 2024-09-16T01:36:06.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.662+0000 7f3affe1e640 1 -- 172.21.15.88:0/1145091278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3af8155800 msgr2=0x7f3af814bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:06.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.662+0000 7f3affe1e640 1 --2- 172.21.15.88:0/1145091278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3af8155800 0x7f3af814bdb0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f3ae802a1c0 tx=0x7f3ae80046c0 comp rx=0 tx=0).stop 2024-09-16T01:36:06.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.662+0000 7f3affe1e640 1 -- 172.21.15.88:0/1145091278 shutdown_connections 2024-09-16T01:36:06.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.662+0000 7f3affe1e640 1 --2- 172.21.15.88:0/1145091278 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3adc06c2b0 0x7f3adc06e770 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:06.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.662+0000 7f3affe1e640 1 --2- 172.21.15.88:0/1145091278 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3af815c330 0x7f3af814c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:06.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.662+0000 7f3affe1e640 1 --2- 172.21.15.88:0/1145091278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3af8155800 0x7f3af814bdb0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:06.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.662+0000 7f3affe1e640 1 --2- 172.21.15.88:0/1145091278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3af80cf570 0x7f3af8143ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:06.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.662+0000 7f3affe1e640 1 -- 172.21.15.88:0/1145091278 >> 172.21.15.88:0/1145091278 conn(0x7f3af806bc10 msgr2=0x7f3af80ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:06.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.662+0000 7f3affe1e640 1 -- 172.21.15.88:0/1145091278 shutdown_connections 2024-09-16T01:36:06.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.662+0000 7f3affe1e640 1 -- 172.21.15.88:0/1145091278 wait complete. 2024-09-16T01:36:06.674 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:36:06.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.771+0000 7f6ad4653640 1 Processor -- start 2024-09-16T01:36:06.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.771+0000 7f6ad4653640 1 -- start start 2024-09-16T01:36:06.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.772+0000 7f6ad4653640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6acc15c550 0x7f6acc0b5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:06.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.772+0000 7f6ad4653640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6acc0b6310 0x7f6acc0b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:06.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.772+0000 7f6ad4653640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6acc0b4cf0 con 0x7f6acc15c170 2024-09-16T01:36:06.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.772+0000 7f6ad4653640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6acc0b4fd0 con 0x7f6acc15c550 2024-09-16T01:36:06.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.772+0000 7f6ad4653640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6acc0b52b0 con 0x7f6acc0b6310 2024-09-16T01:36:06.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.772+0000 7f6ad1bee640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6acc15c550 0x7f6acc0b5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:06.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.772+0000 7f6ad1bee640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6acc15c550 0x7f6acc0b5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50986/0 (socket says 172.21.15.88:50986) 2024-09-16T01:36:06.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.772+0000 7f6ad1bee640 1 -- 172.21.15.88:0/2534482492 learned_addr learned my addr 172.21.15.88:0/2534482492 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:06.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.772+0000 7f6ad2bf0640 1 --2- 172.21.15.88:0/2534482492 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6acc0b6310 0x7f6acc0b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:06.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.772+0000 7f6ad1bee640 1 -- 172.21.15.88:0/2534482492 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6acc0b6310 msgr2=0x7f6acc0b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:06.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.772+0000 7f6ad1bee640 1 --2- 172.21.15.88:0/2534482492 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6acc0b6310 0x7f6acc0b39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:06.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.772+0000 7f6ad1bee640 1 -- 172.21.15.88:0/2534482492 >> v1:172.21.15.154:6789/0 conn(0x7f6acc15c170 legacy=0x7f6acc0cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:06.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.772+0000 7f6ad1bee640 1 -- 172.21.15.88:0/2534482492 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6acc0b5590 con 0x7f6acc15c550 2024-09-16T01:36:06.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.773+0000 7f6ad1bee640 1 --2- 172.21.15.88:0/2534482492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6acc15c550 0x7f6acc0b5dd0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6ac00098a0 tx=0x7f6ac002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bc65df1696b10a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:06.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.773+0000 7f6ad13ed640 1 -- 172.21.15.88:0/2534482492 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac0037070 con 0x7f6acc15c550 2024-09-16T01:36:06.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.773+0000 7f6ad13ed640 1 -- 172.21.15.88:0/2534482492 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ac0032440 con 0x7f6acc15c550 2024-09-16T01:36:06.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.773+0000 7f6ad13ed640 1 -- 172.21.15.88:0/2534482492 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac0032810 con 0x7f6acc15c550 2024-09-16T01:36:06.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.774+0000 7f6ad4653640 1 -- 172.21.15.88:0/2534482492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6acc15c550 msgr2=0x7f6acc0b5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:06.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.774+0000 7f6ad4653640 1 --2- 172.21.15.88:0/2534482492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6acc15c550 0x7f6acc0b5dd0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6ac00098a0 tx=0x7f6ac002a250 comp rx=0 tx=0).stop 2024-09-16T01:36:06.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.774+0000 7f6ad4653640 1 -- 172.21.15.88:0/2534482492 shutdown_connections 2024-09-16T01:36:06.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.774+0000 7f6ad4653640 1 --2- 172.21.15.88:0/2534482492 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6acc0b6310 0x7f6acc0b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:06.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.774+0000 7f6ad4653640 1 --2- 172.21.15.88:0/2534482492 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6acc15c550 0x7f6acc0b5dd0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:06.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.774+0000 7f6ad4653640 1 -- 172.21.15.88:0/2534482492 >> 172.21.15.88:0/2534482492 conn(0x7f6acc06bc10 msgr2=0x7f6acc06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:06.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.774+0000 7f6ad4653640 1 -- 172.21.15.88:0/2534482492 shutdown_connections 2024-09-16T01:36:06.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.774+0000 7f6ad4653640 1 -- 172.21.15.88:0/2534482492 wait complete. 2024-09-16T01:36:06.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.775+0000 7f6ad4653640 1 Processor -- start 2024-09-16T01:36:06.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.775+0000 7f6ad4653640 1 -- start start 2024-09-16T01:36:06.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.776+0000 7f6ad4653640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6acc0b6310 0x7f6acc13dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:06.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.776+0000 7f6ad4653640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6acc15c170 0x7f6acc14f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:06.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.776+0000 7f6ad4653640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6acc15c550 0x7f6acc14f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:06.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.776+0000 7f6ad4653640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6acc15c9c0 con 0x7f6acc0b6310 2024-09-16T01:36:06.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.776+0000 7f6ad4653640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6acc0b5b50 con 0x7f6acc15c170 2024-09-16T01:36:06.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.776+0000 7f6ad4653640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6acc0cfdd0 con 0x7f6acc15c550 2024-09-16T01:36:06.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.776+0000 7f6ad23ef640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6acc0b6310 0x7f6acc13dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:06.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.776+0000 7f6ad23ef640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6acc0b6310 0x7f6acc13dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51000/0 (socket says 172.21.15.88:51000) 2024-09-16T01:36:06.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.776+0000 7f6ad23ef640 1 -- 172.21.15.88:0/3885146009 learned_addr learned my addr 172.21.15.88:0/3885146009 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:06.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.776+0000 7f6ad1bee640 1 --2- 172.21.15.88:0/3885146009 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6acc15c170 0x7f6acc14f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:06.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.776+0000 7f6ad2bf0640 1 --2- 172.21.15.88:0/3885146009 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6acc15c550 0x7f6acc14f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:06.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.776+0000 7f6ad23ef640 1 -- 172.21.15.88:0/3885146009 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6acc15c550 msgr2=0x7f6acc14f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:06.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.776+0000 7f6ad23ef640 1 --2- 172.21.15.88:0/3885146009 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6acc15c550 0x7f6acc14f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:06.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.776+0000 7f6ad23ef640 1 -- 172.21.15.88:0/3885146009 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6acc15c170 msgr2=0x7f6acc14f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:06.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.776+0000 7f6ad23ef640 1 --2- 172.21.15.88:0/3885146009 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6acc15c170 0x7f6acc14f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:06.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.776+0000 7f6ad23ef640 1 -- 172.21.15.88:0/3885146009 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6acc14e800 con 0x7f6acc0b6310 2024-09-16T01:36:06.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.776+0000 7f6ad2bf0640 1 --2- 172.21.15.88:0/3885146009 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6acc15c550 0x7f6acc14f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:06.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.777+0000 7f6ad23ef640 1 --2- 172.21.15.88:0/3885146009 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6acc0b6310 0x7f6acc13dce0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6abc00c7c0 tx=0x7f6abc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:06.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.777+0000 7f6acaffd640 1 -- 172.21.15.88:0/3885146009 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6abc016020 con 0x7f6acc0b6310 2024-09-16T01:36:06.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.777+0000 7f6acaffd640 1 -- 172.21.15.88:0/3885146009 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6abc00aa50 con 0x7f6acc0b6310 2024-09-16T01:36:06.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.777+0000 7f6acaffd640 1 -- 172.21.15.88:0/3885146009 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6abc00ae40 con 0x7f6acc0b6310 2024-09-16T01:36:06.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.777+0000 7f6ad4653640 1 -- 172.21.15.88:0/3885146009 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6acc14eae0 con 0x7f6acc0b6310 2024-09-16T01:36:06.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.777+0000 7f6ad4653640 1 -- 172.21.15.88:0/3885146009 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6acc14bf80 con 0x7f6acc0b6310 2024-09-16T01:36:06.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.778+0000 7f6ad4653640 1 -- 172.21.15.88:0/3885146009 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6acc068fa0 con 0x7f6acc0b6310 2024-09-16T01:36:06.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.778+0000 7f6acaffd640 1 -- 172.21.15.88:0/3885146009 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6abc019020 con 0x7f6acc0b6310 2024-09-16T01:36:06.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.779+0000 7f6acaffd640 1 --2- 172.21.15.88:0/3885146009 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ab806c2b0 0x7f6ab806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:06.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.779+0000 7f6acaffd640 1 -- 172.21.15.88:0/3885146009 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f6abc08b4f0 con 0x7f6acc0b6310 2024-09-16T01:36:06.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.779+0000 7f6ad1bee640 1 --2- 172.21.15.88:0/3885146009 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ab806c2b0 0x7f6ab806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:06.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.779+0000 7f6ad1bee640 1 --2- 172.21.15.88:0/3885146009 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ab806c2b0 0x7f6ab806e770 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0034000 tx=0x7f6ac002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:06.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.783+0000 7f6acaffd640 1 -- 172.21.15.88:0/3885146009 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6acc068fa0 con 0x7f6acc0b6310 2024-09-16T01:36:06.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.949+0000 7f6ad4653640 1 -- 172.21.15.88:0/3885146009 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6acc0c65c0 con 0x7f6acc0b6310 2024-09-16T01:36:06.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.950+0000 7f6acaffd640 1 -- 172.21.15.88:0/3885146009 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+16662 (secure 0 0 0) 0x7f6acc0c65c0 con 0x7f6acc0b6310 2024-09-16T01:36:06.952 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:06.952 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":44,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:36:05.691238+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:36:04.035328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-16T01:36:04.663046+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:36:06.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.952+0000 7f6ad4653640 1 -- 172.21.15.88:0/3885146009 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ab806c2b0 msgr2=0x7f6ab806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:06.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.952+0000 7f6ad4653640 1 --2- 172.21.15.88:0/3885146009 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ab806c2b0 0x7f6ab806e770 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0034000 tx=0x7f6ac002e000 comp rx=0 tx=0).stop 2024-09-16T01:36:06.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.952+0000 7f6ad4653640 1 -- 172.21.15.88:0/3885146009 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6acc0b6310 msgr2=0x7f6acc13dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:06.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.952+0000 7f6ad4653640 1 --2- 172.21.15.88:0/3885146009 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6acc0b6310 0x7f6acc13dce0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6abc00c7c0 tx=0x7f6abc00cc80 comp rx=0 tx=0).stop 2024-09-16T01:36:06.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.952+0000 7f6ad4653640 1 -- 172.21.15.88:0/3885146009 shutdown_connections 2024-09-16T01:36:06.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.952+0000 7f6ad4653640 1 --2- 172.21.15.88:0/3885146009 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ab806c2b0 0x7f6ab806e770 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:06.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.952+0000 7f6ad4653640 1 --2- 172.21.15.88:0/3885146009 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6acc15c550 0x7f6acc14f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:06.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.952+0000 7f6ad4653640 1 --2- 172.21.15.88:0/3885146009 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6acc15c170 0x7f6acc14f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:06.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.952+0000 7f6ad4653640 1 --2- 172.21.15.88:0/3885146009 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6acc0b6310 0x7f6acc13dce0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:06.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.952+0000 7f6ad4653640 1 -- 172.21.15.88:0/3885146009 >> 172.21.15.88:0/3885146009 conn(0x7f6acc06bc10 msgr2=0x7f6acc06d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:06.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.952+0000 7f6ad4653640 1 -- 172.21.15.88:0/3885146009 shutdown_connections 2024-09-16T01:36:06.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:06.952+0000 7f6ad4653640 1 -- 172.21.15.88:0/3885146009 wait complete. 2024-09-16T01:36:06.965 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-16T01:36:07.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.064+0000 7f73f8e23640 1 Processor -- start 2024-09-16T01:36:07.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.064+0000 7f73f8e23640 1 -- start start 2024-09-16T01:36:07.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.065+0000 7f73f8e23640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f73f40b54d0 0x7f73f406fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:07.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.065+0000 7f73f8e23640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f40b7450 0x7f73f415f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:07.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.065+0000 7f73f8e23640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73f406e990 con 0x7f73f415e6f0 2024-09-16T01:36:07.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.065+0000 7f73f8e23640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f73f406ec70 con 0x7f73f40b54d0 2024-09-16T01:36:07.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.065+0000 7f73f8e23640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f73f415ec70 con 0x7f73f40b7450 2024-09-16T01:36:07.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.065+0000 7f73f1d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f73f40b54d0 0x7f73f406fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:07.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.065+0000 7f73f1d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f73f40b54d0 0x7f73f406fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51004/0 (socket says 172.21.15.88:51004) 2024-09-16T01:36:07.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.065+0000 7f73f1d9b640 1 -- 172.21.15.88:0/3391315218 learned_addr learned my addr 172.21.15.88:0/3391315218 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:07.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.065+0000 7f73f259c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f73f415e6f0 0x7f73f40b6d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59380/0 (socket says 172.21.15.88:59380) 2024-09-16T01:36:07.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.066+0000 7f73f2d9d640 1 --2- 172.21.15.88:0/3391315218 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f40b7450 0x7f73f415f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:07.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.066+0000 7f73f1d9b640 1 -- 172.21.15.88:0/3391315218 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f40b7450 msgr2=0x7f73f415f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:07.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.066+0000 7f73f1d9b640 1 --2- 172.21.15.88:0/3391315218 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f40b7450 0x7f73f415f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:07.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.066+0000 7f73f1d9b640 1 -- 172.21.15.88:0/3391315218 >> v1:172.21.15.88:6789/0 conn(0x7f73f415e6f0 legacy=0x7f73f40b6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:07.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.066+0000 7f73f1d9b640 1 -- 172.21.15.88:0/3391315218 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f415ef50 con 0x7f73f40b54d0 2024-09-16T01:36:07.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.066+0000 7f73f159a640 1 -- 172.21.15.88:0/3391315218 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2963179192 0 0) 0x7f73f406e990 con 0x7f73f415e6f0 2024-09-16T01:36:07.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.066+0000 7f73f1d9b640 1 --2- 172.21.15.88:0/3391315218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73f40b54d0 0x7f73f406fd00 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f73e8009870 tx=0x7f73e802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35f7dec570a6f54 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:07.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.066+0000 7f73f159a640 1 -- 172.21.15.88:0/3391315218 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e8037070 con 0x7f73f40b54d0 2024-09-16T01:36:07.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.066+0000 7f73f159a640 1 -- 172.21.15.88:0/3391315218 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73e8032440 con 0x7f73f40b54d0 2024-09-16T01:36:07.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.066+0000 7f73f159a640 1 -- 172.21.15.88:0/3391315218 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e8032810 con 0x7f73f40b54d0 2024-09-16T01:36:07.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.067+0000 7f73f8e23640 1 -- 172.21.15.88:0/3391315218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73f40b54d0 msgr2=0x7f73f406fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:07.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.067+0000 7f73f8e23640 1 --2- 172.21.15.88:0/3391315218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73f40b54d0 0x7f73f406fd00 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f73e8009870 tx=0x7f73e802a270 comp rx=0 tx=0).stop 2024-09-16T01:36:07.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.067+0000 7f73f8e23640 1 -- 172.21.15.88:0/3391315218 shutdown_connections 2024-09-16T01:36:07.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.067+0000 7f73f8e23640 1 --2- 172.21.15.88:0/3391315218 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f40b7450 0x7f73f415f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:07.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.067+0000 7f73f8e23640 1 --2- 172.21.15.88:0/3391315218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73f40b54d0 0x7f73f406fd00 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:07.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.067+0000 7f73f8e23640 1 -- 172.21.15.88:0/3391315218 >> 172.21.15.88:0/3391315218 conn(0x7f73f40d2430 msgr2=0x7f73f40d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:07.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.067+0000 7f73f8e23640 1 -- 172.21.15.88:0/3391315218 shutdown_connections 2024-09-16T01:36:07.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.067+0000 7f73f8e23640 1 -- 172.21.15.88:0/3391315218 wait complete. 2024-09-16T01:36:07.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.068+0000 7f73f8e23640 1 Processor -- start 2024-09-16T01:36:07.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.068+0000 7f73f8e23640 1 -- start start 2024-09-16T01:36:07.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.069+0000 7f73f8e23640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73f40b54d0 0x7f73f41624b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:07.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.069+0000 7f73f8e23640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f40b7450 0x7f73f415d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:07.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.069+0000 7f73f8e23640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f415e6f0 0x7f73f415d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:07.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.069+0000 7f73f8e23640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f73f406e730 con 0x7f73f40b54d0 2024-09-16T01:36:07.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.069+0000 7f73f8e23640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f73f406e5b0 con 0x7f73f40b7450 2024-09-16T01:36:07.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.069+0000 7f73f8e23640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f73f4070830 con 0x7f73f415e6f0 2024-09-16T01:36:07.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.069+0000 7f73f259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73f40b54d0 0x7f73f41624b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:07.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.069+0000 7f73f259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73f40b54d0 0x7f73f41624b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51010/0 (socket says 172.21.15.88:51010) 2024-09-16T01:36:07.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.069+0000 7f73f259c640 1 -- 172.21.15.88:0/823205294 learned_addr learned my addr 172.21.15.88:0/823205294 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:07.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.069+0000 7f73f259c640 1 -- 172.21.15.88:0/823205294 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f415e6f0 msgr2=0x7f73f415d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:07.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.069+0000 7f73f259c640 1 --2- 172.21.15.88:0/823205294 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f415e6f0 0x7f73f415d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:07.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.069+0000 7f73f259c640 1 -- 172.21.15.88:0/823205294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f40b7450 msgr2=0x7f73f415d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:07.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.069+0000 7f73f1d9b640 1 --2- 172.21.15.88:0/823205294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f40b7450 0x7f73f415d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:07.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.070+0000 7f73f259c640 1 --2- 172.21.15.88:0/823205294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f40b7450 0x7f73f415d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:07.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.070+0000 7f73f259c640 1 -- 172.21.15.88:0/823205294 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f41616f0 con 0x7f73f40b54d0 2024-09-16T01:36:07.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.070+0000 7f73f1d9b640 1 --2- 172.21.15.88:0/823205294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f40b7450 0x7f73f415d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:07.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.070+0000 7f73f259c640 1 --2- 172.21.15.88:0/823205294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73f40b54d0 0x7f73f41624b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f73e400c7c0 tx=0x7f73e400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:07.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.070+0000 7f73e2ffd640 1 -- 172.21.15.88:0/823205294 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e4018070 con 0x7f73f40b54d0 2024-09-16T01:36:07.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.070+0000 7f73e2ffd640 1 -- 172.21.15.88:0/823205294 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73e400a640 con 0x7f73f40b54d0 2024-09-16T01:36:07.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.070+0000 7f73e2ffd640 1 -- 172.21.15.88:0/823205294 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e400aa30 con 0x7f73f40b54d0 2024-09-16T01:36:07.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.070+0000 7f73f8e23640 1 -- 172.21.15.88:0/823205294 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73f41619d0 con 0x7f73f40b54d0 2024-09-16T01:36:07.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.070+0000 7f73f8e23640 1 -- 172.21.15.88:0/823205294 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73f4161f90 con 0x7f73f40b54d0 2024-09-16T01:36:07.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.071+0000 7f73f8e23640 1 -- 172.21.15.88:0/823205294 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73f406c6c0 con 0x7f73f40b54d0 2024-09-16T01:36:07.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.071+0000 7f73e2ffd640 1 -- 172.21.15.88:0/823205294 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f73e401b070 con 0x7f73f40b54d0 2024-09-16T01:36:07.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.072+0000 7f73e2ffd640 1 --2- 172.21.15.88:0/823205294 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f73d406c2e0 0x7f73d406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:07.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.072+0000 7f73e2ffd640 1 -- 172.21.15.88:0/823205294 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f73e408a9d0 con 0x7f73f40b54d0 2024-09-16T01:36:07.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.072+0000 7f73f1d9b640 1 --2- 172.21.15.88:0/823205294 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f73d406c2e0 0x7f73d406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:07.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.072+0000 7f73f1d9b640 1 --2- 172.21.15.88:0/823205294 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f73d406c2e0 0x7f73d406e7a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f73e8034000 tx=0x7f73e802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:07.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.076+0000 7f73e2ffd640 1 -- 172.21.15.88:0/823205294 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f73e40591d0 con 0x7f73f40b54d0 2024-09-16T01:36:07.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.275+0000 7f73f8e23640 1 -- 172.21.15.88:0/823205294 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f73f406e1b0 con 0x7f73f40b54d0 2024-09-16T01:36:07.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.364+0000 7f73e2ffd640 1 -- 172.21.15.88:0/823205294 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) ==== 118+0+0 (secure 0 0 0) 0x7f73e4058ad0 con 0x7f73f40b54d0 2024-09-16T01:36:07.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.366+0000 7f73f8e23640 1 -- 172.21.15.88:0/823205294 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f73d406c2e0 msgr2=0x7f73d406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:07.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.366+0000 7f73f8e23640 1 --2- 172.21.15.88:0/823205294 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f73d406c2e0 0x7f73d406e7a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f73e8034000 tx=0x7f73e802e000 comp rx=0 tx=0).stop 2024-09-16T01:36:07.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.366+0000 7f73f8e23640 1 -- 172.21.15.88:0/823205294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73f40b54d0 msgr2=0x7f73f41624b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:07.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.366+0000 7f73f8e23640 1 --2- 172.21.15.88:0/823205294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73f40b54d0 0x7f73f41624b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f73e400c7c0 tx=0x7f73e400cc80 comp rx=0 tx=0).stop 2024-09-16T01:36:07.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.366+0000 7f73f8e23640 1 -- 172.21.15.88:0/823205294 shutdown_connections 2024-09-16T01:36:07.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.366+0000 7f73f8e23640 1 --2- 172.21.15.88:0/823205294 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f73d406c2e0 0x7f73d406e7a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:07.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.366+0000 7f73f8e23640 1 --2- 172.21.15.88:0/823205294 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f415e6f0 0x7f73f415d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:07.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.366+0000 7f73f8e23640 1 --2- 172.21.15.88:0/823205294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f40b7450 0x7f73f415d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:07.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.366+0000 7f73f8e23640 1 --2- 172.21.15.88:0/823205294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73f40b54d0 0x7f73f41624b0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:07.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.366+0000 7f73f8e23640 1 -- 172.21.15.88:0/823205294 >> 172.21.15.88:0/823205294 conn(0x7f73f40d2430 msgr2=0x7f73f406daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:07.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.366+0000 7f73f8e23640 1 -- 172.21.15.88:0/823205294 shutdown_connections 2024-09-16T01:36:07.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.366+0000 7f73f8e23640 1 -- 172.21.15.88:0/823205294 wait complete. 2024-09-16T01:36:07.378 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-16T01:36:07.378 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-16T01:36:07.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.479+0000 7f14d77fd640 1 Processor -- start 2024-09-16T01:36:07.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.479+0000 7f14d77fd640 1 -- start start 2024-09-16T01:36:07.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.480+0000 7f14d77fd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14d015e350 0x7f14d006f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:07.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.480+0000 7f14d77fd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14d006bd40 con 0x7f14d0073f30 2024-09-16T01:36:07.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.480+0000 7f14d77fd640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14d006c020 con 0x7f14d015e730 2024-09-16T01:36:07.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.480+0000 7f14d77fd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f14d006c300 con 0x7f14d015e350 2024-09-16T01:36:07.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.480+0000 7f14d5d9a640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f14d0073f30 0x7f14d00ae990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59396/0 (socket says 172.21.15.88:59396) 2024-09-16T01:36:07.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.480+0000 7f14d5d9a640 1 -- 172.21.15.88:0/2950716187 learned_addr learned my addr 172.21.15.88:0/2950716187 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:07.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.480+0000 7f14d5599640 1 --2- 172.21.15.88:0/2950716187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14d015e350 0x7f14d006f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:07.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.480+0000 7f14c7fff640 1 -- 172.21.15.88:0/2950716187 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 938327860 0 0) 0x7f14d006bd40 con 0x7f14d0073f30 2024-09-16T01:36:07.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.480+0000 7f14c7fff640 1 -- 172.21.15.88:0/2950716187 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14bc003520 con 0x7f14d0073f30 2024-09-16T01:36:07.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.480+0000 7f14c7fff640 1 -- 172.21.15.88:0/2950716187 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2208491040 0 0) 0x7f14bc003520 con 0x7f14d0073f30 2024-09-16T01:36:07.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.480+0000 7f14c7fff640 1 -- 172.21.15.88:0/2950716187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14d015e350 msgr2=0x7f14d006f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:07.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.480+0000 7f14c7fff640 1 --2- 172.21.15.88:0/2950716187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14d015e350 0x7f14d006f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:07.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.480+0000 7f14c7fff640 1 -- 172.21.15.88:0/2950716187 >> v1:172.21.15.154:6789/0 conn(0x7f14d015e730 legacy=0x7f14d0073820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:07.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.480+0000 7f14c7fff640 1 -- 172.21.15.88:0/2950716187 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14d006c5e0 con 0x7f14d0073f30 2024-09-16T01:36:07.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.480+0000 7f14c7fff640 1 -- 172.21.15.88:0/2950716187 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f14c80019f0 con 0x7f14d0073f30 2024-09-16T01:36:07.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.480+0000 7f14c7fff640 1 -- 172.21.15.88:0/2950716187 >> v1:172.21.15.88:6789/0 conn(0x7f14d0073f30 legacy=0x7f14d00ae990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:07.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.481+0000 7f14c7fff640 1 --2- 172.21.15.88:0/2950716187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14bc0039e0 0x7f14bc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:07.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.481+0000 7f14c7fff640 1 --2- 172.21.15.88:0/2950716187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14bc004800 0x7f14bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:07.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.481+0000 7f14c7fff640 1 --2- 172.21.15.88:0/2950716187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14d015e350 0x7f14bc005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:07.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.481+0000 7f14c7fff640 1 -- 172.21.15.88:0/2950716187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f14d006c300 con 0x7f14bc0039e0 2024-09-16T01:36:07.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.481+0000 7f14c7fff640 1 -- 172.21.15.88:0/2950716187 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc00baf0 con 0x7f14bc004800 2024-09-16T01:36:07.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.481+0000 7f14c7fff640 1 -- 172.21.15.88:0/2950716187 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f14bc00bcf0 con 0x7f14d015e350 2024-09-16T01:36:07.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.481+0000 7f14d5599640 1 --2- 172.21.15.88:0/2950716187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14bc0039e0 0x7f14bc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:07.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.481+0000 7f14d4d98640 1 --2- 172.21.15.88:0/2950716187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14bc004800 0x7f14bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:07.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.481+0000 7f14d5d9a640 1 --2- 172.21.15.88:0/2950716187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14d015e350 0x7f14bc005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:07.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.481+0000 7f14d5599640 1 -- 172.21.15.88:0/2950716187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14d015e350 msgr2=0x7f14bc005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:07.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.481+0000 7f14d5599640 1 --2- 172.21.15.88:0/2950716187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14d015e350 0x7f14bc005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:07.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.481+0000 7f14d5599640 1 -- 172.21.15.88:0/2950716187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14bc004800 msgr2=0x7f14bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:07.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.481+0000 7f14d5599640 1 --2- 172.21.15.88:0/2950716187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14bc004800 0x7f14bc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:07.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.481+0000 7f14d5599640 1 -- 172.21.15.88:0/2950716187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14bc003520 con 0x7f14bc0039e0 2024-09-16T01:36:07.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.481+0000 7f14d4d98640 1 --2- 172.21.15.88:0/2950716187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14bc004800 0x7f14bc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:07.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.481+0000 7f14d5d9a640 1 --2- 172.21.15.88:0/2950716187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14d015e350 0x7f14bc005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:07.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.482+0000 7f14d5599640 1 --2- 172.21.15.88:0/2950716187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14bc0039e0 0x7f14bc003e00 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f14cc009ac0 tx=0x7f14bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1af3590550e5e963 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:07.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.482+0000 7f14c7fff640 1 -- 172.21.15.88:0/2950716187 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14cc036070 con 0x7f14bc0039e0 2024-09-16T01:36:07.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.482+0000 7f14c7fff640 1 -- 172.21.15.88:0/2950716187 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14cc031400 con 0x7f14bc0039e0 2024-09-16T01:36:07.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.482+0000 7f14c7fff640 1 -- 172.21.15.88:0/2950716187 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14cc009690 con 0x7f14bc0039e0 2024-09-16T01:36:07.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.482+0000 7f14d77fd640 1 -- 172.21.15.88:0/2950716187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14bc0039e0 msgr2=0x7f14bc003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:07.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.482+0000 7f14d77fd640 1 --2- 172.21.15.88:0/2950716187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14bc0039e0 0x7f14bc003e00 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f14cc009ac0 tx=0x7f14bc005830 comp rx=0 tx=0).stop 2024-09-16T01:36:07.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.482+0000 7f14d5d9a640 1 -- 172.21.15.88:0/2950716187 reap_dead start 2024-09-16T01:36:07.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.482+0000 7f14d77fd640 1 -- 172.21.15.88:0/2950716187 shutdown_connections 2024-09-16T01:36:07.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.482+0000 7f14d77fd640 1 -- 172.21.15.88:0/2950716187 >> 172.21.15.88:0/2950716187 conn(0x7f14d00b3070 msgr2=0x7f14d00b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:07.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.482+0000 7f14d77fd640 1 -- 172.21.15.88:0/2950716187 shutdown_connections 2024-09-16T01:36:07.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.482+0000 7f14d77fd640 1 -- 172.21.15.88:0/2950716187 wait complete. 2024-09-16T01:36:07.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.483+0000 7f14d77fd640 1 Processor -- start 2024-09-16T01:36:07.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.484+0000 7f14d77fd640 1 -- start start 2024-09-16T01:36:07.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.484+0000 7f14d77fd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14bc0039e0 0x7f14d015eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:07.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.484+0000 7f14d77fd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14d015f0a0 0x7f14d01f7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:07.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.484+0000 7f14d77fd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14d0142b50 0x7f14d01f7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:07.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.484+0000 7f14d77fd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f14d006f890 con 0x7f14d015f0a0 2024-09-16T01:36:07.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.484+0000 7f14d77fd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f14d006f710 con 0x7f14d0142b50 2024-09-16T01:36:07.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.484+0000 7f14d77fd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f14d006fa10 con 0x7f14bc0039e0 2024-09-16T01:36:07.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.484+0000 7f14d4d98640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14d015f0a0 0x7f14d01f7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:07.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.484+0000 7f14d4d98640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14d015f0a0 0x7f14d01f7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51032/0 (socket says 172.21.15.88:51032) 2024-09-16T01:36:07.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.484+0000 7f14d4d98640 1 -- 172.21.15.88:0/1349009914 learned_addr learned my addr 172.21.15.88:0/1349009914 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:07.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.485+0000 7f14d5599640 1 --2- 172.21.15.88:0/1349009914 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14bc0039e0 0x7f14d015eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:07.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.485+0000 7f14d5d9a640 1 --2- 172.21.15.88:0/1349009914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14d0142b50 0x7f14d01f7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:07.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.485+0000 7f14d4d98640 1 -- 172.21.15.88:0/1349009914 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14bc0039e0 msgr2=0x7f14d015eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:07.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.485+0000 7f14d4d98640 1 --2- 172.21.15.88:0/1349009914 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14bc0039e0 0x7f14d015eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:07.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.485+0000 7f14d4d98640 1 -- 172.21.15.88:0/1349009914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14d0142b50 msgr2=0x7f14d01f7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:07.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.485+0000 7f14d4d98640 1 --2- 172.21.15.88:0/1349009914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14d0142b50 0x7f14d01f7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:07.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.485+0000 7f14d4d98640 1 -- 172.21.15.88:0/1349009914 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14d015f520 con 0x7f14d015f0a0 2024-09-16T01:36:07.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.485+0000 7f14d4d98640 1 --2- 172.21.15.88:0/1349009914 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14d015f0a0 0x7f14d01f7670 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f14c000e570 tx=0x7f14c000ea30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:07.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.485+0000 7f14c5ffb640 1 -- 172.21.15.88:0/1349009914 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c00052e0 con 0x7f14d015f0a0 2024-09-16T01:36:07.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.485+0000 7f14c5ffb640 1 -- 172.21.15.88:0/1349009914 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14c0018070 con 0x7f14d015f0a0 2024-09-16T01:36:07.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.485+0000 7f14d77fd640 1 -- 172.21.15.88:0/1349009914 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14d015f800 con 0x7f14d015f0a0 2024-09-16T01:36:07.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.485+0000 7f14c5ffb640 1 -- 172.21.15.88:0/1349009914 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c0003810 con 0x7f14d015f0a0 2024-09-16T01:36:07.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.485+0000 7f14d77fd640 1 -- 172.21.15.88:0/1349009914 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14d015fdc0 con 0x7f14d015f0a0 2024-09-16T01:36:07.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.486+0000 7f14d77fd640 1 -- 172.21.15.88:0/1349009914 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14d006fe40 con 0x7f14d015f0a0 2024-09-16T01:36:07.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.486+0000 7f14c5ffb640 1 -- 172.21.15.88:0/1349009914 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f14c00039b0 con 0x7f14d015f0a0 2024-09-16T01:36:07.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.487+0000 7f14c5ffb640 1 --2- 172.21.15.88:0/1349009914 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f14b406c2b0 0x7f14b406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:07.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.487+0000 7f14c5ffb640 1 -- 172.21.15.88:0/1349009914 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f14c008b140 con 0x7f14d015f0a0 2024-09-16T01:36:07.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.487+0000 7f14d5599640 1 --2- 172.21.15.88:0/1349009914 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f14b406c2b0 0x7f14b406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:07.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.487+0000 7f14d5599640 1 --2- 172.21.15.88:0/1349009914 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f14b406c2b0 0x7f14b406e770 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f14cc033000 tx=0x7f14cc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:07.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.491+0000 7f14c5ffb640 1 -- 172.21.15.88:0/1349009914 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f14c0059940 con 0x7f14d015f0a0 2024-09-16T01:36:07.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:07.692+0000 7f14d77fd640 1 -- 172.21.15.88:0/1349009914 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f14d00b5390 con 0x7f14d015f0a0 2024-09-16T01:36:08.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.369+0000 7f14c5ffb640 1 -- 172.21.15.88:0/1349009914 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) ==== 122+0+0 (secure 0 0 0) 0x7f14c0059240 con 0x7f14d015f0a0 2024-09-16T01:36:08.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.372+0000 7f14d77fd640 1 -- 172.21.15.88:0/1349009914 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f14b406c2b0 msgr2=0x7f14b406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:08.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.372+0000 7f14d77fd640 1 --2- 172.21.15.88:0/1349009914 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f14b406c2b0 0x7f14b406e770 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f14cc033000 tx=0x7f14cc008030 comp rx=0 tx=0).stop 2024-09-16T01:36:08.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.372+0000 7f14d77fd640 1 -- 172.21.15.88:0/1349009914 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14d015f0a0 msgr2=0x7f14d01f7670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:08.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.372+0000 7f14d77fd640 1 --2- 172.21.15.88:0/1349009914 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14d015f0a0 0x7f14d01f7670 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f14c000e570 tx=0x7f14c000ea30 comp rx=0 tx=0).stop 2024-09-16T01:36:08.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.372+0000 7f14d77fd640 1 -- 172.21.15.88:0/1349009914 shutdown_connections 2024-09-16T01:36:08.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.372+0000 7f14d77fd640 1 --2- 172.21.15.88:0/1349009914 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f14b406c2b0 0x7f14b406e770 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:08.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.372+0000 7f14d77fd640 1 --2- 172.21.15.88:0/1349009914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14d0142b50 0x7f14d01f7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:08.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.372+0000 7f14d77fd640 1 --2- 172.21.15.88:0/1349009914 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14d015f0a0 0x7f14d01f7670 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:08.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.372+0000 7f14d77fd640 1 --2- 172.21.15.88:0/1349009914 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14bc0039e0 0x7f14d015eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:08.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.372+0000 7f14d77fd640 1 -- 172.21.15.88:0/1349009914 >> 172.21.15.88:0/1349009914 conn(0x7f14d00b3070 msgr2=0x7f14d00b5ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:08.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.372+0000 7f14d77fd640 1 -- 172.21.15.88:0/1349009914 shutdown_connections 2024-09-16T01:36:08.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.372+0000 7f14d77fd640 1 -- 172.21.15.88:0/1349009914 wait complete. 2024-09-16T01:36:08.385 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-16T01:36:08.386 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:36:08.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.486+0000 7f880e7e8640 1 Processor -- start 2024-09-16T01:36:08.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.486+0000 7f880e7e8640 1 -- start start 2024-09-16T01:36:08.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.487+0000 7f880e7e8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f88080cf370 0x7f88080cf800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:08.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.487+0000 7f880e7e8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f88080b52b0 0x7f88080b5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:08.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.487+0000 7f880e7e8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f880815c850 con 0x7f880815c470 2024-09-16T01:36:08.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.487+0000 7f880e7e8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f88080b4160 con 0x7f88080b52b0 2024-09-16T01:36:08.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.487+0000 7f880e7e8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f88080b4440 con 0x7f88080cf370 2024-09-16T01:36:08.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.487+0000 7f880cd85640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f88080b52b0 0x7f88080b5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:08.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.487+0000 7f880cd85640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f88080b52b0 0x7f88080b5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51046/0 (socket says 172.21.15.88:51046) 2024-09-16T01:36:08.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.487+0000 7f880cd85640 1 -- 172.21.15.88:0/2912115196 learned_addr learned my addr 172.21.15.88:0/2912115196 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:08.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.488+0000 7f88077fe640 1 --2- 172.21.15.88:0/2912115196 >> v2:172.21.15.154:3300/0 conn(0x7f88080cf370 0x7f88080cf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:08.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.488+0000 7f880cd85640 1 -- 172.21.15.88:0/2912115196 >> v2:172.21.15.154:3300/0 conn(0x7f88080cf370 msgr2=0x7f88080cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:08.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.488+0000 7f880cd85640 1 --2- 172.21.15.88:0/2912115196 >> v2:172.21.15.154:3300/0 conn(0x7f88080cf370 0x7f88080cf800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:08.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.488+0000 7f880cd85640 1 -- 172.21.15.88:0/2912115196 >> v1:172.21.15.154:6789/0 conn(0x7f880815c470 legacy=0x7f88080b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:08.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.488+0000 7f880cd85640 1 -- 172.21.15.88:0/2912115196 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88080b4720 con 0x7f88080b52b0 2024-09-16T01:36:08.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.488+0000 7f880cd85640 1 --2- 172.21.15.88:0/2912115196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88080b52b0 0x7f88080b5720 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f8800009e50 tx=0x7f880002a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee7ba6b6b6115b03 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:08.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.489+0000 7f8806ffd640 1 -- 172.21.15.88:0/2912115196 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8800036070 con 0x7f88080b52b0 2024-09-16T01:36:08.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.489+0000 7f8806ffd640 1 -- 172.21.15.88:0/2912115196 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8800031600 con 0x7f88080b52b0 2024-09-16T01:36:08.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.489+0000 7f8806ffd640 1 -- 172.21.15.88:0/2912115196 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88000319d0 con 0x7f88080b52b0 2024-09-16T01:36:08.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.489+0000 7f880e7e8640 1 -- 172.21.15.88:0/2912115196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88080b52b0 msgr2=0x7f88080b5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:08.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.489+0000 7f880e7e8640 1 --2- 172.21.15.88:0/2912115196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88080b52b0 0x7f88080b5720 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f8800009e50 tx=0x7f880002a760 comp rx=0 tx=0).stop 2024-09-16T01:36:08.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.489+0000 7f880e7e8640 1 -- 172.21.15.88:0/2912115196 shutdown_connections 2024-09-16T01:36:08.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.489+0000 7f880e7e8640 1 --2- 172.21.15.88:0/2912115196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f88080b52b0 0x7f88080b5720 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:08.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.489+0000 7f880e7e8640 1 --2- 172.21.15.88:0/2912115196 >> v2:172.21.15.154:3300/0 conn(0x7f88080cf370 0x7f88080cf800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:08.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.489+0000 7f880e7e8640 1 -- 172.21.15.88:0/2912115196 >> 172.21.15.88:0/2912115196 conn(0x7f880806bc10 msgr2=0x7f880806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:08.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.489+0000 7f880e7e8640 1 -- 172.21.15.88:0/2912115196 shutdown_connections 2024-09-16T01:36:08.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.489+0000 7f880e7e8640 1 -- 172.21.15.88:0/2912115196 wait complete. 2024-09-16T01:36:08.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.490+0000 7f880e7e8640 1 Processor -- start 2024-09-16T01:36:08.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.491+0000 7f880e7e8640 1 -- start start 2024-09-16T01:36:08.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.491+0000 7f880e7e8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88080b52b0 0x7f8808158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:08.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.491+0000 7f880e7e8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88080cf370 0x7f8808153e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:08.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.491+0000 7f880e7e8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f880815c470 0x7f8808147e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:08.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.491+0000 7f880e7e8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f88080b3e20 con 0x7f880815c470 2024-09-16T01:36:08.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.491+0000 7f880e7e8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f88080b4d90 con 0x7f88080b52b0 2024-09-16T01:36:08.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.491+0000 7f880e7e8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f880806e1b0 con 0x7f88080cf370 2024-09-16T01:36:08.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.491+0000 7f880cd85640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f880815c470 0x7f8808147e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:08.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.491+0000 7f880cd85640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f880815c470 0x7f8808147e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51056/0 (socket says 172.21.15.88:51056) 2024-09-16T01:36:08.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.491+0000 7f880cd85640 1 -- 172.21.15.88:0/4075472484 learned_addr learned my addr 172.21.15.88:0/4075472484 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:08.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.491+0000 7f8807fff640 1 --2- 172.21.15.88:0/4075472484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88080b52b0 0x7f8808158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:08.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.491+0000 7f88077fe640 1 --2- 172.21.15.88:0/4075472484 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88080cf370 0x7f8808153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:08.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.492+0000 7f880cd85640 1 -- 172.21.15.88:0/4075472484 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88080cf370 msgr2=0x7f8808153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:08.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.492+0000 7f880cd85640 1 --2- 172.21.15.88:0/4075472484 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88080cf370 0x7f8808153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:08.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.492+0000 7f880cd85640 1 -- 172.21.15.88:0/4075472484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88080b52b0 msgr2=0x7f8808158130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:08.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.492+0000 7f880cd85640 1 --2- 172.21.15.88:0/4075472484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88080b52b0 0x7f8808158130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:08.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.492+0000 7f880cd85640 1 -- 172.21.15.88:0/4075472484 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8808153260 con 0x7f880815c470 2024-09-16T01:36:08.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.492+0000 7f880cd85640 1 --2- 172.21.15.88:0/4075472484 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f880815c470 0x7f8808147e90 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f8800009f70 tx=0x7f8800004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:08.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.492+0000 7f8804ff9640 1 -- 172.21.15.88:0/4075472484 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880003f020 con 0x7f880815c470 2024-09-16T01:36:08.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.492+0000 7f8804ff9640 1 -- 172.21.15.88:0/4075472484 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8800037040 con 0x7f880815c470 2024-09-16T01:36:08.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.492+0000 7f880e7e8640 1 -- 172.21.15.88:0/4075472484 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8808157090 con 0x7f880815c470 2024-09-16T01:36:08.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.492+0000 7f8804ff9640 1 -- 172.21.15.88:0/4075472484 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8800036070 con 0x7f880815c470 2024-09-16T01:36:08.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.492+0000 7f880e7e8640 1 -- 172.21.15.88:0/4075472484 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8808157650 con 0x7f880815c470 2024-09-16T01:36:08.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.493+0000 7f880e7e8640 1 -- 172.21.15.88:0/4075472484 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88080cf800 con 0x7f880815c470 2024-09-16T01:36:08.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.493+0000 7f8804ff9640 1 -- 172.21.15.88:0/4075472484 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8800042020 con 0x7f880815c470 2024-09-16T01:36:08.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.494+0000 7f8804ff9640 1 --2- 172.21.15.88:0/4075472484 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f87f406c2b0 0x7f87f406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:08.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.494+0000 7f8804ff9640 1 -- 172.21.15.88:0/4075472484 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f88000ab640 con 0x7f880815c470 2024-09-16T01:36:08.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.494+0000 7f8807fff640 1 --2- 172.21.15.88:0/4075472484 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f87f406c2b0 0x7f87f406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:08.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.494+0000 7f8807fff640 1 --2- 172.21.15.88:0/4075472484 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f87f406c2b0 0x7f87f406e770 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f87f8001780 tx=0x7f87f8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:08.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.498+0000 7f8804ff9640 1 -- 172.21.15.88:0/4075472484 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8800049030 con 0x7f880815c470 2024-09-16T01:36:08.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.699+0000 7f880e7e8640 1 -- 172.21.15.88:0/4075472484 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88080b3af0 con 0x7f880815c470 2024-09-16T01:36:08.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.699+0000 7f8804ff9640 1 -- 172.21.15.88:0/4075472484 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5226 (secure 0 0 0) 0x7f8800079760 con 0x7f880815c470 2024-09-16T01:36:08.701 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:08.701 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":25,"btime":"2024-09-16T01:36:08:365397+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4877,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1769276566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":1769276566},{"type":"v1","addr":"172.21.15.154:6835","nonce":1769276566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/725621378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":725621378},{"type":"v1","addr":"172.21.15.154:6834","nonce":725621378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4199111872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4199111872},{"type":"v1","addr":"172.21.15.88:6837","nonce":4199111872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:05.691540+0000","modified":"2024-09-16T01:36:07.693602+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5406},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5406":{"gid":5406,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6836/1610810484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1610810484},{"type":"v1","addr":"172.21.15.88:6836","nonce":1610810484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5406,"qdb_cluster":[5406]},"id":3}]} 2024-09-16T01:36:08.701 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 25 2024-09-16T01:36:08.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.701+0000 7f880e7e8640 1 -- 172.21.15.88:0/4075472484 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f87f406c2b0 msgr2=0x7f87f406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:08.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.701+0000 7f880e7e8640 1 --2- 172.21.15.88:0/4075472484 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f87f406c2b0 0x7f87f406e770 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f87f8001780 tx=0x7f87f8009040 comp rx=0 tx=0).stop 2024-09-16T01:36:08.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.701+0000 7f880e7e8640 1 -- 172.21.15.88:0/4075472484 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f880815c470 msgr2=0x7f8808147e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:08.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.701+0000 7f880e7e8640 1 --2- 172.21.15.88:0/4075472484 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f880815c470 0x7f8808147e90 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f8800009f70 tx=0x7f8800004a40 comp rx=0 tx=0).stop 2024-09-16T01:36:08.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.701+0000 7f880e7e8640 1 -- 172.21.15.88:0/4075472484 shutdown_connections 2024-09-16T01:36:08.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.701+0000 7f880e7e8640 1 --2- 172.21.15.88:0/4075472484 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f87f406c2b0 0x7f87f406e770 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:08.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.701+0000 7f880e7e8640 1 --2- 172.21.15.88:0/4075472484 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f880815c470 0x7f8808147e90 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:08.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.701+0000 7f880e7e8640 1 --2- 172.21.15.88:0/4075472484 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88080cf370 0x7f8808153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:08.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.701+0000 7f880e7e8640 1 --2- 172.21.15.88:0/4075472484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88080b52b0 0x7f8808158130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:08.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.701+0000 7f880e7e8640 1 -- 172.21.15.88:0/4075472484 >> 172.21.15.88:0/4075472484 conn(0x7f880806bc10 msgr2=0x7f880806d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:08.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.702+0000 7f880e7e8640 1 -- 172.21.15.88:0/4075472484 shutdown_connections 2024-09-16T01:36:08.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.702+0000 7f880e7e8640 1 -- 172.21.15.88:0/4075472484 wait complete. 2024-09-16T01:36:08.714 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:36:08.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.810+0000 7f24a7f49640 1 Processor -- start 2024-09-16T01:36:08.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.811+0000 7f24a7f49640 1 -- start start 2024-09-16T01:36:08.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.811+0000 7f24a7f49640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24a0073180 0x7f24a00735d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:08.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.811+0000 7f24a7f49640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24a015e920 con 0x7f24a015e540 2024-09-16T01:36:08.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.811+0000 7f24a7f49640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24a0162a50 con 0x7f24a00b01f0 2024-09-16T01:36:08.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.811+0000 7f24a7f49640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f24a0073b10 con 0x7f24a0073180 2024-09-16T01:36:08.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.812+0000 7f24a5ce5640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f24a015e540 0x7f24a00b52c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59410/0 (socket says 172.21.15.88:59410) 2024-09-16T01:36:08.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.812+0000 7f24a5ce5640 1 -- 172.21.15.88:0/1666538965 learned_addr learned my addr 172.21.15.88:0/1666538965 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:08.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.812+0000 7f24a64e6640 1 --2- 172.21.15.88:0/1666538965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24a0073180 0x7f24a00735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:08.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.812+0000 7f24a4ce3640 1 -- 172.21.15.88:0/1666538965 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2639757199 0 0) 0x7f24a015e920 con 0x7f24a015e540 2024-09-16T01:36:08.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.812+0000 7f24a4ce3640 1 -- 172.21.15.88:0/1666538965 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f248c003520 con 0x7f24a015e540 2024-09-16T01:36:08.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.812+0000 7f24a4ce3640 1 -- 172.21.15.88:0/1666538965 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 990964128 0 0) 0x7f24a0162a50 con 0x7f24a00b01f0 2024-09-16T01:36:08.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.812+0000 7f24a4ce3640 1 -- 172.21.15.88:0/1666538965 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24a015e920 con 0x7f24a00b01f0 2024-09-16T01:36:08.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.812+0000 7f24a4ce3640 1 -- 172.21.15.88:0/1666538965 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1962138689 0 0) 0x7f248c003520 con 0x7f24a015e540 2024-09-16T01:36:08.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.812+0000 7f24a4ce3640 1 -- 172.21.15.88:0/1666538965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24a0073180 msgr2=0x7f24a00735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:08.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.812+0000 7f24a4ce3640 1 --2- 172.21.15.88:0/1666538965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24a0073180 0x7f24a00735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:08.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.812+0000 7f24a4ce3640 1 -- 172.21.15.88:0/1666538965 >> v1:172.21.15.154:6789/0 conn(0x7f24a00b01f0 legacy=0x7f24a01620e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:08.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.812+0000 7f24a4ce3640 1 -- 172.21.15.88:0/1666538965 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a0162d30 con 0x7f24a015e540 2024-09-16T01:36:08.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.812+0000 7f24a4ce3640 1 -- 172.21.15.88:0/1666538965 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f249c001670 con 0x7f24a015e540 2024-09-16T01:36:08.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.812+0000 7f24a4ce3640 1 -- 172.21.15.88:0/1666538965 >> v1:172.21.15.88:6789/0 conn(0x7f24a015e540 legacy=0x7f24a00b52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:08.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.812+0000 7f24a4ce3640 1 --2- 172.21.15.88:0/1666538965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f248c003b70 0x7f248c003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:08.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.812+0000 7f24a4ce3640 1 --2- 172.21.15.88:0/1666538965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24a0073180 0x7f248c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:08.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.812+0000 7f24a4ce3640 1 --2- 172.21.15.88:0/1666538965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f248c004d40 0x7f248c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:08.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.812+0000 7f24a4ce3640 1 -- 172.21.15.88:0/1666538965 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f24a0073b10 con 0x7f248c004d40 2024-09-16T01:36:08.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.812+0000 7f24a4ce3640 1 -- 172.21.15.88:0/1666538965 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f248c00bbb0 con 0x7f248c003b70 2024-09-16T01:36:08.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.812+0000 7f24a4ce3640 1 -- 172.21.15.88:0/1666538965 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f248c00bdb0 con 0x7f24a0073180 2024-09-16T01:36:08.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.812+0000 7f24a64e6640 1 --2- 172.21.15.88:0/1666538965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f248c004d40 0x7f248c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:08.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.812+0000 7f24a5ce5640 1 --2- 172.21.15.88:0/1666538965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f248c003b70 0x7f248c003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:08.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.812+0000 7f24a54e4640 1 --2- 172.21.15.88:0/1666538965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24a0073180 0x7f248c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:08.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.813+0000 7f24a64e6640 1 -- 172.21.15.88:0/1666538965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24a0073180 msgr2=0x7f248c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:08.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.813+0000 7f24a64e6640 1 --2- 172.21.15.88:0/1666538965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24a0073180 0x7f248c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:08.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.813+0000 7f24a64e6640 1 -- 172.21.15.88:0/1666538965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f248c003b70 msgr2=0x7f248c003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:08.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.813+0000 7f24a64e6640 1 --2- 172.21.15.88:0/1666538965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f248c003b70 0x7f248c003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:08.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.813+0000 7f24a64e6640 1 -- 172.21.15.88:0/1666538965 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f248c003520 con 0x7f248c004d40 2024-09-16T01:36:08.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.813+0000 7f24a64e6640 1 --2- 172.21.15.88:0/1666538965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f248c004d40 0x7f248c0051c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f2498009fc0 tx=0x7f248c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c600c59ba15b0d0a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:08.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.813+0000 7f24a4ce3640 1 -- 172.21.15.88:0/1666538965 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2498009c10 con 0x7f248c004d40 2024-09-16T01:36:08.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.813+0000 7f24a4ce3640 1 -- 172.21.15.88:0/1666538965 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2498036070 con 0x7f248c004d40 2024-09-16T01:36:08.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.813+0000 7f24a4ce3640 1 -- 172.21.15.88:0/1666538965 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2498031880 con 0x7f248c004d40 2024-09-16T01:36:08.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.814+0000 7f24a7f49640 1 -- 172.21.15.88:0/1666538965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f248c004d40 msgr2=0x7f248c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:08.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.814+0000 7f24a7f49640 1 --2- 172.21.15.88:0/1666538965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f248c004d40 0x7f248c0051c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f2498009fc0 tx=0x7f248c005830 comp rx=0 tx=0).stop 2024-09-16T01:36:08.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.814+0000 7f24a64e6640 1 -- 172.21.15.88:0/1666538965 reap_dead start 2024-09-16T01:36:08.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.814+0000 7f24a7f49640 1 -- 172.21.15.88:0/1666538965 shutdown_connections 2024-09-16T01:36:08.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.814+0000 7f24a7f49640 1 -- 172.21.15.88:0/1666538965 >> 172.21.15.88:0/1666538965 conn(0x7f24a006bc10 msgr2=0x7f24a006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:08.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.814+0000 7f24a7f49640 1 -- 172.21.15.88:0/1666538965 shutdown_connections 2024-09-16T01:36:08.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.814+0000 7f24a7f49640 1 -- 172.21.15.88:0/1666538965 wait complete. 2024-09-16T01:36:08.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.815+0000 7f24a7f49640 1 Processor -- start 2024-09-16T01:36:08.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.815+0000 7f24a7f49640 1 -- start start 2024-09-16T01:36:08.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.816+0000 7f24a7f49640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f248c004d40 0x7f24a013cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:08.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.816+0000 7f24a7f49640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24a0160960 0x7f24a01396c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:08.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.816+0000 7f24a7f49640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24a00717c0 0x7f24a0071c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:08.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.816+0000 7f24a7f49640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f24a01627f0 con 0x7f24a0160960 2024-09-16T01:36:08.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.816+0000 7f24a7f49640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f24a0163010 con 0x7f24a00717c0 2024-09-16T01:36:08.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.816+0000 7f24a7f49640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f24a00aeca0 con 0x7f248c004d40 2024-09-16T01:36:08.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.816+0000 7f24a54e4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24a0160960 0x7f24a01396c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:08.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.816+0000 7f24a54e4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24a0160960 0x7f24a01396c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51066/0 (socket says 172.21.15.88:51066) 2024-09-16T01:36:08.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.816+0000 7f24a54e4640 1 -- 172.21.15.88:0/617924814 learned_addr learned my addr 172.21.15.88:0/617924814 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:08.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.816+0000 7f24a64e6640 1 --2- 172.21.15.88:0/617924814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24a00717c0 0x7f24a0071c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:08.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.816+0000 7f24a5ce5640 1 --2- 172.21.15.88:0/617924814 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f248c004d40 0x7f24a013cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:08.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.816+0000 7f24a54e4640 1 -- 172.21.15.88:0/617924814 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f248c004d40 msgr2=0x7f24a013cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:08.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.816+0000 7f24a54e4640 1 --2- 172.21.15.88:0/617924814 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f248c004d40 0x7f24a013cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:08.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.816+0000 7f24a54e4640 1 -- 172.21.15.88:0/617924814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24a00717c0 msgr2=0x7f24a0071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:08.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.816+0000 7f24a54e4640 1 --2- 172.21.15.88:0/617924814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24a00717c0 0x7f24a0071c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:08.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.816+0000 7f24a54e4640 1 -- 172.21.15.88:0/617924814 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a0139c00 con 0x7f24a0160960 2024-09-16T01:36:08.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.817+0000 7f24a54e4640 1 --2- 172.21.15.88:0/617924814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24a0160960 0x7f24a01396c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f249400f770 tx=0x7f249400fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:08.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.817+0000 7f24927fc640 1 -- 172.21.15.88:0/617924814 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24940050d0 con 0x7f24a0160960 2024-09-16T01:36:08.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.817+0000 7f24927fc640 1 -- 172.21.15.88:0/617924814 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2494018070 con 0x7f24a0160960 2024-09-16T01:36:08.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.817+0000 7f24927fc640 1 -- 172.21.15.88:0/617924814 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249400d730 con 0x7f24a0160960 2024-09-16T01:36:08.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.817+0000 7f24a7f49640 1 -- 172.21.15.88:0/617924814 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24a0161cc0 con 0x7f24a0160960 2024-09-16T01:36:08.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.817+0000 7f24a7f49640 1 -- 172.21.15.88:0/617924814 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24a0161420 con 0x7f24a0160960 2024-09-16T01:36:08.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.818+0000 7f24a7f49640 1 -- 172.21.15.88:0/617924814 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24a0072180 con 0x7f24a0160960 2024-09-16T01:36:08.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.818+0000 7f24927fc640 1 -- 172.21.15.88:0/617924814 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f249400dae0 con 0x7f24a0160960 2024-09-16T01:36:08.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.819+0000 7f24927fc640 1 --2- 172.21.15.88:0/617924814 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f248406c2e0 0x7f248406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:08.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.819+0000 7f24927fc640 1 -- 172.21.15.88:0/617924814 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f249408b4a0 con 0x7f24a0160960 2024-09-16T01:36:08.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.819+0000 7f24a5ce5640 1 --2- 172.21.15.88:0/617924814 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f248406c2e0 0x7f248406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:08.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.819+0000 7f24a5ce5640 1 --2- 172.21.15.88:0/617924814 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f248406c2e0 0x7f248406e7a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f249c001eb0 tx=0x7f249c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:08.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.823+0000 7f24927fc640 1 -- 172.21.15.88:0/617924814 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2494059ca0 con 0x7f24a0160960 2024-09-16T01:36:08.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.988+0000 7f24a7f49640 1 -- 172.21.15.88:0/617924814 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f24a0073660 con 0x7f24a0160960 2024-09-16T01:36:08.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.989+0000 7f24927fc640 1 -- 172.21.15.88:0/617924814 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+16649 (secure 0 0 0) 0x7f24940595a0 con 0x7f24a0160960 2024-09-16T01:36:08.991 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:08.991 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":45,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:36:07.348833+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:36:04.035328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-16T01:36:04.663046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:36:08.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.991+0000 7f24a7f49640 1 -- 172.21.15.88:0/617924814 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f248406c2e0 msgr2=0x7f248406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:08.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.991+0000 7f24a7f49640 1 --2- 172.21.15.88:0/617924814 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f248406c2e0 0x7f248406e7a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f249c001eb0 tx=0x7f249c009040 comp rx=0 tx=0).stop 2024-09-16T01:36:08.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.991+0000 7f24a7f49640 1 -- 172.21.15.88:0/617924814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24a0160960 msgr2=0x7f24a01396c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:08.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.991+0000 7f24a7f49640 1 --2- 172.21.15.88:0/617924814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24a0160960 0x7f24a01396c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f249400f770 tx=0x7f249400fc30 comp rx=0 tx=0).stop 2024-09-16T01:36:08.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.991+0000 7f24a7f49640 1 -- 172.21.15.88:0/617924814 shutdown_connections 2024-09-16T01:36:08.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.991+0000 7f24a7f49640 1 --2- 172.21.15.88:0/617924814 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f248406c2e0 0x7f248406e7a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:08.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.991+0000 7f24a7f49640 1 --2- 172.21.15.88:0/617924814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24a00717c0 0x7f24a0071c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:08.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.991+0000 7f24a7f49640 1 --2- 172.21.15.88:0/617924814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24a0160960 0x7f24a01396c0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:08.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.991+0000 7f24a7f49640 1 --2- 172.21.15.88:0/617924814 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f248c004d40 0x7f24a013cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:08.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.991+0000 7f24a7f49640 1 -- 172.21.15.88:0/617924814 >> 172.21.15.88:0/617924814 conn(0x7f24a006bc10 msgr2=0x7f24a006cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:08.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.991+0000 7f24a7f49640 1 -- 172.21.15.88:0/617924814 shutdown_connections 2024-09-16T01:36:08.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:08.991+0000 7f24a7f49640 1 -- 172.21.15.88:0/617924814 wait complete. 2024-09-16T01:36:09.004 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T01:36:09.004 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:36:09.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.101+0000 7f6d796ae640 1 Processor -- start 2024-09-16T01:36:09.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.101+0000 7f6d796ae640 1 -- start start 2024-09-16T01:36:09.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.101+0000 7f6d796ae640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6d740d3310 0x7f6d740d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:09.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.101+0000 7f6d796ae640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d7415ea40 con 0x7f6d7415e660 2024-09-16T01:36:09.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.101+0000 7f6d796ae640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d740d3870 con 0x7f6d740d1650 2024-09-16T01:36:09.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.101+0000 7f6d796ae640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6d740d1ac0 con 0x7f6d740d3310 2024-09-16T01:36:09.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.102+0000 7f6d727fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6d740d3310 0x7f6d740d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:09.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.102+0000 7f6d727fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6d740d3310 0x7f6d740d1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51076/0 (socket says 172.21.15.88:51076) 2024-09-16T01:36:09.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.102+0000 7f6d727fc640 1 -- 172.21.15.88:0/3632903889 learned_addr learned my addr 172.21.15.88:0/3632903889 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:09.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.102+0000 7f6d72ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6d7415e660 0x7f6d740cf730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59418/0 (socket says 172.21.15.88:59418) 2024-09-16T01:36:09.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.102+0000 7f6d727fc640 1 -- 172.21.15.88:0/3632903889 >> v1:172.21.15.154:6789/0 conn(0x7f6d740d1650 legacy=0x7f6d740b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:09.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.102+0000 7f6d727fc640 1 -- 172.21.15.88:0/3632903889 >> v1:172.21.15.88:6789/0 conn(0x7f6d7415e660 legacy=0x7f6d740cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:09.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.102+0000 7f6d727fc640 1 -- 172.21.15.88:0/3632903889 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d740d3b50 con 0x7f6d740d3310 2024-09-16T01:36:09.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.102+0000 7f6d71ffb640 1 -- 172.21.15.88:0/3632903889 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4057412055 0 0) 0x7f6d7415ea40 con 0x7f6d7415e660 2024-09-16T01:36:09.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.103+0000 7f6d727fc640 1 --2- 172.21.15.88:0/3632903889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d740d3310 0x7f6d740d1110 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c009870 tx=0x7f6d5c02a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cc8566fd62907a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:09.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.103+0000 7f6d71ffb640 1 -- 172.21.15.88:0/3632903889 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c037070 con 0x7f6d740d3310 2024-09-16T01:36:09.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.103+0000 7f6d71ffb640 1 -- 172.21.15.88:0/3632903889 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d5c032440 con 0x7f6d740d3310 2024-09-16T01:36:09.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.103+0000 7f6d71ffb640 1 -- 172.21.15.88:0/3632903889 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c032810 con 0x7f6d740d3310 2024-09-16T01:36:09.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.103+0000 7f6d796ae640 1 -- 172.21.15.88:0/3632903889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d740d3310 msgr2=0x7f6d740d1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:09.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.103+0000 7f6d796ae640 1 --2- 172.21.15.88:0/3632903889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d740d3310 0x7f6d740d1110 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c009870 tx=0x7f6d5c02a2a0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.103+0000 7f6d796ae640 1 -- 172.21.15.88:0/3632903889 shutdown_connections 2024-09-16T01:36:09.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.103+0000 7f6d796ae640 1 --2- 172.21.15.88:0/3632903889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d740d3310 0x7f6d740d1110 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.103+0000 7f6d796ae640 1 -- 172.21.15.88:0/3632903889 >> 172.21.15.88:0/3632903889 conn(0x7f6d740b6410 msgr2=0x7f6d740b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:09.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.103+0000 7f6d796ae640 1 -- 172.21.15.88:0/3632903889 shutdown_connections 2024-09-16T01:36:09.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.103+0000 7f6d796ae640 1 -- 172.21.15.88:0/3632903889 wait complete. 2024-09-16T01:36:09.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.105+0000 7f6d796ae640 1 Processor -- start 2024-09-16T01:36:09.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.105+0000 7f6d796ae640 1 -- start start 2024-09-16T01:36:09.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.105+0000 7f6d796ae640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d740d1650 0x7f6d740ce320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:09.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.105+0000 7f6d796ae640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d740d3310 0x7f6d740cf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:09.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.105+0000 7f6d796ae640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d7415e660 0x7f6d740b9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:09.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.105+0000 7f6d796ae640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6d740d43f0 con 0x7f6d740d3310 2024-09-16T01:36:09.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.105+0000 7f6d796ae640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6d740d4110 con 0x7f6d740d1650 2024-09-16T01:36:09.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.105+0000 7f6d796ae640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6d740b8280 con 0x7f6d7415e660 2024-09-16T01:36:09.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.106+0000 7f6d727fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d740d3310 0x7f6d740cf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:09.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.106+0000 7f6d727fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d740d3310 0x7f6d740cf9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51080/0 (socket says 172.21.15.88:51080) 2024-09-16T01:36:09.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.106+0000 7f6d727fc640 1 -- 172.21.15.88:0/611213071 learned_addr learned my addr 172.21.15.88:0/611213071 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:09.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.106+0000 7f6d72ffd640 1 --2- 172.21.15.88:0/611213071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d740d1650 0x7f6d740ce320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:09.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.106+0000 7f6d727fc640 1 -- 172.21.15.88:0/611213071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d7415e660 msgr2=0x7f6d740b9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:09.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.106+0000 7f6d737fe640 1 --2- 172.21.15.88:0/611213071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d7415e660 0x7f6d740b9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:09.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.106+0000 7f6d727fc640 1 --2- 172.21.15.88:0/611213071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d7415e660 0x7f6d740b9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.106+0000 7f6d727fc640 1 -- 172.21.15.88:0/611213071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d740d1650 msgr2=0x7f6d740ce320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:09.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.106+0000 7f6d727fc640 1 --2- 172.21.15.88:0/611213071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d740d1650 0x7f6d740ce320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.106+0000 7f6d727fc640 1 -- 172.21.15.88:0/611213071 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d740ce900 con 0x7f6d740d3310 2024-09-16T01:36:09.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.106+0000 7f6d72ffd640 1 --2- 172.21.15.88:0/611213071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d740d1650 0x7f6d740ce320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:09.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.106+0000 7f6d737fe640 1 --2- 172.21.15.88:0/611213071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d7415e660 0x7f6d740b9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:09.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.106+0000 7f6d727fc640 1 --2- 172.21.15.88:0/611213071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d740d3310 0x7f6d740cf9a0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c009870 tx=0x7f6d5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:09.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.106+0000 7f6d6b7fe640 1 -- 172.21.15.88:0/611213071 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c037040 con 0x7f6d740d3310 2024-09-16T01:36:09.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.106+0000 7f6d6b7fe640 1 -- 172.21.15.88:0/611213071 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d5c004250 con 0x7f6d740d3310 2024-09-16T01:36:09.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.106+0000 7f6d796ae640 1 -- 172.21.15.88:0/611213071 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d740cebe0 con 0x7f6d740d3310 2024-09-16T01:36:09.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.106+0000 7f6d6b7fe640 1 -- 172.21.15.88:0/611213071 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c03c670 con 0x7f6d740d3310 2024-09-16T01:36:09.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.106+0000 7f6d796ae640 1 -- 172.21.15.88:0/611213071 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d740cf1a0 con 0x7f6d740d3310 2024-09-16T01:36:09.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.107+0000 7f6d796ae640 1 -- 172.21.15.88:0/611213071 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d740cf680 con 0x7f6d740d3310 2024-09-16T01:36:09.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.108+0000 7f6d6b7fe640 1 -- 172.21.15.88:0/611213071 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6d5c03a070 con 0x7f6d740d3310 2024-09-16T01:36:09.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.108+0000 7f6d6b7fe640 1 --2- 172.21.15.88:0/611213071 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6d5406c2e0 0x7f6d5406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:09.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.108+0000 7f6d6b7fe640 1 -- 172.21.15.88:0/611213071 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f6d5c0abdc0 con 0x7f6d740d3310 2024-09-16T01:36:09.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.108+0000 7f6d72ffd640 1 --2- 172.21.15.88:0/611213071 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6d5406c2e0 0x7f6d5406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:09.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.109+0000 7f6d72ffd640 1 --2- 172.21.15.88:0/611213071 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6d5406c2e0 0x7f6d5406e7a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6d64000f60 tx=0x7f6d64009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:09.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.112+0000 7f6d6b7fe640 1 -- 172.21.15.88:0/611213071 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6d740cf680 con 0x7f6d740d3310 2024-09-16T01:36:09.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.274+0000 7f6d796ae640 1 -- 172.21.15.88:0/611213071 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6d740d1ed0 con 0x7f6d5406c2e0 2024-09-16T01:36:09.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.280+0000 7f6d6b7fe640 1 -- 172.21.15.88:0/611213071 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+247134 (secure 0 0 0) 0x7f6d740d1ed0 con 0x7f6d5406c2e0 2024-09-16T01:36:09.284 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:09.286 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:36:09.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.283+0000 7f6d796ae640 1 -- 172.21.15.88:0/611213071 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6d5406c2e0 msgr2=0x7f6d5406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:09.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.283+0000 7f6d796ae640 1 --2- 172.21.15.88:0/611213071 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6d5406c2e0 0x7f6d5406e7a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6d64000f60 tx=0x7f6d64009040 comp rx=0 tx=0).stop 2024-09-16T01:36:09.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.283+0000 7f6d796ae640 1 -- 172.21.15.88:0/611213071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d740d3310 msgr2=0x7f6d740cf9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:09.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.283+0000 7f6d796ae640 1 --2- 172.21.15.88:0/611213071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d740d3310 0x7f6d740cf9a0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c009870 tx=0x7f6d5c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.283+0000 7f6d796ae640 1 -- 172.21.15.88:0/611213071 shutdown_connections 2024-09-16T01:36:09.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.283+0000 7f6d796ae640 1 --2- 172.21.15.88:0/611213071 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6d5406c2e0 0x7f6d5406e7a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.283+0000 7f6d796ae640 1 --2- 172.21.15.88:0/611213071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d7415e660 0x7f6d740b9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.283+0000 7f6d796ae640 1 --2- 172.21.15.88:0/611213071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d740d3310 0x7f6d740cf9a0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.283+0000 7f6d796ae640 1 --2- 172.21.15.88:0/611213071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d740d1650 0x7f6d740ce320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.283+0000 7f6d796ae640 1 -- 172.21.15.88:0/611213071 >> 172.21.15.88:0/611213071 conn(0x7f6d740b6410 msgr2=0x7f6d7415bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:09.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.283+0000 7f6d796ae640 1 -- 172.21.15.88:0/611213071 shutdown_connections 2024-09-16T01:36:09.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.284+0000 7f6d796ae640 1 -- 172.21.15.88:0/611213071 wait complete. 2024-09-16T01:36:09.296 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":97,"stamp":"2024-09-16T01:36:08.538350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590438,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":163,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":25,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221240,"kb_used_data":4980,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753480,"statfs":{"total":773094113280,"available":772867563520,"internally_reserved":0,"allocated":5099520,"data_stored":2729763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.195069"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.359868+0000","last_change":"2024-09-16T01:36:05.359868+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.359868+0000","last_undegraded":"2024-09-16T01:36:05.359868+0000","last_fullsized":"2024-09-16T01:36:05.359868+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.370012+0000","last_change":"2024-09-16T01:36:06.366967+0000","last_active":"2024-09-16T01:36:06.370012+0000","last_peered":"2024-09-16T01:36:06.370012+0000","last_clean":"2024-09-16T01:36:06.370012+0000","last_became_active":"2024-09-16T01:36:06.366688+0000","last_became_peered":"2024-09-16T01:36:06.366688+0000","last_unstale":"2024-09-16T01:36:06.370012+0000","last_undegraded":"2024-09-16T01:36:06.370012+0000","last_fullsized":"2024-09-16T01:36:06.370012+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:40:52.576920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352493+0000","last_change":"2024-09-16T01:36:05.352493+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352493+0000","last_undegraded":"2024-09-16T01:36:05.352493+0000","last_fullsized":"2024-09-16T01:36:05.352493+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358690+0000","last_change":"2024-09-16T01:36:05.358690+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358690+0000","last_undegraded":"2024-09-16T01:36:05.358690+0000","last_fullsized":"2024-09-16T01:36:05.358690+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.371287+0000","last_change":"2024-09-16T01:36:06.369570+0000","last_active":"2024-09-16T01:36:06.371287+0000","last_peered":"2024-09-16T01:36:06.371287+0000","last_clean":"2024-09-16T01:36:06.371287+0000","last_became_active":"2024-09-16T01:36:06.369276+0000","last_became_peered":"2024-09-16T01:36:06.369276+0000","last_unstale":"2024-09-16T01:36:06.371287+0000","last_undegraded":"2024-09-16T01:36:06.371287+0000","last_fullsized":"2024-09-16T01:36:06.371287+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:39:28.693098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358381+0000","last_change":"2024-09-16T01:36:05.358381+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358381+0000","last_undegraded":"2024-09-16T01:36:05.358381+0000","last_fullsized":"2024-09-16T01:36:05.358381+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358435+0000","last_change":"2024-09-16T01:36:05.358435+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358435+0000","last_undegraded":"2024-09-16T01:36:05.358435+0000","last_fullsized":"2024-09-16T01:36:05.358435+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358875+0000","last_change":"2024-09-16T01:36:05.358875+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358875+0000","last_undegraded":"2024-09-16T01:36:05.358875+0000","last_fullsized":"2024-09-16T01:36:05.358875+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358586+0000","last_change":"2024-09-16T01:36:05.358586+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358586+0000","last_undegraded":"2024-09-16T01:36:05.358586+0000","last_fullsized":"2024-09-16T01:36:05.358586+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358111+0000","last_change":"2024-09-16T01:36:05.358111+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358111+0000","last_undegraded":"2024-09-16T01:36:05.358111+0000","last_fullsized":"2024-09-16T01:36:05.358111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.357648+0000","last_change":"2024-09-16T01:36:05.357648+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.357648+0000","last_undegraded":"2024-09-16T01:36:05.357648+0000","last_fullsized":"2024-09-16T01:36:05.357648+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.376012+0000","last_change":"2024-09-16T01:36:06.374437+0000","last_active":"2024-09-16T01:36:06.376012+0000","last_peered":"2024-09-16T01:36:06.376012+0000","last_clean":"2024-09-16T01:36:06.376012+0000","last_became_active":"2024-09-16T01:36:06.374156+0000","last_became_peered":"2024-09-16T01:36:06.374156+0000","last_unstale":"2024-09-16T01:36:06.376012+0000","last_undegraded":"2024-09-16T01:36:06.376012+0000","last_fullsized":"2024-09-16T01:36:06.376012+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:14:35.252633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352763+0000","last_change":"2024-09-16T01:36:05.352763+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352763+0000","last_undegraded":"2024-09-16T01:36:05.352763+0000","last_fullsized":"2024-09-16T01:36:05.352763+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356941+0000","last_change":"2024-09-16T01:36:05.356941+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356941+0000","last_undegraded":"2024-09-16T01:36:05.356941+0000","last_fullsized":"2024-09-16T01:36:05.356941+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358500+0000","last_change":"2024-09-16T01:36:05.358500+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358500+0000","last_undegraded":"2024-09-16T01:36:05.358500+0000","last_fullsized":"2024-09-16T01:36:05.358500+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352230+0000","last_change":"2024-09-16T01:36:05.352230+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352230+0000","last_undegraded":"2024-09-16T01:36:05.352230+0000","last_fullsized":"2024-09-16T01:36:05.352230+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351446+0000","last_change":"2024-09-16T01:36:05.351446+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351446+0000","last_undegraded":"2024-09-16T01:36:05.351446+0000","last_fullsized":"2024-09-16T01:36:05.351446+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351350+0000","last_change":"2024-09-16T01:36:05.351350+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351350+0000","last_undegraded":"2024-09-16T01:36:05.351350+0000","last_fullsized":"2024-09-16T01:36:05.351350+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.350520+0000","last_change":"2024-09-16T01:36:05.350520+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.350520+0000","last_undegraded":"2024-09-16T01:36:05.350520+0000","last_fullsized":"2024-09-16T01:36:05.350520+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.354933+0000","last_change":"2024-09-16T01:36:05.354933+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.354933+0000","last_undegraded":"2024-09-16T01:36:05.354933+0000","last_fullsized":"2024-09-16T01:36:05.354933+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358886+0000","last_change":"2024-09-16T01:36:05.358886+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358886+0000","last_undegraded":"2024-09-16T01:36:05.358886+0000","last_fullsized":"2024-09-16T01:36:05.358886+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356521+0000","last_change":"2024-09-16T01:36:05.356521+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356521+0000","last_undegraded":"2024-09-16T01:36:05.356521+0000","last_fullsized":"2024-09-16T01:36:05.356521+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351493+0000","last_change":"2024-09-16T01:36:05.351493+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351493+0000","last_undegraded":"2024-09-16T01:36:05.351493+0000","last_fullsized":"2024-09-16T01:36:05.351493+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.350478+0000","last_change":"2024-09-16T01:36:05.350478+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.350478+0000","last_undegraded":"2024-09-16T01:36:05.350478+0000","last_fullsized":"2024-09-16T01:36:05.350478+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.350770+0000","last_change":"2024-09-16T01:36:05.350770+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.350770+0000","last_undegraded":"2024-09-16T01:36:05.350770+0000","last_fullsized":"2024-09-16T01:36:05.350770+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353568+0000","last_change":"2024-09-16T01:36:05.353568+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353568+0000","last_undegraded":"2024-09-16T01:36:05.353568+0000","last_fullsized":"2024-09-16T01:36:05.353568+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.350845+0000","last_change":"2024-09-16T01:36:05.350845+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.350845+0000","last_undegraded":"2024-09-16T01:36:05.350845+0000","last_fullsized":"2024-09-16T01:36:05.350845+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.370131+0000","last_change":"2024-09-16T01:36:06.365094+0000","last_active":"2024-09-16T01:36:06.370131+0000","last_peered":"2024-09-16T01:36:06.370131+0000","last_clean":"2024-09-16T01:36:06.370131+0000","last_became_active":"2024-09-16T01:36:06.364790+0000","last_became_peered":"2024-09-16T01:36:06.364790+0000","last_unstale":"2024-09-16T01:36:06.370131+0000","last_undegraded":"2024-09-16T01:36:06.370131+0000","last_fullsized":"2024-09-16T01:36:06.370131+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:29:16.206145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351073+0000","last_change":"2024-09-16T01:36:05.351073+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351073+0000","last_undegraded":"2024-09-16T01:36:05.351073+0000","last_fullsized":"2024-09-16T01:36:05.351073+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353729+0000","last_change":"2024-09-16T01:36:05.353729+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353729+0000","last_undegraded":"2024-09-16T01:36:05.353729+0000","last_fullsized":"2024-09-16T01:36:05.353729+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356913+0000","last_change":"2024-09-16T01:36:05.356913+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356913+0000","last_undegraded":"2024-09-16T01:36:05.356913+0000","last_fullsized":"2024-09-16T01:36:05.356913+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351956+0000","last_change":"2024-09-16T01:36:05.351956+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351956+0000","last_undegraded":"2024-09-16T01:36:05.351956+0000","last_fullsized":"2024-09-16T01:36:05.351956+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.354055+0000","last_change":"2024-09-16T01:36:05.354055+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.354055+0000","last_undegraded":"2024-09-16T01:36:05.354055+0000","last_fullsized":"2024-09-16T01:36:05.354055+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.360688+0000","last_change":"2024-09-16T01:36:05.357580+0000","last_active":"2024-09-16T01:36:05.360688+0000","last_peered":"2024-09-16T01:36:05.360688+0000","last_clean":"2024-09-16T01:36:05.360688+0000","last_became_active":"2024-09-16T01:36:05.357331+0000","last_became_peered":"2024-09-16T01:36:05.357331+0000","last_unstale":"2024-09-16T01:36:05.360688+0000","last_undegraded":"2024-09-16T01:36:05.360688+0000","last_fullsized":"2024-09-16T01:36:05.360688+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:28:41.466172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351699+0000","last_change":"2024-09-16T01:36:05.351699+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351699+0000","last_undegraded":"2024-09-16T01:36:05.351699+0000","last_fullsized":"2024-09-16T01:36:05.351699+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.365028+0000","last_change":"2024-09-16T01:36:05.357954+0000","last_active":"2024-09-16T01:36:05.365028+0000","last_peered":"2024-09-16T01:36:05.365028+0000","last_clean":"2024-09-16T01:36:05.365028+0000","last_became_active":"2024-09-16T01:36:05.357018+0000","last_became_peered":"2024-09-16T01:36:05.357018+0000","last_unstale":"2024-09-16T01:36:05.365028+0000","last_undegraded":"2024-09-16T01:36:05.365028+0000","last_fullsized":"2024-09-16T01:36:05.365028+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:53.565250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.378364+0000","last_change":"2024-09-16T01:36:06.377925+0000","last_active":"2024-09-16T01:36:06.378364+0000","last_peered":"2024-09-16T01:36:06.378364+0000","last_clean":"2024-09-16T01:36:06.378364+0000","last_became_active":"2024-09-16T01:36:06.377631+0000","last_became_peered":"2024-09-16T01:36:06.377631+0000","last_unstale":"2024-09-16T01:36:06.378364+0000","last_undegraded":"2024-09-16T01:36:06.378364+0000","last_fullsized":"2024-09-16T01:36:06.378364+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:09:13.085891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368709+0000","last_change":"2024-09-16T01:36:05.365158+0000","last_active":"2024-09-16T01:36:05.368709+0000","last_peered":"2024-09-16T01:36:05.368709+0000","last_clean":"2024-09-16T01:36:05.368709+0000","last_became_active":"2024-09-16T01:36:05.364885+0000","last_became_peered":"2024-09-16T01:36:05.364885+0000","last_unstale":"2024-09-16T01:36:05.368709+0000","last_undegraded":"2024-09-16T01:36:05.368709+0000","last_fullsized":"2024-09-16T01:36:05.368709+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:06:24.080498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.354065+0000","last_change":"2024-09-16T01:36:05.354065+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.354065+0000","last_undegraded":"2024-09-16T01:36:05.354065+0000","last_fullsized":"2024-09-16T01:36:05.354065+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373524+0000","last_change":"2024-09-16T01:36:05.368193+0000","last_active":"2024-09-16T01:36:05.373524+0000","last_peered":"2024-09-16T01:36:05.373524+0000","last_clean":"2024-09-16T01:36:05.373524+0000","last_became_active":"2024-09-16T01:36:05.367907+0000","last_became_peered":"2024-09-16T01:36:05.367907+0000","last_unstale":"2024-09-16T01:36:05.373524+0000","last_undegraded":"2024-09-16T01:36:05.373524+0000","last_fullsized":"2024-09-16T01:36:05.373524+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:40:04.647251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353341+0000","last_change":"2024-09-16T01:36:05.353341+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353341+0000","last_undegraded":"2024-09-16T01:36:05.353341+0000","last_fullsized":"2024-09-16T01:36:05.353341+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368806+0000","last_change":"2024-09-16T01:36:05.361916+0000","last_active":"2024-09-16T01:36:05.368806+0000","last_peered":"2024-09-16T01:36:05.368806+0000","last_clean":"2024-09-16T01:36:05.368806+0000","last_became_active":"2024-09-16T01:36:05.361658+0000","last_became_peered":"2024-09-16T01:36:05.361658+0000","last_unstale":"2024-09-16T01:36:05.368806+0000","last_undegraded":"2024-09-16T01:36:05.368806+0000","last_fullsized":"2024-09-16T01:36:05.368806+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:41.613259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356484+0000","last_change":"2024-09-16T01:36:05.356484+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356484+0000","last_undegraded":"2024-09-16T01:36:05.356484+0000","last_fullsized":"2024-09-16T01:36:05.356484+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373139+0000","last_change":"2024-09-16T01:36:05.365323+0000","last_active":"2024-09-16T01:36:05.373139+0000","last_peered":"2024-09-16T01:36:05.373139+0000","last_clean":"2024-09-16T01:36:05.373139+0000","last_became_active":"2024-09-16T01:36:05.364940+0000","last_became_peered":"2024-09-16T01:36:05.364940+0000","last_unstale":"2024-09-16T01:36:05.373139+0000","last_undegraded":"2024-09-16T01:36:05.373139+0000","last_fullsized":"2024-09-16T01:36:05.373139+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:38:11.910238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353237+0000","last_change":"2024-09-16T01:36:05.353237+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353237+0000","last_undegraded":"2024-09-16T01:36:05.353237+0000","last_fullsized":"2024-09-16T01:36:05.353237+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.357443+0000","last_change":"2024-09-16T01:36:05.359801+0000","last_active":"2024-09-16T01:36:06.357443+0000","last_peered":"2024-09-16T01:36:06.357443+0000","last_clean":"2024-09-16T01:36:06.357443+0000","last_became_active":"2024-09-16T01:36:05.359542+0000","last_became_peered":"2024-09-16T01:36:05.359542+0000","last_unstale":"2024-09-16T01:36:06.357443+0000","last_undegraded":"2024-09-16T01:36:06.357443+0000","last_fullsized":"2024-09-16T01:36:06.357443+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:22.128838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351989+0000","last_change":"2024-09-16T01:36:05.351989+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351989+0000","last_undegraded":"2024-09-16T01:36:05.351989+0000","last_fullsized":"2024-09-16T01:36:05.351989+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.366391+0000","last_change":"2024-09-16T01:36:05.364586+0000","last_active":"2024-09-16T01:36:05.366391+0000","last_peered":"2024-09-16T01:36:05.366391+0000","last_clean":"2024-09-16T01:36:05.366391+0000","last_became_active":"2024-09-16T01:36:05.364271+0000","last_became_peered":"2024-09-16T01:36:05.364271+0000","last_unstale":"2024-09-16T01:36:05.366391+0000","last_undegraded":"2024-09-16T01:36:05.366391+0000","last_fullsized":"2024-09-16T01:36:05.366391+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:15:05.347700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.350838+0000","last_change":"2024-09-16T01:36:05.350838+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.350838+0000","last_undegraded":"2024-09-16T01:36:05.350838+0000","last_fullsized":"2024-09-16T01:36:05.350838+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.370306+0000","last_change":"2024-09-16T01:36:05.364097+0000","last_active":"2024-09-16T01:36:05.370306+0000","last_peered":"2024-09-16T01:36:05.370306+0000","last_clean":"2024-09-16T01:36:05.370306+0000","last_became_active":"2024-09-16T01:36:05.363758+0000","last_became_peered":"2024-09-16T01:36:05.363758+0000","last_unstale":"2024-09-16T01:36:05.370306+0000","last_undegraded":"2024-09-16T01:36:05.370306+0000","last_fullsized":"2024-09-16T01:36:05.370306+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:59.822221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.359339+0000","last_change":"2024-09-16T01:36:05.359339+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.359339+0000","last_undegraded":"2024-09-16T01:36:05.359339+0000","last_fullsized":"2024-09-16T01:36:05.359339+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.365112+0000","last_change":"2024-09-16T01:36:05.363830+0000","last_active":"2024-09-16T01:36:05.365112+0000","last_peered":"2024-09-16T01:36:05.365112+0000","last_clean":"2024-09-16T01:36:05.365112+0000","last_became_active":"2024-09-16T01:36:05.363572+0000","last_became_peered":"2024-09-16T01:36:05.363572+0000","last_unstale":"2024-09-16T01:36:05.365112+0000","last_undegraded":"2024-09-16T01:36:05.365112+0000","last_fullsized":"2024-09-16T01:36:05.365112+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:53.565250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356478+0000","last_change":"2024-09-16T01:36:05.356478+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356478+0000","last_undegraded":"2024-09-16T01:36:05.356478+0000","last_fullsized":"2024-09-16T01:36:05.356478+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373551+0000","last_change":"2024-09-16T01:36:05.368744+0000","last_active":"2024-09-16T01:36:05.373551+0000","last_peered":"2024-09-16T01:36:05.373551+0000","last_clean":"2024-09-16T01:36:05.373551+0000","last_became_active":"2024-09-16T01:36:05.368069+0000","last_became_peered":"2024-09-16T01:36:05.368069+0000","last_unstale":"2024-09-16T01:36:05.373551+0000","last_undegraded":"2024-09-16T01:36:05.373551+0000","last_fullsized":"2024-09-16T01:36:05.373551+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:58.552340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.360574+0000","last_change":"2024-09-16T01:36:05.360574+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.360574+0000","last_undegraded":"2024-09-16T01:36:05.360574+0000","last_fullsized":"2024-09-16T01:36:05.360574+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368680+0000","last_change":"2024-09-16T01:36:05.365975+0000","last_active":"2024-09-16T01:36:05.368680+0000","last_peered":"2024-09-16T01:36:05.368680+0000","last_clean":"2024-09-16T01:36:05.368680+0000","last_became_active":"2024-09-16T01:36:05.365674+0000","last_became_peered":"2024-09-16T01:36:05.365674+0000","last_unstale":"2024-09-16T01:36:05.368680+0000","last_undegraded":"2024-09-16T01:36:05.368680+0000","last_fullsized":"2024-09-16T01:36:05.368680+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:21:23.387226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352330+0000","last_change":"2024-09-16T01:36:05.352330+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352330+0000","last_undegraded":"2024-09-16T01:36:05.352330+0000","last_fullsized":"2024-09-16T01:36:05.352330+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368728+0000","last_change":"2024-09-16T01:36:05.360882+0000","last_active":"2024-09-16T01:36:05.368728+0000","last_peered":"2024-09-16T01:36:05.368728+0000","last_clean":"2024-09-16T01:36:05.368728+0000","last_became_active":"2024-09-16T01:36:05.360521+0000","last_became_peered":"2024-09-16T01:36:05.360521+0000","last_unstale":"2024-09-16T01:36:05.368728+0000","last_undegraded":"2024-09-16T01:36:05.368728+0000","last_fullsized":"2024-09-16T01:36:05.368728+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:05:47.000282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.354770+0000","last_change":"2024-09-16T01:36:05.354770+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.354770+0000","last_undegraded":"2024-09-16T01:36:05.354770+0000","last_fullsized":"2024-09-16T01:36:05.354770+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373578+0000","last_change":"2024-09-16T01:36:05.369476+0000","last_active":"2024-09-16T01:36:05.373578+0000","last_peered":"2024-09-16T01:36:05.373578+0000","last_clean":"2024-09-16T01:36:05.373578+0000","last_became_active":"2024-09-16T01:36:05.369155+0000","last_became_peered":"2024-09-16T01:36:05.369155+0000","last_unstale":"2024-09-16T01:36:05.373578+0000","last_undegraded":"2024-09-16T01:36:05.373578+0000","last_fullsized":"2024-09-16T01:36:05.373578+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:22.128838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356076+0000","last_change":"2024-09-16T01:36:05.356076+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356076+0000","last_undegraded":"2024-09-16T01:36:05.356076+0000","last_fullsized":"2024-09-16T01:36:05.356076+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.366290+0000","last_change":"2024-09-16T01:36:05.359803+0000","last_active":"2024-09-16T01:36:05.366290+0000","last_peered":"2024-09-16T01:36:05.366290+0000","last_clean":"2024-09-16T01:36:05.366290+0000","last_became_active":"2024-09-16T01:36:05.359552+0000","last_became_peered":"2024-09-16T01:36:05.359552+0000","last_unstale":"2024-09-16T01:36:05.366290+0000","last_undegraded":"2024-09-16T01:36:05.366290+0000","last_fullsized":"2024-09-16T01:36:05.366290+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:32:35.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.357949+0000","last_change":"2024-09-16T01:36:05.357949+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.357949+0000","last_undegraded":"2024-09-16T01:36:05.357949+0000","last_fullsized":"2024-09-16T01:36:05.357949+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.362430+0000","last_change":"2024-09-16T01:36:05.359125+0000","last_active":"2024-09-16T01:36:05.362430+0000","last_peered":"2024-09-16T01:36:05.362430+0000","last_clean":"2024-09-16T01:36:05.362430+0000","last_became_active":"2024-09-16T01:36:05.358486+0000","last_became_peered":"2024-09-16T01:36:05.358486+0000","last_unstale":"2024-09-16T01:36:05.362430+0000","last_undegraded":"2024-09-16T01:36:05.362430+0000","last_fullsized":"2024-09-16T01:36:05.362430+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:22:44.509769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353180+0000","last_change":"2024-09-16T01:36:05.353180+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353180+0000","last_undegraded":"2024-09-16T01:36:05.353180+0000","last_fullsized":"2024-09-16T01:36:05.353180+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.364379+0000","last_change":"2024-09-16T01:36:05.361659+0000","last_active":"2024-09-16T01:36:05.364379+0000","last_peered":"2024-09-16T01:36:05.364379+0000","last_clean":"2024-09-16T01:36:05.364379+0000","last_became_active":"2024-09-16T01:36:05.361408+0000","last_became_peered":"2024-09-16T01:36:05.361408+0000","last_unstale":"2024-09-16T01:36:05.364379+0000","last_undegraded":"2024-09-16T01:36:05.364379+0000","last_fullsized":"2024-09-16T01:36:05.364379+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:30:10.270905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352248+0000","last_change":"2024-09-16T01:36:05.352248+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352248+0000","last_undegraded":"2024-09-16T01:36:05.352248+0000","last_fullsized":"2024-09-16T01:36:05.352248+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.365065+0000","last_change":"2024-09-16T01:36:05.363734+0000","last_active":"2024-09-16T01:36:05.365065+0000","last_peered":"2024-09-16T01:36:05.365065+0000","last_clean":"2024-09-16T01:36:05.365065+0000","last_became_active":"2024-09-16T01:36:05.363407+0000","last_became_peered":"2024-09-16T01:36:05.363407+0000","last_unstale":"2024-09-16T01:36:05.365065+0000","last_undegraded":"2024-09-16T01:36:05.365065+0000","last_fullsized":"2024-09-16T01:36:05.365065+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:56:30.024164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351047+0000","last_change":"2024-09-16T01:36:05.351047+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351047+0000","last_undegraded":"2024-09-16T01:36:05.351047+0000","last_fullsized":"2024-09-16T01:36:05.351047+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"44'1","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.374827+0000","last_change":"2024-09-16T01:36:05.360643+0000","last_active":"2024-09-16T01:36:06.374827+0000","last_peered":"2024-09-16T01:36:06.374827+0000","last_clean":"2024-09-16T01:36:06.374827+0000","last_became_active":"2024-09-16T01:36:05.360204+0000","last_became_peered":"2024-09-16T01:36:05.360204+0000","last_unstale":"2024-09-16T01:36:06.374827+0000","last_undegraded":"2024-09-16T01:36:06.374827+0000","last_fullsized":"2024-09-16T01:36:06.374827+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:54.497479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351796+0000","last_change":"2024-09-16T01:36:05.351796+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351796+0000","last_undegraded":"2024-09-16T01:36:05.351796+0000","last_fullsized":"2024-09-16T01:36:05.351796+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368891+0000","last_change":"2024-09-16T01:36:05.363498+0000","last_active":"2024-09-16T01:36:05.368891+0000","last_peered":"2024-09-16T01:36:05.368891+0000","last_clean":"2024-09-16T01:36:05.368891+0000","last_became_active":"2024-09-16T01:36:05.363185+0000","last_became_peered":"2024-09-16T01:36:05.363185+0000","last_unstale":"2024-09-16T01:36:05.368891+0000","last_undegraded":"2024-09-16T01:36:05.368891+0000","last_fullsized":"2024-09-16T01:36:05.368891+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:29:00.841831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356095+0000","last_change":"2024-09-16T01:36:05.356095+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356095+0000","last_undegraded":"2024-09-16T01:36:05.356095+0000","last_fullsized":"2024-09-16T01:36:05.356095+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368786+0000","last_change":"2024-09-16T01:36:05.361868+0000","last_active":"2024-09-16T01:36:05.368786+0000","last_peered":"2024-09-16T01:36:05.368786+0000","last_clean":"2024-09-16T01:36:05.368786+0000","last_became_active":"2024-09-16T01:36:05.361605+0000","last_became_peered":"2024-09-16T01:36:05.361605+0000","last_unstale":"2024-09-16T01:36:05.368786+0000","last_undegraded":"2024-09-16T01:36:05.368786+0000","last_fullsized":"2024-09-16T01:36:05.368786+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:52:52.798634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353015+0000","last_change":"2024-09-16T01:36:05.353015+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353015+0000","last_undegraded":"2024-09-16T01:36:05.353015+0000","last_fullsized":"2024-09-16T01:36:05.353015+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.357525+0000","last_change":"2024-09-16T01:36:05.356369+0000","last_active":"2024-09-16T01:36:06.357525+0000","last_peered":"2024-09-16T01:36:06.357525+0000","last_clean":"2024-09-16T01:36:06.357525+0000","last_became_active":"2024-09-16T01:36:05.356027+0000","last_became_peered":"2024-09-16T01:36:05.356027+0000","last_unstale":"2024-09-16T01:36:06.357525+0000","last_undegraded":"2024-09-16T01:36:06.357525+0000","last_fullsized":"2024-09-16T01:36:06.357525+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:02.446309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352450+0000","last_change":"2024-09-16T01:36:05.352450+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352450+0000","last_undegraded":"2024-09-16T01:36:05.352450+0000","last_fullsized":"2024-09-16T01:36:05.352450+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.364524+0000","last_change":"2024-09-16T01:36:05.363731+0000","last_active":"2024-09-16T01:36:05.364524+0000","last_peered":"2024-09-16T01:36:05.364524+0000","last_clean":"2024-09-16T01:36:05.364524+0000","last_became_active":"2024-09-16T01:36:05.363475+0000","last_became_peered":"2024-09-16T01:36:05.363475+0000","last_unstale":"2024-09-16T01:36:05.364524+0000","last_undegraded":"2024-09-16T01:36:05.364524+0000","last_fullsized":"2024-09-16T01:36:05.364524+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:28:42.397852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353294+0000","last_change":"2024-09-16T01:36:05.353294+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353294+0000","last_undegraded":"2024-09-16T01:36:05.353294+0000","last_fullsized":"2024-09-16T01:36:05.353294+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373466+0000","last_change":"2024-09-16T01:36:05.367836+0000","last_active":"2024-09-16T01:36:05.373466+0000","last_peered":"2024-09-16T01:36:05.373466+0000","last_clean":"2024-09-16T01:36:05.373466+0000","last_became_active":"2024-09-16T01:36:05.367513+0000","last_became_peered":"2024-09-16T01:36:05.367513+0000","last_unstale":"2024-09-16T01:36:05.373466+0000","last_undegraded":"2024-09-16T01:36:05.373466+0000","last_fullsized":"2024-09-16T01:36:05.373466+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:53:48.685089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351224+0000","last_change":"2024-09-16T01:36:05.351224+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351224+0000","last_undegraded":"2024-09-16T01:36:05.351224+0000","last_fullsized":"2024-09-16T01:36:05.351224+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368934+0000","last_change":"2024-09-16T01:36:05.365994+0000","last_active":"2024-09-16T01:36:05.368934+0000","last_peered":"2024-09-16T01:36:05.368934+0000","last_clean":"2024-09-16T01:36:05.368934+0000","last_became_active":"2024-09-16T01:36:05.365629+0000","last_became_peered":"2024-09-16T01:36:05.365629+0000","last_unstale":"2024-09-16T01:36:05.368934+0000","last_undegraded":"2024-09-16T01:36:05.368934+0000","last_fullsized":"2024-09-16T01:36:05.368934+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:44:58.819992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352527+0000","last_change":"2024-09-16T01:36:05.352527+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352527+0000","last_undegraded":"2024-09-16T01:36:05.352527+0000","last_fullsized":"2024-09-16T01:36:05.352527+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373583+0000","last_change":"2024-09-16T01:36:05.370944+0000","last_active":"2024-09-16T01:36:05.373583+0000","last_peered":"2024-09-16T01:36:05.373583+0000","last_clean":"2024-09-16T01:36:05.373583+0000","last_became_active":"2024-09-16T01:36:05.370631+0000","last_became_peered":"2024-09-16T01:36:05.370631+0000","last_unstale":"2024-09-16T01:36:05.373583+0000","last_undegraded":"2024-09-16T01:36:05.373583+0000","last_fullsized":"2024-09-16T01:36:05.373583+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:02.446309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.359667+0000","last_change":"2024-09-16T01:36:05.359667+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.359667+0000","last_undegraded":"2024-09-16T01:36:05.359667+0000","last_fullsized":"2024-09-16T01:36:05.359667+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368831+0000","last_change":"2024-09-16T01:36:05.365868+0000","last_active":"2024-09-16T01:36:05.368831+0000","last_peered":"2024-09-16T01:36:05.368831+0000","last_clean":"2024-09-16T01:36:05.368831+0000","last_became_active":"2024-09-16T01:36:05.365481+0000","last_became_peered":"2024-09-16T01:36:05.365481+0000","last_unstale":"2024-09-16T01:36:05.368831+0000","last_undegraded":"2024-09-16T01:36:05.368831+0000","last_fullsized":"2024-09-16T01:36:05.368831+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:26:50.407545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.354612+0000","last_change":"2024-09-16T01:36:05.354612+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.354612+0000","last_undegraded":"2024-09-16T01:36:05.354612+0000","last_fullsized":"2024-09-16T01:36:05.354612+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373531+0000","last_change":"2024-09-16T01:36:05.368272+0000","last_active":"2024-09-16T01:36:05.373531+0000","last_peered":"2024-09-16T01:36:05.373531+0000","last_clean":"2024-09-16T01:36:05.373531+0000","last_became_active":"2024-09-16T01:36:05.367973+0000","last_became_peered":"2024-09-16T01:36:05.367973+0000","last_unstale":"2024-09-16T01:36:05.373531+0000","last_undegraded":"2024-09-16T01:36:05.373531+0000","last_fullsized":"2024-09-16T01:36:05.373531+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:09:24.703051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.355177+0000","last_change":"2024-09-16T01:36:05.355177+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.355177+0000","last_undegraded":"2024-09-16T01:36:05.355177+0000","last_fullsized":"2024-09-16T01:36:05.355177+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.365212+0000","last_change":"2024-09-16T01:36:05.362224+0000","last_active":"2024-09-16T01:36:05.365212+0000","last_peered":"2024-09-16T01:36:05.365212+0000","last_clean":"2024-09-16T01:36:05.365212+0000","last_became_active":"2024-09-16T01:36:05.361853+0000","last_became_peered":"2024-09-16T01:36:05.361853+0000","last_unstale":"2024-09-16T01:36:05.365212+0000","last_undegraded":"2024-09-16T01:36:05.365212+0000","last_fullsized":"2024-09-16T01:36:05.365212+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:15:05.347700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.369963+0000","last_change":"2024-09-16T01:36:06.363427+0000","last_active":"2024-09-16T01:36:06.369963+0000","last_peered":"2024-09-16T01:36:06.369963+0000","last_clean":"2024-09-16T01:36:06.369963+0000","last_became_active":"2024-09-16T01:36:06.363142+0000","last_became_peered":"2024-09-16T01:36:06.363142+0000","last_unstale":"2024-09-16T01:36:06.369963+0000","last_undegraded":"2024-09-16T01:36:06.369963+0000","last_fullsized":"2024-09-16T01:36:06.369963+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:22.762911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.366175+0000","last_change":"2024-09-16T01:36:05.362621+0000","last_active":"2024-09-16T01:36:05.366175+0000","last_peered":"2024-09-16T01:36:05.366175+0000","last_clean":"2024-09-16T01:36:05.366175+0000","last_became_active":"2024-09-16T01:36:05.362300+0000","last_became_peered":"2024-09-16T01:36:05.362300+0000","last_unstale":"2024-09-16T01:36:05.366175+0000","last_undegraded":"2024-09-16T01:36:05.366175+0000","last_fullsized":"2024-09-16T01:36:05.366175+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:31:24.708078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.350199+0000","last_change":"2024-09-16T01:36:05.350199+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.350199+0000","last_undegraded":"2024-09-16T01:36:05.350199+0000","last_fullsized":"2024-09-16T01:36:05.350199+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.363008+0000","last_change":"2024-09-16T01:36:05.362040+0000","last_active":"2024-09-16T01:36:05.363008+0000","last_peered":"2024-09-16T01:36:05.363008+0000","last_clean":"2024-09-16T01:36:05.363008+0000","last_became_active":"2024-09-16T01:36:05.361767+0000","last_became_peered":"2024-09-16T01:36:05.361767+0000","last_unstale":"2024-09-16T01:36:05.363008+0000","last_undegraded":"2024-09-16T01:36:05.363008+0000","last_fullsized":"2024-09-16T01:36:05.363008+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:49:52.496809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353674+0000","last_change":"2024-09-16T01:36:05.353674+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353674+0000","last_undegraded":"2024-09-16T01:36:05.353674+0000","last_fullsized":"2024-09-16T01:36:05.353674+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"44'2","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.377763+0000","last_change":"2024-09-16T01:36:05.361909+0000","last_active":"2024-09-16T01:36:06.377763+0000","last_peered":"2024-09-16T01:36:06.377763+0000","last_clean":"2024-09-16T01:36:06.377763+0000","last_became_active":"2024-09-16T01:36:05.361632+0000","last_became_peered":"2024-09-16T01:36:05.361632+0000","last_unstale":"2024-09-16T01:36:06.377763+0000","last_undegraded":"2024-09-16T01:36:06.377763+0000","last_fullsized":"2024-09-16T01:36:06.377763+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:55:37.303541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":209,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.361317+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:36:05.361317+0000","last_peered":"2024-09-16T01:36:05.361317+0000","last_clean":"2024-09-16T01:36:05.361317+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:36:05.361317+0000","last_undegraded":"2024-09-16T01:36:05.361317+0000","last_fullsized":"2024-09-16T01:36:05.361317+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":70,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509468,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27520,"kb_used_data":488,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344320,"statfs":{"total":96636764160,"available":96608583680,"internally_reserved":0,"allocated":499712,"data_stored":199112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542171,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28088,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343752,"statfs":{"total":96636764160,"available":96608002048,"internally_reserved":0,"allocated":1081344,"data_stored":783213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542171,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":487424,"data_stored":192845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542171,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":487424,"data_stored":192845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542170,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27512,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344328,"statfs":{"total":96636764160,"available":96608591872,"internally_reserved":0,"allocated":487424,"data_stored":192845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542171,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":487424,"data_stored":192845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542170,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27504,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344336,"statfs":{"total":96636764160,"available":96608600064,"internally_reserved":0,"allocated":487424,"data_stored":192845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542170,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28092,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343748,"statfs":{"total":96636764160,"available":96607997952,"internally_reserved":0,"allocated":1081344,"data_stored":783213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:36:09.300 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:36:09.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.434+0000 7fae42952640 1 Processor -- start 2024-09-16T01:36:09.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.434+0000 7fae42952640 1 -- start start 2024-09-16T01:36:09.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.435+0000 7fae42952640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fae3c15c2b0 0x7fae3c0b6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:09.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.435+0000 7fae42952640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae3c0b72f0 0x7fae3c15eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:09.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.435+0000 7fae42952640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae3c15fb10 con 0x7fae3c15f030 2024-09-16T01:36:09.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.435+0000 7fae42952640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fae3c0b5900 con 0x7fae3c15c2b0 2024-09-16T01:36:09.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.435+0000 7fae42952640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fae3c15f830 con 0x7fae3c0b72f0 2024-09-16T01:36:09.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.435+0000 7fae40eef640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fae3c15f030 0x7fae3c06f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59430/0 (socket says 172.21.15.88:59430) 2024-09-16T01:36:09.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.435+0000 7fae40eef640 1 -- 172.21.15.88:0/2031527432 learned_addr learned my addr 172.21.15.88:0/2031527432 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:09.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.435+0000 7fae3affd640 1 -- 172.21.15.88:0/2031527432 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1467339531 0 0) 0x7fae3c15fb10 con 0x7fae3c15f030 2024-09-16T01:36:09.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.435+0000 7fae3affd640 1 -- 172.21.15.88:0/2031527432 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae24003520 con 0x7fae3c15f030 2024-09-16T01:36:09.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.435+0000 7fae3bfff640 1 --2- 172.21.15.88:0/2031527432 >> v2:172.21.15.154:3300/0 conn(0x7fae3c15c2b0 0x7fae3c0b6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:09.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.435+0000 7fae3b7fe640 1 --2- 172.21.15.88:0/2031527432 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae3c0b72f0 0x7fae3c15eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:09.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.436+0000 7fae3affd640 1 -- 172.21.15.88:0/2031527432 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2700465627 0 0) 0x7fae24003520 con 0x7fae3c15f030 2024-09-16T01:36:09.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.436+0000 7fae3affd640 1 -- 172.21.15.88:0/2031527432 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae3c0b72f0 msgr2=0x7fae3c15eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:09.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.436+0000 7fae3affd640 1 --2- 172.21.15.88:0/2031527432 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae3c0b72f0 0x7fae3c15eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.436+0000 7fae3affd640 1 -- 172.21.15.88:0/2031527432 >> v2:172.21.15.154:3300/0 conn(0x7fae3c15c2b0 msgr2=0x7fae3c0b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:09.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.436+0000 7fae3affd640 1 --2- 172.21.15.88:0/2031527432 >> v2:172.21.15.154:3300/0 conn(0x7fae3c15c2b0 0x7fae3c0b6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.436+0000 7fae3affd640 1 -- 172.21.15.88:0/2031527432 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae3c06ce30 con 0x7fae3c15f030 2024-09-16T01:36:09.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.436+0000 7fae3affd640 1 -- 172.21.15.88:0/2031527432 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fae340019f0 con 0x7fae3c15f030 2024-09-16T01:36:09.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.436+0000 7fae3affd640 1 -- 172.21.15.88:0/2031527432 >> v1:172.21.15.88:6789/0 conn(0x7fae3c15f030 legacy=0x7fae3c06f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:09.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.436+0000 7fae3affd640 1 --2- 172.21.15.88:0/2031527432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae240039b0 0x7fae24003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:09.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.436+0000 7fae3affd640 1 --2- 172.21.15.88:0/2031527432 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae3c0b72f0 0x7fae24004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:09.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.436+0000 7fae3affd640 1 --2- 172.21.15.88:0/2031527432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae24004d40 0x7fae240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:09.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.436+0000 7fae3affd640 1 -- 172.21.15.88:0/2031527432 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fae3c0b5900 con 0x7fae240039b0 2024-09-16T01:36:09.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.436+0000 7fae3affd640 1 -- 172.21.15.88:0/2031527432 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fae3c15f830 con 0x7fae24004d40 2024-09-16T01:36:09.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.436+0000 7fae3affd640 1 -- 172.21.15.88:0/2031527432 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fae2400b9c0 con 0x7fae3c0b72f0 2024-09-16T01:36:09.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.436+0000 7fae3bfff640 1 --2- 172.21.15.88:0/2031527432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae240039b0 0x7fae24003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:09.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.436+0000 7fae40eef640 1 --2- 172.21.15.88:0/2031527432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae24004d40 0x7fae240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:09.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.436+0000 7fae3b7fe640 1 --2- 172.21.15.88:0/2031527432 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae3c0b72f0 0x7fae24004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:09.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.436+0000 7fae3bfff640 1 -- 172.21.15.88:0/2031527432 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae3c0b72f0 msgr2=0x7fae24004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:09.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.436+0000 7fae3bfff640 1 --2- 172.21.15.88:0/2031527432 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae3c0b72f0 0x7fae24004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.436+0000 7fae3bfff640 1 -- 172.21.15.88:0/2031527432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae24004d40 msgr2=0x7fae240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:09.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.436+0000 7fae3bfff640 1 --2- 172.21.15.88:0/2031527432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae24004d40 0x7fae240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.436+0000 7fae3bfff640 1 -- 172.21.15.88:0/2031527432 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae24003520 con 0x7fae240039b0 2024-09-16T01:36:09.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.437+0000 7fae3bfff640 1 --2- 172.21.15.88:0/2031527432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae240039b0 0x7fae24003dd0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fae2c009ac0 tx=0x7fae24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12c655d8eb46862 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:09.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.437+0000 7fae3affd640 1 -- 172.21.15.88:0/2031527432 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2c036070 con 0x7fae240039b0 2024-09-16T01:36:09.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.437+0000 7fae3affd640 1 -- 172.21.15.88:0/2031527432 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae2c031400 con 0x7fae240039b0 2024-09-16T01:36:09.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.437+0000 7fae3affd640 1 -- 172.21.15.88:0/2031527432 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2c009690 con 0x7fae240039b0 2024-09-16T01:36:09.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.437+0000 7fae42952640 1 -- 172.21.15.88:0/2031527432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae240039b0 msgr2=0x7fae24003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:09.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.437+0000 7fae42952640 1 --2- 172.21.15.88:0/2031527432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae240039b0 0x7fae24003dd0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fae2c009ac0 tx=0x7fae24005830 comp rx=0 tx=0).stop 2024-09-16T01:36:09.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.437+0000 7fae40eef640 1 -- 172.21.15.88:0/2031527432 reap_dead start 2024-09-16T01:36:09.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.437+0000 7fae42952640 1 -- 172.21.15.88:0/2031527432 shutdown_connections 2024-09-16T01:36:09.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.437+0000 7fae42952640 1 -- 172.21.15.88:0/2031527432 >> 172.21.15.88:0/2031527432 conn(0x7fae3c0b4240 msgr2=0x7fae3c0b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:09.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.438+0000 7fae42952640 1 -- 172.21.15.88:0/2031527432 shutdown_connections 2024-09-16T01:36:09.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.438+0000 7fae42952640 1 -- 172.21.15.88:0/2031527432 wait complete. 2024-09-16T01:36:09.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.439+0000 7fae42952640 1 Processor -- start 2024-09-16T01:36:09.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.439+0000 7fae42952640 1 -- start start 2024-09-16T01:36:09.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.440+0000 7fae42952640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae240039b0 0x7fae3c14aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:09.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.440+0000 7fae42952640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae3c0aae90 0x7fae3c0abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:09.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.440+0000 7fae42952640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae3c0ab310 0x7fae3c0ac500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:09.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.440+0000 7fae42952640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fae3c0b7730 con 0x7fae3c0aae90 2024-09-16T01:36:09.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.440+0000 7fae42952640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fae3c06d3f0 con 0x7fae240039b0 2024-09-16T01:36:09.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.440+0000 7fae42952640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fae3c15f4c0 con 0x7fae3c0ab310 2024-09-16T01:36:09.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.440+0000 7fae3b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae3c0aae90 0x7fae3c0abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:09.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.440+0000 7fae3b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae3c0aae90 0x7fae3c0abfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51098/0 (socket says 172.21.15.88:51098) 2024-09-16T01:36:09.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.440+0000 7fae3b7fe640 1 -- 172.21.15.88:0/485189465 learned_addr learned my addr 172.21.15.88:0/485189465 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:09.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.440+0000 7fae40eef640 1 --2- 172.21.15.88:0/485189465 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae3c0ab310 0x7fae3c0ac500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:09.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.440+0000 7fae3bfff640 1 --2- 172.21.15.88:0/485189465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae240039b0 0x7fae3c14aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:09.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.440+0000 7fae3b7fe640 1 -- 172.21.15.88:0/485189465 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae3c0ab310 msgr2=0x7fae3c0ac500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:09.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.440+0000 7fae3b7fe640 1 --2- 172.21.15.88:0/485189465 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae3c0ab310 0x7fae3c0ac500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.440+0000 7fae3b7fe640 1 -- 172.21.15.88:0/485189465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae240039b0 msgr2=0x7fae3c14aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:09.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.440+0000 7fae3b7fe640 1 --2- 172.21.15.88:0/485189465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae240039b0 0x7fae3c14aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.440+0000 7fae3b7fe640 1 -- 172.21.15.88:0/485189465 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae3c0ab8e0 con 0x7fae3c0aae90 2024-09-16T01:36:09.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.440+0000 7fae3b7fe640 1 --2- 172.21.15.88:0/485189465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae3c0aae90 0x7fae3c0abfc0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fae30011600 tx=0x7fae30011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:09.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.440+0000 7fae38ff9640 1 -- 172.21.15.88:0/485189465 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae3000f920 con 0x7fae3c0aae90 2024-09-16T01:36:09.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.440+0000 7fae38ff9640 1 -- 172.21.15.88:0/485189465 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae3000fac0 con 0x7fae3c0aae90 2024-09-16T01:36:09.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.440+0000 7fae38ff9640 1 -- 172.21.15.88:0/485189465 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae3000fe70 con 0x7fae3c0aae90 2024-09-16T01:36:09.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.440+0000 7fae42952640 1 -- 172.21.15.88:0/485189465 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae3c0abbc0 con 0x7fae3c0aae90 2024-09-16T01:36:09.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.440+0000 7fae42952640 1 -- 172.21.15.88:0/485189465 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae3c14b7a0 con 0x7fae3c0aae90 2024-09-16T01:36:09.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.441+0000 7fae42952640 1 -- 172.21.15.88:0/485189465 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae3c0cfd00 con 0x7fae3c0aae90 2024-09-16T01:36:09.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.442+0000 7fae38ff9640 1 -- 172.21.15.88:0/485189465 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fae30015460 con 0x7fae3c0aae90 2024-09-16T01:36:09.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.442+0000 7fae38ff9640 1 --2- 172.21.15.88:0/485189465 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fae2806c2e0 0x7fae2806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:09.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.442+0000 7fae3bfff640 1 --2- 172.21.15.88:0/485189465 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fae2806c2e0 0x7fae2806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:09.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.442+0000 7fae38ff9640 1 -- 172.21.15.88:0/485189465 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fae3008be40 con 0x7fae3c0aae90 2024-09-16T01:36:09.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.443+0000 7fae3bfff640 1 --2- 172.21.15.88:0/485189465 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fae2806c2e0 0x7fae2806e7a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fae2c033000 tx=0x7fae2c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:09.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.447+0000 7fae38ff9640 1 -- 172.21.15.88:0/485189465 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fae30019030 con 0x7fae3c0aae90 2024-09-16T01:36:09.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.609+0000 7fae42952640 1 -- 172.21.15.88:0/485189465 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fae3c0abd90 con 0x7fae2806c2e0 2024-09-16T01:36:09.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.615+0000 7fae38ff9640 1 -- 172.21.15.88:0/485189465 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+247134 (secure 0 0 0) 0x7fae2c004040 con 0x7fae2806c2e0 2024-09-16T01:36:09.619 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:09.620 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:36:09.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.618+0000 7fae42952640 1 -- 172.21.15.88:0/485189465 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fae2806c2e0 msgr2=0x7fae2806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:09.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.618+0000 7fae42952640 1 --2- 172.21.15.88:0/485189465 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fae2806c2e0 0x7fae2806e7a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fae2c033000 tx=0x7fae2c008030 comp rx=0 tx=0).stop 2024-09-16T01:36:09.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.618+0000 7fae42952640 1 -- 172.21.15.88:0/485189465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae3c0aae90 msgr2=0x7fae3c0abfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:09.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.618+0000 7fae42952640 1 --2- 172.21.15.88:0/485189465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae3c0aae90 0x7fae3c0abfc0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fae30011600 tx=0x7fae30011ac0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.618+0000 7fae42952640 1 -- 172.21.15.88:0/485189465 shutdown_connections 2024-09-16T01:36:09.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.618+0000 7fae42952640 1 --2- 172.21.15.88:0/485189465 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fae2806c2e0 0x7fae2806e7a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.618+0000 7fae42952640 1 --2- 172.21.15.88:0/485189465 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae3c0ab310 0x7fae3c0ac500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.618+0000 7fae42952640 1 --2- 172.21.15.88:0/485189465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae3c0aae90 0x7fae3c0abfc0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.618+0000 7fae42952640 1 --2- 172.21.15.88:0/485189465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae240039b0 0x7fae3c14aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.618+0000 7fae42952640 1 -- 172.21.15.88:0/485189465 >> 172.21.15.88:0/485189465 conn(0x7fae3c0b4240 msgr2=0x7fae3c0cd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:09.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.618+0000 7fae42952640 1 -- 172.21.15.88:0/485189465 shutdown_connections 2024-09-16T01:36:09.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.618+0000 7fae42952640 1 -- 172.21.15.88:0/485189465 wait complete. 2024-09-16T01:36:09.631 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":97,"stamp":"2024-09-16T01:36:08.538350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590438,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":163,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":25,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221240,"kb_used_data":4980,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753480,"statfs":{"total":773094113280,"available":772867563520,"internally_reserved":0,"allocated":5099520,"data_stored":2729763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.195069"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.359868+0000","last_change":"2024-09-16T01:36:05.359868+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.359868+0000","last_undegraded":"2024-09-16T01:36:05.359868+0000","last_fullsized":"2024-09-16T01:36:05.359868+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.370012+0000","last_change":"2024-09-16T01:36:06.366967+0000","last_active":"2024-09-16T01:36:06.370012+0000","last_peered":"2024-09-16T01:36:06.370012+0000","last_clean":"2024-09-16T01:36:06.370012+0000","last_became_active":"2024-09-16T01:36:06.366688+0000","last_became_peered":"2024-09-16T01:36:06.366688+0000","last_unstale":"2024-09-16T01:36:06.370012+0000","last_undegraded":"2024-09-16T01:36:06.370012+0000","last_fullsized":"2024-09-16T01:36:06.370012+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:40:52.576920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352493+0000","last_change":"2024-09-16T01:36:05.352493+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352493+0000","last_undegraded":"2024-09-16T01:36:05.352493+0000","last_fullsized":"2024-09-16T01:36:05.352493+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358690+0000","last_change":"2024-09-16T01:36:05.358690+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358690+0000","last_undegraded":"2024-09-16T01:36:05.358690+0000","last_fullsized":"2024-09-16T01:36:05.358690+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.371287+0000","last_change":"2024-09-16T01:36:06.369570+0000","last_active":"2024-09-16T01:36:06.371287+0000","last_peered":"2024-09-16T01:36:06.371287+0000","last_clean":"2024-09-16T01:36:06.371287+0000","last_became_active":"2024-09-16T01:36:06.369276+0000","last_became_peered":"2024-09-16T01:36:06.369276+0000","last_unstale":"2024-09-16T01:36:06.371287+0000","last_undegraded":"2024-09-16T01:36:06.371287+0000","last_fullsized":"2024-09-16T01:36:06.371287+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:39:28.693098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358381+0000","last_change":"2024-09-16T01:36:05.358381+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358381+0000","last_undegraded":"2024-09-16T01:36:05.358381+0000","last_fullsized":"2024-09-16T01:36:05.358381+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358435+0000","last_change":"2024-09-16T01:36:05.358435+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358435+0000","last_undegraded":"2024-09-16T01:36:05.358435+0000","last_fullsized":"2024-09-16T01:36:05.358435+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358875+0000","last_change":"2024-09-16T01:36:05.358875+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358875+0000","last_undegraded":"2024-09-16T01:36:05.358875+0000","last_fullsized":"2024-09-16T01:36:05.358875+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358586+0000","last_change":"2024-09-16T01:36:05.358586+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358586+0000","last_undegraded":"2024-09-16T01:36:05.358586+0000","last_fullsized":"2024-09-16T01:36:05.358586+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358111+0000","last_change":"2024-09-16T01:36:05.358111+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358111+0000","last_undegraded":"2024-09-16T01:36:05.358111+0000","last_fullsized":"2024-09-16T01:36:05.358111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.357648+0000","last_change":"2024-09-16T01:36:05.357648+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.357648+0000","last_undegraded":"2024-09-16T01:36:05.357648+0000","last_fullsized":"2024-09-16T01:36:05.357648+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.376012+0000","last_change":"2024-09-16T01:36:06.374437+0000","last_active":"2024-09-16T01:36:06.376012+0000","last_peered":"2024-09-16T01:36:06.376012+0000","last_clean":"2024-09-16T01:36:06.376012+0000","last_became_active":"2024-09-16T01:36:06.374156+0000","last_became_peered":"2024-09-16T01:36:06.374156+0000","last_unstale":"2024-09-16T01:36:06.376012+0000","last_undegraded":"2024-09-16T01:36:06.376012+0000","last_fullsized":"2024-09-16T01:36:06.376012+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:14:35.252633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352763+0000","last_change":"2024-09-16T01:36:05.352763+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352763+0000","last_undegraded":"2024-09-16T01:36:05.352763+0000","last_fullsized":"2024-09-16T01:36:05.352763+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356941+0000","last_change":"2024-09-16T01:36:05.356941+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356941+0000","last_undegraded":"2024-09-16T01:36:05.356941+0000","last_fullsized":"2024-09-16T01:36:05.356941+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358500+0000","last_change":"2024-09-16T01:36:05.358500+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358500+0000","last_undegraded":"2024-09-16T01:36:05.358500+0000","last_fullsized":"2024-09-16T01:36:05.358500+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352230+0000","last_change":"2024-09-16T01:36:05.352230+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352230+0000","last_undegraded":"2024-09-16T01:36:05.352230+0000","last_fullsized":"2024-09-16T01:36:05.352230+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351446+0000","last_change":"2024-09-16T01:36:05.351446+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351446+0000","last_undegraded":"2024-09-16T01:36:05.351446+0000","last_fullsized":"2024-09-16T01:36:05.351446+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351350+0000","last_change":"2024-09-16T01:36:05.351350+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351350+0000","last_undegraded":"2024-09-16T01:36:05.351350+0000","last_fullsized":"2024-09-16T01:36:05.351350+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.350520+0000","last_change":"2024-09-16T01:36:05.350520+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.350520+0000","last_undegraded":"2024-09-16T01:36:05.350520+0000","last_fullsized":"2024-09-16T01:36:05.350520+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.354933+0000","last_change":"2024-09-16T01:36:05.354933+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.354933+0000","last_undegraded":"2024-09-16T01:36:05.354933+0000","last_fullsized":"2024-09-16T01:36:05.354933+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358886+0000","last_change":"2024-09-16T01:36:05.358886+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358886+0000","last_undegraded":"2024-09-16T01:36:05.358886+0000","last_fullsized":"2024-09-16T01:36:05.358886+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356521+0000","last_change":"2024-09-16T01:36:05.356521+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356521+0000","last_undegraded":"2024-09-16T01:36:05.356521+0000","last_fullsized":"2024-09-16T01:36:05.356521+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351493+0000","last_change":"2024-09-16T01:36:05.351493+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351493+0000","last_undegraded":"2024-09-16T01:36:05.351493+0000","last_fullsized":"2024-09-16T01:36:05.351493+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.350478+0000","last_change":"2024-09-16T01:36:05.350478+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.350478+0000","last_undegraded":"2024-09-16T01:36:05.350478+0000","last_fullsized":"2024-09-16T01:36:05.350478+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.350770+0000","last_change":"2024-09-16T01:36:05.350770+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.350770+0000","last_undegraded":"2024-09-16T01:36:05.350770+0000","last_fullsized":"2024-09-16T01:36:05.350770+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353568+0000","last_change":"2024-09-16T01:36:05.353568+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353568+0000","last_undegraded":"2024-09-16T01:36:05.353568+0000","last_fullsized":"2024-09-16T01:36:05.353568+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.350845+0000","last_change":"2024-09-16T01:36:05.350845+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.350845+0000","last_undegraded":"2024-09-16T01:36:05.350845+0000","last_fullsized":"2024-09-16T01:36:05.350845+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.370131+0000","last_change":"2024-09-16T01:36:06.365094+0000","last_active":"2024-09-16T01:36:06.370131+0000","last_peered":"2024-09-16T01:36:06.370131+0000","last_clean":"2024-09-16T01:36:06.370131+0000","last_became_active":"2024-09-16T01:36:06.364790+0000","last_became_peered":"2024-09-16T01:36:06.364790+0000","last_unstale":"2024-09-16T01:36:06.370131+0000","last_undegraded":"2024-09-16T01:36:06.370131+0000","last_fullsized":"2024-09-16T01:36:06.370131+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:29:16.206145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351073+0000","last_change":"2024-09-16T01:36:05.351073+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351073+0000","last_undegraded":"2024-09-16T01:36:05.351073+0000","last_fullsized":"2024-09-16T01:36:05.351073+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353729+0000","last_change":"2024-09-16T01:36:05.353729+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353729+0000","last_undegraded":"2024-09-16T01:36:05.353729+0000","last_fullsized":"2024-09-16T01:36:05.353729+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356913+0000","last_change":"2024-09-16T01:36:05.356913+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356913+0000","last_undegraded":"2024-09-16T01:36:05.356913+0000","last_fullsized":"2024-09-16T01:36:05.356913+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351956+0000","last_change":"2024-09-16T01:36:05.351956+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351956+0000","last_undegraded":"2024-09-16T01:36:05.351956+0000","last_fullsized":"2024-09-16T01:36:05.351956+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.354055+0000","last_change":"2024-09-16T01:36:05.354055+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.354055+0000","last_undegraded":"2024-09-16T01:36:05.354055+0000","last_fullsized":"2024-09-16T01:36:05.354055+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.360688+0000","last_change":"2024-09-16T01:36:05.357580+0000","last_active":"2024-09-16T01:36:05.360688+0000","last_peered":"2024-09-16T01:36:05.360688+0000","last_clean":"2024-09-16T01:36:05.360688+0000","last_became_active":"2024-09-16T01:36:05.357331+0000","last_became_peered":"2024-09-16T01:36:05.357331+0000","last_unstale":"2024-09-16T01:36:05.360688+0000","last_undegraded":"2024-09-16T01:36:05.360688+0000","last_fullsized":"2024-09-16T01:36:05.360688+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:28:41.466172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351699+0000","last_change":"2024-09-16T01:36:05.351699+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351699+0000","last_undegraded":"2024-09-16T01:36:05.351699+0000","last_fullsized":"2024-09-16T01:36:05.351699+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.365028+0000","last_change":"2024-09-16T01:36:05.357954+0000","last_active":"2024-09-16T01:36:05.365028+0000","last_peered":"2024-09-16T01:36:05.365028+0000","last_clean":"2024-09-16T01:36:05.365028+0000","last_became_active":"2024-09-16T01:36:05.357018+0000","last_became_peered":"2024-09-16T01:36:05.357018+0000","last_unstale":"2024-09-16T01:36:05.365028+0000","last_undegraded":"2024-09-16T01:36:05.365028+0000","last_fullsized":"2024-09-16T01:36:05.365028+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:53.565250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.378364+0000","last_change":"2024-09-16T01:36:06.377925+0000","last_active":"2024-09-16T01:36:06.378364+0000","last_peered":"2024-09-16T01:36:06.378364+0000","last_clean":"2024-09-16T01:36:06.378364+0000","last_became_active":"2024-09-16T01:36:06.377631+0000","last_became_peered":"2024-09-16T01:36:06.377631+0000","last_unstale":"2024-09-16T01:36:06.378364+0000","last_undegraded":"2024-09-16T01:36:06.378364+0000","last_fullsized":"2024-09-16T01:36:06.378364+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:09:13.085891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368709+0000","last_change":"2024-09-16T01:36:05.365158+0000","last_active":"2024-09-16T01:36:05.368709+0000","last_peered":"2024-09-16T01:36:05.368709+0000","last_clean":"2024-09-16T01:36:05.368709+0000","last_became_active":"2024-09-16T01:36:05.364885+0000","last_became_peered":"2024-09-16T01:36:05.364885+0000","last_unstale":"2024-09-16T01:36:05.368709+0000","last_undegraded":"2024-09-16T01:36:05.368709+0000","last_fullsized":"2024-09-16T01:36:05.368709+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:06:24.080498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.354065+0000","last_change":"2024-09-16T01:36:05.354065+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.354065+0000","last_undegraded":"2024-09-16T01:36:05.354065+0000","last_fullsized":"2024-09-16T01:36:05.354065+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373524+0000","last_change":"2024-09-16T01:36:05.368193+0000","last_active":"2024-09-16T01:36:05.373524+0000","last_peered":"2024-09-16T01:36:05.373524+0000","last_clean":"2024-09-16T01:36:05.373524+0000","last_became_active":"2024-09-16T01:36:05.367907+0000","last_became_peered":"2024-09-16T01:36:05.367907+0000","last_unstale":"2024-09-16T01:36:05.373524+0000","last_undegraded":"2024-09-16T01:36:05.373524+0000","last_fullsized":"2024-09-16T01:36:05.373524+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:40:04.647251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353341+0000","last_change":"2024-09-16T01:36:05.353341+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353341+0000","last_undegraded":"2024-09-16T01:36:05.353341+0000","last_fullsized":"2024-09-16T01:36:05.353341+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368806+0000","last_change":"2024-09-16T01:36:05.361916+0000","last_active":"2024-09-16T01:36:05.368806+0000","last_peered":"2024-09-16T01:36:05.368806+0000","last_clean":"2024-09-16T01:36:05.368806+0000","last_became_active":"2024-09-16T01:36:05.361658+0000","last_became_peered":"2024-09-16T01:36:05.361658+0000","last_unstale":"2024-09-16T01:36:05.368806+0000","last_undegraded":"2024-09-16T01:36:05.368806+0000","last_fullsized":"2024-09-16T01:36:05.368806+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:41.613259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356484+0000","last_change":"2024-09-16T01:36:05.356484+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356484+0000","last_undegraded":"2024-09-16T01:36:05.356484+0000","last_fullsized":"2024-09-16T01:36:05.356484+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373139+0000","last_change":"2024-09-16T01:36:05.365323+0000","last_active":"2024-09-16T01:36:05.373139+0000","last_peered":"2024-09-16T01:36:05.373139+0000","last_clean":"2024-09-16T01:36:05.373139+0000","last_became_active":"2024-09-16T01:36:05.364940+0000","last_became_peered":"2024-09-16T01:36:05.364940+0000","last_unstale":"2024-09-16T01:36:05.373139+0000","last_undegraded":"2024-09-16T01:36:05.373139+0000","last_fullsized":"2024-09-16T01:36:05.373139+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:38:11.910238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353237+0000","last_change":"2024-09-16T01:36:05.353237+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353237+0000","last_undegraded":"2024-09-16T01:36:05.353237+0000","last_fullsized":"2024-09-16T01:36:05.353237+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.357443+0000","last_change":"2024-09-16T01:36:05.359801+0000","last_active":"2024-09-16T01:36:06.357443+0000","last_peered":"2024-09-16T01:36:06.357443+0000","last_clean":"2024-09-16T01:36:06.357443+0000","last_became_active":"2024-09-16T01:36:05.359542+0000","last_became_peered":"2024-09-16T01:36:05.359542+0000","last_unstale":"2024-09-16T01:36:06.357443+0000","last_undegraded":"2024-09-16T01:36:06.357443+0000","last_fullsized":"2024-09-16T01:36:06.357443+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:22.128838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351989+0000","last_change":"2024-09-16T01:36:05.351989+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351989+0000","last_undegraded":"2024-09-16T01:36:05.351989+0000","last_fullsized":"2024-09-16T01:36:05.351989+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.366391+0000","last_change":"2024-09-16T01:36:05.364586+0000","last_active":"2024-09-16T01:36:05.366391+0000","last_peered":"2024-09-16T01:36:05.366391+0000","last_clean":"2024-09-16T01:36:05.366391+0000","last_became_active":"2024-09-16T01:36:05.364271+0000","last_became_peered":"2024-09-16T01:36:05.364271+0000","last_unstale":"2024-09-16T01:36:05.366391+0000","last_undegraded":"2024-09-16T01:36:05.366391+0000","last_fullsized":"2024-09-16T01:36:05.366391+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:15:05.347700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.350838+0000","last_change":"2024-09-16T01:36:05.350838+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.350838+0000","last_undegraded":"2024-09-16T01:36:05.350838+0000","last_fullsized":"2024-09-16T01:36:05.350838+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.370306+0000","last_change":"2024-09-16T01:36:05.364097+0000","last_active":"2024-09-16T01:36:05.370306+0000","last_peered":"2024-09-16T01:36:05.370306+0000","last_clean":"2024-09-16T01:36:05.370306+0000","last_became_active":"2024-09-16T01:36:05.363758+0000","last_became_peered":"2024-09-16T01:36:05.363758+0000","last_unstale":"2024-09-16T01:36:05.370306+0000","last_undegraded":"2024-09-16T01:36:05.370306+0000","last_fullsized":"2024-09-16T01:36:05.370306+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:59.822221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.359339+0000","last_change":"2024-09-16T01:36:05.359339+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.359339+0000","last_undegraded":"2024-09-16T01:36:05.359339+0000","last_fullsized":"2024-09-16T01:36:05.359339+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.365112+0000","last_change":"2024-09-16T01:36:05.363830+0000","last_active":"2024-09-16T01:36:05.365112+0000","last_peered":"2024-09-16T01:36:05.365112+0000","last_clean":"2024-09-16T01:36:05.365112+0000","last_became_active":"2024-09-16T01:36:05.363572+0000","last_became_peered":"2024-09-16T01:36:05.363572+0000","last_unstale":"2024-09-16T01:36:05.365112+0000","last_undegraded":"2024-09-16T01:36:05.365112+0000","last_fullsized":"2024-09-16T01:36:05.365112+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:53.565250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356478+0000","last_change":"2024-09-16T01:36:05.356478+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356478+0000","last_undegraded":"2024-09-16T01:36:05.356478+0000","last_fullsized":"2024-09-16T01:36:05.356478+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373551+0000","last_change":"2024-09-16T01:36:05.368744+0000","last_active":"2024-09-16T01:36:05.373551+0000","last_peered":"2024-09-16T01:36:05.373551+0000","last_clean":"2024-09-16T01:36:05.373551+0000","last_became_active":"2024-09-16T01:36:05.368069+0000","last_became_peered":"2024-09-16T01:36:05.368069+0000","last_unstale":"2024-09-16T01:36:05.373551+0000","last_undegraded":"2024-09-16T01:36:05.373551+0000","last_fullsized":"2024-09-16T01:36:05.373551+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:58.552340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.360574+0000","last_change":"2024-09-16T01:36:05.360574+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.360574+0000","last_undegraded":"2024-09-16T01:36:05.360574+0000","last_fullsized":"2024-09-16T01:36:05.360574+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368680+0000","last_change":"2024-09-16T01:36:05.365975+0000","last_active":"2024-09-16T01:36:05.368680+0000","last_peered":"2024-09-16T01:36:05.368680+0000","last_clean":"2024-09-16T01:36:05.368680+0000","last_became_active":"2024-09-16T01:36:05.365674+0000","last_became_peered":"2024-09-16T01:36:05.365674+0000","last_unstale":"2024-09-16T01:36:05.368680+0000","last_undegraded":"2024-09-16T01:36:05.368680+0000","last_fullsized":"2024-09-16T01:36:05.368680+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:21:23.387226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352330+0000","last_change":"2024-09-16T01:36:05.352330+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352330+0000","last_undegraded":"2024-09-16T01:36:05.352330+0000","last_fullsized":"2024-09-16T01:36:05.352330+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368728+0000","last_change":"2024-09-16T01:36:05.360882+0000","last_active":"2024-09-16T01:36:05.368728+0000","last_peered":"2024-09-16T01:36:05.368728+0000","last_clean":"2024-09-16T01:36:05.368728+0000","last_became_active":"2024-09-16T01:36:05.360521+0000","last_became_peered":"2024-09-16T01:36:05.360521+0000","last_unstale":"2024-09-16T01:36:05.368728+0000","last_undegraded":"2024-09-16T01:36:05.368728+0000","last_fullsized":"2024-09-16T01:36:05.368728+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:05:47.000282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.354770+0000","last_change":"2024-09-16T01:36:05.354770+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.354770+0000","last_undegraded":"2024-09-16T01:36:05.354770+0000","last_fullsized":"2024-09-16T01:36:05.354770+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373578+0000","last_change":"2024-09-16T01:36:05.369476+0000","last_active":"2024-09-16T01:36:05.373578+0000","last_peered":"2024-09-16T01:36:05.373578+0000","last_clean":"2024-09-16T01:36:05.373578+0000","last_became_active":"2024-09-16T01:36:05.369155+0000","last_became_peered":"2024-09-16T01:36:05.369155+0000","last_unstale":"2024-09-16T01:36:05.373578+0000","last_undegraded":"2024-09-16T01:36:05.373578+0000","last_fullsized":"2024-09-16T01:36:05.373578+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:22.128838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356076+0000","last_change":"2024-09-16T01:36:05.356076+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356076+0000","last_undegraded":"2024-09-16T01:36:05.356076+0000","last_fullsized":"2024-09-16T01:36:05.356076+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.366290+0000","last_change":"2024-09-16T01:36:05.359803+0000","last_active":"2024-09-16T01:36:05.366290+0000","last_peered":"2024-09-16T01:36:05.366290+0000","last_clean":"2024-09-16T01:36:05.366290+0000","last_became_active":"2024-09-16T01:36:05.359552+0000","last_became_peered":"2024-09-16T01:36:05.359552+0000","last_unstale":"2024-09-16T01:36:05.366290+0000","last_undegraded":"2024-09-16T01:36:05.366290+0000","last_fullsized":"2024-09-16T01:36:05.366290+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:32:35.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.357949+0000","last_change":"2024-09-16T01:36:05.357949+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.357949+0000","last_undegraded":"2024-09-16T01:36:05.357949+0000","last_fullsized":"2024-09-16T01:36:05.357949+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.362430+0000","last_change":"2024-09-16T01:36:05.359125+0000","last_active":"2024-09-16T01:36:05.362430+0000","last_peered":"2024-09-16T01:36:05.362430+0000","last_clean":"2024-09-16T01:36:05.362430+0000","last_became_active":"2024-09-16T01:36:05.358486+0000","last_became_peered":"2024-09-16T01:36:05.358486+0000","last_unstale":"2024-09-16T01:36:05.362430+0000","last_undegraded":"2024-09-16T01:36:05.362430+0000","last_fullsized":"2024-09-16T01:36:05.362430+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:22:44.509769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353180+0000","last_change":"2024-09-16T01:36:05.353180+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353180+0000","last_undegraded":"2024-09-16T01:36:05.353180+0000","last_fullsized":"2024-09-16T01:36:05.353180+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.364379+0000","last_change":"2024-09-16T01:36:05.361659+0000","last_active":"2024-09-16T01:36:05.364379+0000","last_peered":"2024-09-16T01:36:05.364379+0000","last_clean":"2024-09-16T01:36:05.364379+0000","last_became_active":"2024-09-16T01:36:05.361408+0000","last_became_peered":"2024-09-16T01:36:05.361408+0000","last_unstale":"2024-09-16T01:36:05.364379+0000","last_undegraded":"2024-09-16T01:36:05.364379+0000","last_fullsized":"2024-09-16T01:36:05.364379+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:30:10.270905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352248+0000","last_change":"2024-09-16T01:36:05.352248+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352248+0000","last_undegraded":"2024-09-16T01:36:05.352248+0000","last_fullsized":"2024-09-16T01:36:05.352248+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.365065+0000","last_change":"2024-09-16T01:36:05.363734+0000","last_active":"2024-09-16T01:36:05.365065+0000","last_peered":"2024-09-16T01:36:05.365065+0000","last_clean":"2024-09-16T01:36:05.365065+0000","last_became_active":"2024-09-16T01:36:05.363407+0000","last_became_peered":"2024-09-16T01:36:05.363407+0000","last_unstale":"2024-09-16T01:36:05.365065+0000","last_undegraded":"2024-09-16T01:36:05.365065+0000","last_fullsized":"2024-09-16T01:36:05.365065+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:56:30.024164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351047+0000","last_change":"2024-09-16T01:36:05.351047+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351047+0000","last_undegraded":"2024-09-16T01:36:05.351047+0000","last_fullsized":"2024-09-16T01:36:05.351047+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"44'1","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.374827+0000","last_change":"2024-09-16T01:36:05.360643+0000","last_active":"2024-09-16T01:36:06.374827+0000","last_peered":"2024-09-16T01:36:06.374827+0000","last_clean":"2024-09-16T01:36:06.374827+0000","last_became_active":"2024-09-16T01:36:05.360204+0000","last_became_peered":"2024-09-16T01:36:05.360204+0000","last_unstale":"2024-09-16T01:36:06.374827+0000","last_undegraded":"2024-09-16T01:36:06.374827+0000","last_fullsized":"2024-09-16T01:36:06.374827+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:54.497479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351796+0000","last_change":"2024-09-16T01:36:05.351796+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351796+0000","last_undegraded":"2024-09-16T01:36:05.351796+0000","last_fullsized":"2024-09-16T01:36:05.351796+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368891+0000","last_change":"2024-09-16T01:36:05.363498+0000","last_active":"2024-09-16T01:36:05.368891+0000","last_peered":"2024-09-16T01:36:05.368891+0000","last_clean":"2024-09-16T01:36:05.368891+0000","last_became_active":"2024-09-16T01:36:05.363185+0000","last_became_peered":"2024-09-16T01:36:05.363185+0000","last_unstale":"2024-09-16T01:36:05.368891+0000","last_undegraded":"2024-09-16T01:36:05.368891+0000","last_fullsized":"2024-09-16T01:36:05.368891+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:29:00.841831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356095+0000","last_change":"2024-09-16T01:36:05.356095+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356095+0000","last_undegraded":"2024-09-16T01:36:05.356095+0000","last_fullsized":"2024-09-16T01:36:05.356095+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368786+0000","last_change":"2024-09-16T01:36:05.361868+0000","last_active":"2024-09-16T01:36:05.368786+0000","last_peered":"2024-09-16T01:36:05.368786+0000","last_clean":"2024-09-16T01:36:05.368786+0000","last_became_active":"2024-09-16T01:36:05.361605+0000","last_became_peered":"2024-09-16T01:36:05.361605+0000","last_unstale":"2024-09-16T01:36:05.368786+0000","last_undegraded":"2024-09-16T01:36:05.368786+0000","last_fullsized":"2024-09-16T01:36:05.368786+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:52:52.798634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353015+0000","last_change":"2024-09-16T01:36:05.353015+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353015+0000","last_undegraded":"2024-09-16T01:36:05.353015+0000","last_fullsized":"2024-09-16T01:36:05.353015+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.357525+0000","last_change":"2024-09-16T01:36:05.356369+0000","last_active":"2024-09-16T01:36:06.357525+0000","last_peered":"2024-09-16T01:36:06.357525+0000","last_clean":"2024-09-16T01:36:06.357525+0000","last_became_active":"2024-09-16T01:36:05.356027+0000","last_became_peered":"2024-09-16T01:36:05.356027+0000","last_unstale":"2024-09-16T01:36:06.357525+0000","last_undegraded":"2024-09-16T01:36:06.357525+0000","last_fullsized":"2024-09-16T01:36:06.357525+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:02.446309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352450+0000","last_change":"2024-09-16T01:36:05.352450+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352450+0000","last_undegraded":"2024-09-16T01:36:05.352450+0000","last_fullsized":"2024-09-16T01:36:05.352450+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.364524+0000","last_change":"2024-09-16T01:36:05.363731+0000","last_active":"2024-09-16T01:36:05.364524+0000","last_peered":"2024-09-16T01:36:05.364524+0000","last_clean":"2024-09-16T01:36:05.364524+0000","last_became_active":"2024-09-16T01:36:05.363475+0000","last_became_peered":"2024-09-16T01:36:05.363475+0000","last_unstale":"2024-09-16T01:36:05.364524+0000","last_undegraded":"2024-09-16T01:36:05.364524+0000","last_fullsized":"2024-09-16T01:36:05.364524+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:28:42.397852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353294+0000","last_change":"2024-09-16T01:36:05.353294+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353294+0000","last_undegraded":"2024-09-16T01:36:05.353294+0000","last_fullsized":"2024-09-16T01:36:05.353294+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373466+0000","last_change":"2024-09-16T01:36:05.367836+0000","last_active":"2024-09-16T01:36:05.373466+0000","last_peered":"2024-09-16T01:36:05.373466+0000","last_clean":"2024-09-16T01:36:05.373466+0000","last_became_active":"2024-09-16T01:36:05.367513+0000","last_became_peered":"2024-09-16T01:36:05.367513+0000","last_unstale":"2024-09-16T01:36:05.373466+0000","last_undegraded":"2024-09-16T01:36:05.373466+0000","last_fullsized":"2024-09-16T01:36:05.373466+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:53:48.685089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351224+0000","last_change":"2024-09-16T01:36:05.351224+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351224+0000","last_undegraded":"2024-09-16T01:36:05.351224+0000","last_fullsized":"2024-09-16T01:36:05.351224+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368934+0000","last_change":"2024-09-16T01:36:05.365994+0000","last_active":"2024-09-16T01:36:05.368934+0000","last_peered":"2024-09-16T01:36:05.368934+0000","last_clean":"2024-09-16T01:36:05.368934+0000","last_became_active":"2024-09-16T01:36:05.365629+0000","last_became_peered":"2024-09-16T01:36:05.365629+0000","last_unstale":"2024-09-16T01:36:05.368934+0000","last_undegraded":"2024-09-16T01:36:05.368934+0000","last_fullsized":"2024-09-16T01:36:05.368934+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:44:58.819992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352527+0000","last_change":"2024-09-16T01:36:05.352527+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352527+0000","last_undegraded":"2024-09-16T01:36:05.352527+0000","last_fullsized":"2024-09-16T01:36:05.352527+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373583+0000","last_change":"2024-09-16T01:36:05.370944+0000","last_active":"2024-09-16T01:36:05.373583+0000","last_peered":"2024-09-16T01:36:05.373583+0000","last_clean":"2024-09-16T01:36:05.373583+0000","last_became_active":"2024-09-16T01:36:05.370631+0000","last_became_peered":"2024-09-16T01:36:05.370631+0000","last_unstale":"2024-09-16T01:36:05.373583+0000","last_undegraded":"2024-09-16T01:36:05.373583+0000","last_fullsized":"2024-09-16T01:36:05.373583+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:02.446309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.359667+0000","last_change":"2024-09-16T01:36:05.359667+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.359667+0000","last_undegraded":"2024-09-16T01:36:05.359667+0000","last_fullsized":"2024-09-16T01:36:05.359667+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368831+0000","last_change":"2024-09-16T01:36:05.365868+0000","last_active":"2024-09-16T01:36:05.368831+0000","last_peered":"2024-09-16T01:36:05.368831+0000","last_clean":"2024-09-16T01:36:05.368831+0000","last_became_active":"2024-09-16T01:36:05.365481+0000","last_became_peered":"2024-09-16T01:36:05.365481+0000","last_unstale":"2024-09-16T01:36:05.368831+0000","last_undegraded":"2024-09-16T01:36:05.368831+0000","last_fullsized":"2024-09-16T01:36:05.368831+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:26:50.407545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.354612+0000","last_change":"2024-09-16T01:36:05.354612+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.354612+0000","last_undegraded":"2024-09-16T01:36:05.354612+0000","last_fullsized":"2024-09-16T01:36:05.354612+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373531+0000","last_change":"2024-09-16T01:36:05.368272+0000","last_active":"2024-09-16T01:36:05.373531+0000","last_peered":"2024-09-16T01:36:05.373531+0000","last_clean":"2024-09-16T01:36:05.373531+0000","last_became_active":"2024-09-16T01:36:05.367973+0000","last_became_peered":"2024-09-16T01:36:05.367973+0000","last_unstale":"2024-09-16T01:36:05.373531+0000","last_undegraded":"2024-09-16T01:36:05.373531+0000","last_fullsized":"2024-09-16T01:36:05.373531+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:09:24.703051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.355177+0000","last_change":"2024-09-16T01:36:05.355177+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.355177+0000","last_undegraded":"2024-09-16T01:36:05.355177+0000","last_fullsized":"2024-09-16T01:36:05.355177+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.365212+0000","last_change":"2024-09-16T01:36:05.362224+0000","last_active":"2024-09-16T01:36:05.365212+0000","last_peered":"2024-09-16T01:36:05.365212+0000","last_clean":"2024-09-16T01:36:05.365212+0000","last_became_active":"2024-09-16T01:36:05.361853+0000","last_became_peered":"2024-09-16T01:36:05.361853+0000","last_unstale":"2024-09-16T01:36:05.365212+0000","last_undegraded":"2024-09-16T01:36:05.365212+0000","last_fullsized":"2024-09-16T01:36:05.365212+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:15:05.347700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.369963+0000","last_change":"2024-09-16T01:36:06.363427+0000","last_active":"2024-09-16T01:36:06.369963+0000","last_peered":"2024-09-16T01:36:06.369963+0000","last_clean":"2024-09-16T01:36:06.369963+0000","last_became_active":"2024-09-16T01:36:06.363142+0000","last_became_peered":"2024-09-16T01:36:06.363142+0000","last_unstale":"2024-09-16T01:36:06.369963+0000","last_undegraded":"2024-09-16T01:36:06.369963+0000","last_fullsized":"2024-09-16T01:36:06.369963+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:22.762911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.366175+0000","last_change":"2024-09-16T01:36:05.362621+0000","last_active":"2024-09-16T01:36:05.366175+0000","last_peered":"2024-09-16T01:36:05.366175+0000","last_clean":"2024-09-16T01:36:05.366175+0000","last_became_active":"2024-09-16T01:36:05.362300+0000","last_became_peered":"2024-09-16T01:36:05.362300+0000","last_unstale":"2024-09-16T01:36:05.366175+0000","last_undegraded":"2024-09-16T01:36:05.366175+0000","last_fullsized":"2024-09-16T01:36:05.366175+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:31:24.708078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.350199+0000","last_change":"2024-09-16T01:36:05.350199+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.350199+0000","last_undegraded":"2024-09-16T01:36:05.350199+0000","last_fullsized":"2024-09-16T01:36:05.350199+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.363008+0000","last_change":"2024-09-16T01:36:05.362040+0000","last_active":"2024-09-16T01:36:05.363008+0000","last_peered":"2024-09-16T01:36:05.363008+0000","last_clean":"2024-09-16T01:36:05.363008+0000","last_became_active":"2024-09-16T01:36:05.361767+0000","last_became_peered":"2024-09-16T01:36:05.361767+0000","last_unstale":"2024-09-16T01:36:05.363008+0000","last_undegraded":"2024-09-16T01:36:05.363008+0000","last_fullsized":"2024-09-16T01:36:05.363008+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:49:52.496809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353674+0000","last_change":"2024-09-16T01:36:05.353674+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353674+0000","last_undegraded":"2024-09-16T01:36:05.353674+0000","last_fullsized":"2024-09-16T01:36:05.353674+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"44'2","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.377763+0000","last_change":"2024-09-16T01:36:05.361909+0000","last_active":"2024-09-16T01:36:06.377763+0000","last_peered":"2024-09-16T01:36:06.377763+0000","last_clean":"2024-09-16T01:36:06.377763+0000","last_became_active":"2024-09-16T01:36:05.361632+0000","last_became_peered":"2024-09-16T01:36:05.361632+0000","last_unstale":"2024-09-16T01:36:06.377763+0000","last_undegraded":"2024-09-16T01:36:06.377763+0000","last_fullsized":"2024-09-16T01:36:06.377763+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:55:37.303541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":209,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.361317+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:36:05.361317+0000","last_peered":"2024-09-16T01:36:05.361317+0000","last_clean":"2024-09-16T01:36:05.361317+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:36:05.361317+0000","last_undegraded":"2024-09-16T01:36:05.361317+0000","last_fullsized":"2024-09-16T01:36:05.361317+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":70,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509468,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27520,"kb_used_data":488,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344320,"statfs":{"total":96636764160,"available":96608583680,"internally_reserved":0,"allocated":499712,"data_stored":199112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542171,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28088,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343752,"statfs":{"total":96636764160,"available":96608002048,"internally_reserved":0,"allocated":1081344,"data_stored":783213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542171,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":487424,"data_stored":192845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542171,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":487424,"data_stored":192845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542170,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27512,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344328,"statfs":{"total":96636764160,"available":96608591872,"internally_reserved":0,"allocated":487424,"data_stored":192845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542171,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":487424,"data_stored":192845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542170,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27504,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344336,"statfs":{"total":96636764160,"available":96608600064,"internally_reserved":0,"allocated":487424,"data_stored":192845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542170,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28092,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343748,"statfs":{"total":96636764160,"available":96607997952,"internally_reserved":0,"allocated":1081344,"data_stored":783213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:36:09.634 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:36:09.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.768+0000 7f4c2f047640 1 Processor -- start 2024-09-16T01:36:09.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.768+0000 7f4c2f047640 1 -- start start 2024-09-16T01:36:09.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.769+0000 7f4c2f047640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4c280715e0 0x7f4c280b6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:09.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.769+0000 7f4c2f047640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c280d0270 con 0x7f4c2815c550 2024-09-16T01:36:09.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.769+0000 7f4c2f047640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c28071a50 con 0x7f4c28073160 2024-09-16T01:36:09.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.769+0000 7f4c2f047640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f4c280b7350 con 0x7f4c280715e0 2024-09-16T01:36:09.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.769+0000 7f4c27fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4c280715e0 0x7f4c280b6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:09.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.769+0000 7f4c27fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4c280715e0 0x7f4c280b6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51112/0 (socket says 172.21.15.88:51112) 2024-09-16T01:36:09.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.769+0000 7f4c27fff640 1 -- 172.21.15.88:0/1508368537 learned_addr learned my addr 172.21.15.88:0/1508368537 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:09.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.769+0000 7f4c27fff640 1 -- 172.21.15.88:0/1508368537 >> v1:172.21.15.154:6789/0 conn(0x7f4c28073160 legacy=0x7f4c280d0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:09.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.769+0000 7f4c27fff640 1 -- 172.21.15.88:0/1508368537 >> v1:172.21.15.88:6789/0 conn(0x7f4c2815c550 legacy=0x7f4c280b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:09.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.770+0000 7f4c27fff640 1 -- 172.21.15.88:0/1508368537 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c280d04f0 con 0x7f4c280715e0 2024-09-16T01:36:09.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.770+0000 7f4c27fff640 1 --2- 172.21.15.88:0/1508368537 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c280715e0 0x7f4c280b6e10 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f4c10009870 tx=0x7f4c1002a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=990c3f4181eb35f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:09.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.770+0000 7f4c277fe640 1 -- 172.21.15.88:0/1508368537 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c10037070 con 0x7f4c280715e0 2024-09-16T01:36:09.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.770+0000 7f4c277fe640 1 -- 172.21.15.88:0/1508368537 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c10032440 con 0x7f4c280715e0 2024-09-16T01:36:09.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.770+0000 7f4c277fe640 1 -- 172.21.15.88:0/1508368537 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c10032810 con 0x7f4c280715e0 2024-09-16T01:36:09.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.771+0000 7f4c2f047640 1 -- 172.21.15.88:0/1508368537 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c280715e0 msgr2=0x7f4c280b6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:09.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.771+0000 7f4c2f047640 1 --2- 172.21.15.88:0/1508368537 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c280715e0 0x7f4c280b6e10 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f4c10009870 tx=0x7f4c1002a2a0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.771+0000 7f4c2f047640 1 -- 172.21.15.88:0/1508368537 shutdown_connections 2024-09-16T01:36:09.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.771+0000 7f4c2f047640 1 --2- 172.21.15.88:0/1508368537 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c280715e0 0x7f4c280b6e10 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.771+0000 7f4c2f047640 1 -- 172.21.15.88:0/1508368537 >> 172.21.15.88:0/1508368537 conn(0x7f4c280b4240 msgr2=0x7f4c280b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:09.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.771+0000 7f4c2f047640 1 -- 172.21.15.88:0/1508368537 shutdown_connections 2024-09-16T01:36:09.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.771+0000 7f4c2f047640 1 -- 172.21.15.88:0/1508368537 wait complete. 2024-09-16T01:36:09.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.772+0000 7f4c2f047640 1 Processor -- start 2024-09-16T01:36:09.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.772+0000 7f4c2f047640 1 -- start start 2024-09-16T01:36:09.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.773+0000 7f4c2f047640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c280715e0 0x7f4c28147f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:09.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.773+0000 7f4c2f047640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c28073160 0x7f4c280d2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:09.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.773+0000 7f4c2f047640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c2815c550 0x7f4c280d29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:09.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.773+0000 7f4c2f047640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4c280b7860 con 0x7f4c280715e0 2024-09-16T01:36:09.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.773+0000 7f4c2f047640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2815c930 con 0x7f4c2815c550 2024-09-16T01:36:09.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.773+0000 7f4c2f047640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4c280d07b0 con 0x7f4c28073160 2024-09-16T01:36:09.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.773+0000 7f4c2cde3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c280715e0 0x7f4c28147f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:09.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.773+0000 7f4c2cde3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c280715e0 0x7f4c28147f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51114/0 (socket says 172.21.15.88:51114) 2024-09-16T01:36:09.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.773+0000 7f4c2cde3640 1 -- 172.21.15.88:0/4291067538 learned_addr learned my addr 172.21.15.88:0/4291067538 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:09.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.773+0000 7f4c2cde3640 1 -- 172.21.15.88:0/4291067538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c28073160 msgr2=0x7f4c280d2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:09.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.773+0000 7f4c2cde3640 1 --2- 172.21.15.88:0/4291067538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c28073160 0x7f4c280d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.773+0000 7f4c2cde3640 1 -- 172.21.15.88:0/4291067538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c2815c550 msgr2=0x7f4c280d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:09.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.773+0000 7f4c2cde3640 1 --2- 172.21.15.88:0/4291067538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c2815c550 0x7f4c280d29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.773+0000 7f4c2cde3640 1 -- 172.21.15.88:0/4291067538 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c281f6540 con 0x7f4c280715e0 2024-09-16T01:36:09.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.774+0000 7f4c2cde3640 1 --2- 172.21.15.88:0/4291067538 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c280715e0 0x7f4c28147f50 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4c1800c790 tx=0x7f4c1800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:09.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.774+0000 7f4c257fa640 1 -- 172.21.15.88:0/4291067538 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c18016020 con 0x7f4c280715e0 2024-09-16T01:36:09.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.774+0000 7f4c257fa640 1 -- 172.21.15.88:0/4291067538 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c18006d50 con 0x7f4c280715e0 2024-09-16T01:36:09.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.774+0000 7f4c2f047640 1 -- 172.21.15.88:0/4291067538 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c281f59c0 con 0x7f4c280715e0 2024-09-16T01:36:09.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.774+0000 7f4c257fa640 1 -- 172.21.15.88:0/4291067538 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c18003bb0 con 0x7f4c280715e0 2024-09-16T01:36:09.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.774+0000 7f4c2f047640 1 -- 172.21.15.88:0/4291067538 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c281f5f80 con 0x7f4c280715e0 2024-09-16T01:36:09.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.775+0000 7f4c2f047640 1 -- 172.21.15.88:0/4291067538 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c280b6e10 con 0x7f4c280715e0 2024-09-16T01:36:09.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.775+0000 7f4c257fa640 1 -- 172.21.15.88:0/4291067538 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4c1800a850 con 0x7f4c280715e0 2024-09-16T01:36:09.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.776+0000 7f4c257fa640 1 --2- 172.21.15.88:0/4291067538 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c0806c2e0 0x7f4c0806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:09.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.776+0000 7f4c257fa640 1 -- 172.21.15.88:0/4291067538 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f4c1808b080 con 0x7f4c280715e0 2024-09-16T01:36:09.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.776+0000 7f4c27fff640 1 --2- 172.21.15.88:0/4291067538 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c0806c2e0 0x7f4c0806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:09.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.776+0000 7f4c27fff640 1 --2- 172.21.15.88:0/4291067538 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c0806c2e0 0x7f4c0806e7a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4c10034000 tx=0x7f4c1002c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:09.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.780+0000 7f4c257fa640 1 -- 172.21.15.88:0/4291067538 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4c18020020 con 0x7f4c280715e0 2024-09-16T01:36:09.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.943+0000 7f4c2f047640 1 -- 172.21.15.88:0/4291067538 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4c280b6ac0 con 0x7f4c0806c2e0 2024-09-16T01:36:09.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.949+0000 7f4c257fa640 1 -- 172.21.15.88:0/4291067538 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+247134 (secure 0 0 0) 0x7f4c280b6ac0 con 0x7f4c0806c2e0 2024-09-16T01:36:09.953 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:09.954 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:36:09.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.952+0000 7f4c2f047640 1 -- 172.21.15.88:0/4291067538 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c0806c2e0 msgr2=0x7f4c0806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:09.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.952+0000 7f4c2f047640 1 --2- 172.21.15.88:0/4291067538 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c0806c2e0 0x7f4c0806e7a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4c10034000 tx=0x7f4c1002c000 comp rx=0 tx=0).stop 2024-09-16T01:36:09.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.952+0000 7f4c2f047640 1 -- 172.21.15.88:0/4291067538 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c280715e0 msgr2=0x7f4c28147f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:09.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.952+0000 7f4c2f047640 1 --2- 172.21.15.88:0/4291067538 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c280715e0 0x7f4c28147f50 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4c1800c790 tx=0x7f4c1800cc50 comp rx=0 tx=0).stop 2024-09-16T01:36:09.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.952+0000 7f4c2f047640 1 -- 172.21.15.88:0/4291067538 shutdown_connections 2024-09-16T01:36:09.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.952+0000 7f4c2f047640 1 --2- 172.21.15.88:0/4291067538 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c0806c2e0 0x7f4c0806e7a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.952+0000 7f4c2f047640 1 --2- 172.21.15.88:0/4291067538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c2815c550 0x7f4c280d29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.952+0000 7f4c2f047640 1 --2- 172.21.15.88:0/4291067538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c28073160 0x7f4c280d2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.952+0000 7f4c2f047640 1 --2- 172.21.15.88:0/4291067538 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c280715e0 0x7f4c28147f50 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:09.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.952+0000 7f4c2f047640 1 -- 172.21.15.88:0/4291067538 >> 172.21.15.88:0/4291067538 conn(0x7f4c280b4240 msgr2=0x7f4c280d1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:09.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.952+0000 7f4c2f047640 1 -- 172.21.15.88:0/4291067538 shutdown_connections 2024-09-16T01:36:09.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:09.952+0000 7f4c2f047640 1 -- 172.21.15.88:0/4291067538 wait complete. 2024-09-16T01:36:09.964 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":97,"stamp":"2024-09-16T01:36:08.538350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590438,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":163,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":25,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221240,"kb_used_data":4980,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753480,"statfs":{"total":773094113280,"available":772867563520,"internally_reserved":0,"allocated":5099520,"data_stored":2729763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.195069"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.359868+0000","last_change":"2024-09-16T01:36:05.359868+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.359868+0000","last_undegraded":"2024-09-16T01:36:05.359868+0000","last_fullsized":"2024-09-16T01:36:05.359868+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.370012+0000","last_change":"2024-09-16T01:36:06.366967+0000","last_active":"2024-09-16T01:36:06.370012+0000","last_peered":"2024-09-16T01:36:06.370012+0000","last_clean":"2024-09-16T01:36:06.370012+0000","last_became_active":"2024-09-16T01:36:06.366688+0000","last_became_peered":"2024-09-16T01:36:06.366688+0000","last_unstale":"2024-09-16T01:36:06.370012+0000","last_undegraded":"2024-09-16T01:36:06.370012+0000","last_fullsized":"2024-09-16T01:36:06.370012+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:40:52.576920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352493+0000","last_change":"2024-09-16T01:36:05.352493+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352493+0000","last_undegraded":"2024-09-16T01:36:05.352493+0000","last_fullsized":"2024-09-16T01:36:05.352493+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358690+0000","last_change":"2024-09-16T01:36:05.358690+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358690+0000","last_undegraded":"2024-09-16T01:36:05.358690+0000","last_fullsized":"2024-09-16T01:36:05.358690+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.371287+0000","last_change":"2024-09-16T01:36:06.369570+0000","last_active":"2024-09-16T01:36:06.371287+0000","last_peered":"2024-09-16T01:36:06.371287+0000","last_clean":"2024-09-16T01:36:06.371287+0000","last_became_active":"2024-09-16T01:36:06.369276+0000","last_became_peered":"2024-09-16T01:36:06.369276+0000","last_unstale":"2024-09-16T01:36:06.371287+0000","last_undegraded":"2024-09-16T01:36:06.371287+0000","last_fullsized":"2024-09-16T01:36:06.371287+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:39:28.693098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358381+0000","last_change":"2024-09-16T01:36:05.358381+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358381+0000","last_undegraded":"2024-09-16T01:36:05.358381+0000","last_fullsized":"2024-09-16T01:36:05.358381+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358435+0000","last_change":"2024-09-16T01:36:05.358435+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358435+0000","last_undegraded":"2024-09-16T01:36:05.358435+0000","last_fullsized":"2024-09-16T01:36:05.358435+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358875+0000","last_change":"2024-09-16T01:36:05.358875+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358875+0000","last_undegraded":"2024-09-16T01:36:05.358875+0000","last_fullsized":"2024-09-16T01:36:05.358875+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358586+0000","last_change":"2024-09-16T01:36:05.358586+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358586+0000","last_undegraded":"2024-09-16T01:36:05.358586+0000","last_fullsized":"2024-09-16T01:36:05.358586+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358111+0000","last_change":"2024-09-16T01:36:05.358111+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358111+0000","last_undegraded":"2024-09-16T01:36:05.358111+0000","last_fullsized":"2024-09-16T01:36:05.358111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.357648+0000","last_change":"2024-09-16T01:36:05.357648+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.357648+0000","last_undegraded":"2024-09-16T01:36:05.357648+0000","last_fullsized":"2024-09-16T01:36:05.357648+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.376012+0000","last_change":"2024-09-16T01:36:06.374437+0000","last_active":"2024-09-16T01:36:06.376012+0000","last_peered":"2024-09-16T01:36:06.376012+0000","last_clean":"2024-09-16T01:36:06.376012+0000","last_became_active":"2024-09-16T01:36:06.374156+0000","last_became_peered":"2024-09-16T01:36:06.374156+0000","last_unstale":"2024-09-16T01:36:06.376012+0000","last_undegraded":"2024-09-16T01:36:06.376012+0000","last_fullsized":"2024-09-16T01:36:06.376012+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:14:35.252633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352763+0000","last_change":"2024-09-16T01:36:05.352763+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352763+0000","last_undegraded":"2024-09-16T01:36:05.352763+0000","last_fullsized":"2024-09-16T01:36:05.352763+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356941+0000","last_change":"2024-09-16T01:36:05.356941+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356941+0000","last_undegraded":"2024-09-16T01:36:05.356941+0000","last_fullsized":"2024-09-16T01:36:05.356941+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358500+0000","last_change":"2024-09-16T01:36:05.358500+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358500+0000","last_undegraded":"2024-09-16T01:36:05.358500+0000","last_fullsized":"2024-09-16T01:36:05.358500+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352230+0000","last_change":"2024-09-16T01:36:05.352230+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352230+0000","last_undegraded":"2024-09-16T01:36:05.352230+0000","last_fullsized":"2024-09-16T01:36:05.352230+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351446+0000","last_change":"2024-09-16T01:36:05.351446+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351446+0000","last_undegraded":"2024-09-16T01:36:05.351446+0000","last_fullsized":"2024-09-16T01:36:05.351446+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351350+0000","last_change":"2024-09-16T01:36:05.351350+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351350+0000","last_undegraded":"2024-09-16T01:36:05.351350+0000","last_fullsized":"2024-09-16T01:36:05.351350+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.350520+0000","last_change":"2024-09-16T01:36:05.350520+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.350520+0000","last_undegraded":"2024-09-16T01:36:05.350520+0000","last_fullsized":"2024-09-16T01:36:05.350520+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.354933+0000","last_change":"2024-09-16T01:36:05.354933+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.354933+0000","last_undegraded":"2024-09-16T01:36:05.354933+0000","last_fullsized":"2024-09-16T01:36:05.354933+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358886+0000","last_change":"2024-09-16T01:36:05.358886+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358886+0000","last_undegraded":"2024-09-16T01:36:05.358886+0000","last_fullsized":"2024-09-16T01:36:05.358886+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356521+0000","last_change":"2024-09-16T01:36:05.356521+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356521+0000","last_undegraded":"2024-09-16T01:36:05.356521+0000","last_fullsized":"2024-09-16T01:36:05.356521+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351493+0000","last_change":"2024-09-16T01:36:05.351493+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351493+0000","last_undegraded":"2024-09-16T01:36:05.351493+0000","last_fullsized":"2024-09-16T01:36:05.351493+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.350478+0000","last_change":"2024-09-16T01:36:05.350478+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.350478+0000","last_undegraded":"2024-09-16T01:36:05.350478+0000","last_fullsized":"2024-09-16T01:36:05.350478+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.350770+0000","last_change":"2024-09-16T01:36:05.350770+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.350770+0000","last_undegraded":"2024-09-16T01:36:05.350770+0000","last_fullsized":"2024-09-16T01:36:05.350770+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353568+0000","last_change":"2024-09-16T01:36:05.353568+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353568+0000","last_undegraded":"2024-09-16T01:36:05.353568+0000","last_fullsized":"2024-09-16T01:36:05.353568+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.350845+0000","last_change":"2024-09-16T01:36:05.350845+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.350845+0000","last_undegraded":"2024-09-16T01:36:05.350845+0000","last_fullsized":"2024-09-16T01:36:05.350845+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.370131+0000","last_change":"2024-09-16T01:36:06.365094+0000","last_active":"2024-09-16T01:36:06.370131+0000","last_peered":"2024-09-16T01:36:06.370131+0000","last_clean":"2024-09-16T01:36:06.370131+0000","last_became_active":"2024-09-16T01:36:06.364790+0000","last_became_peered":"2024-09-16T01:36:06.364790+0000","last_unstale":"2024-09-16T01:36:06.370131+0000","last_undegraded":"2024-09-16T01:36:06.370131+0000","last_fullsized":"2024-09-16T01:36:06.370131+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:29:16.206145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351073+0000","last_change":"2024-09-16T01:36:05.351073+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351073+0000","last_undegraded":"2024-09-16T01:36:05.351073+0000","last_fullsized":"2024-09-16T01:36:05.351073+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353729+0000","last_change":"2024-09-16T01:36:05.353729+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353729+0000","last_undegraded":"2024-09-16T01:36:05.353729+0000","last_fullsized":"2024-09-16T01:36:05.353729+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356913+0000","last_change":"2024-09-16T01:36:05.356913+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356913+0000","last_undegraded":"2024-09-16T01:36:05.356913+0000","last_fullsized":"2024-09-16T01:36:05.356913+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351956+0000","last_change":"2024-09-16T01:36:05.351956+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351956+0000","last_undegraded":"2024-09-16T01:36:05.351956+0000","last_fullsized":"2024-09-16T01:36:05.351956+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.354055+0000","last_change":"2024-09-16T01:36:05.354055+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.354055+0000","last_undegraded":"2024-09-16T01:36:05.354055+0000","last_fullsized":"2024-09-16T01:36:05.354055+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.360688+0000","last_change":"2024-09-16T01:36:05.357580+0000","last_active":"2024-09-16T01:36:05.360688+0000","last_peered":"2024-09-16T01:36:05.360688+0000","last_clean":"2024-09-16T01:36:05.360688+0000","last_became_active":"2024-09-16T01:36:05.357331+0000","last_became_peered":"2024-09-16T01:36:05.357331+0000","last_unstale":"2024-09-16T01:36:05.360688+0000","last_undegraded":"2024-09-16T01:36:05.360688+0000","last_fullsized":"2024-09-16T01:36:05.360688+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:28:41.466172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351699+0000","last_change":"2024-09-16T01:36:05.351699+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351699+0000","last_undegraded":"2024-09-16T01:36:05.351699+0000","last_fullsized":"2024-09-16T01:36:05.351699+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.365028+0000","last_change":"2024-09-16T01:36:05.357954+0000","last_active":"2024-09-16T01:36:05.365028+0000","last_peered":"2024-09-16T01:36:05.365028+0000","last_clean":"2024-09-16T01:36:05.365028+0000","last_became_active":"2024-09-16T01:36:05.357018+0000","last_became_peered":"2024-09-16T01:36:05.357018+0000","last_unstale":"2024-09-16T01:36:05.365028+0000","last_undegraded":"2024-09-16T01:36:05.365028+0000","last_fullsized":"2024-09-16T01:36:05.365028+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:53.565250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.378364+0000","last_change":"2024-09-16T01:36:06.377925+0000","last_active":"2024-09-16T01:36:06.378364+0000","last_peered":"2024-09-16T01:36:06.378364+0000","last_clean":"2024-09-16T01:36:06.378364+0000","last_became_active":"2024-09-16T01:36:06.377631+0000","last_became_peered":"2024-09-16T01:36:06.377631+0000","last_unstale":"2024-09-16T01:36:06.378364+0000","last_undegraded":"2024-09-16T01:36:06.378364+0000","last_fullsized":"2024-09-16T01:36:06.378364+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:09:13.085891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368709+0000","last_change":"2024-09-16T01:36:05.365158+0000","last_active":"2024-09-16T01:36:05.368709+0000","last_peered":"2024-09-16T01:36:05.368709+0000","last_clean":"2024-09-16T01:36:05.368709+0000","last_became_active":"2024-09-16T01:36:05.364885+0000","last_became_peered":"2024-09-16T01:36:05.364885+0000","last_unstale":"2024-09-16T01:36:05.368709+0000","last_undegraded":"2024-09-16T01:36:05.368709+0000","last_fullsized":"2024-09-16T01:36:05.368709+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:06:24.080498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.354065+0000","last_change":"2024-09-16T01:36:05.354065+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.354065+0000","last_undegraded":"2024-09-16T01:36:05.354065+0000","last_fullsized":"2024-09-16T01:36:05.354065+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373524+0000","last_change":"2024-09-16T01:36:05.368193+0000","last_active":"2024-09-16T01:36:05.373524+0000","last_peered":"2024-09-16T01:36:05.373524+0000","last_clean":"2024-09-16T01:36:05.373524+0000","last_became_active":"2024-09-16T01:36:05.367907+0000","last_became_peered":"2024-09-16T01:36:05.367907+0000","last_unstale":"2024-09-16T01:36:05.373524+0000","last_undegraded":"2024-09-16T01:36:05.373524+0000","last_fullsized":"2024-09-16T01:36:05.373524+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:40:04.647251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353341+0000","last_change":"2024-09-16T01:36:05.353341+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353341+0000","last_undegraded":"2024-09-16T01:36:05.353341+0000","last_fullsized":"2024-09-16T01:36:05.353341+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368806+0000","last_change":"2024-09-16T01:36:05.361916+0000","last_active":"2024-09-16T01:36:05.368806+0000","last_peered":"2024-09-16T01:36:05.368806+0000","last_clean":"2024-09-16T01:36:05.368806+0000","last_became_active":"2024-09-16T01:36:05.361658+0000","last_became_peered":"2024-09-16T01:36:05.361658+0000","last_unstale":"2024-09-16T01:36:05.368806+0000","last_undegraded":"2024-09-16T01:36:05.368806+0000","last_fullsized":"2024-09-16T01:36:05.368806+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:41.613259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356484+0000","last_change":"2024-09-16T01:36:05.356484+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356484+0000","last_undegraded":"2024-09-16T01:36:05.356484+0000","last_fullsized":"2024-09-16T01:36:05.356484+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373139+0000","last_change":"2024-09-16T01:36:05.365323+0000","last_active":"2024-09-16T01:36:05.373139+0000","last_peered":"2024-09-16T01:36:05.373139+0000","last_clean":"2024-09-16T01:36:05.373139+0000","last_became_active":"2024-09-16T01:36:05.364940+0000","last_became_peered":"2024-09-16T01:36:05.364940+0000","last_unstale":"2024-09-16T01:36:05.373139+0000","last_undegraded":"2024-09-16T01:36:05.373139+0000","last_fullsized":"2024-09-16T01:36:05.373139+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:38:11.910238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353237+0000","last_change":"2024-09-16T01:36:05.353237+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353237+0000","last_undegraded":"2024-09-16T01:36:05.353237+0000","last_fullsized":"2024-09-16T01:36:05.353237+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.357443+0000","last_change":"2024-09-16T01:36:05.359801+0000","last_active":"2024-09-16T01:36:06.357443+0000","last_peered":"2024-09-16T01:36:06.357443+0000","last_clean":"2024-09-16T01:36:06.357443+0000","last_became_active":"2024-09-16T01:36:05.359542+0000","last_became_peered":"2024-09-16T01:36:05.359542+0000","last_unstale":"2024-09-16T01:36:06.357443+0000","last_undegraded":"2024-09-16T01:36:06.357443+0000","last_fullsized":"2024-09-16T01:36:06.357443+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:22.128838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351989+0000","last_change":"2024-09-16T01:36:05.351989+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351989+0000","last_undegraded":"2024-09-16T01:36:05.351989+0000","last_fullsized":"2024-09-16T01:36:05.351989+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.366391+0000","last_change":"2024-09-16T01:36:05.364586+0000","last_active":"2024-09-16T01:36:05.366391+0000","last_peered":"2024-09-16T01:36:05.366391+0000","last_clean":"2024-09-16T01:36:05.366391+0000","last_became_active":"2024-09-16T01:36:05.364271+0000","last_became_peered":"2024-09-16T01:36:05.364271+0000","last_unstale":"2024-09-16T01:36:05.366391+0000","last_undegraded":"2024-09-16T01:36:05.366391+0000","last_fullsized":"2024-09-16T01:36:05.366391+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:15:05.347700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.350838+0000","last_change":"2024-09-16T01:36:05.350838+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.350838+0000","last_undegraded":"2024-09-16T01:36:05.350838+0000","last_fullsized":"2024-09-16T01:36:05.350838+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.370306+0000","last_change":"2024-09-16T01:36:05.364097+0000","last_active":"2024-09-16T01:36:05.370306+0000","last_peered":"2024-09-16T01:36:05.370306+0000","last_clean":"2024-09-16T01:36:05.370306+0000","last_became_active":"2024-09-16T01:36:05.363758+0000","last_became_peered":"2024-09-16T01:36:05.363758+0000","last_unstale":"2024-09-16T01:36:05.370306+0000","last_undegraded":"2024-09-16T01:36:05.370306+0000","last_fullsized":"2024-09-16T01:36:05.370306+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:59.822221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.359339+0000","last_change":"2024-09-16T01:36:05.359339+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.359339+0000","last_undegraded":"2024-09-16T01:36:05.359339+0000","last_fullsized":"2024-09-16T01:36:05.359339+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.365112+0000","last_change":"2024-09-16T01:36:05.363830+0000","last_active":"2024-09-16T01:36:05.365112+0000","last_peered":"2024-09-16T01:36:05.365112+0000","last_clean":"2024-09-16T01:36:05.365112+0000","last_became_active":"2024-09-16T01:36:05.363572+0000","last_became_peered":"2024-09-16T01:36:05.363572+0000","last_unstale":"2024-09-16T01:36:05.365112+0000","last_undegraded":"2024-09-16T01:36:05.365112+0000","last_fullsized":"2024-09-16T01:36:05.365112+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:53.565250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356478+0000","last_change":"2024-09-16T01:36:05.356478+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356478+0000","last_undegraded":"2024-09-16T01:36:05.356478+0000","last_fullsized":"2024-09-16T01:36:05.356478+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373551+0000","last_change":"2024-09-16T01:36:05.368744+0000","last_active":"2024-09-16T01:36:05.373551+0000","last_peered":"2024-09-16T01:36:05.373551+0000","last_clean":"2024-09-16T01:36:05.373551+0000","last_became_active":"2024-09-16T01:36:05.368069+0000","last_became_peered":"2024-09-16T01:36:05.368069+0000","last_unstale":"2024-09-16T01:36:05.373551+0000","last_undegraded":"2024-09-16T01:36:05.373551+0000","last_fullsized":"2024-09-16T01:36:05.373551+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:58.552340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.360574+0000","last_change":"2024-09-16T01:36:05.360574+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.360574+0000","last_undegraded":"2024-09-16T01:36:05.360574+0000","last_fullsized":"2024-09-16T01:36:05.360574+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368680+0000","last_change":"2024-09-16T01:36:05.365975+0000","last_active":"2024-09-16T01:36:05.368680+0000","last_peered":"2024-09-16T01:36:05.368680+0000","last_clean":"2024-09-16T01:36:05.368680+0000","last_became_active":"2024-09-16T01:36:05.365674+0000","last_became_peered":"2024-09-16T01:36:05.365674+0000","last_unstale":"2024-09-16T01:36:05.368680+0000","last_undegraded":"2024-09-16T01:36:05.368680+0000","last_fullsized":"2024-09-16T01:36:05.368680+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:21:23.387226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352330+0000","last_change":"2024-09-16T01:36:05.352330+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352330+0000","last_undegraded":"2024-09-16T01:36:05.352330+0000","last_fullsized":"2024-09-16T01:36:05.352330+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368728+0000","last_change":"2024-09-16T01:36:05.360882+0000","last_active":"2024-09-16T01:36:05.368728+0000","last_peered":"2024-09-16T01:36:05.368728+0000","last_clean":"2024-09-16T01:36:05.368728+0000","last_became_active":"2024-09-16T01:36:05.360521+0000","last_became_peered":"2024-09-16T01:36:05.360521+0000","last_unstale":"2024-09-16T01:36:05.368728+0000","last_undegraded":"2024-09-16T01:36:05.368728+0000","last_fullsized":"2024-09-16T01:36:05.368728+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:05:47.000282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.354770+0000","last_change":"2024-09-16T01:36:05.354770+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.354770+0000","last_undegraded":"2024-09-16T01:36:05.354770+0000","last_fullsized":"2024-09-16T01:36:05.354770+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373578+0000","last_change":"2024-09-16T01:36:05.369476+0000","last_active":"2024-09-16T01:36:05.373578+0000","last_peered":"2024-09-16T01:36:05.373578+0000","last_clean":"2024-09-16T01:36:05.373578+0000","last_became_active":"2024-09-16T01:36:05.369155+0000","last_became_peered":"2024-09-16T01:36:05.369155+0000","last_unstale":"2024-09-16T01:36:05.373578+0000","last_undegraded":"2024-09-16T01:36:05.373578+0000","last_fullsized":"2024-09-16T01:36:05.373578+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:22.128838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356076+0000","last_change":"2024-09-16T01:36:05.356076+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356076+0000","last_undegraded":"2024-09-16T01:36:05.356076+0000","last_fullsized":"2024-09-16T01:36:05.356076+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.366290+0000","last_change":"2024-09-16T01:36:05.359803+0000","last_active":"2024-09-16T01:36:05.366290+0000","last_peered":"2024-09-16T01:36:05.366290+0000","last_clean":"2024-09-16T01:36:05.366290+0000","last_became_active":"2024-09-16T01:36:05.359552+0000","last_became_peered":"2024-09-16T01:36:05.359552+0000","last_unstale":"2024-09-16T01:36:05.366290+0000","last_undegraded":"2024-09-16T01:36:05.366290+0000","last_fullsized":"2024-09-16T01:36:05.366290+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:32:35.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.357949+0000","last_change":"2024-09-16T01:36:05.357949+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.357949+0000","last_undegraded":"2024-09-16T01:36:05.357949+0000","last_fullsized":"2024-09-16T01:36:05.357949+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.362430+0000","last_change":"2024-09-16T01:36:05.359125+0000","last_active":"2024-09-16T01:36:05.362430+0000","last_peered":"2024-09-16T01:36:05.362430+0000","last_clean":"2024-09-16T01:36:05.362430+0000","last_became_active":"2024-09-16T01:36:05.358486+0000","last_became_peered":"2024-09-16T01:36:05.358486+0000","last_unstale":"2024-09-16T01:36:05.362430+0000","last_undegraded":"2024-09-16T01:36:05.362430+0000","last_fullsized":"2024-09-16T01:36:05.362430+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:22:44.509769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353180+0000","last_change":"2024-09-16T01:36:05.353180+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353180+0000","last_undegraded":"2024-09-16T01:36:05.353180+0000","last_fullsized":"2024-09-16T01:36:05.353180+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.364379+0000","last_change":"2024-09-16T01:36:05.361659+0000","last_active":"2024-09-16T01:36:05.364379+0000","last_peered":"2024-09-16T01:36:05.364379+0000","last_clean":"2024-09-16T01:36:05.364379+0000","last_became_active":"2024-09-16T01:36:05.361408+0000","last_became_peered":"2024-09-16T01:36:05.361408+0000","last_unstale":"2024-09-16T01:36:05.364379+0000","last_undegraded":"2024-09-16T01:36:05.364379+0000","last_fullsized":"2024-09-16T01:36:05.364379+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:30:10.270905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352248+0000","last_change":"2024-09-16T01:36:05.352248+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352248+0000","last_undegraded":"2024-09-16T01:36:05.352248+0000","last_fullsized":"2024-09-16T01:36:05.352248+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.365065+0000","last_change":"2024-09-16T01:36:05.363734+0000","last_active":"2024-09-16T01:36:05.365065+0000","last_peered":"2024-09-16T01:36:05.365065+0000","last_clean":"2024-09-16T01:36:05.365065+0000","last_became_active":"2024-09-16T01:36:05.363407+0000","last_became_peered":"2024-09-16T01:36:05.363407+0000","last_unstale":"2024-09-16T01:36:05.365065+0000","last_undegraded":"2024-09-16T01:36:05.365065+0000","last_fullsized":"2024-09-16T01:36:05.365065+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:56:30.024164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351047+0000","last_change":"2024-09-16T01:36:05.351047+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351047+0000","last_undegraded":"2024-09-16T01:36:05.351047+0000","last_fullsized":"2024-09-16T01:36:05.351047+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"44'1","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.374827+0000","last_change":"2024-09-16T01:36:05.360643+0000","last_active":"2024-09-16T01:36:06.374827+0000","last_peered":"2024-09-16T01:36:06.374827+0000","last_clean":"2024-09-16T01:36:06.374827+0000","last_became_active":"2024-09-16T01:36:05.360204+0000","last_became_peered":"2024-09-16T01:36:05.360204+0000","last_unstale":"2024-09-16T01:36:06.374827+0000","last_undegraded":"2024-09-16T01:36:06.374827+0000","last_fullsized":"2024-09-16T01:36:06.374827+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:54.497479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351796+0000","last_change":"2024-09-16T01:36:05.351796+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351796+0000","last_undegraded":"2024-09-16T01:36:05.351796+0000","last_fullsized":"2024-09-16T01:36:05.351796+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368891+0000","last_change":"2024-09-16T01:36:05.363498+0000","last_active":"2024-09-16T01:36:05.368891+0000","last_peered":"2024-09-16T01:36:05.368891+0000","last_clean":"2024-09-16T01:36:05.368891+0000","last_became_active":"2024-09-16T01:36:05.363185+0000","last_became_peered":"2024-09-16T01:36:05.363185+0000","last_unstale":"2024-09-16T01:36:05.368891+0000","last_undegraded":"2024-09-16T01:36:05.368891+0000","last_fullsized":"2024-09-16T01:36:05.368891+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:29:00.841831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356095+0000","last_change":"2024-09-16T01:36:05.356095+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356095+0000","last_undegraded":"2024-09-16T01:36:05.356095+0000","last_fullsized":"2024-09-16T01:36:05.356095+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368786+0000","last_change":"2024-09-16T01:36:05.361868+0000","last_active":"2024-09-16T01:36:05.368786+0000","last_peered":"2024-09-16T01:36:05.368786+0000","last_clean":"2024-09-16T01:36:05.368786+0000","last_became_active":"2024-09-16T01:36:05.361605+0000","last_became_peered":"2024-09-16T01:36:05.361605+0000","last_unstale":"2024-09-16T01:36:05.368786+0000","last_undegraded":"2024-09-16T01:36:05.368786+0000","last_fullsized":"2024-09-16T01:36:05.368786+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:52:52.798634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353015+0000","last_change":"2024-09-16T01:36:05.353015+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353015+0000","last_undegraded":"2024-09-16T01:36:05.353015+0000","last_fullsized":"2024-09-16T01:36:05.353015+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.357525+0000","last_change":"2024-09-16T01:36:05.356369+0000","last_active":"2024-09-16T01:36:06.357525+0000","last_peered":"2024-09-16T01:36:06.357525+0000","last_clean":"2024-09-16T01:36:06.357525+0000","last_became_active":"2024-09-16T01:36:05.356027+0000","last_became_peered":"2024-09-16T01:36:05.356027+0000","last_unstale":"2024-09-16T01:36:06.357525+0000","last_undegraded":"2024-09-16T01:36:06.357525+0000","last_fullsized":"2024-09-16T01:36:06.357525+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:02.446309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352450+0000","last_change":"2024-09-16T01:36:05.352450+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352450+0000","last_undegraded":"2024-09-16T01:36:05.352450+0000","last_fullsized":"2024-09-16T01:36:05.352450+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.364524+0000","last_change":"2024-09-16T01:36:05.363731+0000","last_active":"2024-09-16T01:36:05.364524+0000","last_peered":"2024-09-16T01:36:05.364524+0000","last_clean":"2024-09-16T01:36:05.364524+0000","last_became_active":"2024-09-16T01:36:05.363475+0000","last_became_peered":"2024-09-16T01:36:05.363475+0000","last_unstale":"2024-09-16T01:36:05.364524+0000","last_undegraded":"2024-09-16T01:36:05.364524+0000","last_fullsized":"2024-09-16T01:36:05.364524+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:28:42.397852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353294+0000","last_change":"2024-09-16T01:36:05.353294+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353294+0000","last_undegraded":"2024-09-16T01:36:05.353294+0000","last_fullsized":"2024-09-16T01:36:05.353294+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373466+0000","last_change":"2024-09-16T01:36:05.367836+0000","last_active":"2024-09-16T01:36:05.373466+0000","last_peered":"2024-09-16T01:36:05.373466+0000","last_clean":"2024-09-16T01:36:05.373466+0000","last_became_active":"2024-09-16T01:36:05.367513+0000","last_became_peered":"2024-09-16T01:36:05.367513+0000","last_unstale":"2024-09-16T01:36:05.373466+0000","last_undegraded":"2024-09-16T01:36:05.373466+0000","last_fullsized":"2024-09-16T01:36:05.373466+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:53:48.685089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351224+0000","last_change":"2024-09-16T01:36:05.351224+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351224+0000","last_undegraded":"2024-09-16T01:36:05.351224+0000","last_fullsized":"2024-09-16T01:36:05.351224+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368934+0000","last_change":"2024-09-16T01:36:05.365994+0000","last_active":"2024-09-16T01:36:05.368934+0000","last_peered":"2024-09-16T01:36:05.368934+0000","last_clean":"2024-09-16T01:36:05.368934+0000","last_became_active":"2024-09-16T01:36:05.365629+0000","last_became_peered":"2024-09-16T01:36:05.365629+0000","last_unstale":"2024-09-16T01:36:05.368934+0000","last_undegraded":"2024-09-16T01:36:05.368934+0000","last_fullsized":"2024-09-16T01:36:05.368934+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:44:58.819992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352527+0000","last_change":"2024-09-16T01:36:05.352527+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352527+0000","last_undegraded":"2024-09-16T01:36:05.352527+0000","last_fullsized":"2024-09-16T01:36:05.352527+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373583+0000","last_change":"2024-09-16T01:36:05.370944+0000","last_active":"2024-09-16T01:36:05.373583+0000","last_peered":"2024-09-16T01:36:05.373583+0000","last_clean":"2024-09-16T01:36:05.373583+0000","last_became_active":"2024-09-16T01:36:05.370631+0000","last_became_peered":"2024-09-16T01:36:05.370631+0000","last_unstale":"2024-09-16T01:36:05.373583+0000","last_undegraded":"2024-09-16T01:36:05.373583+0000","last_fullsized":"2024-09-16T01:36:05.373583+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:02.446309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.359667+0000","last_change":"2024-09-16T01:36:05.359667+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.359667+0000","last_undegraded":"2024-09-16T01:36:05.359667+0000","last_fullsized":"2024-09-16T01:36:05.359667+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368831+0000","last_change":"2024-09-16T01:36:05.365868+0000","last_active":"2024-09-16T01:36:05.368831+0000","last_peered":"2024-09-16T01:36:05.368831+0000","last_clean":"2024-09-16T01:36:05.368831+0000","last_became_active":"2024-09-16T01:36:05.365481+0000","last_became_peered":"2024-09-16T01:36:05.365481+0000","last_unstale":"2024-09-16T01:36:05.368831+0000","last_undegraded":"2024-09-16T01:36:05.368831+0000","last_fullsized":"2024-09-16T01:36:05.368831+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:26:50.407545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.354612+0000","last_change":"2024-09-16T01:36:05.354612+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.354612+0000","last_undegraded":"2024-09-16T01:36:05.354612+0000","last_fullsized":"2024-09-16T01:36:05.354612+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373531+0000","last_change":"2024-09-16T01:36:05.368272+0000","last_active":"2024-09-16T01:36:05.373531+0000","last_peered":"2024-09-16T01:36:05.373531+0000","last_clean":"2024-09-16T01:36:05.373531+0000","last_became_active":"2024-09-16T01:36:05.367973+0000","last_became_peered":"2024-09-16T01:36:05.367973+0000","last_unstale":"2024-09-16T01:36:05.373531+0000","last_undegraded":"2024-09-16T01:36:05.373531+0000","last_fullsized":"2024-09-16T01:36:05.373531+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:09:24.703051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.355177+0000","last_change":"2024-09-16T01:36:05.355177+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.355177+0000","last_undegraded":"2024-09-16T01:36:05.355177+0000","last_fullsized":"2024-09-16T01:36:05.355177+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.365212+0000","last_change":"2024-09-16T01:36:05.362224+0000","last_active":"2024-09-16T01:36:05.365212+0000","last_peered":"2024-09-16T01:36:05.365212+0000","last_clean":"2024-09-16T01:36:05.365212+0000","last_became_active":"2024-09-16T01:36:05.361853+0000","last_became_peered":"2024-09-16T01:36:05.361853+0000","last_unstale":"2024-09-16T01:36:05.365212+0000","last_undegraded":"2024-09-16T01:36:05.365212+0000","last_fullsized":"2024-09-16T01:36:05.365212+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:15:05.347700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.369963+0000","last_change":"2024-09-16T01:36:06.363427+0000","last_active":"2024-09-16T01:36:06.369963+0000","last_peered":"2024-09-16T01:36:06.369963+0000","last_clean":"2024-09-16T01:36:06.369963+0000","last_became_active":"2024-09-16T01:36:06.363142+0000","last_became_peered":"2024-09-16T01:36:06.363142+0000","last_unstale":"2024-09-16T01:36:06.369963+0000","last_undegraded":"2024-09-16T01:36:06.369963+0000","last_fullsized":"2024-09-16T01:36:06.369963+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:22.762911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.366175+0000","last_change":"2024-09-16T01:36:05.362621+0000","last_active":"2024-09-16T01:36:05.366175+0000","last_peered":"2024-09-16T01:36:05.366175+0000","last_clean":"2024-09-16T01:36:05.366175+0000","last_became_active":"2024-09-16T01:36:05.362300+0000","last_became_peered":"2024-09-16T01:36:05.362300+0000","last_unstale":"2024-09-16T01:36:05.366175+0000","last_undegraded":"2024-09-16T01:36:05.366175+0000","last_fullsized":"2024-09-16T01:36:05.366175+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:31:24.708078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.350199+0000","last_change":"2024-09-16T01:36:05.350199+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.350199+0000","last_undegraded":"2024-09-16T01:36:05.350199+0000","last_fullsized":"2024-09-16T01:36:05.350199+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.363008+0000","last_change":"2024-09-16T01:36:05.362040+0000","last_active":"2024-09-16T01:36:05.363008+0000","last_peered":"2024-09-16T01:36:05.363008+0000","last_clean":"2024-09-16T01:36:05.363008+0000","last_became_active":"2024-09-16T01:36:05.361767+0000","last_became_peered":"2024-09-16T01:36:05.361767+0000","last_unstale":"2024-09-16T01:36:05.363008+0000","last_undegraded":"2024-09-16T01:36:05.363008+0000","last_fullsized":"2024-09-16T01:36:05.363008+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:49:52.496809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353674+0000","last_change":"2024-09-16T01:36:05.353674+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353674+0000","last_undegraded":"2024-09-16T01:36:05.353674+0000","last_fullsized":"2024-09-16T01:36:05.353674+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"44'2","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.377763+0000","last_change":"2024-09-16T01:36:05.361909+0000","last_active":"2024-09-16T01:36:06.377763+0000","last_peered":"2024-09-16T01:36:06.377763+0000","last_clean":"2024-09-16T01:36:06.377763+0000","last_became_active":"2024-09-16T01:36:05.361632+0000","last_became_peered":"2024-09-16T01:36:05.361632+0000","last_unstale":"2024-09-16T01:36:06.377763+0000","last_undegraded":"2024-09-16T01:36:06.377763+0000","last_fullsized":"2024-09-16T01:36:06.377763+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:55:37.303541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":209,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.361317+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:36:05.361317+0000","last_peered":"2024-09-16T01:36:05.361317+0000","last_clean":"2024-09-16T01:36:05.361317+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:36:05.361317+0000","last_undegraded":"2024-09-16T01:36:05.361317+0000","last_fullsized":"2024-09-16T01:36:05.361317+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":70,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509468,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27520,"kb_used_data":488,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344320,"statfs":{"total":96636764160,"available":96608583680,"internally_reserved":0,"allocated":499712,"data_stored":199112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542171,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28088,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343752,"statfs":{"total":96636764160,"available":96608002048,"internally_reserved":0,"allocated":1081344,"data_stored":783213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542171,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":487424,"data_stored":192845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542171,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":487424,"data_stored":192845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542170,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27512,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344328,"statfs":{"total":96636764160,"available":96608591872,"internally_reserved":0,"allocated":487424,"data_stored":192845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542171,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":487424,"data_stored":192845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542170,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27504,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344336,"statfs":{"total":96636764160,"available":96608600064,"internally_reserved":0,"allocated":487424,"data_stored":192845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542170,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28092,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343748,"statfs":{"total":96636764160,"available":96607997952,"internally_reserved":0,"allocated":1081344,"data_stored":783213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:36:09.968 INFO:tasks.ceph.ceph_manager.ceph:PG 7.3f is not active+clean 2024-09-16T01:36:09.968 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.359868+0000', 'last_change': '2024-09-16T01:36:05.359868+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.359868+0000', 'last_undegraded': '2024-09-16T01:36:05.359868+0000', 'last_fullsized': '2024-09-16T01:36:05.359868+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T01:36:09.968 INFO:tasks.ceph.ceph_manager.ceph:PG 7.3d is not active+clean 2024-09-16T01:36:09.968 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.352493+0000', 'last_change': '2024-09-16T01:36:05.352493+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.352493+0000', 'last_undegraded': '2024-09-16T01:36:05.352493+0000', 'last_fullsized': '2024-09-16T01:36:05.352493+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T01:36:09.968 INFO:tasks.ceph.ceph_manager.ceph:PG 7.3c is not active+clean 2024-09-16T01:36:09.968 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.358690+0000', 'last_change': '2024-09-16T01:36:05.358690+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.358690+0000', 'last_undegraded': '2024-09-16T01:36:05.358690+0000', 'last_fullsized': '2024-09-16T01:36:05.358690+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T01:36:09.968 INFO:tasks.ceph.ceph_manager.ceph:PG 7.3a is not active+clean 2024-09-16T01:36:09.968 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.358381+0000', 'last_change': '2024-09-16T01:36:05.358381+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.358381+0000', 'last_undegraded': '2024-09-16T01:36:05.358381+0000', 'last_fullsized': '2024-09-16T01:36:05.358381+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T01:36:09.968 INFO:tasks.ceph.ceph_manager.ceph:PG 7.39 is not active+clean 2024-09-16T01:36:09.968 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.358435+0000', 'last_change': '2024-09-16T01:36:05.358435+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.358435+0000', 'last_undegraded': '2024-09-16T01:36:05.358435+0000', 'last_fullsized': '2024-09-16T01:36:05.358435+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T01:36:09.969 INFO:tasks.ceph.ceph_manager.ceph:PG 7.38 is not active+clean 2024-09-16T01:36:09.969 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.358875+0000', 'last_change': '2024-09-16T01:36:05.358875+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.358875+0000', 'last_undegraded': '2024-09-16T01:36:05.358875+0000', 'last_fullsized': '2024-09-16T01:36:05.358875+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T01:36:09.969 INFO:tasks.ceph.ceph_manager.ceph:PG 7.37 is not active+clean 2024-09-16T01:36:09.969 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.358586+0000', 'last_change': '2024-09-16T01:36:05.358586+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.358586+0000', 'last_undegraded': '2024-09-16T01:36:05.358586+0000', 'last_fullsized': '2024-09-16T01:36:05.358586+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T01:36:09.969 INFO:tasks.ceph.ceph_manager.ceph:PG 7.36 is not active+clean 2024-09-16T01:36:09.969 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.358111+0000', 'last_change': '2024-09-16T01:36:05.358111+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.358111+0000', 'last_undegraded': '2024-09-16T01:36:05.358111+0000', 'last_fullsized': '2024-09-16T01:36:05.358111+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T01:36:09.969 INFO:tasks.ceph.ceph_manager.ceph:PG 7.35 is not active+clean 2024-09-16T01:36:09.969 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.357648+0000', 'last_change': '2024-09-16T01:36:05.357648+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.357648+0000', 'last_undegraded': '2024-09-16T01:36:05.357648+0000', 'last_fullsized': '2024-09-16T01:36:05.357648+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T01:36:09.969 INFO:tasks.ceph.ceph_manager.ceph:PG 7.33 is not active+clean 2024-09-16T01:36:09.969 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.352763+0000', 'last_change': '2024-09-16T01:36:05.352763+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.352763+0000', 'last_undegraded': '2024-09-16T01:36:05.352763+0000', 'last_fullsized': '2024-09-16T01:36:05.352763+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T01:36:09.969 INFO:tasks.ceph.ceph_manager.ceph:PG 7.32 is not active+clean 2024-09-16T01:36:09.970 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.356941+0000', 'last_change': '2024-09-16T01:36:05.356941+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.356941+0000', 'last_undegraded': '2024-09-16T01:36:05.356941+0000', 'last_fullsized': '2024-09-16T01:36:05.356941+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T01:36:09.970 INFO:tasks.ceph.ceph_manager.ceph:PG 7.31 is not active+clean 2024-09-16T01:36:09.970 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.358500+0000', 'last_change': '2024-09-16T01:36:05.358500+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.358500+0000', 'last_undegraded': '2024-09-16T01:36:05.358500+0000', 'last_fullsized': '2024-09-16T01:36:05.358500+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T01:36:09.970 INFO:tasks.ceph.ceph_manager.ceph:PG 7.30 is not active+clean 2024-09-16T01:36:09.970 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.352230+0000', 'last_change': '2024-09-16T01:36:05.352230+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.352230+0000', 'last_undegraded': '2024-09-16T01:36:05.352230+0000', 'last_fullsized': '2024-09-16T01:36:05.352230+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T01:36:09.970 INFO:tasks.ceph.ceph_manager.ceph:PG 7.2f is not active+clean 2024-09-16T01:36:09.970 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.351446+0000', 'last_change': '2024-09-16T01:36:05.351446+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.351446+0000', 'last_undegraded': '2024-09-16T01:36:05.351446+0000', 'last_fullsized': '2024-09-16T01:36:05.351446+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T01:36:09.970 INFO:tasks.ceph.ceph_manager.ceph:PG 7.2e is not active+clean 2024-09-16T01:36:09.970 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.351350+0000', 'last_change': '2024-09-16T01:36:05.351350+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.351350+0000', 'last_undegraded': '2024-09-16T01:36:05.351350+0000', 'last_fullsized': '2024-09-16T01:36:05.351350+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T01:36:09.970 INFO:tasks.ceph.ceph_manager.ceph:PG 7.2d is not active+clean 2024-09-16T01:36:09.970 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.350520+0000', 'last_change': '2024-09-16T01:36:05.350520+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.350520+0000', 'last_undegraded': '2024-09-16T01:36:05.350520+0000', 'last_fullsized': '2024-09-16T01:36:05.350520+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T01:36:09.971 INFO:tasks.ceph.ceph_manager.ceph:PG 7.2c is not active+clean 2024-09-16T01:36:09.971 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.354933+0000', 'last_change': '2024-09-16T01:36:05.354933+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.354933+0000', 'last_undegraded': '2024-09-16T01:36:05.354933+0000', 'last_fullsized': '2024-09-16T01:36:05.354933+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T01:36:09.971 INFO:tasks.ceph.ceph_manager.ceph:PG 7.2b is not active+clean 2024-09-16T01:36:09.971 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.358886+0000', 'last_change': '2024-09-16T01:36:05.358886+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.358886+0000', 'last_undegraded': '2024-09-16T01:36:05.358886+0000', 'last_fullsized': '2024-09-16T01:36:05.358886+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T01:36:09.971 INFO:tasks.ceph.ceph_manager.ceph:PG 7.2a is not active+clean 2024-09-16T01:36:09.971 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.356521+0000', 'last_change': '2024-09-16T01:36:05.356521+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.356521+0000', 'last_undegraded': '2024-09-16T01:36:05.356521+0000', 'last_fullsized': '2024-09-16T01:36:05.356521+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T01:36:09.971 INFO:tasks.ceph.ceph_manager.ceph:PG 7.29 is not active+clean 2024-09-16T01:36:09.971 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.351493+0000', 'last_change': '2024-09-16T01:36:05.351493+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.351493+0000', 'last_undegraded': '2024-09-16T01:36:05.351493+0000', 'last_fullsized': '2024-09-16T01:36:05.351493+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T01:36:09.971 INFO:tasks.ceph.ceph_manager.ceph:PG 7.28 is not active+clean 2024-09-16T01:36:09.971 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.350478+0000', 'last_change': '2024-09-16T01:36:05.350478+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.350478+0000', 'last_undegraded': '2024-09-16T01:36:05.350478+0000', 'last_fullsized': '2024-09-16T01:36:05.350478+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T01:36:09.972 INFO:tasks.ceph.ceph_manager.ceph:PG 7.27 is not active+clean 2024-09-16T01:36:09.972 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.350770+0000', 'last_change': '2024-09-16T01:36:05.350770+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.350770+0000', 'last_undegraded': '2024-09-16T01:36:05.350770+0000', 'last_fullsized': '2024-09-16T01:36:05.350770+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T01:36:09.972 INFO:tasks.ceph.ceph_manager.ceph:PG 7.26 is not active+clean 2024-09-16T01:36:09.972 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.353568+0000', 'last_change': '2024-09-16T01:36:05.353568+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.353568+0000', 'last_undegraded': '2024-09-16T01:36:05.353568+0000', 'last_fullsized': '2024-09-16T01:36:05.353568+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T01:36:09.972 INFO:tasks.ceph.ceph_manager.ceph:PG 7.25 is not active+clean 2024-09-16T01:36:09.972 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.350845+0000', 'last_change': '2024-09-16T01:36:05.350845+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.350845+0000', 'last_undegraded': '2024-09-16T01:36:05.350845+0000', 'last_fullsized': '2024-09-16T01:36:05.350845+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T01:36:09.972 INFO:tasks.ceph.ceph_manager.ceph:PG 7.23 is not active+clean 2024-09-16T01:36:09.972 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.351073+0000', 'last_change': '2024-09-16T01:36:05.351073+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.351073+0000', 'last_undegraded': '2024-09-16T01:36:05.351073+0000', 'last_fullsized': '2024-09-16T01:36:05.351073+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T01:36:09.972 INFO:tasks.ceph.ceph_manager.ceph:PG 7.22 is not active+clean 2024-09-16T01:36:09.972 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.353729+0000', 'last_change': '2024-09-16T01:36:05.353729+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.353729+0000', 'last_undegraded': '2024-09-16T01:36:05.353729+0000', 'last_fullsized': '2024-09-16T01:36:05.353729+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T01:36:09.972 INFO:tasks.ceph.ceph_manager.ceph:PG 7.21 is not active+clean 2024-09-16T01:36:09.973 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.356913+0000', 'last_change': '2024-09-16T01:36:05.356913+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.356913+0000', 'last_undegraded': '2024-09-16T01:36:05.356913+0000', 'last_fullsized': '2024-09-16T01:36:05.356913+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T01:36:09.973 INFO:tasks.ceph.ceph_manager.ceph:PG 7.20 is not active+clean 2024-09-16T01:36:09.973 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.351956+0000', 'last_change': '2024-09-16T01:36:05.351956+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.351956+0000', 'last_undegraded': '2024-09-16T01:36:05.351956+0000', 'last_fullsized': '2024-09-16T01:36:05.351956+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T01:36:09.973 INFO:tasks.ceph.ceph_manager.ceph:PG 7.1e is not active+clean 2024-09-16T01:36:09.973 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.354055+0000', 'last_change': '2024-09-16T01:36:05.354055+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.354055+0000', 'last_undegraded': '2024-09-16T01:36:05.354055+0000', 'last_fullsized': '2024-09-16T01:36:05.354055+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T01:36:09.973 INFO:tasks.ceph.ceph_manager.ceph:PG 7.1f is not active+clean 2024-09-16T01:36:09.973 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.351699+0000', 'last_change': '2024-09-16T01:36:05.351699+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.351699+0000', 'last_undegraded': '2024-09-16T01:36:05.351699+0000', 'last_fullsized': '2024-09-16T01:36:05.351699+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T01:36:09.973 INFO:tasks.ceph.ceph_manager.ceph:PG 7.1d is not active+clean 2024-09-16T01:36:09.973 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.354065+0000', 'last_change': '2024-09-16T01:36:05.354065+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.354065+0000', 'last_undegraded': '2024-09-16T01:36:05.354065+0000', 'last_fullsized': '2024-09-16T01:36:05.354065+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T01:36:09.973 INFO:tasks.ceph.ceph_manager.ceph:PG 7.1a is not active+clean 2024-09-16T01:36:09.973 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.353341+0000', 'last_change': '2024-09-16T01:36:05.353341+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.353341+0000', 'last_undegraded': '2024-09-16T01:36:05.353341+0000', 'last_fullsized': '2024-09-16T01:36:05.353341+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T01:36:09.974 INFO:tasks.ceph.ceph_manager.ceph:PG 7.1b is not active+clean 2024-09-16T01:36:09.974 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.356484+0000', 'last_change': '2024-09-16T01:36:05.356484+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.356484+0000', 'last_undegraded': '2024-09-16T01:36:05.356484+0000', 'last_fullsized': '2024-09-16T01:36:05.356484+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T01:36:09.974 INFO:tasks.ceph.ceph_manager.ceph:PG 7.18 is not active+clean 2024-09-16T01:36:09.974 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.353237+0000', 'last_change': '2024-09-16T01:36:05.353237+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.353237+0000', 'last_undegraded': '2024-09-16T01:36:05.353237+0000', 'last_fullsized': '2024-09-16T01:36:05.353237+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T01:36:09.974 INFO:tasks.ceph.ceph_manager.ceph:PG 7.19 is not active+clean 2024-09-16T01:36:09.974 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.351989+0000', 'last_change': '2024-09-16T01:36:05.351989+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.351989+0000', 'last_undegraded': '2024-09-16T01:36:05.351989+0000', 'last_fullsized': '2024-09-16T01:36:05.351989+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T01:36:09.974 INFO:tasks.ceph.ceph_manager.ceph:PG 7.16 is not active+clean 2024-09-16T01:36:09.974 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.350838+0000', 'last_change': '2024-09-16T01:36:05.350838+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.350838+0000', 'last_undegraded': '2024-09-16T01:36:05.350838+0000', 'last_fullsized': '2024-09-16T01:36:05.350838+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T01:36:09.974 INFO:tasks.ceph.ceph_manager.ceph:PG 7.17 is not active+clean 2024-09-16T01:36:09.974 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.359339+0000', 'last_change': '2024-09-16T01:36:05.359339+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.359339+0000', 'last_undegraded': '2024-09-16T01:36:05.359339+0000', 'last_fullsized': '2024-09-16T01:36:05.359339+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T01:36:09.974 INFO:tasks.ceph.ceph_manager.ceph:PG 7.14 is not active+clean 2024-09-16T01:36:09.974 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.356478+0000', 'last_change': '2024-09-16T01:36:05.356478+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.356478+0000', 'last_undegraded': '2024-09-16T01:36:05.356478+0000', 'last_fullsized': '2024-09-16T01:36:05.356478+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T01:36:09.975 INFO:tasks.ceph.ceph_manager.ceph:PG 7.15 is not active+clean 2024-09-16T01:36:09.975 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.360574+0000', 'last_change': '2024-09-16T01:36:05.360574+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.360574+0000', 'last_undegraded': '2024-09-16T01:36:05.360574+0000', 'last_fullsized': '2024-09-16T01:36:05.360574+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T01:36:09.975 INFO:tasks.ceph.ceph_manager.ceph:PG 7.12 is not active+clean 2024-09-16T01:36:09.975 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.352330+0000', 'last_change': '2024-09-16T01:36:05.352330+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.352330+0000', 'last_undegraded': '2024-09-16T01:36:05.352330+0000', 'last_fullsized': '2024-09-16T01:36:05.352330+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T01:36:09.975 INFO:tasks.ceph.ceph_manager.ceph:PG 7.13 is not active+clean 2024-09-16T01:36:09.975 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.354770+0000', 'last_change': '2024-09-16T01:36:05.354770+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.354770+0000', 'last_undegraded': '2024-09-16T01:36:05.354770+0000', 'last_fullsized': '2024-09-16T01:36:05.354770+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T01:36:09.975 INFO:tasks.ceph.ceph_manager.ceph:PG 7.10 is not active+clean 2024-09-16T01:36:09.975 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.356076+0000', 'last_change': '2024-09-16T01:36:05.356076+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.356076+0000', 'last_undegraded': '2024-09-16T01:36:05.356076+0000', 'last_fullsized': '2024-09-16T01:36:05.356076+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T01:36:09.975 INFO:tasks.ceph.ceph_manager.ceph:PG 7.11 is not active+clean 2024-09-16T01:36:09.975 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.357949+0000', 'last_change': '2024-09-16T01:36:05.357949+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.357949+0000', 'last_undegraded': '2024-09-16T01:36:05.357949+0000', 'last_fullsized': '2024-09-16T01:36:05.357949+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T01:36:09.975 INFO:tasks.ceph.ceph_manager.ceph:PG 7.e is not active+clean 2024-09-16T01:36:09.975 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.353180+0000', 'last_change': '2024-09-16T01:36:05.353180+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.353180+0000', 'last_undegraded': '2024-09-16T01:36:05.353180+0000', 'last_fullsized': '2024-09-16T01:36:05.353180+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T01:36:09.976 INFO:tasks.ceph.ceph_manager.ceph:PG 7.f is not active+clean 2024-09-16T01:36:09.976 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.352248+0000', 'last_change': '2024-09-16T01:36:05.352248+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.352248+0000', 'last_undegraded': '2024-09-16T01:36:05.352248+0000', 'last_fullsized': '2024-09-16T01:36:05.352248+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T01:36:09.976 INFO:tasks.ceph.ceph_manager.ceph:PG 7.c is not active+clean 2024-09-16T01:36:09.976 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.351047+0000', 'last_change': '2024-09-16T01:36:05.351047+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.351047+0000', 'last_undegraded': '2024-09-16T01:36:05.351047+0000', 'last_fullsized': '2024-09-16T01:36:05.351047+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T01:36:09.976 INFO:tasks.ceph.ceph_manager.ceph:PG 7.d is not active+clean 2024-09-16T01:36:09.976 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.351796+0000', 'last_change': '2024-09-16T01:36:05.351796+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.351796+0000', 'last_undegraded': '2024-09-16T01:36:05.351796+0000', 'last_fullsized': '2024-09-16T01:36:05.351796+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T01:36:09.976 INFO:tasks.ceph.ceph_manager.ceph:PG 7.a is not active+clean 2024-09-16T01:36:09.976 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.356095+0000', 'last_change': '2024-09-16T01:36:05.356095+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.356095+0000', 'last_undegraded': '2024-09-16T01:36:05.356095+0000', 'last_fullsized': '2024-09-16T01:36:05.356095+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T01:36:09.976 INFO:tasks.ceph.ceph_manager.ceph:PG 7.b is not active+clean 2024-09-16T01:36:09.976 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.353015+0000', 'last_change': '2024-09-16T01:36:05.353015+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.353015+0000', 'last_undegraded': '2024-09-16T01:36:05.353015+0000', 'last_fullsized': '2024-09-16T01:36:05.353015+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T01:36:09.976 INFO:tasks.ceph.ceph_manager.ceph:PG 7.8 is not active+clean 2024-09-16T01:36:09.977 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.352450+0000', 'last_change': '2024-09-16T01:36:05.352450+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.352450+0000', 'last_undegraded': '2024-09-16T01:36:05.352450+0000', 'last_fullsized': '2024-09-16T01:36:05.352450+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T01:36:09.977 INFO:tasks.ceph.ceph_manager.ceph:PG 7.9 is not active+clean 2024-09-16T01:36:09.977 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.353294+0000', 'last_change': '2024-09-16T01:36:05.353294+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.353294+0000', 'last_undegraded': '2024-09-16T01:36:05.353294+0000', 'last_fullsized': '2024-09-16T01:36:05.353294+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T01:36:09.977 INFO:tasks.ceph.ceph_manager.ceph:PG 7.7 is not active+clean 2024-09-16T01:36:09.977 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.351224+0000', 'last_change': '2024-09-16T01:36:05.351224+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.351224+0000', 'last_undegraded': '2024-09-16T01:36:05.351224+0000', 'last_fullsized': '2024-09-16T01:36:05.351224+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T01:36:09.977 INFO:tasks.ceph.ceph_manager.ceph:PG 7.4 is not active+clean 2024-09-16T01:36:09.977 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.352527+0000', 'last_change': '2024-09-16T01:36:05.352527+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.352527+0000', 'last_undegraded': '2024-09-16T01:36:05.352527+0000', 'last_fullsized': '2024-09-16T01:36:05.352527+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T01:36:09.977 INFO:tasks.ceph.ceph_manager.ceph:PG 7.5 is not active+clean 2024-09-16T01:36:09.977 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.359667+0000', 'last_change': '2024-09-16T01:36:05.359667+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.359667+0000', 'last_undegraded': '2024-09-16T01:36:05.359667+0000', 'last_fullsized': '2024-09-16T01:36:05.359667+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-16T01:36:09.977 INFO:tasks.ceph.ceph_manager.ceph:PG 7.2 is not active+clean 2024-09-16T01:36:09.977 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.354612+0000', 'last_change': '2024-09-16T01:36:05.354612+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.354612+0000', 'last_undegraded': '2024-09-16T01:36:05.354612+0000', 'last_fullsized': '2024-09-16T01:36:05.354612+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T01:36:09.978 INFO:tasks.ceph.ceph_manager.ceph:PG 7.3 is not active+clean 2024-09-16T01:36:09.978 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.355177+0000', 'last_change': '2024-09-16T01:36:05.355177+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.355177+0000', 'last_undegraded': '2024-09-16T01:36:05.355177+0000', 'last_fullsized': '2024-09-16T01:36:05.355177+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-16T01:36:09.978 INFO:tasks.ceph.ceph_manager.ceph:PG 7.1 is not active+clean 2024-09-16T01:36:09.978 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.350199+0000', 'last_change': '2024-09-16T01:36:05.350199+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.350199+0000', 'last_undegraded': '2024-09-16T01:36:05.350199+0000', 'last_fullsized': '2024-09-16T01:36:05.350199+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-16T01:36:09.978 INFO:tasks.ceph.ceph_manager.ceph:PG 7.6 is not active+clean 2024-09-16T01:36:09.978 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-16T01:36:05.353674+0000', 'last_change': '2024-09-16T01:36:05.353674+0000', 'last_active': '2024-09-16T01:36:05.342630+0000', 'last_peered': '2024-09-16T01:36:05.342630+0000', 'last_clean': '2024-09-16T01:36:05.342630+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-16T01:36:05.353674+0000', 'last_undegraded': '2024-09-16T01:36:05.353674+0000', 'last_fullsized': '2024-09-16T01:36:05.353674+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'last_clean_scrub_stamp': '2024-09-16T01:36:05.342630+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T01:36:09.978 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T01:36:10.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.099+0000 7fc6ac7d6640 1 Processor -- start 2024-09-16T01:36:10.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.099+0000 7fc6ac7d6640 1 -- start start 2024-09-16T01:36:10.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.100+0000 7fc6ac7d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6a415e560 0x7fc6a406e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:10.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.100+0000 7fc6ac7d6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc6a4072220 0x7fc6a40ae890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:10.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.100+0000 7fc6ac7d6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc6a4073130 0x7fc6a40735c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:10.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.100+0000 7fc6ac7d6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc6a40b5220 con 0x7fc6a4073130 2024-09-16T01:36:10.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.100+0000 7fc6ac7d6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc6a40b54c0 con 0x7fc6a4072220 2024-09-16T01:36:10.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.100+0000 7fc6ac7d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc6a40b57a0 con 0x7fc6a415e560 2024-09-16T01:36:10.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.100+0000 7fc6aad73640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc6a4073130 0x7fc6a40735c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:10.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.100+0000 7fc6aad73640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc6a4073130 0x7fc6a40735c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51124/0 (socket says 172.21.15.88:51124) 2024-09-16T01:36:10.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.100+0000 7fc6aad73640 1 -- 172.21.15.88:0/1585469505 learned_addr learned my addr 172.21.15.88:0/1585469505 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:10.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.100+0000 7fc6aad73640 1 -- 172.21.15.88:0/1585469505 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6a415e560 msgr2=0x7fc6a406e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:10.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.100+0000 7fc6aad73640 1 --2- 172.21.15.88:0/1585469505 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6a415e560 0x7fc6a406e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:10.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.100+0000 7fc6a9d71640 1 --2- 172.21.15.88:0/1585469505 >> v2:172.21.15.154:3300/0 conn(0x7fc6a4072220 0x7fc6a40ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:10.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.100+0000 7fc6aad73640 1 -- 172.21.15.88:0/1585469505 >> v2:172.21.15.154:3300/0 conn(0x7fc6a4072220 msgr2=0x7fc6a40ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:10.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.100+0000 7fc6aad73640 1 --2- 172.21.15.88:0/1585469505 >> v2:172.21.15.154:3300/0 conn(0x7fc6a4072220 0x7fc6a40ae890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:10.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.100+0000 7fc6aad73640 1 -- 172.21.15.88:0/1585469505 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6a40b5a80 con 0x7fc6a4073130 2024-09-16T01:36:10.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.101+0000 7fc6aad73640 1 --2- 172.21.15.88:0/1585469505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6a4073130 0x7fc6a40735c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4062c10 tx=0x7fc69c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=737ac3bbb3a7870b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:10.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.101+0000 7fc6a9570640 1 -- 172.21.15.88:0/1585469505 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69c036070 con 0x7fc6a4073130 2024-09-16T01:36:10.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.101+0000 7fc6a9570640 1 -- 172.21.15.88:0/1585469505 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc69c031600 con 0x7fc6a4073130 2024-09-16T01:36:10.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.101+0000 7fc6a9570640 1 -- 172.21.15.88:0/1585469505 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69c0319d0 con 0x7fc6a4073130 2024-09-16T01:36:10.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.102+0000 7fc6ac7d6640 1 -- 172.21.15.88:0/1585469505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6a4073130 msgr2=0x7fc6a40735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:10.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.102+0000 7fc6ac7d6640 1 --2- 172.21.15.88:0/1585469505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6a4073130 0x7fc6a40735c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4062c10 tx=0x7fc69c02a6a0 comp rx=0 tx=0).stop 2024-09-16T01:36:10.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.102+0000 7fc6ac7d6640 1 -- 172.21.15.88:0/1585469505 shutdown_connections 2024-09-16T01:36:10.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.102+0000 7fc6ac7d6640 1 --2- 172.21.15.88:0/1585469505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6a4073130 0x7fc6a40735c0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:10.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.102+0000 7fc6ac7d6640 1 --2- 172.21.15.88:0/1585469505 >> v2:172.21.15.154:3300/0 conn(0x7fc6a4072220 0x7fc6a40ae890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:10.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.102+0000 7fc6ac7d6640 1 --2- 172.21.15.88:0/1585469505 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6a415e560 0x7fc6a406e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:10.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.102+0000 7fc6ac7d6640 1 -- 172.21.15.88:0/1585469505 >> 172.21.15.88:0/1585469505 conn(0x7fc6a406bc10 msgr2=0x7fc6a406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:10.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.102+0000 7fc6ac7d6640 1 -- 172.21.15.88:0/1585469505 shutdown_connections 2024-09-16T01:36:10.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.102+0000 7fc6ac7d6640 1 -- 172.21.15.88:0/1585469505 wait complete. 2024-09-16T01:36:10.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.103+0000 7fc6ac7d6640 1 Processor -- start 2024-09-16T01:36:10.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.103+0000 7fc6ac7d6640 1 -- start start 2024-09-16T01:36:10.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.104+0000 7fc6ac7d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6a4072220 0x7fc6a414d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:10.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.104+0000 7fc6ac7d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6a4073130 0x7fc6a4156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:10.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.104+0000 7fc6ac7d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6a415e560 0x7fc6a414fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:10.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.104+0000 7fc6ac7d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a415e940 con 0x7fc6a4072220 2024-09-16T01:36:10.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.104+0000 7fc6ac7d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a4072660 con 0x7fc6a4073130 2024-09-16T01:36:10.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.104+0000 7fc6ac7d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc6a415eac0 con 0x7fc6a415e560 2024-09-16T01:36:10.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.104+0000 7fc6aa572640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6a4072220 0x7fc6a414d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:10.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.104+0000 7fc6aa572640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6a4072220 0x7fc6a414d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:51128/0 (socket says 172.21.15.88:51128) 2024-09-16T01:36:10.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.104+0000 7fc6aa572640 1 -- 172.21.15.88:0/1336363859 learned_addr learned my addr 172.21.15.88:0/1336363859 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:10.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.104+0000 7fc6a9d71640 1 --2- 172.21.15.88:0/1336363859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6a4073130 0x7fc6a4156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:10.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.104+0000 7fc6aad73640 1 --2- 172.21.15.88:0/1336363859 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6a415e560 0x7fc6a414fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:10.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.104+0000 7fc6aa572640 1 -- 172.21.15.88:0/1336363859 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6a415e560 msgr2=0x7fc6a414fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:10.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.104+0000 7fc6aa572640 1 --2- 172.21.15.88:0/1336363859 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6a415e560 0x7fc6a414fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:10.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.104+0000 7fc6aa572640 1 -- 172.21.15.88:0/1336363859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6a4073130 msgr2=0x7fc6a4156d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:10.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.104+0000 7fc6aa572640 1 --2- 172.21.15.88:0/1336363859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6a4073130 0x7fc6a4156d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:10.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.104+0000 7fc6aa572640 1 -- 172.21.15.88:0/1336363859 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6a4157640 con 0x7fc6a4072220 2024-09-16T01:36:10.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.104+0000 7fc6a9d71640 1 --2- 172.21.15.88:0/1336363859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6a4073130 0x7fc6a4156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:10.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.104+0000 7fc6aa572640 1 --2- 172.21.15.88:0/1336363859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6a4072220 0x7fc6a414d450 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000b4a0 tx=0x7fc6a000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:10.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.104+0000 7fc69affd640 1 -- 172.21.15.88:0/1336363859 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0014070 con 0x7fc6a4072220 2024-09-16T01:36:10.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.104+0000 7fc69affd640 1 -- 172.21.15.88:0/1336363859 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6a0010610 con 0x7fc6a4072220 2024-09-16T01:36:10.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.104+0000 7fc6ac7d6640 1 -- 172.21.15.88:0/1336363859 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6a414ef50 con 0x7fc6a4072220 2024-09-16T01:36:10.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.104+0000 7fc69affd640 1 -- 172.21.15.88:0/1336363859 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0010a00 con 0x7fc6a4072220 2024-09-16T01:36:10.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.104+0000 7fc6ac7d6640 1 -- 172.21.15.88:0/1336363859 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6a414f510 con 0x7fc6a4072220 2024-09-16T01:36:10.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.105+0000 7fc6ac7d6640 1 -- 172.21.15.88:0/1336363859 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6a406e780 con 0x7fc6a4072220 2024-09-16T01:36:10.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.106+0000 7fc69affd640 1 -- 172.21.15.88:0/1336363859 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc6a0017070 con 0x7fc6a4072220 2024-09-16T01:36:10.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.106+0000 7fc69affd640 1 --2- 172.21.15.88:0/1336363859 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc68806c2b0 0x7fc68806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:10.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.106+0000 7fc6a9d71640 1 --2- 172.21.15.88:0/1336363859 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc68806c2b0 0x7fc68806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:10.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.106+0000 7fc69affd640 1 -- 172.21.15.88:0/1336363859 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fc6a008adb0 con 0x7fc6a4072220 2024-09-16T01:36:10.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.107+0000 7fc6a9d71640 1 --2- 172.21.15.88:0/1336363859 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc68806c2b0 0x7fc68806e770 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc6a41f7e50 tx=0x7fc694008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:10.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.111+0000 7fc69affd640 1 -- 172.21.15.88:0/1336363859 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc6a00595b0 con 0x7fc6a4072220 2024-09-16T01:36:10.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.338+0000 7fc6ac7d6640 1 -- 172.21.15.88:0/1336363859 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7fc6a406e4d0 con 0x7fc6a4072220 2024-09-16T01:36:10.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.339+0000 7fc69affd640 1 -- 172.21.15.88:0/1336363859 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1139 (secure 0 0 0) 0x7fc6a0058eb0 con 0x7fc6a4072220 2024-09-16T01:36:10.340 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:10.340 INFO:teuthology.orchestra.run.smithi088.stdout:{"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":131,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":8,"num_up_osds":8,"osd_up_since":1726450450,"num_in_osds":8,"osd_in_since":1726450440,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":58},{"state_name":"active+clean","count":39}],"num_pgs":97,"num_pools":3,"num_objects":4,"data_bytes":590438,"bytes_used":226549760,"bytes_avail":772867563520,"bytes_total":773094113280,"inactive_pgs_ratio":0.59793812036514282,"write_bytes_sec":244,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":25,"btime":"2024-09-16T01:36:08:365397+0000","id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"c","status":"up:active","gid":5406}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T01:35:56.536609+0000","services":{}},"progress_events":{}} 2024-09-16T01:36:10.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.340+0000 7fc6ac7d6640 1 -- 172.21.15.88:0/1336363859 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc68806c2b0 msgr2=0x7fc68806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:10.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.340+0000 7fc6ac7d6640 1 --2- 172.21.15.88:0/1336363859 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc68806c2b0 0x7fc68806e770 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc6a41f7e50 tx=0x7fc694008040 comp rx=0 tx=0).stop 2024-09-16T01:36:10.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.340+0000 7fc6ac7d6640 1 -- 172.21.15.88:0/1336363859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6a4072220 msgr2=0x7fc6a414d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:10.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.340+0000 7fc6ac7d6640 1 --2- 172.21.15.88:0/1336363859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6a4072220 0x7fc6a414d450 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000b4a0 tx=0x7fc6a000b960 comp rx=0 tx=0).stop 2024-09-16T01:36:10.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.341+0000 7fc6ac7d6640 1 -- 172.21.15.88:0/1336363859 shutdown_connections 2024-09-16T01:36:10.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.341+0000 7fc6ac7d6640 1 --2- 172.21.15.88:0/1336363859 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc68806c2b0 0x7fc68806e770 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:10.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.341+0000 7fc6ac7d6640 1 --2- 172.21.15.88:0/1336363859 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6a415e560 0x7fc6a414fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:10.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.341+0000 7fc6ac7d6640 1 --2- 172.21.15.88:0/1336363859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6a4073130 0x7fc6a4156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:10.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.341+0000 7fc6ac7d6640 1 --2- 172.21.15.88:0/1336363859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6a4072220 0x7fc6a414d450 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:10.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.341+0000 7fc6ac7d6640 1 -- 172.21.15.88:0/1336363859 >> 172.21.15.88:0/1336363859 conn(0x7fc6a406bc10 msgr2=0x7fc6a40717a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:10.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.341+0000 7fc6ac7d6640 1 -- 172.21.15.88:0/1336363859 shutdown_connections 2024-09-16T01:36:10.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.341+0000 7fc6ac7d6640 1 -- 172.21.15.88:0/1336363859 wait complete. 2024-09-16T01:36:10.353 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-16T01:36:10.353 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:36:10.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.449+0000 7f2e4c386640 1 Processor -- start 2024-09-16T01:36:10.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.449+0000 7f2e4c386640 1 -- start start 2024-09-16T01:36:10.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.450+0000 7f2e4c386640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2e440b64e0 0x7f2e44167110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:10.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.450+0000 7f2e4c386640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e440b49c0 con 0x7f2e4415c170 2024-09-16T01:36:10.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.450+0000 7f2e4c386640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e440b4cf0 con 0x7f2e4415c550 2024-09-16T01:36:10.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.450+0000 7f2e4c386640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2e440b4fd0 con 0x7f2e440b64e0 2024-09-16T01:36:10.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.450+0000 7f2e4a923640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2e440b64e0 0x7f2e44167110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:10.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.450+0000 7f2e4a923640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2e440b64e0 0x7f2e44167110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48658/0 (socket says 172.21.15.88:48658) 2024-09-16T01:36:10.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.450+0000 7f2e4a122640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f2e4415c170 0x7f2e440cf6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:51568/0 (socket says 172.21.15.88:51568) 2024-09-16T01:36:10.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.450+0000 7f2e4a923640 1 -- 172.21.15.88:0/2686543548 learned_addr learned my addr 172.21.15.88:0/2686543548 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:10.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.451+0000 7f2e4a923640 1 -- 172.21.15.88:0/2686543548 >> v1:172.21.15.154:6789/0 conn(0x7f2e4415c550 legacy=0x7f2e440b5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:10.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.451+0000 7f2e4a923640 1 -- 172.21.15.88:0/2686543548 >> v1:172.21.15.88:6789/0 conn(0x7f2e4415c170 legacy=0x7f2e440cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:10.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.451+0000 7f2e4a923640 1 -- 172.21.15.88:0/2686543548 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e440b52b0 con 0x7f2e440b64e0 2024-09-16T01:36:10.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.451+0000 7f2e4a923640 1 --2- 172.21.15.88:0/2686543548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e440b64e0 0x7f2e44167110 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c009e80 tx=0x7f2e3c02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94185236411c0ab5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:10.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.451+0000 7f2e49120640 1 -- 172.21.15.88:0/2686543548 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c036070 con 0x7f2e440b64e0 2024-09-16T01:36:10.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.451+0000 7f2e49120640 1 -- 172.21.15.88:0/2686543548 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e3c031600 con 0x7f2e440b64e0 2024-09-16T01:36:10.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.451+0000 7f2e49120640 1 -- 172.21.15.88:0/2686543548 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c0319d0 con 0x7f2e440b64e0 2024-09-16T01:36:10.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.452+0000 7f2e4c386640 1 -- 172.21.15.88:0/2686543548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e440b64e0 msgr2=0x7f2e44167110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:10.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.452+0000 7f2e4c386640 1 --2- 172.21.15.88:0/2686543548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e440b64e0 0x7f2e44167110 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c009e80 tx=0x7f2e3c02a710 comp rx=0 tx=0).stop 2024-09-16T01:36:10.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.452+0000 7f2e4c386640 1 -- 172.21.15.88:0/2686543548 shutdown_connections 2024-09-16T01:36:10.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.452+0000 7f2e4c386640 1 --2- 172.21.15.88:0/2686543548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e440b64e0 0x7f2e44167110 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:10.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.452+0000 7f2e4c386640 1 -- 172.21.15.88:0/2686543548 >> 172.21.15.88:0/2686543548 conn(0x7f2e4406bc10 msgr2=0x7f2e4406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:10.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.452+0000 7f2e4c386640 1 -- 172.21.15.88:0/2686543548 shutdown_connections 2024-09-16T01:36:10.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.452+0000 7f2e4c386640 1 -- 172.21.15.88:0/2686543548 wait complete. 2024-09-16T01:36:10.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.453+0000 7f2e4c386640 1 Processor -- start 2024-09-16T01:36:10.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.453+0000 7f2e4c386640 1 -- start start 2024-09-16T01:36:10.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.454+0000 7f2e4c386640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e440b64e0 0x7f2e44147490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:10.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.454+0000 7f2e4c386640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e4415c170 0x7f2e44159760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:10.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.454+0000 7f2e4c386640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e4415c550 0x7f2e4414e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:10.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.454+0000 7f2e4c386640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2e440b5b50 con 0x7f2e4415c550 2024-09-16T01:36:10.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.454+0000 7f2e4c386640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2e440b5870 con 0x7f2e440b64e0 2024-09-16T01:36:10.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.454+0000 7f2e4c386640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2e4415c9c0 con 0x7f2e4415c170 2024-09-16T01:36:10.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.454+0000 7f2e4a923640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e4415c550 0x7f2e4414e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:10.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.454+0000 7f2e4a923640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e4415c550 0x7f2e4414e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48660/0 (socket says 172.21.15.88:48660) 2024-09-16T01:36:10.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.454+0000 7f2e4a923640 1 -- 172.21.15.88:0/236789812 learned_addr learned my addr 172.21.15.88:0/236789812 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:10.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.454+0000 7f2e4a122640 1 --2- 172.21.15.88:0/236789812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e440b64e0 0x7f2e44147490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:10.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.454+0000 7f2e4a923640 1 -- 172.21.15.88:0/236789812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e4415c170 msgr2=0x7f2e44159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:10.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.454+0000 7f2e4a923640 1 --2- 172.21.15.88:0/236789812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e4415c170 0x7f2e44159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:10.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.454+0000 7f2e4a923640 1 -- 172.21.15.88:0/236789812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e440b64e0 msgr2=0x7f2e44147490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:10.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.454+0000 7f2e4a923640 1 --2- 172.21.15.88:0/236789812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e440b64e0 0x7f2e44147490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:10.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.454+0000 7f2e4a923640 1 -- 172.21.15.88:0/236789812 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e4414dfb0 con 0x7f2e4415c550 2024-09-16T01:36:10.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.454+0000 7f2e4a122640 1 --2- 172.21.15.88:0/236789812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e440b64e0 0x7f2e44147490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:10.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.454+0000 7f2e4a923640 1 --2- 172.21.15.88:0/236789812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e4415c550 0x7f2e4414e470 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c034040 tx=0x7f2e3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:10.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.455+0000 7f2e3affd640 1 -- 172.21.15.88:0/236789812 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c036050 con 0x7f2e4415c550 2024-09-16T01:36:10.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.455+0000 7f2e3affd640 1 -- 172.21.15.88:0/236789812 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e3c042070 con 0x7f2e4415c550 2024-09-16T01:36:10.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.455+0000 7f2e4c386640 1 -- 172.21.15.88:0/236789812 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e44149e50 con 0x7f2e4415c550 2024-09-16T01:36:10.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.455+0000 7f2e3affd640 1 -- 172.21.15.88:0/236789812 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c009510 con 0x7f2e4415c550 2024-09-16T01:36:10.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.455+0000 7f2e4c386640 1 -- 172.21.15.88:0/236789812 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e44148c10 con 0x7f2e4415c550 2024-09-16T01:36:10.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.456+0000 7f2e4c386640 1 -- 172.21.15.88:0/236789812 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e440b5dd0 con 0x7f2e4415c550 2024-09-16T01:36:10.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.456+0000 7f2e3affd640 1 -- 172.21.15.88:0/236789812 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2e3c007bd0 con 0x7f2e4415c550 2024-09-16T01:36:10.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.457+0000 7f2e3affd640 1 --2- 172.21.15.88:0/236789812 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2e2806c2b0 0x7f2e2806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:10.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.457+0000 7f2e4a122640 1 --2- 172.21.15.88:0/236789812 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2e2806c2b0 0x7f2e2806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:10.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.457+0000 7f2e3affd640 1 -- 172.21.15.88:0/236789812 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f2e3c0ac240 con 0x7f2e4415c550 2024-09-16T01:36:10.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.457+0000 7f2e4a122640 1 --2- 172.21.15.88:0/236789812 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2e2806c2b0 0x7f2e2806e770 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2e40000f60 tx=0x7f2e40009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:10.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.461+0000 7f2e3affd640 1 -- 172.21.15.88:0/236789812 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2e4414dfb0 con 0x7f2e4415c550 2024-09-16T01:36:10.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.624+0000 7f2e4c386640 1 -- 172.21.15.88:0/236789812 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2e4414a130 con 0x7f2e2806c2b0 2024-09-16T01:36:10.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.630+0000 7f2e3affd640 1 -- 172.21.15.88:0/236789812 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+247134 (secure 0 0 0) 0x7f2e4414a130 con 0x7f2e2806c2b0 2024-09-16T01:36:10.634 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:10.635 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:36:10.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.632+0000 7f2e4c386640 1 -- 172.21.15.88:0/236789812 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2e2806c2b0 msgr2=0x7f2e2806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:10.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.632+0000 7f2e4c386640 1 --2- 172.21.15.88:0/236789812 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2e2806c2b0 0x7f2e2806e770 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2e40000f60 tx=0x7f2e40009040 comp rx=0 tx=0).stop 2024-09-16T01:36:10.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.632+0000 7f2e4c386640 1 -- 172.21.15.88:0/236789812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e4415c550 msgr2=0x7f2e4414e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:10.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.632+0000 7f2e4c386640 1 --2- 172.21.15.88:0/236789812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e4415c550 0x7f2e4414e470 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c034040 tx=0x7f2e3c004a40 comp rx=0 tx=0).stop 2024-09-16T01:36:10.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.633+0000 7f2e4c386640 1 -- 172.21.15.88:0/236789812 shutdown_connections 2024-09-16T01:36:10.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.633+0000 7f2e4c386640 1 --2- 172.21.15.88:0/236789812 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2e2806c2b0 0x7f2e2806e770 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:10.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.633+0000 7f2e4c386640 1 --2- 172.21.15.88:0/236789812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e4415c550 0x7f2e4414e470 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:10.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.633+0000 7f2e4c386640 1 --2- 172.21.15.88:0/236789812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e4415c170 0x7f2e44159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:10.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.633+0000 7f2e4c386640 1 --2- 172.21.15.88:0/236789812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e440b64e0 0x7f2e44147490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:10.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.633+0000 7f2e4c386640 1 -- 172.21.15.88:0/236789812 >> 172.21.15.88:0/236789812 conn(0x7f2e4406bc10 msgr2=0x7f2e4406d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:10.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.633+0000 7f2e4c386640 1 -- 172.21.15.88:0/236789812 shutdown_connections 2024-09-16T01:36:10.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:10.633+0000 7f2e4c386640 1 -- 172.21.15.88:0/236789812 wait complete. 2024-09-16T01:36:10.645 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2024-09-16T01:36:10.538670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590438,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":163,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":25,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221240,"kb_used_data":4980,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753480,"statfs":{"total":773094113280,"available":772867563520,"internally_reserved":0,"allocated":5099520,"data_stored":2729763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000928"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.359868+0000","last_change":"2024-09-16T01:36:05.359868+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.359868+0000","last_undegraded":"2024-09-16T01:36:05.359868+0000","last_fullsized":"2024-09-16T01:36:05.359868+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.370012+0000","last_change":"2024-09-16T01:36:06.366967+0000","last_active":"2024-09-16T01:36:06.370012+0000","last_peered":"2024-09-16T01:36:06.370012+0000","last_clean":"2024-09-16T01:36:06.370012+0000","last_became_active":"2024-09-16T01:36:06.366688+0000","last_became_peered":"2024-09-16T01:36:06.366688+0000","last_unstale":"2024-09-16T01:36:06.370012+0000","last_undegraded":"2024-09-16T01:36:06.370012+0000","last_fullsized":"2024-09-16T01:36:06.370012+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:40:52.576920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352493+0000","last_change":"2024-09-16T01:36:05.352493+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352493+0000","last_undegraded":"2024-09-16T01:36:05.352493+0000","last_fullsized":"2024-09-16T01:36:05.352493+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358690+0000","last_change":"2024-09-16T01:36:05.358690+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358690+0000","last_undegraded":"2024-09-16T01:36:05.358690+0000","last_fullsized":"2024-09-16T01:36:05.358690+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.371287+0000","last_change":"2024-09-16T01:36:06.369570+0000","last_active":"2024-09-16T01:36:06.371287+0000","last_peered":"2024-09-16T01:36:06.371287+0000","last_clean":"2024-09-16T01:36:06.371287+0000","last_became_active":"2024-09-16T01:36:06.369276+0000","last_became_peered":"2024-09-16T01:36:06.369276+0000","last_unstale":"2024-09-16T01:36:06.371287+0000","last_undegraded":"2024-09-16T01:36:06.371287+0000","last_fullsized":"2024-09-16T01:36:06.371287+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:39:28.693098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358381+0000","last_change":"2024-09-16T01:36:05.358381+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358381+0000","last_undegraded":"2024-09-16T01:36:05.358381+0000","last_fullsized":"2024-09-16T01:36:05.358381+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358435+0000","last_change":"2024-09-16T01:36:05.358435+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358435+0000","last_undegraded":"2024-09-16T01:36:05.358435+0000","last_fullsized":"2024-09-16T01:36:05.358435+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358875+0000","last_change":"2024-09-16T01:36:05.358875+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358875+0000","last_undegraded":"2024-09-16T01:36:05.358875+0000","last_fullsized":"2024-09-16T01:36:05.358875+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358586+0000","last_change":"2024-09-16T01:36:05.358586+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358586+0000","last_undegraded":"2024-09-16T01:36:05.358586+0000","last_fullsized":"2024-09-16T01:36:05.358586+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358111+0000","last_change":"2024-09-16T01:36:05.358111+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358111+0000","last_undegraded":"2024-09-16T01:36:05.358111+0000","last_fullsized":"2024-09-16T01:36:05.358111+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.357648+0000","last_change":"2024-09-16T01:36:05.357648+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.357648+0000","last_undegraded":"2024-09-16T01:36:05.357648+0000","last_fullsized":"2024-09-16T01:36:05.357648+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.376012+0000","last_change":"2024-09-16T01:36:06.374437+0000","last_active":"2024-09-16T01:36:06.376012+0000","last_peered":"2024-09-16T01:36:06.376012+0000","last_clean":"2024-09-16T01:36:06.376012+0000","last_became_active":"2024-09-16T01:36:06.374156+0000","last_became_peered":"2024-09-16T01:36:06.374156+0000","last_unstale":"2024-09-16T01:36:06.376012+0000","last_undegraded":"2024-09-16T01:36:06.376012+0000","last_fullsized":"2024-09-16T01:36:06.376012+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:14:35.252633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352763+0000","last_change":"2024-09-16T01:36:05.352763+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352763+0000","last_undegraded":"2024-09-16T01:36:05.352763+0000","last_fullsized":"2024-09-16T01:36:05.352763+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356941+0000","last_change":"2024-09-16T01:36:05.356941+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356941+0000","last_undegraded":"2024-09-16T01:36:05.356941+0000","last_fullsized":"2024-09-16T01:36:05.356941+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358500+0000","last_change":"2024-09-16T01:36:05.358500+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358500+0000","last_undegraded":"2024-09-16T01:36:05.358500+0000","last_fullsized":"2024-09-16T01:36:05.358500+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352230+0000","last_change":"2024-09-16T01:36:05.352230+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352230+0000","last_undegraded":"2024-09-16T01:36:05.352230+0000","last_fullsized":"2024-09-16T01:36:05.352230+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351446+0000","last_change":"2024-09-16T01:36:05.351446+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351446+0000","last_undegraded":"2024-09-16T01:36:05.351446+0000","last_fullsized":"2024-09-16T01:36:05.351446+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351350+0000","last_change":"2024-09-16T01:36:05.351350+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351350+0000","last_undegraded":"2024-09-16T01:36:05.351350+0000","last_fullsized":"2024-09-16T01:36:05.351350+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.350520+0000","last_change":"2024-09-16T01:36:05.350520+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.350520+0000","last_undegraded":"2024-09-16T01:36:05.350520+0000","last_fullsized":"2024-09-16T01:36:05.350520+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.354933+0000","last_change":"2024-09-16T01:36:05.354933+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.354933+0000","last_undegraded":"2024-09-16T01:36:05.354933+0000","last_fullsized":"2024-09-16T01:36:05.354933+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.358886+0000","last_change":"2024-09-16T01:36:05.358886+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.358886+0000","last_undegraded":"2024-09-16T01:36:05.358886+0000","last_fullsized":"2024-09-16T01:36:05.358886+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356521+0000","last_change":"2024-09-16T01:36:05.356521+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356521+0000","last_undegraded":"2024-09-16T01:36:05.356521+0000","last_fullsized":"2024-09-16T01:36:05.356521+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351493+0000","last_change":"2024-09-16T01:36:05.351493+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351493+0000","last_undegraded":"2024-09-16T01:36:05.351493+0000","last_fullsized":"2024-09-16T01:36:05.351493+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.350478+0000","last_change":"2024-09-16T01:36:05.350478+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.350478+0000","last_undegraded":"2024-09-16T01:36:05.350478+0000","last_fullsized":"2024-09-16T01:36:05.350478+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.350770+0000","last_change":"2024-09-16T01:36:05.350770+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.350770+0000","last_undegraded":"2024-09-16T01:36:05.350770+0000","last_fullsized":"2024-09-16T01:36:05.350770+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353568+0000","last_change":"2024-09-16T01:36:05.353568+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353568+0000","last_undegraded":"2024-09-16T01:36:05.353568+0000","last_fullsized":"2024-09-16T01:36:05.353568+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.350845+0000","last_change":"2024-09-16T01:36:05.350845+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.350845+0000","last_undegraded":"2024-09-16T01:36:05.350845+0000","last_fullsized":"2024-09-16T01:36:05.350845+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.370131+0000","last_change":"2024-09-16T01:36:06.365094+0000","last_active":"2024-09-16T01:36:06.370131+0000","last_peered":"2024-09-16T01:36:06.370131+0000","last_clean":"2024-09-16T01:36:06.370131+0000","last_became_active":"2024-09-16T01:36:06.364790+0000","last_became_peered":"2024-09-16T01:36:06.364790+0000","last_unstale":"2024-09-16T01:36:06.370131+0000","last_undegraded":"2024-09-16T01:36:06.370131+0000","last_fullsized":"2024-09-16T01:36:06.370131+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:29:16.206145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351073+0000","last_change":"2024-09-16T01:36:05.351073+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351073+0000","last_undegraded":"2024-09-16T01:36:05.351073+0000","last_fullsized":"2024-09-16T01:36:05.351073+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353729+0000","last_change":"2024-09-16T01:36:05.353729+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353729+0000","last_undegraded":"2024-09-16T01:36:05.353729+0000","last_fullsized":"2024-09-16T01:36:05.353729+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356913+0000","last_change":"2024-09-16T01:36:05.356913+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356913+0000","last_undegraded":"2024-09-16T01:36:05.356913+0000","last_fullsized":"2024-09-16T01:36:05.356913+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351956+0000","last_change":"2024-09-16T01:36:05.351956+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351956+0000","last_undegraded":"2024-09-16T01:36:05.351956+0000","last_fullsized":"2024-09-16T01:36:05.351956+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.354055+0000","last_change":"2024-09-16T01:36:05.354055+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.354055+0000","last_undegraded":"2024-09-16T01:36:05.354055+0000","last_fullsized":"2024-09-16T01:36:05.354055+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.360688+0000","last_change":"2024-09-16T01:36:05.357580+0000","last_active":"2024-09-16T01:36:05.360688+0000","last_peered":"2024-09-16T01:36:05.360688+0000","last_clean":"2024-09-16T01:36:05.360688+0000","last_became_active":"2024-09-16T01:36:05.357331+0000","last_became_peered":"2024-09-16T01:36:05.357331+0000","last_unstale":"2024-09-16T01:36:05.360688+0000","last_undegraded":"2024-09-16T01:36:05.360688+0000","last_fullsized":"2024-09-16T01:36:05.360688+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:28:41.466172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351699+0000","last_change":"2024-09-16T01:36:05.351699+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351699+0000","last_undegraded":"2024-09-16T01:36:05.351699+0000","last_fullsized":"2024-09-16T01:36:05.351699+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.365028+0000","last_change":"2024-09-16T01:36:05.357954+0000","last_active":"2024-09-16T01:36:05.365028+0000","last_peered":"2024-09-16T01:36:05.365028+0000","last_clean":"2024-09-16T01:36:05.365028+0000","last_became_active":"2024-09-16T01:36:05.357018+0000","last_became_peered":"2024-09-16T01:36:05.357018+0000","last_unstale":"2024-09-16T01:36:05.365028+0000","last_undegraded":"2024-09-16T01:36:05.365028+0000","last_fullsized":"2024-09-16T01:36:05.365028+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:53.565250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.378364+0000","last_change":"2024-09-16T01:36:06.377925+0000","last_active":"2024-09-16T01:36:06.378364+0000","last_peered":"2024-09-16T01:36:06.378364+0000","last_clean":"2024-09-16T01:36:06.378364+0000","last_became_active":"2024-09-16T01:36:06.377631+0000","last_became_peered":"2024-09-16T01:36:06.377631+0000","last_unstale":"2024-09-16T01:36:06.378364+0000","last_undegraded":"2024-09-16T01:36:06.378364+0000","last_fullsized":"2024-09-16T01:36:06.378364+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:09:13.085891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368709+0000","last_change":"2024-09-16T01:36:05.365158+0000","last_active":"2024-09-16T01:36:05.368709+0000","last_peered":"2024-09-16T01:36:05.368709+0000","last_clean":"2024-09-16T01:36:05.368709+0000","last_became_active":"2024-09-16T01:36:05.364885+0000","last_became_peered":"2024-09-16T01:36:05.364885+0000","last_unstale":"2024-09-16T01:36:05.368709+0000","last_undegraded":"2024-09-16T01:36:05.368709+0000","last_fullsized":"2024-09-16T01:36:05.368709+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:06:24.080498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.354065+0000","last_change":"2024-09-16T01:36:05.354065+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.354065+0000","last_undegraded":"2024-09-16T01:36:05.354065+0000","last_fullsized":"2024-09-16T01:36:05.354065+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373524+0000","last_change":"2024-09-16T01:36:05.368193+0000","last_active":"2024-09-16T01:36:05.373524+0000","last_peered":"2024-09-16T01:36:05.373524+0000","last_clean":"2024-09-16T01:36:05.373524+0000","last_became_active":"2024-09-16T01:36:05.367907+0000","last_became_peered":"2024-09-16T01:36:05.367907+0000","last_unstale":"2024-09-16T01:36:05.373524+0000","last_undegraded":"2024-09-16T01:36:05.373524+0000","last_fullsized":"2024-09-16T01:36:05.373524+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:40:04.647251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353341+0000","last_change":"2024-09-16T01:36:05.353341+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353341+0000","last_undegraded":"2024-09-16T01:36:05.353341+0000","last_fullsized":"2024-09-16T01:36:05.353341+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368806+0000","last_change":"2024-09-16T01:36:05.361916+0000","last_active":"2024-09-16T01:36:05.368806+0000","last_peered":"2024-09-16T01:36:05.368806+0000","last_clean":"2024-09-16T01:36:05.368806+0000","last_became_active":"2024-09-16T01:36:05.361658+0000","last_became_peered":"2024-09-16T01:36:05.361658+0000","last_unstale":"2024-09-16T01:36:05.368806+0000","last_undegraded":"2024-09-16T01:36:05.368806+0000","last_fullsized":"2024-09-16T01:36:05.368806+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:41.613259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356484+0000","last_change":"2024-09-16T01:36:05.356484+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356484+0000","last_undegraded":"2024-09-16T01:36:05.356484+0000","last_fullsized":"2024-09-16T01:36:05.356484+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373139+0000","last_change":"2024-09-16T01:36:05.365323+0000","last_active":"2024-09-16T01:36:05.373139+0000","last_peered":"2024-09-16T01:36:05.373139+0000","last_clean":"2024-09-16T01:36:05.373139+0000","last_became_active":"2024-09-16T01:36:05.364940+0000","last_became_peered":"2024-09-16T01:36:05.364940+0000","last_unstale":"2024-09-16T01:36:05.373139+0000","last_undegraded":"2024-09-16T01:36:05.373139+0000","last_fullsized":"2024-09-16T01:36:05.373139+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:38:11.910238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353237+0000","last_change":"2024-09-16T01:36:05.353237+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353237+0000","last_undegraded":"2024-09-16T01:36:05.353237+0000","last_fullsized":"2024-09-16T01:36:05.353237+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.357443+0000","last_change":"2024-09-16T01:36:05.359801+0000","last_active":"2024-09-16T01:36:06.357443+0000","last_peered":"2024-09-16T01:36:06.357443+0000","last_clean":"2024-09-16T01:36:06.357443+0000","last_became_active":"2024-09-16T01:36:05.359542+0000","last_became_peered":"2024-09-16T01:36:05.359542+0000","last_unstale":"2024-09-16T01:36:06.357443+0000","last_undegraded":"2024-09-16T01:36:06.357443+0000","last_fullsized":"2024-09-16T01:36:06.357443+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:22.128838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351989+0000","last_change":"2024-09-16T01:36:05.351989+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351989+0000","last_undegraded":"2024-09-16T01:36:05.351989+0000","last_fullsized":"2024-09-16T01:36:05.351989+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.366391+0000","last_change":"2024-09-16T01:36:05.364586+0000","last_active":"2024-09-16T01:36:05.366391+0000","last_peered":"2024-09-16T01:36:05.366391+0000","last_clean":"2024-09-16T01:36:05.366391+0000","last_became_active":"2024-09-16T01:36:05.364271+0000","last_became_peered":"2024-09-16T01:36:05.364271+0000","last_unstale":"2024-09-16T01:36:05.366391+0000","last_undegraded":"2024-09-16T01:36:05.366391+0000","last_fullsized":"2024-09-16T01:36:05.366391+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:15:05.347700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.350838+0000","last_change":"2024-09-16T01:36:05.350838+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.350838+0000","last_undegraded":"2024-09-16T01:36:05.350838+0000","last_fullsized":"2024-09-16T01:36:05.350838+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.370306+0000","last_change":"2024-09-16T01:36:05.364097+0000","last_active":"2024-09-16T01:36:05.370306+0000","last_peered":"2024-09-16T01:36:05.370306+0000","last_clean":"2024-09-16T01:36:05.370306+0000","last_became_active":"2024-09-16T01:36:05.363758+0000","last_became_peered":"2024-09-16T01:36:05.363758+0000","last_unstale":"2024-09-16T01:36:05.370306+0000","last_undegraded":"2024-09-16T01:36:05.370306+0000","last_fullsized":"2024-09-16T01:36:05.370306+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:59.822221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.359339+0000","last_change":"2024-09-16T01:36:05.359339+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.359339+0000","last_undegraded":"2024-09-16T01:36:05.359339+0000","last_fullsized":"2024-09-16T01:36:05.359339+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.365112+0000","last_change":"2024-09-16T01:36:05.363830+0000","last_active":"2024-09-16T01:36:05.365112+0000","last_peered":"2024-09-16T01:36:05.365112+0000","last_clean":"2024-09-16T01:36:05.365112+0000","last_became_active":"2024-09-16T01:36:05.363572+0000","last_became_peered":"2024-09-16T01:36:05.363572+0000","last_unstale":"2024-09-16T01:36:05.365112+0000","last_undegraded":"2024-09-16T01:36:05.365112+0000","last_fullsized":"2024-09-16T01:36:05.365112+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:53.565250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356478+0000","last_change":"2024-09-16T01:36:05.356478+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356478+0000","last_undegraded":"2024-09-16T01:36:05.356478+0000","last_fullsized":"2024-09-16T01:36:05.356478+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373551+0000","last_change":"2024-09-16T01:36:05.368744+0000","last_active":"2024-09-16T01:36:05.373551+0000","last_peered":"2024-09-16T01:36:05.373551+0000","last_clean":"2024-09-16T01:36:05.373551+0000","last_became_active":"2024-09-16T01:36:05.368069+0000","last_became_peered":"2024-09-16T01:36:05.368069+0000","last_unstale":"2024-09-16T01:36:05.373551+0000","last_undegraded":"2024-09-16T01:36:05.373551+0000","last_fullsized":"2024-09-16T01:36:05.373551+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:58.552340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.360574+0000","last_change":"2024-09-16T01:36:05.360574+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.360574+0000","last_undegraded":"2024-09-16T01:36:05.360574+0000","last_fullsized":"2024-09-16T01:36:05.360574+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368680+0000","last_change":"2024-09-16T01:36:05.365975+0000","last_active":"2024-09-16T01:36:05.368680+0000","last_peered":"2024-09-16T01:36:05.368680+0000","last_clean":"2024-09-16T01:36:05.368680+0000","last_became_active":"2024-09-16T01:36:05.365674+0000","last_became_peered":"2024-09-16T01:36:05.365674+0000","last_unstale":"2024-09-16T01:36:05.368680+0000","last_undegraded":"2024-09-16T01:36:05.368680+0000","last_fullsized":"2024-09-16T01:36:05.368680+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:21:23.387226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352330+0000","last_change":"2024-09-16T01:36:05.352330+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352330+0000","last_undegraded":"2024-09-16T01:36:05.352330+0000","last_fullsized":"2024-09-16T01:36:05.352330+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368728+0000","last_change":"2024-09-16T01:36:05.360882+0000","last_active":"2024-09-16T01:36:05.368728+0000","last_peered":"2024-09-16T01:36:05.368728+0000","last_clean":"2024-09-16T01:36:05.368728+0000","last_became_active":"2024-09-16T01:36:05.360521+0000","last_became_peered":"2024-09-16T01:36:05.360521+0000","last_unstale":"2024-09-16T01:36:05.368728+0000","last_undegraded":"2024-09-16T01:36:05.368728+0000","last_fullsized":"2024-09-16T01:36:05.368728+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:05:47.000282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.354770+0000","last_change":"2024-09-16T01:36:05.354770+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.354770+0000","last_undegraded":"2024-09-16T01:36:05.354770+0000","last_fullsized":"2024-09-16T01:36:05.354770+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373578+0000","last_change":"2024-09-16T01:36:05.369476+0000","last_active":"2024-09-16T01:36:05.373578+0000","last_peered":"2024-09-16T01:36:05.373578+0000","last_clean":"2024-09-16T01:36:05.373578+0000","last_became_active":"2024-09-16T01:36:05.369155+0000","last_became_peered":"2024-09-16T01:36:05.369155+0000","last_unstale":"2024-09-16T01:36:05.373578+0000","last_undegraded":"2024-09-16T01:36:05.373578+0000","last_fullsized":"2024-09-16T01:36:05.373578+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:22.128838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356076+0000","last_change":"2024-09-16T01:36:05.356076+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356076+0000","last_undegraded":"2024-09-16T01:36:05.356076+0000","last_fullsized":"2024-09-16T01:36:05.356076+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.366290+0000","last_change":"2024-09-16T01:36:05.359803+0000","last_active":"2024-09-16T01:36:05.366290+0000","last_peered":"2024-09-16T01:36:05.366290+0000","last_clean":"2024-09-16T01:36:05.366290+0000","last_became_active":"2024-09-16T01:36:05.359552+0000","last_became_peered":"2024-09-16T01:36:05.359552+0000","last_unstale":"2024-09-16T01:36:05.366290+0000","last_undegraded":"2024-09-16T01:36:05.366290+0000","last_fullsized":"2024-09-16T01:36:05.366290+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:32:35.415486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.357949+0000","last_change":"2024-09-16T01:36:05.357949+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.357949+0000","last_undegraded":"2024-09-16T01:36:05.357949+0000","last_fullsized":"2024-09-16T01:36:05.357949+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.362430+0000","last_change":"2024-09-16T01:36:05.359125+0000","last_active":"2024-09-16T01:36:05.362430+0000","last_peered":"2024-09-16T01:36:05.362430+0000","last_clean":"2024-09-16T01:36:05.362430+0000","last_became_active":"2024-09-16T01:36:05.358486+0000","last_became_peered":"2024-09-16T01:36:05.358486+0000","last_unstale":"2024-09-16T01:36:05.362430+0000","last_undegraded":"2024-09-16T01:36:05.362430+0000","last_fullsized":"2024-09-16T01:36:05.362430+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:22:44.509769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353180+0000","last_change":"2024-09-16T01:36:05.353180+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353180+0000","last_undegraded":"2024-09-16T01:36:05.353180+0000","last_fullsized":"2024-09-16T01:36:05.353180+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.364379+0000","last_change":"2024-09-16T01:36:05.361659+0000","last_active":"2024-09-16T01:36:05.364379+0000","last_peered":"2024-09-16T01:36:05.364379+0000","last_clean":"2024-09-16T01:36:05.364379+0000","last_became_active":"2024-09-16T01:36:05.361408+0000","last_became_peered":"2024-09-16T01:36:05.361408+0000","last_unstale":"2024-09-16T01:36:05.364379+0000","last_undegraded":"2024-09-16T01:36:05.364379+0000","last_fullsized":"2024-09-16T01:36:05.364379+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:30:10.270905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352248+0000","last_change":"2024-09-16T01:36:05.352248+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352248+0000","last_undegraded":"2024-09-16T01:36:05.352248+0000","last_fullsized":"2024-09-16T01:36:05.352248+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.365065+0000","last_change":"2024-09-16T01:36:05.363734+0000","last_active":"2024-09-16T01:36:05.365065+0000","last_peered":"2024-09-16T01:36:05.365065+0000","last_clean":"2024-09-16T01:36:05.365065+0000","last_became_active":"2024-09-16T01:36:05.363407+0000","last_became_peered":"2024-09-16T01:36:05.363407+0000","last_unstale":"2024-09-16T01:36:05.365065+0000","last_undegraded":"2024-09-16T01:36:05.365065+0000","last_fullsized":"2024-09-16T01:36:05.365065+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:56:30.024164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351047+0000","last_change":"2024-09-16T01:36:05.351047+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351047+0000","last_undegraded":"2024-09-16T01:36:05.351047+0000","last_fullsized":"2024-09-16T01:36:05.351047+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"44'1","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.374827+0000","last_change":"2024-09-16T01:36:05.360643+0000","last_active":"2024-09-16T01:36:06.374827+0000","last_peered":"2024-09-16T01:36:06.374827+0000","last_clean":"2024-09-16T01:36:06.374827+0000","last_became_active":"2024-09-16T01:36:05.360204+0000","last_became_peered":"2024-09-16T01:36:05.360204+0000","last_unstale":"2024-09-16T01:36:06.374827+0000","last_undegraded":"2024-09-16T01:36:06.374827+0000","last_fullsized":"2024-09-16T01:36:06.374827+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:54.497479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351796+0000","last_change":"2024-09-16T01:36:05.351796+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351796+0000","last_undegraded":"2024-09-16T01:36:05.351796+0000","last_fullsized":"2024-09-16T01:36:05.351796+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368891+0000","last_change":"2024-09-16T01:36:05.363498+0000","last_active":"2024-09-16T01:36:05.368891+0000","last_peered":"2024-09-16T01:36:05.368891+0000","last_clean":"2024-09-16T01:36:05.368891+0000","last_became_active":"2024-09-16T01:36:05.363185+0000","last_became_peered":"2024-09-16T01:36:05.363185+0000","last_unstale":"2024-09-16T01:36:05.368891+0000","last_undegraded":"2024-09-16T01:36:05.368891+0000","last_fullsized":"2024-09-16T01:36:05.368891+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:29:00.841831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.356095+0000","last_change":"2024-09-16T01:36:05.356095+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.356095+0000","last_undegraded":"2024-09-16T01:36:05.356095+0000","last_fullsized":"2024-09-16T01:36:05.356095+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368786+0000","last_change":"2024-09-16T01:36:05.361868+0000","last_active":"2024-09-16T01:36:05.368786+0000","last_peered":"2024-09-16T01:36:05.368786+0000","last_clean":"2024-09-16T01:36:05.368786+0000","last_became_active":"2024-09-16T01:36:05.361605+0000","last_became_peered":"2024-09-16T01:36:05.361605+0000","last_unstale":"2024-09-16T01:36:05.368786+0000","last_undegraded":"2024-09-16T01:36:05.368786+0000","last_fullsized":"2024-09-16T01:36:05.368786+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:52:52.798634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353015+0000","last_change":"2024-09-16T01:36:05.353015+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353015+0000","last_undegraded":"2024-09-16T01:36:05.353015+0000","last_fullsized":"2024-09-16T01:36:05.353015+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.357525+0000","last_change":"2024-09-16T01:36:05.356369+0000","last_active":"2024-09-16T01:36:06.357525+0000","last_peered":"2024-09-16T01:36:06.357525+0000","last_clean":"2024-09-16T01:36:06.357525+0000","last_became_active":"2024-09-16T01:36:05.356027+0000","last_became_peered":"2024-09-16T01:36:05.356027+0000","last_unstale":"2024-09-16T01:36:06.357525+0000","last_undegraded":"2024-09-16T01:36:06.357525+0000","last_fullsized":"2024-09-16T01:36:06.357525+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:02.446309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352450+0000","last_change":"2024-09-16T01:36:05.352450+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352450+0000","last_undegraded":"2024-09-16T01:36:05.352450+0000","last_fullsized":"2024-09-16T01:36:05.352450+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.364524+0000","last_change":"2024-09-16T01:36:05.363731+0000","last_active":"2024-09-16T01:36:05.364524+0000","last_peered":"2024-09-16T01:36:05.364524+0000","last_clean":"2024-09-16T01:36:05.364524+0000","last_became_active":"2024-09-16T01:36:05.363475+0000","last_became_peered":"2024-09-16T01:36:05.363475+0000","last_unstale":"2024-09-16T01:36:05.364524+0000","last_undegraded":"2024-09-16T01:36:05.364524+0000","last_fullsized":"2024-09-16T01:36:05.364524+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:28:42.397852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353294+0000","last_change":"2024-09-16T01:36:05.353294+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353294+0000","last_undegraded":"2024-09-16T01:36:05.353294+0000","last_fullsized":"2024-09-16T01:36:05.353294+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373466+0000","last_change":"2024-09-16T01:36:05.367836+0000","last_active":"2024-09-16T01:36:05.373466+0000","last_peered":"2024-09-16T01:36:05.373466+0000","last_clean":"2024-09-16T01:36:05.373466+0000","last_became_active":"2024-09-16T01:36:05.367513+0000","last_became_peered":"2024-09-16T01:36:05.367513+0000","last_unstale":"2024-09-16T01:36:05.373466+0000","last_undegraded":"2024-09-16T01:36:05.373466+0000","last_fullsized":"2024-09-16T01:36:05.373466+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:53:48.685089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.351224+0000","last_change":"2024-09-16T01:36:05.351224+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.351224+0000","last_undegraded":"2024-09-16T01:36:05.351224+0000","last_fullsized":"2024-09-16T01:36:05.351224+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368934+0000","last_change":"2024-09-16T01:36:05.365994+0000","last_active":"2024-09-16T01:36:05.368934+0000","last_peered":"2024-09-16T01:36:05.368934+0000","last_clean":"2024-09-16T01:36:05.368934+0000","last_became_active":"2024-09-16T01:36:05.365629+0000","last_became_peered":"2024-09-16T01:36:05.365629+0000","last_unstale":"2024-09-16T01:36:05.368934+0000","last_undegraded":"2024-09-16T01:36:05.368934+0000","last_fullsized":"2024-09-16T01:36:05.368934+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:44:58.819992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.352527+0000","last_change":"2024-09-16T01:36:05.352527+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.352527+0000","last_undegraded":"2024-09-16T01:36:05.352527+0000","last_fullsized":"2024-09-16T01:36:05.352527+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373583+0000","last_change":"2024-09-16T01:36:05.370944+0000","last_active":"2024-09-16T01:36:05.373583+0000","last_peered":"2024-09-16T01:36:05.373583+0000","last_clean":"2024-09-16T01:36:05.373583+0000","last_became_active":"2024-09-16T01:36:05.370631+0000","last_became_peered":"2024-09-16T01:36:05.370631+0000","last_unstale":"2024-09-16T01:36:05.373583+0000","last_undegraded":"2024-09-16T01:36:05.373583+0000","last_fullsized":"2024-09-16T01:36:05.373583+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:02.446309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.359667+0000","last_change":"2024-09-16T01:36:05.359667+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.359667+0000","last_undegraded":"2024-09-16T01:36:05.359667+0000","last_fullsized":"2024-09-16T01:36:05.359667+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.368831+0000","last_change":"2024-09-16T01:36:05.365868+0000","last_active":"2024-09-16T01:36:05.368831+0000","last_peered":"2024-09-16T01:36:05.368831+0000","last_clean":"2024-09-16T01:36:05.368831+0000","last_became_active":"2024-09-16T01:36:05.365481+0000","last_became_peered":"2024-09-16T01:36:05.365481+0000","last_unstale":"2024-09-16T01:36:05.368831+0000","last_undegraded":"2024-09-16T01:36:05.368831+0000","last_fullsized":"2024-09-16T01:36:05.368831+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:26:50.407545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.354612+0000","last_change":"2024-09-16T01:36:05.354612+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.354612+0000","last_undegraded":"2024-09-16T01:36:05.354612+0000","last_fullsized":"2024-09-16T01:36:05.354612+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.373531+0000","last_change":"2024-09-16T01:36:05.368272+0000","last_active":"2024-09-16T01:36:05.373531+0000","last_peered":"2024-09-16T01:36:05.373531+0000","last_clean":"2024-09-16T01:36:05.373531+0000","last_became_active":"2024-09-16T01:36:05.367973+0000","last_became_peered":"2024-09-16T01:36:05.367973+0000","last_unstale":"2024-09-16T01:36:05.373531+0000","last_undegraded":"2024-09-16T01:36:05.373531+0000","last_fullsized":"2024-09-16T01:36:05.373531+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:09:24.703051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.355177+0000","last_change":"2024-09-16T01:36:05.355177+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.355177+0000","last_undegraded":"2024-09-16T01:36:05.355177+0000","last_fullsized":"2024-09-16T01:36:05.355177+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.365212+0000","last_change":"2024-09-16T01:36:05.362224+0000","last_active":"2024-09-16T01:36:05.365212+0000","last_peered":"2024-09-16T01:36:05.365212+0000","last_clean":"2024-09-16T01:36:05.365212+0000","last_became_active":"2024-09-16T01:36:05.361853+0000","last_became_peered":"2024-09-16T01:36:05.361853+0000","last_unstale":"2024-09-16T01:36:05.365212+0000","last_undegraded":"2024-09-16T01:36:05.365212+0000","last_fullsized":"2024-09-16T01:36:05.365212+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:15:05.347700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.369963+0000","last_change":"2024-09-16T01:36:06.363427+0000","last_active":"2024-09-16T01:36:06.369963+0000","last_peered":"2024-09-16T01:36:06.369963+0000","last_clean":"2024-09-16T01:36:06.369963+0000","last_became_active":"2024-09-16T01:36:06.363142+0000","last_became_peered":"2024-09-16T01:36:06.363142+0000","last_unstale":"2024-09-16T01:36:06.369963+0000","last_undegraded":"2024-09-16T01:36:06.369963+0000","last_fullsized":"2024-09-16T01:36:06.369963+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:22.762911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.366175+0000","last_change":"2024-09-16T01:36:05.362621+0000","last_active":"2024-09-16T01:36:05.366175+0000","last_peered":"2024-09-16T01:36:05.366175+0000","last_clean":"2024-09-16T01:36:05.366175+0000","last_became_active":"2024-09-16T01:36:05.362300+0000","last_became_peered":"2024-09-16T01:36:05.362300+0000","last_unstale":"2024-09-16T01:36:05.366175+0000","last_undegraded":"2024-09-16T01:36:05.366175+0000","last_fullsized":"2024-09-16T01:36:05.366175+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:31:24.708078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.350199+0000","last_change":"2024-09-16T01:36:05.350199+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.350199+0000","last_undegraded":"2024-09-16T01:36:05.350199+0000","last_fullsized":"2024-09-16T01:36:05.350199+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.363008+0000","last_change":"2024-09-16T01:36:05.362040+0000","last_active":"2024-09-16T01:36:05.363008+0000","last_peered":"2024-09-16T01:36:05.363008+0000","last_clean":"2024-09-16T01:36:05.363008+0000","last_became_active":"2024-09-16T01:36:05.361767+0000","last_became_peered":"2024-09-16T01:36:05.361767+0000","last_unstale":"2024-09-16T01:36:05.363008+0000","last_undegraded":"2024-09-16T01:36:05.363008+0000","last_fullsized":"2024-09-16T01:36:05.363008+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:49:52.496809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-16T01:36:05.353674+0000","last_change":"2024-09-16T01:36:05.353674+0000","last_active":"2024-09-16T01:36:05.342630+0000","last_peered":"2024-09-16T01:36:05.342630+0000","last_clean":"2024-09-16T01:36:05.342630+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:36:05.353674+0000","last_undegraded":"2024-09-16T01:36:05.353674+0000","last_fullsized":"2024-09-16T01:36:05.353674+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"44'2","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T01:36:06.377763+0000","last_change":"2024-09-16T01:36:05.361909+0000","last_active":"2024-09-16T01:36:06.377763+0000","last_peered":"2024-09-16T01:36:06.377763+0000","last_clean":"2024-09-16T01:36:06.377763+0000","last_became_active":"2024-09-16T01:36:05.361632+0000","last_became_peered":"2024-09-16T01:36:05.361632+0000","last_unstale":"2024-09-16T01:36:06.377763+0000","last_undegraded":"2024-09-16T01:36:06.377763+0000","last_fullsized":"2024-09-16T01:36:06.377763+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:55:37.303541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":209,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T01:36:05.361317+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:36:05.361317+0000","last_peered":"2024-09-16T01:36:05.361317+0000","last_clean":"2024-09-16T01:36:05.361317+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:36:05.361317+0000","last_undegraded":"2024-09-16T01:36:05.361317+0000","last_fullsized":"2024-09-16T01:36:05.361317+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":70,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509468,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27520,"kb_used_data":488,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344320,"statfs":{"total":96636764160,"available":96608583680,"internally_reserved":0,"allocated":499712,"data_stored":199112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542171,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28088,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343752,"statfs":{"total":96636764160,"available":96608002048,"internally_reserved":0,"allocated":1081344,"data_stored":783213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542171,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":487424,"data_stored":192845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542171,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":487424,"data_stored":192845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542170,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27512,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344328,"statfs":{"total":96636764160,"available":96608591872,"internally_reserved":0,"allocated":487424,"data_stored":192845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542171,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27508,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344332,"statfs":{"total":96636764160,"available":96608595968,"internally_reserved":0,"allocated":487424,"data_stored":192845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542170,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27504,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344336,"statfs":{"total":96636764160,"available":96608600064,"internally_reserved":0,"allocated":487424,"data_stored":192845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542170,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28092,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343748,"statfs":{"total":96636764160,"available":96607997952,"internally_reserved":0,"allocated":1081344,"data_stored":783213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:36:12.345 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:36:12.345+0000 7efdf135a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:36:13.652 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:36:13.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.763+0000 7f4002b11640 1 Processor -- start 2024-09-16T01:36:13.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.763+0000 7f4002b11640 1 -- start start 2024-09-16T01:36:13.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.764+0000 7f4002b11640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3ffc0b63d0 0x7f3ffc0d39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:13.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.764+0000 7f4002b11640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3ffc0cf6b0 0x7f3ffc0b7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:13.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.764+0000 7f4002b11640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ffc0cfc20 con 0x7f3ffc0d1c80 2024-09-16T01:36:13.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.764+0000 7f4002b11640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f3ffc0d2ed0 con 0x7f3ffc0b63d0 2024-09-16T01:36:13.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.764+0000 7f4002b11640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3ffc0d31b0 con 0x7f3ffc0cf6b0 2024-09-16T01:36:13.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.764+0000 7f40008ad640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3ffc0b63d0 0x7f3ffc0d39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:13.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.764+0000 7f40008ad640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3ffc0b63d0 0x7f3ffc0d39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48670/0 (socket says 172.21.15.88:48670) 2024-09-16T01:36:13.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.764+0000 7f40008ad640 1 -- 172.21.15.88:0/2149901758 learned_addr learned my addr 172.21.15.88:0/2149901758 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:13.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.765+0000 7f3ffbfff640 1 --2- 172.21.15.88:0/2149901758 >> v2:172.21.15.154:3300/0 conn(0x7f3ffc0cf6b0 0x7f3ffc0b7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:13.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.765+0000 7f40008ad640 1 -- 172.21.15.88:0/2149901758 >> v2:172.21.15.154:3300/0 conn(0x7f3ffc0cf6b0 msgr2=0x7f3ffc0b7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:13.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.765+0000 7f40008ad640 1 --2- 172.21.15.88:0/2149901758 >> v2:172.21.15.154:3300/0 conn(0x7f3ffc0cf6b0 0x7f3ffc0b7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:13.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.765+0000 7f40008ad640 1 -- 172.21.15.88:0/2149901758 >> v1:172.21.15.154:6789/0 conn(0x7f3ffc0d1c80 legacy=0x7f3ffc0b7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:13.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.765+0000 7f40008ad640 1 -- 172.21.15.88:0/2149901758 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ffc0d3490 con 0x7f3ffc0b63d0 2024-09-16T01:36:13.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.765+0000 7f40008ad640 1 --2- 172.21.15.88:0/2149901758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ffc0b63d0 0x7f3ffc0d39b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc0b40e0 tx=0x7f3fec02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ae5ac26cd7e22b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:13.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.766+0000 7f3ffb7fe640 1 -- 172.21.15.88:0/2149901758 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fec037070 con 0x7f3ffc0b63d0 2024-09-16T01:36:13.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.766+0000 7f3ffb7fe640 1 -- 172.21.15.88:0/2149901758 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fec032440 con 0x7f3ffc0b63d0 2024-09-16T01:36:13.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.766+0000 7f3ffb7fe640 1 -- 172.21.15.88:0/2149901758 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fec032810 con 0x7f3ffc0b63d0 2024-09-16T01:36:13.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.766+0000 7f4002b11640 1 -- 172.21.15.88:0/2149901758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ffc0b63d0 msgr2=0x7f3ffc0d39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:13.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.766+0000 7f4002b11640 1 --2- 172.21.15.88:0/2149901758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ffc0b63d0 0x7f3ffc0d39b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc0b40e0 tx=0x7f3fec02a1a0 comp rx=0 tx=0).stop 2024-09-16T01:36:13.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.766+0000 7f4002b11640 1 -- 172.21.15.88:0/2149901758 shutdown_connections 2024-09-16T01:36:13.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.766+0000 7f4002b11640 1 --2- 172.21.15.88:0/2149901758 >> v2:172.21.15.154:3300/0 conn(0x7f3ffc0cf6b0 0x7f3ffc0b7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:13.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.766+0000 7f4002b11640 1 --2- 172.21.15.88:0/2149901758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ffc0b63d0 0x7f3ffc0d39b0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:13.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.766+0000 7f4002b11640 1 -- 172.21.15.88:0/2149901758 >> 172.21.15.88:0/2149901758 conn(0x7f3ffc0d1160 msgr2=0x7f3ffc0d1570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:13.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.766+0000 7f4002b11640 1 -- 172.21.15.88:0/2149901758 shutdown_connections 2024-09-16T01:36:13.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.766+0000 7f4002b11640 1 -- 172.21.15.88:0/2149901758 wait complete. 2024-09-16T01:36:13.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.767+0000 7f4002b11640 1 Processor -- start 2024-09-16T01:36:13.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.768+0000 7f4002b11640 1 -- start start 2024-09-16T01:36:13.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.768+0000 7f4002b11640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ffc0b63d0 0x7f3ffc1f77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:13.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.768+0000 7f4002b11640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ffc0cf6b0 0x7f3ffc0b98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:13.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.768+0000 7f4002b11640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ffc0d1c80 0x7f3ffc0b9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:13.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.768+0000 7f4002b11640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3ffc0d4070 con 0x7f3ffc0cf6b0 2024-09-16T01:36:13.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.768+0000 7f4002b11640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3ffc0d3ef0 con 0x7f3ffc0d1c80 2024-09-16T01:36:13.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.768+0000 7f4002b11640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3ffc0b67b0 con 0x7f3ffc0b63d0 2024-09-16T01:36:13.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.769+0000 7f3ffbfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ffc0cf6b0 0x7f3ffc0b98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:13.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.769+0000 7f3ffbfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ffc0cf6b0 0x7f3ffc0b98c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48678/0 (socket says 172.21.15.88:48678) 2024-09-16T01:36:13.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.769+0000 7f3ffbfff640 1 -- 172.21.15.88:0/589360541 learned_addr learned my addr 172.21.15.88:0/589360541 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:13.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.769+0000 7f40008ad640 1 --2- 172.21.15.88:0/589360541 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ffc0b63d0 0x7f3ffc1f77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:13.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.769+0000 7f40010ae640 1 --2- 172.21.15.88:0/589360541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ffc0d1c80 0x7f3ffc0b9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:13.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.769+0000 7f3ffbfff640 1 -- 172.21.15.88:0/589360541 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ffc0b63d0 msgr2=0x7f3ffc1f77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:13.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.769+0000 7f3ffbfff640 1 --2- 172.21.15.88:0/589360541 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ffc0b63d0 0x7f3ffc1f77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:13.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.769+0000 7f3ffbfff640 1 -- 172.21.15.88:0/589360541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ffc0d1c80 msgr2=0x7f3ffc0b9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:13.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.769+0000 7f3ffbfff640 1 --2- 172.21.15.88:0/589360541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ffc0d1c80 0x7f3ffc0b9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:13.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.769+0000 7f3ffbfff640 1 -- 172.21.15.88:0/589360541 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ffc1e82e0 con 0x7f3ffc0cf6b0 2024-09-16T01:36:13.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.769+0000 7f3ffbfff640 1 --2- 172.21.15.88:0/589360541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ffc0cf6b0 0x7f3ffc0b98c0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f3ff000d800 tx=0x7f3ff000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:13.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.769+0000 7f3ff97fa640 1 -- 172.21.15.88:0/589360541 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff000b6c0 con 0x7f3ffc0cf6b0 2024-09-16T01:36:13.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.769+0000 7f3ff97fa640 1 -- 172.21.15.88:0/589360541 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ff0019070 con 0x7f3ffc0cf6b0 2024-09-16T01:36:13.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.769+0000 7f4002b11640 1 -- 172.21.15.88:0/589360541 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ffc1e8580 con 0x7f3ffc0cf6b0 2024-09-16T01:36:13.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.769+0000 7f3ff97fa640 1 -- 172.21.15.88:0/589360541 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff000ba80 con 0x7f3ffc0cf6b0 2024-09-16T01:36:13.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.769+0000 7f4002b11640 1 -- 172.21.15.88:0/589360541 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ffc1e8b40 con 0x7f3ffc0cf6b0 2024-09-16T01:36:13.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.770+0000 7f4002b11640 1 -- 172.21.15.88:0/589360541 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ffc0d39b0 con 0x7f3ffc0cf6b0 2024-09-16T01:36:13.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.771+0000 7f3ff97fa640 1 -- 172.21.15.88:0/589360541 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3ff000bc20 con 0x7f3ffc0cf6b0 2024-09-16T01:36:13.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.771+0000 7f3ff97fa640 1 --2- 172.21.15.88:0/589360541 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3fe806c2b0 0x7f3fe806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:13.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.771+0000 7f3ff97fa640 1 -- 172.21.15.88:0/589360541 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f3ff008b8c0 con 0x7f3ffc0cf6b0 2024-09-16T01:36:13.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.771+0000 7f40008ad640 1 --2- 172.21.15.88:0/589360541 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3fe806c2b0 0x7f3fe806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:13.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.772+0000 7f40008ad640 1 --2- 172.21.15.88:0/589360541 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3fe806c2b0 0x7f3fe806e770 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3fec034000 tx=0x7f3fec02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:13.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.776+0000 7f3ff97fa640 1 -- 172.21.15.88:0/589360541 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3ff005a030 con 0x7f3ffc0cf6b0 2024-09-16T01:36:13.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.938+0000 7f4002b11640 1 -- 172.21.15.88:0/589360541 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3ffc0cf280 con 0x7f3fe806c2b0 2024-09-16T01:36:13.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.943+0000 7f3ff97fa640 1 -- 172.21.15.88:0/589360541 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251947 (secure 0 0 0) 0x7f3fec07f840 con 0x7f3fe806c2b0 2024-09-16T01:36:13.947 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:13.949 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:36:13.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.946+0000 7f4002b11640 1 -- 172.21.15.88:0/589360541 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3fe806c2b0 msgr2=0x7f3fe806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:13.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.946+0000 7f4002b11640 1 --2- 172.21.15.88:0/589360541 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3fe806c2b0 0x7f3fe806e770 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3fec034000 tx=0x7f3fec02e000 comp rx=0 tx=0).stop 2024-09-16T01:36:13.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.946+0000 7f4002b11640 1 -- 172.21.15.88:0/589360541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ffc0cf6b0 msgr2=0x7f3ffc0b98c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:13.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.946+0000 7f4002b11640 1 --2- 172.21.15.88:0/589360541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ffc0cf6b0 0x7f3ffc0b98c0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f3ff000d800 tx=0x7f3ff000dcc0 comp rx=0 tx=0).stop 2024-09-16T01:36:13.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.946+0000 7f4002b11640 1 -- 172.21.15.88:0/589360541 shutdown_connections 2024-09-16T01:36:13.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.946+0000 7f4002b11640 1 --2- 172.21.15.88:0/589360541 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3fe806c2b0 0x7f3fe806e770 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:13.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.946+0000 7f4002b11640 1 --2- 172.21.15.88:0/589360541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ffc0d1c80 0x7f3ffc0b9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:13.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.946+0000 7f4002b11640 1 --2- 172.21.15.88:0/589360541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ffc0cf6b0 0x7f3ffc0b98c0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:13.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.946+0000 7f4002b11640 1 --2- 172.21.15.88:0/589360541 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ffc0b63d0 0x7f3ffc1f77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:13.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.946+0000 7f4002b11640 1 -- 172.21.15.88:0/589360541 >> 172.21.15.88:0/589360541 conn(0x7f3ffc0d1160 msgr2=0x7f3ffc162a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:13.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.946+0000 7f4002b11640 1 -- 172.21.15.88:0/589360541 shutdown_connections 2024-09-16T01:36:13.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:13.946+0000 7f4002b11640 1 -- 172.21.15.88:0/589360541 wait complete. 2024-09-16T01:36:13.959 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-09-16T01:36:12.539003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221584,"kb_used_data":5320,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753136,"statfs":{"total":773094113280,"available":772867211264,"internally_reserved":0,"allocated":5447680,"data_stored":2883052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.191112"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.353335+0000","last_change":"2024-09-16T01:36:06.373417+0000","last_active":"2024-09-16T01:36:07.353335+0000","last_peered":"2024-09-16T01:36:07.353335+0000","last_clean":"2024-09-16T01:36:07.353335+0000","last_became_active":"2024-09-16T01:36:06.372711+0000","last_became_peered":"2024-09-16T01:36:06.372711+0000","last_unstale":"2024-09-16T01:36:07.353335+0000","last_undegraded":"2024-09-16T01:36:07.353335+0000","last_fullsized":"2024-09-16T01:36:07.353335+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:07:44.372170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725530+0000","last_change":"2024-09-16T01:36:06.366967+0000","last_active":"2024-09-16T01:36:07.725530+0000","last_peered":"2024-09-16T01:36:07.725530+0000","last_clean":"2024-09-16T01:36:07.725530+0000","last_became_active":"2024-09-16T01:36:06.366688+0000","last_became_peered":"2024-09-16T01:36:06.366688+0000","last_unstale":"2024-09-16T01:36:07.725530+0000","last_undegraded":"2024-09-16T01:36:07.725530+0000","last_fullsized":"2024-09-16T01:36:07.725530+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:34:57.959052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725100+0000","last_change":"2024-09-16T01:36:06.362381+0000","last_active":"2024-09-16T01:36:07.725100+0000","last_peered":"2024-09-16T01:36:07.725100+0000","last_clean":"2024-09-16T01:36:07.725100+0000","last_became_active":"2024-09-16T01:36:06.362110+0000","last_became_peered":"2024-09-16T01:36:06.362110+0000","last_unstale":"2024-09-16T01:36:07.725100+0000","last_undegraded":"2024-09-16T01:36:07.725100+0000","last_fullsized":"2024-09-16T01:36:07.725100+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:44:59.821905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.352021+0000","last_change":"2024-09-16T01:36:06.359820+0000","last_active":"2024-09-16T01:36:07.352021+0000","last_peered":"2024-09-16T01:36:07.352021+0000","last_clean":"2024-09-16T01:36:07.352021+0000","last_became_active":"2024-09-16T01:36:06.359551+0000","last_became_peered":"2024-09-16T01:36:06.359551+0000","last_unstale":"2024-09-16T01:36:07.352021+0000","last_undegraded":"2024-09-16T01:36:07.352021+0000","last_fullsized":"2024-09-16T01:36:07.352021+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:33:13.629797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725121+0000","last_change":"2024-09-16T01:36:06.369570+0000","last_active":"2024-09-16T01:36:07.725121+0000","last_peered":"2024-09-16T01:36:07.725121+0000","last_clean":"2024-09-16T01:36:07.725121+0000","last_became_active":"2024-09-16T01:36:06.369276+0000","last_became_peered":"2024-09-16T01:36:06.369276+0000","last_unstale":"2024-09-16T01:36:07.725121+0000","last_undegraded":"2024-09-16T01:36:07.725121+0000","last_fullsized":"2024-09-16T01:36:07.725121+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:58:02.976688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725504+0000","last_change":"2024-09-16T01:36:06.371408+0000","last_active":"2024-09-16T01:36:07.725504+0000","last_peered":"2024-09-16T01:36:07.725504+0000","last_clean":"2024-09-16T01:36:07.725504+0000","last_became_active":"2024-09-16T01:36:06.371148+0000","last_became_peered":"2024-09-16T01:36:06.371148+0000","last_unstale":"2024-09-16T01:36:07.725504+0000","last_undegraded":"2024-09-16T01:36:07.725504+0000","last_fullsized":"2024-09-16T01:36:07.725504+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:50:25.840032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.352867+0000","last_change":"2024-09-16T01:36:06.380097+0000","last_active":"2024-09-16T01:36:07.352867+0000","last_peered":"2024-09-16T01:36:07.352867+0000","last_clean":"2024-09-16T01:36:07.352867+0000","last_became_active":"2024-09-16T01:36:06.379761+0000","last_became_peered":"2024-09-16T01:36:06.379761+0000","last_unstale":"2024-09-16T01:36:07.352867+0000","last_undegraded":"2024-09-16T01:36:07.352867+0000","last_fullsized":"2024-09-16T01:36:07.352867+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:34:57.959052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.351560+0000","last_change":"2024-09-16T01:36:06.365164+0000","last_active":"2024-09-16T01:36:07.351560+0000","last_peered":"2024-09-16T01:36:07.351560+0000","last_clean":"2024-09-16T01:36:07.351560+0000","last_became_active":"2024-09-16T01:36:06.364912+0000","last_became_peered":"2024-09-16T01:36:06.364912+0000","last_unstale":"2024-09-16T01:36:07.351560+0000","last_undegraded":"2024-09-16T01:36:07.351560+0000","last_fullsized":"2024-09-16T01:36:07.351560+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:08:44.163318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.351987+0000","last_change":"2024-09-16T01:36:06.364708+0000","last_active":"2024-09-16T01:36:07.351987+0000","last_peered":"2024-09-16T01:36:07.351987+0000","last_clean":"2024-09-16T01:36:07.351987+0000","last_became_active":"2024-09-16T01:36:06.364425+0000","last_became_peered":"2024-09-16T01:36:06.364425+0000","last_unstale":"2024-09-16T01:36:07.351987+0000","last_undegraded":"2024-09-16T01:36:07.351987+0000","last_fullsized":"2024-09-16T01:36:07.351987+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:12:18.089962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.352147+0000","last_change":"2024-09-16T01:36:06.373263+0000","last_active":"2024-09-16T01:36:07.352147+0000","last_peered":"2024-09-16T01:36:07.352147+0000","last_clean":"2024-09-16T01:36:07.352147+0000","last_became_active":"2024-09-16T01:36:06.365961+0000","last_became_peered":"2024-09-16T01:36:06.365961+0000","last_unstale":"2024-09-16T01:36:07.352147+0000","last_undegraded":"2024-09-16T01:36:07.352147+0000","last_fullsized":"2024-09-16T01:36:07.352147+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:26:09.111280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.351862+0000","last_change":"2024-09-16T01:36:06.378937+0000","last_active":"2024-09-16T01:36:07.351862+0000","last_peered":"2024-09-16T01:36:07.351862+0000","last_clean":"2024-09-16T01:36:07.351862+0000","last_became_active":"2024-09-16T01:36:06.378656+0000","last_became_peered":"2024-09-16T01:36:06.378656+0000","last_unstale":"2024-09-16T01:36:07.351862+0000","last_undegraded":"2024-09-16T01:36:07.351862+0000","last_fullsized":"2024-09-16T01:36:07.351862+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:33:24.405285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725200+0000","last_change":"2024-09-16T01:36:06.374437+0000","last_active":"2024-09-16T01:36:07.725200+0000","last_peered":"2024-09-16T01:36:07.725200+0000","last_clean":"2024-09-16T01:36:07.725200+0000","last_became_active":"2024-09-16T01:36:06.374156+0000","last_became_peered":"2024-09-16T01:36:06.374156+0000","last_unstale":"2024-09-16T01:36:07.725200+0000","last_undegraded":"2024-09-16T01:36:07.725200+0000","last_fullsized":"2024-09-16T01:36:07.725200+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:33:29.649570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725141+0000","last_change":"2024-09-16T01:36:06.379721+0000","last_active":"2024-09-16T01:36:07.725141+0000","last_peered":"2024-09-16T01:36:07.725141+0000","last_clean":"2024-09-16T01:36:07.725141+0000","last_became_active":"2024-09-16T01:36:06.379293+0000","last_became_peered":"2024-09-16T01:36:06.379293+0000","last_unstale":"2024-09-16T01:36:07.725141+0000","last_undegraded":"2024-09-16T01:36:07.725141+0000","last_fullsized":"2024-09-16T01:36:07.725141+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:03:19.840230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.351778+0000","last_change":"2024-09-16T01:36:06.364397+0000","last_active":"2024-09-16T01:36:07.351778+0000","last_peered":"2024-09-16T01:36:07.351778+0000","last_clean":"2024-09-16T01:36:07.351778+0000","last_became_active":"2024-09-16T01:36:06.363992+0000","last_became_peered":"2024-09-16T01:36:06.363992+0000","last_unstale":"2024-09-16T01:36:07.351778+0000","last_undegraded":"2024-09-16T01:36:07.351778+0000","last_fullsized":"2024-09-16T01:36:07.351778+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:04:00.844684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725558+0000","last_change":"2024-09-16T01:36:06.372795+0000","last_active":"2024-09-16T01:36:07.725558+0000","last_peered":"2024-09-16T01:36:07.725558+0000","last_clean":"2024-09-16T01:36:07.725558+0000","last_became_active":"2024-09-16T01:36:06.372527+0000","last_became_peered":"2024-09-16T01:36:06.372527+0000","last_unstale":"2024-09-16T01:36:07.725558+0000","last_undegraded":"2024-09-16T01:36:07.725558+0000","last_fullsized":"2024-09-16T01:36:07.725558+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:41:40.403487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.353167+0000","last_change":"2024-09-16T01:36:06.368044+0000","last_active":"2024-09-16T01:36:07.353167+0000","last_peered":"2024-09-16T01:36:07.353167+0000","last_clean":"2024-09-16T01:36:07.353167+0000","last_became_active":"2024-09-16T01:36:06.367811+0000","last_became_peered":"2024-09-16T01:36:06.367811+0000","last_unstale":"2024-09-16T01:36:07.353167+0000","last_undegraded":"2024-09-16T01:36:07.353167+0000","last_fullsized":"2024-09-16T01:36:07.353167+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:39.734019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725172+0000","last_change":"2024-09-16T01:36:06.361689+0000","last_active":"2024-09-16T01:36:07.725172+0000","last_peered":"2024-09-16T01:36:07.725172+0000","last_clean":"2024-09-16T01:36:07.725172+0000","last_became_active":"2024-09-16T01:36:06.361181+0000","last_became_peered":"2024-09-16T01:36:06.361181+0000","last_unstale":"2024-09-16T01:36:07.725172+0000","last_undegraded":"2024-09-16T01:36:07.725172+0000","last_fullsized":"2024-09-16T01:36:07.725172+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:09:19.116540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725413+0000","last_change":"2024-09-16T01:36:06.367112+0000","last_active":"2024-09-16T01:36:07.725413+0000","last_peered":"2024-09-16T01:36:07.725413+0000","last_clean":"2024-09-16T01:36:07.725413+0000","last_became_active":"2024-09-16T01:36:06.366569+0000","last_became_peered":"2024-09-16T01:36:06.366569+0000","last_unstale":"2024-09-16T01:36:07.725413+0000","last_undegraded":"2024-09-16T01:36:07.725413+0000","last_fullsized":"2024-09-16T01:36:07.725413+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:46:44.236233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.354227+0000","last_change":"2024-09-16T01:36:06.380702+0000","last_active":"2024-09-16T01:36:07.354227+0000","last_peered":"2024-09-16T01:36:07.354227+0000","last_clean":"2024-09-16T01:36:07.354227+0000","last_became_active":"2024-09-16T01:36:06.380456+0000","last_became_peered":"2024-09-16T01:36:06.380456+0000","last_unstale":"2024-09-16T01:36:07.354227+0000","last_undegraded":"2024-09-16T01:36:07.354227+0000","last_fullsized":"2024-09-16T01:36:07.354227+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:52:53.800547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.724861+0000","last_change":"2024-09-16T01:36:06.374209+0000","last_active":"2024-09-16T01:36:07.724861+0000","last_peered":"2024-09-16T01:36:07.724861+0000","last_clean":"2024-09-16T01:36:07.724861+0000","last_became_active":"2024-09-16T01:36:06.373951+0000","last_became_peered":"2024-09-16T01:36:06.373951+0000","last_unstale":"2024-09-16T01:36:07.724861+0000","last_undegraded":"2024-09-16T01:36:07.724861+0000","last_fullsized":"2024-09-16T01:36:07.724861+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:06:25.082410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725108+0000","last_change":"2024-09-16T01:36:06.370902+0000","last_active":"2024-09-16T01:36:07.725108+0000","last_peered":"2024-09-16T01:36:07.725108+0000","last_clean":"2024-09-16T01:36:07.725108+0000","last_became_active":"2024-09-16T01:36:06.370627+0000","last_became_peered":"2024-09-16T01:36:06.370627+0000","last_unstale":"2024-09-16T01:36:07.725108+0000","last_undegraded":"2024-09-16T01:36:07.725108+0000","last_fullsized":"2024-09-16T01:36:07.725108+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:37:14.997459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.351731+0000","last_change":"2024-09-16T01:36:06.369634+0000","last_active":"2024-09-16T01:36:07.351731+0000","last_peered":"2024-09-16T01:36:07.351731+0000","last_clean":"2024-09-16T01:36:07.351731+0000","last_became_active":"2024-09-16T01:36:06.369360+0000","last_became_peered":"2024-09-16T01:36:06.369360+0000","last_unstale":"2024-09-16T01:36:07.351731+0000","last_undegraded":"2024-09-16T01:36:07.351731+0000","last_fullsized":"2024-09-16T01:36:07.351731+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:46:44.236233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.353907+0000","last_change":"2024-09-16T01:36:06.370492+0000","last_active":"2024-09-16T01:36:07.353907+0000","last_peered":"2024-09-16T01:36:07.353907+0000","last_clean":"2024-09-16T01:36:07.353907+0000","last_became_active":"2024-09-16T01:36:06.370202+0000","last_became_peered":"2024-09-16T01:36:06.370202+0000","last_unstale":"2024-09-16T01:36:07.353907+0000","last_undegraded":"2024-09-16T01:36:07.353907+0000","last_fullsized":"2024-09-16T01:36:07.353907+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:00:49.423918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.353808+0000","last_change":"2024-09-16T01:36:06.372638+0000","last_active":"2024-09-16T01:36:07.353808+0000","last_peered":"2024-09-16T01:36:07.353808+0000","last_clean":"2024-09-16T01:36:07.353808+0000","last_became_active":"2024-09-16T01:36:06.372320+0000","last_became_peered":"2024-09-16T01:36:06.372320+0000","last_unstale":"2024-09-16T01:36:07.353808+0000","last_undegraded":"2024-09-16T01:36:07.353808+0000","last_fullsized":"2024-09-16T01:36:07.353808+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:31:32.278177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.354129+0000","last_change":"2024-09-16T01:36:06.364252+0000","last_active":"2024-09-16T01:36:07.354129+0000","last_peered":"2024-09-16T01:36:07.354129+0000","last_clean":"2024-09-16T01:36:07.354129+0000","last_became_active":"2024-09-16T01:36:06.363965+0000","last_became_peered":"2024-09-16T01:36:06.363965+0000","last_unstale":"2024-09-16T01:36:07.354129+0000","last_undegraded":"2024-09-16T01:36:07.354129+0000","last_fullsized":"2024-09-16T01:36:07.354129+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:32:36.417399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.724651+0000","last_change":"2024-09-16T01:36:06.369699+0000","last_active":"2024-09-16T01:36:07.724651+0000","last_peered":"2024-09-16T01:36:07.724651+0000","last_clean":"2024-09-16T01:36:07.724651+0000","last_became_active":"2024-09-16T01:36:06.369453+0000","last_became_peered":"2024-09-16T01:36:06.369453+0000","last_unstale":"2024-09-16T01:36:07.724651+0000","last_undegraded":"2024-09-16T01:36:07.724651+0000","last_fullsized":"2024-09-16T01:36:07.724651+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:08:51.755748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.724876+0000","last_change":"2024-09-16T01:36:06.365305+0000","last_active":"2024-09-16T01:36:07.724876+0000","last_peered":"2024-09-16T01:36:07.724876+0000","last_clean":"2024-09-16T01:36:07.724876+0000","last_became_active":"2024-09-16T01:36:06.365032+0000","last_became_peered":"2024-09-16T01:36:06.365032+0000","last_unstale":"2024-09-16T01:36:07.724876+0000","last_undegraded":"2024-09-16T01:36:07.724876+0000","last_fullsized":"2024-09-16T01:36:07.724876+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:43:42.615172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725359+0000","last_change":"2024-09-16T01:36:06.365094+0000","last_active":"2024-09-16T01:36:07.725359+0000","last_peered":"2024-09-16T01:36:07.725359+0000","last_clean":"2024-09-16T01:36:07.725359+0000","last_became_active":"2024-09-16T01:36:06.364790+0000","last_became_peered":"2024-09-16T01:36:06.364790+0000","last_unstale":"2024-09-16T01:36:07.725359+0000","last_undegraded":"2024-09-16T01:36:07.725359+0000","last_fullsized":"2024-09-16T01:36:07.725359+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:00:06.815324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.353987+0000","last_change":"2024-09-16T01:36:06.369172+0000","last_active":"2024-09-16T01:36:07.353987+0000","last_peered":"2024-09-16T01:36:07.353987+0000","last_clean":"2024-09-16T01:36:07.353987+0000","last_became_active":"2024-09-16T01:36:06.365272+0000","last_became_peered":"2024-09-16T01:36:06.365272+0000","last_unstale":"2024-09-16T01:36:07.353987+0000","last_undegraded":"2024-09-16T01:36:07.353987+0000","last_fullsized":"2024-09-16T01:36:07.353987+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:50:55.284759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.724759+0000","last_change":"2024-09-16T01:36:06.370164+0000","last_active":"2024-09-16T01:36:07.724759+0000","last_peered":"2024-09-16T01:36:07.724759+0000","last_clean":"2024-09-16T01:36:07.724759+0000","last_became_active":"2024-09-16T01:36:06.369904+0000","last_became_peered":"2024-09-16T01:36:06.369904+0000","last_unstale":"2024-09-16T01:36:07.724759+0000","last_undegraded":"2024-09-16T01:36:07.724759+0000","last_fullsized":"2024-09-16T01:36:07.724759+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:52:53.800547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725689+0000","last_change":"2024-09-16T01:36:06.368341+0000","last_active":"2024-09-16T01:36:07.725689+0000","last_peered":"2024-09-16T01:36:07.725689+0000","last_clean":"2024-09-16T01:36:07.725689+0000","last_became_active":"2024-09-16T01:36:06.368081+0000","last_became_peered":"2024-09-16T01:36:06.368081+0000","last_unstale":"2024-09-16T01:36:07.725689+0000","last_undegraded":"2024-09-16T01:36:07.725689+0000","last_fullsized":"2024-09-16T01:36:07.725689+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:03:10.024235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.724734+0000","last_change":"2024-09-16T01:36:06.362104+0000","last_active":"2024-09-16T01:36:07.724734+0000","last_peered":"2024-09-16T01:36:07.724734+0000","last_clean":"2024-09-16T01:36:07.724734+0000","last_became_active":"2024-09-16T01:36:06.361837+0000","last_became_peered":"2024-09-16T01:36:06.361837+0000","last_unstale":"2024-09-16T01:36:07.724734+0000","last_undegraded":"2024-09-16T01:36:07.724734+0000","last_fullsized":"2024-09-16T01:36:07.724734+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:09.577954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725154+0000","last_change":"2024-09-16T01:36:06.379168+0000","last_active":"2024-09-16T01:36:07.725154+0000","last_peered":"2024-09-16T01:36:07.725154+0000","last_clean":"2024-09-16T01:36:07.725154+0000","last_became_active":"2024-09-16T01:36:06.378851+0000","last_became_peered":"2024-09-16T01:36:06.378851+0000","last_unstale":"2024-09-16T01:36:07.725154+0000","last_undegraded":"2024-09-16T01:36:07.725154+0000","last_fullsized":"2024-09-16T01:36:07.725154+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:59.554252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":18,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725303+0000","last_change":"2024-09-16T01:36:05.357580+0000","last_active":"2024-09-16T01:36:07.725303+0000","last_peered":"2024-09-16T01:36:07.725303+0000","last_clean":"2024-09-16T01:36:07.725303+0000","last_became_active":"2024-09-16T01:36:05.357331+0000","last_became_peered":"2024-09-16T01:36:05.357331+0000","last_unstale":"2024-09-16T01:36:07.725303+0000","last_undegraded":"2024-09-16T01:36:07.725303+0000","last_fullsized":"2024-09-16T01:36:07.725303+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:09:18.114627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.353612+0000","last_change":"2024-09-16T01:36:06.373531+0000","last_active":"2024-09-16T01:36:07.353612+0000","last_peered":"2024-09-16T01:36:07.353612+0000","last_clean":"2024-09-16T01:36:07.353612+0000","last_became_active":"2024-09-16T01:36:06.372510+0000","last_became_peered":"2024-09-16T01:36:06.372510+0000","last_unstale":"2024-09-16T01:36:07.353612+0000","last_undegraded":"2024-09-16T01:36:07.353612+0000","last_fullsized":"2024-09-16T01:36:07.353612+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:40:30.748678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725256+0000","last_change":"2024-09-16T01:36:05.357954+0000","last_active":"2024-09-16T01:36:07.725256+0000","last_peered":"2024-09-16T01:36:07.725256+0000","last_clean":"2024-09-16T01:36:07.725256+0000","last_became_active":"2024-09-16T01:36:05.357018+0000","last_became_peered":"2024-09-16T01:36:05.357018+0000","last_unstale":"2024-09-16T01:36:07.725256+0000","last_undegraded":"2024-09-16T01:36:07.725256+0000","last_fullsized":"2024-09-16T01:36:07.725256+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:59:41.248781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725421+0000","last_change":"2024-09-16T01:36:06.377925+0000","last_active":"2024-09-16T01:36:07.725421+0000","last_peered":"2024-09-16T01:36:07.725421+0000","last_clean":"2024-09-16T01:36:07.725421+0000","last_became_active":"2024-09-16T01:36:06.377631+0000","last_became_peered":"2024-09-16T01:36:06.377631+0000","last_unstale":"2024-09-16T01:36:07.725421+0000","last_undegraded":"2024-09-16T01:36:07.725421+0000","last_fullsized":"2024-09-16T01:36:07.725421+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:46:34.269180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725554+0000","last_change":"2024-09-16T01:36:05.365158+0000","last_active":"2024-09-16T01:36:07.725554+0000","last_peered":"2024-09-16T01:36:07.725554+0000","last_clean":"2024-09-16T01:36:07.725554+0000","last_became_active":"2024-09-16T01:36:05.364885+0000","last_became_peered":"2024-09-16T01:36:05.364885+0000","last_unstale":"2024-09-16T01:36:07.725554+0000","last_undegraded":"2024-09-16T01:36:07.725554+0000","last_fullsized":"2024-09-16T01:36:07.725554+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:26:15.812866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725482+0000","last_change":"2024-09-16T01:36:06.373894+0000","last_active":"2024-09-16T01:36:07.725482+0000","last_peered":"2024-09-16T01:36:07.725482+0000","last_clean":"2024-09-16T01:36:07.725482+0000","last_became_active":"2024-09-16T01:36:06.373617+0000","last_became_peered":"2024-09-16T01:36:06.373617+0000","last_unstale":"2024-09-16T01:36:07.725482+0000","last_undegraded":"2024-09-16T01:36:07.725482+0000","last_fullsized":"2024-09-16T01:36:07.725482+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:23.130751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.352872+0000","last_change":"2024-09-16T01:36:05.368193+0000","last_active":"2024-09-16T01:36:07.352872+0000","last_peered":"2024-09-16T01:36:07.352872+0000","last_clean":"2024-09-16T01:36:07.352872+0000","last_became_active":"2024-09-16T01:36:05.367907+0000","last_became_peered":"2024-09-16T01:36:05.367907+0000","last_unstale":"2024-09-16T01:36:07.352872+0000","last_undegraded":"2024-09-16T01:36:07.352872+0000","last_fullsized":"2024-09-16T01:36:07.352872+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:55:37.303541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.724871+0000","last_change":"2024-09-16T01:36:06.370957+0000","last_active":"2024-09-16T01:36:07.724871+0000","last_peered":"2024-09-16T01:36:07.724871+0000","last_clean":"2024-09-16T01:36:07.724871+0000","last_became_active":"2024-09-16T01:36:06.369782+0000","last_became_peered":"2024-09-16T01:36:06.369782+0000","last_unstale":"2024-09-16T01:36:07.724871+0000","last_undegraded":"2024-09-16T01:36:07.724871+0000","last_fullsized":"2024-09-16T01:36:07.724871+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:53:07.260559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725334+0000","last_change":"2024-09-16T01:36:05.361916+0000","last_active":"2024-09-16T01:36:07.725334+0000","last_peered":"2024-09-16T01:36:07.725334+0000","last_clean":"2024-09-16T01:36:07.725334+0000","last_became_active":"2024-09-16T01:36:05.361658+0000","last_became_peered":"2024-09-16T01:36:05.361658+0000","last_unstale":"2024-09-16T01:36:07.725334+0000","last_undegraded":"2024-09-16T01:36:07.725334+0000","last_fullsized":"2024-09-16T01:36:07.725334+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:08:43.161405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.351831+0000","last_change":"2024-09-16T01:36:06.369501+0000","last_active":"2024-09-16T01:36:07.351831+0000","last_peered":"2024-09-16T01:36:07.351831+0000","last_clean":"2024-09-16T01:36:07.351831+0000","last_became_active":"2024-09-16T01:36:06.369239+0000","last_became_peered":"2024-09-16T01:36:06.369239+0000","last_unstale":"2024-09-16T01:36:07.351831+0000","last_undegraded":"2024-09-16T01:36:07.351831+0000","last_fullsized":"2024-09-16T01:36:07.351831+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:19:40.032848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.724886+0000","last_change":"2024-09-16T01:36:05.365323+0000","last_active":"2024-09-16T01:36:07.724886+0000","last_peered":"2024-09-16T01:36:07.724886+0000","last_clean":"2024-09-16T01:36:07.724886+0000","last_became_active":"2024-09-16T01:36:05.364940+0000","last_became_peered":"2024-09-16T01:36:05.364940+0000","last_unstale":"2024-09-16T01:36:07.724886+0000","last_undegraded":"2024-09-16T01:36:07.724886+0000","last_fullsized":"2024-09-16T01:36:07.724886+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:31:31.276265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.351597+0000","last_change":"2024-09-16T01:36:06.364685+0000","last_active":"2024-09-16T01:36:07.351597+0000","last_peered":"2024-09-16T01:36:07.351597+0000","last_clean":"2024-09-16T01:36:07.351597+0000","last_became_active":"2024-09-16T01:36:06.364344+0000","last_became_peered":"2024-09-16T01:36:06.364344+0000","last_unstale":"2024-09-16T01:36:07.351597+0000","last_undegraded":"2024-09-16T01:36:07.351597+0000","last_fullsized":"2024-09-16T01:36:07.351597+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:09:19.116540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.724806+0000","last_change":"2024-09-16T01:36:05.359801+0000","last_active":"2024-09-16T01:36:07.724806+0000","last_peered":"2024-09-16T01:36:07.724806+0000","last_clean":"2024-09-16T01:36:07.724806+0000","last_became_active":"2024-09-16T01:36:05.359542+0000","last_became_peered":"2024-09-16T01:36:05.359542+0000","last_unstale":"2024-09-16T01:36:07.724806+0000","last_undegraded":"2024-09-16T01:36:07.724806+0000","last_fullsized":"2024-09-16T01:36:07.724806+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:22.128838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.353403+0000","last_change":"2024-09-16T01:36:06.372426+0000","last_active":"2024-09-16T01:36:07.353403+0000","last_peered":"2024-09-16T01:36:07.353403+0000","last_clean":"2024-09-16T01:36:07.353403+0000","last_became_active":"2024-09-16T01:36:06.370576+0000","last_became_peered":"2024-09-16T01:36:06.370576+0000","last_unstale":"2024-09-16T01:36:07.353403+0000","last_undegraded":"2024-09-16T01:36:07.353403+0000","last_fullsized":"2024-09-16T01:36:07.353403+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:19:59.213995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.351709+0000","last_change":"2024-09-16T01:36:05.364586+0000","last_active":"2024-09-16T01:36:07.351709+0000","last_peered":"2024-09-16T01:36:07.351709+0000","last_clean":"2024-09-16T01:36:07.351709+0000","last_became_active":"2024-09-16T01:36:05.364271+0000","last_became_peered":"2024-09-16T01:36:05.364271+0000","last_unstale":"2024-09-16T01:36:07.351709+0000","last_undegraded":"2024-09-16T01:36:07.351709+0000","last_fullsized":"2024-09-16T01:36:07.351709+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:59.822221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725159+0000","last_change":"2024-09-16T01:36:06.364448+0000","last_active":"2024-09-16T01:36:07.725159+0000","last_peered":"2024-09-16T01:36:07.725159+0000","last_clean":"2024-09-16T01:36:07.725159+0000","last_became_active":"2024-09-16T01:36:06.363823+0000","last_became_peered":"2024-09-16T01:36:06.363823+0000","last_unstale":"2024-09-16T01:36:07.725159+0000","last_undegraded":"2024-09-16T01:36:07.725159+0000","last_fullsized":"2024-09-16T01:36:07.725159+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:08:44.163318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725278+0000","last_change":"2024-09-16T01:36:05.364097+0000","last_active":"2024-09-16T01:36:07.725278+0000","last_peered":"2024-09-16T01:36:07.725278+0000","last_clean":"2024-09-16T01:36:07.725278+0000","last_became_active":"2024-09-16T01:36:05.363758+0000","last_became_peered":"2024-09-16T01:36:05.363758+0000","last_unstale":"2024-09-16T01:36:07.725278+0000","last_undegraded":"2024-09-16T01:36:07.725278+0000","last_fullsized":"2024-09-16T01:36:07.725278+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:45:08.629221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.353494+0000","last_change":"2024-09-16T01:36:06.370903+0000","last_active":"2024-09-16T01:36:07.353494+0000","last_peered":"2024-09-16T01:36:07.353494+0000","last_clean":"2024-09-16T01:36:07.353494+0000","last_became_active":"2024-09-16T01:36:06.370604+0000","last_became_peered":"2024-09-16T01:36:06.370604+0000","last_unstale":"2024-09-16T01:36:07.353494+0000","last_undegraded":"2024-09-16T01:36:07.353494+0000","last_fullsized":"2024-09-16T01:36:07.353494+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:43.832867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.352198+0000","last_change":"2024-09-16T01:36:05.363830+0000","last_active":"2024-09-16T01:36:07.352198+0000","last_peered":"2024-09-16T01:36:07.352198+0000","last_clean":"2024-09-16T01:36:07.352198+0000","last_became_active":"2024-09-16T01:36:05.363572+0000","last_became_peered":"2024-09-16T01:36:05.363572+0000","last_unstale":"2024-09-16T01:36:07.352198+0000","last_undegraded":"2024-09-16T01:36:07.352198+0000","last_fullsized":"2024-09-16T01:36:07.352198+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:45:08.629221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725808+0000","last_change":"2024-09-16T01:36:06.370848+0000","last_active":"2024-09-16T01:36:07.725808+0000","last_peered":"2024-09-16T01:36:07.725808+0000","last_clean":"2024-09-16T01:36:07.725808+0000","last_became_active":"2024-09-16T01:36:06.368675+0000","last_became_peered":"2024-09-16T01:36:06.368675+0000","last_unstale":"2024-09-16T01:36:07.725808+0000","last_undegraded":"2024-09-16T01:36:07.725808+0000","last_fullsized":"2024-09-16T01:36:07.725808+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:20.613752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"44'3","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.353342+0000","last_change":"2024-09-16T01:36:05.368744+0000","last_active":"2024-09-16T01:36:07.353342+0000","last_peered":"2024-09-16T01:36:07.353342+0000","last_clean":"2024-09-16T01:36:07.353342+0000","last_became_active":"2024-09-16T01:36:05.368069+0000","last_became_peered":"2024-09-16T01:36:05.368069+0000","last_unstale":"2024-09-16T01:36:07.353342+0000","last_undegraded":"2024-09-16T01:36:07.353342+0000","last_fullsized":"2024-09-16T01:36:07.353342+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:40:51.575008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.353052+0000","last_change":"2024-09-16T01:36:06.374871+0000","last_active":"2024-09-16T01:36:07.353052+0000","last_peered":"2024-09-16T01:36:07.353052+0000","last_clean":"2024-09-16T01:36:07.353052+0000","last_became_active":"2024-09-16T01:36:06.374616+0000","last_became_peered":"2024-09-16T01:36:06.374616+0000","last_unstale":"2024-09-16T01:36:07.353052+0000","last_undegraded":"2024-09-16T01:36:07.353052+0000","last_fullsized":"2024-09-16T01:36:07.353052+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:43:39.597563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":20,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.724849+0000","last_change":"2024-09-16T01:36:05.365975+0000","last_active":"2024-09-16T01:36:07.724849+0000","last_peered":"2024-09-16T01:36:07.724849+0000","last_clean":"2024-09-16T01:36:07.724849+0000","last_became_active":"2024-09-16T01:36:05.365674+0000","last_became_peered":"2024-09-16T01:36:05.365674+0000","last_unstale":"2024-09-16T01:36:07.724849+0000","last_undegraded":"2024-09-16T01:36:07.724849+0000","last_fullsized":"2024-09-16T01:36:07.724849+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:04:24.849669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.353405+0000","last_change":"2024-09-16T01:36:06.365792+0000","last_active":"2024-09-16T01:36:07.353405+0000","last_peered":"2024-09-16T01:36:07.353405+0000","last_clean":"2024-09-16T01:36:07.353405+0000","last_became_active":"2024-09-16T01:36:06.365472+0000","last_became_peered":"2024-09-16T01:36:06.365472+0000","last_unstale":"2024-09-16T01:36:07.353405+0000","last_undegraded":"2024-09-16T01:36:07.353405+0000","last_fullsized":"2024-09-16T01:36:07.353405+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:18.730297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725430+0000","last_change":"2024-09-16T01:36:05.360882+0000","last_active":"2024-09-16T01:36:07.725430+0000","last_peered":"2024-09-16T01:36:07.725430+0000","last_clean":"2024-09-16T01:36:07.725430+0000","last_became_active":"2024-09-16T01:36:05.360521+0000","last_became_peered":"2024-09-16T01:36:05.360521+0000","last_unstale":"2024-09-16T01:36:07.725430+0000","last_undegraded":"2024-09-16T01:36:07.725430+0000","last_fullsized":"2024-09-16T01:36:07.725430+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:46:43.234321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.352130+0000","last_change":"2024-09-16T01:36:06.374056+0000","last_active":"2024-09-16T01:36:07.352130+0000","last_peered":"2024-09-16T01:36:07.352130+0000","last_clean":"2024-09-16T01:36:07.352130+0000","last_became_active":"2024-09-16T01:36:06.373802+0000","last_became_peered":"2024-09-16T01:36:06.373802+0000","last_unstale":"2024-09-16T01:36:07.352130+0000","last_undegraded":"2024-09-16T01:36:07.352130+0000","last_fullsized":"2024-09-16T01:36:07.352130+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:32:43.773389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.353166+0000","last_change":"2024-09-16T01:36:05.369476+0000","last_active":"2024-09-16T01:36:07.353166+0000","last_peered":"2024-09-16T01:36:07.353166+0000","last_clean":"2024-09-16T01:36:07.353166+0000","last_became_active":"2024-09-16T01:36:05.369155+0000","last_became_peered":"2024-09-16T01:36:05.369155+0000","last_unstale":"2024-09-16T01:36:07.353166+0000","last_undegraded":"2024-09-16T01:36:07.353166+0000","last_fullsized":"2024-09-16T01:36:07.353166+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:21.760998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.724549+0000","last_change":"2024-09-16T01:36:06.366890+0000","last_active":"2024-09-16T01:36:07.724549+0000","last_peered":"2024-09-16T01:36:07.724549+0000","last_clean":"2024-09-16T01:36:07.724549+0000","last_became_active":"2024-09-16T01:36:06.366638+0000","last_became_peered":"2024-09-16T01:36:06.366638+0000","last_unstale":"2024-09-16T01:36:07.724549+0000","last_undegraded":"2024-09-16T01:36:07.724549+0000","last_fullsized":"2024-09-16T01:36:07.724549+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:06:31.968687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725658+0000","last_change":"2024-09-16T01:36:05.359803+0000","last_active":"2024-09-16T01:36:07.725658+0000","last_peered":"2024-09-16T01:36:07.725658+0000","last_clean":"2024-09-16T01:36:07.725658+0000","last_became_active":"2024-09-16T01:36:05.359552+0000","last_became_peered":"2024-09-16T01:36:05.359552+0000","last_unstale":"2024-09-16T01:36:07.725658+0000","last_undegraded":"2024-09-16T01:36:07.725658+0000","last_fullsized":"2024-09-16T01:36:07.725658+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:33:23.403372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.352883+0000","last_change":"2024-09-16T01:36:06.375825+0000","last_active":"2024-09-16T01:36:07.352883+0000","last_peered":"2024-09-16T01:36:07.352883+0000","last_clean":"2024-09-16T01:36:07.352883+0000","last_became_active":"2024-09-16T01:36:06.375457+0000","last_became_peered":"2024-09-16T01:36:06.375457+0000","last_unstale":"2024-09-16T01:36:07.352883+0000","last_undegraded":"2024-09-16T01:36:07.352883+0000","last_fullsized":"2024-09-16T01:36:07.352883+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:41:23.177683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"44'8","reported_seq":22,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.351748+0000","last_change":"2024-09-16T01:36:05.359125+0000","last_active":"2024-09-16T01:36:07.351748+0000","last_peered":"2024-09-16T01:36:07.351748+0000","last_clean":"2024-09-16T01:36:07.351748+0000","last_became_active":"2024-09-16T01:36:05.358486+0000","last_became_peered":"2024-09-16T01:36:05.358486+0000","last_unstale":"2024-09-16T01:36:07.351748+0000","last_undegraded":"2024-09-16T01:36:07.351748+0000","last_fullsized":"2024-09-16T01:36:07.351748+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:59:41.248781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.724936+0000","last_change":"2024-09-16T01:36:06.362322+0000","last_active":"2024-09-16T01:36:07.724936+0000","last_peered":"2024-09-16T01:36:07.724936+0000","last_clean":"2024-09-16T01:36:07.724936+0000","last_became_active":"2024-09-16T01:36:06.362064+0000","last_became_peered":"2024-09-16T01:36:06.362064+0000","last_unstale":"2024-09-16T01:36:07.724936+0000","last_undegraded":"2024-09-16T01:36:07.724936+0000","last_fullsized":"2024-09-16T01:36:07.724936+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:26:51.409458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.724642+0000","last_change":"2024-09-16T01:36:05.361659+0000","last_active":"2024-09-16T01:36:07.724642+0000","last_peered":"2024-09-16T01:36:07.724642+0000","last_clean":"2024-09-16T01:36:07.724642+0000","last_became_active":"2024-09-16T01:36:05.361408+0000","last_became_peered":"2024-09-16T01:36:05.361408+0000","last_unstale":"2024-09-16T01:36:07.724642+0000","last_undegraded":"2024-09-16T01:36:07.724642+0000","last_fullsized":"2024-09-16T01:36:07.724642+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:18:06.587339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.352066+0000","last_change":"2024-09-16T01:36:06.376373+0000","last_active":"2024-09-16T01:36:07.352066+0000","last_peered":"2024-09-16T01:36:07.352066+0000","last_clean":"2024-09-16T01:36:07.352066+0000","last_became_active":"2024-09-16T01:36:06.375344+0000","last_became_peered":"2024-09-16T01:36:06.375344+0000","last_unstale":"2024-09-16T01:36:07.352066+0000","last_undegraded":"2024-09-16T01:36:07.352066+0000","last_fullsized":"2024-09-16T01:36:07.352066+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:56:50.007758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725378+0000","last_change":"2024-09-16T01:36:05.363734+0000","last_active":"2024-09-16T01:36:07.725378+0000","last_peered":"2024-09-16T01:36:07.725378+0000","last_clean":"2024-09-16T01:36:07.725378+0000","last_became_active":"2024-09-16T01:36:05.363407+0000","last_became_peered":"2024-09-16T01:36:05.363407+0000","last_unstale":"2024-09-16T01:36:07.725378+0000","last_undegraded":"2024-09-16T01:36:07.725378+0000","last_fullsized":"2024-09-16T01:36:07.725378+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:38.732106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.353607+0000","last_change":"2024-09-16T01:36:06.378982+0000","last_active":"2024-09-16T01:36:07.353607+0000","last_peered":"2024-09-16T01:36:07.353607+0000","last_clean":"2024-09-16T01:36:07.353607+0000","last_became_active":"2024-09-16T01:36:06.378732+0000","last_became_peered":"2024-09-16T01:36:06.378732+0000","last_unstale":"2024-09-16T01:36:07.353607+0000","last_undegraded":"2024-09-16T01:36:07.353607+0000","last_fullsized":"2024-09-16T01:36:07.353607+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:57:34.963379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"44'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725328+0000","last_change":"2024-09-16T01:36:05.360643+0000","last_active":"2024-09-16T01:36:07.725328+0000","last_peered":"2024-09-16T01:36:07.725328+0000","last_clean":"2024-09-16T01:36:07.725328+0000","last_became_active":"2024-09-16T01:36:05.360204+0000","last_became_peered":"2024-09-16T01:36:05.360204+0000","last_unstale":"2024-09-16T01:36:07.725328+0000","last_undegraded":"2024-09-16T01:36:07.725328+0000","last_fullsized":"2024-09-16T01:36:07.725328+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:54.497479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.354166+0000","last_change":"2024-09-16T01:36:06.373173+0000","last_active":"2024-09-16T01:36:07.354166+0000","last_peered":"2024-09-16T01:36:07.354166+0000","last_clean":"2024-09-16T01:36:07.354166+0000","last_became_active":"2024-09-16T01:36:06.372890+0000","last_became_peered":"2024-09-16T01:36:06.372890+0000","last_unstale":"2024-09-16T01:36:07.354166+0000","last_undegraded":"2024-09-16T01:36:07.354166+0000","last_fullsized":"2024-09-16T01:36:07.354166+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:05:48.002195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725796+0000","last_change":"2024-09-16T01:36:05.363498+0000","last_active":"2024-09-16T01:36:07.725796+0000","last_peered":"2024-09-16T01:36:07.725796+0000","last_clean":"2024-09-16T01:36:07.725796+0000","last_became_active":"2024-09-16T01:36:05.363185+0000","last_became_peered":"2024-09-16T01:36:05.363185+0000","last_unstale":"2024-09-16T01:36:07.725796+0000","last_undegraded":"2024-09-16T01:36:07.725796+0000","last_fullsized":"2024-09-16T01:36:07.725796+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:53:06.258646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725656+0000","last_change":"2024-09-16T01:36:06.368593+0000","last_active":"2024-09-16T01:36:07.725656+0000","last_peered":"2024-09-16T01:36:07.725656+0000","last_clean":"2024-09-16T01:36:07.725656+0000","last_became_active":"2024-09-16T01:36:06.368314+0000","last_became_peered":"2024-09-16T01:36:06.368314+0000","last_unstale":"2024-09-16T01:36:07.725656+0000","last_undegraded":"2024-09-16T01:36:07.725656+0000","last_fullsized":"2024-09-16T01:36:07.725656+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:43.488298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725586+0000","last_change":"2024-09-16T01:36:05.361868+0000","last_active":"2024-09-16T01:36:07.725586+0000","last_peered":"2024-09-16T01:36:07.725586+0000","last_clean":"2024-09-16T01:36:07.725586+0000","last_became_active":"2024-09-16T01:36:05.361605+0000","last_became_peered":"2024-09-16T01:36:05.361605+0000","last_unstale":"2024-09-16T01:36:07.725586+0000","last_undegraded":"2024-09-16T01:36:07.725586+0000","last_fullsized":"2024-09-16T01:36:07.725586+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:19:39.030935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.724932+0000","last_change":"2024-09-16T01:36:06.368514+0000","last_active":"2024-09-16T01:36:07.724932+0000","last_peered":"2024-09-16T01:36:07.724932+0000","last_clean":"2024-09-16T01:36:07.724932+0000","last_became_active":"2024-09-16T01:36:06.368255+0000","last_became_peered":"2024-09-16T01:36:06.368255+0000","last_unstale":"2024-09-16T01:36:07.724932+0000","last_undegraded":"2024-09-16T01:36:07.724932+0000","last_fullsized":"2024-09-16T01:36:07.724932+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:09:25.704964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.724949+0000","last_change":"2024-09-16T01:36:05.356369+0000","last_active":"2024-09-16T01:36:07.724949+0000","last_peered":"2024-09-16T01:36:07.724949+0000","last_clean":"2024-09-16T01:36:07.724949+0000","last_became_active":"2024-09-16T01:36:05.356027+0000","last_became_peered":"2024-09-16T01:36:05.356027+0000","last_unstale":"2024-09-16T01:36:07.724949+0000","last_undegraded":"2024-09-16T01:36:07.724949+0000","last_fullsized":"2024-09-16T01:36:07.724949+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:02.446309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.724798+0000","last_change":"2024-09-16T01:36:06.373300+0000","last_active":"2024-09-16T01:36:07.724798+0000","last_peered":"2024-09-16T01:36:07.724798+0000","last_clean":"2024-09-16T01:36:07.724798+0000","last_became_active":"2024-09-16T01:36:06.373014+0000","last_became_peered":"2024-09-16T01:36:06.373014+0000","last_unstale":"2024-09-16T01:36:07.724798+0000","last_undegraded":"2024-09-16T01:36:07.724798+0000","last_fullsized":"2024-09-16T01:36:07.724798+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:31:25.709991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.351604+0000","last_change":"2024-09-16T01:36:05.363731+0000","last_active":"2024-09-16T01:36:07.351604+0000","last_peered":"2024-09-16T01:36:07.351604+0000","last_clean":"2024-09-16T01:36:07.351604+0000","last_became_active":"2024-09-16T01:36:05.363475+0000","last_became_peered":"2024-09-16T01:36:05.363475+0000","last_unstale":"2024-09-16T01:36:07.351604+0000","last_undegraded":"2024-09-16T01:36:07.351604+0000","last_fullsized":"2024-09-16T01:36:07.351604+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:56:30.024164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.351628+0000","last_change":"2024-09-16T01:36:06.363890+0000","last_active":"2024-09-16T01:36:07.351628+0000","last_peered":"2024-09-16T01:36:07.351628+0000","last_clean":"2024-09-16T01:36:07.351628+0000","last_became_active":"2024-09-16T01:36:06.363500+0000","last_became_peered":"2024-09-16T01:36:06.363500+0000","last_unstale":"2024-09-16T01:36:07.351628+0000","last_undegraded":"2024-09-16T01:36:07.351628+0000","last_fullsized":"2024-09-16T01:36:07.351628+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:53:07.260559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.352748+0000","last_change":"2024-09-16T01:36:05.367836+0000","last_active":"2024-09-16T01:36:07.352748+0000","last_peered":"2024-09-16T01:36:07.352748+0000","last_clean":"2024-09-16T01:36:07.352748+0000","last_became_active":"2024-09-16T01:36:05.367513+0000","last_became_peered":"2024-09-16T01:36:05.367513+0000","last_unstale":"2024-09-16T01:36:07.352748+0000","last_undegraded":"2024-09-16T01:36:07.352748+0000","last_fullsized":"2024-09-16T01:36:07.352748+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:04:54.497479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.354093+0000","last_change":"2024-09-16T01:36:06.365983+0000","last_active":"2024-09-16T01:36:07.354093+0000","last_peered":"2024-09-16T01:36:07.354093+0000","last_clean":"2024-09-16T01:36:07.354093+0000","last_became_active":"2024-09-16T01:36:06.365631+0000","last_became_peered":"2024-09-16T01:36:06.365631+0000","last_unstale":"2024-09-16T01:36:07.354093+0000","last_undegraded":"2024-09-16T01:36:07.354093+0000","last_fullsized":"2024-09-16T01:36:07.354093+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:28:42.468084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"45'1","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725067+0000","last_change":"2024-09-16T01:36:05.365994+0000","last_active":"2024-09-16T01:36:07.725067+0000","last_peered":"2024-09-16T01:36:07.725067+0000","last_clean":"2024-09-16T01:36:07.725067+0000","last_became_active":"2024-09-16T01:36:05.365629+0000","last_became_peered":"2024-09-16T01:36:05.365629+0000","last_unstale":"2024-09-16T01:36:07.725067+0000","last_undegraded":"2024-09-16T01:36:07.725067+0000","last_fullsized":"2024-09-16T01:36:07.725067+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:03:59.842772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.351906+0000","last_change":"2024-09-16T01:36:06.363663+0000","last_active":"2024-09-16T01:36:07.351906+0000","last_peered":"2024-09-16T01:36:07.351906+0000","last_clean":"2024-09-16T01:36:07.351906+0000","last_became_active":"2024-09-16T01:36:06.363384+0000","last_became_peered":"2024-09-16T01:36:06.363384+0000","last_unstale":"2024-09-16T01:36:07.351906+0000","last_undegraded":"2024-09-16T01:36:07.351906+0000","last_fullsized":"2024-09-16T01:36:07.351906+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:17:49.000756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.353454+0000","last_change":"2024-09-16T01:36:05.370944+0000","last_active":"2024-09-16T01:36:07.353454+0000","last_peered":"2024-09-16T01:36:07.353454+0000","last_clean":"2024-09-16T01:36:07.353454+0000","last_became_active":"2024-09-16T01:36:05.370631+0000","last_became_peered":"2024-09-16T01:36:05.370631+0000","last_unstale":"2024-09-16T01:36:07.353454+0000","last_undegraded":"2024-09-16T01:36:07.353454+0000","last_fullsized":"2024-09-16T01:36:07.353454+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:39:27.691186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.353237+0000","last_change":"2024-09-16T01:36:06.376156+0000","last_active":"2024-09-16T01:36:07.353237+0000","last_peered":"2024-09-16T01:36:07.353237+0000","last_clean":"2024-09-16T01:36:07.353237+0000","last_became_active":"2024-09-16T01:36:06.375889+0000","last_became_peered":"2024-09-16T01:36:06.375889+0000","last_unstale":"2024-09-16T01:36:07.353237+0000","last_undegraded":"2024-09-16T01:36:07.353237+0000","last_fullsized":"2024-09-16T01:36:07.353237+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:46:02.810323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.724957+0000","last_change":"2024-09-16T01:36:05.365868+0000","last_active":"2024-09-16T01:36:07.724957+0000","last_peered":"2024-09-16T01:36:07.724957+0000","last_clean":"2024-09-16T01:36:07.724957+0000","last_became_active":"2024-09-16T01:36:05.365481+0000","last_became_peered":"2024-09-16T01:36:05.365481+0000","last_unstale":"2024-09-16T01:36:07.724957+0000","last_undegraded":"2024-09-16T01:36:07.724957+0000","last_fullsized":"2024-09-16T01:36:07.724957+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:03:18.838317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725699+0000","last_change":"2024-09-16T01:36:06.370691+0000","last_active":"2024-09-16T01:36:07.725699+0000","last_peered":"2024-09-16T01:36:07.725699+0000","last_clean":"2024-09-16T01:36:07.725699+0000","last_became_active":"2024-09-16T01:36:06.370427+0000","last_became_peered":"2024-09-16T01:36:06.370427+0000","last_unstale":"2024-09-16T01:36:07.725699+0000","last_undegraded":"2024-09-16T01:36:07.725699+0000","last_fullsized":"2024-09-16T01:36:07.725699+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:05:15.782141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.353250+0000","last_change":"2024-09-16T01:36:05.368272+0000","last_active":"2024-09-16T01:36:07.353250+0000","last_peered":"2024-09-16T01:36:07.353250+0000","last_clean":"2024-09-16T01:36:07.353250+0000","last_became_active":"2024-09-16T01:36:05.367973+0000","last_became_peered":"2024-09-16T01:36:05.367973+0000","last_unstale":"2024-09-16T01:36:07.353250+0000","last_undegraded":"2024-09-16T01:36:07.353250+0000","last_fullsized":"2024-09-16T01:36:07.353250+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:29:15.204232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725250+0000","last_change":"2024-09-16T01:36:06.368993+0000","last_active":"2024-09-16T01:36:07.725250+0000","last_peered":"2024-09-16T01:36:07.725250+0000","last_clean":"2024-09-16T01:36:07.725250+0000","last_became_active":"2024-09-16T01:36:06.368285+0000","last_became_peered":"2024-09-16T01:36:06.368285+0000","last_unstale":"2024-09-16T01:36:07.725250+0000","last_undegraded":"2024-09-16T01:36:07.725250+0000","last_fullsized":"2024-09-16T01:36:07.725250+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:02:59.407011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725001+0000","last_change":"2024-09-16T01:36:05.362224+0000","last_active":"2024-09-16T01:36:07.725001+0000","last_peered":"2024-09-16T01:36:07.725001+0000","last_clean":"2024-09-16T01:36:07.725001+0000","last_became_active":"2024-09-16T01:36:05.361853+0000","last_became_peered":"2024-09-16T01:36:05.361853+0000","last_unstale":"2024-09-16T01:36:07.725001+0000","last_undegraded":"2024-09-16T01:36:07.725001+0000","last_fullsized":"2024-09-16T01:36:07.725001+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:17.728385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.724735+0000","last_change":"2024-09-16T01:36:06.363427+0000","last_active":"2024-09-16T01:36:07.724735+0000","last_peered":"2024-09-16T01:36:07.724735+0000","last_clean":"2024-09-16T01:36:07.724735+0000","last_became_active":"2024-09-16T01:36:06.363142+0000","last_became_peered":"2024-09-16T01:36:06.363142+0000","last_unstale":"2024-09-16T01:36:07.724735+0000","last_undegraded":"2024-09-16T01:36:07.724735+0000","last_fullsized":"2024-09-16T01:36:07.724735+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:21:30.939720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725196+0000","last_change":"2024-09-16T01:36:05.362621+0000","last_active":"2024-09-16T01:36:07.725196+0000","last_peered":"2024-09-16T01:36:07.725196+0000","last_clean":"2024-09-16T01:36:07.725196+0000","last_became_active":"2024-09-16T01:36:05.362300+0000","last_became_peered":"2024-09-16T01:36:05.362300+0000","last_unstale":"2024-09-16T01:36:07.725196+0000","last_undegraded":"2024-09-16T01:36:07.725196+0000","last_fullsized":"2024-09-16T01:36:07.725196+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:29:24.035031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.724740+0000","last_change":"2024-09-16T01:36:06.377934+0000","last_active":"2024-09-16T01:36:07.724740+0000","last_peered":"2024-09-16T01:36:07.724740+0000","last_clean":"2024-09-16T01:36:07.724740+0000","last_became_active":"2024-09-16T01:36:06.377653+0000","last_became_peered":"2024-09-16T01:36:06.377653+0000","last_unstale":"2024-09-16T01:36:07.724740+0000","last_undegraded":"2024-09-16T01:36:07.724740+0000","last_fullsized":"2024-09-16T01:36:07.724740+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:29:01.843744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.351845+0000","last_change":"2024-09-16T01:36:05.362040+0000","last_active":"2024-09-16T01:36:07.351845+0000","last_peered":"2024-09-16T01:36:07.351845+0000","last_clean":"2024-09-16T01:36:07.351845+0000","last_became_active":"2024-09-16T01:36:05.361767+0000","last_became_peered":"2024-09-16T01:36:05.361767+0000","last_unstale":"2024-09-16T01:36:07.351845+0000","last_undegraded":"2024-09-16T01:36:07.351845+0000","last_fullsized":"2024-09-16T01:36:07.351845+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:38:11.910238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.352184+0000","last_change":"2024-09-16T01:36:06.374436+0000","last_active":"2024-09-16T01:36:07.352184+0000","last_peered":"2024-09-16T01:36:07.352184+0000","last_clean":"2024-09-16T01:36:07.352184+0000","last_became_active":"2024-09-16T01:36:06.373454+0000","last_became_peered":"2024-09-16T01:36:06.373454+0000","last_unstale":"2024-09-16T01:36:07.352184+0000","last_undegraded":"2024-09-16T01:36:07.352184+0000","last_fullsized":"2024-09-16T01:36:07.352184+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:05.342630+0000","last_clean_scrub_stamp":"2024-09-16T01:36:05.342630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:58:57.577279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"44'2","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.725238+0000","last_change":"2024-09-16T01:36:05.361909+0000","last_active":"2024-09-16T01:36:07.725238+0000","last_peered":"2024-09-16T01:36:07.725238+0000","last_clean":"2024-09-16T01:36:07.725238+0000","last_became_active":"2024-09-16T01:36:05.361632+0000","last_became_peered":"2024-09-16T01:36:05.361632+0000","last_unstale":"2024-09-16T01:36:07.725238+0000","last_undegraded":"2024-09-16T01:36:07.725238+0000","last_fullsized":"2024-09-16T01:36:07.725238+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:04.340718+0000","last_clean_scrub_stamp":"2024-09-16T01:36:04.340718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:55:37.303541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":213,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T01:36:07.353001+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:36:07.353001+0000","last_peered":"2024-09-16T01:36:07.353001+0000","last_clean":"2024-09-16T01:36:07.353001+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:36:07.353001+0000","last_undegraded":"2024-09-16T01:36:07.353001+0000","last_fullsized":"2024-09-16T01:36:07.353001+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509469,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27544,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344296,"statfs":{"total":96636764160,"available":96608559104,"internally_reserved":0,"allocated":528384,"data_stored":212257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542172,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28132,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343708,"statfs":{"total":96636764160,"available":96607956992,"internally_reserved":0,"allocated":1122304,"data_stored":802645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542172,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27552,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344288,"statfs":{"total":96636764160,"available":96608550912,"internally_reserved":0,"allocated":532480,"data_stored":212859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542172,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27556,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344284,"statfs":{"total":96636764160,"available":96608546816,"internally_reserved":0,"allocated":536576,"data_stored":213799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542171,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27560,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344280,"statfs":{"total":96636764160,"available":96608542720,"internally_reserved":0,"allocated":536576,"data_stored":212825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542172,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27560,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344280,"statfs":{"total":96636764160,"available":96608542720,"internally_reserved":0,"allocated":540672,"data_stored":213821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542171,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27548,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344292,"statfs":{"total":96636764160,"available":96608555008,"internally_reserved":0,"allocated":528384,"data_stored":212257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542171,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28132,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343708,"statfs":{"total":96636764160,"available":96607956992,"internally_reserved":0,"allocated":1122304,"data_stored":802589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:36:13.963 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T01:36:13.963 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-16T01:36:14.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.104+0000 7fd8f6e5e640 1 Processor -- start 2024-09-16T01:36:14.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.104+0000 7fd8f6e5e640 1 -- start start 2024-09-16T01:36:14.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.105+0000 7fd8f6e5e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd8f015a050 0x7fd8f015d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:14.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.105+0000 7fd8f6e5e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd8f015ee90 0x7fd8f0073600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:14.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.105+0000 7fd8f6e5e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8f015a430 con 0x7fd8f00731c0 2024-09-16T01:36:14.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.105+0000 7fd8f6e5e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd8f0073b40 con 0x7fd8f015ee90 2024-09-16T01:36:14.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.105+0000 7fd8f6e5e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd8f015d9a0 con 0x7fd8f015a050 2024-09-16T01:36:14.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.105+0000 7fd8f53fb640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd8f015ee90 0x7fd8f0073600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:14.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.105+0000 7fd8f53fb640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd8f015ee90 0x7fd8f0073600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48692/0 (socket says 172.21.15.88:48692) 2024-09-16T01:36:14.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.105+0000 7fd8f53fb640 1 -- 172.21.15.88:0/2559376553 learned_addr learned my addr 172.21.15.88:0/2559376553 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:14.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.105+0000 7fd8f4bfa640 1 --2- 172.21.15.88:0/2559376553 >> v2:172.21.15.154:3300/0 conn(0x7fd8f015a050 0x7fd8f015d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:14.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.105+0000 7fd8f53fb640 1 -- 172.21.15.88:0/2559376553 >> v2:172.21.15.154:3300/0 conn(0x7fd8f015a050 msgr2=0x7fd8f015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:14.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.105+0000 7fd8f53fb640 1 --2- 172.21.15.88:0/2559376553 >> v2:172.21.15.154:3300/0 conn(0x7fd8f015a050 0x7fd8f015d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:14.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.105+0000 7fd8f53fb640 1 -- 172.21.15.88:0/2559376553 >> v1:172.21.15.88:6789/0 conn(0x7fd8f00731c0 legacy=0x7fd8f015ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:14.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.105+0000 7fd8ef7fe640 1 -- 172.21.15.88:0/2559376553 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 270872792 0 0) 0x7fd8f015a430 con 0x7fd8f00731c0 2024-09-16T01:36:14.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.105+0000 7fd8f53fb640 1 -- 172.21.15.88:0/2559376553 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8f00cd0a0 con 0x7fd8f015ee90 2024-09-16T01:36:14.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.106+0000 7fd8f53fb640 1 --2- 172.21.15.88:0/2559376553 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8f015ee90 0x7fd8f0073600 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0069470 tx=0x7fd8e802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81eb5591fad2c303 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:14.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.106+0000 7fd8ef7fe640 1 -- 172.21.15.88:0/2559376553 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8036070 con 0x7fd8f015ee90 2024-09-16T01:36:14.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.106+0000 7fd8ef7fe640 1 -- 172.21.15.88:0/2559376553 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8e8031600 con 0x7fd8f015ee90 2024-09-16T01:36:14.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.106+0000 7fd8ef7fe640 1 -- 172.21.15.88:0/2559376553 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e80319d0 con 0x7fd8f015ee90 2024-09-16T01:36:14.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.107+0000 7fd8f6e5e640 1 -- 172.21.15.88:0/2559376553 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8f015ee90 msgr2=0x7fd8f0073600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:14.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.107+0000 7fd8f6e5e640 1 --2- 172.21.15.88:0/2559376553 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8f015ee90 0x7fd8f0073600 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0069470 tx=0x7fd8e802a6a0 comp rx=0 tx=0).stop 2024-09-16T01:36:14.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.107+0000 7fd8f6e5e640 1 -- 172.21.15.88:0/2559376553 shutdown_connections 2024-09-16T01:36:14.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.107+0000 7fd8f6e5e640 1 --2- 172.21.15.88:0/2559376553 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8f015ee90 0x7fd8f0073600 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:14.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.107+0000 7fd8f6e5e640 1 --2- 172.21.15.88:0/2559376553 >> v2:172.21.15.154:3300/0 conn(0x7fd8f015a050 0x7fd8f015d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:14.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.107+0000 7fd8f6e5e640 1 -- 172.21.15.88:0/2559376553 >> 172.21.15.88:0/2559376553 conn(0x7fd8f006d0f0 msgr2=0x7fd8f006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:14.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.107+0000 7fd8f6e5e640 1 -- 172.21.15.88:0/2559376553 shutdown_connections 2024-09-16T01:36:14.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.107+0000 7fd8f6e5e640 1 -- 172.21.15.88:0/2559376553 wait complete. 2024-09-16T01:36:14.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.108+0000 7fd8f6e5e640 1 Processor -- start 2024-09-16T01:36:14.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.108+0000 7fd8f6e5e640 1 -- start start 2024-09-16T01:36:14.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.109+0000 7fd8f6e5e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8f00731c0 0x7fd8f01565d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:14.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.109+0000 7fd8f6e5e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8f015a050 0x7fd8f01e4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:14.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.109+0000 7fd8f6e5e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8f015ee90 0x7fd8f0145c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:14.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.109+0000 7fd8f6e5e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f00cd940 con 0x7fd8f015ee90 2024-09-16T01:36:14.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.109+0000 7fd8f6e5e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f00cd660 con 0x7fd8f00731c0 2024-09-16T01:36:14.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.109+0000 7fd8f6e5e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd8f00cdc20 con 0x7fd8f015a050 2024-09-16T01:36:14.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.109+0000 7fd8f53fb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8f015ee90 0x7fd8f0145c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:14.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.109+0000 7fd8f53fb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8f015ee90 0x7fd8f0145c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48704/0 (socket says 172.21.15.88:48704) 2024-09-16T01:36:14.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.109+0000 7fd8f53fb640 1 -- 172.21.15.88:0/841653202 learned_addr learned my addr 172.21.15.88:0/841653202 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:14.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.109+0000 7fd8f4bfa640 1 --2- 172.21.15.88:0/841653202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8f00731c0 0x7fd8f01565d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:14.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.109+0000 7fd8f53fb640 1 -- 172.21.15.88:0/841653202 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8f015a050 msgr2=0x7fd8f01e4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:14.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.109+0000 7fd8f53fb640 1 --2- 172.21.15.88:0/841653202 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8f015a050 0x7fd8f01e4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:14.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.109+0000 7fd8f53fb640 1 -- 172.21.15.88:0/841653202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8f00731c0 msgr2=0x7fd8f01565d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:14.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.109+0000 7fd8f53fb640 1 --2- 172.21.15.88:0/841653202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8f00731c0 0x7fd8f01565d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:14.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.109+0000 7fd8f53fb640 1 -- 172.21.15.88:0/841653202 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8f01e56f0 con 0x7fd8f015ee90 2024-09-16T01:36:14.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.109+0000 7fd8f53fb640 1 --2- 172.21.15.88:0/841653202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8f015ee90 0x7fd8f0145c70 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8034040 tx=0x7fd8e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:14.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.110+0000 7fd8ed7fa640 1 -- 172.21.15.88:0/841653202 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8041020 con 0x7fd8f015ee90 2024-09-16T01:36:14.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.110+0000 7fd8ed7fa640 1 -- 172.21.15.88:0/841653202 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8e8037040 con 0x7fd8f015ee90 2024-09-16T01:36:14.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.110+0000 7fd8f6e5e640 1 -- 172.21.15.88:0/841653202 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8f01e59d0 con 0x7fd8f015ee90 2024-09-16T01:36:14.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.110+0000 7fd8ed7fa640 1 -- 172.21.15.88:0/841653202 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8036050 con 0x7fd8f015ee90 2024-09-16T01:36:14.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.110+0000 7fd8f6e5e640 1 -- 172.21.15.88:0/841653202 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8f01e5f90 con 0x7fd8f015ee90 2024-09-16T01:36:14.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.111+0000 7fd8f6e5e640 1 -- 172.21.15.88:0/841653202 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8f00b36e0 con 0x7fd8f015ee90 2024-09-16T01:36:14.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.111+0000 7fd8ed7fa640 1 -- 172.21.15.88:0/841653202 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd8e8007c30 con 0x7fd8f015ee90 2024-09-16T01:36:14.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.111+0000 7fd8ed7fa640 1 --2- 172.21.15.88:0/841653202 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd8dc06c2b0 0x7fd8dc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:14.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.112+0000 7fd8ed7fa640 1 -- 172.21.15.88:0/841653202 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fd8e80abc40 con 0x7fd8f015ee90 2024-09-16T01:36:14.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.112+0000 7fd8f4bfa640 1 --2- 172.21.15.88:0/841653202 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd8dc06c2b0 0x7fd8dc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:14.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.112+0000 7fd8f4bfa640 1 --2- 172.21.15.88:0/841653202 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd8dc06c2b0 0x7fd8dc06e770 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0006fd0 tx=0x7fd8e0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:14.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.116+0000 7fd8ed7fa640 1 -- 172.21.15.88:0/841653202 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd8e804b060 con 0x7fd8f015ee90 2024-09-16T01:36:14.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.345+0000 7fd8f6e5e640 1 -- 172.21.15.88:0/841653202 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fd8f00d0d70 con 0x7fd8f015ee90 2024-09-16T01:36:14.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.347+0000 7fd8ed7fa640 1 -- 172.21.15.88:0/841653202 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) ==== 190+0+147 (secure 0 0 0) 0x7fd8e807a440 con 0x7fd8f015ee90 2024-09-16T01:36:14.349 INFO:teuthology.orchestra.run.smithi088.stdout:[client.0] 2024-09-16T01:36:14.350 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDeiudmvPikIBAAyHxNNnwQKygxg09PNQLc/Q== 2024-09-16T01:36:14.350 INFO:teuthology.orchestra.run.smithi088.stdout: caps mds = "allow" 2024-09-16T01:36:14.350 INFO:teuthology.orchestra.run.smithi088.stdout: caps mon = "allow r" 2024-09-16T01:36:14.350 INFO:teuthology.orchestra.run.smithi088.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-16T01:36:14.350 INFO:teuthology.orchestra.run.smithi088.stderr:updated caps for client.0 2024-09-16T01:36:14.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.349+0000 7fd8f6e5e640 1 -- 172.21.15.88:0/841653202 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd8dc06c2b0 msgr2=0x7fd8dc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:14.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.349+0000 7fd8f6e5e640 1 --2- 172.21.15.88:0/841653202 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd8dc06c2b0 0x7fd8dc06e770 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0006fd0 tx=0x7fd8e0008040 comp rx=0 tx=0).stop 2024-09-16T01:36:14.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.349+0000 7fd8f6e5e640 1 -- 172.21.15.88:0/841653202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8f015ee90 msgr2=0x7fd8f0145c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:14.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.349+0000 7fd8f6e5e640 1 --2- 172.21.15.88:0/841653202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8f015ee90 0x7fd8f0145c70 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8034040 tx=0x7fd8e8004a40 comp rx=0 tx=0).stop 2024-09-16T01:36:14.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.349+0000 7fd8f6e5e640 1 -- 172.21.15.88:0/841653202 shutdown_connections 2024-09-16T01:36:14.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.349+0000 7fd8f6e5e640 1 --2- 172.21.15.88:0/841653202 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd8dc06c2b0 0x7fd8dc06e770 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:14.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.349+0000 7fd8f6e5e640 1 --2- 172.21.15.88:0/841653202 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8f015ee90 0x7fd8f0145c70 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:14.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.349+0000 7fd8f6e5e640 1 --2- 172.21.15.88:0/841653202 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8f015a050 0x7fd8f01e4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:14.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.349+0000 7fd8f6e5e640 1 --2- 172.21.15.88:0/841653202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8f00731c0 0x7fd8f01565d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:14.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.349+0000 7fd8f6e5e640 1 -- 172.21.15.88:0/841653202 >> 172.21.15.88:0/841653202 conn(0x7fd8f006d0f0 msgr2=0x7fd8f015dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:14.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.349+0000 7fd8f6e5e640 1 -- 172.21.15.88:0/841653202 shutdown_connections 2024-09-16T01:36:14.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.350+0000 7fd8f6e5e640 1 -- 172.21.15.88:0/841653202 wait complete. 2024-09-16T01:36:14.363 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:36:14.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.459+0000 7f4f7334b640 1 Processor -- start 2024-09-16T01:36:14.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.459+0000 7f4f7334b640 1 -- start start 2024-09-16T01:36:14.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.460+0000 7f4f7334b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4f6c15c2b0 0x7f4f6c0b6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:14.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.460+0000 7f4f7334b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f6c0b72f0 0x7f4f6c15eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:14.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.460+0000 7f4f7334b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4f6c15f030 0x7f4f6c06f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:14.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.460+0000 7f4f7334b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f4f6c15fb10 con 0x7f4f6c15f030 2024-09-16T01:36:14.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.460+0000 7f4f7334b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f4f6c0b5900 con 0x7f4f6c15c2b0 2024-09-16T01:36:14.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.460+0000 7f4f7334b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4f6c15f830 con 0x7f4f6c0b72f0 2024-09-16T01:36:14.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.460+0000 7f4f718e8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4f6c15f030 0x7f4f6c06f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:14.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.460+0000 7f4f718e8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4f6c15f030 0x7f4f6c06f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48706/0 (socket says 172.21.15.88:48706) 2024-09-16T01:36:14.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.460+0000 7f4f718e8640 1 -- 172.21.15.88:0/2322187108 learned_addr learned my addr 172.21.15.88:0/2322187108 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:14.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.460+0000 7f4f710e7640 1 --2- 172.21.15.88:0/2322187108 >> v2:172.21.15.154:3300/0 conn(0x7f4f6c15c2b0 0x7f4f6c0b6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:14.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.460+0000 7f4f708e6640 1 --2- 172.21.15.88:0/2322187108 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f6c0b72f0 0x7f4f6c15eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:14.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.460+0000 7f4f718e8640 1 -- 172.21.15.88:0/2322187108 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f6c0b72f0 msgr2=0x7f4f6c15eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:14.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.460+0000 7f4f718e8640 1 --2- 172.21.15.88:0/2322187108 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f6c0b72f0 0x7f4f6c15eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:14.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.460+0000 7f4f718e8640 1 -- 172.21.15.88:0/2322187108 >> v2:172.21.15.154:3300/0 conn(0x7f4f6c15c2b0 msgr2=0x7f4f6c0b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:14.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.461+0000 7f4f718e8640 1 --2- 172.21.15.88:0/2322187108 >> v2:172.21.15.154:3300/0 conn(0x7f4f6c15c2b0 0x7f4f6c0b6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:14.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.461+0000 7f4f718e8640 1 -- 172.21.15.88:0/2322187108 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f6c06ce30 con 0x7f4f6c15f030 2024-09-16T01:36:14.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.461+0000 7f4f718e8640 1 --2- 172.21.15.88:0/2322187108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f6c15f030 0x7f4f6c06f3d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c06f040 tx=0x7f4f6402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f8513aab58a9f9a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:14.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.461+0000 7f4f63fff640 1 -- 172.21.15.88:0/2322187108 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f64037070 con 0x7f4f6c15f030 2024-09-16T01:36:14.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.461+0000 7f4f63fff640 1 -- 172.21.15.88:0/2322187108 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f64032440 con 0x7f4f6c15f030 2024-09-16T01:36:14.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.461+0000 7f4f63fff640 1 -- 172.21.15.88:0/2322187108 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f64032810 con 0x7f4f6c15f030 2024-09-16T01:36:14.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.462+0000 7f4f7334b640 1 -- 172.21.15.88:0/2322187108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f6c15f030 msgr2=0x7f4f6c06f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:14.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.462+0000 7f4f7334b640 1 --2- 172.21.15.88:0/2322187108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f6c15f030 0x7f4f6c06f3d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c06f040 tx=0x7f4f6402a1e0 comp rx=0 tx=0).stop 2024-09-16T01:36:14.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.462+0000 7f4f7334b640 1 -- 172.21.15.88:0/2322187108 shutdown_connections 2024-09-16T01:36:14.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.462+0000 7f4f7334b640 1 --2- 172.21.15.88:0/2322187108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f6c15f030 0x7f4f6c06f3d0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:14.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.462+0000 7f4f7334b640 1 --2- 172.21.15.88:0/2322187108 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f6c0b72f0 0x7f4f6c15eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:14.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.462+0000 7f4f7334b640 1 --2- 172.21.15.88:0/2322187108 >> v2:172.21.15.154:3300/0 conn(0x7f4f6c15c2b0 0x7f4f6c0b6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:14.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.462+0000 7f4f7334b640 1 -- 172.21.15.88:0/2322187108 >> 172.21.15.88:0/2322187108 conn(0x7f4f6c0b4240 msgr2=0x7f4f6c0b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:14.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.462+0000 7f4f7334b640 1 -- 172.21.15.88:0/2322187108 shutdown_connections 2024-09-16T01:36:14.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.462+0000 7f4f7334b640 1 -- 172.21.15.88:0/2322187108 wait complete. 2024-09-16T01:36:14.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.463+0000 7f4f7334b640 1 Processor -- start 2024-09-16T01:36:14.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.463+0000 7f4f7334b640 1 -- start start 2024-09-16T01:36:14.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.464+0000 7f4f7334b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f6c0b72f0 0x7f4f6c149ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:14.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.464+0000 7f4f7334b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f6c15c2b0 0x7f4f6c158360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:14.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.464+0000 7f4f7334b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f6c15f030 0x7f4f6c0aa6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:14.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.464+0000 7f4f7334b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6c0b7730 con 0x7f4f6c15c2b0 2024-09-16T01:36:14.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.464+0000 7f4f7334b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6c06d3f0 con 0x7f4f6c15f030 2024-09-16T01:36:14.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.464+0000 7f4f7334b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4f6c15f4c0 con 0x7f4f6c0b72f0 2024-09-16T01:36:14.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.464+0000 7f4f708e6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f6c15c2b0 0x7f4f6c158360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:14.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.464+0000 7f4f708e6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f6c15c2b0 0x7f4f6c158360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48720/0 (socket says 172.21.15.88:48720) 2024-09-16T01:36:14.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.464+0000 7f4f708e6640 1 -- 172.21.15.88:0/3491451668 learned_addr learned my addr 172.21.15.88:0/3491451668 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:14.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.464+0000 7f4f710e7640 1 --2- 172.21.15.88:0/3491451668 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f6c0b72f0 0x7f4f6c149ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:14.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.464+0000 7f4f718e8640 1 --2- 172.21.15.88:0/3491451668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f6c15f030 0x7f4f6c0aa6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:14.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.464+0000 7f4f708e6640 1 -- 172.21.15.88:0/3491451668 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f6c0b72f0 msgr2=0x7f4f6c149ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:14.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.464+0000 7f4f708e6640 1 --2- 172.21.15.88:0/3491451668 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f6c0b72f0 0x7f4f6c149ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:14.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.464+0000 7f4f708e6640 1 -- 172.21.15.88:0/3491451668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f6c15f030 msgr2=0x7f4f6c0aa6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:14.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.464+0000 7f4f708e6640 1 --2- 172.21.15.88:0/3491451668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f6c15f030 0x7f4f6c0aa6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:14.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.464+0000 7f4f708e6640 1 -- 172.21.15.88:0/3491451668 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f6c14a550 con 0x7f4f6c15c2b0 2024-09-16T01:36:14.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.464+0000 7f4f708e6640 1 --2- 172.21.15.88:0/3491451668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f6c15c2b0 0x7f4f6c158360 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c00d800 tx=0x7f4f5c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:14.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.465+0000 7f4f61ffb640 1 -- 172.21.15.88:0/3491451668 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c00b6c0 con 0x7f4f6c15c2b0 2024-09-16T01:36:14.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.465+0000 7f4f61ffb640 1 -- 172.21.15.88:0/3491451668 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f5c019070 con 0x7f4f6c15c2b0 2024-09-16T01:36:14.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.465+0000 7f4f7334b640 1 -- 172.21.15.88:0/3491451668 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f6c14a830 con 0x7f4f6c15c2b0 2024-09-16T01:36:14.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.465+0000 7f4f61ffb640 1 -- 172.21.15.88:0/3491451668 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c00ba80 con 0x7f4f6c15c2b0 2024-09-16T01:36:14.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.465+0000 7f4f7334b640 1 -- 172.21.15.88:0/3491451668 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f6c14adf0 con 0x7f4f6c15c2b0 2024-09-16T01:36:14.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.466+0000 7f4f7334b640 1 -- 172.21.15.88:0/3491451668 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f6c15eaf0 con 0x7f4f6c15c2b0 2024-09-16T01:36:14.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.466+0000 7f4f61ffb640 1 -- 172.21.15.88:0/3491451668 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4f5c00bc20 con 0x7f4f6c15c2b0 2024-09-16T01:36:14.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.467+0000 7f4f61ffb640 1 --2- 172.21.15.88:0/3491451668 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f5006c2b0 0x7f4f5006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:14.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.467+0000 7f4f61ffb640 1 -- 172.21.15.88:0/3491451668 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f4f5c08b7b0 con 0x7f4f6c15c2b0 2024-09-16T01:36:14.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.467+0000 7f4f710e7640 1 --2- 172.21.15.88:0/3491451668 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f5006c2b0 0x7f4f5006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:14.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.467+0000 7f4f710e7640 1 --2- 172.21.15.88:0/3491451668 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f5006c2b0 0x7f4f5006e770 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c158b70 tx=0x7f4f680098d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:14.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.471+0000 7f4f61ffb640 1 -- 172.21.15.88:0/3491451668 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4f5c0598d0 con 0x7f4f6c15c2b0 2024-09-16T01:36:14.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.670+0000 7f4f7334b640 1 -- 172.21.15.88:0/3491451668 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f6c15d680 con 0x7f4f6c15c2b0 2024-09-16T01:36:14.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.671+0000 7f4f61ffb640 1 -- 172.21.15.88:0/3491451668 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5226 (secure 0 0 0) 0x7f4f5c05dbd0 con 0x7f4f6c15c2b0 2024-09-16T01:36:14.673 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:14.673 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":25,"btime":"2024-09-16T01:36:08:365397+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4877,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1769276566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":1769276566},{"type":"v1","addr":"172.21.15.154:6835","nonce":1769276566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/725621378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":725621378},{"type":"v1","addr":"172.21.15.154:6834","nonce":725621378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4199111872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4199111872},{"type":"v1","addr":"172.21.15.88:6837","nonce":4199111872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:05.691540+0000","modified":"2024-09-16T01:36:07.693602+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5406},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5406":{"gid":5406,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6836/1610810484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1610810484},{"type":"v1","addr":"172.21.15.88:6836","nonce":1610810484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5406,"qdb_cluster":[5406]},"id":3}]} 2024-09-16T01:36:14.673 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 25 2024-09-16T01:36:14.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.673+0000 7f4f7334b640 1 -- 172.21.15.88:0/3491451668 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f5006c2b0 msgr2=0x7f4f5006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:14.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.673+0000 7f4f7334b640 1 --2- 172.21.15.88:0/3491451668 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f5006c2b0 0x7f4f5006e770 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c158b70 tx=0x7f4f680098d0 comp rx=0 tx=0).stop 2024-09-16T01:36:14.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.673+0000 7f4f7334b640 1 -- 172.21.15.88:0/3491451668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f6c15c2b0 msgr2=0x7f4f6c158360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:14.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.673+0000 7f4f7334b640 1 --2- 172.21.15.88:0/3491451668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f6c15c2b0 0x7f4f6c158360 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c00d800 tx=0x7f4f5c00dcc0 comp rx=0 tx=0).stop 2024-09-16T01:36:14.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.673+0000 7f4f7334b640 1 -- 172.21.15.88:0/3491451668 shutdown_connections 2024-09-16T01:36:14.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.673+0000 7f4f7334b640 1 --2- 172.21.15.88:0/3491451668 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f5006c2b0 0x7f4f5006e770 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:14.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.673+0000 7f4f7334b640 1 --2- 172.21.15.88:0/3491451668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f6c15f030 0x7f4f6c0aa6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:14.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.673+0000 7f4f7334b640 1 --2- 172.21.15.88:0/3491451668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f6c15c2b0 0x7f4f6c158360 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:14.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.673+0000 7f4f7334b640 1 --2- 172.21.15.88:0/3491451668 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f6c0b72f0 0x7f4f6c149ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:14.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.673+0000 7f4f7334b640 1 -- 172.21.15.88:0/3491451668 >> 172.21.15.88:0/3491451668 conn(0x7f4f6c0b4240 msgr2=0x7f4f6c15cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:14.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.673+0000 7f4f7334b640 1 -- 172.21.15.88:0/3491451668 shutdown_connections 2024-09-16T01:36:14.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:14.673+0000 7f4f7334b640 1 -- 172.21.15.88:0/3491451668 wait complete. 2024-09-16T01:36:15.088 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:36:05.691540+0000', 'modified': '2024-09-16T01:36:07.693602+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5406}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5406': {'gid': 5406, 'name': 'c', 'rank': 0, 'incarnation': 23, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.88:6836/1610810484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 1610810484}, {'type': 'v1', 'addr': '172.21.15.88:6836', 'nonce': 1610810484}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5406, 'qdb_cluster': [5406]} 2024-09-16T01:36:15.088 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T01:36:15.088 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-16T01:36:15.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.186+0000 7f8cf5e80640 1 Processor -- start 2024-09-16T01:36:15.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.186+0000 7f8cf5e80640 1 -- start start 2024-09-16T01:36:15.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.186+0000 7f8cf5e80640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8cf00b68f0 0x7f8cf01696f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:15.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.186+0000 7f8cf5e80640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cf00cf860 con 0x7f8cf015e7a0 2024-09-16T01:36:15.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.186+0000 7f8cf5e80640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cf00cfb40 con 0x7f8cf00cf1f0 2024-09-16T01:36:15.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.187+0000 7f8cf5e80640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f8cf00d2ca0 con 0x7f8cf00b68f0 2024-09-16T01:36:15.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.187+0000 7f8ceffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8cf00b68f0 0x7f8cf01696f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:15.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.187+0000 7f8ceffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8cf00b68f0 0x7f8cf01696f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48734/0 (socket says 172.21.15.88:48734) 2024-09-16T01:36:15.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.187+0000 7f8cef7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8cf015e7a0 0x7f8cf006caa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:51588/0 (socket says 172.21.15.88:51588) 2024-09-16T01:36:15.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.187+0000 7f8ceffff640 1 -- 172.21.15.88:0/1430531581 learned_addr learned my addr 172.21.15.88:0/1430531581 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:15.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.187+0000 7f8ceffff640 1 -- 172.21.15.88:0/1430531581 >> v1:172.21.15.154:6789/0 conn(0x7f8cf00cf1f0 legacy=0x7f8cf00b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:15.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.187+0000 7f8ceffff640 1 -- 172.21.15.88:0/1430531581 >> v1:172.21.15.88:6789/0 conn(0x7f8cf015e7a0 legacy=0x7f8cf006caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:15.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.187+0000 7f8ceffff640 1 -- 172.21.15.88:0/1430531581 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cf00cfe20 con 0x7f8cf00b68f0 2024-09-16T01:36:15.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.187+0000 7f8cee7fc640 1 -- 172.21.15.88:0/1430531581 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 699088179 0 0) 0x7f8cf00cf860 con 0x7f8cf015e7a0 2024-09-16T01:36:15.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.188+0000 7f8ceffff640 1 --2- 172.21.15.88:0/1430531581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cf00b68f0 0x7f8cf01696f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8009e50 tx=0x7f8ce802a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bd1e65b938e1932 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:15.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.188+0000 7f8cee7fc640 1 -- 172.21.15.88:0/1430531581 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8036070 con 0x7f8cf00b68f0 2024-09-16T01:36:15.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.188+0000 7f8cee7fc640 1 -- 172.21.15.88:0/1430531581 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ce8031600 con 0x7f8cf00b68f0 2024-09-16T01:36:15.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.188+0000 7f8cee7fc640 1 -- 172.21.15.88:0/1430531581 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce80319d0 con 0x7f8cf00b68f0 2024-09-16T01:36:15.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.188+0000 7f8cf5e80640 1 -- 172.21.15.88:0/1430531581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cf00b68f0 msgr2=0x7f8cf01696f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:15.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.188+0000 7f8cf5e80640 1 --2- 172.21.15.88:0/1430531581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cf00b68f0 0x7f8cf01696f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8009e50 tx=0x7f8ce802a6d0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.188+0000 7f8cf5e80640 1 -- 172.21.15.88:0/1430531581 shutdown_connections 2024-09-16T01:36:15.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.188+0000 7f8cf5e80640 1 --2- 172.21.15.88:0/1430531581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cf00b68f0 0x7f8cf01696f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.188+0000 7f8cf5e80640 1 -- 172.21.15.88:0/1430531581 >> 172.21.15.88:0/1430531581 conn(0x7f8cf00d11e0 msgr2=0x7f8cf00d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:15.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.188+0000 7f8cf5e80640 1 -- 172.21.15.88:0/1430531581 shutdown_connections 2024-09-16T01:36:15.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.188+0000 7f8cf5e80640 1 -- 172.21.15.88:0/1430531581 wait complete. 2024-09-16T01:36:15.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.190+0000 7f8cf5e80640 1 Processor -- start 2024-09-16T01:36:15.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.190+0000 7f8cf5e80640 1 -- start start 2024-09-16T01:36:15.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.190+0000 7f8cf5e80640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cf00b68f0 0x7f8cf0160f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:15.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.191+0000 7f8cf5e80640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cf00cf1f0 0x7f8cf0151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:15.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.191+0000 7f8cf5e80640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cf015e7a0 0x7f8cf015f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:15.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.191+0000 7f8cf5e80640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf00d4420 con 0x7f8cf015e7a0 2024-09-16T01:36:15.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.191+0000 7f8cf5e80640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf00b7a00 con 0x7f8cf00cf1f0 2024-09-16T01:36:15.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.191+0000 7f8cf5e80640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8cf015fe80 con 0x7f8cf00b68f0 2024-09-16T01:36:15.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.191+0000 7f8ceffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cf015e7a0 0x7f8cf015f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:15.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.191+0000 7f8ceffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cf015e7a0 0x7f8cf015f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48736/0 (socket says 172.21.15.88:48736) 2024-09-16T01:36:15.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.191+0000 7f8ceffff640 1 -- 172.21.15.88:0/3421372586 learned_addr learned my addr 172.21.15.88:0/3421372586 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:15.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.191+0000 7f8cef7fe640 1 --2- 172.21.15.88:0/3421372586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cf00b68f0 0x7f8cf0160f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:15.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.191+0000 7f8ceeffd640 1 --2- 172.21.15.88:0/3421372586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cf00cf1f0 0x7f8cf0151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:15.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.191+0000 7f8ceffff640 1 -- 172.21.15.88:0/3421372586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cf00b68f0 msgr2=0x7f8cf0160f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:15.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.191+0000 7f8ceffff640 1 --2- 172.21.15.88:0/3421372586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cf00b68f0 0x7f8cf0160f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.191+0000 7f8ceffff640 1 -- 172.21.15.88:0/3421372586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cf00cf1f0 msgr2=0x7f8cf0151cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:15.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.191+0000 7f8ceffff640 1 --2- 172.21.15.88:0/3421372586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cf00cf1f0 0x7f8cf0151cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.191+0000 7f8ceffff640 1 -- 172.21.15.88:0/3421372586 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cf0160160 con 0x7f8cf015e7a0 2024-09-16T01:36:15.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.191+0000 7f8ceffff640 1 --2- 172.21.15.88:0/3421372586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cf015e7a0 0x7f8cf015f750 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8009e50 tx=0x7f8ce8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:15.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.191+0000 7f8cd7fff640 1 -- 172.21.15.88:0/3421372586 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8041070 con 0x7f8cf015e7a0 2024-09-16T01:36:15.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.191+0000 7f8cd7fff640 1 -- 172.21.15.88:0/3421372586 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ce8003990 con 0x7f8cf015e7a0 2024-09-16T01:36:15.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.191+0000 7f8cf5e80640 1 -- 172.21.15.88:0/3421372586 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cf0160440 con 0x7f8cf015e7a0 2024-09-16T01:36:15.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.191+0000 7f8cd7fff640 1 -- 172.21.15.88:0/3421372586 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8036040 con 0x7f8cf015e7a0 2024-09-16T01:36:15.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.191+0000 7f8cf5e80640 1 -- 172.21.15.88:0/3421372586 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cf0160a00 con 0x7f8cf015e7a0 2024-09-16T01:36:15.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.193+0000 7f8cd7fff640 1 -- 172.21.15.88:0/3421372586 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8ce8007800 con 0x7f8cf015e7a0 2024-09-16T01:36:15.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.193+0000 7f8cd7fff640 1 --2- 172.21.15.88:0/3421372586 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8cdc06c2b0 0x7f8cdc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:15.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.193+0000 7f8cd7fff640 1 -- 172.21.15.88:0/3421372586 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f8ce80ac0e0 con 0x7f8cf015e7a0 2024-09-16T01:36:15.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.193+0000 7f8cef7fe640 1 --2- 172.21.15.88:0/3421372586 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8cdc06c2b0 0x7f8cdc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:15.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.194+0000 7f8cef7fe640 1 --2- 172.21.15.88:0/3421372586 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8cdc06c2b0 0x7f8cdc06e770 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0000f60 tx=0x7f8ce0007c00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:15.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.194+0000 7f8cf8c14740 1 Processor -- start 2024-09-16T01:36:15.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.194+0000 7f8cf8c14740 1 -- start start 2024-09-16T01:36:15.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.195+0000 7f8cf8c14740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55683fba52c0 0x55683fbeaa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:15.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.195+0000 7f8cf8c14740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55683fba2a80 0x55683fba56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:15.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.195+0000 7f8cf8c14740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fba6a50 0x55683fb9f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:15.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.195+0000 7f8cf8c14740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55683fb07520 con 0x55683fba6a50 2024-09-16T01:36:15.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.195+0000 7f8cf8c14740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55683faaf3e0 con 0x55683fba52c0 2024-09-16T01:36:15.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.195+0000 7f8cf8c14740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55683fb034e0 con 0x55683fba2a80 2024-09-16T01:36:15.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.195+0000 7f8ceffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fba6a50 0x55683fb9f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:15.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.195+0000 7f8ceffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fba6a50 0x55683fb9f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48744/0 (socket says 172.21.15.88:48744) 2024-09-16T01:36:15.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.195+0000 7f8ceffff640 1 -- 172.21.15.88:0/1033299587 learned_addr learned my addr 172.21.15.88:0/1033299587 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:15.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.195+0000 7f8cef7fe640 1 --2- 172.21.15.88:0/1033299587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55683fba52c0 0x55683fbeaa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:15.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.195+0000 7f8ceeffd640 1 --2- 172.21.15.88:0/1033299587 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55683fba2a80 0x55683fba56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:15.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.195+0000 7f8ceffff640 1 -- 172.21.15.88:0/1033299587 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55683fba2a80 msgr2=0x55683fba56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:15.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.195+0000 7f8ceffff640 1 --2- 172.21.15.88:0/1033299587 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55683fba2a80 0x55683fba56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.195+0000 7f8ceffff640 1 -- 172.21.15.88:0/1033299587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55683fba52c0 msgr2=0x55683fbeaa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:15.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.195+0000 7f8ceffff640 1 --2- 172.21.15.88:0/1033299587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55683fba52c0 0x55683fbeaa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.195+0000 7f8ceffff640 1 -- 172.21.15.88:0/1033299587 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55683faca2f0 con 0x55683fba6a50 2024-09-16T01:36:15.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.195+0000 7f8ceeffd640 1 --2- 172.21.15.88:0/1033299587 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55683fba2a80 0x55683fba56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:15.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.196+0000 7f8ceffff640 1 --2- 172.21.15.88:0/1033299587 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fba6a50 0x55683fb9f8b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f8ce80765d0 tx=0x7f8ce8031740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bcc0d2073b5d83f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:15.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.196+0000 7f8cd57fa640 1 -- 172.21.15.88:0/1033299587 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce80a7070 con 0x55683fba6a50 2024-09-16T01:36:15.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.196+0000 7f8cd57fa640 1 -- 172.21.15.88:0/1033299587 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ce807a170 con 0x55683fba6a50 2024-09-16T01:36:15.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.196+0000 7f8cd57fa640 1 -- 172.21.15.88:0/1033299587 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce807e520 con 0x55683fba6a50 2024-09-16T01:36:15.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.196+0000 7f8cf8c14740 1 -- 172.21.15.88:0/1033299587 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fba6a50 msgr2=0x55683fb9f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:15.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.196+0000 7f8cf8c14740 1 --2- 172.21.15.88:0/1033299587 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fba6a50 0x55683fb9f8b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f8ce80765d0 tx=0x7f8ce8031740 comp rx=0 tx=0).stop 2024-09-16T01:36:15.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.196+0000 7f8cf8c14740 1 -- 172.21.15.88:0/1033299587 shutdown_connections 2024-09-16T01:36:15.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.196+0000 7f8cf8c14740 1 --2- 172.21.15.88:0/1033299587 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fba6a50 0x55683fb9f8b0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.196+0000 7f8cf8c14740 1 --2- 172.21.15.88:0/1033299587 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55683fba2a80 0x55683fba56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.196+0000 7f8cf8c14740 1 --2- 172.21.15.88:0/1033299587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55683fba52c0 0x55683fbeaa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.196+0000 7f8cf8c14740 1 -- 172.21.15.88:0/1033299587 >> 172.21.15.88:0/1033299587 conn(0x55683fba1a80 msgr2=0x55683fbe9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:15.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.196+0000 7f8cf8c14740 1 -- 172.21.15.88:0/1033299587 shutdown_connections 2024-09-16T01:36:15.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.197+0000 7f8cf8c14740 1 -- 172.21.15.88:0/1033299587 wait complete. 2024-09-16T01:36:15.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.197+0000 7f8cf8c14740 1 -- start start 2024-09-16T01:36:15.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.197+0000 7f8cf8c14740 1 Processor -- start 2024-09-16T01:36:15.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.198+0000 7f8cf8c14740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55683fba52c0 0x55683fbccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:15.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.198+0000 7f8cf8c14740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fba6a50 0x55683fbcd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:15.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.198+0000 7f8cf8c14740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55683fbcec30 0x55683fb9e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:15.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.198+0000 7f8cf8c14740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55683fba0f20 con 0x55683fba6a50 2024-09-16T01:36:15.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.198+0000 7f8cf8c14740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55683fa90450 con 0x55683fbcec30 2024-09-16T01:36:15.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.198+0000 7f8cf8c14740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55683fbeb290 con 0x55683fba52c0 2024-09-16T01:36:15.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.198+0000 7f8cf8c14740 10 client.0 _ll_register_callbacks cb 0x55683f819ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:36:15.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.198+0000 7f8ceeffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fba6a50 0x55683fbcd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:15.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.198+0000 7f8ceeffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fba6a50 0x55683fbcd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48748/0 (socket says 172.21.15.88:48748) 2024-09-16T01:36:15.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.198+0000 7f8ceeffd640 1 -- 172.21.15.88:0/3872077410 learned_addr learned my addr 172.21.15.88:0/3872077410 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:15.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.198+0000 7f8ceffff640 1 --2- 172.21.15.88:0/3872077410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55683fbcec30 0x55683fb9e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:15.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.198+0000 7f8cef7fe640 1 --2- 172.21.15.88:0/3872077410 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55683fba52c0 0x55683fbccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:15.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.198+0000 7f8ceeffd640 1 -- 172.21.15.88:0/3872077410 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55683fba52c0 msgr2=0x55683fbccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:15.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.198+0000 7f8ceeffd640 1 --2- 172.21.15.88:0/3872077410 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55683fba52c0 0x55683fbccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.198+0000 7f8ceeffd640 1 -- 172.21.15.88:0/3872077410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55683fbcec30 msgr2=0x55683fb9e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:15.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.198+0000 7f8ceeffd640 1 --2- 172.21.15.88:0/3872077410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55683fbcec30 0x55683fb9e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.198+0000 7f8ceeffd640 1 -- 172.21.15.88:0/3872077410 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55683f81a2d0 con 0x55683fba6a50 2024-09-16T01:36:15.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.198+0000 7f8ceffff640 1 --2- 172.21.15.88:0/3872077410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55683fbcec30 0x55683fb9e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:15.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.198+0000 7f8cef7fe640 1 --2- 172.21.15.88:0/3872077410 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55683fba52c0 0x55683fbccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:15.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.198+0000 7f8ceeffd640 1 --2- 172.21.15.88:0/3872077410 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fba6a50 0x55683fbcd1e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8ce40054c0 tx=0x7f8ce40105b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcc27ba95582ee6e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:15.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.198+0000 7f8cce7fc640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:36:15.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.199+0000 7f8cce7fc640 1 -- 172.21.15.88:0/3872077410 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce4019020 con 0x55683fba6a50 2024-09-16T01:36:15.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.199+0000 7f8cce7fc640 1 -- 172.21.15.88:0/3872077410 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f8ce4015660 con 0x55683fba6a50 2024-09-16T01:36:15.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.199+0000 7f8cf8c14740 1 -- 172.21.15.88:0/3872077410 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ce807a170 con 0x55683fba6a50 2024-09-16T01:36:15.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.199+0000 7f8cce7fc640 1 -- 172.21.15.88:0/3872077410 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8ce4023050 con 0x55683fba6a50 2024-09-16T01:36:15.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.199+0000 7f8cf8c14740 10 client.5565 fetch_fsmap learned FSMap version 25 2024-09-16T01:36:15.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.199+0000 7f8cf8c14740 1 -- 172.21.15.88:0/3872077410 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x55683f8e46b0 con 0x55683fba6a50 2024-09-16T01:36:15.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.199+0000 7f8cce7fc640 1 -- 172.21.15.88:0/3872077410 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ce807a170 con 0x55683fba6a50 2024-09-16T01:36:15.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.199+0000 7f8cce7fc640 1 -- 172.21.15.88:0/3872077410 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 25) ==== 3502+0+0 (secure 0 0 0) 0x7f8ce4026020 con 0x55683fba6a50 2024-09-16T01:36:15.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.199+0000 7f8cf8c14740 10 client.5565 fetch_fsmap finished waiting for FSMap version 25 2024-09-16T01:36:15.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.199+0000 7f8cf8c14740 10 client.5565 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T01:36:15.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.199+0000 7f8cf8c14740 20 client.5565 populate_metadata read hostname 'smithi088' 2024-09-16T01:36:15.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.199+0000 7f8cf8c14740 1 --2- 172.21.15.88:0/3872077410 >> [v2:172.21.15.88:6834/1610810484,v1:172.21.15.88:6836/1610810484] conn(0x55683fbc5eb0 0x55683fbc6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:15.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.199+0000 7f8cf8c14740 4 client.5565 mds_command: new command op to 5406 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:36:15.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.199+0000 7f8cf8c14740 1 -- 172.21.15.88:0/3872077410 --> [v2:172.21.15.88:6834/1610810484,v1:172.21.15.88:6836/1610810484] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55683fb14c50 con 0x55683fbc5eb0 2024-09-16T01:36:15.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.199+0000 7f8cef7fe640 1 --2- 172.21.15.88:0/3872077410 >> [v2:172.21.15.88:6834/1610810484,v1:172.21.15.88:6836/1610810484] conn(0x55683fbc5eb0 0x55683fbc6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:15.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.199+0000 7f8cef7fe640 1 --2- 172.21.15.88:0/3872077410 >> [v2:172.21.15.88:6834/1610810484,v1:172.21.15.88:6836/1610810484] conn(0x55683fbc5eb0 0x55683fbc6290 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f021a0169ea79274 server_cookie=7ef2d42874c3d7ab in_seq=0 out_seq=0 2024-09-16T01:36:15.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.199+0000 7f8cce7fc640 10 client.5565 ms_handle_connect on v2:172.21.15.88:6834/1610810484 2024-09-16T01:36:15.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.200+0000 7f8cce7fc640 1 -- 172.21.15.88:0/3872077410 <== mds.0 v2:172.21.15.88:6834/1610810484 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55683fb14c50 con 0x55683fbc5eb0 2024-09-16T01:36:15.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.200+0000 7f8cce7fc640 10 client.5565 handle_command_reply: tid=0 2024-09-16T01:36:15.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.200+0000 7f8cf8c14740 1 client.5565 shutdown 2024-09-16T01:36:15.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.200+0000 7f8cf8c14740 1 -- 172.21.15.88:0/3872077410 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fba6a50 msgr2=0x55683fbcd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:15.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.200+0000 7f8cf8c14740 1 --2- 172.21.15.88:0/3872077410 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fba6a50 0x55683fbcd1e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8ce40054c0 tx=0x7f8ce40105b0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.200+0000 7f8cf8c14740 1 -- 172.21.15.88:0/3872077410 shutdown_connections 2024-09-16T01:36:15.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.200+0000 7f8cf8c14740 1 --2- 172.21.15.88:0/3872077410 >> [v2:172.21.15.88:6834/1610810484,v1:172.21.15.88:6836/1610810484] conn(0x55683fbc5eb0 0x55683fbc6290 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.200+0000 7f8cf8c14740 1 --2- 172.21.15.88:0/3872077410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55683fbcec30 0x55683fb9e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.200+0000 7f8cce7fc640 0 client.5565 ms_handle_reset on v2:172.21.15.88:6834/1610810484 2024-09-16T01:36:15.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.200+0000 7f8cf8c14740 1 --2- 172.21.15.88:0/3872077410 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fba6a50 0x55683fbcd1e0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.200+0000 7f8cf8c14740 1 --2- 172.21.15.88:0/3872077410 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55683fba52c0 0x55683fbccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.200+0000 7f8cf8c14740 1 -- 172.21.15.88:0/3872077410 >> 172.21.15.88:0/3872077410 conn(0x55683fba2a80 msgr2=0x55683fbe8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:15.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.200+0000 7f8cf8c14740 1 -- 172.21.15.88:0/3872077410 shutdown_connections 2024-09-16T01:36:15.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.200+0000 7f8cf8c14740 1 -- 172.21.15.88:0/3872077410 wait complete. 2024-09-16T01:36:15.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.201+0000 7f8cf8c14740 20 client.5565 trim_cache size 0 max 16384 2024-09-16T01:36:15.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.218+0000 7f8cf8c14740 1 Processor -- start 2024-09-16T01:36:15.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.218+0000 7f8cf8c14740 1 -- start start 2024-09-16T01:36:15.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.219+0000 7f8cf8c14740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55683fba52c0 0x55683fbe74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:15.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.219+0000 7f8cf8c14740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55683fba6a50 0x55683fc0dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:15.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.219+0000 7f8cf8c14740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fbc5eb0 0x55683fb9f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:15.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.219+0000 7f8cf8c14740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf01ffab0 con 0x55683fbc5eb0 2024-09-16T01:36:15.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.219+0000 7f8cf8c14740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8cf01fee90 con 0x55683fba6a50 2024-09-16T01:36:15.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.219+0000 7f8cf8c14740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8cf01ff460 con 0x55683fba52c0 2024-09-16T01:36:15.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.219+0000 7f8ceffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fbc5eb0 0x55683fb9f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:15.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.219+0000 7f8ceffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fbc5eb0 0x55683fb9f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48764/0 (socket says 172.21.15.88:48764) 2024-09-16T01:36:15.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.219+0000 7f8ceffff640 1 -- 172.21.15.88:0/1316579160 learned_addr learned my addr 172.21.15.88:0/1316579160 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:15.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.219+0000 7f8ceffff640 1 -- 172.21.15.88:0/1316579160 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55683fba52c0 msgr2=0x55683fbe74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:15.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.219+0000 7f8ceffff640 1 --2- 172.21.15.88:0/1316579160 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55683fba52c0 0x55683fbe74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.219+0000 7f8ceeffd640 1 --2- 172.21.15.88:0/1316579160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55683fba6a50 0x55683fc0dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:15.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.219+0000 7f8ceffff640 1 -- 172.21.15.88:0/1316579160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55683fba6a50 msgr2=0x55683fc0dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:15.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.219+0000 7f8ceffff640 1 --2- 172.21.15.88:0/1316579160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55683fba6a50 0x55683fc0dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.219+0000 7f8ceffff640 1 -- 172.21.15.88:0/1316579160 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55683fba4190 con 0x55683fbc5eb0 2024-09-16T01:36:15.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.219+0000 7f8ceeffd640 1 --2- 172.21.15.88:0/1316579160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55683fba6a50 0x55683fc0dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:15.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.220+0000 7f8ceffff640 1 --2- 172.21.15.88:0/1316579160 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fbc5eb0 0x55683fb9f9d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8079e20 tx=0x7f8ce807d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e32dc5e017913a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:15.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.220+0000 7f8cd4ff9640 1 -- 172.21.15.88:0/1316579160 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce8099070 con 0x55683fbc5eb0 2024-09-16T01:36:15.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.220+0000 7f8cd4ff9640 1 -- 172.21.15.88:0/1316579160 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ce8082d50 con 0x55683fbc5eb0 2024-09-16T01:36:15.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.220+0000 7f8cd4ff9640 1 -- 172.21.15.88:0/1316579160 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce80a7070 con 0x55683fbc5eb0 2024-09-16T01:36:15.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.220+0000 7f8cf8c14740 1 -- 172.21.15.88:0/1316579160 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fbc5eb0 msgr2=0x55683fb9f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:15.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.220+0000 7f8cf8c14740 1 --2- 172.21.15.88:0/1316579160 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fbc5eb0 0x55683fb9f9d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8079e20 tx=0x7f8ce807d000 comp rx=0 tx=0).stop 2024-09-16T01:36:15.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.220+0000 7f8cf8c14740 1 -- 172.21.15.88:0/1316579160 shutdown_connections 2024-09-16T01:36:15.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.220+0000 7f8cf8c14740 1 --2- 172.21.15.88:0/1316579160 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fbc5eb0 0x55683fb9f9d0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.220+0000 7f8cf8c14740 1 --2- 172.21.15.88:0/1316579160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55683fba6a50 0x55683fc0dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.220+0000 7f8cf8c14740 1 --2- 172.21.15.88:0/1316579160 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55683fba52c0 0x55683fbe74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.220+0000 7f8cf8c14740 1 -- 172.21.15.88:0/1316579160 >> 172.21.15.88:0/1316579160 conn(0x55683fba2a80 msgr2=0x55683fbce4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:15.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.220+0000 7f8cf8c14740 1 -- 172.21.15.88:0/1316579160 shutdown_connections 2024-09-16T01:36:15.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.220+0000 7f8cf8c14740 1 -- 172.21.15.88:0/1316579160 wait complete. 2024-09-16T01:36:15.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.221+0000 7f8cf8c14740 1 -- start start 2024-09-16T01:36:15.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.221+0000 7f8cf8c14740 1 Processor -- start 2024-09-16T01:36:15.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.222+0000 7f8cf8c14740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55683fba52c0 0x55683fbe6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:15.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.222+0000 7f8cf8c14740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fba6a50 0x55683fc4e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:15.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.222+0000 7f8cf8c14740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55683fbc5eb0 0x55683fb9f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:15.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.222+0000 7f8cf8c14740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55683fbce350 con 0x55683fba6a50 2024-09-16T01:36:15.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.222+0000 7f8cf8c14740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55683fb9f610 con 0x55683fbc5eb0 2024-09-16T01:36:15.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.222+0000 7f8cf8c14740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55683fb9ed10 con 0x55683fba52c0 2024-09-16T01:36:15.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.222+0000 7f8cf8c14740 10 client.0 _ll_register_callbacks cb 0x7f8ce4007f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:36:15.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.222+0000 7f8ceeffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fba6a50 0x55683fc4e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:15.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.222+0000 7f8ceeffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fba6a50 0x55683fc4e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48772/0 (socket says 172.21.15.88:48772) 2024-09-16T01:36:15.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.222+0000 7f8ceeffd640 1 -- 172.21.15.88:0/3883921089 learned_addr learned my addr 172.21.15.88:0/3883921089 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:15.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.222+0000 7f8cef7fe640 1 --2- 172.21.15.88:0/3883921089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55683fba52c0 0x55683fbe6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:15.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.222+0000 7f8ceffff640 1 --2- 172.21.15.88:0/3883921089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55683fbc5eb0 0x55683fb9f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:15.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.222+0000 7f8ceeffd640 1 -- 172.21.15.88:0/3883921089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55683fba52c0 msgr2=0x55683fbe6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:15.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.222+0000 7f8ceeffd640 1 --2- 172.21.15.88:0/3883921089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55683fba52c0 0x55683fbe6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.222+0000 7f8ceeffd640 1 -- 172.21.15.88:0/3883921089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55683fbc5eb0 msgr2=0x55683fb9f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:15.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.222+0000 7f8ceeffd640 1 --2- 172.21.15.88:0/3883921089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55683fbc5eb0 0x55683fb9f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.222+0000 7f8ceeffd640 1 -- 172.21.15.88:0/3883921089 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55683fbc6290 con 0x55683fba6a50 2024-09-16T01:36:15.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.222+0000 7f8ceeffd640 1 --2- 172.21.15.88:0/3883921089 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fba6a50 0x55683fc4e1b0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f8ce4009010 tx=0x7f8ce4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81b096d05a3a58b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:15.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.222+0000 7f8cce7fc640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:36:15.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.222+0000 7f8cce7fc640 1 -- 172.21.15.88:0/3883921089 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ce4019030 con 0x55683fba6a50 2024-09-16T01:36:15.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.222+0000 7f8cce7fc640 1 -- 172.21.15.88:0/3883921089 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f8ce40273f0 con 0x55683fba6a50 2024-09-16T01:36:15.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.222+0000 7f8cf8c14740 1 -- 172.21.15.88:0/3883921089 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ce8082d50 con 0x55683fba6a50 2024-09-16T01:36:15.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.223+0000 7f8cce7fc640 1 -- 172.21.15.88:0/3883921089 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8ce4010ab0 con 0x55683fba6a50 2024-09-16T01:36:15.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.223+0000 7f8cf8c14740 10 client.5571 fetch_fsmap learned FSMap version 25 2024-09-16T01:36:15.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.223+0000 7f8cf8c14740 1 -- 172.21.15.88:0/3883921089 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x55683fbc6f60 con 0x55683fba6a50 2024-09-16T01:36:15.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.223+0000 7f8cce7fc640 1 -- 172.21.15.88:0/3883921089 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ce8082d50 con 0x55683fba6a50 2024-09-16T01:36:15.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.223+0000 7f8cce7fc640 1 -- 172.21.15.88:0/3883921089 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 25) ==== 3502+0+0 (secure 0 0 0) 0x7f8ce4026040 con 0x55683fba6a50 2024-09-16T01:36:15.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.223+0000 7f8cf8c14740 10 client.5571 fetch_fsmap finished waiting for FSMap version 25 2024-09-16T01:36:15.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.223+0000 7f8cf8c14740 10 client.5571 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T01:36:15.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.223+0000 7f8cf8c14740 20 client.5571 populate_metadata read hostname 'smithi088' 2024-09-16T01:36:15.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.223+0000 7f8cf8c14740 1 --2- 172.21.15.88:0/3883921089 >> [v2:172.21.15.88:6834/1610810484,v1:172.21.15.88:6836/1610810484] conn(0x55683fbcec30 0x55683fbfcb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:15.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.223+0000 7f8cf8c14740 4 client.5571 mds_command: new command op to 5406 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:36:15.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.223+0000 7f8cf8c14740 1 -- 172.21.15.88:0/3883921089 --> [v2:172.21.15.88:6834/1610810484,v1:172.21.15.88:6836/1610810484] -- command(tid 0: {"prefix": "status"}) -- 0x55683fba43f0 con 0x55683fbcec30 2024-09-16T01:36:15.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.223+0000 7f8cef7fe640 1 --2- 172.21.15.88:0/3883921089 >> [v2:172.21.15.88:6834/1610810484,v1:172.21.15.88:6836/1610810484] conn(0x55683fbcec30 0x55683fbfcb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:15.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.223+0000 7f8cef7fe640 1 --2- 172.21.15.88:0/3883921089 >> [v2:172.21.15.88:6834/1610810484,v1:172.21.15.88:6836/1610810484] conn(0x55683fbcec30 0x55683fbfcb30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d8954252fc39e6b server_cookie=ad0c0627aca2bc9c in_seq=0 out_seq=0 2024-09-16T01:36:15.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.223+0000 7f8cce7fc640 10 client.5571 ms_handle_connect on v2:172.21.15.88:6834/1610810484 2024-09-16T01:36:15.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.223+0000 7f8cce7fc640 1 -- 172.21.15.88:0/3883921089 <== mds.0 v2:172.21.15.88:6834/1610810484 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55683fba43f0 con 0x55683fbcec30 2024-09-16T01:36:15.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.223+0000 7f8cce7fc640 10 client.5571 handle_command_reply: tid=0 2024-09-16T01:36:15.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.223+0000 7f8cf8c14740 1 client.5571 shutdown 2024-09-16T01:36:15.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.224+0000 7f8cf8c14740 1 -- 172.21.15.88:0/3883921089 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fba6a50 msgr2=0x55683fc4e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:15.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.224+0000 7f8cf8c14740 1 --2- 172.21.15.88:0/3883921089 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fba6a50 0x55683fc4e1b0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f8ce4009010 tx=0x7f8ce4013000 comp rx=0 tx=0).stop 2024-09-16T01:36:15.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.224+0000 7f8cf8c14740 1 -- 172.21.15.88:0/3883921089 shutdown_connections 2024-09-16T01:36:15.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.224+0000 7f8cf8c14740 1 --2- 172.21.15.88:0/3883921089 >> [v2:172.21.15.88:6834/1610810484,v1:172.21.15.88:6836/1610810484] conn(0x55683fbcec30 0x55683fbfcb30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.224+0000 7f8cf8c14740 1 --2- 172.21.15.88:0/3883921089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55683fbc5eb0 0x55683fb9f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.224+0000 7f8cce7fc640 0 client.5571 ms_handle_reset on v2:172.21.15.88:6834/1610810484 2024-09-16T01:36:15.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.224+0000 7f8cf8c14740 1 --2- 172.21.15.88:0/3883921089 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55683fba6a50 0x55683fc4e1b0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.224+0000 7f8cf8c14740 1 --2- 172.21.15.88:0/3883921089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55683fba52c0 0x55683fbe6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.224+0000 7f8cf8c14740 1 -- 172.21.15.88:0/3883921089 >> 172.21.15.88:0/3883921089 conn(0x55683fba2a80 msgr2=0x55683fb9b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:15.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.224+0000 7f8cf8c14740 1 -- 172.21.15.88:0/3883921089 shutdown_connections 2024-09-16T01:36:15.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.224+0000 7f8cf8c14740 1 -- 172.21.15.88:0/3883921089 wait complete. 2024-09-16T01:36:15.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.224+0000 7f8cf8c14740 20 client.5571 trim_cache size 0 max 16384 2024-09-16T01:36:15.234 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:36:15.235 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "5ace2c91-46f9-482b-8a89-731fbdc171bf", 2024-09-16T01:36:15.235 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2024-09-16T01:36:15.235 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 5406, 2024-09-16T01:36:15.235 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2024-09-16T01:36:15.235 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2024-09-16T01:36:15.235 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2024-09-16T01:36:15.235 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 8.8762308480000005, 2024-09-16T01:36:15.235 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 25, 2024-09-16T01:36:15.235 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 45, 2024-09-16T01:36:15.235 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 45, 2024-09-16T01:36:15.235 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 13.216012897000001 2024-09-16T01:36:15.235 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:36:15.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.224+0000 7f8cf5e80640 1 -- 172.21.15.88:0/3421372586 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8cdc06c2b0 msgr2=0x7f8cdc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:15.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.224+0000 7f8cf5e80640 1 --2- 172.21.15.88:0/3421372586 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8cdc06c2b0 0x7f8cdc06e770 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8ce0000f60 tx=0x7f8ce0007c00 comp rx=0 tx=0).stop 2024-09-16T01:36:15.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.224+0000 7f8cf5e80640 1 -- 172.21.15.88:0/3421372586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cf015e7a0 msgr2=0x7f8cf015f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:15.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.224+0000 7f8cf5e80640 1 --2- 172.21.15.88:0/3421372586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cf015e7a0 0x7f8cf015f750 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f8ce8009e50 tx=0x7f8ce8004a40 comp rx=0 tx=0).stop 2024-09-16T01:36:15.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.224+0000 7f8cf5e80640 1 -- 172.21.15.88:0/3421372586 shutdown_connections 2024-09-16T01:36:15.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.224+0000 7f8cf5e80640 1 --2- 172.21.15.88:0/3421372586 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8cdc06c2b0 0x7f8cdc06e770 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.224+0000 7f8cf5e80640 1 --2- 172.21.15.88:0/3421372586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8cf015e7a0 0x7f8cf015f750 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.224+0000 7f8cf5e80640 1 --2- 172.21.15.88:0/3421372586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cf00cf1f0 0x7f8cf0151cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.224+0000 7f8cf5e80640 1 --2- 172.21.15.88:0/3421372586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cf00b68f0 0x7f8cf0160f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.224+0000 7f8cf5e80640 1 -- 172.21.15.88:0/3421372586 >> 172.21.15.88:0/3421372586 conn(0x7f8cf00d11e0 msgr2=0x7f8cf01588a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:15.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.224+0000 7f8cf5e80640 1 -- 172.21.15.88:0/3421372586 shutdown_connections 2024-09-16T01:36:15.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.224+0000 7f8cf5e80640 1 -- 172.21.15.88:0/3421372586 wait complete. 2024-09-16T01:36:15.238 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:36:15.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.334+0000 7f98bed0a640 1 Processor -- start 2024-09-16T01:36:15.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.334+0000 7f98bed0a640 1 -- start start 2024-09-16T01:36:15.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.335+0000 7f98bed0a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f98b815c430 0x7f98b81607f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:15.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.335+0000 7f98bed0a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98b806e170 0x7f98b806e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:15.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.335+0000 7f98bed0a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98b80d0960 con 0x7f98b8062990 2024-09-16T01:36:15.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.335+0000 7f98bed0a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f98b80d0180 con 0x7f98b815c430 2024-09-16T01:36:15.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.335+0000 7f98bed0a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f98b80d03a0 con 0x7f98b806e170 2024-09-16T01:36:15.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.335+0000 7f98bcaa6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f98b815c430 0x7f98b81607f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:15.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.335+0000 7f98b7fff640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f98b8062990 0x7f98b80cfa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:52720/0 (socket says 172.21.15.88:52720) 2024-09-16T01:36:15.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.335+0000 7f98bd2a7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98b806e170 0x7f98b806e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:15.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.335+0000 7f98bcaa6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f98b815c430 0x7f98b81607f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:60248/0 (socket says 172.21.15.88:60248) 2024-09-16T01:36:15.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.335+0000 7f98b7fff640 1 -- 172.21.15.88:0/3420641845 learned_addr learned my addr 172.21.15.88:0/3420641845 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:15.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.335+0000 7f98bcaa6640 1 -- 172.21.15.88:0/3420641845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98b806e170 msgr2=0x7f98b806e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:15.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.335+0000 7f98bcaa6640 1 --2- 172.21.15.88:0/3420641845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98b806e170 0x7f98b806e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.335+0000 7f98bcaa6640 1 -- 172.21.15.88:0/3420641845 >> v1:172.21.15.154:6789/0 conn(0x7f98b8062990 legacy=0x7f98b80cfa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:15.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.335+0000 7f98b77fe640 1 -- 172.21.15.88:0/3420641845 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3136551910 0 0) 0x7f98b80d0960 con 0x7f98b8062990 2024-09-16T01:36:15.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.335+0000 7f98bcaa6640 1 -- 172.21.15.88:0/3420641845 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b80d0680 con 0x7f98b815c430 2024-09-16T01:36:15.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.336+0000 7f98bcaa6640 1 --2- 172.21.15.88:0/3420641845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b815c430 0x7f98b81607f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f98ac0097a0 tx=0x7f98ac02a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef2b1b5931783a30 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:15.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.336+0000 7f98b77fe640 1 -- 172.21.15.88:0/3420641845 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac037070 con 0x7f98b815c430 2024-09-16T01:36:15.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.336+0000 7f98b77fe640 1 -- 172.21.15.88:0/3420641845 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98ac032440 con 0x7f98b815c430 2024-09-16T01:36:15.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.336+0000 7f98b77fe640 1 -- 172.21.15.88:0/3420641845 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac032810 con 0x7f98b815c430 2024-09-16T01:36:15.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.337+0000 7f98bed0a640 1 -- 172.21.15.88:0/3420641845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b815c430 msgr2=0x7f98b81607f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:15.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.337+0000 7f98bed0a640 1 --2- 172.21.15.88:0/3420641845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b815c430 0x7f98b81607f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f98ac0097a0 tx=0x7f98ac02a180 comp rx=0 tx=0).stop 2024-09-16T01:36:15.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.337+0000 7f98bed0a640 1 -- 172.21.15.88:0/3420641845 shutdown_connections 2024-09-16T01:36:15.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.337+0000 7f98bed0a640 1 --2- 172.21.15.88:0/3420641845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98b806e170 0x7f98b806e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.337+0000 7f98bed0a640 1 --2- 172.21.15.88:0/3420641845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b815c430 0x7f98b81607f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.337+0000 7f98bed0a640 1 -- 172.21.15.88:0/3420641845 >> 172.21.15.88:0/3420641845 conn(0x7f98b806bc10 msgr2=0x7f98b806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:15.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.337+0000 7f98bed0a640 1 -- 172.21.15.88:0/3420641845 shutdown_connections 2024-09-16T01:36:15.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.337+0000 7f98bed0a640 1 -- 172.21.15.88:0/3420641845 wait complete. 2024-09-16T01:36:15.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.338+0000 7f98bed0a640 1 Processor -- start 2024-09-16T01:36:15.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.338+0000 7f98bed0a640 1 -- start start 2024-09-16T01:36:15.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.339+0000 7f98bed0a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98b8062990 0x7f98b80aeb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:15.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.339+0000 7f98bed0a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98b806e170 0x7f98b8158bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:15.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.339+0000 7f98bed0a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b815c430 0x7f98b81f4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:15.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.339+0000 7f98bed0a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f98b8160d30 con 0x7f98b8062990 2024-09-16T01:36:15.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.339+0000 7f98bed0a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f98b81602d0 con 0x7f98b815c430 2024-09-16T01:36:15.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.339+0000 7f98bed0a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f98b815c810 con 0x7f98b806e170 2024-09-16T01:36:15.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.339+0000 7f98bcaa6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98b8062990 0x7f98b80aeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:15.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.339+0000 7f98bcaa6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98b8062990 0x7f98b80aeb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48776/0 (socket says 172.21.15.88:48776) 2024-09-16T01:36:15.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.339+0000 7f98bcaa6640 1 -- 172.21.15.88:0/200530658 learned_addr learned my addr 172.21.15.88:0/200530658 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:15.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.340+0000 7f98b7fff640 1 --2- 172.21.15.88:0/200530658 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98b806e170 0x7f98b8158bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:15.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.340+0000 7f98bd2a7640 1 --2- 172.21.15.88:0/200530658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b815c430 0x7f98b81f4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:15.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.340+0000 7f98bcaa6640 1 -- 172.21.15.88:0/200530658 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98b806e170 msgr2=0x7f98b8158bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:15.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.340+0000 7f98bcaa6640 1 --2- 172.21.15.88:0/200530658 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98b806e170 0x7f98b8158bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.340+0000 7f98bcaa6640 1 -- 172.21.15.88:0/200530658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b815c430 msgr2=0x7f98b81f4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:15.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.340+0000 7f98bcaa6640 1 --2- 172.21.15.88:0/200530658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b815c430 0x7f98b81f4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.340+0000 7f98bcaa6640 1 -- 172.21.15.88:0/200530658 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b80727d0 con 0x7f98b8062990 2024-09-16T01:36:15.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.340+0000 7f98b7fff640 1 --2- 172.21.15.88:0/200530658 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98b806e170 0x7f98b8158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:15.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.340+0000 7f98bd2a7640 1 --2- 172.21.15.88:0/200530658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b815c430 0x7f98b81f4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:15.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.340+0000 7f98bcaa6640 1 --2- 172.21.15.88:0/200530658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98b8062990 0x7f98b80aeb00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f98ac035040 tx=0x7f98ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:15.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.340+0000 7f98b57fa640 1 -- 172.21.15.88:0/200530658 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac037040 con 0x7f98b8062990 2024-09-16T01:36:15.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.340+0000 7f98b57fa640 1 -- 172.21.15.88:0/200530658 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98ac030500 con 0x7f98b8062990 2024-09-16T01:36:15.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.340+0000 7f98bed0a640 1 -- 172.21.15.88:0/200530658 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98b8072ab0 con 0x7f98b8062990 2024-09-16T01:36:15.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.340+0000 7f98b57fa640 1 -- 172.21.15.88:0/200530658 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac0308d0 con 0x7f98b8062990 2024-09-16T01:36:15.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.340+0000 7f98bed0a640 1 -- 172.21.15.88:0/200530658 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98b8158690 con 0x7f98b8062990 2024-09-16T01:36:15.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.341+0000 7f98bed0a640 1 -- 172.21.15.88:0/200530658 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98b81607f0 con 0x7f98b8062990 2024-09-16T01:36:15.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.341+0000 7f98b57fa640 1 -- 172.21.15.88:0/200530658 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f98ac044020 con 0x7f98b8062990 2024-09-16T01:36:15.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.342+0000 7f98b57fa640 1 --2- 172.21.15.88:0/200530658 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f989c06c2b0 0x7f989c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:15.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.342+0000 7f98b57fa640 1 -- 172.21.15.88:0/200530658 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f98ac0ab6a0 con 0x7f98b8062990 2024-09-16T01:36:15.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.342+0000 7f98b7fff640 1 --2- 172.21.15.88:0/200530658 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f989c06c2b0 0x7f989c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:15.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.342+0000 7f98b7fff640 1 --2- 172.21.15.88:0/200530658 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f989c06c2b0 0x7f989c06e770 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f98b80739f0 tx=0x7f98a4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:15.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.346+0000 7f98b57fa640 1 -- 172.21.15.88:0/200530658 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f98ac079e10 con 0x7f98b8062990 2024-09-16T01:36:15.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.548+0000 7f98bed0a640 1 -- 172.21.15.88:0/200530658 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f98b80cdfc0 con 0x7f98b8062990 2024-09-16T01:36:15.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.549+0000 7f98b57fa640 1 -- 172.21.15.88:0/200530658 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7f98ac079710 con 0x7f98b8062990 2024-09-16T01:36:15.552 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:36:15.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.551+0000 7f98bed0a640 1 -- 172.21.15.88:0/200530658 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f989c06c2b0 msgr2=0x7f989c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:15.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.551+0000 7f98bed0a640 1 --2- 172.21.15.88:0/200530658 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f989c06c2b0 0x7f989c06e770 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f98b80739f0 tx=0x7f98a4009040 comp rx=0 tx=0).stop 2024-09-16T01:36:15.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.551+0000 7f98bed0a640 1 -- 172.21.15.88:0/200530658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98b8062990 msgr2=0x7f98b80aeb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:15.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.551+0000 7f98bed0a640 1 --2- 172.21.15.88:0/200530658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98b8062990 0x7f98b80aeb00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f98ac035040 tx=0x7f98ac0046c0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.551+0000 7f98bed0a640 1 -- 172.21.15.88:0/200530658 shutdown_connections 2024-09-16T01:36:15.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.551+0000 7f98bed0a640 1 --2- 172.21.15.88:0/200530658 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f989c06c2b0 0x7f989c06e770 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.551+0000 7f98bed0a640 1 --2- 172.21.15.88:0/200530658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b815c430 0x7f98b81f4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.551+0000 7f98bed0a640 1 --2- 172.21.15.88:0/200530658 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98b806e170 0x7f98b8158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.551+0000 7f98bed0a640 1 --2- 172.21.15.88:0/200530658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98b8062990 0x7f98b80aeb00 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:15.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.551+0000 7f98bed0a640 1 -- 172.21.15.88:0/200530658 >> 172.21.15.88:0/200530658 conn(0x7f98b806bc10 msgr2=0x7f98b80b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:15.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.551+0000 7f98bed0a640 1 -- 172.21.15.88:0/200530658 shutdown_connections 2024-09-16T01:36:15.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15.551+0000 7f98bed0a640 1 -- 172.21.15.88:0/200530658 wait complete. 2024-09-16T01:36:15.564 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-16T01:36:15.564 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:36:15.564 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2024-09-16T01:36:15.564 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T01:36:15.564 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T01:36:15.564 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-16T01:36:15.564 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-16T01:36:15.564 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:36:15.564 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:36:15.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:15.565 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2024-09-16T01:36:15.581 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:36:15.581 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:36:15.581 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:36:15.581 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:36:15.581 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-09-16T01:36:15.581 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:36:15.581 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:36:15.581 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:36:15.581 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2024-09-16T01:36:15.581 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:36:15.582 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:36:15.582 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2024-09-16T01:36:15.582 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:36:15.582 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2024-09-16T01:36:15.582 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:36:15.582 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2024-09-16T01:36:15.582 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:36:15.582 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42648sec preferred_lft 42648sec 2024-09-16T01:36:15.582 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link noprefixroute 2024-09-16T01:36:15.582 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:36:15.582 INFO:teuthology.orchestra.run.smithi088.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:36:15.582 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff 2024-09-16T01:36:15.582 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T01:36:15.582 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:36:15.583 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::e486:6eff:fe81:a1c/64 scope link 2024-09-16T01:36:15.583 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:36:15.583 INFO:teuthology.orchestra.run.smithi088.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:36:15.583 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 12:6f:29:be:95:48 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:36:15.583 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::106f:29ff:febe:9548/64 scope link 2024-09-16T01:36:15.583 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:36:15.583 INFO:teuthology.orchestra.run.smithi088.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:36:15.583 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T01:36:15.583 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::c63:b4ff:feb3:ad9e/64 scope link 2024-09-16T01:36:15.583 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:36:15.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:15.584 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:36:15.644 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-09-16T01:36:15.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:15.646 DEBUG:teuthology.orchestra.run.smithi088:> route 2024-09-16T01:36:15.702 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2024-09-16T01:36:15.702 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:36:15.702 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:36:15.703 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:36:15.703 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:36:15.703 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:36:15.703 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:36:15.703 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:36:15.703 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:36:15.703 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:36:15.703 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:36:15.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:36:15.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:36:15.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:15.828 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:36:15.843 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T01:36:15.843 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T01:36:15.844 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:36:15.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:15.844 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2024-09-16T01:36:15.907 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:36:15.908 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T01:36:15.963 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:36:15.964 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:36:15.964 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:36:16.019 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-16T01:36:16.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:16.019 DEBUG:teuthology.orchestra.run.smithi088:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:36:16.103 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:36:16.103 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:36:16.103 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:36:16.112 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:36:16.112 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:36:16.112 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:36:16.112 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:36:16.112 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:36:16.117 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:36:16.117 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:36:16.117 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:36:16.117 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:36:16.117 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:36:16.117 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790" 2024-09-16T01:36:16.118 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2024-09-16T01:36:16.118 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5ace2c91-46f9-482b-8a89-731fbdc171bf" will pass to kernel 2024-09-16T01:36:16.118 INFO:teuthology.orchestra.run.smithi088.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-16T01:36:16.118 INFO:teuthology.orchestra.run.smithi088.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:36:16.118 INFO:teuthology.orchestra.run.smithi088.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:36:16.118 INFO:teuthology.orchestra.run.smithi088.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:36:16.118 INFO:teuthology.orchestra.run.smithi088.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:36:16.120 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:36:16.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:16.120 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:36:16.145 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:36:16.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:36:16.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:36:16.244 INFO:teuthology.orchestra.run.smithi088.stdout:client5143 2024-09-16T01:36:16.245 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:36:16.245 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:36:16.260 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'mkdir -p a/b') 2024-09-16T01:36:16.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:36:16.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:36:16.363 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 flush journal 2024-09-16T01:36:16.396 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:36:16.408 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:36:16.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.463+0000 7f0d04c7b640 1 Processor -- start 2024-09-16T01:36:16.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.463+0000 7f0d04c7b640 1 -- start start 2024-09-16T01:36:16.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.464+0000 7f0d04c7b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0d000d3060 0x7f0d000b5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.464+0000 7f0d04c7b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d000b5a80 0x7f0d00073160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.464+0000 7f0d04c7b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d000d0370 con 0x7f0d0015c550 2024-09-16T01:36:16.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.464+0000 7f0d04c7b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f0d00071530 con 0x7f0d000d3060 2024-09-16T01:36:16.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.464+0000 7f0d04c7b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0d0015c930 con 0x7f0d000b5a80 2024-09-16T01:36:16.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.465+0000 7f0cfdd9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0d000d3060 0x7f0d000b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.465+0000 7f0cfdd9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0d000d3060 0x7f0d000b5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48782/0 (socket says 172.21.15.88:48782) 2024-09-16T01:36:16.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.465+0000 7f0cfdd9b640 1 -- 172.21.15.88:0/3882032001 learned_addr learned my addr 172.21.15.88:0/3882032001 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:16.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.465+0000 7f0cfed9d640 1 --2- 172.21.15.88:0/3882032001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d000b5a80 0x7f0d00073160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.465+0000 7f0cfdd9b640 1 -- 172.21.15.88:0/3882032001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d000b5a80 msgr2=0x7f0d00073160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:16.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.465+0000 7f0cfdd9b640 1 --2- 172.21.15.88:0/3882032001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d000b5a80 0x7f0d00073160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.465+0000 7f0cfdd9b640 1 -- 172.21.15.88:0/3882032001 >> v1:172.21.15.154:6789/0 conn(0x7f0d0015c550 legacy=0x7f0d000b7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:16.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.465+0000 7f0cfdd9b640 1 -- 172.21.15.88:0/3882032001 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d00070920 con 0x7f0d000d3060 2024-09-16T01:36:16.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.466+0000 7f0cfdd9b640 1 --2- 172.21.15.88:0/3882032001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d000d3060 0x7f0d000b5540 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8009870 tx=0x7f0ce802a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4f7f28510c9e672 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:16.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.466+0000 7f0cfd59a640 1 -- 172.21.15.88:0/3882032001 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8037070 con 0x7f0d000d3060 2024-09-16T01:36:16.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.466+0000 7f0cfd59a640 1 -- 172.21.15.88:0/3882032001 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ce8032440 con 0x7f0d000d3060 2024-09-16T01:36:16.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.466+0000 7f0cfd59a640 1 -- 172.21.15.88:0/3882032001 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8032810 con 0x7f0d000d3060 2024-09-16T01:36:16.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.466+0000 7f0d04c7b640 1 -- 172.21.15.88:0/3882032001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d000d3060 msgr2=0x7f0d000b5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:16.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.466+0000 7f0d04c7b640 1 --2- 172.21.15.88:0/3882032001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d000d3060 0x7f0d000b5540 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8009870 tx=0x7f0ce802a2a0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.466+0000 7f0d04c7b640 1 -- 172.21.15.88:0/3882032001 shutdown_connections 2024-09-16T01:36:16.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.466+0000 7f0d04c7b640 1 --2- 172.21.15.88:0/3882032001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d000b5a80 0x7f0d00073160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.466+0000 7f0d04c7b640 1 --2- 172.21.15.88:0/3882032001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d000d3060 0x7f0d000b5540 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.466+0000 7f0d04c7b640 1 -- 172.21.15.88:0/3882032001 >> 172.21.15.88:0/3882032001 conn(0x7f0d000d24d0 msgr2=0x7f0d000d28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:16.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.466+0000 7f0d04c7b640 1 -- 172.21.15.88:0/3882032001 shutdown_connections 2024-09-16T01:36:16.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.466+0000 7f0d04c7b640 1 -- 172.21.15.88:0/3882032001 wait complete. 2024-09-16T01:36:16.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.468+0000 7f0d04c7b640 1 Processor -- start 2024-09-16T01:36:16.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.468+0000 7f0d04c7b640 1 -- start start 2024-09-16T01:36:16.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.468+0000 7f0d04c7b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d000b5a80 0x7f0d000b14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.468+0000 7f0d04c7b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d000d3060 0x7f0d000ae060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.468+0000 7f0d04c7b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d0015c550 0x7f0d000af9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.468+0000 7f0d04c7b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0d000711c0 con 0x7f0d000d3060 2024-09-16T01:36:16.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.469+0000 7f0d04c7b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0d00070ee0 con 0x7f0d0015c550 2024-09-16T01:36:16.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.469+0000 7f0d04c7b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0d000736a0 con 0x7f0d000b5a80 2024-09-16T01:36:16.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.469+0000 7f0cfdd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d000d3060 0x7f0d000ae060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.469+0000 7f0cfdd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d000d3060 0x7f0d000ae060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48794/0 (socket says 172.21.15.88:48794) 2024-09-16T01:36:16.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.469+0000 7f0cfdd9b640 1 -- 172.21.15.88:0/1331909273 learned_addr learned my addr 172.21.15.88:0/1331909273 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:16.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.469+0000 7f0cfed9d640 1 --2- 172.21.15.88:0/1331909273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d0015c550 0x7f0d000af9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.469+0000 7f0cfe59c640 1 --2- 172.21.15.88:0/1331909273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d000b5a80 0x7f0d000b14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.469+0000 7f0cfdd9b640 1 -- 172.21.15.88:0/1331909273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d000b5a80 msgr2=0x7f0d000b14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:16.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.469+0000 7f0cfdd9b640 1 --2- 172.21.15.88:0/1331909273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d000b5a80 0x7f0d000b14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.469+0000 7f0cfdd9b640 1 -- 172.21.15.88:0/1331909273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d0015c550 msgr2=0x7f0d000af9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:16.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.469+0000 7f0cfdd9b640 1 --2- 172.21.15.88:0/1331909273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d0015c550 0x7f0d000af9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.469+0000 7f0cfdd9b640 1 -- 172.21.15.88:0/1331909273 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d000ad2a0 con 0x7f0d000d3060 2024-09-16T01:36:16.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.469+0000 7f0cfdd9b640 1 --2- 172.21.15.88:0/1331909273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d000d3060 0x7f0d000ae060 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8009870 tx=0x7f0ce80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:16.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.469+0000 7f0cf6ffd640 1 -- 172.21.15.88:0/1331909273 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8041020 con 0x7f0d000d3060 2024-09-16T01:36:16.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.469+0000 7f0cf6ffd640 1 -- 172.21.15.88:0/1331909273 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ce8030970 con 0x7f0d000d3060 2024-09-16T01:36:16.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.469+0000 7f0d04c7b640 1 -- 172.21.15.88:0/1331909273 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d000ad580 con 0x7f0d000d3060 2024-09-16T01:36:16.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.469+0000 7f0cf6ffd640 1 -- 172.21.15.88:0/1331909273 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8037040 con 0x7f0d000d3060 2024-09-16T01:36:16.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.469+0000 7f0d04c7b640 1 -- 172.21.15.88:0/1331909273 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d000adb40 con 0x7f0d000d3060 2024-09-16T01:36:16.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.471+0000 7f0cf6ffd640 1 -- 172.21.15.88:0/1331909273 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0ce8044020 con 0x7f0d000d3060 2024-09-16T01:36:16.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.471+0000 7f0cf6ffd640 1 --2- 172.21.15.88:0/1331909273 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0ce006c2b0 0x7f0ce006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.471+0000 7f0cf6ffd640 1 -- 172.21.15.88:0/1331909273 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f0ce80abc30 con 0x7f0d000d3060 2024-09-16T01:36:16.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.471+0000 7f0cfe59c640 1 --2- 172.21.15.88:0/1331909273 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0ce006c2b0 0x7f0ce006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.472+0000 7f0cfe59c640 1 --2- 172.21.15.88:0/1331909273 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0ce006c2b0 0x7f0ce006e770 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0d000b3650 tx=0x7f0cf0009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:16.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.472+0000 7f0d07a1a740 1 Processor -- start 2024-09-16T01:36:16.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.472+0000 7f0d07a1a740 1 -- start start 2024-09-16T01:36:16.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.473+0000 7f0d07a1a740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556d80be3920 0x556d80c290c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.473+0000 7f0d07a1a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d80be10e0 0x556d80be3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.473+0000 7f0d07a1a740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80be50b0 0x556d80bde1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.473+0000 7f0d07a1a740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x556d80b466f0 con 0x556d80be50b0 2024-09-16T01:36:16.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.473+0000 7f0d07a1a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556d80aee330 con 0x556d80be10e0 2024-09-16T01:36:16.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.473+0000 7f0d07a1a740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x556d80b426b0 con 0x556d80be3920 2024-09-16T01:36:16.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.473+0000 7f0cfed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80be50b0 0x556d80bde1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.473+0000 7f0cfed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80be50b0 0x556d80bde1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48806/0 (socket says 172.21.15.88:48806) 2024-09-16T01:36:16.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.473+0000 7f0cfed9d640 1 -- 172.21.15.88:0/3012854691 learned_addr learned my addr 172.21.15.88:0/3012854691 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:16.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.473+0000 7f0cfe59c640 1 --2- 172.21.15.88:0/3012854691 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556d80be3920 0x556d80c290c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.473+0000 7f0cfdd9b640 1 --2- 172.21.15.88:0/3012854691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d80be10e0 0x556d80be3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.474+0000 7f0cfed9d640 1 -- 172.21.15.88:0/3012854691 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556d80be3920 msgr2=0x556d80c290c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:16.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.474+0000 7f0cfed9d640 1 --2- 172.21.15.88:0/3012854691 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556d80be3920 0x556d80c290c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.474+0000 7f0cfed9d640 1 -- 172.21.15.88:0/3012854691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d80be10e0 msgr2=0x556d80be3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:16.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.474+0000 7f0cfed9d640 1 --2- 172.21.15.88:0/3012854691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d80be10e0 0x556d80be3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.474+0000 7f0cfed9d640 1 -- 172.21.15.88:0/3012854691 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d80b140d0 con 0x556d80be50b0 2024-09-16T01:36:16.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.474+0000 7f0cfed9d640 1 --2- 172.21.15.88:0/3012854691 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80be50b0 0x556d80bde1a0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f0cf800f510 tx=0x7f0cf800f9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0b4455cf39db864 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:16.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.474+0000 7f0cdffff640 1 -- 172.21.15.88:0/3012854691 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf801c070 con 0x556d80be50b0 2024-09-16T01:36:16.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.474+0000 7f0cdffff640 1 -- 172.21.15.88:0/3012854691 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cf800d790 con 0x556d80be50b0 2024-09-16T01:36:16.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.474+0000 7f0cdffff640 1 -- 172.21.15.88:0/3012854691 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf800db40 con 0x556d80be50b0 2024-09-16T01:36:16.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.474+0000 7f0d07a1a740 1 -- 172.21.15.88:0/3012854691 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80be50b0 msgr2=0x556d80bde1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:16.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.474+0000 7f0d07a1a740 1 --2- 172.21.15.88:0/3012854691 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80be50b0 0x556d80bde1a0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f0cf800f510 tx=0x7f0cf800f9d0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.474+0000 7f0d07a1a740 1 -- 172.21.15.88:0/3012854691 shutdown_connections 2024-09-16T01:36:16.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.474+0000 7f0d07a1a740 1 --2- 172.21.15.88:0/3012854691 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80be50b0 0x556d80bde1a0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.475+0000 7f0d07a1a740 1 --2- 172.21.15.88:0/3012854691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d80be10e0 0x556d80be3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.475+0000 7f0d07a1a740 1 --2- 172.21.15.88:0/3012854691 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556d80be3920 0x556d80c290c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.475+0000 7f0d07a1a740 1 -- 172.21.15.88:0/3012854691 >> 172.21.15.88:0/3012854691 conn(0x556d80be0370 msgr2=0x556d80c28590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:16.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.475+0000 7f0d07a1a740 1 -- 172.21.15.88:0/3012854691 shutdown_connections 2024-09-16T01:36:16.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.475+0000 7f0d07a1a740 1 -- 172.21.15.88:0/3012854691 wait complete. 2024-09-16T01:36:16.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.475+0000 7f0d07a1a740 1 -- start start 2024-09-16T01:36:16.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.475+0000 7f0d07a1a740 1 Processor -- start 2024-09-16T01:36:16.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.476+0000 7f0d07a1a740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80be3920 0x556d80c03bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.476+0000 7f0d07a1a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d80be50b0 0x556d80c0e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.476+0000 7f0d07a1a740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556d80c04aa0 0x556d80c09a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.476+0000 7f0d07a1a740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x556d80bdf810 con 0x556d80be3920 2024-09-16T01:36:16.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.476+0000 7f0d07a1a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556d80acf4f0 con 0x556d80be50b0 2024-09-16T01:36:16.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.476+0000 7f0d07a1a740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x556d80c298f0 con 0x556d80c04aa0 2024-09-16T01:36:16.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.476+0000 7f0d07a1a740 10 client.0 _ll_register_callbacks cb 0x556d80af0b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:36:16.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.476+0000 7f0cfe59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80be3920 0x556d80c03bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.476+0000 7f0cfe59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80be3920 0x556d80c03bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48808/0 (socket says 172.21.15.88:48808) 2024-09-16T01:36:16.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.476+0000 7f0cfe59c640 1 -- 172.21.15.88:0/18600716 learned_addr learned my addr 172.21.15.88:0/18600716 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:16.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.476+0000 7f0cfe59c640 1 -- 172.21.15.88:0/18600716 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556d80c04aa0 msgr2=0x556d80c09a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:16.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.476+0000 7f0cfe59c640 1 --2- 172.21.15.88:0/18600716 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556d80c04aa0 0x556d80c09a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.476+0000 7f0cfe59c640 1 -- 172.21.15.88:0/18600716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d80be50b0 msgr2=0x556d80c0e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:16.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.476+0000 7f0cfe59c640 1 --2- 172.21.15.88:0/18600716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d80be50b0 0x556d80c0e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.476+0000 7f0cfe59c640 1 -- 172.21.15.88:0/18600716 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d809237d0 con 0x556d80be3920 2024-09-16T01:36:16.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.477+0000 7f0cfe59c640 1 --2- 172.21.15.88:0/18600716 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80be3920 0x556d80c03bb0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0015760 tx=0x7f0cf0004680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=885d76f73ee71fd1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:16.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.477+0000 7f0cdd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:36:16.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.477+0000 7f0cdd7fa640 1 -- 172.21.15.88:0/18600716 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf0025020 con 0x556d80be3920 2024-09-16T01:36:16.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.477+0000 7f0cdd7fa640 1 -- 172.21.15.88:0/18600716 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f0cf0013690 con 0x556d80be3920 2024-09-16T01:36:16.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.477+0000 7f0d07a1a740 1 -- 172.21.15.88:0/18600716 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cf800d790 con 0x556d80be3920 2024-09-16T01:36:16.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.477+0000 7f0cdd7fa640 1 -- 172.21.15.88:0/18600716 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0cf001b040 con 0x556d80be3920 2024-09-16T01:36:16.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.477+0000 7f0d07a1a740 10 client.5586 fetch_fsmap learned FSMap version 25 2024-09-16T01:36:16.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.477+0000 7f0d07a1a740 1 -- 172.21.15.88:0/18600716 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x556d808593f0 con 0x556d80be3920 2024-09-16T01:36:16.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.477+0000 7f0cdd7fa640 1 -- 172.21.15.88:0/18600716 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cf800d790 con 0x556d80be3920 2024-09-16T01:36:16.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.477+0000 7f0cdd7fa640 1 -- 172.21.15.88:0/18600716 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 25) ==== 3502+0+0 (secure 0 0 0) 0x7f0cf0004020 con 0x556d80be3920 2024-09-16T01:36:16.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.477+0000 7f0d07a1a740 10 client.5586 fetch_fsmap finished waiting for FSMap version 25 2024-09-16T01:36:16.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.477+0000 7f0d07a1a740 10 client.5586 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-16T01:36:16.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.477+0000 7f0d07a1a740 20 client.5586 populate_metadata read hostname 'smithi088' 2024-09-16T01:36:16.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.477+0000 7f0d07a1a740 1 --2- 172.21.15.88:0/18600716 >> [v2:172.21.15.88:6834/1610810484,v1:172.21.15.88:6836/1610810484] conn(0x556d80bdd760 0x556d80bddbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.477+0000 7f0d07a1a740 4 client.5586 mds_command: new command op to 5406 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:36:16.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.477+0000 7f0d07a1a740 1 -- 172.21.15.88:0/18600716 --> [v2:172.21.15.88:6834/1610810484,v1:172.21.15.88:6836/1610810484] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d80b53e20 con 0x556d80bdd760 2024-09-16T01:36:16.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.477+0000 7f0cfdd9b640 1 --2- 172.21.15.88:0/18600716 >> [v2:172.21.15.88:6834/1610810484,v1:172.21.15.88:6836/1610810484] conn(0x556d80bdd760 0x556d80bddbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.478+0000 7f0cdd7fa640 10 client.5586 ms_handle_connect on v2:172.21.15.88:6834/1610810484 2024-09-16T01:36:16.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.478+0000 7f0cfdd9b640 1 --2- 172.21.15.88:0/18600716 >> [v2:172.21.15.88:6834/1610810484,v1:172.21.15.88:6836/1610810484] conn(0x556d80bdd760 0x556d80bddbf0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3dde50572560c28e server_cookie=8e312f144551352d in_seq=0 out_seq=0 2024-09-16T01:36:16.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.478+0000 7f0cdd7fa640 1 -- 172.21.15.88:0/18600716 <== mds.0 v2:172.21.15.88:6834/1610810484 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556d80b53e20 con 0x556d80bdd760 2024-09-16T01:36:16.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.478+0000 7f0cdd7fa640 10 client.5586 handle_command_reply: tid=0 2024-09-16T01:36:16.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.478+0000 7f0d07a1a740 1 client.5586 shutdown 2024-09-16T01:36:16.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.478+0000 7f0d07a1a740 1 -- 172.21.15.88:0/18600716 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80be3920 msgr2=0x556d80c03bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:16.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.478+0000 7f0d07a1a740 1 --2- 172.21.15.88:0/18600716 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80be3920 0x556d80c03bb0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0015760 tx=0x7f0cf0004680 comp rx=0 tx=0).stop 2024-09-16T01:36:16.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.478+0000 7f0d07a1a740 1 -- 172.21.15.88:0/18600716 shutdown_connections 2024-09-16T01:36:16.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.479+0000 7f0d07a1a740 1 --2- 172.21.15.88:0/18600716 >> [v2:172.21.15.88:6834/1610810484,v1:172.21.15.88:6836/1610810484] conn(0x556d80bdd760 0x556d80bddbf0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.479+0000 7f0d07a1a740 1 --2- 172.21.15.88:0/18600716 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556d80c04aa0 0x556d80c09a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.479+0000 7f0d07a1a740 1 --2- 172.21.15.88:0/18600716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d80be50b0 0x556d80c0e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.479+0000 7f0cdd7fa640 0 client.5586 ms_handle_reset on v2:172.21.15.88:6834/1610810484 2024-09-16T01:36:16.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.479+0000 7f0d07a1a740 1 --2- 172.21.15.88:0/18600716 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80be3920 0x556d80c03bb0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.479+0000 7f0d07a1a740 1 -- 172.21.15.88:0/18600716 >> 172.21.15.88:0/18600716 conn(0x556d80be10e0 msgr2=0x556d80c26610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:16.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.479+0000 7f0d07a1a740 1 -- 172.21.15.88:0/18600716 shutdown_connections 2024-09-16T01:36:16.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.479+0000 7f0d07a1a740 1 -- 172.21.15.88:0/18600716 wait complete. 2024-09-16T01:36:16.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.479+0000 7f0d07a1a740 20 client.5586 trim_cache size 0 max 16384 2024-09-16T01:36:16.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.493+0000 7f0d07a1a740 1 Processor -- start 2024-09-16T01:36:16.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.493+0000 7f0d07a1a740 1 -- start start 2024-09-16T01:36:16.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.494+0000 7f0d07a1a740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80be3920 0x556d80c05b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.494+0000 7f0d07a1a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d80be50b0 0x556d80bd9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.494+0000 7f0d07a1a740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556d80c04aa0 0x556d80c286b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.494+0000 7f0d07a1a740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0d001febc0 con 0x556d80be3920 2024-09-16T01:36:16.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.494+0000 7f0d07a1a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0d000abed0 con 0x556d80be50b0 2024-09-16T01:36:16.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.494+0000 7f0d07a1a740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0d000c8190 con 0x556d80c04aa0 2024-09-16T01:36:16.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.494+0000 7f0cfe59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80be3920 0x556d80c05b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.494+0000 7f0cfe59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80be3920 0x556d80c05b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48822/0 (socket says 172.21.15.88:48822) 2024-09-16T01:36:16.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.494+0000 7f0cfe59c640 1 -- 172.21.15.88:0/2637564548 learned_addr learned my addr 172.21.15.88:0/2637564548 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:16.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.494+0000 7f0cfed9d640 1 --2- 172.21.15.88:0/2637564548 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556d80c04aa0 0x556d80c286b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.494+0000 7f0cfdd9b640 1 --2- 172.21.15.88:0/2637564548 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d80be50b0 0x556d80bd9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.494+0000 7f0cfe59c640 1 -- 172.21.15.88:0/2637564548 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556d80c04aa0 msgr2=0x556d80c286b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:16.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.494+0000 7f0cfe59c640 1 --2- 172.21.15.88:0/2637564548 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556d80c04aa0 0x556d80c286b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.494+0000 7f0cfe59c640 1 -- 172.21.15.88:0/2637564548 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d80be50b0 msgr2=0x556d80bd9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:16.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.494+0000 7f0cfe59c640 1 --2- 172.21.15.88:0/2637564548 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d80be50b0 0x556d80bd9b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.494+0000 7f0cfe59c640 1 -- 172.21.15.88:0/2637564548 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d80c09dc0 con 0x556d80be3920 2024-09-16T01:36:16.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.494+0000 7f0cfdd9b640 1 --2- 172.21.15.88:0/2637564548 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d80be50b0 0x556d80bd9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:16.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.494+0000 7f0cfe59c640 1 --2- 172.21.15.88:0/2637564548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80be3920 0x556d80c05b50 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f0cf000ec80 tx=0x7f0cf001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=879228d37b265e3e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:16.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.494+0000 7f0cdf7fe640 1 -- 172.21.15.88:0/2637564548 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf0038070 con 0x556d80be3920 2024-09-16T01:36:16.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.494+0000 7f0cdf7fe640 1 -- 172.21.15.88:0/2637564548 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cf00138b0 con 0x556d80be3920 2024-09-16T01:36:16.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.494+0000 7f0cdf7fe640 1 -- 172.21.15.88:0/2637564548 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf0025030 con 0x556d80be3920 2024-09-16T01:36:16.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.495+0000 7f0d07a1a740 1 -- 172.21.15.88:0/2637564548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80be3920 msgr2=0x556d80c05b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:16.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.495+0000 7f0d07a1a740 1 --2- 172.21.15.88:0/2637564548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80be3920 0x556d80c05b50 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f0cf000ec80 tx=0x7f0cf001f000 comp rx=0 tx=0).stop 2024-09-16T01:36:16.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.495+0000 7f0d07a1a740 1 -- 172.21.15.88:0/2637564548 shutdown_connections 2024-09-16T01:36:16.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.495+0000 7f0d07a1a740 1 --2- 172.21.15.88:0/2637564548 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556d80c04aa0 0x556d80c286b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.495+0000 7f0d07a1a740 1 --2- 172.21.15.88:0/2637564548 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d80be50b0 0x556d80bd9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.495+0000 7f0d07a1a740 1 --2- 172.21.15.88:0/2637564548 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80be3920 0x556d80c05b50 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.495+0000 7f0d07a1a740 1 -- 172.21.15.88:0/2637564548 >> 172.21.15.88:0/2637564548 conn(0x556d80be10e0 msgr2=0x556d80c4c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:16.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.495+0000 7f0d07a1a740 1 -- 172.21.15.88:0/2637564548 shutdown_connections 2024-09-16T01:36:16.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.495+0000 7f0d07a1a740 1 -- 172.21.15.88:0/2637564548 wait complete. 2024-09-16T01:36:16.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.496+0000 7f0d07a1a740 1 -- start start 2024-09-16T01:36:16.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.496+0000 7f0d07a1a740 1 Processor -- start 2024-09-16T01:36:16.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.497+0000 7f0d07a1a740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d80be3920 0x556d80c4c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.497+0000 7f0d07a1a740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556d80be50b0 0x556d80c90ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.497+0000 7f0d07a1a740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80c04aa0 0x556d80c90ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.497+0000 7f0d07a1a740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0d000ab880 con 0x556d80c04aa0 2024-09-16T01:36:16.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.497+0000 7f0d07a1a740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556d80c91f70 con 0x556d80be3920 2024-09-16T01:36:16.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.497+0000 7f0d07a1a740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x556d80c03aa0 con 0x556d80be50b0 2024-09-16T01:36:16.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.497+0000 7f0d07a1a740 10 client.0 _ll_register_callbacks cb 0x7f0cf0009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:36:16.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.497+0000 7f0cfed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80c04aa0 0x556d80c90ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.497+0000 7f0cfed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80c04aa0 0x556d80c90ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48828/0 (socket says 172.21.15.88:48828) 2024-09-16T01:36:16.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.497+0000 7f0cfed9d640 1 -- 172.21.15.88:0/1171847390 learned_addr learned my addr 172.21.15.88:0/1171847390 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:16.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.497+0000 7f0cfdd9b640 1 --2- 172.21.15.88:0/1171847390 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556d80be50b0 0x556d80c90ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.497+0000 7f0cfe59c640 1 --2- 172.21.15.88:0/1171847390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d80be3920 0x556d80c4c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.497+0000 7f0cfed9d640 1 -- 172.21.15.88:0/1171847390 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556d80be50b0 msgr2=0x556d80c90ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:16.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.497+0000 7f0cfed9d640 1 --2- 172.21.15.88:0/1171847390 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556d80be50b0 0x556d80c90ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.497+0000 7f0cfed9d640 1 -- 172.21.15.88:0/1171847390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d80be3920 msgr2=0x556d80c4c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:16.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.497+0000 7f0cfed9d640 1 --2- 172.21.15.88:0/1171847390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d80be3920 0x556d80c4c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.497+0000 7f0cfed9d640 1 -- 172.21.15.88:0/1171847390 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d808190f0 con 0x556d80c04aa0 2024-09-16T01:36:16.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.497+0000 7f0cfed9d640 1 --2- 172.21.15.88:0/1171847390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80c04aa0 0x556d80c90ff0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8011010 tx=0x7f0cf8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4465fc2a7585f08b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:16.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.497+0000 7f0cdd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:36:16.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.497+0000 7f0cdd7fa640 1 -- 172.21.15.88:0/1171847390 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8025020 con 0x556d80c04aa0 2024-09-16T01:36:16.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.497+0000 7f0cdd7fa640 1 -- 172.21.15.88:0/1171847390 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f0cf8003b90 con 0x556d80c04aa0 2024-09-16T01:36:16.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.497+0000 7f0d07a1a740 1 -- 172.21.15.88:0/1171847390 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cf00138b0 con 0x556d80c04aa0 2024-09-16T01:36:16.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.497+0000 7f0cdd7fa640 1 -- 172.21.15.88:0/1171847390 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0cf802f050 con 0x556d80c04aa0 2024-09-16T01:36:16.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.497+0000 7f0d07a1a740 10 client.5592 fetch_fsmap learned FSMap version 25 2024-09-16T01:36:16.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.497+0000 7f0d07a1a740 1 -- 172.21.15.88:0/1171847390 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x556d80c91910 con 0x556d80c04aa0 2024-09-16T01:36:16.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.498+0000 7f0cdd7fa640 1 -- 172.21.15.88:0/1171847390 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cf00138b0 con 0x556d80c04aa0 2024-09-16T01:36:16.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.498+0000 7f0cdd7fa640 1 -- 172.21.15.88:0/1171847390 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 25) ==== 3502+0+0 (secure 0 0 0) 0x7f0cf8032020 con 0x556d80c04aa0 2024-09-16T01:36:16.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.498+0000 7f0d07a1a740 10 client.5592 fetch_fsmap finished waiting for FSMap version 25 2024-09-16T01:36:16.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.498+0000 7f0d07a1a740 10 client.5592 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-16T01:36:16.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.498+0000 7f0d07a1a740 20 client.5592 populate_metadata read hostname 'smithi088' 2024-09-16T01:36:16.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.498+0000 7f0d07a1a740 1 --2- 172.21.15.88:0/1171847390 >> [v2:172.21.15.88:6834/1610810484,v1:172.21.15.88:6836/1610810484] conn(0x556d80c05ae0 0x556d80c08d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.498+0000 7f0d07a1a740 4 client.5592 mds_command: new command op to 5406 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:36:16.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.498+0000 7f0d07a1a740 1 -- 172.21.15.88:0/1171847390 --> [v2:172.21.15.88:6834/1610810484,v1:172.21.15.88:6836/1610810484] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556d80b4e170 con 0x556d80c05ae0 2024-09-16T01:36:16.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.498+0000 7f0cfe59c640 1 --2- 172.21.15.88:0/1171847390 >> [v2:172.21.15.88:6834/1610810484,v1:172.21.15.88:6836/1610810484] conn(0x556d80c05ae0 0x556d80c08d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.498+0000 7f0cdd7fa640 10 client.5592 ms_handle_connect on v2:172.21.15.88:6834/1610810484 2024-09-16T01:36:16.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.498+0000 7f0cfe59c640 1 --2- 172.21.15.88:0/1171847390 >> [v2:172.21.15.88:6834/1610810484,v1:172.21.15.88:6836/1610810484] conn(0x556d80c05ae0 0x556d80c08d40 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37b03978465a02c1 server_cookie=f892228aa1eeee4a in_seq=0 out_seq=0 2024-09-16T01:36:16.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.509+0000 7f0cdd7fa640 1 -- 172.21.15.88:0/1171847390 <== mds.0 v2:172.21.15.88:6834/1610810484 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556d80b4e170 con 0x556d80c05ae0 2024-09-16T01:36:16.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.509+0000 7f0cdd7fa640 10 client.5592 handle_command_reply: tid=0 2024-09-16T01:36:16.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.509+0000 7f0d07a1a740 1 client.5592 shutdown 2024-09-16T01:36:16.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.509+0000 7f0d07a1a740 1 -- 172.21.15.88:0/1171847390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80c04aa0 msgr2=0x556d80c90ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:16.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.509+0000 7f0d07a1a740 1 --2- 172.21.15.88:0/1171847390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80c04aa0 0x556d80c90ff0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8011010 tx=0x7f0cf8004a40 comp rx=0 tx=0).stop 2024-09-16T01:36:16.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.509+0000 7f0d07a1a740 1 -- 172.21.15.88:0/1171847390 shutdown_connections 2024-09-16T01:36:16.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.509+0000 7f0d07a1a740 1 --2- 172.21.15.88:0/1171847390 >> [v2:172.21.15.88:6834/1610810484,v1:172.21.15.88:6836/1610810484] conn(0x556d80c05ae0 0x556d80c08d40 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.509+0000 7f0d07a1a740 1 --2- 172.21.15.88:0/1171847390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556d80c04aa0 0x556d80c90ff0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.509+0000 7f0d07a1a740 1 --2- 172.21.15.88:0/1171847390 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556d80be50b0 0x556d80c90ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.509+0000 7f0cdd7fa640 0 client.5592 ms_handle_reset on v2:172.21.15.88:6834/1610810484 2024-09-16T01:36:16.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.509+0000 7f0d07a1a740 1 --2- 172.21.15.88:0/1171847390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d80be3920 0x556d80c4c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.509+0000 7f0d07a1a740 1 -- 172.21.15.88:0/1171847390 >> 172.21.15.88:0/1171847390 conn(0x556d80be10e0 msgr2=0x556d80bd9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:16.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.509+0000 7f0d07a1a740 1 -- 172.21.15.88:0/1171847390 shutdown_connections 2024-09-16T01:36:16.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.509+0000 7f0d07a1a740 1 -- 172.21.15.88:0/1171847390 wait complete. 2024-09-16T01:36:16.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.509+0000 7f0d07a1a740 20 client.5592 trim_cache size 0 max 16384 2024-09-16T01:36:16.513 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:36:16.513 INFO:teuthology.orchestra.run.smithi088.stdout: "message": "", 2024-09-16T01:36:16.513 INFO:teuthology.orchestra.run.smithi088.stdout: "return_code": 0 2024-09-16T01:36:16.513 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:36:16.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.510+0000 7f0d04c7b640 1 -- 172.21.15.88:0/1331909273 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0ce006c2b0 msgr2=0x7f0ce006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:16.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.510+0000 7f0d04c7b640 1 --2- 172.21.15.88:0/1331909273 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0ce006c2b0 0x7f0ce006e770 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0d000b3650 tx=0x7f0cf0009060 comp rx=0 tx=0).stop 2024-09-16T01:36:16.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.510+0000 7f0d04c7b640 1 -- 172.21.15.88:0/1331909273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d000d3060 msgr2=0x7f0d000ae060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:16.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.510+0000 7f0d04c7b640 1 --2- 172.21.15.88:0/1331909273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d000d3060 0x7f0d000ae060 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8009870 tx=0x7f0ce80046c0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.510+0000 7f0d04c7b640 1 -- 172.21.15.88:0/1331909273 shutdown_connections 2024-09-16T01:36:16.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.510+0000 7f0d04c7b640 1 --2- 172.21.15.88:0/1331909273 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0ce006c2b0 0x7f0ce006e770 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.510+0000 7f0d04c7b640 1 --2- 172.21.15.88:0/1331909273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d0015c550 0x7f0d000af9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.510+0000 7f0d04c7b640 1 --2- 172.21.15.88:0/1331909273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d000d3060 0x7f0d000ae060 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.510+0000 7f0d04c7b640 1 --2- 172.21.15.88:0/1331909273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d000b5a80 0x7f0d000b14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.510+0000 7f0d04c7b640 1 -- 172.21.15.88:0/1331909273 >> 172.21.15.88:0/1331909273 conn(0x7f0d000d24d0 msgr2=0x7f0d000d0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:16.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.510+0000 7f0d04c7b640 1 -- 172.21.15.88:0/1331909273 shutdown_connections 2024-09-16T01:36:16.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.510+0000 7f0d04c7b640 1 -- 172.21.15.88:0/1331909273 wait complete. 2024-09-16T01:36:16.523 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_abort_on_newly_corrupt_dentry False 2024-09-16T01:36:16.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.663+0000 7f7a455cd640 1 Processor -- start 2024-09-16T01:36:16.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.663+0000 7f7a455cd640 1 -- start start 2024-09-16T01:36:16.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.664+0000 7f7a455cd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7a400b6d50 0x7f7a40073b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.664+0000 7f7a455cd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7a4006e5b0 0x7f7a4015f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.664+0000 7f7a455cd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a4006eae0 con 0x7f7a400b7130 2024-09-16T01:36:16.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.664+0000 7f7a455cd640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7a4015ed10 con 0x7f7a400b6d50 2024-09-16T01:36:16.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.664+0000 7f7a455cd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7a4015eff0 con 0x7f7a4006e5b0 2024-09-16T01:36:16.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.664+0000 7f7a3effd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7a400b6d50 0x7f7a40073b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.664+0000 7f7a3f7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7a400b7130 0x7f7a4006cd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:51592/0 (socket says 172.21.15.88:51592) 2024-09-16T01:36:16.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.664+0000 7f7a3f7fe640 1 -- 172.21.15.88:0/2002899417 learned_addr learned my addr 172.21.15.88:0/2002899417 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:16.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.665+0000 7f7a3effd640 1 -- 172.21.15.88:0/2002899417 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7a4006e5b0 msgr2=0x7f7a4015f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:16.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.665+0000 7f7a3e7fc640 1 --2- 172.21.15.88:0/2002899417 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7a4006e5b0 0x7f7a4015f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.665+0000 7f7a3effd640 1 --2- 172.21.15.88:0/2002899417 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7a4006e5b0 0x7f7a4015f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.665+0000 7f7a3dffb640 1 -- 172.21.15.88:0/2002899417 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 871020878 0 0) 0x7f7a4006eae0 con 0x7f7a400b7130 2024-09-16T01:36:16.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.665+0000 7f7a3effd640 1 -- 172.21.15.88:0/2002899417 >> v1:172.21.15.88:6789/0 conn(0x7f7a400b7130 legacy=0x7f7a4006cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:16.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.665+0000 7f7a3effd640 1 -- 172.21.15.88:0/2002899417 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a4015f2d0 con 0x7f7a400b6d50 2024-09-16T01:36:16.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.665+0000 7f7a3e7fc640 1 --2- 172.21.15.88:0/2002899417 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7a4006e5b0 0x7f7a4015f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:16.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.665+0000 7f7a3effd640 1 --2- 172.21.15.88:0/2002899417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a400b6d50 0x7f7a40073b70 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7a300097a0 tx=0x7f7a3002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3500079456114be server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:16.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.666+0000 7f7a3dffb640 1 -- 172.21.15.88:0/2002899417 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a30037070 con 0x7f7a400b6d50 2024-09-16T01:36:16.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.666+0000 7f7a3dffb640 1 -- 172.21.15.88:0/2002899417 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a30032440 con 0x7f7a400b6d50 2024-09-16T01:36:16.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.666+0000 7f7a3dffb640 1 -- 172.21.15.88:0/2002899417 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a30032810 con 0x7f7a400b6d50 2024-09-16T01:36:16.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.666+0000 7f7a455cd640 1 -- 172.21.15.88:0/2002899417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a400b6d50 msgr2=0x7f7a40073b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:16.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.666+0000 7f7a455cd640 1 --2- 172.21.15.88:0/2002899417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a400b6d50 0x7f7a40073b70 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7a300097a0 tx=0x7f7a3002a180 comp rx=0 tx=0).stop 2024-09-16T01:36:16.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.666+0000 7f7a455cd640 1 -- 172.21.15.88:0/2002899417 shutdown_connections 2024-09-16T01:36:16.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.666+0000 7f7a455cd640 1 --2- 172.21.15.88:0/2002899417 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7a4006e5b0 0x7f7a4015f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.666+0000 7f7a455cd640 1 --2- 172.21.15.88:0/2002899417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a400b6d50 0x7f7a40073b70 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.666+0000 7f7a455cd640 1 -- 172.21.15.88:0/2002899417 >> 172.21.15.88:0/2002899417 conn(0x7f7a400d24d0 msgr2=0x7f7a400d28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:16.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.666+0000 7f7a455cd640 1 -- 172.21.15.88:0/2002899417 shutdown_connections 2024-09-16T01:36:16.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.666+0000 7f7a455cd640 1 -- 172.21.15.88:0/2002899417 wait complete. 2024-09-16T01:36:16.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.667+0000 7f7a455cd640 1 Processor -- start 2024-09-16T01:36:16.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.668+0000 7f7a455cd640 1 -- start start 2024-09-16T01:36:16.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.668+0000 7f7a455cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a4006e5b0 0x7f7a40152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.668+0000 7f7a455cd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a400b6d50 0x7f7a40161460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.668+0000 7f7a455cd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7a400b7130 0x7f7a40145f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.668+0000 7f7a455cd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7a400b75a0 con 0x7f7a4006e5b0 2024-09-16T01:36:16.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.668+0000 7f7a455cd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4006edc0 con 0x7f7a400b6d50 2024-09-16T01:36:16.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.668+0000 7f7a455cd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7a4015e8b0 con 0x7f7a400b7130 2024-09-16T01:36:16.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.668+0000 7f7a3effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a4006e5b0 0x7f7a40152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.668+0000 7f7a3effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a4006e5b0 0x7f7a40152090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48848/0 (socket says 172.21.15.88:48848) 2024-09-16T01:36:16.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.668+0000 7f7a3effd640 1 -- 172.21.15.88:0/2088319144 learned_addr learned my addr 172.21.15.88:0/2088319144 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:16.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.668+0000 7f7a3f7fe640 1 --2- 172.21.15.88:0/2088319144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7a400b7130 0x7f7a40145f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.668+0000 7f7a3e7fc640 1 --2- 172.21.15.88:0/2088319144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a400b6d50 0x7f7a40161460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.669+0000 7f7a3effd640 1 -- 172.21.15.88:0/2088319144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7a400b7130 msgr2=0x7f7a40145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:16.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.669+0000 7f7a3effd640 1 --2- 172.21.15.88:0/2088319144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7a400b7130 0x7f7a40145f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.669+0000 7f7a3effd640 1 -- 172.21.15.88:0/2088319144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a400b6d50 msgr2=0x7f7a40161460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:16.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.669+0000 7f7a3effd640 1 --2- 172.21.15.88:0/2088319144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a400b6d50 0x7f7a40161460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.669+0000 7f7a3effd640 1 -- 172.21.15.88:0/2088319144 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a40152b60 con 0x7f7a4006e5b0 2024-09-16T01:36:16.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.669+0000 7f7a3effd640 1 --2- 172.21.15.88:0/2088319144 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a4006e5b0 0x7f7a40152090 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7a30035040 tx=0x7f7a300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:16.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.669+0000 7f7a277fe640 1 -- 172.21.15.88:0/2088319144 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a30037050 con 0x7f7a4006e5b0 2024-09-16T01:36:16.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.669+0000 7f7a277fe640 1 -- 172.21.15.88:0/2088319144 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a30038040 con 0x7f7a4006e5b0 2024-09-16T01:36:16.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.669+0000 7f7a455cd640 1 -- 172.21.15.88:0/2088319144 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a40152e00 con 0x7f7a4006e5b0 2024-09-16T01:36:16.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.669+0000 7f7a277fe640 1 -- 172.21.15.88:0/2088319144 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a30004490 con 0x7f7a4006e5b0 2024-09-16T01:36:16.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.669+0000 7f7a455cd640 1 -- 172.21.15.88:0/2088319144 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a401533c0 con 0x7f7a4006e5b0 2024-09-16T01:36:16.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.670+0000 7f7a455cd640 1 -- 172.21.15.88:0/2088319144 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a4015f7f0 con 0x7f7a4006e5b0 2024-09-16T01:36:16.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.671+0000 7f7a277fe640 1 -- 172.21.15.88:0/2088319144 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7a30030980 con 0x7f7a4006e5b0 2024-09-16T01:36:16.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.671+0000 7f7a277fe640 1 --2- 172.21.15.88:0/2088319144 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7a2c06c2b0 0x7f7a2c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.671+0000 7f7a277fe640 1 -- 172.21.15.88:0/2088319144 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f7a300ac000 con 0x7f7a4006e5b0 2024-09-16T01:36:16.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.671+0000 7f7a3e7fc640 1 --2- 172.21.15.88:0/2088319144 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7a2c06c2b0 0x7f7a2c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.671+0000 7f7a3e7fc640 1 --2- 172.21.15.88:0/2088319144 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7a2c06c2b0 0x7f7a2c06e770 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7a4015cab0 tx=0x7f7a34008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:16.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.675+0000 7f7a277fe640 1 -- 172.21.15.88:0/2088319144 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7a3007a920 con 0x7f7a4006e5b0 2024-09-16T01:36:16.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.842+0000 7f7a455cd640 1 -- 172.21.15.88:0/2088319144 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command([{prefix=config set, name=mds_abort_on_newly_corrupt_dentry}] v 0) -- 0x7f7a40071310 con 0x7f7a4006e5b0 2024-09-16T01:36:16.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.845+0000 7f7a277fe640 1 -- 172.21.15.88:0/2088319144 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_abort_on_newly_corrupt_dentry}]=0 v13)=0 v13) ==== 135+0+0 (secure 0 0 0) 0x7f7a3007a220 con 0x7f7a4006e5b0 2024-09-16T01:36:16.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.847+0000 7f7a455cd640 1 -- 172.21.15.88:0/2088319144 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7a2c06c2b0 msgr2=0x7f7a2c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:16.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.847+0000 7f7a455cd640 1 --2- 172.21.15.88:0/2088319144 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7a2c06c2b0 0x7f7a2c06e770 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7a4015cab0 tx=0x7f7a34008040 comp rx=0 tx=0).stop 2024-09-16T01:36:16.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.847+0000 7f7a455cd640 1 -- 172.21.15.88:0/2088319144 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a4006e5b0 msgr2=0x7f7a40152090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:16.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.847+0000 7f7a455cd640 1 --2- 172.21.15.88:0/2088319144 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a4006e5b0 0x7f7a40152090 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7a30035040 tx=0x7f7a300046c0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.847+0000 7f7a455cd640 1 -- 172.21.15.88:0/2088319144 shutdown_connections 2024-09-16T01:36:16.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.847+0000 7f7a455cd640 1 --2- 172.21.15.88:0/2088319144 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7a2c06c2b0 0x7f7a2c06e770 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.848+0000 7f7a455cd640 1 --2- 172.21.15.88:0/2088319144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7a400b7130 0x7f7a40145f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.848+0000 7f7a455cd640 1 --2- 172.21.15.88:0/2088319144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a400b6d50 0x7f7a40161460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.848+0000 7f7a455cd640 1 --2- 172.21.15.88:0/2088319144 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a4006e5b0 0x7f7a40152090 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.848+0000 7f7a455cd640 1 -- 172.21.15.88:0/2088319144 >> 172.21.15.88:0/2088319144 conn(0x7f7a400d24d0 msgr2=0x7f7a4006d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:16.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.848+0000 7f7a455cd640 1 -- 172.21.15.88:0/2088319144 shutdown_connections 2024-09-16T01:36:16.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.848+0000 7f7a455cd640 1 -- 172.21.15.88:0/2088319144 wait complete. 2024-09-16T01:36:16.861 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_inject_rename_corrupt_dentry_first 1.0 2024-09-16T01:36:16.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.960+0000 7fcbf69f6640 1 Processor -- start 2024-09-16T01:36:16.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.961+0000 7fcbf69f6640 1 -- start start 2024-09-16T01:36:16.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.961+0000 7fcbf69f6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcbf015e7a0 0x7fcbf006caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.961+0000 7fcbf69f6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcbf00b68f0 0x7fcbf00d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.961+0000 7fcbf69f6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbf00cf860 con 0x7fcbf00cf1f0 2024-09-16T01:36:16.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.961+0000 7fcbf69f6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fcbf00d2ca0 con 0x7fcbf00b68f0 2024-09-16T01:36:16.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.961+0000 7fcbf69f6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcbf00cfb40 con 0x7fcbf015e7a0 2024-09-16T01:36:16.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.962+0000 7fcbeffff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcbf015e7a0 0x7fcbf006caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.962+0000 7fcbf4f93640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcbf00b68f0 0x7fcbf00d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.962+0000 7fcbeffff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcbf015e7a0 0x7fcbf006caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:35302/0 (socket says 172.21.15.88:35302) 2024-09-16T01:36:16.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.962+0000 7fcbef7fe640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fcbf00cf1f0 0x7fcbf00b61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:52736/0 (socket says 172.21.15.88:52736) 2024-09-16T01:36:16.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.962+0000 7fcbf4f93640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcbf00b68f0 0x7fcbf00d3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:60306/0 (socket says 172.21.15.88:60306) 2024-09-16T01:36:16.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.962+0000 7fcbeffff640 1 -- 172.21.15.88:0/4051578775 learned_addr learned my addr 172.21.15.88:0/4051578775 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:16.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.962+0000 7fcbf4f93640 1 -- 172.21.15.88:0/4051578775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcbf015e7a0 msgr2=0x7fcbf006caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:16.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.962+0000 7fcbf4f93640 1 --2- 172.21.15.88:0/4051578775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcbf015e7a0 0x7fcbf006caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.962+0000 7fcbf4f93640 1 -- 172.21.15.88:0/4051578775 >> v1:172.21.15.154:6789/0 conn(0x7fcbf00cf1f0 legacy=0x7fcbf00b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:16.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.962+0000 7fcbf4f93640 1 -- 172.21.15.88:0/4051578775 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbf00cfe20 con 0x7fcbf00b68f0 2024-09-16T01:36:16.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.962+0000 7fcbeeffd640 1 -- 172.21.15.88:0/4051578775 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3720290870 0 0) 0x7fcbf00cf860 con 0x7fcbf00cf1f0 2024-09-16T01:36:16.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.962+0000 7fcbeffff640 1 --2- 172.21.15.88:0/4051578775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcbf015e7a0 0x7fcbf006caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:36:16.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.963+0000 7fcbf4f93640 1 --2- 172.21.15.88:0/4051578775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcbf00b68f0 0x7fcbf00d3410 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009d90 tx=0x7fcbe802a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3a788e6297f108c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:16.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.963+0000 7fcbeeffd640 1 -- 172.21.15.88:0/4051578775 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe8031600 con 0x7fcbf00b68f0 2024-09-16T01:36:16.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.963+0000 7fcbeeffd640 1 -- 172.21.15.88:0/4051578775 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbe80317a0 con 0x7fcbf00b68f0 2024-09-16T01:36:16.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.963+0000 7fcbeeffd640 1 -- 172.21.15.88:0/4051578775 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe8031b70 con 0x7fcbf00b68f0 2024-09-16T01:36:16.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.964+0000 7fcbf69f6640 1 -- 172.21.15.88:0/4051578775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcbf00b68f0 msgr2=0x7fcbf00d3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:16.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.964+0000 7fcbf69f6640 1 --2- 172.21.15.88:0/4051578775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcbf00b68f0 0x7fcbf00d3410 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009d90 tx=0x7fcbe802a5d0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.964+0000 7fcbf69f6640 1 -- 172.21.15.88:0/4051578775 shutdown_connections 2024-09-16T01:36:16.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.964+0000 7fcbf69f6640 1 --2- 172.21.15.88:0/4051578775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcbf00b68f0 0x7fcbf00d3410 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.964+0000 7fcbf69f6640 1 --2- 172.21.15.88:0/4051578775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcbf015e7a0 0x7fcbf006caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.964+0000 7fcbf69f6640 1 -- 172.21.15.88:0/4051578775 >> 172.21.15.88:0/4051578775 conn(0x7fcbf00d11e0 msgr2=0x7fcbf00d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:16.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.964+0000 7fcbf69f6640 1 -- 172.21.15.88:0/4051578775 shutdown_connections 2024-09-16T01:36:16.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.964+0000 7fcbf69f6640 1 -- 172.21.15.88:0/4051578775 wait complete. 2024-09-16T01:36:16.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.965+0000 7fcbf69f6640 1 Processor -- start 2024-09-16T01:36:16.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.965+0000 7fcbf69f6640 1 -- start start 2024-09-16T01:36:16.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.966+0000 7fcbf69f6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcbf00b68f0 0x7fcbf00cab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.966+0000 7fcbf69f6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcbf00cf1f0 0x7fcbf0157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.966+0000 7fcbf69f6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbf015e7a0 0x7fcbf00d3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.966+0000 7fcbf69f6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf006cfe0 con 0x7fcbf015e7a0 2024-09-16T01:36:16.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.966+0000 7fcbf69f6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf00b7a00 con 0x7fcbf00cf1f0 2024-09-16T01:36:16.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.966+0000 7fcbf69f6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcbf015eb80 con 0x7fcbf00b68f0 2024-09-16T01:36:16.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.966+0000 7fcbf4f93640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbf015e7a0 0x7fcbf00d3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.966+0000 7fcbf4f93640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbf015e7a0 0x7fcbf00d3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48862/0 (socket says 172.21.15.88:48862) 2024-09-16T01:36:16.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.966+0000 7fcbf4f93640 1 -- 172.21.15.88:0/844381517 learned_addr learned my addr 172.21.15.88:0/844381517 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:16.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.966+0000 7fcbef7fe640 1 --2- 172.21.15.88:0/844381517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcbf00cf1f0 0x7fcbf0157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.966+0000 7fcbeffff640 1 --2- 172.21.15.88:0/844381517 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcbf00b68f0 0x7fcbf00cab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.966+0000 7fcbf4f93640 1 -- 172.21.15.88:0/844381517 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcbf00b68f0 msgr2=0x7fcbf00cab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:16.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.966+0000 7fcbf4f93640 1 --2- 172.21.15.88:0/844381517 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcbf00b68f0 0x7fcbf00cab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.966+0000 7fcbf4f93640 1 -- 172.21.15.88:0/844381517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcbf00cf1f0 msgr2=0x7fcbf0157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:16.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.966+0000 7fcbf4f93640 1 --2- 172.21.15.88:0/844381517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcbf00cf1f0 0x7fcbf0157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:16.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.966+0000 7fcbf4f93640 1 -- 172.21.15.88:0/844381517 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbf00cb830 con 0x7fcbf015e7a0 2024-09-16T01:36:16.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.966+0000 7fcbeffff640 1 --2- 172.21.15.88:0/844381517 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcbf00b68f0 0x7fcbf00cab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:16.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.967+0000 7fcbf4f93640 1 --2- 172.21.15.88:0/844381517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbf015e7a0 0x7fcbf00d3700 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009eb0 tx=0x7fcbe8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:16.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.967+0000 7fcbecff9640 1 -- 172.21.15.88:0/844381517 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe803f070 con 0x7fcbf015e7a0 2024-09-16T01:36:16.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.967+0000 7fcbecff9640 1 -- 172.21.15.88:0/844381517 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbe8030b60 con 0x7fcbf015e7a0 2024-09-16T01:36:16.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.967+0000 7fcbecff9640 1 -- 172.21.15.88:0/844381517 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe8009450 con 0x7fcbf015e7a0 2024-09-16T01:36:16.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.967+0000 7fcbf69f6640 1 -- 172.21.15.88:0/844381517 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbf00cbb10 con 0x7fcbf015e7a0 2024-09-16T01:36:16.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.967+0000 7fcbf69f6640 1 -- 172.21.15.88:0/844381517 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbf00cc3b0 con 0x7fcbf015e7a0 2024-09-16T01:36:16.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.968+0000 7fcbf69f6640 1 -- 172.21.15.88:0/844381517 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbf006caa0 con 0x7fcbf015e7a0 2024-09-16T01:36:16.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.968+0000 7fcbecff9640 1 -- 172.21.15.88:0/844381517 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fcbe803b410 con 0x7fcbf015e7a0 2024-09-16T01:36:16.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.969+0000 7fcbecff9640 1 --2- 172.21.15.88:0/844381517 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcbdc06c2b0 0x7fcbdc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:16.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.969+0000 7fcbecff9640 1 -- 172.21.15.88:0/844381517 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fcbe80ac8d0 con 0x7fcbf015e7a0 2024-09-16T01:36:16.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.969+0000 7fcbeffff640 1 --2- 172.21.15.88:0/844381517 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcbdc06c2b0 0x7fcbdc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:16.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.969+0000 7fcbeffff640 1 --2- 172.21.15.88:0/844381517 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcbdc06c2b0 0x7fcbdc06e770 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0009630 tx=0x7fcbe0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:16.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:16.973+0000 7fcbecff9640 1 -- 172.21.15.88:0/844381517 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fcbe807b040 con 0x7fcbf015e7a0 2024-09-16T01:36:17.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:17.140+0000 7fcbf69f6640 1 -- 172.21.15.88:0/844381517 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command([{prefix=config set, name=mds_inject_rename_corrupt_dentry_first}] v 0) -- 0x7fcbf00d2a80 con 0x7fcbf015e7a0 2024-09-16T01:36:17.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:17.145+0000 7fcbecff9640 1 -- 172.21.15.88:0/844381517 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_inject_rename_corrupt_dentry_first}]=0 v14)=0 v14) ==== 138+0+0 (secure 0 0 0) 0x7fcbe807a940 con 0x7fcbf015e7a0 2024-09-16T01:36:17.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:17.147+0000 7fcbf69f6640 1 -- 172.21.15.88:0/844381517 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcbdc06c2b0 msgr2=0x7fcbdc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:17.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:17.148+0000 7fcbf69f6640 1 --2- 172.21.15.88:0/844381517 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcbdc06c2b0 0x7fcbdc06e770 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0009630 tx=0x7fcbe0008040 comp rx=0 tx=0).stop 2024-09-16T01:36:17.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:17.148+0000 7fcbf69f6640 1 -- 172.21.15.88:0/844381517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbf015e7a0 msgr2=0x7fcbf00d3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:17.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:17.148+0000 7fcbf69f6640 1 --2- 172.21.15.88:0/844381517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbf015e7a0 0x7fcbf00d3700 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009eb0 tx=0x7fcbe8004a40 comp rx=0 tx=0).stop 2024-09-16T01:36:17.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:17.148+0000 7fcbf69f6640 1 -- 172.21.15.88:0/844381517 shutdown_connections 2024-09-16T01:36:17.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:17.148+0000 7fcbf69f6640 1 --2- 172.21.15.88:0/844381517 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcbdc06c2b0 0x7fcbdc06e770 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:17.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:17.148+0000 7fcbf69f6640 1 --2- 172.21.15.88:0/844381517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbf015e7a0 0x7fcbf00d3700 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:17.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:17.148+0000 7fcbf69f6640 1 --2- 172.21.15.88:0/844381517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcbf00cf1f0 0x7fcbf0157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:17.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:17.148+0000 7fcbf69f6640 1 --2- 172.21.15.88:0/844381517 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcbf00b68f0 0x7fcbf00cab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:17.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:17.148+0000 7fcbf69f6640 1 -- 172.21.15.88:0/844381517 >> 172.21.15.88:0/844381517 conn(0x7fcbf00d11e0 msgr2=0x7fcbf00d4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:17.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:17.148+0000 7fcbf69f6640 1 -- 172.21.15.88:0/844381517 shutdown_connections 2024-09-16T01:36:17.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:17.148+0000 7fcbf69f6640 1 -- 172.21.15.88:0/844381517 wait complete. 2024-09-16T01:36:22.163 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'mv a/b a/c; sync .') 2024-09-16T01:36:22.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:36:22.223 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:36:22.221+0000 7efdf3b5f640 -1 mds.0.cache.den(0x10000000000 c) newly corrupt dentry to be committed: [dentry #0x1/a/c [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.5143:44 nref=7 cr=0x55ba84962e00)) (dversion lock w=1 last_client=5143) pv=4 v=3 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x55ba8484b400] 2024-09-16T01:36:22.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:36:22.232 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:36:22.232 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:36:22.249 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:36:22.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.394+0000 7f639c07b640 1 Processor -- start 2024-09-16T01:36:22.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.394+0000 7f639c07b640 1 -- start start 2024-09-16T01:36:22.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.396+0000 7f639c07b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f639415e6f0 0x7f639406fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:22.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.396+0000 7f639c07b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63940b54d0 0x7f639415f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:22.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.396+0000 7f639c07b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f639406e990 con 0x7f63940b6d40 2024-09-16T01:36:22.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.396+0000 7f639c07b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f639406ec70 con 0x7f639415e6f0 2024-09-16T01:36:22.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.396+0000 7f639c07b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f639415ec70 con 0x7f63940b54d0 2024-09-16T01:36:22.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.396+0000 7f6399e17640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f639415e6f0 0x7f639406fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:22.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.396+0000 7f6399e17640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f639415e6f0 0x7f639406fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60946/0 (socket says 172.21.15.88:60946) 2024-09-16T01:36:22.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.396+0000 7f6399e17640 1 -- 172.21.15.88:0/2804950787 learned_addr learned my addr 172.21.15.88:0/2804950787 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:22.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.396+0000 7f6399e17640 1 -- 172.21.15.88:0/2804950787 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63940b54d0 msgr2=0x7f639415f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:22.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.396+0000 7f6399e17640 1 --2- 172.21.15.88:0/2804950787 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63940b54d0 0x7f639415f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:22.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.396+0000 7f6399e17640 1 -- 172.21.15.88:0/2804950787 >> v1:172.21.15.154:6789/0 conn(0x7f63940b6d40 legacy=0x7f63940b71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:22.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.396+0000 7f6399e17640 1 -- 172.21.15.88:0/2804950787 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f639415ef50 con 0x7f639415e6f0 2024-09-16T01:36:22.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.396+0000 7f6399e17640 1 --2- 172.21.15.88:0/2804950787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f639415e6f0 0x7f639406fd00 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f6390009870 tx=0x7f639002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=474029266d9be850 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:22.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.396+0000 7f6398e15640 1 -- 172.21.15.88:0/2804950787 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6390037070 con 0x7f639415e6f0 2024-09-16T01:36:22.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.396+0000 7f6398e15640 1 -- 172.21.15.88:0/2804950787 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6390032440 con 0x7f639415e6f0 2024-09-16T01:36:22.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.396+0000 7f6398e15640 1 -- 172.21.15.88:0/2804950787 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6390032810 con 0x7f639415e6f0 2024-09-16T01:36:22.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.396+0000 7f639c07b640 1 -- 172.21.15.88:0/2804950787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f639415e6f0 msgr2=0x7f639406fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:22.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.396+0000 7f639c07b640 1 --2- 172.21.15.88:0/2804950787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f639415e6f0 0x7f639406fd00 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f6390009870 tx=0x7f639002a210 comp rx=0 tx=0).stop 2024-09-16T01:36:22.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.396+0000 7f639c07b640 1 -- 172.21.15.88:0/2804950787 shutdown_connections 2024-09-16T01:36:22.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.396+0000 7f639c07b640 1 --2- 172.21.15.88:0/2804950787 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63940b54d0 0x7f639415f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:22.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.396+0000 7f639c07b640 1 --2- 172.21.15.88:0/2804950787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f639415e6f0 0x7f639406fd00 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:22.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.396+0000 7f639c07b640 1 -- 172.21.15.88:0/2804950787 >> 172.21.15.88:0/2804950787 conn(0x7f63940d2430 msgr2=0x7f63940d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:22.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.397+0000 7f639c07b640 1 -- 172.21.15.88:0/2804950787 shutdown_connections 2024-09-16T01:36:22.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.397+0000 7f639c07b640 1 -- 172.21.15.88:0/2804950787 wait complete. 2024-09-16T01:36:22.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.398+0000 7f639c07b640 1 Processor -- start 2024-09-16T01:36:22.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.398+0000 7f639c07b640 1 -- start start 2024-09-16T01:36:22.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.398+0000 7f639c07b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63940b54d0 0x7f6394142a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:22.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.398+0000 7f639c07b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63940b6d40 0x7f639415f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:22.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.398+0000 7f639c07b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f639415e6f0 0x7f63941450b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:22.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.398+0000 7f639c07b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f639406e730 con 0x7f639415e6f0 2024-09-16T01:36:22.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.398+0000 7f639c07b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f639406e5b0 con 0x7f63940b54d0 2024-09-16T01:36:22.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.398+0000 7f639c07b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6394070830 con 0x7f63940b6d40 2024-09-16T01:36:22.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.398+0000 7f639a618640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f639415e6f0 0x7f63941450b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:22.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.399+0000 7f639a618640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f639415e6f0 0x7f63941450b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60958/0 (socket says 172.21.15.88:60958) 2024-09-16T01:36:22.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.399+0000 7f639a618640 1 -- 172.21.15.88:0/3904699054 learned_addr learned my addr 172.21.15.88:0/3904699054 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:22.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.399+0000 7f6399e17640 1 --2- 172.21.15.88:0/3904699054 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63940b54d0 0x7f6394142a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.399+0000 7f6399616640 1 --2- 172.21.15.88:0/3904699054 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63940b6d40 0x7f639415f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.399+0000 7f639a618640 1 -- 172.21.15.88:0/3904699054 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63940b6d40 msgr2=0x7f639415f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.399+0000 7f639a618640 1 --2- 172.21.15.88:0/3904699054 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63940b6d40 0x7f639415f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.399+0000 7f639a618640 1 -- 172.21.15.88:0/3904699054 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63940b54d0 msgr2=0x7f6394142a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.399+0000 7f639a618640 1 --2- 172.21.15.88:0/3904699054 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63940b54d0 0x7f6394142a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.399+0000 7f639a618640 1 -- 172.21.15.88:0/3904699054 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f639415c140 con 0x7f639415e6f0 2024-09-16T01:36:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.399+0000 7f639a618640 1 --2- 172.21.15.88:0/3904699054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f639415e6f0 0x7f63941450b0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f638c00c780 tx=0x7f638c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.399+0000 7f638a7fc640 1 -- 172.21.15.88:0/3904699054 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c017070 con 0x7f639415e6f0 2024-09-16T01:36:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.399+0000 7f638a7fc640 1 -- 172.21.15.88:0/3904699054 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f638c003da0 con 0x7f639415e6f0 2024-09-16T01:36:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.399+0000 7f639c07b640 1 -- 172.21.15.88:0/3904699054 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f639415c420 con 0x7f639415e6f0 2024-09-16T01:36:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.399+0000 7f638a7fc640 1 -- 172.21.15.88:0/3904699054 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c0052b0 con 0x7f639415e6f0 2024-09-16T01:36:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.399+0000 7f639c07b640 1 -- 172.21.15.88:0/3904699054 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f639415c9e0 con 0x7f639415e6f0 2024-09-16T01:36:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.400+0000 7f639c07b640 1 -- 172.21.15.88:0/3904699054 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f639406c6c0 con 0x7f639415e6f0 2024-09-16T01:36:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.401+0000 7f638a7fc640 1 -- 172.21.15.88:0/3904699054 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f638c01a070 con 0x7f639415e6f0 2024-09-16T01:36:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.401+0000 7f638a7fc640 1 --2- 172.21.15.88:0/3904699054 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f637806c2b0 0x7f637806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.401+0000 7f638a7fc640 1 -- 172.21.15.88:0/3904699054 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f638c08ab10 con 0x7f639415e6f0 2024-09-16T01:36:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.401+0000 7f6399e17640 1 --2- 172.21.15.88:0/3904699054 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f637806c2b0 0x7f637806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.401+0000 7f6399e17640 1 --2- 172.21.15.88:0/3904699054 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f637806c2b0 0x7f637806e770 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6390034000 tx=0x7f639002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:22.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.405+0000 7f638a7fc640 1 -- 172.21.15.88:0/3904699054 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f638c0591f0 con 0x7f639415e6f0 2024-09-16T01:36:22.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.583+0000 7f639c07b640 1 -- 172.21.15.88:0/3904699054 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f639406d910 con 0x7f639415e6f0 2024-09-16T01:36:22.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.583+0000 7f638a7fc640 1 -- 172.21.15.88:0/3904699054 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v45) ==== 80+0+0 (secure 0 0 0) 0x7f638c058af0 con 0x7f639415e6f0 2024-09-16T01:36:22.586 INFO:teuthology.orchestra.run.smithi088.stderr:listed 0 entries 2024-09-16T01:36:22.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.585+0000 7f639c07b640 1 -- 172.21.15.88:0/3904699054 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f637806c2b0 msgr2=0x7f637806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:22.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.585+0000 7f639c07b640 1 --2- 172.21.15.88:0/3904699054 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f637806c2b0 0x7f637806e770 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6390034000 tx=0x7f639002e000 comp rx=0 tx=0).stop 2024-09-16T01:36:22.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.585+0000 7f639c07b640 1 -- 172.21.15.88:0/3904699054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f639415e6f0 msgr2=0x7f63941450b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:22.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.585+0000 7f639c07b640 1 --2- 172.21.15.88:0/3904699054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f639415e6f0 0x7f63941450b0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f638c00c780 tx=0x7f638c00cc40 comp rx=0 tx=0).stop 2024-09-16T01:36:22.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.585+0000 7f639c07b640 1 -- 172.21.15.88:0/3904699054 shutdown_connections 2024-09-16T01:36:22.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.585+0000 7f639c07b640 1 --2- 172.21.15.88:0/3904699054 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f637806c2b0 0x7f637806e770 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:22.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.585+0000 7f639c07b640 1 --2- 172.21.15.88:0/3904699054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f639415e6f0 0x7f63941450b0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:22.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.585+0000 7f639c07b640 1 --2- 172.21.15.88:0/3904699054 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63940b6d40 0x7f639415f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:22.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.585+0000 7f639c07b640 1 --2- 172.21.15.88:0/3904699054 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63940b54d0 0x7f6394142a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:22.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.585+0000 7f639c07b640 1 -- 172.21.15.88:0/3904699054 >> 172.21.15.88:0/3904699054 conn(0x7f63940d2430 msgr2=0x7f639415abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:22.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.585+0000 7f639c07b640 1 -- 172.21.15.88:0/3904699054 shutdown_connections 2024-09-16T01:36:22.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.585+0000 7f639c07b640 1 -- 172.21.15.88:0/3904699054 wait complete. 2024-09-16T01:36:22.599 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-16T01:36:22.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:22.599 DEBUG:teuthology.orchestra.run.smithi088:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-16T01:36:22.628 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:36:22.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:22.628 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:36:22.685 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:36:22.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.821+0000 7fc1a419f640 1 Processor -- start 2024-09-16T01:36:22.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.821+0000 7fc1a419f640 1 -- start start 2024-09-16T01:36:22.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.822+0000 7fc1a419f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc19c0b4d00 0x7fc19c0b5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:22.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.822+0000 7fc1a419f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc19c0cd780 0x7fc19c0cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:22.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.822+0000 7fc1a419f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc19c0ce4f0 con 0x7fc19c0b4920 2024-09-16T01:36:22.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.822+0000 7fc1a419f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc19c0cf100 con 0x7fc19c0b4d00 2024-09-16T01:36:22.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.822+0000 7fc1a419f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc19c0ce7d0 con 0x7fc19c0cd780 2024-09-16T01:36:22.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.822+0000 7fc1a173a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc19c0b4d00 0x7fc19c0b5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:22.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.822+0000 7fc1a173a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc19c0b4d00 0x7fc19c0b5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60966/0 (socket says 172.21.15.88:60966) 2024-09-16T01:36:22.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.822+0000 7fc1a173a640 1 -- 172.21.15.88:0/278821222 learned_addr learned my addr 172.21.15.88:0/278821222 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:22.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.822+0000 7fc1a273c640 1 --2- 172.21.15.88:0/278821222 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc19c0cd780 0x7fc19c0cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:22.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.822+0000 7fc1a173a640 1 -- 172.21.15.88:0/278821222 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc19c0cd780 msgr2=0x7fc19c0cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:22.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.822+0000 7fc1a173a640 1 --2- 172.21.15.88:0/278821222 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc19c0cd780 0x7fc19c0cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:22.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.822+0000 7fc1a173a640 1 -- 172.21.15.88:0/278821222 >> v1:172.21.15.154:6789/0 conn(0x7fc19c0b4920 legacy=0x7fc19c15b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:22.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.822+0000 7fc1a173a640 1 -- 172.21.15.88:0/278821222 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc19c0ceab0 con 0x7fc19c0b4d00 2024-09-16T01:36:22.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.822+0000 7fc1a273c640 1 --2- 172.21.15.88:0/278821222 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc19c0cd780 0x7fc19c0cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:22.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.823+0000 7fc1a173a640 1 --2- 172.21.15.88:0/278821222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc19c0b4d00 0x7fc19c0b5170 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc18c00bb10 tx=0x7fc18c02c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecc2b34c120ab81 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:22.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.823+0000 7fc1a0f39640 1 -- 172.21.15.88:0/278821222 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18c036020 con 0x7fc19c0b4d00 2024-09-16T01:36:22.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.823+0000 7fc1a0f39640 1 -- 172.21.15.88:0/278821222 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc18c032820 con 0x7fc19c0b4d00 2024-09-16T01:36:22.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.823+0000 7fc1a0f39640 1 -- 172.21.15.88:0/278821222 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18c032bf0 con 0x7fc19c0b4d00 2024-09-16T01:36:22.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.824+0000 7fc1a419f640 1 -- 172.21.15.88:0/278821222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc19c0b4d00 msgr2=0x7fc19c0b5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:22.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.824+0000 7fc1a419f640 1 --2- 172.21.15.88:0/278821222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc19c0b4d00 0x7fc19c0b5170 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc18c00bb10 tx=0x7fc18c02c920 comp rx=0 tx=0).stop 2024-09-16T01:36:22.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.824+0000 7fc1a419f640 1 -- 172.21.15.88:0/278821222 shutdown_connections 2024-09-16T01:36:22.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.824+0000 7fc1a419f640 1 --2- 172.21.15.88:0/278821222 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc19c0cd780 0x7fc19c0cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:22.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.824+0000 7fc1a419f640 1 --2- 172.21.15.88:0/278821222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc19c0b4d00 0x7fc19c0b5170 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:22.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.824+0000 7fc1a419f640 1 -- 172.21.15.88:0/278821222 >> 172.21.15.88:0/278821222 conn(0x7fc19c06bc10 msgr2=0x7fc19c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:22.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.824+0000 7fc1a419f640 1 -- 172.21.15.88:0/278821222 shutdown_connections 2024-09-16T01:36:22.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.824+0000 7fc1a419f640 1 -- 172.21.15.88:0/278821222 wait complete. 2024-09-16T01:36:22.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.825+0000 7fc1a419f640 1 Processor -- start 2024-09-16T01:36:22.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.825+0000 7fc1a419f640 1 -- start start 2024-09-16T01:36:22.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.826+0000 7fc1a419f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc19c0b4920 0x7fc19c1f0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:22.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.826+0000 7fc1a419f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc19c0b4d00 0x7fc19c15a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:22.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.826+0000 7fc1a419f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc19c0cd780 0x7fc19c150ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:22.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.826+0000 7fc1a419f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc19c0ce110 con 0x7fc19c0cd780 2024-09-16T01:36:22.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.826+0000 7fc1a419f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc19c0b56b0 con 0x7fc19c0b4920 2024-09-16T01:36:22.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.826+0000 7fc1a419f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc19c0ce290 con 0x7fc19c0b4d00 2024-09-16T01:36:22.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.826+0000 7fc1a273c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc19c0cd780 0x7fc19c150ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:22.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.826+0000 7fc1a273c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc19c0cd780 0x7fc19c150ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60974/0 (socket says 172.21.15.88:60974) 2024-09-16T01:36:22.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.826+0000 7fc1a273c640 1 -- 172.21.15.88:0/2184649982 learned_addr learned my addr 172.21.15.88:0/2184649982 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:22.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.826+0000 7fc1a1f3b640 1 --2- 172.21.15.88:0/2184649982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc19c0b4920 0x7fc19c1f0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:22.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.826+0000 7fc1a173a640 1 --2- 172.21.15.88:0/2184649982 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc19c0b4d00 0x7fc19c15a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:22.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.826+0000 7fc1a273c640 1 -- 172.21.15.88:0/2184649982 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc19c0b4d00 msgr2=0x7fc19c15a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:22.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.826+0000 7fc1a273c640 1 --2- 172.21.15.88:0/2184649982 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc19c0b4d00 0x7fc19c15a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:22.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.826+0000 7fc1a273c640 1 -- 172.21.15.88:0/2184649982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc19c0b4920 msgr2=0x7fc19c1f0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:22.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.826+0000 7fc1a273c640 1 --2- 172.21.15.88:0/2184649982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc19c0b4920 0x7fc19c1f0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:22.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.826+0000 7fc1a273c640 1 -- 172.21.15.88:0/2184649982 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc19c1f12b0 con 0x7fc19c0cd780 2024-09-16T01:36:22.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.826+0000 7fc1a173a640 1 --2- 172.21.15.88:0/2184649982 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc19c0b4d00 0x7fc19c15a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:22.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.826+0000 7fc1a273c640 1 --2- 172.21.15.88:0/2184649982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc19c0cd780 0x7fc19c150ee0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fc194007e90 tx=0x7fc19400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:22.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.827+0000 7fc1927fc640 1 -- 172.21.15.88:0/2184649982 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19401a070 con 0x7fc19c0cd780 2024-09-16T01:36:22.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.827+0000 7fc1927fc640 1 -- 172.21.15.88:0/2184649982 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc194003990 con 0x7fc19c0cd780 2024-09-16T01:36:22.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.827+0000 7fc1a419f640 1 -- 172.21.15.88:0/2184649982 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc19c157f40 con 0x7fc19c0cd780 2024-09-16T01:36:22.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.827+0000 7fc1927fc640 1 -- 172.21.15.88:0/2184649982 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc194003d60 con 0x7fc19c0cd780 2024-09-16T01:36:22.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.827+0000 7fc1a419f640 1 -- 172.21.15.88:0/2184649982 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc19c158950 con 0x7fc19c0cd780 2024-09-16T01:36:22.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.828+0000 7fc1a419f640 1 -- 172.21.15.88:0/2184649982 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc19c0b5170 con 0x7fc19c0cd780 2024-09-16T01:36:22.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.828+0000 7fc1927fc640 1 -- 172.21.15.88:0/2184649982 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc19401d070 con 0x7fc19c0cd780 2024-09-16T01:36:22.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.829+0000 7fc1927fc640 1 --2- 172.21.15.88:0/2184649982 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc18006c2b0 0x7fc18006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:22.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.829+0000 7fc1a1f3b640 1 --2- 172.21.15.88:0/2184649982 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc18006c2b0 0x7fc18006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:22.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.829+0000 7fc1927fc640 1 -- 172.21.15.88:0/2184649982 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fc19408b560 con 0x7fc19c0cd780 2024-09-16T01:36:22.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.829+0000 7fc1a1f3b640 1 --2- 172.21.15.88:0/2184649982 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc18006c2b0 0x7fc18006e770 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc198001780 tx=0x7fc198009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:22.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:22.834+0000 7fc1927fc640 1 -- 172.21.15.88:0/2184649982 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc19c1f12b0 con 0x7fc19c0cd780 2024-09-16T01:36:23.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.032+0000 7fc1a419f640 1 -- 172.21.15.88:0/2184649982 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc19c15bca0 con 0x7fc19c0cd780 2024-09-16T01:36:23.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.032+0000 7fc1927fc640 1 -- 172.21.15.88:0/2184649982 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5226 (secure 0 0 0) 0x7fc194010070 con 0x7fc19c0cd780 2024-09-16T01:36:23.035 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:23.035 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":25,"btime":"2024-09-16T01:36:08:365397+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4877,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1769276566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":1769276566},{"type":"v1","addr":"172.21.15.154:6835","nonce":1769276566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/725621378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":725621378},{"type":"v1","addr":"172.21.15.154:6834","nonce":725621378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4199111872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4199111872},{"type":"v1","addr":"172.21.15.88:6837","nonce":4199111872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:05.691540+0000","modified":"2024-09-16T01:36:07.693602+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5406},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5406":{"gid":5406,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6836/1610810484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1610810484},{"type":"v1","addr":"172.21.15.88:6836","nonce":1610810484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5406,"qdb_cluster":[5406]},"id":3}]} 2024-09-16T01:36:23.035 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 25 2024-09-16T01:36:23.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.034+0000 7fc1a419f640 1 -- 172.21.15.88:0/2184649982 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc18006c2b0 msgr2=0x7fc18006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:23.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.034+0000 7fc1a419f640 1 --2- 172.21.15.88:0/2184649982 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc18006c2b0 0x7fc18006e770 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc198001780 tx=0x7fc198009040 comp rx=0 tx=0).stop 2024-09-16T01:36:23.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.034+0000 7fc1a419f640 1 -- 172.21.15.88:0/2184649982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc19c0cd780 msgr2=0x7fc19c150ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:23.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.034+0000 7fc1a419f640 1 --2- 172.21.15.88:0/2184649982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc19c0cd780 0x7fc19c150ee0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fc194007e90 tx=0x7fc19400d4a0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.034+0000 7fc1a419f640 1 -- 172.21.15.88:0/2184649982 shutdown_connections 2024-09-16T01:36:23.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.034+0000 7fc1a419f640 1 --2- 172.21.15.88:0/2184649982 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc18006c2b0 0x7fc18006e770 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.034+0000 7fc1a419f640 1 --2- 172.21.15.88:0/2184649982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc19c0cd780 0x7fc19c150ee0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.034+0000 7fc1a419f640 1 --2- 172.21.15.88:0/2184649982 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc19c0b4d00 0x7fc19c15a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.034+0000 7fc1a419f640 1 --2- 172.21.15.88:0/2184649982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc19c0b4920 0x7fc19c1f0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.034+0000 7fc1a419f640 1 -- 172.21.15.88:0/2184649982 >> 172.21.15.88:0/2184649982 conn(0x7fc19c06bc10 msgr2=0x7fc19c15c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:23.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.034+0000 7fc1a419f640 1 -- 172.21.15.88:0/2184649982 shutdown_connections 2024-09-16T01:36:23.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.034+0000 7fc1a419f640 1 -- 172.21.15.88:0/2184649982 wait complete. 2024-09-16T01:36:23.048 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:36:23.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.143+0000 7fb31ff23640 1 Processor -- start 2024-09-16T01:36:23.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.143+0000 7fb31ff23640 1 -- start start 2024-09-16T01:36:23.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.144+0000 7fb31ff23640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb3180b64e0 0x7fb318167110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:23.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.144+0000 7fb31ff23640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3180b49c0 con 0x7fb31815c550 2024-09-16T01:36:23.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.144+0000 7fb31ff23640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3180b4cf0 con 0x7fb31815c170 2024-09-16T01:36:23.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.144+0000 7fb31ff23640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb3180b4fd0 con 0x7fb3180b64e0 2024-09-16T01:36:23.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.144+0000 7fb31d4be640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb31815c550 0x7fb3180b5dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42722/0 (socket says 172.21.15.88:42722) 2024-09-16T01:36:23.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.144+0000 7fb31d4be640 1 -- 172.21.15.88:0/3004424418 learned_addr learned my addr 172.21.15.88:0/3004424418 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:23.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31e4c0640 1 --2- 172.21.15.88:0/3004424418 >> v2:172.21.15.154:3300/0 conn(0x7fb3180b64e0 0x7fb318167110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:23.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31ccbd640 1 -- 172.21.15.88:0/3004424418 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3765021035 0 0) 0x7fb3180b49c0 con 0x7fb31815c550 2024-09-16T01:36:23.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31ccbd640 1 -- 172.21.15.88:0/3004424418 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb300003520 con 0x7fb31815c550 2024-09-16T01:36:23.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31ccbd640 1 -- 172.21.15.88:0/3004424418 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1981611585 0 0) 0x7fb3180b4cf0 con 0x7fb31815c170 2024-09-16T01:36:23.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31ccbd640 1 -- 172.21.15.88:0/3004424418 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3180b49c0 con 0x7fb31815c170 2024-09-16T01:36:23.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31ccbd640 1 -- 172.21.15.88:0/3004424418 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2677045704 0 0) 0x7fb300003520 con 0x7fb31815c550 2024-09-16T01:36:23.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31ccbd640 1 -- 172.21.15.88:0/3004424418 >> v2:172.21.15.154:3300/0 conn(0x7fb3180b64e0 msgr2=0x7fb318167110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:23.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31ccbd640 1 --2- 172.21.15.88:0/3004424418 >> v2:172.21.15.154:3300/0 conn(0x7fb3180b64e0 0x7fb318167110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31ccbd640 1 -- 172.21.15.88:0/3004424418 >> v1:172.21.15.154:6789/0 conn(0x7fb31815c170 legacy=0x7fb3180cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:23.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31ccbd640 1 -- 172.21.15.88:0/3004424418 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3180b52b0 con 0x7fb31815c550 2024-09-16T01:36:23.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31ccbd640 1 -- 172.21.15.88:0/3004424418 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fb30c001670 con 0x7fb31815c550 2024-09-16T01:36:23.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31ccbd640 1 -- 172.21.15.88:0/3004424418 >> v1:172.21.15.88:6789/0 conn(0x7fb31815c550 legacy=0x7fb3180b5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:23.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31ccbd640 1 --2- 172.21.15.88:0/3004424418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb300003b90 0x7fb300003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:23.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31ccbd640 1 --2- 172.21.15.88:0/3004424418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb300004800 0x7fb300004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:23.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31ccbd640 1 --2- 172.21.15.88:0/3004424418 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb300005210 0x7fb30000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:23.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31ccbd640 1 -- 172.21.15.88:0/3004424418 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb3180b4fd0 con 0x7fb300004800 2024-09-16T01:36:23.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31ccbd640 1 -- 172.21.15.88:0/3004424418 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb30000c140 con 0x7fb300003b90 2024-09-16T01:36:23.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31ccbd640 1 -- 172.21.15.88:0/3004424418 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb30000c340 con 0x7fb300005210 2024-09-16T01:36:23.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31d4be640 1 --2- 172.21.15.88:0/3004424418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb300004800 0x7fb300004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:23.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31dcbf640 1 --2- 172.21.15.88:0/3004424418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb300003b90 0x7fb300003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:23.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31e4c0640 1 --2- 172.21.15.88:0/3004424418 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb300005210 0x7fb30000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:23.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31d4be640 1 -- 172.21.15.88:0/3004424418 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb300005210 msgr2=0x7fb30000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:23.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31d4be640 1 --2- 172.21.15.88:0/3004424418 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb300005210 0x7fb30000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31d4be640 1 -- 172.21.15.88:0/3004424418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb300003b90 msgr2=0x7fb300003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:23.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31d4be640 1 --2- 172.21.15.88:0/3004424418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb300003b90 0x7fb300003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31e4c0640 1 -- 172.21.15.88:0/3004424418 reap_dead start 2024-09-16T01:36:23.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31d4be640 1 -- 172.21.15.88:0/3004424418 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb300003520 con 0x7fb300004800 2024-09-16T01:36:23.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.145+0000 7fb31dcbf640 1 --2- 172.21.15.88:0/3004424418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb300003b90 0x7fb300003f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:23.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.146+0000 7fb31d4be640 1 --2- 172.21.15.88:0/3004424418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb300004800 0x7fb300004c80 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb30c007fc0 tx=0x7fb300005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8576f7edf544c93e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:23.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.146+0000 7fb31ccbd640 1 -- 172.21.15.88:0/3004424418 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30c007be0 con 0x7fb300004800 2024-09-16T01:36:23.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.146+0000 7fb31ccbd640 1 -- 172.21.15.88:0/3004424418 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb30c005af0 con 0x7fb300004800 2024-09-16T01:36:23.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.146+0000 7fb31ccbd640 1 -- 172.21.15.88:0/3004424418 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30c007070 con 0x7fb300004800 2024-09-16T01:36:23.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.147+0000 7fb31ff23640 1 -- 172.21.15.88:0/3004424418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb300004800 msgr2=0x7fb300004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:23.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.147+0000 7fb31ff23640 1 --2- 172.21.15.88:0/3004424418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb300004800 0x7fb300004c80 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb30c007fc0 tx=0x7fb300005830 comp rx=0 tx=0).stop 2024-09-16T01:36:23.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.147+0000 7fb31ff23640 1 -- 172.21.15.88:0/3004424418 shutdown_connections 2024-09-16T01:36:23.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.147+0000 7fb31ff23640 1 --2- 172.21.15.88:0/3004424418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb300004800 0x7fb300004c80 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.147+0000 7fb31ff23640 1 -- 172.21.15.88:0/3004424418 >> 172.21.15.88:0/3004424418 conn(0x7fb31806bc10 msgr2=0x7fb31806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:23.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.147+0000 7fb31ff23640 1 -- 172.21.15.88:0/3004424418 shutdown_connections 2024-09-16T01:36:23.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.147+0000 7fb31ff23640 1 -- 172.21.15.88:0/3004424418 wait complete. 2024-09-16T01:36:23.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.148+0000 7fb31ff23640 1 Processor -- start 2024-09-16T01:36:23.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.148+0000 7fb31ff23640 1 -- start start 2024-09-16T01:36:23.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.149+0000 7fb31ff23640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb300004800 0x7fb31813ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:23.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.149+0000 7fb31ff23640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb318140870 0x7fb31814ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:23.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.149+0000 7fb31ff23640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb31814cf60 0x7fb3180cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:23.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.149+0000 7fb31ff23640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb3180b5b50 con 0x7fb318140870 2024-09-16T01:36:23.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.149+0000 7fb31ff23640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb3180b5870 con 0x7fb300004800 2024-09-16T01:36:23.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.149+0000 7fb31ff23640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb31815c9c0 con 0x7fb31814cf60 2024-09-16T01:36:23.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.149+0000 7fb31d4be640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb318140870 0x7fb31814ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:23.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.149+0000 7fb31d4be640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb318140870 0x7fb31814ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:32772/0 (socket says 172.21.15.88:32772) 2024-09-16T01:36:23.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.149+0000 7fb31d4be640 1 -- 172.21.15.88:0/3951119213 learned_addr learned my addr 172.21.15.88:0/3951119213 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:23.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.149+0000 7fb31dcbf640 1 --2- 172.21.15.88:0/3951119213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb300004800 0x7fb31813ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:23.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.149+0000 7fb31e4c0640 1 --2- 172.21.15.88:0/3951119213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb31814cf60 0x7fb3180cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:23.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.149+0000 7fb31d4be640 1 -- 172.21.15.88:0/3951119213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb31814cf60 msgr2=0x7fb3180cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:23.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.149+0000 7fb31d4be640 1 --2- 172.21.15.88:0/3951119213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb31814cf60 0x7fb3180cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.149+0000 7fb31d4be640 1 -- 172.21.15.88:0/3951119213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb300004800 msgr2=0x7fb31813ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:23.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.149+0000 7fb31d4be640 1 --2- 172.21.15.88:0/3951119213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb300004800 0x7fb31813ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.149+0000 7fb31d4be640 1 -- 172.21.15.88:0/3951119213 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb31814d450 con 0x7fb318140870 2024-09-16T01:36:23.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.149+0000 7fb31dcbf640 1 --2- 172.21.15.88:0/3951119213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb300004800 0x7fb31813ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:23.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.149+0000 7fb31e4c0640 1 --2- 172.21.15.88:0/3951119213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb31814cf60 0x7fb3180cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:23.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.149+0000 7fb31d4be640 1 --2- 172.21.15.88:0/3951119213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb318140870 0x7fb31814ca20 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00c1d0 tx=0x7fb3180b5480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:23.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.150+0000 7fb3167fc640 1 -- 172.21.15.88:0/3951119213 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30c041020 con 0x7fb318140870 2024-09-16T01:36:23.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.150+0000 7fb3167fc640 1 -- 172.21.15.88:0/3951119213 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb30c038040 con 0x7fb318140870 2024-09-16T01:36:23.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.150+0000 7fb31ff23640 1 -- 172.21.15.88:0/3951119213 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3180cdaa0 con 0x7fb318140870 2024-09-16T01:36:23.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.150+0000 7fb3167fc640 1 -- 172.21.15.88:0/3951119213 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30c001b20 con 0x7fb318140870 2024-09-16T01:36:23.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.150+0000 7fb31ff23640 1 -- 172.21.15.88:0/3951119213 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb318140cd0 con 0x7fb318140870 2024-09-16T01:36:23.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.151+0000 7fb31ff23640 1 -- 172.21.15.88:0/3951119213 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3180b5dd0 con 0x7fb318140870 2024-09-16T01:36:23.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.151+0000 7fb3167fc640 1 -- 172.21.15.88:0/3951119213 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb30c044020 con 0x7fb318140870 2024-09-16T01:36:23.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.152+0000 7fb3167fc640 1 --2- 172.21.15.88:0/3951119213 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb30406c2b0 0x7fb30406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:23.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.152+0000 7fb3167fc640 1 -- 172.21.15.88:0/3951119213 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fb30c0ab6b0 con 0x7fb318140870 2024-09-16T01:36:23.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.152+0000 7fb31dcbf640 1 --2- 172.21.15.88:0/3951119213 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb30406c2b0 0x7fb30406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:23.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.152+0000 7fb31dcbf640 1 --2- 172.21.15.88:0/3951119213 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb30406c2b0 0x7fb30406e770 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb3180c6e10 tx=0x7fb308009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:23.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.156+0000 7fb3167fc640 1 -- 172.21.15.88:0/3951119213 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb30c079e30 con 0x7fb318140870 2024-09-16T01:36:23.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.353+0000 7fb31ff23640 1 -- 172.21.15.88:0/3951119213 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb31815fbf0 con 0x7fb318140870 2024-09-16T01:36:23.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.353+0000 7fb3167fc640 1 -- 172.21.15.88:0/3951119213 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5226 (secure 0 0 0) 0x7fb30c079730 con 0x7fb318140870 2024-09-16T01:36:23.356 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:23.356 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":25,"btime":"2024-09-16T01:36:08:365397+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4877,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1769276566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":1769276566},{"type":"v1","addr":"172.21.15.154:6835","nonce":1769276566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/725621378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":725621378},{"type":"v1","addr":"172.21.15.154:6834","nonce":725621378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4199111872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4199111872},{"type":"v1","addr":"172.21.15.88:6837","nonce":4199111872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:05.691540+0000","modified":"2024-09-16T01:36:07.693602+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5406},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5406":{"gid":5406,"name":"c","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6836/1610810484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1610810484},{"type":"v1","addr":"172.21.15.88:6836","nonce":1610810484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5406,"qdb_cluster":[5406]},"id":3}]} 2024-09-16T01:36:23.356 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 25 2024-09-16T01:36:23.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.356+0000 7fb31ff23640 1 -- 172.21.15.88:0/3951119213 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb30406c2b0 msgr2=0x7fb30406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:23.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.356+0000 7fb31ff23640 1 --2- 172.21.15.88:0/3951119213 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb30406c2b0 0x7fb30406e770 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb3180c6e10 tx=0x7fb308009040 comp rx=0 tx=0).stop 2024-09-16T01:36:23.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.356+0000 7fb31ff23640 1 -- 172.21.15.88:0/3951119213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb318140870 msgr2=0x7fb31814ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:23.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.356+0000 7fb31ff23640 1 --2- 172.21.15.88:0/3951119213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb318140870 0x7fb31814ca20 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00c1d0 tx=0x7fb3180b5480 comp rx=0 tx=0).stop 2024-09-16T01:36:23.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.356+0000 7fb31ff23640 1 -- 172.21.15.88:0/3951119213 shutdown_connections 2024-09-16T01:36:23.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.356+0000 7fb31ff23640 1 --2- 172.21.15.88:0/3951119213 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb30406c2b0 0x7fb30406e770 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.356+0000 7fb31ff23640 1 --2- 172.21.15.88:0/3951119213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb31814cf60 0x7fb3180cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.356+0000 7fb31ff23640 1 --2- 172.21.15.88:0/3951119213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb318140870 0x7fb31814ca20 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.356+0000 7fb31ff23640 1 --2- 172.21.15.88:0/3951119213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb300004800 0x7fb31813ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.356+0000 7fb31ff23640 1 -- 172.21.15.88:0/3951119213 >> 172.21.15.88:0/3951119213 conn(0x7fb31806bc10 msgr2=0x7fb3180b3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:23.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.356+0000 7fb31ff23640 1 -- 172.21.15.88:0/3951119213 shutdown_connections 2024-09-16T01:36:23.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.356+0000 7fb31ff23640 1 -- 172.21.15.88:0/3951119213 wait complete. 2024-09-16T01:36:23.369 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json dump tree /a 0 2024-09-16T01:36:23.481 INFO:teuthology.orchestra.run.smithi088.stdout:[{"ino":1099511627776,"rdev":0,"ctime":"2024-09-16T01:36:22.220837+0000","btime":"2024-09-16T01:36:16.352953+0000","mode":16877,"uid":1000,"gid":1333,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":0,"stripe_count":0,"object_size":0,"pool_id":-1,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2024-09-16T01:36:22.220837+0000","atime":"2024-09-16T01:36:16.352953+0000","time_warp_seq":0,"change_attr":3,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"quiesce_block":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"2024-09-16T01:36:22.220837+0000","num_files":0,"num_subdirs":1,"change_attr":3},"rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":2,"rsnaps":0,"rctime":"2024-09-16T01:36:22.220837+0000"},"accounted_rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":2,"rsnaps":0,"rctime":"2024-09-16T01:36:22.220837+0000"},"version":9,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[{"key":"security.selinux","val":"unconfined_u:object_r:unlabeled_t:s0\u0000"}],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"request":0,"lock":0,"dirfrag":1,"caps":0,"dirtyparent":0,"dirty":1,"authpin":0},"nref":2,"versionlock":{"gather_set":[],"state":"lock","type":"iversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"authlock":{},"linklock":{},"dirfragtreelock":{},"filelock":{},"xattrlock":{},"snaplock":{},"nestlock":{"gather_set":[],"state":"lock","type":"inest","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"flocklock":{},"policylock":{},"quiescelock":{"gather_set":[],"state":"lock","type":"iquiesce","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"],"client_caps":[],"loner":-1,"want_loner":-1,"mds_caps_wanted":[],"dirfrags":[{"path":"/a","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"3","committed_version":"3","is_rep":false,"dir_auth":"","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"dirty":1,"authpin":0},"nref":2,"dentries":[{"path":"a/c","alternate_name":"","path_ino":1,"snap_first":18446744073709551606,"snap_last":18446744073709551614,"is_primary":true,"is_remote":false,"is_null":false,"is_new":false,"inode":1099511627777,"remote_type":"","version":4,"projected_version":4,"auth_pins":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"request":0,"lock":0,"inodepin":1,"dirty":1,"authpin":0},"nref":2,"lock":{},"versionlock":{"gather_set":[],"state":"lock","type":"dversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"]},{"path":"a/b","alternate_name":"","path_ino":1,"snap_first":2,"snap_last":18446744073709551614,"is_primary":false,"is_remote":false,"is_null":true,"is_new":false,"inode":0,"remote_type":"","version":5,"projected_version":5,"auth_pins":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"request":0,"lock":0,"inodepin":0,"dirty":1,"authpin":0},"nref":1,"lock":{},"versionlock":{"gather_set":[],"state":"lock","type":"dversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"]}]}],"is_subvolume":false}] 2024-09-16T01:36:23.482 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "accounted_rstat": { "rbytes": 0, "rctime": "2024-09-16T01:36:22.220837+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 2, "version": 0 }, "atime": "2024-09-16T01:36:16.352953+0000", "auth_pins": 0, "auth_state": { "replicas": {} }, "authlock": {}, "backtrace_version": 2, "btime": "2024-09-16T01:36:16.352953+0000", "change_attr": 3, "client_caps": [], "client_ranges": [], "ctime": "2024-09-16T01:36:22.220837+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfrags": [ { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "3", "committing_version": "3", "dentries": [ { "alternate_name": "", "auth_pins": 0, "auth_state": { "replicas": {} }, "inode": 1099511627777, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_new": false, "is_null": false, "is_primary": true, "is_remote": false, "lock": {}, "nref": 2, "path": "a/c", "path_ino": 1, "pins": { "authpin": 0, "dirty": 1, "inodepin": 1, "lock": 0, "request": 0 }, "projected_version": 4, "remote_type": "", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snap_first": 18446744073709551606, "snap_last": 18446744073709551614, "states": [ "auth", "dirty" ], "version": 4, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "dversion", "xlock_by": {} } }, { "alternate_name": "", "auth_pins": 0, "auth_state": { "replicas": {} }, "inode": 0, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_new": false, "is_null": true, "is_primary": false, "is_remote": false, "lock": {}, "nref": 1, "path": "a/b", "path_ino": 1, "pins": { "authpin": 0, "dirty": 1, "inodepin": 0, "lock": 0, "request": 0 }, "projected_version": 5, "remote_type": "", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snap_first": 2, "snap_last": 18446744073709551614, "states": [ "auth", "dirty" ], "version": 5, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "dversion", "xlock_by": {} } } ], "dir_auth": "", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/a", "pins": { "authpin": 0, "child": 1, "dirty": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" } ], "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 3, "mtime": "2024-09-16T01:36:22.220837+0000", "num_files": 0, "num_subdirs": 1, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": {}, "flocklock": {}, "gid": 1333, "ino": 1099511627776, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_subvolume": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 0, "pool_id": -1, "pool_ns": "", "stripe_count": 0, "stripe_unit": 0 }, "linklock": {}, "loner": -1, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16877, "mtime": "2024-09-16T01:36:22.220837+0000", "nestlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "inest", "xlock_by": {} }, "nlink": 1, "nref": 2, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "pins": { "authpin": 0, "caps": 0, "dirfrag": 1, "dirty": 1, "dirtyparent": 0, "lock": 0, "request": 0 }, "policylock": {}, "quiesce_block": false, "quiescelock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "iquiesce", "xlock_by": {} }, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 0, "rctime": "2024-09-16T01:36:22.220837+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 2, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth", "dirty" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 1000, "version": 9, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "iversion", "xlock_by": {} }, "want_loner": -1, "xattr_version": 1, "xattrlock": {}, "xattrs": [ { "key": "security.selinux", "val": "unconfined_u:object_r:unlabeled_t:s0\u0000" } ] } ] 2024-09-16T01:36:23.482 DEBUG:tasks.cephfs.test_damage:[{'ino': 1099511627776, 'rdev': 0, 'ctime': '2024-09-16T01:36:22.220837+0000', 'btime': '2024-09-16T01:36:16.352953+0000', 'mode': 16877, 'uid': 1000, 'gid': 1333, 'nlink': 1, 'dir_layout': {'dir_hash': 2, 'unused1': 0, 'unused2': 0, 'unused3': 0}, 'layout': {'stripe_unit': 0, 'stripe_count': 0, 'object_size': 0, 'pool_id': -1, 'pool_ns': ''}, 'old_pools': [], 'size': 0, 'truncate_seq': 1, 'truncate_size': 18446744073709551615, 'truncate_from': 0, 'truncate_pending': 0, 'mtime': '2024-09-16T01:36:22.220837+0000', 'atime': '2024-09-16T01:36:16.352953+0000', 'time_warp_seq': 0, 'change_attr': 3, 'export_pin': -1, 'export_ephemeral_random_pin': 0, 'export_ephemeral_distributed_pin': False, 'quiesce_block': False, 'client_ranges': [], 'dirstat': {'version': 0, 'mtime': '2024-09-16T01:36:22.220837+0000', 'num_files': 0, 'num_subdirs': 1, 'change_attr': 3}, 'rstat': {'version': 0, 'rbytes': 0, 'rfiles': 0, 'rsubdirs': 2, 'rsnaps': 0, 'rctime': '2024-09-16T01:36:22.220837+0000'}, 'accounted_rstat': {'version': 0, 'rbytes': 0, 'rfiles': 0, 'rsubdirs': 2, 'rsnaps': 0, 'rctime': '2024-09-16T01:36:22.220837+0000'}, 'version': 9, 'file_data_version': 0, 'xattr_version': 1, 'backtrace_version': 2, 'stray_prior_path': '', 'max_size_ever': 0, 'quota': {'max_bytes': 0, 'max_files': 0}, 'last_scrub_stamp': '0.000000', 'last_scrub_version': 0, 'symlink': '', 'xattrs': [{'key': 'security.selinux', 'val': 'unconfined_u:object_r:unlabeled_t:s0\x00'}], 'dirfragtree': {'splits': []}, 'old_inodes': [], 'oldest_snap': 18446744073709551614, 'damage_flags': 0, 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'request': 0, 'lock': 0, 'dirfrag': 1, 'caps': 0, 'dirtyparent': 0, 'dirty': 1, 'authpin': 0}, 'nref': 2, 'versionlock': {'gather_set': [], 'state': 'lock', 'type': 'iversion', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'authlock': {}, 'linklock': {}, 'dirfragtreelock': {}, 'filelock': {}, 'xattrlock': {}, 'snaplock': {}, 'nestlock': {'gather_set': [], 'state': 'lock', 'type': 'inest', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'flocklock': {}, 'policylock': {}, 'quiescelock': {'gather_set': [], 'state': 'lock', 'type': 'iquiesce', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'states': ['auth', 'dirty'], 'client_caps': [], 'loner': -1, 'want_loner': -1, 'mds_caps_wanted': [], 'dirfrags': [{'path': '/a', 'dirfrag': '0x10000000000', 'snapid_first': 2, 'projected_version': '7', 'version': '7', 'committing_version': '3', 'committed_version': '3', 'is_rep': False, 'dir_auth': '', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1, 'dirty': 1, 'authpin': 0}, 'nref': 2, 'dentries': [{'path': 'a/c', 'alternate_name': '', 'path_ino': 1, 'snap_first': 18446744073709551606, 'snap_last': 18446744073709551614, 'is_primary': True, 'is_remote': False, 'is_null': False, 'is_new': False, 'inode': 1099511627777, 'remote_type': '', 'version': 4, 'projected_version': 4, 'auth_pins': 0, 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'is_frozen': False, 'is_freezing': False, 'pins': {'request': 0, 'lock': 0, 'inodepin': 1, 'dirty': 1, 'authpin': 0}, 'nref': 2, 'lock': {}, 'versionlock': {'gather_set': [], 'state': 'lock', 'type': 'dversion', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'states': ['auth', 'dirty']}, {'path': 'a/b', 'alternate_name': '', 'path_ino': 1, 'snap_first': 2, 'snap_last': 18446744073709551614, 'is_primary': False, 'is_remote': False, 'is_null': True, 'is_new': False, 'inode': 0, 'remote_type': '', 'version': 5, 'projected_version': 5, 'auth_pins': 0, 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'is_frozen': False, 'is_freezing': False, 'pins': {'request': 0, 'lock': 0, 'inodepin': 0, 'dirty': 1, 'authpin': 0}, 'nref': 1, 'lock': {}, 'versionlock': {'gather_set': [], 'state': 'lock', 'type': 'dversion', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'states': ['auth', 'dirty']}]}], 'is_subvolume': False}] 2024-09-16T01:36:23.483 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:36:23.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.581+0000 7fba5fa77640 1 Processor -- start 2024-09-16T01:36:23.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.581+0000 7fba5fa77640 1 -- start start 2024-09-16T01:36:23.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.582+0000 7fba5fa77640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fba580b4e60 0x7fba580d0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:23.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.582+0000 7fba5fa77640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba580b5c00 0x7fba580b5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:23.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.582+0000 7fba5fa77640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba580cd9b0 con 0x7fba5815ffc0 2024-09-16T01:36:23.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.582+0000 7fba5fa77640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fba580ce5c0 con 0x7fba580b4e60 2024-09-16T01:36:23.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.582+0000 7fba5fa77640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fba580cdc90 con 0x7fba580b5c00 2024-09-16T01:36:23.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.582+0000 7fba5d813640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fba580b4e60 0x7fba580d0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:23.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.582+0000 7fba5d813640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fba580b4e60 0x7fba580d0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:32786/0 (socket says 172.21.15.88:32786) 2024-09-16T01:36:23.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.582+0000 7fba5d813640 1 -- 172.21.15.88:0/3239856916 learned_addr learned my addr 172.21.15.88:0/3239856916 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:23.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.582+0000 7fba5d012640 1 --2- 172.21.15.88:0/3239856916 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba580b5c00 0x7fba580b5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:23.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.582+0000 7fba5d813640 1 -- 172.21.15.88:0/3239856916 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba580b5c00 msgr2=0x7fba580b5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:23.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.582+0000 7fba5d813640 1 --2- 172.21.15.88:0/3239856916 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba580b5c00 0x7fba580b5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.582+0000 7fba5d813640 1 -- 172.21.15.88:0/3239856916 >> v1:172.21.15.154:6789/0 conn(0x7fba5815ffc0 legacy=0x7fba58160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:23.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.582+0000 7fba5d813640 1 -- 172.21.15.88:0/3239856916 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba580cdf70 con 0x7fba580b4e60 2024-09-16T01:36:23.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.583+0000 7fba5d813640 1 --2- 172.21.15.88:0/3239856916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba580b4e60 0x7fba580d0760 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fba54009e50 tx=0x7fba5402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82c286f2574b299f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:23.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.583+0000 7fba5c811640 1 -- 172.21.15.88:0/3239856916 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba54036070 con 0x7fba580b4e60 2024-09-16T01:36:23.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.583+0000 7fba5c811640 1 -- 172.21.15.88:0/3239856916 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba54031600 con 0x7fba580b4e60 2024-09-16T01:36:23.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.583+0000 7fba5c811640 1 -- 172.21.15.88:0/3239856916 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba540319d0 con 0x7fba580b4e60 2024-09-16T01:36:23.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.583+0000 7fba5fa77640 1 -- 172.21.15.88:0/3239856916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba580b4e60 msgr2=0x7fba580d0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:23.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.584+0000 7fba5fa77640 1 --2- 172.21.15.88:0/3239856916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba580b4e60 0x7fba580d0760 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fba54009e50 tx=0x7fba5402a710 comp rx=0 tx=0).stop 2024-09-16T01:36:23.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.584+0000 7fba5fa77640 1 -- 172.21.15.88:0/3239856916 shutdown_connections 2024-09-16T01:36:23.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.584+0000 7fba5fa77640 1 --2- 172.21.15.88:0/3239856916 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba580b5c00 0x7fba580b5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.584+0000 7fba5fa77640 1 --2- 172.21.15.88:0/3239856916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba580b4e60 0x7fba580d0760 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.584+0000 7fba5fa77640 1 -- 172.21.15.88:0/3239856916 >> 172.21.15.88:0/3239856916 conn(0x7fba5806cb50 msgr2=0x7fba5806cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:23.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.584+0000 7fba5fa77640 1 -- 172.21.15.88:0/3239856916 shutdown_connections 2024-09-16T01:36:23.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.584+0000 7fba5fa77640 1 -- 172.21.15.88:0/3239856916 wait complete. 2024-09-16T01:36:23.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.585+0000 7fba5fa77640 1 Processor -- start 2024-09-16T01:36:23.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.585+0000 7fba5fa77640 1 -- start start 2024-09-16T01:36:23.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.586+0000 7fba5fa77640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba580b4e60 0x7fba581ef710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:23.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.586+0000 7fba5fa77640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba580b5c00 0x7fba58151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:23.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.586+0000 7fba5fa77640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba5815ffc0 0x7fba58159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:23.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.586+0000 7fba5fa77640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fba5815fc00 con 0x7fba580b4e60 2024-09-16T01:36:23.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.586+0000 7fba5fa77640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fba580b5780 con 0x7fba5815ffc0 2024-09-16T01:36:23.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.586+0000 7fba5fa77640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fba5815fd80 con 0x7fba580b5c00 2024-09-16T01:36:23.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.586+0000 7fba5d813640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba580b4e60 0x7fba581ef710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:23.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.586+0000 7fba5d813640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba580b4e60 0x7fba581ef710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:32798/0 (socket says 172.21.15.88:32798) 2024-09-16T01:36:23.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.586+0000 7fba5d813640 1 -- 172.21.15.88:0/1841607703 learned_addr learned my addr 172.21.15.88:0/1841607703 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:23.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.586+0000 7fba5d012640 1 --2- 172.21.15.88:0/1841607703 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba580b5c00 0x7fba58151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:23.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.586+0000 7fba5e014640 1 --2- 172.21.15.88:0/1841607703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba5815ffc0 0x7fba58159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:23.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.586+0000 7fba5d813640 1 -- 172.21.15.88:0/1841607703 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba580b5c00 msgr2=0x7fba58151fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:23.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.586+0000 7fba5d813640 1 --2- 172.21.15.88:0/1841607703 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba580b5c00 0x7fba58151fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.586+0000 7fba5d813640 1 -- 172.21.15.88:0/1841607703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba5815ffc0 msgr2=0x7fba58159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:23.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.586+0000 7fba5d813640 1 --2- 172.21.15.88:0/1841607703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba5815ffc0 0x7fba58159930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.586+0000 7fba5d813640 1 -- 172.21.15.88:0/1841607703 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba58152980 con 0x7fba580b4e60 2024-09-16T01:36:23.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.586+0000 7fba5d012640 1 --2- 172.21.15.88:0/1841607703 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba580b5c00 0x7fba58151fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:23.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.586+0000 7fba5e014640 1 --2- 172.21.15.88:0/1841607703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba5815ffc0 0x7fba58159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:23.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.586+0000 7fba5d813640 1 --2- 172.21.15.88:0/1841607703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba580b4e60 0x7fba581ef710 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fba54034040 tx=0x7fba54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:23.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.586+0000 7fba4e7fc640 1 -- 172.21.15.88:0/1841607703 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba54036050 con 0x7fba580b4e60 2024-09-16T01:36:23.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.586+0000 7fba4e7fc640 1 -- 172.21.15.88:0/1841607703 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba54042070 con 0x7fba580b4e60 2024-09-16T01:36:23.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.586+0000 7fba5fa77640 1 -- 172.21.15.88:0/1841607703 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba58152c60 con 0x7fba580b4e60 2024-09-16T01:36:23.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.586+0000 7fba4e7fc640 1 -- 172.21.15.88:0/1841607703 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba54009510 con 0x7fba580b4e60 2024-09-16T01:36:23.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.586+0000 7fba5fa77640 1 -- 172.21.15.88:0/1841607703 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba58153500 con 0x7fba580b4e60 2024-09-16T01:36:23.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.587+0000 7fba5fa77640 1 -- 172.21.15.88:0/1841607703 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba58152e30 con 0x7fba580b4e60 2024-09-16T01:36:23.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.588+0000 7fba4e7fc640 1 -- 172.21.15.88:0/1841607703 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fba54007c50 con 0x7fba580b4e60 2024-09-16T01:36:23.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.588+0000 7fba4e7fc640 1 --2- 172.21.15.88:0/1841607703 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba3c06c2b0 0x7fba3c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:23.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.589+0000 7fba5d012640 1 --2- 172.21.15.88:0/1841607703 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba3c06c2b0 0x7fba3c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:23.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.589+0000 7fba4e7fc640 1 -- 172.21.15.88:0/1841607703 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fba540ac300 con 0x7fba580b4e60 2024-09-16T01:36:23.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.589+0000 7fba5d012640 1 --2- 172.21.15.88:0/1841607703 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba3c06c2b0 0x7fba3c06e770 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fba58070220 tx=0x7fba48008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:23.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.593+0000 7fba4e7fc640 1 -- 172.21.15.88:0/1841607703 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fba5407aa70 con 0x7fba580b4e60 2024-09-16T01:36:23.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.794+0000 7fba5fa77640 1 -- 172.21.15.88:0/1841607703 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fba5815c3a0 con 0x7fba580b4e60 2024-09-16T01:36:23.796 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:36:23.795+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:36:23.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.796+0000 7fba4e7fc640 1 -- 172.21.15.88:0/1841607703 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) ==== 161+0+0 (secure 0 0 0) 0x7fba5815c3a0 con 0x7fba580b4e60 2024-09-16T01:36:23.797 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:36:23.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.799+0000 7fba5fa77640 1 -- 172.21.15.88:0/1841607703 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba3c06c2b0 msgr2=0x7fba3c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:23.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.799+0000 7fba5fa77640 1 --2- 172.21.15.88:0/1841607703 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba3c06c2b0 0x7fba3c06e770 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fba58070220 tx=0x7fba48008040 comp rx=0 tx=0).stop 2024-09-16T01:36:23.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.799+0000 7fba5fa77640 1 -- 172.21.15.88:0/1841607703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba580b4e60 msgr2=0x7fba581ef710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:23.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.799+0000 7fba5fa77640 1 --2- 172.21.15.88:0/1841607703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba580b4e60 0x7fba581ef710 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fba54034040 tx=0x7fba54004a40 comp rx=0 tx=0).stop 2024-09-16T01:36:23.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.799+0000 7fba5fa77640 1 -- 172.21.15.88:0/1841607703 shutdown_connections 2024-09-16T01:36:23.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.799+0000 7fba5fa77640 1 --2- 172.21.15.88:0/1841607703 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba3c06c2b0 0x7fba3c06e770 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.799+0000 7fba5fa77640 1 --2- 172.21.15.88:0/1841607703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba5815ffc0 0x7fba58159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.799+0000 7fba5fa77640 1 --2- 172.21.15.88:0/1841607703 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba580b5c00 0x7fba58151fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.799+0000 7fba5fa77640 1 --2- 172.21.15.88:0/1841607703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba580b4e60 0x7fba581ef710 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.799+0000 7fba5fa77640 1 -- 172.21.15.88:0/1841607703 >> 172.21.15.88:0/1841607703 conn(0x7fba5806cb50 msgr2=0x7fba5815ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:23.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.799+0000 7fba5fa77640 1 -- 172.21.15.88:0/1841607703 shutdown_connections 2024-09-16T01:36:23.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.799+0000 7fba5fa77640 1 -- 172.21.15.88:0/1841607703 wait complete. 2024-09-16T01:36:23.803 INFO:tasks.ceph.mds.c.smithi088.stderr: -4108> 2024-09-16T01:36:12.345+0000 7efdf135a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:36:23.810 INFO:tasks.ceph.mds.c.smithi088.stderr: -347> 2024-09-16T01:36:22.221+0000 7efdf3b5f640 -1 mds.0.cache.den(0x10000000000 c) newly corrupt dentry to be committed: [dentry #0x1/a/c [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.5143:44 nref=7 cr=0x55ba84962e00)) (dversion lock w=1 last_client=5143) pv=4 v=3 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x55ba8484b400] 2024-09-16T01:36:23.815 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_inject_rename_corrupt_dentry_first 2024-09-16T01:36:23.837 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:36:23.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.916+0000 7f17007f6640 1 Processor -- start 2024-09-16T01:36:23.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.916+0000 7f17007f6640 1 -- start start 2024-09-16T01:36:23.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.917+0000 7f17007f6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f16f815a060 0x7f16f806f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:23.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.917+0000 7f17007f6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16f8071770 con 0x7f16f815d3d0 2024-09-16T01:36:23.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.917+0000 7f17007f6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16f8160d60 con 0x7f16f80b1990 2024-09-16T01:36:23.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.917+0000 7f17007f6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f16f80712e0 con 0x7f16f815a060 2024-09-16T01:36:23.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.917+0000 7f16fe592640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f16f815a060 0x7f16f806f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:23.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.917+0000 7f16fe592640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f16f815a060 0x7f16f806f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:32820/0 (socket says 172.21.15.88:32820) 2024-09-16T01:36:23.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.917+0000 7f16fdd91640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f16f815d3d0 0x7f16f815ff40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42736/0 (socket says 172.21.15.88:42736) 2024-09-16T01:36:23.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.917+0000 7f16fe592640 1 -- 172.21.15.88:0/1389105272 learned_addr learned my addr 172.21.15.88:0/1389105272 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:23.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.917+0000 7f16fe592640 1 -- 172.21.15.88:0/1389105272 >> v1:172.21.15.154:6789/0 conn(0x7f16f80b1990 legacy=0x7f16f8160650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:23.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.917+0000 7f16fe592640 1 -- 172.21.15.88:0/1389105272 >> v1:172.21.15.88:6789/0 conn(0x7f16f815d3d0 legacy=0x7f16f815ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:23.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.917+0000 7f16fe592640 1 -- 172.21.15.88:0/1389105272 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f80709b0 con 0x7f16f815a060 2024-09-16T01:36:23.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.918+0000 7f16fe592640 1 --2- 172.21.15.88:0/1389105272 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16f815a060 0x7f16f806f2f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f16f4009870 tx=0x7f16f402a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad106dc0eeec3b80 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:23.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.918+0000 7f16fd590640 1 -- 172.21.15.88:0/1389105272 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f4037070 con 0x7f16f815a060 2024-09-16T01:36:23.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.918+0000 7f16fd590640 1 -- 172.21.15.88:0/1389105272 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16f4032440 con 0x7f16f815a060 2024-09-16T01:36:23.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.918+0000 7f16fd590640 1 -- 172.21.15.88:0/1389105272 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f4032810 con 0x7f16f815a060 2024-09-16T01:36:23.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.919+0000 7f17007f6640 1 -- 172.21.15.88:0/1389105272 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16f815a060 msgr2=0x7f16f806f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:23.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.919+0000 7f17007f6640 1 --2- 172.21.15.88:0/1389105272 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16f815a060 0x7f16f806f2f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f16f4009870 tx=0x7f16f402a2a0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.919+0000 7f17007f6640 1 -- 172.21.15.88:0/1389105272 shutdown_connections 2024-09-16T01:36:23.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.919+0000 7f17007f6640 1 --2- 172.21.15.88:0/1389105272 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16f815a060 0x7f16f806f2f0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.919+0000 7f17007f6640 1 -- 172.21.15.88:0/1389105272 >> 172.21.15.88:0/1389105272 conn(0x7f16f80d2430 msgr2=0x7f16f80d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:23.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.919+0000 7f17007f6640 1 -- 172.21.15.88:0/1389105272 shutdown_connections 2024-09-16T01:36:23.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.919+0000 7f17007f6640 1 -- 172.21.15.88:0/1389105272 wait complete. 2024-09-16T01:36:23.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.920+0000 7f17007f6640 1 Processor -- start 2024-09-16T01:36:23.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.920+0000 7f17007f6640 1 -- start start 2024-09-16T01:36:23.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.921+0000 7f17007f6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16f80b1990 0x7f16f81f8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:23.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.921+0000 7f17007f6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16f815a060 0x7f16f815ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:23.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.921+0000 7f17007f6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16f815d3d0 0x7f16f80cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:23.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.921+0000 7f17007f6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f16f80d12e0 con 0x7f16f815d3d0 2024-09-16T01:36:23.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.921+0000 7f17007f6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f16f8070f70 con 0x7f16f815a060 2024-09-16T01:36:23.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.921+0000 7f17007f6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f16f80d1460 con 0x7f16f80b1990 2024-09-16T01:36:23.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.921+0000 7f16fed93640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16f815d3d0 0x7f16f80cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:23.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.921+0000 7f16fed93640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16f815d3d0 0x7f16f80cb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:32826/0 (socket says 172.21.15.88:32826) 2024-09-16T01:36:23.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.921+0000 7f16fed93640 1 -- 172.21.15.88:0/3644473281 learned_addr learned my addr 172.21.15.88:0/3644473281 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:23.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.921+0000 7f16fdd91640 1 --2- 172.21.15.88:0/3644473281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16f815a060 0x7f16f815ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:23.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.921+0000 7f16fe592640 1 --2- 172.21.15.88:0/3644473281 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16f80b1990 0x7f16f81f8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:23.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.921+0000 7f16fed93640 1 -- 172.21.15.88:0/3644473281 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16f80b1990 msgr2=0x7f16f81f8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:23.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.921+0000 7f16fed93640 1 --2- 172.21.15.88:0/3644473281 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16f80b1990 0x7f16f81f8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.921+0000 7f16fed93640 1 -- 172.21.15.88:0/3644473281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16f815a060 msgr2=0x7f16f815ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:23.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.921+0000 7f16fed93640 1 --2- 172.21.15.88:0/3644473281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16f815a060 0x7f16f815ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:23.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.921+0000 7f16fed93640 1 -- 172.21.15.88:0/3644473281 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f81f8cc0 con 0x7f16f815d3d0 2024-09-16T01:36:23.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.921+0000 7f16fed93640 1 --2- 172.21.15.88:0/3644473281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16f815d3d0 0x7f16f80cb8c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f16f000c780 tx=0x7f16f000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:23.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.921+0000 7f16eeffd640 1 -- 172.21.15.88:0/3644473281 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f0004b60 con 0x7f16f815d3d0 2024-09-16T01:36:23.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.921+0000 7f16eeffd640 1 -- 172.21.15.88:0/3644473281 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16f0004d00 con 0x7f16f815d3d0 2024-09-16T01:36:23.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.921+0000 7f16eeffd640 1 -- 172.21.15.88:0/3644473281 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f000aa90 con 0x7f16f815d3d0 2024-09-16T01:36:23.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.921+0000 7f17007f6640 1 -- 172.21.15.88:0/3644473281 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16f80cdf90 con 0x7f16f815d3d0 2024-09-16T01:36:23.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.921+0000 7f17007f6640 1 -- 172.21.15.88:0/3644473281 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16f80cc400 con 0x7f16f815d3d0 2024-09-16T01:36:23.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.922+0000 7f17007f6640 1 -- 172.21.15.88:0/3644473281 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16f806c6c0 con 0x7f16f815d3d0 2024-09-16T01:36:23.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.923+0000 7f16eeffd640 1 -- 172.21.15.88:0/3644473281 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f16f000ac30 con 0x7f16f815d3d0 2024-09-16T01:36:23.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.923+0000 7f16eeffd640 1 --2- 172.21.15.88:0/3644473281 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f16dc06c2b0 0x7f16dc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:23.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.923+0000 7f16eeffd640 1 -- 172.21.15.88:0/3644473281 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f16f008b5c0 con 0x7f16f815d3d0 2024-09-16T01:36:23.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.923+0000 7f16fe592640 1 --2- 172.21.15.88:0/3644473281 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f16dc06c2b0 0x7f16dc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:23.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.924+0000 7f16fe592640 1 --2- 172.21.15.88:0/3644473281 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f16dc06c2b0 0x7f16dc06e770 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f16f4034000 tx=0x7f16f402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:23.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:23.929+0000 7f16eeffd640 1 -- 172.21.15.88:0/3644473281 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f16f0020070 con 0x7f16f815d3d0 2024-09-16T01:36:24.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.094+0000 7f17007f6640 1 -- 172.21.15.88:0/3644473281 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"} v 0) -- 0x7f16f806feb0 con 0x7f16f815d3d0 2024-09-16T01:36:24.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.097+0000 7f16eeffd640 1 -- 172.21.15.88:0/3644473281 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"}]=0 v15) ==== 121+0+0 (secure 0 0 0) 0x7f16f81f8cc0 con 0x7f16f815d3d0 2024-09-16T01:36:24.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.099+0000 7f17007f6640 1 -- 172.21.15.88:0/3644473281 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f16dc06c2b0 msgr2=0x7f16dc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:24.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.099+0000 7f17007f6640 1 --2- 172.21.15.88:0/3644473281 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f16dc06c2b0 0x7f16dc06e770 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f16f4034000 tx=0x7f16f402e000 comp rx=0 tx=0).stop 2024-09-16T01:36:24.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.099+0000 7f17007f6640 1 -- 172.21.15.88:0/3644473281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16f815d3d0 msgr2=0x7f16f80cb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:24.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.099+0000 7f17007f6640 1 --2- 172.21.15.88:0/3644473281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16f815d3d0 0x7f16f80cb8c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f16f000c780 tx=0x7f16f000cc40 comp rx=0 tx=0).stop 2024-09-16T01:36:24.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.099+0000 7f17007f6640 1 -- 172.21.15.88:0/3644473281 shutdown_connections 2024-09-16T01:36:24.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.099+0000 7f17007f6640 1 --2- 172.21.15.88:0/3644473281 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f16dc06c2b0 0x7f16dc06e770 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.099+0000 7f17007f6640 1 --2- 172.21.15.88:0/3644473281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16f815d3d0 0x7f16f80cb8c0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.099+0000 7f17007f6640 1 --2- 172.21.15.88:0/3644473281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16f815a060 0x7f16f815ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.099+0000 7f17007f6640 1 --2- 172.21.15.88:0/3644473281 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16f80b1990 0x7f16f81f8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.100+0000 7f17007f6640 1 -- 172.21.15.88:0/3644473281 >> 172.21.15.88:0/3644473281 conn(0x7f16f80d2430 msgr2=0x7f16f80b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:24.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.100+0000 7f17007f6640 1 -- 172.21.15.88:0/3644473281 shutdown_connections 2024-09-16T01:36:24.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.100+0000 7f17007f6640 1 -- 172.21.15.88:0/3644473281 wait complete. 2024-09-16T01:36:24.112 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_abort_on_newly_corrupt_dentry False 2024-09-16T01:36:24.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.253+0000 7f18ad647640 1 Processor -- start 2024-09-16T01:36:24.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.253+0000 7f18ad647640 1 -- start start 2024-09-16T01:36:24.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.254+0000 7f18ad647640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f18a80b54d0 0x7f18a806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:24.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.254+0000 7f18ad647640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18a806e990 con 0x7f18a80b7450 2024-09-16T01:36:24.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.254+0000 7f18ad647640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18a806ec70 con 0x7f18a815e6f0 2024-09-16T01:36:24.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.254+0000 7f18ad647640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f18a815ec70 con 0x7f18a80b54d0 2024-09-16T01:36:24.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.254+0000 7f18a67fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f18a80b54d0 0x7f18a806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:24.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.254+0000 7f18a77fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f18a80b7450 0x7f18a8070830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42744/0 (socket says 172.21.15.88:42744) 2024-09-16T01:36:24.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.254+0000 7f18a67fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f18a80b54d0 0x7f18a806fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:32842/0 (socket says 172.21.15.88:32842) 2024-09-16T01:36:24.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.254+0000 7f18a77fe640 1 -- 172.21.15.88:0/2008032616 learned_addr learned my addr 172.21.15.88:0/2008032616 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:24.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.254+0000 7f18a67fc640 1 -- 172.21.15.88:0/2008032616 >> v1:172.21.15.154:6789/0 conn(0x7f18a815e6f0 legacy=0x7f18a80b6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:24.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.254+0000 7f18a67fc640 1 -- 172.21.15.88:0/2008032616 >> v1:172.21.15.88:6789/0 conn(0x7f18a80b7450 legacy=0x7f18a8070830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:24.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.254+0000 7f18a67fc640 1 -- 172.21.15.88:0/2008032616 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a815ef50 con 0x7f18a80b54d0 2024-09-16T01:36:24.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.255+0000 7f18a67fc640 1 --2- 172.21.15.88:0/2008032616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18a80b54d0 0x7f18a806fd00 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f189c0097a0 tx=0x7f189c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92354fda0b480488 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:24.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.255+0000 7f18a5ffb640 1 -- 172.21.15.88:0/2008032616 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189c037070 con 0x7f18a80b54d0 2024-09-16T01:36:24.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.255+0000 7f18a5ffb640 1 -- 172.21.15.88:0/2008032616 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f189c032440 con 0x7f18a80b54d0 2024-09-16T01:36:24.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.255+0000 7f18a5ffb640 1 -- 172.21.15.88:0/2008032616 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189c032810 con 0x7f18a80b54d0 2024-09-16T01:36:24.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.256+0000 7f18ad647640 1 -- 172.21.15.88:0/2008032616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18a80b54d0 msgr2=0x7f18a806fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:24.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.256+0000 7f18ad647640 1 --2- 172.21.15.88:0/2008032616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18a80b54d0 0x7f18a806fd00 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f189c0097a0 tx=0x7f189c02a1e0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.256+0000 7f18ad647640 1 -- 172.21.15.88:0/2008032616 shutdown_connections 2024-09-16T01:36:24.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.256+0000 7f18ad647640 1 --2- 172.21.15.88:0/2008032616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18a80b54d0 0x7f18a806fd00 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.256+0000 7f18ad647640 1 -- 172.21.15.88:0/2008032616 >> 172.21.15.88:0/2008032616 conn(0x7f18a80d2430 msgr2=0x7f18a80d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:24.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.256+0000 7f18ad647640 1 -- 172.21.15.88:0/2008032616 shutdown_connections 2024-09-16T01:36:24.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.256+0000 7f18ad647640 1 -- 172.21.15.88:0/2008032616 wait complete. 2024-09-16T01:36:24.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.257+0000 7f18ad647640 1 Processor -- start 2024-09-16T01:36:24.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.257+0000 7f18ad647640 1 -- start start 2024-09-16T01:36:24.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.258+0000 7f18ad647640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18a80b54d0 0x7f18a81f4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:24.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.258+0000 7f18ad647640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18a80b7450 0x7f18a815d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:24.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.258+0000 7f18ad647640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18a815e6f0 0x7f18a80b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:24.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.258+0000 7f18ad647640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f18a806e730 con 0x7f18a815e6f0 2024-09-16T01:36:24.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.258+0000 7f18ad647640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f18a806e5b0 con 0x7f18a80b54d0 2024-09-16T01:36:24.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.258+0000 7f18ad647640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f18a815f750 con 0x7f18a80b7450 2024-09-16T01:36:24.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.258+0000 7f18a77fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18a815e6f0 0x7f18a80b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:24.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.258+0000 7f18a77fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18a815e6f0 0x7f18a80b1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:32852/0 (socket says 172.21.15.88:32852) 2024-09-16T01:36:24.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.258+0000 7f18a77fe640 1 -- 172.21.15.88:0/2651631660 learned_addr learned my addr 172.21.15.88:0/2651631660 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:24.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.258+0000 7f18a77fe640 1 -- 172.21.15.88:0/2651631660 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18a80b7450 msgr2=0x7f18a815d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:24.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.258+0000 7f18a77fe640 1 --2- 172.21.15.88:0/2651631660 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18a80b7450 0x7f18a815d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.258+0000 7f18a77fe640 1 -- 172.21.15.88:0/2651631660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18a80b54d0 msgr2=0x7f18a81f4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:24.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.258+0000 7f18a77fe640 1 --2- 172.21.15.88:0/2651631660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18a80b54d0 0x7f18a81f4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.258+0000 7f18a77fe640 1 -- 172.21.15.88:0/2651631660 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a81f5250 con 0x7f18a815e6f0 2024-09-16T01:36:24.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.258+0000 7f18a77fe640 1 --2- 172.21.15.88:0/2651631660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18a815e6f0 0x7f18a80b1c20 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f18a000cbc0 tx=0x7f18a000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:24.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.259+0000 7f188f7fe640 1 -- 172.21.15.88:0/2651631660 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a0017070 con 0x7f18a815e6f0 2024-09-16T01:36:24.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.259+0000 7f188f7fe640 1 -- 172.21.15.88:0/2651631660 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18a000ad40 con 0x7f18a815e6f0 2024-09-16T01:36:24.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.259+0000 7f188f7fe640 1 -- 172.21.15.88:0/2651631660 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a0012600 con 0x7f18a815e6f0 2024-09-16T01:36:24.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.259+0000 7f18ad647640 1 -- 172.21.15.88:0/2651631660 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18a81f5530 con 0x7f18a815e6f0 2024-09-16T01:36:24.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.259+0000 7f18ad647640 1 -- 172.21.15.88:0/2651631660 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18a81f8300 con 0x7f18a815e6f0 2024-09-16T01:36:24.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.260+0000 7f18ad647640 1 -- 172.21.15.88:0/2651631660 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18a806c6c0 con 0x7f18a815e6f0 2024-09-16T01:36:24.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.260+0000 7f188f7fe640 1 -- 172.21.15.88:0/2651631660 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f18a001a070 con 0x7f18a815e6f0 2024-09-16T01:36:24.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.261+0000 7f188f7fe640 1 --2- 172.21.15.88:0/2651631660 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f189406c2e0 0x7f189406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:24.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.261+0000 7f188f7fe640 1 -- 172.21.15.88:0/2651631660 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f18a008b100 con 0x7f18a815e6f0 2024-09-16T01:36:24.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.261+0000 7f18a6ffd640 1 --2- 172.21.15.88:0/2651631660 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f189406c2e0 0x7f189406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:24.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.261+0000 7f18a6ffd640 1 --2- 172.21.15.88:0/2651631660 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f189406c2e0 0x7f189406e7a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1898006fd0 tx=0x7f1898006bc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:24.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.265+0000 7f188f7fe640 1 -- 172.21.15.88:0/2651631660 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f18a00591c0 con 0x7f18a815e6f0 2024-09-16T01:36:24.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.433+0000 7f18ad647640 1 -- 172.21.15.88:0/2651631660 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command([{prefix=config set, name=mds_abort_on_newly_corrupt_dentry}] v 0) -- 0x7f18a8068e10 con 0x7f18a815e6f0 2024-09-16T01:36:24.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.433+0000 7f188f7fe640 1 -- 172.21.15.88:0/2651631660 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_abort_on_newly_corrupt_dentry}]=0 v15)=0 v15) ==== 135+0+0 (secure 0 0 0) 0x7f18a005d4c0 con 0x7f18a815e6f0 2024-09-16T01:36:24.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.435+0000 7f18ad647640 1 -- 172.21.15.88:0/2651631660 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f189406c2e0 msgr2=0x7f189406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:24.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.435+0000 7f18ad647640 1 --2- 172.21.15.88:0/2651631660 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f189406c2e0 0x7f189406e7a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1898006fd0 tx=0x7f1898006bc0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.435+0000 7f18ad647640 1 -- 172.21.15.88:0/2651631660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18a815e6f0 msgr2=0x7f18a80b1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:24.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.435+0000 7f18ad647640 1 --2- 172.21.15.88:0/2651631660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18a815e6f0 0x7f18a80b1c20 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f18a000cbc0 tx=0x7f18a000a430 comp rx=0 tx=0).stop 2024-09-16T01:36:24.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.435+0000 7f18ad647640 1 -- 172.21.15.88:0/2651631660 shutdown_connections 2024-09-16T01:36:24.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.435+0000 7f18ad647640 1 --2- 172.21.15.88:0/2651631660 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f189406c2e0 0x7f189406e7a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.435+0000 7f18ad647640 1 --2- 172.21.15.88:0/2651631660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18a815e6f0 0x7f18a80b1c20 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.435+0000 7f18ad647640 1 --2- 172.21.15.88:0/2651631660 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18a80b7450 0x7f18a815d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.435+0000 7f18ad647640 1 --2- 172.21.15.88:0/2651631660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18a80b54d0 0x7f18a81f4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.435+0000 7f18ad647640 1 -- 172.21.15.88:0/2651631660 >> 172.21.15.88:0/2651631660 conn(0x7f18a80d2430 msgr2=0x7f18a806da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:24.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.435+0000 7f18ad647640 1 -- 172.21.15.88:0/2651631660 shutdown_connections 2024-09-16T01:36:24.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.435+0000 7f18ad647640 1 -- 172.21.15.88:0/2651631660 wait complete. 2024-09-16T01:36:24.448 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:36:24.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.548+0000 7f50a6fc4640 1 Processor -- start 2024-09-16T01:36:24.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.548+0000 7f50a6fc4640 1 -- start start 2024-09-16T01:36:24.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.549+0000 7f50a6fc4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f50a015a060 0x7f50a006f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:24.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.549+0000 7f50a6fc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50a00b1990 0x7f50a006f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:24.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.549+0000 7f50a6fc4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50a0071770 con 0x7f50a015d3d0 2024-09-16T01:36:24.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.549+0000 7f50a6fc4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f50a00712e0 con 0x7f50a015a060 2024-09-16T01:36:24.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.549+0000 7f50a6fc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f50a00709b0 con 0x7f50a00b1990 2024-09-16T01:36:24.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.549+0000 7f5097fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f50a015d3d0 0x7f50a015ff40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42754/0 (socket says 172.21.15.88:42754) 2024-09-16T01:36:24.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.549+0000 7f5097fff640 1 -- 172.21.15.88:0/1918427489 learned_addr learned my addr 172.21.15.88:0/1918427489 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:24.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.549+0000 7f50a5561640 1 --2- 172.21.15.88:0/1918427489 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50a00b1990 0x7f50a006f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:24.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.549+0000 7f50a4d60640 1 --2- 172.21.15.88:0/1918427489 >> v2:172.21.15.154:3300/0 conn(0x7f50a015a060 0x7f50a006f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:24.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.549+0000 7f50977fe640 1 -- 172.21.15.88:0/1918427489 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4244884604 0 0) 0x7f50a0071770 con 0x7f50a015d3d0 2024-09-16T01:36:24.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.549+0000 7f50977fe640 1 -- 172.21.15.88:0/1918427489 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5088003520 con 0x7f50a015d3d0 2024-09-16T01:36:24.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.549+0000 7f50977fe640 1 -- 172.21.15.88:0/1918427489 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3962595360 0 0) 0x7f5088003520 con 0x7f50a015d3d0 2024-09-16T01:36:24.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.549+0000 7f50977fe640 1 -- 172.21.15.88:0/1918427489 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50a00b1990 msgr2=0x7f50a006f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:24.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.549+0000 7f50977fe640 1 --2- 172.21.15.88:0/1918427489 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50a00b1990 0x7f50a006f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.549+0000 7f50977fe640 1 -- 172.21.15.88:0/1918427489 >> v2:172.21.15.154:3300/0 conn(0x7f50a015a060 msgr2=0x7f50a006f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:24.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.549+0000 7f50977fe640 1 --2- 172.21.15.88:0/1918427489 >> v2:172.21.15.154:3300/0 conn(0x7f50a015a060 0x7f50a006f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.549+0000 7f50977fe640 1 -- 172.21.15.88:0/1918427489 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50a0070c90 con 0x7f50a015d3d0 2024-09-16T01:36:24.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.549+0000 7f50977fe640 1 -- 172.21.15.88:0/1918427489 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f5090001670 con 0x7f50a015d3d0 2024-09-16T01:36:24.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.549+0000 7f50977fe640 1 -- 172.21.15.88:0/1918427489 >> v1:172.21.15.88:6789/0 conn(0x7f50a015d3d0 legacy=0x7f50a015ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:24.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.549+0000 7f50977fe640 1 --2- 172.21.15.88:0/1918427489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50880039b0 0x7f5088003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:24.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.549+0000 7f50977fe640 1 --2- 172.21.15.88:0/1918427489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5088004800 0x7f5088004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:24.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.549+0000 7f50977fe640 1 --2- 172.21.15.88:0/1918427489 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50a00b1990 0x7f50880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:24.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.549+0000 7f50977fe640 1 -- 172.21.15.88:0/1918427489 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f50a00712e0 con 0x7f50880039b0 2024-09-16T01:36:24.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.549+0000 7f50977fe640 1 -- 172.21.15.88:0/1918427489 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f50a00709b0 con 0x7f5088004800 2024-09-16T01:36:24.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.549+0000 7f50977fe640 1 -- 172.21.15.88:0/1918427489 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f508800b9c0 con 0x7f50a00b1990 2024-09-16T01:36:24.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.550+0000 7f50a4d60640 1 --2- 172.21.15.88:0/1918427489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50880039b0 0x7f5088003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:24.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.550+0000 7f5097fff640 1 --2- 172.21.15.88:0/1918427489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5088004800 0x7f5088004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:24.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.550+0000 7f50a5561640 1 --2- 172.21.15.88:0/1918427489 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50a00b1990 0x7f50880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:24.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.550+0000 7f50a4d60640 1 -- 172.21.15.88:0/1918427489 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50a00b1990 msgr2=0x7f50880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:24.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.550+0000 7f50a4d60640 1 --2- 172.21.15.88:0/1918427489 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50a00b1990 0x7f50880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.550+0000 7f50a4d60640 1 -- 172.21.15.88:0/1918427489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5088004800 msgr2=0x7f5088004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:24.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.550+0000 7f50a4d60640 1 --2- 172.21.15.88:0/1918427489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5088004800 0x7f5088004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.550+0000 7f50a4d60640 1 -- 172.21.15.88:0/1918427489 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5088003520 con 0x7f50880039b0 2024-09-16T01:36:24.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.551+0000 7f50a4d60640 1 --2- 172.21.15.88:0/1918427489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50880039b0 0x7f5088003dd0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f509c009ac0 tx=0x7f5088005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaf3442be65ffe server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:24.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.551+0000 7f50977fe640 1 -- 172.21.15.88:0/1918427489 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f509c036070 con 0x7f50880039b0 2024-09-16T01:36:24.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.551+0000 7f50977fe640 1 -- 172.21.15.88:0/1918427489 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f509c031400 con 0x7f50880039b0 2024-09-16T01:36:24.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.551+0000 7f50977fe640 1 -- 172.21.15.88:0/1918427489 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f509c009690 con 0x7f50880039b0 2024-09-16T01:36:24.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.551+0000 7f50a6fc4640 1 -- 172.21.15.88:0/1918427489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50880039b0 msgr2=0x7f5088003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:24.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.551+0000 7f50a6fc4640 1 --2- 172.21.15.88:0/1918427489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50880039b0 0x7f5088003dd0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f509c009ac0 tx=0x7f5088005830 comp rx=0 tx=0).stop 2024-09-16T01:36:24.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.551+0000 7f50a5561640 1 -- 172.21.15.88:0/1918427489 reap_dead start 2024-09-16T01:36:24.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.551+0000 7f50a6fc4640 1 -- 172.21.15.88:0/1918427489 shutdown_connections 2024-09-16T01:36:24.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.551+0000 7f50a6fc4640 1 -- 172.21.15.88:0/1918427489 >> 172.21.15.88:0/1918427489 conn(0x7f50a00d2430 msgr2=0x7f50a00d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:24.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.551+0000 7f50a6fc4640 1 -- 172.21.15.88:0/1918427489 shutdown_connections 2024-09-16T01:36:24.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.551+0000 7f50a6fc4640 1 -- 172.21.15.88:0/1918427489 wait complete. 2024-09-16T01:36:24.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.552+0000 7f50a6fc4640 1 Processor -- start 2024-09-16T01:36:24.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.552+0000 7f50a6fc4640 1 -- start start 2024-09-16T01:36:24.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.553+0000 7f50a6fc4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50880039b0 0x7f50a0156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:24.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.553+0000 7f50a6fc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50a006cd20 0x7f50a00afd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:24.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.553+0000 7f50a6fc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50a014da00 0x7f50a006e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:24.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.553+0000 7f50a6fc4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f50a00d1460 con 0x7f50880039b0 2024-09-16T01:36:24.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.553+0000 7f50a6fc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f50a00d12e0 con 0x7f50a014da00 2024-09-16T01:36:24.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.553+0000 7f50a6fc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f50a006feb0 con 0x7f50a006cd20 2024-09-16T01:36:24.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.553+0000 7f50a4d60640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50880039b0 0x7f50a0156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:24.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.553+0000 7f50a4d60640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50880039b0 0x7f50a0156260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:32874/0 (socket says 172.21.15.88:32874) 2024-09-16T01:36:24.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.553+0000 7f50a4d60640 1 -- 172.21.15.88:0/1383873484 learned_addr learned my addr 172.21.15.88:0/1383873484 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:24.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.554+0000 7f50a5561640 1 --2- 172.21.15.88:0/1383873484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50a014da00 0x7f50a006e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:24.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.554+0000 7f5097fff640 1 --2- 172.21.15.88:0/1383873484 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50a006cd20 0x7f50a00afd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:24.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.554+0000 7f50a4d60640 1 -- 172.21.15.88:0/1383873484 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50a006cd20 msgr2=0x7f50a00afd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:24.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.554+0000 7f50a4d60640 1 --2- 172.21.15.88:0/1383873484 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50a006cd20 0x7f50a00afd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.554+0000 7f50a4d60640 1 -- 172.21.15.88:0/1383873484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50a014da00 msgr2=0x7f50a006e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:24.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.554+0000 7f50a4d60640 1 --2- 172.21.15.88:0/1383873484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50a014da00 0x7f50a006e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.554+0000 7f50a4d60640 1 -- 172.21.15.88:0/1383873484 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50a01f7dd0 con 0x7f50880039b0 2024-09-16T01:36:24.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.554+0000 7f50a4d60640 1 --2- 172.21.15.88:0/1383873484 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50880039b0 0x7f50a0156260 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f509c034040 tx=0x7f509c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:24.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.554+0000 7f50957fa640 1 -- 172.21.15.88:0/1383873484 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f509c040020 con 0x7f50880039b0 2024-09-16T01:36:24.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.554+0000 7f50957fa640 1 -- 172.21.15.88:0/1383873484 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f509c003b40 con 0x7f50880039b0 2024-09-16T01:36:24.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.554+0000 7f50a6fc4640 1 -- 172.21.15.88:0/1383873484 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50a006d270 con 0x7f50880039b0 2024-09-16T01:36:24.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.554+0000 7f50957fa640 1 -- 172.21.15.88:0/1383873484 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f509c036040 con 0x7f50880039b0 2024-09-16T01:36:24.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.554+0000 7f50a6fc4640 1 -- 172.21.15.88:0/1383873484 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50a006d830 con 0x7f50880039b0 2024-09-16T01:36:24.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.555+0000 7f50a6fc4640 1 -- 172.21.15.88:0/1383873484 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50a006f2f0 con 0x7f50880039b0 2024-09-16T01:36:24.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.555+0000 7f50957fa640 1 -- 172.21.15.88:0/1383873484 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f509c043020 con 0x7f50880039b0 2024-09-16T01:36:24.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.556+0000 7f50957fa640 1 --2- 172.21.15.88:0/1383873484 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f508c06c2e0 0x7f508c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:24.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.556+0000 7f50957fa640 1 -- 172.21.15.88:0/1383873484 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f509c0ab8d0 con 0x7f50880039b0 2024-09-16T01:36:24.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.556+0000 7f5097fff640 1 --2- 172.21.15.88:0/1383873484 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f508c06c2e0 0x7f508c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:24.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.556+0000 7f5097fff640 1 --2- 172.21.15.88:0/1383873484 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f508c06c2e0 0x7f508c06e7a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f50a00b1960 tx=0x7f5090009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:24.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.560+0000 7f50957fa640 1 -- 172.21.15.88:0/1383873484 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f509c04a030 con 0x7f50880039b0 2024-09-16T01:36:24.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.760+0000 7f50a6fc4640 1 -- 172.21.15.88:0/1383873484 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f50a00b3b30 con 0x7f50880039b0 2024-09-16T01:36:24.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.798+0000 7f50957fa640 1 -- 172.21.15.88:0/1383873484 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v27) ==== 162+0+0 (secure 0 0 0) 0x7f509c07ee20 con 0x7f50880039b0 2024-09-16T01:36:24.799 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:36:24.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.800+0000 7f50a6fc4640 1 -- 172.21.15.88:0/1383873484 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f508c06c2e0 msgr2=0x7f508c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:24.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.800+0000 7f50a6fc4640 1 --2- 172.21.15.88:0/1383873484 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f508c06c2e0 0x7f508c06e7a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f50a00b1960 tx=0x7f5090009040 comp rx=0 tx=0).stop 2024-09-16T01:36:24.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.800+0000 7f50a6fc4640 1 -- 172.21.15.88:0/1383873484 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50880039b0 msgr2=0x7f50a0156260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:24.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.800+0000 7f50a6fc4640 1 --2- 172.21.15.88:0/1383873484 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50880039b0 0x7f50a0156260 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f509c034040 tx=0x7f509c004730 comp rx=0 tx=0).stop 2024-09-16T01:36:24.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.800+0000 7f50a6fc4640 1 -- 172.21.15.88:0/1383873484 shutdown_connections 2024-09-16T01:36:24.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.800+0000 7f50a6fc4640 1 --2- 172.21.15.88:0/1383873484 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f508c06c2e0 0x7f508c06e7a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.800+0000 7f50a6fc4640 1 --2- 172.21.15.88:0/1383873484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50a014da00 0x7f50a006e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.800+0000 7f50a6fc4640 1 --2- 172.21.15.88:0/1383873484 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50a006cd20 0x7f50a00afd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.800+0000 7f50a6fc4640 1 --2- 172.21.15.88:0/1383873484 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50880039b0 0x7f50a0156260 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.800+0000 7f50a6fc4640 1 -- 172.21.15.88:0/1383873484 >> 172.21.15.88:0/1383873484 conn(0x7f50a00d2430 msgr2=0x7f50a00b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:24.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.800+0000 7f50a6fc4640 1 -- 172.21.15.88:0/1383873484 shutdown_connections 2024-09-16T01:36:24.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.800+0000 7f50a6fc4640 1 -- 172.21.15.88:0/1383873484 wait complete. 2024-09-16T01:36:24.815 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:36:24.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.914+0000 7f4251453640 1 Processor -- start 2024-09-16T01:36:24.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.914+0000 7f4251453640 1 -- start start 2024-09-16T01:36:24.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.915+0000 7f4251453640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f424c0d3310 0x7f424c0d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:24.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.915+0000 7f4251453640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f424c0d1650 0x7f424c0b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:24.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.915+0000 7f4251453640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f424c15ea40 con 0x7f424c15e660 2024-09-16T01:36:24.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.915+0000 7f4251453640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f424c0d1ac0 con 0x7f424c0d1650 2024-09-16T01:36:24.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.915+0000 7f4251453640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f424c0d3870 con 0x7f424c0d3310 2024-09-16T01:36:24.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.915+0000 7f424b7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f424c0d1650 0x7f424c0b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:24.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.915+0000 7f424b7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f424c0d1650 0x7f424c0b7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:32890/0 (socket says 172.21.15.88:32890) 2024-09-16T01:36:24.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.915+0000 7f424b7fe640 1 -- 172.21.15.88:0/3241216957 learned_addr learned my addr 172.21.15.88:0/3241216957 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:24.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.915+0000 7f424affd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f424c15e660 0x7f424c0cf730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42764/0 (socket says 172.21.15.88:42764) 2024-09-16T01:36:24.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.915+0000 7f42427fc640 1 --2- 172.21.15.88:0/3241216957 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f424c0d3310 0x7f424c0d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:24.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.915+0000 7f424b7fe640 1 -- 172.21.15.88:0/3241216957 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f424c0d3310 msgr2=0x7f424c0d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:24.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.915+0000 7f424b7fe640 1 --2- 172.21.15.88:0/3241216957 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f424c0d3310 0x7f424c0d1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.915+0000 7f424b7fe640 1 -- 172.21.15.88:0/3241216957 >> v1:172.21.15.88:6789/0 conn(0x7f424c15e660 legacy=0x7f424c0cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.915+0000 7f424b7fe640 1 -- 172.21.15.88:0/3241216957 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f424c0d3b50 con 0x7f424c0d1650 2024-09-16T01:36:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.915+0000 7f424a7fc640 1 -- 172.21.15.88:0/3241216957 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1723023246 0 0) 0x7f424c15ea40 con 0x7f424c15e660 2024-09-16T01:36:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.915+0000 7f42427fc640 1 --2- 172.21.15.88:0/3241216957 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f424c0d3310 0x7f424c0d1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.916+0000 7f424b7fe640 1 --2- 172.21.15.88:0/3241216957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f424c0d1650 0x7f424c0b7b70 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4244009e50 tx=0x7f424402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cef9c43ea3a21a5c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.916+0000 7f424a7fc640 1 -- 172.21.15.88:0/3241216957 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4244036070 con 0x7f424c0d1650 2024-09-16T01:36:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.916+0000 7f424a7fc640 1 -- 172.21.15.88:0/3241216957 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4244031600 con 0x7f424c0d1650 2024-09-16T01:36:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.916+0000 7f424a7fc640 1 -- 172.21.15.88:0/3241216957 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42440319d0 con 0x7f424c0d1650 2024-09-16T01:36:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.917+0000 7f4251453640 1 -- 172.21.15.88:0/3241216957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f424c0d1650 msgr2=0x7f424c0b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.917+0000 7f4251453640 1 --2- 172.21.15.88:0/3241216957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f424c0d1650 0x7f424c0b7b70 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4244009e50 tx=0x7f424402a710 comp rx=0 tx=0).stop 2024-09-16T01:36:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.917+0000 7f4251453640 1 -- 172.21.15.88:0/3241216957 shutdown_connections 2024-09-16T01:36:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.917+0000 7f4251453640 1 --2- 172.21.15.88:0/3241216957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f424c0d1650 0x7f424c0b7b70 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.917+0000 7f4251453640 1 --2- 172.21.15.88:0/3241216957 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f424c0d3310 0x7f424c0d1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.917+0000 7f4251453640 1 -- 172.21.15.88:0/3241216957 >> 172.21.15.88:0/3241216957 conn(0x7f424c0b6410 msgr2=0x7f424c0b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:24.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.917+0000 7f4251453640 1 -- 172.21.15.88:0/3241216957 shutdown_connections 2024-09-16T01:36:24.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.917+0000 7f4251453640 1 -- 172.21.15.88:0/3241216957 wait complete. 2024-09-16T01:36:24.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.918+0000 7f4251453640 1 Processor -- start 2024-09-16T01:36:24.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.918+0000 7f4251453640 1 -- start start 2024-09-16T01:36:24.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.919+0000 7f4251453640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f424c0d1650 0x7f424c1f41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:24.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.919+0000 7f4251453640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f424c0d3310 0x7f424c0c9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:24.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.919+0000 7f4251453640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f424c15e660 0x7f424c0b9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:24.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.919+0000 7f4251453640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f424c0d43f0 con 0x7f424c0d1650 2024-09-16T01:36:24.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.919+0000 7f4251453640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f424c0d4110 con 0x7f424c15e660 2024-09-16T01:36:24.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.919+0000 7f4251453640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f424c0b80b0 con 0x7f424c0d3310 2024-09-16T01:36:24.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.919+0000 7f424affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f424c0d1650 0x7f424c1f41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:24.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.919+0000 7f424affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f424c0d1650 0x7f424c1f41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:32900/0 (socket says 172.21.15.88:32900) 2024-09-16T01:36:24.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.919+0000 7f424affd640 1 -- 172.21.15.88:0/3407583063 learned_addr learned my addr 172.21.15.88:0/3407583063 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:24.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.919+0000 7f424b7fe640 1 --2- 172.21.15.88:0/3407583063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f424c15e660 0x7f424c0b9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.919+0000 7f42427fc640 1 --2- 172.21.15.88:0/3407583063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f424c0d3310 0x7f424c0c9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.919+0000 7f424affd640 1 -- 172.21.15.88:0/3407583063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f424c0d3310 msgr2=0x7f424c0c9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.919+0000 7f424affd640 1 --2- 172.21.15.88:0/3407583063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f424c0d3310 0x7f424c0c9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.919+0000 7f424affd640 1 -- 172.21.15.88:0/3407583063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f424c15e660 msgr2=0x7f424c0b9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.919+0000 7f424affd640 1 --2- 172.21.15.88:0/3407583063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f424c15e660 0x7f424c0b9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.919+0000 7f424affd640 1 -- 172.21.15.88:0/3407583063 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f424c0c8290 con 0x7f424c0d1650 2024-09-16T01:36:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.919+0000 7f42427fc640 1 --2- 172.21.15.88:0/3407583063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f424c0d3310 0x7f424c0c9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.919+0000 7f424b7fe640 1 --2- 172.21.15.88:0/3407583063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f424c15e660 0x7f424c0b9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.919+0000 7f424affd640 1 --2- 172.21.15.88:0/3407583063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f424c0d1650 0x7f424c1f41d0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f423c00c460 tx=0x7f423c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.919+0000 7f4243fff640 1 -- 172.21.15.88:0/3407583063 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423c018070 con 0x7f424c0d1650 2024-09-16T01:36:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.919+0000 7f4243fff640 1 -- 172.21.15.88:0/3407583063 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f423c007500 con 0x7f424c0d1650 2024-09-16T01:36:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.919+0000 7f4251453640 1 -- 172.21.15.88:0/3407583063 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f424c0c8570 con 0x7f424c0d1650 2024-09-16T01:36:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.919+0000 7f4243fff640 1 -- 172.21.15.88:0/3407583063 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423c004a20 con 0x7f424c0d1650 2024-09-16T01:36:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.919+0000 7f4251453640 1 -- 172.21.15.88:0/3407583063 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f424c0c8b30 con 0x7f424c0d1650 2024-09-16T01:36:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.920+0000 7f4251453640 1 -- 172.21.15.88:0/3407583063 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f424c0b7b70 con 0x7f424c0d1650 2024-09-16T01:36:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.921+0000 7f4243fff640 1 -- 172.21.15.88:0/3407583063 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f423c01b070 con 0x7f424c0d1650 2024-09-16T01:36:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.921+0000 7f4243fff640 1 --2- 172.21.15.88:0/3407583063 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f422c06c2b0 0x7f422c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.921+0000 7f4243fff640 1 -- 172.21.15.88:0/3407583063 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f423c08a690 con 0x7f424c0d1650 2024-09-16T01:36:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.921+0000 7f42427fc640 1 --2- 172.21.15.88:0/3407583063 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f422c06c2b0 0x7f422c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.922+0000 7f42427fc640 1 --2- 172.21.15.88:0/3407583063 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f422c06c2b0 0x7f422c06e770 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4234006fd0 tx=0x7f4234008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:24.925+0000 7f4243fff640 1 -- 172.21.15.88:0/3407583063 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f423c058e30 con 0x7f424c0d1650 2024-09-16T01:36:25.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.126+0000 7f4251453640 1 -- 172.21.15.88:0/3407583063 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f424c0b6ad0 con 0x7f424c0d1650 2024-09-16T01:36:25.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.127+0000 7f4243fff640 1 -- 172.21.15.88:0/3407583063 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5220 (secure 0 0 0) 0x7f423c058730 con 0x7f424c0d1650 2024-09-16T01:36:25.128 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:25.128 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":28,"btime":"2024-09-16T01:36:24:800958+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4877,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1769276566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":1769276566},{"type":"v1","addr":"172.21.15.154:6835","nonce":1769276566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/725621378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":725621378},{"type":"v1","addr":"172.21.15.154:6834","nonce":725621378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4199111872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4199111872},{"type":"v1","addr":"172.21.15.88:6837","nonce":4199111872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:05.691540+0000","modified":"2024-09-16T01:36:24.800947+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"c","rank":0,"incarnation":28,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6836/2700280540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2700280540},{"type":"v1","addr":"172.21.15.88:6836","nonce":2700280540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-09-16T01:36:25.128 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 28 2024-09-16T01:36:25.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.129+0000 7f4251453640 1 -- 172.21.15.88:0/3407583063 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f422c06c2b0 msgr2=0x7f422c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:25.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.129+0000 7f4251453640 1 --2- 172.21.15.88:0/3407583063 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f422c06c2b0 0x7f422c06e770 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4234006fd0 tx=0x7f4234008040 comp rx=0 tx=0).stop 2024-09-16T01:36:25.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.129+0000 7f4251453640 1 -- 172.21.15.88:0/3407583063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f424c0d1650 msgr2=0x7f424c1f41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:25.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.129+0000 7f4251453640 1 --2- 172.21.15.88:0/3407583063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f424c0d1650 0x7f424c1f41d0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f423c00c460 tx=0x7f423c00c920 comp rx=0 tx=0).stop 2024-09-16T01:36:25.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.129+0000 7f4251453640 1 -- 172.21.15.88:0/3407583063 shutdown_connections 2024-09-16T01:36:25.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.129+0000 7f4251453640 1 --2- 172.21.15.88:0/3407583063 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f422c06c2b0 0x7f422c06e770 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.129+0000 7f4251453640 1 --2- 172.21.15.88:0/3407583063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f424c15e660 0x7f424c0b9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.129+0000 7f4251453640 1 --2- 172.21.15.88:0/3407583063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f424c0d3310 0x7f424c0c9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.129+0000 7f4251453640 1 --2- 172.21.15.88:0/3407583063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f424c0d1650 0x7f424c1f41d0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.129+0000 7f4251453640 1 -- 172.21.15.88:0/3407583063 >> 172.21.15.88:0/3407583063 conn(0x7f424c0b6410 msgr2=0x7f424c0b57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:25.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.129+0000 7f4251453640 1 -- 172.21.15.88:0/3407583063 shutdown_connections 2024-09-16T01:36:25.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.129+0000 7f4251453640 1 -- 172.21.15.88:0/3407583063 wait complete. 2024-09-16T01:36:25.142 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 flush journal 2024-09-16T01:36:25.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.241+0000 7ff4cc642640 1 Processor -- start 2024-09-16T01:36:25.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.241+0000 7ff4cc642640 1 -- start start 2024-09-16T01:36:25.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.242+0000 7ff4cc642640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff4c406f710 0x7ff4c406fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.242+0000 7ff4cc642640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4c4074120 con 0x7ff4c415c2a0 2024-09-16T01:36:25.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.242+0000 7ff4cc642640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4c415ef00 con 0x7ff4c406bc10 2024-09-16T01:36:25.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.242+0000 7ff4cc642640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff4c415c680 con 0x7ff4c406f710 2024-09-16T01:36:25.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.242+0000 7ff4c9bdd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff4c406f710 0x7ff4c406fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.242+0000 7ff4c9bdd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff4c406f710 0x7ff4c406fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:32912/0 (socket says 172.21.15.88:32912) 2024-09-16T01:36:25.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.242+0000 7ff4ca3de640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff4c415c2a0 0x7ff4c40735d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42766/0 (socket says 172.21.15.88:42766) 2024-09-16T01:36:25.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.242+0000 7ff4c9bdd640 1 -- 172.21.15.88:0/2100801642 learned_addr learned my addr 172.21.15.88:0/2100801642 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:25.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.243+0000 7ff4c9bdd640 1 -- 172.21.15.88:0/2100801642 >> v1:172.21.15.154:6789/0 conn(0x7ff4c406bc10 legacy=0x7ff4c406c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:25.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.243+0000 7ff4c9bdd640 1 -- 172.21.15.88:0/2100801642 >> v1:172.21.15.88:6789/0 conn(0x7ff4c415c2a0 legacy=0x7ff4c40735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:25.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.243+0000 7ff4c9bdd640 1 -- 172.21.15.88:0/2100801642 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c415f1e0 con 0x7ff4c406f710 2024-09-16T01:36:25.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.243+0000 7ff4c93dc640 1 -- 172.21.15.88:0/2100801642 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3122117054 0 0) 0x7ff4c4074120 con 0x7ff4c415c2a0 2024-09-16T01:36:25.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.243+0000 7ff4c9bdd640 1 --2- 172.21.15.88:0/2100801642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c406f710 0x7ff4c406fba0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4009870 tx=0x7ff4b402a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80a9051bae7b5f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:25.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.243+0000 7ff4c93dc640 1 -- 172.21.15.88:0/2100801642 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4037070 con 0x7ff4c406f710 2024-09-16T01:36:25.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.243+0000 7ff4c93dc640 1 -- 172.21.15.88:0/2100801642 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b4032440 con 0x7ff4c406f710 2024-09-16T01:36:25.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.243+0000 7ff4c93dc640 1 -- 172.21.15.88:0/2100801642 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4032810 con 0x7ff4c406f710 2024-09-16T01:36:25.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.244+0000 7ff4cc642640 1 -- 172.21.15.88:0/2100801642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c406f710 msgr2=0x7ff4c406fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:25.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.244+0000 7ff4cc642640 1 --2- 172.21.15.88:0/2100801642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c406f710 0x7ff4c406fba0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4009870 tx=0x7ff4b402a2a0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.244+0000 7ff4cc642640 1 -- 172.21.15.88:0/2100801642 shutdown_connections 2024-09-16T01:36:25.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.244+0000 7ff4cc642640 1 --2- 172.21.15.88:0/2100801642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c406f710 0x7ff4c406fba0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.244+0000 7ff4cc642640 1 -- 172.21.15.88:0/2100801642 >> 172.21.15.88:0/2100801642 conn(0x7ff4c40b3070 msgr2=0x7ff4c40b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:25.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.244+0000 7ff4cc642640 1 -- 172.21.15.88:0/2100801642 shutdown_connections 2024-09-16T01:36:25.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.244+0000 7ff4cc642640 1 -- 172.21.15.88:0/2100801642 wait complete. 2024-09-16T01:36:25.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.245+0000 7ff4cc642640 1 Processor -- start 2024-09-16T01:36:25.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.245+0000 7ff4cc642640 1 -- start start 2024-09-16T01:36:25.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.246+0000 7ff4cc642640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4c406bc10 0x7ff4c40cdd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.246+0000 7ff4cc642640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4c406f710 0x7ff4c40d0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.246+0000 7ff4cc642640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c415c2a0 0x7ff4c40cf890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.246+0000 7ff4cc642640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c40b5f20 con 0x7ff4c415c2a0 2024-09-16T01:36:25.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.246+0000 7ff4cc642640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c415ebf0 con 0x7ff4c406bc10 2024-09-16T01:36:25.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.246+0000 7ff4cc642640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff4c40b60a0 con 0x7ff4c406f710 2024-09-16T01:36:25.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.246+0000 7ff4cabdf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c415c2a0 0x7ff4c40cf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.246+0000 7ff4cabdf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c415c2a0 0x7ff4c40cf890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:32920/0 (socket says 172.21.15.88:32920) 2024-09-16T01:36:25.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.246+0000 7ff4cabdf640 1 -- 172.21.15.88:0/80816074 learned_addr learned my addr 172.21.15.88:0/80816074 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:25.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.246+0000 7ff4ca3de640 1 --2- 172.21.15.88:0/80816074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4c406bc10 0x7ff4c40cdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.246+0000 7ff4c9bdd640 1 --2- 172.21.15.88:0/80816074 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4c406f710 0x7ff4c40d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.246+0000 7ff4cabdf640 1 -- 172.21.15.88:0/80816074 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4c406f710 msgr2=0x7ff4c40d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:25.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.246+0000 7ff4cabdf640 1 --2- 172.21.15.88:0/80816074 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4c406f710 0x7ff4c40d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.246+0000 7ff4cabdf640 1 -- 172.21.15.88:0/80816074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4c406bc10 msgr2=0x7ff4c40cdd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:25.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.246+0000 7ff4cabdf640 1 --2- 172.21.15.88:0/80816074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4c406bc10 0x7ff4c40cdd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.247+0000 7ff4cabdf640 1 -- 172.21.15.88:0/80816074 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c4158840 con 0x7ff4c415c2a0 2024-09-16T01:36:25.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.247+0000 7ff4cabdf640 1 --2- 172.21.15.88:0/80816074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c415c2a0 0x7ff4c40cf890 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc00c410 tx=0x7ff4bc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:25.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.247+0000 7ff4baffd640 1 -- 172.21.15.88:0/80816074 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc011790 con 0x7ff4c415c2a0 2024-09-16T01:36:25.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.247+0000 7ff4baffd640 1 -- 172.21.15.88:0/80816074 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4bc017070 con 0x7ff4c415c2a0 2024-09-16T01:36:25.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.247+0000 7ff4cc642640 1 -- 172.21.15.88:0/80816074 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4c4158b20 con 0x7ff4c415c2a0 2024-09-16T01:36:25.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.247+0000 7ff4baffd640 1 -- 172.21.15.88:0/80816074 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc011bb0 con 0x7ff4c415c2a0 2024-09-16T01:36:25.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.247+0000 7ff4cc642640 1 -- 172.21.15.88:0/80816074 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4c414a930 con 0x7ff4c415c2a0 2024-09-16T01:36:25.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.248+0000 7ff4baffd640 1 -- 172.21.15.88:0/80816074 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff4bc01c460 con 0x7ff4c415c2a0 2024-09-16T01:36:25.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.249+0000 7ff4baffd640 1 --2- 172.21.15.88:0/80816074 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff4a806c2e0 0x7ff4a806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.249+0000 7ff4baffd640 1 -- 172.21.15.88:0/80816074 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7ff4bc08b020 con 0x7ff4c415c2a0 2024-09-16T01:36:25.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.249+0000 7ff4ca3de640 1 --2- 172.21.15.88:0/80816074 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff4a806c2e0 0x7ff4a806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.249+0000 7ff4ca3de640 1 --2- 172.21.15.88:0/80816074 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff4a806c2e0 0x7ff4a806e7a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff4c00012e0 tx=0x7ff4c0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.250+0000 7ff4cf3d1740 1 Processor -- start 2024-09-16T01:36:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.250+0000 7ff4cf3d1740 1 -- start start 2024-09-16T01:36:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.251+0000 7ff4cf3d1740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55634315a920 0x5563431a00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.251+0000 7ff4cf3d1740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563431580e0 0x55634315ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.251+0000 7ff4cf3d1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55634315c0b0 0x5563431551a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.251+0000 7ff4cf3d1740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5563430bd6f0 con 0x5563431580e0 2024-09-16T01:36:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.251+0000 7ff4cf3d1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556343065330 con 0x55634315c0b0 2024-09-16T01:36:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.251+0000 7ff4cf3d1740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5563430b96b0 con 0x55634315a920 2024-09-16T01:36:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.251+0000 7ff4c9bdd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563431580e0 0x55634315ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.251+0000 7ff4c9bdd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563431580e0 0x55634315ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:32924/0 (socket says 172.21.15.88:32924) 2024-09-16T01:36:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.251+0000 7ff4c9bdd640 1 -- 172.21.15.88:0/3799219280 learned_addr learned my addr 172.21.15.88:0/3799219280 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.251+0000 7ff4cabdf640 1 --2- 172.21.15.88:0/3799219280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55634315c0b0 0x5563431551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.251+0000 7ff4ca3de640 1 --2- 172.21.15.88:0/3799219280 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55634315a920 0x5563431a00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.251+0000 7ff4c9bdd640 1 -- 172.21.15.88:0/3799219280 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55634315a920 msgr2=0x5563431a00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.251+0000 7ff4c9bdd640 1 --2- 172.21.15.88:0/3799219280 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55634315a920 0x5563431a00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.251+0000 7ff4c9bdd640 1 -- 172.21.15.88:0/3799219280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55634315c0b0 msgr2=0x5563431551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.251+0000 7ff4c9bdd640 1 --2- 172.21.15.88:0/3799219280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55634315c0b0 0x5563431551a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.251+0000 7ff4c9bdd640 1 -- 172.21.15.88:0/3799219280 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55634308b0d0 con 0x5563431580e0 2024-09-16T01:36:25.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.251+0000 7ff4c9bdd640 1 --2- 172.21.15.88:0/3799219280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563431580e0 0x55634315ad00 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ff4b402a890 tx=0x7ff4b402c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=845d8fd5a3a82788 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:25.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.252+0000 7ff4a7fff640 1 -- 172.21.15.88:0/3799219280 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4037040 con 0x5563431580e0 2024-09-16T01:36:25.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.252+0000 7ff4a7fff640 1 -- 172.21.15.88:0/3799219280 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b403e660 con 0x5563431580e0 2024-09-16T01:36:25.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.252+0000 7ff4a7fff640 1 -- 172.21.15.88:0/3799219280 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b403ea10 con 0x5563431580e0 2024-09-16T01:36:25.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.252+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/3799219280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563431580e0 msgr2=0x55634315ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:25.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.252+0000 7ff4cf3d1740 1 --2- 172.21.15.88:0/3799219280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563431580e0 0x55634315ad00 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ff4b402a890 tx=0x7ff4b402c000 comp rx=0 tx=0).stop 2024-09-16T01:36:25.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.252+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/3799219280 shutdown_connections 2024-09-16T01:36:25.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.252+0000 7ff4cf3d1740 1 --2- 172.21.15.88:0/3799219280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55634315c0b0 0x5563431551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.252+0000 7ff4cf3d1740 1 --2- 172.21.15.88:0/3799219280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563431580e0 0x55634315ad00 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.252+0000 7ff4cf3d1740 1 --2- 172.21.15.88:0/3799219280 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55634315a920 0x5563431a00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.252+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/3799219280 >> 172.21.15.88:0/3799219280 conn(0x556343157370 msgr2=0x55634319f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:25.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.252+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/3799219280 shutdown_connections 2024-09-16T01:36:25.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.252+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/3799219280 wait complete. 2024-09-16T01:36:25.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.253+0000 7ff4cf3d1740 1 -- start start 2024-09-16T01:36:25.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.253+0000 7ff4cf3d1740 1 Processor -- start 2024-09-16T01:36:25.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.254+0000 7ff4cf3d1740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55634315a920 0x5563431768e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.254+0000 7ff4cf3d1740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55634315c0b0 0x556343186350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.254+0000 7ff4cf3d1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556343183f90 0x556343184410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.254+0000 7ff4cf3d1740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x556343156810 con 0x55634315c0b0 2024-09-16T01:36:25.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.254+0000 7ff4cf3d1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5563430464f0 con 0x556343183f90 2024-09-16T01:36:25.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.254+0000 7ff4cf3d1740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5563431a08f0 con 0x55634315a920 2024-09-16T01:36:25.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.254+0000 7ff4cf3d1740 10 client.0 _ll_register_callbacks cb 0x556343067b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:36:25.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.254+0000 7ff4c9bdd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55634315c0b0 0x556343186350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.254+0000 7ff4c9bdd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55634315c0b0 0x556343186350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:32932/0 (socket says 172.21.15.88:32932) 2024-09-16T01:36:25.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.254+0000 7ff4c9bdd640 1 -- 172.21.15.88:0/1760726846 learned_addr learned my addr 172.21.15.88:0/1760726846 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:25.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.254+0000 7ff4cabdf640 1 --2- 172.21.15.88:0/1760726846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556343183f90 0x556343184410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.254+0000 7ff4ca3de640 1 --2- 172.21.15.88:0/1760726846 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55634315a920 0x5563431768e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.254+0000 7ff4c9bdd640 1 -- 172.21.15.88:0/1760726846 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55634315a920 msgr2=0x5563431768e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:25.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.254+0000 7ff4c9bdd640 1 --2- 172.21.15.88:0/1760726846 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55634315a920 0x5563431768e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.254+0000 7ff4c9bdd640 1 -- 172.21.15.88:0/1760726846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556343183f90 msgr2=0x556343184410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:25.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.254+0000 7ff4c9bdd640 1 --2- 172.21.15.88:0/1760726846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556343183f90 0x556343184410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.254+0000 7ff4c9bdd640 1 -- 172.21.15.88:0/1760726846 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556342e9a7d0 con 0x55634315c0b0 2024-09-16T01:36:25.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.254+0000 7ff4cabdf640 1 --2- 172.21.15.88:0/1760726846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556343183f90 0x556343184410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:25.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.254+0000 7ff4c9bdd640 1 --2- 172.21.15.88:0/1760726846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55634315c0b0 0x556343186350 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4009960 tx=0x7ff4b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da8b99fc9c2d7ea server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:25.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.255+0000 7ff4a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:36:25.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.255+0000 7ff4a57fa640 1 -- 172.21.15.88:0/1760726846 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4032650 con 0x55634315c0b0 2024-09-16T01:36:25.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.255+0000 7ff4a57fa640 1 -- 172.21.15.88:0/1760726846 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7ff4b4051b20 con 0x55634315c0b0 2024-09-16T01:36:25.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.255+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/1760726846 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4b403e660 con 0x55634315c0b0 2024-09-16T01:36:25.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.255+0000 7ff4a57fa640 1 -- 172.21.15.88:0/1760726846 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff4b4051d20 con 0x55634315c0b0 2024-09-16T01:36:25.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.255+0000 7ff4cf3d1740 10 client.5682 fetch_fsmap learned FSMap version 28 2024-09-16T01:36:25.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.255+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/1760726846 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) -- 0x556342dd03f0 con 0x55634315c0b0 2024-09-16T01:36:25.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.255+0000 7ff4a57fa640 1 -- 172.21.15.88:0/1760726846 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b403e660 con 0x55634315c0b0 2024-09-16T01:36:25.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.255+0000 7ff4a57fa640 1 -- 172.21.15.88:0/1760726846 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 28) ==== 3494+0+0 (secure 0 0 0) 0x7ff4b40327f0 con 0x55634315c0b0 2024-09-16T01:36:25.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.255+0000 7ff4cf3d1740 10 client.5682 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:36:25.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.255+0000 7ff4cf3d1740 10 client.5682 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-16T01:36:25.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.255+0000 7ff4cf3d1740 20 client.5682 populate_metadata read hostname 'smithi088' 2024-09-16T01:36:25.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.255+0000 7ff4cf3d1740 1 --2- 172.21.15.88:0/1760726846 >> [v2:172.21.15.88:6834/2700280540,v1:172.21.15.88:6836/2700280540] conn(0x5563431b3370 0x5563431b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.255+0000 7ff4cf3d1740 4 client.5682 mds_command: new command op to 5634 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:36:25.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.255+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/1760726846 --> [v2:172.21.15.88:6834/2700280540,v1:172.21.15.88:6836/2700280540] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563430cae20 con 0x5563431b3370 2024-09-16T01:36:25.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.255+0000 7ff4ca3de640 1 --2- 172.21.15.88:0/1760726846 >> [v2:172.21.15.88:6834/2700280540,v1:172.21.15.88:6836/2700280540] conn(0x5563431b3370 0x5563431b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.256+0000 7ff4a57fa640 10 client.5682 ms_handle_connect on v2:172.21.15.88:6834/2700280540 2024-09-16T01:36:25.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.256+0000 7ff4ca3de640 1 --2- 172.21.15.88:0/1760726846 >> [v2:172.21.15.88:6834/2700280540,v1:172.21.15.88:6836/2700280540] conn(0x5563431b3370 0x5563431b3750 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91e7ec37daed5165 server_cookie=e3f4fab7411fd9dc in_seq=0 out_seq=0 2024-09-16T01:36:25.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.256+0000 7ff4a57fa640 1 -- 172.21.15.88:0/1760726846 <== mds.0 v2:172.21.15.88:6834/2700280540 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5563430cae20 con 0x5563431b3370 2024-09-16T01:36:25.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.256+0000 7ff4a57fa640 10 client.5682 handle_command_reply: tid=0 2024-09-16T01:36:25.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.256+0000 7ff4cf3d1740 1 client.5682 shutdown 2024-09-16T01:36:25.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.256+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/1760726846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55634315c0b0 msgr2=0x556343186350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:25.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.256+0000 7ff4cf3d1740 1 --2- 172.21.15.88:0/1760726846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55634315c0b0 0x556343186350 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4009960 tx=0x7ff4b4004730 comp rx=0 tx=0).stop 2024-09-16T01:36:25.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.256+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/1760726846 shutdown_connections 2024-09-16T01:36:25.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.256+0000 7ff4cf3d1740 1 --2- 172.21.15.88:0/1760726846 >> [v2:172.21.15.88:6834/2700280540,v1:172.21.15.88:6836/2700280540] conn(0x5563431b3370 0x5563431b3750 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.256+0000 7ff4cf3d1740 1 --2- 172.21.15.88:0/1760726846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556343183f90 0x556343184410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.256+0000 7ff4cf3d1740 1 --2- 172.21.15.88:0/1760726846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55634315c0b0 0x556343186350 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.256+0000 7ff4cf3d1740 1 --2- 172.21.15.88:0/1760726846 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55634315a920 0x5563431768e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.256+0000 7ff4a57fa640 0 client.5682 ms_handle_reset on v2:172.21.15.88:6834/2700280540 2024-09-16T01:36:25.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.256+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/1760726846 >> 172.21.15.88:0/1760726846 conn(0x5563431580e0 msgr2=0x5563431a10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:25.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.256+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/1760726846 shutdown_connections 2024-09-16T01:36:25.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.257+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/1760726846 wait complete. 2024-09-16T01:36:25.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.257+0000 7ff4cf3d1740 20 client.5682 trim_cache size 0 max 16384 2024-09-16T01:36:25.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.271+0000 7ff4cf3d1740 1 Processor -- start 2024-09-16T01:36:25.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.271+0000 7ff4cf3d1740 1 -- start start 2024-09-16T01:36:25.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.272+0000 7ff4cf3d1740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55634315a920 0x5563431af390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.272+0000 7ff4cf3d1740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55634315c0b0 0x5563431b3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.272+0000 7ff4cf3d1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556343183f90 0x556343175ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.272+0000 7ff4cf3d1740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c4200b00 con 0x55634315c0b0 2024-09-16T01:36:25.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.272+0000 7ff4cf3d1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c41ffb40 con 0x556343183f90 2024-09-16T01:36:25.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.272+0000 7ff4cf3d1740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff4c4157140 con 0x55634315a920 2024-09-16T01:36:25.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.272+0000 7ff4c9bdd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55634315c0b0 0x5563431b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.272+0000 7ff4c9bdd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55634315c0b0 0x5563431b3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:32940/0 (socket says 172.21.15.88:32940) 2024-09-16T01:36:25.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.272+0000 7ff4c9bdd640 1 -- 172.21.15.88:0/2398809257 learned_addr learned my addr 172.21.15.88:0/2398809257 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:25.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.272+0000 7ff4cabdf640 1 --2- 172.21.15.88:0/2398809257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556343183f90 0x556343175ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.272+0000 7ff4ca3de640 1 --2- 172.21.15.88:0/2398809257 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55634315a920 0x5563431af390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.272+0000 7ff4c9bdd640 1 -- 172.21.15.88:0/2398809257 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55634315a920 msgr2=0x5563431af390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:25.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.272+0000 7ff4c9bdd640 1 --2- 172.21.15.88:0/2398809257 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55634315a920 0x5563431af390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.272+0000 7ff4c9bdd640 1 -- 172.21.15.88:0/2398809257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556343183f90 msgr2=0x556343175ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:25.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.272+0000 7ff4c9bdd640 1 --2- 172.21.15.88:0/2398809257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556343183f90 0x556343175ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.272+0000 7ff4c9bdd640 1 -- 172.21.15.88:0/2398809257 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556342d900f0 con 0x55634315c0b0 2024-09-16T01:36:25.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.272+0000 7ff4c9bdd640 1 --2- 172.21.15.88:0/2398809257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55634315c0b0 0x5563431b3750 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff4b40090b0 tx=0x7ff4b4040040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cc71f62769f3b9d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:25.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.272+0000 7ff4a77fe640 1 -- 172.21.15.88:0/2398809257 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b405e070 con 0x55634315c0b0 2024-09-16T01:36:25.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.272+0000 7ff4a77fe640 1 -- 172.21.15.88:0/2398809257 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b403eb20 con 0x55634315c0b0 2024-09-16T01:36:25.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.272+0000 7ff4a77fe640 1 -- 172.21.15.88:0/2398809257 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4037070 con 0x55634315c0b0 2024-09-16T01:36:25.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.273+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/2398809257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55634315c0b0 msgr2=0x5563431b3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:25.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.273+0000 7ff4cf3d1740 1 --2- 172.21.15.88:0/2398809257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55634315c0b0 0x5563431b3750 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff4b40090b0 tx=0x7ff4b4040040 comp rx=0 tx=0).stop 2024-09-16T01:36:25.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.273+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/2398809257 shutdown_connections 2024-09-16T01:36:25.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.273+0000 7ff4cf3d1740 1 --2- 172.21.15.88:0/2398809257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556343183f90 0x556343175ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.273+0000 7ff4cf3d1740 1 --2- 172.21.15.88:0/2398809257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55634315c0b0 0x5563431b3750 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.273+0000 7ff4cf3d1740 1 --2- 172.21.15.88:0/2398809257 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55634315a920 0x5563431af390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.273+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/2398809257 >> 172.21.15.88:0/2398809257 conn(0x5563431580e0 msgr2=0x55634318ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:25.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.273+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/2398809257 shutdown_connections 2024-09-16T01:36:25.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.273+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/2398809257 wait complete. 2024-09-16T01:36:25.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.273+0000 7ff4cf3d1740 1 -- start start 2024-09-16T01:36:25.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.274+0000 7ff4cf3d1740 1 Processor -- start 2024-09-16T01:36:25.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.274+0000 7ff4cf3d1740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55634315a920 0x556343182d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.274+0000 7ff4cf3d1740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55634315c0b0 0x5563431b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.274+0000 7ff4cf3d1740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556343183f90 0x5563431b1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.274+0000 7ff4cf3d1740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55634315bf10 con 0x556343183f90 2024-09-16T01:36:25.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.274+0000 7ff4cf3d1740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c4157790 con 0x55634315a920 2024-09-16T01:36:25.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.274+0000 7ff4cf3d1740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x556343088df0 con 0x55634315c0b0 2024-09-16T01:36:25.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.274+0000 7ff4cf3d1740 10 client.0 _ll_register_callbacks cb 0x7ff4bc055350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:36:25.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.275+0000 7ff4cabdf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556343183f90 0x5563431b1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.275+0000 7ff4cabdf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556343183f90 0x5563431b1ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:32952/0 (socket says 172.21.15.88:32952) 2024-09-16T01:36:25.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.275+0000 7ff4cabdf640 1 -- 172.21.15.88:0/1717040526 learned_addr learned my addr 172.21.15.88:0/1717040526 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:25.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.275+0000 7ff4ca3de640 1 --2- 172.21.15.88:0/1717040526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55634315a920 0x556343182d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.275+0000 7ff4c9bdd640 1 --2- 172.21.15.88:0/1717040526 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55634315c0b0 0x5563431b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.275+0000 7ff4cabdf640 1 -- 172.21.15.88:0/1717040526 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55634315c0b0 msgr2=0x5563431b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:25.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.275+0000 7ff4cabdf640 1 --2- 172.21.15.88:0/1717040526 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55634315c0b0 0x5563431b19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.275+0000 7ff4cabdf640 1 -- 172.21.15.88:0/1717040526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55634315a920 msgr2=0x556343182d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:25.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.275+0000 7ff4cabdf640 1 --2- 172.21.15.88:0/1717040526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55634315a920 0x556343182d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.275+0000 7ff4cabdf640 1 -- 172.21.15.88:0/1717040526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55634319cae0 con 0x556343183f90 2024-09-16T01:36:25.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.275+0000 7ff4cabdf640 1 --2- 172.21.15.88:0/1717040526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556343183f90 0x5563431b1ef0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc00a040 tx=0x7ff4bc05d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d544072d757430b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:25.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.275+0000 7ff4a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:36:25.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.275+0000 7ff4a57fa640 1 -- 172.21.15.88:0/1717040526 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc087060 con 0x556343183f90 2024-09-16T01:36:25.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.275+0000 7ff4a57fa640 1 -- 172.21.15.88:0/1717040526 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7ff4bc061780 con 0x556343183f90 2024-09-16T01:36:25.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.275+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/1717040526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4b403eb20 con 0x556343183f90 2024-09-16T01:36:25.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.275+0000 7ff4a57fa640 1 -- 172.21.15.88:0/1717040526 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff4bc0619a0 con 0x556343183f90 2024-09-16T01:36:25.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.275+0000 7ff4cf3d1740 10 client.5688 fetch_fsmap learned FSMap version 28 2024-09-16T01:36:25.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.275+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/1717040526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) -- 0x5563431c2c50 con 0x556343183f90 2024-09-16T01:36:25.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.275+0000 7ff4a57fa640 1 -- 172.21.15.88:0/1717040526 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b403eb20 con 0x556343183f90 2024-09-16T01:36:25.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.275+0000 7ff4a57fa640 1 -- 172.21.15.88:0/1717040526 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 28) ==== 3494+0+0 (secure 0 0 0) 0x7ff4bc073070 con 0x556343183f90 2024-09-16T01:36:25.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.275+0000 7ff4cf3d1740 10 client.5688 fetch_fsmap finished waiting for FSMap version 28 2024-09-16T01:36:25.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.275+0000 7ff4cf3d1740 10 client.5688 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-16T01:36:25.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.275+0000 7ff4cf3d1740 20 client.5688 populate_metadata read hostname 'smithi088' 2024-09-16T01:36:25.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.276+0000 7ff4cf3d1740 1 --2- 172.21.15.88:0/1717040526 >> [v2:172.21.15.88:6834/2700280540,v1:172.21.15.88:6836/2700280540] conn(0x55634321c5e0 0x556343229040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.276+0000 7ff4cf3d1740 4 client.5688 mds_command: new command op to 5634 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:36:25.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.276+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/1717040526 --> [v2:172.21.15.88:6834/2700280540,v1:172.21.15.88:6836/2700280540] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5563430c5170 con 0x55634321c5e0 2024-09-16T01:36:25.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.276+0000 7ff4ca3de640 1 --2- 172.21.15.88:0/1717040526 >> [v2:172.21.15.88:6834/2700280540,v1:172.21.15.88:6836/2700280540] conn(0x55634321c5e0 0x556343229040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.276+0000 7ff4a57fa640 10 client.5688 ms_handle_connect on v2:172.21.15.88:6834/2700280540 2024-09-16T01:36:25.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.276+0000 7ff4ca3de640 1 --2- 172.21.15.88:0/1717040526 >> [v2:172.21.15.88:6834/2700280540,v1:172.21.15.88:6836/2700280540] conn(0x55634321c5e0 0x556343229040 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cb0e1914e8f07a4 server_cookie=ac1d87e3dd15b1a5 in_seq=0 out_seq=0 2024-09-16T01:36:25.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.276+0000 7ff4a57fa640 1 -- 172.21.15.88:0/1717040526 <== mds.0 v2:172.21.15.88:6834/2700280540 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5563430c5170 con 0x55634321c5e0 2024-09-16T01:36:25.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.276+0000 7ff4a57fa640 10 client.5688 handle_command_reply: tid=0 2024-09-16T01:36:25.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.276+0000 7ff4cf3d1740 1 client.5688 shutdown 2024-09-16T01:36:25.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.276+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/1717040526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556343183f90 msgr2=0x5563431b1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:25.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.276+0000 7ff4cf3d1740 1 --2- 172.21.15.88:0/1717040526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556343183f90 0x5563431b1ef0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc00a040 tx=0x7ff4bc05d520 comp rx=0 tx=0).stop 2024-09-16T01:36:25.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.276+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/1717040526 shutdown_connections 2024-09-16T01:36:25.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.276+0000 7ff4cf3d1740 1 --2- 172.21.15.88:0/1717040526 >> [v2:172.21.15.88:6834/2700280540,v1:172.21.15.88:6836/2700280540] conn(0x55634321c5e0 0x556343229040 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.276+0000 7ff4cf3d1740 1 --2- 172.21.15.88:0/1717040526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556343183f90 0x5563431b1ef0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.276+0000 7ff4a57fa640 0 client.5688 ms_handle_reset on v2:172.21.15.88:6834/2700280540 2024-09-16T01:36:25.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.276+0000 7ff4cf3d1740 1 --2- 172.21.15.88:0/1717040526 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55634315c0b0 0x5563431b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.276+0000 7ff4cf3d1740 1 --2- 172.21.15.88:0/1717040526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55634315a920 0x556343182d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.276+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/1717040526 >> 172.21.15.88:0/1717040526 conn(0x5563431580e0 msgr2=0x556343175ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:25.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.276+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/1717040526 shutdown_connections 2024-09-16T01:36:25.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.276+0000 7ff4cf3d1740 1 -- 172.21.15.88:0/1717040526 wait complete. 2024-09-16T01:36:25.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.277+0000 7ff4cf3d1740 20 client.5688 trim_cache size 0 max 16384 2024-09-16T01:36:25.295 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:36:25.296 INFO:teuthology.orchestra.run.smithi088.stdout: "message": "", 2024-09-16T01:36:25.296 INFO:teuthology.orchestra.run.smithi088.stdout: "return_code": 0 2024-09-16T01:36:25.296 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:36:25.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.277+0000 7ff4cc642640 1 -- 172.21.15.88:0/80816074 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff4a806c2e0 msgr2=0x7ff4a806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:25.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.277+0000 7ff4cc642640 1 --2- 172.21.15.88:0/80816074 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff4a806c2e0 0x7ff4a806e7a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff4c00012e0 tx=0x7ff4c0009040 comp rx=0 tx=0).stop 2024-09-16T01:36:25.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.277+0000 7ff4cc642640 1 -- 172.21.15.88:0/80816074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c415c2a0 msgr2=0x7ff4c40cf890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:25.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.277+0000 7ff4cc642640 1 --2- 172.21.15.88:0/80816074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c415c2a0 0x7ff4c40cf890 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc00c410 tx=0x7ff4bc00c8d0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.277+0000 7ff4cc642640 1 -- 172.21.15.88:0/80816074 shutdown_connections 2024-09-16T01:36:25.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.277+0000 7ff4cc642640 1 --2- 172.21.15.88:0/80816074 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff4a806c2e0 0x7ff4a806e7a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.277+0000 7ff4cc642640 1 --2- 172.21.15.88:0/80816074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c415c2a0 0x7ff4c40cf890 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.277+0000 7ff4cc642640 1 --2- 172.21.15.88:0/80816074 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4c406f710 0x7ff4c40d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.277+0000 7ff4cc642640 1 --2- 172.21.15.88:0/80816074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4c406bc10 0x7ff4c40cdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.277+0000 7ff4cc642640 1 -- 172.21.15.88:0/80816074 >> 172.21.15.88:0/80816074 conn(0x7ff4c40b3070 msgr2=0x7ff4c40af6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:25.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.277+0000 7ff4cc642640 1 -- 172.21.15.88:0/80816074 shutdown_connections 2024-09-16T01:36:25.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.277+0000 7ff4cc642640 1 -- 172.21.15.88:0/80816074 wait complete. 2024-09-16T01:36:25.299 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:36:25.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.396+0000 7f9bd69b7640 1 Processor -- start 2024-09-16T01:36:25.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.396+0000 7f9bd69b7640 1 -- start start 2024-09-16T01:36:25.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.397+0000 7f9bd69b7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9bd00b4920 0x7f9bd015b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.397+0000 7f9bd69b7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9bd00b4d00 0x7f9bd00b5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.397+0000 7f9bd69b7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9bd00cd780 0x7f9bd00cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.397+0000 7f9bd69b7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f9bd00cf100 con 0x7f9bd00b4d00 2024-09-16T01:36:25.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.397+0000 7f9bd69b7640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9bd00ce4f0 con 0x7f9bd00b4920 2024-09-16T01:36:25.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.397+0000 7f9bd69b7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9bd00ce7d0 con 0x7f9bd00cd780 2024-09-16T01:36:25.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.397+0000 7f9bcf7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9bd00b4d00 0x7f9bd00b5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.397+0000 7f9bcf7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9bd00b4d00 0x7f9bd00b5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:32960/0 (socket says 172.21.15.88:32960) 2024-09-16T01:36:25.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.397+0000 7f9bcf7fe640 1 -- 172.21.15.88:0/1343477331 learned_addr learned my addr 172.21.15.88:0/1343477331 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:25.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.397+0000 7f9bcf7fe640 1 -- 172.21.15.88:0/1343477331 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9bd00cd780 msgr2=0x7f9bd00cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:25.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.397+0000 7f9bcf7fe640 1 --2- 172.21.15.88:0/1343477331 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9bd00cd780 0x7f9bd00cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.397+0000 7f9bcf7fe640 1 -- 172.21.15.88:0/1343477331 >> v2:172.21.15.154:3300/0 conn(0x7f9bd00b4920 msgr2=0x7f9bd015b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:25.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.397+0000 7f9bcffff640 1 --2- 172.21.15.88:0/1343477331 >> v2:172.21.15.154:3300/0 conn(0x7f9bd00b4920 0x7f9bd015b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.397+0000 7f9bcf7fe640 1 --2- 172.21.15.88:0/1343477331 >> v2:172.21.15.154:3300/0 conn(0x7f9bd00b4920 0x7f9bd015b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.397+0000 7f9bcf7fe640 1 -- 172.21.15.88:0/1343477331 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd00ceab0 con 0x7f9bd00b4d00 2024-09-16T01:36:25.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.397+0000 7f9bcffff640 1 --2- 172.21.15.88:0/1343477331 >> v2:172.21.15.154:3300/0 conn(0x7f9bd00b4920 0x7f9bd015b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:25.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.398+0000 7f9bcf7fe640 1 --2- 172.21.15.88:0/1343477331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd00b4d00 0x7f9bd00b5170 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9bd006cb00 tx=0x7f9bc402c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27a24cb3822c7df server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:25.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.398+0000 7f9bceffd640 1 -- 172.21.15.88:0/1343477331 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4032790 con 0x7f9bd00b4d00 2024-09-16T01:36:25.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.398+0000 7f9bceffd640 1 -- 172.21.15.88:0/1343477331 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bc4038070 con 0x7f9bd00b4d00 2024-09-16T01:36:25.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.398+0000 7f9bceffd640 1 -- 172.21.15.88:0/1343477331 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4032b50 con 0x7f9bd00b4d00 2024-09-16T01:36:25.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.399+0000 7f9bd69b7640 1 -- 172.21.15.88:0/1343477331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd00b4d00 msgr2=0x7f9bd00b5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:25.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.399+0000 7f9bd69b7640 1 --2- 172.21.15.88:0/1343477331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd00b4d00 0x7f9bd00b5170 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9bd006cb00 tx=0x7f9bc402c890 comp rx=0 tx=0).stop 2024-09-16T01:36:25.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.399+0000 7f9bd69b7640 1 -- 172.21.15.88:0/1343477331 shutdown_connections 2024-09-16T01:36:25.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.399+0000 7f9bd69b7640 1 --2- 172.21.15.88:0/1343477331 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9bd00cd780 0x7f9bd00cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.399+0000 7f9bd69b7640 1 --2- 172.21.15.88:0/1343477331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd00b4d00 0x7f9bd00b5170 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.399+0000 7f9bd69b7640 1 --2- 172.21.15.88:0/1343477331 >> v2:172.21.15.154:3300/0 conn(0x7f9bd00b4920 0x7f9bd015b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.399+0000 7f9bd69b7640 1 -- 172.21.15.88:0/1343477331 >> 172.21.15.88:0/1343477331 conn(0x7f9bd006bc10 msgr2=0x7f9bd006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:25.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.399+0000 7f9bd69b7640 1 -- 172.21.15.88:0/1343477331 shutdown_connections 2024-09-16T01:36:25.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.399+0000 7f9bd69b7640 1 -- 172.21.15.88:0/1343477331 wait complete. 2024-09-16T01:36:25.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.400+0000 7f9bd69b7640 1 Processor -- start 2024-09-16T01:36:25.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.400+0000 7f9bd69b7640 1 -- start start 2024-09-16T01:36:25.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.401+0000 7f9bd69b7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9bd00b4920 0x7f9bd01410e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.401+0000 7f9bd69b7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd00b4d00 0x7f9bd00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.401+0000 7f9bd69b7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9bd00cd780 0x7f9bd00cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.401+0000 7f9bd69b7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd015bd10 con 0x7f9bd00b4d00 2024-09-16T01:36:25.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.401+0000 7f9bd69b7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd00b56b0 con 0x7f9bd00cd780 2024-09-16T01:36:25.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.401+0000 7f9bd69b7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9bd00ce110 con 0x7f9bd00b4920 2024-09-16T01:36:25.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.401+0000 7f9bcf7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd00b4d00 0x7f9bd00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.401+0000 7f9bcf7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd00b4d00 0x7f9bd00cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:32966/0 (socket says 172.21.15.88:32966) 2024-09-16T01:36:25.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.401+0000 7f9bcf7fe640 1 -- 172.21.15.88:0/3940787288 learned_addr learned my addr 172.21.15.88:0/3940787288 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:25.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.401+0000 7f9bd4f54640 1 --2- 172.21.15.88:0/3940787288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9bd00cd780 0x7f9bd00cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.401+0000 7f9bcffff640 1 --2- 172.21.15.88:0/3940787288 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9bd00b4920 0x7f9bd01410e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.401+0000 7f9bcf7fe640 1 -- 172.21.15.88:0/3940787288 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9bd00b4920 msgr2=0x7f9bd01410e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:25.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.401+0000 7f9bcf7fe640 1 --2- 172.21.15.88:0/3940787288 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9bd00b4920 0x7f9bd01410e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.401+0000 7f9bcf7fe640 1 -- 172.21.15.88:0/3940787288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9bd00cd780 msgr2=0x7f9bd00cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:25.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.401+0000 7f9bcf7fe640 1 --2- 172.21.15.88:0/3940787288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9bd00cd780 0x7f9bd00cfad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.401+0000 7f9bcf7fe640 1 -- 172.21.15.88:0/3940787288 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd01508d0 con 0x7f9bd00b4d00 2024-09-16T01:36:25.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.401+0000 7f9bcf7fe640 1 --2- 172.21.15.88:0/3940787288 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd00b4d00 0x7f9bd00cf590 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4000c00 tx=0x7f9bc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:25.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.402+0000 7f9bccff9640 1 -- 172.21.15.88:0/3940787288 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4033620 con 0x7f9bd00b4d00 2024-09-16T01:36:25.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.402+0000 7f9bccff9640 1 -- 172.21.15.88:0/3940787288 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bc40337e0 con 0x7f9bd00b4d00 2024-09-16T01:36:25.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.402+0000 7f9bd69b7640 1 -- 172.21.15.88:0/3940787288 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bd0158bd0 con 0x7f9bd00b4d00 2024-09-16T01:36:25.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.402+0000 7f9bccff9640 1 -- 172.21.15.88:0/3940787288 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4033bd0 con 0x7f9bd00b4d00 2024-09-16T01:36:25.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.402+0000 7f9bd69b7640 1 -- 172.21.15.88:0/3940787288 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bd0159190 con 0x7f9bd00b4d00 2024-09-16T01:36:25.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.403+0000 7f9bd69b7640 1 -- 172.21.15.88:0/3940787288 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bd00b5170 con 0x7f9bd00b4d00 2024-09-16T01:36:25.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.403+0000 7f9bccff9640 1 -- 172.21.15.88:0/3940787288 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9bc404d060 con 0x7f9bd00b4d00 2024-09-16T01:36:25.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.403+0000 7f9bccff9640 1 --2- 172.21.15.88:0/3940787288 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9bbc06c2b0 0x7f9bbc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.404+0000 7f9bccff9640 1 -- 172.21.15.88:0/3940787288 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f9bc40adbf0 con 0x7f9bd00b4d00 2024-09-16T01:36:25.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.404+0000 7f9bcffff640 1 --2- 172.21.15.88:0/3940787288 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9bbc06c2b0 0x7f9bbc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.404+0000 7f9bcffff640 1 --2- 172.21.15.88:0/3940787288 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9bbc06c2b0 0x7f9bbc06e770 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9bc000a990 tx=0x7f9bc0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:25.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.408+0000 7f9bccff9640 1 -- 172.21.15.88:0/3940787288 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9bc4041070 con 0x7f9bd00b4d00 2024-09-16T01:36:25.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.609+0000 7f9bd69b7640 1 -- 172.21.15.88:0/3940787288 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9bd0071710 con 0x7f9bd00b4d00 2024-09-16T01:36:25.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.609+0000 7f9bccff9640 1 -- 172.21.15.88:0/3940787288 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5220 (secure 0 0 0) 0x7f9bc407c390 con 0x7f9bd00b4d00 2024-09-16T01:36:25.611 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:25.611 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":28,"btime":"2024-09-16T01:36:24:800958+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4877,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1769276566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":1769276566},{"type":"v1","addr":"172.21.15.154:6835","nonce":1769276566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/725621378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":725621378},{"type":"v1","addr":"172.21.15.154:6834","nonce":725621378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4199111872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4199111872},{"type":"v1","addr":"172.21.15.88:6837","nonce":4199111872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:05.691540+0000","modified":"2024-09-16T01:36:24.800947+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"c","rank":0,"incarnation":28,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6836/2700280540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2700280540},{"type":"v1","addr":"172.21.15.88:6836","nonce":2700280540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-09-16T01:36:25.611 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 28 2024-09-16T01:36:25.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.611+0000 7f9bd69b7640 1 -- 172.21.15.88:0/3940787288 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9bbc06c2b0 msgr2=0x7f9bbc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:25.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.611+0000 7f9bd69b7640 1 --2- 172.21.15.88:0/3940787288 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9bbc06c2b0 0x7f9bbc06e770 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9bc000a990 tx=0x7f9bc0008040 comp rx=0 tx=0).stop 2024-09-16T01:36:25.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.611+0000 7f9bd69b7640 1 -- 172.21.15.88:0/3940787288 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd00b4d00 msgr2=0x7f9bd00cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:25.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.611+0000 7f9bd69b7640 1 --2- 172.21.15.88:0/3940787288 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd00b4d00 0x7f9bd00cf590 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4000c00 tx=0x7f9bc40046c0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.611+0000 7f9bd69b7640 1 -- 172.21.15.88:0/3940787288 shutdown_connections 2024-09-16T01:36:25.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.611+0000 7f9bd69b7640 1 --2- 172.21.15.88:0/3940787288 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9bbc06c2b0 0x7f9bbc06e770 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.611+0000 7f9bd69b7640 1 --2- 172.21.15.88:0/3940787288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9bd00cd780 0x7f9bd00cfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.611+0000 7f9bd69b7640 1 --2- 172.21.15.88:0/3940787288 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd00b4d00 0x7f9bd00cf590 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.611+0000 7f9bd69b7640 1 --2- 172.21.15.88:0/3940787288 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9bd00b4920 0x7f9bd01410e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.611+0000 7f9bd69b7640 1 -- 172.21.15.88:0/3940787288 >> 172.21.15.88:0/3940787288 conn(0x7f9bd006bc10 msgr2=0x7f9bd0070f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:25.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.612+0000 7f9bd69b7640 1 -- 172.21.15.88:0/3940787288 shutdown_connections 2024-09-16T01:36:25.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.612+0000 7f9bd69b7640 1 -- 172.21.15.88:0/3940787288 wait complete. 2024-09-16T01:36:25.624 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:36:25.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.721+0000 7f9ad8a5c640 1 Processor -- start 2024-09-16T01:36:25.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.721+0000 7f9ad8a5c640 1 -- start start 2024-09-16T01:36:25.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.722+0000 7f9ad8a5c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9ad40b5be0 0x7f9ad415ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.722+0000 7f9ad8a5c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ad40b6830 con 0x7f9ad40b29a0 2024-09-16T01:36:25.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.722+0000 7f9ad8a5c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ad40cd0b0 con 0x7f9ad415a9c0 2024-09-16T01:36:25.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.722+0000 7f9ad8a5c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f9ad415f410 con 0x7f9ad40b5be0 2024-09-16T01:36:25.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.722+0000 7f9ac9d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9ad40b5be0 0x7f9ad415ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.722+0000 7f9ac9d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9ad40b5be0 0x7f9ad415ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:32982/0 (socket says 172.21.15.88:32982) 2024-09-16T01:36:25.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.722+0000 7f9ad2d9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9ad40b29a0 0x7f9ad40cde70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42770/0 (socket says 172.21.15.88:42770) 2024-09-16T01:36:25.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.722+0000 7f9ac9d9b640 1 -- 172.21.15.88:0/3383823060 learned_addr learned my addr 172.21.15.88:0/3383823060 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:25.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.722+0000 7f9ac9d9b640 1 -- 172.21.15.88:0/3383823060 >> v1:172.21.15.154:6789/0 conn(0x7f9ad415a9c0 legacy=0x7f9ad415e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:25.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.722+0000 7f9ac9d9b640 1 -- 172.21.15.88:0/3383823060 >> v1:172.21.15.88:6789/0 conn(0x7f9ad40b29a0 legacy=0x7f9ad40cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:25.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.722+0000 7f9ac9d9b640 1 -- 172.21.15.88:0/3383823060 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ad40cd390 con 0x7f9ad40b5be0 2024-09-16T01:36:25.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.722+0000 7f9ad1d9b640 1 -- 172.21.15.88:0/3383823060 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 83493000 0 0) 0x7f9ad40b6830 con 0x7f9ad40b29a0 2024-09-16T01:36:25.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.723+0000 7f9ac9d9b640 1 --2- 172.21.15.88:0/3383823060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ad40b5be0 0x7f9ad415ed10 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9abc00aa10 tx=0x7f9abc02b6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fa720222bc15398 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:25.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.723+0000 7f9ad1d9b640 1 -- 172.21.15.88:0/3383823060 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9abc00b040 con 0x7f9ad40b5be0 2024-09-16T01:36:25.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.723+0000 7f9ad1d9b640 1 -- 172.21.15.88:0/3383823060 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9abc030710 con 0x7f9ad40b5be0 2024-09-16T01:36:25.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.723+0000 7f9ad1d9b640 1 -- 172.21.15.88:0/3383823060 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9abc030ae0 con 0x7f9ad40b5be0 2024-09-16T01:36:25.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.723+0000 7f9ad8a5c640 1 -- 172.21.15.88:0/3383823060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ad40b5be0 msgr2=0x7f9ad415ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:25.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.723+0000 7f9ad8a5c640 1 --2- 172.21.15.88:0/3383823060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ad40b5be0 0x7f9ad415ed10 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9abc00aa10 tx=0x7f9abc02b6d0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.723+0000 7f9ad8a5c640 1 -- 172.21.15.88:0/3383823060 shutdown_connections 2024-09-16T01:36:25.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.723+0000 7f9ad8a5c640 1 --2- 172.21.15.88:0/3383823060 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ad40b5be0 0x7f9ad415ed10 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.723+0000 7f9ad8a5c640 1 -- 172.21.15.88:0/3383823060 >> 172.21.15.88:0/3383823060 conn(0x7f9ad406bc10 msgr2=0x7f9ad406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:25.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.723+0000 7f9ad8a5c640 1 -- 172.21.15.88:0/3383823060 shutdown_connections 2024-09-16T01:36:25.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.723+0000 7f9ad8a5c640 1 -- 172.21.15.88:0/3383823060 wait complete. 2024-09-16T01:36:25.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.725+0000 7f9ad8a5c640 1 Processor -- start 2024-09-16T01:36:25.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.725+0000 7f9ad8a5c640 1 -- start start 2024-09-16T01:36:25.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.726+0000 7f9ad8a5c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ad40b29a0 0x7f9ad414ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.726+0000 7f9ad8a5c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ad40b5be0 0x7f9ad414eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.726+0000 7f9ad8a5c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ad415a9c0 0x7f9ad40cb710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.726+0000 7f9ad8a5c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad415ada0 con 0x7f9ad40b5be0 2024-09-16T01:36:25.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.726+0000 7f9ad8a5c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad40cd950 con 0x7f9ad415a9c0 2024-09-16T01:36:25.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.726+0000 7f9ad8a5c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9ad406de40 con 0x7f9ad40b29a0 2024-09-16T01:36:25.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.726+0000 7f9ac9d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ad40b5be0 0x7f9ad414eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.726+0000 7f9ac9d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ad40b5be0 0x7f9ad414eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:32988/0 (socket says 172.21.15.88:32988) 2024-09-16T01:36:25.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.726+0000 7f9ac9d9b640 1 -- 172.21.15.88:0/1030495724 learned_addr learned my addr 172.21.15.88:0/1030495724 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:25.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.726+0000 7f9ad259c640 1 --2- 172.21.15.88:0/1030495724 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ad40b29a0 0x7f9ad414ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.726+0000 7f9ad2d9d640 1 --2- 172.21.15.88:0/1030495724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ad415a9c0 0x7f9ad40cb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.726+0000 7f9ac9d9b640 1 -- 172.21.15.88:0/1030495724 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ad40b29a0 msgr2=0x7f9ad414ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:25.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.726+0000 7f9ac9d9b640 1 --2- 172.21.15.88:0/1030495724 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ad40b29a0 0x7f9ad414ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.726+0000 7f9ac9d9b640 1 -- 172.21.15.88:0/1030495724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ad415a9c0 msgr2=0x7f9ad40cb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:25.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.726+0000 7f9ac9d9b640 1 --2- 172.21.15.88:0/1030495724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ad415a9c0 0x7f9ad40cb710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.726+0000 7f9ac9d9b640 1 -- 172.21.15.88:0/1030495724 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ad414e330 con 0x7f9ad40b5be0 2024-09-16T01:36:25.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.726+0000 7f9ac9d9b640 1 --2- 172.21.15.88:0/1030495724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ad40b5be0 0x7f9ad414eb20 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9abc00aa10 tx=0x7f9abc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:25.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.726+0000 7f9acb7fe640 1 -- 172.21.15.88:0/1030495724 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9abc03f070 con 0x7f9ad40b5be0 2024-09-16T01:36:25.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.726+0000 7f9acb7fe640 1 -- 172.21.15.88:0/1030495724 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9abc030e10 con 0x7f9ad40b5be0 2024-09-16T01:36:25.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.726+0000 7f9ad8a5c640 1 -- 172.21.15.88:0/1030495724 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ad41f4aa0 con 0x7f9ad40b5be0 2024-09-16T01:36:25.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.726+0000 7f9acb7fe640 1 -- 172.21.15.88:0/1030495724 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9abc004030 con 0x7f9ad40b5be0 2024-09-16T01:36:25.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.727+0000 7f9ad8a5c640 1 -- 172.21.15.88:0/1030495724 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ad41f5060 con 0x7f9ad40b5be0 2024-09-16T01:36:25.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.727+0000 7f9ad8a5c640 1 -- 172.21.15.88:0/1030495724 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ad406e210 con 0x7f9ad40b5be0 2024-09-16T01:36:25.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.728+0000 7f9acb7fe640 1 -- 172.21.15.88:0/1030495724 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9abc042070 con 0x7f9ad40b5be0 2024-09-16T01:36:25.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.728+0000 7f9acb7fe640 1 --2- 172.21.15.88:0/1030495724 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9ab406c2b0 0x7f9ab406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:25.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.729+0000 7f9ad259c640 1 --2- 172.21.15.88:0/1030495724 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9ab406c2b0 0x7f9ab406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:25.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.729+0000 7f9acb7fe640 1 -- 172.21.15.88:0/1030495724 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f9abc0abe10 con 0x7f9ad40b5be0 2024-09-16T01:36:25.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.729+0000 7f9ad259c640 1 --2- 172.21.15.88:0/1030495724 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9ab406c2b0 0x7f9ab406e770 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4000f60 tx=0x7f9ac4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:25.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.733+0000 7f9acb7fe640 1 -- 172.21.15.88:0/1030495724 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9abc079ee0 con 0x7f9ad40b5be0 2024-09-16T01:36:25.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.936+0000 7f9ad8a5c640 1 -- 172.21.15.88:0/1030495724 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ad4076200 con 0x7f9ad40b5be0 2024-09-16T01:36:25.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.937+0000 7f9acb7fe640 1 -- 172.21.15.88:0/1030495724 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+5223 (secure 0 0 0) 0x7f9abc07e1e0 con 0x7f9ad40b5be0 2024-09-16T01:36:25.938 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:25.939 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"btime":"2024-09-16T01:36:25:802538+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4877,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1769276566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":1769276566},{"type":"v1","addr":"172.21.15.154:6835","nonce":1769276566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/725621378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":725621378},{"type":"v1","addr":"172.21.15.154:6834","nonce":725621378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4199111872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4199111872},{"type":"v1","addr":"172.21.15.88:6837","nonce":4199111872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:05.691540+0000","modified":"2024-09-16T01:36:24.810188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"c","rank":0,"incarnation":28,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.88:6836/2700280540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2700280540},{"type":"v1","addr":"172.21.15.88:6836","nonce":2700280540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-09-16T01:36:25.939 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 29 2024-09-16T01:36:25.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.939+0000 7f9ad8a5c640 1 -- 172.21.15.88:0/1030495724 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9ab406c2b0 msgr2=0x7f9ab406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:25.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.939+0000 7f9ad8a5c640 1 --2- 172.21.15.88:0/1030495724 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9ab406c2b0 0x7f9ab406e770 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4000f60 tx=0x7f9ac4009040 comp rx=0 tx=0).stop 2024-09-16T01:36:25.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.939+0000 7f9ad8a5c640 1 -- 172.21.15.88:0/1030495724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ad40b5be0 msgr2=0x7f9ad414eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:25.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.939+0000 7f9ad8a5c640 1 --2- 172.21.15.88:0/1030495724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ad40b5be0 0x7f9ad414eb20 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9abc00aa10 tx=0x7f9abc0046c0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.939+0000 7f9ad8a5c640 1 -- 172.21.15.88:0/1030495724 shutdown_connections 2024-09-16T01:36:25.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.939+0000 7f9ad8a5c640 1 --2- 172.21.15.88:0/1030495724 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9ab406c2b0 0x7f9ab406e770 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.939+0000 7f9ad8a5c640 1 --2- 172.21.15.88:0/1030495724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ad415a9c0 0x7f9ad40cb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.939+0000 7f9ad8a5c640 1 --2- 172.21.15.88:0/1030495724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ad40b5be0 0x7f9ad414eb20 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.939+0000 7f9ad8a5c640 1 --2- 172.21.15.88:0/1030495724 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ad40b29a0 0x7f9ad414ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:25.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.939+0000 7f9ad8a5c640 1 -- 172.21.15.88:0/1030495724 >> 172.21.15.88:0/1030495724 conn(0x7f9ad406bc10 msgr2=0x7f9ad40cc400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:25.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.939+0000 7f9ad8a5c640 1 -- 172.21.15.88:0/1030495724 shutdown_connections 2024-09-16T01:36:25.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:25.939+0000 7f9ad8a5c640 1 -- 172.21.15.88:0/1030495724 wait complete. 2024-09-16T01:36:25.951 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:36:26.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.048+0000 7f586b549640 1 Processor -- start 2024-09-16T01:36:26.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.048+0000 7f586b549640 1 -- start start 2024-09-16T01:36:26.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.049+0000 7f586b549640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58640b4920 0x7f586415b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:26.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.049+0000 7f586b549640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f58640cd780 0x7f58640cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:26.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.049+0000 7f586b549640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58640ce4f0 con 0x7f58640b4d00 2024-09-16T01:36:26.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.049+0000 7f586b549640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f58640cf100 con 0x7f58640cd780 2024-09-16T01:36:26.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.049+0000 7f586b549640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f58640ce7d0 con 0x7f58640b4920 2024-09-16T01:36:26.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.049+0000 7f5869ae6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f58640cd780 0x7f58640cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:26.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.049+0000 7f5869ae6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f58640cd780 0x7f58640cdbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:32998/0 (socket says 172.21.15.88:32998) 2024-09-16T01:36:26.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.049+0000 7f5869ae6640 1 -- 172.21.15.88:0/359189599 learned_addr learned my addr 172.21.15.88:0/359189599 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:26.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.049+0000 7f58692e5640 1 --2- 172.21.15.88:0/359189599 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58640b4920 0x7f586415b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:26.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.050+0000 7f5869ae6640 1 -- 172.21.15.88:0/359189599 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58640b4920 msgr2=0x7f586415b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:26.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.050+0000 7f5869ae6640 1 --2- 172.21.15.88:0/359189599 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58640b4920 0x7f586415b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:26.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.050+0000 7f5869ae6640 1 -- 172.21.15.88:0/359189599 >> v1:172.21.15.154:6789/0 conn(0x7f58640b4d00 legacy=0x7f58640b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:26.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.050+0000 7f5869ae6640 1 -- 172.21.15.88:0/359189599 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58640ceab0 con 0x7f58640cd780 2024-09-16T01:36:26.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.050+0000 7f5869ae6640 1 --2- 172.21.15.88:0/359189599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58640cd780 0x7f58640cdbf0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f585c009e50 tx=0x7f585c02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54f8c24b5b562a15 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:26.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.050+0000 7f585bfff640 1 -- 172.21.15.88:0/359189599 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585c036070 con 0x7f58640cd780 2024-09-16T01:36:26.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.050+0000 7f585bfff640 1 -- 172.21.15.88:0/359189599 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f585c031600 con 0x7f58640cd780 2024-09-16T01:36:26.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.050+0000 7f585bfff640 1 -- 172.21.15.88:0/359189599 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585c0319d0 con 0x7f58640cd780 2024-09-16T01:36:26.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.051+0000 7f586b549640 1 -- 172.21.15.88:0/359189599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58640cd780 msgr2=0x7f58640cdbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:26.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.051+0000 7f586b549640 1 --2- 172.21.15.88:0/359189599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58640cd780 0x7f58640cdbf0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f585c009e50 tx=0x7f585c02a710 comp rx=0 tx=0).stop 2024-09-16T01:36:26.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.051+0000 7f586b549640 1 -- 172.21.15.88:0/359189599 shutdown_connections 2024-09-16T01:36:26.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.051+0000 7f586b549640 1 --2- 172.21.15.88:0/359189599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58640cd780 0x7f58640cdbf0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:26.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.051+0000 7f586b549640 1 --2- 172.21.15.88:0/359189599 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58640b4920 0x7f586415b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:26.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.051+0000 7f586b549640 1 -- 172.21.15.88:0/359189599 >> 172.21.15.88:0/359189599 conn(0x7f586406bc10 msgr2=0x7f586406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:26.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.051+0000 7f586b549640 1 -- 172.21.15.88:0/359189599 shutdown_connections 2024-09-16T01:36:26.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.051+0000 7f586b549640 1 -- 172.21.15.88:0/359189599 wait complete. 2024-09-16T01:36:26.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.052+0000 7f586b549640 1 Processor -- start 2024-09-16T01:36:26.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.052+0000 7f586b549640 1 -- start start 2024-09-16T01:36:26.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.053+0000 7f586b549640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58640b4920 0x7f5864142f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:26.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.053+0000 7f586b549640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58640b4d00 0x7f5864145f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:26.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.053+0000 7f586b549640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58640cd780 0x7f58641f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:26.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.053+0000 7f586b549640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f58640ce130 con 0x7f58640cd780 2024-09-16T01:36:26.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.053+0000 7f586b549640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f586415bd10 con 0x7f58640b4d00 2024-09-16T01:36:26.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.053+0000 7f586b549640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f58640ce2b0 con 0x7f58640b4920 2024-09-16T01:36:26.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.053+0000 7f5869ae6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58640cd780 0x7f58641f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:26.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.053+0000 7f5869ae6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58640cd780 0x7f58641f43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33000/0 (socket says 172.21.15.88:33000) 2024-09-16T01:36:26.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.053+0000 7f5869ae6640 1 -- 172.21.15.88:0/2513176793 learned_addr learned my addr 172.21.15.88:0/2513176793 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:26.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.053+0000 7f5868ae4640 1 --2- 172.21.15.88:0/2513176793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58640b4d00 0x7f5864145f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:26.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.053+0000 7f58692e5640 1 --2- 172.21.15.88:0/2513176793 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58640b4920 0x7f5864142f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:26.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.053+0000 7f5869ae6640 1 -- 172.21.15.88:0/2513176793 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58640b4920 msgr2=0x7f5864142f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:26.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.053+0000 7f5869ae6640 1 --2- 172.21.15.88:0/2513176793 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58640b4920 0x7f5864142f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:26.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.053+0000 7f5869ae6640 1 -- 172.21.15.88:0/2513176793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58640b4d00 msgr2=0x7f5864145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:26.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.053+0000 7f5869ae6640 1 --2- 172.21.15.88:0/2513176793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58640b4d00 0x7f5864145f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:26.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.053+0000 7f5869ae6640 1 -- 172.21.15.88:0/2513176793 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58641f5370 con 0x7f58640cd780 2024-09-16T01:36:26.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.053+0000 7f58692e5640 1 --2- 172.21.15.88:0/2513176793 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58640b4920 0x7f5864142f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:26.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.053+0000 7f5868ae4640 1 --2- 172.21.15.88:0/2513176793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58640b4d00 0x7f5864145f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:26.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.054+0000 7f5869ae6640 1 --2- 172.21.15.88:0/2513176793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58640cd780 0x7f58641f43b0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f585c009f70 tx=0x7f585c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:26.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.054+0000 7f5859ffb640 1 -- 172.21.15.88:0/2513176793 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585c03f020 con 0x7f58640cd780 2024-09-16T01:36:26.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.054+0000 7f5859ffb640 1 -- 172.21.15.88:0/2513176793 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f585c037040 con 0x7f58640cd780 2024-09-16T01:36:26.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.054+0000 7f586b549640 1 -- 172.21.15.88:0/2513176793 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58641f4db0 con 0x7f58640cd780 2024-09-16T01:36:26.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.054+0000 7f5859ffb640 1 -- 172.21.15.88:0/2513176793 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585c036070 con 0x7f58640cd780 2024-09-16T01:36:26.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.054+0000 7f586b549640 1 -- 172.21.15.88:0/2513176793 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5864145730 con 0x7f58640cd780 2024-09-16T01:36:26.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.055+0000 7f586b549640 1 -- 172.21.15.88:0/2513176793 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f586415b7d0 con 0x7f58640cd780 2024-09-16T01:36:26.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.055+0000 7f5859ffb640 1 -- 172.21.15.88:0/2513176793 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f585c042020 con 0x7f58640cd780 2024-09-16T01:36:26.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.056+0000 7f5859ffb640 1 --2- 172.21.15.88:0/2513176793 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f584806c2b0 0x7f584806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:26.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.056+0000 7f5859ffb640 1 -- 172.21.15.88:0/2513176793 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f585c0ab860 con 0x7f58640cd780 2024-09-16T01:36:26.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.056+0000 7f58692e5640 1 --2- 172.21.15.88:0/2513176793 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f584806c2b0 0x7f584806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:26.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.056+0000 7f58692e5640 1 --2- 172.21.15.88:0/2513176793 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f584806c2b0 0x7f584806e770 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5860006fd0 tx=0x7f5860008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:26.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.060+0000 7f5859ffb640 1 -- 172.21.15.88:0/2513176793 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f585c049030 con 0x7f58640cd780 2024-09-16T01:36:26.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.260+0000 7f586b549640 1 -- 172.21.15.88:0/2513176793 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f58640b5610 con 0x7f58640cd780 2024-09-16T01:36:26.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.261+0000 7f5859ffb640 1 -- 172.21.15.88:0/2513176793 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+5223 (secure 0 0 0) 0x7f585c079f70 con 0x7f58640cd780 2024-09-16T01:36:26.262 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:26.262 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":29,"btime":"2024-09-16T01:36:25:802538+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4877,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1769276566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":1769276566},{"type":"v1","addr":"172.21.15.154:6835","nonce":1769276566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/725621378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":725621378},{"type":"v1","addr":"172.21.15.154:6834","nonce":725621378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4199111872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4199111872},{"type":"v1","addr":"172.21.15.88:6837","nonce":4199111872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:05.691540+0000","modified":"2024-09-16T01:36:24.810188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"c","rank":0,"incarnation":28,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.88:6836/2700280540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2700280540},{"type":"v1","addr":"172.21.15.88:6836","nonce":2700280540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-09-16T01:36:26.262 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 29 2024-09-16T01:36:26.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.263+0000 7f586b549640 1 -- 172.21.15.88:0/2513176793 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f584806c2b0 msgr2=0x7f584806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:26.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.263+0000 7f586b549640 1 --2- 172.21.15.88:0/2513176793 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f584806c2b0 0x7f584806e770 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5860006fd0 tx=0x7f5860008040 comp rx=0 tx=0).stop 2024-09-16T01:36:26.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.263+0000 7f586b549640 1 -- 172.21.15.88:0/2513176793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58640cd780 msgr2=0x7f58641f43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:26.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.263+0000 7f586b549640 1 --2- 172.21.15.88:0/2513176793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58640cd780 0x7f58641f43b0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f585c009f70 tx=0x7f585c004a40 comp rx=0 tx=0).stop 2024-09-16T01:36:26.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.263+0000 7f586b549640 1 -- 172.21.15.88:0/2513176793 shutdown_connections 2024-09-16T01:36:26.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.263+0000 7f586b549640 1 --2- 172.21.15.88:0/2513176793 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f584806c2b0 0x7f584806e770 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:26.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.263+0000 7f586b549640 1 --2- 172.21.15.88:0/2513176793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f58640cd780 0x7f58641f43b0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:26.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.263+0000 7f586b549640 1 --2- 172.21.15.88:0/2513176793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58640b4d00 0x7f5864145f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:26.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.263+0000 7f586b549640 1 --2- 172.21.15.88:0/2513176793 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58640b4920 0x7f5864142f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:26.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.263+0000 7f586b549640 1 -- 172.21.15.88:0/2513176793 >> 172.21.15.88:0/2513176793 conn(0x7f586406bc10 msgr2=0x7f58640cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:26.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.263+0000 7f586b549640 1 -- 172.21.15.88:0/2513176793 shutdown_connections 2024-09-16T01:36:26.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.263+0000 7f586b549640 1 -- 172.21.15.88:0/2513176793 wait complete. 2024-09-16T01:36:26.275 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json dump tree /a 0 2024-09-16T01:36:26.388 INFO:teuthology.orchestra.run.smithi088.stdout:[{"ino":1099511627776,"rdev":0,"ctime":"2024-09-16T01:36:22.220837+0000","btime":"2024-09-16T01:36:16.352953+0000","mode":16877,"uid":1000,"gid":1333,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":0,"stripe_count":0,"object_size":0,"pool_id":-1,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2024-09-16T01:36:22.220837+0000","atime":"2024-09-16T01:36:16.352953+0000","time_warp_seq":0,"change_attr":3,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"quiesce_block":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"2024-09-16T01:36:22.220837+0000","num_files":0,"num_subdirs":1,"change_attr":3},"rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":2,"rsnaps":0,"rctime":"2024-09-16T01:36:22.220837+0000"},"accounted_rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":2,"rsnaps":0,"rctime":"2024-09-16T01:36:22.220837+0000"},"version":9,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[{"key":"security.selinux","val":"unconfined_u:object_r:unlabeled_t:s0\u0000"}],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"dirfrag":1,"dirty":1},"nref":2,"versionlock":{"gather_set":[],"state":"lock","type":"iversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"authlock":{},"linklock":{},"dirfragtreelock":{},"filelock":{},"xattrlock":{},"snaplock":{},"nestlock":{},"flocklock":{},"policylock":{},"quiescelock":{"gather_set":[],"state":"lock","type":"iquiesce","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"],"client_caps":[],"loner":-1,"want_loner":-1,"mds_caps_wanted":[],"dirfrags":[{"path":"/a","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"","states":["auth"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1},"nref":1,"dentries":[{"path":"a/c","alternate_name":"","path_ino":1,"snap_first":18446744073709551606,"snap_last":18446744073709551614,"is_primary":true,"is_remote":false,"is_null":false,"is_new":false,"inode":1099511627777,"remote_type":"","version":4,"projected_version":4,"auth_pins":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"inodepin":1,"dirty":1},"nref":2,"lock":{},"versionlock":{"gather_set":[],"state":"lock","type":"dversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"]},{"path":"a/b","alternate_name":"","path_ino":1,"snap_first":2,"snap_last":18446744073709551614,"is_primary":false,"is_remote":false,"is_null":true,"is_new":false,"inode":0,"remote_type":"","version":5,"projected_version":5,"auth_pins":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"dirty":1},"nref":1,"lock":{},"versionlock":{"gather_set":[],"state":"lock","type":"dversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"]}]}],"is_subvolume":false}] 2024-09-16T01:36:26.389 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "accounted_rstat": { "rbytes": 0, "rctime": "2024-09-16T01:36:22.220837+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 2, "version": 0 }, "atime": "2024-09-16T01:36:16.352953+0000", "auth_pins": 0, "auth_state": { "replicas": {} }, "authlock": {}, "backtrace_version": 2, "btime": "2024-09-16T01:36:16.352953+0000", "change_attr": 3, "client_caps": [], "client_ranges": [], "ctime": "2024-09-16T01:36:22.220837+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfrags": [ { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dentries": [ { "alternate_name": "", "auth_pins": 0, "auth_state": { "replicas": {} }, "inode": 1099511627777, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_new": false, "is_null": false, "is_primary": true, "is_remote": false, "lock": {}, "nref": 2, "path": "a/c", "path_ino": 1, "pins": { "dirty": 1, "inodepin": 1 }, "projected_version": 4, "remote_type": "", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snap_first": 18446744073709551606, "snap_last": 18446744073709551614, "states": [ "auth", "dirty" ], "version": 4, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "dversion", "xlock_by": {} } }, { "alternate_name": "", "auth_pins": 0, "auth_state": { "replicas": {} }, "inode": 0, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_new": false, "is_null": true, "is_primary": false, "is_remote": false, "lock": {}, "nref": 1, "path": "a/b", "path_ino": 1, "pins": { "dirty": 1 }, "projected_version": 5, "remote_type": "", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snap_first": 2, "snap_last": 18446744073709551614, "states": [ "auth", "dirty" ], "version": 5, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "dversion", "xlock_by": {} } } ], "dir_auth": "", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/a", "pins": { "child": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth" ], "version": "7" } ], "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 3, "mtime": "2024-09-16T01:36:22.220837+0000", "num_files": 0, "num_subdirs": 1, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": {}, "flocklock": {}, "gid": 1333, "ino": 1099511627776, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_subvolume": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 0, "pool_id": -1, "pool_ns": "", "stripe_count": 0, "stripe_unit": 0 }, "linklock": {}, "loner": -1, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16877, "mtime": "2024-09-16T01:36:22.220837+0000", "nestlock": {}, "nlink": 1, "nref": 2, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "pins": { "dirfrag": 1, "dirty": 1 }, "policylock": {}, "quiesce_block": false, "quiescelock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "iquiesce", "xlock_by": {} }, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 0, "rctime": "2024-09-16T01:36:22.220837+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 2, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth", "dirty" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 1000, "version": 9, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "iversion", "xlock_by": {} }, "want_loner": -1, "xattr_version": 1, "xattrlock": {}, "xattrs": [ { "key": "security.selinux", "val": "unconfined_u:object_r:unlabeled_t:s0\u0000" } ] } ] 2024-09-16T01:36:26.389 DEBUG:tasks.cephfs.test_damage:[{'ino': 1099511627776, 'rdev': 0, 'ctime': '2024-09-16T01:36:22.220837+0000', 'btime': '2024-09-16T01:36:16.352953+0000', 'mode': 16877, 'uid': 1000, 'gid': 1333, 'nlink': 1, 'dir_layout': {'dir_hash': 2, 'unused1': 0, 'unused2': 0, 'unused3': 0}, 'layout': {'stripe_unit': 0, 'stripe_count': 0, 'object_size': 0, 'pool_id': -1, 'pool_ns': ''}, 'old_pools': [], 'size': 0, 'truncate_seq': 1, 'truncate_size': 18446744073709551615, 'truncate_from': 0, 'truncate_pending': 0, 'mtime': '2024-09-16T01:36:22.220837+0000', 'atime': '2024-09-16T01:36:16.352953+0000', 'time_warp_seq': 0, 'change_attr': 3, 'export_pin': -1, 'export_ephemeral_random_pin': 0, 'export_ephemeral_distributed_pin': False, 'quiesce_block': False, 'client_ranges': [], 'dirstat': {'version': 0, 'mtime': '2024-09-16T01:36:22.220837+0000', 'num_files': 0, 'num_subdirs': 1, 'change_attr': 3}, 'rstat': {'version': 0, 'rbytes': 0, 'rfiles': 0, 'rsubdirs': 2, 'rsnaps': 0, 'rctime': '2024-09-16T01:36:22.220837+0000'}, 'accounted_rstat': {'version': 0, 'rbytes': 0, 'rfiles': 0, 'rsubdirs': 2, 'rsnaps': 0, 'rctime': '2024-09-16T01:36:22.220837+0000'}, 'version': 9, 'file_data_version': 0, 'xattr_version': 1, 'backtrace_version': 2, 'stray_prior_path': '', 'max_size_ever': 0, 'quota': {'max_bytes': 0, 'max_files': 0}, 'last_scrub_stamp': '0.000000', 'last_scrub_version': 0, 'symlink': '', 'xattrs': [{'key': 'security.selinux', 'val': 'unconfined_u:object_r:unlabeled_t:s0\x00'}], 'dirfragtree': {'splits': []}, 'old_inodes': [], 'oldest_snap': 18446744073709551614, 'damage_flags': 0, 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'dirfrag': 1, 'dirty': 1}, 'nref': 2, 'versionlock': {'gather_set': [], 'state': 'lock', 'type': 'iversion', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'authlock': {}, 'linklock': {}, 'dirfragtreelock': {}, 'filelock': {}, 'xattrlock': {}, 'snaplock': {}, 'nestlock': {}, 'flocklock': {}, 'policylock': {}, 'quiescelock': {'gather_set': [], 'state': 'lock', 'type': 'iquiesce', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'states': ['auth', 'dirty'], 'client_caps': [], 'loner': -1, 'want_loner': -1, 'mds_caps_wanted': [], 'dirfrags': [{'path': '/a', 'dirfrag': '0x10000000000', 'snapid_first': 2, 'projected_version': '7', 'version': '7', 'committing_version': '0', 'committed_version': '0', 'is_rep': False, 'dir_auth': '', 'states': ['auth'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1}, 'nref': 1, 'dentries': [{'path': 'a/c', 'alternate_name': '', 'path_ino': 1, 'snap_first': 18446744073709551606, 'snap_last': 18446744073709551614, 'is_primary': True, 'is_remote': False, 'is_null': False, 'is_new': False, 'inode': 1099511627777, 'remote_type': '', 'version': 4, 'projected_version': 4, 'auth_pins': 0, 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'is_frozen': False, 'is_freezing': False, 'pins': {'inodepin': 1, 'dirty': 1}, 'nref': 2, 'lock': {}, 'versionlock': {'gather_set': [], 'state': 'lock', 'type': 'dversion', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'states': ['auth', 'dirty']}, {'path': 'a/b', 'alternate_name': '', 'path_ino': 1, 'snap_first': 2, 'snap_last': 18446744073709551614, 'is_primary': False, 'is_remote': False, 'is_null': True, 'is_new': False, 'inode': 0, 'remote_type': '', 'version': 5, 'projected_version': 5, 'auth_pins': 0, 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'is_frozen': False, 'is_freezing': False, 'pins': {'dirty': 1}, 'nref': 1, 'lock': {}, 'versionlock': {'gather_set': [], 'state': 'lock', 'type': 'dversion', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'states': ['auth', 'dirty']}]}], 'is_subvolume': False}] 2024-09-16T01:36:26.389 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T01:36:26.389 DEBUG:teuthology.orchestra.run.smithi088:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T01:36:26.433 DEBUG:teuthology.orchestra.run.smithi154:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T01:36:26.487 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T01:36:26.487 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T01:36:26.487 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:36:26.487 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:36:26.503 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:36:26.503 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T01:36:26.503 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:36:26.503 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:36:26.521 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056026,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1347 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T01:36:26.522 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T01:36:26.522 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:36:26.522 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:36:26.559 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:36:26.559 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T01:36:26.559 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:36:26.559 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:36:26.581 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056026,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1347 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T01:36:26.581 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:36:26.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.696+0000 7f0f13931640 1 Processor -- start 2024-09-16T01:36:26.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.696+0000 7f0f13931640 1 -- start start 2024-09-16T01:36:26.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.697+0000 7f0f13931640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f0c0cfc80 0x7f0f0c0b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:26.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.697+0000 7f0f13931640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f0c15c780 con 0x7f0f0c15c330 2024-09-16T01:36:26.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.697+0000 7f0f13931640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f0c0b50a0 con 0x7f0f0c155800 2024-09-16T01:36:26.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.697+0000 7f0f13931640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0f0c0b5380 con 0x7f0f0c0cfc80 2024-09-16T01:36:26.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.697+0000 7f0f11ece640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0f0c15c330 0x7f0f0c15c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42778/0 (socket says 172.21.15.88:42778) 2024-09-16T01:36:26.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.697+0000 7f0f11ece640 1 -- 172.21.15.88:0/328038474 learned_addr learned my addr 172.21.15.88:0/328038474 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:26.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.697+0000 7f0f03fff640 1 -- 172.21.15.88:0/328038474 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4282403066 0 0) 0x7f0f0c15c780 con 0x7f0f0c15c330 2024-09-16T01:36:26.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.697+0000 7f0f03fff640 1 -- 172.21.15.88:0/328038474 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ef8003520 con 0x7f0f0c15c330 2024-09-16T01:36:26.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.697+0000 7f0f10ecc640 1 --2- 172.21.15.88:0/328038474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f0c0cfc80 0x7f0f0c0b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:26.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.697+0000 7f0f03fff640 1 -- 172.21.15.88:0/328038474 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1583653150 0 0) 0x7f0ef8003520 con 0x7f0f0c15c330 2024-09-16T01:36:26.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.697+0000 7f0f03fff640 1 -- 172.21.15.88:0/328038474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f0c0cfc80 msgr2=0x7f0f0c0b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:26.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.697+0000 7f0f03fff640 1 --2- 172.21.15.88:0/328038474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f0c0cfc80 0x7f0f0c0b6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:26.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.697+0000 7f0f03fff640 1 -- 172.21.15.88:0/328038474 >> v1:172.21.15.154:6789/0 conn(0x7f0f0c155800 legacy=0x7f0f0c0cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:26.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.698+0000 7f0f03fff640 1 -- 172.21.15.88:0/328038474 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f0c0b5660 con 0x7f0f0c15c330 2024-09-16T01:36:26.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.698+0000 7f0f03fff640 1 -- 172.21.15.88:0/328038474 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f0f040019f0 con 0x7f0f0c15c330 2024-09-16T01:36:26.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.698+0000 7f0f03fff640 1 -- 172.21.15.88:0/328038474 >> v1:172.21.15.88:6789/0 conn(0x7f0f0c15c330 legacy=0x7f0f0c15c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:26.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.698+0000 7f0f03fff640 1 --2- 172.21.15.88:0/328038474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ef80039e0 0x7f0ef8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:26.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.698+0000 7f0f03fff640 1 --2- 172.21.15.88:0/328038474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f0c0cfc80 0x7f0ef8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:26.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.698+0000 7f0f03fff640 1 --2- 172.21.15.88:0/328038474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ef8004d40 0x7f0ef80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:26.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.698+0000 7f0f03fff640 1 -- 172.21.15.88:0/328038474 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c0b5380 con 0x7f0ef8004d40 2024-09-16T01:36:26.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.698+0000 7f0f03fff640 1 -- 172.21.15.88:0/328038474 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef800bb00 con 0x7f0ef80039e0 2024-09-16T01:36:26.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.698+0000 7f0f03fff640 1 -- 172.21.15.88:0/328038474 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0ef800bd00 con 0x7f0f0c0cfc80 2024-09-16T01:36:26.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.698+0000 7f0f11ece640 1 --2- 172.21.15.88:0/328038474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ef8004d40 0x7f0ef80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:26.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.698+0000 7f0f116cd640 1 --2- 172.21.15.88:0/328038474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ef80039e0 0x7f0ef8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:26.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.698+0000 7f0f10ecc640 1 --2- 172.21.15.88:0/328038474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f0c0cfc80 0x7f0ef8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:26.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.698+0000 7f0f11ece640 1 -- 172.21.15.88:0/328038474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f0c0cfc80 msgr2=0x7f0ef8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:26.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.698+0000 7f0f11ece640 1 --2- 172.21.15.88:0/328038474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f0c0cfc80 0x7f0ef8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:26.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.698+0000 7f0f11ece640 1 -- 172.21.15.88:0/328038474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ef80039e0 msgr2=0x7f0ef8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:26.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.698+0000 7f0f11ece640 1 --2- 172.21.15.88:0/328038474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ef80039e0 0x7f0ef8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:26.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.698+0000 7f0f11ece640 1 -- 172.21.15.88:0/328038474 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef8003520 con 0x7f0ef8004d40 2024-09-16T01:36:26.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.698+0000 7f0f10ecc640 1 --2- 172.21.15.88:0/328038474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f0c0cfc80 0x7f0ef8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:26.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.698+0000 7f0f116cd640 1 --2- 172.21.15.88:0/328038474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ef80039e0 0x7f0ef8003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:26.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.699+0000 7f0f11ece640 1 --2- 172.21.15.88:0/328038474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ef8004d40 0x7f0ef80051c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0f0400a770 tx=0x7f0ef8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87d3eff0370d823e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:26.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.699+0000 7f0f03fff640 1 -- 172.21.15.88:0/328038474 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0400a390 con 0x7f0ef8004d40 2024-09-16T01:36:26.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.699+0000 7f0f03fff640 1 -- 172.21.15.88:0/328038474 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f0402f070 con 0x7f0ef8004d40 2024-09-16T01:36:26.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.699+0000 7f0f03fff640 1 -- 172.21.15.88:0/328038474 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0402bbe0 con 0x7f0ef8004d40 2024-09-16T01:36:26.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.699+0000 7f0f13931640 1 -- 172.21.15.88:0/328038474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ef8004d40 msgr2=0x7f0ef80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:26.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.699+0000 7f0f13931640 1 --2- 172.21.15.88:0/328038474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ef8004d40 0x7f0ef80051c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0f0400a770 tx=0x7f0ef8005830 comp rx=0 tx=0).stop 2024-09-16T01:36:26.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.699+0000 7f0f11ece640 1 -- 172.21.15.88:0/328038474 reap_dead start 2024-09-16T01:36:26.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.699+0000 7f0f13931640 1 -- 172.21.15.88:0/328038474 shutdown_connections 2024-09-16T01:36:26.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.699+0000 7f0f13931640 1 -- 172.21.15.88:0/328038474 >> 172.21.15.88:0/328038474 conn(0x7f0f0c06bc10 msgr2=0x7f0f0c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:26.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.699+0000 7f0f13931640 1 -- 172.21.15.88:0/328038474 shutdown_connections 2024-09-16T01:36:26.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.699+0000 7f0f13931640 1 -- 172.21.15.88:0/328038474 wait complete. 2024-09-16T01:36:26.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.700+0000 7f0f13931640 1 Processor -- start 2024-09-16T01:36:26.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.701+0000 7f0f13931640 1 -- start start 2024-09-16T01:36:26.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.701+0000 7f0f13931640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ef8004d40 0x7f0f0c1498a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:26.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.701+0000 7f0f13931640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f0c148d30 0x7f0f0c14e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:26.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.701+0000 7f0f13931640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f0c15d760 0x7f0f0c15dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:26.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.701+0000 7f0f13931640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c155be0 con 0x7f0f0c148d30 2024-09-16T01:36:26.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.701+0000 7f0f13931640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c0b5c20 con 0x7f0f0c15d760 2024-09-16T01:36:26.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.701+0000 7f0f13931640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0f0c0b6140 con 0x7f0ef8004d40 2024-09-16T01:36:26.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.701+0000 7f0f10ecc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f0c148d30 0x7f0f0c14e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:26.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.701+0000 7f0f10ecc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f0c148d30 0x7f0f0c14e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33010/0 (socket says 172.21.15.88:33010) 2024-09-16T01:36:26.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.701+0000 7f0f10ecc640 1 -- 172.21.15.88:0/2755401511 learned_addr learned my addr 172.21.15.88:0/2755401511 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:26.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.702+0000 7f0f10ecc640 1 -- 172.21.15.88:0/2755401511 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ef8004d40 msgr2=0x7f0f0c1498a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:26.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.702+0000 7f0f10ecc640 1 --2- 172.21.15.88:0/2755401511 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ef8004d40 0x7f0f0c1498a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:26.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.702+0000 7f0f10ecc640 1 -- 172.21.15.88:0/2755401511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f0c15d760 msgr2=0x7f0f0c15dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:26.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.702+0000 7f0f10ecc640 1 --2- 172.21.15.88:0/2755401511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f0c15d760 0x7f0f0c15dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:26.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.702+0000 7f0f10ecc640 1 -- 172.21.15.88:0/2755401511 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f0c14ec90 con 0x7f0f0c148d30 2024-09-16T01:36:26.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.702+0000 7f0f10ecc640 1 --2- 172.21.15.88:0/2755401511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f0c148d30 0x7f0f0c14e750 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0efc0116d0 tx=0x7f0efc011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:26.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.702+0000 7f0f01ffb640 1 -- 172.21.15.88:0/2755401511 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0efc00f940 con 0x7f0f0c148d30 2024-09-16T01:36:26.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.702+0000 7f0f01ffb640 1 -- 172.21.15.88:0/2755401511 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0efc00fae0 con 0x7f0f0c148d30 2024-09-16T01:36:26.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.702+0000 7f0f13931640 1 -- 172.21.15.88:0/2755401511 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f0c157420 con 0x7f0f0c148d30 2024-09-16T01:36:26.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.702+0000 7f0f01ffb640 1 -- 172.21.15.88:0/2755401511 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0efc015460 con 0x7f0f0c148d30 2024-09-16T01:36:26.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.702+0000 7f0f13931640 1 -- 172.21.15.88:0/2755401511 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f0c1579e0 con 0x7f0f0c148d30 2024-09-16T01:36:26.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.703+0000 7f0f13931640 1 -- 172.21.15.88:0/2755401511 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f0c0cf570 con 0x7f0f0c148d30 2024-09-16T01:36:26.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.704+0000 7f0f01ffb640 1 -- 172.21.15.88:0/2755401511 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0efc015600 con 0x7f0f0c148d30 2024-09-16T01:36:26.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.704+0000 7f0f01ffb640 1 --2- 172.21.15.88:0/2755401511 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0ef006c220 0x7f0ef006e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:26.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.704+0000 7f0f116cd640 1 --2- 172.21.15.88:0/2755401511 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0ef006c220 0x7f0ef006e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:26.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.704+0000 7f0f01ffb640 1 -- 172.21.15.88:0/2755401511 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f0efc08bdb0 con 0x7f0f0c148d30 2024-09-16T01:36:26.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.705+0000 7f0f116cd640 1 --2- 172.21.15.88:0/2755401511 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0ef006c220 0x7f0ef006e6e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0f08007fd0 tx=0x7f0f08009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:26.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.709+0000 7f0f01ffb640 1 -- 172.21.15.88:0/2755401511 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0efc019030 con 0x7f0f0c148d30 2024-09-16T01:36:26.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.908+0000 7f0f13931640 1 -- 172.21.15.88:0/2755401511 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f0c0cd2b0 con 0x7f0f0c148d30 2024-09-16T01:36:26.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.908+0000 7f0f01ffb640 1 -- 172.21.15.88:0/2755401511 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+5220 (secure 0 0 0) 0x7f0efc05a4c0 con 0x7f0f0c148d30 2024-09-16T01:36:26.910 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:26.910 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":30,"btime":"2024-09-16T01:36:26:803638+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4877,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1769276566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":1769276566},{"type":"v1","addr":"172.21.15.154:6835","nonce":1769276566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/725621378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":725621378},{"type":"v1","addr":"172.21.15.154:6834","nonce":725621378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4199111872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4199111872},{"type":"v1","addr":"172.21.15.88:6837","nonce":4199111872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:05.691540+0000","modified":"2024-09-16T01:36:25.805504+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"c","rank":0,"incarnation":28,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.88:6836/2700280540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2700280540},{"type":"v1","addr":"172.21.15.88:6836","nonce":2700280540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-09-16T01:36:26.910 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 30 2024-09-16T01:36:26.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.910+0000 7f0f13931640 1 -- 172.21.15.88:0/2755401511 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0ef006c220 msgr2=0x7f0ef006e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:26.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.910+0000 7f0f13931640 1 --2- 172.21.15.88:0/2755401511 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0ef006c220 0x7f0ef006e6e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0f08007fd0 tx=0x7f0f08009040 comp rx=0 tx=0).stop 2024-09-16T01:36:26.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.910+0000 7f0f13931640 1 -- 172.21.15.88:0/2755401511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f0c148d30 msgr2=0x7f0f0c14e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:26.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.910+0000 7f0f13931640 1 --2- 172.21.15.88:0/2755401511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f0c148d30 0x7f0f0c14e750 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0efc0116d0 tx=0x7f0efc011b90 comp rx=0 tx=0).stop 2024-09-16T01:36:26.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.910+0000 7f0f13931640 1 -- 172.21.15.88:0/2755401511 shutdown_connections 2024-09-16T01:36:26.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.910+0000 7f0f13931640 1 --2- 172.21.15.88:0/2755401511 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0ef006c220 0x7f0ef006e6e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:26.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.910+0000 7f0f13931640 1 --2- 172.21.15.88:0/2755401511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f0c15d760 0x7f0f0c15dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:26.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.910+0000 7f0f13931640 1 --2- 172.21.15.88:0/2755401511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f0c148d30 0x7f0f0c14e750 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:26.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.910+0000 7f0f13931640 1 --2- 172.21.15.88:0/2755401511 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ef8004d40 0x7f0f0c1498a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:26.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.910+0000 7f0f13931640 1 -- 172.21.15.88:0/2755401511 >> 172.21.15.88:0/2755401511 conn(0x7f0f0c06bc10 msgr2=0x7f0f0c0b3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:26.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.910+0000 7f0f13931640 1 -- 172.21.15.88:0/2755401511 shutdown_connections 2024-09-16T01:36:26.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:26.910+0000 7f0f13931640 1 -- 172.21.15.88:0/2755401511 wait complete. 2024-09-16T01:36:26.923 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:36:27.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.019+0000 7fed24896640 1 Processor -- start 2024-09-16T01:36:27.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.019+0000 7fed24896640 1 -- start start 2024-09-16T01:36:27.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.020+0000 7fed24896640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed2015c670 0x7fed20073460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:27.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.020+0000 7fed24896640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fed200b1990 0x7fed200d10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:27.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.020+0000 7fed24896640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed20071750 con 0x7fed20073080 2024-09-16T01:36:27.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.020+0000 7fed24896640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fed200d0c60 con 0x7fed200b1990 2024-09-16T01:36:27.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.020+0000 7fed24896640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fed200d0370 con 0x7fed2015c670 2024-09-16T01:36:27.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.021+0000 7fed1ed9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fed200b1990 0x7fed200d10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:27.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.021+0000 7fed1e59c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fed20073080 0x7fed200b5460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42788/0 (socket says 172.21.15.88:42788) 2024-09-16T01:36:27.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.021+0000 7fed1ed9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fed200b1990 0x7fed200d10f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33020/0 (socket says 172.21.15.88:33020) 2024-09-16T01:36:27.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.021+0000 7fed1e59c640 1 -- 172.21.15.88:0/2843042874 learned_addr learned my addr 172.21.15.88:0/2843042874 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:27.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.021+0000 7fed1ed9d640 1 -- 172.21.15.88:0/2843042874 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed2015c670 msgr2=0x7fed20073460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:27.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.021+0000 7fed1dd9b640 1 --2- 172.21.15.88:0/2843042874 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed2015c670 0x7fed20073460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:27.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.021+0000 7fed1d59a640 1 -- 172.21.15.88:0/2843042874 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1130773739 0 0) 0x7fed20071750 con 0x7fed20073080 2024-09-16T01:36:27.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.021+0000 7fed1ed9d640 1 --2- 172.21.15.88:0/2843042874 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed2015c670 0x7fed20073460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.021+0000 7fed1ed9d640 1 -- 172.21.15.88:0/2843042874 >> v1:172.21.15.88:6789/0 conn(0x7fed20073080 legacy=0x7fed200b5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:27.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.021+0000 7fed1ed9d640 1 -- 172.21.15.88:0/2843042874 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed200739a0 con 0x7fed200b1990 2024-09-16T01:36:27.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.022+0000 7fed1ed9d640 1 --2- 172.21.15.88:0/2843042874 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed200b1990 0x7fed200d10f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fed10009d90 tx=0x7fed1002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba6b52734e39bc4c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:27.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.022+0000 7fed1d59a640 1 -- 172.21.15.88:0/2843042874 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed10007740 con 0x7fed200b1990 2024-09-16T01:36:27.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.022+0000 7fed1d59a640 1 -- 172.21.15.88:0/2843042874 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed100078e0 con 0x7fed200b1990 2024-09-16T01:36:27.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.022+0000 7fed1d59a640 1 -- 172.21.15.88:0/2843042874 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed10007cb0 con 0x7fed200b1990 2024-09-16T01:36:27.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.022+0000 7fed24896640 1 -- 172.21.15.88:0/2843042874 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed200b1990 msgr2=0x7fed200d10f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:27.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.022+0000 7fed24896640 1 --2- 172.21.15.88:0/2843042874 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed200b1990 0x7fed200d10f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fed10009d90 tx=0x7fed1002a660 comp rx=0 tx=0).stop 2024-09-16T01:36:27.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.022+0000 7fed24896640 1 -- 172.21.15.88:0/2843042874 shutdown_connections 2024-09-16T01:36:27.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.022+0000 7fed24896640 1 --2- 172.21.15.88:0/2843042874 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed200b1990 0x7fed200d10f0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.022+0000 7fed24896640 1 --2- 172.21.15.88:0/2843042874 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed2015c670 0x7fed20073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.022+0000 7fed24896640 1 -- 172.21.15.88:0/2843042874 >> 172.21.15.88:0/2843042874 conn(0x7fed200d24d0 msgr2=0x7fed200d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:27.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.022+0000 7fed24896640 1 -- 172.21.15.88:0/2843042874 shutdown_connections 2024-09-16T01:36:27.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.022+0000 7fed24896640 1 -- 172.21.15.88:0/2843042874 wait complete. 2024-09-16T01:36:27.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.024+0000 7fed24896640 1 Processor -- start 2024-09-16T01:36:27.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.024+0000 7fed24896640 1 -- start start 2024-09-16T01:36:27.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.025+0000 7fed24896640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed20073080 0x7fed2006d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:27.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.025+0000 7fed24896640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed200b1990 0x7fed201511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:27.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.025+0000 7fed24896640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2015c670 0x7fed200afa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:27.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.025+0000 7fed24896640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fed200d0610 con 0x7fed2015c670 2024-09-16T01:36:27.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.025+0000 7fed24896640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fed20071470 con 0x7fed200b1990 2024-09-16T01:36:27.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.025+0000 7fed24896640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fed200d08f0 con 0x7fed20073080 2024-09-16T01:36:27.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.025+0000 7fed1ed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2015c670 0x7fed200afa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:27.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.025+0000 7fed1ed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2015c670 0x7fed200afa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33030/0 (socket says 172.21.15.88:33030) 2024-09-16T01:36:27.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.025+0000 7fed1ed9d640 1 -- 172.21.15.88:0/621549039 learned_addr learned my addr 172.21.15.88:0/621549039 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:27.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.025+0000 7fed1e59c640 1 --2- 172.21.15.88:0/621549039 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed20073080 0x7fed2006d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:27.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.025+0000 7fed1dd9b640 1 --2- 172.21.15.88:0/621549039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed200b1990 0x7fed201511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:27.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.025+0000 7fed1ed9d640 1 -- 172.21.15.88:0/621549039 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed20073080 msgr2=0x7fed2006d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:27.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.025+0000 7fed1ed9d640 1 --2- 172.21.15.88:0/621549039 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed20073080 0x7fed2006d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.025+0000 7fed1ed9d640 1 -- 172.21.15.88:0/621549039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed200b1990 msgr2=0x7fed201511e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:27.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.025+0000 7fed1ed9d640 1 --2- 172.21.15.88:0/621549039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed200b1990 0x7fed201511e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.025+0000 7fed1ed9d640 1 -- 172.21.15.88:0/621549039 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed2006df00 con 0x7fed2015c670 2024-09-16T01:36:27.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.025+0000 7fed1e59c640 1 --2- 172.21.15.88:0/621549039 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed20073080 0x7fed2006d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:27.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.025+0000 7fed1dd9b640 1 --2- 172.21.15.88:0/621549039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed200b1990 0x7fed201511e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:27.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.025+0000 7fed1ed9d640 1 --2- 172.21.15.88:0/621549039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2015c670 0x7fed200afa50 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fed10009d90 tx=0x7fed10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:27.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.025+0000 7fed16ffd640 1 -- 172.21.15.88:0/621549039 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed10040070 con 0x7fed2015c670 2024-09-16T01:36:27.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.025+0000 7fed16ffd640 1 -- 172.21.15.88:0/621549039 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed10031c40 con 0x7fed2015c670 2024-09-16T01:36:27.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.025+0000 7fed24896640 1 -- 172.21.15.88:0/621549039 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed2006e1e0 con 0x7fed2015c670 2024-09-16T01:36:27.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.025+0000 7fed16ffd640 1 -- 172.21.15.88:0/621549039 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed10003990 con 0x7fed2015c670 2024-09-16T01:36:27.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.025+0000 7fed24896640 1 -- 172.21.15.88:0/621549039 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed2006e7a0 con 0x7fed2015c670 2024-09-16T01:36:27.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.026+0000 7fed24896640 1 -- 172.21.15.88:0/621549039 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed200d1f00 con 0x7fed2015c670 2024-09-16T01:36:27.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.027+0000 7fed16ffd640 1 -- 172.21.15.88:0/621549039 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fed10039040 con 0x7fed2015c670 2024-09-16T01:36:27.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.027+0000 7fed16ffd640 1 --2- 172.21.15.88:0/621549039 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fed0006c2b0 0x7fed0006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:27.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.027+0000 7fed16ffd640 1 -- 172.21.15.88:0/621549039 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7fed100ac800 con 0x7fed2015c670 2024-09-16T01:36:27.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.027+0000 7fed1e59c640 1 --2- 172.21.15.88:0/621549039 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fed0006c2b0 0x7fed0006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:27.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.028+0000 7fed1e59c640 1 --2- 172.21.15.88:0/621549039 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fed0006c2b0 0x7fed0006e770 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fed18000f60 tx=0x7fed18009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:27.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.032+0000 7fed16ffd640 1 -- 172.21.15.88:0/621549039 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fed1007af10 con 0x7fed2015c670 2024-09-16T01:36:27.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.233+0000 7fed24896640 1 -- 172.21.15.88:0/621549039 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed200d3050 con 0x7fed2015c670 2024-09-16T01:36:27.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.233+0000 7fed16ffd640 1 -- 172.21.15.88:0/621549039 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+5220 (secure 0 0 0) 0x7fed1007a810 con 0x7fed2015c670 2024-09-16T01:36:27.235 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:27.235 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":30,"btime":"2024-09-16T01:36:26:803638+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4877,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1769276566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":1769276566},{"type":"v1","addr":"172.21.15.154:6835","nonce":1769276566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/725621378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":725621378},{"type":"v1","addr":"172.21.15.154:6834","nonce":725621378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4199111872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4199111872},{"type":"v1","addr":"172.21.15.88:6837","nonce":4199111872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:05.691540+0000","modified":"2024-09-16T01:36:25.805504+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"c","rank":0,"incarnation":28,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.88:6836/2700280540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2700280540},{"type":"v1","addr":"172.21.15.88:6836","nonce":2700280540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-09-16T01:36:27.235 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 30 2024-09-16T01:36:27.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.235+0000 7fed24896640 1 -- 172.21.15.88:0/621549039 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fed0006c2b0 msgr2=0x7fed0006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:27.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.235+0000 7fed24896640 1 --2- 172.21.15.88:0/621549039 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fed0006c2b0 0x7fed0006e770 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fed18000f60 tx=0x7fed18009040 comp rx=0 tx=0).stop 2024-09-16T01:36:27.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.235+0000 7fed24896640 1 -- 172.21.15.88:0/621549039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2015c670 msgr2=0x7fed200afa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:27.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.235+0000 7fed24896640 1 --2- 172.21.15.88:0/621549039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2015c670 0x7fed200afa50 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fed10009d90 tx=0x7fed10004a40 comp rx=0 tx=0).stop 2024-09-16T01:36:27.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.235+0000 7fed24896640 1 -- 172.21.15.88:0/621549039 shutdown_connections 2024-09-16T01:36:27.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.235+0000 7fed24896640 1 --2- 172.21.15.88:0/621549039 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fed0006c2b0 0x7fed0006e770 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.235+0000 7fed24896640 1 --2- 172.21.15.88:0/621549039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2015c670 0x7fed200afa50 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.235+0000 7fed24896640 1 --2- 172.21.15.88:0/621549039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed200b1990 0x7fed201511e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.235+0000 7fed24896640 1 --2- 172.21.15.88:0/621549039 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed20073080 0x7fed2006d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.235+0000 7fed24896640 1 -- 172.21.15.88:0/621549039 >> 172.21.15.88:0/621549039 conn(0x7fed200d24d0 msgr2=0x7fed200d10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:27.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.235+0000 7fed24896640 1 -- 172.21.15.88:0/621549039 shutdown_connections 2024-09-16T01:36:27.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.235+0000 7fed24896640 1 -- 172.21.15.88:0/621549039 wait complete. 2024-09-16T01:36:27.247 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:36:27.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.344+0000 7f521de77640 1 Processor -- start 2024-09-16T01:36:27.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.344+0000 7f521de77640 1 -- start start 2024-09-16T01:36:27.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.345+0000 7f521de77640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f52180cfc80 0x7f52180b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:27.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.345+0000 7f521de77640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f521815c330 0x7f52180b4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:27.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.345+0000 7f521de77640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f521815c740 con 0x7f5218155800 2024-09-16T01:36:27.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.345+0000 7f521de77640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f52180b4e60 con 0x7f52180cfc80 2024-09-16T01:36:27.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.345+0000 7f521de77640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f52180b50a0 con 0x7f521815c330 2024-09-16T01:36:27.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.345+0000 7f5216ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f52180cfc80 0x7f52180b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:27.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.345+0000 7f52177fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5218155800 0x7f52180cf570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42790/0 (socket says 172.21.15.88:42790) 2024-09-16T01:36:27.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.345+0000 7f52177fe640 1 -- 172.21.15.88:0/1406971562 learned_addr learned my addr 172.21.15.88:0/1406971562 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:27.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.345+0000 7f5217fff640 1 --2- 172.21.15.88:0/1406971562 >> v2:172.21.15.154:3300/0 conn(0x7f521815c330 0x7f52180b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:27.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.345+0000 7f5216ffd640 1 -- 172.21.15.88:0/1406971562 >> v2:172.21.15.154:3300/0 conn(0x7f521815c330 msgr2=0x7f52180b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:27.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.345+0000 7f5216ffd640 1 --2- 172.21.15.88:0/1406971562 >> v2:172.21.15.154:3300/0 conn(0x7f521815c330 0x7f52180b4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.345+0000 7f5216ffd640 1 -- 172.21.15.88:0/1406971562 >> v1:172.21.15.88:6789/0 conn(0x7f5218155800 legacy=0x7f52180cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:27.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.345+0000 7f5216ffd640 1 -- 172.21.15.88:0/1406971562 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52180b5380 con 0x7f52180cfc80 2024-09-16T01:36:27.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.345+0000 7f52167fc640 1 -- 172.21.15.88:0/1406971562 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3223253325 0 0) 0x7f521815c740 con 0x7f5218155800 2024-09-16T01:36:27.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.346+0000 7f5216ffd640 1 --2- 172.21.15.88:0/1406971562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52180cfc80 0x7f52180b6140 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f520800ba40 tx=0x7f520802c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee1fccbc9af96a61 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:27.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.346+0000 7f52167fc640 1 -- 172.21.15.88:0/1406971562 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5208032770 con 0x7f52180cfc80 2024-09-16T01:36:27.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.346+0000 7f52167fc640 1 -- 172.21.15.88:0/1406971562 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5208038070 con 0x7f52180cfc80 2024-09-16T01:36:27.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.346+0000 7f52167fc640 1 -- 172.21.15.88:0/1406971562 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5208032b30 con 0x7f52180cfc80 2024-09-16T01:36:27.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.346+0000 7f521de77640 1 -- 172.21.15.88:0/1406971562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52180cfc80 msgr2=0x7f52180b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:27.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.346+0000 7f521de77640 1 --2- 172.21.15.88:0/1406971562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52180cfc80 0x7f52180b6140 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f520800ba40 tx=0x7f520802c8b0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.347+0000 7f521de77640 1 -- 172.21.15.88:0/1406971562 shutdown_connections 2024-09-16T01:36:27.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.347+0000 7f521de77640 1 --2- 172.21.15.88:0/1406971562 >> v2:172.21.15.154:3300/0 conn(0x7f521815c330 0x7f52180b4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.347+0000 7f521de77640 1 --2- 172.21.15.88:0/1406971562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52180cfc80 0x7f52180b6140 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.347+0000 7f521de77640 1 -- 172.21.15.88:0/1406971562 >> 172.21.15.88:0/1406971562 conn(0x7f521806bc10 msgr2=0x7f521806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:27.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.347+0000 7f521de77640 1 -- 172.21.15.88:0/1406971562 shutdown_connections 2024-09-16T01:36:27.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.347+0000 7f521de77640 1 -- 172.21.15.88:0/1406971562 wait complete. 2024-09-16T01:36:27.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.348+0000 7f521de77640 1 Processor -- start 2024-09-16T01:36:27.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.348+0000 7f521de77640 1 -- start start 2024-09-16T01:36:27.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.349+0000 7f521de77640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52180cfc80 0x7f521814ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:27.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.349+0000 7f521de77640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5218155800 0x7f5218071cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:27.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.349+0000 7f521de77640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f521815c330 0x7f5218154710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:27.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.349+0000 7f521de77640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f52180b5c20 con 0x7f521815c330 2024-09-16T01:36:27.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.349+0000 7f521de77640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f52180b5940 con 0x7f52180cfc80 2024-09-16T01:36:27.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.349+0000 7f521de77640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5218155be0 con 0x7f5218155800 2024-09-16T01:36:27.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.349+0000 7f5217fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f521815c330 0x7f5218154710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:27.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.349+0000 7f5217fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f521815c330 0x7f5218154710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33050/0 (socket says 172.21.15.88:33050) 2024-09-16T01:36:27.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.349+0000 7f5217fff640 1 -- 172.21.15.88:0/297834706 learned_addr learned my addr 172.21.15.88:0/297834706 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:27.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.349+0000 7f52177fe640 1 --2- 172.21.15.88:0/297834706 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52180cfc80 0x7f521814ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:27.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.349+0000 7f5216ffd640 1 --2- 172.21.15.88:0/297834706 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5218155800 0x7f5218071cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:27.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.349+0000 7f5217fff640 1 -- 172.21.15.88:0/297834706 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5218155800 msgr2=0x7f5218071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:27.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.349+0000 7f5217fff640 1 --2- 172.21.15.88:0/297834706 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5218155800 0x7f5218071cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.349+0000 7f5217fff640 1 -- 172.21.15.88:0/297834706 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52180cfc80 msgr2=0x7f521814ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:27.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.349+0000 7f5217fff640 1 --2- 172.21.15.88:0/297834706 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52180cfc80 0x7f521814ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.349+0000 7f5217fff640 1 -- 172.21.15.88:0/297834706 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f521814f870 con 0x7f521815c330 2024-09-16T01:36:27.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.349+0000 7f5217fff640 1 --2- 172.21.15.88:0/297834706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f521815c330 0x7f5218154710 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f5210007cf0 tx=0x7f521000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:27.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.349+0000 7f520ffff640 1 -- 172.21.15.88:0/297834706 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5210014420 con 0x7f521815c330 2024-09-16T01:36:27.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.349+0000 7f520ffff640 1 -- 172.21.15.88:0/297834706 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5210019070 con 0x7f521815c330 2024-09-16T01:36:27.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.349+0000 7f520ffff640 1 -- 172.21.15.88:0/297834706 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5210014820 con 0x7f521815c330 2024-09-16T01:36:27.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.349+0000 7f521de77640 1 -- 172.21.15.88:0/297834706 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5218158260 con 0x7f521815c330 2024-09-16T01:36:27.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.350+0000 7f521de77640 1 -- 172.21.15.88:0/297834706 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5218158820 con 0x7f521815c330 2024-09-16T01:36:27.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.350+0000 7f521de77640 1 -- 172.21.15.88:0/297834706 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52180b6140 con 0x7f521815c330 2024-09-16T01:36:27.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.351+0000 7f520ffff640 1 -- 172.21.15.88:0/297834706 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f52100149c0 con 0x7f521815c330 2024-09-16T01:36:27.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.351+0000 7f520ffff640 1 --2- 172.21.15.88:0/297834706 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f51f806c2b0 0x7f51f806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:27.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.351+0000 7f520ffff640 1 -- 172.21.15.88:0/297834706 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f521008ba30 con 0x7f521815c330 2024-09-16T01:36:27.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.351+0000 7f52177fe640 1 --2- 172.21.15.88:0/297834706 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f51f806c2b0 0x7f51f806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:27.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.352+0000 7f52177fe640 1 --2- 172.21.15.88:0/297834706 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f51f806c2b0 0x7f51f806e770 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5200000f60 tx=0x7f5200007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:27.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.356+0000 7f520ffff640 1 -- 172.21.15.88:0/297834706 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5210059b80 con 0x7f521815c330 2024-09-16T01:36:27.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.523+0000 7f521de77640 1 -- 172.21.15.88:0/297834706 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f52181544f0 con 0x7f521815c330 2024-09-16T01:36:27.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.524+0000 7f520ffff640 1 -- 172.21.15.88:0/297834706 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) ==== 74+0+16778 (secure 0 0 0) 0x7f521005de80 con 0x7f521815c330 2024-09-16T01:36:27.526 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:27.526 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":46,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:36:23.795684+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:36:04.035328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-16T01:36:04.663046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:6836/1610810484":"2024-09-17T01:36:23.795661+0000","172.21.15.88:6834/1610810484":"2024-09-17T01:36:23.795661+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:36:27.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.526+0000 7f521de77640 1 -- 172.21.15.88:0/297834706 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f51f806c2b0 msgr2=0x7f51f806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:27.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.527+0000 7f521de77640 1 --2- 172.21.15.88:0/297834706 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f51f806c2b0 0x7f51f806e770 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5200000f60 tx=0x7f5200007d50 comp rx=0 tx=0).stop 2024-09-16T01:36:27.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.527+0000 7f521de77640 1 -- 172.21.15.88:0/297834706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f521815c330 msgr2=0x7f5218154710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:27.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.527+0000 7f521de77640 1 --2- 172.21.15.88:0/297834706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f521815c330 0x7f5218154710 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f5210007cf0 tx=0x7f521000e4b0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.527+0000 7f521de77640 1 -- 172.21.15.88:0/297834706 shutdown_connections 2024-09-16T01:36:27.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.527+0000 7f521de77640 1 --2- 172.21.15.88:0/297834706 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f51f806c2b0 0x7f51f806e770 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.527+0000 7f521de77640 1 --2- 172.21.15.88:0/297834706 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f521815c330 0x7f5218154710 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.527+0000 7f521de77640 1 --2- 172.21.15.88:0/297834706 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5218155800 0x7f5218071cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.527+0000 7f521de77640 1 --2- 172.21.15.88:0/297834706 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52180cfc80 0x7f521814ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.527+0000 7f521de77640 1 -- 172.21.15.88:0/297834706 >> 172.21.15.88:0/297834706 conn(0x7f521806bc10 msgr2=0x7f52180ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:27.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.527+0000 7f521de77640 1 -- 172.21.15.88:0/297834706 shutdown_connections 2024-09-16T01:36:27.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.527+0000 7f521de77640 1 -- 172.21.15.88:0/297834706 wait complete. 2024-09-16T01:36:27.539 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-16T01:36:27.539 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:36:27.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.636+0000 7fd868ca5640 1 Processor -- start 2024-09-16T01:36:27.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.636+0000 7fd868ca5640 1 -- start start 2024-09-16T01:36:27.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.637+0000 7fd868ca5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd86415c170 0x7fd86415c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:27.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.637+0000 7fd868ca5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8640b64e0 0x7fd8640cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:27.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.637+0000 7fd868ca5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8640b4cf0 con 0x7fd8640cf6c0 2024-09-16T01:36:27.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.637+0000 7fd868ca5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd86415ca90 con 0x7fd86415c170 2024-09-16T01:36:27.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.637+0000 7fd868ca5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd8640b49c0 con 0x7fd8640b64e0 2024-09-16T01:36:27.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.637+0000 7fd86259c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd86415c170 0x7fd86415c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:27.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.637+0000 7fd86259c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd86415c170 0x7fd86415c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33060/0 (socket says 172.21.15.88:33060) 2024-09-16T01:36:27.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.637+0000 7fd86259c640 1 -- 172.21.15.88:0/164822766 learned_addr learned my addr 172.21.15.88:0/164822766 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:27.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.637+0000 7fd86259c640 1 -- 172.21.15.88:0/164822766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8640b64e0 msgr2=0x7fd8640cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:27.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.637+0000 7fd86259c640 1 --2- 172.21.15.88:0/164822766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8640b64e0 0x7fd8640cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.637+0000 7fd86259c640 1 -- 172.21.15.88:0/164822766 >> v1:172.21.15.154:6789/0 conn(0x7fd8640cf6c0 legacy=0x7fd8640b5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:27.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.637+0000 7fd86259c640 1 -- 172.21.15.88:0/164822766 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8640b4fd0 con 0x7fd86415c170 2024-09-16T01:36:27.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.638+0000 7fd86259c640 1 --2- 172.21.15.88:0/164822766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd86415c170 0x7fd86415c550 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fd8540098a0 tx=0x7fd85402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81b308e11b4e049f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:27.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.638+0000 7fd86159a640 1 -- 172.21.15.88:0/164822766 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd854037070 con 0x7fd86415c170 2024-09-16T01:36:27.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.638+0000 7fd86159a640 1 -- 172.21.15.88:0/164822766 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd854032440 con 0x7fd86415c170 2024-09-16T01:36:27.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.638+0000 7fd86159a640 1 -- 172.21.15.88:0/164822766 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd854032810 con 0x7fd86415c170 2024-09-16T01:36:27.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.638+0000 7fd868ca5640 1 -- 172.21.15.88:0/164822766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd86415c170 msgr2=0x7fd86415c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:27.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.638+0000 7fd868ca5640 1 --2- 172.21.15.88:0/164822766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd86415c170 0x7fd86415c550 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fd8540098a0 tx=0x7fd85402a250 comp rx=0 tx=0).stop 2024-09-16T01:36:27.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.638+0000 7fd868ca5640 1 -- 172.21.15.88:0/164822766 shutdown_connections 2024-09-16T01:36:27.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.638+0000 7fd868ca5640 1 --2- 172.21.15.88:0/164822766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8640b64e0 0x7fd8640cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.638+0000 7fd868ca5640 1 --2- 172.21.15.88:0/164822766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd86415c170 0x7fd86415c550 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.638+0000 7fd868ca5640 1 -- 172.21.15.88:0/164822766 >> 172.21.15.88:0/164822766 conn(0x7fd86406bc10 msgr2=0x7fd86406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:27.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.639+0000 7fd868ca5640 1 -- 172.21.15.88:0/164822766 shutdown_connections 2024-09-16T01:36:27.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.639+0000 7fd868ca5640 1 -- 172.21.15.88:0/164822766 wait complete. 2024-09-16T01:36:27.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.640+0000 7fd868ca5640 1 Processor -- start 2024-09-16T01:36:27.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.640+0000 7fd868ca5640 1 -- start start 2024-09-16T01:36:27.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.641+0000 7fd868ca5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8640b64e0 0x7fd864148890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:27.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.641+0000 7fd868ca5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8640cf6c0 0x7fd8641ef4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:27.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.641+0000 7fd868ca5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd86415c170 0x7fd8641f2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:27.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.641+0000 7fd868ca5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd8640b5870 con 0x7fd86415c170 2024-09-16T01:36:27.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.641+0000 7fd868ca5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd8640b5590 con 0x7fd8640b64e0 2024-09-16T01:36:27.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.641+0000 7fd868ca5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd8640b5b50 con 0x7fd8640cf6c0 2024-09-16T01:36:27.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.641+0000 7fd862d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd86415c170 0x7fd8641f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:27.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.641+0000 7fd862d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd86415c170 0x7fd8641f2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33068/0 (socket says 172.21.15.88:33068) 2024-09-16T01:36:27.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.641+0000 7fd862d9d640 1 -- 172.21.15.88:0/972598030 learned_addr learned my addr 172.21.15.88:0/972598030 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:27.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.641+0000 7fd86259c640 1 --2- 172.21.15.88:0/972598030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8640b64e0 0x7fd864148890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:27.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.641+0000 7fd861d9b640 1 --2- 172.21.15.88:0/972598030 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8640cf6c0 0x7fd8641ef4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:27.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.641+0000 7fd862d9d640 1 -- 172.21.15.88:0/972598030 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8640cf6c0 msgr2=0x7fd8641ef4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:27.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.641+0000 7fd862d9d640 1 --2- 172.21.15.88:0/972598030 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8640cf6c0 0x7fd8641ef4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.641+0000 7fd862d9d640 1 -- 172.21.15.88:0/972598030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8640b64e0 msgr2=0x7fd864148890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:27.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.641+0000 7fd862d9d640 1 --2- 172.21.15.88:0/972598030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8640b64e0 0x7fd864148890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.641+0000 7fd862d9d640 1 -- 172.21.15.88:0/972598030 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd864148dd0 con 0x7fd86415c170 2024-09-16T01:36:27.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.641+0000 7fd862d9d640 1 --2- 172.21.15.88:0/972598030 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd86415c170 0x7fd8641f2090 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fd85c00b9e0 tx=0x7fd85c00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:27.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.641+0000 7fd85affd640 1 -- 172.21.15.88:0/972598030 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c019070 con 0x7fd86415c170 2024-09-16T01:36:27.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.641+0000 7fd85affd640 1 -- 172.21.15.88:0/972598030 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd85c00ca60 con 0x7fd86415c170 2024-09-16T01:36:27.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.641+0000 7fd868ca5640 1 -- 172.21.15.88:0/972598030 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8641f2c60 con 0x7fd86415c170 2024-09-16T01:36:27.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.641+0000 7fd85affd640 1 -- 172.21.15.88:0/972598030 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c00ce50 con 0x7fd86415c170 2024-09-16T01:36:27.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.641+0000 7fd868ca5640 1 -- 172.21.15.88:0/972598030 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd86414c060 con 0x7fd86415c170 2024-09-16T01:36:27.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.642+0000 7fd868ca5640 1 -- 172.21.15.88:0/972598030 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd86414be70 con 0x7fd86415c170 2024-09-16T01:36:27.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.643+0000 7fd85affd640 1 -- 172.21.15.88:0/972598030 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd85c01c070 con 0x7fd86415c170 2024-09-16T01:36:27.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.643+0000 7fd85affd640 1 --2- 172.21.15.88:0/972598030 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd84406c450 0x7fd84406e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:27.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.643+0000 7fd85affd640 1 -- 172.21.15.88:0/972598030 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7fd85c08b160 con 0x7fd86415c170 2024-09-16T01:36:27.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.643+0000 7fd86259c640 1 --2- 172.21.15.88:0/972598030 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd84406c450 0x7fd84406e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:27.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.644+0000 7fd86259c640 1 --2- 172.21.15.88:0/972598030 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd84406c450 0x7fd84406e910 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd854034000 tx=0x7fd85402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:27.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.647+0000 7fd85affd640 1 -- 172.21.15.88:0/972598030 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd85c059870 con 0x7fd86415c170 2024-09-16T01:36:27.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.849+0000 7fd868ca5640 1 -- 172.21.15.88:0/972598030 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8640cfb30 con 0x7fd86415c170 2024-09-16T01:36:27.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.849+0000 7fd85affd640 1 -- 172.21.15.88:0/972598030 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+5227 (secure 0 0 0) 0x7fd85c059170 con 0x7fd86415c170 2024-09-16T01:36:27.851 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:27.851 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":31,"btime":"2024-09-16T01:36:27:804942+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4877,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1769276566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":1769276566},{"type":"v1","addr":"172.21.15.154:6835","nonce":1769276566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/725621378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":725621378},{"type":"v1","addr":"172.21.15.154:6834","nonce":725621378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4199111872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4199111872},{"type":"v1","addr":"172.21.15.88:6837","nonce":4199111872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:05.691540+0000","modified":"2024-09-16T01:36:27.804937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"c","rank":0,"incarnation":28,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6836/2700280540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2700280540},{"type":"v1","addr":"172.21.15.88:6836","nonce":2700280540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5634,"qdb_cluster":[5634]},"id":3}]} 2024-09-16T01:36:27.851 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 31 2024-09-16T01:36:27.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.851+0000 7fd868ca5640 1 -- 172.21.15.88:0/972598030 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd84406c450 msgr2=0x7fd84406e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:27.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.851+0000 7fd868ca5640 1 --2- 172.21.15.88:0/972598030 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd84406c450 0x7fd84406e910 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd854034000 tx=0x7fd85402e000 comp rx=0 tx=0).stop 2024-09-16T01:36:27.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.851+0000 7fd868ca5640 1 -- 172.21.15.88:0/972598030 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd86415c170 msgr2=0x7fd8641f2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:27.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.851+0000 7fd868ca5640 1 --2- 172.21.15.88:0/972598030 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd86415c170 0x7fd8641f2090 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fd85c00b9e0 tx=0x7fd85c00bea0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.851+0000 7fd868ca5640 1 -- 172.21.15.88:0/972598030 shutdown_connections 2024-09-16T01:36:27.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.851+0000 7fd868ca5640 1 --2- 172.21.15.88:0/972598030 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd84406c450 0x7fd84406e910 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.851+0000 7fd868ca5640 1 --2- 172.21.15.88:0/972598030 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd86415c170 0x7fd8641f2090 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.851+0000 7fd868ca5640 1 --2- 172.21.15.88:0/972598030 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd8640cf6c0 0x7fd8641ef4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.851+0000 7fd868ca5640 1 --2- 172.21.15.88:0/972598030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8640b64e0 0x7fd864148890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.851+0000 7fd868ca5640 1 -- 172.21.15.88:0/972598030 >> 172.21.15.88:0/972598030 conn(0x7fd86406bc10 msgr2=0x7fd86415ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:27.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.851+0000 7fd868ca5640 1 -- 172.21.15.88:0/972598030 shutdown_connections 2024-09-16T01:36:27.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.851+0000 7fd868ca5640 1 -- 172.21.15.88:0/972598030 wait complete. 2024-09-16T01:36:27.864 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:36:27.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.961+0000 7f806853b640 1 Processor -- start 2024-09-16T01:36:27.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.961+0000 7f806853b640 1 -- start start 2024-09-16T01:36:27.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.962+0000 7f806853b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f806015c170 0x7f806015c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:27.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.962+0000 7f806853b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f80600cf6c0 0x7f80600cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:27.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.962+0000 7f806853b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80600b49c0 con 0x7f80600b5dd0 2024-09-16T01:36:27.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.962+0000 7f806853b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f806015ca90 con 0x7f80600cf6c0 2024-09-16T01:36:27.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.962+0000 7f806853b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f80600b4cf0 con 0x7f806015c170 2024-09-16T01:36:27.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.962+0000 7f8065ad6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f80600cf6c0 0x7f80600cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:27.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.962+0000 7f8066ad8640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f80600b5dd0 0x7f806015ffb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42796/0 (socket says 172.21.15.88:42796) 2024-09-16T01:36:27.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.962+0000 7f8065ad6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f80600cf6c0 0x7f80600cfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33070/0 (socket says 172.21.15.88:33070) 2024-09-16T01:36:27.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.962+0000 7f8066ad8640 1 -- 172.21.15.88:0/3363580736 learned_addr learned my addr 172.21.15.88:0/3363580736 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:27.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.962+0000 7f80662d7640 1 --2- 172.21.15.88:0/3363580736 >> v2:172.21.15.154:3300/0 conn(0x7f806015c170 0x7f806015c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:27.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.962+0000 7f8065ad6640 1 -- 172.21.15.88:0/3363580736 >> v2:172.21.15.154:3300/0 conn(0x7f806015c170 msgr2=0x7f806015c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:27.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.962+0000 7f8065ad6640 1 --2- 172.21.15.88:0/3363580736 >> v2:172.21.15.154:3300/0 conn(0x7f806015c170 0x7f806015c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.962+0000 7f8065ad6640 1 -- 172.21.15.88:0/3363580736 >> v1:172.21.15.88:6789/0 conn(0x7f80600b5dd0 legacy=0x7f806015ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:27.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.962+0000 7f8065ad6640 1 -- 172.21.15.88:0/3363580736 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80600b4fd0 con 0x7f80600cf6c0 2024-09-16T01:36:27.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.962+0000 7f80652d5640 1 -- 172.21.15.88:0/3363580736 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2270276326 0 0) 0x7f80600b49c0 con 0x7f80600b5dd0 2024-09-16T01:36:27.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.963+0000 7f8065ad6640 1 --2- 172.21.15.88:0/3363580736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80600cf6c0 0x7f80600cfb30 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f80500097a0 tx=0x7f805002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cba81927e382d70 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:27.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.963+0000 7f80652d5640 1 -- 172.21.15.88:0/3363580736 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8050037070 con 0x7f80600cf6c0 2024-09-16T01:36:27.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.963+0000 7f80652d5640 1 -- 172.21.15.88:0/3363580736 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8050032440 con 0x7f80600cf6c0 2024-09-16T01:36:27.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.963+0000 7f80652d5640 1 -- 172.21.15.88:0/3363580736 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8050032810 con 0x7f80600cf6c0 2024-09-16T01:36:27.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.964+0000 7f806853b640 1 -- 172.21.15.88:0/3363580736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80600cf6c0 msgr2=0x7f80600cfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:27.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.964+0000 7f806853b640 1 --2- 172.21.15.88:0/3363580736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80600cf6c0 0x7f80600cfb30 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f80500097a0 tx=0x7f805002a1e0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.964+0000 7f806853b640 1 -- 172.21.15.88:0/3363580736 shutdown_connections 2024-09-16T01:36:27.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.964+0000 7f806853b640 1 --2- 172.21.15.88:0/3363580736 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f80600cf6c0 0x7f80600cfb30 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.964+0000 7f806853b640 1 --2- 172.21.15.88:0/3363580736 >> v2:172.21.15.154:3300/0 conn(0x7f806015c170 0x7f806015c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.964+0000 7f806853b640 1 -- 172.21.15.88:0/3363580736 >> 172.21.15.88:0/3363580736 conn(0x7f806006bc10 msgr2=0x7f806006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:27.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.964+0000 7f806853b640 1 -- 172.21.15.88:0/3363580736 shutdown_connections 2024-09-16T01:36:27.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.964+0000 7f806853b640 1 -- 172.21.15.88:0/3363580736 wait complete. 2024-09-16T01:36:27.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.965+0000 7f806853b640 1 Processor -- start 2024-09-16T01:36:27.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.965+0000 7f806853b640 1 -- start start 2024-09-16T01:36:27.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.966+0000 7f806853b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80600b5dd0 0x7f80601f2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:27.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.966+0000 7f806853b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80600cf6c0 0x7f8060157f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:27.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.966+0000 7f806853b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f806015c170 0x7f806006d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:27.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.966+0000 7f806853b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f80600b5870 con 0x7f806015c170 2024-09-16T01:36:27.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.966+0000 7f806853b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f80600b5590 con 0x7f80600cf6c0 2024-09-16T01:36:27.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.966+0000 7f806853b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f80600b5b50 con 0x7f80600b5dd0 2024-09-16T01:36:27.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.966+0000 7f8066ad8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f806015c170 0x7f806006d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:27.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.966+0000 7f8066ad8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f806015c170 0x7f806006d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33086/0 (socket says 172.21.15.88:33086) 2024-09-16T01:36:27.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.966+0000 7f8066ad8640 1 -- 172.21.15.88:0/2525038096 learned_addr learned my addr 172.21.15.88:0/2525038096 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:27.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.966+0000 7f80662d7640 1 --2- 172.21.15.88:0/2525038096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80600b5dd0 0x7f80601f2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:27.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.966+0000 7f8065ad6640 1 --2- 172.21.15.88:0/2525038096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80600cf6c0 0x7f8060157f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:27.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.966+0000 7f8066ad8640 1 -- 172.21.15.88:0/2525038096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80600b5dd0 msgr2=0x7f80601f2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:27.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.966+0000 7f8066ad8640 1 --2- 172.21.15.88:0/2525038096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80600b5dd0 0x7f80601f2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.966+0000 7f8066ad8640 1 -- 172.21.15.88:0/2525038096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80600cf6c0 msgr2=0x7f8060157f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:27.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.966+0000 7f8066ad8640 1 --2- 172.21.15.88:0/2525038096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80600cf6c0 0x7f8060157f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:27.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.966+0000 7f8066ad8640 1 -- 172.21.15.88:0/2525038096 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8060156730 con 0x7f806015c170 2024-09-16T01:36:27.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.966+0000 7f8065ad6640 1 --2- 172.21.15.88:0/2525038096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80600cf6c0 0x7f8060157f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:27.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.966+0000 7f80662d7640 1 --2- 172.21.15.88:0/2525038096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80600b5dd0 0x7f80601f2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:27.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.966+0000 7f8066ad8640 1 --2- 172.21.15.88:0/2525038096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f806015c170 0x7f806006d0a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f805800c800 tx=0x7f805800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:27.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.966+0000 7f8056ffd640 1 -- 172.21.15.88:0/2525038096 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8058018070 con 0x7f806015c170 2024-09-16T01:36:27.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.966+0000 7f8056ffd640 1 -- 172.21.15.88:0/2525038096 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8058004d80 con 0x7f806015c170 2024-09-16T01:36:27.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.966+0000 7f806853b640 1 -- 172.21.15.88:0/2525038096 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8060156a10 con 0x7f806015c170 2024-09-16T01:36:27.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.966+0000 7f8056ffd640 1 -- 172.21.15.88:0/2525038096 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805800a880 con 0x7f806015c170 2024-09-16T01:36:27.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.966+0000 7f806853b640 1 -- 172.21.15.88:0/2525038096 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8060156fd0 con 0x7f806015c170 2024-09-16T01:36:27.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.967+0000 7f806853b640 1 -- 172.21.15.88:0/2525038096 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80600cfb30 con 0x7f806015c170 2024-09-16T01:36:27.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.968+0000 7f8056ffd640 1 -- 172.21.15.88:0/2525038096 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f805801b070 con 0x7f806015c170 2024-09-16T01:36:27.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.968+0000 7f8056ffd640 1 --2- 172.21.15.88:0/2525038096 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f804406c2b0 0x7f804406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:27.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.968+0000 7f8056ffd640 1 -- 172.21.15.88:0/2525038096 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f805808aa40 con 0x7f806015c170 2024-09-16T01:36:27.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.968+0000 7f80662d7640 1 --2- 172.21.15.88:0/2525038096 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f804406c2b0 0x7f804406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:27.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.969+0000 7f80662d7640 1 --2- 172.21.15.88:0/2525038096 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f804406c2b0 0x7f804406e770 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f805c006fd0 tx=0x7f805c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:27.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:27.973+0000 7f8056ffd640 1 -- 172.21.15.88:0/2525038096 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8058059210 con 0x7f806015c170 2024-09-16T01:36:28.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.172+0000 7f806853b640 1 -- 172.21.15.88:0/2525038096 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80600b2c40 con 0x7f806015c170 2024-09-16T01:36:28.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.172+0000 7f8056ffd640 1 -- 172.21.15.88:0/2525038096 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+5227 (secure 0 0 0) 0x7f8058058b10 con 0x7f806015c170 2024-09-16T01:36:28.174 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:28.174 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":31,"btime":"2024-09-16T01:36:27:804942+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4877,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1769276566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":1769276566},{"type":"v1","addr":"172.21.15.154:6835","nonce":1769276566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/725621378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":725621378},{"type":"v1","addr":"172.21.15.154:6834","nonce":725621378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4199111872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4199111872},{"type":"v1","addr":"172.21.15.88:6837","nonce":4199111872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:05.691540+0000","modified":"2024-09-16T01:36:27.804937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5634":{"gid":5634,"name":"c","rank":0,"incarnation":28,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6836/2700280540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2700280540},{"type":"v1","addr":"172.21.15.88:6836","nonce":2700280540}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5634,"qdb_cluster":[5634]},"id":3}]} 2024-09-16T01:36:28.174 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 31 2024-09-16T01:36:28.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.174+0000 7f806853b640 1 -- 172.21.15.88:0/2525038096 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f804406c2b0 msgr2=0x7f804406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:28.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.174+0000 7f806853b640 1 --2- 172.21.15.88:0/2525038096 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f804406c2b0 0x7f804406e770 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f805c006fd0 tx=0x7f805c008040 comp rx=0 tx=0).stop 2024-09-16T01:36:28.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.174+0000 7f806853b640 1 -- 172.21.15.88:0/2525038096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f806015c170 msgr2=0x7f806006d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:28.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.174+0000 7f806853b640 1 --2- 172.21.15.88:0/2525038096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f806015c170 0x7f806006d0a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f805800c800 tx=0x7f805800ccc0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.174+0000 7f806853b640 1 -- 172.21.15.88:0/2525038096 shutdown_connections 2024-09-16T01:36:28.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.174+0000 7f806853b640 1 --2- 172.21.15.88:0/2525038096 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f804406c2b0 0x7f804406e770 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.174+0000 7f806853b640 1 --2- 172.21.15.88:0/2525038096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f806015c170 0x7f806006d0a0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.174+0000 7f806853b640 1 --2- 172.21.15.88:0/2525038096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80600cf6c0 0x7f8060157f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.174+0000 7f806853b640 1 --2- 172.21.15.88:0/2525038096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80600b5dd0 0x7f80601f2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.174+0000 7f806853b640 1 -- 172.21.15.88:0/2525038096 >> 172.21.15.88:0/2525038096 conn(0x7f806006bc10 msgr2=0x7f8060155190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:28.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.175+0000 7f806853b640 1 -- 172.21.15.88:0/2525038096 shutdown_connections 2024-09-16T01:36:28.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.175+0000 7f806853b640 1 -- 172.21.15.88:0/2525038096 wait complete. 2024-09-16T01:36:28.187 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:36:28.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.283+0000 7f92488d0640 1 Processor -- start 2024-09-16T01:36:28.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.283+0000 7f92488d0640 1 -- start start 2024-09-16T01:36:28.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.284+0000 7f92488d0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f924415c670 0x7f9244073460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:28.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.284+0000 7f92488d0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9244071750 con 0x7f92440b1990 2024-09-16T01:36:28.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.284+0000 7f92488d0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92440d0370 con 0x7f9244073080 2024-09-16T01:36:28.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.284+0000 7f92488d0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f92440d0c60 con 0x7f924415c670 2024-09-16T01:36:28.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.285+0000 7f9241d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f924415c670 0x7f9244073460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:28.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.285+0000 7f9241d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f924415c670 0x7f9244073460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33096/0 (socket says 172.21.15.88:33096) 2024-09-16T01:36:28.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.285+0000 7f9241d9b640 1 -- 172.21.15.88:0/2155230839 learned_addr learned my addr 172.21.15.88:0/2155230839 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:28.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.285+0000 7f9242d9d640 1 --1- 172.21.15.88:0/2155230839 >> v1:172.21.15.88:6789/0 conn(0x7f92440b1990 0x7f92440d10f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42808/0 (socket says 172.21.15.88:42808) 2024-09-16T01:36:28.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.285+0000 7f9241d9b640 1 -- 172.21.15.88:0/2155230839 >> v1:172.21.15.154:6789/0 conn(0x7f9244073080 legacy=0x7f92440b5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:28.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.285+0000 7f9241d9b640 1 -- 172.21.15.88:0/2155230839 >> v1:172.21.15.88:6789/0 conn(0x7f92440b1990 legacy=0x7f92440d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:28.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.285+0000 7f9241d9b640 1 -- 172.21.15.88:0/2155230839 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92440739a0 con 0x7f924415c670 2024-09-16T01:36:28.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.285+0000 7f924159a640 1 -- 172.21.15.88:0/2155230839 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 334481833 0 0) 0x7f9244071750 con 0x7f92440b1990 2024-09-16T01:36:28.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.286+0000 7f9241d9b640 1 --2- 172.21.15.88:0/2155230839 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f924415c670 0x7f9244073460 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9238009870 tx=0x7f923802a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42c3bf14775e0482 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:28.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.286+0000 7f924159a640 1 -- 172.21.15.88:0/2155230839 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9238037070 con 0x7f924415c670 2024-09-16T01:36:28.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.286+0000 7f924159a640 1 -- 172.21.15.88:0/2155230839 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9238032440 con 0x7f924415c670 2024-09-16T01:36:28.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.286+0000 7f924159a640 1 -- 172.21.15.88:0/2155230839 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9238032810 con 0x7f924415c670 2024-09-16T01:36:28.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.286+0000 7f92488d0640 1 -- 172.21.15.88:0/2155230839 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f924415c670 msgr2=0x7f9244073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:28.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.286+0000 7f92488d0640 1 --2- 172.21.15.88:0/2155230839 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f924415c670 0x7f9244073460 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9238009870 tx=0x7f923802a2a0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.286+0000 7f92488d0640 1 -- 172.21.15.88:0/2155230839 shutdown_connections 2024-09-16T01:36:28.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.286+0000 7f92488d0640 1 --2- 172.21.15.88:0/2155230839 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f924415c670 0x7f9244073460 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.286+0000 7f92488d0640 1 -- 172.21.15.88:0/2155230839 >> 172.21.15.88:0/2155230839 conn(0x7f92440d24d0 msgr2=0x7f92440d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:28.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.286+0000 7f92488d0640 1 -- 172.21.15.88:0/2155230839 shutdown_connections 2024-09-16T01:36:28.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.286+0000 7f92488d0640 1 -- 172.21.15.88:0/2155230839 wait complete. 2024-09-16T01:36:28.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.288+0000 7f92488d0640 1 Processor -- start 2024-09-16T01:36:28.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.288+0000 7f92488d0640 1 -- start start 2024-09-16T01:36:28.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.288+0000 7f92488d0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9244073080 0x7f924414f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:28.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.288+0000 7f92488d0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92440b1990 0x7f924414c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:28.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.288+0000 7f92488d0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f924415c670 0x7f924414d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:28.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.288+0000 7f92488d0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f92440d0610 con 0x7f924415c670 2024-09-16T01:36:28.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.288+0000 7f92488d0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9244071470 con 0x7f92440b1990 2024-09-16T01:36:28.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.288+0000 7f92488d0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f92440d08f0 con 0x7f9244073080 2024-09-16T01:36:28.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.289+0000 7f9242d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f924415c670 0x7f924414d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:28.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.289+0000 7f9242d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f924415c670 0x7f924414d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33102/0 (socket says 172.21.15.88:33102) 2024-09-16T01:36:28.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.289+0000 7f9242d9d640 1 -- 172.21.15.88:0/2567549145 learned_addr learned my addr 172.21.15.88:0/2567549145 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:28.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.289+0000 7f9241d9b640 1 --2- 172.21.15.88:0/2567549145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92440b1990 0x7f924414c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:28.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.289+0000 7f924259c640 1 --2- 172.21.15.88:0/2567549145 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9244073080 0x7f924414f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:28.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.289+0000 7f9242d9d640 1 -- 172.21.15.88:0/2567549145 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9244073080 msgr2=0x7f924414f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:28.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.289+0000 7f9242d9d640 1 --2- 172.21.15.88:0/2567549145 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9244073080 0x7f924414f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.289+0000 7f9242d9d640 1 -- 172.21.15.88:0/2567549145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92440b1990 msgr2=0x7f924414c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:28.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.289+0000 7f9242d9d640 1 --2- 172.21.15.88:0/2567549145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92440b1990 0x7f924414c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.289+0000 7f9242d9d640 1 -- 172.21.15.88:0/2567549145 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9244159960 con 0x7f924415c670 2024-09-16T01:36:28.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.289+0000 7f9242d9d640 1 --2- 172.21.15.88:0/2567549145 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f924415c670 0x7f924414d950 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f923c00c780 tx=0x7f923c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:28.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.289+0000 7f9232ffd640 1 -- 172.21.15.88:0/2567549145 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923c018070 con 0x7f924415c670 2024-09-16T01:36:28.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.289+0000 7f9232ffd640 1 -- 172.21.15.88:0/2567549145 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f923c00a640 con 0x7f924415c670 2024-09-16T01:36:28.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.289+0000 7f9232ffd640 1 -- 172.21.15.88:0/2567549145 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923c00aa70 con 0x7f924415c670 2024-09-16T01:36:28.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.289+0000 7f92488d0640 1 -- 172.21.15.88:0/2567549145 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9244159c40 con 0x7f924415c670 2024-09-16T01:36:28.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.289+0000 7f92488d0640 1 -- 172.21.15.88:0/2567549145 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f924414de90 con 0x7f924415c670 2024-09-16T01:36:28.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.290+0000 7f92488d0640 1 -- 172.21.15.88:0/2567549145 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f924415a260 con 0x7f924415c670 2024-09-16T01:36:28.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.291+0000 7f9232ffd640 1 -- 172.21.15.88:0/2567549145 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f923c01b070 con 0x7f924415c670 2024-09-16T01:36:28.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.291+0000 7f9232ffd640 1 --2- 172.21.15.88:0/2567549145 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f922406c2b0 0x7f922406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:28.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.291+0000 7f924259c640 1 --2- 172.21.15.88:0/2567549145 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f922406c2b0 0x7f922406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:28.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.291+0000 7f9232ffd640 1 -- 172.21.15.88:0/2567549145 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f923c08aa50 con 0x7f924415c670 2024-09-16T01:36:28.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.292+0000 7f924259c640 1 --2- 172.21.15.88:0/2567549145 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f922406c2b0 0x7f922406e770 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9234006fd0 tx=0x7f9234008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:28.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.295+0000 7f9232ffd640 1 -- 172.21.15.88:0/2567549145 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f924415a260 con 0x7f924415c670 2024-09-16T01:36:28.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.463+0000 7f92488d0640 1 -- 172.21.15.88:0/2567549145 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f92440d1f00 con 0x7f924415c670 2024-09-16T01:36:28.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.464+0000 7f9232ffd640 1 -- 172.21.15.88:0/2567549145 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) ==== 74+0+16778 (secure 0 0 0) 0x7f923c0591f0 con 0x7f924415c670 2024-09-16T01:36:28.466 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:28.466 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":46,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:36:23.795684+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:36:04.035328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-16T01:36:04.663046+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:6836/1610810484":"2024-09-17T01:36:23.795661+0000","172.21.15.88:6834/1610810484":"2024-09-17T01:36:23.795661+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:36:28.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.466+0000 7f92488d0640 1 -- 172.21.15.88:0/2567549145 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f922406c2b0 msgr2=0x7f922406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:28.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.466+0000 7f92488d0640 1 --2- 172.21.15.88:0/2567549145 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f922406c2b0 0x7f922406e770 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9234006fd0 tx=0x7f9234008040 comp rx=0 tx=0).stop 2024-09-16T01:36:28.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.466+0000 7f92488d0640 1 -- 172.21.15.88:0/2567549145 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f924415c670 msgr2=0x7f924414d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:28.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.466+0000 7f92488d0640 1 --2- 172.21.15.88:0/2567549145 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f924415c670 0x7f924414d950 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f923c00c780 tx=0x7f923c00cc40 comp rx=0 tx=0).stop 2024-09-16T01:36:28.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.466+0000 7f92488d0640 1 -- 172.21.15.88:0/2567549145 shutdown_connections 2024-09-16T01:36:28.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.466+0000 7f92488d0640 1 --2- 172.21.15.88:0/2567549145 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f922406c2b0 0x7f922406e770 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.466+0000 7f92488d0640 1 --2- 172.21.15.88:0/2567549145 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f924415c670 0x7f924414d950 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.466+0000 7f92488d0640 1 --2- 172.21.15.88:0/2567549145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92440b1990 0x7f924414c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.466+0000 7f92488d0640 1 --2- 172.21.15.88:0/2567549145 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9244073080 0x7f924414f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.466+0000 7f92488d0640 1 -- 172.21.15.88:0/2567549145 >> 172.21.15.88:0/2567549145 conn(0x7f92440d24d0 msgr2=0x7f9244157630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:28.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.467+0000 7f92488d0640 1 -- 172.21.15.88:0/2567549145 shutdown_connections 2024-09-16T01:36:28.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.467+0000 7f92488d0640 1 -- 172.21.15.88:0/2567549145 wait complete. 2024-09-16T01:36:28.479 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:36:28.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.578+0000 7f40d5131640 1 Processor -- start 2024-09-16T01:36:28.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.578+0000 7f40d5131640 1 -- start start 2024-09-16T01:36:28.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.578+0000 7f40d5131640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f40d015e7a0 0x7f40d006caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:28.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.578+0000 7f40d5131640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40d00cf1f0 0x7f40d00b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:28.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.578+0000 7f40d5131640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f40d00b6720 0x7f40d01623c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:28.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.578+0000 7f40d5131640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f40d00d2ca0 con 0x7f40d00b6720 2024-09-16T01:36:28.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.578+0000 7f40d5131640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f40d00cf860 con 0x7f40d015e7a0 2024-09-16T01:36:28.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.578+0000 7f40d5131640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f40d00cfb40 con 0x7f40d00cf1f0 2024-09-16T01:36:28.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.579+0000 7f40cf59e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f40d00b6720 0x7f40d01623c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:28.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.579+0000 7f40cf59e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f40d00b6720 0x7f40d01623c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33114/0 (socket says 172.21.15.88:33114) 2024-09-16T01:36:28.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.579+0000 7f40cf59e640 1 -- 172.21.15.88:0/2378087351 learned_addr learned my addr 172.21.15.88:0/2378087351 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:28.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.579+0000 7f40ce59c640 1 --2- 172.21.15.88:0/2378087351 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40d00cf1f0 0x7f40d00b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:28.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.579+0000 7f40ced9d640 1 --2- 172.21.15.88:0/2378087351 >> v2:172.21.15.154:3300/0 conn(0x7f40d015e7a0 0x7f40d006caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:28.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.579+0000 7f40cf59e640 1 -- 172.21.15.88:0/2378087351 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40d00cf1f0 msgr2=0x7f40d00b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:28.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.579+0000 7f40cf59e640 1 --2- 172.21.15.88:0/2378087351 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40d00cf1f0 0x7f40d00b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.579+0000 7f40cf59e640 1 -- 172.21.15.88:0/2378087351 >> v2:172.21.15.154:3300/0 conn(0x7f40d015e7a0 msgr2=0x7f40d006caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:28.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.579+0000 7f40cf59e640 1 --2- 172.21.15.88:0/2378087351 >> v2:172.21.15.154:3300/0 conn(0x7f40d015e7a0 0x7f40d006caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.579+0000 7f40cf59e640 1 -- 172.21.15.88:0/2378087351 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d00cfe20 con 0x7f40d00b6720 2024-09-16T01:36:28.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.579+0000 7f40ce59c640 1 --2- 172.21.15.88:0/2378087351 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40d00cf1f0 0x7f40d00b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:28.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.580+0000 7f40cf59e640 1 --2- 172.21.15.88:0/2378087351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40d00b6720 0x7f40d01623c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f40d00d5820 tx=0x7f40c002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb1bfbd347f745b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:28.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.580+0000 7f40cdd9b640 1 -- 172.21.15.88:0/2378087351 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c0036070 con 0x7f40d00b6720 2024-09-16T01:36:28.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.580+0000 7f40cdd9b640 1 -- 172.21.15.88:0/2378087351 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40c0031600 con 0x7f40d00b6720 2024-09-16T01:36:28.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.580+0000 7f40cdd9b640 1 -- 172.21.15.88:0/2378087351 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c00319d0 con 0x7f40d00b6720 2024-09-16T01:36:28.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.580+0000 7f40d5131640 1 -- 172.21.15.88:0/2378087351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40d00b6720 msgr2=0x7f40d01623c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:28.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.580+0000 7f40d5131640 1 --2- 172.21.15.88:0/2378087351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40d00b6720 0x7f40d01623c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f40d00d5820 tx=0x7f40c002a610 comp rx=0 tx=0).stop 2024-09-16T01:36:28.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.580+0000 7f40d5131640 1 -- 172.21.15.88:0/2378087351 shutdown_connections 2024-09-16T01:36:28.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.580+0000 7f40d5131640 1 --2- 172.21.15.88:0/2378087351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40d00b6720 0x7f40d01623c0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.580+0000 7f40d5131640 1 --2- 172.21.15.88:0/2378087351 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40d00cf1f0 0x7f40d00b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.580+0000 7f40d5131640 1 --2- 172.21.15.88:0/2378087351 >> v2:172.21.15.154:3300/0 conn(0x7f40d015e7a0 0x7f40d006caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.580+0000 7f40d5131640 1 -- 172.21.15.88:0/2378087351 >> 172.21.15.88:0/2378087351 conn(0x7f40d00d11e0 msgr2=0x7f40d00d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:28.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.580+0000 7f40d5131640 1 -- 172.21.15.88:0/2378087351 shutdown_connections 2024-09-16T01:36:28.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.580+0000 7f40d5131640 1 -- 172.21.15.88:0/2378087351 wait complete. 2024-09-16T01:36:28.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.582+0000 7f40d5131640 1 Processor -- start 2024-09-16T01:36:28.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.582+0000 7f40d5131640 1 -- start start 2024-09-16T01:36:28.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.583+0000 7f40d5131640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40d00b6720 0x7f40d00c83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:28.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.583+0000 7f40d5131640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40d00cf1f0 0x7f40d0152e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:28.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.583+0000 7f40d5131640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40d015e7a0 0x7f40d01e8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:28.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.583+0000 7f40d5131640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f40d006cfe0 con 0x7f40d015e7a0 2024-09-16T01:36:28.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.583+0000 7f40d5131640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f40d00b7a00 con 0x7f40d00b6720 2024-09-16T01:36:28.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.583+0000 7f40d5131640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f40d015eb80 con 0x7f40d00cf1f0 2024-09-16T01:36:28.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.583+0000 7f40cf59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40d015e7a0 0x7f40d01e8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:28.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.583+0000 7f40ced9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40d00b6720 0x7f40d00c83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:28.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.583+0000 7f40cf59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40d015e7a0 0x7f40d01e8bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33126/0 (socket says 172.21.15.88:33126) 2024-09-16T01:36:28.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.583+0000 7f40cf59e640 1 -- 172.21.15.88:0/1082836102 learned_addr learned my addr 172.21.15.88:0/1082836102 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:28.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.583+0000 7f40ce59c640 1 --2- 172.21.15.88:0/1082836102 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40d00cf1f0 0x7f40d0152e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:28.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.583+0000 7f40cf59e640 1 -- 172.21.15.88:0/1082836102 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40d00cf1f0 msgr2=0x7f40d0152e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:28.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.583+0000 7f40cf59e640 1 --2- 172.21.15.88:0/1082836102 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40d00cf1f0 0x7f40d0152e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.583+0000 7f40cf59e640 1 -- 172.21.15.88:0/1082836102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40d00b6720 msgr2=0x7f40d00c83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:28.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.583+0000 7f40cf59e640 1 --2- 172.21.15.88:0/1082836102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40d00b6720 0x7f40d00c83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.583+0000 7f40cf59e640 1 -- 172.21.15.88:0/1082836102 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d01e90f0 con 0x7f40d015e7a0 2024-09-16T01:36:28.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.583+0000 7f40cf59e640 1 --2- 172.21.15.88:0/1082836102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40d015e7a0 0x7f40d01e8bb0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f40c0034040 tx=0x7f40c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:28.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.583+0000 7f40c77fe640 1 -- 172.21.15.88:0/1082836102 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c0041020 con 0x7f40d015e7a0 2024-09-16T01:36:28.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.583+0000 7f40c77fe640 1 -- 172.21.15.88:0/1082836102 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40c00077a0 con 0x7f40d015e7a0 2024-09-16T01:36:28.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.583+0000 7f40d5131640 1 -- 172.21.15.88:0/1082836102 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40d00c89b0 con 0x7f40d015e7a0 2024-09-16T01:36:28.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.583+0000 7f40c77fe640 1 -- 172.21.15.88:0/1082836102 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c0036040 con 0x7f40d015e7a0 2024-09-16T01:36:28.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.584+0000 7f40d5131640 1 -- 172.21.15.88:0/1082836102 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40d00c8f70 con 0x7f40d015e7a0 2024-09-16T01:36:28.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.585+0000 7f40d5131640 1 -- 172.21.15.88:0/1082836102 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40d00b61e0 con 0x7f40d015e7a0 2024-09-16T01:36:28.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.585+0000 7f40c77fe640 1 -- 172.21.15.88:0/1082836102 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f40c0044020 con 0x7f40d015e7a0 2024-09-16T01:36:28.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.585+0000 7f40c77fe640 1 --2- 172.21.15.88:0/1082836102 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f40b006c2b0 0x7f40b006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:28.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.586+0000 7f40c77fe640 1 -- 172.21.15.88:0/1082836102 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f40c00abfc0 con 0x7f40d015e7a0 2024-09-16T01:36:28.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.586+0000 7f40ced9d640 1 --2- 172.21.15.88:0/1082836102 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f40b006c2b0 0x7f40b006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:28.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.586+0000 7f40ced9d640 1 --2- 172.21.15.88:0/1082836102 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f40b006c2b0 0x7f40b006e770 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f40c8006fd0 tx=0x7f40c8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:28.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.590+0000 7f40c77fe640 1 -- 172.21.15.88:0/1082836102 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f40c004b030 con 0x7f40d015e7a0 2024-09-16T01:36:28.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.788+0000 7f40d5131640 1 -- 172.21.15.88:0/1082836102 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f40d00d2a80 con 0x7f40d015e7a0 2024-09-16T01:36:28.789 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:36:28.788+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:36:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.790+0000 7f40c77fe640 1 -- 172.21.15.88:0/1082836102 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v32) ==== 161+0+0 (secure 0 0 0) 0x7f40c007a110 con 0x7f40d015e7a0 2024-09-16T01:36:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:36:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.793+0000 7f40d5131640 1 -- 172.21.15.88:0/1082836102 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f40b006c2b0 msgr2=0x7f40b006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.793+0000 7f40d5131640 1 --2- 172.21.15.88:0/1082836102 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f40b006c2b0 0x7f40b006e770 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f40c8006fd0 tx=0x7f40c8008040 comp rx=0 tx=0).stop 2024-09-16T01:36:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.793+0000 7f40d5131640 1 -- 172.21.15.88:0/1082836102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40d015e7a0 msgr2=0x7f40d01e8bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.793+0000 7f40d5131640 1 --2- 172.21.15.88:0/1082836102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40d015e7a0 0x7f40d01e8bb0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f40c0034040 tx=0x7f40c0004a40 comp rx=0 tx=0).stop 2024-09-16T01:36:28.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.793+0000 7f40d5131640 1 -- 172.21.15.88:0/1082836102 shutdown_connections 2024-09-16T01:36:28.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.793+0000 7f40d5131640 1 --2- 172.21.15.88:0/1082836102 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f40b006c2b0 0x7f40b006e770 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.793+0000 7f40d5131640 1 --2- 172.21.15.88:0/1082836102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40d015e7a0 0x7f40d01e8bb0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.793+0000 7f40d5131640 1 --2- 172.21.15.88:0/1082836102 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40d00cf1f0 0x7f40d0152e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.793+0000 7f40d5131640 1 --2- 172.21.15.88:0/1082836102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40d00b6720 0x7f40d00c83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.793+0000 7f40d5131640 1 -- 172.21.15.88:0/1082836102 >> 172.21.15.88:0/1082836102 conn(0x7f40d00d11e0 msgr2=0x7f40d00d15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:28.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.793+0000 7f40d5131640 1 -- 172.21.15.88:0/1082836102 shutdown_connections 2024-09-16T01:36:28.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.793+0000 7f40d5131640 1 -- 172.21.15.88:0/1082836102 wait complete. 2024-09-16T01:36:28.809 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-16T01:36:28.824 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:36:28.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.908+0000 7f8fcfb42640 1 Processor -- start 2024-09-16T01:36:28.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.909+0000 7f8fcfb42640 1 -- start start 2024-09-16T01:36:28.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.910+0000 7f8fcfb42640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8fc8158a00 0x7f8fc8071f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:28.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.910+0000 7f8fcfb42640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8fc8158de0 0x7f8fc806eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:28.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.910+0000 7f8fcfb42640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fc80728f0 con 0x7f8fc8072470 2024-09-16T01:36:28.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.910+0000 7f8fcfb42640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f8fc8072bd0 con 0x7f8fc8158de0 2024-09-16T01:36:28.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.910+0000 7f8fcfb42640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8fc8072eb0 con 0x7f8fc8158a00 2024-09-16T01:36:28.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.910+0000 7f8fce0df640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8fc8158de0 0x7f8fc806eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:28.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.910+0000 7f8fce0df640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8fc8158de0 0x7f8fc806eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33158/0 (socket says 172.21.15.88:33158) 2024-09-16T01:36:28.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.910+0000 7f8fce0df640 1 -- 172.21.15.88:0/432113477 learned_addr learned my addr 172.21.15.88:0/432113477 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:28.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.910+0000 7f8fcd8de640 1 --2- 172.21.15.88:0/432113477 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8fc8158a00 0x7f8fc8071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:28.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.910+0000 7f8fce0df640 1 -- 172.21.15.88:0/432113477 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8fc8158a00 msgr2=0x7f8fc8071f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:28.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.910+0000 7f8fce0df640 1 --2- 172.21.15.88:0/432113477 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8fc8158a00 0x7f8fc8071f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.910+0000 7f8fce0df640 1 -- 172.21.15.88:0/432113477 >> v1:172.21.15.88:6789/0 conn(0x7f8fc8072470 legacy=0x7f8fc80b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:28.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.910+0000 7f8fce0df640 1 -- 172.21.15.88:0/432113477 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fc8073190 con 0x7f8fc8158de0 2024-09-16T01:36:28.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.910+0000 7f8fcc8dc640 1 -- 172.21.15.88:0/432113477 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1385344463 0 0) 0x7f8fc80728f0 con 0x7f8fc8072470 2024-09-16T01:36:28.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.911+0000 7f8fce0df640 1 --2- 172.21.15.88:0/432113477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fc8158de0 0x7f8fc806eae0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009e50 tx=0x7f8fc402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18a185afc61a0fcb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:28.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.911+0000 7f8fcc8dc640 1 -- 172.21.15.88:0/432113477 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4036070 con 0x7f8fc8158de0 2024-09-16T01:36:28.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.911+0000 7f8fcc8dc640 1 -- 172.21.15.88:0/432113477 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fc4031600 con 0x7f8fc8158de0 2024-09-16T01:36:28.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.911+0000 7f8fcc8dc640 1 -- 172.21.15.88:0/432113477 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc40319d0 con 0x7f8fc8158de0 2024-09-16T01:36:28.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.911+0000 7f8fcfb42640 1 -- 172.21.15.88:0/432113477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fc8158de0 msgr2=0x7f8fc806eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:28.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.911+0000 7f8fcfb42640 1 --2- 172.21.15.88:0/432113477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fc8158de0 0x7f8fc806eae0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009e50 tx=0x7f8fc402a710 comp rx=0 tx=0).stop 2024-09-16T01:36:28.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.911+0000 7f8fcfb42640 1 -- 172.21.15.88:0/432113477 shutdown_connections 2024-09-16T01:36:28.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.911+0000 7f8fcfb42640 1 --2- 172.21.15.88:0/432113477 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fc8158de0 0x7f8fc806eae0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.911+0000 7f8fcfb42640 1 --2- 172.21.15.88:0/432113477 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8fc8158a00 0x7f8fc8071f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.911+0000 7f8fcfb42640 1 -- 172.21.15.88:0/432113477 >> 172.21.15.88:0/432113477 conn(0x7f8fc806cb50 msgr2=0x7f8fc806cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:28.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.912+0000 7f8fcfb42640 1 -- 172.21.15.88:0/432113477 shutdown_connections 2024-09-16T01:36:28.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.912+0000 7f8fcfb42640 1 -- 172.21.15.88:0/432113477 wait complete. 2024-09-16T01:36:28.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.913+0000 7f8fcfb42640 1 Processor -- start 2024-09-16T01:36:28.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.913+0000 7f8fcfb42640 1 -- start start 2024-09-16T01:36:28.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.913+0000 7f8fcfb42640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8fc8072470 0x7f8fc814d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:28.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.913+0000 7f8fcfb42640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fc8158a00 0x7f8fc80cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:28.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.913+0000 7f8fcfb42640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fc8158de0 0x7f8fc814afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:28.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.913+0000 7f8fcfb42640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc80b5ca0 con 0x7f8fc8158a00 2024-09-16T01:36:28.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.913+0000 7f8fcfb42640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc80b5b20 con 0x7f8fc8158de0 2024-09-16T01:36:28.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.913+0000 7f8fcfb42640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8fc80b5570 con 0x7f8fc8072470 2024-09-16T01:36:28.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.913+0000 7f8fcd0dd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fc8158a00 0x7f8fc80cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:28.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.913+0000 7f8fcd0dd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fc8158a00 0x7f8fc80cffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33160/0 (socket says 172.21.15.88:33160) 2024-09-16T01:36:28.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.914+0000 7f8fcd0dd640 1 -- 172.21.15.88:0/3225929410 learned_addr learned my addr 172.21.15.88:0/3225929410 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:28.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.914+0000 7f8fcd0dd640 1 -- 172.21.15.88:0/3225929410 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8fc8072470 msgr2=0x7f8fc814d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:28.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.914+0000 7f8fcd0dd640 1 --2- 172.21.15.88:0/3225929410 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8fc8072470 0x7f8fc814d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.914+0000 7f8fcd0dd640 1 -- 172.21.15.88:0/3225929410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fc8158de0 msgr2=0x7f8fc814afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:28.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.914+0000 7f8fcd0dd640 1 --2- 172.21.15.88:0/3225929410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fc8158de0 0x7f8fc814afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:28.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.914+0000 7f8fcd0dd640 1 -- 172.21.15.88:0/3225929410 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fc80cef40 con 0x7f8fc8158a00 2024-09-16T01:36:28.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.914+0000 7f8fcd0dd640 1 --2- 172.21.15.88:0/3225929410 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fc8158a00 0x7f8fc80cffe0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f8fb800c4b0 tx=0x7f8fb800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:28.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.914+0000 7f8fbe7fc640 1 -- 172.21.15.88:0/3225929410 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb8017070 con 0x7f8fc8158a00 2024-09-16T01:36:28.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.914+0000 7f8fbe7fc640 1 -- 172.21.15.88:0/3225929410 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fb8005480 con 0x7f8fc8158a00 2024-09-16T01:36:28.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.914+0000 7f8fbe7fc640 1 -- 172.21.15.88:0/3225929410 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb8003830 con 0x7f8fc8158a00 2024-09-16T01:36:28.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.914+0000 7f8fcfb42640 1 -- 172.21.15.88:0/3225929410 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fc80cf220 con 0x7f8fc8158a00 2024-09-16T01:36:28.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.914+0000 7f8fcfb42640 1 -- 172.21.15.88:0/3225929410 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fc80cf7e0 con 0x7f8fc8158a00 2024-09-16T01:36:28.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.915+0000 7f8fcfb42640 1 -- 172.21.15.88:0/3225929410 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fc8071f30 con 0x7f8fc8158a00 2024-09-16T01:36:28.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.915+0000 7f8fbe7fc640 1 -- 172.21.15.88:0/3225929410 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8fb801a070 con 0x7f8fc8158a00 2024-09-16T01:36:28.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.916+0000 7f8fbe7fc640 1 --2- 172.21.15.88:0/3225929410 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8fac06c2b0 0x7f8fac06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:28.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.916+0000 7f8fcd8de640 1 --2- 172.21.15.88:0/3225929410 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8fac06c2b0 0x7f8fac06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:28.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.916+0000 7f8fbe7fc640 1 -- 172.21.15.88:0/3225929410 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5881+0+0 (secure 0 0 0) 0x7f8fb808a900 con 0x7f8fc8158a00 2024-09-16T01:36:28.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.916+0000 7f8fcd8de640 1 --2- 172.21.15.88:0/3225929410 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8fac06c2b0 0x7f8fac06e770 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8fc806f1b0 tx=0x7f8fc0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:28.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:28.920+0000 7f8fbe7fc640 1 -- 172.21.15.88:0/3225929410 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8fb8058fc0 con 0x7f8fc8158a00 2024-09-16T01:36:29.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.125+0000 7f8fcfb42640 1 -- 172.21.15.88:0/3225929410 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f8fc80cfcc0 con 0x7f8fc8158a00 2024-09-16T01:36:29.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.791+0000 7f8fbe7fc640 1 -- 172.21.15.88:0/3225929410 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v33) ==== 104+0+0 (secure 0 0 0) 0x7f8fc80cfcc0 con 0x7f8fc8158a00 2024-09-16T01:36:29.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.794+0000 7f8fcfb42640 1 -- 172.21.15.88:0/3225929410 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8fac06c2b0 msgr2=0x7f8fac06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:29.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.794+0000 7f8fcfb42640 1 --2- 172.21.15.88:0/3225929410 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8fac06c2b0 0x7f8fac06e770 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8fc806f1b0 tx=0x7f8fc0008040 comp rx=0 tx=0).stop 2024-09-16T01:36:29.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.794+0000 7f8fcfb42640 1 -- 172.21.15.88:0/3225929410 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fc8158a00 msgr2=0x7f8fc80cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:29.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.794+0000 7f8fcfb42640 1 --2- 172.21.15.88:0/3225929410 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fc8158a00 0x7f8fc80cffe0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f8fb800c4b0 tx=0x7f8fb800c970 comp rx=0 tx=0).stop 2024-09-16T01:36:29.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.794+0000 7f8fcfb42640 1 -- 172.21.15.88:0/3225929410 shutdown_connections 2024-09-16T01:36:29.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.794+0000 7f8fcfb42640 1 --2- 172.21.15.88:0/3225929410 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8fac06c2b0 0x7f8fac06e770 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:29.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.794+0000 7f8fcfb42640 1 --2- 172.21.15.88:0/3225929410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fc8158de0 0x7f8fc814afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:29.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.794+0000 7f8fcfb42640 1 --2- 172.21.15.88:0/3225929410 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fc8158a00 0x7f8fc80cffe0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:29.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.794+0000 7f8fcfb42640 1 --2- 172.21.15.88:0/3225929410 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8fc8072470 0x7f8fc814d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:29.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.794+0000 7f8fcfb42640 1 -- 172.21.15.88:0/3225929410 >> 172.21.15.88:0/3225929410 conn(0x7f8fc806cb50 msgr2=0x7f8fc80b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:29.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.794+0000 7f8fcfb42640 1 -- 172.21.15.88:0/3225929410 shutdown_connections 2024-09-16T01:36:29.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.794+0000 7f8fcfb42640 1 -- 172.21.15.88:0/3225929410 wait complete. 2024-09-16T01:36:29.808 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-16T01:36:29.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.951+0000 7fdbaea53640 1 Processor -- start 2024-09-16T01:36:29.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.951+0000 7fdbaea53640 1 -- start start 2024-09-16T01:36:29.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.951+0000 7fdbaea53640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdba815b480 0x7fdba80b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:29.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.951+0000 7fdbaea53640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdba80b60c0 0x7fdba8071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:29.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.951+0000 7fdbaea53640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba80ccff0 con 0x7fdba8072530 2024-09-16T01:36:29.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.951+0000 7fdbaea53640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fdba80b6500 con 0x7fdba80b60c0 2024-09-16T01:36:29.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.951+0000 7fdbaea53640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdba8072990 con 0x7fdba815b480 2024-09-16T01:36:29.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.952+0000 7fdba77fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdba80b60c0 0x7fdba8071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:29.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.952+0000 7fdba77fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdba80b60c0 0x7fdba8071ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33164/0 (socket says 172.21.15.88:33164) 2024-09-16T01:36:29.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.952+0000 7fdba77fe640 1 -- 172.21.15.88:0/3902319193 learned_addr learned my addr 172.21.15.88:0/3902319193 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:29.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.952+0000 7fdba77fe640 1 -- 172.21.15.88:0/3902319193 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdba815b480 msgr2=0x7fdba80b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:29.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.952+0000 7fdba77fe640 1 --2- 172.21.15.88:0/3902319193 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdba815b480 0x7fdba80b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:29.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.952+0000 7fdba77fe640 1 -- 172.21.15.88:0/3902319193 >> v1:172.21.15.154:6789/0 conn(0x7fdba8072530 legacy=0x7fdba8070fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:29.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.952+0000 7fdba77fe640 1 -- 172.21.15.88:0/3902319193 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba80cd2d0 con 0x7fdba80b60c0 2024-09-16T01:36:29.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.953+0000 7fdba77fe640 1 --2- 172.21.15.88:0/3902319193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdba80b60c0 0x7fdba8071ff0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c009870 tx=0x7fdb9c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bf9e5f0f6e72903 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:29.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.953+0000 7fdba6ffd640 1 -- 172.21.15.88:0/3902319193 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c037070 con 0x7fdba80b60c0 2024-09-16T01:36:29.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.953+0000 7fdba6ffd640 1 -- 172.21.15.88:0/3902319193 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb9c032440 con 0x7fdba80b60c0 2024-09-16T01:36:29.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.953+0000 7fdba6ffd640 1 -- 172.21.15.88:0/3902319193 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c032810 con 0x7fdba80b60c0 2024-09-16T01:36:29.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.953+0000 7fdbaea53640 1 -- 172.21.15.88:0/3902319193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdba80b60c0 msgr2=0x7fdba8071ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:29.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.953+0000 7fdbaea53640 1 --2- 172.21.15.88:0/3902319193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdba80b60c0 0x7fdba8071ff0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c009870 tx=0x7fdb9c02a180 comp rx=0 tx=0).stop 2024-09-16T01:36:29.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.953+0000 7fdbaea53640 1 -- 172.21.15.88:0/3902319193 shutdown_connections 2024-09-16T01:36:29.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.953+0000 7fdbaea53640 1 --2- 172.21.15.88:0/3902319193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdba80b60c0 0x7fdba8071ff0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:29.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.953+0000 7fdbaea53640 1 --2- 172.21.15.88:0/3902319193 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdba815b480 0x7fdba80b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:29.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.953+0000 7fdbaea53640 1 -- 172.21.15.88:0/3902319193 >> 172.21.15.88:0/3902319193 conn(0x7fdba806d0f0 msgr2=0x7fdba806d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:29.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.953+0000 7fdbaea53640 1 -- 172.21.15.88:0/3902319193 shutdown_connections 2024-09-16T01:36:29.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.953+0000 7fdbaea53640 1 -- 172.21.15.88:0/3902319193 wait complete. 2024-09-16T01:36:29.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.955+0000 7fdbaea53640 1 Processor -- start 2024-09-16T01:36:29.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.955+0000 7fdbaea53640 1 -- start start 2024-09-16T01:36:29.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.955+0000 7fdbaea53640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdba8072530 0x7fdba81efe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:29.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.955+0000 7fdbaea53640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdba80b60c0 0x7fdba81f2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:29.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.956+0000 7fdbaea53640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdba815b480 0x7fdba81f1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:29.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.956+0000 7fdbaea53640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdba80cdb70 con 0x7fdba815b480 2024-09-16T01:36:29.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.956+0000 7fdbaea53640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdba80cd890 con 0x7fdba80b60c0 2024-09-16T01:36:29.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.956+0000 7fdbaea53640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdba815b860 con 0x7fdba8072530 2024-09-16T01:36:29.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.956+0000 7fdbacff0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdba815b480 0x7fdba81f1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:29.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.956+0000 7fdbacff0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdba815b480 0x7fdba81f1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:33176/0 (socket says 172.21.15.88:33176) 2024-09-16T01:36:29.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.956+0000 7fdbacff0640 1 -- 172.21.15.88:0/2340294344 learned_addr learned my addr 172.21.15.88:0/2340294344 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:29.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.956+0000 7fdba77fe640 1 --2- 172.21.15.88:0/2340294344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdba80b60c0 0x7fdba81f2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:29.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.956+0000 7fdba7fff640 1 --2- 172.21.15.88:0/2340294344 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdba8072530 0x7fdba81efe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:29.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.956+0000 7fdbacff0640 1 -- 172.21.15.88:0/2340294344 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdba8072530 msgr2=0x7fdba81efe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:29.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.956+0000 7fdbacff0640 1 --2- 172.21.15.88:0/2340294344 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdba8072530 0x7fdba81efe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:29.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.956+0000 7fdbacff0640 1 -- 172.21.15.88:0/2340294344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdba80b60c0 msgr2=0x7fdba81f2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:29.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.956+0000 7fdbacff0640 1 --2- 172.21.15.88:0/2340294344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdba80b60c0 0x7fdba81f2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:29.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.956+0000 7fdbacff0640 1 -- 172.21.15.88:0/2340294344 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba81f31e0 con 0x7fdba815b480 2024-09-16T01:36:29.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.956+0000 7fdbacff0640 1 --2- 172.21.15.88:0/2340294344 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdba815b480 0x7fdba81f1970 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fdba000c780 tx=0x7fdba000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:29.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.956+0000 7fdba4ff9640 1 -- 172.21.15.88:0/2340294344 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba0017070 con 0x7fdba815b480 2024-09-16T01:36:29.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.956+0000 7fdba4ff9640 1 -- 172.21.15.88:0/2340294344 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdba0003da0 con 0x7fdba815b480 2024-09-16T01:36:29.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.956+0000 7fdbaea53640 1 -- 172.21.15.88:0/2340294344 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdba81f3830 con 0x7fdba815b480 2024-09-16T01:36:29.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.956+0000 7fdba4ff9640 1 -- 172.21.15.88:0/2340294344 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba00052b0 con 0x7fdba815b480 2024-09-16T01:36:29.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.956+0000 7fdbaea53640 1 -- 172.21.15.88:0/2340294344 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdba81f3df0 con 0x7fdba815b480 2024-09-16T01:36:29.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.957+0000 7fdbaea53640 1 -- 172.21.15.88:0/2340294344 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdba8071ff0 con 0x7fdba815b480 2024-09-16T01:36:29.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.958+0000 7fdba4ff9640 1 -- 172.21.15.88:0/2340294344 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fdba001a070 con 0x7fdba815b480 2024-09-16T01:36:29.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.958+0000 7fdba4ff9640 1 --2- 172.21.15.88:0/2340294344 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdb9406c2b0 0x7fdb9406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:29.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.958+0000 7fdba4ff9640 1 -- 172.21.15.88:0/2340294344 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5881+0+0 (secure 0 0 0) 0x7fdba008ac00 con 0x7fdba815b480 2024-09-16T01:36:29.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.958+0000 7fdba7fff640 1 --2- 172.21.15.88:0/2340294344 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdb9406c2b0 0x7fdb9406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:29.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.958+0000 7fdba7fff640 1 --2- 172.21.15.88:0/2340294344 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdb9406c2b0 0x7fdb9406e770 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdba8157dc0 tx=0x7fdb98008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:29.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:29.962+0000 7fdba4ff9640 1 -- 172.21.15.88:0/2340294344 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fdba81f31e0 con 0x7fdba815b480 2024-09-16T01:36:30.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:30.129+0000 7fdbaea53640 1 -- 172.21.15.88:0/2340294344 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fdba806d790 con 0x7fdba815b480 2024-09-16T01:36:31.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.118+0000 7fdba4ff9640 1 -- 172.21.15.88:0/2340294344 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v48) ==== 181+0+0 (secure 0 0 0) 0x7fdba0059220 con 0x7fdba815b480 2024-09-16T01:36:31.119 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_metadata' removed 2024-09-16T01:36:31.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.121+0000 7fdbaea53640 1 -- 172.21.15.88:0/2340294344 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdb9406c2b0 msgr2=0x7fdb9406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:31.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.122+0000 7fdbaea53640 1 --2- 172.21.15.88:0/2340294344 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdb9406c2b0 0x7fdb9406e770 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdba8157dc0 tx=0x7fdb98008040 comp rx=0 tx=0).stop 2024-09-16T01:36:31.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.122+0000 7fdbaea53640 1 -- 172.21.15.88:0/2340294344 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdba815b480 msgr2=0x7fdba81f1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:31.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.122+0000 7fdbaea53640 1 --2- 172.21.15.88:0/2340294344 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdba815b480 0x7fdba81f1970 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fdba000c780 tx=0x7fdba000cc40 comp rx=0 tx=0).stop 2024-09-16T01:36:31.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.122+0000 7fdbaea53640 1 -- 172.21.15.88:0/2340294344 shutdown_connections 2024-09-16T01:36:31.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.122+0000 7fdbaea53640 1 --2- 172.21.15.88:0/2340294344 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdb9406c2b0 0x7fdb9406e770 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:31.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.122+0000 7fdbaea53640 1 --2- 172.21.15.88:0/2340294344 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdba815b480 0x7fdba81f1970 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:31.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.122+0000 7fdbaea53640 1 --2- 172.21.15.88:0/2340294344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdba80b60c0 0x7fdba81f2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:31.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.122+0000 7fdbaea53640 1 --2- 172.21.15.88:0/2340294344 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdba8072530 0x7fdba81efe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:31.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.122+0000 7fdbaea53640 1 -- 172.21.15.88:0/2340294344 >> 172.21.15.88:0/2340294344 conn(0x7fdba806d0f0 msgr2=0x7fdba80d0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:31.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.122+0000 7fdbaea53640 1 -- 172.21.15.88:0/2340294344 shutdown_connections 2024-09-16T01:36:31.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.122+0000 7fdbaea53640 1 -- 172.21.15.88:0/2340294344 wait complete. 2024-09-16T01:36:31.138 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-16T01:36:31.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.245+0000 7f35ef3fd640 1 Processor -- start 2024-09-16T01:36:31.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.245+0000 7f35ef3fd640 1 -- start start 2024-09-16T01:36:31.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.246+0000 7f35ef3fd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35e8147eb0 0x7f35e80d0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:31.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.246+0000 7f35ef3fd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f35e80b5560 0x7f35e806bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:31.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.246+0000 7f35ef3fd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f35e806c150 0x7f35e806f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:31.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.246+0000 7f35ef3fd640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f35e8148290 con 0x7f35e80b5560 2024-09-16T01:36:31.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.246+0000 7f35ef3fd640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f35e80b5d90 con 0x7f35e806c150 2024-09-16T01:36:31.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.246+0000 7f35ef3fd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f35e80b6070 con 0x7f35e8147eb0 2024-09-16T01:36:31.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.246+0000 7f35ec998640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f35e80b5560 0x7f35e806bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:31.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.246+0000 7f35ec998640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f35e80b5560 0x7f35e806bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42626/0 (socket says 172.21.15.88:42626) 2024-09-16T01:36:31.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.247+0000 7f35ec998640 1 -- 172.21.15.88:0/1849868324 learned_addr learned my addr 172.21.15.88:0/1849868324 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:31.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.247+0000 7f35ed99a640 1 --2- 172.21.15.88:0/1849868324 >> v2:172.21.15.154:3300/0 conn(0x7f35e806c150 0x7f35e806f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:31.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.247+0000 7f35ed199640 1 --2- 172.21.15.88:0/1849868324 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35e8147eb0 0x7f35e80d0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:31.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.247+0000 7f35ec998640 1 -- 172.21.15.88:0/1849868324 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35e8147eb0 msgr2=0x7f35e80d0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:31.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.247+0000 7f35ec998640 1 --2- 172.21.15.88:0/1849868324 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35e8147eb0 0x7f35e80d0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:31.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.247+0000 7f35ec998640 1 -- 172.21.15.88:0/1849868324 >> v2:172.21.15.154:3300/0 conn(0x7f35e806c150 msgr2=0x7f35e806f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:31.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.247+0000 7f35ec998640 1 --2- 172.21.15.88:0/1849868324 >> v2:172.21.15.154:3300/0 conn(0x7f35e806c150 0x7f35e806f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:31.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.247+0000 7f35ec998640 1 -- 172.21.15.88:0/1849868324 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e80b6350 con 0x7f35e80b5560 2024-09-16T01:36:31.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.247+0000 7f35ed99a640 1 --2- 172.21.15.88:0/1849868324 >> v2:172.21.15.154:3300/0 conn(0x7f35e806c150 0x7f35e806f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:31.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.247+0000 7f35ed199640 1 --2- 172.21.15.88:0/1849868324 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35e8147eb0 0x7f35e80d0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:31.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.247+0000 7f35ec998640 1 --2- 172.21.15.88:0/1849868324 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35e80b5560 0x7f35e806bc10 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f35e80cfb90 tx=0x7f35d802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=827cf2e8a86d8219 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:31.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.248+0000 7f35dffff640 1 -- 172.21.15.88:0/1849868324 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d8037070 con 0x7f35e80b5560 2024-09-16T01:36:31.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.248+0000 7f35dffff640 1 -- 172.21.15.88:0/1849868324 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35d8032440 con 0x7f35e80b5560 2024-09-16T01:36:31.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.248+0000 7f35dffff640 1 -- 172.21.15.88:0/1849868324 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d8032810 con 0x7f35e80b5560 2024-09-16T01:36:31.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.248+0000 7f35ef3fd640 1 -- 172.21.15.88:0/1849868324 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35e80b5560 msgr2=0x7f35e806bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:31.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.248+0000 7f35ef3fd640 1 --2- 172.21.15.88:0/1849868324 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35e80b5560 0x7f35e806bc10 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f35e80cfb90 tx=0x7f35d802a1e0 comp rx=0 tx=0).stop 2024-09-16T01:36:31.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.248+0000 7f35ef3fd640 1 -- 172.21.15.88:0/1849868324 shutdown_connections 2024-09-16T01:36:31.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.248+0000 7f35ef3fd640 1 --2- 172.21.15.88:0/1849868324 >> v2:172.21.15.154:3300/0 conn(0x7f35e806c150 0x7f35e806f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:31.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.248+0000 7f35ef3fd640 1 --2- 172.21.15.88:0/1849868324 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35e80b5560 0x7f35e806bc10 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:31.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.248+0000 7f35ef3fd640 1 --2- 172.21.15.88:0/1849868324 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35e8147eb0 0x7f35e80d0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:31.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.248+0000 7f35ef3fd640 1 -- 172.21.15.88:0/1849868324 >> 172.21.15.88:0/1849868324 conn(0x7f35e806e140 msgr2=0x7f35e806e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:31.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.248+0000 7f35ef3fd640 1 -- 172.21.15.88:0/1849868324 shutdown_connections 2024-09-16T01:36:31.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.248+0000 7f35ef3fd640 1 -- 172.21.15.88:0/1849868324 wait complete. 2024-09-16T01:36:31.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.249+0000 7f35ef3fd640 1 Processor -- start 2024-09-16T01:36:31.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.250+0000 7f35ef3fd640 1 -- start start 2024-09-16T01:36:31.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.250+0000 7f35ef3fd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35e806c150 0x7f35e81427b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:31.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.250+0000 7f35ef3fd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35e80b5560 0x7f35e815dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:31.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.250+0000 7f35ef3fd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35e8147eb0 0x7f35e81602a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:31.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.251+0000 7f35ef3fd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f35e80b59a0 con 0x7f35e80b5560 2024-09-16T01:36:31.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.251+0000 7f35ef3fd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f35e80b6910 con 0x7f35e806c150 2024-09-16T01:36:31.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.251+0000 7f35ef3fd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f35e80b5b20 con 0x7f35e8147eb0 2024-09-16T01:36:31.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.251+0000 7f35ec998640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35e80b5560 0x7f35e815dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:31.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.251+0000 7f35ed199640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35e806c150 0x7f35e81427b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:31.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.251+0000 7f35ec998640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35e80b5560 0x7f35e815dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42642/0 (socket says 172.21.15.88:42642) 2024-09-16T01:36:31.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.251+0000 7f35ec998640 1 -- 172.21.15.88:0/2472906733 learned_addr learned my addr 172.21.15.88:0/2472906733 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:31.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.251+0000 7f35ed99a640 1 --2- 172.21.15.88:0/2472906733 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35e8147eb0 0x7f35e81602a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:31.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.251+0000 7f35ec998640 1 -- 172.21.15.88:0/2472906733 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35e8147eb0 msgr2=0x7f35e81602a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:31.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.251+0000 7f35ec998640 1 --2- 172.21.15.88:0/2472906733 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35e8147eb0 0x7f35e81602a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:31.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.251+0000 7f35ec998640 1 -- 172.21.15.88:0/2472906733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35e806c150 msgr2=0x7f35e81427b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:31.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.251+0000 7f35ec998640 1 --2- 172.21.15.88:0/2472906733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35e806c150 0x7f35e81427b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:31.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.251+0000 7f35ec998640 1 -- 172.21.15.88:0/2472906733 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e8160da0 con 0x7f35e80b5560 2024-09-16T01:36:31.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.253+0000 7f35ec998640 1 --2- 172.21.15.88:0/2472906733 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35e80b5560 0x7f35e815dd20 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f35d802a220 tx=0x7f35d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:31.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.254+0000 7f35ddffb640 1 -- 172.21.15.88:0/2472906733 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d8037040 con 0x7f35e80b5560 2024-09-16T01:36:31.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.254+0000 7f35ddffb640 1 -- 172.21.15.88:0/2472906733 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35d8042070 con 0x7f35e80b5560 2024-09-16T01:36:31.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.254+0000 7f35ddffb640 1 -- 172.21.15.88:0/2472906733 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d8030bb0 con 0x7f35e80b5560 2024-09-16T01:36:31.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.254+0000 7f35ef3fd640 1 -- 172.21.15.88:0/2472906733 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35e8161080 con 0x7f35e80b5560 2024-09-16T01:36:31.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.254+0000 7f35ef3fd640 1 -- 172.21.15.88:0/2472906733 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35e813e7c0 con 0x7f35e80b5560 2024-09-16T01:36:31.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.255+0000 7f35ef3fd640 1 -- 172.21.15.88:0/2472906733 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35e813e9c0 con 0x7f35e80b5560 2024-09-16T01:36:31.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.256+0000 7f35ddffb640 1 -- 172.21.15.88:0/2472906733 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f35d8047460 con 0x7f35e80b5560 2024-09-16T01:36:31.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.256+0000 7f35ddffb640 1 --2- 172.21.15.88:0/2472906733 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f35cc06c300 0x7f35cc06e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:31.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.256+0000 7f35ddffb640 1 -- 172.21.15.88:0/2472906733 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5443+0+0 (secure 0 0 0) 0x7f35d80abd60 con 0x7f35e80b5560 2024-09-16T01:36:31.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.256+0000 7f35ed199640 1 --2- 172.21.15.88:0/2472906733 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f35cc06c300 0x7f35cc06e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:31.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.257+0000 7f35ed199640 1 --2- 172.21.15.88:0/2472906733 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f35cc06c300 0x7f35cc06e7c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f35e400aac0 tx=0x7f35e4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:31.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.261+0000 7f35ddffb640 1 -- 172.21.15.88:0/2472906733 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f35e813e9c0 con 0x7f35e80b5560 2024-09-16T01:36:31.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:31.425+0000 7f35ef3fd640 1 -- 172.21.15.88:0/2472906733 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f35e806bc10 con 0x7f35e80b5560 2024-09-16T01:36:32.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.119+0000 7f35ddffb640 1 -- 172.21.15.88:0/2472906733 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v49) ==== 169+0+0 (secure 0 0 0) 0x7f35d807a060 con 0x7f35e80b5560 2024-09-16T01:36:32.121 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_data' removed 2024-09-16T01:36:32.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.123+0000 7f35ef3fd640 1 -- 172.21.15.88:0/2472906733 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f35cc06c300 msgr2=0x7f35cc06e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:32.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.123+0000 7f35ef3fd640 1 --2- 172.21.15.88:0/2472906733 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f35cc06c300 0x7f35cc06e7c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f35e400aac0 tx=0x7f35e4008040 comp rx=0 tx=0).stop 2024-09-16T01:36:32.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.123+0000 7f35ef3fd640 1 -- 172.21.15.88:0/2472906733 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35e80b5560 msgr2=0x7f35e815dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:32.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.123+0000 7f35ef3fd640 1 --2- 172.21.15.88:0/2472906733 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35e80b5560 0x7f35e815dd20 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f35d802a220 tx=0x7f35d80046c0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.124+0000 7f35ef3fd640 1 -- 172.21.15.88:0/2472906733 shutdown_connections 2024-09-16T01:36:32.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.124+0000 7f35ef3fd640 1 --2- 172.21.15.88:0/2472906733 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f35cc06c300 0x7f35cc06e7c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.124+0000 7f35ef3fd640 1 --2- 172.21.15.88:0/2472906733 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f35e8147eb0 0x7f35e81602a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.124+0000 7f35ef3fd640 1 --2- 172.21.15.88:0/2472906733 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35e80b5560 0x7f35e815dd20 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.124+0000 7f35ef3fd640 1 --2- 172.21.15.88:0/2472906733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35e806c150 0x7f35e81427b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.124+0000 7f35ef3fd640 1 -- 172.21.15.88:0/2472906733 >> 172.21.15.88:0/2472906733 conn(0x7f35e806e140 msgr2=0x7f35e80b4ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:32.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.124+0000 7f35ef3fd640 1 -- 172.21.15.88:0/2472906733 shutdown_connections 2024-09-16T01:36:32.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.124+0000 7f35ef3fd640 1 -- 172.21.15.88:0/2472906733 wait complete. 2024-09-16T01:36:32.140 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-16T01:36:32.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.245+0000 7fbaa6a7b640 1 Processor -- start 2024-09-16T01:36:32.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.245+0000 7fbaa6a7b640 1 -- start start 2024-09-16T01:36:32.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.246+0000 7fbaa6a7b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbaa015a050 0x7fbaa015d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:32.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.246+0000 7fbaa6a7b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbaa015a430 con 0x7fbaa015ee90 2024-09-16T01:36:32.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.246+0000 7fbaa6a7b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbaa0160df0 con 0x7fbaa00731c0 2024-09-16T01:36:32.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.246+0000 7fbaa6a7b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fbaa015d9a0 con 0x7fbaa015a050 2024-09-16T01:36:32.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.246+0000 7fbaa4817640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbaa015a050 0x7fbaa015d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:32.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.246+0000 7fbaa5018640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fbaa015ee90 0x7fbaa01606e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60876/0 (socket says 172.21.15.88:60876) 2024-09-16T01:36:32.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.246+0000 7fbaa4817640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbaa015a050 0x7fbaa015d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42654/0 (socket says 172.21.15.88:42654) 2024-09-16T01:36:32.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.246+0000 7fbaa5018640 1 -- 172.21.15.88:0/967405653 learned_addr learned my addr 172.21.15.88:0/967405653 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:32.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.246+0000 7fbaa4817640 1 -- 172.21.15.88:0/967405653 >> v1:172.21.15.154:6789/0 conn(0x7fbaa00731c0 legacy=0x7fbaa015ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:32.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.246+0000 7fbaa4817640 1 -- 172.21.15.88:0/967405653 >> v1:172.21.15.88:6789/0 conn(0x7fbaa015ee90 legacy=0x7fbaa01606e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:32.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.246+0000 7fba977fe640 1 -- 172.21.15.88:0/967405653 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 337614472 0 0) 0x7fbaa015a430 con 0x7fbaa015ee90 2024-09-16T01:36:32.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.246+0000 7fbaa4817640 1 -- 172.21.15.88:0/967405653 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaa00cd0a0 con 0x7fbaa015a050 2024-09-16T01:36:32.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.247+0000 7fbaa4817640 1 --2- 172.21.15.88:0/967405653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaa015a050 0x7fbaa015d460 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fba9c0097a0 tx=0x7fba9c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5076e9ad9bf4adc9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:32.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.247+0000 7fba977fe640 1 -- 172.21.15.88:0/967405653 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c037070 con 0x7fbaa015a050 2024-09-16T01:36:32.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.247+0000 7fba977fe640 1 -- 172.21.15.88:0/967405653 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba9c02ae50 con 0x7fbaa015a050 2024-09-16T01:36:32.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.247+0000 7fba977fe640 1 -- 172.21.15.88:0/967405653 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c032660 con 0x7fbaa015a050 2024-09-16T01:36:32.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.248+0000 7fbaa6a7b640 1 -- 172.21.15.88:0/967405653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaa015a050 msgr2=0x7fbaa015d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:32.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.248+0000 7fbaa6a7b640 1 --2- 172.21.15.88:0/967405653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaa015a050 0x7fbaa015d460 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fba9c0097a0 tx=0x7fba9c02a140 comp rx=0 tx=0).stop 2024-09-16T01:36:32.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.248+0000 7fbaa6a7b640 1 -- 172.21.15.88:0/967405653 shutdown_connections 2024-09-16T01:36:32.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.248+0000 7fbaa6a7b640 1 --2- 172.21.15.88:0/967405653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaa015a050 0x7fbaa015d460 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.248+0000 7fbaa6a7b640 1 -- 172.21.15.88:0/967405653 >> 172.21.15.88:0/967405653 conn(0x7fbaa006d0f0 msgr2=0x7fbaa006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:32.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.248+0000 7fbaa6a7b640 1 -- 172.21.15.88:0/967405653 shutdown_connections 2024-09-16T01:36:32.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.248+0000 7fbaa6a7b640 1 -- 172.21.15.88:0/967405653 wait complete. 2024-09-16T01:36:32.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.249+0000 7fbaa6a7b640 1 Processor -- start 2024-09-16T01:36:32.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.249+0000 7fbaa6a7b640 1 -- start start 2024-09-16T01:36:32.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.250+0000 7fbaa6a7b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaa00731c0 0x7fbaa0140720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:32.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.250+0000 7fbaa6a7b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbaa015a050 0x7fbaa0153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:32.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.250+0000 7fbaa6a7b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbaa015ee90 0x7fbaa015b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:32.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.250+0000 7fbaa6a7b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa00cd940 con 0x7fbaa00731c0 2024-09-16T01:36:32.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.250+0000 7fbaa6a7b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa00cd660 con 0x7fbaa015ee90 2024-09-16T01:36:32.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.250+0000 7fbaa6a7b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbaa00cdc20 con 0x7fbaa015a050 2024-09-16T01:36:32.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.250+0000 7fbaa4817640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaa00731c0 0x7fbaa0140720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:32.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.250+0000 7fbaa4817640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaa00731c0 0x7fbaa0140720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42666/0 (socket says 172.21.15.88:42666) 2024-09-16T01:36:32.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.250+0000 7fbaa4817640 1 -- 172.21.15.88:0/3997638634 learned_addr learned my addr 172.21.15.88:0/3997638634 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:32.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.250+0000 7fbaa5018640 1 --2- 172.21.15.88:0/3997638634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbaa015ee90 0x7fbaa015b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:32.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.250+0000 7fbaa4817640 1 -- 172.21.15.88:0/3997638634 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbaa015a050 msgr2=0x7fbaa0153120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:32.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.250+0000 7fba97fff640 1 --2- 172.21.15.88:0/3997638634 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbaa015a050 0x7fbaa0153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:32.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.250+0000 7fbaa4817640 1 --2- 172.21.15.88:0/3997638634 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbaa015a050 0x7fbaa0153120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.250+0000 7fbaa4817640 1 -- 172.21.15.88:0/3997638634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbaa015ee90 msgr2=0x7fbaa015b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:32.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.250+0000 7fbaa4817640 1 --2- 172.21.15.88:0/3997638634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbaa015ee90 0x7fbaa015b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.250+0000 7fbaa4817640 1 -- 172.21.15.88:0/3997638634 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaa0155f90 con 0x7fbaa00731c0 2024-09-16T01:36:32.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.250+0000 7fbaa5018640 1 --2- 172.21.15.88:0/3997638634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbaa015ee90 0x7fbaa015b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:32.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.250+0000 7fbaa4817640 1 --2- 172.21.15.88:0/3997638634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaa00731c0 0x7fbaa0140720 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fba9c0097a0 tx=0x7fba9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:32.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.251+0000 7fba957fa640 1 -- 172.21.15.88:0/3997638634 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c037040 con 0x7fbaa00731c0 2024-09-16T01:36:32.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.251+0000 7fba957fa640 1 -- 172.21.15.88:0/3997638634 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba9c004030 con 0x7fbaa00731c0 2024-09-16T01:36:32.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.251+0000 7fbaa6a7b640 1 -- 172.21.15.88:0/3997638634 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaa01551d0 con 0x7fbaa00731c0 2024-09-16T01:36:32.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.251+0000 7fba957fa640 1 -- 172.21.15.88:0/3997638634 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c006e10 con 0x7fbaa00731c0 2024-09-16T01:36:32.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.251+0000 7fbaa6a7b640 1 -- 172.21.15.88:0/3997638634 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaa0140c60 con 0x7fbaa00731c0 2024-09-16T01:36:32.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.252+0000 7fbaa6a7b640 1 -- 172.21.15.88:0/3997638634 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaa01555c0 con 0x7fbaa00731c0 2024-09-16T01:36:32.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.252+0000 7fba957fa640 1 -- 172.21.15.88:0/3997638634 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fba9c03d650 con 0x7fbaa00731c0 2024-09-16T01:36:32.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.252+0000 7fba957fa640 1 --2- 172.21.15.88:0/3997638634 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba8c06c2b0 0x7fba8c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:32.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.253+0000 7fba957fa640 1 -- 172.21.15.88:0/3997638634 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5029+0+0 (secure 0 0 0) 0x7fba9c0ab970 con 0x7fbaa00731c0 2024-09-16T01:36:32.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.253+0000 7fba97fff640 1 --2- 172.21.15.88:0/3997638634 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba8c06c2b0 0x7fba8c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:32.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.253+0000 7fba97fff640 1 --2- 172.21.15.88:0/3997638634 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba8c06c2b0 0x7fba8c06e770 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbaa00cc9b0 tx=0x7fba90009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:32.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.257+0000 7fba957fa640 1 -- 172.21.15.88:0/3997638634 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fba9c042020 con 0x7fbaa00731c0 2024-09-16T01:36:32.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.422+0000 7fbaa6a7b640 1 -- 172.21.15.88:0/3997638634 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fbaa01553a0 con 0x7fbaa00731c0 2024-09-16T01:36:32.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.425+0000 7fba957fa640 1 -- 172.21.15.88:0/3997638634 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v16) ==== 104+0+0 (secure 0 0 0) 0x7fba9c07a410 con 0x7fbaa00731c0 2024-09-16T01:36:32.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.427+0000 7fbaa6a7b640 1 -- 172.21.15.88:0/3997638634 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba8c06c2b0 msgr2=0x7fba8c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:32.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.427+0000 7fbaa6a7b640 1 --2- 172.21.15.88:0/3997638634 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba8c06c2b0 0x7fba8c06e770 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbaa00cc9b0 tx=0x7fba90009040 comp rx=0 tx=0).stop 2024-09-16T01:36:32.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.428+0000 7fbaa6a7b640 1 -- 172.21.15.88:0/3997638634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaa00731c0 msgr2=0x7fbaa0140720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:32.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.428+0000 7fbaa6a7b640 1 --2- 172.21.15.88:0/3997638634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaa00731c0 0x7fbaa0140720 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fba9c0097a0 tx=0x7fba9c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.428+0000 7fbaa6a7b640 1 -- 172.21.15.88:0/3997638634 shutdown_connections 2024-09-16T01:36:32.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.428+0000 7fbaa6a7b640 1 --2- 172.21.15.88:0/3997638634 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba8c06c2b0 0x7fba8c06e770 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.428+0000 7fbaa6a7b640 1 --2- 172.21.15.88:0/3997638634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbaa015ee90 0x7fbaa015b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.428+0000 7fbaa6a7b640 1 --2- 172.21.15.88:0/3997638634 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbaa015a050 0x7fbaa0153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.428+0000 7fbaa6a7b640 1 --2- 172.21.15.88:0/3997638634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbaa00731c0 0x7fbaa0140720 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.428+0000 7fbaa6a7b640 1 -- 172.21.15.88:0/3997638634 >> 172.21.15.88:0/3997638634 conn(0x7fbaa006d0f0 msgr2=0x7fbaa00ce2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:32.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.428+0000 7fbaa6a7b640 1 -- 172.21.15.88:0/3997638634 shutdown_connections 2024-09-16T01:36:32.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.428+0000 7fbaa6a7b640 1 -- 172.21.15.88:0/3997638634 wait complete. 2024-09-16T01:36:32.440 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_inject_rename_corrupt_dentry_first 2024-09-16T01:36:32.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.538+0000 7f57a7ea5640 1 Processor -- start 2024-09-16T01:36:32.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.539+0000 7f57a7ea5640 1 -- start start 2024-09-16T01:36:32.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.539+0000 7f57a7ea5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f57a00b6290 0x7f57a0071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:32.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.539+0000 7f57a7ea5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57a0072530 0x7f57a015bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:32.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.539+0000 7f57a7ea5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57a00ccff0 con 0x7f57a015b480 2024-09-16T01:36:32.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.539+0000 7f57a7ea5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f57a00cd2d0 con 0x7f57a00b6290 2024-09-16T01:36:32.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.539+0000 7f57a7ea5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f57a00cd5b0 con 0x7f57a0072530 2024-09-16T01:36:32.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.540+0000 7f57a5c41640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f57a015b480 0x7f57a00b5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60880/0 (socket says 172.21.15.88:60880) 2024-09-16T01:36:32.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.540+0000 7f57a5c41640 1 -- 172.21.15.88:0/1103022974 learned_addr learned my addr 172.21.15.88:0/1103022974 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:32.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.540+0000 7f57a4c3f640 1 -- 172.21.15.88:0/1103022974 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1363025729 0 0) 0x7f57a00ccff0 con 0x7f57a015b480 2024-09-16T01:36:32.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.540+0000 7f57a6442640 1 --2- 172.21.15.88:0/1103022974 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57a0072530 0x7f57a015bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:32.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.540+0000 7f57a5440640 1 --2- 172.21.15.88:0/1103022974 >> v2:172.21.15.154:3300/0 conn(0x7f57a00b6290 0x7f57a0071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:32.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.540+0000 7f57a4c3f640 1 -- 172.21.15.88:0/1103022974 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f578c003520 con 0x7f57a015b480 2024-09-16T01:36:32.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.540+0000 7f57a4c3f640 1 -- 172.21.15.88:0/1103022974 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1815177 0 0) 0x7f578c003520 con 0x7f57a015b480 2024-09-16T01:36:32.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.540+0000 7f57a4c3f640 1 -- 172.21.15.88:0/1103022974 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57a0072530 msgr2=0x7f57a015bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:32.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.540+0000 7f57a4c3f640 1 --2- 172.21.15.88:0/1103022974 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57a0072530 0x7f57a015bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.540+0000 7f57a4c3f640 1 -- 172.21.15.88:0/1103022974 >> v2:172.21.15.154:3300/0 conn(0x7f57a00b6290 msgr2=0x7f57a0071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:32.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.540+0000 7f57a4c3f640 1 --2- 172.21.15.88:0/1103022974 >> v2:172.21.15.154:3300/0 conn(0x7f57a00b6290 0x7f57a0071ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.540+0000 7f57a4c3f640 1 -- 172.21.15.88:0/1103022974 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57a00cd890 con 0x7f57a015b480 2024-09-16T01:36:32.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.540+0000 7f57a4c3f640 1 -- 172.21.15.88:0/1103022974 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f579c001670 con 0x7f57a015b480 2024-09-16T01:36:32.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.540+0000 7f57a6442640 1 --2- 172.21.15.88:0/1103022974 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57a0072530 0x7f57a015bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:32.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.540+0000 7f57a4c3f640 1 -- 172.21.15.88:0/1103022974 >> v1:172.21.15.88:6789/0 conn(0x7f57a015b480 legacy=0x7f57a00b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:32.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.540+0000 7f57a4c3f640 1 --2- 172.21.15.88:0/1103022974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f578c0039b0 0x7f578c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:32.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.540+0000 7f57a4c3f640 1 --2- 172.21.15.88:0/1103022974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f578c004800 0x7f578c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:32.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.540+0000 7f57a4c3f640 1 --2- 172.21.15.88:0/1103022974 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57a0072530 0x7f578c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:32.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.540+0000 7f57a4c3f640 1 -- 172.21.15.88:0/1103022974 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f57a00cd2d0 con 0x7f578c0039b0 2024-09-16T01:36:32.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.540+0000 7f57a4c3f640 1 -- 172.21.15.88:0/1103022974 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f57a00cd5b0 con 0x7f578c004800 2024-09-16T01:36:32.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.540+0000 7f57a4c3f640 1 -- 172.21.15.88:0/1103022974 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f578c00ba40 con 0x7f57a0072530 2024-09-16T01:36:32.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.540+0000 7f57a5c41640 1 --2- 172.21.15.88:0/1103022974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f578c0039b0 0x7f578c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:32.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.540+0000 7f57a5440640 1 --2- 172.21.15.88:0/1103022974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f578c004800 0x7f578c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:32.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.540+0000 7f57a6442640 1 --2- 172.21.15.88:0/1103022974 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57a0072530 0x7f578c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:32.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.541+0000 7f57a5c41640 1 -- 172.21.15.88:0/1103022974 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57a0072530 msgr2=0x7f578c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:32.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.541+0000 7f57a5c41640 1 --2- 172.21.15.88:0/1103022974 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57a0072530 0x7f578c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.541+0000 7f57a5c41640 1 -- 172.21.15.88:0/1103022974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f578c004800 msgr2=0x7f578c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:32.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.541+0000 7f57a5c41640 1 --2- 172.21.15.88:0/1103022974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f578c004800 0x7f578c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.541+0000 7f57a5c41640 1 -- 172.21.15.88:0/1103022974 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f578c003520 con 0x7f578c0039b0 2024-09-16T01:36:32.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.541+0000 7f57a6442640 1 --2- 172.21.15.88:0/1103022974 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57a0072530 0x7f578c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:32.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.541+0000 7f57a5c41640 1 --2- 172.21.15.88:0/1103022974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f578c0039b0 0x7f578c003dd0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f579c00c290 tx=0x7f578c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5743cda4ebfdbeb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:32.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.541+0000 7f57a4c3f640 1 -- 172.21.15.88:0/1103022974 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579c007d50 con 0x7f578c0039b0 2024-09-16T01:36:32.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.541+0000 7f57a4c3f640 1 -- 172.21.15.88:0/1103022974 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f579c005af0 con 0x7f578c0039b0 2024-09-16T01:36:32.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.541+0000 7f57a4c3f640 1 -- 172.21.15.88:0/1103022974 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579c007070 con 0x7f578c0039b0 2024-09-16T01:36:32.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.542+0000 7f57a7ea5640 1 -- 172.21.15.88:0/1103022974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f578c0039b0 msgr2=0x7f578c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:32.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.542+0000 7f57a7ea5640 1 --2- 172.21.15.88:0/1103022974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f578c0039b0 0x7f578c003dd0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f579c00c290 tx=0x7f578c005830 comp rx=0 tx=0).stop 2024-09-16T01:36:32.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.542+0000 7f57a6442640 1 -- 172.21.15.88:0/1103022974 reap_dead start 2024-09-16T01:36:32.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.542+0000 7f57a7ea5640 1 -- 172.21.15.88:0/1103022974 shutdown_connections 2024-09-16T01:36:32.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.542+0000 7f57a7ea5640 1 -- 172.21.15.88:0/1103022974 >> 172.21.15.88:0/1103022974 conn(0x7f57a006d0f0 msgr2=0x7f57a006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:32.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.542+0000 7f57a7ea5640 1 -- 172.21.15.88:0/1103022974 shutdown_connections 2024-09-16T01:36:32.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.542+0000 7f57a7ea5640 1 -- 172.21.15.88:0/1103022974 wait complete. 2024-09-16T01:36:32.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.543+0000 7f57a7ea5640 1 Processor -- start 2024-09-16T01:36:32.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.543+0000 7f57a7ea5640 1 -- start start 2024-09-16T01:36:32.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.544+0000 7f57a7ea5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f578c0039b0 0x7f57a0152970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:32.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.544+0000 7f57a7ea5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57a0155420 0x7f57a01558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:32.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.544+0000 7f57a7ea5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57a01e3d80 0x7f57a01e4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:32.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.544+0000 7f57a7ea5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f57a015b9e0 con 0x7f57a0155420 2024-09-16T01:36:32.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.544+0000 7f57a7ea5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f57a015b860 con 0x7f57a01e3d80 2024-09-16T01:36:32.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.544+0000 7f57a7ea5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f57a015bc60 con 0x7f578c0039b0 2024-09-16T01:36:32.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.544+0000 7f57a5440640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57a0155420 0x7f57a01558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:32.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.544+0000 7f57a5440640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57a0155420 0x7f57a01558a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42686/0 (socket says 172.21.15.88:42686) 2024-09-16T01:36:32.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.544+0000 7f57a5440640 1 -- 172.21.15.88:0/3988619869 learned_addr learned my addr 172.21.15.88:0/3988619869 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:32.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.544+0000 7f57a6442640 1 --2- 172.21.15.88:0/3988619869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57a01e3d80 0x7f57a01e4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:32.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.544+0000 7f57a5c41640 1 --2- 172.21.15.88:0/3988619869 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f578c0039b0 0x7f57a0152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:32.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.544+0000 7f57a5440640 1 -- 172.21.15.88:0/3988619869 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f578c0039b0 msgr2=0x7f57a0152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:32.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.544+0000 7f57a5440640 1 --2- 172.21.15.88:0/3988619869 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f578c0039b0 0x7f57a0152970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.544+0000 7f57a5440640 1 -- 172.21.15.88:0/3988619869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57a01e3d80 msgr2=0x7f57a01e4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:32.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.545+0000 7f57a5440640 1 --2- 172.21.15.88:0/3988619869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57a01e3d80 0x7f57a01e4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.545+0000 7f57a5440640 1 -- 172.21.15.88:0/3988619869 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57a014f520 con 0x7f57a0155420 2024-09-16T01:36:32.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.545+0000 7f57a6442640 1 --2- 172.21.15.88:0/3988619869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57a01e3d80 0x7f57a01e4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:32.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.545+0000 7f57a5c41640 1 --2- 172.21.15.88:0/3988619869 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f578c0039b0 0x7f57a0152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:32.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.545+0000 7f57a5440640 1 --2- 172.21.15.88:0/3988619869 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57a0155420 0x7f57a01558a0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f5794011980 tx=0x7f5794011e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:32.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.545+0000 7f57927fc640 1 -- 172.21.15.88:0/3988619869 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579400faf0 con 0x7f57a0155420 2024-09-16T01:36:32.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.545+0000 7f57927fc640 1 -- 172.21.15.88:0/3988619869 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f579400fc90 con 0x7f57a0155420 2024-09-16T01:36:32.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.545+0000 7f57927fc640 1 -- 172.21.15.88:0/3988619869 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5794014440 con 0x7f57a0155420 2024-09-16T01:36:32.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.545+0000 7f57a7ea5640 1 -- 172.21.15.88:0/3988619869 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57a0155f70 con 0x7f57a0155420 2024-09-16T01:36:32.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.545+0000 7f57a7ea5640 1 -- 172.21.15.88:0/3988619869 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57a013dd90 con 0x7f57a0155420 2024-09-16T01:36:32.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.546+0000 7f57a7ea5640 1 -- 172.21.15.88:0/3988619869 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57a0071ff0 con 0x7f57a0155420 2024-09-16T01:36:32.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.547+0000 7f57927fc640 1 -- 172.21.15.88:0/3988619869 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5794012040 con 0x7f57a0155420 2024-09-16T01:36:32.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.547+0000 7f57927fc640 1 --2- 172.21.15.88:0/3988619869 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f578406c2e0 0x7f578406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:32.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.547+0000 7f57a5c41640 1 --2- 172.21.15.88:0/3988619869 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f578406c2e0 0x7f578406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:32.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.547+0000 7f57927fc640 1 -- 172.21.15.88:0/3988619869 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5029+0+0 (secure 0 0 0) 0x7f579408b6e0 con 0x7f57a0155420 2024-09-16T01:36:32.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.548+0000 7f57a5c41640 1 --2- 172.21.15.88:0/3988619869 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f578406c2e0 0x7f578406e7a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f579c033000 tx=0x7f579c009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:32.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.552+0000 7f57927fc640 1 -- 172.21.15.88:0/3988619869 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5794019030 con 0x7f57a0155420 2024-09-16T01:36:32.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.718+0000 7f57a7ea5640 1 -- 172.21.15.88:0/3988619869 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"} v 0) -- 0x7f57a0157300 con 0x7f57a0155420 2024-09-16T01:36:32.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.719+0000 7f57927fc640 1 -- 172.21.15.88:0/3988619869 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"}]=0 v16) ==== 121+0+0 (secure 0 0 0) 0x7f579405a180 con 0x7f57a0155420 2024-09-16T01:36:32.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.721+0000 7f57a7ea5640 1 -- 172.21.15.88:0/3988619869 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f578406c2e0 msgr2=0x7f578406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:32.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.721+0000 7f57a7ea5640 1 --2- 172.21.15.88:0/3988619869 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f578406c2e0 0x7f578406e7a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f579c033000 tx=0x7f579c009030 comp rx=0 tx=0).stop 2024-09-16T01:36:32.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.721+0000 7f57a7ea5640 1 -- 172.21.15.88:0/3988619869 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57a0155420 msgr2=0x7f57a01558a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:32.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.721+0000 7f57a7ea5640 1 --2- 172.21.15.88:0/3988619869 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57a0155420 0x7f57a01558a0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f5794011980 tx=0x7f5794011e40 comp rx=0 tx=0).stop 2024-09-16T01:36:32.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.721+0000 7f57a7ea5640 1 -- 172.21.15.88:0/3988619869 shutdown_connections 2024-09-16T01:36:32.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.721+0000 7f57a7ea5640 1 --2- 172.21.15.88:0/3988619869 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f578406c2e0 0x7f578406e7a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.721+0000 7f57a7ea5640 1 --2- 172.21.15.88:0/3988619869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57a01e3d80 0x7f57a01e4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.721+0000 7f57a7ea5640 1 --2- 172.21.15.88:0/3988619869 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57a0155420 0x7f57a01558a0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.721+0000 7f57a7ea5640 1 --2- 172.21.15.88:0/3988619869 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f578c0039b0 0x7f57a0152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.721+0000 7f57a7ea5640 1 -- 172.21.15.88:0/3988619869 >> 172.21.15.88:0/3988619869 conn(0x7f57a006d0f0 msgr2=0x7f57a0157820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:32.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.721+0000 7f57a7ea5640 1 -- 172.21.15.88:0/3988619869 shutdown_connections 2024-09-16T01:36:32.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.721+0000 7f57a7ea5640 1 -- 172.21.15.88:0/3988619869 wait complete. 2024-09-16T01:36:32.733 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_abort_on_newly_corrupt_dentry 2024-09-16T01:36:32.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.833+0000 7f6300d13640 1 Processor -- start 2024-09-16T01:36:32.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.833+0000 7f6300d13640 1 -- start start 2024-09-16T01:36:32.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f6300d13640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62fc15b480 0x7f62fc0b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:32.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f6300d13640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f62fc072700 0x7f62fc15bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:32.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f6300d13640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62fc0ccff0 con 0x7f62fc0b60c0 2024-09-16T01:36:32.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f6300d13640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f62fc0b6500 con 0x7f62fc072700 2024-09-16T01:36:32.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f6300d13640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f62fc0cd2d0 con 0x7f62fc15b480 2024-09-16T01:36:32.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f62f1d9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f62fc0b60c0 0x7f62fc071ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60892/0 (socket says 172.21.15.88:60892) 2024-09-16T01:36:32.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f62f1d9b640 1 -- 172.21.15.88:0/2907004947 learned_addr learned my addr 172.21.15.88:0/2907004947 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:32.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f62fa59c640 1 --2- 172.21.15.88:0/2907004947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62fc15b480 0x7f62fc0b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:32.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f62fad9d640 1 --2- 172.21.15.88:0/2907004947 >> v2:172.21.15.154:3300/0 conn(0x7f62fc072700 0x7f62fc15bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:32.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f62f9d9b640 1 -- 172.21.15.88:0/2907004947 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2006832199 0 0) 0x7f62fc0ccff0 con 0x7f62fc0b60c0 2024-09-16T01:36:32.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f62f9d9b640 1 -- 172.21.15.88:0/2907004947 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62e8003520 con 0x7f62fc0b60c0 2024-09-16T01:36:32.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f62f9d9b640 1 -- 172.21.15.88:0/2907004947 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2708582050 0 0) 0x7f62e8003520 con 0x7f62fc0b60c0 2024-09-16T01:36:32.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f62f9d9b640 1 -- 172.21.15.88:0/2907004947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62fc15b480 msgr2=0x7f62fc0b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:32.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f62f9d9b640 1 --2- 172.21.15.88:0/2907004947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62fc15b480 0x7f62fc0b5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f62f9d9b640 1 -- 172.21.15.88:0/2907004947 >> v2:172.21.15.154:3300/0 conn(0x7f62fc072700 msgr2=0x7f62fc15bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:32.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f62f9d9b640 1 --2- 172.21.15.88:0/2907004947 >> v2:172.21.15.154:3300/0 conn(0x7f62fc072700 0x7f62fc15bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f62f9d9b640 1 -- 172.21.15.88:0/2907004947 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62fc0cd5b0 con 0x7f62fc0b60c0 2024-09-16T01:36:32.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f62f9d9b640 1 -- 172.21.15.88:0/2907004947 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f62e4001670 con 0x7f62fc0b60c0 2024-09-16T01:36:32.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f62f9d9b640 1 -- 172.21.15.88:0/2907004947 >> v1:172.21.15.88:6789/0 conn(0x7f62fc0b60c0 legacy=0x7f62fc071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:32.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f62f9d9b640 1 --2- 172.21.15.88:0/2907004947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62e80039b0 0x7f62e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:32.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f62f9d9b640 1 --2- 172.21.15.88:0/2907004947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62e8004800 0x7f62e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:32.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f62f9d9b640 1 --2- 172.21.15.88:0/2907004947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62fc15b480 0x7f62e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:32.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f62f9d9b640 1 -- 172.21.15.88:0/2907004947 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f62fc0b6500 con 0x7f62e80039b0 2024-09-16T01:36:32.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f62f9d9b640 1 -- 172.21.15.88:0/2907004947 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f62fc0cd2d0 con 0x7f62e8004800 2024-09-16T01:36:32.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f62f9d9b640 1 -- 172.21.15.88:0/2907004947 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f62e800b9c0 con 0x7f62fc15b480 2024-09-16T01:36:32.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.834+0000 7f62fa59c640 1 --2- 172.21.15.88:0/2907004947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62e80039b0 0x7f62e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:32.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.835+0000 7f62f1d9b640 1 --2- 172.21.15.88:0/2907004947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62e8004800 0x7f62e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:32.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.835+0000 7f62fad9d640 1 --2- 172.21.15.88:0/2907004947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62fc15b480 0x7f62e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:32.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.835+0000 7f62fa59c640 1 -- 172.21.15.88:0/2907004947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62fc15b480 msgr2=0x7f62e80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:32.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.835+0000 7f62fa59c640 1 --2- 172.21.15.88:0/2907004947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62fc15b480 0x7f62e80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.835+0000 7f62fa59c640 1 -- 172.21.15.88:0/2907004947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62e8004800 msgr2=0x7f62e8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:32.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.835+0000 7f62fa59c640 1 --2- 172.21.15.88:0/2907004947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62e8004800 0x7f62e8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.835+0000 7f62fa59c640 1 -- 172.21.15.88:0/2907004947 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62e8003520 con 0x7f62e80039b0 2024-09-16T01:36:32.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.835+0000 7f62f1d9b640 1 --2- 172.21.15.88:0/2907004947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62e8004800 0x7f62e8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:32.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.835+0000 7f62fa59c640 1 --2- 172.21.15.88:0/2907004947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62e80039b0 0x7f62e8003dd0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f62ec009af0 tx=0x7f62e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5276d003a2cb46b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:32.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.836+0000 7f62f9d9b640 1 -- 172.21.15.88:0/2907004947 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ec036070 con 0x7f62e80039b0 2024-09-16T01:36:32.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.836+0000 7f62f9d9b640 1 -- 172.21.15.88:0/2907004947 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62ec031400 con 0x7f62e80039b0 2024-09-16T01:36:32.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.836+0000 7f62f9d9b640 1 -- 172.21.15.88:0/2907004947 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ec009690 con 0x7f62e80039b0 2024-09-16T01:36:32.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.836+0000 7f6300d13640 1 -- 172.21.15.88:0/2907004947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62e80039b0 msgr2=0x7f62e8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:32.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.836+0000 7f6300d13640 1 --2- 172.21.15.88:0/2907004947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62e80039b0 0x7f62e8003dd0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f62ec009af0 tx=0x7f62e8005830 comp rx=0 tx=0).stop 2024-09-16T01:36:32.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.836+0000 7f62fad9d640 1 -- 172.21.15.88:0/2907004947 reap_dead start 2024-09-16T01:36:32.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.836+0000 7f6300d13640 1 -- 172.21.15.88:0/2907004947 shutdown_connections 2024-09-16T01:36:32.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.836+0000 7f6300d13640 1 -- 172.21.15.88:0/2907004947 >> 172.21.15.88:0/2907004947 conn(0x7f62fc06d0f0 msgr2=0x7f62fc06d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:32.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.836+0000 7f6300d13640 1 -- 172.21.15.88:0/2907004947 shutdown_connections 2024-09-16T01:36:32.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.836+0000 7f6300d13640 1 -- 172.21.15.88:0/2907004947 wait complete. 2024-09-16T01:36:32.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.838+0000 7f6300d13640 1 Processor -- start 2024-09-16T01:36:32.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.838+0000 7f6300d13640 1 -- start start 2024-09-16T01:36:32.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.838+0000 7f6300d13640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62e80039b0 0x7f62fc1452c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:32.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.838+0000 7f6300d13640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62fc146fb0 0x7f62fc0d0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:32.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.838+0000 7f6300d13640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62fc147430 0x7f62fc15c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:32.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.838+0000 7f6300d13640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f62fc15b860 con 0x7f62fc146fb0 2024-09-16T01:36:32.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.838+0000 7f6300d13640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f62fc0cdb70 con 0x7f62fc147430 2024-09-16T01:36:32.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.838+0000 7f6300d13640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f62fc15b9e0 con 0x7f62e80039b0 2024-09-16T01:36:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.838+0000 7f62f1d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62fc146fb0 0x7f62fc0d0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.839+0000 7f62f1d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62fc146fb0 0x7f62fc0d0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42698/0 (socket says 172.21.15.88:42698) 2024-09-16T01:36:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.839+0000 7f62f1d9b640 1 -- 172.21.15.88:0/2206171680 learned_addr learned my addr 172.21.15.88:0/2206171680 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.839+0000 7f62fad9d640 1 --2- 172.21.15.88:0/2206171680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62fc147430 0x7f62fc15c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.839+0000 7f62fa59c640 1 --2- 172.21.15.88:0/2206171680 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62e80039b0 0x7f62fc1452c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.839+0000 7f62f1d9b640 1 -- 172.21.15.88:0/2206171680 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62e80039b0 msgr2=0x7f62fc1452c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.839+0000 7f62f1d9b640 1 --2- 172.21.15.88:0/2206171680 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62e80039b0 0x7f62fc1452c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.839+0000 7f62f1d9b640 1 -- 172.21.15.88:0/2206171680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62fc147430 msgr2=0x7f62fc15c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.839+0000 7f62f1d9b640 1 --2- 172.21.15.88:0/2206171680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62fc147430 0x7f62fc15c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.839+0000 7f62f1d9b640 1 -- 172.21.15.88:0/2206171680 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62fc1461f0 con 0x7f62fc146fb0 2024-09-16T01:36:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.839+0000 7f62f1d9b640 1 --2- 172.21.15.88:0/2206171680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62fc146fb0 0x7f62fc0d0b10 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f62e400e820 tx=0x7f62e400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:32.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.839+0000 7f62f37fe640 1 -- 172.21.15.88:0/2206171680 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e400c870 con 0x7f62fc146fb0 2024-09-16T01:36:32.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.839+0000 7f62f37fe640 1 -- 172.21.15.88:0/2206171680 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62e4018070 con 0x7f62fc146fb0 2024-09-16T01:36:32.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.839+0000 7f6300d13640 1 -- 172.21.15.88:0/2206171680 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62fc1464d0 con 0x7f62fc146fb0 2024-09-16T01:36:32.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.839+0000 7f62f37fe640 1 -- 172.21.15.88:0/2206171680 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e400cc10 con 0x7f62fc146fb0 2024-09-16T01:36:32.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.839+0000 7f6300d13640 1 -- 172.21.15.88:0/2206171680 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62fc146a90 con 0x7f62fc146fb0 2024-09-16T01:36:32.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.840+0000 7f6300d13640 1 -- 172.21.15.88:0/2206171680 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62fc076170 con 0x7f62fc146fb0 2024-09-16T01:36:32.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.841+0000 7f62f37fe640 1 -- 172.21.15.88:0/2206171680 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f62e401d460 con 0x7f62fc146fb0 2024-09-16T01:36:32.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.841+0000 7f62f37fe640 1 --2- 172.21.15.88:0/2206171680 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f62dc06c2b0 0x7f62dc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:32.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.841+0000 7f62f37fe640 1 -- 172.21.15.88:0/2206171680 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5029+0+0 (secure 0 0 0) 0x7f62e408aef0 con 0x7f62fc146fb0 2024-09-16T01:36:32.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.841+0000 7f62fa59c640 1 --2- 172.21.15.88:0/2206171680 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f62dc06c2b0 0x7f62dc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:32.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.842+0000 7f62fa59c640 1 --2- 172.21.15.88:0/2206171680 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f62dc06c2b0 0x7f62dc06e770 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f62ec033000 tx=0x7f62ec008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:32.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:32.846+0000 7f62f37fe640 1 -- 172.21.15.88:0/2206171680 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f62fc076170 con 0x7f62fc146fb0 2024-09-16T01:36:33.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.020+0000 7f6300d13640 1 -- 172.21.15.88:0/2206171680 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_abort_on_newly_corrupt_dentry"} v 0) -- 0x7f62fc0b5b80 con 0x7f62fc146fb0 2024-09-16T01:36:33.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.023+0000 7f62f37fe640 1 -- 172.21.15.88:0/2206171680 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_abort_on_newly_corrupt_dentry"}]=0 v17) ==== 116+0+0 (secure 0 0 0) 0x7f62e4059340 con 0x7f62fc146fb0 2024-09-16T01:36:33.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.025+0000 7f6300d13640 1 -- 172.21.15.88:0/2206171680 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f62dc06c2b0 msgr2=0x7f62dc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:33.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.025+0000 7f6300d13640 1 --2- 172.21.15.88:0/2206171680 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f62dc06c2b0 0x7f62dc06e770 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f62ec033000 tx=0x7f62ec008030 comp rx=0 tx=0).stop 2024-09-16T01:36:33.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.025+0000 7f6300d13640 1 -- 172.21.15.88:0/2206171680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62fc146fb0 msgr2=0x7f62fc0d0b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:33.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.025+0000 7f6300d13640 1 --2- 172.21.15.88:0/2206171680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62fc146fb0 0x7f62fc0d0b10 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f62e400e820 tx=0x7f62e400ece0 comp rx=0 tx=0).stop 2024-09-16T01:36:33.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.025+0000 7f6300d13640 1 -- 172.21.15.88:0/2206171680 shutdown_connections 2024-09-16T01:36:33.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.025+0000 7f6300d13640 1 --2- 172.21.15.88:0/2206171680 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f62dc06c2b0 0x7f62dc06e770 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:33.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.025+0000 7f6300d13640 1 --2- 172.21.15.88:0/2206171680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62fc147430 0x7f62fc15c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:33.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.025+0000 7f6300d13640 1 --2- 172.21.15.88:0/2206171680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62fc146fb0 0x7f62fc0d0b10 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:33.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.025+0000 7f6300d13640 1 --2- 172.21.15.88:0/2206171680 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62e80039b0 0x7f62fc1452c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:33.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.025+0000 7f6300d13640 1 -- 172.21.15.88:0/2206171680 >> 172.21.15.88:0/2206171680 conn(0x7f62fc06d0f0 msgr2=0x7f62fc0b4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:33.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.025+0000 7f6300d13640 1 -- 172.21.15.88:0/2206171680 shutdown_connections 2024-09-16T01:36:33.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.025+0000 7f6300d13640 1 -- 172.21.15.88:0/2206171680 wait complete. 2024-09-16T01:36:33.038 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_existing' 2024-09-16T01:36:33.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.135+0000 7ff865198640 1 Processor -- start 2024-09-16T01:36:33.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.135+0000 7ff865198640 1 -- start start 2024-09-16T01:36:33.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.136+0000 7ff865198640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff8600d0fe0 0x7ff86006cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:33.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.136+0000 7ff865198640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff86006d260 0x7ff8600b5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:33.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.136+0000 7ff865198640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff86015f130 con 0x7ff86015b460 2024-09-16T01:36:33.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.136+0000 7ff865198640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff86006d6b0 con 0x7ff86006d260 2024-09-16T01:36:33.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.136+0000 7ff865198640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff86015eee0 con 0x7ff8600d0fe0 2024-09-16T01:36:33.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.136+0000 7ff85f59e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff86006d260 0x7ff8600b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:33.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.136+0000 7ff85f59e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff86006d260 0x7ff8600b5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42706/0 (socket says 172.21.15.88:42706) 2024-09-16T01:36:33.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.136+0000 7ff85f59e640 1 -- 172.21.15.88:0/706477362 learned_addr learned my addr 172.21.15.88:0/706477362 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:33.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.137+0000 7ff85f59e640 1 -- 172.21.15.88:0/706477362 >> v2:172.21.15.154:3300/0 conn(0x7ff8600d0fe0 msgr2=0x7ff86006cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:33.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.137+0000 7ff85f59e640 1 --2- 172.21.15.88:0/706477362 >> v2:172.21.15.154:3300/0 conn(0x7ff8600d0fe0 0x7ff86006cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:33.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.137+0000 7ff85f59e640 1 -- 172.21.15.88:0/706477362 >> v1:172.21.15.154:6789/0 conn(0x7ff86015b460 legacy=0x7ff8600b44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:33.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.137+0000 7ff85f59e640 1 -- 172.21.15.88:0/706477362 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff86015f410 con 0x7ff86006d260 2024-09-16T01:36:33.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.137+0000 7ff85f59e640 1 --2- 172.21.15.88:0/706477362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff86006d260 0x7ff8600b5080 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ff8600b7610 tx=0x7ff85802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dff39f3066fe2ba server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:33.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.137+0000 7ff85dd9b640 1 -- 172.21.15.88:0/706477362 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff858036070 con 0x7ff86006d260 2024-09-16T01:36:33.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.137+0000 7ff85dd9b640 1 -- 172.21.15.88:0/706477362 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff858031600 con 0x7ff86006d260 2024-09-16T01:36:33.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.137+0000 7ff85dd9b640 1 -- 172.21.15.88:0/706477362 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8580319d0 con 0x7ff86006d260 2024-09-16T01:36:33.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.138+0000 7ff865198640 1 -- 172.21.15.88:0/706477362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff86006d260 msgr2=0x7ff8600b5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:33.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.138+0000 7ff865198640 1 --2- 172.21.15.88:0/706477362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff86006d260 0x7ff8600b5080 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ff8600b7610 tx=0x7ff85802a6f0 comp rx=0 tx=0).stop 2024-09-16T01:36:33.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.138+0000 7ff865198640 1 -- 172.21.15.88:0/706477362 shutdown_connections 2024-09-16T01:36:33.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.138+0000 7ff865198640 1 --2- 172.21.15.88:0/706477362 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff86006d260 0x7ff8600b5080 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:33.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.138+0000 7ff865198640 1 --2- 172.21.15.88:0/706477362 >> v2:172.21.15.154:3300/0 conn(0x7ff8600d0fe0 0x7ff86006cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:33.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.138+0000 7ff865198640 1 -- 172.21.15.88:0/706477362 >> 172.21.15.88:0/706477362 conn(0x7ff8600b6ec0 msgr2=0x7ff8600b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:33.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.138+0000 7ff865198640 1 -- 172.21.15.88:0/706477362 shutdown_connections 2024-09-16T01:36:33.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.138+0000 7ff865198640 1 -- 172.21.15.88:0/706477362 wait complete. 2024-09-16T01:36:33.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.140+0000 7ff865198640 1 Processor -- start 2024-09-16T01:36:33.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.140+0000 7ff865198640 1 -- start start 2024-09-16T01:36:33.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.140+0000 7ff865198640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff86006d260 0x7ff860154690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:33.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.140+0000 7ff865198640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8600d0fe0 0x7ff860145370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:33.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.140+0000 7ff865198640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff86015b460 0x7ff86014d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:33.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.140+0000 7ff865198640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff8600b55c0 con 0x7ff86006d260 2024-09-16T01:36:33.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.140+0000 7ff865198640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff86015f9d0 con 0x7ff8600d0fe0 2024-09-16T01:36:33.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.140+0000 7ff865198640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff8600b5740 con 0x7ff86015b460 2024-09-16T01:36:33.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.140+0000 7ff85ed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff86006d260 0x7ff860154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:33.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.140+0000 7ff85ed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff86006d260 0x7ff860154690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42708/0 (socket says 172.21.15.88:42708) 2024-09-16T01:36:33.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.140+0000 7ff85ed9d640 1 -- 172.21.15.88:0/914502868 learned_addr learned my addr 172.21.15.88:0/914502868 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:33.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.141+0000 7ff85ed9d640 1 -- 172.21.15.88:0/914502868 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff86015b460 msgr2=0x7ff86014d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:33.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.141+0000 7ff85ed9d640 1 --2- 172.21.15.88:0/914502868 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff86015b460 0x7ff86014d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:33.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.141+0000 7ff85ed9d640 1 -- 172.21.15.88:0/914502868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8600d0fe0 msgr2=0x7ff860145370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:33.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.141+0000 7ff85ed9d640 1 --2- 172.21.15.88:0/914502868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8600d0fe0 0x7ff860145370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:33.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.141+0000 7ff85ed9d640 1 -- 172.21.15.88:0/914502868 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8601e7140 con 0x7ff86006d260 2024-09-16T01:36:33.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.141+0000 7ff85ed9d640 1 --2- 172.21.15.88:0/914502868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff86006d260 0x7ff860154690 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7ff85000b3e0 tx=0x7ff85000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:33.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.141+0000 7ff8477fe640 1 -- 172.21.15.88:0/914502868 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff850015020 con 0x7ff86006d260 2024-09-16T01:36:33.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.141+0000 7ff8477fe640 1 -- 172.21.15.88:0/914502868 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff850011850 con 0x7ff86006d260 2024-09-16T01:36:33.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.141+0000 7ff865198640 1 -- 172.21.15.88:0/914502868 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff860158f80 con 0x7ff86006d260 2024-09-16T01:36:33.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.141+0000 7ff8477fe640 1 -- 172.21.15.88:0/914502868 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff850011c40 con 0x7ff86006d260 2024-09-16T01:36:33.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.141+0000 7ff865198640 1 -- 172.21.15.88:0/914502868 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8601e76c0 con 0x7ff86006d260 2024-09-16T01:36:33.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.142+0000 7ff865198640 1 -- 172.21.15.88:0/914502868 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8600b4c20 con 0x7ff86006d260 2024-09-16T01:36:33.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.142+0000 7ff8477fe640 1 -- 172.21.15.88:0/914502868 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff850018020 con 0x7ff86006d260 2024-09-16T01:36:33.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.143+0000 7ff8477fe640 1 --2- 172.21.15.88:0/914502868 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff84c06c2e0 0x7ff84c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:33.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.143+0000 7ff8477fe640 1 -- 172.21.15.88:0/914502868 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5029+0+0 (secure 0 0 0) 0x7ff85008b0c0 con 0x7ff86006d260 2024-09-16T01:36:33.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.143+0000 7ff85e59c640 1 --2- 172.21.15.88:0/914502868 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff84c06c2e0 0x7ff84c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:33.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.143+0000 7ff85e59c640 1 --2- 172.21.15.88:0/914502868 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff84c06c2e0 0x7ff84c06e7a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff860157990 tx=0x7ff854006c90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:33.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.147+0000 7ff8477fe640 1 -- 172.21.15.88:0/914502868 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff85001f020 con 0x7ff86006d260 2024-09-16T01:36:33.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:33.377+0000 7ff865198640 1 -- 172.21.15.88:0/914502868 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_existing"]} v 0) -- 0x7ff8600b5060 con 0x7ff86006d260 2024-09-16T01:36:34.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.131+0000 7ff8477fe640 1 -- 172.21.15.88:0/914502868 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_existing"]}]=0 v135) ==== 141+0+0 (secure 0 0 0) 0x7ff8500172c0 con 0x7ff86006d260 2024-09-16T01:36:34.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.133+0000 7ff865198640 1 -- 172.21.15.88:0/914502868 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff84c06c2e0 msgr2=0x7ff84c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:34.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.134+0000 7ff865198640 1 --2- 172.21.15.88:0/914502868 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff84c06c2e0 0x7ff84c06e7a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff860157990 tx=0x7ff854006c90 comp rx=0 tx=0).stop 2024-09-16T01:36:34.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.134+0000 7ff865198640 1 -- 172.21.15.88:0/914502868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff86006d260 msgr2=0x7ff860154690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:34.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.134+0000 7ff865198640 1 --2- 172.21.15.88:0/914502868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff86006d260 0x7ff860154690 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7ff85000b3e0 tx=0x7ff85000b8a0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.134+0000 7ff865198640 1 -- 172.21.15.88:0/914502868 shutdown_connections 2024-09-16T01:36:34.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.134+0000 7ff865198640 1 --2- 172.21.15.88:0/914502868 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff84c06c2e0 0x7ff84c06e7a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.134+0000 7ff865198640 1 --2- 172.21.15.88:0/914502868 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff86015b460 0x7ff86014d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.134+0000 7ff865198640 1 --2- 172.21.15.88:0/914502868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8600d0fe0 0x7ff860145370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.134+0000 7ff865198640 1 --2- 172.21.15.88:0/914502868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff86006d260 0x7ff860154690 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.134+0000 7ff865198640 1 -- 172.21.15.88:0/914502868 >> 172.21.15.88:0/914502868 conn(0x7ff8600b6ec0 msgr2=0x7ff8600b5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:34.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.134+0000 7ff865198640 1 -- 172.21.15.88:0/914502868 shutdown_connections 2024-09-16T01:36:34.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.134+0000 7ff865198640 1 -- 172.21.15.88:0/914502868 wait complete. 2024-09-16T01:36:34.149 INFO:tasks.cephfs_test_runner:That the MDS won't abort when the dentry is already known to be damaged. ... ok 2024-09-16T01:36:34.149 INFO:tasks.cephfs_test_runner:Starting test: test_dentry_first_precommit (tasks.cephfs.test_damage.TestDamage) That the MDS won't write a dentry with new damage to CDentry::first 2024-09-16T01:36:34.149 INFO:tasks.cephfs_test_runner:test_dentry_first_precommit (tasks.cephfs.test_damage.TestDamage) 2024-09-16T01:36:34.149 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:36:34.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.252+0000 7f022d427640 1 Processor -- start 2024-09-16T01:36:34.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.253+0000 7f022d427640 1 -- start start 2024-09-16T01:36:34.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.253+0000 7f022d427640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f022815fad0 0x7f022806c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:34.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.253+0000 7f022d427640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f022806e910 0x7f02280d0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:34.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.253+0000 7f022d427640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02280cf450 con 0x7f022806cb60 2024-09-16T01:36:34.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.253+0000 7f022d427640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f022806d070 con 0x7f022815fad0 2024-09-16T01:36:34.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.253+0000 7f022d427640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f02280cfd80 con 0x7f022806e910 2024-09-16T01:36:34.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.254+0000 7f0226ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f022815fad0 0x7f022806c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:34.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.254+0000 7f02267fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f022806cb60 0x7f022806e200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60908/0 (socket says 172.21.15.88:60908) 2024-09-16T01:36:34.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.254+0000 7f02267fc640 1 -- 172.21.15.88:0/2066329882 learned_addr learned my addr 172.21.15.88:0/2066329882 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:34.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.254+0000 7f0226ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f022815fad0 0x7f022806c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42724/0 (socket says 172.21.15.88:42724) 2024-09-16T01:36:34.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.254+0000 7f02277fe640 1 --2- 172.21.15.88:0/2066329882 >> v2:172.21.15.154:3300/0 conn(0x7f022806e910 0x7f02280d0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:34.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.254+0000 7f0226ffd640 1 -- 172.21.15.88:0/2066329882 >> v2:172.21.15.154:3300/0 conn(0x7f022806e910 msgr2=0x7f02280d0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:34.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.254+0000 7f0226ffd640 1 --2- 172.21.15.88:0/2066329882 >> v2:172.21.15.154:3300/0 conn(0x7f022806e910 0x7f02280d0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.254+0000 7f0226ffd640 1 -- 172.21.15.88:0/2066329882 >> v1:172.21.15.88:6789/0 conn(0x7f022806cb60 legacy=0x7f022806e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:34.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.254+0000 7f0225ffb640 1 -- 172.21.15.88:0/2066329882 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3183507226 0 0) 0x7f02280cf450 con 0x7f022806cb60 2024-09-16T01:36:34.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.254+0000 7f0226ffd640 1 -- 172.21.15.88:0/2066329882 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02280cf730 con 0x7f022815fad0 2024-09-16T01:36:34.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.254+0000 7f02277fe640 1 --2- 172.21.15.88:0/2066329882 >> v2:172.21.15.154:3300/0 conn(0x7f022806e910 0x7f02280d0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:34.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.255+0000 7f0226ffd640 1 --2- 172.21.15.88:0/2066329882 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f022815fad0 0x7f022806c620 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f02280b0f90 tx=0x7f021802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4442249213d694d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:34.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.255+0000 7f0225ffb640 1 -- 172.21.15.88:0/2066329882 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0218007740 con 0x7f022815fad0 2024-09-16T01:36:34.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.255+0000 7f0225ffb640 1 -- 172.21.15.88:0/2066329882 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02180078e0 con 0x7f022815fad0 2024-09-16T01:36:34.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.255+0000 7f0225ffb640 1 -- 172.21.15.88:0/2066329882 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0218007cb0 con 0x7f022815fad0 2024-09-16T01:36:34.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.255+0000 7f022d427640 1 -- 172.21.15.88:0/2066329882 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f022815fad0 msgr2=0x7f022806c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:34.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.255+0000 7f022d427640 1 --2- 172.21.15.88:0/2066329882 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f022815fad0 0x7f022806c620 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f02280b0f90 tx=0x7f021802a680 comp rx=0 tx=0).stop 2024-09-16T01:36:34.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.255+0000 7f022d427640 1 -- 172.21.15.88:0/2066329882 shutdown_connections 2024-09-16T01:36:34.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.255+0000 7f022d427640 1 --2- 172.21.15.88:0/2066329882 >> v2:172.21.15.154:3300/0 conn(0x7f022806e910 0x7f02280d0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.255+0000 7f022d427640 1 --2- 172.21.15.88:0/2066329882 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f022815fad0 0x7f022806c620 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.255+0000 7f022d427640 1 -- 172.21.15.88:0/2066329882 >> 172.21.15.88:0/2066329882 conn(0x7f02280b6ec0 msgr2=0x7f02280b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:34.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.255+0000 7f022d427640 1 -- 172.21.15.88:0/2066329882 shutdown_connections 2024-09-16T01:36:34.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.255+0000 7f022d427640 1 -- 172.21.15.88:0/2066329882 wait complete. 2024-09-16T01:36:34.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.257+0000 7f022d427640 1 Processor -- start 2024-09-16T01:36:34.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.257+0000 7f022d427640 1 -- start start 2024-09-16T01:36:34.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.257+0000 7f022d427640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f022806cb60 0x7f022815d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:34.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.257+0000 7f022d427640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f022806e910 0x7f022813fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:34.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.257+0000 7f022d427640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f022815fad0 0x7f0228156930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:34.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.257+0000 7f022d427640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f022806d7a0 con 0x7f022806e910 2024-09-16T01:36:34.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.257+0000 7f022d427640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f022806d620 con 0x7f022815fad0 2024-09-16T01:36:34.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.257+0000 7f022d427640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0228071f80 con 0x7f022806cb60 2024-09-16T01:36:34.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.257+0000 7f02267fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f022806e910 0x7f022813fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:34.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.258+0000 7f02267fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f022806e910 0x7f022813fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42738/0 (socket says 172.21.15.88:42738) 2024-09-16T01:36:34.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.258+0000 7f02267fc640 1 -- 172.21.15.88:0/1538034354 learned_addr learned my addr 172.21.15.88:0/1538034354 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:34.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.258+0000 7f02267fc640 1 -- 172.21.15.88:0/1538034354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f022806cb60 msgr2=0x7f022815d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:34.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.258+0000 7f02267fc640 1 --2- 172.21.15.88:0/1538034354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f022806cb60 0x7f022815d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.258+0000 7f02267fc640 1 -- 172.21.15.88:0/1538034354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f022815fad0 msgr2=0x7f0228156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:34.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.258+0000 7f02267fc640 1 --2- 172.21.15.88:0/1538034354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f022815fad0 0x7f0228156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.258+0000 7f02267fc640 1 -- 172.21.15.88:0/1538034354 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0228152300 con 0x7f022806e910 2024-09-16T01:36:34.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.258+0000 7f02267fc640 1 --2- 172.21.15.88:0/1538034354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f022806e910 0x7f022813fff0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f021000c4e0 tx=0x7f021000c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:34.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.258+0000 7f021f7fe640 1 -- 172.21.15.88:0/1538034354 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0210018070 con 0x7f022806e910 2024-09-16T01:36:34.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.258+0000 7f021f7fe640 1 -- 172.21.15.88:0/1538034354 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0210007500 con 0x7f022806e910 2024-09-16T01:36:34.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.258+0000 7f021f7fe640 1 -- 172.21.15.88:0/1538034354 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0210004a20 con 0x7f022806e910 2024-09-16T01:36:34.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.258+0000 7f022d427640 1 -- 172.21.15.88:0/1538034354 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f022815be50 con 0x7f022806e910 2024-09-16T01:36:34.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.258+0000 7f022d427640 1 -- 172.21.15.88:0/1538034354 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0228151cb0 con 0x7f022806e910 2024-09-16T01:36:34.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.259+0000 7f022d427640 1 -- 172.21.15.88:0/1538034354 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02280cf450 con 0x7f022806e910 2024-09-16T01:36:34.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.260+0000 7f021f7fe640 1 -- 172.21.15.88:0/1538034354 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f021001b070 con 0x7f022806e910 2024-09-16T01:36:34.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.260+0000 7f021f7fe640 1 --2- 172.21.15.88:0/1538034354 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f020806c2b0 0x7f020806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:34.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.260+0000 7f021f7fe640 1 -- 172.21.15.88:0/1538034354 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5029+0+0 (secure 0 0 0) 0x7f021008a4a0 con 0x7f022806e910 2024-09-16T01:36:34.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.260+0000 7f0226ffd640 1 --2- 172.21.15.88:0/1538034354 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f020806c2b0 0x7f020806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:34.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.261+0000 7f0226ffd640 1 --2- 172.21.15.88:0/1538034354 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f020806c2b0 0x7f020806e770 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0218034000 tx=0x7f021802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:34.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.265+0000 7f021f7fe640 1 -- 172.21.15.88:0/1538034354 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0210059000 con 0x7f022806e910 2024-09-16T01:36:34.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.433+0000 7f022d427640 1 -- 172.21.15.88:0/1538034354 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f02280b1be0 con 0x7f022806e910 2024-09-16T01:36:34.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.434+0000 7f021f7fe640 1 -- 172.21.15.88:0/1538034354 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) ==== 74+0+13138 (secure 0 0 0) 0x7f0210058900 con 0x7f022806e910 2024-09-16T01:36:34.435 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:34.435 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":49,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:36:32.118803+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:6836/2700280540":"2024-09-17T01:36:28.789211+0000","172.21.15.88:6834/2700280540":"2024-09-17T01:36:28.789211+0000","172.21.15.88:6836/1610810484":"2024-09-17T01:36:23.795661+0000","172.21.15.88:6834/1610810484":"2024-09-17T01:36:23.795661+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:36:34.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.436+0000 7f022d427640 1 -- 172.21.15.88:0/1538034354 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f020806c2b0 msgr2=0x7f020806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:34.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.436+0000 7f022d427640 1 --2- 172.21.15.88:0/1538034354 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f020806c2b0 0x7f020806e770 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0218034000 tx=0x7f021802e000 comp rx=0 tx=0).stop 2024-09-16T01:36:34.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.436+0000 7f022d427640 1 -- 172.21.15.88:0/1538034354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f022806e910 msgr2=0x7f022813fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:34.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.436+0000 7f022d427640 1 --2- 172.21.15.88:0/1538034354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f022806e910 0x7f022813fff0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f021000c4e0 tx=0x7f021000c9a0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.436+0000 7f022d427640 1 -- 172.21.15.88:0/1538034354 shutdown_connections 2024-09-16T01:36:34.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.436+0000 7f022d427640 1 --2- 172.21.15.88:0/1538034354 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f020806c2b0 0x7f020806e770 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.436+0000 7f022d427640 1 --2- 172.21.15.88:0/1538034354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f022815fad0 0x7f0228156930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.436+0000 7f022d427640 1 --2- 172.21.15.88:0/1538034354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f022806e910 0x7f022813fff0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.436+0000 7f022d427640 1 --2- 172.21.15.88:0/1538034354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f022806cb60 0x7f022815d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.436+0000 7f022d427640 1 -- 172.21.15.88:0/1538034354 >> 172.21.15.88:0/1538034354 conn(0x7f02280b6ec0 msgr2=0x7f02280d0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:34.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.436+0000 7f022d427640 1 -- 172.21.15.88:0/1538034354 shutdown_connections 2024-09-16T01:36:34.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.436+0000 7f022d427640 1 -- 172.21.15.88:0/1538034354 wait complete. 2024-09-16T01:36:34.448 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T01:34:10.547178+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T01:36:34.449 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T01:36:34.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.548+0000 7ff4c8f2a640 1 Processor -- start 2024-09-16T01:36:34.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.548+0000 7ff4c8f2a640 1 -- start start 2024-09-16T01:36:34.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.549+0000 7ff4c8f2a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff4c4155880 0x7ff4c415f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:34.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.549+0000 7ff4c8f2a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff4c415dc30 0x7ff4c40b51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:34.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.549+0000 7ff4c8f2a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4c415bb60 con 0x7ff4c40b4d60 2024-09-16T01:36:34.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.549+0000 7ff4c8f2a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff4c4155c60 con 0x7ff4c415dc30 2024-09-16T01:36:34.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.549+0000 7ff4c8f2a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff4c415fcb0 con 0x7ff4c4155880 2024-09-16T01:36:34.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.549+0000 7ff4c2d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff4c415dc30 0x7ff4c40b51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:34.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.549+0000 7ff4c1d9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff4c40b4d60 0x7ff4c406f630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60916/0 (socket says 172.21.15.88:60916) 2024-09-16T01:36:34.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.549+0000 7ff4c1d9b640 1 -- 172.21.15.88:0/689182509 learned_addr learned my addr 172.21.15.88:0/689182509 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:34.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.549+0000 7ff4c2d9d640 1 --2- 172.21.15.88:0/689182509 >> v2:172.21.15.88:3300/0 conn(0x7ff4c415dc30 0x7ff4c40b51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42754/0 (socket says 172.21.15.88:42754) 2024-09-16T01:36:34.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.550+0000 7ff4c2d9d640 1 -- 172.21.15.88:0/689182509 >> v2:172.21.15.154:3300/0 conn(0x7ff4c4155880 msgr2=0x7ff4c415f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:34.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.550+0000 7ff4c259c640 1 --2- 172.21.15.88:0/689182509 >> v2:172.21.15.154:3300/0 conn(0x7ff4c4155880 0x7ff4c415f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:34.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.550+0000 7ff4c159a640 1 -- 172.21.15.88:0/689182509 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 185381197 0 0) 0x7ff4c415bb60 con 0x7ff4c40b4d60 2024-09-16T01:36:34.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.550+0000 7ff4c2d9d640 1 --2- 172.21.15.88:0/689182509 >> v2:172.21.15.154:3300/0 conn(0x7ff4c4155880 0x7ff4c415f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.550+0000 7ff4c2d9d640 1 -- 172.21.15.88:0/689182509 >> v1:172.21.15.88:6789/0 conn(0x7ff4c40b4d60 legacy=0x7ff4c406f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:34.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.550+0000 7ff4c2d9d640 1 -- 172.21.15.88:0/689182509 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c415be40 con 0x7ff4c415dc30 2024-09-16T01:36:34.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.550+0000 7ff4c259c640 1 --2- 172.21.15.88:0/689182509 >> v2:172.21.15.154:3300/0 conn(0x7ff4c4155880 0x7ff4c415f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:34.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.550+0000 7ff4c2d9d640 1 --2- 172.21.15.88:0/689182509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c415dc30 0x7ff4c40b51f0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff4c40b7560 tx=0x7ff4bc02c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f5e78f2e935be42 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:34.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.551+0000 7ff4c159a640 1 -- 172.21.15.88:0/689182509 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc032770 con 0x7ff4c415dc30 2024-09-16T01:36:34.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.551+0000 7ff4c159a640 1 -- 172.21.15.88:0/689182509 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4bc038070 con 0x7ff4c415dc30 2024-09-16T01:36:34.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.551+0000 7ff4c159a640 1 -- 172.21.15.88:0/689182509 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc032b30 con 0x7ff4c415dc30 2024-09-16T01:36:34.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.551+0000 7ff4c8f2a640 1 -- 172.21.15.88:0/689182509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c415dc30 msgr2=0x7ff4c40b51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:34.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.551+0000 7ff4c8f2a640 1 --2- 172.21.15.88:0/689182509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c415dc30 0x7ff4c40b51f0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff4c40b7560 tx=0x7ff4bc02c870 comp rx=0 tx=0).stop 2024-09-16T01:36:34.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.551+0000 7ff4c8f2a640 1 -- 172.21.15.88:0/689182509 shutdown_connections 2024-09-16T01:36:34.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.551+0000 7ff4c8f2a640 1 --2- 172.21.15.88:0/689182509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c415dc30 0x7ff4c40b51f0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.551+0000 7ff4c8f2a640 1 --2- 172.21.15.88:0/689182509 >> v2:172.21.15.154:3300/0 conn(0x7ff4c4155880 0x7ff4c415f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.551+0000 7ff4c8f2a640 1 -- 172.21.15.88:0/689182509 >> 172.21.15.88:0/689182509 conn(0x7ff4c40d1390 msgr2=0x7ff4c40d17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:34.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.551+0000 7ff4c8f2a640 1 -- 172.21.15.88:0/689182509 shutdown_connections 2024-09-16T01:36:34.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.551+0000 7ff4c8f2a640 1 -- 172.21.15.88:0/689182509 wait complete. 2024-09-16T01:36:34.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.553+0000 7ff4c8f2a640 1 Processor -- start 2024-09-16T01:36:34.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.553+0000 7ff4c8f2a640 1 -- start start 2024-09-16T01:36:34.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.553+0000 7ff4c8f2a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c40b4d60 0x7ff4c40738c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:34.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.553+0000 7ff4c8f2a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4c4155880 0x7ff4c40b0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:34.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.553+0000 7ff4c8f2a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4c415dc30 0x7ff4c40af490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:34.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.553+0000 7ff4c8f2a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c415c6e0 con 0x7ff4c40b4d60 2024-09-16T01:36:34.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.553+0000 7ff4c8f2a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c415c400 con 0x7ff4c415dc30 2024-09-16T01:36:34.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.553+0000 7ff4c8f2a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff4c415b770 con 0x7ff4c4155880 2024-09-16T01:36:34.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.553+0000 7ff4c259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c40b4d60 0x7ff4c40738c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:34.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.553+0000 7ff4c259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c40b4d60 0x7ff4c40738c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42760/0 (socket says 172.21.15.88:42760) 2024-09-16T01:36:34.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.553+0000 7ff4c259c640 1 -- 172.21.15.88:0/3119150946 learned_addr learned my addr 172.21.15.88:0/3119150946 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:34.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.554+0000 7ff4c2d9d640 1 --2- 172.21.15.88:0/3119150946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4c415dc30 0x7ff4c40af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:34.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.554+0000 7ff4c1d9b640 1 --2- 172.21.15.88:0/3119150946 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4c4155880 0x7ff4c40b0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:34.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.554+0000 7ff4c259c640 1 -- 172.21.15.88:0/3119150946 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4c4155880 msgr2=0x7ff4c40b0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:34.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.554+0000 7ff4c259c640 1 --2- 172.21.15.88:0/3119150946 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4c4155880 0x7ff4c40b0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.554+0000 7ff4c259c640 1 -- 172.21.15.88:0/3119150946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4c415dc30 msgr2=0x7ff4c40af490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:34.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.554+0000 7ff4c259c640 1 --2- 172.21.15.88:0/3119150946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4c415dc30 0x7ff4c40af490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.554+0000 7ff4c259c640 1 -- 172.21.15.88:0/3119150946 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c40b1950 con 0x7ff4c40b4d60 2024-09-16T01:36:34.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.554+0000 7ff4c2d9d640 1 --2- 172.21.15.88:0/3119150946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4c415dc30 0x7ff4c40af490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:34.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.554+0000 7ff4c1d9b640 1 --2- 172.21.15.88:0/3119150946 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4c4155880 0x7ff4c40b0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:34.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.554+0000 7ff4c259c640 1 --2- 172.21.15.88:0/3119150946 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c40b4d60 0x7ff4c40738c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ff4b400cf90 tx=0x7ff4b400a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:34.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.554+0000 7ff4baffd640 1 -- 172.21.15.88:0/3119150946 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b401a070 con 0x7ff4c40b4d60 2024-09-16T01:36:34.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.554+0000 7ff4baffd640 1 -- 172.21.15.88:0/3119150946 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b400d840 con 0x7ff4c40b4d60 2024-09-16T01:36:34.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.554+0000 7ff4baffd640 1 -- 172.21.15.88:0/3119150946 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b400dc10 con 0x7ff4c40b4d60 2024-09-16T01:36:34.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.554+0000 7ff4c8f2a640 1 -- 172.21.15.88:0/3119150946 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4c40b1c30 con 0x7ff4c40b4d60 2024-09-16T01:36:34.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.554+0000 7ff4c8f2a640 1 -- 172.21.15.88:0/3119150946 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4c40b21f0 con 0x7ff4c40b4d60 2024-09-16T01:36:34.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.555+0000 7ff4c8f2a640 1 -- 172.21.15.88:0/3119150946 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4c40cd8a0 con 0x7ff4c40b4d60 2024-09-16T01:36:34.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.555+0000 7ff4baffd640 1 -- 172.21.15.88:0/3119150946 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff4b401d070 con 0x7ff4c40b4d60 2024-09-16T01:36:34.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.556+0000 7ff4baffd640 1 --2- 172.21.15.88:0/3119150946 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff4a406c2b0 0x7ff4a406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:34.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.556+0000 7ff4baffd640 1 -- 172.21.15.88:0/3119150946 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5029+0+0 (secure 0 0 0) 0x7ff4b408b8d0 con 0x7ff4c40b4d60 2024-09-16T01:36:34.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.556+0000 7ff4c1d9b640 1 --2- 172.21.15.88:0/3119150946 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff4a406c2b0 0x7ff4a406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:34.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.556+0000 7ff4c1d9b640 1 --2- 172.21.15.88:0/3119150946 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff4a406c2b0 0x7ff4a406e770 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac000f60 tx=0x7ff4ac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:34.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.560+0000 7ff4baffd640 1 -- 172.21.15.88:0/3119150946 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff4b4059d20 con 0x7ff4c40b4d60 2024-09-16T01:36:34.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.724+0000 7ff4c8f2a640 1 -- 172.21.15.88:0/3119150946 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff4c415e180 con 0x7ff4c40b4d60 2024-09-16T01:36:34.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.724+0000 7ff4baffd640 1 -- 172.21.15.88:0/3119150946 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v49) ==== 93+0+10 (secure 0 0 0) 0x7ff4b4060070 con 0x7ff4c40b4d60 2024-09-16T01:36:34.726 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2024-09-16T01:36:34.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.726+0000 7ff4c8f2a640 1 -- 172.21.15.88:0/3119150946 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff4a406c2b0 msgr2=0x7ff4a406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:34.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.726+0000 7ff4c8f2a640 1 --2- 172.21.15.88:0/3119150946 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff4a406c2b0 0x7ff4a406e770 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac000f60 tx=0x7ff4ac009040 comp rx=0 tx=0).stop 2024-09-16T01:36:34.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.726+0000 7ff4c8f2a640 1 -- 172.21.15.88:0/3119150946 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c40b4d60 msgr2=0x7ff4c40738c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:34.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.726+0000 7ff4c8f2a640 1 --2- 172.21.15.88:0/3119150946 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c40b4d60 0x7ff4c40738c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ff4b400cf90 tx=0x7ff4b400a7d0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.727+0000 7ff4c8f2a640 1 -- 172.21.15.88:0/3119150946 shutdown_connections 2024-09-16T01:36:34.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.727+0000 7ff4c8f2a640 1 --2- 172.21.15.88:0/3119150946 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff4a406c2b0 0x7ff4a406e770 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.727+0000 7ff4c8f2a640 1 --2- 172.21.15.88:0/3119150946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4c415dc30 0x7ff4c40af490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.727+0000 7ff4c8f2a640 1 --2- 172.21.15.88:0/3119150946 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4c4155880 0x7ff4c40b0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.727+0000 7ff4c8f2a640 1 --2- 172.21.15.88:0/3119150946 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4c40b4d60 0x7ff4c40738c0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.727+0000 7ff4c8f2a640 1 -- 172.21.15.88:0/3119150946 >> 172.21.15.88:0/3119150946 conn(0x7ff4c40d1390 msgr2=0x7ff4c406fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:34.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.727+0000 7ff4c8f2a640 1 -- 172.21.15.88:0/3119150946 shutdown_connections 2024-09-16T01:36:34.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.727+0000 7ff4c8f2a640 1 -- 172.21.15.88:0/3119150946 wait complete. 2024-09-16T01:36:34.739 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_precommit' 2024-09-16T01:36:34.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.837+0000 7f1a078a6640 1 Processor -- start 2024-09-16T01:36:34.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.837+0000 7f1a078a6640 1 -- start start 2024-09-16T01:36:34.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.838+0000 7f1a078a6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a000cf6b0 0x7f1a001629f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:34.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.838+0000 7f1a078a6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1a00162f30 0x7f1a000b7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:34.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.838+0000 7f1a078a6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a000cfc20 con 0x7f1a000b63d0 2024-09-16T01:36:34.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.838+0000 7f1a078a6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f1a000d40c0 con 0x7f1a00162f30 2024-09-16T01:36:34.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.838+0000 7f1a078a6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1a000d2ed0 con 0x7f1a000cf6b0 2024-09-16T01:36:34.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.838+0000 7f1a05e43640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1a00162f30 0x7f1a000b7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:34.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.838+0000 7f1a05e43640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1a00162f30 0x7f1a000b7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42764/0 (socket says 172.21.15.88:42764) 2024-09-16T01:36:34.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.838+0000 7f1a05e43640 1 -- 172.21.15.88:0/3236292097 learned_addr learned my addr 172.21.15.88:0/3236292097 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:34.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.838+0000 7f1a04e41640 1 --2- 172.21.15.88:0/3236292097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a000cf6b0 0x7f1a001629f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:34.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.838+0000 7f1a05e43640 1 -- 172.21.15.88:0/3236292097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a000cf6b0 msgr2=0x7f1a001629f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:34.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.838+0000 7f1a05e43640 1 --2- 172.21.15.88:0/3236292097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a000cf6b0 0x7f1a001629f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.838+0000 7f1a05e43640 1 -- 172.21.15.88:0/3236292097 >> v1:172.21.15.154:6789/0 conn(0x7f1a000b63d0 legacy=0x7f1a000d39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:34.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.838+0000 7f1a05e43640 1 -- 172.21.15.88:0/3236292097 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a000d31b0 con 0x7f1a00162f30 2024-09-16T01:36:34.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.838+0000 7f1a04e41640 1 --2- 172.21.15.88:0/3236292097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a000cf6b0 0x7f1a001629f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:34.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.839+0000 7f1a05e43640 1 --2- 172.21.15.88:0/3236292097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a00162f30 0x7f1a000b7880 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f19f8009e50 tx=0x7f19f802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61a0868ddd42f9c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:34.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.839+0000 7f19f7fff640 1 -- 172.21.15.88:0/3236292097 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f8036070 con 0x7f1a00162f30 2024-09-16T01:36:34.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.839+0000 7f19f7fff640 1 -- 172.21.15.88:0/3236292097 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19f8031600 con 0x7f1a00162f30 2024-09-16T01:36:34.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.839+0000 7f19f7fff640 1 -- 172.21.15.88:0/3236292097 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f80319d0 con 0x7f1a00162f30 2024-09-16T01:36:34.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.839+0000 7f1a078a6640 1 -- 172.21.15.88:0/3236292097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a00162f30 msgr2=0x7f1a000b7880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:34.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.839+0000 7f1a078a6640 1 --2- 172.21.15.88:0/3236292097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a00162f30 0x7f1a000b7880 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f19f8009e50 tx=0x7f19f802a6c0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.839+0000 7f1a078a6640 1 -- 172.21.15.88:0/3236292097 shutdown_connections 2024-09-16T01:36:34.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.839+0000 7f1a078a6640 1 --2- 172.21.15.88:0/3236292097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a00162f30 0x7f1a000b7880 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.839+0000 7f1a078a6640 1 --2- 172.21.15.88:0/3236292097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a000cf6b0 0x7f1a001629f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.839+0000 7f1a078a6640 1 -- 172.21.15.88:0/3236292097 >> 172.21.15.88:0/3236292097 conn(0x7f1a000d1160 msgr2=0x7f1a000d1570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:34.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.840+0000 7f1a078a6640 1 -- 172.21.15.88:0/3236292097 shutdown_connections 2024-09-16T01:36:34.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.840+0000 7f1a078a6640 1 -- 172.21.15.88:0/3236292097 wait complete. 2024-09-16T01:36:34.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.841+0000 7f1a078a6640 1 Processor -- start 2024-09-16T01:36:34.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.841+0000 7f1a078a6640 1 -- start start 2024-09-16T01:36:34.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.841+0000 7f1a078a6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a000b63d0 0x7f1a001f4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:34.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.841+0000 7f1a078a6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a000cf6b0 0x7f1a000cab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:34.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.841+0000 7f1a078a6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a00162f30 0x7f1a000cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:34.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.841+0000 7f1a078a6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1a000cb5c0 con 0x7f1a000b63d0 2024-09-16T01:36:34.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.841+0000 7f1a078a6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1a000cb440 con 0x7f1a000cf6b0 2024-09-16T01:36:34.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.841+0000 7f1a078a6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1a000d1c80 con 0x7f1a00162f30 2024-09-16T01:36:34.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.842+0000 7f1a05642640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a000b63d0 0x7f1a001f4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:34.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.842+0000 7f1a05642640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a000b63d0 0x7f1a001f4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42770/0 (socket says 172.21.15.88:42770) 2024-09-16T01:36:34.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.842+0000 7f1a05642640 1 -- 172.21.15.88:0/2346018979 learned_addr learned my addr 172.21.15.88:0/2346018979 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:34.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.842+0000 7f1a04e41640 1 --2- 172.21.15.88:0/2346018979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a000cf6b0 0x7f1a000cab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:34.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.842+0000 7f1a05e43640 1 --2- 172.21.15.88:0/2346018979 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a00162f30 0x7f1a000cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:34.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.842+0000 7f1a05642640 1 -- 172.21.15.88:0/2346018979 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a00162f30 msgr2=0x7f1a000cb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:34.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.842+0000 7f1a05642640 1 --2- 172.21.15.88:0/2346018979 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a00162f30 0x7f1a000cb7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.842+0000 7f1a05642640 1 -- 172.21.15.88:0/2346018979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a000cf6b0 msgr2=0x7f1a000cab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:34.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.842+0000 7f1a05642640 1 --2- 172.21.15.88:0/2346018979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a000cf6b0 0x7f1a000cab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:34.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.842+0000 7f1a05642640 1 -- 172.21.15.88:0/2346018979 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a000c9300 con 0x7f1a000b63d0 2024-09-16T01:36:34.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.842+0000 7f1a05e43640 1 --2- 172.21.15.88:0/2346018979 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a00162f30 0x7f1a000cb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:34.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.842+0000 7f1a04e41640 1 --2- 172.21.15.88:0/2346018979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a000cf6b0 0x7f1a000cab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:34.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.842+0000 7f1a05642640 1 --2- 172.21.15.88:0/2346018979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a000b63d0 0x7f1a001f4250 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f19fc00c430 tx=0x7f19fc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:34.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.842+0000 7f19f5ffb640 1 -- 172.21.15.88:0/2346018979 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19fc011790 con 0x7f1a000b63d0 2024-09-16T01:36:34.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.842+0000 7f19f5ffb640 1 -- 172.21.15.88:0/2346018979 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19fc017070 con 0x7f1a000b63d0 2024-09-16T01:36:34.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.842+0000 7f19f5ffb640 1 -- 172.21.15.88:0/2346018979 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19fc011bb0 con 0x7f1a000b63d0 2024-09-16T01:36:34.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.842+0000 7f1a078a6640 1 -- 172.21.15.88:0/2346018979 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a001f4b80 con 0x7f1a000b63d0 2024-09-16T01:36:34.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.842+0000 7f1a078a6640 1 -- 172.21.15.88:0/2346018979 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a000cb070 con 0x7f1a000b63d0 2024-09-16T01:36:34.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.843+0000 7f1a078a6640 1 -- 172.21.15.88:0/2346018979 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a000b67b0 con 0x7f1a000b63d0 2024-09-16T01:36:34.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.844+0000 7f19f5ffb640 1 -- 172.21.15.88:0/2346018979 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f19fc01c460 con 0x7f1a000b63d0 2024-09-16T01:36:34.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.844+0000 7f19f5ffb640 1 --2- 172.21.15.88:0/2346018979 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f19e406c2b0 0x7f19e406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:34.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.844+0000 7f1a04e41640 1 --2- 172.21.15.88:0/2346018979 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f19e406c2b0 0x7f19e406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:34.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.844+0000 7f19f5ffb640 1 -- 172.21.15.88:0/2346018979 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5029+0+0 (secure 0 0 0) 0x7f19fc08b070 con 0x7f1a000b63d0 2024-09-16T01:36:34.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.845+0000 7f1a04e41640 1 --2- 172.21.15.88:0/2346018979 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f19e406c2b0 0x7f19e406e770 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1a000cec00 tx=0x7f19f0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:34.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:34.849+0000 7f19f5ffb640 1 -- 172.21.15.88:0/2346018979 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f19fc059a80 con 0x7f1a000b63d0 2024-09-16T01:36:35.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.076+0000 7f1a078a6640 1 -- 172.21.15.88:0/2346018979 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_precommit"]} v 0) -- 0x7f1a000cf280 con 0x7f1a000b63d0 2024-09-16T01:36:35.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.137+0000 7f19f5ffb640 1 -- 172.21.15.88:0/2346018979 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_precommit"]}]=0 v136) ==== 145+0+0 (secure 0 0 0) 0x7f19fc059380 con 0x7f1a000b63d0 2024-09-16T01:36:35.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.140+0000 7f1a078a6640 1 -- 172.21.15.88:0/2346018979 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f19e406c2b0 msgr2=0x7f19e406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:35.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.140+0000 7f1a078a6640 1 --2- 172.21.15.88:0/2346018979 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f19e406c2b0 0x7f19e406e770 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1a000cec00 tx=0x7f19f0008040 comp rx=0 tx=0).stop 2024-09-16T01:36:35.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.140+0000 7f1a078a6640 1 -- 172.21.15.88:0/2346018979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a000b63d0 msgr2=0x7f1a001f4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:35.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.140+0000 7f1a078a6640 1 --2- 172.21.15.88:0/2346018979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a000b63d0 0x7f1a001f4250 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f19fc00c430 tx=0x7f19fc00c8f0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.140+0000 7f1a078a6640 1 -- 172.21.15.88:0/2346018979 shutdown_connections 2024-09-16T01:36:35.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.140+0000 7f1a078a6640 1 --2- 172.21.15.88:0/2346018979 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f19e406c2b0 0x7f19e406e770 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.140+0000 7f1a078a6640 1 --2- 172.21.15.88:0/2346018979 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a00162f30 0x7f1a000cb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.140+0000 7f1a078a6640 1 --2- 172.21.15.88:0/2346018979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a000cf6b0 0x7f1a000cab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.140+0000 7f1a078a6640 1 --2- 172.21.15.88:0/2346018979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a000b63d0 0x7f1a001f4250 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.140+0000 7f1a078a6640 1 -- 172.21.15.88:0/2346018979 >> 172.21.15.88:0/2346018979 conn(0x7f1a000d1160 msgr2=0x7f1a00162710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:35.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.140+0000 7f1a078a6640 1 -- 172.21.15.88:0/2346018979 shutdown_connections 2024-09-16T01:36:35.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.140+0000 7f1a078a6640 1 -- 172.21.15.88:0/2346018979 wait complete. 2024-09-16T01:36:35.152 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T01:36:35.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.252+0000 7fdbd4be4640 1 Processor -- start 2024-09-16T01:36:35.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.252+0000 7fdbd4be4640 1 -- start start 2024-09-16T01:36:35.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.252+0000 7fdbd4be4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdbd00ceda0 0x7fdbd0073490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:35.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.252+0000 7fdbd4be4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbd00cf180 0x7fdbd00cf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:35.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.253+0000 7fdbd4be4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdbd006dc10 0x7fdbd006e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:35.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.253+0000 7fdbd4be4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fdbd00722d0 con 0x7fdbd006dc10 2024-09-16T01:36:35.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.253+0000 7fdbd4be4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fdbd0072590 con 0x7fdbd00ceda0 2024-09-16T01:36:35.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.253+0000 7fdbd4be4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdbd0072870 con 0x7fdbd00cf180 2024-09-16T01:36:35.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.253+0000 7fdbced9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdbd006dc10 0x7fdbd006e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:35.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.253+0000 7fdbced9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdbd006dc10 0x7fdbd006e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42786/0 (socket says 172.21.15.88:42786) 2024-09-16T01:36:35.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.253+0000 7fdbced9d640 1 -- 172.21.15.88:0/279077799 learned_addr learned my addr 172.21.15.88:0/279077799 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:35.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.253+0000 7fdbce59c640 1 --2- 172.21.15.88:0/279077799 >> v2:172.21.15.154:3300/0 conn(0x7fdbd00ceda0 0x7fdbd0073490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:35.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.253+0000 7fdbc5d9b640 1 --2- 172.21.15.88:0/279077799 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbd00cf180 0x7fdbd00cf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:35.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.253+0000 7fdbced9d640 1 -- 172.21.15.88:0/279077799 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbd00cf180 msgr2=0x7fdbd00cf5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:35.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.253+0000 7fdbced9d640 1 --2- 172.21.15.88:0/279077799 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbd00cf180 0x7fdbd00cf5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.253+0000 7fdbced9d640 1 -- 172.21.15.88:0/279077799 >> v2:172.21.15.154:3300/0 conn(0x7fdbd00ceda0 msgr2=0x7fdbd0073490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:35.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.253+0000 7fdbced9d640 1 --2- 172.21.15.88:0/279077799 >> v2:172.21.15.154:3300/0 conn(0x7fdbd00ceda0 0x7fdbd0073490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.253+0000 7fdbced9d640 1 -- 172.21.15.88:0/279077799 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbd0072b50 con 0x7fdbd006dc10 2024-09-16T01:36:35.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.253+0000 7fdbce59c640 1 --2- 172.21.15.88:0/279077799 >> v2:172.21.15.154:3300/0 conn(0x7fdbd00ceda0 0x7fdbd0073490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:35.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.253+0000 7fdbc5d9b640 1 --2- 172.21.15.88:0/279077799 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbd00cf180 0x7fdbd00cf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:35.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.254+0000 7fdbced9d640 1 --2- 172.21.15.88:0/279077799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbd006dc10 0x7fdbd006e0a0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fdbd00b3b10 tx=0x7fdbc802c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34ef2bfa30a49f82 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:35.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.254+0000 7fdbcdd9b640 1 -- 172.21.15.88:0/279077799 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc8033400 con 0x7fdbd006dc10 2024-09-16T01:36:35.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.254+0000 7fdbcdd9b640 1 -- 172.21.15.88:0/279077799 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbc80335a0 con 0x7fdbd006dc10 2024-09-16T01:36:35.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.254+0000 7fdbcdd9b640 1 -- 172.21.15.88:0/279077799 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc8033970 con 0x7fdbd006dc10 2024-09-16T01:36:35.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.254+0000 7fdbd4be4640 1 -- 172.21.15.88:0/279077799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbd006dc10 msgr2=0x7fdbd006e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:35.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.254+0000 7fdbd4be4640 1 --2- 172.21.15.88:0/279077799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbd006dc10 0x7fdbd006e0a0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fdbd00b3b10 tx=0x7fdbc802c8b0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.254+0000 7fdbd4be4640 1 -- 172.21.15.88:0/279077799 shutdown_connections 2024-09-16T01:36:35.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.254+0000 7fdbd4be4640 1 --2- 172.21.15.88:0/279077799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbd006dc10 0x7fdbd006e0a0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.254+0000 7fdbd4be4640 1 --2- 172.21.15.88:0/279077799 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbd00cf180 0x7fdbd00cf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.254+0000 7fdbd4be4640 1 --2- 172.21.15.88:0/279077799 >> v2:172.21.15.154:3300/0 conn(0x7fdbd00ceda0 0x7fdbd0073490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.254+0000 7fdbd4be4640 1 -- 172.21.15.88:0/279077799 >> 172.21.15.88:0/279077799 conn(0x7fdbd006cb50 msgr2=0x7fdbd006cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:35.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.254+0000 7fdbd4be4640 1 -- 172.21.15.88:0/279077799 shutdown_connections 2024-09-16T01:36:35.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.254+0000 7fdbd4be4640 1 -- 172.21.15.88:0/279077799 wait complete. 2024-09-16T01:36:35.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.256+0000 7fdbd4be4640 1 Processor -- start 2024-09-16T01:36:35.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.256+0000 7fdbd4be4640 1 -- start start 2024-09-16T01:36:35.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.257+0000 7fdbd4be4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbd006dc10 0x7fdbd0142390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:35.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.257+0000 7fdbd4be4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbd00ceda0 0x7fdbd00b0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:35.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.257+0000 7fdbd4be4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbd00cf180 0x7fdbd0144e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:35.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.257+0000 7fdbd4be4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd00739d0 con 0x7fdbd006dc10 2024-09-16T01:36:35.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.257+0000 7fdbd4be4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdbd00d0510 con 0x7fdbd00cf180 2024-09-16T01:36:35.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.257+0000 7fdbd4be4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdbd0073b50 con 0x7fdbd00ceda0 2024-09-16T01:36:35.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.257+0000 7fdbce59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbd006dc10 0x7fdbd0142390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:35.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.257+0000 7fdbce59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbd006dc10 0x7fdbd0142390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42798/0 (socket says 172.21.15.88:42798) 2024-09-16T01:36:35.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.257+0000 7fdbce59c640 1 -- 172.21.15.88:0/3644803288 learned_addr learned my addr 172.21.15.88:0/3644803288 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:35.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.257+0000 7fdbced9d640 1 --2- 172.21.15.88:0/3644803288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbd00cf180 0x7fdbd0144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:35.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.257+0000 7fdbc5d9b640 1 --2- 172.21.15.88:0/3644803288 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbd00ceda0 0x7fdbd00b0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:35.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.257+0000 7fdbce59c640 1 -- 172.21.15.88:0/3644803288 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbd00ceda0 msgr2=0x7fdbd00b0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:35.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.257+0000 7fdbce59c640 1 --2- 172.21.15.88:0/3644803288 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbd00ceda0 0x7fdbd00b0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.257+0000 7fdbce59c640 1 -- 172.21.15.88:0/3644803288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbd00cf180 msgr2=0x7fdbd0144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:35.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.257+0000 7fdbce59c640 1 --2- 172.21.15.88:0/3644803288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbd00cf180 0x7fdbd0144e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.257+0000 7fdbce59c640 1 -- 172.21.15.88:0/3644803288 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbd0145370 con 0x7fdbd006dc10 2024-09-16T01:36:35.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.257+0000 7fdbced9d640 1 --2- 172.21.15.88:0/3644803288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbd00cf180 0x7fdbd0144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:35.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.257+0000 7fdbce59c640 1 --2- 172.21.15.88:0/3644803288 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbd006dc10 0x7fdbd0142390 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fdbc000d800 tx=0x7fdbc000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:35.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.257+0000 7fdbc77fe640 1 -- 172.21.15.88:0/3644803288 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc0018070 con 0x7fdbd006dc10 2024-09-16T01:36:35.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.257+0000 7fdbc77fe640 1 -- 172.21.15.88:0/3644803288 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbc0004030 con 0x7fdbd006dc10 2024-09-16T01:36:35.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.257+0000 7fdbd4be4640 1 -- 172.21.15.88:0/3644803288 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbd0142970 con 0x7fdbd006dc10 2024-09-16T01:36:35.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.257+0000 7fdbc77fe640 1 -- 172.21.15.88:0/3644803288 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc0004400 con 0x7fdbd006dc10 2024-09-16T01:36:35.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.257+0000 7fdbd4be4640 1 -- 172.21.15.88:0/3644803288 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbd0142f30 con 0x7fdbd006dc10 2024-09-16T01:36:35.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.259+0000 7fdbd4be4640 1 -- 172.21.15.88:0/3644803288 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbd0073490 con 0x7fdbd006dc10 2024-09-16T01:36:35.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.259+0000 7fdbc77fe640 1 -- 172.21.15.88:0/3644803288 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fdbc00096a0 con 0x7fdbd006dc10 2024-09-16T01:36:35.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.259+0000 7fdbc77fe640 1 --2- 172.21.15.88:0/3644803288 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdbb006c2e0 0x7fdbb006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:35.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.260+0000 7fdbc77fe640 1 -- 172.21.15.88:0/3644803288 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5029+0+0 (secure 0 0 0) 0x7fdbc008b4f0 con 0x7fdbd006dc10 2024-09-16T01:36:35.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.260+0000 7fdbc5d9b640 1 --2- 172.21.15.88:0/3644803288 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdbb006c2e0 0x7fdbb006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:35.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.260+0000 7fdbc5d9b640 1 --2- 172.21.15.88:0/3644803288 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdbb006c2e0 0x7fdbb006e7a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdbd006f410 tx=0x7fdbb8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:35.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.264+0000 7fdbc77fe640 1 -- 172.21.15.88:0/3644803288 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fdbd0145370 con 0x7fdbd006dc10 2024-09-16T01:36:35.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.430+0000 7fdbd4be4640 1 -- 172.21.15.88:0/3644803288 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fdbd00ccf60 con 0x7fdbd006dc10 2024-09-16T01:36:35.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.435+0000 7fdbc77fe640 1 -- 172.21.15.88:0/3644803288 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) ==== 122+0+0 (secure 0 0 0) 0x7fdbc0059f00 con 0x7fdbd006dc10 2024-09-16T01:36:35.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.438+0000 7fdbd4be4640 1 -- 172.21.15.88:0/3644803288 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdbb006c2e0 msgr2=0x7fdbb006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:35.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.438+0000 7fdbd4be4640 1 --2- 172.21.15.88:0/3644803288 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdbb006c2e0 0x7fdbb006e7a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdbd006f410 tx=0x7fdbb8008040 comp rx=0 tx=0).stop 2024-09-16T01:36:35.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.438+0000 7fdbd4be4640 1 -- 172.21.15.88:0/3644803288 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbd006dc10 msgr2=0x7fdbd0142390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:35.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.438+0000 7fdbd4be4640 1 --2- 172.21.15.88:0/3644803288 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbd006dc10 0x7fdbd0142390 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fdbc000d800 tx=0x7fdbc000dcc0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.438+0000 7fdbd4be4640 1 -- 172.21.15.88:0/3644803288 shutdown_connections 2024-09-16T01:36:35.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.438+0000 7fdbd4be4640 1 --2- 172.21.15.88:0/3644803288 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdbb006c2e0 0x7fdbb006e7a0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.438+0000 7fdbd4be4640 1 --2- 172.21.15.88:0/3644803288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbd00cf180 0x7fdbd0144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.438+0000 7fdbd4be4640 1 --2- 172.21.15.88:0/3644803288 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbd00ceda0 0x7fdbd00b0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.438+0000 7fdbd4be4640 1 --2- 172.21.15.88:0/3644803288 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbd006dc10 0x7fdbd0142390 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.438+0000 7fdbd4be4640 1 -- 172.21.15.88:0/3644803288 >> 172.21.15.88:0/3644803288 conn(0x7fdbd006cb50 msgr2=0x7fdbd00b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:35.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.438+0000 7fdbd4be4640 1 -- 172.21.15.88:0/3644803288 shutdown_connections 2024-09-16T01:36:35.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.438+0000 7fdbd4be4640 1 -- 172.21.15.88:0/3644803288 wait complete. 2024-09-16T01:36:35.450 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T01:36:35.450 DEBUG:teuthology.orchestra.run.smithi088:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T01:36:35.496 DEBUG:teuthology.orchestra.run.smithi154:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T01:36:35.538 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T01:36:35.538 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:36:35.538 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:36:35.554 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:36:35.554 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:36:35.554 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:36:35.596 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056026,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1347 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T01:36:35.597 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:36:35.597 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:36:35.612 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:36:35.613 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:36:35.613 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:36:35.654 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056026,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1347 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T01:36:35.655 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-16T01:36:35.655 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:36:35.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.751+0000 7f4c03e37640 1 Processor -- start 2024-09-16T01:36:35.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.751+0000 7f4c03e37640 1 -- start start 2024-09-16T01:36:35.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.752+0000 7f4c03e37640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4bfc155800 0x7f4bfc0b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:35.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.752+0000 7f4c03e37640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4bfc0cfc80 0x7f4bfc0b4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:35.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.752+0000 7f4c03e37640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bfc15c770 con 0x7f4bfc15c330 2024-09-16T01:36:35.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.752+0000 7f4c03e37640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f4bfc0b4e60 con 0x7f4bfc0cfc80 2024-09-16T01:36:35.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.752+0000 7f4c03e37640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4bfc0b50a0 con 0x7f4bfc155800 2024-09-16T01:36:35.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.752+0000 7f4c023d4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4bfc0cfc80 0x7f4bfc0b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:35.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.752+0000 7f4c01bd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4bfc155800 0x7f4bfc0b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:35.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.752+0000 7f4c013d2640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f4bfc15c330 0x7f4bfc0cf570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:50516/0 (socket says 172.21.15.88:50516) 2024-09-16T01:36:35.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.752+0000 7f4c013d2640 1 -- 172.21.15.88:0/2619162083 learned_addr learned my addr 172.21.15.88:0/2619162083 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:35.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.753+0000 7f4c01bd3640 1 -- 172.21.15.88:0/2619162083 >> v2:172.21.15.154:3300/0 conn(0x7f4bfc0cfc80 msgr2=0x7f4bfc0b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:35.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.753+0000 7f4c01bd3640 1 --2- 172.21.15.88:0/2619162083 >> v2:172.21.15.154:3300/0 conn(0x7f4bfc0cfc80 0x7f4bfc0b4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.753+0000 7f4c01bd3640 1 -- 172.21.15.88:0/2619162083 >> v1:172.21.15.154:6789/0 conn(0x7f4bfc15c330 legacy=0x7f4bfc0cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:35.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.753+0000 7f4c01bd3640 1 -- 172.21.15.88:0/2619162083 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bfc0b5380 con 0x7f4bfc155800 2024-09-16T01:36:35.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.753+0000 7f4c023d4640 1 --2- 172.21.15.88:0/2619162083 >> v2:172.21.15.154:3300/0 conn(0x7f4bfc0cfc80 0x7f4bfc0b4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:36:35.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.754+0000 7f4c01bd3640 1 --2- 172.21.15.88:0/2619162083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4bfc155800 0x7f4bfc0b6140 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4bf80097a0 tx=0x7f4bf802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=932a033709cc7565 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:35.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.754+0000 7f4c00bd1640 1 -- 172.21.15.88:0/2619162083 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf8037070 con 0x7f4bfc155800 2024-09-16T01:36:35.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.754+0000 7f4c00bd1640 1 -- 172.21.15.88:0/2619162083 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bf8032440 con 0x7f4bfc155800 2024-09-16T01:36:35.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.754+0000 7f4c00bd1640 1 -- 172.21.15.88:0/2619162083 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf8032810 con 0x7f4bfc155800 2024-09-16T01:36:35.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.754+0000 7f4c03e37640 1 -- 172.21.15.88:0/2619162083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4bfc155800 msgr2=0x7f4bfc0b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:35.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.754+0000 7f4c03e37640 1 --2- 172.21.15.88:0/2619162083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4bfc155800 0x7f4bfc0b6140 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4bf80097a0 tx=0x7f4bf802a180 comp rx=0 tx=0).stop 2024-09-16T01:36:35.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.754+0000 7f4c03e37640 1 -- 172.21.15.88:0/2619162083 shutdown_connections 2024-09-16T01:36:35.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.754+0000 7f4c03e37640 1 --2- 172.21.15.88:0/2619162083 >> v2:172.21.15.154:3300/0 conn(0x7f4bfc0cfc80 0x7f4bfc0b4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.754+0000 7f4c03e37640 1 --2- 172.21.15.88:0/2619162083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4bfc155800 0x7f4bfc0b6140 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.754+0000 7f4c03e37640 1 -- 172.21.15.88:0/2619162083 >> 172.21.15.88:0/2619162083 conn(0x7f4bfc06bc10 msgr2=0x7f4bfc06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:35.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.754+0000 7f4c03e37640 1 -- 172.21.15.88:0/2619162083 shutdown_connections 2024-09-16T01:36:35.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.754+0000 7f4c03e37640 1 -- 172.21.15.88:0/2619162083 wait complete. 2024-09-16T01:36:35.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.755+0000 7f4c03e37640 1 Processor -- start 2024-09-16T01:36:35.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.756+0000 7f4c03e37640 1 -- start start 2024-09-16T01:36:35.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.756+0000 7f4c03e37640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4bfc0cfc80 0x7f4bfc151a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:35.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.756+0000 7f4c03e37640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4bfc155800 0x7f4bfc1488c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:35.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.756+0000 7f4c03e37640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bfc15c330 0x7f4bfc152320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:35.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.756+0000 7f4c03e37640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc0b5c20 con 0x7f4bfc15c330 2024-09-16T01:36:35.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.756+0000 7f4c03e37640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc0b5940 con 0x7f4bfc155800 2024-09-16T01:36:35.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.756+0000 7f4c03e37640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4bfc155be0 con 0x7f4bfc0cfc80 2024-09-16T01:36:35.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.756+0000 7f4c023d4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bfc15c330 0x7f4bfc152320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:35.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.756+0000 7f4c023d4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bfc15c330 0x7f4bfc152320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42810/0 (socket says 172.21.15.88:42810) 2024-09-16T01:36:35.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.756+0000 7f4c023d4640 1 -- 172.21.15.88:0/2828219022 learned_addr learned my addr 172.21.15.88:0/2828219022 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:35.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.756+0000 7f4c013d2640 1 --2- 172.21.15.88:0/2828219022 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4bfc155800 0x7f4bfc1488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:35.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.756+0000 7f4c01bd3640 1 --2- 172.21.15.88:0/2828219022 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4bfc0cfc80 0x7f4bfc151a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:35.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.757+0000 7f4c023d4640 1 -- 172.21.15.88:0/2828219022 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4bfc0cfc80 msgr2=0x7f4bfc151a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:35.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.757+0000 7f4c023d4640 1 --2- 172.21.15.88:0/2828219022 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4bfc0cfc80 0x7f4bfc151a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.757+0000 7f4c023d4640 1 -- 172.21.15.88:0/2828219022 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4bfc155800 msgr2=0x7f4bfc1488c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:35.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.757+0000 7f4c023d4640 1 --2- 172.21.15.88:0/2828219022 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4bfc155800 0x7f4bfc1488c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.757+0000 7f4c023d4640 1 -- 172.21.15.88:0/2828219022 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bfc1509e0 con 0x7f4bfc15c330 2024-09-16T01:36:35.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.757+0000 7f4c023d4640 1 --2- 172.21.15.88:0/2828219022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bfc15c330 0x7f4bfc152320 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f4bf400efc0 tx=0x7f4bf400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:35.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.757+0000 7f4bf27fc640 1 -- 172.21.15.88:0/2828219022 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf4018020 con 0x7f4bfc15c330 2024-09-16T01:36:35.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.757+0000 7f4bf27fc640 1 -- 172.21.15.88:0/2828219022 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bf4014480 con 0x7f4bfc15c330 2024-09-16T01:36:35.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.757+0000 7f4c03e37640 1 -- 172.21.15.88:0/2828219022 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bfc150cc0 con 0x7f4bfc15c330 2024-09-16T01:36:35.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.757+0000 7f4bf27fc640 1 -- 172.21.15.88:0/2828219022 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf4014870 con 0x7f4bfc15c330 2024-09-16T01:36:35.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.757+0000 7f4c03e37640 1 -- 172.21.15.88:0/2828219022 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bfc151280 con 0x7f4bfc15c330 2024-09-16T01:36:35.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.758+0000 7f4c03e37640 1 -- 172.21.15.88:0/2828219022 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bfc0b4920 con 0x7f4bfc15c330 2024-09-16T01:36:35.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.758+0000 7f4bf27fc640 1 -- 172.21.15.88:0/2828219022 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4bf401b020 con 0x7f4bfc15c330 2024-09-16T01:36:35.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.759+0000 7f4bf27fc640 1 --2- 172.21.15.88:0/2828219022 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4be006c2b0 0x7f4be006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:35.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.759+0000 7f4bf27fc640 1 -- 172.21.15.88:0/2828219022 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5029+0+0 (secure 0 0 0) 0x7f4bf408be20 con 0x7f4bfc15c330 2024-09-16T01:36:35.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.759+0000 7f4c01bd3640 1 --2- 172.21.15.88:0/2828219022 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4be006c2b0 0x7f4be006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:35.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.759+0000 7f4c01bd3640 1 --2- 172.21.15.88:0/2828219022 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4be006c2b0 0x7f4be006e770 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8034000 tx=0x7f4bf802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:35.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.763+0000 7f4bf27fc640 1 -- 172.21.15.88:0/2828219022 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4bf4022020 con 0x7f4bfc15c330 2024-09-16T01:36:35.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.960+0000 7f4c03e37640 1 -- 172.21.15.88:0/2828219022 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4bfc0cf980 con 0x7f4bfc15c330 2024-09-16T01:36:35.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.961+0000 7f4bf27fc640 1 -- 172.21.15.88:0/2828219022 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+3911 (secure 0 0 0) 0x7f4bf4010040 con 0x7f4bfc15c330 2024-09-16T01:36:35.962 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:35.963 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":33,"btime":"2024-09-16T01:36:29:790787+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4877,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1769276566","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":1769276566},{"type":"v1","addr":"172.21.15.154:6835","nonce":1769276566}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5035,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/725621378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":725621378},{"type":"v1","addr":"172.21.15.154:6834","nonce":725621378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5403,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4199111872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4199111872},{"type":"v1","addr":"172.21.15.88:6837","nonce":4199111872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22},{"gid":5769,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/498293121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":498293121},{"type":"v1","addr":"172.21.15.88:6836","nonce":498293121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[]} 2024-09-16T01:36:35.963 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 33 2024-09-16T01:36:35.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.963+0000 7f4c03e37640 1 -- 172.21.15.88:0/2828219022 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4be006c2b0 msgr2=0x7f4be006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:35.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.963+0000 7f4c03e37640 1 --2- 172.21.15.88:0/2828219022 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4be006c2b0 0x7f4be006e770 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8034000 tx=0x7f4bf802e000 comp rx=0 tx=0).stop 2024-09-16T01:36:35.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.963+0000 7f4c03e37640 1 -- 172.21.15.88:0/2828219022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bfc15c330 msgr2=0x7f4bfc152320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:35.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.963+0000 7f4c03e37640 1 --2- 172.21.15.88:0/2828219022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bfc15c330 0x7f4bfc152320 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f4bf400efc0 tx=0x7f4bf400c490 comp rx=0 tx=0).stop 2024-09-16T01:36:35.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.963+0000 7f4c03e37640 1 -- 172.21.15.88:0/2828219022 shutdown_connections 2024-09-16T01:36:35.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.963+0000 7f4c03e37640 1 --2- 172.21.15.88:0/2828219022 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4be006c2b0 0x7f4be006e770 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.963+0000 7f4c03e37640 1 --2- 172.21.15.88:0/2828219022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4bfc15c330 0x7f4bfc152320 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.963+0000 7f4c03e37640 1 --2- 172.21.15.88:0/2828219022 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4bfc155800 0x7f4bfc1488c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.963+0000 7f4c03e37640 1 --2- 172.21.15.88:0/2828219022 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4bfc0cfc80 0x7f4bfc151a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:35.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.963+0000 7f4c03e37640 1 -- 172.21.15.88:0/2828219022 >> 172.21.15.88:0/2828219022 conn(0x7f4bfc06bc10 msgr2=0x7f4bfc0cd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:35.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.963+0000 7f4c03e37640 1 -- 172.21.15.88:0/2828219022 shutdown_connections 2024-09-16T01:36:35.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:35.963+0000 7f4c03e37640 1 -- 172.21.15.88:0/2828219022 wait complete. 2024-09-16T01:36:35.976 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-16T01:36:35.976 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-16T01:36:35.977 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-16T01:36:35.977 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:36:35.977 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-16T01:36:35.977 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-16T01:36:35.977 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-16T01:36:35.977 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:36:35.977 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-16T01:36:35.978 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-16T01:36:35.978 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-16T01:36:35.978 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:36:35.978 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-16T01:36:35.978 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-16T01:36:35.978 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-16T01:36:35.978 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:36:35.986 INFO:tasks.ceph.mds.a:Stopped 2024-09-16T01:36:35.986 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-16T01:36:35.987 INFO:tasks.ceph.mds.c:Stopped 2024-09-16T01:36:35.987 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-16T01:36:35.989 INFO:tasks.ceph.mds.b:Stopped 2024-09-16T01:36:35.990 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-16T01:36:35.990 INFO:tasks.ceph.mds.d:Stopped 2024-09-16T01:36:35.990 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-16T01:36:36.028 INFO:tasks.ceph.mds.a:Started 2024-09-16T01:36:36.029 DEBUG:teuthology.parallel:result is None 2024-09-16T01:36:36.029 INFO:tasks.ceph.mds.c:Started 2024-09-16T01:36:36.029 DEBUG:teuthology.parallel:result is None 2024-09-16T01:36:36.032 INFO:tasks.ceph.mds.b:Started 2024-09-16T01:36:36.032 DEBUG:teuthology.parallel:result is None 2024-09-16T01:36:36.033 INFO:tasks.ceph.mds.d:Started 2024-09-16T01:36:36.033 DEBUG:teuthology.parallel:result is None 2024-09-16T01:36:36.033 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-16T01:36:36.103 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:36:36.103 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:36:36.115 INFO:tasks.ceph.mds.b.smithi154.stdout:starting mds.b at 2024-09-16T01:36:36.115 INFO:tasks.ceph.mds.d.smithi154.stdout:starting mds.d at 2024-09-16T01:36:36.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.136+0000 7fbc4bc9b640 1 Processor -- start 2024-09-16T01:36:36.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.136+0000 7fbc4bc9b640 1 -- start start 2024-09-16T01:36:36.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.137+0000 7fbc4bc9b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbc4415b480 0x7fbc440b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:36.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.137+0000 7fbc4bc9b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc44072700 0x7fbc4415bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:36.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.137+0000 7fbc4bc9b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc440ccff0 con 0x7fbc440b60c0 2024-09-16T01:36:36.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.137+0000 7fbc4bc9b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbc440cd2d0 con 0x7fbc4415b480 2024-09-16T01:36:36.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.137+0000 7fbc4bc9b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbc440cd5b0 con 0x7fbc44072700 2024-09-16T01:36:36.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.137+0000 7fbc49236640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fbc440b60c0 0x7fbc44071ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60936/0 (socket says 172.21.15.88:60936) 2024-09-16T01:36:36.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.137+0000 7fbc49236640 1 -- 172.21.15.88:0/477215651 learned_addr learned my addr 172.21.15.88:0/477215651 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:36.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.137+0000 7fbc49a37640 1 --2- 172.21.15.88:0/477215651 >> v2:172.21.15.154:3300/0 conn(0x7fbc4415b480 0x7fbc440b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:36.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.137+0000 7fbc4a238640 1 --2- 172.21.15.88:0/477215651 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc44072700 0x7fbc4415bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:36.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.137+0000 7fbc48a35640 1 -- 172.21.15.88:0/477215651 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2686094566 0 0) 0x7fbc440ccff0 con 0x7fbc440b60c0 2024-09-16T01:36:36.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.138+0000 7fbc48a35640 1 -- 172.21.15.88:0/477215651 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc30003520 con 0x7fbc440b60c0 2024-09-16T01:36:36.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.138+0000 7fbc49a37640 1 -- 172.21.15.88:0/477215651 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc44072700 msgr2=0x7fbc4415bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:36.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.138+0000 7fbc49a37640 1 --2- 172.21.15.88:0/477215651 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc44072700 0x7fbc4415bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:36.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.138+0000 7fbc49a37640 1 -- 172.21.15.88:0/477215651 >> v1:172.21.15.88:6789/0 conn(0x7fbc440b60c0 legacy=0x7fbc44071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:36.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.138+0000 7fbc49a37640 1 -- 172.21.15.88:0/477215651 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc440cd890 con 0x7fbc4415b480 2024-09-16T01:36:36.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.139+0000 7fbc49a37640 1 --2- 172.21.15.88:0/477215651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc4415b480 0x7fbc440b5b80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbc400097a0 tx=0x7fbc4002a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3da8d439379c9cf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:36.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.142+0000 7fbc48a35640 1 -- 172.21.15.88:0/477215651 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc40037070 con 0x7fbc4415b480 2024-09-16T01:36:36.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.142+0000 7fbc48a35640 1 -- 172.21.15.88:0/477215651 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc4002ae50 con 0x7fbc4415b480 2024-09-16T01:36:36.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.142+0000 7fbc48a35640 1 -- 172.21.15.88:0/477215651 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc40032660 con 0x7fbc4415b480 2024-09-16T01:36:36.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.143+0000 7fbc4bc9b640 1 -- 172.21.15.88:0/477215651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc4415b480 msgr2=0x7fbc440b5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:36.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.143+0000 7fbc4bc9b640 1 --2- 172.21.15.88:0/477215651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc4415b480 0x7fbc440b5b80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbc400097a0 tx=0x7fbc4002a100 comp rx=0 tx=0).stop 2024-09-16T01:36:36.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.143+0000 7fbc4bc9b640 1 -- 172.21.15.88:0/477215651 shutdown_connections 2024-09-16T01:36:36.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.143+0000 7fbc4bc9b640 1 --2- 172.21.15.88:0/477215651 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc44072700 0x7fbc4415bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:36.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.143+0000 7fbc4bc9b640 1 --2- 172.21.15.88:0/477215651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc4415b480 0x7fbc440b5b80 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:36.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.143+0000 7fbc4bc9b640 1 -- 172.21.15.88:0/477215651 >> 172.21.15.88:0/477215651 conn(0x7fbc4406d0f0 msgr2=0x7fbc4406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:36.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.143+0000 7fbc4bc9b640 1 -- 172.21.15.88:0/477215651 shutdown_connections 2024-09-16T01:36:36.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.143+0000 7fbc4bc9b640 1 -- 172.21.15.88:0/477215651 wait complete. 2024-09-16T01:36:36.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.144+0000 7fbc4bc9b640 1 Processor -- start 2024-09-16T01:36:36.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.144+0000 7fbc4bc9b640 1 -- start start 2024-09-16T01:36:36.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.145+0000 7fbc4bc9b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc44072700 0x7fbc44150ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:36.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.145+0000 7fbc4bc9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc440b60c0 0x7fbc441469a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:36.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.145+0000 7fbc4bc9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc4415b480 0x7fbc4414eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:36.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.145+0000 7fbc4bc9b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4415b9e0 con 0x7fbc440b60c0 2024-09-16T01:36:36.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.145+0000 7fbc4bc9b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4415b860 con 0x7fbc4415b480 2024-09-16T01:36:36.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.145+0000 7fbc4bc9b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbc4415c1a0 con 0x7fbc44072700 2024-09-16T01:36:36.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.145+0000 7fbc49236640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc440b60c0 0x7fbc441469a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:36.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.146+0000 7fbc49236640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc440b60c0 0x7fbc441469a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42844/0 (socket says 172.21.15.88:42844) 2024-09-16T01:36:36.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.146+0000 7fbc49236640 1 -- 172.21.15.88:0/947320054 learned_addr learned my addr 172.21.15.88:0/947320054 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:36.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.146+0000 7fbc49a37640 1 --2- 172.21.15.88:0/947320054 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc44072700 0x7fbc44150ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:36.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.146+0000 7fbc4a238640 1 --2- 172.21.15.88:0/947320054 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc4415b480 0x7fbc4414eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:36.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.146+0000 7fbc49236640 1 -- 172.21.15.88:0/947320054 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc44072700 msgr2=0x7fbc44150ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:36.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.146+0000 7fbc49236640 1 --2- 172.21.15.88:0/947320054 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc44072700 0x7fbc44150ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:36.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.146+0000 7fbc49236640 1 -- 172.21.15.88:0/947320054 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc4415b480 msgr2=0x7fbc4414eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:36.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.146+0000 7fbc49236640 1 --2- 172.21.15.88:0/947320054 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc4415b480 0x7fbc4414eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:36.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.146+0000 7fbc49236640 1 -- 172.21.15.88:0/947320054 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc44146ee0 con 0x7fbc440b60c0 2024-09-16T01:36:36.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.146+0000 7fbc49236640 1 --2- 172.21.15.88:0/947320054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc440b60c0 0x7fbc441469a0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fbc3400c430 tx=0x7fbc3400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:36.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.147+0000 7fbc3a7fc640 1 -- 172.21.15.88:0/947320054 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc34017070 con 0x7fbc440b60c0 2024-09-16T01:36:36.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.147+0000 7fbc3a7fc640 1 -- 172.21.15.88:0/947320054 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc34005ce0 con 0x7fbc440b60c0 2024-09-16T01:36:36.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.147+0000 7fbc4bc9b640 1 -- 172.21.15.88:0/947320054 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc4414f700 con 0x7fbc440b60c0 2024-09-16T01:36:36.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.147+0000 7fbc3a7fc640 1 -- 172.21.15.88:0/947320054 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc340072f0 con 0x7fbc440b60c0 2024-09-16T01:36:36.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.147+0000 7fbc4bc9b640 1 -- 172.21.15.88:0/947320054 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc4414fcc0 con 0x7fbc440b60c0 2024-09-16T01:36:36.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.148+0000 7fbc4bc9b640 1 -- 172.21.15.88:0/947320054 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc4415bc60 con 0x7fbc440b60c0 2024-09-16T01:36:36.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.149+0000 7fbc3a7fc640 1 -- 172.21.15.88:0/947320054 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fbc3401a070 con 0x7fbc440b60c0 2024-09-16T01:36:36.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.150+0000 7fbc3a7fc640 1 --2- 172.21.15.88:0/947320054 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc2806c2b0 0x7fbc2806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:36.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.150+0000 7fbc3a7fc640 1 -- 172.21.15.88:0/947320054 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5029+0+0 (secure 0 0 0) 0x7fbc3408a600 con 0x7fbc440b60c0 2024-09-16T01:36:36.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.150+0000 7fbc49a37640 1 --2- 172.21.15.88:0/947320054 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc2806c2b0 0x7fbc2806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:36.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.150+0000 7fbc49a37640 1 --2- 172.21.15.88:0/947320054 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc2806c2b0 0x7fbc2806e770 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fbc40034000 tx=0x7fbc4002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:36.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.156+0000 7fbc3a7fc640 1 -- 172.21.15.88:0/947320054 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fbc34058930 con 0x7fbc440b60c0 2024-09-16T01:36:36.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:36.332+0000 7fbc4bc9b640 1 -- 172.21.15.88:0/947320054 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fbc440b5b80 con 0x7fbc440b60c0 2024-09-16T01:36:37.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.151+0000 7fbc3a7fc640 1 -- 172.21.15.88:0/947320054 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v54) ==== 97+0+0 (secure 0 0 0) 0x7fbc3405cc30 con 0x7fbc440b60c0 2024-09-16T01:36:37.153 INFO:teuthology.orchestra.run.smithi088.stderr: removed all blocklist entries 2024-09-16T01:36:37.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.154+0000 7fbc4bc9b640 1 -- 172.21.15.88:0/947320054 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc2806c2b0 msgr2=0x7fbc2806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:37.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.154+0000 7fbc4bc9b640 1 --2- 172.21.15.88:0/947320054 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc2806c2b0 0x7fbc2806e770 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fbc40034000 tx=0x7fbc4002e000 comp rx=0 tx=0).stop 2024-09-16T01:36:37.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.154+0000 7fbc4bc9b640 1 -- 172.21.15.88:0/947320054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc440b60c0 msgr2=0x7fbc441469a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:37.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.154+0000 7fbc4bc9b640 1 --2- 172.21.15.88:0/947320054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc440b60c0 0x7fbc441469a0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fbc3400c430 tx=0x7fbc3400c8f0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.154+0000 7fbc4bc9b640 1 -- 172.21.15.88:0/947320054 shutdown_connections 2024-09-16T01:36:37.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.154+0000 7fbc4bc9b640 1 --2- 172.21.15.88:0/947320054 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc2806c2b0 0x7fbc2806e770 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.154+0000 7fbc4bc9b640 1 --2- 172.21.15.88:0/947320054 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc4415b480 0x7fbc4414eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.154+0000 7fbc4bc9b640 1 --2- 172.21.15.88:0/947320054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc440b60c0 0x7fbc441469a0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.154+0000 7fbc4bc9b640 1 --2- 172.21.15.88:0/947320054 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc44072700 0x7fbc44150ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.154+0000 7fbc4bc9b640 1 -- 172.21.15.88:0/947320054 >> 172.21.15.88:0/947320054 conn(0x7fbc4406d0f0 msgr2=0x7fbc4415d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:37.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.154+0000 7fbc4bc9b640 1 -- 172.21.15.88:0/947320054 shutdown_connections 2024-09-16T01:36:37.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.154+0000 7fbc4bc9b640 1 -- 172.21.15.88:0/947320054 wait complete. 2024-09-16T01:36:37.169 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-16T01:36:37.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.272+0000 7fe26d99f640 1 Processor -- start 2024-09-16T01:36:37.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.272+0000 7fe26d99f640 1 -- start start 2024-09-16T01:36:37.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.272+0000 7fe26d99f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe268159f90 0x7fe2680ce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:37.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.272+0000 7fe26d99f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe2680b5310 0x7fe2680b6b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:37.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.272+0000 7fe26d99f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe26815cda0 con 0x7fe26815c8a0 2024-09-16T01:36:37.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.272+0000 7fe26d99f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe2680ced40 con 0x7fe2680b5310 2024-09-16T01:36:37.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.272+0000 7fe26d99f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe26815d080 con 0x7fe268159f90 2024-09-16T01:36:37.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.273+0000 7fe267fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe2680b5310 0x7fe2680b6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:37.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.273+0000 7fe267fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe2680b5310 0x7fe2680b6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42848/0 (socket says 172.21.15.88:42848) 2024-09-16T01:36:37.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.273+0000 7fe267fff640 1 -- 172.21.15.88:0/4074028451 learned_addr learned my addr 172.21.15.88:0/4074028451 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:37.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.273+0000 7fe26c99d640 1 --2- 172.21.15.88:0/4074028451 >> v2:172.21.15.154:3300/0 conn(0x7fe268159f90 0x7fe2680ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:37.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.273+0000 7fe267fff640 1 -- 172.21.15.88:0/4074028451 >> v2:172.21.15.154:3300/0 conn(0x7fe268159f90 msgr2=0x7fe2680ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:37.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.273+0000 7fe267fff640 1 --2- 172.21.15.88:0/4074028451 >> v2:172.21.15.154:3300/0 conn(0x7fe268159f90 0x7fe2680ce800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.273+0000 7fe267fff640 1 -- 172.21.15.88:0/4074028451 >> v1:172.21.15.154:6789/0 conn(0x7fe26815c8a0 legacy=0x7fe26815fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:37.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.273+0000 7fe267fff640 1 -- 172.21.15.88:0/4074028451 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe26815d360 con 0x7fe2680b5310 2024-09-16T01:36:37.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.274+0000 7fe267fff640 1 --2- 172.21.15.88:0/4074028451 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2680b5310 0x7fe2680b6b10 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe268050130 tx=0x7fe25402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cf146c794f78fa0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:37.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.274+0000 7fe2677fe640 1 -- 172.21.15.88:0/4074028451 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe254037070 con 0x7fe2680b5310 2024-09-16T01:36:37.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.274+0000 7fe2677fe640 1 -- 172.21.15.88:0/4074028451 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe254032440 con 0x7fe2680b5310 2024-09-16T01:36:37.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.274+0000 7fe2677fe640 1 -- 172.21.15.88:0/4074028451 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe254032810 con 0x7fe2680b5310 2024-09-16T01:36:37.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.274+0000 7fe26d99f640 1 -- 172.21.15.88:0/4074028451 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2680b5310 msgr2=0x7fe2680b6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:37.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.274+0000 7fe26d99f640 1 --2- 172.21.15.88:0/4074028451 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2680b5310 0x7fe2680b6b10 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe268050130 tx=0x7fe25402a230 comp rx=0 tx=0).stop 2024-09-16T01:36:37.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.274+0000 7fe26d99f640 1 -- 172.21.15.88:0/4074028451 shutdown_connections 2024-09-16T01:36:37.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.274+0000 7fe26d99f640 1 --2- 172.21.15.88:0/4074028451 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2680b5310 0x7fe2680b6b10 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.274+0000 7fe26d99f640 1 --2- 172.21.15.88:0/4074028451 >> v2:172.21.15.154:3300/0 conn(0x7fe268159f90 0x7fe2680ce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.274+0000 7fe26d99f640 1 -- 172.21.15.88:0/4074028451 >> 172.21.15.88:0/4074028451 conn(0x7fe2680b4010 msgr2=0x7fe2680b4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:37.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.274+0000 7fe26d99f640 1 -- 172.21.15.88:0/4074028451 shutdown_connections 2024-09-16T01:36:37.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.274+0000 7fe26d99f640 1 -- 172.21.15.88:0/4074028451 wait complete. 2024-09-16T01:36:37.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.276+0000 7fe26d99f640 1 Processor -- start 2024-09-16T01:36:37.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.276+0000 7fe26d99f640 1 -- start start 2024-09-16T01:36:37.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.276+0000 7fe26d99f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2680b5310 0x7fe26814ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:37.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.276+0000 7fe26d99f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe268159f90 0x7fe26815dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:37.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.276+0000 7fe26d99f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe26815c8a0 0x7fe26815e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:37.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.276+0000 7fe26d99f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe2680b4e40 con 0x7fe2680b5310 2024-09-16T01:36:37.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.276+0000 7fe26d99f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe2680b4cc0 con 0x7fe26815c8a0 2024-09-16T01:36:37.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.276+0000 7fe26d99f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe2680b7050 con 0x7fe268159f90 2024-09-16T01:36:37.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.277+0000 7fe26c99d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2680b5310 0x7fe26814ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:37.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.277+0000 7fe26c99d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2680b5310 0x7fe26814ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42852/0 (socket says 172.21.15.88:42852) 2024-09-16T01:36:37.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.277+0000 7fe26c99d640 1 -- 172.21.15.88:0/2325233396 learned_addr learned my addr 172.21.15.88:0/2325233396 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:37.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.277+0000 7fe26d19e640 1 --2- 172.21.15.88:0/2325233396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe26815c8a0 0x7fe26815e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:37.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.277+0000 7fe267fff640 1 --2- 172.21.15.88:0/2325233396 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe268159f90 0x7fe26815dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:37.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.277+0000 7fe26c99d640 1 -- 172.21.15.88:0/2325233396 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe268159f90 msgr2=0x7fe26815dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:37.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.277+0000 7fe26c99d640 1 --2- 172.21.15.88:0/2325233396 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe268159f90 0x7fe26815dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.277+0000 7fe26c99d640 1 -- 172.21.15.88:0/2325233396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe26815c8a0 msgr2=0x7fe26815e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:37.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.277+0000 7fe26c99d640 1 --2- 172.21.15.88:0/2325233396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe26815c8a0 0x7fe26815e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.277+0000 7fe26c99d640 1 -- 172.21.15.88:0/2325233396 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2681577b0 con 0x7fe2680b5310 2024-09-16T01:36:37.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.277+0000 7fe26c99d640 1 --2- 172.21.15.88:0/2325233396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2680b5310 0x7fe26814ffa0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00d800 tx=0x7fe25c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:37.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.277+0000 7fe2657fa640 1 -- 172.21.15.88:0/2325233396 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25c00b6c0 con 0x7fe2680b5310 2024-09-16T01:36:37.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.277+0000 7fe2657fa640 1 -- 172.21.15.88:0/2325233396 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe25c019070 con 0x7fe2680b5310 2024-09-16T01:36:37.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.277+0000 7fe2657fa640 1 -- 172.21.15.88:0/2325233396 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25c00ba80 con 0x7fe2680b5310 2024-09-16T01:36:37.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.277+0000 7fe26d99f640 1 -- 172.21.15.88:0/2325233396 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe268157a50 con 0x7fe2680b5310 2024-09-16T01:36:37.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.277+0000 7fe26d99f640 1 -- 172.21.15.88:0/2325233396 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe268158010 con 0x7fe2680b5310 2024-09-16T01:36:37.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.278+0000 7fe26d99f640 1 -- 172.21.15.88:0/2325233396 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2680b73e0 con 0x7fe2680b5310 2024-09-16T01:36:37.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.279+0000 7fe2657fa640 1 -- 172.21.15.88:0/2325233396 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe25c00bc20 con 0x7fe2680b5310 2024-09-16T01:36:37.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.279+0000 7fe2657fa640 1 --2- 172.21.15.88:0/2325233396 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe24c06c2b0 0x7fe24c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:37.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.279+0000 7fe267fff640 1 --2- 172.21.15.88:0/2325233396 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe24c06c2b0 0x7fe24c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:37.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.279+0000 7fe2657fa640 1 -- 172.21.15.88:0/2325233396 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 4857+0+0 (secure 0 0 0) 0x7fe25c08b480 con 0x7fe2680b5310 2024-09-16T01:36:37.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.280+0000 7fe267fff640 1 --2- 172.21.15.88:0/2325233396 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe24c06c2b0 0x7fe24c06e770 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe254034000 tx=0x7fe25402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:37.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.284+0000 7fe2657fa640 1 -- 172.21.15.88:0/2325233396 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe25c059fd0 con 0x7fe2680b5310 2024-09-16T01:36:37.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.516+0000 7fe26d99f640 1 -- 172.21.15.88:0/2325233396 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fe2680b1c60 con 0x7fe2680b5310 2024-09-16T01:36:37.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.516+0000 7fe2657fa640 1 -- 172.21.15.88:0/2325233396 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) ==== 80+0+5304 (secure 0 0 0) 0x7fe25c0598d0 con 0x7fe2680b5310 2024-09-16T01:36:37.518 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:37.518 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:36:37.518 INFO:teuthology.orchestra.run.smithi088.stdout: "auth_dump": [ 2024-09-16T01:36:37.518 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:37.518 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mds.a", 2024-09-16T01:36:37.518 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmy+jJCRAAhnzX/li6D3D07NmThrjXkA==", 2024-09-16T01:36:37.519 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:37.519 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:36:37.519 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:36:37.519 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:36:37.519 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:37.519 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:37.519 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:37.519 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:37.519 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mds.b", 2024-09-16T01:36:37.519 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmx7p+FRAApxKUSZoR/nGUZsaYKlrYUg==", 2024-09-16T01:36:37.519 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:37.519 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:36:37.519 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:36:37.520 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:36:37.520 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:37.520 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:37.520 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:37.520 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:37.520 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mds.c", 2024-09-16T01:36:37.520 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmaYHUEBAAzM0qE53LEgHi+hThHJX3Tw==", 2024-09-16T01:36:37.520 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:37.520 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:36:37.520 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:36:37.520 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:36:37.521 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:37.521 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:37.521 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:37.521 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:37.521 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mds.d", 2024-09-16T01:36:37.521 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudm3TSLHBAAJzlR6H09cc1Pki2FjyoFiA==", 2024-09-16T01:36:37.521 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:37.521 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:36:37.521 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:36:37.521 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:36:37.521 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:37.521 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:37.521 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:37.521 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:37.522 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.0", 2024-09-16T01:36:37.522 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDhiudmZR7MORAAwDKvphubA+CbzBUWuaA80A==", 2024-09-16T01:36:37.522 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:37.522 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:36:37.522 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:36:37.522 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:37.522 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:37.522 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:37.522 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:37.522 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.1", 2024-09-16T01:36:37.522 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDliudmNd+SLBAArOUdoZ226c5jlAsKwqERmw==", 2024-09-16T01:36:37.522 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:37.522 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:36:37.522 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:36:37.523 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:37.523 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:37.523 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:37.523 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:37.523 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.2", 2024-09-16T01:36:37.523 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDpiudm75CUARAAWHtYOg9pE8g+M6IL+8XZ6g==", 2024-09-16T01:36:37.523 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:37.523 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:36:37.523 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:36:37.523 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:37.523 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:37.523 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:37.523 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:37.523 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.3", 2024-09-16T01:36:37.524 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDtiudm0fgWAxAAf2NOWrpXbOhb3dFXozJagQ==", 2024-09-16T01:36:37.524 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:37.524 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:36:37.524 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:36:37.524 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:37.524 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:37.524 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:37.524 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:37.524 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.4", 2024-09-16T01:36:37.524 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQD0iudmdddMCxAAtr2qfj4Nxj3/GHUpxRmtFg==", 2024-09-16T01:36:37.524 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:37.524 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:36:37.524 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:36:37.524 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:37.525 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:37.525 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:37.525 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:37.525 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.5", 2024-09-16T01:36:37.525 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQD3iudmT5RjOhAAcKwca6BG60gsxYnmy4twGg==", 2024-09-16T01:36:37.525 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:37.525 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:36:37.525 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:36:37.525 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:37.525 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:37.525 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:37.525 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:37.525 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.6", 2024-09-16T01:36:37.525 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQD7iudmT7daLRAAdfLFijKQku3rJcJ5xcfLsw==", 2024-09-16T01:36:37.526 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:37.526 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:36:37.526 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:36:37.526 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:37.526 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:37.526 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:37.526 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:37.526 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.7", 2024-09-16T01:36:37.526 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQD/iudmmvtxLhAArYKhQ4zyFHtLgh/+bs5ZhA==", 2024-09-16T01:36:37.526 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:37.526 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:36:37.526 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:36:37.526 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:37.526 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:37.526 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:37.527 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:37.527 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.0", 2024-09-16T01:36:37.527 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmvPikIBAAyHxNNnwQKygxg09PNQLc/Q==", 2024-09-16T01:36:37.527 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:37.527 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:36:37.527 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow r", 2024-09-16T01:36:37.527 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-16T01:36:37.527 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:37.527 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:37.527 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:37.527 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.1", 2024-09-16T01:36:37.527 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmA69iKRAA8Abwe0Lg2PnZDErgJ1gEbQ==", 2024-09-16T01:36:37.527 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:37.527 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:36:37.528 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow r", 2024-09-16T01:36:37.528 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow rw", 2024-09-16T01:36:37.528 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow rwx" 2024-09-16T01:36:37.528 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:37.528 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:37.528 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:37.528 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.2", 2024-09-16T01:36:37.528 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmB+w9JhAAv63Is5T/cvDbbLOjIkI0wA==", 2024-09-16T01:36:37.528 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:37.528 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:36:37.528 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow r", 2024-09-16T01:36:37.528 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow rw", 2024-09-16T01:36:37.529 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow rwx" 2024-09-16T01:36:37.529 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:37.529 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:37.529 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:37.529 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.3", 2024-09-16T01:36:37.529 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudm4WoFLxAAnYGHGqHyBHHajQ8f3rCjEA==", 2024-09-16T01:36:37.529 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:37.529 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:36:37.529 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow r", 2024-09-16T01:36:37.529 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow rw", 2024-09-16T01:36:37.529 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow rwx" 2024-09-16T01:36:37.529 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:37.530 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:37.530 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:37.530 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.admin", 2024-09-16T01:36:37.530 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDdiudmhcE0KBAANauQKyUY01/KlWMrN2x/5Q==", 2024-09-16T01:36:37.530 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:37.530 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow *", 2024-09-16T01:36:37.530 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:36:37.530 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:36:37.530 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:37.530 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:37.531 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:37.531 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:37.531 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mgr.x", 2024-09-16T01:36:37.531 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmToGYBhAAKwXS34PtfAVtEtdtHKHlQQ==", 2024-09-16T01:36:37.531 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:37.531 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow *", 2024-09-16T01:36:37.531 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile mgr", 2024-09-16T01:36:37.531 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:37.531 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:37.531 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:36:37.531 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:37.532 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mgr.y", 2024-09-16T01:36:37.532 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudm/BN5ARAAxbCopbsvxFUZP3drso32qw==", 2024-09-16T01:36:37.532 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:36:37.532 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow *", 2024-09-16T01:36:37.532 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile mgr", 2024-09-16T01:36:37.532 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:36:37.532 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:37.532 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:37.532 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-09-16T01:36:37.532 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:36:37.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.519+0000 7fe26d99f640 1 -- 172.21.15.88:0/2325233396 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe24c06c2b0 msgr2=0x7fe24c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:37.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.519+0000 7fe26d99f640 1 --2- 172.21.15.88:0/2325233396 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe24c06c2b0 0x7fe24c06e770 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe254034000 tx=0x7fe25402e000 comp rx=0 tx=0).stop 2024-09-16T01:36:37.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.519+0000 7fe26d99f640 1 -- 172.21.15.88:0/2325233396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2680b5310 msgr2=0x7fe26814ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:37.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.519+0000 7fe26d99f640 1 --2- 172.21.15.88:0/2325233396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2680b5310 0x7fe26814ffa0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00d800 tx=0x7fe25c00dcc0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.519+0000 7fe26d99f640 1 -- 172.21.15.88:0/2325233396 shutdown_connections 2024-09-16T01:36:37.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.519+0000 7fe26d99f640 1 --2- 172.21.15.88:0/2325233396 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe24c06c2b0 0x7fe24c06e770 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.519+0000 7fe26d99f640 1 --2- 172.21.15.88:0/2325233396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe26815c8a0 0x7fe26815e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.519+0000 7fe26d99f640 1 --2- 172.21.15.88:0/2325233396 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe268159f90 0x7fe26815dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.519+0000 7fe26d99f640 1 --2- 172.21.15.88:0/2325233396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2680b5310 0x7fe26814ffa0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.519+0000 7fe26d99f640 1 -- 172.21.15.88:0/2325233396 >> 172.21.15.88:0/2325233396 conn(0x7fe2680b4010 msgr2=0x7fe2680b3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:37.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.519+0000 7fe26d99f640 1 -- 172.21.15.88:0/2325233396 shutdown_connections 2024-09-16T01:36:37.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.519+0000 7fe26d99f640 1 -- 172.21.15.88:0/2325233396 wait complete. 2024-09-16T01:36:37.535 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-16T01:36:37.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.632+0000 7ff3b843c640 1 Processor -- start 2024-09-16T01:36:37.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.632+0000 7ff3b843c640 1 -- start start 2024-09-16T01:36:37.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.633+0000 7ff3b843c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff3b013f400 0x7ff3b00cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:37.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.633+0000 7ff3b843c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3b0145bb0 0x7ff3b006e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:37.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.633+0000 7ff3b843c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3b00d03c0 con 0x7ff3b006e5f0 2024-09-16T01:36:37.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.633+0000 7ff3b843c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff3b006ea40 con 0x7ff3b013f400 2024-09-16T01:36:37.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.633+0000 7ff3b843c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff3b00d06a0 con 0x7ff3b0145bb0 2024-09-16T01:36:37.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.633+0000 7ff3b61d8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff3b013f400 0x7ff3b00cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:37.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.633+0000 7ff3b61d8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff3b013f400 0x7ff3b00cfab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42862/0 (socket says 172.21.15.88:42862) 2024-09-16T01:36:37.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.633+0000 7ff3b61d8640 1 -- 172.21.15.88:0/1809075845 learned_addr learned my addr 172.21.15.88:0/1809075845 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:37.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.633+0000 7ff3b61d8640 1 -- 172.21.15.88:0/1809075845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3b0145bb0 msgr2=0x7ff3b006e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:37.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.633+0000 7ff3b61d8640 1 --2- 172.21.15.88:0/1809075845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3b0145bb0 0x7ff3b006e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.633+0000 7ff3b61d8640 1 -- 172.21.15.88:0/1809075845 >> v1:172.21.15.154:6789/0 conn(0x7ff3b006e5f0 legacy=0x7ff3b015ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:37.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.633+0000 7ff3b61d8640 1 -- 172.21.15.88:0/1809075845 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3b01431e0 con 0x7ff3b013f400 2024-09-16T01:36:37.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.634+0000 7ff3b61d8640 1 --2- 172.21.15.88:0/1809075845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b013f400 0x7ff3b00cfab0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac009870 tx=0x7ff3ac02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82984394aa9ad7ed server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:37.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.634+0000 7ff3b51d6640 1 -- 172.21.15.88:0/1809075845 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac037070 con 0x7ff3b013f400 2024-09-16T01:36:37.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.634+0000 7ff3b51d6640 1 -- 172.21.15.88:0/1809075845 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3ac032440 con 0x7ff3b013f400 2024-09-16T01:36:37.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.634+0000 7ff3b51d6640 1 -- 172.21.15.88:0/1809075845 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac032810 con 0x7ff3b013f400 2024-09-16T01:36:37.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.635+0000 7ff3b843c640 1 -- 172.21.15.88:0/1809075845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b013f400 msgr2=0x7ff3b00cfab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:37.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.635+0000 7ff3b843c640 1 --2- 172.21.15.88:0/1809075845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b013f400 0x7ff3b00cfab0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac009870 tx=0x7ff3ac02a180 comp rx=0 tx=0).stop 2024-09-16T01:36:37.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.635+0000 7ff3b843c640 1 -- 172.21.15.88:0/1809075845 shutdown_connections 2024-09-16T01:36:37.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.635+0000 7ff3b843c640 1 --2- 172.21.15.88:0/1809075845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3b0145bb0 0x7ff3b006e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.635+0000 7ff3b843c640 1 --2- 172.21.15.88:0/1809075845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b013f400 0x7ff3b00cfab0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.635+0000 7ff3b843c640 1 -- 172.21.15.88:0/1809075845 >> 172.21.15.88:0/1809075845 conn(0x7ff3b006bc10 msgr2=0x7ff3b006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:37.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.635+0000 7ff3b843c640 1 -- 172.21.15.88:0/1809075845 shutdown_connections 2024-09-16T01:36:37.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.635+0000 7ff3b843c640 1 -- 172.21.15.88:0/1809075845 wait complete. 2024-09-16T01:36:37.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.636+0000 7ff3b843c640 1 Processor -- start 2024-09-16T01:36:37.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.636+0000 7ff3b843c640 1 -- start start 2024-09-16T01:36:37.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.637+0000 7ff3b843c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b006e5f0 0x7ff3b0158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:37.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.637+0000 7ff3b843c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3b013f400 0x7ff3b01f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:37.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.637+0000 7ff3b843c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3b0145bb0 0x7ff3b01e61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:37.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.637+0000 7ff3b843c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b00d0170 con 0x7ff3b006e5f0 2024-09-16T01:36:37.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.637+0000 7ff3b843c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b00cfff0 con 0x7ff3b013f400 2024-09-16T01:36:37.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.637+0000 7ff3b843c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff3b013f7e0 con 0x7ff3b0145bb0 2024-09-16T01:36:37.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.637+0000 7ff3b61d8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b006e5f0 0x7ff3b0158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:37.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.637+0000 7ff3b61d8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b006e5f0 0x7ff3b0158a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42878/0 (socket says 172.21.15.88:42878) 2024-09-16T01:36:37.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.637+0000 7ff3b61d8640 1 -- 172.21.15.88:0/3059774315 learned_addr learned my addr 172.21.15.88:0/3059774315 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:37.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.637+0000 7ff3b69d9640 1 --2- 172.21.15.88:0/3059774315 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3b0145bb0 0x7ff3b01e61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:37.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.637+0000 7ff3b59d7640 1 --2- 172.21.15.88:0/3059774315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3b013f400 0x7ff3b01f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:37.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.637+0000 7ff3b61d8640 1 -- 172.21.15.88:0/3059774315 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3b0145bb0 msgr2=0x7ff3b01e61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:37.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.637+0000 7ff3b61d8640 1 --2- 172.21.15.88:0/3059774315 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3b0145bb0 0x7ff3b01e61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.637+0000 7ff3b61d8640 1 -- 172.21.15.88:0/3059774315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3b013f400 msgr2=0x7ff3b01f29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:37.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.637+0000 7ff3b61d8640 1 --2- 172.21.15.88:0/3059774315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3b013f400 0x7ff3b01f29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.637+0000 7ff3b61d8640 1 -- 172.21.15.88:0/3059774315 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3b01e6730 con 0x7ff3b006e5f0 2024-09-16T01:36:37.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.638+0000 7ff3b61d8640 1 --2- 172.21.15.88:0/3059774315 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b006e5f0 0x7ff3b0158a80 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac009870 tx=0x7ff3ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:37.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.638+0000 7ff3a6ffd640 1 -- 172.21.15.88:0/3059774315 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac041070 con 0x7ff3b006e5f0 2024-09-16T01:36:37.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.638+0000 7ff3a6ffd640 1 -- 172.21.15.88:0/3059774315 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3ac03c870 con 0x7ff3b006e5f0 2024-09-16T01:36:37.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.638+0000 7ff3b843c640 1 -- 172.21.15.88:0/3059774315 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3b01f5d00 con 0x7ff3b006e5f0 2024-09-16T01:36:37.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.638+0000 7ff3a6ffd640 1 -- 172.21.15.88:0/3059774315 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac037040 con 0x7ff3b006e5f0 2024-09-16T01:36:37.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.638+0000 7ff3b843c640 1 -- 172.21.15.88:0/3059774315 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3b00b4990 con 0x7ff3b006e5f0 2024-09-16T01:36:37.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.639+0000 7ff3b843c640 1 -- 172.21.15.88:0/3059774315 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3b0076170 con 0x7ff3b006e5f0 2024-09-16T01:36:37.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.639+0000 7ff3a6ffd640 1 -- 172.21.15.88:0/3059774315 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff3ac044070 con 0x7ff3b006e5f0 2024-09-16T01:36:37.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.640+0000 7ff3a6ffd640 1 --2- 172.21.15.88:0/3059774315 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff39406c2b0 0x7ff39406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:37.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.640+0000 7ff3a6ffd640 1 -- 172.21.15.88:0/3059774315 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 4857+0+0 (secure 0 0 0) 0x7ff3ac0abe30 con 0x7ff3b006e5f0 2024-09-16T01:36:37.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.640+0000 7ff3b59d7640 1 --2- 172.21.15.88:0/3059774315 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff39406c2b0 0x7ff39406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:37.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.640+0000 7ff3b59d7640 1 --2- 172.21.15.88:0/3059774315 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff39406c2b0 0x7ff39406e770 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff3b015ccc0 tx=0x7ff3a0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:37.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.644+0000 7ff3a6ffd640 1 -- 172.21.15.88:0/3059774315 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff3b0076170 con 0x7ff3b006e5f0 2024-09-16T01:36:37.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.809+0000 7ff3b843c640 1 -- 172.21.15.88:0/3059774315 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7ff3b00cfab0 con 0x7ff3b006e5f0 2024-09-16T01:36:37.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.810+0000 7ff3a6ffd640 1 -- 172.21.15.88:0/3059774315 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v54) ==== 84+0+65 (secure 0 0 0) 0x7ff3ac07aa10 con 0x7ff3b006e5f0 2024-09-16T01:36:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:37.811 INFO:teuthology.orchestra.run.smithi088.stdout:[ 2024-09-16T01:36:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:36:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: "poolnum": 1, 2024-09-16T01:36:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: "poolname": ".mgr" 2024-09-16T01:36:37.811 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:36:37.812 INFO:teuthology.orchestra.run.smithi088.stdout:] 2024-09-16T01:36:37.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.811+0000 7ff3b843c640 1 -- 172.21.15.88:0/3059774315 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff39406c2b0 msgr2=0x7ff39406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:37.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.811+0000 7ff3b843c640 1 --2- 172.21.15.88:0/3059774315 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff39406c2b0 0x7ff39406e770 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff3b015ccc0 tx=0x7ff3a0008040 comp rx=0 tx=0).stop 2024-09-16T01:36:37.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.811+0000 7ff3b843c640 1 -- 172.21.15.88:0/3059774315 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b006e5f0 msgr2=0x7ff3b0158a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:37.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.811+0000 7ff3b843c640 1 --2- 172.21.15.88:0/3059774315 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b006e5f0 0x7ff3b0158a80 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac009870 tx=0x7ff3ac0046c0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.812+0000 7ff3b843c640 1 -- 172.21.15.88:0/3059774315 shutdown_connections 2024-09-16T01:36:37.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.812+0000 7ff3b843c640 1 --2- 172.21.15.88:0/3059774315 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff39406c2b0 0x7ff39406e770 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.812+0000 7ff3b843c640 1 --2- 172.21.15.88:0/3059774315 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3b0145bb0 0x7ff3b01e61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.812+0000 7ff3b843c640 1 --2- 172.21.15.88:0/3059774315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3b013f400 0x7ff3b01f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.812+0000 7ff3b843c640 1 --2- 172.21.15.88:0/3059774315 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b006e5f0 0x7ff3b0158a80 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.812+0000 7ff3b843c640 1 -- 172.21.15.88:0/3059774315 >> 172.21.15.88:0/3059774315 conn(0x7ff3b006bc10 msgr2=0x7ff3b006bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:37.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.812+0000 7ff3b843c640 1 -- 172.21.15.88:0/3059774315 shutdown_connections 2024-09-16T01:36:37.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.812+0000 7ff3b843c640 1 -- 172.21.15.88:0/3059774315 wait complete. 2024-09-16T01:36:37.824 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-16T01:36:37.824 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-16T01:36:37.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.925+0000 7fef08768640 1 Processor -- start 2024-09-16T01:36:37.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.925+0000 7fef08768640 1 -- start start 2024-09-16T01:36:37.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.926+0000 7fef08768640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fef0006fb10 0x7fef000b7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:37.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.926+0000 7fef08768640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef00071450 0x7fef000b5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:37.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.926+0000 7fef08768640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef000719c0 con 0x7fef0015c490 2024-09-16T01:36:37.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.926+0000 7fef08768640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fef000b6860 con 0x7fef0006fb10 2024-09-16T01:36:37.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.926+0000 7fef08768640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fef000b6490 con 0x7fef00071450 2024-09-16T01:36:37.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.926+0000 7fef05d03640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fef0006fb10 0x7fef000b7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:37.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.926+0000 7fef05d03640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fef0006fb10 0x7fef000b7590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42880/0 (socket says 172.21.15.88:42880) 2024-09-16T01:36:37.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.926+0000 7fef05d03640 1 -- 172.21.15.88:0/3968131317 learned_addr learned my addr 172.21.15.88:0/3968131317 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:37.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.926+0000 7fef06d05640 1 --2- 172.21.15.88:0/3968131317 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef00071450 0x7fef000b5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:37.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.927+0000 7fef05d03640 1 -- 172.21.15.88:0/3968131317 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef00071450 msgr2=0x7fef000b5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:37.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.927+0000 7fef05d03640 1 --2- 172.21.15.88:0/3968131317 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef00071450 0x7fef000b5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.927+0000 7fef05d03640 1 -- 172.21.15.88:0/3968131317 >> v1:172.21.15.154:6789/0 conn(0x7fef0015c490 legacy=0x7fef0006f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:37.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.927+0000 7fef05d03640 1 -- 172.21.15.88:0/3968131317 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef000b6ab0 con 0x7fef0006fb10 2024-09-16T01:36:37.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.928+0000 7fef05d03640 1 --2- 172.21.15.88:0/3968131317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef0006fb10 0x7fef000b7590 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7feef4009870 tx=0x7feef402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=210e332ed24c3c25 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:37.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.928+0000 7fef05502640 1 -- 172.21.15.88:0/3968131317 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef4037070 con 0x7fef0006fb10 2024-09-16T01:36:37.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.928+0000 7fef05502640 1 -- 172.21.15.88:0/3968131317 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feef4032440 con 0x7fef0006fb10 2024-09-16T01:36:37.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.928+0000 7fef05502640 1 -- 172.21.15.88:0/3968131317 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef4032810 con 0x7fef0006fb10 2024-09-16T01:36:37.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.929+0000 7fef08768640 1 -- 172.21.15.88:0/3968131317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef0006fb10 msgr2=0x7fef000b7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:37.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.929+0000 7fef08768640 1 --2- 172.21.15.88:0/3968131317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef0006fb10 0x7fef000b7590 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7feef4009870 tx=0x7feef402a270 comp rx=0 tx=0).stop 2024-09-16T01:36:37.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.929+0000 7fef08768640 1 -- 172.21.15.88:0/3968131317 shutdown_connections 2024-09-16T01:36:37.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.929+0000 7fef08768640 1 --2- 172.21.15.88:0/3968131317 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef00071450 0x7fef000b5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.929+0000 7fef08768640 1 --2- 172.21.15.88:0/3968131317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef0006fb10 0x7fef000b7590 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.929+0000 7fef08768640 1 -- 172.21.15.88:0/3968131317 >> 172.21.15.88:0/3968131317 conn(0x7fef000b4240 msgr2=0x7fef000b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:37.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.929+0000 7fef08768640 1 -- 172.21.15.88:0/3968131317 shutdown_connections 2024-09-16T01:36:37.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.929+0000 7fef08768640 1 -- 172.21.15.88:0/3968131317 wait complete. 2024-09-16T01:36:37.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.930+0000 7fef08768640 1 Processor -- start 2024-09-16T01:36:37.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.930+0000 7fef08768640 1 -- start start 2024-09-16T01:36:37.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.931+0000 7fef08768640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef0006fb10 0x7fef00157c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:37.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.931+0000 7fef08768640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef00071450 0x7fef001581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:37.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.931+0000 7fef08768640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef0015c490 0x7fef000aab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:37.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.931+0000 7fef08768640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fef0015c870 con 0x7fef0015c490 2024-09-16T01:36:37.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.931+0000 7fef08768640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fef000b7070 con 0x7fef00071450 2024-09-16T01:36:37.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.931+0000 7fef08768640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fef000d0270 con 0x7fef0006fb10 2024-09-16T01:36:37.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.931+0000 7fef06d05640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef0015c490 0x7fef000aab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:37.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.931+0000 7fef06d05640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef0015c490 0x7fef000aab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42894/0 (socket says 172.21.15.88:42894) 2024-09-16T01:36:37.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.931+0000 7fef06d05640 1 -- 172.21.15.88:0/486883220 learned_addr learned my addr 172.21.15.88:0/486883220 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:37.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.931+0000 7fef06504640 1 --2- 172.21.15.88:0/486883220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef0006fb10 0x7fef00157c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:37.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.931+0000 7fef05d03640 1 --2- 172.21.15.88:0/486883220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef00071450 0x7fef001581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:37.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.931+0000 7fef06d05640 1 -- 172.21.15.88:0/486883220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef0006fb10 msgr2=0x7fef00157c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:37.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.931+0000 7fef06d05640 1 --2- 172.21.15.88:0/486883220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef0006fb10 0x7fef00157c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.931+0000 7fef06d05640 1 -- 172.21.15.88:0/486883220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef00071450 msgr2=0x7fef001581b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:37.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.931+0000 7fef06d05640 1 --2- 172.21.15.88:0/486883220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef00071450 0x7fef001581b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:37.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.931+0000 7fef06d05640 1 -- 172.21.15.88:0/486883220 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef000b3740 con 0x7fef0015c490 2024-09-16T01:36:37.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.931+0000 7fef06d05640 1 --2- 172.21.15.88:0/486883220 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef0015c490 0x7fef000aab20 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7feef800efc0 tx=0x7feef800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:37.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.931+0000 7feefeffd640 1 -- 172.21.15.88:0/486883220 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef8018020 con 0x7fef0015c490 2024-09-16T01:36:37.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.931+0000 7feefeffd640 1 -- 172.21.15.88:0/486883220 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feef8014480 con 0x7fef0015c490 2024-09-16T01:36:37.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.931+0000 7feefeffd640 1 -- 172.21.15.88:0/486883220 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feef8014870 con 0x7fef0015c490 2024-09-16T01:36:37.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.931+0000 7fef08768640 1 -- 172.21.15.88:0/486883220 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef000b3a20 con 0x7fef0015c490 2024-09-16T01:36:37.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.932+0000 7fef08768640 1 -- 172.21.15.88:0/486883220 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef0015e890 con 0x7fef0015c490 2024-09-16T01:36:37.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.933+0000 7fef08768640 1 -- 172.21.15.88:0/486883220 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef000b5f50 con 0x7fef0015c490 2024-09-16T01:36:37.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.933+0000 7feefeffd640 1 -- 172.21.15.88:0/486883220 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7feef8005d10 con 0x7fef0015c490 2024-09-16T01:36:37.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.933+0000 7feefeffd640 1 --2- 172.21.15.88:0/486883220 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7feeec06c2b0 0x7feeec06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:37.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.933+0000 7feefeffd640 1 -- 172.21.15.88:0/486883220 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 4857+0+0 (secure 0 0 0) 0x7feef808bbe0 con 0x7fef0015c490 2024-09-16T01:36:37.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.934+0000 7fef06504640 1 --2- 172.21.15.88:0/486883220 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7feeec06c2b0 0x7feeec06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:37.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.934+0000 7fef06504640 1 --2- 172.21.15.88:0/486883220 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7feeec06c2b0 0x7feeec06e770 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7feef0007fd0 tx=0x7feef0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:37.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:37.938+0000 7feefeffd640 1 -- 172.21.15.88:0/486883220 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7feef8022070 con 0x7fef0015c490 2024-09-16T01:36:38.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.102+0000 7fef08768640 1 -- 172.21.15.88:0/486883220 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fef000b3bf0 con 0x7fef0015c490 2024-09-16T01:36:38.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.438+0000 7feefeffd640 1 -- 172.21.15.88:0/486883220 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v55) ==== 138+0+0 (secure 0 0 0) 0x7feef8010040 con 0x7fef0015c490 2024-09-16T01:36:38.440 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_metadata' created 2024-09-16T01:36:38.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.440+0000 7fef08768640 1 -- 172.21.15.88:0/486883220 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7feeec06c2b0 msgr2=0x7feeec06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:38.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.440+0000 7fef08768640 1 --2- 172.21.15.88:0/486883220 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7feeec06c2b0 0x7feeec06e770 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7feef0007fd0 tx=0x7feef0009040 comp rx=0 tx=0).stop 2024-09-16T01:36:38.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.440+0000 7fef08768640 1 -- 172.21.15.88:0/486883220 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef0015c490 msgr2=0x7fef000aab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:38.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.440+0000 7fef08768640 1 --2- 172.21.15.88:0/486883220 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef0015c490 0x7fef000aab20 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7feef800efc0 tx=0x7feef800c490 comp rx=0 tx=0).stop 2024-09-16T01:36:38.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.441+0000 7fef08768640 1 -- 172.21.15.88:0/486883220 shutdown_connections 2024-09-16T01:36:38.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.441+0000 7fef08768640 1 --2- 172.21.15.88:0/486883220 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7feeec06c2b0 0x7feeec06e770 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:38.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.441+0000 7fef08768640 1 --2- 172.21.15.88:0/486883220 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef0015c490 0x7fef000aab20 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:38.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.441+0000 7fef08768640 1 --2- 172.21.15.88:0/486883220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef00071450 0x7fef001581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:38.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.441+0000 7fef08768640 1 --2- 172.21.15.88:0/486883220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef0006fb10 0x7fef00157c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:38.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.441+0000 7fef08768640 1 -- 172.21.15.88:0/486883220 >> 172.21.15.88:0/486883220 conn(0x7fef000b4240 msgr2=0x7fef000d06d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:38.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.441+0000 7fef08768640 1 -- 172.21.15.88:0/486883220 shutdown_connections 2024-09-16T01:36:38.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.441+0000 7fef08768640 1 -- 172.21.15.88:0/486883220 wait complete. 2024-09-16T01:36:38.456 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-16T01:36:38.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.560+0000 7f27d430c640 1 Processor -- start 2024-09-16T01:36:38.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.560+0000 7f27d430c640 1 -- start start 2024-09-16T01:36:38.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.561+0000 7f27d430c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f27cc0b53f0 0x7f27cc0d2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:38.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.561+0000 7f27d430c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27cc0d2650 0x7f27cc0ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:38.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.561+0000 7f27d430c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27cc0cdf50 con 0x7f27cc15c710 2024-09-16T01:36:38.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.561+0000 7f27d430c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f27cc0ce230 con 0x7f27cc0b53f0 2024-09-16T01:36:38.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.561+0000 7f27d430c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f27cc0ce510 con 0x7f27cc0d2650 2024-09-16T01:36:38.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.561+0000 7f27d18a7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f27cc0b53f0 0x7f27cc0d2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:38.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.561+0000 7f27d18a7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f27cc0b53f0 0x7f27cc0d2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42898/0 (socket says 172.21.15.88:42898) 2024-09-16T01:36:38.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.561+0000 7f27d18a7640 1 -- 172.21.15.88:0/573841274 learned_addr learned my addr 172.21.15.88:0/573841274 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:38.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.562+0000 7f27d18a7640 1 -- 172.21.15.88:0/573841274 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27cc0d2650 msgr2=0x7f27cc0ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:38.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.562+0000 7f27d28a9640 1 --2- 172.21.15.88:0/573841274 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27cc0d2650 0x7f27cc0ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:38.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.562+0000 7f27d18a7640 1 --2- 172.21.15.88:0/573841274 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27cc0d2650 0x7f27cc0ceff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:38.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.562+0000 7f27d18a7640 1 -- 172.21.15.88:0/573841274 >> v1:172.21.15.154:6789/0 conn(0x7f27cc15c710 legacy=0x7f27cc073970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:38.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.562+0000 7f27d18a7640 1 -- 172.21.15.88:0/573841274 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27cc0ce7f0 con 0x7f27cc0b53f0 2024-09-16T01:36:38.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.562+0000 7f27d18a7640 1 --2- 172.21.15.88:0/573841274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27cc0b53f0 0x7f27cc0d2110 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f27c0009870 tx=0x7f27c002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2775a86a1d31986a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:38.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.562+0000 7f27d10a6640 1 -- 172.21.15.88:0/573841274 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c0037070 con 0x7f27cc0b53f0 2024-09-16T01:36:38.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.563+0000 7f27d10a6640 1 -- 172.21.15.88:0/573841274 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27c0032440 con 0x7f27cc0b53f0 2024-09-16T01:36:38.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.563+0000 7f27d10a6640 1 -- 172.21.15.88:0/573841274 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c0032810 con 0x7f27cc0b53f0 2024-09-16T01:36:38.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.563+0000 7f27d430c640 1 -- 172.21.15.88:0/573841274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27cc0b53f0 msgr2=0x7f27cc0d2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:38.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.563+0000 7f27d430c640 1 --2- 172.21.15.88:0/573841274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27cc0b53f0 0x7f27cc0d2110 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f27c0009870 tx=0x7f27c002a180 comp rx=0 tx=0).stop 2024-09-16T01:36:38.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.563+0000 7f27d430c640 1 -- 172.21.15.88:0/573841274 shutdown_connections 2024-09-16T01:36:38.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.563+0000 7f27d430c640 1 --2- 172.21.15.88:0/573841274 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27cc0d2650 0x7f27cc0ceff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:38.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.563+0000 7f27d430c640 1 --2- 172.21.15.88:0/573841274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27cc0b53f0 0x7f27cc0d2110 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:38.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.563+0000 7f27d430c640 1 -- 172.21.15.88:0/573841274 >> 172.21.15.88:0/573841274 conn(0x7f27cc062a20 msgr2=0x7f27cc062e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:38.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.563+0000 7f27d430c640 1 -- 172.21.15.88:0/573841274 shutdown_connections 2024-09-16T01:36:38.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.563+0000 7f27d430c640 1 -- 172.21.15.88:0/573841274 wait complete. 2024-09-16T01:36:38.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.564+0000 7f27d430c640 1 Processor -- start 2024-09-16T01:36:38.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.564+0000 7f27d430c640 1 -- start start 2024-09-16T01:36:38.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.565+0000 7f27d430c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27cc0b53f0 0x7f27cc0c8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:38.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.565+0000 7f27d430c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27cc0d2650 0x7f27cc154210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:38.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.565+0000 7f27d430c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27cc15c710 0x7f27cc15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:38.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.565+0000 7f27d430c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f27cc074080 con 0x7f27cc0b53f0 2024-09-16T01:36:38.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.565+0000 7f27d430c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f27cc0cdc00 con 0x7f27cc15c710 2024-09-16T01:36:38.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.565+0000 7f27d430c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f27cc074200 con 0x7f27cc0d2650 2024-09-16T01:36:38.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.565+0000 7f27d20a8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27cc0b53f0 0x7f27cc0c8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:38.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.565+0000 7f27d20a8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27cc0b53f0 0x7f27cc0c8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42900/0 (socket says 172.21.15.88:42900) 2024-09-16T01:36:38.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.565+0000 7f27d20a8640 1 -- 172.21.15.88:0/468941066 learned_addr learned my addr 172.21.15.88:0/468941066 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:38.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.565+0000 7f27d28a9640 1 --2- 172.21.15.88:0/468941066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27cc15c710 0x7f27cc15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:38.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.565+0000 7f27d18a7640 1 --2- 172.21.15.88:0/468941066 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27cc0d2650 0x7f27cc154210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:38.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.565+0000 7f27d20a8640 1 -- 172.21.15.88:0/468941066 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27cc0d2650 msgr2=0x7f27cc154210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:38.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.565+0000 7f27d20a8640 1 --2- 172.21.15.88:0/468941066 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27cc0d2650 0x7f27cc154210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:38.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.565+0000 7f27d20a8640 1 -- 172.21.15.88:0/468941066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27cc15c710 msgr2=0x7f27cc15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:38.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.565+0000 7f27d20a8640 1 --2- 172.21.15.88:0/468941066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27cc15c710 0x7f27cc15ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:38.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.565+0000 7f27d20a8640 1 -- 172.21.15.88:0/468941066 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27cc1e4dc0 con 0x7f27cc0b53f0 2024-09-16T01:36:38.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.565+0000 7f27d28a9640 1 --2- 172.21.15.88:0/468941066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27cc15c710 0x7f27cc15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:38.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.566+0000 7f27d20a8640 1 --2- 172.21.15.88:0/468941066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27cc0b53f0 0x7f27cc0c8c60 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f27bc00c7c0 tx=0x7f27bc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:38.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.566+0000 7f27caffd640 1 -- 172.21.15.88:0/468941066 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc016020 con 0x7f27cc0b53f0 2024-09-16T01:36:38.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.566+0000 7f27caffd640 1 -- 172.21.15.88:0/468941066 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27bc00a9f0 con 0x7f27cc0b53f0 2024-09-16T01:36:38.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.566+0000 7f27d430c640 1 -- 172.21.15.88:0/468941066 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27cc1e5030 con 0x7f27cc0b53f0 2024-09-16T01:36:38.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.566+0000 7f27caffd640 1 -- 172.21.15.88:0/468941066 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc00ae20 con 0x7f27cc0b53f0 2024-09-16T01:36:38.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.566+0000 7f27d430c640 1 -- 172.21.15.88:0/468941066 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27cc1e55f0 con 0x7f27cc0b53f0 2024-09-16T01:36:38.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.567+0000 7f27d430c640 1 -- 172.21.15.88:0/468941066 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27cc0d0300 con 0x7f27cc0b53f0 2024-09-16T01:36:38.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.568+0000 7f27caffd640 1 -- 172.21.15.88:0/468941066 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f27bc019020 con 0x7f27cc0b53f0 2024-09-16T01:36:38.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.568+0000 7f27caffd640 1 --2- 172.21.15.88:0/468941066 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f27b806c2b0 0x7f27b806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:38.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.568+0000 7f27caffd640 1 -- 172.21.15.88:0/468941066 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5227+0+0 (secure 0 0 0) 0x7f27bc08b290 con 0x7f27cc0b53f0 2024-09-16T01:36:38.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.568+0000 7f27d18a7640 1 --2- 172.21.15.88:0/468941066 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f27b806c2b0 0x7f27b806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:38.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.568+0000 7f27d18a7640 1 --2- 172.21.15.88:0/468941066 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f27b806c2b0 0x7f27b806e770 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f27c0034000 tx=0x7f27c002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:38.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.572+0000 7f27caffd640 1 -- 172.21.15.88:0/468941066 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f27bc020030 con 0x7f27cc0b53f0 2024-09-16T01:36:38.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:38.737+0000 7f27d430c640 1 -- 172.21.15.88:0/468941066 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f27cc15bb30 con 0x7f27cc0b53f0 2024-09-16T01:36:39.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.444+0000 7f27caffd640 1 -- 172.21.15.88:0/468941066 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v56) ==== 170+0+0 (secure 0 0 0) 0x7f27bc059be0 con 0x7f27cc0b53f0 2024-09-16T01:36:39.445 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_data' created 2024-09-16T01:36:39.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.446+0000 7f27d430c640 1 -- 172.21.15.88:0/468941066 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f27b806c2b0 msgr2=0x7f27b806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:39.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.446+0000 7f27d430c640 1 --2- 172.21.15.88:0/468941066 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f27b806c2b0 0x7f27b806e770 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f27c0034000 tx=0x7f27c002e000 comp rx=0 tx=0).stop 2024-09-16T01:36:39.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.446+0000 7f27d430c640 1 -- 172.21.15.88:0/468941066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27cc0b53f0 msgr2=0x7f27cc0c8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:39.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.446+0000 7f27d430c640 1 --2- 172.21.15.88:0/468941066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27cc0b53f0 0x7f27cc0c8c60 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f27bc00c7c0 tx=0x7f27bc00cc80 comp rx=0 tx=0).stop 2024-09-16T01:36:39.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.447+0000 7f27d430c640 1 -- 172.21.15.88:0/468941066 shutdown_connections 2024-09-16T01:36:39.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.447+0000 7f27d430c640 1 --2- 172.21.15.88:0/468941066 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f27b806c2b0 0x7f27b806e770 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:39.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.447+0000 7f27d430c640 1 --2- 172.21.15.88:0/468941066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27cc15c710 0x7f27cc15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:39.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.447+0000 7f27d430c640 1 --2- 172.21.15.88:0/468941066 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f27cc0d2650 0x7f27cc154210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:39.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.447+0000 7f27d430c640 1 --2- 172.21.15.88:0/468941066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27cc0b53f0 0x7f27cc0c8c60 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:39.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.447+0000 7f27d430c640 1 -- 172.21.15.88:0/468941066 >> 172.21.15.88:0/468941066 conn(0x7f27cc062a20 msgr2=0x7f27cc062e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:39.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.447+0000 7f27d430c640 1 -- 172.21.15.88:0/468941066 shutdown_connections 2024-09-16T01:36:39.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.448+0000 7f27d430c640 1 -- 172.21.15.88:0/468941066 wait complete. 2024-09-16T01:36:39.472 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-16T01:36:39.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.573+0000 7f5deb8a6640 1 Processor -- start 2024-09-16T01:36:39.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.574+0000 7f5deb8a6640 1 -- start start 2024-09-16T01:36:39.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.574+0000 7f5deb8a6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5de40b53f0 0x7f5de40d2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:39.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.574+0000 7f5deb8a6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5de40d2650 0x7f5de40ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:39.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.574+0000 7f5deb8a6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5de40cdf50 con 0x7f5de415c710 2024-09-16T01:36:39.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.574+0000 7f5deb8a6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f5de40ce230 con 0x7f5de40d2650 2024-09-16T01:36:39.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.575+0000 7f5deb8a6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5de40ce510 con 0x7f5de40b53f0 2024-09-16T01:36:39.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.575+0000 7f5de9e43640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5de40d2650 0x7f5de40ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:39.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.575+0000 7f5de9e43640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5de40d2650 0x7f5de40ceff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42906/0 (socket says 172.21.15.88:42906) 2024-09-16T01:36:39.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.575+0000 7f5de9e43640 1 -- 172.21.15.88:0/3784732018 learned_addr learned my addr 172.21.15.88:0/3784732018 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:39.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.575+0000 7f5de8e41640 1 --2- 172.21.15.88:0/3784732018 >> v2:172.21.15.154:3300/0 conn(0x7f5de40b53f0 0x7f5de40d2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:39.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.575+0000 7f5de9e43640 1 -- 172.21.15.88:0/3784732018 >> v2:172.21.15.154:3300/0 conn(0x7f5de40b53f0 msgr2=0x7f5de40d2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:39.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.575+0000 7f5de9e43640 1 --2- 172.21.15.88:0/3784732018 >> v2:172.21.15.154:3300/0 conn(0x7f5de40b53f0 0x7f5de40d2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:39.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.575+0000 7f5de9e43640 1 -- 172.21.15.88:0/3784732018 >> v1:172.21.15.154:6789/0 conn(0x7f5de415c710 legacy=0x7f5de4073970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:39.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.575+0000 7f5de9e43640 1 -- 172.21.15.88:0/3784732018 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de40ce7f0 con 0x7f5de40d2650 2024-09-16T01:36:39.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.576+0000 7f5de9e43640 1 --2- 172.21.15.88:0/3784732018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5de40d2650 0x7f5de40ceff0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f5de40d0000 tx=0x7f5de002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d148cbccbf5b4b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:39.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.576+0000 7f5ddbfff640 1 -- 172.21.15.88:0/3784732018 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de0036070 con 0x7f5de40d2650 2024-09-16T01:36:39.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.576+0000 7f5ddbfff640 1 -- 172.21.15.88:0/3784732018 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5de0031600 con 0x7f5de40d2650 2024-09-16T01:36:39.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.576+0000 7f5ddbfff640 1 -- 172.21.15.88:0/3784732018 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de00319d0 con 0x7f5de40d2650 2024-09-16T01:36:39.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.576+0000 7f5deb8a6640 1 -- 172.21.15.88:0/3784732018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5de40d2650 msgr2=0x7f5de40ceff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:39.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.576+0000 7f5deb8a6640 1 --2- 172.21.15.88:0/3784732018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5de40d2650 0x7f5de40ceff0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f5de40d0000 tx=0x7f5de002a660 comp rx=0 tx=0).stop 2024-09-16T01:36:39.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.576+0000 7f5deb8a6640 1 -- 172.21.15.88:0/3784732018 shutdown_connections 2024-09-16T01:36:39.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.576+0000 7f5deb8a6640 1 --2- 172.21.15.88:0/3784732018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5de40d2650 0x7f5de40ceff0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:39.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.576+0000 7f5deb8a6640 1 --2- 172.21.15.88:0/3784732018 >> v2:172.21.15.154:3300/0 conn(0x7f5de40b53f0 0x7f5de40d2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:39.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.576+0000 7f5deb8a6640 1 -- 172.21.15.88:0/3784732018 >> 172.21.15.88:0/3784732018 conn(0x7f5de4062a20 msgr2=0x7f5de4062e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:39.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.576+0000 7f5deb8a6640 1 -- 172.21.15.88:0/3784732018 shutdown_connections 2024-09-16T01:36:39.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.576+0000 7f5deb8a6640 1 -- 172.21.15.88:0/3784732018 wait complete. 2024-09-16T01:36:39.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.577+0000 7f5deb8a6640 1 Processor -- start 2024-09-16T01:36:39.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.578+0000 7f5deb8a6640 1 -- start start 2024-09-16T01:36:39.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.578+0000 7f5deb8a6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5de40b53f0 0x7f5de4154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:39.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.578+0000 7f5deb8a6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5de40d2650 0x7f5de4153880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:39.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.578+0000 7f5deb8a6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5de415c710 0x7f5de415d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:39.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.578+0000 7f5deb8a6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5de4074080 con 0x7f5de415c710 2024-09-16T01:36:39.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.578+0000 7f5deb8a6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5de40cdc00 con 0x7f5de40b53f0 2024-09-16T01:36:39.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.578+0000 7f5deb8a6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5de4074200 con 0x7f5de40d2650 2024-09-16T01:36:39.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.578+0000 7f5de9e43640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5de415c710 0x7f5de415d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:39.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.578+0000 7f5de9e43640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5de415c710 0x7f5de415d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42912/0 (socket says 172.21.15.88:42912) 2024-09-16T01:36:39.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.578+0000 7f5de9e43640 1 -- 172.21.15.88:0/1322704615 learned_addr learned my addr 172.21.15.88:0/1322704615 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:39.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.579+0000 7f5de9642640 1 --2- 172.21.15.88:0/1322704615 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5de40b53f0 0x7f5de4154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:39.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.579+0000 7f5de8e41640 1 --2- 172.21.15.88:0/1322704615 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5de40d2650 0x7f5de4153880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:39.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.579+0000 7f5de9e43640 1 -- 172.21.15.88:0/1322704615 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5de40d2650 msgr2=0x7f5de4153880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:39.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.579+0000 7f5de9e43640 1 --2- 172.21.15.88:0/1322704615 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5de40d2650 0x7f5de4153880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:39.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.579+0000 7f5de9e43640 1 -- 172.21.15.88:0/1322704615 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5de40b53f0 msgr2=0x7f5de4154fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:39.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.579+0000 7f5de9e43640 1 --2- 172.21.15.88:0/1322704615 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5de40b53f0 0x7f5de4154fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:39.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.579+0000 7f5de9e43640 1 -- 172.21.15.88:0/1322704615 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de4155580 con 0x7f5de415c710 2024-09-16T01:36:39.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.579+0000 7f5de8e41640 1 --2- 172.21.15.88:0/1322704615 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5de40d2650 0x7f5de4153880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:39.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.579+0000 7f5de9e43640 1 --2- 172.21.15.88:0/1322704615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5de415c710 0x7f5de415d6e0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f5de0034040 tx=0x7f5de0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:39.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.579+0000 7f5dd9ffb640 1 -- 172.21.15.88:0/1322704615 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de0036050 con 0x7f5de415c710 2024-09-16T01:36:39.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.579+0000 7f5dd9ffb640 1 -- 172.21.15.88:0/1322704615 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5de0042070 con 0x7f5de415c710 2024-09-16T01:36:39.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.579+0000 7f5dd9ffb640 1 -- 172.21.15.88:0/1322704615 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de0009510 con 0x7f5de415c710 2024-09-16T01:36:39.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.579+0000 7f5deb8a6640 1 -- 172.21.15.88:0/1322704615 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5de4155860 con 0x7f5de415c710 2024-09-16T01:36:39.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.579+0000 7f5deb8a6640 1 -- 172.21.15.88:0/1322704615 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5de415dc20 con 0x7f5de415c710 2024-09-16T01:36:39.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.580+0000 7f5deb8a6640 1 -- 172.21.15.88:0/1322704615 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5de40d2a60 con 0x7f5de415c710 2024-09-16T01:36:39.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.581+0000 7f5dd9ffb640 1 -- 172.21.15.88:0/1322704615 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5de0007bd0 con 0x7f5de415c710 2024-09-16T01:36:39.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.581+0000 7f5dd9ffb640 1 --2- 172.21.15.88:0/1322704615 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5dc806c2b0 0x7f5dc806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:39.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.581+0000 7f5dd9ffb640 1 -- 172.21.15.88:0/1322704615 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5609+0+0 (secure 0 0 0) 0x7f5de00ac270 con 0x7f5de415c710 2024-09-16T01:36:39.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.581+0000 7f5de9642640 1 --2- 172.21.15.88:0/1322704615 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5dc806c2b0 0x7f5dc806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:39.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.582+0000 7f5de9642640 1 --2- 172.21.15.88:0/1322704615 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5dc806c2b0 0x7f5dc806e770 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc006fd0 tx=0x7f5ddc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:39.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.586+0000 7f5dd9ffb640 1 -- 172.21.15.88:0/1322704615 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5de007aa40 con 0x7f5de415c710 2024-09-16T01:36:39.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:39.784+0000 7f5deb8a6640 1 -- 172.21.15.88:0/1322704615 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f5de4155a30 con 0x7f5de415c710 2024-09-16T01:36:40.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.445+0000 7f5dd9ffb640 1 -- 172.21.15.88:0/1322704615 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v35) ==== 172+0+0 (secure 0 0 0) 0x7f5de007a340 con 0x7f5de415c710 2024-09-16T01:36:40.447 INFO:teuthology.orchestra.run.smithi088.stderr:new fs with metadata pool 8 and data pool 9 2024-09-16T01:36:40.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.449+0000 7f5deb8a6640 1 -- 172.21.15.88:0/1322704615 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5dc806c2b0 msgr2=0x7f5dc806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:40.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.449+0000 7f5deb8a6640 1 --2- 172.21.15.88:0/1322704615 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5dc806c2b0 0x7f5dc806e770 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc006fd0 tx=0x7f5ddc008040 comp rx=0 tx=0).stop 2024-09-16T01:36:40.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.449+0000 7f5deb8a6640 1 -- 172.21.15.88:0/1322704615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5de415c710 msgr2=0x7f5de415d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:40.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.449+0000 7f5deb8a6640 1 --2- 172.21.15.88:0/1322704615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5de415c710 0x7f5de415d6e0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f5de0034040 tx=0x7f5de0004a40 comp rx=0 tx=0).stop 2024-09-16T01:36:40.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.450+0000 7f5deb8a6640 1 -- 172.21.15.88:0/1322704615 shutdown_connections 2024-09-16T01:36:40.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.450+0000 7f5deb8a6640 1 --2- 172.21.15.88:0/1322704615 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5dc806c2b0 0x7f5dc806e770 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:40.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.450+0000 7f5deb8a6640 1 --2- 172.21.15.88:0/1322704615 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5de415c710 0x7f5de415d6e0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:40.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.450+0000 7f5deb8a6640 1 --2- 172.21.15.88:0/1322704615 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5de40d2650 0x7f5de4153880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:40.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.450+0000 7f5deb8a6640 1 --2- 172.21.15.88:0/1322704615 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5de40b53f0 0x7f5de4154fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:40.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.450+0000 7f5deb8a6640 1 -- 172.21.15.88:0/1322704615 >> 172.21.15.88:0/1322704615 conn(0x7f5de4062a20 msgr2=0x7f5de415ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:40.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.452+0000 7f5deb8a6640 1 -- 172.21.15.88:0/1322704615 shutdown_connections 2024-09-16T01:36:40.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.454+0000 7f5deb8a6640 1 -- 172.21.15.88:0/1322704615 wait complete. 2024-09-16T01:36:40.478 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:36:40.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.578+0000 7f0472b4b640 1 Processor -- start 2024-09-16T01:36:40.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.578+0000 7f0472b4b640 1 -- start start 2024-09-16T01:36:40.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.579+0000 7f0472b4b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f046c06e800 0x7f046c0ae890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:40.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.579+0000 7f0472b4b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f046c072d50 0x7f046c0b57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:40.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.579+0000 7f0472b4b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f046c0b52c0 con 0x7f046c15e540 2024-09-16T01:36:40.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.579+0000 7f0472b4b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f046c0aedd0 con 0x7f046c072d50 2024-09-16T01:36:40.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.579+0000 7f0472b4b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f046c0af000 con 0x7f046c06e800 2024-09-16T01:36:40.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.579+0000 7f046bfff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f046c06e800 0x7f046c0ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:40.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.579+0000 7f04710e8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f046c072d50 0x7f046c0b57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:40.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.579+0000 7f046bfff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f046c06e800 0x7f046c0ae890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:34902/0 (socket says 172.21.15.88:34902) 2024-09-16T01:36:40.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.579+0000 7f046bfff640 1 -- 172.21.15.88:0/3166217652 learned_addr learned my addr 172.21.15.88:0/3166217652 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:40.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.579+0000 7f04710e8640 1 -- 172.21.15.88:0/3166217652 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f046c06e800 msgr2=0x7f046c0ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:40.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.579+0000 7f04710e8640 1 --2- 172.21.15.88:0/3166217652 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f046c06e800 0x7f046c0ae890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:40.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.579+0000 7f04710e8640 1 -- 172.21.15.88:0/3166217652 >> v1:172.21.15.154:6789/0 conn(0x7f046c15e540 legacy=0x7f046c06e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:40.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.579+0000 7f04710e8640 1 -- 172.21.15.88:0/3166217652 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f046c0af2e0 con 0x7f046c072d50 2024-09-16T01:36:40.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.580+0000 7f04710e8640 1 --2- 172.21.15.88:0/3166217652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f046c072d50 0x7f046c0b57a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0464009d90 tx=0x7f046402a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf2af0403b5d456d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:40.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.580+0000 7f046b7fe640 1 -- 172.21.15.88:0/3166217652 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0464031600 con 0x7f046c072d50 2024-09-16T01:36:40.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.580+0000 7f046b7fe640 1 -- 172.21.15.88:0/3166217652 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04640317a0 con 0x7f046c072d50 2024-09-16T01:36:40.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.580+0000 7f046b7fe640 1 -- 172.21.15.88:0/3166217652 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0464031b70 con 0x7f046c072d50 2024-09-16T01:36:40.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.581+0000 7f0472b4b640 1 -- 172.21.15.88:0/3166217652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f046c072d50 msgr2=0x7f046c0b57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:40.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.581+0000 7f0472b4b640 1 --2- 172.21.15.88:0/3166217652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f046c072d50 0x7f046c0b57a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0464009d90 tx=0x7f046402a5d0 comp rx=0 tx=0).stop 2024-09-16T01:36:40.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.581+0000 7f0472b4b640 1 -- 172.21.15.88:0/3166217652 shutdown_connections 2024-09-16T01:36:40.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.581+0000 7f0472b4b640 1 --2- 172.21.15.88:0/3166217652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f046c072d50 0x7f046c0b57a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:40.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.581+0000 7f0472b4b640 1 --2- 172.21.15.88:0/3166217652 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f046c06e800 0x7f046c0ae890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:40.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.581+0000 7f0472b4b640 1 -- 172.21.15.88:0/3166217652 >> 172.21.15.88:0/3166217652 conn(0x7f046c06bc10 msgr2=0x7f046c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:40.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.581+0000 7f0472b4b640 1 -- 172.21.15.88:0/3166217652 shutdown_connections 2024-09-16T01:36:40.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.581+0000 7f0472b4b640 1 -- 172.21.15.88:0/3166217652 wait complete. 2024-09-16T01:36:40.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.582+0000 7f0472b4b640 1 Processor -- start 2024-09-16T01:36:40.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.582+0000 7f0472b4b640 1 -- start start 2024-09-16T01:36:40.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.583+0000 7f0472b4b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f046c06e800 0x7f046c1425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:40.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.583+0000 7f0472b4b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f046c072d50 0x7f046c1f8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:40.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.583+0000 7f0472b4b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f046c15e540 0x7f046c1e37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:40.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.583+0000 7f0472b4b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f046c15e920 con 0x7f046c15e540 2024-09-16T01:36:40.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.583+0000 7f0472b4b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f046c0af8a0 con 0x7f046c072d50 2024-09-16T01:36:40.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.583+0000 7f0472b4b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f046c0b5ce0 con 0x7f046c06e800 2024-09-16T01:36:40.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.583+0000 7f04710e8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f046c15e540 0x7f046c1e37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:40.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.583+0000 7f04710e8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f046c15e540 0x7f046c1e37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41240/0 (socket says 172.21.15.88:41240) 2024-09-16T01:36:40.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.583+0000 7f04710e8640 1 -- 172.21.15.88:0/1219450623 learned_addr learned my addr 172.21.15.88:0/1219450623 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:40.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.583+0000 7f046bfff640 1 --2- 172.21.15.88:0/1219450623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f046c072d50 0x7f046c1f8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:40.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.583+0000 7f04708e7640 1 --2- 172.21.15.88:0/1219450623 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f046c06e800 0x7f046c1425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:40.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.583+0000 7f04710e8640 1 -- 172.21.15.88:0/1219450623 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f046c06e800 msgr2=0x7f046c1425e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:40.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.583+0000 7f04710e8640 1 --2- 172.21.15.88:0/1219450623 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f046c06e800 0x7f046c1425e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:40.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.583+0000 7f04710e8640 1 -- 172.21.15.88:0/1219450623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f046c072d50 msgr2=0x7f046c1f8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:40.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.583+0000 7f04710e8640 1 --2- 172.21.15.88:0/1219450623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f046c072d50 0x7f046c1f8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:40.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.583+0000 7f04710e8640 1 -- 172.21.15.88:0/1219450623 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f046c1e3ee0 con 0x7f046c15e540 2024-09-16T01:36:40.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.583+0000 7f046bfff640 1 --2- 172.21.15.88:0/1219450623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f046c072d50 0x7f046c1f8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:36:40.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.584+0000 7f04710e8640 1 --2- 172.21.15.88:0/1219450623 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f046c15e540 0x7f046c1e37b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f0464009eb0 tx=0x7f0464004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:40.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.584+0000 7f04697fa640 1 -- 172.21.15.88:0/1219450623 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0464040070 con 0x7f046c15e540 2024-09-16T01:36:40.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.584+0000 7f04697fa640 1 -- 172.21.15.88:0/1219450623 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0464030b60 con 0x7f046c15e540 2024-09-16T01:36:40.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.584+0000 7f0472b4b640 1 -- 172.21.15.88:0/1219450623 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f046c1e41c0 con 0x7f046c15e540 2024-09-16T01:36:40.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.584+0000 7f04697fa640 1 -- 172.21.15.88:0/1219450623 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0464009450 con 0x7f046c15e540 2024-09-16T01:36:40.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.584+0000 7f0472b4b640 1 -- 172.21.15.88:0/1219450623 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f046c1e4780 con 0x7f046c15e540 2024-09-16T01:36:40.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.585+0000 7f0472b4b640 1 -- 172.21.15.88:0/1219450623 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f046c076170 con 0x7f046c15e540 2024-09-16T01:36:40.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.585+0000 7f04697fa640 1 -- 172.21.15.88:0/1219450623 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0464039040 con 0x7f046c15e540 2024-09-16T01:36:40.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.586+0000 7f04697fa640 1 --2- 172.21.15.88:0/1219450623 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f045006c2b0 0x7f045006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:40.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.586+0000 7f04708e7640 1 --2- 172.21.15.88:0/1219450623 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f045006c2b0 0x7f045006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:40.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.586+0000 7f04697fa640 1 -- 172.21.15.88:0/1219450623 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f04640ac9a0 con 0x7f046c15e540 2024-09-16T01:36:40.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.586+0000 7f04708e7640 1 --2- 172.21.15.88:0/1219450623 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f045006c2b0 0x7f045006e770 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f04600016e0 tx=0x7f0460009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:40.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.590+0000 7f04697fa640 1 -- 172.21.15.88:0/1219450623 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f046c076170 con 0x7f046c15e540 2024-09-16T01:36:40.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.757+0000 7f0472b4b640 1 -- 172.21.15.88:0/1219450623 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f046c0ae890 con 0x7f046c15e540 2024-09-16T01:36:40.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.758+0000 7f04697fa640 1 -- 172.21.15.88:0/1219450623 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+16714 (secure 0 0 0) 0x7f046407aac0 con 0x7f046c15e540 2024-09-16T01:36:40.760 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:40.760 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":57,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:36:39.786092+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:36:38.121335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-16T01:36:38.755908+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:36:40.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.760+0000 7f0472b4b640 1 -- 172.21.15.88:0/1219450623 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f045006c2b0 msgr2=0x7f045006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:40.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.760+0000 7f0472b4b640 1 --2- 172.21.15.88:0/1219450623 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f045006c2b0 0x7f045006e770 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f04600016e0 tx=0x7f0460009040 comp rx=0 tx=0).stop 2024-09-16T01:36:40.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.760+0000 7f0472b4b640 1 -- 172.21.15.88:0/1219450623 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f046c15e540 msgr2=0x7f046c1e37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:40.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.760+0000 7f0472b4b640 1 --2- 172.21.15.88:0/1219450623 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f046c15e540 0x7f046c1e37b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f0464009eb0 tx=0x7f0464004a40 comp rx=0 tx=0).stop 2024-09-16T01:36:40.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.760+0000 7f0472b4b640 1 -- 172.21.15.88:0/1219450623 shutdown_connections 2024-09-16T01:36:40.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.760+0000 7f0472b4b640 1 --2- 172.21.15.88:0/1219450623 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f045006c2b0 0x7f045006e770 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:40.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.760+0000 7f0472b4b640 1 --2- 172.21.15.88:0/1219450623 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f046c15e540 0x7f046c1e37b0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:40.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.760+0000 7f0472b4b640 1 --2- 172.21.15.88:0/1219450623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f046c072d50 0x7f046c1f8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:40.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.760+0000 7f0472b4b640 1 --2- 172.21.15.88:0/1219450623 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f046c06e800 0x7f046c1425e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:40.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.760+0000 7f0472b4b640 1 -- 172.21.15.88:0/1219450623 >> 172.21.15.88:0/1219450623 conn(0x7f046c06bc10 msgr2=0x7f046c0cde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:40.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.761+0000 7f0472b4b640 1 -- 172.21.15.88:0/1219450623 shutdown_connections 2024-09-16T01:36:40.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.761+0000 7f0472b4b640 1 -- 172.21.15.88:0/1219450623 wait complete. 2024-09-16T01:36:40.773 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:36:40.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.872+0000 7f744d05c640 1 Processor -- start 2024-09-16T01:36:40.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.872+0000 7f744d05c640 1 -- start start 2024-09-16T01:36:40.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.873+0000 7f744d05c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f744815c2b0 0x7f74480b6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:40.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.873+0000 7f744d05c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f744815fb10 con 0x7f744815f200 2024-09-16T01:36:40.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.873+0000 7f744d05c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74480b5900 con 0x7f74480b72f0 2024-09-16T01:36:40.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.873+0000 7f744d05c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f744815f830 con 0x7f744815c2b0 2024-09-16T01:36:40.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.873+0000 7f7446d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f744815c2b0 0x7f74480b6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:40.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.873+0000 7f744759e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f744815f200 0x7f744806d910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54292/0 (socket says 172.21.15.88:54292) 2024-09-16T01:36:40.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.873+0000 7f7446d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f744815c2b0 0x7f74480b6db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41242/0 (socket says 172.21.15.88:41242) 2024-09-16T01:36:40.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.873+0000 7f744759e640 1 -- 172.21.15.88:0/2793442408 learned_addr learned my addr 172.21.15.88:0/2793442408 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:40.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.873+0000 7f7446d9d640 1 -- 172.21.15.88:0/2793442408 >> v1:172.21.15.154:6789/0 conn(0x7f74480b72f0 legacy=0x7f744815eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:40.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.873+0000 7f7446d9d640 1 -- 172.21.15.88:0/2793442408 >> v1:172.21.15.88:6789/0 conn(0x7f744815f200 legacy=0x7f744806d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:40.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.873+0000 7f7446d9d640 1 -- 172.21.15.88:0/2793442408 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f744806ce30 con 0x7f744815c2b0 2024-09-16T01:36:40.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.874+0000 7f7446d9d640 1 --2- 172.21.15.88:0/2793442408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f744815c2b0 0x7f74480b6db0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f74380097a0 tx=0x7f743802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=592953aaebc4127c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:40.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.874+0000 7f744659c640 1 -- 172.21.15.88:0/2793442408 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7438037070 con 0x7f744815c2b0 2024-09-16T01:36:40.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.874+0000 7f744659c640 1 -- 172.21.15.88:0/2793442408 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7438032440 con 0x7f744815c2b0 2024-09-16T01:36:40.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.874+0000 7f744659c640 1 -- 172.21.15.88:0/2793442408 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7438032810 con 0x7f744815c2b0 2024-09-16T01:36:40.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.875+0000 7f744d05c640 1 -- 172.21.15.88:0/2793442408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f744815c2b0 msgr2=0x7f74480b6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:40.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.875+0000 7f744d05c640 1 --2- 172.21.15.88:0/2793442408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f744815c2b0 0x7f74480b6db0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f74380097a0 tx=0x7f743802a1e0 comp rx=0 tx=0).stop 2024-09-16T01:36:40.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.875+0000 7f744d05c640 1 -- 172.21.15.88:0/2793442408 shutdown_connections 2024-09-16T01:36:40.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.875+0000 7f744d05c640 1 --2- 172.21.15.88:0/2793442408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f744815c2b0 0x7f74480b6db0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:40.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.875+0000 7f744d05c640 1 -- 172.21.15.88:0/2793442408 >> 172.21.15.88:0/2793442408 conn(0x7f74480b4240 msgr2=0x7f74480b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:40.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.875+0000 7f744d05c640 1 -- 172.21.15.88:0/2793442408 shutdown_connections 2024-09-16T01:36:40.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.875+0000 7f744d05c640 1 -- 172.21.15.88:0/2793442408 wait complete. 2024-09-16T01:36:40.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.876+0000 7f744d05c640 1 Processor -- start 2024-09-16T01:36:40.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.876+0000 7f744d05c640 1 -- start start 2024-09-16T01:36:40.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.877+0000 7f744d05c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74480b72f0 0x7f7448148380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:40.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.877+0000 7f744d05c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f744815c2b0 0x7f7448154330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:40.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.877+0000 7f744d05c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f744815f200 0x7f74481554e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:40.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.877+0000 7f744d05c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f74480b7760 con 0x7f744815f200 2024-09-16T01:36:40.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.877+0000 7f744d05c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f744806d3f0 con 0x7f74480b72f0 2024-09-16T01:36:40.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.877+0000 7f744d05c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f744806e020 con 0x7f744815c2b0 2024-09-16T01:36:40.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.877+0000 7f744759e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f744815f200 0x7f74481554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:40.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.877+0000 7f744759e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f744815f200 0x7f74481554e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41254/0 (socket says 172.21.15.88:41254) 2024-09-16T01:36:40.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.877+0000 7f744759e640 1 -- 172.21.15.88:0/3974534862 learned_addr learned my addr 172.21.15.88:0/3974534862 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:40.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.877+0000 7f7446d9d640 1 --2- 172.21.15.88:0/3974534862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74480b72f0 0x7f7448148380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:40.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.877+0000 7f743ffff640 1 --2- 172.21.15.88:0/3974534862 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f744815c2b0 0x7f7448154330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:40.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.877+0000 7f744759e640 1 -- 172.21.15.88:0/3974534862 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f744815c2b0 msgr2=0x7f7448154330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:40.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.877+0000 7f744759e640 1 --2- 172.21.15.88:0/3974534862 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f744815c2b0 0x7f7448154330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:40.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.877+0000 7f744759e640 1 -- 172.21.15.88:0/3974534862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74480b72f0 msgr2=0x7f7448148380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:40.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.877+0000 7f744759e640 1 --2- 172.21.15.88:0/3974534862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74480b72f0 0x7f7448148380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:40.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.877+0000 7f744759e640 1 -- 172.21.15.88:0/3974534862 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7448154e20 con 0x7f744815f200 2024-09-16T01:36:40.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.877+0000 7f744759e640 1 --2- 172.21.15.88:0/3974534862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f744815f200 0x7f74481554e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f744000cab0 tx=0x7f744000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:40.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.877+0000 7f743f7fe640 1 -- 172.21.15.88:0/3974534862 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7440017070 con 0x7f744815f200 2024-09-16T01:36:40.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.877+0000 7f743f7fe640 1 -- 172.21.15.88:0/3974534862 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f744000ac30 con 0x7f744815f200 2024-09-16T01:36:40.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.877+0000 7f744d05c640 1 -- 172.21.15.88:0/3974534862 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7448154870 con 0x7f744815f200 2024-09-16T01:36:40.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.877+0000 7f743f7fe640 1 -- 172.21.15.88:0/3974534862 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7440012400 con 0x7f744815f200 2024-09-16T01:36:40.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.878+0000 7f744d05c640 1 -- 172.21.15.88:0/3974534862 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74480cb960 con 0x7f744815f200 2024-09-16T01:36:40.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.878+0000 7f744d05c640 1 -- 172.21.15.88:0/3974534862 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74480b6db0 con 0x7f744815f200 2024-09-16T01:36:40.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.879+0000 7f743f7fe640 1 -- 172.21.15.88:0/3974534862 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f744001a070 con 0x7f744815f200 2024-09-16T01:36:40.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.879+0000 7f743f7fe640 1 --2- 172.21.15.88:0/3974534862 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f742806c2e0 0x7f742806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:40.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.879+0000 7f743f7fe640 1 -- 172.21.15.88:0/3974534862 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f744008af50 con 0x7f744815f200 2024-09-16T01:36:40.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.879+0000 7f7446d9d640 1 --2- 172.21.15.88:0/3974534862 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f742806c2e0 0x7f742806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:40.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.880+0000 7f7446d9d640 1 --2- 172.21.15.88:0/3974534862 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f742806c2e0 0x7f742806e7a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7438034000 tx=0x7f743802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:40.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:40.884+0000 7f743f7fe640 1 -- 172.21.15.88:0/3974534862 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7440059100 con 0x7f744815f200 2024-09-16T01:36:41.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.052+0000 7f744d05c640 1 -- 172.21.15.88:0/3974534862 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f744806d9b0 con 0x7f744815f200 2024-09-16T01:36:41.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.053+0000 7f743f7fe640 1 -- 172.21.15.88:0/3974534862 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+16714 (secure 0 0 0) 0x7f744005d400 con 0x7f744815f200 2024-09-16T01:36:41.055 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:41.055 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":57,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:36:39.786092+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:36:38.121335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-16T01:36:38.755908+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:36:41.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.055+0000 7f744d05c640 1 -- 172.21.15.88:0/3974534862 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f742806c2e0 msgr2=0x7f742806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:41.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.055+0000 7f744d05c640 1 --2- 172.21.15.88:0/3974534862 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f742806c2e0 0x7f742806e7a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7438034000 tx=0x7f743802e000 comp rx=0 tx=0).stop 2024-09-16T01:36:41.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.055+0000 7f744d05c640 1 -- 172.21.15.88:0/3974534862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f744815f200 msgr2=0x7f74481554e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:41.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.055+0000 7f744d05c640 1 --2- 172.21.15.88:0/3974534862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f744815f200 0x7f74481554e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f744000cab0 tx=0x7f744000cf70 comp rx=0 tx=0).stop 2024-09-16T01:36:41.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.055+0000 7f744d05c640 1 -- 172.21.15.88:0/3974534862 shutdown_connections 2024-09-16T01:36:41.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.055+0000 7f744d05c640 1 --2- 172.21.15.88:0/3974534862 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f742806c2e0 0x7f742806e7a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:41.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.055+0000 7f744d05c640 1 --2- 172.21.15.88:0/3974534862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f744815f200 0x7f74481554e0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:41.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.055+0000 7f744d05c640 1 --2- 172.21.15.88:0/3974534862 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f744815c2b0 0x7f7448154330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:41.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.055+0000 7f744d05c640 1 --2- 172.21.15.88:0/3974534862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74480b72f0 0x7f7448148380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:41.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.055+0000 7f744d05c640 1 -- 172.21.15.88:0/3974534862 >> 172.21.15.88:0/3974534862 conn(0x7f74480b4240 msgr2=0x7f7448160570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:41.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.055+0000 7f744d05c640 1 -- 172.21.15.88:0/3974534862 shutdown_connections 2024-09-16T01:36:41.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.055+0000 7f744d05c640 1 -- 172.21.15.88:0/3974534862 wait complete. 2024-09-16T01:36:41.068 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-16T01:36:41.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.167+0000 7fe05de84640 1 Processor -- start 2024-09-16T01:36:41.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.167+0000 7fe05de84640 1 -- start start 2024-09-16T01:36:41.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.168+0000 7fe05de84640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe05815b480 0x7fe0580b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:41.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.168+0000 7fe05de84640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe058072700 0x7fe05815bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:41.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.168+0000 7fe05de84640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0580ccff0 con 0x7fe0580b60c0 2024-09-16T01:36:41.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.168+0000 7fe05de84640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe0580cd2d0 con 0x7fe05815b480 2024-09-16T01:36:41.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.168+0000 7fe05de84640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe0580cd5b0 con 0x7fe058072700 2024-09-16T01:36:41.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.168+0000 7fe0577fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe05815b480 0x7fe0580b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:41.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.168+0000 7fe0577fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe05815b480 0x7fe0580b5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41264/0 (socket says 172.21.15.88:41264) 2024-09-16T01:36:41.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.168+0000 7fe0577fe640 1 -- 172.21.15.88:0/4181982820 learned_addr learned my addr 172.21.15.88:0/4181982820 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:41.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.168+0000 7fe057fff640 1 --2- 172.21.15.88:0/4181982820 >> v2:172.21.15.154:3300/0 conn(0x7fe058072700 0x7fe05815bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:41.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.168+0000 7fe0577fe640 1 -- 172.21.15.88:0/4181982820 >> v2:172.21.15.154:3300/0 conn(0x7fe058072700 msgr2=0x7fe05815bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:41.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.168+0000 7fe0577fe640 1 --2- 172.21.15.88:0/4181982820 >> v2:172.21.15.154:3300/0 conn(0x7fe058072700 0x7fe05815bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:41.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.169+0000 7fe0577fe640 1 -- 172.21.15.88:0/4181982820 >> v1:172.21.15.88:6789/0 conn(0x7fe0580b60c0 legacy=0x7fe058071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:41.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.169+0000 7fe0577fe640 1 -- 172.21.15.88:0/4181982820 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0580cd890 con 0x7fe05815b480 2024-09-16T01:36:41.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.169+0000 7fe0567fc640 1 -- 172.21.15.88:0/4181982820 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2132149296 0 0) 0x7fe0580ccff0 con 0x7fe0580b60c0 2024-09-16T01:36:41.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.169+0000 7fe0577fe640 1 --2- 172.21.15.88:0/4181982820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe05815b480 0x7fe0580b5b80 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe0580cdf80 tx=0x7fe04802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7d5f83421e06d08 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:41.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.169+0000 7fe0567fc640 1 -- 172.21.15.88:0/4181982820 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe048037070 con 0x7fe05815b480 2024-09-16T01:36:41.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.169+0000 7fe0567fc640 1 -- 172.21.15.88:0/4181982820 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe048032440 con 0x7fe05815b480 2024-09-16T01:36:41.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.169+0000 7fe0567fc640 1 -- 172.21.15.88:0/4181982820 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe048032810 con 0x7fe05815b480 2024-09-16T01:36:41.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.170+0000 7fe05de84640 1 -- 172.21.15.88:0/4181982820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe05815b480 msgr2=0x7fe0580b5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:41.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.170+0000 7fe05de84640 1 --2- 172.21.15.88:0/4181982820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe05815b480 0x7fe0580b5b80 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe0580cdf80 tx=0x7fe04802a1e0 comp rx=0 tx=0).stop 2024-09-16T01:36:41.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.170+0000 7fe05de84640 1 -- 172.21.15.88:0/4181982820 shutdown_connections 2024-09-16T01:36:41.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.170+0000 7fe05de84640 1 --2- 172.21.15.88:0/4181982820 >> v2:172.21.15.154:3300/0 conn(0x7fe058072700 0x7fe05815bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:41.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.170+0000 7fe05de84640 1 --2- 172.21.15.88:0/4181982820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe05815b480 0x7fe0580b5b80 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:41.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.170+0000 7fe05de84640 1 -- 172.21.15.88:0/4181982820 >> 172.21.15.88:0/4181982820 conn(0x7fe05806d0f0 msgr2=0x7fe05806d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:41.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.170+0000 7fe05de84640 1 -- 172.21.15.88:0/4181982820 shutdown_connections 2024-09-16T01:36:41.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.170+0000 7fe05de84640 1 -- 172.21.15.88:0/4181982820 wait complete. 2024-09-16T01:36:41.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.171+0000 7fe05de84640 1 Processor -- start 2024-09-16T01:36:41.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.171+0000 7fe05de84640 1 -- start start 2024-09-16T01:36:41.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.172+0000 7fe05de84640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe058072700 0x7fe058148fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:41.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.172+0000 7fe05de84640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0580b60c0 0x7fe05814e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:41.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.172+0000 7fe05de84640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe05815b480 0x7fe05814ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:41.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.172+0000 7fe05de84640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe05815b9e0 con 0x7fe058072700 2024-09-16T01:36:41.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.172+0000 7fe05de84640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe05815b860 con 0x7fe0580b60c0 2024-09-16T01:36:41.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.172+0000 7fe05de84640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe05815c1a0 con 0x7fe05815b480 2024-09-16T01:36:41.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.172+0000 7fe0577fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe058072700 0x7fe058148fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:41.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.172+0000 7fe0577fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe058072700 0x7fe058148fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41266/0 (socket says 172.21.15.88:41266) 2024-09-16T01:36:41.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.172+0000 7fe0577fe640 1 -- 172.21.15.88:0/113665840 learned_addr learned my addr 172.21.15.88:0/113665840 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:41.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.172+0000 7fe056ffd640 1 --2- 172.21.15.88:0/113665840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0580b60c0 0x7fe05814e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:41.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.172+0000 7fe057fff640 1 --2- 172.21.15.88:0/113665840 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe05815b480 0x7fe05814ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:41.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.172+0000 7fe0577fe640 1 -- 172.21.15.88:0/113665840 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe05815b480 msgr2=0x7fe05814ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:41.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.172+0000 7fe0577fe640 1 --2- 172.21.15.88:0/113665840 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe05815b480 0x7fe05814ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:41.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.172+0000 7fe0577fe640 1 -- 172.21.15.88:0/113665840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0580b60c0 msgr2=0x7fe05814e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:41.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.172+0000 7fe0577fe640 1 --2- 172.21.15.88:0/113665840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0580b60c0 0x7fe05814e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:41.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.172+0000 7fe0577fe640 1 -- 172.21.15.88:0/113665840 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe058150580 con 0x7fe058072700 2024-09-16T01:36:41.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.172+0000 7fe056ffd640 1 --2- 172.21.15.88:0/113665840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0580b60c0 0x7fe05814e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:41.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.173+0000 7fe0577fe640 1 --2- 172.21.15.88:0/113665840 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe058072700 0x7fe058148fa0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe04802a220 tx=0x7fe0480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:41.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.173+0000 7fe03ffff640 1 -- 172.21.15.88:0/113665840 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe048041020 con 0x7fe058072700 2024-09-16T01:36:41.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.173+0000 7fe03ffff640 1 -- 172.21.15.88:0/113665840 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0480306c0 con 0x7fe058072700 2024-09-16T01:36:41.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.173+0000 7fe05de84640 1 -- 172.21.15.88:0/113665840 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe058150860 con 0x7fe058072700 2024-09-16T01:36:41.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.173+0000 7fe03ffff640 1 -- 172.21.15.88:0/113665840 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe048037040 con 0x7fe058072700 2024-09-16T01:36:41.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.173+0000 7fe05de84640 1 -- 172.21.15.88:0/113665840 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe058150e20 con 0x7fe058072700 2024-09-16T01:36:41.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.174+0000 7fe05de84640 1 -- 172.21.15.88:0/113665840 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe05815bc60 con 0x7fe058072700 2024-09-16T01:36:41.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.174+0000 7fe03ffff640 1 -- 172.21.15.88:0/113665840 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe048044020 con 0x7fe058072700 2024-09-16T01:36:41.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.175+0000 7fe03ffff640 1 --2- 172.21.15.88:0/113665840 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe04406c2b0 0x7fe04406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:41.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.175+0000 7fe03ffff640 1 -- 172.21.15.88:0/113665840 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fe0480abc80 con 0x7fe058072700 2024-09-16T01:36:41.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.175+0000 7fe056ffd640 1 --2- 172.21.15.88:0/113665840 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe04406c2b0 0x7fe04406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:41.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.175+0000 7fe056ffd640 1 --2- 172.21.15.88:0/113665840 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe04406c2b0 0x7fe04406e770 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe058139850 tx=0x7fe04c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:41.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.179+0000 7fe03ffff640 1 -- 172.21.15.88:0/113665840 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe04804b030 con 0x7fe058072700 2024-09-16T01:36:41.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.379+0000 7fe05de84640 1 -- 172.21.15.88:0/113665840 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fe058071de0 con 0x7fe058072700 2024-09-16T01:36:41.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.460+0000 7fe03ffff640 1 -- 172.21.15.88:0/113665840 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v36) ==== 118+0+0 (secure 0 0 0) 0x7fe04807a3f0 con 0x7fe058072700 2024-09-16T01:36:41.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.463+0000 7fe05de84640 1 -- 172.21.15.88:0/113665840 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe04406c2b0 msgr2=0x7fe04406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:41.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.463+0000 7fe05de84640 1 --2- 172.21.15.88:0/113665840 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe04406c2b0 0x7fe04406e770 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe058139850 tx=0x7fe04c009040 comp rx=0 tx=0).stop 2024-09-16T01:36:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.463+0000 7fe05de84640 1 -- 172.21.15.88:0/113665840 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe058072700 msgr2=0x7fe058148fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.463+0000 7fe05de84640 1 --2- 172.21.15.88:0/113665840 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe058072700 0x7fe058148fa0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe04802a220 tx=0x7fe0480046c0 comp rx=0 tx=0).stop 2024-09-16T01:36:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.463+0000 7fe05de84640 1 -- 172.21.15.88:0/113665840 shutdown_connections 2024-09-16T01:36:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.463+0000 7fe05de84640 1 --2- 172.21.15.88:0/113665840 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe04406c2b0 0x7fe04406e770 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.463+0000 7fe05de84640 1 --2- 172.21.15.88:0/113665840 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe05815b480 0x7fe05814ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.463+0000 7fe05de84640 1 --2- 172.21.15.88:0/113665840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0580b60c0 0x7fe05814e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.463+0000 7fe05de84640 1 --2- 172.21.15.88:0/113665840 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe058072700 0x7fe058148fa0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.463+0000 7fe05de84640 1 -- 172.21.15.88:0/113665840 >> 172.21.15.88:0/113665840 conn(0x7fe05806d0f0 msgr2=0x7fe05806d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.463+0000 7fe05de84640 1 -- 172.21.15.88:0/113665840 shutdown_connections 2024-09-16T01:36:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.463+0000 7fe05de84640 1 -- 172.21.15.88:0/113665840 wait complete. 2024-09-16T01:36:41.476 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-16T01:36:41.477 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-16T01:36:41.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.576+0000 7ff890d31640 1 Processor -- start 2024-09-16T01:36:41.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.576+0000 7ff890d31640 1 -- start start 2024-09-16T01:36:41.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.577+0000 7ff890d31640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff88c15b480 0x7ff88c0b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:41.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.577+0000 7ff890d31640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff88c0b60c0 0x7ff88c071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:41.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.577+0000 7ff890d31640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff88c072530 0x7ff88c070fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:41.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.577+0000 7ff890d31640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff88c0b6500 con 0x7ff88c15b480 2024-09-16T01:36:41.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.577+0000 7ff890d31640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff88c072970 con 0x7ff88c072530 2024-09-16T01:36:41.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.577+0000 7ff890d31640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff88c0ccff0 con 0x7ff88c0b60c0 2024-09-16T01:36:41.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.577+0000 7ff88a59c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff88c15b480 0x7ff88c0b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:41.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.577+0000 7ff88a59c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff88c15b480 0x7ff88c0b5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41268/0 (socket says 172.21.15.88:41268) 2024-09-16T01:36:41.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.577+0000 7ff88a59c640 1 -- 172.21.15.88:0/3416626599 learned_addr learned my addr 172.21.15.88:0/3416626599 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:41.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.578+0000 7ff88a59c640 1 -- 172.21.15.88:0/3416626599 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff88c0b60c0 msgr2=0x7ff88c071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:41.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.578+0000 7ff88a59c640 1 --2- 172.21.15.88:0/3416626599 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff88c0b60c0 0x7ff88c071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:41.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.578+0000 7ff88a59c640 1 -- 172.21.15.88:0/3416626599 >> v2:172.21.15.154:3300/0 conn(0x7ff88c072530 msgr2=0x7ff88c070fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:41.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.578+0000 7ff88a59c640 1 --2- 172.21.15.88:0/3416626599 >> v2:172.21.15.154:3300/0 conn(0x7ff88c072530 0x7ff88c070fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:41.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.578+0000 7ff88a59c640 1 -- 172.21.15.88:0/3416626599 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff88c0cd2d0 con 0x7ff88c15b480 2024-09-16T01:36:41.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.578+0000 7ff88a59c640 1 --2- 172.21.15.88:0/3416626599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff88c15b480 0x7ff88c0b5b80 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7ff88c0b2eb0 tx=0x7ff87c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=650b532a00d803a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:41.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.578+0000 7ff88959a640 1 -- 172.21.15.88:0/3416626599 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c037070 con 0x7ff88c15b480 2024-09-16T01:36:41.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.578+0000 7ff88959a640 1 -- 172.21.15.88:0/3416626599 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff87c032440 con 0x7ff88c15b480 2024-09-16T01:36:41.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.578+0000 7ff88959a640 1 -- 172.21.15.88:0/3416626599 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c032810 con 0x7ff88c15b480 2024-09-16T01:36:41.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.579+0000 7ff890d31640 1 -- 172.21.15.88:0/3416626599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff88c15b480 msgr2=0x7ff88c0b5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:41.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.579+0000 7ff890d31640 1 --2- 172.21.15.88:0/3416626599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff88c15b480 0x7ff88c0b5b80 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7ff88c0b2eb0 tx=0x7ff87c02a140 comp rx=0 tx=0).stop 2024-09-16T01:36:41.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.579+0000 7ff890d31640 1 -- 172.21.15.88:0/3416626599 shutdown_connections 2024-09-16T01:36:41.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.579+0000 7ff890d31640 1 --2- 172.21.15.88:0/3416626599 >> v2:172.21.15.154:3300/0 conn(0x7ff88c072530 0x7ff88c070fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:41.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.579+0000 7ff890d31640 1 --2- 172.21.15.88:0/3416626599 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff88c0b60c0 0x7ff88c071ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:41.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.579+0000 7ff890d31640 1 --2- 172.21.15.88:0/3416626599 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff88c15b480 0x7ff88c0b5b80 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:41.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.579+0000 7ff890d31640 1 -- 172.21.15.88:0/3416626599 >> 172.21.15.88:0/3416626599 conn(0x7ff88c06d0f0 msgr2=0x7ff88c06d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:41.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.579+0000 7ff890d31640 1 -- 172.21.15.88:0/3416626599 shutdown_connections 2024-09-16T01:36:41.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.579+0000 7ff890d31640 1 -- 172.21.15.88:0/3416626599 wait complete. 2024-09-16T01:36:41.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.580+0000 7ff890d31640 1 Processor -- start 2024-09-16T01:36:41.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.581+0000 7ff890d31640 1 -- start start 2024-09-16T01:36:41.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.581+0000 7ff890d31640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff88c072530 0x7ff88c149fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:41.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.581+0000 7ff890d31640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff88c0b60c0 0x7ff88c0cb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:41.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.581+0000 7ff890d31640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff88c15b480 0x7ff88c14a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:41.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.581+0000 7ff890d31640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff88c0cdb70 con 0x7ff88c15b480 2024-09-16T01:36:41.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.581+0000 7ff890d31640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff88c0cd890 con 0x7ff88c072530 2024-09-16T01:36:41.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.581+0000 7ff890d31640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff88c15b860 con 0x7ff88c0b60c0 2024-09-16T01:36:41.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.581+0000 7ff88ad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff88c15b480 0x7ff88c14a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:41.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.581+0000 7ff88ad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff88c15b480 0x7ff88c14a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41274/0 (socket says 172.21.15.88:41274) 2024-09-16T01:36:41.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.581+0000 7ff88ad9d640 1 -- 172.21.15.88:0/241963464 learned_addr learned my addr 172.21.15.88:0/241963464 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:41.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.582+0000 7ff88a59c640 1 --2- 172.21.15.88:0/241963464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff88c072530 0x7ff88c149fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:41.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.582+0000 7ff88ad9d640 1 -- 172.21.15.88:0/241963464 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff88c0b60c0 msgr2=0x7ff88c0cb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:41.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.582+0000 7ff889d9b640 1 --2- 172.21.15.88:0/241963464 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff88c0b60c0 0x7ff88c0cb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:41.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.582+0000 7ff88ad9d640 1 --2- 172.21.15.88:0/241963464 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff88c0b60c0 0x7ff88c0cb450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:41.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.582+0000 7ff88ad9d640 1 -- 172.21.15.88:0/241963464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff88c072530 msgr2=0x7ff88c149fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:41.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.582+0000 7ff88ad9d640 1 --2- 172.21.15.88:0/241963464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff88c072530 0x7ff88c149fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:41.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.582+0000 7ff88ad9d640 1 -- 172.21.15.88:0/241963464 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff88c0cbdc0 con 0x7ff88c15b480 2024-09-16T01:36:41.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.582+0000 7ff88a59c640 1 --2- 172.21.15.88:0/241963464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff88c072530 0x7ff88c149fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:41.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.582+0000 7ff889d9b640 1 --2- 172.21.15.88:0/241963464 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff88c0b60c0 0x7ff88c0cb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:41.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.582+0000 7ff88ad9d640 1 --2- 172.21.15.88:0/241963464 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff88c15b480 0x7ff88c14a980 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7ff88400b970 tx=0x7ff88400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:41.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.582+0000 7ff87affd640 1 -- 172.21.15.88:0/241963464 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff884019070 con 0x7ff88c15b480 2024-09-16T01:36:41.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.582+0000 7ff87affd640 1 -- 172.21.15.88:0/241963464 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff88400cb60 con 0x7ff88c15b480 2024-09-16T01:36:41.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.582+0000 7ff87affd640 1 -- 172.21.15.88:0/241963464 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff884014440 con 0x7ff88c15b480 2024-09-16T01:36:41.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.582+0000 7ff890d31640 1 -- 172.21.15.88:0/241963464 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff88c0c99b0 con 0x7ff88c15b480 2024-09-16T01:36:41.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.582+0000 7ff890d31640 1 -- 172.21.15.88:0/241963464 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff88c14a500 con 0x7ff88c15b480 2024-09-16T01:36:41.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.583+0000 7ff890d31640 1 -- 172.21.15.88:0/241963464 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff88c071ff0 con 0x7ff88c15b480 2024-09-16T01:36:41.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.583+0000 7ff87affd640 1 -- 172.21.15.88:0/241963464 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff88401c070 con 0x7ff88c15b480 2024-09-16T01:36:41.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.584+0000 7ff87affd640 1 --2- 172.21.15.88:0/241963464 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff86c06c2b0 0x7ff86c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:41.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.584+0000 7ff87affd640 1 -- 172.21.15.88:0/241963464 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7ff88408b1a0 con 0x7ff88c15b480 2024-09-16T01:36:41.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.584+0000 7ff88a59c640 1 --2- 172.21.15.88:0/241963464 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff86c06c2b0 0x7ff86c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:41.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.584+0000 7ff88a59c640 1 --2- 172.21.15.88:0/241963464 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff86c06c2b0 0x7ff86c06e770 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff87c034000 tx=0x7ff87c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:41.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.588+0000 7ff87affd640 1 -- 172.21.15.88:0/241963464 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff884059910 con 0x7ff88c15b480 2024-09-16T01:36:41.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:41.789+0000 7ff890d31640 1 -- 172.21.15.88:0/241963464 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7ff88c06d6e0 con 0x7ff88c15b480 2024-09-16T01:36:42.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.467+0000 7ff87affd640 1 -- 172.21.15.88:0/241963464 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v37) ==== 122+0+0 (secure 0 0 0) 0x7ff884059210 con 0x7ff88c15b480 2024-09-16T01:36:42.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.470+0000 7ff890d31640 1 -- 172.21.15.88:0/241963464 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff86c06c2b0 msgr2=0x7ff86c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:42.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.470+0000 7ff890d31640 1 --2- 172.21.15.88:0/241963464 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff86c06c2b0 0x7ff86c06e770 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff87c034000 tx=0x7ff87c02e000 comp rx=0 tx=0).stop 2024-09-16T01:36:42.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.470+0000 7ff890d31640 1 -- 172.21.15.88:0/241963464 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff88c15b480 msgr2=0x7ff88c14a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:42.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.470+0000 7ff890d31640 1 --2- 172.21.15.88:0/241963464 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff88c15b480 0x7ff88c14a980 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7ff88400b970 tx=0x7ff88400be30 comp rx=0 tx=0).stop 2024-09-16T01:36:42.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.470+0000 7ff890d31640 1 -- 172.21.15.88:0/241963464 shutdown_connections 2024-09-16T01:36:42.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.470+0000 7ff890d31640 1 --2- 172.21.15.88:0/241963464 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff86c06c2b0 0x7ff86c06e770 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:42.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.470+0000 7ff890d31640 1 --2- 172.21.15.88:0/241963464 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff88c15b480 0x7ff88c14a980 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:42.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.470+0000 7ff890d31640 1 --2- 172.21.15.88:0/241963464 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff88c0b60c0 0x7ff88c0cb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:42.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.470+0000 7ff890d31640 1 --2- 172.21.15.88:0/241963464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff88c072530 0x7ff88c149fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:42.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.470+0000 7ff890d31640 1 -- 172.21.15.88:0/241963464 >> 172.21.15.88:0/241963464 conn(0x7ff88c06d0f0 msgr2=0x7ff88c0ce860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:42.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.470+0000 7ff890d31640 1 -- 172.21.15.88:0/241963464 shutdown_connections 2024-09-16T01:36:42.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.470+0000 7ff890d31640 1 -- 172.21.15.88:0/241963464 wait complete. 2024-09-16T01:36:42.484 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-16T01:36:42.484 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:36:42.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.583+0000 7f78febb7640 1 Processor -- start 2024-09-16T01:36:42.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.583+0000 7f78febb7640 1 -- start start 2024-09-16T01:36:42.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.584+0000 7f78febb7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f78f806ef30 0x7f78f806bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:42.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.584+0000 7f78febb7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78f80ceef0 con 0x7f78f806c150 2024-09-16T01:36:42.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.584+0000 7f78febb7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78f80cf1d0 con 0x7f78f815c610 2024-09-16T01:36:42.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.584+0000 7f78febb7640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f78f806e7b0 con 0x7f78f806ef30 2024-09-16T01:36:42.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.584+0000 7f78fd154640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f78f806c150 0x7f78f81660c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54310/0 (socket says 172.21.15.88:54310) 2024-09-16T01:36:42.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.584+0000 7f78fd154640 1 -- 172.21.15.88:0/3992320149 learned_addr learned my addr 172.21.15.88:0/3992320149 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:42.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.584+0000 7f78f7fff640 1 --2- 172.21.15.88:0/3992320149 >> v2:172.21.15.154:3300/0 conn(0x7f78f806ef30 0x7f78f806bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:42.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.584+0000 7f78f77fe640 1 -- 172.21.15.88:0/3992320149 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3702860638 0 0) 0x7f78f80ceef0 con 0x7f78f806c150 2024-09-16T01:36:42.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.585+0000 7f78f77fe640 1 -- 172.21.15.88:0/3992320149 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78e0003520 con 0x7f78f806c150 2024-09-16T01:36:42.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.585+0000 7f78f77fe640 1 -- 172.21.15.88:0/3992320149 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 283113097 0 0) 0x7f78e0003520 con 0x7f78f806c150 2024-09-16T01:36:42.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.585+0000 7f78f77fe640 1 -- 172.21.15.88:0/3992320149 >> v2:172.21.15.154:3300/0 conn(0x7f78f806ef30 msgr2=0x7f78f806bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:42.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.585+0000 7f78f77fe640 1 --2- 172.21.15.88:0/3992320149 >> v2:172.21.15.154:3300/0 conn(0x7f78f806ef30 0x7f78f806bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:42.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.585+0000 7f78f77fe640 1 -- 172.21.15.88:0/3992320149 >> v1:172.21.15.154:6789/0 conn(0x7f78f815c610 legacy=0x7f78f80ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:42.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.585+0000 7f78f77fe640 1 -- 172.21.15.88:0/3992320149 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f80cf4b0 con 0x7f78f806c150 2024-09-16T01:36:42.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.585+0000 7f78f77fe640 1 -- 172.21.15.88:0/3992320149 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f78f00019f0 con 0x7f78f806c150 2024-09-16T01:36:42.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.585+0000 7f78f77fe640 1 -- 172.21.15.88:0/3992320149 >> v1:172.21.15.88:6789/0 conn(0x7f78f806c150 legacy=0x7f78f81660c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:42.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.585+0000 7f78f77fe640 1 --2- 172.21.15.88:0/3992320149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78e00039b0 0x7f78e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:42.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.585+0000 7f78f77fe640 1 --2- 172.21.15.88:0/3992320149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78e0004800 0x7f78e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:42.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.585+0000 7f78f77fe640 1 --2- 172.21.15.88:0/3992320149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f78e00051c0 0x7f78e000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:42.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.585+0000 7f78f77fe640 1 -- 172.21.15.88:0/3992320149 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f78e000c030 con 0x7f78e00039b0 2024-09-16T01:36:42.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.585+0000 7f78f77fe640 1 -- 172.21.15.88:0/3992320149 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f78e000c1e0 con 0x7f78e0004800 2024-09-16T01:36:42.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.585+0000 7f78f77fe640 1 -- 172.21.15.88:0/3992320149 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f78e000c3e0 con 0x7f78e00051c0 2024-09-16T01:36:42.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.585+0000 7f78fc953640 1 --2- 172.21.15.88:0/3992320149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78e00039b0 0x7f78e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:42.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.585+0000 7f78f7fff640 1 --2- 172.21.15.88:0/3992320149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78e0004800 0x7f78e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:42.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.585+0000 7f78fd154640 1 --2- 172.21.15.88:0/3992320149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f78e00051c0 0x7f78e000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:42.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.585+0000 7f78fc953640 1 -- 172.21.15.88:0/3992320149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f78e00051c0 msgr2=0x7f78e000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:42.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.585+0000 7f78fc953640 1 --2- 172.21.15.88:0/3992320149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f78e00051c0 0x7f78e000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:42.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.585+0000 7f78fc953640 1 -- 172.21.15.88:0/3992320149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78e0004800 msgr2=0x7f78e0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:42.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.585+0000 7f78fc953640 1 --2- 172.21.15.88:0/3992320149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78e0004800 0x7f78e0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:42.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.585+0000 7f78fc953640 1 -- 172.21.15.88:0/3992320149 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78e0003520 con 0x7f78e00039b0 2024-09-16T01:36:42.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.585+0000 7f78f7fff640 1 --2- 172.21.15.88:0/3992320149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78e0004800 0x7f78e0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:42.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.585+0000 7f78fd154640 1 -- 172.21.15.88:0/3992320149 reap_dead start 2024-09-16T01:36:42.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.586+0000 7f78fc953640 1 --2- 172.21.15.88:0/3992320149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78e00039b0 0x7f78e0003dd0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f78e8001750 tx=0x7f78e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba39696d35a834e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:42.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.586+0000 7f78f77fe640 1 -- 172.21.15.88:0/3992320149 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e8001370 con 0x7f78e00039b0 2024-09-16T01:36:42.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.586+0000 7f78f77fe640 1 -- 172.21.15.88:0/3992320149 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78e80052d0 con 0x7f78e00039b0 2024-09-16T01:36:42.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.586+0000 7f78f77fe640 1 -- 172.21.15.88:0/3992320149 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e802abe0 con 0x7f78e00039b0 2024-09-16T01:36:42.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.587+0000 7f78febb7640 1 -- 172.21.15.88:0/3992320149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78e00039b0 msgr2=0x7f78e0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:42.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.587+0000 7f78febb7640 1 --2- 172.21.15.88:0/3992320149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78e00039b0 0x7f78e0003dd0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f78e8001750 tx=0x7f78e0005830 comp rx=0 tx=0).stop 2024-09-16T01:36:42.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.587+0000 7f78febb7640 1 -- 172.21.15.88:0/3992320149 shutdown_connections 2024-09-16T01:36:42.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.587+0000 7f78febb7640 1 --2- 172.21.15.88:0/3992320149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78e00039b0 0x7f78e0003dd0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:42.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.587+0000 7f78febb7640 1 -- 172.21.15.88:0/3992320149 >> 172.21.15.88:0/3992320149 conn(0x7f78f80b2430 msgr2=0x7f78f80b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:42.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.587+0000 7f78febb7640 1 -- 172.21.15.88:0/3992320149 shutdown_connections 2024-09-16T01:36:42.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.587+0000 7f78febb7640 1 -- 172.21.15.88:0/3992320149 wait complete. 2024-09-16T01:36:42.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.588+0000 7f78febb7640 1 Processor -- start 2024-09-16T01:36:42.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.588+0000 7f78febb7640 1 -- start start 2024-09-16T01:36:42.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.589+0000 7f78febb7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f78e00039b0 0x7f78f8159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:42.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.589+0000 7f78febb7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78f8152020 0x7f78f81f5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:42.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.589+0000 7f78febb7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78f81f5ec0 0x7f78f81f6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:42.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.589+0000 7f78febb7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f78f80713c0 con 0x7f78f81f5ec0 2024-09-16T01:36:42.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.589+0000 7f78febb7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f78f806c580 con 0x7f78f8152020 2024-09-16T01:36:42.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.589+0000 7f78febb7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f78f8071540 con 0x7f78e00039b0 2024-09-16T01:36:42.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.589+0000 7f78fd154640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78f81f5ec0 0x7f78f81f6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:42.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.589+0000 7f78fd154640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78f81f5ec0 0x7f78f81f6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41304/0 (socket says 172.21.15.88:41304) 2024-09-16T01:36:42.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.589+0000 7f78fd154640 1 -- 172.21.15.88:0/2532333168 learned_addr learned my addr 172.21.15.88:0/2532333168 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:42.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.589+0000 7f78fc953640 1 --2- 172.21.15.88:0/2532333168 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f78e00039b0 0x7f78f8159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:42.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.589+0000 7f78f7fff640 1 --2- 172.21.15.88:0/2532333168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78f8152020 0x7f78f81f5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:42.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.589+0000 7f78fd154640 1 -- 172.21.15.88:0/2532333168 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f78e00039b0 msgr2=0x7f78f8159350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:42.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.589+0000 7f78fd154640 1 --2- 172.21.15.88:0/2532333168 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f78e00039b0 0x7f78f8159350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:42.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.589+0000 7f78fd154640 1 -- 172.21.15.88:0/2532333168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78f8152020 msgr2=0x7f78f81f5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:42.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.589+0000 7f78fd154640 1 --2- 172.21.15.88:0/2532333168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78f8152020 0x7f78f81f5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:42.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.589+0000 7f78fd154640 1 -- 172.21.15.88:0/2532333168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f8159890 con 0x7f78f81f5ec0 2024-09-16T01:36:42.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.589+0000 7f78f7fff640 1 --2- 172.21.15.88:0/2532333168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78f8152020 0x7f78f81f5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:42.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.589+0000 7f78fd154640 1 --2- 172.21.15.88:0/2532333168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78f81f5ec0 0x7f78f81f6340 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f78f80b0330 tx=0x7f78f0007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:42.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.590+0000 7f78f57fa640 1 -- 172.21.15.88:0/2532333168 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f000e9a0 con 0x7f78f81f5ec0 2024-09-16T01:36:42.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.590+0000 7f78f57fa640 1 -- 172.21.15.88:0/2532333168 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78f000eb60 con 0x7f78f81f5ec0 2024-09-16T01:36:42.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.590+0000 7f78febb7640 1 -- 172.21.15.88:0/2532333168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78f8159b30 con 0x7f78f81f5ec0 2024-09-16T01:36:42.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.590+0000 7f78f57fa640 1 -- 172.21.15.88:0/2532333168 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f0014430 con 0x7f78f81f5ec0 2024-09-16T01:36:42.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.590+0000 7f78febb7640 1 -- 172.21.15.88:0/2532333168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78f81531d0 con 0x7f78f81f5ec0 2024-09-16T01:36:42.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.591+0000 7f78febb7640 1 -- 172.21.15.88:0/2532333168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78f806bc10 con 0x7f78f81f5ec0 2024-09-16T01:36:42.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.591+0000 7f78f57fa640 1 -- 172.21.15.88:0/2532333168 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f78f00145d0 con 0x7f78f81f5ec0 2024-09-16T01:36:42.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.592+0000 7f78f57fa640 1 --2- 172.21.15.88:0/2532333168 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f78e406c2e0 0x7f78e406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:42.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.592+0000 7f78f57fa640 1 -- 172.21.15.88:0/2532333168 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f78f008af40 con 0x7f78f81f5ec0 2024-09-16T01:36:42.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.592+0000 7f78fc953640 1 --2- 172.21.15.88:0/2532333168 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f78e406c2e0 0x7f78e406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:42.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.592+0000 7f78fc953640 1 --2- 172.21.15.88:0/2532333168 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f78e406c2e0 0x7f78e406e7a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f78e8032000 tx=0x7f78e802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:42.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.596+0000 7f78f57fa640 1 -- 172.21.15.88:0/2532333168 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f78f0059060 con 0x7f78f81f5ec0 2024-09-16T01:36:42.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.795+0000 7f78febb7640 1 -- 172.21.15.88:0/2532333168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78f8156a50 con 0x7f78f81f5ec0 2024-09-16T01:36:42.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.796+0000 7f78f57fa640 1 -- 172.21.15.88:0/2532333168 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5220 (secure 0 0 0) 0x7f78f005d360 con 0x7f78f81f5ec0 2024-09-16T01:36:42.797 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:42.798 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":37,"btime":"2024-09-16T01:36:42:461977+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/581241194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":581241194},{"type":"v1","addr":"172.21.15.154:6834","nonce":581241194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2104508521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2104508521},{"type":"v1","addr":"172.21.15.154:6835","nonce":2104508521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5859,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/998283126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":998283126},{"type":"v1","addr":"172.21.15.88:6836","nonce":998283126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:39.786438+0000","modified":"2024-09-16T01:36:41.790966+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5862},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5862":{"gid":5862,"name":"a","rank":0,"incarnation":35,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6837/574169866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":574169866},{"type":"v1","addr":"172.21.15.88:6837","nonce":574169866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5862,"qdb_cluster":[5862]},"id":4}]} 2024-09-16T01:36:42.798 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 37 2024-09-16T01:36:42.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.798+0000 7f78febb7640 1 -- 172.21.15.88:0/2532333168 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f78e406c2e0 msgr2=0x7f78e406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:42.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.798+0000 7f78febb7640 1 --2- 172.21.15.88:0/2532333168 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f78e406c2e0 0x7f78e406e7a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f78e8032000 tx=0x7f78e802e000 comp rx=0 tx=0).stop 2024-09-16T01:36:42.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.798+0000 7f78febb7640 1 -- 172.21.15.88:0/2532333168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78f81f5ec0 msgr2=0x7f78f81f6340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:42.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.798+0000 7f78febb7640 1 --2- 172.21.15.88:0/2532333168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78f81f5ec0 0x7f78f81f6340 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f78f80b0330 tx=0x7f78f0007bc0 comp rx=0 tx=0).stop 2024-09-16T01:36:42.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.798+0000 7f78febb7640 1 -- 172.21.15.88:0/2532333168 shutdown_connections 2024-09-16T01:36:42.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.798+0000 7f78febb7640 1 --2- 172.21.15.88:0/2532333168 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f78e406c2e0 0x7f78e406e7a0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:42.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.798+0000 7f78febb7640 1 --2- 172.21.15.88:0/2532333168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78f81f5ec0 0x7f78f81f6340 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:42.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.798+0000 7f78febb7640 1 --2- 172.21.15.88:0/2532333168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78f8152020 0x7f78f81f5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:42.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.798+0000 7f78febb7640 1 --2- 172.21.15.88:0/2532333168 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f78e00039b0 0x7f78f8159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:42.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.798+0000 7f78febb7640 1 -- 172.21.15.88:0/2532333168 >> 172.21.15.88:0/2532333168 conn(0x7f78f80b2430 msgr2=0x7f78f815f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:42.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.798+0000 7f78febb7640 1 -- 172.21.15.88:0/2532333168 shutdown_connections 2024-09-16T01:36:42.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.798+0000 7f78febb7640 1 -- 172.21.15.88:0/2532333168 wait complete. 2024-09-16T01:36:42.810 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:36:42.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.907+0000 7f4fb3f86640 1 Processor -- start 2024-09-16T01:36:42.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.908+0000 7f4fb3f86640 1 -- start start 2024-09-16T01:36:42.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.908+0000 7f4fb3f86640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4fac15e800 0x7f4fac0b5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:42.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.908+0000 7f4fb3f86640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fac15ebe0 0x7f4fac15f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:42.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.908+0000 7f4fb3f86640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fac06ea60 con 0x7f4fac0b6e50 2024-09-16T01:36:42.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.908+0000 7f4fb3f86640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f4fac06ed40 con 0x7f4fac15e800 2024-09-16T01:36:42.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.908+0000 7f4fb3f86640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4fac06cde0 con 0x7f4fac15ebe0 2024-09-16T01:36:42.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.909+0000 7f4fb1d22640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4fac15e800 0x7f4fac0b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:42.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.909+0000 7f4fb1521640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fac15ebe0 0x7f4fac15f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:42.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.909+0000 7f4fb2523640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f4fac0b6e50 0x7f4fac0b7280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:54142/0 (socket says 172.21.15.88:54142) 2024-09-16T01:36:42.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.909+0000 7f4fb1521640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fac15ebe0 0x7f4fac15f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:34992/0 (socket says 172.21.15.88:34992) 2024-09-16T01:36:42.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.909+0000 7f4fb1d22640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4fac15e800 0x7f4fac0b5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:46186/0 (socket says 172.21.15.88:46186) 2024-09-16T01:36:42.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.909+0000 7f4fb2523640 1 -- 172.21.15.88:0/1711709367 learned_addr learned my addr 172.21.15.88:0/1711709367 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:42.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.909+0000 7f4fb0d20640 1 -- 172.21.15.88:0/1711709367 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2586252413 0 0) 0x7f4fac06ea60 con 0x7f4fac0b6e50 2024-09-16T01:36:42.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.909+0000 7f4fb0d20640 1 -- 172.21.15.88:0/1711709367 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f94003520 con 0x7f4fac0b6e50 2024-09-16T01:36:42.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.909+0000 7f4fb1521640 1 -- 172.21.15.88:0/1711709367 >> v2:172.21.15.154:3300/0 conn(0x7f4fac15e800 msgr2=0x7f4fac0b5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:42.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.909+0000 7f4fb1521640 1 --2- 172.21.15.88:0/1711709367 >> v2:172.21.15.154:3300/0 conn(0x7f4fac15e800 0x7f4fac0b5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:42.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.909+0000 7f4fb1521640 1 -- 172.21.15.88:0/1711709367 >> v1:172.21.15.154:6789/0 conn(0x7f4fac0b6e50 legacy=0x7f4fac0b7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:42.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.909+0000 7f4fb1521640 1 -- 172.21.15.88:0/1711709367 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fac06d0c0 con 0x7f4fac15ebe0 2024-09-16T01:36:42.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.909+0000 7f4fb1d22640 1 --2- 172.21.15.88:0/1711709367 >> v2:172.21.15.154:3300/0 conn(0x7f4fac15e800 0x7f4fac0b5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:36:42.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.910+0000 7f4fb1521640 1 --2- 172.21.15.88:0/1711709367 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fac15ebe0 0x7f4fac15f070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4fa00097a0 tx=0x7f4fa002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a65216f5957cfb42 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:42.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.910+0000 7f4fb0d20640 1 -- 172.21.15.88:0/1711709367 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa0037070 con 0x7f4fac15ebe0 2024-09-16T01:36:42.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.910+0000 7f4fb0d20640 1 -- 172.21.15.88:0/1711709367 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fa0032440 con 0x7f4fac15ebe0 2024-09-16T01:36:42.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.910+0000 7f4fb0d20640 1 -- 172.21.15.88:0/1711709367 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa00327f0 con 0x7f4fac15ebe0 2024-09-16T01:36:42.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.911+0000 7f4fb3f86640 1 -- 172.21.15.88:0/1711709367 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fac15ebe0 msgr2=0x7f4fac15f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:42.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.911+0000 7f4fb3f86640 1 --2- 172.21.15.88:0/1711709367 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fac15ebe0 0x7f4fac15f070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4fa00097a0 tx=0x7f4fa002a180 comp rx=0 tx=0).stop 2024-09-16T01:36:42.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.911+0000 7f4fb3f86640 1 -- 172.21.15.88:0/1711709367 shutdown_connections 2024-09-16T01:36:42.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.911+0000 7f4fb3f86640 1 --2- 172.21.15.88:0/1711709367 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fac15ebe0 0x7f4fac15f070 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:42.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.911+0000 7f4fb3f86640 1 --2- 172.21.15.88:0/1711709367 >> v2:172.21.15.154:3300/0 conn(0x7f4fac15e800 0x7f4fac0b5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:42.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.911+0000 7f4fb3f86640 1 -- 172.21.15.88:0/1711709367 >> 172.21.15.88:0/1711709367 conn(0x7f4fac0d24d0 msgr2=0x7f4fac0d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:42.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.911+0000 7f4fb3f86640 1 -- 172.21.15.88:0/1711709367 shutdown_connections 2024-09-16T01:36:42.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.911+0000 7f4fb3f86640 1 -- 172.21.15.88:0/1711709367 wait complete. 2024-09-16T01:36:42.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.912+0000 7f4fb3f86640 1 Processor -- start 2024-09-16T01:36:42.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.912+0000 7f4fb3f86640 1 -- start start 2024-09-16T01:36:42.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.913+0000 7f4fb3f86640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fac0b6e50 0x7f4fac1552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:42.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.913+0000 7f4fb3f86640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fac15e800 0x7f4fac15a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:42.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.913+0000 7f4fb3f86640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fac15ebe0 0x7f4fac15ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:42.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.913+0000 7f4fb3f86640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4fac06e680 con 0x7f4fac15ebe0 2024-09-16T01:36:42.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.913+0000 7f4fb3f86640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4fac0d31e0 con 0x7f4fac0b6e50 2024-09-16T01:36:42.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.913+0000 7f4fb3f86640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4fac06e800 con 0x7f4fac15e800 2024-09-16T01:36:42.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.913+0000 7f4fb2523640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fac15ebe0 0x7f4fac15ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:42.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.913+0000 7f4fb2523640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fac15ebe0 0x7f4fac15ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41320/0 (socket says 172.21.15.88:41320) 2024-09-16T01:36:42.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.913+0000 7f4fb2523640 1 -- 172.21.15.88:0/1382095609 learned_addr learned my addr 172.21.15.88:0/1382095609 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:42.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.913+0000 7f4fb1521640 1 --2- 172.21.15.88:0/1382095609 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fac15e800 0x7f4fac15a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:42.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.913+0000 7f4fb1d22640 1 --2- 172.21.15.88:0/1382095609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fac0b6e50 0x7f4fac1552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:42.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.913+0000 7f4fb2523640 1 -- 172.21.15.88:0/1382095609 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fac15e800 msgr2=0x7f4fac15a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:42.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.913+0000 7f4fb2523640 1 --2- 172.21.15.88:0/1382095609 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fac15e800 0x7f4fac15a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:42.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.913+0000 7f4fb2523640 1 -- 172.21.15.88:0/1382095609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fac0b6e50 msgr2=0x7f4fac1552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:42.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.913+0000 7f4fb2523640 1 --2- 172.21.15.88:0/1382095609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fac0b6e50 0x7f4fac1552b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:42.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.913+0000 7f4fb2523640 1 -- 172.21.15.88:0/1382095609 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fac156430 con 0x7f4fac15ebe0 2024-09-16T01:36:42.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.913+0000 7f4fb2523640 1 --2- 172.21.15.88:0/1382095609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fac15ebe0 0x7f4fac15ae10 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400c430 tx=0x7f4fa400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:42.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.913+0000 7f4faa7fc640 1 -- 172.21.15.88:0/1382095609 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa4017070 con 0x7f4fac15ebe0 2024-09-16T01:36:42.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.913+0000 7f4faa7fc640 1 -- 172.21.15.88:0/1382095609 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fa4005ce0 con 0x7f4fac15ebe0 2024-09-16T01:36:42.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.913+0000 7f4fb3f86640 1 -- 172.21.15.88:0/1382095609 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fac156710 con 0x7f4fac15ebe0 2024-09-16T01:36:42.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.913+0000 7f4faa7fc640 1 -- 172.21.15.88:0/1382095609 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa4007310 con 0x7f4fac15ebe0 2024-09-16T01:36:42.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.913+0000 7f4fb3f86640 1 -- 172.21.15.88:0/1382095609 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fac0cd0c0 con 0x7f4fac15ebe0 2024-09-16T01:36:42.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.914+0000 7f4fb3f86640 1 -- 172.21.15.88:0/1382095609 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fac15f070 con 0x7f4fac15ebe0 2024-09-16T01:36:42.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.915+0000 7f4faa7fc640 1 -- 172.21.15.88:0/1382095609 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4fa401a070 con 0x7f4fac15ebe0 2024-09-16T01:36:42.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.915+0000 7f4faa7fc640 1 --2- 172.21.15.88:0/1382095609 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f9806c2b0 0x7f4f9806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:42.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.915+0000 7f4faa7fc640 1 -- 172.21.15.88:0/1382095609 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f4fa408a7b0 con 0x7f4fac15ebe0 2024-09-16T01:36:42.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.915+0000 7f4fb1d22640 1 --2- 172.21.15.88:0/1382095609 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f9806c2b0 0x7f4f9806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:42.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.916+0000 7f4fb1d22640 1 --2- 172.21.15.88:0/1382095609 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f9806c2b0 0x7f4f9806e770 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c009bd0 tx=0x7f4f9c00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:42.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:42.920+0000 7f4faa7fc640 1 -- 172.21.15.88:0/1382095609 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4fa4058960 con 0x7f4fac15ebe0 2024-09-16T01:36:43.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.085+0000 7f4fb3f86640 1 -- 172.21.15.88:0/1382095609 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4fac0b7560 con 0x7f4fac15ebe0 2024-09-16T01:36:43.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.086+0000 7f4faa7fc640 1 -- 172.21.15.88:0/1382095609 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) ==== 74+0+16701 (secure 0 0 0) 0x7f4fa405cc60 con 0x7f4fac15ebe0 2024-09-16T01:36:43.088 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:43.088 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":58,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:36:41.455722+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:36:38.121335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-16T01:36:38.755908+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:36:43.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.088+0000 7f4fb3f86640 1 -- 172.21.15.88:0/1382095609 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f9806c2b0 msgr2=0x7f4f9806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:43.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.088+0000 7f4fb3f86640 1 --2- 172.21.15.88:0/1382095609 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f9806c2b0 0x7f4f9806e770 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c009bd0 tx=0x7f4f9c00b040 comp rx=0 tx=0).stop 2024-09-16T01:36:43.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.088+0000 7f4fb3f86640 1 -- 172.21.15.88:0/1382095609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fac15ebe0 msgr2=0x7f4fac15ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:43.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.088+0000 7f4fb3f86640 1 --2- 172.21.15.88:0/1382095609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fac15ebe0 0x7f4fac15ae10 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400c430 tx=0x7f4fa400c8f0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.089+0000 7f4fb3f86640 1 -- 172.21.15.88:0/1382095609 shutdown_connections 2024-09-16T01:36:43.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.089+0000 7f4fb3f86640 1 --2- 172.21.15.88:0/1382095609 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f9806c2b0 0x7f4f9806e770 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.089+0000 7f4fb3f86640 1 --2- 172.21.15.88:0/1382095609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fac15ebe0 0x7f4fac15ae10 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.089+0000 7f4fb3f86640 1 --2- 172.21.15.88:0/1382095609 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fac15e800 0x7f4fac15a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.089+0000 7f4fb3f86640 1 --2- 172.21.15.88:0/1382095609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fac0b6e50 0x7f4fac1552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.089+0000 7f4fb3f86640 1 -- 172.21.15.88:0/1382095609 >> 172.21.15.88:0/1382095609 conn(0x7f4fac0d24d0 msgr2=0x7f4fac06f400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:43.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.089+0000 7f4fb3f86640 1 -- 172.21.15.88:0/1382095609 shutdown_connections 2024-09-16T01:36:43.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.089+0000 7f4fb3f86640 1 -- 172.21.15.88:0/1382095609 wait complete. 2024-09-16T01:36:43.101 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T01:36:43.101 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:36:43.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.198+0000 7f85fa650640 1 Processor -- start 2024-09-16T01:36:43.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.199+0000 7f85fa650640 1 -- start start 2024-09-16T01:36:43.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.199+0000 7f85fa650640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f85f406d430 0x7f85f4166250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:43.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.199+0000 7f85fa650640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85f415f130 con 0x7f85f415b460 2024-09-16T01:36:43.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.199+0000 7f85fa650640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85f415f410 con 0x7f85f40d0fe0 2024-09-16T01:36:43.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.199+0000 7f85fa650640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f85f415eee0 con 0x7f85f406d430 2024-09-16T01:36:43.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f3fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f85f415b460 0x7f85f40b44b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54316/0 (socket says 172.21.15.88:54316) 2024-09-16T01:36:43.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f3fff640 1 -- 172.21.15.88:0/2938970104 learned_addr learned my addr 172.21.15.88:0/2938970104 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:43.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f8bed640 1 --2- 172.21.15.88:0/2938970104 >> v2:172.21.15.154:3300/0 conn(0x7f85f406d430 0x7f85f4166250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:43.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f37fe640 1 -- 172.21.15.88:0/2938970104 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2327641646 0 0) 0x7f85f415f130 con 0x7f85f415b460 2024-09-16T01:36:43.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f37fe640 1 -- 172.21.15.88:0/2938970104 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85e0005730 con 0x7f85f415b460 2024-09-16T01:36:43.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f37fe640 1 -- 172.21.15.88:0/2938970104 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3936545292 0 0) 0x7f85e0005730 con 0x7f85f415b460 2024-09-16T01:36:43.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f37fe640 1 -- 172.21.15.88:0/2938970104 >> v2:172.21.15.154:3300/0 conn(0x7f85f406d430 msgr2=0x7f85f4166250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:43.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f37fe640 1 --2- 172.21.15.88:0/2938970104 >> v2:172.21.15.154:3300/0 conn(0x7f85f406d430 0x7f85f4166250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f37fe640 1 -- 172.21.15.88:0/2938970104 >> v1:172.21.15.154:6789/0 conn(0x7f85f40d0fe0 legacy=0x7f85f406cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:43.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f37fe640 1 -- 172.21.15.88:0/2938970104 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85f415f6f0 con 0x7f85f415b460 2024-09-16T01:36:43.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f37fe640 1 -- 172.21.15.88:0/2938970104 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f85e4001670 con 0x7f85f415b460 2024-09-16T01:36:43.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f37fe640 1 -- 172.21.15.88:0/2938970104 >> v1:172.21.15.88:6789/0 conn(0x7f85f415b460 legacy=0x7f85f40b44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:43.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f37fe640 1 --2- 172.21.15.88:0/2938970104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85e0005bc0 0x7f85e0005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:43.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f37fe640 1 --2- 172.21.15.88:0/2938970104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85e0006a10 0x7f85e0006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:43.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f37fe640 1 --2- 172.21.15.88:0/2938970104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85e00073d0 0x7f85e000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:43.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f37fe640 1 -- 172.21.15.88:0/2938970104 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f85f415eee0 con 0x7f85e0005bc0 2024-09-16T01:36:43.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f37fe640 1 -- 172.21.15.88:0/2938970104 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f85e000e270 con 0x7f85e00073d0 2024-09-16T01:36:43.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f37fe640 1 -- 172.21.15.88:0/2938970104 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f85e000e470 con 0x7f85e0006a10 2024-09-16T01:36:43.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f3fff640 1 --2- 172.21.15.88:0/2938970104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85e0005bc0 0x7f85e0005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:43.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f8bed640 1 --2- 172.21.15.88:0/2938970104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85e00073d0 0x7f85e000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:43.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85ebfff640 1 --2- 172.21.15.88:0/2938970104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85e0006a10 0x7f85e0006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:43.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f3fff640 1 -- 172.21.15.88:0/2938970104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85e0006a10 msgr2=0x7f85e0006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:43.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f3fff640 1 --2- 172.21.15.88:0/2938970104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85e0006a10 0x7f85e0006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f3fff640 1 -- 172.21.15.88:0/2938970104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85e00073d0 msgr2=0x7f85e000dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:43.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f3fff640 1 --2- 172.21.15.88:0/2938970104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85e00073d0 0x7f85e000dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f3fff640 1 -- 172.21.15.88:0/2938970104 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85e0005730 con 0x7f85e0005bc0 2024-09-16T01:36:43.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f8bed640 1 --2- 172.21.15.88:0/2938970104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85e00073d0 0x7f85e000dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:43.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.200+0000 7f85f8bed640 1 -- 172.21.15.88:0/2938970104 reap_dead start 2024-09-16T01:36:43.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.201+0000 7f85f3fff640 1 --2- 172.21.15.88:0/2938970104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85e0005bc0 0x7f85e0005fe0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f85e400c210 tx=0x7f85e0007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd5e651d0b0a2277 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:43.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.201+0000 7f85f37fe640 1 -- 172.21.15.88:0/2938970104 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e4007d30 con 0x7f85e0005bc0 2024-09-16T01:36:43.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.201+0000 7f85f37fe640 1 -- 172.21.15.88:0/2938970104 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85e40058d0 con 0x7f85e0005bc0 2024-09-16T01:36:43.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.201+0000 7f85f37fe640 1 -- 172.21.15.88:0/2938970104 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e4005d20 con 0x7f85e0005bc0 2024-09-16T01:36:43.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.202+0000 7f85fa650640 1 -- 172.21.15.88:0/2938970104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85e0005bc0 msgr2=0x7f85e0005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:43.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.202+0000 7f85fa650640 1 --2- 172.21.15.88:0/2938970104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85e0005bc0 0x7f85e0005fe0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f85e400c210 tx=0x7f85e0007a40 comp rx=0 tx=0).stop 2024-09-16T01:36:43.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.202+0000 7f85fa650640 1 -- 172.21.15.88:0/2938970104 shutdown_connections 2024-09-16T01:36:43.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.202+0000 7f85fa650640 1 --2- 172.21.15.88:0/2938970104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85e0005bc0 0x7f85e0005fe0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.202+0000 7f85fa650640 1 -- 172.21.15.88:0/2938970104 >> 172.21.15.88:0/2938970104 conn(0x7f85f40b6ec0 msgr2=0x7f85f40b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:43.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.202+0000 7f85fa650640 1 -- 172.21.15.88:0/2938970104 shutdown_connections 2024-09-16T01:36:43.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.202+0000 7f85fa650640 1 -- 172.21.15.88:0/2938970104 wait complete. 2024-09-16T01:36:43.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.203+0000 7f85fa650640 1 Processor -- start 2024-09-16T01:36:43.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.203+0000 7f85fa650640 1 -- start start 2024-09-16T01:36:43.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.204+0000 7f85fa650640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85f40ce610 0x7f85f40b03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:43.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.204+0000 7f85fa650640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85f415b840 0x7f85f414ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:43.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.204+0000 7f85fa650640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85f40ce9f0 0x7f85f40cee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:43.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.204+0000 7f85fa650640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f85f40b4630 con 0x7f85f415b840 2024-09-16T01:36:43.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.204+0000 7f85fa650640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f85f40b44b0 con 0x7f85f40ce9f0 2024-09-16T01:36:43.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.204+0000 7f85fa650640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f85f40b47b0 con 0x7f85f40ce610 2024-09-16T01:36:43.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.204+0000 7f85ebfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85f415b840 0x7f85f414ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:43.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.204+0000 7f85ebfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85f415b840 0x7f85f414ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41334/0 (socket says 172.21.15.88:41334) 2024-09-16T01:36:43.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.204+0000 7f85ebfff640 1 -- 172.21.15.88:0/3985331329 learned_addr learned my addr 172.21.15.88:0/3985331329 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:43.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.204+0000 7f85f3fff640 1 --2- 172.21.15.88:0/3985331329 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85f40ce610 0x7f85f40b03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:43.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.204+0000 7f85f8bed640 1 --2- 172.21.15.88:0/3985331329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85f40ce9f0 0x7f85f40cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:43.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.204+0000 7f85ebfff640 1 -- 172.21.15.88:0/3985331329 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85f40ce610 msgr2=0x7f85f40b03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:43.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.204+0000 7f85ebfff640 1 --2- 172.21.15.88:0/3985331329 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85f40ce610 0x7f85f40b03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.204+0000 7f85ebfff640 1 -- 172.21.15.88:0/3985331329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85f40ce9f0 msgr2=0x7f85f40cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:43.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.204+0000 7f85ebfff640 1 --2- 172.21.15.88:0/3985331329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85f40ce9f0 0x7f85f40cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.204+0000 7f85ebfff640 1 -- 172.21.15.88:0/3985331329 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85f40b08f0 con 0x7f85f415b840 2024-09-16T01:36:43.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.204+0000 7f85ebfff640 1 --2- 172.21.15.88:0/3985331329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85f415b840 0x7f85f414ff70 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f85dc00e500 tx=0x7f85dc00e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:43.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.204+0000 7f85f17fa640 1 -- 172.21.15.88:0/3985331329 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85dc0053a0 con 0x7f85f415b840 2024-09-16T01:36:43.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.205+0000 7f85f17fa640 1 -- 172.21.15.88:0/3985331329 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85dc018070 con 0x7f85f415b840 2024-09-16T01:36:43.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.205+0000 7f85fa650640 1 -- 172.21.15.88:0/3985331329 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85f415be00 con 0x7f85f415b840 2024-09-16T01:36:43.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.205+0000 7f85f17fa640 1 -- 172.21.15.88:0/3985331329 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85dc003810 con 0x7f85f415b840 2024-09-16T01:36:43.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.205+0000 7f85fa650640 1 -- 172.21.15.88:0/3985331329 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85f415c3c0 con 0x7f85f415b840 2024-09-16T01:36:43.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.205+0000 7f85fa650640 1 -- 172.21.15.88:0/3985331329 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85f40b4c20 con 0x7f85f415b840 2024-09-16T01:36:43.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.206+0000 7f85f17fa640 1 -- 172.21.15.88:0/3985331329 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f85dc0039b0 con 0x7f85f415b840 2024-09-16T01:36:43.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.206+0000 7f85f17fa640 1 --2- 172.21.15.88:0/3985331329 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f85d406c2e0 0x7f85d406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:43.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.206+0000 7f85f17fa640 1 -- 172.21.15.88:0/3985331329 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f85dc08b170 con 0x7f85f415b840 2024-09-16T01:36:43.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.206+0000 7f85f3fff640 1 --2- 172.21.15.88:0/3985331329 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f85d406c2e0 0x7f85d406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:43.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.207+0000 7f85f3fff640 1 --2- 172.21.15.88:0/3985331329 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f85d406c2e0 0x7f85d406e7a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f85e4033000 tx=0x7f85e402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:43.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.211+0000 7f85f17fa640 1 -- 172.21.15.88:0/3985331329 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f85dc0598f0 con 0x7f85f415b840 2024-09-16T01:36:43.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.372+0000 7f85fa650640 1 -- 172.21.15.88:0/3985331329 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f85f40b5540 con 0x7f85d406c2e0 2024-09-16T01:36:43.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.378+0000 7f85f17fa640 1 -- 172.21.15.88:0/3985331329 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251883 (secure 0 0 0) 0x7f85f40b5540 con 0x7f85d406c2e0 2024-09-16T01:36:43.382 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:43.384 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:36:43.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.381+0000 7f85fa650640 1 -- 172.21.15.88:0/3985331329 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f85d406c2e0 msgr2=0x7f85d406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:43.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.381+0000 7f85fa650640 1 --2- 172.21.15.88:0/3985331329 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f85d406c2e0 0x7f85d406e7a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f85e4033000 tx=0x7f85e402e000 comp rx=0 tx=0).stop 2024-09-16T01:36:43.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.381+0000 7f85fa650640 1 -- 172.21.15.88:0/3985331329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85f415b840 msgr2=0x7f85f414ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:43.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.381+0000 7f85fa650640 1 --2- 172.21.15.88:0/3985331329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85f415b840 0x7f85f414ff70 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f85dc00e500 tx=0x7f85dc00e9c0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.382+0000 7f85fa650640 1 -- 172.21.15.88:0/3985331329 shutdown_connections 2024-09-16T01:36:43.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.382+0000 7f85fa650640 1 --2- 172.21.15.88:0/3985331329 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f85d406c2e0 0x7f85d406e7a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.382+0000 7f85fa650640 1 --2- 172.21.15.88:0/3985331329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85f40ce9f0 0x7f85f40cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.382+0000 7f85fa650640 1 --2- 172.21.15.88:0/3985331329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85f415b840 0x7f85f414ff70 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.382+0000 7f85fa650640 1 --2- 172.21.15.88:0/3985331329 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85f40ce610 0x7f85f40b03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.382+0000 7f85fa650640 1 -- 172.21.15.88:0/3985331329 >> 172.21.15.88:0/3985331329 conn(0x7f85f40b6ec0 msgr2=0x7f85f4073070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:43.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.382+0000 7f85fa650640 1 -- 172.21.15.88:0/3985331329 shutdown_connections 2024-09-16T01:36:43.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.382+0000 7f85fa650640 1 -- 172.21.15.88:0/3985331329 wait complete. 2024-09-16T01:36:43.394 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-09-16T01:36:42.545047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":72,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":226796,"kb_used_data":6396,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754747924,"statfs":{"total":773094113280,"available":772861874176,"internally_reserved":0,"allocated":6549504,"data_stored":3383139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":225496626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.105244"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.474906+0000","last_change":"2024-09-16T01:36:40.473004+0000","last_active":"2024-09-16T01:36:40.474906+0000","last_peered":"2024-09-16T01:36:40.474906+0000","last_clean":"2024-09-16T01:36:40.474906+0000","last_became_active":"2024-09-16T01:36:40.472704+0000","last_became_peered":"2024-09-16T01:36:40.472704+0000","last_unstale":"2024-09-16T01:36:40.474906+0000","last_undegraded":"2024-09-16T01:36:40.474906+0000","last_fullsized":"2024-09-16T01:36:40.474906+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:29:50.303208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.472667+0000","last_change":"2024-09-16T01:36:40.466966+0000","last_active":"2024-09-16T01:36:40.472667+0000","last_peered":"2024-09-16T01:36:40.472667+0000","last_clean":"2024-09-16T01:36:40.472667+0000","last_became_active":"2024-09-16T01:36:40.466714+0000","last_became_peered":"2024-09-16T01:36:40.466714+0000","last_unstale":"2024-09-16T01:36:40.472667+0000","last_undegraded":"2024-09-16T01:36:40.472667+0000","last_fullsized":"2024-09-16T01:36:40.472667+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:21.800309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478466+0000","last_change":"2024-09-16T01:36:40.473300+0000","last_active":"2024-09-16T01:36:40.478466+0000","last_peered":"2024-09-16T01:36:40.478466+0000","last_clean":"2024-09-16T01:36:40.478466+0000","last_became_active":"2024-09-16T01:36:40.472936+0000","last_became_peered":"2024-09-16T01:36:40.472936+0000","last_unstale":"2024-09-16T01:36:40.478466+0000","last_undegraded":"2024-09-16T01:36:40.478466+0000","last_fullsized":"2024-09-16T01:36:40.478466+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:31:33.950166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.470583+0000","last_change":"2024-09-16T01:36:40.464619+0000","last_active":"2024-09-16T01:36:40.470583+0000","last_peered":"2024-09-16T01:36:40.470583+0000","last_clean":"2024-09-16T01:36:40.470583+0000","last_became_active":"2024-09-16T01:36:40.464296+0000","last_became_peered":"2024-09-16T01:36:40.464296+0000","last_unstale":"2024-09-16T01:36:40.470583+0000","last_undegraded":"2024-09-16T01:36:40.470583+0000","last_fullsized":"2024-09-16T01:36:40.470583+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:30:03.124326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.471650+0000","last_change":"2024-09-16T01:36:40.465279+0000","last_active":"2024-09-16T01:36:40.471650+0000","last_peered":"2024-09-16T01:36:40.471650+0000","last_clean":"2024-09-16T01:36:40.471650+0000","last_became_active":"2024-09-16T01:36:40.465013+0000","last_became_peered":"2024-09-16T01:36:40.465013+0000","last_unstale":"2024-09-16T01:36:40.471650+0000","last_undegraded":"2024-09-16T01:36:40.471650+0000","last_fullsized":"2024-09-16T01:36:40.471650+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:29.596455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475195+0000","last_change":"2024-09-16T01:36:40.463614+0000","last_active":"2024-09-16T01:36:40.475195+0000","last_peered":"2024-09-16T01:36:40.475195+0000","last_clean":"2024-09-16T01:36:40.475195+0000","last_became_active":"2024-09-16T01:36:40.462404+0000","last_became_peered":"2024-09-16T01:36:40.462404+0000","last_unstale":"2024-09-16T01:36:40.475195+0000","last_undegraded":"2024-09-16T01:36:40.475195+0000","last_fullsized":"2024-09-16T01:36:40.475195+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:32.557899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.469842+0000","last_change":"2024-09-16T01:36:40.464510+0000","last_active":"2024-09-16T01:36:40.469842+0000","last_peered":"2024-09-16T01:36:40.469842+0000","last_clean":"2024-09-16T01:36:40.469842+0000","last_became_active":"2024-09-16T01:36:40.464235+0000","last_became_peered":"2024-09-16T01:36:40.464235+0000","last_unstale":"2024-09-16T01:36:40.469842+0000","last_undegraded":"2024-09-16T01:36:40.469842+0000","last_fullsized":"2024-09-16T01:36:40.469842+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:33:26.780863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476901+0000","last_change":"2024-09-16T01:36:40.473294+0000","last_active":"2024-09-16T01:36:40.476901+0000","last_peered":"2024-09-16T01:36:40.476901+0000","last_clean":"2024-09-16T01:36:40.476901+0000","last_became_active":"2024-09-16T01:36:40.472894+0000","last_became_peered":"2024-09-16T01:36:40.472894+0000","last_unstale":"2024-09-16T01:36:40.476901+0000","last_undegraded":"2024-09-16T01:36:40.476901+0000","last_fullsized":"2024-09-16T01:36:40.476901+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:56:24.891884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.472198+0000","last_change":"2024-09-16T01:36:40.468454+0000","last_active":"2024-09-16T01:36:40.472198+0000","last_peered":"2024-09-16T01:36:40.472198+0000","last_clean":"2024-09-16T01:36:40.472198+0000","last_became_active":"2024-09-16T01:36:40.468194+0000","last_became_peered":"2024-09-16T01:36:40.468194+0000","last_unstale":"2024-09-16T01:36:40.472198+0000","last_undegraded":"2024-09-16T01:36:40.472198+0000","last_fullsized":"2024-09-16T01:36:40.472198+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:43.168711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.465959+0000","last_change":"2024-09-16T01:36:40.462236+0000","last_active":"2024-09-16T01:36:40.465959+0000","last_peered":"2024-09-16T01:36:40.465959+0000","last_clean":"2024-09-16T01:36:40.465959+0000","last_became_active":"2024-09-16T01:36:40.461959+0000","last_became_peered":"2024-09-16T01:36:40.461959+0000","last_unstale":"2024-09-16T01:36:40.465959+0000","last_undegraded":"2024-09-16T01:36:40.465959+0000","last_fullsized":"2024-09-16T01:36:40.465959+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:41:57.274746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.477409+0000","last_change":"2024-09-16T01:36:40.473246+0000","last_active":"2024-09-16T01:36:40.477409+0000","last_peered":"2024-09-16T01:36:40.477409+0000","last_clean":"2024-09-16T01:36:40.477409+0000","last_became_active":"2024-09-16T01:36:40.473013+0000","last_became_peered":"2024-09-16T01:36:40.473013+0000","last_unstale":"2024-09-16T01:36:40.477409+0000","last_undegraded":"2024-09-16T01:36:40.477409+0000","last_fullsized":"2024-09-16T01:36:40.477409+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:15.468906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.470623+0000","last_change":"2024-09-16T01:36:40.467774+0000","last_active":"2024-09-16T01:36:40.470623+0000","last_peered":"2024-09-16T01:36:40.470623+0000","last_clean":"2024-09-16T01:36:40.470623+0000","last_became_active":"2024-09-16T01:36:40.467511+0000","last_became_peered":"2024-09-16T01:36:40.467511+0000","last_unstale":"2024-09-16T01:36:40.470623+0000","last_undegraded":"2024-09-16T01:36:40.470623+0000","last_fullsized":"2024-09-16T01:36:40.470623+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:54:18.427256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.472213+0000","last_change":"2024-09-16T01:36:40.468718+0000","last_active":"2024-09-16T01:36:40.472213+0000","last_peered":"2024-09-16T01:36:40.472213+0000","last_clean":"2024-09-16T01:36:40.472213+0000","last_became_active":"2024-09-16T01:36:40.468428+0000","last_became_peered":"2024-09-16T01:36:40.468428+0000","last_unstale":"2024-09-16T01:36:40.472213+0000","last_undegraded":"2024-09-16T01:36:40.472213+0000","last_fullsized":"2024-09-16T01:36:40.472213+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:06.065750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.466193+0000","last_change":"2024-09-16T01:36:40.461782+0000","last_active":"2024-09-16T01:36:40.466193+0000","last_peered":"2024-09-16T01:36:40.466193+0000","last_clean":"2024-09-16T01:36:40.466193+0000","last_became_active":"2024-09-16T01:36:40.461472+0000","last_became_peered":"2024-09-16T01:36:40.461472+0000","last_unstale":"2024-09-16T01:36:40.466193+0000","last_undegraded":"2024-09-16T01:36:40.466193+0000","last_fullsized":"2024-09-16T01:36:40.466193+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:54.710815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.459407+0000","last_change":"2024-09-16T01:36:40.457421+0000","last_active":"2024-09-16T01:36:40.459407+0000","last_peered":"2024-09-16T01:36:40.459407+0000","last_clean":"2024-09-16T01:36:40.459407+0000","last_became_active":"2024-09-16T01:36:40.457141+0000","last_became_peered":"2024-09-16T01:36:40.457141+0000","last_unstale":"2024-09-16T01:36:40.459407+0000","last_undegraded":"2024-09-16T01:36:40.459407+0000","last_fullsized":"2024-09-16T01:36:40.459407+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:03:44.121298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475567+0000","last_change":"2024-09-16T01:36:40.469272+0000","last_active":"2024-09-16T01:36:40.475567+0000","last_peered":"2024-09-16T01:36:40.475567+0000","last_clean":"2024-09-16T01:36:40.475567+0000","last_became_active":"2024-09-16T01:36:40.469005+0000","last_became_peered":"2024-09-16T01:36:40.469005+0000","last_unstale":"2024-09-16T01:36:40.475567+0000","last_undegraded":"2024-09-16T01:36:40.475567+0000","last_fullsized":"2024-09-16T01:36:40.475567+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:23:27.003444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476834+0000","last_change":"2024-09-16T01:36:40.471483+0000","last_active":"2024-09-16T01:36:40.476834+0000","last_peered":"2024-09-16T01:36:40.476834+0000","last_clean":"2024-09-16T01:36:40.476834+0000","last_became_active":"2024-09-16T01:36:40.471173+0000","last_became_peered":"2024-09-16T01:36:40.471173+0000","last_unstale":"2024-09-16T01:36:40.476834+0000","last_undegraded":"2024-09-16T01:36:40.476834+0000","last_fullsized":"2024-09-16T01:36:40.476834+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:54:15.217078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478361+0000","last_change":"2024-09-16T01:36:40.471975+0000","last_active":"2024-09-16T01:36:40.478361+0000","last_peered":"2024-09-16T01:36:40.478361+0000","last_clean":"2024-09-16T01:36:40.478361+0000","last_became_active":"2024-09-16T01:36:40.471284+0000","last_became_peered":"2024-09-16T01:36:40.471284+0000","last_unstale":"2024-09-16T01:36:40.478361+0000","last_undegraded":"2024-09-16T01:36:40.478361+0000","last_fullsized":"2024-09-16T01:36:40.478361+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:06:01.651652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476619+0000","last_change":"2024-09-16T01:36:40.468666+0000","last_active":"2024-09-16T01:36:40.476619+0000","last_peered":"2024-09-16T01:36:40.476619+0000","last_clean":"2024-09-16T01:36:40.476619+0000","last_became_active":"2024-09-16T01:36:40.468416+0000","last_became_peered":"2024-09-16T01:36:40.468416+0000","last_unstale":"2024-09-16T01:36:40.476619+0000","last_undegraded":"2024-09-16T01:36:40.476619+0000","last_fullsized":"2024-09-16T01:36:40.476619+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:56.859974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476853+0000","last_change":"2024-09-16T01:36:40.471268+0000","last_active":"2024-09-16T01:36:40.476853+0000","last_peered":"2024-09-16T01:36:40.476853+0000","last_clean":"2024-09-16T01:36:40.476853+0000","last_became_active":"2024-09-16T01:36:40.469952+0000","last_became_peered":"2024-09-16T01:36:40.469952+0000","last_unstale":"2024-09-16T01:36:40.476853+0000","last_undegraded":"2024-09-16T01:36:40.476853+0000","last_fullsized":"2024-09-16T01:36:40.476853+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:59:37.550735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475487+0000","last_change":"2024-09-16T01:36:40.465607+0000","last_active":"2024-09-16T01:36:40.475487+0000","last_peered":"2024-09-16T01:36:40.475487+0000","last_clean":"2024-09-16T01:36:40.475487+0000","last_became_active":"2024-09-16T01:36:40.465303+0000","last_became_peered":"2024-09-16T01:36:40.465303+0000","last_unstale":"2024-09-16T01:36:40.475487+0000","last_undegraded":"2024-09-16T01:36:40.475487+0000","last_fullsized":"2024-09-16T01:36:40.475487+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:24:53.760653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.469870+0000","last_change":"2024-09-16T01:36:40.461798+0000","last_active":"2024-09-16T01:36:40.469870+0000","last_peered":"2024-09-16T01:36:40.469870+0000","last_clean":"2024-09-16T01:36:40.469870+0000","last_became_active":"2024-09-16T01:36:40.461500+0000","last_became_peered":"2024-09-16T01:36:40.461500+0000","last_unstale":"2024-09-16T01:36:40.469870+0000","last_undegraded":"2024-09-16T01:36:40.469870+0000","last_fullsized":"2024-09-16T01:36:40.469870+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:57.808314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478526+0000","last_change":"2024-09-16T01:36:40.475000+0000","last_active":"2024-09-16T01:36:40.478526+0000","last_peered":"2024-09-16T01:36:40.478526+0000","last_clean":"2024-09-16T01:36:40.478526+0000","last_became_active":"2024-09-16T01:36:40.474699+0000","last_became_peered":"2024-09-16T01:36:40.474699+0000","last_unstale":"2024-09-16T01:36:40.478526+0000","last_undegraded":"2024-09-16T01:36:40.478526+0000","last_fullsized":"2024-09-16T01:36:40.478526+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:37:37.379239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.472699+0000","last_change":"2024-09-16T01:36:40.468753+0000","last_active":"2024-09-16T01:36:40.472699+0000","last_peered":"2024-09-16T01:36:40.472699+0000","last_clean":"2024-09-16T01:36:40.472699+0000","last_became_active":"2024-09-16T01:36:40.468446+0000","last_became_peered":"2024-09-16T01:36:40.468446+0000","last_unstale":"2024-09-16T01:36:40.472699+0000","last_undegraded":"2024-09-16T01:36:40.472699+0000","last_fullsized":"2024-09-16T01:36:40.472699+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:37:49.094522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.461034+0000","last_change":"2024-09-16T01:36:40.459338+0000","last_active":"2024-09-16T01:36:40.461034+0000","last_peered":"2024-09-16T01:36:40.461034+0000","last_clean":"2024-09-16T01:36:40.461034+0000","last_became_active":"2024-09-16T01:36:40.458954+0000","last_became_peered":"2024-09-16T01:36:40.458954+0000","last_unstale":"2024-09-16T01:36:40.461034+0000","last_undegraded":"2024-09-16T01:36:40.461034+0000","last_fullsized":"2024-09-16T01:36:40.461034+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:37.545285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.470459+0000","last_change":"2024-09-16T01:36:40.464646+0000","last_active":"2024-09-16T01:36:40.470459+0000","last_peered":"2024-09-16T01:36:40.470459+0000","last_clean":"2024-09-16T01:36:40.470459+0000","last_became_active":"2024-09-16T01:36:40.464348+0000","last_became_peered":"2024-09-16T01:36:40.464348+0000","last_unstale":"2024-09-16T01:36:40.470459+0000","last_undegraded":"2024-09-16T01:36:40.470459+0000","last_fullsized":"2024-09-16T01:36:40.470459+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:31:33.950166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475724+0000","last_change":"2024-09-16T01:36:40.465788+0000","last_active":"2024-09-16T01:36:40.475724+0000","last_peered":"2024-09-16T01:36:40.475724+0000","last_clean":"2024-09-16T01:36:40.475724+0000","last_became_active":"2024-09-16T01:36:40.464511+0000","last_became_peered":"2024-09-16T01:36:40.464511+0000","last_unstale":"2024-09-16T01:36:40.475724+0000","last_undegraded":"2024-09-16T01:36:40.475724+0000","last_fullsized":"2024-09-16T01:36:40.475724+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:09:16.216025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.474615+0000","last_change":"2024-09-16T01:36:40.466280+0000","last_active":"2024-09-16T01:36:40.474615+0000","last_peered":"2024-09-16T01:36:40.474615+0000","last_clean":"2024-09-16T01:36:40.474615+0000","last_became_active":"2024-09-16T01:36:40.464657+0000","last_became_peered":"2024-09-16T01:36:40.464657+0000","last_unstale":"2024-09-16T01:36:40.474615+0000","last_undegraded":"2024-09-16T01:36:40.474615+0000","last_fullsized":"2024-09-16T01:36:40.474615+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:56:12.402516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476854+0000","last_change":"2024-09-16T01:36:40.472186+0000","last_active":"2024-09-16T01:36:40.476854+0000","last_peered":"2024-09-16T01:36:40.476854+0000","last_clean":"2024-09-16T01:36:40.476854+0000","last_became_active":"2024-09-16T01:36:40.471918+0000","last_became_peered":"2024-09-16T01:36:40.471918+0000","last_unstale":"2024-09-16T01:36:40.476854+0000","last_undegraded":"2024-09-16T01:36:40.476854+0000","last_fullsized":"2024-09-16T01:36:40.476854+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:32:32.193049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478292+0000","last_change":"2024-09-16T01:36:40.471732+0000","last_active":"2024-09-16T01:36:40.478292+0000","last_peered":"2024-09-16T01:36:40.478292+0000","last_clean":"2024-09-16T01:36:40.478292+0000","last_became_active":"2024-09-16T01:36:40.471441+0000","last_became_peered":"2024-09-16T01:36:40.471441+0000","last_unstale":"2024-09-16T01:36:40.478292+0000","last_undegraded":"2024-09-16T01:36:40.478292+0000","last_fullsized":"2024-09-16T01:36:40.478292+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:58:01.503914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.458508+0000","last_change":"2024-09-16T01:36:40.457225+0000","last_active":"2024-09-16T01:36:40.458508+0000","last_peered":"2024-09-16T01:36:40.458508+0000","last_clean":"2024-09-16T01:36:40.458508+0000","last_became_active":"2024-09-16T01:36:40.456967+0000","last_became_peered":"2024-09-16T01:36:40.456967+0000","last_unstale":"2024-09-16T01:36:40.458508+0000","last_undegraded":"2024-09-16T01:36:40.458508+0000","last_fullsized":"2024-09-16T01:36:40.458508+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:57.227814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.477879+0000","last_change":"2024-09-16T01:36:40.474827+0000","last_active":"2024-09-16T01:36:40.477879+0000","last_peered":"2024-09-16T01:36:40.477879+0000","last_clean":"2024-09-16T01:36:40.477879+0000","last_became_active":"2024-09-16T01:36:40.474479+0000","last_became_peered":"2024-09-16T01:36:40.474479+0000","last_unstale":"2024-09-16T01:36:40.477879+0000","last_undegraded":"2024-09-16T01:36:40.477879+0000","last_fullsized":"2024-09-16T01:36:40.477879+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:47.059215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475710+0000","last_change":"2024-09-16T01:36:40.469807+0000","last_active":"2024-09-16T01:36:40.475710+0000","last_peered":"2024-09-16T01:36:40.475710+0000","last_clean":"2024-09-16T01:36:40.475710+0000","last_became_active":"2024-09-16T01:36:40.469542+0000","last_became_peered":"2024-09-16T01:36:40.469542+0000","last_unstale":"2024-09-16T01:36:40.475710+0000","last_undegraded":"2024-09-16T01:36:40.475710+0000","last_fullsized":"2024-09-16T01:36:40.475710+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:30:00.881153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"57'4","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476066+0000","last_change":"2024-09-16T01:36:39.464332+0000","last_active":"2024-09-16T01:36:40.476066+0000","last_peered":"2024-09-16T01:36:40.476066+0000","last_clean":"2024-09-16T01:36:40.476066+0000","last_became_active":"2024-09-16T01:36:39.464067+0000","last_became_peered":"2024-09-16T01:36:39.464067+0000","last_unstale":"2024-09-16T01:36:40.476066+0000","last_undegraded":"2024-09-16T01:36:40.476066+0000","last_fullsized":"2024-09-16T01:36:40.476066+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:38.563765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476505+0000","last_change":"2024-09-16T01:36:40.471935+0000","last_active":"2024-09-16T01:36:40.476505+0000","last_peered":"2024-09-16T01:36:40.476505+0000","last_clean":"2024-09-16T01:36:40.476505+0000","last_became_active":"2024-09-16T01:36:40.471348+0000","last_became_peered":"2024-09-16T01:36:40.471348+0000","last_unstale":"2024-09-16T01:36:40.476505+0000","last_undegraded":"2024-09-16T01:36:40.476505+0000","last_fullsized":"2024-09-16T01:36:40.476505+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:57.903347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476472+0000","last_change":"2024-09-16T01:36:39.463823+0000","last_active":"2024-09-16T01:36:40.476472+0000","last_peered":"2024-09-16T01:36:40.476472+0000","last_clean":"2024-09-16T01:36:40.476472+0000","last_became_active":"2024-09-16T01:36:39.463549+0000","last_became_peered":"2024-09-16T01:36:39.463549+0000","last_unstale":"2024-09-16T01:36:40.476472+0000","last_undegraded":"2024-09-16T01:36:40.476472+0000","last_fullsized":"2024-09-16T01:36:40.476472+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:29:59.879003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475035+0000","last_change":"2024-09-16T01:36:40.463367+0000","last_active":"2024-09-16T01:36:40.475035+0000","last_peered":"2024-09-16T01:36:40.475035+0000","last_clean":"2024-09-16T01:36:40.475035+0000","last_became_active":"2024-09-16T01:36:40.462287+0000","last_became_peered":"2024-09-16T01:36:40.462287+0000","last_unstale":"2024-09-16T01:36:40.475035+0000","last_undegraded":"2024-09-16T01:36:40.475035+0000","last_fullsized":"2024-09-16T01:36:40.475035+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:15.940785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.477088+0000","last_change":"2024-09-16T01:36:39.461073+0000","last_active":"2024-09-16T01:36:40.477088+0000","last_peered":"2024-09-16T01:36:40.477088+0000","last_clean":"2024-09-16T01:36:40.477088+0000","last_became_active":"2024-09-16T01:36:39.460821+0000","last_became_peered":"2024-09-16T01:36:39.460821+0000","last_unstale":"2024-09-16T01:36:40.477088+0000","last_undegraded":"2024-09-16T01:36:40.477088+0000","last_fullsized":"2024-09-16T01:36:40.477088+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:09:57.224365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475781+0000","last_change":"2024-09-16T01:36:40.469776+0000","last_active":"2024-09-16T01:36:40.475781+0000","last_peered":"2024-09-16T01:36:40.475781+0000","last_clean":"2024-09-16T01:36:40.475781+0000","last_became_active":"2024-09-16T01:36:40.465433+0000","last_became_peered":"2024-09-16T01:36:40.465433+0000","last_unstale":"2024-09-16T01:36:40.475781+0000","last_undegraded":"2024-09-16T01:36:40.475781+0000","last_fullsized":"2024-09-16T01:36:40.475781+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:47:43.290566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476032+0000","last_change":"2024-09-16T01:36:39.460421+0000","last_active":"2024-09-16T01:36:40.476032+0000","last_peered":"2024-09-16T01:36:40.476032+0000","last_clean":"2024-09-16T01:36:40.476032+0000","last_became_active":"2024-09-16T01:36:39.460146+0000","last_became_peered":"2024-09-16T01:36:39.460146+0000","last_unstale":"2024-09-16T01:36:40.476032+0000","last_undegraded":"2024-09-16T01:36:40.476032+0000","last_fullsized":"2024-09-16T01:36:40.476032+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:59:06.387430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.469846+0000","last_change":"2024-09-16T01:36:40.461743+0000","last_active":"2024-09-16T01:36:40.469846+0000","last_peered":"2024-09-16T01:36:40.469846+0000","last_clean":"2024-09-16T01:36:40.469846+0000","last_became_active":"2024-09-16T01:36:40.461455+0000","last_became_peered":"2024-09-16T01:36:40.461455+0000","last_unstale":"2024-09-16T01:36:40.469846+0000","last_undegraded":"2024-09-16T01:36:40.469846+0000","last_fullsized":"2024-09-16T01:36:40.469846+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:23:19.741838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.474803+0000","last_change":"2024-09-16T01:36:39.458301+0000","last_active":"2024-09-16T01:36:40.474803+0000","last_peered":"2024-09-16T01:36:40.474803+0000","last_clean":"2024-09-16T01:36:40.474803+0000","last_became_active":"2024-09-16T01:36:39.457987+0000","last_became_peered":"2024-09-16T01:36:39.457987+0000","last_unstale":"2024-09-16T01:36:40.474803+0000","last_undegraded":"2024-09-16T01:36:40.474803+0000","last_fullsized":"2024-09-16T01:36:40.474803+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:28:16.583211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.479343+0000","last_change":"2024-09-16T01:36:40.476287+0000","last_active":"2024-09-16T01:36:40.479343+0000","last_peered":"2024-09-16T01:36:40.479343+0000","last_clean":"2024-09-16T01:36:40.479343+0000","last_became_active":"2024-09-16T01:36:40.476026+0000","last_became_peered":"2024-09-16T01:36:40.476026+0000","last_unstale":"2024-09-16T01:36:40.479343+0000","last_undegraded":"2024-09-16T01:36:40.479343+0000","last_fullsized":"2024-09-16T01:36:40.479343+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:05:50.979850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475673+0000","last_change":"2024-09-16T01:36:39.460688+0000","last_active":"2024-09-16T01:36:40.475673+0000","last_peered":"2024-09-16T01:36:40.475673+0000","last_clean":"2024-09-16T01:36:40.475673+0000","last_became_active":"2024-09-16T01:36:39.460340+0000","last_became_peered":"2024-09-16T01:36:39.460340+0000","last_unstale":"2024-09-16T01:36:40.475673+0000","last_undegraded":"2024-09-16T01:36:40.475673+0000","last_fullsized":"2024-09-16T01:36:40.475673+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:46:36.645856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.472679+0000","last_change":"2024-09-16T01:36:40.468028+0000","last_active":"2024-09-16T01:36:40.472679+0000","last_peered":"2024-09-16T01:36:40.472679+0000","last_clean":"2024-09-16T01:36:40.472679+0000","last_became_active":"2024-09-16T01:36:40.467726+0000","last_became_peered":"2024-09-16T01:36:40.467726+0000","last_unstale":"2024-09-16T01:36:40.472679+0000","last_undegraded":"2024-09-16T01:36:40.472679+0000","last_fullsized":"2024-09-16T01:36:40.472679+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:20.116406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.461077+0000","last_change":"2024-09-16T01:36:39.461252+0000","last_active":"2024-09-16T01:36:40.461077+0000","last_peered":"2024-09-16T01:36:40.461077+0000","last_clean":"2024-09-16T01:36:40.461077+0000","last_became_active":"2024-09-16T01:36:39.460960+0000","last_became_peered":"2024-09-16T01:36:39.460960+0000","last_unstale":"2024-09-16T01:36:40.461077+0000","last_undegraded":"2024-09-16T01:36:40.461077+0000","last_fullsized":"2024-09-16T01:36:40.461077+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:24:52.758504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476741+0000","last_change":"2024-09-16T01:36:40.469674+0000","last_active":"2024-09-16T01:36:40.476741+0000","last_peered":"2024-09-16T01:36:40.476741+0000","last_clean":"2024-09-16T01:36:40.476741+0000","last_became_active":"2024-09-16T01:36:40.469399+0000","last_became_peered":"2024-09-16T01:36:40.469399+0000","last_unstale":"2024-09-16T01:36:40.476741+0000","last_undegraded":"2024-09-16T01:36:40.476741+0000","last_fullsized":"2024-09-16T01:36:40.476741+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:32:32.869150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.451968+0000","last_change":"2024-09-16T01:36:39.465116+0000","last_active":"2024-09-16T01:36:40.451968+0000","last_peered":"2024-09-16T01:36:40.451968+0000","last_clean":"2024-09-16T01:36:40.451968+0000","last_became_active":"2024-09-16T01:36:39.464777+0000","last_became_peered":"2024-09-16T01:36:39.464777+0000","last_unstale":"2024-09-16T01:36:40.451968+0000","last_undegraded":"2024-09-16T01:36:40.451968+0000","last_fullsized":"2024-09-16T01:36:40.451968+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:50:58.934945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.466221+0000","last_change":"2024-09-16T01:36:40.461894+0000","last_active":"2024-09-16T01:36:40.466221+0000","last_peered":"2024-09-16T01:36:40.466221+0000","last_clean":"2024-09-16T01:36:40.466221+0000","last_became_active":"2024-09-16T01:36:40.461602+0000","last_became_peered":"2024-09-16T01:36:40.461602+0000","last_unstale":"2024-09-16T01:36:40.466221+0000","last_undegraded":"2024-09-16T01:36:40.466221+0000","last_fullsized":"2024-09-16T01:36:40.466221+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:02:20.953972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.462154+0000","last_change":"2024-09-16T01:36:39.463527+0000","last_active":"2024-09-16T01:36:40.462154+0000","last_peered":"2024-09-16T01:36:40.462154+0000","last_clean":"2024-09-16T01:36:40.462154+0000","last_became_active":"2024-09-16T01:36:39.463194+0000","last_became_peered":"2024-09-16T01:36:39.463194+0000","last_unstale":"2024-09-16T01:36:40.462154+0000","last_undegraded":"2024-09-16T01:36:40.462154+0000","last_fullsized":"2024-09-16T01:36:40.462154+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:20:36.477427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478234+0000","last_change":"2024-09-16T01:36:40.471438+0000","last_active":"2024-09-16T01:36:40.478234+0000","last_peered":"2024-09-16T01:36:40.478234+0000","last_clean":"2024-09-16T01:36:40.478234+0000","last_became_active":"2024-09-16T01:36:40.471177+0000","last_became_peered":"2024-09-16T01:36:40.471177+0000","last_unstale":"2024-09-16T01:36:40.478234+0000","last_undegraded":"2024-09-16T01:36:40.478234+0000","last_fullsized":"2024-09-16T01:36:40.478234+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:59:07.389579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.457422+0000","last_change":"2024-09-16T01:36:39.460934+0000","last_active":"2024-09-16T01:36:40.457422+0000","last_peered":"2024-09-16T01:36:40.457422+0000","last_clean":"2024-09-16T01:36:40.457422+0000","last_became_active":"2024-09-16T01:36:39.460651+0000","last_became_peered":"2024-09-16T01:36:39.460651+0000","last_unstale":"2024-09-16T01:36:40.457422+0000","last_undegraded":"2024-09-16T01:36:40.457422+0000","last_fullsized":"2024-09-16T01:36:40.457422+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:04.432074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.466094+0000","last_change":"2024-09-16T01:36:40.463248+0000","last_active":"2024-09-16T01:36:40.466094+0000","last_peered":"2024-09-16T01:36:40.466094+0000","last_clean":"2024-09-16T01:36:40.466094+0000","last_became_active":"2024-09-16T01:36:40.462992+0000","last_became_peered":"2024-09-16T01:36:40.462992+0000","last_unstale":"2024-09-16T01:36:40.466094+0000","last_undegraded":"2024-09-16T01:36:40.466094+0000","last_fullsized":"2024-09-16T01:36:40.466094+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:16:03.996778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"57'3","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478311+0000","last_change":"2024-09-16T01:36:39.459827+0000","last_active":"2024-09-16T01:36:40.478311+0000","last_peered":"2024-09-16T01:36:40.478311+0000","last_clean":"2024-09-16T01:36:40.478311+0000","last_became_active":"2024-09-16T01:36:39.459136+0000","last_became_peered":"2024-09-16T01:36:39.459136+0000","last_unstale":"2024-09-16T01:36:40.478311+0000","last_undegraded":"2024-09-16T01:36:40.478311+0000","last_fullsized":"2024-09-16T01:36:40.478311+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:17:11.701945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478377+0000","last_change":"2024-09-16T01:36:40.473080+0000","last_active":"2024-09-16T01:36:40.478377+0000","last_peered":"2024-09-16T01:36:40.478377+0000","last_clean":"2024-09-16T01:36:40.478377+0000","last_became_active":"2024-09-16T01:36:40.472390+0000","last_became_peered":"2024-09-16T01:36:40.472390+0000","last_unstale":"2024-09-16T01:36:40.478377+0000","last_undegraded":"2024-09-16T01:36:40.478377+0000","last_fullsized":"2024-09-16T01:36:40.478377+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:55:25.810834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"57'6","reported_seq":18,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.480875+0000","last_change":"2024-09-16T01:36:39.463970+0000","last_active":"2024-09-16T01:36:40.480875+0000","last_peered":"2024-09-16T01:36:40.480875+0000","last_clean":"2024-09-16T01:36:40.480875+0000","last_became_active":"2024-09-16T01:36:39.463578+0000","last_became_peered":"2024-09-16T01:36:39.463578+0000","last_unstale":"2024-09-16T01:36:40.480875+0000","last_undegraded":"2024-09-16T01:36:40.480875+0000","last_fullsized":"2024-09-16T01:36:40.480875+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:42:13.498400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.466657+0000","last_change":"2024-09-16T01:36:40.460384+0000","last_active":"2024-09-16T01:36:40.466657+0000","last_peered":"2024-09-16T01:36:40.466657+0000","last_clean":"2024-09-16T01:36:40.466657+0000","last_became_active":"2024-09-16T01:36:40.460100+0000","last_became_peered":"2024-09-16T01:36:40.460100+0000","last_unstale":"2024-09-16T01:36:40.466657+0000","last_undegraded":"2024-09-16T01:36:40.466657+0000","last_fullsized":"2024-09-16T01:36:40.466657+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:20:37.479577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475364+0000","last_change":"2024-09-16T01:36:39.463203+0000","last_active":"2024-09-16T01:36:40.475364+0000","last_peered":"2024-09-16T01:36:40.475364+0000","last_clean":"2024-09-16T01:36:40.475364+0000","last_became_active":"2024-09-16T01:36:39.462964+0000","last_became_peered":"2024-09-16T01:36:39.462964+0000","last_unstale":"2024-09-16T01:36:40.475364+0000","last_undegraded":"2024-09-16T01:36:40.475364+0000","last_fullsized":"2024-09-16T01:36:40.475364+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:25:47.151356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475549+0000","last_change":"2024-09-16T01:36:40.466527+0000","last_active":"2024-09-16T01:36:40.475549+0000","last_peered":"2024-09-16T01:36:40.475549+0000","last_clean":"2024-09-16T01:36:40.475549+0000","last_became_active":"2024-09-16T01:36:40.465922+0000","last_became_peered":"2024-09-16T01:36:40.465922+0000","last_unstale":"2024-09-16T01:36:40.475549+0000","last_undegraded":"2024-09-16T01:36:40.475549+0000","last_fullsized":"2024-09-16T01:36:40.475549+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:20:36.054563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.456156+0000","last_change":"2024-09-16T01:36:39.458436+0000","last_active":"2024-09-16T01:36:40.456156+0000","last_peered":"2024-09-16T01:36:40.456156+0000","last_clean":"2024-09-16T01:36:40.456156+0000","last_became_active":"2024-09-16T01:36:39.458026+0000","last_became_peered":"2024-09-16T01:36:39.458026+0000","last_unstale":"2024-09-16T01:36:40.456156+0000","last_undegraded":"2024-09-16T01:36:40.456156+0000","last_fullsized":"2024-09-16T01:36:40.456156+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:59:30.672192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.473542+0000","last_change":"2024-09-16T01:36:40.471647+0000","last_active":"2024-09-16T01:36:40.473542+0000","last_peered":"2024-09-16T01:36:40.473542+0000","last_clean":"2024-09-16T01:36:40.473542+0000","last_became_active":"2024-09-16T01:36:40.471346+0000","last_became_peered":"2024-09-16T01:36:40.471346+0000","last_unstale":"2024-09-16T01:36:40.473542+0000","last_undegraded":"2024-09-16T01:36:40.473542+0000","last_fullsized":"2024-09-16T01:36:40.473542+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:09:58.226514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.461233+0000","last_change":"2024-09-16T01:36:39.461751+0000","last_active":"2024-09-16T01:36:40.461233+0000","last_peered":"2024-09-16T01:36:40.461233+0000","last_clean":"2024-09-16T01:36:40.461233+0000","last_became_active":"2024-09-16T01:36:39.461513+0000","last_became_peered":"2024-09-16T01:36:39.461513+0000","last_unstale":"2024-09-16T01:36:40.461233+0000","last_undegraded":"2024-09-16T01:36:40.461233+0000","last_fullsized":"2024-09-16T01:36:40.461233+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:20:35.052413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478579+0000","last_change":"2024-09-16T01:36:40.477324+0000","last_active":"2024-09-16T01:36:40.478579+0000","last_peered":"2024-09-16T01:36:40.478579+0000","last_clean":"2024-09-16T01:36:40.478579+0000","last_became_active":"2024-09-16T01:36:40.477120+0000","last_became_peered":"2024-09-16T01:36:40.477120+0000","last_unstale":"2024-09-16T01:36:40.478579+0000","last_undegraded":"2024-09-16T01:36:40.478579+0000","last_fullsized":"2024-09-16T01:36:40.478579+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:41:31.417859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"57'8","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478199+0000","last_change":"2024-09-16T01:36:39.463560+0000","last_active":"2024-09-16T01:36:40.478199+0000","last_peered":"2024-09-16T01:36:40.478199+0000","last_clean":"2024-09-16T01:36:40.478199+0000","last_became_active":"2024-09-16T01:36:39.462861+0000","last_became_peered":"2024-09-16T01:36:39.462861+0000","last_unstale":"2024-09-16T01:36:40.478199+0000","last_undegraded":"2024-09-16T01:36:40.478199+0000","last_fullsized":"2024-09-16T01:36:40.478199+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:20:05.524194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478437+0000","last_change":"2024-09-16T01:36:40.472319+0000","last_active":"2024-09-16T01:36:40.478437+0000","last_peered":"2024-09-16T01:36:40.478437+0000","last_clean":"2024-09-16T01:36:40.478437+0000","last_became_active":"2024-09-16T01:36:40.471860+0000","last_became_peered":"2024-09-16T01:36:40.471860+0000","last_unstale":"2024-09-16T01:36:40.478437+0000","last_undegraded":"2024-09-16T01:36:40.478437+0000","last_fullsized":"2024-09-16T01:36:40.478437+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:21:05.023637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475280+0000","last_change":"2024-09-16T01:36:39.462890+0000","last_active":"2024-09-16T01:36:40.475280+0000","last_peered":"2024-09-16T01:36:40.475280+0000","last_clean":"2024-09-16T01:36:40.475280+0000","last_became_active":"2024-09-16T01:36:39.462530+0000","last_became_peered":"2024-09-16T01:36:39.462530+0000","last_unstale":"2024-09-16T01:36:40.475280+0000","last_undegraded":"2024-09-16T01:36:40.475280+0000","last_fullsized":"2024-09-16T01:36:40.475280+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:52:52.113632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.470495+0000","last_change":"2024-09-16T01:36:40.464108+0000","last_active":"2024-09-16T01:36:40.470495+0000","last_peered":"2024-09-16T01:36:40.470495+0000","last_clean":"2024-09-16T01:36:40.470495+0000","last_became_active":"2024-09-16T01:36:40.463824+0000","last_became_peered":"2024-09-16T01:36:40.463824+0000","last_unstale":"2024-09-16T01:36:40.470495+0000","last_undegraded":"2024-09-16T01:36:40.470495+0000","last_fullsized":"2024-09-16T01:36:40.470495+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:55:25.810834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475356+0000","last_change":"2024-09-16T01:36:39.460547+0000","last_active":"2024-09-16T01:36:40.475356+0000","last_peered":"2024-09-16T01:36:40.475356+0000","last_clean":"2024-09-16T01:36:40.475356+0000","last_became_active":"2024-09-16T01:36:39.460250+0000","last_became_peered":"2024-09-16T01:36:39.460250+0000","last_unstale":"2024-09-16T01:36:40.475356+0000","last_undegraded":"2024-09-16T01:36:40.475356+0000","last_fullsized":"2024-09-16T01:36:40.475356+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:42:36.045032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.477483+0000","last_change":"2024-09-16T01:36:40.473969+0000","last_active":"2024-09-16T01:36:40.477483+0000","last_peered":"2024-09-16T01:36:40.477483+0000","last_clean":"2024-09-16T01:36:40.477483+0000","last_became_active":"2024-09-16T01:36:40.473666+0000","last_became_peered":"2024-09-16T01:36:40.473666+0000","last_unstale":"2024-09-16T01:36:40.477483+0000","last_undegraded":"2024-09-16T01:36:40.477483+0000","last_fullsized":"2024-09-16T01:36:40.477483+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:55:18.946116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475994+0000","last_change":"2024-09-16T01:36:39.463153+0000","last_active":"2024-09-16T01:36:40.475994+0000","last_peered":"2024-09-16T01:36:40.475994+0000","last_clean":"2024-09-16T01:36:40.475994+0000","last_became_active":"2024-09-16T01:36:39.462877+0000","last_became_peered":"2024-09-16T01:36:39.462877+0000","last_unstale":"2024-09-16T01:36:40.475994+0000","last_undegraded":"2024-09-16T01:36:40.475994+0000","last_fullsized":"2024-09-16T01:36:40.475994+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:06:00.649503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478446+0000","last_change":"2024-09-16T01:36:40.473180+0000","last_active":"2024-09-16T01:36:40.478446+0000","last_peered":"2024-09-16T01:36:40.478446+0000","last_clean":"2024-09-16T01:36:40.478446+0000","last_became_active":"2024-09-16T01:36:40.472662+0000","last_became_peered":"2024-09-16T01:36:40.472662+0000","last_unstale":"2024-09-16T01:36:40.478446+0000","last_undegraded":"2024-09-16T01:36:40.478446+0000","last_fullsized":"2024-09-16T01:36:40.478446+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:30:03.124326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.454495+0000","last_change":"2024-09-16T01:36:39.462536+0000","last_active":"2024-09-16T01:36:40.454495+0000","last_peered":"2024-09-16T01:36:40.454495+0000","last_clean":"2024-09-16T01:36:40.454495+0000","last_became_active":"2024-09-16T01:36:39.462284+0000","last_became_peered":"2024-09-16T01:36:39.462284+0000","last_unstale":"2024-09-16T01:36:40.454495+0000","last_undegraded":"2024-09-16T01:36:40.454495+0000","last_fullsized":"2024-09-16T01:36:40.454495+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:26:22.279806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.462168+0000","last_change":"2024-09-16T01:36:40.458778+0000","last_active":"2024-09-16T01:36:40.462168+0000","last_peered":"2024-09-16T01:36:40.462168+0000","last_clean":"2024-09-16T01:36:40.462168+0000","last_became_active":"2024-09-16T01:36:40.458188+0000","last_became_peered":"2024-09-16T01:36:40.458188+0000","last_unstale":"2024-09-16T01:36:40.462168+0000","last_undegraded":"2024-09-16T01:36:40.462168+0000","last_fullsized":"2024-09-16T01:36:40.462168+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:20:06.526343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.473804+0000","last_change":"2024-09-16T01:36:39.462685+0000","last_active":"2024-09-16T01:36:40.473804+0000","last_peered":"2024-09-16T01:36:40.473804+0000","last_clean":"2024-09-16T01:36:40.473804+0000","last_became_active":"2024-09-16T01:36:39.462437+0000","last_became_peered":"2024-09-16T01:36:39.462437+0000","last_unstale":"2024-09-16T01:36:40.473804+0000","last_undegraded":"2024-09-16T01:36:40.473804+0000","last_fullsized":"2024-09-16T01:36:40.473804+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:26.733779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475794+0000","last_change":"2024-09-16T01:36:40.464106+0000","last_active":"2024-09-16T01:36:40.475794+0000","last_peered":"2024-09-16T01:36:40.475794+0000","last_clean":"2024-09-16T01:36:40.475794+0000","last_became_active":"2024-09-16T01:36:40.463825+0000","last_became_peered":"2024-09-16T01:36:40.463825+0000","last_unstale":"2024-09-16T01:36:40.475794+0000","last_undegraded":"2024-09-16T01:36:40.475794+0000","last_fullsized":"2024-09-16T01:36:40.475794+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:47:14.256132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.454549+0000","last_change":"2024-09-16T01:36:39.456580+0000","last_active":"2024-09-16T01:36:40.454549+0000","last_peered":"2024-09-16T01:36:40.454549+0000","last_clean":"2024-09-16T01:36:40.454549+0000","last_became_active":"2024-09-16T01:36:39.456335+0000","last_became_peered":"2024-09-16T01:36:39.456335+0000","last_unstale":"2024-09-16T01:36:40.454549+0000","last_undegraded":"2024-09-16T01:36:40.454549+0000","last_fullsized":"2024-09-16T01:36:40.454549+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:46:35.905236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.477768+0000","last_change":"2024-09-16T01:36:40.472405+0000","last_active":"2024-09-16T01:36:40.477768+0000","last_peered":"2024-09-16T01:36:40.477768+0000","last_clean":"2024-09-16T01:36:40.477768+0000","last_became_active":"2024-09-16T01:36:40.472159+0000","last_became_peered":"2024-09-16T01:36:40.472159+0000","last_unstale":"2024-09-16T01:36:40.477768+0000","last_undegraded":"2024-09-16T01:36:40.477768+0000","last_fullsized":"2024-09-16T01:36:40.477768+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:05.434223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.459880+0000","last_change":"2024-09-16T01:36:39.461853+0000","last_active":"2024-09-16T01:36:40.459880+0000","last_peered":"2024-09-16T01:36:40.459880+0000","last_clean":"2024-09-16T01:36:40.459880+0000","last_became_active":"2024-09-16T01:36:39.461332+0000","last_became_peered":"2024-09-16T01:36:39.461332+0000","last_unstale":"2024-09-16T01:36:40.459880+0000","last_undegraded":"2024-09-16T01:36:40.459880+0000","last_fullsized":"2024-09-16T01:36:40.459880+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:17:11.701945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.470436+0000","last_change":"2024-09-16T01:36:40.463367+0000","last_active":"2024-09-16T01:36:40.470436+0000","last_peered":"2024-09-16T01:36:40.470436+0000","last_clean":"2024-09-16T01:36:40.470436+0000","last_became_active":"2024-09-16T01:36:40.463095+0000","last_became_peered":"2024-09-16T01:36:40.463095+0000","last_unstale":"2024-09-16T01:36:40.470436+0000","last_undegraded":"2024-09-16T01:36:40.470436+0000","last_fullsized":"2024-09-16T01:36:40.470436+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:21:05.023637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.477568+0000","last_change":"2024-09-16T01:36:39.460751+0000","last_active":"2024-09-16T01:36:40.477568+0000","last_peered":"2024-09-16T01:36:40.477568+0000","last_clean":"2024-09-16T01:36:40.477568+0000","last_became_active":"2024-09-16T01:36:39.460478+0000","last_became_peered":"2024-09-16T01:36:39.460478+0000","last_unstale":"2024-09-16T01:36:40.477568+0000","last_undegraded":"2024-09-16T01:36:40.477568+0000","last_fullsized":"2024-09-16T01:36:40.477568+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:25:16.927780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478393+0000","last_change":"2024-09-16T01:36:40.473726+0000","last_active":"2024-09-16T01:36:40.478393+0000","last_peered":"2024-09-16T01:36:40.478393+0000","last_clean":"2024-09-16T01:36:40.478393+0000","last_became_active":"2024-09-16T01:36:40.473241+0000","last_became_peered":"2024-09-16T01:36:40.473241+0000","last_unstale":"2024-09-16T01:36:40.478393+0000","last_undegraded":"2024-09-16T01:36:40.478393+0000","last_fullsized":"2024-09-16T01:36:40.478393+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:37:37.379239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.455105+0000","last_change":"2024-09-16T01:36:39.460521+0000","last_active":"2024-09-16T01:36:40.455105+0000","last_peered":"2024-09-16T01:36:40.455105+0000","last_clean":"2024-09-16T01:36:40.455105+0000","last_became_active":"2024-09-16T01:36:39.460221+0000","last_became_peered":"2024-09-16T01:36:39.460221+0000","last_unstale":"2024-09-16T01:36:40.455105+0000","last_undegraded":"2024-09-16T01:36:40.455105+0000","last_fullsized":"2024-09-16T01:36:40.455105+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:59:36.053696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.466072+0000","last_change":"2024-09-16T01:36:40.462925+0000","last_active":"2024-09-16T01:36:40.466072+0000","last_peered":"2024-09-16T01:36:40.466072+0000","last_clean":"2024-09-16T01:36:40.466072+0000","last_became_active":"2024-09-16T01:36:40.462608+0000","last_became_peered":"2024-09-16T01:36:40.462608+0000","last_unstale":"2024-09-16T01:36:40.466072+0000","last_undegraded":"2024-09-16T01:36:40.466072+0000","last_fullsized":"2024-09-16T01:36:40.466072+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:38.209013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.456363+0000","last_change":"2024-09-16T01:36:39.457230+0000","last_active":"2024-09-16T01:36:40.456363+0000","last_peered":"2024-09-16T01:36:40.456363+0000","last_clean":"2024-09-16T01:36:40.456363+0000","last_became_active":"2024-09-16T01:36:39.456929+0000","last_became_peered":"2024-09-16T01:36:39.456929+0000","last_unstale":"2024-09-16T01:36:40.456363+0000","last_undegraded":"2024-09-16T01:36:40.456363+0000","last_fullsized":"2024-09-16T01:36:40.456363+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:42:21.156826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475702+0000","last_change":"2024-09-16T01:36:40.465659+0000","last_active":"2024-09-16T01:36:40.475702+0000","last_peered":"2024-09-16T01:36:40.475702+0000","last_clean":"2024-09-16T01:36:40.475702+0000","last_became_active":"2024-09-16T01:36:40.465102+0000","last_became_peered":"2024-09-16T01:36:40.465102+0000","last_unstale":"2024-09-16T01:36:40.475702+0000","last_undegraded":"2024-09-16T01:36:40.475702+0000","last_fullsized":"2024-09-16T01:36:40.475702+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:50:35.003419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.461484+0000","last_change":"2024-09-16T01:36:39.461697+0000","last_active":"2024-09-16T01:36:40.461484+0000","last_peered":"2024-09-16T01:36:40.461484+0000","last_clean":"2024-09-16T01:36:40.461484+0000","last_became_active":"2024-09-16T01:36:39.461417+0000","last_became_peered":"2024-09-16T01:36:39.461417+0000","last_unstale":"2024-09-16T01:36:40.461484+0000","last_undegraded":"2024-09-16T01:36:40.461484+0000","last_fullsized":"2024-09-16T01:36:40.461484+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:58:00.501765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476660+0000","last_change":"2024-09-16T01:36:40.470075+0000","last_active":"2024-09-16T01:36:40.476660+0000","last_peered":"2024-09-16T01:36:40.476660+0000","last_clean":"2024-09-16T01:36:40.476660+0000","last_became_active":"2024-09-16T01:36:40.469755+0000","last_became_peered":"2024-09-16T01:36:40.469755+0000","last_unstale":"2024-09-16T01:36:40.476660+0000","last_undegraded":"2024-09-16T01:36:40.476660+0000","last_fullsized":"2024-09-16T01:36:40.476660+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:59.945976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.452422+0000","last_change":"2024-09-16T01:36:39.459118+0000","last_active":"2024-09-16T01:36:40.452422+0000","last_peered":"2024-09-16T01:36:40.452422+0000","last_clean":"2024-09-16T01:36:40.452422+0000","last_became_active":"2024-09-16T01:36:39.458865+0000","last_became_peered":"2024-09-16T01:36:39.458865+0000","last_unstale":"2024-09-16T01:36:40.452422+0000","last_undegraded":"2024-09-16T01:36:40.452422+0000","last_fullsized":"2024-09-16T01:36:40.452422+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:12.692476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478471+0000","last_change":"2024-09-16T01:36:40.474300+0000","last_active":"2024-09-16T01:36:40.478471+0000","last_peered":"2024-09-16T01:36:40.478471+0000","last_clean":"2024-09-16T01:36:40.478471+0000","last_became_active":"2024-09-16T01:36:40.474010+0000","last_became_peered":"2024-09-16T01:36:40.474010+0000","last_unstale":"2024-09-16T01:36:40.478471+0000","last_undegraded":"2024-09-16T01:36:40.478471+0000","last_fullsized":"2024-09-16T01:36:40.478471+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:54:18.427256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.452204+0000","last_change":"2024-09-16T01:36:39.463181+0000","last_active":"2024-09-16T01:36:40.452204+0000","last_peered":"2024-09-16T01:36:40.452204+0000","last_clean":"2024-09-16T01:36:40.452204+0000","last_became_active":"2024-09-16T01:36:39.462934+0000","last_became_peered":"2024-09-16T01:36:39.462934+0000","last_unstale":"2024-09-16T01:36:40.452204+0000","last_undegraded":"2024-09-16T01:36:40.452204+0000","last_fullsized":"2024-09-16T01:36:40.452204+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:05:48.877054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.466872+0000","last_change":"2024-09-16T01:36:40.463959+0000","last_active":"2024-09-16T01:36:40.466872+0000","last_peered":"2024-09-16T01:36:40.466872+0000","last_clean":"2024-09-16T01:36:40.466872+0000","last_became_active":"2024-09-16T01:36:40.463692+0000","last_became_peered":"2024-09-16T01:36:40.463692+0000","last_unstale":"2024-09-16T01:36:40.466872+0000","last_undegraded":"2024-09-16T01:36:40.466872+0000","last_fullsized":"2024-09-16T01:36:40.466872+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:09:11.772325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.454666+0000","last_change":"2024-09-16T01:36:39.464029+0000","last_active":"2024-09-16T01:36:40.454666+0000","last_peered":"2024-09-16T01:36:40.454666+0000","last_clean":"2024-09-16T01:36:40.454666+0000","last_became_active":"2024-09-16T01:36:39.463751+0000","last_became_peered":"2024-09-16T01:36:39.463751+0000","last_unstale":"2024-09-16T01:36:40.454666+0000","last_undegraded":"2024-09-16T01:36:40.454666+0000","last_fullsized":"2024-09-16T01:36:40.454666+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:08:17.467084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476762+0000","last_change":"2024-09-16T01:36:40.469615+0000","last_active":"2024-09-16T01:36:40.476762+0000","last_peered":"2024-09-16T01:36:40.476762+0000","last_clean":"2024-09-16T01:36:40.476762+0000","last_became_active":"2024-09-16T01:36:40.469371+0000","last_became_peered":"2024-09-16T01:36:40.469371+0000","last_unstale":"2024-09-16T01:36:40.476762+0000","last_undegraded":"2024-09-16T01:36:40.476762+0000","last_fullsized":"2024-09-16T01:36:40.476762+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:37:40.239240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.457964+0000","last_change":"2024-09-16T01:36:39.464984+0000","last_active":"2024-09-16T01:36:40.457964+0000","last_peered":"2024-09-16T01:36:40.457964+0000","last_clean":"2024-09-16T01:36:40.457964+0000","last_became_active":"2024-09-16T01:36:39.464726+0000","last_became_peered":"2024-09-16T01:36:39.464726+0000","last_unstale":"2024-09-16T01:36:40.457964+0000","last_undegraded":"2024-09-16T01:36:40.457964+0000","last_fullsized":"2024-09-16T01:36:40.457964+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:46.057066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476882+0000","last_change":"2024-09-16T01:36:40.471212+0000","last_active":"2024-09-16T01:36:40.476882+0000","last_peered":"2024-09-16T01:36:40.476882+0000","last_clean":"2024-09-16T01:36:40.476882+0000","last_became_active":"2024-09-16T01:36:40.470000+0000","last_became_peered":"2024-09-16T01:36:40.470000+0000","last_unstale":"2024-09-16T01:36:40.476882+0000","last_undegraded":"2024-09-16T01:36:40.476882+0000","last_fullsized":"2024-09-16T01:36:40.476882+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:06:50.678750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.455934+0000","last_change":"2024-09-16T01:36:39.459267+0000","last_active":"2024-09-16T01:36:40.455934+0000","last_peered":"2024-09-16T01:36:40.455934+0000","last_clean":"2024-09-16T01:36:40.455934+0000","last_became_active":"2024-09-16T01:36:39.458993+0000","last_became_peered":"2024-09-16T01:36:39.458993+0000","last_unstale":"2024-09-16T01:36:40.455934+0000","last_undegraded":"2024-09-16T01:36:40.455934+0000","last_fullsized":"2024-09-16T01:36:40.455934+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:26:42.206194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":235,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.458337+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:36:40.458337+0000","last_peered":"2024-09-16T01:36:40.458337+0000","last_clean":"2024-09-16T01:36:40.458337+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:36:40.458337+0000","last_undegraded":"2024-09-16T01:36:40.458337+0000","last_fullsized":"2024-09-16T01:36:40.458337+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509475,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":688128,"data_stored":281799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32424,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339416,"statfs":{"total":96636764160,"available":96603561984,"internally_reserved":0,"allocated":1261568,"data_stored":864740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542178,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":667648,"data_stored":274372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542178,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":667648,"data_stored":274372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542177,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":667648,"data_stored":274372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542178,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":667648,"data_stored":274372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":667648,"data_stored":274372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542177,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28268,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343572,"statfs":{"total":96636764160,"available":96607817728,"internally_reserved":0,"allocated":1261568,"data_stored":864740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:36:43.398 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:36:43.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.532+0000 7f679c4f1640 1 Processor -- start 2024-09-16T01:36:43.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.532+0000 7f679c4f1640 1 -- start start 2024-09-16T01:36:43.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.533+0000 7f679c4f1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f67940cfdb0 0x7f67940cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:43.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.533+0000 7f679c4f1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f67940d2fd0 0x7f67940d3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:43.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.533+0000 7f679c4f1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6794163330 con 0x7f679415ec10 2024-09-16T01:36:43.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.533+0000 7f679c4f1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f67940d22a0 con 0x7f67940cfdb0 2024-09-16T01:36:43.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.533+0000 7f679c4f1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f67940d24f0 con 0x7f67940d2fd0 2024-09-16T01:36:43.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.533+0000 7f6799a8c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f67940cfdb0 0x7f67940cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:43.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.533+0000 7f6799a8c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f67940cfdb0 0x7f67940cefc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41348/0 (socket says 172.21.15.88:41348) 2024-09-16T01:36:43.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.533+0000 7f6799a8c640 1 -- 172.21.15.88:0/3611102235 learned_addr learned my addr 172.21.15.88:0/3611102235 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:43.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.533+0000 7f679a28d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f679415ec10 0x7f67940cf6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54320/0 (socket says 172.21.15.88:54320) 2024-09-16T01:36:43.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.534+0000 7f679aa8e640 1 --2- 172.21.15.88:0/3611102235 >> v2:172.21.15.154:3300/0 conn(0x7f67940d2fd0 0x7f67940d3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:43.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.534+0000 7f6799a8c640 1 -- 172.21.15.88:0/3611102235 >> v2:172.21.15.154:3300/0 conn(0x7f67940d2fd0 msgr2=0x7f67940d3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:43.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.534+0000 7f6799a8c640 1 --2- 172.21.15.88:0/3611102235 >> v2:172.21.15.154:3300/0 conn(0x7f67940d2fd0 0x7f67940d3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.534+0000 7f6799a8c640 1 -- 172.21.15.88:0/3611102235 >> v1:172.21.15.88:6789/0 conn(0x7f679415ec10 legacy=0x7f67940cf6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:43.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.534+0000 7f679928b640 1 -- 172.21.15.88:0/3611102235 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4224241817 0 0) 0x7f6794163330 con 0x7f679415ec10 2024-09-16T01:36:43.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.534+0000 7f6799a8c640 1 -- 172.21.15.88:0/3611102235 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67940d27d0 con 0x7f67940cfdb0 2024-09-16T01:36:43.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.534+0000 7f6799a8c640 1 --2- 172.21.15.88:0/3611102235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67940cfdb0 0x7f67940cefc0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f67940754e0 tx=0x7f678402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=105b73530673216c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:43.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.535+0000 7f679928b640 1 -- 172.21.15.88:0/3611102235 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6784037070 con 0x7f67940cfdb0 2024-09-16T01:36:43.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.535+0000 7f679928b640 1 -- 172.21.15.88:0/3611102235 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6784032440 con 0x7f67940cfdb0 2024-09-16T01:36:43.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.535+0000 7f679928b640 1 -- 172.21.15.88:0/3611102235 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6784032810 con 0x7f67940cfdb0 2024-09-16T01:36:43.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.535+0000 7f679c4f1640 1 -- 172.21.15.88:0/3611102235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67940cfdb0 msgr2=0x7f67940cefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:43.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.535+0000 7f679c4f1640 1 --2- 172.21.15.88:0/3611102235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67940cfdb0 0x7f67940cefc0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f67940754e0 tx=0x7f678402a1e0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.535+0000 7f679c4f1640 1 -- 172.21.15.88:0/3611102235 shutdown_connections 2024-09-16T01:36:43.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.535+0000 7f679c4f1640 1 --2- 172.21.15.88:0/3611102235 >> v2:172.21.15.154:3300/0 conn(0x7f67940d2fd0 0x7f67940d3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.535+0000 7f679c4f1640 1 --2- 172.21.15.88:0/3611102235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67940cfdb0 0x7f67940cefc0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.535+0000 7f679c4f1640 1 -- 172.21.15.88:0/3611102235 >> 172.21.15.88:0/3611102235 conn(0x7f67940d46d0 msgr2=0x7f67940d4ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:43.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.535+0000 7f679c4f1640 1 -- 172.21.15.88:0/3611102235 shutdown_connections 2024-09-16T01:36:43.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.535+0000 7f679c4f1640 1 -- 172.21.15.88:0/3611102235 wait complete. 2024-09-16T01:36:43.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.536+0000 7f679c4f1640 1 Processor -- start 2024-09-16T01:36:43.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.536+0000 7f679c4f1640 1 -- start start 2024-09-16T01:36:43.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.537+0000 7f679c4f1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67940cfdb0 0x7f67940ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:43.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.537+0000 7f679c4f1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67940d2fd0 0x7f6794156eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:43.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.537+0000 7f679c4f1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f679415ec10 0x7f679415f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:43.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.537+0000 7f679c4f1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f67940b3700 con 0x7f679415ec10 2024-09-16T01:36:43.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.537+0000 7f679c4f1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6794162fc0 con 0x7f67940d2fd0 2024-09-16T01:36:43.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.537+0000 7f679c4f1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f67940b3880 con 0x7f67940cfdb0 2024-09-16T01:36:43.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.537+0000 7f679aa8e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f679415ec10 0x7f679415f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:43.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.537+0000 7f679aa8e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f679415ec10 0x7f679415f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41358/0 (socket says 172.21.15.88:41358) 2024-09-16T01:36:43.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.537+0000 7f679aa8e640 1 -- 172.21.15.88:0/1431715415 learned_addr learned my addr 172.21.15.88:0/1431715415 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:43.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.537+0000 7f6799a8c640 1 --2- 172.21.15.88:0/1431715415 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67940d2fd0 0x7f6794156eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:43.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.537+0000 7f679a28d640 1 --2- 172.21.15.88:0/1431715415 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67940cfdb0 0x7f67940ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:43.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.537+0000 7f679aa8e640 1 -- 172.21.15.88:0/1431715415 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67940cfdb0 msgr2=0x7f67940ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:43.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.537+0000 7f679aa8e640 1 --2- 172.21.15.88:0/1431715415 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67940cfdb0 0x7f67940ce3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.537+0000 7f679aa8e640 1 -- 172.21.15.88:0/1431715415 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67940d2fd0 msgr2=0x7f6794156eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:43.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.537+0000 7f679aa8e640 1 --2- 172.21.15.88:0/1431715415 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67940d2fd0 0x7f6794156eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.537+0000 7f679aa8e640 1 -- 172.21.15.88:0/1431715415 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67940cbca0 con 0x7f679415ec10 2024-09-16T01:36:43.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.538+0000 7f679aa8e640 1 --2- 172.21.15.88:0/1431715415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f679415ec10 0x7f679415f2a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f678c007e60 tx=0x7f678c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:43.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.538+0000 7f678affd640 1 -- 172.21.15.88:0/1431715415 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f678c014420 con 0x7f679415ec10 2024-09-16T01:36:43.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.538+0000 7f678affd640 1 -- 172.21.15.88:0/1431715415 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f678c019070 con 0x7f679415ec10 2024-09-16T01:36:43.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.538+0000 7f678affd640 1 -- 172.21.15.88:0/1431715415 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f678c014820 con 0x7f679415ec10 2024-09-16T01:36:43.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.538+0000 7f679c4f1640 1 -- 172.21.15.88:0/1431715415 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67940cbf80 con 0x7f679415ec10 2024-09-16T01:36:43.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.538+0000 7f679c4f1640 1 -- 172.21.15.88:0/1431715415 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67940cc540 con 0x7f679415ec10 2024-09-16T01:36:43.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.539+0000 7f679c4f1640 1 -- 172.21.15.88:0/1431715415 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67940cefc0 con 0x7f679415ec10 2024-09-16T01:36:43.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.539+0000 7f678affd640 1 -- 172.21.15.88:0/1431715415 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f678c0149c0 con 0x7f679415ec10 2024-09-16T01:36:43.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.540+0000 7f678affd640 1 --2- 172.21.15.88:0/1431715415 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f677806c2b0 0x7f677806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:43.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.540+0000 7f678affd640 1 -- 172.21.15.88:0/1431715415 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f678c08bab0 con 0x7f679415ec10 2024-09-16T01:36:43.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.540+0000 7f679a28d640 1 --2- 172.21.15.88:0/1431715415 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f677806c2b0 0x7f677806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:43.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.540+0000 7f679a28d640 1 --2- 172.21.15.88:0/1431715415 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f677806c2b0 0x7f677806e770 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6790000f60 tx=0x7f6790007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:43.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.544+0000 7f678affd640 1 -- 172.21.15.88:0/1431715415 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f678c059c60 con 0x7f679415ec10 2024-09-16T01:36:43.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.707+0000 7f679c4f1640 1 -- 172.21.15.88:0/1431715415 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f67941573f0 con 0x7f677806c2b0 2024-09-16T01:36:43.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.712+0000 7f678affd640 1 -- 172.21.15.88:0/1431715415 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251883 (secure 0 0 0) 0x7f6790004c70 con 0x7f677806c2b0 2024-09-16T01:36:43.716 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:43.717 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:36:43.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.715+0000 7f679c4f1640 1 -- 172.21.15.88:0/1431715415 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f677806c2b0 msgr2=0x7f677806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:43.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.715+0000 7f679c4f1640 1 --2- 172.21.15.88:0/1431715415 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f677806c2b0 0x7f677806e770 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6790000f60 tx=0x7f6790007d50 comp rx=0 tx=0).stop 2024-09-16T01:36:43.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.715+0000 7f679c4f1640 1 -- 172.21.15.88:0/1431715415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f679415ec10 msgr2=0x7f679415f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:43.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.715+0000 7f679c4f1640 1 --2- 172.21.15.88:0/1431715415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f679415ec10 0x7f679415f2a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f678c007e60 tx=0x7f678c00e4b0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.716+0000 7f679c4f1640 1 -- 172.21.15.88:0/1431715415 shutdown_connections 2024-09-16T01:36:43.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.716+0000 7f679c4f1640 1 --2- 172.21.15.88:0/1431715415 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f677806c2b0 0x7f677806e770 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.716+0000 7f679c4f1640 1 --2- 172.21.15.88:0/1431715415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f679415ec10 0x7f679415f2a0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.716+0000 7f679c4f1640 1 --2- 172.21.15.88:0/1431715415 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67940d2fd0 0x7f6794156eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.716+0000 7f679c4f1640 1 --2- 172.21.15.88:0/1431715415 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67940cfdb0 0x7f67940ce3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.716+0000 7f679c4f1640 1 -- 172.21.15.88:0/1431715415 >> 172.21.15.88:0/1431715415 conn(0x7f67940d46d0 msgr2=0x7f67940cf1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:43.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.716+0000 7f679c4f1640 1 -- 172.21.15.88:0/1431715415 shutdown_connections 2024-09-16T01:36:43.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.716+0000 7f679c4f1640 1 -- 172.21.15.88:0/1431715415 wait complete. 2024-09-16T01:36:43.728 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-09-16T01:36:42.545047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":72,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":226796,"kb_used_data":6396,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754747924,"statfs":{"total":773094113280,"available":772861874176,"internally_reserved":0,"allocated":6549504,"data_stored":3383139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":225496626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.105244"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.474906+0000","last_change":"2024-09-16T01:36:40.473004+0000","last_active":"2024-09-16T01:36:40.474906+0000","last_peered":"2024-09-16T01:36:40.474906+0000","last_clean":"2024-09-16T01:36:40.474906+0000","last_became_active":"2024-09-16T01:36:40.472704+0000","last_became_peered":"2024-09-16T01:36:40.472704+0000","last_unstale":"2024-09-16T01:36:40.474906+0000","last_undegraded":"2024-09-16T01:36:40.474906+0000","last_fullsized":"2024-09-16T01:36:40.474906+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:29:50.303208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.472667+0000","last_change":"2024-09-16T01:36:40.466966+0000","last_active":"2024-09-16T01:36:40.472667+0000","last_peered":"2024-09-16T01:36:40.472667+0000","last_clean":"2024-09-16T01:36:40.472667+0000","last_became_active":"2024-09-16T01:36:40.466714+0000","last_became_peered":"2024-09-16T01:36:40.466714+0000","last_unstale":"2024-09-16T01:36:40.472667+0000","last_undegraded":"2024-09-16T01:36:40.472667+0000","last_fullsized":"2024-09-16T01:36:40.472667+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:18:21.800309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478466+0000","last_change":"2024-09-16T01:36:40.473300+0000","last_active":"2024-09-16T01:36:40.478466+0000","last_peered":"2024-09-16T01:36:40.478466+0000","last_clean":"2024-09-16T01:36:40.478466+0000","last_became_active":"2024-09-16T01:36:40.472936+0000","last_became_peered":"2024-09-16T01:36:40.472936+0000","last_unstale":"2024-09-16T01:36:40.478466+0000","last_undegraded":"2024-09-16T01:36:40.478466+0000","last_fullsized":"2024-09-16T01:36:40.478466+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:31:33.950166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.470583+0000","last_change":"2024-09-16T01:36:40.464619+0000","last_active":"2024-09-16T01:36:40.470583+0000","last_peered":"2024-09-16T01:36:40.470583+0000","last_clean":"2024-09-16T01:36:40.470583+0000","last_became_active":"2024-09-16T01:36:40.464296+0000","last_became_peered":"2024-09-16T01:36:40.464296+0000","last_unstale":"2024-09-16T01:36:40.470583+0000","last_undegraded":"2024-09-16T01:36:40.470583+0000","last_fullsized":"2024-09-16T01:36:40.470583+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:30:03.124326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.471650+0000","last_change":"2024-09-16T01:36:40.465279+0000","last_active":"2024-09-16T01:36:40.471650+0000","last_peered":"2024-09-16T01:36:40.471650+0000","last_clean":"2024-09-16T01:36:40.471650+0000","last_became_active":"2024-09-16T01:36:40.465013+0000","last_became_peered":"2024-09-16T01:36:40.465013+0000","last_unstale":"2024-09-16T01:36:40.471650+0000","last_undegraded":"2024-09-16T01:36:40.471650+0000","last_fullsized":"2024-09-16T01:36:40.471650+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:29.596455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475195+0000","last_change":"2024-09-16T01:36:40.463614+0000","last_active":"2024-09-16T01:36:40.475195+0000","last_peered":"2024-09-16T01:36:40.475195+0000","last_clean":"2024-09-16T01:36:40.475195+0000","last_became_active":"2024-09-16T01:36:40.462404+0000","last_became_peered":"2024-09-16T01:36:40.462404+0000","last_unstale":"2024-09-16T01:36:40.475195+0000","last_undegraded":"2024-09-16T01:36:40.475195+0000","last_fullsized":"2024-09-16T01:36:40.475195+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:32.557899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.469842+0000","last_change":"2024-09-16T01:36:40.464510+0000","last_active":"2024-09-16T01:36:40.469842+0000","last_peered":"2024-09-16T01:36:40.469842+0000","last_clean":"2024-09-16T01:36:40.469842+0000","last_became_active":"2024-09-16T01:36:40.464235+0000","last_became_peered":"2024-09-16T01:36:40.464235+0000","last_unstale":"2024-09-16T01:36:40.469842+0000","last_undegraded":"2024-09-16T01:36:40.469842+0000","last_fullsized":"2024-09-16T01:36:40.469842+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:33:26.780863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476901+0000","last_change":"2024-09-16T01:36:40.473294+0000","last_active":"2024-09-16T01:36:40.476901+0000","last_peered":"2024-09-16T01:36:40.476901+0000","last_clean":"2024-09-16T01:36:40.476901+0000","last_became_active":"2024-09-16T01:36:40.472894+0000","last_became_peered":"2024-09-16T01:36:40.472894+0000","last_unstale":"2024-09-16T01:36:40.476901+0000","last_undegraded":"2024-09-16T01:36:40.476901+0000","last_fullsized":"2024-09-16T01:36:40.476901+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:56:24.891884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.472198+0000","last_change":"2024-09-16T01:36:40.468454+0000","last_active":"2024-09-16T01:36:40.472198+0000","last_peered":"2024-09-16T01:36:40.472198+0000","last_clean":"2024-09-16T01:36:40.472198+0000","last_became_active":"2024-09-16T01:36:40.468194+0000","last_became_peered":"2024-09-16T01:36:40.468194+0000","last_unstale":"2024-09-16T01:36:40.472198+0000","last_undegraded":"2024-09-16T01:36:40.472198+0000","last_fullsized":"2024-09-16T01:36:40.472198+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:43.168711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.465959+0000","last_change":"2024-09-16T01:36:40.462236+0000","last_active":"2024-09-16T01:36:40.465959+0000","last_peered":"2024-09-16T01:36:40.465959+0000","last_clean":"2024-09-16T01:36:40.465959+0000","last_became_active":"2024-09-16T01:36:40.461959+0000","last_became_peered":"2024-09-16T01:36:40.461959+0000","last_unstale":"2024-09-16T01:36:40.465959+0000","last_undegraded":"2024-09-16T01:36:40.465959+0000","last_fullsized":"2024-09-16T01:36:40.465959+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:41:57.274746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.477409+0000","last_change":"2024-09-16T01:36:40.473246+0000","last_active":"2024-09-16T01:36:40.477409+0000","last_peered":"2024-09-16T01:36:40.477409+0000","last_clean":"2024-09-16T01:36:40.477409+0000","last_became_active":"2024-09-16T01:36:40.473013+0000","last_became_peered":"2024-09-16T01:36:40.473013+0000","last_unstale":"2024-09-16T01:36:40.477409+0000","last_undegraded":"2024-09-16T01:36:40.477409+0000","last_fullsized":"2024-09-16T01:36:40.477409+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:06:15.468906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.470623+0000","last_change":"2024-09-16T01:36:40.467774+0000","last_active":"2024-09-16T01:36:40.470623+0000","last_peered":"2024-09-16T01:36:40.470623+0000","last_clean":"2024-09-16T01:36:40.470623+0000","last_became_active":"2024-09-16T01:36:40.467511+0000","last_became_peered":"2024-09-16T01:36:40.467511+0000","last_unstale":"2024-09-16T01:36:40.470623+0000","last_undegraded":"2024-09-16T01:36:40.470623+0000","last_fullsized":"2024-09-16T01:36:40.470623+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:54:18.427256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.472213+0000","last_change":"2024-09-16T01:36:40.468718+0000","last_active":"2024-09-16T01:36:40.472213+0000","last_peered":"2024-09-16T01:36:40.472213+0000","last_clean":"2024-09-16T01:36:40.472213+0000","last_became_active":"2024-09-16T01:36:40.468428+0000","last_became_peered":"2024-09-16T01:36:40.468428+0000","last_unstale":"2024-09-16T01:36:40.472213+0000","last_undegraded":"2024-09-16T01:36:40.472213+0000","last_fullsized":"2024-09-16T01:36:40.472213+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:07:06.065750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.466193+0000","last_change":"2024-09-16T01:36:40.461782+0000","last_active":"2024-09-16T01:36:40.466193+0000","last_peered":"2024-09-16T01:36:40.466193+0000","last_clean":"2024-09-16T01:36:40.466193+0000","last_became_active":"2024-09-16T01:36:40.461472+0000","last_became_peered":"2024-09-16T01:36:40.461472+0000","last_unstale":"2024-09-16T01:36:40.466193+0000","last_undegraded":"2024-09-16T01:36:40.466193+0000","last_fullsized":"2024-09-16T01:36:40.466193+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:20:54.710815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.459407+0000","last_change":"2024-09-16T01:36:40.457421+0000","last_active":"2024-09-16T01:36:40.459407+0000","last_peered":"2024-09-16T01:36:40.459407+0000","last_clean":"2024-09-16T01:36:40.459407+0000","last_became_active":"2024-09-16T01:36:40.457141+0000","last_became_peered":"2024-09-16T01:36:40.457141+0000","last_unstale":"2024-09-16T01:36:40.459407+0000","last_undegraded":"2024-09-16T01:36:40.459407+0000","last_fullsized":"2024-09-16T01:36:40.459407+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:03:44.121298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475567+0000","last_change":"2024-09-16T01:36:40.469272+0000","last_active":"2024-09-16T01:36:40.475567+0000","last_peered":"2024-09-16T01:36:40.475567+0000","last_clean":"2024-09-16T01:36:40.475567+0000","last_became_active":"2024-09-16T01:36:40.469005+0000","last_became_peered":"2024-09-16T01:36:40.469005+0000","last_unstale":"2024-09-16T01:36:40.475567+0000","last_undegraded":"2024-09-16T01:36:40.475567+0000","last_fullsized":"2024-09-16T01:36:40.475567+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:23:27.003444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476834+0000","last_change":"2024-09-16T01:36:40.471483+0000","last_active":"2024-09-16T01:36:40.476834+0000","last_peered":"2024-09-16T01:36:40.476834+0000","last_clean":"2024-09-16T01:36:40.476834+0000","last_became_active":"2024-09-16T01:36:40.471173+0000","last_became_peered":"2024-09-16T01:36:40.471173+0000","last_unstale":"2024-09-16T01:36:40.476834+0000","last_undegraded":"2024-09-16T01:36:40.476834+0000","last_fullsized":"2024-09-16T01:36:40.476834+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:54:15.217078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478361+0000","last_change":"2024-09-16T01:36:40.471975+0000","last_active":"2024-09-16T01:36:40.478361+0000","last_peered":"2024-09-16T01:36:40.478361+0000","last_clean":"2024-09-16T01:36:40.478361+0000","last_became_active":"2024-09-16T01:36:40.471284+0000","last_became_peered":"2024-09-16T01:36:40.471284+0000","last_unstale":"2024-09-16T01:36:40.478361+0000","last_undegraded":"2024-09-16T01:36:40.478361+0000","last_fullsized":"2024-09-16T01:36:40.478361+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:06:01.651652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476619+0000","last_change":"2024-09-16T01:36:40.468666+0000","last_active":"2024-09-16T01:36:40.476619+0000","last_peered":"2024-09-16T01:36:40.476619+0000","last_clean":"2024-09-16T01:36:40.476619+0000","last_became_active":"2024-09-16T01:36:40.468416+0000","last_became_peered":"2024-09-16T01:36:40.468416+0000","last_unstale":"2024-09-16T01:36:40.476619+0000","last_undegraded":"2024-09-16T01:36:40.476619+0000","last_fullsized":"2024-09-16T01:36:40.476619+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:56.859974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476853+0000","last_change":"2024-09-16T01:36:40.471268+0000","last_active":"2024-09-16T01:36:40.476853+0000","last_peered":"2024-09-16T01:36:40.476853+0000","last_clean":"2024-09-16T01:36:40.476853+0000","last_became_active":"2024-09-16T01:36:40.469952+0000","last_became_peered":"2024-09-16T01:36:40.469952+0000","last_unstale":"2024-09-16T01:36:40.476853+0000","last_undegraded":"2024-09-16T01:36:40.476853+0000","last_fullsized":"2024-09-16T01:36:40.476853+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:59:37.550735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475487+0000","last_change":"2024-09-16T01:36:40.465607+0000","last_active":"2024-09-16T01:36:40.475487+0000","last_peered":"2024-09-16T01:36:40.475487+0000","last_clean":"2024-09-16T01:36:40.475487+0000","last_became_active":"2024-09-16T01:36:40.465303+0000","last_became_peered":"2024-09-16T01:36:40.465303+0000","last_unstale":"2024-09-16T01:36:40.475487+0000","last_undegraded":"2024-09-16T01:36:40.475487+0000","last_fullsized":"2024-09-16T01:36:40.475487+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:24:53.760653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.469870+0000","last_change":"2024-09-16T01:36:40.461798+0000","last_active":"2024-09-16T01:36:40.469870+0000","last_peered":"2024-09-16T01:36:40.469870+0000","last_clean":"2024-09-16T01:36:40.469870+0000","last_became_active":"2024-09-16T01:36:40.461500+0000","last_became_peered":"2024-09-16T01:36:40.461500+0000","last_unstale":"2024-09-16T01:36:40.469870+0000","last_undegraded":"2024-09-16T01:36:40.469870+0000","last_fullsized":"2024-09-16T01:36:40.469870+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:26:57.808314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478526+0000","last_change":"2024-09-16T01:36:40.475000+0000","last_active":"2024-09-16T01:36:40.478526+0000","last_peered":"2024-09-16T01:36:40.478526+0000","last_clean":"2024-09-16T01:36:40.478526+0000","last_became_active":"2024-09-16T01:36:40.474699+0000","last_became_peered":"2024-09-16T01:36:40.474699+0000","last_unstale":"2024-09-16T01:36:40.478526+0000","last_undegraded":"2024-09-16T01:36:40.478526+0000","last_fullsized":"2024-09-16T01:36:40.478526+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:37:37.379239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.472699+0000","last_change":"2024-09-16T01:36:40.468753+0000","last_active":"2024-09-16T01:36:40.472699+0000","last_peered":"2024-09-16T01:36:40.472699+0000","last_clean":"2024-09-16T01:36:40.472699+0000","last_became_active":"2024-09-16T01:36:40.468446+0000","last_became_peered":"2024-09-16T01:36:40.468446+0000","last_unstale":"2024-09-16T01:36:40.472699+0000","last_undegraded":"2024-09-16T01:36:40.472699+0000","last_fullsized":"2024-09-16T01:36:40.472699+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:37:49.094522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.461034+0000","last_change":"2024-09-16T01:36:40.459338+0000","last_active":"2024-09-16T01:36:40.461034+0000","last_peered":"2024-09-16T01:36:40.461034+0000","last_clean":"2024-09-16T01:36:40.461034+0000","last_became_active":"2024-09-16T01:36:40.458954+0000","last_became_peered":"2024-09-16T01:36:40.458954+0000","last_unstale":"2024-09-16T01:36:40.461034+0000","last_undegraded":"2024-09-16T01:36:40.461034+0000","last_fullsized":"2024-09-16T01:36:40.461034+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:37.545285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.470459+0000","last_change":"2024-09-16T01:36:40.464646+0000","last_active":"2024-09-16T01:36:40.470459+0000","last_peered":"2024-09-16T01:36:40.470459+0000","last_clean":"2024-09-16T01:36:40.470459+0000","last_became_active":"2024-09-16T01:36:40.464348+0000","last_became_peered":"2024-09-16T01:36:40.464348+0000","last_unstale":"2024-09-16T01:36:40.470459+0000","last_undegraded":"2024-09-16T01:36:40.470459+0000","last_fullsized":"2024-09-16T01:36:40.470459+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:31:33.950166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475724+0000","last_change":"2024-09-16T01:36:40.465788+0000","last_active":"2024-09-16T01:36:40.475724+0000","last_peered":"2024-09-16T01:36:40.475724+0000","last_clean":"2024-09-16T01:36:40.475724+0000","last_became_active":"2024-09-16T01:36:40.464511+0000","last_became_peered":"2024-09-16T01:36:40.464511+0000","last_unstale":"2024-09-16T01:36:40.475724+0000","last_undegraded":"2024-09-16T01:36:40.475724+0000","last_fullsized":"2024-09-16T01:36:40.475724+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:09:16.216025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.474615+0000","last_change":"2024-09-16T01:36:40.466280+0000","last_active":"2024-09-16T01:36:40.474615+0000","last_peered":"2024-09-16T01:36:40.474615+0000","last_clean":"2024-09-16T01:36:40.474615+0000","last_became_active":"2024-09-16T01:36:40.464657+0000","last_became_peered":"2024-09-16T01:36:40.464657+0000","last_unstale":"2024-09-16T01:36:40.474615+0000","last_undegraded":"2024-09-16T01:36:40.474615+0000","last_fullsized":"2024-09-16T01:36:40.474615+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:56:12.402516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476854+0000","last_change":"2024-09-16T01:36:40.472186+0000","last_active":"2024-09-16T01:36:40.476854+0000","last_peered":"2024-09-16T01:36:40.476854+0000","last_clean":"2024-09-16T01:36:40.476854+0000","last_became_active":"2024-09-16T01:36:40.471918+0000","last_became_peered":"2024-09-16T01:36:40.471918+0000","last_unstale":"2024-09-16T01:36:40.476854+0000","last_undegraded":"2024-09-16T01:36:40.476854+0000","last_fullsized":"2024-09-16T01:36:40.476854+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:32:32.193049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478292+0000","last_change":"2024-09-16T01:36:40.471732+0000","last_active":"2024-09-16T01:36:40.478292+0000","last_peered":"2024-09-16T01:36:40.478292+0000","last_clean":"2024-09-16T01:36:40.478292+0000","last_became_active":"2024-09-16T01:36:40.471441+0000","last_became_peered":"2024-09-16T01:36:40.471441+0000","last_unstale":"2024-09-16T01:36:40.478292+0000","last_undegraded":"2024-09-16T01:36:40.478292+0000","last_fullsized":"2024-09-16T01:36:40.478292+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:58:01.503914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.458508+0000","last_change":"2024-09-16T01:36:40.457225+0000","last_active":"2024-09-16T01:36:40.458508+0000","last_peered":"2024-09-16T01:36:40.458508+0000","last_clean":"2024-09-16T01:36:40.458508+0000","last_became_active":"2024-09-16T01:36:40.456967+0000","last_became_peered":"2024-09-16T01:36:40.456967+0000","last_unstale":"2024-09-16T01:36:40.458508+0000","last_undegraded":"2024-09-16T01:36:40.458508+0000","last_fullsized":"2024-09-16T01:36:40.458508+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:57.227814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.477879+0000","last_change":"2024-09-16T01:36:40.474827+0000","last_active":"2024-09-16T01:36:40.477879+0000","last_peered":"2024-09-16T01:36:40.477879+0000","last_clean":"2024-09-16T01:36:40.477879+0000","last_became_active":"2024-09-16T01:36:40.474479+0000","last_became_peered":"2024-09-16T01:36:40.474479+0000","last_unstale":"2024-09-16T01:36:40.477879+0000","last_undegraded":"2024-09-16T01:36:40.477879+0000","last_fullsized":"2024-09-16T01:36:40.477879+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:47.059215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475710+0000","last_change":"2024-09-16T01:36:40.469807+0000","last_active":"2024-09-16T01:36:40.475710+0000","last_peered":"2024-09-16T01:36:40.475710+0000","last_clean":"2024-09-16T01:36:40.475710+0000","last_became_active":"2024-09-16T01:36:40.469542+0000","last_became_peered":"2024-09-16T01:36:40.469542+0000","last_unstale":"2024-09-16T01:36:40.475710+0000","last_undegraded":"2024-09-16T01:36:40.475710+0000","last_fullsized":"2024-09-16T01:36:40.475710+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:30:00.881153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"57'4","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476066+0000","last_change":"2024-09-16T01:36:39.464332+0000","last_active":"2024-09-16T01:36:40.476066+0000","last_peered":"2024-09-16T01:36:40.476066+0000","last_clean":"2024-09-16T01:36:40.476066+0000","last_became_active":"2024-09-16T01:36:39.464067+0000","last_became_peered":"2024-09-16T01:36:39.464067+0000","last_unstale":"2024-09-16T01:36:40.476066+0000","last_undegraded":"2024-09-16T01:36:40.476066+0000","last_fullsized":"2024-09-16T01:36:40.476066+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:38.563765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476505+0000","last_change":"2024-09-16T01:36:40.471935+0000","last_active":"2024-09-16T01:36:40.476505+0000","last_peered":"2024-09-16T01:36:40.476505+0000","last_clean":"2024-09-16T01:36:40.476505+0000","last_became_active":"2024-09-16T01:36:40.471348+0000","last_became_peered":"2024-09-16T01:36:40.471348+0000","last_unstale":"2024-09-16T01:36:40.476505+0000","last_undegraded":"2024-09-16T01:36:40.476505+0000","last_fullsized":"2024-09-16T01:36:40.476505+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:31:57.903347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476472+0000","last_change":"2024-09-16T01:36:39.463823+0000","last_active":"2024-09-16T01:36:40.476472+0000","last_peered":"2024-09-16T01:36:40.476472+0000","last_clean":"2024-09-16T01:36:40.476472+0000","last_became_active":"2024-09-16T01:36:39.463549+0000","last_became_peered":"2024-09-16T01:36:39.463549+0000","last_unstale":"2024-09-16T01:36:40.476472+0000","last_undegraded":"2024-09-16T01:36:40.476472+0000","last_fullsized":"2024-09-16T01:36:40.476472+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:29:59.879003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475035+0000","last_change":"2024-09-16T01:36:40.463367+0000","last_active":"2024-09-16T01:36:40.475035+0000","last_peered":"2024-09-16T01:36:40.475035+0000","last_clean":"2024-09-16T01:36:40.475035+0000","last_became_active":"2024-09-16T01:36:40.462287+0000","last_became_peered":"2024-09-16T01:36:40.462287+0000","last_unstale":"2024-09-16T01:36:40.475035+0000","last_undegraded":"2024-09-16T01:36:40.475035+0000","last_fullsized":"2024-09-16T01:36:40.475035+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:15.940785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.477088+0000","last_change":"2024-09-16T01:36:39.461073+0000","last_active":"2024-09-16T01:36:40.477088+0000","last_peered":"2024-09-16T01:36:40.477088+0000","last_clean":"2024-09-16T01:36:40.477088+0000","last_became_active":"2024-09-16T01:36:39.460821+0000","last_became_peered":"2024-09-16T01:36:39.460821+0000","last_unstale":"2024-09-16T01:36:40.477088+0000","last_undegraded":"2024-09-16T01:36:40.477088+0000","last_fullsized":"2024-09-16T01:36:40.477088+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:09:57.224365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475781+0000","last_change":"2024-09-16T01:36:40.469776+0000","last_active":"2024-09-16T01:36:40.475781+0000","last_peered":"2024-09-16T01:36:40.475781+0000","last_clean":"2024-09-16T01:36:40.475781+0000","last_became_active":"2024-09-16T01:36:40.465433+0000","last_became_peered":"2024-09-16T01:36:40.465433+0000","last_unstale":"2024-09-16T01:36:40.475781+0000","last_undegraded":"2024-09-16T01:36:40.475781+0000","last_fullsized":"2024-09-16T01:36:40.475781+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:47:43.290566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476032+0000","last_change":"2024-09-16T01:36:39.460421+0000","last_active":"2024-09-16T01:36:40.476032+0000","last_peered":"2024-09-16T01:36:40.476032+0000","last_clean":"2024-09-16T01:36:40.476032+0000","last_became_active":"2024-09-16T01:36:39.460146+0000","last_became_peered":"2024-09-16T01:36:39.460146+0000","last_unstale":"2024-09-16T01:36:40.476032+0000","last_undegraded":"2024-09-16T01:36:40.476032+0000","last_fullsized":"2024-09-16T01:36:40.476032+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:59:06.387430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.469846+0000","last_change":"2024-09-16T01:36:40.461743+0000","last_active":"2024-09-16T01:36:40.469846+0000","last_peered":"2024-09-16T01:36:40.469846+0000","last_clean":"2024-09-16T01:36:40.469846+0000","last_became_active":"2024-09-16T01:36:40.461455+0000","last_became_peered":"2024-09-16T01:36:40.461455+0000","last_unstale":"2024-09-16T01:36:40.469846+0000","last_undegraded":"2024-09-16T01:36:40.469846+0000","last_fullsized":"2024-09-16T01:36:40.469846+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:23:19.741838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.474803+0000","last_change":"2024-09-16T01:36:39.458301+0000","last_active":"2024-09-16T01:36:40.474803+0000","last_peered":"2024-09-16T01:36:40.474803+0000","last_clean":"2024-09-16T01:36:40.474803+0000","last_became_active":"2024-09-16T01:36:39.457987+0000","last_became_peered":"2024-09-16T01:36:39.457987+0000","last_unstale":"2024-09-16T01:36:40.474803+0000","last_undegraded":"2024-09-16T01:36:40.474803+0000","last_fullsized":"2024-09-16T01:36:40.474803+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:28:16.583211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.479343+0000","last_change":"2024-09-16T01:36:40.476287+0000","last_active":"2024-09-16T01:36:40.479343+0000","last_peered":"2024-09-16T01:36:40.479343+0000","last_clean":"2024-09-16T01:36:40.479343+0000","last_became_active":"2024-09-16T01:36:40.476026+0000","last_became_peered":"2024-09-16T01:36:40.476026+0000","last_unstale":"2024-09-16T01:36:40.479343+0000","last_undegraded":"2024-09-16T01:36:40.479343+0000","last_fullsized":"2024-09-16T01:36:40.479343+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:05:50.979850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475673+0000","last_change":"2024-09-16T01:36:39.460688+0000","last_active":"2024-09-16T01:36:40.475673+0000","last_peered":"2024-09-16T01:36:40.475673+0000","last_clean":"2024-09-16T01:36:40.475673+0000","last_became_active":"2024-09-16T01:36:39.460340+0000","last_became_peered":"2024-09-16T01:36:39.460340+0000","last_unstale":"2024-09-16T01:36:40.475673+0000","last_undegraded":"2024-09-16T01:36:40.475673+0000","last_fullsized":"2024-09-16T01:36:40.475673+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:46:36.645856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.472679+0000","last_change":"2024-09-16T01:36:40.468028+0000","last_active":"2024-09-16T01:36:40.472679+0000","last_peered":"2024-09-16T01:36:40.472679+0000","last_clean":"2024-09-16T01:36:40.472679+0000","last_became_active":"2024-09-16T01:36:40.467726+0000","last_became_peered":"2024-09-16T01:36:40.467726+0000","last_unstale":"2024-09-16T01:36:40.472679+0000","last_undegraded":"2024-09-16T01:36:40.472679+0000","last_fullsized":"2024-09-16T01:36:40.472679+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:20.116406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.461077+0000","last_change":"2024-09-16T01:36:39.461252+0000","last_active":"2024-09-16T01:36:40.461077+0000","last_peered":"2024-09-16T01:36:40.461077+0000","last_clean":"2024-09-16T01:36:40.461077+0000","last_became_active":"2024-09-16T01:36:39.460960+0000","last_became_peered":"2024-09-16T01:36:39.460960+0000","last_unstale":"2024-09-16T01:36:40.461077+0000","last_undegraded":"2024-09-16T01:36:40.461077+0000","last_fullsized":"2024-09-16T01:36:40.461077+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:24:52.758504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476741+0000","last_change":"2024-09-16T01:36:40.469674+0000","last_active":"2024-09-16T01:36:40.476741+0000","last_peered":"2024-09-16T01:36:40.476741+0000","last_clean":"2024-09-16T01:36:40.476741+0000","last_became_active":"2024-09-16T01:36:40.469399+0000","last_became_peered":"2024-09-16T01:36:40.469399+0000","last_unstale":"2024-09-16T01:36:40.476741+0000","last_undegraded":"2024-09-16T01:36:40.476741+0000","last_fullsized":"2024-09-16T01:36:40.476741+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:32:32.869150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.451968+0000","last_change":"2024-09-16T01:36:39.465116+0000","last_active":"2024-09-16T01:36:40.451968+0000","last_peered":"2024-09-16T01:36:40.451968+0000","last_clean":"2024-09-16T01:36:40.451968+0000","last_became_active":"2024-09-16T01:36:39.464777+0000","last_became_peered":"2024-09-16T01:36:39.464777+0000","last_unstale":"2024-09-16T01:36:40.451968+0000","last_undegraded":"2024-09-16T01:36:40.451968+0000","last_fullsized":"2024-09-16T01:36:40.451968+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:50:58.934945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.466221+0000","last_change":"2024-09-16T01:36:40.461894+0000","last_active":"2024-09-16T01:36:40.466221+0000","last_peered":"2024-09-16T01:36:40.466221+0000","last_clean":"2024-09-16T01:36:40.466221+0000","last_became_active":"2024-09-16T01:36:40.461602+0000","last_became_peered":"2024-09-16T01:36:40.461602+0000","last_unstale":"2024-09-16T01:36:40.466221+0000","last_undegraded":"2024-09-16T01:36:40.466221+0000","last_fullsized":"2024-09-16T01:36:40.466221+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:02:20.953972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.462154+0000","last_change":"2024-09-16T01:36:39.463527+0000","last_active":"2024-09-16T01:36:40.462154+0000","last_peered":"2024-09-16T01:36:40.462154+0000","last_clean":"2024-09-16T01:36:40.462154+0000","last_became_active":"2024-09-16T01:36:39.463194+0000","last_became_peered":"2024-09-16T01:36:39.463194+0000","last_unstale":"2024-09-16T01:36:40.462154+0000","last_undegraded":"2024-09-16T01:36:40.462154+0000","last_fullsized":"2024-09-16T01:36:40.462154+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:20:36.477427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478234+0000","last_change":"2024-09-16T01:36:40.471438+0000","last_active":"2024-09-16T01:36:40.478234+0000","last_peered":"2024-09-16T01:36:40.478234+0000","last_clean":"2024-09-16T01:36:40.478234+0000","last_became_active":"2024-09-16T01:36:40.471177+0000","last_became_peered":"2024-09-16T01:36:40.471177+0000","last_unstale":"2024-09-16T01:36:40.478234+0000","last_undegraded":"2024-09-16T01:36:40.478234+0000","last_fullsized":"2024-09-16T01:36:40.478234+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:59:07.389579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.457422+0000","last_change":"2024-09-16T01:36:39.460934+0000","last_active":"2024-09-16T01:36:40.457422+0000","last_peered":"2024-09-16T01:36:40.457422+0000","last_clean":"2024-09-16T01:36:40.457422+0000","last_became_active":"2024-09-16T01:36:39.460651+0000","last_became_peered":"2024-09-16T01:36:39.460651+0000","last_unstale":"2024-09-16T01:36:40.457422+0000","last_undegraded":"2024-09-16T01:36:40.457422+0000","last_fullsized":"2024-09-16T01:36:40.457422+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:04.432074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.466094+0000","last_change":"2024-09-16T01:36:40.463248+0000","last_active":"2024-09-16T01:36:40.466094+0000","last_peered":"2024-09-16T01:36:40.466094+0000","last_clean":"2024-09-16T01:36:40.466094+0000","last_became_active":"2024-09-16T01:36:40.462992+0000","last_became_peered":"2024-09-16T01:36:40.462992+0000","last_unstale":"2024-09-16T01:36:40.466094+0000","last_undegraded":"2024-09-16T01:36:40.466094+0000","last_fullsized":"2024-09-16T01:36:40.466094+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:16:03.996778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"57'3","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478311+0000","last_change":"2024-09-16T01:36:39.459827+0000","last_active":"2024-09-16T01:36:40.478311+0000","last_peered":"2024-09-16T01:36:40.478311+0000","last_clean":"2024-09-16T01:36:40.478311+0000","last_became_active":"2024-09-16T01:36:39.459136+0000","last_became_peered":"2024-09-16T01:36:39.459136+0000","last_unstale":"2024-09-16T01:36:40.478311+0000","last_undegraded":"2024-09-16T01:36:40.478311+0000","last_fullsized":"2024-09-16T01:36:40.478311+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:17:11.701945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478377+0000","last_change":"2024-09-16T01:36:40.473080+0000","last_active":"2024-09-16T01:36:40.478377+0000","last_peered":"2024-09-16T01:36:40.478377+0000","last_clean":"2024-09-16T01:36:40.478377+0000","last_became_active":"2024-09-16T01:36:40.472390+0000","last_became_peered":"2024-09-16T01:36:40.472390+0000","last_unstale":"2024-09-16T01:36:40.478377+0000","last_undegraded":"2024-09-16T01:36:40.478377+0000","last_fullsized":"2024-09-16T01:36:40.478377+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:55:25.810834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"57'6","reported_seq":18,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.480875+0000","last_change":"2024-09-16T01:36:39.463970+0000","last_active":"2024-09-16T01:36:40.480875+0000","last_peered":"2024-09-16T01:36:40.480875+0000","last_clean":"2024-09-16T01:36:40.480875+0000","last_became_active":"2024-09-16T01:36:39.463578+0000","last_became_peered":"2024-09-16T01:36:39.463578+0000","last_unstale":"2024-09-16T01:36:40.480875+0000","last_undegraded":"2024-09-16T01:36:40.480875+0000","last_fullsized":"2024-09-16T01:36:40.480875+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:42:13.498400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.466657+0000","last_change":"2024-09-16T01:36:40.460384+0000","last_active":"2024-09-16T01:36:40.466657+0000","last_peered":"2024-09-16T01:36:40.466657+0000","last_clean":"2024-09-16T01:36:40.466657+0000","last_became_active":"2024-09-16T01:36:40.460100+0000","last_became_peered":"2024-09-16T01:36:40.460100+0000","last_unstale":"2024-09-16T01:36:40.466657+0000","last_undegraded":"2024-09-16T01:36:40.466657+0000","last_fullsized":"2024-09-16T01:36:40.466657+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:20:37.479577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475364+0000","last_change":"2024-09-16T01:36:39.463203+0000","last_active":"2024-09-16T01:36:40.475364+0000","last_peered":"2024-09-16T01:36:40.475364+0000","last_clean":"2024-09-16T01:36:40.475364+0000","last_became_active":"2024-09-16T01:36:39.462964+0000","last_became_peered":"2024-09-16T01:36:39.462964+0000","last_unstale":"2024-09-16T01:36:40.475364+0000","last_undegraded":"2024-09-16T01:36:40.475364+0000","last_fullsized":"2024-09-16T01:36:40.475364+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:25:47.151356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475549+0000","last_change":"2024-09-16T01:36:40.466527+0000","last_active":"2024-09-16T01:36:40.475549+0000","last_peered":"2024-09-16T01:36:40.475549+0000","last_clean":"2024-09-16T01:36:40.475549+0000","last_became_active":"2024-09-16T01:36:40.465922+0000","last_became_peered":"2024-09-16T01:36:40.465922+0000","last_unstale":"2024-09-16T01:36:40.475549+0000","last_undegraded":"2024-09-16T01:36:40.475549+0000","last_fullsized":"2024-09-16T01:36:40.475549+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:20:36.054563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.456156+0000","last_change":"2024-09-16T01:36:39.458436+0000","last_active":"2024-09-16T01:36:40.456156+0000","last_peered":"2024-09-16T01:36:40.456156+0000","last_clean":"2024-09-16T01:36:40.456156+0000","last_became_active":"2024-09-16T01:36:39.458026+0000","last_became_peered":"2024-09-16T01:36:39.458026+0000","last_unstale":"2024-09-16T01:36:40.456156+0000","last_undegraded":"2024-09-16T01:36:40.456156+0000","last_fullsized":"2024-09-16T01:36:40.456156+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:59:30.672192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.473542+0000","last_change":"2024-09-16T01:36:40.471647+0000","last_active":"2024-09-16T01:36:40.473542+0000","last_peered":"2024-09-16T01:36:40.473542+0000","last_clean":"2024-09-16T01:36:40.473542+0000","last_became_active":"2024-09-16T01:36:40.471346+0000","last_became_peered":"2024-09-16T01:36:40.471346+0000","last_unstale":"2024-09-16T01:36:40.473542+0000","last_undegraded":"2024-09-16T01:36:40.473542+0000","last_fullsized":"2024-09-16T01:36:40.473542+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:09:58.226514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.461233+0000","last_change":"2024-09-16T01:36:39.461751+0000","last_active":"2024-09-16T01:36:40.461233+0000","last_peered":"2024-09-16T01:36:40.461233+0000","last_clean":"2024-09-16T01:36:40.461233+0000","last_became_active":"2024-09-16T01:36:39.461513+0000","last_became_peered":"2024-09-16T01:36:39.461513+0000","last_unstale":"2024-09-16T01:36:40.461233+0000","last_undegraded":"2024-09-16T01:36:40.461233+0000","last_fullsized":"2024-09-16T01:36:40.461233+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:20:35.052413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478579+0000","last_change":"2024-09-16T01:36:40.477324+0000","last_active":"2024-09-16T01:36:40.478579+0000","last_peered":"2024-09-16T01:36:40.478579+0000","last_clean":"2024-09-16T01:36:40.478579+0000","last_became_active":"2024-09-16T01:36:40.477120+0000","last_became_peered":"2024-09-16T01:36:40.477120+0000","last_unstale":"2024-09-16T01:36:40.478579+0000","last_undegraded":"2024-09-16T01:36:40.478579+0000","last_fullsized":"2024-09-16T01:36:40.478579+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:41:31.417859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"57'8","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478199+0000","last_change":"2024-09-16T01:36:39.463560+0000","last_active":"2024-09-16T01:36:40.478199+0000","last_peered":"2024-09-16T01:36:40.478199+0000","last_clean":"2024-09-16T01:36:40.478199+0000","last_became_active":"2024-09-16T01:36:39.462861+0000","last_became_peered":"2024-09-16T01:36:39.462861+0000","last_unstale":"2024-09-16T01:36:40.478199+0000","last_undegraded":"2024-09-16T01:36:40.478199+0000","last_fullsized":"2024-09-16T01:36:40.478199+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:20:05.524194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478437+0000","last_change":"2024-09-16T01:36:40.472319+0000","last_active":"2024-09-16T01:36:40.478437+0000","last_peered":"2024-09-16T01:36:40.478437+0000","last_clean":"2024-09-16T01:36:40.478437+0000","last_became_active":"2024-09-16T01:36:40.471860+0000","last_became_peered":"2024-09-16T01:36:40.471860+0000","last_unstale":"2024-09-16T01:36:40.478437+0000","last_undegraded":"2024-09-16T01:36:40.478437+0000","last_fullsized":"2024-09-16T01:36:40.478437+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:21:05.023637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475280+0000","last_change":"2024-09-16T01:36:39.462890+0000","last_active":"2024-09-16T01:36:40.475280+0000","last_peered":"2024-09-16T01:36:40.475280+0000","last_clean":"2024-09-16T01:36:40.475280+0000","last_became_active":"2024-09-16T01:36:39.462530+0000","last_became_peered":"2024-09-16T01:36:39.462530+0000","last_unstale":"2024-09-16T01:36:40.475280+0000","last_undegraded":"2024-09-16T01:36:40.475280+0000","last_fullsized":"2024-09-16T01:36:40.475280+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:52:52.113632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.470495+0000","last_change":"2024-09-16T01:36:40.464108+0000","last_active":"2024-09-16T01:36:40.470495+0000","last_peered":"2024-09-16T01:36:40.470495+0000","last_clean":"2024-09-16T01:36:40.470495+0000","last_became_active":"2024-09-16T01:36:40.463824+0000","last_became_peered":"2024-09-16T01:36:40.463824+0000","last_unstale":"2024-09-16T01:36:40.470495+0000","last_undegraded":"2024-09-16T01:36:40.470495+0000","last_fullsized":"2024-09-16T01:36:40.470495+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:55:25.810834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475356+0000","last_change":"2024-09-16T01:36:39.460547+0000","last_active":"2024-09-16T01:36:40.475356+0000","last_peered":"2024-09-16T01:36:40.475356+0000","last_clean":"2024-09-16T01:36:40.475356+0000","last_became_active":"2024-09-16T01:36:39.460250+0000","last_became_peered":"2024-09-16T01:36:39.460250+0000","last_unstale":"2024-09-16T01:36:40.475356+0000","last_undegraded":"2024-09-16T01:36:40.475356+0000","last_fullsized":"2024-09-16T01:36:40.475356+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:42:36.045032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.477483+0000","last_change":"2024-09-16T01:36:40.473969+0000","last_active":"2024-09-16T01:36:40.477483+0000","last_peered":"2024-09-16T01:36:40.477483+0000","last_clean":"2024-09-16T01:36:40.477483+0000","last_became_active":"2024-09-16T01:36:40.473666+0000","last_became_peered":"2024-09-16T01:36:40.473666+0000","last_unstale":"2024-09-16T01:36:40.477483+0000","last_undegraded":"2024-09-16T01:36:40.477483+0000","last_fullsized":"2024-09-16T01:36:40.477483+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:55:18.946116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475994+0000","last_change":"2024-09-16T01:36:39.463153+0000","last_active":"2024-09-16T01:36:40.475994+0000","last_peered":"2024-09-16T01:36:40.475994+0000","last_clean":"2024-09-16T01:36:40.475994+0000","last_became_active":"2024-09-16T01:36:39.462877+0000","last_became_peered":"2024-09-16T01:36:39.462877+0000","last_unstale":"2024-09-16T01:36:40.475994+0000","last_undegraded":"2024-09-16T01:36:40.475994+0000","last_fullsized":"2024-09-16T01:36:40.475994+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:06:00.649503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478446+0000","last_change":"2024-09-16T01:36:40.473180+0000","last_active":"2024-09-16T01:36:40.478446+0000","last_peered":"2024-09-16T01:36:40.478446+0000","last_clean":"2024-09-16T01:36:40.478446+0000","last_became_active":"2024-09-16T01:36:40.472662+0000","last_became_peered":"2024-09-16T01:36:40.472662+0000","last_unstale":"2024-09-16T01:36:40.478446+0000","last_undegraded":"2024-09-16T01:36:40.478446+0000","last_fullsized":"2024-09-16T01:36:40.478446+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:30:03.124326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.454495+0000","last_change":"2024-09-16T01:36:39.462536+0000","last_active":"2024-09-16T01:36:40.454495+0000","last_peered":"2024-09-16T01:36:40.454495+0000","last_clean":"2024-09-16T01:36:40.454495+0000","last_became_active":"2024-09-16T01:36:39.462284+0000","last_became_peered":"2024-09-16T01:36:39.462284+0000","last_unstale":"2024-09-16T01:36:40.454495+0000","last_undegraded":"2024-09-16T01:36:40.454495+0000","last_fullsized":"2024-09-16T01:36:40.454495+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:26:22.279806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.462168+0000","last_change":"2024-09-16T01:36:40.458778+0000","last_active":"2024-09-16T01:36:40.462168+0000","last_peered":"2024-09-16T01:36:40.462168+0000","last_clean":"2024-09-16T01:36:40.462168+0000","last_became_active":"2024-09-16T01:36:40.458188+0000","last_became_peered":"2024-09-16T01:36:40.458188+0000","last_unstale":"2024-09-16T01:36:40.462168+0000","last_undegraded":"2024-09-16T01:36:40.462168+0000","last_fullsized":"2024-09-16T01:36:40.462168+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:20:06.526343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.473804+0000","last_change":"2024-09-16T01:36:39.462685+0000","last_active":"2024-09-16T01:36:40.473804+0000","last_peered":"2024-09-16T01:36:40.473804+0000","last_clean":"2024-09-16T01:36:40.473804+0000","last_became_active":"2024-09-16T01:36:39.462437+0000","last_became_peered":"2024-09-16T01:36:39.462437+0000","last_unstale":"2024-09-16T01:36:40.473804+0000","last_undegraded":"2024-09-16T01:36:40.473804+0000","last_fullsized":"2024-09-16T01:36:40.473804+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:26.733779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475794+0000","last_change":"2024-09-16T01:36:40.464106+0000","last_active":"2024-09-16T01:36:40.475794+0000","last_peered":"2024-09-16T01:36:40.475794+0000","last_clean":"2024-09-16T01:36:40.475794+0000","last_became_active":"2024-09-16T01:36:40.463825+0000","last_became_peered":"2024-09-16T01:36:40.463825+0000","last_unstale":"2024-09-16T01:36:40.475794+0000","last_undegraded":"2024-09-16T01:36:40.475794+0000","last_fullsized":"2024-09-16T01:36:40.475794+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:47:14.256132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.454549+0000","last_change":"2024-09-16T01:36:39.456580+0000","last_active":"2024-09-16T01:36:40.454549+0000","last_peered":"2024-09-16T01:36:40.454549+0000","last_clean":"2024-09-16T01:36:40.454549+0000","last_became_active":"2024-09-16T01:36:39.456335+0000","last_became_peered":"2024-09-16T01:36:39.456335+0000","last_unstale":"2024-09-16T01:36:40.454549+0000","last_undegraded":"2024-09-16T01:36:40.454549+0000","last_fullsized":"2024-09-16T01:36:40.454549+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:46:35.905236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.477768+0000","last_change":"2024-09-16T01:36:40.472405+0000","last_active":"2024-09-16T01:36:40.477768+0000","last_peered":"2024-09-16T01:36:40.477768+0000","last_clean":"2024-09-16T01:36:40.477768+0000","last_became_active":"2024-09-16T01:36:40.472159+0000","last_became_peered":"2024-09-16T01:36:40.472159+0000","last_unstale":"2024-09-16T01:36:40.477768+0000","last_undegraded":"2024-09-16T01:36:40.477768+0000","last_fullsized":"2024-09-16T01:36:40.477768+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:05.434223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.459880+0000","last_change":"2024-09-16T01:36:39.461853+0000","last_active":"2024-09-16T01:36:40.459880+0000","last_peered":"2024-09-16T01:36:40.459880+0000","last_clean":"2024-09-16T01:36:40.459880+0000","last_became_active":"2024-09-16T01:36:39.461332+0000","last_became_peered":"2024-09-16T01:36:39.461332+0000","last_unstale":"2024-09-16T01:36:40.459880+0000","last_undegraded":"2024-09-16T01:36:40.459880+0000","last_fullsized":"2024-09-16T01:36:40.459880+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:17:11.701945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.470436+0000","last_change":"2024-09-16T01:36:40.463367+0000","last_active":"2024-09-16T01:36:40.470436+0000","last_peered":"2024-09-16T01:36:40.470436+0000","last_clean":"2024-09-16T01:36:40.470436+0000","last_became_active":"2024-09-16T01:36:40.463095+0000","last_became_peered":"2024-09-16T01:36:40.463095+0000","last_unstale":"2024-09-16T01:36:40.470436+0000","last_undegraded":"2024-09-16T01:36:40.470436+0000","last_fullsized":"2024-09-16T01:36:40.470436+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:21:05.023637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.477568+0000","last_change":"2024-09-16T01:36:39.460751+0000","last_active":"2024-09-16T01:36:40.477568+0000","last_peered":"2024-09-16T01:36:40.477568+0000","last_clean":"2024-09-16T01:36:40.477568+0000","last_became_active":"2024-09-16T01:36:39.460478+0000","last_became_peered":"2024-09-16T01:36:39.460478+0000","last_unstale":"2024-09-16T01:36:40.477568+0000","last_undegraded":"2024-09-16T01:36:40.477568+0000","last_fullsized":"2024-09-16T01:36:40.477568+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:25:16.927780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478393+0000","last_change":"2024-09-16T01:36:40.473726+0000","last_active":"2024-09-16T01:36:40.478393+0000","last_peered":"2024-09-16T01:36:40.478393+0000","last_clean":"2024-09-16T01:36:40.478393+0000","last_became_active":"2024-09-16T01:36:40.473241+0000","last_became_peered":"2024-09-16T01:36:40.473241+0000","last_unstale":"2024-09-16T01:36:40.478393+0000","last_undegraded":"2024-09-16T01:36:40.478393+0000","last_fullsized":"2024-09-16T01:36:40.478393+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:37:37.379239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.455105+0000","last_change":"2024-09-16T01:36:39.460521+0000","last_active":"2024-09-16T01:36:40.455105+0000","last_peered":"2024-09-16T01:36:40.455105+0000","last_clean":"2024-09-16T01:36:40.455105+0000","last_became_active":"2024-09-16T01:36:39.460221+0000","last_became_peered":"2024-09-16T01:36:39.460221+0000","last_unstale":"2024-09-16T01:36:40.455105+0000","last_undegraded":"2024-09-16T01:36:40.455105+0000","last_fullsized":"2024-09-16T01:36:40.455105+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:59:36.053696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.466072+0000","last_change":"2024-09-16T01:36:40.462925+0000","last_active":"2024-09-16T01:36:40.466072+0000","last_peered":"2024-09-16T01:36:40.466072+0000","last_clean":"2024-09-16T01:36:40.466072+0000","last_became_active":"2024-09-16T01:36:40.462608+0000","last_became_peered":"2024-09-16T01:36:40.462608+0000","last_unstale":"2024-09-16T01:36:40.466072+0000","last_undegraded":"2024-09-16T01:36:40.466072+0000","last_fullsized":"2024-09-16T01:36:40.466072+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:38.209013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.456363+0000","last_change":"2024-09-16T01:36:39.457230+0000","last_active":"2024-09-16T01:36:40.456363+0000","last_peered":"2024-09-16T01:36:40.456363+0000","last_clean":"2024-09-16T01:36:40.456363+0000","last_became_active":"2024-09-16T01:36:39.456929+0000","last_became_peered":"2024-09-16T01:36:39.456929+0000","last_unstale":"2024-09-16T01:36:40.456363+0000","last_undegraded":"2024-09-16T01:36:40.456363+0000","last_fullsized":"2024-09-16T01:36:40.456363+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:42:21.156826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.475702+0000","last_change":"2024-09-16T01:36:40.465659+0000","last_active":"2024-09-16T01:36:40.475702+0000","last_peered":"2024-09-16T01:36:40.475702+0000","last_clean":"2024-09-16T01:36:40.475702+0000","last_became_active":"2024-09-16T01:36:40.465102+0000","last_became_peered":"2024-09-16T01:36:40.465102+0000","last_unstale":"2024-09-16T01:36:40.475702+0000","last_undegraded":"2024-09-16T01:36:40.475702+0000","last_fullsized":"2024-09-16T01:36:40.475702+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:50:35.003419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.461484+0000","last_change":"2024-09-16T01:36:39.461697+0000","last_active":"2024-09-16T01:36:40.461484+0000","last_peered":"2024-09-16T01:36:40.461484+0000","last_clean":"2024-09-16T01:36:40.461484+0000","last_became_active":"2024-09-16T01:36:39.461417+0000","last_became_peered":"2024-09-16T01:36:39.461417+0000","last_unstale":"2024-09-16T01:36:40.461484+0000","last_undegraded":"2024-09-16T01:36:40.461484+0000","last_fullsized":"2024-09-16T01:36:40.461484+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:58:00.501765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476660+0000","last_change":"2024-09-16T01:36:40.470075+0000","last_active":"2024-09-16T01:36:40.476660+0000","last_peered":"2024-09-16T01:36:40.476660+0000","last_clean":"2024-09-16T01:36:40.476660+0000","last_became_active":"2024-09-16T01:36:40.469755+0000","last_became_peered":"2024-09-16T01:36:40.469755+0000","last_unstale":"2024-09-16T01:36:40.476660+0000","last_undegraded":"2024-09-16T01:36:40.476660+0000","last_fullsized":"2024-09-16T01:36:40.476660+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:09:59.945976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.452422+0000","last_change":"2024-09-16T01:36:39.459118+0000","last_active":"2024-09-16T01:36:40.452422+0000","last_peered":"2024-09-16T01:36:40.452422+0000","last_clean":"2024-09-16T01:36:40.452422+0000","last_became_active":"2024-09-16T01:36:39.458865+0000","last_became_peered":"2024-09-16T01:36:39.458865+0000","last_unstale":"2024-09-16T01:36:40.452422+0000","last_undegraded":"2024-09-16T01:36:40.452422+0000","last_fullsized":"2024-09-16T01:36:40.452422+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:12.692476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.478471+0000","last_change":"2024-09-16T01:36:40.474300+0000","last_active":"2024-09-16T01:36:40.478471+0000","last_peered":"2024-09-16T01:36:40.478471+0000","last_clean":"2024-09-16T01:36:40.478471+0000","last_became_active":"2024-09-16T01:36:40.474010+0000","last_became_peered":"2024-09-16T01:36:40.474010+0000","last_unstale":"2024-09-16T01:36:40.478471+0000","last_undegraded":"2024-09-16T01:36:40.478471+0000","last_fullsized":"2024-09-16T01:36:40.478471+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:54:18.427256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.452204+0000","last_change":"2024-09-16T01:36:39.463181+0000","last_active":"2024-09-16T01:36:40.452204+0000","last_peered":"2024-09-16T01:36:40.452204+0000","last_clean":"2024-09-16T01:36:40.452204+0000","last_became_active":"2024-09-16T01:36:39.462934+0000","last_became_peered":"2024-09-16T01:36:39.462934+0000","last_unstale":"2024-09-16T01:36:40.452204+0000","last_undegraded":"2024-09-16T01:36:40.452204+0000","last_fullsized":"2024-09-16T01:36:40.452204+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:05:48.877054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.466872+0000","last_change":"2024-09-16T01:36:40.463959+0000","last_active":"2024-09-16T01:36:40.466872+0000","last_peered":"2024-09-16T01:36:40.466872+0000","last_clean":"2024-09-16T01:36:40.466872+0000","last_became_active":"2024-09-16T01:36:40.463692+0000","last_became_peered":"2024-09-16T01:36:40.463692+0000","last_unstale":"2024-09-16T01:36:40.466872+0000","last_undegraded":"2024-09-16T01:36:40.466872+0000","last_fullsized":"2024-09-16T01:36:40.466872+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:09:11.772325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.454666+0000","last_change":"2024-09-16T01:36:39.464029+0000","last_active":"2024-09-16T01:36:40.454666+0000","last_peered":"2024-09-16T01:36:40.454666+0000","last_clean":"2024-09-16T01:36:40.454666+0000","last_became_active":"2024-09-16T01:36:39.463751+0000","last_became_peered":"2024-09-16T01:36:39.463751+0000","last_unstale":"2024-09-16T01:36:40.454666+0000","last_undegraded":"2024-09-16T01:36:40.454666+0000","last_fullsized":"2024-09-16T01:36:40.454666+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:08:17.467084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476762+0000","last_change":"2024-09-16T01:36:40.469615+0000","last_active":"2024-09-16T01:36:40.476762+0000","last_peered":"2024-09-16T01:36:40.476762+0000","last_clean":"2024-09-16T01:36:40.476762+0000","last_became_active":"2024-09-16T01:36:40.469371+0000","last_became_peered":"2024-09-16T01:36:40.469371+0000","last_unstale":"2024-09-16T01:36:40.476762+0000","last_undegraded":"2024-09-16T01:36:40.476762+0000","last_fullsized":"2024-09-16T01:36:40.476762+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:37:40.239240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.457964+0000","last_change":"2024-09-16T01:36:39.464984+0000","last_active":"2024-09-16T01:36:40.457964+0000","last_peered":"2024-09-16T01:36:40.457964+0000","last_clean":"2024-09-16T01:36:40.457964+0000","last_became_active":"2024-09-16T01:36:39.464726+0000","last_became_peered":"2024-09-16T01:36:39.464726+0000","last_unstale":"2024-09-16T01:36:40.457964+0000","last_undegraded":"2024-09-16T01:36:40.457964+0000","last_fullsized":"2024-09-16T01:36:40.457964+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:37:46.057066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.476882+0000","last_change":"2024-09-16T01:36:40.471212+0000","last_active":"2024-09-16T01:36:40.476882+0000","last_peered":"2024-09-16T01:36:40.476882+0000","last_clean":"2024-09-16T01:36:40.476882+0000","last_became_active":"2024-09-16T01:36:40.470000+0000","last_became_peered":"2024-09-16T01:36:40.470000+0000","last_unstale":"2024-09-16T01:36:40.476882+0000","last_undegraded":"2024-09-16T01:36:40.476882+0000","last_fullsized":"2024-09-16T01:36:40.476882+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:39.439693+0000","last_clean_scrub_stamp":"2024-09-16T01:36:39.439693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:06:50.678750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.455934+0000","last_change":"2024-09-16T01:36:39.459267+0000","last_active":"2024-09-16T01:36:40.455934+0000","last_peered":"2024-09-16T01:36:40.455934+0000","last_clean":"2024-09-16T01:36:40.455934+0000","last_became_active":"2024-09-16T01:36:39.458993+0000","last_became_peered":"2024-09-16T01:36:39.458993+0000","last_unstale":"2024-09-16T01:36:40.455934+0000","last_undegraded":"2024-09-16T01:36:40.455934+0000","last_fullsized":"2024-09-16T01:36:40.455934+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:36:38.437544+0000","last_clean_scrub_stamp":"2024-09-16T01:36:38.437544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:26:42.206194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":235,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T01:36:40.458337+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:36:40.458337+0000","last_peered":"2024-09-16T01:36:40.458337+0000","last_clean":"2024-09-16T01:36:40.458337+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:36:40.458337+0000","last_undegraded":"2024-09-16T01:36:40.458337+0000","last_fullsized":"2024-09-16T01:36:40.458337+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509475,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":688128,"data_stored":281799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32424,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339416,"statfs":{"total":96636764160,"available":96603561984,"internally_reserved":0,"allocated":1261568,"data_stored":864740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542178,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":667648,"data_stored":274372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542178,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":667648,"data_stored":274372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542177,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":667648,"data_stored":274372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542178,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":667648,"data_stored":274372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":667648,"data_stored":274372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542177,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28268,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343572,"statfs":{"total":96636764160,"available":96607817728,"internally_reserved":0,"allocated":1261568,"data_stored":864740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:36:43.732 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T01:36:43.732 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-16T01:36:43.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.872+0000 7fa911027640 1 Processor -- start 2024-09-16T01:36:43.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.872+0000 7fa911027640 1 -- start start 2024-09-16T01:36:43.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.873+0000 7fa911027640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa90c06f710 0x7fa90c06fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:43.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.873+0000 7fa911027640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa90c06bc10 0x7fa90c06c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:43.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.873+0000 7fa911027640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa90c074120 con 0x7fa90c15c2a0 2024-09-16T01:36:43.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.873+0000 7fa911027640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa90c15c680 con 0x7fa90c06f710 2024-09-16T01:36:43.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.873+0000 7fa911027640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa90c15ef00 con 0x7fa90c06bc10 2024-09-16T01:36:43.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.873+0000 7fa90a59c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa90c06f710 0x7fa90c06fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:43.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.873+0000 7fa90ad9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa90c15c2a0 0x7fa90c0735d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54322/0 (socket says 172.21.15.88:54322) 2024-09-16T01:36:43.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.873+0000 7fa90a59c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa90c06f710 0x7fa90c06fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41362/0 (socket says 172.21.15.88:41362) 2024-09-16T01:36:43.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.873+0000 7fa90ad9d640 1 -- 172.21.15.88:0/3492442142 learned_addr learned my addr 172.21.15.88:0/3492442142 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:43.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.873+0000 7fa90a59c640 1 -- 172.21.15.88:0/3492442142 >> v2:172.21.15.154:3300/0 conn(0x7fa90c06bc10 msgr2=0x7fa90c06c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:43.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.873+0000 7fa90a59c640 1 --2- 172.21.15.88:0/3492442142 >> v2:172.21.15.154:3300/0 conn(0x7fa90c06bc10 0x7fa90c06c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.873+0000 7fa90a59c640 1 -- 172.21.15.88:0/3492442142 >> v1:172.21.15.88:6789/0 conn(0x7fa90c15c2a0 legacy=0x7fa90c0735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:43.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.874+0000 7fa90a59c640 1 -- 172.21.15.88:0/3492442142 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa90c15f1e0 con 0x7fa90c06f710 2024-09-16T01:36:43.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.874+0000 7fa90a59c640 1 --2- 172.21.15.88:0/3492442142 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa90c06f710 0x7fa90c06fba0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fa90c0705f0 tx=0x7fa90002a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a844a395c3c51d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:43.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.874+0000 7fa909d9b640 1 -- 172.21.15.88:0/3492442142 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa900037070 con 0x7fa90c06f710 2024-09-16T01:36:43.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.874+0000 7fa909d9b640 1 -- 172.21.15.88:0/3492442142 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa90002ad30 con 0x7fa90c06f710 2024-09-16T01:36:43.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.874+0000 7fa909d9b640 1 -- 172.21.15.88:0/3492442142 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa900032660 con 0x7fa90c06f710 2024-09-16T01:36:43.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.875+0000 7fa911027640 1 -- 172.21.15.88:0/3492442142 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa90c06f710 msgr2=0x7fa90c06fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:43.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.875+0000 7fa911027640 1 --2- 172.21.15.88:0/3492442142 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa90c06f710 0x7fa90c06fba0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fa90c0705f0 tx=0x7fa90002a0a0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.875+0000 7fa911027640 1 -- 172.21.15.88:0/3492442142 shutdown_connections 2024-09-16T01:36:43.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.875+0000 7fa911027640 1 --2- 172.21.15.88:0/3492442142 >> v2:172.21.15.154:3300/0 conn(0x7fa90c06bc10 0x7fa90c06c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.875+0000 7fa911027640 1 --2- 172.21.15.88:0/3492442142 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa90c06f710 0x7fa90c06fba0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.875+0000 7fa911027640 1 -- 172.21.15.88:0/3492442142 >> 172.21.15.88:0/3492442142 conn(0x7fa90c0b3070 msgr2=0x7fa90c0b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:43.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.875+0000 7fa911027640 1 -- 172.21.15.88:0/3492442142 shutdown_connections 2024-09-16T01:36:43.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.875+0000 7fa911027640 1 -- 172.21.15.88:0/3492442142 wait complete. 2024-09-16T01:36:43.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.876+0000 7fa911027640 1 Processor -- start 2024-09-16T01:36:43.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.876+0000 7fa911027640 1 -- start start 2024-09-16T01:36:43.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.877+0000 7fa911027640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa90c06bc10 0x7fa90c14b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:43.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.877+0000 7fa911027640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa90c06f710 0x7fa90c1619c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:43.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.877+0000 7fa911027640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa90c15c2a0 0x7fa90c0ce810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:43.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.877+0000 7fa911027640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa90c0b5f20 con 0x7fa90c06bc10 2024-09-16T01:36:43.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.877+0000 7fa911027640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa90c15ebf0 con 0x7fa90c15c2a0 2024-09-16T01:36:43.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.877+0000 7fa911027640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa90c0b60a0 con 0x7fa90c06f710 2024-09-16T01:36:43.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.877+0000 7fa90ad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa90c06bc10 0x7fa90c14b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:43.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.877+0000 7fa90ad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa90c06bc10 0x7fa90c14b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41378/0 (socket says 172.21.15.88:41378) 2024-09-16T01:36:43.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.877+0000 7fa90ad9d640 1 -- 172.21.15.88:0/1378176498 learned_addr learned my addr 172.21.15.88:0/1378176498 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:43.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.877+0000 7fa90b59e640 1 --2- 172.21.15.88:0/1378176498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa90c15c2a0 0x7fa90c0ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:43.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.877+0000 7fa90a59c640 1 --2- 172.21.15.88:0/1378176498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa90c06f710 0x7fa90c1619c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:43.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.878+0000 7fa90ad9d640 1 -- 172.21.15.88:0/1378176498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa90c06f710 msgr2=0x7fa90c1619c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:43.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.878+0000 7fa90ad9d640 1 --2- 172.21.15.88:0/1378176498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa90c06f710 0x7fa90c1619c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.878+0000 7fa90ad9d640 1 -- 172.21.15.88:0/1378176498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa90c15c2a0 msgr2=0x7fa90c0ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:43.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.878+0000 7fa90ad9d640 1 --2- 172.21.15.88:0/1378176498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa90c15c2a0 0x7fa90c0ce810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:43.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.878+0000 7fa90ad9d640 1 -- 172.21.15.88:0/1378176498 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa90c161f00 con 0x7fa90c06bc10 2024-09-16T01:36:43.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.878+0000 7fa90ad9d640 1 --2- 172.21.15.88:0/1378176498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa90c06bc10 0x7fa90c14b020 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc00c570 tx=0x7fa8fc00ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:43.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.878+0000 7fa8f37fe640 1 -- 172.21.15.88:0/1378176498 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc018070 con 0x7fa90c06bc10 2024-09-16T01:36:43.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.878+0000 7fa8f37fe640 1 -- 172.21.15.88:0/1378176498 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8fc0074e0 con 0x7fa90c06bc10 2024-09-16T01:36:43.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.878+0000 7fa911027640 1 -- 172.21.15.88:0/1378176498 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa90c15e010 con 0x7fa90c06bc10 2024-09-16T01:36:43.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.878+0000 7fa8f37fe640 1 -- 172.21.15.88:0/1378176498 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc004a20 con 0x7fa90c06bc10 2024-09-16T01:36:43.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.878+0000 7fa911027640 1 -- 172.21.15.88:0/1378176498 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa90c1611c0 con 0x7fa90c06bc10 2024-09-16T01:36:43.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.879+0000 7fa911027640 1 -- 172.21.15.88:0/1378176498 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa90c15f9e0 con 0x7fa90c06bc10 2024-09-16T01:36:43.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.880+0000 7fa8f37fe640 1 -- 172.21.15.88:0/1378176498 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa8fc01b070 con 0x7fa90c06bc10 2024-09-16T01:36:43.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.880+0000 7fa8f37fe640 1 --2- 172.21.15.88:0/1378176498 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa8f806c2b0 0x7fa8f806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:43.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.880+0000 7fa8f37fe640 1 -- 172.21.15.88:0/1378176498 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7fa8fc08a640 con 0x7fa90c06bc10 2024-09-16T01:36:43.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.880+0000 7fa90a59c640 1 --2- 172.21.15.88:0/1378176498 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa8f806c2b0 0x7fa8f806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:43.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.880+0000 7fa90a59c640 1 --2- 172.21.15.88:0/1378176498 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa8f806c2b0 0x7fa8f806e770 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa900034000 tx=0x7fa90002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:43.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:43.884+0000 7fa8f37fe640 1 -- 172.21.15.88:0/1378176498 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa8fc0587f0 con 0x7fa90c06bc10 2024-09-16T01:36:44.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.114+0000 7fa911027640 1 -- 172.21.15.88:0/1378176498 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fa90c0b3860 con 0x7fa90c06bc10 2024-09-16T01:36:44.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.116+0000 7fa8f37fe640 1 -- 172.21.15.88:0/1378176498 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) ==== 190+0+147 (secure 0 0 0) 0x7fa8fc05caf0 con 0x7fa90c06bc10 2024-09-16T01:36:44.117 INFO:teuthology.orchestra.run.smithi088.stdout:[client.0] 2024-09-16T01:36:44.117 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDeiudmvPikIBAAyHxNNnwQKygxg09PNQLc/Q== 2024-09-16T01:36:44.117 INFO:teuthology.orchestra.run.smithi088.stdout: caps mds = "allow" 2024-09-16T01:36:44.117 INFO:teuthology.orchestra.run.smithi088.stdout: caps mon = "allow r" 2024-09-16T01:36:44.117 INFO:teuthology.orchestra.run.smithi088.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-16T01:36:44.118 INFO:teuthology.orchestra.run.smithi088.stderr:updated caps for client.0 2024-09-16T01:36:44.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.118+0000 7fa911027640 1 -- 172.21.15.88:0/1378176498 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa8f806c2b0 msgr2=0x7fa8f806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:44.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.118+0000 7fa911027640 1 --2- 172.21.15.88:0/1378176498 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa8f806c2b0 0x7fa8f806e770 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa900034000 tx=0x7fa90002e000 comp rx=0 tx=0).stop 2024-09-16T01:36:44.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.118+0000 7fa911027640 1 -- 172.21.15.88:0/1378176498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa90c06bc10 msgr2=0x7fa90c14b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:44.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.118+0000 7fa911027640 1 --2- 172.21.15.88:0/1378176498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa90c06bc10 0x7fa90c14b020 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc00c570 tx=0x7fa8fc00ca30 comp rx=0 tx=0).stop 2024-09-16T01:36:44.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.118+0000 7fa911027640 1 -- 172.21.15.88:0/1378176498 shutdown_connections 2024-09-16T01:36:44.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.118+0000 7fa911027640 1 --2- 172.21.15.88:0/1378176498 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa8f806c2b0 0x7fa8f806e770 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.118+0000 7fa911027640 1 --2- 172.21.15.88:0/1378176498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa90c15c2a0 0x7fa90c0ce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.118+0000 7fa911027640 1 --2- 172.21.15.88:0/1378176498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa90c06f710 0x7fa90c1619c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.118+0000 7fa911027640 1 --2- 172.21.15.88:0/1378176498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa90c06bc10 0x7fa90c14b020 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.118+0000 7fa911027640 1 -- 172.21.15.88:0/1378176498 >> 172.21.15.88:0/1378176498 conn(0x7fa90c0b3070 msgr2=0x7fa90c0d1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:44.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.119+0000 7fa911027640 1 -- 172.21.15.88:0/1378176498 shutdown_connections 2024-09-16T01:36:44.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.119+0000 7fa911027640 1 -- 172.21.15.88:0/1378176498 wait complete. 2024-09-16T01:36:44.131 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:36:44.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.227+0000 7f46ac123640 1 Processor -- start 2024-09-16T01:36:44.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.228+0000 7f46ac123640 1 -- start start 2024-09-16T01:36:44.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.228+0000 7f46ac123640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f46a40b6140 0x7f46a415c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:44.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.228+0000 7f46ac123640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46a40b50a0 con 0x7f46a40cfc80 2024-09-16T01:36:44.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.228+0000 7f46ac123640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46a40b5380 con 0x7f46a4155800 2024-09-16T01:36:44.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.228+0000 7f46ac123640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f46a40b6550 con 0x7f46a40b6140 2024-09-16T01:36:44.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.229+0000 7f46a96be640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f46a40cfc80 0x7f46a415c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54332/0 (socket says 172.21.15.88:54332) 2024-09-16T01:36:44.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.229+0000 7f46a96be640 1 -- 172.21.15.88:0/997971352 learned_addr learned my addr 172.21.15.88:0/997971352 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:44.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.229+0000 7f46aa6c0640 1 --2- 172.21.15.88:0/997971352 >> v2:172.21.15.88:3300/0 conn(0x7f46a40b6140 0x7f46a415c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:44.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.229+0000 7f46aa6c0640 1 -- 172.21.15.88:0/997971352 >> v1:172.21.15.154:6789/0 conn(0x7f46a4155800 legacy=0x7f46a40cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:44.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.229+0000 7f46aa6c0640 1 -- 172.21.15.88:0/997971352 >> v1:172.21.15.88:6789/0 conn(0x7f46a40cfc80 legacy=0x7f46a415c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:44.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.229+0000 7f46a8ebd640 1 -- 172.21.15.88:0/997971352 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1661851395 0 0) 0x7f46a40b50a0 con 0x7f46a40cfc80 2024-09-16T01:36:44.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.229+0000 7f46aa6c0640 1 -- 172.21.15.88:0/997971352 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a40b5660 con 0x7f46a40b6140 2024-09-16T01:36:44.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.230+0000 7f46aa6c0640 1 --2- 172.21.15.88:0/997971352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46a40b6140 0x7f46a415c330 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f4694009d90 tx=0x7f469402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3927bf175c470138 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:44.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.230+0000 7f46a8ebd640 1 -- 172.21.15.88:0/997971352 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4694007740 con 0x7f46a40b6140 2024-09-16T01:36:44.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.230+0000 7f46a8ebd640 1 -- 172.21.15.88:0/997971352 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46940078e0 con 0x7f46a40b6140 2024-09-16T01:36:44.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.230+0000 7f46a8ebd640 1 -- 172.21.15.88:0/997971352 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4694007cb0 con 0x7f46a40b6140 2024-09-16T01:36:44.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.230+0000 7f46ac123640 1 -- 172.21.15.88:0/997971352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46a40b6140 msgr2=0x7f46a415c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:44.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.230+0000 7f46ac123640 1 --2- 172.21.15.88:0/997971352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46a40b6140 0x7f46a415c330 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f4694009d90 tx=0x7f469402a6f0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.230+0000 7f46ac123640 1 -- 172.21.15.88:0/997971352 shutdown_connections 2024-09-16T01:36:44.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.230+0000 7f46ac123640 1 --2- 172.21.15.88:0/997971352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46a40b6140 0x7f46a415c330 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.230+0000 7f46ac123640 1 -- 172.21.15.88:0/997971352 >> 172.21.15.88:0/997971352 conn(0x7f46a406bc10 msgr2=0x7f46a406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:44.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.230+0000 7f46ac123640 1 -- 172.21.15.88:0/997971352 shutdown_connections 2024-09-16T01:36:44.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.230+0000 7f46ac123640 1 -- 172.21.15.88:0/997971352 wait complete. 2024-09-16T01:36:44.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.231+0000 7f46ac123640 1 Processor -- start 2024-09-16T01:36:44.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.232+0000 7f46ac123640 1 -- start start 2024-09-16T01:36:44.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.232+0000 7f46ac123640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46a40b6140 0x7f46a414f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:44.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.232+0000 7f46ac123640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46a40cfc80 0x7f46a414e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:44.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.232+0000 7f46ac123640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f46a4155800 0x7f46a4071850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:44.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.232+0000 7f46ac123640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f46a4155be0 con 0x7f46a40b6140 2024-09-16T01:36:44.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.232+0000 7f46ac123640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f46a40b5c20 con 0x7f46a40cfc80 2024-09-16T01:36:44.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.232+0000 7f46ac123640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f46a40b4920 con 0x7f46a4155800 2024-09-16T01:36:44.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.232+0000 7f46a9ebf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46a40b6140 0x7f46a414f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:44.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.232+0000 7f46a9ebf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46a40b6140 0x7f46a414f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41394/0 (socket says 172.21.15.88:41394) 2024-09-16T01:36:44.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.232+0000 7f46a9ebf640 1 -- 172.21.15.88:0/3035080941 learned_addr learned my addr 172.21.15.88:0/3035080941 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:44.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.233+0000 7f46a96be640 1 --2- 172.21.15.88:0/3035080941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46a40cfc80 0x7f46a414e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:44.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.233+0000 7f46aa6c0640 1 --2- 172.21.15.88:0/3035080941 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f46a4155800 0x7f46a4071850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:44.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.233+0000 7f46a9ebf640 1 -- 172.21.15.88:0/3035080941 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f46a4155800 msgr2=0x7f46a4071850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:44.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.233+0000 7f46a9ebf640 1 --2- 172.21.15.88:0/3035080941 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f46a4155800 0x7f46a4071850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.233+0000 7f46a9ebf640 1 -- 172.21.15.88:0/3035080941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46a40cfc80 msgr2=0x7f46a414e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:44.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.233+0000 7f46a9ebf640 1 --2- 172.21.15.88:0/3035080941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46a40cfc80 0x7f46a414e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.233+0000 7f46a9ebf640 1 -- 172.21.15.88:0/3035080941 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a414e640 con 0x7f46a40b6140 2024-09-16T01:36:44.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.233+0000 7f46a9ebf640 1 --2- 172.21.15.88:0/3035080941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46a40b6140 0x7f46a414f870 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f469c00c400 tx=0x7f469c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:44.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.233+0000 7f46a27fc640 1 -- 172.21.15.88:0/3035080941 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469c017070 con 0x7f46a40b6140 2024-09-16T01:36:44.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.233+0000 7f46a27fc640 1 -- 172.21.15.88:0/3035080941 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f469c0052d0 con 0x7f46a40b6140 2024-09-16T01:36:44.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.233+0000 7f46ac123640 1 -- 172.21.15.88:0/3035080941 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46a4151590 con 0x7f46a40b6140 2024-09-16T01:36:44.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.233+0000 7f46a27fc640 1 -- 172.21.15.88:0/3035080941 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469c003830 con 0x7f46a40b6140 2024-09-16T01:36:44.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.233+0000 7f46ac123640 1 -- 172.21.15.88:0/3035080941 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46a414fe50 con 0x7f46a40b6140 2024-09-16T01:36:44.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.234+0000 7f46ac123640 1 -- 172.21.15.88:0/3035080941 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46a40b4cb0 con 0x7f46a40b6140 2024-09-16T01:36:44.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.234+0000 7f46a27fc640 1 -- 172.21.15.88:0/3035080941 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f469c01a070 con 0x7f46a40b6140 2024-09-16T01:36:44.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.235+0000 7f46a27fc640 1 --2- 172.21.15.88:0/3035080941 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f469006c2b0 0x7f469006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:44.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.235+0000 7f46a27fc640 1 -- 172.21.15.88:0/3035080941 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f469c08a7c0 con 0x7f46a40b6140 2024-09-16T01:36:44.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.235+0000 7f46a96be640 1 --2- 172.21.15.88:0/3035080941 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f469006c2b0 0x7f469006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:44.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.235+0000 7f46a96be640 1 --2- 172.21.15.88:0/3035080941 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f469006c2b0 0x7f469006e770 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f46a41510f0 tx=0x7f4698007bb0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:44.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.239+0000 7f46a27fc640 1 -- 172.21.15.88:0/3035080941 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f46a414e640 con 0x7f46a40b6140 2024-09-16T01:36:44.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.439+0000 7f46ac123640 1 -- 172.21.15.88:0/3035080941 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46a415ab40 con 0x7f46a40b6140 2024-09-16T01:36:44.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.439+0000 7f46a27fc640 1 -- 172.21.15.88:0/3035080941 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5220 (secure 0 0 0) 0x7f469c058f30 con 0x7f46a40b6140 2024-09-16T01:36:44.441 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:44.441 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":37,"btime":"2024-09-16T01:36:42:461977+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/581241194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":581241194},{"type":"v1","addr":"172.21.15.154:6834","nonce":581241194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2104508521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2104508521},{"type":"v1","addr":"172.21.15.154:6835","nonce":2104508521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5859,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/998283126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":998283126},{"type":"v1","addr":"172.21.15.88:6836","nonce":998283126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:39.786438+0000","modified":"2024-09-16T01:36:41.790966+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5862},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5862":{"gid":5862,"name":"a","rank":0,"incarnation":35,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6837/574169866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":574169866},{"type":"v1","addr":"172.21.15.88:6837","nonce":574169866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5862,"qdb_cluster":[5862]},"id":4}]} 2024-09-16T01:36:44.441 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 37 2024-09-16T01:36:44.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.441+0000 7f46ac123640 1 -- 172.21.15.88:0/3035080941 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f469006c2b0 msgr2=0x7f469006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:44.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.441+0000 7f46ac123640 1 --2- 172.21.15.88:0/3035080941 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f469006c2b0 0x7f469006e770 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f46a41510f0 tx=0x7f4698007bb0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.441+0000 7f46ac123640 1 -- 172.21.15.88:0/3035080941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46a40b6140 msgr2=0x7f46a414f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:44.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.441+0000 7f46ac123640 1 --2- 172.21.15.88:0/3035080941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46a40b6140 0x7f46a414f870 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f469c00c400 tx=0x7f469c00c8c0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.441+0000 7f46ac123640 1 -- 172.21.15.88:0/3035080941 shutdown_connections 2024-09-16T01:36:44.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.441+0000 7f46ac123640 1 --2- 172.21.15.88:0/3035080941 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f469006c2b0 0x7f469006e770 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.441+0000 7f46ac123640 1 --2- 172.21.15.88:0/3035080941 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f46a4155800 0x7f46a4071850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.441+0000 7f46ac123640 1 --2- 172.21.15.88:0/3035080941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46a40cfc80 0x7f46a414e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.441+0000 7f46ac123640 1 --2- 172.21.15.88:0/3035080941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46a40b6140 0x7f46a414f870 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.441+0000 7f46ac123640 1 -- 172.21.15.88:0/3035080941 >> 172.21.15.88:0/3035080941 conn(0x7f46a406bc10 msgr2=0x7f46a406d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:44.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.441+0000 7f46ac123640 1 -- 172.21.15.88:0/3035080941 shutdown_connections 2024-09-16T01:36:44.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.441+0000 7f46ac123640 1 -- 172.21.15.88:0/3035080941 wait complete. 2024-09-16T01:36:44.857 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 37, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:36:39.786438+0000', 'modified': '2024-09-16T01:36:41.790966+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5862}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5862': {'gid': 5862, 'name': 'a', 'rank': 0, 'incarnation': 35, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.88:6837/574169866', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6835', 'nonce': 574169866}, {'type': 'v1', 'addr': '172.21.15.88:6837', 'nonce': 574169866}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5862, 'qdb_cluster': [5862]} 2024-09-16T01:36:44.857 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T01:36:44.858 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-16T01:36:44.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.960+0000 7f9c06a7b640 1 Processor -- start 2024-09-16T01:36:44.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.960+0000 7f9c06a7b640 1 -- start start 2024-09-16T01:36:44.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.961+0000 7f9c06a7b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c0015b480 0x7f9c000b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:44.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.961+0000 7f9c06a7b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9c000b60c0 0x7f9c00071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:44.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.961+0000 7f9c06a7b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c000ccff0 con 0x7f9c00072530 2024-09-16T01:36:44.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.961+0000 7f9c06a7b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9c000b6500 con 0x7f9c000b60c0 2024-09-16T01:36:44.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.961+0000 7f9c06a7b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9c00072990 con 0x7f9c0015b480 2024-09-16T01:36:44.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.961+0000 7f9c05018640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9c00072530 0x7f9c00070fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54340/0 (socket says 172.21.15.88:54340) 2024-09-16T01:36:44.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.961+0000 7f9c05018640 1 -- 172.21.15.88:0/592319095 learned_addr learned my addr 172.21.15.88:0/592319095 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:44.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.961+0000 7f9bff7fe640 1 -- 172.21.15.88:0/592319095 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2666795349 0 0) 0x7f9c000ccff0 con 0x7f9c00072530 2024-09-16T01:36:44.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.961+0000 7f9c04817640 1 --2- 172.21.15.88:0/592319095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c0015b480 0x7f9c000b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:44.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.961+0000 7f9bff7fe640 1 -- 172.21.15.88:0/592319095 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9be8003520 con 0x7f9c00072530 2024-09-16T01:36:44.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.961+0000 7f9bfffff640 1 --2- 172.21.15.88:0/592319095 >> v2:172.21.15.154:3300/0 conn(0x7f9c000b60c0 0x7f9c00071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:44.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.962+0000 7f9bff7fe640 1 -- 172.21.15.88:0/592319095 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3184432847 0 0) 0x7f9be8003520 con 0x7f9c00072530 2024-09-16T01:36:44.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.962+0000 7f9bff7fe640 1 -- 172.21.15.88:0/592319095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c0015b480 msgr2=0x7f9c000b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:44.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.962+0000 7f9bff7fe640 1 --2- 172.21.15.88:0/592319095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c0015b480 0x7f9c000b5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.962+0000 7f9bff7fe640 1 -- 172.21.15.88:0/592319095 >> v2:172.21.15.154:3300/0 conn(0x7f9c000b60c0 msgr2=0x7f9c00071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:44.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.962+0000 7f9bff7fe640 1 --2- 172.21.15.88:0/592319095 >> v2:172.21.15.154:3300/0 conn(0x7f9c000b60c0 0x7f9c00071ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.962+0000 7f9bff7fe640 1 -- 172.21.15.88:0/592319095 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c000cd2d0 con 0x7f9c00072530 2024-09-16T01:36:44.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.962+0000 7f9bff7fe640 1 -- 172.21.15.88:0/592319095 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f9bf80019f0 con 0x7f9c00072530 2024-09-16T01:36:44.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.962+0000 7f9bff7fe640 1 -- 172.21.15.88:0/592319095 >> v1:172.21.15.88:6789/0 conn(0x7f9c00072530 legacy=0x7f9c00070fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:44.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.962+0000 7f9bff7fe640 1 --2- 172.21.15.88:0/592319095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9be80039b0 0x7f9be8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:44.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.962+0000 7f9bff7fe640 1 --2- 172.21.15.88:0/592319095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9be8004800 0x7f9be8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:44.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.962+0000 7f9bff7fe640 1 --2- 172.21.15.88:0/592319095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c0015b480 0x7f9be80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:44.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.962+0000 7f9bff7fe640 1 -- 172.21.15.88:0/592319095 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9c000b6500 con 0x7f9be80039b0 2024-09-16T01:36:44.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.962+0000 7f9bff7fe640 1 -- 172.21.15.88:0/592319095 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9be800b9f0 con 0x7f9be8004800 2024-09-16T01:36:44.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.962+0000 7f9bff7fe640 1 -- 172.21.15.88:0/592319095 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9be800bbf0 con 0x7f9c0015b480 2024-09-16T01:36:44.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.962+0000 7f9c04817640 1 --2- 172.21.15.88:0/592319095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9be80039b0 0x7f9be8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:44.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.962+0000 7f9bfffff640 1 --2- 172.21.15.88:0/592319095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9be8004800 0x7f9be8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:44.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.962+0000 7f9c05018640 1 --2- 172.21.15.88:0/592319095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c0015b480 0x7f9be80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:44.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.962+0000 7f9c04817640 1 -- 172.21.15.88:0/592319095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c0015b480 msgr2=0x7f9be80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:44.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.962+0000 7f9c04817640 1 --2- 172.21.15.88:0/592319095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c0015b480 0x7f9be80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.962+0000 7f9c04817640 1 -- 172.21.15.88:0/592319095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9be8004800 msgr2=0x7f9be8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:44.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.962+0000 7f9c04817640 1 --2- 172.21.15.88:0/592319095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9be8004800 0x7f9be8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.962+0000 7f9c04817640 1 -- 172.21.15.88:0/592319095 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9be8003520 con 0x7f9be80039b0 2024-09-16T01:36:44.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.963+0000 7f9c04817640 1 --2- 172.21.15.88:0/592319095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9be80039b0 0x7f9be8003dd0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f9bf000ba00 tx=0x7f9be8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e16355a9fb907a78 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:44.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.963+0000 7f9bff7fe640 1 -- 172.21.15.88:0/592319095 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf0030070 con 0x7f9be80039b0 2024-09-16T01:36:44.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.963+0000 7f9bff7fe640 1 -- 172.21.15.88:0/592319095 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bf002acc0 con 0x7f9be80039b0 2024-09-16T01:36:44.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.963+0000 7f9bff7fe640 1 -- 172.21.15.88:0/592319095 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf000b600 con 0x7f9be80039b0 2024-09-16T01:36:44.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.963+0000 7f9c06a7b640 1 -- 172.21.15.88:0/592319095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9be80039b0 msgr2=0x7f9be8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:44.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.963+0000 7f9c06a7b640 1 --2- 172.21.15.88:0/592319095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9be80039b0 0x7f9be8003dd0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f9bf000ba00 tx=0x7f9be8005830 comp rx=0 tx=0).stop 2024-09-16T01:36:44.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.963+0000 7f9c05018640 1 -- 172.21.15.88:0/592319095 reap_dead start 2024-09-16T01:36:44.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.963+0000 7f9c06a7b640 1 -- 172.21.15.88:0/592319095 shutdown_connections 2024-09-16T01:36:44.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.963+0000 7f9c06a7b640 1 -- 172.21.15.88:0/592319095 >> 172.21.15.88:0/592319095 conn(0x7f9c0006d0f0 msgr2=0x7f9c0006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:44.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.964+0000 7f9c06a7b640 1 -- 172.21.15.88:0/592319095 shutdown_connections 2024-09-16T01:36:44.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.964+0000 7f9c06a7b640 1 -- 172.21.15.88:0/592319095 wait complete. 2024-09-16T01:36:44.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.965+0000 7f9c06a7b640 1 Processor -- start 2024-09-16T01:36:44.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.965+0000 7f9c06a7b640 1 -- start start 2024-09-16T01:36:44.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.966+0000 7f9c06a7b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9be80039b0 0x7f9c0014e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:44.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.966+0000 7f9c06a7b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c000cc1d0 0x7f9c00151620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:44.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.966+0000 7f9c06a7b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c000cc650 0x7f9c00151b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:44.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.966+0000 7f9c06a7b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9c000cdb70 con 0x7f9c000cc650 2024-09-16T01:36:44.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.966+0000 7f9c06a7b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9c000cd890 con 0x7f9c000cc1d0 2024-09-16T01:36:44.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.966+0000 7f9c06a7b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9c0015b860 con 0x7f9be80039b0 2024-09-16T01:36:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.966+0000 7f9c05018640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c000cc650 0x7f9c00151b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.966+0000 7f9c05018640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c000cc650 0x7f9c00151b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41410/0 (socket says 172.21.15.88:41410) 2024-09-16T01:36:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.966+0000 7f9c05018640 1 -- 172.21.15.88:0/105627439 learned_addr learned my addr 172.21.15.88:0/105627439 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.966+0000 7f9bfffff640 1 --2- 172.21.15.88:0/105627439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c000cc1d0 0x7f9c00151620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.966+0000 7f9c04817640 1 --2- 172.21.15.88:0/105627439 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9be80039b0 0x7f9c0014e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.966+0000 7f9c05018640 1 -- 172.21.15.88:0/105627439 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9be80039b0 msgr2=0x7f9c0014e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.966+0000 7f9c05018640 1 --2- 172.21.15.88:0/105627439 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9be80039b0 0x7f9c0014e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.966+0000 7f9c05018640 1 -- 172.21.15.88:0/105627439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c000cc1d0 msgr2=0x7f9c00151620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.966+0000 7f9c05018640 1 --2- 172.21.15.88:0/105627439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c000cc1d0 0x7f9c00151620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.966+0000 7f9c05018640 1 -- 172.21.15.88:0/105627439 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c001f7c30 con 0x7f9c000cc650 2024-09-16T01:36:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.966+0000 7f9c05018640 1 --2- 172.21.15.88:0/105627439 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c000cc650 0x7f9c00151b60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f9c000694d0 tx=0x7f9bf80079d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.966+0000 7f9bfd7fa640 1 -- 172.21.15.88:0/105627439 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf8017020 con 0x7f9c000cc650 2024-09-16T01:36:44.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.966+0000 7f9bfd7fa640 1 -- 172.21.15.88:0/105627439 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bf800aac0 con 0x7f9c000cc650 2024-09-16T01:36:44.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.966+0000 7f9c06a7b640 1 -- 172.21.15.88:0/105627439 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c001f7f10 con 0x7f9c000cc650 2024-09-16T01:36:44.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.966+0000 7f9bfd7fa640 1 -- 172.21.15.88:0/105627439 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf80133f0 con 0x7f9c000cc650 2024-09-16T01:36:44.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.966+0000 7f9c06a7b640 1 -- 172.21.15.88:0/105627439 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c001f84d0 con 0x7f9c000cc650 2024-09-16T01:36:44.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.968+0000 7f9bfd7fa640 1 -- 172.21.15.88:0/105627439 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9bf801a020 con 0x7f9c000cc650 2024-09-16T01:36:44.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.968+0000 7f9bfd7fa640 1 --2- 172.21.15.88:0/105627439 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9bec06c2b0 0x7f9bec06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:44.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.968+0000 7f9bfd7fa640 1 -- 172.21.15.88:0/105627439 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f9bf808bb00 con 0x7f9c000cc650 2024-09-16T01:36:44.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.968+0000 7f9c04817640 1 --2- 172.21.15.88:0/105627439 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9bec06c2b0 0x7f9bec06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:44.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.969+0000 7f9c04817640 1 --2- 172.21.15.88:0/105627439 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9bec06c2b0 0x7f9bec06e770 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9be80057e0 tx=0x7f9bf0030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:44.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.969+0000 7f9c09818740 1 Processor -- start 2024-09-16T01:36:44.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.969+0000 7f9c09818740 1 -- start start 2024-09-16T01:36:44.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.970+0000 7f9c09818740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646e2f382c0 0x5646e2f7da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:44.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.970+0000 7f9c09818740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646e2f35a80 0x5646e2f386a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:44.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.970+0000 7f9c09818740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 0x5646e2f328b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:44.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.970+0000 7f9c09818740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5646e2e9a520 con 0x5646e2f39a50 2024-09-16T01:36:44.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.970+0000 7f9c09818740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5646e2e423e0 con 0x5646e2f35a80 2024-09-16T01:36:44.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.970+0000 7f9c09818740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5646e2e964e0 con 0x5646e2f382c0 2024-09-16T01:36:44.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.970+0000 7f9c05018640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 0x5646e2f328b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:44.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.970+0000 7f9c05018640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 0x5646e2f328b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41426/0 (socket says 172.21.15.88:41426) 2024-09-16T01:36:44.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.970+0000 7f9c05018640 1 -- 172.21.15.88:0/158348876 learned_addr learned my addr 172.21.15.88:0/158348876 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:44.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.970+0000 7f9bfffff640 1 --2- 172.21.15.88:0/158348876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646e2f35a80 0x5646e2f386a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:44.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.970+0000 7f9c04817640 1 --2- 172.21.15.88:0/158348876 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646e2f382c0 0x5646e2f7da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:44.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.970+0000 7f9c05018640 1 -- 172.21.15.88:0/158348876 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646e2f382c0 msgr2=0x5646e2f7da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:44.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.970+0000 7f9c05018640 1 --2- 172.21.15.88:0/158348876 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646e2f382c0 0x5646e2f7da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.970+0000 7f9c05018640 1 -- 172.21.15.88:0/158348876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646e2f35a80 msgr2=0x5646e2f386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:44.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.970+0000 7f9c05018640 1 --2- 172.21.15.88:0/158348876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646e2f35a80 0x5646e2f386a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.970+0000 7f9c05018640 1 -- 172.21.15.88:0/158348876 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646e2e5d2f0 con 0x5646e2f39a50 2024-09-16T01:36:44.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.970+0000 7f9bfffff640 1 --2- 172.21.15.88:0/158348876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646e2f35a80 0x5646e2f386a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:44.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.970+0000 7f9c05018640 1 --2- 172.21.15.88:0/158348876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 0x5646e2f328b0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f9bf805a2e0 tx=0x7f9bf805a3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=796fa930bddcbff9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:44.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.970+0000 7f9be67fc640 1 -- 172.21.15.88:0/158348876 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf8088070 con 0x5646e2f39a50 2024-09-16T01:36:44.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.970+0000 7f9be67fc640 1 -- 172.21.15.88:0/158348876 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bf8062740 con 0x5646e2f39a50 2024-09-16T01:36:44.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.970+0000 7f9be67fc640 1 -- 172.21.15.88:0/158348876 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf80561d0 con 0x5646e2f39a50 2024-09-16T01:36:44.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.971+0000 7f9c09818740 1 -- 172.21.15.88:0/158348876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 msgr2=0x5646e2f328b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:44.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.971+0000 7f9c09818740 1 --2- 172.21.15.88:0/158348876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 0x5646e2f328b0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f9bf805a2e0 tx=0x7f9bf805a3d0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.971+0000 7f9c09818740 1 -- 172.21.15.88:0/158348876 shutdown_connections 2024-09-16T01:36:44.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.971+0000 7f9c09818740 1 --2- 172.21.15.88:0/158348876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 0x5646e2f328b0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.971+0000 7f9c09818740 1 --2- 172.21.15.88:0/158348876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646e2f35a80 0x5646e2f386a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.971+0000 7f9c09818740 1 --2- 172.21.15.88:0/158348876 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646e2f382c0 0x5646e2f7da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.971+0000 7f9c09818740 1 -- 172.21.15.88:0/158348876 >> 172.21.15.88:0/158348876 conn(0x5646e2f34a80 msgr2=0x5646e2f7cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:44.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.971+0000 7f9c09818740 1 -- 172.21.15.88:0/158348876 shutdown_connections 2024-09-16T01:36:44.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.971+0000 7f9c09818740 1 -- 172.21.15.88:0/158348876 wait complete. 2024-09-16T01:36:44.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.972+0000 7f9c09818740 1 -- start start 2024-09-16T01:36:44.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.972+0000 7f9c09818740 1 Processor -- start 2024-09-16T01:36:44.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.973+0000 7f9c09818740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646e2f382c0 0x5646e2f56a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:44.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.973+0000 7f9c09818740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 0x5646e2f57bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:44.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.973+0000 7f9c09818740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646e2f5a6c0 0x5646e2f561b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:44.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.973+0000 7f9c09818740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5646e2f33f20 con 0x5646e2f39a50 2024-09-16T01:36:44.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.973+0000 7f9c09818740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5646e2e23450 con 0x5646e2f5a6c0 2024-09-16T01:36:44.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.973+0000 7f9c09818740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5646e2f7e290 con 0x5646e2f382c0 2024-09-16T01:36:44.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.973+0000 7f9c09818740 10 client.0 _ll_register_callbacks cb 0x5646e2bacac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:36:44.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.973+0000 7f9bfffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 0x5646e2f57bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:44.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.973+0000 7f9bfffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 0x5646e2f57bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41436/0 (socket says 172.21.15.88:41436) 2024-09-16T01:36:44.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.973+0000 7f9bfffff640 1 -- 172.21.15.88:0/4060305015 learned_addr learned my addr 172.21.15.88:0/4060305015 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:44.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.973+0000 7f9c04817640 1 --2- 172.21.15.88:0/4060305015 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646e2f382c0 0x5646e2f56a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:44.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.973+0000 7f9c05018640 1 --2- 172.21.15.88:0/4060305015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646e2f5a6c0 0x5646e2f561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:44.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.973+0000 7f9bfffff640 1 -- 172.21.15.88:0/4060305015 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646e2f382c0 msgr2=0x5646e2f56a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:44.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.973+0000 7f9bfffff640 1 --2- 172.21.15.88:0/4060305015 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646e2f382c0 0x5646e2f56a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.973+0000 7f9bfffff640 1 -- 172.21.15.88:0/4060305015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646e2f5a6c0 msgr2=0x5646e2f561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:44.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.973+0000 7f9bfffff640 1 --2- 172.21.15.88:0/4060305015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646e2f5a6c0 0x5646e2f561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.973+0000 7f9bfffff640 1 -- 172.21.15.88:0/4060305015 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646e2bad2d0 con 0x5646e2f39a50 2024-09-16T01:36:44.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.973+0000 7f9bfffff640 1 --2- 172.21.15.88:0/4060305015 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 0x5646e2f57bd0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f9bf40177a0 tx=0x7f9bf4017c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e0187f0f50305b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:44.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.973+0000 7f9bdf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:36:44.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.973+0000 7f9bdf7fe640 1 -- 172.21.15.88:0/4060305015 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf4024060 con 0x5646e2f39a50 2024-09-16T01:36:44.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.973+0000 7f9bdf7fe640 1 -- 172.21.15.88:0/4060305015 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f9bf4015bb0 con 0x5646e2f39a50 2024-09-16T01:36:44.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.973+0000 7f9c09818740 1 -- 172.21.15.88:0/4060305015 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bf8062740 con 0x5646e2f39a50 2024-09-16T01:36:44.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.973+0000 7f9bdf7fe640 1 -- 172.21.15.88:0/4060305015 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9bf402a050 con 0x5646e2f39a50 2024-09-16T01:36:44.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.973+0000 7f9c09818740 10 client.5991 fetch_fsmap learned FSMap version 37 2024-09-16T01:36:44.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.973+0000 7f9c09818740 1 -- 172.21.15.88:0/4060305015 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) -- 0x5646e2c776b0 con 0x5646e2f39a50 2024-09-16T01:36:44.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.974+0000 7f9bdf7fe640 1 -- 172.21.15.88:0/4060305015 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bf8062740 con 0x5646e2f39a50 2024-09-16T01:36:44.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.974+0000 7f9bdf7fe640 1 -- 172.21.15.88:0/4060305015 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 37) ==== 3502+0+0 (secure 0 0 0) 0x7f9bf402d020 con 0x5646e2f39a50 2024-09-16T01:36:44.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.974+0000 7f9c09818740 10 client.5991 fetch_fsmap finished waiting for FSMap version 37 2024-09-16T01:36:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.974+0000 7f9c09818740 10 client.5991 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:36:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.974+0000 7f9c09818740 20 client.5991 populate_metadata read hostname 'smithi088' 2024-09-16T01:36:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.974+0000 7f9c09818740 1 --2- 172.21.15.88:0/4060305015 >> [v2:172.21.15.88:6835/574169866,v1:172.21.15.88:6837/574169866] conn(0x5646e2f60760 0x5646e2f317d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.974+0000 7f9c09818740 4 client.5991 mds_command: new command op to 5862 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:36:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.974+0000 7f9c09818740 1 -- 172.21.15.88:0/4060305015 --> [v2:172.21.15.88:6835/574169866,v1:172.21.15.88:6837/574169866] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5646e2ea7c50 con 0x5646e2f60760 2024-09-16T01:36:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.974+0000 7f9c04817640 1 --2- 172.21.15.88:0/4060305015 >> [v2:172.21.15.88:6835/574169866,v1:172.21.15.88:6837/574169866] conn(0x5646e2f60760 0x5646e2f317d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.974+0000 7f9c04817640 1 --2- 172.21.15.88:0/4060305015 >> [v2:172.21.15.88:6835/574169866,v1:172.21.15.88:6837/574169866] conn(0x5646e2f60760 0x5646e2f317d0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98dfe4c142c98cca server_cookie=e578d7af2b526eb in_seq=0 out_seq=0 2024-09-16T01:36:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.974+0000 7f9bdf7fe640 10 client.5991 ms_handle_connect on v2:172.21.15.88:6835/574169866 2024-09-16T01:36:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.975+0000 7f9bdf7fe640 1 -- 172.21.15.88:0/4060305015 <== mds.0 v2:172.21.15.88:6835/574169866 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5646e2ea7c50 con 0x5646e2f60760 2024-09-16T01:36:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.975+0000 7f9bdf7fe640 10 client.5991 handle_command_reply: tid=0 2024-09-16T01:36:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.975+0000 7f9c09818740 1 client.5991 shutdown 2024-09-16T01:36:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.975+0000 7f9c09818740 1 -- 172.21.15.88:0/4060305015 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 msgr2=0x5646e2f57bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.975+0000 7f9c09818740 1 --2- 172.21.15.88:0/4060305015 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 0x5646e2f57bd0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f9bf40177a0 tx=0x7f9bf4017c60 comp rx=0 tx=0).stop 2024-09-16T01:36:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.975+0000 7f9c09818740 1 -- 172.21.15.88:0/4060305015 shutdown_connections 2024-09-16T01:36:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.975+0000 7f9c09818740 1 --2- 172.21.15.88:0/4060305015 >> [v2:172.21.15.88:6835/574169866,v1:172.21.15.88:6837/574169866] conn(0x5646e2f60760 0x5646e2f317d0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.975+0000 7f9c09818740 1 --2- 172.21.15.88:0/4060305015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646e2f5a6c0 0x5646e2f561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.975+0000 7f9c09818740 1 --2- 172.21.15.88:0/4060305015 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 0x5646e2f57bd0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.975+0000 7f9bdf7fe640 0 client.5991 ms_handle_reset on v2:172.21.15.88:6835/574169866 2024-09-16T01:36:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.975+0000 7f9c09818740 1 --2- 172.21.15.88:0/4060305015 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646e2f382c0 0x5646e2f56a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.975+0000 7f9c09818740 1 -- 172.21.15.88:0/4060305015 >> 172.21.15.88:0/4060305015 conn(0x5646e2f35a80 msgr2=0x5646e2f7afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.975+0000 7f9c09818740 1 -- 172.21.15.88:0/4060305015 shutdown_connections 2024-09-16T01:36:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.975+0000 7f9c09818740 1 -- 172.21.15.88:0/4060305015 wait complete. 2024-09-16T01:36:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.975+0000 7f9c09818740 20 client.5991 trim_cache size 0 max 16384 2024-09-16T01:36:44.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.993+0000 7f9c09818740 1 Processor -- start 2024-09-16T01:36:44.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.993+0000 7f9c09818740 1 -- start start 2024-09-16T01:36:44.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.994+0000 7f9c09818740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646e2f382c0 0x5646e2f7ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:44.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.994+0000 7f9c09818740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 0x5646e2f7a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:44.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.994+0000 7f9c09818740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646e2f5a6c0 0x5646e2f61710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:44.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.994+0000 7f9c09818740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9c00200100 con 0x5646e2f39a50 2024-09-16T01:36:44.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.994+0000 7f9c09818740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9c000d0680 con 0x5646e2f382c0 2024-09-16T01:36:44.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.994+0000 7f9c09818740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9c000d0cd0 con 0x5646e2f5a6c0 2024-09-16T01:36:44.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.994+0000 7f9bfffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 0x5646e2f7a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:44.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.994+0000 7f9bfffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 0x5646e2f7a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41450/0 (socket says 172.21.15.88:41450) 2024-09-16T01:36:44.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.994+0000 7f9bfffff640 1 -- 172.21.15.88:0/4250941887 learned_addr learned my addr 172.21.15.88:0/4250941887 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:44.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.994+0000 7f9bfffff640 1 -- 172.21.15.88:0/4250941887 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646e2f5a6c0 msgr2=0x5646e2f61710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:44.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.994+0000 7f9bfffff640 1 --2- 172.21.15.88:0/4250941887 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646e2f5a6c0 0x5646e2f61710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.994+0000 7f9bfffff640 1 -- 172.21.15.88:0/4250941887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646e2f382c0 msgr2=0x5646e2f7ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:44.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.994+0000 7f9bfffff640 1 --2- 172.21.15.88:0/4250941887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646e2f382c0 0x5646e2f7ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.994+0000 7f9bfffff640 1 -- 172.21.15.88:0/4250941887 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646e2e9c9c0 con 0x5646e2f39a50 2024-09-16T01:36:44.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.994+0000 7f9bfffff640 1 --2- 172.21.15.88:0/4250941887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 0x5646e2f7a430 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f9bf4015e00 tx=0x7f9bf401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=117cab84d9cef709 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:44.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.994+0000 7f9be5ffb640 1 -- 172.21.15.88:0/4250941887 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf4024030 con 0x5646e2f39a50 2024-09-16T01:36:44.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.994+0000 7f9be5ffb640 1 -- 172.21.15.88:0/4250941887 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bf4017e50 con 0x5646e2f39a50 2024-09-16T01:36:44.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.994+0000 7f9be5ffb640 1 -- 172.21.15.88:0/4250941887 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf4004270 con 0x5646e2f39a50 2024-09-16T01:36:44.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.995+0000 7f9c09818740 1 -- 172.21.15.88:0/4250941887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 msgr2=0x5646e2f7a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:44.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.995+0000 7f9c09818740 1 --2- 172.21.15.88:0/4250941887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 0x5646e2f7a430 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f9bf4015e00 tx=0x7f9bf401a000 comp rx=0 tx=0).stop 2024-09-16T01:36:44.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.995+0000 7f9c09818740 1 -- 172.21.15.88:0/4250941887 shutdown_connections 2024-09-16T01:36:44.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.995+0000 7f9c09818740 1 --2- 172.21.15.88:0/4250941887 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646e2f5a6c0 0x5646e2f61710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.995+0000 7f9c09818740 1 --2- 172.21.15.88:0/4250941887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 0x5646e2f7a430 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.995+0000 7f9c09818740 1 --2- 172.21.15.88:0/4250941887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646e2f382c0 0x5646e2f7ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:44.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.995+0000 7f9c09818740 1 -- 172.21.15.88:0/4250941887 >> 172.21.15.88:0/4250941887 conn(0x5646e2f35a80 msgr2=0x5646e2f57600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:44.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.995+0000 7f9c09818740 1 -- 172.21.15.88:0/4250941887 shutdown_connections 2024-09-16T01:36:44.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.995+0000 7f9c09818740 1 -- 172.21.15.88:0/4250941887 wait complete. 2024-09-16T01:36:44.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.995+0000 7f9c09818740 1 -- start start 2024-09-16T01:36:45.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.996+0000 7f9c09818740 1 Processor -- start 2024-09-16T01:36:45.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.996+0000 7f9c09818740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646e2f382c0 0x5646e2fe16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:45.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.996+0000 7f9c09818740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 0x5646e2fe45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:45.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.996+0000 7f9c09818740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646e2f5a6c0 0x5646e2f7afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:45.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.996+0000 7f9c09818740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5646e2f56880 con 0x5646e2f39a50 2024-09-16T01:36:45.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.996+0000 7f9c09818740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5646e2f60cf0 con 0x5646e2f5a6c0 2024-09-16T01:36:45.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.996+0000 7f9c09818740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5646e2f594d0 con 0x5646e2f382c0 2024-09-16T01:36:45.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.996+0000 7f9c09818740 10 client.0 _ll_register_callbacks cb 0x7f9bf4009db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:36:45.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.996+0000 7f9bfffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 0x5646e2fe45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:45.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.996+0000 7f9bfffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 0x5646e2fe45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41454/0 (socket says 172.21.15.88:41454) 2024-09-16T01:36:45.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.996+0000 7f9bfffff640 1 -- 172.21.15.88:0/1771497269 learned_addr learned my addr 172.21.15.88:0/1771497269 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:45.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.997+0000 7f9c04817640 1 --2- 172.21.15.88:0/1771497269 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646e2f382c0 0x5646e2fe16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:45.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.997+0000 7f9c05018640 1 --2- 172.21.15.88:0/1771497269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646e2f5a6c0 0x5646e2f7afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:45.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.997+0000 7f9bfffff640 1 -- 172.21.15.88:0/1771497269 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646e2f382c0 msgr2=0x5646e2fe16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:45.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.997+0000 7f9bfffff640 1 --2- 172.21.15.88:0/1771497269 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646e2f382c0 0x5646e2fe16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:45.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.997+0000 7f9bfffff640 1 -- 172.21.15.88:0/1771497269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646e2f5a6c0 msgr2=0x5646e2f7afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:45.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.997+0000 7f9bfffff640 1 --2- 172.21.15.88:0/1771497269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646e2f5a6c0 0x5646e2f7afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:45.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.997+0000 7f9bfffff640 1 -- 172.21.15.88:0/1771497269 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646e2fe1bf0 con 0x5646e2f39a50 2024-09-16T01:36:45.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.997+0000 7f9c04817640 1 --2- 172.21.15.88:0/1771497269 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646e2f382c0 0x5646e2fe16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:45.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.997+0000 7f9bfffff640 1 --2- 172.21.15.88:0/1771497269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 0x5646e2fe45b0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f9bf40177a0 tx=0x7f9bf4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=766ae4896dda2df5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:45.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.997+0000 7f9bdf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:36:45.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.997+0000 7f9bdf7fe640 1 -- 172.21.15.88:0/1771497269 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf4015bb0 con 0x5646e2f39a50 2024-09-16T01:36:45.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.997+0000 7f9bdf7fe640 1 -- 172.21.15.88:0/1771497269 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f9bf4004250 con 0x5646e2f39a50 2024-09-16T01:36:45.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.997+0000 7f9c09818740 1 -- 172.21.15.88:0/1771497269 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bf4017e50 con 0x5646e2f39a50 2024-09-16T01:36:45.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.997+0000 7f9bdf7fe640 1 -- 172.21.15.88:0/1771497269 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9bf4043050 con 0x5646e2f39a50 2024-09-16T01:36:45.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.997+0000 7f9c09818740 10 client.5997 fetch_fsmap learned FSMap version 37 2024-09-16T01:36:45.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.997+0000 7f9c09818740 1 -- 172.21.15.88:0/1771497269 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) -- 0x5646e2f5f130 con 0x5646e2f39a50 2024-09-16T01:36:45.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.997+0000 7f9bdf7fe640 1 -- 172.21.15.88:0/1771497269 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bf4017e50 con 0x5646e2f39a50 2024-09-16T01:36:45.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.997+0000 7f9bdf7fe640 1 -- 172.21.15.88:0/1771497269 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 37) ==== 3502+0+0 (secure 0 0 0) 0x7f9bf4046020 con 0x5646e2f39a50 2024-09-16T01:36:45.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.997+0000 7f9c09818740 10 client.5997 fetch_fsmap finished waiting for FSMap version 37 2024-09-16T01:36:45.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.997+0000 7f9c09818740 10 client.5997 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:36:45.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.997+0000 7f9c09818740 20 client.5997 populate_metadata read hostname 'smithi088' 2024-09-16T01:36:45.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.997+0000 7f9c09818740 1 --2- 172.21.15.88:0/1771497269 >> [v2:172.21.15.88:6835/574169866,v1:172.21.15.88:6837/574169866] conn(0x5646e2f80a10 0x5646e2f61710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:45.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.997+0000 7f9c09818740 4 client.5997 mds_command: new command op to 5862 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:36:45.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.997+0000 7f9c09818740 1 -- 172.21.15.88:0/1771497269 --> [v2:172.21.15.88:6835/574169866,v1:172.21.15.88:6837/574169866] -- command(tid 0: {"prefix": "status"}) -- 0x5646e2ea1fa0 con 0x5646e2f80a10 2024-09-16T01:36:45.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.998+0000 7f9c04817640 1 --2- 172.21.15.88:0/1771497269 >> [v2:172.21.15.88:6835/574169866,v1:172.21.15.88:6837/574169866] conn(0x5646e2f80a10 0x5646e2f61710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:45.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.998+0000 7f9bdf7fe640 10 client.5997 ms_handle_connect on v2:172.21.15.88:6835/574169866 2024-09-16T01:36:45.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.998+0000 7f9c04817640 1 --2- 172.21.15.88:0/1771497269 >> [v2:172.21.15.88:6835/574169866,v1:172.21.15.88:6837/574169866] conn(0x5646e2f80a10 0x5646e2f61710 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fd39119f668a1d0 server_cookie=7964a653183651c1 in_seq=0 out_seq=0 2024-09-16T01:36:45.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.998+0000 7f9bdf7fe640 1 -- 172.21.15.88:0/1771497269 <== mds.0 v2:172.21.15.88:6835/574169866 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x5646e2ea1fa0 con 0x5646e2f80a10 2024-09-16T01:36:45.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.998+0000 7f9bdf7fe640 10 client.5997 handle_command_reply: tid=0 2024-09-16T01:36:45.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.998+0000 7f9c09818740 1 client.5997 shutdown 2024-09-16T01:36:45.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.998+0000 7f9c09818740 1 -- 172.21.15.88:0/1771497269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 msgr2=0x5646e2fe45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:45.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.998+0000 7f9c09818740 1 --2- 172.21.15.88:0/1771497269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 0x5646e2fe45b0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f9bf40177a0 tx=0x7f9bf4004730 comp rx=0 tx=0).stop 2024-09-16T01:36:45.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.998+0000 7f9c09818740 1 -- 172.21.15.88:0/1771497269 shutdown_connections 2024-09-16T01:36:45.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.998+0000 7f9c09818740 1 --2- 172.21.15.88:0/1771497269 >> [v2:172.21.15.88:6835/574169866,v1:172.21.15.88:6837/574169866] conn(0x5646e2f80a10 0x5646e2f61710 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:45.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.998+0000 7f9c09818740 1 --2- 172.21.15.88:0/1771497269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646e2f5a6c0 0x5646e2f7afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:45.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.998+0000 7f9c09818740 1 --2- 172.21.15.88:0/1771497269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5646e2f39a50 0x5646e2fe45b0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:45.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.998+0000 7f9bdf7fe640 0 client.5997 ms_handle_reset on v2:172.21.15.88:6835/574169866 2024-09-16T01:36:45.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.998+0000 7f9c09818740 1 --2- 172.21.15.88:0/1771497269 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646e2f382c0 0x5646e2fe16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:45.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.998+0000 7f9c09818740 1 -- 172.21.15.88:0/1771497269 >> 172.21.15.88:0/1771497269 conn(0x5646e2f35a80 msgr2=0x5646e2f56a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:45.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.998+0000 7f9c09818740 1 -- 172.21.15.88:0/1771497269 shutdown_connections 2024-09-16T01:36:45.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.998+0000 7f9c09818740 1 -- 172.21.15.88:0/1771497269 wait complete. 2024-09-16T01:36:45.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.998+0000 7f9c09818740 20 client.5997 trim_cache size 0 max 16384 2024-09-16T01:36:45.006 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:36:45.007 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "5ace2c91-46f9-482b-8a89-731fbdc171bf", 2024-09-16T01:36:45.007 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2024-09-16T01:36:45.007 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 5862, 2024-09-16T01:36:45.007 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2024-09-16T01:36:45.007 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2024-09-16T01:36:45.007 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2024-09-16T01:36:45.007 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 4.5523900660000001, 2024-09-16T01:36:45.007 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 37, 2024-09-16T01:36:45.007 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 58, 2024-09-16T01:36:45.007 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 58, 2024-09-16T01:36:45.007 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 8.8947515710000005 2024-09-16T01:36:45.007 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:36:45.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.999+0000 7f9c06a7b640 1 -- 172.21.15.88:0/105627439 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9bec06c2b0 msgr2=0x7f9bec06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:45.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.999+0000 7f9c06a7b640 1 --2- 172.21.15.88:0/105627439 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9bec06c2b0 0x7f9bec06e770 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9be80057e0 tx=0x7f9bf0030000 comp rx=0 tx=0).stop 2024-09-16T01:36:45.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.999+0000 7f9c06a7b640 1 -- 172.21.15.88:0/105627439 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c000cc650 msgr2=0x7f9c00151b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:45.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.999+0000 7f9c06a7b640 1 --2- 172.21.15.88:0/105627439 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c000cc650 0x7f9c00151b60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f9c000694d0 tx=0x7f9bf80079d0 comp rx=0 tx=0).stop 2024-09-16T01:36:45.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.999+0000 7f9c06a7b640 1 -- 172.21.15.88:0/105627439 shutdown_connections 2024-09-16T01:36:45.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.999+0000 7f9c06a7b640 1 --2- 172.21.15.88:0/105627439 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9bec06c2b0 0x7f9bec06e770 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:45.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.999+0000 7f9c06a7b640 1 --2- 172.21.15.88:0/105627439 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c000cc650 0x7f9c00151b60 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:45.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.999+0000 7f9c06a7b640 1 --2- 172.21.15.88:0/105627439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c000cc1d0 0x7f9c00151620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:45.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.999+0000 7f9c06a7b640 1 --2- 172.21.15.88:0/105627439 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9be80039b0 0x7f9c0014e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:45.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.999+0000 7f9c06a7b640 1 -- 172.21.15.88:0/105627439 >> 172.21.15.88:0/105627439 conn(0x7f9c0006d0f0 msgr2=0x7f9c00159010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:45.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.999+0000 7f9c06a7b640 1 -- 172.21.15.88:0/105627439 shutdown_connections 2024-09-16T01:36:45.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:44.999+0000 7f9c06a7b640 1 -- 172.21.15.88:0/105627439 wait complete. 2024-09-16T01:36:45.011 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:36:45.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.108+0000 7fd3c6b31640 1 Processor -- start 2024-09-16T01:36:45.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.108+0000 7fd3c6b31640 1 -- start start 2024-09-16T01:36:45.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3c6b31640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3c015e660 0x7fd3c00cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:45.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3c6b31640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd3c00d3310 0x7fd3c00d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:45.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3c6b31640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3c015ea40 con 0x7fd3c00d1650 2024-09-16T01:36:45.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3c6b31640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd3c00d3870 con 0x7fd3c00d3310 2024-09-16T01:36:45.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3c6b31640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd3c00d3b50 con 0x7fd3c015e660 2024-09-16T01:36:45.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3c50ce640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd3c00d1650 0x7fd3c006bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54356/0 (socket says 172.21.15.88:54356) 2024-09-16T01:36:45.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3c50ce640 1 -- 172.21.15.88:0/3149178613 learned_addr learned my addr 172.21.15.88:0/3149178613 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:45.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3c48cd640 1 --2- 172.21.15.88:0/3149178613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3c015e660 0x7fd3c00cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:45.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3bffff640 1 --2- 172.21.15.88:0/3149178613 >> v2:172.21.15.154:3300/0 conn(0x7fd3c00d3310 0x7fd3c00d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:45.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3bf7fe640 1 -- 172.21.15.88:0/3149178613 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3671937611 0 0) 0x7fd3c015ea40 con 0x7fd3c00d1650 2024-09-16T01:36:45.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3bf7fe640 1 -- 172.21.15.88:0/3149178613 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3a8003520 con 0x7fd3c00d1650 2024-09-16T01:36:45.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3bf7fe640 1 -- 172.21.15.88:0/3149178613 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1971950928 0 0) 0x7fd3a8003520 con 0x7fd3c00d1650 2024-09-16T01:36:45.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3bf7fe640 1 -- 172.21.15.88:0/3149178613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3c015e660 msgr2=0x7fd3c00cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:45.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3bf7fe640 1 --2- 172.21.15.88:0/3149178613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3c015e660 0x7fd3c00cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:45.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3bf7fe640 1 -- 172.21.15.88:0/3149178613 >> v2:172.21.15.154:3300/0 conn(0x7fd3c00d3310 msgr2=0x7fd3c00d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:45.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3bf7fe640 1 --2- 172.21.15.88:0/3149178613 >> v2:172.21.15.154:3300/0 conn(0x7fd3c00d3310 0x7fd3c00d1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:45.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3bf7fe640 1 -- 172.21.15.88:0/3149178613 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3c00d3e30 con 0x7fd3c00d1650 2024-09-16T01:36:45.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3bf7fe640 1 -- 172.21.15.88:0/3149178613 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fd3b8001670 con 0x7fd3c00d1650 2024-09-16T01:36:45.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3bf7fe640 1 -- 172.21.15.88:0/3149178613 >> v1:172.21.15.88:6789/0 conn(0x7fd3c00d1650 legacy=0x7fd3c006bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:45.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3bf7fe640 1 --2- 172.21.15.88:0/3149178613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3a80039b0 0x7fd3a8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:45.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3bf7fe640 1 --2- 172.21.15.88:0/3149178613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3a8004800 0x7fd3a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:45.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3bf7fe640 1 --2- 172.21.15.88:0/3149178613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3a80051c0 0x7fd3a800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:45.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3bf7fe640 1 -- 172.21.15.88:0/3149178613 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c00d3870 con 0x7fd3a80039b0 2024-09-16T01:36:45.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3bf7fe640 1 -- 172.21.15.88:0/3149178613 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c00d3b50 con 0x7fd3a80051c0 2024-09-16T01:36:45.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3bf7fe640 1 -- 172.21.15.88:0/3149178613 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd3a800bf00 con 0x7fd3a8004800 2024-09-16T01:36:45.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3c48cd640 1 --2- 172.21.15.88:0/3149178613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3c015e660 0x7fd3c00cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:36:45.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.109+0000 7fd3bffff640 1 --2- 172.21.15.88:0/3149178613 >> v2:172.21.15.154:3300/0 conn(0x7fd3c00d3310 0x7fd3c00d1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:36:45.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.110+0000 7fd3c48cd640 1 --2- 172.21.15.88:0/3149178613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3a80039b0 0x7fd3a8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:45.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.110+0000 7fd3c50ce640 1 --2- 172.21.15.88:0/3149178613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3a80051c0 0x7fd3a800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:45.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.110+0000 7fd3bffff640 1 --2- 172.21.15.88:0/3149178613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3a8004800 0x7fd3a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:45.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.110+0000 7fd3c48cd640 1 -- 172.21.15.88:0/3149178613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3a8004800 msgr2=0x7fd3a8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:45.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.110+0000 7fd3c48cd640 1 --2- 172.21.15.88:0/3149178613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3a8004800 0x7fd3a8004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:45.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.110+0000 7fd3c48cd640 1 -- 172.21.15.88:0/3149178613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3a80051c0 msgr2=0x7fd3a800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:45.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.110+0000 7fd3c48cd640 1 --2- 172.21.15.88:0/3149178613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3a80051c0 0x7fd3a800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:45.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.110+0000 7fd3c48cd640 1 -- 172.21.15.88:0/3149178613 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3a8003520 con 0x7fd3a80039b0 2024-09-16T01:36:45.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.110+0000 7fd3c48cd640 1 --2- 172.21.15.88:0/3149178613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3a80039b0 0x7fd3a8003dd0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0009b00 tx=0x7fd3a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7894130a208da477 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:45.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.111+0000 7fd3bf7fe640 1 -- 172.21.15.88:0/3149178613 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b0036070 con 0x7fd3a80039b0 2024-09-16T01:36:45.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.111+0000 7fd3bf7fe640 1 -- 172.21.15.88:0/3149178613 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3b0004030 con 0x7fd3a80039b0 2024-09-16T01:36:45.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.111+0000 7fd3bf7fe640 1 -- 172.21.15.88:0/3149178613 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b00097d0 con 0x7fd3a80039b0 2024-09-16T01:36:45.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.111+0000 7fd3c6b31640 1 -- 172.21.15.88:0/3149178613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3a80039b0 msgr2=0x7fd3a8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:45.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.111+0000 7fd3c6b31640 1 --2- 172.21.15.88:0/3149178613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3a80039b0 0x7fd3a8003dd0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0009b00 tx=0x7fd3a8005830 comp rx=0 tx=0).stop 2024-09-16T01:36:45.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.111+0000 7fd3c50ce640 1 -- 172.21.15.88:0/3149178613 reap_dead start 2024-09-16T01:36:45.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.111+0000 7fd3c6b31640 1 -- 172.21.15.88:0/3149178613 shutdown_connections 2024-09-16T01:36:45.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.111+0000 7fd3c6b31640 1 -- 172.21.15.88:0/3149178613 >> 172.21.15.88:0/3149178613 conn(0x7fd3c00b6410 msgr2=0x7fd3c00b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:45.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.111+0000 7fd3c6b31640 1 -- 172.21.15.88:0/3149178613 shutdown_connections 2024-09-16T01:36:45.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.111+0000 7fd3c6b31640 1 -- 172.21.15.88:0/3149178613 wait complete. 2024-09-16T01:36:45.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.112+0000 7fd3c6b31640 1 Processor -- start 2024-09-16T01:36:45.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.112+0000 7fd3c6b31640 1 -- start start 2024-09-16T01:36:45.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.113+0000 7fd3c6b31640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3a80039b0 0x7fd3c0150880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:45.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.113+0000 7fd3c6b31640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3c01600a0 0x7fd3c01526e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:45.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.113+0000 7fd3c6b31640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3c0152c20 0x7fd3c0156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:45.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.113+0000 7fd3c6b31640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c00b8b80 con 0x7fd3c0152c20 2024-09-16T01:36:45.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.113+0000 7fd3c6b31640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c00d43f0 con 0x7fd3c01600a0 2024-09-16T01:36:45.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.113+0000 7fd3c6b31640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd3c00b8d00 con 0x7fd3a80039b0 2024-09-16T01:36:45.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.113+0000 7fd3c50ce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3c0152c20 0x7fd3c0156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:45.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.113+0000 7fd3c50ce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3c0152c20 0x7fd3c0156d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41458/0 (socket says 172.21.15.88:41458) 2024-09-16T01:36:45.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.113+0000 7fd3c50ce640 1 -- 172.21.15.88:0/940220755 learned_addr learned my addr 172.21.15.88:0/940220755 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:45.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.113+0000 7fd3c48cd640 1 --2- 172.21.15.88:0/940220755 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3a80039b0 0x7fd3c0150880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:45.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.113+0000 7fd3bffff640 1 --2- 172.21.15.88:0/940220755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3c01600a0 0x7fd3c01526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:45.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.114+0000 7fd3c50ce640 1 -- 172.21.15.88:0/940220755 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3a80039b0 msgr2=0x7fd3c0150880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:45.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.114+0000 7fd3c50ce640 1 --2- 172.21.15.88:0/940220755 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3a80039b0 0x7fd3c0150880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:45.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.114+0000 7fd3c50ce640 1 -- 172.21.15.88:0/940220755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3c01600a0 msgr2=0x7fd3c01526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:45.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.114+0000 7fd3c50ce640 1 --2- 172.21.15.88:0/940220755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3c01600a0 0x7fd3c01526e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:45.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.114+0000 7fd3c50ce640 1 -- 172.21.15.88:0/940220755 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3c015f000 con 0x7fd3c0152c20 2024-09-16T01:36:45.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.114+0000 7fd3bffff640 1 --2- 172.21.15.88:0/940220755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3c01600a0 0x7fd3c01526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:45.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.114+0000 7fd3c48cd640 1 --2- 172.21.15.88:0/940220755 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3a80039b0 0x7fd3c0150880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:45.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.114+0000 7fd3c50ce640 1 --2- 172.21.15.88:0/940220755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3c0152c20 0x7fd3c0156d30 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fd3b800e700 tx=0x7fd3b800ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:45.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.114+0000 7fd3bd7fa640 1 -- 172.21.15.88:0/940220755 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b8019070 con 0x7fd3c0152c20 2024-09-16T01:36:45.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.114+0000 7fd3bd7fa640 1 -- 172.21.15.88:0/940220755 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3b800c5e0 con 0x7fd3c0152c20 2024-09-16T01:36:45.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.114+0000 7fd3c6b31640 1 -- 172.21.15.88:0/940220755 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3c015f2e0 con 0x7fd3c0152c20 2024-09-16T01:36:45.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.114+0000 7fd3bd7fa640 1 -- 172.21.15.88:0/940220755 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b800c9d0 con 0x7fd3c0152c20 2024-09-16T01:36:45.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.114+0000 7fd3c6b31640 1 -- 172.21.15.88:0/940220755 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3c015f8a0 con 0x7fd3c0152c20 2024-09-16T01:36:45.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.115+0000 7fd3c6b31640 1 -- 172.21.15.88:0/940220755 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3c00d1110 con 0x7fd3c0152c20 2024-09-16T01:36:45.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.115+0000 7fd3bd7fa640 1 -- 172.21.15.88:0/940220755 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd3b801c070 con 0x7fd3c0152c20 2024-09-16T01:36:45.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.116+0000 7fd3bd7fa640 1 --2- 172.21.15.88:0/940220755 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd3ac06c2b0 0x7fd3ac06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:45.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.116+0000 7fd3c48cd640 1 --2- 172.21.15.88:0/940220755 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd3ac06c2b0 0x7fd3ac06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:45.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.116+0000 7fd3bd7fa640 1 -- 172.21.15.88:0/940220755 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7fd3b808a870 con 0x7fd3c0152c20 2024-09-16T01:36:45.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.116+0000 7fd3c48cd640 1 --2- 172.21.15.88:0/940220755 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd3ac06c2b0 0x7fd3ac06e770 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0033000 tx=0x7fd3b0008070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:45.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.120+0000 7fd3bd7fa640 1 -- 172.21.15.88:0/940220755 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd3b8058fe0 con 0x7fd3c0152c20 2024-09-16T01:36:45.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.324+0000 7fd3c6b31640 1 -- 172.21.15.88:0/940220755 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd3c00b9e80 con 0x7fd3c0152c20 2024-09-16T01:36:45.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.325+0000 7fd3bd7fa640 1 -- 172.21.15.88:0/940220755 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v37) ==== 53+0+73 (secure 0 0 0) 0x7fd3b805de20 con 0x7fd3c0152c20 2024-09-16T01:36:45.326 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:36:45.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.327+0000 7fd3c6b31640 1 -- 172.21.15.88:0/940220755 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd3ac06c2b0 msgr2=0x7fd3ac06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:45.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.327+0000 7fd3c6b31640 1 --2- 172.21.15.88:0/940220755 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd3ac06c2b0 0x7fd3ac06e770 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0033000 tx=0x7fd3b0008070 comp rx=0 tx=0).stop 2024-09-16T01:36:45.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.327+0000 7fd3c6b31640 1 -- 172.21.15.88:0/940220755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3c0152c20 msgr2=0x7fd3c0156d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:45.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.327+0000 7fd3c6b31640 1 --2- 172.21.15.88:0/940220755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3c0152c20 0x7fd3c0156d30 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fd3b800e700 tx=0x7fd3b800ebc0 comp rx=0 tx=0).stop 2024-09-16T01:36:45.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.327+0000 7fd3c6b31640 1 -- 172.21.15.88:0/940220755 shutdown_connections 2024-09-16T01:36:45.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.327+0000 7fd3c6b31640 1 --2- 172.21.15.88:0/940220755 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd3ac06c2b0 0x7fd3ac06e770 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:45.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.327+0000 7fd3c6b31640 1 --2- 172.21.15.88:0/940220755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3c0152c20 0x7fd3c0156d30 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:45.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.327+0000 7fd3c6b31640 1 --2- 172.21.15.88:0/940220755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3c01600a0 0x7fd3c01526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:45.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.327+0000 7fd3c6b31640 1 --2- 172.21.15.88:0/940220755 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3a80039b0 0x7fd3c0150880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:45.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.327+0000 7fd3c6b31640 1 -- 172.21.15.88:0/940220755 >> 172.21.15.88:0/940220755 conn(0x7fd3c00b6410 msgr2=0x7fd3c00b8ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:45.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.327+0000 7fd3c6b31640 1 -- 172.21.15.88:0/940220755 shutdown_connections 2024-09-16T01:36:45.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45.327+0000 7fd3c6b31640 1 -- 172.21.15.88:0/940220755 wait complete. 2024-09-16T01:36:45.339 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-16T01:36:45.339 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:36:45.339 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2024-09-16T01:36:45.339 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T01:36:45.339 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T01:36:45.339 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-16T01:36:45.339 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-16T01:36:45.339 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:36:45.340 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:36:45.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:45.340 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2024-09-16T01:36:45.356 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:36:45.356 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:36:45.356 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:36:45.356 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:36:45.356 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-09-16T01:36:45.356 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:36:45.357 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:36:45.357 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:36:45.357 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2024-09-16T01:36:45.357 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:36:45.357 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:36:45.357 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2024-09-16T01:36:45.357 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:36:45.357 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2024-09-16T01:36:45.357 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:36:45.357 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2024-09-16T01:36:45.357 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:36:45.357 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42619sec preferred_lft 42619sec 2024-09-16T01:36:45.357 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link noprefixroute 2024-09-16T01:36:45.358 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:36:45.358 INFO:teuthology.orchestra.run.smithi088.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:36:45.358 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff 2024-09-16T01:36:45.358 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T01:36:45.358 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:36:45.358 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::e486:6eff:fe81:a1c/64 scope link 2024-09-16T01:36:45.358 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:36:45.358 INFO:teuthology.orchestra.run.smithi088.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:36:45.358 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 12:6f:29:be:95:48 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:36:45.358 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::106f:29ff:febe:9548/64 scope link 2024-09-16T01:36:45.358 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:36:45.358 INFO:teuthology.orchestra.run.smithi088.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:36:45.358 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T01:36:45.359 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::c63:b4ff:feb3:ad9e/64 scope link 2024-09-16T01:36:45.359 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:36:45.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:45.359 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:36:45.419 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-09-16T01:36:45.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:45.421 DEBUG:teuthology.orchestra.run.smithi088:> route 2024-09-16T01:36:45.477 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2024-09-16T01:36:45.477 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:36:45.477 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:36:45.477 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:36:45.477 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:36:45.478 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:36:45.478 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:36:45.478 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:36:45.478 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:36:45.478 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:36:45.478 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:36:45.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:36:45.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:36:45.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:45.603 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:36:45.619 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T01:36:45.619 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T01:36:45.619 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:36:45.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:45.619 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2024-09-16T01:36:45.684 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:36:45.684 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T01:36:45.739 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:36:45.739 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:36:45.740 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:36:45.795 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-16T01:36:45.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:45.795 DEBUG:teuthology.orchestra.run.smithi088:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:36:45.878 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:36:45.878 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:36:45.878 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:36:45.886 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:36:45.886 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:36:45.887 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:36:45.887 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:36:45.887 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:36:45.892 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:36:45.892 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:36:45.892 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:36:45.892 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:36:45.892 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:36:45.892 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790" 2024-09-16T01:36:45.892 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2024-09-16T01:36:45.892 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5ace2c91-46f9-482b-8a89-731fbdc171bf" will pass to kernel 2024-09-16T01:36:45.893 INFO:teuthology.orchestra.run.smithi088.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-16T01:36:45.893 INFO:teuthology.orchestra.run.smithi088.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:36:45.893 INFO:teuthology.orchestra.run.smithi088.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:36:45.893 INFO:teuthology.orchestra.run.smithi088.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:36:45.893 INFO:teuthology.orchestra.run.smithi088.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:36:45.894 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:36:45.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:45.894 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:36:45.960 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:36:46.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:36:46.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:36:46.059 INFO:teuthology.orchestra.run.smithi088.stdout:client6006 2024-09-16T01:36:46.060 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:36:46.060 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:36:46.075 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'mkdir -p a/{b,c}/d; sync .') 2024-09-16T01:36:46.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:36:46.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:36:46.177 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:36:46.177 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:36:46.193 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:36:46.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.331+0000 7ff19ad31640 1 Processor -- start 2024-09-16T01:36:46.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.331+0000 7ff19ad31640 1 -- start start 2024-09-16T01:36:46.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.331+0000 7ff19ad31640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff1940cf1f0 0x7ff1940b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.331+0000 7ff19ad31640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1940cf860 con 0x7ff19415e7a0 2024-09-16T01:36:46.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.331+0000 7ff19ad31640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1940cfb40 con 0x7ff1940b6720 2024-09-16T01:36:46.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.331+0000 7ff19ad31640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff1940b6b90 con 0x7ff1940cf1f0 2024-09-16T01:36:46.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.332+0000 7ff193fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff1940cf1f0 0x7ff1940b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.332+0000 7ff193fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff1940cf1f0 0x7ff1940b61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41470/0 (socket says 172.21.15.88:41470) 2024-09-16T01:36:46.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.332+0000 7ff198acd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff19415e7a0 0x7ff19406caa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54362/0 (socket says 172.21.15.88:54362) 2024-09-16T01:36:46.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.332+0000 7ff193fff640 1 -- 172.21.15.88:0/511850292 learned_addr learned my addr 172.21.15.88:0/511850292 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:46.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.332+0000 7ff193fff640 1 -- 172.21.15.88:0/511850292 >> v1:172.21.15.154:6789/0 conn(0x7ff1940b6720 legacy=0x7ff1940d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:46.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.332+0000 7ff193fff640 1 -- 172.21.15.88:0/511850292 >> v1:172.21.15.88:6789/0 conn(0x7ff19415e7a0 legacy=0x7ff19406caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:46.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.332+0000 7ff193fff640 1 -- 172.21.15.88:0/511850292 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1940cfe20 con 0x7ff1940cf1f0 2024-09-16T01:36:46.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.332+0000 7ff1937fe640 1 -- 172.21.15.88:0/511850292 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2715964924 0 0) 0x7ff1940cf860 con 0x7ff19415e7a0 2024-09-16T01:36:46.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.333+0000 7ff193fff640 1 --2- 172.21.15.88:0/511850292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1940cf1f0 0x7ff1940b61e0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7ff180009870 tx=0x7ff18002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c17e07496c2bcd10 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:46.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.333+0000 7ff1937fe640 1 -- 172.21.15.88:0/511850292 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff180037070 con 0x7ff1940cf1f0 2024-09-16T01:36:46.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.333+0000 7ff1937fe640 1 -- 172.21.15.88:0/511850292 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff180032440 con 0x7ff1940cf1f0 2024-09-16T01:36:46.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.333+0000 7ff1937fe640 1 -- 172.21.15.88:0/511850292 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff180032810 con 0x7ff1940cf1f0 2024-09-16T01:36:46.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.333+0000 7ff19ad31640 1 -- 172.21.15.88:0/511850292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1940cf1f0 msgr2=0x7ff1940b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:46.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.333+0000 7ff19ad31640 1 --2- 172.21.15.88:0/511850292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1940cf1f0 0x7ff1940b61e0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7ff180009870 tx=0x7ff18002a210 comp rx=0 tx=0).stop 2024-09-16T01:36:46.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.333+0000 7ff19ad31640 1 -- 172.21.15.88:0/511850292 shutdown_connections 2024-09-16T01:36:46.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.333+0000 7ff19ad31640 1 --2- 172.21.15.88:0/511850292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1940cf1f0 0x7ff1940b61e0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.333+0000 7ff19ad31640 1 -- 172.21.15.88:0/511850292 >> 172.21.15.88:0/511850292 conn(0x7ff1940d11e0 msgr2=0x7ff1940d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:46.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.333+0000 7ff19ad31640 1 -- 172.21.15.88:0/511850292 shutdown_connections 2024-09-16T01:36:46.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.333+0000 7ff19ad31640 1 -- 172.21.15.88:0/511850292 wait complete. 2024-09-16T01:36:46.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.334+0000 7ff19ad31640 1 Processor -- start 2024-09-16T01:36:46.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.334+0000 7ff19ad31640 1 -- start start 2024-09-16T01:36:46.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.335+0000 7ff19ad31640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1940b6720 0x7ff194159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.335+0000 7ff19ad31640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1940cf1f0 0x7ff19414da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.335+0000 7ff19ad31640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff19415e7a0 0x7ff194154ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.335+0000 7ff19ad31640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff1940b7a00 con 0x7ff1940b6720 2024-09-16T01:36:46.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.335+0000 7ff19ad31640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff1940d2ca0 con 0x7ff19415e7a0 2024-09-16T01:36:46.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.335+0000 7ff19ad31640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff19415eb80 con 0x7ff1940cf1f0 2024-09-16T01:36:46.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.335+0000 7ff198acd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1940b6720 0x7ff194159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.335+0000 7ff198acd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1940b6720 0x7ff194159220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41472/0 (socket says 172.21.15.88:41472) 2024-09-16T01:36:46.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.335+0000 7ff198acd640 1 -- 172.21.15.88:0/2628106229 learned_addr learned my addr 172.21.15.88:0/2628106229 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:46.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.336+0000 7ff1992ce640 1 --2- 172.21.15.88:0/2628106229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff19415e7a0 0x7ff194154ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.336+0000 7ff193fff640 1 --2- 172.21.15.88:0/2628106229 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1940cf1f0 0x7ff19414da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.336+0000 7ff198acd640 1 -- 172.21.15.88:0/2628106229 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1940cf1f0 msgr2=0x7ff19414da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:46.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.336+0000 7ff198acd640 1 --2- 172.21.15.88:0/2628106229 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1940cf1f0 0x7ff19414da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.336+0000 7ff198acd640 1 -- 172.21.15.88:0/2628106229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff19415e7a0 msgr2=0x7ff194154ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:46.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.336+0000 7ff198acd640 1 --2- 172.21.15.88:0/2628106229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff19415e7a0 0x7ff194154ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.336+0000 7ff198acd640 1 -- 172.21.15.88:0/2628106229 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff19415a2d0 con 0x7ff1940b6720 2024-09-16T01:36:46.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.336+0000 7ff198acd640 1 --2- 172.21.15.88:0/2628106229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1940b6720 0x7ff194159220 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7ff18800c4d0 tx=0x7ff18800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:46.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.336+0000 7ff1917fa640 1 -- 172.21.15.88:0/2628106229 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff188018070 con 0x7ff1940b6720 2024-09-16T01:36:46.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.336+0000 7ff1917fa640 1 -- 172.21.15.88:0/2628106229 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff188007500 con 0x7ff1940b6720 2024-09-16T01:36:46.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.336+0000 7ff1917fa640 1 -- 172.21.15.88:0/2628106229 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff188004a20 con 0x7ff1940b6720 2024-09-16T01:36:46.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.336+0000 7ff19ad31640 1 -- 172.21.15.88:0/2628106229 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff19415a5b0 con 0x7ff1940b6720 2024-09-16T01:36:46.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.336+0000 7ff19ad31640 1 -- 172.21.15.88:0/2628106229 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff194160210 con 0x7ff1940b6720 2024-09-16T01:36:46.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.337+0000 7ff19ad31640 1 -- 172.21.15.88:0/2628106229 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1940b61e0 con 0x7ff1940b6720 2024-09-16T01:36:46.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.338+0000 7ff1917fa640 1 -- 172.21.15.88:0/2628106229 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff188004bc0 con 0x7ff1940b6720 2024-09-16T01:36:46.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.338+0000 7ff1917fa640 1 --2- 172.21.15.88:0/2628106229 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff17806c2b0 0x7ff17806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.338+0000 7ff1917fa640 1 -- 172.21.15.88:0/2628106229 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7ff18808a980 con 0x7ff1940b6720 2024-09-16T01:36:46.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.338+0000 7ff193fff640 1 --2- 172.21.15.88:0/2628106229 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff17806c2b0 0x7ff17806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.339+0000 7ff193fff640 1 --2- 172.21.15.88:0/2628106229 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff17806c2b0 0x7ff17806e770 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff180034000 tx=0x7ff18002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:46.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.342+0000 7ff1917fa640 1 -- 172.21.15.88:0/2628106229 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff188059260 con 0x7ff1940b6720 2024-09-16T01:36:46.442 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:36:46.441+0000 7f32886f9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:36:46.488 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:36:46.490 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:36:46.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.521+0000 7ff19ad31640 1 -- 172.21.15.88:0/2628106229 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff1940d2a80 con 0x7ff1940b6720 2024-09-16T01:36:46.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.521+0000 7ff1917fa640 1 -- 172.21.15.88:0/2628106229 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v58) ==== 80+0+0 (secure 0 0 0) 0x7ff188058b60 con 0x7ff1940b6720 2024-09-16T01:36:46.522 INFO:teuthology.orchestra.run.smithi088.stderr:listed 0 entries 2024-09-16T01:36:46.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.523+0000 7ff19ad31640 1 -- 172.21.15.88:0/2628106229 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff17806c2b0 msgr2=0x7ff17806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:46.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.523+0000 7ff19ad31640 1 --2- 172.21.15.88:0/2628106229 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff17806c2b0 0x7ff17806e770 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff180034000 tx=0x7ff18002e000 comp rx=0 tx=0).stop 2024-09-16T01:36:46.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.523+0000 7ff19ad31640 1 -- 172.21.15.88:0/2628106229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1940b6720 msgr2=0x7ff194159220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:46.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.523+0000 7ff19ad31640 1 --2- 172.21.15.88:0/2628106229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1940b6720 0x7ff194159220 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7ff18800c4d0 tx=0x7ff18800c990 comp rx=0 tx=0).stop 2024-09-16T01:36:46.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.523+0000 7ff19ad31640 1 -- 172.21.15.88:0/2628106229 shutdown_connections 2024-09-16T01:36:46.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.523+0000 7ff19ad31640 1 --2- 172.21.15.88:0/2628106229 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff17806c2b0 0x7ff17806e770 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.523+0000 7ff19ad31640 1 --2- 172.21.15.88:0/2628106229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff19415e7a0 0x7ff194154ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.523+0000 7ff19ad31640 1 --2- 172.21.15.88:0/2628106229 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1940cf1f0 0x7ff19414da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.523+0000 7ff19ad31640 1 --2- 172.21.15.88:0/2628106229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1940b6720 0x7ff194159220 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.523+0000 7ff19ad31640 1 -- 172.21.15.88:0/2628106229 >> 172.21.15.88:0/2628106229 conn(0x7ff1940d11e0 msgr2=0x7ff194157320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:46.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.523+0000 7ff19ad31640 1 -- 172.21.15.88:0/2628106229 shutdown_connections 2024-09-16T01:36:46.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.523+0000 7ff19ad31640 1 -- 172.21.15.88:0/2628106229 wait complete. 2024-09-16T01:36:46.535 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-16T01:36:46.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:46.535 DEBUG:teuthology.orchestra.run.smithi088:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-16T01:36:46.606 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:36:46.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:46.607 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:36:46.663 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 flush journal 2024-09-16T01:36:46.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.801+0000 7fe1b4e1e640 1 Processor -- start 2024-09-16T01:36:46.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.801+0000 7fe1b4e1e640 1 -- start start 2024-09-16T01:36:46.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.802+0000 7fe1b4e1e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b015a050 0x7fe1b015d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.802+0000 7fe1b4e1e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1b015a430 con 0x7fe1b00731c0 2024-09-16T01:36:46.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.802+0000 7fe1b4e1e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1b0160df0 con 0x7fe1b015ee90 2024-09-16T01:36:46.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.802+0000 7fe1b4e1e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe1b015d9a0 con 0x7fe1b015a050 2024-09-16T01:36:46.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.802+0000 7fe1add9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe1b00731c0 0x7fe1b015ffd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54366/0 (socket says 172.21.15.88:54366) 2024-09-16T01:36:46.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.802+0000 7fe1add9b640 1 -- 172.21.15.88:0/986590525 learned_addr learned my addr 172.21.15.88:0/986590525 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:46.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.802+0000 7fe1ae59c640 1 --2- 172.21.15.88:0/986590525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b015a050 0x7fe1b015d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.803+0000 7fe1ad59a640 1 -- 172.21.15.88:0/986590525 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3919459777 0 0) 0x7fe1b015a430 con 0x7fe1b00731c0 2024-09-16T01:36:46.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.803+0000 7fe1ad59a640 1 -- 172.21.15.88:0/986590525 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe198003520 con 0x7fe1b00731c0 2024-09-16T01:36:46.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.803+0000 7fe1ad59a640 1 -- 172.21.15.88:0/986590525 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2040691451 0 0) 0x7fe198003520 con 0x7fe1b00731c0 2024-09-16T01:36:46.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.803+0000 7fe1ad59a640 1 -- 172.21.15.88:0/986590525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b015a050 msgr2=0x7fe1b015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:46.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.803+0000 7fe1ad59a640 1 --2- 172.21.15.88:0/986590525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b015a050 0x7fe1b015d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.803+0000 7fe1ad59a640 1 -- 172.21.15.88:0/986590525 >> v1:172.21.15.154:6789/0 conn(0x7fe1b015ee90 legacy=0x7fe1b01606e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:46.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.803+0000 7fe1ad59a640 1 -- 172.21.15.88:0/986590525 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b00cd0a0 con 0x7fe1b00731c0 2024-09-16T01:36:46.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.803+0000 7fe1ad59a640 1 -- 172.21.15.88:0/986590525 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fe1a4001670 con 0x7fe1b00731c0 2024-09-16T01:36:46.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.803+0000 7fe1ad59a640 1 -- 172.21.15.88:0/986590525 >> v1:172.21.15.88:6789/0 conn(0x7fe1b00731c0 legacy=0x7fe1b015ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:46.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.803+0000 7fe1ad59a640 1 --2- 172.21.15.88:0/986590525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b015a050 0x7fe1980039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.803+0000 7fe1ad59a640 1 --2- 172.21.15.88:0/986590525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe198003f60 0x7fe198004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.803+0000 7fe1ad59a640 1 --2- 172.21.15.88:0/986590525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe198004d40 0x7fe1980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.803+0000 7fe1ad59a640 1 -- 172.21.15.88:0/986590525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b015d9a0 con 0x7fe198004d40 2024-09-16T01:36:46.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.803+0000 7fe1ad59a640 1 -- 172.21.15.88:0/986590525 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe19800bb50 con 0x7fe198003f60 2024-09-16T01:36:46.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.803+0000 7fe1ad59a640 1 -- 172.21.15.88:0/986590525 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe19800bd50 con 0x7fe1b015a050 2024-09-16T01:36:46.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.803+0000 7fe1aed9d640 1 --2- 172.21.15.88:0/986590525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe198004d40 0x7fe1980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.803+0000 7fe1add9b640 1 --2- 172.21.15.88:0/986590525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe198003f60 0x7fe198004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.803+0000 7fe1ae59c640 1 --2- 172.21.15.88:0/986590525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b015a050 0x7fe1980039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.803+0000 7fe1aed9d640 1 -- 172.21.15.88:0/986590525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b015a050 msgr2=0x7fe1980039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:46.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.803+0000 7fe1aed9d640 1 --2- 172.21.15.88:0/986590525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b015a050 0x7fe1980039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.803+0000 7fe1aed9d640 1 -- 172.21.15.88:0/986590525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe198003f60 msgr2=0x7fe198004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:46.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.803+0000 7fe1aed9d640 1 --2- 172.21.15.88:0/986590525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe198003f60 0x7fe198004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.803+0000 7fe1aed9d640 1 -- 172.21.15.88:0/986590525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe198003520 con 0x7fe198004d40 2024-09-16T01:36:46.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.804+0000 7fe1aed9d640 1 --2- 172.21.15.88:0/986590525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe198004d40 0x7fe1980051c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8001720 tx=0x7fe198005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e069d09904584f8e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:46.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.804+0000 7fe1ad59a640 1 -- 172.21.15.88:0/986590525 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a8001370 con 0x7fe198004d40 2024-09-16T01:36:46.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.804+0000 7fe1ad59a640 1 -- 172.21.15.88:0/986590525 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1a80052d0 con 0x7fe198004d40 2024-09-16T01:36:46.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.804+0000 7fe1ad59a640 1 -- 172.21.15.88:0/986590525 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a802abc0 con 0x7fe198004d40 2024-09-16T01:36:46.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.804+0000 7fe1b4e1e640 1 -- 172.21.15.88:0/986590525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe198004d40 msgr2=0x7fe1980051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:46.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.805+0000 7fe1b4e1e640 1 --2- 172.21.15.88:0/986590525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe198004d40 0x7fe1980051c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8001720 tx=0x7fe198005830 comp rx=0 tx=0).stop 2024-09-16T01:36:46.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.805+0000 7fe1aed9d640 1 -- 172.21.15.88:0/986590525 reap_dead start 2024-09-16T01:36:46.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.805+0000 7fe1b4e1e640 1 -- 172.21.15.88:0/986590525 shutdown_connections 2024-09-16T01:36:46.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.805+0000 7fe1b4e1e640 1 -- 172.21.15.88:0/986590525 >> 172.21.15.88:0/986590525 conn(0x7fe1b006d0f0 msgr2=0x7fe1b006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:46.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.805+0000 7fe1b4e1e640 1 -- 172.21.15.88:0/986590525 shutdown_connections 2024-09-16T01:36:46.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.805+0000 7fe1b4e1e640 1 -- 172.21.15.88:0/986590525 wait complete. 2024-09-16T01:36:46.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.806+0000 7fe1b4e1e640 1 Processor -- start 2024-09-16T01:36:46.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.806+0000 7fe1b4e1e640 1 -- start start 2024-09-16T01:36:46.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.807+0000 7fe1b4e1e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe198004d40 0x7fe1b01521f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.807+0000 7fe1b4e1e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1b0071d80 0x7fe1b00721e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.807+0000 7fe1b4e1e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b015b760 0x7fe1b015bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.807+0000 7fe1b4e1e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b00cd940 con 0x7fe198004d40 2024-09-16T01:36:46.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.807+0000 7fe1b4e1e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b00cd660 con 0x7fe1b0071d80 2024-09-16T01:36:46.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.807+0000 7fe1b4e1e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe1b00cdc20 con 0x7fe1b015b760 2024-09-16T01:36:46.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.807+0000 7fe1ae59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe198004d40 0x7fe1b01521f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.807+0000 7fe1ae59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe198004d40 0x7fe1b01521f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41488/0 (socket says 172.21.15.88:41488) 2024-09-16T01:36:46.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.807+0000 7fe1ae59c640 1 -- 172.21.15.88:0/2112179131 learned_addr learned my addr 172.21.15.88:0/2112179131 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:46.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.807+0000 7fe1add9b640 1 --2- 172.21.15.88:0/2112179131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1b0071d80 0x7fe1b00721e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.807+0000 7fe1aed9d640 1 --2- 172.21.15.88:0/2112179131 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b015b760 0x7fe1b015bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.807+0000 7fe1ae59c640 1 -- 172.21.15.88:0/2112179131 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b015b760 msgr2=0x7fe1b015bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:46.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.807+0000 7fe1ae59c640 1 --2- 172.21.15.88:0/2112179131 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b015b760 0x7fe1b015bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.807+0000 7fe1ae59c640 1 -- 172.21.15.88:0/2112179131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1b0071d80 msgr2=0x7fe1b00721e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:46.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.807+0000 7fe1ae59c640 1 --2- 172.21.15.88:0/2112179131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1b0071d80 0x7fe1b00721e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.807+0000 7fe1ae59c640 1 -- 172.21.15.88:0/2112179131 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b00cb6d0 con 0x7fe198004d40 2024-09-16T01:36:46.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.807+0000 7fe1aed9d640 1 --2- 172.21.15.88:0/2112179131 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b015b760 0x7fe1b015bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:46.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.807+0000 7fe1ae59c640 1 --2- 172.21.15.88:0/2112179131 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe198004d40 0x7fe1b01521f0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0011e30 tx=0x7fe1a000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:46.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.807+0000 7fe19effd640 1 -- 172.21.15.88:0/2112179131 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a0015630 con 0x7fe198004d40 2024-09-16T01:36:46.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.807+0000 7fe19effd640 1 -- 172.21.15.88:0/2112179131 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1a00157d0 con 0x7fe198004d40 2024-09-16T01:36:46.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.807+0000 7fe1b4e1e640 1 -- 172.21.15.88:0/2112179131 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1b00cb9b0 con 0x7fe198004d40 2024-09-16T01:36:46.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.807+0000 7fe19effd640 1 -- 172.21.15.88:0/2112179131 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a0015ba0 con 0x7fe198004d40 2024-09-16T01:36:46.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.807+0000 7fe1b4e1e640 1 -- 172.21.15.88:0/2112179131 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1b00cbf70 con 0x7fe198004d40 2024-09-16T01:36:46.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.809+0000 7fe19effd640 1 -- 172.21.15.88:0/2112179131 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe1a001d400 con 0x7fe198004d40 2024-09-16T01:36:46.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.810+0000 7fe19effd640 1 --2- 172.21.15.88:0/2112179131 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe19006c2b0 0x7fe19006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.810+0000 7fe1add9b640 1 --2- 172.21.15.88:0/2112179131 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe19006c2b0 0x7fe19006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.810+0000 7fe19effd640 1 -- 172.21.15.88:0/2112179131 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7fe1a008c720 con 0x7fe198004d40 2024-09-16T01:36:46.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.810+0000 7fe1add9b640 1 --2- 172.21.15.88:0/2112179131 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe19006c2b0 0x7fe19006e770 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe1b00b3170 tx=0x7fe1a4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:46.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.810+0000 7fe1b7c8b740 1 Processor -- start 2024-09-16T01:36:46.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.811+0000 7fe1b7c8b740 1 -- start start 2024-09-16T01:36:46.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.811+0000 7fe1b7c8b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55962b8d1920 0x55962b9170c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.811+0000 7fe1b7c8b740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55962b8cf0e0 0x55962b8d1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.811+0000 7fe1b7c8b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b8d30b0 0x55962b8cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.811+0000 7fe1b7c8b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55962b8346f0 con 0x55962b8d30b0 2024-09-16T01:36:46.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.811+0000 7fe1b7c8b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55962b7dc330 con 0x55962b8d1920 2024-09-16T01:36:46.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.811+0000 7fe1b7c8b740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55962b8306b0 con 0x55962b8cf0e0 2024-09-16T01:36:46.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.811+0000 7fe1aed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b8d30b0 0x55962b8cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.812+0000 7fe1aed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b8d30b0 0x55962b8cc1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41500/0 (socket says 172.21.15.88:41500) 2024-09-16T01:36:46.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.812+0000 7fe1aed9d640 1 -- 172.21.15.88:0/1533539190 learned_addr learned my addr 172.21.15.88:0/1533539190 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:46.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.812+0000 7fe1ae59c640 1 --2- 172.21.15.88:0/1533539190 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55962b8d1920 0x55962b9170c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.812+0000 7fe1add9b640 1 --2- 172.21.15.88:0/1533539190 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55962b8cf0e0 0x55962b8d1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.812+0000 7fe1aed9d640 1 -- 172.21.15.88:0/1533539190 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55962b8cf0e0 msgr2=0x55962b8d1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:46.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.812+0000 7fe1aed9d640 1 --2- 172.21.15.88:0/1533539190 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55962b8cf0e0 0x55962b8d1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.812+0000 7fe1aed9d640 1 -- 172.21.15.88:0/1533539190 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55962b8d1920 msgr2=0x55962b9170c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:46.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.812+0000 7fe1aed9d640 1 --2- 172.21.15.88:0/1533539190 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55962b8d1920 0x55962b9170c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.812+0000 7fe1aed9d640 1 -- 172.21.15.88:0/1533539190 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55962b8020d0 con 0x55962b8d30b0 2024-09-16T01:36:46.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.812+0000 7fe1aed9d640 1 --2- 172.21.15.88:0/1533539190 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b8d30b0 0x55962b8cc1a0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8001810 tx=0x7fe1a802c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=992c1efada9dce37 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:46.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.812+0000 7fe18ffff640 1 -- 172.21.15.88:0/1533539190 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a8041070 con 0x55962b8d30b0 2024-09-16T01:36:46.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.812+0000 7fe18ffff640 1 -- 172.21.15.88:0/1533539190 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1a80052d0 con 0x55962b8d30b0 2024-09-16T01:36:46.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.812+0000 7fe18ffff640 1 -- 172.21.15.88:0/1533539190 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a8037040 con 0x55962b8d30b0 2024-09-16T01:36:46.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.813+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/1533539190 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b8d30b0 msgr2=0x55962b8cc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:46.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.813+0000 7fe1b7c8b740 1 --2- 172.21.15.88:0/1533539190 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b8d30b0 0x55962b8cc1a0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8001810 tx=0x7fe1a802c000 comp rx=0 tx=0).stop 2024-09-16T01:36:46.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.813+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/1533539190 shutdown_connections 2024-09-16T01:36:46.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.813+0000 7fe1b7c8b740 1 --2- 172.21.15.88:0/1533539190 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b8d30b0 0x55962b8cc1a0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.813+0000 7fe1b7c8b740 1 --2- 172.21.15.88:0/1533539190 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55962b8cf0e0 0x55962b8d1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.813+0000 7fe1b7c8b740 1 --2- 172.21.15.88:0/1533539190 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55962b8d1920 0x55962b9170c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.813+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/1533539190 >> 172.21.15.88:0/1533539190 conn(0x55962b8ce370 msgr2=0x55962b916590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:46.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.813+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/1533539190 shutdown_connections 2024-09-16T01:36:46.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.813+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/1533539190 wait complete. 2024-09-16T01:36:46.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.813+0000 7fe1b7c8b740 1 -- start start 2024-09-16T01:36:46.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.814+0000 7fe1b7c8b740 1 Processor -- start 2024-09-16T01:36:46.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.814+0000 7fe1b7c8b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55962b8d1920 0x55962b8e8350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.814+0000 7fe1b7c8b740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55962b8d30b0 0x55962b8f9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.814+0000 7fe1b7c8b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b911d00 0x55962b912180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.814+0000 7fe1b7c8b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55962b8cd810 con 0x55962b911d00 2024-09-16T01:36:46.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.814+0000 7fe1b7c8b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55962b7bd4f0 con 0x55962b8d1920 2024-09-16T01:36:46.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.814+0000 7fe1b7c8b740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55962b9178f0 con 0x55962b8d30b0 2024-09-16T01:36:46.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.815+0000 7fe1b7c8b740 10 client.0 _ll_register_callbacks cb 0x55962b7deb50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:36:46.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.815+0000 7fe1aed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b911d00 0x55962b912180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.815+0000 7fe1aed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b911d00 0x55962b912180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41514/0 (socket says 172.21.15.88:41514) 2024-09-16T01:36:46.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.815+0000 7fe1aed9d640 1 -- 172.21.15.88:0/911779567 learned_addr learned my addr 172.21.15.88:0/911779567 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:46.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.815+0000 7fe1ae59c640 1 --2- 172.21.15.88:0/911779567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55962b8d1920 0x55962b8e8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.815+0000 7fe1add9b640 1 --2- 172.21.15.88:0/911779567 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55962b8d30b0 0x55962b8f9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.815+0000 7fe1aed9d640 1 -- 172.21.15.88:0/911779567 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55962b8d30b0 msgr2=0x55962b8f9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:46.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.815+0000 7fe1aed9d640 1 --2- 172.21.15.88:0/911779567 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55962b8d30b0 0x55962b8f9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.815+0000 7fe1aed9d640 1 -- 172.21.15.88:0/911779567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55962b8d1920 msgr2=0x55962b8e8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:46.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.815+0000 7fe1aed9d640 1 --2- 172.21.15.88:0/911779567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55962b8d1920 0x55962b8e8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.815+0000 7fe1aed9d640 1 -- 172.21.15.88:0/911779567 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55962b6117d0 con 0x55962b911d00 2024-09-16T01:36:46.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.815+0000 7fe1add9b640 1 --2- 172.21.15.88:0/911779567 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55962b8d30b0 0x55962b8f9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:46.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.815+0000 7fe1aed9d640 1 --2- 172.21.15.88:0/911779567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b911d00 0x55962b912180 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8008fd0 tx=0x7fe1a8001190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee5dca7465b9e9d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:46.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.815+0000 7fe18d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:36:46.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.815+0000 7fe18d7fa640 1 -- 172.21.15.88:0/911779567 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a8041040 con 0x55962b911d00 2024-09-16T01:36:46.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.815+0000 7fe18d7fa640 1 -- 172.21.15.88:0/911779567 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7fe1a8005070 con 0x55962b911d00 2024-09-16T01:36:46.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.815+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/911779567 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1a80052d0 con 0x55962b911d00 2024-09-16T01:36:46.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.815+0000 7fe18d7fa640 1 -- 172.21.15.88:0/911779567 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe1a8052050 con 0x55962b911d00 2024-09-16T01:36:46.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.815+0000 7fe1b7c8b740 10 client.6027 fetch_fsmap learned FSMap version 37 2024-09-16T01:36:46.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.815+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/911779567 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) -- 0x55962b5473f0 con 0x55962b911d00 2024-09-16T01:36:46.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.815+0000 7fe18d7fa640 1 -- 172.21.15.88:0/911779567 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1a80052d0 con 0x55962b911d00 2024-09-16T01:36:46.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.816+0000 7fe18d7fa640 1 -- 172.21.15.88:0/911779567 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 37) ==== 3502+0+0 (secure 0 0 0) 0x7fe1a8055020 con 0x55962b911d00 2024-09-16T01:36:46.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.816+0000 7fe1b7c8b740 10 client.6027 fetch_fsmap finished waiting for FSMap version 37 2024-09-16T01:36:46.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.816+0000 7fe1b7c8b740 10 client.6027 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-09-16T01:36:46.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.816+0000 7fe1b7c8b740 20 client.6027 populate_metadata read hostname 'smithi088' 2024-09-16T01:36:46.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.816+0000 7fe1b7c8b740 1 --2- 172.21.15.88:0/911779567 >> [v2:172.21.15.88:6835/574169866,v1:172.21.15.88:6837/574169866] conn(0x55962b913070 0x55962b8fb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.816+0000 7fe1b7c8b740 4 client.6027 mds_command: new command op to 5862 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:36:46.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.816+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/911779567 --> [v2:172.21.15.88:6835/574169866,v1:172.21.15.88:6837/574169866] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55962b841e20 con 0x55962b913070 2024-09-16T01:36:46.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.816+0000 7fe1ae59c640 1 --2- 172.21.15.88:0/911779567 >> [v2:172.21.15.88:6835/574169866,v1:172.21.15.88:6837/574169866] conn(0x55962b913070 0x55962b8fb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.816+0000 7fe18d7fa640 10 client.6027 ms_handle_connect on v2:172.21.15.88:6835/574169866 2024-09-16T01:36:46.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.816+0000 7fe1ae59c640 1 --2- 172.21.15.88:0/911779567 >> [v2:172.21.15.88:6835/574169866,v1:172.21.15.88:6837/574169866] conn(0x55962b913070 0x55962b8fb9c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4cdd50d778834a4 server_cookie=18ea9f0c4335ba53 in_seq=0 out_seq=0 2024-09-16T01:36:46.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.817+0000 7fe18d7fa640 1 -- 172.21.15.88:0/911779567 <== mds.0 v2:172.21.15.88:6835/574169866 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55962b841e20 con 0x55962b913070 2024-09-16T01:36:46.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.817+0000 7fe18d7fa640 10 client.6027 handle_command_reply: tid=0 2024-09-16T01:36:46.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.817+0000 7fe1b7c8b740 1 client.6027 shutdown 2024-09-16T01:36:46.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.817+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/911779567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b911d00 msgr2=0x55962b912180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:46.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.817+0000 7fe1b7c8b740 1 --2- 172.21.15.88:0/911779567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b911d00 0x55962b912180 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8008fd0 tx=0x7fe1a8001190 comp rx=0 tx=0).stop 2024-09-16T01:36:46.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.817+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/911779567 shutdown_connections 2024-09-16T01:36:46.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.817+0000 7fe1b7c8b740 1 --2- 172.21.15.88:0/911779567 >> [v2:172.21.15.88:6835/574169866,v1:172.21.15.88:6837/574169866] conn(0x55962b913070 0x55962b8fb9c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.817+0000 7fe1b7c8b740 1 --2- 172.21.15.88:0/911779567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b911d00 0x55962b912180 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.817+0000 7fe1b7c8b740 1 --2- 172.21.15.88:0/911779567 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55962b8d30b0 0x55962b8f9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.817+0000 7fe1b7c8b740 1 --2- 172.21.15.88:0/911779567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55962b8d1920 0x55962b8e8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.817+0000 7fe18d7fa640 0 client.6027 ms_handle_reset on v2:172.21.15.88:6835/574169866 2024-09-16T01:36:46.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.817+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/911779567 >> 172.21.15.88:0/911779567 conn(0x55962b8cf0e0 msgr2=0x55962b9180f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:46.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.817+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/911779567 shutdown_connections 2024-09-16T01:36:46.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.817+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/911779567 wait complete. 2024-09-16T01:36:46.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.817+0000 7fe1b7c8b740 20 client.6027 trim_cache size 0 max 16384 2024-09-16T01:36:46.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.831+0000 7fe1b7c8b740 1 Processor -- start 2024-09-16T01:36:46.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.831+0000 7fe1b7c8b740 1 -- start start 2024-09-16T01:36:46.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.832+0000 7fe1b7c8b740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55962b8d1920 0x55962b9180f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.832+0000 7fe1b7c8b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b8d30b0 0x55962b918630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.832+0000 7fe1b7c8b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55962b911d00 0x55962b924cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.832+0000 7fe1b7c8b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b02012b0 con 0x55962b8d30b0 2024-09-16T01:36:46.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.832+0000 7fe1b7c8b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b02002d0 con 0x55962b911d00 2024-09-16T01:36:46.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.832+0000 7fe1b7c8b740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe1b01ff800 con 0x55962b8d1920 2024-09-16T01:36:46.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.832+0000 7fe1add9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b8d30b0 0x55962b918630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.832+0000 7fe1add9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b8d30b0 0x55962b918630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41524/0 (socket says 172.21.15.88:41524) 2024-09-16T01:36:46.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.832+0000 7fe1add9b640 1 -- 172.21.15.88:0/2062123220 learned_addr learned my addr 172.21.15.88:0/2062123220 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:46.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.832+0000 7fe1add9b640 1 -- 172.21.15.88:0/2062123220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55962b8d1920 msgr2=0x55962b9180f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:46.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.832+0000 7fe1add9b640 1 --2- 172.21.15.88:0/2062123220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55962b8d1920 0x55962b9180f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.832+0000 7fe1add9b640 1 -- 172.21.15.88:0/2062123220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55962b911d00 msgr2=0x55962b924cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:46.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.832+0000 7fe1add9b640 1 --2- 172.21.15.88:0/2062123220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55962b911d00 0x55962b924cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.832+0000 7fe1add9b640 1 -- 172.21.15.88:0/2062123220 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55962b5070f0 con 0x55962b8d30b0 2024-09-16T01:36:46.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.832+0000 7fe1add9b640 1 --2- 172.21.15.88:0/2062123220 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b8d30b0 0x55962b918630 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4001740 tx=0x7fe1a4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17f006a0bec6a7bf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:46.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.833+0000 7fe18f7fe640 1 -- 172.21.15.88:0/2062123220 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a4022610 con 0x55962b8d30b0 2024-09-16T01:36:46.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.833+0000 7fe18f7fe640 1 -- 172.21.15.88:0/2062123220 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1a4027020 con 0x55962b8d30b0 2024-09-16T01:36:46.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.833+0000 7fe18f7fe640 1 -- 172.21.15.88:0/2062123220 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a40227b0 con 0x55962b8d30b0 2024-09-16T01:36:46.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.833+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/2062123220 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b8d30b0 msgr2=0x55962b918630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:46.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.833+0000 7fe1b7c8b740 1 --2- 172.21.15.88:0/2062123220 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b8d30b0 0x55962b918630 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4001740 tx=0x7fe1a4014000 comp rx=0 tx=0).stop 2024-09-16T01:36:46.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.833+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/2062123220 shutdown_connections 2024-09-16T01:36:46.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.833+0000 7fe1b7c8b740 1 --2- 172.21.15.88:0/2062123220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55962b911d00 0x55962b924cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.833+0000 7fe1b7c8b740 1 --2- 172.21.15.88:0/2062123220 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b8d30b0 0x55962b918630 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.833+0000 7fe1b7c8b740 1 --2- 172.21.15.88:0/2062123220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55962b8d1920 0x55962b9180f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.833+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/2062123220 >> 172.21.15.88:0/2062123220 conn(0x55962b8cf0e0 msgr2=0x55962b8e7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:46.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.833+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/2062123220 shutdown_connections 2024-09-16T01:36:46.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.833+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/2062123220 wait complete. 2024-09-16T01:36:46.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.834+0000 7fe1b7c8b740 1 -- start start 2024-09-16T01:36:46.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.834+0000 7fe1b7c8b740 1 Processor -- start 2024-09-16T01:36:46.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.835+0000 7fe1b7c8b740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b8d1920 0x55962b924f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.835+0000 7fe1b7c8b740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55962b8d30b0 0x55962b993d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.835+0000 7fe1b7c8b740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55962b911d00 0x55962b8fa6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.835+0000 7fe1b7c8b740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b01ffcb0 con 0x55962b8d1920 2024-09-16T01:36:46.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.835+0000 7fe1b7c8b740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55962b8fb3e0 con 0x55962b8d30b0 2024-09-16T01:36:46.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.835+0000 7fe1b7c8b740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55962b925ca0 con 0x55962b911d00 2024-09-16T01:36:46.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.835+0000 7fe1b7c8b740 10 client.0 _ll_register_callbacks cb 0x7fe1a803f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:36:46.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.835+0000 7fe1ae59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b8d1920 0x55962b924f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.835+0000 7fe1ae59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b8d1920 0x55962b924f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41528/0 (socket says 172.21.15.88:41528) 2024-09-16T01:36:46.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.835+0000 7fe1ae59c640 1 -- 172.21.15.88:0/1690377628 learned_addr learned my addr 172.21.15.88:0/1690377628 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:46.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.835+0000 7fe1add9b640 1 --2- 172.21.15.88:0/1690377628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55962b8d30b0 0x55962b993d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.835+0000 7fe1aed9d640 1 --2- 172.21.15.88:0/1690377628 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55962b911d00 0x55962b8fa6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.835+0000 7fe1ae59c640 1 -- 172.21.15.88:0/1690377628 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55962b911d00 msgr2=0x55962b8fa6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:46.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.835+0000 7fe1ae59c640 1 --2- 172.21.15.88:0/1690377628 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55962b911d00 0x55962b8fa6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.835+0000 7fe1ae59c640 1 -- 172.21.15.88:0/1690377628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55962b8d30b0 msgr2=0x55962b993d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:46.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.835+0000 7fe1ae59c640 1 --2- 172.21.15.88:0/1690377628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55962b8d30b0 0x55962b993d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.835+0000 7fe1ae59c640 1 -- 172.21.15.88:0/1690377628 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55962b92fef0 con 0x55962b8d1920 2024-09-16T01:36:46.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.835+0000 7fe1aed9d640 1 --2- 172.21.15.88:0/1690377628 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55962b911d00 0x55962b8fa6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:46.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.835+0000 7fe1ae59c640 1 --2- 172.21.15.88:0/1690377628 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b8d1920 0x55962b924f90 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe1a001c000 tx=0x7fe1a0063430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94be369954eeb8fd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:46.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.835+0000 7fe18d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:36:46.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.836+0000 7fe18d7fa640 1 -- 172.21.15.88:0/1690377628 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a005aa10 con 0x55962b8d1920 2024-09-16T01:36:46.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.836+0000 7fe18d7fa640 1 -- 172.21.15.88:0/1690377628 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7fe1a005abb0 con 0x55962b8d1920 2024-09-16T01:36:46.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.836+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/1690377628 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1a4027020 con 0x55962b8d1920 2024-09-16T01:36:46.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.836+0000 7fe18d7fa640 1 -- 172.21.15.88:0/1690377628 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fe1a0080050 con 0x55962b8d1920 2024-09-16T01:36:46.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.836+0000 7fe1b7c8b740 10 client.6033 fetch_fsmap learned FSMap version 37 2024-09-16T01:36:46.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.836+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/1690377628 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) -- 0x55962b8f5360 con 0x55962b8d1920 2024-09-16T01:36:46.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.836+0000 7fe18d7fa640 1 -- 172.21.15.88:0/1690377628 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1a4027020 con 0x55962b8d1920 2024-09-16T01:36:46.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.836+0000 7fe18d7fa640 1 -- 172.21.15.88:0/1690377628 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 37) ==== 3502+0+0 (secure 0 0 0) 0x7fe1a0083020 con 0x55962b8d1920 2024-09-16T01:36:46.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.836+0000 7fe1b7c8b740 10 client.6033 fetch_fsmap finished waiting for FSMap version 37 2024-09-16T01:36:46.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.836+0000 7fe1b7c8b740 10 client.6033 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-09-16T01:36:46.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.836+0000 7fe1b7c8b740 20 client.6033 populate_metadata read hostname 'smithi088' 2024-09-16T01:36:46.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.836+0000 7fe1b7c8b740 1 --2- 172.21.15.88:0/1690377628 >> [v2:172.21.15.88:6835/574169866,v1:172.21.15.88:6837/574169866] conn(0x55962b8fad10 0x55962b919d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.836+0000 7fe1b7c8b740 4 client.6033 mds_command: new command op to 5862 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:36:46.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.836+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/1690377628 --> [v2:172.21.15.88:6835/574169866,v1:172.21.15.88:6837/574169866] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55962b8fce50 con 0x55962b8fad10 2024-09-16T01:36:46.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.836+0000 7fe1add9b640 1 --2- 172.21.15.88:0/1690377628 >> [v2:172.21.15.88:6835/574169866,v1:172.21.15.88:6837/574169866] conn(0x55962b8fad10 0x55962b919d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.836+0000 7fe18d7fa640 10 client.6033 ms_handle_connect on v2:172.21.15.88:6835/574169866 2024-09-16T01:36:46.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.836+0000 7fe1add9b640 1 --2- 172.21.15.88:0/1690377628 >> [v2:172.21.15.88:6835/574169866,v1:172.21.15.88:6837/574169866] conn(0x55962b8fad10 0x55962b919d80 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e72ff5845e990f9 server_cookie=a6dc465e73b16b2a in_seq=0 out_seq=0 2024-09-16T01:36:46.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.854+0000 7fe18d7fa640 1 -- 172.21.15.88:0/1690377628 <== mds.0 v2:172.21.15.88:6835/574169866 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55962b8fce50 con 0x55962b8fad10 2024-09-16T01:36:46.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.855+0000 7fe18d7fa640 10 client.6033 handle_command_reply: tid=0 2024-09-16T01:36:46.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.855+0000 7fe1b7c8b740 1 client.6033 shutdown 2024-09-16T01:36:46.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.855+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/1690377628 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b8d1920 msgr2=0x55962b924f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:46.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.855+0000 7fe1b7c8b740 1 --2- 172.21.15.88:0/1690377628 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b8d1920 0x55962b924f90 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe1a001c000 tx=0x7fe1a0063430 comp rx=0 tx=0).stop 2024-09-16T01:36:46.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.855+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/1690377628 shutdown_connections 2024-09-16T01:36:46.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.855+0000 7fe1b7c8b740 1 --2- 172.21.15.88:0/1690377628 >> [v2:172.21.15.88:6835/574169866,v1:172.21.15.88:6837/574169866] conn(0x55962b8fad10 0x55962b919d80 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.855+0000 7fe1b7c8b740 1 --2- 172.21.15.88:0/1690377628 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55962b911d00 0x55962b8fa6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.855+0000 7fe1b7c8b740 1 --2- 172.21.15.88:0/1690377628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55962b8d30b0 0x55962b993d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.855+0000 7fe1b7c8b740 1 --2- 172.21.15.88:0/1690377628 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55962b8d1920 0x55962b924f90 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.855+0000 7fe18d7fa640 0 client.6033 ms_handle_reset on v2:172.21.15.88:6835/574169866 2024-09-16T01:36:46.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.855+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/1690377628 >> 172.21.15.88:0/1690377628 conn(0x55962b8cf0e0 msgr2=0x55962b8e7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:46.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.855+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/1690377628 shutdown_connections 2024-09-16T01:36:46.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.855+0000 7fe1b7c8b740 1 -- 172.21.15.88:0/1690377628 wait complete. 2024-09-16T01:36:46.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.855+0000 7fe1b7c8b740 20 client.6033 trim_cache size 0 max 16384 2024-09-16T01:36:46.858 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:36:46.858 INFO:teuthology.orchestra.run.smithi088.stdout: "message": "", 2024-09-16T01:36:46.858 INFO:teuthology.orchestra.run.smithi088.stdout: "return_code": 0 2024-09-16T01:36:46.858 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:36:46.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.856+0000 7fe1b4e1e640 1 -- 172.21.15.88:0/2112179131 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe19006c2b0 msgr2=0x7fe19006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:46.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.856+0000 7fe1b4e1e640 1 --2- 172.21.15.88:0/2112179131 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe19006c2b0 0x7fe19006e770 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe1b00b3170 tx=0x7fe1a4009040 comp rx=0 tx=0).stop 2024-09-16T01:36:46.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.856+0000 7fe1b4e1e640 1 -- 172.21.15.88:0/2112179131 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe198004d40 msgr2=0x7fe1b01521f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:46.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.856+0000 7fe1b4e1e640 1 --2- 172.21.15.88:0/2112179131 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe198004d40 0x7fe1b01521f0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0011e30 tx=0x7fe1a000f580 comp rx=0 tx=0).stop 2024-09-16T01:36:46.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.856+0000 7fe1b4e1e640 1 -- 172.21.15.88:0/2112179131 shutdown_connections 2024-09-16T01:36:46.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.856+0000 7fe1b4e1e640 1 --2- 172.21.15.88:0/2112179131 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe19006c2b0 0x7fe19006e770 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.856+0000 7fe1b4e1e640 1 --2- 172.21.15.88:0/2112179131 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1b015b760 0x7fe1b015bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.856+0000 7fe1b4e1e640 1 --2- 172.21.15.88:0/2112179131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1b0071d80 0x7fe1b00721e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.856+0000 7fe1b4e1e640 1 --2- 172.21.15.88:0/2112179131 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe198004d40 0x7fe1b01521f0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.856+0000 7fe1b4e1e640 1 -- 172.21.15.88:0/2112179131 >> 172.21.15.88:0/2112179131 conn(0x7fe1b006d0f0 msgr2=0x7fe1b00d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:46.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.856+0000 7fe1b4e1e640 1 -- 172.21.15.88:0/2112179131 shutdown_connections 2024-09-16T01:36:46.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.856+0000 7fe1b4e1e640 1 -- 172.21.15.88:0/2112179131 wait complete. 2024-09-16T01:36:46.868 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:36:46.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.964+0000 7f2046998640 1 Processor -- start 2024-09-16T01:36:46.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.964+0000 7f2046998640 1 -- start start 2024-09-16T01:36:46.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.965+0000 7f2046998640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f20400b4920 0x7f204015b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.965+0000 7f2046998640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20400b4d00 0x7f20400b5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.965+0000 7f2046998640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20400ce4f0 con 0x7f20400cd780 2024-09-16T01:36:46.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.965+0000 7f2046998640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f20400cf100 con 0x7f20400b4920 2024-09-16T01:36:46.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.965+0000 7f2046998640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f20400ce7d0 con 0x7f20400b4d00 2024-09-16T01:36:46.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.965+0000 7f203ffff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f20400b4920 0x7f204015b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.965+0000 7f203f7fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20400b4d00 0x7f20400b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.965+0000 7f2044f35640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f20400cd780 0x7f20400cdbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:54182/0 (socket says 172.21.15.88:54182) 2024-09-16T01:36:46.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.965+0000 7f2044f35640 1 -- 172.21.15.88:0/453243457 learned_addr learned my addr 172.21.15.88:0/453243457 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:46.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.966+0000 7f203f7fe640 1 -- 172.21.15.88:0/453243457 >> v2:172.21.15.154:3300/0 conn(0x7f20400b4920 msgr2=0x7f204015b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:46.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.966+0000 7f203f7fe640 1 --2- 172.21.15.88:0/453243457 >> v2:172.21.15.154:3300/0 conn(0x7f20400b4920 0x7f204015b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.966+0000 7f203f7fe640 1 -- 172.21.15.88:0/453243457 >> v1:172.21.15.154:6789/0 conn(0x7f20400cd780 legacy=0x7f20400cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:46.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.966+0000 7f203f7fe640 1 -- 172.21.15.88:0/453243457 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20400ceab0 con 0x7f20400b4d00 2024-09-16T01:36:46.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.966+0000 7f203effd640 1 -- 172.21.15.88:0/453243457 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3137774725 0 0) 0x7f20400ce4f0 con 0x7f20400cd780 2024-09-16T01:36:46.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.966+0000 7f203ffff640 1 --2- 172.21.15.88:0/453243457 >> v2:172.21.15.154:3300/0 conn(0x7f20400b4920 0x7f204015b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:36:46.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.967+0000 7f203f7fe640 1 --2- 172.21.15.88:0/453243457 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20400b4d00 0x7f20400b5140 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f20340097a0 tx=0x7f203402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c2a0fcb4a3f28df server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:46.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.967+0000 7f203effd640 1 -- 172.21.15.88:0/453243457 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2034037070 con 0x7f20400b4d00 2024-09-16T01:36:46.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.967+0000 7f203effd640 1 -- 172.21.15.88:0/453243457 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2034032440 con 0x7f20400b4d00 2024-09-16T01:36:46.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.967+0000 7f203effd640 1 -- 172.21.15.88:0/453243457 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2034032810 con 0x7f20400b4d00 2024-09-16T01:36:46.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.967+0000 7f2046998640 1 -- 172.21.15.88:0/453243457 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20400b4d00 msgr2=0x7f20400b5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:46.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.967+0000 7f2046998640 1 --2- 172.21.15.88:0/453243457 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20400b4d00 0x7f20400b5140 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f20340097a0 tx=0x7f203402a180 comp rx=0 tx=0).stop 2024-09-16T01:36:46.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.967+0000 7f2046998640 1 -- 172.21.15.88:0/453243457 shutdown_connections 2024-09-16T01:36:46.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.967+0000 7f2046998640 1 --2- 172.21.15.88:0/453243457 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20400b4d00 0x7f20400b5140 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.967+0000 7f2046998640 1 --2- 172.21.15.88:0/453243457 >> v2:172.21.15.154:3300/0 conn(0x7f20400b4920 0x7f204015b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.967+0000 7f2046998640 1 -- 172.21.15.88:0/453243457 >> 172.21.15.88:0/453243457 conn(0x7f204006bc10 msgr2=0x7f204006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:46.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.967+0000 7f2046998640 1 -- 172.21.15.88:0/453243457 shutdown_connections 2024-09-16T01:36:46.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.967+0000 7f2046998640 1 -- 172.21.15.88:0/453243457 wait complete. 2024-09-16T01:36:46.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.968+0000 7f2046998640 1 Processor -- start 2024-09-16T01:36:46.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.969+0000 7f2046998640 1 -- start start 2024-09-16T01:36:46.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.969+0000 7f2046998640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20400b4920 0x7f204014ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.969+0000 7f2046998640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20400b4d00 0x7f2040154750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.969+0000 7f2046998640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20400cd780 0x7f2040156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.969+0000 7f2046998640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f20400b5680 con 0x7f20400b4920 2024-09-16T01:36:46.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.969+0000 7f2046998640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f204015bd10 con 0x7f20400cd780 2024-09-16T01:36:46.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.969+0000 7f2046998640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f20400cf590 con 0x7f20400b4d00 2024-09-16T01:36:46.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.969+0000 7f203ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20400b4920 0x7f204014ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.970+0000 7f203ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20400b4920 0x7f204014ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41538/0 (socket says 172.21.15.88:41538) 2024-09-16T01:36:46.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.970+0000 7f203ffff640 1 -- 172.21.15.88:0/102257044 learned_addr learned my addr 172.21.15.88:0/102257044 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:46.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.970+0000 7f203f7fe640 1 --2- 172.21.15.88:0/102257044 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20400b4d00 0x7f2040154750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.970+0000 7f2044f35640 1 --2- 172.21.15.88:0/102257044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20400cd780 0x7f2040156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.970+0000 7f203ffff640 1 -- 172.21.15.88:0/102257044 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20400b4d00 msgr2=0x7f2040154750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:46.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.970+0000 7f203ffff640 1 --2- 172.21.15.88:0/102257044 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20400b4d00 0x7f2040154750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.970+0000 7f203ffff640 1 -- 172.21.15.88:0/102257044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20400cd780 msgr2=0x7f2040156040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:46.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.970+0000 7f203ffff640 1 --2- 172.21.15.88:0/102257044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20400cd780 0x7f2040156040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:46.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.970+0000 7f203ffff640 1 -- 172.21.15.88:0/102257044 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20401532f0 con 0x7f20400b4920 2024-09-16T01:36:46.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.970+0000 7f203ffff640 1 --2- 172.21.15.88:0/102257044 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20400b4920 0x7f204014ddd0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f203000ebe0 tx=0x7f203000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:46.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.970+0000 7f203cff9640 1 -- 172.21.15.88:0/102257044 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2030017070 con 0x7f20400b4920 2024-09-16T01:36:46.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.970+0000 7f203cff9640 1 -- 172.21.15.88:0/102257044 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f203000cc30 con 0x7f20400b4920 2024-09-16T01:36:46.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.970+0000 7f2046998640 1 -- 172.21.15.88:0/102257044 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20401f45a0 con 0x7f20400b4920 2024-09-16T01:36:46.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.970+0000 7f203cff9640 1 -- 172.21.15.88:0/102257044 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2030013470 con 0x7f20400b4920 2024-09-16T01:36:46.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.970+0000 7f2046998640 1 -- 172.21.15.88:0/102257044 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20401f4b60 con 0x7f20400b4920 2024-09-16T01:36:46.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.971+0000 7f2046998640 1 -- 172.21.15.88:0/102257044 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20400cb220 con 0x7f20400b4920 2024-09-16T01:36:46.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.972+0000 7f203cff9640 1 -- 172.21.15.88:0/102257044 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f203001a070 con 0x7f20400b4920 2024-09-16T01:36:46.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.972+0000 7f203cff9640 1 --2- 172.21.15.88:0/102257044 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f202c06c2b0 0x7f202c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:46.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.972+0000 7f203cff9640 1 -- 172.21.15.88:0/102257044 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f203008b3e0 con 0x7f20400b4920 2024-09-16T01:36:46.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.972+0000 7f203f7fe640 1 --2- 172.21.15.88:0/102257044 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f202c06c2b0 0x7f202c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:46.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.972+0000 7f203f7fe640 1 --2- 172.21.15.88:0/102257044 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f202c06c2b0 0x7f202c06e770 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2034034000 tx=0x7f203402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:46.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:46.976+0000 7f203cff9640 1 -- 172.21.15.88:0/102257044 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f20401532f0 con 0x7f20400b4920 2024-09-16T01:36:47.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:47.175+0000 7f2046998640 1 -- 172.21.15.88:0/102257044 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f204015b7d0 con 0x7f20400b4920 2024-09-16T01:36:47.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:47.175+0000 7f203cff9640 1 -- 172.21.15.88:0/102257044 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v37) ==== 53+0+73 (secure 0 0 0) 0x7f2030059b50 con 0x7f20400b4920 2024-09-16T01:36:47.176 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:36:47.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:47.177+0000 7f2046998640 1 -- 172.21.15.88:0/102257044 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f202c06c2b0 msgr2=0x7f202c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:47.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:47.177+0000 7f2046998640 1 --2- 172.21.15.88:0/102257044 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f202c06c2b0 0x7f202c06e770 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2034034000 tx=0x7f203402e000 comp rx=0 tx=0).stop 2024-09-16T01:36:47.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:47.177+0000 7f2046998640 1 -- 172.21.15.88:0/102257044 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20400b4920 msgr2=0x7f204014ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:47.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:47.177+0000 7f2046998640 1 --2- 172.21.15.88:0/102257044 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20400b4920 0x7f204014ddd0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f203000ebe0 tx=0x7f203000c520 comp rx=0 tx=0).stop 2024-09-16T01:36:47.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:47.177+0000 7f2046998640 1 -- 172.21.15.88:0/102257044 shutdown_connections 2024-09-16T01:36:47.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:47.177+0000 7f2046998640 1 --2- 172.21.15.88:0/102257044 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f202c06c2b0 0x7f202c06e770 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:47.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:47.177+0000 7f2046998640 1 --2- 172.21.15.88:0/102257044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20400cd780 0x7f2040156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:47.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:47.177+0000 7f2046998640 1 --2- 172.21.15.88:0/102257044 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20400b4d00 0x7f2040154750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:47.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:47.177+0000 7f2046998640 1 --2- 172.21.15.88:0/102257044 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20400b4920 0x7f204014ddd0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:47.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:47.177+0000 7f2046998640 1 -- 172.21.15.88:0/102257044 >> 172.21.15.88:0/102257044 conn(0x7f204006bc10 msgr2=0x7f20400cf9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:47.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:47.177+0000 7f2046998640 1 -- 172.21.15.88:0/102257044 shutdown_connections 2024-09-16T01:36:47.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:47.177+0000 7f2046998640 1 -- 172.21.15.88:0/102257044 wait complete. 2024-09-16T01:36:47.189 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-16T01:36:47.189 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:36:47.189 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2024-09-16T01:36:47.189 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T01:36:47.189 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T01:36:47.189 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-16T01:36:47.190 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-16T01:36:47.190 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:36:47.190 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:36:47.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:47.190 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2024-09-16T01:36:47.205 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:36:47.205 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:36:47.205 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:36:47.206 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:36:47.206 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-09-16T01:36:47.206 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:36:47.206 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:36:47.206 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:36:47.206 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2024-09-16T01:36:47.206 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:36:47.206 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:36:47.206 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2024-09-16T01:36:47.206 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:36:47.206 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2024-09-16T01:36:47.206 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:36:47.206 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2024-09-16T01:36:47.207 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:36:47.207 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42617sec preferred_lft 42617sec 2024-09-16T01:36:47.207 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link noprefixroute 2024-09-16T01:36:47.207 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:36:47.207 INFO:teuthology.orchestra.run.smithi088.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:36:47.207 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff 2024-09-16T01:36:47.207 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T01:36:47.207 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:36:47.207 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::e486:6eff:fe81:a1c/64 scope link 2024-09-16T01:36:47.207 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:36:47.207 INFO:teuthology.orchestra.run.smithi088.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:36:47.207 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 12:6f:29:be:95:48 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:36:47.207 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::106f:29ff:febe:9548/64 scope link 2024-09-16T01:36:47.207 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:36:47.208 INFO:teuthology.orchestra.run.smithi088.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:36:47.208 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T01:36:47.208 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::c63:b4ff:feb3:ad9e/64 scope link 2024-09-16T01:36:47.208 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:36:47.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:47.208 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:36:47.269 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-09-16T01:36:47.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:47.271 DEBUG:teuthology.orchestra.run.smithi088:> route 2024-09-16T01:36:47.327 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2024-09-16T01:36:47.327 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:36:47.327 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:36:47.327 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:36:47.327 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:36:47.328 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:36:47.328 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:36:47.328 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:36:47.328 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:36:47.328 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:36:47.328 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:36:47.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:36:47.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:36:47.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:47.453 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:36:47.469 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T01:36:47.469 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T01:36:47.469 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:36:47.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:47.470 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2024-09-16T01:36:47.533 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:36:47.534 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T01:36:47.589 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:36:47.590 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:36:47.590 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:36:47.645 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-16T01:36:47.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:47.646 DEBUG:teuthology.orchestra.run.smithi088:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:36:47.729 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:36:47.729 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:36:47.729 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:36:47.737 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:36:47.737 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:36:47.738 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:36:47.738 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:36:47.738 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:36:47.743 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:36:47.743 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:36:47.743 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:36:47.743 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:36:47.743 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:36:47.743 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790" 2024-09-16T01:36:47.743 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2024-09-16T01:36:47.743 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5ace2c91-46f9-482b-8a89-731fbdc171bf" will pass to kernel 2024-09-16T01:36:47.744 INFO:teuthology.orchestra.run.smithi088.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-16T01:36:47.744 INFO:teuthology.orchestra.run.smithi088.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:36:47.744 INFO:teuthology.orchestra.run.smithi088.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:36:47.744 INFO:teuthology.orchestra.run.smithi088.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:36:47.744 INFO:teuthology.orchestra.run.smithi088.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:36:47.745 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:36:47.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:47.746 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:36:47.810 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:36:47.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:47 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:36:47.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:36:47.907 INFO:teuthology.orchestra.run.smithi088.stdout:client5455 2024-09-16T01:36:47.907 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:36:47.907 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:36:47.924 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'chmod 711 a/b/d; sync .') 2024-09-16T01:36:48.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:36:48.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:36:48.021 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:36:48.021 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:36:48.037 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:36:48.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.175+0000 7f71eb366640 1 Processor -- start 2024-09-16T01:36:48.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.176+0000 7f71eb366640 1 -- start start 2024-09-16T01:36:48.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.176+0000 7f71eb366640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e415e4f0 0x7f71e40ce680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:48.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.176+0000 7f71eb366640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f71e415e8d0 0x7f71e40b3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:48.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.176+0000 7f71eb366640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f71e40afcd0 0x7f71e40b4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:48.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.176+0000 7f71eb366640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f71e40717d0 con 0x7f71e40afcd0 2024-09-16T01:36:48.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.176+0000 7f71eb366640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f71e40af8c0 con 0x7f71e415e8d0 2024-09-16T01:36:48.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.176+0000 7f71eb366640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f71e4071a90 con 0x7f71e415e4f0 2024-09-16T01:36:48.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.177+0000 7f71e9903640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f71e40afcd0 0x7f71e40b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:48.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.177+0000 7f71e9903640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f71e40afcd0 0x7f71e40b4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41544/0 (socket says 172.21.15.88:41544) 2024-09-16T01:36:48.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.177+0000 7f71e9903640 1 -- 172.21.15.88:0/966097570 learned_addr learned my addr 172.21.15.88:0/966097570 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:48.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.177+0000 7f71e9102640 1 --2- 172.21.15.88:0/966097570 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e415e4f0 0x7f71e40ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:48.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.177+0000 7f71e8901640 1 --2- 172.21.15.88:0/966097570 >> v2:172.21.15.154:3300/0 conn(0x7f71e415e8d0 0x7f71e40b3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:48.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.177+0000 7f71e9903640 1 -- 172.21.15.88:0/966097570 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e415e4f0 msgr2=0x7f71e40ce680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:48.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.177+0000 7f71e9903640 1 --2- 172.21.15.88:0/966097570 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e415e4f0 0x7f71e40ce680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:48.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.177+0000 7f71e9903640 1 -- 172.21.15.88:0/966097570 >> v2:172.21.15.154:3300/0 conn(0x7f71e415e8d0 msgr2=0x7f71e40b3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:48.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.177+0000 7f71e9903640 1 --2- 172.21.15.88:0/966097570 >> v2:172.21.15.154:3300/0 conn(0x7f71e415e8d0 0x7f71e40b3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:48.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.177+0000 7f71e9903640 1 -- 172.21.15.88:0/966097570 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71e40af340 con 0x7f71e40afcd0 2024-09-16T01:36:48.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.177+0000 7f71e9102640 1 --2- 172.21.15.88:0/966097570 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e415e4f0 0x7f71e40ce680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:48.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.177+0000 7f71e8901640 1 --2- 172.21.15.88:0/966097570 >> v2:172.21.15.154:3300/0 conn(0x7f71e415e8d0 0x7f71e40b3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:48.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.178+0000 7f71e9903640 1 --2- 172.21.15.88:0/966097570 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e40afcd0 0x7f71e40b4340 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f71e406f8c0 tx=0x7f71dc02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd9b7b63df18a696 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:48.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.178+0000 7f71d7fff640 1 -- 172.21.15.88:0/966097570 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71dc036070 con 0x7f71e40afcd0 2024-09-16T01:36:48.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.178+0000 7f71d7fff640 1 -- 172.21.15.88:0/966097570 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71dc031600 con 0x7f71e40afcd0 2024-09-16T01:36:48.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.178+0000 7f71d7fff640 1 -- 172.21.15.88:0/966097570 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71dc0319d0 con 0x7f71e40afcd0 2024-09-16T01:36:48.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.178+0000 7f71eb366640 1 -- 172.21.15.88:0/966097570 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e40afcd0 msgr2=0x7f71e40b4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:48.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.178+0000 7f71eb366640 1 --2- 172.21.15.88:0/966097570 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e40afcd0 0x7f71e40b4340 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f71e406f8c0 tx=0x7f71dc02a6a0 comp rx=0 tx=0).stop 2024-09-16T01:36:48.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.178+0000 7f71eb366640 1 -- 172.21.15.88:0/966097570 shutdown_connections 2024-09-16T01:36:48.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.178+0000 7f71eb366640 1 --2- 172.21.15.88:0/966097570 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e40afcd0 0x7f71e40b4340 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:48.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.178+0000 7f71eb366640 1 --2- 172.21.15.88:0/966097570 >> v2:172.21.15.154:3300/0 conn(0x7f71e415e8d0 0x7f71e40b3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:48.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.178+0000 7f71eb366640 1 --2- 172.21.15.88:0/966097570 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e415e4f0 0x7f71e40ce680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:48.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.178+0000 7f71eb366640 1 -- 172.21.15.88:0/966097570 >> 172.21.15.88:0/966097570 conn(0x7f71e406bc10 msgr2=0x7f71e406bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:48.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.178+0000 7f71eb366640 1 -- 172.21.15.88:0/966097570 shutdown_connections 2024-09-16T01:36:48.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.178+0000 7f71eb366640 1 -- 172.21.15.88:0/966097570 wait complete. 2024-09-16T01:36:48.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.180+0000 7f71eb366640 1 Processor -- start 2024-09-16T01:36:48.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.180+0000 7f71eb366640 1 -- start start 2024-09-16T01:36:48.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.180+0000 7f71eb366640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71e40afcd0 0x7f71e414c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:48.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.180+0000 7f71eb366640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e415e4f0 0x7f71e40c66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:48.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.180+0000 7f71eb366640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e415e8d0 0x7f71e415bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:48.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.180+0000 7f71eb366640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f71e40b4880 con 0x7f71e415e8d0 2024-09-16T01:36:48.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.180+0000 7f71eb366640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f71e40b0240 con 0x7f71e40afcd0 2024-09-16T01:36:48.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.180+0000 7f71eb366640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f71e40b4a00 con 0x7f71e415e4f0 2024-09-16T01:36:48.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.180+0000 7f71e9903640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e415e8d0 0x7f71e415bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:48.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.180+0000 7f71e9903640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e415e8d0 0x7f71e415bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41548/0 (socket says 172.21.15.88:41548) 2024-09-16T01:36:48.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.180+0000 7f71e9903640 1 -- 172.21.15.88:0/730209142 learned_addr learned my addr 172.21.15.88:0/730209142 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:48.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.181+0000 7f71e9903640 1 -- 172.21.15.88:0/730209142 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e415e4f0 msgr2=0x7f71e40c66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:48.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.181+0000 7f71e8901640 1 --2- 172.21.15.88:0/730209142 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e415e4f0 0x7f71e40c66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:48.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.181+0000 7f71e9102640 1 --2- 172.21.15.88:0/730209142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71e40afcd0 0x7f71e414c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:48.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.181+0000 7f71e9903640 1 --2- 172.21.15.88:0/730209142 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e415e4f0 0x7f71e40c66c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:48.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.181+0000 7f71e9903640 1 -- 172.21.15.88:0/730209142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71e40afcd0 msgr2=0x7f71e414c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:48.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.181+0000 7f71e9903640 1 --2- 172.21.15.88:0/730209142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71e40afcd0 0x7f71e414c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:48.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.181+0000 7f71e9903640 1 -- 172.21.15.88:0/730209142 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71e415aa20 con 0x7f71e415e8d0 2024-09-16T01:36:48.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.181+0000 7f71e8901640 1 --2- 172.21.15.88:0/730209142 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e415e4f0 0x7f71e40c66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:48.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.181+0000 7f71e9903640 1 --2- 172.21.15.88:0/730209142 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e415e8d0 0x7f71e415bac0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f71dc034040 tx=0x7f71dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:48.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.181+0000 7f71d5ffb640 1 -- 172.21.15.88:0/730209142 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71dc036050 con 0x7f71e415e8d0 2024-09-16T01:36:48.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.181+0000 7f71d5ffb640 1 -- 172.21.15.88:0/730209142 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71dc042070 con 0x7f71e415e8d0 2024-09-16T01:36:48.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.181+0000 7f71eb366640 1 -- 172.21.15.88:0/730209142 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71e415ad00 con 0x7f71e415e8d0 2024-09-16T01:36:48.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.181+0000 7f71d5ffb640 1 -- 172.21.15.88:0/730209142 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71dc009510 con 0x7f71e415e8d0 2024-09-16T01:36:48.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.181+0000 7f71eb366640 1 -- 172.21.15.88:0/730209142 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71e415b2c0 con 0x7f71e415e8d0 2024-09-16T01:36:48.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.182+0000 7f71eb366640 1 -- 172.21.15.88:0/730209142 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71e415c000 con 0x7f71e415e8d0 2024-09-16T01:36:48.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.183+0000 7f71d5ffb640 1 -- 172.21.15.88:0/730209142 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f71dc007bd0 con 0x7f71e415e8d0 2024-09-16T01:36:48.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.183+0000 7f71d5ffb640 1 --2- 172.21.15.88:0/730209142 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f71c806c2b0 0x7f71c806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:48.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.183+0000 7f71d5ffb640 1 -- 172.21.15.88:0/730209142 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7f71dc0ac280 con 0x7f71e415e8d0 2024-09-16T01:36:48.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.183+0000 7f71e9102640 1 --2- 172.21.15.88:0/730209142 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f71c806c2b0 0x7f71c806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:48.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.184+0000 7f71e9102640 1 --2- 172.21.15.88:0/730209142 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f71c806c2b0 0x7f71c806e770 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f71d800d8b0 tx=0x7f71d8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:48.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.188+0000 7f71d5ffb640 1 -- 172.21.15.88:0/730209142 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f71e415c000 con 0x7f71e415e8d0 2024-09-16T01:36:48.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.364+0000 7f71eb366640 1 -- 172.21.15.88:0/730209142 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f71e40d02d0 con 0x7f71e415e8d0 2024-09-16T01:36:48.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.364+0000 7f71d5ffb640 1 -- 172.21.15.88:0/730209142 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v58) ==== 80+0+0 (secure 0 0 0) 0x7f71dc07aa80 con 0x7f71e415e8d0 2024-09-16T01:36:48.366 INFO:teuthology.orchestra.run.smithi088.stderr:listed 0 entries 2024-09-16T01:36:48.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.366+0000 7f71eb366640 1 -- 172.21.15.88:0/730209142 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f71c806c2b0 msgr2=0x7f71c806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:48.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.366+0000 7f71eb366640 1 --2- 172.21.15.88:0/730209142 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f71c806c2b0 0x7f71c806e770 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f71d800d8b0 tx=0x7f71d8008040 comp rx=0 tx=0).stop 2024-09-16T01:36:48.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.366+0000 7f71eb366640 1 -- 172.21.15.88:0/730209142 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e415e8d0 msgr2=0x7f71e415bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:48.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.366+0000 7f71eb366640 1 --2- 172.21.15.88:0/730209142 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e415e8d0 0x7f71e415bac0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f71dc034040 tx=0x7f71dc004a40 comp rx=0 tx=0).stop 2024-09-16T01:36:48.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.366+0000 7f71eb366640 1 -- 172.21.15.88:0/730209142 shutdown_connections 2024-09-16T01:36:48.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.366+0000 7f71eb366640 1 --2- 172.21.15.88:0/730209142 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f71c806c2b0 0x7f71c806e770 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:48.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.366+0000 7f71eb366640 1 --2- 172.21.15.88:0/730209142 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71e415e8d0 0x7f71e415bac0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:48.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.366+0000 7f71eb366640 1 --2- 172.21.15.88:0/730209142 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71e415e4f0 0x7f71e40c66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:48.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.366+0000 7f71eb366640 1 --2- 172.21.15.88:0/730209142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71e40afcd0 0x7f71e414c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:48.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.366+0000 7f71eb366640 1 -- 172.21.15.88:0/730209142 >> 172.21.15.88:0/730209142 conn(0x7f71e406bc10 msgr2=0x7f71e40b11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:48.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.367+0000 7f71eb366640 1 -- 172.21.15.88:0/730209142 shutdown_connections 2024-09-16T01:36:48.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.367+0000 7f71eb366640 1 -- 172.21.15.88:0/730209142 wait complete. 2024-09-16T01:36:48.378 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-16T01:36:48.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:48.379 DEBUG:teuthology.orchestra.run.smithi088:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-16T01:36:48.411 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:36:48.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:36:48.411 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:36:48.466 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:36:48.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.607+0000 7feca24e6640 1 Processor -- start 2024-09-16T01:36:48.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.607+0000 7feca24e6640 1 -- start start 2024-09-16T01:36:48.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.607+0000 7feca24e6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fec9c15aa00 0x7fec9c0ce390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:48.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.608+0000 7feca24e6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec9c0ce9a0 con 0x7fec9c0b5830 2024-09-16T01:36:48.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.608+0000 7feca24e6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec9c0cec80 con 0x7fec9c071720 2024-09-16T01:36:48.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.608+0000 7feca24e6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fec9c071b90 con 0x7fec9c15aa00 2024-09-16T01:36:48.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.608+0000 7fec9bfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fec9c15aa00 0x7fec9c0ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:48.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.608+0000 7feca0a83640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fec9c0b5830 0x7fec9c0cfa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54372/0 (socket says 172.21.15.88:54372) 2024-09-16T01:36:48.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.608+0000 7fec9bfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fec9c15aa00 0x7fec9c0ce390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41564/0 (socket says 172.21.15.88:41564) 2024-09-16T01:36:48.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.608+0000 7feca0a83640 1 -- 172.21.15.88:0/1638884770 learned_addr learned my addr 172.21.15.88:0/1638884770 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:48.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.608+0000 7fec9bfff640 1 -- 172.21.15.88:0/1638884770 >> v1:172.21.15.154:6789/0 conn(0x7fec9c071720 legacy=0x7fec9c0b4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:48.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.608+0000 7fec9bfff640 1 -- 172.21.15.88:0/1638884770 >> v1:172.21.15.88:6789/0 conn(0x7fec9c0b5830 legacy=0x7fec9c0cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:48.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.608+0000 7fec9bfff640 1 -- 172.21.15.88:0/1638884770 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec9c0cef60 con 0x7fec9c15aa00 2024-09-16T01:36:48.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.609+0000 7fec9bfff640 1 --2- 172.21.15.88:0/1638884770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec9c15aa00 0x7fec9c0ce390 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fec8c0097a0 tx=0x7fec8c02b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=347ae3558ddb3058 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:48.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.609+0000 7fec9b7fe640 1 -- 172.21.15.88:0/1638884770 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec8c038070 con 0x7fec9c15aa00 2024-09-16T01:36:48.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.609+0000 7fec9b7fe640 1 -- 172.21.15.88:0/1638884770 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec8c033640 con 0x7fec9c15aa00 2024-09-16T01:36:48.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.609+0000 7fec9b7fe640 1 -- 172.21.15.88:0/1638884770 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec8c033a10 con 0x7fec9c15aa00 2024-09-16T01:36:48.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.609+0000 7feca24e6640 1 -- 172.21.15.88:0/1638884770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec9c15aa00 msgr2=0x7fec9c0ce390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:48.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.609+0000 7feca24e6640 1 --2- 172.21.15.88:0/1638884770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec9c15aa00 0x7fec9c0ce390 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fec8c0097a0 tx=0x7fec8c02b340 comp rx=0 tx=0).stop 2024-09-16T01:36:48.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.609+0000 7feca24e6640 1 -- 172.21.15.88:0/1638884770 shutdown_connections 2024-09-16T01:36:48.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.609+0000 7feca24e6640 1 --2- 172.21.15.88:0/1638884770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec9c15aa00 0x7fec9c0ce390 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:48.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.609+0000 7feca24e6640 1 -- 172.21.15.88:0/1638884770 >> 172.21.15.88:0/1638884770 conn(0x7fec9c0b0aa0 msgr2=0x7fec9c0b0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:48.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.610+0000 7feca24e6640 1 -- 172.21.15.88:0/1638884770 shutdown_connections 2024-09-16T01:36:48.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.610+0000 7feca24e6640 1 -- 172.21.15.88:0/1638884770 wait complete. 2024-09-16T01:36:48.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.611+0000 7feca24e6640 1 Processor -- start 2024-09-16T01:36:48.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.611+0000 7feca24e6640 1 -- start start 2024-09-16T01:36:48.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.611+0000 7feca24e6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fec9c071720 0x7fec9c152520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:48.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.611+0000 7feca24e6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec9c0b5830 0x7fec9c1f1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:48.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.611+0000 7feca24e6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec9c15aa00 0x7fec9c158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:48.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.611+0000 7feca24e6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fec9c15ade0 con 0x7fec9c15aa00 2024-09-16T01:36:48.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.611+0000 7feca24e6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fec9c0cf5b0 con 0x7fec9c0b5830 2024-09-16T01:36:48.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.611+0000 7feca24e6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fec9c15af60 con 0x7fec9c071720 2024-09-16T01:36:48.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.611+0000 7feca0a83640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec9c15aa00 0x7fec9c158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:48.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.612+0000 7feca0a83640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec9c15aa00 0x7fec9c158fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41572/0 (socket says 172.21.15.88:41572) 2024-09-16T01:36:48.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.612+0000 7feca0a83640 1 -- 172.21.15.88:0/2357740373 learned_addr learned my addr 172.21.15.88:0/2357740373 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:48.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.612+0000 7fec9bfff640 1 --2- 172.21.15.88:0/2357740373 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fec9c071720 0x7fec9c152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:48.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.612+0000 7fec93fff640 1 --2- 172.21.15.88:0/2357740373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec9c0b5830 0x7fec9c1f1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:48.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.612+0000 7feca0a83640 1 -- 172.21.15.88:0/2357740373 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fec9c071720 msgr2=0x7fec9c152520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:48.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.612+0000 7feca0a83640 1 --2- 172.21.15.88:0/2357740373 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fec9c071720 0x7fec9c152520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:48.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.612+0000 7feca0a83640 1 -- 172.21.15.88:0/2357740373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec9c0b5830 msgr2=0x7fec9c1f1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:48.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.612+0000 7feca0a83640 1 --2- 172.21.15.88:0/2357740373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec9c0b5830 0x7fec9c1f1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:48.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.612+0000 7feca0a83640 1 -- 172.21.15.88:0/2357740373 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec9c159ab0 con 0x7fec9c15aa00 2024-09-16T01:36:48.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.612+0000 7feca0a83640 1 --2- 172.21.15.88:0/2357740373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec9c15aa00 0x7fec9c158fd0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fec9400cbf0 tx=0x7fec9400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:48.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.612+0000 7fec997fa640 1 -- 172.21.15.88:0/2357740373 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec94017070 con 0x7fec9c15aa00 2024-09-16T01:36:48.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.612+0000 7fec997fa640 1 -- 172.21.15.88:0/2357740373 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec9400ad40 con 0x7fec9c15aa00 2024-09-16T01:36:48.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.612+0000 7feca24e6640 1 -- 172.21.15.88:0/2357740373 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec9c14d720 con 0x7fec9c15aa00 2024-09-16T01:36:48.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.612+0000 7fec997fa640 1 -- 172.21.15.88:0/2357740373 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec94012600 con 0x7fec9c15aa00 2024-09-16T01:36:48.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.612+0000 7feca24e6640 1 -- 172.21.15.88:0/2357740373 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec9c14dce0 con 0x7fec9c15aa00 2024-09-16T01:36:48.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.613+0000 7feca24e6640 1 -- 172.21.15.88:0/2357740373 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec9c0ce390 con 0x7fec9c15aa00 2024-09-16T01:36:48.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.614+0000 7fec997fa640 1 -- 172.21.15.88:0/2357740373 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fec9401a070 con 0x7fec9c15aa00 2024-09-16T01:36:48.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.614+0000 7fec997fa640 1 --2- 172.21.15.88:0/2357740373 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fec7c06c2b0 0x7fec7c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:48.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.614+0000 7fec9bfff640 1 --2- 172.21.15.88:0/2357740373 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fec7c06c2b0 0x7fec7c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:48.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.614+0000 7fec997fa640 1 -- 172.21.15.88:0/2357740373 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5709+0+0 (secure 0 0 0) 0x7fec9408b0f0 con 0x7fec9c15aa00 2024-09-16T01:36:48.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.615+0000 7fec9bfff640 1 --2- 172.21.15.88:0/2357740373 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fec7c06c2b0 0x7fec7c06e770 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fec8c035000 tx=0x7fec8c02f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:48.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.618+0000 7fec997fa640 1 -- 172.21.15.88:0/2357740373 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fec94059210 con 0x7fec9c15aa00 2024-09-16T01:36:48.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.819+0000 7feca24e6640 1 -- 172.21.15.88:0/2357740373 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fec9c071d10 con 0x7fec9c15aa00 2024-09-16T01:36:48.821 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:36:48.820+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:36:48.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.821+0000 7fec997fa640 1 -- 172.21.15.88:0/2357740373 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v38) ==== 161+0+0 (secure 0 0 0) 0x7fec9405d510 con 0x7fec9c15aa00 2024-09-16T01:36:48.822 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:36:48.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.824+0000 7feca24e6640 1 -- 172.21.15.88:0/2357740373 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fec7c06c2b0 msgr2=0x7fec7c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:48.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.824+0000 7feca24e6640 1 --2- 172.21.15.88:0/2357740373 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fec7c06c2b0 0x7fec7c06e770 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fec8c035000 tx=0x7fec8c02f000 comp rx=0 tx=0).stop 2024-09-16T01:36:48.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.824+0000 7feca24e6640 1 -- 172.21.15.88:0/2357740373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec9c15aa00 msgr2=0x7fec9c158fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:48.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.824+0000 7feca24e6640 1 --2- 172.21.15.88:0/2357740373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec9c15aa00 0x7fec9c158fd0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fec9400cbf0 tx=0x7fec9400a430 comp rx=0 tx=0).stop 2024-09-16T01:36:48.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.824+0000 7feca24e6640 1 -- 172.21.15.88:0/2357740373 shutdown_connections 2024-09-16T01:36:48.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.824+0000 7feca24e6640 1 --2- 172.21.15.88:0/2357740373 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fec7c06c2b0 0x7fec7c06e770 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:48.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.824+0000 7feca24e6640 1 --2- 172.21.15.88:0/2357740373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec9c15aa00 0x7fec9c158fd0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:48.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.824+0000 7feca24e6640 1 --2- 172.21.15.88:0/2357740373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec9c0b5830 0x7fec9c1f1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:48.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.824+0000 7feca24e6640 1 --2- 172.21.15.88:0/2357740373 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fec9c071720 0x7fec9c152520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:48.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.824+0000 7feca24e6640 1 -- 172.21.15.88:0/2357740373 >> 172.21.15.88:0/2357740373 conn(0x7fec9c0b0aa0 msgr2=0x7fec9c0b3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:48.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.824+0000 7feca24e6640 1 -- 172.21.15.88:0/2357740373 shutdown_connections 2024-09-16T01:36:48.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.824+0000 7feca24e6640 1 -- 172.21.15.88:0/2357740373 wait complete. 2024-09-16T01:36:48.831 INFO:tasks.ceph.mds.a.smithi088.stderr: -1848> 2024-09-16T01:36:46.441+0000 7f32886f9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:36:48.837 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_inject_journal_corrupt_dentry_first 1.0 2024-09-16T01:36:48.858 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:36:48.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.937+0000 7f3bc5dc4640 1 Processor -- start 2024-09-16T01:36:48.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.937+0000 7f3bc5dc4640 1 -- start start 2024-09-16T01:36:48.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.938+0000 7f3bc5dc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc006f710 0x7f3bc006fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:48.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.938+0000 7f3bc5dc4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3bc006bc10 0x7f3bc006c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:48.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.938+0000 7f3bc5dc4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bc0074120 con 0x7f3bc015c2a0 2024-09-16T01:36:48.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.938+0000 7f3bc5dc4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f3bc015c680 con 0x7f3bc006bc10 2024-09-16T01:36:48.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.938+0000 7f3bc5dc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3bc00700b0 con 0x7f3bc006f710 2024-09-16T01:36:48.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.938+0000 7f3bbffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3bc006bc10 0x7f3bc006c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:48.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.938+0000 7f3bbffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3bc006bc10 0x7f3bc006c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41592/0 (socket says 172.21.15.88:41592) 2024-09-16T01:36:48.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.938+0000 7f3bbffff640 1 -- 172.21.15.88:0/579537897 learned_addr learned my addr 172.21.15.88:0/579537897 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:48.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.939+0000 7f3bbeffd640 1 --2- 172.21.15.88:0/579537897 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc006f710 0x7f3bc006fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:48.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.939+0000 7f3bbffff640 1 -- 172.21.15.88:0/579537897 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc006f710 msgr2=0x7f3bc006fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:48.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.939+0000 7f3bbffff640 1 --2- 172.21.15.88:0/579537897 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc006f710 0x7f3bc006fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:48.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.939+0000 7f3bbffff640 1 -- 172.21.15.88:0/579537897 >> v1:172.21.15.154:6789/0 conn(0x7f3bc015c2a0 legacy=0x7f3bc00735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:48.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.939+0000 7f3bbffff640 1 -- 172.21.15.88:0/579537897 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc015ef00 con 0x7f3bc006bc10 2024-09-16T01:36:48.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.939+0000 7f3bbffff640 1 --2- 172.21.15.88:0/579537897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc006bc10 0x7f3bc006c0a0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8009e50 tx=0x7f3bb802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88adb97d68252e85 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:48.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.940+0000 7f3bbe7fc640 1 -- 172.21.15.88:0/579537897 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8036070 con 0x7f3bc006bc10 2024-09-16T01:36:48.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.940+0000 7f3bbe7fc640 1 -- 172.21.15.88:0/579537897 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bb8031600 con 0x7f3bc006bc10 2024-09-16T01:36:48.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.940+0000 7f3bbe7fc640 1 -- 172.21.15.88:0/579537897 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb80319d0 con 0x7f3bc006bc10 2024-09-16T01:36:48.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.940+0000 7f3bc5dc4640 1 -- 172.21.15.88:0/579537897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc006bc10 msgr2=0x7f3bc006c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:48.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.940+0000 7f3bc5dc4640 1 --2- 172.21.15.88:0/579537897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc006bc10 0x7f3bc006c0a0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8009e50 tx=0x7f3bb802a710 comp rx=0 tx=0).stop 2024-09-16T01:36:48.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.940+0000 7f3bc5dc4640 1 -- 172.21.15.88:0/579537897 shutdown_connections 2024-09-16T01:36:48.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.940+0000 7f3bc5dc4640 1 --2- 172.21.15.88:0/579537897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc006bc10 0x7f3bc006c0a0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:48.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.940+0000 7f3bc5dc4640 1 --2- 172.21.15.88:0/579537897 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc006f710 0x7f3bc006fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:48.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.940+0000 7f3bc5dc4640 1 -- 172.21.15.88:0/579537897 >> 172.21.15.88:0/579537897 conn(0x7f3bc00b3070 msgr2=0x7f3bc00b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:48.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.940+0000 7f3bc5dc4640 1 -- 172.21.15.88:0/579537897 shutdown_connections 2024-09-16T01:36:48.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.940+0000 7f3bc5dc4640 1 -- 172.21.15.88:0/579537897 wait complete. 2024-09-16T01:36:48.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.941+0000 7f3bc5dc4640 1 Processor -- start 2024-09-16T01:36:48.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.942+0000 7f3bc5dc4640 1 -- start start 2024-09-16T01:36:48.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.942+0000 7f3bc5dc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bc006bc10 0x7f3bc0152e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:48.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.942+0000 7f3bc5dc4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc006f710 0x7f3bc015b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:48.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.942+0000 7f3bc5dc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc015c2a0 0x7f3bc0160c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:48.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.942+0000 7f3bc5dc4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc015ebf0 con 0x7f3bc006f710 2024-09-16T01:36:48.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.942+0000 7f3bc5dc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc015f4c0 con 0x7f3bc006bc10 2024-09-16T01:36:48.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.942+0000 7f3bc5dc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3bc00b5f20 con 0x7f3bc015c2a0 2024-09-16T01:36:48.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.942+0000 7f3bbeffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc006f710 0x7f3bc015b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:48.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.942+0000 7f3bbeffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc006f710 0x7f3bc015b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41604/0 (socket says 172.21.15.88:41604) 2024-09-16T01:36:48.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.942+0000 7f3bbeffd640 1 -- 172.21.15.88:0/2309099910 learned_addr learned my addr 172.21.15.88:0/2309099910 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:48.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.942+0000 7f3bbf7fe640 1 --2- 172.21.15.88:0/2309099910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bc006bc10 0x7f3bc0152e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:48.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.943+0000 7f3bbffff640 1 --2- 172.21.15.88:0/2309099910 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc015c2a0 0x7f3bc0160c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:48.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.943+0000 7f3bbeffd640 1 -- 172.21.15.88:0/2309099910 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc015c2a0 msgr2=0x7f3bc0160c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:48.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.943+0000 7f3bbeffd640 1 --2- 172.21.15.88:0/2309099910 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc015c2a0 0x7f3bc0160c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:48.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.943+0000 7f3bbeffd640 1 -- 172.21.15.88:0/2309099910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bc006bc10 msgr2=0x7f3bc0152e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:48.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.943+0000 7f3bbeffd640 1 --2- 172.21.15.88:0/2309099910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bc006bc10 0x7f3bc0152e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:48.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.943+0000 7f3bbeffd640 1 -- 172.21.15.88:0/2309099910 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc0159ae0 con 0x7f3bc006f710 2024-09-16T01:36:48.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.943+0000 7f3bbeffd640 1 --2- 172.21.15.88:0/2309099910 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc006f710 0x7f3bc015b2c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f3bb400d800 tx=0x7f3bb400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:48.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.943+0000 7f3ba7fff640 1 -- 172.21.15.88:0/2309099910 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb400b6c0 con 0x7f3bc006f710 2024-09-16T01:36:48.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.943+0000 7f3ba7fff640 1 -- 172.21.15.88:0/2309099910 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bb4019070 con 0x7f3bc006f710 2024-09-16T01:36:48.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.943+0000 7f3bc5dc4640 1 -- 172.21.15.88:0/2309099910 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bc0159dc0 con 0x7f3bc006f710 2024-09-16T01:36:48.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.943+0000 7f3ba7fff640 1 -- 172.21.15.88:0/2309099910 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb400bac0 con 0x7f3bc006f710 2024-09-16T01:36:48.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.943+0000 7f3bc5dc4640 1 -- 172.21.15.88:0/2309099910 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bc015a380 con 0x7f3bc006f710 2024-09-16T01:36:48.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.944+0000 7f3bc5dc4640 1 -- 172.21.15.88:0/2309099910 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bc015f9e0 con 0x7f3bc006f710 2024-09-16T01:36:48.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.945+0000 7f3ba7fff640 1 -- 172.21.15.88:0/2309099910 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3bb400bc60 con 0x7f3bc006f710 2024-09-16T01:36:48.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.945+0000 7f3ba7fff640 1 --2- 172.21.15.88:0/2309099910 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3bac06c2b0 0x7f3bac06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:48.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.945+0000 7f3bbf7fe640 1 --2- 172.21.15.88:0/2309099910 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3bac06c2b0 0x7f3bac06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:48.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.945+0000 7f3ba7fff640 1 -- 172.21.15.88:0/2309099910 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5795+0+0 (secure 0 0 0) 0x7f3bb408b8d0 con 0x7f3bc006f710 2024-09-16T01:36:48.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.946+0000 7f3bbf7fe640 1 --2- 172.21.15.88:0/2309099910 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3bac06c2b0 0x7f3bac06e770 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3bc00b2cf0 tx=0x7f3bb0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:48.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:48.949+0000 7f3ba7fff640 1 -- 172.21.15.88:0/2309099910 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3bb4059fe0 con 0x7f3bc006f710 2024-09-16T01:36:49.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:49.118+0000 7f3bc5dc4640 1 -- 172.21.15.88:0/2309099910 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command([{prefix=config set, name=mds_inject_journal_corrupt_dentry_first}] v 0) -- 0x7f3bc00aff00 con 0x7f3bc006f710 2024-09-16T01:36:49.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:49.121+0000 7f3ba7fff640 1 -- 172.21.15.88:0/2309099910 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_inject_journal_corrupt_dentry_first}]=0 v19)=0 v19) ==== 139+0+0 (secure 0 0 0) 0x7f3bb405ee20 con 0x7f3bc006f710 2024-09-16T01:36:49.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:49.123+0000 7f3bc5dc4640 1 -- 172.21.15.88:0/2309099910 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3bac06c2b0 msgr2=0x7f3bac06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:49.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:49.123+0000 7f3bc5dc4640 1 --2- 172.21.15.88:0/2309099910 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3bac06c2b0 0x7f3bac06e770 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3bc00b2cf0 tx=0x7f3bb0009040 comp rx=0 tx=0).stop 2024-09-16T01:36:49.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:49.123+0000 7f3bc5dc4640 1 -- 172.21.15.88:0/2309099910 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc006f710 msgr2=0x7f3bc015b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:49.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:49.123+0000 7f3bc5dc4640 1 --2- 172.21.15.88:0/2309099910 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc006f710 0x7f3bc015b2c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f3bb400d800 tx=0x7f3bb400dcc0 comp rx=0 tx=0).stop 2024-09-16T01:36:49.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:49.123+0000 7f3bc5dc4640 1 -- 172.21.15.88:0/2309099910 shutdown_connections 2024-09-16T01:36:49.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:49.123+0000 7f3bc5dc4640 1 --2- 172.21.15.88:0/2309099910 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3bac06c2b0 0x7f3bac06e770 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:49.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:49.123+0000 7f3bc5dc4640 1 --2- 172.21.15.88:0/2309099910 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc015c2a0 0x7f3bc0160c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:49.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:49.123+0000 7f3bc5dc4640 1 --2- 172.21.15.88:0/2309099910 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc006f710 0x7f3bc015b2c0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:49.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:49.123+0000 7f3bc5dc4640 1 --2- 172.21.15.88:0/2309099910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bc006bc10 0x7f3bc0152e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:49.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:49.123+0000 7f3bc5dc4640 1 -- 172.21.15.88:0/2309099910 >> 172.21.15.88:0/2309099910 conn(0x7f3bc00b3070 msgr2=0x7f3bc00b0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:49.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:49.123+0000 7f3bc5dc4640 1 -- 172.21.15.88:0/2309099910 shutdown_connections 2024-09-16T01:36:49.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:49.123+0000 7f3bc5dc4640 1 -- 172.21.15.88:0/2309099910 wait complete. 2024-09-16T01:36:54.136 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:36:54.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.249+0000 7f74e3cd0640 1 Processor -- start 2024-09-16T01:36:54.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.249+0000 7f74e3cd0640 1 -- start start 2024-09-16T01:36:54.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.250+0000 7f74e3cd0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74dc15e6f0 0x7f74dc06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:54.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.250+0000 7f74e3cd0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f74dc0b54d0 0x7f74dc15f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:54.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.250+0000 7f74e3cd0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74dc06e990 con 0x7f74dc0b6d40 2024-09-16T01:36:54.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.250+0000 7f74e3cd0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f74dc06ec70 con 0x7f74dc0b54d0 2024-09-16T01:36:54.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.250+0000 7f74e3cd0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f74dc15ec70 con 0x7f74dc15e6f0 2024-09-16T01:36:54.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.250+0000 7f74e126b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f74dc0b54d0 0x7f74dc15f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:54.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.250+0000 7f74e1a6c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74dc15e6f0 0x7f74dc06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:54.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.250+0000 7f74e226d640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f74dc0b6d40 0x7f74dc0b71f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:38940/0 (socket says 172.21.15.88:38940) 2024-09-16T01:36:54.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.250+0000 7f74e226d640 1 -- 172.21.15.88:0/79411243 learned_addr learned my addr 172.21.15.88:0/79411243 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.251+0000 7f74e1a6c640 1 -- 172.21.15.88:0/79411243 >> v2:172.21.15.154:3300/0 conn(0x7f74dc0b54d0 msgr2=0x7f74dc15f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.251+0000 7f74e1a6c640 1 --2- 172.21.15.88:0/79411243 >> v2:172.21.15.154:3300/0 conn(0x7f74dc0b54d0 0x7f74dc15f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.251+0000 7f74e1a6c640 1 -- 172.21.15.88:0/79411243 >> v1:172.21.15.154:6789/0 conn(0x7f74dc0b6d40 legacy=0x7f74dc0b71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.251+0000 7f74e1a6c640 1 -- 172.21.15.88:0/79411243 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74dc15ef50 con 0x7f74dc15e6f0 2024-09-16T01:36:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.251+0000 7f74e126b640 1 --2- 172.21.15.88:0/79411243 >> v2:172.21.15.154:3300/0 conn(0x7f74dc0b54d0 0x7f74dc15f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:36:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.251+0000 7f74e1a6c640 1 --2- 172.21.15.88:0/79411243 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74dc15e6f0 0x7f74dc06fd00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f74d80097a0 tx=0x7f74d802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d08a83d8a9854c7b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.252+0000 7f74e0a6a640 1 -- 172.21.15.88:0/79411243 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d8037070 con 0x7f74dc15e6f0 2024-09-16T01:36:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.252+0000 7f74e0a6a640 1 -- 172.21.15.88:0/79411243 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74d8032440 con 0x7f74dc15e6f0 2024-09-16T01:36:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.252+0000 7f74e0a6a640 1 -- 172.21.15.88:0/79411243 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d8032810 con 0x7f74dc15e6f0 2024-09-16T01:36:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.252+0000 7f74e3cd0640 1 -- 172.21.15.88:0/79411243 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74dc15e6f0 msgr2=0x7f74dc06fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.252+0000 7f74e3cd0640 1 --2- 172.21.15.88:0/79411243 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74dc15e6f0 0x7f74dc06fd00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f74d80097a0 tx=0x7f74d802a180 comp rx=0 tx=0).stop 2024-09-16T01:36:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.252+0000 7f74e3cd0640 1 -- 172.21.15.88:0/79411243 shutdown_connections 2024-09-16T01:36:54.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.252+0000 7f74e3cd0640 1 --2- 172.21.15.88:0/79411243 >> v2:172.21.15.154:3300/0 conn(0x7f74dc0b54d0 0x7f74dc15f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:54.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.252+0000 7f74e3cd0640 1 --2- 172.21.15.88:0/79411243 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74dc15e6f0 0x7f74dc06fd00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:54.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.252+0000 7f74e3cd0640 1 -- 172.21.15.88:0/79411243 >> 172.21.15.88:0/79411243 conn(0x7f74dc0d2430 msgr2=0x7f74dc0d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:54.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.252+0000 7f74e3cd0640 1 -- 172.21.15.88:0/79411243 shutdown_connections 2024-09-16T01:36:54.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.252+0000 7f74e3cd0640 1 -- 172.21.15.88:0/79411243 wait complete. 2024-09-16T01:36:54.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.254+0000 7f74e3cd0640 1 Processor -- start 2024-09-16T01:36:54.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.254+0000 7f74e3cd0640 1 -- start start 2024-09-16T01:36:54.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.254+0000 7f74e3cd0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74dc0b54d0 0x7f74dc153930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:54.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.254+0000 7f74e3cd0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74dc0b6d40 0x7f74dc1550d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:54.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.254+0000 7f74e3cd0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74dc15e6f0 0x7f74dc161700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:54.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.254+0000 7f74e3cd0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc06e730 con 0x7f74dc0b6d40 2024-09-16T01:36:54.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.254+0000 7f74e3cd0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc06e5b0 con 0x7f74dc0b54d0 2024-09-16T01:36:54.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.254+0000 7f74e3cd0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f74dc070830 con 0x7f74dc15e6f0 2024-09-16T01:36:54.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.254+0000 7f74e126b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74dc0b6d40 0x7f74dc1550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:54.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.254+0000 7f74e126b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74dc0b6d40 0x7f74dc1550d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44032/0 (socket says 172.21.15.88:44032) 2024-09-16T01:36:54.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.254+0000 7f74e126b640 1 -- 172.21.15.88:0/2260173234 learned_addr learned my addr 172.21.15.88:0/2260173234 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:54.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.255+0000 7f74e226d640 1 --2- 172.21.15.88:0/2260173234 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74dc15e6f0 0x7f74dc161700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:54.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.255+0000 7f74e126b640 1 -- 172.21.15.88:0/2260173234 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74dc15e6f0 msgr2=0x7f74dc161700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:54.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.255+0000 7f74e126b640 1 --2- 172.21.15.88:0/2260173234 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74dc15e6f0 0x7f74dc161700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:54.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.255+0000 7f74e1a6c640 1 --2- 172.21.15.88:0/2260173234 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74dc0b54d0 0x7f74dc153930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:54.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.255+0000 7f74e126b640 1 -- 172.21.15.88:0/2260173234 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74dc0b54d0 msgr2=0x7f74dc153930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:54.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.255+0000 7f74e126b640 1 --2- 172.21.15.88:0/2260173234 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74dc0b54d0 0x7f74dc153930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:54.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.255+0000 7f74e126b640 1 -- 172.21.15.88:0/2260173234 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74dc155610 con 0x7f74dc0b6d40 2024-09-16T01:36:54.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.255+0000 7f74e1a6c640 1 --2- 172.21.15.88:0/2260173234 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74dc0b54d0 0x7f74dc153930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:54.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.255+0000 7f74e126b640 1 --2- 172.21.15.88:0/2260173234 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74dc0b6d40 0x7f74dc1550d0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f74cc00ebe0 tx=0x7f74cc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:54.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.255+0000 7f74d27fc640 1 -- 172.21.15.88:0/2260173234 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74cc017070 con 0x7f74dc0b6d40 2024-09-16T01:36:54.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.255+0000 7f74d27fc640 1 -- 172.21.15.88:0/2260173234 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74cc00cbd0 con 0x7f74dc0b6d40 2024-09-16T01:36:54.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.255+0000 7f74e3cd0640 1 -- 172.21.15.88:0/2260173234 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74dc154030 con 0x7f74dc0b6d40 2024-09-16T01:36:54.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.255+0000 7f74d27fc640 1 -- 172.21.15.88:0/2260173234 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74cc013470 con 0x7f74dc0b6d40 2024-09-16T01:36:54.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.255+0000 7f74e3cd0640 1 -- 172.21.15.88:0/2260173234 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74dc1545f0 con 0x7f74dc0b6d40 2024-09-16T01:36:54.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.256+0000 7f74e3cd0640 1 -- 172.21.15.88:0/2260173234 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74dc0d1b00 con 0x7f74dc0b6d40 2024-09-16T01:36:54.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.257+0000 7f74d27fc640 1 -- 172.21.15.88:0/2260173234 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f74cc01a070 con 0x7f74dc0b6d40 2024-09-16T01:36:54.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.257+0000 7f74d27fc640 1 --2- 172.21.15.88:0/2260173234 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f74c006c2b0 0x7f74c006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:54.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.257+0000 7f74d27fc640 1 -- 172.21.15.88:0/2260173234 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5795+0+0 (secure 0 0 0) 0x7f74cc08b3d0 con 0x7f74dc0b6d40 2024-09-16T01:36:54.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.257+0000 7f74e1a6c640 1 --2- 172.21.15.88:0/2260173234 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f74c006c2b0 0x7f74c006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:54.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.258+0000 7f74e1a6c640 1 --2- 172.21.15.88:0/2260173234 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f74c006c2b0 0x7f74c006e770 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f74d8034000 tx=0x7f74d802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:54.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.262+0000 7f74d27fc640 1 -- 172.21.15.88:0/2260173234 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f74cc059ae0 con 0x7f74dc0b6d40 2024-09-16T01:36:54.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.464+0000 7f74e3cd0640 1 -- 172.21.15.88:0/2260173234 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f74dc159b80 con 0x7f74dc0b6d40 2024-09-16T01:36:54.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.839+0000 7f74d27fc640 1 -- 172.21.15.88:0/2260173234 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v40) ==== 162+0+0 (secure 0 0 0) 0x7f74cc0593e0 con 0x7f74dc0b6d40 2024-09-16T01:36:54.840 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:36:54.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.841+0000 7f74e3cd0640 1 -- 172.21.15.88:0/2260173234 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f74c006c2b0 msgr2=0x7f74c006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:54.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.841+0000 7f74e3cd0640 1 --2- 172.21.15.88:0/2260173234 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f74c006c2b0 0x7f74c006e770 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f74d8034000 tx=0x7f74d802e000 comp rx=0 tx=0).stop 2024-09-16T01:36:54.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.841+0000 7f74e3cd0640 1 -- 172.21.15.88:0/2260173234 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74dc0b6d40 msgr2=0x7f74dc1550d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:54.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.841+0000 7f74e3cd0640 1 --2- 172.21.15.88:0/2260173234 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74dc0b6d40 0x7f74dc1550d0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f74cc00ebe0 tx=0x7f74cc00c520 comp rx=0 tx=0).stop 2024-09-16T01:36:54.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.841+0000 7f74e3cd0640 1 -- 172.21.15.88:0/2260173234 shutdown_connections 2024-09-16T01:36:54.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.841+0000 7f74e3cd0640 1 --2- 172.21.15.88:0/2260173234 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f74c006c2b0 0x7f74c006e770 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:54.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.841+0000 7f74e3cd0640 1 --2- 172.21.15.88:0/2260173234 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74dc15e6f0 0x7f74dc161700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:54.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.841+0000 7f74e3cd0640 1 --2- 172.21.15.88:0/2260173234 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74dc0b6d40 0x7f74dc1550d0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:54.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.841+0000 7f74e3cd0640 1 --2- 172.21.15.88:0/2260173234 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74dc0b54d0 0x7f74dc153930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:54.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.841+0000 7f74e3cd0640 1 -- 172.21.15.88:0/2260173234 >> 172.21.15.88:0/2260173234 conn(0x7f74dc0d2430 msgr2=0x7f74dc0b20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:54.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.841+0000 7f74e3cd0640 1 -- 172.21.15.88:0/2260173234 shutdown_connections 2024-09-16T01:36:54.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.841+0000 7f74e3cd0640 1 -- 172.21.15.88:0/2260173234 wait complete. 2024-09-16T01:36:54.856 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:36:54.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.951+0000 7f757460e640 1 Processor -- start 2024-09-16T01:36:54.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.951+0000 7f757460e640 1 -- start start 2024-09-16T01:36:54.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.952+0000 7f757460e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f756c15e950 0x7f756c06ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:54.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.952+0000 7f757460e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f756c072050 0x7f756c06d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:54.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.952+0000 7f757460e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f756c06faf0 con 0x7f756c0735b0 2024-09-16T01:36:54.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.952+0000 7f757460e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f756c06fdd0 con 0x7f756c072050 2024-09-16T01:36:54.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.952+0000 7f757460e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f756c0700b0 con 0x7f756c15e950 2024-09-16T01:36:54.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.952+0000 7f7572bab640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f756c072050 0x7f756c06d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:54.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.953+0000 7f7572bab640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f756c072050 0x7f756c06d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44048/0 (socket says 172.21.15.88:44048) 2024-09-16T01:36:54.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.953+0000 7f7572bab640 1 -- 172.21.15.88:0/2173411064 learned_addr learned my addr 172.21.15.88:0/2173411064 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:54.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.953+0000 7f75723aa640 1 --2- 172.21.15.88:0/2173411064 >> v2:172.21.15.154:3300/0 conn(0x7f756c15e950 0x7f756c06ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:54.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.953+0000 7f7572bab640 1 -- 172.21.15.88:0/2173411064 >> v2:172.21.15.154:3300/0 conn(0x7f756c15e950 msgr2=0x7f756c06ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:54.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.953+0000 7f7572bab640 1 --2- 172.21.15.88:0/2173411064 >> v2:172.21.15.154:3300/0 conn(0x7f756c15e950 0x7f756c06ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:54.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.953+0000 7f7572bab640 1 -- 172.21.15.88:0/2173411064 >> v1:172.21.15.154:6789/0 conn(0x7f756c0735b0 legacy=0x7f756c06bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:54.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.953+0000 7f7572bab640 1 -- 172.21.15.88:0/2173411064 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f756c070390 con 0x7f756c072050 2024-09-16T01:36:54.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.953+0000 7f7572bab640 1 --2- 172.21.15.88:0/2173411064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f756c072050 0x7f756c06d370 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f756c069190 tx=0x7f756402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bb222ed095eb3c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:54.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.954+0000 7f75713a8640 1 -- 172.21.15.88:0/2173411064 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7564037070 con 0x7f756c072050 2024-09-16T01:36:54.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.954+0000 7f75713a8640 1 -- 172.21.15.88:0/2173411064 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7564032440 con 0x7f756c072050 2024-09-16T01:36:54.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.954+0000 7f75713a8640 1 -- 172.21.15.88:0/2173411064 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7564032810 con 0x7f756c072050 2024-09-16T01:36:54.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.954+0000 7f757460e640 1 -- 172.21.15.88:0/2173411064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f756c072050 msgr2=0x7f756c06d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:54.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.954+0000 7f757460e640 1 --2- 172.21.15.88:0/2173411064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f756c072050 0x7f756c06d370 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f756c069190 tx=0x7f756402a230 comp rx=0 tx=0).stop 2024-09-16T01:36:54.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.954+0000 7f757460e640 1 -- 172.21.15.88:0/2173411064 shutdown_connections 2024-09-16T01:36:54.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.954+0000 7f757460e640 1 --2- 172.21.15.88:0/2173411064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f756c072050 0x7f756c06d370 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:54.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.954+0000 7f757460e640 1 --2- 172.21.15.88:0/2173411064 >> v2:172.21.15.154:3300/0 conn(0x7f756c15e950 0x7f756c06ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:54.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.954+0000 7f757460e640 1 -- 172.21.15.88:0/2173411064 >> 172.21.15.88:0/2173411064 conn(0x7f756c0d14d0 msgr2=0x7f756c0d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:54.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.954+0000 7f757460e640 1 -- 172.21.15.88:0/2173411064 shutdown_connections 2024-09-16T01:36:54.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.954+0000 7f757460e640 1 -- 172.21.15.88:0/2173411064 wait complete. 2024-09-16T01:36:54.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.955+0000 7f757460e640 1 Processor -- start 2024-09-16T01:36:54.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.955+0000 7f757460e640 1 -- start start 2024-09-16T01:36:54.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.956+0000 7f757460e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f756c072050 0x7f756c158ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:54.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.956+0000 7f757460e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f756c0735b0 0x7f756c157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:54.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.956+0000 7f757460e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f756c15e950 0x7f756c153c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:54.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.956+0000 7f757460e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f756c0d0230 con 0x7f756c072050 2024-09-16T01:36:54.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.956+0000 7f757460e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f756c0d00b0 con 0x7f756c15e950 2024-09-16T01:36:54.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.956+0000 7f757460e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f756c0d03b0 con 0x7f756c0735b0 2024-09-16T01:36:54.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.956+0000 7f75723aa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f756c072050 0x7f756c158ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:54.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.956+0000 7f75723aa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f756c072050 0x7f756c158ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44056/0 (socket says 172.21.15.88:44056) 2024-09-16T01:36:54.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.956+0000 7f75723aa640 1 -- 172.21.15.88:0/19658103 learned_addr learned my addr 172.21.15.88:0/19658103 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:54.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.956+0000 7f7572bab640 1 --2- 172.21.15.88:0/19658103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f756c15e950 0x7f756c153c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:54.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.957+0000 7f75723aa640 1 -- 172.21.15.88:0/19658103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f756c0735b0 msgr2=0x7f756c157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:54.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.957+0000 7f75723aa640 1 --2- 172.21.15.88:0/19658103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f756c0735b0 0x7f756c157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:54.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.957+0000 7f75723aa640 1 -- 172.21.15.88:0/19658103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f756c15e950 msgr2=0x7f756c153c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:54.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.957+0000 7f75723aa640 1 --2- 172.21.15.88:0/19658103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f756c15e950 0x7f756c153c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:54.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.957+0000 7f75723aa640 1 -- 172.21.15.88:0/19658103 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f756c0cb620 con 0x7f756c072050 2024-09-16T01:36:54.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.957+0000 7f7572bab640 1 --2- 172.21.15.88:0/19658103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f756c15e950 0x7f756c153c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:54.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.957+0000 7f75723aa640 1 --2- 172.21.15.88:0/19658103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f756c072050 0x7f756c158ba0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7568007ef0 tx=0x7f756800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:54.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.957+0000 7f7562ffd640 1 -- 172.21.15.88:0/19658103 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7568014620 con 0x7f756c072050 2024-09-16T01:36:54.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.957+0000 7f7562ffd640 1 -- 172.21.15.88:0/19658103 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7568019070 con 0x7f756c072050 2024-09-16T01:36:54.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.957+0000 7f757460e640 1 -- 172.21.15.88:0/19658103 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f756c157f60 con 0x7f756c072050 2024-09-16T01:36:54.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.957+0000 7f7562ffd640 1 -- 172.21.15.88:0/19658103 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7568014a00 con 0x7f756c072050 2024-09-16T01:36:54.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.957+0000 7f757460e640 1 -- 172.21.15.88:0/19658103 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f756c159460 con 0x7f756c072050 2024-09-16T01:36:54.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.958+0000 7f757460e640 1 -- 172.21.15.88:0/19658103 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f756c06ce30 con 0x7f756c072050 2024-09-16T01:36:54.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.958+0000 7f7562ffd640 1 -- 172.21.15.88:0/19658103 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7568014ba0 con 0x7f756c072050 2024-09-16T01:36:54.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.959+0000 7f7562ffd640 1 --2- 172.21.15.88:0/19658103 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f755006c2b0 0x7f755006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:54.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.959+0000 7f7562ffd640 1 -- 172.21.15.88:0/19658103 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5795+0+0 (secure 0 0 0) 0x7f756808bd30 con 0x7f756c072050 2024-09-16T01:36:54.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.959+0000 7f7571ba9640 1 --2- 172.21.15.88:0/19658103 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f755006c2b0 0x7f755006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:54.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.959+0000 7f7571ba9640 1 --2- 172.21.15.88:0/19658103 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f755006c2b0 0x7f755006e770 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f756c15ba10 tx=0x7f755c0079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:54.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:54.963+0000 7f7562ffd640 1 -- 172.21.15.88:0/19658103 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f756c0cb620 con 0x7f756c072050 2024-09-16T01:36:55.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:55.163+0000 7f757460e640 1 -- 172.21.15.88:0/19658103 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f756c0b4c70 con 0x7f756c072050 2024-09-16T01:36:55.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:55.163+0000 7f7562ffd640 1 -- 172.21.15.88:0/19658103 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+5217 (secure 0 0 0) 0x7f756805a4d0 con 0x7f756c072050 2024-09-16T01:36:55.165 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:55.166 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":41,"btime":"2024-09-16T01:36:54:841659+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/581241194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":581241194},{"type":"v1","addr":"172.21.15.154:6834","nonce":581241194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2104508521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2104508521},{"type":"v1","addr":"172.21.15.154:6835","nonce":2104508521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5859,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/998283126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":998283126},{"type":"v1","addr":"172.21.15.88:6836","nonce":998283126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:39.786438+0000","modified":"2024-09-16T01:36:54.841644+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"a","rank":0,"incarnation":41,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6837/4130074102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4130074102},{"type":"v1","addr":"172.21.15.88:6837","nonce":4130074102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":4}]} 2024-09-16T01:36:55.166 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 41 2024-09-16T01:36:55.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:55.165+0000 7f757460e640 1 -- 172.21.15.88:0/19658103 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f755006c2b0 msgr2=0x7f755006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:55.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:55.165+0000 7f757460e640 1 --2- 172.21.15.88:0/19658103 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f755006c2b0 0x7f755006e770 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f756c15ba10 tx=0x7f755c0079b0 comp rx=0 tx=0).stop 2024-09-16T01:36:55.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:55.166+0000 7f757460e640 1 -- 172.21.15.88:0/19658103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f756c072050 msgr2=0x7f756c158ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:55.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:55.166+0000 7f757460e640 1 --2- 172.21.15.88:0/19658103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f756c072050 0x7f756c158ba0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7568007ef0 tx=0x7f756800e4b0 comp rx=0 tx=0).stop 2024-09-16T01:36:55.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:55.166+0000 7f757460e640 1 -- 172.21.15.88:0/19658103 shutdown_connections 2024-09-16T01:36:55.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:55.166+0000 7f757460e640 1 --2- 172.21.15.88:0/19658103 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f755006c2b0 0x7f755006e770 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:55.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:55.166+0000 7f757460e640 1 --2- 172.21.15.88:0/19658103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f756c15e950 0x7f756c153c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:55.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:55.166+0000 7f757460e640 1 --2- 172.21.15.88:0/19658103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f756c0735b0 0x7f756c157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:55.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:55.166+0000 7f757460e640 1 --2- 172.21.15.88:0/19658103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f756c072050 0x7f756c158ba0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:55.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:55.166+0000 7f757460e640 1 -- 172.21.15.88:0/19658103 >> 172.21.15.88:0/19658103 conn(0x7f756c0d14d0 msgr2=0x7f756c0b3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:55.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:55.166+0000 7f757460e640 1 -- 172.21.15.88:0/19658103 shutdown_connections 2024-09-16T01:36:55.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:55.166+0000 7f757460e640 1 -- 172.21.15.88:0/19658103 wait complete. 2024-09-16T01:36:55.581 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 41, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:36:39.786438+0000', 'modified': '2024-09-16T01:36:54.841644+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 59, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6054}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6054': {'gid': 6054, 'name': 'a', 'rank': 0, 'incarnation': 41, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.88:6837/4130074102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6835', 'nonce': 4130074102}, {'type': 'v1', 'addr': '172.21.15.88:6837', 'nonce': 4130074102}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-16T01:36:55.581 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6054:up:replay 2024-09-16T01:36:56.581 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:36:56.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.689+0000 7fdf8c065640 1 Processor -- start 2024-09-16T01:36:56.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.690+0000 7fdf8c065640 1 -- start start 2024-09-16T01:36:56.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.690+0000 7fdf8c065640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdf840b5be0 0x7fdf8415a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:56.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.690+0000 7fdf8c065640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf840b29a0 0x7fdf8406de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:56.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.690+0000 7fdf8c065640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf840b6830 con 0x7fdf8415e600 2024-09-16T01:36:56.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.690+0000 7fdf8c065640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fdf8415f410 con 0x7fdf840b5be0 2024-09-16T01:36:56.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.690+0000 7fdf8c065640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdf8415f1c0 con 0x7fdf840b29a0 2024-09-16T01:36:56.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.691+0000 7fdf8a602640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fdf8415e600 0x7fdf8415eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58494/0 (socket says 172.21.15.88:58494) 2024-09-16T01:36:56.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.691+0000 7fdf8a602640 1 -- 172.21.15.88:0/3263395141 learned_addr learned my addr 172.21.15.88:0/3263395141 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:56.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.691+0000 7fdf89e01640 1 --2- 172.21.15.88:0/3263395141 >> v2:172.21.15.154:3300/0 conn(0x7fdf840b5be0 0x7fdf8415a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:56.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.691+0000 7fdf89600640 1 --2- 172.21.15.88:0/3263395141 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf840b29a0 0x7fdf8406de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:56.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.691+0000 7fdf88dff640 1 -- 172.21.15.88:0/3263395141 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3008737966 0 0) 0x7fdf840b6830 con 0x7fdf8415e600 2024-09-16T01:36:56.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.691+0000 7fdf88dff640 1 -- 172.21.15.88:0/3263395141 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf70003520 con 0x7fdf8415e600 2024-09-16T01:36:56.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.691+0000 7fdf88dff640 1 -- 172.21.15.88:0/3263395141 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4274136004 0 0) 0x7fdf70003520 con 0x7fdf8415e600 2024-09-16T01:36:56.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.691+0000 7fdf89e01640 1 -- 172.21.15.88:0/3263395141 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf840b29a0 msgr2=0x7fdf8406de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:56.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.691+0000 7fdf89e01640 1 --2- 172.21.15.88:0/3263395141 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf840b29a0 0x7fdf8406de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:56.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.691+0000 7fdf89e01640 1 -- 172.21.15.88:0/3263395141 >> v1:172.21.15.88:6789/0 conn(0x7fdf8415e600 legacy=0x7fdf8415eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:56.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.691+0000 7fdf89e01640 1 -- 172.21.15.88:0/3263395141 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf840cd0b0 con 0x7fdf840b5be0 2024-09-16T01:36:56.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.691+0000 7fdf89600640 1 --2- 172.21.15.88:0/3263395141 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf840b29a0 0x7fdf8406de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:36:56.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.692+0000 7fdf89e01640 1 --2- 172.21.15.88:0/3263395141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf840b5be0 0x7fdf8415a9c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdf80009d90 tx=0x7fdf8002a560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97e8e4a31653e77b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:56.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.692+0000 7fdf88dff640 1 -- 172.21.15.88:0/3263395141 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c001670 con 0x7fdf840b5be0 2024-09-16T01:36:56.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.692+0000 7fdf88dff640 1 -- 172.21.15.88:0/3263395141 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf80007540 con 0x7fdf840b5be0 2024-09-16T01:36:56.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.692+0000 7fdf88dff640 1 -- 172.21.15.88:0/3263395141 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf800078f0 con 0x7fdf840b5be0 2024-09-16T01:36:56.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.693+0000 7fdf8c065640 1 -- 172.21.15.88:0/3263395141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf840b5be0 msgr2=0x7fdf8415a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:56.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.693+0000 7fdf8c065640 1 --2- 172.21.15.88:0/3263395141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf840b5be0 0x7fdf8415a9c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdf80009d90 tx=0x7fdf8002a560 comp rx=0 tx=0).stop 2024-09-16T01:36:56.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.693+0000 7fdf8c065640 1 -- 172.21.15.88:0/3263395141 shutdown_connections 2024-09-16T01:36:56.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.693+0000 7fdf8c065640 1 --2- 172.21.15.88:0/3263395141 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf840b29a0 0x7fdf8406de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:56.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.693+0000 7fdf8c065640 1 --2- 172.21.15.88:0/3263395141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf840b5be0 0x7fdf8415a9c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:56.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.693+0000 7fdf8c065640 1 -- 172.21.15.88:0/3263395141 >> 172.21.15.88:0/3263395141 conn(0x7fdf8406bc10 msgr2=0x7fdf8406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:56.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.693+0000 7fdf8c065640 1 -- 172.21.15.88:0/3263395141 shutdown_connections 2024-09-16T01:36:56.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.693+0000 7fdf8c065640 1 -- 172.21.15.88:0/3263395141 wait complete. 2024-09-16T01:36:56.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.694+0000 7fdf8c065640 1 Processor -- start 2024-09-16T01:36:56.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.694+0000 7fdf8c065640 1 -- start start 2024-09-16T01:36:56.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.695+0000 7fdf8c065640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf840b29a0 0x7fdf84144e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:56.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.695+0000 7fdf8c065640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf840b5be0 0x7fdf8415ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:56.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.695+0000 7fdf8c065640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf8415e600 0x7fdf8414b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:56.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.695+0000 7fdf8c065640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdf840cd950 con 0x7fdf840b29a0 2024-09-16T01:36:56.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.695+0000 7fdf8c065640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdf840cd670 con 0x7fdf840b5be0 2024-09-16T01:36:56.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.695+0000 7fdf8c065640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdf8406e380 con 0x7fdf8415e600 2024-09-16T01:36:56.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.695+0000 7fdf89e01640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf840b29a0 0x7fdf84144e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:56.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.695+0000 7fdf89e01640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf840b29a0 0x7fdf84144e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44060/0 (socket says 172.21.15.88:44060) 2024-09-16T01:36:56.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.695+0000 7fdf89e01640 1 -- 172.21.15.88:0/1854046955 learned_addr learned my addr 172.21.15.88:0/1854046955 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:56.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.695+0000 7fdf8a602640 1 --2- 172.21.15.88:0/1854046955 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf8415e600 0x7fdf8414b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:56.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.695+0000 7fdf89600640 1 --2- 172.21.15.88:0/1854046955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf840b5be0 0x7fdf8415ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:56.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.696+0000 7fdf89e01640 1 -- 172.21.15.88:0/1854046955 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf8415e600 msgr2=0x7fdf8414b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:56.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.696+0000 7fdf89e01640 1 --2- 172.21.15.88:0/1854046955 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf8415e600 0x7fdf8414b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:56.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.696+0000 7fdf89e01640 1 -- 172.21.15.88:0/1854046955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf840b5be0 msgr2=0x7fdf8415ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:56.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.696+0000 7fdf89e01640 1 --2- 172.21.15.88:0/1854046955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf840b5be0 0x7fdf8415ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:56.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.696+0000 7fdf89e01640 1 -- 172.21.15.88:0/1854046955 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf8414dbd0 con 0x7fdf840b29a0 2024-09-16T01:36:56.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.696+0000 7fdf89e01640 1 --2- 172.21.15.88:0/1854046955 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf840b29a0 0x7fdf84144e20 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fdf80009eb0 tx=0x7fdf80004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:56.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.696+0000 7fdf7a7fc640 1 -- 172.21.15.88:0/1854046955 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf80040070 con 0x7fdf840b29a0 2024-09-16T01:36:56.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.696+0000 7fdf8c065640 1 -- 172.21.15.88:0/1854046955 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf8415d690 con 0x7fdf840b29a0 2024-09-16T01:36:56.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.696+0000 7fdf7a7fc640 1 -- 172.21.15.88:0/1854046955 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf80031c40 con 0x7fdf840b29a0 2024-09-16T01:36:56.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.696+0000 7fdf7a7fc640 1 -- 172.21.15.88:0/1854046955 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf80009450 con 0x7fdf840b29a0 2024-09-16T01:36:56.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.696+0000 7fdf8c065640 1 -- 172.21.15.88:0/1854046955 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf8414e150 con 0x7fdf840b29a0 2024-09-16T01:36:56.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.697+0000 7fdf8c065640 1 -- 172.21.15.88:0/1854046955 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf84074df0 con 0x7fdf840b29a0 2024-09-16T01:36:56.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.698+0000 7fdf7a7fc640 1 -- 172.21.15.88:0/1854046955 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fdf80039040 con 0x7fdf840b29a0 2024-09-16T01:36:56.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.698+0000 7fdf7a7fc640 1 --2- 172.21.15.88:0/1854046955 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdf6806c2b0 0x7fdf6806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:56.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.698+0000 7fdf7a7fc640 1 -- 172.21.15.88:0/1854046955 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5795+0+0 (secure 0 0 0) 0x7fdf800ac940 con 0x7fdf840b29a0 2024-09-16T01:36:56.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.698+0000 7fdf89600640 1 --2- 172.21.15.88:0/1854046955 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdf6806c2b0 0x7fdf6806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:56.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.698+0000 7fdf89600640 1 --2- 172.21.15.88:0/1854046955 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdf6806c2b0 0x7fdf6806e770 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fdf74009560 tx=0x7fdf74008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:56.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.702+0000 7fdf7a7fc640 1 -- 172.21.15.88:0/1854046955 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fdf80007e50 con 0x7fdf840b29a0 2024-09-16T01:36:56.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.901+0000 7fdf8c065640 1 -- 172.21.15.88:0/1854046955 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf8406de40 con 0x7fdf840b29a0 2024-09-16T01:36:56.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.902+0000 7fdf7a7fc640 1 -- 172.21.15.88:0/1854046955 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+5217 (secure 0 0 0) 0x7fdf8007afd0 con 0x7fdf840b29a0 2024-09-16T01:36:56.903 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:56.904 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":43,"btime":"2024-09-16T01:36:56:847572+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/581241194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":581241194},{"type":"v1","addr":"172.21.15.154:6834","nonce":581241194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2104508521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2104508521},{"type":"v1","addr":"172.21.15.154:6835","nonce":2104508521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5859,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/998283126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":998283126},{"type":"v1","addr":"172.21.15.88:6836","nonce":998283126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:39.786438+0000","modified":"2024-09-16T01:36:55.848660+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"a","rank":0,"incarnation":41,"state":"up:rejoin","state_seq":4,"addr":"172.21.15.88:6837/4130074102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4130074102},{"type":"v1","addr":"172.21.15.88:6837","nonce":4130074102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":4}]} 2024-09-16T01:36:56.904 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 43 2024-09-16T01:36:56.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.904+0000 7fdf8c065640 1 -- 172.21.15.88:0/1854046955 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdf6806c2b0 msgr2=0x7fdf6806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:56.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.904+0000 7fdf8c065640 1 --2- 172.21.15.88:0/1854046955 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdf6806c2b0 0x7fdf6806e770 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fdf74009560 tx=0x7fdf74008040 comp rx=0 tx=0).stop 2024-09-16T01:36:56.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.904+0000 7fdf8c065640 1 -- 172.21.15.88:0/1854046955 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf840b29a0 msgr2=0x7fdf84144e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:56.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.904+0000 7fdf8c065640 1 --2- 172.21.15.88:0/1854046955 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf840b29a0 0x7fdf84144e20 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fdf80009eb0 tx=0x7fdf80004a40 comp rx=0 tx=0).stop 2024-09-16T01:36:56.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.904+0000 7fdf8c065640 1 -- 172.21.15.88:0/1854046955 shutdown_connections 2024-09-16T01:36:56.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.904+0000 7fdf8c065640 1 --2- 172.21.15.88:0/1854046955 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdf6806c2b0 0x7fdf6806e770 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:56.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.904+0000 7fdf8c065640 1 --2- 172.21.15.88:0/1854046955 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf8415e600 0x7fdf8414b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:56.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.904+0000 7fdf8c065640 1 --2- 172.21.15.88:0/1854046955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf840b5be0 0x7fdf8415ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:56.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.904+0000 7fdf8c065640 1 --2- 172.21.15.88:0/1854046955 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf840b29a0 0x7fdf84144e20 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:56.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.904+0000 7fdf8c065640 1 -- 172.21.15.88:0/1854046955 >> 172.21.15.88:0/1854046955 conn(0x7fdf8406bc10 msgr2=0x7fdf840b3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:56.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.904+0000 7fdf8c065640 1 -- 172.21.15.88:0/1854046955 shutdown_connections 2024-09-16T01:36:56.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:56.904+0000 7fdf8c065640 1 -- 172.21.15.88:0/1854046955 wait complete. 2024-09-16T01:36:57.319 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 43, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:36:39.786438+0000', 'modified': '2024-09-16T01:36:55.848660+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 59, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6054}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6054': {'gid': 6054, 'name': 'a', 'rank': 0, 'incarnation': 41, 'state': 'up:rejoin', 'state_seq': 4, 'addr': '172.21.15.88:6837/4130074102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6835', 'nonce': 4130074102}, {'type': 'v1', 'addr': '172.21.15.88:6837', 'nonce': 4130074102}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-16T01:36:57.319 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6054:up:rejoin 2024-09-16T01:36:58.320 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:36:58.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.430+0000 7f2a0fe5e640 1 Processor -- start 2024-09-16T01:36:58.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.430+0000 7f2a0fe5e640 1 -- start start 2024-09-16T01:36:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.431+0000 7f2a0fe5e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2a080b5be0 0x7f2a0815ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.431+0000 7f2a0fe5e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2a080b29a0 0x7f2a080cde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.431+0000 7f2a0fe5e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a080b6830 con 0x7f2a0815a9c0 2024-09-16T01:36:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.431+0000 7f2a0fe5e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2a0815f410 con 0x7f2a080b29a0 2024-09-16T01:36:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.431+0000 7f2a0fe5e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2a080cd0b0 con 0x7f2a080b5be0 2024-09-16T01:36:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.431+0000 7f2a0dbfa640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f2a0815a9c0 0x7f2a0815e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58504/0 (socket says 172.21.15.88:58504) 2024-09-16T01:36:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.431+0000 7f2a0dbfa640 1 -- 172.21.15.88:0/3850514415 learned_addr learned my addr 172.21.15.88:0/3850514415 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.431+0000 7f2a0e3fb640 1 --2- 172.21.15.88:0/3850514415 >> v2:172.21.15.88:3300/0 conn(0x7f2a080b29a0 0x7f2a080cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.431+0000 7f2a0e3fb640 1 -- 172.21.15.88:0/3850514415 >> v2:172.21.15.154:3300/0 conn(0x7f2a080b5be0 msgr2=0x7f2a0815ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.431+0000 7f2a0e3fb640 1 --2- 172.21.15.88:0/3850514415 >> v2:172.21.15.154:3300/0 conn(0x7f2a080b5be0 0x7f2a0815ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.431+0000 7f2a0e3fb640 1 -- 172.21.15.88:0/3850514415 >> v1:172.21.15.88:6789/0 conn(0x7f2a0815a9c0 legacy=0x7f2a0815e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.431+0000 7f2a0cbf8640 1 -- 172.21.15.88:0/3850514415 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2217969057 0 0) 0x7f2a080b6830 con 0x7f2a0815a9c0 2024-09-16T01:36:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.431+0000 7f2a0e3fb640 1 -- 172.21.15.88:0/3850514415 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a080cd390 con 0x7f2a080b29a0 2024-09-16T01:36:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.432+0000 7f2a0e3fb640 1 --2- 172.21.15.88:0/3850514415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a080b29a0 0x7f2a080cde70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f2a080ccd90 tx=0x7f2a0002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28fbb01f23e1734c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.432+0000 7f2a0cbf8640 1 -- 172.21.15.88:0/3850514415 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a00007740 con 0x7f2a080b29a0 2024-09-16T01:36:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.432+0000 7f2a0cbf8640 1 -- 172.21.15.88:0/3850514415 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a000078e0 con 0x7f2a080b29a0 2024-09-16T01:36:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.432+0000 7f2a0cbf8640 1 -- 172.21.15.88:0/3850514415 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a00007cb0 con 0x7f2a080b29a0 2024-09-16T01:36:58.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.433+0000 7f2a0fe5e640 1 -- 172.21.15.88:0/3850514415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a080b29a0 msgr2=0x7f2a080cde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:58.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.433+0000 7f2a0fe5e640 1 --2- 172.21.15.88:0/3850514415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a080b29a0 0x7f2a080cde70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f2a080ccd90 tx=0x7f2a0002a680 comp rx=0 tx=0).stop 2024-09-16T01:36:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.433+0000 7f2a0fe5e640 1 -- 172.21.15.88:0/3850514415 shutdown_connections 2024-09-16T01:36:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.433+0000 7f2a0fe5e640 1 --2- 172.21.15.88:0/3850514415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a080b29a0 0x7f2a080cde70 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.433+0000 7f2a0fe5e640 1 --2- 172.21.15.88:0/3850514415 >> v2:172.21.15.154:3300/0 conn(0x7f2a080b5be0 0x7f2a0815ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.433+0000 7f2a0fe5e640 1 -- 172.21.15.88:0/3850514415 >> 172.21.15.88:0/3850514415 conn(0x7f2a0806bc10 msgr2=0x7f2a0806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.433+0000 7f2a0fe5e640 1 -- 172.21.15.88:0/3850514415 shutdown_connections 2024-09-16T01:36:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.433+0000 7f2a0fe5e640 1 -- 172.21.15.88:0/3850514415 wait complete. 2024-09-16T01:36:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.434+0000 7f2a0fe5e640 1 Processor -- start 2024-09-16T01:36:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.434+0000 7f2a0fe5e640 1 -- start start 2024-09-16T01:36:58.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.435+0000 7f2a0fe5e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a080b29a0 0x7f2a0814d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:58.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.435+0000 7f2a0fe5e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a080b5be0 0x7f2a0814e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:58.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.435+0000 7f2a0fe5e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a0815a9c0 0x7f2a081595b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:58.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.435+0000 7f2a0fe5e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2a0815ada0 con 0x7f2a080b29a0 2024-09-16T01:36:58.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.435+0000 7f2a0fe5e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2a080cd950 con 0x7f2a080b5be0 2024-09-16T01:36:58.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.435+0000 7f2a0fe5e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2a080ce3b0 con 0x7f2a0815a9c0 2024-09-16T01:36:58.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.435+0000 7f2a0dbfa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a080b29a0 0x7f2a0814d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.435+0000 7f2a0dbfa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a080b29a0 0x7f2a0814d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44082/0 (socket says 172.21.15.88:44082) 2024-09-16T01:36:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.435+0000 7f2a0dbfa640 1 -- 172.21.15.88:0/204936549 learned_addr learned my addr 172.21.15.88:0/204936549 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.435+0000 7f2a0e3fb640 1 --2- 172.21.15.88:0/204936549 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a0815a9c0 0x7f2a081595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.435+0000 7f2a0d3f9640 1 --2- 172.21.15.88:0/204936549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a080b5be0 0x7f2a0814e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.435+0000 7f2a0dbfa640 1 -- 172.21.15.88:0/204936549 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a0815a9c0 msgr2=0x7f2a081595b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.435+0000 7f2a0dbfa640 1 --2- 172.21.15.88:0/204936549 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a0815a9c0 0x7f2a081595b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.435+0000 7f2a0dbfa640 1 -- 172.21.15.88:0/204936549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a080b5be0 msgr2=0x7f2a0814e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.435+0000 7f2a0dbfa640 1 --2- 172.21.15.88:0/204936549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a080b5be0 0x7f2a0814e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.436+0000 7f2a0dbfa640 1 -- 172.21.15.88:0/204936549 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a0814e610 con 0x7f2a080b29a0 2024-09-16T01:36:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.436+0000 7f2a0e3fb640 1 --2- 172.21.15.88:0/204936549 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a0815a9c0 0x7f2a081595b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.436+0000 7f2a0d3f9640 1 --2- 172.21.15.88:0/204936549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a080b5be0 0x7f2a0814e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.436+0000 7f2a0dbfa640 1 --2- 172.21.15.88:0/204936549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a080b29a0 0x7f2a0814d180 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f2a0400c430 tx=0x7f2a0400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.436+0000 7f29fe7fc640 1 -- 172.21.15.88:0/204936549 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a040074e0 con 0x7f2a080b29a0 2024-09-16T01:36:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.436+0000 7f29fe7fc640 1 -- 172.21.15.88:0/204936549 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a040047e0 con 0x7f2a080b29a0 2024-09-16T01:36:58.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.436+0000 7f2a0fe5e640 1 -- 172.21.15.88:0/204936549 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a08158350 con 0x7f2a080b29a0 2024-09-16T01:36:58.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.436+0000 7f29fe7fc640 1 -- 172.21.15.88:0/204936549 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a04004bd0 con 0x7f2a080b29a0 2024-09-16T01:36:58.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.436+0000 7f2a0fe5e640 1 -- 172.21.15.88:0/204936549 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a08158910 con 0x7f2a080b29a0 2024-09-16T01:36:58.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.437+0000 7f2a0fe5e640 1 -- 172.21.15.88:0/204936549 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a080cde70 con 0x7f2a080b29a0 2024-09-16T01:36:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.437+0000 7f29fe7fc640 1 -- 172.21.15.88:0/204936549 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2a04011070 con 0x7f2a080b29a0 2024-09-16T01:36:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.438+0000 7f29fe7fc640 1 --2- 172.21.15.88:0/204936549 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f29ec06c2b0 0x7f29ec06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.438+0000 7f29fe7fc640 1 -- 172.21.15.88:0/204936549 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5795+0+0 (secure 0 0 0) 0x7f2a0408b0f0 con 0x7f2a080b29a0 2024-09-16T01:36:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.438+0000 7f2a0d3f9640 1 --2- 172.21.15.88:0/204936549 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f29ec06c2b0 0x7f29ec06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.438+0000 7f2a0d3f9640 1 --2- 172.21.15.88:0/204936549 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f29ec06c2b0 0x7f29ec06e770 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f29f8006fd0 tx=0x7f29f8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:58.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.442+0000 7f29fe7fc640 1 -- 172.21.15.88:0/204936549 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2a0400e280 con 0x7f2a080b29a0 2024-09-16T01:36:58.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.643+0000 7f2a0fe5e640 1 -- 172.21.15.88:0/204936549 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a08154bd0 con 0x7f2a080b29a0 2024-09-16T01:36:58.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.644+0000 7f29fe7fc640 1 -- 172.21.15.88:0/204936549 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+5224 (secure 0 0 0) 0x7f2a04059800 con 0x7f2a080b29a0 2024-09-16T01:36:58.645 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:58.646 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":44,"btime":"2024-09-16T01:36:57:852065+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/581241194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":581241194},{"type":"v1","addr":"172.21.15.154:6834","nonce":581241194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2104508521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2104508521},{"type":"v1","addr":"172.21.15.154:6835","nonce":2104508521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5859,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/998283126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":998283126},{"type":"v1","addr":"172.21.15.88:6836","nonce":998283126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:39.786438+0000","modified":"2024-09-16T01:36:57.852059+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":5,"addr":"172.21.15.88:6837/4130074102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4130074102},{"type":"v1","addr":"172.21.15.88:6837","nonce":4130074102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":4}]} 2024-09-16T01:36:58.646 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 44 2024-09-16T01:36:58.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.646+0000 7f2a0fe5e640 1 -- 172.21.15.88:0/204936549 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f29ec06c2b0 msgr2=0x7f29ec06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:58.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.646+0000 7f2a0fe5e640 1 --2- 172.21.15.88:0/204936549 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f29ec06c2b0 0x7f29ec06e770 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f29f8006fd0 tx=0x7f29f8008040 comp rx=0 tx=0).stop 2024-09-16T01:36:58.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.646+0000 7f2a0fe5e640 1 -- 172.21.15.88:0/204936549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a080b29a0 msgr2=0x7f2a0814d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:58.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.646+0000 7f2a0fe5e640 1 --2- 172.21.15.88:0/204936549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a080b29a0 0x7f2a0814d180 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f2a0400c430 tx=0x7f2a0400c8f0 comp rx=0 tx=0).stop 2024-09-16T01:36:58.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.646+0000 7f2a0fe5e640 1 -- 172.21.15.88:0/204936549 shutdown_connections 2024-09-16T01:36:58.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.646+0000 7f2a0fe5e640 1 --2- 172.21.15.88:0/204936549 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f29ec06c2b0 0x7f29ec06e770 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:58.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.646+0000 7f2a0fe5e640 1 --2- 172.21.15.88:0/204936549 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a0815a9c0 0x7f2a081595b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:58.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.646+0000 7f2a0fe5e640 1 --2- 172.21.15.88:0/204936549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a080b5be0 0x7f2a0814e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:58.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.646+0000 7f2a0fe5e640 1 --2- 172.21.15.88:0/204936549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a080b29a0 0x7f2a0814d180 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:58.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.646+0000 7f2a0fe5e640 1 -- 172.21.15.88:0/204936549 >> 172.21.15.88:0/204936549 conn(0x7f2a0806bc10 msgr2=0x7f2a0814d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:58.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.646+0000 7f2a0fe5e640 1 -- 172.21.15.88:0/204936549 shutdown_connections 2024-09-16T01:36:58.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:58.646+0000 7f2a0fe5e640 1 -- 172.21.15.88:0/204936549 wait complete. 2024-09-16T01:36:59.061 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 44, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:36:39.786438+0000', 'modified': '2024-09-16T01:36:57.852059+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 59, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6054}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6054': {'gid': 6054, 'name': 'a', 'rank': 0, 'incarnation': 41, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.88:6837/4130074102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6835', 'nonce': 4130074102}, {'type': 'v1', 'addr': '172.21.15.88:6837', 'nonce': 4130074102}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6054, 'qdb_cluster': [6054]} 2024-09-16T01:36:59.061 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T01:36:59.062 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-16T01:36:59.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.160+0000 7f254c7a6640 1 Processor -- start 2024-09-16T01:36:59.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.160+0000 7f254c7a6640 1 -- start start 2024-09-16T01:36:59.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.161+0000 7f254c7a6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f254415e7a0 0x7f254406caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.161+0000 7f254c7a6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25440cf1f0 0x7f25440b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.161+0000 7f254c7a6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f25440b6720 0x7f25441623c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.161+0000 7f254c7a6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f25440d2ca0 con 0x7f254415e7a0 2024-09-16T01:36:59.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.161+0000 7f254c7a6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f25440cf860 con 0x7f25440b6720 2024-09-16T01:36:59.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.161+0000 7f254c7a6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f25440cfb40 con 0x7f25440cf1f0 2024-09-16T01:36:59.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.161+0000 7f254a542640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f254415e7a0 0x7f254406caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.161+0000 7f254a542640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f254415e7a0 0x7f254406caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44088/0 (socket says 172.21.15.88:44088) 2024-09-16T01:36:59.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.161+0000 7f254a542640 1 -- 172.21.15.88:0/1102819434 learned_addr learned my addr 172.21.15.88:0/1102819434 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:59.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.162+0000 7f254ad43640 1 --2- 172.21.15.88:0/1102819434 >> v2:172.21.15.154:3300/0 conn(0x7f25440b6720 0x7f25441623c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.162+0000 7f2549d41640 1 --2- 172.21.15.88:0/1102819434 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25440cf1f0 0x7f25440b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.162+0000 7f254a542640 1 -- 172.21.15.88:0/1102819434 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25440cf1f0 msgr2=0x7f25440b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:59.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.162+0000 7f254a542640 1 --2- 172.21.15.88:0/1102819434 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25440cf1f0 0x7f25440b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.162+0000 7f254a542640 1 -- 172.21.15.88:0/1102819434 >> v2:172.21.15.154:3300/0 conn(0x7f25440b6720 msgr2=0x7f25441623c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:59.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.162+0000 7f254a542640 1 --2- 172.21.15.88:0/1102819434 >> v2:172.21.15.154:3300/0 conn(0x7f25440b6720 0x7f25441623c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.162+0000 7f254a542640 1 -- 172.21.15.88:0/1102819434 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25440cfe20 con 0x7f254415e7a0 2024-09-16T01:36:59.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.162+0000 7f254a542640 1 --2- 172.21.15.88:0/1102819434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f254415e7a0 0x7f254406caa0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f2544163460 tx=0x7f2540032e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5355605646fb7895 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:59.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.163+0000 7f2549540640 1 -- 172.21.15.88:0/1102819434 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2540033960 con 0x7f254415e7a0 2024-09-16T01:36:59.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.163+0000 7f2549540640 1 -- 172.21.15.88:0/1102819434 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2540033b00 con 0x7f254415e7a0 2024-09-16T01:36:59.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.163+0000 7f2549540640 1 -- 172.21.15.88:0/1102819434 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254003b3f0 con 0x7f254415e7a0 2024-09-16T01:36:59.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.163+0000 7f254c7a6640 1 -- 172.21.15.88:0/1102819434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f254415e7a0 msgr2=0x7f254406caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:59.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.163+0000 7f254c7a6640 1 --2- 172.21.15.88:0/1102819434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f254415e7a0 0x7f254406caa0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f2544163460 tx=0x7f2540032e20 comp rx=0 tx=0).stop 2024-09-16T01:36:59.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.163+0000 7f254c7a6640 1 -- 172.21.15.88:0/1102819434 shutdown_connections 2024-09-16T01:36:59.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.163+0000 7f254c7a6640 1 --2- 172.21.15.88:0/1102819434 >> v2:172.21.15.154:3300/0 conn(0x7f25440b6720 0x7f25441623c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.163+0000 7f254c7a6640 1 --2- 172.21.15.88:0/1102819434 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25440cf1f0 0x7f25440b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.163+0000 7f254c7a6640 1 --2- 172.21.15.88:0/1102819434 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f254415e7a0 0x7f254406caa0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.163+0000 7f254c7a6640 1 -- 172.21.15.88:0/1102819434 >> 172.21.15.88:0/1102819434 conn(0x7f25440d11e0 msgr2=0x7f25440d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:59.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.163+0000 7f254c7a6640 1 -- 172.21.15.88:0/1102819434 shutdown_connections 2024-09-16T01:36:59.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.163+0000 7f254c7a6640 1 -- 172.21.15.88:0/1102819434 wait complete. 2024-09-16T01:36:59.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.164+0000 7f254c7a6640 1 Processor -- start 2024-09-16T01:36:59.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.164+0000 7f254c7a6640 1 -- start start 2024-09-16T01:36:59.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.165+0000 7f254c7a6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25440b6720 0x7f25440d4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.165+0000 7f254c7a6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25440cf1f0 0x7f25440c8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.165+0000 7f254c7a6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f254415e7a0 0x7f25440b8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.165+0000 7f254c7a6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f254406cfe0 con 0x7f25440b6720 2024-09-16T01:36:59.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.165+0000 7f254c7a6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f25440b7a00 con 0x7f25440cf1f0 2024-09-16T01:36:59.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.165+0000 7f254c7a6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f254415eb80 con 0x7f254415e7a0 2024-09-16T01:36:59.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.165+0000 7f254a542640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25440b6720 0x7f25440d4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.165+0000 7f254a542640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25440b6720 0x7f25440d4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44100/0 (socket says 172.21.15.88:44100) 2024-09-16T01:36:59.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.165+0000 7f254a542640 1 -- 172.21.15.88:0/4284034164 learned_addr learned my addr 172.21.15.88:0/4284034164 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:59.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.165+0000 7f2549d41640 1 --2- 172.21.15.88:0/4284034164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25440cf1f0 0x7f25440c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.165+0000 7f254ad43640 1 --2- 172.21.15.88:0/4284034164 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f254415e7a0 0x7f25440b8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.166+0000 7f254a542640 1 -- 172.21.15.88:0/4284034164 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f254415e7a0 msgr2=0x7f25440b8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:59.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.166+0000 7f254a542640 1 --2- 172.21.15.88:0/4284034164 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f254415e7a0 0x7f25440b8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.166+0000 7f254a542640 1 -- 172.21.15.88:0/4284034164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25440cf1f0 msgr2=0x7f25440c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:59.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.166+0000 7f254a542640 1 --2- 172.21.15.88:0/4284034164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25440cf1f0 0x7f25440c8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.166+0000 7f254a542640 1 -- 172.21.15.88:0/4284034164 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2544157950 con 0x7f25440b6720 2024-09-16T01:36:59.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.166+0000 7f254ad43640 1 --2- 172.21.15.88:0/4284034164 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f254415e7a0 0x7f25440b8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:59.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.166+0000 7f254a542640 1 --2- 172.21.15.88:0/4284034164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25440b6720 0x7f25440d4d00 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f2540000f80 tx=0x7f2540004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:59.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.166+0000 7f253affd640 1 -- 172.21.15.88:0/4284034164 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254004a020 con 0x7f25440b6720 2024-09-16T01:36:59.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.166+0000 7f253affd640 1 -- 172.21.15.88:0/4284034164 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2540045450 con 0x7f25440b6720 2024-09-16T01:36:59.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.166+0000 7f254c7a6640 1 -- 172.21.15.88:0/4284034164 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2544157bf0 con 0x7f25440b6720 2024-09-16T01:36:59.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.166+0000 7f253affd640 1 -- 172.21.15.88:0/4284034164 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2540009510 con 0x7f25440b6720 2024-09-16T01:36:59.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.166+0000 7f254c7a6640 1 -- 172.21.15.88:0/4284034164 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25441581b0 con 0x7f25440b6720 2024-09-16T01:36:59.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.167+0000 7f253affd640 1 -- 172.21.15.88:0/4284034164 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f254004d020 con 0x7f25440b6720 2024-09-16T01:36:59.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.168+0000 7f253affd640 1 --2- 172.21.15.88:0/4284034164 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f252806c2b0 0x7f252806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.168+0000 7f2549d41640 1 --2- 172.21.15.88:0/4284034164 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f252806c2b0 0x7f252806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.168+0000 7f253affd640 1 -- 172.21.15.88:0/4284034164 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5795+0+0 (secure 0 0 0) 0x7f25400b4d10 con 0x7f25440b6720 2024-09-16T01:36:59.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.169+0000 7f2549d41640 1 --2- 172.21.15.88:0/4284034164 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f252806c2b0 0x7f252806e770 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f25440b9f50 tx=0x7f2534008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:59.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.169+0000 7f254f620740 1 Processor -- start 2024-09-16T01:36:59.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.169+0000 7f254f620740 1 -- start start 2024-09-16T01:36:59.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.170+0000 7f254f620740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650fea932c0 0x5650fead8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.170+0000 7f254f620740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650fea90a80 0x5650fea936a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.170+0000 7f254f620740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fea94a50 0x5650fea8d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.170+0000 7f254f620740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5650fe9f5520 con 0x5650fea94a50 2024-09-16T01:36:59.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.170+0000 7f254f620740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5650fe99d3e0 con 0x5650fea90a80 2024-09-16T01:36:59.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.170+0000 7f254f620740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5650fe9f14e0 con 0x5650fea932c0 2024-09-16T01:36:59.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.170+0000 7f254ad43640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fea94a50 0x5650fea8d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.170+0000 7f254ad43640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fea94a50 0x5650fea8d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44106/0 (socket says 172.21.15.88:44106) 2024-09-16T01:36:59.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.170+0000 7f254ad43640 1 -- 172.21.15.88:0/945352118 learned_addr learned my addr 172.21.15.88:0/945352118 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:59.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.170+0000 7f254ad43640 1 -- 172.21.15.88:0/945352118 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650fea932c0 msgr2=0x5650fead8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:59.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.170+0000 7f254a542640 1 --2- 172.21.15.88:0/945352118 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650fea932c0 0x5650fead8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.170+0000 7f2549d41640 1 --2- 172.21.15.88:0/945352118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650fea90a80 0x5650fea936a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.170+0000 7f254ad43640 1 --2- 172.21.15.88:0/945352118 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650fea932c0 0x5650fead8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.170+0000 7f254ad43640 1 -- 172.21.15.88:0/945352118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650fea90a80 msgr2=0x5650fea936a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:59.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.170+0000 7f254ad43640 1 --2- 172.21.15.88:0/945352118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650fea90a80 0x5650fea936a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.170+0000 7f254ad43640 1 -- 172.21.15.88:0/945352118 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650fe9b82f0 con 0x5650fea94a50 2024-09-16T01:36:59.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.171+0000 7f254ad43640 1 --2- 172.21.15.88:0/945352118 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fea94a50 0x5650fea8d8b0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f253c00b960 tx=0x7f253c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ba543e5ef85a6ad server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:59.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.171+0000 7f2527fff640 1 -- 172.21.15.88:0/945352118 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c018020 con 0x5650fea94a50 2024-09-16T01:36:59.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.171+0000 7f2527fff640 1 -- 172.21.15.88:0/945352118 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f253c00fa80 con 0x5650fea94a50 2024-09-16T01:36:59.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.171+0000 7f2527fff640 1 -- 172.21.15.88:0/945352118 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c00fe70 con 0x5650fea94a50 2024-09-16T01:36:59.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.171+0000 7f254f620740 1 -- 172.21.15.88:0/945352118 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fea94a50 msgr2=0x5650fea8d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:59.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.171+0000 7f254f620740 1 --2- 172.21.15.88:0/945352118 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fea94a50 0x5650fea8d8b0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f253c00b960 tx=0x7f253c00be20 comp rx=0 tx=0).stop 2024-09-16T01:36:59.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.171+0000 7f254f620740 1 -- 172.21.15.88:0/945352118 shutdown_connections 2024-09-16T01:36:59.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.171+0000 7f254f620740 1 --2- 172.21.15.88:0/945352118 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fea94a50 0x5650fea8d8b0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.171+0000 7f254f620740 1 --2- 172.21.15.88:0/945352118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650fea90a80 0x5650fea936a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.171+0000 7f254f620740 1 --2- 172.21.15.88:0/945352118 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650fea932c0 0x5650fead8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.171+0000 7f254f620740 1 -- 172.21.15.88:0/945352118 >> 172.21.15.88:0/945352118 conn(0x5650fea8fa80 msgr2=0x5650fead7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:59.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.172+0000 7f254f620740 1 -- 172.21.15.88:0/945352118 shutdown_connections 2024-09-16T01:36:59.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.172+0000 7f254f620740 1 -- 172.21.15.88:0/945352118 wait complete. 2024-09-16T01:36:59.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.172+0000 7f254f620740 1 -- start start 2024-09-16T01:36:59.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.172+0000 7f254f620740 1 Processor -- start 2024-09-16T01:36:59.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.173+0000 7f254f620740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650fea932c0 0x5650fea8ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.173+0000 7f254f620740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650fea94a50 0x5650feaebc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.173+0000 7f254f620740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fead5870 0x5650feabd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.173+0000 7f254f620740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5650fea8ef20 con 0x5650fead5870 2024-09-16T01:36:59.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.173+0000 7f254f620740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5650fe97e450 con 0x5650fea94a50 2024-09-16T01:36:59.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.173+0000 7f254f620740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5650fead9290 con 0x5650fea932c0 2024-09-16T01:36:59.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.173+0000 7f254f620740 10 client.0 _ll_register_callbacks cb 0x5650fe707ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:36:59.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.173+0000 7f254ad43640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fead5870 0x5650feabd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.173+0000 7f254ad43640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fead5870 0x5650feabd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44122/0 (socket says 172.21.15.88:44122) 2024-09-16T01:36:59.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.173+0000 7f254ad43640 1 -- 172.21.15.88:0/2428100846 learned_addr learned my addr 172.21.15.88:0/2428100846 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:59.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.173+0000 7f254a542640 1 --2- 172.21.15.88:0/2428100846 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650fea932c0 0x5650fea8ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.173+0000 7f2549d41640 1 --2- 172.21.15.88:0/2428100846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650fea94a50 0x5650feaebc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.173+0000 7f254ad43640 1 -- 172.21.15.88:0/2428100846 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650fea932c0 msgr2=0x5650fea8ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:59.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.173+0000 7f254ad43640 1 --2- 172.21.15.88:0/2428100846 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650fea932c0 0x5650fea8ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.173+0000 7f254ad43640 1 -- 172.21.15.88:0/2428100846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650fea94a50 msgr2=0x5650feaebc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:59.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.173+0000 7f254ad43640 1 --2- 172.21.15.88:0/2428100846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650fea94a50 0x5650feaebc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.173+0000 7f254ad43640 1 -- 172.21.15.88:0/2428100846 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650fe7082d0 con 0x5650fead5870 2024-09-16T01:36:59.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.173+0000 7f2549d41640 1 --2- 172.21.15.88:0/2428100846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650fea94a50 0x5650feaebc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:59.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.174+0000 7f254ad43640 1 --2- 172.21.15.88:0/2428100846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fead5870 0x5650feabd6c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f253c014670 tx=0x7f253c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f3eed61fbaa8880 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:59.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.174+0000 7f25257fa640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:36:59.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.174+0000 7f25257fa640 1 -- 172.21.15.88:0/2428100846 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c018070 con 0x5650fead5870 2024-09-16T01:36:59.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.174+0000 7f25257fa640 1 -- 172.21.15.88:0/2428100846 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(59..59 src has 1..59) ==== 5795+0+0 (secure 0 0 0) 0x7f253c00f880 con 0x5650fead5870 2024-09-16T01:36:59.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.174+0000 7f254f620740 1 -- 172.21.15.88:0/2428100846 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f253c00fa80 con 0x5650fead5870 2024-09-16T01:36:59.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.174+0000 7f25257fa640 1 -- 172.21.15.88:0/2428100846 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f253c02f050 con 0x5650fead5870 2024-09-16T01:36:59.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.174+0000 7f254f620740 10 client.6096 fetch_fsmap learned FSMap version 44 2024-09-16T01:36:59.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.174+0000 7f254f620740 1 -- 172.21.15.88:0/2428100846 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=2+}) -- 0x5650fe7d26b0 con 0x5650fead5870 2024-09-16T01:36:59.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.174+0000 7f25257fa640 1 -- 172.21.15.88:0/2428100846 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f253c00fa80 con 0x5650fead5870 2024-09-16T01:36:59.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.174+0000 7f25257fa640 1 -- 172.21.15.88:0/2428100846 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 44) ==== 3502+0+0 (secure 0 0 0) 0x7f253c032020 con 0x5650fead5870 2024-09-16T01:36:59.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.174+0000 7f254f620740 10 client.6096 fetch_fsmap finished waiting for FSMap version 44 2024-09-16T01:36:59.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.174+0000 7f254f620740 10 client.6096 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:36:59.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.174+0000 7f254f620740 20 client.6096 populate_metadata read hostname 'smithi088' 2024-09-16T01:36:59.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.174+0000 7f254f620740 1 --2- 172.21.15.88:0/2428100846 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x5650feab2930 0x5650feab2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.174+0000 7f254f620740 4 client.6096 mds_command: new command op to 6054 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:36:59.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.174+0000 7f254f620740 1 -- 172.21.15.88:0/2428100846 --> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5650fea02c50 con 0x5650feab2930 2024-09-16T01:36:59.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.174+0000 7f254a542640 1 --2- 172.21.15.88:0/2428100846 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x5650feab2930 0x5650feab2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.174+0000 7f25257fa640 10 client.6096 ms_handle_connect on v2:172.21.15.88:6835/4130074102 2024-09-16T01:36:59.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.175+0000 7f254a542640 1 --2- 172.21.15.88:0/2428100846 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x5650feab2930 0x5650feab2d10 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43f625709a50b054 server_cookie=2102316f29eebc3e in_seq=0 out_seq=0 2024-09-16T01:36:59.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.175+0000 7f25257fa640 1 -- 172.21.15.88:0/2428100846 <== mds.0 v2:172.21.15.88:6835/4130074102 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5650fea02c50 con 0x5650feab2930 2024-09-16T01:36:59.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.175+0000 7f25257fa640 10 client.6096 handle_command_reply: tid=0 2024-09-16T01:36:59.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.175+0000 7f254f620740 1 client.6096 shutdown 2024-09-16T01:36:59.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.175+0000 7f254f620740 1 -- 172.21.15.88:0/2428100846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fead5870 msgr2=0x5650feabd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:59.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.175+0000 7f254f620740 1 --2- 172.21.15.88:0/2428100846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fead5870 0x5650feabd6c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f253c014670 tx=0x7f253c004730 comp rx=0 tx=0).stop 2024-09-16T01:36:59.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.175+0000 7f254f620740 1 -- 172.21.15.88:0/2428100846 shutdown_connections 2024-09-16T01:36:59.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.175+0000 7f254f620740 1 --2- 172.21.15.88:0/2428100846 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x5650feab2930 0x5650feab2d10 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.175+0000 7f254f620740 1 --2- 172.21.15.88:0/2428100846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fead5870 0x5650feabd6c0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.175+0000 7f25257fa640 0 client.6096 ms_handle_reset on v2:172.21.15.88:6835/4130074102 2024-09-16T01:36:59.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.175+0000 7f254f620740 1 --2- 172.21.15.88:0/2428100846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650fea94a50 0x5650feaebc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.175+0000 7f254f620740 1 --2- 172.21.15.88:0/2428100846 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650fea932c0 0x5650fea8ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.175+0000 7f254f620740 1 -- 172.21.15.88:0/2428100846 >> 172.21.15.88:0/2428100846 conn(0x5650fea90a80 msgr2=0x5650feadb440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:59.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.176+0000 7f254f620740 1 -- 172.21.15.88:0/2428100846 shutdown_connections 2024-09-16T01:36:59.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.176+0000 7f254f620740 1 -- 172.21.15.88:0/2428100846 wait complete. 2024-09-16T01:36:59.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.176+0000 7f254f620740 20 client.6096 trim_cache size 0 max 16384 2024-09-16T01:36:59.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.193+0000 7f254f620740 1 Processor -- start 2024-09-16T01:36:59.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.193+0000 7f254f620740 1 -- start start 2024-09-16T01:36:59.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.194+0000 7f254f620740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650fea932c0 0x5650feadb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.194+0000 7f254f620740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fea94a50 0x5650feadb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.194+0000 7f254f620740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650fead5870 0x5650fead65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.194+0000 7f254f620740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2544159ca0 con 0x5650fea94a50 2024-09-16T01:36:59.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.194+0000 7f254f620740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f25440cb000 con 0x5650fea932c0 2024-09-16T01:36:59.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.194+0000 7f254f620740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2544159650 con 0x5650fead5870 2024-09-16T01:36:59.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.194+0000 7f2549d41640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fea94a50 0x5650feadb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.194+0000 7f2549d41640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fea94a50 0x5650feadb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44130/0 (socket says 172.21.15.88:44130) 2024-09-16T01:36:59.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.194+0000 7f2549d41640 1 -- 172.21.15.88:0/3189846977 learned_addr learned my addr 172.21.15.88:0/3189846977 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:59.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.194+0000 7f2549d41640 1 -- 172.21.15.88:0/3189846977 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650fead5870 msgr2=0x5650fead65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:59.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.194+0000 7f2549d41640 1 --2- 172.21.15.88:0/3189846977 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650fead5870 0x5650fead65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.194+0000 7f2549d41640 1 -- 172.21.15.88:0/3189846977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650fea932c0 msgr2=0x5650feadb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:59.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.194+0000 7f2549d41640 1 --2- 172.21.15.88:0/3189846977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650fea932c0 0x5650feadb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.194+0000 7f2549d41640 1 -- 172.21.15.88:0/3189846977 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650feabca70 con 0x5650fea94a50 2024-09-16T01:36:59.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.194+0000 7f2549d41640 1 --2- 172.21.15.88:0/3189846977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fea94a50 0x5650feadb980 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f2534010a80 tx=0x7f2534016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b64d616acb3ab21e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:59.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.194+0000 7f25277fe640 1 -- 172.21.15.88:0/3189846977 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2534016b90 con 0x5650fea94a50 2024-09-16T01:36:59.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.194+0000 7f25277fe640 1 -- 172.21.15.88:0/3189846977 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2534028070 con 0x5650fea94a50 2024-09-16T01:36:59.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.194+0000 7f25277fe640 1 -- 172.21.15.88:0/3189846977 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2534016d50 con 0x5650fea94a50 2024-09-16T01:36:59.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.195+0000 7f254f620740 1 -- 172.21.15.88:0/3189846977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fea94a50 msgr2=0x5650feadb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:59.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.195+0000 7f254f620740 1 --2- 172.21.15.88:0/3189846977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fea94a50 0x5650feadb980 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f2534010a80 tx=0x7f2534016000 comp rx=0 tx=0).stop 2024-09-16T01:36:59.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.195+0000 7f254f620740 1 -- 172.21.15.88:0/3189846977 shutdown_connections 2024-09-16T01:36:59.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.195+0000 7f254f620740 1 --2- 172.21.15.88:0/3189846977 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650fead5870 0x5650fead65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.195+0000 7f254f620740 1 --2- 172.21.15.88:0/3189846977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fea94a50 0x5650feadb980 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.195+0000 7f254f620740 1 --2- 172.21.15.88:0/3189846977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650fea932c0 0x5650feadb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.195+0000 7f254f620740 1 -- 172.21.15.88:0/3189846977 >> 172.21.15.88:0/3189846977 conn(0x5650fea90a80 msgr2=0x5650fea8c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:59.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.195+0000 7f254f620740 1 -- 172.21.15.88:0/3189846977 shutdown_connections 2024-09-16T01:36:59.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.195+0000 7f254f620740 1 -- 172.21.15.88:0/3189846977 wait complete. 2024-09-16T01:36:59.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.196+0000 7f254f620740 1 -- start start 2024-09-16T01:36:59.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.196+0000 7f254f620740 1 Processor -- start 2024-09-16T01:36:59.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f254f620740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fea932c0 0x5650feab2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f254f620740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650fea94a50 0x5650feb3a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f254f620740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650fead5870 0x5650fead6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f254f620740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5650feaec200 con 0x5650fea932c0 2024-09-16T01:36:59.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f254f620740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5650feabd260 con 0x5650fead5870 2024-09-16T01:36:59.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f254f620740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5650feabdc00 con 0x5650fea94a50 2024-09-16T01:36:59.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f254f620740 10 client.0 _ll_register_callbacks cb 0x7f253c00b750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:36:59.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f254a542640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fea932c0 0x5650feab2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f254a542640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fea932c0 0x5650feab2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44140/0 (socket says 172.21.15.88:44140) 2024-09-16T01:36:59.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f254a542640 1 -- 172.21.15.88:0/3215382608 learned_addr learned my addr 172.21.15.88:0/3215382608 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:59.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f2549d41640 1 --2- 172.21.15.88:0/3215382608 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650fea94a50 0x5650feb3a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f254ad43640 1 --2- 172.21.15.88:0/3215382608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650fead5870 0x5650fead6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f254a542640 1 -- 172.21.15.88:0/3215382608 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650fea94a50 msgr2=0x5650feb3a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:59.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f254a542640 1 --2- 172.21.15.88:0/3215382608 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650fea94a50 0x5650feb3a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f254a542640 1 -- 172.21.15.88:0/3215382608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650fead5870 msgr2=0x5650fead6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:59.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f254a542640 1 --2- 172.21.15.88:0/3215382608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650fead5870 0x5650fead6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f254a542640 1 -- 172.21.15.88:0/3215382608 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650feab37a0 con 0x5650fea932c0 2024-09-16T01:36:59.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f2549d41640 1 --2- 172.21.15.88:0/3215382608 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650fea94a50 0x5650feb3a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:36:59.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f254a542640 1 --2- 172.21.15.88:0/3215382608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fea932c0 0x5650feab2e30 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f2540084fd0 tx=0x7f25400826b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fffef9e27996fbb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:59.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f25257fa640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:36:59.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f25257fa640 1 -- 172.21.15.88:0/3215382608 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254009f060 con 0x5650fea932c0 2024-09-16T01:36:59.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f25257fa640 1 -- 172.21.15.88:0/3215382608 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(59..59 src has 1..59) ==== 5795+0+0 (secure 0 0 0) 0x7f254007f300 con 0x5650fea932c0 2024-09-16T01:36:59.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f254f620740 1 -- 172.21.15.88:0/3215382608 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2534028070 con 0x5650fea932c0 2024-09-16T01:36:59.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f25257fa640 1 -- 172.21.15.88:0/3215382608 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f25400b1050 con 0x5650fea932c0 2024-09-16T01:36:59.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f254f620740 10 client.6102 fetch_fsmap learned FSMap version 44 2024-09-16T01:36:59.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f254f620740 1 -- 172.21.15.88:0/3215382608 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=2+}) -- 0x5650feab2300 con 0x5650fea932c0 2024-09-16T01:36:59.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.197+0000 7f25257fa640 1 -- 172.21.15.88:0/3215382608 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2534028070 con 0x5650fea932c0 2024-09-16T01:36:59.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.198+0000 7f25257fa640 1 -- 172.21.15.88:0/3215382608 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 44) ==== 3502+0+0 (secure 0 0 0) 0x7f254009a030 con 0x5650fea932c0 2024-09-16T01:36:59.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.198+0000 7f254f620740 10 client.6102 fetch_fsmap finished waiting for FSMap version 44 2024-09-16T01:36:59.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.198+0000 7f254f620740 10 client.6102 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:36:59.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.198+0000 7f254f620740 20 client.6102 populate_metadata read hostname 'smithi088' 2024-09-16T01:36:59.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.198+0000 7f254f620740 1 --2- 172.21.15.88:0/3215382608 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x5650feab19b0 0x5650feab1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.198+0000 7f254f620740 4 client.6102 mds_command: new command op to 6054 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:36:59.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.198+0000 7f254f620740 1 -- 172.21.15.88:0/3215382608 --> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] -- command(tid 0: {"prefix": "status"}) -- 0x5650fe9fcfa0 con 0x5650feab19b0 2024-09-16T01:36:59.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.198+0000 7f2549d41640 1 --2- 172.21.15.88:0/3215382608 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x5650feab19b0 0x5650feab1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.198+0000 7f25257fa640 10 client.6102 ms_handle_connect on v2:172.21.15.88:6835/4130074102 2024-09-16T01:36:59.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.198+0000 7f2549d41640 1 --2- 172.21.15.88:0/3215382608 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x5650feab19b0 0x5650feab1d90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9526ccf4f81b025b server_cookie=e4142ab2bfb16b2f in_seq=0 out_seq=0 2024-09-16T01:36:59.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.198+0000 7f25257fa640 1 -- 172.21.15.88:0/3215382608 <== mds.0 v2:172.21.15.88:6835/4130074102 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x5650fe9fcfa0 con 0x5650feab19b0 2024-09-16T01:36:59.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.198+0000 7f25257fa640 10 client.6102 handle_command_reply: tid=0 2024-09-16T01:36:59.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.198+0000 7f254f620740 1 client.6102 shutdown 2024-09-16T01:36:59.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.198+0000 7f254f620740 1 -- 172.21.15.88:0/3215382608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fea932c0 msgr2=0x5650feab2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:59.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.198+0000 7f254f620740 1 --2- 172.21.15.88:0/3215382608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fea932c0 0x5650feab2e30 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f2540084fd0 tx=0x7f25400826b0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.198+0000 7f254f620740 1 -- 172.21.15.88:0/3215382608 shutdown_connections 2024-09-16T01:36:59.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.198+0000 7f254f620740 1 --2- 172.21.15.88:0/3215382608 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x5650feab19b0 0x5650feab1d90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.198+0000 7f254f620740 1 --2- 172.21.15.88:0/3215382608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650fead5870 0x5650fead6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.198+0000 7f254f620740 1 --2- 172.21.15.88:0/3215382608 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650fea94a50 0x5650feb3a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.198+0000 7f254f620740 1 --2- 172.21.15.88:0/3215382608 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5650fea932c0 0x5650feab2e30 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.198+0000 7f25257fa640 0 client.6102 ms_handle_reset on v2:172.21.15.88:6835/4130074102 2024-09-16T01:36:59.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.198+0000 7f254f620740 1 -- 172.21.15.88:0/3215382608 >> 172.21.15.88:0/3215382608 conn(0x5650fea90a80 msgr2=0x5650feadb440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:59.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.198+0000 7f254f620740 1 -- 172.21.15.88:0/3215382608 shutdown_connections 2024-09-16T01:36:59.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.198+0000 7f254f620740 1 -- 172.21.15.88:0/3215382608 wait complete. 2024-09-16T01:36:59.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.199+0000 7f254f620740 20 client.6102 trim_cache size 0 max 16384 2024-09-16T01:36:59.212 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:36:59.212 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "5ace2c91-46f9-482b-8a89-731fbdc171bf", 2024-09-16T01:36:59.212 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2024-09-16T01:36:59.213 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 6054, 2024-09-16T01:36:59.213 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2024-09-16T01:36:59.213 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2024-09-16T01:36:59.213 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2024-09-16T01:36:59.213 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 4.3561067060000003, 2024-09-16T01:36:59.213 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 44, 2024-09-16T01:36:59.213 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 59, 2024-09-16T01:36:59.213 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 59, 2024-09-16T01:36:59.213 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 10.339575965 2024-09-16T01:36:59.213 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:36:59.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.199+0000 7f254c7a6640 1 -- 172.21.15.88:0/4284034164 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f252806c2b0 msgr2=0x7f252806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:59.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.199+0000 7f254c7a6640 1 --2- 172.21.15.88:0/4284034164 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f252806c2b0 0x7f252806e770 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f25440b9f50 tx=0x7f2534008040 comp rx=0 tx=0).stop 2024-09-16T01:36:59.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.199+0000 7f254c7a6640 1 -- 172.21.15.88:0/4284034164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25440b6720 msgr2=0x7f25440d4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:59.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.199+0000 7f254c7a6640 1 --2- 172.21.15.88:0/4284034164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25440b6720 0x7f25440d4d00 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f2540000f80 tx=0x7f2540004a40 comp rx=0 tx=0).stop 2024-09-16T01:36:59.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.199+0000 7f254c7a6640 1 -- 172.21.15.88:0/4284034164 shutdown_connections 2024-09-16T01:36:59.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.199+0000 7f254c7a6640 1 --2- 172.21.15.88:0/4284034164 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f252806c2b0 0x7f252806e770 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.199+0000 7f254c7a6640 1 --2- 172.21.15.88:0/4284034164 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f254415e7a0 0x7f25440b8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.199+0000 7f254c7a6640 1 --2- 172.21.15.88:0/4284034164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25440cf1f0 0x7f25440c8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.199+0000 7f254c7a6640 1 --2- 172.21.15.88:0/4284034164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25440b6720 0x7f25440d4d00 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.199+0000 7f254c7a6640 1 -- 172.21.15.88:0/4284034164 >> 172.21.15.88:0/4284034164 conn(0x7f25440d11e0 msgr2=0x7f25440d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:59.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.199+0000 7f254c7a6640 1 -- 172.21.15.88:0/4284034164 shutdown_connections 2024-09-16T01:36:59.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.199+0000 7f254c7a6640 1 -- 172.21.15.88:0/4284034164 wait complete. 2024-09-16T01:36:59.216 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:36:59.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.313+0000 7f953545c640 1 Processor -- start 2024-09-16T01:36:59.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.313+0000 7f953545c640 1 -- start start 2024-09-16T01:36:59.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.314+0000 7f953545c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f953015e7a0 0x7f95300ba240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.314+0000 7f953545c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95300cf2f0 0x7f95300cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.314+0000 7f953545c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95300d4430 con 0x7f95300d3fe0 2024-09-16T01:36:59.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.314+0000 7f953545c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f95300d2f40 con 0x7f953015e7a0 2024-09-16T01:36:59.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.314+0000 7f953545c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f95300d3220 con 0x7f95300cf2f0 2024-09-16T01:36:59.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.314+0000 7f952effd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f953015e7a0 0x7f95300ba240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.314+0000 7f95267fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f95300d3fe0 0x7f95300d2780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58508/0 (socket says 172.21.15.88:58508) 2024-09-16T01:36:59.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.314+0000 7f952effd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f953015e7a0 0x7f95300ba240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44146/0 (socket says 172.21.15.88:44146) 2024-09-16T01:36:59.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.314+0000 7f95267fc640 1 -- 172.21.15.88:0/3079588129 learned_addr learned my addr 172.21.15.88:0/3079588129 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:59.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.315+0000 7f952effd640 1 -- 172.21.15.88:0/3079588129 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95300cf2f0 msgr2=0x7f95300cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:36:59.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.315+0000 7f952effd640 1 --2- 172.21.15.88:0/3079588129 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95300cf2f0 0x7f95300cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.315+0000 7f952effd640 1 -- 172.21.15.88:0/3079588129 >> v1:172.21.15.88:6789/0 conn(0x7f95300d3fe0 legacy=0x7f95300d2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:59.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.315+0000 7f952effd640 1 -- 172.21.15.88:0/3079588129 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95300d3500 con 0x7f953015e7a0 2024-09-16T01:36:59.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.315+0000 7f952e7fc640 1 -- 172.21.15.88:0/3079588129 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1000025289 0 0) 0x7f95300d4430 con 0x7f95300d3fe0 2024-09-16T01:36:59.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.315+0000 7f952effd640 1 --2- 172.21.15.88:0/3079588129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f953015e7a0 0x7f95300ba240 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f95200097a0 tx=0x7f952002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9619ef3d28ee05ca server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:59.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.315+0000 7f952e7fc640 1 -- 172.21.15.88:0/3079588129 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9520037070 con 0x7f953015e7a0 2024-09-16T01:36:59.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.315+0000 7f952e7fc640 1 -- 172.21.15.88:0/3079588129 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9520032440 con 0x7f953015e7a0 2024-09-16T01:36:59.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.315+0000 7f952e7fc640 1 -- 172.21.15.88:0/3079588129 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9520032810 con 0x7f953015e7a0 2024-09-16T01:36:59.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.316+0000 7f953545c640 1 -- 172.21.15.88:0/3079588129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f953015e7a0 msgr2=0x7f95300ba240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:59.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.316+0000 7f953545c640 1 --2- 172.21.15.88:0/3079588129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f953015e7a0 0x7f95300ba240 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f95200097a0 tx=0x7f952002a210 comp rx=0 tx=0).stop 2024-09-16T01:36:59.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.316+0000 7f953545c640 1 -- 172.21.15.88:0/3079588129 shutdown_connections 2024-09-16T01:36:59.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.316+0000 7f953545c640 1 --2- 172.21.15.88:0/3079588129 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95300cf2f0 0x7f95300cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.316+0000 7f953545c640 1 --2- 172.21.15.88:0/3079588129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f953015e7a0 0x7f95300ba240 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.316+0000 7f953545c640 1 -- 172.21.15.88:0/3079588129 >> 172.21.15.88:0/3079588129 conn(0x7f95300d1550 msgr2=0x7f95300d1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:59.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.316+0000 7f953545c640 1 -- 172.21.15.88:0/3079588129 shutdown_connections 2024-09-16T01:36:59.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.316+0000 7f953545c640 1 -- 172.21.15.88:0/3079588129 wait complete. 2024-09-16T01:36:59.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.317+0000 7f953545c640 1 Processor -- start 2024-09-16T01:36:59.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.317+0000 7f953545c640 1 -- start start 2024-09-16T01:36:59.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.318+0000 7f953545c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95300cf2f0 0x7f95301f8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.318+0000 7f953545c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95300d3fe0 0x7f95300b90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.318+0000 7f953545c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f953015e7a0 0x7f95300b9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.318+0000 7f953545c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f953015eb80 con 0x7f953015e7a0 2024-09-16T01:36:59.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.318+0000 7f953545c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f95300d3ac0 con 0x7f95300d3fe0 2024-09-16T01:36:59.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.318+0000 7f953545c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f953015ed00 con 0x7f95300cf2f0 2024-09-16T01:36:59.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.318+0000 7f952f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f953015e7a0 0x7f95300b9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.318+0000 7f952f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f953015e7a0 0x7f95300b9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44150/0 (socket says 172.21.15.88:44150) 2024-09-16T01:36:59.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.318+0000 7f952f7fe640 1 -- 172.21.15.88:0/802687906 learned_addr learned my addr 172.21.15.88:0/802687906 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:59.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.318+0000 7f952effd640 1 --2- 172.21.15.88:0/802687906 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95300cf2f0 0x7f95301f8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.318+0000 7f95267fc640 1 --2- 172.21.15.88:0/802687906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95300d3fe0 0x7f95300b90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.319+0000 7f952f7fe640 1 -- 172.21.15.88:0/802687906 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95300cf2f0 msgr2=0x7f95301f8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:59.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.319+0000 7f952f7fe640 1 --2- 172.21.15.88:0/802687906 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95300cf2f0 0x7f95301f8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.319+0000 7f952f7fe640 1 -- 172.21.15.88:0/802687906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95300d3fe0 msgr2=0x7f95300b90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:59.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.319+0000 7f952f7fe640 1 --2- 172.21.15.88:0/802687906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95300d3fe0 0x7f95300b90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.319+0000 7f952f7fe640 1 -- 172.21.15.88:0/802687906 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95301f4520 con 0x7f953015e7a0 2024-09-16T01:36:59.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.319+0000 7f952effd640 1 --2- 172.21.15.88:0/802687906 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95300cf2f0 0x7f95301f8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:59.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.319+0000 7f95267fc640 1 --2- 172.21.15.88:0/802687906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95300d3fe0 0x7f95300b90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:36:59.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.319+0000 7f952f7fe640 1 --2- 172.21.15.88:0/802687906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f953015e7a0 0x7f95300b9600 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f952800b970 tx=0x7f952800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:59.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.319+0000 7f9527fff640 1 -- 172.21.15.88:0/802687906 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9528019070 con 0x7f953015e7a0 2024-09-16T01:36:59.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.319+0000 7f9527fff640 1 -- 172.21.15.88:0/802687906 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f952800cb60 con 0x7f953015e7a0 2024-09-16T01:36:59.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.319+0000 7f953545c640 1 -- 172.21.15.88:0/802687906 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95301f4800 con 0x7f953015e7a0 2024-09-16T01:36:59.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.319+0000 7f9527fff640 1 -- 172.21.15.88:0/802687906 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9528014440 con 0x7f953015e7a0 2024-09-16T01:36:59.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.319+0000 7f953545c640 1 -- 172.21.15.88:0/802687906 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95301f52e0 con 0x7f953015e7a0 2024-09-16T01:36:59.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.320+0000 7f953545c640 1 -- 172.21.15.88:0/802687906 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95300ba240 con 0x7f953015e7a0 2024-09-16T01:36:59.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.320+0000 7f9527fff640 1 -- 172.21.15.88:0/802687906 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f952801c070 con 0x7f953015e7a0 2024-09-16T01:36:59.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.321+0000 7f9527fff640 1 --2- 172.21.15.88:0/802687906 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f951006c2e0 0x7f951006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.321+0000 7f9527fff640 1 -- 172.21.15.88:0/802687906 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5795+0+0 (secure 0 0 0) 0x7f952808b190 con 0x7f953015e7a0 2024-09-16T01:36:59.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.321+0000 7f952effd640 1 --2- 172.21.15.88:0/802687906 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f951006c2e0 0x7f951006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.321+0000 7f952effd640 1 --2- 172.21.15.88:0/802687906 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f951006c2e0 0x7f951006e7a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9520034000 tx=0x7f952002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:59.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.325+0000 7f9527fff640 1 -- 172.21.15.88:0/802687906 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f95280598d0 con 0x7f953015e7a0 2024-09-16T01:36:59.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.525+0000 7f953545c640 1 -- 172.21.15.88:0/802687906 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95300cf9d0 con 0x7f953015e7a0 2024-09-16T01:36:59.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.526+0000 7f9527fff640 1 -- 172.21.15.88:0/802687906 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+5224 (secure 0 0 0) 0x7f95280591d0 con 0x7f953015e7a0 2024-09-16T01:36:59.527 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:36:59.527 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":44,"btime":"2024-09-16T01:36:57:852065+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/581241194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":581241194},{"type":"v1","addr":"172.21.15.154:6834","nonce":581241194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2104508521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2104508521},{"type":"v1","addr":"172.21.15.154:6835","nonce":2104508521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5859,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/998283126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":998283126},{"type":"v1","addr":"172.21.15.88:6836","nonce":998283126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:39.786438+0000","modified":"2024-09-16T01:36:57.852059+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":5,"addr":"172.21.15.88:6837/4130074102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4130074102},{"type":"v1","addr":"172.21.15.88:6837","nonce":4130074102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":4}]} 2024-09-16T01:36:59.528 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 44 2024-09-16T01:36:59.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.528+0000 7f953545c640 1 -- 172.21.15.88:0/802687906 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f951006c2e0 msgr2=0x7f951006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:59.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.528+0000 7f953545c640 1 --2- 172.21.15.88:0/802687906 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f951006c2e0 0x7f951006e7a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9520034000 tx=0x7f952002e000 comp rx=0 tx=0).stop 2024-09-16T01:36:59.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.528+0000 7f953545c640 1 -- 172.21.15.88:0/802687906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f953015e7a0 msgr2=0x7f95300b9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:59.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.528+0000 7f953545c640 1 --2- 172.21.15.88:0/802687906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f953015e7a0 0x7f95300b9600 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f952800b970 tx=0x7f952800be30 comp rx=0 tx=0).stop 2024-09-16T01:36:59.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.528+0000 7f953545c640 1 -- 172.21.15.88:0/802687906 shutdown_connections 2024-09-16T01:36:59.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.528+0000 7f953545c640 1 --2- 172.21.15.88:0/802687906 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f951006c2e0 0x7f951006e7a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.528+0000 7f953545c640 1 --2- 172.21.15.88:0/802687906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f953015e7a0 0x7f95300b9600 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.528+0000 7f953545c640 1 --2- 172.21.15.88:0/802687906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95300d3fe0 0x7f95300b90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.528+0000 7f953545c640 1 --2- 172.21.15.88:0/802687906 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95300cf2f0 0x7f95301f8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.528+0000 7f953545c640 1 -- 172.21.15.88:0/802687906 >> 172.21.15.88:0/802687906 conn(0x7f95300d1550 msgr2=0x7f95300b7700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:59.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.528+0000 7f953545c640 1 -- 172.21.15.88:0/802687906 shutdown_connections 2024-09-16T01:36:59.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.528+0000 7f953545c640 1 -- 172.21.15.88:0/802687906 wait complete. 2024-09-16T01:36:59.540 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds freeze 4:0 true 2024-09-16T01:36:59.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.639+0000 7f7e35c8d640 1 Processor -- start 2024-09-16T01:36:59.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.639+0000 7f7e35c8d640 1 -- start start 2024-09-16T01:36:59.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.640+0000 7f7e35c8d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7e3006fa00 0x7f7e300d02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.640+0000 7f7e35c8d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e30071770 con 0x7f7e3015a060 2024-09-16T01:36:59.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.640+0000 7f7e35c8d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e300709b0 con 0x7f7e3015d3d0 2024-09-16T01:36:59.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.640+0000 7f7e35c8d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7e300712e0 con 0x7f7e3006fa00 2024-09-16T01:36:59.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.640+0000 7f7e2f7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7e3015a060 0x7f7e3006f2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58512/0 (socket says 172.21.15.88:58512) 2024-09-16T01:36:59.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.640+0000 7f7e2f7fe640 1 -- 172.21.15.88:0/4196872381 learned_addr learned my addr 172.21.15.88:0/4196872381 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:36:59.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.640+0000 7f7e2effd640 1 --2- 172.21.15.88:0/4196872381 >> v2:172.21.15.154:3300/0 conn(0x7f7e3006fa00 0x7f7e300d02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.640+0000 7f7e2e7fc640 1 -- 172.21.15.88:0/4196872381 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 345964032 0 0) 0x7f7e30071770 con 0x7f7e3015a060 2024-09-16T01:36:59.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.640+0000 7f7e2e7fc640 1 -- 172.21.15.88:0/4196872381 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e18003520 con 0x7f7e3015a060 2024-09-16T01:36:59.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.640+0000 7f7e2effd640 1 -- 172.21.15.88:0/4196872381 >> v1:172.21.15.154:6789/0 conn(0x7f7e3015d3d0 legacy=0x7f7e300d0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:59.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.640+0000 7f7e2effd640 1 -- 172.21.15.88:0/4196872381 >> v1:172.21.15.88:6789/0 conn(0x7f7e3015a060 legacy=0x7f7e3006f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:59.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.640+0000 7f7e2effd640 1 -- 172.21.15.88:0/4196872381 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e30070c90 con 0x7f7e3006fa00 2024-09-16T01:36:59.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.641+0000 7f7e2effd640 1 --2- 172.21.15.88:0/4196872381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e3006fa00 0x7f7e300d02d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7e240097a0 tx=0x7f7e2402a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2f98ccf4f335532 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:59.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.641+0000 7f7e2e7fc640 1 -- 172.21.15.88:0/4196872381 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e24037070 con 0x7f7e3006fa00 2024-09-16T01:36:59.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.642+0000 7f7e2e7fc640 1 -- 172.21.15.88:0/4196872381 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e24032440 con 0x7f7e3006fa00 2024-09-16T01:36:59.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.642+0000 7f7e2e7fc640 1 -- 172.21.15.88:0/4196872381 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e240327f0 con 0x7f7e3006fa00 2024-09-16T01:36:59.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.642+0000 7f7e35c8d640 1 -- 172.21.15.88:0/4196872381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e3006fa00 msgr2=0x7f7e300d02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:59.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.642+0000 7f7e35c8d640 1 --2- 172.21.15.88:0/4196872381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e3006fa00 0x7f7e300d02d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7e240097a0 tx=0x7f7e2402a230 comp rx=0 tx=0).stop 2024-09-16T01:36:59.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.642+0000 7f7e35c8d640 1 -- 172.21.15.88:0/4196872381 shutdown_connections 2024-09-16T01:36:59.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.642+0000 7f7e35c8d640 1 --2- 172.21.15.88:0/4196872381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e3006fa00 0x7f7e300d02d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.642+0000 7f7e35c8d640 1 -- 172.21.15.88:0/4196872381 >> 172.21.15.88:0/4196872381 conn(0x7f7e300d2430 msgr2=0x7f7e300d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:36:59.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.642+0000 7f7e35c8d640 1 -- 172.21.15.88:0/4196872381 shutdown_connections 2024-09-16T01:36:59.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.642+0000 7f7e35c8d640 1 -- 172.21.15.88:0/4196872381 wait complete. 2024-09-16T01:36:59.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.643+0000 7f7e35c8d640 1 Processor -- start 2024-09-16T01:36:59.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.643+0000 7f7e35c8d640 1 -- start start 2024-09-16T01:36:59.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.644+0000 7f7e35c8d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e3006fa00 0x7f7e3015d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.644+0000 7f7e35c8d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e3015a060 0x7f7e30155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.644+0000 7f7e35c8d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e3015d3d0 0x7f7e30151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.644+0000 7f7e35c8d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7e30070030 con 0x7f7e3015d3d0 2024-09-16T01:36:59.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.644+0000 7f7e35c8d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3006feb0 con 0x7f7e3006fa00 2024-09-16T01:36:59.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.644+0000 7f7e35c8d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7e300b1990 con 0x7f7e3015a060 2024-09-16T01:36:59.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.644+0000 7f7e2ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e3015d3d0 0x7f7e30151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.644+0000 7f7e2ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e3015d3d0 0x7f7e30151d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44162/0 (socket says 172.21.15.88:44162) 2024-09-16T01:36:59.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.644+0000 7f7e2ffff640 1 -- 172.21.15.88:0/722334696 learned_addr learned my addr 172.21.15.88:0/722334696 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:36:59.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.644+0000 7f7e2f7fe640 1 --2- 172.21.15.88:0/722334696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e3006fa00 0x7f7e3015d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.644+0000 7f7e2effd640 1 --2- 172.21.15.88:0/722334696 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e3015a060 0x7f7e30155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.645+0000 7f7e2ffff640 1 -- 172.21.15.88:0/722334696 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e3015a060 msgr2=0x7f7e30155670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:59.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.645+0000 7f7e2ffff640 1 --2- 172.21.15.88:0/722334696 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e3015a060 0x7f7e30155670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.645+0000 7f7e2ffff640 1 -- 172.21.15.88:0/722334696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e3006fa00 msgr2=0x7f7e3015d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:36:59.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.645+0000 7f7e2ffff640 1 --2- 172.21.15.88:0/722334696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e3006fa00 0x7f7e3015d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:36:59.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.645+0000 7f7e2ffff640 1 -- 172.21.15.88:0/722334696 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e301522b0 con 0x7f7e3015d3d0 2024-09-16T01:36:59.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.645+0000 7f7e2ffff640 1 --2- 172.21.15.88:0/722334696 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e3015d3d0 0x7f7e30151d70 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800c3c0 tx=0x7f7e2800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:59.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.645+0000 7f7e17fff640 1 -- 172.21.15.88:0/722334696 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e28017070 con 0x7f7e3015d3d0 2024-09-16T01:36:59.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.645+0000 7f7e17fff640 1 -- 172.21.15.88:0/722334696 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e280052d0 con 0x7f7e3015d3d0 2024-09-16T01:36:59.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.645+0000 7f7e35c8d640 1 -- 172.21.15.88:0/722334696 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e3015eb50 con 0x7f7e3015d3d0 2024-09-16T01:36:59.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.645+0000 7f7e17fff640 1 -- 172.21.15.88:0/722334696 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e28003810 con 0x7f7e3015d3d0 2024-09-16T01:36:59.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.645+0000 7f7e35c8d640 1 -- 172.21.15.88:0/722334696 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e3015e2b0 con 0x7f7e3015d3d0 2024-09-16T01:36:59.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.646+0000 7f7e35c8d640 1 -- 172.21.15.88:0/722334696 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e3006c6c0 con 0x7f7e3015d3d0 2024-09-16T01:36:59.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.646+0000 7f7e17fff640 1 -- 172.21.15.88:0/722334696 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7e2801a070 con 0x7f7e3015d3d0 2024-09-16T01:36:59.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.647+0000 7f7e17fff640 1 --2- 172.21.15.88:0/722334696 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7e1c06c2b0 0x7f7e1c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:36:59.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.647+0000 7f7e17fff640 1 -- 172.21.15.88:0/722334696 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5795+0+0 (secure 0 0 0) 0x7f7e2808a650 con 0x7f7e3015d3d0 2024-09-16T01:36:59.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.647+0000 7f7e2f7fe640 1 --2- 172.21.15.88:0/722334696 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7e1c06c2b0 0x7f7e1c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:36:59.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.647+0000 7f7e2f7fe640 1 --2- 172.21.15.88:0/722334696 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7e1c06c2b0 0x7f7e1c06e770 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7e20001560 tx=0x7f7e20001a90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:36:59.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.651+0000 7f7e17fff640 1 -- 172.21.15.88:0/722334696 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7e301522b0 con 0x7f7e3015d3d0 2024-09-16T01:36:59.829 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:36:59.828+0000 7fc924ccf640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:36:59.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:36:59.883+0000 7f7e35c8d640 1 -- 172.21.15.88:0/722334696 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds freeze", "role_or_gid": "4:0", "val": "true"} v 0) -- 0x7f7e3015f000 con 0x7f7e3015d3d0 2024-09-16T01:37:00.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.863+0000 7f7e17fff640 1 -- 172.21.15.88:0/722334696 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds freeze", "role_or_gid": "4:0", "val": "true"}]=0 freezing mds.6054 v45) ==== 112+0+0 (secure 0 0 0) 0x7f7e280587a0 con 0x7f7e3015d3d0 2024-09-16T01:37:00.865 INFO:teuthology.orchestra.run.smithi088.stderr:freezing mds.6054 2024-09-16T01:37:00.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.866+0000 7f7e35c8d640 1 -- 172.21.15.88:0/722334696 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7e1c06c2b0 msgr2=0x7f7e1c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:00.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.866+0000 7f7e35c8d640 1 --2- 172.21.15.88:0/722334696 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7e1c06c2b0 0x7f7e1c06e770 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7e20001560 tx=0x7f7e20001a90 comp rx=0 tx=0).stop 2024-09-16T01:37:00.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.866+0000 7f7e35c8d640 1 -- 172.21.15.88:0/722334696 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e3015d3d0 msgr2=0x7f7e30151d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:00.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.866+0000 7f7e35c8d640 1 --2- 172.21.15.88:0/722334696 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e3015d3d0 0x7f7e30151d70 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f7e2800c3c0 tx=0x7f7e2800c880 comp rx=0 tx=0).stop 2024-09-16T01:37:00.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.866+0000 7f7e35c8d640 1 -- 172.21.15.88:0/722334696 shutdown_connections 2024-09-16T01:37:00.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.866+0000 7f7e35c8d640 1 --2- 172.21.15.88:0/722334696 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7e1c06c2b0 0x7f7e1c06e770 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:00.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.866+0000 7f7e35c8d640 1 --2- 172.21.15.88:0/722334696 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e3015d3d0 0x7f7e30151d70 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:00.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.867+0000 7f7e35c8d640 1 --2- 172.21.15.88:0/722334696 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e3015a060 0x7f7e30155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:00.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.867+0000 7f7e35c8d640 1 --2- 172.21.15.88:0/722334696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e3006fa00 0x7f7e3015d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:00.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.867+0000 7f7e35c8d640 1 -- 172.21.15.88:0/722334696 >> 172.21.15.88:0/722334696 conn(0x7f7e300d2430 msgr2=0x7f7e300b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:00.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.867+0000 7f7e35c8d640 1 -- 172.21.15.88:0/722334696 shutdown_connections 2024-09-16T01:37:00.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.867+0000 7f7e35c8d640 1 -- 172.21.15.88:0/722334696 wait complete. 2024-09-16T01:37:00.881 DEBUG:teuthology.orchestra.run.smithi088:> sudo daemon-helper kill ceph --cluster ceph -w 2024-09-16T01:37:00.884 DEBUG:teuthology.orchestra.run.smithi088:> timeout 30 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --connect-timeout=60 tell mds.4:0 flush journal 2024-09-16T01:37:00.886 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok --format=json config get mds_beacon_grace 2024-09-16T01:37:00.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.991+0000 7fc92d96d640 1 Processor -- start 2024-09-16T01:37:00.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.991+0000 7fc92d96d640 1 -- start start 2024-09-16T01:37:00.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.992+0000 7fc92d96d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc92815c500 0x7fc9280b09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:00.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.992+0000 7fc92d96d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc9280b3350 0x7fc9280b3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:00.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.992+0000 7fc92d96d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9280cf1a0 con 0x7fc92815fee0 2024-09-16T01:37:00.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.992+0000 7fc92d96d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc9280b0ee0 con 0x7fc9280b3350 2024-09-16T01:37:00.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.992+0000 7fc92d96d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc92806bc10 con 0x7fc92815c500 2024-09-16T01:37:00.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.992+0000 7fc926ffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc92815c500 0x7fc9280b09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:00.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.992+0000 7fc9267fc640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc9280b3350 0x7fc9280b3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:00.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.992+0000 7fc926ffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc92815c500 0x7fc9280b09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:58612/0 (socket says 172.21.15.88:58612) 2024-09-16T01:37:00.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.992+0000 7fc926ffd640 1 -- 172.21.15.88:0/1877397152 learned_addr learned my addr 172.21.15.88:0/1877397152 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:00.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.993+0000 7fc926ffd640 1 -- 172.21.15.88:0/1877397152 >> v2:172.21.15.154:3300/0 conn(0x7fc9280b3350 msgr2=0x7fc9280b3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:00.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.993+0000 7fc926ffd640 1 --2- 172.21.15.88:0/1877397152 >> v2:172.21.15.154:3300/0 conn(0x7fc9280b3350 0x7fc9280b3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.993+0000 7fc926ffd640 1 -- 172.21.15.88:0/1877397152 >> v1:172.21.15.154:6789/0 conn(0x7fc92815fee0 legacy=0x7fc928160370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.993+0000 7fc926ffd640 1 -- 172.21.15.88:0/1877397152 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9280b2520 con 0x7fc92815c500 2024-09-16T01:37:00.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.993+0000 7fc926ffd640 1 --2- 172.21.15.88:0/1877397152 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc92815c500 0x7fc9280b09a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc918009870 tx=0x7fc91802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe922865e0efb6bf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:00.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.994+0000 7fc925ffb640 1 -- 172.21.15.88:0/1877397152 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc918037070 con 0x7fc92815c500 2024-09-16T01:37:00.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.994+0000 7fc925ffb640 1 -- 172.21.15.88:0/1877397152 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc918032440 con 0x7fc92815c500 2024-09-16T01:37:00.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.994+0000 7fc925ffb640 1 -- 172.21.15.88:0/1877397152 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc918032810 con 0x7fc92815c500 2024-09-16T01:37:00.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.994+0000 7fc92d96d640 1 -- 172.21.15.88:0/1877397152 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc92815c500 msgr2=0x7fc9280b09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:00.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.994+0000 7fc92d96d640 1 --2- 172.21.15.88:0/1877397152 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc92815c500 0x7fc9280b09a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc918009870 tx=0x7fc91802a270 comp rx=0 tx=0).stop 2024-09-16T01:37:00.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.994+0000 7fc92d96d640 1 -- 172.21.15.88:0/1877397152 shutdown_connections 2024-09-16T01:37:00.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.994+0000 7fc92d96d640 1 --2- 172.21.15.88:0/1877397152 >> v2:172.21.15.154:3300/0 conn(0x7fc9280b3350 0x7fc9280b3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:00.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.994+0000 7fc92d96d640 1 --2- 172.21.15.88:0/1877397152 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc92815c500 0x7fc9280b09a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:00.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.994+0000 7fc92d96d640 1 -- 172.21.15.88:0/1877397152 >> 172.21.15.88:0/1877397152 conn(0x7fc928062a50 msgr2=0x7fc92806e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:00.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.994+0000 7fc92d96d640 1 -- 172.21.15.88:0/1877397152 shutdown_connections 2024-09-16T01:37:00.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.994+0000 7fc92d96d640 1 -- 172.21.15.88:0/1877397152 wait complete. 2024-09-16T01:37:00.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.996+0000 7fc92d96d640 1 Processor -- start 2024-09-16T01:37:00.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.996+0000 7fc92d96d640 1 -- start start 2024-09-16T01:37:00.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.996+0000 7fc92d96d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9280b3350 0x7fc928156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:00.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.996+0000 7fc92d96d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc92815fee0 0x7fc9280b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:00.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.996+0000 7fc92d96d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc928159bf0 0x7fc9281f28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:00.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.996+0000 7fc92d96d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc9280b3cd0 con 0x7fc9280b3350 2024-09-16T01:37:01.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.997+0000 7fc92d96d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc92806bef0 con 0x7fc92815fee0 2024-09-16T01:37:01.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.997+0000 7fc92d96d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc9280b2ae0 con 0x7fc928159bf0 2024-09-16T01:37:01.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.997+0000 7fc926ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9280b3350 0x7fc928156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.997+0000 7fc926ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9280b3350 0x7fc928156ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46600/0 (socket says 172.21.15.88:46600) 2024-09-16T01:37:01.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.997+0000 7fc926ffd640 1 -- 172.21.15.88:0/3512768961 learned_addr learned my addr 172.21.15.88:0/3512768961 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:01.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.997+0000 7fc9277fe640 1 --2- 172.21.15.88:0/3512768961 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc928159bf0 0x7fc9281f28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.997+0000 7fc9267fc640 1 --2- 172.21.15.88:0/3512768961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc92815fee0 0x7fc9280b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.997+0000 7fc926ffd640 1 -- 172.21.15.88:0/3512768961 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc928159bf0 msgr2=0x7fc9281f28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:01.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.997+0000 7fc926ffd640 1 --2- 172.21.15.88:0/3512768961 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc928159bf0 0x7fc9281f28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.997+0000 7fc926ffd640 1 -- 172.21.15.88:0/3512768961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc92815fee0 msgr2=0x7fc9280b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:01.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.997+0000 7fc926ffd640 1 --2- 172.21.15.88:0/3512768961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc92815fee0 0x7fc9280b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.997+0000 7fc926ffd640 1 -- 172.21.15.88:0/3512768961 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc928157480 con 0x7fc9280b3350 2024-09-16T01:37:01.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.997+0000 7fc926ffd640 1 --2- 172.21.15.88:0/3512768961 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9280b3350 0x7fc928156ea0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fc91802a2b0 tx=0x7fc9180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:01.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.997+0000 7fc90ffff640 1 -- 172.21.15.88:0/3512768961 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc918037040 con 0x7fc9280b3350 2024-09-16T01:37:01.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.997+0000 7fc90ffff640 1 -- 172.21.15.88:0/3512768961 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc918042070 con 0x7fc9280b3350 2024-09-16T01:37:01.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.997+0000 7fc92d96d640 1 -- 172.21.15.88:0/3512768961 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc928157760 con 0x7fc9280b3350 2024-09-16T01:37:01.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.997+0000 7fc90ffff640 1 -- 172.21.15.88:0/3512768961 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc918030860 con 0x7fc9280b3350 2024-09-16T01:37:01.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.998+0000 7fc92d96d640 1 -- 172.21.15.88:0/3512768961 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc928157d20 con 0x7fc9280b3350 2024-09-16T01:37:01.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:00.999+0000 7fc90ffff640 1 -- 172.21.15.88:0/3512768961 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc918030a00 con 0x7fc9280b3350 2024-09-16T01:37:01.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.000+0000 7fc90ffff640 1 --2- 172.21.15.88:0/3512768961 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc91406c2b0 0x7fc91406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.000+0000 7fc90ffff640 1 -- 172.21.15.88:0/3512768961 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5795+0+0 (secure 0 0 0) 0x7fc9180abc30 con 0x7fc9280b3350 2024-09-16T01:37:01.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.000+0000 7fc9267fc640 1 --2- 172.21.15.88:0/3512768961 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc91406c2b0 0x7fc91406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.001+0000 7fc9267fc640 1 --2- 172.21.15.88:0/3512768961 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc91406c2b0 0x7fc91406e770 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc92806d9d0 tx=0x7fc91c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:01.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.001+0000 7fc9307e6740 1 Processor -- start 2024-09-16T01:37:01.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.001+0000 7fc9307e6740 1 -- start start 2024-09-16T01:37:01.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.002+0000 7fc9307e6740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7d1370530 0x55e7d1370910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.002+0000 7fc9307e6740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55e7d136bc30 0x55e7d136ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.002+0000 7fc9307e6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7d136dcf0 0x55e7d136b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.002+0000 7fc9307e6740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55e7d12d35e0 con 0x55e7d1370530 2024-09-16T01:37:01.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.002+0000 7fc9307e6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e7d127b360 con 0x55e7d136dcf0 2024-09-16T01:37:01.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.002+0000 7fc9307e6740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55e7d12cf5a0 con 0x55e7d136bc30 2024-09-16T01:37:01.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.002+0000 7fc926ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7d1370530 0x55e7d1370910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.002+0000 7fc926ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7d1370530 0x55e7d1370910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46606/0 (socket says 172.21.15.88:46606) 2024-09-16T01:37:01.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.002+0000 7fc926ffd640 1 -- 172.21.15.88:0/3989866604 learned_addr learned my addr 172.21.15.88:0/3989866604 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:01.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.002+0000 7fc9277fe640 1 --2- 172.21.15.88:0/3989866604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7d136dcf0 0x55e7d136b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.002+0000 7fc9267fc640 1 --2- 172.21.15.88:0/3989866604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55e7d136bc30 0x55e7d136ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.002+0000 7fc926ffd640 1 -- 172.21.15.88:0/3989866604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55e7d136bc30 msgr2=0x55e7d136ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:01.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.002+0000 7fc926ffd640 1 --2- 172.21.15.88:0/3989866604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55e7d136bc30 0x55e7d136ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.002+0000 7fc926ffd640 1 -- 172.21.15.88:0/3989866604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7d136dcf0 msgr2=0x55e7d136b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:01.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.002+0000 7fc926ffd640 1 --2- 172.21.15.88:0/3989866604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7d136dcf0 0x55e7d136b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.002+0000 7fc926ffd640 1 -- 172.21.15.88:0/3989866604 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7d13374c0 con 0x55e7d1370530 2024-09-16T01:37:01.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.002+0000 7fc9277fe640 1 --2- 172.21.15.88:0/3989866604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7d136dcf0 0x55e7d136b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:37:01.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.002+0000 7fc926ffd640 1 --2- 172.21.15.88:0/3989866604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7d1370530 0x55e7d1370910 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc9180822f0 tx=0x7fc918079710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dda4ab3cd53d2a79 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:01.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.002+0000 7fc90d7fa640 1 -- 172.21.15.88:0/3989866604 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc918090020 con 0x55e7d1370530 2024-09-16T01:37:01.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.002+0000 7fc90d7fa640 1 -- 172.21.15.88:0/3989866604 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc918080040 con 0x55e7d1370530 2024-09-16T01:37:01.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.003+0000 7fc90d7fa640 1 -- 172.21.15.88:0/3989866604 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc918082820 con 0x55e7d1370530 2024-09-16T01:37:01.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.003+0000 7fc9307e6740 1 -- 172.21.15.88:0/3989866604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7d1370530 msgr2=0x55e7d1370910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:01.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.003+0000 7fc9307e6740 1 --2- 172.21.15.88:0/3989866604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7d1370530 0x55e7d1370910 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc9180822f0 tx=0x7fc918079710 comp rx=0 tx=0).stop 2024-09-16T01:37:01.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.004+0000 7fc9307e6740 1 -- 172.21.15.88:0/3989866604 shutdown_connections 2024-09-16T01:37:01.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.004+0000 7fc9307e6740 1 --2- 172.21.15.88:0/3989866604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7d136dcf0 0x55e7d136b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.004+0000 7fc9307e6740 1 --2- 172.21.15.88:0/3989866604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55e7d136bc30 0x55e7d136ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.004+0000 7fc9307e6740 1 --2- 172.21.15.88:0/3989866604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7d1370530 0x55e7d1370910 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.004+0000 7fc9307e6740 1 -- 172.21.15.88:0/3989866604 >> 172.21.15.88:0/3989866604 conn(0x55e7d136ccf0 msgr2=0x55e7d13b53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:01.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.004+0000 7fc9307e6740 1 -- 172.21.15.88:0/3989866604 shutdown_connections 2024-09-16T01:37:01.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.004+0000 7fc9307e6740 1 -- 172.21.15.88:0/3989866604 wait complete. 2024-09-16T01:37:01.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.005+0000 7fc9307e6740 1 -- start start 2024-09-16T01:37:01.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.005+0000 7fc9307e6740 1 Processor -- start 2024-09-16T01:37:01.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.006+0000 7fc9307e6740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7d136dcf0 0x55e7d13c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.006+0000 7fc9307e6740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55e7d1370530 0x55e7d136a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.006+0000 7fc9307e6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7d136ae80 0x55e7d137f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.006+0000 7fc9307e6740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55e7d12d35e0 con 0x55e7d136dcf0 2024-09-16T01:37:01.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.006+0000 7fc9307e6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e7d125c4b0 con 0x55e7d136ae80 2024-09-16T01:37:01.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.006+0000 7fc9307e6740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55e7d139adc0 con 0x55e7d1370530 2024-09-16T01:37:01.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.006+0000 7fc9307e6740 10 client.0 _ll_register_callbacks cb 0x55e7d1258c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:37:01.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.006+0000 7fc926ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7d136dcf0 0x55e7d13c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.006+0000 7fc926ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7d136dcf0 0x55e7d13c6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46618/0 (socket says 172.21.15.88:46618) 2024-09-16T01:37:01.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.006+0000 7fc926ffd640 1 -- 172.21.15.88:0/1735600071 learned_addr learned my addr 172.21.15.88:0/1735600071 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:01.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.006+0000 7fc9267fc640 1 --2- 172.21.15.88:0/1735600071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55e7d1370530 0x55e7d136a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.006+0000 7fc9277fe640 1 --2- 172.21.15.88:0/1735600071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7d136ae80 0x55e7d137f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.006+0000 7fc926ffd640 1 -- 172.21.15.88:0/1735600071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55e7d1370530 msgr2=0x55e7d136a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:01.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.006+0000 7fc926ffd640 1 --2- 172.21.15.88:0/1735600071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55e7d1370530 0x55e7d136a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.006+0000 7fc926ffd640 1 -- 172.21.15.88:0/1735600071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7d136ae80 msgr2=0x55e7d137f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:01.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.006+0000 7fc926ffd640 1 --2- 172.21.15.88:0/1735600071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7d136ae80 0x55e7d137f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.006+0000 7fc926ffd640 1 -- 172.21.15.88:0/1735600071 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7d10b2a30 con 0x55e7d136dcf0 2024-09-16T01:37:01.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.007+0000 7fc926ffd640 1 --2- 172.21.15.88:0/1735600071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7d136dcf0 0x55e7d13c6d50 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fc918044ff0 tx=0x7fc918032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8fe394d70d58a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:01.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.007+0000 7fc9067fc640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:37:01.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.007+0000 7fc9067fc640 1 -- 172.21.15.88:0/1735600071 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc918090040 con 0x55e7d136dcf0 2024-09-16T01:37:01.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.007+0000 7fc9067fc640 1 -- 172.21.15.88:0/1735600071 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(59..59 src has 1..59) ==== 5795+0+0 (secure 0 0 0) 0x7fc918082390 con 0x55e7d136dcf0 2024-09-16T01:37:01.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.007+0000 7fc9307e6740 1 -- 172.21.15.88:0/1735600071 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc918080040 con 0x55e7d136dcf0 2024-09-16T01:37:01.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.007+0000 7fc9067fc640 1 -- 172.21.15.88:0/1735600071 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fc918098050 con 0x55e7d136dcf0 2024-09-16T01:37:01.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.007+0000 7fc9307e6740 10 client.6126 fetch_fsmap learned FSMap version 45 2024-09-16T01:37:01.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.007+0000 7fc9307e6740 1 -- 172.21.15.88:0/1735600071 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=2+}) -- 0x55e7d0fe6620 con 0x55e7d136dcf0 2024-09-16T01:37:01.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.007+0000 7fc9067fc640 1 -- 172.21.15.88:0/1735600071 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc918080040 con 0x55e7d136dcf0 2024-09-16T01:37:01.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.007+0000 7fc9067fc640 1 -- 172.21.15.88:0/1735600071 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 45) ==== 3502+0+0 (secure 0 0 0) 0x7fc91803a070 con 0x55e7d136dcf0 2024-09-16T01:37:01.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.007+0000 7fc9307e6740 10 client.6126 fetch_fsmap finished waiting for FSMap version 45 2024-09-16T01:37:01.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.007+0000 7fc9307e6740 10 client.6126 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-09-16T01:37:01.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.007+0000 7fc9307e6740 20 client.6126 populate_metadata read hostname 'smithi088' 2024-09-16T01:37:01.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.007+0000 7fc9307e6740 1 --2- 172.21.15.88:0/1735600071 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13999c0 0x55e7d1383320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.007+0000 7fc9307e6740 4 client.6126 mds_command: new command op to 6054 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:37:01.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.007+0000 7fc9307e6740 1 -- 172.21.15.88:0/1735600071 --> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7d12e0d10 con 0x55e7d13999c0 2024-09-16T01:37:01.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.007+0000 7fc9267fc640 1 --2- 172.21.15.88:0/1735600071 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13999c0 0x55e7d1383320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.008+0000 7fc9067fc640 10 client.6126 ms_handle_connect on v2:172.21.15.88:6835/4130074102 2024-09-16T01:37:01.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.008+0000 7fc9267fc640 1 --2- 172.21.15.88:0/1735600071 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13999c0 0x55e7d1383320 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64a32ca39704413c server_cookie=97aa3ed1755dc42c in_seq=0 out_seq=0 2024-09-16T01:37:01.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.008+0000 7fc9067fc640 1 -- 172.21.15.88:0/1735600071 <== mds.0 v2:172.21.15.88:6835/4130074102 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e7d12e0d10 con 0x55e7d13999c0 2024-09-16T01:37:01.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.008+0000 7fc9067fc640 10 client.6126 handle_command_reply: tid=0 2024-09-16T01:37:01.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.008+0000 7fc9307e6740 1 client.6126 shutdown 2024-09-16T01:37:01.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.009+0000 7fc9307e6740 1 -- 172.21.15.88:0/1735600071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7d136dcf0 msgr2=0x55e7d13c6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:01.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.009+0000 7fc9307e6740 1 --2- 172.21.15.88:0/1735600071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7d136dcf0 0x55e7d13c6d50 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fc918044ff0 tx=0x7fc918032f90 comp rx=0 tx=0).stop 2024-09-16T01:37:01.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.009+0000 7fc9307e6740 1 -- 172.21.15.88:0/1735600071 shutdown_connections 2024-09-16T01:37:01.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.009+0000 7fc9307e6740 1 --2- 172.21.15.88:0/1735600071 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13999c0 0x55e7d1383320 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.009+0000 7fc9307e6740 1 --2- 172.21.15.88:0/1735600071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7d136ae80 0x55e7d137f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.009+0000 7fc9307e6740 1 --2- 172.21.15.88:0/1735600071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55e7d1370530 0x55e7d136a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.009+0000 7fc9307e6740 1 --2- 172.21.15.88:0/1735600071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7d136dcf0 0x55e7d13c6d50 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.009+0000 7fc9307e6740 1 -- 172.21.15.88:0/1735600071 >> 172.21.15.88:0/1735600071 conn(0x55e7d136bc30 msgr2=0x55e7d13b4970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:01.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.009+0000 7fc9067fc640 0 client.6126 ms_handle_reset on v2:172.21.15.88:6835/4130074102 2024-09-16T01:37:01.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.009+0000 7fc9307e6740 1 -- 172.21.15.88:0/1735600071 shutdown_connections 2024-09-16T01:37:01.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.009+0000 7fc9307e6740 1 -- 172.21.15.88:0/1735600071 wait complete. 2024-09-16T01:37:01.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.009+0000 7fc9307e6740 20 client.6126 trim_cache size 0 max 16384 2024-09-16T01:37:01.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.013+0000 7f2bad24c640 1 Processor -- start 2024-09-16T01:37:01.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.013+0000 7f2bad24c640 1 -- start start 2024-09-16T01:37:01.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.014+0000 7f2bad24c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2ba80b44e0 0x7f2ba80b48c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.014+0000 7f2bad24c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2ba80d2bd0 0x7f2ba8167090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.014+0000 7f2bad24c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ba8073360 con 0x7f2ba80b7000 2024-09-16T01:37:01.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.014+0000 7f2bad24c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2ba8073640 con 0x7f2ba80d2bd0 2024-09-16T01:37:01.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.014+0000 7f2bad24c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2ba8073920 con 0x7f2ba80b44e0 2024-09-16T01:37:01.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.014+0000 7f2ba759e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2ba80d2bd0 0x7f2ba8167090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.014+0000 7f2ba759e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2ba80d2bd0 0x7f2ba8167090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46626/0 (socket says 172.21.15.88:46626) 2024-09-16T01:37:01.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.014+0000 7f2b9ffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f2ba80b7000 0x7f2ba80d24c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:39372/0 (socket says 172.21.15.88:39372) 2024-09-16T01:37:01.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.014+0000 7f2ba759e640 1 -- 172.21.15.88:0/1827083140 learned_addr learned my addr 172.21.15.88:0/1827083140 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:01.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.014+0000 7f2ba6d9d640 1 --2- 172.21.15.88:0/1827083140 >> v2:172.21.15.154:3300/0 conn(0x7f2ba80b44e0 0x7f2ba80b48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.014+0000 7f2ba759e640 1 -- 172.21.15.88:0/1827083140 >> v2:172.21.15.154:3300/0 conn(0x7f2ba80b44e0 msgr2=0x7f2ba80b48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:01.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.014+0000 7f2ba759e640 1 --2- 172.21.15.88:0/1827083140 >> v2:172.21.15.154:3300/0 conn(0x7f2ba80b44e0 0x7f2ba80b48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.014+0000 7f2ba759e640 1 -- 172.21.15.88:0/1827083140 >> v1:172.21.15.88:6789/0 conn(0x7f2ba80b7000 legacy=0x7f2ba80d24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:01.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.014+0000 7f2ba759e640 1 -- 172.21.15.88:0/1827083140 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ba8073c00 con 0x7f2ba80d2bd0 2024-09-16T01:37:01.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.014+0000 7f2ba659c640 1 -- 172.21.15.88:0/1827083140 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3113351803 0 0) 0x7f2ba8073360 con 0x7f2ba80b7000 2024-09-16T01:37:01.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.015+0000 7f2ba759e640 1 --2- 172.21.15.88:0/1827083140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ba80d2bd0 0x7f2ba8167090 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80b79a0 tx=0x7f2ba002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ace77dd770526557 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:01.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.015+0000 7f2ba659c640 1 -- 172.21.15.88:0/1827083140 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba0036070 con 0x7f2ba80d2bd0 2024-09-16T01:37:01.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.015+0000 7f2ba659c640 1 -- 172.21.15.88:0/1827083140 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ba0031600 con 0x7f2ba80d2bd0 2024-09-16T01:37:01.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.015+0000 7f2ba659c640 1 -- 172.21.15.88:0/1827083140 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba00319d0 con 0x7f2ba80d2bd0 2024-09-16T01:37:01.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.016+0000 7f2bad24c640 1 -- 172.21.15.88:0/1827083140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ba80d2bd0 msgr2=0x7f2ba8167090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:01.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.016+0000 7f2bad24c640 1 --2- 172.21.15.88:0/1827083140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ba80d2bd0 0x7f2ba8167090 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80b79a0 tx=0x7f2ba002a730 comp rx=0 tx=0).stop 2024-09-16T01:37:01.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.016+0000 7f2bad24c640 1 -- 172.21.15.88:0/1827083140 shutdown_connections 2024-09-16T01:37:01.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.016+0000 7f2bad24c640 1 --2- 172.21.15.88:0/1827083140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ba80d2bd0 0x7f2ba8167090 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.016+0000 7f2bad24c640 1 --2- 172.21.15.88:0/1827083140 >> v2:172.21.15.154:3300/0 conn(0x7f2ba80b44e0 0x7f2ba80b48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.016+0000 7f2bad24c640 1 -- 172.21.15.88:0/1827083140 >> 172.21.15.88:0/1827083140 conn(0x7f2ba80b79f0 msgr2=0x7f2ba80b7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:01.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.016+0000 7f2bad24c640 1 -- 172.21.15.88:0/1827083140 shutdown_connections 2024-09-16T01:37:01.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.016+0000 7f2bad24c640 1 -- 172.21.15.88:0/1827083140 wait complete. 2024-09-16T01:37:01.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.017+0000 7f2bad24c640 1 Processor -- start 2024-09-16T01:37:01.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.017+0000 7f2bad24c640 1 -- start start 2024-09-16T01:37:01.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.018+0000 7f2bad24c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ba80b44e0 0x7f2ba8150c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.018+0000 7f2bad24c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ba80b7000 0x7f2ba81f6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.018+0000 7f2bad24c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ba80d2bd0 0x7f2ba81f39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.018+0000 7f2bad24c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba80d01d0 con 0x7f2ba80b44e0 2024-09-16T01:37:01.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.018+0000 7f2bad24c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba80b4e00 con 0x7f2ba80b7000 2024-09-16T01:37:01.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.018+0000 7f2bad24c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2ba80b64e0 con 0x7f2ba80d2bd0 2024-09-16T01:37:01.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.018+0000 7f2ba6d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ba80b44e0 0x7f2ba8150c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.018+0000 7f2ba6d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ba80b44e0 0x7f2ba8150c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46642/0 (socket says 172.21.15.88:46642) 2024-09-16T01:37:01.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.018+0000 7f2ba6d9d640 1 -- 172.21.15.88:0/61510493 learned_addr learned my addr 172.21.15.88:0/61510493 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:01.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.018+0000 7f2b9ffff640 1 --2- 172.21.15.88:0/61510493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ba80b7000 0x7f2ba81f6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.018+0000 7f2ba759e640 1 --2- 172.21.15.88:0/61510493 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ba80d2bd0 0x7f2ba81f39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.018+0000 7f2ba6d9d640 1 -- 172.21.15.88:0/61510493 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ba80d2bd0 msgr2=0x7f2ba81f39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:01.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.019+0000 7f2ba6d9d640 1 --2- 172.21.15.88:0/61510493 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ba80d2bd0 0x7f2ba81f39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.019+0000 7f2ba6d9d640 1 -- 172.21.15.88:0/61510493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ba80b7000 msgr2=0x7f2ba81f6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:01.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.019+0000 7f2ba6d9d640 1 --2- 172.21.15.88:0/61510493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ba80b7000 0x7f2ba81f6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.019+0000 7f2ba6d9d640 1 -- 172.21.15.88:0/61510493 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ba806d3d0 con 0x7f2ba80b44e0 2024-09-16T01:37:01.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.019+0000 7f2ba6d9d640 1 --2- 172.21.15.88:0/61510493 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ba80b44e0 0x7f2ba8150c00 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f2b9800d5f0 tx=0x7f2b9800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:01.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.019+0000 7f2b9f7fe640 1 -- 172.21.15.88:0/61510493 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b980125f0 con 0x7f2ba80b44e0 2024-09-16T01:37:01.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.019+0000 7f2b9f7fe640 1 -- 172.21.15.88:0/61510493 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b98019070 con 0x7f2ba80b44e0 2024-09-16T01:37:01.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.019+0000 7f2bad24c640 1 -- 172.21.15.88:0/61510493 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ba81f7820 con 0x7f2ba80b44e0 2024-09-16T01:37:01.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.019+0000 7f2b9f7fe640 1 -- 172.21.15.88:0/61510493 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b980129d0 con 0x7f2ba80b44e0 2024-09-16T01:37:01.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.019+0000 7f2bad24c640 1 -- 172.21.15.88:0/61510493 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ba806d640 con 0x7f2ba80b44e0 2024-09-16T01:37:01.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.020+0000 7f2bad24c640 1 -- 172.21.15.88:0/61510493 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f2ba80d3080 con 0x7f2ba80b44e0 2024-09-16T01:37:01.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.021+0000 7f2b9f7fe640 1 -- 172.21.15.88:0/61510493 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2b98012b70 con 0x7f2ba80b44e0 2024-09-16T01:37:01.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.021+0000 7f2b9f7fe640 1 --2- 172.21.15.88:0/61510493 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b8806c2b0 0x7f2b8806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.021+0000 7f2b9f7fe640 1 -- 172.21.15.88:0/61510493 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5795+0+0 (secure 0 0 0) 0x7f2b9808c270 con 0x7f2ba80b44e0 2024-09-16T01:37:01.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.021+0000 7f2b9ffff640 1 --2- 172.21.15.88:0/61510493 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b8806c2b0 0x7f2b8806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.021+0000 7f2b9f7fe640 1 -- 172.21.15.88:0/61510493 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+515 (secure 0 0 0) 0x7f2b9808c720 con 0x7f2ba80b44e0 2024-09-16T01:37:01.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.022+0000 7f2bad24c640 1 -- 172.21.15.88:0/61510493 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({log-info=0+}) -- 0x7f2ba81f7510 con 0x7f2ba80b44e0 2024-09-16T01:37:01.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.022+0000 7f2b9ffff640 1 --2- 172.21.15.88:0/61510493 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b8806c2b0 0x7f2b8806e770 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2ba81f6f30 tx=0x7f2b90007b50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:01.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.022+0000 7f2b9f7fe640 1 -- 172.21.15.88:0/61510493 <== mon.0 v2:172.21.15.88:3300/0 7 ==== log(1 entries from seq 639 at 2024-09-16T01:36:59.884109+0000) ==== 308+0+0 (secure 0 0 0) 0x7f2b9808a850 con 0x7f2ba80b44e0 2024-09-16T01:37:01.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.025+0000 7fc9307e6740 1 Processor -- start 2024-09-16T01:37:01.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.025+0000 7fc9307e6740 1 -- start start 2024-09-16T01:37:01.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.026+0000 7fc9307e6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7d136ae80 0x55e7d13986f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.026+0000 7fc9307e6740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7d136dcf0 0x55e7d13999c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.026+0000 7fc9307e6740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55e7d1370530 0x55e7d1399f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.026+0000 7fc9307e6740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc9281fec30 con 0x55e7d136dcf0 2024-09-16T01:37:01.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.026+0000 7fc9307e6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc92815d970 con 0x55e7d136ae80 2024-09-16T01:37:01.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.026+0000 7fc9307e6740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc92815ccd0 con 0x55e7d1370530 2024-09-16T01:37:01.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.026+0000 7fc9267fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7d136dcf0 0x55e7d13999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.026+0000 7fc9267fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7d136dcf0 0x55e7d13999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46644/0 (socket says 172.21.15.88:46644) 2024-09-16T01:37:01.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.026+0000 7fc9267fc640 1 -- 172.21.15.88:0/630553438 learned_addr learned my addr 172.21.15.88:0/630553438 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:01.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.026+0000 7fc9277fe640 1 --2- 172.21.15.88:0/630553438 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55e7d1370530 0x55e7d1399f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.026+0000 7fc926ffd640 1 --2- 172.21.15.88:0/630553438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7d136ae80 0x55e7d13986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.026+0000 7fc9267fc640 1 -- 172.21.15.88:0/630553438 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55e7d1370530 msgr2=0x55e7d1399f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:01.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.026+0000 7fc9267fc640 1 --2- 172.21.15.88:0/630553438 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55e7d1370530 0x55e7d1399f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.026+0000 7fc9267fc640 1 -- 172.21.15.88:0/630553438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7d136ae80 msgr2=0x55e7d13986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:01.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.026+0000 7fc9267fc640 1 --2- 172.21.15.88:0/630553438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7d136ae80 0x55e7d13986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.026+0000 7fc9267fc640 1 -- 172.21.15.88:0/630553438 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7d13abb70 con 0x55e7d136dcf0 2024-09-16T01:37:01.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.026+0000 7fc9267fc640 1 --2- 172.21.15.88:0/630553438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7d136dcf0 0x55e7d13999c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fc91c0117d0 tx=0x7fc91c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89528ba1ef529b3b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:01.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.027+0000 7fc90cff9640 1 -- 172.21.15.88:0/630553438 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc91c014c00 con 0x55e7d136dcf0 2024-09-16T01:37:01.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.027+0000 7fc90cff9640 1 -- 172.21.15.88:0/630553438 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc91c031070 con 0x55e7d136dcf0 2024-09-16T01:37:01.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.027+0000 7fc90cff9640 1 -- 172.21.15.88:0/630553438 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc91c02c3b0 con 0x55e7d136dcf0 2024-09-16T01:37:01.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.027+0000 7fc9307e6740 1 -- 172.21.15.88:0/630553438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7d136dcf0 msgr2=0x55e7d13999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:01.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.027+0000 7fc9307e6740 1 --2- 172.21.15.88:0/630553438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7d136dcf0 0x55e7d13999c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fc91c0117d0 tx=0x7fc91c019000 comp rx=0 tx=0).stop 2024-09-16T01:37:01.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.027+0000 7fc9307e6740 1 -- 172.21.15.88:0/630553438 shutdown_connections 2024-09-16T01:37:01.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.027+0000 7fc9307e6740 1 --2- 172.21.15.88:0/630553438 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55e7d1370530 0x55e7d1399f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.027+0000 7fc9307e6740 1 --2- 172.21.15.88:0/630553438 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7d136dcf0 0x55e7d13999c0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.027+0000 7fc9307e6740 1 --2- 172.21.15.88:0/630553438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7d136ae80 0x55e7d13986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.027+0000 7fc9307e6740 1 -- 172.21.15.88:0/630553438 >> 172.21.15.88:0/630553438 conn(0x55e7d136bc30 msgr2=0x55e7d1381240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:01.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.028+0000 7fc9307e6740 1 -- 172.21.15.88:0/630553438 shutdown_connections 2024-09-16T01:37:01.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.028+0000 7fc9307e6740 1 -- 172.21.15.88:0/630553438 wait complete. 2024-09-16T01:37:01.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.028+0000 7fc9307e6740 1 -- start start 2024-09-16T01:37:01.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.028+0000 7fc9307e6740 1 Processor -- start 2024-09-16T01:37:01.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.029+0000 7fc9307e6740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7d136ae80 0x55e7d1433780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.029+0000 7fc9307e6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7d136dcf0 0x55e7d13b52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.029+0000 7fc9307e6740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55e7d1370530 0x55e7d142a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.029+0000 7fc9307e6740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc92815d320 con 0x55e7d136ae80 2024-09-16T01:37:01.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.029+0000 7fc9307e6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55e7d13ceeb0 con 0x55e7d136dcf0 2024-09-16T01:37:01.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.029+0000 7fc9307e6740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55e7d1433fd0 con 0x55e7d1370530 2024-09-16T01:37:01.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.029+0000 7fc9307e6740 10 client.0 _ll_register_callbacks cb 0x7fc918079830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:37:01.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.029+0000 7fc926ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7d136ae80 0x55e7d1433780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.029+0000 7fc926ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7d136ae80 0x55e7d1433780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46646/0 (socket says 172.21.15.88:46646) 2024-09-16T01:37:01.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.029+0000 7fc926ffd640 1 -- 172.21.15.88:0/2464331926 learned_addr learned my addr 172.21.15.88:0/2464331926 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:01.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.029+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7d136dcf0 0x55e7d13b52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.029+0000 7fc9277fe640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55e7d1370530 0x55e7d142a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.029+0000 7fc926ffd640 1 -- 172.21.15.88:0/2464331926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55e7d1370530 msgr2=0x55e7d142a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:01.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.029+0000 7fc926ffd640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55e7d1370530 0x55e7d142a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.029+0000 7fc926ffd640 1 -- 172.21.15.88:0/2464331926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7d136dcf0 msgr2=0x55e7d13b52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:01.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.029+0000 7fc926ffd640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7d136dcf0 0x55e7d13b52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.029+0000 7fc926ffd640 1 -- 172.21.15.88:0/2464331926 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7d0fa6320 con 0x55e7d136ae80 2024-09-16T01:37:01.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.029+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7d136dcf0 0x55e7d13b52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:37:01.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.029+0000 7fc9277fe640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55e7d1370530 0x55e7d142a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:37:01.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.029+0000 7fc926ffd640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e7d136ae80 0x55e7d1433780 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fc918076150 tx=0x7fc918075f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443dda1160790882 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:01.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.029+0000 7fc9067fc640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:37:01.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.030+0000 7fc9067fc640 1 -- 172.21.15.88:0/2464331926 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc918090070 con 0x55e7d136ae80 2024-09-16T01:37:01.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.030+0000 7fc9067fc640 1 -- 172.21.15.88:0/2464331926 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(59..59 src has 1..59) ==== 5795+0+0 (secure 0 0 0) 0x7fc9180801f0 con 0x55e7d136ae80 2024-09-16T01:37:01.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.030+0000 7fc9307e6740 1 -- 172.21.15.88:0/2464331926 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc91c031070 con 0x55e7d136ae80 2024-09-16T01:37:01.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.030+0000 7fc9067fc640 1 -- 172.21.15.88:0/2464331926 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fc9180803f0 con 0x55e7d136ae80 2024-09-16T01:37:01.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.030+0000 7fc9307e6740 10 client.6141 fetch_fsmap learned FSMap version 45 2024-09-16T01:37:01.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.030+0000 7fc9307e6740 1 -- 172.21.15.88:0/2464331926 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=2+}) -- 0x55e7d0fa6320 con 0x55e7d136ae80 2024-09-16T01:37:01.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.030+0000 7fc9067fc640 1 -- 172.21.15.88:0/2464331926 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc91c031070 con 0x55e7d136ae80 2024-09-16T01:37:01.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.030+0000 7fc9067fc640 1 -- 172.21.15.88:0/2464331926 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 45) ==== 3502+0+0 (secure 0 0 0) 0x7fc91803a040 con 0x55e7d136ae80 2024-09-16T01:37:01.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.030+0000 7fc9307e6740 10 client.6141 fetch_fsmap finished waiting for FSMap version 45 2024-09-16T01:37:01.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.030+0000 7fc9307e6740 10 client.6141 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-09-16T01:37:01.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.030+0000 7fc9307e6740 20 client.6141 populate_metadata read hostname 'smithi088' 2024-09-16T01:37:01.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.030+0000 7fc9307e6740 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.030+0000 7fc9307e6740 4 client.6141 mds_command: new command op to 6054 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:37:01.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.030+0000 7fc9307e6740 1 -- 172.21.15.88:0/2464331926 --> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e7d12db060 con 0x55e7d13b7850 2024-09-16T01:37:01.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.030+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.030+0000 7fc9067fc640 10 client.6141 ms_handle_connect on v2:172.21.15.88:6835/4130074102 2024-09-16T01:37:01.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.031+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7dd0774a0b77e8d server_cookie=2e62afdfe88729a1 in_seq=0 out_seq=0 2024-09-16T01:37:01.045 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:37:01.033+0000 7fc920cc7640 -1 mds.0.cache.den(0x1 a) newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x56123098d180] 2024-09-16T01:37:01.045 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:37:01.033+0000 7fc920cc7640 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x56123098d180] 2024-09-16T01:37:01.045 INFO:tasks.ceph.mds.a.smithi088.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-326-g8921c8ef/rpm/el9/BUILD/ceph-19.1.1-326-g8921c8ef/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7fc920cc7640 time 2024-09-16T01:37:01.034429+0000 2024-09-16T01:37:01.046 INFO:tasks.ceph.mds.a.smithi088.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-326-g8921c8ef/rpm/el9/BUILD/ceph-19.1.1-326-g8921c8ef/src/mds/MDSRank.cc: 952: ceph_abort_msg("abort() called") 2024-09-16T01:37:01.046 INFO:tasks.ceph.mds.a.smithi088.stderr: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T01:37:01.046 INFO:tasks.ceph.mds.a.smithi088.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7fc92d183417] 2024-09-16T01:37:01.046 INFO:tasks.ceph.mds.a.smithi088.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x61) [0x56122d2e72d1] 2024-09-16T01:37:01.046 INFO:tasks.ceph.mds.a.smithi088.stderr: 3: (CDentry::check_corruption(bool)+0x4d4) [0x56122d5124e4] 2024-09-16T01:37:01.046 INFO:tasks.ceph.mds.a.smithi088.stderr: 4: ceph-mds(+0x3cd2fc) [0x56122d52c2fc] 2024-09-16T01:37:01.046 INFO:tasks.ceph.mds.a.smithi088.stderr: 5: (CDir::_omap_commit(int)+0x93a) [0x56122d52dc2a] 2024-09-16T01:37:01.046 INFO:tasks.ceph.mds.a.smithi088.stderr: 6: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x265) [0x56122d52f205] 2024-09-16T01:37:01.046 INFO:tasks.ceph.mds.a.smithi088.stderr: 7: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0xaf7) [0x56122d334417] 2024-09-16T01:37:01.046 INFO:tasks.ceph.mds.a.smithi088.stderr: 8: (MDLog::try_expire(LogSegment*, int)+0x62) [0x56122d5b34d2] 2024-09-16T01:37:01.046 INFO:tasks.ceph.mds.a.smithi088.stderr: 9: (MDLog::trim_all()+0x769) [0x56122d5b6399] 2024-09-16T01:37:01.046 INFO:tasks.ceph.mds.a.smithi088.stderr: 10: ceph-mds(+0x1bfd71) [0x56122d31ed71] 2024-09-16T01:37:01.046 INFO:tasks.ceph.mds.a.smithi088.stderr: 11: ceph-mds(+0x16033d) [0x56122d2bf33d] 2024-09-16T01:37:01.047 INFO:tasks.ceph.mds.a.smithi088.stderr: 12: (MDSContext::complete(int)+0x5c) [0x56122d5ab8ec] 2024-09-16T01:37:01.047 INFO:tasks.ceph.mds.a.smithi088.stderr: 13: ceph-mds(+0x160602) [0x56122d2bf602] 2024-09-16T01:37:01.047 INFO:tasks.ceph.mds.a.smithi088.stderr: 14: (MDSContext::complete(int)+0x5c) [0x56122d5ab8ec] 2024-09-16T01:37:01.047 INFO:tasks.ceph.mds.a.smithi088.stderr: 15: (MDSIOContextBase::complete(int)+0x31c) [0x56122d5abd3c] 2024-09-16T01:37:01.047 INFO:tasks.ceph.mds.a.smithi088.stderr: 16: (Finisher::finisher_thread_entry()+0x175) [0x7fc92d241b65] 2024-09-16T01:37:01.047 INFO:tasks.ceph.mds.a.smithi088.stderr: 17: /lib64/libc.so.6(+0x89c52) [0x7fc92c889c52] 2024-09-16T01:37:01.047 INFO:tasks.ceph.mds.a.smithi088.stderr: 18: /lib64/libc.so.6(+0x10ec80) [0x7fc92c90ec80] 2024-09-16T01:37:01.047 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:37:01.034+0000 7fc920cc7640 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-326-g8921c8ef/rpm/el9/BUILD/ceph-19.1.1-326-g8921c8ef/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7fc920cc7640 time 2024-09-16T01:37:01.034429+0000 2024-09-16T01:37:01.047 INFO:tasks.ceph.mds.a.smithi088.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-326-g8921c8ef/rpm/el9/BUILD/ceph-19.1.1-326-g8921c8ef/src/mds/MDSRank.cc: 952: ceph_abort_msg("abort() called") 2024-09-16T01:37:01.047 INFO:tasks.ceph.mds.a.smithi088.stderr: 2024-09-16T01:37:01.047 INFO:tasks.ceph.mds.a.smithi088.stderr: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T01:37:01.048 INFO:tasks.ceph.mds.a.smithi088.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7fc92d183417] 2024-09-16T01:37:01.048 INFO:tasks.ceph.mds.a.smithi088.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x61) [0x56122d2e72d1] 2024-09-16T01:37:01.048 INFO:tasks.ceph.mds.a.smithi088.stderr: 3: (CDentry::check_corruption(bool)+0x4d4) [0x56122d5124e4] 2024-09-16T01:37:01.048 INFO:tasks.ceph.mds.a.smithi088.stderr: 4: ceph-mds(+0x3cd2fc) [0x56122d52c2fc] 2024-09-16T01:37:01.048 INFO:tasks.ceph.mds.a.smithi088.stderr: 5: (CDir::_omap_commit(int)+0x93a) [0x56122d52dc2a] 2024-09-16T01:37:01.048 INFO:tasks.ceph.mds.a.smithi088.stderr: 6: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x265) [0x56122d52f205] 2024-09-16T01:37:01.048 INFO:tasks.ceph.mds.a.smithi088.stderr: 7: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0xaf7) [0x56122d334417] 2024-09-16T01:37:01.048 INFO:tasks.ceph.mds.a.smithi088.stderr: 8: (MDLog::try_expire(LogSegment*, int)+0x62) [0x56122d5b34d2] 2024-09-16T01:37:01.048 INFO:tasks.ceph.mds.a.smithi088.stderr: 9: (MDLog::trim_all()+0x769) [0x56122d5b6399] 2024-09-16T01:37:01.048 INFO:tasks.ceph.mds.a.smithi088.stderr: 10: ceph-mds(+0x1bfd71) [0x56122d31ed71] 2024-09-16T01:37:01.048 INFO:tasks.ceph.mds.a.smithi088.stderr: 11: ceph-mds(+0x16033d) [0x56122d2bf33d] 2024-09-16T01:37:01.049 INFO:tasks.ceph.mds.a.smithi088.stderr: 12: (MDSContext::complete(int)+0x5c) [0x56122d5ab8ec] 2024-09-16T01:37:01.049 INFO:tasks.ceph.mds.a.smithi088.stderr: 13: ceph-mds(+0x160602) [0x56122d2bf602] 2024-09-16T01:37:01.049 INFO:tasks.ceph.mds.a.smithi088.stderr: 14: (MDSContext::complete(int)+0x5c) [0x56122d5ab8ec] 2024-09-16T01:37:01.049 INFO:tasks.ceph.mds.a.smithi088.stderr: 15: (MDSIOContextBase::complete(int)+0x31c) [0x56122d5abd3c] 2024-09-16T01:37:01.049 INFO:tasks.ceph.mds.a.smithi088.stderr: 16: (Finisher::finisher_thread_entry()+0x175) [0x7fc92d241b65] 2024-09-16T01:37:01.049 INFO:tasks.ceph.mds.a.smithi088.stderr: 17: /lib64/libc.so.6(+0x89c52) [0x7fc92c889c52] 2024-09-16T01:37:01.049 INFO:tasks.ceph.mds.a.smithi088.stderr: 18: /lib64/libc.so.6(+0x10ec80) [0x7fc92c90ec80] 2024-09-16T01:37:01.049 INFO:tasks.ceph.mds.a.smithi088.stderr: 2024-09-16T01:37:01.049 INFO:tasks.ceph.mds.a.smithi088.stderr:*** Caught signal (Aborted) ** 2024-09-16T01:37:01.049 INFO:tasks.ceph.mds.a.smithi088.stderr: in thread 7fc920cc7640 thread_name:MR_Finisher 2024-09-16T01:37:01.049 INFO:tasks.ceph.mds.a.smithi088.stderr: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T01:37:01.049 INFO:tasks.ceph.mds.a.smithi088.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc92c83e6f0] 2024-09-16T01:37:01.050 INFO:tasks.ceph.mds.a.smithi088.stderr: 2: /lib64/libc.so.6(+0x8b99c) [0x7fc92c88b99c] 2024-09-16T01:37:01.050 INFO:tasks.ceph.mds.a.smithi088.stderr: 3: raise() 2024-09-16T01:37:01.050 INFO:tasks.ceph.mds.a.smithi088.stderr: 4: abort() 2024-09-16T01:37:01.050 INFO:tasks.ceph.mds.a.smithi088.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x190) [0x7fc92d1834d4] 2024-09-16T01:37:01.050 INFO:tasks.ceph.mds.a.smithi088.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x61) [0x56122d2e72d1] 2024-09-16T01:37:01.050 INFO:tasks.ceph.mds.a.smithi088.stderr: 7: (CDentry::check_corruption(bool)+0x4d4) [0x56122d5124e4] 2024-09-16T01:37:01.050 INFO:tasks.ceph.mds.a.smithi088.stderr: 8: ceph-mds(+0x3cd2fc) [0x56122d52c2fc] 2024-09-16T01:37:01.050 INFO:tasks.ceph.mds.a.smithi088.stderr: 9: (CDir::_omap_commit(int)+0x93a) [0x56122d52dc2a] 2024-09-16T01:37:01.050 INFO:tasks.ceph.mds.a.smithi088.stderr: 10: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x265) [0x56122d52f205] 2024-09-16T01:37:01.050 INFO:tasks.ceph.mds.a.smithi088.stderr: 11: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0xaf7) [0x56122d334417] 2024-09-16T01:37:01.050 INFO:tasks.ceph.mds.a.smithi088.stderr: 12: (MDLog::try_expire(LogSegment*, int)+0x62) [0x56122d5b34d2] 2024-09-16T01:37:01.050 INFO:tasks.ceph.mds.a.smithi088.stderr: 13: (MDLog::trim_all()+0x769) [0x56122d5b6399] 2024-09-16T01:37:01.050 INFO:tasks.ceph.mds.a.smithi088.stderr: 14: ceph-mds(+0x1bfd71) [0x56122d31ed71] 2024-09-16T01:37:01.051 INFO:tasks.ceph.mds.a.smithi088.stderr: 15: ceph-mds(+0x16033d) [0x56122d2bf33d] 2024-09-16T01:37:01.051 INFO:tasks.ceph.mds.a.smithi088.stderr: 16: (MDSContext::complete(int)+0x5c) [0x56122d5ab8ec] 2024-09-16T01:37:01.051 INFO:tasks.ceph.mds.a.smithi088.stderr: 17: ceph-mds(+0x160602) [0x56122d2bf602] 2024-09-16T01:37:01.051 INFO:tasks.ceph.mds.a.smithi088.stderr: 18: (MDSContext::complete(int)+0x5c) [0x56122d5ab8ec] 2024-09-16T01:37:01.051 INFO:tasks.ceph.mds.a.smithi088.stderr: 19: (MDSIOContextBase::complete(int)+0x31c) [0x56122d5abd3c] 2024-09-16T01:37:01.051 INFO:tasks.ceph.mds.a.smithi088.stderr: 20: (Finisher::finisher_thread_entry()+0x175) [0x7fc92d241b65] 2024-09-16T01:37:01.051 INFO:tasks.ceph.mds.a.smithi088.stderr: 21: /lib64/libc.so.6(+0x89c52) [0x7fc92c889c52] 2024-09-16T01:37:01.051 INFO:tasks.ceph.mds.a.smithi088.stderr: 22: /lib64/libc.so.6(+0x10ec80) [0x7fc92c90ec80] 2024-09-16T01:37:01.051 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:37:01.035+0000 7fc920cc7640 -1 *** Caught signal (Aborted) ** 2024-09-16T01:37:01.051 INFO:tasks.ceph.mds.a.smithi088.stderr: in thread 7fc920cc7640 thread_name:MR_Finisher 2024-09-16T01:37:01.051 INFO:tasks.ceph.mds.a.smithi088.stderr: 2024-09-16T01:37:01.051 INFO:tasks.ceph.mds.a.smithi088.stderr: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T01:37:01.051 INFO:tasks.ceph.mds.a.smithi088.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc92c83e6f0] 2024-09-16T01:37:01.052 INFO:tasks.ceph.mds.a.smithi088.stderr: 2: /lib64/libc.so.6(+0x8b99c) [0x7fc92c88b99c] 2024-09-16T01:37:01.052 INFO:tasks.ceph.mds.a.smithi088.stderr: 3: raise() 2024-09-16T01:37:01.052 INFO:tasks.ceph.mds.a.smithi088.stderr: 4: abort() 2024-09-16T01:37:01.052 INFO:tasks.ceph.mds.a.smithi088.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x190) [0x7fc92d1834d4] 2024-09-16T01:37:01.052 INFO:tasks.ceph.mds.a.smithi088.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x61) [0x56122d2e72d1] 2024-09-16T01:37:01.052 INFO:tasks.ceph.mds.a.smithi088.stderr: 7: (CDentry::check_corruption(bool)+0x4d4) [0x56122d5124e4] 2024-09-16T01:37:01.052 INFO:tasks.ceph.mds.a.smithi088.stderr: 8: ceph-mds(+0x3cd2fc) [0x56122d52c2fc] 2024-09-16T01:37:01.052 INFO:tasks.ceph.mds.a.smithi088.stderr: 9: (CDir::_omap_commit(int)+0x93a) [0x56122d52dc2a] 2024-09-16T01:37:01.052 INFO:tasks.ceph.mds.a.smithi088.stderr: 10: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x265) [0x56122d52f205] 2024-09-16T01:37:01.052 INFO:tasks.ceph.mds.a.smithi088.stderr: 11: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0xaf7) [0x56122d334417] 2024-09-16T01:37:01.052 INFO:tasks.ceph.mds.a.smithi088.stderr: 12: (MDLog::try_expire(LogSegment*, int)+0x62) [0x56122d5b34d2] 2024-09-16T01:37:01.052 INFO:tasks.ceph.mds.a.smithi088.stderr: 13: (MDLog::trim_all()+0x769) [0x56122d5b6399] 2024-09-16T01:37:01.053 INFO:tasks.ceph.mds.a.smithi088.stderr: 14: ceph-mds(+0x1bfd71) [0x56122d31ed71] 2024-09-16T01:37:01.053 INFO:tasks.ceph.mds.a.smithi088.stderr: 15: ceph-mds(+0x16033d) [0x56122d2bf33d] 2024-09-16T01:37:01.053 INFO:tasks.ceph.mds.a.smithi088.stderr: 16: (MDSContext::complete(int)+0x5c) [0x56122d5ab8ec] 2024-09-16T01:37:01.053 INFO:tasks.ceph.mds.a.smithi088.stderr: 17: ceph-mds(+0x160602) [0x56122d2bf602] 2024-09-16T01:37:01.053 INFO:tasks.ceph.mds.a.smithi088.stderr: 18: (MDSContext::complete(int)+0x5c) [0x56122d5ab8ec] 2024-09-16T01:37:01.053 INFO:tasks.ceph.mds.a.smithi088.stderr: 19: (MDSIOContextBase::complete(int)+0x31c) [0x56122d5abd3c] 2024-09-16T01:37:01.053 INFO:tasks.ceph.mds.a.smithi088.stderr: 20: (Finisher::finisher_thread_entry()+0x175) [0x7fc92d241b65] 2024-09-16T01:37:01.053 INFO:tasks.ceph.mds.a.smithi088.stderr: 21: /lib64/libc.so.6(+0x89c52) [0x7fc92c889c52] 2024-09-16T01:37:01.053 INFO:tasks.ceph.mds.a.smithi088.stderr: 22: /lib64/libc.so.6(+0x10ec80) [0x7fc92c90ec80] 2024-09-16T01:37:01.053 INFO:tasks.ceph.mds.a.smithi088.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-09-16T01:37:01.053 INFO:tasks.ceph.mds.a.smithi088.stderr: 2024-09-16T01:37:01.054 INFO:teuthology.orchestra.run.smithi088.stdout:{"mds_beacon_grace":"15.000000"} 2024-09-16T01:37:01.054 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_beacon_grace": "15.000000" } 2024-09-16T01:37:01.055 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:37:01.056 INFO:tasks.ceph.mds.a.smithi088.stderr: -191> 2024-09-16T01:36:59.828+0000 7fc924ccf640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:37:01.056 INFO:tasks.ceph.mds.a.smithi088.stderr: -10> 2024-09-16T01:37:01.033+0000 7fc920cc7640 -1 mds.0.cache.den(0x1 a) newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x56123098d180] 2024-09-16T01:37:01.056 INFO:tasks.ceph.mds.a.smithi088.stderr: -8> 2024-09-16T01:37:01.033+0000 7fc920cc7640 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x56123098d180] 2024-09-16T01:37:01.056 INFO:tasks.ceph.mds.a.smithi088.stderr: -2> 2024-09-16T01:37:01.034+0000 7fc920cc7640 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-326-g8921c8ef/rpm/el9/BUILD/ceph-19.1.1-326-g8921c8ef/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7fc920cc7640 time 2024-09-16T01:37:01.034429+0000 2024-09-16T01:37:01.056 INFO:tasks.ceph.mds.a.smithi088.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-326-g8921c8ef/rpm/el9/BUILD/ceph-19.1.1-326-g8921c8ef/src/mds/MDSRank.cc: 952: ceph_abort_msg("abort() called") 2024-09-16T01:37:01.056 INFO:tasks.ceph.mds.a.smithi088.stderr: 2024-09-16T01:37:01.056 INFO:tasks.ceph.mds.a.smithi088.stderr: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T01:37:01.056 INFO:tasks.ceph.mds.a.smithi088.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7fc92d183417] 2024-09-16T01:37:01.057 INFO:tasks.ceph.mds.a.smithi088.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x61) [0x56122d2e72d1] 2024-09-16T01:37:01.057 INFO:tasks.ceph.mds.a.smithi088.stderr: 3: (CDentry::check_corruption(bool)+0x4d4) [0x56122d5124e4] 2024-09-16T01:37:01.057 INFO:tasks.ceph.mds.a.smithi088.stderr: 4: ceph-mds(+0x3cd2fc) [0x56122d52c2fc] 2024-09-16T01:37:01.057 INFO:tasks.ceph.mds.a.smithi088.stderr: 5: (CDir::_omap_commit(int)+0x93a) [0x56122d52dc2a] 2024-09-16T01:37:01.057 INFO:tasks.ceph.mds.a.smithi088.stderr: 6: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x265) [0x56122d52f205] 2024-09-16T01:37:01.057 INFO:tasks.ceph.mds.a.smithi088.stderr: 7: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0xaf7) [0x56122d334417] 2024-09-16T01:37:01.057 INFO:tasks.ceph.mds.a.smithi088.stderr: 8: (MDLog::try_expire(LogSegment*, int)+0x62) [0x56122d5b34d2] 2024-09-16T01:37:01.057 INFO:tasks.ceph.mds.a.smithi088.stderr: 9: (MDLog::trim_all()+0x769) [0x56122d5b6399] 2024-09-16T01:37:01.057 INFO:tasks.ceph.mds.a.smithi088.stderr: 10: ceph-mds(+0x1bfd71) [0x56122d31ed71] 2024-09-16T01:37:01.057 INFO:tasks.ceph.mds.a.smithi088.stderr: 11: ceph-mds(+0x16033d) [0x56122d2bf33d] 2024-09-16T01:37:01.057 INFO:tasks.ceph.mds.a.smithi088.stderr: 12: (MDSContext::complete(int)+0x5c) [0x56122d5ab8ec] 2024-09-16T01:37:01.057 INFO:tasks.ceph.mds.a.smithi088.stderr: 13: ceph-mds(+0x160602) [0x56122d2bf602] 2024-09-16T01:37:01.057 INFO:tasks.ceph.mds.a.smithi088.stderr: 14: (MDSContext::complete(int)+0x5c) [0x56122d5ab8ec] 2024-09-16T01:37:01.058 INFO:tasks.ceph.mds.a.smithi088.stderr: 15: (MDSIOContextBase::complete(int)+0x31c) [0x56122d5abd3c] 2024-09-16T01:37:01.058 INFO:tasks.ceph.mds.a.smithi088.stderr: 16: (Finisher::finisher_thread_entry()+0x175) [0x7fc92d241b65] 2024-09-16T01:37:01.058 INFO:tasks.ceph.mds.a.smithi088.stderr: 17: /lib64/libc.so.6(+0x89c52) [0x7fc92c889c52] 2024-09-16T01:37:01.058 INFO:tasks.ceph.mds.a.smithi088.stderr: 18: /lib64/libc.so.6(+0x10ec80) [0x7fc92c90ec80] 2024-09-16T01:37:01.058 INFO:tasks.ceph.mds.a.smithi088.stderr: 2024-09-16T01:37:01.058 INFO:tasks.ceph.mds.a.smithi088.stderr: 0> 2024-09-16T01:37:01.035+0000 7fc920cc7640 -1 *** Caught signal (Aborted) ** 2024-09-16T01:37:01.058 INFO:tasks.ceph.mds.a.smithi088.stderr: in thread 7fc920cc7640 thread_name:MR_Finisher 2024-09-16T01:37:01.058 INFO:tasks.ceph.mds.a.smithi088.stderr: 2024-09-16T01:37:01.059 INFO:tasks.ceph.mds.a.smithi088.stderr: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T01:37:01.059 INFO:tasks.ceph.mds.a.smithi088.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc92c83e6f0] 2024-09-16T01:37:01.059 INFO:tasks.ceph.mds.a.smithi088.stderr: 2: /lib64/libc.so.6(+0x8b99c) [0x7fc92c88b99c] 2024-09-16T01:37:01.059 INFO:tasks.ceph.mds.a.smithi088.stderr: 3: raise() 2024-09-16T01:37:01.059 INFO:tasks.ceph.mds.a.smithi088.stderr: 4: abort() 2024-09-16T01:37:01.059 INFO:tasks.ceph.mds.a.smithi088.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x190) [0x7fc92d1834d4] 2024-09-16T01:37:01.059 INFO:tasks.ceph.mds.a.smithi088.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x61) [0x56122d2e72d1] 2024-09-16T01:37:01.059 INFO:tasks.ceph.mds.a.smithi088.stderr: 7: (CDentry::check_corruption(bool)+0x4d4) [0x56122d5124e4] 2024-09-16T01:37:01.059 INFO:tasks.ceph.mds.a.smithi088.stderr: 8: ceph-mds(+0x3cd2fc) [0x56122d52c2fc] 2024-09-16T01:37:01.059 INFO:tasks.ceph.mds.a.smithi088.stderr: 9: (CDir::_omap_commit(int)+0x93a) [0x56122d52dc2a] 2024-09-16T01:37:01.059 INFO:tasks.ceph.mds.a.smithi088.stderr: 10: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x265) [0x56122d52f205] 2024-09-16T01:37:01.060 INFO:tasks.ceph.mds.a.smithi088.stderr: 11: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0xaf7) [0x56122d334417] 2024-09-16T01:37:01.060 INFO:tasks.ceph.mds.a.smithi088.stderr: 12: (MDLog::try_expire(LogSegment*, int)+0x62) [0x56122d5b34d2] 2024-09-16T01:37:01.060 INFO:tasks.ceph.mds.a.smithi088.stderr: 13: (MDLog::trim_all()+0x769) [0x56122d5b6399] 2024-09-16T01:37:01.060 INFO:tasks.ceph.mds.a.smithi088.stderr: 14: ceph-mds(+0x1bfd71) [0x56122d31ed71] 2024-09-16T01:37:01.060 INFO:tasks.ceph.mds.a.smithi088.stderr: 15: ceph-mds(+0x16033d) [0x56122d2bf33d] 2024-09-16T01:37:01.060 INFO:tasks.ceph.mds.a.smithi088.stderr: 16: (MDSContext::complete(int)+0x5c) [0x56122d5ab8ec] 2024-09-16T01:37:01.060 INFO:tasks.ceph.mds.a.smithi088.stderr: 17: ceph-mds(+0x160602) [0x56122d2bf602] 2024-09-16T01:37:01.060 INFO:tasks.ceph.mds.a.smithi088.stderr: 18: (MDSContext::complete(int)+0x5c) [0x56122d5ab8ec] 2024-09-16T01:37:01.060 INFO:tasks.ceph.mds.a.smithi088.stderr: 19: (MDSIOContextBase::complete(int)+0x31c) [0x56122d5abd3c] 2024-09-16T01:37:01.060 INFO:tasks.ceph.mds.a.smithi088.stderr: 20: (Finisher::finisher_thread_entry()+0x175) [0x7fc92d241b65] 2024-09-16T01:37:01.060 INFO:tasks.ceph.mds.a.smithi088.stderr: 21: /lib64/libc.so.6(+0x89c52) [0x7fc92c889c52] 2024-09-16T01:37:01.060 INFO:tasks.ceph.mds.a.smithi088.stderr: 22: /lib64/libc.so.6(+0x10ec80) [0x7fc92c90ec80] 2024-09-16T01:37:01.060 INFO:tasks.ceph.mds.a.smithi088.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-09-16T01:37:01.061 INFO:tasks.ceph.mds.a.smithi088.stderr: 2024-09-16T01:37:01.061 INFO:tasks.ceph.mds.a.smithi088.stderr: -191> 2024-09-16T01:36:59.828+0000 7fc924ccf640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:37:01.061 INFO:tasks.ceph.mds.a.smithi088.stderr: -10> 2024-09-16T01:37:01.033+0000 7fc920cc7640 -1 mds.0.cache.den(0x1 a) newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x56123098d180] 2024-09-16T01:37:01.061 INFO:tasks.ceph.mds.a.smithi088.stderr: -8> 2024-09-16T01:37:01.033+0000 7fc920cc7640 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x56123098d180] 2024-09-16T01:37:01.061 INFO:tasks.ceph.mds.a.smithi088.stderr: -2> 2024-09-16T01:37:01.034+0000 7fc920cc7640 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-326-g8921c8ef/rpm/el9/BUILD/ceph-19.1.1-326-g8921c8ef/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7fc920cc7640 time 2024-09-16T01:37:01.034429+0000 2024-09-16T01:37:01.061 INFO:tasks.ceph.mds.a.smithi088.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-326-g8921c8ef/rpm/el9/BUILD/ceph-19.1.1-326-g8921c8ef/src/mds/MDSRank.cc: 952: ceph_abort_msg("abort() called") 2024-09-16T01:37:01.061 INFO:tasks.ceph.mds.a.smithi088.stderr: 2024-09-16T01:37:01.061 INFO:tasks.ceph.mds.a.smithi088.stderr: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T01:37:01.061 INFO:tasks.ceph.mds.a.smithi088.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7fc92d183417] 2024-09-16T01:37:01.061 INFO:tasks.ceph.mds.a.smithi088.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x61) [0x56122d2e72d1] 2024-09-16T01:37:01.061 INFO:tasks.ceph.mds.a.smithi088.stderr: 3: (CDentry::check_corruption(bool)+0x4d4) [0x56122d5124e4] 2024-09-16T01:37:01.061 INFO:tasks.ceph.mds.a.smithi088.stderr: 4: ceph-mds(+0x3cd2fc) [0x56122d52c2fc] 2024-09-16T01:37:01.062 INFO:tasks.ceph.mds.a.smithi088.stderr: 5: (CDir::_omap_commit(int)+0x93a) [0x56122d52dc2a] 2024-09-16T01:37:01.062 INFO:tasks.ceph.mds.a.smithi088.stderr: 6: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x265) [0x56122d52f205] 2024-09-16T01:37:01.062 INFO:tasks.ceph.mds.a.smithi088.stderr: 7: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0xaf7) [0x56122d334417] 2024-09-16T01:37:01.062 INFO:tasks.ceph.mds.a.smithi088.stderr: 8: (MDLog::try_expire(LogSegment*, int)+0x62) [0x56122d5b34d2] 2024-09-16T01:37:01.062 INFO:tasks.ceph.mds.a.smithi088.stderr: 9: (MDLog::trim_all()+0x769) [0x56122d5b6399] 2024-09-16T01:37:01.062 INFO:tasks.ceph.mds.a.smithi088.stderr: 10: ceph-mds(+0x1bfd71) [0x56122d31ed71] 2024-09-16T01:37:01.062 INFO:tasks.ceph.mds.a.smithi088.stderr: 11: ceph-mds(+0x16033d) [0x56122d2bf33d] 2024-09-16T01:37:01.062 INFO:tasks.ceph.mds.a.smithi088.stderr: 12: (MDSContext::complete(int)+0x5c) [0x56122d5ab8ec] 2024-09-16T01:37:01.062 INFO:tasks.ceph.mds.a.smithi088.stderr: 13: ceph-mds(+0x160602) [0x56122d2bf602] 2024-09-16T01:37:01.062 INFO:tasks.ceph.mds.a.smithi088.stderr: 14: (MDSContext::complete(int)+0x5c) [0x56122d5ab8ec] 2024-09-16T01:37:01.062 INFO:tasks.ceph.mds.a.smithi088.stderr: 15: (MDSIOContextBase::complete(int)+0x31c) [0x56122d5abd3c] 2024-09-16T01:37:01.062 INFO:tasks.ceph.mds.a.smithi088.stderr: 16: (Finisher::finisher_thread_entry()+0x175) [0x7fc92d241b65] 2024-09-16T01:37:01.062 INFO:tasks.ceph.mds.a.smithi088.stderr: 17: /lib64/libc.so.6(+0x89c52) [0x7fc92c889c52] 2024-09-16T01:37:01.062 INFO:tasks.ceph.mds.a.smithi088.stderr: 18: /lib64/libc.so.6(+0x10ec80) [0x7fc92c90ec80] 2024-09-16T01:37:01.063 INFO:tasks.ceph.mds.a.smithi088.stderr: 2024-09-16T01:37:01.063 INFO:tasks.ceph.mds.a.smithi088.stderr: 0> 2024-09-16T01:37:01.035+0000 7fc920cc7640 -1 *** Caught signal (Aborted) ** 2024-09-16T01:37:01.063 INFO:tasks.ceph.mds.a.smithi088.stderr: in thread 7fc920cc7640 thread_name:MR_Finisher 2024-09-16T01:37:01.063 INFO:tasks.ceph.mds.a.smithi088.stderr: 2024-09-16T01:37:01.063 INFO:tasks.ceph.mds.a.smithi088.stderr: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T01:37:01.063 INFO:tasks.ceph.mds.a.smithi088.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc92c83e6f0] 2024-09-16T01:37:01.063 INFO:tasks.ceph.mds.a.smithi088.stderr: 2: /lib64/libc.so.6(+0x8b99c) [0x7fc92c88b99c] 2024-09-16T01:37:01.063 INFO:tasks.ceph.mds.a.smithi088.stderr: 3: raise() 2024-09-16T01:37:01.063 INFO:tasks.ceph.mds.a.smithi088.stderr: 4: abort() 2024-09-16T01:37:01.063 INFO:tasks.ceph.mds.a.smithi088.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x190) [0x7fc92d1834d4] 2024-09-16T01:37:01.063 INFO:tasks.ceph.mds.a.smithi088.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x61) [0x56122d2e72d1] 2024-09-16T01:37:01.063 INFO:tasks.ceph.mds.a.smithi088.stderr: 7: (CDentry::check_corruption(bool)+0x4d4) [0x56122d5124e4] 2024-09-16T01:37:01.064 INFO:tasks.ceph.mds.a.smithi088.stderr: 8: ceph-mds(+0x3cd2fc) [0x56122d52c2fc] 2024-09-16T01:37:01.064 INFO:tasks.ceph.mds.a.smithi088.stderr: 9: (CDir::_omap_commit(int)+0x93a) [0x56122d52dc2a] 2024-09-16T01:37:01.064 INFO:tasks.ceph.mds.a.smithi088.stderr: 10: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x265) [0x56122d52f205] 2024-09-16T01:37:01.064 INFO:tasks.ceph.mds.a.smithi088.stderr: 11: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0xaf7) [0x56122d334417] 2024-09-16T01:37:01.064 INFO:tasks.ceph.mds.a.smithi088.stderr: 12: (MDLog::try_expire(LogSegment*, int)+0x62) [0x56122d5b34d2] 2024-09-16T01:37:01.064 INFO:tasks.ceph.mds.a.smithi088.stderr: 13: (MDLog::trim_all()+0x769) [0x56122d5b6399] 2024-09-16T01:37:01.064 INFO:tasks.ceph.mds.a.smithi088.stderr: 14: ceph-mds(+0x1bfd71) [0x56122d31ed71] 2024-09-16T01:37:01.064 INFO:tasks.ceph.mds.a.smithi088.stderr: 15: ceph-mds(+0x16033d) [0x56122d2bf33d] 2024-09-16T01:37:01.064 INFO:tasks.ceph.mds.a.smithi088.stderr: 16: (MDSContext::complete(int)+0x5c) [0x56122d5ab8ec] 2024-09-16T01:37:01.064 INFO:tasks.ceph.mds.a.smithi088.stderr: 17: ceph-mds(+0x160602) [0x56122d2bf602] 2024-09-16T01:37:01.064 INFO:tasks.ceph.mds.a.smithi088.stderr: 18: (MDSContext::complete(int)+0x5c) [0x56122d5ab8ec] 2024-09-16T01:37:01.064 INFO:tasks.ceph.mds.a.smithi088.stderr: 19: (MDSIOContextBase::complete(int)+0x31c) [0x56122d5abd3c] 2024-09-16T01:37:01.064 INFO:tasks.ceph.mds.a.smithi088.stderr: 20: (Finisher::finisher_thread_entry()+0x175) [0x7fc92d241b65] 2024-09-16T01:37:01.065 INFO:tasks.ceph.mds.a.smithi088.stderr: 21: /lib64/libc.so.6(+0x89c52) [0x7fc92c889c52] 2024-09-16T01:37:01.065 INFO:tasks.ceph.mds.a.smithi088.stderr: 22: /lib64/libc.so.6(+0x10ec80) [0x7fc92c90ec80] 2024-09-16T01:37:01.065 INFO:tasks.ceph.mds.a.smithi088.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-09-16T01:37:01.065 INFO:tasks.ceph.mds.a.smithi088.stderr: 2024-09-16T01:37:01.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.062+0000 7fc9267fc640 1 -- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 msgr2=0x55e7d13cdc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2024-09-16T01:37:01.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.062+0000 7fc9267fc640 1 -- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 msgr2=0x55e7d13cdc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T01:37:01.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.062+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T01:37:01.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.063+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 unknown :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-09-16T01:37:01.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.063+0000 7fc9267fc640 1 -- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 msgr2=0x55e7d13cdc40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:6835/4130074102 2024-09-16T01:37:01.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.063+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 unknown :-1 s=START_CONNECT pgs=14 cs=2 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T01:37:01.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.063+0000 7fc9067fc640 1 client.6141 ms_handle_refused on v2:172.21.15.88:6835/4130074102 2024-09-16T01:37:01.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.162+0000 7f7e3b61e640 1 Processor -- start 2024-09-16T01:37:01.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.162+0000 7f7e3b61e640 1 -- start start 2024-09-16T01:37:01.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.163+0000 7f7e3b61e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e340d1c80 0x7f7e340b7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.163+0000 7f7e3b61e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e340cfc20 con 0x7f7e340cf6b0 2024-09-16T01:37:01.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.163+0000 7f7e3b61e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e340d2ed0 con 0x7f7e340b63d0 2024-09-16T01:37:01.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.163+0000 7f7e3b61e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7e340d40c0 con 0x7f7e340d1c80 2024-09-16T01:37:01.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.163+0000 7f7e38bb9640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7e340cf6b0 0x7f7e341629f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:39378/0 (socket says 172.21.15.88:39378) 2024-09-16T01:37:01.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.163+0000 7f7e38bb9640 1 -- 172.21.15.88:0/1612963320 learned_addr learned my addr 172.21.15.88:0/1612963320 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:37:01.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.163+0000 7f7e39bbb640 1 --2- 172.21.15.88:0/1612963320 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e340d1c80 0x7f7e340b7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.163+0000 7f7e2bfff640 1 -- 172.21.15.88:0/1612963320 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1768445450 0 0) 0x7f7e340cfc20 con 0x7f7e340cf6b0 2024-09-16T01:37:01.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.163+0000 7f7e2bfff640 1 -- 172.21.15.88:0/1612963320 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e20003520 con 0x7f7e340cf6b0 2024-09-16T01:37:01.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.163+0000 7f7e2bfff640 1 -- 172.21.15.88:0/1612963320 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 416775498 0 0) 0x7f7e340d2ed0 con 0x7f7e340b63d0 2024-09-16T01:37:01.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.163+0000 7f7e2bfff640 1 -- 172.21.15.88:0/1612963320 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e340cfc20 con 0x7f7e340b63d0 2024-09-16T01:37:01.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.163+0000 7f7e2bfff640 1 -- 172.21.15.88:0/1612963320 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1140927849 0 0) 0x7f7e20003520 con 0x7f7e340cf6b0 2024-09-16T01:37:01.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.164+0000 7f7e39bbb640 1 -- 172.21.15.88:0/1612963320 >> v1:172.21.15.154:6789/0 conn(0x7f7e340b63d0 legacy=0x7f7e340d39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:01.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.164+0000 7f7e39bbb640 1 -- 172.21.15.88:0/1612963320 >> v1:172.21.15.88:6789/0 conn(0x7f7e340cf6b0 legacy=0x7f7e341629f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:01.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.164+0000 7f7e39bbb640 1 -- 172.21.15.88:0/1612963320 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e340d31b0 con 0x7f7e340d1c80 2024-09-16T01:37:01.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.164+0000 7f7e39bbb640 1 --2- 172.21.15.88:0/1612963320 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e340d1c80 0x7f7e340b7880 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c009dc0 tx=0x7f7e2c02a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5866a80715a9884c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:01.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.164+0000 7f7e2bfff640 1 -- 172.21.15.88:0/1612963320 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e24001670 con 0x7f7e340d1c80 2024-09-16T01:37:01.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.165+0000 7f7e2bfff640 1 -- 172.21.15.88:0/1612963320 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e2c007540 con 0x7f7e340d1c80 2024-09-16T01:37:01.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.165+0000 7f7e2bfff640 1 -- 172.21.15.88:0/1612963320 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c007910 con 0x7f7e340d1c80 2024-09-16T01:37:01.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.165+0000 7f7e3b61e640 1 -- 172.21.15.88:0/1612963320 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e340d1c80 msgr2=0x7f7e340b7880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:01.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.165+0000 7f7e3b61e640 1 --2- 172.21.15.88:0/1612963320 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e340d1c80 0x7f7e340b7880 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c009dc0 tx=0x7f7e2c02a5d0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.165+0000 7f7e3b61e640 1 -- 172.21.15.88:0/1612963320 shutdown_connections 2024-09-16T01:37:01.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.165+0000 7f7e3b61e640 1 --2- 172.21.15.88:0/1612963320 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e340d1c80 0x7f7e340b7880 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.165+0000 7f7e3b61e640 1 -- 172.21.15.88:0/1612963320 >> 172.21.15.88:0/1612963320 conn(0x7f7e340d1160 msgr2=0x7f7e340d1570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.165+0000 7f7e3b61e640 1 -- 172.21.15.88:0/1612963320 shutdown_connections 2024-09-16T01:37:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.165+0000 7f7e3b61e640 1 -- 172.21.15.88:0/1612963320 wait complete. 2024-09-16T01:37:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.166+0000 7f7e3b61e640 1 Processor -- start 2024-09-16T01:37:01.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.166+0000 7f7e3b61e640 1 -- start start 2024-09-16T01:37:01.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.167+0000 7f7e3b61e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e340b63d0 0x7f7e34157e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.167+0000 7f7e3b61e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e340cf6b0 0x7f7e34159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.167+0000 7f7e3b61e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e340d1c80 0x7f7e3415a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.167+0000 7f7e3b61e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7e340cb440 con 0x7f7e340d1c80 2024-09-16T01:37:01.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.167+0000 7f7e3b61e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7e34163100 con 0x7f7e340cf6b0 2024-09-16T01:37:01.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.167+0000 7f7e3b61e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7e340cb5c0 con 0x7f7e340b63d0 2024-09-16T01:37:01.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.167+0000 7f7e39bbb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e340d1c80 0x7f7e3415a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.167+0000 7f7e39bbb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e340d1c80 0x7f7e3415a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46648/0 (socket says 172.21.15.88:46648) 2024-09-16T01:37:01.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.167+0000 7f7e39bbb640 1 -- 172.21.15.88:0/685785753 learned_addr learned my addr 172.21.15.88:0/685785753 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:01.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.167+0000 7f7e38bb9640 1 --2- 172.21.15.88:0/685785753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e340cf6b0 0x7f7e34159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.167+0000 7f7e393ba640 1 --2- 172.21.15.88:0/685785753 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e340b63d0 0x7f7e34157e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.167+0000 7f7e39bbb640 1 -- 172.21.15.88:0/685785753 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e340b63d0 msgr2=0x7f7e34157e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.167+0000 7f7e39bbb640 1 --2- 172.21.15.88:0/685785753 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e340b63d0 0x7f7e34157e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.167+0000 7f7e39bbb640 1 -- 172.21.15.88:0/685785753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e340cf6b0 msgr2=0x7f7e34159ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.167+0000 7f7e39bbb640 1 --2- 172.21.15.88:0/685785753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e340cf6b0 0x7f7e34159ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.167+0000 7f7e39bbb640 1 -- 172.21.15.88:0/685785753 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e341583a0 con 0x7f7e340d1c80 2024-09-16T01:37:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.167+0000 7f7e39bbb640 1 --2- 172.21.15.88:0/685785753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e340d1c80 0x7f7e3415a000 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c009ee0 tx=0x7f7e2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.168+0000 7f7e29ffb640 1 -- 172.21.15.88:0/685785753 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c03f070 con 0x7f7e340d1c80 2024-09-16T01:37:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.168+0000 7f7e29ffb640 1 -- 172.21.15.88:0/685785753 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e2c031bc0 con 0x7f7e340d1c80 2024-09-16T01:37:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.168+0000 7f7e3b61e640 1 -- 172.21.15.88:0/685785753 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e34158960 con 0x7f7e340d1c80 2024-09-16T01:37:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.168+0000 7f7e29ffb640 1 -- 172.21.15.88:0/685785753 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c009450 con 0x7f7e340d1c80 2024-09-16T01:37:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.168+0000 7f7e3b61e640 1 -- 172.21.15.88:0/685785753 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e34158f20 con 0x7f7e340d1c80 2024-09-16T01:37:01.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.169+0000 7f7e3b61e640 1 -- 172.21.15.88:0/685785753 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e340b67b0 con 0x7f7e340d1c80 2024-09-16T01:37:01.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.169+0000 7f7e29ffb640 1 -- 172.21.15.88:0/685785753 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7e2c043410 con 0x7f7e340d1c80 2024-09-16T01:37:01.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.170+0000 7f7e29ffb640 1 --2- 172.21.15.88:0/685785753 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7e1806c2b0 0x7f7e1806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:01.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.170+0000 7f7e29ffb640 1 -- 172.21.15.88:0/685785753 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5795+0+0 (secure 0 0 0) 0x7f7e2c0ac880 con 0x7f7e340d1c80 2024-09-16T01:37:01.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.170+0000 7f7e393ba640 1 --2- 172.21.15.88:0/685785753 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7e1806c2b0 0x7f7e1806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:01.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.170+0000 7f7e393ba640 1 --2- 172.21.15.88:0/685785753 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7e1806c2b0 0x7f7e1806e770 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7e300011e0 tx=0x7f7e30009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:01.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.174+0000 7f7e29ffb640 1 -- 172.21.15.88:0/685785753 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7e2c07af00 con 0x7f7e340d1c80 2024-09-16T01:37:01.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.264+0000 7fc9267fc640 1 -- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 msgr2=0x55e7d13cdc40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:6835/4130074102 2024-09-16T01:37:01.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.264+0000 7fc9067fc640 1 client.6141 ms_handle_refused on v2:172.21.15.88:6835/4130074102 2024-09-16T01:37:01.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.264+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 unknown :-1 s=START_CONNECT pgs=14 cs=3 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T01:37:01.313 INFO:tasks.ceph.mds.a.smithi088.stderr:daemon-helper: command crashed with signal 6 2024-09-16T01:37:01.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.377+0000 7f7e3b61e640 1 -- 172.21.15.88:0/685785753 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e340cf280 con 0x7f7e340d1c80 2024-09-16T01:37:01.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.378+0000 7f7e29ffb640 1 -- 172.21.15.88:0/685785753 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5224 (secure 0 0 0) 0x7f7e2c07a800 con 0x7f7e340d1c80 2024-09-16T01:37:01.379 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:37:01.379 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":45,"btime":"2024-09-16T01:37:00:860056+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/581241194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":581241194},{"type":"v1","addr":"172.21.15.154:6834","nonce":581241194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2104508521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2104508521},{"type":"v1","addr":"172.21.15.154:6835","nonce":2104508521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5859,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/998283126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":998283126},{"type":"v1","addr":"172.21.15.88:6836","nonce":998283126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:39.786438+0000","modified":"2024-09-16T01:36:59.884181+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":5,"addr":"172.21.15.88:6837/4130074102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4130074102},{"type":"v1","addr":"172.21.15.88:6837","nonce":4130074102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":4}]} 2024-09-16T01:37:01.380 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 45 2024-09-16T01:37:01.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.380+0000 7f7e3b61e640 1 -- 172.21.15.88:0/685785753 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7e1806c2b0 msgr2=0x7f7e1806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:01.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.380+0000 7f7e3b61e640 1 --2- 172.21.15.88:0/685785753 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7e1806c2b0 0x7f7e1806e770 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7e300011e0 tx=0x7f7e30009040 comp rx=0 tx=0).stop 2024-09-16T01:37:01.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.380+0000 7f7e3b61e640 1 -- 172.21.15.88:0/685785753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e340d1c80 msgr2=0x7f7e3415a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:01.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.380+0000 7f7e3b61e640 1 --2- 172.21.15.88:0/685785753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e340d1c80 0x7f7e3415a000 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c009ee0 tx=0x7f7e2c004a40 comp rx=0 tx=0).stop 2024-09-16T01:37:01.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.380+0000 7f7e3b61e640 1 -- 172.21.15.88:0/685785753 shutdown_connections 2024-09-16T01:37:01.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.380+0000 7f7e3b61e640 1 --2- 172.21.15.88:0/685785753 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7e1806c2b0 0x7f7e1806e770 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.380+0000 7f7e3b61e640 1 --2- 172.21.15.88:0/685785753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e340d1c80 0x7f7e3415a000 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.380+0000 7f7e3b61e640 1 --2- 172.21.15.88:0/685785753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e340cf6b0 0x7f7e34159ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.380+0000 7f7e3b61e640 1 --2- 172.21.15.88:0/685785753 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e340b63d0 0x7f7e34157e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:01.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.380+0000 7f7e3b61e640 1 -- 172.21.15.88:0/685785753 >> 172.21.15.88:0/685785753 conn(0x7f7e340d1160 msgr2=0x7f7e340b9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:01.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.380+0000 7f7e3b61e640 1 -- 172.21.15.88:0/685785753 shutdown_connections 2024-09-16T01:37:01.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.380+0000 7f7e3b61e640 1 -- 172.21.15.88:0/685785753 wait complete. 2024-09-16T01:37:01.392 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-09-16T01:37:01.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.665+0000 7fc9267fc640 1 -- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 msgr2=0x55e7d13cdc40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:6835/4130074102 2024-09-16T01:37:01.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.665+0000 7fc9067fc640 1 client.6141 ms_handle_refused on v2:172.21.15.88:6835/4130074102 2024-09-16T01:37:01.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.665+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 unknown :-1 s=START_CONNECT pgs=14 cs=4 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T01:37:01.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:01.866+0000 7f2b9f7fe640 1 -- 172.21.15.88:0/61510493 <== mon.0 v2:172.21.15.88:3300/0 8 ==== log(2 entries from seq 640 at 2024-09-16T01:37:00.864614+0000) ==== 634+0+0 (secure 0 0 0) 0x7f2b98012e60 con 0x7f2ba80b44e0 2024-09-16T01:37:01.867 INFO:teuthology.orchestra.run.smithi088.stdout: cluster: 2024-09-16T01:37:01.867 INFO:teuthology.orchestra.run.smithi088.stdout: id: 5ace2c91-46f9-482b-8a89-731fbdc171bf 2024-09-16T01:37:01.867 INFO:teuthology.orchestra.run.smithi088.stdout: health: HEALTH_OK 2024-09-16T01:37:01.868 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:37:01.868 INFO:teuthology.orchestra.run.smithi088.stdout: services: 2024-09-16T01:37:01.868 INFO:teuthology.orchestra.run.smithi088.stdout: mon: 3 daemons, quorum a,b,c (age 3m) 2024-09-16T01:37:01.868 INFO:teuthology.orchestra.run.smithi088.stdout: mgr: y(active, since 2m), standbys: x 2024-09-16T01:37:01.868 INFO:teuthology.orchestra.run.smithi088.stdout: mds: 1/1 daemons up, 3 standby 2024-09-16T01:37:01.868 INFO:teuthology.orchestra.run.smithi088.stdout: osd: 8 osds: 8 up (since 2m), 8 in (since 3m) 2024-09-16T01:37:01.868 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:37:01.868 INFO:teuthology.orchestra.run.smithi088.stdout: data: 2024-09-16T01:37:01.868 INFO:teuthology.orchestra.run.smithi088.stdout: volumes: 1/1 healthy 2024-09-16T01:37:01.868 INFO:teuthology.orchestra.run.smithi088.stdout: pools: 3 pools, 97 pgs 2024-09-16T01:37:01.868 INFO:teuthology.orchestra.run.smithi088.stdout: objects: 29 objects, 602 KiB 2024-09-16T01:37:01.868 INFO:teuthology.orchestra.run.smithi088.stdout: usage: 222 MiB used, 720 GiB / 720 GiB avail 2024-09-16T01:37:01.869 INFO:teuthology.orchestra.run.smithi088.stdout: pgs: 97 active+clean 2024-09-16T01:37:01.869 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:37:01.869 INFO:teuthology.orchestra.run.smithi088.stdout: io: 2024-09-16T01:37:01.869 INFO:teuthology.orchestra.run.smithi088.stdout: client: 1.9 KiB/s rd, 3.6 KiB/s wr, 1 op/s rd, 3 op/s wr 2024-09-16T01:37:01.869 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:37:01.869 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:37:01.869 INFO:teuthology.orchestra.run.smithi088.stdout:2024-09-16T01:37:01.034356+0000 mds.a [ERR] MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x56123098d180] 2024-09-16T01:37:02.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:02.466+0000 7fc9267fc640 1 -- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 msgr2=0x55e7d13cdc40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:6835/4130074102 2024-09-16T01:37:02.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:02.466+0000 7fc9067fc640 1 client.6141 ms_handle_refused on v2:172.21.15.88:6835/4130074102 2024-09-16T01:37:02.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:02.466+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 unknown :-1 s=START_CONNECT pgs=14 cs=5 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T01:37:04.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:04.068+0000 7fc9267fc640 1 -- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 msgr2=0x55e7d13cdc40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:6835/4130074102 2024-09-16T01:37:04.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:04.068+0000 7fc9067fc640 1 client.6141 ms_handle_refused on v2:172.21.15.88:6835/4130074102 2024-09-16T01:37:04.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:04.068+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 unknown :-1 s=START_CONNECT pgs=14 cs=6 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-09-16T01:37:05.797 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-09-16T01:37:06.393 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:37:06.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.504+0000 7f5f886a5640 1 Processor -- start 2024-09-16T01:37:06.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.504+0000 7f5f886a5640 1 -- start start 2024-09-16T01:37:06.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.505+0000 7f5f886a5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5f8015e660 0x7f5f800cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:06.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.505+0000 7f5f886a5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f800d1820 0x7f5f800b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:06.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.505+0000 7f5f886a5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f8015ea40 con 0x7f5f800d3310 2024-09-16T01:37:06.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.505+0000 7f5f886a5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5f800d3870 con 0x7f5f8015e660 2024-09-16T01:37:06.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.505+0000 7f5f886a5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5f800d3b50 con 0x7f5f800d1820 2024-09-16T01:37:06.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.505+0000 7f5f85c40640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5f800d3310 0x7f5f800d1110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:39386/0 (socket says 172.21.15.88:39386) 2024-09-16T01:37:06.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.505+0000 7f5f85c40640 1 -- 172.21.15.88:0/3414515334 learned_addr learned my addr 172.21.15.88:0/3414515334 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:37:06.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.505+0000 7f5f8543f640 1 -- 172.21.15.88:0/3414515334 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1873778382 0 0) 0x7f5f8015ea40 con 0x7f5f800d3310 2024-09-16T01:37:06.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.505+0000 7f5f86c42640 1 --2- 172.21.15.88:0/3414515334 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f800d1820 0x7f5f800b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:06.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.505+0000 7f5f8543f640 1 -- 172.21.15.88:0/3414515334 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f68003520 con 0x7f5f800d3310 2024-09-16T01:37:06.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.505+0000 7f5f86441640 1 --2- 172.21.15.88:0/3414515334 >> v2:172.21.15.154:3300/0 conn(0x7f5f8015e660 0x7f5f800cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:06.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.505+0000 7f5f8543f640 1 -- 172.21.15.88:0/3414515334 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3797522641 0 0) 0x7f5f68003520 con 0x7f5f800d3310 2024-09-16T01:37:06.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.505+0000 7f5f8543f640 1 -- 172.21.15.88:0/3414515334 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f800d1820 msgr2=0x7f5f800b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:06.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.505+0000 7f5f8543f640 1 --2- 172.21.15.88:0/3414515334 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f800d1820 0x7f5f800b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:06.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.505+0000 7f5f8543f640 1 -- 172.21.15.88:0/3414515334 >> v2:172.21.15.154:3300/0 conn(0x7f5f8015e660 msgr2=0x7f5f800cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:06.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.505+0000 7f5f8543f640 1 --2- 172.21.15.88:0/3414515334 >> v2:172.21.15.154:3300/0 conn(0x7f5f8015e660 0x7f5f800cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:06.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.505+0000 7f5f8543f640 1 -- 172.21.15.88:0/3414515334 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f800d3e30 con 0x7f5f800d3310 2024-09-16T01:37:06.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.505+0000 7f5f8543f640 1 -- 172.21.15.88:0/3414515334 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f5f74001670 con 0x7f5f800d3310 2024-09-16T01:37:06.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.505+0000 7f5f8543f640 1 -- 172.21.15.88:0/3414515334 >> v1:172.21.15.88:6789/0 conn(0x7f5f800d3310 legacy=0x7f5f800d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:06.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.505+0000 7f5f8543f640 1 --2- 172.21.15.88:0/3414515334 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f800d1820 0x7f5f680039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:06.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.505+0000 7f5f8543f640 1 --2- 172.21.15.88:0/3414515334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f68003f80 0x7f5f68004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:06.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.506+0000 7f5f8543f640 1 --2- 172.21.15.88:0/3414515334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f68004d40 0x7f5f680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:06.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.506+0000 7f5f8543f640 1 -- 172.21.15.88:0/3414515334 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5f800d3870 con 0x7f5f68004d40 2024-09-16T01:37:06.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.506+0000 7f5f8543f640 1 -- 172.21.15.88:0/3414515334 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5f800d3b50 con 0x7f5f68003f80 2024-09-16T01:37:06.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.506+0000 7f5f8543f640 1 -- 172.21.15.88:0/3414515334 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5f6800b9c0 con 0x7f5f800d1820 2024-09-16T01:37:06.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.506+0000 7f5f86c42640 1 --2- 172.21.15.88:0/3414515334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f68004d40 0x7f5f680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:06.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.506+0000 7f5f85c40640 1 --2- 172.21.15.88:0/3414515334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f68003f80 0x7f5f68004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:06.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.506+0000 7f5f86441640 1 --2- 172.21.15.88:0/3414515334 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f800d1820 0x7f5f680039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:06.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.506+0000 7f5f86c42640 1 -- 172.21.15.88:0/3414515334 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f800d1820 msgr2=0x7f5f680039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:06.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.506+0000 7f5f86c42640 1 --2- 172.21.15.88:0/3414515334 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f800d1820 0x7f5f680039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:06.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.506+0000 7f5f86c42640 1 -- 172.21.15.88:0/3414515334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f68003f80 msgr2=0x7f5f68004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:06.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.506+0000 7f5f86c42640 1 --2- 172.21.15.88:0/3414515334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f68003f80 0x7f5f68004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:06.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.506+0000 7f5f86c42640 1 -- 172.21.15.88:0/3414515334 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f68003520 con 0x7f5f68004d40 2024-09-16T01:37:06.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.507+0000 7f5f86c42640 1 --2- 172.21.15.88:0/3414515334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f68004d40 0x7f5f680051c0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f5f7800e170 tx=0x7f5f68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4901fbf11eb4453 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:06.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.507+0000 7f5f8543f640 1 -- 172.21.15.88:0/3414515334 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f78034020 con 0x7f5f68004d40 2024-09-16T01:37:06.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.507+0000 7f5f8543f640 1 -- 172.21.15.88:0/3414515334 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f780305f0 con 0x7f5f68004d40 2024-09-16T01:37:06.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.507+0000 7f5f8543f640 1 -- 172.21.15.88:0/3414515334 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f78009c10 con 0x7f5f68004d40 2024-09-16T01:37:06.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.507+0000 7f5f886a5640 1 -- 172.21.15.88:0/3414515334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f68004d40 msgr2=0x7f5f680051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:06.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.507+0000 7f5f886a5640 1 --2- 172.21.15.88:0/3414515334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f68004d40 0x7f5f680051c0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f5f7800e170 tx=0x7f5f68005830 comp rx=0 tx=0).stop 2024-09-16T01:37:06.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.507+0000 7f5f86c42640 1 -- 172.21.15.88:0/3414515334 reap_dead start 2024-09-16T01:37:06.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.507+0000 7f5f886a5640 1 -- 172.21.15.88:0/3414515334 shutdown_connections 2024-09-16T01:37:06.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.507+0000 7f5f886a5640 1 -- 172.21.15.88:0/3414515334 >> 172.21.15.88:0/3414515334 conn(0x7f5f800b6410 msgr2=0x7f5f800b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:06.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.507+0000 7f5f886a5640 1 -- 172.21.15.88:0/3414515334 shutdown_connections 2024-09-16T01:37:06.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.507+0000 7f5f886a5640 1 -- 172.21.15.88:0/3414515334 wait complete. 2024-09-16T01:37:06.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.509+0000 7f5f886a5640 1 Processor -- start 2024-09-16T01:37:06.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.509+0000 7f5f886a5640 1 -- start start 2024-09-16T01:37:06.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.509+0000 7f5f886a5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f68004d40 0x7f5f801f5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:06.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.509+0000 7f5f886a5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f80158400 0x7f5f80158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:06.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.509+0000 7f5f886a5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f80160b30 0x7f5f80160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:06.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.509+0000 7f5f886a5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5f800cfc70 con 0x7f5f68004d40 2024-09-16T01:37:06.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.509+0000 7f5f886a5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5f800d43f0 con 0x7f5f80160b30 2024-09-16T01:37:06.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.509+0000 7f5f886a5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5f800b7b70 con 0x7f5f80158400 2024-09-16T01:37:06.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.509+0000 7f5f86441640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f68004d40 0x7f5f801f5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:06.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.509+0000 7f5f86441640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f68004d40 0x7f5f801f5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46676/0 (socket says 172.21.15.88:46676) 2024-09-16T01:37:06.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.509+0000 7f5f86441640 1 -- 172.21.15.88:0/3447058552 learned_addr learned my addr 172.21.15.88:0/3447058552 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:06.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.510+0000 7f5f86441640 1 -- 172.21.15.88:0/3447058552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f80158400 msgr2=0x7f5f80158880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:06.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.510+0000 7f5f86441640 1 --2- 172.21.15.88:0/3447058552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f80158400 0x7f5f80158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:06.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.510+0000 7f5f86441640 1 -- 172.21.15.88:0/3447058552 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f80160b30 msgr2=0x7f5f80160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:06.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.510+0000 7f5f86441640 1 --2- 172.21.15.88:0/3447058552 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f80160b30 0x7f5f80160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:06.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.510+0000 7f5f86441640 1 -- 172.21.15.88:0/3447058552 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f80158fe0 con 0x7f5f68004d40 2024-09-16T01:37:06.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.510+0000 7f5f86441640 1 --2- 172.21.15.88:0/3447058552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f68004d40 0x7f5f801f5100 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f5f70011ad0 tx=0x7f5f70011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:06.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.510+0000 7f5f7effd640 1 -- 172.21.15.88:0/3447058552 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7000fad0 con 0x7f5f68004d40 2024-09-16T01:37:06.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.510+0000 7f5f7effd640 1 -- 172.21.15.88:0/3447058552 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f7000fc70 con 0x7f5f68004d40 2024-09-16T01:37:06.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.510+0000 7f5f7effd640 1 -- 172.21.15.88:0/3447058552 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f70014440 con 0x7f5f68004d40 2024-09-16T01:37:06.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.510+0000 7f5f886a5640 1 -- 172.21.15.88:0/3447058552 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f8015fd70 con 0x7f5f68004d40 2024-09-16T01:37:06.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.510+0000 7f5f886a5640 1 -- 172.21.15.88:0/3447058552 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f80160330 con 0x7f5f68004d40 2024-09-16T01:37:06.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.511+0000 7f5f886a5640 1 -- 172.21.15.88:0/3447058552 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f800cf730 con 0x7f5f68004d40 2024-09-16T01:37:06.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.511+0000 7f5f7effd640 1 -- 172.21.15.88:0/3447058552 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5f70012040 con 0x7f5f68004d40 2024-09-16T01:37:06.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.512+0000 7f5f7effd640 1 --2- 172.21.15.88:0/3447058552 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5f6c06c2b0 0x7f5f6c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:06.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.512+0000 7f5f7effd640 1 -- 172.21.15.88:0/3447058552 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5795+0+0 (secure 0 0 0) 0x7f5f7008baf0 con 0x7f5f68004d40 2024-09-16T01:37:06.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.512+0000 7f5f85c40640 1 --2- 172.21.15.88:0/3447058552 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5f6c06c2b0 0x7f5f6c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:06.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.512+0000 7f5f85c40640 1 --2- 172.21.15.88:0/3447058552 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5f6c06c2b0 0x7f5f6c06e770 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5f74000fb0 tx=0x7f5f74000bc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:06.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.516+0000 7f5f7effd640 1 -- 172.21.15.88:0/3447058552 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5f70019030 con 0x7f5f68004d40 2024-09-16T01:37:06.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.716+0000 7f5f886a5640 1 -- 172.21.15.88:0/3447058552 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f80159450 con 0x7f5f68004d40 2024-09-16T01:37:06.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.717+0000 7f5f7effd640 1 -- 172.21.15.88:0/3447058552 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5224 (secure 0 0 0) 0x7f5f7005a200 con 0x7f5f68004d40 2024-09-16T01:37:06.719 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:37:06.719 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":45,"btime":"2024-09-16T01:37:00:860056+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/581241194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":581241194},{"type":"v1","addr":"172.21.15.154:6834","nonce":581241194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2104508521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2104508521},{"type":"v1","addr":"172.21.15.154:6835","nonce":2104508521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5859,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/998283126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":998283126},{"type":"v1","addr":"172.21.15.88:6836","nonce":998283126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:39.786438+0000","modified":"2024-09-16T01:36:59.884181+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":5,"addr":"172.21.15.88:6837/4130074102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4130074102},{"type":"v1","addr":"172.21.15.88:6837","nonce":4130074102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":4}]} 2024-09-16T01:37:06.720 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 45 2024-09-16T01:37:06.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.719+0000 7f5f886a5640 1 -- 172.21.15.88:0/3447058552 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5f6c06c2b0 msgr2=0x7f5f6c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:06.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.719+0000 7f5f886a5640 1 --2- 172.21.15.88:0/3447058552 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5f6c06c2b0 0x7f5f6c06e770 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5f74000fb0 tx=0x7f5f74000bc0 comp rx=0 tx=0).stop 2024-09-16T01:37:06.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.719+0000 7f5f886a5640 1 -- 172.21.15.88:0/3447058552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f68004d40 msgr2=0x7f5f801f5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:06.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.719+0000 7f5f886a5640 1 --2- 172.21.15.88:0/3447058552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f68004d40 0x7f5f801f5100 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f5f70011ad0 tx=0x7f5f70011f90 comp rx=0 tx=0).stop 2024-09-16T01:37:06.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.719+0000 7f5f886a5640 1 -- 172.21.15.88:0/3447058552 shutdown_connections 2024-09-16T01:37:06.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.719+0000 7f5f886a5640 1 --2- 172.21.15.88:0/3447058552 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5f6c06c2b0 0x7f5f6c06e770 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:06.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.719+0000 7f5f886a5640 1 --2- 172.21.15.88:0/3447058552 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f80160b30 0x7f5f80160fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:06.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.719+0000 7f5f886a5640 1 --2- 172.21.15.88:0/3447058552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f80158400 0x7f5f80158880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:06.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.719+0000 7f5f886a5640 1 --2- 172.21.15.88:0/3447058552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5f68004d40 0x7f5f801f5100 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:06.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.719+0000 7f5f886a5640 1 -- 172.21.15.88:0/3447058552 >> 172.21.15.88:0/3447058552 conn(0x7f5f800b6410 msgr2=0x7f5f800ba060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:06.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.719+0000 7f5f886a5640 1 -- 172.21.15.88:0/3447058552 shutdown_connections 2024-09-16T01:37:06.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:06.719+0000 7f5f886a5640 1 -- 172.21.15.88:0/3447058552 wait complete. 2024-09-16T01:37:06.732 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-09-16T01:37:07.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:07.270+0000 7fc9267fc640 1 -- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 msgr2=0x55e7d13cdc40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:6835/4130074102 2024-09-16T01:37:07.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:07.270+0000 7fc9067fc640 1 client.6141 ms_handle_refused on v2:172.21.15.88:6835/4130074102 2024-09-16T01:37:07.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:07.270+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 unknown :-1 s=START_CONNECT pgs=14 cs=7 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-09-16T01:37:11.733 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:37:11.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.842+0000 7fee2721e640 1 Processor -- start 2024-09-16T01:37:11.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.842+0000 7fee2721e640 1 -- start start 2024-09-16T01:37:11.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.843+0000 7fee2721e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee2015e950 0x7fee2006ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:11.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.843+0000 7fee2721e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee2006ca30 con 0x7fee200735b0 2024-09-16T01:37:11.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.843+0000 7fee2721e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee2006faf0 con 0x7fee20072050 2024-09-16T01:37:11.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.843+0000 7fee2721e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fee2006fdd0 con 0x7fee2015e950 2024-09-16T01:37:11.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.843+0000 7fee1ffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fee200735b0 0x7fee2006bc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35328/0 (socket says 172.21.15.88:35328) 2024-09-16T01:37:11.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.843+0000 7fee1ffff640 1 -- 172.21.15.88:0/3971561540 learned_addr learned my addr 172.21.15.88:0/3971561540 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:37:11.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.844+0000 7fee24fba640 1 --2- 172.21.15.88:0/3971561540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee2015e950 0x7fee2006ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:11.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.844+0000 7fee1f7fe640 1 -- 172.21.15.88:0/3971561540 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2097730853 0 0) 0x7fee2006ca30 con 0x7fee200735b0 2024-09-16T01:37:11.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.844+0000 7fee1f7fe640 1 -- 172.21.15.88:0/3971561540 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee08003520 con 0x7fee200735b0 2024-09-16T01:37:11.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.844+0000 7fee1f7fe640 1 -- 172.21.15.88:0/3971561540 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2628151784 0 0) 0x7fee08003520 con 0x7fee200735b0 2024-09-16T01:37:11.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.844+0000 7fee1f7fe640 1 -- 172.21.15.88:0/3971561540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee2015e950 msgr2=0x7fee2006ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:11.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.844+0000 7fee1f7fe640 1 --2- 172.21.15.88:0/3971561540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee2015e950 0x7fee2006ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:11.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.844+0000 7fee1f7fe640 1 -- 172.21.15.88:0/3971561540 >> v1:172.21.15.154:6789/0 conn(0x7fee20072050 legacy=0x7fee2006c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:11.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.844+0000 7fee1f7fe640 1 -- 172.21.15.88:0/3971561540 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee200700b0 con 0x7fee200735b0 2024-09-16T01:37:11.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.844+0000 7fee1f7fe640 1 -- 172.21.15.88:0/3971561540 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fee14001670 con 0x7fee200735b0 2024-09-16T01:37:11.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.844+0000 7fee1f7fe640 1 -- 172.21.15.88:0/3971561540 >> v1:172.21.15.88:6789/0 conn(0x7fee200735b0 legacy=0x7fee2006bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:11.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.844+0000 7fee1f7fe640 1 --2- 172.21.15.88:0/3971561540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee2015e950 0x7fee080039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:11.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.844+0000 7fee1f7fe640 1 --2- 172.21.15.88:0/3971561540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee08003f20 0x7fee08004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:11.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.844+0000 7fee1f7fe640 1 --2- 172.21.15.88:0/3971561540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee08004d40 0x7fee080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:11.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.844+0000 7fee1f7fe640 1 -- 172.21.15.88:0/3971561540 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fee2006fdd0 con 0x7fee08003f20 2024-09-16T01:37:11.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.844+0000 7fee1f7fe640 1 -- 172.21.15.88:0/3971561540 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fee0800ba20 con 0x7fee08004d40 2024-09-16T01:37:11.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.844+0000 7fee1f7fe640 1 -- 172.21.15.88:0/3971561540 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fee0800bc20 con 0x7fee2015e950 2024-09-16T01:37:11.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.844+0000 7fee1ffff640 1 --2- 172.21.15.88:0/3971561540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee08003f20 0x7fee08004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:11.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.844+0000 7fee24fba640 1 --2- 172.21.15.88:0/3971561540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee2015e950 0x7fee080039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:11.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.844+0000 7fee257bb640 1 --2- 172.21.15.88:0/3971561540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee08004d40 0x7fee080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:11.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.844+0000 7fee1ffff640 1 -- 172.21.15.88:0/3971561540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee2015e950 msgr2=0x7fee080039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:11.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.844+0000 7fee1ffff640 1 --2- 172.21.15.88:0/3971561540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee2015e950 0x7fee080039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:11.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.844+0000 7fee1ffff640 1 -- 172.21.15.88:0/3971561540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee08004d40 msgr2=0x7fee080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:11.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.844+0000 7fee1ffff640 1 --2- 172.21.15.88:0/3971561540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee08004d40 0x7fee080051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:11.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.844+0000 7fee1ffff640 1 -- 172.21.15.88:0/3971561540 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee08003520 con 0x7fee08003f20 2024-09-16T01:37:11.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.844+0000 7fee257bb640 1 --2- 172.21.15.88:0/3971561540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee08004d40 0x7fee080051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:37:11.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.845+0000 7fee1ffff640 1 --2- 172.21.15.88:0/3971561540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee08003f20 0x7fee08004800 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fee20069100 tx=0x7fee08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cde6a2ce8a079be server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:11.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.845+0000 7fee1f7fe640 1 -- 172.21.15.88:0/3971561540 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee14007cf0 con 0x7fee08003f20 2024-09-16T01:37:11.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.845+0000 7fee1f7fe640 1 -- 172.21.15.88:0/3971561540 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee14005ad0 con 0x7fee08003f20 2024-09-16T01:37:11.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.845+0000 7fee1f7fe640 1 -- 172.21.15.88:0/3971561540 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee14007070 con 0x7fee08003f20 2024-09-16T01:37:11.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.846+0000 7fee2721e640 1 -- 172.21.15.88:0/3971561540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee08003f20 msgr2=0x7fee08004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:11.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.846+0000 7fee2721e640 1 --2- 172.21.15.88:0/3971561540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee08003f20 0x7fee08004800 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fee20069100 tx=0x7fee08005830 comp rx=0 tx=0).stop 2024-09-16T01:37:11.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.846+0000 7fee257bb640 1 -- 172.21.15.88:0/3971561540 reap_dead start 2024-09-16T01:37:11.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.846+0000 7fee2721e640 1 -- 172.21.15.88:0/3971561540 shutdown_connections 2024-09-16T01:37:11.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.846+0000 7fee2721e640 1 -- 172.21.15.88:0/3971561540 >> 172.21.15.88:0/3971561540 conn(0x7fee200d14d0 msgr2=0x7fee200d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:11.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.846+0000 7fee2721e640 1 -- 172.21.15.88:0/3971561540 shutdown_connections 2024-09-16T01:37:11.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.846+0000 7fee2721e640 1 -- 172.21.15.88:0/3971561540 wait complete. 2024-09-16T01:37:11.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.847+0000 7fee2721e640 1 Processor -- start 2024-09-16T01:37:11.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.847+0000 7fee2721e640 1 -- start start 2024-09-16T01:37:11.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.848+0000 7fee2721e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee08003f20 0x7fee20147a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:11.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.848+0000 7fee2721e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee200aa480 0x7fee20158f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:11.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.848+0000 7fee2721e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee20159470 0x7fee201598f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:11.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.848+0000 7fee2721e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fee2006f8b0 con 0x7fee200aa480 2024-09-16T01:37:11.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.848+0000 7fee2721e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fee2006f730 con 0x7fee20159470 2024-09-16T01:37:11.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.848+0000 7fee2721e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fee200d00b0 con 0x7fee08003f20 2024-09-16T01:37:11.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.848+0000 7fee1ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee200aa480 0x7fee20158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:11.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.848+0000 7fee1ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee200aa480 0x7fee20158f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36894/0 (socket says 172.21.15.88:36894) 2024-09-16T01:37:11.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.848+0000 7fee1ffff640 1 -- 172.21.15.88:0/2748604468 learned_addr learned my addr 172.21.15.88:0/2748604468 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:11.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.848+0000 7fee1ffff640 1 -- 172.21.15.88:0/2748604468 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee08003f20 msgr2=0x7fee20147a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:11.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.848+0000 7fee1ffff640 1 --2- 172.21.15.88:0/2748604468 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee08003f20 0x7fee20147a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:11.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.848+0000 7fee1ffff640 1 -- 172.21.15.88:0/2748604468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee20159470 msgr2=0x7fee201598f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:11.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.848+0000 7fee1ffff640 1 --2- 172.21.15.88:0/2748604468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee20159470 0x7fee201598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:11.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.848+0000 7fee1ffff640 1 -- 172.21.15.88:0/2748604468 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee200aa900 con 0x7fee200aa480 2024-09-16T01:37:11.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.849+0000 7fee1ffff640 1 --2- 172.21.15.88:0/2748604468 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee200aa480 0x7fee20158f30 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fee200743c0 tx=0x7fee140019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:11.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.849+0000 7fee1d7fa640 1 -- 172.21.15.88:0/2748604468 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee14040020 con 0x7fee200aa480 2024-09-16T01:37:11.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.849+0000 7fee1d7fa640 1 -- 172.21.15.88:0/2748604468 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee14001430 con 0x7fee200aa480 2024-09-16T01:37:11.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.849+0000 7fee1d7fa640 1 -- 172.21.15.88:0/2748604468 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee14037040 con 0x7fee200aa480 2024-09-16T01:37:11.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.849+0000 7fee2721e640 1 -- 172.21.15.88:0/2748604468 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee201481d0 con 0x7fee200aa480 2024-09-16T01:37:11.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.849+0000 7fee2721e640 1 -- 172.21.15.88:0/2748604468 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee20148790 con 0x7fee200aa480 2024-09-16T01:37:11.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.850+0000 7fee2721e640 1 -- 172.21.15.88:0/2748604468 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee2006ce30 con 0x7fee200aa480 2024-09-16T01:37:11.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.851+0000 7fee1d7fa640 1 -- 172.21.15.88:0/2748604468 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fee14043020 con 0x7fee200aa480 2024-09-16T01:37:11.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.851+0000 7fee1d7fa640 1 --2- 172.21.15.88:0/2748604468 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fee0c06c2b0 0x7fee0c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:11.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.851+0000 7fee1d7fa640 1 -- 172.21.15.88:0/2748604468 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5795+0+0 (secure 0 0 0) 0x7fee140ab0c0 con 0x7fee200aa480 2024-09-16T01:37:11.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.851+0000 7fee24fba640 1 --2- 172.21.15.88:0/2748604468 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fee0c06c2b0 0x7fee0c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:11.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.852+0000 7fee24fba640 1 --2- 172.21.15.88:0/2748604468 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fee0c06c2b0 0x7fee0c06e770 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fee10009ab0 tx=0x7fee100095d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:11.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:11.855+0000 7fee1d7fa640 1 -- 172.21.15.88:0/2748604468 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fee14005aa0 con 0x7fee200aa480 2024-09-16T01:37:12.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:12.054+0000 7fee2721e640 1 -- 172.21.15.88:0/2748604468 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee200b58d0 con 0x7fee200aa480 2024-09-16T01:37:12.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:12.055+0000 7fee1d7fa640 1 -- 172.21.15.88:0/2748604468 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5224 (secure 0 0 0) 0x7fee14079940 con 0x7fee200aa480 2024-09-16T01:37:12.057 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:37:12.057 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":45,"btime":"2024-09-16T01:37:00:860056+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/581241194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":581241194},{"type":"v1","addr":"172.21.15.154:6834","nonce":581241194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2104508521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2104508521},{"type":"v1","addr":"172.21.15.154:6835","nonce":2104508521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5859,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/998283126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":998283126},{"type":"v1","addr":"172.21.15.88:6836","nonce":998283126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:39.786438+0000","modified":"2024-09-16T01:36:59.884181+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":5,"addr":"172.21.15.88:6837/4130074102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4130074102},{"type":"v1","addr":"172.21.15.88:6837","nonce":4130074102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":4}]} 2024-09-16T01:37:12.058 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 45 2024-09-16T01:37:12.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:12.057+0000 7fee2721e640 1 -- 172.21.15.88:0/2748604468 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fee0c06c2b0 msgr2=0x7fee0c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:12.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:12.057+0000 7fee2721e640 1 --2- 172.21.15.88:0/2748604468 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fee0c06c2b0 0x7fee0c06e770 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fee10009ab0 tx=0x7fee100095d0 comp rx=0 tx=0).stop 2024-09-16T01:37:12.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:12.057+0000 7fee2721e640 1 -- 172.21.15.88:0/2748604468 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee200aa480 msgr2=0x7fee20158f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:12.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:12.057+0000 7fee2721e640 1 --2- 172.21.15.88:0/2748604468 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee200aa480 0x7fee20158f30 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fee200743c0 tx=0x7fee140019c0 comp rx=0 tx=0).stop 2024-09-16T01:37:12.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:12.057+0000 7fee2721e640 1 -- 172.21.15.88:0/2748604468 shutdown_connections 2024-09-16T01:37:12.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:12.057+0000 7fee2721e640 1 --2- 172.21.15.88:0/2748604468 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fee0c06c2b0 0x7fee0c06e770 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:12.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:12.057+0000 7fee2721e640 1 --2- 172.21.15.88:0/2748604468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee20159470 0x7fee201598f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:12.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:12.057+0000 7fee2721e640 1 --2- 172.21.15.88:0/2748604468 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fee200aa480 0x7fee20158f30 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:12.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:12.057+0000 7fee2721e640 1 --2- 172.21.15.88:0/2748604468 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee08003f20 0x7fee20147a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:12.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:12.057+0000 7fee2721e640 1 -- 172.21.15.88:0/2748604468 >> 172.21.15.88:0/2748604468 conn(0x7fee200d14d0 msgr2=0x7fee2015ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:12.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:12.057+0000 7fee2721e640 1 -- 172.21.15.88:0/2748604468 shutdown_connections 2024-09-16T01:37:12.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:12.057+0000 7fee2721e640 1 -- 172.21.15.88:0/2748604468 wait complete. 2024-09-16T01:37:12.070 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-09-16T01:37:12.410 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~7s 2024-09-16T01:37:13.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:13.677+0000 7fc9267fc640 1 -- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 msgr2=0x55e7d13cdc40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.88:6835/4130074102 2024-09-16T01:37:13.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:13.677+0000 7fc9067fc640 1 client.6141 ms_handle_refused on v2:172.21.15.88:6835/4130074102 2024-09-16T01:37:13.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:13.677+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 unknown :-1 s=START_CONNECT pgs=14 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-09-16T01:37:16.598 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:37:16.601 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:37:17.071 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:37:17.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.180+0000 7f112ef6d640 1 Processor -- start 2024-09-16T01:37:17.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.180+0000 7f112ef6d640 1 -- start start 2024-09-16T01:37:17.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.180+0000 7f112ef6d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f112815e660 0x7f11280cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:17.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.181+0000 7f112ef6d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f11280d3310 0x7f11280d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:17.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.181+0000 7f112ef6d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f112815ea40 con 0x7f11280d1650 2024-09-16T01:37:17.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.181+0000 7f112ef6d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f11280d3870 con 0x7f11280d3310 2024-09-16T01:37:17.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.181+0000 7f112ef6d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f11280d3b50 con 0x7f112815e660 2024-09-16T01:37:17.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.181+0000 7f1127fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f11280d3310 0x7f11280d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:17.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.181+0000 7f112d50a640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f11280d1650 0x7f112806bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35330/0 (socket says 172.21.15.88:35330) 2024-09-16T01:37:17.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.181+0000 7f112d50a640 1 -- 172.21.15.88:0/1687914836 learned_addr learned my addr 172.21.15.88:0/1687914836 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:37:17.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.181+0000 7f112cd09640 1 --2- 172.21.15.88:0/1687914836 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f112815e660 0x7f11280cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:17.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.181+0000 7f1127fff640 1 -- 172.21.15.88:0/1687914836 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f112815e660 msgr2=0x7f11280cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:17.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.181+0000 7f1127fff640 1 --2- 172.21.15.88:0/1687914836 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f112815e660 0x7f11280cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:17.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.181+0000 7f1127fff640 1 -- 172.21.15.88:0/1687914836 >> v1:172.21.15.88:6789/0 conn(0x7f11280d1650 legacy=0x7f112806bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:17.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.181+0000 7f1127fff640 1 -- 172.21.15.88:0/1687914836 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11280d3e30 con 0x7f11280d3310 2024-09-16T01:37:17.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.181+0000 7f11277fe640 1 -- 172.21.15.88:0/1687914836 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2312440102 0 0) 0x7f112815ea40 con 0x7f11280d1650 2024-09-16T01:37:17.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.181+0000 7f112cd09640 1 --2- 172.21.15.88:0/1687914836 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f112815e660 0x7f11280cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:37:17.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.182+0000 7f1127fff640 1 --2- 172.21.15.88:0/1687914836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11280d3310 0x7f11280d1110 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f11140097a0 tx=0x7f111402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=834b462578a9465 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:17.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.182+0000 7f11277fe640 1 -- 172.21.15.88:0/1687914836 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1114037070 con 0x7f11280d3310 2024-09-16T01:37:17.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.182+0000 7f11277fe640 1 -- 172.21.15.88:0/1687914836 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1114032440 con 0x7f11280d3310 2024-09-16T01:37:17.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.182+0000 7f11277fe640 1 -- 172.21.15.88:0/1687914836 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1114032810 con 0x7f11280d3310 2024-09-16T01:37:17.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.182+0000 7f112ef6d640 1 -- 172.21.15.88:0/1687914836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11280d3310 msgr2=0x7f11280d1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:17.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.182+0000 7f112ef6d640 1 --2- 172.21.15.88:0/1687914836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11280d3310 0x7f11280d1110 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f11140097a0 tx=0x7f111402a180 comp rx=0 tx=0).stop 2024-09-16T01:37:17.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.183+0000 7f112ef6d640 1 -- 172.21.15.88:0/1687914836 shutdown_connections 2024-09-16T01:37:17.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.183+0000 7f112ef6d640 1 --2- 172.21.15.88:0/1687914836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11280d3310 0x7f11280d1110 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:17.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.183+0000 7f112ef6d640 1 --2- 172.21.15.88:0/1687914836 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f112815e660 0x7f11280cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:17.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.183+0000 7f112ef6d640 1 -- 172.21.15.88:0/1687914836 >> 172.21.15.88:0/1687914836 conn(0x7f11280b6410 msgr2=0x7f11280b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:17.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.183+0000 7f112ef6d640 1 -- 172.21.15.88:0/1687914836 shutdown_connections 2024-09-16T01:37:17.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.183+0000 7f112ef6d640 1 -- 172.21.15.88:0/1687914836 wait complete. 2024-09-16T01:37:17.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.184+0000 7f112ef6d640 1 Processor -- start 2024-09-16T01:37:17.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.184+0000 7f112ef6d640 1 -- start start 2024-09-16T01:37:17.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.185+0000 7f112ef6d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11280d1650 0x7f1128150710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:17.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.185+0000 7f112ef6d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11280d3310 0x7f11281f82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:17.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.185+0000 7f112ef6d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f112815e660 0x7f1128157480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:17.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.185+0000 7f112ef6d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f11280cfc70 con 0x7f11280d3310 2024-09-16T01:37:17.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.185+0000 7f112ef6d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f11280d43f0 con 0x7f112815e660 2024-09-16T01:37:17.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.185+0000 7f112ef6d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f11280b8b80 con 0x7f11280d1650 2024-09-16T01:37:17.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.185+0000 7f1127fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11280d3310 0x7f11281f82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:17.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.185+0000 7f1127fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11280d3310 0x7f11281f82a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36900/0 (socket says 172.21.15.88:36900) 2024-09-16T01:37:17.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.185+0000 7f1127fff640 1 -- 172.21.15.88:0/2489957553 learned_addr learned my addr 172.21.15.88:0/2489957553 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:17.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.185+0000 7f112cd09640 1 --2- 172.21.15.88:0/2489957553 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11280d1650 0x7f1128150710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:17.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.185+0000 7f1127fff640 1 -- 172.21.15.88:0/2489957553 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11280d1650 msgr2=0x7f1128150710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:17.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.185+0000 7f1127fff640 1 --2- 172.21.15.88:0/2489957553 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11280d1650 0x7f1128150710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:17.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.185+0000 7f1127fff640 1 -- 172.21.15.88:0/2489957553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f112815e660 msgr2=0x7f1128157480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:17.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.185+0000 7f1127fff640 1 --2- 172.21.15.88:0/2489957553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f112815e660 0x7f1128157480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:17.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.185+0000 7f1127fff640 1 -- 172.21.15.88:0/2489957553 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11281f87e0 con 0x7f11280d3310 2024-09-16T01:37:17.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.185+0000 7f112cd09640 1 --2- 172.21.15.88:0/2489957553 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11280d1650 0x7f1128150710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:37:17.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.185+0000 7f1127fff640 1 --2- 172.21.15.88:0/2489957553 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11280d3310 0x7f11281f82a0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f11140097a0 tx=0x7f11140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:17.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.185+0000 7f11257fa640 1 -- 172.21.15.88:0/2489957553 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1114037050 con 0x7f11280d3310 2024-09-16T01:37:17.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.185+0000 7f11257fa640 1 -- 172.21.15.88:0/2489957553 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1114038040 con 0x7f11280d3310 2024-09-16T01:37:17.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.185+0000 7f112ef6d640 1 -- 172.21.15.88:0/2489957553 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f112815ed80 con 0x7f11280d3310 2024-09-16T01:37:17.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.185+0000 7f11257fa640 1 -- 172.21.15.88:0/2489957553 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1114004490 con 0x7f11280d3310 2024-09-16T01:37:17.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.186+0000 7f112ef6d640 1 -- 172.21.15.88:0/2489957553 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f112815f340 con 0x7f11280d3310 2024-09-16T01:37:17.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.186+0000 7f112ef6d640 1 -- 172.21.15.88:0/2489957553 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11280d1110 con 0x7f11280d3310 2024-09-16T01:37:17.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.187+0000 7f11257fa640 1 -- 172.21.15.88:0/2489957553 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1114030980 con 0x7f11280d3310 2024-09-16T01:37:17.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.188+0000 7f11257fa640 1 --2- 172.21.15.88:0/2489957553 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f110c06c2b0 0x7f110c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:17.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.188+0000 7f112cd09640 1 --2- 172.21.15.88:0/2489957553 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f110c06c2b0 0x7f110c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:17.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.188+0000 7f11257fa640 1 -- 172.21.15.88:0/2489957553 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5795+0+0 (secure 0 0 0) 0x7f11140ac040 con 0x7f11280d3310 2024-09-16T01:37:17.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.188+0000 7f112cd09640 1 --2- 172.21.15.88:0/2489957553 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f110c06c2b0 0x7f110c06e770 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f111c006fd0 tx=0x7f111c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:17.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.192+0000 7f11257fa640 1 -- 172.21.15.88:0/2489957553 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f111407a900 con 0x7f11280d3310 2024-09-16T01:37:17.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.393+0000 7f112ef6d640 1 -- 172.21.15.88:0/2489957553 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f112806aea0 con 0x7f11280d3310 2024-09-16T01:37:17.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.393+0000 7f11257fa640 1 -- 172.21.15.88:0/2489957553 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5224 (secure 0 0 0) 0x7f111407a200 con 0x7f11280d3310 2024-09-16T01:37:17.395 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:37:17.395 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":45,"btime":"2024-09-16T01:37:00:860056+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/581241194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":581241194},{"type":"v1","addr":"172.21.15.154:6834","nonce":581241194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2104508521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2104508521},{"type":"v1","addr":"172.21.15.154:6835","nonce":2104508521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5859,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/998283126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":998283126},{"type":"v1","addr":"172.21.15.88:6836","nonce":998283126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:39.786438+0000","modified":"2024-09-16T01:36:59.884181+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":5,"addr":"172.21.15.88:6837/4130074102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4130074102},{"type":"v1","addr":"172.21.15.88:6837","nonce":4130074102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":4}]} 2024-09-16T01:37:17.395 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 45 2024-09-16T01:37:17.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.395+0000 7f112ef6d640 1 -- 172.21.15.88:0/2489957553 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f110c06c2b0 msgr2=0x7f110c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:17.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.395+0000 7f112ef6d640 1 --2- 172.21.15.88:0/2489957553 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f110c06c2b0 0x7f110c06e770 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f111c006fd0 tx=0x7f111c008040 comp rx=0 tx=0).stop 2024-09-16T01:37:17.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.395+0000 7f112ef6d640 1 -- 172.21.15.88:0/2489957553 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11280d3310 msgr2=0x7f11281f82a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:17.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.395+0000 7f112ef6d640 1 --2- 172.21.15.88:0/2489957553 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11280d3310 0x7f11281f82a0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f11140097a0 tx=0x7f11140046c0 comp rx=0 tx=0).stop 2024-09-16T01:37:17.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.396+0000 7f112ef6d640 1 -- 172.21.15.88:0/2489957553 shutdown_connections 2024-09-16T01:37:17.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.396+0000 7f112ef6d640 1 --2- 172.21.15.88:0/2489957553 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f110c06c2b0 0x7f110c06e770 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:17.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.396+0000 7f112ef6d640 1 --2- 172.21.15.88:0/2489957553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f112815e660 0x7f1128157480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:17.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.396+0000 7f112ef6d640 1 --2- 172.21.15.88:0/2489957553 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11280d3310 0x7f11281f82a0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:17.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.396+0000 7f112ef6d640 1 --2- 172.21.15.88:0/2489957553 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11280d1650 0x7f1128150710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:17.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.396+0000 7f112ef6d640 1 -- 172.21.15.88:0/2489957553 >> 172.21.15.88:0/2489957553 conn(0x7f11280b6410 msgr2=0x7f11280b41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:17.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.396+0000 7f112ef6d640 1 -- 172.21.15.88:0/2489957553 shutdown_connections 2024-09-16T01:37:17.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:17.396+0000 7f112ef6d640 1 -- 172.21.15.88:0/2489957553 wait complete. 2024-09-16T01:37:17.409 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-09-16T01:37:19.023 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~13s 2024-09-16T01:37:22.409 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:37:22.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.519+0000 7f7e99954640 1 Processor -- start 2024-09-16T01:37:22.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.519+0000 7f7e99954640 1 -- start start 2024-09-16T01:37:22.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.519+0000 7f7e99954640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e940d3310 0x7f7e940d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:22.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.520+0000 7f7e99954640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7e940d1650 0x7f7e940b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:22.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.520+0000 7f7e99954640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e9415ea40 con 0x7f7e9415e660 2024-09-16T01:37:22.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.520+0000 7f7e99954640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7e940d1ac0 con 0x7f7e940d1650 2024-09-16T01:37:22.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.520+0000 7f7e99954640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7e940d3870 con 0x7f7e940d3310 2024-09-16T01:37:22.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.520+0000 7f7e937fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7e940d1650 0x7f7e940b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:22.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.520+0000 7f7e92ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7e9415e660 0x7f7e940cf730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58822/0 (socket says 172.21.15.88:58822) 2024-09-16T01:37:22.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.520+0000 7f7e92ffd640 1 -- 172.21.15.88:0/311861600 learned_addr learned my addr 172.21.15.88:0/311861600 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:37:22.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.520+0000 7f7e8a7fc640 1 --2- 172.21.15.88:0/311861600 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e940d3310 0x7f7e940d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:22.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.520+0000 7f7e937fe640 1 -- 172.21.15.88:0/311861600 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e940d3310 msgr2=0x7f7e940d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:22.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.520+0000 7f7e937fe640 1 --2- 172.21.15.88:0/311861600 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e940d3310 0x7f7e940d1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:22.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.520+0000 7f7e937fe640 1 -- 172.21.15.88:0/311861600 >> v1:172.21.15.88:6789/0 conn(0x7f7e9415e660 legacy=0x7f7e940cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:22.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.520+0000 7f7e937fe640 1 -- 172.21.15.88:0/311861600 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e940d3b50 con 0x7f7e940d1650 2024-09-16T01:37:22.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.520+0000 7f7e927fc640 1 -- 172.21.15.88:0/311861600 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3864399006 0 0) 0x7f7e9415ea40 con 0x7f7e9415e660 2024-09-16T01:37:22.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.521+0000 7f7e937fe640 1 --2- 172.21.15.88:0/311861600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e940d1650 0x7f7e940b7b70 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c009d90 tx=0x7f7e8c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c718acb52d8a027 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:22.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.521+0000 7f7e927fc640 1 -- 172.21.15.88:0/311861600 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c007740 con 0x7f7e940d1650 2024-09-16T01:37:22.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.521+0000 7f7e927fc640 1 -- 172.21.15.88:0/311861600 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e8c0078e0 con 0x7f7e940d1650 2024-09-16T01:37:22.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.521+0000 7f7e927fc640 1 -- 172.21.15.88:0/311861600 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c007cb0 con 0x7f7e940d1650 2024-09-16T01:37:22.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.521+0000 7f7e99954640 1 -- 172.21.15.88:0/311861600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e940d1650 msgr2=0x7f7e940b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:22.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.521+0000 7f7e99954640 1 --2- 172.21.15.88:0/311861600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e940d1650 0x7f7e940b7b70 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c009d90 tx=0x7f7e8c02a660 comp rx=0 tx=0).stop 2024-09-16T01:37:22.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.522+0000 7f7e99954640 1 -- 172.21.15.88:0/311861600 shutdown_connections 2024-09-16T01:37:22.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.522+0000 7f7e99954640 1 --2- 172.21.15.88:0/311861600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e940d1650 0x7f7e940b7b70 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:22.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.522+0000 7f7e99954640 1 --2- 172.21.15.88:0/311861600 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e940d3310 0x7f7e940d1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:22.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.522+0000 7f7e99954640 1 -- 172.21.15.88:0/311861600 >> 172.21.15.88:0/311861600 conn(0x7f7e940b6410 msgr2=0x7f7e940b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:22.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.522+0000 7f7e99954640 1 -- 172.21.15.88:0/311861600 shutdown_connections 2024-09-16T01:37:22.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.522+0000 7f7e99954640 1 -- 172.21.15.88:0/311861600 wait complete. 2024-09-16T01:37:22.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.523+0000 7f7e99954640 1 Processor -- start 2024-09-16T01:37:22.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.523+0000 7f7e99954640 1 -- start start 2024-09-16T01:37:22.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.524+0000 7f7e99954640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e940d1650 0x7f7e940c9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:22.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.524+0000 7f7e99954640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e940d3310 0x7f7e9414fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:22.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.524+0000 7f7e99954640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e9415e660 0x7f7e941603d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:22.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.524+0000 7f7e99954640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7e940d43f0 con 0x7f7e9415e660 2024-09-16T01:37:22.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.524+0000 7f7e99954640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7e940d4110 con 0x7f7e940d3310 2024-09-16T01:37:22.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.524+0000 7f7e99954640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7e940b80b0 con 0x7f7e940d1650 2024-09-16T01:37:22.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.524+0000 7f7e937fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e9415e660 0x7f7e941603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:22.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.524+0000 7f7e937fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e9415e660 0x7f7e941603d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53636/0 (socket says 172.21.15.88:53636) 2024-09-16T01:37:22.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.524+0000 7f7e937fe640 1 -- 172.21.15.88:0/460907211 learned_addr learned my addr 172.21.15.88:0/460907211 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:22.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.524+0000 7f7e92ffd640 1 --2- 172.21.15.88:0/460907211 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e940d1650 0x7f7e940c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:22.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.524+0000 7f7e8a7fc640 1 --2- 172.21.15.88:0/460907211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e940d3310 0x7f7e9414fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:22.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.524+0000 7f7e937fe640 1 -- 172.21.15.88:0/460907211 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e940d1650 msgr2=0x7f7e940c9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:22.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.524+0000 7f7e937fe640 1 --2- 172.21.15.88:0/460907211 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e940d1650 0x7f7e940c9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:22.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.524+0000 7f7e937fe640 1 -- 172.21.15.88:0/460907211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e940d3310 msgr2=0x7f7e9414fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:22.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.524+0000 7f7e937fe640 1 --2- 172.21.15.88:0/460907211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e940d3310 0x7f7e9414fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:22.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.524+0000 7f7e937fe640 1 -- 172.21.15.88:0/460907211 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e941f4c60 con 0x7f7e9415e660 2024-09-16T01:37:22.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.524+0000 7f7e937fe640 1 --2- 172.21.15.88:0/460907211 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e9415e660 0x7f7e941603d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c009eb0 tx=0x7f7e8c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:22.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.524+0000 7f7e90ff9640 1 -- 172.21.15.88:0/460907211 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c040070 con 0x7f7e9415e660 2024-09-16T01:37:22.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.524+0000 7f7e90ff9640 1 -- 172.21.15.88:0/460907211 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e8c031bc0 con 0x7f7e9415e660 2024-09-16T01:37:22.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.524+0000 7f7e99954640 1 -- 172.21.15.88:0/460907211 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e940ce5c0 con 0x7f7e9415e660 2024-09-16T01:37:22.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.524+0000 7f7e90ff9640 1 -- 172.21.15.88:0/460907211 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c009450 con 0x7f7e9415e660 2024-09-16T01:37:22.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.525+0000 7f7e99954640 1 -- 172.21.15.88:0/460907211 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e94160910 con 0x7f7e9415e660 2024-09-16T01:37:22.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.526+0000 7f7e99954640 1 -- 172.21.15.88:0/460907211 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e940b7b70 con 0x7f7e9415e660 2024-09-16T01:37:22.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.526+0000 7f7e90ff9640 1 -- 172.21.15.88:0/460907211 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7e8c039040 con 0x7f7e9415e660 2024-09-16T01:37:22.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.526+0000 7f7e90ff9640 1 --2- 172.21.15.88:0/460907211 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7e7406c2e0 0x7f7e7406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:22.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.526+0000 7f7e90ff9640 1 -- 172.21.15.88:0/460907211 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5795+0+0 (secure 0 0 0) 0x7f7e8c0acaa0 con 0x7f7e9415e660 2024-09-16T01:37:22.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.526+0000 7f7e92ffd640 1 --2- 172.21.15.88:0/460907211 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7e7406c2e0 0x7f7e7406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:22.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.527+0000 7f7e92ffd640 1 --2- 172.21.15.88:0/460907211 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7e7406c2e0 0x7f7e7406e7a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7e84007fd0 tx=0x7f7e84007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:22.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.531+0000 7f7e90ff9640 1 -- 172.21.15.88:0/460907211 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7e8c07abf0 con 0x7f7e9415e660 2024-09-16T01:37:22.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.729+0000 7f7e99954640 1 -- 172.21.15.88:0/460907211 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e940ce390 con 0x7f7e9415e660 2024-09-16T01:37:22.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.729+0000 7f7e90ff9640 1 -- 172.21.15.88:0/460907211 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+5272 (secure 0 0 0) 0x7f7e8c07eef0 con 0x7f7e9415e660 2024-09-16T01:37:22.731 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:37:22.731 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":46,"btime":"2024-09-16T01:37:18:217401+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/581241194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":581241194},{"type":"v1","addr":"172.21.15.154:6834","nonce":581241194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2104508521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2104508521},{"type":"v1","addr":"172.21.15.154:6835","nonce":2104508521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5859,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/998283126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":998283126},{"type":"v1","addr":"172.21.15.88:6836","nonce":998283126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:39.786438+0000","modified":"2024-09-16T01:37:18.217362+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"a","rank":0,"incarnation":41,"state":"up:active","state_seq":5,"addr":"172.21.15.88:6837/4130074102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4130074102},{"type":"v1","addr":"172.21.15.88:6837","nonce":4130074102}]},"join_fscid":-1,"laggy_since":"2024-09-16T01:37:18.217362+0000","export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":4}]} 2024-09-16T01:37:22.731 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 46 2024-09-16T01:37:22.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.731+0000 7f7e99954640 1 -- 172.21.15.88:0/460907211 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7e7406c2e0 msgr2=0x7f7e7406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:22.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.731+0000 7f7e99954640 1 --2- 172.21.15.88:0/460907211 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7e7406c2e0 0x7f7e7406e7a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7e84007fd0 tx=0x7f7e84007de0 comp rx=0 tx=0).stop 2024-09-16T01:37:22.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.731+0000 7f7e99954640 1 -- 172.21.15.88:0/460907211 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e9415e660 msgr2=0x7f7e941603d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:22.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.731+0000 7f7e99954640 1 --2- 172.21.15.88:0/460907211 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e9415e660 0x7f7e941603d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c009eb0 tx=0x7f7e8c004a40 comp rx=0 tx=0).stop 2024-09-16T01:37:22.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.731+0000 7f7e99954640 1 -- 172.21.15.88:0/460907211 shutdown_connections 2024-09-16T01:37:22.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.731+0000 7f7e99954640 1 --2- 172.21.15.88:0/460907211 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7e7406c2e0 0x7f7e7406e7a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:22.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.731+0000 7f7e99954640 1 --2- 172.21.15.88:0/460907211 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e9415e660 0x7f7e941603d0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:22.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.731+0000 7f7e99954640 1 --2- 172.21.15.88:0/460907211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e940d3310 0x7f7e9414fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:22.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.731+0000 7f7e99954640 1 --2- 172.21.15.88:0/460907211 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e940d1650 0x7f7e940c9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:22.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.731+0000 7f7e99954640 1 -- 172.21.15.88:0/460907211 >> 172.21.15.88:0/460907211 conn(0x7f7e940b6410 msgr2=0x7f7e940b5a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:22.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.732+0000 7f7e99954640 1 -- 172.21.15.88:0/460907211 shutdown_connections 2024-09-16T01:37:22.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.732+0000 7f7e99954640 1 -- 172.21.15.88:0/460907211 wait complete. 2024-09-16T01:37:22.744 DEBUG:tasks.ceph_test_case:wait_until_true: success in 20s and 0 retries 2024-09-16T01:37:22.845 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_inject_journal_corrupt_dentry_first 2024-09-16T01:37:22.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.944+0000 7ff5a70e4640 1 Processor -- start 2024-09-16T01:37:22.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.944+0000 7ff5a70e4640 1 -- start start 2024-09-16T01:37:22.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.945+0000 7ff5a70e4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff5a006fa00 0x7ff5a00d02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:22.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.945+0000 7ff5a70e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5a015d3d0 0x7ff5a00d0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:22.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.945+0000 7ff5a70e4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5a0071770 con 0x7ff5a015a060 2024-09-16T01:37:22.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.945+0000 7ff5a70e4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff5a00712e0 con 0x7ff5a006fa00 2024-09-16T01:37:22.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.945+0000 7ff5a70e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff5a00709b0 con 0x7ff5a015d3d0 2024-09-16T01:37:22.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.945+0000 7ff5a4e80640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff5a015a060 0x7ff5a006f2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58838/0 (socket says 172.21.15.88:58838) 2024-09-16T01:37:22.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.945+0000 7ff5a4e80640 1 -- 172.21.15.88:0/2530708592 learned_addr learned my addr 172.21.15.88:0/2530708592 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:37:22.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.945+0000 7ff597fff640 1 --2- 172.21.15.88:0/2530708592 >> v2:172.21.15.154:3300/0 conn(0x7ff5a006fa00 0x7ff5a00d02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:22.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.945+0000 7ff5a5681640 1 --2- 172.21.15.88:0/2530708592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5a015d3d0 0x7ff5a00d0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:22.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.945+0000 7ff5977fe640 1 -- 172.21.15.88:0/2530708592 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3992985404 0 0) 0x7ff5a0071770 con 0x7ff5a015a060 2024-09-16T01:37:22.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.945+0000 7ff5977fe640 1 -- 172.21.15.88:0/2530708592 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff588003520 con 0x7ff5a015a060 2024-09-16T01:37:22.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.945+0000 7ff5977fe640 1 -- 172.21.15.88:0/2530708592 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2133237547 0 0) 0x7ff588003520 con 0x7ff5a015a060 2024-09-16T01:37:22.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.945+0000 7ff5977fe640 1 -- 172.21.15.88:0/2530708592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5a015d3d0 msgr2=0x7ff5a00d0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:22.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.945+0000 7ff5977fe640 1 --2- 172.21.15.88:0/2530708592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5a015d3d0 0x7ff5a00d0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:22.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.945+0000 7ff5977fe640 1 -- 172.21.15.88:0/2530708592 >> v2:172.21.15.154:3300/0 conn(0x7ff5a006fa00 msgr2=0x7ff5a00d02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:22.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.945+0000 7ff5977fe640 1 --2- 172.21.15.88:0/2530708592 >> v2:172.21.15.154:3300/0 conn(0x7ff5a006fa00 0x7ff5a00d02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:22.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.945+0000 7ff5977fe640 1 -- 172.21.15.88:0/2530708592 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a0070c90 con 0x7ff5a015a060 2024-09-16T01:37:22.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.945+0000 7ff5977fe640 1 -- 172.21.15.88:0/2530708592 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7ff59c001670 con 0x7ff5a015a060 2024-09-16T01:37:22.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.945+0000 7ff597fff640 1 --2- 172.21.15.88:0/2530708592 >> v2:172.21.15.154:3300/0 conn(0x7ff5a006fa00 0x7ff5a00d02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:37:22.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.945+0000 7ff5977fe640 1 -- 172.21.15.88:0/2530708592 >> v1:172.21.15.88:6789/0 conn(0x7ff5a015a060 legacy=0x7ff5a006f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:22.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.945+0000 7ff5977fe640 1 --2- 172.21.15.88:0/2530708592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5880039b0 0x7ff588003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:22.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.945+0000 7ff5977fe640 1 --2- 172.21.15.88:0/2530708592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff588004800 0x7ff588004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:22.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.945+0000 7ff5977fe640 1 --2- 172.21.15.88:0/2530708592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5a015d3d0 0x7ff5880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:22.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.946+0000 7ff5977fe640 1 -- 172.21.15.88:0/2530708592 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a00712e0 con 0x7ff5880039b0 2024-09-16T01:37:22.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.946+0000 7ff5977fe640 1 -- 172.21.15.88:0/2530708592 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a00709b0 con 0x7ff588004800 2024-09-16T01:37:22.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.946+0000 7ff5977fe640 1 -- 172.21.15.88:0/2530708592 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff58800b9c0 con 0x7ff5a015d3d0 2024-09-16T01:37:22.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.946+0000 7ff5a4e80640 1 --2- 172.21.15.88:0/2530708592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5880039b0 0x7ff588003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:22.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.946+0000 7ff597fff640 1 --2- 172.21.15.88:0/2530708592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff588004800 0x7ff588004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:22.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.946+0000 7ff5a5681640 1 --2- 172.21.15.88:0/2530708592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5a015d3d0 0x7ff5880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:22.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.946+0000 7ff5a4e80640 1 -- 172.21.15.88:0/2530708592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5a015d3d0 msgr2=0x7ff5880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:22.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.946+0000 7ff5a4e80640 1 --2- 172.21.15.88:0/2530708592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5a015d3d0 0x7ff5880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:22.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.946+0000 7ff5a4e80640 1 -- 172.21.15.88:0/2530708592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff588004800 msgr2=0x7ff588004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:22.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.946+0000 7ff5a4e80640 1 --2- 172.21.15.88:0/2530708592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff588004800 0x7ff588004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:22.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.946+0000 7ff5a4e80640 1 -- 172.21.15.88:0/2530708592 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff588003520 con 0x7ff5880039b0 2024-09-16T01:37:22.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.947+0000 7ff5a4e80640 1 --2- 172.21.15.88:0/2530708592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5880039b0 0x7ff588003dd0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7ff59c00c190 tx=0x7ff588005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63c5822c68b5d34d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:22.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.947+0000 7ff5977fe640 1 -- 172.21.15.88:0/2530708592 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59c007c60 con 0x7ff5880039b0 2024-09-16T01:37:22.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.947+0000 7ff5977fe640 1 -- 172.21.15.88:0/2530708592 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff59c005af0 con 0x7ff5880039b0 2024-09-16T01:37:22.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.947+0000 7ff5977fe640 1 -- 172.21.15.88:0/2530708592 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59c007070 con 0x7ff5880039b0 2024-09-16T01:37:22.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.947+0000 7ff5a70e4640 1 -- 172.21.15.88:0/2530708592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5880039b0 msgr2=0x7ff588003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:22.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.947+0000 7ff5a70e4640 1 --2- 172.21.15.88:0/2530708592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5880039b0 0x7ff588003dd0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7ff59c00c190 tx=0x7ff588005830 comp rx=0 tx=0).stop 2024-09-16T01:37:22.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.947+0000 7ff5a5681640 1 -- 172.21.15.88:0/2530708592 reap_dead start 2024-09-16T01:37:22.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.947+0000 7ff5a70e4640 1 -- 172.21.15.88:0/2530708592 shutdown_connections 2024-09-16T01:37:22.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.947+0000 7ff5a70e4640 1 -- 172.21.15.88:0/2530708592 >> 172.21.15.88:0/2530708592 conn(0x7ff5a00d2430 msgr2=0x7ff5a00d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:22.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.947+0000 7ff5a70e4640 1 -- 172.21.15.88:0/2530708592 shutdown_connections 2024-09-16T01:37:22.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.947+0000 7ff5a70e4640 1 -- 172.21.15.88:0/2530708592 wait complete. 2024-09-16T01:37:22.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.948+0000 7ff5a70e4640 1 Processor -- start 2024-09-16T01:37:22.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.948+0000 7ff5a70e4640 1 -- start start 2024-09-16T01:37:22.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.949+0000 7ff5a70e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5880039b0 0x7ff5a0152a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:22.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.949+0000 7ff5a70e4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5a01f86e0 0x7ff5a0155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:22.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.949+0000 7ff5a70e4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5a0146d50 0x7ff5a01f7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:22.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.949+0000 7ff5a70e4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a0070030 con 0x7ff5a0146d50 2024-09-16T01:37:22.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.949+0000 7ff5a70e4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a006feb0 con 0x7ff5a01f86e0 2024-09-16T01:37:22.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.949+0000 7ff5a70e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff5a00b1990 con 0x7ff5880039b0 2024-09-16T01:37:22.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.949+0000 7ff5a5681640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5a0146d50 0x7ff5a01f7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:22.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.949+0000 7ff5a5681640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5a0146d50 0x7ff5a01f7a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53650/0 (socket says 172.21.15.88:53650) 2024-09-16T01:37:22.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.949+0000 7ff5a5681640 1 -- 172.21.15.88:0/1660182938 learned_addr learned my addr 172.21.15.88:0/1660182938 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:22.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.950+0000 7ff597fff640 1 --2- 172.21.15.88:0/1660182938 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5a01f86e0 0x7ff5a0155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:22.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.950+0000 7ff5a4e80640 1 --2- 172.21.15.88:0/1660182938 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5880039b0 0x7ff5a0152a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:22.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.950+0000 7ff5a5681640 1 -- 172.21.15.88:0/1660182938 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5880039b0 msgr2=0x7ff5a0152a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:22.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.950+0000 7ff5a5681640 1 --2- 172.21.15.88:0/1660182938 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5880039b0 0x7ff5a0152a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:22.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.950+0000 7ff5a5681640 1 -- 172.21.15.88:0/1660182938 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5a01f86e0 msgr2=0x7ff5a0155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:22.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.950+0000 7ff5a5681640 1 --2- 172.21.15.88:0/1660182938 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5a01f86e0 0x7ff5a0155820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:22.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.950+0000 7ff5a5681640 1 -- 172.21.15.88:0/1660182938 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a0154780 con 0x7ff5a0146d50 2024-09-16T01:37:22.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.950+0000 7ff5a5681640 1 --2- 172.21.15.88:0/1660182938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5a0146d50 0x7ff5a01f7a90 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0075a00 tx=0x7ff59800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:22.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.950+0000 7ff5957fa640 1 -- 172.21.15.88:0/1660182938 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff598015430 con 0x7ff5a0146d50 2024-09-16T01:37:22.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.950+0000 7ff5957fa640 1 -- 172.21.15.88:0/1660182938 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5980155f0 con 0x7ff5a0146d50 2024-09-16T01:37:22.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.950+0000 7ff5a70e4640 1 -- 172.21.15.88:0/1660182938 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5a0154a60 con 0x7ff5a0146d50 2024-09-16T01:37:22.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.950+0000 7ff5957fa640 1 -- 172.21.15.88:0/1660182938 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5980159e0 con 0x7ff5a0146d50 2024-09-16T01:37:22.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.950+0000 7ff5a70e4640 1 -- 172.21.15.88:0/1660182938 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5a0155020 con 0x7ff5a0146d50 2024-09-16T01:37:22.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.951+0000 7ff5a70e4640 1 -- 172.21.15.88:0/1660182938 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5a006c6c0 con 0x7ff5a0146d50 2024-09-16T01:37:22.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.952+0000 7ff5957fa640 1 -- 172.21.15.88:0/1660182938 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff598015b80 con 0x7ff5a0146d50 2024-09-16T01:37:22.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.952+0000 7ff5957fa640 1 --2- 172.21.15.88:0/1660182938 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff58c06c2b0 0x7ff58c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:22.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.952+0000 7ff5a4e80640 1 --2- 172.21.15.88:0/1660182938 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff58c06c2b0 0x7ff58c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:22.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.952+0000 7ff5957fa640 1 -- 172.21.15.88:0/1660182938 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5795+0+0 (secure 0 0 0) 0x7ff59808c290 con 0x7ff5a0146d50 2024-09-16T01:37:22.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.953+0000 7ff5a4e80640 1 --2- 172.21.15.88:0/1660182938 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff58c06c2b0 0x7ff58c06e770 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff59c032000 tx=0x7ff59c009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:22.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:22.957+0000 7ff5957fa640 1 -- 172.21.15.88:0/1660182938 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff598019030 con 0x7ff5a0146d50 2024-09-16T01:37:23.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.122+0000 7ff5a70e4640 1 -- 172.21.15.88:0/1660182938 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_inject_journal_corrupt_dentry_first"} v 0) -- 0x7ff5a015c4a0 con 0x7ff5a0146d50 2024-09-16T01:37:23.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.125+0000 7ff5957fa640 1 -- 172.21.15.88:0/1660182938 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_inject_journal_corrupt_dentry_first"}]=0 v20) ==== 122+0+0 (secure 0 0 0) 0x7ff598010040 con 0x7ff5a0146d50 2024-09-16T01:37:23.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.128+0000 7ff5a70e4640 1 -- 172.21.15.88:0/1660182938 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff58c06c2b0 msgr2=0x7ff58c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:23.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.128+0000 7ff5a70e4640 1 --2- 172.21.15.88:0/1660182938 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff58c06c2b0 0x7ff58c06e770 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7ff59c032000 tx=0x7ff59c009030 comp rx=0 tx=0).stop 2024-09-16T01:37:23.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.128+0000 7ff5a70e4640 1 -- 172.21.15.88:0/1660182938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5a0146d50 msgr2=0x7ff5a01f7a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:23.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.128+0000 7ff5a70e4640 1 --2- 172.21.15.88:0/1660182938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5a0146d50 0x7ff5a01f7a90 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0075a00 tx=0x7ff59800f580 comp rx=0 tx=0).stop 2024-09-16T01:37:23.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.128+0000 7ff5a70e4640 1 -- 172.21.15.88:0/1660182938 shutdown_connections 2024-09-16T01:37:23.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.128+0000 7ff5a70e4640 1 --2- 172.21.15.88:0/1660182938 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff58c06c2b0 0x7ff58c06e770 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:23.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.128+0000 7ff5a70e4640 1 --2- 172.21.15.88:0/1660182938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5a0146d50 0x7ff5a01f7a90 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:23.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.128+0000 7ff5a70e4640 1 --2- 172.21.15.88:0/1660182938 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5a01f86e0 0x7ff5a0155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:23.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.128+0000 7ff5a70e4640 1 --2- 172.21.15.88:0/1660182938 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5880039b0 0x7ff5a0152a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:23.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.128+0000 7ff5a70e4640 1 -- 172.21.15.88:0/1660182938 >> 172.21.15.88:0/1660182938 conn(0x7ff5a00d2430 msgr2=0x7ff5a0158420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:23.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.128+0000 7ff5a70e4640 1 -- 172.21.15.88:0/1660182938 shutdown_connections 2024-09-16T01:37:23.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.128+0000 7ff5a70e4640 1 -- 172.21.15.88:0/1660182938 wait complete. 2024-09-16T01:37:23.140 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds freeze 4:0 false 2024-09-16T01:37:23.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.240+0000 7fc3edbaf640 1 Processor -- start 2024-09-16T01:37:23.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.240+0000 7fc3edbaf640 1 -- start start 2024-09-16T01:37:23.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.241+0000 7fc3edbaf640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc3e80b4300 0x7fc3e806bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:23.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.241+0000 7fc3edbaf640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc3e806c150 0x7fc3e80d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:23.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.241+0000 7fc3edbaf640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3e806c560 con 0x7fc3e815c520 2024-09-16T01:37:23.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.241+0000 7fc3edbaf640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc3e80cffe0 con 0x7fc3e80b4300 2024-09-16T01:37:23.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.241+0000 7fc3edbaf640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc3e80b4760 con 0x7fc3e806c150 2024-09-16T01:37:23.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.241+0000 7fc3e6ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc3e80b4300 0x7fc3e806bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:23.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.241+0000 7fc3e6ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc3e80b4300 0x7fc3e806bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53660/0 (socket says 172.21.15.88:53660) 2024-09-16T01:37:23.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.241+0000 7fc3e6ffd640 1 -- 172.21.15.88:0/1978228781 learned_addr learned my addr 172.21.15.88:0/1978228781 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:23.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.241+0000 7fc3e7fff640 1 --2- 172.21.15.88:0/1978228781 >> v2:172.21.15.154:3300/0 conn(0x7fc3e806c150 0x7fc3e80d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:23.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.242+0000 7fc3e6ffd640 1 -- 172.21.15.88:0/1978228781 >> v2:172.21.15.154:3300/0 conn(0x7fc3e806c150 msgr2=0x7fc3e80d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:23.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.242+0000 7fc3e6ffd640 1 --2- 172.21.15.88:0/1978228781 >> v2:172.21.15.154:3300/0 conn(0x7fc3e806c150 0x7fc3e80d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:23.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.242+0000 7fc3e6ffd640 1 -- 172.21.15.88:0/1978228781 >> v1:172.21.15.154:6789/0 conn(0x7fc3e815c520 legacy=0x7fc3e8070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:23.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.242+0000 7fc3e6ffd640 1 -- 172.21.15.88:0/1978228781 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e80cf3d0 con 0x7fc3e80b4300 2024-09-16T01:37:23.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.242+0000 7fc3e6ffd640 1 --2- 172.21.15.88:0/1978228781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3e80b4300 0x7fc3e806bc10 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fc3e80d0160 tx=0x7fc3d002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12bf47b64cfd7323 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:23.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.242+0000 7fc3e67fc640 1 -- 172.21.15.88:0/1978228781 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d0037070 con 0x7fc3e80b4300 2024-09-16T01:37:23.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.242+0000 7fc3e67fc640 1 -- 172.21.15.88:0/1978228781 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3d0032440 con 0x7fc3e80b4300 2024-09-16T01:37:23.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.242+0000 7fc3e67fc640 1 -- 172.21.15.88:0/1978228781 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d0032810 con 0x7fc3e80b4300 2024-09-16T01:37:23.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.243+0000 7fc3edbaf640 1 -- 172.21.15.88:0/1978228781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3e80b4300 msgr2=0x7fc3e806bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:23.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.243+0000 7fc3edbaf640 1 --2- 172.21.15.88:0/1978228781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3e80b4300 0x7fc3e806bc10 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fc3e80d0160 tx=0x7fc3d002a1e0 comp rx=0 tx=0).stop 2024-09-16T01:37:23.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.243+0000 7fc3edbaf640 1 -- 172.21.15.88:0/1978228781 shutdown_connections 2024-09-16T01:37:23.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.243+0000 7fc3edbaf640 1 --2- 172.21.15.88:0/1978228781 >> v2:172.21.15.154:3300/0 conn(0x7fc3e806c150 0x7fc3e80d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:23.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.243+0000 7fc3edbaf640 1 --2- 172.21.15.88:0/1978228781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3e80b4300 0x7fc3e806bc10 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:23.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.243+0000 7fc3edbaf640 1 -- 172.21.15.88:0/1978228781 >> 172.21.15.88:0/1978228781 conn(0x7fc3e80d1390 msgr2=0x7fc3e80d17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:23.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.243+0000 7fc3edbaf640 1 -- 172.21.15.88:0/1978228781 shutdown_connections 2024-09-16T01:37:23.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.243+0000 7fc3edbaf640 1 -- 172.21.15.88:0/1978228781 wait complete. 2024-09-16T01:37:23.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.244+0000 7fc3edbaf640 1 Processor -- start 2024-09-16T01:37:23.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.244+0000 7fc3edbaf640 1 -- start start 2024-09-16T01:37:23.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.245+0000 7fc3edbaf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3e806c150 0x7fc3e8159710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:23.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.245+0000 7fc3edbaf640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3e80b4300 0x7fc3e8149eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:23.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.245+0000 7fc3edbaf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3e815c520 0x7fc3e814ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:23.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.245+0000 7fc3edbaf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e80cfc70 con 0x7fc3e815c520 2024-09-16T01:37:23.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.245+0000 7fc3edbaf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e80cf990 con 0x7fc3e806c150 2024-09-16T01:37:23.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.245+0000 7fc3edbaf640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc3e80cdb00 con 0x7fc3e80b4300 2024-09-16T01:37:23.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.245+0000 7fc3e7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3e815c520 0x7fc3e814ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:23.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.245+0000 7fc3e7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3e815c520 0x7fc3e814ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53670/0 (socket says 172.21.15.88:53670) 2024-09-16T01:37:23.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.245+0000 7fc3e7fff640 1 -- 172.21.15.88:0/3343800925 learned_addr learned my addr 172.21.15.88:0/3343800925 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:23.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.245+0000 7fc3e77fe640 1 --2- 172.21.15.88:0/3343800925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3e806c150 0x7fc3e8159710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:23.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.245+0000 7fc3e6ffd640 1 --2- 172.21.15.88:0/3343800925 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3e80b4300 0x7fc3e8149eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:23.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.245+0000 7fc3e7fff640 1 -- 172.21.15.88:0/3343800925 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3e80b4300 msgr2=0x7fc3e8149eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:23.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.245+0000 7fc3e7fff640 1 --2- 172.21.15.88:0/3343800925 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3e80b4300 0x7fc3e8149eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:23.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.245+0000 7fc3e7fff640 1 -- 172.21.15.88:0/3343800925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3e806c150 msgr2=0x7fc3e8159710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:23.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.245+0000 7fc3e7fff640 1 --2- 172.21.15.88:0/3343800925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3e806c150 0x7fc3e8159710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:23.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.245+0000 7fc3e7fff640 1 -- 172.21.15.88:0/3343800925 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e8159c50 con 0x7fc3e815c520 2024-09-16T01:37:23.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.246+0000 7fc3e6ffd640 1 --2- 172.21.15.88:0/3343800925 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3e80b4300 0x7fc3e8149eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:37:23.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.246+0000 7fc3e7fff640 1 --2- 172.21.15.88:0/3343800925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3e815c520 0x7fc3e814ca60 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fc3e000e820 tx=0x7fc3e000ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:23.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.246+0000 7fc3dffff640 1 -- 172.21.15.88:0/3343800925 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e0017070 con 0x7fc3e815c520 2024-09-16T01:37:23.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.246+0000 7fc3dffff640 1 -- 172.21.15.88:0/3343800925 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3e000ee70 con 0x7fc3e815c520 2024-09-16T01:37:23.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.246+0000 7fc3dffff640 1 -- 172.21.15.88:0/3343800925 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e000cb60 con 0x7fc3e815c520 2024-09-16T01:37:23.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.246+0000 7fc3edbaf640 1 -- 172.21.15.88:0/3343800925 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3e81f2380 con 0x7fc3e815c520 2024-09-16T01:37:23.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.246+0000 7fc3edbaf640 1 -- 172.21.15.88:0/3343800925 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3e81f2bc0 con 0x7fc3e815c520 2024-09-16T01:37:23.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.247+0000 7fc3edbaf640 1 -- 172.21.15.88:0/3343800925 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3e806bc10 con 0x7fc3e815c520 2024-09-16T01:37:23.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.248+0000 7fc3dffff640 1 -- 172.21.15.88:0/3343800925 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc3e001a070 con 0x7fc3e815c520 2024-09-16T01:37:23.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.248+0000 7fc3dffff640 1 --2- 172.21.15.88:0/3343800925 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc3c806c2b0 0x7fc3c806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:23.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.248+0000 7fc3dffff640 1 -- 172.21.15.88:0/3343800925 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5795+0+0 (secure 0 0 0) 0x7fc3e008b1d0 con 0x7fc3e815c520 2024-09-16T01:37:23.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.248+0000 7fc3e77fe640 1 --2- 172.21.15.88:0/3343800925 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc3c806c2b0 0x7fc3c806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:23.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.248+0000 7fc3e77fe640 1 --2- 172.21.15.88:0/3343800925 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc3c806c2b0 0x7fc3c806e770 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8001f10 tx=0x7fc3d8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:23.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.252+0000 7fc3dffff640 1 -- 172.21.15.88:0/3343800925 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc3e81f2380 con 0x7fc3e815c520 2024-09-16T01:37:23.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:23.484+0000 7fc3edbaf640 1 -- 172.21.15.88:0/3343800925 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds freeze", "role_or_gid": "4:0", "val": "false"} v 0) -- 0x7fc3e815dbd0 con 0x7fc3e815c520 2024-09-16T01:37:24.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.237+0000 7fc3dffff640 1 -- 172.21.15.88:0/3343800925 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds freeze", "role_or_gid": "4:0", "val": "false"}]=0 unfreezing mds.6054 v47) ==== 115+0+0 (secure 0 0 0) 0x7fc3e0059320 con 0x7fc3e815c520 2024-09-16T01:37:24.239 INFO:teuthology.orchestra.run.smithi088.stderr:unfreezing mds.6054 2024-09-16T01:37:24.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.239+0000 7fc3edbaf640 1 -- 172.21.15.88:0/3343800925 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc3c806c2b0 msgr2=0x7fc3c806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:24.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.239+0000 7fc3edbaf640 1 --2- 172.21.15.88:0/3343800925 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc3c806c2b0 0x7fc3c806e770 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8001f10 tx=0x7fc3d8009040 comp rx=0 tx=0).stop 2024-09-16T01:37:24.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.239+0000 7fc3edbaf640 1 -- 172.21.15.88:0/3343800925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3e815c520 msgr2=0x7fc3e814ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:24.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.239+0000 7fc3edbaf640 1 --2- 172.21.15.88:0/3343800925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3e815c520 0x7fc3e814ca60 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fc3e000e820 tx=0x7fc3e000ec20 comp rx=0 tx=0).stop 2024-09-16T01:37:24.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.240+0000 7fc3edbaf640 1 -- 172.21.15.88:0/3343800925 shutdown_connections 2024-09-16T01:37:24.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.240+0000 7fc3edbaf640 1 --2- 172.21.15.88:0/3343800925 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc3c806c2b0 0x7fc3c806e770 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:24.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.240+0000 7fc3edbaf640 1 --2- 172.21.15.88:0/3343800925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3e815c520 0x7fc3e814ca60 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:24.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.240+0000 7fc3edbaf640 1 --2- 172.21.15.88:0/3343800925 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3e80b4300 0x7fc3e8149eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:24.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.240+0000 7fc3edbaf640 1 --2- 172.21.15.88:0/3343800925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3e806c150 0x7fc3e8159710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:24.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.240+0000 7fc3edbaf640 1 -- 172.21.15.88:0/3343800925 >> 172.21.15.88:0/3343800925 conn(0x7fc3e80d1390 msgr2=0x7fc3e80b0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:24.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.240+0000 7fc3edbaf640 1 -- 172.21.15.88:0/3343800925 shutdown_connections 2024-09-16T01:37:24.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.240+0000 7fc3edbaf640 1 -- 172.21.15.88:0/3343800925 wait complete. 2024-09-16T01:37:24.256 DEBUG:teuthology.orchestra.run.smithi088:> sudo sysctl -n kernel.core_pattern 2024-09-16T01:37:24.279 INFO:teuthology.orchestra.run.smithi088.stdout:/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T01:37:24.280 INFO:tasks.cephfs.cephfs_test_case:Clearing core from directory: /home/ubuntu/cephtest/archive/coredump 2024-09-16T01:37:24.280 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest/archive/coredump && sudo ls | sudo xargs file 2024-09-16T01:37:24.347 INFO:teuthology.orchestra.run.smithi088.stdout:1726450621.53729.core: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph-mds -f --cluster ceph -i a', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/proc/self/exe', platform: 'x86_64' 2024-09-16T01:37:24.349 INFO:tasks.cephfs.cephfs_test_case:Enumerated cores: ['1726450621.53729.core'] 2024-09-16T01:37:24.349 INFO:tasks.cephfs.cephfs_test_case:Found core file 1726450621.53729.core, deleting it 2024-09-16T01:37:24.349 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest/archive/coredump && sudo rm -f 1726450621.53729.core 2024-09-16T01:37:24.416 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-16T01:37:24.416 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-16T01:37:24.417 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-16T01:37:24.417 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:37:24.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:37:24.417 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a' 2024-09-16T01:37:24.418 INFO:tasks.ceph.mds.a:Stopped 2024-09-16T01:37:24.419 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-16T01:37:24.458 INFO:tasks.ceph.mds.a:Started 2024-09-16T01:37:24.458 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:37:24.531 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:37:24.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.557+0000 7f2472d13640 1 Processor -- start 2024-09-16T01:37:24.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.557+0000 7f2472d13640 1 -- start start 2024-09-16T01:37:24.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f2472d13640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f246c0cf6b0 0x7f246c1629f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:24.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f2472d13640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f246c162f30 0x7f246c0b7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:24.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f2472d13640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f246c0cfc20 con 0x7f246c0b63d0 2024-09-16T01:37:24.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f2472d13640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f246c0d40c0 con 0x7f246c162f30 2024-09-16T01:37:24.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f2472d13640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f246c0d2ed0 con 0x7f246c0cf6b0 2024-09-16T01:37:24.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f2470aaf640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f246c0b63d0 0x7f246c0d39b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58864/0 (socket says 172.21.15.88:58864) 2024-09-16T01:37:24.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f2470aaf640 1 -- 172.21.15.88:0/3828996994 learned_addr learned my addr 172.21.15.88:0/3828996994 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:37:24.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f24712b0640 1 --2- 172.21.15.88:0/3828996994 >> v2:172.21.15.154:3300/0 conn(0x7f246c162f30 0x7f246c0b7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:24.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f246bfff640 1 --2- 172.21.15.88:0/3828996994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f246c0cf6b0 0x7f246c1629f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:24.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f246b7fe640 1 -- 172.21.15.88:0/3828996994 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2322758355 0 0) 0x7f246c0cfc20 con 0x7f246c0b63d0 2024-09-16T01:37:24.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f246b7fe640 1 -- 172.21.15.88:0/3828996994 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f245c003520 con 0x7f246c0b63d0 2024-09-16T01:37:24.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f246b7fe640 1 -- 172.21.15.88:0/3828996994 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1617333635 0 0) 0x7f245c003520 con 0x7f246c0b63d0 2024-09-16T01:37:24.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f246b7fe640 1 -- 172.21.15.88:0/3828996994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f246c0cf6b0 msgr2=0x7f246c1629f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:24.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f246b7fe640 1 --2- 172.21.15.88:0/3828996994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f246c0cf6b0 0x7f246c1629f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:24.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f246b7fe640 1 -- 172.21.15.88:0/3828996994 >> v2:172.21.15.154:3300/0 conn(0x7f246c162f30 msgr2=0x7f246c0b7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:24.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f246b7fe640 1 --2- 172.21.15.88:0/3828996994 >> v2:172.21.15.154:3300/0 conn(0x7f246c162f30 0x7f246c0b7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:24.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f246b7fe640 1 -- 172.21.15.88:0/3828996994 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f246c0d31b0 con 0x7f246c0b63d0 2024-09-16T01:37:24.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f246b7fe640 1 -- 172.21.15.88:0/3828996994 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f2460001670 con 0x7f246c0b63d0 2024-09-16T01:37:24.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f246b7fe640 1 -- 172.21.15.88:0/3828996994 >> v1:172.21.15.88:6789/0 conn(0x7f246c0b63d0 legacy=0x7f246c0d39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:24.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f246b7fe640 1 --2- 172.21.15.88:0/3828996994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f246c0cf6b0 0x7f245c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:24.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f246b7fe640 1 --2- 172.21.15.88:0/3828996994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f245c003f30 0x7f245c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:24.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f246b7fe640 1 --2- 172.21.15.88:0/3828996994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f245c004d40 0x7f245c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:24.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f246b7fe640 1 -- 172.21.15.88:0/3828996994 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f246c0d40c0 con 0x7f245c003f30 2024-09-16T01:37:24.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f246b7fe640 1 -- 172.21.15.88:0/3828996994 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f246c0d2ed0 con 0x7f245c004d40 2024-09-16T01:37:24.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f246b7fe640 1 -- 172.21.15.88:0/3828996994 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f245c00b9c0 con 0x7f246c0cf6b0 2024-09-16T01:37:24.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f246bfff640 1 --2- 172.21.15.88:0/3828996994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f245c003f30 0x7f245c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:24.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f2470aaf640 1 --2- 172.21.15.88:0/3828996994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f246c0cf6b0 0x7f245c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:24.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.558+0000 7f24712b0640 1 --2- 172.21.15.88:0/3828996994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f245c004d40 0x7f245c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:24.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.559+0000 7f246bfff640 1 -- 172.21.15.88:0/3828996994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f246c0cf6b0 msgr2=0x7f245c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:24.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.559+0000 7f246bfff640 1 --2- 172.21.15.88:0/3828996994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f246c0cf6b0 0x7f245c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:24.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.559+0000 7f246bfff640 1 -- 172.21.15.88:0/3828996994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f245c004d40 msgr2=0x7f245c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:24.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.559+0000 7f246bfff640 1 --2- 172.21.15.88:0/3828996994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f245c004d40 0x7f245c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:24.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.559+0000 7f246bfff640 1 -- 172.21.15.88:0/3828996994 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f245c003520 con 0x7f245c003f30 2024-09-16T01:37:24.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.559+0000 7f246bfff640 1 --2- 172.21.15.88:0/3828996994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f245c003f30 0x7f245c004800 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f2458009ac0 tx=0x7f245c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91a827df27050469 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:24.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.559+0000 7f246b7fe640 1 -- 172.21.15.88:0/3828996994 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2458036070 con 0x7f245c003f30 2024-09-16T01:37:24.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.559+0000 7f246b7fe640 1 -- 172.21.15.88:0/3828996994 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2458031400 con 0x7f245c003f30 2024-09-16T01:37:24.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.559+0000 7f246b7fe640 1 -- 172.21.15.88:0/3828996994 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2458009690 con 0x7f245c003f30 2024-09-16T01:37:24.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.560+0000 7f2472d13640 1 -- 172.21.15.88:0/3828996994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f245c003f30 msgr2=0x7f245c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:24.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.560+0000 7f2472d13640 1 --2- 172.21.15.88:0/3828996994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f245c003f30 0x7f245c004800 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f2458009ac0 tx=0x7f245c005830 comp rx=0 tx=0).stop 2024-09-16T01:37:24.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.560+0000 7f24712b0640 1 -- 172.21.15.88:0/3828996994 reap_dead start 2024-09-16T01:37:24.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.560+0000 7f2472d13640 1 -- 172.21.15.88:0/3828996994 shutdown_connections 2024-09-16T01:37:24.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.560+0000 7f2472d13640 1 -- 172.21.15.88:0/3828996994 >> 172.21.15.88:0/3828996994 conn(0x7f246c0d1160 msgr2=0x7f246c0d1570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:24.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.560+0000 7f2472d13640 1 -- 172.21.15.88:0/3828996994 shutdown_connections 2024-09-16T01:37:24.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.560+0000 7f2472d13640 1 -- 172.21.15.88:0/3828996994 wait complete. 2024-09-16T01:37:24.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.561+0000 7f2472d13640 1 Processor -- start 2024-09-16T01:37:24.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.561+0000 7f2472d13640 1 -- start start 2024-09-16T01:37:24.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.562+0000 7f2472d13640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f245c003f30 0x7f246c14caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:24.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.562+0000 7f2472d13640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f246c0cafb0 0x7f246c0ca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:24.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.562+0000 7f2472d13640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f246c15f790 0x7f246c15fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:24.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.562+0000 7f2472d13640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f246c0cb5c0 con 0x7f246c0cafb0 2024-09-16T01:37:24.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.562+0000 7f2472d13640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f246c0cb440 con 0x7f246c15f790 2024-09-16T01:37:24.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.562+0000 7f2472d13640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f246c0d1c80 con 0x7f245c003f30 2024-09-16T01:37:24.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.562+0000 7f246bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f246c0cafb0 0x7f246c0ca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:24.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.562+0000 7f246bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f246c0cafb0 0x7f246c0ca5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53718/0 (socket says 172.21.15.88:53718) 2024-09-16T01:37:24.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.562+0000 7f246bfff640 1 -- 172.21.15.88:0/623502840 learned_addr learned my addr 172.21.15.88:0/623502840 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:24.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.562+0000 7f24712b0640 1 --2- 172.21.15.88:0/623502840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f246c15f790 0x7f246c15fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:24.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.562+0000 7f2470aaf640 1 --2- 172.21.15.88:0/623502840 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f245c003f30 0x7f246c14caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:24.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.563+0000 7f246bfff640 1 -- 172.21.15.88:0/623502840 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f245c003f30 msgr2=0x7f246c14caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:24.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.563+0000 7f246bfff640 1 --2- 172.21.15.88:0/623502840 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f245c003f30 0x7f246c14caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:24.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.563+0000 7f246bfff640 1 -- 172.21.15.88:0/623502840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f246c15f790 msgr2=0x7f246c15fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:24.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.563+0000 7f246bfff640 1 --2- 172.21.15.88:0/623502840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f246c15f790 0x7f246c15fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:24.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.563+0000 7f246bfff640 1 -- 172.21.15.88:0/623502840 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f246c0c8cc0 con 0x7f246c0cafb0 2024-09-16T01:37:24.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.563+0000 7f246bfff640 1 --2- 172.21.15.88:0/623502840 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f246c0cafb0 0x7f246c0ca5b0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f2458000c00 tx=0x7f2458004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:24.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.563+0000 7f24697fa640 1 -- 172.21.15.88:0/623502840 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2458040020 con 0x7f246c0cafb0 2024-09-16T01:37:24.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.563+0000 7f24697fa640 1 -- 172.21.15.88:0/623502840 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2458003c60 con 0x7f246c0cafb0 2024-09-16T01:37:24.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.563+0000 7f24697fa640 1 -- 172.21.15.88:0/623502840 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2458036040 con 0x7f246c0cafb0 2024-09-16T01:37:24.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.563+0000 7f2472d13640 1 -- 172.21.15.88:0/623502840 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f246c0c8fa0 con 0x7f246c0cafb0 2024-09-16T01:37:24.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.563+0000 7f2472d13640 1 -- 172.21.15.88:0/623502840 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f246c0c9560 con 0x7f246c0cafb0 2024-09-16T01:37:24.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.564+0000 7f2472d13640 1 -- 172.21.15.88:0/623502840 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f246c0b67b0 con 0x7f246c0cafb0 2024-09-16T01:37:24.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.564+0000 7f24697fa640 1 -- 172.21.15.88:0/623502840 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2458043020 con 0x7f246c0cafb0 2024-09-16T01:37:24.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.565+0000 7f24697fa640 1 --2- 172.21.15.88:0/623502840 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f245006c2b0 0x7f245006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:24.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.565+0000 7f24697fa640 1 -- 172.21.15.88:0/623502840 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5881+0+0 (secure 0 0 0) 0x7f24580ab820 con 0x7f246c0cafb0 2024-09-16T01:37:24.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.565+0000 7f2470aaf640 1 --2- 172.21.15.88:0/623502840 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f245006c2b0 0x7f245006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:24.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.565+0000 7f2470aaf640 1 --2- 172.21.15.88:0/623502840 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f245006c2b0 0x7f245006e770 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f24600015d0 tx=0x7f2460009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:24.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.569+0000 7f24697fa640 1 -- 172.21.15.88:0/623502840 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f245804a030 con 0x7f246c0cafb0 2024-09-16T01:37:24.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.769+0000 7f2472d13640 1 -- 172.21.15.88:0/623502840 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f246c0cf280 con 0x7f246c0cafb0 2024-09-16T01:37:24.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.769+0000 7f24697fa640 1 -- 172.21.15.88:0/623502840 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+4392 (secure 0 0 0) 0x7f24580044e0 con 0x7f246c0cafb0 2024-09-16T01:37:24.771 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:37:24.771 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":48,"btime":"2024-09-16T01:37:24:240944+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/581241194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":581241194},{"type":"v1","addr":"172.21.15.154:6834","nonce":581241194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2104508521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2104508521},{"type":"v1","addr":"172.21.15.154:6835","nonce":2104508521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:39.786438+0000","modified":"2024-09-16T01:37:24.240941+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5859},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5859":{"gid":5859,"name":"c","rank":0,"incarnation":48,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6836/998283126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":998283126},{"type":"v1","addr":"172.21.15.88:6836","nonce":998283126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":4}]} 2024-09-16T01:37:24.771 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 48 2024-09-16T01:37:24.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.771+0000 7f2472d13640 1 -- 172.21.15.88:0/623502840 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f245006c2b0 msgr2=0x7f245006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:24.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.771+0000 7f2472d13640 1 --2- 172.21.15.88:0/623502840 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f245006c2b0 0x7f245006e770 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f24600015d0 tx=0x7f2460009040 comp rx=0 tx=0).stop 2024-09-16T01:37:24.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.771+0000 7f2472d13640 1 -- 172.21.15.88:0/623502840 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f246c0cafb0 msgr2=0x7f246c0ca5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:24.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.771+0000 7f2472d13640 1 --2- 172.21.15.88:0/623502840 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f246c0cafb0 0x7f246c0ca5b0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f2458000c00 tx=0x7f2458004730 comp rx=0 tx=0).stop 2024-09-16T01:37:24.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.771+0000 7f2472d13640 1 -- 172.21.15.88:0/623502840 shutdown_connections 2024-09-16T01:37:24.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.771+0000 7f2472d13640 1 --2- 172.21.15.88:0/623502840 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f245006c2b0 0x7f245006e770 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:24.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.771+0000 7f2472d13640 1 --2- 172.21.15.88:0/623502840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f246c15f790 0x7f246c15fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:24.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.771+0000 7f2472d13640 1 --2- 172.21.15.88:0/623502840 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f246c0cafb0 0x7f246c0ca5b0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:24.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.771+0000 7f2472d13640 1 --2- 172.21.15.88:0/623502840 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f245c003f30 0x7f246c14caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:24.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.771+0000 7f2472d13640 1 -- 172.21.15.88:0/623502840 >> 172.21.15.88:0/623502840 conn(0x7f246c0d1160 msgr2=0x7f246c0b7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:24.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.771+0000 7f2472d13640 1 -- 172.21.15.88:0/623502840 shutdown_connections 2024-09-16T01:37:24.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:24.771+0000 7f2472d13640 1 -- 172.21.15.88:0/623502840 wait complete. 2024-09-16T01:37:25.186 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 48, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:36:39.786438+0000', 'modified': '2024-09-16T01:37:24.240941+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 60, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5859}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5859': {'gid': 5859, 'name': 'c', 'rank': 0, 'incarnation': 48, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.88:6836/998283126', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 998283126}, {'type': 'v1', 'addr': '172.21.15.88:6836', 'nonce': 998283126}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-16T01:37:25.186 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5859:up:replay 2024-09-16T01:37:25.736 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-09-16T01:37:26.187 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:37:26.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.299+0000 7fa214d0a640 1 Processor -- start 2024-09-16T01:37:26.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.299+0000 7fa214d0a640 1 -- start start 2024-09-16T01:37:26.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.300+0000 7fa214d0a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa21015e540 0x7fa2100b52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:26.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.300+0000 7fa214d0a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa2100b01f0 0x7fa2101620e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:26.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.300+0000 7fa214d0a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa210073180 0x7fa2100735d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:26.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.300+0000 7fa214d0a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa210073b10 con 0x7fa2100b01f0 2024-09-16T01:37:26.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.300+0000 7fa214d0a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa210163010 con 0x7fa210073180 2024-09-16T01:37:26.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.300+0000 7fa214d0a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa2100b5800 con 0x7fa21015e540 2024-09-16T01:37:26.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.300+0000 7fa207fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa2100b01f0 0x7fa2101620e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:26.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.300+0000 7fa207fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa2100b01f0 0x7fa2101620e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53724/0 (socket says 172.21.15.88:53724) 2024-09-16T01:37:26.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.300+0000 7fa207fff640 1 -- 172.21.15.88:0/3481591221 learned_addr learned my addr 172.21.15.88:0/3481591221 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:26.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.300+0000 7fa20e59c640 1 --2- 172.21.15.88:0/3481591221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa21015e540 0x7fa2100b52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:26.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.300+0000 7fa20ed9d640 1 --2- 172.21.15.88:0/3481591221 >> v2:172.21.15.154:3300/0 conn(0x7fa210073180 0x7fa2100735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:26.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.300+0000 7fa207fff640 1 -- 172.21.15.88:0/3481591221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa21015e540 msgr2=0x7fa2100b52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:26.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.300+0000 7fa207fff640 1 --2- 172.21.15.88:0/3481591221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa21015e540 0x7fa2100b52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:26.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.300+0000 7fa207fff640 1 -- 172.21.15.88:0/3481591221 >> v2:172.21.15.154:3300/0 conn(0x7fa210073180 msgr2=0x7fa2100735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:26.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.300+0000 7fa207fff640 1 --2- 172.21.15.88:0/3481591221 >> v2:172.21.15.154:3300/0 conn(0x7fa210073180 0x7fa2100735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:26.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.300+0000 7fa207fff640 1 -- 172.21.15.88:0/3481591221 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa21015e920 con 0x7fa2100b01f0 2024-09-16T01:37:26.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.301+0000 7fa207fff640 1 --2- 172.21.15.88:0/3481591221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2100b01f0 0x7fa2101620e0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fa2100b6e80 tx=0x7fa1f802a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af5beb2e39ed465c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:26.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.301+0000 7fa20dd9b640 1 -- 172.21.15.88:0/3481591221 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f8037070 con 0x7fa2100b01f0 2024-09-16T01:37:26.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.301+0000 7fa20dd9b640 1 -- 172.21.15.88:0/3481591221 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1f802ae00 con 0x7fa2100b01f0 2024-09-16T01:37:26.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.301+0000 7fa20dd9b640 1 -- 172.21.15.88:0/3481591221 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f8032660 con 0x7fa2100b01f0 2024-09-16T01:37:26.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.302+0000 7fa214d0a640 1 -- 172.21.15.88:0/3481591221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2100b01f0 msgr2=0x7fa2101620e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:26.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.302+0000 7fa214d0a640 1 --2- 172.21.15.88:0/3481591221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2100b01f0 0x7fa2101620e0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fa2100b6e80 tx=0x7fa1f802a0b0 comp rx=0 tx=0).stop 2024-09-16T01:37:26.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.302+0000 7fa214d0a640 1 -- 172.21.15.88:0/3481591221 shutdown_connections 2024-09-16T01:37:26.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.302+0000 7fa214d0a640 1 --2- 172.21.15.88:0/3481591221 >> v2:172.21.15.154:3300/0 conn(0x7fa210073180 0x7fa2100735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:26.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.302+0000 7fa214d0a640 1 --2- 172.21.15.88:0/3481591221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2100b01f0 0x7fa2101620e0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:26.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.302+0000 7fa214d0a640 1 --2- 172.21.15.88:0/3481591221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa21015e540 0x7fa2100b52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:26.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.302+0000 7fa214d0a640 1 -- 172.21.15.88:0/3481591221 >> 172.21.15.88:0/3481591221 conn(0x7fa21006bc10 msgr2=0x7fa21006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:26.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.302+0000 7fa214d0a640 1 -- 172.21.15.88:0/3481591221 shutdown_connections 2024-09-16T01:37:26.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.302+0000 7fa214d0a640 1 -- 172.21.15.88:0/3481591221 wait complete. 2024-09-16T01:37:26.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.303+0000 7fa214d0a640 1 Processor -- start 2024-09-16T01:37:26.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.303+0000 7fa214d0a640 1 -- start start 2024-09-16T01:37:26.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.304+0000 7fa214d0a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa210073180 0x7fa2101f4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:26.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.304+0000 7fa214d0a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2100b01f0 0x7fa21014c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:26.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.304+0000 7fa214d0a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa21015e540 0x7fa21015f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:26.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.304+0000 7fa214d0a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa2100ae9c0 con 0x7fa210073180 2024-09-16T01:37:26.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.304+0000 7fa214d0a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa210162d30 con 0x7fa2100b01f0 2024-09-16T01:37:26.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.304+0000 7fa214d0a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa2100aeca0 con 0x7fa21015e540 2024-09-16T01:37:26.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.304+0000 7fa20e59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa210073180 0x7fa2101f4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:26.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.304+0000 7fa20e59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa210073180 0x7fa2101f4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53740/0 (socket says 172.21.15.88:53740) 2024-09-16T01:37:26.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.304+0000 7fa20e59c640 1 -- 172.21.15.88:0/489283261 learned_addr learned my addr 172.21.15.88:0/489283261 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:26.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.304+0000 7fa20e59c640 1 -- 172.21.15.88:0/489283261 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa21015e540 msgr2=0x7fa21015f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:26.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.304+0000 7fa20e59c640 1 --2- 172.21.15.88:0/489283261 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa21015e540 0x7fa21015f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:26.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.304+0000 7fa20e59c640 1 -- 172.21.15.88:0/489283261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2100b01f0 msgr2=0x7fa21014c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:26.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.304+0000 7fa20e59c640 1 --2- 172.21.15.88:0/489283261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2100b01f0 0x7fa21014c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:26.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.304+0000 7fa20e59c640 1 -- 172.21.15.88:0/489283261 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2101601f0 con 0x7fa210073180 2024-09-16T01:37:26.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.305+0000 7fa20e59c640 1 --2- 172.21.15.88:0/489283261 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa210073180 0x7fa2101f4440 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fa20000d5f0 tx=0x7fa20000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:26.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.305+0000 7fa206ffd640 1 -- 172.21.15.88:0/489283261 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2000125f0 con 0x7fa210073180 2024-09-16T01:37:26.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.305+0000 7fa206ffd640 1 -- 172.21.15.88:0/489283261 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa200019070 con 0x7fa210073180 2024-09-16T01:37:26.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.305+0000 7fa214d0a640 1 -- 172.21.15.88:0/489283261 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa21015c590 con 0x7fa210073180 2024-09-16T01:37:26.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.305+0000 7fa206ffd640 1 -- 172.21.15.88:0/489283261 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2000129d0 con 0x7fa210073180 2024-09-16T01:37:26.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.305+0000 7fa214d0a640 1 -- 172.21.15.88:0/489283261 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa21015cb50 con 0x7fa210073180 2024-09-16T01:37:26.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.306+0000 7fa214d0a640 1 -- 172.21.15.88:0/489283261 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2100735d0 con 0x7fa210073180 2024-09-16T01:37:26.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.306+0000 7fa206ffd640 1 -- 172.21.15.88:0/489283261 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa200012b70 con 0x7fa210073180 2024-09-16T01:37:26.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.307+0000 7fa206ffd640 1 --2- 172.21.15.88:0/489283261 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa1f006c2b0 0x7fa1f006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:26.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.307+0000 7fa206ffd640 1 -- 172.21.15.88:0/489283261 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5881+0+0 (secure 0 0 0) 0x7fa20008b800 con 0x7fa210073180 2024-09-16T01:37:26.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.307+0000 7fa207fff640 1 --2- 172.21.15.88:0/489283261 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa1f006c2b0 0x7fa1f006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:26.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.307+0000 7fa207fff640 1 --2- 172.21.15.88:0/489283261 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa1f006c2b0 0x7fa1f006e770 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8034000 tx=0x7fa1f802c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:26.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.311+0000 7fa206ffd640 1 -- 172.21.15.88:0/489283261 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa200059f50 con 0x7fa210073180 2024-09-16T01:37:26.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.490+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 unknown :-1 s=BANNER_CONNECTING pgs=14 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:26.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.491+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 crc :-1 s=SESSION_RECONNECTING pgs=14 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_session_reset received session reset full=1 2024-09-16T01:37:26.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.491+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 crc :-1 s=SESSION_RECONNECTING pgs=14 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).reset_session 2024-09-16T01:37:26.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.491+0000 7fc9067fc640 0 client.6141 ms_handle_remote_reset on v2:172.21.15.88:6835/4130074102 2024-09-16T01:37:26.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.491+0000 7fc9267fc640 1 -- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 msgr2=0x55e7d13cdc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2024-09-16T01:37:26.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.491+0000 7fc9267fc640 1 -- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 msgr2=0x55e7d13cdc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T01:37:26.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.491+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T01:37:26.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.491+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-16T01:37:26.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.510+0000 7fa214d0a640 1 -- 172.21.15.88:0/489283261 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa210157eb0 con 0x7fa210073180 2024-09-16T01:37:26.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.510+0000 7fa206ffd640 1 -- 172.21.15.88:0/489283261 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+5218 (secure 0 0 0) 0x7fa200059850 con 0x7fa210073180 2024-09-16T01:37:26.512 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:37:26.513 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":50,"btime":"2024-09-16T01:37:26:250328+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/581241194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":581241194},{"type":"v1","addr":"172.21.15.154:6834","nonce":581241194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2104508521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2104508521},{"type":"v1","addr":"172.21.15.154:6835","nonce":2104508521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":6195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2920926674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2920926674},{"type":"v1","addr":"172.21.15.88:6837","nonce":2920926674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:39.786438+0000","modified":"2024-09-16T01:37:25.251856+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5859},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5859":{"gid":5859,"name":"c","rank":0,"incarnation":48,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.88:6836/998283126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":998283126},{"type":"v1","addr":"172.21.15.88:6836","nonce":998283126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":4}]} 2024-09-16T01:37:26.515 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 50 2024-09-16T01:37:26.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.512+0000 7fa214d0a640 1 -- 172.21.15.88:0/489283261 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa1f006c2b0 msgr2=0x7fa1f006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:26.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.512+0000 7fa214d0a640 1 --2- 172.21.15.88:0/489283261 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa1f006c2b0 0x7fa1f006e770 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8034000 tx=0x7fa1f802c000 comp rx=0 tx=0).stop 2024-09-16T01:37:26.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.512+0000 7fa214d0a640 1 -- 172.21.15.88:0/489283261 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa210073180 msgr2=0x7fa2101f4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:26.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.512+0000 7fa214d0a640 1 --2- 172.21.15.88:0/489283261 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa210073180 0x7fa2101f4440 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fa20000d5f0 tx=0x7fa20000dab0 comp rx=0 tx=0).stop 2024-09-16T01:37:26.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.512+0000 7fa214d0a640 1 -- 172.21.15.88:0/489283261 shutdown_connections 2024-09-16T01:37:26.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.512+0000 7fa214d0a640 1 --2- 172.21.15.88:0/489283261 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa1f006c2b0 0x7fa1f006e770 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:26.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.512+0000 7fa214d0a640 1 --2- 172.21.15.88:0/489283261 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa21015e540 0x7fa21015f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:26.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.512+0000 7fa214d0a640 1 --2- 172.21.15.88:0/489283261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2100b01f0 0x7fa21014c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:26.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.512+0000 7fa214d0a640 1 --2- 172.21.15.88:0/489283261 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa210073180 0x7fa2101f4440 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:26.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.512+0000 7fa214d0a640 1 -- 172.21.15.88:0/489283261 >> 172.21.15.88:0/489283261 conn(0x7fa21006bc10 msgr2=0x7fa210072a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:26.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.512+0000 7fa214d0a640 1 -- 172.21.15.88:0/489283261 shutdown_connections 2024-09-16T01:37:26.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:26.512+0000 7fa214d0a640 1 -- 172.21.15.88:0/489283261 wait complete. 2024-09-16T01:37:26.929 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 50, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:36:39.786438+0000', 'modified': '2024-09-16T01:37:25.251856+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 60, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5859}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5859': {'gid': 5859, 'name': 'c', 'rank': 0, 'incarnation': 48, 'state': 'up:rejoin', 'state_seq': 15, 'addr': '172.21.15.88:6836/998283126', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 998283126}, {'type': 'v1', 'addr': '172.21.15.88:6836', 'nonce': 998283126}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-16T01:37:26.930 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5859:up:rejoin 2024-09-16T01:37:27.930 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:37:28.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.039+0000 7fb63a93b640 1 Processor -- start 2024-09-16T01:37:28.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.039+0000 7fb63a93b640 1 -- start start 2024-09-16T01:37:28.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.040+0000 7fb63a93b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6340b4920 0x7fb63415b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.040+0000 7fb63a93b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb6340b4d00 0x7fb6340b5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.040+0000 7fb63a93b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb6340cd780 0x7fb6340cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.040+0000 7fb63a93b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb6340cf100 con 0x7fb6340cd780 2024-09-16T01:37:28.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.040+0000 7fb63a93b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb6340ce4f0 con 0x7fb6340b4d00 2024-09-16T01:37:28.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.040+0000 7fb63a93b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb6340ce7d0 con 0x7fb6340b4920 2024-09-16T01:37:28.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.040+0000 7fb638ed8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb6340cd780 0x7fb6340cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.040+0000 7fb638ed8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb6340cd780 0x7fb6340cdbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53750/0 (socket says 172.21.15.88:53750) 2024-09-16T01:37:28.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.040+0000 7fb638ed8640 1 -- 172.21.15.88:0/1957417877 learned_addr learned my addr 172.21.15.88:0/1957417877 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:28.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.041+0000 7fb6337fe640 1 --2- 172.21.15.88:0/1957417877 >> v2:172.21.15.154:3300/0 conn(0x7fb6340b4d00 0x7fb6340b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.041+0000 7fb633fff640 1 --2- 172.21.15.88:0/1957417877 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6340b4920 0x7fb63415b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.041+0000 7fb638ed8640 1 -- 172.21.15.88:0/1957417877 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6340b4920 msgr2=0x7fb63415b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:28.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.041+0000 7fb638ed8640 1 --2- 172.21.15.88:0/1957417877 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6340b4920 0x7fb63415b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.041+0000 7fb638ed8640 1 -- 172.21.15.88:0/1957417877 >> v2:172.21.15.154:3300/0 conn(0x7fb6340b4d00 msgr2=0x7fb6340b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:28.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.041+0000 7fb638ed8640 1 --2- 172.21.15.88:0/1957417877 >> v2:172.21.15.154:3300/0 conn(0x7fb6340b4d00 0x7fb6340b5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.041+0000 7fb638ed8640 1 -- 172.21.15.88:0/1957417877 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6340ceab0 con 0x7fb6340cd780 2024-09-16T01:37:28.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.041+0000 7fb6337fe640 1 --2- 172.21.15.88:0/1957417877 >> v2:172.21.15.154:3300/0 conn(0x7fb6340b4d00 0x7fb6340b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:37:28.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.041+0000 7fb638ed8640 1 --2- 172.21.15.88:0/1957417877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6340cd780 0x7fb6340cdbf0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fb63406c7d0 tx=0x7fb62c02c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af4b9515d9261b5c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:28.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.042+0000 7fb632ffd640 1 -- 172.21.15.88:0/1957417877 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb62c00b2e0 con 0x7fb6340cd780 2024-09-16T01:37:28.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.042+0000 7fb632ffd640 1 -- 172.21.15.88:0/1957417877 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb62c009600 con 0x7fb6340cd780 2024-09-16T01:37:28.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.042+0000 7fb632ffd640 1 -- 172.21.15.88:0/1957417877 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb62c00b480 con 0x7fb6340cd780 2024-09-16T01:37:28.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.042+0000 7fb63a93b640 1 -- 172.21.15.88:0/1957417877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6340cd780 msgr2=0x7fb6340cdbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:28.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.042+0000 7fb63a93b640 1 --2- 172.21.15.88:0/1957417877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6340cd780 0x7fb6340cdbf0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fb63406c7d0 tx=0x7fb62c02c8a0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.042+0000 7fb63a93b640 1 -- 172.21.15.88:0/1957417877 shutdown_connections 2024-09-16T01:37:28.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.042+0000 7fb63a93b640 1 --2- 172.21.15.88:0/1957417877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6340cd780 0x7fb6340cdbf0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.042+0000 7fb63a93b640 1 --2- 172.21.15.88:0/1957417877 >> v2:172.21.15.154:3300/0 conn(0x7fb6340b4d00 0x7fb6340b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.042+0000 7fb63a93b640 1 --2- 172.21.15.88:0/1957417877 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6340b4920 0x7fb63415b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.042+0000 7fb63a93b640 1 -- 172.21.15.88:0/1957417877 >> 172.21.15.88:0/1957417877 conn(0x7fb63406bc10 msgr2=0x7fb63406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:28.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.042+0000 7fb63a93b640 1 -- 172.21.15.88:0/1957417877 shutdown_connections 2024-09-16T01:37:28.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.042+0000 7fb63a93b640 1 -- 172.21.15.88:0/1957417877 wait complete. 2024-09-16T01:37:28.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.043+0000 7fb63a93b640 1 Processor -- start 2024-09-16T01:37:28.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.044+0000 7fb63a93b640 1 -- start start 2024-09-16T01:37:28.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.044+0000 7fb63a93b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6340b4920 0x7fb63414ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.044+0000 7fb63a93b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6340b4d00 0x7fb63406ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.044+0000 7fb63a93b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6340cd780 0x7fb63406d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.044+0000 7fb63a93b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb6340b5680 con 0x7fb6340cd780 2024-09-16T01:37:28.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.044+0000 7fb63a93b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb63415bd10 con 0x7fb6340b4d00 2024-09-16T01:37:28.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.044+0000 7fb63a93b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb6340ce130 con 0x7fb6340b4920 2024-09-16T01:37:28.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.044+0000 7fb638ed8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6340cd780 0x7fb63406d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.044+0000 7fb638ed8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6340cd780 0x7fb63406d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53758/0 (socket says 172.21.15.88:53758) 2024-09-16T01:37:28.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.045+0000 7fb638ed8640 1 -- 172.21.15.88:0/2907879591 learned_addr learned my addr 172.21.15.88:0/2907879591 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:28.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.045+0000 7fb6337fe640 1 --2- 172.21.15.88:0/2907879591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6340b4d00 0x7fb63406ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.045+0000 7fb638ed8640 1 -- 172.21.15.88:0/2907879591 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6340b4920 msgr2=0x7fb63414ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:28.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.045+0000 7fb633fff640 1 --2- 172.21.15.88:0/2907879591 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6340b4920 0x7fb63414ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.045+0000 7fb638ed8640 1 --2- 172.21.15.88:0/2907879591 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6340b4920 0x7fb63414ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.045+0000 7fb638ed8640 1 -- 172.21.15.88:0/2907879591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6340b4d00 msgr2=0x7fb63406ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:28.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.045+0000 7fb638ed8640 1 --2- 172.21.15.88:0/2907879591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6340b4d00 0x7fb63406ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.045+0000 7fb638ed8640 1 -- 172.21.15.88:0/2907879591 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb634159b30 con 0x7fb6340cd780 2024-09-16T01:37:28.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.045+0000 7fb6337fe640 1 --2- 172.21.15.88:0/2907879591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6340b4d00 0x7fb63406ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:37:28.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.045+0000 7fb638ed8640 1 --2- 172.21.15.88:0/2907879591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6340cd780 0x7fb63406d370 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fb62c02ce90 tx=0x7fb62c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:28.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.045+0000 7fb630ff9640 1 -- 172.21.15.88:0/2907879591 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb62c041030 con 0x7fb6340cd780 2024-09-16T01:37:28.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.045+0000 7fb630ff9640 1 -- 172.21.15.88:0/2907879591 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb62c03d650 con 0x7fb6340cd780 2024-09-16T01:37:28.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.045+0000 7fb63a93b640 1 -- 172.21.15.88:0/2907879591 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb634158d70 con 0x7fb6340cd780 2024-09-16T01:37:28.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.045+0000 7fb630ff9640 1 -- 172.21.15.88:0/2907879591 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb62c00b720 con 0x7fb6340cd780 2024-09-16T01:37:28.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.045+0000 7fb63a93b640 1 -- 172.21.15.88:0/2907879591 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb634159330 con 0x7fb6340cd780 2024-09-16T01:37:28.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.046+0000 7fb63a93b640 1 -- 172.21.15.88:0/2907879591 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6340b5140 con 0x7fb6340cd780 2024-09-16T01:37:28.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.047+0000 7fb630ff9640 1 -- 172.21.15.88:0/2907879591 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb62c03b040 con 0x7fb6340cd780 2024-09-16T01:37:28.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.047+0000 7fb630ff9640 1 --2- 172.21.15.88:0/2907879591 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb62006c2e0 0x7fb62006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.047+0000 7fb630ff9640 1 -- 172.21.15.88:0/2907879591 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5881+0+0 (secure 0 0 0) 0x7fb62c030020 con 0x7fb6340cd780 2024-09-16T01:37:28.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.047+0000 7fb633fff640 1 --2- 172.21.15.88:0/2907879591 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb62006c2e0 0x7fb62006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.048+0000 7fb633fff640 1 --2- 172.21.15.88:0/2907879591 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb62006c2e0 0x7fb62006e7a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb624006fd0 tx=0x7fb624008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:28.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.052+0000 7fb630ff9640 1 -- 172.21.15.88:0/2907879591 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb62c04b030 con 0x7fb6340cd780 2024-09-16T01:37:28.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.253+0000 7fb63a93b640 1 -- 172.21.15.88:0/2907879591 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6340710c0 con 0x7fb6340cd780 2024-09-16T01:37:28.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.254+0000 7fb630ff9640 1 -- 172.21.15.88:0/2907879591 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+5225 (secure 0 0 0) 0x7fb62c07c620 con 0x7fb6340cd780 2024-09-16T01:37:28.256 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:37:28.256 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":51,"btime":"2024-09-16T01:37:27:255277+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/581241194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":581241194},{"type":"v1","addr":"172.21.15.154:6834","nonce":581241194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2104508521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2104508521},{"type":"v1","addr":"172.21.15.154:6835","nonce":2104508521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":6195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2920926674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2920926674},{"type":"v1","addr":"172.21.15.88:6837","nonce":2920926674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:39.786438+0000","modified":"2024-09-16T01:37:27.255273+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5859},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5859":{"gid":5859,"name":"c","rank":0,"incarnation":48,"state":"up:active","state_seq":16,"addr":"172.21.15.88:6836/998283126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":998283126},{"type":"v1","addr":"172.21.15.88:6836","nonce":998283126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5859,"qdb_cluster":[5859]},"id":4}]} 2024-09-16T01:37:28.256 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 51 2024-09-16T01:37:28.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.256+0000 7fb63a93b640 1 -- 172.21.15.88:0/2907879591 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb62006c2e0 msgr2=0x7fb62006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:28.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.256+0000 7fb63a93b640 1 --2- 172.21.15.88:0/2907879591 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb62006c2e0 0x7fb62006e7a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb624006fd0 tx=0x7fb624008040 comp rx=0 tx=0).stop 2024-09-16T01:37:28.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.256+0000 7fb63a93b640 1 -- 172.21.15.88:0/2907879591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6340cd780 msgr2=0x7fb63406d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:28.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.256+0000 7fb63a93b640 1 --2- 172.21.15.88:0/2907879591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6340cd780 0x7fb63406d370 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fb62c02ce90 tx=0x7fb62c004a40 comp rx=0 tx=0).stop 2024-09-16T01:37:28.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.256+0000 7fb63a93b640 1 -- 172.21.15.88:0/2907879591 shutdown_connections 2024-09-16T01:37:28.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.256+0000 7fb63a93b640 1 --2- 172.21.15.88:0/2907879591 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb62006c2e0 0x7fb62006e7a0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.256+0000 7fb63a93b640 1 --2- 172.21.15.88:0/2907879591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6340cd780 0x7fb63406d370 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.256+0000 7fb63a93b640 1 --2- 172.21.15.88:0/2907879591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6340b4d00 0x7fb63406ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.256+0000 7fb63a93b640 1 --2- 172.21.15.88:0/2907879591 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6340b4920 0x7fb63414ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.256+0000 7fb63a93b640 1 -- 172.21.15.88:0/2907879591 >> 172.21.15.88:0/2907879591 conn(0x7fb63406bc10 msgr2=0x7fb63406bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:28.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.256+0000 7fb63a93b640 1 -- 172.21.15.88:0/2907879591 shutdown_connections 2024-09-16T01:37:28.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.256+0000 7fb63a93b640 1 -- 172.21.15.88:0/2907879591 wait complete. 2024-09-16T01:37:28.672 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:36:39.786438+0000', 'modified': '2024-09-16T01:37:27.255273+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 60, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5859}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5859': {'gid': 5859, 'name': 'c', 'rank': 0, 'incarnation': 48, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.88:6836/998283126', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 998283126}, {'type': 'v1', 'addr': '172.21.15.88:6836', 'nonce': 998283126}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5859, 'qdb_cluster': [5859]} 2024-09-16T01:37:28.672 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T01:37:28.672 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-16T01:37:28.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.771+0000 7fdf767f8640 1 Processor -- start 2024-09-16T01:37:28.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.771+0000 7fdf767f8640 1 -- start start 2024-09-16T01:37:28.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.772+0000 7fdf767f8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdf7015e7c0 0x7fdf700d1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.772+0000 7fdf767f8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdf700ce9e0 0x7fdf700b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.772+0000 7fdf767f8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf70074090 con 0x7fdf700ce3b0 2024-09-16T01:37:28.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.772+0000 7fdf767f8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fdf700ccd10 con 0x7fdf700ce9e0 2024-09-16T01:37:28.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.772+0000 7fdf767f8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fdf700ccff0 con 0x7fdf7015e7c0 2024-09-16T01:37:28.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.772+0000 7fdf6f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdf700ce9e0 0x7fdf700b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.772+0000 7fdf74d95640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fdf700ce3b0 0x7fdf700cd7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58870/0 (socket says 172.21.15.88:58870) 2024-09-16T01:37:28.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.772+0000 7fdf74d95640 1 -- 172.21.15.88:0/3396339236 learned_addr learned my addr 172.21.15.88:0/3396339236 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:37:28.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.772+0000 7fdf6f7fe640 1 -- 172.21.15.88:0/3396339236 >> v2:172.21.15.154:3300/0 conn(0x7fdf7015e7c0 msgr2=0x7fdf700d1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:28.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.772+0000 7fdf6ffff640 1 --2- 172.21.15.88:0/3396339236 >> v2:172.21.15.154:3300/0 conn(0x7fdf7015e7c0 0x7fdf700d1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.772+0000 7fdf6effd640 1 -- 172.21.15.88:0/3396339236 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1542339253 0 0) 0x7fdf70074090 con 0x7fdf700ce3b0 2024-09-16T01:37:28.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.772+0000 7fdf6f7fe640 1 --2- 172.21.15.88:0/3396339236 >> v2:172.21.15.154:3300/0 conn(0x7fdf7015e7c0 0x7fdf700d1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.772+0000 7fdf6f7fe640 1 -- 172.21.15.88:0/3396339236 >> v1:172.21.15.88:6789/0 conn(0x7fdf700ce3b0 legacy=0x7fdf700cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:28.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.772+0000 7fdf6f7fe640 1 -- 172.21.15.88:0/3396339236 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf700cd2d0 con 0x7fdf700ce9e0 2024-09-16T01:37:28.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.773+0000 7fdf6f7fe640 1 --2- 172.21.15.88:0/3396339236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf700ce9e0 0x7fdf700b44e0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fdf700cf620 tx=0x7fdf6402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f12ebdaca2af87c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:28.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.773+0000 7fdf6effd640 1 -- 172.21.15.88:0/3396339236 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf64037070 con 0x7fdf700ce9e0 2024-09-16T01:37:28.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.773+0000 7fdf6effd640 1 -- 172.21.15.88:0/3396339236 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf64032440 con 0x7fdf700ce9e0 2024-09-16T01:37:28.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.773+0000 7fdf6effd640 1 -- 172.21.15.88:0/3396339236 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf64032810 con 0x7fdf700ce9e0 2024-09-16T01:37:28.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.773+0000 7fdf767f8640 1 -- 172.21.15.88:0/3396339236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf700ce9e0 msgr2=0x7fdf700b44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:28.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.773+0000 7fdf767f8640 1 --2- 172.21.15.88:0/3396339236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf700ce9e0 0x7fdf700b44e0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fdf700cf620 tx=0x7fdf6402a1a0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.773+0000 7fdf767f8640 1 -- 172.21.15.88:0/3396339236 shutdown_connections 2024-09-16T01:37:28.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.773+0000 7fdf767f8640 1 --2- 172.21.15.88:0/3396339236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf700ce9e0 0x7fdf700b44e0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.773+0000 7fdf767f8640 1 --2- 172.21.15.88:0/3396339236 >> v2:172.21.15.154:3300/0 conn(0x7fdf7015e7c0 0x7fdf700d1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.773+0000 7fdf767f8640 1 -- 172.21.15.88:0/3396339236 >> 172.21.15.88:0/3396339236 conn(0x7fdf70062a20 msgr2=0x7fdf70062e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:28.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.774+0000 7fdf767f8640 1 -- 172.21.15.88:0/3396339236 shutdown_connections 2024-09-16T01:37:28.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.774+0000 7fdf767f8640 1 -- 172.21.15.88:0/3396339236 wait complete. 2024-09-16T01:37:28.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.775+0000 7fdf767f8640 1 Processor -- start 2024-09-16T01:37:28.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.775+0000 7fdf767f8640 1 -- start start 2024-09-16T01:37:28.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.776+0000 7fdf767f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf700ce3b0 0x7fdf701f51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.776+0000 7fdf767f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf700ce9e0 0x7fdf701f4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.776+0000 7fdf767f8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf7015e7c0 0x7fdf700b6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.776+0000 7fdf767f8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdf70073da0 con 0x7fdf7015e7c0 2024-09-16T01:37:28.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.776+0000 7fdf767f8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdf70073c20 con 0x7fdf700ce9e0 2024-09-16T01:37:28.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.776+0000 7fdf767f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdf700cdf00 con 0x7fdf700ce3b0 2024-09-16T01:37:28.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.776+0000 7fdf74d95640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf7015e7c0 0x7fdf700b6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.776+0000 7fdf74d95640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf7015e7c0 0x7fdf700b6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53780/0 (socket says 172.21.15.88:53780) 2024-09-16T01:37:28.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.776+0000 7fdf74d95640 1 -- 172.21.15.88:0/408456440 learned_addr learned my addr 172.21.15.88:0/408456440 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:28.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.776+0000 7fdf6ffff640 1 --2- 172.21.15.88:0/408456440 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf700ce3b0 0x7fdf701f51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.776+0000 7fdf6f7fe640 1 --2- 172.21.15.88:0/408456440 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf700ce9e0 0x7fdf701f4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.776+0000 7fdf74d95640 1 -- 172.21.15.88:0/408456440 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf700ce3b0 msgr2=0x7fdf701f51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:28.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.776+0000 7fdf74d95640 1 --2- 172.21.15.88:0/408456440 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf700ce3b0 0x7fdf701f51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.776+0000 7fdf74d95640 1 -- 172.21.15.88:0/408456440 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf700ce9e0 msgr2=0x7fdf701f4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:28.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.776+0000 7fdf74d95640 1 --2- 172.21.15.88:0/408456440 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf700ce9e0 0x7fdf701f4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.776+0000 7fdf74d95640 1 -- 172.21.15.88:0/408456440 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf7015efc0 con 0x7fdf7015e7c0 2024-09-16T01:37:28.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.776+0000 7fdf6f7fe640 1 --2- 172.21.15.88:0/408456440 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf700ce9e0 0x7fdf701f4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:37:28.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.776+0000 7fdf74d95640 1 --2- 172.21.15.88:0/408456440 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf7015e7c0 0x7fdf700b6e40 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800c7a0 tx=0x7fdf6800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:28.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.776+0000 7fdf6cff9640 1 -- 172.21.15.88:0/408456440 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf68018070 con 0x7fdf7015e7c0 2024-09-16T01:37:28.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.776+0000 7fdf6cff9640 1 -- 172.21.15.88:0/408456440 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf6800a640 con 0x7fdf7015e7c0 2024-09-16T01:37:28.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.776+0000 7fdf6cff9640 1 -- 172.21.15.88:0/408456440 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf6800aa70 con 0x7fdf7015e7c0 2024-09-16T01:37:28.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.776+0000 7fdf767f8640 1 -- 172.21.15.88:0/408456440 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf7015f2a0 con 0x7fdf7015e7c0 2024-09-16T01:37:28.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.776+0000 7fdf767f8640 1 -- 172.21.15.88:0/408456440 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf7015f860 con 0x7fdf7015e7c0 2024-09-16T01:37:28.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.778+0000 7fdf6cff9640 1 -- 172.21.15.88:0/408456440 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fdf6801b070 con 0x7fdf7015e7c0 2024-09-16T01:37:28.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.778+0000 7fdf6cff9640 1 --2- 172.21.15.88:0/408456440 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdf5c06c2b0 0x7fdf5c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.778+0000 7fdf6cff9640 1 -- 172.21.15.88:0/408456440 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5881+0+0 (secure 0 0 0) 0x7fdf6808ab40 con 0x7fdf7015e7c0 2024-09-16T01:37:28.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.778+0000 7fdf6ffff640 1 --2- 172.21.15.88:0/408456440 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdf5c06c2b0 0x7fdf5c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.779+0000 7fdf6ffff640 1 --2- 172.21.15.88:0/408456440 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdf5c06c2b0 0x7fdf5c06e770 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fdf60006fd0 tx=0x7fdf60008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:28.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.779+0000 7fdf79668740 1 Processor -- start 2024-09-16T01:37:28.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.779+0000 7fdf79668740 1 -- start start 2024-09-16T01:37:28.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.780+0000 7fdf79668740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd57042c0 0x560bd5749a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.780+0000 7fdf79668740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560bd5701a80 0x560bd57046a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.780+0000 7fdf79668740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd5705a50 0x560bd56fe8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.780+0000 7fdf79668740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x560bd5666520 con 0x560bd57042c0 2024-09-16T01:37:28.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.780+0000 7fdf79668740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560bd560e3e0 con 0x560bd5705a50 2024-09-16T01:37:28.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.780+0000 7fdf79668740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x560bd56624e0 con 0x560bd5701a80 2024-09-16T01:37:28.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.780+0000 7fdf6ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd57042c0 0x560bd5749a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.780+0000 7fdf6ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd57042c0 0x560bd5749a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53782/0 (socket says 172.21.15.88:53782) 2024-09-16T01:37:28.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.780+0000 7fdf6ffff640 1 -- 172.21.15.88:0/2630560643 learned_addr learned my addr 172.21.15.88:0/2630560643 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:28.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.780+0000 7fdf74d95640 1 --2- 172.21.15.88:0/2630560643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd5705a50 0x560bd56fe8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.780+0000 7fdf6f7fe640 1 --2- 172.21.15.88:0/2630560643 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560bd5701a80 0x560bd57046a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.780+0000 7fdf6ffff640 1 -- 172.21.15.88:0/2630560643 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560bd5701a80 msgr2=0x560bd57046a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:28.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.780+0000 7fdf6ffff640 1 --2- 172.21.15.88:0/2630560643 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560bd5701a80 0x560bd57046a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.780+0000 7fdf6ffff640 1 -- 172.21.15.88:0/2630560643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd5705a50 msgr2=0x560bd56fe8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:28.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.780+0000 7fdf6ffff640 1 --2- 172.21.15.88:0/2630560643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd5705a50 0x560bd56fe8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.780+0000 7fdf6ffff640 1 -- 172.21.15.88:0/2630560643 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bd56292f0 con 0x560bd57042c0 2024-09-16T01:37:28.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.780+0000 7fdf74d95640 1 --2- 172.21.15.88:0/2630560643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd5705a50 0x560bd56fe8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:37:28.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.780+0000 7fdf6f7fe640 1 --2- 172.21.15.88:0/2630560643 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560bd5701a80 0x560bd57046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:37:28.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.780+0000 7fdf6ffff640 1 --2- 172.21.15.88:0/2630560643 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd57042c0 0x560bd5749a60 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fdf60004300 tx=0x7fdf60011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae557a3aa244046c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:28.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.781+0000 7fdf55ffb640 1 -- 172.21.15.88:0/2630560643 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf60025070 con 0x560bd57042c0 2024-09-16T01:37:28.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.781+0000 7fdf55ffb640 1 -- 172.21.15.88:0/2630560643 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf6000ecb0 con 0x560bd57042c0 2024-09-16T01:37:28.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.781+0000 7fdf55ffb640 1 -- 172.21.15.88:0/2630560643 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf60003d90 con 0x560bd57042c0 2024-09-16T01:37:28.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.781+0000 7fdf79668740 1 -- 172.21.15.88:0/2630560643 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd57042c0 msgr2=0x560bd5749a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:28.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.781+0000 7fdf79668740 1 --2- 172.21.15.88:0/2630560643 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd57042c0 0x560bd5749a60 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fdf60004300 tx=0x7fdf60011000 comp rx=0 tx=0).stop 2024-09-16T01:37:28.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.781+0000 7fdf79668740 1 -- 172.21.15.88:0/2630560643 shutdown_connections 2024-09-16T01:37:28.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.781+0000 7fdf79668740 1 --2- 172.21.15.88:0/2630560643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd5705a50 0x560bd56fe8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.781+0000 7fdf79668740 1 --2- 172.21.15.88:0/2630560643 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560bd5701a80 0x560bd57046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.781+0000 7fdf79668740 1 --2- 172.21.15.88:0/2630560643 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd57042c0 0x560bd5749a60 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.781+0000 7fdf79668740 1 -- 172.21.15.88:0/2630560643 >> 172.21.15.88:0/2630560643 conn(0x560bd5700a80 msgr2=0x560bd5748f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:28.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.781+0000 7fdf79668740 1 -- 172.21.15.88:0/2630560643 shutdown_connections 2024-09-16T01:37:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.781+0000 7fdf79668740 1 -- 172.21.15.88:0/2630560643 wait complete. 2024-09-16T01:37:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.782+0000 7fdf79668740 1 -- start start 2024-09-16T01:37:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.782+0000 7fdf79668740 1 Processor -- start 2024-09-16T01:37:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.783+0000 7fdf79668740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560bd57042c0 0x560bd57254f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.783+0000 7fdf79668740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd5705a50 0x560bd5722f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.783+0000 7fdf79668740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd56fd7d0 0x560bd5702ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.783+0000 7fdf79668740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x560bd56fff20 con 0x560bd56fd7d0 2024-09-16T01:37:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.783+0000 7fdf79668740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560bd55ef450 con 0x560bd5705a50 2024-09-16T01:37:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.783+0000 7fdf79668740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x560bd574a290 con 0x560bd57042c0 2024-09-16T01:37:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.783+0000 7fdf79668740 10 client.0 _ll_register_callbacks cb 0x560bd5378ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:37:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.783+0000 7fdf74d95640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd56fd7d0 0x560bd5702ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.783+0000 7fdf74d95640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd56fd7d0 0x560bd5702ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53784/0 (socket says 172.21.15.88:53784) 2024-09-16T01:37:28.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.783+0000 7fdf74d95640 1 -- 172.21.15.88:0/1495460660 learned_addr learned my addr 172.21.15.88:0/1495460660 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:28.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.783+0000 7fdf6f7fe640 1 --2- 172.21.15.88:0/1495460660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd5705a50 0x560bd5722f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.783+0000 7fdf6ffff640 1 --2- 172.21.15.88:0/1495460660 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560bd57042c0 0x560bd57254f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.783+0000 7fdf74d95640 1 -- 172.21.15.88:0/1495460660 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560bd57042c0 msgr2=0x560bd57254f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:28.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.783+0000 7fdf74d95640 1 --2- 172.21.15.88:0/1495460660 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560bd57042c0 0x560bd57254f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.783+0000 7fdf74d95640 1 -- 172.21.15.88:0/1495460660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd5705a50 msgr2=0x560bd5722f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:28.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.783+0000 7fdf74d95640 1 --2- 172.21.15.88:0/1495460660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd5705a50 0x560bd5722f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.783+0000 7fdf74d95640 1 -- 172.21.15.88:0/1495460660 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bd53792d0 con 0x560bd56fd7d0 2024-09-16T01:37:28.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.783+0000 7fdf74d95640 1 --2- 172.21.15.88:0/1495460660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd56fd7d0 0x560bd5702ac0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fdf68088f60 tx=0x7fdf68059200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9031fc98c2b7aeae server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:28.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.783+0000 7fdf4effd640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:37:28.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.783+0000 7fdf4effd640 1 -- 172.21.15.88:0/1495460660 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf68061450 con 0x560bd56fd7d0 2024-09-16T01:37:28.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.783+0000 7fdf4effd640 1 -- 172.21.15.88:0/1495460660 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(60..60 src has 1..60) ==== 5881+0+0 (secure 0 0 0) 0x7fdf68057f50 con 0x560bd56fd7d0 2024-09-16T01:37:28.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.783+0000 7fdf79668740 1 -- 172.21.15.88:0/1495460660 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf6000ecb0 con 0x560bd56fd7d0 2024-09-16T01:37:28.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.784+0000 7fdf4effd640 1 -- 172.21.15.88:0/1495460660 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fdf6805ceb0 con 0x560bd56fd7d0 2024-09-16T01:37:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.784+0000 7fdf79668740 10 client.6228 fetch_fsmap learned FSMap version 51 2024-09-16T01:37:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.784+0000 7fdf79668740 1 -- 172.21.15.88:0/1495460660 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x560bd54436b0 con 0x560bd56fd7d0 2024-09-16T01:37:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.784+0000 7fdf4effd640 1 -- 172.21.15.88:0/1495460660 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf6000ecb0 con 0x560bd56fd7d0 2024-09-16T01:37:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.784+0000 7fdf4effd640 1 -- 172.21.15.88:0/1495460660 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 51) ==== 3502+0+0 (secure 0 0 0) 0x7fdf6805f0b0 con 0x560bd56fd7d0 2024-09-16T01:37:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.784+0000 7fdf79668740 10 client.6228 fetch_fsmap finished waiting for FSMap version 51 2024-09-16T01:37:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.784+0000 7fdf79668740 10 client.6228 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T01:37:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.784+0000 7fdf79668740 20 client.6228 populate_metadata read hostname 'smithi088' 2024-09-16T01:37:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.784+0000 7fdf79668740 1 --2- 172.21.15.88:0/1495460660 >> [v2:172.21.15.88:6834/998283126,v1:172.21.15.88:6836/998283126] conn(0x560bd572d8d0 0x560bd572dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.784+0000 7fdf79668740 4 client.6228 mds_command: new command op to 5859 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:37:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.784+0000 7fdf79668740 1 -- 172.21.15.88:0/1495460660 --> [v2:172.21.15.88:6834/998283126,v1:172.21.15.88:6836/998283126] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560bd5673c50 con 0x560bd572d8d0 2024-09-16T01:37:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.784+0000 7fdf6ffff640 1 --2- 172.21.15.88:0/1495460660 >> [v2:172.21.15.88:6834/998283126,v1:172.21.15.88:6836/998283126] conn(0x560bd572d8d0 0x560bd572dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.784+0000 7fdf4effd640 10 client.6228 ms_handle_connect on v2:172.21.15.88:6834/998283126 2024-09-16T01:37:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.784+0000 7fdf6ffff640 1 --2- 172.21.15.88:0/1495460660 >> [v2:172.21.15.88:6834/998283126,v1:172.21.15.88:6836/998283126] conn(0x560bd572d8d0 0x560bd572dcb0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cafe6242d9d41114 server_cookie=eedd6f6ddc0b46f3 in_seq=0 out_seq=0 2024-09-16T01:37:28.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.785+0000 7fdf4effd640 1 -- 172.21.15.88:0/1495460660 <== mds.0 v2:172.21.15.88:6834/998283126 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560bd5673c50 con 0x560bd572d8d0 2024-09-16T01:37:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.785+0000 7fdf4effd640 10 client.6228 handle_command_reply: tid=0 2024-09-16T01:37:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.785+0000 7fdf79668740 1 client.6228 shutdown 2024-09-16T01:37:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.785+0000 7fdf79668740 1 -- 172.21.15.88:0/1495460660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd56fd7d0 msgr2=0x560bd5702ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.785+0000 7fdf79668740 1 --2- 172.21.15.88:0/1495460660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd56fd7d0 0x560bd5702ac0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fdf68088f60 tx=0x7fdf68059200 comp rx=0 tx=0).stop 2024-09-16T01:37:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.785+0000 7fdf79668740 1 -- 172.21.15.88:0/1495460660 shutdown_connections 2024-09-16T01:37:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.785+0000 7fdf79668740 1 --2- 172.21.15.88:0/1495460660 >> [v2:172.21.15.88:6834/998283126,v1:172.21.15.88:6836/998283126] conn(0x560bd572d8d0 0x560bd572dcb0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.785+0000 7fdf79668740 1 --2- 172.21.15.88:0/1495460660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd56fd7d0 0x560bd5702ac0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.785+0000 7fdf79668740 1 --2- 172.21.15.88:0/1495460660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd5705a50 0x560bd5722f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.785+0000 7fdf4effd640 0 client.6228 ms_handle_reset on v2:172.21.15.88:6834/998283126 2024-09-16T01:37:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.785+0000 7fdf79668740 1 --2- 172.21.15.88:0/1495460660 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560bd57042c0 0x560bd57254f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.785+0000 7fdf79668740 1 -- 172.21.15.88:0/1495460660 >> 172.21.15.88:0/1495460660 conn(0x560bd5701a80 msgr2=0x560bd5747e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.785+0000 7fdf79668740 1 -- 172.21.15.88:0/1495460660 shutdown_connections 2024-09-16T01:37:28.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.785+0000 7fdf79668740 1 -- 172.21.15.88:0/1495460660 wait complete. 2024-09-16T01:37:28.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.785+0000 7fdf79668740 20 client.6228 trim_cache size 0 max 16384 2024-09-16T01:37:28.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.803+0000 7fdf79668740 1 Processor -- start 2024-09-16T01:37:28.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.803+0000 7fdf79668740 1 -- start start 2024-09-16T01:37:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.804+0000 7fdf79668740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd56fd7d0 0x560bd572e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.804+0000 7fdf79668740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd57042c0 0x560bd576cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.804+0000 7fdf79668740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560bd5705a50 0x560bd576d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.804+0000 7fdf79668740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdf701ffc30 con 0x560bd57042c0 2024-09-16T01:37:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.804+0000 7fdf79668740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdf70160cc0 con 0x560bd56fd7d0 2024-09-16T01:37:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.804+0000 7fdf79668740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdf700b6180 con 0x560bd5705a50 2024-09-16T01:37:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.804+0000 7fdf6f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd57042c0 0x560bd576cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.804+0000 7fdf6f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd57042c0 0x560bd576cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53800/0 (socket says 172.21.15.88:53800) 2024-09-16T01:37:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.804+0000 7fdf6f7fe640 1 -- 172.21.15.88:0/3307253206 learned_addr learned my addr 172.21.15.88:0/3307253206 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.804+0000 7fdf6ffff640 1 --2- 172.21.15.88:0/3307253206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd56fd7d0 0x560bd572e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.804+0000 7fdf74d95640 1 --2- 172.21.15.88:0/3307253206 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560bd5705a50 0x560bd576d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.804+0000 7fdf6f7fe640 1 -- 172.21.15.88:0/3307253206 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560bd5705a50 msgr2=0x560bd576d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.804+0000 7fdf6f7fe640 1 --2- 172.21.15.88:0/3307253206 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560bd5705a50 0x560bd576d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.804+0000 7fdf6f7fe640 1 -- 172.21.15.88:0/3307253206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd56fd7d0 msgr2=0x560bd572e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.804+0000 7fdf6f7fe640 1 --2- 172.21.15.88:0/3307253206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd56fd7d0 0x560bd572e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.804+0000 7fdf6f7fe640 1 -- 172.21.15.88:0/3307253206 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bd57468e0 con 0x560bd57042c0 2024-09-16T01:37:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.804+0000 7fdf74d95640 1 --2- 172.21.15.88:0/3307253206 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560bd5705a50 0x560bd576d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:37:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.804+0000 7fdf6f7fe640 1 --2- 172.21.15.88:0/3307253206 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd57042c0 0x560bd576cb90 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fdf64000c00 tx=0x7fdf6402c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd25fa5b9c1a41ce server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.804+0000 7fdf557fa640 1 -- 172.21.15.88:0/3307253206 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf64043030 con 0x560bd57042c0 2024-09-16T01:37:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.804+0000 7fdf557fa640 1 -- 172.21.15.88:0/3307253206 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf64004140 con 0x560bd57042c0 2024-09-16T01:37:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.804+0000 7fdf557fa640 1 -- 172.21.15.88:0/3307253206 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf64037040 con 0x560bd57042c0 2024-09-16T01:37:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.805+0000 7fdf79668740 1 -- 172.21.15.88:0/3307253206 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd57042c0 msgr2=0x560bd576cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.805+0000 7fdf79668740 1 --2- 172.21.15.88:0/3307253206 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd57042c0 0x560bd576cb90 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fdf64000c00 tx=0x7fdf6402c000 comp rx=0 tx=0).stop 2024-09-16T01:37:28.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.805+0000 7fdf79668740 1 -- 172.21.15.88:0/3307253206 shutdown_connections 2024-09-16T01:37:28.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.805+0000 7fdf79668740 1 --2- 172.21.15.88:0/3307253206 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560bd5705a50 0x560bd576d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.805+0000 7fdf79668740 1 --2- 172.21.15.88:0/3307253206 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd57042c0 0x560bd576cb90 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.805+0000 7fdf79668740 1 --2- 172.21.15.88:0/3307253206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd56fd7d0 0x560bd572e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.805+0000 7fdf79668740 1 -- 172.21.15.88:0/3307253206 >> 172.21.15.88:0/3307253206 conn(0x560bd5701a80 msgr2=0x560bd57588a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:28.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.805+0000 7fdf79668740 1 -- 172.21.15.88:0/3307253206 shutdown_connections 2024-09-16T01:37:28.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.805+0000 7fdf79668740 1 -- 172.21.15.88:0/3307253206 wait complete. 2024-09-16T01:37:28.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.805+0000 7fdf79668740 1 -- start start 2024-09-16T01:37:28.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.806+0000 7fdf79668740 1 Processor -- start 2024-09-16T01:37:28.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.806+0000 7fdf79668740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560bd56fd7d0 0x560bd5702020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.806+0000 7fdf79668740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd57042c0 0x560bd57c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.806+0000 7fdf79668740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd5705a50 0x560bd5745b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.806+0000 7fdf79668740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x560bd572e950 con 0x560bd57042c0 2024-09-16T01:37:28.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.806+0000 7fdf79668740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560bd56fe500 con 0x560bd5705a50 2024-09-16T01:37:28.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.806+0000 7fdf79668740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x560bd5724050 con 0x560bd56fd7d0 2024-09-16T01:37:28.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.807+0000 7fdf79668740 10 client.0 _ll_register_callbacks cb 0x7fdf68058640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:37:28.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.807+0000 7fdf6f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd57042c0 0x560bd57c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.807+0000 7fdf6f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd57042c0 0x560bd57c6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53810/0 (socket says 172.21.15.88:53810) 2024-09-16T01:37:28.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.807+0000 7fdf6f7fe640 1 -- 172.21.15.88:0/1646163618 learned_addr learned my addr 172.21.15.88:0/1646163618 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:37:28.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.807+0000 7fdf6ffff640 1 --2- 172.21.15.88:0/1646163618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560bd56fd7d0 0x560bd5702020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.807+0000 7fdf74d95640 1 --2- 172.21.15.88:0/1646163618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd5705a50 0x560bd5745b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.807+0000 7fdf6f7fe640 1 -- 172.21.15.88:0/1646163618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560bd56fd7d0 msgr2=0x560bd5702020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:28.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.807+0000 7fdf6f7fe640 1 --2- 172.21.15.88:0/1646163618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560bd56fd7d0 0x560bd5702020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.807+0000 7fdf6f7fe640 1 -- 172.21.15.88:0/1646163618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd5705a50 msgr2=0x560bd5745b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:37:28.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.807+0000 7fdf6f7fe640 1 --2- 172.21.15.88:0/1646163618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd5705a50 0x560bd5745b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.807+0000 7fdf6f7fe640 1 -- 172.21.15.88:0/1646163618 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bd5702560 con 0x560bd57042c0 2024-09-16T01:37:28.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.807+0000 7fdf6f7fe640 1 --2- 172.21.15.88:0/1646163618 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd57042c0 0x560bd57c6ae0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fdf64009860 tx=0x7fdf64004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=248bcad81935d339 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:37:28.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.807+0000 7fdf4effd640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:37:28.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.807+0000 7fdf4effd640 1 -- 172.21.15.88:0/1646163618 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf64037040 con 0x560bd57042c0 2024-09-16T01:37:28.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.807+0000 7fdf4effd640 1 -- 172.21.15.88:0/1646163618 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(60..60 src has 1..60) ==== 5881+0+0 (secure 0 0 0) 0x7fdf64003a20 con 0x560bd57042c0 2024-09-16T01:37:28.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.807+0000 7fdf79668740 1 -- 172.21.15.88:0/1646163618 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf64004140 con 0x560bd57042c0 2024-09-16T01:37:28.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.807+0000 7fdf4effd640 1 -- 172.21.15.88:0/1646163618 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fdf64056050 con 0x560bd57042c0 2024-09-16T01:37:28.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.807+0000 7fdf79668740 10 client.6234 fetch_fsmap learned FSMap version 51 2024-09-16T01:37:28.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.807+0000 7fdf79668740 1 -- 172.21.15.88:0/1646163618 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x560bd57c7080 con 0x560bd57042c0 2024-09-16T01:37:28.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.807+0000 7fdf4effd640 1 -- 172.21.15.88:0/1646163618 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf64004140 con 0x560bd57042c0 2024-09-16T01:37:28.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.808+0000 7fdf4effd640 1 -- 172.21.15.88:0/1646163618 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 51) ==== 3502+0+0 (secure 0 0 0) 0x7fdf64059020 con 0x560bd57042c0 2024-09-16T01:37:28.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.808+0000 7fdf79668740 10 client.6234 fetch_fsmap finished waiting for FSMap version 51 2024-09-16T01:37:28.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.808+0000 7fdf79668740 10 client.6234 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T01:37:28.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.808+0000 7fdf79668740 20 client.6234 populate_metadata read hostname 'smithi088' 2024-09-16T01:37:28.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.808+0000 7fdf79668740 1 --2- 172.21.15.88:0/1646163618 >> [v2:172.21.15.88:6834/998283126,v1:172.21.15.88:6836/998283126] conn(0x560bd576ccf0 0x560bd576d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:37:28.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.808+0000 7fdf79668740 4 client.6234 mds_command: new command op to 5859 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:37:28.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.808+0000 7fdf79668740 1 -- 172.21.15.88:0/1646163618 --> [v2:172.21.15.88:6834/998283126,v1:172.21.15.88:6836/998283126] -- command(tid 0: {"prefix": "status"}) -- 0x560bd566dfa0 con 0x560bd576ccf0 2024-09-16T01:37:28.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.808+0000 7fdf6ffff640 1 --2- 172.21.15.88:0/1646163618 >> [v2:172.21.15.88:6834/998283126,v1:172.21.15.88:6836/998283126] conn(0x560bd576ccf0 0x560bd576d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:28.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.808+0000 7fdf4effd640 10 client.6234 ms_handle_connect on v2:172.21.15.88:6834/998283126 2024-09-16T01:37:28.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.808+0000 7fdf6ffff640 1 --2- 172.21.15.88:0/1646163618 >> [v2:172.21.15.88:6834/998283126,v1:172.21.15.88:6836/998283126] conn(0x560bd576ccf0 0x560bd576d0d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e10306c397647d3 server_cookie=30a269a22b012403 in_seq=0 out_seq=0 2024-09-16T01:37:28.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.808+0000 7fdf4effd640 1 -- 172.21.15.88:0/1646163618 <== mds.0 v2:172.21.15.88:6834/998283126 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x560bd566dfa0 con 0x560bd576ccf0 2024-09-16T01:37:28.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.808+0000 7fdf4effd640 10 client.6234 handle_command_reply: tid=0 2024-09-16T01:37:28.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.808+0000 7fdf79668740 1 client.6234 shutdown 2024-09-16T01:37:28.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.808+0000 7fdf79668740 1 -- 172.21.15.88:0/1646163618 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd57042c0 msgr2=0x560bd57c6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:28.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.808+0000 7fdf79668740 1 --2- 172.21.15.88:0/1646163618 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd57042c0 0x560bd57c6ae0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fdf64009860 tx=0x7fdf64004730 comp rx=0 tx=0).stop 2024-09-16T01:37:28.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.808+0000 7fdf79668740 1 -- 172.21.15.88:0/1646163618 shutdown_connections 2024-09-16T01:37:28.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.808+0000 7fdf79668740 1 --2- 172.21.15.88:0/1646163618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560bd5705a50 0x560bd5745b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.808+0000 7fdf79668740 1 --2- 172.21.15.88:0/1646163618 >> [v2:172.21.15.88:6834/998283126,v1:172.21.15.88:6836/998283126] conn(0x560bd576ccf0 0x560bd576d0d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.808+0000 7fdf79668740 1 --2- 172.21.15.88:0/1646163618 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560bd57042c0 0x560bd57c6ae0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.808+0000 7fdf79668740 1 --2- 172.21.15.88:0/1646163618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560bd56fd7d0 0x560bd5702020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.808+0000 7fdf4effd640 0 client.6234 ms_handle_reset on v2:172.21.15.88:6834/998283126 2024-09-16T01:37:28.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.808+0000 7fdf79668740 1 -- 172.21.15.88:0/1646163618 >> 172.21.15.88:0/1646163618 conn(0x560bd5701a80 msgr2=0x560bd57588a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:28.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.809+0000 7fdf79668740 1 -- 172.21.15.88:0/1646163618 shutdown_connections 2024-09-16T01:37:28.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.809+0000 7fdf79668740 1 -- 172.21.15.88:0/1646163618 wait complete. 2024-09-16T01:37:28.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.809+0000 7fdf79668740 20 client.6234 trim_cache size 0 max 16384 2024-09-16T01:37:28.817 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:37:28.817 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "5ace2c91-46f9-482b-8a89-731fbdc171bf", 2024-09-16T01:37:28.817 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2024-09-16T01:37:28.817 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 5859, 2024-09-16T01:37:28.817 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2024-09-16T01:37:28.817 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2024-09-16T01:37:28.817 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2024-09-16T01:37:28.817 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 4.5668219600000004, 2024-09-16T01:37:28.817 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 51, 2024-09-16T01:37:28.817 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 60, 2024-09-16T01:37:28.818 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 60, 2024-09-16T01:37:28.818 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 52.705385606 2024-09-16T01:37:28.818 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:37:28.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.809+0000 7fdf767f8640 1 -- 172.21.15.88:0/408456440 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdf5c06c2b0 msgr2=0x7fdf5c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:28.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.809+0000 7fdf767f8640 1 --2- 172.21.15.88:0/408456440 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdf5c06c2b0 0x7fdf5c06e770 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fdf60006fd0 tx=0x7fdf60008040 comp rx=0 tx=0).stop 2024-09-16T01:37:28.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.809+0000 7fdf767f8640 1 -- 172.21.15.88:0/408456440 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf7015e7c0 msgr2=0x7fdf700b6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:37:28.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.809+0000 7fdf767f8640 1 --2- 172.21.15.88:0/408456440 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf7015e7c0 0x7fdf700b6e40 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800c7a0 tx=0x7fdf6800cc60 comp rx=0 tx=0).stop 2024-09-16T01:37:28.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.809+0000 7fdf767f8640 1 -- 172.21.15.88:0/408456440 shutdown_connections 2024-09-16T01:37:28.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.809+0000 7fdf767f8640 1 --2- 172.21.15.88:0/408456440 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdf5c06c2b0 0x7fdf5c06e770 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.809+0000 7fdf767f8640 1 --2- 172.21.15.88:0/408456440 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf7015e7c0 0x7fdf700b6e40 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.809+0000 7fdf767f8640 1 --2- 172.21.15.88:0/408456440 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf700ce9e0 0x7fdf701f4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.809+0000 7fdf767f8640 1 --2- 172.21.15.88:0/408456440 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf700ce3b0 0x7fdf701f51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:37:28.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.809+0000 7fdf767f8640 1 -- 172.21.15.88:0/408456440 >> 172.21.15.88:0/408456440 conn(0x7fdf70062a20 msgr2=0x7fdf700733d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:37:28.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.809+0000 7fdf767f8640 1 -- 172.21.15.88:0/408456440 shutdown_connections 2024-09-16T01:37:28.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:28.809+0000 7fdf767f8640 1 -- 172.21.15.88:0/408456440 wait complete. 2024-09-16T01:37:31.443 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:37:31.442+0000 7f1648390640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:37:41.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:41.506+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:41.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:41.507+0000 7fc9267fc640 1 -- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 msgr2=0x55e7d13cdc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2024-09-16T01:37:41.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:41.507+0000 7fc9267fc640 1 -- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 msgr2=0x55e7d13cdc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T01:37:41.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:41.507+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T01:37:41.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:41.507+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-16T01:37:46.686 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:37:46.689 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:37:56.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:56.521+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:37:56.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:56.521+0000 7fc9267fc640 1 -- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 msgr2=0x55e7d13cdc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2024-09-16T01:37:56.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:56.521+0000 7fc9267fc640 1 -- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 msgr2=0x55e7d13cdc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T01:37:56.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:56.521+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T01:37:56.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:37:56.522+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-16T01:38:11.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:38:11.524+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:38:11.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:38:11.525+0000 7fc9267fc640 1 -- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 msgr2=0x55e7d13cdc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2024-09-16T01:38:11.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:38:11.525+0000 7fc9267fc640 1 -- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 msgr2=0x55e7d13cdc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T01:38:11.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:38:11.525+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T01:38:11.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:38:11.525+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-16T01:38:16.841 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:38:16.844 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:38:26.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:38:26.531+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:38:26.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:38:26.531+0000 7fc9267fc640 1 -- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 msgr2=0x55e7d13cdc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2024-09-16T01:38:26.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:38:26.531+0000 7fc9267fc640 1 -- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 msgr2=0x55e7d13cdc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T01:38:26.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:38:26.531+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T01:38:26.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:38:26.531+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-16T01:38:41.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:38:41.535+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:38:41.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:38:41.535+0000 7fc9267fc640 1 -- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 msgr2=0x55e7d13cdc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2024-09-16T01:38:41.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:38:41.535+0000 7fc9267fc640 1 -- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 msgr2=0x55e7d13cdc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T01:38:41.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:38:41.535+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T01:38:41.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:38:41.535+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-16T01:38:46.987 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:38:46.990 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:38:56.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:38:56.550+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:38:56.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:38:56.550+0000 7fc9267fc640 1 -- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 msgr2=0x55e7d13cdc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2024-09-16T01:38:56.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:38:56.550+0000 7fc9267fc640 1 -- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 msgr2=0x55e7d13cdc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T01:38:56.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:38:56.550+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T01:38:56.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:38:56.551+0000 7fc9267fc640 1 --2- 172.21.15.88:0/2464331926 >> [v2:172.21.15.88:6835/4130074102,v1:172.21.15.88:6837/4130074102] conn(0x55e7d13b7850 0x55e7d13cdc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-09-16T01:39:00.917 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-09-16T01:39:00.917 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:39:01.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.025+0000 7eff9de65640 1 Processor -- start 2024-09-16T01:39:01.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.025+0000 7eff9de65640 1 -- start start 2024-09-16T01:39:01.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.026+0000 7eff9de65640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7eff98155800 0x7eff980b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:01.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.026+0000 7eff9de65640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff980b50a0 con 0x7eff980cfc80 2024-09-16T01:39:01.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.026+0000 7eff9de65640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff980b5380 con 0x7eff9815c330 2024-09-16T01:39:01.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.026+0000 7eff9de65640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7eff9815c7a0 con 0x7eff98155800 2024-09-16T01:39:01.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.026+0000 7eff977fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7eff98155800 0x7eff980b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:01.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.026+0000 7eff97fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7eff980cfc80 0x7eff9815c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57352/0 (socket says 172.21.15.88:57352) 2024-09-16T01:39:01.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.026+0000 7eff97fff640 1 -- 172.21.15.88:0/1503545079 learned_addr learned my addr 172.21.15.88:0/1503545079 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:01.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.027+0000 7eff977fe640 1 -- 172.21.15.88:0/1503545079 >> v1:172.21.15.154:6789/0 conn(0x7eff9815c330 legacy=0x7eff980cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:01.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.027+0000 7eff977fe640 1 -- 172.21.15.88:0/1503545079 >> v1:172.21.15.88:6789/0 conn(0x7eff980cfc80 legacy=0x7eff9815c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:01.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.027+0000 7eff967fc640 1 -- 172.21.15.88:0/1503545079 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3193159047 0 0) 0x7eff980b50a0 con 0x7eff980cfc80 2024-09-16T01:39:01.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.027+0000 7eff977fe640 1 -- 172.21.15.88:0/1503545079 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff980b5660 con 0x7eff98155800 2024-09-16T01:39:01.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.027+0000 7eff977fe640 1 --2- 172.21.15.88:0/1503545079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff98155800 0x7eff980b6140 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7eff880097a0 tx=0x7eff8802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a56d55dec869f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:01.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.028+0000 7eff967fc640 1 -- 172.21.15.88:0/1503545079 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff88037070 con 0x7eff98155800 2024-09-16T01:39:01.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.028+0000 7eff967fc640 1 -- 172.21.15.88:0/1503545079 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff88032440 con 0x7eff98155800 2024-09-16T01:39:01.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.028+0000 7eff967fc640 1 -- 172.21.15.88:0/1503545079 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff88032810 con 0x7eff98155800 2024-09-16T01:39:01.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.028+0000 7eff9de65640 1 -- 172.21.15.88:0/1503545079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff98155800 msgr2=0x7eff980b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:01.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.028+0000 7eff9de65640 1 --2- 172.21.15.88:0/1503545079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff98155800 0x7eff980b6140 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7eff880097a0 tx=0x7eff8802a1e0 comp rx=0 tx=0).stop 2024-09-16T01:39:01.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.028+0000 7eff9de65640 1 -- 172.21.15.88:0/1503545079 shutdown_connections 2024-09-16T01:39:01.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.028+0000 7eff9de65640 1 --2- 172.21.15.88:0/1503545079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff98155800 0x7eff980b6140 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:01.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.028+0000 7eff9de65640 1 -- 172.21.15.88:0/1503545079 >> 172.21.15.88:0/1503545079 conn(0x7eff9806bc10 msgr2=0x7eff9806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:01.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.028+0000 7eff9de65640 1 -- 172.21.15.88:0/1503545079 shutdown_connections 2024-09-16T01:39:01.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.028+0000 7eff9de65640 1 -- 172.21.15.88:0/1503545079 wait complete. 2024-09-16T01:39:01.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.029+0000 7eff9de65640 1 Processor -- start 2024-09-16T01:39:01.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.030+0000 7eff9de65640 1 -- start start 2024-09-16T01:39:01.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.030+0000 7eff9de65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff980cfc80 0x7eff9814cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:01.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.030+0000 7eff9de65640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff98155800 0x7eff981f25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:01.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.030+0000 7eff9de65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff9815c330 0x7eff981f2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:01.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.030+0000 7eff9de65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7eff98155be0 con 0x7eff9815c330 2024-09-16T01:39:01.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.030+0000 7eff9de65640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7eff980b5c20 con 0x7eff98155800 2024-09-16T01:39:01.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.030+0000 7eff9de65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7eff980b4920 con 0x7eff980cfc80 2024-09-16T01:39:01.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.031+0000 7eff97fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff9815c330 0x7eff981f2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:01.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.031+0000 7eff97fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff9815c330 0x7eff981f2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52672/0 (socket says 172.21.15.88:52672) 2024-09-16T01:39:01.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.031+0000 7eff97fff640 1 -- 172.21.15.88:0/1209708544 learned_addr learned my addr 172.21.15.88:0/1209708544 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:01.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.031+0000 7eff96ffd640 1 --2- 172.21.15.88:0/1209708544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff98155800 0x7eff981f25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:01.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.031+0000 7eff977fe640 1 --2- 172.21.15.88:0/1209708544 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff980cfc80 0x7eff9814cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:01.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.031+0000 7eff97fff640 1 -- 172.21.15.88:0/1209708544 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff980cfc80 msgr2=0x7eff9814cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:01.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.031+0000 7eff97fff640 1 --2- 172.21.15.88:0/1209708544 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff980cfc80 0x7eff9814cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:01.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.031+0000 7eff97fff640 1 -- 172.21.15.88:0/1209708544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff98155800 msgr2=0x7eff981f25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:01.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.031+0000 7eff97fff640 1 --2- 172.21.15.88:0/1209708544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff98155800 0x7eff981f25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:01.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.031+0000 7eff97fff640 1 -- 172.21.15.88:0/1209708544 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff9814c970 con 0x7eff9815c330 2024-09-16T01:39:01.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.031+0000 7eff97fff640 1 --2- 172.21.15.88:0/1209708544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff9815c330 0x7eff981f2b30 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7eff9000c830 tx=0x7eff9000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:01.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.031+0000 7eff7ffff640 1 -- 172.21.15.88:0/1209708544 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff90018070 con 0x7eff9815c330 2024-09-16T01:39:01.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.031+0000 7eff7ffff640 1 -- 172.21.15.88:0/1209708544 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff9000ce20 con 0x7eff9815c330 2024-09-16T01:39:01.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.031+0000 7eff7ffff640 1 -- 172.21.15.88:0/1209708544 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff9000a880 con 0x7eff9815c330 2024-09-16T01:39:01.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.031+0000 7eff9de65640 1 -- 172.21.15.88:0/1209708544 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff9814d4e0 con 0x7eff9815c330 2024-09-16T01:39:01.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.032+0000 7eff9de65640 1 -- 172.21.15.88:0/1209708544 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff9814dd80 con 0x7eff9815c330 2024-09-16T01:39:01.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.033+0000 7eff9de65640 1 -- 172.21.15.88:0/1209708544 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff98071bf0 con 0x7eff9815c330 2024-09-16T01:39:01.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.033+0000 7eff7ffff640 1 -- 172.21.15.88:0/1209708544 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7eff9001b070 con 0x7eff9815c330 2024-09-16T01:39:01.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.033+0000 7eff7ffff640 1 --2- 172.21.15.88:0/1209708544 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff8406c2b0 0x7eff8406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:01.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.033+0000 7eff7ffff640 1 -- 172.21.15.88:0/1209708544 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5881+0+0 (secure 0 0 0) 0x7eff9008a9e0 con 0x7eff9815c330 2024-09-16T01:39:01.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.033+0000 7eff977fe640 1 --2- 172.21.15.88:0/1209708544 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff8406c2b0 0x7eff8406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:01.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.034+0000 7eff977fe640 1 --2- 172.21.15.88:0/1209708544 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff8406c2b0 0x7eff8406e770 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7eff88034000 tx=0x7eff8802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:01.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.038+0000 7eff7ffff640 1 -- 172.21.15.88:0/1209708544 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7eff98071bf0 con 0x7eff9815c330 2024-09-16T01:39:01.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.239+0000 7eff9de65640 1 -- 172.21.15.88:0/1209708544 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7eff980b4cb0 con 0x7eff9815c330 2024-09-16T01:39:01.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.239+0000 7eff7ffff640 1 -- 172.21.15.88:0/1209708544 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v51) ==== 53+0+73 (secure 0 0 0) 0x7eff9814c970 con 0x7eff9815c330 2024-09-16T01:39:01.241 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:39:01.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.242+0000 7eff9de65640 1 -- 172.21.15.88:0/1209708544 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff8406c2b0 msgr2=0x7eff8406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:01.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.242+0000 7eff9de65640 1 --2- 172.21.15.88:0/1209708544 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff8406c2b0 0x7eff8406e770 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7eff88034000 tx=0x7eff8802e000 comp rx=0 tx=0).stop 2024-09-16T01:39:01.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.242+0000 7eff9de65640 1 -- 172.21.15.88:0/1209708544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff9815c330 msgr2=0x7eff981f2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:01.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.242+0000 7eff9de65640 1 --2- 172.21.15.88:0/1209708544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff9815c330 0x7eff981f2b30 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7eff9000c830 tx=0x7eff9000ccf0 comp rx=0 tx=0).stop 2024-09-16T01:39:01.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.242+0000 7eff9de65640 1 -- 172.21.15.88:0/1209708544 shutdown_connections 2024-09-16T01:39:01.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.242+0000 7eff9de65640 1 --2- 172.21.15.88:0/1209708544 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff8406c2b0 0x7eff8406e770 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:01.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.242+0000 7eff9de65640 1 --2- 172.21.15.88:0/1209708544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff9815c330 0x7eff981f2b30 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:01.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.242+0000 7eff9de65640 1 --2- 172.21.15.88:0/1209708544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff98155800 0x7eff981f25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:01.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.242+0000 7eff9de65640 1 --2- 172.21.15.88:0/1209708544 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff980cfc80 0x7eff9814cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:01.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.242+0000 7eff9de65640 1 -- 172.21.15.88:0/1209708544 >> 172.21.15.88:0/1209708544 conn(0x7eff9806bc10 msgr2=0x7eff980714e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:01.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.242+0000 7eff9de65640 1 -- 172.21.15.88:0/1209708544 shutdown_connections 2024-09-16T01:39:01.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01.242+0000 7eff9de65640 1 -- 172.21.15.88:0/1209708544 wait complete. 2024-09-16T01:39:01.254 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-16T01:39:01.254 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:39:01.254 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2024-09-16T01:39:01.255 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T01:39:01.255 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T01:39:01.255 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-16T01:39:01.255 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-16T01:39:01.255 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:39:01.255 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:39:01.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:39:01.255 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2024-09-16T01:39:01.271 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:39:01.271 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:39:01.271 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:39:01.272 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:39:01.272 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-09-16T01:39:01.272 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:39:01.272 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:39:01.272 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:39:01.272 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2024-09-16T01:39:01.272 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:39:01.272 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:39:01.272 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2024-09-16T01:39:01.272 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:39:01.272 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2024-09-16T01:39:01.272 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:39:01.272 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2024-09-16T01:39:01.273 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:39:01.273 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42483sec preferred_lft 42483sec 2024-09-16T01:39:01.273 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link noprefixroute 2024-09-16T01:39:01.273 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:39:01.273 INFO:teuthology.orchestra.run.smithi088.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:39:01.273 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff 2024-09-16T01:39:01.273 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T01:39:01.273 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:39:01.273 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::e486:6eff:fe81:a1c/64 scope link 2024-09-16T01:39:01.273 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:39:01.273 INFO:teuthology.orchestra.run.smithi088.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:39:01.273 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 12:6f:29:be:95:48 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:39:01.273 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::106f:29ff:febe:9548/64 scope link 2024-09-16T01:39:01.274 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:39:01.274 INFO:teuthology.orchestra.run.smithi088.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:39:01.274 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T01:39:01.274 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::c63:b4ff:feb3:ad9e/64 scope link 2024-09-16T01:39:01.274 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:39:01.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:39:01.274 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:39:01.335 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-09-16T01:39:01.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:39:01.337 DEBUG:teuthology.orchestra.run.smithi088:> route 2024-09-16T01:39:01.393 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2024-09-16T01:39:01.394 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:39:01.394 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:39:01.394 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:39:01.394 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:39:01.394 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:39:01.394 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:39:01.394 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:39:01.395 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:39:01.395 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:39:01.395 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:39:01.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:39:01.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:39:01.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:39:01.520 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:39:01.535 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T01:39:01.535 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T01:39:01.536 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:39:01.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:39:01.536 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2024-09-16T01:39:01.600 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:39:01.601 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T01:39:01.656 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:39:01.656 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:39:01.657 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:39:01.711 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-16T01:39:01.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:39:01.712 DEBUG:teuthology.orchestra.run.smithi088:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:39:01.795 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:39:01.795 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:39:01.795 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:39:01.803 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:39:01.804 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:39:01.804 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:39:01.804 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:39:01.804 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:39:01.813 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:39:01.814 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:39:01.814 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:39:01.814 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:39:01.814 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:39:01.814 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790" 2024-09-16T01:39:01.814 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2024-09-16T01:39:01.814 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5ace2c91-46f9-482b-8a89-731fbdc171bf" will pass to kernel 2024-09-16T01:39:01.815 INFO:teuthology.orchestra.run.smithi088.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-16T01:39:01.815 INFO:teuthology.orchestra.run.smithi088.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:39:01.815 INFO:teuthology.orchestra.run.smithi088.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:39:01.815 INFO:teuthology.orchestra.run.smithi088.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:39:01.815 INFO:teuthology.orchestra.run.smithi088.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:39:01.816 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:39:01.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:39:01.816 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:39:01.881 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:39:01.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:39:01.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:39:01.977 INFO:teuthology.orchestra.run.smithi088.stdout:client5315 2024-09-16T01:39:01.977 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:39:01.977 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:39:01.993 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat a/ && find a/') 2024-09-16T01:39:02.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:39:02.083 INFO:teuthology.orchestra.run.smithi088.stdout: File: a/ 2024-09-16T01:39:02.083 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 2 Blocks: 0 IO Block: 65536 directory 2024-09-16T01:39:02.083 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 2ah/42d Inode: 1099511627776 Links: 4 2024-09-16T01:39:02.084 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1333/ ubuntu) 2024-09-16T01:39:02.084 INFO:teuthology.orchestra.run.smithi088.stdout:Context: unconfined_u:object_r:unlabeled_t:s0 2024-09-16T01:39:02.084 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-09-16 01:36:46.163362476 +0000 2024-09-16T01:39:02.084 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-09-16 01:36:46.167362396 +0000 2024-09-16T01:39:02.084 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-09-16 01:36:46.167362396 +0000 2024-09-16T01:39:02.084 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-09-16 01:36:46.163362476 +0000 2024-09-16T01:39:02.089 INFO:teuthology.orchestra.run.smithi088.stdout:a/ 2024-09-16T01:39:02.089 INFO:teuthology.orchestra.run.smithi088.stdout:a/c 2024-09-16T01:39:02.089 INFO:teuthology.orchestra.run.smithi088.stdout:a/c/d 2024-09-16T01:39:02.089 INFO:teuthology.orchestra.run.smithi088.stdout:a/b 2024-09-16T01:39:02.089 INFO:teuthology.orchestra.run.smithi088.stdout:a/b/d 2024-09-16T01:39:02.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:39:02.095 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 flush journal 2024-09-16T01:39:02.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.194+0000 7f6758d4b640 1 Processor -- start 2024-09-16T01:39:02.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.194+0000 7f6758d4b640 1 -- start start 2024-09-16T01:39:02.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.195+0000 7f6758d4b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f675415b480 0x7f67540b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.195+0000 7f6758d4b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67540ccff0 con 0x7f67540b60c0 2024-09-16T01:39:02.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.195+0000 7f6758d4b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67540cd2d0 con 0x7f6754072700 2024-09-16T01:39:02.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.195+0000 7f6758d4b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f67540cd5b0 con 0x7f675415b480 2024-09-16T01:39:02.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.195+0000 7f6751d9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f67540b60c0 0x7f6754071ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57360/0 (socket says 172.21.15.88:57360) 2024-09-16T01:39:02.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.195+0000 7f6751d9b640 1 -- 172.21.15.88:0/1970965082 learned_addr learned my addr 172.21.15.88:0/1970965082 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:02.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.195+0000 7f675259c640 1 --2- 172.21.15.88:0/1970965082 >> v2:172.21.15.154:3300/0 conn(0x7f675415b480 0x7f67540b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.196+0000 7f675159a640 1 -- 172.21.15.88:0/1970965082 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1378195340 0 0) 0x7f67540ccff0 con 0x7f67540b60c0 2024-09-16T01:39:02.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.196+0000 7f675159a640 1 -- 172.21.15.88:0/1970965082 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f673c003520 con 0x7f67540b60c0 2024-09-16T01:39:02.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.196+0000 7f675159a640 1 -- 172.21.15.88:0/1970965082 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2802840440 0 0) 0x7f673c003520 con 0x7f67540b60c0 2024-09-16T01:39:02.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.196+0000 7f675159a640 1 -- 172.21.15.88:0/1970965082 >> v2:172.21.15.154:3300/0 conn(0x7f675415b480 msgr2=0x7f67540b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:02.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.196+0000 7f675159a640 1 --2- 172.21.15.88:0/1970965082 >> v2:172.21.15.154:3300/0 conn(0x7f675415b480 0x7f67540b5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.196+0000 7f675159a640 1 -- 172.21.15.88:0/1970965082 >> v1:172.21.15.154:6789/0 conn(0x7f6754072700 legacy=0x7f675415bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:02.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.196+0000 7f675159a640 1 -- 172.21.15.88:0/1970965082 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67540cd890 con 0x7f67540b60c0 2024-09-16T01:39:02.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.196+0000 7f675159a640 1 -- 172.21.15.88:0/1970965082 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f6748001670 con 0x7f67540b60c0 2024-09-16T01:39:02.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.196+0000 7f675159a640 1 -- 172.21.15.88:0/1970965082 >> v1:172.21.15.88:6789/0 conn(0x7f67540b60c0 legacy=0x7f6754071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:02.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.196+0000 7f675159a640 1 --2- 172.21.15.88:0/1970965082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f673c0039b0 0x7f673c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.196+0000 7f675159a640 1 --2- 172.21.15.88:0/1970965082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f673c004800 0x7f673c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.196+0000 7f675159a640 1 --2- 172.21.15.88:0/1970965082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f673c0051c0 0x7f673c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.196+0000 7f675159a640 1 -- 172.21.15.88:0/1970965082 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f67540cd5b0 con 0x7f673c0051c0 2024-09-16T01:39:02.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.196+0000 7f675159a640 1 -- 172.21.15.88:0/1970965082 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f673c00c160 con 0x7f673c0039b0 2024-09-16T01:39:02.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.196+0000 7f675159a640 1 -- 172.21.15.88:0/1970965082 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f673c00e570 con 0x7f673c004800 2024-09-16T01:39:02.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.196+0000 7f6752d9d640 1 --2- 172.21.15.88:0/1970965082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f673c0051c0 0x7f673c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.196+0000 7f675259c640 1 --2- 172.21.15.88:0/1970965082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f673c0039b0 0x7f673c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.196+0000 7f6751d9b640 1 --2- 172.21.15.88:0/1970965082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f673c004800 0x7f673c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.196+0000 7f6752d9d640 1 -- 172.21.15.88:0/1970965082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f673c004800 msgr2=0x7f673c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:02.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.196+0000 7f6752d9d640 1 --2- 172.21.15.88:0/1970965082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f673c004800 0x7f673c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.196+0000 7f6752d9d640 1 -- 172.21.15.88:0/1970965082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f673c0039b0 msgr2=0x7f673c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:02.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.196+0000 7f6752d9d640 1 --2- 172.21.15.88:0/1970965082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f673c0039b0 0x7f673c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.196+0000 7f6752d9d640 1 -- 172.21.15.88:0/1970965082 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f673c003520 con 0x7f673c0051c0 2024-09-16T01:39:02.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.196+0000 7f6751d9b640 1 --2- 172.21.15.88:0/1970965082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f673c004800 0x7f673c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:02.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.197+0000 7f6752d9d640 1 -- 172.21.15.88:0/1970965082 reap_dead start 2024-09-16T01:39:02.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.197+0000 7f6752d9d640 1 --2- 172.21.15.88:0/1970965082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f673c0051c0 0x7f673c00b9c0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f674c001750 tx=0x7f673c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db1e9ed78225a20b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:02.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.197+0000 7f675159a640 1 -- 172.21.15.88:0/1970965082 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c001370 con 0x7f673c0051c0 2024-09-16T01:39:02.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.197+0000 7f675159a640 1 -- 172.21.15.88:0/1970965082 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f674c005310 con 0x7f673c0051c0 2024-09-16T01:39:02.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.197+0000 7f675159a640 1 -- 172.21.15.88:0/1970965082 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c02abe0 con 0x7f673c0051c0 2024-09-16T01:39:02.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.198+0000 7f6758d4b640 1 -- 172.21.15.88:0/1970965082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f673c0051c0 msgr2=0x7f673c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:02.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.198+0000 7f6758d4b640 1 --2- 172.21.15.88:0/1970965082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f673c0051c0 0x7f673c00b9c0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f674c001750 tx=0x7f673c005830 comp rx=0 tx=0).stop 2024-09-16T01:39:02.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.198+0000 7f6758d4b640 1 -- 172.21.15.88:0/1970965082 shutdown_connections 2024-09-16T01:39:02.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.198+0000 7f6758d4b640 1 --2- 172.21.15.88:0/1970965082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f673c0051c0 0x7f673c00b9c0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.198+0000 7f6758d4b640 1 -- 172.21.15.88:0/1970965082 >> 172.21.15.88:0/1970965082 conn(0x7f675406d0f0 msgr2=0x7f675406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:02.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.198+0000 7f6758d4b640 1 -- 172.21.15.88:0/1970965082 shutdown_connections 2024-09-16T01:39:02.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.198+0000 7f6758d4b640 1 -- 172.21.15.88:0/1970965082 wait complete. 2024-09-16T01:39:02.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.199+0000 7f6758d4b640 1 Processor -- start 2024-09-16T01:39:02.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.199+0000 7f6758d4b640 1 -- start start 2024-09-16T01:39:02.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.200+0000 7f6758d4b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f673c0051c0 0x7f675415e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.200+0000 7f6758d4b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6754150d20 0x7f675413f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.200+0000 7f6758d4b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f675413fc00 0x7f6754142180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.200+0000 7f6758d4b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f675415b9e0 con 0x7f6754150d20 2024-09-16T01:39:02.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.200+0000 7f6758d4b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f675415b860 con 0x7f675413fc00 2024-09-16T01:39:02.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.200+0000 7f6758d4b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f67540b5b80 con 0x7f673c0051c0 2024-09-16T01:39:02.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.200+0000 7f6751d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6754150d20 0x7f675413f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.200+0000 7f6751d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6754150d20 0x7f675413f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52702/0 (socket says 172.21.15.88:52702) 2024-09-16T01:39:02.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.200+0000 7f6751d9b640 1 -- 172.21.15.88:0/3286794228 learned_addr learned my addr 172.21.15.88:0/3286794228 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:02.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.200+0000 7f6752d9d640 1 --2- 172.21.15.88:0/3286794228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f675413fc00 0x7f6754142180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.200+0000 7f675259c640 1 --2- 172.21.15.88:0/3286794228 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f673c0051c0 0x7f675415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.200+0000 7f6751d9b640 1 -- 172.21.15.88:0/3286794228 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f673c0051c0 msgr2=0x7f675415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:02.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.200+0000 7f6751d9b640 1 --2- 172.21.15.88:0/3286794228 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f673c0051c0 0x7f675415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.200+0000 7f6751d9b640 1 -- 172.21.15.88:0/3286794228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f675413fc00 msgr2=0x7f6754142180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:02.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.200+0000 7f6751d9b640 1 --2- 172.21.15.88:0/3286794228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f675413fc00 0x7f6754142180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.200+0000 7f6751d9b640 1 -- 172.21.15.88:0/3286794228 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6754150a20 con 0x7f6754150d20 2024-09-16T01:39:02.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.200+0000 7f6752d9d640 1 --2- 172.21.15.88:0/3286794228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f675413fc00 0x7f6754142180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:02.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.200+0000 7f6751d9b640 1 --2- 172.21.15.88:0/3286794228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6754150d20 0x7f675413f6c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6748001200 tx=0x7f674800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:02.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.200+0000 7f6742ffd640 1 -- 172.21.15.88:0/3286794228 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674800c8d0 con 0x7f6754150d20 2024-09-16T01:39:02.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.201+0000 7f6742ffd640 1 -- 172.21.15.88:0/3286794228 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6748018070 con 0x7f6754150d20 2024-09-16T01:39:02.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.201+0000 7f6758d4b640 1 -- 172.21.15.88:0/3286794228 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f675415e970 con 0x7f6754150d20 2024-09-16T01:39:02.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.201+0000 7f6742ffd640 1 -- 172.21.15.88:0/3286794228 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674800cc70 con 0x7f6754150d20 2024-09-16T01:39:02.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.201+0000 7f6758d4b640 1 -- 172.21.15.88:0/3286794228 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67541407b0 con 0x7f6754150d20 2024-09-16T01:39:02.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.202+0000 7f6742ffd640 1 -- 172.21.15.88:0/3286794228 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f674801d460 con 0x7f6754150d20 2024-09-16T01:39:02.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.202+0000 7f6742ffd640 1 --2- 172.21.15.88:0/3286794228 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f673406c2b0 0x7f673406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.203+0000 7f6742ffd640 1 -- 172.21.15.88:0/3286794228 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5881+0+0 (secure 0 0 0) 0x7f674808b3f0 con 0x7f6754150d20 2024-09-16T01:39:02.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.203+0000 7f675259c640 1 --2- 172.21.15.88:0/3286794228 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f673406c2b0 0x7f673406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.203+0000 7f675259c640 1 --2- 172.21.15.88:0/3286794228 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f673406c2b0 0x7f673406e770 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f67541f3620 tx=0x7f674400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:02.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.203+0000 7f675bbc2740 1 Processor -- start 2024-09-16T01:39:02.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.204+0000 7f675bbc2740 1 -- start start 2024-09-16T01:39:02.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.204+0000 7f675bbc2740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56060633a920 0x5606063800c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.204+0000 7f675bbc2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5606063380e0 0x56060633ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.204+0000 7f675bbc2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56060633c0b0 0x5606063351a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.204+0000 7f675bbc2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56060629d6f0 con 0x5606063380e0 2024-09-16T01:39:02.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.204+0000 7f675bbc2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560606245330 con 0x56060633c0b0 2024-09-16T01:39:02.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.204+0000 7f675bbc2740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5606062996b0 con 0x56060633a920 2024-09-16T01:39:02.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.204+0000 7f6751d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5606063380e0 0x56060633ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.204+0000 7f6751d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5606063380e0 0x56060633ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52708/0 (socket says 172.21.15.88:52708) 2024-09-16T01:39:02.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.204+0000 7f6751d9b640 1 -- 172.21.15.88:0/2178520654 learned_addr learned my addr 172.21.15.88:0/2178520654 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:02.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.205+0000 7f675259c640 1 --2- 172.21.15.88:0/2178520654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56060633a920 0x5606063800c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.205+0000 7f6752d9d640 1 --2- 172.21.15.88:0/2178520654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56060633c0b0 0x5606063351a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.205+0000 7f6751d9b640 1 -- 172.21.15.88:0/2178520654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56060633a920 msgr2=0x5606063800c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:02.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.205+0000 7f6751d9b640 1 --2- 172.21.15.88:0/2178520654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56060633a920 0x5606063800c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.205+0000 7f6751d9b640 1 -- 172.21.15.88:0/2178520654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56060633c0b0 msgr2=0x5606063351a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:02.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.205+0000 7f6751d9b640 1 --2- 172.21.15.88:0/2178520654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56060633c0b0 0x5606063351a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.205+0000 7f6751d9b640 1 -- 172.21.15.88:0/2178520654 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56060626b0d0 con 0x5606063380e0 2024-09-16T01:39:02.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.205+0000 7f675259c640 1 --2- 172.21.15.88:0/2178520654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56060633a920 0x5606063800c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:02.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.205+0000 7f6751d9b640 1 --2- 172.21.15.88:0/2178520654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5606063380e0 0x56060633ad00 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f67480592a0 tx=0x7f6748058f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b7fb4b52c6535f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:02.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.205+0000 7f6733fff640 1 -- 172.21.15.88:0/2178520654 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6748061ef0 con 0x5606063380e0 2024-09-16T01:39:02.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.205+0000 7f6733fff640 1 -- 172.21.15.88:0/2178520654 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f674800ce10 con 0x5606063380e0 2024-09-16T01:39:02.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.205+0000 7f6733fff640 1 -- 172.21.15.88:0/2178520654 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674808b0c0 con 0x5606063380e0 2024-09-16T01:39:02.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.206+0000 7f675bbc2740 1 -- 172.21.15.88:0/2178520654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5606063380e0 msgr2=0x56060633ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:02.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.206+0000 7f675bbc2740 1 --2- 172.21.15.88:0/2178520654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5606063380e0 0x56060633ad00 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f67480592a0 tx=0x7f6748058f80 comp rx=0 tx=0).stop 2024-09-16T01:39:02.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.206+0000 7f675bbc2740 1 -- 172.21.15.88:0/2178520654 shutdown_connections 2024-09-16T01:39:02.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.206+0000 7f675bbc2740 1 --2- 172.21.15.88:0/2178520654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56060633c0b0 0x5606063351a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.206+0000 7f675bbc2740 1 --2- 172.21.15.88:0/2178520654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5606063380e0 0x56060633ad00 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.206+0000 7f675bbc2740 1 --2- 172.21.15.88:0/2178520654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56060633a920 0x5606063800c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.206+0000 7f675bbc2740 1 -- 172.21.15.88:0/2178520654 >> 172.21.15.88:0/2178520654 conn(0x560606337370 msgr2=0x56060637f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:02.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.206+0000 7f675bbc2740 1 -- 172.21.15.88:0/2178520654 shutdown_connections 2024-09-16T01:39:02.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.206+0000 7f675bbc2740 1 -- 172.21.15.88:0/2178520654 wait complete. 2024-09-16T01:39:02.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.206+0000 7f675bbc2740 1 -- start start 2024-09-16T01:39:02.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.206+0000 7f675bbc2740 1 Processor -- start 2024-09-16T01:39:02.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.207+0000 7f675bbc2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56060633a920 0x56060636a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.207+0000 7f675bbc2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56060633c0b0 0x560606361090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.207+0000 7f675bbc2740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5606063615d0 0x56060638fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.207+0000 7f675bbc2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x560606336810 con 0x56060633c0b0 2024-09-16T01:39:02.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.207+0000 7f675bbc2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5606062264f0 con 0x56060633a920 2024-09-16T01:39:02.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.207+0000 7f675bbc2740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5606063808f0 con 0x5606063615d0 2024-09-16T01:39:02.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.207+0000 7f675bbc2740 10 client.0 _ll_register_callbacks cb 0x560606247b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:39:02.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.207+0000 7f6751d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56060633c0b0 0x560606361090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.207+0000 7f6751d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56060633c0b0 0x560606361090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52710/0 (socket says 172.21.15.88:52710) 2024-09-16T01:39:02.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.207+0000 7f6751d9b640 1 -- 172.21.15.88:0/301877306 learned_addr learned my addr 172.21.15.88:0/301877306 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:02.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.208+0000 7f675259c640 1 --2- 172.21.15.88:0/301877306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56060633a920 0x56060636a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.208+0000 7f6752d9d640 1 --2- 172.21.15.88:0/301877306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5606063615d0 0x56060638fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.208+0000 7f6751d9b640 1 -- 172.21.15.88:0/301877306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5606063615d0 msgr2=0x56060638fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:02.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.208+0000 7f6751d9b640 1 --2- 172.21.15.88:0/301877306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5606063615d0 0x56060638fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.208+0000 7f6751d9b640 1 -- 172.21.15.88:0/301877306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56060633a920 msgr2=0x56060636a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:02.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.208+0000 7f6751d9b640 1 --2- 172.21.15.88:0/301877306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56060633a920 0x56060636a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.208+0000 7f6751d9b640 1 -- 172.21.15.88:0/301877306 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56060607a7d0 con 0x56060633c0b0 2024-09-16T01:39:02.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.208+0000 7f6752d9d640 1 --2- 172.21.15.88:0/301877306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5606063615d0 0x56060638fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:02.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.208+0000 7f6751d9b640 1 --2- 172.21.15.88:0/301877306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56060633c0b0 0x560606361090 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f674800c4b0 tx=0x7f6748000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=488b518442fe73d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:02.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.208+0000 7f67317fa640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:39:02.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.208+0000 7f67317fa640 1 -- 172.21.15.88:0/301877306 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6748075030 con 0x56060633c0b0 2024-09-16T01:39:02.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.208+0000 7f67317fa640 1 -- 172.21.15.88:0/301877306 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(60..60 src has 1..60) ==== 5881+0+0 (secure 0 0 0) 0x7f6748078320 con 0x56060633c0b0 2024-09-16T01:39:02.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.208+0000 7f675bbc2740 1 -- 172.21.15.88:0/301877306 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f674800ce10 con 0x56060633c0b0 2024-09-16T01:39:02.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.208+0000 7f67317fa640 1 -- 172.21.15.88:0/301877306 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f6748057610 con 0x56060633c0b0 2024-09-16T01:39:02.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.208+0000 7f675bbc2740 10 client.6255 fetch_fsmap learned FSMap version 51 2024-09-16T01:39:02.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.208+0000 7f675bbc2740 1 -- 172.21.15.88:0/301877306 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x560605fb03f0 con 0x56060633c0b0 2024-09-16T01:39:02.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.208+0000 7f67317fa640 1 -- 172.21.15.88:0/301877306 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f674800ce10 con 0x56060633c0b0 2024-09-16T01:39:02.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.208+0000 7f67317fa640 1 -- 172.21.15.88:0/301877306 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 51) ==== 3502+0+0 (secure 0 0 0) 0x7f6748055980 con 0x56060633c0b0 2024-09-16T01:39:02.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.208+0000 7f675bbc2740 10 client.6255 fetch_fsmap finished waiting for FSMap version 51 2024-09-16T01:39:02.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.208+0000 7f675bbc2740 10 client.6255 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-09-16T01:39:02.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.208+0000 7f675bbc2740 20 client.6255 populate_metadata read hostname 'smithi088' 2024-09-16T01:39:02.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.208+0000 7f675bbc2740 1 --2- 172.21.15.88:0/301877306 >> [v2:172.21.15.88:6834/998283126,v1:172.21.15.88:6836/998283126] conn(0x5606063672b0 0x560606367690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.208+0000 7f675bbc2740 4 client.6255 mds_command: new command op to 5859 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:39:02.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.208+0000 7f675bbc2740 1 -- 172.21.15.88:0/301877306 --> [v2:172.21.15.88:6834/998283126,v1:172.21.15.88:6836/998283126] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5606062aae20 con 0x5606063672b0 2024-09-16T01:39:02.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.209+0000 7f675259c640 1 --2- 172.21.15.88:0/301877306 >> [v2:172.21.15.88:6834/998283126,v1:172.21.15.88:6836/998283126] conn(0x5606063672b0 0x560606367690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.209+0000 7f67317fa640 10 client.6255 ms_handle_connect on v2:172.21.15.88:6834/998283126 2024-09-16T01:39:02.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.209+0000 7f675259c640 1 --2- 172.21.15.88:0/301877306 >> [v2:172.21.15.88:6834/998283126,v1:172.21.15.88:6836/998283126] conn(0x5606063672b0 0x560606367690 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd08fbf07d88583c server_cookie=30c450728decdcd in_seq=0 out_seq=0 2024-09-16T01:39:02.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.209+0000 7f67317fa640 1 -- 172.21.15.88:0/301877306 <== mds.0 v2:172.21.15.88:6834/998283126 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5606062aae20 con 0x5606063672b0 2024-09-16T01:39:02.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.209+0000 7f67317fa640 10 client.6255 handle_command_reply: tid=0 2024-09-16T01:39:02.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.209+0000 7f675bbc2740 1 client.6255 shutdown 2024-09-16T01:39:02.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.210+0000 7f675bbc2740 1 -- 172.21.15.88:0/301877306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56060633c0b0 msgr2=0x560606361090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:02.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.210+0000 7f675bbc2740 1 --2- 172.21.15.88:0/301877306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56060633c0b0 0x560606361090 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f674800c4b0 tx=0x7f6748000ed0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.210+0000 7f675bbc2740 1 -- 172.21.15.88:0/301877306 shutdown_connections 2024-09-16T01:39:02.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.210+0000 7f675bbc2740 1 --2- 172.21.15.88:0/301877306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5606063615d0 0x56060638fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.210+0000 7f675bbc2740 1 --2- 172.21.15.88:0/301877306 >> [v2:172.21.15.88:6834/998283126,v1:172.21.15.88:6836/998283126] conn(0x5606063672b0 0x560606367690 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.210+0000 7f675bbc2740 1 --2- 172.21.15.88:0/301877306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56060633c0b0 0x560606361090 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.210+0000 7f675bbc2740 1 --2- 172.21.15.88:0/301877306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56060633a920 0x56060636a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.210+0000 7f67317fa640 0 client.6255 ms_handle_reset on v2:172.21.15.88:6834/998283126 2024-09-16T01:39:02.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.210+0000 7f675bbc2740 1 -- 172.21.15.88:0/301877306 >> 172.21.15.88:0/301877306 conn(0x5606063380e0 msgr2=0x560606338680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:02.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.210+0000 7f675bbc2740 1 -- 172.21.15.88:0/301877306 shutdown_connections 2024-09-16T01:39:02.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.210+0000 7f675bbc2740 1 -- 172.21.15.88:0/301877306 wait complete. 2024-09-16T01:39:02.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.210+0000 7f675bbc2740 20 client.6255 trim_cache size 0 max 16384 2024-09-16T01:39:02.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.224+0000 7f675bbc2740 1 Processor -- start 2024-09-16T01:39:02.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.224+0000 7f675bbc2740 1 -- start start 2024-09-16T01:39:02.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.225+0000 7f675bbc2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56060633a920 0x56060637ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.225+0000 7f675bbc2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56060633c0b0 0x56060637e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.225+0000 7f675bbc2740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5606063615d0 0x56060637ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.225+0000 7f675bbc2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f67541fce90 con 0x56060633c0b0 2024-09-16T01:39:02.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.225+0000 7f675bbc2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f67541fbf10 con 0x56060633a920 2024-09-16T01:39:02.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.225+0000 7f675bbc2740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f675415c4c0 con 0x5606063615d0 2024-09-16T01:39:02.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.225+0000 7f6751d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56060633c0b0 0x56060637e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.225+0000 7f6751d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56060633c0b0 0x56060637e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52712/0 (socket says 172.21.15.88:52712) 2024-09-16T01:39:02.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.225+0000 7f6751d9b640 1 -- 172.21.15.88:0/3189065033 learned_addr learned my addr 172.21.15.88:0/3189065033 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:02.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.225+0000 7f6751d9b640 1 -- 172.21.15.88:0/3189065033 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5606063615d0 msgr2=0x56060637ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:02.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.225+0000 7f6751d9b640 1 --2- 172.21.15.88:0/3189065033 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5606063615d0 0x56060637ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.225+0000 7f6751d9b640 1 -- 172.21.15.88:0/3189065033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56060633a920 msgr2=0x56060637ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:02.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.225+0000 7f6751d9b640 1 --2- 172.21.15.88:0/3189065033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56060633a920 0x56060637ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.225+0000 7f6751d9b640 1 -- 172.21.15.88:0/3189065033 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560606361400 con 0x56060633c0b0 2024-09-16T01:39:02.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.225+0000 7f6751d9b640 1 --2- 172.21.15.88:0/3189065033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56060633c0b0 0x56060637e770 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f67480619c0 tx=0x7f674806f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b000169aeb9b1ee server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:02.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.225+0000 7f67337fe640 1 -- 172.21.15.88:0/3189065033 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6748075070 con 0x56060633c0b0 2024-09-16T01:39:02.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.225+0000 7f67337fe640 1 -- 172.21.15.88:0/3189065033 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f674800ce10 con 0x56060633c0b0 2024-09-16T01:39:02.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.225+0000 7f67337fe640 1 -- 172.21.15.88:0/3189065033 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6748058e50 con 0x56060633c0b0 2024-09-16T01:39:02.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.226+0000 7f675bbc2740 1 -- 172.21.15.88:0/3189065033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56060633c0b0 msgr2=0x56060637e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:02.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.226+0000 7f675bbc2740 1 --2- 172.21.15.88:0/3189065033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56060633c0b0 0x56060637e770 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f67480619c0 tx=0x7f674806f000 comp rx=0 tx=0).stop 2024-09-16T01:39:02.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.226+0000 7f675bbc2740 1 -- 172.21.15.88:0/3189065033 shutdown_connections 2024-09-16T01:39:02.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.226+0000 7f675bbc2740 1 --2- 172.21.15.88:0/3189065033 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5606063615d0 0x56060637ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.226+0000 7f675bbc2740 1 --2- 172.21.15.88:0/3189065033 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56060633c0b0 0x56060637e770 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.226+0000 7f675bbc2740 1 --2- 172.21.15.88:0/3189065033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56060633a920 0x56060637ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.226+0000 7f675bbc2740 1 -- 172.21.15.88:0/3189065033 >> 172.21.15.88:0/3189065033 conn(0x5606063380e0 msgr2=0x560606338710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:02.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.226+0000 7f675bbc2740 1 -- 172.21.15.88:0/3189065033 shutdown_connections 2024-09-16T01:39:02.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.226+0000 7f675bbc2740 1 -- 172.21.15.88:0/3189065033 wait complete. 2024-09-16T01:39:02.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.226+0000 7f675bbc2740 1 -- start start 2024-09-16T01:39:02.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.227+0000 7f675bbc2740 1 Processor -- start 2024-09-16T01:39:02.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.227+0000 7f675bbc2740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56060633a920 0x56060637bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.227+0000 7f675bbc2740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56060633c0b0 0x56060637c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.227+0000 7f675bbc2740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5606063615d0 0x5606063a3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.227+0000 7f675bbc2740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x560606338e20 con 0x56060633c0b0 2024-09-16T01:39:02.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.227+0000 7f675bbc2740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f67541fb910 con 0x56060633a920 2024-09-16T01:39:02.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.227+0000 7f675bbc2740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x560606366740 con 0x5606063615d0 2024-09-16T01:39:02.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.227+0000 7f675bbc2740 10 client.0 _ll_register_callbacks cb 0x7f6748058d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:39:02.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.228+0000 7f6751d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56060633c0b0 0x56060637c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.228+0000 7f6751d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56060633c0b0 0x56060637c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52722/0 (socket says 172.21.15.88:52722) 2024-09-16T01:39:02.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.228+0000 7f6751d9b640 1 -- 172.21.15.88:0/778580609 learned_addr learned my addr 172.21.15.88:0/778580609 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:02.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.228+0000 7f675259c640 1 --2- 172.21.15.88:0/778580609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56060633a920 0x56060637bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.228+0000 7f6752d9d640 1 --2- 172.21.15.88:0/778580609 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5606063615d0 0x5606063a3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.228+0000 7f6751d9b640 1 -- 172.21.15.88:0/778580609 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5606063615d0 msgr2=0x5606063a3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:02.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.228+0000 7f6751d9b640 1 --2- 172.21.15.88:0/778580609 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5606063615d0 0x5606063a3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.228+0000 7f6751d9b640 1 -- 172.21.15.88:0/778580609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56060633a920 msgr2=0x56060637bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:02.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.228+0000 7f6751d9b640 1 --2- 172.21.15.88:0/778580609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56060633a920 0x56060637bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.228+0000 7f6751d9b640 1 -- 172.21.15.88:0/778580609 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560605f700f0 con 0x56060633c0b0 2024-09-16T01:39:02.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.228+0000 7f6751d9b640 1 --2- 172.21.15.88:0/778580609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56060633c0b0 0x56060637c200 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f67480897d0 tx=0x7f67480556e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c19f5eaa85917e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:02.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.228+0000 7f67317fa640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:39:02.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.228+0000 7f67317fa640 1 -- 172.21.15.88:0/778580609 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6748075040 con 0x56060633c0b0 2024-09-16T01:39:02.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.228+0000 7f67317fa640 1 -- 172.21.15.88:0/778580609 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(60..60 src has 1..60) ==== 5881+0+0 (secure 0 0 0) 0x7f6748061c70 con 0x56060633c0b0 2024-09-16T01:39:02.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.228+0000 7f675bbc2740 1 -- 172.21.15.88:0/778580609 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f674800ce10 con 0x56060633c0b0 2024-09-16T01:39:02.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.228+0000 7f67317fa640 1 -- 172.21.15.88:0/778580609 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f674804b050 con 0x56060633c0b0 2024-09-16T01:39:02.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.228+0000 7f675bbc2740 10 client.6261 fetch_fsmap learned FSMap version 51 2024-09-16T01:39:02.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.228+0000 7f675bbc2740 1 -- 172.21.15.88:0/778580609 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x560605f700f0 con 0x56060633c0b0 2024-09-16T01:39:02.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.228+0000 7f67317fa640 1 -- 172.21.15.88:0/778580609 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f674800ce10 con 0x56060633c0b0 2024-09-16T01:39:02.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.228+0000 7f67317fa640 1 -- 172.21.15.88:0/778580609 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 51) ==== 3502+0+0 (secure 0 0 0) 0x7f6748059570 con 0x56060633c0b0 2024-09-16T01:39:02.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.229+0000 7f675bbc2740 10 client.6261 fetch_fsmap finished waiting for FSMap version 51 2024-09-16T01:39:02.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.229+0000 7f675bbc2740 10 client.6261 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-09-16T01:39:02.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.229+0000 7f675bbc2740 20 client.6261 populate_metadata read hostname 'smithi088' 2024-09-16T01:39:02.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.229+0000 7f675bbc2740 1 --2- 172.21.15.88:0/778580609 >> [v2:172.21.15.88:6834/998283126,v1:172.21.15.88:6836/998283126] conn(0x56060638eda0 0x5606063a3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.229+0000 7f675bbc2740 4 client.6261 mds_command: new command op to 5859 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:39:02.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.229+0000 7f675bbc2740 1 -- 172.21.15.88:0/778580609 --> [v2:172.21.15.88:6834/998283126,v1:172.21.15.88:6836/998283126] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56060636b960 con 0x56060638eda0 2024-09-16T01:39:02.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.229+0000 7f675259c640 1 --2- 172.21.15.88:0/778580609 >> [v2:172.21.15.88:6834/998283126,v1:172.21.15.88:6836/998283126] conn(0x56060638eda0 0x5606063a3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.229+0000 7f67317fa640 10 client.6261 ms_handle_connect on v2:172.21.15.88:6834/998283126 2024-09-16T01:39:02.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.229+0000 7f675259c640 1 --2- 172.21.15.88:0/778580609 >> [v2:172.21.15.88:6834/998283126,v1:172.21.15.88:6836/998283126] conn(0x56060638eda0 0x5606063a3540 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47bab4204625c16c server_cookie=7e71a0c5b21f952e in_seq=0 out_seq=0 2024-09-16T01:39:02.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.240+0000 7f67317fa640 1 -- 172.21.15.88:0/778580609 <== mds.0 v2:172.21.15.88:6834/998283126 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56060636b960 con 0x56060638eda0 2024-09-16T01:39:02.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.240+0000 7f67317fa640 10 client.6261 handle_command_reply: tid=0 2024-09-16T01:39:02.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.240+0000 7f675bbc2740 1 client.6261 shutdown 2024-09-16T01:39:02.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.240+0000 7f675bbc2740 1 -- 172.21.15.88:0/778580609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56060633c0b0 msgr2=0x56060637c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:02.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.240+0000 7f675bbc2740 1 --2- 172.21.15.88:0/778580609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56060633c0b0 0x56060637c200 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f67480897d0 tx=0x7f67480556e0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.240+0000 7f675bbc2740 1 -- 172.21.15.88:0/778580609 shutdown_connections 2024-09-16T01:39:02.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.240+0000 7f675bbc2740 1 --2- 172.21.15.88:0/778580609 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5606063615d0 0x5606063a3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.240+0000 7f675bbc2740 1 --2- 172.21.15.88:0/778580609 >> [v2:172.21.15.88:6834/998283126,v1:172.21.15.88:6836/998283126] conn(0x56060638eda0 0x5606063a3540 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.240+0000 7f675bbc2740 1 --2- 172.21.15.88:0/778580609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56060633c0b0 0x56060637c200 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.240+0000 7f67317fa640 0 client.6261 ms_handle_reset on v2:172.21.15.88:6834/998283126 2024-09-16T01:39:02.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.240+0000 7f675bbc2740 1 --2- 172.21.15.88:0/778580609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56060633a920 0x56060637bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.240+0000 7f675bbc2740 1 -- 172.21.15.88:0/778580609 >> 172.21.15.88:0/778580609 conn(0x5606063380e0 msgr2=0x560606399600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:02.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.240+0000 7f675bbc2740 1 -- 172.21.15.88:0/778580609 shutdown_connections 2024-09-16T01:39:02.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.240+0000 7f675bbc2740 1 -- 172.21.15.88:0/778580609 wait complete. 2024-09-16T01:39:02.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.240+0000 7f675bbc2740 20 client.6261 trim_cache size 0 max 16384 2024-09-16T01:39:02.243 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:39:02.243 INFO:teuthology.orchestra.run.smithi088.stdout: "message": "", 2024-09-16T01:39:02.243 INFO:teuthology.orchestra.run.smithi088.stdout: "return_code": 0 2024-09-16T01:39:02.244 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:39:02.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.241+0000 7f6758d4b640 1 -- 172.21.15.88:0/3286794228 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f673406c2b0 msgr2=0x7f673406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:02.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.241+0000 7f6758d4b640 1 --2- 172.21.15.88:0/3286794228 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f673406c2b0 0x7f673406e770 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f67541f3620 tx=0x7f674400b040 comp rx=0 tx=0).stop 2024-09-16T01:39:02.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.241+0000 7f6758d4b640 1 -- 172.21.15.88:0/3286794228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6754150d20 msgr2=0x7f675413f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:02.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.241+0000 7f6758d4b640 1 --2- 172.21.15.88:0/3286794228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6754150d20 0x7f675413f6c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6748001200 tx=0x7f674800ec60 comp rx=0 tx=0).stop 2024-09-16T01:39:02.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.241+0000 7f6758d4b640 1 -- 172.21.15.88:0/3286794228 shutdown_connections 2024-09-16T01:39:02.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.241+0000 7f6758d4b640 1 --2- 172.21.15.88:0/3286794228 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f673406c2b0 0x7f673406e770 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.241+0000 7f6758d4b640 1 --2- 172.21.15.88:0/3286794228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f675413fc00 0x7f6754142180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.241+0000 7f6758d4b640 1 --2- 172.21.15.88:0/3286794228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6754150d20 0x7f675413f6c0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.241+0000 7f6758d4b640 1 --2- 172.21.15.88:0/3286794228 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f673c0051c0 0x7f675415e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.241+0000 7f6758d4b640 1 -- 172.21.15.88:0/3286794228 >> 172.21.15.88:0/3286794228 conn(0x7f675406d0f0 msgr2=0x7f67540ce2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:02.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.241+0000 7f6758d4b640 1 -- 172.21.15.88:0/3286794228 shutdown_connections 2024-09-16T01:39:02.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.241+0000 7f6758d4b640 1 -- 172.21.15.88:0/3286794228 wait complete. 2024-09-16T01:39:02.253 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T01:39:02.253 DEBUG:teuthology.orchestra.run.smithi088:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T01:39:02.300 DEBUG:teuthology.orchestra.run.smithi154:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T01:39:02.347 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T01:39:02.347 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T01:39:02.347 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:39:02.347 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:39:02.363 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:39:02.363 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:39:02.419 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:39:02.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.560+0000 7f3864e82640 1 Processor -- start 2024-09-16T01:39:02.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.560+0000 7f3864e82640 1 -- start start 2024-09-16T01:39:02.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.561+0000 7f3864e82640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38600b4e60 0x7f38600d0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.561+0000 7f3864e82640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f386015cb20 0x7f386015cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.561+0000 7f3864e82640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38600cd9b0 con 0x7f38600b5c00 2024-09-16T01:39:02.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.561+0000 7f3864e82640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f38600ce5c0 con 0x7f386015cb20 2024-09-16T01:39:02.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.561+0000 7f3864e82640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f38600cdc90 con 0x7f38600b4e60 2024-09-16T01:39:02.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.561+0000 7f385ed9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f386015cb20 0x7f386015cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.561+0000 7f385dd9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f38600b5c00 0x7f38600b5240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57370/0 (socket says 172.21.15.88:57370) 2024-09-16T01:39:02.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.561+0000 7f385dd9b640 1 -- 172.21.15.88:0/2639558316 learned_addr learned my addr 172.21.15.88:0/2639558316 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:02.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.561+0000 7f385ed9d640 1 -- 172.21.15.88:0/2639558316 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38600b4e60 msgr2=0x7f38600d0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:02.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.561+0000 7f385e59c640 1 --2- 172.21.15.88:0/2639558316 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38600b4e60 0x7f38600d0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.561+0000 7f385d59a640 1 -- 172.21.15.88:0/2639558316 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3401310105 0 0) 0x7f38600cd9b0 con 0x7f38600b5c00 2024-09-16T01:39:02.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.561+0000 7f385ed9d640 1 --2- 172.21.15.88:0/2639558316 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38600b4e60 0x7f38600d0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.561+0000 7f385ed9d640 1 -- 172.21.15.88:0/2639558316 >> v1:172.21.15.88:6789/0 conn(0x7f38600b5c00 legacy=0x7f38600b5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:02.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.561+0000 7f385ed9d640 1 -- 172.21.15.88:0/2639558316 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38600cdf70 con 0x7f386015cb20 2024-09-16T01:39:02.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.561+0000 7f385e59c640 1 --2- 172.21.15.88:0/2639558316 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38600b4e60 0x7f38600d0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:02.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.562+0000 7f385ed9d640 1 --2- 172.21.15.88:0/2639558316 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f386015cb20 0x7f386015cfb0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f3858009d90 tx=0x7f385802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbccbcbbcdc6344b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:02.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.562+0000 7f385d59a640 1 -- 172.21.15.88:0/2639558316 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3858007740 con 0x7f386015cb20 2024-09-16T01:39:02.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.562+0000 7f385d59a640 1 -- 172.21.15.88:0/2639558316 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38580078e0 con 0x7f386015cb20 2024-09-16T01:39:02.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.562+0000 7f385d59a640 1 -- 172.21.15.88:0/2639558316 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3858007cb0 con 0x7f386015cb20 2024-09-16T01:39:02.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.563+0000 7f3864e82640 1 -- 172.21.15.88:0/2639558316 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f386015cb20 msgr2=0x7f386015cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:02.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.563+0000 7f3864e82640 1 --2- 172.21.15.88:0/2639558316 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f386015cb20 0x7f386015cfb0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f3858009d90 tx=0x7f385802a660 comp rx=0 tx=0).stop 2024-09-16T01:39:02.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.563+0000 7f3864e82640 1 -- 172.21.15.88:0/2639558316 shutdown_connections 2024-09-16T01:39:02.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.563+0000 7f3864e82640 1 --2- 172.21.15.88:0/2639558316 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f386015cb20 0x7f386015cfb0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.563+0000 7f3864e82640 1 --2- 172.21.15.88:0/2639558316 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38600b4e60 0x7f38600d0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.563+0000 7f3864e82640 1 -- 172.21.15.88:0/2639558316 >> 172.21.15.88:0/2639558316 conn(0x7f386006cb50 msgr2=0x7f386006cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:02.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.563+0000 7f3864e82640 1 -- 172.21.15.88:0/2639558316 shutdown_connections 2024-09-16T01:39:02.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.563+0000 7f3864e82640 1 -- 172.21.15.88:0/2639558316 wait complete. 2024-09-16T01:39:02.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.564+0000 7f3864e82640 1 Processor -- start 2024-09-16T01:39:02.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.564+0000 7f3864e82640 1 -- start start 2024-09-16T01:39:02.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.565+0000 7f3864e82640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38600b4e60 0x7f386014fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.565+0000 7f3864e82640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38600b5c00 0x7f3860150f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.565+0000 7f3864e82640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f386015cb20 0x7f3860151450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.565+0000 7f3864e82640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f386015fd80 con 0x7f386015cb20 2024-09-16T01:39:02.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.565+0000 7f3864e82640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f386015fc00 con 0x7f38600b4e60 2024-09-16T01:39:02.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.565+0000 7f3864e82640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f38600b6060 con 0x7f38600b5c00 2024-09-16T01:39:02.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.565+0000 7f385ed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f386015cb20 0x7f3860151450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.565+0000 7f385ed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f386015cb20 0x7f3860151450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52742/0 (socket says 172.21.15.88:52742) 2024-09-16T01:39:02.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.565+0000 7f385ed9d640 1 -- 172.21.15.88:0/1078253226 learned_addr learned my addr 172.21.15.88:0/1078253226 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:02.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.565+0000 7f385e59c640 1 --2- 172.21.15.88:0/1078253226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38600b4e60 0x7f386014fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.565+0000 7f385dd9b640 1 --2- 172.21.15.88:0/1078253226 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38600b5c00 0x7f3860150f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.565+0000 7f385ed9d640 1 -- 172.21.15.88:0/1078253226 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38600b5c00 msgr2=0x7f3860150f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:02.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.565+0000 7f385ed9d640 1 --2- 172.21.15.88:0/1078253226 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38600b5c00 0x7f3860150f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.565+0000 7f385ed9d640 1 -- 172.21.15.88:0/1078253226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38600b4e60 msgr2=0x7f386014fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:02.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.565+0000 7f385ed9d640 1 --2- 172.21.15.88:0/1078253226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38600b4e60 0x7f386014fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.565+0000 7f385ed9d640 1 -- 172.21.15.88:0/1078253226 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f386014eb80 con 0x7f386015cb20 2024-09-16T01:39:02.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.565+0000 7f385e59c640 1 --2- 172.21.15.88:0/1078253226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38600b4e60 0x7f386014fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:02.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.565+0000 7f385dd9b640 1 --2- 172.21.15.88:0/1078253226 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38600b5c00 0x7f3860150f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:02.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.566+0000 7f385ed9d640 1 --2- 172.21.15.88:0/1078253226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f386015cb20 0x7f3860151450 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f3858009d90 tx=0x7f3858004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:02.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.566+0000 7f384effd640 1 -- 172.21.15.88:0/1078253226 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385803f070 con 0x7f386015cb20 2024-09-16T01:39:02.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.566+0000 7f384effd640 1 -- 172.21.15.88:0/1078253226 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3858031c40 con 0x7f386015cb20 2024-09-16T01:39:02.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.566+0000 7f3864e82640 1 -- 172.21.15.88:0/1078253226 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f386014ee60 con 0x7f386015cb20 2024-09-16T01:39:02.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.566+0000 7f384effd640 1 -- 172.21.15.88:0/1078253226 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3858003990 con 0x7f386015cb20 2024-09-16T01:39:02.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.566+0000 7f3864e82640 1 -- 172.21.15.88:0/1078253226 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f386014f420 con 0x7f386015cb20 2024-09-16T01:39:02.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.567+0000 7f3864e82640 1 -- 172.21.15.88:0/1078253226 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38600d0760 con 0x7f386015cb20 2024-09-16T01:39:02.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.567+0000 7f384effd640 1 -- 172.21.15.88:0/1078253226 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3858003b30 con 0x7f386015cb20 2024-09-16T01:39:02.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.568+0000 7f384effd640 1 --2- 172.21.15.88:0/1078253226 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f384006c2b0 0x7f384006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:02.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.568+0000 7f385e59c640 1 --2- 172.21.15.88:0/1078253226 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f384006c2b0 0x7f384006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:02.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.568+0000 7f384effd640 1 -- 172.21.15.88:0/1078253226 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5881+0+0 (secure 0 0 0) 0x7f38580ac3e0 con 0x7f386015cb20 2024-09-16T01:39:02.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.568+0000 7f385e59c640 1 --2- 172.21.15.88:0/1078253226 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f384006c2b0 0x7f384006e770 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3850006fd0 tx=0x7f3850008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:02.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.572+0000 7f384effd640 1 -- 172.21.15.88:0/1078253226 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f385807aaa0 con 0x7f386015cb20 2024-09-16T01:39:02.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.749+0000 7f3864e82640 1 -- 172.21.15.88:0/1078253226 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f38600b2410 con 0x7f386015cb20 2024-09-16T01:39:02.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.750+0000 7f384effd640 1 -- 172.21.15.88:0/1078253226 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v60) ==== 80+0+242 (secure 0 0 0) 0x7f385807a3a0 con 0x7f386015cb20 2024-09-16T01:39:02.751 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/4130074102 2024-09-17T01:37:24.240722+0000 2024-09-16T01:39:02.751 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/4130074102 2024-09-17T01:37:24.240722+0000 2024-09-16T01:39:02.751 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/574169866 2024-09-17T01:36:48.820862+0000 2024-09-16T01:39:02.752 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/574169866 2024-09-17T01:36:48.820862+0000 2024-09-16T01:39:02.752 INFO:teuthology.orchestra.run.smithi088.stderr:listed 4 entries 2024-09-16T01:39:02.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.752+0000 7f3864e82640 1 -- 172.21.15.88:0/1078253226 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f384006c2b0 msgr2=0x7f384006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:02.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.752+0000 7f3864e82640 1 --2- 172.21.15.88:0/1078253226 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f384006c2b0 0x7f384006e770 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3850006fd0 tx=0x7f3850008040 comp rx=0 tx=0).stop 2024-09-16T01:39:02.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.752+0000 7f3864e82640 1 -- 172.21.15.88:0/1078253226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f386015cb20 msgr2=0x7f3860151450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:02.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.752+0000 7f3864e82640 1 --2- 172.21.15.88:0/1078253226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f386015cb20 0x7f3860151450 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f3858009d90 tx=0x7f3858004a40 comp rx=0 tx=0).stop 2024-09-16T01:39:02.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.752+0000 7f3864e82640 1 -- 172.21.15.88:0/1078253226 shutdown_connections 2024-09-16T01:39:02.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.752+0000 7f3864e82640 1 --2- 172.21.15.88:0/1078253226 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f384006c2b0 0x7f384006e770 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.752+0000 7f3864e82640 1 --2- 172.21.15.88:0/1078253226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f386015cb20 0x7f3860151450 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.752+0000 7f3864e82640 1 --2- 172.21.15.88:0/1078253226 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38600b5c00 0x7f3860150f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.752+0000 7f3864e82640 1 --2- 172.21.15.88:0/1078253226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38600b4e60 0x7f386014fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:02.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.752+0000 7f3864e82640 1 -- 172.21.15.88:0/1078253226 >> 172.21.15.88:0/1078253226 conn(0x7f386006cb50 msgr2=0x7f386006fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:02.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.752+0000 7f3864e82640 1 -- 172.21.15.88:0/1078253226 shutdown_connections 2024-09-16T01:39:02.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:02.752+0000 7f3864e82640 1 -- 172.21.15.88:0/1078253226 wait complete. 2024-09-16T01:39:02.764 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-16T01:39:02.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:39:02.764 DEBUG:teuthology.orchestra.run.smithi088:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-16T01:39:02.793 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:39:02.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:39:02.794 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:39:02.849 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T01:39:02.849 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:39:02.849 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:39:02.869 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056026,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1347 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T01:39:02.869 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T01:39:02.869 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:39:02.869 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:39:02.905 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:39:02.905 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T01:39:02.905 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:39:02.905 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:39:02.928 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056026,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1347 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T01:39:02.928 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:39:03.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.041+0000 7f1e75082640 1 Processor -- start 2024-09-16T01:39:03.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.041+0000 7f1e75082640 1 -- start start 2024-09-16T01:39:03.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.042+0000 7f1e75082640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e70155ac0 0x7f1e700ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:03.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.042+0000 7f1e75082640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1e7015c2a0 0x7f1e700cf570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:03.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.042+0000 7f1e75082640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e700b4920 con 0x7f1e701556e0 2024-09-16T01:39:03.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.042+0000 7f1e75082640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f1e700b54f0 con 0x7f1e7015c2a0 2024-09-16T01:39:03.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.042+0000 7f1e75082640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1e7015c730 con 0x7f1e70155ac0 2024-09-16T01:39:03.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.042+0000 7f1e6f59e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1e7015c2a0 0x7f1e700cf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:03.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.042+0000 7f1e6f59e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1e7015c2a0 0x7f1e700cf570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52754/0 (socket says 172.21.15.88:52754) 2024-09-16T01:39:03.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.042+0000 7f1e6f59e640 1 -- 172.21.15.88:0/835025721 learned_addr learned my addr 172.21.15.88:0/835025721 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:03.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.042+0000 7f1e67fff640 1 --2- 172.21.15.88:0/835025721 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e70155ac0 0x7f1e700ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:03.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.042+0000 7f1e6f59e640 1 -- 172.21.15.88:0/835025721 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e70155ac0 msgr2=0x7f1e700ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:03.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.042+0000 7f1e6f59e640 1 --2- 172.21.15.88:0/835025721 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e70155ac0 0x7f1e700ccf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.042+0000 7f1e6f59e640 1 -- 172.21.15.88:0/835025721 >> v1:172.21.15.154:6789/0 conn(0x7f1e701556e0 legacy=0x7f1e700cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:03.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.042+0000 7f1e6f59e640 1 -- 172.21.15.88:0/835025721 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e700b4bc0 con 0x7f1e7015c2a0 2024-09-16T01:39:03.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.043+0000 7f1e6f59e640 1 --2- 172.21.15.88:0/835025721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e7015c2a0 0x7f1e700cf570 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800d150 tx=0x7f1e6802c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=903014d0488f1537 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:03.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.043+0000 7f1e6e59c640 1 -- 172.21.15.88:0/835025721 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e68009600 con 0x7f1e7015c2a0 2024-09-16T01:39:03.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.043+0000 7f1e6e59c640 1 -- 172.21.15.88:0/835025721 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e68037070 con 0x7f1e7015c2a0 2024-09-16T01:39:03.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.043+0000 7f1e6e59c640 1 -- 172.21.15.88:0/835025721 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e680099c0 con 0x7f1e7015c2a0 2024-09-16T01:39:03.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.044+0000 7f1e75082640 1 -- 172.21.15.88:0/835025721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e7015c2a0 msgr2=0x7f1e700cf570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:03.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.044+0000 7f1e75082640 1 --2- 172.21.15.88:0/835025721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e7015c2a0 0x7f1e700cf570 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800d150 tx=0x7f1e6802c930 comp rx=0 tx=0).stop 2024-09-16T01:39:03.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.044+0000 7f1e75082640 1 -- 172.21.15.88:0/835025721 shutdown_connections 2024-09-16T01:39:03.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.044+0000 7f1e75082640 1 --2- 172.21.15.88:0/835025721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e7015c2a0 0x7f1e700cf570 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.044+0000 7f1e75082640 1 --2- 172.21.15.88:0/835025721 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e70155ac0 0x7f1e700ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.044+0000 7f1e75082640 1 -- 172.21.15.88:0/835025721 >> 172.21.15.88:0/835025721 conn(0x7f1e7006bc10 msgr2=0x7f1e7006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:03.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.044+0000 7f1e75082640 1 -- 172.21.15.88:0/835025721 shutdown_connections 2024-09-16T01:39:03.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.044+0000 7f1e75082640 1 -- 172.21.15.88:0/835025721 wait complete. 2024-09-16T01:39:03.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.045+0000 7f1e75082640 1 Processor -- start 2024-09-16T01:39:03.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.045+0000 7f1e75082640 1 -- start start 2024-09-16T01:39:03.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.046+0000 7f1e75082640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e701556e0 0x7f1e7014cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:03.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.046+0000 7f1e75082640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e70155ac0 0x7f1e7014ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:03.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.046+0000 7f1e75082640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e7015c2a0 0x7f1e701543f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:03.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.046+0000 7f1e75082640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1e700cdd40 con 0x7f1e7015c2a0 2024-09-16T01:39:03.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.046+0000 7f1e75082640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1e700b5180 con 0x7f1e701556e0 2024-09-16T01:39:03.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.046+0000 7f1e75082640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1e700cfab0 con 0x7f1e70155ac0 2024-09-16T01:39:03.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.046+0000 7f1e6f59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e7015c2a0 0x7f1e701543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:03.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.046+0000 7f1e6f59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e7015c2a0 0x7f1e701543f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52766/0 (socket says 172.21.15.88:52766) 2024-09-16T01:39:03.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.046+0000 7f1e6f59e640 1 -- 172.21.15.88:0/958066082 learned_addr learned my addr 172.21.15.88:0/958066082 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:03.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.046+0000 7f1e6ed9d640 1 --2- 172.21.15.88:0/958066082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e701556e0 0x7f1e7014cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:03.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.046+0000 7f1e67fff640 1 --2- 172.21.15.88:0/958066082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e70155ac0 0x7f1e7014ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:03.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.046+0000 7f1e6f59e640 1 -- 172.21.15.88:0/958066082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e70155ac0 msgr2=0x7f1e7014ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:03.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.046+0000 7f1e6f59e640 1 --2- 172.21.15.88:0/958066082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e70155ac0 0x7f1e7014ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.046+0000 7f1e6f59e640 1 -- 172.21.15.88:0/958066082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e701556e0 msgr2=0x7f1e7014cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:03.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.046+0000 7f1e6f59e640 1 --2- 172.21.15.88:0/958066082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e701556e0 0x7f1e7014cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.046+0000 7f1e6f59e640 1 -- 172.21.15.88:0/958066082 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e7014d4a0 con 0x7f1e7015c2a0 2024-09-16T01:39:03.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.046+0000 7f1e6f59e640 1 --2- 172.21.15.88:0/958066082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e7015c2a0 0x7f1e701543f0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800d150 tx=0x7f1e68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:03.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.047+0000 7f1e677fe640 1 -- 172.21.15.88:0/958066082 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e68009dc0 con 0x7f1e7015c2a0 2024-09-16T01:39:03.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.047+0000 7f1e677fe640 1 -- 172.21.15.88:0/958066082 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e68047480 con 0x7f1e7015c2a0 2024-09-16T01:39:03.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.047+0000 7f1e677fe640 1 -- 172.21.15.88:0/958066082 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e68047870 con 0x7f1e7015c2a0 2024-09-16T01:39:03.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.047+0000 7f1e75082640 1 -- 172.21.15.88:0/958066082 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e701560f0 con 0x7f1e7015c2a0 2024-09-16T01:39:03.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.047+0000 7f1e75082640 1 -- 172.21.15.88:0/958066082 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e701566b0 con 0x7f1e7015c2a0 2024-09-16T01:39:03.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.048+0000 7f1e75082640 1 -- 172.21.15.88:0/958066082 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e700cfe30 con 0x7f1e7015c2a0 2024-09-16T01:39:03.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.048+0000 7f1e677fe640 1 -- 172.21.15.88:0/958066082 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1e68032420 con 0x7f1e7015c2a0 2024-09-16T01:39:03.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.048+0000 7f1e677fe640 1 --2- 172.21.15.88:0/958066082 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1e5006c2b0 0x7f1e5006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:03.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.049+0000 7f1e677fe640 1 -- 172.21.15.88:0/958066082 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5881+0+0 (secure 0 0 0) 0x7f1e680ae550 con 0x7f1e7015c2a0 2024-09-16T01:39:03.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.049+0000 7f1e6ed9d640 1 --2- 172.21.15.88:0/958066082 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1e5006c2b0 0x7f1e5006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:03.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.049+0000 7f1e6ed9d640 1 --2- 172.21.15.88:0/958066082 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1e5006c2b0 0x7f1e5006e770 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1e7015e320 tx=0x7f1e60009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:03.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.053+0000 7f1e677fe640 1 -- 172.21.15.88:0/958066082 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1e7014d4a0 con 0x7f1e7015c2a0 2024-09-16T01:39:03.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.255+0000 7f1e75082640 1 -- 172.21.15.88:0/958066082 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e700cd190 con 0x7f1e7015c2a0 2024-09-16T01:39:03.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.255+0000 7f1e677fe640 1 -- 172.21.15.88:0/958066082 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+5225 (secure 0 0 0) 0x7f1e6807c5c0 con 0x7f1e7015c2a0 2024-09-16T01:39:03.257 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:03.257 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":51,"btime":"2024-09-16T01:37:27:255277+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/581241194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":581241194},{"type":"v1","addr":"172.21.15.154:6834","nonce":581241194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2104508521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2104508521},{"type":"v1","addr":"172.21.15.154:6835","nonce":2104508521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":6195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2920926674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2920926674},{"type":"v1","addr":"172.21.15.88:6837","nonce":2920926674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:39.786438+0000","modified":"2024-09-16T01:37:27.255273+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5859},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5859":{"gid":5859,"name":"c","rank":0,"incarnation":48,"state":"up:active","state_seq":16,"addr":"172.21.15.88:6836/998283126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":998283126},{"type":"v1","addr":"172.21.15.88:6836","nonce":998283126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5859,"qdb_cluster":[5859]},"id":4}]} 2024-09-16T01:39:03.257 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 51 2024-09-16T01:39:03.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.257+0000 7f1e75082640 1 -- 172.21.15.88:0/958066082 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1e5006c2b0 msgr2=0x7f1e5006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:03.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.257+0000 7f1e75082640 1 --2- 172.21.15.88:0/958066082 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1e5006c2b0 0x7f1e5006e770 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1e7015e320 tx=0x7f1e60009040 comp rx=0 tx=0).stop 2024-09-16T01:39:03.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.257+0000 7f1e75082640 1 -- 172.21.15.88:0/958066082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e7015c2a0 msgr2=0x7f1e701543f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:03.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.257+0000 7f1e75082640 1 --2- 172.21.15.88:0/958066082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e7015c2a0 0x7f1e701543f0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f1e6800d150 tx=0x7f1e68004a40 comp rx=0 tx=0).stop 2024-09-16T01:39:03.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.257+0000 7f1e75082640 1 -- 172.21.15.88:0/958066082 shutdown_connections 2024-09-16T01:39:03.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.257+0000 7f1e75082640 1 --2- 172.21.15.88:0/958066082 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1e5006c2b0 0x7f1e5006e770 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.257+0000 7f1e75082640 1 --2- 172.21.15.88:0/958066082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e7015c2a0 0x7f1e701543f0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.257+0000 7f1e75082640 1 --2- 172.21.15.88:0/958066082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e70155ac0 0x7f1e7014ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.258+0000 7f1e75082640 1 --2- 172.21.15.88:0/958066082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e701556e0 0x7f1e7014cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.258+0000 7f1e75082640 1 -- 172.21.15.88:0/958066082 >> 172.21.15.88:0/958066082 conn(0x7f1e7006bc10 msgr2=0x7f1e700b2730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:03.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.258+0000 7f1e75082640 1 -- 172.21.15.88:0/958066082 shutdown_connections 2024-09-16T01:39:03.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.258+0000 7f1e75082640 1 -- 172.21.15.88:0/958066082 wait complete. 2024-09-16T01:39:03.270 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:39:03.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.366+0000 7f9e10e3c640 1 Processor -- start 2024-09-16T01:39:03.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.366+0000 7f9e10e3c640 1 -- start start 2024-09-16T01:39:03.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.367+0000 7f9e10e3c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9e0c073560 0x7f9e0c06ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:03.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.367+0000 7f9e10e3c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e0c15eaa0 con 0x7f9e0c06e200 2024-09-16T01:39:03.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.367+0000 7f9e10e3c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e0c073a90 con 0x7f9e0c15f230 2024-09-16T01:39:03.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.367+0000 7f9e10e3c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9e0c15eda0 con 0x7f9e0c073560 2024-09-16T01:39:03.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.367+0000 7f9e0a59c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9e0c06e200 0x7f9e0c06e5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57386/0 (socket says 172.21.15.88:57386) 2024-09-16T01:39:03.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.367+0000 7f9e0a59c640 1 -- 172.21.15.88:0/3633420765 learned_addr learned my addr 172.21.15.88:0/3633420765 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:03.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.368+0000 7f9e09d9b640 1 --2- 172.21.15.88:0/3633420765 >> v2:172.21.15.154:3300/0 conn(0x7f9e0c073560 0x7f9e0c06ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:03.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.368+0000 7f9e0959a640 1 -- 172.21.15.88:0/3633420765 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2341824136 0 0) 0x7f9e0c15eaa0 con 0x7f9e0c06e200 2024-09-16T01:39:03.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.368+0000 7f9e0959a640 1 -- 172.21.15.88:0/3633420765 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9df8003520 con 0x7f9e0c06e200 2024-09-16T01:39:03.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.368+0000 7f9e09d9b640 1 -- 172.21.15.88:0/3633420765 >> v1:172.21.15.154:6789/0 conn(0x7f9e0c15f230 legacy=0x7f9e0c0b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:03.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.368+0000 7f9e0959a640 1 -- 172.21.15.88:0/3633420765 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2069012468 0 0) 0x7f9df8003520 con 0x7f9e0c06e200 2024-09-16T01:39:03.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.368+0000 7f9e09d9b640 1 -- 172.21.15.88:0/3633420765 >> v1:172.21.15.88:6789/0 conn(0x7f9e0c06e200 legacy=0x7f9e0c06e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:03.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.368+0000 7f9e09d9b640 1 -- 172.21.15.88:0/3633420765 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e0c073d70 con 0x7f9e0c073560 2024-09-16T01:39:03.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.369+0000 7f9e09d9b640 1 --2- 172.21.15.88:0/3633420765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e0c073560 0x7f9e0c06ccd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9df40097a0 tx=0x7f9df402a130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6c469ceb6ef0009 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:03.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.369+0000 7f9e0959a640 1 -- 172.21.15.88:0/3633420765 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df4037070 con 0x7f9e0c073560 2024-09-16T01:39:03.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.369+0000 7f9e0959a640 1 -- 172.21.15.88:0/3633420765 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9df402abc0 con 0x7f9e0c073560 2024-09-16T01:39:03.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.369+0000 7f9e0959a640 1 -- 172.21.15.88:0/3633420765 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc0019f0 con 0x7f9e0c073560 2024-09-16T01:39:03.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.369+0000 7f9e10e3c640 1 -- 172.21.15.88:0/3633420765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e0c073560 msgr2=0x7f9e0c06ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:03.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.369+0000 7f9e10e3c640 1 --2- 172.21.15.88:0/3633420765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e0c073560 0x7f9e0c06ccd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9df40097a0 tx=0x7f9df402a130 comp rx=0 tx=0).stop 2024-09-16T01:39:03.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.369+0000 7f9e10e3c640 1 -- 172.21.15.88:0/3633420765 shutdown_connections 2024-09-16T01:39:03.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.369+0000 7f9e10e3c640 1 --2- 172.21.15.88:0/3633420765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e0c073560 0x7f9e0c06ccd0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.370+0000 7f9e10e3c640 1 -- 172.21.15.88:0/3633420765 >> 172.21.15.88:0/3633420765 conn(0x7f9e0c0b6ec0 msgr2=0x7f9e0c0b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:03.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.370+0000 7f9e10e3c640 1 -- 172.21.15.88:0/3633420765 shutdown_connections 2024-09-16T01:39:03.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.370+0000 7f9e10e3c640 1 -- 172.21.15.88:0/3633420765 wait complete. 2024-09-16T01:39:03.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.371+0000 7f9e10e3c640 1 Processor -- start 2024-09-16T01:39:03.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.371+0000 7f9e10e3c640 1 -- start start 2024-09-16T01:39:03.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.372+0000 7f9e10e3c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e0c06e200 0x7f9e0c0cfd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:03.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.372+0000 7f9e10e3c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e0c073560 0x7f9e0c148ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:03.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.372+0000 7f9e10e3c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e0c15f230 0x7f9e0c159140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:03.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.372+0000 7f9e10e3c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0c0b4c00 con 0x7f9e0c15f230 2024-09-16T01:39:03.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.372+0000 7f9e10e3c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0c0b4a80 con 0x7f9e0c073560 2024-09-16T01:39:03.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.372+0000 7f9e10e3c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9e0c158e10 con 0x7f9e0c06e200 2024-09-16T01:39:03.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.372+0000 7f9e0ad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e0c15f230 0x7f9e0c159140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:03.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.372+0000 7f9e0ad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e0c15f230 0x7f9e0c159140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52778/0 (socket says 172.21.15.88:52778) 2024-09-16T01:39:03.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.372+0000 7f9e0ad9d640 1 -- 172.21.15.88:0/716297357 learned_addr learned my addr 172.21.15.88:0/716297357 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:03.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.372+0000 7f9e09d9b640 1 --2- 172.21.15.88:0/716297357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e0c073560 0x7f9e0c148ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:03.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.372+0000 7f9e0a59c640 1 --2- 172.21.15.88:0/716297357 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e0c06e200 0x7f9e0c0cfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:03.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.372+0000 7f9e0ad9d640 1 -- 172.21.15.88:0/716297357 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e0c06e200 msgr2=0x7f9e0c0cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:03.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.372+0000 7f9e0ad9d640 1 --2- 172.21.15.88:0/716297357 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e0c06e200 0x7f9e0c0cfd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.372+0000 7f9e0ad9d640 1 -- 172.21.15.88:0/716297357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e0c073560 msgr2=0x7f9e0c148ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:03.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.372+0000 7f9e0ad9d640 1 --2- 172.21.15.88:0/716297357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e0c073560 0x7f9e0c148ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.372+0000 7f9e0ad9d640 1 -- 172.21.15.88:0/716297357 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e0c0af210 con 0x7f9e0c15f230 2024-09-16T01:39:03.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.372+0000 7f9e0ad9d640 1 --2- 172.21.15.88:0/716297357 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e0c15f230 0x7f9e0c159140 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f9e0400c390 tx=0x7f9e0400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:03.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.372+0000 7f9e02ffd640 1 -- 172.21.15.88:0/716297357 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e04017070 con 0x7f9e0c15f230 2024-09-16T01:39:03.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.372+0000 7f9e02ffd640 1 -- 172.21.15.88:0/716297357 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e040052d0 con 0x7f9e0c15f230 2024-09-16T01:39:03.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.372+0000 7f9e10e3c640 1 -- 172.21.15.88:0/716297357 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e0c0af4b0 con 0x7f9e0c15f230 2024-09-16T01:39:03.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.372+0000 7f9e02ffd640 1 -- 172.21.15.88:0/716297357 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e04003810 con 0x7f9e0c15f230 2024-09-16T01:39:03.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.372+0000 7f9e10e3c640 1 -- 172.21.15.88:0/716297357 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e0c0afa70 con 0x7f9e0c15f230 2024-09-16T01:39:03.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.373+0000 7f9e10e3c640 1 -- 172.21.15.88:0/716297357 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e0c15f6e0 con 0x7f9e0c15f230 2024-09-16T01:39:03.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.374+0000 7f9e02ffd640 1 -- 172.21.15.88:0/716297357 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9e0401a070 con 0x7f9e0c15f230 2024-09-16T01:39:03.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.374+0000 7f9e02ffd640 1 --2- 172.21.15.88:0/716297357 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9dec06c2b0 0x7f9dec06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:03.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.374+0000 7f9e0a59c640 1 --2- 172.21.15.88:0/716297357 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9dec06c2b0 0x7f9dec06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:03.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.374+0000 7f9e02ffd640 1 -- 172.21.15.88:0/716297357 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5881+0+0 (secure 0 0 0) 0x7f9e0408a730 con 0x7f9e0c15f230 2024-09-16T01:39:03.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.375+0000 7f9e0a59c640 1 --2- 172.21.15.88:0/716297357 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9dec06c2b0 0x7f9dec06e770 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc005c00 tx=0x7f9dfc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:03.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.379+0000 7f9e02ffd640 1 -- 172.21.15.88:0/716297357 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9e040587a0 con 0x7f9e0c15f230 2024-09-16T01:39:03.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.583+0000 7f9e10e3c640 1 -- 172.21.15.88:0/716297357 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e0c0b5cc0 con 0x7f9e0c15f230 2024-09-16T01:39:03.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.584+0000 7f9e02ffd640 1 -- 172.21.15.88:0/716297357 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+5225 (secure 0 0 0) 0x7f9e0405caa0 con 0x7f9e0c15f230 2024-09-16T01:39:03.585 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:03.586 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":51,"btime":"2024-09-16T01:37:27:255277+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/581241194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":581241194},{"type":"v1","addr":"172.21.15.154:6834","nonce":581241194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2104508521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2104508521},{"type":"v1","addr":"172.21.15.154:6835","nonce":2104508521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":6195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2920926674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2920926674},{"type":"v1","addr":"172.21.15.88:6837","nonce":2920926674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:39.786438+0000","modified":"2024-09-16T01:37:27.255273+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5859},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5859":{"gid":5859,"name":"c","rank":0,"incarnation":48,"state":"up:active","state_seq":16,"addr":"172.21.15.88:6836/998283126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":998283126},{"type":"v1","addr":"172.21.15.88:6836","nonce":998283126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5859,"qdb_cluster":[5859]},"id":4}]} 2024-09-16T01:39:03.586 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 51 2024-09-16T01:39:03.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.586+0000 7f9e10e3c640 1 -- 172.21.15.88:0/716297357 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9dec06c2b0 msgr2=0x7f9dec06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:03.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.586+0000 7f9e10e3c640 1 --2- 172.21.15.88:0/716297357 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9dec06c2b0 0x7f9dec06e770 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc005c00 tx=0x7f9dfc009040 comp rx=0 tx=0).stop 2024-09-16T01:39:03.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.586+0000 7f9e10e3c640 1 -- 172.21.15.88:0/716297357 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e0c15f230 msgr2=0x7f9e0c159140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:03.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.586+0000 7f9e10e3c640 1 --2- 172.21.15.88:0/716297357 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e0c15f230 0x7f9e0c159140 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f9e0400c390 tx=0x7f9e0400c850 comp rx=0 tx=0).stop 2024-09-16T01:39:03.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.586+0000 7f9e10e3c640 1 -- 172.21.15.88:0/716297357 shutdown_connections 2024-09-16T01:39:03.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.586+0000 7f9e10e3c640 1 --2- 172.21.15.88:0/716297357 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9dec06c2b0 0x7f9dec06e770 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.586+0000 7f9e10e3c640 1 --2- 172.21.15.88:0/716297357 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e0c15f230 0x7f9e0c159140 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.586+0000 7f9e10e3c640 1 --2- 172.21.15.88:0/716297357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e0c073560 0x7f9e0c148ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.586+0000 7f9e10e3c640 1 --2- 172.21.15.88:0/716297357 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e0c06e200 0x7f9e0c0cfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.586+0000 7f9e10e3c640 1 -- 172.21.15.88:0/716297357 >> 172.21.15.88:0/716297357 conn(0x7f9e0c0b6ec0 msgr2=0x7f9e0c0d0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:03.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.586+0000 7f9e10e3c640 1 -- 172.21.15.88:0/716297357 shutdown_connections 2024-09-16T01:39:03.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.586+0000 7f9e10e3c640 1 -- 172.21.15.88:0/716297357 wait complete. 2024-09-16T01:39:03.598 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:39:03.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.695+0000 7f60c4ac9640 1 Processor -- start 2024-09-16T01:39:03.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.695+0000 7f60c4ac9640 1 -- start start 2024-09-16T01:39:03.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.696+0000 7f60c4ac9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60c00735b0 0x7f60c00cf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:03.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.696+0000 7f60c4ac9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60c006faf0 con 0x7f60c006d540 2024-09-16T01:39:03.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.696+0000 7f60c4ac9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60c006fdd0 con 0x7f60c015e950 2024-09-16T01:39:03.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.696+0000 7f60c4ac9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f60c00700b0 con 0x7f60c00735b0 2024-09-16T01:39:03.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.696+0000 7f60b7fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f60c006d540 0x7f60c00cf0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57402/0 (socket says 172.21.15.88:57402) 2024-09-16T01:39:03.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.696+0000 7f60b7fff640 1 -- 172.21.15.88:0/3604816337 learned_addr learned my addr 172.21.15.88:0/3604816337 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:03.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.696+0000 7f60bed9d640 1 --2- 172.21.15.88:0/3604816337 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60c00735b0 0x7f60c00cf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:03.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.696+0000 7f60bdd9b640 1 -- 172.21.15.88:0/3604816337 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1686257651 0 0) 0x7f60c006faf0 con 0x7f60c006d540 2024-09-16T01:39:03.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.696+0000 7f60bdd9b640 1 -- 172.21.15.88:0/3604816337 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60ac003520 con 0x7f60c006d540 2024-09-16T01:39:03.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.696+0000 7f60bdd9b640 1 -- 172.21.15.88:0/3604816337 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1794924260 0 0) 0x7f60c006fdd0 con 0x7f60c015e950 2024-09-16T01:39:03.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.696+0000 7f60bdd9b640 1 -- 172.21.15.88:0/3604816337 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60c006faf0 con 0x7f60c015e950 2024-09-16T01:39:03.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.696+0000 7f60bdd9b640 1 -- 172.21.15.88:0/3604816337 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1661764196 0 0) 0x7f60ac003520 con 0x7f60c006d540 2024-09-16T01:39:03.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.696+0000 7f60bdd9b640 1 -- 172.21.15.88:0/3604816337 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60c00735b0 msgr2=0x7f60c00cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:03.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.696+0000 7f60bdd9b640 1 --2- 172.21.15.88:0/3604816337 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60c00735b0 0x7f60c00cf7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.696+0000 7f60bdd9b640 1 -- 172.21.15.88:0/3604816337 >> v1:172.21.15.154:6789/0 conn(0x7f60c015e950 legacy=0x7f60c006ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:03.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.696+0000 7f60bdd9b640 1 -- 172.21.15.88:0/3604816337 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60c0070390 con 0x7f60c006d540 2024-09-16T01:39:03.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.696+0000 7f60bdd9b640 1 -- 172.21.15.88:0/3604816337 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f60a8001670 con 0x7f60c006d540 2024-09-16T01:39:03.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.696+0000 7f60bdd9b640 1 -- 172.21.15.88:0/3604816337 >> v1:172.21.15.88:6789/0 conn(0x7f60c006d540 legacy=0x7f60c00cf0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:03.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.696+0000 7f60bdd9b640 1 --2- 172.21.15.88:0/3604816337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60ac003b70 0x7f60ac003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:03.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.696+0000 7f60bdd9b640 1 --2- 172.21.15.88:0/3604816337 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60c00735b0 0x7f60ac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:03.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.697+0000 7f60bdd9b640 1 --2- 172.21.15.88:0/3604816337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60ac004d40 0x7f60ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:03.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.697+0000 7f60bdd9b640 1 -- 172.21.15.88:0/3604816337 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f60c00700b0 con 0x7f60ac003b70 2024-09-16T01:39:03.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.697+0000 7f60bdd9b640 1 -- 172.21.15.88:0/3604816337 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac00baa0 con 0x7f60ac004d40 2024-09-16T01:39:03.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.697+0000 7f60bdd9b640 1 -- 172.21.15.88:0/3604816337 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f60ac00bca0 con 0x7f60c00735b0 2024-09-16T01:39:03.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.697+0000 7f60be59c640 1 --2- 172.21.15.88:0/3604816337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60ac003b70 0x7f60ac003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:03.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.697+0000 7f60b7fff640 1 --2- 172.21.15.88:0/3604816337 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60c00735b0 0x7f60ac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:03.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.697+0000 7f60bed9d640 1 --2- 172.21.15.88:0/3604816337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60ac004d40 0x7f60ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:03.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.697+0000 7f60be59c640 1 -- 172.21.15.88:0/3604816337 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60c00735b0 msgr2=0x7f60ac004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:03.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.697+0000 7f60be59c640 1 --2- 172.21.15.88:0/3604816337 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60c00735b0 0x7f60ac004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.697+0000 7f60be59c640 1 -- 172.21.15.88:0/3604816337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60ac004d40 msgr2=0x7f60ac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:03.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.697+0000 7f60be59c640 1 --2- 172.21.15.88:0/3604816337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60ac004d40 0x7f60ac0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.697+0000 7f60be59c640 1 -- 172.21.15.88:0/3604816337 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60ac003520 con 0x7f60ac003b70 2024-09-16T01:39:03.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.698+0000 7f60be59c640 1 --2- 172.21.15.88:0/3604816337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60ac003b70 0x7f60ac003f70 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f60b00018d0 tx=0x7f60ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4219156e7138c631 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:03.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.698+0000 7f60bdd9b640 1 -- 172.21.15.88:0/3604816337 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b00014f0 con 0x7f60ac003b70 2024-09-16T01:39:03.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.698+0000 7f60bdd9b640 1 -- 172.21.15.88:0/3604816337 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60b00052d0 con 0x7f60ac003b70 2024-09-16T01:39:03.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.698+0000 7f60bdd9b640 1 -- 172.21.15.88:0/3604816337 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b002aca0 con 0x7f60ac003b70 2024-09-16T01:39:03.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.698+0000 7f60c4ac9640 1 -- 172.21.15.88:0/3604816337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60ac003b70 msgr2=0x7f60ac003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:03.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.698+0000 7f60c4ac9640 1 --2- 172.21.15.88:0/3604816337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60ac003b70 0x7f60ac003f70 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f60b00018d0 tx=0x7f60ac005830 comp rx=0 tx=0).stop 2024-09-16T01:39:03.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.698+0000 7f60bed9d640 1 -- 172.21.15.88:0/3604816337 reap_dead start 2024-09-16T01:39:03.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.698+0000 7f60c4ac9640 1 -- 172.21.15.88:0/3604816337 shutdown_connections 2024-09-16T01:39:03.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.698+0000 7f60c4ac9640 1 -- 172.21.15.88:0/3604816337 >> 172.21.15.88:0/3604816337 conn(0x7f60c00d14d0 msgr2=0x7f60c00d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:03.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.698+0000 7f60c4ac9640 1 -- 172.21.15.88:0/3604816337 shutdown_connections 2024-09-16T01:39:03.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.698+0000 7f60c4ac9640 1 -- 172.21.15.88:0/3604816337 wait complete. 2024-09-16T01:39:03.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.700+0000 7f60c4ac9640 1 Processor -- start 2024-09-16T01:39:03.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.700+0000 7f60c4ac9640 1 -- start start 2024-09-16T01:39:03.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.700+0000 7f60c4ac9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60ac003b70 0x7f60c0153910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:03.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.700+0000 7f60c4ac9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60c0154490 0x7f60c0156fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:03.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.700+0000 7f60c4ac9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60c0153ed0 0x7f60c0159ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:03.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.700+0000 7f60c4ac9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f60c00721d0 con 0x7f60c0154490 2024-09-16T01:39:03.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.700+0000 7f60c4ac9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f60c0072050 con 0x7f60ac003b70 2024-09-16T01:39:03.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.700+0000 7f60c4ac9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f60c0072350 con 0x7f60c0153ed0 2024-09-16T01:39:03.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.700+0000 7f60b7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60c0154490 0x7f60c0156fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:03.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.700+0000 7f60b7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60c0154490 0x7f60c0156fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52794/0 (socket says 172.21.15.88:52794) 2024-09-16T01:39:03.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.700+0000 7f60b7fff640 1 -- 172.21.15.88:0/2966331185 learned_addr learned my addr 172.21.15.88:0/2966331185 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:03.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.701+0000 7f60be59c640 1 --2- 172.21.15.88:0/2966331185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60ac003b70 0x7f60c0153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:03.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.701+0000 7f60bed9d640 1 --2- 172.21.15.88:0/2966331185 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60c0153ed0 0x7f60c0159ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:03.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.701+0000 7f60b7fff640 1 -- 172.21.15.88:0/2966331185 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60c0153ed0 msgr2=0x7f60c0159ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:03.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.701+0000 7f60b7fff640 1 --2- 172.21.15.88:0/2966331185 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60c0153ed0 0x7f60c0159ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.701+0000 7f60b7fff640 1 -- 172.21.15.88:0/2966331185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60ac003b70 msgr2=0x7f60c0153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:03.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.701+0000 7f60b7fff640 1 --2- 172.21.15.88:0/2966331185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60ac003b70 0x7f60c0153910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.701+0000 7f60b7fff640 1 -- 172.21.15.88:0/2966331185 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60c0155f20 con 0x7f60c0154490 2024-09-16T01:39:03.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.701+0000 7f60b7fff640 1 --2- 172.21.15.88:0/2966331185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60c0154490 0x7f60c0156fc0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f60a800e790 tx=0x7f60a800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:03.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.701+0000 7f60b6ffd640 1 -- 172.21.15.88:0/2966331185 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a800ee40 con 0x7f60c0154490 2024-09-16T01:39:03.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.701+0000 7f60b6ffd640 1 -- 172.21.15.88:0/2966331185 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60a8018070 con 0x7f60c0154490 2024-09-16T01:39:03.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.701+0000 7f60c4ac9640 1 -- 172.21.15.88:0/2966331185 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60c0156200 con 0x7f60c0154490 2024-09-16T01:39:03.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.701+0000 7f60b6ffd640 1 -- 172.21.15.88:0/2966331185 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a800cab0 con 0x7f60c0154490 2024-09-16T01:39:03.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.701+0000 7f60c4ac9640 1 -- 172.21.15.88:0/2966331185 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60c01567c0 con 0x7f60c0154490 2024-09-16T01:39:03.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.702+0000 7f60c4ac9640 1 -- 172.21.15.88:0/2966331185 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60c006ce30 con 0x7f60c0154490 2024-09-16T01:39:03.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.703+0000 7f60b6ffd640 1 -- 172.21.15.88:0/2966331185 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f60a800cc50 con 0x7f60c0154490 2024-09-16T01:39:03.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.703+0000 7f60b6ffd640 1 --2- 172.21.15.88:0/2966331185 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f60a006c2e0 0x7f60a006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:03.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.703+0000 7f60b6ffd640 1 -- 172.21.15.88:0/2966331185 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5881+0+0 (secure 0 0 0) 0x7f60a808ae00 con 0x7f60c0154490 2024-09-16T01:39:03.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.703+0000 7f60be59c640 1 --2- 172.21.15.88:0/2966331185 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f60a006c2e0 0x7f60a006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:03.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.703+0000 7f60be59c640 1 --2- 172.21.15.88:0/2966331185 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f60a006c2e0 0x7f60a006e7a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f60b0032000 tx=0x7f60b0009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:03.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.707+0000 7f60b6ffd640 1 -- 172.21.15.88:0/2966331185 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f60a8058f00 con 0x7f60c0154490 2024-09-16T01:39:03.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.874+0000 7f60c4ac9640 1 -- 172.21.15.88:0/2966331185 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f60c00b3c50 con 0x7f60c0154490 2024-09-16T01:39:03.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.875+0000 7f60b6ffd640 1 -- 172.21.15.88:0/2966331185 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) ==== 74+0+16958 (secure 0 0 0) 0x7f60a805d200 con 0x7f60c0154490 2024-09-16T01:39:03.878 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:03.878 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":60,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:37:24.240775+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:36:38.121335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-16T01:36:38.755908+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:6837/4130074102":"2024-09-17T01:37:24.240722+0000","172.21.15.88:6835/4130074102":"2024-09-17T01:37:24.240722+0000","172.21.15.88:6837/574169866":"2024-09-17T01:36:48.820862+0000","172.21.15.88:6835/574169866":"2024-09-17T01:36:48.820862+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:39:03.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.877+0000 7f60c4ac9640 1 -- 172.21.15.88:0/2966331185 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f60a006c2e0 msgr2=0x7f60a006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:03.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.877+0000 7f60c4ac9640 1 --2- 172.21.15.88:0/2966331185 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f60a006c2e0 0x7f60a006e7a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f60b0032000 tx=0x7f60b0009030 comp rx=0 tx=0).stop 2024-09-16T01:39:03.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.877+0000 7f60c4ac9640 1 -- 172.21.15.88:0/2966331185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60c0154490 msgr2=0x7f60c0156fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:03.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.877+0000 7f60c4ac9640 1 --2- 172.21.15.88:0/2966331185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60c0154490 0x7f60c0156fc0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f60a800e790 tx=0x7f60a800ec50 comp rx=0 tx=0).stop 2024-09-16T01:39:03.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.877+0000 7f60c4ac9640 1 -- 172.21.15.88:0/2966331185 shutdown_connections 2024-09-16T01:39:03.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.877+0000 7f60c4ac9640 1 --2- 172.21.15.88:0/2966331185 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f60a006c2e0 0x7f60a006e7a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.877+0000 7f60c4ac9640 1 --2- 172.21.15.88:0/2966331185 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60c0153ed0 0x7f60c0159ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.877+0000 7f60c4ac9640 1 --2- 172.21.15.88:0/2966331185 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60c0154490 0x7f60c0156fc0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.877+0000 7f60c4ac9640 1 --2- 172.21.15.88:0/2966331185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60ac003b70 0x7f60c0153910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.877+0000 7f60c4ac9640 1 -- 172.21.15.88:0/2966331185 >> 172.21.15.88:0/2966331185 conn(0x7f60c00d14d0 msgr2=0x7f60c0072500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:03.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.877+0000 7f60c4ac9640 1 -- 172.21.15.88:0/2966331185 shutdown_connections 2024-09-16T01:39:03.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.877+0000 7f60c4ac9640 1 -- 172.21.15.88:0/2966331185 wait complete. 2024-09-16T01:39:03.893 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-16T01:39:03.893 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:39:03.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.992+0000 7f390b342640 1 Processor -- start 2024-09-16T01:39:03.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.992+0000 7f390b342640 1 -- start start 2024-09-16T01:39:03.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.993+0000 7f390b342640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f390415c550 0x7f39040b5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:03.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.993+0000 7f390b342640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f39040b6310 0x7f39040b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:03.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.993+0000 7f390b342640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39040b4cf0 con 0x7f390415c170 2024-09-16T01:39:03.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.993+0000 7f390b342640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f39040b4fd0 con 0x7f39040b6310 2024-09-16T01:39:03.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.993+0000 7f390b342640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f39040b52b0 con 0x7f390415c550 2024-09-16T01:39:03.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.993+0000 7f39098df640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f39040b6310 0x7f39040b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:03.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.993+0000 7f39098df640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f39040b6310 0x7f39040b39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52806/0 (socket says 172.21.15.88:52806) 2024-09-16T01:39:03.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.993+0000 7f39098df640 1 -- 172.21.15.88:0/1896871136 learned_addr learned my addr 172.21.15.88:0/1896871136 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:03.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.993+0000 7f39088dd640 1 --2- 172.21.15.88:0/1896871136 >> v2:172.21.15.154:3300/0 conn(0x7f390415c550 0x7f39040b5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:03.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.993+0000 7f39098df640 1 -- 172.21.15.88:0/1896871136 >> v2:172.21.15.154:3300/0 conn(0x7f390415c550 msgr2=0x7f39040b5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:03.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.993+0000 7f39098df640 1 --2- 172.21.15.88:0/1896871136 >> v2:172.21.15.154:3300/0 conn(0x7f390415c550 0x7f39040b5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:03.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.993+0000 7f39098df640 1 -- 172.21.15.88:0/1896871136 >> v1:172.21.15.154:6789/0 conn(0x7f390415c170 legacy=0x7f39040cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:03.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.993+0000 7f39098df640 1 -- 172.21.15.88:0/1896871136 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39040b5590 con 0x7f39040b6310 2024-09-16T01:39:03.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.994+0000 7f39098df640 1 --2- 172.21.15.88:0/1896871136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39040b6310 0x7f39040b39b0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f38fc009e50 tx=0x7f38fc02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cff5bf045dcbea9b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:03.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.994+0000 7f38fbfff640 1 -- 172.21.15.88:0/1896871136 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc036070 con 0x7f39040b6310 2024-09-16T01:39:03.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.994+0000 7f38fbfff640 1 -- 172.21.15.88:0/1896871136 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38fc031600 con 0x7f39040b6310 2024-09-16T01:39:03.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.994+0000 7f38fbfff640 1 -- 172.21.15.88:0/1896871136 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc0319d0 con 0x7f39040b6310 2024-09-16T01:39:03.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.994+0000 7f390b342640 1 -- 172.21.15.88:0/1896871136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39040b6310 msgr2=0x7f39040b39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:03.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.994+0000 7f390b342640 1 --2- 172.21.15.88:0/1896871136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39040b6310 0x7f39040b39b0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f38fc009e50 tx=0x7f38fc02a710 comp rx=0 tx=0).stop 2024-09-16T01:39:03.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.994+0000 7f390b342640 1 -- 172.21.15.88:0/1896871136 shutdown_connections 2024-09-16T01:39:04.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.994+0000 7f390b342640 1 --2- 172.21.15.88:0/1896871136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39040b6310 0x7f39040b39b0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.994+0000 7f390b342640 1 --2- 172.21.15.88:0/1896871136 >> v2:172.21.15.154:3300/0 conn(0x7f390415c550 0x7f39040b5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.994+0000 7f390b342640 1 -- 172.21.15.88:0/1896871136 >> 172.21.15.88:0/1896871136 conn(0x7f390406bc10 msgr2=0x7f390406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:04.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.995+0000 7f390b342640 1 -- 172.21.15.88:0/1896871136 shutdown_connections 2024-09-16T01:39:04.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.995+0000 7f390b342640 1 -- 172.21.15.88:0/1896871136 wait complete. 2024-09-16T01:39:04.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.996+0000 7f390b342640 1 Processor -- start 2024-09-16T01:39:04.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.996+0000 7f390b342640 1 -- start start 2024-09-16T01:39:04.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.996+0000 7f390b342640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39040b6310 0x7f390415f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.996+0000 7f390b342640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f390415c170 0x7f3904152c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.996+0000 7f390b342640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f390415c550 0x7f39041f5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.996+0000 7f390b342640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f390415c9c0 con 0x7f390415c550 2024-09-16T01:39:04.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.996+0000 7f390b342640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f39040b5b50 con 0x7f390415c170 2024-09-16T01:39:04.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.996+0000 7f390b342640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f39040cfdd0 con 0x7f39040b6310 2024-09-16T01:39:04.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.996+0000 7f39098df640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f390415c550 0x7f39041f5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.997+0000 7f39098df640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f390415c550 0x7f39041f5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52812/0 (socket says 172.21.15.88:52812) 2024-09-16T01:39:04.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.997+0000 7f39098df640 1 -- 172.21.15.88:0/2842592914 learned_addr learned my addr 172.21.15.88:0/2842592914 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:04.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.997+0000 7f39090de640 1 --2- 172.21.15.88:0/2842592914 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39040b6310 0x7f390415f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.997+0000 7f39088dd640 1 --2- 172.21.15.88:0/2842592914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f390415c170 0x7f3904152c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.997+0000 7f39098df640 1 -- 172.21.15.88:0/2842592914 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39040b6310 msgr2=0x7f390415f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:04.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.997+0000 7f39098df640 1 --2- 172.21.15.88:0/2842592914 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39040b6310 0x7f390415f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.997+0000 7f39098df640 1 -- 172.21.15.88:0/2842592914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f390415c170 msgr2=0x7f3904152c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:04.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.997+0000 7f39098df640 1 --2- 172.21.15.88:0/2842592914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f390415c170 0x7f3904152c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.997+0000 7f39098df640 1 -- 172.21.15.88:0/2842592914 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f390414b930 con 0x7f390415c550 2024-09-16T01:39:04.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.997+0000 7f39088dd640 1 --2- 172.21.15.88:0/2842592914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f390415c170 0x7f3904152c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:04.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.997+0000 7f39090de640 1 --2- 172.21.15.88:0/2842592914 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39040b6310 0x7f390415f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:04.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.997+0000 7f39098df640 1 --2- 172.21.15.88:0/2842592914 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f390415c550 0x7f39041f5c40 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f38fc009e50 tx=0x7f38fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:04.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.997+0000 7f38f9ffb640 1 -- 172.21.15.88:0/2842592914 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc036050 con 0x7f390415c550 2024-09-16T01:39:04.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.997+0000 7f38f9ffb640 1 -- 172.21.15.88:0/2842592914 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38fc042070 con 0x7f390415c550 2024-09-16T01:39:04.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.997+0000 7f390b342640 1 -- 172.21.15.88:0/2842592914 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f390415fc30 con 0x7f390415c550 2024-09-16T01:39:04.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.997+0000 7f38f9ffb640 1 -- 172.21.15.88:0/2842592914 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc009510 con 0x7f390415c550 2024-09-16T01:39:04.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.997+0000 7f390b342640 1 -- 172.21.15.88:0/2842592914 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3904155630 con 0x7f390415c550 2024-09-16T01:39:04.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.998+0000 7f390b342640 1 -- 172.21.15.88:0/2842592914 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39040b5dd0 con 0x7f390415c550 2024-09-16T01:39:04.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.999+0000 7f38f9ffb640 1 -- 172.21.15.88:0/2842592914 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f38fc007bf0 con 0x7f390415c550 2024-09-16T01:39:04.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.999+0000 7f38f9ffb640 1 --2- 172.21.15.88:0/2842592914 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f38e806c2b0 0x7f38e806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.999+0000 7f38f9ffb640 1 -- 172.21.15.88:0/2842592914 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5881+0+0 (secure 0 0 0) 0x7f38fc0ac2b0 con 0x7f390415c550 2024-09-16T01:39:04.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:03.999+0000 7f39090de640 1 --2- 172.21.15.88:0/2842592914 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f38e806c2b0 0x7f38e806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.000+0000 7f39090de640 1 --2- 172.21.15.88:0/2842592914 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f38e806c2b0 0x7f38e806e770 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3900003eb0 tx=0x7f3900009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:04.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.003+0000 7f38f9ffb640 1 -- 172.21.15.88:0/2842592914 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f390414b930 con 0x7f390415c550 2024-09-16T01:39:04.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.203+0000 7f390b342640 1 -- 172.21.15.88:0/2842592914 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39040761e0 con 0x7f390415c550 2024-09-16T01:39:04.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.204+0000 7f38f9ffb640 1 -- 172.21.15.88:0/2842592914 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+5225 (secure 0 0 0) 0x7f38fc07a970 con 0x7f390415c550 2024-09-16T01:39:04.205 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:04.205 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":51,"btime":"2024-09-16T01:37:27:255277+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/581241194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":581241194},{"type":"v1","addr":"172.21.15.154:6834","nonce":581241194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2104508521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2104508521},{"type":"v1","addr":"172.21.15.154:6835","nonce":2104508521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":6195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2920926674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2920926674},{"type":"v1","addr":"172.21.15.88:6837","nonce":2920926674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:39.786438+0000","modified":"2024-09-16T01:37:27.255273+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5859},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5859":{"gid":5859,"name":"c","rank":0,"incarnation":48,"state":"up:active","state_seq":16,"addr":"172.21.15.88:6836/998283126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":998283126},{"type":"v1","addr":"172.21.15.88:6836","nonce":998283126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5859,"qdb_cluster":[5859]},"id":4}]} 2024-09-16T01:39:04.206 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 51 2024-09-16T01:39:04.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.206+0000 7f390b342640 1 -- 172.21.15.88:0/2842592914 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f38e806c2b0 msgr2=0x7f38e806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:04.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.206+0000 7f390b342640 1 --2- 172.21.15.88:0/2842592914 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f38e806c2b0 0x7f38e806e770 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3900003eb0 tx=0x7f3900009040 comp rx=0 tx=0).stop 2024-09-16T01:39:04.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.206+0000 7f390b342640 1 -- 172.21.15.88:0/2842592914 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f390415c550 msgr2=0x7f39041f5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:04.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.206+0000 7f390b342640 1 --2- 172.21.15.88:0/2842592914 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f390415c550 0x7f39041f5c40 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f38fc009e50 tx=0x7f38fc004a40 comp rx=0 tx=0).stop 2024-09-16T01:39:04.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.206+0000 7f390b342640 1 -- 172.21.15.88:0/2842592914 shutdown_connections 2024-09-16T01:39:04.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.206+0000 7f390b342640 1 --2- 172.21.15.88:0/2842592914 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f38e806c2b0 0x7f38e806e770 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.206+0000 7f390b342640 1 --2- 172.21.15.88:0/2842592914 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f390415c550 0x7f39041f5c40 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.206+0000 7f390b342640 1 --2- 172.21.15.88:0/2842592914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f390415c170 0x7f3904152c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.206+0000 7f390b342640 1 --2- 172.21.15.88:0/2842592914 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39040b6310 0x7f390415f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.206+0000 7f390b342640 1 -- 172.21.15.88:0/2842592914 >> 172.21.15.88:0/2842592914 conn(0x7f390406bc10 msgr2=0x7f3904070f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:04.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.206+0000 7f390b342640 1 -- 172.21.15.88:0/2842592914 shutdown_connections 2024-09-16T01:39:04.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.206+0000 7f390b342640 1 -- 172.21.15.88:0/2842592914 wait complete. 2024-09-16T01:39:04.218 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:39:04.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.315+0000 7f06a3331640 1 Processor -- start 2024-09-16T01:39:04.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.316+0000 7f06a3331640 1 -- start start 2024-09-16T01:39:04.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.316+0000 7f06a3331640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f069c15e800 0x7f069c0b5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.316+0000 7f06a3331640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f069c15ebe0 0x7f069c15f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.317+0000 7f06a3331640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f069c06ea60 con 0x7f069c0b6e50 2024-09-16T01:39:04.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.317+0000 7f06a3331640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f069c06ed40 con 0x7f069c15e800 2024-09-16T01:39:04.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.317+0000 7f06a3331640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f069c06cde0 con 0x7f069c15ebe0 2024-09-16T01:39:04.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.317+0000 7f06a10cd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f069c15e800 0x7f069c0b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.317+0000 7f06a10cd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f069c15e800 0x7f069c0b5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52818/0 (socket says 172.21.15.88:52818) 2024-09-16T01:39:04.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.317+0000 7f06a10cd640 1 -- 172.21.15.88:0/324595787 learned_addr learned my addr 172.21.15.88:0/324595787 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:04.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.317+0000 7f06a08cc640 1 --2- 172.21.15.88:0/324595787 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f069c15ebe0 0x7f069c15f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.317+0000 7f06a10cd640 1 -- 172.21.15.88:0/324595787 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f069c15ebe0 msgr2=0x7f069c15f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:04.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.317+0000 7f06a10cd640 1 --2- 172.21.15.88:0/324595787 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f069c15ebe0 0x7f069c15f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.317+0000 7f06a10cd640 1 -- 172.21.15.88:0/324595787 >> v1:172.21.15.154:6789/0 conn(0x7f069c0b6e50 legacy=0x7f069c0b7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:04.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.317+0000 7f06a10cd640 1 -- 172.21.15.88:0/324595787 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f069c06d0c0 con 0x7f069c15e800 2024-09-16T01:39:04.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.318+0000 7f06a10cd640 1 --2- 172.21.15.88:0/324595787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f069c15e800 0x7f069c0b5460 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f0698009870 tx=0x7f069802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72d2bcfb87a11139 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:04.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.318+0000 7f068ffff640 1 -- 172.21.15.88:0/324595787 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0698037070 con 0x7f069c15e800 2024-09-16T01:39:04.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.318+0000 7f068ffff640 1 -- 172.21.15.88:0/324595787 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0698032440 con 0x7f069c15e800 2024-09-16T01:39:04.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.318+0000 7f068ffff640 1 -- 172.21.15.88:0/324595787 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0698032810 con 0x7f069c15e800 2024-09-16T01:39:04.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.318+0000 7f06a3331640 1 -- 172.21.15.88:0/324595787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f069c15e800 msgr2=0x7f069c0b5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:04.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.318+0000 7f06a3331640 1 --2- 172.21.15.88:0/324595787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f069c15e800 0x7f069c0b5460 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f0698009870 tx=0x7f069802a270 comp rx=0 tx=0).stop 2024-09-16T01:39:04.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.318+0000 7f06a3331640 1 -- 172.21.15.88:0/324595787 shutdown_connections 2024-09-16T01:39:04.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.318+0000 7f06a3331640 1 --2- 172.21.15.88:0/324595787 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f069c15ebe0 0x7f069c15f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.318+0000 7f06a3331640 1 --2- 172.21.15.88:0/324595787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f069c15e800 0x7f069c0b5460 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.318+0000 7f06a3331640 1 -- 172.21.15.88:0/324595787 >> 172.21.15.88:0/324595787 conn(0x7f069c0d24d0 msgr2=0x7f069c0d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:04.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.319+0000 7f06a3331640 1 -- 172.21.15.88:0/324595787 shutdown_connections 2024-09-16T01:39:04.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.319+0000 7f06a3331640 1 -- 172.21.15.88:0/324595787 wait complete. 2024-09-16T01:39:04.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.320+0000 7f06a3331640 1 Processor -- start 2024-09-16T01:39:04.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.320+0000 7f06a3331640 1 -- start start 2024-09-16T01:39:04.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.320+0000 7f06a3331640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f069c0b6e50 0x7f069c0cc140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.320+0000 7f06a3331640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f069c15e800 0x7f069c14bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.320+0000 7f06a3331640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f069c15ebe0 0x7f069c14c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.320+0000 7f06a3331640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f069c06e680 con 0x7f069c15ebe0 2024-09-16T01:39:04.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.320+0000 7f06a3331640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f069c0d31e0 con 0x7f069c15e800 2024-09-16T01:39:04.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.320+0000 7f06a3331640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f069c06e800 con 0x7f069c0b6e50 2024-09-16T01:39:04.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.320+0000 7f06a18ce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f069c15ebe0 0x7f069c14c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.321+0000 7f06a18ce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f069c15ebe0 0x7f069c14c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52830/0 (socket says 172.21.15.88:52830) 2024-09-16T01:39:04.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.321+0000 7f06a18ce640 1 -- 172.21.15.88:0/851297290 learned_addr learned my addr 172.21.15.88:0/851297290 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:04.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.321+0000 7f06a08cc640 1 --2- 172.21.15.88:0/851297290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f069c15e800 0x7f069c14bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.321+0000 7f06a10cd640 1 --2- 172.21.15.88:0/851297290 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f069c0b6e50 0x7f069c0cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.321+0000 7f06a18ce640 1 -- 172.21.15.88:0/851297290 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f069c0b6e50 msgr2=0x7f069c0cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:04.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.321+0000 7f06a18ce640 1 --2- 172.21.15.88:0/851297290 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f069c0b6e50 0x7f069c0cc140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.321+0000 7f06a18ce640 1 -- 172.21.15.88:0/851297290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f069c15e800 msgr2=0x7f069c14bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:04.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.321+0000 7f06a18ce640 1 --2- 172.21.15.88:0/851297290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f069c15e800 0x7f069c14bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.321+0000 7f06a18ce640 1 -- 172.21.15.88:0/851297290 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f069c159050 con 0x7f069c15ebe0 2024-09-16T01:39:04.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.321+0000 7f06a08cc640 1 --2- 172.21.15.88:0/851297290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f069c15e800 0x7f069c14bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:04.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.321+0000 7f06a10cd640 1 --2- 172.21.15.88:0/851297290 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f069c0b6e50 0x7f069c0cc140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:04.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.321+0000 7f06a18ce640 1 --2- 172.21.15.88:0/851297290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f069c15ebe0 0x7f069c14c0c0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f069400c7a0 tx=0x7f069400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:04.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.321+0000 7f068dffb640 1 -- 172.21.15.88:0/851297290 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0694003b70 con 0x7f069c15ebe0 2024-09-16T01:39:04.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.321+0000 7f068dffb640 1 -- 172.21.15.88:0/851297290 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0694003d30 con 0x7f069c15ebe0 2024-09-16T01:39:04.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.321+0000 7f06a3331640 1 -- 172.21.15.88:0/851297290 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f069c158a90 con 0x7f069c15ebe0 2024-09-16T01:39:04.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.321+0000 7f068dffb640 1 -- 172.21.15.88:0/851297290 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069400aa90 con 0x7f069c15ebe0 2024-09-16T01:39:04.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.321+0000 7f06a3331640 1 -- 172.21.15.88:0/851297290 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f069c159a30 con 0x7f069c15ebe0 2024-09-16T01:39:04.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.322+0000 7f06a3331640 1 -- 172.21.15.88:0/851297290 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f069c15f070 con 0x7f069c15ebe0 2024-09-16T01:39:04.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.322+0000 7f068dffb640 1 -- 172.21.15.88:0/851297290 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f069400ac30 con 0x7f069c15ebe0 2024-09-16T01:39:04.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.323+0000 7f068dffb640 1 --2- 172.21.15.88:0/851297290 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f068006c2b0 0x7f068006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.323+0000 7f068dffb640 1 -- 172.21.15.88:0/851297290 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5881+0+0 (secure 0 0 0) 0x7f069408b490 con 0x7f069c15ebe0 2024-09-16T01:39:04.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.323+0000 7f06a10cd640 1 --2- 172.21.15.88:0/851297290 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f068006c2b0 0x7f068006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.323+0000 7f06a10cd640 1 --2- 172.21.15.88:0/851297290 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f068006c2b0 0x7f068006e770 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0698034000 tx=0x7f069802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:04.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.327+0000 7f068dffb640 1 -- 172.21.15.88:0/851297290 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0694059b30 con 0x7f069c15ebe0 2024-09-16T01:39:04.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.528+0000 7f06a3331640 1 -- 172.21.15.88:0/851297290 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f069c070f20 con 0x7f069c15ebe0 2024-09-16T01:39:04.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.529+0000 7f068dffb640 1 -- 172.21.15.88:0/851297290 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+5225 (secure 0 0 0) 0x7f0694059430 con 0x7f069c15ebe0 2024-09-16T01:39:04.531 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:04.531 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":51,"btime":"2024-09-16T01:37:27:255277+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/581241194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":581241194},{"type":"v1","addr":"172.21.15.154:6834","nonce":581241194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2104508521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2104508521},{"type":"v1","addr":"172.21.15.154:6835","nonce":2104508521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":6195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2920926674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2920926674},{"type":"v1","addr":"172.21.15.88:6837","nonce":2920926674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:36:39.786438+0000","modified":"2024-09-16T01:37:27.255273+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5859},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5859":{"gid":5859,"name":"c","rank":0,"incarnation":48,"state":"up:active","state_seq":16,"addr":"172.21.15.88:6836/998283126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":998283126},{"type":"v1","addr":"172.21.15.88:6836","nonce":998283126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5859,"qdb_cluster":[5859]},"id":4}]} 2024-09-16T01:39:04.531 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 51 2024-09-16T01:39:04.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.531+0000 7f06a3331640 1 -- 172.21.15.88:0/851297290 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f068006c2b0 msgr2=0x7f068006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:04.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.531+0000 7f06a3331640 1 --2- 172.21.15.88:0/851297290 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f068006c2b0 0x7f068006e770 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0698034000 tx=0x7f069802e000 comp rx=0 tx=0).stop 2024-09-16T01:39:04.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.531+0000 7f06a3331640 1 -- 172.21.15.88:0/851297290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f069c15ebe0 msgr2=0x7f069c14c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:04.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.531+0000 7f06a3331640 1 --2- 172.21.15.88:0/851297290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f069c15ebe0 0x7f069c14c0c0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f069400c7a0 tx=0x7f069400cc60 comp rx=0 tx=0).stop 2024-09-16T01:39:04.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.531+0000 7f06a3331640 1 -- 172.21.15.88:0/851297290 shutdown_connections 2024-09-16T01:39:04.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.531+0000 7f06a3331640 1 --2- 172.21.15.88:0/851297290 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f068006c2b0 0x7f068006e770 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.531+0000 7f06a3331640 1 --2- 172.21.15.88:0/851297290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f069c15ebe0 0x7f069c14c0c0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.531+0000 7f06a3331640 1 --2- 172.21.15.88:0/851297290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f069c15e800 0x7f069c14bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.531+0000 7f06a3331640 1 --2- 172.21.15.88:0/851297290 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f069c0b6e50 0x7f069c0cc140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.531+0000 7f06a3331640 1 -- 172.21.15.88:0/851297290 >> 172.21.15.88:0/851297290 conn(0x7f069c0d24d0 msgr2=0x7f069c06f400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:04.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.531+0000 7f06a3331640 1 -- 172.21.15.88:0/851297290 shutdown_connections 2024-09-16T01:39:04.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.531+0000 7f06a3331640 1 -- 172.21.15.88:0/851297290 wait complete. 2024-09-16T01:39:04.543 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:39:04.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.640+0000 7f648d398640 1 Processor -- start 2024-09-16T01:39:04.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.640+0000 7f648d398640 1 -- start start 2024-09-16T01:39:04.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.641+0000 7f648d398640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f648806e910 0x7f64880735d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.641+0000 7f648d398640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6488073b10 0x7f64880b4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.641+0000 7f648d398640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f648815eb10 con 0x7f648815c100 2024-09-16T01:39:04.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.641+0000 7f648d398640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f648815edd0 con 0x7f648806e910 2024-09-16T01:39:04.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.641+0000 7f648d398640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f648815f0b0 con 0x7f6488073b10 2024-09-16T01:39:04.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.641+0000 7f6486ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f648815c100 0x7f648806e200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57414/0 (socket says 172.21.15.88:57414) 2024-09-16T01:39:04.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.641+0000 7f6486ffd640 1 -- 172.21.15.88:0/2831371747 learned_addr learned my addr 172.21.15.88:0/2831371747 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:04.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.641+0000 7f64867fc640 1 --2- 172.21.15.88:0/2831371747 >> v2:172.21.15.154:3300/0 conn(0x7f648806e910 0x7f64880735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.641+0000 7f64877fe640 1 --2- 172.21.15.88:0/2831371747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6488073b10 0x7f64880b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.641+0000 7f6485ffb640 1 -- 172.21.15.88:0/2831371747 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4015861893 0 0) 0x7f648815eb10 con 0x7f648815c100 2024-09-16T01:39:04.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.641+0000 7f6485ffb640 1 -- 172.21.15.88:0/2831371747 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6470003520 con 0x7f648815c100 2024-09-16T01:39:04.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.642+0000 7f6485ffb640 1 -- 172.21.15.88:0/2831371747 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1359051745 0 0) 0x7f6470003520 con 0x7f648815c100 2024-09-16T01:39:04.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.642+0000 7f6485ffb640 1 -- 172.21.15.88:0/2831371747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6488073b10 msgr2=0x7f64880b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:04.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.642+0000 7f6485ffb640 1 --2- 172.21.15.88:0/2831371747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6488073b10 0x7f64880b4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.642+0000 7f6485ffb640 1 -- 172.21.15.88:0/2831371747 >> v2:172.21.15.154:3300/0 conn(0x7f648806e910 msgr2=0x7f64880735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:04.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.642+0000 7f6485ffb640 1 --2- 172.21.15.88:0/2831371747 >> v2:172.21.15.154:3300/0 conn(0x7f648806e910 0x7f64880735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.642+0000 7f6485ffb640 1 -- 172.21.15.88:0/2831371747 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648815f390 con 0x7f648815c100 2024-09-16T01:39:04.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.642+0000 7f6485ffb640 1 -- 172.21.15.88:0/2831371747 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f6478001670 con 0x7f648815c100 2024-09-16T01:39:04.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.642+0000 7f6485ffb640 1 -- 172.21.15.88:0/2831371747 >> v1:172.21.15.88:6789/0 conn(0x7f648815c100 legacy=0x7f648806e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:04.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.642+0000 7f6485ffb640 1 --2- 172.21.15.88:0/2831371747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6488073b10 0x7f64700039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.642+0000 7f6485ffb640 1 --2- 172.21.15.88:0/2831371747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6470003f30 0x7f6470004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.642+0000 7f6485ffb640 1 --2- 172.21.15.88:0/2831371747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6470004d40 0x7f64700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.642+0000 7f6485ffb640 1 -- 172.21.15.88:0/2831371747 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f648815edd0 con 0x7f6470003f30 2024-09-16T01:39:04.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.642+0000 7f6485ffb640 1 -- 172.21.15.88:0/2831371747 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f648815f0b0 con 0x7f6470004d40 2024-09-16T01:39:04.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.642+0000 7f6485ffb640 1 -- 172.21.15.88:0/2831371747 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f647000b9c0 con 0x7f6488073b10 2024-09-16T01:39:04.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.642+0000 7f64867fc640 1 --2- 172.21.15.88:0/2831371747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6470003f30 0x7f6470004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.642+0000 7f6486ffd640 1 --2- 172.21.15.88:0/2831371747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6488073b10 0x7f64700039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.642+0000 7f64877fe640 1 --2- 172.21.15.88:0/2831371747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6470004d40 0x7f64700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.642+0000 7f64867fc640 1 -- 172.21.15.88:0/2831371747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6488073b10 msgr2=0x7f64700039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:04.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.642+0000 7f64867fc640 1 --2- 172.21.15.88:0/2831371747 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6488073b10 0x7f64700039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.642+0000 7f64867fc640 1 -- 172.21.15.88:0/2831371747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6470004d40 msgr2=0x7f64700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:04.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.642+0000 7f64867fc640 1 --2- 172.21.15.88:0/2831371747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6470004d40 0x7f64700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.642+0000 7f64867fc640 1 -- 172.21.15.88:0/2831371747 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6470003520 con 0x7f6470003f30 2024-09-16T01:39:04.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.642+0000 7f64877fe640 1 --2- 172.21.15.88:0/2831371747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6470004d40 0x7f64700051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:04.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.643+0000 7f64867fc640 1 --2- 172.21.15.88:0/2831371747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6470003f30 0x7f6470004800 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f647c009ac0 tx=0x7f6470005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75c441bc78603708 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:04.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.643+0000 7f6485ffb640 1 -- 172.21.15.88:0/2831371747 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c036070 con 0x7f6470003f30 2024-09-16T01:39:04.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.643+0000 7f6485ffb640 1 -- 172.21.15.88:0/2831371747 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f647c031400 con 0x7f6470003f30 2024-09-16T01:39:04.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.643+0000 7f6485ffb640 1 -- 172.21.15.88:0/2831371747 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c009690 con 0x7f6470003f30 2024-09-16T01:39:04.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.644+0000 7f648d398640 1 -- 172.21.15.88:0/2831371747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6470003f30 msgr2=0x7f6470004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:04.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.644+0000 7f648d398640 1 --2- 172.21.15.88:0/2831371747 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6470003f30 0x7f6470004800 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f647c009ac0 tx=0x7f6470005830 comp rx=0 tx=0).stop 2024-09-16T01:39:04.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.644+0000 7f64877fe640 1 -- 172.21.15.88:0/2831371747 reap_dead start 2024-09-16T01:39:04.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.644+0000 7f648d398640 1 -- 172.21.15.88:0/2831371747 shutdown_connections 2024-09-16T01:39:04.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.644+0000 7f648d398640 1 -- 172.21.15.88:0/2831371747 >> 172.21.15.88:0/2831371747 conn(0x7f64880b6ec0 msgr2=0x7f64880b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:04.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.644+0000 7f648d398640 1 -- 172.21.15.88:0/2831371747 shutdown_connections 2024-09-16T01:39:04.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.644+0000 7f648d398640 1 -- 172.21.15.88:0/2831371747 wait complete. 2024-09-16T01:39:04.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.645+0000 7f648d398640 1 Processor -- start 2024-09-16T01:39:04.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.645+0000 7f648d398640 1 -- start start 2024-09-16T01:39:04.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.646+0000 7f648d398640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6470003f30 0x7f648815fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.646+0000 7f648d398640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64881601f0 0x7f648815c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.646+0000 7f648d398640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6488140760 0x7f648815cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.646+0000 7f648d398640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6488073fc0 con 0x7f64881601f0 2024-09-16T01:39:04.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.646+0000 7f648d398640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f648815c4e0 con 0x7f6488140760 2024-09-16T01:39:04.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.646+0000 7f648d398640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6488074140 con 0x7f6470003f30 2024-09-16T01:39:04.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.646+0000 7f64867fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64881601f0 0x7f648815c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.646+0000 7f64867fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64881601f0 0x7f648815c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52854/0 (socket says 172.21.15.88:52854) 2024-09-16T01:39:04.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.646+0000 7f64867fc640 1 -- 172.21.15.88:0/3987043593 learned_addr learned my addr 172.21.15.88:0/3987043593 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:04.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.646+0000 7f64877fe640 1 --2- 172.21.15.88:0/3987043593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6488140760 0x7f648815cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.646+0000 7f6486ffd640 1 --2- 172.21.15.88:0/3987043593 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6470003f30 0x7f648815fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.646+0000 7f64867fc640 1 -- 172.21.15.88:0/3987043593 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6470003f30 msgr2=0x7f648815fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:04.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.646+0000 7f64867fc640 1 --2- 172.21.15.88:0/3987043593 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6470003f30 0x7f648815fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.646+0000 7f64867fc640 1 -- 172.21.15.88:0/3987043593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6488140760 msgr2=0x7f648815cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:04.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.646+0000 7f64867fc640 1 --2- 172.21.15.88:0/3987043593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6488140760 0x7f648815cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.646+0000 7f64867fc640 1 -- 172.21.15.88:0/3987043593 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6488162e10 con 0x7f64881601f0 2024-09-16T01:39:04.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.646+0000 7f64867fc640 1 --2- 172.21.15.88:0/3987043593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64881601f0 0x7f648815c660 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f647c034040 tx=0x7f647c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:04.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.646+0000 7f646f7fe640 1 -- 172.21.15.88:0/3987043593 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c040020 con 0x7f64881601f0 2024-09-16T01:39:04.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.646+0000 7f646f7fe640 1 -- 172.21.15.88:0/3987043593 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f647c003c60 con 0x7f64881601f0 2024-09-16T01:39:04.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.646+0000 7f648d398640 1 -- 172.21.15.88:0/3987043593 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6488161d70 con 0x7f64881601f0 2024-09-16T01:39:04.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.646+0000 7f646f7fe640 1 -- 172.21.15.88:0/3987043593 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c036040 con 0x7f64881601f0 2024-09-16T01:39:04.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.646+0000 7f648d398640 1 -- 172.21.15.88:0/3987043593 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6488162330 con 0x7f64881601f0 2024-09-16T01:39:04.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.647+0000 7f648d398640 1 -- 172.21.15.88:0/3987043593 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64880735d0 con 0x7f64881601f0 2024-09-16T01:39:04.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.648+0000 7f646f7fe640 1 -- 172.21.15.88:0/3987043593 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f647c043020 con 0x7f64881601f0 2024-09-16T01:39:04.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.648+0000 7f646f7fe640 1 --2- 172.21.15.88:0/3987043593 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f647406c2e0 0x7f647406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.648+0000 7f646f7fe640 1 -- 172.21.15.88:0/3987043593 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5881+0+0 (secure 0 0 0) 0x7f647c0aba20 con 0x7f64881601f0 2024-09-16T01:39:04.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.648+0000 7f6486ffd640 1 --2- 172.21.15.88:0/3987043593 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f647406c2e0 0x7f647406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.649+0000 7f6486ffd640 1 --2- 172.21.15.88:0/3987043593 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f647406c2e0 0x7f647406e7a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6478007fd0 tx=0x7f6478009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:04.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.652+0000 7f646f7fe640 1 -- 172.21.15.88:0/3987043593 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f647c04a030 con 0x7f64881601f0 2024-09-16T01:39:04.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.818+0000 7f648d398640 1 -- 172.21.15.88:0/3987043593 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f648806ceb0 con 0x7f64881601f0 2024-09-16T01:39:04.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.819+0000 7f646f7fe640 1 -- 172.21.15.88:0/3987043593 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) ==== 74+0+16958 (secure 0 0 0) 0x7f647c0044e0 con 0x7f64881601f0 2024-09-16T01:39:04.821 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:04.821 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":60,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:37:24.240775+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:36:38.121335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-16T01:36:38.755908+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:6837/4130074102":"2024-09-17T01:37:24.240722+0000","172.21.15.88:6835/4130074102":"2024-09-17T01:37:24.240722+0000","172.21.15.88:6837/574169866":"2024-09-17T01:36:48.820862+0000","172.21.15.88:6835/574169866":"2024-09-17T01:36:48.820862+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:39:04.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.821+0000 7f648d398640 1 -- 172.21.15.88:0/3987043593 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f647406c2e0 msgr2=0x7f647406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:04.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.821+0000 7f648d398640 1 --2- 172.21.15.88:0/3987043593 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f647406c2e0 0x7f647406e7a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6478007fd0 tx=0x7f6478009040 comp rx=0 tx=0).stop 2024-09-16T01:39:04.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.821+0000 7f648d398640 1 -- 172.21.15.88:0/3987043593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64881601f0 msgr2=0x7f648815c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:04.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.821+0000 7f648d398640 1 --2- 172.21.15.88:0/3987043593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64881601f0 0x7f648815c660 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f647c034040 tx=0x7f647c004730 comp rx=0 tx=0).stop 2024-09-16T01:39:04.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.821+0000 7f648d398640 1 -- 172.21.15.88:0/3987043593 shutdown_connections 2024-09-16T01:39:04.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.821+0000 7f648d398640 1 --2- 172.21.15.88:0/3987043593 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f647406c2e0 0x7f647406e7a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.821+0000 7f648d398640 1 --2- 172.21.15.88:0/3987043593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6488140760 0x7f648815cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.821+0000 7f648d398640 1 --2- 172.21.15.88:0/3987043593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64881601f0 0x7f648815c660 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.821+0000 7f648d398640 1 --2- 172.21.15.88:0/3987043593 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6470003f30 0x7f648815fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.821+0000 7f648d398640 1 -- 172.21.15.88:0/3987043593 >> 172.21.15.88:0/3987043593 conn(0x7f64880b6ec0 msgr2=0x7f648815aae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:04.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.821+0000 7f648d398640 1 -- 172.21.15.88:0/3987043593 shutdown_connections 2024-09-16T01:39:04.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.821+0000 7f648d398640 1 -- 172.21.15.88:0/3987043593 wait complete. 2024-09-16T01:39:04.834 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:39:04.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.931+0000 7f6cfeb9d640 1 Processor -- start 2024-09-16T01:39:04.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.931+0000 7f6cfeb9d640 1 -- start start 2024-09-16T01:39:04.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.932+0000 7f6cfeb9d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6cf80d1dc0 0x7f6cf81692b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.932+0000 7f6cfeb9d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cf806ed10 con 0x7f6cf815e700 2024-09-16T01:39:04.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.932+0000 7f6cfeb9d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cf815eae0 con 0x7f6cf80b53f0 2024-09-16T01:39:04.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.932+0000 7f6cfeb9d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6cf8070820 con 0x7f6cf80d1dc0 2024-09-16T01:39:04.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.932+0000 7f6cfdb9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6cf815e700 0x7f6cf806e530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57426/0 (socket says 172.21.15.88:57426) 2024-09-16T01:39:04.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.932+0000 7f6cfdb9b640 1 -- 172.21.15.88:0/2111890140 learned_addr learned my addr 172.21.15.88:0/2111890140 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:04.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.933+0000 7f6cfcb99640 1 -- 172.21.15.88:0/2111890140 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3011727585 0 0) 0x7f6cf806ed10 con 0x7f6cf815e700 2024-09-16T01:39:04.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.933+0000 7f6cfcb99640 1 -- 172.21.15.88:0/2111890140 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ce0003520 con 0x7f6cf815e700 2024-09-16T01:39:04.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.933+0000 7f6cfe39c640 1 --2- 172.21.15.88:0/2111890140 >> v2:172.21.15.154:3300/0 conn(0x7f6cf80d1dc0 0x7f6cf81692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.933+0000 7f6cfcb99640 1 -- 172.21.15.88:0/2111890140 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2411828431 0 0) 0x7f6ce0003520 con 0x7f6cf815e700 2024-09-16T01:39:04.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.933+0000 7f6cfcb99640 1 -- 172.21.15.88:0/2111890140 >> v2:172.21.15.154:3300/0 conn(0x7f6cf80d1dc0 msgr2=0x7f6cf81692b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:04.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.933+0000 7f6cfcb99640 1 --2- 172.21.15.88:0/2111890140 >> v2:172.21.15.154:3300/0 conn(0x7f6cf80d1dc0 0x7f6cf81692b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.933+0000 7f6cfcb99640 1 -- 172.21.15.88:0/2111890140 >> v1:172.21.15.154:6789/0 conn(0x7f6cf80b53f0 legacy=0x7f6cf80d16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:04.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.933+0000 7f6cfcb99640 1 -- 172.21.15.88:0/2111890140 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf8070b00 con 0x7f6cf815e700 2024-09-16T01:39:04.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.933+0000 7f6cfcb99640 1 -- 172.21.15.88:0/2111890140 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f6ce80019f0 con 0x7f6cf815e700 2024-09-16T01:39:04.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.933+0000 7f6cfcb99640 1 -- 172.21.15.88:0/2111890140 >> v1:172.21.15.88:6789/0 conn(0x7f6cf815e700 legacy=0x7f6cf806e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:04.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.933+0000 7f6cfcb99640 1 --2- 172.21.15.88:0/2111890140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ce00039b0 0x7f6ce0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.933+0000 7f6cfcb99640 1 --2- 172.21.15.88:0/2111890140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ce0004800 0x7f6ce0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.933+0000 7f6cfcb99640 1 --2- 172.21.15.88:0/2111890140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ce00051c0 0x7f6ce000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.933+0000 7f6cfcb99640 1 -- 172.21.15.88:0/2111890140 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf8070820 con 0x7f6ce0004800 2024-09-16T01:39:04.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.933+0000 7f6cfcb99640 1 -- 172.21.15.88:0/2111890140 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce000c110 con 0x7f6ce00039b0 2024-09-16T01:39:04.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.933+0000 7f6cfcb99640 1 -- 172.21.15.88:0/2111890140 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6ce000c310 con 0x7f6ce00051c0 2024-09-16T01:39:04.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.933+0000 7f6cfd39a640 1 --2- 172.21.15.88:0/2111890140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ce0004800 0x7f6ce0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.933+0000 7f6cfdb9b640 1 --2- 172.21.15.88:0/2111890140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ce00039b0 0x7f6ce0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.933+0000 7f6cfe39c640 1 --2- 172.21.15.88:0/2111890140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ce00051c0 0x7f6ce000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.933+0000 7f6cfd39a640 1 -- 172.21.15.88:0/2111890140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ce00051c0 msgr2=0x7f6ce000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:04.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.933+0000 7f6cfd39a640 1 --2- 172.21.15.88:0/2111890140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ce00051c0 0x7f6ce000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.933+0000 7f6cfd39a640 1 -- 172.21.15.88:0/2111890140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ce00039b0 msgr2=0x7f6ce0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:04.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.933+0000 7f6cfd39a640 1 --2- 172.21.15.88:0/2111890140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ce00039b0 0x7f6ce0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.933+0000 7f6cfe39c640 1 -- 172.21.15.88:0/2111890140 reap_dead start 2024-09-16T01:39:04.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.933+0000 7f6cfd39a640 1 -- 172.21.15.88:0/2111890140 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ce0003520 con 0x7f6ce0004800 2024-09-16T01:39:04.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.934+0000 7f6cfd39a640 1 --2- 172.21.15.88:0/2111890140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ce0004800 0x7f6ce0004c80 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f6cec001750 tx=0x7f6ce0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9167115145aa16c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:04.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.934+0000 7f6cfcb99640 1 -- 172.21.15.88:0/2111890140 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cec001370 con 0x7f6ce0004800 2024-09-16T01:39:04.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.934+0000 7f6cfcb99640 1 -- 172.21.15.88:0/2111890140 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cec0052d0 con 0x7f6ce0004800 2024-09-16T01:39:04.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.934+0000 7f6cfcb99640 1 -- 172.21.15.88:0/2111890140 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cec02abc0 con 0x7f6ce0004800 2024-09-16T01:39:04.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.935+0000 7f6cfeb9d640 1 -- 172.21.15.88:0/2111890140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ce0004800 msgr2=0x7f6ce0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:04.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.935+0000 7f6cfeb9d640 1 --2- 172.21.15.88:0/2111890140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ce0004800 0x7f6ce0004c80 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f6cec001750 tx=0x7f6ce0005830 comp rx=0 tx=0).stop 2024-09-16T01:39:04.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.935+0000 7f6cfeb9d640 1 -- 172.21.15.88:0/2111890140 shutdown_connections 2024-09-16T01:39:04.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.935+0000 7f6cfeb9d640 1 --2- 172.21.15.88:0/2111890140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ce0004800 0x7f6ce0004c80 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.935+0000 7f6cfeb9d640 1 -- 172.21.15.88:0/2111890140 >> 172.21.15.88:0/2111890140 conn(0x7f6cf80d2340 msgr2=0x7f6cf80d2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:04.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.935+0000 7f6cfeb9d640 1 -- 172.21.15.88:0/2111890140 shutdown_connections 2024-09-16T01:39:04.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.935+0000 7f6cfeb9d640 1 -- 172.21.15.88:0/2111890140 wait complete. 2024-09-16T01:39:04.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.936+0000 7f6cfeb9d640 1 Processor -- start 2024-09-16T01:39:04.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.936+0000 7f6cfeb9d640 1 -- start start 2024-09-16T01:39:04.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.937+0000 7f6cfeb9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ce0004800 0x7f6cf814cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.937+0000 7f6cfeb9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf80b2540 0x7f6cf80b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.937+0000 7f6cfeb9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6cf8158290 0x7f6cf8158710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.937+0000 7f6cfeb9d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf80d16b0 con 0x7f6ce0004800 2024-09-16T01:39:04.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.937+0000 7f6cfeb9d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf80710c0 con 0x7f6cf80b2540 2024-09-16T01:39:04.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.937+0000 7f6cfeb9d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6cf80d1830 con 0x7f6cf8158290 2024-09-16T01:39:04.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.937+0000 7f6cfdb9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ce0004800 0x7f6cf814cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.937+0000 7f6cfdb9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ce0004800 0x7f6cf814cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52876/0 (socket says 172.21.15.88:52876) 2024-09-16T01:39:04.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.937+0000 7f6cfdb9b640 1 -- 172.21.15.88:0/2463288245 learned_addr learned my addr 172.21.15.88:0/2463288245 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:04.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.937+0000 7f6cfd39a640 1 --2- 172.21.15.88:0/2463288245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf80b2540 0x7f6cf80b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.937+0000 7f6cfe39c640 1 --2- 172.21.15.88:0/2463288245 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6cf8158290 0x7f6cf8158710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.937+0000 7f6cfdb9b640 1 -- 172.21.15.88:0/2463288245 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6cf8158290 msgr2=0x7f6cf8158710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:04.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.937+0000 7f6cfdb9b640 1 --2- 172.21.15.88:0/2463288245 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6cf8158290 0x7f6cf8158710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.937+0000 7f6cfdb9b640 1 -- 172.21.15.88:0/2463288245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf80b2540 msgr2=0x7f6cf80b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:04.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.937+0000 7f6cfdb9b640 1 --2- 172.21.15.88:0/2463288245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf80b2540 0x7f6cf80b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:04.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.937+0000 7f6cfdb9b640 1 -- 172.21.15.88:0/2463288245 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf8158e70 con 0x7f6ce0004800 2024-09-16T01:39:04.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.937+0000 7f6cfd39a640 1 --2- 172.21.15.88:0/2463288245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf80b2540 0x7f6cf80b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:04.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.937+0000 7f6cfe39c640 1 --2- 172.21.15.88:0/2463288245 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6cf8158290 0x7f6cf8158710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:04.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.937+0000 7f6cfdb9b640 1 --2- 172.21.15.88:0/2463288245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ce0004800 0x7f6cf814cdd0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8001560 tx=0x7f6ce800a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:04.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.937+0000 7f6cf67fc640 1 -- 172.21.15.88:0/2463288245 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8017020 con 0x7f6ce0004800 2024-09-16T01:39:04.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.937+0000 7f6cf67fc640 1 -- 172.21.15.88:0/2463288245 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ce800ad60 con 0x7f6ce0004800 2024-09-16T01:39:04.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.937+0000 7f6cfeb9d640 1 -- 172.21.15.88:0/2463288245 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cf8159150 con 0x7f6ce0004800 2024-09-16T01:39:04.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.937+0000 7f6cf67fc640 1 -- 172.21.15.88:0/2463288245 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8013630 con 0x7f6ce0004800 2024-09-16T01:39:04.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.937+0000 7f6cfeb9d640 1 -- 172.21.15.88:0/2463288245 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cf8159710 con 0x7f6ce0004800 2024-09-16T01:39:04.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.938+0000 7f6cfeb9d640 1 -- 172.21.15.88:0/2463288245 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cf806ae10 con 0x7f6ce0004800 2024-09-16T01:39:04.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.939+0000 7f6cf67fc640 1 -- 172.21.15.88:0/2463288245 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6ce801a020 con 0x7f6ce0004800 2024-09-16T01:39:04.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.939+0000 7f6cf67fc640 1 --2- 172.21.15.88:0/2463288245 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ce406c2b0 0x7f6ce406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:04.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.939+0000 7f6cf67fc640 1 -- 172.21.15.88:0/2463288245 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5881+0+0 (secure 0 0 0) 0x7f6ce808bfb0 con 0x7f6ce0004800 2024-09-16T01:39:04.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.939+0000 7f6cfd39a640 1 --2- 172.21.15.88:0/2463288245 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ce406c2b0 0x7f6ce406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:04.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.940+0000 7f6cfd39a640 1 --2- 172.21.15.88:0/2463288245 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ce406c2b0 0x7f6ce406e770 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6cec032000 tx=0x7f6cec02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:04.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:04.943+0000 7f6cf67fc640 1 -- 172.21.15.88:0/2463288245 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6ce8021020 con 0x7f6ce0004800 2024-09-16T01:39:05.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.143+0000 7f6cfeb9d640 1 -- 172.21.15.88:0/2463288245 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f6cf80b3ed0 con 0x7f6ce0004800 2024-09-16T01:39:05.145 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:39:05.144+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:39:05.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.145+0000 7f6cf67fc640 1 -- 172.21.15.88:0/2463288245 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v52) ==== 161+0+0 (secure 0 0 0) 0x7f6ce805a670 con 0x7f6ce0004800 2024-09-16T01:39:05.146 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:39:05.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.148+0000 7f6cfeb9d640 1 -- 172.21.15.88:0/2463288245 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ce406c2b0 msgr2=0x7f6ce406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:05.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.148+0000 7f6cfeb9d640 1 --2- 172.21.15.88:0/2463288245 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ce406c2b0 0x7f6ce406e770 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6cec032000 tx=0x7f6cec02e000 comp rx=0 tx=0).stop 2024-09-16T01:39:05.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.148+0000 7f6cfeb9d640 1 -- 172.21.15.88:0/2463288245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ce0004800 msgr2=0x7f6cf814cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:05.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.148+0000 7f6cfeb9d640 1 --2- 172.21.15.88:0/2463288245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ce0004800 0x7f6cf814cdd0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8001560 tx=0x7f6ce800a730 comp rx=0 tx=0).stop 2024-09-16T01:39:05.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.148+0000 7f6cfeb9d640 1 -- 172.21.15.88:0/2463288245 shutdown_connections 2024-09-16T01:39:05.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.148+0000 7f6cfeb9d640 1 --2- 172.21.15.88:0/2463288245 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ce406c2b0 0x7f6ce406e770 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:05.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.148+0000 7f6cfeb9d640 1 --2- 172.21.15.88:0/2463288245 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6cf8158290 0x7f6cf8158710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:05.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.148+0000 7f6cfeb9d640 1 --2- 172.21.15.88:0/2463288245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf80b2540 0x7f6cf80b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:05.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.148+0000 7f6cfeb9d640 1 --2- 172.21.15.88:0/2463288245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ce0004800 0x7f6cf814cdd0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:05.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.149+0000 7f6cfeb9d640 1 -- 172.21.15.88:0/2463288245 >> 172.21.15.88:0/2463288245 conn(0x7f6cf80d2340 msgr2=0x7f6cf806f3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:05.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.149+0000 7f6cfeb9d640 1 -- 172.21.15.88:0/2463288245 shutdown_connections 2024-09-16T01:39:05.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.149+0000 7f6cfeb9d640 1 -- 172.21.15.88:0/2463288245 wait complete. 2024-09-16T01:39:05.152 INFO:tasks.ceph.mds.c.smithi088.stderr: -3992> 2024-09-16T01:37:31.442+0000 7f1648390640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:39:05.161 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-16T01:39:05.181 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:39:05.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.261+0000 7fde9a653640 1 Processor -- start 2024-09-16T01:39:05.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.261+0000 7fde9a653640 1 -- start start 2024-09-16T01:39:05.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.262+0000 7fde9a653640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fde9406e5b0 0x7fde94073b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:05.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.262+0000 7fde9a653640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fde9415f7f0 0x7fde9406cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:05.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.262+0000 7fde9a653640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde9406eae0 con 0x7fde940b6d50 2024-09-16T01:39:05.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.262+0000 7fde9a653640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fde9415ed10 con 0x7fde9406e5b0 2024-09-16T01:39:05.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.262+0000 7fde9a653640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fde9415eff0 con 0x7fde9415f7f0 2024-09-16T01:39:05.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.262+0000 7fde937fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fde9406e5b0 0x7fde94073b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:05.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.263+0000 7fde937fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fde9406e5b0 0x7fde94073b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52904/0 (socket says 172.21.15.88:52904) 2024-09-16T01:39:05.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.263+0000 7fde937fe640 1 -- 172.21.15.88:0/2107153633 learned_addr learned my addr 172.21.15.88:0/2107153633 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:05.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.263+0000 7fde98bf0640 1 --2- 172.21.15.88:0/2107153633 >> v2:172.21.15.154:3300/0 conn(0x7fde9415f7f0 0x7fde9406cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:05.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.263+0000 7fde937fe640 1 -- 172.21.15.88:0/2107153633 >> v2:172.21.15.154:3300/0 conn(0x7fde9415f7f0 msgr2=0x7fde9406cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:05.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.263+0000 7fde937fe640 1 --2- 172.21.15.88:0/2107153633 >> v2:172.21.15.154:3300/0 conn(0x7fde9415f7f0 0x7fde9406cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:05.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.263+0000 7fde937fe640 1 -- 172.21.15.88:0/2107153633 >> v1:172.21.15.154:6789/0 conn(0x7fde940b6d50 legacy=0x7fde940b7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:05.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.263+0000 7fde937fe640 1 -- 172.21.15.88:0/2107153633 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde9415f2d0 con 0x7fde9406e5b0 2024-09-16T01:39:05.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.263+0000 7fde937fe640 1 --2- 172.21.15.88:0/2107153633 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde9406e5b0 0x7fde94073b70 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fde94072f20 tx=0x7fde8802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e2be2e58de3720 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:05.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.264+0000 7fde92ffd640 1 -- 172.21.15.88:0/2107153633 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde88037070 con 0x7fde9406e5b0 2024-09-16T01:39:05.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.264+0000 7fde92ffd640 1 -- 172.21.15.88:0/2107153633 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde88032440 con 0x7fde9406e5b0 2024-09-16T01:39:05.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.264+0000 7fde92ffd640 1 -- 172.21.15.88:0/2107153633 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde88032810 con 0x7fde9406e5b0 2024-09-16T01:39:05.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.264+0000 7fde9a653640 1 -- 172.21.15.88:0/2107153633 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde9406e5b0 msgr2=0x7fde94073b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:05.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.264+0000 7fde9a653640 1 --2- 172.21.15.88:0/2107153633 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde9406e5b0 0x7fde94073b70 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fde94072f20 tx=0x7fde8802a270 comp rx=0 tx=0).stop 2024-09-16T01:39:05.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.264+0000 7fde9a653640 1 -- 172.21.15.88:0/2107153633 shutdown_connections 2024-09-16T01:39:05.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.264+0000 7fde9a653640 1 --2- 172.21.15.88:0/2107153633 >> v2:172.21.15.154:3300/0 conn(0x7fde9415f7f0 0x7fde9406cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:05.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.264+0000 7fde9a653640 1 --2- 172.21.15.88:0/2107153633 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde9406e5b0 0x7fde94073b70 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:05.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.264+0000 7fde9a653640 1 -- 172.21.15.88:0/2107153633 >> 172.21.15.88:0/2107153633 conn(0x7fde940d24d0 msgr2=0x7fde940d28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:05.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.264+0000 7fde9a653640 1 -- 172.21.15.88:0/2107153633 shutdown_connections 2024-09-16T01:39:05.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.264+0000 7fde9a653640 1 -- 172.21.15.88:0/2107153633 wait complete. 2024-09-16T01:39:05.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.265+0000 7fde9a653640 1 Processor -- start 2024-09-16T01:39:05.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.266+0000 7fde9a653640 1 -- start start 2024-09-16T01:39:05.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.266+0000 7fde9a653640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde9406e5b0 0x7fde9415c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:05.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.266+0000 7fde9a653640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde940b6d50 0x7fde941585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:05.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.266+0000 7fde9a653640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde9415f7f0 0x7fde94145a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:05.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.266+0000 7fde9a653640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fde9415e8b0 con 0x7fde940b6d50 2024-09-16T01:39:05.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.266+0000 7fde9a653640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fde9406edc0 con 0x7fde9406e5b0 2024-09-16T01:39:05.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.266+0000 7fde9a653640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fde9415ea30 con 0x7fde9415f7f0 2024-09-16T01:39:05.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.266+0000 7fde937fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde940b6d50 0x7fde941585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:05.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.266+0000 7fde937fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde940b6d50 0x7fde941585b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52916/0 (socket says 172.21.15.88:52916) 2024-09-16T01:39:05.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.266+0000 7fde937fe640 1 -- 172.21.15.88:0/1857395837 learned_addr learned my addr 172.21.15.88:0/1857395837 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:05.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.267+0000 7fde93fff640 1 --2- 172.21.15.88:0/1857395837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde9406e5b0 0x7fde9415c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:05.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.267+0000 7fde98bf0640 1 --2- 172.21.15.88:0/1857395837 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde9415f7f0 0x7fde94145a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:05.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.267+0000 7fde937fe640 1 -- 172.21.15.88:0/1857395837 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde9415f7f0 msgr2=0x7fde94145a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:05.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.267+0000 7fde937fe640 1 --2- 172.21.15.88:0/1857395837 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde9415f7f0 0x7fde94145a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:05.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.267+0000 7fde937fe640 1 -- 172.21.15.88:0/1857395837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde9406e5b0 msgr2=0x7fde9415c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:05.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.267+0000 7fde937fe640 1 --2- 172.21.15.88:0/1857395837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde9406e5b0 0x7fde9415c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:05.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.267+0000 7fde937fe640 1 -- 172.21.15.88:0/1857395837 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde94156c90 con 0x7fde940b6d50 2024-09-16T01:39:05.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.267+0000 7fde93fff640 1 --2- 172.21.15.88:0/1857395837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde9406e5b0 0x7fde9415c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:05.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.267+0000 7fde937fe640 1 --2- 172.21.15.88:0/1857395837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde940b6d50 0x7fde941585b0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fde8802a2b0 tx=0x7fde880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:05.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.267+0000 7fde90ff9640 1 -- 172.21.15.88:0/1857395837 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde88041020 con 0x7fde940b6d50 2024-09-16T01:39:05.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.267+0000 7fde90ff9640 1 -- 172.21.15.88:0/1857395837 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde880306c0 con 0x7fde940b6d50 2024-09-16T01:39:05.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.267+0000 7fde9a653640 1 -- 172.21.15.88:0/1857395837 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde9415acd0 con 0x7fde940b6d50 2024-09-16T01:39:05.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.267+0000 7fde90ff9640 1 -- 172.21.15.88:0/1857395837 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde88037040 con 0x7fde940b6d50 2024-09-16T01:39:05.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.267+0000 7fde9a653640 1 -- 172.21.15.88:0/1857395837 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde941e8450 con 0x7fde940b6d50 2024-09-16T01:39:05.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.268+0000 7fde9a653640 1 -- 172.21.15.88:0/1857395837 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde94073b70 con 0x7fde940b6d50 2024-09-16T01:39:05.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.268+0000 7fde90ff9640 1 -- 172.21.15.88:0/1857395837 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fde88044020 con 0x7fde940b6d50 2024-09-16T01:39:05.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.269+0000 7fde90ff9640 1 --2- 172.21.15.88:0/1857395837 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fde8006c2e0 0x7fde8006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:05.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.269+0000 7fde90ff9640 1 -- 172.21.15.88:0/1857395837 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5967+0+0 (secure 0 0 0) 0x7fde880abcc0 con 0x7fde940b6d50 2024-09-16T01:39:05.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.269+0000 7fde93fff640 1 --2- 172.21.15.88:0/1857395837 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fde8006c2e0 0x7fde8006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:05.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.269+0000 7fde93fff640 1 --2- 172.21.15.88:0/1857395837 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fde8006c2e0 0x7fde8006e7a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fde84001700 tx=0x7fde84009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:05.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.273+0000 7fde90ff9640 1 -- 172.21.15.88:0/1857395837 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fde8804b030 con 0x7fde940b6d50 2024-09-16T01:39:05.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:05.474+0000 7fde9a653640 1 -- 172.21.15.88:0/1857395837 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fde940693c0 con 0x7fde940b6d50 2024-09-16T01:39:06.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.146+0000 7fde90ff9640 1 -- 172.21.15.88:0/1857395837 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v53) ==== 104+0+0 (secure 0 0 0) 0x7fde8807a3c0 con 0x7fde940b6d50 2024-09-16T01:39:06.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.149+0000 7fde9a653640 1 -- 172.21.15.88:0/1857395837 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fde8006c2e0 msgr2=0x7fde8006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:06.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.149+0000 7fde9a653640 1 --2- 172.21.15.88:0/1857395837 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fde8006c2e0 0x7fde8006e7a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fde84001700 tx=0x7fde84009040 comp rx=0 tx=0).stop 2024-09-16T01:39:06.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.149+0000 7fde9a653640 1 -- 172.21.15.88:0/1857395837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde940b6d50 msgr2=0x7fde941585b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:06.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.149+0000 7fde9a653640 1 --2- 172.21.15.88:0/1857395837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde940b6d50 0x7fde941585b0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fde8802a2b0 tx=0x7fde880046c0 comp rx=0 tx=0).stop 2024-09-16T01:39:06.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.149+0000 7fde9a653640 1 -- 172.21.15.88:0/1857395837 shutdown_connections 2024-09-16T01:39:06.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.149+0000 7fde9a653640 1 --2- 172.21.15.88:0/1857395837 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fde8006c2e0 0x7fde8006e7a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:06.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.149+0000 7fde9a653640 1 --2- 172.21.15.88:0/1857395837 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde9415f7f0 0x7fde94145a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:06.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.149+0000 7fde9a653640 1 --2- 172.21.15.88:0/1857395837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde940b6d50 0x7fde941585b0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:06.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.149+0000 7fde9a653640 1 --2- 172.21.15.88:0/1857395837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde9406e5b0 0x7fde9415c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:06.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.149+0000 7fde9a653640 1 -- 172.21.15.88:0/1857395837 >> 172.21.15.88:0/1857395837 conn(0x7fde940d24d0 msgr2=0x7fde94073160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:06.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.149+0000 7fde9a653640 1 -- 172.21.15.88:0/1857395837 shutdown_connections 2024-09-16T01:39:06.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.149+0000 7fde9a653640 1 -- 172.21.15.88:0/1857395837 wait complete. 2024-09-16T01:39:06.163 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-16T01:39:06.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.306+0000 7f9551b8d640 1 Processor -- start 2024-09-16T01:39:06.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.306+0000 7f9551b8d640 1 -- start start 2024-09-16T01:39:06.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.307+0000 7f9551b8d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f954c158a00 0x7f954c071f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:06.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.307+0000 7f9551b8d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f954c0728f0 con 0x7f954c158de0 2024-09-16T01:39:06.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.307+0000 7f9551b8d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f954c072bd0 con 0x7f954c072470 2024-09-16T01:39:06.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.307+0000 7f9551b8d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f954c072eb0 con 0x7f954c158a00 2024-09-16T01:39:06.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.307+0000 7f954b7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f954c158a00 0x7f954c071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:06.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.307+0000 7f954b7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f954c158a00 0x7f954c071f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52932/0 (socket says 172.21.15.88:52932) 2024-09-16T01:39:06.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.307+0000 7f954bfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f954c158de0 0x7f954c06eae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57446/0 (socket says 172.21.15.88:57446) 2024-09-16T01:39:06.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.307+0000 7f954b7fe640 1 -- 172.21.15.88:0/3362618466 learned_addr learned my addr 172.21.15.88:0/3362618466 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:06.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.307+0000 7f954b7fe640 1 -- 172.21.15.88:0/3362618466 >> v1:172.21.15.154:6789/0 conn(0x7f954c072470 legacy=0x7f954c0b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:06.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.307+0000 7f954b7fe640 1 -- 172.21.15.88:0/3362618466 >> v1:172.21.15.88:6789/0 conn(0x7f954c158de0 legacy=0x7f954c06eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:06.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.307+0000 7f954b7fe640 1 -- 172.21.15.88:0/3362618466 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f954c073190 con 0x7f954c158a00 2024-09-16T01:39:06.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.308+0000 7f954b7fe640 1 --2- 172.21.15.88:0/3362618466 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f954c158a00 0x7f954c071f30 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f953c009870 tx=0x7f953c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a98a6b3a56e467dc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:06.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.308+0000 7f954a7fc640 1 -- 172.21.15.88:0/3362618466 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f953c037070 con 0x7f954c158a00 2024-09-16T01:39:06.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.308+0000 7f954a7fc640 1 -- 172.21.15.88:0/3362618466 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f953c02ae30 con 0x7f954c158a00 2024-09-16T01:39:06.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.308+0000 7f954a7fc640 1 -- 172.21.15.88:0/3362618466 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f953c032640 con 0x7f954c158a00 2024-09-16T01:39:06.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.308+0000 7f9551b8d640 1 -- 172.21.15.88:0/3362618466 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f954c158a00 msgr2=0x7f954c071f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:06.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.308+0000 7f9551b8d640 1 --2- 172.21.15.88:0/3362618466 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f954c158a00 0x7f954c071f30 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f953c009870 tx=0x7f953c02a1a0 comp rx=0 tx=0).stop 2024-09-16T01:39:06.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.308+0000 7f9551b8d640 1 -- 172.21.15.88:0/3362618466 shutdown_connections 2024-09-16T01:39:06.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.308+0000 7f9551b8d640 1 --2- 172.21.15.88:0/3362618466 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f954c158a00 0x7f954c071f30 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:06.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.308+0000 7f9551b8d640 1 -- 172.21.15.88:0/3362618466 >> 172.21.15.88:0/3362618466 conn(0x7f954c06cb50 msgr2=0x7f954c06cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:06.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.309+0000 7f9551b8d640 1 -- 172.21.15.88:0/3362618466 shutdown_connections 2024-09-16T01:39:06.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.309+0000 7f9551b8d640 1 -- 172.21.15.88:0/3362618466 wait complete. 2024-09-16T01:39:06.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.310+0000 7f9551b8d640 1 Processor -- start 2024-09-16T01:39:06.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.310+0000 7f9551b8d640 1 -- start start 2024-09-16T01:39:06.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.310+0000 7f9551b8d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f954c072470 0x7f954c1e9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:06.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.311+0000 7f9551b8d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f954c158a00 0x7f954c14dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:06.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.311+0000 7f9551b8d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f954c158de0 0x7f954c1610c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:06.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.311+0000 7f9551b8d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f954c0b5ca0 con 0x7f954c158a00 2024-09-16T01:39:06.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.311+0000 7f9551b8d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f954c0b5b20 con 0x7f954c158de0 2024-09-16T01:39:06.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.311+0000 7f9551b8d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f954c0b5570 con 0x7f954c072470 2024-09-16T01:39:06.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.311+0000 7f954affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f954c158a00 0x7f954c14dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:06.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.311+0000 7f954affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f954c158a00 0x7f954c14dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52934/0 (socket says 172.21.15.88:52934) 2024-09-16T01:39:06.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.311+0000 7f954affd640 1 -- 172.21.15.88:0/2478373119 learned_addr learned my addr 172.21.15.88:0/2478373119 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:06.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.311+0000 7f954bfff640 1 --2- 172.21.15.88:0/2478373119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f954c158de0 0x7f954c1610c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:06.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.311+0000 7f954b7fe640 1 --2- 172.21.15.88:0/2478373119 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f954c072470 0x7f954c1e9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:06.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.311+0000 7f954affd640 1 -- 172.21.15.88:0/2478373119 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f954c072470 msgr2=0x7f954c1e9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:06.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.311+0000 7f954affd640 1 --2- 172.21.15.88:0/2478373119 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f954c072470 0x7f954c1e9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:06.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.311+0000 7f954affd640 1 -- 172.21.15.88:0/2478373119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f954c158de0 msgr2=0x7f954c1610c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:06.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.311+0000 7f954affd640 1 --2- 172.21.15.88:0/2478373119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f954c158de0 0x7f954c1610c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:06.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.311+0000 7f954affd640 1 -- 172.21.15.88:0/2478373119 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f954c14e520 con 0x7f954c158a00 2024-09-16T01:39:06.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.311+0000 7f954affd640 1 --2- 172.21.15.88:0/2478373119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f954c158a00 0x7f954c14dfe0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f954000c400 tx=0x7f954000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:06.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.311+0000 7f9533fff640 1 -- 172.21.15.88:0/2478373119 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9540011790 con 0x7f954c158a00 2024-09-16T01:39:06.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.311+0000 7f9533fff640 1 -- 172.21.15.88:0/2478373119 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9540017070 con 0x7f954c158a00 2024-09-16T01:39:06.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.311+0000 7f9533fff640 1 -- 172.21.15.88:0/2478373119 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9540011bb0 con 0x7f954c158a00 2024-09-16T01:39:06.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.311+0000 7f9551b8d640 1 -- 172.21.15.88:0/2478373119 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f954c1ea310 con 0x7f954c158a00 2024-09-16T01:39:06.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.311+0000 7f9551b8d640 1 -- 172.21.15.88:0/2478373119 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f954c1ea8d0 con 0x7f954c158a00 2024-09-16T01:39:06.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.312+0000 7f9551b8d640 1 -- 172.21.15.88:0/2478373119 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f954c071f30 con 0x7f954c158a00 2024-09-16T01:39:06.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.313+0000 7f9533fff640 1 -- 172.21.15.88:0/2478373119 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f954001c460 con 0x7f954c158a00 2024-09-16T01:39:06.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.313+0000 7f9533fff640 1 --2- 172.21.15.88:0/2478373119 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f953806c2b0 0x7f953806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:06.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.313+0000 7f9533fff640 1 -- 172.21.15.88:0/2478373119 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5967+0+0 (secure 0 0 0) 0x7f954008b350 con 0x7f954c158a00 2024-09-16T01:39:06.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.313+0000 7f954b7fe640 1 --2- 172.21.15.88:0/2478373119 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f953806c2b0 0x7f953806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:06.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.314+0000 7f954b7fe640 1 --2- 172.21.15.88:0/2478373119 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f953806c2b0 0x7f953806e770 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f953c034000 tx=0x7f953c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:06.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.317+0000 7f9533fff640 1 -- 172.21.15.88:0/2478373119 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9540059a50 con 0x7f954c158a00 2024-09-16T01:39:06.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:06.487+0000 7f9551b8d640 1 -- 172.21.15.88:0/2478373119 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f954c158080 con 0x7f954c158a00 2024-09-16T01:39:07.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.416+0000 7f9533fff640 1 -- 172.21.15.88:0/2478373119 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v62) ==== 181+0+0 (secure 0 0 0) 0x7f9540059350 con 0x7f954c158a00 2024-09-16T01:39:07.417 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_metadata' removed 2024-09-16T01:39:07.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.420+0000 7f9551b8d640 1 -- 172.21.15.88:0/2478373119 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f953806c2b0 msgr2=0x7f953806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:07.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.420+0000 7f9551b8d640 1 --2- 172.21.15.88:0/2478373119 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f953806c2b0 0x7f953806e770 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f953c034000 tx=0x7f953c02e000 comp rx=0 tx=0).stop 2024-09-16T01:39:07.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.420+0000 7f9551b8d640 1 -- 172.21.15.88:0/2478373119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f954c158a00 msgr2=0x7f954c14dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:07.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.420+0000 7f9551b8d640 1 --2- 172.21.15.88:0/2478373119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f954c158a00 0x7f954c14dfe0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f954000c400 tx=0x7f954000c8c0 comp rx=0 tx=0).stop 2024-09-16T01:39:07.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.420+0000 7f9551b8d640 1 -- 172.21.15.88:0/2478373119 shutdown_connections 2024-09-16T01:39:07.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.420+0000 7f9551b8d640 1 --2- 172.21.15.88:0/2478373119 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f953806c2b0 0x7f953806e770 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:07.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.420+0000 7f9551b8d640 1 --2- 172.21.15.88:0/2478373119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f954c158de0 0x7f954c1610c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:07.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.420+0000 7f9551b8d640 1 --2- 172.21.15.88:0/2478373119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f954c158a00 0x7f954c14dfe0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:07.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.420+0000 7f9551b8d640 1 --2- 172.21.15.88:0/2478373119 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f954c072470 0x7f954c1e9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:07.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.420+0000 7f9551b8d640 1 -- 172.21.15.88:0/2478373119 >> 172.21.15.88:0/2478373119 conn(0x7f954c06cb50 msgr2=0x7f954c0b0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:07.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.420+0000 7f9551b8d640 1 -- 172.21.15.88:0/2478373119 shutdown_connections 2024-09-16T01:39:07.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.420+0000 7f9551b8d640 1 -- 172.21.15.88:0/2478373119 wait complete. 2024-09-16T01:39:07.435 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-16T01:39:07.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.544+0000 7f4706d6d640 1 Processor -- start 2024-09-16T01:39:07.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.544+0000 7f4706d6d640 1 -- start start 2024-09-16T01:39:07.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.544+0000 7f4706d6d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f470015e990 0x7f47000b3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:07.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.544+0000 7f4706d6d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4700162fa0 0x7f47000d29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:07.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.544+0000 7f4706d6d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f470015ed70 con 0x7f4700162450 2024-09-16T01:39:07.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.544+0000 7f4706d6d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f47000cff20 con 0x7f4700162fa0 2024-09-16T01:39:07.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.544+0000 7f4706d6d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f47000d1fa0 con 0x7f470015e990 2024-09-16T01:39:07.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.545+0000 7f46fffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4700162450 0x7f4700162890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57454/0 (socket says 172.21.15.88:57454) 2024-09-16T01:39:07.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.545+0000 7f46fffff640 1 -- 172.21.15.88:0/1116885718 learned_addr learned my addr 172.21.15.88:0/1116885718 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:07.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.545+0000 7f46ff7fe640 1 -- 172.21.15.88:0/1116885718 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3896384045 0 0) 0x7f46ec001110 con 0x7f4700162450 2024-09-16T01:39:07.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.545+0000 7f470530a640 1 --2- 172.21.15.88:0/1116885718 >> v2:172.21.15.154:3300/0 conn(0x7f4700162fa0 0x7f47000d29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:07.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.545+0000 7f4704b09640 1 --2- 172.21.15.88:0/1116885718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f470015e990 0x7f47000b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:07.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.545+0000 7f46ff7fe640 1 -- 172.21.15.88:0/1116885718 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46f0003520 con 0x7f4700162450 2024-09-16T01:39:07.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.545+0000 7f46ff7fe640 1 -- 172.21.15.88:0/1116885718 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 900783875 0 0) 0x7f46f0003520 con 0x7f4700162450 2024-09-16T01:39:07.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.545+0000 7f46ff7fe640 1 -- 172.21.15.88:0/1116885718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f470015e990 msgr2=0x7f47000b3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:07.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.545+0000 7f46ff7fe640 1 --2- 172.21.15.88:0/1116885718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f470015e990 0x7f47000b3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:07.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.545+0000 7f46ff7fe640 1 -- 172.21.15.88:0/1116885718 >> v2:172.21.15.154:3300/0 conn(0x7f4700162fa0 msgr2=0x7f47000d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:07.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.545+0000 7f46ff7fe640 1 --2- 172.21.15.88:0/1116885718 >> v2:172.21.15.154:3300/0 conn(0x7f4700162fa0 0x7f47000d29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:07.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.545+0000 7f46ff7fe640 1 -- 172.21.15.88:0/1116885718 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47000cf960 con 0x7f4700162450 2024-09-16T01:39:07.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.545+0000 7f46ff7fe640 1 -- 172.21.15.88:0/1116885718 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f46ec001840 con 0x7f4700162450 2024-09-16T01:39:07.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.545+0000 7f46ff7fe640 1 -- 172.21.15.88:0/1116885718 >> v1:172.21.15.88:6789/0 conn(0x7f4700162450 legacy=0x7f4700162890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:07.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.545+0000 7f46ff7fe640 1 --2- 172.21.15.88:0/1116885718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f470015e990 0x7f46f00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:07.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.545+0000 7f46ff7fe640 1 --2- 172.21.15.88:0/1116885718 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46f0003f80 0x7f46f0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:07.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.545+0000 7f46ff7fe640 1 --2- 172.21.15.88:0/1116885718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f0004d40 0x7f46f00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:07.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.545+0000 7f46ff7fe640 1 -- 172.21.15.88:0/1116885718 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f47000cff20 con 0x7f46f0003f80 2024-09-16T01:39:07.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.545+0000 7f46ff7fe640 1 -- 172.21.15.88:0/1116885718 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f47000d1fa0 con 0x7f46f0004d40 2024-09-16T01:39:07.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.545+0000 7f46ff7fe640 1 -- 172.21.15.88:0/1116885718 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f46f000ba70 con 0x7f470015e990 2024-09-16T01:39:07.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.545+0000 7f46fffff640 1 --2- 172.21.15.88:0/1116885718 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46f0003f80 0x7f46f0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:07.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.545+0000 7f4704b09640 1 --2- 172.21.15.88:0/1116885718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f470015e990 0x7f46f00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:07.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.545+0000 7f470530a640 1 --2- 172.21.15.88:0/1116885718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f0004d40 0x7f46f00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:07.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.546+0000 7f46fffff640 1 -- 172.21.15.88:0/1116885718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f470015e990 msgr2=0x7f46f00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:07.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.546+0000 7f46fffff640 1 --2- 172.21.15.88:0/1116885718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f470015e990 0x7f46f00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:07.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.546+0000 7f46fffff640 1 -- 172.21.15.88:0/1116885718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f0004d40 msgr2=0x7f46f00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:07.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.546+0000 7f46fffff640 1 --2- 172.21.15.88:0/1116885718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46f0004d40 0x7f46f00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:07.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.546+0000 7f46fffff640 1 -- 172.21.15.88:0/1116885718 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f0003520 con 0x7f46f0003f80 2024-09-16T01:39:07.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.546+0000 7f46fffff640 1 --2- 172.21.15.88:0/1116885718 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46f0003f80 0x7f46f0004800 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f4700074790 tx=0x7f46f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a4afd647958092c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:07.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.546+0000 7f46ff7fe640 1 -- 172.21.15.88:0/1116885718 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec036070 con 0x7f46f0003f80 2024-09-16T01:39:07.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.547+0000 7f46ff7fe640 1 -- 172.21.15.88:0/1116885718 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46ec005ad0 con 0x7f46f0003f80 2024-09-16T01:39:07.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.547+0000 7f46ff7fe640 1 -- 172.21.15.88:0/1116885718 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec007e60 con 0x7f46f0003f80 2024-09-16T01:39:07.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.547+0000 7f4706d6d640 1 -- 172.21.15.88:0/1116885718 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46f0003f80 msgr2=0x7f46f0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:07.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.547+0000 7f4706d6d640 1 --2- 172.21.15.88:0/1116885718 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46f0003f80 0x7f46f0004800 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f4700074790 tx=0x7f46f0005830 comp rx=0 tx=0).stop 2024-09-16T01:39:07.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.547+0000 7f470530a640 1 -- 172.21.15.88:0/1116885718 reap_dead start 2024-09-16T01:39:07.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.547+0000 7f4706d6d640 1 -- 172.21.15.88:0/1116885718 shutdown_connections 2024-09-16T01:39:07.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.547+0000 7f4706d6d640 1 -- 172.21.15.88:0/1116885718 >> 172.21.15.88:0/1116885718 conn(0x7f47000d1200 msgr2=0x7f47000d1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:07.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.547+0000 7f4706d6d640 1 -- 172.21.15.88:0/1116885718 shutdown_connections 2024-09-16T01:39:07.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.547+0000 7f4706d6d640 1 -- 172.21.15.88:0/1116885718 wait complete. 2024-09-16T01:39:07.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.548+0000 7f4706d6d640 1 Processor -- start 2024-09-16T01:39:07.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.548+0000 7f4706d6d640 1 -- start start 2024-09-16T01:39:07.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.549+0000 7f4706d6d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f46f0003f80 0x7f47000ce6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:07.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.549+0000 7f4706d6d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47001f3bf0 0x7f47000ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:07.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.549+0000 7f4706d6d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47000cab10 0x7f47001f7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:07.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.549+0000 7f4706d6d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f47000b3aa0 con 0x7f47001f3bf0 2024-09-16T01:39:07.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.549+0000 7f4706d6d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f47000b3920 con 0x7f47000cab10 2024-09-16T01:39:07.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.549+0000 7f4706d6d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f47000b3c20 con 0x7f46f0003f80 2024-09-16T01:39:07.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.549+0000 7f46fffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47001f3bf0 0x7f47000ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:07.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.549+0000 7f46fffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47001f3bf0 0x7f47000ca5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52950/0 (socket says 172.21.15.88:52950) 2024-09-16T01:39:07.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.549+0000 7f46fffff640 1 -- 172.21.15.88:0/485216439 learned_addr learned my addr 172.21.15.88:0/485216439 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:07.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.549+0000 7f4704b09640 1 --2- 172.21.15.88:0/485216439 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f46f0003f80 0x7f47000ce6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:07.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.549+0000 7f470530a640 1 --2- 172.21.15.88:0/485216439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47000cab10 0x7f47001f7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:07.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.550+0000 7f46fffff640 1 -- 172.21.15.88:0/485216439 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f46f0003f80 msgr2=0x7f47000ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:07.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.550+0000 7f46fffff640 1 --2- 172.21.15.88:0/485216439 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f46f0003f80 0x7f47000ce6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:07.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.550+0000 7f46fffff640 1 -- 172.21.15.88:0/485216439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47000cab10 msgr2=0x7f47001f7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:07.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.550+0000 7f46fffff640 1 --2- 172.21.15.88:0/485216439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47000cab10 0x7f47001f7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:07.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.550+0000 7f46fffff640 1 -- 172.21.15.88:0/485216439 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47000cc600 con 0x7f47001f3bf0 2024-09-16T01:39:07.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.550+0000 7f4704b09640 1 --2- 172.21.15.88:0/485216439 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f46f0003f80 0x7f47000ce6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:07.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.550+0000 7f470530a640 1 --2- 172.21.15.88:0/485216439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47000cab10 0x7f47001f7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:07.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.550+0000 7f46fffff640 1 --2- 172.21.15.88:0/485216439 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47001f3bf0 0x7f47000ca5d0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f46ec02af00 tx=0x7f47000b7760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:07.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.550+0000 7f46fd7fa640 1 -- 172.21.15.88:0/485216439 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec041020 con 0x7f47001f3bf0 2024-09-16T01:39:07.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.550+0000 7f46fd7fa640 1 -- 172.21.15.88:0/485216439 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46ec0018d0 con 0x7f47001f3bf0 2024-09-16T01:39:07.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.550+0000 7f4706d6d640 1 -- 172.21.15.88:0/485216439 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47000cc8e0 con 0x7f47001f3bf0 2024-09-16T01:39:07.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.550+0000 7f46fd7fa640 1 -- 172.21.15.88:0/485216439 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec036030 con 0x7f47001f3bf0 2024-09-16T01:39:07.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.550+0000 7f4706d6d640 1 -- 172.21.15.88:0/485216439 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47000ccea0 con 0x7f47001f3bf0 2024-09-16T01:39:07.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.551+0000 7f4706d6d640 1 -- 172.21.15.88:0/485216439 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47000b3e90 con 0x7f47001f3bf0 2024-09-16T01:39:07.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.551+0000 7f46fd7fa640 1 -- 172.21.15.88:0/485216439 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f46ec044020 con 0x7f47001f3bf0 2024-09-16T01:39:07.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.552+0000 7f46fd7fa640 1 --2- 172.21.15.88:0/485216439 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f46e406c2e0 0x7f46e406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:07.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.552+0000 7f4704b09640 1 --2- 172.21.15.88:0/485216439 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f46e406c2e0 0x7f46e406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:07.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.552+0000 7f46fd7fa640 1 -- 172.21.15.88:0/485216439 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5529+0+0 (secure 0 0 0) 0x7f46ec0ab3f0 con 0x7f47001f3bf0 2024-09-16T01:39:07.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.552+0000 7f4704b09640 1 --2- 172.21.15.88:0/485216439 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f46e406c2e0 0x7f46e406e7a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f46f4009a60 tx=0x7f46f4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:07.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.556+0000 7f46fd7fa640 1 -- 172.21.15.88:0/485216439 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f470015ed70 con 0x7f47001f3bf0 2024-09-16T01:39:07.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:07.722+0000 7f4706d6d640 1 -- 172.21.15.88:0/485216439 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f47000caef0 con 0x7f47001f3bf0 2024-09-16T01:39:08.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.422+0000 7f46fd7fa640 1 -- 172.21.15.88:0/485216439 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v63) ==== 169+0+0 (secure 0 0 0) 0x7f46ec079c10 con 0x7f47001f3bf0 2024-09-16T01:39:08.424 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_data' removed 2024-09-16T01:39:08.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.425+0000 7f4706d6d640 1 -- 172.21.15.88:0/485216439 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f46e406c2e0 msgr2=0x7f46e406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:08.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.425+0000 7f4706d6d640 1 --2- 172.21.15.88:0/485216439 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f46e406c2e0 0x7f46e406e7a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f46f4009a60 tx=0x7f46f4008040 comp rx=0 tx=0).stop 2024-09-16T01:39:08.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.425+0000 7f4706d6d640 1 -- 172.21.15.88:0/485216439 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47001f3bf0 msgr2=0x7f47000ca5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:08.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.425+0000 7f4706d6d640 1 --2- 172.21.15.88:0/485216439 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47001f3bf0 0x7f47000ca5d0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f46ec02af00 tx=0x7f47000b7760 comp rx=0 tx=0).stop 2024-09-16T01:39:08.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.425+0000 7f4706d6d640 1 -- 172.21.15.88:0/485216439 shutdown_connections 2024-09-16T01:39:08.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.425+0000 7f4706d6d640 1 --2- 172.21.15.88:0/485216439 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f46e406c2e0 0x7f46e406e7a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:08.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.425+0000 7f4706d6d640 1 --2- 172.21.15.88:0/485216439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47000cab10 0x7f47001f7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:08.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.425+0000 7f4706d6d640 1 --2- 172.21.15.88:0/485216439 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47001f3bf0 0x7f47000ca5d0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:08.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.425+0000 7f4706d6d640 1 --2- 172.21.15.88:0/485216439 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f46f0003f80 0x7f47000ce6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:08.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.425+0000 7f4706d6d640 1 -- 172.21.15.88:0/485216439 >> 172.21.15.88:0/485216439 conn(0x7f47000d1200 msgr2=0x7f47000b7900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:08.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.425+0000 7f4706d6d640 1 -- 172.21.15.88:0/485216439 shutdown_connections 2024-09-16T01:39:08.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.425+0000 7f4706d6d640 1 -- 172.21.15.88:0/485216439 wait complete. 2024-09-16T01:39:08.440 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-16T01:39:08.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.545+0000 7fce572c9640 1 Processor -- start 2024-09-16T01:39:08.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.545+0000 7fce572c9640 1 -- start start 2024-09-16T01:39:08.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.546+0000 7fce572c9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fce500b7530 0x7fce50162ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:08.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.546+0000 7fce572c9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce500b57d0 con 0x7fce500d1bf0 2024-09-16T01:39:08.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.546+0000 7fce572c9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce5006e8c0 con 0x7fce5015e790 2024-09-16T01:39:08.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.546+0000 7fce572c9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fce5006d430 con 0x7fce500b7530 2024-09-16T01:39:08.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.546+0000 7fce54864640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fce500d1bf0 0x7fce500b6e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57458/0 (socket says 172.21.15.88:57458) 2024-09-16T01:39:08.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.546+0000 7fce55866640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fce500b7530 0x7fce50162ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:08.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.546+0000 7fce54864640 1 -- 172.21.15.88:0/3500034052 learned_addr learned my addr 172.21.15.88:0/3500034052 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:08.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.547+0000 7fce55866640 1 -- 172.21.15.88:0/3500034052 >> v1:172.21.15.154:6789/0 conn(0x7fce5015e790 legacy=0x7fce5006cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:08.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.547+0000 7fce55866640 1 -- 172.21.15.88:0/3500034052 >> v1:172.21.15.88:6789/0 conn(0x7fce500d1bf0 legacy=0x7fce500b6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:08.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.547+0000 7fce55866640 1 -- 172.21.15.88:0/3500034052 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce5006eba0 con 0x7fce500b7530 2024-09-16T01:39:08.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.547+0000 7fce55866640 1 --2- 172.21.15.88:0/3500034052 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce500b7530 0x7fce50162ca0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fce48009d90 tx=0x7fce4802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d37e6ce9544a616d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:08.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.547+0000 7fce47fff640 1 -- 172.21.15.88:0/3500034052 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce48007740 con 0x7fce500b7530 2024-09-16T01:39:08.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.548+0000 7fce47fff640 1 -- 172.21.15.88:0/3500034052 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce480078e0 con 0x7fce500b7530 2024-09-16T01:39:08.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.548+0000 7fce47fff640 1 -- 172.21.15.88:0/3500034052 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce48007cb0 con 0x7fce500b7530 2024-09-16T01:39:08.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.548+0000 7fce572c9640 1 -- 172.21.15.88:0/3500034052 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce500b7530 msgr2=0x7fce50162ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:08.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.548+0000 7fce572c9640 1 --2- 172.21.15.88:0/3500034052 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce500b7530 0x7fce50162ca0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fce48009d90 tx=0x7fce4802a730 comp rx=0 tx=0).stop 2024-09-16T01:39:08.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.548+0000 7fce572c9640 1 -- 172.21.15.88:0/3500034052 shutdown_connections 2024-09-16T01:39:08.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.548+0000 7fce572c9640 1 --2- 172.21.15.88:0/3500034052 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce500b7530 0x7fce50162ca0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:08.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.548+0000 7fce572c9640 1 -- 172.21.15.88:0/3500034052 >> 172.21.15.88:0/3500034052 conn(0x7fce500d24d0 msgr2=0x7fce500d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:08.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.549+0000 7fce572c9640 1 -- 172.21.15.88:0/3500034052 shutdown_connections 2024-09-16T01:39:08.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.549+0000 7fce572c9640 1 -- 172.21.15.88:0/3500034052 wait complete. 2024-09-16T01:39:08.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.550+0000 7fce572c9640 1 Processor -- start 2024-09-16T01:39:08.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.551+0000 7fce572c9640 1 -- start start 2024-09-16T01:39:08.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.552+0000 7fce572c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce500b7530 0x7fce50160d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:08.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.552+0000 7fce572c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce500d1bf0 0x7fce50208be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:08.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.552+0000 7fce572c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce5015e790 0x7fce50209120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:08.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.552+0000 7fce572c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fce50073c70 con 0x7fce500b7530 2024-09-16T01:39:08.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.552+0000 7fce572c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fce50073af0 con 0x7fce5015e790 2024-09-16T01:39:08.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.552+0000 7fce572c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fce50073df0 con 0x7fce500d1bf0 2024-09-16T01:39:08.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.552+0000 7fce55065640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce500b7530 0x7fce50160d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:08.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.553+0000 7fce55065640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce500b7530 0x7fce50160d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52968/0 (socket says 172.21.15.88:52968) 2024-09-16T01:39:08.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.553+0000 7fce55065640 1 -- 172.21.15.88:0/474524844 learned_addr learned my addr 172.21.15.88:0/474524844 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:08.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.554+0000 7fce55065640 1 -- 172.21.15.88:0/474524844 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce500d1bf0 msgr2=0x7fce50208be0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:39:08.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.554+0000 7fce55065640 1 --2- 172.21.15.88:0/474524844 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce500d1bf0 0x7fce50208be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:08.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.554+0000 7fce55065640 1 -- 172.21.15.88:0/474524844 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce5015e790 msgr2=0x7fce50209120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T01:39:08.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.554+0000 7fce55065640 1 --2- 172.21.15.88:0/474524844 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce5015e790 0x7fce50209120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:08.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.554+0000 7fce55065640 1 -- 172.21.15.88:0/474524844 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce50161800 con 0x7fce500b7530 2024-09-16T01:39:08.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.554+0000 7fce55065640 1 --2- 172.21.15.88:0/474524844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce500b7530 0x7fce50160d40 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fce4c00b3e0 tx=0x7fce4c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:08.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.557+0000 7fce45ffb640 1 -- 172.21.15.88:0/474524844 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4c015020 con 0x7fce500b7530 2024-09-16T01:39:08.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.557+0000 7fce45ffb640 1 -- 172.21.15.88:0/474524844 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce4c011850 con 0x7fce500b7530 2024-09-16T01:39:08.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.557+0000 7fce45ffb640 1 -- 172.21.15.88:0/474524844 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4c011c40 con 0x7fce500b7530 2024-09-16T01:39:08.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.557+0000 7fce572c9640 1 -- 172.21.15.88:0/474524844 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce50161ae0 con 0x7fce500b7530 2024-09-16T01:39:08.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.557+0000 7fce572c9640 1 -- 172.21.15.88:0/474524844 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce501620a0 con 0x7fce500b7530 2024-09-16T01:39:08.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.558+0000 7fce572c9640 1 -- 172.21.15.88:0/474524844 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce500741b0 con 0x7fce500b7530 2024-09-16T01:39:08.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.559+0000 7fce45ffb640 1 -- 172.21.15.88:0/474524844 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fce4c018020 con 0x7fce500b7530 2024-09-16T01:39:08.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.560+0000 7fce45ffb640 1 --2- 172.21.15.88:0/474524844 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fce3406c300 0x7fce3406e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:08.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.560+0000 7fce45ffb640 1 -- 172.21.15.88:0/474524844 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5115+0+0 (secure 0 0 0) 0x7fce4c08b170 con 0x7fce500b7530 2024-09-16T01:39:08.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.560+0000 7fce54864640 1 --2- 172.21.15.88:0/474524844 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fce3406c300 0x7fce3406e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:08.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.561+0000 7fce54864640 1 --2- 172.21.15.88:0/474524844 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fce3406c300 0x7fce3406e7c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fce40000f60 tx=0x7fce40007a00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:08.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.566+0000 7fce45ffb640 1 -- 172.21.15.88:0/474524844 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fce4c01f020 con 0x7fce500b7530 2024-09-16T01:39:08.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.733+0000 7fce572c9640 1 -- 172.21.15.88:0/474524844 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fce5006fa60 con 0x7fce500b7530 2024-09-16T01:39:08.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.736+0000 7fce45ffb640 1 -- 172.21.15.88:0/474524844 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) ==== 104+0+0 (secure 0 0 0) 0x7fce4c0172c0 con 0x7fce500b7530 2024-09-16T01:39:08.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.738+0000 7fce572c9640 1 -- 172.21.15.88:0/474524844 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fce3406c300 msgr2=0x7fce3406e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:08.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.738+0000 7fce572c9640 1 --2- 172.21.15.88:0/474524844 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fce3406c300 0x7fce3406e7c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fce40000f60 tx=0x7fce40007a00 comp rx=0 tx=0).stop 2024-09-16T01:39:08.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.738+0000 7fce572c9640 1 -- 172.21.15.88:0/474524844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce500b7530 msgr2=0x7fce50160d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:08.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.738+0000 7fce572c9640 1 --2- 172.21.15.88:0/474524844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce500b7530 0x7fce50160d40 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fce4c00b3e0 tx=0x7fce4c00b8a0 comp rx=0 tx=0).stop 2024-09-16T01:39:08.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.738+0000 7fce572c9640 1 -- 172.21.15.88:0/474524844 shutdown_connections 2024-09-16T01:39:08.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.738+0000 7fce572c9640 1 --2- 172.21.15.88:0/474524844 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fce3406c300 0x7fce3406e7c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:08.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.738+0000 7fce572c9640 1 --2- 172.21.15.88:0/474524844 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce5015e790 0x7fce50209120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:08.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.738+0000 7fce572c9640 1 --2- 172.21.15.88:0/474524844 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce500d1bf0 0x7fce50208be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:08.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.738+0000 7fce572c9640 1 --2- 172.21.15.88:0/474524844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce500b7530 0x7fce50160d40 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:08.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.738+0000 7fce572c9640 1 -- 172.21.15.88:0/474524844 >> 172.21.15.88:0/474524844 conn(0x7fce500d24d0 msgr2=0x7fce500b6c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:08.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.738+0000 7fce572c9640 1 -- 172.21.15.88:0/474524844 shutdown_connections 2024-09-16T01:39:08.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.738+0000 7fce572c9640 1 -- 172.21.15.88:0/474524844 wait complete. 2024-09-16T01:39:08.751 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_inject_journal_corrupt_dentry_first 2024-09-16T01:39:08.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.849+0000 7f3effeba640 1 Processor -- start 2024-09-16T01:39:08.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.849+0000 7f3effeba640 1 -- start start 2024-09-16T01:39:08.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.850+0000 7f3effeba640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3ef815e6f0 0x7f3ef806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:08.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.850+0000 7f3effeba640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3ef80b54d0 0x7f3ef815f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:08.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.850+0000 7f3effeba640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ef806e990 con 0x7f3ef80b6d40 2024-09-16T01:39:08.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.850+0000 7f3effeba640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f3ef806ec70 con 0x7f3ef80b54d0 2024-09-16T01:39:08.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.850+0000 7f3effeba640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3ef815ec70 con 0x7f3ef815e6f0 2024-09-16T01:39:08.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.850+0000 7f3efd455640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3ef80b54d0 0x7f3ef815f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:08.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.850+0000 7f3efd455640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3ef80b54d0 0x7f3ef815f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52974/0 (socket says 172.21.15.88:52974) 2024-09-16T01:39:08.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.850+0000 7f3efd455640 1 -- 172.21.15.88:0/2844283856 learned_addr learned my addr 172.21.15.88:0/2844283856 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:08.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.851+0000 7f3efdc56640 1 --2- 172.21.15.88:0/2844283856 >> v2:172.21.15.154:3300/0 conn(0x7f3ef815e6f0 0x7f3ef806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:08.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.851+0000 7f3efd455640 1 -- 172.21.15.88:0/2844283856 >> v2:172.21.15.154:3300/0 conn(0x7f3ef815e6f0 msgr2=0x7f3ef806fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:08.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.851+0000 7f3efd455640 1 --2- 172.21.15.88:0/2844283856 >> v2:172.21.15.154:3300/0 conn(0x7f3ef815e6f0 0x7f3ef806fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:08.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.851+0000 7f3efd455640 1 -- 172.21.15.88:0/2844283856 >> v1:172.21.15.88:6789/0 conn(0x7f3ef80b6d40 legacy=0x7f3ef80b71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:08.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.851+0000 7f3efcc54640 1 -- 172.21.15.88:0/2844283856 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 915840613 0 0) 0x7f3ef806e990 con 0x7f3ef80b6d40 2024-09-16T01:39:08.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.851+0000 7f3efd455640 1 -- 172.21.15.88:0/2844283856 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ef815ef50 con 0x7f3ef80b54d0 2024-09-16T01:39:08.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.851+0000 7f3efd455640 1 --2- 172.21.15.88:0/2844283856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ef80b54d0 0x7f3ef815f750 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8069920 tx=0x7f3ee802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36338ae20af88639 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:08.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.852+0000 7f3efcc54640 1 -- 172.21.15.88:0/2844283856 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee8037070 con 0x7f3ef80b54d0 2024-09-16T01:39:08.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.852+0000 7f3efcc54640 1 -- 172.21.15.88:0/2844283856 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ee8032440 con 0x7f3ef80b54d0 2024-09-16T01:39:08.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.852+0000 7f3efcc54640 1 -- 172.21.15.88:0/2844283856 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee8032810 con 0x7f3ef80b54d0 2024-09-16T01:39:08.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.852+0000 7f3effeba640 1 -- 172.21.15.88:0/2844283856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ef80b54d0 msgr2=0x7f3ef815f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:08.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.852+0000 7f3effeba640 1 --2- 172.21.15.88:0/2844283856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ef80b54d0 0x7f3ef815f750 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8069920 tx=0x7f3ee802a270 comp rx=0 tx=0).stop 2024-09-16T01:39:08.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.852+0000 7f3effeba640 1 -- 172.21.15.88:0/2844283856 shutdown_connections 2024-09-16T01:39:08.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.852+0000 7f3effeba640 1 --2- 172.21.15.88:0/2844283856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ef80b54d0 0x7f3ef815f750 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:08.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.852+0000 7f3effeba640 1 --2- 172.21.15.88:0/2844283856 >> v2:172.21.15.154:3300/0 conn(0x7f3ef815e6f0 0x7f3ef806fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:08.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.852+0000 7f3effeba640 1 -- 172.21.15.88:0/2844283856 >> 172.21.15.88:0/2844283856 conn(0x7f3ef80d2430 msgr2=0x7f3ef80d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:08.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.852+0000 7f3effeba640 1 -- 172.21.15.88:0/2844283856 shutdown_connections 2024-09-16T01:39:08.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.852+0000 7f3effeba640 1 -- 172.21.15.88:0/2844283856 wait complete. 2024-09-16T01:39:08.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.853+0000 7f3effeba640 1 Processor -- start 2024-09-16T01:39:08.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.853+0000 7f3effeba640 1 -- start start 2024-09-16T01:39:08.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.854+0000 7f3effeba640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ef80b54d0 0x7f3ef8161820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:08.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.854+0000 7f3effeba640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ef80b6d40 0x7f3ef815f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:08.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.854+0000 7f3effeba640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ef815e6f0 0x7f3ef815fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:08.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.854+0000 7f3effeba640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef806e730 con 0x7f3ef815e6f0 2024-09-16T01:39:08.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.854+0000 7f3effeba640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef806e5b0 con 0x7f3ef80b54d0 2024-09-16T01:39:08.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.854+0000 7f3effeba640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3ef8070830 con 0x7f3ef80b6d40 2024-09-16T01:39:08.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.854+0000 7f3efe457640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ef815e6f0 0x7f3ef815fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:08.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.854+0000 7f3efe457640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ef815e6f0 0x7f3ef815fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:52986/0 (socket says 172.21.15.88:52986) 2024-09-16T01:39:08.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.854+0000 7f3efe457640 1 -- 172.21.15.88:0/2920035801 learned_addr learned my addr 172.21.15.88:0/2920035801 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:08.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.854+0000 7f3efd455640 1 --2- 172.21.15.88:0/2920035801 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ef80b6d40 0x7f3ef815f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:08.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.854+0000 7f3efdc56640 1 --2- 172.21.15.88:0/2920035801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ef80b54d0 0x7f3ef8161820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:08.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.854+0000 7f3efe457640 1 -- 172.21.15.88:0/2920035801 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ef80b6d40 msgr2=0x7f3ef815f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:08.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.855+0000 7f3efe457640 1 --2- 172.21.15.88:0/2920035801 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ef80b6d40 0x7f3ef815f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:08.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.855+0000 7f3efe457640 1 -- 172.21.15.88:0/2920035801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ef80b54d0 msgr2=0x7f3ef8161820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:08.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.855+0000 7f3efe457640 1 --2- 172.21.15.88:0/2920035801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ef80b54d0 0x7f3ef8161820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:08.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.855+0000 7f3efe457640 1 -- 172.21.15.88:0/2920035801 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ef8154510 con 0x7f3ef815e6f0 2024-09-16T01:39:08.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.855+0000 7f3efdc56640 1 --2- 172.21.15.88:0/2920035801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ef80b54d0 0x7f3ef8161820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:08.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.855+0000 7f3efe457640 1 --2- 172.21.15.88:0/2920035801 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ef815e6f0 0x7f3ef815fea0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f3ef000c760 tx=0x7f3ef000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:08.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.855+0000 7f3eee7fc640 1 -- 172.21.15.88:0/2920035801 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef0003da0 con 0x7f3ef815e6f0 2024-09-16T01:39:08.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.855+0000 7f3eee7fc640 1 -- 172.21.15.88:0/2920035801 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ef0005070 con 0x7f3ef815e6f0 2024-09-16T01:39:08.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.855+0000 7f3effeba640 1 -- 172.21.15.88:0/2920035801 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ef8162460 con 0x7f3ef815e6f0 2024-09-16T01:39:08.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.855+0000 7f3eee7fc640 1 -- 172.21.15.88:0/2920035801 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef0005460 con 0x7f3ef815e6f0 2024-09-16T01:39:08.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.855+0000 7f3effeba640 1 -- 172.21.15.88:0/2920035801 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ef8158bc0 con 0x7f3ef815e6f0 2024-09-16T01:39:08.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.856+0000 7f3effeba640 1 -- 172.21.15.88:0/2920035801 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ef815f750 con 0x7f3ef815e6f0 2024-09-16T01:39:08.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.856+0000 7f3eee7fc640 1 -- 172.21.15.88:0/2920035801 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3ef0005600 con 0x7f3ef815e6f0 2024-09-16T01:39:08.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.857+0000 7f3eee7fc640 1 --2- 172.21.15.88:0/2920035801 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3edc06c2b0 0x7f3edc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:08.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.857+0000 7f3eee7fc640 1 -- 172.21.15.88:0/2920035801 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5115+0+0 (secure 0 0 0) 0x7f3ef008afc0 con 0x7f3ef815e6f0 2024-09-16T01:39:08.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.857+0000 7f3efdc56640 1 --2- 172.21.15.88:0/2920035801 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3edc06c2b0 0x7f3edc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:08.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.857+0000 7f3efdc56640 1 --2- 172.21.15.88:0/2920035801 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3edc06c2b0 0x7f3edc06e770 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4006fd0 tx=0x7f3ef4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:08.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:08.861+0000 7f3eee7fc640 1 -- 172.21.15.88:0/2920035801 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3ef0059a10 con 0x7f3ef815e6f0 2024-09-16T01:39:09.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.025+0000 7f3effeba640 1 -- 172.21.15.88:0/2920035801 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_inject_journal_corrupt_dentry_first"} v 0) -- 0x7f3ef80b1b80 con 0x7f3ef815e6f0 2024-09-16T01:39:09.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.026+0000 7f3eee7fc640 1 -- 172.21.15.88:0/2920035801 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_inject_journal_corrupt_dentry_first"}]=0 v21) ==== 122+0+0 (secure 0 0 0) 0x7f3ef0059310 con 0x7f3ef815e6f0 2024-09-16T01:39:09.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.028+0000 7f3effeba640 1 -- 172.21.15.88:0/2920035801 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3edc06c2b0 msgr2=0x7f3edc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:09.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.028+0000 7f3effeba640 1 --2- 172.21.15.88:0/2920035801 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3edc06c2b0 0x7f3edc06e770 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4006fd0 tx=0x7f3ef4008040 comp rx=0 tx=0).stop 2024-09-16T01:39:09.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.028+0000 7f3effeba640 1 -- 172.21.15.88:0/2920035801 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ef815e6f0 msgr2=0x7f3ef815fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:09.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.028+0000 7f3effeba640 1 --2- 172.21.15.88:0/2920035801 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ef815e6f0 0x7f3ef815fea0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f3ef000c760 tx=0x7f3ef000cc20 comp rx=0 tx=0).stop 2024-09-16T01:39:09.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.028+0000 7f3effeba640 1 -- 172.21.15.88:0/2920035801 shutdown_connections 2024-09-16T01:39:09.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.028+0000 7f3effeba640 1 --2- 172.21.15.88:0/2920035801 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3edc06c2b0 0x7f3edc06e770 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.028+0000 7f3effeba640 1 --2- 172.21.15.88:0/2920035801 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ef815e6f0 0x7f3ef815fea0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.028+0000 7f3effeba640 1 --2- 172.21.15.88:0/2920035801 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ef80b6d40 0x7f3ef815f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.028+0000 7f3effeba640 1 --2- 172.21.15.88:0/2920035801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ef80b54d0 0x7f3ef8161820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.028+0000 7f3effeba640 1 -- 172.21.15.88:0/2920035801 >> 172.21.15.88:0/2920035801 conn(0x7f3ef80d2430 msgr2=0x7f3ef81574c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:09.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.028+0000 7f3effeba640 1 -- 172.21.15.88:0/2920035801 shutdown_connections 2024-09-16T01:39:09.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.028+0000 7f3effeba640 1 -- 172.21.15.88:0/2920035801 wait complete. 2024-09-16T01:39:09.040 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_precommit' 2024-09-16T01:39:09.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.137+0000 7fc30b749640 1 Processor -- start 2024-09-16T01:39:09.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.138+0000 7fc30b749640 1 -- start start 2024-09-16T01:39:09.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.138+0000 7fc30b749640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc30415e660 0x7fc3040cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:09.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.138+0000 7fc30b749640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc3040d1820 0x7fc3040b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:09.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.138+0000 7fc30b749640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc30415ea40 con 0x7fc3040d3310 2024-09-16T01:39:09.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.138+0000 7fc30b749640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc3040d3870 con 0x7fc30415e660 2024-09-16T01:39:09.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.138+0000 7fc30b749640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc3040d3b50 con 0x7fc3040d1820 2024-09-16T01:39:09.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.138+0000 7fc3094e5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc30415e660 0x7fc3040cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:09.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.139+0000 7fc3094e5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc30415e660 0x7fc3040cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53000/0 (socket says 172.21.15.88:53000) 2024-09-16T01:39:09.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.139+0000 7fc3094e5640 1 -- 172.21.15.88:0/558578266 learned_addr learned my addr 172.21.15.88:0/558578266 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:09.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.139+0000 7fc309ce6640 1 --2- 172.21.15.88:0/558578266 >> v2:172.21.15.154:3300/0 conn(0x7fc3040d1820 0x7fc3040b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:09.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.139+0000 7fc3094e5640 1 -- 172.21.15.88:0/558578266 >> v2:172.21.15.154:3300/0 conn(0x7fc3040d1820 msgr2=0x7fc3040b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:09.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.139+0000 7fc3094e5640 1 --2- 172.21.15.88:0/558578266 >> v2:172.21.15.154:3300/0 conn(0x7fc3040d1820 0x7fc3040b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.139+0000 7fc3094e5640 1 -- 172.21.15.88:0/558578266 >> v1:172.21.15.154:6789/0 conn(0x7fc3040d3310 legacy=0x7fc3040d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:09.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.139+0000 7fc3094e5640 1 -- 172.21.15.88:0/558578266 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3040d3e30 con 0x7fc30415e660 2024-09-16T01:39:09.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.139+0000 7fc309ce6640 1 --2- 172.21.15.88:0/558578266 >> v2:172.21.15.154:3300/0 conn(0x7fc3040d1820 0x7fc3040b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:09.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.139+0000 7fc3094e5640 1 --2- 172.21.15.88:0/558578266 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc30415e660 0x7fc3040cf730 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fc3040d26b0 tx=0x7fc30002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=329e3dd69b635ab9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:09.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.140+0000 7fc2fbfff640 1 -- 172.21.15.88:0/558578266 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc300037070 con 0x7fc30415e660 2024-09-16T01:39:09.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.140+0000 7fc2fbfff640 1 -- 172.21.15.88:0/558578266 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc300032440 con 0x7fc30415e660 2024-09-16T01:39:09.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.140+0000 7fc2fbfff640 1 -- 172.21.15.88:0/558578266 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc300032810 con 0x7fc30415e660 2024-09-16T01:39:09.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.140+0000 7fc30b749640 1 -- 172.21.15.88:0/558578266 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc30415e660 msgr2=0x7fc3040cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:09.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.140+0000 7fc30b749640 1 --2- 172.21.15.88:0/558578266 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc30415e660 0x7fc3040cf730 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fc3040d26b0 tx=0x7fc30002a230 comp rx=0 tx=0).stop 2024-09-16T01:39:09.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.140+0000 7fc30b749640 1 -- 172.21.15.88:0/558578266 shutdown_connections 2024-09-16T01:39:09.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.140+0000 7fc30b749640 1 --2- 172.21.15.88:0/558578266 >> v2:172.21.15.154:3300/0 conn(0x7fc3040d1820 0x7fc3040b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.140+0000 7fc30b749640 1 --2- 172.21.15.88:0/558578266 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc30415e660 0x7fc3040cf730 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.140+0000 7fc30b749640 1 -- 172.21.15.88:0/558578266 >> 172.21.15.88:0/558578266 conn(0x7fc3040b6410 msgr2=0x7fc3040b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:09.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.140+0000 7fc30b749640 1 -- 172.21.15.88:0/558578266 shutdown_connections 2024-09-16T01:39:09.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.140+0000 7fc30b749640 1 -- 172.21.15.88:0/558578266 wait complete. 2024-09-16T01:39:09.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.141+0000 7fc30b749640 1 Processor -- start 2024-09-16T01:39:09.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.142+0000 7fc30b749640 1 -- start start 2024-09-16T01:39:09.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.142+0000 7fc30b749640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3040d1820 0x7fc3040b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:09.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.142+0000 7fc30b749640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3040d3310 0x7fc3040cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:09.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.142+0000 7fc30b749640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc30415e660 0x7fc3041f44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:09.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.142+0000 7fc30b749640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc3040cfc70 con 0x7fc3040d3310 2024-09-16T01:39:09.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.142+0000 7fc30b749640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc3040d43f0 con 0x7fc30415e660 2024-09-16T01:39:09.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.142+0000 7fc30b749640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc3040b80b0 con 0x7fc3040d1820 2024-09-16T01:39:09.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.142+0000 7fc308ce4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3040d3310 0x7fc3040cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:09.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.142+0000 7fc308ce4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3040d3310 0x7fc3040cc1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53008/0 (socket says 172.21.15.88:53008) 2024-09-16T01:39:09.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.142+0000 7fc308ce4640 1 -- 172.21.15.88:0/302499563 learned_addr learned my addr 172.21.15.88:0/302499563 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:09.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.142+0000 7fc3094e5640 1 --2- 172.21.15.88:0/302499563 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3040d1820 0x7fc3040b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:09.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.142+0000 7fc309ce6640 1 --2- 172.21.15.88:0/302499563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc30415e660 0x7fc3041f44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:09.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.143+0000 7fc308ce4640 1 -- 172.21.15.88:0/302499563 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3040d1820 msgr2=0x7fc3040b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:09.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.143+0000 7fc308ce4640 1 --2- 172.21.15.88:0/302499563 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3040d1820 0x7fc3040b3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.143+0000 7fc308ce4640 1 -- 172.21.15.88:0/302499563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc30415e660 msgr2=0x7fc3041f44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:09.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.143+0000 7fc308ce4640 1 --2- 172.21.15.88:0/302499563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc30415e660 0x7fc3041f44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.143+0000 7fc308ce4640 1 -- 172.21.15.88:0/302499563 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3040c8410 con 0x7fc3040d3310 2024-09-16T01:39:09.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.143+0000 7fc3094e5640 1 --2- 172.21.15.88:0/302499563 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3040d1820 0x7fc3040b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:09.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.143+0000 7fc309ce6640 1 --2- 172.21.15.88:0/302499563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc30415e660 0x7fc3041f44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:09.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.143+0000 7fc308ce4640 1 --2- 172.21.15.88:0/302499563 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3040d3310 0x7fc3040cc1e0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fc2f400c430 tx=0x7fc2f400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:09.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.143+0000 7fc2f9ffb640 1 -- 172.21.15.88:0/302499563 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f4011790 con 0x7fc3040d3310 2024-09-16T01:39:09.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.143+0000 7fc2f9ffb640 1 -- 172.21.15.88:0/302499563 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2f4017070 con 0x7fc3040d3310 2024-09-16T01:39:09.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.143+0000 7fc30b749640 1 -- 172.21.15.88:0/302499563 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3040c86f0 con 0x7fc3040d3310 2024-09-16T01:39:09.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.143+0000 7fc2f9ffb640 1 -- 172.21.15.88:0/302499563 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f4011bb0 con 0x7fc3040d3310 2024-09-16T01:39:09.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.143+0000 7fc30b749640 1 -- 172.21.15.88:0/302499563 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3040c8cb0 con 0x7fc3040d3310 2024-09-16T01:39:09.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.144+0000 7fc30b749640 1 -- 172.21.15.88:0/302499563 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3040b7b70 con 0x7fc3040d3310 2024-09-16T01:39:09.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.144+0000 7fc2f9ffb640 1 -- 172.21.15.88:0/302499563 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc2f401c460 con 0x7fc3040d3310 2024-09-16T01:39:09.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.145+0000 7fc2f9ffb640 1 --2- 172.21.15.88:0/302499563 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc2e806c2b0 0x7fc2e806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:09.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.145+0000 7fc3094e5640 1 --2- 172.21.15.88:0/302499563 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc2e806c2b0 0x7fc2e806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:09.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.145+0000 7fc2f9ffb640 1 -- 172.21.15.88:0/302499563 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5115+0+0 (secure 0 0 0) 0x7fc2f408b050 con 0x7fc3040d3310 2024-09-16T01:39:09.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.146+0000 7fc3094e5640 1 --2- 172.21.15.88:0/302499563 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc2e806c2b0 0x7fc2e806e770 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc300034000 tx=0x7fc30002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:09.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.149+0000 7fc2f9ffb640 1 -- 172.21.15.88:0/302499563 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc2f4059aa0 con 0x7fc3040d3310 2024-09-16T01:39:09.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.379+0000 7fc30b749640 1 -- 172.21.15.88:0/302499563 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_precommit"]} v 0) -- 0x7fc30415c780 con 0x7fc3040d3310 2024-09-16T01:39:09.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.427+0000 7fc2f9ffb640 1 -- 172.21.15.88:0/302499563 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_precommit"]}]=0 v235) ==== 142+0+0 (secure 0 0 0) 0x7fc2f40593a0 con 0x7fc3040d3310 2024-09-16T01:39:09.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.429+0000 7fc30b749640 1 -- 172.21.15.88:0/302499563 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc2e806c2b0 msgr2=0x7fc2e806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:09.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.429+0000 7fc30b749640 1 --2- 172.21.15.88:0/302499563 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc2e806c2b0 0x7fc2e806e770 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc300034000 tx=0x7fc30002e000 comp rx=0 tx=0).stop 2024-09-16T01:39:09.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.429+0000 7fc30b749640 1 -- 172.21.15.88:0/302499563 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3040d3310 msgr2=0x7fc3040cc1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:09.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.429+0000 7fc30b749640 1 --2- 172.21.15.88:0/302499563 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3040d3310 0x7fc3040cc1e0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fc2f400c430 tx=0x7fc2f400c8f0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.429+0000 7fc30b749640 1 -- 172.21.15.88:0/302499563 shutdown_connections 2024-09-16T01:39:09.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.429+0000 7fc30b749640 1 --2- 172.21.15.88:0/302499563 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc2e806c2b0 0x7fc2e806e770 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.429+0000 7fc30b749640 1 --2- 172.21.15.88:0/302499563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc30415e660 0x7fc3041f44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.429+0000 7fc30b749640 1 --2- 172.21.15.88:0/302499563 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3040d3310 0x7fc3040cc1e0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.429+0000 7fc30b749640 1 --2- 172.21.15.88:0/302499563 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3040d1820 0x7fc3040b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.429+0000 7fc30b749640 1 -- 172.21.15.88:0/302499563 >> 172.21.15.88:0/302499563 conn(0x7fc3040b6410 msgr2=0x7fc30406bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:09.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.429+0000 7fc30b749640 1 -- 172.21.15.88:0/302499563 shutdown_connections 2024-09-16T01:39:09.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.429+0000 7fc30b749640 1 -- 172.21.15.88:0/302499563 wait complete. 2024-09-16T01:39:09.441 INFO:tasks.cephfs_test_runner:That the MDS won't write a dentry with new damage to CDentry::first ... ok 2024-09-16T01:39:09.442 INFO:tasks.cephfs_test_runner:Starting test: test_dentry_first_preflush (tasks.cephfs.test_damage.TestDamage) That the MDS won't write a dentry with new damage to CDentry::first 2024-09-16T01:39:09.442 INFO:tasks.cephfs_test_runner:test_dentry_first_preflush (tasks.cephfs.test_damage.TestDamage) 2024-09-16T01:39:09.442 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:39:09.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.540+0000 7f4d81598640 1 Processor -- start 2024-09-16T01:39:09.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.540+0000 7f4d81598640 1 -- start start 2024-09-16T01:39:09.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.541+0000 7f4d81598640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c0b7290 0x7f4d7c169240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:09.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.541+0000 7f4d81598640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d7c0708d0 con 0x7f4d7c0d1b40 2024-09-16T01:39:09.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.541+0000 7f4d81598640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d7c070bb0 con 0x7f4d7c0b6eb0 2024-09-16T01:39:09.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.541+0000 7f4d81598640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4d7c070e90 con 0x7f4d7c0b7290 2024-09-16T01:39:09.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.541+0000 7f4d727fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4d7c0d1b40 0x7f4d7c1618a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57484/0 (socket says 172.21.15.88:57484) 2024-09-16T01:39:09.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.541+0000 7f4d727fc640 1 -- 172.21.15.88:0/548829648 learned_addr learned my addr 172.21.15.88:0/548829648 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:09.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.541+0000 7f4d7a7fc640 1 -- 172.21.15.88:0/548829648 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 74264028 0 0) 0x7f4d7c0708d0 con 0x7f4d7c0d1b40 2024-09-16T01:39:09.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.541+0000 7f4d7a7fc640 1 -- 172.21.15.88:0/548829648 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d68003520 con 0x7f4d7c0d1b40 2024-09-16T01:39:09.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.541+0000 7f4d7b7fe640 1 --2- 172.21.15.88:0/548829648 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c0b7290 0x7f4d7c169240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:09.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.542+0000 7f4d7a7fc640 1 -- 172.21.15.88:0/548829648 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3617900444 0 0) 0x7f4d68003520 con 0x7f4d7c0d1b40 2024-09-16T01:39:09.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.542+0000 7f4d7a7fc640 1 -- 172.21.15.88:0/548829648 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c0b7290 msgr2=0x7f4d7c169240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:09.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.542+0000 7f4d7a7fc640 1 --2- 172.21.15.88:0/548829648 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c0b7290 0x7f4d7c169240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.542+0000 7f4d7a7fc640 1 -- 172.21.15.88:0/548829648 >> v1:172.21.15.154:6789/0 conn(0x7f4d7c0b6eb0 legacy=0x7f4d7c06fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:09.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.542+0000 7f4d7a7fc640 1 -- 172.21.15.88:0/548829648 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d7c071170 con 0x7f4d7c0d1b40 2024-09-16T01:39:09.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.542+0000 7f4d7b7fe640 1 --2- 172.21.15.88:0/548829648 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c0b7290 0x7f4d7c169240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:09.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.542+0000 7f4d7a7fc640 1 -- 172.21.15.88:0/548829648 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f4d64001690 con 0x7f4d7c0d1b40 2024-09-16T01:39:09.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.542+0000 7f4d7a7fc640 1 -- 172.21.15.88:0/548829648 >> v1:172.21.15.88:6789/0 conn(0x7f4d7c0d1b40 legacy=0x7f4d7c1618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:09.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.542+0000 7f4d7a7fc640 1 --2- 172.21.15.88:0/548829648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d680039e0 0x7f4d68003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:09.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.542+0000 7f4d7a7fc640 1 --2- 172.21.15.88:0/548829648 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d68004800 0x7f4d68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:09.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.542+0000 7f4d7a7fc640 1 --2- 172.21.15.88:0/548829648 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c0b7290 0x7f4d6800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:09.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.542+0000 7f4d7a7fc640 1 -- 172.21.15.88:0/548829648 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c070e90 con 0x7f4d68004800 2024-09-16T01:39:09.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.542+0000 7f4d7a7fc640 1 -- 172.21.15.88:0/548829648 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4d680055f0 con 0x7f4d680039e0 2024-09-16T01:39:09.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.542+0000 7f4d7a7fc640 1 -- 172.21.15.88:0/548829648 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4d6800bf00 con 0x7f4d7c0b7290 2024-09-16T01:39:09.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.542+0000 7f4d727fc640 1 --2- 172.21.15.88:0/548829648 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d68004800 0x7f4d68004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:09.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.542+0000 7f4d7b7fe640 1 --2- 172.21.15.88:0/548829648 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c0b7290 0x7f4d6800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:09.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.542+0000 7f4d7affd640 1 --2- 172.21.15.88:0/548829648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d680039e0 0x7f4d68003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:09.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.542+0000 7f4d727fc640 1 -- 172.21.15.88:0/548829648 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c0b7290 msgr2=0x7f4d6800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:09.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.542+0000 7f4d727fc640 1 --2- 172.21.15.88:0/548829648 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c0b7290 0x7f4d6800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.542+0000 7f4d727fc640 1 -- 172.21.15.88:0/548829648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d680039e0 msgr2=0x7f4d68003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:09.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.542+0000 7f4d727fc640 1 --2- 172.21.15.88:0/548829648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d680039e0 0x7f4d68003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.542+0000 7f4d727fc640 1 -- 172.21.15.88:0/548829648 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d68003520 con 0x7f4d68004800 2024-09-16T01:39:09.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.543+0000 7f4d727fc640 1 --2- 172.21.15.88:0/548829648 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d68004800 0x7f4d68004c80 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f4d6400c160 tx=0x7f4d68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a066ac7de7f4c67f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:09.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.543+0000 7f4d7a7fc640 1 -- 172.21.15.88:0/548829648 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d64007c60 con 0x7f4d68004800 2024-09-16T01:39:09.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.543+0000 7f4d7a7fc640 1 -- 172.21.15.88:0/548829648 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d64005af0 con 0x7f4d68004800 2024-09-16T01:39:09.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.543+0000 7f4d7a7fc640 1 -- 172.21.15.88:0/548829648 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d64007070 con 0x7f4d68004800 2024-09-16T01:39:09.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.543+0000 7f4d81598640 1 -- 172.21.15.88:0/548829648 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d68004800 msgr2=0x7f4d68004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:09.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.543+0000 7f4d81598640 1 --2- 172.21.15.88:0/548829648 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d68004800 0x7f4d68004c80 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f4d6400c160 tx=0x7f4d68005830 comp rx=0 tx=0).stop 2024-09-16T01:39:09.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.544+0000 7f4d7b7fe640 1 -- 172.21.15.88:0/548829648 reap_dead start 2024-09-16T01:39:09.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.544+0000 7f4d81598640 1 -- 172.21.15.88:0/548829648 shutdown_connections 2024-09-16T01:39:09.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.544+0000 7f4d81598640 1 -- 172.21.15.88:0/548829648 >> 172.21.15.88:0/548829648 conn(0x7f4d7c0d2450 msgr2=0x7f4d7c0d2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:09.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.544+0000 7f4d81598640 1 -- 172.21.15.88:0/548829648 shutdown_connections 2024-09-16T01:39:09.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.544+0000 7f4d81598640 1 -- 172.21.15.88:0/548829648 wait complete. 2024-09-16T01:39:09.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.545+0000 7f4d81598640 1 Processor -- start 2024-09-16T01:39:09.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.545+0000 7f4d81598640 1 -- start start 2024-09-16T01:39:09.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.546+0000 7f4d81598640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d68004800 0x7f4d7c140ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:09.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.546+0000 7f4d81598640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c14f290 0x7f4d7c15ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.546+0000 7f4d81598640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d7c155f10 0x7f4d7c15b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.546+0000 7f4d81598640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c071af0 con 0x7f4d7c155f10 2024-09-16T01:39:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.546+0000 7f4d81598640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c071970 con 0x7f4d68004800 2024-09-16T01:39:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.546+0000 7f4d81598640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4d7c167230 con 0x7f4d7c14f290 2024-09-16T01:39:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.546+0000 7f4d7b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d7c155f10 0x7f4d7c15b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.546+0000 7f4d7b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d7c155f10 0x7f4d7c15b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53032/0 (socket says 172.21.15.88:53032) 2024-09-16T01:39:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.546+0000 7f4d7b7fe640 1 -- 172.21.15.88:0/3116371267 learned_addr learned my addr 172.21.15.88:0/3116371267 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.546+0000 7f4d7affd640 1 --2- 172.21.15.88:0/3116371267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d68004800 0x7f4d7c140ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.546+0000 7f4d727fc640 1 --2- 172.21.15.88:0/3116371267 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c14f290 0x7f4d7c15ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.546+0000 7f4d7b7fe640 1 -- 172.21.15.88:0/3116371267 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c14f290 msgr2=0x7f4d7c15ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.546+0000 7f4d7b7fe640 1 --2- 172.21.15.88:0/3116371267 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c14f290 0x7f4d7c15ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.546+0000 7f4d7b7fe640 1 -- 172.21.15.88:0/3116371267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d68004800 msgr2=0x7f4d7c140ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.546+0000 7f4d7b7fe640 1 --2- 172.21.15.88:0/3116371267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d68004800 0x7f4d7c140ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.546+0000 7f4d7b7fe640 1 -- 172.21.15.88:0/3116371267 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d7c153750 con 0x7f4d7c155f10 2024-09-16T01:39:09.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.546+0000 7f4d727fc640 1 --2- 172.21.15.88:0/3116371267 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c14f290 0x7f4d7c15ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:09.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.546+0000 7f4d7b7fe640 1 --2- 172.21.15.88:0/3116371267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d7c155f10 0x7f4d7c15b150 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c0d1f20 tx=0x7f4d74010530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:09.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.546+0000 7f4d73fff640 1 -- 172.21.15.88:0/3116371267 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d74010d20 con 0x7f4d7c155f10 2024-09-16T01:39:09.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.546+0000 7f4d73fff640 1 -- 172.21.15.88:0/3116371267 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d74003990 con 0x7f4d7c155f10 2024-09-16T01:39:09.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.546+0000 7f4d81598640 1 -- 172.21.15.88:0/3116371267 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d7c153a30 con 0x7f4d7c155f10 2024-09-16T01:39:09.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.546+0000 7f4d73fff640 1 -- 172.21.15.88:0/3116371267 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d74003b80 con 0x7f4d7c155f10 2024-09-16T01:39:09.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.546+0000 7f4d81598640 1 -- 172.21.15.88:0/3116371267 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d7c1533b0 con 0x7f4d7c155f10 2024-09-16T01:39:09.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.548+0000 7f4d81598640 1 -- 172.21.15.88:0/3116371267 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d7c06fd00 con 0x7f4d7c155f10 2024-09-16T01:39:09.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.548+0000 7f4d73fff640 1 -- 172.21.15.88:0/3116371267 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4d7401e020 con 0x7f4d7c155f10 2024-09-16T01:39:09.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.548+0000 7f4d73fff640 1 --2- 172.21.15.88:0/3116371267 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d5c06c2b0 0x7f4d5c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:09.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.548+0000 7f4d73fff640 1 -- 172.21.15.88:0/3116371267 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5115+0+0 (secure 0 0 0) 0x7f4d7408b760 con 0x7f4d7c155f10 2024-09-16T01:39:09.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.548+0000 7f4d7affd640 1 --2- 172.21.15.88:0/3116371267 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d5c06c2b0 0x7f4d5c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:09.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.549+0000 7f4d7affd640 1 --2- 172.21.15.88:0/3116371267 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d5c06c2b0 0x7f4d5c06e770 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c1f7ae0 tx=0x7f4d6c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:09.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.553+0000 7f4d73fff640 1 -- 172.21.15.88:0/3116371267 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4d74025020 con 0x7f4d7c155f10 2024-09-16T01:39:09.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.718+0000 7f4d81598640 1 -- 172.21.15.88:0/3116371267 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4d7c0b19b0 con 0x7f4d7c155f10 2024-09-16T01:39:09.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.719+0000 7f4d73fff640 1 -- 172.21.15.88:0/3116371267 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) ==== 74+0+13264 (secure 0 0 0) 0x7f4d7405a1b0 con 0x7f4d7c155f10 2024-09-16T01:39:09.721 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:09.721 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":63,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:39:08.417410+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:6834/998283126":"2024-09-17T01:39:05.144665+0000","172.21.15.88:6837/4130074102":"2024-09-17T01:37:24.240722+0000","172.21.15.88:6836/998283126":"2024-09-17T01:39:05.144665+0000","172.21.15.88:6835/4130074102":"2024-09-17T01:37:24.240722+0000","172.21.15.88:6837/574169866":"2024-09-17T01:36:48.820862+0000","172.21.15.88:6835/574169866":"2024-09-17T01:36:48.820862+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:39:09.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.721+0000 7f4d81598640 1 -- 172.21.15.88:0/3116371267 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d5c06c2b0 msgr2=0x7f4d5c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:09.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.721+0000 7f4d81598640 1 --2- 172.21.15.88:0/3116371267 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d5c06c2b0 0x7f4d5c06e770 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c1f7ae0 tx=0x7f4d6c009040 comp rx=0 tx=0).stop 2024-09-16T01:39:09.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.721+0000 7f4d81598640 1 -- 172.21.15.88:0/3116371267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d7c155f10 msgr2=0x7f4d7c15b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:09.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.721+0000 7f4d81598640 1 --2- 172.21.15.88:0/3116371267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d7c155f10 0x7f4d7c15b150 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c0d1f20 tx=0x7f4d74010530 comp rx=0 tx=0).stop 2024-09-16T01:39:09.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.721+0000 7f4d81598640 1 -- 172.21.15.88:0/3116371267 shutdown_connections 2024-09-16T01:39:09.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.721+0000 7f4d81598640 1 --2- 172.21.15.88:0/3116371267 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d5c06c2b0 0x7f4d5c06e770 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.721+0000 7f4d81598640 1 --2- 172.21.15.88:0/3116371267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d7c155f10 0x7f4d7c15b150 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.721+0000 7f4d81598640 1 --2- 172.21.15.88:0/3116371267 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c14f290 0x7f4d7c15ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.721+0000 7f4d81598640 1 --2- 172.21.15.88:0/3116371267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d68004800 0x7f4d7c140ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.721+0000 7f4d81598640 1 -- 172.21.15.88:0/3116371267 >> 172.21.15.88:0/3116371267 conn(0x7f4d7c0d2450 msgr2=0x7f4d7c0b2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:09.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.721+0000 7f4d81598640 1 -- 172.21.15.88:0/3116371267 shutdown_connections 2024-09-16T01:39:09.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.721+0000 7f4d81598640 1 -- 172.21.15.88:0/3116371267 wait complete. 2024-09-16T01:39:09.733 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T01:34:10.547178+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T01:39:09.734 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T01:39:09.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.833+0000 7fdd3c853640 1 Processor -- start 2024-09-16T01:39:09.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.833+0000 7fdd3c853640 1 -- start start 2024-09-16T01:39:09.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.834+0000 7fdd3c853640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdd38158a00 0x7fdd38071f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:09.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.834+0000 7fdd3c853640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdd38072470 0x7fdd380b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:09.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.834+0000 7fdd3c853640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd380728f0 con 0x7fdd38158de0 2024-09-16T01:39:09.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.834+0000 7fdd3c853640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fdd38072bd0 con 0x7fdd38158a00 2024-09-16T01:39:09.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.834+0000 7fdd3c853640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fdd38072eb0 con 0x7fdd38072470 2024-09-16T01:39:09.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.834+0000 7fdd3659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdd38158a00 0x7fdd38071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:09.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.834+0000 7fdd3659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdd38158a00 0x7fdd38071f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53044/0 (socket says 172.21.15.88:53044) 2024-09-16T01:39:09.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.834+0000 7fdd3659c640 1 -- 172.21.15.88:0/2658466826 learned_addr learned my addr 172.21.15.88:0/2658466826 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:09.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.834+0000 7fdd35d9b640 1 --2- 172.21.15.88:0/2658466826 >> v2:172.21.15.154:3300/0 conn(0x7fdd38072470 0x7fdd380b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:09.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.835+0000 7fdd3659c640 1 -- 172.21.15.88:0/2658466826 >> v2:172.21.15.154:3300/0 conn(0x7fdd38072470 msgr2=0x7fdd380b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:09.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.835+0000 7fdd3659c640 1 --2- 172.21.15.88:0/2658466826 >> v2:172.21.15.154:3300/0 conn(0x7fdd38072470 0x7fdd380b4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.835+0000 7fdd3659c640 1 -- 172.21.15.88:0/2658466826 >> v1:172.21.15.154:6789/0 conn(0x7fdd38158de0 legacy=0x7fdd38162110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:09.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.835+0000 7fdd3659c640 1 -- 172.21.15.88:0/2658466826 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd38073190 con 0x7fdd38158a00 2024-09-16T01:39:09.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.835+0000 7fdd3659c640 1 --2- 172.21.15.88:0/2658466826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd38158a00 0x7fdd38071f30 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fdd3815eb20 tx=0x7fdd2802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=523de146f02bce7b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:09.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.835+0000 7fdd3559a640 1 -- 172.21.15.88:0/2658466826 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd28036070 con 0x7fdd38158a00 2024-09-16T01:39:09.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.835+0000 7fdd3559a640 1 -- 172.21.15.88:0/2658466826 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd28031600 con 0x7fdd38158a00 2024-09-16T01:39:09.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.835+0000 7fdd3559a640 1 -- 172.21.15.88:0/2658466826 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd280319d0 con 0x7fdd38158a00 2024-09-16T01:39:09.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.836+0000 7fdd3c853640 1 -- 172.21.15.88:0/2658466826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd38158a00 msgr2=0x7fdd38071f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:09.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.836+0000 7fdd3c853640 1 --2- 172.21.15.88:0/2658466826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd38158a00 0x7fdd38071f30 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fdd3815eb20 tx=0x7fdd2802a6f0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.836+0000 7fdd3c853640 1 -- 172.21.15.88:0/2658466826 shutdown_connections 2024-09-16T01:39:09.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.836+0000 7fdd3c853640 1 --2- 172.21.15.88:0/2658466826 >> v2:172.21.15.154:3300/0 conn(0x7fdd38072470 0x7fdd380b4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.836+0000 7fdd3c853640 1 --2- 172.21.15.88:0/2658466826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd38158a00 0x7fdd38071f30 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.836+0000 7fdd3c853640 1 -- 172.21.15.88:0/2658466826 >> 172.21.15.88:0/2658466826 conn(0x7fdd3806cb50 msgr2=0x7fdd3806cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:09.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.836+0000 7fdd3c853640 1 -- 172.21.15.88:0/2658466826 shutdown_connections 2024-09-16T01:39:09.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.836+0000 7fdd3c853640 1 -- 172.21.15.88:0/2658466826 wait complete. 2024-09-16T01:39:09.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.837+0000 7fdd3c853640 1 Processor -- start 2024-09-16T01:39:09.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.837+0000 7fdd3c853640 1 -- start start 2024-09-16T01:39:09.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.838+0000 7fdd3c853640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd38072470 0x7fdd3814e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:09.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.838+0000 7fdd3c853640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd38158a00 0x7fdd38155800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:09.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.838+0000 7fdd3c853640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd38158de0 0x7fdd381e8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:09.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.838+0000 7fdd3c853640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdd380b5ca0 con 0x7fdd38158de0 2024-09-16T01:39:09.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.838+0000 7fdd3c853640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdd380b5b20 con 0x7fdd38072470 2024-09-16T01:39:09.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.838+0000 7fdd3c853640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdd3806faf0 con 0x7fdd38158a00 2024-09-16T01:39:09.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.838+0000 7fdd36d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd38158de0 0x7fdd381e8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:09.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.838+0000 7fdd36d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd38158de0 0x7fdd381e8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53058/0 (socket says 172.21.15.88:53058) 2024-09-16T01:39:09.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.838+0000 7fdd36d9d640 1 -- 172.21.15.88:0/2347901207 learned_addr learned my addr 172.21.15.88:0/2347901207 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:09.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.838+0000 7fdd35d9b640 1 --2- 172.21.15.88:0/2347901207 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd38158a00 0x7fdd38155800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:09.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.838+0000 7fdd3659c640 1 --2- 172.21.15.88:0/2347901207 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd38072470 0x7fdd3814e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:09.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.839+0000 7fdd36d9d640 1 -- 172.21.15.88:0/2347901207 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd38158a00 msgr2=0x7fdd38155800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:09.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.839+0000 7fdd36d9d640 1 --2- 172.21.15.88:0/2347901207 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd38158a00 0x7fdd38155800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.839+0000 7fdd36d9d640 1 -- 172.21.15.88:0/2347901207 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd38072470 msgr2=0x7fdd3814e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:09.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.839+0000 7fdd36d9d640 1 --2- 172.21.15.88:0/2347901207 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd38072470 0x7fdd3814e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:09.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.839+0000 7fdd36d9d640 1 -- 172.21.15.88:0/2347901207 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd3814e8f0 con 0x7fdd38158de0 2024-09-16T01:39:09.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.839+0000 7fdd36d9d640 1 --2- 172.21.15.88:0/2347901207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd38158de0 0x7fdd381e8fa0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fdd3000b440 tx=0x7fdd3000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:09.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.839+0000 7fdd26ffd640 1 -- 172.21.15.88:0/2347901207 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30015020 con 0x7fdd38158de0 2024-09-16T01:39:09.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.839+0000 7fdd26ffd640 1 -- 172.21.15.88:0/2347901207 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd30011830 con 0x7fdd38158de0 2024-09-16T01:39:09.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.839+0000 7fdd3c853640 1 -- 172.21.15.88:0/2347901207 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd3815fac0 con 0x7fdd38158de0 2024-09-16T01:39:09.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.839+0000 7fdd26ffd640 1 -- 172.21.15.88:0/2347901207 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30011c20 con 0x7fdd38158de0 2024-09-16T01:39:09.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.839+0000 7fdd3c853640 1 -- 172.21.15.88:0/2347901207 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd38160040 con 0x7fdd38158de0 2024-09-16T01:39:09.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.840+0000 7fdd3c853640 1 -- 172.21.15.88:0/2347901207 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd38071f30 con 0x7fdd38158de0 2024-09-16T01:39:09.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.840+0000 7fdd26ffd640 1 -- 172.21.15.88:0/2347901207 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fdd30018020 con 0x7fdd38158de0 2024-09-16T01:39:09.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.841+0000 7fdd26ffd640 1 --2- 172.21.15.88:0/2347901207 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdd1806c2b0 0x7fdd1806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:09.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.841+0000 7fdd3659c640 1 --2- 172.21.15.88:0/2347901207 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdd1806c2b0 0x7fdd1806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:09.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.841+0000 7fdd26ffd640 1 -- 172.21.15.88:0/2347901207 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5115+0+0 (secure 0 0 0) 0x7fdd3008ac90 con 0x7fdd38158de0 2024-09-16T01:39:09.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.841+0000 7fdd3659c640 1 --2- 172.21.15.88:0/2347901207 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdd1806c2b0 0x7fdd1806e770 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fdd28033000 tx=0x7fdd28009000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:09.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:09.845+0000 7fdd26ffd640 1 -- 172.21.15.88:0/2347901207 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fdd3001f030 con 0x7fdd38158de0 2024-09-16T01:39:10.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.017+0000 7fdd3c853640 1 -- 172.21.15.88:0/2347901207 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdd38154650 con 0x7fdd38158de0 2024-09-16T01:39:10.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.018+0000 7fdd26ffd640 1 -- 172.21.15.88:0/2347901207 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v63) ==== 93+0+10 (secure 0 0 0) 0x7fdd3814e8f0 con 0x7fdd38158de0 2024-09-16T01:39:10.019 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2024-09-16T01:39:10.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.020+0000 7fdd3c853640 1 -- 172.21.15.88:0/2347901207 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdd1806c2b0 msgr2=0x7fdd1806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:10.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.020+0000 7fdd3c853640 1 --2- 172.21.15.88:0/2347901207 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdd1806c2b0 0x7fdd1806e770 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fdd28033000 tx=0x7fdd28009000 comp rx=0 tx=0).stop 2024-09-16T01:39:10.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.020+0000 7fdd3c853640 1 -- 172.21.15.88:0/2347901207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd38158de0 msgr2=0x7fdd381e8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:10.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.020+0000 7fdd3c853640 1 --2- 172.21.15.88:0/2347901207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd38158de0 0x7fdd381e8fa0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fdd3000b440 tx=0x7fdd3000b900 comp rx=0 tx=0).stop 2024-09-16T01:39:10.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.020+0000 7fdd3c853640 1 -- 172.21.15.88:0/2347901207 shutdown_connections 2024-09-16T01:39:10.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.020+0000 7fdd3c853640 1 --2- 172.21.15.88:0/2347901207 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdd1806c2b0 0x7fdd1806e770 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:10.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.020+0000 7fdd3c853640 1 --2- 172.21.15.88:0/2347901207 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd38158de0 0x7fdd381e8fa0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:10.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.020+0000 7fdd3c853640 1 --2- 172.21.15.88:0/2347901207 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd38158a00 0x7fdd38155800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:10.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.020+0000 7fdd3c853640 1 --2- 172.21.15.88:0/2347901207 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd38072470 0x7fdd3814e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:10.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.020+0000 7fdd3c853640 1 -- 172.21.15.88:0/2347901207 >> 172.21.15.88:0/2347901207 conn(0x7fdd3806cb50 msgr2=0x7fdd38161cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:10.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.020+0000 7fdd3c853640 1 -- 172.21.15.88:0/2347901207 shutdown_connections 2024-09-16T01:39:10.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.020+0000 7fdd3c853640 1 -- 172.21.15.88:0/2347901207 wait complete. 2024-09-16T01:39:10.032 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_preflush' 2024-09-16T01:39:10.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.129+0000 7f11b7ffd640 1 Processor -- start 2024-09-16T01:39:10.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.130+0000 7f11b7ffd640 1 -- start start 2024-09-16T01:39:10.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.130+0000 7f11b7ffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11b00b7560 0x7f11b006d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:10.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.130+0000 7f11b7ffd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11b00b58f0 con 0x7f11b015e800 2024-09-16T01:39:10.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.130+0000 7f11b7ffd640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11b006ea60 con 0x7f11b00b5460 2024-09-16T01:39:10.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.130+0000 7f11b7ffd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f11b006ed40 con 0x7f11b00b7560 2024-09-16T01:39:10.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b5d99640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f11b015e800 0x7f11b015ebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57490/0 (socket says 172.21.15.88:57490) 2024-09-16T01:39:10.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b5d99640 1 -- 172.21.15.88:0/3290095698 learned_addr learned my addr 172.21.15.88:0/3290095698 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:10.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b4d97640 1 -- 172.21.15.88:0/3290095698 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2288028073 0 0) 0x7f11b00b58f0 con 0x7f11b015e800 2024-09-16T01:39:10.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b659a640 1 --2- 172.21.15.88:0/3290095698 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11b00b7560 0x7f11b006d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:10.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b4d97640 1 -- 172.21.15.88:0/3290095698 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1198003520 con 0x7f11b015e800 2024-09-16T01:39:10.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b4d97640 1 -- 172.21.15.88:0/3290095698 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1743915496 0 0) 0x7f1198003520 con 0x7f11b015e800 2024-09-16T01:39:10.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b4d97640 1 -- 172.21.15.88:0/3290095698 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11b00b7560 msgr2=0x7f11b006d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:10.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b4d97640 1 --2- 172.21.15.88:0/3290095698 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11b00b7560 0x7f11b006d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:10.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b4d97640 1 -- 172.21.15.88:0/3290095698 >> v1:172.21.15.154:6789/0 conn(0x7f11b00b5460 legacy=0x7f11b00b6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:10.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b4d97640 1 -- 172.21.15.88:0/3290095698 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11b006cde0 con 0x7f11b015e800 2024-09-16T01:39:10.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b4d97640 1 -- 172.21.15.88:0/3290095698 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f11a00019f0 con 0x7f11b015e800 2024-09-16T01:39:10.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b4d97640 1 -- 172.21.15.88:0/3290095698 >> v1:172.21.15.88:6789/0 conn(0x7f11b015e800 legacy=0x7f11b015ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:10.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b4d97640 1 --2- 172.21.15.88:0/3290095698 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11980039e0 0x7f1198003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:10.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b4d97640 1 --2- 172.21.15.88:0/3290095698 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11b00b7560 0x7f1198004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:10.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b4d97640 1 --2- 172.21.15.88:0/3290095698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1198004d40 0x7f11980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:10.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b4d97640 1 -- 172.21.15.88:0/3290095698 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f11b006ed40 con 0x7f11980039e0 2024-09-16T01:39:10.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b4d97640 1 -- 172.21.15.88:0/3290095698 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f119800ba70 con 0x7f1198004d40 2024-09-16T01:39:10.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b4d97640 1 -- 172.21.15.88:0/3290095698 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f119800bc70 con 0x7f11b00b7560 2024-09-16T01:39:10.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b5d99640 1 --2- 172.21.15.88:0/3290095698 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11980039e0 0x7f1198003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:10.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b5598640 1 --2- 172.21.15.88:0/3290095698 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11b00b7560 0x7f1198004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:10.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b659a640 1 --2- 172.21.15.88:0/3290095698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1198004d40 0x7f11980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:10.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b5d99640 1 -- 172.21.15.88:0/3290095698 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11b00b7560 msgr2=0x7f1198004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:10.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b5d99640 1 --2- 172.21.15.88:0/3290095698 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11b00b7560 0x7f1198004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:10.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b5d99640 1 -- 172.21.15.88:0/3290095698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1198004d40 msgr2=0x7f11980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:10.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b5d99640 1 --2- 172.21.15.88:0/3290095698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1198004d40 0x7f11980051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:10.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.131+0000 7f11b5d99640 1 -- 172.21.15.88:0/3290095698 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1198003520 con 0x7f11980039e0 2024-09-16T01:39:10.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.132+0000 7f11b5d99640 1 --2- 172.21.15.88:0/3290095698 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11980039e0 0x7f1198003e00 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f11a000a830 tx=0x7f1198005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa155d72b902aee7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:10.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.132+0000 7f11b4d97640 1 -- 172.21.15.88:0/3290095698 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a000a480 con 0x7f11980039e0 2024-09-16T01:39:10.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.132+0000 7f11b4d97640 1 -- 172.21.15.88:0/3290095698 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11a002f070 con 0x7f11980039e0 2024-09-16T01:39:10.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.132+0000 7f11b4d97640 1 -- 172.21.15.88:0/3290095698 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a002bcd0 con 0x7f11980039e0 2024-09-16T01:39:10.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.133+0000 7f11b7ffd640 1 -- 172.21.15.88:0/3290095698 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11980039e0 msgr2=0x7f1198003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:10.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.133+0000 7f11b7ffd640 1 --2- 172.21.15.88:0/3290095698 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11980039e0 0x7f1198003e00 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f11a000a830 tx=0x7f1198005830 comp rx=0 tx=0).stop 2024-09-16T01:39:10.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.133+0000 7f11b659a640 1 -- 172.21.15.88:0/3290095698 reap_dead start 2024-09-16T01:39:10.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.133+0000 7f11b7ffd640 1 -- 172.21.15.88:0/3290095698 shutdown_connections 2024-09-16T01:39:10.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.133+0000 7f11b7ffd640 1 -- 172.21.15.88:0/3290095698 >> 172.21.15.88:0/3290095698 conn(0x7f11b00d24d0 msgr2=0x7f11b00d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:10.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.133+0000 7f11b7ffd640 1 -- 172.21.15.88:0/3290095698 shutdown_connections 2024-09-16T01:39:10.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.133+0000 7f11b7ffd640 1 -- 172.21.15.88:0/3290095698 wait complete. 2024-09-16T01:39:10.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.134+0000 7f11b7ffd640 1 Processor -- start 2024-09-16T01:39:10.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.134+0000 7f11b7ffd640 1 -- start start 2024-09-16T01:39:10.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.135+0000 7f11b7ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11980039e0 0x7f11b01584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:10.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.135+0000 7f11b7ffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11b014be60 0x7f11b00b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:10.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.135+0000 7f11b7ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11b00b31a0 0x7f11b00ce280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:10.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.135+0000 7f11b7ffd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f11b00d31e0 con 0x7f11980039e0 2024-09-16T01:39:10.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.135+0000 7f11b7ffd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f11b006d3a0 con 0x7f11b00b31a0 2024-09-16T01:39:10.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.135+0000 7f11b7ffd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f11b015f2f0 con 0x7f11b014be60 2024-09-16T01:39:10.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.135+0000 7f11b5d99640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11980039e0 0x7f11b01584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:10.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.135+0000 7f11b5d99640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11980039e0 0x7f11b01584d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53074/0 (socket says 172.21.15.88:53074) 2024-09-16T01:39:10.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.135+0000 7f11b5d99640 1 -- 172.21.15.88:0/1674152773 learned_addr learned my addr 172.21.15.88:0/1674152773 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:10.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.135+0000 7f11b5d99640 1 -- 172.21.15.88:0/1674152773 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11b014be60 msgr2=0x7f11b00b2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:10.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.135+0000 7f11b5d99640 1 --2- 172.21.15.88:0/1674152773 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11b014be60 0x7f11b00b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:10.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.135+0000 7f11b5d99640 1 -- 172.21.15.88:0/1674152773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11b00b31a0 msgr2=0x7f11b00ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:10.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.135+0000 7f11b5d99640 1 --2- 172.21.15.88:0/1674152773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11b00b31a0 0x7f11b00ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:10.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.135+0000 7f11b5d99640 1 -- 172.21.15.88:0/1674152773 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11b00ccc30 con 0x7f11980039e0 2024-09-16T01:39:10.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.135+0000 7f11b5d99640 1 --2- 172.21.15.88:0/1674152773 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11980039e0 0x7f11b01584d0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f11a000a950 tx=0x7f11b006ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:10.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.135+0000 7f11ae7fc640 1 -- 172.21.15.88:0/1674152773 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a0044070 con 0x7f11980039e0 2024-09-16T01:39:10.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.135+0000 7f11ae7fc640 1 -- 172.21.15.88:0/1674152773 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11a0034670 con 0x7f11980039e0 2024-09-16T01:39:10.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.136+0000 7f11b7ffd640 1 -- 172.21.15.88:0/1674152773 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11b0158b60 con 0x7f11980039e0 2024-09-16T01:39:10.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.136+0000 7f11ae7fc640 1 -- 172.21.15.88:0/1674152773 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a000a480 con 0x7f11980039e0 2024-09-16T01:39:10.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.136+0000 7f11b7ffd640 1 -- 172.21.15.88:0/1674152773 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11b0159120 con 0x7f11980039e0 2024-09-16T01:39:10.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.137+0000 7f11b7ffd640 1 -- 172.21.15.88:0/1674152773 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11b00b35d0 con 0x7f11980039e0 2024-09-16T01:39:10.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.137+0000 7f11ae7fc640 1 -- 172.21.15.88:0/1674152773 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f11a0047070 con 0x7f11980039e0 2024-09-16T01:39:10.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.137+0000 7f11ae7fc640 1 --2- 172.21.15.88:0/1674152773 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f119c06c2b0 0x7f119c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:10.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.137+0000 7f11ae7fc640 1 -- 172.21.15.88:0/1674152773 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5115+0+0 (secure 0 0 0) 0x7f11a00ab1c0 con 0x7f11980039e0 2024-09-16T01:39:10.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.137+0000 7f11b5598640 1 --2- 172.21.15.88:0/1674152773 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f119c06c2b0 0x7f119c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:10.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.138+0000 7f11b5598640 1 --2- 172.21.15.88:0/1674152773 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f119c06c2b0 0x7f119c06e770 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f11a4001750 tx=0x7f11a4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:10.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.142+0000 7f11ae7fc640 1 -- 172.21.15.88:0/1674152773 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f11a0079b20 con 0x7f11980039e0 2024-09-16T01:39:10.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.371+0000 7f11b7ffd640 1 -- 172.21.15.88:0/1674152773 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_preflush"]} v 0) -- 0x7f11b00b2460 con 0x7f11980039e0 2024-09-16T01:39:10.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.428+0000 7f11ae7fc640 1 -- 172.21.15.88:0/1674152773 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_preflush"]}]=0 v236) ==== 144+0+0 (secure 0 0 0) 0x7f11a002ce20 con 0x7f11980039e0 2024-09-16T01:39:10.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.430+0000 7f11b7ffd640 1 -- 172.21.15.88:0/1674152773 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f119c06c2b0 msgr2=0x7f119c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:10.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.430+0000 7f11b7ffd640 1 --2- 172.21.15.88:0/1674152773 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f119c06c2b0 0x7f119c06e770 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f11a4001750 tx=0x7f11a4009040 comp rx=0 tx=0).stop 2024-09-16T01:39:10.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.430+0000 7f11b7ffd640 1 -- 172.21.15.88:0/1674152773 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11980039e0 msgr2=0x7f11b01584d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:10.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.430+0000 7f11b7ffd640 1 --2- 172.21.15.88:0/1674152773 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11980039e0 0x7f11b01584d0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f11a000a950 tx=0x7f11b006ec30 comp rx=0 tx=0).stop 2024-09-16T01:39:10.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.430+0000 7f11b7ffd640 1 -- 172.21.15.88:0/1674152773 shutdown_connections 2024-09-16T01:39:10.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.430+0000 7f11b7ffd640 1 --2- 172.21.15.88:0/1674152773 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f119c06c2b0 0x7f119c06e770 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:10.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.430+0000 7f11b7ffd640 1 --2- 172.21.15.88:0/1674152773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11b00b31a0 0x7f11b00ce280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:10.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.430+0000 7f11b7ffd640 1 --2- 172.21.15.88:0/1674152773 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11b014be60 0x7f11b00b2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:10.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.430+0000 7f11b7ffd640 1 --2- 172.21.15.88:0/1674152773 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f11980039e0 0x7f11b01584d0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:10.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.430+0000 7f11b7ffd640 1 -- 172.21.15.88:0/1674152773 >> 172.21.15.88:0/1674152773 conn(0x7f11b00d24d0 msgr2=0x7f11b0070570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:10.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.430+0000 7f11b7ffd640 1 -- 172.21.15.88:0/1674152773 shutdown_connections 2024-09-16T01:39:10.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.430+0000 7f11b7ffd640 1 -- 172.21.15.88:0/1674152773 wait complete. 2024-09-16T01:39:10.443 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T01:39:10.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.543+0000 7f5634d2a640 1 Processor -- start 2024-09-16T01:39:10.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.543+0000 7f5634d2a640 1 -- start start 2024-09-16T01:39:10.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.544+0000 7f5634d2a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5630073600 0x7f563015e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:10.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.544+0000 7f5634d2a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f563015a430 con 0x7f56300731c0 2024-09-16T01:39:10.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.544+0000 7f5634d2a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56300cd0a0 con 0x7f563015a050 2024-09-16T01:39:10.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.544+0000 7f5634d2a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f56300cd380 con 0x7f5630073600 2024-09-16T01:39:10.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.544+0000 7f562ed9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5630073600 0x7f563015e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:10.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.544+0000 7f562dd9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f56300731c0 0x7f563015de80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41976/0 (socket says 172.21.15.88:41976) 2024-09-16T01:39:10.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.544+0000 7f562dd9b640 1 -- 172.21.15.88:0/4132428000 learned_addr learned my addr 172.21.15.88:0/4132428000 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:10.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.545+0000 7f562ed9d640 1 -- 172.21.15.88:0/4132428000 >> v1:172.21.15.154:6789/0 conn(0x7f563015a050 legacy=0x7f563015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:10.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.545+0000 7f562ed9d640 1 -- 172.21.15.88:0/4132428000 >> v1:172.21.15.88:6789/0 conn(0x7f56300731c0 legacy=0x7f563015de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:10.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.545+0000 7f562ed9d640 1 -- 172.21.15.88:0/4132428000 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56300cd660 con 0x7f5630073600 2024-09-16T01:39:10.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.545+0000 7f562d59a640 1 -- 172.21.15.88:0/4132428000 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4088269429 0 0) 0x7f563015a430 con 0x7f56300731c0 2024-09-16T01:39:10.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.545+0000 7f562ed9d640 1 --2- 172.21.15.88:0/4132428000 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5630073600 0x7f563015e590 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f56280097a0 tx=0x7f562802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=496cee3277620403 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:10.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.545+0000 7f562d59a640 1 -- 172.21.15.88:0/4132428000 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5628037070 con 0x7f5630073600 2024-09-16T01:39:10.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.545+0000 7f562d59a640 1 -- 172.21.15.88:0/4132428000 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5628032440 con 0x7f5630073600 2024-09-16T01:39:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.545+0000 7f562d59a640 1 -- 172.21.15.88:0/4132428000 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5628032810 con 0x7f5630073600 2024-09-16T01:39:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.546+0000 7f5634d2a640 1 -- 172.21.15.88:0/4132428000 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5630073600 msgr2=0x7f563015e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.546+0000 7f5634d2a640 1 --2- 172.21.15.88:0/4132428000 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5630073600 0x7f563015e590 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f56280097a0 tx=0x7f562802a1a0 comp rx=0 tx=0).stop 2024-09-16T01:39:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.546+0000 7f5634d2a640 1 -- 172.21.15.88:0/4132428000 shutdown_connections 2024-09-16T01:39:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.546+0000 7f5634d2a640 1 --2- 172.21.15.88:0/4132428000 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5630073600 0x7f563015e590 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.546+0000 7f5634d2a640 1 -- 172.21.15.88:0/4132428000 >> 172.21.15.88:0/4132428000 conn(0x7f563006d0f0 msgr2=0x7f563006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.546+0000 7f5634d2a640 1 -- 172.21.15.88:0/4132428000 shutdown_connections 2024-09-16T01:39:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.546+0000 7f5634d2a640 1 -- 172.21.15.88:0/4132428000 wait complete. 2024-09-16T01:39:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.547+0000 7f5634d2a640 1 Processor -- start 2024-09-16T01:39:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.547+0000 7f5634d2a640 1 -- start start 2024-09-16T01:39:10.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.548+0000 7f5634d2a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56300731c0 0x7f5630148030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:10.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.548+0000 7f5634d2a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5630073600 0x7f56300b5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:10.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.548+0000 7f5634d2a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f563015a050 0x7f563015e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.548+0000 7f5634d2a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5630073a90 con 0x7f563015a050 2024-09-16T01:39:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.548+0000 7f5634d2a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f56300cdc20 con 0x7f5630073600 2024-09-16T01:39:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.548+0000 7f5634d2a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f56300b36e0 con 0x7f56300731c0 2024-09-16T01:39:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.548+0000 7f562ed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f563015a050 0x7f563015e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.548+0000 7f562ed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f563015a050 0x7f563015e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37466/0 (socket says 172.21.15.88:37466) 2024-09-16T01:39:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.548+0000 7f562ed9d640 1 -- 172.21.15.88:0/115313305 learned_addr learned my addr 172.21.15.88:0/115313305 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.548+0000 7f562dd9b640 1 --2- 172.21.15.88:0/115313305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5630073600 0x7f56300b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.548+0000 7f562e59c640 1 --2- 172.21.15.88:0/115313305 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56300731c0 0x7f5630148030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.548+0000 7f562ed9d640 1 -- 172.21.15.88:0/115313305 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56300731c0 msgr2=0x7f5630148030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.548+0000 7f562ed9d640 1 --2- 172.21.15.88:0/115313305 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56300731c0 0x7f5630148030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.548+0000 7f562ed9d640 1 -- 172.21.15.88:0/115313305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5630073600 msgr2=0x7f56300b5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.548+0000 7f562ed9d640 1 --2- 172.21.15.88:0/115313305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5630073600 0x7f56300b5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:10.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.548+0000 7f562ed9d640 1 -- 172.21.15.88:0/115313305 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5630148570 con 0x7f563015a050 2024-09-16T01:39:10.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.549+0000 7f562ed9d640 1 --2- 172.21.15.88:0/115313305 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f563015a050 0x7f563015e780 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f5628035040 tx=0x7f56280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:10.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.549+0000 7f561effd640 1 -- 172.21.15.88:0/115313305 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5628037050 con 0x7f563015a050 2024-09-16T01:39:10.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.549+0000 7f561effd640 1 -- 172.21.15.88:0/115313305 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5628038040 con 0x7f563015a050 2024-09-16T01:39:10.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.549+0000 7f5634d2a640 1 -- 172.21.15.88:0/115313305 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5630154630 con 0x7f563015a050 2024-09-16T01:39:10.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.549+0000 7f561effd640 1 -- 172.21.15.88:0/115313305 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5628004490 con 0x7f563015a050 2024-09-16T01:39:10.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.549+0000 7f5634d2a640 1 -- 172.21.15.88:0/115313305 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56300b4a20 con 0x7f563015a050 2024-09-16T01:39:10.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.550+0000 7f5634d2a640 1 -- 172.21.15.88:0/115313305 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56300ccd00 con 0x7f563015a050 2024-09-16T01:39:10.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.550+0000 7f561effd640 1 -- 172.21.15.88:0/115313305 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5628030980 con 0x7f563015a050 2024-09-16T01:39:10.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.551+0000 7f561effd640 1 --2- 172.21.15.88:0/115313305 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f561006c2b0 0x7f561006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:10.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.551+0000 7f561effd640 1 -- 172.21.15.88:0/115313305 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5115+0+0 (secure 0 0 0) 0x7f56280abe90 con 0x7f563015a050 2024-09-16T01:39:10.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.551+0000 7f562e59c640 1 --2- 172.21.15.88:0/115313305 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f561006c2b0 0x7f561006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:10.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.551+0000 7f562e59c640 1 --2- 172.21.15.88:0/115313305 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f561006c2b0 0x7f561006e770 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f56200012e0 tx=0x7f5620009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:10.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.555+0000 7f561effd640 1 -- 172.21.15.88:0/115313305 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5630148570 con 0x7f563015a050 2024-09-16T01:39:10.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.722+0000 7f5634d2a640 1 -- 172.21.15.88:0/115313305 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f5630153740 con 0x7f563015a050 2024-09-16T01:39:10.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.725+0000 7f561effd640 1 -- 172.21.15.88:0/115313305 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) ==== 122+0+0 (secure 0 0 0) 0x7f5628032e20 con 0x7f563015a050 2024-09-16T01:39:10.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.727+0000 7f5634d2a640 1 -- 172.21.15.88:0/115313305 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f561006c2b0 msgr2=0x7f561006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:10.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.727+0000 7f5634d2a640 1 --2- 172.21.15.88:0/115313305 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f561006c2b0 0x7f561006e770 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f56200012e0 tx=0x7f5620009040 comp rx=0 tx=0).stop 2024-09-16T01:39:10.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.727+0000 7f5634d2a640 1 -- 172.21.15.88:0/115313305 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f563015a050 msgr2=0x7f563015e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:10.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.727+0000 7f5634d2a640 1 --2- 172.21.15.88:0/115313305 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f563015a050 0x7f563015e780 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f5628035040 tx=0x7f56280046c0 comp rx=0 tx=0).stop 2024-09-16T01:39:10.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.727+0000 7f5634d2a640 1 -- 172.21.15.88:0/115313305 shutdown_connections 2024-09-16T01:39:10.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.727+0000 7f5634d2a640 1 --2- 172.21.15.88:0/115313305 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f561006c2b0 0x7f561006e770 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:10.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.727+0000 7f5634d2a640 1 --2- 172.21.15.88:0/115313305 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f563015a050 0x7f563015e780 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:10.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.727+0000 7f5634d2a640 1 --2- 172.21.15.88:0/115313305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5630073600 0x7f56300b5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:10.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.727+0000 7f5634d2a640 1 --2- 172.21.15.88:0/115313305 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56300731c0 0x7f5630148030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:10.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.727+0000 7f5634d2a640 1 -- 172.21.15.88:0/115313305 >> 172.21.15.88:0/115313305 conn(0x7f563006d0f0 msgr2=0x7f56300cfac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:10.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.727+0000 7f5634d2a640 1 -- 172.21.15.88:0/115313305 shutdown_connections 2024-09-16T01:39:10.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:10.727+0000 7f5634d2a640 1 -- 172.21.15.88:0/115313305 wait complete. 2024-09-16T01:39:10.740 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T01:39:10.741 DEBUG:teuthology.orchestra.run.smithi088:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T01:39:10.789 DEBUG:teuthology.orchestra.run.smithi154:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T01:39:10.831 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T01:39:10.831 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:39:10.831 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:39:10.846 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:39:10.847 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:39:10.847 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:39:10.890 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056026,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1347 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T01:39:10.890 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:39:10.890 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:39:10.906 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:39:10.906 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:39:10.906 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:39:10.948 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056026,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1347 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T01:39:10.948 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-16T01:39:10.948 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:39:11.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.044+0000 7fb26af91640 1 Processor -- start 2024-09-16T01:39:11.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.044+0000 7fb26af91640 1 -- start start 2024-09-16T01:39:11.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.045+0000 7fb26af91640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb264071430 0x7fb2641582b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:11.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.045+0000 7fb26af91640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb26415ec10 0x7fb26415beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:11.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.045+0000 7fb26af91640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb264073130 con 0x7fb264069ae0 2024-09-16T01:39:11.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.045+0000 7fb26af91640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb264072f00 con 0x7fb26415ec10 2024-09-16T01:39:11.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.045+0000 7fb26af91640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb2640736f0 con 0x7fb264071430 2024-09-16T01:39:11.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.045+0000 7fb26952e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb26415ec10 0x7fb26415beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:11.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.045+0000 7fb26952e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb26415ec10 0x7fb26415beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37472/0 (socket says 172.21.15.88:37472) 2024-09-16T01:39:11.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.045+0000 7fb26952e640 1 -- 172.21.15.88:0/2504235759 learned_addr learned my addr 172.21.15.88:0/2504235759 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:11.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.045+0000 7fb26952e640 1 -- 172.21.15.88:0/2504235759 >> v2:172.21.15.154:3300/0 conn(0x7fb264071430 msgr2=0x7fb2641582b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:11.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.045+0000 7fb26952e640 1 --2- 172.21.15.88:0/2504235759 >> v2:172.21.15.154:3300/0 conn(0x7fb264071430 0x7fb2641582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:11.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.045+0000 7fb26952e640 1 -- 172.21.15.88:0/2504235759 >> v1:172.21.15.154:6789/0 conn(0x7fb264069ae0 legacy=0x7fb26415b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:11.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.045+0000 7fb26952e640 1 -- 172.21.15.88:0/2504235759 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb264073410 con 0x7fb26415ec10 2024-09-16T01:39:11.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.046+0000 7fb26952e640 1 --2- 172.21.15.88:0/2504235759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb26415ec10 0x7fb26415beb0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fb26406e810 tx=0x7fb25c02c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3543e8102561fe8f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:11.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.046+0000 7fb2637fe640 1 -- 172.21.15.88:0/2504235759 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb25c038070 con 0x7fb26415ec10 2024-09-16T01:39:11.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.046+0000 7fb2637fe640 1 -- 172.21.15.88:0/2504235759 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb25c009600 con 0x7fb26415ec10 2024-09-16T01:39:11.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.046+0000 7fb2637fe640 1 -- 172.21.15.88:0/2504235759 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb25c0099d0 con 0x7fb26415ec10 2024-09-16T01:39:11.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.047+0000 7fb26af91640 1 -- 172.21.15.88:0/2504235759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb26415ec10 msgr2=0x7fb26415beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:11.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.047+0000 7fb26af91640 1 --2- 172.21.15.88:0/2504235759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb26415ec10 0x7fb26415beb0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fb26406e810 tx=0x7fb25c02c7e0 comp rx=0 tx=0).stop 2024-09-16T01:39:11.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.047+0000 7fb26af91640 1 -- 172.21.15.88:0/2504235759 shutdown_connections 2024-09-16T01:39:11.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.047+0000 7fb26af91640 1 --2- 172.21.15.88:0/2504235759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb26415ec10 0x7fb26415beb0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:11.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.047+0000 7fb26af91640 1 --2- 172.21.15.88:0/2504235759 >> v2:172.21.15.154:3300/0 conn(0x7fb264071430 0x7fb2641582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:11.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.047+0000 7fb26af91640 1 -- 172.21.15.88:0/2504235759 >> 172.21.15.88:0/2504235759 conn(0x7fb2640b2430 msgr2=0x7fb2640b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:11.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.047+0000 7fb26af91640 1 -- 172.21.15.88:0/2504235759 shutdown_connections 2024-09-16T01:39:11.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.047+0000 7fb26af91640 1 -- 172.21.15.88:0/2504235759 wait complete. 2024-09-16T01:39:11.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.048+0000 7fb26af91640 1 Processor -- start 2024-09-16T01:39:11.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.048+0000 7fb26af91640 1 -- start start 2024-09-16T01:39:11.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.049+0000 7fb26af91640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb264069ae0 0x7fb2640aadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:11.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.049+0000 7fb26af91640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb264071430 0x7fb26415b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:11.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.049+0000 7fb26af91640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb26415ec10 0x7fb26415bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:11.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.049+0000 7fb26af91640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb26415f060 con 0x7fb264071430 2024-09-16T01:39:11.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.049+0000 7fb26af91640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb26415e6f0 con 0x7fb26415ec10 2024-09-16T01:39:11.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.049+0000 7fb26af91640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb26415f1e0 con 0x7fb264069ae0 2024-09-16T01:39:11.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.049+0000 7fb263fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb264071430 0x7fb26415b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:11.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.049+0000 7fb263fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb264071430 0x7fb26415b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37488/0 (socket says 172.21.15.88:37488) 2024-09-16T01:39:11.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.049+0000 7fb263fff640 1 -- 172.21.15.88:0/2191094613 learned_addr learned my addr 172.21.15.88:0/2191094613 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:11.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.049+0000 7fb26952e640 1 --2- 172.21.15.88:0/2191094613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb26415ec10 0x7fb26415bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:11.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.049+0000 7fb268d2d640 1 --2- 172.21.15.88:0/2191094613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb264069ae0 0x7fb2640aadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:11.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.049+0000 7fb263fff640 1 -- 172.21.15.88:0/2191094613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb264069ae0 msgr2=0x7fb2640aadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:11.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.049+0000 7fb263fff640 1 --2- 172.21.15.88:0/2191094613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb264069ae0 0x7fb2640aadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:11.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.049+0000 7fb263fff640 1 -- 172.21.15.88:0/2191094613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb26415ec10 msgr2=0x7fb26415bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:11.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.049+0000 7fb263fff640 1 --2- 172.21.15.88:0/2191094613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb26415ec10 0x7fb26415bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:11.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.049+0000 7fb263fff640 1 -- 172.21.15.88:0/2191094613 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb264151720 con 0x7fb264071430 2024-09-16T01:39:11.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.049+0000 7fb26952e640 1 --2- 172.21.15.88:0/2191094613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb26415ec10 0x7fb26415bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:11.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.049+0000 7fb263fff640 1 --2- 172.21.15.88:0/2191094613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb264071430 0x7fb26415b5d0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fb25800b440 tx=0x7fb25800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:11.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.050+0000 7fb2617fa640 1 -- 172.21.15.88:0/2191094613 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb258015020 con 0x7fb264071430 2024-09-16T01:39:11.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.050+0000 7fb2617fa640 1 -- 172.21.15.88:0/2191094613 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2580118b0 con 0x7fb264071430 2024-09-16T01:39:11.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.050+0000 7fb26af91640 1 -- 172.21.15.88:0/2191094613 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2640af540 con 0x7fb264071430 2024-09-16T01:39:11.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.050+0000 7fb2617fa640 1 -- 172.21.15.88:0/2191094613 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb258011ca0 con 0x7fb264071430 2024-09-16T01:39:11.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.050+0000 7fb26af91640 1 -- 172.21.15.88:0/2191094613 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb264158710 con 0x7fb264071430 2024-09-16T01:39:11.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.051+0000 7fb26af91640 1 -- 172.21.15.88:0/2191094613 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2640b2d40 con 0x7fb264071430 2024-09-16T01:39:11.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.051+0000 7fb2617fa640 1 -- 172.21.15.88:0/2191094613 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb258018020 con 0x7fb264071430 2024-09-16T01:39:11.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.051+0000 7fb2617fa640 1 --2- 172.21.15.88:0/2191094613 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb25006c2b0 0x7fb25006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:11.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.052+0000 7fb2617fa640 1 -- 172.21.15.88:0/2191094613 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5115+0+0 (secure 0 0 0) 0x7fb25808b330 con 0x7fb264071430 2024-09-16T01:39:11.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.052+0000 7fb268d2d640 1 --2- 172.21.15.88:0/2191094613 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb25006c2b0 0x7fb25006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:11.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.052+0000 7fb268d2d640 1 --2- 172.21.15.88:0/2191094613 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb25006c2b0 0x7fb25006e770 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb254006fd0 tx=0x7fb254008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:11.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.056+0000 7fb2617fa640 1 -- 172.21.15.88:0/2191094613 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb2640b2d40 con 0x7fb264071430 2024-09-16T01:39:11.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.253+0000 7fb26af91640 1 -- 172.21.15.88:0/2191094613 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2640afa60 con 0x7fb264071430 2024-09-16T01:39:11.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.253+0000 7fb2617fa640 1 -- 172.21.15.88:0/2191094613 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+3914 (secure 0 0 0) 0x7fb25801f020 con 0x7fb264071430 2024-09-16T01:39:11.255 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:11.255 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":53,"btime":"2024-09-16T01:39:06:146356+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/581241194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":581241194},{"type":"v1","addr":"172.21.15.154:6834","nonce":581241194}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":5320,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2104508521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2104508521},{"type":"v1","addr":"172.21.15.154:6835","nonce":2104508521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34},{"gid":6195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2920926674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2920926674},{"type":"v1","addr":"172.21.15.88:6837","nonce":2920926674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":6321,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2119830489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2119830489},{"type":"v1","addr":"172.21.15.88:6836","nonce":2119830489}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":53}],"filesystems":[]} 2024-09-16T01:39:11.255 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 53 2024-09-16T01:39:11.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.255+0000 7fb26af91640 1 -- 172.21.15.88:0/2191094613 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb25006c2b0 msgr2=0x7fb25006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:11.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.255+0000 7fb26af91640 1 --2- 172.21.15.88:0/2191094613 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb25006c2b0 0x7fb25006e770 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb254006fd0 tx=0x7fb254008040 comp rx=0 tx=0).stop 2024-09-16T01:39:11.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.255+0000 7fb26af91640 1 -- 172.21.15.88:0/2191094613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb264071430 msgr2=0x7fb26415b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:11.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.255+0000 7fb26af91640 1 --2- 172.21.15.88:0/2191094613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb264071430 0x7fb26415b5d0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fb25800b440 tx=0x7fb25800b900 comp rx=0 tx=0).stop 2024-09-16T01:39:11.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.255+0000 7fb26af91640 1 -- 172.21.15.88:0/2191094613 shutdown_connections 2024-09-16T01:39:11.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.255+0000 7fb26af91640 1 --2- 172.21.15.88:0/2191094613 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb25006c2b0 0x7fb25006e770 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:11.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.255+0000 7fb26af91640 1 --2- 172.21.15.88:0/2191094613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb26415ec10 0x7fb26415bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:11.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.255+0000 7fb26af91640 1 --2- 172.21.15.88:0/2191094613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb264071430 0x7fb26415b5d0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:11.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.255+0000 7fb26af91640 1 --2- 172.21.15.88:0/2191094613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb264069ae0 0x7fb2640aadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:11.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.255+0000 7fb26af91640 1 -- 172.21.15.88:0/2191094613 >> 172.21.15.88:0/2191094613 conn(0x7fb2640b2430 msgr2=0x7fb2640b3830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:11.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.255+0000 7fb26af91640 1 -- 172.21.15.88:0/2191094613 shutdown_connections 2024-09-16T01:39:11.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.255+0000 7fb26af91640 1 -- 172.21.15.88:0/2191094613 wait complete. 2024-09-16T01:39:11.268 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-16T01:39:11.268 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-16T01:39:11.268 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-16T01:39:11.268 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:39:11.268 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-16T01:39:11.268 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-16T01:39:11.269 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-16T01:39:11.269 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:39:11.269 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-16T01:39:11.269 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-16T01:39:11.269 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-16T01:39:11.269 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:39:11.269 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-16T01:39:11.270 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-16T01:39:11.270 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-16T01:39:11.270 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:39:11.278 INFO:tasks.ceph.mds.a:Stopped 2024-09-16T01:39:11.278 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-16T01:39:11.279 INFO:tasks.ceph.mds.c:Stopped 2024-09-16T01:39:11.279 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-16T01:39:11.281 INFO:tasks.ceph.mds.b:Stopped 2024-09-16T01:39:11.281 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-16T01:39:11.282 INFO:tasks.ceph.mds.d:Stopped 2024-09-16T01:39:11.282 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-16T01:39:11.321 INFO:tasks.ceph.mds.a:Started 2024-09-16T01:39:11.321 DEBUG:teuthology.parallel:result is None 2024-09-16T01:39:11.321 INFO:tasks.ceph.mds.c:Started 2024-09-16T01:39:11.322 DEBUG:teuthology.parallel:result is None 2024-09-16T01:39:11.324 INFO:tasks.ceph.mds.b:Started 2024-09-16T01:39:11.325 DEBUG:teuthology.parallel:result is None 2024-09-16T01:39:11.325 INFO:tasks.ceph.mds.d:Started 2024-09-16T01:39:11.326 DEBUG:teuthology.parallel:result is None 2024-09-16T01:39:11.326 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-16T01:39:11.395 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:39:11.395 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:39:11.395 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:39:11.394+0000 7efd86ca0640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-09-16T01:39:11.409 INFO:tasks.ceph.mds.b.smithi154.stdout:starting mds.b at 2024-09-16T01:39:11.409 INFO:tasks.ceph.mds.d.smithi154.stdout:starting mds.d at 2024-09-16T01:39:11.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.428+0000 7ff83a62c640 1 Processor -- start 2024-09-16T01:39:11.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.428+0000 7ff83a62c640 1 -- start start 2024-09-16T01:39:11.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.429+0000 7ff83a62c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff83415e7a0 0x7ff83406caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:11.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.429+0000 7ff83a62c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8340cf860 con 0x7ff8340b68f0 2024-09-16T01:39:11.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.429+0000 7ff83a62c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8340cfb40 con 0x7ff8340cf1f0 2024-09-16T01:39:11.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.429+0000 7ff83a62c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff8340d2ca0 con 0x7ff83415e7a0 2024-09-16T01:39:11.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.429+0000 7ff838bc9640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff8340b68f0 0x7ff8340d3410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41992/0 (socket says 172.21.15.88:41992) 2024-09-16T01:39:11.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.430+0000 7ff838bc9640 1 -- 172.21.15.88:0/4265635210 learned_addr learned my addr 172.21.15.88:0/4265635210 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:11.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.430+0000 7ff833fff640 1 --2- 172.21.15.88:0/4265635210 >> v2:172.21.15.154:3300/0 conn(0x7ff83415e7a0 0x7ff83406caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:11.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.430+0000 7ff833fff640 1 -- 172.21.15.88:0/4265635210 >> v1:172.21.15.154:6789/0 conn(0x7ff8340cf1f0 legacy=0x7ff8340b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:11.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.430+0000 7ff833fff640 1 -- 172.21.15.88:0/4265635210 >> v1:172.21.15.88:6789/0 conn(0x7ff8340b68f0 legacy=0x7ff8340d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:11.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.430+0000 7ff833fff640 1 -- 172.21.15.88:0/4265635210 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8340cfe20 con 0x7ff83415e7a0 2024-09-16T01:39:11.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.431+0000 7ff833fff640 1 --2- 172.21.15.88:0/4265635210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff83415e7a0 0x7ff83406caa0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff8240097a0 tx=0x7ff82402a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d56f1ce70debfb31 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:11.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.433+0000 7ff8337fe640 1 -- 172.21.15.88:0/4265635210 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff824037070 con 0x7ff83415e7a0 2024-09-16T01:39:11.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.433+0000 7ff8337fe640 1 -- 172.21.15.88:0/4265635210 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff824032440 con 0x7ff83415e7a0 2024-09-16T01:39:11.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.433+0000 7ff8337fe640 1 -- 172.21.15.88:0/4265635210 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff824037070 con 0x7ff83415e7a0 2024-09-16T01:39:11.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.434+0000 7ff83a62c640 1 -- 172.21.15.88:0/4265635210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff83415e7a0 msgr2=0x7ff83406caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:11.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.434+0000 7ff83a62c640 1 --2- 172.21.15.88:0/4265635210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff83415e7a0 0x7ff83406caa0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff8240097a0 tx=0x7ff82402a1a0 comp rx=0 tx=0).stop 2024-09-16T01:39:11.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.434+0000 7ff83a62c640 1 -- 172.21.15.88:0/4265635210 shutdown_connections 2024-09-16T01:39:11.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.434+0000 7ff83a62c640 1 --2- 172.21.15.88:0/4265635210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff83415e7a0 0x7ff83406caa0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:11.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.434+0000 7ff83a62c640 1 -- 172.21.15.88:0/4265635210 >> 172.21.15.88:0/4265635210 conn(0x7ff8340d11e0 msgr2=0x7ff8340d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:11.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.434+0000 7ff83a62c640 1 -- 172.21.15.88:0/4265635210 shutdown_connections 2024-09-16T01:39:11.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.434+0000 7ff83a62c640 1 -- 172.21.15.88:0/4265635210 wait complete. 2024-09-16T01:39:11.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.435+0000 7ff83a62c640 1 Processor -- start 2024-09-16T01:39:11.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.435+0000 7ff83a62c640 1 -- start start 2024-09-16T01:39:11.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.436+0000 7ff83a62c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8340b68f0 0x7ff83414cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:11.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.436+0000 7ff83a62c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8340cf1f0 0x7ff8341614f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:11.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.436+0000 7ff83a62c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff83415e7a0 0x7ff83415d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:11.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.436+0000 7ff83a62c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff83406cfe0 con 0x7ff83415e7a0 2024-09-16T01:39:11.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.436+0000 7ff83a62c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff8340b7a00 con 0x7ff8340b68f0 2024-09-16T01:39:11.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.436+0000 7ff83a62c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff83415eb80 con 0x7ff8340cf1f0 2024-09-16T01:39:11.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.436+0000 7ff838bc9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff83415e7a0 0x7ff83415d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:11.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.436+0000 7ff838bc9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff83415e7a0 0x7ff83415d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37554/0 (socket says 172.21.15.88:37554) 2024-09-16T01:39:11.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.436+0000 7ff838bc9640 1 -- 172.21.15.88:0/2528169089 learned_addr learned my addr 172.21.15.88:0/2528169089 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:11.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.436+0000 7ff833fff640 1 --2- 172.21.15.88:0/2528169089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8340b68f0 0x7ff83414cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:11.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.436+0000 7ff82bfff640 1 --2- 172.21.15.88:0/2528169089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8340cf1f0 0x7ff8341614f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:11.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.437+0000 7ff838bc9640 1 -- 172.21.15.88:0/2528169089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8340cf1f0 msgr2=0x7ff8341614f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:11.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.437+0000 7ff838bc9640 1 --2- 172.21.15.88:0/2528169089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8340cf1f0 0x7ff8341614f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:11.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.437+0000 7ff838bc9640 1 -- 172.21.15.88:0/2528169089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8340b68f0 msgr2=0x7ff83414cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:11.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.437+0000 7ff838bc9640 1 --2- 172.21.15.88:0/2528169089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8340b68f0 0x7ff83414cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:11.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.437+0000 7ff838bc9640 1 -- 172.21.15.88:0/2528169089 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8340c8640 con 0x7ff83415e7a0 2024-09-16T01:39:11.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.437+0000 7ff838bc9640 1 --2- 172.21.15.88:0/2528169089 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff83415e7a0 0x7ff83415d240 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7ff82c00ca20 tx=0x7ff82c00cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:11.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.437+0000 7ff8317fa640 1 -- 172.21.15.88:0/2528169089 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c017070 con 0x7ff83415e7a0 2024-09-16T01:39:11.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.437+0000 7ff8317fa640 1 -- 172.21.15.88:0/2528169089 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff82c00ab70 con 0x7ff83415e7a0 2024-09-16T01:39:11.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.437+0000 7ff83a62c640 1 -- 172.21.15.88:0/2528169089 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff834161df0 con 0x7ff83415e7a0 2024-09-16T01:39:11.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.437+0000 7ff8317fa640 1 -- 172.21.15.88:0/2528169089 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c012400 con 0x7ff83415e7a0 2024-09-16T01:39:11.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.437+0000 7ff83a62c640 1 -- 172.21.15.88:0/2528169089 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8340c8bc0 con 0x7ff83415e7a0 2024-09-16T01:39:11.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.438+0000 7ff83a62c640 1 -- 172.21.15.88:0/2528169089 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff83406caa0 con 0x7ff83415e7a0 2024-09-16T01:39:11.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.438+0000 7ff8317fa640 1 -- 172.21.15.88:0/2528169089 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff82c01a070 con 0x7ff83415e7a0 2024-09-16T01:39:11.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.439+0000 7ff8317fa640 1 --2- 172.21.15.88:0/2528169089 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff81406c2b0 0x7ff81406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:11.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.439+0000 7ff833fff640 1 --2- 172.21.15.88:0/2528169089 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff81406c2b0 0x7ff81406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:11.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.439+0000 7ff8317fa640 1 -- 172.21.15.88:0/2528169089 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5115+0+0 (secure 0 0 0) 0x7ff82c08ad20 con 0x7ff83415e7a0 2024-09-16T01:39:11.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.439+0000 7ff833fff640 1 --2- 172.21.15.88:0/2528169089 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff81406c2b0 0x7ff81406e770 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff824034000 tx=0x7ff82402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:11.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.443+0000 7ff8317fa640 1 -- 172.21.15.88:0/2528169089 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff82c059090 con 0x7ff83415e7a0 2024-09-16T01:39:11.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:11.622+0000 7ff83a62c640 1 -- 172.21.15.88:0/2528169089 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7ff8340d2a80 con 0x7ff83415e7a0 2024-09-16T01:39:12.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.450+0000 7ff8317fa640 1 -- 172.21.15.88:0/2528169089 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v68) ==== 97+0+0 (secure 0 0 0) 0x7ff82c05d390 con 0x7ff83415e7a0 2024-09-16T01:39:12.451 INFO:teuthology.orchestra.run.smithi088.stderr: removed all blocklist entries 2024-09-16T01:39:12.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.452+0000 7ff83a62c640 1 -- 172.21.15.88:0/2528169089 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff81406c2b0 msgr2=0x7ff81406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:12.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.452+0000 7ff83a62c640 1 --2- 172.21.15.88:0/2528169089 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff81406c2b0 0x7ff81406e770 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff824034000 tx=0x7ff82402e000 comp rx=0 tx=0).stop 2024-09-16T01:39:12.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.452+0000 7ff83a62c640 1 -- 172.21.15.88:0/2528169089 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff83415e7a0 msgr2=0x7ff83415d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:12.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.452+0000 7ff83a62c640 1 --2- 172.21.15.88:0/2528169089 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff83415e7a0 0x7ff83415d240 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7ff82c00ca20 tx=0x7ff82c00cee0 comp rx=0 tx=0).stop 2024-09-16T01:39:12.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.452+0000 7ff83a62c640 1 -- 172.21.15.88:0/2528169089 shutdown_connections 2024-09-16T01:39:12.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.452+0000 7ff83a62c640 1 --2- 172.21.15.88:0/2528169089 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff81406c2b0 0x7ff81406e770 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:12.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.452+0000 7ff83a62c640 1 --2- 172.21.15.88:0/2528169089 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff83415e7a0 0x7ff83415d240 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:12.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.452+0000 7ff83a62c640 1 --2- 172.21.15.88:0/2528169089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8340cf1f0 0x7ff8341614f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:12.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.452+0000 7ff83a62c640 1 --2- 172.21.15.88:0/2528169089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8340b68f0 0x7ff83414cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:12.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.452+0000 7ff83a62c640 1 -- 172.21.15.88:0/2528169089 >> 172.21.15.88:0/2528169089 conn(0x7ff8340d11e0 msgr2=0x7ff8340d15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:12.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.452+0000 7ff83a62c640 1 -- 172.21.15.88:0/2528169089 shutdown_connections 2024-09-16T01:39:12.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.452+0000 7ff83a62c640 1 -- 172.21.15.88:0/2528169089 wait complete. 2024-09-16T01:39:12.465 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-16T01:39:12.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.563+0000 7f7b42923640 1 Processor -- start 2024-09-16T01:39:12.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.563+0000 7f7b42923640 1 -- start start 2024-09-16T01:39:12.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.564+0000 7f7b42923640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c0b74c0 0x7f7b3c15eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:12.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.564+0000 7f7b42923640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7b3c15f030 0x7f7b3c06d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:12.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.564+0000 7f7b42923640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b3c15fb10 con 0x7f7b3c15c2b0 2024-09-16T01:39:12.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.564+0000 7f7b42923640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7b3c0b5900 con 0x7f7b3c15f030 2024-09-16T01:39:12.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.564+0000 7f7b42923640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7b3c15f830 con 0x7f7b3c0b74c0 2024-09-16T01:39:12.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.564+0000 7f7b3bfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7b3c15c2b0 0x7f7b3c0b6db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41998/0 (socket says 172.21.15.88:41998) 2024-09-16T01:39:12.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.564+0000 7f7b3bfff640 1 -- 172.21.15.88:0/3135703887 learned_addr learned my addr 172.21.15.88:0/3135703887 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:12.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.564+0000 7f7b3b7fe640 1 --2- 172.21.15.88:0/3135703887 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c0b74c0 0x7f7b3c15eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:12.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.564+0000 7f7b40ec0640 1 --2- 172.21.15.88:0/3135703887 >> v2:172.21.15.154:3300/0 conn(0x7f7b3c15f030 0x7f7b3c06d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:12.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.564+0000 7f7b3affd640 1 -- 172.21.15.88:0/3135703887 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2267794047 0 0) 0x7f7b3c15fb10 con 0x7f7b3c15c2b0 2024-09-16T01:39:12.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.564+0000 7f7b3affd640 1 -- 172.21.15.88:0/3135703887 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b24003520 con 0x7f7b3c15c2b0 2024-09-16T01:39:12.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.564+0000 7f7b3affd640 1 -- 172.21.15.88:0/3135703887 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3255928996 0 0) 0x7f7b24003520 con 0x7f7b3c15c2b0 2024-09-16T01:39:12.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.564+0000 7f7b3affd640 1 -- 172.21.15.88:0/3135703887 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c0b74c0 msgr2=0x7f7b3c15eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:12.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.564+0000 7f7b3affd640 1 --2- 172.21.15.88:0/3135703887 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c0b74c0 0x7f7b3c15eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:12.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.564+0000 7f7b3affd640 1 -- 172.21.15.88:0/3135703887 >> v2:172.21.15.154:3300/0 conn(0x7f7b3c15f030 msgr2=0x7f7b3c06d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:12.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.564+0000 7f7b3affd640 1 --2- 172.21.15.88:0/3135703887 >> v2:172.21.15.154:3300/0 conn(0x7f7b3c15f030 0x7f7b3c06d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:12.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.564+0000 7f7b3affd640 1 -- 172.21.15.88:0/3135703887 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b3c06ce30 con 0x7f7b3c15c2b0 2024-09-16T01:39:12.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.564+0000 7f7b3affd640 1 -- 172.21.15.88:0/3135703887 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f7b2c001670 con 0x7f7b3c15c2b0 2024-09-16T01:39:12.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.564+0000 7f7b3affd640 1 -- 172.21.15.88:0/3135703887 >> v1:172.21.15.88:6789/0 conn(0x7f7b3c15c2b0 legacy=0x7f7b3c0b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:12.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.564+0000 7f7b3affd640 1 --2- 172.21.15.88:0/3135703887 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c0b74c0 0x7f7b240039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:12.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.564+0000 7f7b3affd640 1 --2- 172.21.15.88:0/3135703887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b24003f30 0x7f7b24004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:12.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.564+0000 7f7b3affd640 1 --2- 172.21.15.88:0/3135703887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b24004d40 0x7f7b240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:12.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.564+0000 7f7b3affd640 1 -- 172.21.15.88:0/3135703887 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3c0b5900 con 0x7f7b24003f30 2024-09-16T01:39:12.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.564+0000 7f7b3affd640 1 -- 172.21.15.88:0/3135703887 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3c15f830 con 0x7f7b24004d40 2024-09-16T01:39:12.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.564+0000 7f7b3affd640 1 -- 172.21.15.88:0/3135703887 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7b2400b9c0 con 0x7f7b3c0b74c0 2024-09-16T01:39:12.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.565+0000 7f7b3b7fe640 1 --2- 172.21.15.88:0/3135703887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b24003f30 0x7f7b24004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:12.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.565+0000 7f7b3bfff640 1 --2- 172.21.15.88:0/3135703887 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c0b74c0 0x7f7b240039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:12.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.565+0000 7f7b40ec0640 1 --2- 172.21.15.88:0/3135703887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b24004d40 0x7f7b240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:12.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.565+0000 7f7b3b7fe640 1 -- 172.21.15.88:0/3135703887 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c0b74c0 msgr2=0x7f7b240039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:12.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.565+0000 7f7b3b7fe640 1 --2- 172.21.15.88:0/3135703887 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c0b74c0 0x7f7b240039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:12.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.565+0000 7f7b3b7fe640 1 -- 172.21.15.88:0/3135703887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b24004d40 msgr2=0x7f7b240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:12.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.565+0000 7f7b3b7fe640 1 --2- 172.21.15.88:0/3135703887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b24004d40 0x7f7b240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:12.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.565+0000 7f7b3b7fe640 1 -- 172.21.15.88:0/3135703887 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b24003520 con 0x7f7b24003f30 2024-09-16T01:39:12.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.565+0000 7f7b3b7fe640 1 --2- 172.21.15.88:0/3135703887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b24003f30 0x7f7b24004800 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f7b30009ac0 tx=0x7f7b24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce6452057efbea49 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:12.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.566+0000 7f7b3affd640 1 -- 172.21.15.88:0/3135703887 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b30036070 con 0x7f7b24003f30 2024-09-16T01:39:12.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.566+0000 7f7b3affd640 1 -- 172.21.15.88:0/3135703887 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b30031400 con 0x7f7b24003f30 2024-09-16T01:39:12.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.566+0000 7f7b3affd640 1 -- 172.21.15.88:0/3135703887 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b30009690 con 0x7f7b24003f30 2024-09-16T01:39:12.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.566+0000 7f7b42923640 1 -- 172.21.15.88:0/3135703887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b24003f30 msgr2=0x7f7b24004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:12.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.566+0000 7f7b42923640 1 --2- 172.21.15.88:0/3135703887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b24003f30 0x7f7b24004800 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f7b30009ac0 tx=0x7f7b24005830 comp rx=0 tx=0).stop 2024-09-16T01:39:12.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.566+0000 7f7b40ec0640 1 -- 172.21.15.88:0/3135703887 reap_dead start 2024-09-16T01:39:12.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.566+0000 7f7b42923640 1 -- 172.21.15.88:0/3135703887 shutdown_connections 2024-09-16T01:39:12.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.566+0000 7f7b42923640 1 -- 172.21.15.88:0/3135703887 >> 172.21.15.88:0/3135703887 conn(0x7f7b3c0b4240 msgr2=0x7f7b3c0b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:12.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.566+0000 7f7b42923640 1 -- 172.21.15.88:0/3135703887 shutdown_connections 2024-09-16T01:39:12.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.566+0000 7f7b42923640 1 -- 172.21.15.88:0/3135703887 wait complete. 2024-09-16T01:39:12.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.567+0000 7f7b42923640 1 Processor -- start 2024-09-16T01:39:12.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.568+0000 7f7b42923640 1 -- start start 2024-09-16T01:39:12.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.568+0000 7f7b42923640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b24003f30 0x7f7b3c1e88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:12.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.568+0000 7f7b42923640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c1e7ec0 0x7f7b3c1e8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:12.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.568+0000 7f7b42923640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b3c158330 0x7f7b3c1e6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:12.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.568+0000 7f7b42923640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3c15f4c0 con 0x7f7b3c158330 2024-09-16T01:39:12.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.568+0000 7f7b42923640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3c06d3f0 con 0x7f7b24003f30 2024-09-16T01:39:12.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.568+0000 7f7b42923640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7b3c15eaf0 con 0x7f7b3c1e7ec0 2024-09-16T01:39:12.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.568+0000 7f7b40ec0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b3c158330 0x7f7b3c1e6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:12.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.568+0000 7f7b40ec0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b3c158330 0x7f7b3c1e6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37578/0 (socket says 172.21.15.88:37578) 2024-09-16T01:39:12.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.568+0000 7f7b40ec0640 1 -- 172.21.15.88:0/3340659518 learned_addr learned my addr 172.21.15.88:0/3340659518 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:12.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.569+0000 7f7b3bfff640 1 --2- 172.21.15.88:0/3340659518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b24003f30 0x7f7b3c1e88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:12.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.569+0000 7f7b3b7fe640 1 --2- 172.21.15.88:0/3340659518 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c1e7ec0 0x7f7b3c1e8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:12.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.569+0000 7f7b40ec0640 1 -- 172.21.15.88:0/3340659518 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c1e7ec0 msgr2=0x7f7b3c1e8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:12.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.569+0000 7f7b40ec0640 1 --2- 172.21.15.88:0/3340659518 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c1e7ec0 0x7f7b3c1e8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:12.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.569+0000 7f7b40ec0640 1 -- 172.21.15.88:0/3340659518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b24003f30 msgr2=0x7f7b3c1e88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:12.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.569+0000 7f7b40ec0640 1 --2- 172.21.15.88:0/3340659518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b24003f30 0x7f7b3c1e88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:12.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.569+0000 7f7b40ec0640 1 -- 172.21.15.88:0/3340659518 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b3c159a00 con 0x7f7b3c158330 2024-09-16T01:39:12.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.569+0000 7f7b3b7fe640 1 --2- 172.21.15.88:0/3340659518 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c1e7ec0 0x7f7b3c1e8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:12.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.569+0000 7f7b3bfff640 1 --2- 172.21.15.88:0/3340659518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b24003f30 0x7f7b3c1e88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:12.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.569+0000 7f7b40ec0640 1 --2- 172.21.15.88:0/3340659518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b3c158330 0x7f7b3c1e6dd0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c0b5fa0 tx=0x7f7b3400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:12.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.569+0000 7f7b38ff9640 1 -- 172.21.15.88:0/3340659518 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3400fe10 con 0x7f7b3c158330 2024-09-16T01:39:12.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.569+0000 7f7b38ff9640 1 -- 172.21.15.88:0/3340659518 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b34015430 con 0x7f7b3c158330 2024-09-16T01:39:12.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.569+0000 7f7b42923640 1 -- 172.21.15.88:0/3340659518 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b3c157850 con 0x7f7b3c158330 2024-09-16T01:39:12.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.569+0000 7f7b38ff9640 1 -- 172.21.15.88:0/3340659518 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b34015820 con 0x7f7b3c158330 2024-09-16T01:39:12.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.569+0000 7f7b42923640 1 -- 172.21.15.88:0/3340659518 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b3c159590 con 0x7f7b3c158330 2024-09-16T01:39:12.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.570+0000 7f7b42923640 1 -- 172.21.15.88:0/3340659518 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b3c0b6db0 con 0x7f7b3c158330 2024-09-16T01:39:12.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.570+0000 7f7b38ff9640 1 -- 172.21.15.88:0/3340659518 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7b340159c0 con 0x7f7b3c158330 2024-09-16T01:39:12.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.571+0000 7f7b38ff9640 1 --2- 172.21.15.88:0/3340659518 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7b2806c2e0 0x7f7b2806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:12.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.571+0000 7f7b38ff9640 1 -- 172.21.15.88:0/3340659518 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 4857+0+0 (secure 0 0 0) 0x7f7b3408bf90 con 0x7f7b3c158330 2024-09-16T01:39:12.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.571+0000 7f7b3bfff640 1 --2- 172.21.15.88:0/3340659518 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7b2806c2e0 0x7f7b2806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:12.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.571+0000 7f7b3bfff640 1 --2- 172.21.15.88:0/3340659518 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7b2806c2e0 0x7f7b2806e7a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c001050 tx=0x7f7b2c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:12.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.575+0000 7f7b38ff9640 1 -- 172.21.15.88:0/3340659518 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7b34019030 con 0x7f7b3c158330 2024-09-16T01:39:12.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.804+0000 7f7b42923640 1 -- 172.21.15.88:0/3340659518 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f7b3c0b38d0 con 0x7f7b3c158330 2024-09-16T01:39:12.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.804+0000 7f7b38ff9640 1 -- 172.21.15.88:0/3340659518 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) ==== 80+0+5304 (secure 0 0 0) 0x7f7b34010040 con 0x7f7b3c158330 2024-09-16T01:39:12.806 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:12.806 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:39:12.806 INFO:teuthology.orchestra.run.smithi088.stdout: "auth_dump": [ 2024-09-16T01:39:12.806 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:39:12.807 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mds.a", 2024-09-16T01:39:12.807 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmy+jJCRAAhnzX/li6D3D07NmThrjXkA==", 2024-09-16T01:39:12.807 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:39:12.807 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:39:12.807 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:39:12.807 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:39:12.807 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:39:12.807 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:39:12.807 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:39:12.807 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:39:12.807 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mds.b", 2024-09-16T01:39:12.808 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmx7p+FRAApxKUSZoR/nGUZsaYKlrYUg==", 2024-09-16T01:39:12.808 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:39:12.808 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:39:12.808 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:39:12.808 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:39:12.808 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:39:12.808 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:39:12.809 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:39:12.809 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:39:12.809 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mds.c", 2024-09-16T01:39:12.809 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmaYHUEBAAzM0qE53LEgHi+hThHJX3Tw==", 2024-09-16T01:39:12.809 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:39:12.809 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:39:12.809 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:39:12.809 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:39:12.809 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:39:12.810 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:39:12.810 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:39:12.810 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:39:12.810 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mds.d", 2024-09-16T01:39:12.810 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudm3TSLHBAAJzlR6H09cc1Pki2FjyoFiA==", 2024-09-16T01:39:12.810 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:39:12.810 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:39:12.810 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:39:12.810 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:39:12.810 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:39:12.810 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:39:12.810 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:39:12.811 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:39:12.811 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.0", 2024-09-16T01:39:12.811 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDhiudmZR7MORAAwDKvphubA+CbzBUWuaA80A==", 2024-09-16T01:39:12.811 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:39:12.811 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:39:12.811 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:39:12.811 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:39:12.811 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:39:12.811 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:39:12.811 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:39:12.811 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.1", 2024-09-16T01:39:12.811 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDliudmNd+SLBAArOUdoZ226c5jlAsKwqERmw==", 2024-09-16T01:39:12.811 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:39:12.811 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:39:12.812 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:39:12.812 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:39:12.812 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:39:12.812 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:39:12.812 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:39:12.812 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.2", 2024-09-16T01:39:12.812 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDpiudm75CUARAAWHtYOg9pE8g+M6IL+8XZ6g==", 2024-09-16T01:39:12.812 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:39:12.812 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:39:12.813 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:39:12.813 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:39:12.813 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:39:12.813 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:39:12.813 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:39:12.813 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.3", 2024-09-16T01:39:12.813 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDtiudm0fgWAxAAf2NOWrpXbOhb3dFXozJagQ==", 2024-09-16T01:39:12.813 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:39:12.813 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:39:12.813 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:39:12.813 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:39:12.813 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:39:12.814 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:39:12.814 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:39:12.814 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.4", 2024-09-16T01:39:12.814 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQD0iudmdddMCxAAtr2qfj4Nxj3/GHUpxRmtFg==", 2024-09-16T01:39:12.814 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:39:12.814 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:39:12.814 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:39:12.814 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:39:12.814 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:39:12.814 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:39:12.814 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:39:12.814 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.5", 2024-09-16T01:39:12.814 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQD3iudmT5RjOhAAcKwca6BG60gsxYnmy4twGg==", 2024-09-16T01:39:12.815 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:39:12.815 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:39:12.815 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:39:12.815 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:39:12.815 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:39:12.815 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:39:12.815 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:39:12.815 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.6", 2024-09-16T01:39:12.815 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQD7iudmT7daLRAAdfLFijKQku3rJcJ5xcfLsw==", 2024-09-16T01:39:12.815 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:39:12.815 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:39:12.816 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:39:12.816 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:39:12.816 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:39:12.816 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:39:12.816 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:39:12.816 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.7", 2024-09-16T01:39:12.816 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQD/iudmmvtxLhAArYKhQ4zyFHtLgh/+bs5ZhA==", 2024-09-16T01:39:12.816 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:39:12.816 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:39:12.816 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:39:12.816 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:39:12.817 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:39:12.817 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:39:12.817 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:39:12.817 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.0", 2024-09-16T01:39:12.817 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmvPikIBAAyHxNNnwQKygxg09PNQLc/Q==", 2024-09-16T01:39:12.817 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:39:12.817 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:39:12.817 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow r", 2024-09-16T01:39:12.817 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-16T01:39:12.817 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:39:12.817 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:39:12.817 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:39:12.817 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.1", 2024-09-16T01:39:12.818 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmA69iKRAA8Abwe0Lg2PnZDErgJ1gEbQ==", 2024-09-16T01:39:12.818 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:39:12.818 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:39:12.818 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow r", 2024-09-16T01:39:12.818 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow rw", 2024-09-16T01:39:12.818 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow rwx" 2024-09-16T01:39:12.818 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:39:12.818 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:39:12.818 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:39:12.818 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.2", 2024-09-16T01:39:12.818 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmB+w9JhAAv63Is5T/cvDbbLOjIkI0wA==", 2024-09-16T01:39:12.818 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:39:12.818 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:39:12.819 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow r", 2024-09-16T01:39:12.819 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow rw", 2024-09-16T01:39:12.819 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow rwx" 2024-09-16T01:39:12.819 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:39:12.819 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:39:12.819 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:39:12.819 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.3", 2024-09-16T01:39:12.819 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudm4WoFLxAAnYGHGqHyBHHajQ8f3rCjEA==", 2024-09-16T01:39:12.819 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:39:12.819 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:39:12.820 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow r", 2024-09-16T01:39:12.820 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow rw", 2024-09-16T01:39:12.820 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow rwx" 2024-09-16T01:39:12.820 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:39:12.820 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:39:12.820 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:39:12.820 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.admin", 2024-09-16T01:39:12.820 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDdiudmhcE0KBAANauQKyUY01/KlWMrN2x/5Q==", 2024-09-16T01:39:12.820 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:39:12.820 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow *", 2024-09-16T01:39:12.821 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:39:12.821 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:39:12.821 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:39:12.821 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:39:12.821 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:39:12.821 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:39:12.821 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mgr.x", 2024-09-16T01:39:12.821 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmToGYBhAAKwXS34PtfAVtEtdtHKHlQQ==", 2024-09-16T01:39:12.821 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:39:12.821 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow *", 2024-09-16T01:39:12.821 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile mgr", 2024-09-16T01:39:12.821 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:39:12.821 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:39:12.822 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:39:12.822 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:39:12.822 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mgr.y", 2024-09-16T01:39:12.822 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudm/BN5ARAAxbCopbsvxFUZP3drso32qw==", 2024-09-16T01:39:12.822 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:39:12.822 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow *", 2024-09-16T01:39:12.822 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile mgr", 2024-09-16T01:39:12.822 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:39:12.822 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:39:12.822 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:39:12.822 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-09-16T01:39:12.822 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:39:12.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.806+0000 7f7b42923640 1 -- 172.21.15.88:0/3340659518 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7b2806c2e0 msgr2=0x7f7b2806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:12.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.806+0000 7f7b42923640 1 --2- 172.21.15.88:0/3340659518 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7b2806c2e0 0x7f7b2806e7a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c001050 tx=0x7f7b2c009040 comp rx=0 tx=0).stop 2024-09-16T01:39:12.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.806+0000 7f7b42923640 1 -- 172.21.15.88:0/3340659518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b3c158330 msgr2=0x7f7b3c1e6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:12.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.806+0000 7f7b42923640 1 --2- 172.21.15.88:0/3340659518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b3c158330 0x7f7b3c1e6dd0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c0b5fa0 tx=0x7f7b3400f580 comp rx=0 tx=0).stop 2024-09-16T01:39:12.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.806+0000 7f7b42923640 1 -- 172.21.15.88:0/3340659518 shutdown_connections 2024-09-16T01:39:12.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.806+0000 7f7b42923640 1 --2- 172.21.15.88:0/3340659518 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7b2806c2e0 0x7f7b2806e7a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:12.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.806+0000 7f7b42923640 1 --2- 172.21.15.88:0/3340659518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b3c158330 0x7f7b3c1e6dd0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:12.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.806+0000 7f7b42923640 1 --2- 172.21.15.88:0/3340659518 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b3c1e7ec0 0x7f7b3c1e8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:12.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.806+0000 7f7b42923640 1 --2- 172.21.15.88:0/3340659518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b24003f30 0x7f7b3c1e88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:12.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.806+0000 7f7b42923640 1 -- 172.21.15.88:0/3340659518 >> 172.21.15.88:0/3340659518 conn(0x7f7b3c0b4240 msgr2=0x7f7b3c06d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:12.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.806+0000 7f7b42923640 1 -- 172.21.15.88:0/3340659518 shutdown_connections 2024-09-16T01:39:12.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.806+0000 7f7b42923640 1 -- 172.21.15.88:0/3340659518 wait complete. 2024-09-16T01:39:12.825 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-16T01:39:12.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.921+0000 7f6272853640 1 Processor -- start 2024-09-16T01:39:12.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.921+0000 7f6272853640 1 -- start start 2024-09-16T01:39:12.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.922+0000 7f6272853640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c15a9c0 0x7f626c15e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:12.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.922+0000 7f6272853640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f626c0b5be0 0x7f626c15eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:12.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.922+0000 7f6272853640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f626c0b6830 con 0x7f626c0b29a0 2024-09-16T01:39:12.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.922+0000 7f6272853640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f626c15f410 con 0x7f626c0b5be0 2024-09-16T01:39:12.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.922+0000 7f6272853640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f626c0cd0b0 con 0x7f626c15a9c0 2024-09-16T01:39:12.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.922+0000 7f6270df0640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f626c0b29a0 0x7f626c0b3710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42012/0 (socket says 172.21.15.88:42012) 2024-09-16T01:39:12.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.922+0000 7f6270df0640 1 -- 172.21.15.88:0/895453968 learned_addr learned my addr 172.21.15.88:0/895453968 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:12.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.922+0000 7f626b7fe640 1 --2- 172.21.15.88:0/895453968 >> v2:172.21.15.154:3300/0 conn(0x7f626c0b5be0 0x7f626c15eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:12.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.922+0000 7f626bfff640 1 --2- 172.21.15.88:0/895453968 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c15a9c0 0x7f626c15e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:12.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.922+0000 7f626affd640 1 -- 172.21.15.88:0/895453968 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3202605117 0 0) 0x7f626c0b6830 con 0x7f626c0b29a0 2024-09-16T01:39:12.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.922+0000 7f626affd640 1 -- 172.21.15.88:0/895453968 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6254003520 con 0x7f626c0b29a0 2024-09-16T01:39:12.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.922+0000 7f626affd640 1 -- 172.21.15.88:0/895453968 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 354215307 0 0) 0x7f6254003520 con 0x7f626c0b29a0 2024-09-16T01:39:12.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.922+0000 7f626affd640 1 -- 172.21.15.88:0/895453968 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c15a9c0 msgr2=0x7f626c15e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:12.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.922+0000 7f626affd640 1 --2- 172.21.15.88:0/895453968 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c15a9c0 0x7f626c15e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.922+0000 7f626affd640 1 -- 172.21.15.88:0/895453968 >> v2:172.21.15.154:3300/0 conn(0x7f626c0b5be0 msgr2=0x7f626c15eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.922+0000 7f626affd640 1 --2- 172.21.15.88:0/895453968 >> v2:172.21.15.154:3300/0 conn(0x7f626c0b5be0 0x7f626c15eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.923+0000 7f626affd640 1 -- 172.21.15.88:0/895453968 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f626c0cd390 con 0x7f626c0b29a0 2024-09-16T01:39:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.923+0000 7f626b7fe640 1 --2- 172.21.15.88:0/895453968 >> v2:172.21.15.154:3300/0 conn(0x7f626c0b5be0 0x7f626c15eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:39:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.923+0000 7f626affd640 1 -- 172.21.15.88:0/895453968 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f62640019f0 con 0x7f626c0b29a0 2024-09-16T01:39:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.923+0000 7f626affd640 1 -- 172.21.15.88:0/895453968 >> v1:172.21.15.88:6789/0 conn(0x7f626c0b29a0 legacy=0x7f626c0b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.923+0000 7f626affd640 1 --2- 172.21.15.88:0/895453968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62540039b0 0x7f6254003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.923+0000 7f626affd640 1 --2- 172.21.15.88:0/895453968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6254004800 0x7f6254004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.923+0000 7f626affd640 1 --2- 172.21.15.88:0/895453968 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c15a9c0 0x7f62540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.923+0000 7f626affd640 1 -- 172.21.15.88:0/895453968 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f626c15f410 con 0x7f62540039b0 2024-09-16T01:39:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.923+0000 7f626affd640 1 -- 172.21.15.88:0/895453968 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f626c0cd0b0 con 0x7f6254004800 2024-09-16T01:39:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.923+0000 7f626affd640 1 -- 172.21.15.88:0/895453968 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f625400b9c0 con 0x7f626c15a9c0 2024-09-16T01:39:12.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.923+0000 7f626bfff640 1 --2- 172.21.15.88:0/895453968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62540039b0 0x7f6254003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:12.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.923+0000 7f626b7fe640 1 --2- 172.21.15.88:0/895453968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6254004800 0x7f6254004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:12.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.923+0000 7f6270df0640 1 --2- 172.21.15.88:0/895453968 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c15a9c0 0x7f62540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:12.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.923+0000 7f626bfff640 1 -- 172.21.15.88:0/895453968 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c15a9c0 msgr2=0x7f62540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:12.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.923+0000 7f626bfff640 1 --2- 172.21.15.88:0/895453968 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c15a9c0 0x7f62540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:12.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.923+0000 7f626bfff640 1 -- 172.21.15.88:0/895453968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6254004800 msgr2=0x7f6254004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:12.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.923+0000 7f626bfff640 1 --2- 172.21.15.88:0/895453968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6254004800 0x7f6254004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:12.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.923+0000 7f626bfff640 1 -- 172.21.15.88:0/895453968 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6254003520 con 0x7f62540039b0 2024-09-16T01:39:12.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.923+0000 7f6270df0640 1 --2- 172.21.15.88:0/895453968 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c15a9c0 0x7f62540051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:12.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.924+0000 7f626bfff640 1 --2- 172.21.15.88:0/895453968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62540039b0 0x7f6254003dd0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f625c009ac0 tx=0x7f6254005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e14772972e4e1ddd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:12.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.924+0000 7f626affd640 1 -- 172.21.15.88:0/895453968 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625c036070 con 0x7f62540039b0 2024-09-16T01:39:12.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.924+0000 7f626affd640 1 -- 172.21.15.88:0/895453968 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f625c031400 con 0x7f62540039b0 2024-09-16T01:39:12.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.924+0000 7f626affd640 1 -- 172.21.15.88:0/895453968 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625c009690 con 0x7f62540039b0 2024-09-16T01:39:12.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.924+0000 7f6272853640 1 -- 172.21.15.88:0/895453968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62540039b0 msgr2=0x7f6254003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:12.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.924+0000 7f6272853640 1 --2- 172.21.15.88:0/895453968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62540039b0 0x7f6254003dd0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f625c009ac0 tx=0x7f6254005830 comp rx=0 tx=0).stop 2024-09-16T01:39:12.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.924+0000 7f6270df0640 1 -- 172.21.15.88:0/895453968 reap_dead start 2024-09-16T01:39:12.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.924+0000 7f6272853640 1 -- 172.21.15.88:0/895453968 shutdown_connections 2024-09-16T01:39:12.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.924+0000 7f6272853640 1 -- 172.21.15.88:0/895453968 >> 172.21.15.88:0/895453968 conn(0x7f626c06bc10 msgr2=0x7f626c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:12.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.924+0000 7f6272853640 1 -- 172.21.15.88:0/895453968 shutdown_connections 2024-09-16T01:39:12.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.924+0000 7f6272853640 1 -- 172.21.15.88:0/895453968 wait complete. 2024-09-16T01:39:12.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.926+0000 7f6272853640 1 Processor -- start 2024-09-16T01:39:12.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.926+0000 7f6272853640 1 -- start start 2024-09-16T01:39:12.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.926+0000 7f6272853640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62540039b0 0x7f626c14a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:12.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.926+0000 7f6272853640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c15d1a0 0x7f626c15c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:12.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.926+0000 7f6272853640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f626c14f380 0x7f626c14f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:12.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.926+0000 7f6272853640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f626c15ada0 con 0x7f62540039b0 2024-09-16T01:39:12.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.926+0000 7f6272853640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f626c0cd950 con 0x7f626c14f380 2024-09-16T01:39:12.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.926+0000 7f6272853640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f626c15f080 con 0x7f626c15d1a0 2024-09-16T01:39:12.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.927+0000 7f626bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62540039b0 0x7f626c14a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:12.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.927+0000 7f626bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62540039b0 0x7f626c14a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37592/0 (socket says 172.21.15.88:37592) 2024-09-16T01:39:12.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.927+0000 7f626bfff640 1 -- 172.21.15.88:0/908356513 learned_addr learned my addr 172.21.15.88:0/908356513 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:12.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.927+0000 7f6270df0640 1 --2- 172.21.15.88:0/908356513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f626c14f380 0x7f626c14f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:12.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.927+0000 7f626b7fe640 1 --2- 172.21.15.88:0/908356513 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c15d1a0 0x7f626c15c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:12.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.927+0000 7f626bfff640 1 -- 172.21.15.88:0/908356513 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c15d1a0 msgr2=0x7f626c15c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:12.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.927+0000 7f626bfff640 1 --2- 172.21.15.88:0/908356513 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c15d1a0 0x7f626c15c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:12.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.927+0000 7f626bfff640 1 -- 172.21.15.88:0/908356513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f626c14f380 msgr2=0x7f626c14f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:12.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.927+0000 7f626bfff640 1 --2- 172.21.15.88:0/908356513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f626c14f380 0x7f626c14f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:12.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.927+0000 7f626bfff640 1 -- 172.21.15.88:0/908356513 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f626c15d620 con 0x7f62540039b0 2024-09-16T01:39:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.927+0000 7f626bfff640 1 --2- 172.21.15.88:0/908356513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62540039b0 0x7f626c14a7e0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f625c009be0 tx=0x7f625c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.927+0000 7f6268ff9640 1 -- 172.21.15.88:0/908356513 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625c040020 con 0x7f62540039b0 2024-09-16T01:39:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.927+0000 7f6268ff9640 1 -- 172.21.15.88:0/908356513 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f625c003bc0 con 0x7f62540039b0 2024-09-16T01:39:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.927+0000 7f6272853640 1 -- 172.21.15.88:0/908356513 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f626c15d8a0 con 0x7f62540039b0 2024-09-16T01:39:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.927+0000 7f6268ff9640 1 -- 172.21.15.88:0/908356513 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625c036040 con 0x7f62540039b0 2024-09-16T01:39:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.927+0000 7f6272853640 1 -- 172.21.15.88:0/908356513 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f626c149d00 con 0x7f62540039b0 2024-09-16T01:39:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.928+0000 7f6272853640 1 -- 172.21.15.88:0/908356513 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f626c06de40 con 0x7f62540039b0 2024-09-16T01:39:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.929+0000 7f6268ff9640 1 -- 172.21.15.88:0/908356513 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f625c043020 con 0x7f62540039b0 2024-09-16T01:39:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.929+0000 7f6268ff9640 1 --2- 172.21.15.88:0/908356513 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f625806c2b0 0x7f625806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.929+0000 7f6268ff9640 1 -- 172.21.15.88:0/908356513 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 4857+0+0 (secure 0 0 0) 0x7f625c0ab230 con 0x7f62540039b0 2024-09-16T01:39:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.929+0000 7f626b7fe640 1 --2- 172.21.15.88:0/908356513 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f625806c2b0 0x7f625806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.930+0000 7f626b7fe640 1 --2- 172.21.15.88:0/908356513 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f625806c2b0 0x7f625806e770 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f626c0cc4f0 tx=0x7f6260008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:12.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:12.933+0000 7f6268ff9640 1 -- 172.21.15.88:0/908356513 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f625c04a070 con 0x7f62540039b0 2024-09-16T01:39:13.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.100+0000 7f6272853640 1 -- 172.21.15.88:0/908356513 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f626c15b130 con 0x7f62540039b0 2024-09-16T01:39:13.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.101+0000 7f6268ff9640 1 -- 172.21.15.88:0/908356513 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v68) ==== 84+0+65 (secure 0 0 0) 0x7f626c15d620 con 0x7f62540039b0 2024-09-16T01:39:13.102 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:13.103 INFO:teuthology.orchestra.run.smithi088.stdout:[ 2024-09-16T01:39:13.103 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:39:13.103 INFO:teuthology.orchestra.run.smithi088.stdout: "poolnum": 1, 2024-09-16T01:39:13.103 INFO:teuthology.orchestra.run.smithi088.stdout: "poolname": ".mgr" 2024-09-16T01:39:13.103 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:39:13.103 INFO:teuthology.orchestra.run.smithi088.stdout:] 2024-09-16T01:39:13.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.103+0000 7f6272853640 1 -- 172.21.15.88:0/908356513 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f625806c2b0 msgr2=0x7f625806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:13.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.103+0000 7f6272853640 1 --2- 172.21.15.88:0/908356513 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f625806c2b0 0x7f625806e770 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f626c0cc4f0 tx=0x7f6260008040 comp rx=0 tx=0).stop 2024-09-16T01:39:13.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.103+0000 7f6272853640 1 -- 172.21.15.88:0/908356513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62540039b0 msgr2=0x7f626c14a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:13.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.103+0000 7f6272853640 1 --2- 172.21.15.88:0/908356513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62540039b0 0x7f626c14a7e0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f625c009be0 tx=0x7f625c004730 comp rx=0 tx=0).stop 2024-09-16T01:39:13.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.103+0000 7f6272853640 1 -- 172.21.15.88:0/908356513 shutdown_connections 2024-09-16T01:39:13.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.103+0000 7f6272853640 1 --2- 172.21.15.88:0/908356513 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f625806c2b0 0x7f625806e770 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:13.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.103+0000 7f6272853640 1 --2- 172.21.15.88:0/908356513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f626c14f380 0x7f626c14f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:13.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.103+0000 7f6272853640 1 --2- 172.21.15.88:0/908356513 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c15d1a0 0x7f626c15c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:13.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.103+0000 7f6272853640 1 --2- 172.21.15.88:0/908356513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62540039b0 0x7f626c14a7e0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:13.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.103+0000 7f6272853640 1 -- 172.21.15.88:0/908356513 >> 172.21.15.88:0/908356513 conn(0x7f626c06bc10 msgr2=0x7f626c071e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:13.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.103+0000 7f6272853640 1 -- 172.21.15.88:0/908356513 shutdown_connections 2024-09-16T01:39:13.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.103+0000 7f6272853640 1 -- 172.21.15.88:0/908356513 wait complete. 2024-09-16T01:39:13.115 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-16T01:39:13.115 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-16T01:39:13.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.216+0000 7f8bafb6f640 1 Processor -- start 2024-09-16T01:39:13.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.216+0000 7f8bafb6f640 1 -- start start 2024-09-16T01:39:13.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.217+0000 7f8bafb6f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ba815e660 0x7f8ba806cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:13.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.217+0000 7f8bafb6f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8ba80d1780 0x7f8ba81627a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:13.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.217+0000 7f8bafb6f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8ba80d1cd0 0x7f8ba8162ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:13.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.217+0000 7f8bafb6f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f8ba8162310 con 0x7f8ba80d1cd0 2024-09-16T01:39:13.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.217+0000 7f8bafb6f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f8ba80b6f90 con 0x7f8ba80d1780 2024-09-16T01:39:13.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.217+0000 7f8bafb6f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8ba80b7270 con 0x7f8ba815e660 2024-09-16T01:39:13.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.217+0000 7f8bae10c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8ba80d1cd0 0x7f8ba8162ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:13.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.217+0000 7f8bae10c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8ba80d1cd0 0x7f8ba8162ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37600/0 (socket says 172.21.15.88:37600) 2024-09-16T01:39:13.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.217+0000 7f8bae10c640 1 -- 172.21.15.88:0/2189040170 learned_addr learned my addr 172.21.15.88:0/2189040170 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:13.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.217+0000 7f8bad10a640 1 --2- 172.21.15.88:0/2189040170 >> v2:172.21.15.154:3300/0 conn(0x7f8ba80d1780 0x7f8ba81627a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:13.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.217+0000 7f8bad90b640 1 --2- 172.21.15.88:0/2189040170 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ba815e660 0x7f8ba806cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:13.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.217+0000 7f8bae10c640 1 -- 172.21.15.88:0/2189040170 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ba815e660 msgr2=0x7f8ba806cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:13.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.217+0000 7f8bae10c640 1 --2- 172.21.15.88:0/2189040170 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ba815e660 0x7f8ba806cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:13.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.217+0000 7f8bae10c640 1 -- 172.21.15.88:0/2189040170 >> v2:172.21.15.154:3300/0 conn(0x7f8ba80d1780 msgr2=0x7f8ba81627a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:13.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.217+0000 7f8bae10c640 1 --2- 172.21.15.88:0/2189040170 >> v2:172.21.15.154:3300/0 conn(0x7f8ba80d1780 0x7f8ba81627a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:13.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.217+0000 7f8bae10c640 1 -- 172.21.15.88:0/2189040170 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba80b7550 con 0x7f8ba80d1cd0 2024-09-16T01:39:13.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.217+0000 7f8bad10a640 1 --2- 172.21.15.88:0/2189040170 >> v2:172.21.15.154:3300/0 conn(0x7f8ba80d1780 0x7f8ba81627a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:13.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.217+0000 7f8bad90b640 1 --2- 172.21.15.88:0/2189040170 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ba815e660 0x7f8ba806cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:13.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.218+0000 7f8bae10c640 1 --2- 172.21.15.88:0/2189040170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ba80d1cd0 0x7f8ba8162ce0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8071bc0 tx=0x7f8ba002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6483823b59a98566 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:13.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.218+0000 7f8bac909640 1 -- 172.21.15.88:0/2189040170 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba0036070 con 0x7f8ba80d1cd0 2024-09-16T01:39:13.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.218+0000 7f8bac909640 1 -- 172.21.15.88:0/2189040170 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ba0031600 con 0x7f8ba80d1cd0 2024-09-16T01:39:13.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.218+0000 7f8bac909640 1 -- 172.21.15.88:0/2189040170 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba00319d0 con 0x7f8ba80d1cd0 2024-09-16T01:39:13.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.218+0000 7f8bafb6f640 1 -- 172.21.15.88:0/2189040170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ba80d1cd0 msgr2=0x7f8ba8162ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:13.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.218+0000 7f8bafb6f640 1 --2- 172.21.15.88:0/2189040170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ba80d1cd0 0x7f8ba8162ce0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8071bc0 tx=0x7f8ba002a6a0 comp rx=0 tx=0).stop 2024-09-16T01:39:13.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.218+0000 7f8bafb6f640 1 -- 172.21.15.88:0/2189040170 shutdown_connections 2024-09-16T01:39:13.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.218+0000 7f8bafb6f640 1 --2- 172.21.15.88:0/2189040170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ba80d1cd0 0x7f8ba8162ce0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:13.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.218+0000 7f8bafb6f640 1 --2- 172.21.15.88:0/2189040170 >> v2:172.21.15.154:3300/0 conn(0x7f8ba80d1780 0x7f8ba81627a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:13.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.218+0000 7f8bafb6f640 1 --2- 172.21.15.88:0/2189040170 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ba815e660 0x7f8ba806cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:13.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.218+0000 7f8bafb6f640 1 -- 172.21.15.88:0/2189040170 >> 172.21.15.88:0/2189040170 conn(0x7f8ba80b4160 msgr2=0x7f8ba80b4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:13.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.219+0000 7f8bafb6f640 1 -- 172.21.15.88:0/2189040170 shutdown_connections 2024-09-16T01:39:13.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.219+0000 7f8bafb6f640 1 -- 172.21.15.88:0/2189040170 wait complete. 2024-09-16T01:39:13.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.220+0000 7f8bafb6f640 1 Processor -- start 2024-09-16T01:39:13.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.220+0000 7f8bafb6f640 1 -- start start 2024-09-16T01:39:13.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.220+0000 7f8bafb6f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ba80d1780 0x7f8ba8155460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:13.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.220+0000 7f8bafb6f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ba80d1cd0 0x7f8ba815d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:13.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.220+0000 7f8bafb6f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ba815e660 0x7f8ba81528a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:13.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.220+0000 7f8bafb6f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba80b6c40 con 0x7f8ba80d1780 2024-09-16T01:39:13.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.220+0000 7f8bafb6f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba815ea40 con 0x7f8ba80d1cd0 2024-09-16T01:39:13.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.220+0000 7f8bafb6f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8ba80b7830 con 0x7f8ba815e660 2024-09-16T01:39:13.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.220+0000 7f8bad90b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ba80d1780 0x7f8ba8155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:13.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.220+0000 7f8bad90b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ba80d1780 0x7f8ba8155460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37614/0 (socket says 172.21.15.88:37614) 2024-09-16T01:39:13.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.220+0000 7f8bad90b640 1 -- 172.21.15.88:0/943839820 learned_addr learned my addr 172.21.15.88:0/943839820 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:13.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.221+0000 7f8bad10a640 1 --2- 172.21.15.88:0/943839820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ba80d1cd0 0x7f8ba815d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:13.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.221+0000 7f8bae10c640 1 --2- 172.21.15.88:0/943839820 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ba815e660 0x7f8ba81528a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:13.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.221+0000 7f8bad90b640 1 -- 172.21.15.88:0/943839820 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ba815e660 msgr2=0x7f8ba81528a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:13.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.221+0000 7f8bad90b640 1 --2- 172.21.15.88:0/943839820 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ba815e660 0x7f8ba81528a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:13.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.221+0000 7f8bad90b640 1 -- 172.21.15.88:0/943839820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ba80d1cd0 msgr2=0x7f8ba815d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:13.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.221+0000 7f8bad90b640 1 --2- 172.21.15.88:0/943839820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ba80d1cd0 0x7f8ba815d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:13.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.221+0000 7f8bad90b640 1 -- 172.21.15.88:0/943839820 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba81543c0 con 0x7f8ba80d1780 2024-09-16T01:39:13.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.221+0000 7f8bad10a640 1 --2- 172.21.15.88:0/943839820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ba80d1cd0 0x7f8ba815d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:13.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.221+0000 7f8bad90b640 1 --2- 172.21.15.88:0/943839820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ba80d1780 0x7f8ba8155460 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f8ba400c7f0 tx=0x7f8ba400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:13.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.221+0000 7f8b9e7fc640 1 -- 172.21.15.88:0/943839820 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4015070 con 0x7f8ba80d1780 2024-09-16T01:39:13.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.221+0000 7f8b9e7fc640 1 -- 172.21.15.88:0/943839820 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ba4007820 con 0x7f8ba80d1780 2024-09-16T01:39:13.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.221+0000 7f8bafb6f640 1 -- 172.21.15.88:0/943839820 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ba81546a0 con 0x7f8ba80d1780 2024-09-16T01:39:13.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.221+0000 7f8b9e7fc640 1 -- 172.21.15.88:0/943839820 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4007bd0 con 0x7f8ba80d1780 2024-09-16T01:39:13.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.221+0000 7f8bafb6f640 1 -- 172.21.15.88:0/943839820 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ba8154c60 con 0x7f8ba80d1780 2024-09-16T01:39:13.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.222+0000 7f8bafb6f640 1 -- 172.21.15.88:0/943839820 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ba806cd20 con 0x7f8ba80d1780 2024-09-16T01:39:13.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.222+0000 7f8b9e7fc640 1 -- 172.21.15.88:0/943839820 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8ba4018070 con 0x7f8ba80d1780 2024-09-16T01:39:13.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.223+0000 7f8b9e7fc640 1 --2- 172.21.15.88:0/943839820 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8b8c06c2b0 0x7f8b8c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:13.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.223+0000 7f8b9e7fc640 1 -- 172.21.15.88:0/943839820 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 4857+0+0 (secure 0 0 0) 0x7f8ba408b020 con 0x7f8ba80d1780 2024-09-16T01:39:13.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.223+0000 7f8bad10a640 1 --2- 172.21.15.88:0/943839820 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8b8c06c2b0 0x7f8b8c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:13.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.223+0000 7f8bad10a640 1 --2- 172.21.15.88:0/943839820 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8b8c06c2b0 0x7f8b8c06e770 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8ba806ae70 tx=0x7f8b98008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:13.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.227+0000 7f8b9e7fc640 1 -- 172.21.15.88:0/943839820 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8ba4059ae0 con 0x7f8ba80d1780 2024-09-16T01:39:13.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.393+0000 7f8bafb6f640 1 -- 172.21.15.88:0/943839820 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f8ba8155060 con 0x7f8ba80d1780 2024-09-16T01:39:13.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.456+0000 7f8b9e7fc640 1 -- 172.21.15.88:0/943839820 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v69) ==== 138+0+0 (secure 0 0 0) 0x7f8ba40593e0 con 0x7f8ba80d1780 2024-09-16T01:39:13.458 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_metadata' created 2024-09-16T01:39:13.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.459+0000 7f8bafb6f640 1 -- 172.21.15.88:0/943839820 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8b8c06c2b0 msgr2=0x7f8b8c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:13.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.459+0000 7f8bafb6f640 1 --2- 172.21.15.88:0/943839820 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8b8c06c2b0 0x7f8b8c06e770 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8ba806ae70 tx=0x7f8b98008040 comp rx=0 tx=0).stop 2024-09-16T01:39:13.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.459+0000 7f8bafb6f640 1 -- 172.21.15.88:0/943839820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ba80d1780 msgr2=0x7f8ba8155460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:13.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.459+0000 7f8bafb6f640 1 --2- 172.21.15.88:0/943839820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ba80d1780 0x7f8ba8155460 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f8ba400c7f0 tx=0x7f8ba400ccb0 comp rx=0 tx=0).stop 2024-09-16T01:39:13.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.459+0000 7f8bafb6f640 1 -- 172.21.15.88:0/943839820 shutdown_connections 2024-09-16T01:39:13.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.459+0000 7f8bafb6f640 1 --2- 172.21.15.88:0/943839820 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8b8c06c2b0 0x7f8b8c06e770 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:13.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.459+0000 7f8bafb6f640 1 --2- 172.21.15.88:0/943839820 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ba815e660 0x7f8ba81528a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:13.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.459+0000 7f8bafb6f640 1 --2- 172.21.15.88:0/943839820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ba80d1cd0 0x7f8ba815d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:13.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.459+0000 7f8bafb6f640 1 --2- 172.21.15.88:0/943839820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ba80d1780 0x7f8ba8155460 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:13.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.459+0000 7f8bafb6f640 1 -- 172.21.15.88:0/943839820 >> 172.21.15.88:0/943839820 conn(0x7f8ba80b4160 msgr2=0x7f8ba80d0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:13.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.459+0000 7f8bafb6f640 1 -- 172.21.15.88:0/943839820 shutdown_connections 2024-09-16T01:39:13.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.459+0000 7f8bafb6f640 1 -- 172.21.15.88:0/943839820 wait complete. 2024-09-16T01:39:13.474 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-16T01:39:13.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.580+0000 7f00c5d31640 1 Processor -- start 2024-09-16T01:39:13.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.580+0000 7f00c5d31640 1 -- start start 2024-09-16T01:39:13.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.580+0000 7f00c5d31640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f00c015e7a0 0x7f00c006caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:13.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.580+0000 7f00c5d31640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f00c00b68f0 0x7f00c00d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:13.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.580+0000 7f00c5d31640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00c00cf860 con 0x7f00c00cf1f0 2024-09-16T01:39:13.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.580+0000 7f00c5d31640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f00c00d2ca0 con 0x7f00c00b68f0 2024-09-16T01:39:13.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.580+0000 7f00c5d31640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f00c00cfb40 con 0x7f00c015e7a0 2024-09-16T01:39:13.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.581+0000 7f00bffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f00c00b68f0 0x7f00c00d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:13.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.581+0000 7f00bffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f00c00b68f0 0x7f00c00d3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37616/0 (socket says 172.21.15.88:37616) 2024-09-16T01:39:13.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.581+0000 7f00bffff640 1 -- 172.21.15.88:0/2943286117 learned_addr learned my addr 172.21.15.88:0/2943286117 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:13.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.581+0000 7f00bf7fe640 1 --2- 172.21.15.88:0/2943286117 >> v2:172.21.15.154:3300/0 conn(0x7f00c015e7a0 0x7f00c006caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:13.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.581+0000 7f00bffff640 1 -- 172.21.15.88:0/2943286117 >> v2:172.21.15.154:3300/0 conn(0x7f00c015e7a0 msgr2=0x7f00c006caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:13.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.581+0000 7f00bffff640 1 --2- 172.21.15.88:0/2943286117 >> v2:172.21.15.154:3300/0 conn(0x7f00c015e7a0 0x7f00c006caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:13.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.581+0000 7f00bffff640 1 -- 172.21.15.88:0/2943286117 >> v1:172.21.15.154:6789/0 conn(0x7f00c00cf1f0 legacy=0x7f00c00b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:13.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.581+0000 7f00bffff640 1 -- 172.21.15.88:0/2943286117 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c00cfe20 con 0x7f00c00b68f0 2024-09-16T01:39:13.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.582+0000 7f00bffff640 1 --2- 172.21.15.88:0/2943286117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c00b68f0 0x7f00c00d3410 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f00c00d5820 tx=0x7f00b802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70017a2621f07c1e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:13.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.582+0000 7f00be7fc640 1 -- 172.21.15.88:0/2943286117 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b8037070 con 0x7f00c00b68f0 2024-09-16T01:39:13.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.582+0000 7f00be7fc640 1 -- 172.21.15.88:0/2943286117 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00b8032440 con 0x7f00c00b68f0 2024-09-16T01:39:13.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.582+0000 7f00be7fc640 1 -- 172.21.15.88:0/2943286117 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b8032810 con 0x7f00c00b68f0 2024-09-16T01:39:13.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.582+0000 7f00c5d31640 1 -- 172.21.15.88:0/2943286117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c00b68f0 msgr2=0x7f00c00d3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:13.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.582+0000 7f00c5d31640 1 --2- 172.21.15.88:0/2943286117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c00b68f0 0x7f00c00d3410 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f00c00d5820 tx=0x7f00b802a1a0 comp rx=0 tx=0).stop 2024-09-16T01:39:13.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.582+0000 7f00c5d31640 1 -- 172.21.15.88:0/2943286117 shutdown_connections 2024-09-16T01:39:13.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.582+0000 7f00c5d31640 1 --2- 172.21.15.88:0/2943286117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c00b68f0 0x7f00c00d3410 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:13.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.582+0000 7f00c5d31640 1 --2- 172.21.15.88:0/2943286117 >> v2:172.21.15.154:3300/0 conn(0x7f00c015e7a0 0x7f00c006caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:13.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.582+0000 7f00c5d31640 1 -- 172.21.15.88:0/2943286117 >> 172.21.15.88:0/2943286117 conn(0x7f00c00d11e0 msgr2=0x7f00c00d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:13.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.582+0000 7f00c5d31640 1 -- 172.21.15.88:0/2943286117 shutdown_connections 2024-09-16T01:39:13.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.582+0000 7f00c5d31640 1 -- 172.21.15.88:0/2943286117 wait complete. 2024-09-16T01:39:13.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.583+0000 7f00c5d31640 1 Processor -- start 2024-09-16T01:39:13.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.584+0000 7f00c5d31640 1 -- start start 2024-09-16T01:39:13.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.584+0000 7f00c5d31640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c00b68f0 0x7f00c00d3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:13.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.584+0000 7f00c5d31640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00c00cf1f0 0x7f00c014ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:13.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.584+0000 7f00c5d31640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00c015e7a0 0x7f00c015ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:13.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.584+0000 7f00c5d31640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f00c006cfe0 con 0x7f00c00b68f0 2024-09-16T01:39:13.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.584+0000 7f00c5d31640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f00c00b7a00 con 0x7f00c015e7a0 2024-09-16T01:39:13.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.584+0000 7f00c5d31640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f00c015eb80 con 0x7f00c00cf1f0 2024-09-16T01:39:13.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.584+0000 7f00bf7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c00b68f0 0x7f00c00d3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:13.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.585+0000 7f00bf7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c00b68f0 0x7f00c00d3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37618/0 (socket says 172.21.15.88:37618) 2024-09-16T01:39:13.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.585+0000 7f00bf7fe640 1 -- 172.21.15.88:0/2448149191 learned_addr learned my addr 172.21.15.88:0/2448149191 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:13.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.585+0000 7f00beffd640 1 --2- 172.21.15.88:0/2448149191 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00c00cf1f0 0x7f00c014ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:13.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.585+0000 7f00bffff640 1 --2- 172.21.15.88:0/2448149191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00c015e7a0 0x7f00c015ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:13.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.585+0000 7f00bf7fe640 1 -- 172.21.15.88:0/2448149191 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00c00cf1f0 msgr2=0x7f00c014ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:13.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.585+0000 7f00bf7fe640 1 --2- 172.21.15.88:0/2448149191 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00c00cf1f0 0x7f00c014ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:13.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.585+0000 7f00bf7fe640 1 -- 172.21.15.88:0/2448149191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00c015e7a0 msgr2=0x7f00c015ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:13.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.585+0000 7f00bf7fe640 1 --2- 172.21.15.88:0/2448149191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00c015e7a0 0x7f00c015ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:13.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.585+0000 7f00bf7fe640 1 -- 172.21.15.88:0/2448149191 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c0161960 con 0x7f00c00b68f0 2024-09-16T01:39:13.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.585+0000 7f00bffff640 1 --2- 172.21.15.88:0/2448149191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00c015e7a0 0x7f00c015ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:13.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.585+0000 7f00bf7fe640 1 --2- 172.21.15.88:0/2448149191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c00b68f0 0x7f00c00d3310 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f00b0007ef0 tx=0x7f00b000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:13.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.585+0000 7f00a7fff640 1 -- 172.21.15.88:0/2448149191 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b0014620 con 0x7f00c00b68f0 2024-09-16T01:39:13.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.585+0000 7f00a7fff640 1 -- 172.21.15.88:0/2448149191 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00b0019070 con 0x7f00c00b68f0 2024-09-16T01:39:13.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.585+0000 7f00c5d31640 1 -- 172.21.15.88:0/2448149191 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00c0161c40 con 0x7f00c00b68f0 2024-09-16T01:39:13.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.585+0000 7f00a7fff640 1 -- 172.21.15.88:0/2448149191 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b0014a00 con 0x7f00c00b68f0 2024-09-16T01:39:13.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.585+0000 7f00c5d31640 1 -- 172.21.15.88:0/2448149191 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00c00c8850 con 0x7f00c00b68f0 2024-09-16T01:39:13.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.586+0000 7f00c5d31640 1 -- 172.21.15.88:0/2448149191 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00c006caa0 con 0x7f00c00b68f0 2024-09-16T01:39:13.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.587+0000 7f00a7fff640 1 -- 172.21.15.88:0/2448149191 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f00b0014ba0 con 0x7f00c00b68f0 2024-09-16T01:39:13.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.587+0000 7f00a7fff640 1 --2- 172.21.15.88:0/2448149191 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f00ac06c2e0 0x7f00ac06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:13.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.587+0000 7f00beffd640 1 --2- 172.21.15.88:0/2448149191 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f00ac06c2e0 0x7f00ac06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:13.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.587+0000 7f00a7fff640 1 -- 172.21.15.88:0/2448149191 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5227+0+0 (secure 0 0 0) 0x7f00b008ba90 con 0x7f00c00b68f0 2024-09-16T01:39:13.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.587+0000 7f00beffd640 1 --2- 172.21.15.88:0/2448149191 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f00ac06c2e0 0x7f00ac06e7a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f00c00cb390 tx=0x7f00b4007e30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:13.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.592+0000 7f00a7fff640 1 -- 172.21.15.88:0/2448149191 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f00b005a3e0 con 0x7f00c00b68f0 2024-09-16T01:39:13.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:13.757+0000 7f00c5d31640 1 -- 172.21.15.88:0/2448149191 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f00c00d2a80 con 0x7f00c00b68f0 2024-09-16T01:39:14.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.463+0000 7f00a7fff640 1 -- 172.21.15.88:0/2448149191 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v70) ==== 170+0+0 (secure 0 0 0) 0x7f00b0059ce0 con 0x7f00c00b68f0 2024-09-16T01:39:14.464 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_data' created 2024-09-16T01:39:14.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.467+0000 7f00c5d31640 1 -- 172.21.15.88:0/2448149191 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f00ac06c2e0 msgr2=0x7f00ac06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:14.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.467+0000 7f00c5d31640 1 --2- 172.21.15.88:0/2448149191 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f00ac06c2e0 0x7f00ac06e7a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f00c00cb390 tx=0x7f00b4007e30 comp rx=0 tx=0).stop 2024-09-16T01:39:14.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.467+0000 7f00c5d31640 1 -- 172.21.15.88:0/2448149191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c00b68f0 msgr2=0x7f00c00d3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:14.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.467+0000 7f00c5d31640 1 --2- 172.21.15.88:0/2448149191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c00b68f0 0x7f00c00d3310 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f00b0007ef0 tx=0x7f00b000e4b0 comp rx=0 tx=0).stop 2024-09-16T01:39:14.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.467+0000 7f00c5d31640 1 -- 172.21.15.88:0/2448149191 shutdown_connections 2024-09-16T01:39:14.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.467+0000 7f00c5d31640 1 --2- 172.21.15.88:0/2448149191 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f00ac06c2e0 0x7f00ac06e7a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:14.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.467+0000 7f00c5d31640 1 --2- 172.21.15.88:0/2448149191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00c015e7a0 0x7f00c015ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:14.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.467+0000 7f00c5d31640 1 --2- 172.21.15.88:0/2448149191 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00c00cf1f0 0x7f00c014ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:14.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.467+0000 7f00c5d31640 1 --2- 172.21.15.88:0/2448149191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c00b68f0 0x7f00c00d3310 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:14.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.467+0000 7f00c5d31640 1 -- 172.21.15.88:0/2448149191 >> 172.21.15.88:0/2448149191 conn(0x7f00c00d11e0 msgr2=0x7f00c00b7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:14.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.468+0000 7f00c5d31640 1 -- 172.21.15.88:0/2448149191 shutdown_connections 2024-09-16T01:39:14.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.468+0000 7f00c5d31640 1 -- 172.21.15.88:0/2448149191 wait complete. 2024-09-16T01:39:14.492 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-16T01:39:14.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.593+0000 7fbccefc6640 1 Processor -- start 2024-09-16T01:39:14.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.593+0000 7fbccefc6640 1 -- start start 2024-09-16T01:39:14.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.594+0000 7fbccefc6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbcc815e7a0 0x7fbcc806caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:14.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.594+0000 7fbccefc6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbcc80b68f0 0x7fbcc80d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:14.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.594+0000 7fbccefc6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcc80cf860 con 0x7fbcc80cf1f0 2024-09-16T01:39:14.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.594+0000 7fbccefc6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fbcc80d2ca0 con 0x7fbcc815e7a0 2024-09-16T01:39:14.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.594+0000 7fbccefc6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbcc80cfb40 con 0x7fbcc80b68f0 2024-09-16T01:39:14.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.594+0000 7fbcccd62640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbcc815e7a0 0x7fbcc806caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:14.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.594+0000 7fbcc7fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fbcc80cf1f0 0x7fbcc80b61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42028/0 (socket says 172.21.15.88:42028) 2024-09-16T01:39:14.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.594+0000 7fbcc7fff640 1 -- 172.21.15.88:0/1303075726 learned_addr learned my addr 172.21.15.88:0/1303075726 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:14.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.595+0000 7fbcccd62640 1 -- 172.21.15.88:0/1303075726 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbcc80b68f0 msgr2=0x7fbcc80d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:14.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.595+0000 7fbcccd62640 1 --2- 172.21.15.88:0/1303075726 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbcc80b68f0 0x7fbcc80d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:14.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.595+0000 7fbcccd62640 1 -- 172.21.15.88:0/1303075726 >> v1:172.21.15.88:6789/0 conn(0x7fbcc80cf1f0 legacy=0x7fbcc80b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:14.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.595+0000 7fbcc77fe640 1 -- 172.21.15.88:0/1303075726 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3131774801 0 0) 0x7fbcc80cf860 con 0x7fbcc80cf1f0 2024-09-16T01:39:14.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.595+0000 7fbcccd62640 1 -- 172.21.15.88:0/1303075726 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcc80cfe20 con 0x7fbcc815e7a0 2024-09-16T01:39:14.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.595+0000 7fbcccd62640 1 --2- 172.21.15.88:0/1303075726 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcc815e7a0 0x7fbcc806caa0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fbcb80097a0 tx=0x7fbcb802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccb5fa1fcb9bce13 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:14.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.596+0000 7fbcc77fe640 1 -- 172.21.15.88:0/1303075726 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8037070 con 0x7fbcc815e7a0 2024-09-16T01:39:14.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.596+0000 7fbcc77fe640 1 -- 172.21.15.88:0/1303075726 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcb802ae50 con 0x7fbcc815e7a0 2024-09-16T01:39:14.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.596+0000 7fbcc77fe640 1 -- 172.21.15.88:0/1303075726 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8032660 con 0x7fbcc815e7a0 2024-09-16T01:39:14.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.596+0000 7fbccefc6640 1 -- 172.21.15.88:0/1303075726 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcc815e7a0 msgr2=0x7fbcc806caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:14.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.596+0000 7fbccefc6640 1 --2- 172.21.15.88:0/1303075726 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcc815e7a0 0x7fbcc806caa0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fbcb80097a0 tx=0x7fbcb802a100 comp rx=0 tx=0).stop 2024-09-16T01:39:14.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.596+0000 7fbccefc6640 1 -- 172.21.15.88:0/1303075726 shutdown_connections 2024-09-16T01:39:14.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.596+0000 7fbccefc6640 1 --2- 172.21.15.88:0/1303075726 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbcc80b68f0 0x7fbcc80d3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:14.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.596+0000 7fbccefc6640 1 --2- 172.21.15.88:0/1303075726 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcc815e7a0 0x7fbcc806caa0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:14.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.596+0000 7fbccefc6640 1 -- 172.21.15.88:0/1303075726 >> 172.21.15.88:0/1303075726 conn(0x7fbcc80d11e0 msgr2=0x7fbcc80d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:14.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.596+0000 7fbccefc6640 1 -- 172.21.15.88:0/1303075726 shutdown_connections 2024-09-16T01:39:14.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.596+0000 7fbccefc6640 1 -- 172.21.15.88:0/1303075726 wait complete. 2024-09-16T01:39:14.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.597+0000 7fbccefc6640 1 Processor -- start 2024-09-16T01:39:14.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.597+0000 7fbccefc6640 1 -- start start 2024-09-16T01:39:14.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.598+0000 7fbccefc6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcc80b68f0 0x7fbcc8159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:14.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.598+0000 7fbccefc6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbcc80cf1f0 0x7fbcc80ca460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:14.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.598+0000 7fbccefc6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbcc815e7a0 0x7fbcc80ca9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:14.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.598+0000 7fbccefc6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbcc806cfe0 con 0x7fbcc80b68f0 2024-09-16T01:39:14.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.598+0000 7fbccefc6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbcc80b7a00 con 0x7fbcc80cf1f0 2024-09-16T01:39:14.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.598+0000 7fbccefc6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbcc815eb80 con 0x7fbcc815e7a0 2024-09-16T01:39:14.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.598+0000 7fbcccd62640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcc80b68f0 0x7fbcc8159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:14.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.598+0000 7fbcccd62640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcc80b68f0 0x7fbcc8159c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37634/0 (socket says 172.21.15.88:37634) 2024-09-16T01:39:14.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.598+0000 7fbcccd62640 1 -- 172.21.15.88:0/2647210856 learned_addr learned my addr 172.21.15.88:0/2647210856 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:14.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.599+0000 7fbcccd62640 1 -- 172.21.15.88:0/2647210856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbcc815e7a0 msgr2=0x7fbcc80ca9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:14.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.599+0000 7fbcccd62640 1 --2- 172.21.15.88:0/2647210856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbcc815e7a0 0x7fbcc80ca9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:14.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.599+0000 7fbcccd62640 1 -- 172.21.15.88:0/2647210856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbcc80cf1f0 msgr2=0x7fbcc80ca460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:14.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.599+0000 7fbcccd62640 1 --2- 172.21.15.88:0/2647210856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbcc80cf1f0 0x7fbcc80ca460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:14.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.599+0000 7fbcccd62640 1 -- 172.21.15.88:0/2647210856 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcc815a190 con 0x7fbcc80b68f0 2024-09-16T01:39:14.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.599+0000 7fbcccd62640 1 --2- 172.21.15.88:0/2647210856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcc80b68f0 0x7fbcc8159c50 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fbcb80098c0 tx=0x7fbcb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:14.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.599+0000 7fbcc57fa640 1 -- 172.21.15.88:0/2647210856 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8042070 con 0x7fbcc80b68f0 2024-09-16T01:39:14.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.599+0000 7fbcc57fa640 1 -- 172.21.15.88:0/2647210856 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcb803e670 con 0x7fbcc80b68f0 2024-09-16T01:39:14.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.599+0000 7fbcc57fa640 1 -- 172.21.15.88:0/2647210856 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8037040 con 0x7fbcc80b68f0 2024-09-16T01:39:14.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.599+0000 7fbccefc6640 1 -- 172.21.15.88:0/2647210856 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcc80cefd0 con 0x7fbcc80b68f0 2024-09-16T01:39:14.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.599+0000 7fbccefc6640 1 -- 172.21.15.88:0/2647210856 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcc80cea10 con 0x7fbcc80b68f0 2024-09-16T01:39:14.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.600+0000 7fbccefc6640 1 -- 172.21.15.88:0/2647210856 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcc806caa0 con 0x7fbcc80b68f0 2024-09-16T01:39:14.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.601+0000 7fbcc57fa640 1 -- 172.21.15.88:0/2647210856 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fbcb8045070 con 0x7fbcc80b68f0 2024-09-16T01:39:14.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.601+0000 7fbcc57fa640 1 --2- 172.21.15.88:0/2647210856 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbcb406c2b0 0x7fbcb406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:14.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.601+0000 7fbcc57fa640 1 -- 172.21.15.88:0/2647210856 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5609+0+0 (secure 0 0 0) 0x7fbcb80abe50 con 0x7fbcc80b68f0 2024-09-16T01:39:14.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.601+0000 7fbcc7fff640 1 --2- 172.21.15.88:0/2647210856 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbcb406c2b0 0x7fbcb406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:14.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.602+0000 7fbcc7fff640 1 --2- 172.21.15.88:0/2647210856 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbcb406c2b0 0x7fbcb406e770 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc007fd0 tx=0x7fbcbc007920 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:14.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.606+0000 7fbcc57fa640 1 -- 172.21.15.88:0/2647210856 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fbcc815a190 con 0x7fbcc80b68f0 2024-09-16T01:39:14.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:14.806+0000 7fbccefc6640 1 -- 172.21.15.88:0/2647210856 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fbcc80d2a80 con 0x7fbcc80b68f0 2024-09-16T01:39:15.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.477+0000 7fbcc57fa640 1 -- 172.21.15.88:0/2647210856 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v55) ==== 174+0+0 (secure 0 0 0) 0x7fbcb807a740 con 0x7fbcc80b68f0 2024-09-16T01:39:15.479 INFO:teuthology.orchestra.run.smithi088.stderr:new fs with metadata pool 10 and data pool 11 2024-09-16T01:39:15.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.483+0000 7fbccefc6640 1 -- 172.21.15.88:0/2647210856 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbcb406c2b0 msgr2=0x7fbcb406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:15.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.483+0000 7fbccefc6640 1 --2- 172.21.15.88:0/2647210856 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbcb406c2b0 0x7fbcb406e770 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc007fd0 tx=0x7fbcbc007920 comp rx=0 tx=0).stop 2024-09-16T01:39:15.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.483+0000 7fbccefc6640 1 -- 172.21.15.88:0/2647210856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcc80b68f0 msgr2=0x7fbcc8159c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:15.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.483+0000 7fbccefc6640 1 --2- 172.21.15.88:0/2647210856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcc80b68f0 0x7fbcc8159c50 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fbcb80098c0 tx=0x7fbcb80046c0 comp rx=0 tx=0).stop 2024-09-16T01:39:15.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.483+0000 7fbccefc6640 1 -- 172.21.15.88:0/2647210856 shutdown_connections 2024-09-16T01:39:15.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.483+0000 7fbccefc6640 1 --2- 172.21.15.88:0/2647210856 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbcb406c2b0 0x7fbcb406e770 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:15.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.483+0000 7fbccefc6640 1 --2- 172.21.15.88:0/2647210856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbcc815e7a0 0x7fbcc80ca9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:15.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.483+0000 7fbccefc6640 1 --2- 172.21.15.88:0/2647210856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbcc80cf1f0 0x7fbcc80ca460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:15.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.483+0000 7fbccefc6640 1 --2- 172.21.15.88:0/2647210856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcc80b68f0 0x7fbcc8159c50 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:15.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.483+0000 7fbccefc6640 1 -- 172.21.15.88:0/2647210856 >> 172.21.15.88:0/2647210856 conn(0x7fbcc80d11e0 msgr2=0x7fbcc80b41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:15.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.484+0000 7fbccefc6640 1 -- 172.21.15.88:0/2647210856 shutdown_connections 2024-09-16T01:39:15.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.484+0000 7fbccefc6640 1 -- 172.21.15.88:0/2647210856 wait complete. 2024-09-16T01:39:15.504 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:39:15.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.605+0000 7ff65b1d8640 1 Processor -- start 2024-09-16T01:39:15.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.605+0000 7ff65b1d8640 1 -- start start 2024-09-16T01:39:15.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.606+0000 7ff65b1d8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff654155800 0x7ff6540b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:15.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.606+0000 7ff65b1d8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff65415c330 0x7ff6540b4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:15.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.606+0000 7ff65b1d8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff65415c770 con 0x7ff6540cf570 2024-09-16T01:39:15.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.606+0000 7ff65b1d8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff6540b4e60 con 0x7ff65415c330 2024-09-16T01:39:15.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.606+0000 7ff65b1d8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff6540b50a0 con 0x7ff654155800 2024-09-16T01:39:15.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.606+0000 7ff653fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff65415c330 0x7ff6540b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:15.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.606+0000 7ff658f74640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff654155800 0x7ff6540b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:15.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.606+0000 7ff659775640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7ff6540cf570 0x7ff65415c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:33526/0 (socket says 172.21.15.88:33526) 2024-09-16T01:39:15.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.606+0000 7ff659775640 1 -- 172.21.15.88:0/1495977624 learned_addr learned my addr 172.21.15.88:0/1495977624 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:15.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.607+0000 7ff6537fe640 1 -- 172.21.15.88:0/1495977624 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3229714781 0 0) 0x7ff63c001110 con 0x7ff6540cf570 2024-09-16T01:39:15.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.607+0000 7ff6537fe640 1 -- 172.21.15.88:0/1495977624 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff640005730 con 0x7ff6540cf570 2024-09-16T01:39:15.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.607+0000 7ff658f74640 1 -- 172.21.15.88:0/1495977624 >> v2:172.21.15.154:3300/0 conn(0x7ff65415c330 msgr2=0x7ff6540b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:15.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.607+0000 7ff658f74640 1 --2- 172.21.15.88:0/1495977624 >> v2:172.21.15.154:3300/0 conn(0x7ff65415c330 0x7ff6540b4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:15.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.607+0000 7ff658f74640 1 -- 172.21.15.88:0/1495977624 >> v1:172.21.15.154:6789/0 conn(0x7ff6540cf570 legacy=0x7ff65415c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:15.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.607+0000 7ff658f74640 1 -- 172.21.15.88:0/1495977624 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6540b5380 con 0x7ff654155800 2024-09-16T01:39:15.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.607+0000 7ff653fff640 1 --2- 172.21.15.88:0/1495977624 >> v2:172.21.15.154:3300/0 conn(0x7ff65415c330 0x7ff6540b4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:39:15.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.608+0000 7ff658f74640 1 --2- 172.21.15.88:0/1495977624 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff654155800 0x7ff6540b6140 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0097a0 tx=0x7ff64c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8019445a754a1ef4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:15.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.608+0000 7ff6537fe640 1 -- 172.21.15.88:0/1495977624 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64c037070 con 0x7ff654155800 2024-09-16T01:39:15.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.608+0000 7ff6537fe640 1 -- 172.21.15.88:0/1495977624 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff64c032440 con 0x7ff654155800 2024-09-16T01:39:15.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.608+0000 7ff6537fe640 1 -- 172.21.15.88:0/1495977624 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64c0327f0 con 0x7ff654155800 2024-09-16T01:39:15.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.608+0000 7ff65b1d8640 1 -- 172.21.15.88:0/1495977624 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff654155800 msgr2=0x7ff6540b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:15.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.608+0000 7ff65b1d8640 1 --2- 172.21.15.88:0/1495977624 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff654155800 0x7ff6540b6140 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0097a0 tx=0x7ff64c02a180 comp rx=0 tx=0).stop 2024-09-16T01:39:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.608+0000 7ff65b1d8640 1 -- 172.21.15.88:0/1495977624 shutdown_connections 2024-09-16T01:39:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.608+0000 7ff65b1d8640 1 --2- 172.21.15.88:0/1495977624 >> v2:172.21.15.154:3300/0 conn(0x7ff65415c330 0x7ff6540b4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.608+0000 7ff65b1d8640 1 --2- 172.21.15.88:0/1495977624 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff654155800 0x7ff6540b6140 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.608+0000 7ff65b1d8640 1 -- 172.21.15.88:0/1495977624 >> 172.21.15.88:0/1495977624 conn(0x7ff65406bc10 msgr2=0x7ff65406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.608+0000 7ff65b1d8640 1 -- 172.21.15.88:0/1495977624 shutdown_connections 2024-09-16T01:39:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.608+0000 7ff65b1d8640 1 -- 172.21.15.88:0/1495977624 wait complete. 2024-09-16T01:39:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.610+0000 7ff65b1d8640 1 Processor -- start 2024-09-16T01:39:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.610+0000 7ff65b1d8640 1 -- start start 2024-09-16T01:39:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.610+0000 7ff65b1d8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6540cf570 0x7ff6541476d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.610+0000 7ff65b1d8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff654155800 0x7ff65415a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.610+0000 7ff65b1d8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff65415c330 0x7ff65415a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.610+0000 7ff65b1d8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff6540b5c20 con 0x7ff65415c330 2024-09-16T01:39:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.610+0000 7ff65b1d8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff6540b5940 con 0x7ff6540cf570 2024-09-16T01:39:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.610+0000 7ff65b1d8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff654155be0 con 0x7ff654155800 2024-09-16T01:39:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.611+0000 7ff659775640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff65415c330 0x7ff65415a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.611+0000 7ff659775640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff65415c330 0x7ff65415a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37642/0 (socket says 172.21.15.88:37642) 2024-09-16T01:39:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.611+0000 7ff659775640 1 -- 172.21.15.88:0/49450218 learned_addr learned my addr 172.21.15.88:0/49450218 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.611+0000 7ff658f74640 1 --2- 172.21.15.88:0/49450218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6540cf570 0x7ff6541476d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.611+0000 7ff653fff640 1 --2- 172.21.15.88:0/49450218 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff654155800 0x7ff65415a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.611+0000 7ff659775640 1 -- 172.21.15.88:0/49450218 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff654155800 msgr2=0x7ff65415a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.611+0000 7ff659775640 1 --2- 172.21.15.88:0/49450218 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff654155800 0x7ff65415a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.611+0000 7ff659775640 1 -- 172.21.15.88:0/49450218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6540cf570 msgr2=0x7ff6541476d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.611+0000 7ff659775640 1 --2- 172.21.15.88:0/49450218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6540cf570 0x7ff6541476d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.611+0000 7ff659775640 1 -- 172.21.15.88:0/49450218 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff65415ca80 con 0x7ff65415c330 2024-09-16T01:39:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.611+0000 7ff659775640 1 --2- 172.21.15.88:0/49450218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff65415c330 0x7ff65415a960 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7ff63c00c3c0 tx=0x7ff63c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.611+0000 7ff6517fa640 1 -- 172.21.15.88:0/49450218 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63c017070 con 0x7ff65415c330 2024-09-16T01:39:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.611+0000 7ff6517fa640 1 -- 172.21.15.88:0/49450218 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff63c005ce0 con 0x7ff65415c330 2024-09-16T01:39:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.611+0000 7ff65b1d8640 1 -- 172.21.15.88:0/49450218 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff65415cd60 con 0x7ff65415c330 2024-09-16T01:39:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.611+0000 7ff6517fa640 1 -- 172.21.15.88:0/49450218 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63c0072f0 con 0x7ff65415c330 2024-09-16T01:39:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.611+0000 7ff65b1d8640 1 -- 172.21.15.88:0/49450218 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff65415d320 con 0x7ff65415c330 2024-09-16T01:39:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.612+0000 7ff65b1d8640 1 -- 172.21.15.88:0/49450218 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6540b6140 con 0x7ff65415c330 2024-09-16T01:39:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.613+0000 7ff6517fa640 1 -- 172.21.15.88:0/49450218 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff63c01a070 con 0x7ff65415c330 2024-09-16T01:39:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.613+0000 7ff6517fa640 1 --2- 172.21.15.88:0/49450218 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff63406c2e0 0x7ff63406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.613+0000 7ff6517fa640 1 -- 172.21.15.88:0/49450218 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5709+0+0 (secure 0 0 0) 0x7ff63c08a6e0 con 0x7ff65415c330 2024-09-16T01:39:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.613+0000 7ff658f74640 1 --2- 172.21.15.88:0/49450218 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff63406c2e0 0x7ff63406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.614+0000 7ff658f74640 1 --2- 172.21.15.88:0/49450218 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff63406c2e0 0x7ff63406e7a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff64c034000 tx=0x7ff64c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.617+0000 7ff6517fa640 1 -- 172.21.15.88:0/49450218 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff65415c770 con 0x7ff65415c330 2024-09-16T01:39:15.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.783+0000 7ff65b1d8640 1 -- 172.21.15.88:0/49450218 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff65406dd70 con 0x7ff65415c330 2024-09-16T01:39:15.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.784+0000 7ff6517fa640 1 -- 172.21.15.88:0/49450218 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+16717 (secure 0 0 0) 0x7ff63c058800 con 0x7ff65415c330 2024-09-16T01:39:15.786 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:15.786 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":71,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:39:14.808056+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:39:13.412390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-16T01:39:13.777039+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:39:15.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.786+0000 7ff65b1d8640 1 -- 172.21.15.88:0/49450218 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff63406c2e0 msgr2=0x7ff63406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:15.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.786+0000 7ff65b1d8640 1 --2- 172.21.15.88:0/49450218 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff63406c2e0 0x7ff63406e7a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff64c034000 tx=0x7ff64c02e000 comp rx=0 tx=0).stop 2024-09-16T01:39:15.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.786+0000 7ff65b1d8640 1 -- 172.21.15.88:0/49450218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff65415c330 msgr2=0x7ff65415a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:15.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.786+0000 7ff65b1d8640 1 --2- 172.21.15.88:0/49450218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff65415c330 0x7ff65415a960 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7ff63c00c3c0 tx=0x7ff63c00c880 comp rx=0 tx=0).stop 2024-09-16T01:39:15.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.786+0000 7ff65b1d8640 1 -- 172.21.15.88:0/49450218 shutdown_connections 2024-09-16T01:39:15.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.786+0000 7ff65b1d8640 1 --2- 172.21.15.88:0/49450218 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff63406c2e0 0x7ff63406e7a0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:15.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.786+0000 7ff65b1d8640 1 --2- 172.21.15.88:0/49450218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff65415c330 0x7ff65415a960 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:15.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.786+0000 7ff65b1d8640 1 --2- 172.21.15.88:0/49450218 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff654155800 0x7ff65415a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:15.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.786+0000 7ff65b1d8640 1 --2- 172.21.15.88:0/49450218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6540cf570 0x7ff6541476d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:15.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.786+0000 7ff65b1d8640 1 -- 172.21.15.88:0/49450218 >> 172.21.15.88:0/49450218 conn(0x7ff65406bc10 msgr2=0x7ff6540714e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:15.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.786+0000 7ff65b1d8640 1 -- 172.21.15.88:0/49450218 shutdown_connections 2024-09-16T01:39:15.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.786+0000 7ff65b1d8640 1 -- 172.21.15.88:0/49450218 wait complete. 2024-09-16T01:39:15.799 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:39:15.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.896+0000 7f2eaf36d640 1 Processor -- start 2024-09-16T01:39:15.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.896+0000 7f2eaf36d640 1 -- start start 2024-09-16T01:39:15.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.897+0000 7f2eaf36d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2ea80cfc80 0x7f2ea80b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:15.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.897+0000 7f2eaf36d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ea815c330 0x7f2ea80b4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:15.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.897+0000 7f2eaf36d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ea815c740 con 0x7f2ea8155800 2024-09-16T01:39:15.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.897+0000 7f2eaf36d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2ea80b4e60 con 0x7f2ea80cfc80 2024-09-16T01:39:15.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.897+0000 7f2eaf36d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2ea80b50a0 con 0x7f2ea815c330 2024-09-16T01:39:15.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.897+0000 7f2ead109640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f2ea8155800 0x7f2ea80cf570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42042/0 (socket says 172.21.15.88:42042) 2024-09-16T01:39:15.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.897+0000 7f2ead109640 1 -- 172.21.15.88:0/2084992414 learned_addr learned my addr 172.21.15.88:0/2084992414 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:15.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.897+0000 7f2e9ffff640 1 -- 172.21.15.88:0/2084992414 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2554948287 0 0) 0x7f2ea815c740 con 0x7f2ea8155800 2024-09-16T01:39:15.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.897+0000 7f2e9ffff640 1 -- 172.21.15.88:0/2084992414 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e94003520 con 0x7f2ea8155800 2024-09-16T01:39:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.897+0000 7f2ead90a640 1 --2- 172.21.15.88:0/2084992414 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ea815c330 0x7f2ea80b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.897+0000 7f2eac908640 1 --2- 172.21.15.88:0/2084992414 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80cfc80 0x7f2ea80b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.897+0000 7f2e9ffff640 1 -- 172.21.15.88:0/2084992414 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2322075348 0 0) 0x7f2e94003520 con 0x7f2ea8155800 2024-09-16T01:39:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.897+0000 7f2e9ffff640 1 -- 172.21.15.88:0/2084992414 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ea815c330 msgr2=0x7f2ea80b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.897+0000 7f2e9ffff640 1 --2- 172.21.15.88:0/2084992414 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ea815c330 0x7f2ea80b4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.897+0000 7f2e9ffff640 1 -- 172.21.15.88:0/2084992414 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80cfc80 msgr2=0x7f2ea80b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.897+0000 7f2e9ffff640 1 --2- 172.21.15.88:0/2084992414 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80cfc80 0x7f2ea80b6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.897+0000 7f2e9ffff640 1 -- 172.21.15.88:0/2084992414 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea80b5380 con 0x7f2ea8155800 2024-09-16T01:39:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.897+0000 7f2e9ffff640 1 -- 172.21.15.88:0/2084992414 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f2ea4001670 con 0x7f2ea8155800 2024-09-16T01:39:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.897+0000 7f2eac908640 1 --2- 172.21.15.88:0/2084992414 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80cfc80 0x7f2ea80b6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.897+0000 7f2e9ffff640 1 -- 172.21.15.88:0/2084992414 >> v1:172.21.15.88:6789/0 conn(0x7f2ea8155800 legacy=0x7f2ea80cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.898+0000 7f2e9ffff640 1 --2- 172.21.15.88:0/2084992414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e940039b0 0x7f2e94003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.898+0000 7f2e9ffff640 1 --2- 172.21.15.88:0/2084992414 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ea815c330 0x7f2e94004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:15.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.898+0000 7f2e9ffff640 1 --2- 172.21.15.88:0/2084992414 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e94004d40 0x7f2e940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:15.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.898+0000 7f2e9ffff640 1 -- 172.21.15.88:0/2084992414 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea80b4e60 con 0x7f2e94004d40 2024-09-16T01:39:15.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.898+0000 7f2e9ffff640 1 -- 172.21.15.88:0/2084992414 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea80b50a0 con 0x7f2e940039b0 2024-09-16T01:39:15.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.898+0000 7f2e9ffff640 1 -- 172.21.15.88:0/2084992414 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2e9400b9c0 con 0x7f2ea815c330 2024-09-16T01:39:15.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.898+0000 7f2ead90a640 1 --2- 172.21.15.88:0/2084992414 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e94004d40 0x7f2e940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:15.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.898+0000 7f2ead109640 1 --2- 172.21.15.88:0/2084992414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e940039b0 0x7f2e94003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:15.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.898+0000 7f2eac908640 1 --2- 172.21.15.88:0/2084992414 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ea815c330 0x7f2e94004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:15.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.898+0000 7f2ead90a640 1 -- 172.21.15.88:0/2084992414 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ea815c330 msgr2=0x7f2e94004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:15.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.898+0000 7f2ead90a640 1 --2- 172.21.15.88:0/2084992414 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ea815c330 0x7f2e94004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:15.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.898+0000 7f2ead90a640 1 -- 172.21.15.88:0/2084992414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e940039b0 msgr2=0x7f2e94003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:15.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.898+0000 7f2ead90a640 1 --2- 172.21.15.88:0/2084992414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e940039b0 0x7f2e94003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:15.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.898+0000 7f2ead90a640 1 -- 172.21.15.88:0/2084992414 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e94003520 con 0x7f2e94004d40 2024-09-16T01:39:15.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.899+0000 7f2ead90a640 1 --2- 172.21.15.88:0/2084992414 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e94004d40 0x7f2e940051c0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f2ea000e170 tx=0x7f2e94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f00756732374e77 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:15.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.899+0000 7f2e9ffff640 1 -- 172.21.15.88:0/2084992414 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0034020 con 0x7f2e94004d40 2024-09-16T01:39:15.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.899+0000 7f2e9ffff640 1 -- 172.21.15.88:0/2084992414 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ea00305f0 con 0x7f2e94004d40 2024-09-16T01:39:15.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.899+0000 7f2e9ffff640 1 -- 172.21.15.88:0/2084992414 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0009c10 con 0x7f2e94004d40 2024-09-16T01:39:15.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.899+0000 7f2eaf36d640 1 -- 172.21.15.88:0/2084992414 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e94004d40 msgr2=0x7f2e940051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:15.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.899+0000 7f2eaf36d640 1 --2- 172.21.15.88:0/2084992414 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e94004d40 0x7f2e940051c0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f2ea000e170 tx=0x7f2e94005830 comp rx=0 tx=0).stop 2024-09-16T01:39:15.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.899+0000 7f2ead90a640 1 -- 172.21.15.88:0/2084992414 reap_dead start 2024-09-16T01:39:15.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.899+0000 7f2eaf36d640 1 -- 172.21.15.88:0/2084992414 shutdown_connections 2024-09-16T01:39:15.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.899+0000 7f2eaf36d640 1 -- 172.21.15.88:0/2084992414 >> 172.21.15.88:0/2084992414 conn(0x7f2ea806bc10 msgr2=0x7f2ea806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:15.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.899+0000 7f2eaf36d640 1 -- 172.21.15.88:0/2084992414 shutdown_connections 2024-09-16T01:39:15.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.899+0000 7f2eaf36d640 1 -- 172.21.15.88:0/2084992414 wait complete. 2024-09-16T01:39:15.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.901+0000 7f2eaf36d640 1 Processor -- start 2024-09-16T01:39:15.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.901+0000 7f2eaf36d640 1 -- start start 2024-09-16T01:39:15.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.901+0000 7f2eaf36d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e94004d40 0x7f2ea81549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:15.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.901+0000 7f2eaf36d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ea814a200 0x7f2ea8152a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:15.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.901+0000 7f2eaf36d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ea81e6e20 0x7f2ea81e72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:15.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.901+0000 7f2eaf36d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea80b5c20 con 0x7f2ea81e6e20 2024-09-16T01:39:15.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.901+0000 7f2eaf36d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea80b5940 con 0x7f2e94004d40 2024-09-16T01:39:15.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.901+0000 7f2eaf36d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2ea8155be0 con 0x7f2ea814a200 2024-09-16T01:39:15.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.901+0000 7f2ead90a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ea81e6e20 0x7f2ea81e72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:15.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.901+0000 7f2ead90a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ea81e6e20 0x7f2ea81e72d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37660/0 (socket says 172.21.15.88:37660) 2024-09-16T01:39:15.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.901+0000 7f2ead90a640 1 -- 172.21.15.88:0/951360093 learned_addr learned my addr 172.21.15.88:0/951360093 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:15.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.902+0000 7f2ead90a640 1 -- 172.21.15.88:0/951360093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ea814a200 msgr2=0x7f2ea8152a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:15.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.902+0000 7f2ead90a640 1 --2- 172.21.15.88:0/951360093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ea814a200 0x7f2ea8152a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:15.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.902+0000 7f2ead90a640 1 -- 172.21.15.88:0/951360093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e94004d40 msgr2=0x7f2ea81549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:15.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.902+0000 7f2ead90a640 1 --2- 172.21.15.88:0/951360093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e94004d40 0x7f2ea81549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:15.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.902+0000 7f2ead90a640 1 -- 172.21.15.88:0/951360093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea8153920 con 0x7f2ea81e6e20 2024-09-16T01:39:15.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.902+0000 7f2ead90a640 1 --2- 172.21.15.88:0/951360093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ea81e6e20 0x7f2ea81e72d0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f2ea000e170 tx=0x7f2ea0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:15.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.902+0000 7f2e9dffb640 1 -- 172.21.15.88:0/951360093 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0040070 con 0x7f2ea81e6e20 2024-09-16T01:39:15.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.902+0000 7f2e9dffb640 1 -- 172.21.15.88:0/951360093 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ea0003990 con 0x7f2ea81e6e20 2024-09-16T01:39:15.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.902+0000 7f2eaf36d640 1 -- 172.21.15.88:0/951360093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ea8153c00 con 0x7f2ea81e6e20 2024-09-16T01:39:15.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.902+0000 7f2e9dffb640 1 -- 172.21.15.88:0/951360093 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0034040 con 0x7f2ea81e6e20 2024-09-16T01:39:15.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.902+0000 7f2eaf36d640 1 -- 172.21.15.88:0/951360093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ea81541c0 con 0x7f2ea81e6e20 2024-09-16T01:39:15.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.903+0000 7f2eaf36d640 1 -- 172.21.15.88:0/951360093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ea8072d20 con 0x7f2ea81e6e20 2024-09-16T01:39:15.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.904+0000 7f2e9dffb640 1 -- 172.21.15.88:0/951360093 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2ea0037020 con 0x7f2ea81e6e20 2024-09-16T01:39:15.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.904+0000 7f2e9dffb640 1 --2- 172.21.15.88:0/951360093 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2e8c06c2e0 0x7f2e8c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:15.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.904+0000 7f2e9dffb640 1 -- 172.21.15.88:0/951360093 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5709+0+0 (secure 0 0 0) 0x7f2ea00ab7e0 con 0x7f2ea81e6e20 2024-09-16T01:39:15.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.904+0000 7f2ead109640 1 --2- 172.21.15.88:0/951360093 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2e8c06c2e0 0x7f2e8c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:15.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.905+0000 7f2ead109640 1 --2- 172.21.15.88:0/951360093 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2e8c06c2e0 0x7f2e8c06e7a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4001be0 tx=0x7f2ea4001b70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:15.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:15.908+0000 7f2e9dffb640 1 -- 172.21.15.88:0/951360093 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2ea8072d20 con 0x7f2ea81e6e20 2024-09-16T01:39:16.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.075+0000 7f2eaf36d640 1 -- 172.21.15.88:0/951360093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2ea80b6140 con 0x7f2ea81e6e20 2024-09-16T01:39:16.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.076+0000 7f2e9dffb640 1 -- 172.21.15.88:0/951360093 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+16717 (secure 0 0 0) 0x7f2ea007ee20 con 0x7f2ea81e6e20 2024-09-16T01:39:16.077 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:16.077 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":71,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:39:14.808056+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:39:13.412390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-16T01:39:13.777039+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:39:16.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.078+0000 7f2eaf36d640 1 -- 172.21.15.88:0/951360093 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2e8c06c2e0 msgr2=0x7f2e8c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:16.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.078+0000 7f2eaf36d640 1 --2- 172.21.15.88:0/951360093 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2e8c06c2e0 0x7f2e8c06e7a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4001be0 tx=0x7f2ea4001b70 comp rx=0 tx=0).stop 2024-09-16T01:39:16.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.078+0000 7f2eaf36d640 1 -- 172.21.15.88:0/951360093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ea81e6e20 msgr2=0x7f2ea81e72d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:16.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.078+0000 7f2eaf36d640 1 --2- 172.21.15.88:0/951360093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ea81e6e20 0x7f2ea81e72d0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f2ea000e170 tx=0x7f2ea0009750 comp rx=0 tx=0).stop 2024-09-16T01:39:16.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.078+0000 7f2eaf36d640 1 -- 172.21.15.88:0/951360093 shutdown_connections 2024-09-16T01:39:16.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.078+0000 7f2eaf36d640 1 --2- 172.21.15.88:0/951360093 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2e8c06c2e0 0x7f2e8c06e7a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:16.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.078+0000 7f2eaf36d640 1 --2- 172.21.15.88:0/951360093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ea81e6e20 0x7f2ea81e72d0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:16.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.078+0000 7f2eaf36d640 1 --2- 172.21.15.88:0/951360093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ea814a200 0x7f2ea8152a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:16.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.078+0000 7f2eaf36d640 1 --2- 172.21.15.88:0/951360093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e94004d40 0x7f2ea81549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:16.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.078+0000 7f2eaf36d640 1 -- 172.21.15.88:0/951360093 >> 172.21.15.88:0/951360093 conn(0x7f2ea806bc10 msgr2=0x7f2ea806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:16.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.078+0000 7f2eaf36d640 1 -- 172.21.15.88:0/951360093 shutdown_connections 2024-09-16T01:39:16.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.078+0000 7f2eaf36d640 1 -- 172.21.15.88:0/951360093 wait complete. 2024-09-16T01:39:16.091 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-16T01:39:16.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.191+0000 7f41f10f1640 1 Processor -- start 2024-09-16T01:39:16.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.191+0000 7f41f10f1640 1 -- start start 2024-09-16T01:39:16.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.192+0000 7f41f10f1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41ec0b54d0 0x7f41ec06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:16.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.192+0000 7f41f10f1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41ec06e990 con 0x7f41ec0b7450 2024-09-16T01:39:16.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.192+0000 7f41f10f1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41ec06ec70 con 0x7f41ec15e6f0 2024-09-16T01:39:16.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.192+0000 7f41f10f1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f41ec15ec70 con 0x7f41ec0b54d0 2024-09-16T01:39:16.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.192+0000 7f41eb59e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f41ec0b7450 0x7f41ec070830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42048/0 (socket says 172.21.15.88:42048) 2024-09-16T01:39:16.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.192+0000 7f41eb59e640 1 -- 172.21.15.88:0/1452013907 learned_addr learned my addr 172.21.15.88:0/1452013907 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:16.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.192+0000 7f41e259c640 1 --2- 172.21.15.88:0/1452013907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41ec0b54d0 0x7f41ec06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:16.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.192+0000 7f41ea59c640 1 -- 172.21.15.88:0/1452013907 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3099185989 0 0) 0x7f41ec06e990 con 0x7f41ec0b7450 2024-09-16T01:39:16.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.192+0000 7f41ea59c640 1 -- 172.21.15.88:0/1452013907 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41d8003520 con 0x7f41ec0b7450 2024-09-16T01:39:16.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.192+0000 7f41ea59c640 1 -- 172.21.15.88:0/1452013907 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3537317880 0 0) 0x7f41d8003520 con 0x7f41ec0b7450 2024-09-16T01:39:16.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.192+0000 7f41ea59c640 1 -- 172.21.15.88:0/1452013907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41ec0b54d0 msgr2=0x7f41ec06fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:16.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.192+0000 7f41ea59c640 1 --2- 172.21.15.88:0/1452013907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41ec0b54d0 0x7f41ec06fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:16.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.192+0000 7f41ea59c640 1 -- 172.21.15.88:0/1452013907 >> v1:172.21.15.154:6789/0 conn(0x7f41ec15e6f0 legacy=0x7f41ec0b6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:16.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.192+0000 7f41ea59c640 1 -- 172.21.15.88:0/1452013907 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41ec15ef50 con 0x7f41ec0b7450 2024-09-16T01:39:16.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.192+0000 7f41ea59c640 1 -- 172.21.15.88:0/1452013907 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f41e40019f0 con 0x7f41ec0b7450 2024-09-16T01:39:16.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.192+0000 7f41ea59c640 1 -- 172.21.15.88:0/1452013907 >> v1:172.21.15.88:6789/0 conn(0x7f41ec0b7450 legacy=0x7f41ec070830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:16.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.192+0000 7f41ea59c640 1 --2- 172.21.15.88:0/1452013907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41d80039e0 0x7f41d8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:16.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.192+0000 7f41ea59c640 1 --2- 172.21.15.88:0/1452013907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41ec0b54d0 0x7f41d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:16.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.192+0000 7f41ea59c640 1 --2- 172.21.15.88:0/1452013907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41d8004d40 0x7f41d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:16.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.192+0000 7f41ea59c640 1 -- 172.21.15.88:0/1452013907 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f41ec15ec70 con 0x7f41d8004d40 2024-09-16T01:39:16.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.192+0000 7f41ea59c640 1 -- 172.21.15.88:0/1452013907 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f41d800bb50 con 0x7f41d80039e0 2024-09-16T01:39:16.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.192+0000 7f41ea59c640 1 -- 172.21.15.88:0/1452013907 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f41d800bd20 con 0x7f41ec0b54d0 2024-09-16T01:39:16.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.192+0000 7f41eb59e640 1 --2- 172.21.15.88:0/1452013907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41d8004d40 0x7f41d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:16.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.192+0000 7f41ead9d640 1 --2- 172.21.15.88:0/1452013907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41d80039e0 0x7f41d8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:16.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.193+0000 7f41e259c640 1 --2- 172.21.15.88:0/1452013907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41ec0b54d0 0x7f41d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:16.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.193+0000 7f41eb59e640 1 -- 172.21.15.88:0/1452013907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41ec0b54d0 msgr2=0x7f41d8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:16.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.193+0000 7f41eb59e640 1 --2- 172.21.15.88:0/1452013907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41ec0b54d0 0x7f41d8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:16.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.193+0000 7f41eb59e640 1 -- 172.21.15.88:0/1452013907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41d80039e0 msgr2=0x7f41d8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:16.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.193+0000 7f41eb59e640 1 --2- 172.21.15.88:0/1452013907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41d80039e0 0x7f41d8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:16.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.193+0000 7f41eb59e640 1 -- 172.21.15.88:0/1452013907 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41d8003520 con 0x7f41d8004d40 2024-09-16T01:39:16.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.193+0000 7f41eb59e640 1 --2- 172.21.15.88:0/1452013907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41d8004d40 0x7f41d80051c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f41e400a870 tx=0x7f41d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a82531df23484120 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:16.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.194+0000 7f41ea59c640 1 -- 172.21.15.88:0/1452013907 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e400a460 con 0x7f41d8004d40 2024-09-16T01:39:16.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.194+0000 7f41ea59c640 1 -- 172.21.15.88:0/1452013907 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41e402f070 con 0x7f41d8004d40 2024-09-16T01:39:16.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.194+0000 7f41ea59c640 1 -- 172.21.15.88:0/1452013907 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e402bca0 con 0x7f41d8004d40 2024-09-16T01:39:16.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.194+0000 7f41f10f1640 1 -- 172.21.15.88:0/1452013907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41d8004d40 msgr2=0x7f41d80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:16.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.194+0000 7f41f10f1640 1 --2- 172.21.15.88:0/1452013907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41d8004d40 0x7f41d80051c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f41e400a870 tx=0x7f41d8005830 comp rx=0 tx=0).stop 2024-09-16T01:39:16.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.194+0000 7f41eb59e640 1 -- 172.21.15.88:0/1452013907 reap_dead start 2024-09-16T01:39:16.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.194+0000 7f41f10f1640 1 -- 172.21.15.88:0/1452013907 shutdown_connections 2024-09-16T01:39:16.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.194+0000 7f41f10f1640 1 -- 172.21.15.88:0/1452013907 >> 172.21.15.88:0/1452013907 conn(0x7f41ec0d2430 msgr2=0x7f41ec0d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:16.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.194+0000 7f41f10f1640 1 -- 172.21.15.88:0/1452013907 shutdown_connections 2024-09-16T01:39:16.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.194+0000 7f41f10f1640 1 -- 172.21.15.88:0/1452013907 wait complete. 2024-09-16T01:39:16.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.195+0000 7f41f10f1640 1 Processor -- start 2024-09-16T01:39:16.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.196+0000 7f41f10f1640 1 -- start start 2024-09-16T01:39:16.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.196+0000 7f41f10f1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41d8004d40 0x7f41ec1f5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:16.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.196+0000 7f41f10f1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41ec162340 0x7f41ec159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:16.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.196+0000 7f41f10f1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41ec1627c0 0x7f41ec15a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:16.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.196+0000 7f41f10f1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f41ec06e730 con 0x7f41ec162340 2024-09-16T01:39:16.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.196+0000 7f41f10f1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f41ec06e5b0 con 0x7f41ec1627c0 2024-09-16T01:39:16.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.196+0000 7f41f10f1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f41ec06fd00 con 0x7f41d8004d40 2024-09-16T01:39:16.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.196+0000 7f41e259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41ec162340 0x7f41ec159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:16.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.196+0000 7f41e259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41ec162340 0x7f41ec159c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37678/0 (socket says 172.21.15.88:37678) 2024-09-16T01:39:16.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.196+0000 7f41e259c640 1 -- 172.21.15.88:0/1066809922 learned_addr learned my addr 172.21.15.88:0/1066809922 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:16.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.197+0000 7f41ead9d640 1 --2- 172.21.15.88:0/1066809922 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41d8004d40 0x7f41ec1f5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:16.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.197+0000 7f41eb59e640 1 --2- 172.21.15.88:0/1066809922 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41ec1627c0 0x7f41ec15a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:16.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.197+0000 7f41e259c640 1 -- 172.21.15.88:0/1066809922 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41d8004d40 msgr2=0x7f41ec1f5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:16.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.197+0000 7f41e259c640 1 --2- 172.21.15.88:0/1066809922 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41d8004d40 0x7f41ec1f5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:16.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.197+0000 7f41e259c640 1 -- 172.21.15.88:0/1066809922 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41ec1627c0 msgr2=0x7f41ec15a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:16.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.197+0000 7f41e259c640 1 --2- 172.21.15.88:0/1066809922 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41ec1627c0 0x7f41ec15a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:16.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.197+0000 7f41e259c640 1 -- 172.21.15.88:0/1066809922 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41ec0d06b0 con 0x7f41ec162340 2024-09-16T01:39:16.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.197+0000 7f41e259c640 1 --2- 172.21.15.88:0/1066809922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41ec162340 0x7f41ec159c10 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f41d4010840 tx=0x7f41d4010d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:16.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.197+0000 7f41e3fff640 1 -- 172.21.15.88:0/1066809922 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d401a020 con 0x7f41ec162340 2024-09-16T01:39:16.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.197+0000 7f41e3fff640 1 -- 172.21.15.88:0/1066809922 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41d400e9c0 con 0x7f41ec162340 2024-09-16T01:39:16.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.197+0000 7f41e3fff640 1 -- 172.21.15.88:0/1066809922 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d400ed70 con 0x7f41ec162340 2024-09-16T01:39:16.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.197+0000 7f41f10f1640 1 -- 172.21.15.88:0/1066809922 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41ec0cd5f0 con 0x7f41ec162340 2024-09-16T01:39:16.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.197+0000 7f41f10f1640 1 -- 172.21.15.88:0/1066809922 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41ec0cdbb0 con 0x7f41ec162340 2024-09-16T01:39:16.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.198+0000 7f41f10f1640 1 -- 172.21.15.88:0/1066809922 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41ec06c6c0 con 0x7f41ec162340 2024-09-16T01:39:16.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.199+0000 7f41e3fff640 1 -- 172.21.15.88:0/1066809922 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f41d401d020 con 0x7f41ec162340 2024-09-16T01:39:16.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.199+0000 7f41e3fff640 1 --2- 172.21.15.88:0/1066809922 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f41cc06c2b0 0x7f41cc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:16.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.199+0000 7f41ead9d640 1 --2- 172.21.15.88:0/1066809922 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f41cc06c2b0 0x7f41cc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:16.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.199+0000 7f41e3fff640 1 -- 172.21.15.88:0/1066809922 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5709+0+0 (secure 0 0 0) 0x7f41d408ba80 con 0x7f41ec162340 2024-09-16T01:39:16.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.200+0000 7f41ead9d640 1 --2- 172.21.15.88:0/1066809922 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f41cc06c2b0 0x7f41cc06e770 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f41dc0035f0 tx=0x7f41dc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:16.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.203+0000 7f41e3fff640 1 -- 172.21.15.88:0/1066809922 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f41d4024030 con 0x7f41ec162340 2024-09-16T01:39:16.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.403+0000 7f41f10f1640 1 -- 172.21.15.88:0/1066809922 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f41ec1608d0 con 0x7f41ec162340 2024-09-16T01:39:16.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.482+0000 7f41e3fff640 1 -- 172.21.15.88:0/1066809922 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v56) ==== 118+0+0 (secure 0 0 0) 0x7f41d405a280 con 0x7f41ec162340 2024-09-16T01:39:16.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.484+0000 7f41f10f1640 1 -- 172.21.15.88:0/1066809922 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f41cc06c2b0 msgr2=0x7f41cc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:16.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.484+0000 7f41f10f1640 1 --2- 172.21.15.88:0/1066809922 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f41cc06c2b0 0x7f41cc06e770 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f41dc0035f0 tx=0x7f41dc009040 comp rx=0 tx=0).stop 2024-09-16T01:39:16.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.484+0000 7f41f10f1640 1 -- 172.21.15.88:0/1066809922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41ec162340 msgr2=0x7f41ec159c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:16.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.484+0000 7f41f10f1640 1 --2- 172.21.15.88:0/1066809922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41ec162340 0x7f41ec159c10 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f41d4010840 tx=0x7f41d4010d00 comp rx=0 tx=0).stop 2024-09-16T01:39:16.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.484+0000 7f41f10f1640 1 -- 172.21.15.88:0/1066809922 shutdown_connections 2024-09-16T01:39:16.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.484+0000 7f41f10f1640 1 --2- 172.21.15.88:0/1066809922 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f41cc06c2b0 0x7f41cc06e770 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:16.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.484+0000 7f41f10f1640 1 --2- 172.21.15.88:0/1066809922 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41ec1627c0 0x7f41ec15a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:16.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.484+0000 7f41f10f1640 1 --2- 172.21.15.88:0/1066809922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41ec162340 0x7f41ec159c10 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:16.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.484+0000 7f41f10f1640 1 --2- 172.21.15.88:0/1066809922 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41d8004d40 0x7f41ec1f5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:16.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.484+0000 7f41f10f1640 1 -- 172.21.15.88:0/1066809922 >> 172.21.15.88:0/1066809922 conn(0x7f41ec0d2430 msgr2=0x7f41ec074c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:16.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.484+0000 7f41f10f1640 1 -- 172.21.15.88:0/1066809922 shutdown_connections 2024-09-16T01:39:16.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.484+0000 7f41f10f1640 1 -- 172.21.15.88:0/1066809922 wait complete. 2024-09-16T01:39:16.497 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-16T01:39:16.497 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-16T01:39:16.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.597+0000 7f1339b2c640 1 Processor -- start 2024-09-16T01:39:16.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.597+0000 7f1339b2c640 1 -- start start 2024-09-16T01:39:16.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.598+0000 7f1339b2c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f13340b6290 0x7f1334071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:16.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.598+0000 7f1339b2c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1334072530 0x7f133415bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:16.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.598+0000 7f1339b2c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13340ccff0 con 0x7f133415b480 2024-09-16T01:39:16.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.598+0000 7f1339b2c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f13340cd2d0 con 0x7f13340b6290 2024-09-16T01:39:16.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.598+0000 7f1339b2c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f13340cd5b0 con 0x7f1334072530 2024-09-16T01:39:16.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.598+0000 7f13337fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f133415b480 0x7f13340b5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42056/0 (socket says 172.21.15.88:42056) 2024-09-16T01:39:16.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.598+0000 7f13337fe640 1 -- 172.21.15.88:0/4283718097 learned_addr learned my addr 172.21.15.88:0/4283718097 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:16.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.598+0000 7f13327fc640 1 -- 172.21.15.88:0/4283718097 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 374297539 0 0) 0x7f13340ccff0 con 0x7f133415b480 2024-09-16T01:39:16.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.598+0000 7f13327fc640 1 -- 172.21.15.88:0/4283718097 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f131c003520 con 0x7f133415b480 2024-09-16T01:39:16.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.599+0000 7f1332ffd640 1 --2- 172.21.15.88:0/4283718097 >> v2:172.21.15.154:3300/0 conn(0x7f13340b6290 0x7f1334071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:16.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.599+0000 7f1333fff640 1 --2- 172.21.15.88:0/4283718097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1334072530 0x7f133415bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:16.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.599+0000 7f13327fc640 1 -- 172.21.15.88:0/4283718097 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1185139707 0 0) 0x7f131c003520 con 0x7f133415b480 2024-09-16T01:39:16.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.599+0000 7f13327fc640 1 -- 172.21.15.88:0/4283718097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1334072530 msgr2=0x7f133415bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:16.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.599+0000 7f13327fc640 1 --2- 172.21.15.88:0/4283718097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1334072530 0x7f133415bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:16.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.599+0000 7f13327fc640 1 -- 172.21.15.88:0/4283718097 >> v2:172.21.15.154:3300/0 conn(0x7f13340b6290 msgr2=0x7f1334071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:16.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.599+0000 7f13327fc640 1 --2- 172.21.15.88:0/4283718097 >> v2:172.21.15.154:3300/0 conn(0x7f13340b6290 0x7f1334071ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:16.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.599+0000 7f13327fc640 1 -- 172.21.15.88:0/4283718097 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13340cd890 con 0x7f133415b480 2024-09-16T01:39:16.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.599+0000 7f13327fc640 1 -- 172.21.15.88:0/4283718097 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f1324001670 con 0x7f133415b480 2024-09-16T01:39:16.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.599+0000 7f13327fc640 1 -- 172.21.15.88:0/4283718097 >> v1:172.21.15.88:6789/0 conn(0x7f133415b480 legacy=0x7f13340b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:16.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.599+0000 7f13327fc640 1 --2- 172.21.15.88:0/4283718097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1334072530 0x7f131c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:16.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.599+0000 7f13327fc640 1 --2- 172.21.15.88:0/4283718097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f131c003fe0 0x7f131c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:16.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.599+0000 7f13327fc640 1 --2- 172.21.15.88:0/4283718097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f131c004d40 0x7f131c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:16.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.599+0000 7f13327fc640 1 -- 172.21.15.88:0/4283718097 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f13340cd2d0 con 0x7f131c004d40 2024-09-16T01:39:16.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.599+0000 7f13327fc640 1 -- 172.21.15.88:0/4283718097 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f13340cd5b0 con 0x7f131c003fe0 2024-09-16T01:39:16.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.599+0000 7f13327fc640 1 -- 172.21.15.88:0/4283718097 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f131c00bb80 con 0x7f1334072530 2024-09-16T01:39:16.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.599+0000 7f1333fff640 1 --2- 172.21.15.88:0/4283718097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f131c004d40 0x7f131c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:16.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.599+0000 7f1332ffd640 1 --2- 172.21.15.88:0/4283718097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f131c003fe0 0x7f131c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:16.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.599+0000 7f13337fe640 1 --2- 172.21.15.88:0/4283718097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1334072530 0x7f131c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:16.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.599+0000 7f1333fff640 1 -- 172.21.15.88:0/4283718097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1334072530 msgr2=0x7f131c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:16.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.599+0000 7f1333fff640 1 --2- 172.21.15.88:0/4283718097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1334072530 0x7f131c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:16.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.599+0000 7f1333fff640 1 -- 172.21.15.88:0/4283718097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f131c003fe0 msgr2=0x7f131c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:16.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.599+0000 7f1333fff640 1 --2- 172.21.15.88:0/4283718097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f131c003fe0 0x7f131c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:16.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.599+0000 7f1333fff640 1 -- 172.21.15.88:0/4283718097 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f131c003520 con 0x7f131c004d40 2024-09-16T01:39:16.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.600+0000 7f1333fff640 1 --2- 172.21.15.88:0/4283718097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f131c004d40 0x7f131c0051c0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f132c007360 tx=0x7f131c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c138cb51ea8f203 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:16.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.600+0000 7f13327fc640 1 -- 172.21.15.88:0/4283718097 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132c03a070 con 0x7f131c004d40 2024-09-16T01:39:16.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.600+0000 7f13327fc640 1 -- 172.21.15.88:0/4283718097 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f132c02d7c0 con 0x7f131c004d40 2024-09-16T01:39:16.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.600+0000 7f13327fc640 1 -- 172.21.15.88:0/4283718097 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132c009be0 con 0x7f131c004d40 2024-09-16T01:39:16.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.601+0000 7f1339b2c640 1 -- 172.21.15.88:0/4283718097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f131c004d40 msgr2=0x7f131c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:16.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.601+0000 7f1339b2c640 1 --2- 172.21.15.88:0/4283718097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f131c004d40 0x7f131c0051c0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f132c007360 tx=0x7f131c005830 comp rx=0 tx=0).stop 2024-09-16T01:39:16.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.601+0000 7f1333fff640 1 -- 172.21.15.88:0/4283718097 reap_dead start 2024-09-16T01:39:16.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.601+0000 7f1339b2c640 1 -- 172.21.15.88:0/4283718097 shutdown_connections 2024-09-16T01:39:16.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.601+0000 7f1339b2c640 1 -- 172.21.15.88:0/4283718097 >> 172.21.15.88:0/4283718097 conn(0x7f133406d0f0 msgr2=0x7f133406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:16.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.601+0000 7f1339b2c640 1 -- 172.21.15.88:0/4283718097 shutdown_connections 2024-09-16T01:39:16.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.601+0000 7f1339b2c640 1 -- 172.21.15.88:0/4283718097 wait complete. 2024-09-16T01:39:16.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.602+0000 7f1339b2c640 1 Processor -- start 2024-09-16T01:39:16.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.602+0000 7f1339b2c640 1 -- start start 2024-09-16T01:39:16.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.603+0000 7f1339b2c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f131c004d40 0x7f13341579b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:16.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.603+0000 7f1339b2c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f133414e330 0x7f1334154e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:16.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.603+0000 7f1339b2c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13340ca430 0x7f13340d0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:16.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.603+0000 7f1339b2c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f133415b9e0 con 0x7f133414e330 2024-09-16T01:39:16.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.603+0000 7f1339b2c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f133415b860 con 0x7f13340ca430 2024-09-16T01:39:16.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.603+0000 7f1339b2c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f13340b3590 con 0x7f131c004d40 2024-09-16T01:39:16.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.603+0000 7f1332ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f133414e330 0x7f1334154e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:16.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.603+0000 7f1332ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f133414e330 0x7f1334154e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37692/0 (socket says 172.21.15.88:37692) 2024-09-16T01:39:16.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.603+0000 7f1332ffd640 1 -- 172.21.15.88:0/4290667801 learned_addr learned my addr 172.21.15.88:0/4290667801 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:16.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.603+0000 7f1333fff640 1 --2- 172.21.15.88:0/4290667801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13340ca430 0x7f13340d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:16.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.603+0000 7f13337fe640 1 --2- 172.21.15.88:0/4290667801 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f131c004d40 0x7f13341579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:16.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.603+0000 7f1332ffd640 1 -- 172.21.15.88:0/4290667801 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f131c004d40 msgr2=0x7f13341579b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:16.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.603+0000 7f1332ffd640 1 --2- 172.21.15.88:0/4290667801 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f131c004d40 0x7f13341579b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:16.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.603+0000 7f1332ffd640 1 -- 172.21.15.88:0/4290667801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13340ca430 msgr2=0x7f13340d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:16.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.603+0000 7f1332ffd640 1 --2- 172.21.15.88:0/4290667801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13340ca430 0x7f13340d0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:16.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.603+0000 7f1332ffd640 1 -- 172.21.15.88:0/4290667801 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13340cf580 con 0x7f133414e330 2024-09-16T01:39:16.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.603+0000 7f13337fe640 1 --2- 172.21.15.88:0/4290667801 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f131c004d40 0x7f13341579b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:16.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.603+0000 7f1332ffd640 1 --2- 172.21.15.88:0/4290667801 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f133414e330 0x7f1334154e60 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f1328010870 tx=0x7f1328010d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:16.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.603+0000 7f131bfff640 1 -- 172.21.15.88:0/4290667801 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132801a020 con 0x7f133414e330 2024-09-16T01:39:16.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.603+0000 7f131bfff640 1 -- 172.21.15.88:0/4290667801 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f132800e9c0 con 0x7f133414e330 2024-09-16T01:39:16.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.603+0000 7f131bfff640 1 -- 172.21.15.88:0/4290667801 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132800ed90 con 0x7f133414e330 2024-09-16T01:39:16.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.603+0000 7f1339b2c640 1 -- 172.21.15.88:0/4290667801 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13340cf860 con 0x7f133414e330 2024-09-16T01:39:16.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.603+0000 7f1339b2c640 1 -- 172.21.15.88:0/4290667801 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13341559c0 con 0x7f133414e330 2024-09-16T01:39:16.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.604+0000 7f1339b2c640 1 -- 172.21.15.88:0/4290667801 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1334071ff0 con 0x7f133414e330 2024-09-16T01:39:16.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.605+0000 7f131bfff640 1 -- 172.21.15.88:0/4290667801 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f132801d020 con 0x7f133414e330 2024-09-16T01:39:16.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.605+0000 7f131bfff640 1 --2- 172.21.15.88:0/4290667801 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f132006c2b0 0x7f132006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:16.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.605+0000 7f131bfff640 1 -- 172.21.15.88:0/4290667801 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f132808bda0 con 0x7f133414e330 2024-09-16T01:39:16.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.605+0000 7f13337fe640 1 --2- 172.21.15.88:0/4290667801 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f132006c2b0 0x7f132006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:16.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.606+0000 7f13337fe640 1 --2- 172.21.15.88:0/4290667801 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f132006c2b0 0x7f132006e770 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1324001050 tx=0x7f1324009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:16.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.609+0000 7f131bfff640 1 -- 172.21.15.88:0/4290667801 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1328024020 con 0x7f133414e330 2024-09-16T01:39:16.733 INFO:tasks.ceph.mds.d.smithi154.stderr:2024-09-16T01:39:16.730+0000 7f8a8ccd1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:39:16.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:16.810+0000 7f1339b2c640 1 -- 172.21.15.88:0/4290667801 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f1334073740 con 0x7f133414e330 2024-09-16T01:39:17.125 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:39:17.128 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:39:17.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.487+0000 7f131bfff640 1 -- 172.21.15.88:0/4290667801 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v57) ==== 122+0+0 (secure 0 0 0) 0x7f132805a5a0 con 0x7f133414e330 2024-09-16T01:39:17.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.489+0000 7f1339b2c640 1 -- 172.21.15.88:0/4290667801 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f132006c2b0 msgr2=0x7f132006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:17.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.489+0000 7f1339b2c640 1 --2- 172.21.15.88:0/4290667801 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f132006c2b0 0x7f132006e770 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1324001050 tx=0x7f1324009040 comp rx=0 tx=0).stop 2024-09-16T01:39:17.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.489+0000 7f1339b2c640 1 -- 172.21.15.88:0/4290667801 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f133414e330 msgr2=0x7f1334154e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:17.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.489+0000 7f1339b2c640 1 --2- 172.21.15.88:0/4290667801 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f133414e330 0x7f1334154e60 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f1328010870 tx=0x7f1328010d30 comp rx=0 tx=0).stop 2024-09-16T01:39:17.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.489+0000 7f1339b2c640 1 -- 172.21.15.88:0/4290667801 shutdown_connections 2024-09-16T01:39:17.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.489+0000 7f1339b2c640 1 --2- 172.21.15.88:0/4290667801 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f132006c2b0 0x7f132006e770 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:17.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.489+0000 7f1339b2c640 1 --2- 172.21.15.88:0/4290667801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13340ca430 0x7f13340d0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:17.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.489+0000 7f1339b2c640 1 --2- 172.21.15.88:0/4290667801 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f133414e330 0x7f1334154e60 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:17.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.489+0000 7f1339b2c640 1 --2- 172.21.15.88:0/4290667801 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f131c004d40 0x7f13341579b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:17.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.489+0000 7f1339b2c640 1 -- 172.21.15.88:0/4290667801 >> 172.21.15.88:0/4290667801 conn(0x7f133406d0f0 msgr2=0x7f133415c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:17.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.489+0000 7f1339b2c640 1 -- 172.21.15.88:0/4290667801 shutdown_connections 2024-09-16T01:39:17.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.489+0000 7f1339b2c640 1 -- 172.21.15.88:0/4290667801 wait complete. 2024-09-16T01:39:17.502 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-16T01:39:17.502 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:39:17.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.600+0000 7f0defb6d640 1 Processor -- start 2024-09-16T01:39:17.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.600+0000 7f0defb6d640 1 -- start start 2024-09-16T01:39:17.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.601+0000 7f0defb6d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0de80b4920 0x7f0de815b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:17.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.601+0000 7f0defb6d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0de80b4d00 0x7f0de80b5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:17.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.601+0000 7f0defb6d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0de80ce4f0 con 0x7f0de80cd780 2024-09-16T01:39:17.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.601+0000 7f0defb6d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0de80cf100 con 0x7f0de80b4920 2024-09-16T01:39:17.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.601+0000 7f0defb6d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0de80ce7d0 con 0x7f0de80b4d00 2024-09-16T01:39:17.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.601+0000 7f0ded108640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0de80b4d00 0x7f0de80b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:17.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.601+0000 7f0ded108640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0de80b4d00 0x7f0de80b5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:35962/0 (socket says 172.21.15.88:35962) 2024-09-16T01:39:17.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.601+0000 7f0ded108640 1 -- 172.21.15.88:0/3730154241 learned_addr learned my addr 172.21.15.88:0/3730154241 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:17.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.601+0000 7f0ded909640 1 --2- 172.21.15.88:0/3730154241 >> v2:172.21.15.154:3300/0 conn(0x7f0de80b4920 0x7f0de815b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:17.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.601+0000 7f0ded909640 1 -- 172.21.15.88:0/3730154241 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0de80b4d00 msgr2=0x7f0de80b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:17.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.601+0000 7f0ded909640 1 --2- 172.21.15.88:0/3730154241 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0de80b4d00 0x7f0de80b5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:17.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.601+0000 7f0ded909640 1 -- 172.21.15.88:0/3730154241 >> v1:172.21.15.154:6789/0 conn(0x7f0de80cd780 legacy=0x7f0de80cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:17.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.601+0000 7f0ded909640 1 -- 172.21.15.88:0/3730154241 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0de80ceab0 con 0x7f0de80b4920 2024-09-16T01:39:17.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.601+0000 7f0ded108640 1 --2- 172.21.15.88:0/3730154241 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0de80b4d00 0x7f0de80b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:39:17.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.602+0000 7f0ded909640 1 --2- 172.21.15.88:0/3730154241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0de80b4920 0x7f0de815b7d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0de40097a0 tx=0x7f0de402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d3d525ed4e1d2b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:17.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.602+0000 7f0dec907640 1 -- 172.21.15.88:0/3730154241 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de4037070 con 0x7f0de80b4920 2024-09-16T01:39:17.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.602+0000 7f0dec907640 1 -- 172.21.15.88:0/3730154241 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0de4032440 con 0x7f0de80b4920 2024-09-16T01:39:17.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.602+0000 7f0dec907640 1 -- 172.21.15.88:0/3730154241 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de4032810 con 0x7f0de80b4920 2024-09-16T01:39:17.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.603+0000 7f0defb6d640 1 -- 172.21.15.88:0/3730154241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0de80b4920 msgr2=0x7f0de815b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:17.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.603+0000 7f0defb6d640 1 --2- 172.21.15.88:0/3730154241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0de80b4920 0x7f0de815b7d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0de40097a0 tx=0x7f0de402a180 comp rx=0 tx=0).stop 2024-09-16T01:39:17.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.603+0000 7f0defb6d640 1 -- 172.21.15.88:0/3730154241 shutdown_connections 2024-09-16T01:39:17.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.603+0000 7f0defb6d640 1 --2- 172.21.15.88:0/3730154241 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0de80b4d00 0x7f0de80b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:17.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.603+0000 7f0defb6d640 1 --2- 172.21.15.88:0/3730154241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0de80b4920 0x7f0de815b7d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:17.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.603+0000 7f0defb6d640 1 -- 172.21.15.88:0/3730154241 >> 172.21.15.88:0/3730154241 conn(0x7f0de806bc10 msgr2=0x7f0de806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:17.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.603+0000 7f0defb6d640 1 -- 172.21.15.88:0/3730154241 shutdown_connections 2024-09-16T01:39:17.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.603+0000 7f0defb6d640 1 -- 172.21.15.88:0/3730154241 wait complete. 2024-09-16T01:39:17.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.604+0000 7f0defb6d640 1 Processor -- start 2024-09-16T01:39:17.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.604+0000 7f0defb6d640 1 -- start start 2024-09-16T01:39:17.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.605+0000 7f0defb6d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0de80b4920 0x7f0de80aa480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:17.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.605+0000 7f0defb6d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0de80b4d00 0x7f0de814e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:17.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.605+0000 7f0defb6d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0de80cd780 0x7f0de8149060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:17.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.605+0000 7f0defb6d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0de80b5680 con 0x7f0de80b4d00 2024-09-16T01:39:17.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.605+0000 7f0defb6d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0de815bd10 con 0x7f0de80b4920 2024-09-16T01:39:17.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.605+0000 7f0defb6d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0de80cf590 con 0x7f0de80cd780 2024-09-16T01:39:17.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.605+0000 7f0ded108640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0de80b4d00 0x7f0de814e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:17.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.605+0000 7f0ded108640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0de80b4d00 0x7f0de814e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37700/0 (socket says 172.21.15.88:37700) 2024-09-16T01:39:17.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.605+0000 7f0ded108640 1 -- 172.21.15.88:0/1806374458 learned_addr learned my addr 172.21.15.88:0/1806374458 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:17.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.605+0000 7f0dee10a640 1 --2- 172.21.15.88:0/1806374458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0de80cd780 0x7f0de8149060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:17.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.605+0000 7f0ded909640 1 --2- 172.21.15.88:0/1806374458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0de80b4920 0x7f0de80aa480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:17.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.605+0000 7f0ded108640 1 -- 172.21.15.88:0/1806374458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0de80cd780 msgr2=0x7f0de8149060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:17.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.605+0000 7f0ded108640 1 --2- 172.21.15.88:0/1806374458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0de80cd780 0x7f0de8149060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:17.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.605+0000 7f0ded108640 1 -- 172.21.15.88:0/1806374458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0de80b4920 msgr2=0x7f0de80aa480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:17.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.605+0000 7f0ded108640 1 --2- 172.21.15.88:0/1806374458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0de80b4920 0x7f0de80aa480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:17.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.605+0000 7f0ded108640 1 -- 172.21.15.88:0/1806374458 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0de80cac10 con 0x7f0de80b4d00 2024-09-16T01:39:17.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.605+0000 7f0ded108640 1 --2- 172.21.15.88:0/1806374458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0de80b4d00 0x7f0de814e880 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f0dd800ea00 tx=0x7f0dd800eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:17.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.606+0000 7f0dde7fc640 1 -- 172.21.15.88:0/1806374458 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8017070 con 0x7f0de80b4d00 2024-09-16T01:39:17.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.606+0000 7f0dde7fc640 1 -- 172.21.15.88:0/1806374458 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dd800ca60 con 0x7f0de80b4d00 2024-09-16T01:39:17.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.606+0000 7f0defb6d640 1 -- 172.21.15.88:0/1806374458 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0de80caef0 con 0x7f0de80b4d00 2024-09-16T01:39:17.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.606+0000 7f0dde7fc640 1 -- 172.21.15.88:0/1806374458 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd800ce10 con 0x7f0de80b4d00 2024-09-16T01:39:17.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.606+0000 7f0defb6d640 1 -- 172.21.15.88:0/1806374458 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0de80cb4b0 con 0x7f0de80b4d00 2024-09-16T01:39:17.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.607+0000 7f0defb6d640 1 -- 172.21.15.88:0/1806374458 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0de815b7d0 con 0x7f0de80b4d00 2024-09-16T01:39:17.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.607+0000 7f0dde7fc640 1 -- 172.21.15.88:0/1806374458 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0dd801a070 con 0x7f0de80b4d00 2024-09-16T01:39:17.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.608+0000 7f0dde7fc640 1 --2- 172.21.15.88:0/1806374458 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0dcc06c2b0 0x7f0dcc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:17.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.608+0000 7f0dde7fc640 1 -- 172.21.15.88:0/1806374458 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f0dd808b610 con 0x7f0de80b4d00 2024-09-16T01:39:17.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.608+0000 7f0ded909640 1 --2- 172.21.15.88:0/1806374458 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0dcc06c2b0 0x7f0dcc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:17.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.608+0000 7f0ded909640 1 --2- 172.21.15.88:0/1806374458 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0dcc06c2b0 0x7f0dcc06e770 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0de4034000 tx=0x7f0de402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:17.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.612+0000 7f0dde7fc640 1 -- 172.21.15.88:0/1806374458 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0dd80597c0 con 0x7f0de80b4d00 2024-09-16T01:39:17.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.812+0000 7f0defb6d640 1 -- 172.21.15.88:0/1806374458 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0de80b45a0 con 0x7f0de80b4d00 2024-09-16T01:39:17.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.812+0000 7f0dde7fc640 1 -- 172.21.15.88:0/1806374458 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+5231 (secure 0 0 0) 0x7f0dd805dac0 con 0x7f0de80b4d00 2024-09-16T01:39:17.814 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:17.814 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":57,"btime":"2024-09-16T01:39:17:483517+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5725,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2696547723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2696547723},{"type":"v1","addr":"172.21.15.154:6834","nonce":2696547723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54},{"gid":6411,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3276199399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3276199399},{"type":"v1","addr":"172.21.15.88:6836","nonce":3276199399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54},{"gid":6414,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2944989249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2944989249},{"type":"v1","addr":"172.21.15.88:6837","nonce":2944989249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:39:14.808422+0000","modified":"2024-09-16T01:39:16.811457+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6429":{"gid":6429,"name":"d","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6835/3482042494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":3482042494},{"type":"v1","addr":"172.21.15.154:6835","nonce":3482042494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6429,"qdb_cluster":[6429]},"id":5}]} 2024-09-16T01:39:17.814 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 57 2024-09-16T01:39:17.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.814+0000 7f0defb6d640 1 -- 172.21.15.88:0/1806374458 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0dcc06c2b0 msgr2=0x7f0dcc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:17.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.814+0000 7f0defb6d640 1 --2- 172.21.15.88:0/1806374458 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0dcc06c2b0 0x7f0dcc06e770 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f0de4034000 tx=0x7f0de402e000 comp rx=0 tx=0).stop 2024-09-16T01:39:17.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.814+0000 7f0defb6d640 1 -- 172.21.15.88:0/1806374458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0de80b4d00 msgr2=0x7f0de814e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:17.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.814+0000 7f0defb6d640 1 --2- 172.21.15.88:0/1806374458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0de80b4d00 0x7f0de814e880 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f0dd800ea00 tx=0x7f0dd800eec0 comp rx=0 tx=0).stop 2024-09-16T01:39:17.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.814+0000 7f0defb6d640 1 -- 172.21.15.88:0/1806374458 shutdown_connections 2024-09-16T01:39:17.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.814+0000 7f0defb6d640 1 --2- 172.21.15.88:0/1806374458 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0dcc06c2b0 0x7f0dcc06e770 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:17.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.814+0000 7f0defb6d640 1 --2- 172.21.15.88:0/1806374458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0de80cd780 0x7f0de8149060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:17.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.814+0000 7f0defb6d640 1 --2- 172.21.15.88:0/1806374458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0de80b4d00 0x7f0de814e880 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:17.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.814+0000 7f0defb6d640 1 --2- 172.21.15.88:0/1806374458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0de80b4920 0x7f0de80aa480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:17.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.814+0000 7f0defb6d640 1 -- 172.21.15.88:0/1806374458 >> 172.21.15.88:0/1806374458 conn(0x7f0de806bc10 msgr2=0x7f0de8070f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:17.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.815+0000 7f0defb6d640 1 -- 172.21.15.88:0/1806374458 shutdown_connections 2024-09-16T01:39:17.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.815+0000 7f0defb6d640 1 -- 172.21.15.88:0/1806374458 wait complete. 2024-09-16T01:39:17.826 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:39:17.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.922+0000 7ff3b716d640 1 Processor -- start 2024-09-16T01:39:17.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.922+0000 7ff3b716d640 1 -- start start 2024-09-16T01:39:17.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.923+0000 7ff3b716d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3b00b4920 0x7ff3b015b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:17.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.923+0000 7ff3b716d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff3b00b4d00 0x7ff3b00b5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:17.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.923+0000 7ff3b716d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3b00ce4f0 con 0x7ff3b00cd780 2024-09-16T01:39:17.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.923+0000 7ff3b716d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff3b00cf100 con 0x7ff3b00b4d00 2024-09-16T01:39:17.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.923+0000 7ff3b716d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff3b00ce7d0 con 0x7ff3b00b4920 2024-09-16T01:39:17.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.923+0000 7ff3a7fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff3b00b4d00 0x7ff3b00b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:17.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.923+0000 7ff3b570a640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff3b00cd780 0x7ff3b00cdbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42064/0 (socket says 172.21.15.88:42064) 2024-09-16T01:39:17.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.923+0000 7ff3b570a640 1 -- 172.21.15.88:0/3137062513 learned_addr learned my addr 172.21.15.88:0/3137062513 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:17.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.923+0000 7ff3a7fff640 1 -- 172.21.15.88:0/3137062513 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3b00b4920 msgr2=0x7ff3b015b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:17.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.923+0000 7ff3b4f09640 1 --2- 172.21.15.88:0/3137062513 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3b00b4920 0x7ff3b015b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:17.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.923+0000 7ff3a77fe640 1 -- 172.21.15.88:0/3137062513 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 97174623 0 0) 0x7ff3b00ce4f0 con 0x7ff3b00cd780 2024-09-16T01:39:17.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.923+0000 7ff3a7fff640 1 --2- 172.21.15.88:0/3137062513 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3b00b4920 0x7ff3b015b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:17.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.923+0000 7ff3a7fff640 1 -- 172.21.15.88:0/3137062513 >> v1:172.21.15.88:6789/0 conn(0x7ff3b00cd780 legacy=0x7ff3b00cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:17.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.923+0000 7ff3a7fff640 1 -- 172.21.15.88:0/3137062513 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3b00ceab0 con 0x7ff3b00b4d00 2024-09-16T01:39:17.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.924+0000 7ff3a7fff640 1 --2- 172.21.15.88:0/3137062513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b00b4d00 0x7ff3b00b5140 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7ff3a00097a0 tx=0x7ff3a002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9af8c58e192d7f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:17.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.924+0000 7ff3a77fe640 1 -- 172.21.15.88:0/3137062513 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0037070 con 0x7ff3b00b4d00 2024-09-16T01:39:17.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.924+0000 7ff3a77fe640 1 -- 172.21.15.88:0/3137062513 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3a0032440 con 0x7ff3b00b4d00 2024-09-16T01:39:17.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.924+0000 7ff3a77fe640 1 -- 172.21.15.88:0/3137062513 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0032810 con 0x7ff3b00b4d00 2024-09-16T01:39:17.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.925+0000 7ff3b716d640 1 -- 172.21.15.88:0/3137062513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b00b4d00 msgr2=0x7ff3b00b5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:17.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.925+0000 7ff3b716d640 1 --2- 172.21.15.88:0/3137062513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b00b4d00 0x7ff3b00b5140 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7ff3a00097a0 tx=0x7ff3a002a180 comp rx=0 tx=0).stop 2024-09-16T01:39:17.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.925+0000 7ff3b716d640 1 -- 172.21.15.88:0/3137062513 shutdown_connections 2024-09-16T01:39:17.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.925+0000 7ff3b716d640 1 --2- 172.21.15.88:0/3137062513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b00b4d00 0x7ff3b00b5140 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:17.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.925+0000 7ff3b716d640 1 --2- 172.21.15.88:0/3137062513 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3b00b4920 0x7ff3b015b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:17.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.925+0000 7ff3b716d640 1 -- 172.21.15.88:0/3137062513 >> 172.21.15.88:0/3137062513 conn(0x7ff3b006bc10 msgr2=0x7ff3b006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:17.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.925+0000 7ff3b716d640 1 -- 172.21.15.88:0/3137062513 shutdown_connections 2024-09-16T01:39:17.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.925+0000 7ff3b716d640 1 -- 172.21.15.88:0/3137062513 wait complete. 2024-09-16T01:39:17.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.926+0000 7ff3b716d640 1 Processor -- start 2024-09-16T01:39:17.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.926+0000 7ff3b716d640 1 -- start start 2024-09-16T01:39:17.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.927+0000 7ff3b716d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b00b4920 0x7ff3b01427a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:17.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.927+0000 7ff3b716d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3b00b4d00 0x7ff3b015d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:17.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.927+0000 7ff3b716d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3b00cd780 0x7ff3b014f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:17.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.927+0000 7ff3b716d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b00b5680 con 0x7ff3b00b4920 2024-09-16T01:39:17.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.927+0000 7ff3b716d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b015bd10 con 0x7ff3b00b4d00 2024-09-16T01:39:17.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.927+0000 7ff3b716d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff3b00cf590 con 0x7ff3b00cd780 2024-09-16T01:39:17.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.927+0000 7ff3b4f09640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b00b4920 0x7ff3b01427a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:17.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.927+0000 7ff3b4f09640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b00b4920 0x7ff3b01427a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37712/0 (socket says 172.21.15.88:37712) 2024-09-16T01:39:17.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.927+0000 7ff3b4f09640 1 -- 172.21.15.88:0/2846000797 learned_addr learned my addr 172.21.15.88:0/2846000797 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:17.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.927+0000 7ff3b570a640 1 --2- 172.21.15.88:0/2846000797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3b00cd780 0x7ff3b014f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:17.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.927+0000 7ff3b4f09640 1 -- 172.21.15.88:0/2846000797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3b00cd780 msgr2=0x7ff3b014f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:17.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.927+0000 7ff3b4f09640 1 --2- 172.21.15.88:0/2846000797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3b00cd780 0x7ff3b014f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:17.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.927+0000 7ff3b4f09640 1 -- 172.21.15.88:0/2846000797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3b00b4d00 msgr2=0x7ff3b015d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:17.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.927+0000 7ff3a7fff640 1 --2- 172.21.15.88:0/2846000797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3b00b4d00 0x7ff3b015d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:17.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.927+0000 7ff3b4f09640 1 --2- 172.21.15.88:0/2846000797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3b00b4d00 0x7ff3b015d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:17.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.927+0000 7ff3b4f09640 1 -- 172.21.15.88:0/2846000797 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3b0150e30 con 0x7ff3b00b4920 2024-09-16T01:39:17.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.927+0000 7ff3b4f09640 1 --2- 172.21.15.88:0/2846000797 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b00b4920 0x7ff3b01427a0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac00c820 tx=0x7ff3ac00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:17.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.927+0000 7ff3a57fa640 1 -- 172.21.15.88:0/2846000797 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac015070 con 0x7ff3b00b4920 2024-09-16T01:39:17.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.927+0000 7ff3a57fa640 1 -- 172.21.15.88:0/2846000797 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3ac004240 con 0x7ff3b00b4920 2024-09-16T01:39:17.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.927+0000 7ff3b716d640 1 -- 172.21.15.88:0/2846000797 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3b0151110 con 0x7ff3b00b4920 2024-09-16T01:39:17.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.928+0000 7ff3a57fa640 1 -- 172.21.15.88:0/2846000797 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac00ce10 con 0x7ff3b00b4920 2024-09-16T01:39:17.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.928+0000 7ff3b716d640 1 -- 172.21.15.88:0/2846000797 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3b01496f0 con 0x7ff3b00b4920 2024-09-16T01:39:17.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.929+0000 7ff3b716d640 1 -- 172.21.15.88:0/2846000797 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3b015b7d0 con 0x7ff3b00b4920 2024-09-16T01:39:17.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.929+0000 7ff3a57fa640 1 -- 172.21.15.88:0/2846000797 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff3ac007830 con 0x7ff3b00b4920 2024-09-16T01:39:17.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.929+0000 7ff3a57fa640 1 --2- 172.21.15.88:0/2846000797 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff39c06c2b0 0x7ff39c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:17.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.930+0000 7ff3a7fff640 1 --2- 172.21.15.88:0/2846000797 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff39c06c2b0 0x7ff39c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:17.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.930+0000 7ff3a57fa640 1 -- 172.21.15.88:0/2846000797 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7ff3ac08b300 con 0x7ff3b00b4920 2024-09-16T01:39:17.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.930+0000 7ff3a7fff640 1 --2- 172.21.15.88:0/2846000797 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff39c06c2b0 0x7ff39c06e770 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0034000 tx=0x7ff3a002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:17.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:17.934+0000 7ff3a57fa640 1 -- 172.21.15.88:0/2846000797 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff3ac059a70 con 0x7ff3b00b4920 2024-09-16T01:39:18.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.099+0000 7ff3b716d640 1 -- 172.21.15.88:0/2846000797 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff3b014df90 con 0x7ff3b00b4920 2024-09-16T01:39:18.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.100+0000 7ff3a57fa640 1 -- 172.21.15.88:0/2846000797 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+16704 (secure 0 0 0) 0x7ff3ac059370 con 0x7ff3b00b4920 2024-09-16T01:39:18.101 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:18.102 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":72,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:39:16.465669+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:39:13.412390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-16T01:39:13.777039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:39:18.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.102+0000 7ff3b716d640 1 -- 172.21.15.88:0/2846000797 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff39c06c2b0 msgr2=0x7ff39c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:18.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.102+0000 7ff3b716d640 1 --2- 172.21.15.88:0/2846000797 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff39c06c2b0 0x7ff39c06e770 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0034000 tx=0x7ff3a002e000 comp rx=0 tx=0).stop 2024-09-16T01:39:18.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.102+0000 7ff3b716d640 1 -- 172.21.15.88:0/2846000797 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b00b4920 msgr2=0x7ff3b01427a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:18.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.102+0000 7ff3b716d640 1 --2- 172.21.15.88:0/2846000797 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b00b4920 0x7ff3b01427a0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac00c820 tx=0x7ff3ac00cce0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.102+0000 7ff3b716d640 1 -- 172.21.15.88:0/2846000797 shutdown_connections 2024-09-16T01:39:18.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.102+0000 7ff3b716d640 1 --2- 172.21.15.88:0/2846000797 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff39c06c2b0 0x7ff39c06e770 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.102+0000 7ff3b716d640 1 --2- 172.21.15.88:0/2846000797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3b00cd780 0x7ff3b014f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.102+0000 7ff3b716d640 1 --2- 172.21.15.88:0/2846000797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3b00b4d00 0x7ff3b015d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.102+0000 7ff3b716d640 1 --2- 172.21.15.88:0/2846000797 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff3b00b4920 0x7ff3b01427a0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.102+0000 7ff3b716d640 1 -- 172.21.15.88:0/2846000797 >> 172.21.15.88:0/2846000797 conn(0x7ff3b006bc10 msgr2=0x7ff3b00cdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:18.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.102+0000 7ff3b716d640 1 -- 172.21.15.88:0/2846000797 shutdown_connections 2024-09-16T01:39:18.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.102+0000 7ff3b716d640 1 -- 172.21.15.88:0/2846000797 wait complete. 2024-09-16T01:39:18.114 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T01:39:18.114 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:39:18.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.211+0000 7f634f191640 1 Processor -- start 2024-09-16T01:39:18.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.211+0000 7f634f191640 1 -- start start 2024-09-16T01:39:18.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.212+0000 7f634f191640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f634815ddc0 0x7f6348167100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:18.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.212+0000 7f634f191640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6348072760 con 0x7f634815c710 2024-09-16T01:39:18.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.212+0000 7f634f191640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6348072a30 con 0x7f634806bc10 2024-09-16T01:39:18.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.212+0000 7f634f191640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6348072d10 con 0x7f634815ddc0 2024-09-16T01:39:18.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.212+0000 7f6347fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f634815c710 0x7f6348072050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42070/0 (socket says 172.21.15.88:42070) 2024-09-16T01:39:18.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.212+0000 7f6347fff640 1 -- 172.21.15.88:0/2337724546 learned_addr learned my addr 172.21.15.88:0/2337724546 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:18.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.212+0000 7f63477fe640 1 -- 172.21.15.88:0/2337724546 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1324071327 0 0) 0x7f6348072760 con 0x7f634815c710 2024-09-16T01:39:18.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.212+0000 7f63477fe640 1 -- 172.21.15.88:0/2337724546 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6334003520 con 0x7f634815c710 2024-09-16T01:39:18.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.212+0000 7f634d72e640 1 --2- 172.21.15.88:0/2337724546 >> v2:172.21.15.154:3300/0 conn(0x7f634815ddc0 0x7f6348167100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:18.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.212+0000 7f63477fe640 1 -- 172.21.15.88:0/2337724546 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3535976735 0 0) 0x7f6334003520 con 0x7f634815c710 2024-09-16T01:39:18.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.212+0000 7f63477fe640 1 -- 172.21.15.88:0/2337724546 >> v2:172.21.15.154:3300/0 conn(0x7f634815ddc0 msgr2=0x7f6348167100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:18.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.212+0000 7f63477fe640 1 --2- 172.21.15.88:0/2337724546 >> v2:172.21.15.154:3300/0 conn(0x7f634815ddc0 0x7f6348167100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.212+0000 7f63477fe640 1 -- 172.21.15.88:0/2337724546 >> v1:172.21.15.154:6789/0 conn(0x7f634806bc10 legacy=0x7f634806bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:18.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.212+0000 7f63477fe640 1 -- 172.21.15.88:0/2337724546 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6348072ff0 con 0x7f634815c710 2024-09-16T01:39:18.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.212+0000 7f63477fe640 1 -- 172.21.15.88:0/2337724546 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f63380019f0 con 0x7f634815c710 2024-09-16T01:39:18.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.212+0000 7f63477fe640 1 -- 172.21.15.88:0/2337724546 >> v1:172.21.15.88:6789/0 conn(0x7f634815c710 legacy=0x7f6348072050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:18.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.212+0000 7f63477fe640 1 --2- 172.21.15.88:0/2337724546 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63340039b0 0x7f6334003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:18.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.212+0000 7f63477fe640 1 --2- 172.21.15.88:0/2337724546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6334004800 0x7f6334004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:18.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.212+0000 7f63477fe640 1 --2- 172.21.15.88:0/2337724546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63340051c0 0x7f633400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:18.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.212+0000 7f63477fe640 1 -- 172.21.15.88:0/2337724546 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6348072d10 con 0x7f63340051c0 2024-09-16T01:39:18.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.212+0000 7f63477fe640 1 -- 172.21.15.88:0/2337724546 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f633400c110 con 0x7f6334004800 2024-09-16T01:39:18.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.212+0000 7f63477fe640 1 -- 172.21.15.88:0/2337724546 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f633400c310 con 0x7f63340039b0 2024-09-16T01:39:18.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.212+0000 7f634d72e640 1 --2- 172.21.15.88:0/2337724546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63340051c0 0x7f633400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:18.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.213+0000 7f634cf2d640 1 --2- 172.21.15.88:0/2337724546 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63340039b0 0x7f6334003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:18.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.213+0000 7f6347fff640 1 --2- 172.21.15.88:0/2337724546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6334004800 0x7f6334004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:18.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.213+0000 7f634d72e640 1 -- 172.21.15.88:0/2337724546 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63340039b0 msgr2=0x7f6334003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:18.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.213+0000 7f634d72e640 1 --2- 172.21.15.88:0/2337724546 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63340039b0 0x7f6334003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.213+0000 7f634d72e640 1 -- 172.21.15.88:0/2337724546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6334004800 msgr2=0x7f6334004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:18.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.213+0000 7f634d72e640 1 --2- 172.21.15.88:0/2337724546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6334004800 0x7f6334004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.213+0000 7f634d72e640 1 -- 172.21.15.88:0/2337724546 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6334003520 con 0x7f63340051c0 2024-09-16T01:39:18.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.213+0000 7f634d72e640 1 -- 172.21.15.88:0/2337724546 reap_dead start 2024-09-16T01:39:18.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.213+0000 7f634d72e640 1 --2- 172.21.15.88:0/2337724546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63340051c0 0x7f633400b9c0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f633000ba10 tx=0x7f6334005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db792467eea92fad server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:18.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.213+0000 7f63477fe640 1 -- 172.21.15.88:0/2337724546 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6330030070 con 0x7f63340051c0 2024-09-16T01:39:18.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.214+0000 7f63477fe640 1 -- 172.21.15.88:0/2337724546 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f633002ae00 con 0x7f63340051c0 2024-09-16T01:39:18.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.214+0000 7f63477fe640 1 -- 172.21.15.88:0/2337724546 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633000b610 con 0x7f63340051c0 2024-09-16T01:39:18.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.214+0000 7f634f191640 1 -- 172.21.15.88:0/2337724546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63340051c0 msgr2=0x7f633400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:18.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.214+0000 7f634f191640 1 --2- 172.21.15.88:0/2337724546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63340051c0 0x7f633400b9c0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f633000ba10 tx=0x7f6334005830 comp rx=0 tx=0).stop 2024-09-16T01:39:18.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.214+0000 7f634f191640 1 -- 172.21.15.88:0/2337724546 shutdown_connections 2024-09-16T01:39:18.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.214+0000 7f634f191640 1 --2- 172.21.15.88:0/2337724546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63340051c0 0x7f633400b9c0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.214+0000 7f634f191640 1 -- 172.21.15.88:0/2337724546 >> 172.21.15.88:0/2337724546 conn(0x7f63480d14d0 msgr2=0x7f63480d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:18.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.214+0000 7f634f191640 1 -- 172.21.15.88:0/2337724546 shutdown_connections 2024-09-16T01:39:18.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.214+0000 7f634f191640 1 -- 172.21.15.88:0/2337724546 wait complete. 2024-09-16T01:39:18.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.215+0000 7f634f191640 1 Processor -- start 2024-09-16T01:39:18.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.215+0000 7f634f191640 1 -- start start 2024-09-16T01:39:18.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.216+0000 7f634f191640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63340051c0 0x7f63481f7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:18.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.216+0000 7f634f191640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63480cc530 0x7f63481f7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:18.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.216+0000 7f634f191640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63480cefe0 0x7f63480cf470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:18.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.216+0000 7f634f191640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f634806f850 con 0x7f63480cc530 2024-09-16T01:39:18.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.216+0000 7f634f191640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f63480735b0 con 0x7f63480cefe0 2024-09-16T01:39:18.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.216+0000 7f634f191640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f634806f9d0 con 0x7f63340051c0 2024-09-16T01:39:18.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.216+0000 7f6347fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63480cc530 0x7f63481f7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:18.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.216+0000 7f6347fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63480cc530 0x7f63481f7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37736/0 (socket says 172.21.15.88:37736) 2024-09-16T01:39:18.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.216+0000 7f6347fff640 1 -- 172.21.15.88:0/2684216785 learned_addr learned my addr 172.21.15.88:0/2684216785 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:18.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.217+0000 7f634d72e640 1 --2- 172.21.15.88:0/2684216785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63480cefe0 0x7f63480cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:18.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.217+0000 7f634cf2d640 1 --2- 172.21.15.88:0/2684216785 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63340051c0 0x7f63481f7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:18.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.217+0000 7f6347fff640 1 -- 172.21.15.88:0/2684216785 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63340051c0 msgr2=0x7f63481f7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:18.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.217+0000 7f6347fff640 1 --2- 172.21.15.88:0/2684216785 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63340051c0 0x7f63481f7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.217+0000 7f6347fff640 1 -- 172.21.15.88:0/2684216785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63480cefe0 msgr2=0x7f63480cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:18.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.217+0000 7f6347fff640 1 --2- 172.21.15.88:0/2684216785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63480cefe0 0x7f63480cf470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.217+0000 7f6347fff640 1 -- 172.21.15.88:0/2684216785 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63480cca30 con 0x7f63480cc530 2024-09-16T01:39:18.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.217+0000 7f634cf2d640 1 --2- 172.21.15.88:0/2684216785 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63340051c0 0x7f63481f7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:18.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.217+0000 7f6347fff640 1 --2- 172.21.15.88:0/2684216785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63480cc530 0x7f63481f7b50 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f6338001330 tx=0x7f633800a360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:18.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.217+0000 7f63457fa640 1 -- 172.21.15.88:0/2684216785 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6338017020 con 0x7f63480cc530 2024-09-16T01:39:18.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.217+0000 7f63457fa640 1 -- 172.21.15.88:0/2684216785 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f633800ada0 con 0x7f63480cc530 2024-09-16T01:39:18.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.217+0000 7f63457fa640 1 -- 172.21.15.88:0/2684216785 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6338013610 con 0x7f63480cc530 2024-09-16T01:39:18.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.217+0000 7f634f191640 1 -- 172.21.15.88:0/2684216785 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63480ccd10 con 0x7f63480cc530 2024-09-16T01:39:18.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.217+0000 7f634f191640 1 -- 172.21.15.88:0/2684216785 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63480cd2d0 con 0x7f63480cc530 2024-09-16T01:39:18.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.218+0000 7f634f191640 1 -- 172.21.15.88:0/2684216785 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f634806bff0 con 0x7f63480cc530 2024-09-16T01:39:18.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.218+0000 7f63457fa640 1 -- 172.21.15.88:0/2684216785 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f633801a020 con 0x7f63480cc530 2024-09-16T01:39:18.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.219+0000 7f63457fa640 1 --2- 172.21.15.88:0/2684216785 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f632806c2b0 0x7f632806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:18.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.219+0000 7f63457fa640 1 -- 172.21.15.88:0/2684216785 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f633808bf20 con 0x7f63480cc530 2024-09-16T01:39:18.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.219+0000 7f634cf2d640 1 --2- 172.21.15.88:0/2684216785 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f632806c2b0 0x7f632806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:18.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.219+0000 7f634cf2d640 1 --2- 172.21.15.88:0/2684216785 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f632806c2b0 0x7f632806e770 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6340000c00 tx=0x7f6340009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:18.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.223+0000 7f63457fa640 1 -- 172.21.15.88:0/2684216785 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6338021020 con 0x7f63480cc530 2024-09-16T01:39:18.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.383+0000 7f634f191640 1 -- 172.21.15.88:0/2684216785 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f634815e800 con 0x7f632806c2b0 2024-09-16T01:39:18.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.389+0000 7f63457fa640 1 -- 172.21.15.88:0/2684216785 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252006 (secure 0 0 0) 0x7f634815e800 con 0x7f632806c2b0 2024-09-16T01:39:18.393 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:18.395 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:39:18.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.392+0000 7f634f191640 1 -- 172.21.15.88:0/2684216785 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f632806c2b0 msgr2=0x7f632806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:18.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.392+0000 7f634f191640 1 --2- 172.21.15.88:0/2684216785 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f632806c2b0 0x7f632806e770 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6340000c00 tx=0x7f6340009040 comp rx=0 tx=0).stop 2024-09-16T01:39:18.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.392+0000 7f634f191640 1 -- 172.21.15.88:0/2684216785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63480cc530 msgr2=0x7f63481f7b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:18.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.392+0000 7f634f191640 1 --2- 172.21.15.88:0/2684216785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63480cc530 0x7f63481f7b50 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f6338001330 tx=0x7f633800a360 comp rx=0 tx=0).stop 2024-09-16T01:39:18.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.392+0000 7f634f191640 1 -- 172.21.15.88:0/2684216785 shutdown_connections 2024-09-16T01:39:18.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.392+0000 7f634f191640 1 --2- 172.21.15.88:0/2684216785 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f632806c2b0 0x7f632806e770 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.392+0000 7f634f191640 1 --2- 172.21.15.88:0/2684216785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63480cefe0 0x7f63480cf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.392+0000 7f634f191640 1 --2- 172.21.15.88:0/2684216785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63480cc530 0x7f63481f7b50 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.392+0000 7f634f191640 1 --2- 172.21.15.88:0/2684216785 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63340051c0 0x7f63481f7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.392+0000 7f634f191640 1 -- 172.21.15.88:0/2684216785 >> 172.21.15.88:0/2684216785 conn(0x7f63480d14d0 msgr2=0x7f634806c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:18.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.392+0000 7f634f191640 1 -- 172.21.15.88:0/2684216785 shutdown_connections 2024-09-16T01:39:18.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.392+0000 7f634f191640 1 -- 172.21.15.88:0/2684216785 wait complete. 2024-09-16T01:39:18.404 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2024-09-16T01:39:16.583575+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":232400,"kb_used_data":7816,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754742320,"statfs":{"total":773094113280,"available":772856135680,"internally_reserved":0,"allocated":8003584,"data_stored":4046278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":229756466},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000521"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.502006+0000","last_change":"2024-09-16T01:39:15.500162+0000","last_active":"2024-09-16T01:39:15.502006+0000","last_peered":"2024-09-16T01:39:15.502006+0000","last_clean":"2024-09-16T01:39:15.502006+0000","last_became_active":"2024-09-16T01:39:15.496318+0000","last_became_peered":"2024-09-16T01:39:15.496318+0000","last_unstale":"2024-09-16T01:39:15.502006+0000","last_undegraded":"2024-09-16T01:39:15.502006+0000","last_fullsized":"2024-09-16T01:39:15.502006+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:44.054894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.497912+0000","last_change":"2024-09-16T01:39:15.495072+0000","last_active":"2024-09-16T01:39:15.497912+0000","last_peered":"2024-09-16T01:39:15.497912+0000","last_clean":"2024-09-16T01:39:15.497912+0000","last_became_active":"2024-09-16T01:39:15.493376+0000","last_became_peered":"2024-09-16T01:39:15.493376+0000","last_unstale":"2024-09-16T01:39:15.497912+0000","last_undegraded":"2024-09-16T01:39:15.497912+0000","last_fullsized":"2024-09-16T01:39:15.497912+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:15:40.953770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.482523+0000","last_change":"2024-09-16T01:39:15.481136+0000","last_active":"2024-09-16T01:39:15.482523+0000","last_peered":"2024-09-16T01:39:15.482523+0000","last_clean":"2024-09-16T01:39:15.482523+0000","last_became_active":"2024-09-16T01:39:15.480891+0000","last_became_peered":"2024-09-16T01:39:15.480891+0000","last_unstale":"2024-09-16T01:39:15.482523+0000","last_undegraded":"2024-09-16T01:39:15.482523+0000","last_fullsized":"2024-09-16T01:39:15.482523+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:10:00.747727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.492345+0000","last_change":"2024-09-16T01:39:15.483406+0000","last_active":"2024-09-16T01:39:15.492345+0000","last_peered":"2024-09-16T01:39:15.492345+0000","last_clean":"2024-09-16T01:39:15.492345+0000","last_became_active":"2024-09-16T01:39:15.483143+0000","last_became_peered":"2024-09-16T01:39:15.483143+0000","last_unstale":"2024-09-16T01:39:15.492345+0000","last_undegraded":"2024-09-16T01:39:15.492345+0000","last_fullsized":"2024-09-16T01:39:15.492345+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:33:51.100506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.496029+0000","last_change":"2024-09-16T01:39:15.488889+0000","last_active":"2024-09-16T01:39:15.496029+0000","last_peered":"2024-09-16T01:39:15.496029+0000","last_clean":"2024-09-16T01:39:15.496029+0000","last_became_active":"2024-09-16T01:39:15.488539+0000","last_became_peered":"2024-09-16T01:39:15.488539+0000","last_unstale":"2024-09-16T01:39:15.496029+0000","last_undegraded":"2024-09-16T01:39:15.496029+0000","last_fullsized":"2024-09-16T01:39:15.496029+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:08:41.278079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.491684+0000","last_change":"2024-09-16T01:39:15.483949+0000","last_active":"2024-09-16T01:39:15.491684+0000","last_peered":"2024-09-16T01:39:15.491684+0000","last_clean":"2024-09-16T01:39:15.491684+0000","last_became_active":"2024-09-16T01:39:15.483408+0000","last_became_peered":"2024-09-16T01:39:15.483408+0000","last_unstale":"2024-09-16T01:39:15.491684+0000","last_undegraded":"2024-09-16T01:39:15.491684+0000","last_fullsized":"2024-09-16T01:39:15.491684+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:02:52.162549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T01:39:16.487082+0000","last_change":"2024-09-16T01:39:15.483151+0000","last_active":"2024-09-16T01:39:16.487082+0000","last_peered":"2024-09-16T01:39:16.487082+0000","last_clean":"2024-09-16T01:39:16.487082+0000","last_became_active":"2024-09-16T01:39:15.482173+0000","last_became_peered":"2024-09-16T01:39:15.482173+0000","last_unstale":"2024-09-16T01:39:16.487082+0000","last_undegraded":"2024-09-16T01:39:16.487082+0000","last_fullsized":"2024-09-16T01:39:16.487082+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:11:02.011376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.485573+0000","last_change":"2024-09-16T01:39:15.480491+0000","last_active":"2024-09-16T01:39:15.485573+0000","last_peered":"2024-09-16T01:39:15.485573+0000","last_clean":"2024-09-16T01:39:15.485573+0000","last_became_active":"2024-09-16T01:39:15.480150+0000","last_became_peered":"2024-09-16T01:39:15.480150+0000","last_unstale":"2024-09-16T01:39:15.485573+0000","last_undegraded":"2024-09-16T01:39:15.485573+0000","last_fullsized":"2024-09-16T01:39:15.485573+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:43.014904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.493553+0000","last_change":"2024-09-16T01:39:15.480553+0000","last_active":"2024-09-16T01:39:15.493553+0000","last_peered":"2024-09-16T01:39:15.493553+0000","last_clean":"2024-09-16T01:39:15.493553+0000","last_became_active":"2024-09-16T01:39:15.478968+0000","last_became_peered":"2024-09-16T01:39:15.478968+0000","last_unstale":"2024-09-16T01:39:15.493553+0000","last_undegraded":"2024-09-16T01:39:15.493553+0000","last_fullsized":"2024-09-16T01:39:15.493553+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:50.764141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.501155+0000","last_change":"2024-09-16T01:39:15.499685+0000","last_active":"2024-09-16T01:39:15.501155+0000","last_peered":"2024-09-16T01:39:15.501155+0000","last_clean":"2024-09-16T01:39:15.501155+0000","last_became_active":"2024-09-16T01:39:15.499409+0000","last_became_peered":"2024-09-16T01:39:15.499409+0000","last_unstale":"2024-09-16T01:39:15.501155+0000","last_undegraded":"2024-09-16T01:39:15.501155+0000","last_fullsized":"2024-09-16T01:39:15.501155+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:34:57.787684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.501512+0000","last_change":"2024-09-16T01:39:15.497417+0000","last_active":"2024-09-16T01:39:15.501512+0000","last_peered":"2024-09-16T01:39:15.501512+0000","last_clean":"2024-09-16T01:39:15.501512+0000","last_became_active":"2024-09-16T01:39:15.496647+0000","last_became_peered":"2024-09-16T01:39:15.496647+0000","last_unstale":"2024-09-16T01:39:15.501512+0000","last_undegraded":"2024-09-16T01:39:15.501512+0000","last_fullsized":"2024-09-16T01:39:15.501512+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:52.651374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.483557+0000","last_change":"2024-09-16T01:39:15.481113+0000","last_active":"2024-09-16T01:39:15.483557+0000","last_peered":"2024-09-16T01:39:15.483557+0000","last_clean":"2024-09-16T01:39:15.483557+0000","last_became_active":"2024-09-16T01:39:15.480810+0000","last_became_peered":"2024-09-16T01:39:15.480810+0000","last_unstale":"2024-09-16T01:39:15.483557+0000","last_undegraded":"2024-09-16T01:39:15.483557+0000","last_fullsized":"2024-09-16T01:39:15.483557+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:05:36.207551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.483326+0000","last_change":"2024-09-16T01:39:15.480642+0000","last_active":"2024-09-16T01:39:15.483326+0000","last_peered":"2024-09-16T01:39:15.483326+0000","last_clean":"2024-09-16T01:39:15.483326+0000","last_became_active":"2024-09-16T01:39:15.480371+0000","last_became_peered":"2024-09-16T01:39:15.480371+0000","last_unstale":"2024-09-16T01:39:15.483326+0000","last_undegraded":"2024-09-16T01:39:15.483326+0000","last_fullsized":"2024-09-16T01:39:15.483326+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:17:04.087123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.493265+0000","last_change":"2024-09-16T01:39:15.477843+0000","last_active":"2024-09-16T01:39:15.493265+0000","last_peered":"2024-09-16T01:39:15.493265+0000","last_clean":"2024-09-16T01:39:15.493265+0000","last_became_active":"2024-09-16T01:39:15.477559+0000","last_became_peered":"2024-09-16T01:39:15.477559+0000","last_unstale":"2024-09-16T01:39:15.493265+0000","last_undegraded":"2024-09-16T01:39:15.493265+0000","last_fullsized":"2024-09-16T01:39:15.493265+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:19.653901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.493941+0000","last_change":"2024-09-16T01:39:15.486422+0000","last_active":"2024-09-16T01:39:15.493941+0000","last_peered":"2024-09-16T01:39:15.493941+0000","last_clean":"2024-09-16T01:39:15.493941+0000","last_became_active":"2024-09-16T01:39:15.486128+0000","last_became_peered":"2024-09-16T01:39:15.486128+0000","last_unstale":"2024-09-16T01:39:15.493941+0000","last_undegraded":"2024-09-16T01:39:15.493941+0000","last_fullsized":"2024-09-16T01:39:15.493941+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:03:08.237538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.490505+0000","last_change":"2024-09-16T01:39:15.486426+0000","last_active":"2024-09-16T01:39:15.490505+0000","last_peered":"2024-09-16T01:39:15.490505+0000","last_clean":"2024-09-16T01:39:15.490505+0000","last_became_active":"2024-09-16T01:39:15.486138+0000","last_became_peered":"2024-09-16T01:39:15.486138+0000","last_unstale":"2024-09-16T01:39:15.490505+0000","last_undegraded":"2024-09-16T01:39:15.490505+0000","last_fullsized":"2024-09-16T01:39:15.490505+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:02:52.162549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.493515+0000","last_change":"2024-09-16T01:39:15.479421+0000","last_active":"2024-09-16T01:39:15.493515+0000","last_peered":"2024-09-16T01:39:15.493515+0000","last_clean":"2024-09-16T01:39:15.493515+0000","last_became_active":"2024-09-16T01:39:15.478127+0000","last_became_peered":"2024-09-16T01:39:15.478127+0000","last_unstale":"2024-09-16T01:39:15.493515+0000","last_undegraded":"2024-09-16T01:39:15.493515+0000","last_fullsized":"2024-09-16T01:39:15.493515+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:47:11.776578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.492177+0000","last_change":"2024-09-16T01:39:15.483906+0000","last_active":"2024-09-16T01:39:15.492177+0000","last_peered":"2024-09-16T01:39:15.492177+0000","last_clean":"2024-09-16T01:39:15.492177+0000","last_became_active":"2024-09-16T01:39:15.483639+0000","last_became_peered":"2024-09-16T01:39:15.483639+0000","last_unstale":"2024-09-16T01:39:15.492177+0000","last_undegraded":"2024-09-16T01:39:15.492177+0000","last_fullsized":"2024-09-16T01:39:15.492177+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:57:42.989890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.487191+0000","last_change":"2024-09-16T01:39:15.482961+0000","last_active":"2024-09-16T01:39:15.487191+0000","last_peered":"2024-09-16T01:39:15.487191+0000","last_clean":"2024-09-16T01:39:15.487191+0000","last_became_active":"2024-09-16T01:39:15.482687+0000","last_became_peered":"2024-09-16T01:39:15.482687+0000","last_unstale":"2024-09-16T01:39:15.487191+0000","last_undegraded":"2024-09-16T01:39:15.487191+0000","last_fullsized":"2024-09-16T01:39:15.487191+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:31:51.302505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.483150+0000","last_change":"2024-09-16T01:39:15.477677+0000","last_active":"2024-09-16T01:39:15.483150+0000","last_peered":"2024-09-16T01:39:15.483150+0000","last_clean":"2024-09-16T01:39:15.483150+0000","last_became_active":"2024-09-16T01:39:15.477388+0000","last_became_peered":"2024-09-16T01:39:15.477388+0000","last_unstale":"2024-09-16T01:39:15.483150+0000","last_undegraded":"2024-09-16T01:39:15.483150+0000","last_fullsized":"2024-09-16T01:39:15.483150+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:26:42.021902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.490931+0000","last_change":"2024-09-16T01:39:15.489962+0000","last_active":"2024-09-16T01:39:15.490931+0000","last_peered":"2024-09-16T01:39:15.490931+0000","last_clean":"2024-09-16T01:39:15.490931+0000","last_became_active":"2024-09-16T01:39:15.489667+0000","last_became_peered":"2024-09-16T01:39:15.489667+0000","last_unstale":"2024-09-16T01:39:15.490931+0000","last_undegraded":"2024-09-16T01:39:15.490931+0000","last_fullsized":"2024-09-16T01:39:15.490931+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:54:37.811306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.498343+0000","last_change":"2024-09-16T01:39:15.495324+0000","last_active":"2024-09-16T01:39:15.498343+0000","last_peered":"2024-09-16T01:39:15.498343+0000","last_clean":"2024-09-16T01:39:15.498343+0000","last_became_active":"2024-09-16T01:39:15.495052+0000","last_became_peered":"2024-09-16T01:39:15.495052+0000","last_unstale":"2024-09-16T01:39:15.498343+0000","last_undegraded":"2024-09-16T01:39:15.498343+0000","last_fullsized":"2024-09-16T01:39:15.498343+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:33:41.800796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.494989+0000","last_change":"2024-09-16T01:39:15.492957+0000","last_active":"2024-09-16T01:39:15.494989+0000","last_peered":"2024-09-16T01:39:15.494989+0000","last_clean":"2024-09-16T01:39:15.494989+0000","last_became_active":"2024-09-16T01:39:15.492700+0000","last_became_peered":"2024-09-16T01:39:15.492700+0000","last_unstale":"2024-09-16T01:39:15.494989+0000","last_undegraded":"2024-09-16T01:39:15.494989+0000","last_fullsized":"2024-09-16T01:39:15.494989+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:49:10.214476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.501988+0000","last_change":"2024-09-16T01:39:15.499849+0000","last_active":"2024-09-16T01:39:15.501988+0000","last_peered":"2024-09-16T01:39:15.501988+0000","last_clean":"2024-09-16T01:39:15.501988+0000","last_became_active":"2024-09-16T01:39:15.495301+0000","last_became_peered":"2024-09-16T01:39:15.495301+0000","last_unstale":"2024-09-16T01:39:15.501988+0000","last_undegraded":"2024-09-16T01:39:15.501988+0000","last_fullsized":"2024-09-16T01:39:15.501988+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:21:14.462992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.497396+0000","last_change":"2024-09-16T01:39:15.496321+0000","last_active":"2024-09-16T01:39:15.497396+0000","last_peered":"2024-09-16T01:39:15.497396+0000","last_clean":"2024-09-16T01:39:15.497396+0000","last_became_active":"2024-09-16T01:39:15.496025+0000","last_became_peered":"2024-09-16T01:39:15.496025+0000","last_unstale":"2024-09-16T01:39:15.497396+0000","last_undegraded":"2024-09-16T01:39:15.497396+0000","last_fullsized":"2024-09-16T01:39:15.497396+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:00:19.023789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T01:39:16.487416+0000","last_change":"2024-09-16T01:39:15.486415+0000","last_active":"2024-09-16T01:39:16.487416+0000","last_peered":"2024-09-16T01:39:16.487416+0000","last_clean":"2024-09-16T01:39:16.487416+0000","last_became_active":"2024-09-16T01:39:15.486130+0000","last_became_peered":"2024-09-16T01:39:15.486130+0000","last_unstale":"2024-09-16T01:39:16.487416+0000","last_undegraded":"2024-09-16T01:39:16.487416+0000","last_fullsized":"2024-09-16T01:39:16.487416+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:15:14.342280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.491782+0000","last_change":"2024-09-16T01:39:15.489875+0000","last_active":"2024-09-16T01:39:15.491782+0000","last_peered":"2024-09-16T01:39:15.491782+0000","last_clean":"2024-09-16T01:39:15.491782+0000","last_became_active":"2024-09-16T01:39:15.489552+0000","last_became_peered":"2024-09-16T01:39:15.489552+0000","last_unstale":"2024-09-16T01:39:15.491782+0000","last_undegraded":"2024-09-16T01:39:15.491782+0000","last_fullsized":"2024-09-16T01:39:15.491782+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:25:24.101060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.496913+0000","last_change":"2024-09-16T01:39:15.494872+0000","last_active":"2024-09-16T01:39:15.496913+0000","last_peered":"2024-09-16T01:39:15.496913+0000","last_clean":"2024-09-16T01:39:15.496913+0000","last_became_active":"2024-09-16T01:39:15.494596+0000","last_became_peered":"2024-09-16T01:39:15.494596+0000","last_unstale":"2024-09-16T01:39:15.496913+0000","last_undegraded":"2024-09-16T01:39:15.496913+0000","last_fullsized":"2024-09-16T01:39:15.496913+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:39:00.982561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.480669+0000","last_change":"2024-09-16T01:39:15.479157+0000","last_active":"2024-09-16T01:39:15.480669+0000","last_peered":"2024-09-16T01:39:15.480669+0000","last_clean":"2024-09-16T01:39:15.480669+0000","last_became_active":"2024-09-16T01:39:15.478839+0000","last_became_peered":"2024-09-16T01:39:15.478839+0000","last_unstale":"2024-09-16T01:39:15.480669+0000","last_undegraded":"2024-09-16T01:39:15.480669+0000","last_fullsized":"2024-09-16T01:39:15.480669+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.906974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T01:39:16.487140+0000","last_change":"2024-09-16T01:39:15.486645+0000","last_active":"2024-09-16T01:39:16.487140+0000","last_peered":"2024-09-16T01:39:16.487140+0000","last_clean":"2024-09-16T01:39:16.487140+0000","last_became_active":"2024-09-16T01:39:15.486343+0000","last_became_peered":"2024-09-16T01:39:15.486343+0000","last_unstale":"2024-09-16T01:39:16.487140+0000","last_undegraded":"2024-09-16T01:39:16.487140+0000","last_fullsized":"2024-09-16T01:39:16.487140+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:58:48.937237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T01:39:16.487293+0000","last_change":"2024-09-16T01:39:15.487946+0000","last_active":"2024-09-16T01:39:16.487293+0000","last_peered":"2024-09-16T01:39:16.487293+0000","last_clean":"2024-09-16T01:39:16.487293+0000","last_became_active":"2024-09-16T01:39:15.487649+0000","last_became_peered":"2024-09-16T01:39:15.487649+0000","last_unstale":"2024-09-16T01:39:16.487293+0000","last_undegraded":"2024-09-16T01:39:16.487293+0000","last_fullsized":"2024-09-16T01:39:16.487293+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:21:14.462992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.485545+0000","last_change":"2024-09-16T01:39:15.479771+0000","last_active":"2024-09-16T01:39:15.485545+0000","last_peered":"2024-09-16T01:39:15.485545+0000","last_clean":"2024-09-16T01:39:15.485545+0000","last_became_active":"2024-09-16T01:39:15.478711+0000","last_became_peered":"2024-09-16T01:39:15.478711+0000","last_unstale":"2024-09-16T01:39:15.485545+0000","last_undegraded":"2024-09-16T01:39:15.485545+0000","last_fullsized":"2024-09-16T01:39:15.485545+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:03:13.587790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.502001+0000","last_change":"2024-09-16T01:39:15.500372+0000","last_active":"2024-09-16T01:39:15.502001+0000","last_peered":"2024-09-16T01:39:15.502001+0000","last_clean":"2024-09-16T01:39:15.502001+0000","last_became_active":"2024-09-16T01:39:15.500129+0000","last_became_peered":"2024-09-16T01:39:15.500129+0000","last_unstale":"2024-09-16T01:39:15.502001+0000","last_undegraded":"2024-09-16T01:39:15.502001+0000","last_fullsized":"2024-09-16T01:39:15.502001+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:34:32.699174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"71'4","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.498317+0000","last_change":"2024-09-16T01:39:14.480906+0000","last_active":"2024-09-16T01:39:15.498317+0000","last_peered":"2024-09-16T01:39:15.498317+0000","last_clean":"2024-09-16T01:39:15.498317+0000","last_became_active":"2024-09-16T01:39:14.480606+0000","last_became_peered":"2024-09-16T01:39:14.480606+0000","last_unstale":"2024-09-16T01:39:15.498317+0000","last_undegraded":"2024-09-16T01:39:15.498317+0000","last_fullsized":"2024-09-16T01:39:15.498317+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:30:39.759873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.493537+0000","last_change":"2024-09-16T01:39:15.480300+0000","last_active":"2024-09-16T01:39:15.493537+0000","last_peered":"2024-09-16T01:39:15.493537+0000","last_clean":"2024-09-16T01:39:15.493537+0000","last_became_active":"2024-09-16T01:39:15.477956+0000","last_became_peered":"2024-09-16T01:39:15.477956+0000","last_unstale":"2024-09-16T01:39:15.493537+0000","last_undegraded":"2024-09-16T01:39:15.493537+0000","last_fullsized":"2024-09-16T01:39:15.493537+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:22.796258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"71'1","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.496417+0000","last_change":"2024-09-16T01:39:14.483678+0000","last_active":"2024-09-16T01:39:15.496417+0000","last_peered":"2024-09-16T01:39:15.496417+0000","last_clean":"2024-09-16T01:39:15.496417+0000","last_became_active":"2024-09-16T01:39:14.483390+0000","last_became_peered":"2024-09-16T01:39:14.483390+0000","last_unstale":"2024-09-16T01:39:15.496417+0000","last_undegraded":"2024-09-16T01:39:15.496417+0000","last_fullsized":"2024-09-16T01:39:15.496417+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:34:56.781236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.499580+0000","last_change":"2024-09-16T01:39:15.497713+0000","last_active":"2024-09-16T01:39:15.499580+0000","last_peered":"2024-09-16T01:39:15.499580+0000","last_clean":"2024-09-16T01:39:15.499580+0000","last_became_active":"2024-09-16T01:39:15.497228+0000","last_became_peered":"2024-09-16T01:39:15.497228+0000","last_unstale":"2024-09-16T01:39:15.499580+0000","last_undegraded":"2024-09-16T01:39:15.499580+0000","last_fullsized":"2024-09-16T01:39:15.499580+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:39:00.982561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"71'2","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.501304+0000","last_change":"2024-09-16T01:39:14.480878+0000","last_active":"2024-09-16T01:39:15.501304+0000","last_peered":"2024-09-16T01:39:15.501304+0000","last_clean":"2024-09-16T01:39:15.501304+0000","last_became_active":"2024-09-16T01:39:14.480593+0000","last_became_peered":"2024-09-16T01:39:14.480593+0000","last_unstale":"2024-09-16T01:39:15.501304+0000","last_undegraded":"2024-09-16T01:39:15.501304+0000","last_fullsized":"2024-09-16T01:39:15.501304+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:31:50.296058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.501533+0000","last_change":"2024-09-16T01:39:15.498590+0000","last_active":"2024-09-16T01:39:15.501533+0000","last_peered":"2024-09-16T01:39:15.501533+0000","last_clean":"2024-09-16T01:39:15.501533+0000","last_became_active":"2024-09-16T01:39:15.498333+0000","last_became_peered":"2024-09-16T01:39:15.498333+0000","last_unstale":"2024-09-16T01:39:15.501533+0000","last_undegraded":"2024-09-16T01:39:15.501533+0000","last_fullsized":"2024-09-16T01:39:15.501533+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:10:21.183290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"71'2","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.499832+0000","last_change":"2024-09-16T01:39:14.482912+0000","last_active":"2024-09-16T01:39:15.499832+0000","last_peered":"2024-09-16T01:39:15.499832+0000","last_clean":"2024-09-16T01:39:15.499832+0000","last_became_active":"2024-09-16T01:39:14.481707+0000","last_became_peered":"2024-09-16T01:39:14.481707+0000","last_unstale":"2024-09-16T01:39:15.499832+0000","last_undegraded":"2024-09-16T01:39:15.499832+0000","last_fullsized":"2024-09-16T01:39:15.499832+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:51:00.179012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T01:39:16.487292+0000","last_change":"2024-09-16T01:39:15.478421+0000","last_active":"2024-09-16T01:39:16.487292+0000","last_peered":"2024-09-16T01:39:16.487292+0000","last_clean":"2024-09-16T01:39:16.487292+0000","last_became_active":"2024-09-16T01:39:15.478145+0000","last_became_peered":"2024-09-16T01:39:15.478145+0000","last_unstale":"2024-09-16T01:39:16.487292+0000","last_undegraded":"2024-09-16T01:39:16.487292+0000","last_fullsized":"2024-09-16T01:39:16.487292+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:44.054894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"71'1","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.497551+0000","last_change":"2024-09-16T01:39:14.479292+0000","last_active":"2024-09-16T01:39:15.497551+0000","last_peered":"2024-09-16T01:39:15.497551+0000","last_clean":"2024-09-16T01:39:15.497551+0000","last_became_active":"2024-09-16T01:39:14.479040+0000","last_became_peered":"2024-09-16T01:39:14.479040+0000","last_unstale":"2024-09-16T01:39:15.497551+0000","last_undegraded":"2024-09-16T01:39:15.497551+0000","last_fullsized":"2024-09-16T01:39:15.497551+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:25:37.315336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.490909+0000","last_change":"2024-09-16T01:39:15.489157+0000","last_active":"2024-09-16T01:39:15.490909+0000","last_peered":"2024-09-16T01:39:15.490909+0000","last_clean":"2024-09-16T01:39:15.490909+0000","last_became_active":"2024-09-16T01:39:15.488852+0000","last_became_peered":"2024-09-16T01:39:15.488852+0000","last_unstale":"2024-09-16T01:39:15.490909+0000","last_undegraded":"2024-09-16T01:39:15.490909+0000","last_fullsized":"2024-09-16T01:39:15.490909+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:27:29.143421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"71'1","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.498123+0000","last_change":"2024-09-16T01:39:14.480930+0000","last_active":"2024-09-16T01:39:15.498123+0000","last_peered":"2024-09-16T01:39:15.498123+0000","last_clean":"2024-09-16T01:39:15.498123+0000","last_became_active":"2024-09-16T01:39:14.480669+0000","last_became_peered":"2024-09-16T01:39:14.480669+0000","last_unstale":"2024-09-16T01:39:15.498123+0000","last_undegraded":"2024-09-16T01:39:15.498123+0000","last_fullsized":"2024-09-16T01:39:15.498123+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:53:36.791253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.493347+0000","last_change":"2024-09-16T01:39:15.480132+0000","last_active":"2024-09-16T01:39:15.493347+0000","last_peered":"2024-09-16T01:39:15.493347+0000","last_clean":"2024-09-16T01:39:15.493347+0000","last_became_active":"2024-09-16T01:39:15.479878+0000","last_became_peered":"2024-09-16T01:39:15.479878+0000","last_unstale":"2024-09-16T01:39:15.493347+0000","last_undegraded":"2024-09-16T01:39:15.493347+0000","last_fullsized":"2024-09-16T01:39:15.493347+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:07:43.338533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.470305+0000","last_change":"2024-09-16T01:39:14.481778+0000","last_active":"2024-09-16T01:39:15.470305+0000","last_peered":"2024-09-16T01:39:15.470305+0000","last_clean":"2024-09-16T01:39:15.470305+0000","last_became_active":"2024-09-16T01:39:14.481515+0000","last_became_peered":"2024-09-16T01:39:14.481515+0000","last_unstale":"2024-09-16T01:39:15.470305+0000","last_undegraded":"2024-09-16T01:39:15.470305+0000","last_fullsized":"2024-09-16T01:39:15.470305+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:54:42.509627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.496113+0000","last_change":"2024-09-16T01:39:15.489802+0000","last_active":"2024-09-16T01:39:15.496113+0000","last_peered":"2024-09-16T01:39:15.496113+0000","last_clean":"2024-09-16T01:39:15.496113+0000","last_became_active":"2024-09-16T01:39:15.489496+0000","last_became_peered":"2024-09-16T01:39:15.489496+0000","last_unstale":"2024-09-16T01:39:15.496113+0000","last_undegraded":"2024-09-16T01:39:15.496113+0000","last_fullsized":"2024-09-16T01:39:15.496113+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:29:47.956916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.473324+0000","last_change":"2024-09-16T01:39:14.477667+0000","last_active":"2024-09-16T01:39:15.473324+0000","last_peered":"2024-09-16T01:39:15.473324+0000","last_clean":"2024-09-16T01:39:15.473324+0000","last_became_active":"2024-09-16T01:39:14.477358+0000","last_became_peered":"2024-09-16T01:39:14.477358+0000","last_unstale":"2024-09-16T01:39:15.473324+0000","last_undegraded":"2024-09-16T01:39:15.473324+0000","last_fullsized":"2024-09-16T01:39:15.473324+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:33.745508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T01:39:16.487387+0000","last_change":"2024-09-16T01:39:15.490784+0000","last_active":"2024-09-16T01:39:16.487387+0000","last_peered":"2024-09-16T01:39:16.487387+0000","last_clean":"2024-09-16T01:39:16.487387+0000","last_became_active":"2024-09-16T01:39:15.490517+0000","last_became_peered":"2024-09-16T01:39:15.490517+0000","last_unstale":"2024-09-16T01:39:16.487387+0000","last_undegraded":"2024-09-16T01:39:16.487387+0000","last_fullsized":"2024-09-16T01:39:16.487387+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:34:32.699174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.477730+0000","last_change":"2024-09-16T01:39:14.479610+0000","last_active":"2024-09-16T01:39:15.477730+0000","last_peered":"2024-09-16T01:39:15.477730+0000","last_clean":"2024-09-16T01:39:15.477730+0000","last_became_active":"2024-09-16T01:39:14.479316+0000","last_became_peered":"2024-09-16T01:39:14.479316+0000","last_unstale":"2024-09-16T01:39:15.477730+0000","last_undegraded":"2024-09-16T01:39:15.477730+0000","last_fullsized":"2024-09-16T01:39:15.477730+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:42.008457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.501710+0000","last_change":"2024-09-16T01:39:15.497698+0000","last_active":"2024-09-16T01:39:15.501710+0000","last_peered":"2024-09-16T01:39:15.501710+0000","last_clean":"2024-09-16T01:39:15.501710+0000","last_became_active":"2024-09-16T01:39:15.496309+0000","last_became_peered":"2024-09-16T01:39:15.496309+0000","last_unstale":"2024-09-16T01:39:15.501710+0000","last_undegraded":"2024-09-16T01:39:15.501710+0000","last_fullsized":"2024-09-16T01:39:15.501710+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:11:02.011376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.478944+0000","last_change":"2024-09-16T01:39:14.483878+0000","last_active":"2024-09-16T01:39:15.478944+0000","last_peered":"2024-09-16T01:39:15.478944+0000","last_clean":"2024-09-16T01:39:15.478944+0000","last_became_active":"2024-09-16T01:39:14.483573+0000","last_became_peered":"2024-09-16T01:39:14.483573+0000","last_unstale":"2024-09-16T01:39:15.478944+0000","last_undegraded":"2024-09-16T01:39:15.478944+0000","last_fullsized":"2024-09-16T01:39:15.478944+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:33:43.742264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.499324+0000","last_change":"2024-09-16T01:39:15.497490+0000","last_active":"2024-09-16T01:39:15.499324+0000","last_peered":"2024-09-16T01:39:15.499324+0000","last_clean":"2024-09-16T01:39:15.499324+0000","last_became_active":"2024-09-16T01:39:15.492295+0000","last_became_peered":"2024-09-16T01:39:15.492295+0000","last_unstale":"2024-09-16T01:39:15.499324+0000","last_undegraded":"2024-09-16T01:39:15.499324+0000","last_fullsized":"2024-09-16T01:39:15.499324+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:54:37.811306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"71'3","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.501031+0000","last_change":"2024-09-16T01:39:14.477914+0000","last_active":"2024-09-16T01:39:15.501031+0000","last_peered":"2024-09-16T01:39:15.501031+0000","last_clean":"2024-09-16T01:39:15.501031+0000","last_became_active":"2024-09-16T01:39:14.477537+0000","last_became_peered":"2024-09-16T01:39:14.477537+0000","last_unstale":"2024-09-16T01:39:15.501031+0000","last_undegraded":"2024-09-16T01:39:15.501031+0000","last_fullsized":"2024-09-16T01:39:15.501031+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:39:02.681571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.493236+0000","last_change":"2024-09-16T01:39:15.477919+0000","last_active":"2024-09-16T01:39:15.493236+0000","last_peered":"2024-09-16T01:39:15.493236+0000","last_clean":"2024-09-16T01:39:15.493236+0000","last_became_active":"2024-09-16T01:39:15.477436+0000","last_became_peered":"2024-09-16T01:39:15.477436+0000","last_unstale":"2024-09-16T01:39:15.493236+0000","last_undegraded":"2024-09-16T01:39:15.493236+0000","last_fullsized":"2024-09-16T01:39:15.493236+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:42:28.285765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"71'6","reported_seq":18,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.500749+0000","last_change":"2024-09-16T01:39:14.480691+0000","last_active":"2024-09-16T01:39:15.500749+0000","last_peered":"2024-09-16T01:39:15.500749+0000","last_clean":"2024-09-16T01:39:15.500749+0000","last_became_active":"2024-09-16T01:39:14.480411+0000","last_became_peered":"2024-09-16T01:39:14.480411+0000","last_unstale":"2024-09-16T01:39:15.500749+0000","last_undegraded":"2024-09-16T01:39:15.500749+0000","last_fullsized":"2024-09-16T01:39:15.500749+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:54:15.531756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.500076+0000","last_change":"2024-09-16T01:39:15.496405+0000","last_active":"2024-09-16T01:39:15.500076+0000","last_peered":"2024-09-16T01:39:15.500076+0000","last_clean":"2024-09-16T01:39:15.500076+0000","last_became_active":"2024-09-16T01:39:15.496106+0000","last_became_peered":"2024-09-16T01:39:15.496106+0000","last_unstale":"2024-09-16T01:39:15.500076+0000","last_undegraded":"2024-09-16T01:39:15.500076+0000","last_fullsized":"2024-09-16T01:39:15.500076+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:12.294726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"71'2","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.495915+0000","last_change":"2024-09-16T01:39:14.477452+0000","last_active":"2024-09-16T01:39:15.495915+0000","last_peered":"2024-09-16T01:39:15.495915+0000","last_clean":"2024-09-16T01:39:15.495915+0000","last_became_active":"2024-09-16T01:39:14.477178+0000","last_became_peered":"2024-09-16T01:39:14.477178+0000","last_unstale":"2024-09-16T01:39:15.495915+0000","last_undegraded":"2024-09-16T01:39:15.495915+0000","last_fullsized":"2024-09-16T01:39:15.495915+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:58:08.056124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T01:39:16.487195+0000","last_change":"2024-09-16T01:39:15.489991+0000","last_active":"2024-09-16T01:39:16.487195+0000","last_peered":"2024-09-16T01:39:16.487195+0000","last_clean":"2024-09-16T01:39:16.487195+0000","last_became_active":"2024-09-16T01:39:15.489713+0000","last_became_peered":"2024-09-16T01:39:15.489713+0000","last_unstale":"2024-09-16T01:39:16.487195+0000","last_undegraded":"2024-09-16T01:39:16.487195+0000","last_fullsized":"2024-09-16T01:39:16.487195+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:10:21.183290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.477291+0000","last_change":"2024-09-16T01:39:14.473788+0000","last_active":"2024-09-16T01:39:15.477291+0000","last_peered":"2024-09-16T01:39:15.477291+0000","last_clean":"2024-09-16T01:39:15.477291+0000","last_became_active":"2024-09-16T01:39:14.473538+0000","last_became_peered":"2024-09-16T01:39:14.473538+0000","last_unstale":"2024-09-16T01:39:15.477291+0000","last_undegraded":"2024-09-16T01:39:15.477291+0000","last_fullsized":"2024-09-16T01:39:15.477291+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:26:57.297516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.488191+0000","last_change":"2024-09-16T01:39:15.485010+0000","last_active":"2024-09-16T01:39:15.488191+0000","last_peered":"2024-09-16T01:39:15.488191+0000","last_clean":"2024-09-16T01:39:15.488191+0000","last_became_active":"2024-09-16T01:39:15.484664+0000","last_became_peered":"2024-09-16T01:39:15.484664+0000","last_unstale":"2024-09-16T01:39:15.488191+0000","last_undegraded":"2024-09-16T01:39:15.488191+0000","last_fullsized":"2024-09-16T01:39:15.488191+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:16.034246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T01:39:16.487130+0000","last_change":"2024-09-16T01:39:14.485795+0000","last_active":"2024-09-16T01:39:16.487130+0000","last_peered":"2024-09-16T01:39:16.487130+0000","last_clean":"2024-09-16T01:39:16.487130+0000","last_became_active":"2024-09-16T01:39:14.485549+0000","last_became_peered":"2024-09-16T01:39:14.485549+0000","last_unstale":"2024-09-16T01:39:16.487130+0000","last_undegraded":"2024-09-16T01:39:16.487130+0000","last_fullsized":"2024-09-16T01:39:16.487130+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:43:33.350600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.485794+0000","last_change":"2024-09-16T01:39:15.476934+0000","last_active":"2024-09-16T01:39:15.485794+0000","last_peered":"2024-09-16T01:39:15.485794+0000","last_clean":"2024-09-16T01:39:15.485794+0000","last_became_active":"2024-09-16T01:39:15.476683+0000","last_became_peered":"2024-09-16T01:39:15.476683+0000","last_unstale":"2024-09-16T01:39:15.485794+0000","last_undegraded":"2024-09-16T01:39:15.485794+0000","last_fullsized":"2024-09-16T01:39:15.485794+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:58:02.701821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"71'8","reported_seq":20,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.502530+0000","last_change":"2024-09-16T01:39:14.475088+0000","last_active":"2024-09-16T01:39:15.502530+0000","last_peered":"2024-09-16T01:39:15.502530+0000","last_clean":"2024-09-16T01:39:15.502530+0000","last_became_active":"2024-09-16T01:39:14.474752+0000","last_became_peered":"2024-09-16T01:39:14.474752+0000","last_unstale":"2024-09-16T01:39:15.502530+0000","last_undegraded":"2024-09-16T01:39:15.502530+0000","last_fullsized":"2024-09-16T01:39:15.502530+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:13.036717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.501415+0000","last_change":"2024-09-16T01:39:15.499795+0000","last_active":"2024-09-16T01:39:15.501415+0000","last_peered":"2024-09-16T01:39:15.501415+0000","last_clean":"2024-09-16T01:39:15.501415+0000","last_became_active":"2024-09-16T01:39:15.499516+0000","last_became_peered":"2024-09-16T01:39:15.499516+0000","last_unstale":"2024-09-16T01:39:15.501415+0000","last_undegraded":"2024-09-16T01:39:15.501415+0000","last_fullsized":"2024-09-16T01:39:15.501415+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:26:42.021902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"71'1","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.498430+0000","last_change":"2024-09-16T01:39:14.478767+0000","last_active":"2024-09-16T01:39:15.498430+0000","last_peered":"2024-09-16T01:39:15.498430+0000","last_clean":"2024-09-16T01:39:15.498430+0000","last_became_active":"2024-09-16T01:39:14.478445+0000","last_became_peered":"2024-09-16T01:39:14.478445+0000","last_unstale":"2024-09-16T01:39:15.498430+0000","last_undegraded":"2024-09-16T01:39:15.498430+0000","last_fullsized":"2024-09-16T01:39:15.498430+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:02:37.714944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.501738+0000","last_change":"2024-09-16T01:39:15.498536+0000","last_active":"2024-09-16T01:39:15.501738+0000","last_peered":"2024-09-16T01:39:15.501738+0000","last_clean":"2024-09-16T01:39:15.501738+0000","last_became_active":"2024-09-16T01:39:15.497432+0000","last_became_peered":"2024-09-16T01:39:15.497432+0000","last_unstale":"2024-09-16T01:39:15.501738+0000","last_undegraded":"2024-09-16T01:39:15.501738+0000","last_fullsized":"2024-09-16T01:39:15.501738+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:58:48.937237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"71'2","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.501354+0000","last_change":"2024-09-16T01:39:14.479568+0000","last_active":"2024-09-16T01:39:15.501354+0000","last_peered":"2024-09-16T01:39:15.501354+0000","last_clean":"2024-09-16T01:39:15.501354+0000","last_became_active":"2024-09-16T01:39:14.479287+0000","last_became_peered":"2024-09-16T01:39:14.479287+0000","last_unstale":"2024-09-16T01:39:15.501354+0000","last_undegraded":"2024-09-16T01:39:15.501354+0000","last_fullsized":"2024-09-16T01:39:15.501354+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:15:39.947322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.493128+0000","last_change":"2024-09-16T01:39:15.486333+0000","last_active":"2024-09-16T01:39:15.493128+0000","last_peered":"2024-09-16T01:39:15.493128+0000","last_clean":"2024-09-16T01:39:15.493128+0000","last_became_active":"2024-09-16T01:39:15.486086+0000","last_became_peered":"2024-09-16T01:39:15.486086+0000","last_unstale":"2024-09-16T01:39:15.493128+0000","last_undegraded":"2024-09-16T01:39:15.493128+0000","last_fullsized":"2024-09-16T01:39:15.493128+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:01:56.501512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"71'1","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.498831+0000","last_change":"2024-09-16T01:39:14.482959+0000","last_active":"2024-09-16T01:39:15.498831+0000","last_peered":"2024-09-16T01:39:15.498831+0000","last_clean":"2024-09-16T01:39:15.498831+0000","last_became_active":"2024-09-16T01:39:14.482511+0000","last_became_peered":"2024-09-16T01:39:14.482511+0000","last_unstale":"2024-09-16T01:39:15.498831+0000","last_undegraded":"2024-09-16T01:39:15.498831+0000","last_fullsized":"2024-09-16T01:39:15.498831+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:58:30.468300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.496120+0000","last_change":"2024-09-16T01:39:15.491483+0000","last_active":"2024-09-16T01:39:15.496120+0000","last_peered":"2024-09-16T01:39:15.496120+0000","last_clean":"2024-09-16T01:39:15.496120+0000","last_became_active":"2024-09-16T01:39:15.491256+0000","last_became_peered":"2024-09-16T01:39:15.491256+0000","last_unstale":"2024-09-16T01:39:15.496120+0000","last_undegraded":"2024-09-16T01:39:15.496120+0000","last_fullsized":"2024-09-16T01:39:15.496120+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:42:14.774404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.480777+0000","last_change":"2024-09-16T01:39:14.482639+0000","last_active":"2024-09-16T01:39:15.480777+0000","last_peered":"2024-09-16T01:39:15.480777+0000","last_clean":"2024-09-16T01:39:15.480777+0000","last_became_active":"2024-09-16T01:39:14.482388+0000","last_became_peered":"2024-09-16T01:39:14.482388+0000","last_unstale":"2024-09-16T01:39:15.480777+0000","last_undegraded":"2024-09-16T01:39:15.480777+0000","last_fullsized":"2024-09-16T01:39:15.480777+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:39:40.424008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.480649+0000","last_change":"2024-09-16T01:39:15.478509+0000","last_active":"2024-09-16T01:39:15.480649+0000","last_peered":"2024-09-16T01:39:15.480649+0000","last_clean":"2024-09-16T01:39:15.480649+0000","last_became_active":"2024-09-16T01:39:15.478016+0000","last_became_peered":"2024-09-16T01:39:15.478016+0000","last_unstale":"2024-09-16T01:39:15.480649+0000","last_undegraded":"2024-09-16T01:39:15.480649+0000","last_fullsized":"2024-09-16T01:39:15.480649+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:46:23.043172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.477522+0000","last_change":"2024-09-16T01:39:14.480323+0000","last_active":"2024-09-16T01:39:15.477522+0000","last_peered":"2024-09-16T01:39:15.477522+0000","last_clean":"2024-09-16T01:39:15.477522+0000","last_became_active":"2024-09-16T01:39:14.480070+0000","last_became_peered":"2024-09-16T01:39:14.480070+0000","last_unstale":"2024-09-16T01:39:15.477522+0000","last_undegraded":"2024-09-16T01:39:15.477522+0000","last_fullsized":"2024-09-16T01:39:15.477522+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:13:58.290073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.483240+0000","last_change":"2024-09-16T01:39:15.482211+0000","last_active":"2024-09-16T01:39:15.483240+0000","last_peered":"2024-09-16T01:39:15.483240+0000","last_clean":"2024-09-16T01:39:15.483240+0000","last_became_active":"2024-09-16T01:39:15.481895+0000","last_became_peered":"2024-09-16T01:39:15.481895+0000","last_unstale":"2024-09-16T01:39:15.483240+0000","last_undegraded":"2024-09-16T01:39:15.483240+0000","last_fullsized":"2024-09-16T01:39:15.483240+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:19:55.217976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T01:39:16.487280+0000","last_change":"2024-09-16T01:39:14.485506+0000","last_active":"2024-09-16T01:39:16.487280+0000","last_peered":"2024-09-16T01:39:16.487280+0000","last_clean":"2024-09-16T01:39:16.487280+0000","last_became_active":"2024-09-16T01:39:14.485324+0000","last_became_peered":"2024-09-16T01:39:14.485324+0000","last_unstale":"2024-09-16T01:39:16.487280+0000","last_undegraded":"2024-09-16T01:39:16.487280+0000","last_fullsized":"2024-09-16T01:39:16.487280+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:08.626483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.483269+0000","last_change":"2024-09-16T01:39:15.482198+0000","last_active":"2024-09-16T01:39:15.483269+0000","last_peered":"2024-09-16T01:39:15.483269+0000","last_clean":"2024-09-16T01:39:15.483269+0000","last_became_active":"2024-09-16T01:39:15.481950+0000","last_became_peered":"2024-09-16T01:39:15.481950+0000","last_unstale":"2024-09-16T01:39:15.483269+0000","last_undegraded":"2024-09-16T01:39:15.483269+0000","last_fullsized":"2024-09-16T01:39:15.483269+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:06:04.402749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.471094+0000","last_change":"2024-09-16T01:39:14.480725+0000","last_active":"2024-09-16T01:39:15.471094+0000","last_peered":"2024-09-16T01:39:15.471094+0000","last_clean":"2024-09-16T01:39:15.471094+0000","last_became_active":"2024-09-16T01:39:14.480462+0000","last_became_peered":"2024-09-16T01:39:14.480462+0000","last_unstale":"2024-09-16T01:39:15.471094+0000","last_undegraded":"2024-09-16T01:39:15.471094+0000","last_fullsized":"2024-09-16T01:39:15.471094+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:33:40.794349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.493943+0000","last_change":"2024-09-16T01:39:15.486880+0000","last_active":"2024-09-16T01:39:15.493943+0000","last_peered":"2024-09-16T01:39:15.493943+0000","last_clean":"2024-09-16T01:39:15.493943+0000","last_became_active":"2024-09-16T01:39:15.486551+0000","last_became_peered":"2024-09-16T01:39:15.486551+0000","last_unstale":"2024-09-16T01:39:15.493943+0000","last_undegraded":"2024-09-16T01:39:15.493943+0000","last_fullsized":"2024-09-16T01:39:15.493943+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:56.770000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"71'2","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.498756+0000","last_change":"2024-09-16T01:39:14.481961+0000","last_active":"2024-09-16T01:39:15.498756+0000","last_peered":"2024-09-16T01:39:15.498756+0000","last_clean":"2024-09-16T01:39:15.498756+0000","last_became_active":"2024-09-16T01:39:14.481683+0000","last_became_peered":"2024-09-16T01:39:14.481683+0000","last_unstale":"2024-09-16T01:39:15.498756+0000","last_undegraded":"2024-09-16T01:39:15.498756+0000","last_fullsized":"2024-09-16T01:39:15.498756+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:44:33.074920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T01:39:16.487443+0000","last_change":"2024-09-16T01:39:15.478929+0000","last_active":"2024-09-16T01:39:16.487443+0000","last_peered":"2024-09-16T01:39:16.487443+0000","last_clean":"2024-09-16T01:39:16.487443+0000","last_became_active":"2024-09-16T01:39:15.478658+0000","last_became_peered":"2024-09-16T01:39:15.478658+0000","last_unstale":"2024-09-16T01:39:16.487443+0000","last_undegraded":"2024-09-16T01:39:16.487443+0000","last_fullsized":"2024-09-16T01:39:16.487443+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:32:50.936209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"72'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T01:39:16.488840+0000","last_change":"2024-09-16T01:39:14.480924+0000","last_active":"2024-09-16T01:39:16.488840+0000","last_peered":"2024-09-16T01:39:16.488840+0000","last_clean":"2024-09-16T01:39:16.488840+0000","last_became_active":"2024-09-16T01:39:14.480660+0000","last_became_peered":"2024-09-16T01:39:14.480660+0000","last_unstale":"2024-09-16T01:39:16.488840+0000","last_undegraded":"2024-09-16T01:39:16.488840+0000","last_fullsized":"2024-09-16T01:39:16.488840+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:45:28.982396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.500130+0000","last_change":"2024-09-16T01:39:15.496128+0000","last_active":"2024-09-16T01:39:15.500130+0000","last_peered":"2024-09-16T01:39:15.500130+0000","last_clean":"2024-09-16T01:39:15.500130+0000","last_became_active":"2024-09-16T01:39:15.495841+0000","last_became_peered":"2024-09-16T01:39:15.495841+0000","last_unstale":"2024-09-16T01:39:15.500130+0000","last_undegraded":"2024-09-16T01:39:15.500130+0000","last_fullsized":"2024-09-16T01:39:15.500130+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:40.261799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.473872+0000","last_change":"2024-09-16T01:39:14.480571+0000","last_active":"2024-09-16T01:39:15.473872+0000","last_peered":"2024-09-16T01:39:15.473872+0000","last_clean":"2024-09-16T01:39:15.473872+0000","last_became_active":"2024-09-16T01:39:14.480331+0000","last_became_peered":"2024-09-16T01:39:14.480331+0000","last_unstale":"2024-09-16T01:39:15.473872+0000","last_undegraded":"2024-09-16T01:39:15.473872+0000","last_fullsized":"2024-09-16T01:39:15.473872+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:44:56.435514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.496889+0000","last_change":"2024-09-16T01:39:15.494048+0000","last_active":"2024-09-16T01:39:15.496889+0000","last_peered":"2024-09-16T01:39:15.496889+0000","last_clean":"2024-09-16T01:39:15.496889+0000","last_became_active":"2024-09-16T01:39:15.493774+0000","last_became_peered":"2024-09-16T01:39:15.493774+0000","last_unstale":"2024-09-16T01:39:15.496889+0000","last_undegraded":"2024-09-16T01:39:15.496889+0000","last_fullsized":"2024-09-16T01:39:15.496889+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:55:10.488556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.471584+0000","last_change":"2024-09-16T01:39:14.482307+0000","last_active":"2024-09-16T01:39:15.471584+0000","last_peered":"2024-09-16T01:39:15.471584+0000","last_clean":"2024-09-16T01:39:15.471584+0000","last_became_active":"2024-09-16T01:39:14.481787+0000","last_became_peered":"2024-09-16T01:39:14.481787+0000","last_unstale":"2024-09-16T01:39:15.471584+0000","last_undegraded":"2024-09-16T01:39:15.471584+0000","last_fullsized":"2024-09-16T01:39:15.471584+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:00:18.017341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.493287+0000","last_change":"2024-09-16T01:39:15.477617+0000","last_active":"2024-09-16T01:39:15.493287+0000","last_peered":"2024-09-16T01:39:15.493287+0000","last_clean":"2024-09-16T01:39:15.493287+0000","last_became_active":"2024-09-16T01:39:15.477332+0000","last_became_peered":"2024-09-16T01:39:15.477332+0000","last_unstale":"2024-09-16T01:39:15.493287+0000","last_undegraded":"2024-09-16T01:39:15.493287+0000","last_fullsized":"2024-09-16T01:39:15.493287+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:28:25.405417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.473593+0000","last_change":"2024-09-16T01:39:14.480491+0000","last_active":"2024-09-16T01:39:15.473593+0000","last_peered":"2024-09-16T01:39:15.473593+0000","last_clean":"2024-09-16T01:39:15.473593+0000","last_became_active":"2024-09-16T01:39:14.480222+0000","last_became_peered":"2024-09-16T01:39:14.480222+0000","last_unstale":"2024-09-16T01:39:15.473593+0000","last_undegraded":"2024-09-16T01:39:15.473593+0000","last_fullsized":"2024-09-16T01:39:15.473593+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:02.334769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.499352+0000","last_change":"2024-09-16T01:39:15.497331+0000","last_active":"2024-09-16T01:39:15.499352+0000","last_peered":"2024-09-16T01:39:15.499352+0000","last_clean":"2024-09-16T01:39:15.499352+0000","last_became_active":"2024-09-16T01:39:15.497045+0000","last_became_peered":"2024-09-16T01:39:15.497045+0000","last_unstale":"2024-09-16T01:39:15.499352+0000","last_undegraded":"2024-09-16T01:39:15.499352+0000","last_fullsized":"2024-09-16T01:39:15.499352+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:27:29.143421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.470692+0000","last_change":"2024-09-16T01:39:14.478513+0000","last_active":"2024-09-16T01:39:15.470692+0000","last_peered":"2024-09-16T01:39:15.470692+0000","last_clean":"2024-09-16T01:39:15.470692+0000","last_became_active":"2024-09-16T01:39:14.478246+0000","last_became_peered":"2024-09-16T01:39:14.478246+0000","last_unstale":"2024-09-16T01:39:15.470692+0000","last_undegraded":"2024-09-16T01:39:15.470692+0000","last_fullsized":"2024-09-16T01:39:15.470692+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:53:12.947025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.480026+0000","last_change":"2024-09-16T01:39:15.476544+0000","last_active":"2024-09-16T01:39:15.480026+0000","last_peered":"2024-09-16T01:39:15.480026+0000","last_clean":"2024-09-16T01:39:15.480026+0000","last_became_active":"2024-09-16T01:39:15.476198+0000","last_became_peered":"2024-09-16T01:39:15.476198+0000","last_unstale":"2024-09-16T01:39:15.480026+0000","last_undegraded":"2024-09-16T01:39:15.480026+0000","last_fullsized":"2024-09-16T01:39:15.480026+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:35:56.620792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.476800+0000","last_change":"2024-09-16T01:39:14.478534+0000","last_active":"2024-09-16T01:39:15.476800+0000","last_peered":"2024-09-16T01:39:15.476800+0000","last_clean":"2024-09-16T01:39:15.476800+0000","last_became_active":"2024-09-16T01:39:14.478266+0000","last_became_peered":"2024-09-16T01:39:14.478266+0000","last_unstale":"2024-09-16T01:39:15.476800+0000","last_undegraded":"2024-09-16T01:39:15.476800+0000","last_fullsized":"2024-09-16T01:39:15.476800+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:03:12.581343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.485609+0000","last_change":"2024-09-16T01:39:15.480158+0000","last_active":"2024-09-16T01:39:15.485609+0000","last_peered":"2024-09-16T01:39:15.485609+0000","last_clean":"2024-09-16T01:39:15.485609+0000","last_became_active":"2024-09-16T01:39:15.479774+0000","last_became_peered":"2024-09-16T01:39:15.479774+0000","last_unstale":"2024-09-16T01:39:15.485609+0000","last_undegraded":"2024-09-16T01:39:15.485609+0000","last_fullsized":"2024-09-16T01:39:15.485609+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:14.043165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.477016+0000","last_change":"2024-09-16T01:39:14.478551+0000","last_active":"2024-09-16T01:39:15.477016+0000","last_peered":"2024-09-16T01:39:15.477016+0000","last_clean":"2024-09-16T01:39:15.477016+0000","last_became_active":"2024-09-16T01:39:14.478222+0000","last_became_peered":"2024-09-16T01:39:14.478222+0000","last_unstale":"2024-09-16T01:39:15.477016+0000","last_undegraded":"2024-09-16T01:39:15.477016+0000","last_fullsized":"2024-09-16T01:39:15.477016+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:55:06.749555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.482481+0000","last_change":"2024-09-16T01:39:15.480517+0000","last_active":"2024-09-16T01:39:15.482481+0000","last_peered":"2024-09-16T01:39:15.482481+0000","last_clean":"2024-09-16T01:39:15.482481+0000","last_became_active":"2024-09-16T01:39:15.480197+0000","last_became_peered":"2024-09-16T01:39:15.480197+0000","last_unstale":"2024-09-16T01:39:15.482481+0000","last_undegraded":"2024-09-16T01:39:15.482481+0000","last_fullsized":"2024-09-16T01:39:15.482481+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:40.532529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"71'2","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.500281+0000","last_change":"2024-09-16T01:39:14.481270+0000","last_active":"2024-09-16T01:39:15.500281+0000","last_peered":"2024-09-16T01:39:15.500281+0000","last_clean":"2024-09-16T01:39:15.500281+0000","last_became_active":"2024-09-16T01:39:14.480982+0000","last_became_peered":"2024-09-16T01:39:14.480982+0000","last_unstale":"2024-09-16T01:39:15.500281+0000","last_undegraded":"2024-09-16T01:39:15.500281+0000","last_fullsized":"2024-09-16T01:39:15.500281+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:32:26.475867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":261,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.482902+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:39:15.482902+0000","last_peered":"2024-09-16T01:39:15.482902+0000","last_clean":"2024-09-16T01:39:15.482902+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:39:15.482902+0000","last_undegraded":"2024-09-16T01:39:15.482902+0000","last_fullsized":"2024-09-16T01:39:15.482902+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509506,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32036,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339804,"statfs":{"total":96636764160,"available":96603959296,"internally_reserved":0,"allocated":864256,"data_stored":363344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542209,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32604,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1441792,"data_stored":946847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":847872,"data_stored":356479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":864256,"data_stored":363324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542208,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":847872,"data_stored":356479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542209,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":847872,"data_stored":356479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542208,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":847872,"data_stored":356479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542208,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28436,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343404,"statfs":{"total":96636764160,"available":96607645696,"internally_reserved":0,"allocated":1441792,"data_stored":946847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:39:18.409 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:39:18.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.542+0000 7f51bb378640 1 Processor -- start 2024-09-16T01:39:18.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.542+0000 7f51bb378640 1 -- start start 2024-09-16T01:39:18.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.543+0000 7f51bb378640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f51b40715e0 0x7f51b40b6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:18.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.543+0000 7f51bb378640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f51b4073160 0x7f51b4073590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:18.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.543+0000 7f51bb378640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51b40d0270 con 0x7f51b415c550 2024-09-16T01:39:18.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.543+0000 7f51bb378640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f51b40b7350 con 0x7f51b40715e0 2024-09-16T01:39:18.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.543+0000 7f51bb378640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f51b415c930 con 0x7f51b4073160 2024-09-16T01:39:18.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.543+0000 7f51b8913640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f51b40715e0 0x7f51b40b6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:18.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.543+0000 7f51b9114640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f51b415c550 0x7f51b40b5540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42076/0 (socket says 172.21.15.88:42076) 2024-09-16T01:39:18.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.543+0000 7f51b9114640 1 -- 172.21.15.88:0/1597697268 learned_addr learned my addr 172.21.15.88:0/1597697268 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:18.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.543+0000 7f51b9915640 1 --2- 172.21.15.88:0/1597697268 >> v2:172.21.15.154:3300/0 conn(0x7f51b4073160 0x7f51b4073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:18.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.543+0000 7f51b8913640 1 -- 172.21.15.88:0/1597697268 >> v2:172.21.15.154:3300/0 conn(0x7f51b4073160 msgr2=0x7f51b4073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:18.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.543+0000 7f51b8913640 1 --2- 172.21.15.88:0/1597697268 >> v2:172.21.15.154:3300/0 conn(0x7f51b4073160 0x7f51b4073590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.543+0000 7f51b8913640 1 -- 172.21.15.88:0/1597697268 >> v1:172.21.15.88:6789/0 conn(0x7f51b415c550 legacy=0x7f51b40b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:18.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.543+0000 7f51abfff640 1 -- 172.21.15.88:0/1597697268 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2580115157 0 0) 0x7f51b0001110 con 0x7f51b415c550 2024-09-16T01:39:18.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.543+0000 7f51b8913640 1 -- 172.21.15.88:0/1597697268 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51b4071a50 con 0x7f51b40715e0 2024-09-16T01:39:18.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.543+0000 7f51b9915640 1 --2- 172.21.15.88:0/1597697268 >> v2:172.21.15.154:3300/0 conn(0x7f51b4073160 0x7f51b4073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:18.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.544+0000 7f51b8913640 1 --2- 172.21.15.88:0/1597697268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b40715e0 0x7f51b40b6e10 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f51b4072d90 tx=0x7f51a402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd91ea6997252259 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:18.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.544+0000 7f51abfff640 1 -- 172.21.15.88:0/1597697268 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a4037070 con 0x7f51b40715e0 2024-09-16T01:39:18.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.544+0000 7f51abfff640 1 -- 172.21.15.88:0/1597697268 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51a4032440 con 0x7f51b40715e0 2024-09-16T01:39:18.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.544+0000 7f51abfff640 1 -- 172.21.15.88:0/1597697268 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a4032810 con 0x7f51b40715e0 2024-09-16T01:39:18.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.545+0000 7f51bb378640 1 -- 172.21.15.88:0/1597697268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b40715e0 msgr2=0x7f51b40b6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:18.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.545+0000 7f51bb378640 1 --2- 172.21.15.88:0/1597697268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b40715e0 0x7f51b40b6e10 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f51b4072d90 tx=0x7f51a402a1a0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.545+0000 7f51bb378640 1 -- 172.21.15.88:0/1597697268 shutdown_connections 2024-09-16T01:39:18.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.545+0000 7f51bb378640 1 --2- 172.21.15.88:0/1597697268 >> v2:172.21.15.154:3300/0 conn(0x7f51b4073160 0x7f51b4073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.545+0000 7f51bb378640 1 --2- 172.21.15.88:0/1597697268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b40715e0 0x7f51b40b6e10 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.545+0000 7f51bb378640 1 -- 172.21.15.88:0/1597697268 >> 172.21.15.88:0/1597697268 conn(0x7f51b40b4240 msgr2=0x7f51b40b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:18.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.545+0000 7f51bb378640 1 -- 172.21.15.88:0/1597697268 shutdown_connections 2024-09-16T01:39:18.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.545+0000 7f51bb378640 1 -- 172.21.15.88:0/1597697268 wait complete. 2024-09-16T01:39:18.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.546+0000 7f51bb378640 1 Processor -- start 2024-09-16T01:39:18.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.546+0000 7f51bb378640 1 -- start start 2024-09-16T01:39:18.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.547+0000 7f51bb378640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51b40715e0 0x7f51b41e7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:18.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.547+0000 7f51bb378640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b4073160 0x7f51b415adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:18.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.547+0000 7f51bb378640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51b415c550 0x7f51b41f4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:18.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.547+0000 7f51bb378640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f51b40b7580 con 0x7f51b4073160 2024-09-16T01:39:18.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.547+0000 7f51bb378640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f51b4073ad0 con 0x7f51b415c550 2024-09-16T01:39:18.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.547+0000 7f51bb378640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f51b40b7860 con 0x7f51b40715e0 2024-09-16T01:39:18.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.547+0000 7f51b8913640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b4073160 0x7f51b415adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:18.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.547+0000 7f51b8913640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b4073160 0x7f51b415adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37756/0 (socket says 172.21.15.88:37756) 2024-09-16T01:39:18.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.547+0000 7f51b8913640 1 -- 172.21.15.88:0/2960928647 learned_addr learned my addr 172.21.15.88:0/2960928647 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:18.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.547+0000 7f51b8913640 1 -- 172.21.15.88:0/2960928647 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51b40715e0 msgr2=0x7f51b41e7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:18.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.547+0000 7f51b9915640 1 --2- 172.21.15.88:0/2960928647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51b415c550 0x7f51b41f4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:18.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.547+0000 7f51b8913640 1 --2- 172.21.15.88:0/2960928647 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51b40715e0 0x7f51b41e7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.547+0000 7f51b8913640 1 -- 172.21.15.88:0/2960928647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51b415c550 msgr2=0x7f51b41f4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:18.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.547+0000 7f51b8913640 1 --2- 172.21.15.88:0/2960928647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51b415c550 0x7f51b41f4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.547+0000 7f51b8913640 1 -- 172.21.15.88:0/2960928647 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51b41e4f50 con 0x7f51b4073160 2024-09-16T01:39:18.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.547+0000 7f51b8913640 1 --2- 172.21.15.88:0/2960928647 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b4073160 0x7f51b415adc0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f51a4000c00 tx=0x7f51a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:18.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.547+0000 7f51a9ffb640 1 -- 172.21.15.88:0/2960928647 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a4037050 con 0x7f51b4073160 2024-09-16T01:39:18.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.548+0000 7f51a9ffb640 1 -- 172.21.15.88:0/2960928647 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51a4038040 con 0x7f51b4073160 2024-09-16T01:39:18.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.548+0000 7f51a9ffb640 1 -- 172.21.15.88:0/2960928647 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a4004450 con 0x7f51b4073160 2024-09-16T01:39:18.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.548+0000 7f51bb378640 1 -- 172.21.15.88:0/2960928647 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51b41e5230 con 0x7f51b4073160 2024-09-16T01:39:18.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.548+0000 7f51bb378640 1 -- 172.21.15.88:0/2960928647 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51b41e57f0 con 0x7f51b4073160 2024-09-16T01:39:18.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.549+0000 7f51bb378640 1 -- 172.21.15.88:0/2960928647 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51b40d07b0 con 0x7f51b4073160 2024-09-16T01:39:18.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.549+0000 7f51a9ffb640 1 -- 172.21.15.88:0/2960928647 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f51a4030980 con 0x7f51b4073160 2024-09-16T01:39:18.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.550+0000 7f51a9ffb640 1 --2- 172.21.15.88:0/2960928647 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f519806c2b0 0x7f519806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:18.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.550+0000 7f51a9ffb640 1 -- 172.21.15.88:0/2960928647 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f51a40ac0d0 con 0x7f51b4073160 2024-09-16T01:39:18.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.550+0000 7f51b9114640 1 --2- 172.21.15.88:0/2960928647 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f519806c2b0 0x7f519806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:18.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.550+0000 7f51b9114640 1 --2- 172.21.15.88:0/2960928647 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f519806c2b0 0x7f519806e770 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f51b0001620 tx=0x7f51b0001310 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:18.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.554+0000 7f51a9ffb640 1 -- 172.21.15.88:0/2960928647 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f51a407a9f0 con 0x7f51b4073160 2024-09-16T01:39:18.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.720+0000 7f51bb378640 1 -- 172.21.15.88:0/2960928647 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f51b41e4d00 con 0x7f519806c2b0 2024-09-16T01:39:18.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.725+0000 7f51a9ffb640 1 -- 172.21.15.88:0/2960928647 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252006 (secure 0 0 0) 0x7f51b41e4d00 con 0x7f519806c2b0 2024-09-16T01:39:18.730 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:18.731 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:39:18.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.728+0000 7f51bb378640 1 -- 172.21.15.88:0/2960928647 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f519806c2b0 msgr2=0x7f519806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:18.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.728+0000 7f51bb378640 1 --2- 172.21.15.88:0/2960928647 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f519806c2b0 0x7f519806e770 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f51b0001620 tx=0x7f51b0001310 comp rx=0 tx=0).stop 2024-09-16T01:39:18.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.728+0000 7f51bb378640 1 -- 172.21.15.88:0/2960928647 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b4073160 msgr2=0x7f51b415adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:18.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.728+0000 7f51bb378640 1 --2- 172.21.15.88:0/2960928647 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b4073160 0x7f51b415adc0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f51a4000c00 tx=0x7f51a40046c0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.728+0000 7f51bb378640 1 -- 172.21.15.88:0/2960928647 shutdown_connections 2024-09-16T01:39:18.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.728+0000 7f51bb378640 1 --2- 172.21.15.88:0/2960928647 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f519806c2b0 0x7f519806e770 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.728+0000 7f51bb378640 1 --2- 172.21.15.88:0/2960928647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51b415c550 0x7f51b41f4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.728+0000 7f51bb378640 1 --2- 172.21.15.88:0/2960928647 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b4073160 0x7f51b415adc0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.728+0000 7f51bb378640 1 --2- 172.21.15.88:0/2960928647 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f51b40715e0 0x7f51b41e7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.728+0000 7f51bb378640 1 -- 172.21.15.88:0/2960928647 >> 172.21.15.88:0/2960928647 conn(0x7f51b40b4240 msgr2=0x7f51b415eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:18.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.728+0000 7f51bb378640 1 -- 172.21.15.88:0/2960928647 shutdown_connections 2024-09-16T01:39:18.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.728+0000 7f51bb378640 1 -- 172.21.15.88:0/2960928647 wait complete. 2024-09-16T01:39:18.740 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-09-16T01:39:18.583866+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":232400,"kb_used_data":7816,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754742320,"statfs":{"total":773094113280,"available":772856135680,"internally_reserved":0,"allocated":8003584,"data_stored":4046278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":229756466},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.123811"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.502006+0000","last_change":"2024-09-16T01:39:15.500162+0000","last_active":"2024-09-16T01:39:15.502006+0000","last_peered":"2024-09-16T01:39:15.502006+0000","last_clean":"2024-09-16T01:39:15.502006+0000","last_became_active":"2024-09-16T01:39:15.496318+0000","last_became_peered":"2024-09-16T01:39:15.496318+0000","last_unstale":"2024-09-16T01:39:15.502006+0000","last_undegraded":"2024-09-16T01:39:15.502006+0000","last_fullsized":"2024-09-16T01:39:15.502006+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:44.054894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.497912+0000","last_change":"2024-09-16T01:39:15.495072+0000","last_active":"2024-09-16T01:39:15.497912+0000","last_peered":"2024-09-16T01:39:15.497912+0000","last_clean":"2024-09-16T01:39:15.497912+0000","last_became_active":"2024-09-16T01:39:15.493376+0000","last_became_peered":"2024-09-16T01:39:15.493376+0000","last_unstale":"2024-09-16T01:39:15.497912+0000","last_undegraded":"2024-09-16T01:39:15.497912+0000","last_fullsized":"2024-09-16T01:39:15.497912+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:15:40.953770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.482523+0000","last_change":"2024-09-16T01:39:15.481136+0000","last_active":"2024-09-16T01:39:15.482523+0000","last_peered":"2024-09-16T01:39:15.482523+0000","last_clean":"2024-09-16T01:39:15.482523+0000","last_became_active":"2024-09-16T01:39:15.480891+0000","last_became_peered":"2024-09-16T01:39:15.480891+0000","last_unstale":"2024-09-16T01:39:15.482523+0000","last_undegraded":"2024-09-16T01:39:15.482523+0000","last_fullsized":"2024-09-16T01:39:15.482523+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:10:00.747727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.492345+0000","last_change":"2024-09-16T01:39:15.483406+0000","last_active":"2024-09-16T01:39:15.492345+0000","last_peered":"2024-09-16T01:39:15.492345+0000","last_clean":"2024-09-16T01:39:15.492345+0000","last_became_active":"2024-09-16T01:39:15.483143+0000","last_became_peered":"2024-09-16T01:39:15.483143+0000","last_unstale":"2024-09-16T01:39:15.492345+0000","last_undegraded":"2024-09-16T01:39:15.492345+0000","last_fullsized":"2024-09-16T01:39:15.492345+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:33:51.100506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.496029+0000","last_change":"2024-09-16T01:39:15.488889+0000","last_active":"2024-09-16T01:39:15.496029+0000","last_peered":"2024-09-16T01:39:15.496029+0000","last_clean":"2024-09-16T01:39:15.496029+0000","last_became_active":"2024-09-16T01:39:15.488539+0000","last_became_peered":"2024-09-16T01:39:15.488539+0000","last_unstale":"2024-09-16T01:39:15.496029+0000","last_undegraded":"2024-09-16T01:39:15.496029+0000","last_fullsized":"2024-09-16T01:39:15.496029+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:08:41.278079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.491684+0000","last_change":"2024-09-16T01:39:15.483949+0000","last_active":"2024-09-16T01:39:15.491684+0000","last_peered":"2024-09-16T01:39:15.491684+0000","last_clean":"2024-09-16T01:39:15.491684+0000","last_became_active":"2024-09-16T01:39:15.483408+0000","last_became_peered":"2024-09-16T01:39:15.483408+0000","last_unstale":"2024-09-16T01:39:15.491684+0000","last_undegraded":"2024-09-16T01:39:15.491684+0000","last_fullsized":"2024-09-16T01:39:15.491684+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:02:52.162549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T01:39:16.487082+0000","last_change":"2024-09-16T01:39:15.483151+0000","last_active":"2024-09-16T01:39:16.487082+0000","last_peered":"2024-09-16T01:39:16.487082+0000","last_clean":"2024-09-16T01:39:16.487082+0000","last_became_active":"2024-09-16T01:39:15.482173+0000","last_became_peered":"2024-09-16T01:39:15.482173+0000","last_unstale":"2024-09-16T01:39:16.487082+0000","last_undegraded":"2024-09-16T01:39:16.487082+0000","last_fullsized":"2024-09-16T01:39:16.487082+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:11:02.011376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.485573+0000","last_change":"2024-09-16T01:39:15.480491+0000","last_active":"2024-09-16T01:39:15.485573+0000","last_peered":"2024-09-16T01:39:15.485573+0000","last_clean":"2024-09-16T01:39:15.485573+0000","last_became_active":"2024-09-16T01:39:15.480150+0000","last_became_peered":"2024-09-16T01:39:15.480150+0000","last_unstale":"2024-09-16T01:39:15.485573+0000","last_undegraded":"2024-09-16T01:39:15.485573+0000","last_fullsized":"2024-09-16T01:39:15.485573+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:43.014904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.493553+0000","last_change":"2024-09-16T01:39:15.480553+0000","last_active":"2024-09-16T01:39:15.493553+0000","last_peered":"2024-09-16T01:39:15.493553+0000","last_clean":"2024-09-16T01:39:15.493553+0000","last_became_active":"2024-09-16T01:39:15.478968+0000","last_became_peered":"2024-09-16T01:39:15.478968+0000","last_unstale":"2024-09-16T01:39:15.493553+0000","last_undegraded":"2024-09-16T01:39:15.493553+0000","last_fullsized":"2024-09-16T01:39:15.493553+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:50.764141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.501155+0000","last_change":"2024-09-16T01:39:15.499685+0000","last_active":"2024-09-16T01:39:15.501155+0000","last_peered":"2024-09-16T01:39:15.501155+0000","last_clean":"2024-09-16T01:39:15.501155+0000","last_became_active":"2024-09-16T01:39:15.499409+0000","last_became_peered":"2024-09-16T01:39:15.499409+0000","last_unstale":"2024-09-16T01:39:15.501155+0000","last_undegraded":"2024-09-16T01:39:15.501155+0000","last_fullsized":"2024-09-16T01:39:15.501155+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:34:57.787684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.501512+0000","last_change":"2024-09-16T01:39:15.497417+0000","last_active":"2024-09-16T01:39:15.501512+0000","last_peered":"2024-09-16T01:39:15.501512+0000","last_clean":"2024-09-16T01:39:15.501512+0000","last_became_active":"2024-09-16T01:39:15.496647+0000","last_became_peered":"2024-09-16T01:39:15.496647+0000","last_unstale":"2024-09-16T01:39:15.501512+0000","last_undegraded":"2024-09-16T01:39:15.501512+0000","last_fullsized":"2024-09-16T01:39:15.501512+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:51:52.651374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.483557+0000","last_change":"2024-09-16T01:39:15.481113+0000","last_active":"2024-09-16T01:39:15.483557+0000","last_peered":"2024-09-16T01:39:15.483557+0000","last_clean":"2024-09-16T01:39:15.483557+0000","last_became_active":"2024-09-16T01:39:15.480810+0000","last_became_peered":"2024-09-16T01:39:15.480810+0000","last_unstale":"2024-09-16T01:39:15.483557+0000","last_undegraded":"2024-09-16T01:39:15.483557+0000","last_fullsized":"2024-09-16T01:39:15.483557+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:05:36.207551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.483326+0000","last_change":"2024-09-16T01:39:15.480642+0000","last_active":"2024-09-16T01:39:15.483326+0000","last_peered":"2024-09-16T01:39:15.483326+0000","last_clean":"2024-09-16T01:39:15.483326+0000","last_became_active":"2024-09-16T01:39:15.480371+0000","last_became_peered":"2024-09-16T01:39:15.480371+0000","last_unstale":"2024-09-16T01:39:15.483326+0000","last_undegraded":"2024-09-16T01:39:15.483326+0000","last_fullsized":"2024-09-16T01:39:15.483326+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:17:04.087123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.493265+0000","last_change":"2024-09-16T01:39:15.477843+0000","last_active":"2024-09-16T01:39:15.493265+0000","last_peered":"2024-09-16T01:39:15.493265+0000","last_clean":"2024-09-16T01:39:15.493265+0000","last_became_active":"2024-09-16T01:39:15.477559+0000","last_became_peered":"2024-09-16T01:39:15.477559+0000","last_unstale":"2024-09-16T01:39:15.493265+0000","last_undegraded":"2024-09-16T01:39:15.493265+0000","last_fullsized":"2024-09-16T01:39:15.493265+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:19.653901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.493941+0000","last_change":"2024-09-16T01:39:15.486422+0000","last_active":"2024-09-16T01:39:15.493941+0000","last_peered":"2024-09-16T01:39:15.493941+0000","last_clean":"2024-09-16T01:39:15.493941+0000","last_became_active":"2024-09-16T01:39:15.486128+0000","last_became_peered":"2024-09-16T01:39:15.486128+0000","last_unstale":"2024-09-16T01:39:15.493941+0000","last_undegraded":"2024-09-16T01:39:15.493941+0000","last_fullsized":"2024-09-16T01:39:15.493941+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:03:08.237538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.490505+0000","last_change":"2024-09-16T01:39:15.486426+0000","last_active":"2024-09-16T01:39:15.490505+0000","last_peered":"2024-09-16T01:39:15.490505+0000","last_clean":"2024-09-16T01:39:15.490505+0000","last_became_active":"2024-09-16T01:39:15.486138+0000","last_became_peered":"2024-09-16T01:39:15.486138+0000","last_unstale":"2024-09-16T01:39:15.490505+0000","last_undegraded":"2024-09-16T01:39:15.490505+0000","last_fullsized":"2024-09-16T01:39:15.490505+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:02:52.162549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.493515+0000","last_change":"2024-09-16T01:39:15.479421+0000","last_active":"2024-09-16T01:39:15.493515+0000","last_peered":"2024-09-16T01:39:15.493515+0000","last_clean":"2024-09-16T01:39:15.493515+0000","last_became_active":"2024-09-16T01:39:15.478127+0000","last_became_peered":"2024-09-16T01:39:15.478127+0000","last_unstale":"2024-09-16T01:39:15.493515+0000","last_undegraded":"2024-09-16T01:39:15.493515+0000","last_fullsized":"2024-09-16T01:39:15.493515+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:47:11.776578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.492177+0000","last_change":"2024-09-16T01:39:15.483906+0000","last_active":"2024-09-16T01:39:15.492177+0000","last_peered":"2024-09-16T01:39:15.492177+0000","last_clean":"2024-09-16T01:39:15.492177+0000","last_became_active":"2024-09-16T01:39:15.483639+0000","last_became_peered":"2024-09-16T01:39:15.483639+0000","last_unstale":"2024-09-16T01:39:15.492177+0000","last_undegraded":"2024-09-16T01:39:15.492177+0000","last_fullsized":"2024-09-16T01:39:15.492177+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:57:42.989890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.487191+0000","last_change":"2024-09-16T01:39:15.482961+0000","last_active":"2024-09-16T01:39:15.487191+0000","last_peered":"2024-09-16T01:39:15.487191+0000","last_clean":"2024-09-16T01:39:15.487191+0000","last_became_active":"2024-09-16T01:39:15.482687+0000","last_became_peered":"2024-09-16T01:39:15.482687+0000","last_unstale":"2024-09-16T01:39:15.487191+0000","last_undegraded":"2024-09-16T01:39:15.487191+0000","last_fullsized":"2024-09-16T01:39:15.487191+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:31:51.302505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.483150+0000","last_change":"2024-09-16T01:39:15.477677+0000","last_active":"2024-09-16T01:39:15.483150+0000","last_peered":"2024-09-16T01:39:15.483150+0000","last_clean":"2024-09-16T01:39:15.483150+0000","last_became_active":"2024-09-16T01:39:15.477388+0000","last_became_peered":"2024-09-16T01:39:15.477388+0000","last_unstale":"2024-09-16T01:39:15.483150+0000","last_undegraded":"2024-09-16T01:39:15.483150+0000","last_fullsized":"2024-09-16T01:39:15.483150+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:26:42.021902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.490931+0000","last_change":"2024-09-16T01:39:15.489962+0000","last_active":"2024-09-16T01:39:15.490931+0000","last_peered":"2024-09-16T01:39:15.490931+0000","last_clean":"2024-09-16T01:39:15.490931+0000","last_became_active":"2024-09-16T01:39:15.489667+0000","last_became_peered":"2024-09-16T01:39:15.489667+0000","last_unstale":"2024-09-16T01:39:15.490931+0000","last_undegraded":"2024-09-16T01:39:15.490931+0000","last_fullsized":"2024-09-16T01:39:15.490931+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:54:37.811306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.498343+0000","last_change":"2024-09-16T01:39:15.495324+0000","last_active":"2024-09-16T01:39:15.498343+0000","last_peered":"2024-09-16T01:39:15.498343+0000","last_clean":"2024-09-16T01:39:15.498343+0000","last_became_active":"2024-09-16T01:39:15.495052+0000","last_became_peered":"2024-09-16T01:39:15.495052+0000","last_unstale":"2024-09-16T01:39:15.498343+0000","last_undegraded":"2024-09-16T01:39:15.498343+0000","last_fullsized":"2024-09-16T01:39:15.498343+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:33:41.800796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.494989+0000","last_change":"2024-09-16T01:39:15.492957+0000","last_active":"2024-09-16T01:39:15.494989+0000","last_peered":"2024-09-16T01:39:15.494989+0000","last_clean":"2024-09-16T01:39:15.494989+0000","last_became_active":"2024-09-16T01:39:15.492700+0000","last_became_peered":"2024-09-16T01:39:15.492700+0000","last_unstale":"2024-09-16T01:39:15.494989+0000","last_undegraded":"2024-09-16T01:39:15.494989+0000","last_fullsized":"2024-09-16T01:39:15.494989+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:49:10.214476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.501988+0000","last_change":"2024-09-16T01:39:15.499849+0000","last_active":"2024-09-16T01:39:15.501988+0000","last_peered":"2024-09-16T01:39:15.501988+0000","last_clean":"2024-09-16T01:39:15.501988+0000","last_became_active":"2024-09-16T01:39:15.495301+0000","last_became_peered":"2024-09-16T01:39:15.495301+0000","last_unstale":"2024-09-16T01:39:15.501988+0000","last_undegraded":"2024-09-16T01:39:15.501988+0000","last_fullsized":"2024-09-16T01:39:15.501988+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:21:14.462992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.497396+0000","last_change":"2024-09-16T01:39:15.496321+0000","last_active":"2024-09-16T01:39:15.497396+0000","last_peered":"2024-09-16T01:39:15.497396+0000","last_clean":"2024-09-16T01:39:15.497396+0000","last_became_active":"2024-09-16T01:39:15.496025+0000","last_became_peered":"2024-09-16T01:39:15.496025+0000","last_unstale":"2024-09-16T01:39:15.497396+0000","last_undegraded":"2024-09-16T01:39:15.497396+0000","last_fullsized":"2024-09-16T01:39:15.497396+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:00:19.023789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T01:39:16.487416+0000","last_change":"2024-09-16T01:39:15.486415+0000","last_active":"2024-09-16T01:39:16.487416+0000","last_peered":"2024-09-16T01:39:16.487416+0000","last_clean":"2024-09-16T01:39:16.487416+0000","last_became_active":"2024-09-16T01:39:15.486130+0000","last_became_peered":"2024-09-16T01:39:15.486130+0000","last_unstale":"2024-09-16T01:39:16.487416+0000","last_undegraded":"2024-09-16T01:39:16.487416+0000","last_fullsized":"2024-09-16T01:39:16.487416+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:15:14.342280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.491782+0000","last_change":"2024-09-16T01:39:15.489875+0000","last_active":"2024-09-16T01:39:15.491782+0000","last_peered":"2024-09-16T01:39:15.491782+0000","last_clean":"2024-09-16T01:39:15.491782+0000","last_became_active":"2024-09-16T01:39:15.489552+0000","last_became_peered":"2024-09-16T01:39:15.489552+0000","last_unstale":"2024-09-16T01:39:15.491782+0000","last_undegraded":"2024-09-16T01:39:15.491782+0000","last_fullsized":"2024-09-16T01:39:15.491782+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:25:24.101060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.496913+0000","last_change":"2024-09-16T01:39:15.494872+0000","last_active":"2024-09-16T01:39:15.496913+0000","last_peered":"2024-09-16T01:39:15.496913+0000","last_clean":"2024-09-16T01:39:15.496913+0000","last_became_active":"2024-09-16T01:39:15.494596+0000","last_became_peered":"2024-09-16T01:39:15.494596+0000","last_unstale":"2024-09-16T01:39:15.496913+0000","last_undegraded":"2024-09-16T01:39:15.496913+0000","last_fullsized":"2024-09-16T01:39:15.496913+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:39:00.982561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.480669+0000","last_change":"2024-09-16T01:39:15.479157+0000","last_active":"2024-09-16T01:39:15.480669+0000","last_peered":"2024-09-16T01:39:15.480669+0000","last_clean":"2024-09-16T01:39:15.480669+0000","last_became_active":"2024-09-16T01:39:15.478839+0000","last_became_peered":"2024-09-16T01:39:15.478839+0000","last_unstale":"2024-09-16T01:39:15.480669+0000","last_undegraded":"2024-09-16T01:39:15.480669+0000","last_fullsized":"2024-09-16T01:39:15.480669+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:29:37.906974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T01:39:16.487140+0000","last_change":"2024-09-16T01:39:15.486645+0000","last_active":"2024-09-16T01:39:16.487140+0000","last_peered":"2024-09-16T01:39:16.487140+0000","last_clean":"2024-09-16T01:39:16.487140+0000","last_became_active":"2024-09-16T01:39:15.486343+0000","last_became_peered":"2024-09-16T01:39:15.486343+0000","last_unstale":"2024-09-16T01:39:16.487140+0000","last_undegraded":"2024-09-16T01:39:16.487140+0000","last_fullsized":"2024-09-16T01:39:16.487140+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:58:48.937237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T01:39:16.487293+0000","last_change":"2024-09-16T01:39:15.487946+0000","last_active":"2024-09-16T01:39:16.487293+0000","last_peered":"2024-09-16T01:39:16.487293+0000","last_clean":"2024-09-16T01:39:16.487293+0000","last_became_active":"2024-09-16T01:39:15.487649+0000","last_became_peered":"2024-09-16T01:39:15.487649+0000","last_unstale":"2024-09-16T01:39:16.487293+0000","last_undegraded":"2024-09-16T01:39:16.487293+0000","last_fullsized":"2024-09-16T01:39:16.487293+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:21:14.462992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.485545+0000","last_change":"2024-09-16T01:39:15.479771+0000","last_active":"2024-09-16T01:39:15.485545+0000","last_peered":"2024-09-16T01:39:15.485545+0000","last_clean":"2024-09-16T01:39:15.485545+0000","last_became_active":"2024-09-16T01:39:15.478711+0000","last_became_peered":"2024-09-16T01:39:15.478711+0000","last_unstale":"2024-09-16T01:39:15.485545+0000","last_undegraded":"2024-09-16T01:39:15.485545+0000","last_fullsized":"2024-09-16T01:39:15.485545+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:03:13.587790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.502001+0000","last_change":"2024-09-16T01:39:15.500372+0000","last_active":"2024-09-16T01:39:15.502001+0000","last_peered":"2024-09-16T01:39:15.502001+0000","last_clean":"2024-09-16T01:39:15.502001+0000","last_became_active":"2024-09-16T01:39:15.500129+0000","last_became_peered":"2024-09-16T01:39:15.500129+0000","last_unstale":"2024-09-16T01:39:15.502001+0000","last_undegraded":"2024-09-16T01:39:15.502001+0000","last_fullsized":"2024-09-16T01:39:15.502001+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:34:32.699174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"71'4","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.498317+0000","last_change":"2024-09-16T01:39:14.480906+0000","last_active":"2024-09-16T01:39:15.498317+0000","last_peered":"2024-09-16T01:39:15.498317+0000","last_clean":"2024-09-16T01:39:15.498317+0000","last_became_active":"2024-09-16T01:39:14.480606+0000","last_became_peered":"2024-09-16T01:39:14.480606+0000","last_unstale":"2024-09-16T01:39:15.498317+0000","last_undegraded":"2024-09-16T01:39:15.498317+0000","last_fullsized":"2024-09-16T01:39:15.498317+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:30:39.759873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.493537+0000","last_change":"2024-09-16T01:39:15.480300+0000","last_active":"2024-09-16T01:39:15.493537+0000","last_peered":"2024-09-16T01:39:15.493537+0000","last_clean":"2024-09-16T01:39:15.493537+0000","last_became_active":"2024-09-16T01:39:15.477956+0000","last_became_peered":"2024-09-16T01:39:15.477956+0000","last_unstale":"2024-09-16T01:39:15.493537+0000","last_undegraded":"2024-09-16T01:39:15.493537+0000","last_fullsized":"2024-09-16T01:39:15.493537+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:22.796258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"71'1","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.496417+0000","last_change":"2024-09-16T01:39:14.483678+0000","last_active":"2024-09-16T01:39:15.496417+0000","last_peered":"2024-09-16T01:39:15.496417+0000","last_clean":"2024-09-16T01:39:15.496417+0000","last_became_active":"2024-09-16T01:39:14.483390+0000","last_became_peered":"2024-09-16T01:39:14.483390+0000","last_unstale":"2024-09-16T01:39:15.496417+0000","last_undegraded":"2024-09-16T01:39:15.496417+0000","last_fullsized":"2024-09-16T01:39:15.496417+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:34:56.781236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.499580+0000","last_change":"2024-09-16T01:39:15.497713+0000","last_active":"2024-09-16T01:39:15.499580+0000","last_peered":"2024-09-16T01:39:15.499580+0000","last_clean":"2024-09-16T01:39:15.499580+0000","last_became_active":"2024-09-16T01:39:15.497228+0000","last_became_peered":"2024-09-16T01:39:15.497228+0000","last_unstale":"2024-09-16T01:39:15.499580+0000","last_undegraded":"2024-09-16T01:39:15.499580+0000","last_fullsized":"2024-09-16T01:39:15.499580+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:39:00.982561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"71'2","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.501304+0000","last_change":"2024-09-16T01:39:14.480878+0000","last_active":"2024-09-16T01:39:15.501304+0000","last_peered":"2024-09-16T01:39:15.501304+0000","last_clean":"2024-09-16T01:39:15.501304+0000","last_became_active":"2024-09-16T01:39:14.480593+0000","last_became_peered":"2024-09-16T01:39:14.480593+0000","last_unstale":"2024-09-16T01:39:15.501304+0000","last_undegraded":"2024-09-16T01:39:15.501304+0000","last_fullsized":"2024-09-16T01:39:15.501304+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:31:50.296058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.501533+0000","last_change":"2024-09-16T01:39:15.498590+0000","last_active":"2024-09-16T01:39:15.501533+0000","last_peered":"2024-09-16T01:39:15.501533+0000","last_clean":"2024-09-16T01:39:15.501533+0000","last_became_active":"2024-09-16T01:39:15.498333+0000","last_became_peered":"2024-09-16T01:39:15.498333+0000","last_unstale":"2024-09-16T01:39:15.501533+0000","last_undegraded":"2024-09-16T01:39:15.501533+0000","last_fullsized":"2024-09-16T01:39:15.501533+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:10:21.183290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"71'2","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.499832+0000","last_change":"2024-09-16T01:39:14.482912+0000","last_active":"2024-09-16T01:39:15.499832+0000","last_peered":"2024-09-16T01:39:15.499832+0000","last_clean":"2024-09-16T01:39:15.499832+0000","last_became_active":"2024-09-16T01:39:14.481707+0000","last_became_peered":"2024-09-16T01:39:14.481707+0000","last_unstale":"2024-09-16T01:39:15.499832+0000","last_undegraded":"2024-09-16T01:39:15.499832+0000","last_fullsized":"2024-09-16T01:39:15.499832+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:51:00.179012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T01:39:16.487292+0000","last_change":"2024-09-16T01:39:15.478421+0000","last_active":"2024-09-16T01:39:16.487292+0000","last_peered":"2024-09-16T01:39:16.487292+0000","last_clean":"2024-09-16T01:39:16.487292+0000","last_became_active":"2024-09-16T01:39:15.478145+0000","last_became_peered":"2024-09-16T01:39:15.478145+0000","last_unstale":"2024-09-16T01:39:16.487292+0000","last_undegraded":"2024-09-16T01:39:16.487292+0000","last_fullsized":"2024-09-16T01:39:16.487292+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:44.054894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"71'1","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.497551+0000","last_change":"2024-09-16T01:39:14.479292+0000","last_active":"2024-09-16T01:39:15.497551+0000","last_peered":"2024-09-16T01:39:15.497551+0000","last_clean":"2024-09-16T01:39:15.497551+0000","last_became_active":"2024-09-16T01:39:14.479040+0000","last_became_peered":"2024-09-16T01:39:14.479040+0000","last_unstale":"2024-09-16T01:39:15.497551+0000","last_undegraded":"2024-09-16T01:39:15.497551+0000","last_fullsized":"2024-09-16T01:39:15.497551+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:25:37.315336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.490909+0000","last_change":"2024-09-16T01:39:15.489157+0000","last_active":"2024-09-16T01:39:15.490909+0000","last_peered":"2024-09-16T01:39:15.490909+0000","last_clean":"2024-09-16T01:39:15.490909+0000","last_became_active":"2024-09-16T01:39:15.488852+0000","last_became_peered":"2024-09-16T01:39:15.488852+0000","last_unstale":"2024-09-16T01:39:15.490909+0000","last_undegraded":"2024-09-16T01:39:15.490909+0000","last_fullsized":"2024-09-16T01:39:15.490909+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:27:29.143421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"71'1","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.498123+0000","last_change":"2024-09-16T01:39:14.480930+0000","last_active":"2024-09-16T01:39:15.498123+0000","last_peered":"2024-09-16T01:39:15.498123+0000","last_clean":"2024-09-16T01:39:15.498123+0000","last_became_active":"2024-09-16T01:39:14.480669+0000","last_became_peered":"2024-09-16T01:39:14.480669+0000","last_unstale":"2024-09-16T01:39:15.498123+0000","last_undegraded":"2024-09-16T01:39:15.498123+0000","last_fullsized":"2024-09-16T01:39:15.498123+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:53:36.791253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.493347+0000","last_change":"2024-09-16T01:39:15.480132+0000","last_active":"2024-09-16T01:39:15.493347+0000","last_peered":"2024-09-16T01:39:15.493347+0000","last_clean":"2024-09-16T01:39:15.493347+0000","last_became_active":"2024-09-16T01:39:15.479878+0000","last_became_peered":"2024-09-16T01:39:15.479878+0000","last_unstale":"2024-09-16T01:39:15.493347+0000","last_undegraded":"2024-09-16T01:39:15.493347+0000","last_fullsized":"2024-09-16T01:39:15.493347+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:07:43.338533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.470305+0000","last_change":"2024-09-16T01:39:14.481778+0000","last_active":"2024-09-16T01:39:15.470305+0000","last_peered":"2024-09-16T01:39:15.470305+0000","last_clean":"2024-09-16T01:39:15.470305+0000","last_became_active":"2024-09-16T01:39:14.481515+0000","last_became_peered":"2024-09-16T01:39:14.481515+0000","last_unstale":"2024-09-16T01:39:15.470305+0000","last_undegraded":"2024-09-16T01:39:15.470305+0000","last_fullsized":"2024-09-16T01:39:15.470305+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:54:42.509627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.496113+0000","last_change":"2024-09-16T01:39:15.489802+0000","last_active":"2024-09-16T01:39:15.496113+0000","last_peered":"2024-09-16T01:39:15.496113+0000","last_clean":"2024-09-16T01:39:15.496113+0000","last_became_active":"2024-09-16T01:39:15.489496+0000","last_became_peered":"2024-09-16T01:39:15.489496+0000","last_unstale":"2024-09-16T01:39:15.496113+0000","last_undegraded":"2024-09-16T01:39:15.496113+0000","last_fullsized":"2024-09-16T01:39:15.496113+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:29:47.956916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.473324+0000","last_change":"2024-09-16T01:39:14.477667+0000","last_active":"2024-09-16T01:39:15.473324+0000","last_peered":"2024-09-16T01:39:15.473324+0000","last_clean":"2024-09-16T01:39:15.473324+0000","last_became_active":"2024-09-16T01:39:14.477358+0000","last_became_peered":"2024-09-16T01:39:14.477358+0000","last_unstale":"2024-09-16T01:39:15.473324+0000","last_undegraded":"2024-09-16T01:39:15.473324+0000","last_fullsized":"2024-09-16T01:39:15.473324+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:33.745508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T01:39:16.487387+0000","last_change":"2024-09-16T01:39:15.490784+0000","last_active":"2024-09-16T01:39:16.487387+0000","last_peered":"2024-09-16T01:39:16.487387+0000","last_clean":"2024-09-16T01:39:16.487387+0000","last_became_active":"2024-09-16T01:39:15.490517+0000","last_became_peered":"2024-09-16T01:39:15.490517+0000","last_unstale":"2024-09-16T01:39:16.487387+0000","last_undegraded":"2024-09-16T01:39:16.487387+0000","last_fullsized":"2024-09-16T01:39:16.487387+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:34:32.699174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.477730+0000","last_change":"2024-09-16T01:39:14.479610+0000","last_active":"2024-09-16T01:39:15.477730+0000","last_peered":"2024-09-16T01:39:15.477730+0000","last_clean":"2024-09-16T01:39:15.477730+0000","last_became_active":"2024-09-16T01:39:14.479316+0000","last_became_peered":"2024-09-16T01:39:14.479316+0000","last_unstale":"2024-09-16T01:39:15.477730+0000","last_undegraded":"2024-09-16T01:39:15.477730+0000","last_fullsized":"2024-09-16T01:39:15.477730+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:42.008457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.501710+0000","last_change":"2024-09-16T01:39:15.497698+0000","last_active":"2024-09-16T01:39:15.501710+0000","last_peered":"2024-09-16T01:39:15.501710+0000","last_clean":"2024-09-16T01:39:15.501710+0000","last_became_active":"2024-09-16T01:39:15.496309+0000","last_became_peered":"2024-09-16T01:39:15.496309+0000","last_unstale":"2024-09-16T01:39:15.501710+0000","last_undegraded":"2024-09-16T01:39:15.501710+0000","last_fullsized":"2024-09-16T01:39:15.501710+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:11:02.011376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.478944+0000","last_change":"2024-09-16T01:39:14.483878+0000","last_active":"2024-09-16T01:39:15.478944+0000","last_peered":"2024-09-16T01:39:15.478944+0000","last_clean":"2024-09-16T01:39:15.478944+0000","last_became_active":"2024-09-16T01:39:14.483573+0000","last_became_peered":"2024-09-16T01:39:14.483573+0000","last_unstale":"2024-09-16T01:39:15.478944+0000","last_undegraded":"2024-09-16T01:39:15.478944+0000","last_fullsized":"2024-09-16T01:39:15.478944+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:33:43.742264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.499324+0000","last_change":"2024-09-16T01:39:15.497490+0000","last_active":"2024-09-16T01:39:15.499324+0000","last_peered":"2024-09-16T01:39:15.499324+0000","last_clean":"2024-09-16T01:39:15.499324+0000","last_became_active":"2024-09-16T01:39:15.492295+0000","last_became_peered":"2024-09-16T01:39:15.492295+0000","last_unstale":"2024-09-16T01:39:15.499324+0000","last_undegraded":"2024-09-16T01:39:15.499324+0000","last_fullsized":"2024-09-16T01:39:15.499324+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:54:37.811306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"71'3","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.501031+0000","last_change":"2024-09-16T01:39:14.477914+0000","last_active":"2024-09-16T01:39:15.501031+0000","last_peered":"2024-09-16T01:39:15.501031+0000","last_clean":"2024-09-16T01:39:15.501031+0000","last_became_active":"2024-09-16T01:39:14.477537+0000","last_became_peered":"2024-09-16T01:39:14.477537+0000","last_unstale":"2024-09-16T01:39:15.501031+0000","last_undegraded":"2024-09-16T01:39:15.501031+0000","last_fullsized":"2024-09-16T01:39:15.501031+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:39:02.681571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.493236+0000","last_change":"2024-09-16T01:39:15.477919+0000","last_active":"2024-09-16T01:39:15.493236+0000","last_peered":"2024-09-16T01:39:15.493236+0000","last_clean":"2024-09-16T01:39:15.493236+0000","last_became_active":"2024-09-16T01:39:15.477436+0000","last_became_peered":"2024-09-16T01:39:15.477436+0000","last_unstale":"2024-09-16T01:39:15.493236+0000","last_undegraded":"2024-09-16T01:39:15.493236+0000","last_fullsized":"2024-09-16T01:39:15.493236+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:42:28.285765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"71'6","reported_seq":18,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.500749+0000","last_change":"2024-09-16T01:39:14.480691+0000","last_active":"2024-09-16T01:39:15.500749+0000","last_peered":"2024-09-16T01:39:15.500749+0000","last_clean":"2024-09-16T01:39:15.500749+0000","last_became_active":"2024-09-16T01:39:14.480411+0000","last_became_peered":"2024-09-16T01:39:14.480411+0000","last_unstale":"2024-09-16T01:39:15.500749+0000","last_undegraded":"2024-09-16T01:39:15.500749+0000","last_fullsized":"2024-09-16T01:39:15.500749+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:54:15.531756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.500076+0000","last_change":"2024-09-16T01:39:15.496405+0000","last_active":"2024-09-16T01:39:15.500076+0000","last_peered":"2024-09-16T01:39:15.500076+0000","last_clean":"2024-09-16T01:39:15.500076+0000","last_became_active":"2024-09-16T01:39:15.496106+0000","last_became_peered":"2024-09-16T01:39:15.496106+0000","last_unstale":"2024-09-16T01:39:15.500076+0000","last_undegraded":"2024-09-16T01:39:15.500076+0000","last_fullsized":"2024-09-16T01:39:15.500076+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:04:12.294726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"71'2","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.495915+0000","last_change":"2024-09-16T01:39:14.477452+0000","last_active":"2024-09-16T01:39:15.495915+0000","last_peered":"2024-09-16T01:39:15.495915+0000","last_clean":"2024-09-16T01:39:15.495915+0000","last_became_active":"2024-09-16T01:39:14.477178+0000","last_became_peered":"2024-09-16T01:39:14.477178+0000","last_unstale":"2024-09-16T01:39:15.495915+0000","last_undegraded":"2024-09-16T01:39:15.495915+0000","last_fullsized":"2024-09-16T01:39:15.495915+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:58:08.056124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T01:39:16.487195+0000","last_change":"2024-09-16T01:39:15.489991+0000","last_active":"2024-09-16T01:39:16.487195+0000","last_peered":"2024-09-16T01:39:16.487195+0000","last_clean":"2024-09-16T01:39:16.487195+0000","last_became_active":"2024-09-16T01:39:15.489713+0000","last_became_peered":"2024-09-16T01:39:15.489713+0000","last_unstale":"2024-09-16T01:39:16.487195+0000","last_undegraded":"2024-09-16T01:39:16.487195+0000","last_fullsized":"2024-09-16T01:39:16.487195+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:10:21.183290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.477291+0000","last_change":"2024-09-16T01:39:14.473788+0000","last_active":"2024-09-16T01:39:15.477291+0000","last_peered":"2024-09-16T01:39:15.477291+0000","last_clean":"2024-09-16T01:39:15.477291+0000","last_became_active":"2024-09-16T01:39:14.473538+0000","last_became_peered":"2024-09-16T01:39:14.473538+0000","last_unstale":"2024-09-16T01:39:15.477291+0000","last_undegraded":"2024-09-16T01:39:15.477291+0000","last_fullsized":"2024-09-16T01:39:15.477291+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:26:57.297516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.488191+0000","last_change":"2024-09-16T01:39:15.485010+0000","last_active":"2024-09-16T01:39:15.488191+0000","last_peered":"2024-09-16T01:39:15.488191+0000","last_clean":"2024-09-16T01:39:15.488191+0000","last_became_active":"2024-09-16T01:39:15.484664+0000","last_became_peered":"2024-09-16T01:39:15.484664+0000","last_unstale":"2024-09-16T01:39:15.488191+0000","last_undegraded":"2024-09-16T01:39:15.488191+0000","last_fullsized":"2024-09-16T01:39:15.488191+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:24:16.034246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T01:39:16.487130+0000","last_change":"2024-09-16T01:39:14.485795+0000","last_active":"2024-09-16T01:39:16.487130+0000","last_peered":"2024-09-16T01:39:16.487130+0000","last_clean":"2024-09-16T01:39:16.487130+0000","last_became_active":"2024-09-16T01:39:14.485549+0000","last_became_peered":"2024-09-16T01:39:14.485549+0000","last_unstale":"2024-09-16T01:39:16.487130+0000","last_undegraded":"2024-09-16T01:39:16.487130+0000","last_fullsized":"2024-09-16T01:39:16.487130+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:43:33.350600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.485794+0000","last_change":"2024-09-16T01:39:15.476934+0000","last_active":"2024-09-16T01:39:15.485794+0000","last_peered":"2024-09-16T01:39:15.485794+0000","last_clean":"2024-09-16T01:39:15.485794+0000","last_became_active":"2024-09-16T01:39:15.476683+0000","last_became_peered":"2024-09-16T01:39:15.476683+0000","last_unstale":"2024-09-16T01:39:15.485794+0000","last_undegraded":"2024-09-16T01:39:15.485794+0000","last_fullsized":"2024-09-16T01:39:15.485794+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:58:02.701821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"71'8","reported_seq":20,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.502530+0000","last_change":"2024-09-16T01:39:14.475088+0000","last_active":"2024-09-16T01:39:15.502530+0000","last_peered":"2024-09-16T01:39:15.502530+0000","last_clean":"2024-09-16T01:39:15.502530+0000","last_became_active":"2024-09-16T01:39:14.474752+0000","last_became_peered":"2024-09-16T01:39:14.474752+0000","last_unstale":"2024-09-16T01:39:15.502530+0000","last_undegraded":"2024-09-16T01:39:15.502530+0000","last_fullsized":"2024-09-16T01:39:15.502530+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:13.036717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.501415+0000","last_change":"2024-09-16T01:39:15.499795+0000","last_active":"2024-09-16T01:39:15.501415+0000","last_peered":"2024-09-16T01:39:15.501415+0000","last_clean":"2024-09-16T01:39:15.501415+0000","last_became_active":"2024-09-16T01:39:15.499516+0000","last_became_peered":"2024-09-16T01:39:15.499516+0000","last_unstale":"2024-09-16T01:39:15.501415+0000","last_undegraded":"2024-09-16T01:39:15.501415+0000","last_fullsized":"2024-09-16T01:39:15.501415+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:26:42.021902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"71'1","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.498430+0000","last_change":"2024-09-16T01:39:14.478767+0000","last_active":"2024-09-16T01:39:15.498430+0000","last_peered":"2024-09-16T01:39:15.498430+0000","last_clean":"2024-09-16T01:39:15.498430+0000","last_became_active":"2024-09-16T01:39:14.478445+0000","last_became_peered":"2024-09-16T01:39:14.478445+0000","last_unstale":"2024-09-16T01:39:15.498430+0000","last_undegraded":"2024-09-16T01:39:15.498430+0000","last_fullsized":"2024-09-16T01:39:15.498430+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:02:37.714944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.501738+0000","last_change":"2024-09-16T01:39:15.498536+0000","last_active":"2024-09-16T01:39:15.501738+0000","last_peered":"2024-09-16T01:39:15.501738+0000","last_clean":"2024-09-16T01:39:15.501738+0000","last_became_active":"2024-09-16T01:39:15.497432+0000","last_became_peered":"2024-09-16T01:39:15.497432+0000","last_unstale":"2024-09-16T01:39:15.501738+0000","last_undegraded":"2024-09-16T01:39:15.501738+0000","last_fullsized":"2024-09-16T01:39:15.501738+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:58:48.937237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"71'2","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.501354+0000","last_change":"2024-09-16T01:39:14.479568+0000","last_active":"2024-09-16T01:39:15.501354+0000","last_peered":"2024-09-16T01:39:15.501354+0000","last_clean":"2024-09-16T01:39:15.501354+0000","last_became_active":"2024-09-16T01:39:14.479287+0000","last_became_peered":"2024-09-16T01:39:14.479287+0000","last_unstale":"2024-09-16T01:39:15.501354+0000","last_undegraded":"2024-09-16T01:39:15.501354+0000","last_fullsized":"2024-09-16T01:39:15.501354+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:15:39.947322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.493128+0000","last_change":"2024-09-16T01:39:15.486333+0000","last_active":"2024-09-16T01:39:15.493128+0000","last_peered":"2024-09-16T01:39:15.493128+0000","last_clean":"2024-09-16T01:39:15.493128+0000","last_became_active":"2024-09-16T01:39:15.486086+0000","last_became_peered":"2024-09-16T01:39:15.486086+0000","last_unstale":"2024-09-16T01:39:15.493128+0000","last_undegraded":"2024-09-16T01:39:15.493128+0000","last_fullsized":"2024-09-16T01:39:15.493128+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:01:56.501512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"71'1","reported_seq":13,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.498831+0000","last_change":"2024-09-16T01:39:14.482959+0000","last_active":"2024-09-16T01:39:15.498831+0000","last_peered":"2024-09-16T01:39:15.498831+0000","last_clean":"2024-09-16T01:39:15.498831+0000","last_became_active":"2024-09-16T01:39:14.482511+0000","last_became_peered":"2024-09-16T01:39:14.482511+0000","last_unstale":"2024-09-16T01:39:15.498831+0000","last_undegraded":"2024-09-16T01:39:15.498831+0000","last_fullsized":"2024-09-16T01:39:15.498831+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:58:30.468300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.496120+0000","last_change":"2024-09-16T01:39:15.491483+0000","last_active":"2024-09-16T01:39:15.496120+0000","last_peered":"2024-09-16T01:39:15.496120+0000","last_clean":"2024-09-16T01:39:15.496120+0000","last_became_active":"2024-09-16T01:39:15.491256+0000","last_became_peered":"2024-09-16T01:39:15.491256+0000","last_unstale":"2024-09-16T01:39:15.496120+0000","last_undegraded":"2024-09-16T01:39:15.496120+0000","last_fullsized":"2024-09-16T01:39:15.496120+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:42:14.774404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.480777+0000","last_change":"2024-09-16T01:39:14.482639+0000","last_active":"2024-09-16T01:39:15.480777+0000","last_peered":"2024-09-16T01:39:15.480777+0000","last_clean":"2024-09-16T01:39:15.480777+0000","last_became_active":"2024-09-16T01:39:14.482388+0000","last_became_peered":"2024-09-16T01:39:14.482388+0000","last_unstale":"2024-09-16T01:39:15.480777+0000","last_undegraded":"2024-09-16T01:39:15.480777+0000","last_fullsized":"2024-09-16T01:39:15.480777+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:39:40.424008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.480649+0000","last_change":"2024-09-16T01:39:15.478509+0000","last_active":"2024-09-16T01:39:15.480649+0000","last_peered":"2024-09-16T01:39:15.480649+0000","last_clean":"2024-09-16T01:39:15.480649+0000","last_became_active":"2024-09-16T01:39:15.478016+0000","last_became_peered":"2024-09-16T01:39:15.478016+0000","last_unstale":"2024-09-16T01:39:15.480649+0000","last_undegraded":"2024-09-16T01:39:15.480649+0000","last_fullsized":"2024-09-16T01:39:15.480649+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:46:23.043172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.477522+0000","last_change":"2024-09-16T01:39:14.480323+0000","last_active":"2024-09-16T01:39:15.477522+0000","last_peered":"2024-09-16T01:39:15.477522+0000","last_clean":"2024-09-16T01:39:15.477522+0000","last_became_active":"2024-09-16T01:39:14.480070+0000","last_became_peered":"2024-09-16T01:39:14.480070+0000","last_unstale":"2024-09-16T01:39:15.477522+0000","last_undegraded":"2024-09-16T01:39:15.477522+0000","last_fullsized":"2024-09-16T01:39:15.477522+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:13:58.290073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.483240+0000","last_change":"2024-09-16T01:39:15.482211+0000","last_active":"2024-09-16T01:39:15.483240+0000","last_peered":"2024-09-16T01:39:15.483240+0000","last_clean":"2024-09-16T01:39:15.483240+0000","last_became_active":"2024-09-16T01:39:15.481895+0000","last_became_peered":"2024-09-16T01:39:15.481895+0000","last_unstale":"2024-09-16T01:39:15.483240+0000","last_undegraded":"2024-09-16T01:39:15.483240+0000","last_fullsized":"2024-09-16T01:39:15.483240+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:19:55.217976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T01:39:16.487280+0000","last_change":"2024-09-16T01:39:14.485506+0000","last_active":"2024-09-16T01:39:16.487280+0000","last_peered":"2024-09-16T01:39:16.487280+0000","last_clean":"2024-09-16T01:39:16.487280+0000","last_became_active":"2024-09-16T01:39:14.485324+0000","last_became_peered":"2024-09-16T01:39:14.485324+0000","last_unstale":"2024-09-16T01:39:16.487280+0000","last_undegraded":"2024-09-16T01:39:16.487280+0000","last_fullsized":"2024-09-16T01:39:16.487280+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:08.626483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.483269+0000","last_change":"2024-09-16T01:39:15.482198+0000","last_active":"2024-09-16T01:39:15.483269+0000","last_peered":"2024-09-16T01:39:15.483269+0000","last_clean":"2024-09-16T01:39:15.483269+0000","last_became_active":"2024-09-16T01:39:15.481950+0000","last_became_peered":"2024-09-16T01:39:15.481950+0000","last_unstale":"2024-09-16T01:39:15.483269+0000","last_undegraded":"2024-09-16T01:39:15.483269+0000","last_fullsized":"2024-09-16T01:39:15.483269+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:06:04.402749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.471094+0000","last_change":"2024-09-16T01:39:14.480725+0000","last_active":"2024-09-16T01:39:15.471094+0000","last_peered":"2024-09-16T01:39:15.471094+0000","last_clean":"2024-09-16T01:39:15.471094+0000","last_became_active":"2024-09-16T01:39:14.480462+0000","last_became_peered":"2024-09-16T01:39:14.480462+0000","last_unstale":"2024-09-16T01:39:15.471094+0000","last_undegraded":"2024-09-16T01:39:15.471094+0000","last_fullsized":"2024-09-16T01:39:15.471094+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:33:40.794349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.493943+0000","last_change":"2024-09-16T01:39:15.486880+0000","last_active":"2024-09-16T01:39:15.493943+0000","last_peered":"2024-09-16T01:39:15.493943+0000","last_clean":"2024-09-16T01:39:15.493943+0000","last_became_active":"2024-09-16T01:39:15.486551+0000","last_became_peered":"2024-09-16T01:39:15.486551+0000","last_unstale":"2024-09-16T01:39:15.493943+0000","last_undegraded":"2024-09-16T01:39:15.493943+0000","last_fullsized":"2024-09-16T01:39:15.493943+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:56.770000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"71'2","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.498756+0000","last_change":"2024-09-16T01:39:14.481961+0000","last_active":"2024-09-16T01:39:15.498756+0000","last_peered":"2024-09-16T01:39:15.498756+0000","last_clean":"2024-09-16T01:39:15.498756+0000","last_became_active":"2024-09-16T01:39:14.481683+0000","last_became_peered":"2024-09-16T01:39:14.481683+0000","last_unstale":"2024-09-16T01:39:15.498756+0000","last_undegraded":"2024-09-16T01:39:15.498756+0000","last_fullsized":"2024-09-16T01:39:15.498756+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:44:33.074920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T01:39:16.487443+0000","last_change":"2024-09-16T01:39:15.478929+0000","last_active":"2024-09-16T01:39:16.487443+0000","last_peered":"2024-09-16T01:39:16.487443+0000","last_clean":"2024-09-16T01:39:16.487443+0000","last_became_active":"2024-09-16T01:39:15.478658+0000","last_became_peered":"2024-09-16T01:39:15.478658+0000","last_unstale":"2024-09-16T01:39:16.487443+0000","last_undegraded":"2024-09-16T01:39:16.487443+0000","last_fullsized":"2024-09-16T01:39:16.487443+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:32:50.936209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"72'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-16T01:39:16.488840+0000","last_change":"2024-09-16T01:39:14.480924+0000","last_active":"2024-09-16T01:39:16.488840+0000","last_peered":"2024-09-16T01:39:16.488840+0000","last_clean":"2024-09-16T01:39:16.488840+0000","last_became_active":"2024-09-16T01:39:14.480660+0000","last_became_peered":"2024-09-16T01:39:14.480660+0000","last_unstale":"2024-09-16T01:39:16.488840+0000","last_undegraded":"2024-09-16T01:39:16.488840+0000","last_fullsized":"2024-09-16T01:39:16.488840+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:45:28.982396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.500130+0000","last_change":"2024-09-16T01:39:15.496128+0000","last_active":"2024-09-16T01:39:15.500130+0000","last_peered":"2024-09-16T01:39:15.500130+0000","last_clean":"2024-09-16T01:39:15.500130+0000","last_became_active":"2024-09-16T01:39:15.495841+0000","last_became_peered":"2024-09-16T01:39:15.495841+0000","last_unstale":"2024-09-16T01:39:15.500130+0000","last_undegraded":"2024-09-16T01:39:15.500130+0000","last_fullsized":"2024-09-16T01:39:15.500130+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:40.261799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.473872+0000","last_change":"2024-09-16T01:39:14.480571+0000","last_active":"2024-09-16T01:39:15.473872+0000","last_peered":"2024-09-16T01:39:15.473872+0000","last_clean":"2024-09-16T01:39:15.473872+0000","last_became_active":"2024-09-16T01:39:14.480331+0000","last_became_peered":"2024-09-16T01:39:14.480331+0000","last_unstale":"2024-09-16T01:39:15.473872+0000","last_undegraded":"2024-09-16T01:39:15.473872+0000","last_fullsized":"2024-09-16T01:39:15.473872+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:44:56.435514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.496889+0000","last_change":"2024-09-16T01:39:15.494048+0000","last_active":"2024-09-16T01:39:15.496889+0000","last_peered":"2024-09-16T01:39:15.496889+0000","last_clean":"2024-09-16T01:39:15.496889+0000","last_became_active":"2024-09-16T01:39:15.493774+0000","last_became_peered":"2024-09-16T01:39:15.493774+0000","last_unstale":"2024-09-16T01:39:15.496889+0000","last_undegraded":"2024-09-16T01:39:15.496889+0000","last_fullsized":"2024-09-16T01:39:15.496889+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:55:10.488556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.471584+0000","last_change":"2024-09-16T01:39:14.482307+0000","last_active":"2024-09-16T01:39:15.471584+0000","last_peered":"2024-09-16T01:39:15.471584+0000","last_clean":"2024-09-16T01:39:15.471584+0000","last_became_active":"2024-09-16T01:39:14.481787+0000","last_became_peered":"2024-09-16T01:39:14.481787+0000","last_unstale":"2024-09-16T01:39:15.471584+0000","last_undegraded":"2024-09-16T01:39:15.471584+0000","last_fullsized":"2024-09-16T01:39:15.471584+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:00:18.017341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.493287+0000","last_change":"2024-09-16T01:39:15.477617+0000","last_active":"2024-09-16T01:39:15.493287+0000","last_peered":"2024-09-16T01:39:15.493287+0000","last_clean":"2024-09-16T01:39:15.493287+0000","last_became_active":"2024-09-16T01:39:15.477332+0000","last_became_peered":"2024-09-16T01:39:15.477332+0000","last_unstale":"2024-09-16T01:39:15.493287+0000","last_undegraded":"2024-09-16T01:39:15.493287+0000","last_fullsized":"2024-09-16T01:39:15.493287+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:28:25.405417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.473593+0000","last_change":"2024-09-16T01:39:14.480491+0000","last_active":"2024-09-16T01:39:15.473593+0000","last_peered":"2024-09-16T01:39:15.473593+0000","last_clean":"2024-09-16T01:39:15.473593+0000","last_became_active":"2024-09-16T01:39:14.480222+0000","last_became_peered":"2024-09-16T01:39:14.480222+0000","last_unstale":"2024-09-16T01:39:15.473593+0000","last_undegraded":"2024-09-16T01:39:15.473593+0000","last_fullsized":"2024-09-16T01:39:15.473593+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:02.334769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.499352+0000","last_change":"2024-09-16T01:39:15.497331+0000","last_active":"2024-09-16T01:39:15.499352+0000","last_peered":"2024-09-16T01:39:15.499352+0000","last_clean":"2024-09-16T01:39:15.499352+0000","last_became_active":"2024-09-16T01:39:15.497045+0000","last_became_peered":"2024-09-16T01:39:15.497045+0000","last_unstale":"2024-09-16T01:39:15.499352+0000","last_undegraded":"2024-09-16T01:39:15.499352+0000","last_fullsized":"2024-09-16T01:39:15.499352+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:27:29.143421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.470692+0000","last_change":"2024-09-16T01:39:14.478513+0000","last_active":"2024-09-16T01:39:15.470692+0000","last_peered":"2024-09-16T01:39:15.470692+0000","last_clean":"2024-09-16T01:39:15.470692+0000","last_became_active":"2024-09-16T01:39:14.478246+0000","last_became_peered":"2024-09-16T01:39:14.478246+0000","last_unstale":"2024-09-16T01:39:15.470692+0000","last_undegraded":"2024-09-16T01:39:15.470692+0000","last_fullsized":"2024-09-16T01:39:15.470692+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:53:12.947025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.480026+0000","last_change":"2024-09-16T01:39:15.476544+0000","last_active":"2024-09-16T01:39:15.480026+0000","last_peered":"2024-09-16T01:39:15.480026+0000","last_clean":"2024-09-16T01:39:15.480026+0000","last_became_active":"2024-09-16T01:39:15.476198+0000","last_became_peered":"2024-09-16T01:39:15.476198+0000","last_unstale":"2024-09-16T01:39:15.480026+0000","last_undegraded":"2024-09-16T01:39:15.480026+0000","last_fullsized":"2024-09-16T01:39:15.480026+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:35:56.620792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.476800+0000","last_change":"2024-09-16T01:39:14.478534+0000","last_active":"2024-09-16T01:39:15.476800+0000","last_peered":"2024-09-16T01:39:15.476800+0000","last_clean":"2024-09-16T01:39:15.476800+0000","last_became_active":"2024-09-16T01:39:14.478266+0000","last_became_peered":"2024-09-16T01:39:14.478266+0000","last_unstale":"2024-09-16T01:39:15.476800+0000","last_undegraded":"2024-09-16T01:39:15.476800+0000","last_fullsized":"2024-09-16T01:39:15.476800+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:03:12.581343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.485609+0000","last_change":"2024-09-16T01:39:15.480158+0000","last_active":"2024-09-16T01:39:15.485609+0000","last_peered":"2024-09-16T01:39:15.485609+0000","last_clean":"2024-09-16T01:39:15.485609+0000","last_became_active":"2024-09-16T01:39:15.479774+0000","last_became_peered":"2024-09-16T01:39:15.479774+0000","last_unstale":"2024-09-16T01:39:15.485609+0000","last_undegraded":"2024-09-16T01:39:15.485609+0000","last_fullsized":"2024-09-16T01:39:15.485609+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:14.043165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.477016+0000","last_change":"2024-09-16T01:39:14.478551+0000","last_active":"2024-09-16T01:39:15.477016+0000","last_peered":"2024-09-16T01:39:15.477016+0000","last_clean":"2024-09-16T01:39:15.477016+0000","last_became_active":"2024-09-16T01:39:14.478222+0000","last_became_peered":"2024-09-16T01:39:14.478222+0000","last_unstale":"2024-09-16T01:39:15.477016+0000","last_undegraded":"2024-09-16T01:39:15.477016+0000","last_fullsized":"2024-09-16T01:39:15.477016+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:55:06.749555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.482481+0000","last_change":"2024-09-16T01:39:15.480517+0000","last_active":"2024-09-16T01:39:15.482481+0000","last_peered":"2024-09-16T01:39:15.482481+0000","last_clean":"2024-09-16T01:39:15.482481+0000","last_became_active":"2024-09-16T01:39:15.480197+0000","last_became_peered":"2024-09-16T01:39:15.480197+0000","last_unstale":"2024-09-16T01:39:15.482481+0000","last_undegraded":"2024-09-16T01:39:15.482481+0000","last_fullsized":"2024-09-16T01:39:15.482481+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:14.457974+0000","last_clean_scrub_stamp":"2024-09-16T01:39:14.457974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:31:40.532529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"71'2","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.500281+0000","last_change":"2024-09-16T01:39:14.481270+0000","last_active":"2024-09-16T01:39:15.500281+0000","last_peered":"2024-09-16T01:39:15.500281+0000","last_clean":"2024-09-16T01:39:15.500281+0000","last_became_active":"2024-09-16T01:39:14.480982+0000","last_became_peered":"2024-09-16T01:39:14.480982+0000","last_unstale":"2024-09-16T01:39:15.500281+0000","last_undegraded":"2024-09-16T01:39:15.500281+0000","last_fullsized":"2024-09-16T01:39:15.500281+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:39:13.451527+0000","last_clean_scrub_stamp":"2024-09-16T01:39:13.451527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:32:26.475867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":261,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-09-16T01:39:15.482902+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:39:15.482902+0000","last_peered":"2024-09-16T01:39:15.482902+0000","last_clean":"2024-09-16T01:39:15.482902+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:39:15.482902+0000","last_undegraded":"2024-09-16T01:39:15.482902+0000","last_fullsized":"2024-09-16T01:39:15.482902+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509506,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32036,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339804,"statfs":{"total":96636764160,"available":96603959296,"internally_reserved":0,"allocated":864256,"data_stored":363344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542209,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32604,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1441792,"data_stored":946847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":847872,"data_stored":356479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":864256,"data_stored":363324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542208,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":847872,"data_stored":356479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542209,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":847872,"data_stored":356479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542208,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":847872,"data_stored":356479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542208,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28436,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343404,"statfs":{"total":96636764160,"available":96607645696,"internally_reserved":0,"allocated":1441792,"data_stored":946847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:39:18.744 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T01:39:18.745 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-16T01:39:18.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.885+0000 7f195bc80640 1 Processor -- start 2024-09-16T01:39:18.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.885+0000 7f195bc80640 1 -- start start 2024-09-16T01:39:18.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.886+0000 7f195bc80640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f195406bc10 0x7f195406c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:18.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.886+0000 7f195bc80640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f195406f630 0x7f195406faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:18.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.886+0000 7f195bc80640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f195415c4d0 con 0x7f195415c0f0 2024-09-16T01:39:18.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.886+0000 7f195bc80640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f19540740e0 con 0x7f195406bc10 2024-09-16T01:39:18.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.886+0000 7f195bc80640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f195415ee60 con 0x7f195406f630 2024-09-16T01:39:18.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.886+0000 7f195921b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f195406bc10 0x7f195406c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:18.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.886+0000 7f195921b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f195406bc10 0x7f195406c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37770/0 (socket says 172.21.15.88:37770) 2024-09-16T01:39:18.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.886+0000 7f195921b640 1 -- 172.21.15.88:0/4270326365 learned_addr learned my addr 172.21.15.88:0/4270326365 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:18.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.887+0000 7f195921b640 1 -- 172.21.15.88:0/4270326365 >> v2:172.21.15.154:3300/0 conn(0x7f195406f630 msgr2=0x7f195406faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:18.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.887+0000 7f195921b640 1 --2- 172.21.15.88:0/4270326365 >> v2:172.21.15.154:3300/0 conn(0x7f195406f630 0x7f195406faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.887+0000 7f195921b640 1 -- 172.21.15.88:0/4270326365 >> v1:172.21.15.154:6789/0 conn(0x7f195415c0f0 legacy=0x7f19540739d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:18.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.887+0000 7f195921b640 1 -- 172.21.15.88:0/4270326365 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f195415f140 con 0x7f195406bc10 2024-09-16T01:39:18.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.887+0000 7f195921b640 1 --2- 172.21.15.88:0/4270326365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f195406bc10 0x7f195406c0a0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f19540758c0 tx=0x7f194802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be86270838c37294 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:18.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.887+0000 7f1958a1a640 1 -- 172.21.15.88:0/4270326365 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1948037070 con 0x7f195406bc10 2024-09-16T01:39:18.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.888+0000 7f1958a1a640 1 -- 172.21.15.88:0/4270326365 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1948032440 con 0x7f195406bc10 2024-09-16T01:39:18.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.888+0000 7f1958a1a640 1 -- 172.21.15.88:0/4270326365 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1948032810 con 0x7f195406bc10 2024-09-16T01:39:18.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.888+0000 7f195bc80640 1 -- 172.21.15.88:0/4270326365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f195406bc10 msgr2=0x7f195406c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:18.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.888+0000 7f195bc80640 1 --2- 172.21.15.88:0/4270326365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f195406bc10 0x7f195406c0a0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f19540758c0 tx=0x7f194802a1a0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.888+0000 7f195bc80640 1 -- 172.21.15.88:0/4270326365 shutdown_connections 2024-09-16T01:39:18.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.888+0000 7f195bc80640 1 --2- 172.21.15.88:0/4270326365 >> v2:172.21.15.154:3300/0 conn(0x7f195406f630 0x7f195406faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.888+0000 7f195bc80640 1 --2- 172.21.15.88:0/4270326365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f195406bc10 0x7f195406c0a0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.888+0000 7f195bc80640 1 -- 172.21.15.88:0/4270326365 >> 172.21.15.88:0/4270326365 conn(0x7f19540d1390 msgr2=0x7f19540d17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:18.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.888+0000 7f195bc80640 1 -- 172.21.15.88:0/4270326365 shutdown_connections 2024-09-16T01:39:18.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.888+0000 7f195bc80640 1 -- 172.21.15.88:0/4270326365 wait complete. 2024-09-16T01:39:18.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.889+0000 7f195bc80640 1 Processor -- start 2024-09-16T01:39:18.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.889+0000 7f195bc80640 1 -- start start 2024-09-16T01:39:18.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.890+0000 7f195bc80640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f195406bc10 0x7f19541520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:18.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.890+0000 7f195bc80640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f195406f630 0x7f195415a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:18.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.890+0000 7f195bc80640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f195415c0f0 0x7f195406e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:18.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.890+0000 7f195bc80640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1954071ef0 con 0x7f195406bc10 2024-09-16T01:39:18.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.890+0000 7f195bc80640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f195415eb50 con 0x7f195406f630 2024-09-16T01:39:18.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.890+0000 7f195bc80640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1954072070 con 0x7f195415c0f0 2024-09-16T01:39:18.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.890+0000 7f1959a1c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f195406bc10 0x7f19541520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:18.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.890+0000 7f1959a1c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f195406bc10 0x7f19541520c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37776/0 (socket says 172.21.15.88:37776) 2024-09-16T01:39:18.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.890+0000 7f1959a1c640 1 -- 172.21.15.88:0/3489837796 learned_addr learned my addr 172.21.15.88:0/3489837796 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:18.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.890+0000 7f1959a1c640 1 -- 172.21.15.88:0/3489837796 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f195415c0f0 msgr2=0x7f195406e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:18.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.890+0000 7f1959a1c640 1 --2- 172.21.15.88:0/3489837796 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f195415c0f0 0x7f195406e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.890+0000 7f1959a1c640 1 -- 172.21.15.88:0/3489837796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f195406f630 msgr2=0x7f195415a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:18.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.890+0000 7f1959a1c640 1 --2- 172.21.15.88:0/3489837796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f195406f630 0x7f195415a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:18.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.890+0000 7f1959a1c640 1 -- 172.21.15.88:0/3489837796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19540b54e0 con 0x7f195406bc10 2024-09-16T01:39:18.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.891+0000 7f1959a1c640 1 --2- 172.21.15.88:0/3489837796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f195406bc10 0x7f19541520c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f194400ba50 tx=0x7f194400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:18.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.891+0000 7f19527fc640 1 -- 172.21.15.88:0/3489837796 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1944019070 con 0x7f195406bc10 2024-09-16T01:39:18.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.891+0000 7f19527fc640 1 -- 172.21.15.88:0/3489837796 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f194400cc10 con 0x7f195406bc10 2024-09-16T01:39:18.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.891+0000 7f19527fc640 1 -- 172.21.15.88:0/3489837796 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1944014440 con 0x7f195406bc10 2024-09-16T01:39:18.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.891+0000 7f195bc80640 1 -- 172.21.15.88:0/3489837796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19540b57c0 con 0x7f195406bc10 2024-09-16T01:39:18.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.891+0000 7f195bc80640 1 -- 172.21.15.88:0/3489837796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19540b5d80 con 0x7f195406bc10 2024-09-16T01:39:18.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.892+0000 7f195bc80640 1 -- 172.21.15.88:0/3489837796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19540b0ba0 con 0x7f195406bc10 2024-09-16T01:39:18.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.892+0000 7f19527fc640 1 -- 172.21.15.88:0/3489837796 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f194401c070 con 0x7f195406bc10 2024-09-16T01:39:18.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.893+0000 7f19527fc640 1 --2- 172.21.15.88:0/3489837796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f194006c2b0 0x7f194006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:18.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.893+0000 7f19527fc640 1 -- 172.21.15.88:0/3489837796 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f194408b270 con 0x7f195406bc10 2024-09-16T01:39:18.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.893+0000 7f195921b640 1 --2- 172.21.15.88:0/3489837796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f194006c2b0 0x7f194006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:18.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.893+0000 7f195921b640 1 --2- 172.21.15.88:0/3489837796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f194006c2b0 0x7f194006e770 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1948034000 tx=0x7f194802c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:18.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:18.897+0000 7f19527fc640 1 -- 172.21.15.88:0/3489837796 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f19440599e0 con 0x7f195406bc10 2024-09-16T01:39:19.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.127+0000 7f195bc80640 1 -- 172.21.15.88:0/3489837796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f1954158f70 con 0x7f195406bc10 2024-09-16T01:39:19.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.129+0000 7f19527fc640 1 -- 172.21.15.88:0/3489837796 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) ==== 190+0+147 (secure 0 0 0) 0x7f19440592e0 con 0x7f195406bc10 2024-09-16T01:39:19.130 INFO:teuthology.orchestra.run.smithi088.stdout:[client.0] 2024-09-16T01:39:19.131 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDeiudmvPikIBAAyHxNNnwQKygxg09PNQLc/Q== 2024-09-16T01:39:19.131 INFO:teuthology.orchestra.run.smithi088.stdout: caps mds = "allow" 2024-09-16T01:39:19.131 INFO:teuthology.orchestra.run.smithi088.stdout: caps mon = "allow r" 2024-09-16T01:39:19.131 INFO:teuthology.orchestra.run.smithi088.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-16T01:39:19.131 INFO:teuthology.orchestra.run.smithi088.stderr:updated caps for client.0 2024-09-16T01:39:19.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.131+0000 7f195bc80640 1 -- 172.21.15.88:0/3489837796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f194006c2b0 msgr2=0x7f194006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:19.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.131+0000 7f195bc80640 1 --2- 172.21.15.88:0/3489837796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f194006c2b0 0x7f194006e770 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1948034000 tx=0x7f194802c000 comp rx=0 tx=0).stop 2024-09-16T01:39:19.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.131+0000 7f195bc80640 1 -- 172.21.15.88:0/3489837796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f195406bc10 msgr2=0x7f19541520c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:19.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.131+0000 7f195bc80640 1 --2- 172.21.15.88:0/3489837796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f195406bc10 0x7f19541520c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f194400ba50 tx=0x7f194400bf10 comp rx=0 tx=0).stop 2024-09-16T01:39:19.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.131+0000 7f195bc80640 1 -- 172.21.15.88:0/3489837796 shutdown_connections 2024-09-16T01:39:19.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.131+0000 7f195bc80640 1 --2- 172.21.15.88:0/3489837796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f194006c2b0 0x7f194006e770 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.131+0000 7f195bc80640 1 --2- 172.21.15.88:0/3489837796 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f195415c0f0 0x7f195406e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.131+0000 7f195bc80640 1 --2- 172.21.15.88:0/3489837796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f195406f630 0x7f195415a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.131+0000 7f195bc80640 1 --2- 172.21.15.88:0/3489837796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f195406bc10 0x7f19541520c0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.131+0000 7f195bc80640 1 -- 172.21.15.88:0/3489837796 >> 172.21.15.88:0/3489837796 conn(0x7f19540d1390 msgr2=0x7f1954072300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:19.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.131+0000 7f195bc80640 1 -- 172.21.15.88:0/3489837796 shutdown_connections 2024-09-16T01:39:19.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.131+0000 7f195bc80640 1 -- 172.21.15.88:0/3489837796 wait complete. 2024-09-16T01:39:19.144 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:39:19.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.241+0000 7f2730ce4640 1 Processor -- start 2024-09-16T01:39:19.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.241+0000 7f2730ce4640 1 -- start start 2024-09-16T01:39:19.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.242+0000 7f2730ce4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f272c15e950 0x7f272c06ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:19.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.242+0000 7f2730ce4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f272c0735b0 0x7f272c06d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:19.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.242+0000 7f2730ce4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f272c06faf0 con 0x7f272c072050 2024-09-16T01:39:19.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.242+0000 7f2730ce4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f272c06fdd0 con 0x7f272c15e950 2024-09-16T01:39:19.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.242+0000 7f2730ce4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f272c0700b0 con 0x7f272c0735b0 2024-09-16T01:39:19.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.242+0000 7f272a59c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f272c15e950 0x7f272c06ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:19.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.242+0000 7f272ad9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f272c072050 0x7f272c06bc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42086/0 (socket says 172.21.15.88:42086) 2024-09-16T01:39:19.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.242+0000 7f272ad9d640 1 -- 172.21.15.88:0/160842321 learned_addr learned my addr 172.21.15.88:0/160842321 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:19.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.242+0000 7f2729d9b640 1 --2- 172.21.15.88:0/160842321 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f272c0735b0 0x7f272c06d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:19.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.242+0000 7f272a59c640 1 -- 172.21.15.88:0/160842321 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f272c0735b0 msgr2=0x7f272c06d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:19.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.242+0000 7f272a59c640 1 --2- 172.21.15.88:0/160842321 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f272c0735b0 0x7f272c06d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.242+0000 7f272a59c640 1 -- 172.21.15.88:0/160842321 >> v1:172.21.15.88:6789/0 conn(0x7f272c072050 legacy=0x7f272c06bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:19.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.242+0000 7f272959a640 1 -- 172.21.15.88:0/160842321 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1361166010 0 0) 0x7f272c06faf0 con 0x7f272c072050 2024-09-16T01:39:19.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.243+0000 7f272a59c640 1 -- 172.21.15.88:0/160842321 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f272c070390 con 0x7f272c15e950 2024-09-16T01:39:19.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.243+0000 7f272a59c640 1 --2- 172.21.15.88:0/160842321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f272c15e950 0x7f272c06ce30 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f271c009d90 tx=0x7f271c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c86eb8bc9891962 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:19.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.243+0000 7f272959a640 1 -- 172.21.15.88:0/160842321 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f271c007740 con 0x7f272c15e950 2024-09-16T01:39:19.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.243+0000 7f272959a640 1 -- 172.21.15.88:0/160842321 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f271c0078e0 con 0x7f272c15e950 2024-09-16T01:39:19.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.243+0000 7f272959a640 1 -- 172.21.15.88:0/160842321 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f271c007cb0 con 0x7f272c15e950 2024-09-16T01:39:19.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.244+0000 7f2730ce4640 1 -- 172.21.15.88:0/160842321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f272c15e950 msgr2=0x7f272c06ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:19.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.244+0000 7f2730ce4640 1 --2- 172.21.15.88:0/160842321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f272c15e950 0x7f272c06ce30 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f271c009d90 tx=0x7f271c02a660 comp rx=0 tx=0).stop 2024-09-16T01:39:19.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.244+0000 7f2730ce4640 1 -- 172.21.15.88:0/160842321 shutdown_connections 2024-09-16T01:39:19.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.244+0000 7f2730ce4640 1 --2- 172.21.15.88:0/160842321 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f272c0735b0 0x7f272c06d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.244+0000 7f2730ce4640 1 --2- 172.21.15.88:0/160842321 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f272c15e950 0x7f272c06ce30 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.244+0000 7f2730ce4640 1 -- 172.21.15.88:0/160842321 >> 172.21.15.88:0/160842321 conn(0x7f272c0d14d0 msgr2=0x7f272c0d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:19.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.244+0000 7f2730ce4640 1 -- 172.21.15.88:0/160842321 shutdown_connections 2024-09-16T01:39:19.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.244+0000 7f2730ce4640 1 -- 172.21.15.88:0/160842321 wait complete. 2024-09-16T01:39:19.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.245+0000 7f2730ce4640 1 Processor -- start 2024-09-16T01:39:19.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.245+0000 7f2730ce4640 1 -- start start 2024-09-16T01:39:19.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.246+0000 7f2730ce4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f272c072050 0x7f272c1596b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:19.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.246+0000 7f2730ce4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f272c0735b0 0x7f272c1570c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:19.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.246+0000 7f2730ce4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f272c15e950 0x7f272c1615d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:19.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.246+0000 7f2730ce4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f272c0d0230 con 0x7f272c15e950 2024-09-16T01:39:19.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.246+0000 7f2730ce4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f272c0d00b0 con 0x7f272c072050 2024-09-16T01:39:19.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.246+0000 7f2730ce4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f272c0d03b0 con 0x7f272c0735b0 2024-09-16T01:39:19.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.246+0000 7f272ad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f272c15e950 0x7f272c1615d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:19.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.246+0000 7f272ad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f272c15e950 0x7f272c1615d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37800/0 (socket says 172.21.15.88:37800) 2024-09-16T01:39:19.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.246+0000 7f272ad9d640 1 -- 172.21.15.88:0/1894987339 learned_addr learned my addr 172.21.15.88:0/1894987339 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:19.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.246+0000 7f272a59c640 1 --2- 172.21.15.88:0/1894987339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f272c072050 0x7f272c1596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:19.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.246+0000 7f272ad9d640 1 -- 172.21.15.88:0/1894987339 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f272c0735b0 msgr2=0x7f272c1570c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:19.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.246+0000 7f2729d9b640 1 --2- 172.21.15.88:0/1894987339 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f272c0735b0 0x7f272c1570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:19.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.246+0000 7f272ad9d640 1 --2- 172.21.15.88:0/1894987339 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f272c0735b0 0x7f272c1570c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.246+0000 7f272ad9d640 1 -- 172.21.15.88:0/1894987339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f272c072050 msgr2=0x7f272c1596b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:19.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.246+0000 7f272ad9d640 1 --2- 172.21.15.88:0/1894987339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f272c072050 0x7f272c1596b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.246+0000 7f272ad9d640 1 -- 172.21.15.88:0/1894987339 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f272c158610 con 0x7f272c15e950 2024-09-16T01:39:19.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.246+0000 7f272a59c640 1 --2- 172.21.15.88:0/1894987339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f272c072050 0x7f272c1596b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:19.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.246+0000 7f2729d9b640 1 --2- 172.21.15.88:0/1894987339 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f272c0735b0 0x7f272c1570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:19.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.247+0000 7f272ad9d640 1 --2- 172.21.15.88:0/1894987339 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f272c15e950 0x7f272c1615d0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f272400c3e0 tx=0x7f272400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:19.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.247+0000 7f271affd640 1 -- 172.21.15.88:0/1894987339 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27240074e0 con 0x7f272c15e950 2024-09-16T01:39:19.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.247+0000 7f271affd640 1 -- 172.21.15.88:0/1894987339 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27240047e0 con 0x7f272c15e950 2024-09-16T01:39:19.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.247+0000 7f2730ce4640 1 -- 172.21.15.88:0/1894987339 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f272c1588f0 con 0x7f272c15e950 2024-09-16T01:39:19.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.247+0000 7f271affd640 1 -- 172.21.15.88:0/1894987339 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2724004bd0 con 0x7f272c15e950 2024-09-16T01:39:19.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.247+0000 7f2730ce4640 1 -- 172.21.15.88:0/1894987339 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f272c158eb0 con 0x7f272c15e950 2024-09-16T01:39:19.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.248+0000 7f2730ce4640 1 -- 172.21.15.88:0/1894987339 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f272c06ce30 con 0x7f272c15e950 2024-09-16T01:39:19.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.248+0000 7f271affd640 1 -- 172.21.15.88:0/1894987339 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2724011070 con 0x7f272c15e950 2024-09-16T01:39:19.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.249+0000 7f271affd640 1 --2- 172.21.15.88:0/1894987339 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f270c06c2b0 0x7f270c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:19.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.249+0000 7f271affd640 1 -- 172.21.15.88:0/1894987339 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f272408afa0 con 0x7f272c15e950 2024-09-16T01:39:19.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.249+0000 7f272a59c640 1 --2- 172.21.15.88:0/1894987339 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f270c06c2b0 0x7f270c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:19.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.249+0000 7f272a59c640 1 --2- 172.21.15.88:0/1894987339 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f270c06c2b0 0x7f270c06e770 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f271c034000 tx=0x7f271c02f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:19.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.253+0000 7f271affd640 1 -- 172.21.15.88:0/1894987339 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f27240597a0 con 0x7f272c15e950 2024-09-16T01:39:19.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.453+0000 7f2730ce4640 1 -- 172.21.15.88:0/1894987339 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f272c15bd10 con 0x7f272c15e950 2024-09-16T01:39:19.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.453+0000 7f271affd640 1 -- 172.21.15.88:0/1894987339 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+5231 (secure 0 0 0) 0x7f27240590a0 con 0x7f272c15e950 2024-09-16T01:39:19.455 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:19.455 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":57,"btime":"2024-09-16T01:39:17:483517+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5725,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2696547723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2696547723},{"type":"v1","addr":"172.21.15.154:6834","nonce":2696547723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54},{"gid":6411,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3276199399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3276199399},{"type":"v1","addr":"172.21.15.88:6836","nonce":3276199399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54},{"gid":6414,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2944989249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2944989249},{"type":"v1","addr":"172.21.15.88:6837","nonce":2944989249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:39:14.808422+0000","modified":"2024-09-16T01:39:16.811457+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6429":{"gid":6429,"name":"d","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6835/3482042494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":3482042494},{"type":"v1","addr":"172.21.15.154:6835","nonce":3482042494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6429,"qdb_cluster":[6429]},"id":5}]} 2024-09-16T01:39:19.456 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 57 2024-09-16T01:39:19.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.455+0000 7f2730ce4640 1 -- 172.21.15.88:0/1894987339 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f270c06c2b0 msgr2=0x7f270c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:19.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.455+0000 7f2730ce4640 1 --2- 172.21.15.88:0/1894987339 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f270c06c2b0 0x7f270c06e770 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f271c034000 tx=0x7f271c02f000 comp rx=0 tx=0).stop 2024-09-16T01:39:19.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.455+0000 7f2730ce4640 1 -- 172.21.15.88:0/1894987339 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f272c15e950 msgr2=0x7f272c1615d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:19.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.455+0000 7f2730ce4640 1 --2- 172.21.15.88:0/1894987339 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f272c15e950 0x7f272c1615d0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f272400c3e0 tx=0x7f272400c8a0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.456+0000 7f2730ce4640 1 -- 172.21.15.88:0/1894987339 shutdown_connections 2024-09-16T01:39:19.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.456+0000 7f2730ce4640 1 --2- 172.21.15.88:0/1894987339 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f270c06c2b0 0x7f270c06e770 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.456+0000 7f2730ce4640 1 --2- 172.21.15.88:0/1894987339 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f272c15e950 0x7f272c1615d0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.456+0000 7f2730ce4640 1 --2- 172.21.15.88:0/1894987339 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f272c0735b0 0x7f272c1570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.456+0000 7f2730ce4640 1 --2- 172.21.15.88:0/1894987339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f272c072050 0x7f272c1596b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.456+0000 7f2730ce4640 1 -- 172.21.15.88:0/1894987339 >> 172.21.15.88:0/1894987339 conn(0x7f272c0d14d0 msgr2=0x7f272c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:19.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.456+0000 7f2730ce4640 1 -- 172.21.15.88:0/1894987339 shutdown_connections 2024-09-16T01:39:19.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.456+0000 7f2730ce4640 1 -- 172.21.15.88:0/1894987339 wait complete. 2024-09-16T01:39:19.871 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:39:14.808422+0000', 'modified': '2024-09-16T01:39:16.811457+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6429}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6429': {'gid': 6429, 'name': 'd', 'rank': 0, 'incarnation': 55, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.154:6835/3482042494', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6833', 'nonce': 3482042494}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 3482042494}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6429, 'qdb_cluster': [6429]} 2024-09-16T01:39:19.871 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T01:39:19.871 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-09-16T01:39:19.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.969+0000 7f411e511640 1 Processor -- start 2024-09-16T01:39:19.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.969+0000 7f411e511640 1 -- start start 2024-09-16T01:39:19.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.970+0000 7f411e511640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f41180cf1f0 0x7f41180b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:19.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.970+0000 7f411e511640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41180cf860 con 0x7f411815e7a0 2024-09-16T01:39:19.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.970+0000 7f411e511640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41180cfb40 con 0x7f41180b6720 2024-09-16T01:39:19.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.970+0000 7f411e511640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f41180b6b90 con 0x7f41180cf1f0 2024-09-16T01:39:19.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.970+0000 7f41177fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f41180cf1f0 0x7f41180b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:19.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.970+0000 7f41177fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f41180cf1f0 0x7f41180b61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37812/0 (socket says 172.21.15.88:37812) 2024-09-16T01:39:19.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.970+0000 7f4117fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f411815e7a0 0x7f411806caa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42092/0 (socket says 172.21.15.88:42092) 2024-09-16T01:39:19.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.970+0000 7f41177fe640 1 -- 172.21.15.88:0/822754107 learned_addr learned my addr 172.21.15.88:0/822754107 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:19.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.971+0000 7f41177fe640 1 -- 172.21.15.88:0/822754107 >> v1:172.21.15.154:6789/0 conn(0x7f41180b6720 legacy=0x7f41180d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:19.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.971+0000 7f41177fe640 1 -- 172.21.15.88:0/822754107 >> v1:172.21.15.88:6789/0 conn(0x7f411815e7a0 legacy=0x7f411806caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:19.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.971+0000 7f41177fe640 1 -- 172.21.15.88:0/822754107 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41180cfe20 con 0x7f41180cf1f0 2024-09-16T01:39:19.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.971+0000 7f41177fe640 1 --2- 172.21.15.88:0/822754107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41180cf1f0 0x7f41180b61e0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f410c009870 tx=0x7f410c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=661da1542e204f45 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:19.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.971+0000 7f4116ffd640 1 -- 172.21.15.88:0/822754107 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410c037070 con 0x7f41180cf1f0 2024-09-16T01:39:19.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.971+0000 7f4116ffd640 1 -- 172.21.15.88:0/822754107 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f410c032440 con 0x7f41180cf1f0 2024-09-16T01:39:19.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.971+0000 7f4116ffd640 1 -- 172.21.15.88:0/822754107 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410c032810 con 0x7f41180cf1f0 2024-09-16T01:39:19.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.972+0000 7f411e511640 1 -- 172.21.15.88:0/822754107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41180cf1f0 msgr2=0x7f41180b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:19.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.972+0000 7f411e511640 1 --2- 172.21.15.88:0/822754107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41180cf1f0 0x7f41180b61e0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f410c009870 tx=0x7f410c02a210 comp rx=0 tx=0).stop 2024-09-16T01:39:19.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.972+0000 7f411e511640 1 -- 172.21.15.88:0/822754107 shutdown_connections 2024-09-16T01:39:19.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.972+0000 7f411e511640 1 --2- 172.21.15.88:0/822754107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41180cf1f0 0x7f41180b61e0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.972+0000 7f411e511640 1 -- 172.21.15.88:0/822754107 >> 172.21.15.88:0/822754107 conn(0x7f41180d11e0 msgr2=0x7f41180d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:19.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.972+0000 7f411e511640 1 -- 172.21.15.88:0/822754107 shutdown_connections 2024-09-16T01:39:19.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.972+0000 7f411e511640 1 -- 172.21.15.88:0/822754107 wait complete. 2024-09-16T01:39:19.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.973+0000 7f411e511640 1 Processor -- start 2024-09-16T01:39:19.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.973+0000 7f411e511640 1 -- start start 2024-09-16T01:39:19.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.974+0000 7f411e511640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41180b6720 0x7f4118149bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:19.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.974+0000 7f411e511640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41180cf1f0 0x7f41180cc750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:19.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.974+0000 7f411e511640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f411815e7a0 0x7f41180d3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:19.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.974+0000 7f411e511640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f41180b7a00 con 0x7f41180b6720 2024-09-16T01:39:19.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.974+0000 7f411e511640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f41180d2ca0 con 0x7f41180cf1f0 2024-09-16T01:39:19.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.974+0000 7f411e511640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f411815eb80 con 0x7f411815e7a0 2024-09-16T01:39:19.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.974+0000 7f4117fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41180b6720 0x7f4118149bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:19.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.974+0000 7f4117fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41180b6720 0x7f4118149bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37820/0 (socket says 172.21.15.88:37820) 2024-09-16T01:39:19.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.974+0000 7f4117fff640 1 -- 172.21.15.88:0/4137203735 learned_addr learned my addr 172.21.15.88:0/4137203735 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:19.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.974+0000 7f4117fff640 1 -- 172.21.15.88:0/4137203735 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f411815e7a0 msgr2=0x7f41180d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:19.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.974+0000 7f4117fff640 1 --2- 172.21.15.88:0/4137203735 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f411815e7a0 0x7f41180d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.974+0000 7f4117fff640 1 -- 172.21.15.88:0/4137203735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41180cf1f0 msgr2=0x7f41180cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:19.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.974+0000 7f4117fff640 1 --2- 172.21.15.88:0/4137203735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41180cf1f0 0x7f41180cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.974+0000 7f4117fff640 1 -- 172.21.15.88:0/4137203735 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41181f4cd0 con 0x7f41180b6720 2024-09-16T01:39:19.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.975+0000 7f4117fff640 1 --2- 172.21.15.88:0/4137203735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41180b6720 0x7f4118149bc0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f410800c400 tx=0x7f410800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:19.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.975+0000 7f4114ff9640 1 -- 172.21.15.88:0/4137203735 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41080074e0 con 0x7f41180b6720 2024-09-16T01:39:19.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.975+0000 7f4114ff9640 1 -- 172.21.15.88:0/4137203735 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4108007680 con 0x7f41180b6720 2024-09-16T01:39:19.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.975+0000 7f411e511640 1 -- 172.21.15.88:0/4137203735 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41181f4fb0 con 0x7f41180b6720 2024-09-16T01:39:19.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.975+0000 7f4114ff9640 1 -- 172.21.15.88:0/4137203735 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4108004a00 con 0x7f41180b6720 2024-09-16T01:39:19.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.975+0000 7f411e511640 1 -- 172.21.15.88:0/4137203735 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41181f5570 con 0x7f41180b6720 2024-09-16T01:39:19.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.976+0000 7f4114ff9640 1 -- 172.21.15.88:0/4137203735 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f410801a020 con 0x7f41180b6720 2024-09-16T01:39:19.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.977+0000 7f4114ff9640 1 --2- 172.21.15.88:0/4137203735 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f410406c2b0 0x7f410406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:19.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.977+0000 7f4114ff9640 1 -- 172.21.15.88:0/4137203735 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f410808ace0 con 0x7f41180b6720 2024-09-16T01:39:19.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.977+0000 7f41177fe640 1 --2- 172.21.15.88:0/4137203735 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f410406c2b0 0x7f410406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:19.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.977+0000 7f41177fe640 1 --2- 172.21.15.88:0/4137203735 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f410406c2b0 0x7f410406e770 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f410c034000 tx=0x7f410c02c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:19.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.977+0000 7f41212aa740 1 Processor -- start 2024-09-16T01:39:19.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.978+0000 7f41212aa740 1 -- start start 2024-09-16T01:39:19.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.978+0000 7f41212aa740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564f185202c0 0x564f18565a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:19.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.978+0000 7f41212aa740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f1851da80 0x564f185206a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:19.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.978+0000 7f41212aa740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f18521a50 0x564f1851a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:19.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.978+0000 7f41212aa740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564f18482520 con 0x564f1851da80 2024-09-16T01:39:19.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.978+0000 7f41212aa740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564f1842a3e0 con 0x564f18521a50 2024-09-16T01:39:19.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.978+0000 7f41212aa740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x564f1847e4e0 con 0x564f185202c0 2024-09-16T01:39:19.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.978+0000 7f41177fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f1851da80 0x564f185206a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:19.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.978+0000 7f41177fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f1851da80 0x564f185206a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37832/0 (socket says 172.21.15.88:37832) 2024-09-16T01:39:19.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.978+0000 7f41177fe640 1 -- 172.21.15.88:0/4138478098 learned_addr learned my addr 172.21.15.88:0/4138478098 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:19.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.979+0000 7f411caae640 1 --2- 172.21.15.88:0/4138478098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f18521a50 0x564f1851a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:19.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.979+0000 7f4117fff640 1 --2- 172.21.15.88:0/4138478098 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564f185202c0 0x564f18565a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:19.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.979+0000 7f41177fe640 1 -- 172.21.15.88:0/4138478098 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564f185202c0 msgr2=0x564f18565a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:19.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.979+0000 7f41177fe640 1 --2- 172.21.15.88:0/4138478098 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564f185202c0 0x564f18565a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.979+0000 7f41177fe640 1 -- 172.21.15.88:0/4138478098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f18521a50 msgr2=0x564f1851a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:19.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.979+0000 7f41177fe640 1 --2- 172.21.15.88:0/4138478098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f18521a50 0x564f1851a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.979+0000 7f41177fe640 1 -- 172.21.15.88:0/4138478098 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f184452f0 con 0x564f1851da80 2024-09-16T01:39:19.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.979+0000 7f411caae640 1 --2- 172.21.15.88:0/4138478098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f18521a50 0x564f1851a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:19.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.979+0000 7f41177fe640 1 --2- 172.21.15.88:0/4138478098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f1851da80 0x564f185206a0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f410c009960 tx=0x7f410c008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96eca7c7ba463e2b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:19.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.979+0000 7f40fdffb640 1 -- 172.21.15.88:0/4138478098 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410c04b070 con 0x564f1851da80 2024-09-16T01:39:19.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.979+0000 7f40fdffb640 1 -- 172.21.15.88:0/4138478098 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f410c032cb0 con 0x564f1851da80 2024-09-16T01:39:19.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.979+0000 7f40fdffb640 1 -- 172.21.15.88:0/4138478098 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410c037040 con 0x564f1851da80 2024-09-16T01:39:19.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.979+0000 7f41212aa740 1 -- 172.21.15.88:0/4138478098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f1851da80 msgr2=0x564f185206a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:19.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.979+0000 7f41212aa740 1 --2- 172.21.15.88:0/4138478098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f1851da80 0x564f185206a0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f410c009960 tx=0x7f410c008030 comp rx=0 tx=0).stop 2024-09-16T01:39:19.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.980+0000 7f41212aa740 1 -- 172.21.15.88:0/4138478098 shutdown_connections 2024-09-16T01:39:19.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.980+0000 7f41212aa740 1 --2- 172.21.15.88:0/4138478098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f18521a50 0x564f1851a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.980+0000 7f41212aa740 1 --2- 172.21.15.88:0/4138478098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f1851da80 0x564f185206a0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.980+0000 7f41212aa740 1 --2- 172.21.15.88:0/4138478098 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564f185202c0 0x564f18565a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.980+0000 7f41212aa740 1 -- 172.21.15.88:0/4138478098 >> 172.21.15.88:0/4138478098 conn(0x564f1851ca80 msgr2=0x564f18564f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:19.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.980+0000 7f41212aa740 1 -- 172.21.15.88:0/4138478098 shutdown_connections 2024-09-16T01:39:19.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.980+0000 7f41212aa740 1 -- 172.21.15.88:0/4138478098 wait complete. 2024-09-16T01:39:19.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.980+0000 7f41212aa740 1 -- start start 2024-09-16T01:39:19.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.980+0000 7f41212aa740 1 Processor -- start 2024-09-16T01:39:19.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.981+0000 7f41212aa740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f185202c0 0x564f1851af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:19.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.981+0000 7f41212aa740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f18521a50 0x564f18578f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:19.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.981+0000 7f41212aa740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564f1856b470 0x564f1854a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:19.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.981+0000 7f41212aa740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564f1851bf20 con 0x564f18521a50 2024-09-16T01:39:19.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.981+0000 7f41212aa740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564f1840b450 con 0x564f185202c0 2024-09-16T01:39:19.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.981+0000 7f41212aa740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x564f18566290 con 0x564f1856b470 2024-09-16T01:39:19.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.981+0000 7f41212aa740 10 client.0 _ll_register_callbacks cb 0x564f18194ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:39:19.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.981+0000 7f41177fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f18521a50 0x564f18578f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:19.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.981+0000 7f41177fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f18521a50 0x564f18578f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37846/0 (socket says 172.21.15.88:37846) 2024-09-16T01:39:19.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.981+0000 7f41177fe640 1 -- 172.21.15.88:0/1318462642 learned_addr learned my addr 172.21.15.88:0/1318462642 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:19.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.981+0000 7f4117fff640 1 --2- 172.21.15.88:0/1318462642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f185202c0 0x564f1851af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:19.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.981+0000 7f411caae640 1 --2- 172.21.15.88:0/1318462642 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564f1856b470 0x564f1854a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:19.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.981+0000 7f41177fe640 1 -- 172.21.15.88:0/1318462642 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564f1856b470 msgr2=0x564f1854a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:19.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.981+0000 7f41177fe640 1 --2- 172.21.15.88:0/1318462642 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564f1856b470 0x564f1854a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.981+0000 7f41177fe640 1 -- 172.21.15.88:0/1318462642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f185202c0 msgr2=0x564f1851af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:19.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.981+0000 7f41177fe640 1 --2- 172.21.15.88:0/1318462642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f185202c0 0x564f1851af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.982+0000 7f41177fe640 1 -- 172.21.15.88:0/1318462642 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f181952d0 con 0x564f18521a50 2024-09-16T01:39:19.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.982+0000 7f41177fe640 1 --2- 172.21.15.88:0/1318462642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f18521a50 0x564f18578f10 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f410c009990 tx=0x7f410c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1973bbf594a457e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:19.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.982+0000 7f40f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:39:19.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.982+0000 7f40f6ffd640 1 -- 172.21.15.88:0/1318462642 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410c056060 con 0x564f18521a50 2024-09-16T01:39:19.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.982+0000 7f41212aa740 1 -- 172.21.15.88:0/1318462642 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f410c032cb0 con 0x564f18521a50 2024-09-16T01:39:19.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.982+0000 7f40f6ffd640 1 -- 172.21.15.88:0/1318462642 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f410c004030 con 0x564f18521a50 2024-09-16T01:39:19.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.982+0000 7f40f6ffd640 1 -- 172.21.15.88:0/1318462642 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f410c05c050 con 0x564f18521a50 2024-09-16T01:39:19.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.982+0000 7f41212aa740 10 client.6546 fetch_fsmap learned FSMap version 57 2024-09-16T01:39:19.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.982+0000 7f41212aa740 1 -- 172.21.15.88:0/1318462642 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) -- 0x564f1825f6b0 con 0x564f18521a50 2024-09-16T01:39:19.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.982+0000 7f40f6ffd640 1 -- 172.21.15.88:0/1318462642 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f410c032cb0 con 0x564f18521a50 2024-09-16T01:39:19.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.982+0000 7f40f6ffd640 1 -- 172.21.15.88:0/1318462642 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 57) ==== 3502+0+0 (secure 0 0 0) 0x7f410c05f020 con 0x564f18521a50 2024-09-16T01:39:19.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.982+0000 7f41212aa740 10 client.6546 fetch_fsmap finished waiting for FSMap version 57 2024-09-16T01:39:19.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.982+0000 7f41212aa740 10 client.6546 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-16T01:39:19.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.982+0000 7f41212aa740 20 client.6546 populate_metadata read hostname 'smithi088' 2024-09-16T01:39:19.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.982+0000 7f41212aa740 1 --2- 172.21.15.88:0/1318462642 >> [v2:172.21.15.154:6833/3482042494,v1:172.21.15.154:6835/3482042494] conn(0x564f18547dc0 0x564f185481a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:19.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.982+0000 7f41212aa740 4 client.6546 mds_command: new command op to 6429 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:39:19.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.982+0000 7f41212aa740 1 -- 172.21.15.88:0/1318462642 --> [v2:172.21.15.154:6833/3482042494,v1:172.21.15.154:6835/3482042494] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f1848fc50 con 0x564f18547dc0 2024-09-16T01:39:19.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.983+0000 7f4117fff640 1 --2- 172.21.15.88:0/1318462642 >> [v2:172.21.15.154:6833/3482042494,v1:172.21.15.154:6835/3482042494] conn(0x564f18547dc0 0x564f185481a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:19.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.983+0000 7f4117fff640 1 --2- 172.21.15.88:0/1318462642 >> [v2:172.21.15.154:6833/3482042494,v1:172.21.15.154:6835/3482042494] conn(0x564f18547dc0 0x564f185481a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d272b8f189d75f3 server_cookie=216c2ba403c8dc in_seq=0 out_seq=0 2024-09-16T01:39:19.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.983+0000 7f40f6ffd640 10 client.6546 ms_handle_connect on v2:172.21.15.154:6833/3482042494 2024-09-16T01:39:19.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.984+0000 7f40f6ffd640 1 -- 172.21.15.88:0/1318462642 <== mds.0 v2:172.21.15.154:6833/3482042494 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564f1848fc50 con 0x564f18547dc0 2024-09-16T01:39:19.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.984+0000 7f40f6ffd640 10 client.6546 handle_command_reply: tid=0 2024-09-16T01:39:19.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.984+0000 7f41212aa740 1 client.6546 shutdown 2024-09-16T01:39:19.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.984+0000 7f41212aa740 1 -- 172.21.15.88:0/1318462642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f18521a50 msgr2=0x564f18578f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:19.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.984+0000 7f41212aa740 1 --2- 172.21.15.88:0/1318462642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f18521a50 0x564f18578f10 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f410c009990 tx=0x7f410c004730 comp rx=0 tx=0).stop 2024-09-16T01:39:19.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.984+0000 7f41212aa740 1 -- 172.21.15.88:0/1318462642 shutdown_connections 2024-09-16T01:39:19.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.984+0000 7f41212aa740 1 --2- 172.21.15.88:0/1318462642 >> [v2:172.21.15.154:6833/3482042494,v1:172.21.15.154:6835/3482042494] conn(0x564f18547dc0 0x564f185481a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.984+0000 7f41212aa740 1 --2- 172.21.15.88:0/1318462642 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564f1856b470 0x564f1854a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.984+0000 7f40f6ffd640 0 client.6546 ms_handle_reset on v2:172.21.15.154:6833/3482042494 2024-09-16T01:39:19.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.984+0000 7f41212aa740 1 --2- 172.21.15.88:0/1318462642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f18521a50 0x564f18578f10 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.984+0000 7f41212aa740 1 --2- 172.21.15.88:0/1318462642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f185202c0 0x564f1851af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:19.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.984+0000 7f41212aa740 1 -- 172.21.15.88:0/1318462642 >> 172.21.15.88:0/1318462642 conn(0x564f1851da80 msgr2=0x564f18564570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:19.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.984+0000 7f41212aa740 1 -- 172.21.15.88:0/1318462642 shutdown_connections 2024-09-16T01:39:19.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.984+0000 7f41212aa740 1 -- 172.21.15.88:0/1318462642 wait complete. 2024-09-16T01:39:19.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:19.984+0000 7f41212aa740 20 client.6546 trim_cache size 0 max 16384 2024-09-16T01:39:20.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.002+0000 7f41212aa740 1 Processor -- start 2024-09-16T01:39:20.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.002+0000 7f41212aa740 1 -- start start 2024-09-16T01:39:20.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.002+0000 7f41212aa740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f185202c0 0x564f18548780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:20.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.002+0000 7f41212aa740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564f18521a50 0x564f18564000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:20.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.002+0000 7f41212aa740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f1856b470 0x564f18564540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:20.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.002+0000 7f41212aa740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f41181ffe10 con 0x564f185202c0 2024-09-16T01:39:20.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.002+0000 7f41212aa740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f41181ff1f0 con 0x564f1856b470 2024-09-16T01:39:20.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.002+0000 7f41212aa740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f41181ff7c0 con 0x564f18521a50 2024-09-16T01:39:20.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.002+0000 7f4117fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f185202c0 0x564f18548780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:20.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.002+0000 7f4117fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f185202c0 0x564f18548780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37862/0 (socket says 172.21.15.88:37862) 2024-09-16T01:39:20.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.002+0000 7f4117fff640 1 -- 172.21.15.88:0/212313557 learned_addr learned my addr 172.21.15.88:0/212313557 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:20.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.003+0000 7f4117fff640 1 -- 172.21.15.88:0/212313557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564f18521a50 msgr2=0x564f18564000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:20.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.003+0000 7f4117fff640 1 --2- 172.21.15.88:0/212313557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564f18521a50 0x564f18564000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.003+0000 7f4117fff640 1 -- 172.21.15.88:0/212313557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f1856b470 msgr2=0x564f18564540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:20.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.003+0000 7f4117fff640 1 --2- 172.21.15.88:0/212313557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f1856b470 0x564f18564540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.003+0000 7f4117fff640 1 -- 172.21.15.88:0/212313557 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f18549db0 con 0x564f185202c0 2024-09-16T01:39:20.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.003+0000 7f4117fff640 1 --2- 172.21.15.88:0/212313557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f185202c0 0x564f18548780 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f4108001eb0 tx=0x7f4108015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0f5a5055a7646cc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:20.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.003+0000 7f40fd7fa640 1 -- 172.21.15.88:0/212313557 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4108057050 con 0x564f185202c0 2024-09-16T01:39:20.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.003+0000 7f40fd7fa640 1 -- 172.21.15.88:0/212313557 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41080553c0 con 0x564f185202c0 2024-09-16T01:39:20.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.003+0000 7f40fd7fa640 1 -- 172.21.15.88:0/212313557 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41080591f0 con 0x564f185202c0 2024-09-16T01:39:20.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.003+0000 7f41212aa740 1 -- 172.21.15.88:0/212313557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f185202c0 msgr2=0x564f18548780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:20.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.003+0000 7f41212aa740 1 --2- 172.21.15.88:0/212313557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f185202c0 0x564f18548780 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f4108001eb0 tx=0x7f4108015040 comp rx=0 tx=0).stop 2024-09-16T01:39:20.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.003+0000 7f41212aa740 1 -- 172.21.15.88:0/212313557 shutdown_connections 2024-09-16T01:39:20.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.003+0000 7f41212aa740 1 --2- 172.21.15.88:0/212313557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f1856b470 0x564f18564540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.003+0000 7f41212aa740 1 --2- 172.21.15.88:0/212313557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564f18521a50 0x564f18564000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.003+0000 7f41212aa740 1 --2- 172.21.15.88:0/212313557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f185202c0 0x564f18548780 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.003+0000 7f41212aa740 1 -- 172.21.15.88:0/212313557 >> 172.21.15.88:0/212313557 conn(0x564f1851da80 msgr2=0x564f185698c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:20.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.004+0000 7f41212aa740 1 -- 172.21.15.88:0/212313557 shutdown_connections 2024-09-16T01:39:20.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.004+0000 7f41212aa740 1 -- 172.21.15.88:0/212313557 wait complete. 2024-09-16T01:39:20.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.004+0000 7f41212aa740 1 -- start start 2024-09-16T01:39:20.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.004+0000 7f41212aa740 1 Processor -- start 2024-09-16T01:39:20.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.005+0000 7f41212aa740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564f185202c0 0x564f185e39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:20.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.005+0000 7f41212aa740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f18521a50 0x564f18564000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:20.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.005+0000 7f41212aa740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f1856b470 0x564f18564540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:20.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.005+0000 7f41212aa740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564f1856bd90 con 0x564f18521a50 2024-09-16T01:39:20.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.005+0000 7f41212aa740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564f1856ac70 con 0x564f1856b470 2024-09-16T01:39:20.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.005+0000 7f41212aa740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x564f18519a40 con 0x564f185202c0 2024-09-16T01:39:20.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.005+0000 7f41212aa740 10 client.0 _ll_register_callbacks cb 0x7f410c049070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:39:20.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.005+0000 7f41177fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f18521a50 0x564f18564000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:20.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.005+0000 7f41177fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f18521a50 0x564f18564000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37876/0 (socket says 172.21.15.88:37876) 2024-09-16T01:39:20.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.005+0000 7f41177fe640 1 -- 172.21.15.88:0/1906435550 learned_addr learned my addr 172.21.15.88:0/1906435550 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:20.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.005+0000 7f4117fff640 1 --2- 172.21.15.88:0/1906435550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564f185202c0 0x564f185e39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:20.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.005+0000 7f411caae640 1 --2- 172.21.15.88:0/1906435550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f1856b470 0x564f18564540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:20.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.005+0000 7f41177fe640 1 -- 172.21.15.88:0/1906435550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564f185202c0 msgr2=0x564f185e39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.005+0000 7f41177fe640 1 --2- 172.21.15.88:0/1906435550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564f185202c0 0x564f185e39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.005+0000 7f41177fe640 1 -- 172.21.15.88:0/1906435550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f1856b470 msgr2=0x564f18564540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.005+0000 7f41177fe640 1 --2- 172.21.15.88:0/1906435550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f1856b470 0x564f18564540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.005+0000 7f41177fe640 1 -- 172.21.15.88:0/1906435550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f185e3f20 con 0x564f18521a50 2024-09-16T01:39:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.005+0000 7f41177fe640 1 --2- 172.21.15.88:0/1906435550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f18521a50 0x564f18564000 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f410c054040 tx=0x7f410c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad88f85140abf99b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.005+0000 7f40f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:39:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.006+0000 7f40f6ffd640 1 -- 172.21.15.88:0/1906435550 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410c04b070 con 0x564f18521a50 2024-09-16T01:39:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.006+0000 7f40f6ffd640 1 -- 172.21.15.88:0/1906435550 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f410c051ac0 con 0x564f18521a50 2024-09-16T01:39:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.006+0000 7f41212aa740 1 -- 172.21.15.88:0/1906435550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41080553c0 con 0x564f18521a50 2024-09-16T01:39:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.006+0000 7f40f6ffd640 1 -- 172.21.15.88:0/1906435550 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7f410c037040 con 0x564f18521a50 2024-09-16T01:39:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.006+0000 7f41212aa740 10 client.6552 fetch_fsmap learned FSMap version 57 2024-09-16T01:39:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.006+0000 7f41212aa740 1 -- 172.21.15.88:0/1906435550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) -- 0x564f18589380 con 0x564f18521a50 2024-09-16T01:39:20.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.006+0000 7f40f6ffd640 1 -- 172.21.15.88:0/1906435550 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41080553c0 con 0x564f18521a50 2024-09-16T01:39:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.006+0000 7f40f6ffd640 1 -- 172.21.15.88:0/1906435550 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 57) ==== 3502+0+0 (secure 0 0 0) 0x7f410c068050 con 0x564f18521a50 2024-09-16T01:39:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.006+0000 7f41212aa740 10 client.6552 fetch_fsmap finished waiting for FSMap version 57 2024-09-16T01:39:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.006+0000 7f41212aa740 10 client.6552 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-16T01:39:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.006+0000 7f41212aa740 20 client.6552 populate_metadata read hostname 'smithi088' 2024-09-16T01:39:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.006+0000 7f41212aa740 1 --2- 172.21.15.88:0/1906435550 >> [v2:172.21.15.154:6833/3482042494,v1:172.21.15.154:6835/3482042494] conn(0x564f18542e70 0x564f18543250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.006+0000 7f41212aa740 4 client.6552 mds_command: new command op to 6429 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:39:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.006+0000 7f41212aa740 1 -- 172.21.15.88:0/1906435550 --> [v2:172.21.15.154:6833/3482042494,v1:172.21.15.154:6835/3482042494] -- command(tid 0: {"prefix": "status"}) -- 0x564f18489fa0 con 0x564f18542e70 2024-09-16T01:39:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.006+0000 7f4117fff640 1 --2- 172.21.15.88:0/1906435550 >> [v2:172.21.15.154:6833/3482042494,v1:172.21.15.154:6835/3482042494] conn(0x564f18542e70 0x564f18543250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.007+0000 7f40f6ffd640 10 client.6552 ms_handle_connect on v2:172.21.15.154:6833/3482042494 2024-09-16T01:39:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.007+0000 7f4117fff640 1 --2- 172.21.15.88:0/1906435550 >> [v2:172.21.15.154:6833/3482042494,v1:172.21.15.154:6835/3482042494] conn(0x564f18542e70 0x564f18543250 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a9bf0f8fd657d1c server_cookie=a48f825d81343d84 in_seq=0 out_seq=0 2024-09-16T01:39:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.007+0000 7f40f6ffd640 1 -- 172.21.15.88:0/1906435550 <== mds.0 v2:172.21.15.154:6833/3482042494 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x564f18489fa0 con 0x564f18542e70 2024-09-16T01:39:20.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.007+0000 7f40f6ffd640 10 client.6552 handle_command_reply: tid=0 2024-09-16T01:39:20.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.007+0000 7f41212aa740 1 client.6552 shutdown 2024-09-16T01:39:20.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.007+0000 7f41212aa740 1 -- 172.21.15.88:0/1906435550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f18521a50 msgr2=0x564f18564000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:20.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.007+0000 7f41212aa740 1 --2- 172.21.15.88:0/1906435550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f18521a50 0x564f18564000 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f410c054040 tx=0x7f410c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.007+0000 7f41212aa740 1 -- 172.21.15.88:0/1906435550 shutdown_connections 2024-09-16T01:39:20.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.007+0000 7f41212aa740 1 --2- 172.21.15.88:0/1906435550 >> [v2:172.21.15.154:6833/3482042494,v1:172.21.15.154:6835/3482042494] conn(0x564f18542e70 0x564f18543250 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.007+0000 7f41212aa740 1 --2- 172.21.15.88:0/1906435550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f1856b470 0x564f18564540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.007+0000 7f40f6ffd640 0 client.6552 ms_handle_reset on v2:172.21.15.154:6833/3482042494 2024-09-16T01:39:20.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.007+0000 7f41212aa740 1 --2- 172.21.15.88:0/1906435550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564f18521a50 0x564f18564000 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.007+0000 7f41212aa740 1 --2- 172.21.15.88:0/1906435550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564f185202c0 0x564f185e39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.007+0000 7f41212aa740 1 -- 172.21.15.88:0/1906435550 >> 172.21.15.88:0/1906435550 conn(0x564f1851da80 msgr2=0x564f185470e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:20.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.007+0000 7f41212aa740 1 -- 172.21.15.88:0/1906435550 shutdown_connections 2024-09-16T01:39:20.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.007+0000 7f41212aa740 1 -- 172.21.15.88:0/1906435550 wait complete. 2024-09-16T01:39:20.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.007+0000 7f41212aa740 20 client.6552 trim_cache size 0 max 16384 2024-09-16T01:39:20.016 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:39:20.016 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "5ace2c91-46f9-482b-8a89-731fbdc171bf", 2024-09-16T01:39:20.016 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2024-09-16T01:39:20.016 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 6429, 2024-09-16T01:39:20.016 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2024-09-16T01:39:20.016 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2024-09-16T01:39:20.016 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2024-09-16T01:39:20.016 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 4.5286789629999999, 2024-09-16T01:39:20.017 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 57, 2024-09-16T01:39:20.017 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 72, 2024-09-16T01:39:20.017 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 72, 2024-09-16T01:39:20.017 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 8.597400596 2024-09-16T01:39:20.017 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:39:20.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.008+0000 7f411e511640 1 -- 172.21.15.88:0/4137203735 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f410406c2b0 msgr2=0x7f410406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:20.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.008+0000 7f411e511640 1 --2- 172.21.15.88:0/4137203735 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f410406c2b0 0x7f410406e770 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f410c034000 tx=0x7f410c02c000 comp rx=0 tx=0).stop 2024-09-16T01:39:20.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.008+0000 7f411e511640 1 -- 172.21.15.88:0/4137203735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41180b6720 msgr2=0x7f4118149bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:20.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.008+0000 7f411e511640 1 --2- 172.21.15.88:0/4137203735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41180b6720 0x7f4118149bc0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f410800c400 tx=0x7f410800c8c0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.008+0000 7f411e511640 1 -- 172.21.15.88:0/4137203735 shutdown_connections 2024-09-16T01:39:20.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.008+0000 7f411e511640 1 --2- 172.21.15.88:0/4137203735 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f410406c2b0 0x7f410406e770 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.008+0000 7f411e511640 1 --2- 172.21.15.88:0/4137203735 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f411815e7a0 0x7f41180d3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.008+0000 7f411e511640 1 --2- 172.21.15.88:0/4137203735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41180cf1f0 0x7f41180cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.008+0000 7f411e511640 1 --2- 172.21.15.88:0/4137203735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f41180b6720 0x7f4118149bc0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.008+0000 7f411e511640 1 -- 172.21.15.88:0/4137203735 >> 172.21.15.88:0/4137203735 conn(0x7f41180d11e0 msgr2=0x7f41180b5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:20.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.008+0000 7f411e511640 1 -- 172.21.15.88:0/4137203735 shutdown_connections 2024-09-16T01:39:20.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.008+0000 7f411e511640 1 -- 172.21.15.88:0/4137203735 wait complete. 2024-09-16T01:39:20.020 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:39:20.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.117+0000 7fa361b0a640 1 Processor -- start 2024-09-16T01:39:20.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.117+0000 7fa361b0a640 1 -- start start 2024-09-16T01:39:20.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.117+0000 7fa361b0a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa35c073080 0x7fa35c0b5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:20.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.117+0000 7fa361b0a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa35c0b1990 0x7fa35c0d10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:20.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.117+0000 7fa361b0a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa35c071750 con 0x7fa35c15c670 2024-09-16T01:39:20.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.117+0000 7fa361b0a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa35c0d0c60 con 0x7fa35c073080 2024-09-16T01:39:20.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.117+0000 7fa361b0a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa35c0d0370 con 0x7fa35c0b1990 2024-09-16T01:39:20.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.118+0000 7fa35b7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa35c073080 0x7fa35c0b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:20.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.118+0000 7fa35b7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa35c073080 0x7fa35c0b5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37892/0 (socket says 172.21.15.88:37892) 2024-09-16T01:39:20.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.118+0000 7fa353fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa35c15c670 0x7fa35c073460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42106/0 (socket says 172.21.15.88:42106) 2024-09-16T01:39:20.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.118+0000 7fa35b7fe640 1 -- 172.21.15.88:0/2546862449 learned_addr learned my addr 172.21.15.88:0/2546862449 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:20.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.118+0000 7fa35bfff640 1 --2- 172.21.15.88:0/2546862449 >> v2:172.21.15.154:3300/0 conn(0x7fa35c0b1990 0x7fa35c0d10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:20.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.118+0000 7fa35b7fe640 1 -- 172.21.15.88:0/2546862449 >> v2:172.21.15.154:3300/0 conn(0x7fa35c0b1990 msgr2=0x7fa35c0d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:20.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.118+0000 7fa35b7fe640 1 --2- 172.21.15.88:0/2546862449 >> v2:172.21.15.154:3300/0 conn(0x7fa35c0b1990 0x7fa35c0d10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.118+0000 7fa35b7fe640 1 -- 172.21.15.88:0/2546862449 >> v1:172.21.15.88:6789/0 conn(0x7fa35c15c670 legacy=0x7fa35c073460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:20.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.118+0000 7fa35affd640 1 -- 172.21.15.88:0/2546862449 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 615910119 0 0) 0x7fa35c071750 con 0x7fa35c15c670 2024-09-16T01:39:20.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.118+0000 7fa35b7fe640 1 -- 172.21.15.88:0/2546862449 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa35c0b59a0 con 0x7fa35c073080 2024-09-16T01:39:20.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.118+0000 7fa35bfff640 1 --2- 172.21.15.88:0/2546862449 >> v2:172.21.15.154:3300/0 conn(0x7fa35c0b1990 0x7fa35c0d10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:20.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.119+0000 7fa35b7fe640 1 --2- 172.21.15.88:0/2546862449 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa35c073080 0x7fa35c0b5460 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fa35c1610c0 tx=0x7fa34c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3198cbb48668f5d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:20.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.119+0000 7fa35affd640 1 -- 172.21.15.88:0/2546862449 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34c037070 con 0x7fa35c073080 2024-09-16T01:39:20.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.119+0000 7fa35affd640 1 -- 172.21.15.88:0/2546862449 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa34c032440 con 0x7fa35c073080 2024-09-16T01:39:20.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.119+0000 7fa35affd640 1 -- 172.21.15.88:0/2546862449 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34c032810 con 0x7fa35c073080 2024-09-16T01:39:20.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.119+0000 7fa361b0a640 1 -- 172.21.15.88:0/2546862449 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa35c073080 msgr2=0x7fa35c0b5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:20.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.119+0000 7fa361b0a640 1 --2- 172.21.15.88:0/2546862449 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa35c073080 0x7fa35c0b5460 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fa35c1610c0 tx=0x7fa34c02a270 comp rx=0 tx=0).stop 2024-09-16T01:39:20.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.119+0000 7fa361b0a640 1 -- 172.21.15.88:0/2546862449 shutdown_connections 2024-09-16T01:39:20.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.119+0000 7fa361b0a640 1 --2- 172.21.15.88:0/2546862449 >> v2:172.21.15.154:3300/0 conn(0x7fa35c0b1990 0x7fa35c0d10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.119+0000 7fa361b0a640 1 --2- 172.21.15.88:0/2546862449 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa35c073080 0x7fa35c0b5460 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.119+0000 7fa361b0a640 1 -- 172.21.15.88:0/2546862449 >> 172.21.15.88:0/2546862449 conn(0x7fa35c0d24d0 msgr2=0x7fa35c0d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:20.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.119+0000 7fa361b0a640 1 -- 172.21.15.88:0/2546862449 shutdown_connections 2024-09-16T01:39:20.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.119+0000 7fa361b0a640 1 -- 172.21.15.88:0/2546862449 wait complete. 2024-09-16T01:39:20.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.121+0000 7fa361b0a640 1 Processor -- start 2024-09-16T01:39:20.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.121+0000 7fa361b0a640 1 -- start start 2024-09-16T01:39:20.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.121+0000 7fa361b0a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa35c073080 0x7fa35c06e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:20.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.121+0000 7fa361b0a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa35c0b1990 0x7fa35c1510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:20.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.121+0000 7fa361b0a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa35c15c670 0x7fa35c159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:20.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.121+0000 7fa361b0a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa35c0d0610 con 0x7fa35c073080 2024-09-16T01:39:20.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.121+0000 7fa361b0a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa35c071470 con 0x7fa35c0b1990 2024-09-16T01:39:20.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.121+0000 7fa361b0a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa35c0d08f0 con 0x7fa35c15c670 2024-09-16T01:39:20.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.121+0000 7fa35b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa35c073080 0x7fa35c06e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:20.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.121+0000 7fa35b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa35c073080 0x7fa35c06e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37904/0 (socket says 172.21.15.88:37904) 2024-09-16T01:39:20.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.121+0000 7fa35b7fe640 1 -- 172.21.15.88:0/110418976 learned_addr learned my addr 172.21.15.88:0/110418976 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:20.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.122+0000 7fa35b7fe640 1 -- 172.21.15.88:0/110418976 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa35c15c670 msgr2=0x7fa35c159aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:20.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.122+0000 7fa35b7fe640 1 --2- 172.21.15.88:0/110418976 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa35c15c670 0x7fa35c159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.122+0000 7fa35b7fe640 1 -- 172.21.15.88:0/110418976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa35c0b1990 msgr2=0x7fa35c1510e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:20.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.122+0000 7fa353fff640 1 --2- 172.21.15.88:0/110418976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa35c0b1990 0x7fa35c1510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:20.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.122+0000 7fa35b7fe640 1 --2- 172.21.15.88:0/110418976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa35c0b1990 0x7fa35c1510e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.122+0000 7fa35b7fe640 1 -- 172.21.15.88:0/110418976 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa35c159fe0 con 0x7fa35c073080 2024-09-16T01:39:20.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.122+0000 7fa353fff640 1 --2- 172.21.15.88:0/110418976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa35c0b1990 0x7fa35c1510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:20.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.122+0000 7fa35b7fe640 1 --2- 172.21.15.88:0/110418976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa35c073080 0x7fa35c06e910 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fa34c02a2b0 tx=0x7fa34c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:20.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.122+0000 7fa358ff9640 1 -- 172.21.15.88:0/110418976 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34c037040 con 0x7fa35c073080 2024-09-16T01:39:20.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.122+0000 7fa358ff9640 1 -- 172.21.15.88:0/110418976 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa34c042070 con 0x7fa35c073080 2024-09-16T01:39:20.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.122+0000 7fa358ff9640 1 -- 172.21.15.88:0/110418976 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34c030bb0 con 0x7fa35c073080 2024-09-16T01:39:20.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.122+0000 7fa361b0a640 1 -- 172.21.15.88:0/110418976 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa35c15a280 con 0x7fa35c073080 2024-09-16T01:39:20.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.122+0000 7fa361b0a640 1 -- 172.21.15.88:0/110418976 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa35c15a840 con 0x7fa35c073080 2024-09-16T01:39:20.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.123+0000 7fa361b0a640 1 -- 172.21.15.88:0/110418976 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa35c15bd90 con 0x7fa35c073080 2024-09-16T01:39:20.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.123+0000 7fa358ff9640 1 -- 172.21.15.88:0/110418976 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa34c047460 con 0x7fa35c073080 2024-09-16T01:39:20.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.124+0000 7fa358ff9640 1 --2- 172.21.15.88:0/110418976 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa33c06c2e0 0x7fa33c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:20.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.124+0000 7fa358ff9640 1 -- 172.21.15.88:0/110418976 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7fa34c0abfe0 con 0x7fa35c073080 2024-09-16T01:39:20.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.124+0000 7fa353fff640 1 --2- 172.21.15.88:0/110418976 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa33c06c2e0 0x7fa33c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:20.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.124+0000 7fa353fff640 1 --2- 172.21.15.88:0/110418976 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa33c06c2e0 0x7fa33c06e7a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa35c06cfc0 tx=0x7fa344009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:20.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.128+0000 7fa358ff9640 1 -- 172.21.15.88:0/110418976 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa35c15bd90 con 0x7fa35c073080 2024-09-16T01:39:20.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.327+0000 7fa361b0a640 1 -- 172.21.15.88:0/110418976 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa35c0d1f00 con 0x7fa35c073080 2024-09-16T01:39:20.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.328+0000 7fa358ff9640 1 -- 172.21.15.88:0/110418976 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v57) ==== 53+0+73 (secure 0 0 0) 0x7fa34c045360 con 0x7fa35c073080 2024-09-16T01:39:20.329 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:39:20.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.329+0000 7fa361b0a640 1 -- 172.21.15.88:0/110418976 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa33c06c2e0 msgr2=0x7fa33c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:20.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.329+0000 7fa361b0a640 1 --2- 172.21.15.88:0/110418976 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa33c06c2e0 0x7fa33c06e7a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa35c06cfc0 tx=0x7fa344009040 comp rx=0 tx=0).stop 2024-09-16T01:39:20.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.329+0000 7fa361b0a640 1 -- 172.21.15.88:0/110418976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa35c073080 msgr2=0x7fa35c06e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:20.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.329+0000 7fa361b0a640 1 --2- 172.21.15.88:0/110418976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa35c073080 0x7fa35c06e910 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fa34c02a2b0 tx=0x7fa34c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.330+0000 7fa361b0a640 1 -- 172.21.15.88:0/110418976 shutdown_connections 2024-09-16T01:39:20.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.330+0000 7fa361b0a640 1 --2- 172.21.15.88:0/110418976 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa33c06c2e0 0x7fa33c06e7a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.330+0000 7fa361b0a640 1 --2- 172.21.15.88:0/110418976 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa35c15c670 0x7fa35c159aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.330+0000 7fa361b0a640 1 --2- 172.21.15.88:0/110418976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa35c0b1990 0x7fa35c1510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.330+0000 7fa361b0a640 1 --2- 172.21.15.88:0/110418976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa35c073080 0x7fa35c06e910 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:20.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.330+0000 7fa361b0a640 1 -- 172.21.15.88:0/110418976 >> 172.21.15.88:0/110418976 conn(0x7fa35c0d24d0 msgr2=0x7fa35c073460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:20.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.330+0000 7fa361b0a640 1 -- 172.21.15.88:0/110418976 shutdown_connections 2024-09-16T01:39:20.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20.330+0000 7fa361b0a640 1 -- 172.21.15.88:0/110418976 wait complete. 2024-09-16T01:39:20.341 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-16T01:39:20.341 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:39:20.342 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2024-09-16T01:39:20.342 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T01:39:20.342 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T01:39:20.342 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-16T01:39:20.342 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-16T01:39:20.342 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:39:20.342 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:39:20.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:39:20.342 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2024-09-16T01:39:20.358 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:39:20.358 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:39:20.358 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:39:20.358 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:39:20.358 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-09-16T01:39:20.358 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:39:20.359 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:39:20.359 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:39:20.359 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2024-09-16T01:39:20.359 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:39:20.359 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:39:20.359 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2024-09-16T01:39:20.359 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:39:20.359 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2024-09-16T01:39:20.359 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:39:20.359 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2024-09-16T01:39:20.359 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:39:20.359 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42464sec preferred_lft 42464sec 2024-09-16T01:39:20.359 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link noprefixroute 2024-09-16T01:39:20.360 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:39:20.360 INFO:teuthology.orchestra.run.smithi088.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:39:20.360 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff 2024-09-16T01:39:20.360 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T01:39:20.360 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:39:20.360 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::e486:6eff:fe81:a1c/64 scope link 2024-09-16T01:39:20.360 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:39:20.360 INFO:teuthology.orchestra.run.smithi088.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:39:20.360 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 12:6f:29:be:95:48 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:39:20.360 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::106f:29ff:febe:9548/64 scope link 2024-09-16T01:39:20.360 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:39:20.360 INFO:teuthology.orchestra.run.smithi088.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:39:20.361 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T01:39:20.361 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::c63:b4ff:feb3:ad9e/64 scope link 2024-09-16T01:39:20.361 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:39:20.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:39:20.361 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:39:20.421 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-09-16T01:39:20.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:39:20.422 DEBUG:teuthology.orchestra.run.smithi088:> route 2024-09-16T01:39:20.478 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2024-09-16T01:39:20.478 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:39:20.478 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:39:20.478 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:39:20.478 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:39:20.479 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:39:20.479 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:39:20.479 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:39:20.479 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:39:20.479 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:39:20.479 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:39:20.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:39:20.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:39:20.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:39:20.603 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:39:20.619 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T01:39:20.619 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T01:39:20.619 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:39:20.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:39:20.620 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2024-09-16T01:39:20.683 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:39:20.683 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T01:39:20.738 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:39:20.739 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:39:20.739 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:39:20.794 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-16T01:39:20.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:39:20.794 DEBUG:teuthology.orchestra.run.smithi088:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:39:20.878 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:39:20.878 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:39:20.878 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:39:20.887 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:39:20.887 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:39:20.887 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:39:20.887 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:39:20.887 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:39:20.893 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:39:20.893 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:39:20.893 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:39:20.893 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:39:20.893 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:39:20.893 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790" 2024-09-16T01:39:20.893 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2024-09-16T01:39:20.893 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5ace2c91-46f9-482b-8a89-731fbdc171bf" will pass to kernel 2024-09-16T01:39:20.894 INFO:teuthology.orchestra.run.smithi088.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-16T01:39:20.894 INFO:teuthology.orchestra.run.smithi088.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:39:20.894 INFO:teuthology.orchestra.run.smithi088.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:39:20.894 INFO:teuthology.orchestra.run.smithi088.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:39:20.894 INFO:teuthology.orchestra.run.smithi088.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:39:20.896 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:39:20.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:39:20.896 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:39:20.921 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:39:21.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:39:21.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:39:21.023 INFO:teuthology.orchestra.run.smithi088.stdout:client6564 2024-09-16T01:39:21.023 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:39:21.023 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:39:21.039 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:39:21.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.175+0000 7f69f01cd640 1 Processor -- start 2024-09-16T01:39:21.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.175+0000 7f69f01cd640 1 -- start start 2024-09-16T01:39:21.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.176+0000 7f69f01cd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f69e8155800 0x7f69e80b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:21.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.176+0000 7f69f01cd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e815c330 0x7f69e80b4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:21.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.176+0000 7f69f01cd640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69e80b50a0 con 0x7f69e80cf570 2024-09-16T01:39:21.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.176+0000 7f69f01cd640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f69e80b4e60 con 0x7f69e8155800 2024-09-16T01:39:21.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.176+0000 7f69f01cd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f69e815c7a0 con 0x7f69e815c330 2024-09-16T01:39:21.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.176+0000 7f69edf69640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f69e8155800 0x7f69e80b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:21.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.176+0000 7f69edf69640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f69e8155800 0x7f69e80b6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49290/0 (socket says 172.21.15.88:49290) 2024-09-16T01:39:21.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.176+0000 7f69edf69640 1 -- 172.21.15.88:0/628348388 learned_addr learned my addr 172.21.15.88:0/628348388 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:21.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.176+0000 7f69ed768640 1 --2- 172.21.15.88:0/628348388 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e815c330 0x7f69e80b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:21.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.176+0000 7f69edf69640 1 -- 172.21.15.88:0/628348388 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e815c330 msgr2=0x7f69e80b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:21.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.176+0000 7f69edf69640 1 --2- 172.21.15.88:0/628348388 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e815c330 0x7f69e80b4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:21.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.176+0000 7f69edf69640 1 -- 172.21.15.88:0/628348388 >> v1:172.21.15.154:6789/0 conn(0x7f69e80cf570 legacy=0x7f69e815c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:21.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.177+0000 7f69edf69640 1 -- 172.21.15.88:0/628348388 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69e80b5380 con 0x7f69e8155800 2024-09-16T01:39:21.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.177+0000 7f69ed768640 1 --2- 172.21.15.88:0/628348388 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e815c330 0x7f69e80b4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:21.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.177+0000 7f69edf69640 1 --2- 172.21.15.88:0/628348388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69e8155800 0x7f69e80b6140 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f69d8009e50 tx=0x7f69d802a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=464bf00b2dca52e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:21.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.177+0000 7f69ecf67640 1 -- 172.21.15.88:0/628348388 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d8036070 con 0x7f69e8155800 2024-09-16T01:39:21.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.177+0000 7f69ecf67640 1 -- 172.21.15.88:0/628348388 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69d8031600 con 0x7f69e8155800 2024-09-16T01:39:21.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.177+0000 7f69ecf67640 1 -- 172.21.15.88:0/628348388 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d80319d0 con 0x7f69e8155800 2024-09-16T01:39:21.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.178+0000 7f69f01cd640 1 -- 172.21.15.88:0/628348388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69e8155800 msgr2=0x7f69e80b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:21.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.178+0000 7f69f01cd640 1 --2- 172.21.15.88:0/628348388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69e8155800 0x7f69e80b6140 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f69d8009e50 tx=0x7f69d802a6d0 comp rx=0 tx=0).stop 2024-09-16T01:39:21.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.178+0000 7f69f01cd640 1 -- 172.21.15.88:0/628348388 shutdown_connections 2024-09-16T01:39:21.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.178+0000 7f69f01cd640 1 --2- 172.21.15.88:0/628348388 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e815c330 0x7f69e80b4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:21.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.178+0000 7f69f01cd640 1 --2- 172.21.15.88:0/628348388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69e8155800 0x7f69e80b6140 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:21.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.178+0000 7f69f01cd640 1 -- 172.21.15.88:0/628348388 >> 172.21.15.88:0/628348388 conn(0x7f69e806bc10 msgr2=0x7f69e806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:21.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.178+0000 7f69f01cd640 1 -- 172.21.15.88:0/628348388 shutdown_connections 2024-09-16T01:39:21.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.178+0000 7f69f01cd640 1 -- 172.21.15.88:0/628348388 wait complete. 2024-09-16T01:39:21.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.179+0000 7f69f01cd640 1 Processor -- start 2024-09-16T01:39:21.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.179+0000 7f69f01cd640 1 -- start start 2024-09-16T01:39:21.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.180+0000 7f69f01cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69e80cf570 0x7f69e8154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:21.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.180+0000 7f69f01cd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69e8155800 0x7f69e8155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:21.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.180+0000 7f69f01cd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e815c330 0x7f69e815a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:21.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.180+0000 7f69f01cd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f69e80b5c20 con 0x7f69e80cf570 2024-09-16T01:39:21.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.180+0000 7f69f01cd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f69e80b5940 con 0x7f69e8155800 2024-09-16T01:39:21.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.180+0000 7f69f01cd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f69e8155be0 con 0x7f69e815c330 2024-09-16T01:39:21.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.180+0000 7f69edf69640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69e80cf570 0x7f69e8154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:21.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.180+0000 7f69edf69640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69e80cf570 0x7f69e8154ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49302/0 (socket says 172.21.15.88:49302) 2024-09-16T01:39:21.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.180+0000 7f69edf69640 1 -- 172.21.15.88:0/1785396253 learned_addr learned my addr 172.21.15.88:0/1785396253 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:21.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.180+0000 7f69ed768640 1 --2- 172.21.15.88:0/1785396253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69e8155800 0x7f69e8155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:21.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.180+0000 7f69ee76a640 1 --2- 172.21.15.88:0/1785396253 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e815c330 0x7f69e815a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:21.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.180+0000 7f69edf69640 1 -- 172.21.15.88:0/1785396253 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e815c330 msgr2=0x7f69e815a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:21.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.180+0000 7f69edf69640 1 --2- 172.21.15.88:0/1785396253 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e815c330 0x7f69e815a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:21.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.180+0000 7f69edf69640 1 -- 172.21.15.88:0/1785396253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69e8155800 msgr2=0x7f69e8155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:21.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.180+0000 7f69edf69640 1 --2- 172.21.15.88:0/1785396253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69e8155800 0x7f69e8155e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:21.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.180+0000 7f69edf69640 1 -- 172.21.15.88:0/1785396253 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69e8159570 con 0x7f69e80cf570 2024-09-16T01:39:21.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.180+0000 7f69edf69640 1 --2- 172.21.15.88:0/1785396253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69e80cf570 0x7f69e8154ab0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f69d8009e50 tx=0x7f69d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:21.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.181+0000 7f69e67fc640 1 -- 172.21.15.88:0/1785396253 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d8036050 con 0x7f69e80cf570 2024-09-16T01:39:21.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.181+0000 7f69e67fc640 1 -- 172.21.15.88:0/1785396253 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69d8042070 con 0x7f69e80cf570 2024-09-16T01:39:21.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.181+0000 7f69f01cd640 1 -- 172.21.15.88:0/1785396253 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69e8159850 con 0x7f69e80cf570 2024-09-16T01:39:21.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.181+0000 7f69e67fc640 1 -- 172.21.15.88:0/1785396253 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d8009510 con 0x7f69e80cf570 2024-09-16T01:39:21.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.181+0000 7f69f01cd640 1 -- 172.21.15.88:0/1785396253 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69e8156390 con 0x7f69e80cf570 2024-09-16T01:39:21.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.182+0000 7f69f01cd640 1 -- 172.21.15.88:0/1785396253 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69e80b6140 con 0x7f69e80cf570 2024-09-16T01:39:21.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.182+0000 7f69e67fc640 1 -- 172.21.15.88:0/1785396253 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f69d8007c50 con 0x7f69e80cf570 2024-09-16T01:39:21.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.182+0000 7f69e67fc640 1 --2- 172.21.15.88:0/1785396253 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f69d406c2b0 0x7f69d406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:21.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.183+0000 7f69e67fc640 1 -- 172.21.15.88:0/1785396253 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f69d80ac470 con 0x7f69e80cf570 2024-09-16T01:39:21.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.183+0000 7f69ed768640 1 --2- 172.21.15.88:0/1785396253 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f69d406c2b0 0x7f69d406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:21.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.183+0000 7f69ed768640 1 --2- 172.21.15.88:0/1785396253 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f69d406c2b0 0x7f69d406e770 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f69e80b4a50 tx=0x7f69dc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:21.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.187+0000 7f69e67fc640 1 -- 172.21.15.88:0/1785396253 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f69d807abe0 con 0x7f69e80cf570 2024-09-16T01:39:21.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.387+0000 7f69f01cd640 1 -- 172.21.15.88:0/1785396253 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69e80cde80 con 0x7f69e80cf570 2024-09-16T01:39:21.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.388+0000 7f69e67fc640 1 -- 172.21.15.88:0/1785396253 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+5231 (secure 0 0 0) 0x7f69d807a4e0 con 0x7f69e80cf570 2024-09-16T01:39:21.389 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:21.389 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":57,"btime":"2024-09-16T01:39:17:483517+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5725,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2696547723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2696547723},{"type":"v1","addr":"172.21.15.154:6834","nonce":2696547723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54},{"gid":6411,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3276199399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3276199399},{"type":"v1","addr":"172.21.15.88:6836","nonce":3276199399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54},{"gid":6414,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2944989249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2944989249},{"type":"v1","addr":"172.21.15.88:6837","nonce":2944989249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:39:14.808422+0000","modified":"2024-09-16T01:39:16.811457+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6429":{"gid":6429,"name":"d","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6835/3482042494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":3482042494},{"type":"v1","addr":"172.21.15.154:6835","nonce":3482042494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6429,"qdb_cluster":[6429]},"id":5}]} 2024-09-16T01:39:21.390 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 57 2024-09-16T01:39:21.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.390+0000 7f69f01cd640 1 -- 172.21.15.88:0/1785396253 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f69d406c2b0 msgr2=0x7f69d406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:21.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.390+0000 7f69f01cd640 1 --2- 172.21.15.88:0/1785396253 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f69d406c2b0 0x7f69d406e770 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f69e80b4a50 tx=0x7f69dc008040 comp rx=0 tx=0).stop 2024-09-16T01:39:21.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.390+0000 7f69f01cd640 1 -- 172.21.15.88:0/1785396253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69e80cf570 msgr2=0x7f69e8154ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:21.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.390+0000 7f69f01cd640 1 --2- 172.21.15.88:0/1785396253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69e80cf570 0x7f69e8154ab0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f69d8009e50 tx=0x7f69d8004a40 comp rx=0 tx=0).stop 2024-09-16T01:39:21.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.390+0000 7f69f01cd640 1 -- 172.21.15.88:0/1785396253 shutdown_connections 2024-09-16T01:39:21.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.390+0000 7f69f01cd640 1 --2- 172.21.15.88:0/1785396253 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f69d406c2b0 0x7f69d406e770 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:21.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.390+0000 7f69f01cd640 1 --2- 172.21.15.88:0/1785396253 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69e815c330 0x7f69e815a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:21.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.390+0000 7f69f01cd640 1 --2- 172.21.15.88:0/1785396253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69e8155800 0x7f69e8155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:21.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.390+0000 7f69f01cd640 1 --2- 172.21.15.88:0/1785396253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69e80cf570 0x7f69e8154ab0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:21.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.390+0000 7f69f01cd640 1 -- 172.21.15.88:0/1785396253 >> 172.21.15.88:0/1785396253 conn(0x7f69e806bc10 msgr2=0x7f69e80ce160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:21.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.390+0000 7f69f01cd640 1 -- 172.21.15.88:0/1785396253 shutdown_connections 2024-09-16T01:39:21.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.390+0000 7f69f01cd640 1 -- 172.21.15.88:0/1785396253 wait complete. 2024-09-16T01:39:21.402 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds freeze 5:0 true 2024-09-16T01:39:21.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.502+0000 7fb3d10d0640 1 Processor -- start 2024-09-16T01:39:21.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.502+0000 7fb3d10d0640 1 -- start start 2024-09-16T01:39:21.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.502+0000 7fb3d10d0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3cc15a060 0x7fb3cc06f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:21.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.502+0000 7fb3d10d0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb3cc0b1990 0x7fb3cc06f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:21.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.502+0000 7fb3d10d0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3cc071770 con 0x7fb3cc15d3d0 2024-09-16T01:39:21.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.502+0000 7fb3d10d0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb3cc0712e0 con 0x7fb3cc0b1990 2024-09-16T01:39:21.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.502+0000 7fb3d10d0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb3cc0709b0 con 0x7fb3cc15a060 2024-09-16T01:39:21.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.503+0000 7fb3c3fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb3cc15d3d0 0x7fb3cc15ff40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50192/0 (socket says 172.21.15.88:50192) 2024-09-16T01:39:21.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.503+0000 7fb3c3fff640 1 -- 172.21.15.88:0/2246819476 learned_addr learned my addr 172.21.15.88:0/2246819476 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:21.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.503+0000 7fb3cb59e640 1 --2- 172.21.15.88:0/2246819476 >> v2:172.21.15.154:3300/0 conn(0x7fb3cc0b1990 0x7fb3cc06f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:21.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.503+0000 7fb3cad9d640 1 --2- 172.21.15.88:0/2246819476 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3cc15a060 0x7fb3cc06f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:21.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.503+0000 7fb3cb59e640 1 -- 172.21.15.88:0/2246819476 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3cc15a060 msgr2=0x7fb3cc06f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:21.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.503+0000 7fb3cb59e640 1 --2- 172.21.15.88:0/2246819476 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3cc15a060 0x7fb3cc06f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:21.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.503+0000 7fb3cb59e640 1 -- 172.21.15.88:0/2246819476 >> v1:172.21.15.88:6789/0 conn(0x7fb3cc15d3d0 legacy=0x7fb3cc15ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:21.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.503+0000 7fb3cb59e640 1 -- 172.21.15.88:0/2246819476 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3cc070c90 con 0x7fb3cc0b1990 2024-09-16T01:39:21.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.503+0000 7fb3cad9d640 1 --2- 172.21.15.88:0/2246819476 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3cc15a060 0x7fb3cc06f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:39:21.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.504+0000 7fb3cb59e640 1 --2- 172.21.15.88:0/2246819476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3cc0b1990 0x7fb3cc06f830 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009d90 tx=0x7fb3c402a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b674fe871c13705e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:21.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.507+0000 7fb3ca59c640 1 -- 172.21.15.88:0/2246819476 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4007740 con 0x7fb3cc0b1990 2024-09-16T01:39:21.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.507+0000 7fb3ca59c640 1 -- 172.21.15.88:0/2246819476 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3c40078e0 con 0x7fb3cc0b1990 2024-09-16T01:39:21.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.507+0000 7fb3ca59c640 1 -- 172.21.15.88:0/2246819476 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4007cb0 con 0x7fb3cc0b1990 2024-09-16T01:39:21.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.508+0000 7fb3d10d0640 1 -- 172.21.15.88:0/2246819476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3cc0b1990 msgr2=0x7fb3cc06f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:21.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.508+0000 7fb3d10d0640 1 --2- 172.21.15.88:0/2246819476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3cc0b1990 0x7fb3cc06f830 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009d90 tx=0x7fb3c402a660 comp rx=0 tx=0).stop 2024-09-16T01:39:21.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.508+0000 7fb3d10d0640 1 -- 172.21.15.88:0/2246819476 shutdown_connections 2024-09-16T01:39:21.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.508+0000 7fb3d10d0640 1 --2- 172.21.15.88:0/2246819476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3cc0b1990 0x7fb3cc06f830 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:21.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.508+0000 7fb3d10d0640 1 --2- 172.21.15.88:0/2246819476 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3cc15a060 0x7fb3cc06f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:21.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.508+0000 7fb3d10d0640 1 -- 172.21.15.88:0/2246819476 >> 172.21.15.88:0/2246819476 conn(0x7fb3cc0d2430 msgr2=0x7fb3cc0d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:21.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.508+0000 7fb3d10d0640 1 -- 172.21.15.88:0/2246819476 shutdown_connections 2024-09-16T01:39:21.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.508+0000 7fb3d10d0640 1 -- 172.21.15.88:0/2246819476 wait complete. 2024-09-16T01:39:21.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.509+0000 7fb3d10d0640 1 Processor -- start 2024-09-16T01:39:21.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.509+0000 7fb3d10d0640 1 -- start start 2024-09-16T01:39:21.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.510+0000 7fb3d10d0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3cc0b1990 0x7fb3cc14bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:21.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.510+0000 7fb3d10d0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3cc15a060 0x7fb3cc1542f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:21.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.510+0000 7fb3d10d0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3cc15d3d0 0x7fb3cc154830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:21.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.510+0000 7fb3d10d0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb3cc0d1460 con 0x7fb3cc15d3d0 2024-09-16T01:39:21.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.510+0000 7fb3d10d0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb3cc0d12e0 con 0x7fb3cc0b1990 2024-09-16T01:39:21.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.510+0000 7fb3d10d0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb3cc06feb0 con 0x7fb3cc15a060 2024-09-16T01:39:21.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.510+0000 7fb3cb59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3cc15d3d0 0x7fb3cc154830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:21.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.510+0000 7fb3cb59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3cc15d3d0 0x7fb3cc154830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49304/0 (socket says 172.21.15.88:49304) 2024-09-16T01:39:21.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.510+0000 7fb3cb59e640 1 -- 172.21.15.88:0/1643677936 learned_addr learned my addr 172.21.15.88:0/1643677936 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:21.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.510+0000 7fb3c3fff640 1 --2- 172.21.15.88:0/1643677936 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3cc15a060 0x7fb3cc1542f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:21.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.510+0000 7fb3cad9d640 1 --2- 172.21.15.88:0/1643677936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3cc0b1990 0x7fb3cc14bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:21.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.511+0000 7fb3cb59e640 1 -- 172.21.15.88:0/1643677936 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3cc15a060 msgr2=0x7fb3cc1542f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:21.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.511+0000 7fb3cb59e640 1 --2- 172.21.15.88:0/1643677936 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3cc15a060 0x7fb3cc1542f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:21.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.511+0000 7fb3cb59e640 1 -- 172.21.15.88:0/1643677936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3cc0b1990 msgr2=0x7fb3cc14bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:21.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.511+0000 7fb3cb59e640 1 --2- 172.21.15.88:0/1643677936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3cc0b1990 0x7fb3cc14bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:21.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.511+0000 7fb3cb59e640 1 -- 172.21.15.88:0/1643677936 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3cc156b90 con 0x7fb3cc15d3d0 2024-09-16T01:39:21.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.511+0000 7fb3c3fff640 1 --2- 172.21.15.88:0/1643677936 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3cc15a060 0x7fb3cc1542f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:21.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.511+0000 7fb3cb59e640 1 --2- 172.21.15.88:0/1643677936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3cc15d3d0 0x7fb3cc154830 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009eb0 tx=0x7fb3c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:21.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.511+0000 7fb3c37fe640 1 -- 172.21.15.88:0/1643677936 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4040070 con 0x7fb3cc15d3d0 2024-09-16T01:39:21.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.511+0000 7fb3c37fe640 1 -- 172.21.15.88:0/1643677936 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3c4031b40 con 0x7fb3cc15d3d0 2024-09-16T01:39:21.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.511+0000 7fb3d10d0640 1 -- 172.21.15.88:0/1643677936 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3cc156e70 con 0x7fb3cc15d3d0 2024-09-16T01:39:21.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.511+0000 7fb3c37fe640 1 -- 172.21.15.88:0/1643677936 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4045460 con 0x7fb3cc15d3d0 2024-09-16T01:39:21.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.511+0000 7fb3d10d0640 1 -- 172.21.15.88:0/1643677936 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3cc157430 con 0x7fb3cc15d3d0 2024-09-16T01:39:21.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.512+0000 7fb3d10d0640 1 -- 172.21.15.88:0/1643677936 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3cc06f2f0 con 0x7fb3cc15d3d0 2024-09-16T01:39:21.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.512+0000 7fb3c37fe640 1 -- 172.21.15.88:0/1643677936 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb3c4039040 con 0x7fb3cc15d3d0 2024-09-16T01:39:21.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.513+0000 7fb3c37fe640 1 --2- 172.21.15.88:0/1643677936 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb3ac06c2b0 0x7fb3ac06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:21.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.513+0000 7fb3c37fe640 1 -- 172.21.15.88:0/1643677936 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7fb3c40acc70 con 0x7fb3cc15d3d0 2024-09-16T01:39:21.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.513+0000 7fb3cad9d640 1 --2- 172.21.15.88:0/1643677936 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb3ac06c2b0 0x7fb3ac06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:21.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.513+0000 7fb3cad9d640 1 --2- 172.21.15.88:0/1643677936 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb3ac06c2b0 0x7fb3ac06e770 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc009560 tx=0x7fb3bc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:21.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.517+0000 7fb3c37fe640 1 -- 172.21.15.88:0/1643677936 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb3c407b3e0 con 0x7fb3cc15d3d0 2024-09-16T01:39:21.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:21.747+0000 7fb3d10d0640 1 -- 172.21.15.88:0/1643677936 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds freeze", "role_or_gid": "5:0", "val": "true"} v 0) -- 0x7fb3cc0b2de0 con 0x7fb3cc15d3d0 2024-09-16T01:39:22.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.508+0000 7fb3c37fe640 1 -- 172.21.15.88:0/1643677936 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds freeze", "role_or_gid": "5:0", "val": "true"}]=0 freezing mds.6429 v58) ==== 112+0+0 (secure 0 0 0) 0x7fb3c407ace0 con 0x7fb3cc15d3d0 2024-09-16T01:39:22.509 INFO:teuthology.orchestra.run.smithi088.stderr:freezing mds.6429 2024-09-16T01:39:22.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.510+0000 7fb3d10d0640 1 -- 172.21.15.88:0/1643677936 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb3ac06c2b0 msgr2=0x7fb3ac06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:22.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.510+0000 7fb3d10d0640 1 --2- 172.21.15.88:0/1643677936 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb3ac06c2b0 0x7fb3ac06e770 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc009560 tx=0x7fb3bc008040 comp rx=0 tx=0).stop 2024-09-16T01:39:22.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.510+0000 7fb3d10d0640 1 -- 172.21.15.88:0/1643677936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3cc15d3d0 msgr2=0x7fb3cc154830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:22.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.510+0000 7fb3d10d0640 1 --2- 172.21.15.88:0/1643677936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3cc15d3d0 0x7fb3cc154830 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009eb0 tx=0x7fb3c4004a40 comp rx=0 tx=0).stop 2024-09-16T01:39:22.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.510+0000 7fb3d10d0640 1 -- 172.21.15.88:0/1643677936 shutdown_connections 2024-09-16T01:39:22.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.510+0000 7fb3d10d0640 1 --2- 172.21.15.88:0/1643677936 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb3ac06c2b0 0x7fb3ac06e770 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.510+0000 7fb3d10d0640 1 --2- 172.21.15.88:0/1643677936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3cc15d3d0 0x7fb3cc154830 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.510+0000 7fb3d10d0640 1 --2- 172.21.15.88:0/1643677936 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb3cc15a060 0x7fb3cc1542f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.510+0000 7fb3d10d0640 1 --2- 172.21.15.88:0/1643677936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb3cc0b1990 0x7fb3cc14bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.510+0000 7fb3d10d0640 1 -- 172.21.15.88:0/1643677936 >> 172.21.15.88:0/1643677936 conn(0x7fb3cc0d2430 msgr2=0x7fb3cc0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:22.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.511+0000 7fb3d10d0640 1 -- 172.21.15.88:0/1643677936 shutdown_connections 2024-09-16T01:39:22.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.511+0000 7fb3d10d0640 1 -- 172.21.15.88:0/1643677936 wait complete. 2024-09-16T01:39:22.525 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'mkdir -p a/{b,c}/d') 2024-09-16T01:39:22.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:39:22.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:39:22.591 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:0 flush journal 2024-09-16T01:39:22.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.690+0000 7f794c33a640 1 Processor -- start 2024-09-16T01:39:22.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.690+0000 7f794c33a640 1 -- start start 2024-09-16T01:39:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.691+0000 7f794c33a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7944158a00 0x7f7944071f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.691+0000 7f794c33a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7944072470 0x7f79440b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.691+0000 7f794c33a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7944158de0 0x7f79440b53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.691+0000 7f794c33a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f79440728f0 con 0x7f7944072470 2024-09-16T01:39:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.691+0000 7f794c33a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7944072bd0 con 0x7f7944158a00 2024-09-16T01:39:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.691+0000 7f794c33a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7944072eb0 con 0x7f7944158de0 2024-09-16T01:39:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.691+0000 7f79498d5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7944072470 0x7f79440b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.691+0000 7f79498d5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7944072470 0x7f79440b4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49308/0 (socket says 172.21.15.88:49308) 2024-09-16T01:39:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.691+0000 7f79498d5640 1 -- 172.21.15.88:0/828637935 learned_addr learned my addr 172.21.15.88:0/828637935 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.691+0000 7f794a0d6640 1 --2- 172.21.15.88:0/828637935 >> v2:172.21.15.154:3300/0 conn(0x7f7944158a00 0x7f7944071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:22.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.691+0000 7f794a8d7640 1 --2- 172.21.15.88:0/828637935 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7944158de0 0x7f79440b53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:22.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.691+0000 7f79498d5640 1 -- 172.21.15.88:0/828637935 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7944158de0 msgr2=0x7f79440b53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:22.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.691+0000 7f79498d5640 1 --2- 172.21.15.88:0/828637935 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7944158de0 0x7f79440b53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.691+0000 7f79498d5640 1 -- 172.21.15.88:0/828637935 >> v2:172.21.15.154:3300/0 conn(0x7f7944158a00 msgr2=0x7f7944071f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:22.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.691+0000 7f79498d5640 1 --2- 172.21.15.88:0/828637935 >> v2:172.21.15.154:3300/0 conn(0x7f7944158a00 0x7f7944071f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.691+0000 7f79498d5640 1 -- 172.21.15.88:0/828637935 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7944073190 con 0x7f7944072470 2024-09-16T01:39:22.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.692+0000 7f79498d5640 1 --2- 172.21.15.88:0/828637935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7944072470 0x7f79440b4e60 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f7944069f90 tx=0x7f793402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed4a24ae083769d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:22.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.692+0000 7f79490d4640 1 -- 172.21.15.88:0/828637935 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7934037070 con 0x7f7944072470 2024-09-16T01:39:22.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.692+0000 7f79490d4640 1 -- 172.21.15.88:0/828637935 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7934032440 con 0x7f7944072470 2024-09-16T01:39:22.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.692+0000 7f79490d4640 1 -- 172.21.15.88:0/828637935 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7934032810 con 0x7f7944072470 2024-09-16T01:39:22.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.693+0000 7f794c33a640 1 -- 172.21.15.88:0/828637935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7944072470 msgr2=0x7f79440b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:22.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.693+0000 7f794c33a640 1 --2- 172.21.15.88:0/828637935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7944072470 0x7f79440b4e60 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f7944069f90 tx=0x7f793402a1e0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.693+0000 7f794c33a640 1 -- 172.21.15.88:0/828637935 shutdown_connections 2024-09-16T01:39:22.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.693+0000 7f794c33a640 1 --2- 172.21.15.88:0/828637935 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7944158de0 0x7f79440b53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.693+0000 7f794c33a640 1 --2- 172.21.15.88:0/828637935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7944072470 0x7f79440b4e60 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.693+0000 7f794c33a640 1 --2- 172.21.15.88:0/828637935 >> v2:172.21.15.154:3300/0 conn(0x7f7944158a00 0x7f7944071f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.693+0000 7f794c33a640 1 -- 172.21.15.88:0/828637935 >> 172.21.15.88:0/828637935 conn(0x7f794406cb50 msgr2=0x7f794406cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:22.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.693+0000 7f794c33a640 1 -- 172.21.15.88:0/828637935 shutdown_connections 2024-09-16T01:39:22.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.693+0000 7f794c33a640 1 -- 172.21.15.88:0/828637935 wait complete. 2024-09-16T01:39:22.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.694+0000 7f794c33a640 1 Processor -- start 2024-09-16T01:39:22.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.694+0000 7f794c33a640 1 -- start start 2024-09-16T01:39:22.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.694+0000 7f794c33a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7944072470 0x7f79441f38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.695+0000 7f794c33a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7944158a00 0x7f79440cf570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.695+0000 7f794c33a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7944158de0 0x7f79441575e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.695+0000 7f794c33a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f79440b5ca0 con 0x7f7944158de0 2024-09-16T01:39:22.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.695+0000 7f794c33a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f79440b5b20 con 0x7f7944072470 2024-09-16T01:39:22.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.695+0000 7f794c33a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7944162110 con 0x7f7944158a00 2024-09-16T01:39:22.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.695+0000 7f794a8d7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7944158de0 0x7f79441575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:22.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.695+0000 7f794a8d7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7944158de0 0x7f79441575e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49322/0 (socket says 172.21.15.88:49322) 2024-09-16T01:39:22.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.695+0000 7f794a8d7640 1 -- 172.21.15.88:0/2962110473 learned_addr learned my addr 172.21.15.88:0/2962110473 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:22.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.695+0000 7f794a0d6640 1 --2- 172.21.15.88:0/2962110473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7944072470 0x7f79441f38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:22.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.695+0000 7f79498d5640 1 --2- 172.21.15.88:0/2962110473 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7944158a00 0x7f79440cf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:22.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.695+0000 7f794a8d7640 1 -- 172.21.15.88:0/2962110473 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7944158a00 msgr2=0x7f79440cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:22.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.695+0000 7f794a8d7640 1 --2- 172.21.15.88:0/2962110473 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7944158a00 0x7f79440cf570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.695+0000 7f794a8d7640 1 -- 172.21.15.88:0/2962110473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7944072470 msgr2=0x7f79441f38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:22.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.695+0000 7f794a8d7640 1 --2- 172.21.15.88:0/2962110473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7944072470 0x7f79441f38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.695+0000 7f794a8d7640 1 -- 172.21.15.88:0/2962110473 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f794415fad0 con 0x7f7944158de0 2024-09-16T01:39:22.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.695+0000 7f794a8d7640 1 --2- 172.21.15.88:0/2962110473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7944158de0 0x7f79441575e0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f793c009fd0 tx=0x7f793c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:22.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.695+0000 7f793affd640 1 -- 172.21.15.88:0/2962110473 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c014420 con 0x7f7944158de0 2024-09-16T01:39:22.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.695+0000 7f793affd640 1 -- 172.21.15.88:0/2962110473 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f793c019070 con 0x7f7944158de0 2024-09-16T01:39:22.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.695+0000 7f793affd640 1 -- 172.21.15.88:0/2962110473 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c014800 con 0x7f7944158de0 2024-09-16T01:39:22.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.695+0000 7f794c33a640 1 -- 172.21.15.88:0/2962110473 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f794415fdb0 con 0x7f7944158de0 2024-09-16T01:39:22.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.695+0000 7f794c33a640 1 -- 172.21.15.88:0/2962110473 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7944160370 con 0x7f7944158de0 2024-09-16T01:39:22.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.697+0000 7f793affd640 1 -- 172.21.15.88:0/2962110473 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f793c0149a0 con 0x7f7944158de0 2024-09-16T01:39:22.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.697+0000 7f793affd640 1 --2- 172.21.15.88:0/2962110473 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f792806c2b0 0x7f792806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.697+0000 7f793affd640 1 -- 172.21.15.88:0/2962110473 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f793c08ba90 con 0x7f7944158de0 2024-09-16T01:39:22.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.697+0000 7f794a0d6640 1 --2- 172.21.15.88:0/2962110473 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f792806c2b0 0x7f792806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:22.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.698+0000 7f794a0d6640 1 --2- 172.21.15.88:0/2962110473 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f792806c2b0 0x7f792806e770 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7940006fd0 tx=0x7f7940008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:22.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.698+0000 7f794f0d4740 1 Processor -- start 2024-09-16T01:39:22.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.699+0000 7f794f0d4740 1 -- start start 2024-09-16T01:39:22.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.699+0000 7f794f0d4740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e5920 0x55ad8162b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.699+0000 7f794f0d4740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ad815e30e0 0x55ad815e5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.699+0000 7f794f0d4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad815e70b0 0x55ad815e01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.699+0000 7f794f0d4740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55ad815486f0 con 0x55ad815e5920 2024-09-16T01:39:22.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.699+0000 7f794f0d4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ad814f0330 con 0x55ad815e70b0 2024-09-16T01:39:22.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.699+0000 7f794f0d4740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55ad815446b0 con 0x55ad815e30e0 2024-09-16T01:39:22.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.699+0000 7f794a0d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e5920 0x55ad8162b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:22.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.699+0000 7f794a0d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e5920 0x55ad8162b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49324/0 (socket says 172.21.15.88:49324) 2024-09-16T01:39:22.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.699+0000 7f794a0d6640 1 -- 172.21.15.88:0/3809558485 learned_addr learned my addr 172.21.15.88:0/3809558485 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:22.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.700+0000 7f79498d5640 1 --2- 172.21.15.88:0/3809558485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ad815e30e0 0x55ad815e5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:22.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.700+0000 7f794a8d7640 1 --2- 172.21.15.88:0/3809558485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad815e70b0 0x55ad815e01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:22.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.700+0000 7f794a0d6640 1 -- 172.21.15.88:0/3809558485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ad815e30e0 msgr2=0x55ad815e5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:22.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.700+0000 7f794a0d6640 1 --2- 172.21.15.88:0/3809558485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ad815e30e0 0x55ad815e5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.700+0000 7f794a0d6640 1 -- 172.21.15.88:0/3809558485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad815e70b0 msgr2=0x55ad815e01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:22.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.700+0000 7f794a0d6640 1 --2- 172.21.15.88:0/3809558485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad815e70b0 0x55ad815e01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.700+0000 7f794a0d6640 1 -- 172.21.15.88:0/3809558485 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad815160d0 con 0x55ad815e5920 2024-09-16T01:39:22.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.700+0000 7f794a0d6640 1 --2- 172.21.15.88:0/3809558485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e5920 0x55ad8162b0c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f7940010bb0 tx=0x7f7940013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de7cafcb1e036808 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:22.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.700+0000 7f7927fff640 1 -- 172.21.15.88:0/3809558485 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794001c050 con 0x55ad815e5920 2024-09-16T01:39:22.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.700+0000 7f7927fff640 1 -- 172.21.15.88:0/3809558485 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7940026070 con 0x55ad815e5920 2024-09-16T01:39:22.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.700+0000 7f7927fff640 1 -- 172.21.15.88:0/3809558485 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7940021440 con 0x55ad815e5920 2024-09-16T01:39:22.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.701+0000 7f794f0d4740 1 -- 172.21.15.88:0/3809558485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e5920 msgr2=0x55ad8162b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:22.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.701+0000 7f794f0d4740 1 --2- 172.21.15.88:0/3809558485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e5920 0x55ad8162b0c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f7940010bb0 tx=0x7f7940013000 comp rx=0 tx=0).stop 2024-09-16T01:39:22.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.701+0000 7f794f0d4740 1 -- 172.21.15.88:0/3809558485 shutdown_connections 2024-09-16T01:39:22.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.701+0000 7f794f0d4740 1 --2- 172.21.15.88:0/3809558485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad815e70b0 0x55ad815e01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.701+0000 7f794f0d4740 1 --2- 172.21.15.88:0/3809558485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ad815e30e0 0x55ad815e5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.701+0000 7f794f0d4740 1 --2- 172.21.15.88:0/3809558485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e5920 0x55ad8162b0c0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.701+0000 7f794f0d4740 1 -- 172.21.15.88:0/3809558485 >> 172.21.15.88:0/3809558485 conn(0x55ad815e2370 msgr2=0x55ad8162a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:22.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.701+0000 7f794f0d4740 1 -- 172.21.15.88:0/3809558485 shutdown_connections 2024-09-16T01:39:22.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.701+0000 7f794f0d4740 1 -- 172.21.15.88:0/3809558485 wait complete. 2024-09-16T01:39:22.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.701+0000 7f794f0d4740 1 -- start start 2024-09-16T01:39:22.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.701+0000 7f794f0d4740 1 Processor -- start 2024-09-16T01:39:22.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.702+0000 7f794f0d4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad815e5920 0x55ad8160d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.702+0000 7f794f0d4740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ad815e70b0 0x55ad81627b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.702+0000 7f794f0d4740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e4540 0x55ad815e49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.702+0000 7f794f0d4740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55ad815e1810 con 0x55ad815e4540 2024-09-16T01:39:22.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.702+0000 7f794f0d4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ad814d14f0 con 0x55ad815e5920 2024-09-16T01:39:22.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.702+0000 7f794f0d4740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55ad8162b8f0 con 0x55ad815e70b0 2024-09-16T01:39:22.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.702+0000 7f794f0d4740 10 client.0 _ll_register_callbacks cb 0x55ad814f2b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:39:22.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.703+0000 7f794a8d7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e4540 0x55ad815e49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:22.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.703+0000 7f79498d5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ad815e70b0 0x55ad81627b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:22.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.703+0000 7f794a0d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad815e5920 0x55ad8160d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:22.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.703+0000 7f794a8d7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e4540 0x55ad815e49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49334/0 (socket says 172.21.15.88:49334) 2024-09-16T01:39:22.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.703+0000 7f794a8d7640 1 -- 172.21.15.88:0/884728459 learned_addr learned my addr 172.21.15.88:0/884728459 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:22.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.703+0000 7f794a8d7640 1 -- 172.21.15.88:0/884728459 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ad815e70b0 msgr2=0x55ad81627b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:22.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.703+0000 7f794a8d7640 1 --2- 172.21.15.88:0/884728459 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ad815e70b0 0x55ad81627b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.703+0000 7f794a8d7640 1 -- 172.21.15.88:0/884728459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad815e5920 msgr2=0x55ad8160d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:22.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.703+0000 7f794a8d7640 1 --2- 172.21.15.88:0/884728459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad815e5920 0x55ad8160d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.703+0000 7f794a8d7640 1 -- 172.21.15.88:0/884728459 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad813257d0 con 0x55ad815e4540 2024-09-16T01:39:22.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.703+0000 7f794a8d7640 1 --2- 172.21.15.88:0/884728459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e4540 0x55ad815e49c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f793c089fe0 tx=0x7f793c05df70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee2ee70ada10a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:22.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.703+0000 7f79257fa640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:39:22.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.703+0000 7f79257fa640 1 -- 172.21.15.88:0/884728459 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c059200 con 0x55ad815e4540 2024-09-16T01:39:22.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.703+0000 7f79257fa640 1 -- 172.21.15.88:0/884728459 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f793c0600b0 con 0x55ad815e4540 2024-09-16T01:39:22.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.703+0000 7f794f0d4740 1 -- 172.21.15.88:0/884728459 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7940026070 con 0x55ad815e4540 2024-09-16T01:39:22.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.703+0000 7f79257fa640 1 -- 172.21.15.88:0/884728459 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f793c0602b0 con 0x55ad815e4540 2024-09-16T01:39:22.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.703+0000 7f794f0d4740 10 client.6585 fetch_fsmap learned FSMap version 58 2024-09-16T01:39:22.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.703+0000 7f794f0d4740 1 -- 172.21.15.88:0/884728459 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=2+}) -- 0x55ad8125b3f0 con 0x55ad815e4540 2024-09-16T01:39:22.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.703+0000 7f79257fa640 1 -- 172.21.15.88:0/884728459 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7940026070 con 0x55ad815e4540 2024-09-16T01:39:22.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.703+0000 7f79257fa640 1 -- 172.21.15.88:0/884728459 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 58) ==== 3502+0+0 (secure 0 0 0) 0x7f793c055900 con 0x55ad815e4540 2024-09-16T01:39:22.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.704+0000 7f794f0d4740 10 client.6585 fetch_fsmap finished waiting for FSMap version 58 2024-09-16T01:39:22.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.704+0000 7f794f0d4740 10 client.6585 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.d 2024-09-16T01:39:22.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.704+0000 7f794f0d4740 20 client.6585 populate_metadata read hostname 'smithi088' 2024-09-16T01:39:22.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.704+0000 7f794f0d4740 1 --2- 172.21.15.88:0/884728459 >> [v2:172.21.15.154:6833/3482042494,v1:172.21.15.154:6835/3482042494] conn(0x55ad81604ed0 0x55ad816052b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.704+0000 7f794f0d4740 4 client.6585 mds_command: new command op to 6429 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:39:22.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.704+0000 7f794f0d4740 1 -- 172.21.15.88:0/884728459 --> [v2:172.21.15.154:6833/3482042494,v1:172.21.15.154:6835/3482042494] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad81555e20 con 0x55ad81604ed0 2024-09-16T01:39:22.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.704+0000 7f794a0d6640 1 --2- 172.21.15.88:0/884728459 >> [v2:172.21.15.154:6833/3482042494,v1:172.21.15.154:6835/3482042494] conn(0x55ad81604ed0 0x55ad816052b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:22.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.704+0000 7f79257fa640 10 client.6585 ms_handle_connect on v2:172.21.15.154:6833/3482042494 2024-09-16T01:39:22.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.704+0000 7f794a0d6640 1 --2- 172.21.15.88:0/884728459 >> [v2:172.21.15.154:6833/3482042494,v1:172.21.15.154:6835/3482042494] conn(0x55ad81604ed0 0x55ad816052b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cbb18758b9b9ec6 server_cookie=7cb54a335161958d in_seq=0 out_seq=0 2024-09-16T01:39:22.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.705+0000 7f79257fa640 1 -- 172.21.15.88:0/884728459 <== mds.0 v2:172.21.15.154:6833/3482042494 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ad81555e20 con 0x55ad81604ed0 2024-09-16T01:39:22.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.705+0000 7f79257fa640 10 client.6585 handle_command_reply: tid=0 2024-09-16T01:39:22.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.705+0000 7f794f0d4740 1 client.6585 shutdown 2024-09-16T01:39:22.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.705+0000 7f794f0d4740 1 -- 172.21.15.88:0/884728459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e4540 msgr2=0x55ad815e49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:22.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.705+0000 7f794f0d4740 1 --2- 172.21.15.88:0/884728459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e4540 0x55ad815e49c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f793c089fe0 tx=0x7f793c05df70 comp rx=0 tx=0).stop 2024-09-16T01:39:22.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.705+0000 7f794f0d4740 1 -- 172.21.15.88:0/884728459 shutdown_connections 2024-09-16T01:39:22.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.705+0000 7f794f0d4740 1 --2- 172.21.15.88:0/884728459 >> [v2:172.21.15.154:6833/3482042494,v1:172.21.15.154:6835/3482042494] conn(0x55ad81604ed0 0x55ad816052b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.705+0000 7f794f0d4740 1 --2- 172.21.15.88:0/884728459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e4540 0x55ad815e49c0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.705+0000 7f794f0d4740 1 --2- 172.21.15.88:0/884728459 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ad815e70b0 0x55ad81627b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.705+0000 7f79257fa640 0 client.6585 ms_handle_reset on v2:172.21.15.154:6833/3482042494 2024-09-16T01:39:22.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.705+0000 7f794f0d4740 1 --2- 172.21.15.88:0/884728459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad815e5920 0x55ad8160d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.705+0000 7f794f0d4740 1 -- 172.21.15.88:0/884728459 >> 172.21.15.88:0/884728459 conn(0x55ad815e30e0 msgr2=0x55ad8162c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:22.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.706+0000 7f794f0d4740 1 -- 172.21.15.88:0/884728459 shutdown_connections 2024-09-16T01:39:22.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.706+0000 7f794f0d4740 1 -- 172.21.15.88:0/884728459 wait complete. 2024-09-16T01:39:22.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.706+0000 7f794f0d4740 20 client.6585 trim_cache size 0 max 16384 2024-09-16T01:39:22.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.720+0000 7f794f0d4740 1 Processor -- start 2024-09-16T01:39:22.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.720+0000 7f794f0d4740 1 -- start start 2024-09-16T01:39:22.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.721+0000 7f794f0d4740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e4540 0x55ad8160e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.721+0000 7f794f0d4740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ad815e5920 0x55ad815e4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.721+0000 7f794f0d4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad815e70b0 0x55ad81627930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.721+0000 7f794f0d4740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7944200b30 con 0x55ad815e4540 2024-09-16T01:39:22.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.721+0000 7f794f0d4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f79441ffb10 con 0x55ad815e70b0 2024-09-16T01:39:22.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.721+0000 7f794f0d4740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f79441f4f40 con 0x55ad815e5920 2024-09-16T01:39:22.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.721+0000 7f794a0d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e4540 0x55ad8160e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:22.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.721+0000 7f794a0d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e4540 0x55ad8160e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49336/0 (socket says 172.21.15.88:49336) 2024-09-16T01:39:22.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.721+0000 7f794a0d6640 1 -- 172.21.15.88:0/3258510838 learned_addr learned my addr 172.21.15.88:0/3258510838 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:22.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.721+0000 7f794a0d6640 1 -- 172.21.15.88:0/3258510838 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ad815e5920 msgr2=0x55ad815e4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:22.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.721+0000 7f794a0d6640 1 --2- 172.21.15.88:0/3258510838 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ad815e5920 0x55ad815e4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.721+0000 7f794a0d6640 1 -- 172.21.15.88:0/3258510838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad815e70b0 msgr2=0x55ad81627930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:22.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.721+0000 7f794a0d6640 1 --2- 172.21.15.88:0/3258510838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad815e70b0 0x55ad81627930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.721+0000 7f794a0d6640 1 -- 172.21.15.88:0/3258510838 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad8121b0f0 con 0x55ad815e4540 2024-09-16T01:39:22.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.721+0000 7f794a0d6640 1 --2- 172.21.15.88:0/3258510838 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e4540 0x55ad8160e270 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f7940010620 tx=0x7f7940015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a21b071ab096e8a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:22.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.721+0000 7f79277fe640 1 -- 172.21.15.88:0/3258510838 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794001c040 con 0x55ad815e4540 2024-09-16T01:39:22.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.721+0000 7f79277fe640 1 -- 172.21.15.88:0/3258510838 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7940026050 con 0x55ad815e4540 2024-09-16T01:39:22.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.721+0000 7f79277fe640 1 -- 172.21.15.88:0/3258510838 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7940005b90 con 0x55ad815e4540 2024-09-16T01:39:22.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.722+0000 7f794f0d4740 1 -- 172.21.15.88:0/3258510838 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e4540 msgr2=0x55ad8160e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:22.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.722+0000 7f794f0d4740 1 --2- 172.21.15.88:0/3258510838 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e4540 0x55ad8160e270 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f7940010620 tx=0x7f7940015000 comp rx=0 tx=0).stop 2024-09-16T01:39:22.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.722+0000 7f794f0d4740 1 -- 172.21.15.88:0/3258510838 shutdown_connections 2024-09-16T01:39:22.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.722+0000 7f794f0d4740 1 --2- 172.21.15.88:0/3258510838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad815e70b0 0x55ad81627930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.722+0000 7f794f0d4740 1 --2- 172.21.15.88:0/3258510838 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ad815e5920 0x55ad815e4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.722+0000 7f794f0d4740 1 --2- 172.21.15.88:0/3258510838 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e4540 0x55ad8160e270 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.722+0000 7f794f0d4740 1 -- 172.21.15.88:0/3258510838 >> 172.21.15.88:0/3258510838 conn(0x55ad815e30e0 msgr2=0x55ad815db9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:22.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.722+0000 7f794f0d4740 1 -- 172.21.15.88:0/3258510838 shutdown_connections 2024-09-16T01:39:22.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.722+0000 7f794f0d4740 1 -- 172.21.15.88:0/3258510838 wait complete. 2024-09-16T01:39:22.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.722+0000 7f794f0d4740 1 -- start start 2024-09-16T01:39:22.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.722+0000 7f794f0d4740 1 Processor -- start 2024-09-16T01:39:22.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.723+0000 7f794f0d4740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ad815e4540 0x55ad816ab180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.723+0000 7f794f0d4740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e5920 0x55ad81629d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.723+0000 7f794f0d4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad815e70b0 0x55ad816b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.723+0000 7f794f0d4740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55ad8150b210 con 0x55ad815e5920 2024-09-16T01:39:22.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.723+0000 7f794f0d4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f79441f5590 con 0x55ad815e70b0 2024-09-16T01:39:22.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.723+0000 7f794f0d4740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55ad8162a820 con 0x55ad815e4540 2024-09-16T01:39:22.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.723+0000 7f794f0d4740 10 client.0 _ll_register_callbacks cb 0x7f793c055f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:39:22.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.723+0000 7f79498d5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e5920 0x55ad81629d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:22.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.723+0000 7f79498d5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e5920 0x55ad81629d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49350/0 (socket says 172.21.15.88:49350) 2024-09-16T01:39:22.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.723+0000 7f79498d5640 1 -- 172.21.15.88:0/3682068339 learned_addr learned my addr 172.21.15.88:0/3682068339 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:22.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.724+0000 7f794a0d6640 1 --2- 172.21.15.88:0/3682068339 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ad815e4540 0x55ad816ab180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:22.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.724+0000 7f79498d5640 1 -- 172.21.15.88:0/3682068339 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ad815e4540 msgr2=0x55ad816ab180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:22.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.724+0000 7f79498d5640 1 --2- 172.21.15.88:0/3682068339 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ad815e4540 0x55ad816ab180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.724+0000 7f79498d5640 1 -- 172.21.15.88:0/3682068339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad815e70b0 msgr2=0x55ad816b1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:22.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.724+0000 7f794a8d7640 1 --2- 172.21.15.88:0/3682068339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad815e70b0 0x55ad816b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:22.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.724+0000 7f79498d5640 1 --2- 172.21.15.88:0/3682068339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad815e70b0 0x55ad816b1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.724+0000 7f79498d5640 1 -- 172.21.15.88:0/3682068339 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad816ab6c0 con 0x55ad815e5920 2024-09-16T01:39:22.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.724+0000 7f794a8d7640 1 --2- 172.21.15.88:0/3682068339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad815e70b0 0x55ad816b1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:22.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.724+0000 7f79498d5640 1 --2- 172.21.15.88:0/3682068339 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e5920 0x55ad81629d70 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f7934035040 tx=0x7f79340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fac81443e0387b08 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:22.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.724+0000 7f79257fa640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:39:22.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.724+0000 7f79257fa640 1 -- 172.21.15.88:0/3682068339 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7934037040 con 0x55ad815e5920 2024-09-16T01:39:22.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.724+0000 7f79257fa640 1 -- 172.21.15.88:0/3682068339 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f7934047aa0 con 0x55ad815e5920 2024-09-16T01:39:22.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.724+0000 7f794f0d4740 1 -- 172.21.15.88:0/3682068339 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7940026050 con 0x55ad815e5920 2024-09-16T01:39:22.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.724+0000 7f79257fa640 1 -- 172.21.15.88:0/3682068339 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f7934032e40 con 0x55ad815e5920 2024-09-16T01:39:22.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.724+0000 7f794f0d4740 10 client.6591 fetch_fsmap learned FSMap version 58 2024-09-16T01:39:22.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.724+0000 7f794f0d4740 1 -- 172.21.15.88:0/3682068339 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=2+}) -- 0x55ad8162a310 con 0x55ad815e5920 2024-09-16T01:39:22.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.724+0000 7f79257fa640 1 -- 172.21.15.88:0/3682068339 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7940026050 con 0x55ad815e5920 2024-09-16T01:39:22.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.724+0000 7f79257fa640 1 -- 172.21.15.88:0/3682068339 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 58) ==== 3502+0+0 (secure 0 0 0) 0x7f7934047ca0 con 0x55ad815e5920 2024-09-16T01:39:22.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.724+0000 7f794f0d4740 10 client.6591 fetch_fsmap finished waiting for FSMap version 58 2024-09-16T01:39:22.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.724+0000 7f794f0d4740 10 client.6591 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.d 2024-09-16T01:39:22.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.725+0000 7f794f0d4740 20 client.6591 populate_metadata read hostname 'smithi088' 2024-09-16T01:39:22.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.725+0000 7f794f0d4740 1 --2- 172.21.15.88:0/3682068339 >> [v2:172.21.15.154:6833/3482042494,v1:172.21.15.154:6835/3482042494] conn(0x55ad81624c70 0x55ad815e4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.725+0000 7f794f0d4740 4 client.6591 mds_command: new command op to 6429 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:39:22.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.725+0000 7f794f0d4740 1 -- 172.21.15.88:0/3682068339 --> [v2:172.21.15.154:6833/3482042494,v1:172.21.15.154:6835/3482042494] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ad8160ec50 con 0x55ad81624c70 2024-09-16T01:39:22.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.725+0000 7f794a0d6640 1 --2- 172.21.15.88:0/3682068339 >> [v2:172.21.15.154:6833/3482042494,v1:172.21.15.154:6835/3482042494] conn(0x55ad81624c70 0x55ad815e4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:22.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.725+0000 7f79257fa640 10 client.6591 ms_handle_connect on v2:172.21.15.154:6833/3482042494 2024-09-16T01:39:22.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.725+0000 7f794a0d6640 1 --2- 172.21.15.88:0/3682068339 >> [v2:172.21.15.154:6833/3482042494,v1:172.21.15.154:6835/3482042494] conn(0x55ad81624c70 0x55ad815e4960 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=651faa33ef417f01 server_cookie=ee5d1aff0bbe1159 in_seq=0 out_seq=0 2024-09-16T01:39:22.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.738+0000 7f79257fa640 1 -- 172.21.15.88:0/3682068339 <== mds.0 v2:172.21.15.154:6833/3482042494 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ad8160ec50 con 0x55ad81624c70 2024-09-16T01:39:22.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.738+0000 7f79257fa640 10 client.6591 handle_command_reply: tid=0 2024-09-16T01:39:22.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.738+0000 7f794f0d4740 1 client.6591 shutdown 2024-09-16T01:39:22.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.739+0000 7f794f0d4740 1 -- 172.21.15.88:0/3682068339 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e5920 msgr2=0x55ad81629d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:22.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.739+0000 7f794f0d4740 1 --2- 172.21.15.88:0/3682068339 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e5920 0x55ad81629d70 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f7934035040 tx=0x7f79340046c0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.739+0000 7f794f0d4740 1 -- 172.21.15.88:0/3682068339 shutdown_connections 2024-09-16T01:39:22.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.739+0000 7f794f0d4740 1 --2- 172.21.15.88:0/3682068339 >> [v2:172.21.15.154:6833/3482042494,v1:172.21.15.154:6835/3482042494] conn(0x55ad81624c70 0x55ad815e4960 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.739+0000 7f794f0d4740 1 --2- 172.21.15.88:0/3682068339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ad815e70b0 0x55ad816b1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.739+0000 7f794f0d4740 1 --2- 172.21.15.88:0/3682068339 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ad815e5920 0x55ad81629d70 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.739+0000 7f79257fa640 0 client.6591 ms_handle_reset on v2:172.21.15.154:6833/3482042494 2024-09-16T01:39:22.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.739+0000 7f794f0d4740 1 --2- 172.21.15.88:0/3682068339 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ad815e4540 0x55ad816ab180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.739+0000 7f794f0d4740 1 -- 172.21.15.88:0/3682068339 >> 172.21.15.88:0/3682068339 conn(0x55ad815e30e0 msgr2=0x55ad815db7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:22.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.739+0000 7f794f0d4740 1 -- 172.21.15.88:0/3682068339 shutdown_connections 2024-09-16T01:39:22.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.739+0000 7f794f0d4740 1 -- 172.21.15.88:0/3682068339 wait complete. 2024-09-16T01:39:22.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.739+0000 7f794f0d4740 20 client.6591 trim_cache size 0 max 16384 2024-09-16T01:39:22.742 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:39:22.742 INFO:teuthology.orchestra.run.smithi088.stdout: "message": "", 2024-09-16T01:39:22.742 INFO:teuthology.orchestra.run.smithi088.stdout: "return_code": 0 2024-09-16T01:39:22.743 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:39:22.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.739+0000 7f794c33a640 1 -- 172.21.15.88:0/2962110473 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f792806c2b0 msgr2=0x7f792806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:22.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.739+0000 7f794c33a640 1 --2- 172.21.15.88:0/2962110473 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f792806c2b0 0x7f792806e770 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7940006fd0 tx=0x7f7940008040 comp rx=0 tx=0).stop 2024-09-16T01:39:22.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.739+0000 7f794c33a640 1 -- 172.21.15.88:0/2962110473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7944158de0 msgr2=0x7f79441575e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:22.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.739+0000 7f794c33a640 1 --2- 172.21.15.88:0/2962110473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7944158de0 0x7f79441575e0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f793c009fd0 tx=0x7f793c00e4b0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.739+0000 7f794c33a640 1 -- 172.21.15.88:0/2962110473 shutdown_connections 2024-09-16T01:39:22.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.739+0000 7f794c33a640 1 --2- 172.21.15.88:0/2962110473 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f792806c2b0 0x7f792806e770 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.740+0000 7f794c33a640 1 --2- 172.21.15.88:0/2962110473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7944158de0 0x7f79441575e0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.740+0000 7f794c33a640 1 --2- 172.21.15.88:0/2962110473 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7944158a00 0x7f79440cf570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.740+0000 7f794c33a640 1 --2- 172.21.15.88:0/2962110473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7944072470 0x7f79441f38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.740+0000 7f794c33a640 1 -- 172.21.15.88:0/2962110473 >> 172.21.15.88:0/2962110473 conn(0x7f794406cb50 msgr2=0x7f79440afcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:22.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.740+0000 7f794c33a640 1 -- 172.21.15.88:0/2962110473 shutdown_connections 2024-09-16T01:39:22.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.740+0000 7f794c33a640 1 -- 172.21.15.88:0/2962110473 wait complete. 2024-09-16T01:39:22.751 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_inject_rename_corrupt_dentry_first 1.0 2024-09-16T01:39:22.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.852+0000 7f5e89359640 1 Processor -- start 2024-09-16T01:39:22.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.852+0000 7f5e89359640 1 -- start start 2024-09-16T01:39:22.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.853+0000 7f5e89359640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e8415a060 0x7f5e8406f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.853+0000 7f5e89359640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5e840b1990 0x7f5e8406f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.853+0000 7f5e89359640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e84071770 con 0x7f5e8415d3d0 2024-09-16T01:39:22.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.853+0000 7f5e89359640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f5e840712e0 con 0x7f5e840b1990 2024-09-16T01:39:22.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.853+0000 7f5e89359640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5e840709b0 con 0x7f5e8415a060 2024-09-16T01:39:22.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.853+0000 7f5e837fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5e840b1990 0x7f5e8406f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:22.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.853+0000 7f5e7a7fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5e8415d3d0 0x7f5e8415ff40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50202/0 (socket says 172.21.15.88:50202) 2024-09-16T01:39:22.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.853+0000 7f5e7a7fc640 1 -- 172.21.15.88:0/1240929670 learned_addr learned my addr 172.21.15.88:0/1240929670 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:22.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.853+0000 7f5e82ffd640 1 --2- 172.21.15.88:0/1240929670 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e8415a060 0x7f5e8406f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:22.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.853+0000 7f5e837fe640 1 -- 172.21.15.88:0/1240929670 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e8415a060 msgr2=0x7f5e8406f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:22.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.853+0000 7f5e837fe640 1 --2- 172.21.15.88:0/1240929670 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e8415a060 0x7f5e8406f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.853+0000 7f5e837fe640 1 -- 172.21.15.88:0/1240929670 >> v1:172.21.15.88:6789/0 conn(0x7f5e8415d3d0 legacy=0x7f5e8415ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:22.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.853+0000 7f5e837fe640 1 -- 172.21.15.88:0/1240929670 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e84070c90 con 0x7f5e840b1990 2024-09-16T01:39:22.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.853+0000 7f5e827fc640 1 -- 172.21.15.88:0/1240929670 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4277525338 0 0) 0x7f5e84071770 con 0x7f5e8415d3d0 2024-09-16T01:39:22.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.854+0000 7f5e837fe640 1 --2- 172.21.15.88:0/1240929670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e840b1990 0x7f5e8406f830 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c009d90 tx=0x7f5e7c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e753adadb013bc12 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:22.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.854+0000 7f5e827fc640 1 -- 172.21.15.88:0/1240929670 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c007740 con 0x7f5e840b1990 2024-09-16T01:39:22.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.854+0000 7f5e827fc640 1 -- 172.21.15.88:0/1240929670 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e7c0078e0 con 0x7f5e840b1990 2024-09-16T01:39:22.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.854+0000 7f5e827fc640 1 -- 172.21.15.88:0/1240929670 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c007cb0 con 0x7f5e840b1990 2024-09-16T01:39:22.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.855+0000 7f5e89359640 1 -- 172.21.15.88:0/1240929670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e840b1990 msgr2=0x7f5e8406f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:22.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.855+0000 7f5e89359640 1 --2- 172.21.15.88:0/1240929670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e840b1990 0x7f5e8406f830 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c009d90 tx=0x7f5e7c02a660 comp rx=0 tx=0).stop 2024-09-16T01:39:22.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.855+0000 7f5e89359640 1 -- 172.21.15.88:0/1240929670 shutdown_connections 2024-09-16T01:39:22.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.855+0000 7f5e89359640 1 --2- 172.21.15.88:0/1240929670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e840b1990 0x7f5e8406f830 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.855+0000 7f5e89359640 1 --2- 172.21.15.88:0/1240929670 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e8415a060 0x7f5e8406f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.855+0000 7f5e89359640 1 -- 172.21.15.88:0/1240929670 >> 172.21.15.88:0/1240929670 conn(0x7f5e840d2430 msgr2=0x7f5e840d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:22.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.855+0000 7f5e89359640 1 -- 172.21.15.88:0/1240929670 shutdown_connections 2024-09-16T01:39:22.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.855+0000 7f5e89359640 1 -- 172.21.15.88:0/1240929670 wait complete. 2024-09-16T01:39:22.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.856+0000 7f5e89359640 1 Processor -- start 2024-09-16T01:39:22.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.856+0000 7f5e89359640 1 -- start start 2024-09-16T01:39:22.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.857+0000 7f5e89359640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e840b1990 0x7f5e84143ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.857+0000 7f5e89359640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e8415a060 0x7f5e84158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.857+0000 7f5e89359640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e8415d3d0 0x7f5e84158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.857+0000 7f5e89359640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5e840d1460 con 0x7f5e8415d3d0 2024-09-16T01:39:22.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.857+0000 7f5e89359640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5e840d12e0 con 0x7f5e8415a060 2024-09-16T01:39:22.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.857+0000 7f5e89359640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5e8406feb0 con 0x7f5e840b1990 2024-09-16T01:39:22.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.857+0000 7f5e837fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e8415d3d0 0x7f5e84158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:22.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.857+0000 7f5e837fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e8415d3d0 0x7f5e84158cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49364/0 (socket says 172.21.15.88:49364) 2024-09-16T01:39:22.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.857+0000 7f5e837fe640 1 -- 172.21.15.88:0/2885573034 learned_addr learned my addr 172.21.15.88:0/2885573034 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:22.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.857+0000 7f5e82ffd640 1 --2- 172.21.15.88:0/2885573034 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e840b1990 0x7f5e84143ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:22.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.857+0000 7f5e7a7fc640 1 --2- 172.21.15.88:0/2885573034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e8415a060 0x7f5e84158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:22.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.857+0000 7f5e837fe640 1 -- 172.21.15.88:0/2885573034 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e840b1990 msgr2=0x7f5e84143ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:22.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.857+0000 7f5e837fe640 1 --2- 172.21.15.88:0/2885573034 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e840b1990 0x7f5e84143ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.857+0000 7f5e837fe640 1 -- 172.21.15.88:0/2885573034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e8415a060 msgr2=0x7f5e84158770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:22.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.857+0000 7f5e837fe640 1 --2- 172.21.15.88:0/2885573034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e8415a060 0x7f5e84158770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:22.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.857+0000 7f5e837fe640 1 -- 172.21.15.88:0/2885573034 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e841557e0 con 0x7f5e8415d3d0 2024-09-16T01:39:22.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.858+0000 7f5e837fe640 1 --2- 172.21.15.88:0/2885573034 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e8415d3d0 0x7f5e84158cb0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c009eb0 tx=0x7f5e7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:22.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.858+0000 7f5e7bfff640 1 -- 172.21.15.88:0/2885573034 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c040070 con 0x7f5e8415d3d0 2024-09-16T01:39:22.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.858+0000 7f5e7bfff640 1 -- 172.21.15.88:0/2885573034 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e7c031bc0 con 0x7f5e8415d3d0 2024-09-16T01:39:22.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.858+0000 7f5e89359640 1 -- 172.21.15.88:0/2885573034 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e84155ac0 con 0x7f5e8415d3d0 2024-09-16T01:39:22.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.858+0000 7f5e7bfff640 1 -- 172.21.15.88:0/2885573034 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c045460 con 0x7f5e8415d3d0 2024-09-16T01:39:22.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.858+0000 7f5e89359640 1 -- 172.21.15.88:0/2885573034 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e841502d0 con 0x7f5e8415d3d0 2024-09-16T01:39:22.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.859+0000 7f5e89359640 1 -- 172.21.15.88:0/2885573034 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e8406c6c0 con 0x7f5e8415d3d0 2024-09-16T01:39:22.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.859+0000 7f5e7bfff640 1 -- 172.21.15.88:0/2885573034 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5e7c039040 con 0x7f5e8415d3d0 2024-09-16T01:39:22.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.860+0000 7f5e7bfff640 1 --2- 172.21.15.88:0/2885573034 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e6406c2b0 0x7f5e6406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:22.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.860+0000 7f5e82ffd640 1 --2- 172.21.15.88:0/2885573034 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e6406c2b0 0x7f5e6406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:22.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.860+0000 7f5e7bfff640 1 -- 172.21.15.88:0/2885573034 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f5e7c0acc10 con 0x7f5e8415d3d0 2024-09-16T01:39:22.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.860+0000 7f5e82ffd640 1 --2- 172.21.15.88:0/2885573034 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e6406c2b0 0x7f5e6406e770 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5e74006fd0 tx=0x7f5e74008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:22.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:22.864+0000 7f5e7bfff640 1 -- 172.21.15.88:0/2885573034 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5e7c07ad30 con 0x7f5e8415d3d0 2024-09-16T01:39:23.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:23.027+0000 7f5e89359640 1 -- 172.21.15.88:0/2885573034 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command([{prefix=config set, name=mds_inject_rename_corrupt_dentry_first}] v 0) -- 0x7f5e8406f2f0 con 0x7f5e8415d3d0 2024-09-16T01:39:23.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:23.031+0000 7f5e7bfff640 1 -- 172.21.15.88:0/2885573034 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_inject_rename_corrupt_dentry_first}]=0 v23)=0 v23) ==== 138+0+0 (secure 0 0 0) 0x7f5e7c07f030 con 0x7f5e8415d3d0 2024-09-16T01:39:23.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:23.033+0000 7f5e89359640 1 -- 172.21.15.88:0/2885573034 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e6406c2b0 msgr2=0x7f5e6406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:23.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:23.033+0000 7f5e89359640 1 --2- 172.21.15.88:0/2885573034 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e6406c2b0 0x7f5e6406e770 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5e74006fd0 tx=0x7f5e74008040 comp rx=0 tx=0).stop 2024-09-16T01:39:23.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:23.033+0000 7f5e89359640 1 -- 172.21.15.88:0/2885573034 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e8415d3d0 msgr2=0x7f5e84158cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:23.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:23.033+0000 7f5e89359640 1 --2- 172.21.15.88:0/2885573034 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e8415d3d0 0x7f5e84158cb0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c009eb0 tx=0x7f5e7c004a40 comp rx=0 tx=0).stop 2024-09-16T01:39:23.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:23.033+0000 7f5e89359640 1 -- 172.21.15.88:0/2885573034 shutdown_connections 2024-09-16T01:39:23.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:23.033+0000 7f5e89359640 1 --2- 172.21.15.88:0/2885573034 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e6406c2b0 0x7f5e6406e770 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:23.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:23.033+0000 7f5e89359640 1 --2- 172.21.15.88:0/2885573034 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e8415d3d0 0x7f5e84158cb0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:23.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:23.033+0000 7f5e89359640 1 --2- 172.21.15.88:0/2885573034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e8415a060 0x7f5e84158770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:23.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:23.033+0000 7f5e89359640 1 --2- 172.21.15.88:0/2885573034 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e840b1990 0x7f5e84143ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:23.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:23.033+0000 7f5e89359640 1 -- 172.21.15.88:0/2885573034 >> 172.21.15.88:0/2885573034 conn(0x7f5e840d2430 msgr2=0x7f5e840d15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:23.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:23.033+0000 7f5e89359640 1 -- 172.21.15.88:0/2885573034 shutdown_connections 2024-09-16T01:39:23.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:23.033+0000 7f5e89359640 1 -- 172.21.15.88:0/2885573034 wait complete. 2024-09-16T01:39:28.047 DEBUG:teuthology.orchestra.run.smithi088:> sudo daemon-helper kill ceph --cluster ceph -w 2024-09-16T01:39:28.050 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'timeout 60 mv a/b a/z') 2024-09-16T01:39:28.052 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok --format=json config get mds_beacon_grace 2024-09-16T01:39:28.112 INFO:tasks.ceph.mds.d.smithi154.stderr:2024-09-16T01:39:28.111+0000 7f8a8f4d6640 -1 mds.0.cache.den(0x10000000000 z) newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.6564:33 nref=7 cr=0x561b4d221c00)) (dversion lock w=1 last_client=6564) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x561b4d3c7680] 2024-09-16T01:39:28.112 INFO:tasks.ceph.mds.d.smithi154.stderr:2024-09-16T01:39:28.111+0000 7f8a8f4d6640 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.6564:33 nref=7 cr=0x561b4d221c00)) (dversion lock w=1 last_client=6564) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x561b4d3c7680] 2024-09-16T01:39:28.113 INFO:tasks.ceph.mds.d.smithi154.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-326-g8921c8ef/rpm/el9/BUILD/ceph-19.1.1-326-g8921c8ef/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f8a8f4d6640 time 2024-09-16T01:39:28.112463+0000 2024-09-16T01:39:28.113 INFO:tasks.ceph.mds.d.smithi154.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-326-g8921c8ef/rpm/el9/BUILD/ceph-19.1.1-326-g8921c8ef/src/mds/MDSRank.cc: 952: ceph_abort_msg("abort() called") 2024-09-16T01:39:28.113 INFO:tasks.ceph.mds.d.smithi154.stderr: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T01:39:28.113 INFO:tasks.ceph.mds.d.smithi154.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f8a95183417] 2024-09-16T01:39:28.113 INFO:tasks.ceph.mds.d.smithi154.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x61) [0x561b4a1472d1] 2024-09-16T01:39:28.113 INFO:tasks.ceph.mds.d.smithi154.stderr: 3: (CDentry::check_corruption(bool)+0x4d4) [0x561b4a3724e4] 2024-09-16T01:39:28.114 INFO:tasks.ceph.mds.d.smithi154.stderr: 4: ceph-mds(+0x1cb3e9) [0x561b4a18a3e9] 2024-09-16T01:39:28.114 INFO:tasks.ceph.mds.d.smithi154.stderr: 5: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x207f) [0x561b4a2152df] 2024-09-16T01:39:28.114 INFO:tasks.ceph.mds.d.smithi154.stderr: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x25e2) [0x561b4a2185b2] 2024-09-16T01:39:28.114 INFO:tasks.ceph.mds.d.smithi154.stderr: 7: (Server::handle_client_request(boost::intrusive_ptr const&)+0x3e3) [0x561b4a1dedc3] 2024-09-16T01:39:28.114 INFO:tasks.ceph.mds.d.smithi154.stderr: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x13c) [0x561b4a1cccec] 2024-09-16T01:39:28.114 INFO:tasks.ceph.mds.d.smithi154.stderr: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x561b4a14e89b] 2024-09-16T01:39:28.114 INFO:tasks.ceph.mds.d.smithi154.stderr: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561b4a14ed6c] 2024-09-16T01:39:28.114 INFO:tasks.ceph.mds.d.smithi154.stderr: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1a6) [0x561b4a133286] 2024-09-16T01:39:28.114 INFO:tasks.ceph.mds.d.smithi154.stderr: 12: (DispatchQueue::entry()+0x4c2) [0x7f8a953905c2] 2024-09-16T01:39:28.114 INFO:tasks.ceph.mds.d.smithi154.stderr: 13: /usr/lib64/ceph/libceph-common.so.2(+0x42a1d1) [0x7f8a9542a1d1] 2024-09-16T01:39:28.114 INFO:tasks.ceph.mds.d.smithi154.stderr: 14: /lib64/libc.so.6(+0x89c52) [0x7f8a94889c52] 2024-09-16T01:39:28.114 INFO:tasks.ceph.mds.d.smithi154.stderr: 15: /lib64/libc.so.6(+0x10ec80) [0x7f8a9490ec80] 2024-09-16T01:39:28.114 INFO:tasks.ceph.mds.d.smithi154.stderr:2024-09-16T01:39:28.112+0000 7f8a8f4d6640 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-326-g8921c8ef/rpm/el9/BUILD/ceph-19.1.1-326-g8921c8ef/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f8a8f4d6640 time 2024-09-16T01:39:28.112463+0000 2024-09-16T01:39:28.114 INFO:tasks.ceph.mds.d.smithi154.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-326-g8921c8ef/rpm/el9/BUILD/ceph-19.1.1-326-g8921c8ef/src/mds/MDSRank.cc: 952: ceph_abort_msg("abort() called") 2024-09-16T01:39:28.115 INFO:tasks.ceph.mds.d.smithi154.stderr: 2024-09-16T01:39:28.115 INFO:tasks.ceph.mds.d.smithi154.stderr: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T01:39:28.115 INFO:tasks.ceph.mds.d.smithi154.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f8a95183417] 2024-09-16T01:39:28.115 INFO:tasks.ceph.mds.d.smithi154.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x61) [0x561b4a1472d1] 2024-09-16T01:39:28.115 INFO:tasks.ceph.mds.d.smithi154.stderr: 3: (CDentry::check_corruption(bool)+0x4d4) [0x561b4a3724e4] 2024-09-16T01:39:28.115 INFO:tasks.ceph.mds.d.smithi154.stderr: 4: ceph-mds(+0x1cb3e9) [0x561b4a18a3e9] 2024-09-16T01:39:28.115 INFO:tasks.ceph.mds.d.smithi154.stderr: 5: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x207f) [0x561b4a2152df] 2024-09-16T01:39:28.115 INFO:tasks.ceph.mds.d.smithi154.stderr: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x25e2) [0x561b4a2185b2] 2024-09-16T01:39:28.115 INFO:tasks.ceph.mds.d.smithi154.stderr: 7: (Server::handle_client_request(boost::intrusive_ptr const&)+0x3e3) [0x561b4a1dedc3] 2024-09-16T01:39:28.115 INFO:tasks.ceph.mds.d.smithi154.stderr: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x13c) [0x561b4a1cccec] 2024-09-16T01:39:28.115 INFO:tasks.ceph.mds.d.smithi154.stderr: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x561b4a14e89b] 2024-09-16T01:39:28.115 INFO:tasks.ceph.mds.d.smithi154.stderr: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561b4a14ed6c] 2024-09-16T01:39:28.115 INFO:tasks.ceph.mds.d.smithi154.stderr: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1a6) [0x561b4a133286] 2024-09-16T01:39:28.115 INFO:tasks.ceph.mds.d.smithi154.stderr: 12: (DispatchQueue::entry()+0x4c2) [0x7f8a953905c2] 2024-09-16T01:39:28.116 INFO:tasks.ceph.mds.d.smithi154.stderr: 13: /usr/lib64/ceph/libceph-common.so.2(+0x42a1d1) [0x7f8a9542a1d1] 2024-09-16T01:39:28.116 INFO:tasks.ceph.mds.d.smithi154.stderr: 14: /lib64/libc.so.6(+0x89c52) [0x7f8a94889c52] 2024-09-16T01:39:28.116 INFO:tasks.ceph.mds.d.smithi154.stderr: 15: /lib64/libc.so.6(+0x10ec80) [0x7f8a9490ec80] 2024-09-16T01:39:28.116 INFO:tasks.ceph.mds.d.smithi154.stderr: 2024-09-16T01:39:28.116 INFO:tasks.ceph.mds.d.smithi154.stderr:*** Caught signal (Aborted) ** 2024-09-16T01:39:28.116 INFO:tasks.ceph.mds.d.smithi154.stderr: in thread 7f8a8f4d6640 thread_name:ms_dispatch 2024-09-16T01:39:28.116 INFO:tasks.ceph.mds.d.smithi154.stderr: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T01:39:28.116 INFO:tasks.ceph.mds.d.smithi154.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f8a9483e6f0] 2024-09-16T01:39:28.117 INFO:tasks.ceph.mds.d.smithi154.stderr: 2: /lib64/libc.so.6(+0x8b99c) [0x7f8a9488b99c] 2024-09-16T01:39:28.117 INFO:tasks.ceph.mds.d.smithi154.stderr: 3: raise() 2024-09-16T01:39:28.117 INFO:tasks.ceph.mds.d.smithi154.stderr: 4: abort() 2024-09-16T01:39:28.117 INFO:tasks.ceph.mds.d.smithi154.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x190) [0x7f8a951834d4] 2024-09-16T01:39:28.117 INFO:tasks.ceph.mds.d.smithi154.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x61) [0x561b4a1472d1] 2024-09-16T01:39:28.117 INFO:tasks.ceph.mds.d.smithi154.stderr: 7: (CDentry::check_corruption(bool)+0x4d4) [0x561b4a3724e4] 2024-09-16T01:39:28.117 INFO:tasks.ceph.mds.d.smithi154.stderr: 8: ceph-mds(+0x1cb3e9) [0x561b4a18a3e9] 2024-09-16T01:39:28.117 INFO:tasks.ceph.mds.d.smithi154.stderr: 9: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x207f) [0x561b4a2152df] 2024-09-16T01:39:28.118 INFO:tasks.ceph.mds.d.smithi154.stderr: 10: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x25e2) [0x561b4a2185b2] 2024-09-16T01:39:28.118 INFO:tasks.ceph.mds.d.smithi154.stderr: 11: (Server::handle_client_request(boost::intrusive_ptr const&)+0x3e3) [0x561b4a1dedc3] 2024-09-16T01:39:28.118 INFO:tasks.ceph.mds.d.smithi154.stderr: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x13c) [0x561b4a1cccec] 2024-09-16T01:39:28.118 INFO:tasks.ceph.mds.d.smithi154.stderr: 13: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x561b4a14e89b] 2024-09-16T01:39:28.118 INFO:tasks.ceph.mds.d.smithi154.stderr: 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561b4a14ed6c] 2024-09-16T01:39:28.118 INFO:tasks.ceph.mds.d.smithi154.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1a6) [0x561b4a133286] 2024-09-16T01:39:28.118 INFO:tasks.ceph.mds.d.smithi154.stderr: 16: (DispatchQueue::entry()+0x4c2) [0x7f8a953905c2] 2024-09-16T01:39:28.118 INFO:tasks.ceph.mds.d.smithi154.stderr: 17: /usr/lib64/ceph/libceph-common.so.2(+0x42a1d1) [0x7f8a9542a1d1] 2024-09-16T01:39:28.118 INFO:tasks.ceph.mds.d.smithi154.stderr: 18: /lib64/libc.so.6(+0x89c52) [0x7f8a94889c52] 2024-09-16T01:39:28.119 INFO:tasks.ceph.mds.d.smithi154.stderr: 19: /lib64/libc.so.6(+0x10ec80) [0x7f8a9490ec80] 2024-09-16T01:39:28.119 INFO:tasks.ceph.mds.d.smithi154.stderr:2024-09-16T01:39:28.113+0000 7f8a8f4d6640 -1 *** Caught signal (Aborted) ** 2024-09-16T01:39:28.119 INFO:tasks.ceph.mds.d.smithi154.stderr: in thread 7f8a8f4d6640 thread_name:ms_dispatch 2024-09-16T01:39:28.119 INFO:tasks.ceph.mds.d.smithi154.stderr: 2024-09-16T01:39:28.119 INFO:tasks.ceph.mds.d.smithi154.stderr: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T01:39:28.119 INFO:tasks.ceph.mds.d.smithi154.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f8a9483e6f0] 2024-09-16T01:39:28.120 INFO:tasks.ceph.mds.d.smithi154.stderr: 2: /lib64/libc.so.6(+0x8b99c) [0x7f8a9488b99c] 2024-09-16T01:39:28.120 INFO:tasks.ceph.mds.d.smithi154.stderr: 3: raise() 2024-09-16T01:39:28.120 INFO:tasks.ceph.mds.d.smithi154.stderr: 4: abort() 2024-09-16T01:39:28.120 INFO:tasks.ceph.mds.d.smithi154.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x190) [0x7f8a951834d4] 2024-09-16T01:39:28.120 INFO:tasks.ceph.mds.d.smithi154.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x61) [0x561b4a1472d1] 2024-09-16T01:39:28.120 INFO:tasks.ceph.mds.d.smithi154.stderr: 7: (CDentry::check_corruption(bool)+0x4d4) [0x561b4a3724e4] 2024-09-16T01:39:28.120 INFO:tasks.ceph.mds.d.smithi154.stderr: 8: ceph-mds(+0x1cb3e9) [0x561b4a18a3e9] 2024-09-16T01:39:28.120 INFO:tasks.ceph.mds.d.smithi154.stderr: 9: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x207f) [0x561b4a2152df] 2024-09-16T01:39:28.120 INFO:tasks.ceph.mds.d.smithi154.stderr: 10: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x25e2) [0x561b4a2185b2] 2024-09-16T01:39:28.120 INFO:tasks.ceph.mds.d.smithi154.stderr: 11: (Server::handle_client_request(boost::intrusive_ptr const&)+0x3e3) [0x561b4a1dedc3] 2024-09-16T01:39:28.120 INFO:tasks.ceph.mds.d.smithi154.stderr: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x13c) [0x561b4a1cccec] 2024-09-16T01:39:28.120 INFO:tasks.ceph.mds.d.smithi154.stderr: 13: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x561b4a14e89b] 2024-09-16T01:39:28.120 INFO:tasks.ceph.mds.d.smithi154.stderr: 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561b4a14ed6c] 2024-09-16T01:39:28.121 INFO:tasks.ceph.mds.d.smithi154.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1a6) [0x561b4a133286] 2024-09-16T01:39:28.121 INFO:tasks.ceph.mds.d.smithi154.stderr: 16: (DispatchQueue::entry()+0x4c2) [0x7f8a953905c2] 2024-09-16T01:39:28.121 INFO:tasks.ceph.mds.d.smithi154.stderr: 17: /usr/lib64/ceph/libceph-common.so.2(+0x42a1d1) [0x7f8a9542a1d1] 2024-09-16T01:39:28.121 INFO:tasks.ceph.mds.d.smithi154.stderr: 18: /lib64/libc.so.6(+0x89c52) [0x7f8a94889c52] 2024-09-16T01:39:28.121 INFO:tasks.ceph.mds.d.smithi154.stderr: 19: /lib64/libc.so.6(+0x10ec80) [0x7f8a9490ec80] 2024-09-16T01:39:28.121 INFO:tasks.ceph.mds.d.smithi154.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-09-16T01:39:28.121 INFO:tasks.ceph.mds.d.smithi154.stderr: 2024-09-16T01:39:28.121 INFO:tasks.ceph.mds.d.smithi154.stderr: -4082> 2024-09-16T01:39:16.730+0000 7f8a8ccd1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:39:28.124 INFO:tasks.ceph.mds.d.smithi154.stderr: -8> 2024-09-16T01:39:28.111+0000 7f8a8f4d6640 -1 mds.0.cache.den(0x10000000000 z) newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.6564:33 nref=7 cr=0x561b4d221c00)) (dversion lock w=1 last_client=6564) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x561b4d3c7680] 2024-09-16T01:39:28.124 INFO:tasks.ceph.mds.d.smithi154.stderr: -6> 2024-09-16T01:39:28.111+0000 7f8a8f4d6640 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.6564:33 nref=7 cr=0x561b4d221c00)) (dversion lock w=1 last_client=6564) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x561b4d3c7680] 2024-09-16T01:39:28.124 INFO:tasks.ceph.mds.d.smithi154.stderr: -1> 2024-09-16T01:39:28.112+0000 7f8a8f4d6640 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-326-g8921c8ef/rpm/el9/BUILD/ceph-19.1.1-326-g8921c8ef/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f8a8f4d6640 time 2024-09-16T01:39:28.112463+0000 2024-09-16T01:39:28.124 INFO:tasks.ceph.mds.d.smithi154.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-326-g8921c8ef/rpm/el9/BUILD/ceph-19.1.1-326-g8921c8ef/src/mds/MDSRank.cc: 952: ceph_abort_msg("abort() called") 2024-09-16T01:39:28.124 INFO:tasks.ceph.mds.d.smithi154.stderr: 2024-09-16T01:39:28.124 INFO:tasks.ceph.mds.d.smithi154.stderr: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T01:39:28.124 INFO:tasks.ceph.mds.d.smithi154.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f8a95183417] 2024-09-16T01:39:28.124 INFO:tasks.ceph.mds.d.smithi154.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x61) [0x561b4a1472d1] 2024-09-16T01:39:28.124 INFO:tasks.ceph.mds.d.smithi154.stderr: 3: (CDentry::check_corruption(bool)+0x4d4) [0x561b4a3724e4] 2024-09-16T01:39:28.124 INFO:tasks.ceph.mds.d.smithi154.stderr: 4: ceph-mds(+0x1cb3e9) [0x561b4a18a3e9] 2024-09-16T01:39:28.124 INFO:tasks.ceph.mds.d.smithi154.stderr: 5: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x207f) [0x561b4a2152df] 2024-09-16T01:39:28.125 INFO:tasks.ceph.mds.d.smithi154.stderr: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x25e2) [0x561b4a2185b2] 2024-09-16T01:39:28.125 INFO:tasks.ceph.mds.d.smithi154.stderr: 7: (Server::handle_client_request(boost::intrusive_ptr const&)+0x3e3) [0x561b4a1dedc3] 2024-09-16T01:39:28.125 INFO:tasks.ceph.mds.d.smithi154.stderr: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x13c) [0x561b4a1cccec] 2024-09-16T01:39:28.125 INFO:tasks.ceph.mds.d.smithi154.stderr: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x561b4a14e89b] 2024-09-16T01:39:28.125 INFO:tasks.ceph.mds.d.smithi154.stderr: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561b4a14ed6c] 2024-09-16T01:39:28.125 INFO:tasks.ceph.mds.d.smithi154.stderr: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1a6) [0x561b4a133286] 2024-09-16T01:39:28.125 INFO:tasks.ceph.mds.d.smithi154.stderr: 12: (DispatchQueue::entry()+0x4c2) [0x7f8a953905c2] 2024-09-16T01:39:28.125 INFO:tasks.ceph.mds.d.smithi154.stderr: 13: /usr/lib64/ceph/libceph-common.so.2(+0x42a1d1) [0x7f8a9542a1d1] 2024-09-16T01:39:28.125 INFO:tasks.ceph.mds.d.smithi154.stderr: 14: /lib64/libc.so.6(+0x89c52) [0x7f8a94889c52] 2024-09-16T01:39:28.125 INFO:tasks.ceph.mds.d.smithi154.stderr: 15: /lib64/libc.so.6(+0x10ec80) [0x7f8a9490ec80] 2024-09-16T01:39:28.125 INFO:tasks.ceph.mds.d.smithi154.stderr: 2024-09-16T01:39:28.125 INFO:tasks.ceph.mds.d.smithi154.stderr: 0> 2024-09-16T01:39:28.113+0000 7f8a8f4d6640 -1 *** Caught signal (Aborted) ** 2024-09-16T01:39:28.125 INFO:tasks.ceph.mds.d.smithi154.stderr: in thread 7f8a8f4d6640 thread_name:ms_dispatch 2024-09-16T01:39:28.126 INFO:tasks.ceph.mds.d.smithi154.stderr: 2024-09-16T01:39:28.126 INFO:tasks.ceph.mds.d.smithi154.stderr: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T01:39:28.126 INFO:tasks.ceph.mds.d.smithi154.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f8a9483e6f0] 2024-09-16T01:39:28.126 INFO:tasks.ceph.mds.d.smithi154.stderr: 2: /lib64/libc.so.6(+0x8b99c) [0x7f8a9488b99c] 2024-09-16T01:39:28.126 INFO:tasks.ceph.mds.d.smithi154.stderr: 3: raise() 2024-09-16T01:39:28.126 INFO:tasks.ceph.mds.d.smithi154.stderr: 4: abort() 2024-09-16T01:39:28.126 INFO:tasks.ceph.mds.d.smithi154.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x190) [0x7f8a951834d4] 2024-09-16T01:39:28.126 INFO:tasks.ceph.mds.d.smithi154.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x61) [0x561b4a1472d1] 2024-09-16T01:39:28.126 INFO:tasks.ceph.mds.d.smithi154.stderr: 7: (CDentry::check_corruption(bool)+0x4d4) [0x561b4a3724e4] 2024-09-16T01:39:28.126 INFO:tasks.ceph.mds.d.smithi154.stderr: 8: ceph-mds(+0x1cb3e9) [0x561b4a18a3e9] 2024-09-16T01:39:28.126 INFO:tasks.ceph.mds.d.smithi154.stderr: 9: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x207f) [0x561b4a2152df] 2024-09-16T01:39:28.126 INFO:tasks.ceph.mds.d.smithi154.stderr: 10: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x25e2) [0x561b4a2185b2] 2024-09-16T01:39:28.126 INFO:tasks.ceph.mds.d.smithi154.stderr: 11: (Server::handle_client_request(boost::intrusive_ptr const&)+0x3e3) [0x561b4a1dedc3] 2024-09-16T01:39:28.126 INFO:tasks.ceph.mds.d.smithi154.stderr: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x13c) [0x561b4a1cccec] 2024-09-16T01:39:28.126 INFO:tasks.ceph.mds.d.smithi154.stderr: 13: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x561b4a14e89b] 2024-09-16T01:39:28.127 INFO:tasks.ceph.mds.d.smithi154.stderr: 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561b4a14ed6c] 2024-09-16T01:39:28.127 INFO:tasks.ceph.mds.d.smithi154.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1a6) [0x561b4a133286] 2024-09-16T01:39:28.127 INFO:tasks.ceph.mds.d.smithi154.stderr: 16: (DispatchQueue::entry()+0x4c2) [0x7f8a953905c2] 2024-09-16T01:39:28.127 INFO:tasks.ceph.mds.d.smithi154.stderr: 17: /usr/lib64/ceph/libceph-common.so.2(+0x42a1d1) [0x7f8a9542a1d1] 2024-09-16T01:39:28.127 INFO:tasks.ceph.mds.d.smithi154.stderr: 18: /lib64/libc.so.6(+0x89c52) [0x7f8a94889c52] 2024-09-16T01:39:28.127 INFO:tasks.ceph.mds.d.smithi154.stderr: 19: /lib64/libc.so.6(+0x10ec80) [0x7f8a9490ec80] 2024-09-16T01:39:28.127 INFO:tasks.ceph.mds.d.smithi154.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-09-16T01:39:28.127 INFO:tasks.ceph.mds.d.smithi154.stderr: 2024-09-16T01:39:28.127 INFO:tasks.ceph.mds.d.smithi154.stderr: -4082> 2024-09-16T01:39:16.730+0000 7f8a8ccd1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:39:28.129 INFO:tasks.ceph.mds.d.smithi154.stderr: -8> 2024-09-16T01:39:28.111+0000 7f8a8f4d6640 -1 mds.0.cache.den(0x10000000000 z) newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.6564:33 nref=7 cr=0x561b4d221c00)) (dversion lock w=1 last_client=6564) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x561b4d3c7680] 2024-09-16T01:39:28.129 INFO:tasks.ceph.mds.d.smithi154.stderr: -6> 2024-09-16T01:39:28.111+0000 7f8a8f4d6640 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.6564:33 nref=7 cr=0x561b4d221c00)) (dversion lock w=1 last_client=6564) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x561b4d3c7680] 2024-09-16T01:39:28.129 INFO:tasks.ceph.mds.d.smithi154.stderr: -1> 2024-09-16T01:39:28.112+0000 7f8a8f4d6640 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-326-g8921c8ef/rpm/el9/BUILD/ceph-19.1.1-326-g8921c8ef/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f8a8f4d6640 time 2024-09-16T01:39:28.112463+0000 2024-09-16T01:39:28.130 INFO:tasks.ceph.mds.d.smithi154.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.1-326-g8921c8ef/rpm/el9/BUILD/ceph-19.1.1-326-g8921c8ef/src/mds/MDSRank.cc: 952: ceph_abort_msg("abort() called") 2024-09-16T01:39:28.130 INFO:tasks.ceph.mds.d.smithi154.stderr: 2024-09-16T01:39:28.130 INFO:tasks.ceph.mds.d.smithi154.stderr: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T01:39:28.130 INFO:tasks.ceph.mds.d.smithi154.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f8a95183417] 2024-09-16T01:39:28.130 INFO:tasks.ceph.mds.d.smithi154.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x61) [0x561b4a1472d1] 2024-09-16T01:39:28.130 INFO:tasks.ceph.mds.d.smithi154.stderr: 3: (CDentry::check_corruption(bool)+0x4d4) [0x561b4a3724e4] 2024-09-16T01:39:28.130 INFO:tasks.ceph.mds.d.smithi154.stderr: 4: ceph-mds(+0x1cb3e9) [0x561b4a18a3e9] 2024-09-16T01:39:28.130 INFO:tasks.ceph.mds.d.smithi154.stderr: 5: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x207f) [0x561b4a2152df] 2024-09-16T01:39:28.130 INFO:tasks.ceph.mds.d.smithi154.stderr: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x25e2) [0x561b4a2185b2] 2024-09-16T01:39:28.130 INFO:tasks.ceph.mds.d.smithi154.stderr: 7: (Server::handle_client_request(boost::intrusive_ptr const&)+0x3e3) [0x561b4a1dedc3] 2024-09-16T01:39:28.130 INFO:tasks.ceph.mds.d.smithi154.stderr: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x13c) [0x561b4a1cccec] 2024-09-16T01:39:28.130 INFO:tasks.ceph.mds.d.smithi154.stderr: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x561b4a14e89b] 2024-09-16T01:39:28.131 INFO:tasks.ceph.mds.d.smithi154.stderr: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561b4a14ed6c] 2024-09-16T01:39:28.131 INFO:tasks.ceph.mds.d.smithi154.stderr: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1a6) [0x561b4a133286] 2024-09-16T01:39:28.131 INFO:tasks.ceph.mds.d.smithi154.stderr: 12: (DispatchQueue::entry()+0x4c2) [0x7f8a953905c2] 2024-09-16T01:39:28.131 INFO:tasks.ceph.mds.d.smithi154.stderr: 13: /usr/lib64/ceph/libceph-common.so.2(+0x42a1d1) [0x7f8a9542a1d1] 2024-09-16T01:39:28.131 INFO:tasks.ceph.mds.d.smithi154.stderr: 14: /lib64/libc.so.6(+0x89c52) [0x7f8a94889c52] 2024-09-16T01:39:28.131 INFO:tasks.ceph.mds.d.smithi154.stderr: 15: /lib64/libc.so.6(+0x10ec80) [0x7f8a9490ec80] 2024-09-16T01:39:28.131 INFO:tasks.ceph.mds.d.smithi154.stderr: 2024-09-16T01:39:28.131 INFO:tasks.ceph.mds.d.smithi154.stderr: 0> 2024-09-16T01:39:28.113+0000 7f8a8f4d6640 -1 *** Caught signal (Aborted) ** 2024-09-16T01:39:28.131 INFO:tasks.ceph.mds.d.smithi154.stderr: in thread 7f8a8f4d6640 thread_name:ms_dispatch 2024-09-16T01:39:28.131 INFO:tasks.ceph.mds.d.smithi154.stderr: 2024-09-16T01:39:28.131 INFO:tasks.ceph.mds.d.smithi154.stderr: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-16T01:39:28.131 INFO:tasks.ceph.mds.d.smithi154.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f8a9483e6f0] 2024-09-16T01:39:28.131 INFO:tasks.ceph.mds.d.smithi154.stderr: 2: /lib64/libc.so.6(+0x8b99c) [0x7f8a9488b99c] 2024-09-16T01:39:28.131 INFO:tasks.ceph.mds.d.smithi154.stderr: 3: raise() 2024-09-16T01:39:28.131 INFO:tasks.ceph.mds.d.smithi154.stderr: 4: abort() 2024-09-16T01:39:28.132 INFO:tasks.ceph.mds.d.smithi154.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x190) [0x7f8a951834d4] 2024-09-16T01:39:28.132 INFO:tasks.ceph.mds.d.smithi154.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x61) [0x561b4a1472d1] 2024-09-16T01:39:28.132 INFO:tasks.ceph.mds.d.smithi154.stderr: 7: (CDentry::check_corruption(bool)+0x4d4) [0x561b4a3724e4] 2024-09-16T01:39:28.132 INFO:tasks.ceph.mds.d.smithi154.stderr: 8: ceph-mds(+0x1cb3e9) [0x561b4a18a3e9] 2024-09-16T01:39:28.132 INFO:tasks.ceph.mds.d.smithi154.stderr: 9: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x207f) [0x561b4a2152df] 2024-09-16T01:39:28.132 INFO:tasks.ceph.mds.d.smithi154.stderr: 10: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x25e2) [0x561b4a2185b2] 2024-09-16T01:39:28.132 INFO:tasks.ceph.mds.d.smithi154.stderr: 11: (Server::handle_client_request(boost::intrusive_ptr const&)+0x3e3) [0x561b4a1dedc3] 2024-09-16T01:39:28.132 INFO:tasks.ceph.mds.d.smithi154.stderr: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x13c) [0x561b4a1cccec] 2024-09-16T01:39:28.132 INFO:tasks.ceph.mds.d.smithi154.stderr: 13: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x561b4a14e89b] 2024-09-16T01:39:28.132 INFO:tasks.ceph.mds.d.smithi154.stderr: 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561b4a14ed6c] 2024-09-16T01:39:28.132 INFO:tasks.ceph.mds.d.smithi154.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1a6) [0x561b4a133286] 2024-09-16T01:39:28.132 INFO:tasks.ceph.mds.d.smithi154.stderr: 16: (DispatchQueue::entry()+0x4c2) [0x7f8a953905c2] 2024-09-16T01:39:28.132 INFO:tasks.ceph.mds.d.smithi154.stderr: 17: /usr/lib64/ceph/libceph-common.so.2(+0x42a1d1) [0x7f8a9542a1d1] 2024-09-16T01:39:28.132 INFO:tasks.ceph.mds.d.smithi154.stderr: 18: /lib64/libc.so.6(+0x89c52) [0x7f8a94889c52] 2024-09-16T01:39:28.132 INFO:tasks.ceph.mds.d.smithi154.stderr: 19: /lib64/libc.so.6(+0x10ec80) [0x7f8a9490ec80] 2024-09-16T01:39:28.133 INFO:tasks.ceph.mds.d.smithi154.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-09-16T01:39:28.133 INFO:tasks.ceph.mds.d.smithi154.stderr: 2024-09-16T01:39:28.173 INFO:teuthology.orchestra.run.smithi088.stdout:{"mds_beacon_grace":"15.000000"} 2024-09-16T01:39:28.173 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_beacon_grace": "15.000000" } 2024-09-16T01:39:28.174 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:39:28.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.184+0000 7f31e6776640 1 Processor -- start 2024-09-16T01:39:28.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.184+0000 7f31e6776640 1 -- start start 2024-09-16T01:39:28.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.184+0000 7f31e6776640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31e00b7000 0x7f31e00b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:28.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.184+0000 7f31e6776640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f31e00b7460 0x7f31e00b5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:28.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.185+0000 7f31e6776640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31e0073360 con 0x7f31e00b44e0 2024-09-16T01:39:28.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.185+0000 7f31e6776640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f31e0073640 con 0x7f31e00b7460 2024-09-16T01:39:28.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.185+0000 7f31e6776640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f31e0073920 con 0x7f31e00b7000 2024-09-16T01:39:28.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.185+0000 7f31e4d13640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f31e00b7460 0x7f31e00b5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:28.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.185+0000 7f31e4d13640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f31e00b7460 0x7f31e00b5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49370/0 (socket says 172.21.15.88:49370) 2024-09-16T01:39:28.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.185+0000 7f31e4d13640 1 -- 172.21.15.88:0/735435644 learned_addr learned my addr 172.21.15.88:0/735435644 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:28.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.185+0000 7f31df7fe640 1 --2- 172.21.15.88:0/735435644 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31e00b7000 0x7f31e00b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:28.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.185+0000 7f31e4d13640 1 -- 172.21.15.88:0/735435644 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31e00b7000 msgr2=0x7f31e00b54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:28.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.185+0000 7f31e4d13640 1 --2- 172.21.15.88:0/735435644 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31e00b7000 0x7f31e00b54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:28.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.185+0000 7f31e4d13640 1 -- 172.21.15.88:0/735435644 >> v1:172.21.15.154:6789/0 conn(0x7f31e00b44e0 legacy=0x7f31e00b48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:28.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.185+0000 7f31e4d13640 1 -- 172.21.15.88:0/735435644 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31e0073c00 con 0x7f31e00b7460 2024-09-16T01:39:28.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.186+0000 7f31e4d13640 1 --2- 172.21.15.88:0/735435644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31e00b7460 0x7f31e00b5a10 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f31d8009e50 tx=0x7f31d802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3140f59e7a9f0d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:28.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.186+0000 7f31deffd640 1 -- 172.21.15.88:0/735435644 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d8036070 con 0x7f31e00b7460 2024-09-16T01:39:28.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.186+0000 7f31deffd640 1 -- 172.21.15.88:0/735435644 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31d8031600 con 0x7f31e00b7460 2024-09-16T01:39:28.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.186+0000 7f31deffd640 1 -- 172.21.15.88:0/735435644 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d80319d0 con 0x7f31e00b7460 2024-09-16T01:39:28.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.186+0000 7f31e6776640 1 -- 172.21.15.88:0/735435644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31e00b7460 msgr2=0x7f31e00b5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:28.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.186+0000 7f31e6776640 1 --2- 172.21.15.88:0/735435644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31e00b7460 0x7f31e00b5a10 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f31d8009e50 tx=0x7f31d802a710 comp rx=0 tx=0).stop 2024-09-16T01:39:28.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.187+0000 7f31e6776640 1 -- 172.21.15.88:0/735435644 shutdown_connections 2024-09-16T01:39:28.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.187+0000 7f31e6776640 1 --2- 172.21.15.88:0/735435644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31e00b7460 0x7f31e00b5a10 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:28.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.187+0000 7f31e6776640 1 --2- 172.21.15.88:0/735435644 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31e00b7000 0x7f31e00b54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:28.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.187+0000 7f31e6776640 1 -- 172.21.15.88:0/735435644 >> 172.21.15.88:0/735435644 conn(0x7f31e00b79f0 msgr2=0x7f31e00b7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:28.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.187+0000 7f31e6776640 1 -- 172.21.15.88:0/735435644 shutdown_connections 2024-09-16T01:39:28.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.187+0000 7f31e6776640 1 -- 172.21.15.88:0/735435644 wait complete. 2024-09-16T01:39:28.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.188+0000 7f31e6776640 1 Processor -- start 2024-09-16T01:39:28.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.188+0000 7f31e6776640 1 -- start start 2024-09-16T01:39:28.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.189+0000 7f31e6776640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31e00b44e0 0x7f31e01f1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:28.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.189+0000 7f31e6776640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31e00b7000 0x7f31e015ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:28.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.189+0000 7f31e6776640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31e00b7460 0x7f31e0144ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:28.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.189+0000 7f31e6776640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f31e0156de0 con 0x7f31e00b44e0 2024-09-16T01:39:28.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.189+0000 7f31e6776640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f31e00d01d0 con 0x7f31e00b7460 2024-09-16T01:39:28.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.189+0000 7f31e6776640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f31e0156f60 con 0x7f31e00b7000 2024-09-16T01:39:28.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.189+0000 7f31dffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31e00b44e0 0x7f31e01f1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:28.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.189+0000 7f31df7fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31e00b7000 0x7f31e015ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:28.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.189+0000 7f31dffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31e00b44e0 0x7f31e01f1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49380/0 (socket says 172.21.15.88:49380) 2024-09-16T01:39:28.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.189+0000 7f31dffff640 1 -- 172.21.15.88:0/607276357 learned_addr learned my addr 172.21.15.88:0/607276357 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:28.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.189+0000 7f31e4d13640 1 --2- 172.21.15.88:0/607276357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31e00b7460 0x7f31e0144ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:28.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.189+0000 7f31dffff640 1 -- 172.21.15.88:0/607276357 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31e00b7000 msgr2=0x7f31e015ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:28.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.189+0000 7f31dffff640 1 --2- 172.21.15.88:0/607276357 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31e00b7000 0x7f31e015ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:28.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.189+0000 7f31dffff640 1 -- 172.21.15.88:0/607276357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31e00b7460 msgr2=0x7f31e0144ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:28.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.189+0000 7f31dffff640 1 --2- 172.21.15.88:0/607276357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31e00b7460 0x7f31e0144ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:28.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.189+0000 7f31dffff640 1 -- 172.21.15.88:0/607276357 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31e0145880 con 0x7f31e00b44e0 2024-09-16T01:39:28.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.189+0000 7f31e4d13640 1 --2- 172.21.15.88:0/607276357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31e00b7460 0x7f31e0144ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:39:28.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.190+0000 7f31dffff640 1 --2- 172.21.15.88:0/607276357 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31e00b44e0 0x7f31e01f1580 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f31d000c400 tx=0x7f31d000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:28.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.190+0000 7f31dcff9640 1 -- 172.21.15.88:0/607276357 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d00052d0 con 0x7f31e00b44e0 2024-09-16T01:39:28.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.190+0000 7f31dcff9640 1 -- 172.21.15.88:0/607276357 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31d0005490 con 0x7f31e00b44e0 2024-09-16T01:39:28.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.190+0000 7f31e6776640 1 -- 172.21.15.88:0/607276357 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31e0145b60 con 0x7f31e00b44e0 2024-09-16T01:39:28.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.190+0000 7f31dcff9640 1 -- 172.21.15.88:0/607276357 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d0007260 con 0x7f31e00b44e0 2024-09-16T01:39:28.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.190+0000 7f31e6776640 1 -- 172.21.15.88:0/607276357 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31e0146120 con 0x7f31e00b44e0 2024-09-16T01:39:28.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.191+0000 7f31e6776640 1 -- 172.21.15.88:0/607276357 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f31e00b6b60 con 0x7f31e00b44e0 2024-09-16T01:39:28.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.191+0000 7f31dcff9640 1 -- 172.21.15.88:0/607276357 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f31d0007400 con 0x7f31e00b44e0 2024-09-16T01:39:28.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.192+0000 7f31dcff9640 1 --2- 172.21.15.88:0/607276357 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f31cc06c2b0 0x7f31cc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:28.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.192+0000 7f31dcff9640 1 -- 172.21.15.88:0/607276357 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f31d008a830 con 0x7f31e00b44e0 2024-09-16T01:39:28.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.192+0000 7f31df7fe640 1 --2- 172.21.15.88:0/607276357 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f31cc06c2b0 0x7f31cc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:28.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.192+0000 7f31dcff9640 1 -- 172.21.15.88:0/607276357 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+511 (secure 0 0 0) 0x7f31d008f240 con 0x7f31e00b44e0 2024-09-16T01:39:28.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.192+0000 7f31e6776640 1 -- 172.21.15.88:0/607276357 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({log-info=0+}) -- 0x7f31e00d04f0 con 0x7f31e00b44e0 2024-09-16T01:39:28.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.192+0000 7f31df7fe640 1 --2- 172.21.15.88:0/607276357 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f31cc06c2b0 0x7f31cc06e770 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f31e00c8ba0 tx=0x7f31d4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:28.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.272+0000 7fe12bd52640 1 Processor -- start 2024-09-16T01:39:28.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.272+0000 7fe12bd52640 1 -- start start 2024-09-16T01:39:28.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.273+0000 7fe12bd52640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe1240b4920 0x7fe12415b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:28.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.273+0000 7fe12bd52640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1240cd780 0x7fe1240cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:28.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.273+0000 7fe12bd52640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1240ce4f0 con 0x7fe1240b4d00 2024-09-16T01:39:28.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.273+0000 7fe12bd52640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe1240cf100 con 0x7fe1240b4920 2024-09-16T01:39:28.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.273+0000 7fe12bd52640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe1240ce7d0 con 0x7fe1240cd780 2024-09-16T01:39:28.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.273+0000 7fe129aee640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe1240b4920 0x7fe12415b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:28.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.273+0000 7fe129aee640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe1240b4920 0x7fe12415b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:43202/0 (socket says 172.21.15.88:43202) 2024-09-16T01:39:28.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.273+0000 7fe129aee640 1 -- 172.21.15.88:0/3559697473 learned_addr learned my addr 172.21.15.88:0/3559697473 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:28.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.274+0000 7fe12a2ef640 1 --2- 172.21.15.88:0/3559697473 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1240cd780 0x7fe1240cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:28.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.274+0000 7fe129aee640 1 -- 172.21.15.88:0/3559697473 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1240cd780 msgr2=0x7fe1240cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:28.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.274+0000 7fe129aee640 1 --2- 172.21.15.88:0/3559697473 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1240cd780 0x7fe1240cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:28.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.274+0000 7fe129aee640 1 -- 172.21.15.88:0/3559697473 >> v1:172.21.15.154:6789/0 conn(0x7fe1240b4d00 legacy=0x7fe1240b5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:28.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.274+0000 7fe129aee640 1 -- 172.21.15.88:0/3559697473 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1240ceab0 con 0x7fe1240b4920 2024-09-16T01:39:28.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.274+0000 7fe12a2ef640 1 --2- 172.21.15.88:0/3559697473 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1240cd780 0x7fe1240cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:28.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.274+0000 7fe129aee640 1 --2- 172.21.15.88:0/3559697473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1240b4920 0x7fe12415b7d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe120009870 tx=0x7fe12002a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f1082e66dc3323d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:28.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.275+0000 7fe128aec640 1 -- 172.21.15.88:0/3559697473 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe120037070 con 0x7fe1240b4920 2024-09-16T01:39:28.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.275+0000 7fe128aec640 1 -- 172.21.15.88:0/3559697473 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe120032440 con 0x7fe1240b4920 2024-09-16T01:39:28.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.275+0000 7fe128aec640 1 -- 172.21.15.88:0/3559697473 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe120032810 con 0x7fe1240b4920 2024-09-16T01:39:28.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.276+0000 7fe12bd52640 1 -- 172.21.15.88:0/3559697473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1240b4920 msgr2=0x7fe12415b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:28.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.276+0000 7fe12bd52640 1 --2- 172.21.15.88:0/3559697473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1240b4920 0x7fe12415b7d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fe120009870 tx=0x7fe12002a210 comp rx=0 tx=0).stop 2024-09-16T01:39:28.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.276+0000 7fe12bd52640 1 -- 172.21.15.88:0/3559697473 shutdown_connections 2024-09-16T01:39:28.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.276+0000 7fe12bd52640 1 --2- 172.21.15.88:0/3559697473 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1240cd780 0x7fe1240cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:28.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.276+0000 7fe12bd52640 1 --2- 172.21.15.88:0/3559697473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1240b4920 0x7fe12415b7d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:28.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.276+0000 7fe12bd52640 1 -- 172.21.15.88:0/3559697473 >> 172.21.15.88:0/3559697473 conn(0x7fe12406bc10 msgr2=0x7fe12406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:28.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.276+0000 7fe12bd52640 1 -- 172.21.15.88:0/3559697473 shutdown_connections 2024-09-16T01:39:28.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.276+0000 7fe12bd52640 1 -- 172.21.15.88:0/3559697473 wait complete. 2024-09-16T01:39:28.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.277+0000 7fe12bd52640 1 Processor -- start 2024-09-16T01:39:28.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.277+0000 7fe12bd52640 1 -- start start 2024-09-16T01:39:28.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.278+0000 7fe12bd52640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1240b4920 0x7fe124144110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:28.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.278+0000 7fe12bd52640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1240b4d00 0x7fe1241e5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:28.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.278+0000 7fe12bd52640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1240cd780 0x7fe12415a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:28.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.278+0000 7fe12bd52640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe1240ce110 con 0x7fe1240b4d00 2024-09-16T01:39:28.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.278+0000 7fe12bd52640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe12415bd10 con 0x7fe1240b4920 2024-09-16T01:39:28.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.278+0000 7fe12bd52640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe1240ce290 con 0x7fe1240cd780 2024-09-16T01:39:28.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.278+0000 7fe1292ed640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1240b4d00 0x7fe1241e5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:28.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.278+0000 7fe1292ed640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1240b4d00 0x7fe1241e5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49384/0 (socket says 172.21.15.88:49384) 2024-09-16T01:39:28.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.278+0000 7fe1292ed640 1 -- 172.21.15.88:0/3397793487 learned_addr learned my addr 172.21.15.88:0/3397793487 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:28.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.278+0000 7fe12a2ef640 1 --2- 172.21.15.88:0/3397793487 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1240cd780 0x7fe12415a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:28.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.278+0000 7fe129aee640 1 --2- 172.21.15.88:0/3397793487 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1240b4920 0x7fe124144110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:28.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.278+0000 7fe1292ed640 1 -- 172.21.15.88:0/3397793487 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1240cd780 msgr2=0x7fe12415a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:28.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.278+0000 7fe1292ed640 1 --2- 172.21.15.88:0/3397793487 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1240cd780 0x7fe12415a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:28.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.278+0000 7fe1292ed640 1 -- 172.21.15.88:0/3397793487 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1240b4920 msgr2=0x7fe124144110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:28.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.278+0000 7fe1292ed640 1 --2- 172.21.15.88:0/3397793487 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1240b4920 0x7fe124144110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:28.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.278+0000 7fe1292ed640 1 -- 172.21.15.88:0/3397793487 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1241e6460 con 0x7fe1240b4d00 2024-09-16T01:39:28.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.278+0000 7fe1292ed640 1 --2- 172.21.15.88:0/3397793487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1240b4d00 0x7fe1241e5f20 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fe11400c430 tx=0x7fe11400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:28.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.278+0000 7fe11a7fc640 1 -- 172.21.15.88:0/3397793487 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe114011790 con 0x7fe1240b4d00 2024-09-16T01:39:28.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.278+0000 7fe11a7fc640 1 -- 172.21.15.88:0/3397793487 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe114017070 con 0x7fe1240b4d00 2024-09-16T01:39:28.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.278+0000 7fe12bd52640 1 -- 172.21.15.88:0/3397793487 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1241e4e80 con 0x7fe1240b4d00 2024-09-16T01:39:28.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.278+0000 7fe11a7fc640 1 -- 172.21.15.88:0/3397793487 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe114011bb0 con 0x7fe1240b4d00 2024-09-16T01:39:28.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.278+0000 7fe12bd52640 1 -- 172.21.15.88:0/3397793487 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1241e5440 con 0x7fe1240b4d00 2024-09-16T01:39:28.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.279+0000 7fe12bd52640 1 -- 172.21.15.88:0/3397793487 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1240cd370 con 0x7fe1240b4d00 2024-09-16T01:39:28.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.280+0000 7fe11a7fc640 1 -- 172.21.15.88:0/3397793487 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe11401c460 con 0x7fe1240b4d00 2024-09-16T01:39:28.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.280+0000 7fe11a7fc640 1 --2- 172.21.15.88:0/3397793487 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe10806c2e0 0x7fe10806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:28.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.281+0000 7fe129aee640 1 --2- 172.21.15.88:0/3397793487 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe10806c2e0 0x7fe10806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:28.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.281+0000 7fe11a7fc640 1 -- 172.21.15.88:0/3397793487 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7fe11408b0b0 con 0x7fe1240b4d00 2024-09-16T01:39:28.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.281+0000 7fe129aee640 1 --2- 172.21.15.88:0/3397793487 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe10806c2e0 0x7fe10806e7a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe120034000 tx=0x7fe12002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:28.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.285+0000 7fe11a7fc640 1 -- 172.21.15.88:0/3397793487 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe1240cd370 con 0x7fe1240b4d00 2024-09-16T01:39:28.411 INFO:tasks.ceph.mds.d.smithi154.stderr:daemon-helper: command crashed with signal 6 2024-09-16T01:39:28.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.488+0000 7fe12bd52640 1 -- 172.21.15.88:0/3397793487 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1240c5750 con 0x7fe1240b4d00 2024-09-16T01:39:28.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.489+0000 7fe11a7fc640 1 -- 172.21.15.88:0/3397793487 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+5231 (secure 0 0 0) 0x7fe114059260 con 0x7fe1240b4d00 2024-09-16T01:39:28.490 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:28.491 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":58,"btime":"2024-09-16T01:39:22:503461+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5725,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2696547723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2696547723},{"type":"v1","addr":"172.21.15.154:6834","nonce":2696547723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54},{"gid":6411,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3276199399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3276199399},{"type":"v1","addr":"172.21.15.88:6836","nonce":3276199399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54},{"gid":6414,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2944989249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2944989249},{"type":"v1","addr":"172.21.15.88:6837","nonce":2944989249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:39:14.808422+0000","modified":"2024-09-16T01:39:21.748489+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6429":{"gid":6429,"name":"d","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6835/3482042494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":3482042494},{"type":"v1","addr":"172.21.15.154:6835","nonce":3482042494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6429,"qdb_cluster":[6429]},"id":5}]} 2024-09-16T01:39:28.491 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 58 2024-09-16T01:39:28.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.491+0000 7fe12bd52640 1 -- 172.21.15.88:0/3397793487 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe10806c2e0 msgr2=0x7fe10806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:28.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.491+0000 7fe12bd52640 1 --2- 172.21.15.88:0/3397793487 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe10806c2e0 0x7fe10806e7a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe120034000 tx=0x7fe12002e000 comp rx=0 tx=0).stop 2024-09-16T01:39:28.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.491+0000 7fe12bd52640 1 -- 172.21.15.88:0/3397793487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1240b4d00 msgr2=0x7fe1241e5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:28.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.491+0000 7fe12bd52640 1 --2- 172.21.15.88:0/3397793487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1240b4d00 0x7fe1241e5f20 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fe11400c430 tx=0x7fe11400c8f0 comp rx=0 tx=0).stop 2024-09-16T01:39:28.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.491+0000 7fe12bd52640 1 -- 172.21.15.88:0/3397793487 shutdown_connections 2024-09-16T01:39:28.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.491+0000 7fe12bd52640 1 --2- 172.21.15.88:0/3397793487 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe10806c2e0 0x7fe10806e7a0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:28.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.491+0000 7fe12bd52640 1 --2- 172.21.15.88:0/3397793487 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1240cd780 0x7fe12415a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:28.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.491+0000 7fe12bd52640 1 --2- 172.21.15.88:0/3397793487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe1240b4d00 0x7fe1241e5f20 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:28.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.491+0000 7fe12bd52640 1 --2- 172.21.15.88:0/3397793487 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1240b4920 0x7fe124144110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:28.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.491+0000 7fe12bd52640 1 -- 172.21.15.88:0/3397793487 >> 172.21.15.88:0/3397793487 conn(0x7fe12406bc10 msgr2=0x7fe12415c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:28.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.491+0000 7fe12bd52640 1 -- 172.21.15.88:0/3397793487 shutdown_connections 2024-09-16T01:39:28.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.491+0000 7fe12bd52640 1 -- 172.21.15.88:0/3397793487 wait complete. 2024-09-16T01:39:28.503 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-09-16T01:39:28.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:28.649+0000 7f31dcff9640 1 -- 172.21.15.88:0/607276357 <== mon.0 v2:172.21.15.88:3300/0 7 ==== log(1 entries from seq 1 at 2024-09-16T01:39:28.112405+0000) ==== 468+0+0 (secure 0 0 0) 0x7f31d0088e70 con 0x7f31e00b44e0 2024-09-16T01:39:28.651 INFO:teuthology.orchestra.run.smithi088.stdout: cluster: 2024-09-16T01:39:28.651 INFO:teuthology.orchestra.run.smithi088.stdout: id: 5ace2c91-46f9-482b-8a89-731fbdc171bf 2024-09-16T01:39:28.651 INFO:teuthology.orchestra.run.smithi088.stdout: health: HEALTH_OK 2024-09-16T01:39:28.651 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:28.651 INFO:teuthology.orchestra.run.smithi088.stdout: services: 2024-09-16T01:39:28.651 INFO:teuthology.orchestra.run.smithi088.stdout: mon: 3 daemons, quorum a,b,c (age 5m) 2024-09-16T01:39:28.651 INFO:teuthology.orchestra.run.smithi088.stdout: mgr: y(active, since 5m), standbys: x 2024-09-16T01:39:28.651 INFO:teuthology.orchestra.run.smithi088.stdout: mds: 1/1 daemons up, 3 standby 2024-09-16T01:39:28.651 INFO:teuthology.orchestra.run.smithi088.stdout: osd: 8 osds: 8 up (since 5m), 8 in (since 5m) 2024-09-16T01:39:28.652 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:28.652 INFO:teuthology.orchestra.run.smithi088.stdout: data: 2024-09-16T01:39:28.652 INFO:teuthology.orchestra.run.smithi088.stdout: volumes: 1/1 healthy 2024-09-16T01:39:28.652 INFO:teuthology.orchestra.run.smithi088.stdout: pools: 3 pools, 97 pgs 2024-09-16T01:39:28.652 INFO:teuthology.orchestra.run.smithi088.stdout: objects: 29 objects, 596 KiB 2024-09-16T01:39:28.653 INFO:teuthology.orchestra.run.smithi088.stdout: usage: 235 MiB used, 720 GiB / 720 GiB avail 2024-09-16T01:39:28.653 INFO:teuthology.orchestra.run.smithi088.stdout: pgs: 97 active+clean 2024-09-16T01:39:28.653 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:28.653 INFO:teuthology.orchestra.run.smithi088.stdout: io: 2024-09-16T01:39:28.653 INFO:teuthology.orchestra.run.smithi088.stdout: client: 0 B/s rd, 3.4 KiB/s wr, 0 op/s rd, 3 op/s wr 2024-09-16T01:39:28.654 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:28.654 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:28.654 INFO:teuthology.orchestra.run.smithi088.stdout:2024-09-16T01:39:28.112405+0000 mds.d [ERR] MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by request(client.6564:33 nref=7 cr=0x561b4d221c00)) (dversion lock w=1 last_client=6564) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x561b4d3c7680] 2024-09-16T01:39:33.188 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-09-16T01:39:33.504 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:39:33.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.613+0000 7ff82b18f640 1 Processor -- start 2024-09-16T01:39:33.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.613+0000 7ff82b18f640 1 -- start start 2024-09-16T01:39:33.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.614+0000 7ff82b18f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff82406e780 0x7ff8240ae890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:33.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.614+0000 7ff82b18f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8240b5220 con 0x7ff82415e560 2024-09-16T01:39:33.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.614+0000 7ff82b18f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8240b54c0 con 0x7ff824073130 2024-09-16T01:39:33.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.614+0000 7ff82b18f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff82406ebc0 con 0x7ff82406e780 2024-09-16T01:39:33.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.614+0000 7ff828f2b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff82415e560 0x7ff824072220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41174/0 (socket says 172.21.15.88:41174) 2024-09-16T01:39:33.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.614+0000 7ff828f2b640 1 -- 172.21.15.88:0/2994841810 learned_addr learned my addr 172.21.15.88:0/2994841810 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:33.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.614+0000 7ff823fff640 1 --2- 172.21.15.88:0/2994841810 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff82406e780 0x7ff8240ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:33.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.614+0000 7ff8237fe640 1 -- 172.21.15.88:0/2994841810 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3045233079 0 0) 0x7ff8240b5220 con 0x7ff82415e560 2024-09-16T01:39:33.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.614+0000 7ff8237fe640 1 -- 172.21.15.88:0/2994841810 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff810003520 con 0x7ff82415e560 2024-09-16T01:39:33.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.614+0000 7ff8237fe640 1 -- 172.21.15.88:0/2994841810 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1040621376 0 0) 0x7ff8240b54c0 con 0x7ff824073130 2024-09-16T01:39:33.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.614+0000 7ff8237fe640 1 -- 172.21.15.88:0/2994841810 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8240b5220 con 0x7ff824073130 2024-09-16T01:39:33.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.614+0000 7ff8237fe640 1 -- 172.21.15.88:0/2994841810 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 443591858 0 0) 0x7ff810003520 con 0x7ff82415e560 2024-09-16T01:39:33.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.614+0000 7ff823fff640 1 -- 172.21.15.88:0/2994841810 >> v1:172.21.15.154:6789/0 conn(0x7ff824073130 legacy=0x7ff8240b6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:33.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.614+0000 7ff823fff640 1 -- 172.21.15.88:0/2994841810 >> v1:172.21.15.88:6789/0 conn(0x7ff82415e560 legacy=0x7ff824072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:33.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.614+0000 7ff823fff640 1 -- 172.21.15.88:0/2994841810 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8240b57a0 con 0x7ff82406e780 2024-09-16T01:39:33.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.615+0000 7ff823fff640 1 --2- 172.21.15.88:0/2994841810 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff82406e780 0x7ff8240ae890 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff814009dc0 tx=0x7ff81402a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8a47572a7a6e8e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:33.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.615+0000 7ff8237fe640 1 -- 172.21.15.88:0/2994841810 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c001690 con 0x7ff82406e780 2024-09-16T01:39:33.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.616+0000 7ff8237fe640 1 -- 172.21.15.88:0/2994841810 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff814007540 con 0x7ff82406e780 2024-09-16T01:39:33.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.616+0000 7ff8237fe640 1 -- 172.21.15.88:0/2994841810 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff814007910 con 0x7ff82406e780 2024-09-16T01:39:33.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.616+0000 7ff82b18f640 1 -- 172.21.15.88:0/2994841810 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff82406e780 msgr2=0x7ff8240ae890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:33.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.616+0000 7ff82b18f640 1 --2- 172.21.15.88:0/2994841810 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff82406e780 0x7ff8240ae890 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff814009dc0 tx=0x7ff81402a5d0 comp rx=0 tx=0).stop 2024-09-16T01:39:33.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.616+0000 7ff82b18f640 1 -- 172.21.15.88:0/2994841810 shutdown_connections 2024-09-16T01:39:33.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.616+0000 7ff82b18f640 1 --2- 172.21.15.88:0/2994841810 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff82406e780 0x7ff8240ae890 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:33.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.616+0000 7ff82b18f640 1 -- 172.21.15.88:0/2994841810 >> 172.21.15.88:0/2994841810 conn(0x7ff82406bc10 msgr2=0x7ff82406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:33.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.616+0000 7ff82b18f640 1 -- 172.21.15.88:0/2994841810 shutdown_connections 2024-09-16T01:39:33.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.616+0000 7ff82b18f640 1 -- 172.21.15.88:0/2994841810 wait complete. 2024-09-16T01:39:33.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.617+0000 7ff82b18f640 1 Processor -- start 2024-09-16T01:39:33.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.618+0000 7ff82b18f640 1 -- start start 2024-09-16T01:39:33.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.618+0000 7ff82b18f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff82406e780 0x7ff82414d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:33.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.618+0000 7ff82b18f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff824073130 0x7ff8240ce610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:33.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.618+0000 7ff82b18f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff82415e560 0x7ff8240ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:33.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.618+0000 7ff82b18f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff82415e940 con 0x7ff82415e560 2024-09-16T01:39:33.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.618+0000 7ff82b18f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff8240b5d60 con 0x7ff82406e780 2024-09-16T01:39:33.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.618+0000 7ff82b18f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff82415eac0 con 0x7ff824073130 2024-09-16T01:39:33.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.619+0000 7ff82972c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff82415e560 0x7ff8240ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:33.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.619+0000 7ff82972c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff82415e560 0x7ff8240ceb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48318/0 (socket says 172.21.15.88:48318) 2024-09-16T01:39:33.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.619+0000 7ff82972c640 1 -- 172.21.15.88:0/3652746469 learned_addr learned my addr 172.21.15.88:0/3652746469 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:33.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.619+0000 7ff828f2b640 1 --2- 172.21.15.88:0/3652746469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff82406e780 0x7ff82414d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:33.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.619+0000 7ff823fff640 1 --2- 172.21.15.88:0/3652746469 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff824073130 0x7ff8240ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:33.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.619+0000 7ff82972c640 1 -- 172.21.15.88:0/3652746469 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff824073130 msgr2=0x7ff8240ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:33.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.619+0000 7ff82972c640 1 --2- 172.21.15.88:0/3652746469 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff824073130 0x7ff8240ce610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:33.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.619+0000 7ff82972c640 1 -- 172.21.15.88:0/3652746469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff82406e780 msgr2=0x7ff82414d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:33.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.619+0000 7ff82972c640 1 --2- 172.21.15.88:0/3652746469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff82406e780 0x7ff82414d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:33.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.619+0000 7ff82972c640 1 -- 172.21.15.88:0/3652746469 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8241511a0 con 0x7ff82415e560 2024-09-16T01:39:33.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.619+0000 7ff82972c640 1 --2- 172.21.15.88:0/3652746469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff82415e560 0x7ff8240ceb50 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7ff80c00c400 tx=0x7ff80c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:33.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.619+0000 7ff8217fa640 1 -- 172.21.15.88:0/3652746469 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c017070 con 0x7ff82415e560 2024-09-16T01:39:33.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.619+0000 7ff8217fa640 1 -- 172.21.15.88:0/3652746469 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff80c0052d0 con 0x7ff82415e560 2024-09-16T01:39:33.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.619+0000 7ff82b18f640 1 -- 172.21.15.88:0/3652746469 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8241559c0 con 0x7ff82415e560 2024-09-16T01:39:33.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.619+0000 7ff8217fa640 1 -- 172.21.15.88:0/3652746469 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80c0037f0 con 0x7ff82415e560 2024-09-16T01:39:33.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.619+0000 7ff82b18f640 1 -- 172.21.15.88:0/3652746469 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff824155f80 con 0x7ff82415e560 2024-09-16T01:39:33.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.620+0000 7ff82b18f640 1 -- 172.21.15.88:0/3652746469 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8240735a0 con 0x7ff82415e560 2024-09-16T01:39:33.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.621+0000 7ff8217fa640 1 -- 172.21.15.88:0/3652746469 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff80c01a070 con 0x7ff82415e560 2024-09-16T01:39:33.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.621+0000 7ff8217fa640 1 --2- 172.21.15.88:0/3652746469 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff80406c2b0 0x7ff80406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:33.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.621+0000 7ff8217fa640 1 -- 172.21.15.88:0/3652746469 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7ff80c08a7b0 con 0x7ff82415e560 2024-09-16T01:39:33.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.621+0000 7ff828f2b640 1 --2- 172.21.15.88:0/3652746469 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff80406c2b0 0x7ff80406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:33.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.622+0000 7ff828f2b640 1 --2- 172.21.15.88:0/3652746469 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff80406c2b0 0x7ff80406e770 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff81c001eb0 tx=0x7ff81c0016d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:33.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.626+0000 7ff8217fa640 1 -- 172.21.15.88:0/3652746469 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff80c058850 con 0x7ff82415e560 2024-09-16T01:39:33.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.824+0000 7ff82b18f640 1 -- 172.21.15.88:0/3652746469 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff82406d440 con 0x7ff82415e560 2024-09-16T01:39:33.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.825+0000 7ff8217fa640 1 -- 172.21.15.88:0/3652746469 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+5231 (secure 0 0 0) 0x7ff80c05cb50 con 0x7ff82415e560 2024-09-16T01:39:33.827 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:33.827 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":58,"btime":"2024-09-16T01:39:22:503461+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5725,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2696547723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2696547723},{"type":"v1","addr":"172.21.15.154:6834","nonce":2696547723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54},{"gid":6411,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3276199399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3276199399},{"type":"v1","addr":"172.21.15.88:6836","nonce":3276199399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54},{"gid":6414,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2944989249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2944989249},{"type":"v1","addr":"172.21.15.88:6837","nonce":2944989249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:39:14.808422+0000","modified":"2024-09-16T01:39:21.748489+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6429":{"gid":6429,"name":"d","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6835/3482042494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":3482042494},{"type":"v1","addr":"172.21.15.154:6835","nonce":3482042494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6429,"qdb_cluster":[6429]},"id":5}]} 2024-09-16T01:39:33.829 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 58 2024-09-16T01:39:33.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.827+0000 7ff82b18f640 1 -- 172.21.15.88:0/3652746469 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff80406c2b0 msgr2=0x7ff80406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:33.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.827+0000 7ff82b18f640 1 --2- 172.21.15.88:0/3652746469 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff80406c2b0 0x7ff80406e770 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff81c001eb0 tx=0x7ff81c0016d0 comp rx=0 tx=0).stop 2024-09-16T01:39:33.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.827+0000 7ff82b18f640 1 -- 172.21.15.88:0/3652746469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff82415e560 msgr2=0x7ff8240ceb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:33.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.827+0000 7ff82b18f640 1 --2- 172.21.15.88:0/3652746469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff82415e560 0x7ff8240ceb50 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7ff80c00c400 tx=0x7ff80c00c8c0 comp rx=0 tx=0).stop 2024-09-16T01:39:33.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.827+0000 7ff82b18f640 1 -- 172.21.15.88:0/3652746469 shutdown_connections 2024-09-16T01:39:33.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.827+0000 7ff82b18f640 1 --2- 172.21.15.88:0/3652746469 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff80406c2b0 0x7ff80406e770 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:33.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.827+0000 7ff82b18f640 1 --2- 172.21.15.88:0/3652746469 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff82415e560 0x7ff8240ceb50 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:33.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.827+0000 7ff82b18f640 1 --2- 172.21.15.88:0/3652746469 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff824073130 0x7ff8240ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:33.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.827+0000 7ff82b18f640 1 --2- 172.21.15.88:0/3652746469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff82406e780 0x7ff82414d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:33.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.827+0000 7ff82b18f640 1 -- 172.21.15.88:0/3652746469 >> 172.21.15.88:0/3652746469 conn(0x7ff82406bc10 msgr2=0x7ff82406c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:33.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.827+0000 7ff82b18f640 1 -- 172.21.15.88:0/3652746469 shutdown_connections 2024-09-16T01:39:33.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:33.827+0000 7ff82b18f640 1 -- 172.21.15.88:0/3652746469 wait complete. 2024-09-16T01:39:33.840 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-09-16T01:39:38.840 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:39:38.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.949+0000 7f675fd78640 1 Processor -- start 2024-09-16T01:39:38.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.949+0000 7f675fd78640 1 -- start start 2024-09-16T01:39:38.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.950+0000 7f675fd78640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f675815e950 0x7f675806ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:38.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.950+0000 7f675fd78640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f675806ca30 con 0x7f67580735b0 2024-09-16T01:39:38.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.950+0000 7f675fd78640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f675806faf0 con 0x7f6758072050 2024-09-16T01:39:38.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.950+0000 7f675fd78640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f675806fdd0 con 0x7f675815e950 2024-09-16T01:39:38.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.950+0000 7f675db14640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f675815e950 0x7f675806ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:38.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.950+0000 7f675d313640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f67580735b0 0x7f675806bc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41188/0 (socket says 172.21.15.88:41188) 2024-09-16T01:39:38.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.950+0000 7f675d313640 1 -- 172.21.15.88:0/3303088243 learned_addr learned my addr 172.21.15.88:0/3303088243 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:38.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.951+0000 7f675db14640 1 -- 172.21.15.88:0/3303088243 >> v1:172.21.15.154:6789/0 conn(0x7f6758072050 legacy=0x7f675806c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:38.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.951+0000 7f675db14640 1 -- 172.21.15.88:0/3303088243 >> v1:172.21.15.88:6789/0 conn(0x7f67580735b0 legacy=0x7f675806bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:38.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.951+0000 7f675cb12640 1 -- 172.21.15.88:0/3303088243 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2182135056 0 0) 0x7f675806ca30 con 0x7f67580735b0 2024-09-16T01:39:38.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.951+0000 7f675db14640 1 -- 172.21.15.88:0/3303088243 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67580700b0 con 0x7f675815e950 2024-09-16T01:39:38.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.951+0000 7f675db14640 1 --2- 172.21.15.88:0/3303088243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f675815e950 0x7f675806ce30 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f67540097a0 tx=0x7f675402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c819786d600035e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:38.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.951+0000 7f675cb12640 1 -- 172.21.15.88:0/3303088243 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6754037070 con 0x7f675815e950 2024-09-16T01:39:38.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.952+0000 7f675cb12640 1 -- 172.21.15.88:0/3303088243 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6754032440 con 0x7f675815e950 2024-09-16T01:39:38.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.952+0000 7f675cb12640 1 -- 172.21.15.88:0/3303088243 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6754032810 con 0x7f675815e950 2024-09-16T01:39:38.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.952+0000 7f675fd78640 1 -- 172.21.15.88:0/3303088243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f675815e950 msgr2=0x7f675806ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:38.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.952+0000 7f675fd78640 1 --2- 172.21.15.88:0/3303088243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f675815e950 0x7f675806ce30 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f67540097a0 tx=0x7f675402a230 comp rx=0 tx=0).stop 2024-09-16T01:39:38.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.952+0000 7f675fd78640 1 -- 172.21.15.88:0/3303088243 shutdown_connections 2024-09-16T01:39:38.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.952+0000 7f675fd78640 1 --2- 172.21.15.88:0/3303088243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f675815e950 0x7f675806ce30 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:38.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.952+0000 7f675fd78640 1 -- 172.21.15.88:0/3303088243 >> 172.21.15.88:0/3303088243 conn(0x7f67580d14d0 msgr2=0x7f67580d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:38.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.952+0000 7f675fd78640 1 -- 172.21.15.88:0/3303088243 shutdown_connections 2024-09-16T01:39:38.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.952+0000 7f675fd78640 1 -- 172.21.15.88:0/3303088243 wait complete. 2024-09-16T01:39:38.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.953+0000 7f675fd78640 1 Processor -- start 2024-09-16T01:39:38.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.953+0000 7f675fd78640 1 -- start start 2024-09-16T01:39:38.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.954+0000 7f675fd78640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6758072050 0x7f67581542d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:38.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.954+0000 7f675fd78640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67580735b0 0x7f67580cbe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:38.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.954+0000 7f675fd78640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f675815e950 0x7f6758159160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:38.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.954+0000 7f675fd78640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f675806f8b0 con 0x7f67580735b0 2024-09-16T01:39:38.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.954+0000 7f675fd78640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f675806f730 con 0x7f675815e950 2024-09-16T01:39:38.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.954+0000 7f675fd78640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f67580d00b0 con 0x7f6758072050 2024-09-16T01:39:38.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.954+0000 7f675d313640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67580735b0 0x7f67580cbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:38.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.954+0000 7f675d313640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67580735b0 0x7f67580cbe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:48328/0 (socket says 172.21.15.88:48328) 2024-09-16T01:39:38.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.954+0000 7f675d313640 1 -- 172.21.15.88:0/1769151236 learned_addr learned my addr 172.21.15.88:0/1769151236 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:38.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.954+0000 7f675db14640 1 --2- 172.21.15.88:0/1769151236 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6758072050 0x7f67581542d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:38.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.954+0000 7f675e315640 1 --2- 172.21.15.88:0/1769151236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f675815e950 0x7f6758159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:38.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.955+0000 7f675d313640 1 -- 172.21.15.88:0/1769151236 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6758072050 msgr2=0x7f67581542d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:38.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.955+0000 7f675d313640 1 --2- 172.21.15.88:0/1769151236 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6758072050 0x7f67581542d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:38.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.955+0000 7f675d313640 1 -- 172.21.15.88:0/1769151236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f675815e950 msgr2=0x7f6758159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:38.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.955+0000 7f675d313640 1 --2- 172.21.15.88:0/1769151236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f675815e950 0x7f6758159160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:38.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.955+0000 7f675d313640 1 -- 172.21.15.88:0/1769151236 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6758154810 con 0x7f67580735b0 2024-09-16T01:39:38.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.955+0000 7f675d313640 1 --2- 172.21.15.88:0/1769151236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67580735b0 0x7f67580cbe20 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f674800c460 tx=0x7f674800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:38.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.955+0000 7f674e7fc640 1 -- 172.21.15.88:0/1769151236 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6748018070 con 0x7f67580735b0 2024-09-16T01:39:38.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.955+0000 7f674e7fc640 1 -- 172.21.15.88:0/1769151236 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6748007520 con 0x7f67580735b0 2024-09-16T01:39:38.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.955+0000 7f675fd78640 1 -- 172.21.15.88:0/1769151236 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6758155ce0 con 0x7f67580735b0 2024-09-16T01:39:38.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.955+0000 7f674e7fc640 1 -- 172.21.15.88:0/1769151236 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6748004a20 con 0x7f67580735b0 2024-09-16T01:39:38.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.955+0000 7f675fd78640 1 -- 172.21.15.88:0/1769151236 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67581562a0 con 0x7f67580735b0 2024-09-16T01:39:38.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.956+0000 7f675fd78640 1 -- 172.21.15.88:0/1769151236 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f675806ce30 con 0x7f67580735b0 2024-09-16T01:39:38.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.957+0000 7f674e7fc640 1 -- 172.21.15.88:0/1769151236 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f674801b070 con 0x7f67580735b0 2024-09-16T01:39:38.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.957+0000 7f674e7fc640 1 --2- 172.21.15.88:0/1769151236 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f673c06c2b0 0x7f673c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:38.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.957+0000 7f675db14640 1 --2- 172.21.15.88:0/1769151236 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f673c06c2b0 0x7f673c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:38.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.957+0000 7f674e7fc640 1 -- 172.21.15.88:0/1769151236 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f674808a740 con 0x7f67580735b0 2024-09-16T01:39:38.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.958+0000 7f675db14640 1 --2- 172.21.15.88:0/1769151236 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f673c06c2b0 0x7f673c06e770 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6754034000 tx=0x7f675402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:38.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:38.962+0000 7f674e7fc640 1 -- 172.21.15.88:0/1769151236 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6758154810 con 0x7f67580735b0 2024-09-16T01:39:39.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:39.161+0000 7f675fd78640 1 -- 172.21.15.88:0/1769151236 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67580b6180 con 0x7f67580735b0 2024-09-16T01:39:39.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:39.162+0000 7f674e7fc640 1 -- 172.21.15.88:0/1769151236 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+5231 (secure 0 0 0) 0x7f6748058860 con 0x7f67580735b0 2024-09-16T01:39:39.164 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:39.164 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":58,"btime":"2024-09-16T01:39:22:503461+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5725,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2696547723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2696547723},{"type":"v1","addr":"172.21.15.154:6834","nonce":2696547723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54},{"gid":6411,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3276199399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3276199399},{"type":"v1","addr":"172.21.15.88:6836","nonce":3276199399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54},{"gid":6414,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2944989249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2944989249},{"type":"v1","addr":"172.21.15.88:6837","nonce":2944989249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:39:14.808422+0000","modified":"2024-09-16T01:39:21.748489+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6429":{"gid":6429,"name":"d","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6835/3482042494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":3482042494},{"type":"v1","addr":"172.21.15.154:6835","nonce":3482042494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6429,"qdb_cluster":[6429]},"id":5}]} 2024-09-16T01:39:39.165 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 58 2024-09-16T01:39:39.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:39.164+0000 7f675fd78640 1 -- 172.21.15.88:0/1769151236 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f673c06c2b0 msgr2=0x7f673c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:39.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:39.164+0000 7f675fd78640 1 --2- 172.21.15.88:0/1769151236 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f673c06c2b0 0x7f673c06e770 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6754034000 tx=0x7f675402e000 comp rx=0 tx=0).stop 2024-09-16T01:39:39.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:39.164+0000 7f675fd78640 1 -- 172.21.15.88:0/1769151236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67580735b0 msgr2=0x7f67580cbe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:39.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:39.164+0000 7f675fd78640 1 --2- 172.21.15.88:0/1769151236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67580735b0 0x7f67580cbe20 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f674800c460 tx=0x7f674800c920 comp rx=0 tx=0).stop 2024-09-16T01:39:39.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:39.164+0000 7f675fd78640 1 -- 172.21.15.88:0/1769151236 shutdown_connections 2024-09-16T01:39:39.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:39.164+0000 7f675fd78640 1 --2- 172.21.15.88:0/1769151236 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f673c06c2b0 0x7f673c06e770 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:39.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:39.164+0000 7f675fd78640 1 --2- 172.21.15.88:0/1769151236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f675815e950 0x7f6758159160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:39.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:39.164+0000 7f675fd78640 1 --2- 172.21.15.88:0/1769151236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67580735b0 0x7f67580cbe20 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:39.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:39.164+0000 7f675fd78640 1 --2- 172.21.15.88:0/1769151236 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6758072050 0x7f67581542d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:39.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:39.164+0000 7f675fd78640 1 -- 172.21.15.88:0/1769151236 >> 172.21.15.88:0/1769151236 conn(0x7f67580d14d0 msgr2=0x7f675815f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:39.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:39.164+0000 7f675fd78640 1 -- 172.21.15.88:0/1769151236 shutdown_connections 2024-09-16T01:39:39.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:39.164+0000 7f675fd78640 1 -- 172.21.15.88:0/1769151236 wait complete. 2024-09-16T01:39:39.177 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-09-16T01:39:39.800 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~7s 2024-09-16T01:39:44.178 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:39:44.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.287+0000 7faa98cf8640 1 Processor -- start 2024-09-16T01:39:44.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.287+0000 7faa98cf8640 1 -- start start 2024-09-16T01:39:44.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.288+0000 7faa98cf8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7faa94072d50 0x7faa940ae890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:44.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.288+0000 7faa98cf8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa940b52c0 con 0x7faa9415e540 2024-09-16T01:39:44.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.288+0000 7faa98cf8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa940af000 con 0x7faa9406e800 2024-09-16T01:39:44.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.288+0000 7faa98cf8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7faa940aedd0 con 0x7faa94072d50 2024-09-16T01:39:44.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.288+0000 7faa9259c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7faa9415e540 0x7faa9406e0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36128/0 (socket says 172.21.15.88:36128) 2024-09-16T01:39:44.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.288+0000 7faa9259c640 1 -- 172.21.15.88:0/1135936950 learned_addr learned my addr 172.21.15.88:0/1135936950 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:44.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa92d9d640 1 --2- 172.21.15.88:0/1135936950 >> v2:172.21.15.154:3300/0 conn(0x7faa94072d50 0x7faa940ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:44.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa9159a640 1 -- 172.21.15.88:0/1135936950 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3818552039 0 0) 0x7faa940b52c0 con 0x7faa9415e540 2024-09-16T01:39:44.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa9159a640 1 -- 172.21.15.88:0/1135936950 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa7c003520 con 0x7faa9415e540 2024-09-16T01:39:44.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa9159a640 1 -- 172.21.15.88:0/1135936950 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4140796986 0 0) 0x7faa940af000 con 0x7faa9406e800 2024-09-16T01:39:44.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa9159a640 1 -- 172.21.15.88:0/1135936950 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa940b52c0 con 0x7faa9406e800 2024-09-16T01:39:44.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa9159a640 1 -- 172.21.15.88:0/1135936950 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 123306997 0 0) 0x7faa7c003520 con 0x7faa9415e540 2024-09-16T01:39:44.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa9159a640 1 -- 172.21.15.88:0/1135936950 >> v2:172.21.15.154:3300/0 conn(0x7faa94072d50 msgr2=0x7faa940ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:44.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa9159a640 1 --2- 172.21.15.88:0/1135936950 >> v2:172.21.15.154:3300/0 conn(0x7faa94072d50 0x7faa940ae890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:44.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa9159a640 1 -- 172.21.15.88:0/1135936950 >> v1:172.21.15.154:6789/0 conn(0x7faa9406e800 legacy=0x7faa940b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:44.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa9159a640 1 -- 172.21.15.88:0/1135936950 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa940af2e0 con 0x7faa9415e540 2024-09-16T01:39:44.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa9159a640 1 -- 172.21.15.88:0/1135936950 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7faa84001670 con 0x7faa9415e540 2024-09-16T01:39:44.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa9159a640 1 -- 172.21.15.88:0/1135936950 >> v1:172.21.15.88:6789/0 conn(0x7faa9415e540 legacy=0x7faa9406e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:44.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa9159a640 1 --2- 172.21.15.88:0/1135936950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa7c003b90 0x7faa7c003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:44.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa9159a640 1 --2- 172.21.15.88:0/1135936950 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa7c004800 0x7faa7c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:44.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa9159a640 1 --2- 172.21.15.88:0/1135936950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa7c005210 0x7faa7c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:44.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa9159a640 1 -- 172.21.15.88:0/1135936950 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7faa7c00c110 con 0x7faa7c005210 2024-09-16T01:39:44.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa9159a640 1 -- 172.21.15.88:0/1135936950 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faa7c00c2c0 con 0x7faa7c003b90 2024-09-16T01:39:44.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa9159a640 1 -- 172.21.15.88:0/1135936950 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7faa7c00c4c0 con 0x7faa7c004800 2024-09-16T01:39:44.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa92d9d640 1 --2- 172.21.15.88:0/1135936950 >> v2:172.21.15.154:3300/0 conn(0x7faa94072d50 0x7faa940ae890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:39:44.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa92d9d640 1 --2- 172.21.15.88:0/1135936950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa7c005210 0x7faa7c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:44.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa9259c640 1 --2- 172.21.15.88:0/1135936950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa7c003b90 0x7faa7c003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:44.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa91d9b640 1 --2- 172.21.15.88:0/1135936950 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa7c004800 0x7faa7c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:44.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa92d9d640 1 -- 172.21.15.88:0/1135936950 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa7c004800 msgr2=0x7faa7c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:44.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa92d9d640 1 --2- 172.21.15.88:0/1135936950 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa7c004800 0x7faa7c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa92d9d640 1 -- 172.21.15.88:0/1135936950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa7c003b90 msgr2=0x7faa7c003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa92d9d640 1 --2- 172.21.15.88:0/1135936950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa7c003b90 0x7faa7c003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.289+0000 7faa92d9d640 1 -- 172.21.15.88:0/1135936950 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa7c003520 con 0x7faa7c005210 2024-09-16T01:39:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.290+0000 7faa92d9d640 1 -- 172.21.15.88:0/1135936950 reap_dead start 2024-09-16T01:39:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.290+0000 7faa92d9d640 1 --2- 172.21.15.88:0/1135936950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa7c005210 0x7faa7c00b9c0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7faa8c00e320 tx=0x7faa7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7a74f2333ded456 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.290+0000 7faa9159a640 1 -- 172.21.15.88:0/1135936950 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8c009de0 con 0x7faa7c005210 2024-09-16T01:39:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.290+0000 7faa9159a640 1 -- 172.21.15.88:0/1135936950 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa8c02e620 con 0x7faa7c005210 2024-09-16T01:39:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.290+0000 7faa9159a640 1 -- 172.21.15.88:0/1135936950 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8c02ea10 con 0x7faa7c005210 2024-09-16T01:39:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.291+0000 7faa98cf8640 1 -- 172.21.15.88:0/1135936950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa7c005210 msgr2=0x7faa7c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.291+0000 7faa98cf8640 1 --2- 172.21.15.88:0/1135936950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa7c005210 0x7faa7c00b9c0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7faa8c00e320 tx=0x7faa7c005830 comp rx=0 tx=0).stop 2024-09-16T01:39:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.291+0000 7faa98cf8640 1 -- 172.21.15.88:0/1135936950 shutdown_connections 2024-09-16T01:39:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.291+0000 7faa98cf8640 1 --2- 172.21.15.88:0/1135936950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa7c005210 0x7faa7c00b9c0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.291+0000 7faa98cf8640 1 -- 172.21.15.88:0/1135936950 >> 172.21.15.88:0/1135936950 conn(0x7faa9406bc10 msgr2=0x7faa9406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:44.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.291+0000 7faa98cf8640 1 -- 172.21.15.88:0/1135936950 shutdown_connections 2024-09-16T01:39:44.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.291+0000 7faa98cf8640 1 -- 172.21.15.88:0/1135936950 wait complete. 2024-09-16T01:39:44.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.292+0000 7faa98cf8640 1 Processor -- start 2024-09-16T01:39:44.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.293+0000 7faa98cf8640 1 -- start start 2024-09-16T01:39:44.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.293+0000 7faa98cf8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa7c005210 0x7faa940b5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:44.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.293+0000 7faa98cf8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa94151640 0x7faa94151ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:44.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.293+0000 7faa98cf8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa94142000 0x7faa94142480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:44.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.293+0000 7faa98cf8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7faa9415e920 con 0x7faa94151640 2024-09-16T01:39:44.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.293+0000 7faa98cf8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faa940af8a0 con 0x7faa94142000 2024-09-16T01:39:44.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.293+0000 7faa98cf8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7faa940ae890 con 0x7faa7c005210 2024-09-16T01:39:44.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.293+0000 7faa91d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa94151640 0x7faa94151ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:44.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.293+0000 7faa91d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa94151640 0x7faa94151ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46448/0 (socket says 172.21.15.88:46448) 2024-09-16T01:39:44.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.293+0000 7faa91d9b640 1 -- 172.21.15.88:0/1211956919 learned_addr learned my addr 172.21.15.88:0/1211956919 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:44.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.293+0000 7faa9259c640 1 --2- 172.21.15.88:0/1211956919 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa7c005210 0x7faa940b5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:44.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.293+0000 7faa92d9d640 1 --2- 172.21.15.88:0/1211956919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa94142000 0x7faa94142480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:44.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.294+0000 7faa91d9b640 1 -- 172.21.15.88:0/1211956919 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa7c005210 msgr2=0x7faa940b5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:44.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.294+0000 7faa91d9b640 1 --2- 172.21.15.88:0/1211956919 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa7c005210 0x7faa940b5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:44.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.294+0000 7faa91d9b640 1 -- 172.21.15.88:0/1211956919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa94142000 msgr2=0x7faa94142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:44.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.294+0000 7faa91d9b640 1 --2- 172.21.15.88:0/1211956919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa94142000 0x7faa94142480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:44.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.294+0000 7faa91d9b640 1 -- 172.21.15.88:0/1211956919 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa941521b0 con 0x7faa94151640 2024-09-16T01:39:44.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.294+0000 7faa91d9b640 1 --2- 172.21.15.88:0/1211956919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa94151640 0x7faa94151ac0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7faa8800e7b0 tx=0x7faa8800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:44.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.294+0000 7faa82ffd640 1 -- 172.21.15.88:0/1211956919 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa880050d0 con 0x7faa94151640 2024-09-16T01:39:44.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.294+0000 7faa82ffd640 1 -- 172.21.15.88:0/1211956919 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa88018070 con 0x7faa94151640 2024-09-16T01:39:44.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.294+0000 7faa82ffd640 1 -- 172.21.15.88:0/1211956919 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8800c6c0 con 0x7faa94151640 2024-09-16T01:39:44.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.294+0000 7faa98cf8640 1 -- 172.21.15.88:0/1211956919 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa941e9e50 con 0x7faa94151640 2024-09-16T01:39:44.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.294+0000 7faa98cf8640 1 -- 172.21.15.88:0/1211956919 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa941ea410 con 0x7faa94151640 2024-09-16T01:39:44.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.295+0000 7faa98cf8640 1 -- 172.21.15.88:0/1211956919 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa940aec20 con 0x7faa94151640 2024-09-16T01:39:44.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.296+0000 7faa82ffd640 1 -- 172.21.15.88:0/1211956919 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7faa8800c860 con 0x7faa94151640 2024-09-16T01:39:44.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.296+0000 7faa82ffd640 1 --2- 172.21.15.88:0/1211956919 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faa7406c2e0 0x7faa7406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:44.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.296+0000 7faa82ffd640 1 -- 172.21.15.88:0/1211956919 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7faa8808b100 con 0x7faa94151640 2024-09-16T01:39:44.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.296+0000 7faa9259c640 1 --2- 172.21.15.88:0/1211956919 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faa7406c2e0 0x7faa7406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:44.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.297+0000 7faa9259c640 1 --2- 172.21.15.88:0/1211956919 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faa7406c2e0 0x7faa7406e7a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7faa84000c00 tx=0x7faa84001d70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:44.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.301+0000 7faa82ffd640 1 -- 172.21.15.88:0/1211956919 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7faa880592e0 con 0x7faa94151640 2024-09-16T01:39:44.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.502+0000 7faa98cf8640 1 -- 172.21.15.88:0/1211956919 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa941ea5e0 con 0x7faa94151640 2024-09-16T01:39:44.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.502+0000 7faa82ffd640 1 -- 172.21.15.88:0/1211956919 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+5279 (secure 0 0 0) 0x7faa8805d5e0 con 0x7faa94151640 2024-09-16T01:39:44.505 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:44.506 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":59,"btime":"2024-09-16T01:39:43:238688+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5725,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2696547723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2696547723},{"type":"v1","addr":"172.21.15.154:6834","nonce":2696547723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54},{"gid":6411,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3276199399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3276199399},{"type":"v1","addr":"172.21.15.88:6836","nonce":3276199399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54},{"gid":6414,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2944989249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2944989249},{"type":"v1","addr":"172.21.15.88:6837","nonce":2944989249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:39:14.808422+0000","modified":"2024-09-16T01:39:43.238660+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6429},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6429":{"gid":6429,"name":"d","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6835/3482042494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":3482042494},{"type":"v1","addr":"172.21.15.154:6835","nonce":3482042494}]},"join_fscid":-1,"laggy_since":"2024-09-16T01:39:43.238660+0000","export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6429,"qdb_cluster":[6429]},"id":5}]} 2024-09-16T01:39:44.506 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 59 2024-09-16T01:39:44.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.504+0000 7faa98cf8640 1 -- 172.21.15.88:0/1211956919 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faa7406c2e0 msgr2=0x7faa7406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:44.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.504+0000 7faa98cf8640 1 --2- 172.21.15.88:0/1211956919 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faa7406c2e0 0x7faa7406e7a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7faa84000c00 tx=0x7faa84001d70 comp rx=0 tx=0).stop 2024-09-16T01:39:44.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.504+0000 7faa98cf8640 1 -- 172.21.15.88:0/1211956919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa94151640 msgr2=0x7faa94151ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:44.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.504+0000 7faa98cf8640 1 --2- 172.21.15.88:0/1211956919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa94151640 0x7faa94151ac0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7faa8800e7b0 tx=0x7faa8800ec70 comp rx=0 tx=0).stop 2024-09-16T01:39:44.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.504+0000 7faa98cf8640 1 -- 172.21.15.88:0/1211956919 shutdown_connections 2024-09-16T01:39:44.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.504+0000 7faa98cf8640 1 --2- 172.21.15.88:0/1211956919 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faa7406c2e0 0x7faa7406e7a0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:44.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.504+0000 7faa98cf8640 1 --2- 172.21.15.88:0/1211956919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa94142000 0x7faa94142480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:44.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.504+0000 7faa98cf8640 1 --2- 172.21.15.88:0/1211956919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa94151640 0x7faa94151ac0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:44.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.504+0000 7faa98cf8640 1 --2- 172.21.15.88:0/1211956919 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa7c005210 0x7faa940b5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:44.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.504+0000 7faa98cf8640 1 -- 172.21.15.88:0/1211956919 >> 172.21.15.88:0/1211956919 conn(0x7faa9406bc10 msgr2=0x7faa940b0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:44.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.505+0000 7faa98cf8640 1 -- 172.21.15.88:0/1211956919 shutdown_connections 2024-09-16T01:39:44.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.505+0000 7faa98cf8640 1 -- 172.21.15.88:0/1211956919 wait complete. 2024-09-16T01:39:44.518 DEBUG:tasks.ceph_test_case:wait_until_true: success in 15s and 0 retries 2024-09-16T01:39:44.619 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_inject_rename_corrupt_dentry_first 2024-09-16T01:39:44.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.718+0000 7f20c839f640 1 Processor -- start 2024-09-16T01:39:44.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.718+0000 7f20c839f640 1 -- start start 2024-09-16T01:39:44.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.719+0000 7f20c839f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f20c0158de0 0x7f20c00b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:44.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.719+0000 7f20c839f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f20c00b53a0 0x7f20c006eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:44.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.719+0000 7f20c839f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20c0072640 con 0x7f20c0158a00 2024-09-16T01:39:44.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.719+0000 7f20c839f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f20c00728f0 con 0x7f20c00b53a0 2024-09-16T01:39:44.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.719+0000 7f20c839f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f20c0072bd0 con 0x7f20c0158de0 2024-09-16T01:39:44.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.719+0000 7f20c693c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f20c00b53a0 0x7f20c006eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:44.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.719+0000 7f20c613b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f20c0158a00 0x7f20c0071f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36142/0 (socket says 172.21.15.88:36142) 2024-09-16T01:39:44.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.719+0000 7f20c613b640 1 -- 172.21.15.88:0/182428396 learned_addr learned my addr 172.21.15.88:0/182428396 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:44.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.719+0000 7f20c593a640 1 --2- 172.21.15.88:0/182428396 >> v2:172.21.15.154:3300/0 conn(0x7f20c0158de0 0x7f20c00b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:44.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.719+0000 7f20c693c640 1 -- 172.21.15.88:0/182428396 >> v2:172.21.15.154:3300/0 conn(0x7f20c0158de0 msgr2=0x7f20c00b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:44.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.719+0000 7f20c693c640 1 --2- 172.21.15.88:0/182428396 >> v2:172.21.15.154:3300/0 conn(0x7f20c0158de0 0x7f20c00b4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:44.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.719+0000 7f20c693c640 1 -- 172.21.15.88:0/182428396 >> v1:172.21.15.88:6789/0 conn(0x7f20c0158a00 legacy=0x7f20c0071f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:44.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.719+0000 7f20c5139640 1 -- 172.21.15.88:0/182428396 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3105589248 0 0) 0x7f20c0072640 con 0x7f20c0158a00 2024-09-16T01:39:44.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.719+0000 7f20c693c640 1 -- 172.21.15.88:0/182428396 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20c0072eb0 con 0x7f20c00b53a0 2024-09-16T01:39:44.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.719+0000 7f20c593a640 1 --2- 172.21.15.88:0/182428396 >> v2:172.21.15.154:3300/0 conn(0x7f20c0158de0 0x7f20c00b4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:44.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.720+0000 7f20c693c640 1 --2- 172.21.15.88:0/182428396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20c00b53a0 0x7f20c006eae0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f20b80097a0 tx=0x7f20b802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4140aeacf642ed6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:44.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.720+0000 7f20c5139640 1 -- 172.21.15.88:0/182428396 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b8037070 con 0x7f20c00b53a0 2024-09-16T01:39:44.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.720+0000 7f20c5139640 1 -- 172.21.15.88:0/182428396 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20b8032440 con 0x7f20c00b53a0 2024-09-16T01:39:44.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.720+0000 7f20c5139640 1 -- 172.21.15.88:0/182428396 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b8032810 con 0x7f20c00b53a0 2024-09-16T01:39:44.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.721+0000 7f20c839f640 1 -- 172.21.15.88:0/182428396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20c00b53a0 msgr2=0x7f20c006eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:44.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.721+0000 7f20c839f640 1 --2- 172.21.15.88:0/182428396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20c00b53a0 0x7f20c006eae0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f20b80097a0 tx=0x7f20b802a1e0 comp rx=0 tx=0).stop 2024-09-16T01:39:44.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.721+0000 7f20c839f640 1 -- 172.21.15.88:0/182428396 shutdown_connections 2024-09-16T01:39:44.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.721+0000 7f20c839f640 1 --2- 172.21.15.88:0/182428396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20c00b53a0 0x7f20c006eae0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:44.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.721+0000 7f20c839f640 1 --2- 172.21.15.88:0/182428396 >> v2:172.21.15.154:3300/0 conn(0x7f20c0158de0 0x7f20c00b4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:44.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.721+0000 7f20c839f640 1 -- 172.21.15.88:0/182428396 >> 172.21.15.88:0/182428396 conn(0x7f20c006cb50 msgr2=0x7f20c006cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:44.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.721+0000 7f20c839f640 1 -- 172.21.15.88:0/182428396 shutdown_connections 2024-09-16T01:39:44.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.721+0000 7f20c839f640 1 -- 172.21.15.88:0/182428396 wait complete. 2024-09-16T01:39:44.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.722+0000 7f20c839f640 1 Processor -- start 2024-09-16T01:39:44.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.722+0000 7f20c839f640 1 -- start start 2024-09-16T01:39:44.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.723+0000 7f20c839f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20c00b53a0 0x7f20c014b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:44.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.723+0000 7f20c839f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20c0158a00 0x7f20c00cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:44.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.723+0000 7f20c839f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c0158de0 0x7f20c014e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:44.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.723+0000 7f20c839f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f20c00b5b20 con 0x7f20c00b53a0 2024-09-16T01:39:44.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.723+0000 7f20c839f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f20c006e550 con 0x7f20c0158de0 2024-09-16T01:39:44.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.723+0000 7f20c839f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f20c00b5ca0 con 0x7f20c0158a00 2024-09-16T01:39:44.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.723+0000 7f20c613b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20c00b53a0 0x7f20c014b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:44.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.723+0000 7f20c613b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20c00b53a0 0x7f20c014b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46478/0 (socket says 172.21.15.88:46478) 2024-09-16T01:39:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.723+0000 7f20c613b640 1 -- 172.21.15.88:0/770149063 learned_addr learned my addr 172.21.15.88:0/770149063 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.723+0000 7f20c693c640 1 --2- 172.21.15.88:0/770149063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c0158de0 0x7f20c014e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.723+0000 7f20c593a640 1 --2- 172.21.15.88:0/770149063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20c0158a00 0x7f20c00cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.723+0000 7f20c613b640 1 -- 172.21.15.88:0/770149063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20c0158a00 msgr2=0x7f20c00cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.723+0000 7f20c613b640 1 --2- 172.21.15.88:0/770149063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20c0158a00 0x7f20c00cdf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.723+0000 7f20c613b640 1 -- 172.21.15.88:0/770149063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c0158de0 msgr2=0x7f20c014e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.723+0000 7f20c613b640 1 --2- 172.21.15.88:0/770149063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c0158de0 0x7f20c014e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.723+0000 7f20c613b640 1 -- 172.21.15.88:0/770149063 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20c01f5900 con 0x7f20c00b53a0 2024-09-16T01:39:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.723+0000 7f20c693c640 1 --2- 172.21.15.88:0/770149063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c0158de0 0x7f20c014e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.723+0000 7f20c613b640 1 --2- 172.21.15.88:0/770149063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20c00b53a0 0x7f20c014b500 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f20bc00c8c0 tx=0x7f20bc00cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:44.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.723+0000 7f20b6ffd640 1 -- 172.21.15.88:0/770149063 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20bc018070 con 0x7f20c00b53a0 2024-09-16T01:39:44.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.723+0000 7f20b6ffd640 1 -- 172.21.15.88:0/770149063 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20bc00a640 con 0x7f20c00b53a0 2024-09-16T01:39:44.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.723+0000 7f20c839f640 1 -- 172.21.15.88:0/770149063 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20c01f5be0 con 0x7f20c00b53a0 2024-09-16T01:39:44.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.723+0000 7f20b6ffd640 1 -- 172.21.15.88:0/770149063 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20bc00aa30 con 0x7f20c00b53a0 2024-09-16T01:39:44.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.723+0000 7f20c839f640 1 -- 172.21.15.88:0/770149063 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20c01f61a0 con 0x7f20c00b53a0 2024-09-16T01:39:44.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.724+0000 7f20c839f640 1 -- 172.21.15.88:0/770149063 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20c00b4e60 con 0x7f20c00b53a0 2024-09-16T01:39:44.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.725+0000 7f20b6ffd640 1 -- 172.21.15.88:0/770149063 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f20bc01b070 con 0x7f20c00b53a0 2024-09-16T01:39:44.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.725+0000 7f20b6ffd640 1 --2- 172.21.15.88:0/770149063 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f20a406c2b0 0x7f20a406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:44.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.725+0000 7f20b6ffd640 1 -- 172.21.15.88:0/770149063 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f20bc08aab0 con 0x7f20c00b53a0 2024-09-16T01:39:44.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.725+0000 7f20c593a640 1 --2- 172.21.15.88:0/770149063 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f20a406c2b0 0x7f20a406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:44.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.726+0000 7f20c593a640 1 --2- 172.21.15.88:0/770149063 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f20a406c2b0 0x7f20a406e770 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f20c00b1490 tx=0x7f20b0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:44.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.729+0000 7f20b6ffd640 1 -- 172.21.15.88:0/770149063 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f20bc0592b0 con 0x7f20c00b53a0 2024-09-16T01:39:44.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.895+0000 7f20c839f640 1 -- 172.21.15.88:0/770149063 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"} v 0) -- 0x7f20c00b14c0 con 0x7f20c00b53a0 2024-09-16T01:39:45.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.898+0000 7f20b6ffd640 1 -- 172.21.15.88:0/770149063 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"}]=0 v24) ==== 121+0+0 (secure 0 0 0) 0x7f20bc058bb0 con 0x7f20c00b53a0 2024-09-16T01:39:45.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.900+0000 7f20c839f640 1 -- 172.21.15.88:0/770149063 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f20a406c2b0 msgr2=0x7f20a406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:45.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.900+0000 7f20c839f640 1 --2- 172.21.15.88:0/770149063 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f20a406c2b0 0x7f20a406e770 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f20c00b1490 tx=0x7f20b0008040 comp rx=0 tx=0).stop 2024-09-16T01:39:45.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.900+0000 7f20c839f640 1 -- 172.21.15.88:0/770149063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20c00b53a0 msgr2=0x7f20c014b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:45.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.900+0000 7f20c839f640 1 --2- 172.21.15.88:0/770149063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20c00b53a0 0x7f20c014b500 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f20bc00c8c0 tx=0x7f20bc00cd80 comp rx=0 tx=0).stop 2024-09-16T01:39:45.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.900+0000 7f20c839f640 1 -- 172.21.15.88:0/770149063 shutdown_connections 2024-09-16T01:39:45.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.900+0000 7f20c839f640 1 --2- 172.21.15.88:0/770149063 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f20a406c2b0 0x7f20a406e770 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:45.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.900+0000 7f20c839f640 1 --2- 172.21.15.88:0/770149063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c0158de0 0x7f20c014e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:45.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.900+0000 7f20c839f640 1 --2- 172.21.15.88:0/770149063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20c0158a00 0x7f20c00cdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:45.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.900+0000 7f20c839f640 1 --2- 172.21.15.88:0/770149063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20c00b53a0 0x7f20c014b500 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:45.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.900+0000 7f20c839f640 1 -- 172.21.15.88:0/770149063 >> 172.21.15.88:0/770149063 conn(0x7f20c006cb50 msgr2=0x7f20c00aea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:45.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.900+0000 7f20c839f640 1 -- 172.21.15.88:0/770149063 shutdown_connections 2024-09-16T01:39:45.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:44.901+0000 7f20c839f640 1 -- 172.21.15.88:0/770149063 wait complete. 2024-09-16T01:39:45.016 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds freeze 5:0 false 2024-09-16T01:39:45.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.115+0000 7f2e76d8d640 1 Processor -- start 2024-09-16T01:39:45.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.115+0000 7f2e76d8d640 1 -- start start 2024-09-16T01:39:45.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.116+0000 7f2e76d8d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e7015a050 0x7f2e7015d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:45.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.116+0000 7f2e76d8d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2e700731c0 0x7f2e70073600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:45.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.116+0000 7f2e76d8d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e7015a430 con 0x7f2e7015ffd0 2024-09-16T01:39:45.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.116+0000 7f2e76d8d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2e70073b40 con 0x7f2e700731c0 2024-09-16T01:39:45.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.116+0000 7f2e76d8d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2e7015d9a0 con 0x7f2e7015a050 2024-09-16T01:39:45.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.116+0000 7f2e7532a640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f2e7015ffd0 0x7f2e70160480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36146/0 (socket says 172.21.15.88:36146) 2024-09-16T01:39:45.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.116+0000 7f2e7532a640 1 -- 172.21.15.88:0/3904701062 learned_addr learned my addr 172.21.15.88:0/3904701062 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:45.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.116+0000 7f2e6ffff640 1 --2- 172.21.15.88:0/3904701062 >> v2:172.21.15.154:3300/0 conn(0x7f2e700731c0 0x7f2e70073600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:45.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.116+0000 7f2e74b29640 1 --2- 172.21.15.88:0/3904701062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e7015a050 0x7f2e7015d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:45.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.116+0000 7f2e6f7fe640 1 -- 172.21.15.88:0/3904701062 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4030840581 0 0) 0x7f2e7015a430 con 0x7f2e7015ffd0 2024-09-16T01:39:45.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.116+0000 7f2e6f7fe640 1 -- 172.21.15.88:0/3904701062 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e58003520 con 0x7f2e7015ffd0 2024-09-16T01:39:45.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.116+0000 7f2e6f7fe640 1 -- 172.21.15.88:0/3904701062 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1788118521 0 0) 0x7f2e58003520 con 0x7f2e7015ffd0 2024-09-16T01:39:45.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.116+0000 7f2e6f7fe640 1 -- 172.21.15.88:0/3904701062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e7015a050 msgr2=0x7f2e7015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:45.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.116+0000 7f2e6f7fe640 1 --2- 172.21.15.88:0/3904701062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e7015a050 0x7f2e7015d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:45.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.116+0000 7f2e6f7fe640 1 -- 172.21.15.88:0/3904701062 >> v2:172.21.15.154:3300/0 conn(0x7f2e700731c0 msgr2=0x7f2e70073600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:45.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.117+0000 7f2e6f7fe640 1 --2- 172.21.15.88:0/3904701062 >> v2:172.21.15.154:3300/0 conn(0x7f2e700731c0 0x7f2e70073600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:45.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.117+0000 7f2e6f7fe640 1 -- 172.21.15.88:0/3904701062 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e700cd0a0 con 0x7f2e7015ffd0 2024-09-16T01:39:45.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.117+0000 7f2e6f7fe640 1 -- 172.21.15.88:0/3904701062 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f2e60001670 con 0x7f2e7015ffd0 2024-09-16T01:39:45.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.117+0000 7f2e6f7fe640 1 -- 172.21.15.88:0/3904701062 >> v1:172.21.15.88:6789/0 conn(0x7f2e7015ffd0 legacy=0x7f2e70160480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:45.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.117+0000 7f2e6f7fe640 1 --2- 172.21.15.88:0/3904701062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e580039b0 0x7f2e58003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:45.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.117+0000 7f2e6f7fe640 1 --2- 172.21.15.88:0/3904701062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e58004800 0x7f2e58004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:45.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.117+0000 7f2e6f7fe640 1 --2- 172.21.15.88:0/3904701062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e7015a050 0x7f2e580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:45.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.117+0000 7f2e6f7fe640 1 -- 172.21.15.88:0/3904701062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2e70073b40 con 0x7f2e58004800 2024-09-16T01:39:45.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.117+0000 7f2e6f7fe640 1 -- 172.21.15.88:0/3904701062 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7015d9a0 con 0x7f2e580039b0 2024-09-16T01:39:45.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.117+0000 7f2e6f7fe640 1 -- 172.21.15.88:0/3904701062 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2e5800b9c0 con 0x7f2e7015a050 2024-09-16T01:39:45.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.117+0000 7f2e6ffff640 1 --2- 172.21.15.88:0/3904701062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e58004800 0x7f2e58004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:45.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.117+0000 7f2e74b29640 1 --2- 172.21.15.88:0/3904701062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e580039b0 0x7f2e58003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:45.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.117+0000 7f2e7532a640 1 --2- 172.21.15.88:0/3904701062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e7015a050 0x7f2e580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:45.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.117+0000 7f2e6ffff640 1 -- 172.21.15.88:0/3904701062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e7015a050 msgr2=0x7f2e580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:45.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.117+0000 7f2e6ffff640 1 --2- 172.21.15.88:0/3904701062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e7015a050 0x7f2e580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:45.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.117+0000 7f2e6ffff640 1 -- 172.21.15.88:0/3904701062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e580039b0 msgr2=0x7f2e58003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:45.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.117+0000 7f2e6ffff640 1 --2- 172.21.15.88:0/3904701062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e580039b0 0x7f2e58003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:45.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.117+0000 7f2e6ffff640 1 -- 172.21.15.88:0/3904701062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e58003520 con 0x7f2e58004800 2024-09-16T01:39:45.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.118+0000 7f2e6ffff640 1 --2- 172.21.15.88:0/3904701062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e58004800 0x7f2e58004c80 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f2e64009ac0 tx=0x7f2e58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e36a5c9f2a41419 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:45.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.118+0000 7f2e6f7fe640 1 -- 172.21.15.88:0/3904701062 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e64036070 con 0x7f2e58004800 2024-09-16T01:39:45.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.118+0000 7f2e6f7fe640 1 -- 172.21.15.88:0/3904701062 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e64031400 con 0x7f2e58004800 2024-09-16T01:39:45.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.118+0000 7f2e6f7fe640 1 -- 172.21.15.88:0/3904701062 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e64009690 con 0x7f2e58004800 2024-09-16T01:39:45.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.118+0000 7f2e76d8d640 1 -- 172.21.15.88:0/3904701062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e58004800 msgr2=0x7f2e58004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:45.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.118+0000 7f2e76d8d640 1 --2- 172.21.15.88:0/3904701062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e58004800 0x7f2e58004c80 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f2e64009ac0 tx=0x7f2e58005830 comp rx=0 tx=0).stop 2024-09-16T01:39:45.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.118+0000 7f2e7532a640 1 -- 172.21.15.88:0/3904701062 reap_dead start 2024-09-16T01:39:45.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.118+0000 7f2e76d8d640 1 -- 172.21.15.88:0/3904701062 shutdown_connections 2024-09-16T01:39:45.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.118+0000 7f2e76d8d640 1 -- 172.21.15.88:0/3904701062 >> 172.21.15.88:0/3904701062 conn(0x7f2e7006d0f0 msgr2=0x7f2e7006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:45.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.118+0000 7f2e76d8d640 1 -- 172.21.15.88:0/3904701062 shutdown_connections 2024-09-16T01:39:45.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.118+0000 7f2e76d8d640 1 -- 172.21.15.88:0/3904701062 wait complete. 2024-09-16T01:39:45.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.120+0000 7f2e76d8d640 1 Processor -- start 2024-09-16T01:39:45.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.120+0000 7f2e76d8d640 1 -- start start 2024-09-16T01:39:45.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.120+0000 7f2e76d8d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e58004800 0x7f2e70151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:45.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.120+0000 7f2e76d8d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e7014f4c0 0x7f2e701562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:45.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.120+0000 7f2e76d8d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e70152c60 0x7f2e700cf750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:45.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.120+0000 7f2e76d8d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2e700cd940 con 0x7f2e7014f4c0 2024-09-16T01:39:45.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.120+0000 7f2e76d8d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2e700cd660 con 0x7f2e58004800 2024-09-16T01:39:45.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.120+0000 7f2e76d8d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2e700cdc20 con 0x7f2e70152c60 2024-09-16T01:39:45.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.121+0000 7f2e6ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e7014f4c0 0x7f2e701562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:45.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.121+0000 7f2e6ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e7014f4c0 0x7f2e701562c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46498/0 (socket says 172.21.15.88:46498) 2024-09-16T01:39:45.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.121+0000 7f2e6ffff640 1 -- 172.21.15.88:0/1489749309 learned_addr learned my addr 172.21.15.88:0/1489749309 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:45.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.121+0000 7f2e6ffff640 1 -- 172.21.15.88:0/1489749309 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e70152c60 msgr2=0x7f2e700cf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:45.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.121+0000 7f2e6ffff640 1 --2- 172.21.15.88:0/1489749309 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e70152c60 0x7f2e700cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:45.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.121+0000 7f2e6ffff640 1 -- 172.21.15.88:0/1489749309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e58004800 msgr2=0x7f2e70151530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:45.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.121+0000 7f2e6ffff640 1 --2- 172.21.15.88:0/1489749309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e58004800 0x7f2e70151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:45.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.121+0000 7f2e6ffff640 1 -- 172.21.15.88:0/1489749309 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e70156800 con 0x7f2e7014f4c0 2024-09-16T01:39:45.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.121+0000 7f2e6ffff640 1 --2- 172.21.15.88:0/1489749309 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e7014f4c0 0x7f2e701562c0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f2e64034040 tx=0x7f2e64004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:45.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.121+0000 7f2e6d7fa640 1 -- 172.21.15.88:0/1489749309 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e64041020 con 0x7f2e7014f4c0 2024-09-16T01:39:45.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.121+0000 7f2e6d7fa640 1 -- 172.21.15.88:0/1489749309 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e6403c470 con 0x7f2e7014f4c0 2024-09-16T01:39:45.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.121+0000 7f2e6d7fa640 1 -- 172.21.15.88:0/1489749309 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e64036040 con 0x7f2e7014f4c0 2024-09-16T01:39:45.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.121+0000 7f2e76d8d640 1 -- 172.21.15.88:0/1489749309 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e70151bc0 con 0x7f2e7014f4c0 2024-09-16T01:39:45.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.121+0000 7f2e76d8d640 1 -- 172.21.15.88:0/1489749309 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e70152180 con 0x7f2e7014f4c0 2024-09-16T01:39:45.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.122+0000 7f2e76d8d640 1 -- 172.21.15.88:0/1489749309 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e700b36e0 con 0x7f2e7014f4c0 2024-09-16T01:39:45.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.123+0000 7f2e6d7fa640 1 -- 172.21.15.88:0/1489749309 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2e64044020 con 0x7f2e7014f4c0 2024-09-16T01:39:45.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.123+0000 7f2e6d7fa640 1 --2- 172.21.15.88:0/1489749309 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2e5c06c2e0 0x7f2e5c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:45.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.123+0000 7f2e6d7fa640 1 -- 172.21.15.88:0/1489749309 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5709+0+0 (secure 0 0 0) 0x7f2e640abc50 con 0x7f2e7014f4c0 2024-09-16T01:39:45.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.123+0000 7f2e74b29640 1 --2- 172.21.15.88:0/1489749309 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2e5c06c2e0 0x7f2e5c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:45.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.124+0000 7f2e74b29640 1 --2- 172.21.15.88:0/1489749309 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2e5c06c2e0 0x7f2e5c06e7a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009ee0 tx=0x7f2e68009750 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:45.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.128+0000 7f2e6d7fa640 1 -- 172.21.15.88:0/1489749309 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2e640b1020 con 0x7f2e7014f4c0 2024-09-16T01:39:45.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:45.361+0000 7f2e76d8d640 1 -- 172.21.15.88:0/1489749309 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds freeze", "role_or_gid": "5:0", "val": "false"} v 0) -- 0x7f2e7015a820 con 0x7f2e7014f4c0 2024-09-16T01:39:46.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.249+0000 7f2e6d7fa640 1 -- 172.21.15.88:0/1489749309 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds freeze", "role_or_gid": "5:0", "val": "false"}]=0 unfreezing mds.6429 v60) ==== 115+0+0 (secure 0 0 0) 0x7f2e6404b030 con 0x7f2e7014f4c0 2024-09-16T01:39:46.251 INFO:teuthology.orchestra.run.smithi088.stderr:unfreezing mds.6429 2024-09-16T01:39:46.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.253+0000 7f2e76d8d640 1 -- 172.21.15.88:0/1489749309 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2e5c06c2e0 msgr2=0x7f2e5c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:46.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.253+0000 7f2e76d8d640 1 --2- 172.21.15.88:0/1489749309 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2e5c06c2e0 0x7f2e5c06e7a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009ee0 tx=0x7f2e68009750 comp rx=0 tx=0).stop 2024-09-16T01:39:46.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.253+0000 7f2e76d8d640 1 -- 172.21.15.88:0/1489749309 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e7014f4c0 msgr2=0x7f2e701562c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:46.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.253+0000 7f2e76d8d640 1 --2- 172.21.15.88:0/1489749309 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e7014f4c0 0x7f2e701562c0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f2e64034040 tx=0x7f2e64004730 comp rx=0 tx=0).stop 2024-09-16T01:39:46.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.253+0000 7f2e76d8d640 1 -- 172.21.15.88:0/1489749309 shutdown_connections 2024-09-16T01:39:46.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.253+0000 7f2e76d8d640 1 --2- 172.21.15.88:0/1489749309 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2e5c06c2e0 0x7f2e5c06e7a0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:46.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.253+0000 7f2e76d8d640 1 --2- 172.21.15.88:0/1489749309 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e70152c60 0x7f2e700cf750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:46.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.253+0000 7f2e76d8d640 1 --2- 172.21.15.88:0/1489749309 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e7014f4c0 0x7f2e701562c0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:46.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.253+0000 7f2e76d8d640 1 --2- 172.21.15.88:0/1489749309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e58004800 0x7f2e70151530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:46.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.253+0000 7f2e76d8d640 1 -- 172.21.15.88:0/1489749309 >> 172.21.15.88:0/1489749309 conn(0x7f2e7006d0f0 msgr2=0x7f2e700b5820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:46.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.253+0000 7f2e76d8d640 1 -- 172.21.15.88:0/1489749309 shutdown_connections 2024-09-16T01:39:46.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.253+0000 7f2e76d8d640 1 -- 172.21.15.88:0/1489749309 wait complete. 2024-09-16T01:39:46.271 DEBUG:teuthology.orchestra.run.smithi154:> sudo sysctl -n kernel.core_pattern 2024-09-16T01:39:46.298 INFO:teuthology.orchestra.run.smithi154.stdout:/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T01:39:46.300 INFO:tasks.cephfs.cephfs_test_case:Clearing core from directory: /home/ubuntu/cephtest/archive/coredump 2024-09-16T01:39:46.300 DEBUG:teuthology.orchestra.run.smithi154:> cd /home/ubuntu/cephtest/archive/coredump && sudo ls | sudo xargs file 2024-09-16T01:39:46.370 INFO:teuthology.orchestra.run.smithi154.stdout:1726450768.43891.core: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph-mds -f --cluster ceph -i d', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/bin/ceph-mds', platform: 'x86_64' 2024-09-16T01:39:46.373 INFO:tasks.cephfs.cephfs_test_case:Enumerated cores: ['1726450768.43891.core'] 2024-09-16T01:39:46.373 INFO:tasks.cephfs.cephfs_test_case:Found core file 1726450768.43891.core, deleting it 2024-09-16T01:39:46.373 DEBUG:teuthology.orchestra.run.smithi154:> cd /home/ubuntu/cephtest/archive/coredump && sudo rm -f 1726450768.43891.core 2024-09-16T01:39:46.444 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-16T01:39:46.444 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-16T01:39:46.445 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-16T01:39:46.445 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:39:46.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:39:46.445 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi154 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' 2024-09-16T01:39:46.445 INFO:tasks.ceph.mds.d:Stopped 2024-09-16T01:39:46.445 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-16T01:39:46.487 INFO:tasks.ceph.mds.d:Started 2024-09-16T01:39:46.488 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:39:46.524 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~13s 2024-09-16T01:39:46.562 INFO:tasks.ceph.mds.d.smithi154.stdout:starting mds.d at 2024-09-16T01:39:46.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.596+0000 7efd68f59640 1 Processor -- start 2024-09-16T01:39:46.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.596+0000 7efd68f59640 1 -- start start 2024-09-16T01:39:46.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.597+0000 7efd68f59640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efd640b5be0 0x7efd6415ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:46.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.597+0000 7efd68f59640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efd640b29a0 0x7efd640cde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:46.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.597+0000 7efd68f59640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd640b6830 con 0x7efd6415a9c0 2024-09-16T01:39:46.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.597+0000 7efd68f59640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7efd6415f410 con 0x7efd640b29a0 2024-09-16T01:39:46.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.597+0000 7efd68f59640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7efd640cd0b0 con 0x7efd640b5be0 2024-09-16T01:39:46.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.597+0000 7efd62d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efd640b29a0 0x7efd640cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:46.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.597+0000 7efd62d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efd640b29a0 0x7efd640cde70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46506/0 (socket says 172.21.15.88:46506) 2024-09-16T01:39:46.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.597+0000 7efd62d9d640 1 -- 172.21.15.88:0/2324620679 learned_addr learned my addr 172.21.15.88:0/2324620679 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:46.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.597+0000 7efd59d9b640 1 --2- 172.21.15.88:0/2324620679 >> v2:172.21.15.154:3300/0 conn(0x7efd640b5be0 0x7efd6415ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:46.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.598+0000 7efd59d9b640 1 -- 172.21.15.88:0/2324620679 >> v2:172.21.15.88:3300/0 conn(0x7efd640b29a0 msgr2=0x7efd640cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:46.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.598+0000 7efd59d9b640 1 --2- 172.21.15.88:0/2324620679 >> v2:172.21.15.88:3300/0 conn(0x7efd640b29a0 0x7efd640cde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:46.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.598+0000 7efd59d9b640 1 -- 172.21.15.88:0/2324620679 >> v1:172.21.15.154:6789/0 conn(0x7efd6415a9c0 legacy=0x7efd6415e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:46.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.598+0000 7efd59d9b640 1 -- 172.21.15.88:0/2324620679 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd640cd390 con 0x7efd640b5be0 2024-09-16T01:39:46.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.598+0000 7efd61d9b640 1 -- 172.21.15.88:0/2324620679 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3467839417 0 0) 0x7efd640b6830 con 0x7efd6415a9c0 2024-09-16T01:39:46.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.599+0000 7efd59d9b640 1 --2- 172.21.15.88:0/2324620679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd640b5be0 0x7efd6415ed10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7efd640ccd90 tx=0x7efd4c02a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d619a8c404ca80c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:46.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.599+0000 7efd61d9b640 1 -- 172.21.15.88:0/2324620679 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd4c037070 con 0x7efd640b5be0 2024-09-16T01:39:46.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.599+0000 7efd61d9b640 1 -- 172.21.15.88:0/2324620679 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd4c032440 con 0x7efd640b5be0 2024-09-16T01:39:46.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.599+0000 7efd61d9b640 1 -- 172.21.15.88:0/2324620679 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd4c032810 con 0x7efd640b5be0 2024-09-16T01:39:46.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.599+0000 7efd68f59640 1 -- 172.21.15.88:0/2324620679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd640b5be0 msgr2=0x7efd6415ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:46.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.599+0000 7efd68f59640 1 --2- 172.21.15.88:0/2324620679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd640b5be0 0x7efd6415ed10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7efd640ccd90 tx=0x7efd4c02a1a0 comp rx=0 tx=0).stop 2024-09-16T01:39:46.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.599+0000 7efd68f59640 1 -- 172.21.15.88:0/2324620679 shutdown_connections 2024-09-16T01:39:46.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.599+0000 7efd68f59640 1 --2- 172.21.15.88:0/2324620679 >> v2:172.21.15.88:3300/0 conn(0x7efd640b29a0 0x7efd640cde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:46.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.599+0000 7efd68f59640 1 --2- 172.21.15.88:0/2324620679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd640b5be0 0x7efd6415ed10 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:46.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.599+0000 7efd68f59640 1 -- 172.21.15.88:0/2324620679 >> 172.21.15.88:0/2324620679 conn(0x7efd6406bc10 msgr2=0x7efd6406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:46.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.599+0000 7efd68f59640 1 -- 172.21.15.88:0/2324620679 shutdown_connections 2024-09-16T01:39:46.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.599+0000 7efd68f59640 1 -- 172.21.15.88:0/2324620679 wait complete. 2024-09-16T01:39:46.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.601+0000 7efd68f59640 1 Processor -- start 2024-09-16T01:39:46.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.601+0000 7efd68f59640 1 -- start start 2024-09-16T01:39:46.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.602+0000 7efd68f59640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd640b29a0 0x7efd6414da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:46.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.602+0000 7efd68f59640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd640b5be0 0x7efd640cb170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:46.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.602+0000 7efd68f59640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd6415a9c0 0x7efd6415d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:46.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.602+0000 7efd68f59640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7efd6415ada0 con 0x7efd640b5be0 2024-09-16T01:39:46.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.602+0000 7efd68f59640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efd640cd950 con 0x7efd6415a9c0 2024-09-16T01:39:46.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.602+0000 7efd68f59640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efd640ce3b0 con 0x7efd640b29a0 2024-09-16T01:39:46.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.602+0000 7efd59d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd640b5be0 0x7efd640cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:46.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.602+0000 7efd59d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd640b5be0 0x7efd640cb170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46508/0 (socket says 172.21.15.88:46508) 2024-09-16T01:39:46.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.602+0000 7efd59d9b640 1 -- 172.21.15.88:0/1295791590 learned_addr learned my addr 172.21.15.88:0/1295791590 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:46.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.602+0000 7efd6259c640 1 --2- 172.21.15.88:0/1295791590 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd640b29a0 0x7efd6414da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:46.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.602+0000 7efd62d9d640 1 --2- 172.21.15.88:0/1295791590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd6415a9c0 0x7efd6415d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:46.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.602+0000 7efd59d9b640 1 -- 172.21.15.88:0/1295791590 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd640b29a0 msgr2=0x7efd6414da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:46.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.602+0000 7efd59d9b640 1 --2- 172.21.15.88:0/1295791590 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd640b29a0 0x7efd6414da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:46.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.602+0000 7efd59d9b640 1 -- 172.21.15.88:0/1295791590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd6415a9c0 msgr2=0x7efd6415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:46.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.602+0000 7efd59d9b640 1 --2- 172.21.15.88:0/1295791590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd6415a9c0 0x7efd6415d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:46.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.602+0000 7efd59d9b640 1 -- 172.21.15.88:0/1295791590 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd641f4160 con 0x7efd640b5be0 2024-09-16T01:39:46.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.602+0000 7efd6259c640 1 --2- 172.21.15.88:0/1295791590 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd640b29a0 0x7efd6414da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:46.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.602+0000 7efd62d9d640 1 --2- 172.21.15.88:0/1295791590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd6415a9c0 0x7efd6415d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:46.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.602+0000 7efd59d9b640 1 --2- 172.21.15.88:0/1295791590 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd640b5be0 0x7efd640cb170 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7efd4c009890 tx=0x7efd4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:46.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.603+0000 7efd5b7fe640 1 -- 172.21.15.88:0/1295791590 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd4c037040 con 0x7efd640b5be0 2024-09-16T01:39:46.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.603+0000 7efd5b7fe640 1 -- 172.21.15.88:0/1295791590 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd4c042070 con 0x7efd640b5be0 2024-09-16T01:39:46.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.603+0000 7efd68f59640 1 -- 172.21.15.88:0/1295791590 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd640cbbf0 con 0x7efd640b5be0 2024-09-16T01:39:46.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.603+0000 7efd5b7fe640 1 -- 172.21.15.88:0/1295791590 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd4c030920 con 0x7efd640b5be0 2024-09-16T01:39:46.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.603+0000 7efd68f59640 1 -- 172.21.15.88:0/1295791590 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd6415c7e0 con 0x7efd640b5be0 2024-09-16T01:39:46.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.604+0000 7efd68f59640 1 -- 172.21.15.88:0/1295791590 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd640cde70 con 0x7efd640b5be0 2024-09-16T01:39:46.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.604+0000 7efd5b7fe640 1 -- 172.21.15.88:0/1295791590 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7efd4c030ac0 con 0x7efd640b5be0 2024-09-16T01:39:46.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.604+0000 7efd5b7fe640 1 --2- 172.21.15.88:0/1295791590 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efd4406c2e0 0x7efd4406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:46.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.605+0000 7efd6259c640 1 --2- 172.21.15.88:0/1295791590 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efd4406c2e0 0x7efd4406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:46.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.605+0000 7efd5b7fe640 1 -- 172.21.15.88:0/1295791590 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5795+0+0 (secure 0 0 0) 0x7efd4c0abda0 con 0x7efd640b5be0 2024-09-16T01:39:46.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.605+0000 7efd6259c640 1 --2- 172.21.15.88:0/1295791590 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efd4406c2e0 0x7efd4406e7a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7efd54000f60 tx=0x7efd54009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:46.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.609+0000 7efd5b7fe640 1 -- 172.21.15.88:0/1295791590 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7efd4c079e60 con 0x7efd640b5be0 2024-09-16T01:39:46.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.808+0000 7efd68f59640 1 -- 172.21.15.88:0/1295791590 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd640b44a0 con 0x7efd640b5be0 2024-09-16T01:39:46.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.808+0000 7efd5b7fe640 1 -- 172.21.15.88:0/1295791590 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+4397 (secure 0 0 0) 0x7efd4c07e160 con 0x7efd640b5be0 2024-09-16T01:39:46.810 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:46.810 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":61,"btime":"2024-09-16T01:39:46:253302+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5725,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2696547723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2696547723},{"type":"v1","addr":"172.21.15.154:6834","nonce":2696547723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54},{"gid":6411,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3276199399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3276199399},{"type":"v1","addr":"172.21.15.88:6836","nonce":3276199399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:39:14.808422+0000","modified":"2024-09-16T01:39:46.253297+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6414},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":61,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6837/2944989249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2944989249},{"type":"v1","addr":"172.21.15.88:6837","nonce":2944989249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":5}]} 2024-09-16T01:39:46.810 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 61 2024-09-16T01:39:46.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.810+0000 7efd68f59640 1 -- 172.21.15.88:0/1295791590 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efd4406c2e0 msgr2=0x7efd4406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:46.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.810+0000 7efd68f59640 1 --2- 172.21.15.88:0/1295791590 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efd4406c2e0 0x7efd4406e7a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7efd54000f60 tx=0x7efd54009040 comp rx=0 tx=0).stop 2024-09-16T01:39:46.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.810+0000 7efd68f59640 1 -- 172.21.15.88:0/1295791590 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd640b5be0 msgr2=0x7efd640cb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:46.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.810+0000 7efd68f59640 1 --2- 172.21.15.88:0/1295791590 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd640b5be0 0x7efd640cb170 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7efd4c009890 tx=0x7efd4c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:39:46.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.810+0000 7efd68f59640 1 -- 172.21.15.88:0/1295791590 shutdown_connections 2024-09-16T01:39:46.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.810+0000 7efd68f59640 1 --2- 172.21.15.88:0/1295791590 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efd4406c2e0 0x7efd4406e7a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:46.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.810+0000 7efd68f59640 1 --2- 172.21.15.88:0/1295791590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd6415a9c0 0x7efd6415d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:46.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.810+0000 7efd68f59640 1 --2- 172.21.15.88:0/1295791590 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd640b5be0 0x7efd640cb170 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:46.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.810+0000 7efd68f59640 1 --2- 172.21.15.88:0/1295791590 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd640b29a0 0x7efd6414da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:46.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.810+0000 7efd68f59640 1 -- 172.21.15.88:0/1295791590 >> 172.21.15.88:0/1295791590 conn(0x7efd6406bc10 msgr2=0x7efd640cc300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:46.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.810+0000 7efd68f59640 1 -- 172.21.15.88:0/1295791590 shutdown_connections 2024-09-16T01:39:46.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:46.810+0000 7efd68f59640 1 -- 172.21.15.88:0/1295791590 wait complete. 2024-09-16T01:39:47.189 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:39:47.192 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:39:47.225 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:39:14.808422+0000', 'modified': '2024-09-16T01:39:46.253297+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 73, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6414}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6414': {'gid': 6414, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.88:6837/2944989249', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6835', 'nonce': 2944989249}, {'type': 'v1', 'addr': '172.21.15.88:6837', 'nonce': 2944989249}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-16T01:39:47.225 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6414:up:replay 2024-09-16T01:39:48.226 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:39:48.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.332+0000 7ff2643c4640 1 Processor -- start 2024-09-16T01:39:48.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.332+0000 7ff2643c4640 1 -- start start 2024-09-16T01:39:48.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.333+0000 7ff2643c4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff25c073080 0x7ff25c0b5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:48.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.333+0000 7ff2643c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff25c15c670 0x7ff25c073460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:48.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.333+0000 7ff2643c4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff25c0b1990 0x7ff25c0d10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:48.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.333+0000 7ff2643c4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff25c0d0c60 con 0x7ff25c0b1990 2024-09-16T01:39:48.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.333+0000 7ff2643c4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff25c071750 con 0x7ff25c073080 2024-09-16T01:39:48.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.333+0000 7ff2643c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff25c0b59a0 con 0x7ff25c15c670 2024-09-16T01:39:48.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.333+0000 7ff262961640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff25c0b1990 0x7ff25c0d10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:48.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.334+0000 7ff262961640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff25c0b1990 0x7ff25c0d10f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46518/0 (socket says 172.21.15.88:46518) 2024-09-16T01:39:48.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.334+0000 7ff262961640 1 -- 172.21.15.88:0/3917883925 learned_addr learned my addr 172.21.15.88:0/3917883925 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:48.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.334+0000 7ff262160640 1 --2- 172.21.15.88:0/3917883925 >> v2:172.21.15.154:3300/0 conn(0x7ff25c073080 0x7ff25c0b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:48.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.334+0000 7ff26195f640 1 --2- 172.21.15.88:0/3917883925 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff25c15c670 0x7ff25c073460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:48.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.334+0000 7ff262961640 1 -- 172.21.15.88:0/3917883925 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff25c15c670 msgr2=0x7ff25c073460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:48.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.334+0000 7ff262961640 1 --2- 172.21.15.88:0/3917883925 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff25c15c670 0x7ff25c073460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:48.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.334+0000 7ff262961640 1 -- 172.21.15.88:0/3917883925 >> v2:172.21.15.154:3300/0 conn(0x7ff25c073080 msgr2=0x7ff25c0b5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:48.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.334+0000 7ff262961640 1 --2- 172.21.15.88:0/3917883925 >> v2:172.21.15.154:3300/0 conn(0x7ff25c073080 0x7ff25c0b5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:48.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.334+0000 7ff262961640 1 -- 172.21.15.88:0/3917883925 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff25c0d0370 con 0x7ff25c0b1990 2024-09-16T01:39:48.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.335+0000 7ff262961640 1 --2- 172.21.15.88:0/3917883925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff25c0b1990 0x7ff25c0d10f0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7ff25c069220 tx=0x7ff25402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e3305b936c80c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:48.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.335+0000 7ff26115e640 1 -- 172.21.15.88:0/3917883925 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff254037070 con 0x7ff25c0b1990 2024-09-16T01:39:48.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.335+0000 7ff26115e640 1 -- 172.21.15.88:0/3917883925 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff254032440 con 0x7ff25c0b1990 2024-09-16T01:39:48.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.335+0000 7ff26115e640 1 -- 172.21.15.88:0/3917883925 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff254032810 con 0x7ff25c0b1990 2024-09-16T01:39:48.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.335+0000 7ff2643c4640 1 -- 172.21.15.88:0/3917883925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff25c0b1990 msgr2=0x7ff25c0d10f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:48.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.335+0000 7ff2643c4640 1 --2- 172.21.15.88:0/3917883925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff25c0b1990 0x7ff25c0d10f0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7ff25c069220 tx=0x7ff25402a1e0 comp rx=0 tx=0).stop 2024-09-16T01:39:48.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.335+0000 7ff2643c4640 1 -- 172.21.15.88:0/3917883925 shutdown_connections 2024-09-16T01:39:48.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.335+0000 7ff2643c4640 1 --2- 172.21.15.88:0/3917883925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff25c0b1990 0x7ff25c0d10f0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:48.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.335+0000 7ff2643c4640 1 --2- 172.21.15.88:0/3917883925 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff25c15c670 0x7ff25c073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:48.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.335+0000 7ff2643c4640 1 --2- 172.21.15.88:0/3917883925 >> v2:172.21.15.154:3300/0 conn(0x7ff25c073080 0x7ff25c0b5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:48.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.335+0000 7ff2643c4640 1 -- 172.21.15.88:0/3917883925 >> 172.21.15.88:0/3917883925 conn(0x7ff25c0d24d0 msgr2=0x7ff25c0d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:48.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.335+0000 7ff2643c4640 1 -- 172.21.15.88:0/3917883925 shutdown_connections 2024-09-16T01:39:48.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.335+0000 7ff2643c4640 1 -- 172.21.15.88:0/3917883925 wait complete. 2024-09-16T01:39:48.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.337+0000 7ff2643c4640 1 Processor -- start 2024-09-16T01:39:48.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.337+0000 7ff2643c4640 1 -- start start 2024-09-16T01:39:48.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.337+0000 7ff2643c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff25c073080 0x7ff25c06dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:48.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.337+0000 7ff2643c4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff25c0b1990 0x7ff25c0afe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:48.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.337+0000 7ff2643c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff25c15c670 0x7ff25c0b0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:48.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.337+0000 7ff2643c4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c071470 con 0x7ff25c0b1990 2024-09-16T01:39:48.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.337+0000 7ff2643c4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c071190 con 0x7ff25c073080 2024-09-16T01:39:48.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.337+0000 7ff2643c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff25c0d0610 con 0x7ff25c15c670 2024-09-16T01:39:48.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.337+0000 7ff26195f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff25c0b1990 0x7ff25c0afe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:48.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.337+0000 7ff26195f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff25c0b1990 0x7ff25c0afe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46534/0 (socket says 172.21.15.88:46534) 2024-09-16T01:39:48.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.337+0000 7ff26195f640 1 -- 172.21.15.88:0/3017434711 learned_addr learned my addr 172.21.15.88:0/3017434711 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:48.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.338+0000 7ff262160640 1 --2- 172.21.15.88:0/3017434711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff25c073080 0x7ff25c06dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:48.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.338+0000 7ff262961640 1 --2- 172.21.15.88:0/3017434711 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff25c15c670 0x7ff25c0b0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:48.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.338+0000 7ff26195f640 1 -- 172.21.15.88:0/3017434711 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff25c15c670 msgr2=0x7ff25c0b0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:48.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.338+0000 7ff26195f640 1 --2- 172.21.15.88:0/3017434711 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff25c15c670 0x7ff25c0b0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:48.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.338+0000 7ff26195f640 1 -- 172.21.15.88:0/3017434711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff25c073080 msgr2=0x7ff25c06dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:48.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.338+0000 7ff26195f640 1 --2- 172.21.15.88:0/3017434711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff25c073080 0x7ff25c06dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:48.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.338+0000 7ff26195f640 1 -- 172.21.15.88:0/3017434711 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff25c06ec50 con 0x7ff25c0b1990 2024-09-16T01:39:48.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.338+0000 7ff26195f640 1 --2- 172.21.15.88:0/3017434711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff25c0b1990 0x7ff25c0afe10 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7ff25000c5c0 tx=0x7ff25000ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:48.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.338+0000 7ff25affd640 1 -- 172.21.15.88:0/3017434711 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff250015070 con 0x7ff25c0b1990 2024-09-16T01:39:48.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.338+0000 7ff25affd640 1 -- 172.21.15.88:0/3017434711 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff250004220 con 0x7ff25c0b1990 2024-09-16T01:39:48.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.338+0000 7ff25affd640 1 -- 172.21.15.88:0/3017434711 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff250003610 con 0x7ff25c0b1990 2024-09-16T01:39:48.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.338+0000 7ff2643c4640 1 -- 172.21.15.88:0/3017434711 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff25c150db0 con 0x7ff25c0b1990 2024-09-16T01:39:48.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.338+0000 7ff2643c4640 1 -- 172.21.15.88:0/3017434711 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff25c06e970 con 0x7ff25c0b1990 2024-09-16T01:39:48.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.339+0000 7ff2643c4640 1 -- 172.21.15.88:0/3017434711 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff25c0d08f0 con 0x7ff25c0b1990 2024-09-16T01:39:48.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.340+0000 7ff25affd640 1 -- 172.21.15.88:0/3017434711 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff2500037b0 con 0x7ff25c0b1990 2024-09-16T01:39:48.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.340+0000 7ff25affd640 1 --2- 172.21.15.88:0/3017434711 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff24806c2e0 0x7ff24806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:48.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.340+0000 7ff25affd640 1 -- 172.21.15.88:0/3017434711 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5795+0+0 (secure 0 0 0) 0x7ff25008b1d0 con 0x7ff25c0b1990 2024-09-16T01:39:48.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.340+0000 7ff262160640 1 --2- 172.21.15.88:0/3017434711 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff24806c2e0 0x7ff24806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:48.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.341+0000 7ff262160640 1 --2- 172.21.15.88:0/3017434711 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff24806c2e0 0x7ff24806e7a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff24c004410 tx=0x7ff24c0039f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:48.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.345+0000 7ff25affd640 1 -- 172.21.15.88:0/3017434711 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff2500598e0 con 0x7ff25c0b1990 2024-09-16T01:39:48.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.543+0000 7ff2643c4640 1 -- 172.21.15.88:0/3017434711 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff25c0d3050 con 0x7ff25c0b1990 2024-09-16T01:39:48.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.544+0000 7ff25affd640 1 -- 172.21.15.88:0/3017434711 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+5226 (secure 0 0 0) 0x7ff2500591e0 con 0x7ff25c0b1990 2024-09-16T01:39:48.546 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:48.546 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":63,"btime":"2024-09-16T01:39:48:265577+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5725,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2696547723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2696547723},{"type":"v1","addr":"172.21.15.154:6834","nonce":2696547723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54},{"gid":5863,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2167482844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2167482844},{"type":"v1","addr":"172.21.15.154:6835","nonce":2167482844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62},{"gid":6411,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3276199399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3276199399},{"type":"v1","addr":"172.21.15.88:6836","nonce":3276199399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:39:14.808422+0000","modified":"2024-09-16T01:39:47.266966+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6414},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":61,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.88:6837/2944989249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2944989249},{"type":"v1","addr":"172.21.15.88:6837","nonce":2944989249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":5}]} 2024-09-16T01:39:48.546 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 63 2024-09-16T01:39:48.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.546+0000 7ff2643c4640 1 -- 172.21.15.88:0/3017434711 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff24806c2e0 msgr2=0x7ff24806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:48.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.546+0000 7ff2643c4640 1 --2- 172.21.15.88:0/3017434711 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff24806c2e0 0x7ff24806e7a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff24c004410 tx=0x7ff24c0039f0 comp rx=0 tx=0).stop 2024-09-16T01:39:48.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.546+0000 7ff2643c4640 1 -- 172.21.15.88:0/3017434711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff25c0b1990 msgr2=0x7ff25c0afe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:48.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.546+0000 7ff2643c4640 1 --2- 172.21.15.88:0/3017434711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff25c0b1990 0x7ff25c0afe10 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7ff25000c5c0 tx=0x7ff25000ca80 comp rx=0 tx=0).stop 2024-09-16T01:39:48.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.546+0000 7ff2643c4640 1 -- 172.21.15.88:0/3017434711 shutdown_connections 2024-09-16T01:39:48.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.546+0000 7ff2643c4640 1 --2- 172.21.15.88:0/3017434711 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff24806c2e0 0x7ff24806e7a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:48.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.546+0000 7ff2643c4640 1 --2- 172.21.15.88:0/3017434711 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff25c15c670 0x7ff25c0b0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:48.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.546+0000 7ff2643c4640 1 --2- 172.21.15.88:0/3017434711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff25c0b1990 0x7ff25c0afe10 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:48.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.546+0000 7ff2643c4640 1 --2- 172.21.15.88:0/3017434711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff25c073080 0x7ff25c06dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:48.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.546+0000 7ff2643c4640 1 -- 172.21.15.88:0/3017434711 >> 172.21.15.88:0/3017434711 conn(0x7ff25c0d24d0 msgr2=0x7ff25c0ce970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:48.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.546+0000 7ff2643c4640 1 -- 172.21.15.88:0/3017434711 shutdown_connections 2024-09-16T01:39:48.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:48.546+0000 7ff2643c4640 1 -- 172.21.15.88:0/3017434711 wait complete. 2024-09-16T01:39:48.960 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:39:14.808422+0000', 'modified': '2024-09-16T01:39:47.266966+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 73, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6414}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6414': {'gid': 6414, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:rejoin', 'state_seq': 11, 'addr': '172.21.15.88:6837/2944989249', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6835', 'nonce': 2944989249}, {'type': 'v1', 'addr': '172.21.15.88:6837', 'nonce': 2944989249}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-16T01:39:48.961 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6414:up:rejoin 2024-09-16T01:39:49.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:39:49.962 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:39:50.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.071+0000 7fe449d11640 1 Processor -- start 2024-09-16T01:39:50.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.071+0000 7fe449d11640 1 -- start start 2024-09-16T01:39:50.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.071+0000 7fe449d11640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe44406e780 0x7fe4440ae890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:50.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.071+0000 7fe449d11640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe444073130 0x7fe444073580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:50.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.071+0000 7fe449d11640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4440b5220 con 0x7fe44415e560 2024-09-16T01:39:50.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.071+0000 7fe449d11640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe44406ebc0 con 0x7fe44406e780 2024-09-16T01:39:50.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.071+0000 7fe449d11640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe444073ac0 con 0x7fe444073130 2024-09-16T01:39:50.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.072+0000 7fe442ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe44406e780 0x7fe4440ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:50.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.072+0000 7fe4437fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe44415e560 0x7fe444072220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36148/0 (socket says 172.21.15.88:36148) 2024-09-16T01:39:50.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.072+0000 7fe4437fe640 1 -- 172.21.15.88:0/3589228018 learned_addr learned my addr 172.21.15.88:0/3589228018 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:50.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.072+0000 7fe442ffd640 1 -- 172.21.15.88:0/3589228018 >> v2:172.21.15.154:3300/0 conn(0x7fe444073130 msgr2=0x7fe444073580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:50.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.072+0000 7fe443fff640 1 --2- 172.21.15.88:0/3589228018 >> v2:172.21.15.154:3300/0 conn(0x7fe444073130 0x7fe444073580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:50.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.072+0000 7fe4427fc640 1 -- 172.21.15.88:0/3589228018 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 148965202 0 0) 0x7fe4440b5220 con 0x7fe44415e560 2024-09-16T01:39:50.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.072+0000 7fe442ffd640 1 --2- 172.21.15.88:0/3589228018 >> v2:172.21.15.154:3300/0 conn(0x7fe444073130 0x7fe444073580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.072+0000 7fe442ffd640 1 -- 172.21.15.88:0/3589228018 >> v1:172.21.15.88:6789/0 conn(0x7fe44415e560 legacy=0x7fe444072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:50.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.072+0000 7fe442ffd640 1 -- 172.21.15.88:0/3589228018 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4440b54c0 con 0x7fe44406e780 2024-09-16T01:39:50.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.072+0000 7fe443fff640 1 --2- 172.21.15.88:0/3589228018 >> v2:172.21.15.154:3300/0 conn(0x7fe444073130 0x7fe444073580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:50.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.073+0000 7fe442ffd640 1 --2- 172.21.15.88:0/3589228018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe44406e780 0x7fe4440ae890 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fe4440b44f0 tx=0x7fe42c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8853e39a55ec74b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:50.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.073+0000 7fe4427fc640 1 -- 172.21.15.88:0/3589228018 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42c037070 con 0x7fe44406e780 2024-09-16T01:39:50.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.073+0000 7fe4427fc640 1 -- 172.21.15.88:0/3589228018 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe42c032440 con 0x7fe44406e780 2024-09-16T01:39:50.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.073+0000 7fe4427fc640 1 -- 172.21.15.88:0/3589228018 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42c02ae60 con 0x7fe44406e780 2024-09-16T01:39:50.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.073+0000 7fe449d11640 1 -- 172.21.15.88:0/3589228018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe44406e780 msgr2=0x7fe4440ae890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:50.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.073+0000 7fe449d11640 1 --2- 172.21.15.88:0/3589228018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe44406e780 0x7fe4440ae890 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fe4440b44f0 tx=0x7fe42c02a110 comp rx=0 tx=0).stop 2024-09-16T01:39:50.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.073+0000 7fe449d11640 1 -- 172.21.15.88:0/3589228018 shutdown_connections 2024-09-16T01:39:50.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.073+0000 7fe449d11640 1 --2- 172.21.15.88:0/3589228018 >> v2:172.21.15.154:3300/0 conn(0x7fe444073130 0x7fe444073580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.073+0000 7fe449d11640 1 --2- 172.21.15.88:0/3589228018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe44406e780 0x7fe4440ae890 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.073+0000 7fe449d11640 1 -- 172.21.15.88:0/3589228018 >> 172.21.15.88:0/3589228018 conn(0x7fe44406bc10 msgr2=0x7fe44406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:50.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.073+0000 7fe449d11640 1 -- 172.21.15.88:0/3589228018 shutdown_connections 2024-09-16T01:39:50.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.073+0000 7fe449d11640 1 -- 172.21.15.88:0/3589228018 wait complete. 2024-09-16T01:39:50.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.075+0000 7fe449d11640 1 Processor -- start 2024-09-16T01:39:50.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.075+0000 7fe449d11640 1 -- start start 2024-09-16T01:39:50.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.076+0000 7fe449d11640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe44406e780 0x7fe444157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:50.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.076+0000 7fe449d11640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe444073130 0x7fe4440cf410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:50.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.076+0000 7fe449d11640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe44415e560 0x7fe4440cd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:50.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.076+0000 7fe449d11640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe4440b5d60 con 0x7fe44415e560 2024-09-16T01:39:50.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.076+0000 7fe449d11640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe4440b5a80 con 0x7fe444073130 2024-09-16T01:39:50.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.076+0000 7fe449d11640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe44415e940 con 0x7fe44406e780 2024-09-16T01:39:50.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.076+0000 7fe443fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe44415e560 0x7fe4440cd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:50.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.076+0000 7fe443fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe44415e560 0x7fe4440cd5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46544/0 (socket says 172.21.15.88:46544) 2024-09-16T01:39:50.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.076+0000 7fe443fff640 1 -- 172.21.15.88:0/1418782278 learned_addr learned my addr 172.21.15.88:0/1418782278 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:50.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.076+0000 7fe442ffd640 1 --2- 172.21.15.88:0/1418782278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe444073130 0x7fe4440cf410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:50.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.076+0000 7fe4437fe640 1 --2- 172.21.15.88:0/1418782278 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe44406e780 0x7fe444157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:50.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.076+0000 7fe443fff640 1 -- 172.21.15.88:0/1418782278 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe44406e780 msgr2=0x7fe444157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:50.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.076+0000 7fe443fff640 1 --2- 172.21.15.88:0/1418782278 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe44406e780 0x7fe444157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.076+0000 7fe443fff640 1 -- 172.21.15.88:0/1418782278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe444073130 msgr2=0x7fe4440cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:50.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.076+0000 7fe443fff640 1 --2- 172.21.15.88:0/1418782278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe444073130 0x7fe4440cf410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.076+0000 7fe443fff640 1 -- 172.21.15.88:0/1418782278 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4440cfe60 con 0x7fe44415e560 2024-09-16T01:39:50.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.076+0000 7fe4437fe640 1 --2- 172.21.15.88:0/1418782278 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe44406e780 0x7fe444157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:50.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.076+0000 7fe443fff640 1 --2- 172.21.15.88:0/1418782278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe44415e560 0x7fe4440cd5c0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fe43c00cf20 tx=0x7fe43c00a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:50.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.076+0000 7fe43bfff640 1 -- 172.21.15.88:0/1418782278 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43c01a070 con 0x7fe44415e560 2024-09-16T01:39:50.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.076+0000 7fe43bfff640 1 -- 172.21.15.88:0/1418782278 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe43c00d840 con 0x7fe44415e560 2024-09-16T01:39:50.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.076+0000 7fe43bfff640 1 -- 172.21.15.88:0/1418782278 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43c00dc30 con 0x7fe44415e560 2024-09-16T01:39:50.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.076+0000 7fe449d11640 1 -- 172.21.15.88:0/1418782278 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4440d0140 con 0x7fe44415e560 2024-09-16T01:39:50.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.077+0000 7fe449d11640 1 -- 172.21.15.88:0/1418782278 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe444155fd0 con 0x7fe44415e560 2024-09-16T01:39:50.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.078+0000 7fe449d11640 1 -- 172.21.15.88:0/1418782278 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe444073580 con 0x7fe44415e560 2024-09-16T01:39:50.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.078+0000 7fe43bfff640 1 -- 172.21.15.88:0/1418782278 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe43c01d070 con 0x7fe44415e560 2024-09-16T01:39:50.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.078+0000 7fe43bfff640 1 --2- 172.21.15.88:0/1418782278 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe42406c2b0 0x7fe42406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:50.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.079+0000 7fe4437fe640 1 --2- 172.21.15.88:0/1418782278 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe42406c2b0 0x7fe42406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:50.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.079+0000 7fe43bfff640 1 -- 172.21.15.88:0/1418782278 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5795+0+0 (secure 0 0 0) 0x7fe43c08bc00 con 0x7fe44415e560 2024-09-16T01:39:50.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.079+0000 7fe4437fe640 1 --2- 172.21.15.88:0/1418782278 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe42406c2b0 0x7fe42406e770 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe434000f60 tx=0x7fe434007ca0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:50.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.083+0000 7fe43bfff640 1 -- 172.21.15.88:0/1418782278 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe43c092080 con 0x7fe44415e560 2024-09-16T01:39:50.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.284+0000 7fe449d11640 1 -- 172.21.15.88:0/1418782278 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe4440b0b10 con 0x7fe44415e560 2024-09-16T01:39:50.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.284+0000 7fe43bfff640 1 -- 172.21.15.88:0/1418782278 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+5233 (secure 0 0 0) 0x7fe43c059d50 con 0x7fe44415e560 2024-09-16T01:39:50.286 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:50.286 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":64,"btime":"2024-09-16T01:39:49:271443+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5725,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2696547723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2696547723},{"type":"v1","addr":"172.21.15.154:6834","nonce":2696547723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54},{"gid":5863,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2167482844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2167482844},{"type":"v1","addr":"172.21.15.154:6835","nonce":2167482844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62},{"gid":6411,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3276199399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3276199399},{"type":"v1","addr":"172.21.15.88:6836","nonce":3276199399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:39:14.808422+0000","modified":"2024-09-16T01:39:49.271439+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6414},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":12,"addr":"172.21.15.88:6837/2944989249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2944989249},{"type":"v1","addr":"172.21.15.88:6837","nonce":2944989249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6414,"qdb_cluster":[6414]},"id":5}]} 2024-09-16T01:39:50.286 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 64 2024-09-16T01:39:50.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.287+0000 7fe449d11640 1 -- 172.21.15.88:0/1418782278 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe42406c2b0 msgr2=0x7fe42406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:50.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.287+0000 7fe449d11640 1 --2- 172.21.15.88:0/1418782278 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe42406c2b0 0x7fe42406e770 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe434000f60 tx=0x7fe434007ca0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.287+0000 7fe449d11640 1 -- 172.21.15.88:0/1418782278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe44415e560 msgr2=0x7fe4440cd5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:50.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.287+0000 7fe449d11640 1 --2- 172.21.15.88:0/1418782278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe44415e560 0x7fe4440cd5c0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fe43c00cf20 tx=0x7fe43c00a7d0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.287+0000 7fe449d11640 1 -- 172.21.15.88:0/1418782278 shutdown_connections 2024-09-16T01:39:50.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.287+0000 7fe449d11640 1 --2- 172.21.15.88:0/1418782278 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe42406c2b0 0x7fe42406e770 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.287+0000 7fe449d11640 1 --2- 172.21.15.88:0/1418782278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe44415e560 0x7fe4440cd5c0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.287+0000 7fe449d11640 1 --2- 172.21.15.88:0/1418782278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe444073130 0x7fe4440cf410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.287+0000 7fe449d11640 1 --2- 172.21.15.88:0/1418782278 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe44406e780 0x7fe444157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.287+0000 7fe449d11640 1 -- 172.21.15.88:0/1418782278 >> 172.21.15.88:0/1418782278 conn(0x7fe44406bc10 msgr2=0x7fe4440b6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:50.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.287+0000 7fe449d11640 1 -- 172.21.15.88:0/1418782278 shutdown_connections 2024-09-16T01:39:50.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.287+0000 7fe449d11640 1 -- 172.21.15.88:0/1418782278 wait complete. 2024-09-16T01:39:50.702 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:39:14.808422+0000', 'modified': '2024-09-16T01:39:49.271439+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 73, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6414}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6414': {'gid': 6414, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.88:6837/2944989249', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6835', 'nonce': 2944989249}, {'type': 'v1', 'addr': '172.21.15.88:6837', 'nonce': 2944989249}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6414, 'qdb_cluster': [6414]} 2024-09-16T01:39:50.703 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T01:39:50.704 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-16T01:39:50.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.805+0000 7fca1d6f8640 1 Processor -- start 2024-09-16T01:39:50.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.805+0000 7fca1d6f8640 1 -- start start 2024-09-16T01:39:50.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.805+0000 7fca1d6f8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fca180cf1f0 0x7fca180b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:50.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.805+0000 7fca1d6f8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca180cf860 con 0x7fca1815e7a0 2024-09-16T01:39:50.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.805+0000 7fca1d6f8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca180cfb40 con 0x7fca180b6720 2024-09-16T01:39:50.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.805+0000 7fca1d6f8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fca180b6b90 con 0x7fca180cf1f0 2024-09-16T01:39:50.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.806+0000 7fca167fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fca180cf1f0 0x7fca180b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:50.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.806+0000 7fca16ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fca1815e7a0 0x7fca1806caa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55912/0 (socket says 172.21.15.88:55912) 2024-09-16T01:39:50.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.806+0000 7fca16ffd640 1 -- 172.21.15.88:0/2518755283 learned_addr learned my addr 172.21.15.88:0/2518755283 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:50.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.806+0000 7fca167fc640 1 -- 172.21.15.88:0/2518755283 >> v1:172.21.15.154:6789/0 conn(0x7fca180b6720 legacy=0x7fca180d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:50.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.806+0000 7fca167fc640 1 -- 172.21.15.88:0/2518755283 >> v1:172.21.15.88:6789/0 conn(0x7fca1815e7a0 legacy=0x7fca1806caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:50.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.806+0000 7fca167fc640 1 -- 172.21.15.88:0/2518755283 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca180cfe20 con 0x7fca180cf1f0 2024-09-16T01:39:50.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.807+0000 7fca167fc640 1 --2- 172.21.15.88:0/2518755283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca180cf1f0 0x7fca180b61e0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fca0c0097a0 tx=0x7fca0c02a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=196bd82008c534fa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:50.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.807+0000 7fca15ffb640 1 -- 172.21.15.88:0/2518755283 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0c037070 con 0x7fca180cf1f0 2024-09-16T01:39:50.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.807+0000 7fca15ffb640 1 -- 172.21.15.88:0/2518755283 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca0c02acd0 con 0x7fca180cf1f0 2024-09-16T01:39:50.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.807+0000 7fca15ffb640 1 -- 172.21.15.88:0/2518755283 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0c032660 con 0x7fca180cf1f0 2024-09-16T01:39:50.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.807+0000 7fca1d6f8640 1 -- 172.21.15.88:0/2518755283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca180cf1f0 msgr2=0x7fca180b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:50.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.807+0000 7fca1d6f8640 1 --2- 172.21.15.88:0/2518755283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca180cf1f0 0x7fca180b61e0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fca0c0097a0 tx=0x7fca0c02a040 comp rx=0 tx=0).stop 2024-09-16T01:39:50.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.807+0000 7fca1d6f8640 1 -- 172.21.15.88:0/2518755283 shutdown_connections 2024-09-16T01:39:50.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.807+0000 7fca1d6f8640 1 --2- 172.21.15.88:0/2518755283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca180cf1f0 0x7fca180b61e0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.807+0000 7fca1d6f8640 1 -- 172.21.15.88:0/2518755283 >> 172.21.15.88:0/2518755283 conn(0x7fca180d11e0 msgr2=0x7fca180d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:50.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.807+0000 7fca1d6f8640 1 -- 172.21.15.88:0/2518755283 shutdown_connections 2024-09-16T01:39:50.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.807+0000 7fca1d6f8640 1 -- 172.21.15.88:0/2518755283 wait complete. 2024-09-16T01:39:50.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.809+0000 7fca1d6f8640 1 Processor -- start 2024-09-16T01:39:50.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.809+0000 7fca1d6f8640 1 -- start start 2024-09-16T01:39:50.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.809+0000 7fca1d6f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca180b6720 0x7fca181626f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:50.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.809+0000 7fca1d6f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca180cf1f0 0x7fca181f8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:50.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.809+0000 7fca1d6f8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca1815e7a0 0x7fca18156ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:50.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.809+0000 7fca1d6f8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fca180b7a00 con 0x7fca1815e7a0 2024-09-16T01:39:50.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.809+0000 7fca1d6f8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fca180d2ca0 con 0x7fca180b6720 2024-09-16T01:39:50.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.809+0000 7fca1d6f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fca1815eb80 con 0x7fca180cf1f0 2024-09-16T01:39:50.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.810+0000 7fca177fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca1815e7a0 0x7fca18156ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:50.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.810+0000 7fca177fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca1815e7a0 0x7fca18156ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59940/0 (socket says 172.21.15.88:59940) 2024-09-16T01:39:50.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.810+0000 7fca177fe640 1 -- 172.21.15.88:0/1384295982 learned_addr learned my addr 172.21.15.88:0/1384295982 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:50.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.810+0000 7fca167fc640 1 --2- 172.21.15.88:0/1384295982 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca180cf1f0 0x7fca181f8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:50.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.810+0000 7fca16ffd640 1 --2- 172.21.15.88:0/1384295982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca180b6720 0x7fca181626f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:50.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.810+0000 7fca177fe640 1 -- 172.21.15.88:0/1384295982 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca180cf1f0 msgr2=0x7fca181f8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:50.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.810+0000 7fca177fe640 1 --2- 172.21.15.88:0/1384295982 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca180cf1f0 0x7fca181f8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.810+0000 7fca177fe640 1 -- 172.21.15.88:0/1384295982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca180b6720 msgr2=0x7fca181626f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:50.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.810+0000 7fca177fe640 1 --2- 172.21.15.88:0/1384295982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca180b6720 0x7fca181626f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.810+0000 7fca177fe640 1 -- 172.21.15.88:0/1384295982 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca181f8f90 con 0x7fca1815e7a0 2024-09-16T01:39:50.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.810+0000 7fca177fe640 1 --2- 172.21.15.88:0/1384295982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca1815e7a0 0x7fca18156ad0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fca1000b970 tx=0x7fca1000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:50.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.810+0000 7fc9ff7fe640 1 -- 172.21.15.88:0/1384295982 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca10019070 con 0x7fca1815e7a0 2024-09-16T01:39:50.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.810+0000 7fc9ff7fe640 1 -- 172.21.15.88:0/1384295982 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca1000cb40 con 0x7fca1815e7a0 2024-09-16T01:39:50.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.810+0000 7fc9ff7fe640 1 -- 172.21.15.88:0/1384295982 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca10014440 con 0x7fca1815e7a0 2024-09-16T01:39:50.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.810+0000 7fca1d6f8640 1 -- 172.21.15.88:0/1384295982 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca18157010 con 0x7fca1815e7a0 2024-09-16T01:39:50.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.810+0000 7fca1d6f8640 1 -- 172.21.15.88:0/1384295982 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca180c86a0 con 0x7fca1815e7a0 2024-09-16T01:39:50.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.812+0000 7fc9ff7fe640 1 -- 172.21.15.88:0/1384295982 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fca1001c070 con 0x7fca1815e7a0 2024-09-16T01:39:50.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.812+0000 7fc9ff7fe640 1 --2- 172.21.15.88:0/1384295982 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fca0406c2b0 0x7fca0406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:50.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.812+0000 7fc9ff7fe640 1 -- 172.21.15.88:0/1384295982 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5795+0+0 (secure 0 0 0) 0x7fca1008b100 con 0x7fca1815e7a0 2024-09-16T01:39:50.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.812+0000 7fca16ffd640 1 --2- 172.21.15.88:0/1384295982 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fca0406c2b0 0x7fca0406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:50.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.813+0000 7fca16ffd640 1 --2- 172.21.15.88:0/1384295982 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fca0406c2b0 0x7fca0406e770 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fca08000f60 tx=0x7fca08007ca0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:50.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.813+0000 7fca20495740 1 Processor -- start 2024-09-16T01:39:50.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.813+0000 7fca20495740 1 -- start start 2024-09-16T01:39:50.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.814+0000 7fca20495740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2d2c0 0x558e0cc72a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:50.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.814+0000 7fca20495740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0cc2aa80 0x558e0cc2d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:50.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.814+0000 7fca20495740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558e0cc2ea50 0x558e0cc278b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:50.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.814+0000 7fca20495740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558e0cb8f520 con 0x558e0cc2d2c0 2024-09-16T01:39:50.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.814+0000 7fca20495740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558e0cb373e0 con 0x558e0cc2aa80 2024-09-16T01:39:50.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.814+0000 7fca20495740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x558e0cb8b4e0 con 0x558e0cc2ea50 2024-09-16T01:39:50.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.814+0000 7fca16ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2d2c0 0x558e0cc72a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:50.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.814+0000 7fca16ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2d2c0 0x558e0cc72a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59956/0 (socket says 172.21.15.88:59956) 2024-09-16T01:39:50.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.814+0000 7fca16ffd640 1 -- 172.21.15.88:0/792947917 learned_addr learned my addr 172.21.15.88:0/792947917 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:50.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.814+0000 7fca167fc640 1 --2- 172.21.15.88:0/792947917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0cc2aa80 0x558e0cc2d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:50.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.814+0000 7fca177fe640 1 --2- 172.21.15.88:0/792947917 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558e0cc2ea50 0x558e0cc278b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:50.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.814+0000 7fca16ffd640 1 -- 172.21.15.88:0/792947917 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558e0cc2ea50 msgr2=0x558e0cc278b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:50.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.814+0000 7fca16ffd640 1 --2- 172.21.15.88:0/792947917 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558e0cc2ea50 0x558e0cc278b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.814+0000 7fca16ffd640 1 -- 172.21.15.88:0/792947917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0cc2aa80 msgr2=0x558e0cc2d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:50.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.814+0000 7fca16ffd640 1 --2- 172.21.15.88:0/792947917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0cc2aa80 0x558e0cc2d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.814+0000 7fca16ffd640 1 -- 172.21.15.88:0/792947917 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e0cb522f0 con 0x558e0cc2d2c0 2024-09-16T01:39:50.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.814+0000 7fca167fc640 1 --2- 172.21.15.88:0/792947917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0cc2aa80 0x558e0cc2d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:50.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.814+0000 7fca16ffd640 1 --2- 172.21.15.88:0/792947917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2d2c0 0x558e0cc72a60 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fca080076b0 tx=0x7fca08012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3fe9e0d527e64d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:50.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.815+0000 7fc9fd7fa640 1 -- 172.21.15.88:0/792947917 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca080150d0 con 0x558e0cc2d2c0 2024-09-16T01:39:50.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.815+0000 7fc9fd7fa640 1 -- 172.21.15.88:0/792947917 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca08015270 con 0x558e0cc2d2c0 2024-09-16T01:39:50.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.815+0000 7fc9fd7fa640 1 -- 172.21.15.88:0/792947917 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca08015dd0 con 0x558e0cc2d2c0 2024-09-16T01:39:50.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.815+0000 7fca20495740 1 -- 172.21.15.88:0/792947917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2d2c0 msgr2=0x558e0cc72a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:50.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.815+0000 7fca20495740 1 --2- 172.21.15.88:0/792947917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2d2c0 0x558e0cc72a60 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fca080076b0 tx=0x7fca08012000 comp rx=0 tx=0).stop 2024-09-16T01:39:50.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.815+0000 7fca20495740 1 -- 172.21.15.88:0/792947917 shutdown_connections 2024-09-16T01:39:50.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.815+0000 7fca20495740 1 --2- 172.21.15.88:0/792947917 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558e0cc2ea50 0x558e0cc278b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.815+0000 7fca20495740 1 --2- 172.21.15.88:0/792947917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0cc2aa80 0x558e0cc2d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.815+0000 7fca20495740 1 --2- 172.21.15.88:0/792947917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2d2c0 0x558e0cc72a60 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.815+0000 7fca20495740 1 -- 172.21.15.88:0/792947917 >> 172.21.15.88:0/792947917 conn(0x558e0cc29a80 msgr2=0x558e0cc71f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:50.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.815+0000 7fca20495740 1 -- 172.21.15.88:0/792947917 shutdown_connections 2024-09-16T01:39:50.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.815+0000 7fca20495740 1 -- 172.21.15.88:0/792947917 wait complete. 2024-09-16T01:39:50.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.816+0000 7fca20495740 1 -- start start 2024-09-16T01:39:50.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.816+0000 7fca20495740 1 Processor -- start 2024-09-16T01:39:50.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.817+0000 7fca20495740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558e0cc2d2c0 0x558e0cc4c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:50.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.817+0000 7fca20495740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0cc2ea50 0x558e0cc2be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:50.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.817+0000 7fca20495740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2c360 0x558e0cc4def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:50.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.817+0000 7fca20495740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558e0cc28f20 con 0x558e0cc2c360 2024-09-16T01:39:50.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.817+0000 7fca20495740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558e0cb18450 con 0x558e0cc2ea50 2024-09-16T01:39:50.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.817+0000 7fca20495740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x558e0cc73290 con 0x558e0cc2d2c0 2024-09-16T01:39:50.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.817+0000 7fca20495740 10 client.0 _ll_register_callbacks cb 0x558e0c8a1ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:39:50.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.817+0000 7fca177fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2c360 0x558e0cc4def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:50.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.817+0000 7fca177fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2c360 0x558e0cc4def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59960/0 (socket says 172.21.15.88:59960) 2024-09-16T01:39:50.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.817+0000 7fca177fe640 1 -- 172.21.15.88:0/1616222499 learned_addr learned my addr 172.21.15.88:0/1616222499 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:50.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.817+0000 7fca16ffd640 1 --2- 172.21.15.88:0/1616222499 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558e0cc2d2c0 0x558e0cc4c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:50.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.817+0000 7fca167fc640 1 --2- 172.21.15.88:0/1616222499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0cc2ea50 0x558e0cc2be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:50.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.817+0000 7fca177fe640 1 -- 172.21.15.88:0/1616222499 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558e0cc2d2c0 msgr2=0x558e0cc4c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:50.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.817+0000 7fca177fe640 1 --2- 172.21.15.88:0/1616222499 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558e0cc2d2c0 0x558e0cc4c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.817+0000 7fca177fe640 1 -- 172.21.15.88:0/1616222499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0cc2ea50 msgr2=0x558e0cc2be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:50.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.817+0000 7fca177fe640 1 --2- 172.21.15.88:0/1616222499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0cc2ea50 0x558e0cc2be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.817+0000 7fca177fe640 1 -- 172.21.15.88:0/1616222499 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e0c8a22d0 con 0x558e0cc2c360 2024-09-16T01:39:50.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.817+0000 7fca177fe640 1 --2- 172.21.15.88:0/1616222499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2c360 0x558e0cc4def0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fca10089530 tx=0x7fca100591c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aaf4fa1ac6eb0eb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:50.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.817+0000 7fc9f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:39:50.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.817+0000 7fc9f5ffb640 1 -- 172.21.15.88:0/1616222499 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca10061850 con 0x558e0cc2c360 2024-09-16T01:39:50.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.817+0000 7fc9f5ffb640 1 -- 172.21.15.88:0/1616222499 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 5795+0+0 (secure 0 0 0) 0x7fca100619f0 con 0x558e0cc2c360 2024-09-16T01:39:50.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.817+0000 7fca20495740 1 -- 172.21.15.88:0/1616222499 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca08015270 con 0x558e0cc2c360 2024-09-16T01:39:50.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.817+0000 7fc9f5ffb640 1 -- 172.21.15.88:0/1616222499 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7fca10089050 con 0x558e0cc2c360 2024-09-16T01:39:50.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.817+0000 7fca20495740 10 client.6687 fetch_fsmap learned FSMap version 64 2024-09-16T01:39:50.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.817+0000 7fca20495740 1 -- 172.21.15.88:0/1616222499 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) -- 0x558e0c96c6b0 con 0x558e0cc2c360 2024-09-16T01:39:50.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.818+0000 7fc9f5ffb640 1 -- 172.21.15.88:0/1616222499 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca08015270 con 0x558e0cc2c360 2024-09-16T01:39:50.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.818+0000 7fc9f5ffb640 1 -- 172.21.15.88:0/1616222499 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 64) ==== 3502+0+0 (secure 0 0 0) 0x7fca10058550 con 0x558e0cc2c360 2024-09-16T01:39:50.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.818+0000 7fca20495740 10 client.6687 fetch_fsmap finished waiting for FSMap version 64 2024-09-16T01:39:50.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.818+0000 7fca20495740 10 client.6687 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:39:50.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.818+0000 7fca20495740 20 client.6687 populate_metadata read hostname 'smithi088' 2024-09-16T01:39:50.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.818+0000 7fca20495740 1 --2- 172.21.15.88:0/1616222499 >> [v2:172.21.15.88:6835/2944989249,v1:172.21.15.88:6837/2944989249] conn(0x558e0cc26920 0x558e0cc26d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:50.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.818+0000 7fca20495740 4 client.6687 mds_command: new command op to 6414 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:39:50.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.818+0000 7fca20495740 1 -- 172.21.15.88:0/1616222499 --> [v2:172.21.15.88:6835/2944989249,v1:172.21.15.88:6837/2944989249] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e0cb9cc50 con 0x558e0cc26920 2024-09-16T01:39:50.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.818+0000 7fca16ffd640 1 --2- 172.21.15.88:0/1616222499 >> [v2:172.21.15.88:6835/2944989249,v1:172.21.15.88:6837/2944989249] conn(0x558e0cc26920 0x558e0cc26d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:50.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.818+0000 7fca16ffd640 1 --2- 172.21.15.88:0/1616222499 >> [v2:172.21.15.88:6835/2944989249,v1:172.21.15.88:6837/2944989249] conn(0x558e0cc26920 0x558e0cc26d00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8d22c9b799848d0 server_cookie=398b0ab148aaaa22 in_seq=0 out_seq=0 2024-09-16T01:39:50.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.818+0000 7fc9f5ffb640 10 client.6687 ms_handle_connect on v2:172.21.15.88:6835/2944989249 2024-09-16T01:39:50.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.819+0000 7fc9f5ffb640 1 -- 172.21.15.88:0/1616222499 <== mds.0 v2:172.21.15.88:6835/2944989249 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558e0cb9cc50 con 0x558e0cc26920 2024-09-16T01:39:50.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.819+0000 7fc9f5ffb640 10 client.6687 handle_command_reply: tid=0 2024-09-16T01:39:50.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.819+0000 7fca20495740 1 client.6687 shutdown 2024-09-16T01:39:50.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.819+0000 7fca20495740 1 -- 172.21.15.88:0/1616222499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2c360 msgr2=0x558e0cc4def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:50.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.819+0000 7fca20495740 1 --2- 172.21.15.88:0/1616222499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2c360 0x558e0cc4def0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fca10089530 tx=0x7fca100591c0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.819+0000 7fca20495740 1 -- 172.21.15.88:0/1616222499 shutdown_connections 2024-09-16T01:39:50.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.819+0000 7fca20495740 1 --2- 172.21.15.88:0/1616222499 >> [v2:172.21.15.88:6835/2944989249,v1:172.21.15.88:6837/2944989249] conn(0x558e0cc26920 0x558e0cc26d00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.819+0000 7fca20495740 1 --2- 172.21.15.88:0/1616222499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2c360 0x558e0cc4def0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.819+0000 7fc9f5ffb640 0 client.6687 ms_handle_reset on v2:172.21.15.88:6835/2944989249 2024-09-16T01:39:50.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.819+0000 7fca20495740 1 --2- 172.21.15.88:0/1616222499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0cc2ea50 0x558e0cc2be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.819+0000 7fca20495740 1 --2- 172.21.15.88:0/1616222499 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558e0cc2d2c0 0x558e0cc4c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.819+0000 7fca20495740 1 -- 172.21.15.88:0/1616222499 >> 172.21.15.88:0/1616222499 conn(0x558e0cc2aa80 msgr2=0x558e0cc6fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:50.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.819+0000 7fca20495740 1 -- 172.21.15.88:0/1616222499 shutdown_connections 2024-09-16T01:39:50.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.819+0000 7fca20495740 1 -- 172.21.15.88:0/1616222499 wait complete. 2024-09-16T01:39:50.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.819+0000 7fca20495740 20 client.6687 trim_cache size 0 max 16384 2024-09-16T01:39:50.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.837+0000 7fca20495740 1 Processor -- start 2024-09-16T01:39:50.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.837+0000 7fca20495740 1 -- start start 2024-09-16T01:39:50.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.837+0000 7fca20495740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0cc2c360 0x558e0cc6ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:50.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.837+0000 7fca20495740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558e0cc2d2c0 0x558e0cc70120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:50.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.837+0000 7fca20495740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2ea50 0x558e0cc4b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:50.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.837+0000 7fca20495740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fca180cc520 con 0x558e0cc2ea50 2024-09-16T01:39:50.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.837+0000 7fca20495740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fca180cb880 con 0x558e0cc2c360 2024-09-16T01:39:50.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.837+0000 7fca20495740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fca180cbed0 con 0x558e0cc2d2c0 2024-09-16T01:39:50.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.838+0000 7fca177fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2ea50 0x558e0cc4b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:50.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.838+0000 7fca177fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2ea50 0x558e0cc4b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59972/0 (socket says 172.21.15.88:59972) 2024-09-16T01:39:50.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.838+0000 7fca177fe640 1 -- 172.21.15.88:0/1110855194 learned_addr learned my addr 172.21.15.88:0/1110855194 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:50.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.838+0000 7fca177fe640 1 -- 172.21.15.88:0/1110855194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558e0cc2d2c0 msgr2=0x558e0cc70120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:50.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.838+0000 7fca16ffd640 1 --2- 172.21.15.88:0/1110855194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0cc2c360 0x558e0cc6ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:50.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.838+0000 7fca167fc640 1 --2- 172.21.15.88:0/1110855194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558e0cc2d2c0 0x558e0cc70120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:50.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.838+0000 7fca177fe640 1 --2- 172.21.15.88:0/1110855194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558e0cc2d2c0 0x558e0cc70120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.838+0000 7fca177fe640 1 -- 172.21.15.88:0/1110855194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0cc2c360 msgr2=0x558e0cc6ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:50.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.838+0000 7fca177fe640 1 --2- 172.21.15.88:0/1110855194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0cc2c360 0x558e0cc6ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.838+0000 7fca177fe640 1 -- 172.21.15.88:0/1110855194 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e0cc4e260 con 0x558e0cc2ea50 2024-09-16T01:39:50.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.838+0000 7fca177fe640 1 --2- 172.21.15.88:0/1110855194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2ea50 0x558e0cc4b3c0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fca10089500 tx=0x7fca1005b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=598038d77c281732 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:50.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.838+0000 7fc9fcff9640 1 -- 172.21.15.88:0/1110855194 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca10058ed0 con 0x558e0cc2ea50 2024-09-16T01:39:50.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.838+0000 7fc9fcff9640 1 -- 172.21.15.88:0/1110855194 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca100891f0 con 0x558e0cc2ea50 2024-09-16T01:39:50.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.838+0000 7fc9fcff9640 1 -- 172.21.15.88:0/1110855194 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca10058550 con 0x558e0cc2ea50 2024-09-16T01:39:50.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.839+0000 7fca20495740 1 -- 172.21.15.88:0/1110855194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2ea50 msgr2=0x558e0cc4b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:50.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.839+0000 7fca20495740 1 --2- 172.21.15.88:0/1110855194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2ea50 0x558e0cc4b3c0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fca10089500 tx=0x7fca1005b000 comp rx=0 tx=0).stop 2024-09-16T01:39:50.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.839+0000 7fca20495740 1 -- 172.21.15.88:0/1110855194 shutdown_connections 2024-09-16T01:39:50.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.839+0000 7fca20495740 1 --2- 172.21.15.88:0/1110855194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2ea50 0x558e0cc4b3c0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.839+0000 7fca20495740 1 --2- 172.21.15.88:0/1110855194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558e0cc2d2c0 0x558e0cc70120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.839+0000 7fca20495740 1 --2- 172.21.15.88:0/1110855194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0cc2c360 0x558e0cc6ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.839+0000 7fca20495740 1 -- 172.21.15.88:0/1110855194 >> 172.21.15.88:0/1110855194 conn(0x558e0cc2aa80 msgr2=0x558e0cc556d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:50.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.839+0000 7fca20495740 1 -- 172.21.15.88:0/1110855194 shutdown_connections 2024-09-16T01:39:50.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.839+0000 7fca20495740 1 -- 172.21.15.88:0/1110855194 wait complete. 2024-09-16T01:39:50.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.839+0000 7fca20495740 1 -- start start 2024-09-16T01:39:50.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.839+0000 7fca20495740 1 Processor -- start 2024-09-16T01:39:50.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.840+0000 7fca20495740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0cc2c360 0x558e0ccefcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:50.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.840+0000 7fca20495740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558e0cc2d2c0 0x558e0cc6ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:50.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.840+0000 7fca20495740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2ea50 0x558e0cc6f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:50.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.840+0000 7fca20495740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558e0cc56a30 con 0x558e0cc2ea50 2024-09-16T01:39:50.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.840+0000 7fca20495740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558e0cc4d350 con 0x558e0cc2c360 2024-09-16T01:39:50.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.840+0000 7fca20495740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x558e0cc82970 con 0x558e0cc2d2c0 2024-09-16T01:39:50.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.840+0000 7fca20495740 10 client.0 _ll_register_callbacks cb 0x7fca10055420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:39:50.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.840+0000 7fca177fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2ea50 0x558e0cc6f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:50.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.840+0000 7fca177fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2ea50 0x558e0cc6f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59984/0 (socket says 172.21.15.88:59984) 2024-09-16T01:39:50.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.840+0000 7fca177fe640 1 -- 172.21.15.88:0/517037209 learned_addr learned my addr 172.21.15.88:0/517037209 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:50.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.840+0000 7fca16ffd640 1 --2- 172.21.15.88:0/517037209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0cc2c360 0x558e0ccefcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:50.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.840+0000 7fca167fc640 1 --2- 172.21.15.88:0/517037209 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558e0cc2d2c0 0x558e0cc6ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:50.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.840+0000 7fca177fe640 1 -- 172.21.15.88:0/517037209 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558e0cc2d2c0 msgr2=0x558e0cc6ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:50.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.840+0000 7fca177fe640 1 --2- 172.21.15.88:0/517037209 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558e0cc2d2c0 0x558e0cc6ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.840+0000 7fca177fe640 1 -- 172.21.15.88:0/517037209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0cc2c360 msgr2=0x558e0ccefcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:50.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.840+0000 7fca177fe640 1 --2- 172.21.15.88:0/517037209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0cc2c360 0x558e0ccefcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.840+0000 7fca177fe640 1 -- 172.21.15.88:0/517037209 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e0cc51ed0 con 0x558e0cc2ea50 2024-09-16T01:39:50.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.840+0000 7fca167fc640 1 --2- 172.21.15.88:0/517037209 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558e0cc2d2c0 0x558e0cc6ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:50.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.841+0000 7fca177fe640 1 --2- 172.21.15.88:0/517037209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2ea50 0x558e0cc6f480 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fca100573b0 tx=0x7fca10089490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b789d9dc985798b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:50.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.841+0000 7fc9f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:39:50.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.841+0000 7fc9f5ffb640 1 -- 172.21.15.88:0/517037209 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca10058ed0 con 0x558e0cc2ea50 2024-09-16T01:39:50.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.841+0000 7fc9f5ffb640 1 -- 172.21.15.88:0/517037209 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 5795+0+0 (secure 0 0 0) 0x7fca10061a50 con 0x558e0cc2ea50 2024-09-16T01:39:50.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.841+0000 7fca20495740 1 -- 172.21.15.88:0/517037209 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca100891f0 con 0x558e0cc2ea50 2024-09-16T01:39:50.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.841+0000 7fc9f5ffb640 1 -- 172.21.15.88:0/517037209 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7fca10054e50 con 0x558e0cc2ea50 2024-09-16T01:39:50.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.841+0000 7fca20495740 10 client.6693 fetch_fsmap learned FSMap version 64 2024-09-16T01:39:50.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.841+0000 7fca20495740 1 -- 172.21.15.88:0/517037209 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) -- 0x558e0cc95840 con 0x558e0cc2ea50 2024-09-16T01:39:50.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.841+0000 7fc9f5ffb640 1 -- 172.21.15.88:0/517037209 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca100891f0 con 0x558e0cc2ea50 2024-09-16T01:39:50.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.841+0000 7fc9f5ffb640 1 -- 172.21.15.88:0/517037209 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 64) ==== 3502+0+0 (secure 0 0 0) 0x7fca10054ff0 con 0x558e0cc2ea50 2024-09-16T01:39:50.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.841+0000 7fca20495740 10 client.6693 fetch_fsmap finished waiting for FSMap version 64 2024-09-16T01:39:50.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.841+0000 7fca20495740 10 client.6693 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:39:50.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.841+0000 7fca20495740 20 client.6693 populate_metadata read hostname 'smithi088' 2024-09-16T01:39:50.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.841+0000 7fca20495740 1 --2- 172.21.15.88:0/517037209 >> [v2:172.21.15.88:6835/2944989249,v1:172.21.15.88:6837/2944989249] conn(0x558e0cc51120 0x558e0ccf04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:50.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.841+0000 7fca20495740 4 client.6693 mds_command: new command op to 6414 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:39:50.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.841+0000 7fca20495740 1 -- 172.21.15.88:0/517037209 --> [v2:172.21.15.88:6835/2944989249,v1:172.21.15.88:6837/2944989249] -- command(tid 0: {"prefix": "status"}) -- 0x558e0cb96fa0 con 0x558e0cc51120 2024-09-16T01:39:50.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.841+0000 7fca16ffd640 1 --2- 172.21.15.88:0/517037209 >> [v2:172.21.15.88:6835/2944989249,v1:172.21.15.88:6837/2944989249] conn(0x558e0cc51120 0x558e0ccf04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:50.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.841+0000 7fca16ffd640 1 --2- 172.21.15.88:0/517037209 >> [v2:172.21.15.88:6835/2944989249,v1:172.21.15.88:6837/2944989249] conn(0x558e0cc51120 0x558e0ccf04f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc7544a8fc0c28f2 server_cookie=732059ced550a12c in_seq=0 out_seq=0 2024-09-16T01:39:50.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.841+0000 7fc9f5ffb640 10 client.6693 ms_handle_connect on v2:172.21.15.88:6835/2944989249 2024-09-16T01:39:50.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.842+0000 7fc9f5ffb640 1 -- 172.21.15.88:0/517037209 <== mds.0 v2:172.21.15.88:6835/2944989249 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x558e0cb96fa0 con 0x558e0cc51120 2024-09-16T01:39:50.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.842+0000 7fc9f5ffb640 10 client.6693 handle_command_reply: tid=0 2024-09-16T01:39:50.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.842+0000 7fca20495740 1 client.6693 shutdown 2024-09-16T01:39:50.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.842+0000 7fca20495740 1 -- 172.21.15.88:0/517037209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2ea50 msgr2=0x558e0cc6f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:50.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.842+0000 7fca20495740 1 --2- 172.21.15.88:0/517037209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2ea50 0x558e0cc6f480 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fca100573b0 tx=0x7fca10089490 comp rx=0 tx=0).stop 2024-09-16T01:39:50.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.842+0000 7fca20495740 1 -- 172.21.15.88:0/517037209 shutdown_connections 2024-09-16T01:39:50.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.842+0000 7fca20495740 1 --2- 172.21.15.88:0/517037209 >> [v2:172.21.15.88:6835/2944989249,v1:172.21.15.88:6837/2944989249] conn(0x558e0cc51120 0x558e0ccf04f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.842+0000 7fca20495740 1 --2- 172.21.15.88:0/517037209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558e0cc2ea50 0x558e0cc6f480 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.842+0000 7fc9f5ffb640 0 client.6693 ms_handle_reset on v2:172.21.15.88:6835/2944989249 2024-09-16T01:39:50.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.842+0000 7fca20495740 1 --2- 172.21.15.88:0/517037209 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558e0cc2d2c0 0x558e0cc6ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.842+0000 7fca20495740 1 --2- 172.21.15.88:0/517037209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e0cc2c360 0x558e0ccefcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.842+0000 7fca20495740 1 -- 172.21.15.88:0/517037209 >> 172.21.15.88:0/517037209 conn(0x558e0cc2aa80 msgr2=0x558e0cc73a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:50.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.842+0000 7fca20495740 1 -- 172.21.15.88:0/517037209 shutdown_connections 2024-09-16T01:39:50.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.842+0000 7fca20495740 1 -- 172.21.15.88:0/517037209 wait complete. 2024-09-16T01:39:50.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.842+0000 7fca20495740 20 client.6693 trim_cache size 0 max 16384 2024-09-16T01:39:50.852 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:39:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "5ace2c91-46f9-482b-8a89-731fbdc171bf", 2024-09-16T01:39:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2024-09-16T01:39:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 6414, 2024-09-16T01:39:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2024-09-16T01:39:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2024-09-16T01:39:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2024-09-16T01:39:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 4.5875383689999998, 2024-09-16T01:39:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 64, 2024-09-16T01:39:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 73, 2024-09-16T01:39:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 73, 2024-09-16T01:39:50.852 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 39.446491776000002 2024-09-16T01:39:50.853 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:39:50.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.842+0000 7fca1d6f8640 1 -- 172.21.15.88:0/1384295982 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fca0406c2b0 msgr2=0x7fca0406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:50.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.842+0000 7fca1d6f8640 1 --2- 172.21.15.88:0/1384295982 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fca0406c2b0 0x7fca0406e770 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fca08000f60 tx=0x7fca08007ca0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.842+0000 7fca1d6f8640 1 -- 172.21.15.88:0/1384295982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca1815e7a0 msgr2=0x7fca18156ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:50.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.842+0000 7fca1d6f8640 1 --2- 172.21.15.88:0/1384295982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca1815e7a0 0x7fca18156ad0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fca1000b970 tx=0x7fca1000be30 comp rx=0 tx=0).stop 2024-09-16T01:39:50.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.843+0000 7fca1d6f8640 1 -- 172.21.15.88:0/1384295982 shutdown_connections 2024-09-16T01:39:50.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.843+0000 7fca1d6f8640 1 --2- 172.21.15.88:0/1384295982 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fca0406c2b0 0x7fca0406e770 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.843+0000 7fca1d6f8640 1 --2- 172.21.15.88:0/1384295982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca1815e7a0 0x7fca18156ad0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.843+0000 7fca1d6f8640 1 --2- 172.21.15.88:0/1384295982 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca180cf1f0 0x7fca181f8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.843+0000 7fca1d6f8640 1 --2- 172.21.15.88:0/1384295982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca180b6720 0x7fca181626f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:50.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.843+0000 7fca1d6f8640 1 -- 172.21.15.88:0/1384295982 >> 172.21.15.88:0/1384295982 conn(0x7fca180d11e0 msgr2=0x7fca180d15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:50.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.843+0000 7fca1d6f8640 1 -- 172.21.15.88:0/1384295982 shutdown_connections 2024-09-16T01:39:50.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50.843+0000 7fca1d6f8640 1 -- 172.21.15.88:0/1384295982 wait complete. 2024-09-16T01:39:50.855 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'stat a/ && find a/') 2024-09-16T01:39:50.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:39:50.906 INFO:teuthology.orchestra.run.smithi088.stdout: File: a/ 2024-09-16T01:39:50.906 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 2 Blocks: 0 IO Block: 65536 directory 2024-09-16T01:39:50.906 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 2ah/42d Inode: 1099511627776 Links: 4 2024-09-16T01:39:50.906 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1333/ ubuntu) 2024-09-16T01:39:50.906 INFO:teuthology.orchestra.run.smithi088.stdout:Context: unconfined_u:object_r:unlabeled_t:s0 2024-09-16T01:39:50.906 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-09-16 01:39:22.577259155 +0000 2024-09-16T01:39:50.906 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-09-16 01:39:28.110149399 +0000 2024-09-16T01:39:50.906 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-09-16 01:39:28.110149399 +0000 2024-09-16T01:39:50.906 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-09-16 01:39:22.577259155 +0000 2024-09-16T01:39:50.912 INFO:teuthology.orchestra.run.smithi088.stdout:a/ 2024-09-16T01:39:50.912 INFO:teuthology.orchestra.run.smithi088.stdout:a/z 2024-09-16T01:39:50.912 INFO:teuthology.orchestra.run.smithi088.stdout:a/z/d 2024-09-16T01:39:50.912 INFO:teuthology.orchestra.run.smithi088.stdout:a/c 2024-09-16T01:39:50.912 INFO:teuthology.orchestra.run.smithi088.stdout:a/c/d 2024-09-16T01:39:50.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:39:50.918 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:0 flush journal 2024-09-16T01:39:51.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.017+0000 7f3f59447640 1 Processor -- start 2024-09-16T01:39:51.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.017+0000 7f3f59447640 1 -- start start 2024-09-16T01:39:51.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.018+0000 7f3f59447640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f540b68f0 0x7f3f541696f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.018+0000 7f3f59447640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f540cf860 con 0x7f3f540cf1f0 2024-09-16T01:39:51.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.018+0000 7f3f59447640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f540cfb40 con 0x7f3f5415e7a0 2024-09-16T01:39:51.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.018+0000 7f3f59447640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3f540d2ca0 con 0x7f3f540b68f0 2024-09-16T01:39:51.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.018+0000 7f3f4bfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3f540cf1f0 0x7f3f540b61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55914/0 (socket says 172.21.15.88:55914) 2024-09-16T01:39:51.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.018+0000 7f3f4bfff640 1 -- 172.21.15.88:0/1061600557 learned_addr learned my addr 172.21.15.88:0/1061600557 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:51.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.018+0000 7f3f537fe640 1 --2- 172.21.15.88:0/1061600557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f540b68f0 0x7f3f541696f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.018+0000 7f3f527fc640 1 -- 172.21.15.88:0/1061600557 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3956053026 0 0) 0x7f3f540cf860 con 0x7f3f540cf1f0 2024-09-16T01:39:51.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.018+0000 7f3f527fc640 1 -- 172.21.15.88:0/1061600557 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f40003520 con 0x7f3f540cf1f0 2024-09-16T01:39:51.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.019+0000 7f3f527fc640 1 -- 172.21.15.88:0/1061600557 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 434923692 0 0) 0x7f3f540cfb40 con 0x7f3f5415e7a0 2024-09-16T01:39:51.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.019+0000 7f3f527fc640 1 -- 172.21.15.88:0/1061600557 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f540cf860 con 0x7f3f5415e7a0 2024-09-16T01:39:51.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.019+0000 7f3f527fc640 1 -- 172.21.15.88:0/1061600557 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1970674713 0 0) 0x7f3f40003520 con 0x7f3f540cf1f0 2024-09-16T01:39:51.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.019+0000 7f3f537fe640 1 -- 172.21.15.88:0/1061600557 >> v1:172.21.15.154:6789/0 conn(0x7f3f5415e7a0 legacy=0x7f3f5406caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:51.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.019+0000 7f3f537fe640 1 -- 172.21.15.88:0/1061600557 >> v1:172.21.15.88:6789/0 conn(0x7f3f540cf1f0 legacy=0x7f3f540b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:51.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.019+0000 7f3f537fe640 1 -- 172.21.15.88:0/1061600557 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f540cfe20 con 0x7f3f540b68f0 2024-09-16T01:39:51.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.019+0000 7f3f537fe640 1 --2- 172.21.15.88:0/1061600557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f540b68f0 0x7f3f541696f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c009dc0 tx=0x7f3f4c02a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0fe8f946d11412b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:51.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.020+0000 7f3f527fc640 1 -- 172.21.15.88:0/1061600557 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c001670 con 0x7f3f540b68f0 2024-09-16T01:39:51.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.020+0000 7f3f527fc640 1 -- 172.21.15.88:0/1061600557 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f4c007540 con 0x7f3f540b68f0 2024-09-16T01:39:51.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.020+0000 7f3f527fc640 1 -- 172.21.15.88:0/1061600557 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c0078f0 con 0x7f3f540b68f0 2024-09-16T01:39:51.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.020+0000 7f3f59447640 1 -- 172.21.15.88:0/1061600557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f540b68f0 msgr2=0x7f3f541696f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:51.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.020+0000 7f3f59447640 1 --2- 172.21.15.88:0/1061600557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f540b68f0 0x7f3f541696f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c009dc0 tx=0x7f3f4c02a5d0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.020+0000 7f3f59447640 1 -- 172.21.15.88:0/1061600557 shutdown_connections 2024-09-16T01:39:51.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.020+0000 7f3f59447640 1 --2- 172.21.15.88:0/1061600557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f540b68f0 0x7f3f541696f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.020+0000 7f3f59447640 1 -- 172.21.15.88:0/1061600557 >> 172.21.15.88:0/1061600557 conn(0x7f3f540d11e0 msgr2=0x7f3f540d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:51.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.020+0000 7f3f59447640 1 -- 172.21.15.88:0/1061600557 shutdown_connections 2024-09-16T01:39:51.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.020+0000 7f3f59447640 1 -- 172.21.15.88:0/1061600557 wait complete. 2024-09-16T01:39:51.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.021+0000 7f3f59447640 1 Processor -- start 2024-09-16T01:39:51.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.021+0000 7f3f59447640 1 -- start start 2024-09-16T01:39:51.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.022+0000 7f3f59447640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f540b68f0 0x7f3f540cb830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.022+0000 7f3f59447640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f540cf1f0 0x7f3f54149ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.022+0000 7f3f59447640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f5415e7a0 0x7f3f540ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.022+0000 7f3f59447640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3f540d4420 con 0x7f3f540cf1f0 2024-09-16T01:39:51.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.022+0000 7f3f59447640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3f540b7a00 con 0x7f3f5415e7a0 2024-09-16T01:39:51.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.022+0000 7f3f59447640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3f540ccad0 con 0x7f3f540b68f0 2024-09-16T01:39:51.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.022+0000 7f3f4bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f540cf1f0 0x7f3f54149ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.022+0000 7f3f4bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f540cf1f0 0x7f3f54149ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59994/0 (socket says 172.21.15.88:59994) 2024-09-16T01:39:51.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.022+0000 7f3f4bfff640 1 -- 172.21.15.88:0/865867758 learned_addr learned my addr 172.21.15.88:0/865867758 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:51.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.022+0000 7f3f4bfff640 1 -- 172.21.15.88:0/865867758 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f540b68f0 msgr2=0x7f3f540cb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:51.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.022+0000 7f3f4bfff640 1 --2- 172.21.15.88:0/865867758 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f540b68f0 0x7f3f540cb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.022+0000 7f3f4bfff640 1 -- 172.21.15.88:0/865867758 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f5415e7a0 msgr2=0x7f3f540ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:51.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.022+0000 7f3f4bfff640 1 --2- 172.21.15.88:0/865867758 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f5415e7a0 0x7f3f540ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.022+0000 7f3f4bfff640 1 -- 172.21.15.88:0/865867758 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f540cacc0 con 0x7f3f540cf1f0 2024-09-16T01:39:51.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.023+0000 7f3f4bfff640 1 --2- 172.21.15.88:0/865867758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f540cf1f0 0x7f3f54149ae0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c00c5c0 tx=0x7f3f3c00ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:51.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.023+0000 7f3f4b7fe640 1 -- 172.21.15.88:0/865867758 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c017070 con 0x7f3f540cf1f0 2024-09-16T01:39:51.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.023+0000 7f3f4b7fe640 1 -- 172.21.15.88:0/865867758 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f3c005ce0 con 0x7f3f540cf1f0 2024-09-16T01:39:51.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.023+0000 7f3f59447640 1 -- 172.21.15.88:0/865867758 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f540cbec0 con 0x7f3f540cf1f0 2024-09-16T01:39:51.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.023+0000 7f3f4b7fe640 1 -- 172.21.15.88:0/865867758 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c0072b0 con 0x7f3f540cf1f0 2024-09-16T01:39:51.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.023+0000 7f3f59447640 1 -- 172.21.15.88:0/865867758 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f540cc480 con 0x7f3f540cf1f0 2024-09-16T01:39:51.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.024+0000 7f3f4b7fe640 1 -- 172.21.15.88:0/865867758 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3f3c01a070 con 0x7f3f540cf1f0 2024-09-16T01:39:51.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.025+0000 7f3f4b7fe640 1 --2- 172.21.15.88:0/865867758 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3f3406c2b0 0x7f3f3406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.025+0000 7f3f52ffd640 1 --2- 172.21.15.88:0/865867758 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3f3406c2b0 0x7f3f3406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.025+0000 7f3f4b7fe640 1 -- 172.21.15.88:0/865867758 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5795+0+0 (secure 0 0 0) 0x7f3f3c08a970 con 0x7f3f540cf1f0 2024-09-16T01:39:51.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.025+0000 7f3f52ffd640 1 --2- 172.21.15.88:0/865867758 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3f3406c2b0 0x7f3f3406e770 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3f440010e0 tx=0x7f3f44007950 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:51.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.026+0000 7f3f5c1dc740 1 Processor -- start 2024-09-16T01:39:51.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.026+0000 7f3f5c1dc740 1 -- start start 2024-09-16T01:39:51.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.027+0000 7f3f5c1dc740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558d05205920 0x558d0524b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.027+0000 7f3f5c1dc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d052030e0 0x558d05205d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.027+0000 7f3f5c1dc740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d052070b0 0x558d052001a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.027+0000 7f3f5c1dc740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558d051686f0 con 0x558d052070b0 2024-09-16T01:39:51.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.027+0000 7f3f5c1dc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558d05110330 con 0x558d052030e0 2024-09-16T01:39:51.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.027+0000 7f3f5c1dc740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x558d051646b0 con 0x558d05205920 2024-09-16T01:39:51.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.027+0000 7f3f537fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d052070b0 0x558d052001a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.027+0000 7f3f537fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d052070b0 0x558d052001a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59998/0 (socket says 172.21.15.88:59998) 2024-09-16T01:39:51.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.027+0000 7f3f537fe640 1 -- 172.21.15.88:0/1333809613 learned_addr learned my addr 172.21.15.88:0/1333809613 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:51.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.027+0000 7f3f4bfff640 1 --2- 172.21.15.88:0/1333809613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d052030e0 0x558d05205d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.027+0000 7f3f52ffd640 1 --2- 172.21.15.88:0/1333809613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558d05205920 0x558d0524b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.027+0000 7f3f537fe640 1 -- 172.21.15.88:0/1333809613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558d05205920 msgr2=0x558d0524b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:51.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.027+0000 7f3f537fe640 1 --2- 172.21.15.88:0/1333809613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558d05205920 0x558d0524b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.027+0000 7f3f537fe640 1 -- 172.21.15.88:0/1333809613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d052030e0 msgr2=0x558d05205d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:51.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.027+0000 7f3f537fe640 1 --2- 172.21.15.88:0/1333809613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d052030e0 0x558d05205d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.027+0000 7f3f537fe640 1 -- 172.21.15.88:0/1333809613 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d051360d0 con 0x558d052070b0 2024-09-16T01:39:51.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.027+0000 7f3f4bfff640 1 --2- 172.21.15.88:0/1333809613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d052030e0 0x558d05205d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:51.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.027+0000 7f3f537fe640 1 --2- 172.21.15.88:0/1333809613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d052070b0 0x558d052001a0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c007110 tx=0x7f3f4c02c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d7108761a02deac server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:51.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.027+0000 7f3f48ff9640 1 -- 172.21.15.88:0/1333809613 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c040070 con 0x558d052070b0 2024-09-16T01:39:51.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.027+0000 7f3f48ff9640 1 -- 172.21.15.88:0/1333809613 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f4c007740 con 0x558d052070b0 2024-09-16T01:39:51.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.027+0000 7f3f48ff9640 1 -- 172.21.15.88:0/1333809613 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c007af0 con 0x558d052070b0 2024-09-16T01:39:51.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.028+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/1333809613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d052070b0 msgr2=0x558d052001a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:51.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.028+0000 7f3f5c1dc740 1 --2- 172.21.15.88:0/1333809613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d052070b0 0x558d052001a0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c007110 tx=0x7f3f4c02c000 comp rx=0 tx=0).stop 2024-09-16T01:39:51.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.028+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/1333809613 shutdown_connections 2024-09-16T01:39:51.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.028+0000 7f3f5c1dc740 1 --2- 172.21.15.88:0/1333809613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d052070b0 0x558d052001a0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.028+0000 7f3f5c1dc740 1 --2- 172.21.15.88:0/1333809613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d052030e0 0x558d05205d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.028+0000 7f3f5c1dc740 1 --2- 172.21.15.88:0/1333809613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558d05205920 0x558d0524b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.028+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/1333809613 >> 172.21.15.88:0/1333809613 conn(0x558d05202370 msgr2=0x558d0524a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:51.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.028+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/1333809613 shutdown_connections 2024-09-16T01:39:51.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.028+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/1333809613 wait complete. 2024-09-16T01:39:51.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.029+0000 7f3f5c1dc740 1 -- start start 2024-09-16T01:39:51.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.029+0000 7f3f5c1dc740 1 Processor -- start 2024-09-16T01:39:51.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.029+0000 7f3f5c1dc740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558d05205920 0x558d05247ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.029+0000 7f3f5c1dc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d052070b0 0x558d05216d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.030+0000 7f3f5c1dc740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d0522d300 0x558d0522fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.030+0000 7f3f5c1dc740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558d05201810 con 0x558d0522d300 2024-09-16T01:39:51.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.030+0000 7f3f5c1dc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558d050f14f0 con 0x558d052070b0 2024-09-16T01:39:51.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.030+0000 7f3f5c1dc740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x558d0524b8f0 con 0x558d05205920 2024-09-16T01:39:51.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.030+0000 7f3f5c1dc740 10 client.0 _ll_register_callbacks cb 0x558d05112b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:39:51.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.030+0000 7f3f537fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d0522d300 0x558d0522fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.030+0000 7f3f537fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d0522d300 0x558d0522fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60002/0 (socket says 172.21.15.88:60002) 2024-09-16T01:39:51.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.030+0000 7f3f537fe640 1 -- 172.21.15.88:0/1716318476 learned_addr learned my addr 172.21.15.88:0/1716318476 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:51.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.030+0000 7f3f52ffd640 1 --2- 172.21.15.88:0/1716318476 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558d05205920 0x558d05247ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.030+0000 7f3f4bfff640 1 --2- 172.21.15.88:0/1716318476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d052070b0 0x558d05216d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.030+0000 7f3f537fe640 1 -- 172.21.15.88:0/1716318476 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558d05205920 msgr2=0x558d05247ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:51.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.030+0000 7f3f537fe640 1 --2- 172.21.15.88:0/1716318476 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558d05205920 0x558d05247ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.030+0000 7f3f537fe640 1 -- 172.21.15.88:0/1716318476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d052070b0 msgr2=0x558d05216d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:51.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.030+0000 7f3f537fe640 1 --2- 172.21.15.88:0/1716318476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d052070b0 0x558d05216d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.030+0000 7f3f537fe640 1 -- 172.21.15.88:0/1716318476 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d04f457d0 con 0x558d0522d300 2024-09-16T01:39:51.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.030+0000 7f3f537fe640 1 --2- 172.21.15.88:0/1716318476 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d0522d300 0x558d0522fec0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c03f040 tx=0x7f3f4c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd31a16f15f9d6e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:51.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.030+0000 7f3f31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:39:51.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.030+0000 7f3f31ffb640 1 -- 172.21.15.88:0/1716318476 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c054060 con 0x558d0522d300 2024-09-16T01:39:51.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.030+0000 7f3f31ffb640 1 -- 172.21.15.88:0/1716318476 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 5795+0+0 (secure 0 0 0) 0x7f3f4c045870 con 0x558d0522d300 2024-09-16T01:39:51.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.030+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/1716318476 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f4c007740 con 0x558d0522d300 2024-09-16T01:39:51.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.030+0000 7f3f31ffb640 1 -- 172.21.15.88:0/1716318476 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f3f4c05a050 con 0x558d0522d300 2024-09-16T01:39:51.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.030+0000 7f3f5c1dc740 10 client.6705 fetch_fsmap learned FSMap version 64 2024-09-16T01:39:51.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.030+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/1716318476 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) -- 0x558d04e7b3f0 con 0x558d0522d300 2024-09-16T01:39:51.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.030+0000 7f3f31ffb640 1 -- 172.21.15.88:0/1716318476 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f4c007740 con 0x558d0522d300 2024-09-16T01:39:51.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.031+0000 7f3f31ffb640 1 -- 172.21.15.88:0/1716318476 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 64) ==== 3502+0+0 (secure 0 0 0) 0x7f3f4c05d020 con 0x558d0522d300 2024-09-16T01:39:51.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.031+0000 7f3f5c1dc740 10 client.6705 fetch_fsmap finished waiting for FSMap version 64 2024-09-16T01:39:51.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.031+0000 7f3f5c1dc740 10 client.6705 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-09-16T01:39:51.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.031+0000 7f3f5c1dc740 20 client.6705 populate_metadata read hostname 'smithi088' 2024-09-16T01:39:51.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.031+0000 7f3f5c1dc740 1 --2- 172.21.15.88:0/1716318476 >> [v2:172.21.15.88:6835/2944989249,v1:172.21.15.88:6837/2944989249] conn(0x558d05214880 0x558d0522be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.031+0000 7f3f5c1dc740 4 client.6705 mds_command: new command op to 6414 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:39:51.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.031+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/1716318476 --> [v2:172.21.15.88:6835/2944989249,v1:172.21.15.88:6837/2944989249] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d05175e20 con 0x558d05214880 2024-09-16T01:39:51.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.031+0000 7f3f52ffd640 1 --2- 172.21.15.88:0/1716318476 >> [v2:172.21.15.88:6835/2944989249,v1:172.21.15.88:6837/2944989249] conn(0x558d05214880 0x558d0522be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.031+0000 7f3f31ffb640 10 client.6705 ms_handle_connect on v2:172.21.15.88:6835/2944989249 2024-09-16T01:39:51.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.031+0000 7f3f52ffd640 1 --2- 172.21.15.88:0/1716318476 >> [v2:172.21.15.88:6835/2944989249,v1:172.21.15.88:6837/2944989249] conn(0x558d05214880 0x558d0522be10 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65b5c037e7e7a71d server_cookie=ee84dba2eecfacea in_seq=0 out_seq=0 2024-09-16T01:39:51.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.032+0000 7f3f31ffb640 1 -- 172.21.15.88:0/1716318476 <== mds.0 v2:172.21.15.88:6835/2944989249 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558d05175e20 con 0x558d05214880 2024-09-16T01:39:51.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.032+0000 7f3f31ffb640 10 client.6705 handle_command_reply: tid=0 2024-09-16T01:39:51.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.032+0000 7f3f5c1dc740 1 client.6705 shutdown 2024-09-16T01:39:51.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.032+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/1716318476 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d0522d300 msgr2=0x558d0522fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:51.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.032+0000 7f3f5c1dc740 1 --2- 172.21.15.88:0/1716318476 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d0522d300 0x558d0522fec0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c03f040 tx=0x7f3f4c004ab0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.032+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/1716318476 shutdown_connections 2024-09-16T01:39:51.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.032+0000 7f3f5c1dc740 1 --2- 172.21.15.88:0/1716318476 >> [v2:172.21.15.88:6835/2944989249,v1:172.21.15.88:6837/2944989249] conn(0x558d05214880 0x558d0522be10 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.032+0000 7f3f5c1dc740 1 --2- 172.21.15.88:0/1716318476 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d0522d300 0x558d0522fec0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.032+0000 7f3f5c1dc740 1 --2- 172.21.15.88:0/1716318476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d052070b0 0x558d05216d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.032+0000 7f3f31ffb640 0 client.6705 ms_handle_reset on v2:172.21.15.88:6835/2944989249 2024-09-16T01:39:51.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.032+0000 7f3f5c1dc740 1 --2- 172.21.15.88:0/1716318476 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558d05205920 0x558d05247ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.032+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/1716318476 >> 172.21.15.88:0/1716318476 conn(0x558d052030e0 msgr2=0x558d05248610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:51.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.032+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/1716318476 shutdown_connections 2024-09-16T01:39:51.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.032+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/1716318476 wait complete. 2024-09-16T01:39:51.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.032+0000 7f3f5c1dc740 20 client.6705 trim_cache size 0 max 16384 2024-09-16T01:39:51.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.046+0000 7f3f5c1dc740 1 Processor -- start 2024-09-16T01:39:51.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.046+0000 7f3f5c1dc740 1 -- start start 2024-09-16T01:39:51.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.047+0000 7f3f5c1dc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d05205920 0x558d052498d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.047+0000 7f3f5c1dc740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d052070b0 0x558d05226120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.047+0000 7f3f5c1dc740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558d0522d300 0x558d05226660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.047+0000 7f3f5c1dc740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3f54200a10 con 0x558d052070b0 2024-09-16T01:39:51.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.047+0000 7f3f5c1dc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3f540c8ba0 con 0x558d05205920 2024-09-16T01:39:51.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.047+0000 7f3f5c1dc740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3f5414aae0 con 0x558d0522d300 2024-09-16T01:39:51.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.047+0000 7f3f4bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d052070b0 0x558d05226120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.047+0000 7f3f4bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d052070b0 0x558d05226120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60004/0 (socket says 172.21.15.88:60004) 2024-09-16T01:39:51.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.047+0000 7f3f4bfff640 1 -- 172.21.15.88:0/3250509583 learned_addr learned my addr 172.21.15.88:0/3250509583 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:51.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.047+0000 7f3f537fe640 1 --2- 172.21.15.88:0/3250509583 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558d0522d300 0x558d05226660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.047+0000 7f3f52ffd640 1 --2- 172.21.15.88:0/3250509583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d05205920 0x558d052498d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.047+0000 7f3f4bfff640 1 -- 172.21.15.88:0/3250509583 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558d0522d300 msgr2=0x558d05226660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:51.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.047+0000 7f3f4bfff640 1 --2- 172.21.15.88:0/3250509583 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558d0522d300 0x558d05226660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.047+0000 7f3f4bfff640 1 -- 172.21.15.88:0/3250509583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d05205920 msgr2=0x558d052498d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:51.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.047+0000 7f3f4bfff640 1 --2- 172.21.15.88:0/3250509583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d05205920 0x558d052498d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.047+0000 7f3f4bfff640 1 -- 172.21.15.88:0/3250509583 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d04e3b0f0 con 0x558d052070b0 2024-09-16T01:39:51.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.047+0000 7f3f537fe640 1 --2- 172.21.15.88:0/3250509583 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558d0522d300 0x558d05226660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:51.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.047+0000 7f3f52ffd640 1 --2- 172.21.15.88:0/3250509583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d05205920 0x558d052498d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:51.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.047+0000 7f3f4bfff640 1 --2- 172.21.15.88:0/3250509583 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d052070b0 0x558d05226120 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c054e70 tx=0x7f3f3c05f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f714a9c67edd7675 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:51.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.048+0000 7f3f33fff640 1 -- 172.21.15.88:0/3250509583 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c087070 con 0x558d052070b0 2024-09-16T01:39:51.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.048+0000 7f3f33fff640 1 -- 172.21.15.88:0/3250509583 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f3c055080 con 0x558d052070b0 2024-09-16T01:39:51.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.048+0000 7f3f33fff640 1 -- 172.21.15.88:0/3250509583 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c058e90 con 0x558d052070b0 2024-09-16T01:39:51.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.048+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/3250509583 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d052070b0 msgr2=0x558d05226120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:51.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.048+0000 7f3f5c1dc740 1 --2- 172.21.15.88:0/3250509583 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d052070b0 0x558d05226120 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c054e70 tx=0x7f3f3c05f040 comp rx=0 tx=0).stop 2024-09-16T01:39:51.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.048+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/3250509583 shutdown_connections 2024-09-16T01:39:51.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.048+0000 7f3f5c1dc740 1 --2- 172.21.15.88:0/3250509583 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558d0522d300 0x558d05226660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.048+0000 7f3f5c1dc740 1 --2- 172.21.15.88:0/3250509583 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d052070b0 0x558d05226120 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.048+0000 7f3f5c1dc740 1 --2- 172.21.15.88:0/3250509583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d05205920 0x558d052498d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.048+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/3250509583 >> 172.21.15.88:0/3250509583 conn(0x558d052030e0 msgr2=0x558d05224090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:51.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.048+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/3250509583 shutdown_connections 2024-09-16T01:39:51.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.048+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/3250509583 wait complete. 2024-09-16T01:39:51.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.049+0000 7f3f5c1dc740 1 -- start start 2024-09-16T01:39:51.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.049+0000 7f3f5c1dc740 1 Processor -- start 2024-09-16T01:39:51.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.049+0000 7f3f5c1dc740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558d05205920 0x558d052c8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.049+0000 7f3f5c1dc740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d052070b0 0x558d05224e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f5c1dc740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d0522d300 0x558d052464e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f5c1dc740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3f540c8550 con 0x558d052070b0 2024-09-16T01:39:51.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f5c1dc740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558d0522bd40 con 0x558d0522d300 2024-09-16T01:39:51.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f5c1dc740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x558d052162c0 con 0x558d05205920 2024-09-16T01:39:51.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f5c1dc740 10 client.0 _ll_register_callbacks cb 0x7f3f40003880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:39:51.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f4bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d052070b0 0x558d05224e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f4bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d052070b0 0x558d05224e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60020/0 (socket says 172.21.15.88:60020) 2024-09-16T01:39:51.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f4bfff640 1 -- 172.21.15.88:0/1441118181 learned_addr learned my addr 172.21.15.88:0/1441118181 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:51.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f52ffd640 1 --2- 172.21.15.88:0/1441118181 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558d05205920 0x558d052c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f537fe640 1 --2- 172.21.15.88:0/1441118181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d0522d300 0x558d052464e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f4bfff640 1 -- 172.21.15.88:0/1441118181 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558d05205920 msgr2=0x558d052c8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:51.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f4bfff640 1 --2- 172.21.15.88:0/1441118181 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558d05205920 0x558d052c8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f4bfff640 1 -- 172.21.15.88:0/1441118181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d0522d300 msgr2=0x558d052464e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:51.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f4bfff640 1 --2- 172.21.15.88:0/1441118181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d0522d300 0x558d052464e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f4bfff640 1 -- 172.21.15.88:0/1441118181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d0521a360 con 0x558d052070b0 2024-09-16T01:39:51.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f4bfff640 1 --2- 172.21.15.88:0/1441118181 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d052070b0 0x558d05224e30 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c058710 tx=0x7f3f3c001150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e19e1fc721fbbe16 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:51.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:39:51.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f31ffb640 1 -- 172.21.15.88:0/1441118181 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c087030 con 0x558d052070b0 2024-09-16T01:39:51.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f31ffb640 1 -- 172.21.15.88:0/1441118181 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(73..73 src has 1..73) ==== 5795+0+0 (secure 0 0 0) 0x7f3f3c061370 con 0x558d052070b0 2024-09-16T01:39:51.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/1441118181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f3c055080 con 0x558d052070b0 2024-09-16T01:39:51.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f31ffb640 1 -- 172.21.15.88:0/1441118181 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) ==== 24+0+0 (secure 0 0 0) 0x7f3f3c07e050 con 0x558d052070b0 2024-09-16T01:39:51.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f5c1dc740 10 client.6711 fetch_fsmap learned FSMap version 64 2024-09-16T01:39:51.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/1441118181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) -- 0x558d0522b820 con 0x558d052070b0 2024-09-16T01:39:51.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f31ffb640 1 -- 172.21.15.88:0/1441118181 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f3c055080 con 0x558d052070b0 2024-09-16T01:39:51.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f31ffb640 1 -- 172.21.15.88:0/1441118181 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 64) ==== 3502+0+0 (secure 0 0 0) 0x7f3f3c007450 con 0x558d052070b0 2024-09-16T01:39:51.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f5c1dc740 10 client.6711 fetch_fsmap finished waiting for FSMap version 64 2024-09-16T01:39:51.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f5c1dc740 10 client.6711 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-09-16T01:39:51.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.050+0000 7f3f5c1dc740 20 client.6711 populate_metadata read hostname 'smithi088' 2024-09-16T01:39:51.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.051+0000 7f3f5c1dc740 1 --2- 172.21.15.88:0/1441118181 >> [v2:172.21.15.88:6835/2944989249,v1:172.21.15.88:6837/2944989249] conn(0x558d0522f6c0 0x558d05246a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.051+0000 7f3f5c1dc740 4 client.6711 mds_command: new command op to 6414 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-16T01:39:51.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.051+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/1441118181 --> [v2:172.21.15.88:6835/2944989249,v1:172.21.15.88:6837/2944989249] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558d05170170 con 0x558d0522f6c0 2024-09-16T01:39:51.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.051+0000 7f3f52ffd640 1 --2- 172.21.15.88:0/1441118181 >> [v2:172.21.15.88:6835/2944989249,v1:172.21.15.88:6837/2944989249] conn(0x558d0522f6c0 0x558d05246a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.051+0000 7f3f31ffb640 10 client.6711 ms_handle_connect on v2:172.21.15.88:6835/2944989249 2024-09-16T01:39:51.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.051+0000 7f3f52ffd640 1 --2- 172.21.15.88:0/1441118181 >> [v2:172.21.15.88:6835/2944989249,v1:172.21.15.88:6837/2944989249] conn(0x558d0522f6c0 0x558d05246a20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a872c8736dc4079 server_cookie=dbe0d09619bcc109 in_seq=0 out_seq=0 2024-09-16T01:39:51.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.060+0000 7f3f31ffb640 1 -- 172.21.15.88:0/1441118181 <== mds.0 v2:172.21.15.88:6835/2944989249 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558d05170170 con 0x558d0522f6c0 2024-09-16T01:39:51.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.060+0000 7f3f31ffb640 10 client.6711 handle_command_reply: tid=0 2024-09-16T01:39:51.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.060+0000 7f3f5c1dc740 1 client.6711 shutdown 2024-09-16T01:39:51.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.060+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/1441118181 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d052070b0 msgr2=0x558d05224e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:51.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.060+0000 7f3f5c1dc740 1 --2- 172.21.15.88:0/1441118181 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d052070b0 0x558d05224e30 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c058710 tx=0x7f3f3c001150 comp rx=0 tx=0).stop 2024-09-16T01:39:51.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.060+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/1441118181 shutdown_connections 2024-09-16T01:39:51.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.060+0000 7f3f5c1dc740 1 --2- 172.21.15.88:0/1441118181 >> [v2:172.21.15.88:6835/2944989249,v1:172.21.15.88:6837/2944989249] conn(0x558d0522f6c0 0x558d05246a20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.060+0000 7f3f5c1dc740 1 --2- 172.21.15.88:0/1441118181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d0522d300 0x558d052464e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.060+0000 7f3f31ffb640 0 client.6711 ms_handle_reset on v2:172.21.15.88:6835/2944989249 2024-09-16T01:39:51.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.060+0000 7f3f5c1dc740 1 --2- 172.21.15.88:0/1441118181 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d052070b0 0x558d05224e30 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.060+0000 7f3f5c1dc740 1 --2- 172.21.15.88:0/1441118181 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558d05205920 0x558d052c8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.060+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/1441118181 >> 172.21.15.88:0/1441118181 conn(0x558d052030e0 msgr2=0x558d05226120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:51.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.060+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/1441118181 shutdown_connections 2024-09-16T01:39:51.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.060+0000 7f3f5c1dc740 1 -- 172.21.15.88:0/1441118181 wait complete. 2024-09-16T01:39:51.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.061+0000 7f3f5c1dc740 20 client.6711 trim_cache size 0 max 16384 2024-09-16T01:39:51.063 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:39:51.064 INFO:teuthology.orchestra.run.smithi088.stdout: "message": "", 2024-09-16T01:39:51.064 INFO:teuthology.orchestra.run.smithi088.stdout: "return_code": 0 2024-09-16T01:39:51.064 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:39:51.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.061+0000 7f3f59447640 1 -- 172.21.15.88:0/865867758 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3f3406c2b0 msgr2=0x7f3f3406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:51.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.061+0000 7f3f59447640 1 --2- 172.21.15.88:0/865867758 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3f3406c2b0 0x7f3f3406e770 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3f440010e0 tx=0x7f3f44007950 comp rx=0 tx=0).stop 2024-09-16T01:39:51.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.061+0000 7f3f59447640 1 -- 172.21.15.88:0/865867758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f540cf1f0 msgr2=0x7f3f54149ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:51.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.061+0000 7f3f59447640 1 --2- 172.21.15.88:0/865867758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f540cf1f0 0x7f3f54149ae0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c00c5c0 tx=0x7f3f3c00ca80 comp rx=0 tx=0).stop 2024-09-16T01:39:51.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.061+0000 7f3f59447640 1 -- 172.21.15.88:0/865867758 shutdown_connections 2024-09-16T01:39:51.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.061+0000 7f3f59447640 1 --2- 172.21.15.88:0/865867758 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3f3406c2b0 0x7f3f3406e770 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.061+0000 7f3f59447640 1 --2- 172.21.15.88:0/865867758 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f5415e7a0 0x7f3f540ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.061+0000 7f3f59447640 1 --2- 172.21.15.88:0/865867758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f540cf1f0 0x7f3f54149ae0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.061+0000 7f3f59447640 1 --2- 172.21.15.88:0/865867758 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f540b68f0 0x7f3f540cb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.061+0000 7f3f59447640 1 -- 172.21.15.88:0/865867758 >> 172.21.15.88:0/865867758 conn(0x7f3f540d11e0 msgr2=0x7f3f5415cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:51.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.061+0000 7f3f59447640 1 -- 172.21.15.88:0/865867758 shutdown_connections 2024-09-16T01:39:51.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.061+0000 7f3f59447640 1 -- 172.21.15.88:0/865867758 wait complete. 2024-09-16T01:39:51.073 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T01:39:51.074 DEBUG:teuthology.orchestra.run.smithi088:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T01:39:51.117 DEBUG:teuthology.orchestra.run.smithi154:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T01:39:51.160 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T01:39:51.160 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T01:39:51.160 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:39:51.160 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:39:51.176 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:39:51.176 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:39:51.232 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:39:51.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.370+0000 7f0ebcd9f640 1 Processor -- start 2024-09-16T01:39:51.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.370+0000 7f0ebcd9f640 1 -- start start 2024-09-16T01:39:51.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.371+0000 7f0ebcd9f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0eb8071150 0x7f0eb80b5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.371+0000 7f0ebcd9f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0eb8072f40 0x7f0eb80d1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.371+0000 7f0ebcd9f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0eb80d0220 con 0x7f0eb8070660 2024-09-16T01:39:51.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.371+0000 7f0ebcd9f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0eb80d04c0 con 0x7f0eb8071150 2024-09-16T01:39:51.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.371+0000 7f0ebcd9f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0eb80d07a0 con 0x7f0eb8072f40 2024-09-16T01:39:51.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.371+0000 7f0eb77fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0eb8070660 0x7f0eb8070a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55930/0 (socket says 172.21.15.88:55930) 2024-09-16T01:39:51.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.371+0000 7f0eb77fe640 1 -- 172.21.15.88:0/1441896966 learned_addr learned my addr 172.21.15.88:0/1441896966 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:51.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb6ffd640 1 --2- 172.21.15.88:0/1441896966 >> v2:172.21.15.154:3300/0 conn(0x7f0eb8071150 0x7f0eb80b5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb7fff640 1 --2- 172.21.15.88:0/1441896966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0eb8072f40 0x7f0eb80d1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb67fc640 1 -- 172.21.15.88:0/1441896966 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 361588873 0 0) 0x7f0eb80d0220 con 0x7f0eb8070660 2024-09-16T01:39:51.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb67fc640 1 -- 172.21.15.88:0/1441896966 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ea0003520 con 0x7f0eb8070660 2024-09-16T01:39:51.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb67fc640 1 -- 172.21.15.88:0/1441896966 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3177138131 0 0) 0x7f0ea0003520 con 0x7f0eb8070660 2024-09-16T01:39:51.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb67fc640 1 -- 172.21.15.88:0/1441896966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0eb8072f40 msgr2=0x7f0eb80d1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:51.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb67fc640 1 --2- 172.21.15.88:0/1441896966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0eb8072f40 0x7f0eb80d1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb67fc640 1 -- 172.21.15.88:0/1441896966 >> v2:172.21.15.154:3300/0 conn(0x7f0eb8071150 msgr2=0x7f0eb80b5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:51.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb67fc640 1 --2- 172.21.15.88:0/1441896966 >> v2:172.21.15.154:3300/0 conn(0x7f0eb8071150 0x7f0eb80b5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb67fc640 1 -- 172.21.15.88:0/1441896966 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eb80d0a80 con 0x7f0eb8070660 2024-09-16T01:39:51.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb67fc640 1 -- 172.21.15.88:0/1441896966 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f0ea80019f0 con 0x7f0eb8070660 2024-09-16T01:39:51.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb67fc640 1 -- 172.21.15.88:0/1441896966 >> v1:172.21.15.88:6789/0 conn(0x7f0eb8070660 legacy=0x7f0eb8070a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:51.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb67fc640 1 --2- 172.21.15.88:0/1441896966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ea00039b0 0x7f0ea0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb67fc640 1 --2- 172.21.15.88:0/1441896966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0eb8072f40 0x7f0ea0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb67fc640 1 --2- 172.21.15.88:0/1441896966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ea0004d40 0x7f0ea00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb67fc640 1 -- 172.21.15.88:0/1441896966 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb80d04c0 con 0x7f0ea0004d40 2024-09-16T01:39:51.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb67fc640 1 -- 172.21.15.88:0/1441896966 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb80d07a0 con 0x7f0ea00039b0 2024-09-16T01:39:51.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb67fc640 1 -- 172.21.15.88:0/1441896966 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0ea000b9c0 con 0x7f0eb8072f40 2024-09-16T01:39:51.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb7fff640 1 --2- 172.21.15.88:0/1441896966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ea0004d40 0x7f0ea00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb77fe640 1 --2- 172.21.15.88:0/1441896966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ea00039b0 0x7f0ea0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb6ffd640 1 --2- 172.21.15.88:0/1441896966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0eb8072f40 0x7f0ea0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb7fff640 1 -- 172.21.15.88:0/1441896966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0eb8072f40 msgr2=0x7f0ea0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:51.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb7fff640 1 --2- 172.21.15.88:0/1441896966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0eb8072f40 0x7f0ea0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb7fff640 1 -- 172.21.15.88:0/1441896966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ea00039b0 msgr2=0x7f0ea0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:51.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb7fff640 1 --2- 172.21.15.88:0/1441896966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ea00039b0 0x7f0ea0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.372+0000 7f0eb7fff640 1 -- 172.21.15.88:0/1441896966 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ea0003520 con 0x7f0ea0004d40 2024-09-16T01:39:51.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.373+0000 7f0eb7fff640 1 --2- 172.21.15.88:0/1441896966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ea0004d40 0x7f0ea00051c0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0009ac0 tx=0x7f0ea0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=694450fc633252b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:51.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.373+0000 7f0eb67fc640 1 -- 172.21.15.88:0/1441896966 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb0036070 con 0x7f0ea0004d40 2024-09-16T01:39:51.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.373+0000 7f0eb67fc640 1 -- 172.21.15.88:0/1441896966 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0eb0031400 con 0x7f0ea0004d40 2024-09-16T01:39:51.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.373+0000 7f0eb67fc640 1 -- 172.21.15.88:0/1441896966 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb0009690 con 0x7f0ea0004d40 2024-09-16T01:39:51.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.374+0000 7f0ebcd9f640 1 -- 172.21.15.88:0/1441896966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ea0004d40 msgr2=0x7f0ea00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:51.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.374+0000 7f0ebcd9f640 1 --2- 172.21.15.88:0/1441896966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ea0004d40 0x7f0ea00051c0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0009ac0 tx=0x7f0ea0005830 comp rx=0 tx=0).stop 2024-09-16T01:39:51.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.374+0000 7f0eb7fff640 1 -- 172.21.15.88:0/1441896966 reap_dead start 2024-09-16T01:39:51.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.374+0000 7f0ebcd9f640 1 -- 172.21.15.88:0/1441896966 shutdown_connections 2024-09-16T01:39:51.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.374+0000 7f0ebcd9f640 1 -- 172.21.15.88:0/1441896966 >> 172.21.15.88:0/1441896966 conn(0x7f0eb80d2300 msgr2=0x7f0eb80d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:51.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.374+0000 7f0ebcd9f640 1 -- 172.21.15.88:0/1441896966 shutdown_connections 2024-09-16T01:39:51.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.374+0000 7f0ebcd9f640 1 -- 172.21.15.88:0/1441896966 wait complete. 2024-09-16T01:39:51.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.375+0000 7f0ebcd9f640 1 Processor -- start 2024-09-16T01:39:51.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.375+0000 7f0ebcd9f640 1 -- start start 2024-09-16T01:39:51.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.376+0000 7f0ebcd9f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ea0004d40 0x7f0eb8154500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.376+0000 7f0ebcd9f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0eb81517a0 0x7f0eb813f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.376+0000 7f0ebcd9f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0eb8151c20 0x7f0eb81f5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.376+0000 7f0ebcd9f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb80ceb60 con 0x7f0eb8151c20 2024-09-16T01:39:51.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.376+0000 7f0ebcd9f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0eb80ce9e0 con 0x7f0eb81517a0 2024-09-16T01:39:51.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.376+0000 7f0ebcd9f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0eb815c630 con 0x7f0ea0004d40 2024-09-16T01:39:51.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.376+0000 7f0eb7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0eb8151c20 0x7f0eb81f5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.376+0000 7f0eb7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0eb8151c20 0x7f0eb81f5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60030/0 (socket says 172.21.15.88:60030) 2024-09-16T01:39:51.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.376+0000 7f0eb7fff640 1 -- 172.21.15.88:0/2218004735 learned_addr learned my addr 172.21.15.88:0/2218004735 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:51.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.376+0000 7f0eb77fe640 1 --2- 172.21.15.88:0/2218004735 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ea0004d40 0x7f0eb8154500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.376+0000 7f0eb6ffd640 1 --2- 172.21.15.88:0/2218004735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0eb81517a0 0x7f0eb813f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.376+0000 7f0eb7fff640 1 -- 172.21.15.88:0/2218004735 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ea0004d40 msgr2=0x7f0eb8154500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:51.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.376+0000 7f0eb7fff640 1 --2- 172.21.15.88:0/2218004735 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ea0004d40 0x7f0eb8154500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.376+0000 7f0eb7fff640 1 -- 172.21.15.88:0/2218004735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0eb81517a0 msgr2=0x7f0eb813f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:51.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.376+0000 7f0eb7fff640 1 --2- 172.21.15.88:0/2218004735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0eb81517a0 0x7f0eb813f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.376+0000 7f0eb7fff640 1 -- 172.21.15.88:0/2218004735 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eb81534c0 con 0x7f0eb8151c20 2024-09-16T01:39:51.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.376+0000 7f0eb7fff640 1 --2- 172.21.15.88:0/2218004735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0eb8151c20 0x7f0eb81f5370 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0009ac0 tx=0x7f0eb0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:51.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.376+0000 7f0e9ffff640 1 -- 172.21.15.88:0/2218004735 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb0036040 con 0x7f0eb8151c20 2024-09-16T01:39:51.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.377+0000 7f0e9ffff640 1 -- 172.21.15.88:0/2218004735 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0eb0041070 con 0x7f0eb8151c20 2024-09-16T01:39:51.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.377+0000 7f0e9ffff640 1 -- 172.21.15.88:0/2218004735 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb0009690 con 0x7f0eb8151c20 2024-09-16T01:39:51.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.377+0000 7f0ebcd9f640 1 -- 172.21.15.88:0/2218004735 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eb8153740 con 0x7f0eb8151c20 2024-09-16T01:39:51.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.377+0000 7f0ebcd9f640 1 -- 172.21.15.88:0/2218004735 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eb8153d00 con 0x7f0eb8151c20 2024-09-16T01:39:51.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.378+0000 7f0ebcd9f640 1 -- 172.21.15.88:0/2218004735 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0eb80b5310 con 0x7f0eb8151c20 2024-09-16T01:39:51.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.378+0000 7f0e9ffff640 1 -- 172.21.15.88:0/2218004735 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0eb0039070 con 0x7f0eb8151c20 2024-09-16T01:39:51.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.379+0000 7f0e9ffff640 1 --2- 172.21.15.88:0/2218004735 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0ea406c2b0 0x7f0ea406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.379+0000 7f0e9ffff640 1 -- 172.21.15.88:0/2218004735 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5795+0+0 (secure 0 0 0) 0x7f0eb00aac50 con 0x7f0eb8151c20 2024-09-16T01:39:51.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.379+0000 7f0eb77fe640 1 --2- 172.21.15.88:0/2218004735 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0ea406c2b0 0x7f0ea406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.379+0000 7f0eb77fe640 1 --2- 172.21.15.88:0/2218004735 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0ea406c2b0 0x7f0ea406e770 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8000f80 tx=0x7f0ea8006040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:51.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.383+0000 7f0e9ffff640 1 -- 172.21.15.88:0/2218004735 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0eb00b0020 con 0x7f0eb8151c20 2024-09-16T01:39:51.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.560+0000 7f0ebcd9f640 1 -- 172.21.15.88:0/2218004735 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0eb8160b40 con 0x7f0eb8151c20 2024-09-16T01:39:51.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.560+0000 7f0e9ffff640 1 -- 172.21.15.88:0/2218004735 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v73) ==== 80+0+124 (secure 0 0 0) 0x7f0eb004a020 con 0x7f0eb8151c20 2024-09-16T01:39:51.562 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.154:6835/3482042494 2024-09-17T01:39:46.253065+0000 2024-09-16T01:39:51.562 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.154:6833/3482042494 2024-09-17T01:39:46.253065+0000 2024-09-16T01:39:51.562 INFO:teuthology.orchestra.run.smithi088.stderr:listed 2 entries 2024-09-16T01:39:51.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.562+0000 7f0ebcd9f640 1 -- 172.21.15.88:0/2218004735 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0ea406c2b0 msgr2=0x7f0ea406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:51.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.562+0000 7f0ebcd9f640 1 --2- 172.21.15.88:0/2218004735 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0ea406c2b0 0x7f0ea406e770 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8000f80 tx=0x7f0ea8006040 comp rx=0 tx=0).stop 2024-09-16T01:39:51.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.562+0000 7f0ebcd9f640 1 -- 172.21.15.88:0/2218004735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0eb8151c20 msgr2=0x7f0eb81f5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:51.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.562+0000 7f0ebcd9f640 1 --2- 172.21.15.88:0/2218004735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0eb8151c20 0x7f0eb81f5370 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0009ac0 tx=0x7f0eb0004730 comp rx=0 tx=0).stop 2024-09-16T01:39:51.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.562+0000 7f0ebcd9f640 1 -- 172.21.15.88:0/2218004735 shutdown_connections 2024-09-16T01:39:51.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.562+0000 7f0ebcd9f640 1 --2- 172.21.15.88:0/2218004735 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0ea406c2b0 0x7f0ea406e770 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.562+0000 7f0ebcd9f640 1 --2- 172.21.15.88:0/2218004735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0eb8151c20 0x7f0eb81f5370 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.562+0000 7f0ebcd9f640 1 --2- 172.21.15.88:0/2218004735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0eb81517a0 0x7f0eb813f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.562+0000 7f0ebcd9f640 1 --2- 172.21.15.88:0/2218004735 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ea0004d40 0x7f0eb8154500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.562+0000 7f0ebcd9f640 1 -- 172.21.15.88:0/2218004735 >> 172.21.15.88:0/2218004735 conn(0x7f0eb80d2300 msgr2=0x7f0eb80d1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:51.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.563+0000 7f0ebcd9f640 1 -- 172.21.15.88:0/2218004735 shutdown_connections 2024-09-16T01:39:51.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.563+0000 7f0ebcd9f640 1 -- 172.21.15.88:0/2218004735 wait complete. 2024-09-16T01:39:51.574 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-16T01:39:51.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:39:51.575 DEBUG:teuthology.orchestra.run.smithi088:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-16T01:39:51.602 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:39:51.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:39:51.603 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:39:51.659 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T01:39:51.659 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:39:51.659 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:39:51.679 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056026,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1347 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T01:39:51.679 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T01:39:51.679 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:39:51.679 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:39:51.715 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:39:51.715 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T01:39:51.715 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:39:51.715 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:39:51.733 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:39:51.732+0000 7efd80c94640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:39:51.738 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056026,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1347 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T01:39:51.738 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:39:51.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.834+0000 7fab4b711640 1 Processor -- start 2024-09-16T01:39:51.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.835+0000 7fab4b711640 1 -- start start 2024-09-16T01:39:51.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.835+0000 7fab4b711640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fab4415e800 0x7fab440b5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.835+0000 7fab4b711640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fab440b7560 0x7fab4415f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.835+0000 7fab4b711640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab4406ea60 con 0x7fab4415ebe0 2024-09-16T01:39:51.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.835+0000 7fab4b711640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fab4406ed40 con 0x7fab440b7560 2024-09-16T01:39:51.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.835+0000 7fab4b711640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fab4406cde0 con 0x7fab4415e800 2024-09-16T01:39:51.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.835+0000 7fab49cae640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fab440b7560 0x7fab4415f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.835+0000 7fab48cac640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fab4415ebe0 0x7fab440b6e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55938/0 (socket says 172.21.15.88:55938) 2024-09-16T01:39:51.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.836+0000 7fab49cae640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fab440b7560 0x7fab4415f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60032/0 (socket says 172.21.15.88:60032) 2024-09-16T01:39:51.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.836+0000 7fab48cac640 1 -- 172.21.15.88:0/1128836677 learned_addr learned my addr 172.21.15.88:0/1128836677 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:51.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.836+0000 7fab49cae640 1 -- 172.21.15.88:0/1128836677 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fab4415e800 msgr2=0x7fab440b5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:51.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.836+0000 7fab49cae640 1 --2- 172.21.15.88:0/1128836677 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fab4415e800 0x7fab440b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.836+0000 7fab49cae640 1 -- 172.21.15.88:0/1128836677 >> v1:172.21.15.88:6789/0 conn(0x7fab4415ebe0 legacy=0x7fab440b6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:51.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.836+0000 7fab49cae640 1 -- 172.21.15.88:0/1128836677 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab4406d0c0 con 0x7fab440b7560 2024-09-16T01:39:51.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.836+0000 7fab3bfff640 1 -- 172.21.15.88:0/1128836677 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4158223857 0 0) 0x7fab4406ea60 con 0x7fab4415ebe0 2024-09-16T01:39:51.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.836+0000 7fab49cae640 1 --2- 172.21.15.88:0/1128836677 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab440b7560 0x7fab4415f040 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fab3c009d90 tx=0x7fab3c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b87d44776e557a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:51.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.837+0000 7fab3bfff640 1 -- 172.21.15.88:0/1128836677 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab3c007740 con 0x7fab440b7560 2024-09-16T01:39:51.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.837+0000 7fab3bfff640 1 -- 172.21.15.88:0/1128836677 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab3c0078e0 con 0x7fab440b7560 2024-09-16T01:39:51.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.837+0000 7fab3bfff640 1 -- 172.21.15.88:0/1128836677 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab3c007cb0 con 0x7fab440b7560 2024-09-16T01:39:51.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.837+0000 7fab4b711640 1 -- 172.21.15.88:0/1128836677 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab440b7560 msgr2=0x7fab4415f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:51.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.837+0000 7fab4b711640 1 --2- 172.21.15.88:0/1128836677 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab440b7560 0x7fab4415f040 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fab3c009d90 tx=0x7fab3c02a660 comp rx=0 tx=0).stop 2024-09-16T01:39:51.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.837+0000 7fab4b711640 1 -- 172.21.15.88:0/1128836677 shutdown_connections 2024-09-16T01:39:51.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.837+0000 7fab4b711640 1 --2- 172.21.15.88:0/1128836677 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab440b7560 0x7fab4415f040 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.837+0000 7fab4b711640 1 --2- 172.21.15.88:0/1128836677 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fab4415e800 0x7fab440b5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.837+0000 7fab4b711640 1 -- 172.21.15.88:0/1128836677 >> 172.21.15.88:0/1128836677 conn(0x7fab440d24d0 msgr2=0x7fab440d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:51.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.837+0000 7fab4b711640 1 -- 172.21.15.88:0/1128836677 shutdown_connections 2024-09-16T01:39:51.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.837+0000 7fab4b711640 1 -- 172.21.15.88:0/1128836677 wait complete. 2024-09-16T01:39:51.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.838+0000 7fab4b711640 1 Processor -- start 2024-09-16T01:39:51.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.838+0000 7fab4b711640 1 -- start start 2024-09-16T01:39:51.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.839+0000 7fab4b711640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab440b7560 0x7fab440cb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.839+0000 7fab4b711640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fab4415e800 0x7fab441f3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.839+0000 7fab4b711640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab4415ebe0 0x7fab44154fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.839+0000 7fab4b711640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fab4406e680 con 0x7fab440b7560 2024-09-16T01:39:51.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.839+0000 7fab4b711640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fab440d31e0 con 0x7fab4415ebe0 2024-09-16T01:39:51.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.839+0000 7fab4b711640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fab4406e800 con 0x7fab4415e800 2024-09-16T01:39:51.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.839+0000 7fab494ad640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab440b7560 0x7fab440cb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.839+0000 7fab494ad640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab440b7560 0x7fab440cb7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60048/0 (socket says 172.21.15.88:60048) 2024-09-16T01:39:51.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.839+0000 7fab494ad640 1 -- 172.21.15.88:0/2080587604 learned_addr learned my addr 172.21.15.88:0/2080587604 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:51.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.839+0000 7fab494ad640 1 -- 172.21.15.88:0/2080587604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fab4415e800 msgr2=0x7fab441f3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:51.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.839+0000 7fab48cac640 1 --2- 172.21.15.88:0/2080587604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fab4415e800 0x7fab441f3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.839+0000 7fab49cae640 1 --2- 172.21.15.88:0/2080587604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab4415ebe0 0x7fab44154fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.839+0000 7fab494ad640 1 --2- 172.21.15.88:0/2080587604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fab4415e800 0x7fab441f3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.840+0000 7fab494ad640 1 -- 172.21.15.88:0/2080587604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab4415ebe0 msgr2=0x7fab44154fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:51.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.840+0000 7fab494ad640 1 --2- 172.21.15.88:0/2080587604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab4415ebe0 0x7fab44154fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:51.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.840+0000 7fab494ad640 1 -- 172.21.15.88:0/2080587604 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab44156300 con 0x7fab440b7560 2024-09-16T01:39:51.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.840+0000 7fab49cae640 1 --2- 172.21.15.88:0/2080587604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab4415ebe0 0x7fab44154fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:51.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.840+0000 7fab48cac640 1 --2- 172.21.15.88:0/2080587604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fab4415e800 0x7fab441f3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:51.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.840+0000 7fab494ad640 1 --2- 172.21.15.88:0/2080587604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab440b7560 0x7fab440cb7e0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fab4000b4a0 tx=0x7fab4000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:51.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.840+0000 7fab39ffb640 1 -- 172.21.15.88:0/2080587604 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab40014070 con 0x7fab440b7560 2024-09-16T01:39:51.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.840+0000 7fab39ffb640 1 -- 172.21.15.88:0/2080587604 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab40010610 con 0x7fab440b7560 2024-09-16T01:39:51.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.840+0000 7fab4b711640 1 -- 172.21.15.88:0/2080587604 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab441f4770 con 0x7fab440b7560 2024-09-16T01:39:51.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.840+0000 7fab39ffb640 1 -- 172.21.15.88:0/2080587604 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab40010a00 con 0x7fab440b7560 2024-09-16T01:39:51.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.840+0000 7fab4b711640 1 -- 172.21.15.88:0/2080587604 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab441f4d30 con 0x7fab440b7560 2024-09-16T01:39:51.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.841+0000 7fab4b711640 1 -- 172.21.15.88:0/2080587604 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab4415f040 con 0x7fab440b7560 2024-09-16T01:39:51.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.841+0000 7fab39ffb640 1 -- 172.21.15.88:0/2080587604 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fab40017070 con 0x7fab440b7560 2024-09-16T01:39:51.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.842+0000 7fab39ffb640 1 --2- 172.21.15.88:0/2080587604 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fab2806c2b0 0x7fab2806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:51.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.842+0000 7fab39ffb640 1 -- 172.21.15.88:0/2080587604 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5795+0+0 (secure 0 0 0) 0x7fab4008ae90 con 0x7fab440b7560 2024-09-16T01:39:51.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.842+0000 7fab48cac640 1 --2- 172.21.15.88:0/2080587604 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fab2806c2b0 0x7fab2806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:51.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.842+0000 7fab48cac640 1 --2- 172.21.15.88:0/2080587604 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fab2806c2b0 0x7fab2806e770 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fab440cbd20 tx=0x7fab34009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:51.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:51.846+0000 7fab39ffb640 1 -- 172.21.15.88:0/2080587604 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fab400595a0 con 0x7fab440b7560 2024-09-16T01:39:52.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.047+0000 7fab4b711640 1 -- 172.21.15.88:0/2080587604 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab441620a0 con 0x7fab440b7560 2024-09-16T01:39:52.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.047+0000 7fab39ffb640 1 -- 172.21.15.88:0/2080587604 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+5233 (secure 0 0 0) 0x7fab40058ea0 con 0x7fab440b7560 2024-09-16T01:39:52.049 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:52.049 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":64,"btime":"2024-09-16T01:39:49:271443+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5725,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2696547723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2696547723},{"type":"v1","addr":"172.21.15.154:6834","nonce":2696547723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54},{"gid":5863,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2167482844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2167482844},{"type":"v1","addr":"172.21.15.154:6835","nonce":2167482844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62},{"gid":6411,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3276199399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3276199399},{"type":"v1","addr":"172.21.15.88:6836","nonce":3276199399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:39:14.808422+0000","modified":"2024-09-16T01:39:49.271439+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6414},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":12,"addr":"172.21.15.88:6837/2944989249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2944989249},{"type":"v1","addr":"172.21.15.88:6837","nonce":2944989249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6414,"qdb_cluster":[6414]},"id":5}]} 2024-09-16T01:39:52.049 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 64 2024-09-16T01:39:52.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.049+0000 7fab4b711640 1 -- 172.21.15.88:0/2080587604 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fab2806c2b0 msgr2=0x7fab2806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:52.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.049+0000 7fab4b711640 1 --2- 172.21.15.88:0/2080587604 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fab2806c2b0 0x7fab2806e770 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fab440cbd20 tx=0x7fab34009040 comp rx=0 tx=0).stop 2024-09-16T01:39:52.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.049+0000 7fab4b711640 1 -- 172.21.15.88:0/2080587604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab440b7560 msgr2=0x7fab440cb7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:52.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.049+0000 7fab4b711640 1 --2- 172.21.15.88:0/2080587604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab440b7560 0x7fab440cb7e0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fab4000b4a0 tx=0x7fab4000b960 comp rx=0 tx=0).stop 2024-09-16T01:39:52.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.049+0000 7fab4b711640 1 -- 172.21.15.88:0/2080587604 shutdown_connections 2024-09-16T01:39:52.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.049+0000 7fab4b711640 1 --2- 172.21.15.88:0/2080587604 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fab2806c2b0 0x7fab2806e770 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.049+0000 7fab4b711640 1 --2- 172.21.15.88:0/2080587604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab4415ebe0 0x7fab44154fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.049+0000 7fab4b711640 1 --2- 172.21.15.88:0/2080587604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fab4415e800 0x7fab441f3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.049+0000 7fab4b711640 1 --2- 172.21.15.88:0/2080587604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab440b7560 0x7fab440cb7e0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.049+0000 7fab4b711640 1 -- 172.21.15.88:0/2080587604 >> 172.21.15.88:0/2080587604 conn(0x7fab440d24d0 msgr2=0x7fab4415d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:52.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.049+0000 7fab4b711640 1 -- 172.21.15.88:0/2080587604 shutdown_connections 2024-09-16T01:39:52.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.049+0000 7fab4b711640 1 -- 172.21.15.88:0/2080587604 wait complete. 2024-09-16T01:39:52.062 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:39:52.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.160+0000 7f6d314af640 1 Processor -- start 2024-09-16T01:39:52.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.160+0000 7f6d314af640 1 -- start start 2024-09-16T01:39:52.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.161+0000 7f6d314af640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6d2c15e660 0x7f6d2c0cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:52.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.161+0000 7f6d314af640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d2c0d3310 0x7f6d2c0d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:52.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.161+0000 7f6d314af640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6d2c0d1650 0x7f6d2c06bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:52.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.161+0000 7f6d314af640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6d2c0d1ac0 con 0x7f6d2c15e660 2024-09-16T01:39:52.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.161+0000 7f6d314af640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6d2c15ea40 con 0x7f6d2c0d1650 2024-09-16T01:39:52.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.161+0000 7f6d314af640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6d2c0d3870 con 0x7f6d2c0d3310 2024-09-16T01:39:52.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.161+0000 7f6d2affd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6d2c15e660 0x7f6d2c0cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:52.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.161+0000 7f6d2affd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6d2c15e660 0x7f6d2c0cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60050/0 (socket says 172.21.15.88:60050) 2024-09-16T01:39:52.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.161+0000 7f6d2affd640 1 -- 172.21.15.88:0/2257297019 learned_addr learned my addr 172.21.15.88:0/2257297019 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:52.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.161+0000 7f6d2affd640 1 -- 172.21.15.88:0/2257297019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d2c0d3310 msgr2=0x7f6d2c0d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:52.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.161+0000 7f6d2a7fc640 1 --2- 172.21.15.88:0/2257297019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d2c0d3310 0x7f6d2c0d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:52.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.161+0000 7f6d2affd640 1 --2- 172.21.15.88:0/2257297019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d2c0d3310 0x7f6d2c0d1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.161+0000 7f6d2affd640 1 -- 172.21.15.88:0/2257297019 >> v2:172.21.15.154:3300/0 conn(0x7f6d2c0d1650 msgr2=0x7f6d2c06bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:52.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.161+0000 7f6d2b7fe640 1 --2- 172.21.15.88:0/2257297019 >> v2:172.21.15.154:3300/0 conn(0x7f6d2c0d1650 0x7f6d2c06bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:52.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.161+0000 7f6d2affd640 1 --2- 172.21.15.88:0/2257297019 >> v2:172.21.15.154:3300/0 conn(0x7f6d2c0d1650 0x7f6d2c06bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.161+0000 7f6d2affd640 1 -- 172.21.15.88:0/2257297019 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d2c0d3b50 con 0x7f6d2c15e660 2024-09-16T01:39:52.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.161+0000 7f6d2b7fe640 1 --2- 172.21.15.88:0/2257297019 >> v2:172.21.15.154:3300/0 conn(0x7f6d2c0d1650 0x7f6d2c06bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:52.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.162+0000 7f6d2affd640 1 --2- 172.21.15.88:0/2257297019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d2c15e660 0x7f6d2c0cf730 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c0cb760 tx=0x7f6d1c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3e907bc6054c9c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:52.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.162+0000 7f6d29ffb640 1 -- 172.21.15.88:0/2257297019 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c036070 con 0x7f6d2c15e660 2024-09-16T01:39:52.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.162+0000 7f6d29ffb640 1 -- 172.21.15.88:0/2257297019 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d1c031600 con 0x7f6d2c15e660 2024-09-16T01:39:52.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.162+0000 7f6d29ffb640 1 -- 172.21.15.88:0/2257297019 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c0319d0 con 0x7f6d2c15e660 2024-09-16T01:39:52.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.163+0000 7f6d314af640 1 -- 172.21.15.88:0/2257297019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d2c15e660 msgr2=0x7f6d2c0cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:52.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.163+0000 7f6d314af640 1 --2- 172.21.15.88:0/2257297019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d2c15e660 0x7f6d2c0cf730 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c0cb760 tx=0x7f6d1c02a6a0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.163+0000 7f6d314af640 1 -- 172.21.15.88:0/2257297019 shutdown_connections 2024-09-16T01:39:52.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.163+0000 7f6d314af640 1 --2- 172.21.15.88:0/2257297019 >> v2:172.21.15.154:3300/0 conn(0x7f6d2c0d1650 0x7f6d2c06bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.163+0000 7f6d314af640 1 --2- 172.21.15.88:0/2257297019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d2c0d3310 0x7f6d2c0d1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.163+0000 7f6d314af640 1 --2- 172.21.15.88:0/2257297019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d2c15e660 0x7f6d2c0cf730 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.163+0000 7f6d314af640 1 -- 172.21.15.88:0/2257297019 >> 172.21.15.88:0/2257297019 conn(0x7f6d2c0b6410 msgr2=0x7f6d2c0b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:52.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.163+0000 7f6d314af640 1 -- 172.21.15.88:0/2257297019 shutdown_connections 2024-09-16T01:39:52.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.163+0000 7f6d314af640 1 -- 172.21.15.88:0/2257297019 wait complete. 2024-09-16T01:39:52.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.164+0000 7f6d314af640 1 Processor -- start 2024-09-16T01:39:52.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.164+0000 7f6d314af640 1 -- start start 2024-09-16T01:39:52.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.165+0000 7f6d314af640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d2c0d1650 0x7f6d2c15ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:52.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.165+0000 7f6d314af640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d2c0d3310 0x7f6d2c15f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:52.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.165+0000 7f6d314af640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d2c15e660 0x7f6d2c159db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:52.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.165+0000 7f6d314af640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2c0d43f0 con 0x7f6d2c0d3310 2024-09-16T01:39:52.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.165+0000 7f6d314af640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2c0d4110 con 0x7f6d2c15e660 2024-09-16T01:39:52.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.165+0000 7f6d314af640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6d2c0cfc70 con 0x7f6d2c0d1650 2024-09-16T01:39:52.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.165+0000 7f6d2a7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d2c0d3310 0x7f6d2c15f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:52.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.165+0000 7f6d2a7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d2c0d3310 0x7f6d2c15f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60060/0 (socket says 172.21.15.88:60060) 2024-09-16T01:39:52.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.165+0000 7f6d2a7fc640 1 -- 172.21.15.88:0/4289175979 learned_addr learned my addr 172.21.15.88:0/4289175979 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:52.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.165+0000 7f6d2affd640 1 --2- 172.21.15.88:0/4289175979 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d2c0d1650 0x7f6d2c15ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:52.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.165+0000 7f6d2b7fe640 1 --2- 172.21.15.88:0/4289175979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d2c15e660 0x7f6d2c159db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:52.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.165+0000 7f6d2a7fc640 1 -- 172.21.15.88:0/4289175979 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d2c0d1650 msgr2=0x7f6d2c15ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:52.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.165+0000 7f6d2a7fc640 1 --2- 172.21.15.88:0/4289175979 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d2c0d1650 0x7f6d2c15ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.165+0000 7f6d2a7fc640 1 -- 172.21.15.88:0/4289175979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d2c15e660 msgr2=0x7f6d2c159db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:52.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.165+0000 7f6d2a7fc640 1 --2- 172.21.15.88:0/4289175979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d2c15e660 0x7f6d2c159db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.165+0000 7f6d2a7fc640 1 -- 172.21.15.88:0/4289175979 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d2c15ab70 con 0x7f6d2c0d3310 2024-09-16T01:39:52.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.165+0000 7f6d2a7fc640 1 --2- 172.21.15.88:0/4289175979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d2c0d3310 0x7f6d2c15f150 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f6d2000c7f0 tx=0x7f6d2000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:52.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.165+0000 7f6d137fe640 1 -- 172.21.15.88:0/4289175979 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d20015070 con 0x7f6d2c0d3310 2024-09-16T01:39:52.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.165+0000 7f6d137fe640 1 -- 172.21.15.88:0/4289175979 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d20004220 con 0x7f6d2c0d3310 2024-09-16T01:39:52.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.166+0000 7f6d137fe640 1 -- 172.21.15.88:0/4289175979 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d20007620 con 0x7f6d2c0d3310 2024-09-16T01:39:52.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.166+0000 7f6d314af640 1 -- 172.21.15.88:0/4289175979 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d2c1e71e0 con 0x7f6d2c0d3310 2024-09-16T01:39:52.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.166+0000 7f6d314af640 1 -- 172.21.15.88:0/4289175979 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d2c1e77a0 con 0x7f6d2c0d3310 2024-09-16T01:39:52.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.167+0000 7f6d314af640 1 -- 172.21.15.88:0/4289175979 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d2c0d1110 con 0x7f6d2c0d3310 2024-09-16T01:39:52.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.167+0000 7f6d137fe640 1 -- 172.21.15.88:0/4289175979 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6d200079d0 con 0x7f6d2c0d3310 2024-09-16T01:39:52.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.167+0000 7f6d137fe640 1 --2- 172.21.15.88:0/4289175979 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6d1806c2b0 0x7f6d1806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:52.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.167+0000 7f6d137fe640 1 -- 172.21.15.88:0/4289175979 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5795+0+0 (secure 0 0 0) 0x7f6d2008b7d0 con 0x7f6d2c0d3310 2024-09-16T01:39:52.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.168+0000 7f6d2affd640 1 --2- 172.21.15.88:0/4289175979 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6d1806c2b0 0x7f6d1806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:52.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.168+0000 7f6d2affd640 1 --2- 172.21.15.88:0/4289175979 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6d1806c2b0 0x7f6d1806e770 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c033000 tx=0x7f6d1c009000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:52.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.172+0000 7f6d137fe640 1 -- 172.21.15.88:0/4289175979 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6d20059f70 con 0x7f6d2c0d3310 2024-09-16T01:39:52.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.373+0000 7f6d314af640 1 -- 172.21.15.88:0/4289175979 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d2c15bad0 con 0x7f6d2c0d3310 2024-09-16T01:39:52.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.374+0000 7f6d137fe640 1 -- 172.21.15.88:0/4289175979 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+5233 (secure 0 0 0) 0x7f6d20059870 con 0x7f6d2c0d3310 2024-09-16T01:39:52.376 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:52.376 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":64,"btime":"2024-09-16T01:39:49:271443+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5725,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2696547723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2696547723},{"type":"v1","addr":"172.21.15.154:6834","nonce":2696547723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54},{"gid":5863,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2167482844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2167482844},{"type":"v1","addr":"172.21.15.154:6835","nonce":2167482844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62},{"gid":6411,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3276199399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3276199399},{"type":"v1","addr":"172.21.15.88:6836","nonce":3276199399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:39:14.808422+0000","modified":"2024-09-16T01:39:49.271439+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6414},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":12,"addr":"172.21.15.88:6837/2944989249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2944989249},{"type":"v1","addr":"172.21.15.88:6837","nonce":2944989249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6414,"qdb_cluster":[6414]},"id":5}]} 2024-09-16T01:39:52.377 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 64 2024-09-16T01:39:52.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.376+0000 7f6d314af640 1 -- 172.21.15.88:0/4289175979 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6d1806c2b0 msgr2=0x7f6d1806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:52.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.376+0000 7f6d314af640 1 --2- 172.21.15.88:0/4289175979 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6d1806c2b0 0x7f6d1806e770 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c033000 tx=0x7f6d1c009000 comp rx=0 tx=0).stop 2024-09-16T01:39:52.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.376+0000 7f6d314af640 1 -- 172.21.15.88:0/4289175979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d2c0d3310 msgr2=0x7f6d2c15f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:52.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.376+0000 7f6d314af640 1 --2- 172.21.15.88:0/4289175979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d2c0d3310 0x7f6d2c15f150 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f6d2000c7f0 tx=0x7f6d2000ccb0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.376+0000 7f6d314af640 1 -- 172.21.15.88:0/4289175979 shutdown_connections 2024-09-16T01:39:52.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.376+0000 7f6d314af640 1 --2- 172.21.15.88:0/4289175979 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6d1806c2b0 0x7f6d1806e770 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.376+0000 7f6d314af640 1 --2- 172.21.15.88:0/4289175979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d2c15e660 0x7f6d2c159db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.376+0000 7f6d314af640 1 --2- 172.21.15.88:0/4289175979 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d2c0d3310 0x7f6d2c15f150 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.376+0000 7f6d314af640 1 --2- 172.21.15.88:0/4289175979 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d2c0d1650 0x7f6d2c15ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.376+0000 7f6d314af640 1 -- 172.21.15.88:0/4289175979 >> 172.21.15.88:0/4289175979 conn(0x7f6d2c0b6410 msgr2=0x7f6d2c0b5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:52.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.376+0000 7f6d314af640 1 -- 172.21.15.88:0/4289175979 shutdown_connections 2024-09-16T01:39:52.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.376+0000 7f6d314af640 1 -- 172.21.15.88:0/4289175979 wait complete. 2024-09-16T01:39:52.390 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:39:52.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.489+0000 7f09b2c4c640 1 Processor -- start 2024-09-16T01:39:52.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.489+0000 7f09b2c4c640 1 -- start start 2024-09-16T01:39:52.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.490+0000 7f09b2c4c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09ac15b460 0x7f09ac0b44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:52.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.490+0000 7f09b2c4c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09ac0b49f0 con 0x7f09ac06d430 2024-09-16T01:39:52.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.490+0000 7f09b2c4c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09ac15f130 con 0x7f09ac0d0fe0 2024-09-16T01:39:52.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.490+0000 7f09b2c4c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f09ac15eee0 con 0x7f09ac15b460 2024-09-16T01:39:52.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.490+0000 7f09b11e9640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f09ac06d430 0x7f09ac0b5080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55940/0 (socket says 172.21.15.88:55940) 2024-09-16T01:39:52.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.490+0000 7f09b11e9640 1 -- 172.21.15.88:0/3842609992 learned_addr learned my addr 172.21.15.88:0/3842609992 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:52.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.490+0000 7f09ab7fe640 1 -- 172.21.15.88:0/3842609992 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2296752861 0 0) 0x7f09ac0b49f0 con 0x7f09ac06d430 2024-09-16T01:39:52.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.490+0000 7f09b09e8640 1 --2- 172.21.15.88:0/3842609992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09ac15b460 0x7f09ac0b44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:52.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.490+0000 7f09ab7fe640 1 -- 172.21.15.88:0/3842609992 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f099c003520 con 0x7f09ac06d430 2024-09-16T01:39:52.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.490+0000 7f09ab7fe640 1 -- 172.21.15.88:0/3842609992 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 417140508 0 0) 0x7f099c003520 con 0x7f09ac06d430 2024-09-16T01:39:52.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.490+0000 7f09ab7fe640 1 -- 172.21.15.88:0/3842609992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09ac15b460 msgr2=0x7f09ac0b44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:52.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.490+0000 7f09ab7fe640 1 --2- 172.21.15.88:0/3842609992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09ac15b460 0x7f09ac0b44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.490+0000 7f09ab7fe640 1 -- 172.21.15.88:0/3842609992 >> v1:172.21.15.154:6789/0 conn(0x7f09ac0d0fe0 legacy=0x7f09ac06cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:52.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.490+0000 7f09ab7fe640 1 -- 172.21.15.88:0/3842609992 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09ac15f410 con 0x7f09ac06d430 2024-09-16T01:39:52.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.490+0000 7f09ab7fe640 1 -- 172.21.15.88:0/3842609992 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f09a40019f0 con 0x7f09ac06d430 2024-09-16T01:39:52.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.490+0000 7f09ab7fe640 1 -- 172.21.15.88:0/3842609992 >> v1:172.21.15.88:6789/0 conn(0x7f09ac06d430 legacy=0x7f09ac0b5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:52.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.490+0000 7f09ab7fe640 1 --2- 172.21.15.88:0/3842609992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09ac15b460 0x7f099c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:52.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.490+0000 7f09ab7fe640 1 --2- 172.21.15.88:0/3842609992 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f099c003f60 0x7f099c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:52.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.490+0000 7f09ab7fe640 1 --2- 172.21.15.88:0/3842609992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f099c004d40 0x7f099c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:52.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.490+0000 7f09ab7fe640 1 -- 172.21.15.88:0/3842609992 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac15eee0 con 0x7f099c003f60 2024-09-16T01:39:52.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.490+0000 7f09ab7fe640 1 -- 172.21.15.88:0/3842609992 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f099c00baa0 con 0x7f099c004d40 2024-09-16T01:39:52.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.490+0000 7f09ab7fe640 1 -- 172.21.15.88:0/3842609992 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f099c00bca0 con 0x7f09ac15b460 2024-09-16T01:39:52.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.490+0000 7f09abfff640 1 --2- 172.21.15.88:0/3842609992 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f099c003f60 0x7f099c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:52.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.491+0000 7f09b09e8640 1 --2- 172.21.15.88:0/3842609992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09ac15b460 0x7f099c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:52.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.491+0000 7f09b11e9640 1 --2- 172.21.15.88:0/3842609992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f099c004d40 0x7f099c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:52.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.491+0000 7f09abfff640 1 -- 172.21.15.88:0/3842609992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09ac15b460 msgr2=0x7f099c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:52.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.491+0000 7f09abfff640 1 --2- 172.21.15.88:0/3842609992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09ac15b460 0x7f099c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.491+0000 7f09abfff640 1 -- 172.21.15.88:0/3842609992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f099c004d40 msgr2=0x7f099c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:52.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.491+0000 7f09abfff640 1 --2- 172.21.15.88:0/3842609992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f099c004d40 0x7f099c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.491+0000 7f09abfff640 1 -- 172.21.15.88:0/3842609992 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f099c003520 con 0x7f099c003f60 2024-09-16T01:39:52.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.491+0000 7f09b11e9640 1 --2- 172.21.15.88:0/3842609992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f099c004d40 0x7f099c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:52.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.491+0000 7f09abfff640 1 --2- 172.21.15.88:0/3842609992 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f099c003f60 0x7f099c004800 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f09ac074140 tx=0x7f099c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e898545b2f3b936 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:52.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.492+0000 7f09ab7fe640 1 -- 172.21.15.88:0/3842609992 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0998001370 con 0x7f099c003f60 2024-09-16T01:39:52.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.492+0000 7f09ab7fe640 1 -- 172.21.15.88:0/3842609992 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09980052d0 con 0x7f099c003f60 2024-09-16T01:39:52.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.492+0000 7f09ab7fe640 1 -- 172.21.15.88:0/3842609992 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099802ab50 con 0x7f099c003f60 2024-09-16T01:39:52.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.492+0000 7f09b2c4c640 1 -- 172.21.15.88:0/3842609992 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f099c003f60 msgr2=0x7f099c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:52.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.492+0000 7f09b2c4c640 1 --2- 172.21.15.88:0/3842609992 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f099c003f60 0x7f099c004800 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f09ac074140 tx=0x7f099c005830 comp rx=0 tx=0).stop 2024-09-16T01:39:52.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.492+0000 7f09b11e9640 1 -- 172.21.15.88:0/3842609992 reap_dead start 2024-09-16T01:39:52.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.492+0000 7f09b2c4c640 1 -- 172.21.15.88:0/3842609992 shutdown_connections 2024-09-16T01:39:52.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.492+0000 7f09b2c4c640 1 -- 172.21.15.88:0/3842609992 >> 172.21.15.88:0/3842609992 conn(0x7f09ac0b6ec0 msgr2=0x7f09ac0b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:52.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.492+0000 7f09b2c4c640 1 -- 172.21.15.88:0/3842609992 shutdown_connections 2024-09-16T01:39:52.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.492+0000 7f09b2c4c640 1 -- 172.21.15.88:0/3842609992 wait complete. 2024-09-16T01:39:52.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.493+0000 7f09b2c4c640 1 Processor -- start 2024-09-16T01:39:52.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.493+0000 7f09b2c4c640 1 -- start start 2024-09-16T01:39:52.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.494+0000 7f09b2c4c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f099c003f60 0x7f09ac0b0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:52.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.494+0000 7f09b2c4c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09ac144eb0 0x7f09ac0c62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:52.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.494+0000 7f09b2c4c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09ac1f5040 0x7f09ac0b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:52.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.494+0000 7f09b2c4c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac0b44b0 con 0x7f09ac1f5040 2024-09-16T01:39:52.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.494+0000 7f09b2c4c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac15f9d0 con 0x7f099c003f60 2024-09-16T01:39:52.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.494+0000 7f09b2c4c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f09ac0b4630 con 0x7f09ac144eb0 2024-09-16T01:39:52.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.494+0000 7f09b11e9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09ac1f5040 0x7f09ac0b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:52.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.494+0000 7f09b11e9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09ac1f5040 0x7f09ac0b5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60070/0 (socket says 172.21.15.88:60070) 2024-09-16T01:39:52.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.494+0000 7f09b11e9640 1 -- 172.21.15.88:0/1735592181 learned_addr learned my addr 172.21.15.88:0/1735592181 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:52.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.494+0000 7f09abfff640 1 --2- 172.21.15.88:0/1735592181 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09ac144eb0 0x7f09ac0c62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:52.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.494+0000 7f09b09e8640 1 --2- 172.21.15.88:0/1735592181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f099c003f60 0x7f09ac0b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:52.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.495+0000 7f09b11e9640 1 -- 172.21.15.88:0/1735592181 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09ac144eb0 msgr2=0x7f09ac0c62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:52.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.495+0000 7f09b11e9640 1 --2- 172.21.15.88:0/1735592181 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09ac144eb0 0x7f09ac0c62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.495+0000 7f09b11e9640 1 -- 172.21.15.88:0/1735592181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f099c003f60 msgr2=0x7f09ac0b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:52.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.495+0000 7f09b11e9640 1 --2- 172.21.15.88:0/1735592181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f099c003f60 0x7f09ac0b0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.495+0000 7f09b11e9640 1 -- 172.21.15.88:0/1735592181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09ac145330 con 0x7f09ac1f5040 2024-09-16T01:39:52.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.495+0000 7f09abfff640 1 --2- 172.21.15.88:0/1735592181 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09ac144eb0 0x7f09ac0c62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:52.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.495+0000 7f09b09e8640 1 --2- 172.21.15.88:0/1735592181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f099c003f60 0x7f09ac0b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:52.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.495+0000 7f09b11e9640 1 --2- 172.21.15.88:0/1735592181 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09ac1f5040 0x7f09ac0b5a60 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f09ac0b7570 tx=0x7f09a400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:52.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.495+0000 7f09a97fa640 1 -- 172.21.15.88:0/1735592181 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a400eda0 con 0x7f09ac1f5040 2024-09-16T01:39:52.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.495+0000 7f09a97fa640 1 -- 172.21.15.88:0/1735592181 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09a4018070 con 0x7f09ac1f5040 2024-09-16T01:39:52.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.495+0000 7f09b2c4c640 1 -- 172.21.15.88:0/1735592181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09ac1f3fc0 con 0x7f09ac1f5040 2024-09-16T01:39:52.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.495+0000 7f09a97fa640 1 -- 172.21.15.88:0/1735592181 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a4013660 con 0x7f09ac1f5040 2024-09-16T01:39:52.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.495+0000 7f09b2c4c640 1 -- 172.21.15.88:0/1735592181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09ac1f4560 con 0x7f09ac1f5040 2024-09-16T01:39:52.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.496+0000 7f09b2c4c640 1 -- 172.21.15.88:0/1735592181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09ac06cd20 con 0x7f09ac1f5040 2024-09-16T01:39:52.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.496+0000 7f09a97fa640 1 -- 172.21.15.88:0/1735592181 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f09a4013800 con 0x7f09ac1f5040 2024-09-16T01:39:52.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.497+0000 7f09a97fa640 1 --2- 172.21.15.88:0/1735592181 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f099006c2e0 0x7f099006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:52.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.497+0000 7f09a97fa640 1 -- 172.21.15.88:0/1735592181 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5795+0+0 (secure 0 0 0) 0x7f09a408b8d0 con 0x7f09ac1f5040 2024-09-16T01:39:52.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.497+0000 7f09b09e8640 1 --2- 172.21.15.88:0/1735592181 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f099006c2e0 0x7f099006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:52.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.497+0000 7f09b09e8640 1 --2- 172.21.15.88:0/1735592181 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f099006c2e0 0x7f099006e7a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009ab0 tx=0x7f09a0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:52.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.501+0000 7f09a97fa640 1 -- 172.21.15.88:0/1735592181 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f09a4059fe0 con 0x7f09ac1f5040 2024-09-16T01:39:52.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.668+0000 7f09b2c4c640 1 -- 172.21.15.88:0/1735592181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f09ac0b5540 con 0x7f09ac1f5040 2024-09-16T01:39:52.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.669+0000 7f09a97fa640 1 -- 172.21.15.88:0/1735592181 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+16835 (secure 0 0 0) 0x7f09a405ee20 con 0x7f09ac1f5040 2024-09-16T01:39:52.671 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:52.671 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":73,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:39:46.253124+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:39:13.412390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-16T01:39:13.777039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.154:6835/3482042494":"2024-09-17T01:39:46.253065+0000","172.21.15.154:6833/3482042494":"2024-09-17T01:39:46.253065+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:39:52.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.671+0000 7f09b2c4c640 1 -- 172.21.15.88:0/1735592181 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f099006c2e0 msgr2=0x7f099006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:52.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.671+0000 7f09b2c4c640 1 --2- 172.21.15.88:0/1735592181 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f099006c2e0 0x7f099006e7a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009ab0 tx=0x7f09a0008040 comp rx=0 tx=0).stop 2024-09-16T01:39:52.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.671+0000 7f09b2c4c640 1 -- 172.21.15.88:0/1735592181 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09ac1f5040 msgr2=0x7f09ac0b5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:52.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.671+0000 7f09b2c4c640 1 --2- 172.21.15.88:0/1735592181 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09ac1f5040 0x7f09ac0b5a60 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f09ac0b7570 tx=0x7f09a400e530 comp rx=0 tx=0).stop 2024-09-16T01:39:52.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.671+0000 7f09b2c4c640 1 -- 172.21.15.88:0/1735592181 shutdown_connections 2024-09-16T01:39:52.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.671+0000 7f09b2c4c640 1 --2- 172.21.15.88:0/1735592181 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f099006c2e0 0x7f099006e7a0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.671+0000 7f09b2c4c640 1 --2- 172.21.15.88:0/1735592181 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09ac1f5040 0x7f09ac0b5a60 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.671+0000 7f09b2c4c640 1 --2- 172.21.15.88:0/1735592181 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09ac144eb0 0x7f09ac0c62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.671+0000 7f09b2c4c640 1 --2- 172.21.15.88:0/1735592181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f099c003f60 0x7f09ac0b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.671+0000 7f09b2c4c640 1 -- 172.21.15.88:0/1735592181 >> 172.21.15.88:0/1735592181 conn(0x7f09ac0b6ec0 msgr2=0x7f09ac06bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:52.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.671+0000 7f09b2c4c640 1 -- 172.21.15.88:0/1735592181 shutdown_connections 2024-09-16T01:39:52.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.671+0000 7f09b2c4c640 1 -- 172.21.15.88:0/1735592181 wait complete. 2024-09-16T01:39:52.684 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-16T01:39:52.684 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:39:52.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.781+0000 7f8ab229b640 1 Processor -- start 2024-09-16T01:39:52.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.781+0000 7f8ab229b640 1 -- start start 2024-09-16T01:39:52.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.782+0000 7f8ab229b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8aac0735b0 0x7f8aac0cf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:52.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.782+0000 7f8ab229b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8aac06faf0 con 0x7f8aac06d540 2024-09-16T01:39:52.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.782+0000 7f8ab229b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8aac06fdd0 con 0x7f8aac15e950 2024-09-16T01:39:52.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.782+0000 7f8ab229b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8aac0700b0 con 0x7f8aac0735b0 2024-09-16T01:39:52.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.782+0000 7f8aab7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8aac06d540 0x7f8aac0cf0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55950/0 (socket says 172.21.15.88:55950) 2024-09-16T01:39:52.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.782+0000 7f8aab7fe640 1 -- 172.21.15.88:0/4271563406 learned_addr learned my addr 172.21.15.88:0/4271563406 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:52.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.782+0000 7f8aaaffd640 1 -- 172.21.15.88:0/4271563406 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1902702884 0 0) 0x7f8aac06faf0 con 0x7f8aac06d540 2024-09-16T01:39:52.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.782+0000 7f8aaaffd640 1 -- 172.21.15.88:0/4271563406 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a94003520 con 0x7f8aac06d540 2024-09-16T01:39:52.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.782+0000 7f8ab0838640 1 --2- 172.21.15.88:0/4271563406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8aac0735b0 0x7f8aac0cf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:52.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.782+0000 7f8aaaffd640 1 -- 172.21.15.88:0/4271563406 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 84155966 0 0) 0x7f8a94003520 con 0x7f8aac06d540 2024-09-16T01:39:52.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.782+0000 7f8aaaffd640 1 -- 172.21.15.88:0/4271563406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8aac0735b0 msgr2=0x7f8aac0cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:52.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.782+0000 7f8aaaffd640 1 --2- 172.21.15.88:0/4271563406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8aac0735b0 0x7f8aac0cf7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.782+0000 7f8aaaffd640 1 -- 172.21.15.88:0/4271563406 >> v1:172.21.15.154:6789/0 conn(0x7f8aac15e950 legacy=0x7f8aac06ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:52.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.782+0000 7f8aaaffd640 1 -- 172.21.15.88:0/4271563406 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aac070390 con 0x7f8aac06d540 2024-09-16T01:39:52.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.782+0000 7f8aaaffd640 1 -- 172.21.15.88:0/4271563406 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f8aa0001670 con 0x7f8aac06d540 2024-09-16T01:39:52.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.783+0000 7f8aaaffd640 1 -- 172.21.15.88:0/4271563406 >> v1:172.21.15.88:6789/0 conn(0x7f8aac06d540 legacy=0x7f8aac0cf0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:52.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.783+0000 7f8aaaffd640 1 --2- 172.21.15.88:0/4271563406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8aac0735b0 0x7f8a940039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:52.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.783+0000 7f8aaaffd640 1 --2- 172.21.15.88:0/4271563406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a94003ff0 0x7f8a94004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:52.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.783+0000 7f8aaaffd640 1 --2- 172.21.15.88:0/4271563406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a94004d40 0x7f8a940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:52.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.783+0000 7f8aaaffd640 1 -- 172.21.15.88:0/4271563406 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8aac0700b0 con 0x7f8a94003ff0 2024-09-16T01:39:52.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.783+0000 7f8aaaffd640 1 -- 172.21.15.88:0/4271563406 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9400bb50 con 0x7f8a94004d40 2024-09-16T01:39:52.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.783+0000 7f8aaaffd640 1 -- 172.21.15.88:0/4271563406 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8a9400bd50 con 0x7f8aac0735b0 2024-09-16T01:39:52.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.783+0000 7f8aab7fe640 1 --2- 172.21.15.88:0/4271563406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a94003ff0 0x7f8a94004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:52.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.783+0000 7f8aabfff640 1 --2- 172.21.15.88:0/4271563406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8aac0735b0 0x7f8a940039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:52.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.783+0000 7f8ab0838640 1 --2- 172.21.15.88:0/4271563406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a94004d40 0x7f8a940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:52.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.783+0000 7f8aab7fe640 1 -- 172.21.15.88:0/4271563406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8aac0735b0 msgr2=0x7f8a940039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:52.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.783+0000 7f8aab7fe640 1 --2- 172.21.15.88:0/4271563406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8aac0735b0 0x7f8a940039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.783+0000 7f8aab7fe640 1 -- 172.21.15.88:0/4271563406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a94004d40 msgr2=0x7f8a940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:52.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.783+0000 7f8aab7fe640 1 --2- 172.21.15.88:0/4271563406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a94004d40 0x7f8a940051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.783+0000 7f8aab7fe640 1 -- 172.21.15.88:0/4271563406 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a94003520 con 0x7f8a94003ff0 2024-09-16T01:39:52.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.784+0000 7f8aab7fe640 1 --2- 172.21.15.88:0/4271563406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a94003ff0 0x7f8a94004800 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0007fd0 tx=0x7f8a94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a527e3ad272114d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:52.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.784+0000 7f8aaaffd640 1 -- 172.21.15.88:0/4271563406 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa0007d50 con 0x7f8a94003ff0 2024-09-16T01:39:52.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.784+0000 7f8aaaffd640 1 -- 172.21.15.88:0/4271563406 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8aa00058d0 con 0x7f8a94003ff0 2024-09-16T01:39:52.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.784+0000 7f8aaaffd640 1 -- 172.21.15.88:0/4271563406 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa0005ce0 con 0x7f8a94003ff0 2024-09-16T01:39:52.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.784+0000 7f8ab229b640 1 -- 172.21.15.88:0/4271563406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a94003ff0 msgr2=0x7f8a94004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:52.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.784+0000 7f8ab229b640 1 --2- 172.21.15.88:0/4271563406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a94003ff0 0x7f8a94004800 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0007fd0 tx=0x7f8a94005830 comp rx=0 tx=0).stop 2024-09-16T01:39:52.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.784+0000 7f8ab0838640 1 -- 172.21.15.88:0/4271563406 reap_dead start 2024-09-16T01:39:52.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.784+0000 7f8ab229b640 1 -- 172.21.15.88:0/4271563406 shutdown_connections 2024-09-16T01:39:52.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.784+0000 7f8ab229b640 1 -- 172.21.15.88:0/4271563406 >> 172.21.15.88:0/4271563406 conn(0x7f8aac0d14d0 msgr2=0x7f8aac0d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:52.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.784+0000 7f8ab229b640 1 -- 172.21.15.88:0/4271563406 shutdown_connections 2024-09-16T01:39:52.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.784+0000 7f8ab229b640 1 -- 172.21.15.88:0/4271563406 wait complete. 2024-09-16T01:39:52.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.785+0000 7f8ab229b640 1 Processor -- start 2024-09-16T01:39:52.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.786+0000 7f8ab229b640 1 -- start start 2024-09-16T01:39:52.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.786+0000 7f8ab229b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a94003ff0 0x7f8aac153cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:52.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.786+0000 7f8ab229b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8aac0ca900 0x7f8aac0cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:52.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.786+0000 7f8ab229b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8aac0cc330 0x7f8aac0cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:52.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.786+0000 7f8ab229b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8aac0721d0 con 0x7f8aac0cc330 2024-09-16T01:39:52.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.786+0000 7f8ab229b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8aac072050 con 0x7f8aac0ca900 2024-09-16T01:39:52.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.786+0000 7f8ab229b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8aac072350 con 0x7f8a94003ff0 2024-09-16T01:39:52.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.786+0000 7f8ab0838640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8aac0cc330 0x7f8aac0cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:52.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.786+0000 7f8ab0838640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8aac0cc330 0x7f8aac0cc7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60098/0 (socket says 172.21.15.88:60098) 2024-09-16T01:39:52.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.787+0000 7f8ab0838640 1 -- 172.21.15.88:0/1336381125 learned_addr learned my addr 172.21.15.88:0/1336381125 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:52.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.787+0000 7f8aabfff640 1 --2- 172.21.15.88:0/1336381125 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a94003ff0 0x7f8aac153cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:52.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.787+0000 7f8aab7fe640 1 --2- 172.21.15.88:0/1336381125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8aac0ca900 0x7f8aac0cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:52.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.787+0000 7f8ab0838640 1 -- 172.21.15.88:0/1336381125 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a94003ff0 msgr2=0x7f8aac153cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:52.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.787+0000 7f8ab0838640 1 --2- 172.21.15.88:0/1336381125 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a94003ff0 0x7f8aac153cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.787+0000 7f8ab0838640 1 -- 172.21.15.88:0/1336381125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8aac0ca900 msgr2=0x7f8aac0cad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:52.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.787+0000 7f8ab0838640 1 --2- 172.21.15.88:0/1336381125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8aac0ca900 0x7f8aac0cad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.787+0000 7f8ab0838640 1 -- 172.21.15.88:0/1336381125 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aac0cb980 con 0x7f8aac0cc330 2024-09-16T01:39:52.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.787+0000 7f8ab0838640 1 --2- 172.21.15.88:0/1336381125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8aac0cc330 0x7f8aac0cc7b0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f8aa40098a0 tx=0x7f8aa4009d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:52.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.787+0000 7f8aa8ff9640 1 -- 172.21.15.88:0/1336381125 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4019070 con 0x7f8aac0cc330 2024-09-16T01:39:52.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.787+0000 7f8aa8ff9640 1 -- 172.21.15.88:0/1336381125 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8aa4007ce0 con 0x7f8aac0cc330 2024-09-16T01:39:52.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.787+0000 7f8aa8ff9640 1 -- 172.21.15.88:0/1336381125 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4014640 con 0x7f8aac0cc330 2024-09-16T01:39:52.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.787+0000 7f8ab229b640 1 -- 172.21.15.88:0/1336381125 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8aac0cb3c0 con 0x7f8aac0cc330 2024-09-16T01:39:52.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.787+0000 7f8ab229b640 1 -- 172.21.15.88:0/1336381125 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8aac15edd0 con 0x7f8aac0cc330 2024-09-16T01:39:52.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.788+0000 7f8ab229b640 1 -- 172.21.15.88:0/1336381125 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8aac06ce30 con 0x7f8aac0cc330 2024-09-16T01:39:52.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.789+0000 7f8aa8ff9640 1 -- 172.21.15.88:0/1336381125 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8aa401c070 con 0x7f8aac0cc330 2024-09-16T01:39:52.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.789+0000 7f8aa8ff9640 1 --2- 172.21.15.88:0/1336381125 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8a9806c2b0 0x7f8a9806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:52.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.789+0000 7f8aa8ff9640 1 -- 172.21.15.88:0/1336381125 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5795+0+0 (secure 0 0 0) 0x7f8aa408af30 con 0x7f8aac0cc330 2024-09-16T01:39:52.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.789+0000 7f8aabfff640 1 --2- 172.21.15.88:0/1336381125 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8a9806c2b0 0x7f8a9806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:52.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.789+0000 7f8aabfff640 1 --2- 172.21.15.88:0/1336381125 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8a9806c2b0 0x7f8a9806e770 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c007fd0 tx=0x7f8a9c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:52.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.793+0000 7f8aa8ff9640 1 -- 172.21.15.88:0/1336381125 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8aa4058ff0 con 0x7f8aac0cc330 2024-09-16T01:39:52.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.994+0000 7f8ab229b640 1 -- 172.21.15.88:0/1336381125 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8aac15e100 con 0x7f8aac0cc330 2024-09-16T01:39:52.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.994+0000 7f8aa8ff9640 1 -- 172.21.15.88:0/1336381125 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+5233 (secure 0 0 0) 0x7f8aa405d2f0 con 0x7f8aac0cc330 2024-09-16T01:39:52.996 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:52.996 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":64,"btime":"2024-09-16T01:39:49:271443+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5725,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2696547723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2696547723},{"type":"v1","addr":"172.21.15.154:6834","nonce":2696547723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54},{"gid":5863,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2167482844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2167482844},{"type":"v1","addr":"172.21.15.154:6835","nonce":2167482844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62},{"gid":6411,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3276199399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3276199399},{"type":"v1","addr":"172.21.15.88:6836","nonce":3276199399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:39:14.808422+0000","modified":"2024-09-16T01:39:49.271439+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6414},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":12,"addr":"172.21.15.88:6837/2944989249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2944989249},{"type":"v1","addr":"172.21.15.88:6837","nonce":2944989249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6414,"qdb_cluster":[6414]},"id":5}]} 2024-09-16T01:39:52.996 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 64 2024-09-16T01:39:52.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.996+0000 7f8ab229b640 1 -- 172.21.15.88:0/1336381125 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8a9806c2b0 msgr2=0x7f8a9806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:52.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.996+0000 7f8ab229b640 1 --2- 172.21.15.88:0/1336381125 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8a9806c2b0 0x7f8a9806e770 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c007fd0 tx=0x7f8a9c009040 comp rx=0 tx=0).stop 2024-09-16T01:39:52.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.996+0000 7f8ab229b640 1 -- 172.21.15.88:0/1336381125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8aac0cc330 msgr2=0x7f8aac0cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:52.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.996+0000 7f8ab229b640 1 --2- 172.21.15.88:0/1336381125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8aac0cc330 0x7f8aac0cc7b0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f8aa40098a0 tx=0x7f8aa4009d60 comp rx=0 tx=0).stop 2024-09-16T01:39:52.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.996+0000 7f8ab229b640 1 -- 172.21.15.88:0/1336381125 shutdown_connections 2024-09-16T01:39:52.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.996+0000 7f8ab229b640 1 --2- 172.21.15.88:0/1336381125 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8a9806c2b0 0x7f8a9806e770 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.996+0000 7f8ab229b640 1 --2- 172.21.15.88:0/1336381125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8aac0cc330 0x7f8aac0cc7b0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.996+0000 7f8ab229b640 1 --2- 172.21.15.88:0/1336381125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8aac0ca900 0x7f8aac0cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.996+0000 7f8ab229b640 1 --2- 172.21.15.88:0/1336381125 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a94003ff0 0x7f8aac153cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:52.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.996+0000 7f8ab229b640 1 -- 172.21.15.88:0/1336381125 >> 172.21.15.88:0/1336381125 conn(0x7f8aac0d14d0 msgr2=0x7f8aac0b6220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:52.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.996+0000 7f8ab229b640 1 -- 172.21.15.88:0/1336381125 shutdown_connections 2024-09-16T01:39:52.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:52.996+0000 7f8ab229b640 1 -- 172.21.15.88:0/1336381125 wait complete. 2024-09-16T01:39:53.008 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:39:53.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.105+0000 7f861a3b7640 1 Processor -- start 2024-09-16T01:39:53.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.105+0000 7f861a3b7640 1 -- start start 2024-09-16T01:39:53.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.106+0000 7f861a3b7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f86140cd780 0x7f86140cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:53.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.106+0000 7f861a3b7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86140ce4f0 con 0x7f86140b4d00 2024-09-16T01:39:53.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.106+0000 7f861a3b7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86140ce7d0 con 0x7f86140b4920 2024-09-16T01:39:53.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.106+0000 7f861a3b7640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f86140cf100 con 0x7f86140cd780 2024-09-16T01:39:53.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.106+0000 7f86137fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f86140b4d00 0x7f86140b5170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55954/0 (socket says 172.21.15.88:55954) 2024-09-16T01:39:53.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.106+0000 7f86137fe640 1 -- 172.21.15.88:0/2235939298 learned_addr learned my addr 172.21.15.88:0/2235939298 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:53.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.106+0000 7f8618954640 1 --2- 172.21.15.88:0/2235939298 >> v2:172.21.15.154:3300/0 conn(0x7f86140cd780 0x7f86140cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:53.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.106+0000 7f8612ffd640 1 -- 172.21.15.88:0/2235939298 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 561262784 0 0) 0x7f86140ce4f0 con 0x7f86140b4d00 2024-09-16T01:39:53.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.106+0000 7f8612ffd640 1 -- 172.21.15.88:0/2235939298 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85fc003520 con 0x7f86140b4d00 2024-09-16T01:39:53.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.106+0000 7f8612ffd640 1 -- 172.21.15.88:0/2235939298 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2764663296 0 0) 0x7f85fc003520 con 0x7f86140b4d00 2024-09-16T01:39:53.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.106+0000 7f8612ffd640 1 -- 172.21.15.88:0/2235939298 >> v2:172.21.15.154:3300/0 conn(0x7f86140cd780 msgr2=0x7f86140cdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:53.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.106+0000 7f8612ffd640 1 --2- 172.21.15.88:0/2235939298 >> v2:172.21.15.154:3300/0 conn(0x7f86140cd780 0x7f86140cdb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.106+0000 7f8612ffd640 1 -- 172.21.15.88:0/2235939298 >> v1:172.21.15.154:6789/0 conn(0x7f86140b4920 legacy=0x7f861415b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:53.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.106+0000 7f8612ffd640 1 -- 172.21.15.88:0/2235939298 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86140ceab0 con 0x7f86140b4d00 2024-09-16T01:39:53.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.106+0000 7f8612ffd640 1 -- 172.21.15.88:0/2235939298 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f8608001670 con 0x7f86140b4d00 2024-09-16T01:39:53.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.106+0000 7f8612ffd640 1 -- 172.21.15.88:0/2235939298 >> v1:172.21.15.88:6789/0 conn(0x7f86140b4d00 legacy=0x7f86140b5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:53.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.106+0000 7f8612ffd640 1 --2- 172.21.15.88:0/2235939298 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85fc0039b0 0x7f85fc003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:53.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.106+0000 7f8612ffd640 1 --2- 172.21.15.88:0/2235939298 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85fc004800 0x7f85fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:53.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.107+0000 7f8612ffd640 1 --2- 172.21.15.88:0/2235939298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85fc0051c0 0x7f85fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:53.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.107+0000 7f8612ffd640 1 -- 172.21.15.88:0/2235939298 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f86140cf100 con 0x7f85fc0039b0 2024-09-16T01:39:53.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.107+0000 7f8612ffd640 1 -- 172.21.15.88:0/2235939298 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f85fc00c090 con 0x7f85fc0051c0 2024-09-16T01:39:53.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.107+0000 7f8612ffd640 1 -- 172.21.15.88:0/2235939298 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f85fc00c290 con 0x7f85fc004800 2024-09-16T01:39:53.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.107+0000 7f8613fff640 1 --2- 172.21.15.88:0/2235939298 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85fc0039b0 0x7f85fc003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:53.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.107+0000 7f86137fe640 1 --2- 172.21.15.88:0/2235939298 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85fc004800 0x7f85fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:53.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.107+0000 7f8618954640 1 --2- 172.21.15.88:0/2235939298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85fc0051c0 0x7f85fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:53.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.107+0000 7f8613fff640 1 -- 172.21.15.88:0/2235939298 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85fc004800 msgr2=0x7f85fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:53.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.107+0000 7f8613fff640 1 --2- 172.21.15.88:0/2235939298 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85fc004800 0x7f85fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.107+0000 7f8613fff640 1 -- 172.21.15.88:0/2235939298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85fc0051c0 msgr2=0x7f85fc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:53.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.107+0000 7f8613fff640 1 --2- 172.21.15.88:0/2235939298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85fc0051c0 0x7f85fc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.107+0000 7f8618954640 1 -- 172.21.15.88:0/2235939298 reap_dead start 2024-09-16T01:39:53.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.107+0000 7f8613fff640 1 -- 172.21.15.88:0/2235939298 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85fc003520 con 0x7f85fc0039b0 2024-09-16T01:39:53.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.107+0000 7f86137fe640 1 --2- 172.21.15.88:0/2235939298 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85fc004800 0x7f85fc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:53.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.108+0000 7f8613fff640 1 --2- 172.21.15.88:0/2235939298 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85fc0039b0 0x7f85fc003df0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f8604001750 tx=0x7f85fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdc8212444d31881 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:53.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.108+0000 7f8612ffd640 1 -- 172.21.15.88:0/2235939298 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8604001370 con 0x7f85fc0039b0 2024-09-16T01:39:53.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.108+0000 7f8612ffd640 1 -- 172.21.15.88:0/2235939298 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86040052d0 con 0x7f85fc0039b0 2024-09-16T01:39:53.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.108+0000 7f8612ffd640 1 -- 172.21.15.88:0/2235939298 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860402abc0 con 0x7f85fc0039b0 2024-09-16T01:39:53.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.108+0000 7f861a3b7640 1 -- 172.21.15.88:0/2235939298 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85fc0039b0 msgr2=0x7f85fc003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:53.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.108+0000 7f861a3b7640 1 --2- 172.21.15.88:0/2235939298 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85fc0039b0 0x7f85fc003df0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f8604001750 tx=0x7f85fc005830 comp rx=0 tx=0).stop 2024-09-16T01:39:53.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.108+0000 7f861a3b7640 1 -- 172.21.15.88:0/2235939298 shutdown_connections 2024-09-16T01:39:53.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.108+0000 7f861a3b7640 1 --2- 172.21.15.88:0/2235939298 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85fc0039b0 0x7f85fc003df0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.108+0000 7f861a3b7640 1 -- 172.21.15.88:0/2235939298 >> 172.21.15.88:0/2235939298 conn(0x7f861406bc10 msgr2=0x7f861406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:53.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.108+0000 7f861a3b7640 1 -- 172.21.15.88:0/2235939298 shutdown_connections 2024-09-16T01:39:53.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.108+0000 7f861a3b7640 1 -- 172.21.15.88:0/2235939298 wait complete. 2024-09-16T01:39:53.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.109+0000 7f861a3b7640 1 Processor -- start 2024-09-16T01:39:53.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.110+0000 7f861a3b7640 1 -- start start 2024-09-16T01:39:53.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.110+0000 7f861a3b7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85fc0039b0 0x7f8614145d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:53.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.110+0000 7f861a3b7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86141596b0 0x7f8614150b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:53.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.110+0000 7f861a3b7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8614151060 0x7f8614152e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:53.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.110+0000 7f861a3b7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f86140b52f0 con 0x7f85fc0039b0 2024-09-16T01:39:53.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.110+0000 7f861a3b7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f86140b5170 con 0x7f8614151060 2024-09-16T01:39:53.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.110+0000 7f861a3b7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f86140b5470 con 0x7f86141596b0 2024-09-16T01:39:53.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.110+0000 7f8613fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85fc0039b0 0x7f8614145d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:53.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.110+0000 7f8613fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85fc0039b0 0x7f8614145d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60120/0 (socket says 172.21.15.88:60120) 2024-09-16T01:39:53.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.110+0000 7f8613fff640 1 -- 172.21.15.88:0/825898937 learned_addr learned my addr 172.21.15.88:0/825898937 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:53.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.111+0000 7f8618954640 1 --2- 172.21.15.88:0/825898937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8614151060 0x7f8614152e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:53.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.111+0000 7f86137fe640 1 --2- 172.21.15.88:0/825898937 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86141596b0 0x7f8614150b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:53.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.111+0000 7f8613fff640 1 -- 172.21.15.88:0/825898937 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86141596b0 msgr2=0x7f8614150b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:53.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.111+0000 7f8613fff640 1 --2- 172.21.15.88:0/825898937 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86141596b0 0x7f8614150b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.111+0000 7f8613fff640 1 -- 172.21.15.88:0/825898937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8614151060 msgr2=0x7f8614152e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:53.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.111+0000 7f8613fff640 1 --2- 172.21.15.88:0/825898937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8614151060 0x7f8614152e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.111+0000 7f8613fff640 1 -- 172.21.15.88:0/825898937 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8614151500 con 0x7f85fc0039b0 2024-09-16T01:39:53.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.111+0000 7f8613fff640 1 --2- 172.21.15.88:0/825898937 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85fc0039b0 0x7f8614145d70 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f8604001870 tx=0x7f8604001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:53.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.111+0000 7f8610ff9640 1 -- 172.21.15.88:0/825898937 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860403f030 con 0x7f85fc0039b0 2024-09-16T01:39:53.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.111+0000 7f8610ff9640 1 -- 172.21.15.88:0/825898937 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8604005480 con 0x7f85fc0039b0 2024-09-16T01:39:53.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.111+0000 7f8610ff9640 1 -- 172.21.15.88:0/825898937 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8604037040 con 0x7f85fc0039b0 2024-09-16T01:39:53.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.111+0000 7f861a3b7640 1 -- 172.21.15.88:0/825898937 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86141517e0 con 0x7f85fc0039b0 2024-09-16T01:39:53.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.111+0000 7f861a3b7640 1 -- 172.21.15.88:0/825898937 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8614151da0 con 0x7f85fc0039b0 2024-09-16T01:39:53.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.112+0000 7f861a3b7640 1 -- 172.21.15.88:0/825898937 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f861415b7d0 con 0x7f85fc0039b0 2024-09-16T01:39:53.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.113+0000 7f8610ff9640 1 -- 172.21.15.88:0/825898937 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8604042070 con 0x7f85fc0039b0 2024-09-16T01:39:53.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.113+0000 7f8610ff9640 1 --2- 172.21.15.88:0/825898937 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f860006c2b0 0x7f860006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:53.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.113+0000 7f86137fe640 1 --2- 172.21.15.88:0/825898937 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f860006c2b0 0x7f860006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:53.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.113+0000 7f8610ff9640 1 -- 172.21.15.88:0/825898937 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5795+0+0 (secure 0 0 0) 0x7f86040aaef0 con 0x7f85fc0039b0 2024-09-16T01:39:53.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.114+0000 7f86137fe640 1 --2- 172.21.15.88:0/825898937 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f860006c2b0 0x7f860006e770 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8608001eb0 tx=0x7f8608009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:53.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.117+0000 7f8610ff9640 1 -- 172.21.15.88:0/825898937 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8604079600 con 0x7f85fc0039b0 2024-09-16T01:39:53.236 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-09-16T01:39:53.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.317+0000 7f861a3b7640 1 -- 172.21.15.88:0/825898937 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8614071680 con 0x7f85fc0039b0 2024-09-16T01:39:53.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.318+0000 7f8610ff9640 1 -- 172.21.15.88:0/825898937 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+5233 (secure 0 0 0) 0x7f8604078f00 con 0x7f85fc0039b0 2024-09-16T01:39:53.320 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:53.320 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":64,"btime":"2024-09-16T01:39:49:271443+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5725,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2696547723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2696547723},{"type":"v1","addr":"172.21.15.154:6834","nonce":2696547723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54},{"gid":5863,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2167482844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2167482844},{"type":"v1","addr":"172.21.15.154:6835","nonce":2167482844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62},{"gid":6411,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3276199399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3276199399},{"type":"v1","addr":"172.21.15.88:6836","nonce":3276199399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:39:14.808422+0000","modified":"2024-09-16T01:39:49.271439+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6414},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6414":{"gid":6414,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":12,"addr":"172.21.15.88:6837/2944989249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2944989249},{"type":"v1","addr":"172.21.15.88:6837","nonce":2944989249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6414,"qdb_cluster":[6414]},"id":5}]} 2024-09-16T01:39:53.320 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 64 2024-09-16T01:39:53.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.320+0000 7f861a3b7640 1 -- 172.21.15.88:0/825898937 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f860006c2b0 msgr2=0x7f860006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:53.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.320+0000 7f861a3b7640 1 --2- 172.21.15.88:0/825898937 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f860006c2b0 0x7f860006e770 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8608001eb0 tx=0x7f8608009040 comp rx=0 tx=0).stop 2024-09-16T01:39:53.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.320+0000 7f861a3b7640 1 -- 172.21.15.88:0/825898937 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85fc0039b0 msgr2=0x7f8614145d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:53.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.320+0000 7f861a3b7640 1 --2- 172.21.15.88:0/825898937 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85fc0039b0 0x7f8614145d70 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f8604001870 tx=0x7f8604001120 comp rx=0 tx=0).stop 2024-09-16T01:39:53.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.320+0000 7f861a3b7640 1 -- 172.21.15.88:0/825898937 shutdown_connections 2024-09-16T01:39:53.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.320+0000 7f861a3b7640 1 --2- 172.21.15.88:0/825898937 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f860006c2b0 0x7f860006e770 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.320+0000 7f861a3b7640 1 --2- 172.21.15.88:0/825898937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8614151060 0x7f8614152e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.320+0000 7f861a3b7640 1 --2- 172.21.15.88:0/825898937 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86141596b0 0x7f8614150b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.320+0000 7f861a3b7640 1 --2- 172.21.15.88:0/825898937 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85fc0039b0 0x7f8614145d70 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.320+0000 7f861a3b7640 1 -- 172.21.15.88:0/825898937 >> 172.21.15.88:0/825898937 conn(0x7f861406bc10 msgr2=0x7f8614071e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:53.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.320+0000 7f861a3b7640 1 -- 172.21.15.88:0/825898937 shutdown_connections 2024-09-16T01:39:53.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.320+0000 7f861a3b7640 1 -- 172.21.15.88:0/825898937 wait complete. 2024-09-16T01:39:53.332 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:39:53.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.428+0000 7fbbcaca6640 1 Processor -- start 2024-09-16T01:39:53.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.428+0000 7fbbcaca6640 1 -- start start 2024-09-16T01:39:53.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.429+0000 7fbbcaca6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbbc40b6140 0x7fbbc415c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:53.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.429+0000 7fbbcaca6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbc40b50a0 con 0x7fbbc40cfc80 2024-09-16T01:39:53.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.429+0000 7fbbcaca6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbc40b5380 con 0x7fbbc4155800 2024-09-16T01:39:53.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.429+0000 7fbbcaca6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbbc40b6550 con 0x7fbbc40b6140 2024-09-16T01:39:53.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.429+0000 7fbbbbfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fbbc40cfc80 0x7fbbc415c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55958/0 (socket says 172.21.15.88:55958) 2024-09-16T01:39:53.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.429+0000 7fbbbbfff640 1 -- 172.21.15.88:0/3296315082 learned_addr learned my addr 172.21.15.88:0/3296315082 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:53.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.429+0000 7fbbc9243640 1 --2- 172.21.15.88:0/3296315082 >> v2:172.21.15.154:3300/0 conn(0x7fbbc40b6140 0x7fbbc415c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:53.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.429+0000 7fbbbb7fe640 1 -- 172.21.15.88:0/3296315082 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3147137905 0 0) 0x7fbbc40b50a0 con 0x7fbbc40cfc80 2024-09-16T01:39:53.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.429+0000 7fbbbb7fe640 1 -- 172.21.15.88:0/3296315082 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbac003520 con 0x7fbbc40cfc80 2024-09-16T01:39:53.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.429+0000 7fbbbb7fe640 1 -- 172.21.15.88:0/3296315082 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1218706179 0 0) 0x7fbbac003520 con 0x7fbbc40cfc80 2024-09-16T01:39:53.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.429+0000 7fbbbb7fe640 1 -- 172.21.15.88:0/3296315082 >> v2:172.21.15.154:3300/0 conn(0x7fbbc40b6140 msgr2=0x7fbbc415c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:53.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.429+0000 7fbbbb7fe640 1 --2- 172.21.15.88:0/3296315082 >> v2:172.21.15.154:3300/0 conn(0x7fbbc40b6140 0x7fbbc415c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.429+0000 7fbbbb7fe640 1 -- 172.21.15.88:0/3296315082 >> v1:172.21.15.154:6789/0 conn(0x7fbbc4155800 legacy=0x7fbbc40cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:53.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.430+0000 7fbbbb7fe640 1 -- 172.21.15.88:0/3296315082 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbc40b5660 con 0x7fbbc40cfc80 2024-09-16T01:39:53.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.430+0000 7fbbbb7fe640 1 -- 172.21.15.88:0/3296315082 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fbbb4001670 con 0x7fbbc40cfc80 2024-09-16T01:39:53.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.430+0000 7fbbbb7fe640 1 -- 172.21.15.88:0/3296315082 >> v1:172.21.15.88:6789/0 conn(0x7fbbc40cfc80 legacy=0x7fbbc415c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:53.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.430+0000 7fbbbb7fe640 1 --2- 172.21.15.88:0/3296315082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbbac0039b0 0x7fbbac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:53.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.430+0000 7fbbbb7fe640 1 --2- 172.21.15.88:0/3296315082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbac004800 0x7fbbac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:53.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.430+0000 7fbbbb7fe640 1 --2- 172.21.15.88:0/3296315082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbac0051c0 0x7fbbac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:53.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.430+0000 7fbbbb7fe640 1 -- 172.21.15.88:0/3296315082 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc40b6550 con 0x7fbbac004800 2024-09-16T01:39:53.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.430+0000 7fbbbb7fe640 1 -- 172.21.15.88:0/3296315082 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbbac00c060 con 0x7fbbac0051c0 2024-09-16T01:39:53.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.430+0000 7fbbbb7fe640 1 -- 172.21.15.88:0/3296315082 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbbac00c260 con 0x7fbbac0039b0 2024-09-16T01:39:53.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.430+0000 7fbbbbfff640 1 --2- 172.21.15.88:0/3296315082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbac004800 0x7fbbac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:53.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.430+0000 7fbbc8a42640 1 --2- 172.21.15.88:0/3296315082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbbac0039b0 0x7fbbac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:53.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.430+0000 7fbbc9243640 1 --2- 172.21.15.88:0/3296315082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbac0051c0 0x7fbbac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:53.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.430+0000 7fbbbbfff640 1 -- 172.21.15.88:0/3296315082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbbac0039b0 msgr2=0x7fbbac003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:53.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.430+0000 7fbbbbfff640 1 --2- 172.21.15.88:0/3296315082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbbac0039b0 0x7fbbac003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.430+0000 7fbbbbfff640 1 -- 172.21.15.88:0/3296315082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbac0051c0 msgr2=0x7fbbac00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:53.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.430+0000 7fbbbbfff640 1 --2- 172.21.15.88:0/3296315082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbac0051c0 0x7fbbac00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.430+0000 7fbbbbfff640 1 -- 172.21.15.88:0/3296315082 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbac003520 con 0x7fbbac004800 2024-09-16T01:39:53.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.430+0000 7fbbc9243640 1 -- 172.21.15.88:0/3296315082 reap_dead start 2024-09-16T01:39:53.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.431+0000 7fbbbbfff640 1 --2- 172.21.15.88:0/3296315082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbac004800 0x7fbbac004c80 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fbbb400c210 tx=0x7fbbac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d5bba810cf5a4b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:53.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.431+0000 7fbbbb7fe640 1 -- 172.21.15.88:0/3296315082 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4007cf0 con 0x7fbbac004800 2024-09-16T01:39:53.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.431+0000 7fbbbb7fe640 1 -- 172.21.15.88:0/3296315082 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbb4005ad0 con 0x7fbbac004800 2024-09-16T01:39:53.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.431+0000 7fbbbb7fe640 1 -- 172.21.15.88:0/3296315082 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4007070 con 0x7fbbac004800 2024-09-16T01:39:53.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.431+0000 7fbbcaca6640 1 -- 172.21.15.88:0/3296315082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbac004800 msgr2=0x7fbbac004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:53.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.431+0000 7fbbcaca6640 1 --2- 172.21.15.88:0/3296315082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbac004800 0x7fbbac004c80 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fbbb400c210 tx=0x7fbbac005830 comp rx=0 tx=0).stop 2024-09-16T01:39:53.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.431+0000 7fbbcaca6640 1 -- 172.21.15.88:0/3296315082 shutdown_connections 2024-09-16T01:39:53.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.431+0000 7fbbcaca6640 1 --2- 172.21.15.88:0/3296315082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbac004800 0x7fbbac004c80 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.431+0000 7fbbcaca6640 1 -- 172.21.15.88:0/3296315082 >> 172.21.15.88:0/3296315082 conn(0x7fbbc406bc10 msgr2=0x7fbbc406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:53.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.431+0000 7fbbcaca6640 1 -- 172.21.15.88:0/3296315082 shutdown_connections 2024-09-16T01:39:53.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.431+0000 7fbbcaca6640 1 -- 172.21.15.88:0/3296315082 wait complete. 2024-09-16T01:39:53.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.433+0000 7fbbcaca6640 1 Processor -- start 2024-09-16T01:39:53.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.433+0000 7fbbcaca6640 1 -- start start 2024-09-16T01:39:53.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.433+0000 7fbbcaca6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbac004800 0x7fbbc413db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:53.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.433+0000 7fbbcaca6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbc41f1ce0 0x7fbbc41f2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:53.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.433+0000 7fbbcaca6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbbc41f3b90 0x7fbbc41f4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:53.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.433+0000 7fbbcaca6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc4155be0 con 0x7fbbac004800 2024-09-16T01:39:53.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.433+0000 7fbbcaca6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc40b5c20 con 0x7fbbc41f1ce0 2024-09-16T01:39:53.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.433+0000 7fbbcaca6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbbc415c330 con 0x7fbbc41f3b90 2024-09-16T01:39:53.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.434+0000 7fbbc8a42640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbac004800 0x7fbbc413db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:53.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.434+0000 7fbbc8a42640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbac004800 0x7fbbc413db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60132/0 (socket says 172.21.15.88:60132) 2024-09-16T01:39:53.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.434+0000 7fbbc8a42640 1 -- 172.21.15.88:0/74956105 learned_addr learned my addr 172.21.15.88:0/74956105 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:53.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.434+0000 7fbbbbfff640 1 --2- 172.21.15.88:0/74956105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbc41f1ce0 0x7fbbc41f2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:53.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.434+0000 7fbbc9243640 1 --2- 172.21.15.88:0/74956105 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbbc41f3b90 0x7fbbc41f4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:53.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.434+0000 7fbbc8a42640 1 -- 172.21.15.88:0/74956105 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbbc41f3b90 msgr2=0x7fbbc41f4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:53.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.434+0000 7fbbc8a42640 1 --2- 172.21.15.88:0/74956105 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbbc41f3b90 0x7fbbc41f4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.434+0000 7fbbc8a42640 1 -- 172.21.15.88:0/74956105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbc41f1ce0 msgr2=0x7fbbc41f2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:53.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.434+0000 7fbbc8a42640 1 --2- 172.21.15.88:0/74956105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbc41f1ce0 0x7fbbc41f2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.434+0000 7fbbc8a42640 1 -- 172.21.15.88:0/74956105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbc41f2af0 con 0x7fbbac004800 2024-09-16T01:39:53.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.434+0000 7fbbc9243640 1 --2- 172.21.15.88:0/74956105 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbbc41f3b90 0x7fbbc41f4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:53.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.434+0000 7fbbbbfff640 1 --2- 172.21.15.88:0/74956105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbc41f1ce0 0x7fbbc41f2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:53.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.434+0000 7fbbc8a42640 1 --2- 172.21.15.88:0/74956105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbac004800 0x7fbbc413db60 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fbbc000e480 tx=0x7fbbc000e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:53.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.434+0000 7fbbb97fa640 1 -- 172.21.15.88:0/74956105 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc00052e0 con 0x7fbbac004800 2024-09-16T01:39:53.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.434+0000 7fbbb97fa640 1 -- 172.21.15.88:0/74956105 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbc0018070 con 0x7fbbac004800 2024-09-16T01:39:53.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.434+0000 7fbbcaca6640 1 -- 172.21.15.88:0/74956105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbc41f2dd0 con 0x7fbbac004800 2024-09-16T01:39:53.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.434+0000 7fbbb97fa640 1 -- 172.21.15.88:0/74956105 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc0003830 con 0x7fbbac004800 2024-09-16T01:39:53.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.434+0000 7fbbcaca6640 1 -- 172.21.15.88:0/74956105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbc41f3390 con 0x7fbbac004800 2024-09-16T01:39:53.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.435+0000 7fbbcaca6640 1 -- 172.21.15.88:0/74956105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbc415c6c0 con 0x7fbbac004800 2024-09-16T01:39:53.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.436+0000 7fbbb97fa640 1 -- 172.21.15.88:0/74956105 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fbbc00039d0 con 0x7fbbac004800 2024-09-16T01:39:53.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.436+0000 7fbbb97fa640 1 --2- 172.21.15.88:0/74956105 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbbb006c2e0 0x7fbbb006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:53.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.436+0000 7fbbb97fa640 1 -- 172.21.15.88:0/74956105 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5795+0+0 (secure 0 0 0) 0x7fbbc008b0e0 con 0x7fbbac004800 2024-09-16T01:39:53.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.436+0000 7fbbbbfff640 1 --2- 172.21.15.88:0/74956105 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbbb006c2e0 0x7fbbb006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:53.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.437+0000 7fbbbbfff640 1 --2- 172.21.15.88:0/74956105 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbbb006c2e0 0x7fbbb006e7a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4070e90 tx=0x7fbbb4009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:53.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.441+0000 7fbbb97fa640 1 -- 172.21.15.88:0/74956105 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fbbc0059880 con 0x7fbbac004800 2024-09-16T01:39:53.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.609+0000 7fbbcaca6640 1 -- 172.21.15.88:0/74956105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbbc40b4de0 con 0x7fbbac004800 2024-09-16T01:39:53.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.610+0000 7fbbb97fa640 1 -- 172.21.15.88:0/74956105 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+16835 (secure 0 0 0) 0x7fbbc0059180 con 0x7fbbac004800 2024-09-16T01:39:53.611 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:53.612 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":73,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:39:46.253124+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:39:13.412390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-16T01:39:13.777039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.154:6835/3482042494":"2024-09-17T01:39:46.253065+0000","172.21.15.154:6833/3482042494":"2024-09-17T01:39:46.253065+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:39:53.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.612+0000 7fbbcaca6640 1 -- 172.21.15.88:0/74956105 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbbb006c2e0 msgr2=0x7fbbb006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:53.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.612+0000 7fbbcaca6640 1 --2- 172.21.15.88:0/74956105 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbbb006c2e0 0x7fbbb006e7a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4070e90 tx=0x7fbbb4009030 comp rx=0 tx=0).stop 2024-09-16T01:39:53.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.612+0000 7fbbcaca6640 1 -- 172.21.15.88:0/74956105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbac004800 msgr2=0x7fbbc413db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:53.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.612+0000 7fbbcaca6640 1 --2- 172.21.15.88:0/74956105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbac004800 0x7fbbc413db60 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fbbc000e480 tx=0x7fbbc000e940 comp rx=0 tx=0).stop 2024-09-16T01:39:53.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.612+0000 7fbbcaca6640 1 -- 172.21.15.88:0/74956105 shutdown_connections 2024-09-16T01:39:53.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.612+0000 7fbbcaca6640 1 --2- 172.21.15.88:0/74956105 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbbb006c2e0 0x7fbbb006e7a0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.612+0000 7fbbcaca6640 1 --2- 172.21.15.88:0/74956105 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbbc41f3b90 0x7fbbc41f4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.612+0000 7fbbcaca6640 1 --2- 172.21.15.88:0/74956105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbc41f1ce0 0x7fbbc41f2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.612+0000 7fbbcaca6640 1 --2- 172.21.15.88:0/74956105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbbac004800 0x7fbbc413db60 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.612+0000 7fbbcaca6640 1 -- 172.21.15.88:0/74956105 >> 172.21.15.88:0/74956105 conn(0x7fbbc406bc10 msgr2=0x7fbbc40ce6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:53.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.612+0000 7fbbcaca6640 1 -- 172.21.15.88:0/74956105 shutdown_connections 2024-09-16T01:39:53.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.612+0000 7fbbcaca6640 1 -- 172.21.15.88:0/74956105 wait complete. 2024-09-16T01:39:53.625 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:39:53.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.722+0000 7f606c7cf640 1 Processor -- start 2024-09-16T01:39:53.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.722+0000 7f606c7cf640 1 -- start start 2024-09-16T01:39:53.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.723+0000 7f606c7cf640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60640b6290 0x7f6064071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:53.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.723+0000 7f606c7cf640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60640ccff0 con 0x7f6064072530 2024-09-16T01:39:53.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.723+0000 7f606c7cf640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60640cd2d0 con 0x7f606415b480 2024-09-16T01:39:53.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.723+0000 7f606c7cf640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f60640cd5b0 con 0x7f60640b6290 2024-09-16T01:39:53.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.724+0000 7f606ad6c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6064072530 0x7f606415bc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55960/0 (socket says 172.21.15.88:55960) 2024-09-16T01:39:53.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.724+0000 7f606ad6c640 1 -- 172.21.15.88:0/4012835105 learned_addr learned my addr 172.21.15.88:0/4012835105 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:53.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.724+0000 7f6069d6a640 1 --2- 172.21.15.88:0/4012835105 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60640b6290 0x7f6064071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:53.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.724+0000 7f6069569640 1 -- 172.21.15.88:0/4012835105 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 727477237 0 0) 0x7f60640ccff0 con 0x7f6064072530 2024-09-16T01:39:53.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.724+0000 7f6069569640 1 -- 172.21.15.88:0/4012835105 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6050003520 con 0x7f6064072530 2024-09-16T01:39:53.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.724+0000 7f6069569640 1 -- 172.21.15.88:0/4012835105 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2428306309 0 0) 0x7f6050003520 con 0x7f6064072530 2024-09-16T01:39:53.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.724+0000 7f6069569640 1 -- 172.21.15.88:0/4012835105 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60640b6290 msgr2=0x7f6064071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:53.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.724+0000 7f6069569640 1 --2- 172.21.15.88:0/4012835105 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60640b6290 0x7f6064071ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.724+0000 7f6069569640 1 -- 172.21.15.88:0/4012835105 >> v1:172.21.15.154:6789/0 conn(0x7f606415b480 legacy=0x7f60640b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:53.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.724+0000 7f6069569640 1 -- 172.21.15.88:0/4012835105 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60640cd890 con 0x7f6064072530 2024-09-16T01:39:53.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.724+0000 7f6069569640 1 -- 172.21.15.88:0/4012835105 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f605c0019f0 con 0x7f6064072530 2024-09-16T01:39:53.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.724+0000 7f6069569640 1 -- 172.21.15.88:0/4012835105 >> v1:172.21.15.88:6789/0 conn(0x7f6064072530 legacy=0x7f606415bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:53.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.724+0000 7f6069569640 1 --2- 172.21.15.88:0/4012835105 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60500039e0 0x7f6050003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:53.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.724+0000 7f6069569640 1 --2- 172.21.15.88:0/4012835105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6050004300 0x7f6050004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:53.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.724+0000 7f6069569640 1 --2- 172.21.15.88:0/4012835105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6050004f90 0x7f605000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:53.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.724+0000 7f6069569640 1 -- 172.21.15.88:0/4012835105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f60640cd5b0 con 0x7f6050004f90 2024-09-16T01:39:53.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.724+0000 7f6069569640 1 -- 172.21.15.88:0/4012835105 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f605000bf00 con 0x7f6050004300 2024-09-16T01:39:53.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.724+0000 7f6069569640 1 -- 172.21.15.88:0/4012835105 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f605000c0b0 con 0x7f60500039e0 2024-09-16T01:39:53.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.724+0000 7f606ad6c640 1 --2- 172.21.15.88:0/4012835105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6050004f90 0x7f605000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:53.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.724+0000 7f606a56b640 1 --2- 172.21.15.88:0/4012835105 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60500039e0 0x7f6050003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:53.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.724+0000 7f6069d6a640 1 --2- 172.21.15.88:0/4012835105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6050004300 0x7f6050004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:53.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.725+0000 7f606ad6c640 1 -- 172.21.15.88:0/4012835105 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60500039e0 msgr2=0x7f6050003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:53.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.725+0000 7f606ad6c640 1 --2- 172.21.15.88:0/4012835105 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60500039e0 0x7f6050003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.725+0000 7f606ad6c640 1 -- 172.21.15.88:0/4012835105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6050004300 msgr2=0x7f6050004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:53.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.725+0000 7f606ad6c640 1 --2- 172.21.15.88:0/4012835105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6050004300 0x7f6050004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.725+0000 7f606ad6c640 1 -- 172.21.15.88:0/4012835105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6050003520 con 0x7f6050004f90 2024-09-16T01:39:53.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.725+0000 7f606ad6c640 1 --2- 172.21.15.88:0/4012835105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6050004f90 0x7f605000b9c0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f605c00a880 tx=0x7f6050005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=677e28ea796985b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:53.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.725+0000 7f6069569640 1 -- 172.21.15.88:0/4012835105 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605c00a4a0 con 0x7f6050004f90 2024-09-16T01:39:53.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.725+0000 7f6069569640 1 -- 172.21.15.88:0/4012835105 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f605c02f070 con 0x7f6050004f90 2024-09-16T01:39:53.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.725+0000 7f6069569640 1 -- 172.21.15.88:0/4012835105 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605c02bd10 con 0x7f6050004f90 2024-09-16T01:39:53.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.726+0000 7f606c7cf640 1 -- 172.21.15.88:0/4012835105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6050004f90 msgr2=0x7f605000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:53.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.726+0000 7f606c7cf640 1 --2- 172.21.15.88:0/4012835105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6050004f90 0x7f605000b9c0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f605c00a880 tx=0x7f6050005830 comp rx=0 tx=0).stop 2024-09-16T01:39:53.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.726+0000 7f606ad6c640 1 -- 172.21.15.88:0/4012835105 reap_dead start 2024-09-16T01:39:53.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.726+0000 7f606c7cf640 1 -- 172.21.15.88:0/4012835105 shutdown_connections 2024-09-16T01:39:53.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.726+0000 7f606c7cf640 1 -- 172.21.15.88:0/4012835105 >> 172.21.15.88:0/4012835105 conn(0x7f606406d0f0 msgr2=0x7f606406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:53.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.726+0000 7f606c7cf640 1 -- 172.21.15.88:0/4012835105 shutdown_connections 2024-09-16T01:39:53.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.726+0000 7f606c7cf640 1 -- 172.21.15.88:0/4012835105 wait complete. 2024-09-16T01:39:53.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.727+0000 7f606c7cf640 1 Processor -- start 2024-09-16T01:39:53.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.727+0000 7f606c7cf640 1 -- start start 2024-09-16T01:39:53.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.728+0000 7f606c7cf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6050004f90 0x7f6064143030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:53.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.728+0000 7f606c7cf640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60640d0330 0x7f60640d07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:53.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.728+0000 7f606c7cf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6064156420 0x7f60641568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:53.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.728+0000 7f606c7cf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f606415b9e0 con 0x7f6064156420 2024-09-16T01:39:53.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.728+0000 7f606c7cf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f606415b860 con 0x7f6050004f90 2024-09-16T01:39:53.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.728+0000 7f606c7cf640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6064071ff0 con 0x7f60640d0330 2024-09-16T01:39:53.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.728+0000 7f606ad6c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6064156420 0x7f60641568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:53.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.728+0000 7f606ad6c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6064156420 0x7f60641568d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60146/0 (socket says 172.21.15.88:60146) 2024-09-16T01:39:53.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.728+0000 7f606ad6c640 1 -- 172.21.15.88:0/555261270 learned_addr learned my addr 172.21.15.88:0/555261270 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:53.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.728+0000 7f606a56b640 1 --2- 172.21.15.88:0/555261270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6050004f90 0x7f6064143030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:53.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.728+0000 7f6069d6a640 1 --2- 172.21.15.88:0/555261270 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60640d0330 0x7f60640d07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:53.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.728+0000 7f606ad6c640 1 -- 172.21.15.88:0/555261270 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60640d0330 msgr2=0x7f60640d07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:53.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.728+0000 7f606ad6c640 1 --2- 172.21.15.88:0/555261270 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60640d0330 0x7f60640d07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.728+0000 7f606ad6c640 1 -- 172.21.15.88:0/555261270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6050004f90 msgr2=0x7f6064143030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:53.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.728+0000 7f606ad6c640 1 --2- 172.21.15.88:0/555261270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6050004f90 0x7f6064143030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.728+0000 7f606ad6c640 1 -- 172.21.15.88:0/555261270 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60641f0af0 con 0x7f6064156420 2024-09-16T01:39:53.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.729+0000 7f606ad6c640 1 --2- 172.21.15.88:0/555261270 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6064156420 0x7f60641568d0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f605c00a9a0 tx=0x7f60640cd1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:53.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.729+0000 7f6056ffd640 1 -- 172.21.15.88:0/555261270 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605c00a4a0 con 0x7f6064156420 2024-09-16T01:39:53.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.729+0000 7f6056ffd640 1 -- 172.21.15.88:0/555261270 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f605c044070 con 0x7f6064156420 2024-09-16T01:39:53.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.729+0000 7f606c7cf640 1 -- 172.21.15.88:0/555261270 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6064156e10 con 0x7f6064156420 2024-09-16T01:39:53.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.729+0000 7f6056ffd640 1 -- 172.21.15.88:0/555261270 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605c034470 con 0x7f6064156420 2024-09-16T01:39:53.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.729+0000 7f606c7cf640 1 -- 172.21.15.88:0/555261270 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60641f1070 con 0x7f6064156420 2024-09-16T01:39:53.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.730+0000 7f606c7cf640 1 -- 172.21.15.88:0/555261270 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60640b5b80 con 0x7f6064156420 2024-09-16T01:39:53.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.730+0000 7f6056ffd640 1 -- 172.21.15.88:0/555261270 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f605c03b040 con 0x7f6064156420 2024-09-16T01:39:53.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.731+0000 7f6056ffd640 1 --2- 172.21.15.88:0/555261270 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f604806c2e0 0x7f604806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:53.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.731+0000 7f6056ffd640 1 -- 172.21.15.88:0/555261270 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5795+0+0 (secure 0 0 0) 0x7f605c0ae080 con 0x7f6064156420 2024-09-16T01:39:53.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.731+0000 7f606a56b640 1 --2- 172.21.15.88:0/555261270 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f604806c2e0 0x7f604806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:53.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.731+0000 7f606a56b640 1 --2- 172.21.15.88:0/555261270 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f604806c2e0 0x7f604806e7a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f6058001910 tx=0x7f6058009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:53.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.735+0000 7f6056ffd640 1 -- 172.21.15.88:0/555261270 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f605c07c790 con 0x7f6064156420 2024-09-16T01:39:53.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.932+0000 7f606c7cf640 1 -- 172.21.15.88:0/555261270 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f6064158480 con 0x7f6064156420 2024-09-16T01:39:53.934 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:39:53.932+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:39:53.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.934+0000 7f6056ffd640 1 -- 172.21.15.88:0/555261270 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v65) ==== 161+0+0 (secure 0 0 0) 0x7f605c07c090 con 0x7f6064156420 2024-09-16T01:39:53.935 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:39:53.938 INFO:tasks.ceph.mds.a.smithi088.stderr: -3558> 2024-09-16T01:39:11.394+0000 7efd86ca0640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-09-16T01:39:53.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.937+0000 7f606c7cf640 1 -- 172.21.15.88:0/555261270 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f604806c2e0 msgr2=0x7f604806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:53.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.937+0000 7f606c7cf640 1 --2- 172.21.15.88:0/555261270 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f604806c2e0 0x7f604806e7a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f6058001910 tx=0x7f6058009040 comp rx=0 tx=0).stop 2024-09-16T01:39:53.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.937+0000 7f606c7cf640 1 -- 172.21.15.88:0/555261270 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6064156420 msgr2=0x7f60641568d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:53.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.937+0000 7f606c7cf640 1 --2- 172.21.15.88:0/555261270 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6064156420 0x7f60641568d0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f605c00a9a0 tx=0x7f60640cd1c0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.937+0000 7f606c7cf640 1 -- 172.21.15.88:0/555261270 shutdown_connections 2024-09-16T01:39:53.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.937+0000 7f606c7cf640 1 --2- 172.21.15.88:0/555261270 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f604806c2e0 0x7f604806e7a0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.937+0000 7f606c7cf640 1 --2- 172.21.15.88:0/555261270 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6064156420 0x7f60641568d0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.937+0000 7f606c7cf640 1 --2- 172.21.15.88:0/555261270 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60640d0330 0x7f60640d07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.937+0000 7f606c7cf640 1 --2- 172.21.15.88:0/555261270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6050004f90 0x7f6064143030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:53.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.937+0000 7f606c7cf640 1 -- 172.21.15.88:0/555261270 >> 172.21.15.88:0/555261270 conn(0x7f606406d0f0 msgr2=0x7f6064157950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:53.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.937+0000 7f606c7cf640 1 -- 172.21.15.88:0/555261270 shutdown_connections 2024-09-16T01:39:53.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:53.937+0000 7f606c7cf640 1 -- 172.21.15.88:0/555261270 wait complete. 2024-09-16T01:39:53.942 INFO:tasks.ceph.mds.a.smithi088.stderr: -74> 2024-09-16T01:39:51.732+0000 7efd80c94640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:39:53.950 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-16T01:39:53.966 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:39:54.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.049+0000 7fb6c41a6640 1 Processor -- start 2024-09-16T01:39:54.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.050+0000 7fb6c41a6640 1 -- start start 2024-09-16T01:39:54.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.050+0000 7fb6c41a6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6bc15e7a0 0x7fb6bc06caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:54.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.050+0000 7fb6c41a6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6bc0cf860 con 0x7fb6bc0b68f0 2024-09-16T01:39:54.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.050+0000 7fb6c41a6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6bc0cfb40 con 0x7fb6bc0cf1f0 2024-09-16T01:39:54.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.050+0000 7fb6c41a6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb6bc0d2ca0 con 0x7fb6bc15e7a0 2024-09-16T01:39:54.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.051+0000 7fb6c2743640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb6bc0b68f0 0x7fb6bc0d3410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55972/0 (socket says 172.21.15.88:55972) 2024-09-16T01:39:54.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.051+0000 7fb6c2743640 1 -- 172.21.15.88:0/3598453960 learned_addr learned my addr 172.21.15.88:0/3598453960 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:54.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.051+0000 7fb6c0f40640 1 -- 172.21.15.88:0/3598453960 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 797260635 0 0) 0x7fb6bc0cf860 con 0x7fb6bc0b68f0 2024-09-16T01:39:54.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.051+0000 7fb6c1f42640 1 --2- 172.21.15.88:0/3598453960 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6bc15e7a0 0x7fb6bc06caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:54.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.051+0000 7fb6c0f40640 1 -- 172.21.15.88:0/3598453960 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6a8003520 con 0x7fb6bc0b68f0 2024-09-16T01:39:54.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.051+0000 7fb6c0f40640 1 -- 172.21.15.88:0/3598453960 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2239474960 0 0) 0x7fb6a8003520 con 0x7fb6bc0b68f0 2024-09-16T01:39:54.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.051+0000 7fb6c0f40640 1 -- 172.21.15.88:0/3598453960 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6bc15e7a0 msgr2=0x7fb6bc06caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:54.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.051+0000 7fb6c0f40640 1 --2- 172.21.15.88:0/3598453960 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6bc15e7a0 0x7fb6bc06caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:54.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.051+0000 7fb6c0f40640 1 -- 172.21.15.88:0/3598453960 >> v1:172.21.15.154:6789/0 conn(0x7fb6bc0cf1f0 legacy=0x7fb6bc0b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:54.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.051+0000 7fb6c0f40640 1 -- 172.21.15.88:0/3598453960 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6bc0cfe20 con 0x7fb6bc0b68f0 2024-09-16T01:39:54.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.051+0000 7fb6c0f40640 1 -- 172.21.15.88:0/3598453960 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fb6b40019f0 con 0x7fb6bc0b68f0 2024-09-16T01:39:54.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.051+0000 7fb6c0f40640 1 -- 172.21.15.88:0/3598453960 >> v1:172.21.15.88:6789/0 conn(0x7fb6bc0b68f0 legacy=0x7fb6bc0d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:54.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.051+0000 7fb6c0f40640 1 --2- 172.21.15.88:0/3598453960 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6a80039e0 0x7fb6a8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:54.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.051+0000 7fb6c0f40640 1 --2- 172.21.15.88:0/3598453960 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6bc15e7a0 0x7fb6a8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:54.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.051+0000 7fb6c0f40640 1 --2- 172.21.15.88:0/3598453960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6a8004d40 0x7fb6a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:54.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.051+0000 7fb6c0f40640 1 -- 172.21.15.88:0/3598453960 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb6bc0d2ca0 con 0x7fb6a80039e0 2024-09-16T01:39:54.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.051+0000 7fb6c0f40640 1 -- 172.21.15.88:0/3598453960 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a800baf0 con 0x7fb6a8004d40 2024-09-16T01:39:54.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.051+0000 7fb6c0f40640 1 -- 172.21.15.88:0/3598453960 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb6a800bcf0 con 0x7fb6bc15e7a0 2024-09-16T01:39:54.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.051+0000 7fb6c1f42640 1 --2- 172.21.15.88:0/3598453960 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6a80039e0 0x7fb6a8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:54.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.051+0000 7fb6c2743640 1 --2- 172.21.15.88:0/3598453960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6a8004d40 0x7fb6a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:54.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.051+0000 7fb6c1741640 1 --2- 172.21.15.88:0/3598453960 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6bc15e7a0 0x7fb6a8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:54.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.052+0000 7fb6c1f42640 1 -- 172.21.15.88:0/3598453960 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6bc15e7a0 msgr2=0x7fb6a8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:54.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.052+0000 7fb6c1f42640 1 --2- 172.21.15.88:0/3598453960 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6bc15e7a0 0x7fb6a8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:54.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.052+0000 7fb6c1f42640 1 -- 172.21.15.88:0/3598453960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6a8004d40 msgr2=0x7fb6a80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:54.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.052+0000 7fb6c1f42640 1 --2- 172.21.15.88:0/3598453960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6a8004d40 0x7fb6a80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:54.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.052+0000 7fb6c1f42640 1 -- 172.21.15.88:0/3598453960 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a8003520 con 0x7fb6a80039e0 2024-09-16T01:39:54.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.052+0000 7fb6c1f42640 1 --2- 172.21.15.88:0/3598453960 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6a80039e0 0x7fb6a8003e00 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8009ac0 tx=0x7fb6a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbe77b78c51b69e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:54.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.052+0000 7fb6c0f40640 1 -- 172.21.15.88:0/3598453960 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b8036070 con 0x7fb6a80039e0 2024-09-16T01:39:54.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.052+0000 7fb6c0f40640 1 -- 172.21.15.88:0/3598453960 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6b8031400 con 0x7fb6a80039e0 2024-09-16T01:39:54.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.052+0000 7fb6c0f40640 1 -- 172.21.15.88:0/3598453960 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b8009690 con 0x7fb6a80039e0 2024-09-16T01:39:54.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.053+0000 7fb6c41a6640 1 -- 172.21.15.88:0/3598453960 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6a80039e0 msgr2=0x7fb6a8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:54.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.053+0000 7fb6c41a6640 1 --2- 172.21.15.88:0/3598453960 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6a80039e0 0x7fb6a8003e00 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8009ac0 tx=0x7fb6a8005830 comp rx=0 tx=0).stop 2024-09-16T01:39:54.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.053+0000 7fb6c2743640 1 -- 172.21.15.88:0/3598453960 reap_dead start 2024-09-16T01:39:54.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.053+0000 7fb6c41a6640 1 -- 172.21.15.88:0/3598453960 shutdown_connections 2024-09-16T01:39:54.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.053+0000 7fb6c41a6640 1 -- 172.21.15.88:0/3598453960 >> 172.21.15.88:0/3598453960 conn(0x7fb6bc0d11e0 msgr2=0x7fb6bc0d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:54.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.053+0000 7fb6c41a6640 1 -- 172.21.15.88:0/3598453960 shutdown_connections 2024-09-16T01:39:54.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.053+0000 7fb6c41a6640 1 -- 172.21.15.88:0/3598453960 wait complete. 2024-09-16T01:39:54.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.054+0000 7fb6c41a6640 1 Processor -- start 2024-09-16T01:39:54.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.054+0000 7fb6c41a6640 1 -- start start 2024-09-16T01:39:54.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.055+0000 7fb6c41a6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6a80039e0 0x7fb6bc14f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:54.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.055+0000 7fb6c41a6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6bc14eba0 0x7fb6bc15ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:54.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.055+0000 7fb6c41a6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6bc15f3c0 0x7fb6bc15f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:54.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.055+0000 7fb6c41a6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb6bc15eb80 con 0x7fb6a80039e0 2024-09-16T01:39:54.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.055+0000 7fb6c41a6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb6bc0b7a00 con 0x7fb6bc14eba0 2024-09-16T01:39:54.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.055+0000 7fb6c41a6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb6bc15ed00 con 0x7fb6bc15f3c0 2024-09-16T01:39:54.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.055+0000 7fb6c1f42640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6a80039e0 0x7fb6bc14f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:54.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.055+0000 7fb6c1f42640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6a80039e0 0x7fb6bc14f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60190/0 (socket says 172.21.15.88:60190) 2024-09-16T01:39:54.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.055+0000 7fb6c1f42640 1 -- 172.21.15.88:0/3399417744 learned_addr learned my addr 172.21.15.88:0/3399417744 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:54.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.055+0000 7fb6c1741640 1 --2- 172.21.15.88:0/3399417744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6bc14eba0 0x7fb6bc15ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:54.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.055+0000 7fb6c2743640 1 --2- 172.21.15.88:0/3399417744 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6bc15f3c0 0x7fb6bc15f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:54.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.055+0000 7fb6c1f42640 1 -- 172.21.15.88:0/3399417744 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6bc15f3c0 msgr2=0x7fb6bc15f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:54.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.055+0000 7fb6c1f42640 1 --2- 172.21.15.88:0/3399417744 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6bc15f3c0 0x7fb6bc15f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:54.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.055+0000 7fb6c1f42640 1 -- 172.21.15.88:0/3399417744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6bc14eba0 msgr2=0x7fb6bc15ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:54.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.055+0000 7fb6c1f42640 1 --2- 172.21.15.88:0/3399417744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6bc14eba0 0x7fb6bc15ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:54.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.055+0000 7fb6c1f42640 1 -- 172.21.15.88:0/3399417744 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6bc14fb80 con 0x7fb6a80039e0 2024-09-16T01:39:54.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.055+0000 7fb6c2743640 1 --2- 172.21.15.88:0/3399417744 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6bc15f3c0 0x7fb6bc15f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:54.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.055+0000 7fb6c1741640 1 --2- 172.21.15.88:0/3399417744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6bc14eba0 0x7fb6bc15ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:54.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.055+0000 7fb6c1f42640 1 --2- 172.21.15.88:0/3399417744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6a80039e0 0x7fb6bc14f3e0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8009ac0 tx=0x7fb6b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:54.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.056+0000 7fb6b27fc640 1 -- 172.21.15.88:0/3399417744 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b8040020 con 0x7fb6a80039e0 2024-09-16T01:39:54.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.056+0000 7fb6b27fc640 1 -- 172.21.15.88:0/3399417744 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6b8003bc0 con 0x7fb6a80039e0 2024-09-16T01:39:54.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.056+0000 7fb6c41a6640 1 -- 172.21.15.88:0/3399417744 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6bc14fe60 con 0x7fb6a80039e0 2024-09-16T01:39:54.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.056+0000 7fb6b27fc640 1 -- 172.21.15.88:0/3399417744 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b8036040 con 0x7fb6a80039e0 2024-09-16T01:39:54.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.056+0000 7fb6c41a6640 1 -- 172.21.15.88:0/3399417744 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6bc150420 con 0x7fb6a80039e0 2024-09-16T01:39:54.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.057+0000 7fb6c41a6640 1 -- 172.21.15.88:0/3399417744 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6bc0b61e0 con 0x7fb6a80039e0 2024-09-16T01:39:54.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.057+0000 7fb6b27fc640 1 -- 172.21.15.88:0/3399417744 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb6b8043020 con 0x7fb6a80039e0 2024-09-16T01:39:54.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.058+0000 7fb6b27fc640 1 --2- 172.21.15.88:0/3399417744 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb6a006c2e0 0x7fb6a006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:54.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.058+0000 7fb6b27fc640 1 -- 172.21.15.88:0/3399417744 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5881+0+0 (secure 0 0 0) 0x7fb6b80ab4e0 con 0x7fb6a80039e0 2024-09-16T01:39:54.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.058+0000 7fb6c1741640 1 --2- 172.21.15.88:0/3399417744 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb6a006c2e0 0x7fb6a006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:54.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.058+0000 7fb6c1741640 1 --2- 172.21.15.88:0/3399417744 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb6a006c2e0 0x7fb6a006e7a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc0d4980 tx=0x7fb6ac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:54.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.062+0000 7fb6b27fc640 1 -- 172.21.15.88:0/3399417744 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb6b804a070 con 0x7fb6a80039e0 2024-09-16T01:39:54.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.261+0000 7fb6c41a6640 1 -- 172.21.15.88:0/3399417744 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fb6bc0b5480 con 0x7fb6a80039e0 2024-09-16T01:39:54.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.935+0000 7fb6b27fc640 1 -- 172.21.15.88:0/3399417744 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v66) ==== 104+0+0 (secure 0 0 0) 0x7fb6b803ae20 con 0x7fb6a80039e0 2024-09-16T01:39:54.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.937+0000 7fb6c41a6640 1 -- 172.21.15.88:0/3399417744 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb6a006c2e0 msgr2=0x7fb6a006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:54.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.937+0000 7fb6c41a6640 1 --2- 172.21.15.88:0/3399417744 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb6a006c2e0 0x7fb6a006e7a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc0d4980 tx=0x7fb6ac009040 comp rx=0 tx=0).stop 2024-09-16T01:39:54.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.937+0000 7fb6c41a6640 1 -- 172.21.15.88:0/3399417744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6a80039e0 msgr2=0x7fb6bc14f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:54.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.937+0000 7fb6c41a6640 1 --2- 172.21.15.88:0/3399417744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6a80039e0 0x7fb6bc14f3e0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8009ac0 tx=0x7fb6b8004730 comp rx=0 tx=0).stop 2024-09-16T01:39:54.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.937+0000 7fb6c41a6640 1 -- 172.21.15.88:0/3399417744 shutdown_connections 2024-09-16T01:39:54.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.937+0000 7fb6c41a6640 1 --2- 172.21.15.88:0/3399417744 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb6a006c2e0 0x7fb6a006e7a0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:54.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.937+0000 7fb6c41a6640 1 --2- 172.21.15.88:0/3399417744 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6bc15f3c0 0x7fb6bc15f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:54.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.937+0000 7fb6c41a6640 1 --2- 172.21.15.88:0/3399417744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6bc14eba0 0x7fb6bc15ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:54.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.937+0000 7fb6c41a6640 1 --2- 172.21.15.88:0/3399417744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6a80039e0 0x7fb6bc14f3e0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:54.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.937+0000 7fb6c41a6640 1 -- 172.21.15.88:0/3399417744 >> 172.21.15.88:0/3399417744 conn(0x7fb6bc0d11e0 msgr2=0x7fb6bc0b93b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:54.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.937+0000 7fb6c41a6640 1 -- 172.21.15.88:0/3399417744 shutdown_connections 2024-09-16T01:39:54.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:54.937+0000 7fb6c41a6640 1 -- 172.21.15.88:0/3399417744 wait complete. 2024-09-16T01:39:54.951 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-16T01:39:55.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.094+0000 7fcf0aeed640 1 Processor -- start 2024-09-16T01:39:55.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.094+0000 7fcf0aeed640 1 -- start start 2024-09-16T01:39:55.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.095+0000 7fcf0aeed640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf0415e990 0x7fcf040b3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:55.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.095+0000 7fcf0aeed640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf04162450 0x7fcf04162890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:55.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.095+0000 7fcf0aeed640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf0415ed70 con 0x7fcf04162dd0 2024-09-16T01:39:55.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.095+0000 7fcf0aeed640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fcf040cff20 con 0x7fcf0415e990 2024-09-16T01:39:55.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.095+0000 7fcf0aeed640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcf04163280 con 0x7fcf04162450 2024-09-16T01:39:55.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.095+0000 7fcf08c89640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf0415e990 0x7fcf040b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:55.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.095+0000 7fcf08c89640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf0415e990 0x7fcf040b3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60204/0 (socket says 172.21.15.88:60204) 2024-09-16T01:39:55.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.095+0000 7fcf08c89640 1 -- 172.21.15.88:0/3421607144 learned_addr learned my addr 172.21.15.88:0/3421607144 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:55.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.096+0000 7fcf03fff640 1 --2- 172.21.15.88:0/3421607144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf04162450 0x7fcf04162890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:55.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.096+0000 7fcf08c89640 1 -- 172.21.15.88:0/3421607144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf04162450 msgr2=0x7fcf04162890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:55.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.096+0000 7fcf08c89640 1 --2- 172.21.15.88:0/3421607144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf04162450 0x7fcf04162890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:55.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.096+0000 7fcf08c89640 1 -- 172.21.15.88:0/3421607144 >> v1:172.21.15.154:6789/0 conn(0x7fcf04162dd0 legacy=0x7fcf040b6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:55.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.096+0000 7fcf08c89640 1 -- 172.21.15.88:0/3421607144 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf040d1fa0 con 0x7fcf0415e990 2024-09-16T01:39:55.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.096+0000 7fcf08c89640 1 --2- 172.21.15.88:0/3421607144 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf0415e990 0x7fcf040b3920 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fcef4009e50 tx=0x7fcef402a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b5d6aa376933df4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:55.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.096+0000 7fcf037fe640 1 -- 172.21.15.88:0/3421607144 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef4036070 con 0x7fcf0415e990 2024-09-16T01:39:55.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.097+0000 7fcf037fe640 1 -- 172.21.15.88:0/3421607144 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcef4031600 con 0x7fcf0415e990 2024-09-16T01:39:55.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.097+0000 7fcf037fe640 1 -- 172.21.15.88:0/3421607144 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef40319d0 con 0x7fcf0415e990 2024-09-16T01:39:55.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.097+0000 7fcf0aeed640 1 -- 172.21.15.88:0/3421607144 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf0415e990 msgr2=0x7fcf040b3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:55.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.097+0000 7fcf0aeed640 1 --2- 172.21.15.88:0/3421607144 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf0415e990 0x7fcf040b3920 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fcef4009e50 tx=0x7fcef402a630 comp rx=0 tx=0).stop 2024-09-16T01:39:55.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.097+0000 7fcf0aeed640 1 -- 172.21.15.88:0/3421607144 shutdown_connections 2024-09-16T01:39:55.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.097+0000 7fcf0aeed640 1 --2- 172.21.15.88:0/3421607144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf04162450 0x7fcf04162890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:55.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.097+0000 7fcf0aeed640 1 --2- 172.21.15.88:0/3421607144 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf0415e990 0x7fcf040b3920 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:55.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.097+0000 7fcf0aeed640 1 -- 172.21.15.88:0/3421607144 >> 172.21.15.88:0/3421607144 conn(0x7fcf040d1200 msgr2=0x7fcf040d1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:55.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.097+0000 7fcf0aeed640 1 -- 172.21.15.88:0/3421607144 shutdown_connections 2024-09-16T01:39:55.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.097+0000 7fcf0aeed640 1 -- 172.21.15.88:0/3421607144 wait complete. 2024-09-16T01:39:55.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.098+0000 7fcf0aeed640 1 Processor -- start 2024-09-16T01:39:55.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.098+0000 7fcf0aeed640 1 -- start start 2024-09-16T01:39:55.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.099+0000 7fcf0aeed640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf0415e990 0x7fcf040b5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:55.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.099+0000 7fcf0aeed640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf04162450 0x7fcf041502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:55.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.099+0000 7fcf0aeed640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf04162dd0 0x7fcf040b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:55.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.099+0000 7fcf0aeed640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcf040d39b0 con 0x7fcf04162dd0 2024-09-16T01:39:55.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.099+0000 7fcf0aeed640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcf040cfc40 con 0x7fcf04162450 2024-09-16T01:39:55.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.099+0000 7fcf0aeed640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcf040d3b30 con 0x7fcf0415e990 2024-09-16T01:39:55.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.099+0000 7fcf0948a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf04162dd0 0x7fcf040b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:55.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.099+0000 7fcf0948a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf04162dd0 0x7fcf040b4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60218/0 (socket says 172.21.15.88:60218) 2024-09-16T01:39:55.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.099+0000 7fcf0948a640 1 -- 172.21.15.88:0/810164622 learned_addr learned my addr 172.21.15.88:0/810164622 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:55.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.099+0000 7fcf08c89640 1 --2- 172.21.15.88:0/810164622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf0415e990 0x7fcf040b5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:55.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.100+0000 7fcf0948a640 1 -- 172.21.15.88:0/810164622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf0415e990 msgr2=0x7fcf040b5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:55.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.100+0000 7fcf0948a640 1 --2- 172.21.15.88:0/810164622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf0415e990 0x7fcf040b5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:55.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.100+0000 7fcf0948a640 1 -- 172.21.15.88:0/810164622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf04162450 msgr2=0x7fcf041502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:55.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.100+0000 7fcf0948a640 1 --2- 172.21.15.88:0/810164622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf04162450 0x7fcf041502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:55.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.100+0000 7fcf0948a640 1 -- 172.21.15.88:0/810164622 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf040b66d0 con 0x7fcf04162dd0 2024-09-16T01:39:55.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.100+0000 7fcf0948a640 1 --2- 172.21.15.88:0/810164622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf04162dd0 0x7fcf040b4580 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fcefc00c410 tx=0x7fcefc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:55.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.100+0000 7fcf017fa640 1 -- 172.21.15.88:0/810164622 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc011790 con 0x7fcf04162dd0 2024-09-16T01:39:55.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.100+0000 7fcf017fa640 1 -- 172.21.15.88:0/810164622 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcefc017070 con 0x7fcf04162dd0 2024-09-16T01:39:55.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.100+0000 7fcf0aeed640 1 -- 172.21.15.88:0/810164622 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf040b69b0 con 0x7fcf04162dd0 2024-09-16T01:39:55.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.100+0000 7fcf017fa640 1 -- 172.21.15.88:0/810164622 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc011bb0 con 0x7fcf04162dd0 2024-09-16T01:39:55.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.100+0000 7fcf0aeed640 1 -- 172.21.15.88:0/810164622 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf040b6f70 con 0x7fcf04162dd0 2024-09-16T01:39:55.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.101+0000 7fcf0aeed640 1 -- 172.21.15.88:0/810164622 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf040b3920 con 0x7fcf04162dd0 2024-09-16T01:39:55.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.101+0000 7fcf017fa640 1 -- 172.21.15.88:0/810164622 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fcefc01c460 con 0x7fcf04162dd0 2024-09-16T01:39:55.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.102+0000 7fcf017fa640 1 --2- 172.21.15.88:0/810164622 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcef006c2b0 0x7fcef006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:55.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.102+0000 7fcf017fa640 1 -- 172.21.15.88:0/810164622 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5881+0+0 (secure 0 0 0) 0x7fcefc08b0f0 con 0x7fcf04162dd0 2024-09-16T01:39:55.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.102+0000 7fcf08c89640 1 --2- 172.21.15.88:0/810164622 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcef006c2b0 0x7fcef006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:55.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.102+0000 7fcf08c89640 1 --2- 172.21.15.88:0/810164622 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcef006c2b0 0x7fcef006e770 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fcef4033000 tx=0x7fcef4009000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:55.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.106+0000 7fcf017fa640 1 -- 172.21.15.88:0/810164622 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fcefc021020 con 0x7fcf04162dd0 2024-09-16T01:39:55.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.273+0000 7fcf0aeed640 1 -- 172.21.15.88:0/810164622 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fcf040ba700 con 0x7fcf04162dd0 2024-09-16T01:39:55.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.937+0000 7fcf017fa640 1 -- 172.21.15.88:0/810164622 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v75) ==== 181+0+0 (secure 0 0 0) 0x7fcefc059720 con 0x7fcf04162dd0 2024-09-16T01:39:55.938 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_metadata' removed 2024-09-16T01:39:55.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.940+0000 7fcf0aeed640 1 -- 172.21.15.88:0/810164622 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcef006c2b0 msgr2=0x7fcef006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:55.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.940+0000 7fcf0aeed640 1 --2- 172.21.15.88:0/810164622 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcef006c2b0 0x7fcef006e770 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fcef4033000 tx=0x7fcef4009000 comp rx=0 tx=0).stop 2024-09-16T01:39:55.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.940+0000 7fcf0aeed640 1 -- 172.21.15.88:0/810164622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf04162dd0 msgr2=0x7fcf040b4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:55.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.940+0000 7fcf0aeed640 1 --2- 172.21.15.88:0/810164622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf04162dd0 0x7fcf040b4580 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fcefc00c410 tx=0x7fcefc00c8d0 comp rx=0 tx=0).stop 2024-09-16T01:39:55.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.940+0000 7fcf0aeed640 1 -- 172.21.15.88:0/810164622 shutdown_connections 2024-09-16T01:39:55.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.940+0000 7fcf0aeed640 1 --2- 172.21.15.88:0/810164622 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcef006c2b0 0x7fcef006e770 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:55.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.940+0000 7fcf0aeed640 1 --2- 172.21.15.88:0/810164622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf04162dd0 0x7fcf040b4580 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:55.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.940+0000 7fcf0aeed640 1 --2- 172.21.15.88:0/810164622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf04162450 0x7fcf041502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:55.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.940+0000 7fcf0aeed640 1 --2- 172.21.15.88:0/810164622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf0415e990 0x7fcf040b5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:55.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.940+0000 7fcf0aeed640 1 -- 172.21.15.88:0/810164622 >> 172.21.15.88:0/810164622 conn(0x7fcf040d1200 msgr2=0x7fcf040d15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:55.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.940+0000 7fcf0aeed640 1 -- 172.21.15.88:0/810164622 shutdown_connections 2024-09-16T01:39:55.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:55.940+0000 7fcf0aeed640 1 -- 172.21.15.88:0/810164622 wait complete. 2024-09-16T01:39:55.955 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-16T01:39:56.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.058+0000 7f0283fff640 1 Processor -- start 2024-09-16T01:39:56.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.058+0000 7f0283fff640 1 -- start start 2024-09-16T01:39:56.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.059+0000 7f0283fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02840d4380 0x7f02840d47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:56.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.059+0000 7f0283fff640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02840d40d0 con 0x7f028415e700 2024-09-16T01:39:56.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.059+0000 7f0283fff640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02840cead0 con 0x7f02840b7510 2024-09-16T01:39:56.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.059+0000 7f0283fff640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f02840cfb70 con 0x7f02840d4380 2024-09-16T01:39:56.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.059+0000 7f0282ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f028415e700 0x7f02840d39c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:55986/0 (socket says 172.21.15.88:55986) 2024-09-16T01:39:56.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.059+0000 7f0282ffd640 1 -- 172.21.15.88:0/1362691926 learned_addr learned my addr 172.21.15.88:0/1362691926 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:56.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.060+0000 7f0281ffb640 1 -- 172.21.15.88:0/1362691926 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1573193215 0 0) 0x7f02840d40d0 con 0x7f028415e700 2024-09-16T01:39:56.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.060+0000 7f02827fc640 1 --2- 172.21.15.88:0/1362691926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02840d4380 0x7f02840d47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:56.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.060+0000 7f0281ffb640 1 -- 172.21.15.88:0/1362691926 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f026c003520 con 0x7f028415e700 2024-09-16T01:39:56.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.060+0000 7f0281ffb640 1 -- 172.21.15.88:0/1362691926 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 514922670 0 0) 0x7f026c003520 con 0x7f028415e700 2024-09-16T01:39:56.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.060+0000 7f0281ffb640 1 -- 172.21.15.88:0/1362691926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02840d4380 msgr2=0x7f02840d47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:56.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.060+0000 7f0281ffb640 1 --2- 172.21.15.88:0/1362691926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02840d4380 0x7f02840d47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:56.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.060+0000 7f0281ffb640 1 -- 172.21.15.88:0/1362691926 >> v1:172.21.15.154:6789/0 conn(0x7f02840b7510 legacy=0x7f02840b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:56.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.060+0000 7f0281ffb640 1 -- 172.21.15.88:0/1362691926 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02840cedb0 con 0x7f028415e700 2024-09-16T01:39:56.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.060+0000 7f0281ffb640 1 -- 172.21.15.88:0/1362691926 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f0274001670 con 0x7f028415e700 2024-09-16T01:39:56.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.060+0000 7f0281ffb640 1 -- 172.21.15.88:0/1362691926 >> v1:172.21.15.88:6789/0 conn(0x7f028415e700 legacy=0x7f02840d39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:56.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.060+0000 7f0281ffb640 1 --2- 172.21.15.88:0/1362691926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02840d4380 0x7f026c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:56.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.060+0000 7f0281ffb640 1 --2- 172.21.15.88:0/1362691926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f026c003f60 0x7f026c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:56.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.060+0000 7f0281ffb640 1 --2- 172.21.15.88:0/1362691926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f026c004d40 0x7f026c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:56.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.060+0000 7f0281ffb640 1 -- 172.21.15.88:0/1362691926 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f026c00ba70 con 0x7f026c003f60 2024-09-16T01:39:56.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.060+0000 7f0281ffb640 1 -- 172.21.15.88:0/1362691926 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f026c00bc20 con 0x7f026c004d40 2024-09-16T01:39:56.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.060+0000 7f0281ffb640 1 -- 172.21.15.88:0/1362691926 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f026c00be20 con 0x7f02840d4380 2024-09-16T01:39:56.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.060+0000 7f02827fc640 1 --2- 172.21.15.88:0/1362691926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f026c003f60 0x7f026c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:56.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.060+0000 7f0282ffd640 1 --2- 172.21.15.88:0/1362691926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02840d4380 0x7f026c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:56.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.060+0000 7f02837fe640 1 --2- 172.21.15.88:0/1362691926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f026c004d40 0x7f026c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:56.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.060+0000 7f02827fc640 1 -- 172.21.15.88:0/1362691926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02840d4380 msgr2=0x7f026c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:56.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.060+0000 7f02827fc640 1 --2- 172.21.15.88:0/1362691926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02840d4380 0x7f026c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:56.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.060+0000 7f02827fc640 1 -- 172.21.15.88:0/1362691926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f026c004d40 msgr2=0x7f026c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:56.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.060+0000 7f02827fc640 1 --2- 172.21.15.88:0/1362691926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f026c004d40 0x7f026c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:56.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.060+0000 7f02827fc640 1 -- 172.21.15.88:0/1362691926 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f026c003520 con 0x7f026c003f60 2024-09-16T01:39:56.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.061+0000 7f02827fc640 1 --2- 172.21.15.88:0/1362691926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f026c003f60 0x7f026c004800 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f0278009ac0 tx=0x7f026c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=943221797da13d2d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:56.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.061+0000 7f0281ffb640 1 -- 172.21.15.88:0/1362691926 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0278036070 con 0x7f026c003f60 2024-09-16T01:39:56.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.061+0000 7f0281ffb640 1 -- 172.21.15.88:0/1362691926 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0278031400 con 0x7f026c003f60 2024-09-16T01:39:56.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.061+0000 7f0281ffb640 1 -- 172.21.15.88:0/1362691926 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0278009690 con 0x7f026c003f60 2024-09-16T01:39:56.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.062+0000 7f0283fff640 1 -- 172.21.15.88:0/1362691926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f026c003f60 msgr2=0x7f026c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:56.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.062+0000 7f0283fff640 1 --2- 172.21.15.88:0/1362691926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f026c003f60 0x7f026c004800 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f0278009ac0 tx=0x7f026c005830 comp rx=0 tx=0).stop 2024-09-16T01:39:56.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.062+0000 7f02837fe640 1 -- 172.21.15.88:0/1362691926 reap_dead start 2024-09-16T01:39:56.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.062+0000 7f0283fff640 1 -- 172.21.15.88:0/1362691926 shutdown_connections 2024-09-16T01:39:56.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.062+0000 7f0283fff640 1 -- 172.21.15.88:0/1362691926 >> 172.21.15.88:0/1362691926 conn(0x7f0284062a20 msgr2=0x7f0284062e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:56.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.062+0000 7f0283fff640 1 -- 172.21.15.88:0/1362691926 shutdown_connections 2024-09-16T01:39:56.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.062+0000 7f0283fff640 1 -- 172.21.15.88:0/1362691926 wait complete. 2024-09-16T01:39:56.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.063+0000 7f0283fff640 1 Processor -- start 2024-09-16T01:39:56.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.063+0000 7f0283fff640 1 -- start start 2024-09-16T01:39:56.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.064+0000 7f0283fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f026c003f60 0x7f0284161470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:56.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.064+0000 7f0283fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02840caab0 0x7f02840c91c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:56.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.064+0000 7f0283fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02841508e0 0x7f02840b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:56.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.064+0000 7f0283fff640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f02840cf650 con 0x7f026c003f60 2024-09-16T01:39:56.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.064+0000 7f0283fff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f02840cf370 con 0x7f02841508e0 2024-09-16T01:39:56.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.064+0000 7f0283fff640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f02840b6c60 con 0x7f02840caab0 2024-09-16T01:39:56.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.064+0000 7f0282ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f026c003f60 0x7f0284161470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:56.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.064+0000 7f0282ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f026c003f60 0x7f0284161470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60232/0 (socket says 172.21.15.88:60232) 2024-09-16T01:39:56.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.064+0000 7f0282ffd640 1 -- 172.21.15.88:0/3191465287 learned_addr learned my addr 172.21.15.88:0/3191465287 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:56.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.064+0000 7f0282ffd640 1 -- 172.21.15.88:0/3191465287 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02840caab0 msgr2=0x7f02840c91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:56.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.064+0000 7f0282ffd640 1 --2- 172.21.15.88:0/3191465287 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02840caab0 0x7f02840c91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:56.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.064+0000 7f0282ffd640 1 -- 172.21.15.88:0/3191465287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02841508e0 msgr2=0x7f02840b3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:56.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.064+0000 7f0282ffd640 1 --2- 172.21.15.88:0/3191465287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02841508e0 0x7f02840b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:56.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.064+0000 7f0282ffd640 1 -- 172.21.15.88:0/3191465287 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02840c9a10 con 0x7f026c003f60 2024-09-16T01:39:56.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.064+0000 7f0282ffd640 1 --2- 172.21.15.88:0/3191465287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f026c003f60 0x7f0284161470 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f02740072c0 tx=0x7f0274007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:56.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.064+0000 7f026b7fe640 1 -- 172.21.15.88:0/3191465287 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027400f9c0 con 0x7f026c003f60 2024-09-16T01:39:56.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.064+0000 7f026b7fe640 1 -- 172.21.15.88:0/3191465287 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0274018070 con 0x7f026c003f60 2024-09-16T01:39:56.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.064+0000 7f0283fff640 1 -- 172.21.15.88:0/3191465287 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02840c9cf0 con 0x7f026c003f60 2024-09-16T01:39:56.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.065+0000 7f026b7fe640 1 -- 172.21.15.88:0/3191465287 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027400fdc0 con 0x7f026c003f60 2024-09-16T01:39:56.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.065+0000 7f0283fff640 1 -- 172.21.15.88:0/3191465287 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02840ca2b0 con 0x7f026c003f60 2024-09-16T01:39:56.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.065+0000 7f0283fff640 1 -- 172.21.15.88:0/3191465287 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02840d47c0 con 0x7f026c003f60 2024-09-16T01:39:56.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.066+0000 7f026b7fe640 1 -- 172.21.15.88:0/3191465287 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f027401d460 con 0x7f026c003f60 2024-09-16T01:39:56.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.066+0000 7f026b7fe640 1 --2- 172.21.15.88:0/3191465287 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f027006c2b0 0x7f027006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:56.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.067+0000 7f026b7fe640 1 -- 172.21.15.88:0/3191465287 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 5443+0+0 (secure 0 0 0) 0x7f027408b500 con 0x7f026c003f60 2024-09-16T01:39:56.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.067+0000 7f02827fc640 1 --2- 172.21.15.88:0/3191465287 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f027006c2b0 0x7f027006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:56.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.067+0000 7f02827fc640 1 --2- 172.21.15.88:0/3191465287 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f027006c2b0 0x7f027006e770 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0278033000 tx=0x7f027802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:56.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.071+0000 7f026b7fe640 1 -- 172.21.15.88:0/3191465287 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0274059d70 con 0x7f026c003f60 2024-09-16T01:39:56.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.236+0000 7f0283fff640 1 -- 172.21.15.88:0/3191465287 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f02840b63f0 con 0x7f026c003f60 2024-09-16T01:39:56.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.938+0000 7f026b7fe640 1 -- 172.21.15.88:0/3191465287 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v76) ==== 169+0+0 (secure 0 0 0) 0x7f0274059670 con 0x7f026c003f60 2024-09-16T01:39:56.940 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_data' removed 2024-09-16T01:39:56.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.941+0000 7f0283fff640 1 -- 172.21.15.88:0/3191465287 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f027006c2b0 msgr2=0x7f027006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:56.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.941+0000 7f0283fff640 1 --2- 172.21.15.88:0/3191465287 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f027006c2b0 0x7f027006e770 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f0278033000 tx=0x7f027802e000 comp rx=0 tx=0).stop 2024-09-16T01:39:56.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.941+0000 7f0283fff640 1 -- 172.21.15.88:0/3191465287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f026c003f60 msgr2=0x7f0284161470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:56.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.941+0000 7f0283fff640 1 --2- 172.21.15.88:0/3191465287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f026c003f60 0x7f0284161470 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f02740072c0 tx=0x7f0274007780 comp rx=0 tx=0).stop 2024-09-16T01:39:56.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.941+0000 7f0283fff640 1 -- 172.21.15.88:0/3191465287 shutdown_connections 2024-09-16T01:39:56.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.941+0000 7f0283fff640 1 --2- 172.21.15.88:0/3191465287 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f027006c2b0 0x7f027006e770 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:56.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.941+0000 7f0283fff640 1 --2- 172.21.15.88:0/3191465287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02841508e0 0x7f02840b3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:56.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.941+0000 7f0283fff640 1 --2- 172.21.15.88:0/3191465287 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02840caab0 0x7f02840c91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:56.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.941+0000 7f0283fff640 1 --2- 172.21.15.88:0/3191465287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f026c003f60 0x7f0284161470 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:56.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.941+0000 7f0283fff640 1 -- 172.21.15.88:0/3191465287 >> 172.21.15.88:0/3191465287 conn(0x7f0284062a20 msgr2=0x7f02840b53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:56.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.941+0000 7f0283fff640 1 -- 172.21.15.88:0/3191465287 shutdown_connections 2024-09-16T01:39:56.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:56.941+0000 7f0283fff640 1 -- 172.21.15.88:0/3191465287 wait complete. 2024-09-16T01:39:56.956 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-16T01:39:57.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.058+0000 7f08875f8640 1 Processor -- start 2024-09-16T01:39:57.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.058+0000 7f08875f8640 1 -- start start 2024-09-16T01:39:57.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.059+0000 7f08875f8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f088015b480 0x7f08800b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:57.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.059+0000 7f08875f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08800b60c0 0x7f0880071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:57.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.059+0000 7f08875f8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0880072530 0x7f0880070fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:57.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.059+0000 7f08875f8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f08800b6500 con 0x7f0880072530 2024-09-16T01:39:57.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.059+0000 7f08875f8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0880072970 con 0x7f088015b480 2024-09-16T01:39:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.059+0000 7f08875f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f08800ccff0 con 0x7f08800b60c0 2024-09-16T01:39:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.059+0000 7f0885b95640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0880072530 0x7f0880070fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.059+0000 7f0885b95640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0880072530 0x7f0880070fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60234/0 (socket says 172.21.15.88:60234) 2024-09-16T01:39:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.059+0000 7f0885b95640 1 -- 172.21.15.88:0/4011017081 learned_addr learned my addr 172.21.15.88:0/4011017081 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.059+0000 7f0885b95640 1 -- 172.21.15.88:0/4011017081 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08800b60c0 msgr2=0x7f0880071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.059+0000 7f0885b95640 1 --2- 172.21.15.88:0/4011017081 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08800b60c0 0x7f0880071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.059+0000 7f0885b95640 1 -- 172.21.15.88:0/4011017081 >> v2:172.21.15.154:3300/0 conn(0x7f088015b480 msgr2=0x7f08800b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.059+0000 7f0885b95640 1 --2- 172.21.15.88:0/4011017081 >> v2:172.21.15.154:3300/0 conn(0x7f088015b480 0x7f08800b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.059+0000 7f0885b95640 1 -- 172.21.15.88:0/4011017081 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08800cd2d0 con 0x7f0880072530 2024-09-16T01:39:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.060+0000 7f0885b95640 1 --2- 172.21.15.88:0/4011017081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0880072530 0x7f0880070fb0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f08800694d0 tx=0x7f087802c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709d56e610f48c3e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.060+0000 7f0877fff640 1 -- 172.21.15.88:0/4011017081 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0878039070 con 0x7f0880072530 2024-09-16T01:39:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.060+0000 7f0877fff640 1 -- 172.21.15.88:0/4011017081 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0878009600 con 0x7f0880072530 2024-09-16T01:39:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.060+0000 7f0877fff640 1 -- 172.21.15.88:0/4011017081 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08780099f0 con 0x7f0880072530 2024-09-16T01:39:57.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.061+0000 7f08875f8640 1 -- 172.21.15.88:0/4011017081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0880072530 msgr2=0x7f0880070fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:57.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.061+0000 7f08875f8640 1 --2- 172.21.15.88:0/4011017081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0880072530 0x7f0880070fb0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f08800694d0 tx=0x7f087802c810 comp rx=0 tx=0).stop 2024-09-16T01:39:57.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.061+0000 7f08875f8640 1 -- 172.21.15.88:0/4011017081 shutdown_connections 2024-09-16T01:39:57.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.061+0000 7f08875f8640 1 --2- 172.21.15.88:0/4011017081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0880072530 0x7f0880070fb0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.061+0000 7f08875f8640 1 --2- 172.21.15.88:0/4011017081 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08800b60c0 0x7f0880071ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.061+0000 7f08875f8640 1 --2- 172.21.15.88:0/4011017081 >> v2:172.21.15.154:3300/0 conn(0x7f088015b480 0x7f08800b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.061+0000 7f08875f8640 1 -- 172.21.15.88:0/4011017081 >> 172.21.15.88:0/4011017081 conn(0x7f088006d0f0 msgr2=0x7f088006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:57.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.061+0000 7f08875f8640 1 -- 172.21.15.88:0/4011017081 shutdown_connections 2024-09-16T01:39:57.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.061+0000 7f08875f8640 1 -- 172.21.15.88:0/4011017081 wait complete. 2024-09-16T01:39:57.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.062+0000 7f08875f8640 1 Processor -- start 2024-09-16T01:39:57.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.062+0000 7f08875f8640 1 -- start start 2024-09-16T01:39:57.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.063+0000 7f08875f8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0880072530 0x7f08801507d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:57.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.063+0000 7f08875f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08800b60c0 0x7f0880153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:57.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.063+0000 7f08875f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f088015b480 0x7f0880153640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:57.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.063+0000 7f08875f8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f08800cdb70 con 0x7f0880072530 2024-09-16T01:39:57.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.063+0000 7f08875f8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f08800cd890 con 0x7f08800b60c0 2024-09-16T01:39:57.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.063+0000 7f08875f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f088015b860 con 0x7f088015b480 2024-09-16T01:39:57.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.063+0000 7f0885394640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0880072530 0x7f08801507d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:57.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.063+0000 7f0885394640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0880072530 0x7f08801507d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60248/0 (socket says 172.21.15.88:60248) 2024-09-16T01:39:57.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.063+0000 7f0885394640 1 -- 172.21.15.88:0/3992368110 learned_addr learned my addr 172.21.15.88:0/3992368110 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:57.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.063+0000 7f0885394640 1 -- 172.21.15.88:0/3992368110 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f088015b480 msgr2=0x7f0880153640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:57.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.063+0000 7f0885394640 1 --2- 172.21.15.88:0/3992368110 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f088015b480 0x7f0880153640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.063+0000 7f0885394640 1 -- 172.21.15.88:0/3992368110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08800b60c0 msgr2=0x7f0880153100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:57.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.063+0000 7f0885394640 1 --2- 172.21.15.88:0/3992368110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08800b60c0 0x7f0880153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.063+0000 7f0885394640 1 -- 172.21.15.88:0/3992368110 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08801549b0 con 0x7f0880072530 2024-09-16T01:39:57.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.063+0000 7f0885394640 1 --2- 172.21.15.88:0/3992368110 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0880072530 0x7f08801507d0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f087c00b410 tx=0x7f087c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:57.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.063+0000 7f0875ffb640 1 -- 172.21.15.88:0/3992368110 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c015020 con 0x7f0880072530 2024-09-16T01:39:57.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.063+0000 7f0875ffb640 1 -- 172.21.15.88:0/3992368110 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f087c011850 con 0x7f0880072530 2024-09-16T01:39:57.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.063+0000 7f0875ffb640 1 -- 172.21.15.88:0/3992368110 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c011c40 con 0x7f0880072530 2024-09-16T01:39:57.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.063+0000 7f08875f8640 1 -- 172.21.15.88:0/3992368110 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08801e53e0 con 0x7f0880072530 2024-09-16T01:39:57.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.064+0000 7f08875f8640 1 -- 172.21.15.88:0/3992368110 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f088015c150 con 0x7f0880072530 2024-09-16T01:39:57.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.065+0000 7f08875f8640 1 -- 172.21.15.88:0/3992368110 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0880071ff0 con 0x7f0880072530 2024-09-16T01:39:57.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.065+0000 7f0875ffb640 1 -- 172.21.15.88:0/3992368110 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f087c018020 con 0x7f0880072530 2024-09-16T01:39:57.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.065+0000 7f0875ffb640 1 --2- 172.21.15.88:0/3992368110 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f086406c2b0 0x7f086406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:57.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.066+0000 7f0875ffb640 1 -- 172.21.15.88:0/3992368110 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5029+0+0 (secure 0 0 0) 0x7f087c08b0b0 con 0x7f0880072530 2024-09-16T01:39:57.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.066+0000 7f0884b93640 1 --2- 172.21.15.88:0/3992368110 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f086406c2b0 0x7f086406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:57.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.066+0000 7f0884b93640 1 --2- 172.21.15.88:0/3992368110 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f086406c2b0 0x7f086406e770 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f088015e190 tx=0x7f0870006c90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:57.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.070+0000 7f0875ffb640 1 -- 172.21.15.88:0/3992368110 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f087c01f020 con 0x7f0880072530 2024-09-16T01:39:57.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.234+0000 7f08875f8640 1 -- 172.21.15.88:0/3992368110 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f0880063360 con 0x7f0880072530 2024-09-16T01:39:57.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.237+0000 7f0875ffb640 1 -- 172.21.15.88:0/3992368110 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) ==== 104+0+0 (secure 0 0 0) 0x7f087c0172c0 con 0x7f0880072530 2024-09-16T01:39:57.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.239+0000 7f08875f8640 1 -- 172.21.15.88:0/3992368110 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f086406c2b0 msgr2=0x7f086406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:57.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.239+0000 7f08875f8640 1 --2- 172.21.15.88:0/3992368110 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f086406c2b0 0x7f086406e770 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f088015e190 tx=0x7f0870006c90 comp rx=0 tx=0).stop 2024-09-16T01:39:57.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.239+0000 7f08875f8640 1 -- 172.21.15.88:0/3992368110 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0880072530 msgr2=0x7f08801507d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:57.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.239+0000 7f08875f8640 1 --2- 172.21.15.88:0/3992368110 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0880072530 0x7f08801507d0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f087c00b410 tx=0x7f087c00b8d0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.239+0000 7f08875f8640 1 -- 172.21.15.88:0/3992368110 shutdown_connections 2024-09-16T01:39:57.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.239+0000 7f08875f8640 1 --2- 172.21.15.88:0/3992368110 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f086406c2b0 0x7f086406e770 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.239+0000 7f08875f8640 1 --2- 172.21.15.88:0/3992368110 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f088015b480 0x7f0880153640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.239+0000 7f08875f8640 1 --2- 172.21.15.88:0/3992368110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08800b60c0 0x7f0880153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.239+0000 7f08875f8640 1 --2- 172.21.15.88:0/3992368110 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0880072530 0x7f08801507d0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.239+0000 7f08875f8640 1 -- 172.21.15.88:0/3992368110 >> 172.21.15.88:0/3992368110 conn(0x7f088006d0f0 msgr2=0x7f08800b4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:57.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.239+0000 7f08875f8640 1 -- 172.21.15.88:0/3992368110 shutdown_connections 2024-09-16T01:39:57.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.239+0000 7f08875f8640 1 -- 172.21.15.88:0/3992368110 wait complete. 2024-09-16T01:39:57.253 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_inject_rename_corrupt_dentry_first 2024-09-16T01:39:57.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.353+0000 7f195c1cf640 1 Processor -- start 2024-09-16T01:39:57.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.353+0000 7f195c1cf640 1 -- start start 2024-09-16T01:39:57.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.354+0000 7f195c1cf640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f195415c2a0 0x7f19540735d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:57.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.354+0000 7f195c1cf640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f195406fe20 0x7f195406bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:57.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.354+0000 7f195c1cf640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1954074120 con 0x7f1954073b10 2024-09-16T01:39:57.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.354+0000 7f195c1cf640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f195415c680 con 0x7f195415c2a0 2024-09-16T01:39:57.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.354+0000 7f195c1cf640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f195415ef00 con 0x7f195406fe20 2024-09-16T01:39:57.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.354+0000 7f1959f6b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f195415c2a0 0x7f19540735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:57.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.354+0000 7f1959f6b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f195415c2a0 0x7f19540735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60254/0 (socket says 172.21.15.88:60254) 2024-09-16T01:39:57.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.354+0000 7f195976a640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1954073b10 0x7f195406f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:56000/0 (socket says 172.21.15.88:56000) 2024-09-16T01:39:57.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.354+0000 7f1959f6b640 1 -- 172.21.15.88:0/3376708972 learned_addr learned my addr 172.21.15.88:0/3376708972 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:57.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.354+0000 7f195a76c640 1 --2- 172.21.15.88:0/3376708972 >> v2:172.21.15.154:3300/0 conn(0x7f195406fe20 0x7f195406bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:57.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.354+0000 7f1959f6b640 1 -- 172.21.15.88:0/3376708972 >> v2:172.21.15.154:3300/0 conn(0x7f195406fe20 msgr2=0x7f195406bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:57.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.354+0000 7f1959f6b640 1 --2- 172.21.15.88:0/3376708972 >> v2:172.21.15.154:3300/0 conn(0x7f195406fe20 0x7f195406bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.354+0000 7f1959f6b640 1 -- 172.21.15.88:0/3376708972 >> v1:172.21.15.88:6789/0 conn(0x7f1954073b10 legacy=0x7f195406f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:57.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.354+0000 7f1959f6b640 1 -- 172.21.15.88:0/3376708972 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f195415f1e0 con 0x7f195415c2a0 2024-09-16T01:39:57.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.354+0000 7f1958f69640 1 -- 172.21.15.88:0/3376708972 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2606623045 0 0) 0x7f1954074120 con 0x7f1954073b10 2024-09-16T01:39:57.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.355+0000 7f1959f6b640 1 --2- 172.21.15.88:0/3376708972 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f195415c2a0 0x7f19540735d0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f19540705f0 tx=0x7f195002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7df57a3106807d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:57.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.355+0000 7f1958f69640 1 -- 172.21.15.88:0/3376708972 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1950037070 con 0x7f195415c2a0 2024-09-16T01:39:57.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.355+0000 7f1958f69640 1 -- 172.21.15.88:0/3376708972 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1950032440 con 0x7f195415c2a0 2024-09-16T01:39:57.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.355+0000 7f1958f69640 1 -- 172.21.15.88:0/3376708972 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1950032810 con 0x7f195415c2a0 2024-09-16T01:39:57.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.355+0000 7f195c1cf640 1 -- 172.21.15.88:0/3376708972 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f195415c2a0 msgr2=0x7f19540735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:57.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.355+0000 7f195c1cf640 1 --2- 172.21.15.88:0/3376708972 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f195415c2a0 0x7f19540735d0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f19540705f0 tx=0x7f195002a270 comp rx=0 tx=0).stop 2024-09-16T01:39:57.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.355+0000 7f195c1cf640 1 -- 172.21.15.88:0/3376708972 shutdown_connections 2024-09-16T01:39:57.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.356+0000 7f195c1cf640 1 --2- 172.21.15.88:0/3376708972 >> v2:172.21.15.154:3300/0 conn(0x7f195406fe20 0x7f195406bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.356+0000 7f195c1cf640 1 --2- 172.21.15.88:0/3376708972 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f195415c2a0 0x7f19540735d0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.356+0000 7f195c1cf640 1 -- 172.21.15.88:0/3376708972 >> 172.21.15.88:0/3376708972 conn(0x7f19540b3070 msgr2=0x7f19540b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:57.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.356+0000 7f195c1cf640 1 -- 172.21.15.88:0/3376708972 shutdown_connections 2024-09-16T01:39:57.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.356+0000 7f195c1cf640 1 -- 172.21.15.88:0/3376708972 wait complete. 2024-09-16T01:39:57.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.357+0000 7f195c1cf640 1 Processor -- start 2024-09-16T01:39:57.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.357+0000 7f195c1cf640 1 -- start start 2024-09-16T01:39:57.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.357+0000 7f195c1cf640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f195406fe20 0x7f1954153080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:57.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.357+0000 7f195c1cf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1954073b10 0x7f19540cf290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:57.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.357+0000 7f195c1cf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f195415c2a0 0x7f1954157d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:57.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.357+0000 7f195c1cf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f19540b5f20 con 0x7f1954073b10 2024-09-16T01:39:57.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.357+0000 7f195c1cf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f195415ebf0 con 0x7f195415c2a0 2024-09-16T01:39:57.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.357+0000 7f195c1cf640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f19540b60a0 con 0x7f195406fe20 2024-09-16T01:39:57.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.358+0000 7f195976a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1954073b10 0x7f19540cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:57.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.358+0000 7f195976a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1954073b10 0x7f19540cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60258/0 (socket says 172.21.15.88:60258) 2024-09-16T01:39:57.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.358+0000 7f195976a640 1 -- 172.21.15.88:0/2557365714 learned_addr learned my addr 172.21.15.88:0/2557365714 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:57.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.358+0000 7f1959f6b640 1 --2- 172.21.15.88:0/2557365714 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f195406fe20 0x7f1954153080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:57.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.358+0000 7f195a76c640 1 --2- 172.21.15.88:0/2557365714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f195415c2a0 0x7f1954157d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:57.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.358+0000 7f195976a640 1 -- 172.21.15.88:0/2557365714 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f195406fe20 msgr2=0x7f1954153080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:57.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.358+0000 7f195976a640 1 --2- 172.21.15.88:0/2557365714 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f195406fe20 0x7f1954153080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.358+0000 7f195976a640 1 -- 172.21.15.88:0/2557365714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f195415c2a0 msgr2=0x7f1954157d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:57.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.358+0000 7f195976a640 1 --2- 172.21.15.88:0/2557365714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f195415c2a0 0x7f1954157d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.358+0000 7f195976a640 1 -- 172.21.15.88:0/2557365714 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f195415a380 con 0x7f1954073b10 2024-09-16T01:39:57.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.358+0000 7f195976a640 1 --2- 172.21.15.88:0/2557365714 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1954073b10 0x7f19540cf290 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f194400c500 tx=0x7f194400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:57.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.358+0000 7f194a7fc640 1 -- 172.21.15.88:0/2557365714 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1944018070 con 0x7f1954073b10 2024-09-16T01:39:57.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.358+0000 7f194a7fc640 1 -- 172.21.15.88:0/2557365714 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1944007580 con 0x7f1954073b10 2024-09-16T01:39:57.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.358+0000 7f194a7fc640 1 -- 172.21.15.88:0/2557365714 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1944004a20 con 0x7f1954073b10 2024-09-16T01:39:57.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.358+0000 7f195c1cf640 1 -- 172.21.15.88:0/2557365714 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f195415fd90 con 0x7f1954073b10 2024-09-16T01:39:57.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.358+0000 7f195c1cf640 1 -- 172.21.15.88:0/2557365714 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19540cf7d0 con 0x7f1954073b10 2024-09-16T01:39:57.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.359+0000 7f195c1cf640 1 -- 172.21.15.88:0/2557365714 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19540d0e10 con 0x7f1954073b10 2024-09-16T01:39:57.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.360+0000 7f194a7fc640 1 -- 172.21.15.88:0/2557365714 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f194401b070 con 0x7f1954073b10 2024-09-16T01:39:57.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.360+0000 7f194a7fc640 1 --2- 172.21.15.88:0/2557365714 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f193806c2b0 0x7f193806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:57.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.360+0000 7f194a7fc640 1 -- 172.21.15.88:0/2557365714 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5029+0+0 (secure 0 0 0) 0x7f194408a3c0 con 0x7f1954073b10 2024-09-16T01:39:57.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.360+0000 7f1959f6b640 1 --2- 172.21.15.88:0/2557365714 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f193806c2b0 0x7f193806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:57.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.361+0000 7f1959f6b640 1 --2- 172.21.15.88:0/2557365714 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f193806c2b0 0x7f193806e770 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1950034000 tx=0x7f195002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:57.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.364+0000 7f194a7fc640 1 -- 172.21.15.88:0/2557365714 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1944058810 con 0x7f1954073b10 2024-09-16T01:39:57.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.531+0000 7f195c1cf640 1 -- 172.21.15.88:0/2557365714 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"} v 0) -- 0x7f19540b2a90 con 0x7f1954073b10 2024-09-16T01:39:57.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.531+0000 7f194a7fc640 1 -- 172.21.15.88:0/2557365714 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"}]=0 v25) ==== 121+0+0 (secure 0 0 0) 0x7f194405cb10 con 0x7f1954073b10 2024-09-16T01:39:57.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.533+0000 7f195c1cf640 1 -- 172.21.15.88:0/2557365714 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f193806c2b0 msgr2=0x7f193806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:57.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.533+0000 7f195c1cf640 1 --2- 172.21.15.88:0/2557365714 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f193806c2b0 0x7f193806e770 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1950034000 tx=0x7f195002e000 comp rx=0 tx=0).stop 2024-09-16T01:39:57.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.533+0000 7f195c1cf640 1 -- 172.21.15.88:0/2557365714 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1954073b10 msgr2=0x7f19540cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:57.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.533+0000 7f195c1cf640 1 --2- 172.21.15.88:0/2557365714 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1954073b10 0x7f19540cf290 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f194400c500 tx=0x7f194400c9c0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.533+0000 7f195c1cf640 1 -- 172.21.15.88:0/2557365714 shutdown_connections 2024-09-16T01:39:57.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.533+0000 7f195c1cf640 1 --2- 172.21.15.88:0/2557365714 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f193806c2b0 0x7f193806e770 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.533+0000 7f195c1cf640 1 --2- 172.21.15.88:0/2557365714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f195415c2a0 0x7f1954157d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.533+0000 7f195c1cf640 1 --2- 172.21.15.88:0/2557365714 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1954073b10 0x7f19540cf290 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.533+0000 7f195c1cf640 1 --2- 172.21.15.88:0/2557365714 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f195406fe20 0x7f1954153080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.533+0000 7f195c1cf640 1 -- 172.21.15.88:0/2557365714 >> 172.21.15.88:0/2557365714 conn(0x7f19540b3070 msgr2=0x7f19540b1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:57.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.533+0000 7f195c1cf640 1 -- 172.21.15.88:0/2557365714 shutdown_connections 2024-09-16T01:39:57.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.533+0000 7f195c1cf640 1 -- 172.21.15.88:0/2557365714 wait complete. 2024-09-16T01:39:57.545 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_preflush' 2024-09-16T01:39:57.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.646+0000 7f322e077640 1 Processor -- start 2024-09-16T01:39:57.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.646+0000 7f322e077640 1 -- start start 2024-09-16T01:39:57.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.647+0000 7f322e077640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f32280cfc80 0x7f32280b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:57.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.647+0000 7f322e077640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f322815c330 0x7f32280b4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:57.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.647+0000 7f322e077640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f322815c740 con 0x7f3228155800 2024-09-16T01:39:57.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.647+0000 7f322e077640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f32280b4e60 con 0x7f32280cfc80 2024-09-16T01:39:57.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.647+0000 7f322e077640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f32280b50a0 con 0x7f322815c330 2024-09-16T01:39:57.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.647+0000 7f3226ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f32280cfc80 0x7f32280b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:57.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.647+0000 7f3226ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f32280cfc80 0x7f32280b6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60266/0 (socket says 172.21.15.88:60266) 2024-09-16T01:39:57.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.647+0000 7f32277fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3228155800 0x7f32280cf570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:56008/0 (socket says 172.21.15.88:56008) 2024-09-16T01:39:57.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.647+0000 7f3226ffd640 1 -- 172.21.15.88:0/3348262897 learned_addr learned my addr 172.21.15.88:0/3348262897 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:57.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.647+0000 7f3227fff640 1 --2- 172.21.15.88:0/3348262897 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f322815c330 0x7f32280b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:57.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.647+0000 7f3226ffd640 1 -- 172.21.15.88:0/3348262897 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f322815c330 msgr2=0x7f32280b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:57.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.647+0000 7f3226ffd640 1 --2- 172.21.15.88:0/3348262897 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f322815c330 0x7f32280b4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.647+0000 7f3226ffd640 1 -- 172.21.15.88:0/3348262897 >> v1:172.21.15.88:6789/0 conn(0x7f3228155800 legacy=0x7f32280cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:57.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.647+0000 7f32267fc640 1 -- 172.21.15.88:0/3348262897 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 819204161 0 0) 0x7f322815c740 con 0x7f3228155800 2024-09-16T01:39:57.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.647+0000 7f3226ffd640 1 -- 172.21.15.88:0/3348262897 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32280b5380 con 0x7f32280cfc80 2024-09-16T01:39:57.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.647+0000 7f3227fff640 1 --2- 172.21.15.88:0/3348262897 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f322815c330 0x7f32280b4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:39:57.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.648+0000 7f3226ffd640 1 --2- 172.21.15.88:0/3348262897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32280cfc80 0x7f32280b6140 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f321c009870 tx=0x7f321c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e5c63866da4f5f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:57.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.648+0000 7f32267fc640 1 -- 172.21.15.88:0/3348262897 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c037070 con 0x7f32280cfc80 2024-09-16T01:39:57.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.648+0000 7f32267fc640 1 -- 172.21.15.88:0/3348262897 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f321c032440 con 0x7f32280cfc80 2024-09-16T01:39:57.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.648+0000 7f32267fc640 1 -- 172.21.15.88:0/3348262897 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c032810 con 0x7f32280cfc80 2024-09-16T01:39:57.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.648+0000 7f322e077640 1 -- 172.21.15.88:0/3348262897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32280cfc80 msgr2=0x7f32280b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:57.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.648+0000 7f322e077640 1 --2- 172.21.15.88:0/3348262897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32280cfc80 0x7f32280b6140 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f321c009870 tx=0x7f321c02a270 comp rx=0 tx=0).stop 2024-09-16T01:39:57.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.649+0000 7f322e077640 1 -- 172.21.15.88:0/3348262897 shutdown_connections 2024-09-16T01:39:57.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.649+0000 7f322e077640 1 --2- 172.21.15.88:0/3348262897 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f322815c330 0x7f32280b4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.649+0000 7f322e077640 1 --2- 172.21.15.88:0/3348262897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32280cfc80 0x7f32280b6140 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.649+0000 7f322e077640 1 -- 172.21.15.88:0/3348262897 >> 172.21.15.88:0/3348262897 conn(0x7f322806bc10 msgr2=0x7f322806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:57.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.649+0000 7f322e077640 1 -- 172.21.15.88:0/3348262897 shutdown_connections 2024-09-16T01:39:57.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.649+0000 7f322e077640 1 -- 172.21.15.88:0/3348262897 wait complete. 2024-09-16T01:39:57.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.650+0000 7f322e077640 1 Processor -- start 2024-09-16T01:39:57.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.650+0000 7f322e077640 1 -- start start 2024-09-16T01:39:57.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.651+0000 7f322e077640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32280cfc80 0x7f3228142e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:57.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.651+0000 7f322e077640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3228155800 0x7f3228157e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:57.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.651+0000 7f322e077640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f322815c330 0x7f3228154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:57.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.651+0000 7f322e077640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f32280b5c20 con 0x7f3228155800 2024-09-16T01:39:57.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.651+0000 7f322e077640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f32280b5940 con 0x7f322815c330 2024-09-16T01:39:57.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.651+0000 7f322e077640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3228155be0 con 0x7f32280cfc80 2024-09-16T01:39:57.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.651+0000 7f3226ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3228155800 0x7f3228157e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:57.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.651+0000 7f3226ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3228155800 0x7f3228157e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60270/0 (socket says 172.21.15.88:60270) 2024-09-16T01:39:57.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.651+0000 7f3226ffd640 1 -- 172.21.15.88:0/4277089920 learned_addr learned my addr 172.21.15.88:0/4277089920 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:57.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.651+0000 7f32277fe640 1 --2- 172.21.15.88:0/4277089920 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32280cfc80 0x7f3228142e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:57.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.651+0000 7f3226ffd640 1 -- 172.21.15.88:0/4277089920 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32280cfc80 msgr2=0x7f3228142e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:57.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.651+0000 7f3226ffd640 1 --2- 172.21.15.88:0/4277089920 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32280cfc80 0x7f3228142e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.651+0000 7f3226ffd640 1 -- 172.21.15.88:0/4277089920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f322815c330 msgr2=0x7f3228154350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:57.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.651+0000 7f3227fff640 1 --2- 172.21.15.88:0/4277089920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f322815c330 0x7f3228154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:57.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.651+0000 7f3226ffd640 1 --2- 172.21.15.88:0/4277089920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f322815c330 0x7f3228154350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.651+0000 7f3226ffd640 1 -- 172.21.15.88:0/4277089920 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3228155fc0 con 0x7f3228155800 2024-09-16T01:39:57.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.651+0000 7f3226ffd640 1 --2- 172.21.15.88:0/4277089920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3228155800 0x7f3228157e70 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f321c035040 tx=0x7f321c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:57.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.651+0000 7f320ffff640 1 -- 172.21.15.88:0/4277089920 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c004030 con 0x7f3228155800 2024-09-16T01:39:57.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.651+0000 7f320ffff640 1 -- 172.21.15.88:0/4277089920 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f321c0041d0 con 0x7f3228155800 2024-09-16T01:39:57.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.651+0000 7f322e077640 1 -- 172.21.15.88:0/4277089920 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32281562a0 con 0x7f3228155800 2024-09-16T01:39:57.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.651+0000 7f320ffff640 1 -- 172.21.15.88:0/4277089920 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c003610 con 0x7f3228155800 2024-09-16T01:39:57.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.651+0000 7f322e077640 1 -- 172.21.15.88:0/4277089920 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3228156860 con 0x7f3228155800 2024-09-16T01:39:57.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.652+0000 7f322e077640 1 -- 172.21.15.88:0/4277089920 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32280b6140 con 0x7f3228155800 2024-09-16T01:39:57.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.653+0000 7f320ffff640 1 -- 172.21.15.88:0/4277089920 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f321c04a060 con 0x7f3228155800 2024-09-16T01:39:57.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.653+0000 7f320ffff640 1 --2- 172.21.15.88:0/4277089920 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f321406c2b0 0x7f321406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:57.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.653+0000 7f320ffff640 1 -- 172.21.15.88:0/4277089920 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5029+0+0 (secure 0 0 0) 0x7f321c0acef0 con 0x7f3228155800 2024-09-16T01:39:57.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.653+0000 7f32277fe640 1 --2- 172.21.15.88:0/4277089920 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f321406c2b0 0x7f321406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:57.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.654+0000 7f32277fe640 1 --2- 172.21.15.88:0/4277089920 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f321406c2b0 0x7f321406e770 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3218007fd0 tx=0x7f3218009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:57.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.658+0000 7f320ffff640 1 -- 172.21.15.88:0/4277089920 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f321c03e070 con 0x7f3228155800 2024-09-16T01:39:57.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.886+0000 7f322e077640 1 -- 172.21.15.88:0/4277089920 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_preflush"]} v 0) -- 0x7f3228157780 con 0x7f3228155800 2024-09-16T01:39:57.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.940+0000 7f320ffff640 1 -- 172.21.15.88:0/4277089920 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_preflush"]}]=0 v274) ==== 141+0+0 (secure 0 0 0) 0x7f321c07b990 con 0x7f3228155800 2024-09-16T01:39:57.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.942+0000 7f322e077640 1 -- 172.21.15.88:0/4277089920 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f321406c2b0 msgr2=0x7f321406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:57.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.942+0000 7f322e077640 1 --2- 172.21.15.88:0/4277089920 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f321406c2b0 0x7f321406e770 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3218007fd0 tx=0x7f3218009040 comp rx=0 tx=0).stop 2024-09-16T01:39:57.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.942+0000 7f322e077640 1 -- 172.21.15.88:0/4277089920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3228155800 msgr2=0x7f3228157e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:57.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.942+0000 7f322e077640 1 --2- 172.21.15.88:0/4277089920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3228155800 0x7f3228157e70 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f321c035040 tx=0x7f321c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.942+0000 7f322e077640 1 -- 172.21.15.88:0/4277089920 shutdown_connections 2024-09-16T01:39:57.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.942+0000 7f322e077640 1 --2- 172.21.15.88:0/4277089920 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f321406c2b0 0x7f321406e770 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.942+0000 7f322e077640 1 --2- 172.21.15.88:0/4277089920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f322815c330 0x7f3228154350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.942+0000 7f322e077640 1 --2- 172.21.15.88:0/4277089920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3228155800 0x7f3228157e70 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.942+0000 7f322e077640 1 --2- 172.21.15.88:0/4277089920 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32280cfc80 0x7f3228142e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:57.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.942+0000 7f322e077640 1 -- 172.21.15.88:0/4277089920 >> 172.21.15.88:0/4277089920 conn(0x7f322806bc10 msgr2=0x7f32280cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:57.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.942+0000 7f322e077640 1 -- 172.21.15.88:0/4277089920 shutdown_connections 2024-09-16T01:39:57.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:57.942+0000 7f322e077640 1 -- 172.21.15.88:0/4277089920 wait complete. 2024-09-16T01:39:57.954 INFO:tasks.cephfs_test_runner:That the MDS won't write a dentry with new damage to CDentry::first ... ok 2024-09-16T01:39:57.954 INFO:tasks.cephfs_test_runner:Starting test: test_object_deletion (tasks.cephfs.test_damage.TestDamage) That the MDS has a clean 'damaged' response to loss of any single metadata object 2024-09-16T01:39:57.954 INFO:tasks.cephfs_test_runner:test_object_deletion (tasks.cephfs.test_damage.TestDamage) 2024-09-16T01:39:57.955 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:39:58.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.052+0000 7f1bfa7f6640 1 Processor -- start 2024-09-16T01:39:58.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.052+0000 7f1bfa7f6640 1 -- start start 2024-09-16T01:39:58.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.053+0000 7f1bfa7f6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1bf415e950 0x7f1bf406ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:58.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.053+0000 7f1bfa7f6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bf406ca30 con 0x7f1bf40735b0 2024-09-16T01:39:58.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.053+0000 7f1bfa7f6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bf406faf0 con 0x7f1bf4072050 2024-09-16T01:39:58.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.053+0000 7f1bfa7f6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f1bf406fdd0 con 0x7f1bf415e950 2024-09-16T01:39:58.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.053+0000 7f1bf37fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1bf40735b0 0x7f1bf406bc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:56020/0 (socket says 172.21.15.88:56020) 2024-09-16T01:39:58.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.053+0000 7f1bf37fe640 1 -- 172.21.15.88:0/243690031 learned_addr learned my addr 172.21.15.88:0/243690031 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:58.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.053+0000 7f1bf2ffd640 1 -- 172.21.15.88:0/243690031 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3991579735 0 0) 0x7f1bf406ca30 con 0x7f1bf40735b0 2024-09-16T01:39:58.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.053+0000 7f1bf2ffd640 1 -- 172.21.15.88:0/243690031 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bdc003520 con 0x7f1bf40735b0 2024-09-16T01:39:58.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.053+0000 7f1bf3fff640 1 --2- 172.21.15.88:0/243690031 >> v2:172.21.15.154:3300/0 conn(0x7f1bf415e950 0x7f1bf406ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:58.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.053+0000 7f1bf2ffd640 1 -- 172.21.15.88:0/243690031 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3200373278 0 0) 0x7f1bdc003520 con 0x7f1bf40735b0 2024-09-16T01:39:58.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.053+0000 7f1bf2ffd640 1 -- 172.21.15.88:0/243690031 >> v2:172.21.15.154:3300/0 conn(0x7f1bf415e950 msgr2=0x7f1bf406ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:58.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.053+0000 7f1bf2ffd640 1 --2- 172.21.15.88:0/243690031 >> v2:172.21.15.154:3300/0 conn(0x7f1bf415e950 0x7f1bf406ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.053+0000 7f1bf2ffd640 1 -- 172.21.15.88:0/243690031 >> v1:172.21.15.154:6789/0 conn(0x7f1bf4072050 legacy=0x7f1bf406c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:58.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.054+0000 7f1bf2ffd640 1 -- 172.21.15.88:0/243690031 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf40700b0 con 0x7f1bf40735b0 2024-09-16T01:39:58.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.054+0000 7f1bf2ffd640 1 -- 172.21.15.88:0/243690031 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f1be8001670 con 0x7f1bf40735b0 2024-09-16T01:39:58.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.054+0000 7f1bf2ffd640 1 -- 172.21.15.88:0/243690031 >> v1:172.21.15.88:6789/0 conn(0x7f1bf40735b0 legacy=0x7f1bf406bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:58.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.054+0000 7f1bf2ffd640 1 --2- 172.21.15.88:0/243690031 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bdc0039b0 0x7f1bdc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:58.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.054+0000 7f1bf2ffd640 1 --2- 172.21.15.88:0/243690031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bdc004800 0x7f1bdc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:58.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.054+0000 7f1bf2ffd640 1 --2- 172.21.15.88:0/243690031 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bdc0051c0 0x7f1bdc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:58.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.054+0000 7f1bf2ffd640 1 -- 172.21.15.88:0/243690031 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf406fdd0 con 0x7f1bdc004800 2024-09-16T01:39:58.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.054+0000 7f1bf2ffd640 1 -- 172.21.15.88:0/243690031 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1bdc00c110 con 0x7f1bdc0039b0 2024-09-16T01:39:58.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.054+0000 7f1bf2ffd640 1 -- 172.21.15.88:0/243690031 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1bdc00c310 con 0x7f1bdc0051c0 2024-09-16T01:39:58.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.054+0000 7f1bf37fe640 1 --2- 172.21.15.88:0/243690031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bdc004800 0x7f1bdc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:58.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.054+0000 7f1bf3fff640 1 --2- 172.21.15.88:0/243690031 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bdc0039b0 0x7f1bdc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:58.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.054+0000 7f1bf8d93640 1 --2- 172.21.15.88:0/243690031 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bdc0051c0 0x7f1bdc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:58.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.054+0000 7f1bf37fe640 1 -- 172.21.15.88:0/243690031 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bdc0051c0 msgr2=0x7f1bdc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:58.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.054+0000 7f1bf37fe640 1 --2- 172.21.15.88:0/243690031 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bdc0051c0 0x7f1bdc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.054+0000 7f1bf37fe640 1 -- 172.21.15.88:0/243690031 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bdc0039b0 msgr2=0x7f1bdc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:58.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.054+0000 7f1bf37fe640 1 --2- 172.21.15.88:0/243690031 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bdc0039b0 0x7f1bdc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.054+0000 7f1bf37fe640 1 -- 172.21.15.88:0/243690031 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bdc003520 con 0x7f1bdc004800 2024-09-16T01:39:58.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.054+0000 7f1bf8d93640 1 -- 172.21.15.88:0/243690031 reap_dead start 2024-09-16T01:39:58.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.055+0000 7f1bf37fe640 1 --2- 172.21.15.88:0/243690031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bdc004800 0x7f1bdc004c80 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f1be800c210 tx=0x7f1bdc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae4be6b439018c01 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:58.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.055+0000 7f1bf2ffd640 1 -- 172.21.15.88:0/243690031 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be8007d10 con 0x7f1bdc004800 2024-09-16T01:39:58.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.055+0000 7f1bf2ffd640 1 -- 172.21.15.88:0/243690031 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1be8005ad0 con 0x7f1bdc004800 2024-09-16T01:39:58.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.055+0000 7f1bf2ffd640 1 -- 172.21.15.88:0/243690031 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be8007070 con 0x7f1bdc004800 2024-09-16T01:39:58.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.055+0000 7f1bfa7f6640 1 -- 172.21.15.88:0/243690031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bdc004800 msgr2=0x7f1bdc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:58.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.055+0000 7f1bfa7f6640 1 --2- 172.21.15.88:0/243690031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bdc004800 0x7f1bdc004c80 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f1be800c210 tx=0x7f1bdc005830 comp rx=0 tx=0).stop 2024-09-16T01:39:58.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.055+0000 7f1bfa7f6640 1 -- 172.21.15.88:0/243690031 shutdown_connections 2024-09-16T01:39:58.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.055+0000 7f1bfa7f6640 1 --2- 172.21.15.88:0/243690031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bdc004800 0x7f1bdc004c80 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.055+0000 7f1bfa7f6640 1 -- 172.21.15.88:0/243690031 >> 172.21.15.88:0/243690031 conn(0x7f1bf40d14d0 msgr2=0x7f1bf40d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:58.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.055+0000 7f1bfa7f6640 1 -- 172.21.15.88:0/243690031 shutdown_connections 2024-09-16T01:39:58.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.055+0000 7f1bfa7f6640 1 -- 172.21.15.88:0/243690031 wait complete. 2024-09-16T01:39:58.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.057+0000 7f1bfa7f6640 1 Processor -- start 2024-09-16T01:39:58.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.057+0000 7f1bfa7f6640 1 -- start start 2024-09-16T01:39:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.057+0000 7f1bfa7f6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bdc004800 0x7f1bf40cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.057+0000 7f1bfa7f6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bf41f4ca0 0x7f1bf4157690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.058+0000 7f1bfa7f6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bf415a030 0x7f1bf415a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.058+0000 7f1bfa7f6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf406f8b0 con 0x7f1bdc004800 2024-09-16T01:39:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.058+0000 7f1bfa7f6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf406f730 con 0x7f1bf41f4ca0 2024-09-16T01:39:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.058+0000 7f1bfa7f6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1bf406ce30 con 0x7f1bf415a030 2024-09-16T01:39:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.058+0000 7f1bf3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bdc004800 0x7f1bf40cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.058+0000 7f1bf3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bdc004800 0x7f1bf40cdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60288/0 (socket says 172.21.15.88:60288) 2024-09-16T01:39:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.058+0000 7f1bf3fff640 1 -- 172.21.15.88:0/1837903915 learned_addr learned my addr 172.21.15.88:0/1837903915 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.058+0000 7f1bf37fe640 1 --2- 172.21.15.88:0/1837903915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bf41f4ca0 0x7f1bf4157690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.058+0000 7f1bf8d93640 1 --2- 172.21.15.88:0/1837903915 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bf415a030 0x7f1bf415a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.058+0000 7f1bf3fff640 1 -- 172.21.15.88:0/1837903915 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bf415a030 msgr2=0x7f1bf415a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.058+0000 7f1bf3fff640 1 --2- 172.21.15.88:0/1837903915 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bf415a030 0x7f1bf415a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.058+0000 7f1bf3fff640 1 -- 172.21.15.88:0/1837903915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bf41f4ca0 msgr2=0x7f1bf4157690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.058+0000 7f1bf3fff640 1 --2- 172.21.15.88:0/1837903915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bf41f4ca0 0x7f1bf4157690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.058+0000 7f1bf3fff640 1 -- 172.21.15.88:0/1837903915 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf41582a0 con 0x7f1bdc004800 2024-09-16T01:39:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.058+0000 7f1bf37fe640 1 --2- 172.21.15.88:0/1837903915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bf41f4ca0 0x7f1bf4157690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.058+0000 7f1bf3fff640 1 --2- 172.21.15.88:0/1837903915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bdc004800 0x7f1bf40cdb50 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f1be400b7f0 tx=0x7f1be400bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.058+0000 7f1bf0ff9640 1 -- 172.21.15.88:0/1837903915 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be4018070 con 0x7f1bdc004800 2024-09-16T01:39:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.058+0000 7f1bf0ff9640 1 -- 172.21.15.88:0/1837903915 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1be4010830 con 0x7f1bdc004800 2024-09-16T01:39:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.058+0000 7f1bfa7f6640 1 -- 172.21.15.88:0/1837903915 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bf4158580 con 0x7f1bdc004800 2024-09-16T01:39:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.058+0000 7f1bf0ff9640 1 -- 172.21.15.88:0/1837903915 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be4010c20 con 0x7f1bdc004800 2024-09-16T01:39:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.058+0000 7f1bfa7f6640 1 -- 172.21.15.88:0/1837903915 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bf4158b40 con 0x7f1bdc004800 2024-09-16T01:39:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.059+0000 7f1bfa7f6640 1 -- 172.21.15.88:0/1837903915 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bf406d1c0 con 0x7f1bdc004800 2024-09-16T01:39:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.060+0000 7f1bf0ff9640 1 -- 172.21.15.88:0/1837903915 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1be401b070 con 0x7f1bdc004800 2024-09-16T01:39:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.060+0000 7f1bf0ff9640 1 --2- 172.21.15.88:0/1837903915 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1be006c2b0 0x7f1be006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:58.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.060+0000 7f1bf0ff9640 1 -- 172.21.15.88:0/1837903915 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5029+0+0 (secure 0 0 0) 0x7f1be408b5b0 con 0x7f1bdc004800 2024-09-16T01:39:58.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.060+0000 7f1bf37fe640 1 --2- 172.21.15.88:0/1837903915 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1be006c2b0 0x7f1be006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:58.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.061+0000 7f1bf37fe640 1 --2- 172.21.15.88:0/1837903915 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1be006c2b0 0x7f1be006e770 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1be8032000 tx=0x7f1be8034040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:58.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.065+0000 7f1bf0ff9640 1 -- 172.21.15.88:0/1837903915 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1be405a050 con 0x7f1bdc004800 2024-09-16T01:39:58.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.231+0000 7f1bfa7f6640 1 -- 172.21.15.88:0/1837903915 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1bf40b56f0 con 0x7f1bdc004800 2024-09-16T01:39:58.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.232+0000 7f1bf0ff9640 1 -- 172.21.15.88:0/1837903915 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) ==== 74+0+13141 (secure 0 0 0) 0x7f1be4059950 con 0x7f1bdc004800 2024-09-16T01:39:58.233 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:58.233 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":76,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:39:56.937874+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:6837/2944989249":"2024-09-17T01:39:53.933424+0000","172.21.15.88:6835/2944989249":"2024-09-17T01:39:53.933424+0000","172.21.15.154:6835/3482042494":"2024-09-17T01:39:46.253065+0000","172.21.15.154:6833/3482042494":"2024-09-17T01:39:46.253065+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:39:58.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.233+0000 7f1bfa7f6640 1 -- 172.21.15.88:0/1837903915 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1be006c2b0 msgr2=0x7f1be006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:58.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.233+0000 7f1bfa7f6640 1 --2- 172.21.15.88:0/1837903915 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1be006c2b0 0x7f1be006e770 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1be8032000 tx=0x7f1be8034040 comp rx=0 tx=0).stop 2024-09-16T01:39:58.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.233+0000 7f1bfa7f6640 1 -- 172.21.15.88:0/1837903915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bdc004800 msgr2=0x7f1bf40cdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:58.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.233+0000 7f1bfa7f6640 1 --2- 172.21.15.88:0/1837903915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bdc004800 0x7f1bf40cdb50 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f1be400b7f0 tx=0x7f1be400bcb0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.233+0000 7f1bfa7f6640 1 -- 172.21.15.88:0/1837903915 shutdown_connections 2024-09-16T01:39:58.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.233+0000 7f1bfa7f6640 1 --2- 172.21.15.88:0/1837903915 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1be006c2b0 0x7f1be006e770 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.233+0000 7f1bfa7f6640 1 --2- 172.21.15.88:0/1837903915 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bf415a030 0x7f1bf415a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.234+0000 7f1bfa7f6640 1 --2- 172.21.15.88:0/1837903915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bf41f4ca0 0x7f1bf4157690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.234+0000 7f1bfa7f6640 1 --2- 172.21.15.88:0/1837903915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1bdc004800 0x7f1bf40cdb50 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.234+0000 7f1bfa7f6640 1 -- 172.21.15.88:0/1837903915 >> 172.21.15.88:0/1837903915 conn(0x7f1bf40d14d0 msgr2=0x7f1bf40b4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:58.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.234+0000 7f1bfa7f6640 1 -- 172.21.15.88:0/1837903915 shutdown_connections 2024-09-16T01:39:58.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.234+0000 7f1bfa7f6640 1 -- 172.21.15.88:0/1837903915 wait complete. 2024-09-16T01:39:58.246 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T01:34:10.547178+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T01:39:58.247 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T01:39:58.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.346+0000 7f8bc91f8640 1 Processor -- start 2024-09-16T01:39:58.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.346+0000 7f8bc91f8640 1 -- start start 2024-09-16T01:39:58.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.347+0000 7f8bc91f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8bc41492f0 0x7f8bc41666a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:58.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.347+0000 7f8bc91f8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8bc40b60e0 0x7f8bc4166be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:58.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.347+0000 7f8bc91f8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8bc4167120 0x7f8bc4072a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:58.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.347+0000 7f8bc91f8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f8bc4070fb0 con 0x7f8bc4167120 2024-09-16T01:39:58.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.347+0000 7f8bc91f8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f8bc40b3f40 con 0x7f8bc40b60e0 2024-09-16T01:39:58.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.347+0000 7f8bc91f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8bc4071200 con 0x7f8bc41492f0 2024-09-16T01:39:58.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.347+0000 7f8bc359e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8bc4167120 0x7f8bc4072a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:58.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.348+0000 7f8bc359e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8bc4167120 0x7f8bc4072a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60304/0 (socket says 172.21.15.88:60304) 2024-09-16T01:39:58.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.348+0000 7f8bc359e640 1 -- 172.21.15.88:0/221382671 learned_addr learned my addr 172.21.15.88:0/221382671 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:58.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.348+0000 7f8bc359e640 1 -- 172.21.15.88:0/221382671 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8bc41492f0 msgr2=0x7f8bc41666a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:58.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.348+0000 7f8bc259c640 1 --2- 172.21.15.88:0/221382671 >> v2:172.21.15.154:3300/0 conn(0x7f8bc40b60e0 0x7f8bc4166be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:58.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.348+0000 7f8bc2d9d640 1 --2- 172.21.15.88:0/221382671 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8bc41492f0 0x7f8bc41666a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:58.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.348+0000 7f8bc359e640 1 --2- 172.21.15.88:0/221382671 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8bc41492f0 0x7f8bc41666a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.348+0000 7f8bc359e640 1 -- 172.21.15.88:0/221382671 >> v2:172.21.15.154:3300/0 conn(0x7f8bc40b60e0 msgr2=0x7f8bc4166be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:58.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.348+0000 7f8bc359e640 1 --2- 172.21.15.88:0/221382671 >> v2:172.21.15.154:3300/0 conn(0x7f8bc40b60e0 0x7f8bc4166be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.348+0000 7f8bc359e640 1 -- 172.21.15.88:0/221382671 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bc4073d90 con 0x7f8bc4167120 2024-09-16T01:39:58.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.349+0000 7f8bc359e640 1 --2- 172.21.15.88:0/221382671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bc4167120 0x7f8bc4072a60 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f8bc40b5b60 tx=0x7f8bbc02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29b7fdeab61f1360 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:58.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.349+0000 7f8bc1d9b640 1 -- 172.21.15.88:0/221382671 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc036070 con 0x7f8bc4167120 2024-09-16T01:39:58.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.349+0000 7f8bc1d9b640 1 -- 172.21.15.88:0/221382671 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bbc031600 con 0x7f8bc4167120 2024-09-16T01:39:58.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.349+0000 7f8bc1d9b640 1 -- 172.21.15.88:0/221382671 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc0319d0 con 0x7f8bc4167120 2024-09-16T01:39:58.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.349+0000 7f8bc91f8640 1 -- 172.21.15.88:0/221382671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bc4167120 msgr2=0x7f8bc4072a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:58.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.349+0000 7f8bc91f8640 1 --2- 172.21.15.88:0/221382671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bc4167120 0x7f8bc4072a60 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f8bc40b5b60 tx=0x7f8bbc02a610 comp rx=0 tx=0).stop 2024-09-16T01:39:58.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.349+0000 7f8bc91f8640 1 -- 172.21.15.88:0/221382671 shutdown_connections 2024-09-16T01:39:58.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.349+0000 7f8bc91f8640 1 --2- 172.21.15.88:0/221382671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bc4167120 0x7f8bc4072a60 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.349+0000 7f8bc91f8640 1 --2- 172.21.15.88:0/221382671 >> v2:172.21.15.154:3300/0 conn(0x7f8bc40b60e0 0x7f8bc4166be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.349+0000 7f8bc91f8640 1 --2- 172.21.15.88:0/221382671 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8bc41492f0 0x7f8bc41666a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.349+0000 7f8bc91f8640 1 -- 172.21.15.88:0/221382671 >> 172.21.15.88:0/221382671 conn(0x7f8bc406d0f0 msgr2=0x7f8bc406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:58.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.349+0000 7f8bc91f8640 1 -- 172.21.15.88:0/221382671 shutdown_connections 2024-09-16T01:39:58.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.349+0000 7f8bc91f8640 1 -- 172.21.15.88:0/221382671 wait complete. 2024-09-16T01:39:58.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.351+0000 7f8bc91f8640 1 Processor -- start 2024-09-16T01:39:58.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.351+0000 7f8bc91f8640 1 -- start start 2024-09-16T01:39:58.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.351+0000 7f8bc91f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8bc40b60e0 0x7f8bc4143410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:58.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.351+0000 7f8bc91f8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bc41492f0 0x7f8bc41ed580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:58.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.351+0000 7f8bc91f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8bc4167120 0x7f8bc41edac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:58.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.351+0000 7f8bc91f8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc4073120 con 0x7f8bc41492f0 2024-09-16T01:39:58.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.351+0000 7f8bc91f8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc4072fa0 con 0x7f8bc4167120 2024-09-16T01:39:58.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.351+0000 7f8bc91f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8bc40732a0 con 0x7f8bc40b60e0 2024-09-16T01:39:58.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.352+0000 7f8bc259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bc41492f0 0x7f8bc41ed580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:58.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.352+0000 7f8bc259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bc41492f0 0x7f8bc41ed580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60320/0 (socket says 172.21.15.88:60320) 2024-09-16T01:39:58.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.352+0000 7f8bc259c640 1 -- 172.21.15.88:0/3540994676 learned_addr learned my addr 172.21.15.88:0/3540994676 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:58.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.352+0000 7f8bc2d9d640 1 --2- 172.21.15.88:0/3540994676 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8bc40b60e0 0x7f8bc4143410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:58.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.352+0000 7f8bc359e640 1 --2- 172.21.15.88:0/3540994676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8bc4167120 0x7f8bc41edac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:58.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.352+0000 7f8bc259c640 1 -- 172.21.15.88:0/3540994676 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8bc40b60e0 msgr2=0x7f8bc4143410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:58.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.352+0000 7f8bc259c640 1 --2- 172.21.15.88:0/3540994676 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8bc40b60e0 0x7f8bc4143410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.352+0000 7f8bc259c640 1 -- 172.21.15.88:0/3540994676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8bc4167120 msgr2=0x7f8bc41edac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:58.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.352+0000 7f8bc259c640 1 --2- 172.21.15.88:0/3540994676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8bc4167120 0x7f8bc41edac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.352+0000 7f8bc259c640 1 -- 172.21.15.88:0/3540994676 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bc4142370 con 0x7f8bc41492f0 2024-09-16T01:39:58.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.352+0000 7f8bc259c640 1 --2- 172.21.15.88:0/3540994676 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bc41492f0 0x7f8bc41ed580 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f8bb800c7f0 tx=0x7f8bb800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:58.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.352+0000 7f8bab7fe640 1 -- 172.21.15.88:0/3540994676 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb8015070 con 0x7f8bc41492f0 2024-09-16T01:39:58.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.352+0000 7f8bab7fe640 1 -- 172.21.15.88:0/3540994676 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bb8004220 con 0x7f8bc41492f0 2024-09-16T01:39:58.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.352+0000 7f8bc91f8640 1 -- 172.21.15.88:0/3540994676 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bc4142650 con 0x7f8bc41492f0 2024-09-16T01:39:58.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.352+0000 7f8bab7fe640 1 -- 172.21.15.88:0/3540994676 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb8007620 con 0x7f8bc41492f0 2024-09-16T01:39:58.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.352+0000 7f8bc91f8640 1 -- 172.21.15.88:0/3540994676 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bc4142c10 con 0x7f8bc41492f0 2024-09-16T01:39:58.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.353+0000 7f8bc91f8640 1 -- 172.21.15.88:0/3540994676 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bc4072a60 con 0x7f8bc41492f0 2024-09-16T01:39:58.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.354+0000 7f8bab7fe640 1 -- 172.21.15.88:0/3540994676 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8bb80079d0 con 0x7f8bc41492f0 2024-09-16T01:39:58.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.354+0000 7f8bab7fe640 1 --2- 172.21.15.88:0/3540994676 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8bb006c2b0 0x7f8bb006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:58.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.354+0000 7f8bc2d9d640 1 --2- 172.21.15.88:0/3540994676 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8bb006c2b0 0x7f8bb006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:58.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.354+0000 7f8bab7fe640 1 -- 172.21.15.88:0/3540994676 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5029+0+0 (secure 0 0 0) 0x7f8bb808b4a0 con 0x7f8bc41492f0 2024-09-16T01:39:58.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.355+0000 7f8bc2d9d640 1 --2- 172.21.15.88:0/3540994676 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8bb006c2b0 0x7f8bb006e770 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f8bb400aac0 tx=0x7f8bb4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:58.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.359+0000 7f8bab7fe640 1 -- 172.21.15.88:0/3540994676 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8bb8059860 con 0x7f8bc41492f0 2024-09-16T01:39:58.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.524+0000 7f8bc91f8640 1 -- 172.21.15.88:0/3540994676 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8bc40b4bc0 con 0x7f8bc41492f0 2024-09-16T01:39:58.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.524+0000 7f8bab7fe640 1 -- 172.21.15.88:0/3540994676 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) ==== 93+0+10 (secure 0 0 0) 0x7f8bb805db60 con 0x7f8bc41492f0 2024-09-16T01:39:58.526 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2024-09-16T01:39:58.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.527+0000 7f8bc91f8640 1 -- 172.21.15.88:0/3540994676 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8bb006c2b0 msgr2=0x7f8bb006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:58.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.527+0000 7f8bc91f8640 1 --2- 172.21.15.88:0/3540994676 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8bb006c2b0 0x7f8bb006e770 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f8bb400aac0 tx=0x7f8bb4008040 comp rx=0 tx=0).stop 2024-09-16T01:39:58.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.527+0000 7f8bc91f8640 1 -- 172.21.15.88:0/3540994676 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bc41492f0 msgr2=0x7f8bc41ed580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:58.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.527+0000 7f8bc91f8640 1 --2- 172.21.15.88:0/3540994676 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bc41492f0 0x7f8bc41ed580 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f8bb800c7f0 tx=0x7f8bb800ccb0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.527+0000 7f8bc91f8640 1 -- 172.21.15.88:0/3540994676 shutdown_connections 2024-09-16T01:39:58.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.527+0000 7f8bc91f8640 1 --2- 172.21.15.88:0/3540994676 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8bb006c2b0 0x7f8bb006e770 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.527+0000 7f8bc91f8640 1 --2- 172.21.15.88:0/3540994676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8bc4167120 0x7f8bc41edac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.527+0000 7f8bc91f8640 1 --2- 172.21.15.88:0/3540994676 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8bc41492f0 0x7f8bc41ed580 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.527+0000 7f8bc91f8640 1 --2- 172.21.15.88:0/3540994676 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8bc40b60e0 0x7f8bc4143410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.527+0000 7f8bc91f8640 1 -- 172.21.15.88:0/3540994676 >> 172.21.15.88:0/3540994676 conn(0x7f8bc406d0f0 msgr2=0x7f8bc40716f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:58.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.527+0000 7f8bc91f8640 1 -- 172.21.15.88:0/3540994676 shutdown_connections 2024-09-16T01:39:58.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.527+0000 7f8bc91f8640 1 -- 172.21.15.88:0/3540994676 wait complete. 2024-09-16T01:39:58.539 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_damage.TestDamage.test_object_deletion' 2024-09-16T01:39:58.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.636+0000 7f0e5e39f640 1 Processor -- start 2024-09-16T01:39:58.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.637+0000 7f0e5e39f640 1 -- start start 2024-09-16T01:39:58.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.637+0000 7f0e5e39f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0e580b7220 0x7f0e580b5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:58.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.637+0000 7f0e5e39f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e581600c0 con 0x7f0e5815c270 2024-09-16T01:39:58.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.637+0000 7f0e5e39f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e58071070 con 0x7f0e5815c650 2024-09-16T01:39:58.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.637+0000 7f0e5e39f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0e58160330 con 0x7f0e580b7220 2024-09-16T01:39:58.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.638+0000 7f0e5d39d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0e5815c270 0x7f0e580b6b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:56036/0 (socket says 172.21.15.88:56036) 2024-09-16T01:39:58.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.638+0000 7f0e5d39d640 1 -- 172.21.15.88:0/3023785557 learned_addr learned my addr 172.21.15.88:0/3023785557 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:58.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.638+0000 7f0e57fff640 1 -- 172.21.15.88:0/3023785557 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3824786437 0 0) 0x7f0e581600c0 con 0x7f0e5815c270 2024-09-16T01:39:58.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.638+0000 7f0e5cb9c640 1 --2- 172.21.15.88:0/3023785557 >> v2:172.21.15.154:3300/0 conn(0x7f0e580b7220 0x7f0e580b5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:58.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.638+0000 7f0e57fff640 1 -- 172.21.15.88:0/3023785557 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e40003520 con 0x7f0e5815c270 2024-09-16T01:39:58.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.638+0000 7f0e57fff640 1 -- 172.21.15.88:0/3023785557 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3193619794 0 0) 0x7f0e40003520 con 0x7f0e5815c270 2024-09-16T01:39:58.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.638+0000 7f0e57fff640 1 -- 172.21.15.88:0/3023785557 >> v2:172.21.15.154:3300/0 conn(0x7f0e580b7220 msgr2=0x7f0e580b5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:58.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.638+0000 7f0e57fff640 1 --2- 172.21.15.88:0/3023785557 >> v2:172.21.15.154:3300/0 conn(0x7f0e580b7220 0x7f0e580b5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.638+0000 7f0e57fff640 1 -- 172.21.15.88:0/3023785557 >> v1:172.21.15.154:6789/0 conn(0x7f0e5815c650 legacy=0x7f0e580b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:58.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.638+0000 7f0e57fff640 1 -- 172.21.15.88:0/3023785557 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e58160610 con 0x7f0e5815c270 2024-09-16T01:39:58.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.638+0000 7f0e57fff640 1 -- 172.21.15.88:0/3023785557 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f0e50001670 con 0x7f0e5815c270 2024-09-16T01:39:58.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.638+0000 7f0e57fff640 1 -- 172.21.15.88:0/3023785557 >> v1:172.21.15.88:6789/0 conn(0x7f0e5815c270 legacy=0x7f0e580b6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:58.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.638+0000 7f0e57fff640 1 --2- 172.21.15.88:0/3023785557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e400039b0 0x7f0e40003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:58.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.638+0000 7f0e57fff640 1 --2- 172.21.15.88:0/3023785557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e40004800 0x7f0e40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:58.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.638+0000 7f0e57fff640 1 --2- 172.21.15.88:0/3023785557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e400051c0 0x7f0e4000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:58.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.638+0000 7f0e57fff640 1 -- 172.21.15.88:0/3023785557 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0e58160330 con 0x7f0e400051c0 2024-09-16T01:39:58.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.638+0000 7f0e57fff640 1 -- 172.21.15.88:0/3023785557 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0e4000c060 con 0x7f0e400039b0 2024-09-16T01:39:58.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.638+0000 7f0e57fff640 1 -- 172.21.15.88:0/3023785557 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0e4000c260 con 0x7f0e40004800 2024-09-16T01:39:58.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.638+0000 7f0e5db9e640 1 --2- 172.21.15.88:0/3023785557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e400051c0 0x7f0e4000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:58.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.638+0000 7f0e5d39d640 1 --2- 172.21.15.88:0/3023785557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e400039b0 0x7f0e40003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:58.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.638+0000 7f0e5cb9c640 1 --2- 172.21.15.88:0/3023785557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e40004800 0x7f0e40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:58.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.639+0000 7f0e5db9e640 1 -- 172.21.15.88:0/3023785557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e40004800 msgr2=0x7f0e40004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:58.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.639+0000 7f0e5db9e640 1 --2- 172.21.15.88:0/3023785557 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e40004800 0x7f0e40004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.639+0000 7f0e5db9e640 1 -- 172.21.15.88:0/3023785557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e400039b0 msgr2=0x7f0e40003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:58.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.639+0000 7f0e5db9e640 1 --2- 172.21.15.88:0/3023785557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e400039b0 0x7f0e40003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.639+0000 7f0e5db9e640 1 -- 172.21.15.88:0/3023785557 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e40003520 con 0x7f0e400051c0 2024-09-16T01:39:58.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.639+0000 7f0e5db9e640 1 -- 172.21.15.88:0/3023785557 reap_dead start 2024-09-16T01:39:58.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.639+0000 7f0e5db9e640 1 --2- 172.21.15.88:0/3023785557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e400051c0 0x7f0e4000b9c0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f0e48001720 tx=0x7f0e40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b830684d83609906 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:58.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.639+0000 7f0e57fff640 1 -- 172.21.15.88:0/3023785557 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e48001370 con 0x7f0e400051c0 2024-09-16T01:39:58.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.639+0000 7f0e57fff640 1 -- 172.21.15.88:0/3023785557 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e480052d0 con 0x7f0e400051c0 2024-09-16T01:39:58.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.639+0000 7f0e57fff640 1 -- 172.21.15.88:0/3023785557 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4802aba0 con 0x7f0e400051c0 2024-09-16T01:39:58.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.640+0000 7f0e5e39f640 1 -- 172.21.15.88:0/3023785557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e400051c0 msgr2=0x7f0e4000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:58.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.640+0000 7f0e5e39f640 1 --2- 172.21.15.88:0/3023785557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e400051c0 0x7f0e4000b9c0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f0e48001720 tx=0x7f0e40005830 comp rx=0 tx=0).stop 2024-09-16T01:39:58.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.640+0000 7f0e5e39f640 1 -- 172.21.15.88:0/3023785557 shutdown_connections 2024-09-16T01:39:58.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.640+0000 7f0e5e39f640 1 --2- 172.21.15.88:0/3023785557 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e400051c0 0x7f0e4000b9c0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.640+0000 7f0e5e39f640 1 -- 172.21.15.88:0/3023785557 >> 172.21.15.88:0/3023785557 conn(0x7f0e580b4010 msgr2=0x7f0e580b43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:58.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.640+0000 7f0e5e39f640 1 -- 172.21.15.88:0/3023785557 shutdown_connections 2024-09-16T01:39:58.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.640+0000 7f0e5e39f640 1 -- 172.21.15.88:0/3023785557 wait complete. 2024-09-16T01:39:58.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.641+0000 7f0e5e39f640 1 Processor -- start 2024-09-16T01:39:58.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.641+0000 7f0e5e39f640 1 -- start start 2024-09-16T01:39:58.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.642+0000 7f0e5e39f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e400051c0 0x7f0e58154dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:58.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.642+0000 7f0e5e39f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e5815e190 0x7f0e5815a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:58.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.642+0000 7f0e5e39f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e5815e5f0 0x7f0e5815adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:58.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.642+0000 7f0e5e39f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0e58071350 con 0x7f0e5815e190 2024-09-16T01:39:58.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.642+0000 7f0e5e39f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0e580ceac0 con 0x7f0e5815e5f0 2024-09-16T01:39:58.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.642+0000 7f0e5e39f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0e580d1fc0 con 0x7f0e400051c0 2024-09-16T01:39:58.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.642+0000 7f0e5cb9c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e5815e190 0x7f0e5815a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:58.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.642+0000 7f0e5cb9c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e5815e190 0x7f0e5815a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60344/0 (socket says 172.21.15.88:60344) 2024-09-16T01:39:58.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.642+0000 7f0e5cb9c640 1 -- 172.21.15.88:0/2101117645 learned_addr learned my addr 172.21.15.88:0/2101117645 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:58.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.642+0000 7f0e5d39d640 1 --2- 172.21.15.88:0/2101117645 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e400051c0 0x7f0e58154dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:58.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.642+0000 7f0e5db9e640 1 --2- 172.21.15.88:0/2101117645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e5815e5f0 0x7f0e5815adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:58.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.642+0000 7f0e5cb9c640 1 -- 172.21.15.88:0/2101117645 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e400051c0 msgr2=0x7f0e58154dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:58.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.642+0000 7f0e5cb9c640 1 --2- 172.21.15.88:0/2101117645 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e400051c0 0x7f0e58154dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.642+0000 7f0e5cb9c640 1 -- 172.21.15.88:0/2101117645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e5815e5f0 msgr2=0x7f0e5815adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:58.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.642+0000 7f0e5cb9c640 1 --2- 172.21.15.88:0/2101117645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e5815e5f0 0x7f0e5815adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.642+0000 7f0e5cb9c640 1 -- 172.21.15.88:0/2101117645 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e5815eba0 con 0x7f0e5815e190 2024-09-16T01:39:58.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.642+0000 7f0e5db9e640 1 --2- 172.21.15.88:0/2101117645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e5815e5f0 0x7f0e5815adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:58.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.643+0000 7f0e5cb9c640 1 --2- 172.21.15.88:0/2101117645 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e5815e190 0x7f0e5815a8b0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c009f90 tx=0x7f0e4c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:58.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.643+0000 7f0e55ffb640 1 -- 172.21.15.88:0/2101117645 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c015430 con 0x7f0e5815e190 2024-09-16T01:39:58.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.643+0000 7f0e55ffb640 1 -- 172.21.15.88:0/2101117645 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e4c0155d0 con 0x7f0e5815e190 2024-09-16T01:39:58.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.643+0000 7f0e5e39f640 1 -- 172.21.15.88:0/2101117645 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e5815ee80 con 0x7f0e5815e190 2024-09-16T01:39:58.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.643+0000 7f0e55ffb640 1 -- 172.21.15.88:0/2101117645 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c0159c0 con 0x7f0e5815e190 2024-09-16T01:39:58.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.643+0000 7f0e5e39f640 1 -- 172.21.15.88:0/2101117645 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e5815f440 con 0x7f0e5815e190 2024-09-16T01:39:58.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.644+0000 7f0e5e39f640 1 -- 172.21.15.88:0/2101117645 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e58160e10 con 0x7f0e5815e190 2024-09-16T01:39:58.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.644+0000 7f0e55ffb640 1 -- 172.21.15.88:0/2101117645 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0e4c015b60 con 0x7f0e5815e190 2024-09-16T01:39:58.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.645+0000 7f0e55ffb640 1 --2- 172.21.15.88:0/2101117645 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0e4406c2e0 0x7f0e4406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:58.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.645+0000 7f0e55ffb640 1 -- 172.21.15.88:0/2101117645 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5029+0+0 (secure 0 0 0) 0x7f0e4c08c130 con 0x7f0e5815e190 2024-09-16T01:39:58.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.645+0000 7f0e5d39d640 1 --2- 172.21.15.88:0/2101117645 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0e4406c2e0 0x7f0e4406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:58.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.645+0000 7f0e5d39d640 1 --2- 172.21.15.88:0/2101117645 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0e4406c2e0 0x7f0e4406e7a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0e500015d0 tx=0x7f0e500078f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:58.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.649+0000 7f0e55ffb640 1 -- 172.21.15.88:0/2101117645 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0e4c019030 con 0x7f0e5815e190 2024-09-16T01:39:58.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.877+0000 7f0e5e39f640 1 -- 172.21.15.88:0/2101117645 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_object_deletion"]} v 0) -- 0x7f0e5815f820 con 0x7f0e5815e190 2024-09-16T01:39:58.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.944+0000 7f0e55ffb640 1 -- 172.21.15.88:0/2101117645 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_object_deletion"]}]=0 v275) ==== 138+0+0 (secure 0 0 0) 0x7f0e4c010040 con 0x7f0e5815e190 2024-09-16T01:39:58.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.947+0000 7f0e5e39f640 1 -- 172.21.15.88:0/2101117645 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0e4406c2e0 msgr2=0x7f0e4406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:58.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.947+0000 7f0e5e39f640 1 --2- 172.21.15.88:0/2101117645 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0e4406c2e0 0x7f0e4406e7a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0e500015d0 tx=0x7f0e500078f0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.947+0000 7f0e5e39f640 1 -- 172.21.15.88:0/2101117645 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e5815e190 msgr2=0x7f0e5815a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:58.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.947+0000 7f0e5e39f640 1 --2- 172.21.15.88:0/2101117645 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e5815e190 0x7f0e5815a8b0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c009f90 tx=0x7f0e4c00f580 comp rx=0 tx=0).stop 2024-09-16T01:39:58.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.947+0000 7f0e5e39f640 1 -- 172.21.15.88:0/2101117645 shutdown_connections 2024-09-16T01:39:58.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.947+0000 7f0e5e39f640 1 --2- 172.21.15.88:0/2101117645 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0e4406c2e0 0x7f0e4406e7a0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.947+0000 7f0e5e39f640 1 --2- 172.21.15.88:0/2101117645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e5815e5f0 0x7f0e5815adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.947+0000 7f0e5e39f640 1 --2- 172.21.15.88:0/2101117645 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e5815e190 0x7f0e5815a8b0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.947+0000 7f0e5e39f640 1 --2- 172.21.15.88:0/2101117645 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e400051c0 0x7f0e58154dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:58.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.947+0000 7f0e5e39f640 1 -- 172.21.15.88:0/2101117645 >> 172.21.15.88:0/2101117645 conn(0x7f0e580b4010 msgr2=0x7f0e580d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:58.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.947+0000 7f0e5e39f640 1 -- 172.21.15.88:0/2101117645 shutdown_connections 2024-09-16T01:39:58.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:58.947+0000 7f0e5e39f640 1 -- 172.21.15.88:0/2101117645 wait complete. 2024-09-16T01:39:58.959 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T01:39:59.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.059+0000 7f4039ad0640 1 Processor -- start 2024-09-16T01:39:59.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.059+0000 7f4039ad0640 1 -- start start 2024-09-16T01:39:59.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.060+0000 7f4039ad0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4034073600 0x7f403415e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:59.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.060+0000 7f4039ad0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f403415a430 con 0x7f403415a050 2024-09-16T01:39:59.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.060+0000 7f4039ad0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40340cd0a0 con 0x7f40340731c0 2024-09-16T01:39:59.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.060+0000 7f4039ad0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f40340cd380 con 0x7f4034073600 2024-09-16T01:39:59.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.060+0000 7f40337fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f403415a050 0x7f403415d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:56052/0 (socket says 172.21.15.88:56052) 2024-09-16T01:39:59.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.060+0000 7f40337fe640 1 -- 172.21.15.88:0/1677998727 learned_addr learned my addr 172.21.15.88:0/1677998727 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:59.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.060+0000 7f40327fc640 1 -- 172.21.15.88:0/1677998727 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2789936923 0 0) 0x7f403415a430 con 0x7f403415a050 2024-09-16T01:39:59.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.060+0000 7f40327fc640 1 -- 172.21.15.88:0/1677998727 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f401c003520 con 0x7f403415a050 2024-09-16T01:39:59.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.060+0000 7f4033fff640 1 --2- 172.21.15.88:0/1677998727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4034073600 0x7f403415e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:59.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.060+0000 7f40327fc640 1 -- 172.21.15.88:0/1677998727 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 677918292 0 0) 0x7f401c003520 con 0x7f403415a050 2024-09-16T01:39:59.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.060+0000 7f40327fc640 1 -- 172.21.15.88:0/1677998727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4034073600 msgr2=0x7f403415e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:59.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.060+0000 7f40327fc640 1 --2- 172.21.15.88:0/1677998727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4034073600 0x7f403415e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:59.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.060+0000 7f40327fc640 1 -- 172.21.15.88:0/1677998727 >> v1:172.21.15.154:6789/0 conn(0x7f40340731c0 legacy=0x7f403415de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:59.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.060+0000 7f40327fc640 1 -- 172.21.15.88:0/1677998727 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40340cd660 con 0x7f403415a050 2024-09-16T01:39:59.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.060+0000 7f40327fc640 1 -- 172.21.15.88:0/1677998727 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f4024001670 con 0x7f403415a050 2024-09-16T01:39:59.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.060+0000 7f40327fc640 1 -- 172.21.15.88:0/1677998727 >> v1:172.21.15.88:6789/0 conn(0x7f403415a050 legacy=0x7f403415d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:59.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.060+0000 7f40327fc640 1 --2- 172.21.15.88:0/1677998727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f401c0039e0 0x7f401c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:59.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.060+0000 7f40327fc640 1 --2- 172.21.15.88:0/1677998727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f401c004800 0x7f401c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:59.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.060+0000 7f40327fc640 1 --2- 172.21.15.88:0/1677998727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4034073600 0x7f401c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:59.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.060+0000 7f40327fc640 1 -- 172.21.15.88:0/1677998727 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f40340cd380 con 0x7f401c004800 2024-09-16T01:39:59.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.060+0000 7f40327fc640 1 -- 172.21.15.88:0/1677998727 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f401c00bb20 con 0x7f401c0039e0 2024-09-16T01:39:59.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.060+0000 7f40327fc640 1 -- 172.21.15.88:0/1677998727 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f401c00bd20 con 0x7f4034073600 2024-09-16T01:39:59.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.061+0000 7f4032ffd640 1 --2- 172.21.15.88:0/1677998727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f401c004800 0x7f401c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:59.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.061+0000 7f40337fe640 1 --2- 172.21.15.88:0/1677998727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f401c0039e0 0x7f401c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:59.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.061+0000 7f4033fff640 1 --2- 172.21.15.88:0/1677998727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4034073600 0x7f401c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:59.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.061+0000 7f4032ffd640 1 -- 172.21.15.88:0/1677998727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4034073600 msgr2=0x7f401c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:59.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.061+0000 7f4032ffd640 1 --2- 172.21.15.88:0/1677998727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4034073600 0x7f401c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:59.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.061+0000 7f4032ffd640 1 -- 172.21.15.88:0/1677998727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f401c0039e0 msgr2=0x7f401c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:59.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.061+0000 7f4032ffd640 1 --2- 172.21.15.88:0/1677998727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f401c0039e0 0x7f401c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:59.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.061+0000 7f4032ffd640 1 -- 172.21.15.88:0/1677998727 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f401c003520 con 0x7f401c004800 2024-09-16T01:39:59.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.061+0000 7f4033fff640 1 --2- 172.21.15.88:0/1677998727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4034073600 0x7f401c005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:59.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.061+0000 7f4032ffd640 1 --2- 172.21.15.88:0/1677998727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f401c004800 0x7f401c004c80 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f4028001750 tx=0x7f401c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91bd288aec6f3a37 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:59.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.062+0000 7f40327fc640 1 -- 172.21.15.88:0/1677998727 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4028001370 con 0x7f401c004800 2024-09-16T01:39:59.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.062+0000 7f40327fc640 1 -- 172.21.15.88:0/1677998727 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40280052d0 con 0x7f401c004800 2024-09-16T01:39:59.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.062+0000 7f40327fc640 1 -- 172.21.15.88:0/1677998727 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402802abe0 con 0x7f401c004800 2024-09-16T01:39:59.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.062+0000 7f4039ad0640 1 -- 172.21.15.88:0/1677998727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f401c004800 msgr2=0x7f401c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:59.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.062+0000 7f4039ad0640 1 --2- 172.21.15.88:0/1677998727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f401c004800 0x7f401c004c80 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f4028001750 tx=0x7f401c005830 comp rx=0 tx=0).stop 2024-09-16T01:39:59.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.062+0000 7f4033fff640 1 -- 172.21.15.88:0/1677998727 reap_dead start 2024-09-16T01:39:59.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.062+0000 7f4039ad0640 1 -- 172.21.15.88:0/1677998727 shutdown_connections 2024-09-16T01:39:59.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.062+0000 7f4039ad0640 1 -- 172.21.15.88:0/1677998727 >> 172.21.15.88:0/1677998727 conn(0x7f403406d0f0 msgr2=0x7f403406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:59.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.062+0000 7f4039ad0640 1 -- 172.21.15.88:0/1677998727 shutdown_connections 2024-09-16T01:39:59.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.062+0000 7f4039ad0640 1 -- 172.21.15.88:0/1677998727 wait complete. 2024-09-16T01:39:59.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.063+0000 7f4039ad0640 1 Processor -- start 2024-09-16T01:39:59.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.064+0000 7f4039ad0640 1 -- start start 2024-09-16T01:39:59.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.064+0000 7f4039ad0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f401c004800 0x7f40340c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:59.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.064+0000 7f4039ad0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40340cc330 0x7f4034156440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:59.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.064+0000 7f4039ad0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4034147a70 0x7f4034147f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:59.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.064+0000 7f4039ad0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4034073a90 con 0x7f4034147a70 2024-09-16T01:39:59.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.064+0000 7f4039ad0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f40340cdc20 con 0x7f401c004800 2024-09-16T01:39:59.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.064+0000 7f4039ad0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f40340b36e0 con 0x7f40340cc330 2024-09-16T01:39:59.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.064+0000 7f4033fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4034147a70 0x7f4034147f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:59.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.064+0000 7f4033fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4034147a70 0x7f4034147f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60370/0 (socket says 172.21.15.88:60370) 2024-09-16T01:39:59.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.064+0000 7f4033fff640 1 -- 172.21.15.88:0/481162332 learned_addr learned my addr 172.21.15.88:0/481162332 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:59.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.065+0000 7f40337fe640 1 --2- 172.21.15.88:0/481162332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f401c004800 0x7f40340c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:59.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.065+0000 7f4032ffd640 1 --2- 172.21.15.88:0/481162332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40340cc330 0x7f4034156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:59.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.065+0000 7f4033fff640 1 -- 172.21.15.88:0/481162332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40340cc330 msgr2=0x7f4034156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:59.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.065+0000 7f4033fff640 1 --2- 172.21.15.88:0/481162332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40340cc330 0x7f4034156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:59.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.065+0000 7f4033fff640 1 -- 172.21.15.88:0/481162332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f401c004800 msgr2=0x7f40340c8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:59.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.065+0000 7f4033fff640 1 --2- 172.21.15.88:0/481162332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f401c004800 0x7f40340c8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:59.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.065+0000 7f4033fff640 1 -- 172.21.15.88:0/481162332 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40340cb290 con 0x7f4034147a70 2024-09-16T01:39:59.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.065+0000 7f40337fe640 1 --2- 172.21.15.88:0/481162332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f401c004800 0x7f40340c8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:59.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.065+0000 7f4033fff640 1 --2- 172.21.15.88:0/481162332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4034147a70 0x7f4034147f20 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f40340b36b0 tx=0x7f402c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:59.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.065+0000 7f401bfff640 1 -- 172.21.15.88:0/481162332 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c010df0 con 0x7f4034147a70 2024-09-16T01:39:59.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.065+0000 7f401bfff640 1 -- 172.21.15.88:0/481162332 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f402c003990 con 0x7f4034147a70 2024-09-16T01:39:59.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.065+0000 7f4039ad0640 1 -- 172.21.15.88:0/481162332 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40340cb570 con 0x7f4034147a70 2024-09-16T01:39:59.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.065+0000 7f401bfff640 1 -- 172.21.15.88:0/481162332 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c003d80 con 0x7f4034147a70 2024-09-16T01:39:59.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.065+0000 7f4039ad0640 1 -- 172.21.15.88:0/481162332 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40340cbb30 con 0x7f4034147a70 2024-09-16T01:39:59.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.066+0000 7f4039ad0640 1 -- 172.21.15.88:0/481162332 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40340ccd00 con 0x7f4034147a70 2024-09-16T01:39:59.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.067+0000 7f401bfff640 1 -- 172.21.15.88:0/481162332 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f402c01e020 con 0x7f4034147a70 2024-09-16T01:39:59.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.067+0000 7f401bfff640 1 --2- 172.21.15.88:0/481162332 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f402006c2e0 0x7f402006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:59.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.067+0000 7f40337fe640 1 --2- 172.21.15.88:0/481162332 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f402006c2e0 0x7f402006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:59.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.067+0000 7f401bfff640 1 -- 172.21.15.88:0/481162332 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5029+0+0 (secure 0 0 0) 0x7f402c08b900 con 0x7f4034147a70 2024-09-16T01:39:59.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.068+0000 7f40337fe640 1 --2- 172.21.15.88:0/481162332 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f402006c2e0 0x7f402006e7a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4024007cb0 tx=0x7f4024009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:59.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.072+0000 7f401bfff640 1 -- 172.21.15.88:0/481162332 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f402c05a280 con 0x7f4034147a70 2024-09-16T01:39:59.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.237+0000 7f4039ad0640 1 -- 172.21.15.88:0/481162332 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f403415eca0 con 0x7f4034147a70 2024-09-16T01:39:59.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.241+0000 7f401bfff640 1 -- 172.21.15.88:0/481162332 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) ==== 122+0+0 (secure 0 0 0) 0x7f402c059b80 con 0x7f4034147a70 2024-09-16T01:39:59.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.243+0000 7f4039ad0640 1 -- 172.21.15.88:0/481162332 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f402006c2e0 msgr2=0x7f402006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:59.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.243+0000 7f4039ad0640 1 --2- 172.21.15.88:0/481162332 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f402006c2e0 0x7f402006e7a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4024007cb0 tx=0x7f4024009040 comp rx=0 tx=0).stop 2024-09-16T01:39:59.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.243+0000 7f4039ad0640 1 -- 172.21.15.88:0/481162332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4034147a70 msgr2=0x7f4034147f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:59.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.243+0000 7f4039ad0640 1 --2- 172.21.15.88:0/481162332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4034147a70 0x7f4034147f20 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f40340b36b0 tx=0x7f402c009f90 comp rx=0 tx=0).stop 2024-09-16T01:39:59.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.243+0000 7f4039ad0640 1 -- 172.21.15.88:0/481162332 shutdown_connections 2024-09-16T01:39:59.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.243+0000 7f4039ad0640 1 --2- 172.21.15.88:0/481162332 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f402006c2e0 0x7f402006e7a0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:59.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.243+0000 7f4039ad0640 1 --2- 172.21.15.88:0/481162332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4034147a70 0x7f4034147f20 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:59.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.243+0000 7f4039ad0640 1 --2- 172.21.15.88:0/481162332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40340cc330 0x7f4034156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:59.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.243+0000 7f4039ad0640 1 --2- 172.21.15.88:0/481162332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f401c004800 0x7f40340c8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:59.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.243+0000 7f4039ad0640 1 -- 172.21.15.88:0/481162332 >> 172.21.15.88:0/481162332 conn(0x7f403406d0f0 msgr2=0x7f403415f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:59.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.243+0000 7f4039ad0640 1 -- 172.21.15.88:0/481162332 shutdown_connections 2024-09-16T01:39:59.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.243+0000 7f4039ad0640 1 -- 172.21.15.88:0/481162332 wait complete. 2024-09-16T01:39:59.255 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T01:39:59.255 DEBUG:teuthology.orchestra.run.smithi088:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T01:39:59.302 DEBUG:teuthology.orchestra.run.smithi154:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T01:39:59.343 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T01:39:59.343 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:39:59.343 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:39:59.359 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:39:59.360 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:39:59.360 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:39:59.401 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056026,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1347 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T01:39:59.401 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:39:59.402 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:39:59.417 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:39:59.417 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:39:59.417 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:39:59.460 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056026,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1347 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T01:39:59.460 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-16T01:39:59.460 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:39:59.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.557+0000 7fd2adeaf640 1 Processor -- start 2024-09-16T01:39:59.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.557+0000 7fd2adeaf640 1 -- start start 2024-09-16T01:39:59.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.558+0000 7fd2adeaf640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd2a80cd780 0x7fd2a80cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:59.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.558+0000 7fd2adeaf640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2a80ce4f0 con 0x7fd2a80b4d00 2024-09-16T01:39:59.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.558+0000 7fd2adeaf640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2a80ce7d0 con 0x7fd2a80b4920 2024-09-16T01:39:59.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.558+0000 7fd2adeaf640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd2a80cf100 con 0x7fd2a80cd780 2024-09-16T01:39:59.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.558+0000 7fd2a6ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd2a80b4d00 0x7fd2a80b5170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:56060/0 (socket says 172.21.15.88:56060) 2024-09-16T01:39:59.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.558+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/3867076262 learned_addr learned my addr 172.21.15.88:0/3867076262 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.558+0000 7fd2a7fff640 1 --2- 172.21.15.88:0/3867076262 >> v2:172.21.15.154:3300/0 conn(0x7fd2a80cd780 0x7fd2a80cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.558+0000 7fd2a67fc640 1 -- 172.21.15.88:0/3867076262 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3664800028 0 0) 0x7fd2a80ce4f0 con 0x7fd2a80b4d00 2024-09-16T01:39:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.558+0000 7fd2a67fc640 1 -- 172.21.15.88:0/3867076262 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd290003520 con 0x7fd2a80b4d00 2024-09-16T01:39:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.558+0000 7fd2a67fc640 1 -- 172.21.15.88:0/3867076262 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3711273391 0 0) 0x7fd290003520 con 0x7fd2a80b4d00 2024-09-16T01:39:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.558+0000 7fd2a67fc640 1 -- 172.21.15.88:0/3867076262 >> v2:172.21.15.154:3300/0 conn(0x7fd2a80cd780 msgr2=0x7fd2a80cdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.558+0000 7fd2a67fc640 1 --2- 172.21.15.88:0/3867076262 >> v2:172.21.15.154:3300/0 conn(0x7fd2a80cd780 0x7fd2a80cdb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.558+0000 7fd2a67fc640 1 -- 172.21.15.88:0/3867076262 >> v1:172.21.15.154:6789/0 conn(0x7fd2a80b4920 legacy=0x7fd2a815b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.558+0000 7fd2a67fc640 1 -- 172.21.15.88:0/3867076262 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2a80ceab0 con 0x7fd2a80b4d00 2024-09-16T01:39:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.558+0000 7fd2a67fc640 1 -- 172.21.15.88:0/3867076262 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fd29c001670 con 0x7fd2a80b4d00 2024-09-16T01:39:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.558+0000 7fd2a67fc640 1 -- 172.21.15.88:0/3867076262 >> v1:172.21.15.88:6789/0 conn(0x7fd2a80b4d00 legacy=0x7fd2a80b5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.558+0000 7fd2a67fc640 1 --2- 172.21.15.88:0/3867076262 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2900039b0 0x7fd290003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.558+0000 7fd2a67fc640 1 --2- 172.21.15.88:0/3867076262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd290004800 0x7fd290004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.558+0000 7fd2a67fc640 1 --2- 172.21.15.88:0/3867076262 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2900051c0 0x7fd29000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:59.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.558+0000 7fd2a67fc640 1 -- 172.21.15.88:0/3867076262 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a80cf100 con 0x7fd2900051c0 2024-09-16T01:39:59.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.558+0000 7fd2a67fc640 1 -- 172.21.15.88:0/3867076262 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd29000c060 con 0x7fd290004800 2024-09-16T01:39:59.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.558+0000 7fd2a67fc640 1 -- 172.21.15.88:0/3867076262 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd29000c260 con 0x7fd2900039b0 2024-09-16T01:39:59.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.559+0000 7fd2a7fff640 1 --2- 172.21.15.88:0/3867076262 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2900051c0 0x7fd29000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:59.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.559+0000 7fd2a6ffd640 1 --2- 172.21.15.88:0/3867076262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd290004800 0x7fd290004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:59.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.559+0000 7fd2a77fe640 1 --2- 172.21.15.88:0/3867076262 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2900039b0 0x7fd290003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:59.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.559+0000 7fd2a7fff640 1 -- 172.21.15.88:0/3867076262 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2900039b0 msgr2=0x7fd290003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:59.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.559+0000 7fd2a7fff640 1 --2- 172.21.15.88:0/3867076262 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2900039b0 0x7fd290003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:59.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.559+0000 7fd2a7fff640 1 -- 172.21.15.88:0/3867076262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd290004800 msgr2=0x7fd290004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:59.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.559+0000 7fd2a7fff640 1 --2- 172.21.15.88:0/3867076262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd290004800 0x7fd290004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:59.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.559+0000 7fd2a7fff640 1 -- 172.21.15.88:0/3867076262 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd290003520 con 0x7fd2900051c0 2024-09-16T01:39:59.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.559+0000 7fd2a77fe640 1 --2- 172.21.15.88:0/3867076262 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2900039b0 0x7fd290003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:39:59.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.559+0000 7fd2a7fff640 1 -- 172.21.15.88:0/3867076262 reap_dead start 2024-09-16T01:39:59.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.560+0000 7fd2a7fff640 1 --2- 172.21.15.88:0/3867076262 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2900051c0 0x7fd29000b9c0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0009fc0 tx=0x7fd290005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21093c12c6697a08 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:59.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.560+0000 7fd2a67fc640 1 -- 172.21.15.88:0/3867076262 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a0035020 con 0x7fd2900051c0 2024-09-16T01:39:59.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.560+0000 7fd2a67fc640 1 -- 172.21.15.88:0/3867076262 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2a0003b90 con 0x7fd2900051c0 2024-09-16T01:39:59.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.560+0000 7fd2a67fc640 1 -- 172.21.15.88:0/3867076262 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a0009c10 con 0x7fd2900051c0 2024-09-16T01:39:59.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.560+0000 7fd2adeaf640 1 -- 172.21.15.88:0/3867076262 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2900051c0 msgr2=0x7fd29000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:59.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.560+0000 7fd2adeaf640 1 --2- 172.21.15.88:0/3867076262 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2900051c0 0x7fd29000b9c0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0009fc0 tx=0x7fd290005830 comp rx=0 tx=0).stop 2024-09-16T01:39:59.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.560+0000 7fd2adeaf640 1 -- 172.21.15.88:0/3867076262 shutdown_connections 2024-09-16T01:39:59.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.560+0000 7fd2adeaf640 1 --2- 172.21.15.88:0/3867076262 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2900051c0 0x7fd29000b9c0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:59.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.560+0000 7fd2adeaf640 1 -- 172.21.15.88:0/3867076262 >> 172.21.15.88:0/3867076262 conn(0x7fd2a806bc10 msgr2=0x7fd2a806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:59.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.560+0000 7fd2adeaf640 1 -- 172.21.15.88:0/3867076262 shutdown_connections 2024-09-16T01:39:59.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.560+0000 7fd2adeaf640 1 -- 172.21.15.88:0/3867076262 wait complete. 2024-09-16T01:39:59.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.561+0000 7fd2adeaf640 1 Processor -- start 2024-09-16T01:39:59.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.562+0000 7fd2adeaf640 1 -- start start 2024-09-16T01:39:59.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.562+0000 7fd2adeaf640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2900051c0 0x7fd2a814ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:59.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.562+0000 7fd2adeaf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2a81530e0 0x7fd2a8153560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:59.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.562+0000 7fd2adeaf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2a81509d0 0x7fd2a8150e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:59.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.562+0000 7fd2adeaf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a80b52f0 con 0x7fd2a81509d0 2024-09-16T01:39:59.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.562+0000 7fd2adeaf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a80b5170 con 0x7fd2a81530e0 2024-09-16T01:39:59.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.562+0000 7fd2adeaf640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd2a80b5470 con 0x7fd2900051c0 2024-09-16T01:39:59.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.562+0000 7fd2a7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2a81509d0 0x7fd2a8150e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:59.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.562+0000 7fd2a7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2a81509d0 0x7fd2a8150e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60390/0 (socket says 172.21.15.88:60390) 2024-09-16T01:39:59.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.562+0000 7fd2a7fff640 1 -- 172.21.15.88:0/3413674774 learned_addr learned my addr 172.21.15.88:0/3413674774 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:59.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.563+0000 7fd2a6ffd640 1 --2- 172.21.15.88:0/3413674774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2a81530e0 0x7fd2a8153560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:59.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.563+0000 7fd2a77fe640 1 --2- 172.21.15.88:0/3413674774 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2900051c0 0x7fd2a814ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:59.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.563+0000 7fd2a7fff640 1 -- 172.21.15.88:0/3413674774 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2900051c0 msgr2=0x7fd2a814ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:59.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.563+0000 7fd2a7fff640 1 --2- 172.21.15.88:0/3413674774 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2900051c0 0x7fd2a814ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:59.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.563+0000 7fd2a7fff640 1 -- 172.21.15.88:0/3413674774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2a81530e0 msgr2=0x7fd2a8153560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:59.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.563+0000 7fd2a7fff640 1 --2- 172.21.15.88:0/3413674774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2a81530e0 0x7fd2a8153560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:59.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.563+0000 7fd2a7fff640 1 -- 172.21.15.88:0/3413674774 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2a8151740 con 0x7fd2a81509d0 2024-09-16T01:39:59.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.563+0000 7fd2a7fff640 1 --2- 172.21.15.88:0/3413674774 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2a81509d0 0x7fd2a8150e50 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fd2a000e230 tx=0x7fd2a0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:59.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.563+0000 7fd28ffff640 1 -- 172.21.15.88:0/3413674774 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a003f030 con 0x7fd2a81509d0 2024-09-16T01:39:59.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.563+0000 7fd28ffff640 1 -- 172.21.15.88:0/3413674774 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2a0003990 con 0x7fd2a81509d0 2024-09-16T01:39:59.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.563+0000 7fd2adeaf640 1 -- 172.21.15.88:0/3413674774 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2a8151a20 con 0x7fd2a81509d0 2024-09-16T01:39:59.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.563+0000 7fd28ffff640 1 -- 172.21.15.88:0/3413674774 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a0035070 con 0x7fd2a81509d0 2024-09-16T01:39:59.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.563+0000 7fd2adeaf640 1 -- 172.21.15.88:0/3413674774 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2a8151fe0 con 0x7fd2a81509d0 2024-09-16T01:39:59.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.564+0000 7fd2adeaf640 1 -- 172.21.15.88:0/3413674774 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2a8152220 con 0x7fd2a81509d0 2024-09-16T01:39:59.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.565+0000 7fd28ffff640 1 -- 172.21.15.88:0/3413674774 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd2a0042030 con 0x7fd2a81509d0 2024-09-16T01:39:59.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.565+0000 7fd28ffff640 1 --2- 172.21.15.88:0/3413674774 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd29406c2e0 0x7fd29406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:59.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.565+0000 7fd28ffff640 1 -- 172.21.15.88:0/3413674774 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 5029+0+0 (secure 0 0 0) 0x7fd2a00ab820 con 0x7fd2a81509d0 2024-09-16T01:39:59.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.565+0000 7fd2a77fe640 1 --2- 172.21.15.88:0/3413674774 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd29406c2e0 0x7fd29406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:59.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.565+0000 7fd2a77fe640 1 --2- 172.21.15.88:0/3413674774 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd29406c2e0 0x7fd29406e7a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd298007fd0 tx=0x7fd298009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:59.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.569+0000 7fd28ffff640 1 -- 172.21.15.88:0/3413674774 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd2a8152220 con 0x7fd2a81509d0 2024-09-16T01:39:59.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.766+0000 7fd2adeaf640 1 -- 172.21.15.88:0/3413674774 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2a815b7d0 con 0x7fd2a81509d0 2024-09-16T01:39:59.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.767+0000 7fd28ffff640 1 -- 172.21.15.88:0/3413674774 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+3917 (secure 0 0 0) 0x7fd2a0079be0 con 0x7fd2a81509d0 2024-09-16T01:39:59.769 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:39:59.769 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":66,"btime":"2024-09-16T01:39:54:934991+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5725,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2696547723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2696547723},{"type":"v1","addr":"172.21.15.154:6834","nonce":2696547723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54},{"gid":5863,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2167482844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2167482844},{"type":"v1","addr":"172.21.15.154:6835","nonce":2167482844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62},{"gid":6411,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3276199399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3276199399},{"type":"v1","addr":"172.21.15.88:6836","nonce":3276199399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54},{"gid":6768,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2345993547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2345993547},{"type":"v1","addr":"172.21.15.88:6837","nonce":2345993547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[]} 2024-09-16T01:39:59.769 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 66 2024-09-16T01:39:59.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.769+0000 7fd2adeaf640 1 -- 172.21.15.88:0/3413674774 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd29406c2e0 msgr2=0x7fd29406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:59.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.769+0000 7fd2adeaf640 1 --2- 172.21.15.88:0/3413674774 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd29406c2e0 0x7fd29406e7a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd298007fd0 tx=0x7fd298009040 comp rx=0 tx=0).stop 2024-09-16T01:39:59.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.769+0000 7fd2adeaf640 1 -- 172.21.15.88:0/3413674774 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2a81509d0 msgr2=0x7fd2a8150e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:59.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.769+0000 7fd2adeaf640 1 --2- 172.21.15.88:0/3413674774 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2a81509d0 0x7fd2a8150e50 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fd2a000e230 tx=0x7fd2a0009750 comp rx=0 tx=0).stop 2024-09-16T01:39:59.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.769+0000 7fd2adeaf640 1 -- 172.21.15.88:0/3413674774 shutdown_connections 2024-09-16T01:39:59.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.769+0000 7fd2adeaf640 1 --2- 172.21.15.88:0/3413674774 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd29406c2e0 0x7fd29406e7a0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:59.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.769+0000 7fd2adeaf640 1 --2- 172.21.15.88:0/3413674774 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2a81509d0 0x7fd2a8150e50 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:59.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.769+0000 7fd2adeaf640 1 --2- 172.21.15.88:0/3413674774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2a81530e0 0x7fd2a8153560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:59.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.769+0000 7fd2adeaf640 1 --2- 172.21.15.88:0/3413674774 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2900051c0 0x7fd2a814ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:59.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.769+0000 7fd2adeaf640 1 -- 172.21.15.88:0/3413674774 >> 172.21.15.88:0/3413674774 conn(0x7fd2a806bc10 msgr2=0x7fd2a806bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:59.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.769+0000 7fd2adeaf640 1 -- 172.21.15.88:0/3413674774 shutdown_connections 2024-09-16T01:39:59.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.769+0000 7fd2adeaf640 1 -- 172.21.15.88:0/3413674774 wait complete. 2024-09-16T01:39:59.782 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-16T01:39:59.782 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-16T01:39:59.782 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-16T01:39:59.782 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:39:59.783 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-16T01:39:59.783 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-16T01:39:59.783 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-16T01:39:59.783 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:39:59.783 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-16T01:39:59.784 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-16T01:39:59.784 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-16T01:39:59.784 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:39:59.784 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-16T01:39:59.785 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-16T01:39:59.785 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-16T01:39:59.785 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:39:59.792 INFO:tasks.ceph.mds.a:Stopped 2024-09-16T01:39:59.792 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-16T01:39:59.793 INFO:tasks.ceph.mds.c:Stopped 2024-09-16T01:39:59.793 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-16T01:39:59.796 INFO:tasks.ceph.mds.b:Stopped 2024-09-16T01:39:59.796 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-16T01:39:59.798 INFO:tasks.ceph.mds.d:Stopped 2024-09-16T01:39:59.798 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-16T01:39:59.835 INFO:tasks.ceph.mds.a:Started 2024-09-16T01:39:59.835 DEBUG:teuthology.parallel:result is None 2024-09-16T01:39:59.836 INFO:tasks.ceph.mds.c:Started 2024-09-16T01:39:59.836 DEBUG:teuthology.parallel:result is None 2024-09-16T01:39:59.840 INFO:tasks.ceph.mds.b:Started 2024-09-16T01:39:59.840 DEBUG:teuthology.parallel:result is None 2024-09-16T01:39:59.841 INFO:tasks.ceph.mds.d:Started 2024-09-16T01:39:59.842 DEBUG:teuthology.parallel:result is None 2024-09-16T01:39:59.842 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-16T01:39:59.909 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:39:59.909 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:39:59.925 INFO:tasks.ceph.mds.b.smithi154.stdout:starting mds.b at 2024-09-16T01:39:59.925 INFO:tasks.ceph.mds.d.smithi154.stdout:starting mds.d at 2024-09-16T01:39:59.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.945+0000 7f244e25e640 1 Processor -- start 2024-09-16T01:39:59.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.945+0000 7f244e25e640 1 -- start start 2024-09-16T01:39:59.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.946+0000 7f244e25e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2448161d70 0x7f24480b3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:59.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.946+0000 7f244e25e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2448162150 0x7f24480b6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:59.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.946+0000 7f244e25e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24480cf2a0 con 0x7f24480d4260 2024-09-16T01:39:59.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.946+0000 7f244e25e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f24480cf760 con 0x7f2448161d70 2024-09-16T01:39:59.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.946+0000 7f244e25e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f24480cf9b0 con 0x7f2448162150 2024-09-16T01:39:59.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.946+0000 7f2447fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f24480d4260 0x7f2448162610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:56072/0 (socket says 172.21.15.88:56072) 2024-09-16T01:39:59.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.946+0000 7f24477fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2448161d70 0x7f24480b3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:59.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.946+0000 7f2447fff640 1 -- 172.21.15.88:0/3627349769 learned_addr learned my addr 172.21.15.88:0/3627349769 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:39:59.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.946+0000 7f2446ffd640 1 --2- 172.21.15.88:0/3627349769 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2448162150 0x7f24480b6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:59.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.946+0000 7f24477fe640 1 -- 172.21.15.88:0/3627349769 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2448162150 msgr2=0x7f24480b6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:39:59.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.946+0000 7f24477fe640 1 --2- 172.21.15.88:0/3627349769 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2448162150 0x7f24480b6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:59.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.946+0000 7f24477fe640 1 -- 172.21.15.88:0/3627349769 >> v1:172.21.15.88:6789/0 conn(0x7f24480d4260 legacy=0x7f2448162610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:59.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.946+0000 7f24477fe640 1 -- 172.21.15.88:0/3627349769 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24480cfc90 con 0x7f2448161d70 2024-09-16T01:39:59.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.947+0000 7f24477fe640 1 --2- 172.21.15.88:0/3627349769 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2448161d70 0x7f24480b3390 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f24380097a0 tx=0x7f243802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a4e1b94b2b2abb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:59.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.950+0000 7f24467fc640 1 -- 172.21.15.88:0/3627349769 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2438037070 con 0x7f2448161d70 2024-09-16T01:39:59.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.950+0000 7f24467fc640 1 -- 172.21.15.88:0/3627349769 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2438032440 con 0x7f2448161d70 2024-09-16T01:39:59.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.950+0000 7f24467fc640 1 -- 172.21.15.88:0/3627349769 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2438032810 con 0x7f2448161d70 2024-09-16T01:39:59.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.950+0000 7f244e25e640 1 -- 172.21.15.88:0/3627349769 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2448161d70 msgr2=0x7f24480b3390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:59.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.950+0000 7f244e25e640 1 --2- 172.21.15.88:0/3627349769 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2448161d70 0x7f24480b3390 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f24380097a0 tx=0x7f243802a180 comp rx=0 tx=0).stop 2024-09-16T01:39:59.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.950+0000 7f244e25e640 1 -- 172.21.15.88:0/3627349769 shutdown_connections 2024-09-16T01:39:59.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.950+0000 7f244e25e640 1 --2- 172.21.15.88:0/3627349769 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2448162150 0x7f24480b6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:59.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.950+0000 7f244e25e640 1 --2- 172.21.15.88:0/3627349769 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2448161d70 0x7f24480b3390 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:59.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.950+0000 7f244e25e640 1 -- 172.21.15.88:0/3627349769 >> 172.21.15.88:0/3627349769 conn(0x7f244806c500 msgr2=0x7f244806c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:39:59.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.950+0000 7f244e25e640 1 -- 172.21.15.88:0/3627349769 shutdown_connections 2024-09-16T01:39:59.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.951+0000 7f244e25e640 1 -- 172.21.15.88:0/3627349769 wait complete. 2024-09-16T01:39:59.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.952+0000 7f244e25e640 1 Processor -- start 2024-09-16T01:39:59.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.952+0000 7f244e25e640 1 -- start start 2024-09-16T01:39:59.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.953+0000 7f244e25e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24480d4260 0x7f2448159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:59.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.953+0000 7f244e25e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2448161d70 0x7f24480b5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:59.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.953+0000 7f244e25e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2448162150 0x7f2448158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:59.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.953+0000 7f244e25e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f24480ceed0 con 0x7f2448161d70 2024-09-16T01:39:59.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.953+0000 7f244e25e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f24480ced50 con 0x7f24480d4260 2024-09-16T01:39:59.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.953+0000 7f244e25e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f24480cf050 con 0x7f2448162150 2024-09-16T01:39:59.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.953+0000 7f2446ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2448161d70 0x7f24480b5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:59.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.953+0000 7f2446ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2448161d70 0x7f24480b5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60464/0 (socket says 172.21.15.88:60464) 2024-09-16T01:39:59.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.953+0000 7f2446ffd640 1 -- 172.21.15.88:0/3452436042 learned_addr learned my addr 172.21.15.88:0/3452436042 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:39:59.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.953+0000 7f2447fff640 1 --2- 172.21.15.88:0/3452436042 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2448162150 0x7f2448158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:59.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.953+0000 7f24477fe640 1 --2- 172.21.15.88:0/3452436042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24480d4260 0x7f2448159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:59.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.953+0000 7f2446ffd640 1 -- 172.21.15.88:0/3452436042 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2448162150 msgr2=0x7f2448158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:59.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.953+0000 7f2446ffd640 1 --2- 172.21.15.88:0/3452436042 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2448162150 0x7f2448158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:59.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.953+0000 7f2446ffd640 1 -- 172.21.15.88:0/3452436042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24480d4260 msgr2=0x7f2448159fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:39:59.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.953+0000 7f2446ffd640 1 --2- 172.21.15.88:0/3452436042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24480d4260 0x7f2448159fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:39:59.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.953+0000 7f2446ffd640 1 -- 172.21.15.88:0/3452436042 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2448158f40 con 0x7f2448161d70 2024-09-16T01:39:59.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.953+0000 7f2446ffd640 1 --2- 172.21.15.88:0/3452436042 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2448161d70 0x7f24480b5bd0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f243c00e8a0 tx=0x7f243c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:59.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.956+0000 7f242ffff640 1 -- 172.21.15.88:0/3452436042 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c017070 con 0x7f2448161d70 2024-09-16T01:39:59.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.956+0000 7f244e25e640 1 -- 172.21.15.88:0/3452436042 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2448159220 con 0x7f2448161d70 2024-09-16T01:39:59.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.956+0000 7f244e25e640 1 -- 172.21.15.88:0/3452436042 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24481597e0 con 0x7f2448161d70 2024-09-16T01:39:59.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.956+0000 7f242ffff640 1 -- 172.21.15.88:0/3452436042 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f243c00ee90 con 0x7f2448161d70 2024-09-16T01:39:59.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.956+0000 7f242ffff640 1 -- 172.21.15.88:0/3452436042 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c004220 con 0x7f2448161d70 2024-09-16T01:39:59.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.957+0000 7f244e25e640 1 -- 172.21.15.88:0/3452436042 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2428005290 con 0x7f2448161d70 2024-09-16T01:39:59.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.957+0000 7f242ffff640 1 -- 172.21.15.88:0/3452436042 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f243c01a070 con 0x7f2448161d70 2024-09-16T01:39:59.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.958+0000 7f242ffff640 1 --2- 172.21.15.88:0/3452436042 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f243406c2b0 0x7f243406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:39:59.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.958+0000 7f242ffff640 1 -- 172.21.15.88:0/3452436042 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 5029+0+0 (secure 0 0 0) 0x7f243c08b250 con 0x7f2448161d70 2024-09-16T01:39:59.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.958+0000 7f24477fe640 1 --2- 172.21.15.88:0/3452436042 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f243406c2b0 0x7f243406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:39:59.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.959+0000 7f24477fe640 1 --2- 172.21.15.88:0/3452436042 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f243406c2b0 0x7f243406e770 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2438034000 tx=0x7f243802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:39:59.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:39:59.962+0000 7f242ffff640 1 -- 172.21.15.88:0/3452436042 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f243c021030 con 0x7f2448161d70 2024-09-16T01:40:00.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:00.140+0000 7f244e25e640 1 -- 172.21.15.88:0/3452436042 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f2428005850 con 0x7f2448161d70 2024-09-16T01:40:00.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:00.964+0000 7f242ffff640 1 -- 172.21.15.88:0/3452436042 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v81) ==== 97+0+0 (secure 0 0 0) 0x7f243c0595e0 con 0x7f2448161d70 2024-09-16T01:40:00.966 INFO:teuthology.orchestra.run.smithi088.stderr: removed all blocklist entries 2024-09-16T01:40:00.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:00.967+0000 7f244e25e640 1 -- 172.21.15.88:0/3452436042 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f243406c2b0 msgr2=0x7f243406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:00.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:00.967+0000 7f244e25e640 1 --2- 172.21.15.88:0/3452436042 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f243406c2b0 0x7f243406e770 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2438034000 tx=0x7f243802e000 comp rx=0 tx=0).stop 2024-09-16T01:40:00.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:00.967+0000 7f244e25e640 1 -- 172.21.15.88:0/3452436042 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2448161d70 msgr2=0x7f24480b5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:00.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:00.967+0000 7f244e25e640 1 --2- 172.21.15.88:0/3452436042 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2448161d70 0x7f24480b5bd0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f243c00e8a0 tx=0x7f243c00ed60 comp rx=0 tx=0).stop 2024-09-16T01:40:00.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:00.967+0000 7f244e25e640 1 -- 172.21.15.88:0/3452436042 shutdown_connections 2024-09-16T01:40:00.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:00.967+0000 7f244e25e640 1 --2- 172.21.15.88:0/3452436042 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f243406c2b0 0x7f243406e770 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:00.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:00.967+0000 7f244e25e640 1 --2- 172.21.15.88:0/3452436042 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2448162150 0x7f2448158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:00.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:00.967+0000 7f244e25e640 1 --2- 172.21.15.88:0/3452436042 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2448161d70 0x7f24480b5bd0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:00.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:00.967+0000 7f244e25e640 1 --2- 172.21.15.88:0/3452436042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24480d4260 0x7f2448159fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:00.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:00.967+0000 7f244e25e640 1 -- 172.21.15.88:0/3452436042 >> 172.21.15.88:0/3452436042 conn(0x7f244806c500 msgr2=0x7f24480d32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:00.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:00.967+0000 7f244e25e640 1 -- 172.21.15.88:0/3452436042 shutdown_connections 2024-09-16T01:40:00.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:00.967+0000 7f244e25e640 1 -- 172.21.15.88:0/3452436042 wait complete. 2024-09-16T01:40:00.982 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-16T01:40:01.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.086+0000 7fc821be4640 1 Processor -- start 2024-09-16T01:40:01.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.086+0000 7fc821be4640 1 -- start start 2024-09-16T01:40:01.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.087+0000 7fc821be4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc81c15e540 0x7fc81c0b52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:01.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.087+0000 7fc821be4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc81c073180 0x7fc81c0735c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:01.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.087+0000 7fc821be4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc81c0b5800 con 0x7fc81c0b01f0 2024-09-16T01:40:01.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.087+0000 7fc821be4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc81c163010 con 0x7fc81c15e540 2024-09-16T01:40:01.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.087+0000 7fc821be4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc81c073b00 con 0x7fc81c073180 2024-09-16T01:40:01.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.087+0000 7fc81affd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc81c0b01f0 0x7fc81c1620e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33240/0 (socket says 172.21.15.88:33240) 2024-09-16T01:40:01.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.087+0000 7fc81affd640 1 -- 172.21.15.88:0/43517413 learned_addr learned my addr 172.21.15.88:0/43517413 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:01.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.087+0000 7fc81b7fe640 1 --2- 172.21.15.88:0/43517413 >> v2:172.21.15.154:3300/0 conn(0x7fc81c15e540 0x7fc81c0b52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:01.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.087+0000 7fc81bfff640 1 --2- 172.21.15.88:0/43517413 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc81c073180 0x7fc81c0735c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:01.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.087+0000 7fc81a7fc640 1 -- 172.21.15.88:0/43517413 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1845163409 0 0) 0x7fc81c0b5800 con 0x7fc81c0b01f0 2024-09-16T01:40:01.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.087+0000 7fc81a7fc640 1 -- 172.21.15.88:0/43517413 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc808003520 con 0x7fc81c0b01f0 2024-09-16T01:40:01.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.087+0000 7fc81a7fc640 1 -- 172.21.15.88:0/43517413 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 113306700 0 0) 0x7fc808003520 con 0x7fc81c0b01f0 2024-09-16T01:40:01.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.087+0000 7fc81a7fc640 1 -- 172.21.15.88:0/43517413 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc81c073180 msgr2=0x7fc81c0735c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:01.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.087+0000 7fc81a7fc640 1 --2- 172.21.15.88:0/43517413 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc81c073180 0x7fc81c0735c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:01.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.087+0000 7fc81a7fc640 1 -- 172.21.15.88:0/43517413 >> v2:172.21.15.154:3300/0 conn(0x7fc81c15e540 msgr2=0x7fc81c0b52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:01.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.087+0000 7fc81a7fc640 1 --2- 172.21.15.88:0/43517413 >> v2:172.21.15.154:3300/0 conn(0x7fc81c15e540 0x7fc81c0b52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:01.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.087+0000 7fc81a7fc640 1 -- 172.21.15.88:0/43517413 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc81c15e920 con 0x7fc81c0b01f0 2024-09-16T01:40:01.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.087+0000 7fc81a7fc640 1 -- 172.21.15.88:0/43517413 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fc804001670 con 0x7fc81c0b01f0 2024-09-16T01:40:01.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.087+0000 7fc81a7fc640 1 -- 172.21.15.88:0/43517413 >> v1:172.21.15.88:6789/0 conn(0x7fc81c0b01f0 legacy=0x7fc81c1620e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:01.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.088+0000 7fc81a7fc640 1 --2- 172.21.15.88:0/43517413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8080039b0 0x7fc808003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:01.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.088+0000 7fc81a7fc640 1 --2- 172.21.15.88:0/43517413 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc808004800 0x7fc808004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:01.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.088+0000 7fc81a7fc640 1 --2- 172.21.15.88:0/43517413 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc81c073180 0x7fc8080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:01.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.088+0000 7fc81a7fc640 1 -- 172.21.15.88:0/43517413 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc81c163010 con 0x7fc808004800 2024-09-16T01:40:01.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.088+0000 7fc81a7fc640 1 -- 172.21.15.88:0/43517413 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc81c073b00 con 0x7fc8080039b0 2024-09-16T01:40:01.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.088+0000 7fc81a7fc640 1 -- 172.21.15.88:0/43517413 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc80800ba40 con 0x7fc81c073180 2024-09-16T01:40:01.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.088+0000 7fc81affd640 1 --2- 172.21.15.88:0/43517413 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc808004800 0x7fc808004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:01.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.088+0000 7fc81b7fe640 1 --2- 172.21.15.88:0/43517413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8080039b0 0x7fc808003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:01.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.088+0000 7fc81bfff640 1 --2- 172.21.15.88:0/43517413 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc81c073180 0x7fc8080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:01.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.088+0000 7fc81affd640 1 -- 172.21.15.88:0/43517413 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc81c073180 msgr2=0x7fc8080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:01.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.088+0000 7fc81affd640 1 --2- 172.21.15.88:0/43517413 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc81c073180 0x7fc8080051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:01.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.088+0000 7fc81affd640 1 -- 172.21.15.88:0/43517413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8080039b0 msgr2=0x7fc808003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:01.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.088+0000 7fc81affd640 1 --2- 172.21.15.88:0/43517413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8080039b0 0x7fc808003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:01.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.088+0000 7fc81affd640 1 -- 172.21.15.88:0/43517413 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc808003520 con 0x7fc808004800 2024-09-16T01:40:01.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.088+0000 7fc81b7fe640 1 --2- 172.21.15.88:0/43517413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8080039b0 0x7fc808003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:01.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.089+0000 7fc81affd640 1 --2- 172.21.15.88:0/43517413 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc808004800 0x7fc808004c80 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fc804007fd0 tx=0x7fc808005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eda110d41c39a2c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:01.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.089+0000 7fc81a7fc640 1 -- 172.21.15.88:0/43517413 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc804007d50 con 0x7fc808004800 2024-09-16T01:40:01.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.089+0000 7fc81a7fc640 1 -- 172.21.15.88:0/43517413 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc804005ad0 con 0x7fc808004800 2024-09-16T01:40:01.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.089+0000 7fc81a7fc640 1 -- 172.21.15.88:0/43517413 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc804007070 con 0x7fc808004800 2024-09-16T01:40:01.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.089+0000 7fc821be4640 1 -- 172.21.15.88:0/43517413 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc808004800 msgr2=0x7fc808004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:01.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.089+0000 7fc821be4640 1 --2- 172.21.15.88:0/43517413 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc808004800 0x7fc808004c80 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fc804007fd0 tx=0x7fc808005830 comp rx=0 tx=0).stop 2024-09-16T01:40:01.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.089+0000 7fc81bfff640 1 -- 172.21.15.88:0/43517413 reap_dead start 2024-09-16T01:40:01.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.089+0000 7fc821be4640 1 -- 172.21.15.88:0/43517413 shutdown_connections 2024-09-16T01:40:01.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.089+0000 7fc821be4640 1 -- 172.21.15.88:0/43517413 >> 172.21.15.88:0/43517413 conn(0x7fc81c06bc10 msgr2=0x7fc81c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:01.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.089+0000 7fc821be4640 1 -- 172.21.15.88:0/43517413 shutdown_connections 2024-09-16T01:40:01.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.089+0000 7fc821be4640 1 -- 172.21.15.88:0/43517413 wait complete. 2024-09-16T01:40:01.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.090+0000 7fc821be4640 1 Processor -- start 2024-09-16T01:40:01.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.091+0000 7fc821be4640 1 -- start start 2024-09-16T01:40:01.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.091+0000 7fc821be4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc808004800 0x7fc81c14f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:01.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.091+0000 7fc821be4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc81c0cf140 0x7fc81c0cf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:01.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.091+0000 7fc821be4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc81c15b210 0x7fc81c15b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:01.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.091+0000 7fc821be4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc81c162d30 con 0x7fc81c0cf140 2024-09-16T01:40:01.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.091+0000 7fc821be4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc81c1627f0 con 0x7fc808004800 2024-09-16T01:40:01.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.091+0000 7fc821be4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc81c0ae9c0 con 0x7fc81c15b210 2024-09-16T01:40:01.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.091+0000 7fc81affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc81c0cf140 0x7fc81c0cf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:01.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.091+0000 7fc81affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc81c0cf140 0x7fc81c0cf5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59870/0 (socket says 172.21.15.88:59870) 2024-09-16T01:40:01.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.091+0000 7fc81affd640 1 -- 172.21.15.88:0/2920038064 learned_addr learned my addr 172.21.15.88:0/2920038064 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:01.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.092+0000 7fc81bfff640 1 --2- 172.21.15.88:0/2920038064 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc81c15b210 0x7fc81c15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:01.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.092+0000 7fc81b7fe640 1 --2- 172.21.15.88:0/2920038064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc808004800 0x7fc81c14f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:01.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.092+0000 7fc81affd640 1 -- 172.21.15.88:0/2920038064 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc81c15b210 msgr2=0x7fc81c15b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:01.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.092+0000 7fc81affd640 1 --2- 172.21.15.88:0/2920038064 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc81c15b210 0x7fc81c15b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:01.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.092+0000 7fc81affd640 1 -- 172.21.15.88:0/2920038064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc808004800 msgr2=0x7fc81c14f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:01.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.092+0000 7fc81affd640 1 --2- 172.21.15.88:0/2920038064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc808004800 0x7fc81c14f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:01.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.092+0000 7fc81affd640 1 -- 172.21.15.88:0/2920038064 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc81c0ce0a0 con 0x7fc81c0cf140 2024-09-16T01:40:01.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.092+0000 7fc81bfff640 1 --2- 172.21.15.88:0/2920038064 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc81c15b210 0x7fc81c15b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:01.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.092+0000 7fc81b7fe640 1 --2- 172.21.15.88:0/2920038064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc808004800 0x7fc81c14f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:01.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.092+0000 7fc81affd640 1 --2- 172.21.15.88:0/2920038064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc81c0cf140 0x7fc81c0cf5c0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fc80402ae20 tx=0x7fc81c15eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:01.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.092+0000 7fc813fff640 1 -- 172.21.15.88:0/2920038064 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc804042020 con 0x7fc81c0cf140 2024-09-16T01:40:01.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.092+0000 7fc813fff640 1 -- 172.21.15.88:0/2920038064 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc804004bf0 con 0x7fc81c0cf140 2024-09-16T01:40:01.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.092+0000 7fc821be4640 1 -- 172.21.15.88:0/2920038064 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc81c0ce380 con 0x7fc81c0cf140 2024-09-16T01:40:01.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.092+0000 7fc813fff640 1 -- 172.21.15.88:0/2920038064 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc804007070 con 0x7fc81c0cf140 2024-09-16T01:40:01.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.092+0000 7fc821be4640 1 -- 172.21.15.88:0/2920038064 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc81c0ce940 con 0x7fc81c0cf140 2024-09-16T01:40:01.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.093+0000 7fc821be4640 1 -- 172.21.15.88:0/2920038064 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc81c0d0550 con 0x7fc81c0cf140 2024-09-16T01:40:01.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.093+0000 7fc813fff640 1 -- 172.21.15.88:0/2920038064 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc804045020 con 0x7fc81c0cf140 2024-09-16T01:40:01.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.094+0000 7fc813fff640 1 --2- 172.21.15.88:0/2920038064 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc7fc06c2b0 0x7fc7fc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:01.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.094+0000 7fc813fff640 1 -- 172.21.15.88:0/2920038064 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 4857+0+0 (secure 0 0 0) 0x7fc8040ab860 con 0x7fc81c0cf140 2024-09-16T01:40:01.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.094+0000 7fc81b7fe640 1 --2- 172.21.15.88:0/2920038064 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc7fc06c2b0 0x7fc7fc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:01.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.094+0000 7fc81b7fe640 1 --2- 172.21.15.88:0/2920038064 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc7fc06c2b0 0x7fc7fc06e770 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc81c156c20 tx=0x7fc80c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:01.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.098+0000 7fc813fff640 1 -- 172.21.15.88:0/2920038064 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc81c0d0550 con 0x7fc81c0cf140 2024-09-16T01:40:01.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.329+0000 7fc821be4640 1 -- 172.21.15.88:0/2920038064 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fc81c0cd000 con 0x7fc81c0cf140 2024-09-16T01:40:01.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.329+0000 7fc813fff640 1 -- 172.21.15.88:0/2920038064 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) ==== 80+0+5304 (secure 0 0 0) 0x7fc81c0cd000 con 0x7fc81c0cf140 2024-09-16T01:40:01.331 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:01.331 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:40:01.331 INFO:teuthology.orchestra.run.smithi088.stdout: "auth_dump": [ 2024-09-16T01:40:01.331 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:40:01.331 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mds.a", 2024-09-16T01:40:01.331 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmy+jJCRAAhnzX/li6D3D07NmThrjXkA==", 2024-09-16T01:40:01.331 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:40:01.331 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:40:01.331 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:40:01.331 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:40:01.331 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:40:01.331 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:40:01.332 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:40:01.332 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:40:01.332 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mds.b", 2024-09-16T01:40:01.332 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmx7p+FRAApxKUSZoR/nGUZsaYKlrYUg==", 2024-09-16T01:40:01.332 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:40:01.332 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:40:01.332 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:40:01.332 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:40:01.332 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:40:01.332 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:40:01.332 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:40:01.332 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:40:01.332 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mds.c", 2024-09-16T01:40:01.333 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmaYHUEBAAzM0qE53LEgHi+hThHJX3Tw==", 2024-09-16T01:40:01.333 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:40:01.333 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:40:01.333 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:40:01.333 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:40:01.333 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:40:01.333 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:40:01.333 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:40:01.333 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:40:01.333 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mds.d", 2024-09-16T01:40:01.333 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudm3TSLHBAAJzlR6H09cc1Pki2FjyoFiA==", 2024-09-16T01:40:01.333 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:40:01.333 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:40:01.334 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:40:01.334 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:40:01.334 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:40:01.334 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:40:01.334 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:40:01.334 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:40:01.334 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.0", 2024-09-16T01:40:01.334 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDhiudmZR7MORAAwDKvphubA+CbzBUWuaA80A==", 2024-09-16T01:40:01.334 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:40:01.334 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:40:01.334 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:40:01.334 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:40:01.334 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:40:01.335 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:40:01.335 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:40:01.335 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.1", 2024-09-16T01:40:01.335 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDliudmNd+SLBAArOUdoZ226c5jlAsKwqERmw==", 2024-09-16T01:40:01.335 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:40:01.335 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:40:01.335 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:40:01.335 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:40:01.335 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:40:01.335 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:40:01.335 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:40:01.335 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.2", 2024-09-16T01:40:01.335 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDpiudm75CUARAAWHtYOg9pE8g+M6IL+8XZ6g==", 2024-09-16T01:40:01.335 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:40:01.336 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:40:01.336 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:40:01.336 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:40:01.336 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:40:01.336 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:40:01.336 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:40:01.336 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.3", 2024-09-16T01:40:01.336 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDtiudm0fgWAxAAf2NOWrpXbOhb3dFXozJagQ==", 2024-09-16T01:40:01.336 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:40:01.336 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:40:01.336 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:40:01.336 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:40:01.337 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:40:01.337 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:40:01.337 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:40:01.337 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.4", 2024-09-16T01:40:01.337 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQD0iudmdddMCxAAtr2qfj4Nxj3/GHUpxRmtFg==", 2024-09-16T01:40:01.337 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:40:01.337 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:40:01.337 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:40:01.337 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:40:01.337 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:40:01.337 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:40:01.337 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:40:01.337 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.5", 2024-09-16T01:40:01.337 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQD3iudmT5RjOhAAcKwca6BG60gsxYnmy4twGg==", 2024-09-16T01:40:01.338 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:40:01.338 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:40:01.338 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:40:01.338 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:40:01.338 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:40:01.338 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:40:01.338 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:40:01.338 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.6", 2024-09-16T01:40:01.338 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQD7iudmT7daLRAAdfLFijKQku3rJcJ5xcfLsw==", 2024-09-16T01:40:01.338 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:40:01.338 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:40:01.338 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:40:01.338 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:40:01.338 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:40:01.339 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:40:01.339 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:40:01.339 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.7", 2024-09-16T01:40:01.339 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQD/iudmmvtxLhAArYKhQ4zyFHtLgh/+bs5ZhA==", 2024-09-16T01:40:01.339 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:40:01.339 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:40:01.339 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:40:01.339 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:40:01.339 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:40:01.339 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:40:01.339 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:40:01.339 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.0", 2024-09-16T01:40:01.339 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmvPikIBAAyHxNNnwQKygxg09PNQLc/Q==", 2024-09-16T01:40:01.340 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:40:01.340 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:40:01.340 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow r", 2024-09-16T01:40:01.340 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-16T01:40:01.340 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:40:01.340 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:40:01.340 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:40:01.340 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.1", 2024-09-16T01:40:01.340 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmA69iKRAA8Abwe0Lg2PnZDErgJ1gEbQ==", 2024-09-16T01:40:01.340 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:40:01.340 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:40:01.340 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow r", 2024-09-16T01:40:01.340 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow rw", 2024-09-16T01:40:01.340 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow rwx" 2024-09-16T01:40:01.341 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:40:01.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:40:01.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:40:01.341 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.2", 2024-09-16T01:40:01.341 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmB+w9JhAAv63Is5T/cvDbbLOjIkI0wA==", 2024-09-16T01:40:01.341 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:40:01.341 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:40:01.341 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow r", 2024-09-16T01:40:01.341 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow rw", 2024-09-16T01:40:01.341 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow rwx" 2024-09-16T01:40:01.341 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:40:01.341 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:40:01.341 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:40:01.341 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.3", 2024-09-16T01:40:01.342 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudm4WoFLxAAnYGHGqHyBHHajQ8f3rCjEA==", 2024-09-16T01:40:01.342 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:40:01.342 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:40:01.342 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow r", 2024-09-16T01:40:01.342 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow rw", 2024-09-16T01:40:01.342 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow rwx" 2024-09-16T01:40:01.342 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:40:01.342 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:40:01.342 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:40:01.342 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.admin", 2024-09-16T01:40:01.342 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDdiudmhcE0KBAANauQKyUY01/KlWMrN2x/5Q==", 2024-09-16T01:40:01.342 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:40:01.342 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow *", 2024-09-16T01:40:01.343 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:40:01.343 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:40:01.343 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:40:01.343 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:40:01.343 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:40:01.343 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:40:01.343 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mgr.x", 2024-09-16T01:40:01.343 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmToGYBhAAKwXS34PtfAVtEtdtHKHlQQ==", 2024-09-16T01:40:01.343 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:40:01.343 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow *", 2024-09-16T01:40:01.344 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile mgr", 2024-09-16T01:40:01.344 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:40:01.344 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:40:01.344 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:40:01.344 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:40:01.344 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mgr.y", 2024-09-16T01:40:01.344 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudm/BN5ARAAxbCopbsvxFUZP3drso32qw==", 2024-09-16T01:40:01.344 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:40:01.344 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow *", 2024-09-16T01:40:01.344 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile mgr", 2024-09-16T01:40:01.344 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:40:01.344 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:40:01.344 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:40:01.345 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-09-16T01:40:01.345 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:40:01.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.331+0000 7fc821be4640 1 -- 172.21.15.88:0/2920038064 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc7fc06c2b0 msgr2=0x7fc7fc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:01.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.331+0000 7fc821be4640 1 --2- 172.21.15.88:0/2920038064 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc7fc06c2b0 0x7fc7fc06e770 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc81c156c20 tx=0x7fc80c008040 comp rx=0 tx=0).stop 2024-09-16T01:40:01.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.331+0000 7fc821be4640 1 -- 172.21.15.88:0/2920038064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc81c0cf140 msgr2=0x7fc81c0cf5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:01.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.331+0000 7fc821be4640 1 --2- 172.21.15.88:0/2920038064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc81c0cf140 0x7fc81c0cf5c0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fc80402ae20 tx=0x7fc81c15eaf0 comp rx=0 tx=0).stop 2024-09-16T01:40:01.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.331+0000 7fc821be4640 1 -- 172.21.15.88:0/2920038064 shutdown_connections 2024-09-16T01:40:01.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.331+0000 7fc821be4640 1 --2- 172.21.15.88:0/2920038064 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc7fc06c2b0 0x7fc7fc06e770 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:01.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.331+0000 7fc821be4640 1 --2- 172.21.15.88:0/2920038064 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc81c15b210 0x7fc81c15b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:01.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.331+0000 7fc821be4640 1 --2- 172.21.15.88:0/2920038064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc81c0cf140 0x7fc81c0cf5c0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:01.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.331+0000 7fc821be4640 1 --2- 172.21.15.88:0/2920038064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc808004800 0x7fc81c14f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:01.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.331+0000 7fc821be4640 1 -- 172.21.15.88:0/2920038064 >> 172.21.15.88:0/2920038064 conn(0x7fc81c06bc10 msgr2=0x7fc81c1620c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:01.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.331+0000 7fc821be4640 1 -- 172.21.15.88:0/2920038064 shutdown_connections 2024-09-16T01:40:01.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.331+0000 7fc821be4640 1 -- 172.21.15.88:0/2920038064 wait complete. 2024-09-16T01:40:01.347 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-16T01:40:01.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.445+0000 7febc2df1640 1 Processor -- start 2024-09-16T01:40:01.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.445+0000 7febc2df1640 1 -- start start 2024-09-16T01:40:01.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.446+0000 7febc2df1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7febbc0d3310 0x7febbc0d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:01.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.446+0000 7febc2df1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febbc15ea40 con 0x7febbc15e660 2024-09-16T01:40:01.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.446+0000 7febc2df1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febbc0d3870 con 0x7febbc0d1650 2024-09-16T01:40:01.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.446+0000 7febc2df1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7febbc0d1ac0 con 0x7febbc0d3310 2024-09-16T01:40:01.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.446+0000 7febbbfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7febbc0d3310 0x7febbc0d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:01.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.446+0000 7febc0b8d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7febbc15e660 0x7febbc0cf730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33254/0 (socket says 172.21.15.88:33254) 2024-09-16T01:40:01.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.446+0000 7febc0b8d640 1 -- 172.21.15.88:0/3032752339 learned_addr learned my addr 172.21.15.88:0/3032752339 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:01.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.446+0000 7febbbfff640 1 -- 172.21.15.88:0/3032752339 >> v1:172.21.15.154:6789/0 conn(0x7febbc0d1650 legacy=0x7febbc0b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:01.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.446+0000 7febbbfff640 1 -- 172.21.15.88:0/3032752339 >> v1:172.21.15.88:6789/0 conn(0x7febbc15e660 legacy=0x7febbc0cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:01.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.446+0000 7febbb7fe640 1 -- 172.21.15.88:0/3032752339 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 636450583 0 0) 0x7febbc15ea40 con 0x7febbc15e660 2024-09-16T01:40:01.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.446+0000 7febbbfff640 1 -- 172.21.15.88:0/3032752339 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febbc0d3b50 con 0x7febbc0d3310 2024-09-16T01:40:01.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.447+0000 7febbbfff640 1 --2- 172.21.15.88:0/3032752339 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7febbc0d3310 0x7febbc0d1110 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7febb00097a0 tx=0x7febb002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=419b232259d3d7b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:01.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.447+0000 7febbb7fe640 1 -- 172.21.15.88:0/3032752339 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb0037070 con 0x7febbc0d3310 2024-09-16T01:40:01.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.447+0000 7febbb7fe640 1 -- 172.21.15.88:0/3032752339 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febb0032440 con 0x7febbc0d3310 2024-09-16T01:40:01.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.447+0000 7febbb7fe640 1 -- 172.21.15.88:0/3032752339 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb0032810 con 0x7febbc0d3310 2024-09-16T01:40:01.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.448+0000 7febc2df1640 1 -- 172.21.15.88:0/3032752339 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7febbc0d3310 msgr2=0x7febbc0d1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:01.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.448+0000 7febc2df1640 1 --2- 172.21.15.88:0/3032752339 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7febbc0d3310 0x7febbc0d1110 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7febb00097a0 tx=0x7febb002a230 comp rx=0 tx=0).stop 2024-09-16T01:40:01.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.448+0000 7febc2df1640 1 -- 172.21.15.88:0/3032752339 shutdown_connections 2024-09-16T01:40:01.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.448+0000 7febc2df1640 1 --2- 172.21.15.88:0/3032752339 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7febbc0d3310 0x7febbc0d1110 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:01.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.448+0000 7febc2df1640 1 -- 172.21.15.88:0/3032752339 >> 172.21.15.88:0/3032752339 conn(0x7febbc0b6410 msgr2=0x7febbc0b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:01.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.448+0000 7febc2df1640 1 -- 172.21.15.88:0/3032752339 shutdown_connections 2024-09-16T01:40:01.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.448+0000 7febc2df1640 1 -- 172.21.15.88:0/3032752339 wait complete. 2024-09-16T01:40:01.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.449+0000 7febc2df1640 1 Processor -- start 2024-09-16T01:40:01.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.449+0000 7febc2df1640 1 -- start start 2024-09-16T01:40:01.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.450+0000 7febc2df1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febbc0d1650 0x7febbc1f4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:01.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.450+0000 7febc2df1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7febbc0d3310 0x7febbc0c8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:01.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.450+0000 7febc2df1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febbc15e660 0x7febbc14c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:01.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.450+0000 7febc2df1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7febbc0d43f0 con 0x7febbc0d3310 2024-09-16T01:40:01.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.450+0000 7febc2df1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7febbc0d4110 con 0x7febbc0d1650 2024-09-16T01:40:01.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.450+0000 7febc2df1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7febbc0b8280 con 0x7febbc15e660 2024-09-16T01:40:01.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.450+0000 7febbbfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7febbc0d3310 0x7febbc0c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:01.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.450+0000 7febbbfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7febbc0d3310 0x7febbc0c8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59888/0 (socket says 172.21.15.88:59888) 2024-09-16T01:40:01.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.450+0000 7febbbfff640 1 -- 172.21.15.88:0/2010425001 learned_addr learned my addr 172.21.15.88:0/2010425001 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:01.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.450+0000 7febc0b8d640 1 --2- 172.21.15.88:0/2010425001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febbc0d1650 0x7febbc1f4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:01.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.450+0000 7febc138e640 1 --2- 172.21.15.88:0/2010425001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febbc15e660 0x7febbc14c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:01.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.450+0000 7febbbfff640 1 -- 172.21.15.88:0/2010425001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febbc15e660 msgr2=0x7febbc14c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:01.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.450+0000 7febbbfff640 1 --2- 172.21.15.88:0/2010425001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febbc15e660 0x7febbc14c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:01.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.450+0000 7febbbfff640 1 -- 172.21.15.88:0/2010425001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febbc0d1650 msgr2=0x7febbc1f4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:01.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.450+0000 7febbbfff640 1 --2- 172.21.15.88:0/2010425001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febbc0d1650 0x7febbc1f4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:01.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.450+0000 7febbbfff640 1 -- 172.21.15.88:0/2010425001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febbc14cf30 con 0x7febbc0d3310 2024-09-16T01:40:01.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.451+0000 7febbbfff640 1 --2- 172.21.15.88:0/2010425001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7febbc0d3310 0x7febbc0c8080 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7febb0000c00 tx=0x7febb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:01.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.451+0000 7febb97fa640 1 -- 172.21.15.88:0/2010425001 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb0037040 con 0x7febbc0d3310 2024-09-16T01:40:01.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.451+0000 7febb97fa640 1 -- 172.21.15.88:0/2010425001 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febb0042070 con 0x7febbc0d3310 2024-09-16T01:40:01.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.451+0000 7febc2df1640 1 -- 172.21.15.88:0/2010425001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febbc0cb120 con 0x7febbc0d3310 2024-09-16T01:40:01.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.451+0000 7febb97fa640 1 -- 172.21.15.88:0/2010425001 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb0030a70 con 0x7febbc0d3310 2024-09-16T01:40:01.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.451+0000 7febc2df1640 1 -- 172.21.15.88:0/2010425001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febbc15b2f0 con 0x7febbc0d3310 2024-09-16T01:40:01.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.452+0000 7febc2df1640 1 -- 172.21.15.88:0/2010425001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febbc0d1110 con 0x7febbc0d3310 2024-09-16T01:40:01.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.452+0000 7febb97fa640 1 -- 172.21.15.88:0/2010425001 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7febb0030c10 con 0x7febbc0d3310 2024-09-16T01:40:01.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.453+0000 7febb97fa640 1 --2- 172.21.15.88:0/2010425001 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7feba806c2b0 0x7feba806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:01.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.453+0000 7febb97fa640 1 -- 172.21.15.88:0/2010425001 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 4857+0+0 (secure 0 0 0) 0x7febb00abae0 con 0x7febbc0d3310 2024-09-16T01:40:01.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.453+0000 7febc0b8d640 1 --2- 172.21.15.88:0/2010425001 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7feba806c2b0 0x7feba806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:01.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.453+0000 7febc0b8d640 1 --2- 172.21.15.88:0/2010425001 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7feba806c2b0 0x7feba806e770 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7febac000f60 tx=0x7febac007e40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:01.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.457+0000 7febb97fa640 1 -- 172.21.15.88:0/2010425001 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7febb007a5b0 con 0x7febbc0d3310 2024-09-16T01:40:01.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.625+0000 7febc2df1640 1 -- 172.21.15.88:0/2010425001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7febbc0b5570 con 0x7febbc0d3310 2024-09-16T01:40:01.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.626+0000 7febb97fa640 1 -- 172.21.15.88:0/2010425001 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7febb0079eb0 con 0x7febbc0d3310 2024-09-16T01:40:01.627 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:01.627 INFO:teuthology.orchestra.run.smithi088.stdout:[ 2024-09-16T01:40:01.627 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:40:01.627 INFO:teuthology.orchestra.run.smithi088.stdout: "poolnum": 1, 2024-09-16T01:40:01.627 INFO:teuthology.orchestra.run.smithi088.stdout: "poolname": ".mgr" 2024-09-16T01:40:01.628 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:40:01.628 INFO:teuthology.orchestra.run.smithi088.stdout:] 2024-09-16T01:40:01.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.627+0000 7febc2df1640 1 -- 172.21.15.88:0/2010425001 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7feba806c2b0 msgr2=0x7feba806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:01.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.627+0000 7febc2df1640 1 --2- 172.21.15.88:0/2010425001 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7feba806c2b0 0x7feba806e770 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7febac000f60 tx=0x7febac007e40 comp rx=0 tx=0).stop 2024-09-16T01:40:01.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.627+0000 7febc2df1640 1 -- 172.21.15.88:0/2010425001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7febbc0d3310 msgr2=0x7febbc0c8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:01.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.627+0000 7febc2df1640 1 --2- 172.21.15.88:0/2010425001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7febbc0d3310 0x7febbc0c8080 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7febb0000c00 tx=0x7febb00046c0 comp rx=0 tx=0).stop 2024-09-16T01:40:01.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.627+0000 7febc2df1640 1 -- 172.21.15.88:0/2010425001 shutdown_connections 2024-09-16T01:40:01.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.627+0000 7febc2df1640 1 --2- 172.21.15.88:0/2010425001 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7feba806c2b0 0x7feba806e770 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:01.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.627+0000 7febc2df1640 1 --2- 172.21.15.88:0/2010425001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7febbc15e660 0x7febbc14c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:01.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.628+0000 7febc2df1640 1 --2- 172.21.15.88:0/2010425001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7febbc0d3310 0x7febbc0c8080 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:01.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.628+0000 7febc2df1640 1 --2- 172.21.15.88:0/2010425001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7febbc0d1650 0x7febbc1f4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:01.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.628+0000 7febc2df1640 1 -- 172.21.15.88:0/2010425001 >> 172.21.15.88:0/2010425001 conn(0x7febbc0b6410 msgr2=0x7febbc0b57d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:01.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.628+0000 7febc2df1640 1 -- 172.21.15.88:0/2010425001 shutdown_connections 2024-09-16T01:40:01.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.628+0000 7febc2df1640 1 -- 172.21.15.88:0/2010425001 wait complete. 2024-09-16T01:40:01.640 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-16T01:40:01.640 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-16T01:40:01.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.741+0000 7f5ceee50640 1 Processor -- start 2024-09-16T01:40:01.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.741+0000 7f5ceee50640 1 -- start start 2024-09-16T01:40:01.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.742+0000 7f5ceee50640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5ce815c470 0x7f5ce80d0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:01.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.742+0000 7f5ceee50640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ce8073c00 0x7f5ce80b4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:01.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.742+0000 7f5ceee50640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ce80b5680 con 0x7f5ce80b46d0 2024-09-16T01:40:01.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.742+0000 7f5ceee50640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5ce80d1560 con 0x7f5ce815c470 2024-09-16T01:40:01.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.742+0000 7f5ceee50640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5ce80d1840 con 0x7f5ce8073c00 2024-09-16T01:40:01.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.742+0000 7f5ced3ed640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5ce80b46d0 0x7f5ce80d23c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33262/0 (socket says 172.21.15.88:33262) 2024-09-16T01:40:01.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.742+0000 7f5ced3ed640 1 -- 172.21.15.88:0/726612590 learned_addr learned my addr 172.21.15.88:0/726612590 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:01.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.743+0000 7f5cecbec640 1 --2- 172.21.15.88:0/726612590 >> v2:172.21.15.154:3300/0 conn(0x7f5ce815c470 0x7f5ce80d0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:01.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.743+0000 7f5ce7fff640 1 --2- 172.21.15.88:0/726612590 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ce8073c00 0x7f5ce80b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:01.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.743+0000 7f5ce77fe640 1 -- 172.21.15.88:0/726612590 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1494538358 0 0) 0x7f5ce80b5680 con 0x7f5ce80b46d0 2024-09-16T01:40:01.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.743+0000 7f5ce77fe640 1 -- 172.21.15.88:0/726612590 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cd0003520 con 0x7f5ce80b46d0 2024-09-16T01:40:01.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.743+0000 7f5ce77fe640 1 -- 172.21.15.88:0/726612590 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3315773433 0 0) 0x7f5cd0003520 con 0x7f5ce80b46d0 2024-09-16T01:40:01.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.743+0000 7f5cecbec640 1 -- 172.21.15.88:0/726612590 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ce8073c00 msgr2=0x7f5ce80b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:01.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.743+0000 7f5cecbec640 1 --2- 172.21.15.88:0/726612590 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ce8073c00 0x7f5ce80b4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:01.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.743+0000 7f5cecbec640 1 -- 172.21.15.88:0/726612590 >> v1:172.21.15.88:6789/0 conn(0x7f5ce80b46d0 legacy=0x7f5ce80d23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:01.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.743+0000 7f5cecbec640 1 -- 172.21.15.88:0/726612590 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce80d1b20 con 0x7f5ce815c470 2024-09-16T01:40:01.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.744+0000 7f5cecbec640 1 --2- 172.21.15.88:0/726612590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ce815c470 0x7f5ce80d0f80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5cd80097a0 tx=0x7f5cd802a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97d3cda1c575be1c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:01.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.744+0000 7f5ce77fe640 1 -- 172.21.15.88:0/726612590 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd8037070 con 0x7f5ce815c470 2024-09-16T01:40:01.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.744+0000 7f5ce77fe640 1 -- 172.21.15.88:0/726612590 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cd802aba0 con 0x7f5ce815c470 2024-09-16T01:40:01.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.744+0000 7f5ce77fe640 1 -- 172.21.15.88:0/726612590 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce0001670 con 0x7f5ce815c470 2024-09-16T01:40:01.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.744+0000 7f5ceee50640 1 -- 172.21.15.88:0/726612590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ce815c470 msgr2=0x7f5ce80d0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:01.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.744+0000 7f5ceee50640 1 --2- 172.21.15.88:0/726612590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ce815c470 0x7f5ce80d0f80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5cd80097a0 tx=0x7f5cd802a110 comp rx=0 tx=0).stop 2024-09-16T01:40:01.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.744+0000 7f5ceee50640 1 -- 172.21.15.88:0/726612590 shutdown_connections 2024-09-16T01:40:01.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.744+0000 7f5ceee50640 1 --2- 172.21.15.88:0/726612590 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ce8073c00 0x7f5ce80b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:01.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.744+0000 7f5ceee50640 1 --2- 172.21.15.88:0/726612590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ce815c470 0x7f5ce80d0f80 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:01.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.744+0000 7f5ceee50640 1 -- 172.21.15.88:0/726612590 >> 172.21.15.88:0/726612590 conn(0x7f5ce8062630 msgr2=0x7f5ce8062a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:01.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.745+0000 7f5ceee50640 1 -- 172.21.15.88:0/726612590 shutdown_connections 2024-09-16T01:40:01.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.745+0000 7f5ceee50640 1 -- 172.21.15.88:0/726612590 wait complete. 2024-09-16T01:40:01.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.746+0000 7f5ceee50640 1 Processor -- start 2024-09-16T01:40:01.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.746+0000 7f5ceee50640 1 -- start start 2024-09-16T01:40:01.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.746+0000 7f5ceee50640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ce8073c00 0x7f5ce80c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:01.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.746+0000 7f5ceee50640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ce80b46d0 0x7f5ce81f26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:01.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.747+0000 7f5ceee50640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ce815c470 0x7f5ce80d1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:01.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.747+0000 7f5ceee50640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce80741c0 con 0x7f5ce815c470 2024-09-16T01:40:01.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.747+0000 7f5ceee50640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce8074040 con 0x7f5ce8073c00 2024-09-16T01:40:01.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.747+0000 7f5ceee50640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5ce80b6b90 con 0x7f5ce80b46d0 2024-09-16T01:40:01.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.747+0000 7f5ced3ed640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ce815c470 0x7f5ce80d1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:01.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.747+0000 7f5ced3ed640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ce815c470 0x7f5ce80d1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59904/0 (socket says 172.21.15.88:59904) 2024-09-16T01:40:01.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.747+0000 7f5ced3ed640 1 -- 172.21.15.88:0/3006242970 learned_addr learned my addr 172.21.15.88:0/3006242970 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:01.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.747+0000 7f5ce7fff640 1 --2- 172.21.15.88:0/3006242970 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ce80b46d0 0x7f5ce81f26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:01.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.747+0000 7f5cecbec640 1 --2- 172.21.15.88:0/3006242970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ce8073c00 0x7f5ce80c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:01.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.747+0000 7f5ced3ed640 1 -- 172.21.15.88:0/3006242970 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ce80b46d0 msgr2=0x7f5ce81f26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:01.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.747+0000 7f5ced3ed640 1 --2- 172.21.15.88:0/3006242970 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ce80b46d0 0x7f5ce81f26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:01.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.747+0000 7f5ced3ed640 1 -- 172.21.15.88:0/3006242970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ce8073c00 msgr2=0x7f5ce80c7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:01.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.747+0000 7f5ced3ed640 1 --2- 172.21.15.88:0/3006242970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ce8073c00 0x7f5ce80c7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:01.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.747+0000 7f5ced3ed640 1 -- 172.21.15.88:0/3006242970 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce80c8890 con 0x7f5ce815c470 2024-09-16T01:40:01.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.747+0000 7f5cecbec640 1 --2- 172.21.15.88:0/3006242970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ce8073c00 0x7f5ce80c7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:01.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.747+0000 7f5ce7fff640 1 --2- 172.21.15.88:0/3006242970 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ce80b46d0 0x7f5ce81f26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:01.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.747+0000 7f5ced3ed640 1 --2- 172.21.15.88:0/3006242970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ce815c470 0x7f5ce80d1d60 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f5ce000c750 tx=0x7f5ce000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:01.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.747+0000 7f5ce57fa640 1 -- 172.21.15.88:0/3006242970 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce0017070 con 0x7f5ce815c470 2024-09-16T01:40:01.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.747+0000 7f5ce57fa640 1 -- 172.21.15.88:0/3006242970 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ce0005ad0 con 0x7f5ce815c470 2024-09-16T01:40:01.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.747+0000 7f5ceee50640 1 -- 172.21.15.88:0/3006242970 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ce81519a0 con 0x7f5ce815c470 2024-09-16T01:40:01.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.747+0000 7f5ce57fa640 1 -- 172.21.15.88:0/3006242970 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce0007070 con 0x7f5ce815c470 2024-09-16T01:40:01.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.747+0000 7f5ceee50640 1 -- 172.21.15.88:0/3006242970 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ce8151f60 con 0x7f5ce815c470 2024-09-16T01:40:01.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.748+0000 7f5ceee50640 1 -- 172.21.15.88:0/3006242970 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ce80b4190 con 0x7f5ce815c470 2024-09-16T01:40:01.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.749+0000 7f5ce57fa640 1 -- 172.21.15.88:0/3006242970 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5ce001a070 con 0x7f5ce815c470 2024-09-16T01:40:01.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.749+0000 7f5ce57fa640 1 --2- 172.21.15.88:0/3006242970 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5cd406c2b0 0x7f5cd406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:01.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.749+0000 7f5cecbec640 1 --2- 172.21.15.88:0/3006242970 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5cd406c2b0 0x7f5cd406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:01.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.749+0000 7f5ce57fa640 1 -- 172.21.15.88:0/3006242970 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 4857+0+0 (secure 0 0 0) 0x7f5ce008a450 con 0x7f5ce815c470 2024-09-16T01:40:01.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.750+0000 7f5cecbec640 1 --2- 172.21.15.88:0/3006242970 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5cd406c2b0 0x7f5cd406e770 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8034000 tx=0x7f5cd802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:01.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.754+0000 7f5ce57fa640 1 -- 172.21.15.88:0/3006242970 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5ce00588c0 con 0x7f5ce815c470 2024-09-16T01:40:01.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:01.923+0000 7f5ceee50640 1 -- 172.21.15.88:0/3006242970 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f5ce8070030 con 0x7f5ce815c470 2024-09-16T01:40:02.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.244+0000 7f5ce57fa640 1 -- 172.21.15.88:0/3006242970 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7f5ce005cbc0 con 0x7f5ce815c470 2024-09-16T01:40:02.245 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_metadata' created 2024-09-16T01:40:02.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.246+0000 7f5ceee50640 1 -- 172.21.15.88:0/3006242970 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5cd406c2b0 msgr2=0x7f5cd406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:02.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.246+0000 7f5ceee50640 1 --2- 172.21.15.88:0/3006242970 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5cd406c2b0 0x7f5cd406e770 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8034000 tx=0x7f5cd802e000 comp rx=0 tx=0).stop 2024-09-16T01:40:02.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.246+0000 7f5ceee50640 1 -- 172.21.15.88:0/3006242970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ce815c470 msgr2=0x7f5ce80d1d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:02.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.246+0000 7f5ceee50640 1 --2- 172.21.15.88:0/3006242970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ce815c470 0x7f5ce80d1d60 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f5ce000c750 tx=0x7f5ce000cc10 comp rx=0 tx=0).stop 2024-09-16T01:40:02.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.246+0000 7f5ceee50640 1 -- 172.21.15.88:0/3006242970 shutdown_connections 2024-09-16T01:40:02.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.246+0000 7f5ceee50640 1 --2- 172.21.15.88:0/3006242970 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5cd406c2b0 0x7f5cd406e770 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:02.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.246+0000 7f5ceee50640 1 --2- 172.21.15.88:0/3006242970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ce815c470 0x7f5ce80d1d60 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:02.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.246+0000 7f5ceee50640 1 --2- 172.21.15.88:0/3006242970 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ce80b46d0 0x7f5ce81f26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:02.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.246+0000 7f5ceee50640 1 --2- 172.21.15.88:0/3006242970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ce8073c00 0x7f5ce80c7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:02.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.246+0000 7f5ceee50640 1 -- 172.21.15.88:0/3006242970 >> 172.21.15.88:0/3006242970 conn(0x7f5ce8062630 msgr2=0x7f5ce80d0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:02.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.246+0000 7f5ceee50640 1 -- 172.21.15.88:0/3006242970 shutdown_connections 2024-09-16T01:40:02.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.246+0000 7f5ceee50640 1 -- 172.21.15.88:0/3006242970 wait complete. 2024-09-16T01:40:02.261 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-16T01:40:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.366+0000 7f9c1ca2e640 1 Processor -- start 2024-09-16T01:40:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.366+0000 7f9c1ca2e640 1 -- start start 2024-09-16T01:40:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.367+0000 7f9c1ca2e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c1815e6f0 0x7f9c1806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.367+0000 7f9c1ca2e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9c180b54d0 0x7f9c1815f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.367+0000 7f9c1ca2e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c1806e990 con 0x7f9c180b6d40 2024-09-16T01:40:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.367+0000 7f9c1ca2e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f9c1806ec70 con 0x7f9c180b54d0 2024-09-16T01:40:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.367+0000 7f9c1ca2e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9c1815ec70 con 0x7f9c1815e6f0 2024-09-16T01:40:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.367+0000 7f9c0dd9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9c180b54d0 0x7f9c1815f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.367+0000 7f9c0dd9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9c180b54d0 0x7f9c1815f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59910/0 (socket says 172.21.15.88:59910) 2024-09-16T01:40:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.367+0000 7f9c0dd9b640 1 -- 172.21.15.88:0/2108219459 learned_addr learned my addr 172.21.15.88:0/2108219459 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.367+0000 7f9c0dd9b640 1 -- 172.21.15.88:0/2108219459 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c1815e6f0 msgr2=0x7f9c1806fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:02.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.367+0000 7f9c1659c640 1 --2- 172.21.15.88:0/2108219459 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c1815e6f0 0x7f9c1806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:02.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.367+0000 7f9c0dd9b640 1 --2- 172.21.15.88:0/2108219459 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c1815e6f0 0x7f9c1806fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:02.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.367+0000 7f9c0dd9b640 1 -- 172.21.15.88:0/2108219459 >> v1:172.21.15.154:6789/0 conn(0x7f9c180b6d40 legacy=0x7f9c180b71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:02.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.367+0000 7f9c0dd9b640 1 -- 172.21.15.88:0/2108219459 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c1815ef50 con 0x7f9c180b54d0 2024-09-16T01:40:02.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.368+0000 7f9c0dd9b640 1 --2- 172.21.15.88:0/2108219459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c180b54d0 0x7f9c1815f750 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f9c00009870 tx=0x7f9c0002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe33ab0a6b79a76b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:02.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.368+0000 7f9c15d9b640 1 -- 172.21.15.88:0/2108219459 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c00037070 con 0x7f9c180b54d0 2024-09-16T01:40:02.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.368+0000 7f9c15d9b640 1 -- 172.21.15.88:0/2108219459 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c00032440 con 0x7f9c180b54d0 2024-09-16T01:40:02.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.368+0000 7f9c15d9b640 1 -- 172.21.15.88:0/2108219459 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c00032810 con 0x7f9c180b54d0 2024-09-16T01:40:02.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.369+0000 7f9c1ca2e640 1 -- 172.21.15.88:0/2108219459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c180b54d0 msgr2=0x7f9c1815f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:02.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.369+0000 7f9c1ca2e640 1 --2- 172.21.15.88:0/2108219459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c180b54d0 0x7f9c1815f750 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f9c00009870 tx=0x7f9c0002a210 comp rx=0 tx=0).stop 2024-09-16T01:40:02.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.369+0000 7f9c1ca2e640 1 -- 172.21.15.88:0/2108219459 shutdown_connections 2024-09-16T01:40:02.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.369+0000 7f9c1ca2e640 1 --2- 172.21.15.88:0/2108219459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c180b54d0 0x7f9c1815f750 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:02.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.369+0000 7f9c1ca2e640 1 --2- 172.21.15.88:0/2108219459 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c1815e6f0 0x7f9c1806fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:02.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.369+0000 7f9c1ca2e640 1 -- 172.21.15.88:0/2108219459 >> 172.21.15.88:0/2108219459 conn(0x7f9c180d2430 msgr2=0x7f9c180d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:02.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.369+0000 7f9c1ca2e640 1 -- 172.21.15.88:0/2108219459 shutdown_connections 2024-09-16T01:40:02.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.369+0000 7f9c1ca2e640 1 -- 172.21.15.88:0/2108219459 wait complete. 2024-09-16T01:40:02.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.370+0000 7f9c1ca2e640 1 Processor -- start 2024-09-16T01:40:02.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.370+0000 7f9c1ca2e640 1 -- start start 2024-09-16T01:40:02.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.371+0000 7f9c1ca2e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c180b54d0 0x7f9c18146f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:02.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.371+0000 7f9c1ca2e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c180b6d40 0x7f9c18153950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:02.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.371+0000 7f9c1ca2e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c1815e6f0 0x7f9c18153e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:02.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.371+0000 7f9c1ca2e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1806e730 con 0x7f9c180b54d0 2024-09-16T01:40:02.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.371+0000 7f9c1ca2e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1806e5b0 con 0x7f9c180b6d40 2024-09-16T01:40:02.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.371+0000 7f9c1ca2e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9c18070830 con 0x7f9c1815e6f0 2024-09-16T01:40:02.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.371+0000 7f9c1659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c180b54d0 0x7f9c18146f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:02.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.371+0000 7f9c1659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c180b54d0 0x7f9c18146f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59916/0 (socket says 172.21.15.88:59916) 2024-09-16T01:40:02.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.371+0000 7f9c1659c640 1 -- 172.21.15.88:0/4288651844 learned_addr learned my addr 172.21.15.88:0/4288651844 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:02.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.371+0000 7f9c0dd9b640 1 --2- 172.21.15.88:0/4288651844 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c180b6d40 0x7f9c18153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:02.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.371+0000 7f9c16d9d640 1 --2- 172.21.15.88:0/4288651844 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c1815e6f0 0x7f9c18153e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:02.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.371+0000 7f9c1659c640 1 -- 172.21.15.88:0/4288651844 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c1815e6f0 msgr2=0x7f9c18153e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:02.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.371+0000 7f9c1659c640 1 --2- 172.21.15.88:0/4288651844 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c1815e6f0 0x7f9c18153e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:02.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.371+0000 7f9c1659c640 1 -- 172.21.15.88:0/4288651844 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c180b6d40 msgr2=0x7f9c18153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:02.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.371+0000 7f9c1659c640 1 --2- 172.21.15.88:0/4288651844 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c180b6d40 0x7f9c18153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:02.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.371+0000 7f9c1659c640 1 -- 172.21.15.88:0/4288651844 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c18152230 con 0x7f9c180b54d0 2024-09-16T01:40:02.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.371+0000 7f9c0dd9b640 1 --2- 172.21.15.88:0/4288651844 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c180b6d40 0x7f9c18153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:02.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.371+0000 7f9c1659c640 1 --2- 172.21.15.88:0/4288651844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c180b54d0 0x7f9c18146f10 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f9c0800c7f0 tx=0x7f9c0800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:02.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.372+0000 7f9c0f7fe640 1 -- 172.21.15.88:0/4288651844 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c08015070 con 0x7f9c180b54d0 2024-09-16T01:40:02.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.372+0000 7f9c0f7fe640 1 -- 172.21.15.88:0/4288651844 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c08004220 con 0x7f9c180b54d0 2024-09-16T01:40:02.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.372+0000 7f9c1ca2e640 1 -- 172.21.15.88:0/4288651844 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c181524d0 con 0x7f9c180b54d0 2024-09-16T01:40:02.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.372+0000 7f9c0f7fe640 1 -- 172.21.15.88:0/4288651844 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c08007620 con 0x7f9c180b54d0 2024-09-16T01:40:02.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.372+0000 7f9c1ca2e640 1 -- 172.21.15.88:0/4288651844 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c18152a90 con 0x7f9c180b54d0 2024-09-16T01:40:02.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.373+0000 7f9c1ca2e640 1 -- 172.21.15.88:0/4288651844 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c1806c6c0 con 0x7f9c180b54d0 2024-09-16T01:40:02.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.373+0000 7f9c0f7fe640 1 -- 172.21.15.88:0/4288651844 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9c080079d0 con 0x7f9c180b54d0 2024-09-16T01:40:02.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.374+0000 7f9c0f7fe640 1 --2- 172.21.15.88:0/4288651844 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9bf806c2e0 0x7f9bf806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:02.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.374+0000 7f9c0dd9b640 1 --2- 172.21.15.88:0/4288651844 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9bf806c2e0 0x7f9bf806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:02.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.374+0000 7f9c0f7fe640 1 -- 172.21.15.88:0/4288651844 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5227+0+0 (secure 0 0 0) 0x7f9c0808b220 con 0x7f9c180b54d0 2024-09-16T01:40:02.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.374+0000 7f9c0dd9b640 1 --2- 172.21.15.88:0/4288651844 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9bf806c2e0 0x7f9bf806e7a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f9c00034000 tx=0x7f9c0002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:02.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.378+0000 7f9c0f7fe640 1 -- 172.21.15.88:0/4288651844 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9c08059b70 con 0x7f9c180b54d0 2024-09-16T01:40:02.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:02.545+0000 7f9c1ca2e640 1 -- 172.21.15.88:0/4288651844 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f9c1815dcf0 con 0x7f9c180b54d0 2024-09-16T01:40:03.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.249+0000 7f9c0f7fe640 1 -- 172.21.15.88:0/4288651844 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v83) ==== 170+0+0 (secure 0 0 0) 0x7f9c08059470 con 0x7f9c180b54d0 2024-09-16T01:40:03.251 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_data' created 2024-09-16T01:40:03.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.252+0000 7f9c1ca2e640 1 -- 172.21.15.88:0/4288651844 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9bf806c2e0 msgr2=0x7f9bf806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:03.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.252+0000 7f9c1ca2e640 1 --2- 172.21.15.88:0/4288651844 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9bf806c2e0 0x7f9bf806e7a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f9c00034000 tx=0x7f9c0002e000 comp rx=0 tx=0).stop 2024-09-16T01:40:03.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.252+0000 7f9c1ca2e640 1 -- 172.21.15.88:0/4288651844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c180b54d0 msgr2=0x7f9c18146f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:03.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.252+0000 7f9c1ca2e640 1 --2- 172.21.15.88:0/4288651844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c180b54d0 0x7f9c18146f10 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f9c0800c7f0 tx=0x7f9c0800ccb0 comp rx=0 tx=0).stop 2024-09-16T01:40:03.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.252+0000 7f9c1ca2e640 1 -- 172.21.15.88:0/4288651844 shutdown_connections 2024-09-16T01:40:03.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.252+0000 7f9c1ca2e640 1 --2- 172.21.15.88:0/4288651844 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9bf806c2e0 0x7f9bf806e7a0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:03.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.252+0000 7f9c1ca2e640 1 --2- 172.21.15.88:0/4288651844 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9c1815e6f0 0x7f9c18153e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:03.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.252+0000 7f9c1ca2e640 1 --2- 172.21.15.88:0/4288651844 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c180b6d40 0x7f9c18153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:03.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.252+0000 7f9c1ca2e640 1 --2- 172.21.15.88:0/4288651844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c180b54d0 0x7f9c18146f10 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:03.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.252+0000 7f9c1ca2e640 1 -- 172.21.15.88:0/4288651844 >> 172.21.15.88:0/4288651844 conn(0x7f9c180d2430 msgr2=0x7f9c1815b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:03.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.252+0000 7f9c1ca2e640 1 -- 172.21.15.88:0/4288651844 shutdown_connections 2024-09-16T01:40:03.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.252+0000 7f9c1ca2e640 1 -- 172.21.15.88:0/4288651844 wait complete. 2024-09-16T01:40:03.278 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-16T01:40:03.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.380+0000 7f7097c5e640 1 Processor -- start 2024-09-16T01:40:03.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.380+0000 7f7097c5e640 1 -- start start 2024-09-16T01:40:03.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.381+0000 7f7097c5e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7090158a00 0x7f7090071f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:03.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.381+0000 7f7097c5e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7090072470 0x7f70900b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:03.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.381+0000 7f7097c5e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70900728f0 con 0x7f7090158de0 2024-09-16T01:40:03.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.381+0000 7f7097c5e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7090072bd0 con 0x7f7090158a00 2024-09-16T01:40:03.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.381+0000 7f7097c5e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7090072eb0 con 0x7f7090072470 2024-09-16T01:40:03.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.381+0000 7f70959fa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7090158a00 0x7f7090071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:03.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.381+0000 7f70959fa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7090158a00 0x7f7090071f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59930/0 (socket says 172.21.15.88:59930) 2024-09-16T01:40:03.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.381+0000 7f70959fa640 1 -- 172.21.15.88:0/852718558 learned_addr learned my addr 172.21.15.88:0/852718558 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:03.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.381+0000 7f70951f9640 1 --2- 172.21.15.88:0/852718558 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7090072470 0x7f70900b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:03.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.381+0000 7f70959fa640 1 -- 172.21.15.88:0/852718558 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7090072470 msgr2=0x7f70900b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:03.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.381+0000 7f70959fa640 1 --2- 172.21.15.88:0/852718558 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7090072470 0x7f70900b4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:03.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.381+0000 7f70959fa640 1 -- 172.21.15.88:0/852718558 >> v1:172.21.15.154:6789/0 conn(0x7f7090158de0 legacy=0x7f7090162110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:03.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.381+0000 7f70959fa640 1 -- 172.21.15.88:0/852718558 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7090073190 con 0x7f7090158a00 2024-09-16T01:40:03.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.382+0000 7f70959fa640 1 --2- 172.21.15.88:0/852718558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7090158a00 0x7f7090071f30 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f7080009e50 tx=0x7f708002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87edc27e75434b0a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:03.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.382+0000 7f70949f8640 1 -- 172.21.15.88:0/852718558 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7080036070 con 0x7f7090158a00 2024-09-16T01:40:03.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.382+0000 7f70949f8640 1 -- 172.21.15.88:0/852718558 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7080031600 con 0x7f7090158a00 2024-09-16T01:40:03.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.382+0000 7f70949f8640 1 -- 172.21.15.88:0/852718558 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70800319d0 con 0x7f7090158a00 2024-09-16T01:40:03.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.383+0000 7f7097c5e640 1 -- 172.21.15.88:0/852718558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7090158a00 msgr2=0x7f7090071f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:03.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.383+0000 7f7097c5e640 1 --2- 172.21.15.88:0/852718558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7090158a00 0x7f7090071f30 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f7080009e50 tx=0x7f708002a710 comp rx=0 tx=0).stop 2024-09-16T01:40:03.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.383+0000 7f7097c5e640 1 -- 172.21.15.88:0/852718558 shutdown_connections 2024-09-16T01:40:03.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.383+0000 7f7097c5e640 1 --2- 172.21.15.88:0/852718558 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7090072470 0x7f70900b4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:03.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.383+0000 7f7097c5e640 1 --2- 172.21.15.88:0/852718558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7090158a00 0x7f7090071f30 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:03.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.383+0000 7f7097c5e640 1 -- 172.21.15.88:0/852718558 >> 172.21.15.88:0/852718558 conn(0x7f709006cb50 msgr2=0x7f709006cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:03.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.383+0000 7f7097c5e640 1 -- 172.21.15.88:0/852718558 shutdown_connections 2024-09-16T01:40:03.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.383+0000 7f7097c5e640 1 -- 172.21.15.88:0/852718558 wait complete. 2024-09-16T01:40:03.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.384+0000 7f7097c5e640 1 Processor -- start 2024-09-16T01:40:03.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.384+0000 7f7097c5e640 1 -- start start 2024-09-16T01:40:03.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.385+0000 7f7097c5e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7090072470 0x7f70901600b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:03.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.385+0000 7f7097c5e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7090158a00 0x7f709014dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:03.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.385+0000 7f7097c5e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7090158de0 0x7f70901ea120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:03.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.385+0000 7f7097c5e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f70900b5ca0 con 0x7f7090158de0 2024-09-16T01:40:03.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.385+0000 7f7097c5e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f70900b5b20 con 0x7f7090158a00 2024-09-16T01:40:03.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.385+0000 7f7097c5e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f709006faf0 con 0x7f7090072470 2024-09-16T01:40:03.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.385+0000 7f70961fb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7090158de0 0x7f70901ea120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:03.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.385+0000 7f70961fb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7090158de0 0x7f70901ea120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59944/0 (socket says 172.21.15.88:59944) 2024-09-16T01:40:03.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.385+0000 7f70961fb640 1 -- 172.21.15.88:0/436413586 learned_addr learned my addr 172.21.15.88:0/436413586 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:03.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.385+0000 7f70959fa640 1 --2- 172.21.15.88:0/436413586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7090072470 0x7f70901600b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:03.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.385+0000 7f70951f9640 1 --2- 172.21.15.88:0/436413586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7090158a00 0x7f709014dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:03.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.385+0000 7f70961fb640 1 -- 172.21.15.88:0/436413586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7090072470 msgr2=0x7f70901600b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:03.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.385+0000 7f70961fb640 1 --2- 172.21.15.88:0/436413586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7090072470 0x7f70901600b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:03.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.385+0000 7f70961fb640 1 -- 172.21.15.88:0/436413586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7090158a00 msgr2=0x7f709014dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:03.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.385+0000 7f70961fb640 1 --2- 172.21.15.88:0/436413586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7090158a00 0x7f709014dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:03.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.385+0000 7f70961fb640 1 -- 172.21.15.88:0/436413586 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7090160970 con 0x7f7090158de0 2024-09-16T01:40:03.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.386+0000 7f70961fb640 1 --2- 172.21.15.88:0/436413586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7090158de0 0x7f70901ea120 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f708800c3f0 tx=0x7f708800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:03.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.386+0000 7f708e7fc640 1 -- 172.21.15.88:0/436413586 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7088017070 con 0x7f7090158de0 2024-09-16T01:40:03.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.386+0000 7f708e7fc640 1 -- 172.21.15.88:0/436413586 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70880052d0 con 0x7f7090158de0 2024-09-16T01:40:03.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.386+0000 7f708e7fc640 1 -- 172.21.15.88:0/436413586 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7088003830 con 0x7f7090158de0 2024-09-16T01:40:03.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.386+0000 7f7097c5e640 1 -- 172.21.15.88:0/436413586 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7090160c50 con 0x7f7090158de0 2024-09-16T01:40:03.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.386+0000 7f7097c5e640 1 -- 172.21.15.88:0/436413586 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70901614f0 con 0x7f7090158de0 2024-09-16T01:40:03.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.387+0000 7f7097c5e640 1 -- 172.21.15.88:0/436413586 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7090071f30 con 0x7f7090158de0 2024-09-16T01:40:03.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.387+0000 7f708e7fc640 1 -- 172.21.15.88:0/436413586 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f708801a070 con 0x7f7090158de0 2024-09-16T01:40:03.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.388+0000 7f708e7fc640 1 --2- 172.21.15.88:0/436413586 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f707c06c2b0 0x7f707c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:03.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.388+0000 7f70959fa640 1 --2- 172.21.15.88:0/436413586 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f707c06c2b0 0x7f707c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:03.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.388+0000 7f708e7fc640 1 -- 172.21.15.88:0/436413586 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5609+0+0 (secure 0 0 0) 0x7f708808a5c0 con 0x7f7090158de0 2024-09-16T01:40:03.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.388+0000 7f70959fa640 1 --2- 172.21.15.88:0/436413586 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f707c06c2b0 0x7f707c06e770 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f7080033000 tx=0x7f7080009000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:03.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.392+0000 7f708e7fc640 1 -- 172.21.15.88:0/436413586 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7088058800 con 0x7f7090158de0 2024-09-16T01:40:03.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:03.596+0000 7f7097c5e640 1 -- 172.21.15.88:0/436413586 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f7090160e20 con 0x7f7090158de0 2024-09-16T01:40:04.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.250+0000 7f708e7fc640 1 -- 172.21.15.88:0/436413586 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v68) ==== 174+0+0 (secure 0 0 0) 0x7f708805cb00 con 0x7f7090158de0 2024-09-16T01:40:04.252 INFO:teuthology.orchestra.run.smithi088.stderr:new fs with metadata pool 12 and data pool 13 2024-09-16T01:40:04.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.253+0000 7f7097c5e640 1 -- 172.21.15.88:0/436413586 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f707c06c2b0 msgr2=0x7f707c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:04.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.253+0000 7f7097c5e640 1 --2- 172.21.15.88:0/436413586 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f707c06c2b0 0x7f707c06e770 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f7080033000 tx=0x7f7080009000 comp rx=0 tx=0).stop 2024-09-16T01:40:04.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.253+0000 7f7097c5e640 1 -- 172.21.15.88:0/436413586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7090158de0 msgr2=0x7f70901ea120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:04.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.253+0000 7f7097c5e640 1 --2- 172.21.15.88:0/436413586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7090158de0 0x7f70901ea120 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f708800c3f0 tx=0x7f708800c7f0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.253+0000 7f7097c5e640 1 -- 172.21.15.88:0/436413586 shutdown_connections 2024-09-16T01:40:04.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.253+0000 7f7097c5e640 1 --2- 172.21.15.88:0/436413586 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f707c06c2b0 0x7f707c06e770 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.253+0000 7f7097c5e640 1 --2- 172.21.15.88:0/436413586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7090158de0 0x7f70901ea120 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.253+0000 7f7097c5e640 1 --2- 172.21.15.88:0/436413586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7090158a00 0x7f709014dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.253+0000 7f7097c5e640 1 --2- 172.21.15.88:0/436413586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7090072470 0x7f70901600b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.253+0000 7f7097c5e640 1 -- 172.21.15.88:0/436413586 >> 172.21.15.88:0/436413586 conn(0x7f709006cb50 msgr2=0x7f70900b1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:04.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.253+0000 7f7097c5e640 1 -- 172.21.15.88:0/436413586 shutdown_connections 2024-09-16T01:40:04.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.253+0000 7f7097c5e640 1 -- 172.21.15.88:0/436413586 wait complete. 2024-09-16T01:40:04.278 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:40:04.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.379+0000 7f6a5d0d7640 1 Processor -- start 2024-09-16T01:40:04.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.379+0000 7f6a5d0d7640 1 -- start start 2024-09-16T01:40:04.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.380+0000 7f6a5d0d7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6a5815e660 0x7f6a580cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:04.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.380+0000 7f6a5d0d7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a580d1820 0x7f6a580b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:04.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.380+0000 7f6a5d0d7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a5815ea40 con 0x7f6a580d3310 2024-09-16T01:40:04.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.380+0000 7f6a5d0d7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6a580d3870 con 0x7f6a5815e660 2024-09-16T01:40:04.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.380+0000 7f6a5d0d7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6a580d3b50 con 0x7f6a580d1820 2024-09-16T01:40:04.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.380+0000 7f6a56d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6a5815e660 0x7f6a580cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:04.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.380+0000 7f6a56d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6a5815e660 0x7f6a580cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59950/0 (socket says 172.21.15.88:59950) 2024-09-16T01:40:04.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.380+0000 7f6a56d9d640 1 -- 172.21.15.88:0/1579673069 learned_addr learned my addr 172.21.15.88:0/1579673069 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:04.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.381+0000 7f6a5759e640 1 --2- 172.21.15.88:0/1579673069 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a580d1820 0x7f6a580b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:04.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.381+0000 7f6a56d9d640 1 -- 172.21.15.88:0/1579673069 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a580d1820 msgr2=0x7f6a580b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.381+0000 7f6a56d9d640 1 --2- 172.21.15.88:0/1579673069 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a580d1820 0x7f6a580b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.381+0000 7f6a56d9d640 1 -- 172.21.15.88:0/1579673069 >> v1:172.21.15.154:6789/0 conn(0x7f6a580d3310 legacy=0x7f6a580d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.381+0000 7f6a56d9d640 1 -- 172.21.15.88:0/1579673069 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a580d3e30 con 0x7f6a5815e660 2024-09-16T01:40:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.381+0000 7f6a5759e640 1 --2- 172.21.15.88:0/1579673069 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a580d1820 0x7f6a580b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.381+0000 7f6a56d9d640 1 --2- 172.21.15.88:0/1579673069 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a5815e660 0x7f6a580cf730 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f6a48009870 tx=0x7f6a4802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e96e1fbf5417abc9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.382+0000 7f6a5659c640 1 -- 172.21.15.88:0/1579673069 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a48037070 con 0x7f6a5815e660 2024-09-16T01:40:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.382+0000 7f6a5659c640 1 -- 172.21.15.88:0/1579673069 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a48032440 con 0x7f6a5815e660 2024-09-16T01:40:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.382+0000 7f6a5659c640 1 -- 172.21.15.88:0/1579673069 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a48032810 con 0x7f6a5815e660 2024-09-16T01:40:04.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.382+0000 7f6a5d0d7640 1 -- 172.21.15.88:0/1579673069 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a5815e660 msgr2=0x7f6a580cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:04.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.382+0000 7f6a5d0d7640 1 --2- 172.21.15.88:0/1579673069 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a5815e660 0x7f6a580cf730 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f6a48009870 tx=0x7f6a4802a270 comp rx=0 tx=0).stop 2024-09-16T01:40:04.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.382+0000 7f6a5d0d7640 1 -- 172.21.15.88:0/1579673069 shutdown_connections 2024-09-16T01:40:04.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.382+0000 7f6a5d0d7640 1 --2- 172.21.15.88:0/1579673069 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a580d1820 0x7f6a580b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.382+0000 7f6a5d0d7640 1 --2- 172.21.15.88:0/1579673069 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a5815e660 0x7f6a580cf730 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.382+0000 7f6a5d0d7640 1 -- 172.21.15.88:0/1579673069 >> 172.21.15.88:0/1579673069 conn(0x7f6a580b6410 msgr2=0x7f6a580b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:04.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.382+0000 7f6a5d0d7640 1 -- 172.21.15.88:0/1579673069 shutdown_connections 2024-09-16T01:40:04.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.382+0000 7f6a5d0d7640 1 -- 172.21.15.88:0/1579673069 wait complete. 2024-09-16T01:40:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.383+0000 7f6a5d0d7640 1 Processor -- start 2024-09-16T01:40:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.384+0000 7f6a5d0d7640 1 -- start start 2024-09-16T01:40:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.384+0000 7f6a5d0d7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a580d1820 0x7f6a5814ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.384+0000 7f6a5d0d7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a580d3310 0x7f6a5814f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.384+0000 7f6a5d0d7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a5815e660 0x7f6a580b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.384+0000 7f6a5d0d7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6a580cfc70 con 0x7f6a5815e660 2024-09-16T01:40:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.384+0000 7f6a5d0d7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6a580d43f0 con 0x7f6a580d1820 2024-09-16T01:40:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.384+0000 7f6a5d0d7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6a580b80b0 con 0x7f6a580d3310 2024-09-16T01:40:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.384+0000 7f6a5759e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a5815e660 0x7f6a580b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.384+0000 7f6a5759e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a5815e660 0x7f6a580b4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59962/0 (socket says 172.21.15.88:59962) 2024-09-16T01:40:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.384+0000 7f6a5759e640 1 -- 172.21.15.88:0/3240459708 learned_addr learned my addr 172.21.15.88:0/3240459708 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.384+0000 7f6a56d9d640 1 --2- 172.21.15.88:0/3240459708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a580d1820 0x7f6a5814ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.385+0000 7f6a4e59c640 1 --2- 172.21.15.88:0/3240459708 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a580d3310 0x7f6a5814f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.385+0000 7f6a5759e640 1 -- 172.21.15.88:0/3240459708 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a580d3310 msgr2=0x7f6a5814f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.385+0000 7f6a5759e640 1 --2- 172.21.15.88:0/3240459708 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a580d3310 0x7f6a5814f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.385+0000 7f6a5759e640 1 -- 172.21.15.88:0/3240459708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a580d1820 msgr2=0x7f6a5814ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.385+0000 7f6a5759e640 1 --2- 172.21.15.88:0/3240459708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a580d1820 0x7f6a5814ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.385+0000 7f6a5759e640 1 -- 172.21.15.88:0/3240459708 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a580c8230 con 0x7f6a5815e660 2024-09-16T01:40:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.385+0000 7f6a5759e640 1 --2- 172.21.15.88:0/3240459708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a5815e660 0x7f6a580b4fe0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f6a50007ef0 tx=0x7f6a5000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.385+0000 7f6a4ffff640 1 -- 172.21.15.88:0/3240459708 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5001a070 con 0x7f6a5815e660 2024-09-16T01:40:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.385+0000 7f6a4ffff640 1 -- 172.21.15.88:0/3240459708 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a50003b90 con 0x7f6a5815e660 2024-09-16T01:40:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.385+0000 7f6a5d0d7640 1 -- 172.21.15.88:0/3240459708 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a580c8510 con 0x7f6a5815e660 2024-09-16T01:40:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.385+0000 7f6a4ffff640 1 -- 172.21.15.88:0/3240459708 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a50004030 con 0x7f6a5815e660 2024-09-16T01:40:04.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.385+0000 7f6a5d0d7640 1 -- 172.21.15.88:0/3240459708 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a580c8ad0 con 0x7f6a5815e660 2024-09-16T01:40:04.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.386+0000 7f6a5d0d7640 1 -- 172.21.15.88:0/3240459708 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a580b7b70 con 0x7f6a5815e660 2024-09-16T01:40:04.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.387+0000 7f6a4ffff640 1 -- 172.21.15.88:0/3240459708 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6a5001d070 con 0x7f6a5815e660 2024-09-16T01:40:04.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.387+0000 7f6a4ffff640 1 --2- 172.21.15.88:0/3240459708 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6a3806c2b0 0x7f6a3806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:04.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.387+0000 7f6a56d9d640 1 --2- 172.21.15.88:0/3240459708 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6a3806c2b0 0x7f6a3806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:04.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.387+0000 7f6a4ffff640 1 -- 172.21.15.88:0/3240459708 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5709+0+0 (secure 0 0 0) 0x7f6a5008b6d0 con 0x7f6a5815e660 2024-09-16T01:40:04.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.388+0000 7f6a56d9d640 1 --2- 172.21.15.88:0/3240459708 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6a3806c2b0 0x7f6a3806e770 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6a48034000 tx=0x7f6a4802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:04.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.392+0000 7f6a4ffff640 1 -- 172.21.15.88:0/3240459708 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6a50010070 con 0x7f6a5815e660 2024-09-16T01:40:04.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.558+0000 7f6a5d0d7640 1 -- 172.21.15.88:0/3240459708 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6a5815bf50 con 0x7f6a5815e660 2024-09-16T01:40:04.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.560+0000 7f6a4ffff640 1 -- 172.21.15.88:0/3240459708 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+16709 (secure 0 0 0) 0x7f6a50059770 con 0x7f6a5815e660 2024-09-16T01:40:04.561 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:04.561 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":84,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:40:03.597487+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:40:01.941882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-16T01:40:02.564641+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:40:04.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.562+0000 7f6a5d0d7640 1 -- 172.21.15.88:0/3240459708 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6a3806c2b0 msgr2=0x7f6a3806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:04.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.562+0000 7f6a5d0d7640 1 --2- 172.21.15.88:0/3240459708 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6a3806c2b0 0x7f6a3806e770 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6a48034000 tx=0x7f6a4802e000 comp rx=0 tx=0).stop 2024-09-16T01:40:04.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.562+0000 7f6a5d0d7640 1 -- 172.21.15.88:0/3240459708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a5815e660 msgr2=0x7f6a580b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:04.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.562+0000 7f6a5d0d7640 1 --2- 172.21.15.88:0/3240459708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a5815e660 0x7f6a580b4fe0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f6a50007ef0 tx=0x7f6a5000d4a0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.562+0000 7f6a5d0d7640 1 -- 172.21.15.88:0/3240459708 shutdown_connections 2024-09-16T01:40:04.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.562+0000 7f6a5d0d7640 1 --2- 172.21.15.88:0/3240459708 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6a3806c2b0 0x7f6a3806e770 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.562+0000 7f6a5d0d7640 1 --2- 172.21.15.88:0/3240459708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a5815e660 0x7f6a580b4fe0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.562+0000 7f6a5d0d7640 1 --2- 172.21.15.88:0/3240459708 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a580d3310 0x7f6a5814f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.562+0000 7f6a5d0d7640 1 --2- 172.21.15.88:0/3240459708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a580d1820 0x7f6a5814ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.562+0000 7f6a5d0d7640 1 -- 172.21.15.88:0/3240459708 >> 172.21.15.88:0/3240459708 conn(0x7f6a580b6410 msgr2=0x7f6a580b99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:04.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.562+0000 7f6a5d0d7640 1 -- 172.21.15.88:0/3240459708 shutdown_connections 2024-09-16T01:40:04.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.562+0000 7f6a5d0d7640 1 -- 172.21.15.88:0/3240459708 wait complete. 2024-09-16T01:40:04.574 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:40:04.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.671+0000 7f3b25cd0640 1 Processor -- start 2024-09-16T01:40:04.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.671+0000 7f3b25cd0640 1 -- start start 2024-09-16T01:40:04.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.672+0000 7f3b25cd0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3b2006e780 0x7f3b200ae890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:04.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.672+0000 7f3b25cd0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b20073130 0x7f3b20073580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:04.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.672+0000 7f3b25cd0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b200b5220 con 0x7f3b2015e560 2024-09-16T01:40:04.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.672+0000 7f3b25cd0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3b2006ebc0 con 0x7f3b2006e780 2024-09-16T01:40:04.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.672+0000 7f3b25cd0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3b20073ac0 con 0x7f3b20073130 2024-09-16T01:40:04.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.672+0000 7f3b1f7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3b2015e560 0x7f3b20072220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33278/0 (socket says 172.21.15.88:33278) 2024-09-16T01:40:04.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.672+0000 7f3b1f7fe640 1 -- 172.21.15.88:0/3352728000 learned_addr learned my addr 172.21.15.88:0/3352728000 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:04.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.672+0000 7f3b1effd640 1 --2- 172.21.15.88:0/3352728000 >> v2:172.21.15.154:3300/0 conn(0x7f3b2006e780 0x7f3b200ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:04.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.672+0000 7f3b1ffff640 1 --2- 172.21.15.88:0/3352728000 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b20073130 0x7f3b20073580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:04.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.673+0000 7f3b1e7fc640 1 -- 172.21.15.88:0/3352728000 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 727538155 0 0) 0x7f3b200b5220 con 0x7f3b2015e560 2024-09-16T01:40:04.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.673+0000 7f3b1e7fc640 1 -- 172.21.15.88:0/3352728000 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b08003520 con 0x7f3b2015e560 2024-09-16T01:40:04.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.673+0000 7f3b1effd640 1 -- 172.21.15.88:0/3352728000 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b20073130 msgr2=0x7f3b20073580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:04.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.673+0000 7f3b1effd640 1 --2- 172.21.15.88:0/3352728000 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b20073130 0x7f3b20073580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.673+0000 7f3b1effd640 1 -- 172.21.15.88:0/3352728000 >> v1:172.21.15.88:6789/0 conn(0x7f3b2015e560 legacy=0x7f3b20072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:04.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.673+0000 7f3b1effd640 1 -- 172.21.15.88:0/3352728000 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b200b54c0 con 0x7f3b2006e780 2024-09-16T01:40:04.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.673+0000 7f3b1effd640 1 --2- 172.21.15.88:0/3352728000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b2006e780 0x7f3b200ae890 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3b140097a0 tx=0x7f3b1402a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=675264100706ceda server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:04.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.674+0000 7f3b1e7fc640 1 -- 172.21.15.88:0/3352728000 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b14037070 con 0x7f3b2006e780 2024-09-16T01:40:04.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.674+0000 7f3b1e7fc640 1 -- 172.21.15.88:0/3352728000 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b1402ae50 con 0x7f3b2006e780 2024-09-16T01:40:04.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.674+0000 7f3b1e7fc640 1 -- 172.21.15.88:0/3352728000 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b14032640 con 0x7f3b2006e780 2024-09-16T01:40:04.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.674+0000 7f3b25cd0640 1 -- 172.21.15.88:0/3352728000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b2006e780 msgr2=0x7f3b200ae890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:04.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.674+0000 7f3b25cd0640 1 --2- 172.21.15.88:0/3352728000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b2006e780 0x7f3b200ae890 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3b140097a0 tx=0x7f3b1402a100 comp rx=0 tx=0).stop 2024-09-16T01:40:04.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.674+0000 7f3b25cd0640 1 -- 172.21.15.88:0/3352728000 shutdown_connections 2024-09-16T01:40:04.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.674+0000 7f3b25cd0640 1 --2- 172.21.15.88:0/3352728000 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b20073130 0x7f3b20073580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.674+0000 7f3b25cd0640 1 --2- 172.21.15.88:0/3352728000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b2006e780 0x7f3b200ae890 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.674+0000 7f3b25cd0640 1 -- 172.21.15.88:0/3352728000 >> 172.21.15.88:0/3352728000 conn(0x7f3b2006bc10 msgr2=0x7f3b2006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:04.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.674+0000 7f3b25cd0640 1 -- 172.21.15.88:0/3352728000 shutdown_connections 2024-09-16T01:40:04.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.674+0000 7f3b25cd0640 1 -- 172.21.15.88:0/3352728000 wait complete. 2024-09-16T01:40:04.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.675+0000 7f3b25cd0640 1 Processor -- start 2024-09-16T01:40:04.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.676+0000 7f3b25cd0640 1 -- start start 2024-09-16T01:40:04.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.676+0000 7f3b25cd0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b2006e780 0x7f3b20150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:04.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.676+0000 7f3b25cd0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b20073130 0x7f3b2014f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:04.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.676+0000 7f3b25cd0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b2015e560 0x7f3b2014d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:04.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.676+0000 7f3b25cd0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3b200b5d60 con 0x7f3b20073130 2024-09-16T01:40:04.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.676+0000 7f3b25cd0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3b200b5a80 con 0x7f3b2006e780 2024-09-16T01:40:04.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.676+0000 7f3b25cd0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3b2015e940 con 0x7f3b2015e560 2024-09-16T01:40:04.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.676+0000 7f3b1effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b20073130 0x7f3b2014f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:04.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.676+0000 7f3b1effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b20073130 0x7f3b2014f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59964/0 (socket says 172.21.15.88:59964) 2024-09-16T01:40:04.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.676+0000 7f3b1effd640 1 -- 172.21.15.88:0/3398659606 learned_addr learned my addr 172.21.15.88:0/3398659606 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:04.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.677+0000 7f3b1f7fe640 1 --2- 172.21.15.88:0/3398659606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b2006e780 0x7f3b20150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:04.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.677+0000 7f3b1effd640 1 -- 172.21.15.88:0/3398659606 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b2015e560 msgr2=0x7f3b2014d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:04.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.677+0000 7f3b1effd640 1 --2- 172.21.15.88:0/3398659606 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b2015e560 0x7f3b2014d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.677+0000 7f3b1effd640 1 -- 172.21.15.88:0/3398659606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b2006e780 msgr2=0x7f3b20150e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:04.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.677+0000 7f3b1effd640 1 --2- 172.21.15.88:0/3398659606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b2006e780 0x7f3b20150e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.677+0000 7f3b1effd640 1 -- 172.21.15.88:0/3398659606 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b2014fdc0 con 0x7f3b20073130 2024-09-16T01:40:04.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.677+0000 7f3b1f7fe640 1 --2- 172.21.15.88:0/3398659606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b2006e780 0x7f3b20150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:04.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.677+0000 7f3b1effd640 1 --2- 172.21.15.88:0/3398659606 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b20073130 0x7f3b2014f6c0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f3b14035040 tx=0x7f3b140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:04.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.677+0000 7f3b07fff640 1 -- 172.21.15.88:0/3398659606 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b14037050 con 0x7f3b20073130 2024-09-16T01:40:04.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.677+0000 7f3b07fff640 1 -- 172.21.15.88:0/3398659606 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b14038040 con 0x7f3b20073130 2024-09-16T01:40:04.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.677+0000 7f3b25cd0640 1 -- 172.21.15.88:0/3398659606 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b201500a0 con 0x7f3b20073130 2024-09-16T01:40:04.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.677+0000 7f3b07fff640 1 -- 172.21.15.88:0/3398659606 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b140044c0 con 0x7f3b20073130 2024-09-16T01:40:04.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.677+0000 7f3b25cd0640 1 -- 172.21.15.88:0/3398659606 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b20150660 con 0x7f3b20073130 2024-09-16T01:40:04.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.678+0000 7f3b25cd0640 1 -- 172.21.15.88:0/3398659606 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b20072220 con 0x7f3b20073130 2024-09-16T01:40:04.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.679+0000 7f3b07fff640 1 -- 172.21.15.88:0/3398659606 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3b14045020 con 0x7f3b20073130 2024-09-16T01:40:04.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.679+0000 7f3b07fff640 1 --2- 172.21.15.88:0/3398659606 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3b0c06c2e0 0x7f3b0c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:04.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.679+0000 7f3b07fff640 1 -- 172.21.15.88:0/3398659606 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5709+0+0 (secure 0 0 0) 0x7f3b140ab530 con 0x7f3b20073130 2024-09-16T01:40:04.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.679+0000 7f3b1f7fe640 1 --2- 172.21.15.88:0/3398659606 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3b0c06c2e0 0x7f3b0c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:04.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.679+0000 7f3b1f7fe640 1 --2- 172.21.15.88:0/3398659606 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3b0c06c2e0 0x7f3b0c06e7a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3b10005880 tx=0x7f3b10009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:04.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.683+0000 7f3b07fff640 1 -- 172.21.15.88:0/3398659606 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3b20072220 con 0x7f3b20073130 2024-09-16T01:40:04.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.851+0000 7f3b25cd0640 1 -- 172.21.15.88:0/3398659606 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3b20073580 con 0x7f3b20073130 2024-09-16T01:40:04.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.852+0000 7f3b07fff640 1 -- 172.21.15.88:0/3398659606 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+16709 (secure 0 0 0) 0x7f3b1403b360 con 0x7f3b20073130 2024-09-16T01:40:04.854 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:04.854 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":84,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:40:03.597487+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:40:01.941882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-16T01:40:02.564641+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:40:04.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.854+0000 7f3b25cd0640 1 -- 172.21.15.88:0/3398659606 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3b0c06c2e0 msgr2=0x7f3b0c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:04.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.854+0000 7f3b25cd0640 1 --2- 172.21.15.88:0/3398659606 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3b0c06c2e0 0x7f3b0c06e7a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3b10005880 tx=0x7f3b10009040 comp rx=0 tx=0).stop 2024-09-16T01:40:04.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.854+0000 7f3b25cd0640 1 -- 172.21.15.88:0/3398659606 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b20073130 msgr2=0x7f3b2014f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:04.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.854+0000 7f3b25cd0640 1 --2- 172.21.15.88:0/3398659606 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b20073130 0x7f3b2014f6c0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f3b14035040 tx=0x7f3b140046c0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.854+0000 7f3b25cd0640 1 -- 172.21.15.88:0/3398659606 shutdown_connections 2024-09-16T01:40:04.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.854+0000 7f3b25cd0640 1 --2- 172.21.15.88:0/3398659606 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3b0c06c2e0 0x7f3b0c06e7a0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.854+0000 7f3b25cd0640 1 --2- 172.21.15.88:0/3398659606 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b2015e560 0x7f3b2014d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.854+0000 7f3b25cd0640 1 --2- 172.21.15.88:0/3398659606 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3b20073130 0x7f3b2014f6c0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.854+0000 7f3b25cd0640 1 --2- 172.21.15.88:0/3398659606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b2006e780 0x7f3b20150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.854+0000 7f3b25cd0640 1 -- 172.21.15.88:0/3398659606 >> 172.21.15.88:0/3398659606 conn(0x7f3b2006bc10 msgr2=0x7f3b200b6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:04.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.855+0000 7f3b25cd0640 1 -- 172.21.15.88:0/3398659606 shutdown_connections 2024-09-16T01:40:04.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.855+0000 7f3b25cd0640 1 -- 172.21.15.88:0/3398659606 wait complete. 2024-09-16T01:40:04.867 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-16T01:40:04.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.967+0000 7fa4149af640 1 Processor -- start 2024-09-16T01:40:04.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.967+0000 7fa4149af640 1 -- start start 2024-09-16T01:40:04.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.968+0000 7fa4149af640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa41015a050 0x7fa41015d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:04.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.968+0000 7fa4149af640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa41015ee90 0x7fa410073600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:04.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.968+0000 7fa4149af640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa41015a430 con 0x7fa4100731c0 2024-09-16T01:40:04.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.968+0000 7fa4149af640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa410073b40 con 0x7fa41015ee90 2024-09-16T01:40:04.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.968+0000 7fa4149af640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa41015d9a0 con 0x7fa41015a050 2024-09-16T01:40:04.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.968+0000 7fa40ed9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa41015ee90 0x7fa410073600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:04.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.968+0000 7fa40ed9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa41015ee90 0x7fa410073600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59976/0 (socket says 172.21.15.88:59976) 2024-09-16T01:40:04.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.968+0000 7fa407fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa4100731c0 0x7fa41015ffd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33284/0 (socket says 172.21.15.88:33284) 2024-09-16T01:40:04.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.968+0000 7fa40ed9d640 1 -- 172.21.15.88:0/3662623483 learned_addr learned my addr 172.21.15.88:0/3662623483 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:04.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.968+0000 7fa40e59c640 1 --2- 172.21.15.88:0/3662623483 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa41015a050 0x7fa41015d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:04.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.968+0000 7fa40ed9d640 1 -- 172.21.15.88:0/3662623483 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa41015a050 msgr2=0x7fa41015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:04.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.968+0000 7fa40ed9d640 1 --2- 172.21.15.88:0/3662623483 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa41015a050 0x7fa41015d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.968+0000 7fa40ed9d640 1 -- 172.21.15.88:0/3662623483 >> v1:172.21.15.88:6789/0 conn(0x7fa4100731c0 legacy=0x7fa41015ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:04.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.968+0000 7fa40ed9d640 1 -- 172.21.15.88:0/3662623483 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4100cd0a0 con 0x7fa41015ee90 2024-09-16T01:40:04.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.968+0000 7fa40dd9b640 1 -- 172.21.15.88:0/3662623483 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1367244299 0 0) 0x7fa41015a430 con 0x7fa4100731c0 2024-09-16T01:40:04.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.969+0000 7fa40ed9d640 1 --2- 172.21.15.88:0/3662623483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa41015ee90 0x7fa410073600 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fa408009e50 tx=0x7fa40802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e7a5e908dc500d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:04.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.969+0000 7fa40dd9b640 1 -- 172.21.15.88:0/3662623483 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa408036070 con 0x7fa41015ee90 2024-09-16T01:40:04.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.969+0000 7fa40dd9b640 1 -- 172.21.15.88:0/3662623483 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa408031600 con 0x7fa41015ee90 2024-09-16T01:40:04.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.969+0000 7fa40dd9b640 1 -- 172.21.15.88:0/3662623483 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4080319d0 con 0x7fa41015ee90 2024-09-16T01:40:04.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.970+0000 7fa4149af640 1 -- 172.21.15.88:0/3662623483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa41015ee90 msgr2=0x7fa410073600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:04.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.970+0000 7fa4149af640 1 --2- 172.21.15.88:0/3662623483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa41015ee90 0x7fa410073600 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fa408009e50 tx=0x7fa40802a680 comp rx=0 tx=0).stop 2024-09-16T01:40:04.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.970+0000 7fa4149af640 1 -- 172.21.15.88:0/3662623483 shutdown_connections 2024-09-16T01:40:04.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.970+0000 7fa4149af640 1 --2- 172.21.15.88:0/3662623483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa41015ee90 0x7fa410073600 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.970+0000 7fa4149af640 1 --2- 172.21.15.88:0/3662623483 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa41015a050 0x7fa41015d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.970+0000 7fa4149af640 1 -- 172.21.15.88:0/3662623483 >> 172.21.15.88:0/3662623483 conn(0x7fa41006d0f0 msgr2=0x7fa41006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:04.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.970+0000 7fa4149af640 1 -- 172.21.15.88:0/3662623483 shutdown_connections 2024-09-16T01:40:04.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.970+0000 7fa4149af640 1 -- 172.21.15.88:0/3662623483 wait complete. 2024-09-16T01:40:04.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.971+0000 7fa4149af640 1 Processor -- start 2024-09-16T01:40:04.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.971+0000 7fa4149af640 1 -- start start 2024-09-16T01:40:04.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.972+0000 7fa4149af640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4100731c0 0x7fa410158260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:04.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.972+0000 7fa4149af640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa41015a050 0x7fa410155a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:04.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.972+0000 7fa4149af640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa41015ee90 0x7fa410155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:04.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.972+0000 7fa4149af640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa4100cd940 con 0x7fa41015a050 2024-09-16T01:40:04.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.972+0000 7fa4149af640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa4100cd660 con 0x7fa4100731c0 2024-09-16T01:40:04.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.972+0000 7fa4149af640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa4100cdc20 con 0x7fa41015ee90 2024-09-16T01:40:04.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.972+0000 7fa407fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa41015a050 0x7fa410155a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:04.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.972+0000 7fa407fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa41015a050 0x7fa410155a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59990/0 (socket says 172.21.15.88:59990) 2024-09-16T01:40:04.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.972+0000 7fa407fff640 1 -- 172.21.15.88:0/3092589410 learned_addr learned my addr 172.21.15.88:0/3092589410 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:04.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.972+0000 7fa40e59c640 1 --2- 172.21.15.88:0/3092589410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4100731c0 0x7fa410158260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:04.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.972+0000 7fa407fff640 1 -- 172.21.15.88:0/3092589410 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa41015ee90 msgr2=0x7fa410155f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:04.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.972+0000 7fa407fff640 1 --2- 172.21.15.88:0/3092589410 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa41015ee90 0x7fa410155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.972+0000 7fa407fff640 1 -- 172.21.15.88:0/3092589410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4100731c0 msgr2=0x7fa410158260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:04.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.972+0000 7fa407fff640 1 --2- 172.21.15.88:0/3092589410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4100731c0 0x7fa410158260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:04.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.972+0000 7fa407fff640 1 -- 172.21.15.88:0/3092589410 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4100c6fd0 con 0x7fa41015a050 2024-09-16T01:40:04.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.972+0000 7fa407fff640 1 --2- 172.21.15.88:0/3092589410 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa41015a050 0x7fa410155a50 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fa3f800c4e0 tx=0x7fa3f800c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:04.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.972+0000 7fa406ffd640 1 -- 172.21.15.88:0/3092589410 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8018070 con 0x7fa41015a050 2024-09-16T01:40:04.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.972+0000 7fa406ffd640 1 -- 172.21.15.88:0/3092589410 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3f8007500 con 0x7fa41015a050 2024-09-16T01:40:04.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.972+0000 7fa406ffd640 1 -- 172.21.15.88:0/3092589410 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8004a20 con 0x7fa41015a050 2024-09-16T01:40:04.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.972+0000 7fa4149af640 1 -- 172.21.15.88:0/3092589410 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4100c72b0 con 0x7fa41015a050 2024-09-16T01:40:04.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.972+0000 7fa4149af640 1 -- 172.21.15.88:0/3092589410 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4100c7870 con 0x7fa41015a050 2024-09-16T01:40:04.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.973+0000 7fa4149af640 1 -- 172.21.15.88:0/3092589410 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4100b36e0 con 0x7fa41015a050 2024-09-16T01:40:04.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.974+0000 7fa406ffd640 1 -- 172.21.15.88:0/3092589410 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa3f8004bc0 con 0x7fa41015a050 2024-09-16T01:40:04.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.974+0000 7fa406ffd640 1 --2- 172.21.15.88:0/3092589410 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa3f006c2b0 0x7fa3f006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:04.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.974+0000 7fa406ffd640 1 -- 172.21.15.88:0/3092589410 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 5709+0+0 (secure 0 0 0) 0x7fa3f808a740 con 0x7fa41015a050 2024-09-16T01:40:04.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.974+0000 7fa40e59c640 1 --2- 172.21.15.88:0/3092589410 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa3f006c2b0 0x7fa3f006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:04.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.975+0000 7fa40e59c640 1 --2- 172.21.15.88:0/3092589410 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa3f006c2b0 0x7fa3f006e770 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa400006fd0 tx=0x7fa400008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:04.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:04.978+0000 7fa406ffd640 1 -- 172.21.15.88:0/3092589410 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa3f8059000 con 0x7fa41015a050 2024-09-16T01:40:05.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.177+0000 7fa4149af640 1 -- 172.21.15.88:0/3092589410 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fa41015b720 con 0x7fa41015a050 2024-09-16T01:40:05.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.271+0000 7fa406ffd640 1 -- 172.21.15.88:0/3092589410 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v69) ==== 118+0+0 (secure 0 0 0) 0x7fa3f8058900 con 0x7fa41015a050 2024-09-16T01:40:05.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.273+0000 7fa4149af640 1 -- 172.21.15.88:0/3092589410 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa3f006c2b0 msgr2=0x7fa3f006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:05.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.273+0000 7fa4149af640 1 --2- 172.21.15.88:0/3092589410 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa3f006c2b0 0x7fa3f006e770 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa400006fd0 tx=0x7fa400008040 comp rx=0 tx=0).stop 2024-09-16T01:40:05.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.273+0000 7fa4149af640 1 -- 172.21.15.88:0/3092589410 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa41015a050 msgr2=0x7fa410155a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:05.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.273+0000 7fa4149af640 1 --2- 172.21.15.88:0/3092589410 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa41015a050 0x7fa410155a50 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fa3f800c4e0 tx=0x7fa3f800c9a0 comp rx=0 tx=0).stop 2024-09-16T01:40:05.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.273+0000 7fa4149af640 1 -- 172.21.15.88:0/3092589410 shutdown_connections 2024-09-16T01:40:05.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.273+0000 7fa4149af640 1 --2- 172.21.15.88:0/3092589410 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa3f006c2b0 0x7fa3f006e770 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:05.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.273+0000 7fa4149af640 1 --2- 172.21.15.88:0/3092589410 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa41015ee90 0x7fa410155f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:05.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.273+0000 7fa4149af640 1 --2- 172.21.15.88:0/3092589410 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa41015a050 0x7fa410155a50 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:05.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.273+0000 7fa4149af640 1 --2- 172.21.15.88:0/3092589410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4100731c0 0x7fa410158260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:05.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.273+0000 7fa4149af640 1 -- 172.21.15.88:0/3092589410 >> 172.21.15.88:0/3092589410 conn(0x7fa41006d0f0 msgr2=0x7fa4100b45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:05.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.273+0000 7fa4149af640 1 -- 172.21.15.88:0/3092589410 shutdown_connections 2024-09-16T01:40:05.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.273+0000 7fa4149af640 1 -- 172.21.15.88:0/3092589410 wait complete. 2024-09-16T01:40:05.285 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-16T01:40:05.286 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-16T01:40:05.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.386+0000 7f6381e53640 1 Processor -- start 2024-09-16T01:40:05.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.386+0000 7f6381e53640 1 -- start start 2024-09-16T01:40:05.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.387+0000 7f6381e53640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f637c15a060 0x7f637c06f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:05.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.387+0000 7f6381e53640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f637c0b1990 0x7f637c06f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:05.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.387+0000 7f6381e53640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f637c071770 con 0x7f637c15d3d0 2024-09-16T01:40:05.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.387+0000 7f6381e53640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f637c0712e0 con 0x7f637c15a060 2024-09-16T01:40:05.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.387+0000 7f6381e53640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f637c0709b0 con 0x7f637c0b1990 2024-09-16T01:40:05.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.387+0000 7f637b7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f637c15a060 0x7f637c06f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:05.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.387+0000 7f637b7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f637c15a060 0x7f637c06f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60000/0 (socket says 172.21.15.88:60000) 2024-09-16T01:40:05.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.387+0000 7f637b7fe640 1 -- 172.21.15.88:0/3623712452 learned_addr learned my addr 172.21.15.88:0/3623712452 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:05.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.387+0000 7f637affd640 1 --1- 172.21.15.88:0/3623712452 >> v1:172.21.15.88:6789/0 conn(0x7f637c15d3d0 0x7f637c15ff40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33288/0 (socket says 172.21.15.88:33288) 2024-09-16T01:40:05.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.388+0000 7f637b7fe640 1 -- 172.21.15.88:0/3623712452 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f637c0b1990 msgr2=0x7f637c06f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:05.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.388+0000 7f637bfff640 1 --2- 172.21.15.88:0/3623712452 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f637c0b1990 0x7f637c06f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:05.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.388+0000 7f637a7fc640 1 -- 172.21.15.88:0/3623712452 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 461414159 0 0) 0x7f637c071770 con 0x7f637c15d3d0 2024-09-16T01:40:05.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.388+0000 7f637b7fe640 1 --2- 172.21.15.88:0/3623712452 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f637c0b1990 0x7f637c06f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:05.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.388+0000 7f637b7fe640 1 -- 172.21.15.88:0/3623712452 >> v1:172.21.15.88:6789/0 conn(0x7f637c15d3d0 legacy=0x7f637c15ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:05.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.388+0000 7f637b7fe640 1 -- 172.21.15.88:0/3623712452 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f637c070c90 con 0x7f637c15a060 2024-09-16T01:40:05.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.388+0000 7f637bfff640 1 --2- 172.21.15.88:0/3623712452 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f637c0b1990 0x7f637c06f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:05.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.388+0000 7f637b7fe640 1 --2- 172.21.15.88:0/3623712452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c15a060 0x7f637c06f2f0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f636c009870 tx=0x7f636c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22876205424c91d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:05.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.388+0000 7f637a7fc640 1 -- 172.21.15.88:0/3623712452 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636c037070 con 0x7f637c15a060 2024-09-16T01:40:05.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.388+0000 7f637a7fc640 1 -- 172.21.15.88:0/3623712452 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f636c032440 con 0x7f637c15a060 2024-09-16T01:40:05.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.388+0000 7f637a7fc640 1 -- 172.21.15.88:0/3623712452 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636c032810 con 0x7f637c15a060 2024-09-16T01:40:05.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.389+0000 7f6381e53640 1 -- 172.21.15.88:0/3623712452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c15a060 msgr2=0x7f637c06f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:05.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.389+0000 7f6381e53640 1 --2- 172.21.15.88:0/3623712452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c15a060 0x7f637c06f2f0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f636c009870 tx=0x7f636c02a270 comp rx=0 tx=0).stop 2024-09-16T01:40:05.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.389+0000 7f6381e53640 1 -- 172.21.15.88:0/3623712452 shutdown_connections 2024-09-16T01:40:05.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.389+0000 7f6381e53640 1 --2- 172.21.15.88:0/3623712452 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f637c0b1990 0x7f637c06f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:05.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.389+0000 7f6381e53640 1 --2- 172.21.15.88:0/3623712452 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c15a060 0x7f637c06f2f0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:05.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.389+0000 7f6381e53640 1 -- 172.21.15.88:0/3623712452 >> 172.21.15.88:0/3623712452 conn(0x7f637c0d2430 msgr2=0x7f637c0d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:05.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.389+0000 7f6381e53640 1 -- 172.21.15.88:0/3623712452 shutdown_connections 2024-09-16T01:40:05.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.389+0000 7f6381e53640 1 -- 172.21.15.88:0/3623712452 wait complete. 2024-09-16T01:40:05.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.390+0000 7f6381e53640 1 Processor -- start 2024-09-16T01:40:05.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.390+0000 7f6381e53640 1 -- start start 2024-09-16T01:40:05.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.391+0000 7f6381e53640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f637c0b1990 0x7f637c153310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:05.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.391+0000 7f6381e53640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f637c15a060 0x7f637c1f5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:05.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.391+0000 7f6381e53640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c15d3d0 0x7f637c160070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:05.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.391+0000 7f6381e53640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f637c0d1460 con 0x7f637c15d3d0 2024-09-16T01:40:05.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.391+0000 7f6381e53640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f637c0d12e0 con 0x7f637c15a060 2024-09-16T01:40:05.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.391+0000 7f6381e53640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f637c06feb0 con 0x7f637c0b1990 2024-09-16T01:40:05.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.391+0000 7f637bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c15d3d0 0x7f637c160070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:05.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.391+0000 7f637bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c15d3d0 0x7f637c160070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60006/0 (socket says 172.21.15.88:60006) 2024-09-16T01:40:05.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.391+0000 7f637bfff640 1 -- 172.21.15.88:0/1077036173 learned_addr learned my addr 172.21.15.88:0/1077036173 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:05.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.391+0000 7f637b7fe640 1 --2- 172.21.15.88:0/1077036173 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f637c0b1990 0x7f637c153310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:05.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.392+0000 7f637affd640 1 --2- 172.21.15.88:0/1077036173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f637c15a060 0x7f637c1f5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:05.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.392+0000 7f637bfff640 1 -- 172.21.15.88:0/1077036173 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f637c0b1990 msgr2=0x7f637c153310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:05.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.392+0000 7f637bfff640 1 --2- 172.21.15.88:0/1077036173 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f637c0b1990 0x7f637c153310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:05.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.392+0000 7f637bfff640 1 -- 172.21.15.88:0/1077036173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f637c15a060 msgr2=0x7f637c1f5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:05.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.392+0000 7f637bfff640 1 --2- 172.21.15.88:0/1077036173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f637c15a060 0x7f637c1f5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:05.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.392+0000 7f637bfff640 1 -- 172.21.15.88:0/1077036173 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f637c153960 con 0x7f637c15d3d0 2024-09-16T01:40:05.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.392+0000 7f637affd640 1 --2- 172.21.15.88:0/1077036173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f637c15a060 0x7f637c1f5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:05.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.392+0000 7f637bfff640 1 --2- 172.21.15.88:0/1077036173 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c15d3d0 0x7f637c160070 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f637400cf20 tx=0x7f637400a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:05.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.392+0000 7f6363fff640 1 -- 172.21.15.88:0/1077036173 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f637401a070 con 0x7f637c15d3d0 2024-09-16T01:40:05.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.392+0000 7f6363fff640 1 -- 172.21.15.88:0/1077036173 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f637400d840 con 0x7f637c15d3d0 2024-09-16T01:40:05.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.392+0000 7f6381e53640 1 -- 172.21.15.88:0/1077036173 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f637c153c40 con 0x7f637c15d3d0 2024-09-16T01:40:05.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.392+0000 7f6363fff640 1 -- 172.21.15.88:0/1077036173 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f637400dc30 con 0x7f637c15d3d0 2024-09-16T01:40:05.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.392+0000 7f6381e53640 1 -- 172.21.15.88:0/1077036173 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f637c154200 con 0x7f637c15d3d0 2024-09-16T01:40:05.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.393+0000 7f6381e53640 1 -- 172.21.15.88:0/1077036173 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f637c06c6c0 con 0x7f637c15d3d0 2024-09-16T01:40:05.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.394+0000 7f6363fff640 1 -- 172.21.15.88:0/1077036173 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f637401d070 con 0x7f637c15d3d0 2024-09-16T01:40:05.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.394+0000 7f6363fff640 1 --2- 172.21.15.88:0/1077036173 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f636806c2b0 0x7f636806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:05.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.394+0000 7f6363fff640 1 -- 172.21.15.88:0/1077036173 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f637408bcb0 con 0x7f637c15d3d0 2024-09-16T01:40:05.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.394+0000 7f637b7fe640 1 --2- 172.21.15.88:0/1077036173 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f636806c2b0 0x7f636806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:05.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.395+0000 7f637b7fe640 1 --2- 172.21.15.88:0/1077036173 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f636806c2b0 0x7f636806e770 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f636c034000 tx=0x7f636c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:05.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.398+0000 7f6363fff640 1 -- 172.21.15.88:0/1077036173 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f637405a420 con 0x7f637c15d3d0 2024-09-16T01:40:05.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:05.601+0000 7f6381e53640 1 -- 172.21.15.88:0/1077036173 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f637c0cf0e0 con 0x7f637c15d3d0 2024-09-16T01:40:06.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.275+0000 7f6363fff640 1 -- 172.21.15.88:0/1077036173 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v70) ==== 122+0+0 (secure 0 0 0) 0x7f637c0cf0e0 con 0x7f637c15d3d0 2024-09-16T01:40:06.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.278+0000 7f6381e53640 1 -- 172.21.15.88:0/1077036173 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f636806c2b0 msgr2=0x7f636806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:06.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.278+0000 7f6381e53640 1 --2- 172.21.15.88:0/1077036173 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f636806c2b0 0x7f636806e770 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f636c034000 tx=0x7f636c02e000 comp rx=0 tx=0).stop 2024-09-16T01:40:06.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.278+0000 7f6381e53640 1 -- 172.21.15.88:0/1077036173 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c15d3d0 msgr2=0x7f637c160070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:06.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.278+0000 7f6381e53640 1 --2- 172.21.15.88:0/1077036173 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c15d3d0 0x7f637c160070 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f637400cf20 tx=0x7f637400a7d0 comp rx=0 tx=0).stop 2024-09-16T01:40:06.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.278+0000 7f6381e53640 1 -- 172.21.15.88:0/1077036173 shutdown_connections 2024-09-16T01:40:06.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.278+0000 7f6381e53640 1 --2- 172.21.15.88:0/1077036173 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f636806c2b0 0x7f636806e770 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:06.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.278+0000 7f6381e53640 1 --2- 172.21.15.88:0/1077036173 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c15d3d0 0x7f637c160070 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:06.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.278+0000 7f6381e53640 1 --2- 172.21.15.88:0/1077036173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f637c15a060 0x7f637c1f5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:06.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.278+0000 7f6381e53640 1 --2- 172.21.15.88:0/1077036173 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f637c0b1990 0x7f637c153310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:06.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.278+0000 7f6381e53640 1 -- 172.21.15.88:0/1077036173 >> 172.21.15.88:0/1077036173 conn(0x7f637c0d2430 msgr2=0x7f637c15ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:06.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.278+0000 7f6381e53640 1 -- 172.21.15.88:0/1077036173 shutdown_connections 2024-09-16T01:40:06.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.278+0000 7f6381e53640 1 -- 172.21.15.88:0/1077036173 wait complete. 2024-09-16T01:40:06.292 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-16T01:40:06.292 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:40:06.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.391+0000 7f4f6808b640 1 Processor -- start 2024-09-16T01:40:06.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.391+0000 7f4f6808b640 1 -- start start 2024-09-16T01:40:06.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.392+0000 7f4f6808b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4f6015c710 0x7f4f60072050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:06.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.392+0000 7f4f6808b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4f60072590 0x7f4f6015cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:06.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.392+0000 7f4f6808b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f60072a30 con 0x7f4f6006bc10 2024-09-16T01:40:06.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.392+0000 7f4f6808b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f4f60072d10 con 0x7f4f60072590 2024-09-16T01:40:06.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.392+0000 7f4f6808b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f4f60072ff0 con 0x7f4f6015c710 2024-09-16T01:40:06.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.392+0000 7f4f66628640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4f60072590 0x7f4f6015cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:06.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.392+0000 7f4f66628640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4f60072590 0x7f4f6015cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60010/0 (socket says 172.21.15.88:60010) 2024-09-16T01:40:06.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.392+0000 7f4f66628640 1 -- 172.21.15.88:0/873412681 learned_addr learned my addr 172.21.15.88:0/873412681 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:06.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.393+0000 7f4f65626640 1 --2- 172.21.15.88:0/873412681 >> v2:172.21.15.154:3300/0 conn(0x7f4f6015c710 0x7f4f60072050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:06.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.393+0000 7f4f66628640 1 -- 172.21.15.88:0/873412681 >> v2:172.21.15.154:3300/0 conn(0x7f4f6015c710 msgr2=0x7f4f60072050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:06.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.393+0000 7f4f66628640 1 --2- 172.21.15.88:0/873412681 >> v2:172.21.15.154:3300/0 conn(0x7f4f6015c710 0x7f4f60072050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:06.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.393+0000 7f4f66628640 1 -- 172.21.15.88:0/873412681 >> v1:172.21.15.154:6789/0 conn(0x7f4f6006bc10 legacy=0x7f4f6006bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:06.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.393+0000 7f4f66628640 1 -- 172.21.15.88:0/873412681 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f600732d0 con 0x7f4f60072590 2024-09-16T01:40:06.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.393+0000 7f4f65626640 1 --2- 172.21.15.88:0/873412681 >> v2:172.21.15.154:3300/0 conn(0x7f4f6015c710 0x7f4f60072050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:06.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.393+0000 7f4f66628640 1 --2- 172.21.15.88:0/873412681 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f60072590 0x7f4f6015cdb0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f4f600b68d0 tx=0x7f4f5802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88a7123dfea357a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:06.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.393+0000 7f4f64e25640 1 -- 172.21.15.88:0/873412681 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f58036070 con 0x7f4f60072590 2024-09-16T01:40:06.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.393+0000 7f4f64e25640 1 -- 172.21.15.88:0/873412681 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f58031600 con 0x7f4f60072590 2024-09-16T01:40:06.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.393+0000 7f4f64e25640 1 -- 172.21.15.88:0/873412681 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f580319d0 con 0x7f4f60072590 2024-09-16T01:40:06.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.394+0000 7f4f6808b640 1 -- 172.21.15.88:0/873412681 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f60072590 msgr2=0x7f4f6015cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:06.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.394+0000 7f4f6808b640 1 --2- 172.21.15.88:0/873412681 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f60072590 0x7f4f6015cdb0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f4f600b68d0 tx=0x7f4f5802a6f0 comp rx=0 tx=0).stop 2024-09-16T01:40:06.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.394+0000 7f4f6808b640 1 -- 172.21.15.88:0/873412681 shutdown_connections 2024-09-16T01:40:06.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.394+0000 7f4f6808b640 1 --2- 172.21.15.88:0/873412681 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f60072590 0x7f4f6015cdb0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:06.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.394+0000 7f4f6808b640 1 --2- 172.21.15.88:0/873412681 >> v2:172.21.15.154:3300/0 conn(0x7f4f6015c710 0x7f4f60072050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:06.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.394+0000 7f4f6808b640 1 -- 172.21.15.88:0/873412681 >> 172.21.15.88:0/873412681 conn(0x7f4f600d14d0 msgr2=0x7f4f600d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:06.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.394+0000 7f4f6808b640 1 -- 172.21.15.88:0/873412681 shutdown_connections 2024-09-16T01:40:06.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.394+0000 7f4f6808b640 1 -- 172.21.15.88:0/873412681 wait complete. 2024-09-16T01:40:06.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.395+0000 7f4f6808b640 1 Processor -- start 2024-09-16T01:40:06.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.395+0000 7f4f6808b640 1 -- start start 2024-09-16T01:40:06.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.396+0000 7f4f6808b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f6006bc10 0x7f4f600c8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:06.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.396+0000 7f4f6808b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f60072590 0x7f4f60149b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:06.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.396+0000 7f4f6808b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f6015c710 0x7f4f6014a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:06.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.396+0000 7f4f6808b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6006f9d0 con 0x7f4f6006bc10 2024-09-16T01:40:06.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.396+0000 7f4f6808b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6006f850 con 0x7f4f6015c710 2024-09-16T01:40:06.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.396+0000 7f4f6808b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4f6015d2f0 con 0x7f4f60072590 2024-09-16T01:40:06.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.396+0000 7f4f65e27640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f6006bc10 0x7f4f600c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:06.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.396+0000 7f4f65e27640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f6006bc10 0x7f4f600c8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60018/0 (socket says 172.21.15.88:60018) 2024-09-16T01:40:06.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.396+0000 7f4f65e27640 1 -- 172.21.15.88:0/1460674482 learned_addr learned my addr 172.21.15.88:0/1460674482 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:06.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.396+0000 7f4f66628640 1 --2- 172.21.15.88:0/1460674482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f6015c710 0x7f4f6014a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:06.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.396+0000 7f4f65626640 1 --2- 172.21.15.88:0/1460674482 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f60072590 0x7f4f60149b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:06.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.396+0000 7f4f65e27640 1 -- 172.21.15.88:0/1460674482 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f60072590 msgr2=0x7f4f60149b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:06.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.396+0000 7f4f65e27640 1 --2- 172.21.15.88:0/1460674482 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f60072590 0x7f4f60149b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:06.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.396+0000 7f4f65e27640 1 -- 172.21.15.88:0/1460674482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f6015c710 msgr2=0x7f4f6014a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:06.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.396+0000 7f4f65e27640 1 --2- 172.21.15.88:0/1460674482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f6015c710 0x7f4f6014a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:06.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.396+0000 7f4f65e27640 1 -- 172.21.15.88:0/1460674482 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f60155a40 con 0x7f4f6006bc10 2024-09-16T01:40:06.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.396+0000 7f4f65626640 1 --2- 172.21.15.88:0/1460674482 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f60072590 0x7f4f60149b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:06.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.397+0000 7f4f65e27640 1 --2- 172.21.15.88:0/1460674482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f6006bc10 0x7f4f600c8950 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c00c460 tx=0x7f4f5c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:06.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.397+0000 7f4f567fc640 1 -- 172.21.15.88:0/1460674482 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c016020 con 0x7f4f6006bc10 2024-09-16T01:40:06.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.397+0000 7f4f567fc640 1 -- 172.21.15.88:0/1460674482 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f5c012870 con 0x7f4f6006bc10 2024-09-16T01:40:06.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.397+0000 7f4f6808b640 1 -- 172.21.15.88:0/1460674482 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f60155d20 con 0x7f4f6006bc10 2024-09-16T01:40:06.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.397+0000 7f4f567fc640 1 -- 172.21.15.88:0/1460674482 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c012c60 con 0x7f4f6006bc10 2024-09-16T01:40:06.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.397+0000 7f4f6808b640 1 -- 172.21.15.88:0/1460674482 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f601562e0 con 0x7f4f6006bc10 2024-09-16T01:40:06.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.398+0000 7f4f6808b640 1 -- 172.21.15.88:0/1460674482 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f6015cdb0 con 0x7f4f6006bc10 2024-09-16T01:40:06.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.398+0000 7f4f567fc640 1 -- 172.21.15.88:0/1460674482 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4f5c019020 con 0x7f4f6006bc10 2024-09-16T01:40:06.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.399+0000 7f4f567fc640 1 --2- 172.21.15.88:0/1460674482 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f4406c2b0 0x7f4f4406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:06.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.399+0000 7f4f567fc640 1 -- 172.21.15.88:0/1460674482 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f4f5c08b4c0 con 0x7f4f6006bc10 2024-09-16T01:40:06.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.399+0000 7f4f65626640 1 --2- 172.21.15.88:0/1460674482 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f4406c2b0 0x7f4f4406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:06.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.399+0000 7f4f65626640 1 --2- 172.21.15.88:0/1460674482 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f4406c2b0 0x7f4f4406e770 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4f50006fd0 tx=0x7f4f50008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:06.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.403+0000 7f4f567fc640 1 -- 172.21.15.88:0/1460674482 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4f5c020020 con 0x7f4f6006bc10 2024-09-16T01:40:06.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.603+0000 7f4f6808b640 1 -- 172.21.15.88:0/1460674482 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f600b5d20 con 0x7f4f6006bc10 2024-09-16T01:40:06.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.603+0000 7f4f567fc640 1 -- 172.21.15.88:0/1460674482 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) ==== 94+0+5228 (secure 0 0 0) 0x7f4f5c0182c0 con 0x7f4f6006bc10 2024-09-16T01:40:06.605 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:06.605 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":70,"btime":"2024-09-16T01:40:06:272351+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6849,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/1289932173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1289932173},{"type":"v1","addr":"172.21.15.88:6836","nonce":1289932173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:40:05.602153+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6852":{"gid":6852,"name":"c","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6837/4162274767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4162274767},{"type":"v1","addr":"172.21.15.88:6837","nonce":4162274767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6852,"qdb_cluster":[6852]},"id":6}]} 2024-09-16T01:40:06.605 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 70 2024-09-16T01:40:06.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.606+0000 7f4f6808b640 1 -- 172.21.15.88:0/1460674482 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f4406c2b0 msgr2=0x7f4f4406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:06.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.606+0000 7f4f6808b640 1 --2- 172.21.15.88:0/1460674482 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f4406c2b0 0x7f4f4406e770 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f4f50006fd0 tx=0x7f4f50008040 comp rx=0 tx=0).stop 2024-09-16T01:40:06.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.606+0000 7f4f6808b640 1 -- 172.21.15.88:0/1460674482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f6006bc10 msgr2=0x7f4f600c8950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:06.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.606+0000 7f4f6808b640 1 --2- 172.21.15.88:0/1460674482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f6006bc10 0x7f4f600c8950 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c00c460 tx=0x7f4f5c00c920 comp rx=0 tx=0).stop 2024-09-16T01:40:06.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.606+0000 7f4f6808b640 1 -- 172.21.15.88:0/1460674482 shutdown_connections 2024-09-16T01:40:06.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.606+0000 7f4f6808b640 1 --2- 172.21.15.88:0/1460674482 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f4406c2b0 0x7f4f4406e770 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:06.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.606+0000 7f4f6808b640 1 --2- 172.21.15.88:0/1460674482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f6015c710 0x7f4f6014a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:06.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.606+0000 7f4f6808b640 1 --2- 172.21.15.88:0/1460674482 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f60072590 0x7f4f60149b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:06.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.606+0000 7f4f6808b640 1 --2- 172.21.15.88:0/1460674482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f6006bc10 0x7f4f600c8950 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:06.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.606+0000 7f4f6808b640 1 -- 172.21.15.88:0/1460674482 >> 172.21.15.88:0/1460674482 conn(0x7f4f600d14d0 msgr2=0x7f4f600b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:06.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.606+0000 7f4f6808b640 1 -- 172.21.15.88:0/1460674482 shutdown_connections 2024-09-16T01:40:06.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.606+0000 7f4f6808b640 1 -- 172.21.15.88:0/1460674482 wait complete. 2024-09-16T01:40:06.618 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:40:06.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.716+0000 7fba641fd640 1 Processor -- start 2024-09-16T01:40:06.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.717+0000 7fba641fd640 1 -- start start 2024-09-16T01:40:06.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.717+0000 7fba641fd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fba5c15e560 0x7fba5c06e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:06.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.717+0000 7fba641fd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba5c073130 0x7fba5c073580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:06.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.717+0000 7fba641fd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba5c0b5220 con 0x7fba5c072220 2024-09-16T01:40:06.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.717+0000 7fba641fd640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fba5c0aefa0 con 0x7fba5c15e560 2024-09-16T01:40:06.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.717+0000 7fba641fd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fba5c073ac0 con 0x7fba5c073130 2024-09-16T01:40:06.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.718+0000 7fba61798640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fba5c072220 0x7fba5c0ae890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33296/0 (socket says 172.21.15.88:33296) 2024-09-16T01:40:06.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.718+0000 7fba61798640 1 -- 172.21.15.88:0/1773297035 learned_addr learned my addr 172.21.15.88:0/1773297035 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:06.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.718+0000 7fba61f99640 1 --2- 172.21.15.88:0/1773297035 >> v2:172.21.15.154:3300/0 conn(0x7fba5c15e560 0x7fba5c06e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:06.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.718+0000 7fba6279a640 1 --2- 172.21.15.88:0/1773297035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba5c073130 0x7fba5c073580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:06.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.718+0000 7fba60f97640 1 -- 172.21.15.88:0/1773297035 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4093995222 0 0) 0x7fba5c0b5220 con 0x7fba5c072220 2024-09-16T01:40:06.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.718+0000 7fba60f97640 1 -- 172.21.15.88:0/1773297035 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba44003520 con 0x7fba5c072220 2024-09-16T01:40:06.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.718+0000 7fba60f97640 1 -- 172.21.15.88:0/1773297035 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1024114461 0 0) 0x7fba44003520 con 0x7fba5c072220 2024-09-16T01:40:06.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.718+0000 7fba60f97640 1 -- 172.21.15.88:0/1773297035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba5c073130 msgr2=0x7fba5c073580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:06.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.718+0000 7fba60f97640 1 --2- 172.21.15.88:0/1773297035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba5c073130 0x7fba5c073580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:06.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.718+0000 7fba60f97640 1 -- 172.21.15.88:0/1773297035 >> v2:172.21.15.154:3300/0 conn(0x7fba5c15e560 msgr2=0x7fba5c06e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:06.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.718+0000 7fba60f97640 1 --2- 172.21.15.88:0/1773297035 >> v2:172.21.15.154:3300/0 conn(0x7fba5c15e560 0x7fba5c06e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:06.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.718+0000 7fba60f97640 1 -- 172.21.15.88:0/1773297035 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba5c0b54c0 con 0x7fba5c072220 2024-09-16T01:40:06.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.718+0000 7fba60f97640 1 -- 172.21.15.88:0/1773297035 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fba50001670 con 0x7fba5c072220 2024-09-16T01:40:06.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.718+0000 7fba60f97640 1 -- 172.21.15.88:0/1773297035 >> v1:172.21.15.88:6789/0 conn(0x7fba5c072220 legacy=0x7fba5c0ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:06.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.718+0000 7fba60f97640 1 --2- 172.21.15.88:0/1773297035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba440039b0 0x7fba44003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:06.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.718+0000 7fba60f97640 1 --2- 172.21.15.88:0/1773297035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba44004800 0x7fba44004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:06.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.718+0000 7fba60f97640 1 --2- 172.21.15.88:0/1773297035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba5c073130 0x7fba440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:06.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.718+0000 7fba60f97640 1 -- 172.21.15.88:0/1773297035 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fba5c0aefa0 con 0x7fba44004800 2024-09-16T01:40:06.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.718+0000 7fba60f97640 1 -- 172.21.15.88:0/1773297035 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fba5c073ac0 con 0x7fba440039b0 2024-09-16T01:40:06.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.718+0000 7fba60f97640 1 -- 172.21.15.88:0/1773297035 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fba4400b9c0 con 0x7fba5c073130 2024-09-16T01:40:06.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.718+0000 7fba61798640 1 --2- 172.21.15.88:0/1773297035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba44004800 0x7fba44004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:06.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.718+0000 7fba61f99640 1 --2- 172.21.15.88:0/1773297035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba440039b0 0x7fba44003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:06.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.718+0000 7fba6279a640 1 --2- 172.21.15.88:0/1773297035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba5c073130 0x7fba440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:06.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.719+0000 7fba61798640 1 -- 172.21.15.88:0/1773297035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba5c073130 msgr2=0x7fba440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:06.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.719+0000 7fba61798640 1 --2- 172.21.15.88:0/1773297035 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba5c073130 0x7fba440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:06.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.719+0000 7fba61798640 1 -- 172.21.15.88:0/1773297035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba440039b0 msgr2=0x7fba44003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:06.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.719+0000 7fba61798640 1 --2- 172.21.15.88:0/1773297035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba440039b0 0x7fba44003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:06.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.719+0000 7fba61798640 1 -- 172.21.15.88:0/1773297035 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba44003520 con 0x7fba44004800 2024-09-16T01:40:06.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.719+0000 7fba61798640 1 --2- 172.21.15.88:0/1773297035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba44004800 0x7fba44004c80 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fba5000c290 tx=0x7fba44005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c673919a80ed8236 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:06.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.719+0000 7fba60f97640 1 -- 172.21.15.88:0/1773297035 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba50007d50 con 0x7fba44004800 2024-09-16T01:40:06.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.719+0000 7fba60f97640 1 -- 172.21.15.88:0/1773297035 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba50005ad0 con 0x7fba44004800 2024-09-16T01:40:06.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.720+0000 7fba60f97640 1 -- 172.21.15.88:0/1773297035 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba50007070 con 0x7fba44004800 2024-09-16T01:40:06.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.720+0000 7fba641fd640 1 -- 172.21.15.88:0/1773297035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba44004800 msgr2=0x7fba44004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:06.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.720+0000 7fba641fd640 1 --2- 172.21.15.88:0/1773297035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba44004800 0x7fba44004c80 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fba5000c290 tx=0x7fba44005830 comp rx=0 tx=0).stop 2024-09-16T01:40:06.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.720+0000 7fba6279a640 1 -- 172.21.15.88:0/1773297035 reap_dead start 2024-09-16T01:40:06.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.720+0000 7fba641fd640 1 -- 172.21.15.88:0/1773297035 shutdown_connections 2024-09-16T01:40:06.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.720+0000 7fba641fd640 1 -- 172.21.15.88:0/1773297035 >> 172.21.15.88:0/1773297035 conn(0x7fba5c06bc10 msgr2=0x7fba5c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:06.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.720+0000 7fba641fd640 1 -- 172.21.15.88:0/1773297035 shutdown_connections 2024-09-16T01:40:06.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.720+0000 7fba641fd640 1 -- 172.21.15.88:0/1773297035 wait complete. 2024-09-16T01:40:06.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.721+0000 7fba641fd640 1 Processor -- start 2024-09-16T01:40:06.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.721+0000 7fba641fd640 1 -- start start 2024-09-16T01:40:06.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.722+0000 7fba641fd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba44004800 0x7fba5c14e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:06.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.722+0000 7fba641fd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba5c1f7da0 0x7fba5c1f8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:06.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.722+0000 7fba641fd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba5c1f8760 0x7fba5c15b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:06.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.722+0000 7fba641fd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fba5c0b5d60 con 0x7fba5c1f8760 2024-09-16T01:40:06.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.722+0000 7fba641fd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fba5c0b5a80 con 0x7fba44004800 2024-09-16T01:40:06.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.722+0000 7fba641fd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fba5c072660 con 0x7fba5c1f7da0 2024-09-16T01:40:06.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.722+0000 7fba6279a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba5c1f8760 0x7fba5c15b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:06.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.722+0000 7fba6279a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba5c1f8760 0x7fba5c15b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60040/0 (socket says 172.21.15.88:60040) 2024-09-16T01:40:06.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.722+0000 7fba6279a640 1 -- 172.21.15.88:0/2506142571 learned_addr learned my addr 172.21.15.88:0/2506142571 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:06.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.722+0000 7fba61798640 1 --2- 172.21.15.88:0/2506142571 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba5c1f7da0 0x7fba5c1f8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:06.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.722+0000 7fba61f99640 1 --2- 172.21.15.88:0/2506142571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba44004800 0x7fba5c14e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:06.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.722+0000 7fba6279a640 1 -- 172.21.15.88:0/2506142571 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba5c1f7da0 msgr2=0x7fba5c1f8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:06.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.722+0000 7fba6279a640 1 --2- 172.21.15.88:0/2506142571 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba5c1f7da0 0x7fba5c1f8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:06.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.722+0000 7fba6279a640 1 -- 172.21.15.88:0/2506142571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba44004800 msgr2=0x7fba5c14e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:06.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.722+0000 7fba6279a640 1 --2- 172.21.15.88:0/2506142571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba44004800 0x7fba5c14e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:06.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.722+0000 7fba6279a640 1 -- 172.21.15.88:0/2506142571 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba5c161250 con 0x7fba5c1f8760 2024-09-16T01:40:06.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.722+0000 7fba61f99640 1 --2- 172.21.15.88:0/2506142571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba44004800 0x7fba5c14e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:06.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.722+0000 7fba6279a640 1 --2- 172.21.15.88:0/2506142571 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba5c1f8760 0x7fba5c15b7c0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fba5c0cd440 tx=0x7fba4c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:06.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.723+0000 7fba5a7fc640 1 -- 172.21.15.88:0/2506142571 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba4c00fe10 con 0x7fba5c1f8760 2024-09-16T01:40:06.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.723+0000 7fba5a7fc640 1 -- 172.21.15.88:0/2506142571 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba4c01d070 con 0x7fba5c1f8760 2024-09-16T01:40:06.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.723+0000 7fba641fd640 1 -- 172.21.15.88:0/2506142571 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba5c161530 con 0x7fba5c1f8760 2024-09-16T01:40:06.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.723+0000 7fba5a7fc640 1 -- 172.21.15.88:0/2506142571 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba4c015670 con 0x7fba5c1f8760 2024-09-16T01:40:06.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.723+0000 7fba641fd640 1 -- 172.21.15.88:0/2506142571 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba5c15afc0 con 0x7fba5c1f8760 2024-09-16T01:40:06.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.724+0000 7fba641fd640 1 -- 172.21.15.88:0/2506142571 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba5c0b6b80 con 0x7fba5c1f8760 2024-09-16T01:40:06.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.724+0000 7fba5a7fc640 1 -- 172.21.15.88:0/2506142571 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fba4c015810 con 0x7fba5c1f8760 2024-09-16T01:40:06.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.724+0000 7fba5a7fc640 1 --2- 172.21.15.88:0/2506142571 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba4806c2b0 0x7fba4806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:06.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.725+0000 7fba5a7fc640 1 -- 172.21.15.88:0/2506142571 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7fba4c08bca0 con 0x7fba5c1f8760 2024-09-16T01:40:06.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.725+0000 7fba61f99640 1 --2- 172.21.15.88:0/2506142571 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba4806c2b0 0x7fba4806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:06.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.725+0000 7fba61f99640 1 --2- 172.21.15.88:0/2506142571 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba4806c2b0 0x7fba4806e770 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fba54009bc0 tx=0x7fba54008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:06.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.729+0000 7fba5a7fc640 1 -- 172.21.15.88:0/2506142571 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fba4c010040 con 0x7fba5c1f8760 2024-09-16T01:40:06.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.896+0000 7fba641fd640 1 -- 172.21.15.88:0/2506142571 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fba5c0b0210 con 0x7fba5c1f8760 2024-09-16T01:40:06.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.897+0000 7fba5a7fc640 1 -- 172.21.15.88:0/2506142571 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+16696 (secure 0 0 0) 0x7fba4c05a410 con 0x7fba5c1f8760 2024-09-16T01:40:06.898 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:06.898 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":85,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:40:05.251607+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:40:01.941882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-16T01:40:02.564641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:40:06.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.899+0000 7fba641fd640 1 -- 172.21.15.88:0/2506142571 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba4806c2b0 msgr2=0x7fba4806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:06.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.899+0000 7fba641fd640 1 --2- 172.21.15.88:0/2506142571 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba4806c2b0 0x7fba4806e770 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fba54009bc0 tx=0x7fba54008040 comp rx=0 tx=0).stop 2024-09-16T01:40:06.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.899+0000 7fba641fd640 1 -- 172.21.15.88:0/2506142571 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba5c1f8760 msgr2=0x7fba5c15b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:06.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.899+0000 7fba641fd640 1 --2- 172.21.15.88:0/2506142571 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba5c1f8760 0x7fba5c15b7c0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fba5c0cd440 tx=0x7fba4c00f580 comp rx=0 tx=0).stop 2024-09-16T01:40:06.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.899+0000 7fba641fd640 1 -- 172.21.15.88:0/2506142571 shutdown_connections 2024-09-16T01:40:06.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.899+0000 7fba641fd640 1 --2- 172.21.15.88:0/2506142571 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba4806c2b0 0x7fba4806e770 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:06.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.899+0000 7fba641fd640 1 --2- 172.21.15.88:0/2506142571 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fba5c1f8760 0x7fba5c15b7c0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:06.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.899+0000 7fba641fd640 1 --2- 172.21.15.88:0/2506142571 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba5c1f7da0 0x7fba5c1f8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:06.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.899+0000 7fba641fd640 1 --2- 172.21.15.88:0/2506142571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba44004800 0x7fba5c14e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:06.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.899+0000 7fba641fd640 1 -- 172.21.15.88:0/2506142571 >> 172.21.15.88:0/2506142571 conn(0x7fba5c06bc10 msgr2=0x7fba5c0af120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:06.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.899+0000 7fba641fd640 1 -- 172.21.15.88:0/2506142571 shutdown_connections 2024-09-16T01:40:06.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:06.899+0000 7fba641fd640 1 -- 172.21.15.88:0/2506142571 wait complete. 2024-09-16T01:40:06.911 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T01:40:06.911 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:40:07.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.008+0000 7fc1737e4640 1 Processor -- start 2024-09-16T01:40:07.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.008+0000 7fc1737e4640 1 -- start start 2024-09-16T01:40:07.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.009+0000 7fc1737e4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc16c0b4920 0x7fc16c15b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:07.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.009+0000 7fc1737e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc16c0b4d00 0x7fc16c0b5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:07.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.009+0000 7fc1737e4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc16c0ce4f0 con 0x7fc16c0cd780 2024-09-16T01:40:07.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.009+0000 7fc1737e4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc16c0cf100 con 0x7fc16c0b4920 2024-09-16T01:40:07.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.009+0000 7fc1737e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc16c0ce7d0 con 0x7fc16c0b4d00 2024-09-16T01:40:07.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.009+0000 7fc171580640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc16c0b4920 0x7fc16c15b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:07.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.009+0000 7fc171580640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc16c0b4920 0x7fc16c15b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60046/0 (socket says 172.21.15.88:60046) 2024-09-16T01:40:07.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.009+0000 7fc171580640 1 -- 172.21.15.88:0/1081190646 learned_addr learned my addr 172.21.15.88:0/1081190646 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:07.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.009+0000 7fc171580640 1 -- 172.21.15.88:0/1081190646 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc16c0b4d00 msgr2=0x7fc16c0b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:07.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.009+0000 7fc171580640 1 --2- 172.21.15.88:0/1081190646 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc16c0b4d00 0x7fc16c0b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.010+0000 7fc171580640 1 -- 172.21.15.88:0/1081190646 >> v1:172.21.15.154:6789/0 conn(0x7fc16c0cd780 legacy=0x7fc16c0cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:07.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.010+0000 7fc171580640 1 -- 172.21.15.88:0/1081190646 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc16c0ceab0 con 0x7fc16c0b4920 2024-09-16T01:40:07.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.010+0000 7fc171580640 1 --2- 172.21.15.88:0/1081190646 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc16c0b4920 0x7fc16c15b7d0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fc168009870 tx=0x7fc16802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58270b20f5fc7b8f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:07.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.010+0000 7fc163fff640 1 -- 172.21.15.88:0/1081190646 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc168037070 con 0x7fc16c0b4920 2024-09-16T01:40:07.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.010+0000 7fc163fff640 1 -- 172.21.15.88:0/1081190646 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc168032440 con 0x7fc16c0b4920 2024-09-16T01:40:07.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.010+0000 7fc163fff640 1 -- 172.21.15.88:0/1081190646 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc168032810 con 0x7fc16c0b4920 2024-09-16T01:40:07.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.011+0000 7fc1737e4640 1 -- 172.21.15.88:0/1081190646 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc16c0b4920 msgr2=0x7fc16c15b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:07.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.011+0000 7fc1737e4640 1 --2- 172.21.15.88:0/1081190646 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc16c0b4920 0x7fc16c15b7d0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fc168009870 tx=0x7fc16802a210 comp rx=0 tx=0).stop 2024-09-16T01:40:07.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.011+0000 7fc1737e4640 1 -- 172.21.15.88:0/1081190646 shutdown_connections 2024-09-16T01:40:07.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.011+0000 7fc1737e4640 1 --2- 172.21.15.88:0/1081190646 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc16c0b4d00 0x7fc16c0b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.011+0000 7fc1737e4640 1 --2- 172.21.15.88:0/1081190646 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc16c0b4920 0x7fc16c15b7d0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.011+0000 7fc1737e4640 1 -- 172.21.15.88:0/1081190646 >> 172.21.15.88:0/1081190646 conn(0x7fc16c06bc10 msgr2=0x7fc16c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:07.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.011+0000 7fc1737e4640 1 -- 172.21.15.88:0/1081190646 shutdown_connections 2024-09-16T01:40:07.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.011+0000 7fc1737e4640 1 -- 172.21.15.88:0/1081190646 wait complete. 2024-09-16T01:40:07.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.012+0000 7fc1737e4640 1 Processor -- start 2024-09-16T01:40:07.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.012+0000 7fc1737e4640 1 -- start start 2024-09-16T01:40:07.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.013+0000 7fc1737e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc16c0b4920 0x7fc16c1548b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:07.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.013+0000 7fc1737e4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc16c0b4d00 0x7fc16c140ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:07.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.013+0000 7fc1737e4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc16c0cd780 0x7fc16c141880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:07.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.013+0000 7fc1737e4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc16c0b5680 con 0x7fc16c0cd780 2024-09-16T01:40:07.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.013+0000 7fc1737e4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc16c15bd10 con 0x7fc16c0b4d00 2024-09-16T01:40:07.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.013+0000 7fc1737e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc16c0cf590 con 0x7fc16c0b4920 2024-09-16T01:40:07.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.013+0000 7fc171d81640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc16c0cd780 0x7fc16c141880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:07.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.013+0000 7fc171d81640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc16c0cd780 0x7fc16c141880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60054/0 (socket says 172.21.15.88:60054) 2024-09-16T01:40:07.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.013+0000 7fc171d81640 1 -- 172.21.15.88:0/3965568229 learned_addr learned my addr 172.21.15.88:0/3965568229 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:07.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.013+0000 7fc171d81640 1 -- 172.21.15.88:0/3965568229 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc16c0b4920 msgr2=0x7fc16c1548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:07.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.013+0000 7fc171d81640 1 --2- 172.21.15.88:0/3965568229 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc16c0b4920 0x7fc16c1548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.013+0000 7fc171d81640 1 -- 172.21.15.88:0/3965568229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc16c0b4d00 msgr2=0x7fc16c140ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:07.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.013+0000 7fc171d81640 1 --2- 172.21.15.88:0/3965568229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc16c0b4d00 0x7fc16c140ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.013+0000 7fc171d81640 1 -- 172.21.15.88:0/3965568229 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc16c157c90 con 0x7fc16c0cd780 2024-09-16T01:40:07.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.013+0000 7fc171d81640 1 --2- 172.21.15.88:0/3965568229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc16c0cd780 0x7fc16c141880 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fc16400b9e0 tx=0x7fc16400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:07.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.014+0000 7fc161ffb640 1 -- 172.21.15.88:0/3965568229 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc164019070 con 0x7fc16c0cd780 2024-09-16T01:40:07.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.014+0000 7fc161ffb640 1 -- 172.21.15.88:0/3965568229 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc16400caa0 con 0x7fc16c0cd780 2024-09-16T01:40:07.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.014+0000 7fc1737e4640 1 -- 172.21.15.88:0/3965568229 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc16c157f70 con 0x7fc16c0cd780 2024-09-16T01:40:07.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.014+0000 7fc161ffb640 1 -- 172.21.15.88:0/3965568229 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc164014440 con 0x7fc16c0cd780 2024-09-16T01:40:07.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.014+0000 7fc1737e4640 1 -- 172.21.15.88:0/3965568229 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc16c158530 con 0x7fc16c0cd780 2024-09-16T01:40:07.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.015+0000 7fc1737e4640 1 -- 172.21.15.88:0/3965568229 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc16c15b7d0 con 0x7fc16c0cd780 2024-09-16T01:40:07.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.015+0000 7fc161ffb640 1 -- 172.21.15.88:0/3965568229 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc16401c070 con 0x7fc16c0cd780 2024-09-16T01:40:07.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.016+0000 7fc161ffb640 1 --2- 172.21.15.88:0/3965568229 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc15006c2b0 0x7fc15006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:07.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.016+0000 7fc161ffb640 1 -- 172.21.15.88:0/3965568229 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7fc16408b0b0 con 0x7fc16c0cd780 2024-09-16T01:40:07.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.016+0000 7fc171580640 1 --2- 172.21.15.88:0/3965568229 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc15006c2b0 0x7fc15006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:07.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.016+0000 7fc171580640 1 --2- 172.21.15.88:0/3965568229 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc15006c2b0 0x7fc15006e770 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc168034000 tx=0x7fc16802c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:07.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.020+0000 7fc161ffb640 1 -- 172.21.15.88:0/3965568229 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc164059820 con 0x7fc16c0cd780 2024-09-16T01:40:07.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.181+0000 7fc1737e4640 1 -- 172.21.15.88:0/3965568229 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc16c072620 con 0x7fc15006c2b0 2024-09-16T01:40:07.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.187+0000 7fc161ffb640 1 -- 172.21.15.88:0/3965568229 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252041 (secure 0 0 0) 0x7fc16c072620 con 0x7fc15006c2b0 2024-09-16T01:40:07.191 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:07.193 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:40:07.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.190+0000 7fc1737e4640 1 -- 172.21.15.88:0/3965568229 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc15006c2b0 msgr2=0x7fc15006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:07.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.190+0000 7fc1737e4640 1 --2- 172.21.15.88:0/3965568229 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc15006c2b0 0x7fc15006e770 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc168034000 tx=0x7fc16802c000 comp rx=0 tx=0).stop 2024-09-16T01:40:07.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.190+0000 7fc1737e4640 1 -- 172.21.15.88:0/3965568229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc16c0cd780 msgr2=0x7fc16c141880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:07.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.190+0000 7fc1737e4640 1 --2- 172.21.15.88:0/3965568229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc16c0cd780 0x7fc16c141880 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fc16400b9e0 tx=0x7fc16400bea0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.190+0000 7fc1737e4640 1 -- 172.21.15.88:0/3965568229 shutdown_connections 2024-09-16T01:40:07.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.190+0000 7fc1737e4640 1 --2- 172.21.15.88:0/3965568229 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc15006c2b0 0x7fc15006e770 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.190+0000 7fc1737e4640 1 --2- 172.21.15.88:0/3965568229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc16c0cd780 0x7fc16c141880 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.190+0000 7fc1737e4640 1 --2- 172.21.15.88:0/3965568229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc16c0b4d00 0x7fc16c140ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.190+0000 7fc1737e4640 1 --2- 172.21.15.88:0/3965568229 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc16c0b4920 0x7fc16c1548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.190+0000 7fc1737e4640 1 -- 172.21.15.88:0/3965568229 >> 172.21.15.88:0/3965568229 conn(0x7fc16c06bc10 msgr2=0x7fc16c074c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:07.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.190+0000 7fc1737e4640 1 -- 172.21.15.88:0/3965568229 shutdown_connections 2024-09-16T01:40:07.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.190+0000 7fc1737e4640 1 -- 172.21.15.88:0/3965568229 wait complete. 2024-09-16T01:40:07.202 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-09-16T01:40:06.593899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":101,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":242172,"kb_used_data":9280,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754732548,"statfs":{"total":773094113280,"available":772846129152,"internally_reserved":0,"allocated":9502720,"data_stored":4706332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.348667"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256764+0000","last_change":"2024-09-16T01:40:04.271300+0000","last_active":"2024-09-16T01:40:05.256764+0000","last_peered":"2024-09-16T01:40:05.256764+0000","last_clean":"2024-09-16T01:40:05.256764+0000","last_became_active":"2024-09-16T01:40:04.270801+0000","last_became_peered":"2024-09-16T01:40:04.270801+0000","last_unstale":"2024-09-16T01:40:05.256764+0000","last_undegraded":"2024-09-16T01:40:05.256764+0000","last_fullsized":"2024-09-16T01:40:05.256764+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:39.441878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.281545+0000","last_change":"2024-09-16T01:40:04.275657+0000","last_active":"2024-09-16T01:40:04.281545+0000","last_peered":"2024-09-16T01:40:04.281545+0000","last_clean":"2024-09-16T01:40:04.281545+0000","last_became_active":"2024-09-16T01:40:04.274280+0000","last_became_peered":"2024-09-16T01:40:04.274280+0000","last_unstale":"2024-09-16T01:40:04.281545+0000","last_undegraded":"2024-09-16T01:40:04.281545+0000","last_fullsized":"2024-09-16T01:40:04.281545+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:33:39.723365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256322+0000","last_change":"2024-09-16T01:40:04.259552+0000","last_active":"2024-09-16T01:40:05.256322+0000","last_peered":"2024-09-16T01:40:05.256322+0000","last_clean":"2024-09-16T01:40:05.256322+0000","last_became_active":"2024-09-16T01:40:04.259252+0000","last_became_peered":"2024-09-16T01:40:04.259252+0000","last_unstale":"2024-09-16T01:40:05.256322+0000","last_undegraded":"2024-09-16T01:40:05.256322+0000","last_fullsized":"2024-09-16T01:40:05.256322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:53.219368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.289678+0000","last_change":"2024-09-16T01:40:04.284529+0000","last_active":"2024-09-16T01:40:04.289678+0000","last_peered":"2024-09-16T01:40:04.289678+0000","last_clean":"2024-09-16T01:40:04.289678+0000","last_became_active":"2024-09-16T01:40:04.284280+0000","last_became_peered":"2024-09-16T01:40:04.284280+0000","last_unstale":"2024-09-16T01:40:04.289678+0000","last_undegraded":"2024-09-16T01:40:04.289678+0000","last_fullsized":"2024-09-16T01:40:04.289678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:49:59.001632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.279416+0000","last_change":"2024-09-16T01:40:04.273647+0000","last_active":"2024-09-16T01:40:04.279416+0000","last_peered":"2024-09-16T01:40:04.279416+0000","last_clean":"2024-09-16T01:40:04.279416+0000","last_became_active":"2024-09-16T01:40:04.273362+0000","last_became_peered":"2024-09-16T01:40:04.273362+0000","last_unstale":"2024-09-16T01:40:04.279416+0000","last_undegraded":"2024-09-16T01:40:04.279416+0000","last_fullsized":"2024-09-16T01:40:04.279416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:16:01.400396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.286268+0000","last_change":"2024-09-16T01:40:04.285106+0000","last_active":"2024-09-16T01:40:04.286268+0000","last_peered":"2024-09-16T01:40:04.286268+0000","last_clean":"2024-09-16T01:40:04.286268+0000","last_became_active":"2024-09-16T01:40:04.284804+0000","last_became_peered":"2024-09-16T01:40:04.284804+0000","last_unstale":"2024-09-16T01:40:04.286268+0000","last_undegraded":"2024-09-16T01:40:04.286268+0000","last_fullsized":"2024-09-16T01:40:04.286268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:06:54.718293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.287590+0000","last_change":"2024-09-16T01:40:04.282323+0000","last_active":"2024-09-16T01:40:04.287590+0000","last_peered":"2024-09-16T01:40:04.287590+0000","last_clean":"2024-09-16T01:40:04.287590+0000","last_became_active":"2024-09-16T01:40:04.282052+0000","last_became_peered":"2024-09-16T01:40:04.282052+0000","last_unstale":"2024-09-16T01:40:04.287590+0000","last_undegraded":"2024-09-16T01:40:04.287590+0000","last_fullsized":"2024-09-16T01:40:04.287590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:58.420086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.270398+0000","last_change":"2024-09-16T01:40:04.268240+0000","last_active":"2024-09-16T01:40:04.270398+0000","last_peered":"2024-09-16T01:40:04.270398+0000","last_clean":"2024-09-16T01:40:04.270398+0000","last_became_active":"2024-09-16T01:40:04.267343+0000","last_became_peered":"2024-09-16T01:40:04.267343+0000","last_unstale":"2024-09-16T01:40:04.270398+0000","last_undegraded":"2024-09-16T01:40:04.270398+0000","last_fullsized":"2024-09-16T01:40:04.270398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:46:06.509496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.281326+0000","last_change":"2024-09-16T01:40:04.280262+0000","last_active":"2024-09-16T01:40:04.281326+0000","last_peered":"2024-09-16T01:40:04.281326+0000","last_clean":"2024-09-16T01:40:04.281326+0000","last_became_active":"2024-09-16T01:40:04.280068+0000","last_became_peered":"2024-09-16T01:40:04.280068+0000","last_unstale":"2024-09-16T01:40:04.281326+0000","last_undegraded":"2024-09-16T01:40:04.281326+0000","last_fullsized":"2024-09-16T01:40:04.281326+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:05:40.988739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256335+0000","last_change":"2024-09-16T01:40:04.269451+0000","last_active":"2024-09-16T01:40:05.256335+0000","last_peered":"2024-09-16T01:40:05.256335+0000","last_clean":"2024-09-16T01:40:05.256335+0000","last_became_active":"2024-09-16T01:40:04.269175+0000","last_became_peered":"2024-09-16T01:40:04.269175+0000","last_unstale":"2024-09-16T01:40:05.256335+0000","last_undegraded":"2024-09-16T01:40:05.256335+0000","last_fullsized":"2024-09-16T01:40:05.256335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:07:13.027761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.271517+0000","last_change":"2024-09-16T01:40:04.265706+0000","last_active":"2024-09-16T01:40:04.271517+0000","last_peered":"2024-09-16T01:40:04.271517+0000","last_clean":"2024-09-16T01:40:04.271517+0000","last_became_active":"2024-09-16T01:40:04.265426+0000","last_became_peered":"2024-09-16T01:40:04.265426+0000","last_unstale":"2024-09-16T01:40:04.271517+0000","last_undegraded":"2024-09-16T01:40:04.271517+0000","last_fullsized":"2024-09-16T01:40:04.271517+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:09.970446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256197+0000","last_change":"2024-09-16T01:40:04.268695+0000","last_active":"2024-09-16T01:40:05.256197+0000","last_peered":"2024-09-16T01:40:05.256197+0000","last_clean":"2024-09-16T01:40:05.256197+0000","last_became_active":"2024-09-16T01:40:04.268433+0000","last_became_peered":"2024-09-16T01:40:04.268433+0000","last_unstale":"2024-09-16T01:40:05.256197+0000","last_undegraded":"2024-09-16T01:40:05.256197+0000","last_fullsized":"2024-09-16T01:40:05.256197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:21:54.216209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.255994+0000","last_change":"2024-09-16T01:40:04.266270+0000","last_active":"2024-09-16T01:40:05.255994+0000","last_peered":"2024-09-16T01:40:05.255994+0000","last_clean":"2024-09-16T01:40:05.255994+0000","last_became_active":"2024-09-16T01:40:04.265944+0000","last_became_peered":"2024-09-16T01:40:04.265944+0000","last_unstale":"2024-09-16T01:40:05.255994+0000","last_undegraded":"2024-09-16T01:40:05.255994+0000","last_fullsized":"2024-09-16T01:40:05.255994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:51:12.899296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.289715+0000","last_change":"2024-09-16T01:40:04.283535+0000","last_active":"2024-09-16T01:40:04.289715+0000","last_peered":"2024-09-16T01:40:04.289715+0000","last_clean":"2024-09-16T01:40:04.289715+0000","last_became_active":"2024-09-16T01:40:04.282926+0000","last_became_peered":"2024-09-16T01:40:04.282926+0000","last_unstale":"2024-09-16T01:40:04.289715+0000","last_undegraded":"2024-09-16T01:40:04.289715+0000","last_fullsized":"2024-09-16T01:40:04.289715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:04.397034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.267235+0000","last_change":"2024-09-16T01:40:04.259774+0000","last_active":"2024-09-16T01:40:04.267235+0000","last_peered":"2024-09-16T01:40:04.267235+0000","last_clean":"2024-09-16T01:40:04.267235+0000","last_became_active":"2024-09-16T01:40:04.259469+0000","last_became_peered":"2024-09-16T01:40:04.259469+0000","last_unstale":"2024-09-16T01:40:04.267235+0000","last_undegraded":"2024-09-16T01:40:04.267235+0000","last_fullsized":"2024-09-16T01:40:04.267235+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:30:35.198807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.279454+0000","last_change":"2024-09-16T01:40:04.274557+0000","last_active":"2024-09-16T01:40:04.279454+0000","last_peered":"2024-09-16T01:40:04.279454+0000","last_clean":"2024-09-16T01:40:04.279454+0000","last_became_active":"2024-09-16T01:40:04.274239+0000","last_became_peered":"2024-09-16T01:40:04.274239+0000","last_unstale":"2024-09-16T01:40:04.279454+0000","last_undegraded":"2024-09-16T01:40:04.279454+0000","last_fullsized":"2024-09-16T01:40:04.279454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:43.843151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.283268+0000","last_change":"2024-09-16T01:40:04.282582+0000","last_active":"2024-09-16T01:40:04.283268+0000","last_peered":"2024-09-16T01:40:04.283268+0000","last_clean":"2024-09-16T01:40:04.283268+0000","last_became_active":"2024-09-16T01:40:04.282313+0000","last_became_peered":"2024-09-16T01:40:04.282313+0000","last_unstale":"2024-09-16T01:40:04.283268+0000","last_undegraded":"2024-09-16T01:40:04.283268+0000","last_fullsized":"2024-09-16T01:40:04.283268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:31:59.317167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.289689+0000","last_change":"2024-09-16T01:40:04.283813+0000","last_active":"2024-09-16T01:40:04.289689+0000","last_peered":"2024-09-16T01:40:04.289689+0000","last_clean":"2024-09-16T01:40:04.289689+0000","last_became_active":"2024-09-16T01:40:04.283442+0000","last_became_peered":"2024-09-16T01:40:04.283442+0000","last_unstale":"2024-09-16T01:40:04.289689+0000","last_undegraded":"2024-09-16T01:40:04.289689+0000","last_fullsized":"2024-09-16T01:40:04.289689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:31:29.553476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.257402+0000","last_change":"2024-09-16T01:40:04.272557+0000","last_active":"2024-09-16T01:40:05.257402+0000","last_peered":"2024-09-16T01:40:05.257402+0000","last_clean":"2024-09-16T01:40:05.257402+0000","last_became_active":"2024-09-16T01:40:04.271776+0000","last_became_peered":"2024-09-16T01:40:04.271776+0000","last_unstale":"2024-09-16T01:40:05.257402+0000","last_undegraded":"2024-09-16T01:40:05.257402+0000","last_fullsized":"2024-09-16T01:40:05.257402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:38:19.894228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.289809+0000","last_change":"2024-09-16T01:40:04.286776+0000","last_active":"2024-09-16T01:40:04.289809+0000","last_peered":"2024-09-16T01:40:04.289809+0000","last_clean":"2024-09-16T01:40:04.289809+0000","last_became_active":"2024-09-16T01:40:04.286485+0000","last_became_peered":"2024-09-16T01:40:04.286485+0000","last_unstale":"2024-09-16T01:40:04.289809+0000","last_undegraded":"2024-09-16T01:40:04.289809+0000","last_fullsized":"2024-09-16T01:40:04.289809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:45:29.048955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.289848+0000","last_change":"2024-09-16T01:40:04.287172+0000","last_active":"2024-09-16T01:40:04.289848+0000","last_peered":"2024-09-16T01:40:04.289848+0000","last_clean":"2024-09-16T01:40:04.289848+0000","last_became_active":"2024-09-16T01:40:04.286921+0000","last_became_peered":"2024-09-16T01:40:04.286921+0000","last_unstale":"2024-09-16T01:40:04.289848+0000","last_undegraded":"2024-09-16T01:40:04.289848+0000","last_fullsized":"2024-09-16T01:40:04.289848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:05:01.081881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.287882+0000","last_change":"2024-09-16T01:40:04.282655+0000","last_active":"2024-09-16T01:40:04.287882+0000","last_peered":"2024-09-16T01:40:04.287882+0000","last_clean":"2024-09-16T01:40:04.287882+0000","last_became_active":"2024-09-16T01:40:04.282366+0000","last_became_peered":"2024-09-16T01:40:04.282366+0000","last_unstale":"2024-09-16T01:40:04.287882+0000","last_undegraded":"2024-09-16T01:40:04.287882+0000","last_fullsized":"2024-09-16T01:40:04.287882+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:38.198140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.255751+0000","last_change":"2024-09-16T01:40:04.262760+0000","last_active":"2024-09-16T01:40:05.255751+0000","last_peered":"2024-09-16T01:40:05.255751+0000","last_clean":"2024-09-16T01:40:05.255751+0000","last_became_active":"2024-09-16T01:40:04.262385+0000","last_became_peered":"2024-09-16T01:40:04.262385+0000","last_unstale":"2024-09-16T01:40:05.255751+0000","last_undegraded":"2024-09-16T01:40:05.255751+0000","last_fullsized":"2024-09-16T01:40:05.255751+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:16:49.578781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.289794+0000","last_change":"2024-09-16T01:40:04.286747+0000","last_active":"2024-09-16T01:40:04.289794+0000","last_peered":"2024-09-16T01:40:04.289794+0000","last_clean":"2024-09-16T01:40:04.289794+0000","last_became_active":"2024-09-16T01:40:04.286520+0000","last_became_peered":"2024-09-16T01:40:04.286520+0000","last_unstale":"2024-09-16T01:40:04.289794+0000","last_undegraded":"2024-09-16T01:40:04.289794+0000","last_fullsized":"2024-09-16T01:40:04.289794+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:55:59.275712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.279010+0000","last_change":"2024-09-16T01:40:04.274554+0000","last_active":"2024-09-16T01:40:04.279010+0000","last_peered":"2024-09-16T01:40:04.279010+0000","last_clean":"2024-09-16T01:40:04.279010+0000","last_became_active":"2024-09-16T01:40:04.274139+0000","last_became_peered":"2024-09-16T01:40:04.274139+0000","last_unstale":"2024-09-16T01:40:04.279010+0000","last_undegraded":"2024-09-16T01:40:04.279010+0000","last_fullsized":"2024-09-16T01:40:04.279010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:16:03.129435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.255775+0000","last_change":"2024-09-16T01:40:04.265405+0000","last_active":"2024-09-16T01:40:05.255775+0000","last_peered":"2024-09-16T01:40:05.255775+0000","last_clean":"2024-09-16T01:40:05.255775+0000","last_became_active":"2024-09-16T01:40:04.265114+0000","last_became_peered":"2024-09-16T01:40:04.265114+0000","last_unstale":"2024-09-16T01:40:05.255775+0000","last_undegraded":"2024-09-16T01:40:05.255775+0000","last_fullsized":"2024-09-16T01:40:05.255775+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:55:25.161381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.275180+0000","last_change":"2024-09-16T01:40:04.268566+0000","last_active":"2024-09-16T01:40:04.275180+0000","last_peered":"2024-09-16T01:40:04.275180+0000","last_clean":"2024-09-16T01:40:04.275180+0000","last_became_active":"2024-09-16T01:40:04.268312+0000","last_became_peered":"2024-09-16T01:40:04.268312+0000","last_unstale":"2024-09-16T01:40:04.275180+0000","last_undegraded":"2024-09-16T01:40:04.275180+0000","last_fullsized":"2024-09-16T01:40:04.275180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:49.395185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256354+0000","last_change":"2024-09-16T01:40:04.271058+0000","last_active":"2024-09-16T01:40:05.256354+0000","last_peered":"2024-09-16T01:40:05.256354+0000","last_clean":"2024-09-16T01:40:05.256354+0000","last_became_active":"2024-09-16T01:40:04.270783+0000","last_became_peered":"2024-09-16T01:40:04.270783+0000","last_unstale":"2024-09-16T01:40:05.256354+0000","last_undegraded":"2024-09-16T01:40:05.256354+0000","last_fullsized":"2024-09-16T01:40:05.256354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:30:35.198807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.275580+0000","last_change":"2024-09-16T01:40:04.269738+0000","last_active":"2024-09-16T01:40:04.275580+0000","last_peered":"2024-09-16T01:40:04.275580+0000","last_clean":"2024-09-16T01:40:04.275580+0000","last_became_active":"2024-09-16T01:40:04.269475+0000","last_became_peered":"2024-09-16T01:40:04.269475+0000","last_unstale":"2024-09-16T01:40:04.275580+0000","last_undegraded":"2024-09-16T01:40:04.275580+0000","last_fullsized":"2024-09-16T01:40:04.275580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:22:03.250147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.289596+0000","last_change":"2024-09-16T01:40:04.287758+0000","last_active":"2024-09-16T01:40:04.289596+0000","last_peered":"2024-09-16T01:40:04.289596+0000","last_clean":"2024-09-16T01:40:04.289596+0000","last_became_active":"2024-09-16T01:40:04.287499+0000","last_became_peered":"2024-09-16T01:40:04.287499+0000","last_unstale":"2024-09-16T01:40:04.289596+0000","last_undegraded":"2024-09-16T01:40:04.289596+0000","last_fullsized":"2024-09-16T01:40:04.289596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:02:23.790489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.267851+0000","last_change":"2024-09-16T01:40:04.263004+0000","last_active":"2024-09-16T01:40:04.267851+0000","last_peered":"2024-09-16T01:40:04.267851+0000","last_clean":"2024-09-16T01:40:04.267851+0000","last_became_active":"2024-09-16T01:40:04.262684+0000","last_became_peered":"2024-09-16T01:40:04.262684+0000","last_unstale":"2024-09-16T01:40:04.267851+0000","last_undegraded":"2024-09-16T01:40:04.267851+0000","last_fullsized":"2024-09-16T01:40:04.267851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:25:07.083159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256623+0000","last_change":"2024-09-16T01:40:04.266455+0000","last_active":"2024-09-16T01:40:05.256623+0000","last_peered":"2024-09-16T01:40:05.256623+0000","last_clean":"2024-09-16T01:40:05.256623+0000","last_became_active":"2024-09-16T01:40:04.266151+0000","last_became_peered":"2024-09-16T01:40:04.266151+0000","last_unstale":"2024-09-16T01:40:05.256623+0000","last_undegraded":"2024-09-16T01:40:05.256623+0000","last_fullsized":"2024-09-16T01:40:05.256623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:11:15.958789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.279402+0000","last_change":"2024-09-16T01:40:04.273836+0000","last_active":"2024-09-16T01:40:04.279402+0000","last_peered":"2024-09-16T01:40:04.279402+0000","last_clean":"2024-09-16T01:40:04.279402+0000","last_became_active":"2024-09-16T01:40:04.273476+0000","last_became_peered":"2024-09-16T01:40:04.273476+0000","last_unstale":"2024-09-16T01:40:04.279402+0000","last_undegraded":"2024-09-16T01:40:04.279402+0000","last_fullsized":"2024-09-16T01:40:04.279402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:12.686302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"84'4","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256872+0000","last_change":"2024-09-16T01:40:03.269292+0000","last_active":"2024-09-16T01:40:05.256872+0000","last_peered":"2024-09-16T01:40:05.256872+0000","last_clean":"2024-09-16T01:40:05.256872+0000","last_became_active":"2024-09-16T01:40:03.269039+0000","last_became_peered":"2024-09-16T01:40:03.269039+0000","last_unstale":"2024-09-16T01:40:05.256872+0000","last_undegraded":"2024-09-16T01:40:05.256872+0000","last_fullsized":"2024-09-16T01:40:05.256872+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:26:31.689093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256916+0000","last_change":"2024-09-16T01:40:04.261753+0000","last_active":"2024-09-16T01:40:05.256916+0000","last_peered":"2024-09-16T01:40:05.256916+0000","last_clean":"2024-09-16T01:40:05.256916+0000","last_became_active":"2024-09-16T01:40:04.261464+0000","last_became_peered":"2024-09-16T01:40:04.261464+0000","last_unstale":"2024-09-16T01:40:05.256916+0000","last_undegraded":"2024-09-16T01:40:05.256916+0000","last_fullsized":"2024-09-16T01:40:05.256916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:46:24.061573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"84'1","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.282759+0000","last_change":"2024-09-16T01:40:03.263453+0000","last_active":"2024-09-16T01:40:04.282759+0000","last_peered":"2024-09-16T01:40:04.282759+0000","last_clean":"2024-09-16T01:40:04.282759+0000","last_became_active":"2024-09-16T01:40:03.262836+0000","last_became_peered":"2024-09-16T01:40:03.262836+0000","last_unstale":"2024-09-16T01:40:04.282759+0000","last_undegraded":"2024-09-16T01:40:04.282759+0000","last_fullsized":"2024-09-16T01:40:04.282759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:26:31.689093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256493+0000","last_change":"2024-09-16T01:40:04.266448+0000","last_active":"2024-09-16T01:40:05.256493+0000","last_peered":"2024-09-16T01:40:05.256493+0000","last_clean":"2024-09-16T01:40:05.256493+0000","last_became_active":"2024-09-16T01:40:04.266183+0000","last_became_peered":"2024-09-16T01:40:04.266183+0000","last_unstale":"2024-09-16T01:40:05.256493+0000","last_undegraded":"2024-09-16T01:40:05.256493+0000","last_fullsized":"2024-09-16T01:40:05.256493+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:52:40.505962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"84'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.257297+0000","last_change":"2024-09-16T01:40:03.264515+0000","last_active":"2024-09-16T01:40:05.257297+0000","last_peered":"2024-09-16T01:40:05.257297+0000","last_clean":"2024-09-16T01:40:05.257297+0000","last_became_active":"2024-09-16T01:40:03.264240+0000","last_became_peered":"2024-09-16T01:40:03.264240+0000","last_unstale":"2024-09-16T01:40:05.257297+0000","last_undegraded":"2024-09-16T01:40:05.257297+0000","last_fullsized":"2024-09-16T01:40:05.257297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:48.392893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256374+0000","last_change":"2024-09-16T01:40:04.287012+0000","last_active":"2024-09-16T01:40:05.256374+0000","last_peered":"2024-09-16T01:40:05.256374+0000","last_clean":"2024-09-16T01:40:05.256374+0000","last_became_active":"2024-09-16T01:40:04.286726+0000","last_became_peered":"2024-09-16T01:40:04.286726+0000","last_unstale":"2024-09-16T01:40:05.256374+0000","last_undegraded":"2024-09-16T01:40:05.256374+0000","last_fullsized":"2024-09-16T01:40:05.256374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:11:22.300784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"84'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.257267+0000","last_change":"2024-09-16T01:40:03.264709+0000","last_active":"2024-09-16T01:40:05.257267+0000","last_peered":"2024-09-16T01:40:05.257267+0000","last_clean":"2024-09-16T01:40:05.257267+0000","last_became_active":"2024-09-16T01:40:03.264193+0000","last_became_peered":"2024-09-16T01:40:03.264193+0000","last_unstale":"2024-09-16T01:40:05.257267+0000","last_undegraded":"2024-09-16T01:40:05.257267+0000","last_fullsized":"2024-09-16T01:40:05.257267+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:16:00.398104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.288336+0000","last_change":"2024-09-16T01:40:04.286614+0000","last_active":"2024-09-16T01:40:04.288336+0000","last_peered":"2024-09-16T01:40:04.288336+0000","last_clean":"2024-09-16T01:40:04.288336+0000","last_became_active":"2024-09-16T01:40:04.286362+0000","last_became_peered":"2024-09-16T01:40:04.286362+0000","last_unstale":"2024-09-16T01:40:04.288336+0000","last_undegraded":"2024-09-16T01:40:04.288336+0000","last_fullsized":"2024-09-16T01:40:04.288336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:53:13.515186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"84'1","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.284670+0000","last_change":"2024-09-16T01:40:03.264527+0000","last_active":"2024-09-16T01:40:04.284670+0000","last_peered":"2024-09-16T01:40:04.284670+0000","last_clean":"2024-09-16T01:40:04.284670+0000","last_became_active":"2024-09-16T01:40:03.264233+0000","last_became_peered":"2024-09-16T01:40:03.264233+0000","last_unstale":"2024-09-16T01:40:04.284670+0000","last_undegraded":"2024-09-16T01:40:04.284670+0000","last_fullsized":"2024-09-16T01:40:04.284670+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:47:23.618181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.286205+0000","last_change":"2024-09-16T01:40:04.284494+0000","last_active":"2024-09-16T01:40:04.286205+0000","last_peered":"2024-09-16T01:40:04.286205+0000","last_clean":"2024-09-16T01:40:04.286205+0000","last_became_active":"2024-09-16T01:40:04.284222+0000","last_became_peered":"2024-09-16T01:40:04.284222+0000","last_unstale":"2024-09-16T01:40:04.286205+0000","last_undegraded":"2024-09-16T01:40:04.286205+0000","last_fullsized":"2024-09-16T01:40:04.286205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:02:29.794001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"84'1","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.282687+0000","last_change":"2024-09-16T01:40:03.263711+0000","last_active":"2024-09-16T01:40:04.282687+0000","last_peered":"2024-09-16T01:40:04.282687+0000","last_clean":"2024-09-16T01:40:04.282687+0000","last_became_active":"2024-09-16T01:40:03.263408+0000","last_became_peered":"2024-09-16T01:40:03.263408+0000","last_unstale":"2024-09-16T01:40:04.282687+0000","last_undegraded":"2024-09-16T01:40:04.282687+0000","last_fullsized":"2024-09-16T01:40:04.282687+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:47:16.451717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256073+0000","last_change":"2024-09-16T01:40:04.263307+0000","last_active":"2024-09-16T01:40:05.256073+0000","last_peered":"2024-09-16T01:40:05.256073+0000","last_clean":"2024-09-16T01:40:05.256073+0000","last_became_active":"2024-09-16T01:40:04.263019+0000","last_became_peered":"2024-09-16T01:40:04.263019+0000","last_unstale":"2024-09-16T01:40:05.256073+0000","last_undegraded":"2024-09-16T01:40:05.256073+0000","last_fullsized":"2024-09-16T01:40:05.256073+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:31:13.781556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.257058+0000","last_change":"2024-09-16T01:40:03.263700+0000","last_active":"2024-09-16T01:40:05.257058+0000","last_peered":"2024-09-16T01:40:05.257058+0000","last_clean":"2024-09-16T01:40:05.257058+0000","last_became_active":"2024-09-16T01:40:03.263397+0000","last_became_peered":"2024-09-16T01:40:03.263397+0000","last_unstale":"2024-09-16T01:40:05.257058+0000","last_undegraded":"2024-09-16T01:40:05.257058+0000","last_fullsized":"2024-09-16T01:40:05.257058+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:42.840859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.267257+0000","last_change":"2024-09-16T01:40:04.265941+0000","last_active":"2024-09-16T01:40:04.267257+0000","last_peered":"2024-09-16T01:40:04.267257+0000","last_clean":"2024-09-16T01:40:04.267257+0000","last_became_active":"2024-09-16T01:40:04.265681+0000","last_became_peered":"2024-09-16T01:40:04.265681+0000","last_unstale":"2024-09-16T01:40:04.267257+0000","last_undegraded":"2024-09-16T01:40:04.267257+0000","last_fullsized":"2024-09-16T01:40:04.267257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:52:40.505962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.263755+0000","last_change":"2024-09-16T01:40:03.268669+0000","last_active":"2024-09-16T01:40:04.263755+0000","last_peered":"2024-09-16T01:40:04.263755+0000","last_clean":"2024-09-16T01:40:04.263755+0000","last_became_active":"2024-09-16T01:40:03.268192+0000","last_became_peered":"2024-09-16T01:40:03.268192+0000","last_unstale":"2024-09-16T01:40:04.263755+0000","last_undegraded":"2024-09-16T01:40:04.263755+0000","last_fullsized":"2024-09-16T01:40:04.263755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:59:36.722101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.268032+0000","last_change":"2024-09-16T01:40:04.266898+0000","last_active":"2024-09-16T01:40:04.268032+0000","last_peered":"2024-09-16T01:40:04.268032+0000","last_clean":"2024-09-16T01:40:04.268032+0000","last_became_active":"2024-09-16T01:40:04.266584+0000","last_became_peered":"2024-09-16T01:40:04.266584+0000","last_unstale":"2024-09-16T01:40:04.268032+0000","last_undegraded":"2024-09-16T01:40:04.268032+0000","last_fullsized":"2024-09-16T01:40:04.268032+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:34:02.972536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256968+0000","last_change":"2024-09-16T01:40:03.264902+0000","last_active":"2024-09-16T01:40:05.256968+0000","last_peered":"2024-09-16T01:40:05.256968+0000","last_clean":"2024-09-16T01:40:05.256968+0000","last_became_active":"2024-09-16T01:40:03.264485+0000","last_became_peered":"2024-09-16T01:40:03.264485+0000","last_unstale":"2024-09-16T01:40:05.256968+0000","last_undegraded":"2024-09-16T01:40:05.256968+0000","last_fullsized":"2024-09-16T01:40:05.256968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:57.117123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.284295+0000","last_change":"2024-09-16T01:40:04.283561+0000","last_active":"2024-09-16T01:40:04.284295+0000","last_peered":"2024-09-16T01:40:04.284295+0000","last_clean":"2024-09-16T01:40:04.284295+0000","last_became_active":"2024-09-16T01:40:04.283238+0000","last_became_peered":"2024-09-16T01:40:04.283238+0000","last_unstale":"2024-09-16T01:40:04.284295+0000","last_undegraded":"2024-09-16T01:40:04.284295+0000","last_fullsized":"2024-09-16T01:40:04.284295+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:23:38.464038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256444+0000","last_change":"2024-09-16T01:40:03.268499+0000","last_active":"2024-09-16T01:40:05.256444+0000","last_peered":"2024-09-16T01:40:05.256444+0000","last_clean":"2024-09-16T01:40:05.256444+0000","last_became_active":"2024-09-16T01:40:03.268248+0000","last_became_peered":"2024-09-16T01:40:03.268248+0000","last_unstale":"2024-09-16T01:40:05.256444+0000","last_undegraded":"2024-09-16T01:40:05.256444+0000","last_fullsized":"2024-09-16T01:40:05.256444+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:37.909549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.257409+0000","last_change":"2024-09-16T01:40:04.267674+0000","last_active":"2024-09-16T01:40:05.257409+0000","last_peered":"2024-09-16T01:40:05.257409+0000","last_clean":"2024-09-16T01:40:05.257409+0000","last_became_active":"2024-09-16T01:40:04.267407+0000","last_became_peered":"2024-09-16T01:40:04.267407+0000","last_unstale":"2024-09-16T01:40:05.257409+0000","last_undegraded":"2024-09-16T01:40:05.257409+0000","last_fullsized":"2024-09-16T01:40:05.257409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:14:45.631031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"84'3","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.255743+0000","last_change":"2024-09-16T01:40:03.271207+0000","last_active":"2024-09-16T01:40:05.255743+0000","last_peered":"2024-09-16T01:40:05.255743+0000","last_clean":"2024-09-16T01:40:05.255743+0000","last_became_active":"2024-09-16T01:40:03.270865+0000","last_became_peered":"2024-09-16T01:40:03.270865+0000","last_unstale":"2024-09-16T01:40:05.255743+0000","last_undegraded":"2024-09-16T01:40:05.255743+0000","last_fullsized":"2024-09-16T01:40:05.255743+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:02:22.041889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.286147+0000","last_change":"2024-09-16T01:40:04.284154+0000","last_active":"2024-09-16T01:40:04.286147+0000","last_peered":"2024-09-16T01:40:04.286147+0000","last_clean":"2024-09-16T01:40:04.286147+0000","last_became_active":"2024-09-16T01:40:04.283815+0000","last_became_peered":"2024-09-16T01:40:04.283815+0000","last_unstale":"2024-09-16T01:40:04.286147+0000","last_undegraded":"2024-09-16T01:40:04.286147+0000","last_fullsized":"2024-09-16T01:40:04.286147+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:32.141921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"84'6","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.287490+0000","last_change":"2024-09-16T01:40:03.262909+0000","last_active":"2024-09-16T01:40:04.287490+0000","last_peered":"2024-09-16T01:40:04.287490+0000","last_clean":"2024-09-16T01:40:04.287490+0000","last_became_active":"2024-09-16T01:40:03.262600+0000","last_became_peered":"2024-09-16T01:40:03.262600+0000","last_unstale":"2024-09-16T01:40:04.287490+0000","last_undegraded":"2024-09-16T01:40:04.287490+0000","last_fullsized":"2024-09-16T01:40:04.287490+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:21:07.670452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.257079+0000","last_change":"2024-09-16T01:40:04.270582+0000","last_active":"2024-09-16T01:40:05.257079+0000","last_peered":"2024-09-16T01:40:05.257079+0000","last_clean":"2024-09-16T01:40:05.257079+0000","last_became_active":"2024-09-16T01:40:04.270277+0000","last_became_peered":"2024-09-16T01:40:04.270277+0000","last_unstale":"2024-09-16T01:40:05.257079+0000","last_undegraded":"2024-09-16T01:40:05.257079+0000","last_fullsized":"2024-09-16T01:40:05.257079+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:56:05.602743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"84'2","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.283409+0000","last_change":"2024-09-16T01:40:03.262692+0000","last_active":"2024-09-16T01:40:04.283409+0000","last_peered":"2024-09-16T01:40:04.283409+0000","last_clean":"2024-09-16T01:40:04.283409+0000","last_became_active":"2024-09-16T01:40:03.262426+0000","last_became_peered":"2024-09-16T01:40:03.262426+0000","last_unstale":"2024-09-16T01:40:04.283409+0000","last_undegraded":"2024-09-16T01:40:04.283409+0000","last_fullsized":"2024-09-16T01:40:04.283409+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:05:00.079589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256140+0000","last_change":"2024-09-16T01:40:04.269975+0000","last_active":"2024-09-16T01:40:05.256140+0000","last_peered":"2024-09-16T01:40:05.256140+0000","last_clean":"2024-09-16T01:40:05.256140+0000","last_became_active":"2024-09-16T01:40:04.269079+0000","last_became_peered":"2024-09-16T01:40:04.269079+0000","last_unstale":"2024-09-16T01:40:05.256140+0000","last_undegraded":"2024-09-16T01:40:05.256140+0000","last_fullsized":"2024-09-16T01:40:05.256140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:16.399868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.256500+0000","last_change":"2024-09-16T01:40:03.267797+0000","last_active":"2024-09-16T01:40:04.256500+0000","last_peered":"2024-09-16T01:40:04.256500+0000","last_clean":"2024-09-16T01:40:04.256500+0000","last_became_active":"2024-09-16T01:40:03.267546+0000","last_became_peered":"2024-09-16T01:40:03.267546+0000","last_unstale":"2024-09-16T01:40:04.256500+0000","last_undegraded":"2024-09-16T01:40:04.256500+0000","last_fullsized":"2024-09-16T01:40:04.256500+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:07:12.025468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.288216+0000","last_change":"2024-09-16T01:40:04.284411+0000","last_active":"2024-09-16T01:40:04.288216+0000","last_peered":"2024-09-16T01:40:04.288216+0000","last_clean":"2024-09-16T01:40:04.288216+0000","last_became_active":"2024-09-16T01:40:04.284159+0000","last_became_peered":"2024-09-16T01:40:04.284159+0000","last_unstale":"2024-09-16T01:40:04.288216+0000","last_undegraded":"2024-09-16T01:40:04.288216+0000","last_fullsized":"2024-09-16T01:40:04.288216+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:47.244592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.262661+0000","last_change":"2024-09-16T01:40:03.268017+0000","last_active":"2024-09-16T01:40:04.262661+0000","last_peered":"2024-09-16T01:40:04.262661+0000","last_clean":"2024-09-16T01:40:04.262661+0000","last_became_active":"2024-09-16T01:40:03.267743+0000","last_became_peered":"2024-09-16T01:40:03.267743+0000","last_unstale":"2024-09-16T01:40:04.262661+0000","last_undegraded":"2024-09-16T01:40:04.262661+0000","last_fullsized":"2024-09-16T01:40:04.262661+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:39:52.543069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.276882+0000","last_change":"2024-09-16T01:40:04.273684+0000","last_active":"2024-09-16T01:40:04.276882+0000","last_peered":"2024-09-16T01:40:04.276882+0000","last_clean":"2024-09-16T01:40:04.276882+0000","last_became_active":"2024-09-16T01:40:04.273447+0000","last_became_peered":"2024-09-16T01:40:04.273447+0000","last_unstale":"2024-09-16T01:40:04.276882+0000","last_undegraded":"2024-09-16T01:40:04.276882+0000","last_fullsized":"2024-09-16T01:40:04.276882+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:21.486329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"84'8","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256968+0000","last_change":"2024-09-16T01:40:03.267394+0000","last_active":"2024-09-16T01:40:05.256968+0000","last_peered":"2024-09-16T01:40:05.256968+0000","last_clean":"2024-09-16T01:40:05.256968+0000","last_became_active":"2024-09-16T01:40:03.267102+0000","last_became_peered":"2024-09-16T01:40:03.267102+0000","last_unstale":"2024-09-16T01:40:05.256968+0000","last_undegraded":"2024-09-16T01:40:05.256968+0000","last_fullsized":"2024-09-16T01:40:05.256968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:26:31.689093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.255936+0000","last_change":"2024-09-16T01:40:04.270698+0000","last_active":"2024-09-16T01:40:05.255936+0000","last_peered":"2024-09-16T01:40:05.255936+0000","last_clean":"2024-09-16T01:40:05.255936+0000","last_became_active":"2024-09-16T01:40:04.270413+0000","last_became_peered":"2024-09-16T01:40:04.270413+0000","last_unstale":"2024-09-16T01:40:05.255936+0000","last_undegraded":"2024-09-16T01:40:05.255936+0000","last_fullsized":"2024-09-16T01:40:05.255936+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:56:44.277657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"84'1","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.257046+0000","last_change":"2024-09-16T01:40:03.266140+0000","last_active":"2024-09-16T01:40:05.257046+0000","last_peered":"2024-09-16T01:40:05.257046+0000","last_clean":"2024-09-16T01:40:05.257046+0000","last_became_active":"2024-09-16T01:40:03.265680+0000","last_became_peered":"2024-09-16T01:40:03.265680+0000","last_unstale":"2024-09-16T01:40:05.257046+0000","last_undegraded":"2024-09-16T01:40:05.257046+0000","last_fullsized":"2024-09-16T01:40:05.257046+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:16:00.398104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256762+0000","last_change":"2024-09-16T01:40:04.260299+0000","last_active":"2024-09-16T01:40:05.256762+0000","last_peered":"2024-09-16T01:40:05.256762+0000","last_clean":"2024-09-16T01:40:05.256762+0000","last_became_active":"2024-09-16T01:40:04.260017+0000","last_became_peered":"2024-09-16T01:40:04.260017+0000","last_unstale":"2024-09-16T01:40:05.256762+0000","last_undegraded":"2024-09-16T01:40:05.256762+0000","last_fullsized":"2024-09-16T01:40:05.256762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:38:50.149048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"84'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.257301+0000","last_change":"2024-09-16T01:40:03.269449+0000","last_active":"2024-09-16T01:40:05.257301+0000","last_peered":"2024-09-16T01:40:05.257301+0000","last_clean":"2024-09-16T01:40:05.257301+0000","last_became_active":"2024-09-16T01:40:03.269187+0000","last_became_peered":"2024-09-16T01:40:03.269187+0000","last_unstale":"2024-09-16T01:40:05.257301+0000","last_undegraded":"2024-09-16T01:40:05.257301+0000","last_fullsized":"2024-09-16T01:40:05.257301+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:57.117123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.279593+0000","last_change":"2024-09-16T01:40:04.274031+0000","last_active":"2024-09-16T01:40:04.279593+0000","last_peered":"2024-09-16T01:40:04.279593+0000","last_clean":"2024-09-16T01:40:04.279593+0000","last_became_active":"2024-09-16T01:40:04.273151+0000","last_became_peered":"2024-09-16T01:40:04.273151+0000","last_unstale":"2024-09-16T01:40:04.279593+0000","last_undegraded":"2024-09-16T01:40:04.279593+0000","last_fullsized":"2024-09-16T01:40:04.279593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:21:23.407416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.257363+0000","last_change":"2024-09-16T01:40:03.265220+0000","last_active":"2024-09-16T01:40:05.257363+0000","last_peered":"2024-09-16T01:40:05.257363+0000","last_clean":"2024-09-16T01:40:05.257363+0000","last_became_active":"2024-09-16T01:40:03.264921+0000","last_became_peered":"2024-09-16T01:40:03.264921+0000","last_unstale":"2024-09-16T01:40:05.257363+0000","last_undegraded":"2024-09-16T01:40:05.257363+0000","last_fullsized":"2024-09-16T01:40:05.257363+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:11.684010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.287691+0000","last_change":"2024-09-16T01:40:04.283264+0000","last_active":"2024-09-16T01:40:04.287691+0000","last_peered":"2024-09-16T01:40:04.287691+0000","last_clean":"2024-09-16T01:40:04.287691+0000","last_became_active":"2024-09-16T01:40:04.282624+0000","last_became_peered":"2024-09-16T01:40:04.282624+0000","last_unstale":"2024-09-16T01:40:04.287691+0000","last_undegraded":"2024-09-16T01:40:04.287691+0000","last_fullsized":"2024-09-16T01:40:04.287691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:39:05.135707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"84'2","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.284428+0000","last_change":"2024-09-16T01:40:03.266541+0000","last_active":"2024-09-16T01:40:04.284428+0000","last_peered":"2024-09-16T01:40:04.284428+0000","last_clean":"2024-09-16T01:40:04.284428+0000","last_became_active":"2024-09-16T01:40:03.266278+0000","last_became_peered":"2024-09-16T01:40:03.266278+0000","last_unstale":"2024-09-16T01:40:04.284428+0000","last_undegraded":"2024-09-16T01:40:04.284428+0000","last_fullsized":"2024-09-16T01:40:04.284428+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:21:53.213917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.270441+0000","last_change":"2024-09-16T01:40:04.267979+0000","last_active":"2024-09-16T01:40:04.270441+0000","last_peered":"2024-09-16T01:40:04.270441+0000","last_clean":"2024-09-16T01:40:04.270441+0000","last_became_active":"2024-09-16T01:40:04.267674+0000","last_became_peered":"2024-09-16T01:40:04.267674+0000","last_unstale":"2024-09-16T01:40:04.270441+0000","last_undegraded":"2024-09-16T01:40:04.270441+0000","last_fullsized":"2024-09-16T01:40:04.270441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:40:37.567970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.271019+0000","last_change":"2024-09-16T01:40:03.269303+0000","last_active":"2024-09-16T01:40:04.271019+0000","last_peered":"2024-09-16T01:40:04.271019+0000","last_clean":"2024-09-16T01:40:04.271019+0000","last_became_active":"2024-09-16T01:40:03.269016+0000","last_became_peered":"2024-09-16T01:40:03.269016+0000","last_unstale":"2024-09-16T01:40:04.271019+0000","last_undegraded":"2024-09-16T01:40:04.271019+0000","last_fullsized":"2024-09-16T01:40:04.271019+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:22.141910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.289640+0000","last_change":"2024-09-16T01:40:04.283679+0000","last_active":"2024-09-16T01:40:04.289640+0000","last_peered":"2024-09-16T01:40:04.289640+0000","last_clean":"2024-09-16T01:40:04.289640+0000","last_became_active":"2024-09-16T01:40:04.283181+0000","last_became_peered":"2024-09-16T01:40:04.283181+0000","last_unstale":"2024-09-16T01:40:04.289640+0000","last_undegraded":"2024-09-16T01:40:04.289640+0000","last_fullsized":"2024-09-16T01:40:04.289640+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:40:30.217611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.266347+0000","last_change":"2024-09-16T01:40:03.267916+0000","last_active":"2024-09-16T01:40:04.266347+0000","last_peered":"2024-09-16T01:40:04.266347+0000","last_clean":"2024-09-16T01:40:04.266347+0000","last_became_active":"2024-09-16T01:40:03.267576+0000","last_became_peered":"2024-09-16T01:40:03.267576+0000","last_unstale":"2024-09-16T01:40:04.266347+0000","last_undegraded":"2024-09-16T01:40:04.266347+0000","last_fullsized":"2024-09-16T01:40:04.266347+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:46:17.773706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.281312+0000","last_change":"2024-09-16T01:40:04.279810+0000","last_active":"2024-09-16T01:40:04.281312+0000","last_peered":"2024-09-16T01:40:04.281312+0000","last_clean":"2024-09-16T01:40:04.281312+0000","last_became_active":"2024-09-16T01:40:04.279547+0000","last_became_peered":"2024-09-16T01:40:04.279547+0000","last_unstale":"2024-09-16T01:40:04.281312+0000","last_undegraded":"2024-09-16T01:40:04.281312+0000","last_fullsized":"2024-09-16T01:40:04.281312+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:12:25.121393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.255965+0000","last_change":"2024-09-16T01:40:03.266456+0000","last_active":"2024-09-16T01:40:04.255965+0000","last_peered":"2024-09-16T01:40:04.255965+0000","last_clean":"2024-09-16T01:40:04.255965+0000","last_became_active":"2024-09-16T01:40:03.266182+0000","last_became_peered":"2024-09-16T01:40:03.266182+0000","last_unstale":"2024-09-16T01:40:04.255965+0000","last_undegraded":"2024-09-16T01:40:04.255965+0000","last_fullsized":"2024-09-16T01:40:04.255965+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:38:08.121960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.279268+0000","last_change":"2024-09-16T01:40:04.271156+0000","last_active":"2024-09-16T01:40:04.279268+0000","last_peered":"2024-09-16T01:40:04.279268+0000","last_clean":"2024-09-16T01:40:04.279268+0000","last_became_active":"2024-09-16T01:40:04.270887+0000","last_became_peered":"2024-09-16T01:40:04.270887+0000","last_unstale":"2024-09-16T01:40:04.279268+0000","last_undegraded":"2024-09-16T01:40:04.279268+0000","last_fullsized":"2024-09-16T01:40:04.279268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:58.119415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"84'2","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.278751+0000","last_change":"2024-09-16T01:40:03.267643+0000","last_active":"2024-09-16T01:40:04.278751+0000","last_peered":"2024-09-16T01:40:04.278751+0000","last_clean":"2024-09-16T01:40:04.278751+0000","last_became_active":"2024-09-16T01:40:03.267277+0000","last_became_peered":"2024-09-16T01:40:03.267277+0000","last_unstale":"2024-09-16T01:40:04.278751+0000","last_undegraded":"2024-09-16T01:40:04.278751+0000","last_fullsized":"2024-09-16T01:40:04.278751+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:11:49.796239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256674+0000","last_change":"2024-09-16T01:40:04.259552+0000","last_active":"2024-09-16T01:40:05.256674+0000","last_peered":"2024-09-16T01:40:05.256674+0000","last_clean":"2024-09-16T01:40:05.256674+0000","last_became_active":"2024-09-16T01:40:04.259274+0000","last_became_peered":"2024-09-16T01:40:04.259274+0000","last_unstale":"2024-09-16T01:40:05.256674+0000","last_undegraded":"2024-09-16T01:40:05.256674+0000","last_fullsized":"2024-09-16T01:40:05.256674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:43.697016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.262748+0000","last_change":"2024-09-16T01:40:03.262794+0000","last_active":"2024-09-16T01:40:04.262748+0000","last_peered":"2024-09-16T01:40:04.262748+0000","last_clean":"2024-09-16T01:40:04.262748+0000","last_became_active":"2024-09-16T01:40:03.262515+0000","last_became_peered":"2024-09-16T01:40:03.262515+0000","last_unstale":"2024-09-16T01:40:04.262748+0000","last_undegraded":"2024-09-16T01:40:04.262748+0000","last_fullsized":"2024-09-16T01:40:04.262748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:52:40.436237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256512+0000","last_change":"2024-09-16T01:40:04.286093+0000","last_active":"2024-09-16T01:40:05.256512+0000","last_peered":"2024-09-16T01:40:05.256512+0000","last_clean":"2024-09-16T01:40:05.256512+0000","last_became_active":"2024-09-16T01:40:04.285766+0000","last_became_peered":"2024-09-16T01:40:04.285766+0000","last_unstale":"2024-09-16T01:40:05.256512+0000","last_undegraded":"2024-09-16T01:40:05.256512+0000","last_fullsized":"2024-09-16T01:40:05.256512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:34:02.972536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.255945+0000","last_change":"2024-09-16T01:40:03.267093+0000","last_active":"2024-09-16T01:40:05.255945+0000","last_peered":"2024-09-16T01:40:05.255945+0000","last_clean":"2024-09-16T01:40:05.255945+0000","last_became_active":"2024-09-16T01:40:03.266690+0000","last_became_peered":"2024-09-16T01:40:03.266690+0000","last_unstale":"2024-09-16T01:40:05.255945+0000","last_undegraded":"2024-09-16T01:40:05.255945+0000","last_fullsized":"2024-09-16T01:40:05.255945+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:25:06.080867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.284931+0000","last_change":"2024-09-16T01:40:04.283450+0000","last_active":"2024-09-16T01:40:04.284931+0000","last_peered":"2024-09-16T01:40:04.284931+0000","last_clean":"2024-09-16T01:40:04.284931+0000","last_became_active":"2024-09-16T01:40:04.283099+0000","last_became_peered":"2024-09-16T01:40:04.283099+0000","last_unstale":"2024-09-16T01:40:04.284931+0000","last_undegraded":"2024-09-16T01:40:04.284931+0000","last_fullsized":"2024-09-16T01:40:04.284931+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:56:42.591248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256602+0000","last_change":"2024-09-16T01:40:03.264409+0000","last_active":"2024-09-16T01:40:05.256602+0000","last_peered":"2024-09-16T01:40:05.256602+0000","last_clean":"2024-09-16T01:40:05.256602+0000","last_became_active":"2024-09-16T01:40:03.263982+0000","last_became_peered":"2024-09-16T01:40:03.263982+0000","last_unstale":"2024-09-16T01:40:05.256602+0000","last_undegraded":"2024-09-16T01:40:05.256602+0000","last_fullsized":"2024-09-16T01:40:05.256602+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:48.392893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.270361+0000","last_change":"2024-09-16T01:40:04.269189+0000","last_active":"2024-09-16T01:40:04.270361+0000","last_peered":"2024-09-16T01:40:04.270361+0000","last_clean":"2024-09-16T01:40:04.270361+0000","last_became_active":"2024-09-16T01:40:04.268898+0000","last_became_peered":"2024-09-16T01:40:04.268898+0000","last_unstale":"2024-09-16T01:40:04.270361+0000","last_undegraded":"2024-09-16T01:40:04.270361+0000","last_fullsized":"2024-09-16T01:40:04.270361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:52:31.738742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.257332+0000","last_change":"2024-09-16T01:40:03.271013+0000","last_active":"2024-09-16T01:40:05.257332+0000","last_peered":"2024-09-16T01:40:05.257332+0000","last_clean":"2024-09-16T01:40:05.257332+0000","last_became_active":"2024-09-16T01:40:03.270739+0000","last_became_peered":"2024-09-16T01:40:03.270739+0000","last_unstale":"2024-09-16T01:40:05.257332+0000","last_undegraded":"2024-09-16T01:40:05.257332+0000","last_fullsized":"2024-09-16T01:40:05.257332+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:21:22.405123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.257247+0000","last_change":"2024-09-16T01:40:04.272964+0000","last_active":"2024-09-16T01:40:05.257247+0000","last_peered":"2024-09-16T01:40:05.257247+0000","last_clean":"2024-09-16T01:40:05.257247+0000","last_became_active":"2024-09-16T01:40:04.272717+0000","last_became_peered":"2024-09-16T01:40:04.272717+0000","last_unstale":"2024-09-16T01:40:05.257247+0000","last_undegraded":"2024-09-16T01:40:05.257247+0000","last_fullsized":"2024-09-16T01:40:05.257247+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:30:25.714992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.260302+0000","last_change":"2024-09-16T01:40:03.264491+0000","last_active":"2024-09-16T01:40:04.260302+0000","last_peered":"2024-09-16T01:40:04.260302+0000","last_clean":"2024-09-16T01:40:04.260302+0000","last_became_active":"2024-09-16T01:40:03.264185+0000","last_became_peered":"2024-09-16T01:40:03.264185+0000","last_unstale":"2024-09-16T01:40:04.260302+0000","last_undegraded":"2024-09-16T01:40:04.260302+0000","last_fullsized":"2024-09-16T01:40:04.260302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:02:22.041889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.255717+0000","last_change":"2024-09-16T01:40:04.265646+0000","last_active":"2024-09-16T01:40:05.255717+0000","last_peered":"2024-09-16T01:40:05.255717+0000","last_clean":"2024-09-16T01:40:05.255717+0000","last_became_active":"2024-09-16T01:40:04.265257+0000","last_became_peered":"2024-09-16T01:40:04.265257+0000","last_unstale":"2024-09-16T01:40:05.255717+0000","last_undegraded":"2024-09-16T01:40:05.255717+0000","last_fullsized":"2024-09-16T01:40:05.255717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:52:48.110074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.255980+0000","last_change":"2024-09-16T01:40:03.263473+0000","last_active":"2024-09-16T01:40:05.255980+0000","last_peered":"2024-09-16T01:40:05.255980+0000","last_clean":"2024-09-16T01:40:05.255980+0000","last_became_active":"2024-09-16T01:40:03.263206+0000","last_became_peered":"2024-09-16T01:40:03.263206+0000","last_unstale":"2024-09-16T01:40:05.255980+0000","last_undegraded":"2024-09-16T01:40:05.255980+0000","last_fullsized":"2024-09-16T01:40:05.255980+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:47:23.618181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.255891+0000","last_change":"2024-09-16T01:40:04.283274+0000","last_active":"2024-09-16T01:40:05.255891+0000","last_peered":"2024-09-16T01:40:05.255891+0000","last_clean":"2024-09-16T01:40:05.255891+0000","last_became_active":"2024-09-16T01:40:04.282915+0000","last_became_peered":"2024-09-16T01:40:04.282915+0000","last_unstale":"2024-09-16T01:40:05.255891+0000","last_undegraded":"2024-09-16T01:40:05.255891+0000","last_fullsized":"2024-09-16T01:40:05.255891+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:38:09.124252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.261118+0000","last_change":"2024-09-16T01:40:03.266672+0000","last_active":"2024-09-16T01:40:04.261118+0000","last_peered":"2024-09-16T01:40:04.261118+0000","last_clean":"2024-09-16T01:40:04.261118+0000","last_became_active":"2024-09-16T01:40:03.266361+0000","last_became_peered":"2024-09-16T01:40:03.266361+0000","last_unstale":"2024-09-16T01:40:04.261118+0000","last_undegraded":"2024-09-16T01:40:04.261118+0000","last_fullsized":"2024-09-16T01:40:04.261118+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:37.909549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.275664+0000","last_change":"2024-09-16T01:40:04.268702+0000","last_active":"2024-09-16T01:40:04.275664+0000","last_peered":"2024-09-16T01:40:04.275664+0000","last_clean":"2024-09-16T01:40:04.275664+0000","last_became_active":"2024-09-16T01:40:04.268402+0000","last_became_peered":"2024-09-16T01:40:04.268402+0000","last_unstale":"2024-09-16T01:40:04.275664+0000","last_undegraded":"2024-09-16T01:40:04.275664+0000","last_fullsized":"2024-09-16T01:40:04.275664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:34:32.842050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"84'1","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256431+0000","last_change":"2024-09-16T01:40:03.273312+0000","last_active":"2024-09-16T01:40:05.256431+0000","last_peered":"2024-09-16T01:40:05.256431+0000","last_clean":"2024-09-16T01:40:05.256431+0000","last_became_active":"2024-09-16T01:40:03.272995+0000","last_became_peered":"2024-09-16T01:40:03.272995+0000","last_unstale":"2024-09-16T01:40:05.256431+0000","last_undegraded":"2024-09-16T01:40:05.256431+0000","last_fullsized":"2024-09-16T01:40:05.256431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:23:37.461745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":286,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.255844+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:40:05.255844+0000","last_peered":"2024-09-16T01:40:05.255844+0000","last_clean":"2024-09-16T01:40:05.255844+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:40:05.255844+0000","last_undegraded":"2024-09-16T01:40:05.255844+0000","last_fullsized":"2024-09-16T01:40:05.255844+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1044480,"data_stored":440156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542219,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32780,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339060,"statfs":{"total":96636764160,"available":96603197440,"internally_reserved":0,"allocated":1626112,"data_stored":1028960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28060,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343780,"statfs":{"total":96636764160,"available":96608030720,"internally_reserved":0,"allocated":1052672,"data_stored":445516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542219,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28032,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343808,"statfs":{"total":96636764160,"available":96608059392,"internally_reserved":0,"allocated":1028096,"data_stored":438522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542218,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28048,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343792,"statfs":{"total":96636764160,"available":96608043008,"internally_reserved":0,"allocated":1040384,"data_stored":440134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542219,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28036,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343804,"statfs":{"total":96636764160,"available":96608055296,"internally_reserved":0,"allocated":1032192,"data_stored":439104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542218,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":1040384,"data_stored":438716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542218,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32796,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339044,"statfs":{"total":96636764160,"available":96603181056,"internally_reserved":0,"allocated":1638400,"data_stored":1035224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:40:07.207 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:40:07.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.340+0000 7f360b8a5640 1 Processor -- start 2024-09-16T01:40:07.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.340+0000 7f360b8a5640 1 -- start start 2024-09-16T01:40:07.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.341+0000 7f360b8a5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3604145bb0 0x7f360406e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:07.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.341+0000 7f360b8a5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f360406ea10 con 0x7f360413f400 2024-09-16T01:40:07.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.341+0000 7f360b8a5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36040d03c0 con 0x7f360406e5f0 2024-09-16T01:40:07.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.341+0000 7f360b8a5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f36040d06a0 con 0x7f3604145bb0 2024-09-16T01:40:07.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.341+0000 7f3608e40640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3604145bb0 0x7f360406e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:07.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.341+0000 7f3609641640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f360413f400 0x7f36040cfab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33306/0 (socket says 172.21.15.88:33306) 2024-09-16T01:40:07.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.341+0000 7f3609641640 1 -- 172.21.15.88:0/1545922466 learned_addr learned my addr 172.21.15.88:0/1545922466 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:07.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.341+0000 7f3608e40640 1 -- 172.21.15.88:0/1545922466 >> v1:172.21.15.154:6789/0 conn(0x7f360406e5f0 legacy=0x7f36041439e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:07.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.341+0000 7f3608e40640 1 -- 172.21.15.88:0/1545922466 >> v1:172.21.15.88:6789/0 conn(0x7f360413f400 legacy=0x7f36040cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:07.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.341+0000 7f3608e40640 1 -- 172.21.15.88:0/1545922466 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36041431e0 con 0x7f3604145bb0 2024-09-16T01:40:07.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.341+0000 7f35fbfff640 1 -- 172.21.15.88:0/1545922466 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3659381691 0 0) 0x7f360406ea10 con 0x7f360413f400 2024-09-16T01:40:07.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.342+0000 7f3608e40640 1 --2- 172.21.15.88:0/1545922466 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3604145bb0 0x7f360406e0b0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f35f40097a0 tx=0x7f35f402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfb49dc276f38744 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:07.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.342+0000 7f35fbfff640 1 -- 172.21.15.88:0/1545922466 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f4037070 con 0x7f3604145bb0 2024-09-16T01:40:07.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.342+0000 7f35fbfff640 1 -- 172.21.15.88:0/1545922466 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35f402ae50 con 0x7f3604145bb0 2024-09-16T01:40:07.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.342+0000 7f35fbfff640 1 -- 172.21.15.88:0/1545922466 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f4032660 con 0x7f3604145bb0 2024-09-16T01:40:07.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.343+0000 7f360b8a5640 1 -- 172.21.15.88:0/1545922466 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3604145bb0 msgr2=0x7f360406e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:07.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.343+0000 7f360b8a5640 1 --2- 172.21.15.88:0/1545922466 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3604145bb0 0x7f360406e0b0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f35f40097a0 tx=0x7f35f402a140 comp rx=0 tx=0).stop 2024-09-16T01:40:07.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.343+0000 7f360b8a5640 1 -- 172.21.15.88:0/1545922466 shutdown_connections 2024-09-16T01:40:07.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.343+0000 7f360b8a5640 1 --2- 172.21.15.88:0/1545922466 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3604145bb0 0x7f360406e0b0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.343+0000 7f360b8a5640 1 -- 172.21.15.88:0/1545922466 >> 172.21.15.88:0/1545922466 conn(0x7f360406bc10 msgr2=0x7f360406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:07.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.343+0000 7f360b8a5640 1 -- 172.21.15.88:0/1545922466 shutdown_connections 2024-09-16T01:40:07.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.343+0000 7f360b8a5640 1 -- 172.21.15.88:0/1545922466 wait complete. 2024-09-16T01:40:07.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.344+0000 7f360b8a5640 1 Processor -- start 2024-09-16T01:40:07.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.344+0000 7f360b8a5640 1 -- start start 2024-09-16T01:40:07.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.345+0000 7f360b8a5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f360406e5f0 0x7f36040aef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:07.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.345+0000 7f360b8a5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f360413f400 0x7f36041f8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:07.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.345+0000 7f360b8a5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3604145bb0 0x7f360413e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:07.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.345+0000 7f360b8a5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f360413f960 con 0x7f360413f400 2024-09-16T01:40:07.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.345+0000 7f360b8a5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f360413f7e0 con 0x7f3604145bb0 2024-09-16T01:40:07.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.345+0000 7f360b8a5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f36041440f0 con 0x7f360406e5f0 2024-09-16T01:40:07.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.345+0000 7f3608e40640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f360413f400 0x7f36041f8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:07.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.345+0000 7f3608e40640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f360413f400 0x7f36041f8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60074/0 (socket says 172.21.15.88:60074) 2024-09-16T01:40:07.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.345+0000 7f3608e40640 1 -- 172.21.15.88:0/402870332 learned_addr learned my addr 172.21.15.88:0/402870332 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:07.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.345+0000 7f3609e42640 1 --2- 172.21.15.88:0/402870332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3604145bb0 0x7f360413e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:07.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.345+0000 7f3609641640 1 --2- 172.21.15.88:0/402870332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f360406e5f0 0x7f36040aef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:07.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.345+0000 7f3608e40640 1 -- 172.21.15.88:0/402870332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f360406e5f0 msgr2=0x7f36040aef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:07.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.345+0000 7f3608e40640 1 --2- 172.21.15.88:0/402870332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f360406e5f0 0x7f36040aef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.345+0000 7f3608e40640 1 -- 172.21.15.88:0/402870332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3604145bb0 msgr2=0x7f360413e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:07.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.345+0000 7f3608e40640 1 --2- 172.21.15.88:0/402870332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3604145bb0 0x7f360413e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.345+0000 7f3608e40640 1 -- 172.21.15.88:0/402870332 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36040af660 con 0x7f360413f400 2024-09-16T01:40:07.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.345+0000 7f3609641640 1 --2- 172.21.15.88:0/402870332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f360406e5f0 0x7f36040aef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:07.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.345+0000 7f3608e40640 1 --2- 172.21.15.88:0/402870332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f360413f400 0x7f36041f8520 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f35f40097a0 tx=0x7f35f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:07.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.345+0000 7f35f9ffb640 1 -- 172.21.15.88:0/402870332 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f4037040 con 0x7f360413f400 2024-09-16T01:40:07.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.345+0000 7f35f9ffb640 1 -- 172.21.15.88:0/402870332 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35f4004030 con 0x7f360413f400 2024-09-16T01:40:07.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.345+0000 7f360b8a5640 1 -- 172.21.15.88:0/402870332 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36040af940 con 0x7f360413f400 2024-09-16T01:40:07.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.345+0000 7f35f9ffb640 1 -- 172.21.15.88:0/402870332 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f4006e10 con 0x7f360413f400 2024-09-16T01:40:07.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.346+0000 7f360b8a5640 1 -- 172.21.15.88:0/402870332 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36040aff00 con 0x7f360413f400 2024-09-16T01:40:07.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.346+0000 7f360b8a5640 1 -- 172.21.15.88:0/402870332 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f360406e0b0 con 0x7f360413f400 2024-09-16T01:40:07.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.347+0000 7f35f9ffb640 1 -- 172.21.15.88:0/402870332 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f35f403d650 con 0x7f360413f400 2024-09-16T01:40:07.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.347+0000 7f35f9ffb640 1 --2- 172.21.15.88:0/402870332 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f35e806c2b0 0x7f35e806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:07.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.347+0000 7f35f9ffb640 1 -- 172.21.15.88:0/402870332 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f35f40abca0 con 0x7f360413f400 2024-09-16T01:40:07.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.347+0000 7f3609641640 1 --2- 172.21.15.88:0/402870332 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f35e806c2b0 0x7f35e806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:07.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.348+0000 7f3609641640 1 --2- 172.21.15.88:0/402870332 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f35e806c2b0 0x7f35e806e770 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3600000f60 tx=0x7f3600009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:07.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.352+0000 7f35f9ffb640 1 -- 172.21.15.88:0/402870332 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f35f4042020 con 0x7f360413f400 2024-09-16T01:40:07.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.515+0000 7f360b8a5640 1 -- 172.21.15.88:0/402870332 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f360406f420 con 0x7f35e806c2b0 2024-09-16T01:40:07.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.521+0000 7f35f9ffb640 1 -- 172.21.15.88:0/402870332 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252041 (secure 0 0 0) 0x7f360406f420 con 0x7f35e806c2b0 2024-09-16T01:40:07.525 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:07.526 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:40:07.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.524+0000 7f360b8a5640 1 -- 172.21.15.88:0/402870332 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f35e806c2b0 msgr2=0x7f35e806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:07.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.524+0000 7f360b8a5640 1 --2- 172.21.15.88:0/402870332 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f35e806c2b0 0x7f35e806e770 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3600000f60 tx=0x7f3600009040 comp rx=0 tx=0).stop 2024-09-16T01:40:07.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.525+0000 7f360b8a5640 1 -- 172.21.15.88:0/402870332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f360413f400 msgr2=0x7f36041f8520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:07.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.525+0000 7f360b8a5640 1 --2- 172.21.15.88:0/402870332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f360413f400 0x7f36041f8520 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f35f40097a0 tx=0x7f35f40046c0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.525+0000 7f360b8a5640 1 -- 172.21.15.88:0/402870332 shutdown_connections 2024-09-16T01:40:07.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.525+0000 7f360b8a5640 1 --2- 172.21.15.88:0/402870332 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f35e806c2b0 0x7f35e806e770 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.525+0000 7f360b8a5640 1 --2- 172.21.15.88:0/402870332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3604145bb0 0x7f360413e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.525+0000 7f360b8a5640 1 --2- 172.21.15.88:0/402870332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f360413f400 0x7f36041f8520 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.525+0000 7f360b8a5640 1 --2- 172.21.15.88:0/402870332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f360406e5f0 0x7f36040aef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.525+0000 7f360b8a5640 1 -- 172.21.15.88:0/402870332 >> 172.21.15.88:0/402870332 conn(0x7f360406bc10 msgr2=0x7f36040b61b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:07.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.525+0000 7f360b8a5640 1 -- 172.21.15.88:0/402870332 shutdown_connections 2024-09-16T01:40:07.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.525+0000 7f360b8a5640 1 -- 172.21.15.88:0/402870332 wait complete. 2024-09-16T01:40:07.537 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-09-16T01:40:06.593899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":101,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":242172,"kb_used_data":9280,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754732548,"statfs":{"total":773094113280,"available":772846129152,"internally_reserved":0,"allocated":9502720,"data_stored":4706332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.348667"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256764+0000","last_change":"2024-09-16T01:40:04.271300+0000","last_active":"2024-09-16T01:40:05.256764+0000","last_peered":"2024-09-16T01:40:05.256764+0000","last_clean":"2024-09-16T01:40:05.256764+0000","last_became_active":"2024-09-16T01:40:04.270801+0000","last_became_peered":"2024-09-16T01:40:04.270801+0000","last_unstale":"2024-09-16T01:40:05.256764+0000","last_undegraded":"2024-09-16T01:40:05.256764+0000","last_fullsized":"2024-09-16T01:40:05.256764+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:39.441878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.281545+0000","last_change":"2024-09-16T01:40:04.275657+0000","last_active":"2024-09-16T01:40:04.281545+0000","last_peered":"2024-09-16T01:40:04.281545+0000","last_clean":"2024-09-16T01:40:04.281545+0000","last_became_active":"2024-09-16T01:40:04.274280+0000","last_became_peered":"2024-09-16T01:40:04.274280+0000","last_unstale":"2024-09-16T01:40:04.281545+0000","last_undegraded":"2024-09-16T01:40:04.281545+0000","last_fullsized":"2024-09-16T01:40:04.281545+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:33:39.723365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256322+0000","last_change":"2024-09-16T01:40:04.259552+0000","last_active":"2024-09-16T01:40:05.256322+0000","last_peered":"2024-09-16T01:40:05.256322+0000","last_clean":"2024-09-16T01:40:05.256322+0000","last_became_active":"2024-09-16T01:40:04.259252+0000","last_became_peered":"2024-09-16T01:40:04.259252+0000","last_unstale":"2024-09-16T01:40:05.256322+0000","last_undegraded":"2024-09-16T01:40:05.256322+0000","last_fullsized":"2024-09-16T01:40:05.256322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:55:53.219368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.289678+0000","last_change":"2024-09-16T01:40:04.284529+0000","last_active":"2024-09-16T01:40:04.289678+0000","last_peered":"2024-09-16T01:40:04.289678+0000","last_clean":"2024-09-16T01:40:04.289678+0000","last_became_active":"2024-09-16T01:40:04.284280+0000","last_became_peered":"2024-09-16T01:40:04.284280+0000","last_unstale":"2024-09-16T01:40:04.289678+0000","last_undegraded":"2024-09-16T01:40:04.289678+0000","last_fullsized":"2024-09-16T01:40:04.289678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:49:59.001632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.279416+0000","last_change":"2024-09-16T01:40:04.273647+0000","last_active":"2024-09-16T01:40:04.279416+0000","last_peered":"2024-09-16T01:40:04.279416+0000","last_clean":"2024-09-16T01:40:04.279416+0000","last_became_active":"2024-09-16T01:40:04.273362+0000","last_became_peered":"2024-09-16T01:40:04.273362+0000","last_unstale":"2024-09-16T01:40:04.279416+0000","last_undegraded":"2024-09-16T01:40:04.279416+0000","last_fullsized":"2024-09-16T01:40:04.279416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:16:01.400396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.286268+0000","last_change":"2024-09-16T01:40:04.285106+0000","last_active":"2024-09-16T01:40:04.286268+0000","last_peered":"2024-09-16T01:40:04.286268+0000","last_clean":"2024-09-16T01:40:04.286268+0000","last_became_active":"2024-09-16T01:40:04.284804+0000","last_became_peered":"2024-09-16T01:40:04.284804+0000","last_unstale":"2024-09-16T01:40:04.286268+0000","last_undegraded":"2024-09-16T01:40:04.286268+0000","last_fullsized":"2024-09-16T01:40:04.286268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:06:54.718293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.287590+0000","last_change":"2024-09-16T01:40:04.282323+0000","last_active":"2024-09-16T01:40:04.287590+0000","last_peered":"2024-09-16T01:40:04.287590+0000","last_clean":"2024-09-16T01:40:04.287590+0000","last_became_active":"2024-09-16T01:40:04.282052+0000","last_became_peered":"2024-09-16T01:40:04.282052+0000","last_unstale":"2024-09-16T01:40:04.287590+0000","last_undegraded":"2024-09-16T01:40:04.287590+0000","last_fullsized":"2024-09-16T01:40:04.287590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:58.420086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.270398+0000","last_change":"2024-09-16T01:40:04.268240+0000","last_active":"2024-09-16T01:40:04.270398+0000","last_peered":"2024-09-16T01:40:04.270398+0000","last_clean":"2024-09-16T01:40:04.270398+0000","last_became_active":"2024-09-16T01:40:04.267343+0000","last_became_peered":"2024-09-16T01:40:04.267343+0000","last_unstale":"2024-09-16T01:40:04.270398+0000","last_undegraded":"2024-09-16T01:40:04.270398+0000","last_fullsized":"2024-09-16T01:40:04.270398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:46:06.509496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.281326+0000","last_change":"2024-09-16T01:40:04.280262+0000","last_active":"2024-09-16T01:40:04.281326+0000","last_peered":"2024-09-16T01:40:04.281326+0000","last_clean":"2024-09-16T01:40:04.281326+0000","last_became_active":"2024-09-16T01:40:04.280068+0000","last_became_peered":"2024-09-16T01:40:04.280068+0000","last_unstale":"2024-09-16T01:40:04.281326+0000","last_undegraded":"2024-09-16T01:40:04.281326+0000","last_fullsized":"2024-09-16T01:40:04.281326+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:05:40.988739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256335+0000","last_change":"2024-09-16T01:40:04.269451+0000","last_active":"2024-09-16T01:40:05.256335+0000","last_peered":"2024-09-16T01:40:05.256335+0000","last_clean":"2024-09-16T01:40:05.256335+0000","last_became_active":"2024-09-16T01:40:04.269175+0000","last_became_peered":"2024-09-16T01:40:04.269175+0000","last_unstale":"2024-09-16T01:40:05.256335+0000","last_undegraded":"2024-09-16T01:40:05.256335+0000","last_fullsized":"2024-09-16T01:40:05.256335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:07:13.027761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.271517+0000","last_change":"2024-09-16T01:40:04.265706+0000","last_active":"2024-09-16T01:40:04.271517+0000","last_peered":"2024-09-16T01:40:04.271517+0000","last_clean":"2024-09-16T01:40:04.271517+0000","last_became_active":"2024-09-16T01:40:04.265426+0000","last_became_peered":"2024-09-16T01:40:04.265426+0000","last_unstale":"2024-09-16T01:40:04.271517+0000","last_undegraded":"2024-09-16T01:40:04.271517+0000","last_fullsized":"2024-09-16T01:40:04.271517+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:09.970446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256197+0000","last_change":"2024-09-16T01:40:04.268695+0000","last_active":"2024-09-16T01:40:05.256197+0000","last_peered":"2024-09-16T01:40:05.256197+0000","last_clean":"2024-09-16T01:40:05.256197+0000","last_became_active":"2024-09-16T01:40:04.268433+0000","last_became_peered":"2024-09-16T01:40:04.268433+0000","last_unstale":"2024-09-16T01:40:05.256197+0000","last_undegraded":"2024-09-16T01:40:05.256197+0000","last_fullsized":"2024-09-16T01:40:05.256197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:21:54.216209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.255994+0000","last_change":"2024-09-16T01:40:04.266270+0000","last_active":"2024-09-16T01:40:05.255994+0000","last_peered":"2024-09-16T01:40:05.255994+0000","last_clean":"2024-09-16T01:40:05.255994+0000","last_became_active":"2024-09-16T01:40:04.265944+0000","last_became_peered":"2024-09-16T01:40:04.265944+0000","last_unstale":"2024-09-16T01:40:05.255994+0000","last_undegraded":"2024-09-16T01:40:05.255994+0000","last_fullsized":"2024-09-16T01:40:05.255994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:51:12.899296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.289715+0000","last_change":"2024-09-16T01:40:04.283535+0000","last_active":"2024-09-16T01:40:04.289715+0000","last_peered":"2024-09-16T01:40:04.289715+0000","last_clean":"2024-09-16T01:40:04.289715+0000","last_became_active":"2024-09-16T01:40:04.282926+0000","last_became_peered":"2024-09-16T01:40:04.282926+0000","last_unstale":"2024-09-16T01:40:04.289715+0000","last_undegraded":"2024-09-16T01:40:04.289715+0000","last_fullsized":"2024-09-16T01:40:04.289715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:04.397034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.267235+0000","last_change":"2024-09-16T01:40:04.259774+0000","last_active":"2024-09-16T01:40:04.267235+0000","last_peered":"2024-09-16T01:40:04.267235+0000","last_clean":"2024-09-16T01:40:04.267235+0000","last_became_active":"2024-09-16T01:40:04.259469+0000","last_became_peered":"2024-09-16T01:40:04.259469+0000","last_unstale":"2024-09-16T01:40:04.267235+0000","last_undegraded":"2024-09-16T01:40:04.267235+0000","last_fullsized":"2024-09-16T01:40:04.267235+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:30:35.198807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.279454+0000","last_change":"2024-09-16T01:40:04.274557+0000","last_active":"2024-09-16T01:40:04.279454+0000","last_peered":"2024-09-16T01:40:04.279454+0000","last_clean":"2024-09-16T01:40:04.279454+0000","last_became_active":"2024-09-16T01:40:04.274239+0000","last_became_peered":"2024-09-16T01:40:04.274239+0000","last_unstale":"2024-09-16T01:40:04.279454+0000","last_undegraded":"2024-09-16T01:40:04.279454+0000","last_fullsized":"2024-09-16T01:40:04.279454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:43.843151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.283268+0000","last_change":"2024-09-16T01:40:04.282582+0000","last_active":"2024-09-16T01:40:04.283268+0000","last_peered":"2024-09-16T01:40:04.283268+0000","last_clean":"2024-09-16T01:40:04.283268+0000","last_became_active":"2024-09-16T01:40:04.282313+0000","last_became_peered":"2024-09-16T01:40:04.282313+0000","last_unstale":"2024-09-16T01:40:04.283268+0000","last_undegraded":"2024-09-16T01:40:04.283268+0000","last_fullsized":"2024-09-16T01:40:04.283268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:31:59.317167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.289689+0000","last_change":"2024-09-16T01:40:04.283813+0000","last_active":"2024-09-16T01:40:04.289689+0000","last_peered":"2024-09-16T01:40:04.289689+0000","last_clean":"2024-09-16T01:40:04.289689+0000","last_became_active":"2024-09-16T01:40:04.283442+0000","last_became_peered":"2024-09-16T01:40:04.283442+0000","last_unstale":"2024-09-16T01:40:04.289689+0000","last_undegraded":"2024-09-16T01:40:04.289689+0000","last_fullsized":"2024-09-16T01:40:04.289689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:31:29.553476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.257402+0000","last_change":"2024-09-16T01:40:04.272557+0000","last_active":"2024-09-16T01:40:05.257402+0000","last_peered":"2024-09-16T01:40:05.257402+0000","last_clean":"2024-09-16T01:40:05.257402+0000","last_became_active":"2024-09-16T01:40:04.271776+0000","last_became_peered":"2024-09-16T01:40:04.271776+0000","last_unstale":"2024-09-16T01:40:05.257402+0000","last_undegraded":"2024-09-16T01:40:05.257402+0000","last_fullsized":"2024-09-16T01:40:05.257402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:38:19.894228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.289809+0000","last_change":"2024-09-16T01:40:04.286776+0000","last_active":"2024-09-16T01:40:04.289809+0000","last_peered":"2024-09-16T01:40:04.289809+0000","last_clean":"2024-09-16T01:40:04.289809+0000","last_became_active":"2024-09-16T01:40:04.286485+0000","last_became_peered":"2024-09-16T01:40:04.286485+0000","last_unstale":"2024-09-16T01:40:04.289809+0000","last_undegraded":"2024-09-16T01:40:04.289809+0000","last_fullsized":"2024-09-16T01:40:04.289809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:45:29.048955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.289848+0000","last_change":"2024-09-16T01:40:04.287172+0000","last_active":"2024-09-16T01:40:04.289848+0000","last_peered":"2024-09-16T01:40:04.289848+0000","last_clean":"2024-09-16T01:40:04.289848+0000","last_became_active":"2024-09-16T01:40:04.286921+0000","last_became_peered":"2024-09-16T01:40:04.286921+0000","last_unstale":"2024-09-16T01:40:04.289848+0000","last_undegraded":"2024-09-16T01:40:04.289848+0000","last_fullsized":"2024-09-16T01:40:04.289848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:05:01.081881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.287882+0000","last_change":"2024-09-16T01:40:04.282655+0000","last_active":"2024-09-16T01:40:04.287882+0000","last_peered":"2024-09-16T01:40:04.287882+0000","last_clean":"2024-09-16T01:40:04.287882+0000","last_became_active":"2024-09-16T01:40:04.282366+0000","last_became_peered":"2024-09-16T01:40:04.282366+0000","last_unstale":"2024-09-16T01:40:04.287882+0000","last_undegraded":"2024-09-16T01:40:04.287882+0000","last_fullsized":"2024-09-16T01:40:04.287882+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:38.198140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.255751+0000","last_change":"2024-09-16T01:40:04.262760+0000","last_active":"2024-09-16T01:40:05.255751+0000","last_peered":"2024-09-16T01:40:05.255751+0000","last_clean":"2024-09-16T01:40:05.255751+0000","last_became_active":"2024-09-16T01:40:04.262385+0000","last_became_peered":"2024-09-16T01:40:04.262385+0000","last_unstale":"2024-09-16T01:40:05.255751+0000","last_undegraded":"2024-09-16T01:40:05.255751+0000","last_fullsized":"2024-09-16T01:40:05.255751+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:16:49.578781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.289794+0000","last_change":"2024-09-16T01:40:04.286747+0000","last_active":"2024-09-16T01:40:04.289794+0000","last_peered":"2024-09-16T01:40:04.289794+0000","last_clean":"2024-09-16T01:40:04.289794+0000","last_became_active":"2024-09-16T01:40:04.286520+0000","last_became_peered":"2024-09-16T01:40:04.286520+0000","last_unstale":"2024-09-16T01:40:04.289794+0000","last_undegraded":"2024-09-16T01:40:04.289794+0000","last_fullsized":"2024-09-16T01:40:04.289794+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:55:59.275712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.279010+0000","last_change":"2024-09-16T01:40:04.274554+0000","last_active":"2024-09-16T01:40:04.279010+0000","last_peered":"2024-09-16T01:40:04.279010+0000","last_clean":"2024-09-16T01:40:04.279010+0000","last_became_active":"2024-09-16T01:40:04.274139+0000","last_became_peered":"2024-09-16T01:40:04.274139+0000","last_unstale":"2024-09-16T01:40:04.279010+0000","last_undegraded":"2024-09-16T01:40:04.279010+0000","last_fullsized":"2024-09-16T01:40:04.279010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:16:03.129435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.255775+0000","last_change":"2024-09-16T01:40:04.265405+0000","last_active":"2024-09-16T01:40:05.255775+0000","last_peered":"2024-09-16T01:40:05.255775+0000","last_clean":"2024-09-16T01:40:05.255775+0000","last_became_active":"2024-09-16T01:40:04.265114+0000","last_became_peered":"2024-09-16T01:40:04.265114+0000","last_unstale":"2024-09-16T01:40:05.255775+0000","last_undegraded":"2024-09-16T01:40:05.255775+0000","last_fullsized":"2024-09-16T01:40:05.255775+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:55:25.161381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.275180+0000","last_change":"2024-09-16T01:40:04.268566+0000","last_active":"2024-09-16T01:40:04.275180+0000","last_peered":"2024-09-16T01:40:04.275180+0000","last_clean":"2024-09-16T01:40:04.275180+0000","last_became_active":"2024-09-16T01:40:04.268312+0000","last_became_peered":"2024-09-16T01:40:04.268312+0000","last_unstale":"2024-09-16T01:40:04.275180+0000","last_undegraded":"2024-09-16T01:40:04.275180+0000","last_fullsized":"2024-09-16T01:40:04.275180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:49.395185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256354+0000","last_change":"2024-09-16T01:40:04.271058+0000","last_active":"2024-09-16T01:40:05.256354+0000","last_peered":"2024-09-16T01:40:05.256354+0000","last_clean":"2024-09-16T01:40:05.256354+0000","last_became_active":"2024-09-16T01:40:04.270783+0000","last_became_peered":"2024-09-16T01:40:04.270783+0000","last_unstale":"2024-09-16T01:40:05.256354+0000","last_undegraded":"2024-09-16T01:40:05.256354+0000","last_fullsized":"2024-09-16T01:40:05.256354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:30:35.198807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.275580+0000","last_change":"2024-09-16T01:40:04.269738+0000","last_active":"2024-09-16T01:40:04.275580+0000","last_peered":"2024-09-16T01:40:04.275580+0000","last_clean":"2024-09-16T01:40:04.275580+0000","last_became_active":"2024-09-16T01:40:04.269475+0000","last_became_peered":"2024-09-16T01:40:04.269475+0000","last_unstale":"2024-09-16T01:40:04.275580+0000","last_undegraded":"2024-09-16T01:40:04.275580+0000","last_fullsized":"2024-09-16T01:40:04.275580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:22:03.250147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.289596+0000","last_change":"2024-09-16T01:40:04.287758+0000","last_active":"2024-09-16T01:40:04.289596+0000","last_peered":"2024-09-16T01:40:04.289596+0000","last_clean":"2024-09-16T01:40:04.289596+0000","last_became_active":"2024-09-16T01:40:04.287499+0000","last_became_peered":"2024-09-16T01:40:04.287499+0000","last_unstale":"2024-09-16T01:40:04.289596+0000","last_undegraded":"2024-09-16T01:40:04.289596+0000","last_fullsized":"2024-09-16T01:40:04.289596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:02:23.790489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.267851+0000","last_change":"2024-09-16T01:40:04.263004+0000","last_active":"2024-09-16T01:40:04.267851+0000","last_peered":"2024-09-16T01:40:04.267851+0000","last_clean":"2024-09-16T01:40:04.267851+0000","last_became_active":"2024-09-16T01:40:04.262684+0000","last_became_peered":"2024-09-16T01:40:04.262684+0000","last_unstale":"2024-09-16T01:40:04.267851+0000","last_undegraded":"2024-09-16T01:40:04.267851+0000","last_fullsized":"2024-09-16T01:40:04.267851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:25:07.083159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256623+0000","last_change":"2024-09-16T01:40:04.266455+0000","last_active":"2024-09-16T01:40:05.256623+0000","last_peered":"2024-09-16T01:40:05.256623+0000","last_clean":"2024-09-16T01:40:05.256623+0000","last_became_active":"2024-09-16T01:40:04.266151+0000","last_became_peered":"2024-09-16T01:40:04.266151+0000","last_unstale":"2024-09-16T01:40:05.256623+0000","last_undegraded":"2024-09-16T01:40:05.256623+0000","last_fullsized":"2024-09-16T01:40:05.256623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:11:15.958789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.279402+0000","last_change":"2024-09-16T01:40:04.273836+0000","last_active":"2024-09-16T01:40:04.279402+0000","last_peered":"2024-09-16T01:40:04.279402+0000","last_clean":"2024-09-16T01:40:04.279402+0000","last_became_active":"2024-09-16T01:40:04.273476+0000","last_became_peered":"2024-09-16T01:40:04.273476+0000","last_unstale":"2024-09-16T01:40:04.279402+0000","last_undegraded":"2024-09-16T01:40:04.279402+0000","last_fullsized":"2024-09-16T01:40:04.279402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:12.686302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"84'4","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256872+0000","last_change":"2024-09-16T01:40:03.269292+0000","last_active":"2024-09-16T01:40:05.256872+0000","last_peered":"2024-09-16T01:40:05.256872+0000","last_clean":"2024-09-16T01:40:05.256872+0000","last_became_active":"2024-09-16T01:40:03.269039+0000","last_became_peered":"2024-09-16T01:40:03.269039+0000","last_unstale":"2024-09-16T01:40:05.256872+0000","last_undegraded":"2024-09-16T01:40:05.256872+0000","last_fullsized":"2024-09-16T01:40:05.256872+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:26:31.689093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256916+0000","last_change":"2024-09-16T01:40:04.261753+0000","last_active":"2024-09-16T01:40:05.256916+0000","last_peered":"2024-09-16T01:40:05.256916+0000","last_clean":"2024-09-16T01:40:05.256916+0000","last_became_active":"2024-09-16T01:40:04.261464+0000","last_became_peered":"2024-09-16T01:40:04.261464+0000","last_unstale":"2024-09-16T01:40:05.256916+0000","last_undegraded":"2024-09-16T01:40:05.256916+0000","last_fullsized":"2024-09-16T01:40:05.256916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:46:24.061573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"84'1","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.282759+0000","last_change":"2024-09-16T01:40:03.263453+0000","last_active":"2024-09-16T01:40:04.282759+0000","last_peered":"2024-09-16T01:40:04.282759+0000","last_clean":"2024-09-16T01:40:04.282759+0000","last_became_active":"2024-09-16T01:40:03.262836+0000","last_became_peered":"2024-09-16T01:40:03.262836+0000","last_unstale":"2024-09-16T01:40:04.282759+0000","last_undegraded":"2024-09-16T01:40:04.282759+0000","last_fullsized":"2024-09-16T01:40:04.282759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:26:31.689093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256493+0000","last_change":"2024-09-16T01:40:04.266448+0000","last_active":"2024-09-16T01:40:05.256493+0000","last_peered":"2024-09-16T01:40:05.256493+0000","last_clean":"2024-09-16T01:40:05.256493+0000","last_became_active":"2024-09-16T01:40:04.266183+0000","last_became_peered":"2024-09-16T01:40:04.266183+0000","last_unstale":"2024-09-16T01:40:05.256493+0000","last_undegraded":"2024-09-16T01:40:05.256493+0000","last_fullsized":"2024-09-16T01:40:05.256493+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:52:40.505962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"84'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.257297+0000","last_change":"2024-09-16T01:40:03.264515+0000","last_active":"2024-09-16T01:40:05.257297+0000","last_peered":"2024-09-16T01:40:05.257297+0000","last_clean":"2024-09-16T01:40:05.257297+0000","last_became_active":"2024-09-16T01:40:03.264240+0000","last_became_peered":"2024-09-16T01:40:03.264240+0000","last_unstale":"2024-09-16T01:40:05.257297+0000","last_undegraded":"2024-09-16T01:40:05.257297+0000","last_fullsized":"2024-09-16T01:40:05.257297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:48.392893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256374+0000","last_change":"2024-09-16T01:40:04.287012+0000","last_active":"2024-09-16T01:40:05.256374+0000","last_peered":"2024-09-16T01:40:05.256374+0000","last_clean":"2024-09-16T01:40:05.256374+0000","last_became_active":"2024-09-16T01:40:04.286726+0000","last_became_peered":"2024-09-16T01:40:04.286726+0000","last_unstale":"2024-09-16T01:40:05.256374+0000","last_undegraded":"2024-09-16T01:40:05.256374+0000","last_fullsized":"2024-09-16T01:40:05.256374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:11:22.300784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"84'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.257267+0000","last_change":"2024-09-16T01:40:03.264709+0000","last_active":"2024-09-16T01:40:05.257267+0000","last_peered":"2024-09-16T01:40:05.257267+0000","last_clean":"2024-09-16T01:40:05.257267+0000","last_became_active":"2024-09-16T01:40:03.264193+0000","last_became_peered":"2024-09-16T01:40:03.264193+0000","last_unstale":"2024-09-16T01:40:05.257267+0000","last_undegraded":"2024-09-16T01:40:05.257267+0000","last_fullsized":"2024-09-16T01:40:05.257267+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:16:00.398104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.288336+0000","last_change":"2024-09-16T01:40:04.286614+0000","last_active":"2024-09-16T01:40:04.288336+0000","last_peered":"2024-09-16T01:40:04.288336+0000","last_clean":"2024-09-16T01:40:04.288336+0000","last_became_active":"2024-09-16T01:40:04.286362+0000","last_became_peered":"2024-09-16T01:40:04.286362+0000","last_unstale":"2024-09-16T01:40:04.288336+0000","last_undegraded":"2024-09-16T01:40:04.288336+0000","last_fullsized":"2024-09-16T01:40:04.288336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:53:13.515186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"84'1","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.284670+0000","last_change":"2024-09-16T01:40:03.264527+0000","last_active":"2024-09-16T01:40:04.284670+0000","last_peered":"2024-09-16T01:40:04.284670+0000","last_clean":"2024-09-16T01:40:04.284670+0000","last_became_active":"2024-09-16T01:40:03.264233+0000","last_became_peered":"2024-09-16T01:40:03.264233+0000","last_unstale":"2024-09-16T01:40:04.284670+0000","last_undegraded":"2024-09-16T01:40:04.284670+0000","last_fullsized":"2024-09-16T01:40:04.284670+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:47:23.618181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.286205+0000","last_change":"2024-09-16T01:40:04.284494+0000","last_active":"2024-09-16T01:40:04.286205+0000","last_peered":"2024-09-16T01:40:04.286205+0000","last_clean":"2024-09-16T01:40:04.286205+0000","last_became_active":"2024-09-16T01:40:04.284222+0000","last_became_peered":"2024-09-16T01:40:04.284222+0000","last_unstale":"2024-09-16T01:40:04.286205+0000","last_undegraded":"2024-09-16T01:40:04.286205+0000","last_fullsized":"2024-09-16T01:40:04.286205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:02:29.794001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"84'1","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.282687+0000","last_change":"2024-09-16T01:40:03.263711+0000","last_active":"2024-09-16T01:40:04.282687+0000","last_peered":"2024-09-16T01:40:04.282687+0000","last_clean":"2024-09-16T01:40:04.282687+0000","last_became_active":"2024-09-16T01:40:03.263408+0000","last_became_peered":"2024-09-16T01:40:03.263408+0000","last_unstale":"2024-09-16T01:40:04.282687+0000","last_undegraded":"2024-09-16T01:40:04.282687+0000","last_fullsized":"2024-09-16T01:40:04.282687+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:47:16.451717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256073+0000","last_change":"2024-09-16T01:40:04.263307+0000","last_active":"2024-09-16T01:40:05.256073+0000","last_peered":"2024-09-16T01:40:05.256073+0000","last_clean":"2024-09-16T01:40:05.256073+0000","last_became_active":"2024-09-16T01:40:04.263019+0000","last_became_peered":"2024-09-16T01:40:04.263019+0000","last_unstale":"2024-09-16T01:40:05.256073+0000","last_undegraded":"2024-09-16T01:40:05.256073+0000","last_fullsized":"2024-09-16T01:40:05.256073+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:31:13.781556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.257058+0000","last_change":"2024-09-16T01:40:03.263700+0000","last_active":"2024-09-16T01:40:05.257058+0000","last_peered":"2024-09-16T01:40:05.257058+0000","last_clean":"2024-09-16T01:40:05.257058+0000","last_became_active":"2024-09-16T01:40:03.263397+0000","last_became_peered":"2024-09-16T01:40:03.263397+0000","last_unstale":"2024-09-16T01:40:05.257058+0000","last_undegraded":"2024-09-16T01:40:05.257058+0000","last_fullsized":"2024-09-16T01:40:05.257058+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:20:42.840859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.267257+0000","last_change":"2024-09-16T01:40:04.265941+0000","last_active":"2024-09-16T01:40:04.267257+0000","last_peered":"2024-09-16T01:40:04.267257+0000","last_clean":"2024-09-16T01:40:04.267257+0000","last_became_active":"2024-09-16T01:40:04.265681+0000","last_became_peered":"2024-09-16T01:40:04.265681+0000","last_unstale":"2024-09-16T01:40:04.267257+0000","last_undegraded":"2024-09-16T01:40:04.267257+0000","last_fullsized":"2024-09-16T01:40:04.267257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:52:40.505962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.263755+0000","last_change":"2024-09-16T01:40:03.268669+0000","last_active":"2024-09-16T01:40:04.263755+0000","last_peered":"2024-09-16T01:40:04.263755+0000","last_clean":"2024-09-16T01:40:04.263755+0000","last_became_active":"2024-09-16T01:40:03.268192+0000","last_became_peered":"2024-09-16T01:40:03.268192+0000","last_unstale":"2024-09-16T01:40:04.263755+0000","last_undegraded":"2024-09-16T01:40:04.263755+0000","last_fullsized":"2024-09-16T01:40:04.263755+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:59:36.722101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.268032+0000","last_change":"2024-09-16T01:40:04.266898+0000","last_active":"2024-09-16T01:40:04.268032+0000","last_peered":"2024-09-16T01:40:04.268032+0000","last_clean":"2024-09-16T01:40:04.268032+0000","last_became_active":"2024-09-16T01:40:04.266584+0000","last_became_peered":"2024-09-16T01:40:04.266584+0000","last_unstale":"2024-09-16T01:40:04.268032+0000","last_undegraded":"2024-09-16T01:40:04.268032+0000","last_fullsized":"2024-09-16T01:40:04.268032+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:34:02.972536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256968+0000","last_change":"2024-09-16T01:40:03.264902+0000","last_active":"2024-09-16T01:40:05.256968+0000","last_peered":"2024-09-16T01:40:05.256968+0000","last_clean":"2024-09-16T01:40:05.256968+0000","last_became_active":"2024-09-16T01:40:03.264485+0000","last_became_peered":"2024-09-16T01:40:03.264485+0000","last_unstale":"2024-09-16T01:40:05.256968+0000","last_undegraded":"2024-09-16T01:40:05.256968+0000","last_fullsized":"2024-09-16T01:40:05.256968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:57.117123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.284295+0000","last_change":"2024-09-16T01:40:04.283561+0000","last_active":"2024-09-16T01:40:04.284295+0000","last_peered":"2024-09-16T01:40:04.284295+0000","last_clean":"2024-09-16T01:40:04.284295+0000","last_became_active":"2024-09-16T01:40:04.283238+0000","last_became_peered":"2024-09-16T01:40:04.283238+0000","last_unstale":"2024-09-16T01:40:04.284295+0000","last_undegraded":"2024-09-16T01:40:04.284295+0000","last_fullsized":"2024-09-16T01:40:04.284295+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:23:38.464038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256444+0000","last_change":"2024-09-16T01:40:03.268499+0000","last_active":"2024-09-16T01:40:05.256444+0000","last_peered":"2024-09-16T01:40:05.256444+0000","last_clean":"2024-09-16T01:40:05.256444+0000","last_became_active":"2024-09-16T01:40:03.268248+0000","last_became_peered":"2024-09-16T01:40:03.268248+0000","last_unstale":"2024-09-16T01:40:05.256444+0000","last_undegraded":"2024-09-16T01:40:05.256444+0000","last_fullsized":"2024-09-16T01:40:05.256444+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:37.909549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.257409+0000","last_change":"2024-09-16T01:40:04.267674+0000","last_active":"2024-09-16T01:40:05.257409+0000","last_peered":"2024-09-16T01:40:05.257409+0000","last_clean":"2024-09-16T01:40:05.257409+0000","last_became_active":"2024-09-16T01:40:04.267407+0000","last_became_peered":"2024-09-16T01:40:04.267407+0000","last_unstale":"2024-09-16T01:40:05.257409+0000","last_undegraded":"2024-09-16T01:40:05.257409+0000","last_fullsized":"2024-09-16T01:40:05.257409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:14:45.631031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"84'3","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.255743+0000","last_change":"2024-09-16T01:40:03.271207+0000","last_active":"2024-09-16T01:40:05.255743+0000","last_peered":"2024-09-16T01:40:05.255743+0000","last_clean":"2024-09-16T01:40:05.255743+0000","last_became_active":"2024-09-16T01:40:03.270865+0000","last_became_peered":"2024-09-16T01:40:03.270865+0000","last_unstale":"2024-09-16T01:40:05.255743+0000","last_undegraded":"2024-09-16T01:40:05.255743+0000","last_fullsized":"2024-09-16T01:40:05.255743+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:02:22.041889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.286147+0000","last_change":"2024-09-16T01:40:04.284154+0000","last_active":"2024-09-16T01:40:04.286147+0000","last_peered":"2024-09-16T01:40:04.286147+0000","last_clean":"2024-09-16T01:40:04.286147+0000","last_became_active":"2024-09-16T01:40:04.283815+0000","last_became_peered":"2024-09-16T01:40:04.283815+0000","last_unstale":"2024-09-16T01:40:04.286147+0000","last_undegraded":"2024-09-16T01:40:04.286147+0000","last_fullsized":"2024-09-16T01:40:04.286147+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:32.141921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"84'6","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.287490+0000","last_change":"2024-09-16T01:40:03.262909+0000","last_active":"2024-09-16T01:40:04.287490+0000","last_peered":"2024-09-16T01:40:04.287490+0000","last_clean":"2024-09-16T01:40:04.287490+0000","last_became_active":"2024-09-16T01:40:03.262600+0000","last_became_peered":"2024-09-16T01:40:03.262600+0000","last_unstale":"2024-09-16T01:40:04.287490+0000","last_undegraded":"2024-09-16T01:40:04.287490+0000","last_fullsized":"2024-09-16T01:40:04.287490+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:21:07.670452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.257079+0000","last_change":"2024-09-16T01:40:04.270582+0000","last_active":"2024-09-16T01:40:05.257079+0000","last_peered":"2024-09-16T01:40:05.257079+0000","last_clean":"2024-09-16T01:40:05.257079+0000","last_became_active":"2024-09-16T01:40:04.270277+0000","last_became_peered":"2024-09-16T01:40:04.270277+0000","last_unstale":"2024-09-16T01:40:05.257079+0000","last_undegraded":"2024-09-16T01:40:05.257079+0000","last_fullsized":"2024-09-16T01:40:05.257079+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:56:05.602743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"84'2","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.283409+0000","last_change":"2024-09-16T01:40:03.262692+0000","last_active":"2024-09-16T01:40:04.283409+0000","last_peered":"2024-09-16T01:40:04.283409+0000","last_clean":"2024-09-16T01:40:04.283409+0000","last_became_active":"2024-09-16T01:40:03.262426+0000","last_became_peered":"2024-09-16T01:40:03.262426+0000","last_unstale":"2024-09-16T01:40:04.283409+0000","last_undegraded":"2024-09-16T01:40:04.283409+0000","last_fullsized":"2024-09-16T01:40:04.283409+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:05:00.079589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256140+0000","last_change":"2024-09-16T01:40:04.269975+0000","last_active":"2024-09-16T01:40:05.256140+0000","last_peered":"2024-09-16T01:40:05.256140+0000","last_clean":"2024-09-16T01:40:05.256140+0000","last_became_active":"2024-09-16T01:40:04.269079+0000","last_became_peered":"2024-09-16T01:40:04.269079+0000","last_unstale":"2024-09-16T01:40:05.256140+0000","last_undegraded":"2024-09-16T01:40:05.256140+0000","last_fullsized":"2024-09-16T01:40:05.256140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:45:16.399868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.256500+0000","last_change":"2024-09-16T01:40:03.267797+0000","last_active":"2024-09-16T01:40:04.256500+0000","last_peered":"2024-09-16T01:40:04.256500+0000","last_clean":"2024-09-16T01:40:04.256500+0000","last_became_active":"2024-09-16T01:40:03.267546+0000","last_became_peered":"2024-09-16T01:40:03.267546+0000","last_unstale":"2024-09-16T01:40:04.256500+0000","last_undegraded":"2024-09-16T01:40:04.256500+0000","last_fullsized":"2024-09-16T01:40:04.256500+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:07:12.025468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.288216+0000","last_change":"2024-09-16T01:40:04.284411+0000","last_active":"2024-09-16T01:40:04.288216+0000","last_peered":"2024-09-16T01:40:04.288216+0000","last_clean":"2024-09-16T01:40:04.288216+0000","last_became_active":"2024-09-16T01:40:04.284159+0000","last_became_peered":"2024-09-16T01:40:04.284159+0000","last_unstale":"2024-09-16T01:40:04.288216+0000","last_undegraded":"2024-09-16T01:40:04.288216+0000","last_fullsized":"2024-09-16T01:40:04.288216+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:00:47.244592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.262661+0000","last_change":"2024-09-16T01:40:03.268017+0000","last_active":"2024-09-16T01:40:04.262661+0000","last_peered":"2024-09-16T01:40:04.262661+0000","last_clean":"2024-09-16T01:40:04.262661+0000","last_became_active":"2024-09-16T01:40:03.267743+0000","last_became_peered":"2024-09-16T01:40:03.267743+0000","last_unstale":"2024-09-16T01:40:04.262661+0000","last_undegraded":"2024-09-16T01:40:04.262661+0000","last_fullsized":"2024-09-16T01:40:04.262661+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:39:52.543069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.276882+0000","last_change":"2024-09-16T01:40:04.273684+0000","last_active":"2024-09-16T01:40:04.276882+0000","last_peered":"2024-09-16T01:40:04.276882+0000","last_clean":"2024-09-16T01:40:04.276882+0000","last_became_active":"2024-09-16T01:40:04.273447+0000","last_became_peered":"2024-09-16T01:40:04.273447+0000","last_unstale":"2024-09-16T01:40:04.276882+0000","last_undegraded":"2024-09-16T01:40:04.276882+0000","last_fullsized":"2024-09-16T01:40:04.276882+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:21.486329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"84'8","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256968+0000","last_change":"2024-09-16T01:40:03.267394+0000","last_active":"2024-09-16T01:40:05.256968+0000","last_peered":"2024-09-16T01:40:05.256968+0000","last_clean":"2024-09-16T01:40:05.256968+0000","last_became_active":"2024-09-16T01:40:03.267102+0000","last_became_peered":"2024-09-16T01:40:03.267102+0000","last_unstale":"2024-09-16T01:40:05.256968+0000","last_undegraded":"2024-09-16T01:40:05.256968+0000","last_fullsized":"2024-09-16T01:40:05.256968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:26:31.689093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.255936+0000","last_change":"2024-09-16T01:40:04.270698+0000","last_active":"2024-09-16T01:40:05.255936+0000","last_peered":"2024-09-16T01:40:05.255936+0000","last_clean":"2024-09-16T01:40:05.255936+0000","last_became_active":"2024-09-16T01:40:04.270413+0000","last_became_peered":"2024-09-16T01:40:04.270413+0000","last_unstale":"2024-09-16T01:40:05.255936+0000","last_undegraded":"2024-09-16T01:40:05.255936+0000","last_fullsized":"2024-09-16T01:40:05.255936+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:56:44.277657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"84'1","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.257046+0000","last_change":"2024-09-16T01:40:03.266140+0000","last_active":"2024-09-16T01:40:05.257046+0000","last_peered":"2024-09-16T01:40:05.257046+0000","last_clean":"2024-09-16T01:40:05.257046+0000","last_became_active":"2024-09-16T01:40:03.265680+0000","last_became_peered":"2024-09-16T01:40:03.265680+0000","last_unstale":"2024-09-16T01:40:05.257046+0000","last_undegraded":"2024-09-16T01:40:05.257046+0000","last_fullsized":"2024-09-16T01:40:05.257046+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:16:00.398104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256762+0000","last_change":"2024-09-16T01:40:04.260299+0000","last_active":"2024-09-16T01:40:05.256762+0000","last_peered":"2024-09-16T01:40:05.256762+0000","last_clean":"2024-09-16T01:40:05.256762+0000","last_became_active":"2024-09-16T01:40:04.260017+0000","last_became_peered":"2024-09-16T01:40:04.260017+0000","last_unstale":"2024-09-16T01:40:05.256762+0000","last_undegraded":"2024-09-16T01:40:05.256762+0000","last_fullsized":"2024-09-16T01:40:05.256762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:38:50.149048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"84'2","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.257301+0000","last_change":"2024-09-16T01:40:03.269449+0000","last_active":"2024-09-16T01:40:05.257301+0000","last_peered":"2024-09-16T01:40:05.257301+0000","last_clean":"2024-09-16T01:40:05.257301+0000","last_became_active":"2024-09-16T01:40:03.269187+0000","last_became_peered":"2024-09-16T01:40:03.269187+0000","last_unstale":"2024-09-16T01:40:05.257301+0000","last_undegraded":"2024-09-16T01:40:05.257301+0000","last_fullsized":"2024-09-16T01:40:05.257301+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:57.117123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.279593+0000","last_change":"2024-09-16T01:40:04.274031+0000","last_active":"2024-09-16T01:40:04.279593+0000","last_peered":"2024-09-16T01:40:04.279593+0000","last_clean":"2024-09-16T01:40:04.279593+0000","last_became_active":"2024-09-16T01:40:04.273151+0000","last_became_peered":"2024-09-16T01:40:04.273151+0000","last_unstale":"2024-09-16T01:40:04.279593+0000","last_undegraded":"2024-09-16T01:40:04.279593+0000","last_fullsized":"2024-09-16T01:40:04.279593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:21:23.407416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.257363+0000","last_change":"2024-09-16T01:40:03.265220+0000","last_active":"2024-09-16T01:40:05.257363+0000","last_peered":"2024-09-16T01:40:05.257363+0000","last_clean":"2024-09-16T01:40:05.257363+0000","last_became_active":"2024-09-16T01:40:03.264921+0000","last_became_peered":"2024-09-16T01:40:03.264921+0000","last_unstale":"2024-09-16T01:40:05.257363+0000","last_undegraded":"2024-09-16T01:40:05.257363+0000","last_fullsized":"2024-09-16T01:40:05.257363+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:11.684010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.287691+0000","last_change":"2024-09-16T01:40:04.283264+0000","last_active":"2024-09-16T01:40:04.287691+0000","last_peered":"2024-09-16T01:40:04.287691+0000","last_clean":"2024-09-16T01:40:04.287691+0000","last_became_active":"2024-09-16T01:40:04.282624+0000","last_became_peered":"2024-09-16T01:40:04.282624+0000","last_unstale":"2024-09-16T01:40:04.287691+0000","last_undegraded":"2024-09-16T01:40:04.287691+0000","last_fullsized":"2024-09-16T01:40:04.287691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:39:05.135707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"84'2","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.284428+0000","last_change":"2024-09-16T01:40:03.266541+0000","last_active":"2024-09-16T01:40:04.284428+0000","last_peered":"2024-09-16T01:40:04.284428+0000","last_clean":"2024-09-16T01:40:04.284428+0000","last_became_active":"2024-09-16T01:40:03.266278+0000","last_became_peered":"2024-09-16T01:40:03.266278+0000","last_unstale":"2024-09-16T01:40:04.284428+0000","last_undegraded":"2024-09-16T01:40:04.284428+0000","last_fullsized":"2024-09-16T01:40:04.284428+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:21:53.213917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.270441+0000","last_change":"2024-09-16T01:40:04.267979+0000","last_active":"2024-09-16T01:40:04.270441+0000","last_peered":"2024-09-16T01:40:04.270441+0000","last_clean":"2024-09-16T01:40:04.270441+0000","last_became_active":"2024-09-16T01:40:04.267674+0000","last_became_peered":"2024-09-16T01:40:04.267674+0000","last_unstale":"2024-09-16T01:40:04.270441+0000","last_undegraded":"2024-09-16T01:40:04.270441+0000","last_fullsized":"2024-09-16T01:40:04.270441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:40:37.567970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.271019+0000","last_change":"2024-09-16T01:40:03.269303+0000","last_active":"2024-09-16T01:40:04.271019+0000","last_peered":"2024-09-16T01:40:04.271019+0000","last_clean":"2024-09-16T01:40:04.271019+0000","last_became_active":"2024-09-16T01:40:03.269016+0000","last_became_peered":"2024-09-16T01:40:03.269016+0000","last_unstale":"2024-09-16T01:40:04.271019+0000","last_undegraded":"2024-09-16T01:40:04.271019+0000","last_fullsized":"2024-09-16T01:40:04.271019+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:22.141910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.289640+0000","last_change":"2024-09-16T01:40:04.283679+0000","last_active":"2024-09-16T01:40:04.289640+0000","last_peered":"2024-09-16T01:40:04.289640+0000","last_clean":"2024-09-16T01:40:04.289640+0000","last_became_active":"2024-09-16T01:40:04.283181+0000","last_became_peered":"2024-09-16T01:40:04.283181+0000","last_unstale":"2024-09-16T01:40:04.289640+0000","last_undegraded":"2024-09-16T01:40:04.289640+0000","last_fullsized":"2024-09-16T01:40:04.289640+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:40:30.217611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.266347+0000","last_change":"2024-09-16T01:40:03.267916+0000","last_active":"2024-09-16T01:40:04.266347+0000","last_peered":"2024-09-16T01:40:04.266347+0000","last_clean":"2024-09-16T01:40:04.266347+0000","last_became_active":"2024-09-16T01:40:03.267576+0000","last_became_peered":"2024-09-16T01:40:03.267576+0000","last_unstale":"2024-09-16T01:40:04.266347+0000","last_undegraded":"2024-09-16T01:40:04.266347+0000","last_fullsized":"2024-09-16T01:40:04.266347+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:46:17.773706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.281312+0000","last_change":"2024-09-16T01:40:04.279810+0000","last_active":"2024-09-16T01:40:04.281312+0000","last_peered":"2024-09-16T01:40:04.281312+0000","last_clean":"2024-09-16T01:40:04.281312+0000","last_became_active":"2024-09-16T01:40:04.279547+0000","last_became_peered":"2024-09-16T01:40:04.279547+0000","last_unstale":"2024-09-16T01:40:04.281312+0000","last_undegraded":"2024-09-16T01:40:04.281312+0000","last_fullsized":"2024-09-16T01:40:04.281312+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:12:25.121393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.255965+0000","last_change":"2024-09-16T01:40:03.266456+0000","last_active":"2024-09-16T01:40:04.255965+0000","last_peered":"2024-09-16T01:40:04.255965+0000","last_clean":"2024-09-16T01:40:04.255965+0000","last_became_active":"2024-09-16T01:40:03.266182+0000","last_became_peered":"2024-09-16T01:40:03.266182+0000","last_unstale":"2024-09-16T01:40:04.255965+0000","last_undegraded":"2024-09-16T01:40:04.255965+0000","last_fullsized":"2024-09-16T01:40:04.255965+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:38:08.121960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.279268+0000","last_change":"2024-09-16T01:40:04.271156+0000","last_active":"2024-09-16T01:40:04.279268+0000","last_peered":"2024-09-16T01:40:04.279268+0000","last_clean":"2024-09-16T01:40:04.279268+0000","last_became_active":"2024-09-16T01:40:04.270887+0000","last_became_peered":"2024-09-16T01:40:04.270887+0000","last_unstale":"2024-09-16T01:40:04.279268+0000","last_undegraded":"2024-09-16T01:40:04.279268+0000","last_fullsized":"2024-09-16T01:40:04.279268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:37:58.119415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"84'2","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.278751+0000","last_change":"2024-09-16T01:40:03.267643+0000","last_active":"2024-09-16T01:40:04.278751+0000","last_peered":"2024-09-16T01:40:04.278751+0000","last_clean":"2024-09-16T01:40:04.278751+0000","last_became_active":"2024-09-16T01:40:03.267277+0000","last_became_peered":"2024-09-16T01:40:03.267277+0000","last_unstale":"2024-09-16T01:40:04.278751+0000","last_undegraded":"2024-09-16T01:40:04.278751+0000","last_fullsized":"2024-09-16T01:40:04.278751+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:11:49.796239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256674+0000","last_change":"2024-09-16T01:40:04.259552+0000","last_active":"2024-09-16T01:40:05.256674+0000","last_peered":"2024-09-16T01:40:05.256674+0000","last_clean":"2024-09-16T01:40:05.256674+0000","last_became_active":"2024-09-16T01:40:04.259274+0000","last_became_peered":"2024-09-16T01:40:04.259274+0000","last_unstale":"2024-09-16T01:40:05.256674+0000","last_undegraded":"2024-09-16T01:40:05.256674+0000","last_fullsized":"2024-09-16T01:40:05.256674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:28:43.697016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.262748+0000","last_change":"2024-09-16T01:40:03.262794+0000","last_active":"2024-09-16T01:40:04.262748+0000","last_peered":"2024-09-16T01:40:04.262748+0000","last_clean":"2024-09-16T01:40:04.262748+0000","last_became_active":"2024-09-16T01:40:03.262515+0000","last_became_peered":"2024-09-16T01:40:03.262515+0000","last_unstale":"2024-09-16T01:40:04.262748+0000","last_undegraded":"2024-09-16T01:40:04.262748+0000","last_fullsized":"2024-09-16T01:40:04.262748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:52:40.436237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256512+0000","last_change":"2024-09-16T01:40:04.286093+0000","last_active":"2024-09-16T01:40:05.256512+0000","last_peered":"2024-09-16T01:40:05.256512+0000","last_clean":"2024-09-16T01:40:05.256512+0000","last_became_active":"2024-09-16T01:40:04.285766+0000","last_became_peered":"2024-09-16T01:40:04.285766+0000","last_unstale":"2024-09-16T01:40:05.256512+0000","last_undegraded":"2024-09-16T01:40:05.256512+0000","last_fullsized":"2024-09-16T01:40:05.256512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:34:02.972536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.255945+0000","last_change":"2024-09-16T01:40:03.267093+0000","last_active":"2024-09-16T01:40:05.255945+0000","last_peered":"2024-09-16T01:40:05.255945+0000","last_clean":"2024-09-16T01:40:05.255945+0000","last_became_active":"2024-09-16T01:40:03.266690+0000","last_became_peered":"2024-09-16T01:40:03.266690+0000","last_unstale":"2024-09-16T01:40:05.255945+0000","last_undegraded":"2024-09-16T01:40:05.255945+0000","last_fullsized":"2024-09-16T01:40:05.255945+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:25:06.080867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.284931+0000","last_change":"2024-09-16T01:40:04.283450+0000","last_active":"2024-09-16T01:40:04.284931+0000","last_peered":"2024-09-16T01:40:04.284931+0000","last_clean":"2024-09-16T01:40:04.284931+0000","last_became_active":"2024-09-16T01:40:04.283099+0000","last_became_peered":"2024-09-16T01:40:04.283099+0000","last_unstale":"2024-09-16T01:40:04.284931+0000","last_undegraded":"2024-09-16T01:40:04.284931+0000","last_fullsized":"2024-09-16T01:40:04.284931+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:56:42.591248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256602+0000","last_change":"2024-09-16T01:40:03.264409+0000","last_active":"2024-09-16T01:40:05.256602+0000","last_peered":"2024-09-16T01:40:05.256602+0000","last_clean":"2024-09-16T01:40:05.256602+0000","last_became_active":"2024-09-16T01:40:03.263982+0000","last_became_peered":"2024-09-16T01:40:03.263982+0000","last_unstale":"2024-09-16T01:40:05.256602+0000","last_undegraded":"2024-09-16T01:40:05.256602+0000","last_fullsized":"2024-09-16T01:40:05.256602+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:48.392893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.270361+0000","last_change":"2024-09-16T01:40:04.269189+0000","last_active":"2024-09-16T01:40:04.270361+0000","last_peered":"2024-09-16T01:40:04.270361+0000","last_clean":"2024-09-16T01:40:04.270361+0000","last_became_active":"2024-09-16T01:40:04.268898+0000","last_became_peered":"2024-09-16T01:40:04.268898+0000","last_unstale":"2024-09-16T01:40:04.270361+0000","last_undegraded":"2024-09-16T01:40:04.270361+0000","last_fullsized":"2024-09-16T01:40:04.270361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:52:31.738742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.257332+0000","last_change":"2024-09-16T01:40:03.271013+0000","last_active":"2024-09-16T01:40:05.257332+0000","last_peered":"2024-09-16T01:40:05.257332+0000","last_clean":"2024-09-16T01:40:05.257332+0000","last_became_active":"2024-09-16T01:40:03.270739+0000","last_became_peered":"2024-09-16T01:40:03.270739+0000","last_unstale":"2024-09-16T01:40:05.257332+0000","last_undegraded":"2024-09-16T01:40:05.257332+0000","last_fullsized":"2024-09-16T01:40:05.257332+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:21:22.405123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.257247+0000","last_change":"2024-09-16T01:40:04.272964+0000","last_active":"2024-09-16T01:40:05.257247+0000","last_peered":"2024-09-16T01:40:05.257247+0000","last_clean":"2024-09-16T01:40:05.257247+0000","last_became_active":"2024-09-16T01:40:04.272717+0000","last_became_peered":"2024-09-16T01:40:04.272717+0000","last_unstale":"2024-09-16T01:40:05.257247+0000","last_undegraded":"2024-09-16T01:40:05.257247+0000","last_fullsized":"2024-09-16T01:40:05.257247+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:30:25.714992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.260302+0000","last_change":"2024-09-16T01:40:03.264491+0000","last_active":"2024-09-16T01:40:04.260302+0000","last_peered":"2024-09-16T01:40:04.260302+0000","last_clean":"2024-09-16T01:40:04.260302+0000","last_became_active":"2024-09-16T01:40:03.264185+0000","last_became_peered":"2024-09-16T01:40:03.264185+0000","last_unstale":"2024-09-16T01:40:04.260302+0000","last_undegraded":"2024-09-16T01:40:04.260302+0000","last_fullsized":"2024-09-16T01:40:04.260302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:02:22.041889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.255717+0000","last_change":"2024-09-16T01:40:04.265646+0000","last_active":"2024-09-16T01:40:05.255717+0000","last_peered":"2024-09-16T01:40:05.255717+0000","last_clean":"2024-09-16T01:40:05.255717+0000","last_became_active":"2024-09-16T01:40:04.265257+0000","last_became_peered":"2024-09-16T01:40:04.265257+0000","last_unstale":"2024-09-16T01:40:05.255717+0000","last_undegraded":"2024-09-16T01:40:05.255717+0000","last_fullsized":"2024-09-16T01:40:05.255717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:52:48.110074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.255980+0000","last_change":"2024-09-16T01:40:03.263473+0000","last_active":"2024-09-16T01:40:05.255980+0000","last_peered":"2024-09-16T01:40:05.255980+0000","last_clean":"2024-09-16T01:40:05.255980+0000","last_became_active":"2024-09-16T01:40:03.263206+0000","last_became_peered":"2024-09-16T01:40:03.263206+0000","last_unstale":"2024-09-16T01:40:05.255980+0000","last_undegraded":"2024-09-16T01:40:05.255980+0000","last_fullsized":"2024-09-16T01:40:05.255980+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:47:23.618181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.255891+0000","last_change":"2024-09-16T01:40:04.283274+0000","last_active":"2024-09-16T01:40:05.255891+0000","last_peered":"2024-09-16T01:40:05.255891+0000","last_clean":"2024-09-16T01:40:05.255891+0000","last_became_active":"2024-09-16T01:40:04.282915+0000","last_became_peered":"2024-09-16T01:40:04.282915+0000","last_unstale":"2024-09-16T01:40:05.255891+0000","last_undegraded":"2024-09-16T01:40:05.255891+0000","last_fullsized":"2024-09-16T01:40:05.255891+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:38:09.124252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.261118+0000","last_change":"2024-09-16T01:40:03.266672+0000","last_active":"2024-09-16T01:40:04.261118+0000","last_peered":"2024-09-16T01:40:04.261118+0000","last_clean":"2024-09-16T01:40:04.261118+0000","last_became_active":"2024-09-16T01:40:03.266361+0000","last_became_peered":"2024-09-16T01:40:03.266361+0000","last_unstale":"2024-09-16T01:40:04.261118+0000","last_undegraded":"2024-09-16T01:40:04.261118+0000","last_fullsized":"2024-09-16T01:40:04.261118+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:02:37.909549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-16T01:40:04.275664+0000","last_change":"2024-09-16T01:40:04.268702+0000","last_active":"2024-09-16T01:40:04.275664+0000","last_peered":"2024-09-16T01:40:04.275664+0000","last_clean":"2024-09-16T01:40:04.275664+0000","last_became_active":"2024-09-16T01:40:04.268402+0000","last_became_peered":"2024-09-16T01:40:04.268402+0000","last_unstale":"2024-09-16T01:40:04.275664+0000","last_undegraded":"2024-09-16T01:40:04.275664+0000","last_fullsized":"2024-09-16T01:40:04.275664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:03.245130+0000","last_clean_scrub_stamp":"2024-09-16T01:40:03.245130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:34:32.842050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"84'1","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.256431+0000","last_change":"2024-09-16T01:40:03.273312+0000","last_active":"2024-09-16T01:40:05.256431+0000","last_peered":"2024-09-16T01:40:05.256431+0000","last_clean":"2024-09-16T01:40:05.256431+0000","last_became_active":"2024-09-16T01:40:03.272995+0000","last_became_peered":"2024-09-16T01:40:03.272995+0000","last_unstale":"2024-09-16T01:40:05.256431+0000","last_undegraded":"2024-09-16T01:40:05.256431+0000","last_fullsized":"2024-09-16T01:40:05.256431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:40:02.242837+0000","last_clean_scrub_stamp":"2024-09-16T01:40:02.242837+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:23:37.461745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":286,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-16T01:40:05.255844+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:40:05.255844+0000","last_peered":"2024-09-16T01:40:05.255844+0000","last_clean":"2024-09-16T01:40:05.255844+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:40:05.255844+0000","last_undegraded":"2024-09-16T01:40:05.255844+0000","last_fullsized":"2024-09-16T01:40:05.255844+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1044480,"data_stored":440156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542219,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32780,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339060,"statfs":{"total":96636764160,"available":96603197440,"internally_reserved":0,"allocated":1626112,"data_stored":1028960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28060,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343780,"statfs":{"total":96636764160,"available":96608030720,"internally_reserved":0,"allocated":1052672,"data_stored":445516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542219,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28032,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343808,"statfs":{"total":96636764160,"available":96608059392,"internally_reserved":0,"allocated":1028096,"data_stored":438522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542218,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28048,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343792,"statfs":{"total":96636764160,"available":96608043008,"internally_reserved":0,"allocated":1040384,"data_stored":440134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542219,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28036,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343804,"statfs":{"total":96636764160,"available":96608055296,"internally_reserved":0,"allocated":1032192,"data_stored":439104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542218,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":1040384,"data_stored":438716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542218,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32796,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339044,"statfs":{"total":96636764160,"available":96603181056,"internally_reserved":0,"allocated":1638400,"data_stored":1035224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:40:07.541 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T01:40:07.541 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-16T01:40:07.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.681+0000 7f91dfdc4640 1 Processor -- start 2024-09-16T01:40:07.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.682+0000 7f91dfdc4640 1 -- start start 2024-09-16T01:40:07.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.682+0000 7f91dfdc4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f91d806fc30 0x7f91d80734b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:07.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.682+0000 7f91dfdc4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91d80739f0 con 0x7f91d80ae890 2024-09-16T01:40:07.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.682+0000 7f91dfdc4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91d80717c0 con 0x7f91d80730d0 2024-09-16T01:40:07.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.682+0000 7f91dfdc4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f91d8071aa0 con 0x7f91d806fc30 2024-09-16T01:40:07.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.683+0000 7f91de361640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f91d80ae890 0x7f91d80aeca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33318/0 (socket says 172.21.15.88:33318) 2024-09-16T01:40:07.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.683+0000 7f91de361640 1 -- 172.21.15.88:0/2056455894 learned_addr learned my addr 172.21.15.88:0/2056455894 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:07.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.683+0000 7f91dd35f640 1 --2- 172.21.15.88:0/2056455894 >> v2:172.21.15.154:3300/0 conn(0x7f91d806fc30 0x7f91d80734b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:07.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.683+0000 7f91dcb5e640 1 -- 172.21.15.88:0/2056455894 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 437195015 0 0) 0x7f91d80739f0 con 0x7f91d80ae890 2024-09-16T01:40:07.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.683+0000 7f91dcb5e640 1 -- 172.21.15.88:0/2056455894 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91c0003520 con 0x7f91d80ae890 2024-09-16T01:40:07.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.683+0000 7f91dcb5e640 1 -- 172.21.15.88:0/2056455894 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 567337549 0 0) 0x7f91c0003520 con 0x7f91d80ae890 2024-09-16T01:40:07.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.683+0000 7f91dcb5e640 1 -- 172.21.15.88:0/2056455894 >> v2:172.21.15.154:3300/0 conn(0x7f91d806fc30 msgr2=0x7f91d80734b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:07.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.683+0000 7f91dcb5e640 1 --2- 172.21.15.88:0/2056455894 >> v2:172.21.15.154:3300/0 conn(0x7f91d806fc30 0x7f91d80734b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.683+0000 7f91dcb5e640 1 -- 172.21.15.88:0/2056455894 >> v1:172.21.15.154:6789/0 conn(0x7f91d80730d0 legacy=0x7f91d815f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:07.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.683+0000 7f91dcb5e640 1 -- 172.21.15.88:0/2056455894 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91d8071d80 con 0x7f91d80ae890 2024-09-16T01:40:07.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.683+0000 7f91dcb5e640 1 -- 172.21.15.88:0/2056455894 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f91d0001670 con 0x7f91d80ae890 2024-09-16T01:40:07.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.683+0000 7f91dcb5e640 1 -- 172.21.15.88:0/2056455894 >> v1:172.21.15.88:6789/0 conn(0x7f91d80ae890 legacy=0x7f91d80aeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:07.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.683+0000 7f91dcb5e640 1 --2- 172.21.15.88:0/2056455894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91c00039b0 0x7f91c0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:07.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.683+0000 7f91dcb5e640 1 --2- 172.21.15.88:0/2056455894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91c0004800 0x7f91c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:07.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.683+0000 7f91dcb5e640 1 --2- 172.21.15.88:0/2056455894 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91c00051c0 0x7f91c000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:07.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.683+0000 7f91dcb5e640 1 -- 172.21.15.88:0/2056455894 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f91d8071aa0 con 0x7f91c0004800 2024-09-16T01:40:07.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.683+0000 7f91dcb5e640 1 -- 172.21.15.88:0/2056455894 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f91c000c060 con 0x7f91c00039b0 2024-09-16T01:40:07.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.683+0000 7f91dcb5e640 1 -- 172.21.15.88:0/2056455894 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f91c000c260 con 0x7f91c00051c0 2024-09-16T01:40:07.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.683+0000 7f91dd35f640 1 --2- 172.21.15.88:0/2056455894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91c0004800 0x7f91c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:07.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.683+0000 7f91ddb60640 1 --2- 172.21.15.88:0/2056455894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91c00039b0 0x7f91c0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:07.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.683+0000 7f91de361640 1 --2- 172.21.15.88:0/2056455894 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91c00051c0 0x7f91c000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:07.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.684+0000 7f91dd35f640 1 -- 172.21.15.88:0/2056455894 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91c00051c0 msgr2=0x7f91c000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:07.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.684+0000 7f91dd35f640 1 --2- 172.21.15.88:0/2056455894 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91c00051c0 0x7f91c000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.684+0000 7f91dd35f640 1 -- 172.21.15.88:0/2056455894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91c00039b0 msgr2=0x7f91c0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:07.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.684+0000 7f91dd35f640 1 --2- 172.21.15.88:0/2056455894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91c00039b0 0x7f91c0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.684+0000 7f91dd35f640 1 -- 172.21.15.88:0/2056455894 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91c0003520 con 0x7f91c0004800 2024-09-16T01:40:07.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.684+0000 7f91de361640 1 -- 172.21.15.88:0/2056455894 reap_dead start 2024-09-16T01:40:07.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.684+0000 7f91dd35f640 1 --2- 172.21.15.88:0/2056455894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91c0004800 0x7f91c0004c80 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f91cc009ac0 tx=0x7f91c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cd4127cbac5697c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:07.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.684+0000 7f91dcb5e640 1 -- 172.21.15.88:0/2056455894 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91cc036070 con 0x7f91c0004800 2024-09-16T01:40:07.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.684+0000 7f91dcb5e640 1 -- 172.21.15.88:0/2056455894 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91cc031400 con 0x7f91c0004800 2024-09-16T01:40:07.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.684+0000 7f91dcb5e640 1 -- 172.21.15.88:0/2056455894 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91cc009690 con 0x7f91c0004800 2024-09-16T01:40:07.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.685+0000 7f91dfdc4640 1 -- 172.21.15.88:0/2056455894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91c0004800 msgr2=0x7f91c0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:07.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.685+0000 7f91dfdc4640 1 --2- 172.21.15.88:0/2056455894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91c0004800 0x7f91c0004c80 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f91cc009ac0 tx=0x7f91c0005830 comp rx=0 tx=0).stop 2024-09-16T01:40:07.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.685+0000 7f91dfdc4640 1 -- 172.21.15.88:0/2056455894 shutdown_connections 2024-09-16T01:40:07.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.685+0000 7f91dfdc4640 1 --2- 172.21.15.88:0/2056455894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91c0004800 0x7f91c0004c80 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.685+0000 7f91dfdc4640 1 -- 172.21.15.88:0/2056455894 >> 172.21.15.88:0/2056455894 conn(0x7f91d80b0aa0 msgr2=0x7f91d80b0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:07.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.685+0000 7f91dfdc4640 1 -- 172.21.15.88:0/2056455894 shutdown_connections 2024-09-16T01:40:07.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.685+0000 7f91dfdc4640 1 -- 172.21.15.88:0/2056455894 wait complete. 2024-09-16T01:40:07.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.686+0000 7f91dfdc4640 1 Processor -- start 2024-09-16T01:40:07.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.686+0000 7f91dfdc4640 1 -- start start 2024-09-16T01:40:07.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.687+0000 7f91dfdc4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91c0004800 0x7f91d8151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:07.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.687+0000 7f91dfdc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91d815e4e0 0x7f91d8152140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:07.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.687+0000 7f91dfdc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91d8153b70 0x7f91d8153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:07.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.687+0000 7f91dfdc4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f91d8159170 con 0x7f91c0004800 2024-09-16T01:40:07.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.687+0000 7f91dfdc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f91d8072340 con 0x7f91d815e4e0 2024-09-16T01:40:07.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.687+0000 7f91dfdc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f91d8072860 con 0x7f91d8153b70 2024-09-16T01:40:07.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.687+0000 7f91ddb60640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91c0004800 0x7f91d8151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:07.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.687+0000 7f91ddb60640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91c0004800 0x7f91d8151b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60092/0 (socket says 172.21.15.88:60092) 2024-09-16T01:40:07.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.687+0000 7f91ddb60640 1 -- 172.21.15.88:0/212448700 learned_addr learned my addr 172.21.15.88:0/212448700 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:07.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.687+0000 7f91de361640 1 --2- 172.21.15.88:0/212448700 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91d8153b70 0x7f91d8153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:07.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.687+0000 7f91dd35f640 1 --2- 172.21.15.88:0/212448700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91d815e4e0 0x7f91d8152140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:07.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.687+0000 7f91ddb60640 1 -- 172.21.15.88:0/212448700 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91d8153b70 msgr2=0x7f91d8153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:07.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.687+0000 7f91ddb60640 1 --2- 172.21.15.88:0/212448700 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91d8153b70 0x7f91d8153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.687+0000 7f91ddb60640 1 -- 172.21.15.88:0/212448700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91d815e4e0 msgr2=0x7f91d8152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:07.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.687+0000 7f91ddb60640 1 --2- 172.21.15.88:0/212448700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91d815e4e0 0x7f91d8152140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.687+0000 7f91ddb60640 1 -- 172.21.15.88:0/212448700 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91d81528b0 con 0x7f91c0004800 2024-09-16T01:40:07.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.687+0000 7f91dd35f640 1 --2- 172.21.15.88:0/212448700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91d815e4e0 0x7f91d8152140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:07.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.687+0000 7f91ddb60640 1 --2- 172.21.15.88:0/212448700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91c0004800 0x7f91d8151b00 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f91c800fbf0 tx=0x7f91c800d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:07.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.688+0000 7f91d67fc640 1 -- 172.21.15.88:0/212448700 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c800dbd0 con 0x7f91c0004800 2024-09-16T01:40:07.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.688+0000 7f91d67fc640 1 -- 172.21.15.88:0/212448700 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91c8018070 con 0x7f91c0004800 2024-09-16T01:40:07.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.688+0000 7f91dfdc4640 1 -- 172.21.15.88:0/212448700 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91d8152b90 con 0x7f91c0004800 2024-09-16T01:40:07.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.688+0000 7f91d67fc640 1 -- 172.21.15.88:0/212448700 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c8003970 con 0x7f91c0004800 2024-09-16T01:40:07.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.688+0000 7f91dfdc4640 1 -- 172.21.15.88:0/212448700 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91d80cd8b0 con 0x7f91c0004800 2024-09-16T01:40:07.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.689+0000 7f91dfdc4640 1 -- 172.21.15.88:0/212448700 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91d80734b0 con 0x7f91c0004800 2024-09-16T01:40:07.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.689+0000 7f91d67fc640 1 -- 172.21.15.88:0/212448700 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f91c8003b10 con 0x7f91c0004800 2024-09-16T01:40:07.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.690+0000 7f91d67fc640 1 --2- 172.21.15.88:0/212448700 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f91c406c2b0 0x7f91c406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:07.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.690+0000 7f91d67fc640 1 -- 172.21.15.88:0/212448700 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f91c808b4d0 con 0x7f91c0004800 2024-09-16T01:40:07.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.690+0000 7f91dd35f640 1 --2- 172.21.15.88:0/212448700 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f91c406c2b0 0x7f91c406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:07.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.690+0000 7f91dd35f640 1 --2- 172.21.15.88:0/212448700 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f91c406c2b0 0x7f91c406e770 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f91cc033000 tx=0x7f91cc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:07.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.694+0000 7f91d67fc640 1 -- 172.21.15.88:0/212448700 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f91c8059cd0 con 0x7f91c0004800 2024-09-16T01:40:07.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.922+0000 7f91dfdc4640 1 -- 172.21.15.88:0/212448700 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f91d80d0af0 con 0x7f91c0004800 2024-09-16T01:40:07.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.924+0000 7f91d67fc640 1 -- 172.21.15.88:0/212448700 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) ==== 190+0+147 (secure 0 0 0) 0x7f91c8089e20 con 0x7f91c0004800 2024-09-16T01:40:07.925 INFO:teuthology.orchestra.run.smithi088.stdout:[client.0] 2024-09-16T01:40:07.925 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDeiudmvPikIBAAyHxNNnwQKygxg09PNQLc/Q== 2024-09-16T01:40:07.925 INFO:teuthology.orchestra.run.smithi088.stdout: caps mds = "allow" 2024-09-16T01:40:07.926 INFO:teuthology.orchestra.run.smithi088.stdout: caps mon = "allow r" 2024-09-16T01:40:07.926 INFO:teuthology.orchestra.run.smithi088.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-16T01:40:07.926 INFO:teuthology.orchestra.run.smithi088.stderr:updated caps for client.0 2024-09-16T01:40:07.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.927+0000 7f91dfdc4640 1 -- 172.21.15.88:0/212448700 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f91c406c2b0 msgr2=0x7f91c406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:07.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.927+0000 7f91dfdc4640 1 --2- 172.21.15.88:0/212448700 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f91c406c2b0 0x7f91c406e770 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f91cc033000 tx=0x7f91cc008030 comp rx=0 tx=0).stop 2024-09-16T01:40:07.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.927+0000 7f91dfdc4640 1 -- 172.21.15.88:0/212448700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91c0004800 msgr2=0x7f91d8151b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:07.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.927+0000 7f91dfdc4640 1 --2- 172.21.15.88:0/212448700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91c0004800 0x7f91d8151b00 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f91c800fbf0 tx=0x7f91c800d520 comp rx=0 tx=0).stop 2024-09-16T01:40:07.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.927+0000 7f91dfdc4640 1 -- 172.21.15.88:0/212448700 shutdown_connections 2024-09-16T01:40:07.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.927+0000 7f91dfdc4640 1 --2- 172.21.15.88:0/212448700 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f91c406c2b0 0x7f91c406e770 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.927+0000 7f91dfdc4640 1 --2- 172.21.15.88:0/212448700 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91d8153b70 0x7f91d8153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.927+0000 7f91dfdc4640 1 --2- 172.21.15.88:0/212448700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91d815e4e0 0x7f91d8152140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.927+0000 7f91dfdc4640 1 --2- 172.21.15.88:0/212448700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91c0004800 0x7f91d8151b00 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:07.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.927+0000 7f91dfdc4640 1 -- 172.21.15.88:0/212448700 >> 172.21.15.88:0/212448700 conn(0x7f91d80b0aa0 msgr2=0x7f91d806dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:07.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.927+0000 7f91dfdc4640 1 -- 172.21.15.88:0/212448700 shutdown_connections 2024-09-16T01:40:07.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:07.927+0000 7f91dfdc4640 1 -- 172.21.15.88:0/212448700 wait complete. 2024-09-16T01:40:07.939 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:40:08.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.036+0000 7f5082e65640 1 Processor -- start 2024-09-16T01:40:08.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.036+0000 7f5082e65640 1 -- start start 2024-09-16T01:40:08.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.037+0000 7f5082e65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507c15e540 0x7f507c0ae890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.037+0000 7f5082e65640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f507c0b52c0 con 0x7f507c06e800 2024-09-16T01:40:08.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.037+0000 7f5082e65640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f507c0af000 con 0x7f507c072d50 2024-09-16T01:40:08.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.037+0000 7f5082e65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f507c0aedd0 con 0x7f507c15e540 2024-09-16T01:40:08.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.037+0000 7f5081402640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f507c06e800 0x7f507c0b57a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33334/0 (socket says 172.21.15.88:33334) 2024-09-16T01:40:08.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.037+0000 7f5081402640 1 -- 172.21.15.88:0/2173521984 learned_addr learned my addr 172.21.15.88:0/2173521984 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:08.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.037+0000 7f5080c01640 1 --2- 172.21.15.88:0/2173521984 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507c15e540 0x7f507c0ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.037+0000 7f507b7fe640 1 -- 172.21.15.88:0/2173521984 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 19358394 0 0) 0x7f507c0b52c0 con 0x7f507c06e800 2024-09-16T01:40:08.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.037+0000 7f507b7fe640 1 -- 172.21.15.88:0/2173521984 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f506c003520 con 0x7f507c06e800 2024-09-16T01:40:08.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.037+0000 7f507b7fe640 1 -- 172.21.15.88:0/2173521984 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2850445907 0 0) 0x7f507c0af000 con 0x7f507c072d50 2024-09-16T01:40:08.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.037+0000 7f507b7fe640 1 -- 172.21.15.88:0/2173521984 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f507c0b52c0 con 0x7f507c072d50 2024-09-16T01:40:08.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.037+0000 7f507b7fe640 1 -- 172.21.15.88:0/2173521984 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2745063909 0 0) 0x7f506c003520 con 0x7f507c06e800 2024-09-16T01:40:08.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.037+0000 7f507b7fe640 1 -- 172.21.15.88:0/2173521984 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507c15e540 msgr2=0x7f507c0ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:08.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.037+0000 7f507b7fe640 1 --2- 172.21.15.88:0/2173521984 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507c15e540 0x7f507c0ae890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.037+0000 7f507b7fe640 1 -- 172.21.15.88:0/2173521984 >> v1:172.21.15.154:6789/0 conn(0x7f507c072d50 legacy=0x7f507c06e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:08.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.037+0000 7f507b7fe640 1 -- 172.21.15.88:0/2173521984 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f507c0af2e0 con 0x7f507c06e800 2024-09-16T01:40:08.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.037+0000 7f507b7fe640 1 -- 172.21.15.88:0/2173521984 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f5074001970 con 0x7f507c06e800 2024-09-16T01:40:08.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.038+0000 7f507b7fe640 1 -- 172.21.15.88:0/2173521984 >> v1:172.21.15.88:6789/0 conn(0x7f507c06e800 legacy=0x7f507c0b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:08.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.038+0000 7f507b7fe640 1 --2- 172.21.15.88:0/2173521984 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507c15e540 0x7f506c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.038+0000 7f507b7fe640 1 --2- 172.21.15.88:0/2173521984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f506c004120 0x7f506c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.038+0000 7f507b7fe640 1 --2- 172.21.15.88:0/2173521984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f506c004d40 0x7f506c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.038+0000 7f507b7fe640 1 -- 172.21.15.88:0/2173521984 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f506c00bb50 con 0x7f506c004d40 2024-09-16T01:40:08.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.038+0000 7f507b7fe640 1 -- 172.21.15.88:0/2173521984 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f506c00bd00 con 0x7f506c004120 2024-09-16T01:40:08.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.038+0000 7f507b7fe640 1 -- 172.21.15.88:0/2173521984 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f506c00bf00 con 0x7f507c15e540 2024-09-16T01:40:08.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.038+0000 7f5081402640 1 --2- 172.21.15.88:0/2173521984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f506c004d40 0x7f506c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.038+0000 7f5080c01640 1 --2- 172.21.15.88:0/2173521984 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507c15e540 0x7f506c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.038+0000 7f507bfff640 1 --2- 172.21.15.88:0/2173521984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f506c004120 0x7f506c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.038+0000 7f5081402640 1 -- 172.21.15.88:0/2173521984 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507c15e540 msgr2=0x7f506c003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:08.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.038+0000 7f5081402640 1 --2- 172.21.15.88:0/2173521984 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507c15e540 0x7f506c003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.038+0000 7f5081402640 1 -- 172.21.15.88:0/2173521984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f506c004120 msgr2=0x7f506c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:08.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.038+0000 7f5081402640 1 --2- 172.21.15.88:0/2173521984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f506c004120 0x7f506c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.038+0000 7f5081402640 1 -- 172.21.15.88:0/2173521984 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f506c003520 con 0x7f506c004d40 2024-09-16T01:40:08.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.038+0000 7f507bfff640 1 --2- 172.21.15.88:0/2173521984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f506c004120 0x7f506c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:08.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.039+0000 7f5081402640 1 --2- 172.21.15.88:0/2173521984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f506c004d40 0x7f506c0051c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f507400a6b0 tx=0x7f506c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ee0c294108aae2a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:08.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.039+0000 7f507b7fe640 1 -- 172.21.15.88:0/2173521984 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507400a2d0 con 0x7f506c004d40 2024-09-16T01:40:08.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.039+0000 7f507b7fe640 1 -- 172.21.15.88:0/2173521984 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f507402f070 con 0x7f506c004d40 2024-09-16T01:40:08.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.039+0000 7f507b7fe640 1 -- 172.21.15.88:0/2173521984 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507402bbc0 con 0x7f506c004d40 2024-09-16T01:40:08.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.039+0000 7f5082e65640 1 -- 172.21.15.88:0/2173521984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f506c004d40 msgr2=0x7f506c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:08.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.039+0000 7f5082e65640 1 --2- 172.21.15.88:0/2173521984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f506c004d40 0x7f506c0051c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f507400a6b0 tx=0x7f506c005830 comp rx=0 tx=0).stop 2024-09-16T01:40:08.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.039+0000 7f5081402640 1 -- 172.21.15.88:0/2173521984 reap_dead start 2024-09-16T01:40:08.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.039+0000 7f5082e65640 1 -- 172.21.15.88:0/2173521984 shutdown_connections 2024-09-16T01:40:08.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.039+0000 7f5082e65640 1 -- 172.21.15.88:0/2173521984 >> 172.21.15.88:0/2173521984 conn(0x7f507c06bc10 msgr2=0x7f507c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:08.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.040+0000 7f5082e65640 1 -- 172.21.15.88:0/2173521984 shutdown_connections 2024-09-16T01:40:08.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.040+0000 7f5082e65640 1 -- 172.21.15.88:0/2173521984 wait complete. 2024-09-16T01:40:08.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.041+0000 7f5082e65640 1 Processor -- start 2024-09-16T01:40:08.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.041+0000 7f5082e65640 1 -- start start 2024-09-16T01:40:08.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.042+0000 7f5082e65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f506c004d40 0x7f507c14d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.042+0000 7f5082e65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507c1f43d0 0x7f507c1e95f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.042+0000 7f5082e65640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f507c1e9b30 0x7f507c160690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.042+0000 7f5082e65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f507c15e920 con 0x7f506c004d40 2024-09-16T01:40:08.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.042+0000 7f5082e65640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f507c0af8a0 con 0x7f507c1e9b30 2024-09-16T01:40:08.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.042+0000 7f5082e65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f507c0ae890 con 0x7f507c1f43d0 2024-09-16T01:40:08.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.042+0000 7f5080c01640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f506c004d40 0x7f507c14d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.042+0000 7f5080c01640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f506c004d40 0x7f507c14d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60112/0 (socket says 172.21.15.88:60112) 2024-09-16T01:40:08.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.042+0000 7f5080c01640 1 -- 172.21.15.88:0/1446746089 learned_addr learned my addr 172.21.15.88:0/1446746089 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:08.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.042+0000 7f5081402640 1 --2- 172.21.15.88:0/1446746089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f507c1e9b30 0x7f507c160690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.042+0000 7f507bfff640 1 --2- 172.21.15.88:0/1446746089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507c1f43d0 0x7f507c1e95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.042+0000 7f5080c01640 1 -- 172.21.15.88:0/1446746089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507c1f43d0 msgr2=0x7f507c1e95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:08.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.042+0000 7f5080c01640 1 --2- 172.21.15.88:0/1446746089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507c1f43d0 0x7f507c1e95f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.042+0000 7f5080c01640 1 -- 172.21.15.88:0/1446746089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f507c1e9b30 msgr2=0x7f507c160690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:08.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.042+0000 7f5080c01640 1 --2- 172.21.15.88:0/1446746089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f507c1e9b30 0x7f507c160690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.042+0000 7f5080c01640 1 -- 172.21.15.88:0/1446746089 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f507c161430 con 0x7f506c004d40 2024-09-16T01:40:08.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.042+0000 7f5080c01640 1 --2- 172.21.15.88:0/1446746089 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f506c004d40 0x7f507c14d290 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f50700119a0 tx=0x7f5070011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:08.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.042+0000 7f50797fa640 1 -- 172.21.15.88:0/1446746089 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507000faf0 con 0x7f506c004d40 2024-09-16T01:40:08.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.042+0000 7f50797fa640 1 -- 172.21.15.88:0/1446746089 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f507000fc90 con 0x7f506c004d40 2024-09-16T01:40:08.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.042+0000 7f5082e65640 1 -- 172.21.15.88:0/1446746089 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f507c1e7060 con 0x7f506c004d40 2024-09-16T01:40:08.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.042+0000 7f50797fa640 1 -- 172.21.15.88:0/1446746089 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5070014440 con 0x7f506c004d40 2024-09-16T01:40:08.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.042+0000 7f5082e65640 1 -- 172.21.15.88:0/1446746089 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f507c1e7620 con 0x7f506c004d40 2024-09-16T01:40:08.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.043+0000 7f5082e65640 1 -- 172.21.15.88:0/1446746089 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f507c1e9f40 con 0x7f506c004d40 2024-09-16T01:40:08.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.044+0000 7f50797fa640 1 -- 172.21.15.88:0/1446746089 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5070012040 con 0x7f506c004d40 2024-09-16T01:40:08.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.044+0000 7f50797fa640 1 --2- 172.21.15.88:0/1446746089 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f506006c2b0 0x7f506006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.044+0000 7f50797fa640 1 -- 172.21.15.88:0/1446746089 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f507008bae0 con 0x7f506c004d40 2024-09-16T01:40:08.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.044+0000 7f507bfff640 1 --2- 172.21.15.88:0/1446746089 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f506006c2b0 0x7f506006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.045+0000 7f507bfff640 1 --2- 172.21.15.88:0/1446746089 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f506006c2b0 0x7f506006e770 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5068007fd0 tx=0x7f5068009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:08.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.049+0000 7f50797fa640 1 -- 172.21.15.88:0/1446746089 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f507c1e9f40 con 0x7f506c004d40 2024-09-16T01:40:08.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.245+0000 7f5082e65640 1 -- 172.21.15.88:0/1446746089 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f507c0739a0 con 0x7f506c004d40 2024-09-16T01:40:08.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.246+0000 7f50797fa640 1 -- 172.21.15.88:0/1446746089 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) ==== 94+0+5228 (secure 0 0 0) 0x7f5070019030 con 0x7f506c004d40 2024-09-16T01:40:08.247 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:08.247 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":70,"btime":"2024-09-16T01:40:06:272351+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6849,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/1289932173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1289932173},{"type":"v1","addr":"172.21.15.88:6836","nonce":1289932173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:40:05.602153+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6852":{"gid":6852,"name":"c","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6837/4162274767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4162274767},{"type":"v1","addr":"172.21.15.88:6837","nonce":4162274767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6852,"qdb_cluster":[6852]},"id":6}]} 2024-09-16T01:40:08.248 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 70 2024-09-16T01:40:08.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.248+0000 7f5082e65640 1 -- 172.21.15.88:0/1446746089 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f506006c2b0 msgr2=0x7f506006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:08.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.248+0000 7f5082e65640 1 --2- 172.21.15.88:0/1446746089 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f506006c2b0 0x7f506006e770 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5068007fd0 tx=0x7f5068009040 comp rx=0 tx=0).stop 2024-09-16T01:40:08.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.248+0000 7f5082e65640 1 -- 172.21.15.88:0/1446746089 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f506c004d40 msgr2=0x7f507c14d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:08.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.248+0000 7f5082e65640 1 --2- 172.21.15.88:0/1446746089 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f506c004d40 0x7f507c14d290 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f50700119a0 tx=0x7f5070011e60 comp rx=0 tx=0).stop 2024-09-16T01:40:08.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.248+0000 7f5082e65640 1 -- 172.21.15.88:0/1446746089 shutdown_connections 2024-09-16T01:40:08.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.248+0000 7f5082e65640 1 --2- 172.21.15.88:0/1446746089 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f506006c2b0 0x7f506006e770 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.248+0000 7f5082e65640 1 --2- 172.21.15.88:0/1446746089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f507c1e9b30 0x7f507c160690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.248+0000 7f5082e65640 1 --2- 172.21.15.88:0/1446746089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f507c1f43d0 0x7f507c1e95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.248+0000 7f5082e65640 1 --2- 172.21.15.88:0/1446746089 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f506c004d40 0x7f507c14d290 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.248+0000 7f5082e65640 1 -- 172.21.15.88:0/1446746089 >> 172.21.15.88:0/1446746089 conn(0x7f507c06bc10 msgr2=0x7f507c0b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:08.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.248+0000 7f5082e65640 1 -- 172.21.15.88:0/1446746089 shutdown_connections 2024-09-16T01:40:08.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.248+0000 7f5082e65640 1 -- 172.21.15.88:0/1446746089 wait complete. 2024-09-16T01:40:08.663 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:40:03.597796+0000', 'modified': '2024-09-16T01:40:05.602153+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6852}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6852': {'gid': 6852, 'name': 'c', 'rank': 0, 'incarnation': 68, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.88:6837/4162274767', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6835', 'nonce': 4162274767}, {'type': 'v1', 'addr': '172.21.15.88:6837', 'nonce': 4162274767}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6852, 'qdb_cluster': [6852]} 2024-09-16T01:40:08.663 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T01:40:08.664 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-16T01:40:08.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.763+0000 7f52385a6640 1 Processor -- start 2024-09-16T01:40:08.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.763+0000 7f52385a6640 1 -- start start 2024-09-16T01:40:08.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.764+0000 7f52385a6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f523015e940 0x7f52300b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.764+0000 7f52385a6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f52300d2290 0x7f52300b6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.764+0000 7f52385a6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f523015ed20 con 0x7f52300ba440 2024-09-16T01:40:08.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.764+0000 7f52385a6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f52300d3ef0 con 0x7f52300d2290 2024-09-16T01:40:08.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.764+0000 7f52385a6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f52300d32e0 con 0x7f523015e940 2024-09-16T01:40:08.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.764+0000 7f5236342640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f523015e940 0x7f52300b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.764+0000 7f5236b43640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f52300d2290 0x7f52300b6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.764+0000 7f5235b41640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f52300ba440 0x7f52300cef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:39126/0 (socket says 172.21.15.88:39126) 2024-09-16T01:40:08.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.764+0000 7f5236342640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f523015e940 0x7f52300b6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:42854/0 (socket says 172.21.15.88:42854) 2024-09-16T01:40:08.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.764+0000 7f5235b41640 1 -- 172.21.15.88:0/3299896277 learned_addr learned my addr 172.21.15.88:0/3299896277 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:08.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.765+0000 7f5236342640 1 -- 172.21.15.88:0/3299896277 >> v2:172.21.15.154:3300/0 conn(0x7f52300d2290 msgr2=0x7f52300b6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:08.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.765+0000 7f5236342640 1 --2- 172.21.15.88:0/3299896277 >> v2:172.21.15.154:3300/0 conn(0x7f52300d2290 0x7f52300b6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.765+0000 7f5236342640 1 -- 172.21.15.88:0/3299896277 >> v1:172.21.15.154:6789/0 conn(0x7f52300ba440 legacy=0x7f52300cef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:08.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.765+0000 7f5236342640 1 -- 172.21.15.88:0/3299896277 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52300d35c0 con 0x7f523015e940 2024-09-16T01:40:08.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.765+0000 7f5235340640 1 -- 172.21.15.88:0/3299896277 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3262166195 0 0) 0x7f523015ed20 con 0x7f52300ba440 2024-09-16T01:40:08.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.765+0000 7f5236b43640 1 --2- 172.21.15.88:0/3299896277 >> v2:172.21.15.154:3300/0 conn(0x7f52300d2290 0x7f52300b6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:40:08.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.765+0000 7f5236342640 1 --2- 172.21.15.88:0/3299896277 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f523015e940 0x7f52300b6140 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f522c0097a0 tx=0x7f522c02a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fad233d3914c090d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:08.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.765+0000 7f5235340640 1 -- 172.21.15.88:0/3299896277 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522c037070 con 0x7f523015e940 2024-09-16T01:40:08.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.765+0000 7f5235340640 1 -- 172.21.15.88:0/3299896277 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f522c032440 con 0x7f523015e940 2024-09-16T01:40:08.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.766+0000 7f5235340640 1 -- 172.21.15.88:0/3299896277 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522c02ae60 con 0x7f523015e940 2024-09-16T01:40:08.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.766+0000 7f52385a6640 1 -- 172.21.15.88:0/3299896277 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f523015e940 msgr2=0x7f52300b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:08.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.766+0000 7f52385a6640 1 --2- 172.21.15.88:0/3299896277 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f523015e940 0x7f52300b6140 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f522c0097a0 tx=0x7f522c02a110 comp rx=0 tx=0).stop 2024-09-16T01:40:08.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.766+0000 7f52385a6640 1 -- 172.21.15.88:0/3299896277 shutdown_connections 2024-09-16T01:40:08.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.766+0000 7f52385a6640 1 --2- 172.21.15.88:0/3299896277 >> v2:172.21.15.154:3300/0 conn(0x7f52300d2290 0x7f52300b6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.766+0000 7f52385a6640 1 --2- 172.21.15.88:0/3299896277 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f523015e940 0x7f52300b6140 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.766+0000 7f52385a6640 1 -- 172.21.15.88:0/3299896277 >> 172.21.15.88:0/3299896277 conn(0x7f52300d1140 msgr2=0x7f52300d1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:08.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.766+0000 7f52385a6640 1 -- 172.21.15.88:0/3299896277 shutdown_connections 2024-09-16T01:40:08.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.766+0000 7f52385a6640 1 -- 172.21.15.88:0/3299896277 wait complete. 2024-09-16T01:40:08.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.767+0000 7f52385a6640 1 Processor -- start 2024-09-16T01:40:08.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.767+0000 7f52385a6640 1 -- start start 2024-09-16T01:40:08.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.768+0000 7f52385a6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52300ba440 0x7f52301f3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.768+0000 7f52385a6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52300d2290 0x7f52301509c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.768+0000 7f52385a6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f523015e940 0x7f5230160450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.768+0000 7f52385a6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f52300b7510 con 0x7f523015e940 2024-09-16T01:40:08.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.768+0000 7f52385a6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f52300d3b80 con 0x7f52300ba440 2024-09-16T01:40:08.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.768+0000 7f52385a6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f52300b8bb0 con 0x7f52300d2290 2024-09-16T01:40:08.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.768+0000 7f5236b43640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f523015e940 0x7f5230160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.768+0000 7f5236b43640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f523015e940 0x7f5230160450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60122/0 (socket says 172.21.15.88:60122) 2024-09-16T01:40:08.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.768+0000 7f5236b43640 1 -- 172.21.15.88:0/3372310236 learned_addr learned my addr 172.21.15.88:0/3372310236 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:08.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.768+0000 7f5236342640 1 --2- 172.21.15.88:0/3372310236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52300ba440 0x7f52301f3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.768+0000 7f5235b41640 1 --2- 172.21.15.88:0/3372310236 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52300d2290 0x7f52301509c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.768+0000 7f5236b43640 1 -- 172.21.15.88:0/3372310236 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52300d2290 msgr2=0x7f52301509c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:08.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.768+0000 7f5236b43640 1 --2- 172.21.15.88:0/3372310236 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52300d2290 0x7f52301509c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.768+0000 7f5236b43640 1 -- 172.21.15.88:0/3372310236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52300ba440 msgr2=0x7f52301f3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:08.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.768+0000 7f5236b43640 1 --2- 172.21.15.88:0/3372310236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52300ba440 0x7f52301f3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.768+0000 7f5236b43640 1 -- 172.21.15.88:0/3372310236 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f523014a9e0 con 0x7f523015e940 2024-09-16T01:40:08.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.769+0000 7f5236b43640 1 --2- 172.21.15.88:0/3372310236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f523015e940 0x7f5230160450 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f522800efc0 tx=0x7f5228009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:08.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.769+0000 7f5226ffd640 1 -- 172.21.15.88:0/3372310236 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5228018030 con 0x7f523015e940 2024-09-16T01:40:08.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.769+0000 7f5226ffd640 1 -- 172.21.15.88:0/3372310236 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f522800cde0 con 0x7f523015e940 2024-09-16T01:40:08.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.769+0000 7f5226ffd640 1 -- 172.21.15.88:0/3372310236 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5228003bb0 con 0x7f523015e940 2024-09-16T01:40:08.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.769+0000 7f52385a6640 1 -- 172.21.15.88:0/3372310236 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52300cde60 con 0x7f523015e940 2024-09-16T01:40:08.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.769+0000 7f52385a6640 1 -- 172.21.15.88:0/3372310236 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52300ce420 con 0x7f523015e940 2024-09-16T01:40:08.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.770+0000 7f5226ffd640 1 -- 172.21.15.88:0/3372310236 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5228005d10 con 0x7f523015e940 2024-09-16T01:40:08.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.771+0000 7f5226ffd640 1 --2- 172.21.15.88:0/3372310236 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f521406c2e0 0x7f521406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.771+0000 7f5236342640 1 --2- 172.21.15.88:0/3372310236 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f521406c2e0 0x7f521406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.771+0000 7f5226ffd640 1 -- 172.21.15.88:0/3372310236 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f522808bf00 con 0x7f523015e940 2024-09-16T01:40:08.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.771+0000 7f5236342640 1 --2- 172.21.15.88:0/3372310236 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f521406c2e0 0x7f521406e7a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f522c034000 tx=0x7f522c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:08.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.772+0000 7f523b422740 1 Processor -- start 2024-09-16T01:40:08.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.772+0000 7f523b422740 1 -- start start 2024-09-16T01:40:08.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.773+0000 7f523b422740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b23a62c0 0x55d9b23eba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.773+0000 7f523b422740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a3a80 0x55d9b23a66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.773+0000 7f523b422740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9b23a7a50 0x55d9b23a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.773+0000 7f523b422740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d9b2308520 con 0x55d9b23a3a80 2024-09-16T01:40:08.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.773+0000 7f523b422740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d9b22b03e0 con 0x55d9b23a62c0 2024-09-16T01:40:08.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.773+0000 7f523b422740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55d9b23044e0 con 0x55d9b23a7a50 2024-09-16T01:40:08.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.773+0000 7f5235b41640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a3a80 0x55d9b23a66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.773+0000 7f5235b41640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a3a80 0x55d9b23a66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60126/0 (socket says 172.21.15.88:60126) 2024-09-16T01:40:08.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.773+0000 7f5235b41640 1 -- 172.21.15.88:0/2709914981 learned_addr learned my addr 172.21.15.88:0/2709914981 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:08.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.773+0000 7f5236342640 1 --2- 172.21.15.88:0/2709914981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b23a62c0 0x55d9b23eba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.773+0000 7f5236b43640 1 --2- 172.21.15.88:0/2709914981 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9b23a7a50 0x55d9b23a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.773+0000 7f5235b41640 1 -- 172.21.15.88:0/2709914981 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9b23a7a50 msgr2=0x55d9b23a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:08.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.773+0000 7f5235b41640 1 --2- 172.21.15.88:0/2709914981 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9b23a7a50 0x55d9b23a08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.773+0000 7f5235b41640 1 -- 172.21.15.88:0/2709914981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b23a62c0 msgr2=0x55d9b23eba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:08.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.773+0000 7f5235b41640 1 --2- 172.21.15.88:0/2709914981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b23a62c0 0x55d9b23eba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.773+0000 7f5235b41640 1 -- 172.21.15.88:0/2709914981 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9b22cb2f0 con 0x55d9b23a3a80 2024-09-16T01:40:08.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.773+0000 7f5235b41640 1 --2- 172.21.15.88:0/2709914981 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a3a80 0x55d9b23a66a0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f52200016d0 tx=0x7f5220001b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=738f0f061c0c2883 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:08.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.774+0000 7f5213fff640 1 -- 172.21.15.88:0/2709914981 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52200074e0 con 0x55d9b23a3a80 2024-09-16T01:40:08.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.774+0000 7f5213fff640 1 -- 172.21.15.88:0/2709914981 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52200047e0 con 0x55d9b23a3a80 2024-09-16T01:40:08.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.774+0000 7f5213fff640 1 -- 172.21.15.88:0/2709914981 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5220004bb0 con 0x55d9b23a3a80 2024-09-16T01:40:08.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.774+0000 7f523b422740 1 -- 172.21.15.88:0/2709914981 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a3a80 msgr2=0x55d9b23a66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:08.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.774+0000 7f523b422740 1 --2- 172.21.15.88:0/2709914981 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a3a80 0x55d9b23a66a0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f52200016d0 tx=0x7f5220001b90 comp rx=0 tx=0).stop 2024-09-16T01:40:08.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.774+0000 7f523b422740 1 -- 172.21.15.88:0/2709914981 shutdown_connections 2024-09-16T01:40:08.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.774+0000 7f523b422740 1 --2- 172.21.15.88:0/2709914981 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9b23a7a50 0x55d9b23a08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.774+0000 7f523b422740 1 --2- 172.21.15.88:0/2709914981 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a3a80 0x55d9b23a66a0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.774+0000 7f523b422740 1 --2- 172.21.15.88:0/2709914981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b23a62c0 0x55d9b23eba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.774+0000 7f523b422740 1 -- 172.21.15.88:0/2709914981 >> 172.21.15.88:0/2709914981 conn(0x55d9b23a2a80 msgr2=0x55d9b23eaf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:08.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.775+0000 7f523b422740 1 -- 172.21.15.88:0/2709914981 shutdown_connections 2024-09-16T01:40:08.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.775+0000 7f523b422740 1 -- 172.21.15.88:0/2709914981 wait complete. 2024-09-16T01:40:08.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.775+0000 7f523b422740 1 -- start start 2024-09-16T01:40:08.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.775+0000 7f523b422740 1 Processor -- start 2024-09-16T01:40:08.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.776+0000 7f523b422740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b23a62c0 0x55d9b23c4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.776+0000 7f523b422740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a7a50 0x55d9b23a1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.776+0000 7f523b422740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9b23a16b0 0x55d9b23a48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.776+0000 7f523b422740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d9b23a1f20 con 0x55d9b23a7a50 2024-09-16T01:40:08.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.776+0000 7f523b422740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d9b2291450 con 0x55d9b23a62c0 2024-09-16T01:40:08.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.776+0000 7f523b422740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55d9b23ec290 con 0x55d9b23a16b0 2024-09-16T01:40:08.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.776+0000 7f523b422740 10 client.0 _ll_register_callbacks cb 0x55d9b201aac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:40:08.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.776+0000 7f5235b41640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a7a50 0x55d9b23a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.776+0000 7f5235b41640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a7a50 0x55d9b23a1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60128/0 (socket says 172.21.15.88:60128) 2024-09-16T01:40:08.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.776+0000 7f5235b41640 1 -- 172.21.15.88:0/1634616967 learned_addr learned my addr 172.21.15.88:0/1634616967 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:08.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.776+0000 7f5235b41640 1 -- 172.21.15.88:0/1634616967 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9b23a16b0 msgr2=0x55d9b23a48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:08.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.776+0000 7f5235b41640 1 --2- 172.21.15.88:0/1634616967 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9b23a16b0 0x55d9b23a48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.776+0000 7f5235b41640 1 -- 172.21.15.88:0/1634616967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b23a62c0 msgr2=0x55d9b23c4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:08.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.776+0000 7f5235b41640 1 --2- 172.21.15.88:0/1634616967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b23a62c0 0x55d9b23c4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.776+0000 7f5235b41640 1 -- 172.21.15.88:0/1634616967 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9b201b2d0 con 0x55d9b23a7a50 2024-09-16T01:40:08.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.777+0000 7f5235b41640 1 --2- 172.21.15.88:0/1634616967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a7a50 0x55d9b23a1170 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f5220011000 tx=0x7f5220000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c1177234d056245 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:08.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.777+0000 7f52117fa640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:40:08.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.777+0000 7f52117fa640 1 -- 172.21.15.88:0/1634616967 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5220024020 con 0x55d9b23a7a50 2024-09-16T01:40:08.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.777+0000 7f523b422740 1 -- 172.21.15.88:0/1634616967 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52200047e0 con 0x55d9b23a7a50 2024-09-16T01:40:08.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.777+0000 7f52117fa640 1 -- 172.21.15.88:0/1634616967 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f5220004b90 con 0x55d9b23a7a50 2024-09-16T01:40:08.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.777+0000 7f52117fa640 1 -- 172.21.15.88:0/1634616967 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f522002e050 con 0x55d9b23a7a50 2024-09-16T01:40:08.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.777+0000 7f523b422740 10 client.6987 fetch_fsmap learned FSMap version 70 2024-09-16T01:40:08.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.777+0000 7f523b422740 1 -- 172.21.15.88:0/1634616967 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) -- 0x55d9b20e56b0 con 0x55d9b23a7a50 2024-09-16T01:40:08.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.777+0000 7f52117fa640 1 -- 172.21.15.88:0/1634616967 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52200047e0 con 0x55d9b23a7a50 2024-09-16T01:40:08.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.777+0000 7f52117fa640 1 -- 172.21.15.88:0/1634616967 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 70) ==== 3502+0+0 (secure 0 0 0) 0x7f5220031020 con 0x55d9b23a7a50 2024-09-16T01:40:08.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.777+0000 7f523b422740 10 client.6987 fetch_fsmap finished waiting for FSMap version 70 2024-09-16T01:40:08.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.777+0000 7f523b422740 10 client.6987 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T01:40:08.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.777+0000 7f523b422740 20 client.6987 populate_metadata read hostname 'smithi088' 2024-09-16T01:40:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.777+0000 7f523b422740 1 --2- 172.21.15.88:0/1634616967 >> [v2:172.21.15.88:6835/4162274767,v1:172.21.15.88:6837/4162274767] conn(0x55d9b23c5920 0x55d9b23c5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.777+0000 7f523b422740 4 client.6987 mds_command: new command op to 6852 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:40:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.777+0000 7f523b422740 1 -- 172.21.15.88:0/1634616967 --> [v2:172.21.15.88:6835/4162274767,v1:172.21.15.88:6837/4162274767] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9b2315c50 con 0x55d9b23c5920 2024-09-16T01:40:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.777+0000 7f5236342640 1 --2- 172.21.15.88:0/1634616967 >> [v2:172.21.15.88:6835/4162274767,v1:172.21.15.88:6837/4162274767] conn(0x55d9b23c5920 0x55d9b23c5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.778+0000 7f52117fa640 10 client.6987 ms_handle_connect on v2:172.21.15.88:6835/4162274767 2024-09-16T01:40:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.778+0000 7f5236342640 1 --2- 172.21.15.88:0/1634616967 >> [v2:172.21.15.88:6835/4162274767,v1:172.21.15.88:6837/4162274767] conn(0x55d9b23c5920 0x55d9b23c5d00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1c51cc4c0b5e37c server_cookie=52167d763cb5428e in_seq=0 out_seq=0 2024-09-16T01:40:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.778+0000 7f52117fa640 1 -- 172.21.15.88:0/1634616967 <== mds.0 v2:172.21.15.88:6835/4162274767 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d9b2315c50 con 0x55d9b23c5920 2024-09-16T01:40:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.778+0000 7f52117fa640 10 client.6987 handle_command_reply: tid=0 2024-09-16T01:40:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.778+0000 7f523b422740 1 client.6987 shutdown 2024-09-16T01:40:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.778+0000 7f523b422740 1 -- 172.21.15.88:0/1634616967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a7a50 msgr2=0x55d9b23a1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.778+0000 7f523b422740 1 --2- 172.21.15.88:0/1634616967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a7a50 0x55d9b23a1170 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f5220011000 tx=0x7f5220000ed0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.778+0000 7f523b422740 1 -- 172.21.15.88:0/1634616967 shutdown_connections 2024-09-16T01:40:08.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.778+0000 7f523b422740 1 --2- 172.21.15.88:0/1634616967 >> [v2:172.21.15.88:6835/4162274767,v1:172.21.15.88:6837/4162274767] conn(0x55d9b23c5920 0x55d9b23c5d00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.778+0000 7f523b422740 1 --2- 172.21.15.88:0/1634616967 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9b23a16b0 0x55d9b23a48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.778+0000 7f523b422740 1 --2- 172.21.15.88:0/1634616967 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a7a50 0x55d9b23a1170 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.778+0000 7f523b422740 1 --2- 172.21.15.88:0/1634616967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b23a62c0 0x55d9b23c4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.778+0000 7f52117fa640 0 client.6987 ms_handle_reset on v2:172.21.15.88:6835/4162274767 2024-09-16T01:40:08.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.778+0000 7f523b422740 1 -- 172.21.15.88:0/1634616967 >> 172.21.15.88:0/1634616967 conn(0x55d9b23a3a80 msgr2=0x55d9b23e8ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:08.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.779+0000 7f523b422740 1 -- 172.21.15.88:0/1634616967 shutdown_connections 2024-09-16T01:40:08.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.779+0000 7f523b422740 1 -- 172.21.15.88:0/1634616967 wait complete. 2024-09-16T01:40:08.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.779+0000 7f523b422740 20 client.6987 trim_cache size 0 max 16384 2024-09-16T01:40:08.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.796+0000 7f523b422740 1 Processor -- start 2024-09-16T01:40:08.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.796+0000 7f523b422740 1 -- start start 2024-09-16T01:40:08.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.797+0000 7f523b422740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b23a16b0 0x55d9b23c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.797+0000 7f523b422740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a62c0 0x55d9b23a1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.797+0000 7f523b422740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9b23a7a50 0x55d9b23c5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.797+0000 7f523b422740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f52301ff970 con 0x55d9b23a62c0 2024-09-16T01:40:08.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.797+0000 7f523b422740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f523014d600 con 0x55d9b23a16b0 2024-09-16T01:40:08.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.797+0000 7f523b422740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f52301ff470 con 0x55d9b23a7a50 2024-09-16T01:40:08.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.797+0000 7f5235b41640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a62c0 0x55d9b23a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.797+0000 7f5235b41640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a62c0 0x55d9b23a1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60136/0 (socket says 172.21.15.88:60136) 2024-09-16T01:40:08.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.797+0000 7f5235b41640 1 -- 172.21.15.88:0/2561161613 learned_addr learned my addr 172.21.15.88:0/2561161613 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:08.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.797+0000 7f5235b41640 1 -- 172.21.15.88:0/2561161613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9b23a7a50 msgr2=0x55d9b23c5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:08.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.797+0000 7f5235b41640 1 --2- 172.21.15.88:0/2561161613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9b23a7a50 0x55d9b23c5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.797+0000 7f5236342640 1 --2- 172.21.15.88:0/2561161613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b23a16b0 0x55d9b23c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.797+0000 7f5235b41640 1 -- 172.21.15.88:0/2561161613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b23a16b0 msgr2=0x55d9b23c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:08.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.797+0000 7f5235b41640 1 --2- 172.21.15.88:0/2561161613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b23a16b0 0x55d9b23c4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.797+0000 7f5235b41640 1 -- 172.21.15.88:0/2561161613 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9b239c310 con 0x55d9b23a62c0 2024-09-16T01:40:08.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.798+0000 7f5235b41640 1 --2- 172.21.15.88:0/2561161613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a62c0 0x55d9b23a1020 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f5220007b40 tx=0x7f5220020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b29e89ccc27ff16 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:08.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.798+0000 7f52137fe640 1 -- 172.21.15.88:0/2561161613 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5220024070 con 0x55d9b23a62c0 2024-09-16T01:40:08.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.798+0000 7f52137fe640 1 -- 172.21.15.88:0/2561161613 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5220004a60 con 0x55d9b23a62c0 2024-09-16T01:40:08.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.798+0000 7f52137fe640 1 -- 172.21.15.88:0/2561161613 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522000d730 con 0x55d9b23a62c0 2024-09-16T01:40:08.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.798+0000 7f523b422740 1 -- 172.21.15.88:0/2561161613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a62c0 msgr2=0x55d9b23a1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:08.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.798+0000 7f523b422740 1 --2- 172.21.15.88:0/2561161613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a62c0 0x55d9b23a1020 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f5220007b40 tx=0x7f5220020000 comp rx=0 tx=0).stop 2024-09-16T01:40:08.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.798+0000 7f523b422740 1 -- 172.21.15.88:0/2561161613 shutdown_connections 2024-09-16T01:40:08.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.798+0000 7f523b422740 1 --2- 172.21.15.88:0/2561161613 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9b23a7a50 0x55d9b23c5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.798+0000 7f523b422740 1 --2- 172.21.15.88:0/2561161613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a62c0 0x55d9b23a1020 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.798+0000 7f523b422740 1 --2- 172.21.15.88:0/2561161613 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b23a16b0 0x55d9b23c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.798+0000 7f523b422740 1 -- 172.21.15.88:0/2561161613 >> 172.21.15.88:0/2561161613 conn(0x55d9b23a3a80 msgr2=0x55d9b23a40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:08.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.798+0000 7f523b422740 1 -- 172.21.15.88:0/2561161613 shutdown_connections 2024-09-16T01:40:08.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.798+0000 7f523b422740 1 -- 172.21.15.88:0/2561161613 wait complete. 2024-09-16T01:40:08.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.799+0000 7f523b422740 1 -- start start 2024-09-16T01:40:08.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.799+0000 7f523b422740 1 Processor -- start 2024-09-16T01:40:08.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.800+0000 7f523b422740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9b23a16b0 0x55d9b23cd390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.800+0000 7f523b422740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b23a62c0 0x55d9b24699f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.800+0000 7f523b422740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a7a50 0x55d9b2469f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.800+0000 7f523b422740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d9b23cf6a0 con 0x55d9b23a7a50 2024-09-16T01:40:08.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.800+0000 7f523b422740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d9b23fc050 con 0x55d9b23a62c0 2024-09-16T01:40:08.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.800+0000 7f523b422740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55d9b239faf0 con 0x55d9b23a16b0 2024-09-16T01:40:08.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.800+0000 7f523b422740 10 client.0 _ll_register_callbacks cb 0x7f522c035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:40:08.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.800+0000 7f5236b43640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a7a50 0x55d9b2469f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.800+0000 7f5236b43640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a7a50 0x55d9b2469f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60148/0 (socket says 172.21.15.88:60148) 2024-09-16T01:40:08.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.800+0000 7f5236b43640 1 -- 172.21.15.88:0/2589739643 learned_addr learned my addr 172.21.15.88:0/2589739643 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:08.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.800+0000 7f5235b41640 1 --2- 172.21.15.88:0/2589739643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b23a62c0 0x55d9b24699f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.800+0000 7f5236342640 1 --2- 172.21.15.88:0/2589739643 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9b23a16b0 0x55d9b23cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.800+0000 7f5236b43640 1 -- 172.21.15.88:0/2589739643 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9b23a16b0 msgr2=0x55d9b23cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:08.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.800+0000 7f5236b43640 1 --2- 172.21.15.88:0/2589739643 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9b23a16b0 0x55d9b23cd390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.800+0000 7f5236b43640 1 -- 172.21.15.88:0/2589739643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b23a62c0 msgr2=0x55d9b24699f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:08.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.800+0000 7f5236b43640 1 --2- 172.21.15.88:0/2589739643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b23a62c0 0x55d9b24699f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.800+0000 7f5236b43640 1 -- 172.21.15.88:0/2589739643 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9b246de40 con 0x55d9b23a7a50 2024-09-16T01:40:08.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.800+0000 7f5236b43640 1 --2- 172.21.15.88:0/2589739643 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a7a50 0x55d9b2469f30 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f52280625a0 tx=0x7f522805a6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8810c8bbebbbcdb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:08.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.800+0000 7f52117fa640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:40:08.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.800+0000 7f52117fa640 1 -- 172.21.15.88:0/2589739643 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5228077d90 con 0x55d9b23a7a50 2024-09-16T01:40:08.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.800+0000 7f52117fa640 1 -- 172.21.15.88:0/2589739643 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f5228059340 con 0x55d9b23a7a50 2024-09-16T01:40:08.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.800+0000 7f523b422740 1 -- 172.21.15.88:0/2589739643 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5220004a60 con 0x55d9b23a7a50 2024-09-16T01:40:08.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.800+0000 7f52117fa640 1 -- 172.21.15.88:0/2589739643 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) ==== 24+0+0 (secure 0 0 0) 0x7f5228062ad0 con 0x55d9b23a7a50 2024-09-16T01:40:08.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.800+0000 7f523b422740 10 client.6993 fetch_fsmap learned FSMap version 70 2024-09-16T01:40:08.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.800+0000 7f523b422740 1 -- 172.21.15.88:0/2589739643 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) -- 0x55d9b23e3d60 con 0x55d9b23a7a50 2024-09-16T01:40:08.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.801+0000 7f52117fa640 1 -- 172.21.15.88:0/2589739643 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5220004a60 con 0x55d9b23a7a50 2024-09-16T01:40:08.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.801+0000 7f52117fa640 1 -- 172.21.15.88:0/2589739643 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 70) ==== 3502+0+0 (secure 0 0 0) 0x7f5228076020 con 0x55d9b23a7a50 2024-09-16T01:40:08.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.801+0000 7f523b422740 10 client.6993 fetch_fsmap finished waiting for FSMap version 70 2024-09-16T01:40:08.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.801+0000 7f523b422740 10 client.6993 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T01:40:08.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.801+0000 7f523b422740 20 client.6993 populate_metadata read hostname 'smithi088' 2024-09-16T01:40:08.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.801+0000 7f523b422740 1 --2- 172.21.15.88:0/2589739643 >> [v2:172.21.15.88:6835/4162274767,v1:172.21.15.88:6837/4162274767] conn(0x55d9b23e4690 0x55d9b23e4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.801+0000 7f523b422740 4 client.6993 mds_command: new command op to 6852 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:40:08.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.801+0000 7f523b422740 1 -- 172.21.15.88:0/2589739643 --> [v2:172.21.15.88:6835/4162274767,v1:172.21.15.88:6837/4162274767] -- command(tid 0: {"prefix": "status"}) -- 0x55d9b230ffa0 con 0x55d9b23e4690 2024-09-16T01:40:08.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.801+0000 7f5236342640 1 --2- 172.21.15.88:0/2589739643 >> [v2:172.21.15.88:6835/4162274767,v1:172.21.15.88:6837/4162274767] conn(0x55d9b23e4690 0x55d9b23e4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.801+0000 7f52117fa640 10 client.6993 ms_handle_connect on v2:172.21.15.88:6835/4162274767 2024-09-16T01:40:08.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.801+0000 7f5236342640 1 --2- 172.21.15.88:0/2589739643 >> [v2:172.21.15.88:6835/4162274767,v1:172.21.15.88:6837/4162274767] conn(0x55d9b23e4690 0x55d9b23e4a70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2db10a94cdc5226 server_cookie=50877a3d9ab8277a in_seq=0 out_seq=0 2024-09-16T01:40:08.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.801+0000 7f52117fa640 1 -- 172.21.15.88:0/2589739643 <== mds.0 v2:172.21.15.88:6835/4162274767 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55d9b230ffa0 con 0x55d9b23e4690 2024-09-16T01:40:08.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.801+0000 7f52117fa640 10 client.6993 handle_command_reply: tid=0 2024-09-16T01:40:08.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.801+0000 7f523b422740 1 client.6993 shutdown 2024-09-16T01:40:08.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.801+0000 7f523b422740 1 -- 172.21.15.88:0/2589739643 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a7a50 msgr2=0x55d9b2469f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:08.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.801+0000 7f523b422740 1 --2- 172.21.15.88:0/2589739643 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a7a50 0x55d9b2469f30 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f52280625a0 tx=0x7f522805a6b0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.802+0000 7f523b422740 1 -- 172.21.15.88:0/2589739643 shutdown_connections 2024-09-16T01:40:08.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.802+0000 7f523b422740 1 --2- 172.21.15.88:0/2589739643 >> [v2:172.21.15.88:6835/4162274767,v1:172.21.15.88:6837/4162274767] conn(0x55d9b23e4690 0x55d9b23e4a70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.802+0000 7f523b422740 1 --2- 172.21.15.88:0/2589739643 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9b23a7a50 0x55d9b2469f30 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.802+0000 7f523b422740 1 --2- 172.21.15.88:0/2589739643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b23a62c0 0x55d9b24699f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.802+0000 7f523b422740 1 --2- 172.21.15.88:0/2589739643 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9b23a16b0 0x55d9b23cd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.802+0000 7f52117fa640 0 client.6993 ms_handle_reset on v2:172.21.15.88:6835/4162274767 2024-09-16T01:40:08.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.802+0000 7f523b422740 1 -- 172.21.15.88:0/2589739643 >> 172.21.15.88:0/2589739643 conn(0x55d9b23a3a80 msgr2=0x55d9b23e8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:08.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.802+0000 7f523b422740 1 -- 172.21.15.88:0/2589739643 shutdown_connections 2024-09-16T01:40:08.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.802+0000 7f523b422740 1 -- 172.21.15.88:0/2589739643 wait complete. 2024-09-16T01:40:08.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.802+0000 7f523b422740 20 client.6993 trim_cache size 0 max 16384 2024-09-16T01:40:08.811 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:40:08.811 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "5ace2c91-46f9-482b-8a89-731fbdc171bf", 2024-09-16T01:40:08.811 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2024-09-16T01:40:08.811 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 6852, 2024-09-16T01:40:08.811 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2024-09-16T01:40:08.811 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2024-09-16T01:40:08.811 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2024-09-16T01:40:08.811 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 4.5503706030000002, 2024-09-16T01:40:08.811 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 70, 2024-09-16T01:40:08.811 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 85, 2024-09-16T01:40:08.812 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 85, 2024-09-16T01:40:08.812 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 8.8915877499999993 2024-09-16T01:40:08.812 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:40:08.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.802+0000 7f52385a6640 1 -- 172.21.15.88:0/3372310236 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f521406c2e0 msgr2=0x7f521406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:08.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.802+0000 7f52385a6640 1 --2- 172.21.15.88:0/3372310236 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f521406c2e0 0x7f521406e7a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f522c034000 tx=0x7f522c02e000 comp rx=0 tx=0).stop 2024-09-16T01:40:08.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.802+0000 7f52385a6640 1 -- 172.21.15.88:0/3372310236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f523015e940 msgr2=0x7f5230160450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:08.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.802+0000 7f52385a6640 1 --2- 172.21.15.88:0/3372310236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f523015e940 0x7f5230160450 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f522800efc0 tx=0x7f5228009f90 comp rx=0 tx=0).stop 2024-09-16T01:40:08.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.802+0000 7f52385a6640 1 -- 172.21.15.88:0/3372310236 shutdown_connections 2024-09-16T01:40:08.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.802+0000 7f52385a6640 1 --2- 172.21.15.88:0/3372310236 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f521406c2e0 0x7f521406e7a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.802+0000 7f52385a6640 1 --2- 172.21.15.88:0/3372310236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f523015e940 0x7f5230160450 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.802+0000 7f52385a6640 1 --2- 172.21.15.88:0/3372310236 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52300d2290 0x7f52301509c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.802+0000 7f52385a6640 1 --2- 172.21.15.88:0/3372310236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52300ba440 0x7f52301f3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.802+0000 7f52385a6640 1 -- 172.21.15.88:0/3372310236 >> 172.21.15.88:0/3372310236 conn(0x7f52300d1140 msgr2=0x7f52300b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:08.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.802+0000 7f52385a6640 1 -- 172.21.15.88:0/3372310236 shutdown_connections 2024-09-16T01:40:08.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.802+0000 7f52385a6640 1 -- 172.21.15.88:0/3372310236 wait complete. 2024-09-16T01:40:08.815 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:40:08.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.911+0000 7f2bec511640 1 Processor -- start 2024-09-16T01:40:08.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.911+0000 7f2bec511640 1 -- start start 2024-09-16T01:40:08.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.912+0000 7f2bec511640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2be415c550 0x7f2be40b5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.912+0000 7f2bec511640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be40b4cf0 con 0x7f2be415c170 2024-09-16T01:40:08.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.912+0000 7f2bec511640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be40b4fd0 con 0x7f2be40b6310 2024-09-16T01:40:08.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.912+0000 7f2bec511640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2be40b52b0 con 0x7f2be415c550 2024-09-16T01:40:08.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.912+0000 7f2bea2ad640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f2be415c170 0x7f2be40cf6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:33336/0 (socket says 172.21.15.88:33336) 2024-09-16T01:40:08.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.912+0000 7f2bea2ad640 1 -- 172.21.15.88:0/2373094157 learned_addr learned my addr 172.21.15.88:0/2373094157 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:08.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.912+0000 7f2be92ab640 1 -- 172.21.15.88:0/2373094157 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2843127582 0 0) 0x7f2be40b4cf0 con 0x7f2be415c170 2024-09-16T01:40:08.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.912+0000 7f2be92ab640 1 -- 172.21.15.88:0/2373094157 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bd0003520 con 0x7f2be415c170 2024-09-16T01:40:08.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.912+0000 7f2be9aac640 1 --2- 172.21.15.88:0/2373094157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2be415c550 0x7f2be40b5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.912+0000 7f2be92ab640 1 -- 172.21.15.88:0/2373094157 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2339186604 0 0) 0x7f2bd0003520 con 0x7f2be415c170 2024-09-16T01:40:08.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.912+0000 7f2be92ab640 1 -- 172.21.15.88:0/2373094157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2be415c550 msgr2=0x7f2be40b5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:08.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.912+0000 7f2be92ab640 1 --2- 172.21.15.88:0/2373094157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2be415c550 0x7f2be40b5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.912+0000 7f2be92ab640 1 -- 172.21.15.88:0/2373094157 >> v1:172.21.15.154:6789/0 conn(0x7f2be40b6310 legacy=0x7f2be40b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:08.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.912+0000 7f2be92ab640 1 -- 172.21.15.88:0/2373094157 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be40b5590 con 0x7f2be415c170 2024-09-16T01:40:08.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.912+0000 7f2be92ab640 1 -- 172.21.15.88:0/2373094157 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f2be0001670 con 0x7f2be415c170 2024-09-16T01:40:08.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.912+0000 7f2be92ab640 1 -- 172.21.15.88:0/2373094157 >> v1:172.21.15.88:6789/0 conn(0x7f2be415c170 legacy=0x7f2be40cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:08.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.912+0000 7f2be92ab640 1 --2- 172.21.15.88:0/2373094157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2be415c550 0x7f2bd00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.912+0000 7f2be92ab640 1 --2- 172.21.15.88:0/2373094157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bd0003f60 0x7f2bd0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.912+0000 7f2be92ab640 1 --2- 172.21.15.88:0/2373094157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bd0004d40 0x7f2bd00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.913+0000 7f2be92ab640 1 -- 172.21.15.88:0/2373094157 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2be40b52b0 con 0x7f2bd0003f60 2024-09-16T01:40:08.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.913+0000 7f2be92ab640 1 -- 172.21.15.88:0/2373094157 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd000baa0 con 0x7f2bd0004d40 2024-09-16T01:40:08.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.913+0000 7f2be92ab640 1 -- 172.21.15.88:0/2373094157 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2bd000bca0 con 0x7f2be415c550 2024-09-16T01:40:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.913+0000 7f2be9aac640 1 --2- 172.21.15.88:0/2373094157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bd0003f60 0x7f2bd0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.913+0000 7f2bea2ad640 1 --2- 172.21.15.88:0/2373094157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2be415c550 0x7f2bd00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.913+0000 7f2beaaae640 1 --2- 172.21.15.88:0/2373094157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bd0004d40 0x7f2bd00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.913+0000 7f2be9aac640 1 -- 172.21.15.88:0/2373094157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2be415c550 msgr2=0x7f2bd00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.913+0000 7f2be9aac640 1 --2- 172.21.15.88:0/2373094157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2be415c550 0x7f2bd00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.913+0000 7f2be9aac640 1 -- 172.21.15.88:0/2373094157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bd0004d40 msgr2=0x7f2bd00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.913+0000 7f2be9aac640 1 --2- 172.21.15.88:0/2373094157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bd0004d40 0x7f2bd00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.913+0000 7f2be9aac640 1 -- 172.21.15.88:0/2373094157 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bd0003520 con 0x7f2bd0003f60 2024-09-16T01:40:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.913+0000 7f2bea2ad640 1 --2- 172.21.15.88:0/2373094157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2be415c550 0x7f2bd00039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.914+0000 7f2be9aac640 1 --2- 172.21.15.88:0/2373094157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bd0003f60 0x7f2bd0004800 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4009ac0 tx=0x7f2bd0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abf6ec369042e8d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.914+0000 7f2be92ab640 1 -- 172.21.15.88:0/2373094157 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4036070 con 0x7f2bd0003f60 2024-09-16T01:40:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.914+0000 7f2be92ab640 1 -- 172.21.15.88:0/2373094157 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bd4031400 con 0x7f2bd0003f60 2024-09-16T01:40:08.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.914+0000 7f2be92ab640 1 -- 172.21.15.88:0/2373094157 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4009690 con 0x7f2bd0003f60 2024-09-16T01:40:08.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.914+0000 7f2bec511640 1 -- 172.21.15.88:0/2373094157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bd0003f60 msgr2=0x7f2bd0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:08.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.914+0000 7f2bec511640 1 --2- 172.21.15.88:0/2373094157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bd0003f60 0x7f2bd0004800 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4009ac0 tx=0x7f2bd0005830 comp rx=0 tx=0).stop 2024-09-16T01:40:08.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.914+0000 7f2beaaae640 1 -- 172.21.15.88:0/2373094157 reap_dead start 2024-09-16T01:40:08.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.914+0000 7f2bec511640 1 -- 172.21.15.88:0/2373094157 shutdown_connections 2024-09-16T01:40:08.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.914+0000 7f2bec511640 1 -- 172.21.15.88:0/2373094157 >> 172.21.15.88:0/2373094157 conn(0x7f2be406bc10 msgr2=0x7f2be406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:08.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.914+0000 7f2bec511640 1 -- 172.21.15.88:0/2373094157 shutdown_connections 2024-09-16T01:40:08.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.914+0000 7f2bec511640 1 -- 172.21.15.88:0/2373094157 wait complete. 2024-09-16T01:40:08.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.915+0000 7f2bec511640 1 Processor -- start 2024-09-16T01:40:08.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.916+0000 7f2bec511640 1 -- start start 2024-09-16T01:40:08.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.916+0000 7f2bec511640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bd0003f60 0x7f2be413f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.916+0000 7f2bec511640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2be414b090 0x7f2be414bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.916+0000 7f2bec511640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2be415e3c0 0x7f2be415e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.916+0000 7f2bec511640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2be415c990 con 0x7f2bd0003f60 2024-09-16T01:40:08.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.916+0000 7f2bec511640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2be40b5b50 con 0x7f2be415e3c0 2024-09-16T01:40:08.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.916+0000 7f2bec511640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2be40cfdd0 con 0x7f2be414b090 2024-09-16T01:40:08.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.916+0000 7f2bea2ad640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bd0003f60 0x7f2be413f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.916+0000 7f2bea2ad640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bd0003f60 0x7f2be413f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60156/0 (socket says 172.21.15.88:60156) 2024-09-16T01:40:08.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.916+0000 7f2bea2ad640 1 -- 172.21.15.88:0/3900213005 learned_addr learned my addr 172.21.15.88:0/3900213005 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:08.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.917+0000 7f2beaaae640 1 --2- 172.21.15.88:0/3900213005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2be415e3c0 0x7f2be415e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.917+0000 7f2be9aac640 1 --2- 172.21.15.88:0/3900213005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2be414b090 0x7f2be414bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.917+0000 7f2bea2ad640 1 -- 172.21.15.88:0/3900213005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2be414b090 msgr2=0x7f2be414bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:08.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.917+0000 7f2bea2ad640 1 --2- 172.21.15.88:0/3900213005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2be414b090 0x7f2be414bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.917+0000 7f2bea2ad640 1 -- 172.21.15.88:0/3900213005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2be415e3c0 msgr2=0x7f2be415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:08.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.917+0000 7f2bea2ad640 1 --2- 172.21.15.88:0/3900213005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2be415e3c0 0x7f2be415e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:08.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.917+0000 7f2bea2ad640 1 -- 172.21.15.88:0/3900213005 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be414c110 con 0x7f2bd0003f60 2024-09-16T01:40:08.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.917+0000 7f2bea2ad640 1 --2- 172.21.15.88:0/3900213005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bd0003f60 0x7f2be413f030 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f2be000e8f0 tx=0x7f2be000edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:08.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.917+0000 7f2bdaffd640 1 -- 172.21.15.88:0/3900213005 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be000c870 con 0x7f2bd0003f60 2024-09-16T01:40:08.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.917+0000 7f2bdaffd640 1 -- 172.21.15.88:0/3900213005 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2be0018070 con 0x7f2bd0003f60 2024-09-16T01:40:08.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.917+0000 7f2bdaffd640 1 -- 172.21.15.88:0/3900213005 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be000cc30 con 0x7f2bd0003f60 2024-09-16T01:40:08.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.917+0000 7f2bec511640 1 -- 172.21.15.88:0/3900213005 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2be41f25e0 con 0x7f2bd0003f60 2024-09-16T01:40:08.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.917+0000 7f2bec511640 1 -- 172.21.15.88:0/3900213005 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2be41f2ba0 con 0x7f2bd0003f60 2024-09-16T01:40:08.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.918+0000 7f2bec511640 1 -- 172.21.15.88:0/3900213005 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2be40b5dd0 con 0x7f2bd0003f60 2024-09-16T01:40:08.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.919+0000 7f2bdaffd640 1 -- 172.21.15.88:0/3900213005 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2be001d460 con 0x7f2bd0003f60 2024-09-16T01:40:08.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.919+0000 7f2bdaffd640 1 --2- 172.21.15.88:0/3900213005 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2bc806c2b0 0x7f2bc806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:08.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.919+0000 7f2bdaffd640 1 -- 172.21.15.88:0/3900213005 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f2be008b280 con 0x7f2bd0003f60 2024-09-16T01:40:08.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.919+0000 7f2be9aac640 1 --2- 172.21.15.88:0/3900213005 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2bc806c2b0 0x7f2bc806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:08.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.919+0000 7f2be9aac640 1 --2- 172.21.15.88:0/3900213005 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2bc806c2b0 0x7f2bc806e770 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4033000 tx=0x7f2bd4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:08.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:08.923+0000 7f2bdaffd640 1 -- 172.21.15.88:0/3900213005 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2be0059a80 con 0x7f2bd0003f60 2024-09-16T01:40:09.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:09.121+0000 7f2bec511640 1 -- 172.21.15.88:0/3900213005 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2be4152090 con 0x7f2bd0003f60 2024-09-16T01:40:09.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:09.121+0000 7f2bdaffd640 1 -- 172.21.15.88:0/3900213005 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v70) ==== 53+0+73 (secure 0 0 0) 0x7f2be0059380 con 0x7f2bd0003f60 2024-09-16T01:40:09.123 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:40:09.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:09.123+0000 7f2bec511640 1 -- 172.21.15.88:0/3900213005 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2bc806c2b0 msgr2=0x7f2bc806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:09.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:09.123+0000 7f2bec511640 1 --2- 172.21.15.88:0/3900213005 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2bc806c2b0 0x7f2bc806e770 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4033000 tx=0x7f2bd4008030 comp rx=0 tx=0).stop 2024-09-16T01:40:09.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:09.123+0000 7f2bec511640 1 -- 172.21.15.88:0/3900213005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bd0003f60 msgr2=0x7f2be413f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:09.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:09.123+0000 7f2bec511640 1 --2- 172.21.15.88:0/3900213005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bd0003f60 0x7f2be413f030 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f2be000e8f0 tx=0x7f2be000edb0 comp rx=0 tx=0).stop 2024-09-16T01:40:09.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:09.123+0000 7f2bec511640 1 -- 172.21.15.88:0/3900213005 shutdown_connections 2024-09-16T01:40:09.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:09.123+0000 7f2bec511640 1 --2- 172.21.15.88:0/3900213005 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2bc806c2b0 0x7f2bc806e770 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:09.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:09.123+0000 7f2bec511640 1 --2- 172.21.15.88:0/3900213005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2be415e3c0 0x7f2be415e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:09.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:09.123+0000 7f2bec511640 1 --2- 172.21.15.88:0/3900213005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2be414b090 0x7f2be414bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:09.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:09.123+0000 7f2bec511640 1 --2- 172.21.15.88:0/3900213005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bd0003f60 0x7f2be413f030 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:09.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:09.123+0000 7f2bec511640 1 -- 172.21.15.88:0/3900213005 >> 172.21.15.88:0/3900213005 conn(0x7f2be406bc10 msgr2=0x7f2be406ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:09.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:09.123+0000 7f2bec511640 1 -- 172.21.15.88:0/3900213005 shutdown_connections 2024-09-16T01:40:09.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:09.123+0000 7f2bec511640 1 -- 172.21.15.88:0/3900213005 wait complete. 2024-09-16T01:40:09.135 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-16T01:40:09.135 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:40:09.135 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2024-09-16T01:40:09.135 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T01:40:09.135 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T01:40:09.136 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-16T01:40:09.136 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-16T01:40:09.136 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:40:09.136 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:40:09.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:09.136 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2024-09-16T01:40:09.155 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:40:09.155 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:40:09.155 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:40:09.155 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:40:09.155 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-09-16T01:40:09.156 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:40:09.156 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:40:09.156 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:40:09.156 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2024-09-16T01:40:09.156 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:40:09.156 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:40:09.156 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2024-09-16T01:40:09.156 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:40:09.156 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2024-09-16T01:40:09.156 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:40:09.156 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2024-09-16T01:40:09.156 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:40:09.156 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42415sec preferred_lft 42415sec 2024-09-16T01:40:09.157 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link noprefixroute 2024-09-16T01:40:09.157 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:40:09.157 INFO:teuthology.orchestra.run.smithi088.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:40:09.157 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff 2024-09-16T01:40:09.157 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T01:40:09.157 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:40:09.157 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::e486:6eff:fe81:a1c/64 scope link 2024-09-16T01:40:09.157 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:40:09.157 INFO:teuthology.orchestra.run.smithi088.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:40:09.157 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 12:6f:29:be:95:48 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:40:09.157 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::106f:29ff:febe:9548/64 scope link 2024-09-16T01:40:09.157 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:40:09.158 INFO:teuthology.orchestra.run.smithi088.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:40:09.158 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T01:40:09.158 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::c63:b4ff:feb3:ad9e/64 scope link 2024-09-16T01:40:09.158 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:40:09.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:09.158 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:40:09.218 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-09-16T01:40:09.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:09.220 DEBUG:teuthology.orchestra.run.smithi088:> route 2024-09-16T01:40:09.276 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2024-09-16T01:40:09.276 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:40:09.277 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:40:09.277 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:40:09.277 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:40:09.277 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:40:09.277 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:40:09.277 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:40:09.277 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:40:09.277 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:40:09.277 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:40:09.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:40:09.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:40:09.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:09.403 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:40:09.418 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T01:40:09.419 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T01:40:09.419 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:40:09.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:09.419 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2024-09-16T01:40:09.484 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:40:09.484 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T01:40:09.540 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:40:09.540 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:40:09.540 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:40:09.595 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-16T01:40:09.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:09.595 DEBUG:teuthology.orchestra.run.smithi088:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:40:09.678 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:40:09.678 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:40:09.678 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:40:09.686 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:40:09.687 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:40:09.687 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:40:09.687 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:40:09.687 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:40:09.697 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:40:09.697 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:40:09.697 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:40:09.697 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:40:09.697 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:40:09.697 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790" 2024-09-16T01:40:09.697 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2024-09-16T01:40:09.697 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5ace2c91-46f9-482b-8a89-731fbdc171bf" will pass to kernel 2024-09-16T01:40:09.702 INFO:teuthology.orchestra.run.smithi088.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-16T01:40:09.702 INFO:teuthology.orchestra.run.smithi088.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:40:09.702 INFO:teuthology.orchestra.run.smithi088.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:40:09.702 INFO:teuthology.orchestra.run.smithi088.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:40:09.702 INFO:teuthology.orchestra.run.smithi088.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:40:09.702 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:40:09.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:09.703 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:40:09.767 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:40:09.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:09 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:40:09.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:40:09.867 INFO:teuthology.orchestra.run.smithi088.stdout:client6091 2024-09-16T01:40:09.868 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:40:09.868 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:40:09.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:09.883 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir) 2024-09-16T01:40:09.943 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:40:09.943 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:40:09.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:09.999 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=subdir/sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-09-16T01:40:10.098 INFO:teuthology.orchestra.run.smithi088.stderr:6+0 records in 2024-09-16T01:40:10.098 INFO:teuthology.orchestra.run.smithi088.stderr:6+0 records out 2024-09-16T01:40:10.098 INFO:teuthology.orchestra.run.smithi088.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.0423178 s, 149 MB/s 2024-09-16T01:40:10.099 DEBUG:teuthology.orchestra.run.smithi088:> stdin-killer -- python3 -c ' 2024-09-16T01:40:10.099 DEBUG:teuthology.orchestra.run.smithi088:> import os 2024-09-16T01:40:10.099 DEBUG:teuthology.orchestra.run.smithi088:> import stat 2024-09-16T01:40:10.099 DEBUG:teuthology.orchestra.run.smithi088:> import json 2024-09-16T01:40:10.099 DEBUG:teuthology.orchestra.run.smithi088:> import sys 2024-09-16T01:40:10.099 DEBUG:teuthology.orchestra.run.smithi088:> 2024-09-16T01:40:10.099 DEBUG:teuthology.orchestra.run.smithi088:> try: 2024-09-16T01:40:10.099 DEBUG:teuthology.orchestra.run.smithi088:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/sixmegs'"'"') 2024-09-16T01:40:10.099 DEBUG:teuthology.orchestra.run.smithi088:> except OSError as e: 2024-09-16T01:40:10.099 DEBUG:teuthology.orchestra.run.smithi088:> sys.exit(e.errno) 2024-09-16T01:40:10.100 DEBUG:teuthology.orchestra.run.smithi088:> 2024-09-16T01:40:10.100 DEBUG:teuthology.orchestra.run.smithi088:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-09-16T01:40:10.100 DEBUG:teuthology.orchestra.run.smithi088:> print(json.dumps( 2024-09-16T01:40:10.100 DEBUG:teuthology.orchestra.run.smithi088:> dict([(a, getattr(s, a)) for a in attrs]), 2024-09-16T01:40:10.100 DEBUG:teuthology.orchestra.run.smithi088:> indent=2)) 2024-09-16T01:40:10.100 DEBUG:teuthology.orchestra.run.smithi088:> ' 2024-09-16T01:40:10.167 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:40:10.167 INFO:teuthology.orchestra.run.smithi088.stdout: "st_mode": 33188, 2024-09-16T01:40:10.167 INFO:teuthology.orchestra.run.smithi088.stdout: "st_ino": 1099511627777, 2024-09-16T01:40:10.167 INFO:teuthology.orchestra.run.smithi088.stdout: "st_dev": 42, 2024-09-16T01:40:10.167 INFO:teuthology.orchestra.run.smithi088.stdout: "st_nlink": 1, 2024-09-16T01:40:10.167 INFO:teuthology.orchestra.run.smithi088.stdout: "st_uid": 1000, 2024-09-16T01:40:10.168 INFO:teuthology.orchestra.run.smithi088.stdout: "st_gid": 1333, 2024-09-16T01:40:10.168 INFO:teuthology.orchestra.run.smithi088.stdout: "st_size": 6291456, 2024-09-16T01:40:10.168 INFO:teuthology.orchestra.run.smithi088.stdout: "st_atime": 1726450810.0533173, 2024-09-16T01:40:10.168 INFO:teuthology.orchestra.run.smithi088.stdout: "st_mtime": 1726450810.073317, 2024-09-16T01:40:10.168 INFO:teuthology.orchestra.run.smithi088.stdout: "st_ctime": 1726450810.073317 2024-09-16T01:40:10.168 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:40:10.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:40:10.176 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:40:10.176 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:40:10.232 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:40:10.232 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:40:10.248 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:40:10.246+0000 7f928a9d9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:40:10.264 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:40:10.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.406+0000 7fd01cf68640 1 Processor -- start 2024-09-16T01:40:10.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.406+0000 7fd01cf68640 1 -- start start 2024-09-16T01:40:10.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.407+0000 7fd01cf68640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd01815b480 0x7fd0180b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:10.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.407+0000 7fd01cf68640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd0180b60c0 0x7fd018071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:10.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.407+0000 7fd01cf68640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0180ccff0 con 0x7fd018072530 2024-09-16T01:40:10.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.407+0000 7fd01cf68640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd018072980 con 0x7fd01815b480 2024-09-16T01:40:10.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.407+0000 7fd01cf68640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd0180cd2d0 con 0x7fd0180b60c0 2024-09-16T01:40:10.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.407+0000 7fd01659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd01815b480 0x7fd0180b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:10.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.407+0000 7fd016d9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd018072530 0x7fd018070fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57810/0 (socket says 172.21.15.88:57810) 2024-09-16T01:40:10.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.407+0000 7fd01659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd01815b480 0x7fd0180b5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:45904/0 (socket says 172.21.15.88:45904) 2024-09-16T01:40:10.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.407+0000 7fd016d9d640 1 -- 172.21.15.88:0/1172815773 learned_addr learned my addr 172.21.15.88:0/1172815773 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:10.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.407+0000 7fd015d9b640 1 --2- 172.21.15.88:0/1172815773 >> v2:172.21.15.154:3300/0 conn(0x7fd0180b60c0 0x7fd018071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:10.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.407+0000 7fd01659c640 1 -- 172.21.15.88:0/1172815773 >> v2:172.21.15.154:3300/0 conn(0x7fd0180b60c0 msgr2=0x7fd018071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:10.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.407+0000 7fd01659c640 1 --2- 172.21.15.88:0/1172815773 >> v2:172.21.15.154:3300/0 conn(0x7fd0180b60c0 0x7fd018071ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:10.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.407+0000 7fd01659c640 1 -- 172.21.15.88:0/1172815773 >> v1:172.21.15.88:6789/0 conn(0x7fd018072530 legacy=0x7fd018070fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:10.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.408+0000 7fd01659c640 1 -- 172.21.15.88:0/1172815773 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0180cd5b0 con 0x7fd01815b480 2024-09-16T01:40:10.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.407+0000 7fd01559a640 1 -- 172.21.15.88:0/1172815773 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1624568707 0 0) 0x7fd0180ccff0 con 0x7fd018072530 2024-09-16T01:40:10.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.408+0000 7fd01659c640 1 --2- 172.21.15.88:0/1172815773 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd01815b480 0x7fd0180b5b80 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fd0180b2ee0 tx=0x7fd00002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dee62fea39124e6a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:10.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.408+0000 7fd01559a640 1 -- 172.21.15.88:0/1172815773 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd000037070 con 0x7fd01815b480 2024-09-16T01:40:10.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.408+0000 7fd01559a640 1 -- 172.21.15.88:0/1172815773 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd000032440 con 0x7fd01815b480 2024-09-16T01:40:10.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.408+0000 7fd01559a640 1 -- 172.21.15.88:0/1172815773 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd000032810 con 0x7fd01815b480 2024-09-16T01:40:10.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.409+0000 7fd01cf68640 1 -- 172.21.15.88:0/1172815773 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd01815b480 msgr2=0x7fd0180b5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:10.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.409+0000 7fd01cf68640 1 --2- 172.21.15.88:0/1172815773 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd01815b480 0x7fd0180b5b80 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fd0180b2ee0 tx=0x7fd00002a1a0 comp rx=0 tx=0).stop 2024-09-16T01:40:10.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.409+0000 7fd01cf68640 1 -- 172.21.15.88:0/1172815773 shutdown_connections 2024-09-16T01:40:10.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.409+0000 7fd01cf68640 1 --2- 172.21.15.88:0/1172815773 >> v2:172.21.15.154:3300/0 conn(0x7fd0180b60c0 0x7fd018071ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:10.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.409+0000 7fd01cf68640 1 --2- 172.21.15.88:0/1172815773 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd01815b480 0x7fd0180b5b80 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:10.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.409+0000 7fd01cf68640 1 -- 172.21.15.88:0/1172815773 >> 172.21.15.88:0/1172815773 conn(0x7fd01806d0f0 msgr2=0x7fd01806d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:10.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.409+0000 7fd01cf68640 1 -- 172.21.15.88:0/1172815773 shutdown_connections 2024-09-16T01:40:10.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.409+0000 7fd01cf68640 1 -- 172.21.15.88:0/1172815773 wait complete. 2024-09-16T01:40:10.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.411+0000 7fd01cf68640 1 Processor -- start 2024-09-16T01:40:10.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.411+0000 7fd01cf68640 1 -- start start 2024-09-16T01:40:10.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.411+0000 7fd01cf68640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd018072530 0x7fd0180d0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:10.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.411+0000 7fd01cf68640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0180b60c0 0x7fd018148c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:10.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.411+0000 7fd01cf68640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd01815b480 0x7fd01815d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:10.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.411+0000 7fd01cf68640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd01815b860 con 0x7fd01815b480 2024-09-16T01:40:10.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.411+0000 7fd01cf68640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd0180cdb70 con 0x7fd0180b60c0 2024-09-16T01:40:10.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.411+0000 7fd01cf68640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd01815b9e0 con 0x7fd018072530 2024-09-16T01:40:10.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.411+0000 7fd016d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd01815b480 0x7fd01815d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:10.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.411+0000 7fd016d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd01815b480 0x7fd01815d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:45910/0 (socket says 172.21.15.88:45910) 2024-09-16T01:40:10.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.411+0000 7fd016d9d640 1 -- 172.21.15.88:0/1024411397 learned_addr learned my addr 172.21.15.88:0/1024411397 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:10.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.412+0000 7fd015d9b640 1 --2- 172.21.15.88:0/1024411397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0180b60c0 0x7fd018148c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:10.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.412+0000 7fd01659c640 1 --2- 172.21.15.88:0/1024411397 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd018072530 0x7fd0180d0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:10.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.412+0000 7fd016d9d640 1 -- 172.21.15.88:0/1024411397 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd018072530 msgr2=0x7fd0180d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:10.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.412+0000 7fd016d9d640 1 --2- 172.21.15.88:0/1024411397 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd018072530 0x7fd0180d0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:10.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.412+0000 7fd016d9d640 1 -- 172.21.15.88:0/1024411397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0180b60c0 msgr2=0x7fd018148c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:10.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.412+0000 7fd016d9d640 1 --2- 172.21.15.88:0/1024411397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0180b60c0 0x7fd018148c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:10.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.412+0000 7fd016d9d640 1 -- 172.21.15.88:0/1024411397 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd018149f50 con 0x7fd01815b480 2024-09-16T01:40:10.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.412+0000 7fd016d9d640 1 --2- 172.21.15.88:0/1024411397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd01815b480 0x7fd01815d2e0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fd01000c7d0 tx=0x7fd01000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:10.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.412+0000 7fd00effd640 1 -- 172.21.15.88:0/1024411397 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd010018070 con 0x7fd01815b480 2024-09-16T01:40:10.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.412+0000 7fd00effd640 1 -- 172.21.15.88:0/1024411397 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd01000a640 con 0x7fd01815b480 2024-09-16T01:40:10.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.412+0000 7fd01cf68640 1 -- 172.21.15.88:0/1024411397 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd01814a230 con 0x7fd01815b480 2024-09-16T01:40:10.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.412+0000 7fd00effd640 1 -- 172.21.15.88:0/1024411397 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd01000aa70 con 0x7fd01815b480 2024-09-16T01:40:10.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.412+0000 7fd01cf68640 1 -- 172.21.15.88:0/1024411397 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd018146130 con 0x7fd01815b480 2024-09-16T01:40:10.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.413+0000 7fd01cf68640 1 -- 172.21.15.88:0/1024411397 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd01814a750 con 0x7fd01815b480 2024-09-16T01:40:10.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.414+0000 7fd00effd640 1 -- 172.21.15.88:0/1024411397 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd01001b070 con 0x7fd01815b480 2024-09-16T01:40:10.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.415+0000 7fd00effd640 1 --2- 172.21.15.88:0/1024411397 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcff806c2b0 0x7fcff806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:10.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.415+0000 7fd00effd640 1 -- 172.21.15.88:0/1024411397 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7fd01008a9b0 con 0x7fd01815b480 2024-09-16T01:40:10.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.415+0000 7fd01659c640 1 --2- 172.21.15.88:0/1024411397 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcff806c2b0 0x7fcff806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:10.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.415+0000 7fd01659c640 1 --2- 172.21.15.88:0/1024411397 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcff806c2b0 0x7fcff806e770 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd000034000 tx=0x7fd00002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:10.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.419+0000 7fd00effd640 1 -- 172.21.15.88:0/1024411397 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd01814a750 con 0x7fd01815b480 2024-09-16T01:40:10.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.601+0000 7fd01cf68640 1 -- 172.21.15.88:0/1024411397 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd018071ff0 con 0x7fd01815b480 2024-09-16T01:40:10.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.602+0000 7fd00effd640 1 -- 172.21.15.88:0/1024411397 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v85) ==== 80+0+0 (secure 0 0 0) 0x7fd010058ad0 con 0x7fd01815b480 2024-09-16T01:40:10.603 INFO:teuthology.orchestra.run.smithi088.stderr:listed 0 entries 2024-09-16T01:40:10.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.604+0000 7fd01cf68640 1 -- 172.21.15.88:0/1024411397 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcff806c2b0 msgr2=0x7fcff806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:10.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.604+0000 7fd01cf68640 1 --2- 172.21.15.88:0/1024411397 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcff806c2b0 0x7fcff806e770 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd000034000 tx=0x7fd00002e000 comp rx=0 tx=0).stop 2024-09-16T01:40:10.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.604+0000 7fd01cf68640 1 -- 172.21.15.88:0/1024411397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd01815b480 msgr2=0x7fd01815d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:10.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.604+0000 7fd01cf68640 1 --2- 172.21.15.88:0/1024411397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd01815b480 0x7fd01815d2e0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fd01000c7d0 tx=0x7fd01000cc90 comp rx=0 tx=0).stop 2024-09-16T01:40:10.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.604+0000 7fd01cf68640 1 -- 172.21.15.88:0/1024411397 shutdown_connections 2024-09-16T01:40:10.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.604+0000 7fd01cf68640 1 --2- 172.21.15.88:0/1024411397 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcff806c2b0 0x7fcff806e770 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:10.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.604+0000 7fd01cf68640 1 --2- 172.21.15.88:0/1024411397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd01815b480 0x7fd01815d2e0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:10.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.604+0000 7fd01cf68640 1 --2- 172.21.15.88:0/1024411397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0180b60c0 0x7fd018148c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:10.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.604+0000 7fd01cf68640 1 --2- 172.21.15.88:0/1024411397 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd018072530 0x7fd0180d0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:10.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.604+0000 7fd01cf68640 1 -- 172.21.15.88:0/1024411397 >> 172.21.15.88:0/1024411397 conn(0x7fd01806d0f0 msgr2=0x7fd0180b46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:10.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.604+0000 7fd01cf68640 1 -- 172.21.15.88:0/1024411397 shutdown_connections 2024-09-16T01:40:10.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.604+0000 7fd01cf68640 1 -- 172.21.15.88:0/1024411397 wait complete. 2024-09-16T01:40:10.616 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-16T01:40:10.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:10.616 DEBUG:teuthology.orchestra.run.smithi088:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-16T01:40:10.656 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:40:10.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:10.656 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:40:10.711 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:40:10.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.849+0000 7f67ead49640 1 Processor -- start 2024-09-16T01:40:10.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.849+0000 7f67ead49640 1 -- start start 2024-09-16T01:40:10.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.850+0000 7f67ead49640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67e40d3310 0x7f67e40d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:10.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.850+0000 7f67ead49640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67e415ea40 con 0x7f67e40d1650 2024-09-16T01:40:10.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.850+0000 7f67ead49640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67e40d3870 con 0x7f67e415e660 2024-09-16T01:40:10.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.850+0000 7f67ead49640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f67e40d1ac0 con 0x7f67e40d3310 2024-09-16T01:40:10.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.850+0000 7f67e92e6640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f67e40d1650 0x7f67e40b7b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57818/0 (socket says 172.21.15.88:57818) 2024-09-16T01:40:10.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.850+0000 7f67e92e6640 1 -- 172.21.15.88:0/872520901 learned_addr learned my addr 172.21.15.88:0/872520901 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:10.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.850+0000 7f67e3fff640 1 --2- 172.21.15.88:0/872520901 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67e40d3310 0x7f67e40d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:10.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.850+0000 7f67e37fe640 1 -- 172.21.15.88:0/872520901 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3075950278 0 0) 0x7f67e415ea40 con 0x7f67e40d1650 2024-09-16T01:40:10.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.850+0000 7f67e37fe640 1 -- 172.21.15.88:0/872520901 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67d4003520 con 0x7f67e40d1650 2024-09-16T01:40:10.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.850+0000 7f67e37fe640 1 -- 172.21.15.88:0/872520901 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1572253672 0 0) 0x7f67e40d3870 con 0x7f67e415e660 2024-09-16T01:40:10.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.850+0000 7f67e37fe640 1 -- 172.21.15.88:0/872520901 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67e415ea40 con 0x7f67e415e660 2024-09-16T01:40:10.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.850+0000 7f67e37fe640 1 -- 172.21.15.88:0/872520901 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 170659008 0 0) 0x7f67d4003520 con 0x7f67e40d1650 2024-09-16T01:40:10.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.850+0000 7f67e37fe640 1 -- 172.21.15.88:0/872520901 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67e40d3310 msgr2=0x7f67e40d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:10.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.850+0000 7f67e37fe640 1 --2- 172.21.15.88:0/872520901 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67e40d3310 0x7f67e40d1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:10.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.850+0000 7f67e37fe640 1 -- 172.21.15.88:0/872520901 >> v1:172.21.15.154:6789/0 conn(0x7f67e415e660 legacy=0x7f67e40cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:10.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.850+0000 7f67e37fe640 1 -- 172.21.15.88:0/872520901 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67e40d3b50 con 0x7f67e40d1650 2024-09-16T01:40:10.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.850+0000 7f67e37fe640 1 -- 172.21.15.88:0/872520901 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f67dc0019f0 con 0x7f67e40d1650 2024-09-16T01:40:10.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.850+0000 7f67e3fff640 1 --2- 172.21.15.88:0/872520901 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67e40d3310 0x7f67e40d1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:40:10.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.850+0000 7f67e37fe640 1 -- 172.21.15.88:0/872520901 >> v1:172.21.15.88:6789/0 conn(0x7f67e40d1650 legacy=0x7f67e40b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:10.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.851+0000 7f67e37fe640 1 --2- 172.21.15.88:0/872520901 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67e40d3310 0x7f67d4003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:10.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.851+0000 7f67e37fe640 1 --2- 172.21.15.88:0/872520901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67d40040b0 0x7f67d4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:10.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.851+0000 7f67e37fe640 1 --2- 172.21.15.88:0/872520901 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67d4004d40 0x7f67d40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:10.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.851+0000 7f67e37fe640 1 -- 172.21.15.88:0/872520901 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f67e40d1ac0 con 0x7f67d4004d40 2024-09-16T01:40:10.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.851+0000 7f67e37fe640 1 -- 172.21.15.88:0/872520901 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f67d400ba70 con 0x7f67d40040b0 2024-09-16T01:40:10.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.851+0000 7f67e37fe640 1 -- 172.21.15.88:0/872520901 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f67d400bc70 con 0x7f67e40d3310 2024-09-16T01:40:10.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.851+0000 7f67e92e6640 1 --2- 172.21.15.88:0/872520901 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67d4004d40 0x7f67d40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:10.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.851+0000 7f67e8ae5640 1 --2- 172.21.15.88:0/872520901 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67e40d3310 0x7f67d4003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:10.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.851+0000 7f67e3fff640 1 --2- 172.21.15.88:0/872520901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67d40040b0 0x7f67d4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:10.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.851+0000 7f67e92e6640 1 -- 172.21.15.88:0/872520901 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67e40d3310 msgr2=0x7f67d4003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:10.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.851+0000 7f67e92e6640 1 --2- 172.21.15.88:0/872520901 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67e40d3310 0x7f67d4003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:10.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.851+0000 7f67e92e6640 1 -- 172.21.15.88:0/872520901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67d40040b0 msgr2=0x7f67d4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:10.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.851+0000 7f67e92e6640 1 --2- 172.21.15.88:0/872520901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67d40040b0 0x7f67d4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:10.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.851+0000 7f67e92e6640 1 -- 172.21.15.88:0/872520901 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d4003520 con 0x7f67d4004d40 2024-09-16T01:40:10.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.852+0000 7f67e92e6640 1 --2- 172.21.15.88:0/872520901 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67d4004d40 0x7f67d40051c0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f67dc00a7c0 tx=0x7f67d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecbd7e70667de2fe server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:10.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.852+0000 7f67e37fe640 1 -- 172.21.15.88:0/872520901 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc00a410 con 0x7f67d4004d40 2024-09-16T01:40:10.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.852+0000 7f67e37fe640 1 -- 172.21.15.88:0/872520901 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67dc02f070 con 0x7f67d4004d40 2024-09-16T01:40:10.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.852+0000 7f67e37fe640 1 -- 172.21.15.88:0/872520901 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc02bc20 con 0x7f67d4004d40 2024-09-16T01:40:10.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.852+0000 7f67ead49640 1 -- 172.21.15.88:0/872520901 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67d4004d40 msgr2=0x7f67d40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:10.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.852+0000 7f67ead49640 1 --2- 172.21.15.88:0/872520901 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67d4004d40 0x7f67d40051c0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f67dc00a7c0 tx=0x7f67d4005830 comp rx=0 tx=0).stop 2024-09-16T01:40:10.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.852+0000 7f67e92e6640 1 -- 172.21.15.88:0/872520901 reap_dead start 2024-09-16T01:40:10.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.852+0000 7f67ead49640 1 -- 172.21.15.88:0/872520901 shutdown_connections 2024-09-16T01:40:10.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.852+0000 7f67ead49640 1 -- 172.21.15.88:0/872520901 >> 172.21.15.88:0/872520901 conn(0x7f67e40b6410 msgr2=0x7f67e40b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:10.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.852+0000 7f67ead49640 1 -- 172.21.15.88:0/872520901 shutdown_connections 2024-09-16T01:40:10.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.852+0000 7f67ead49640 1 -- 172.21.15.88:0/872520901 wait complete. 2024-09-16T01:40:10.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.854+0000 7f67ead49640 1 Processor -- start 2024-09-16T01:40:10.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.854+0000 7f67ead49640 1 -- start start 2024-09-16T01:40:10.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.854+0000 7f67ead49640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67d4004d40 0x7f67e4151c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:10.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.854+0000 7f67ead49640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67e415fcd0 0x7f67e4160150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:10.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.854+0000 7f67ead49640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67e4160690 0x7f67e40c9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:10.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.855+0000 7f67ead49640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f67e40d43f0 con 0x7f67e4160690 2024-09-16T01:40:10.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.855+0000 7f67ead49640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f67e40d4110 con 0x7f67d4004d40 2024-09-16T01:40:10.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.855+0000 7f67ead49640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f67e40d1110 con 0x7f67e415fcd0 2024-09-16T01:40:10.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.855+0000 7f67e92e6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67e4160690 0x7f67e40c9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:10.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.855+0000 7f67e92e6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67e4160690 0x7f67e40c9400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:45934/0 (socket says 172.21.15.88:45934) 2024-09-16T01:40:10.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.855+0000 7f67e92e6640 1 -- 172.21.15.88:0/1025164800 learned_addr learned my addr 172.21.15.88:0/1025164800 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:10.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.855+0000 7f67e8ae5640 1 --2- 172.21.15.88:0/1025164800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67d4004d40 0x7f67e4151c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:10.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.855+0000 7f67e3fff640 1 --2- 172.21.15.88:0/1025164800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67e415fcd0 0x7f67e4160150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:10.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.855+0000 7f67e92e6640 1 -- 172.21.15.88:0/1025164800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67e415fcd0 msgr2=0x7f67e4160150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:10.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.855+0000 7f67e92e6640 1 --2- 172.21.15.88:0/1025164800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67e415fcd0 0x7f67e4160150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:10.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.855+0000 7f67e92e6640 1 -- 172.21.15.88:0/1025164800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67d4004d40 msgr2=0x7f67e4151c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:10.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.855+0000 7f67e92e6640 1 --2- 172.21.15.88:0/1025164800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67d4004d40 0x7f67e4151c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:10.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.855+0000 7f67e92e6640 1 -- 172.21.15.88:0/1025164800 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67e4160b10 con 0x7f67e4160690 2024-09-16T01:40:10.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.855+0000 7f67e92e6640 1 --2- 172.21.15.88:0/1025164800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67e4160690 0x7f67e40c9400 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f67dc02b470 tx=0x7f67e40d3a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:10.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.855+0000 7f67e17fa640 1 -- 172.21.15.88:0/1025164800 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc044070 con 0x7f67e4160690 2024-09-16T01:40:10.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.855+0000 7f67e17fa640 1 -- 172.21.15.88:0/1025164800 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67dc00a3f0 con 0x7f67e4160690 2024-09-16T01:40:10.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.855+0000 7f67ead49640 1 -- 172.21.15.88:0/1025164800 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67e40c8360 con 0x7f67e4160690 2024-09-16T01:40:10.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.855+0000 7f67e17fa640 1 -- 172.21.15.88:0/1025164800 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc001f70 con 0x7f67e4160690 2024-09-16T01:40:10.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.855+0000 7f67ead49640 1 -- 172.21.15.88:0/1025164800 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67e40c8920 con 0x7f67e4160690 2024-09-16T01:40:10.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.857+0000 7f67ead49640 1 -- 172.21.15.88:0/1025164800 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67e40cf730 con 0x7f67e4160690 2024-09-16T01:40:10.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.857+0000 7f67e17fa640 1 -- 172.21.15.88:0/1025164800 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f67dc047070 con 0x7f67e4160690 2024-09-16T01:40:10.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.857+0000 7f67e17fa640 1 --2- 172.21.15.88:0/1025164800 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f67c806c2b0 0x7f67c806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:10.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.857+0000 7f67e17fa640 1 -- 172.21.15.88:0/1025164800 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7f67dc0ab270 con 0x7f67e4160690 2024-09-16T01:40:10.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.857+0000 7f67e8ae5640 1 --2- 172.21.15.88:0/1025164800 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f67c806c2b0 0x7f67c806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:10.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.858+0000 7f67e8ae5640 1 --2- 172.21.15.88:0/1025164800 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f67c806c2b0 0x7f67c806e770 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f67e415bc00 tx=0x7f67d8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:10.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:10.861+0000 7f67e17fa640 1 -- 172.21.15.88:0/1025164800 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f67dc0794e0 con 0x7f67e4160690 2024-09-16T01:40:11.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.067+0000 7f67ead49640 1 -- 172.21.15.88:0/1025164800 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67e4159a20 con 0x7f67e4160690 2024-09-16T01:40:11.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.068+0000 7f67e17fa640 1 -- 172.21.15.88:0/1025164800 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) ==== 94+0+5228 (secure 0 0 0) 0x7f67dc07d7e0 con 0x7f67e4160690 2024-09-16T01:40:11.070 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:11.070 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":70,"btime":"2024-09-16T01:40:06:272351+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6849,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/1289932173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1289932173},{"type":"v1","addr":"172.21.15.88:6836","nonce":1289932173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:40:05.602153+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6852":{"gid":6852,"name":"c","rank":0,"incarnation":68,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6837/4162274767","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4162274767},{"type":"v1","addr":"172.21.15.88:6837","nonce":4162274767}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6852,"qdb_cluster":[6852]},"id":6}]} 2024-09-16T01:40:11.070 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 70 2024-09-16T01:40:11.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.070+0000 7f67ead49640 1 -- 172.21.15.88:0/1025164800 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f67c806c2b0 msgr2=0x7f67c806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.070+0000 7f67ead49640 1 --2- 172.21.15.88:0/1025164800 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f67c806c2b0 0x7f67c806e770 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f67e415bc00 tx=0x7f67d8009040 comp rx=0 tx=0).stop 2024-09-16T01:40:11.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.070+0000 7f67ead49640 1 -- 172.21.15.88:0/1025164800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67e4160690 msgr2=0x7f67e40c9400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.070+0000 7f67ead49640 1 --2- 172.21.15.88:0/1025164800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67e4160690 0x7f67e40c9400 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f67dc02b470 tx=0x7f67e40d3a40 comp rx=0 tx=0).stop 2024-09-16T01:40:11.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.070+0000 7f67ead49640 1 -- 172.21.15.88:0/1025164800 shutdown_connections 2024-09-16T01:40:11.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.070+0000 7f67ead49640 1 --2- 172.21.15.88:0/1025164800 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f67c806c2b0 0x7f67c806e770 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.070+0000 7f67ead49640 1 --2- 172.21.15.88:0/1025164800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67e4160690 0x7f67e40c9400 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.070+0000 7f67ead49640 1 --2- 172.21.15.88:0/1025164800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67e415fcd0 0x7f67e4160150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.070+0000 7f67ead49640 1 --2- 172.21.15.88:0/1025164800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67d4004d40 0x7f67e4151c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.070+0000 7f67ead49640 1 -- 172.21.15.88:0/1025164800 >> 172.21.15.88:0/1025164800 conn(0x7f67e40b6410 msgr2=0x7f67e415cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:11.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.070+0000 7f67ead49640 1 -- 172.21.15.88:0/1025164800 shutdown_connections 2024-09-16T01:40:11.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.070+0000 7f67ead49640 1 -- 172.21.15.88:0/1025164800 wait complete. 2024-09-16T01:40:11.082 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-09-16T01:40:11.209 INFO:teuthology.orchestra.run.smithi088.stdout:{"message":"","return_code":0} 2024-09-16T01:40:11.209 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-16T01:40:11.209 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:40:11.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.309+0000 7fc12a1f8640 1 Processor -- start 2024-09-16T01:40:11.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.310+0000 7fc12a1f8640 1 -- start start 2024-09-16T01:40:11.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.310+0000 7fc12a1f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc12415c260 0x7fc1240d1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.310+0000 7fc12a1f8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc1240b3570 0x7fc1240b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.310+0000 7fc12a1f8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc124072c70 con 0x7fc124071f80 2024-09-16T01:40:11.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.310+0000 7fc12a1f8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc1240d21b0 con 0x7fc1240b3570 2024-09-16T01:40:11.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.310+0000 7fc12a1f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc124072f50 con 0x7fc12415c260 2024-09-16T01:40:11.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.310+0000 7fc11affd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc1240b3570 0x7fc1240b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.310+0000 7fc11affd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc1240b3570 0x7fc1240b39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:45938/0 (socket says 172.21.15.88:45938) 2024-09-16T01:40:11.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.311+0000 7fc11affd640 1 -- 172.21.15.88:0/3213584956 learned_addr learned my addr 172.21.15.88:0/3213584956 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:11.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.311+0000 7fc1237fe640 1 --2- 172.21.15.88:0/3213584956 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc12415c260 0x7fc1240d1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.311+0000 7fc11affd640 1 -- 172.21.15.88:0/3213584956 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc12415c260 msgr2=0x7fc1240d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:11.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.311+0000 7fc11affd640 1 --2- 172.21.15.88:0/3213584956 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc12415c260 0x7fc1240d1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.311+0000 7fc11affd640 1 -- 172.21.15.88:0/3213584956 >> v1:172.21.15.154:6789/0 conn(0x7fc124071f80 legacy=0x7fc124072430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:11.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.311+0000 7fc11affd640 1 -- 172.21.15.88:0/3213584956 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1240d1190 con 0x7fc1240b3570 2024-09-16T01:40:11.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.311+0000 7fc11affd640 1 --2- 172.21.15.88:0/3213584956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1240b3570 0x7fc1240b39b0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fc10c009870 tx=0x7fc10c02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3586701f4468152 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.312+0000 7fc122ffd640 1 -- 172.21.15.88:0/3213584956 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc10c037070 con 0x7fc1240b3570 2024-09-16T01:40:11.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.312+0000 7fc122ffd640 1 -- 172.21.15.88:0/3213584956 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc10c02ae10 con 0x7fc1240b3570 2024-09-16T01:40:11.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.312+0000 7fc122ffd640 1 -- 172.21.15.88:0/3213584956 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc10c032660 con 0x7fc1240b3570 2024-09-16T01:40:11.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.312+0000 7fc12a1f8640 1 -- 172.21.15.88:0/3213584956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1240b3570 msgr2=0x7fc1240b39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.312+0000 7fc12a1f8640 1 --2- 172.21.15.88:0/3213584956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1240b3570 0x7fc1240b39b0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fc10c009870 tx=0x7fc10c02a100 comp rx=0 tx=0).stop 2024-09-16T01:40:11.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.312+0000 7fc12a1f8640 1 -- 172.21.15.88:0/3213584956 shutdown_connections 2024-09-16T01:40:11.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.312+0000 7fc12a1f8640 1 --2- 172.21.15.88:0/3213584956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1240b3570 0x7fc1240b39b0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.312+0000 7fc12a1f8640 1 --2- 172.21.15.88:0/3213584956 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc12415c260 0x7fc1240d1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.312+0000 7fc12a1f8640 1 -- 172.21.15.88:0/3213584956 >> 172.21.15.88:0/3213584956 conn(0x7fc1240b6b50 msgr2=0x7fc1240b6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:11.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.312+0000 7fc12a1f8640 1 -- 172.21.15.88:0/3213584956 shutdown_connections 2024-09-16T01:40:11.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.312+0000 7fc12a1f8640 1 -- 172.21.15.88:0/3213584956 wait complete. 2024-09-16T01:40:11.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.313+0000 7fc12a1f8640 1 Processor -- start 2024-09-16T01:40:11.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.314+0000 7fc12a1f8640 1 -- start start 2024-09-16T01:40:11.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.314+0000 7fc12a1f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc124071f80 0x7fc1241512e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.314+0000 7fc12a1f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1240b3570 0x7fc124155ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.314+0000 7fc12a1f8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc12415c260 0x7fc124147bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.314+0000 7fc12a1f8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc1240b3ef0 con 0x7fc12415c260 2024-09-16T01:40:11.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.314+0000 7fc12a1f8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc1240d17e0 con 0x7fc124071f80 2024-09-16T01:40:11.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.314+0000 7fc12a1f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc12415c640 con 0x7fc1240b3570 2024-09-16T01:40:11.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.314+0000 7fc123fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc12415c260 0x7fc124147bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.314+0000 7fc123fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc12415c260 0x7fc124147bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:45940/0 (socket says 172.21.15.88:45940) 2024-09-16T01:40:11.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.314+0000 7fc123fff640 1 -- 172.21.15.88:0/3892691460 learned_addr learned my addr 172.21.15.88:0/3892691460 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:11.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.314+0000 7fc11affd640 1 --2- 172.21.15.88:0/3892691460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1240b3570 0x7fc124155ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.314+0000 7fc1237fe640 1 --2- 172.21.15.88:0/3892691460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc124071f80 0x7fc1241512e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.315+0000 7fc123fff640 1 -- 172.21.15.88:0/3892691460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1240b3570 msgr2=0x7fc124155ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.315+0000 7fc123fff640 1 --2- 172.21.15.88:0/3892691460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1240b3570 0x7fc124155ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.315+0000 7fc123fff640 1 -- 172.21.15.88:0/3892691460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc124071f80 msgr2=0x7fc1241512e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.315+0000 7fc123fff640 1 --2- 172.21.15.88:0/3892691460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc124071f80 0x7fc1241512e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.315+0000 7fc123fff640 1 -- 172.21.15.88:0/3892691460 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc124158f00 con 0x7fc12415c260 2024-09-16T01:40:11.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.315+0000 7fc1237fe640 1 --2- 172.21.15.88:0/3892691460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc124071f80 0x7fc1241512e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:11.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.315+0000 7fc11affd640 1 --2- 172.21.15.88:0/3892691460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1240b3570 0x7fc124155ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:11.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.315+0000 7fc123fff640 1 --2- 172.21.15.88:0/3892691460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc12415c260 0x7fc124147bb0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fc11c00c780 tx=0x7fc11c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.315+0000 7fc120ff9640 1 -- 172.21.15.88:0/3892691460 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11c004b60 con 0x7fc12415c260 2024-09-16T01:40:11.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.315+0000 7fc120ff9640 1 -- 172.21.15.88:0/3892691460 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc11c004d00 con 0x7fc12415c260 2024-09-16T01:40:11.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.315+0000 7fc12a1f8640 1 -- 172.21.15.88:0/3892691460 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1241591e0 con 0x7fc12415c260 2024-09-16T01:40:11.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.315+0000 7fc120ff9640 1 -- 172.21.15.88:0/3892691460 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11c00aa90 con 0x7fc12415c260 2024-09-16T01:40:11.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.315+0000 7fc12a1f8640 1 -- 172.21.15.88:0/3892691460 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc124152a70 con 0x7fc12415c260 2024-09-16T01:40:11.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.316+0000 7fc12a1f8640 1 -- 172.21.15.88:0/3892691460 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1240b39b0 con 0x7fc12415c260 2024-09-16T01:40:11.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.317+0000 7fc120ff9640 1 -- 172.21.15.88:0/3892691460 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc11c00ac30 con 0x7fc12415c260 2024-09-16T01:40:11.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.317+0000 7fc120ff9640 1 --2- 172.21.15.88:0/3892691460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc10406c2b0 0x7fc10406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.317+0000 7fc120ff9640 1 -- 172.21.15.88:0/3892691460 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5709+0+0 (secure 0 0 0) 0x7fc11c08b660 con 0x7fc12415c260 2024-09-16T01:40:11.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.317+0000 7fc1237fe640 1 --2- 172.21.15.88:0/3892691460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc10406c2b0 0x7fc10406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.318+0000 7fc1237fe640 1 --2- 172.21.15.88:0/3892691460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc10406c2b0 0x7fc10406e770 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc114006fd0 tx=0x7fc114008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.321+0000 7fc120ff9640 1 -- 172.21.15.88:0/3892691460 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc11c020070 con 0x7fc12415c260 2024-09-16T01:40:11.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.520+0000 7fc12a1f8640 1 -- 172.21.15.88:0/3892691460 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc124073160 con 0x7fc12415c260 2024-09-16T01:40:11.522 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:40:11.521+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:40:11.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.522+0000 7fc120ff9640 1 -- 172.21.15.88:0/3892691460 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v71) ==== 161+0+0 (secure 0 0 0) 0x7fc11c059e60 con 0x7fc12415c260 2024-09-16T01:40:11.524 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:40:11.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.525+0000 7fc12a1f8640 1 -- 172.21.15.88:0/3892691460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc10406c2b0 msgr2=0x7fc10406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.525+0000 7fc12a1f8640 1 --2- 172.21.15.88:0/3892691460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc10406c2b0 0x7fc10406e770 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fc114006fd0 tx=0x7fc114008040 comp rx=0 tx=0).stop 2024-09-16T01:40:11.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.525+0000 7fc12a1f8640 1 -- 172.21.15.88:0/3892691460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc12415c260 msgr2=0x7fc124147bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.525+0000 7fc12a1f8640 1 --2- 172.21.15.88:0/3892691460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc12415c260 0x7fc124147bb0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fc11c00c780 tx=0x7fc11c00cc40 comp rx=0 tx=0).stop 2024-09-16T01:40:11.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.525+0000 7fc12a1f8640 1 -- 172.21.15.88:0/3892691460 shutdown_connections 2024-09-16T01:40:11.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.525+0000 7fc12a1f8640 1 --2- 172.21.15.88:0/3892691460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc10406c2b0 0x7fc10406e770 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.525+0000 7fc12a1f8640 1 --2- 172.21.15.88:0/3892691460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc12415c260 0x7fc124147bb0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.525+0000 7fc12a1f8640 1 --2- 172.21.15.88:0/3892691460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1240b3570 0x7fc124155ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.525+0000 7fc12a1f8640 1 --2- 172.21.15.88:0/3892691460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc124071f80 0x7fc1241512e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.525+0000 7fc12a1f8640 1 -- 172.21.15.88:0/3892691460 >> 172.21.15.88:0/3892691460 conn(0x7fc1240b6b50 msgr2=0x7fc1240b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:11.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.525+0000 7fc12a1f8640 1 -- 172.21.15.88:0/3892691460 shutdown_connections 2024-09-16T01:40:11.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.525+0000 7fc12a1f8640 1 -- 172.21.15.88:0/3892691460 wait complete. 2024-09-16T01:40:11.534 INFO:tasks.ceph.mds.c.smithi088.stderr: -1668> 2024-09-16T01:40:10.246+0000 7f928a9d9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:40:11.540 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata export - 2024-09-16T01:40:11.563 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:40:11.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.583+0000 7f0fd301cf40 1 Processor -- start 2024-09-16T01:40:11.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.583+0000 7f0fd301cf40 1 -- start start 2024-09-16T01:40:11.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.584+0000 7f0fd301cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fc0002810 0x5650f6734100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.584+0000 7f0fd301cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650f673c9f0 0x5650f673b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.584+0000 7f0fd301cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650f673b9d0 0x5650f666bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.584+0000 7f0fd301cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5650f65be510 con 0x7f0fc0002810 2024-09-16T01:40:11.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.584+0000 7f0fd301cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5650f6617c50 con 0x5650f673c9f0 2024-09-16T01:40:11.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.584+0000 7f0fd301cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5650f665b570 con 0x5650f673b9d0 2024-09-16T01:40:11.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.584+0000 7f0fd0b12640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fc0002810 0x5650f6734100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.584+0000 7f0fd0b12640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fc0002810 0x5650f6734100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:45992/0 (socket says 172.21.15.88:45992) 2024-09-16T01:40:11.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.584+0000 7f0fd0b12640 1 -- 172.21.15.88:0/2449411561 learned_addr learned my addr 172.21.15.88:0/2449411561 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:11.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.584+0000 7f0fcbfff640 1 --2- 172.21.15.88:0/2449411561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650f673c9f0 0x5650f673b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.584+0000 7f0fd1313640 1 --2- 172.21.15.88:0/2449411561 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650f673b9d0 0x5650f666bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.584+0000 7f0fd0b12640 1 -- 172.21.15.88:0/2449411561 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650f673b9d0 msgr2=0x5650f666bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:11.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.584+0000 7f0fd0b12640 1 --2- 172.21.15.88:0/2449411561 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650f673b9d0 0x5650f666bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.584+0000 7f0fd0b12640 1 -- 172.21.15.88:0/2449411561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650f673c9f0 msgr2=0x5650f673b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:11.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.584+0000 7f0fd0b12640 1 --2- 172.21.15.88:0/2449411561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650f673c9f0 0x5650f673b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.584+0000 7f0fd0b12640 1 -- 172.21.15.88:0/2449411561 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650f66b7490 con 0x7f0fc0002810 2024-09-16T01:40:11.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.584+0000 7f0fd0b12640 1 --2- 172.21.15.88:0/2449411561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fc0002810 0x5650f6734100 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f0fc400d560 tx=0x7f0fc400da20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=252019d3a87f5b04 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.584+0000 7f0fcaffd640 1 -- 172.21.15.88:0/2449411561 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4013070 con 0x7f0fc0002810 2024-09-16T01:40:11.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.584+0000 7f0fcaffd640 1 -- 172.21.15.88:0/2449411561 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fc400b770 con 0x7f0fc0002810 2024-09-16T01:40:11.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.584+0000 7f0fcaffd640 1 -- 172.21.15.88:0/2449411561 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc400bb60 con 0x7f0fc0002810 2024-09-16T01:40:11.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.585+0000 7f0fd301cf40 1 -- 172.21.15.88:0/2449411561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fc0002810 msgr2=0x5650f6734100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.585+0000 7f0fd301cf40 1 --2- 172.21.15.88:0/2449411561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fc0002810 0x5650f6734100 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f0fc400d560 tx=0x7f0fc400da20 comp rx=0 tx=0).stop 2024-09-16T01:40:11.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.585+0000 7f0fd301cf40 1 -- 172.21.15.88:0/2449411561 shutdown_connections 2024-09-16T01:40:11.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.585+0000 7f0fd301cf40 1 --2- 172.21.15.88:0/2449411561 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650f673b9d0 0x5650f666bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.585+0000 7f0fd301cf40 1 --2- 172.21.15.88:0/2449411561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650f673c9f0 0x5650f673b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.585+0000 7f0fd301cf40 1 --2- 172.21.15.88:0/2449411561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fc0002810 0x5650f6734100 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.585+0000 7f0fd301cf40 1 -- 172.21.15.88:0/2449411561 >> 172.21.15.88:0/2449411561 conn(0x5650f66b7750 msgr2=0x5650f6698b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:11.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.585+0000 7f0fd301cf40 1 -- 172.21.15.88:0/2449411561 shutdown_connections 2024-09-16T01:40:11.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.585+0000 7f0fd301cf40 1 -- 172.21.15.88:0/2449411561 wait complete. 2024-09-16T01:40:11.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.585+0000 7f0fd301cf40 1 Processor -- start 2024-09-16T01:40:11.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.585+0000 7f0fd301cf40 1 -- start start 2024-09-16T01:40:11.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.586+0000 7f0fd301cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650f673b9d0 0x5650f66b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.586+0000 7f0fd301cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650f673c9f0 0x5650f6743f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.586+0000 7f0fd301cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fc0002810 0x5650f67220f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.586+0000 7f0fd301cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5650f673dae0 con 0x7f0fc0002810 2024-09-16T01:40:11.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.586+0000 7f0fd301cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5650f673e130 con 0x5650f673c9f0 2024-09-16T01:40:11.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.586+0000 7f0fd301cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5650f673ddc0 con 0x5650f673b9d0 2024-09-16T01:40:11.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.586+0000 7f0fd1313640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fc0002810 0x5650f67220f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.586+0000 7f0fd1313640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fc0002810 0x5650f67220f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46004/0 (socket says 172.21.15.88:46004) 2024-09-16T01:40:11.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.586+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 learned_addr learned my addr 172.21.15.88:0/914797419 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:11.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.586+0000 7f0fd0b12640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650f673b9d0 0x5650f66b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.586+0000 7f0fcbfff640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650f673c9f0 0x5650f6743f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.586+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650f673b9d0 msgr2=0x5650f66b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.586+0000 7f0fd1313640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5650f673b9d0 0x5650f66b2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.586+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650f673c9f0 msgr2=0x5650f6743f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.586+0000 7f0fd1313640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5650f673c9f0 0x5650f6743f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.586+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650f6722630 con 0x7f0fc0002810 2024-09-16T01:40:11.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.587+0000 7f0fd1313640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fc0002810 0x5650f67220f0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x5650f6741530 tx=0x7f0fcc00e7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.587+0000 7f0fc9ffb640 1 -- 172.21.15.88:0/914797419 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc016020 con 0x7f0fc0002810 2024-09-16T01:40:11.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.587+0000 7f0fc9ffb640 1 -- 172.21.15.88:0/914797419 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fcc0123f0 con 0x7f0fc0002810 2024-09-16T01:40:11.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.587+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5650f66b1c80 con 0x7f0fc0002810 2024-09-16T01:40:11.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.587+0000 7f0fc9ffb640 1 -- 172.21.15.88:0/914797419 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc0127c0 con 0x7f0fc0002810 2024-09-16T01:40:11.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.587+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5650f66b2240 con 0x7f0fc0002810 2024-09-16T01:40:11.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.588+0000 7f0fc9ffb640 1 -- 172.21.15.88:0/914797419 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0fcc019020 con 0x7f0fc0002810 2024-09-16T01:40:11.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.588+0000 7f0fc9ffb640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0fb406ac90 0x7f0fb406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.589+0000 7f0fd0b12640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0fb406ac90 0x7f0fb406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.589+0000 7f0fc9ffb640 1 -- 172.21.15.88:0/914797419 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f0fcc089940 con 0x7f0fc0002810 2024-09-16T01:40:11.594 INFO:teuthology.orchestra.run.smithi088.stdout:���� 2024-09-16T01:40:11.594 INFO:teuthology.orchestra.run.smithi088.stdout: �� 2024-09-16T01:40:11.594 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:11.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.589+0000 7f0fd301cf40 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] conn(0x5650f666fce0 0x5650f6722b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.589+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] -- osd_op(unknown.0.0:1 12.0 12:00000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x5650f666fce0 2024-09-16T01:40:11.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.589+0000 7f0fd0b12640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0fb406ac90 0x7f0fb406d150 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0fc400da60 tx=0x7f0fc4009000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.589+0000 7f0fcbfff640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] conn(0x5650f666fce0 0x5650f6722b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.589+0000 7f0fcbfff640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] conn(0x5650f666fce0 0x5650f6722b50 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.590+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.4 v2:172.21.15.154:6816/2520717231 1 ==== osd_op_reply(1 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0fbc01a320 con 0x5650f666fce0 2024-09-16T01:40:11.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.590+0000 7f0fcbfff640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x7f0fbc058910 0x7f0fbc064390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.590+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:08000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fbc058910 2024-09-16T01:40:11.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.590+0000 7f0fd0b12640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x7f0fbc058910 0x7f0fbc064390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.590+0000 7f0fd0b12640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x7f0fbc058910 0x7f0fbc064390 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.592+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(2 [pgnls start_epoch 86 out=177b] v85'10 uv10 ondisk = 1) ==== 144+0+177 (crc 0 0 0) 0x7f0fc4015030 con 0x7f0fbc058910 2024-09-16T01:40:11.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.592+0000 7f0fd0b12640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x7f0fc4018070 0x7f0fc4018960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.592+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:3 12.8 12:10000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fc4018070 2024-09-16T01:40:11.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.592+0000 7f0fcbfff640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x7f0fc4018070 0x7f0fc4018960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.593+0000 7f0fcbfff640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x7f0fc4018070 0x7f0fc4018960 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.593+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(3 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0fbc01a320 con 0x7f0fc4018070 2024-09-16T01:40:11.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.593+0000 7f0fcbfff640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x7f0fbc067920 0x7f0fbc006380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.593+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:4 12.18 12:18000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fbc067920 2024-09-16T01:40:11.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.593+0000 7f0fd1313640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x7f0fbc067920 0x7f0fbc006380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.593+0000 7f0fd1313640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x7f0fbc067920 0x7f0fbc006380 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.595+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(4 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0fcc054100 con 0x7f0fbc067920 2024-09-16T01:40:11.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.595+0000 7f0fd1313640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x7f0fcc01b020 0x7f0fcc056950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.595+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:5 12.4 12:20000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fcc01b020 2024-09-16T01:40:11.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.595+0000 7f0fd0b12640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x7f0fcc01b020 0x7f0fcc056950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.596+0000 7f0fd0b12640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x7f0fcc01b020 0x7f0fcc056950 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.596+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(5 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0fc4015030 con 0x7f0fcc01b020 2024-09-16T01:40:11.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.596+0000 7f0fd0b12640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x7f0fc40243f0 0x7f0fc4024850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.596+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:6 12.14 12:28000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fc40243f0 2024-09-16T01:40:11.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.596+0000 7f0fcbfff640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x7f0fc40243f0 0x7f0fc4024850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.597+0000 7f0fcbfff640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x7f0fc40243f0 0x7f0fc4024850 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.599+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(6 [pgnls start_epoch 86 out=149b] v85'20 uv20 ondisk = 1) ==== 144+0+149 (crc 0 0 0) 0x7f0fbc01a320 con 0x7f0fc40243f0 2024-09-16T01:40:11.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.600+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.c 12:30000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fbc058910 2024-09-16T01:40:11.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.600+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(7 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0fc4015030 con 0x7f0fbc058910 2024-09-16T01:40:11.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.600+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.1c 12:38000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fbc058910 2024-09-16T01:40:11.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.600+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(8 [pgnls start_epoch 86 out=73b] v84'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f0fc4015030 con 0x7f0fbc058910 2024-09-16T01:40:11.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.600+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] -- osd_op(unknown.0.0:9 12.2 12:40000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x5650f666fce0 2024-09-16T01:40:11.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.601+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.4 v2:172.21.15.154:6816/2520717231 2 ==== osd_op_reply(9 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0fbc01a320 con 0x5650f666fce0 2024-09-16T01:40:11.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.601+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:10 12.12 12:48000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fc4018070 2024-09-16T01:40:11.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.601+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(10 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0fbc01a320 con 0x7f0fc4018070 2024-09-16T01:40:11.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.601+0000 7f0fcbfff640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] conn(0x7f0fbc077040 0x7f0fbc078430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.601+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] -- osd_op(unknown.0.0:11 12.a 12:50000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fbc077040 2024-09-16T01:40:11.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.601+0000 7f0fd1313640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] conn(0x7f0fbc077040 0x7f0fbc078430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.601+0000 7f0fd1313640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] conn(0x7f0fbc077040 0x7f0fbc078430 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.603+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 <== osd.2 v2:172.21.15.88:6808/2918260303 1 ==== osd_op_reply(11 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0fcc054100 con 0x7f0fbc077040 2024-09-16T01:40:11.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.603+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] -- osd_op(unknown.0.0:12 12.1a 12:58000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x5650f666fce0 2024-09-16T01:40:11.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.604+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.4 v2:172.21.15.154:6816/2520717231 3 ==== osd_op_reply(12 [pgnls start_epoch 86 out=49b] v84'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0fbc01a320 con 0x5650f666fce0 2024-09-16T01:40:11.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.604+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:13 12.6 12:60000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fbc067920 2024-09-16T01:40:11.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.604+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(13 [pgnls start_epoch 86 out=77b] v85'3 uv3 ondisk = 1) ==== 144+0+77 (crc 0 0 0) 0x7f0fcc054100 con 0x7f0fbc067920 2024-09-16T01:40:11.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.604+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:14 12.16 12:68000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fcc01b020 2024-09-16T01:40:11.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.604+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(14 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0fc4015030 con 0x7f0fcc01b020 2024-09-16T01:40:11.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.604+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.e 12:70000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fbc058910 2024-09-16T01:40:11.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.605+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(15 [pgnls start_epoch 86 out=73b] v84'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f0fc4015030 con 0x7f0fbc058910 2024-09-16T01:40:11.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.605+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] -- osd_op(unknown.0.0:16 12.1e 12:78000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x5650f666fce0 2024-09-16T01:40:11.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.605+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.4 v2:172.21.15.154:6816/2520717231 4 ==== osd_op_reply(16 [pgnls start_epoch 86 out=49b] v84'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0fbc01a320 con 0x5650f666fce0 2024-09-16T01:40:11.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.605+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:17 12.1 12:80000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fcc01b020 2024-09-16T01:40:11.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.605+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(17 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0fc4015030 con 0x7f0fcc01b020 2024-09-16T01:40:11.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.605+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:18 12.11 12:88000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fc40243f0 2024-09-16T01:40:11.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.606+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(18 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0fbc01a320 con 0x7f0fc40243f0 2024-09-16T01:40:11.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.606+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] -- osd_op(unknown.0.0:19 12.9 12:90000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fbc077040 2024-09-16T01:40:11.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.606+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 <== osd.2 v2:172.21.15.88:6808/2918260303 2 ==== osd_op_reply(19 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0fcc054100 con 0x7f0fbc077040 2024-09-16T01:40:11.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.606+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.19 12:98000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fbc058910 2024-09-16T01:40:11.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.606+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(20 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0fc4015030 con 0x7f0fbc058910 2024-09-16T01:40:11.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.606+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:21 12.5 12:a0000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fcc01b020 2024-09-16T01:40:11.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.607+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(21 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0fc4015030 con 0x7f0fcc01b020 2024-09-16T01:40:11.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.607+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:22 12.15 12:a8000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fcc01b020 2024-09-16T01:40:11.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.607+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(22 [pgnls start_epoch 86 out=97b] v84'3 uv3 ondisk = 1) ==== 144+0+97 (crc 0 0 0) 0x7f0fc4015030 con 0x7f0fcc01b020 2024-09-16T01:40:11.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.607+0000 7f0fd0b12640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x7f0fc40348f0 0x7f0fc400e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.607+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:23 12.d 12:b0000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fc40348f0 2024-09-16T01:40:11.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.607+0000 7f0fd0b12640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x7f0fc40348f0 0x7f0fc400e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.607+0000 7f0fd0b12640 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x7f0fc40348f0 0x7f0fc400e3e0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.610+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(23 [pgnls start_epoch 86 out=74b] v84'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f0fc4015030 con 0x7f0fc40348f0 2024-09-16T01:40:11.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.610+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.1d 12:b8000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fbc058910 2024-09-16T01:40:11.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.610+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(24 [pgnls start_epoch 86 out=73b] v84'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f0fc4015030 con 0x7f0fbc058910 2024-09-16T01:40:11.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.610+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.3 12:c0000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fbc058910 2024-09-16T01:40:11.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.610+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(25 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0fc4015030 con 0x7f0fbc058910 2024-09-16T01:40:11.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.610+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:26 12.13 12:c8000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fbc067920 2024-09-16T01:40:11.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.611+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(26 [pgnls start_epoch 86 out=98b] v85'3 uv3 ondisk = 1) ==== 144+0+98 (crc 0 0 0) 0x7f0fcc054100 con 0x7f0fbc067920 2024-09-16T01:40:11.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.611+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:27 12.b 12:d0000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fc4018070 2024-09-16T01:40:11.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.611+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(27 [pgnls start_epoch 86 out=76b] v85'3 uv3 ondisk = 1) ==== 144+0+76 (crc 0 0 0) 0x7f0fbc01a320 con 0x7f0fc4018070 2024-09-16T01:40:11.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.611+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] -- osd_op(unknown.0.0:28 12.1b 12:d8000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x5650f666fce0 2024-09-16T01:40:11.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.611+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.4 v2:172.21.15.154:6816/2520717231 5 ==== osd_op_reply(28 [pgnls start_epoch 86 out=49b] v84'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0fbc01a320 con 0x5650f666fce0 2024-09-16T01:40:11.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.611+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:29 12.7 12:e0000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fbc067920 2024-09-16T01:40:11.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.612+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(29 [pgnls start_epoch 86 out=73b] v84'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f0fcc054100 con 0x7f0fbc067920 2024-09-16T01:40:11.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.612+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:30 12.17 12:e8000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fcc01b020 2024-09-16T01:40:11.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.612+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(30 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0fc4015030 con 0x7f0fcc01b020 2024-09-16T01:40:11.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.612+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:31 12.f 12:f0000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fcc01b020 2024-09-16T01:40:11.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.612+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(31 [pgnls start_epoch 86 out=73b] v84'1 uv1 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f0fc4015030 con 0x7f0fcc01b020 2024-09-16T01:40:11.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.612+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:32 12.1f 12:f8000000:%01:::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723090 con 0x7f0fcc01b020 2024-09-16T01:40:11.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.613+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(32 [pgnls start_epoch 86 out=101b] v85'6 uv6 ondisk = 1) ==== 144+0+101 (crc 0 0 0) 0x7f0fc4015030 con 0x7f0fcc01b020 2024-09-16T01:40:11.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.613+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:33 12.10 12:09401a81:::601.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f6723430 con 0x7f0fbc058910 2024-09-16T01:40:11.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.613+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(33 601.00000000 [read 0~0] v0'0 uv4 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0fc4015030 con 0x7f0fbc058910 2024-09-16T01:40:11.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.613+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:34 12.10 12:09401a81:::601.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f66b0b40 con 0x7f0fbc058910 2024-09-16T01:40:11.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.613+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(34 601.00000000 [getxattrs out=124b] v0'0 uv4 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f0fc4015030 con 0x7f0fbc058910 2024-09-16T01:40:11.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.613+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:35 12.10 12:09401a81:::601.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f66b0ee0 con 0x7f0fbc058910 2024-09-16T01:40:11.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.613+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(35 601.00000000 [omap-get-header out=274b] v0'0 uv4 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f0fc405e040 con 0x7f0fbc058910 2024-09-16T01:40:11.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.613+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:36 12.10 12:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f66b1280 con 0x7f0fbc058910 2024-09-16T01:40:11.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.614+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(36 601.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f0fc405e040 con 0x7f0fbc058910 2024-09-16T01:40:11.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.614+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:37 12.10 12:0bd6d154:::602.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f66b1620 con 0x7f0fbc058910 2024-09-16T01:40:11.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.614+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(37 602.00000000 [read 0~0] v0'0 uv6 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0fc405e040 con 0x7f0fbc058910 2024-09-16T01:40:11.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.614+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:38 12.10 12:0bd6d154:::602.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e7dd0 con 0x7f0fbc058910 2024-09-16T01:40:11.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.614+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(38 602.00000000 [getxattrs out=124b] v0'0 uv6 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f0fc405e040 con 0x7f0fbc058910 2024-09-16T01:40:11.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.614+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:39 12.10 12:0bd6d154:::602.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e7dd0 con 0x7f0fbc058910 2024-09-16T01:40:11.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.614+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(39 602.00000000 [omap-get-header out=274b] v0'0 uv6 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f0fc405e040 con 0x7f0fbc058910 2024-09-16T01:40:11.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.614+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:40 12.10 12:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e8de0 con 0x7f0fbc058910 2024-09-16T01:40:11.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.614+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(40 602.00000000 [omap-get-vals out=5b] v0'0 uv6 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f0fc405e040 con 0x7f0fbc058910 2024-09-16T01:40:11.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.615+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:41 12.10 12:0cf6ad0f:::10000000000.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e7dd0 con 0x7f0fbc058910 2024-09-16T01:40:11.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.615+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(41 10000000000.00000000 [read 0~0] v0'0 uv10 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0fc405e040 con 0x7f0fbc058910 2024-09-16T01:40:11.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.615+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:42 12.10 12:0cf6ad0f:::10000000000.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e7dd0 con 0x7f0fbc058910 2024-09-16T01:40:11.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.615+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(42 10000000000.00000000 [getxattrs out=124b] v0'0 uv10 ondisk = 0) ==== 164+0+124 (crc 0 0 0) 0x7f0fc405e040 con 0x7f0fbc058910 2024-09-16T01:40:11.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.615+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:43 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e7dd0 con 0x7f0fbc058910 2024-09-16T01:40:11.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.615+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(43 10000000000.00000000 [omap-get-header out=274b] v0'0 uv10 ondisk = 0) ==== 164+0+274 (crc 0 0 0) 0x7f0fc405e040 con 0x7f0fbc058910 2024-09-16T01:40:11.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.615+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:44 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e8de0 con 0x7f0fbc058910 2024-09-16T01:40:11.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.615+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(44 10000000000.00000000 [omap-get-vals out=586b] v0'0 uv10 ondisk = 0) ==== 164+0+586 (crc 0 0 0) 0x7f0fc405e040 con 0x7f0fbc058910 2024-09-16T01:40:11.622 INFO:teuthology.orchestra.run.smithi088.stdout: ��*$- 601.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray1 ���� �� ((88���� �� ��*$- 602.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray2 ���� �� ((88���� �� ��2,510000000000.00000000�������������������!�������������������������� ���~_layout��������_parent>8subdir ���� ��  (z��f�-(z��f�-8`z��fw�^8`z��fw�^���� ��OI sixmegs_head1i"�z��fw�^���5@@ `��������z��fw�^z��f�-(8`z��fw�^8`z��fw�^ 2024-09-16T01:40:11.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.615+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:45 12.10 12:0d82a743:::600.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e7dd0 con 0x7f0fbc058910 2024-09-16T01:40:11.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.616+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(45 600.00000000 [read 0~0] v0'0 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0fc405e040 con 0x7f0fbc058910 2024-09-16T01:40:11.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.616+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:46 12.10 12:0d82a743:::600.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e7dd0 con 0x7f0fbc058910 2024-09-16T01:40:11.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.616+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(46 600.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fbc058910 2024-09-16T01:40:11.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.616+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:47 12.10 12:0d82a743:::600.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e7dd0 con 0x7f0fbc058910 2024-09-16T01:40:11.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.616+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(47 600.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fbc058910 2024-09-16T01:40:11.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.616+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:48 12.10 12:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e8de0 con 0x7f0fbc058910 2024-09-16T01:40:11.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.616+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(48 600.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fbc058910 2024-09-16T01:40:11.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.616+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:49 12.10 12:0d89b25e:::603.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e7dd0 con 0x7f0fbc058910 2024-09-16T01:40:11.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.616+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(49 603.00000000 [read 0~0] v0'0 uv8 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fbc058910 2024-09-16T01:40:11.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.616+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:50 12.10 12:0d89b25e:::603.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e7dd0 con 0x7f0fbc058910 2024-09-16T01:40:11.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.617+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(50 603.00000000 [getxattrs out=124b] v0'0 uv8 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fbc058910 2024-09-16T01:40:11.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.617+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:51 12.10 12:0d89b25e:::603.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e7dd0 con 0x7f0fbc058910 2024-09-16T01:40:11.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.617+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(51 603.00000000 [omap-get-header out=274b] v0'0 uv8 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fbc058910 2024-09-16T01:40:11.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.617+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:52 12.10 12:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e8de0 con 0x7f0fbc058910 2024-09-16T01:40:11.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.617+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(52 603.00000000 [omap-get-vals out=5b] v0'0 uv8 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fbc058910 2024-09-16T01:40:11.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.617+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:53 12.14 12:287034c4:::1.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e7dd0 con 0x7f0fc40243f0 2024-09-16T01:40:11.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.617+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(53 1.00000000.inode [read 0~582 out=582b] v0'0 uv16 ondisk = 0) ==== 160+0+582 (crc 0 0 0) 0x7f0fbc01a320 con 0x7f0fc40243f0 2024-09-16T01:40:11.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.617+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:54 12.14 12:287034c4:::1.00000000.inode:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e8170 con 0x7f0fc40243f0 2024-09-16T01:40:11.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.618+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(54 1.00000000.inode [getxattrs out=4b] v0'0 uv16 ondisk = 0) ==== 160+0+4 (crc 0 0 0) 0x7f0fbc01a320 con 0x7f0fc40243f0 2024-09-16T01:40:11.628 INFO:teuthology.orchestra.run.smithi088.stdout:��������z��f�-����security.selinux%unconfined_u:object_r:unlabeled_t:s0������������ �� ��*$- 600.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray0 ���� �� ((88���� �� ��*$- 603.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray3 ���� �� ((88���� �� ��.(11.00000000.inode�������������������!�������������������������� ��`ZFFceph fs volume v011)�y��f,��7�C@@ ��������y��f,��7(y��f,��78y��f,��78t��f��t��f������NHt��f�������������� �� 2024-09-16T01:40:11.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.618+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:55 12.14 12:287034c4:::1.00000000.inode:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e8510 con 0x7f0fc40243f0 2024-09-16T01:40:11.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.618+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(55 1.00000000.inode [omap-get-header] v0'0 uv16 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0fbc01a320 con 0x7f0fc40243f0 2024-09-16T01:40:11.629 INFO:teuthology.orchestra.run.smithi088.stdout:���� �� 2024-09-16T01:40:11.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.618+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:56 12.14 12:287034c4:::1.00000000.inode:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e98c0 con 0x7f0fc40243f0 2024-09-16T01:40:11.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.618+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(56 1.00000000.inode [omap-get-vals out=5b] v0'0 uv16 ondisk = 0) ==== 160+0+5 (crc 0 0 0) 0x7f0fbc01a320 con 0x7f0fc40243f0 2024-09-16T01:40:11.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.618+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:57 12.14 12:292cf221:::200.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e88b0 con 0x7f0fc40243f0 2024-09-16T01:40:11.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.619+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(57 200.00000000 [read 0~90 out=90b] v0'0 uv20 ondisk = 0) ==== 156+0+90 (crc 0 0 0) 0x7f0fbc01a320 con 0x7f0fc40243f0 2024-09-16T01:40:11.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.619+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:58 12.14 12:292cf221:::200.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e8ca0 con 0x7f0fc40243f0 2024-09-16T01:40:11.631 INFO:teuthology.orchestra.run.smithi088.stdout:���� �� ��*$- 200.00000000�������������������!�������������������������� ��tnZZTceph fs volume v011@d<@d<@RD@@@ ���� �� 2024-09-16T01:40:11.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.619+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(58 200.00000000 [getxattrs out=4b] v0'0 uv20 ondisk = 0) ==== 156+0+4 (crc 0 0 0) 0x7f0fbc01a320 con 0x7f0fc40243f0 2024-09-16T01:40:11.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.619+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:59 12.14 12:292cf221:::200.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e90c0 con 0x7f0fc40243f0 2024-09-16T01:40:11.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.619+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(59 200.00000000 [omap-get-header] v0'0 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0fbc01a320 con 0x7f0fc40243f0 2024-09-16T01:40:11.632 INFO:teuthology.orchestra.run.smithi088.stdout:���� �� 2024-09-16T01:40:11.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.619+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:60 12.14 12:292cf221:::200.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e9460 con 0x7f0fc40243f0 2024-09-16T01:40:11.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.619+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(60 200.00000000 [omap-get-vals out=5b] v0'0 uv20 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f0fbc01a320 con 0x7f0fc40243f0 2024-09-16T01:40:11.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.619+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:61 12.14 12:2e2fa760:::200.00000001:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e9460 con 0x7f0fc40243f0 2024-09-16T01:40:11.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.620+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(61 200.00000001 [read 0~17490 out=17490b] v0'0 uv17 ondisk = 0) ==== 156+0+17490 (crc 0 0 0) 0x7f0fbc01a320 con 0x7f0fc40243f0 2024-09-16T01:40:11.633 INFO:teuthology.orchestra.run.smithi088.stdout:���� �� ��*$- 200.00000001�������������������!�������������������������� ��lDfDRDRD��XS&YA1e@��XS&YA1h^Tt��f��(2 ((88 2 ( 2024-09-16T01:40:11.633 INFO:teuthology.orchestra.run.smithi088.stdout:( 2024-09-16T01:40:11.633 INFO:teuthology.orchestra.run.smithi088.stdout:8 2024-09-16T01:40:11.633 INFO:teuthology.orchestra.run.smithi088.stdout:t��f�: 8 2024-09-16T01:40:11.634 INFO:teuthology.orchestra.run.smithi088.stdout:t��f�:  ��������@0@��XS&YA1�� 2024-09-16T01:40:11.634 INFO:teuthology.orchestra.run.smithi088.stdout:�y��f��)���4�X� entity_id0hostname smithi088kernel_version5.14.0-505.el9.x86_64root/�� ����@��XS&YA1���y��f= �-n���������@��XS&YA1y��f-��-setattr� C���������y��f(��-�C@@ ��������t��f��(8y��f(��-8t��f��t��f������NHt��f�������������������X@��XS&YA1��y��f�b8mkdir�p (y��f,��7(y��f,��78y��f,��78y��f,��7J 8subdir���������y��f,��7�A�5����������������y��f,��7y��f,��7(88��������y��f,��7����security.selinux%unconfined_u:object_r:unlabeled_t:s0��������2 ((88  C���������y��f,��7�C@@ ��������y��f,��7(y��f,��78y��f,��78t��f��t��f������NHt��f����������������������@��XS&YA1�z��f{Jopenc�p (y��f,��7(y��f,��78z��f�-8y��f,��7J 8subdir���������z��f�-�A�5����������������z��f�-y��f,��7(z��f�-8z��f�-8z��f�-��������y��f,��7����security.selinux%unconfined_u:object_r:unlabeled_t:s0��������� (z��f�-(z��f�-8z��f�-8z��f�-m [sixmegs���������z��f�-���5@@ ��������z��f�-z��f�-�@(88��������z��f�-����security.selinux%unconfined_u:object_r:unlabeled_t:s0�������������������@��XS&YA1 2024-09-16T01:40:11.634 INFO:teuthology.orchestra.run.smithi088.stdout:z��f��� 2024-09-16T01:40:11.634 INFO:teuthology.orchestra.run.smithi088.stdout:cap update�p (y��f,��7(y��f,��78 z��f��8y��f,��7J 8subdir���������z��f�-�A�5����������������z��f�-y��f,��7(z��f�-8 z��f��8 z��f����������y��f,��7����security.selinux%unconfined_u:object_r:unlabeled_t:s0��������� (z��f�-(z��f�-8 z��f��8 z��f��m [sixmegs���������z��f�����5@@ ��������z��f��z��f�-��(8 z��f��8 z��f����������z��f�-����security.selinux%unconfined_u:object_r:unlabeled_t:s0������������������@��XS&YA1 2024-09-16T01:40:11.634 INFO:teuthology.orchestra.run.smithi088.stdout:z��f� O 2024-09-16T01:40:11.634 INFO:teuthology.orchestra.run.smithi088.stdout:cap update�p  (y��f,��7(y��f,��78Pz��f��08y��f,��7J 8subdir���������z��f�-�A�5����������������z��f�-y��f,��7(z��f�-8Pz��f��08Pz��f��0��������y��f,��7����security.selinux%unconfined_u:object_r:unlabeled_t:s0��������� (z��f�-(z��f�-8Pz��f��08Pz��f��0m [sixmegs���������z��f��0���5@@ P��������z��f��0z��f�-��(8Pz��f��08Pz��f��0��������z��f�-����security.selinux%unconfined_u:object_r:unlabeled_t:s0������������������ @��XS&YA1 2024-09-16T01:40:11.634 INFO:teuthology.orchestra.run.smithi088.stdout:z��f��*& 2024-09-16T01:40:11.634 INFO:teuthology.orchestra.run.smithi088.stdout:cap update�p  (y��f,��7(y��f,��78`z��fw�^8y��f,��7J 8subdir�������� 2024-09-16T01:40:11.634 INFO:teuthology.orchestra.run.smithi088.stdout:�z��f�-�A�5����������������z��f�-y��f,��7(z��f�-8`z��fw�^8`z��fw�^ 2024-09-16T01:40:11.634 INFO:teuthology.orchestra.run.smithi088.stdout:��������y��f,��7����security.selinux%unconfined_u:object_r:unlabeled_t:s0���������  (z��f�-(z��f�-8`z��fw�^8`z��fw�^m [sixmegs���������z��fw�^���5@@ `��������z��fw�^z��f�-�(8`z��fw�^8`z��fw�^��������z��f�-����security.selinux%unconfined_u:object_r:unlabeled_t:s0�����������������)@��XS&YA1�� 2024-09-16T01:40:11.635 INFO:teuthology.orchestra.run.smithi088.stdout:�z��f�eS&���4�X���=1@��XS&YA1 2024-09-16T01:40:11.635 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:11.635 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:11.635 INFO:teuthology.orchestra.run.smithi088.stdout:z��f���&check_inode_max_size� p  (y��f,��7(y��f,��78`z��fw�^8y��f,��7J 8subdir�������� �z��f�-�A�5����������������z��f�-y��f,��7(z��f�-8`z��fw�^8`z��fw�^ ��������y��f,��7����security.selinux%unconfined_u:object_r:unlabeled_t:s0���������  (z��f�-(z��f�-8`z��fw�^8`z��fw�^� 5sixmegs�������� 2024-09-16T01:40:11.635 INFO:teuthology.orchestra.run.smithi088.stdout:�z��fw�^���5@@ `��������z��fw�^z��f�-(8`z��fw�^8`z��fw�^ 2024-09-16T01:40:11.635 INFO:teuthology.orchestra.run.smithi088.stdout:��������z��f�-����security.selinux%unconfined_u:object_r:unlabeled_t:s0�������� 5sixmegs�������� 2024-09-16T01:40:11.635 INFO:teuthology.orchestra.run.smithi088.stdout:�z��fw�^���5@@ `��������z��fw�^z��f�-(8`z��fw�^8`z��fw�^ 2024-09-16T01:40:11.635 INFO:teuthology.orchestra.run.smithi088.stdout:��������z��f�-����security.selinux%unconfined_u:object_r:unlabeled_t:s0�����������������1@��XS&YA18.$�<@��XS&YA1h^T{��f�} 2  (y��f,��7(y��f,��78`z��fw�^8y��f,��7 2 ( 2024-09-16T01:40:11.635 INFO:teuthology.orchestra.run.smithi088.stdout:( 2024-09-16T01:40:11.635 INFO:teuthology.orchestra.run.smithi088.stdout:8 2024-09-16T01:40:11.635 INFO:teuthology.orchestra.run.smithi088.stdout:t��f�: 8 2024-09-16T01:40:11.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.620+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:62 12.14 12:2e2fa760:::200.00000001:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e9460 con 0x7f0fc40243f0 2024-09-16T01:40:11.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.620+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(62 200.00000001 [getxattrs out=4b] v0'0 uv17 ondisk = 0) ==== 156+0+4 (crc 0 0 0) 0x7f0fbc01a320 con 0x7f0fc40243f0 2024-09-16T01:40:11.636 INFO:teuthology.orchestra.run.smithi088.stdout:t��f�:  ��������@d<@��XS&YA1^TJ{��fR8> scatter_writebehind2 (y��f,��7(y��f,��78`z��fw�^8`z��fw�^  C���������y��f,��7�C@@ ��������y��f,��7(y��f,��78`z��fw�^8t��f��t��f������NHt��f�������������������?@���� �� 2024-09-16T01:40:11.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.620+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:63 12.14 12:2e2fa760:::200.00000001:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e9460 con 0x7f0fc40243f0 2024-09-16T01:40:11.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.620+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(63 200.00000001 [omap-get-header] v0'0 uv17 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0fbc01a320 con 0x7f0fc40243f0 2024-09-16T01:40:11.637 INFO:teuthology.orchestra.run.smithi088.stdout:���� �� 2024-09-16T01:40:11.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.621+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:64 12.14 12:2e2fa760:::200.00000001:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e9460 con 0x7f0fc40243f0 2024-09-16T01:40:11.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.621+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(64 200.00000001 [omap-get-vals out=5b] v0'0 uv17 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f0fbc01a320 con 0x7f0fc40243f0 2024-09-16T01:40:11.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.621+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:65 12.14 12:2f55791f:::606.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e9460 con 0x7f0fc40243f0 2024-09-16T01:40:11.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.621+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(65 606.00000000 [read 0~0] v0'0 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0fbc01a320 con 0x7f0fc40243f0 2024-09-16T01:40:11.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.621+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:66 12.14 12:2f55791f:::606.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e9460 con 0x7f0fc40243f0 2024-09-16T01:40:11.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.621+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(66 606.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f0fbc01a320 con 0x7f0fc40243f0 2024-09-16T01:40:11.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.621+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:67 12.14 12:2f55791f:::606.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e9460 con 0x7f0fc40243f0 2024-09-16T01:40:11.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.622+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(67 606.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f0fbc01a320 con 0x7f0fc40243f0 2024-09-16T01:40:11.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.622+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:68 12.14 12:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e9460 con 0x7f0fc40243f0 2024-09-16T01:40:11.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.622+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(68 606.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f0fbc01a320 con 0x7f0fc40243f0 2024-09-16T01:40:11.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.622+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:69 12.1c 12:3ed09add:::607.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e9460 con 0x7f0fbc058910 2024-09-16T01:40:11.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.622+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(69 607.00000000 [read 0~0] v0'0 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fbc058910 2024-09-16T01:40:11.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.622+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:70 12.1c 12:3ed09add:::607.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e9460 con 0x7f0fbc058910 2024-09-16T01:40:11.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.622+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(70 607.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fbc058910 2024-09-16T01:40:11.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.622+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:71 12.1c 12:3ed09add:::607.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e9460 con 0x7f0fbc058910 2024-09-16T01:40:11.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.623+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(71 607.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fbc058910 2024-09-16T01:40:11.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.623+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:72 12.1c 12:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e9460 con 0x7f0fbc058910 2024-09-16T01:40:11.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.623+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(72 607.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fbc058910 2024-09-16T01:40:11.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.623+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:73 12.6 12:654134d2:::mds0_openfiles.0:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e9460 con 0x7f0fbc067920 2024-09-16T01:40:11.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.623+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(73 mds0_openfiles.0 [read 0~0] v0'0 uv3 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0fcc054100 con 0x7f0fbc067920 2024-09-16T01:40:11.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.623+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:74 12.6 12:654134d2:::mds0_openfiles.0:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67e9c60 con 0x7f0fbc067920 2024-09-16T01:40:11.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.624+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(74 mds0_openfiles.0 [getxattrs out=4b] v0'0 uv3 ondisk = 0) ==== 160+0+4 (crc 0 0 0) 0x7f0fcc054100 con 0x7f0fbc067920 2024-09-16T01:40:11.644 INFO:teuthology.orchestra.run.smithi088.stdout:���� �� ��*$- 606.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray6 ���� �� ((88���� �� ��*$- 607.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray7 ���� �� ((88���� �� ��.(1mds0_openfiles.0�������������������!�������������������������� �� 2024-09-16T01:40:11.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.624+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:75 12.6 12:654134d2:::mds0_openfiles.0:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67ea000 con 0x7f0fbc067920 2024-09-16T01:40:11.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.624+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(75 mds0_openfiles.0 [omap-get-header out=42b] v0'0 uv3 ondisk = 0) ==== 160+0+42 (crc 0 0 0) 0x7f0fcc054100 con 0x7f0fbc067920 2024-09-16T01:40:11.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.624+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:76 12.6 12:654134d2:::mds0_openfiles.0:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eb3b0 con 0x7f0fbc067920 2024-09-16T01:40:11.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.624+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(76 mds0_openfiles.0 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 160+0+5 (crc 0 0 0) 0x7f0fcc054100 con 0x7f0fbc067920 2024-09-16T01:40:11.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.624+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:77 12.e 12:717a0223:::608.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67ea3a0 con 0x7f0fbc058910 2024-09-16T01:40:11.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.624+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(77 608.00000000 [read 0~0] v0'0 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fbc058910 2024-09-16T01:40:11.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.624+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:78 12.e 12:717a0223:::608.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67ea3a0 con 0x7f0fbc058910 2024-09-16T01:40:11.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.624+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(78 608.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fbc058910 2024-09-16T01:40:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.624+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:79 12.e 12:717a0223:::608.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67ea3a0 con 0x7f0fbc058910 2024-09-16T01:40:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.625+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(79 608.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fbc058910 2024-09-16T01:40:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.625+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:80 12.e 12:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eb750 con 0x7f0fbc058910 2024-09-16T01:40:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.625+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(80 608.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fbc058910 2024-09-16T01:40:11.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.625+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:81 12.15 12:a93a17c2:::604.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67ea3a0 con 0x7f0fcc01b020 2024-09-16T01:40:11.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.626+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(81 604.00000000 [read 0~0] v0'0 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fcc01b020 2024-09-16T01:40:11.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.626+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:82 12.15 12:a93a17c2:::604.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67ea3a0 con 0x7f0fcc01b020 2024-09-16T01:40:11.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.626+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(82 604.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fcc01b020 2024-09-16T01:40:11.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.626+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:83 12.15 12:a93a17c2:::604.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67ea3a0 con 0x7f0fcc01b020 2024-09-16T01:40:11.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.626+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(83 604.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fcc01b020 2024-09-16T01:40:11.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.626+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:84 12.15 12:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67ea3a0 con 0x7f0fcc01b020 2024-09-16T01:40:11.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.627+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(84 604.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fcc01b020 2024-09-16T01:40:11.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.627+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:85 12.15 12:aa448500:::500.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67ea3a0 con 0x7f0fcc01b020 2024-09-16T01:40:11.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.627+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(85 500.00000000 [read 0~90 out=90b] v0'0 uv3 ondisk = 0) ==== 156+0+90 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fcc01b020 2024-09-16T01:40:11.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.627+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:86 12.15 12:aa448500:::500.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67ea3a0 con 0x7f0fcc01b020 2024-09-16T01:40:11.651 INFO:teuthology.orchestra.run.smithi088.stdout:���� ��4.*ceph fs volume v011 ���� �� ��*$- 608.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray8 ���� �� ((88���� �� ��*$- 604.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray4 ���� �� ((88���� �� ��*$- 500.00000000�������������������!�������������������������� ��tnZZTceph fs volume v011@@@@@@ ���� �� 2024-09-16T01:40:11.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.628+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(86 500.00000000 [getxattrs out=4b] v0'0 uv3 ondisk = 0) ==== 156+0+4 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fcc01b020 2024-09-16T01:40:11.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.628+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:87 12.15 12:aa448500:::500.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67ea3a0 con 0x7f0fcc01b020 2024-09-16T01:40:11.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.628+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(87 500.00000000 [omap-get-header] v0'0 uv3 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fcc01b020 2024-09-16T01:40:11.652 INFO:teuthology.orchestra.run.smithi088.stdout:���� �� 2024-09-16T01:40:11.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.628+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:88 12.15 12:aa448500:::500.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67ea3a0 con 0x7f0fcc01b020 2024-09-16T01:40:11.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.628+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(88 500.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fcc01b020 2024-09-16T01:40:11.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.628+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:89 12.d 12:b50e409b:::mds_snaptable:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67ea3a0 con 0x7f0fc40348f0 2024-09-16T01:40:11.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.629+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(89 mds_snaptable [read 0~70 out=70b] v0'0 uv1 ondisk = 0) ==== 157+0+70 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fc40348f0 2024-09-16T01:40:11.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.629+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:90 12.d 12:b50e409b:::mds_snaptable:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67ea3a0 con 0x7f0fc40348f0 2024-09-16T01:40:11.654 INFO:teuthology.orchestra.run.smithi088.stdout:���� �� ��+%. mds_snaptable�������������������!�������������������������� ��`ZFF4���� �� 2024-09-16T01:40:11.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.630+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(90 mds_snaptable [getxattrs out=4b] v0'0 uv1 ondisk = 0) ==== 157+0+4 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fc40348f0 2024-09-16T01:40:11.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.630+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:91 12.d 12:b50e409b:::mds_snaptable:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67ea3a0 con 0x7f0fc40348f0 2024-09-16T01:40:11.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.630+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(91 mds_snaptable [omap-get-header] v0'0 uv1 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fc40348f0 2024-09-16T01:40:11.655 INFO:teuthology.orchestra.run.smithi088.stdout:���� �� 2024-09-16T01:40:11.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.630+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:92 12.d 12:b50e409b:::mds_snaptable:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67ea3a0 con 0x7f0fc40348f0 2024-09-16T01:40:11.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.630+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(92 mds_snaptable [omap-get-vals out=5b] v0'0 uv1 ondisk = 0) ==== 157+0+5 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fc40348f0 2024-09-16T01:40:11.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.630+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:93 12.1d 12:b871830b:::605.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67ea3a0 con 0x7f0fbc058910 2024-09-16T01:40:11.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.631+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(93 605.00000000 [read 0~0] v0'0 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fbc058910 2024-09-16T01:40:11.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.631+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:94 12.1d 12:b871830b:::605.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67ea3a0 con 0x7f0fbc058910 2024-09-16T01:40:11.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.631+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(94 605.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fbc058910 2024-09-16T01:40:11.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.631+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:95 12.1d 12:b871830b:::605.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67ea3a0 con 0x7f0fbc058910 2024-09-16T01:40:11.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.631+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(95 605.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fbc058910 2024-09-16T01:40:11.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.631+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:96 12.1d 12:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eb750 con 0x7f0fbc058910 2024-09-16T01:40:11.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.632+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(96 605.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fbc058910 2024-09-16T01:40:11.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.632+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:c91d4a1d:::mds0_inotable:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67ea3a0 con 0x7f0fbc067920 2024-09-16T01:40:11.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.632+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(97 mds0_inotable [read 0~34 out=34b] v0'0 uv3 ondisk = 0) ==== 157+0+34 (crc 0 0 0) 0x7f0fcc054100 con 0x7f0fbc067920 2024-09-16T01:40:11.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.632+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:c91d4a1d:::mds0_inotable:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67ea790 con 0x7f0fbc067920 2024-09-16T01:40:11.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.632+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(98 mds0_inotable [getxattrs out=4b] v0'0 uv3 ondisk = 0) ==== 157+0+4 (crc 0 0 0) 0x7f0fcc054100 con 0x7f0fbc067920 2024-09-16T01:40:11.659 INFO:teuthology.orchestra.run.smithi088.stdout:���� �� ��*$- 605.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray5 ���� �� ((88���� �� ��+%. mds0_inotable�������������������!�������������������������� ��<6""��������� �� 2024-09-16T01:40:11.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.632+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:c91d4a1d:::mds0_inotable:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eab80 con 0x7f0fbc067920 2024-09-16T01:40:11.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.632+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(99 mds0_inotable [omap-get-header] v0'0 uv3 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0fcc054100 con 0x7f0fbc067920 2024-09-16T01:40:11.660 INFO:teuthology.orchestra.run.smithi088.stdout:���� �� 2024-09-16T01:40:11.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.632+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:c91d4a1d:::mds0_inotable:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eaf70 con 0x7f0fbc067920 2024-09-16T01:40:11.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.633+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(100 mds0_inotable [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 157+0+5 (crc 0 0 0) 0x7f0fcc054100 con 0x7f0fbc067920 2024-09-16T01:40:11.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.633+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:101 12.13 12:cd5a64a3:::100.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eaf70 con 0x7f0fbc067920 2024-09-16T01:40:11.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.633+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(101 100.00000000 [read 0~0] v0'0 uv1 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0fcc054100 con 0x7f0fbc067920 2024-09-16T01:40:11.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.633+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:102 12.13 12:cd5a64a3:::100.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eaf70 con 0x7f0fbc067920 2024-09-16T01:40:11.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.633+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(102 100.00000000 [getxattrs out=4b] v0'0 uv1 ondisk = 0) ==== 156+0+4 (crc 0 0 0) 0x7f0fcc054100 con 0x7f0fbc067920 2024-09-16T01:40:11.662 INFO:teuthology.orchestra.run.smithi088.stdout:���� �� ��*$- 100.00000000�������������������!�������������������������� �� 2024-09-16T01:40:11.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.633+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:103 12.13 12:cd5a64a3:::100.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eaf70 con 0x7f0fbc067920 2024-09-16T01:40:11.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.633+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(103 100.00000000 [omap-get-header out=274b] v0'0 uv1 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f0fcc054100 con 0x7f0fbc067920 2024-09-16T01:40:11.663 INFO:teuthology.orchestra.run.smithi088.stdout:���� �� ( 2024-09-16T01:40:11.663 INFO:teuthology.orchestra.run.smithi088.stdout:( 2024-09-16T01:40:11.663 INFO:teuthology.orchestra.run.smithi088.stdout:8 2024-09-16T01:40:11.663 INFO:teuthology.orchestra.run.smithi088.stdout:t��f�: 8 2024-09-16T01:40:11.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.633+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:104 12.13 12:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eaf70 con 0x7f0fbc067920 2024-09-16T01:40:11.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.634+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(104 100.00000000 [omap-get-vals out=5195b] v0'0 uv1 ondisk = 0) ==== 156+0+5195 (crc 0 0 0) 0x7f0fcc054100 con 0x7f0fbc067920 2024-09-16T01:40:11.664 INFO:teuthology.orchestra.run.smithi088.stdout:t��f�: ���� ��PJ 2024-09-16T01:40:11.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.634+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:105 12.b 12:d0630e4c:::mds0_sessionmap:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eaf70 con 0x7f0fc4018070 2024-09-16T01:40:11.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.634+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(105 mds0_sessionmap [read 0~0] v0'0 uv3 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f0fbc01a320 con 0x7f0fc4018070 2024-09-16T01:40:11.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.634+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:106 12.b 12:d0630e4c:::mds0_sessionmap:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eaf70 con 0x7f0fc4018070 2024-09-16T01:40:11.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.634+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.3 v2:172.21.15.88:6800/697609987 5 ==== osd_op_reply(106 mds0_sessionmap [getxattrs out=4b] v0'0 uv3 ondisk = 0) ==== 159+0+4 (crc 0 0 0) 0x7f0fbc01a320 con 0x7f0fc4018070 2024-09-16T01:40:11.665 INFO:teuthology.orchestra.run.smithi088.stdout: stray0_head�i���t��f�<@A����������������t��f�<(8t��f�<8t��f�<t��f�<������������ stray1_head�i���t��f��@A����������������t��f��(8t��f��8t��f��t��f�������������� stray2_head�i���t��f��@A����������������t��f��(8t��f��8t��f��t��f�������������� stray3_head�i���t��fQ�@A����������������t��fQ�(8t��fQ�8t��fQ�t��fQ������������� stray4_head�i���t��f4.@A����������������t��f4.(8t��f4.8t��f4.t��f4.������������ stray5_head�i���t��f�E@A����������������t��f�E(8t��f�E8t��f�Et��f�E������������ stray6_head�i���t��f�{@A����������������t��f�{(8t��f�{8t��f�{t��f�{������������ stray7_head�i���t��f (@A����������������t��f ((8t��f (8t��f (t��f (������������ stray8_head�i���t��f 5@A����������������t��f 5(8t��f 58t��f 5t��f 5������������ stray9_head�i��� t��f�: @A����������������t��f�: (8t��f�: 8t��f�: t��f�: ���������������� �� ��-'0mds0_sessionmap�������������������!�������������������������� �� 2024-09-16T01:40:11.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.634+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:107 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eaf70 con 0x7f0fc4018070 2024-09-16T01:40:11.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.634+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.3 v2:172.21.15.88:6800/697609987 6 ==== osd_op_reply(107 mds0_sessionmap [omap-get-header out=14b] v0'0 uv3 ondisk = 0) ==== 159+0+14 (crc 0 0 0) 0x7f0fbc01a320 con 0x7f0fc4018070 2024-09-16T01:40:11.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.634+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:108 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eaf70 con 0x7f0fc4018070 2024-09-16T01:40:11.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.635+0000 7f0fcbfff640 1 -- 172.21.15.88:0/914797419 <== osd.3 v2:172.21.15.88:6800/697609987 7 ==== osd_op_reply(108 mds0_sessionmap [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 159+0+5 (crc 0 0 0) 0x7f0fbc01a320 con 0x7f0fc4018070 2024-09-16T01:40:11.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.635+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eaf70 con 0x7f0fbc067920 2024-09-16T01:40:11.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.635+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(109 609.00000000 [read 0~0] v0'0 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0fcc054100 con 0x7f0fbc067920 2024-09-16T01:40:11.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.635+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eaf70 con 0x7f0fbc067920 2024-09-16T01:40:11.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.635+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(110 609.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) ==== 156+0+124 (crc 0 0 0) 0x7f0fcc054100 con 0x7f0fbc067920 2024-09-16T01:40:11.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.635+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:111 12.7 12:e0b41d06:::609.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eaf70 con 0x7f0fbc067920 2024-09-16T01:40:11.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.635+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(111 609.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f0fcc054100 con 0x7f0fbc067920 2024-09-16T01:40:11.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.635+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:112 12.7 12:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eaf70 con 0x7f0fbc067920 2024-09-16T01:40:11.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.636+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(112 609.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f0fcc054100 con 0x7f0fbc067920 2024-09-16T01:40:11.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.636+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eaf70 con 0x7f0fcc01b020 2024-09-16T01:40:11.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.637+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(113 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) ==== 156+0+22 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fcc01b020 2024-09-16T01:40:11.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.637+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eaf70 con 0x7f0fcc01b020 2024-09-16T01:40:11.670 INFO:teuthology.orchestra.run.smithi088.stdout:���� ������ �� ��*$- 609.00000000�������������������!�������������������������� ���~_layout��������_parent>8 stray9 ���� �� ((88���� �� ��*$- 400.00000000�������������������!�������������������������� ��0*���� �� 2024-09-16T01:40:11.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.637+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(114 400.00000000 [getxattrs out=4b] v0'0 uv1 ondisk = 0) ==== 156+0+4 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fcc01b020 2024-09-16T01:40:11.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.637+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eaf70 con 0x7f0fcc01b020 2024-09-16T01:40:11.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.637+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(115 400.00000000 [omap-get-header] v0'0 uv1 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fcc01b020 2024-09-16T01:40:11.671 INFO:teuthology.orchestra.run.smithi088.stdout:���� �� 2024-09-16T01:40:11.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.637+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.f 12:f1f69726:::400.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eaf70 con 0x7f0fcc01b020 2024-09-16T01:40:11.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.638+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(116 400.00000000 [omap-get-vals out=5b] v0'0 uv1 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fcc01b020 2024-09-16T01:40:11.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.638+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eaf70 con 0x7f0fcc01b020 2024-09-16T01:40:11.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.638+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(117 100.00000000.inode [read 0~582 out=582b] v0'0 uv3 ondisk = 0) ==== 162+0+582 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fcc01b020 2024-09-16T01:40:11.673 INFO:teuthology.orchestra.run.smithi088.stdout:���� �� ��0*3100.00000000.inode�������������������!�������������������������� ��`ZFFceph fs volume v011)�t��f��@A����������������t��f��( 2024-09-16T01:40:11.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.638+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eaf70 con 0x7f0fcc01b020 2024-09-16T01:40:11.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.639+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(118 100.00000000.inode [getxattrs out=4b] v0'0 uv3 ondisk = 0) ==== 162+0+4 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fcc01b020 2024-09-16T01:40:11.673 INFO:teuthology.orchestra.run.smithi088.stdout:8 t��f�: 8 t��f�: t��f������NH������������ �� 2024-09-16T01:40:11.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.639+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eaf70 con 0x7f0fcc01b020 2024-09-16T01:40:11.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.639+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(119 100.00000000.inode [omap-get-header] v0'0 uv3 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fcc01b020 2024-09-16T01:40:11.674 INFO:teuthology.orchestra.run.smithi088.stdout:���� �� 2024-09-16T01:40:11.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.639+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eaf70 con 0x7f0fcc01b020 2024-09-16T01:40:11.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.640+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(120 100.00000000.inode [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 162+0+5 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fcc01b020 2024-09-16T01:40:11.675 INFO:teuthology.orchestra.run.smithi088.stdout:���� �� ��("+ 2024-09-16T01:40:11.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.640+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eaf70 con 0x7f0fcc01b020 2024-09-16T01:40:11.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.640+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(121 1.00000000 [read 0~0] v0'0 uv6 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fcc01b020 2024-09-16T01:40:11.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.640+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eaf70 con 0x7f0fcc01b020 2024-09-16T01:40:11.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.641+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(122 1.00000000 [getxattrs out=92b] v0'0 uv6 ondisk = 0) ==== 154+0+92 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fcc01b020 2024-09-16T01:40:11.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.641+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eaf70 con 0x7f0fcc01b020 2024-09-16T01:40:11.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.641+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(123 1.00000000 [omap-get-header out=274b] v0'0 uv6 ondisk = 0) ==== 154+0+274 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fcc01b020 2024-09-16T01:40:11.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.641+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5650f67eaf70 con 0x7f0fcc01b020 2024-09-16T01:40:11.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd0b12640 1 -- 172.21.15.88:0/914797419 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(124 1.00000000 [omap-get-vals out=585b] v0'0 uv6 ondisk = 0) ==== 154+0+585 (crc 0 0 0) 0x7f0fc4064090 con 0x7f0fcc01b020 2024-09-16T01:40:11.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x7f0fbc058910 msgr2=0x7f0fbc064390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x7f0fbc058910 0x7f0fbc064390 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x7f0fbc067920 msgr2=0x7f0fbc006380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x7f0fbc067920 0x7f0fbc006380 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] conn(0x7f0fbc077040 msgr2=0x7f0fbc078430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] conn(0x7f0fbc077040 0x7f0fbc078430 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x7f0fc4018070 msgr2=0x7f0fc4018960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x7f0fc4018070 0x7f0fc4018960 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] conn(0x5650f666fce0 msgr2=0x5650f6722b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] conn(0x5650f666fce0 0x5650f6722b50 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x7f0fc40243f0 msgr2=0x7f0fc4024850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x7f0fc40243f0 0x7f0fc4024850 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x7f0fcc01b020 msgr2=0x7f0fcc056950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x7f0fcc01b020 0x7f0fcc056950 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x7f0fc40348f0 msgr2=0x7f0fc400e3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x7f0fc40348f0 0x7f0fc400e3e0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 reap_dead start 2024-09-16T01:40:11.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd1313640 1 -- 172.21.15.88:0/914797419 reap_dead start 2024-09-16T01:40:11.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0fb406ac90 msgr2=0x7f0fb406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0fb406ac90 0x7f0fb406d150 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0fc400da60 tx=0x7f0fc4009000 comp rx=0 tx=0).stop 2024-09-16T01:40:11.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fc0002810 msgr2=0x5650f67220f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fc0002810 0x5650f67220f0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x5650f6741530 tx=0x7f0fcc00e7d0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 shutdown_connections 2024-09-16T01:40:11.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0fb406ac90 0x7f0fb406d150 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 --2- 172.21.15.88:0/914797419 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fc0002810 0x5650f67220f0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 >> 172.21.15.88:0/914797419 conn(0x5650f66b7750 msgr2=0x5650f66b7b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:11.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 shutdown_connections 2024-09-16T01:40:11.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.642+0000 7f0fd301cf40 1 -- 172.21.15.88:0/914797419 wait complete. 2024-09-16T01:40:11.681 INFO:teuthology.orchestra.run.smithi088.stdout:1.00000000�������������������!�������������������������� ��d^_layout@@ _parent ���� ��  (y��f,��7(y��f,��78`z��fw�^8y��f,��7���� ��NH subdir_head1i"�z��f�-�A�5����������������z��f�-y��f,��7(z��f�-8`z��fw�^8`z��fw�^ ��������y��f,��7����security.selinux%unconfined_u:object_r:unlabeled_t:s0������������ �� �� 2024-09-16T01:40:11.682 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata ls 2024-09-16T01:40:11.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.759+0000 7f325dc2ff40 1 Processor -- start 2024-09-16T01:40:11.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.759+0000 7f325dc2ff40 1 -- start start 2024-09-16T01:40:11.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.760+0000 7f325dc2ff40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56330dad4df0 0x56330db77080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.760+0000 7f325dc2ff40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56330dad5a40 0x56330dc0bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.760+0000 7f325dc2ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56330daef230 0x56330dc0c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.760+0000 7f325dc2ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56330d9f7510 con 0x56330daef230 2024-09-16T01:40:11.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.760+0000 7f325dc2ff40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56330dad5200 con 0x56330dad4df0 2024-09-16T01:40:11.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.760+0000 7f325dc2ff40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56330daf0100 con 0x56330dad5a40 2024-09-16T01:40:11.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.760+0000 7f325bf26640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56330daef230 0x56330dc0c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.760+0000 7f325bf26640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56330daef230 0x56330dc0c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46036/0 (socket says 172.21.15.88:46036) 2024-09-16T01:40:11.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.760+0000 7f325bf26640 1 -- 172.21.15.88:0/1424654604 learned_addr learned my addr 172.21.15.88:0/1424654604 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:11.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.760+0000 7f325bf26640 1 -- 172.21.15.88:0/1424654604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56330dad5a40 msgr2=0x56330dc0bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:11.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.760+0000 7f325bf26640 1 --2- 172.21.15.88:0/1424654604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56330dad5a40 0x56330dc0bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.760+0000 7f325b725640 1 --2- 172.21.15.88:0/1424654604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56330dad4df0 0x56330db77080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.760+0000 7f325bf26640 1 -- 172.21.15.88:0/1424654604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56330dad4df0 msgr2=0x56330db77080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:11.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.760+0000 7f325bf26640 1 --2- 172.21.15.88:0/1424654604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56330dad4df0 0x56330db77080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.760+0000 7f325bf26640 1 -- 172.21.15.88:0/1424654604 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56330db7a050 con 0x56330daef230 2024-09-16T01:40:11.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.760+0000 7f325b725640 1 --2- 172.21.15.88:0/1424654604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56330dad4df0 0x56330db77080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:11.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.760+0000 7f325bf26640 1 --2- 172.21.15.88:0/1424654604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56330daef230 0x56330dc0c1d0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f32540015c0 tx=0x7f3254001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fe34536041d479e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.760+0000 7f325a723640 1 -- 172.21.15.88:0/1424654604 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3254003650 con 0x56330daef230 2024-09-16T01:40:11.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.760+0000 7f325a723640 1 -- 172.21.15.88:0/1424654604 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32540037f0 con 0x56330daef230 2024-09-16T01:40:11.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.760+0000 7f325a723640 1 -- 172.21.15.88:0/1424654604 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3254003c20 con 0x56330daef230 2024-09-16T01:40:11.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.761+0000 7f325dc2ff40 1 -- 172.21.15.88:0/1424654604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56330daef230 msgr2=0x56330dc0c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.761+0000 7f325dc2ff40 1 --2- 172.21.15.88:0/1424654604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56330daef230 0x56330dc0c1d0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f32540015c0 tx=0x7f3254001a80 comp rx=0 tx=0).stop 2024-09-16T01:40:11.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.761+0000 7f325dc2ff40 1 -- 172.21.15.88:0/1424654604 shutdown_connections 2024-09-16T01:40:11.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.761+0000 7f325dc2ff40 1 --2- 172.21.15.88:0/1424654604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56330daef230 0x56330dc0c1d0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.761+0000 7f325dc2ff40 1 --2- 172.21.15.88:0/1424654604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56330dad5a40 0x56330dc0bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.761+0000 7f325dc2ff40 1 --2- 172.21.15.88:0/1424654604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56330dad4df0 0x56330db77080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.761+0000 7f325dc2ff40 1 -- 172.21.15.88:0/1424654604 >> 172.21.15.88:0/1424654604 conn(0x56330daf0730 msgr2=0x56330db78ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:11.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.761+0000 7f325dc2ff40 1 -- 172.21.15.88:0/1424654604 shutdown_connections 2024-09-16T01:40:11.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.761+0000 7f325dc2ff40 1 -- 172.21.15.88:0/1424654604 wait complete. 2024-09-16T01:40:11.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.761+0000 7f325dc2ff40 1 Processor -- start 2024-09-16T01:40:11.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.762+0000 7f325dc2ff40 1 -- start start 2024-09-16T01:40:11.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.762+0000 7f325dc2ff40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56330dad4df0 0x56330dc08fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.762+0000 7f325dc2ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56330dad5a40 0x56330db75840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.762+0000 7f325dc2ff40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56330daef230 0x56330daeb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.762+0000 7f325dc2ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56330daf0350 con 0x56330dad5a40 2024-09-16T01:40:11.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.762+0000 7f325dc2ff40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56330db7a380 con 0x56330daef230 2024-09-16T01:40:11.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.762+0000 7f325dc2ff40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56330db7cec0 con 0x56330dad4df0 2024-09-16T01:40:11.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.762+0000 7f3253fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56330dad5a40 0x56330db75840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.762+0000 7f3253fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56330dad5a40 0x56330db75840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46052/0 (socket says 172.21.15.88:46052) 2024-09-16T01:40:11.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.762+0000 7f3253fff640 1 -- 172.21.15.88:0/1729242982 learned_addr learned my addr 172.21.15.88:0/1729242982 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:11.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.763+0000 7f325b725640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56330dad4df0 0x56330dc08fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.763+0000 7f325bf26640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56330daef230 0x56330daeb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.763+0000 7f3253fff640 1 -- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56330dad4df0 msgr2=0x56330dc08fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.763+0000 7f3253fff640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56330dad4df0 0x56330dc08fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.763+0000 7f3253fff640 1 -- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56330daef230 msgr2=0x56330daeb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.763+0000 7f3253fff640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56330daef230 0x56330daeb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.763+0000 7f3253fff640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56330dae9460 con 0x56330dad5a40 2024-09-16T01:40:11.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.763+0000 7f325bf26640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56330daef230 0x56330daeb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:11.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.763+0000 7f3253fff640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56330dad5a40 0x56330db75840 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f32440083b0 tx=0x7f3244008870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.763+0000 7f3259721640 1 -- 172.21.15.88:0/1729242982 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32440067a0 con 0x56330dad5a40 2024-09-16T01:40:11.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.763+0000 7f3259721640 1 -- 172.21.15.88:0/1729242982 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3244006960 con 0x56330dad5a40 2024-09-16T01:40:11.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.763+0000 7f3259721640 1 -- 172.21.15.88:0/1729242982 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3244006d50 con 0x56330dad5a40 2024-09-16T01:40:11.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.763+0000 7f325dc2ff40 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56330dae9740 con 0x56330dad5a40 2024-09-16T01:40:11.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.763+0000 7f325dc2ff40 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56330dae9d00 con 0x56330dad5a40 2024-09-16T01:40:11.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.764+0000 7f3259721640 1 -- 172.21.15.88:0/1729242982 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f324401c440 con 0x56330dad5a40 2024-09-16T01:40:11.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.765+0000 7f3259721640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f323c06ac90 0x7f323c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.765+0000 7f3259721640 1 -- 172.21.15.88:0/1729242982 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f324408a150 con 0x56330dad5a40 2024-09-16T01:40:11.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.765+0000 7f325b725640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f323c06ac90 0x7f323c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.765+0000 7f325dc2ff40 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] conn(0x56330dc07410 0x56330daea4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.765+0000 7f325dc2ff40 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] -- osd_op(unknown.0.0:1 12.0 12:00000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x56330dc07410 2024-09-16T01:40:11.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.765+0000 7f325b725640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f323c06ac90 0x7f323c06d150 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f324c057000 tx=0x7f324c05a040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.765+0000 7f325bf26640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] conn(0x56330dc07410 0x56330daea4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.766+0000 7f325bf26640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] conn(0x56330dc07410 0x56330daea4e0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.766+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 <== osd.4 v2:172.21.15.154:6816/2520717231 1 ==== osd_op_reply(1 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f325400f030 con 0x56330dc07410 2024-09-16T01:40:11.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.766+0000 7f325bf26640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x7f3254013930 0x7f32540123c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.766+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:08000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f3254013930 2024-09-16T01:40:11.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.766+0000 7f325b725640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x7f3254013930 0x7f32540123c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.766+0000 7f325b725640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x7f3254013930 0x7f32540123c0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.767+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(2 [pgnls start_epoch 86 out=177b] v85'10 uv10 ondisk = 1) ==== 144+0+177 (crc 0 0 0) 0x7f324c050bf0 con 0x7f3254013930 2024-09-16T01:40:11.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.767+0000 7f325b725640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x7f324c065bd0 0x7f324c058910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.767+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:3 12.8 12:10000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f324c065bd0 2024-09-16T01:40:11.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.767+0000 7f3253fff640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x7f324c065bd0 0x7f324c058910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.767+0000 7f3253fff640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x7f324c065bd0 0x7f324c058910 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.767+0000 7f3253fff640 1 -- 172.21.15.88:0/1729242982 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(3 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3244019020 con 0x7f324c065bd0 2024-09-16T01:40:11.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.767+0000 7f3253fff640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x7f3244009040 0x7f32440570b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.767+0000 7f3253fff640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:4 12.18 12:18000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f3244009040 2024-09-16T01:40:11.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.768+0000 7f325bf26640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x7f3244009040 0x7f32440570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.768+0000 7f325bf26640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x7f3244009040 0x7f32440570b0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.768+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(4 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f325400f030 con 0x7f3244009040 2024-09-16T01:40:11.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.768+0000 7f325bf26640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x7f3254021440 0x7f3254021880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.768+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:5 12.4 12:20000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f3254021440 2024-09-16T01:40:11.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.768+0000 7f325b725640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x7f3254021440 0x7f3254021880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.769+0000 7f325b725640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x7f3254021440 0x7f3254021880 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.769+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(5 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f324c050bf0 con 0x7f3254021440 2024-09-16T01:40:11.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.769+0000 7f325b725640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x7f324c04f3e0 0x7f324c04f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.769+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:6 12.14 12:28000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f324c04f3e0 2024-09-16T01:40:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.769+0000 7f3253fff640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x7f324c04f3e0 0x7f324c04f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.770+0000 7f3253fff640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x7f324c04f3e0 0x7f324c04f820 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.770+0000 7f3253fff640 1 -- 172.21.15.88:0/1729242982 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(6 [pgnls start_epoch 86 out=149b] v85'20 uv20 ondisk = 1) ==== 144+0+149 (crc 0 0 0) 0x7f3244019020 con 0x7f324c04f3e0 2024-09-16T01:40:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.770+0000 7f3253fff640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.c 12:30000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f3254013930 2024-09-16T01:40:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.770+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(7 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f324c050bf0 con 0x7f3254013930 2024-09-16T01:40:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.770+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.1c 12:38000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f3254013930 2024-09-16T01:40:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.771+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(8 [pgnls start_epoch 86 out=73b] v84'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f324c050bf0 con 0x7f3254013930 2024-09-16T01:40:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.771+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] -- osd_op(unknown.0.0:9 12.2 12:40000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x56330dc07410 2024-09-16T01:40:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.771+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 <== osd.4 v2:172.21.15.154:6816/2520717231 2 ==== osd_op_reply(9 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f325400f030 con 0x56330dc07410 2024-09-16T01:40:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.771+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:10 12.12 12:48000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f324c065bd0 2024-09-16T01:40:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.771+0000 7f3253fff640 1 -- 172.21.15.88:0/1729242982 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(10 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3244019020 con 0x7f324c065bd0 2024-09-16T01:40:11.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.771+0000 7f3253fff640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] conn(0x7f3244088190 0x7f324405d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.771+0000 7f3253fff640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] -- osd_op(unknown.0.0:11 12.a 12:50000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f3244088190 2024-09-16T01:40:11.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.771+0000 7f325bf26640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] conn(0x7f3244088190 0x7f324405d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.772+0000 7f325bf26640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] conn(0x7f3244088190 0x7f324405d730 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.772+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 <== osd.2 v2:172.21.15.88:6808/2918260303 1 ==== osd_op_reply(11 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f325400f030 con 0x7f3244088190 2024-09-16T01:40:11.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.772+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] -- osd_op(unknown.0.0:12 12.1a 12:58000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x56330dc07410 2024-09-16T01:40:11.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.772+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 <== osd.4 v2:172.21.15.154:6816/2520717231 3 ==== osd_op_reply(12 [pgnls start_epoch 86 out=49b] v84'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f325400f030 con 0x56330dc07410 2024-09-16T01:40:11.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.772+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:13 12.6 12:60000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f3244009040 2024-09-16T01:40:11.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.772+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(13 [pgnls start_epoch 86 out=77b] v85'3 uv3 ondisk = 1) ==== 144+0+77 (crc 0 0 0) 0x7f325400f030 con 0x7f3244009040 2024-09-16T01:40:11.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.772+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:14 12.16 12:68000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f3254021440 2024-09-16T01:40:11.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.773+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(14 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f324c050bf0 con 0x7f3254021440 2024-09-16T01:40:11.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.773+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.e 12:70000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f3254013930 2024-09-16T01:40:11.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.773+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(15 [pgnls start_epoch 86 out=73b] v84'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f324c050bf0 con 0x7f3254013930 2024-09-16T01:40:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.773+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] -- osd_op(unknown.0.0:16 12.1e 12:78000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x56330dc07410 2024-09-16T01:40:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.773+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 <== osd.4 v2:172.21.15.154:6816/2520717231 4 ==== osd_op_reply(16 [pgnls start_epoch 86 out=49b] v84'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f325400f030 con 0x56330dc07410 2024-09-16T01:40:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.773+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:17 12.1 12:80000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f3254021440 2024-09-16T01:40:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.774+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(17 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f324c050bf0 con 0x7f3254021440 2024-09-16T01:40:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.774+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:18 12.11 12:88000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f324c04f3e0 2024-09-16T01:40:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.774+0000 7f3253fff640 1 -- 172.21.15.88:0/1729242982 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(18 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f3244019020 con 0x7f324c04f3e0 2024-09-16T01:40:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.774+0000 7f3253fff640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] -- osd_op(unknown.0.0:19 12.9 12:90000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f3244088190 2024-09-16T01:40:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.774+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 <== osd.2 v2:172.21.15.88:6808/2918260303 2 ==== osd_op_reply(19 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f325400f030 con 0x7f3244088190 2024-09-16T01:40:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.774+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.19 12:98000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f3254013930 2024-09-16T01:40:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.775+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(20 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f324c050bf0 con 0x7f3254013930 2024-09-16T01:40:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.775+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:21 12.5 12:a0000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f3254021440 2024-09-16T01:40:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.775+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(21 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f324c050bf0 con 0x7f3254021440 2024-09-16T01:40:11.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.775+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:22 12.15 12:a8000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f3254021440 2024-09-16T01:40:11.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.775+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(22 [pgnls start_epoch 86 out=97b] v84'3 uv3 ondisk = 1) ==== 144+0+97 (crc 0 0 0) 0x7f324c050bf0 con 0x7f3254021440 2024-09-16T01:40:11.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.775+0000 7f325b725640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x7f324c0718b0 0x7f324c0633d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.775+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:23 12.d 12:b0000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f324c0718b0 2024-09-16T01:40:11.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.775+0000 7f325b725640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x7f324c0718b0 0x7f324c0633d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.776+0000 7f325b725640 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x7f324c0718b0 0x7f324c0633d0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.776+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(23 [pgnls start_epoch 86 out=74b] v84'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7f324c08f040 con 0x7f324c0718b0 2024-09-16T01:40:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.776+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.1d 12:b8000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f3254013930 2024-09-16T01:40:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.776+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(24 [pgnls start_epoch 86 out=73b] v84'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f324c08f040 con 0x7f3254013930 2024-09-16T01:40:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.776+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.3 12:c0000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f3254013930 2024-09-16T01:40:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.777+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(25 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f324c08f040 con 0x7f3254013930 2024-09-16T01:40:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.777+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:26 12.13 12:c8000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f3244009040 2024-09-16T01:40:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.777+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(26 [pgnls start_epoch 86 out=98b] v85'3 uv3 ondisk = 1) ==== 144+0+98 (crc 0 0 0) 0x7f325400f030 con 0x7f3244009040 2024-09-16T01:40:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.777+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:27 12.b 12:d0000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f324c065bd0 2024-09-16T01:40:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.777+0000 7f3253fff640 1 -- 172.21.15.88:0/1729242982 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(27 [pgnls start_epoch 86 out=76b] v85'3 uv3 ondisk = 1) ==== 144+0+76 (crc 0 0 0) 0x7f3244019020 con 0x7f324c065bd0 2024-09-16T01:40:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.777+0000 7f3253fff640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] -- osd_op(unknown.0.0:28 12.1b 12:d8000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x56330dc07410 2024-09-16T01:40:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.777+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 <== osd.4 v2:172.21.15.154:6816/2520717231 5 ==== osd_op_reply(28 [pgnls start_epoch 86 out=49b] v84'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f325400f030 con 0x56330dc07410 2024-09-16T01:40:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.778+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:29 12.7 12:e0000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f3244009040 2024-09-16T01:40:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.778+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(29 [pgnls start_epoch 86 out=73b] v84'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f325400f030 con 0x7f3244009040 2024-09-16T01:40:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.778+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:30 12.17 12:e8000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f3254021440 2024-09-16T01:40:11.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.778+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(30 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f324c08f040 con 0x7f3254021440 2024-09-16T01:40:11.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.778+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:31 12.f 12:f0000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f3254021440 2024-09-16T01:40:11.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(31 [pgnls start_epoch 86 out=73b] v84'1 uv1 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7f324c08f040 con 0x7f3254021440 2024-09-16T01:40:11.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:32 12.1f 12:f8000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x56330daeaa20 con 0x7f3254021440 2024-09-16T01:40:11.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325b725640 1 -- 172.21.15.88:0/1729242982 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(32 [pgnls start_epoch 86 out=101b] v85'6 uv6 ondisk = 1) ==== 144+0+101 (crc 0 0 0) 0x7f324c08f040 con 0x7f3254021440 2024-09-16T01:40:11.786 INFO:teuthology.orchestra.run.smithi088.stdout:601.00000000 2024-09-16T01:40:11.786 INFO:teuthology.orchestra.run.smithi088.stdout:602.00000000 2024-09-16T01:40:11.787 INFO:teuthology.orchestra.run.smithi088.stdout:10000000000.00000000 2024-09-16T01:40:11.787 INFO:teuthology.orchestra.run.smithi088.stdout:600.00000000 2024-09-16T01:40:11.787 INFO:teuthology.orchestra.run.smithi088.stdout:603.00000000 2024-09-16T01:40:11.787 INFO:teuthology.orchestra.run.smithi088.stdout:1.00000000.inode 2024-09-16T01:40:11.787 INFO:teuthology.orchestra.run.smithi088.stdout:200.00000000 2024-09-16T01:40:11.787 INFO:teuthology.orchestra.run.smithi088.stdout:200.00000001 2024-09-16T01:40:11.787 INFO:teuthology.orchestra.run.smithi088.stdout:606.00000000 2024-09-16T01:40:11.787 INFO:teuthology.orchestra.run.smithi088.stdout:607.00000000 2024-09-16T01:40:11.787 INFO:teuthology.orchestra.run.smithi088.stdout:mds0_openfiles.0 2024-09-16T01:40:11.787 INFO:teuthology.orchestra.run.smithi088.stdout:608.00000000 2024-09-16T01:40:11.787 INFO:teuthology.orchestra.run.smithi088.stdout:604.00000000 2024-09-16T01:40:11.787 INFO:teuthology.orchestra.run.smithi088.stdout:500.00000000 2024-09-16T01:40:11.787 INFO:teuthology.orchestra.run.smithi088.stdout:mds_snaptable 2024-09-16T01:40:11.788 INFO:teuthology.orchestra.run.smithi088.stdout:605.00000000 2024-09-16T01:40:11.788 INFO:teuthology.orchestra.run.smithi088.stdout:mds0_inotable 2024-09-16T01:40:11.788 INFO:teuthology.orchestra.run.smithi088.stdout:100.00000000 2024-09-16T01:40:11.788 INFO:teuthology.orchestra.run.smithi088.stdout:mds0_sessionmap 2024-09-16T01:40:11.788 INFO:teuthology.orchestra.run.smithi088.stdout:609.00000000 2024-09-16T01:40:11.788 INFO:teuthology.orchestra.run.smithi088.stdout:400.00000000 2024-09-16T01:40:11.788 INFO:teuthology.orchestra.run.smithi088.stdout:100.00000000.inode 2024-09-16T01:40:11.788 INFO:teuthology.orchestra.run.smithi088.stdout:1.00000000 2024-09-16T01:40:11.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 -- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x7f3254013930 msgr2=0x7f32540123c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x7f3254013930 0x7f32540123c0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 -- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x7f3244009040 msgr2=0x7f32440570b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x7f3244009040 0x7f32440570b0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 -- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] conn(0x7f3244088190 msgr2=0x7f324405d730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] conn(0x7f3244088190 0x7f324405d730 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 -- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x7f324c065bd0 msgr2=0x7f324c058910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x7f324c065bd0 0x7f324c058910 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 -- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] conn(0x56330dc07410 msgr2=0x56330daea4e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] conn(0x56330dc07410 0x56330daea4e0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 -- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x7f324c04f3e0 msgr2=0x7f324c04f820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x7f324c04f3e0 0x7f324c04f820 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 -- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x7f3254021440 msgr2=0x7f3254021880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x7f3254021440 0x7f3254021880 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 -- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x7f324c0718b0 msgr2=0x7f324c0633d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x7f324c0718b0 0x7f324c0633d0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 reap_dead start 2024-09-16T01:40:11.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 reap_dead start 2024-09-16T01:40:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 reap_dead start 2024-09-16T01:40:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325bf26640 1 -- 172.21.15.88:0/1729242982 reap_dead start 2024-09-16T01:40:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 -- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f323c06ac90 msgr2=0x7f323c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f323c06ac90 0x7f323c06d150 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f324c057000 tx=0x7f324c05a040 comp rx=0 tx=0).stop 2024-09-16T01:40:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 -- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56330dad5a40 msgr2=0x56330db75840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56330dad5a40 0x56330db75840 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f32440083b0 tx=0x7f3244008870 comp rx=0 tx=0).stop 2024-09-16T01:40:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 -- 172.21.15.88:0/1729242982 shutdown_connections 2024-09-16T01:40:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f323c06ac90 0x7f323c06d150 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 --2- 172.21.15.88:0/1729242982 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56330dad5a40 0x56330db75840 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 -- 172.21.15.88:0/1729242982 >> 172.21.15.88:0/1729242982 conn(0x56330daf0730 msgr2=0x56330db77080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 -- 172.21.15.88:0/1729242982 shutdown_connections 2024-09-16T01:40:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.779+0000 7f325dc2ff40 1 -- 172.21.15.88:0/1729242982 wait complete. 2024-09-16T01:40:11.796 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 10000000000.00000000 2024-09-16T01:40:11.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.871+0000 7f3c0483cf40 1 Processor -- start 2024-09-16T01:40:11.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.871+0000 7f3c0483cf40 1 -- start start 2024-09-16T01:40:11.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.872+0000 7f3c0483cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bec0022f0 0x564532ae8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.872+0000 7f3c0483cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564532ac9840 0x564532ac7020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.872+0000 7f3c0483cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564532a5b2a0 0x564532a5b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.872+0000 7f3c0483cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564532965510 con 0x564532ac9840 2024-09-16T01:40:11.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.872+0000 7f3c0483cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5645329bec50 con 0x7f3bec0022f0 2024-09-16T01:40:11.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.872+0000 7f3c0483cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x564532acadb0 con 0x564532a5b2a0 2024-09-16T01:40:11.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.872+0000 7f3c01b33640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564532ac9840 0x564532ac7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.872+0000 7f3c01b33640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564532ac9840 0x564532ac7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46064/0 (socket says 172.21.15.88:46064) 2024-09-16T01:40:11.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.872+0000 7f3c01b33640 1 -- 172.21.15.88:0/2392387538 learned_addr learned my addr 172.21.15.88:0/2392387538 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:11.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.872+0000 7f3c01b33640 1 -- 172.21.15.88:0/2392387538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564532a5b2a0 msgr2=0x564532a5b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:11.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.872+0000 7f3c01b33640 1 --2- 172.21.15.88:0/2392387538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564532a5b2a0 0x564532a5b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.872+0000 7f3c01b33640 1 -- 172.21.15.88:0/2392387538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bec0022f0 msgr2=0x564532ae8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:11.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.872+0000 7f3c01b33640 1 --2- 172.21.15.88:0/2392387538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bec0022f0 0x564532ae8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.872+0000 7f3c01b33640 1 -- 172.21.15.88:0/2392387538 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564532ac9d80 con 0x564532ac9840 2024-09-16T01:40:11.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.872+0000 7f3c01b33640 1 --2- 172.21.15.88:0/2392387538 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564532ac9840 0x564532ac7020 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f3bf800d560 tx=0x7f3bf800da20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad5a2ac9bff839c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.872+0000 7f3c00b31640 1 -- 172.21.15.88:0/2392387538 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf8013070 con 0x564532ac9840 2024-09-16T01:40:11.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.873+0000 7f3c00b31640 1 -- 172.21.15.88:0/2392387538 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bf800f730 con 0x564532ac9840 2024-09-16T01:40:11.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.873+0000 7f3c00b31640 1 -- 172.21.15.88:0/2392387538 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf800fb20 con 0x564532ac9840 2024-09-16T01:40:11.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.873+0000 7f3c0483cf40 1 -- 172.21.15.88:0/2392387538 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564532ac9840 msgr2=0x564532ac7020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.873+0000 7f3c0483cf40 1 --2- 172.21.15.88:0/2392387538 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564532ac9840 0x564532ac7020 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f3bf800d560 tx=0x7f3bf800da20 comp rx=0 tx=0).stop 2024-09-16T01:40:11.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.873+0000 7f3c0483cf40 1 -- 172.21.15.88:0/2392387538 shutdown_connections 2024-09-16T01:40:11.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.873+0000 7f3c0483cf40 1 --2- 172.21.15.88:0/2392387538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564532a5b2a0 0x564532a5b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.873+0000 7f3c0483cf40 1 --2- 172.21.15.88:0/2392387538 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564532ac9840 0x564532ac7020 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.873+0000 7f3c0483cf40 1 --2- 172.21.15.88:0/2392387538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bec0022f0 0x564532ae8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.873+0000 7f3c0483cf40 1 -- 172.21.15.88:0/2392387538 >> 172.21.15.88:0/2392387538 conn(0x564532a12be0 msgr2=0x564532aebc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:11.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.873+0000 7f3c0483cf40 1 -- 172.21.15.88:0/2392387538 shutdown_connections 2024-09-16T01:40:11.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.873+0000 7f3c0483cf40 1 -- 172.21.15.88:0/2392387538 wait complete. 2024-09-16T01:40:11.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.874+0000 7f3c0483cf40 1 Processor -- start 2024-09-16T01:40:11.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.874+0000 7f3c0483cf40 1 -- start start 2024-09-16T01:40:11.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.874+0000 7f3c0483cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564532a5b2a0 0x564532a5b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.874+0000 7f3c0483cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532ac9840 0x564532ae3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.874+0000 7f3c0483cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bec0022f0 0x564532acc550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.874+0000 7f3c0483cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564532a58550 con 0x7f3bec0022f0 2024-09-16T01:40:11.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.874+0000 7f3c0483cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564532acb000 con 0x564532ac9840 2024-09-16T01:40:11.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.874+0000 7f3c0483cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x564532aeb6f0 con 0x564532a5b2a0 2024-09-16T01:40:11.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.874+0000 7f3c02b35640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bec0022f0 0x564532acc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.874+0000 7f3c02b35640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bec0022f0 0x564532acc550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46078/0 (socket says 172.21.15.88:46078) 2024-09-16T01:40:11.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.874+0000 7f3c02b35640 1 -- 172.21.15.88:0/331171520 learned_addr learned my addr 172.21.15.88:0/331171520 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:11.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.875+0000 7f3c01b33640 1 --2- 172.21.15.88:0/331171520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532ac9840 0x564532ae3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.875+0000 7f3c02334640 1 --2- 172.21.15.88:0/331171520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564532a5b2a0 0x564532a5b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.875+0000 7f3c02b35640 1 -- 172.21.15.88:0/331171520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564532a5b2a0 msgr2=0x564532a5b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.875+0000 7f3c02b35640 1 --2- 172.21.15.88:0/331171520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564532a5b2a0 0x564532a5b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.875+0000 7f3c02b35640 1 -- 172.21.15.88:0/331171520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532ac9840 msgr2=0x564532ae3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.875+0000 7f3c02b35640 1 --2- 172.21.15.88:0/331171520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532ac9840 0x564532ae3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.875+0000 7f3c02b35640 1 -- 172.21.15.88:0/331171520 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564532a5bc60 con 0x7f3bec0022f0 2024-09-16T01:40:11.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.875+0000 7f3c01b33640 1 --2- 172.21.15.88:0/331171520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532ac9840 0x564532ae3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:11.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.875+0000 7f3c02b35640 1 --2- 172.21.15.88:0/331171520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bec0022f0 0x564532acc550 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x564532a19120 tx=0x7f3bfc00ddd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.875+0000 7f3bf37fe640 1 -- 172.21.15.88:0/331171520 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc00b770 con 0x7f3bec0022f0 2024-09-16T01:40:11.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.875+0000 7f3bf37fe640 1 -- 172.21.15.88:0/331171520 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bfc016070 con 0x7f3bec0022f0 2024-09-16T01:40:11.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.875+0000 7f3c0483cf40 1 -- 172.21.15.88:0/331171520 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564532ae7260 con 0x7f3bec0022f0 2024-09-16T01:40:11.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.875+0000 7f3bf37fe640 1 -- 172.21.15.88:0/331171520 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc00bb50 con 0x7f3bec0022f0 2024-09-16T01:40:11.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.875+0000 7f3c0483cf40 1 -- 172.21.15.88:0/331171520 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564532ae7820 con 0x7f3bec0022f0 2024-09-16T01:40:11.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.876+0000 7f3bf37fe640 1 -- 172.21.15.88:0/331171520 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3bfc00bcf0 con 0x7f3bec0022f0 2024-09-16T01:40:11.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.877+0000 7f3bf37fe640 1 --2- 172.21.15.88:0/331171520 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3be406ac90 0x7f3be406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.877+0000 7f3bf37fe640 1 -- 172.21.15.88:0/331171520 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f3bfc089c10 con 0x7f3bec0022f0 2024-09-16T01:40:11.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.877+0000 7f3c02334640 1 --2- 172.21.15.88:0/331171520 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3be406ac90 0x7f3be406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.877+0000 7f3c0483cf40 1 --2- 172.21.15.88:0/331171520 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x564532a5bfb0 0x564532a55e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.877+0000 7f3c0483cf40 1 -- 172.21.15.88:0/331171520 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x564532a56340 con 0x564532a5bfb0 2024-09-16T01:40:11.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.877+0000 7f3c01b33640 1 --2- 172.21.15.88:0/331171520 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x564532a5bfb0 0x564532a55e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.877+0000 7f3c02334640 1 --2- 172.21.15.88:0/331171520 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3be406ac90 0x7f3be406d150 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4058000 tx=0x7f3bf404f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.878+0000 7f3c01b33640 1 --2- 172.21.15.88:0/331171520 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x564532a5bfb0 0x564532a55e00 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.878+0000 7f3c01b33640 1 -- 172.21.15.88:0/331171520 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 10000000000.00000000 [omap-get-header out=274b] v0'0 uv10 ondisk = 0) ==== 164+0+274 (crc 0 0 0) 0x7f3bf8015030 con 0x564532a5bfb0 2024-09-16T01:40:11.883 INFO:teuthology.orchestra.run.smithi088.stdout:header (274 bytes) : 2024-09-16T01:40:11.883 INFO:teuthology.orchestra.run.smithi088.stdout:00000000 04 03 0c 01 00 00 0b 00 00 00 00 00 00 00 00 00 |................| 2024-09-16T01:40:11.883 INFO:teuthology.orchestra.run.smithi088.stdout:00000010 00 00 00 00 00 00 03 02 28 00 00 00 00 00 00 00 |........(.......| 2024-09-16T01:40:11.883 INFO:teuthology.orchestra.run.smithi088.stdout:00000020 00 00 00 00 7a 8c e7 66 03 8f 2d 03 01 00 00 00 |....z..f..-.....| 2024-09-16T01:40:11.883 INFO:teuthology.orchestra.run.smithi088.stdout:00000030 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 |................| 2024-09-16T01:40:11.883 INFO:teuthology.orchestra.run.smithi088.stdout:00000040 00 00 00 00 03 02 28 00 00 00 00 00 00 00 00 00 |......(.........| 2024-09-16T01:40:11.883 INFO:teuthology.orchestra.run.smithi088.stdout:00000050 00 00 7a 8c e7 66 03 8f 2d 03 01 00 00 00 00 00 |..z..f..-.......| 2024-09-16T01:40:11.884 INFO:teuthology.orchestra.run.smithi088.stdout:00000060 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 |................| 2024-09-16T01:40:11.884 INFO:teuthology.orchestra.run.smithi088.stdout:00000070 00 00 03 02 38 00 00 00 00 00 00 00 00 00 00 00 |....8...........| 2024-09-16T01:40:11.884 INFO:teuthology.orchestra.run.smithi088.stdout:00000080 00 00 60 00 00 00 00 00 01 00 00 00 00 00 00 00 |..`.............| 2024-09-16T01:40:11.884 INFO:teuthology.orchestra.run.smithi088.stdout:00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-16T01:40:11.884 INFO:teuthology.orchestra.run.smithi088.stdout:000000a0 00 00 00 00 00 00 00 00 7a 8c e7 66 77 ba 5e 04 |........z..fw.^.| 2024-09-16T01:40:11.884 INFO:teuthology.orchestra.run.smithi088.stdout:000000b0 03 02 38 00 00 00 00 00 00 00 00 00 00 00 00 00 |..8.............| 2024-09-16T01:40:11.884 INFO:teuthology.orchestra.run.smithi088.stdout:000000c0 60 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 |`...............| 2024-09-16T01:40:11.884 INFO:teuthology.orchestra.run.smithi088.stdout:000000d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-16T01:40:11.884 INFO:teuthology.orchestra.run.smithi088.stdout:000000e0 00 00 00 00 00 00 7a 8c e7 66 77 ba 5e 04 00 00 |......z..fw.^...| 2024-09-16T01:40:11.884 INFO:teuthology.orchestra.run.smithi088.stdout:000000f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-16T01:40:11.884 INFO:teuthology.orchestra.run.smithi088.stdout:* 2024-09-16T01:40:11.884 INFO:teuthology.orchestra.run.smithi088.stdout:00000110 00 00 |..| 2024-09-16T01:40:11.884 INFO:teuthology.orchestra.run.smithi088.stdout:00000112 2024-09-16T01:40:11.885 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:11.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.878+0000 7f3c0483cf40 1 -- 172.21.15.88:0/331171520 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x564532a5bfb0 msgr2=0x564532a55e00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.878+0000 7f3c0483cf40 1 --2- 172.21.15.88:0/331171520 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x564532a5bfb0 0x564532a55e00 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.878+0000 7f3c0483cf40 1 -- 172.21.15.88:0/331171520 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3be406ac90 msgr2=0x7f3be406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.878+0000 7f3c0483cf40 1 --2- 172.21.15.88:0/331171520 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3be406ac90 0x7f3be406d150 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4058000 tx=0x7f3bf404f000 comp rx=0 tx=0).stop 2024-09-16T01:40:11.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.878+0000 7f3c0483cf40 1 -- 172.21.15.88:0/331171520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bec0022f0 msgr2=0x564532acc550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.878+0000 7f3c0483cf40 1 --2- 172.21.15.88:0/331171520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bec0022f0 0x564532acc550 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x564532a19120 tx=0x7f3bfc00ddd0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.878+0000 7f3c02b35640 1 -- 172.21.15.88:0/331171520 reap_dead start 2024-09-16T01:40:11.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.878+0000 7f3c0483cf40 1 -- 172.21.15.88:0/331171520 shutdown_connections 2024-09-16T01:40:11.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.878+0000 7f3c0483cf40 1 -- 172.21.15.88:0/331171520 >> 172.21.15.88:0/331171520 conn(0x564532a12be0 msgr2=0x564532ac8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:11.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.878+0000 7f3c0483cf40 1 -- 172.21.15.88:0/331171520 shutdown_connections 2024-09-16T01:40:11.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.878+0000 7f3c0483cf40 1 -- 172.21.15.88:0/331171520 wait complete. 2024-09-16T01:40:11.887 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 600.00000000 2024-09-16T01:40:11.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.968+0000 7f40b2b25f40 1 Processor -- start 2024-09-16T01:40:11.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.968+0000 7f40b2b25f40 1 -- start start 2024-09-16T01:40:11.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.968+0000 7f40b2b25f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f409c003ab0 0x558843154c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.968+0000 7f40b2b25f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588431d6700 0x5588431551a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.969+0000 7f40b2b25f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5588431e2540 0x55884310cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.969+0000 7f40b2b25f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55884305f510 con 0x5588431e2540 2024-09-16T01:40:11.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.969+0000 7f40b2b25f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5588430b8c50 con 0x5588431d6700 2024-09-16T01:40:11.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.969+0000 7f40b2b25f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5588431e1270 con 0x7f409c003ab0 2024-09-16T01:40:11.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.969+0000 7f40b0504640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5588431e2540 0x55884310cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.969+0000 7f40b0504640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5588431e2540 0x55884310cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46106/0 (socket says 172.21.15.88:46106) 2024-09-16T01:40:11.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.969+0000 7f40b0504640 1 -- 172.21.15.88:0/170457467 learned_addr learned my addr 172.21.15.88:0/170457467 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.969+0000 7f40afd03640 1 --2- 172.21.15.88:0/170457467 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f409c003ab0 0x558843154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.969+0000 7f40af502640 1 --2- 172.21.15.88:0/170457467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588431d6700 0x5588431551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.969+0000 7f40b0504640 1 -- 172.21.15.88:0/170457467 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f409c003ab0 msgr2=0x558843154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.969+0000 7f40b0504640 1 --2- 172.21.15.88:0/170457467 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f409c003ab0 0x558843154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.969+0000 7f40b0504640 1 -- 172.21.15.88:0/170457467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588431d6700 msgr2=0x5588431551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.969+0000 7f40b0504640 1 --2- 172.21.15.88:0/170457467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588431d6700 0x5588431551a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.969+0000 7f40b0504640 1 -- 172.21.15.88:0/170457467 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588431e3e20 con 0x5588431e2540 2024-09-16T01:40:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.969+0000 7f40b0504640 1 --2- 172.21.15.88:0/170457467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5588431e2540 0x55884310cbe0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f40a0006d50 tx=0x7f40a0008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=580615058e8544fa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.969+0000 7f40ae500640 1 -- 172.21.15.88:0/170457467 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a00063b0 con 0x5588431e2540 2024-09-16T01:40:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.969+0000 7f40ae500640 1 -- 172.21.15.88:0/170457467 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40a00017b0 con 0x5588431e2540 2024-09-16T01:40:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.969+0000 7f40ae500640 1 -- 172.21.15.88:0/170457467 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a0001be0 con 0x5588431e2540 2024-09-16T01:40:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.970+0000 7f40b2b25f40 1 -- 172.21.15.88:0/170457467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5588431e2540 msgr2=0x55884310cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.970+0000 7f40b2b25f40 1 --2- 172.21.15.88:0/170457467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5588431e2540 0x55884310cbe0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f40a0006d50 tx=0x7f40a0008030 comp rx=0 tx=0).stop 2024-09-16T01:40:11.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.970+0000 7f40b2b25f40 1 -- 172.21.15.88:0/170457467 shutdown_connections 2024-09-16T01:40:11.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.970+0000 7f40b2b25f40 1 --2- 172.21.15.88:0/170457467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5588431e2540 0x55884310cbe0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.970+0000 7f40b2b25f40 1 --2- 172.21.15.88:0/170457467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5588431d6700 0x5588431551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.970+0000 7f40b2b25f40 1 --2- 172.21.15.88:0/170457467 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f409c003ab0 0x558843154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.970+0000 7f40b2b25f40 1 -- 172.21.15.88:0/170457467 >> 172.21.15.88:0/170457467 conn(0x558843158750 msgr2=0x558843139a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:11.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.970+0000 7f40b2b25f40 1 -- 172.21.15.88:0/170457467 shutdown_connections 2024-09-16T01:40:11.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.970+0000 7f40b2b25f40 1 -- 172.21.15.88:0/170457467 wait complete. 2024-09-16T01:40:11.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.970+0000 7f40b2b25f40 1 Processor -- start 2024-09-16T01:40:11.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.970+0000 7f40b2b25f40 1 -- start start 2024-09-16T01:40:11.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.971+0000 7f40b2b25f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588431d6700 0x5588431ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.971+0000 7f40b2b25f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5588431e2540 0x5588431d1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.971+0000 7f40b2b25f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f409c003ab0 0x5588431d1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.971+0000 7f40b2b25f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5588431e0f00 con 0x5588431e2540 2024-09-16T01:40:11.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.971+0000 7f40b2b25f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5588431e3930 con 0x7f409c003ab0 2024-09-16T01:40:11.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.971+0000 7f40b2b25f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5588431e5130 con 0x5588431d6700 2024-09-16T01:40:11.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.971+0000 7f40af502640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5588431e2540 0x5588431d1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.971+0000 7f40af502640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5588431e2540 0x5588431d1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46114/0 (socket says 172.21.15.88:46114) 2024-09-16T01:40:11.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.971+0000 7f40af502640 1 -- 172.21.15.88:0/3639146693 learned_addr learned my addr 172.21.15.88:0/3639146693 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:11.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.971+0000 7f40b0504640 1 --2- 172.21.15.88:0/3639146693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f409c003ab0 0x5588431d1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.971+0000 7f40afd03640 1 --2- 172.21.15.88:0/3639146693 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588431d6700 0x5588431ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.972+0000 7f40af502640 1 -- 172.21.15.88:0/3639146693 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588431d6700 msgr2=0x5588431ca3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.972+0000 7f40af502640 1 --2- 172.21.15.88:0/3639146693 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5588431d6700 0x5588431ca3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.972+0000 7f40af502640 1 -- 172.21.15.88:0/3639146693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f409c003ab0 msgr2=0x5588431d1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.972+0000 7f40af502640 1 --2- 172.21.15.88:0/3639146693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f409c003ab0 0x5588431d1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.972+0000 7f40af502640 1 -- 172.21.15.88:0/3639146693 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588431e5510 con 0x5588431e2540 2024-09-16T01:40:11.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.972+0000 7f40af502640 1 --2- 172.21.15.88:0/3639146693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5588431e2540 0x5588431d1a00 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f4098000c00 tx=0x7f409800e6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.972+0000 7f40ad4fe640 1 -- 172.21.15.88:0/3639146693 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4098003110 con 0x5588431e2540 2024-09-16T01:40:11.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.972+0000 7f40ad4fe640 1 -- 172.21.15.88:0/3639146693 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4098017020 con 0x5588431e2540 2024-09-16T01:40:11.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.972+0000 7f40ad4fe640 1 -- 172.21.15.88:0/3639146693 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40980032d0 con 0x5588431e2540 2024-09-16T01:40:11.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.972+0000 7f40b2b25f40 1 -- 172.21.15.88:0/3639146693 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5588431d58a0 con 0x5588431e2540 2024-09-16T01:40:11.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.972+0000 7f40b2b25f40 1 -- 172.21.15.88:0/3639146693 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588431d5e60 con 0x5588431e2540 2024-09-16T01:40:11.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.973+0000 7f40ad4fe640 1 -- 172.21.15.88:0/3639146693 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f40980014c0 con 0x5588431e2540 2024-09-16T01:40:11.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.974+0000 7f40ad4fe640 1 --2- 172.21.15.88:0/3639146693 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f409006ac90 0x7f409006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.974+0000 7f40ad4fe640 1 -- 172.21.15.88:0/3639146693 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f409808a4d0 con 0x5588431e2540 2024-09-16T01:40:11.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.974+0000 7f40afd03640 1 --2- 172.21.15.88:0/3639146693 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f409006ac90 0x7f409006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.974+0000 7f40b2b25f40 1 --2- 172.21.15.88:0/3639146693 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x558843158d70 0x558843280400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:11.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.974+0000 7f40b2b25f40 1 -- 172.21.15.88:0/3639146693 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:0d82a743:::600.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5588431e3490 con 0x558843158d70 2024-09-16T01:40:11.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.974+0000 7f40b0504640 1 --2- 172.21.15.88:0/3639146693 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x558843158d70 0x558843280400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:11.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.974+0000 7f40afd03640 1 --2- 172.21.15.88:0/3639146693 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f409006ac90 0x7f409006d150 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f40a8005400 tx=0x7f40a800c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.974+0000 7f40b0504640 1 --2- 172.21.15.88:0/3639146693 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x558843158d70 0x558843280400 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:11.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.975+0000 7f40b0504640 1 -- 172.21.15.88:0/3639146693 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 600.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f40a0001960 con 0x558843158d70 2024-09-16T01:40:11.980 INFO:teuthology.orchestra.run.smithi088.stdout:header (274 bytes) : 2024-09-16T01:40:11.980 INFO:teuthology.orchestra.run.smithi088.stdout:00000000 04 03 0c 01 00 00 01 00 00 00 00 00 00 00 00 00 |................| 2024-09-16T01:40:11.980 INFO:teuthology.orchestra.run.smithi088.stdout:00000010 00 00 00 00 00 00 03 02 28 00 00 00 00 00 00 00 |........(.......| 2024-09-16T01:40:11.980 INFO:teuthology.orchestra.run.smithi088.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-16T01:40:11.980 INFO:teuthology.orchestra.run.smithi088.stdout:* 2024-09-16T01:40:11.980 INFO:teuthology.orchestra.run.smithi088.stdout:00000040 00 00 00 00 03 02 28 00 00 00 00 00 00 00 00 00 |......(.........| 2024-09-16T01:40:11.980 INFO:teuthology.orchestra.run.smithi088.stdout:00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-16T01:40:11.980 INFO:teuthology.orchestra.run.smithi088.stdout:* 2024-09-16T01:40:11.980 INFO:teuthology.orchestra.run.smithi088.stdout:00000070 00 00 03 02 38 00 00 00 00 00 00 00 00 00 00 00 |....8...........| 2024-09-16T01:40:11.981 INFO:teuthology.orchestra.run.smithi088.stdout:00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-16T01:40:11.981 INFO:teuthology.orchestra.run.smithi088.stdout:* 2024-09-16T01:40:11.981 INFO:teuthology.orchestra.run.smithi088.stdout:000000b0 03 02 38 00 00 00 00 00 00 00 00 00 00 00 00 00 |..8.............| 2024-09-16T01:40:11.981 INFO:teuthology.orchestra.run.smithi088.stdout:000000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-16T01:40:11.981 INFO:teuthology.orchestra.run.smithi088.stdout:* 2024-09-16T01:40:11.981 INFO:teuthology.orchestra.run.smithi088.stdout:00000110 00 00 |..| 2024-09-16T01:40:11.981 INFO:teuthology.orchestra.run.smithi088.stdout:00000112 2024-09-16T01:40:11.981 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:11.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.975+0000 7f40b2b25f40 1 -- 172.21.15.88:0/3639146693 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x558843158d70 msgr2=0x558843280400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.975+0000 7f40b2b25f40 1 --2- 172.21.15.88:0/3639146693 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x558843158d70 0x558843280400 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.975+0000 7f40b2b25f40 1 -- 172.21.15.88:0/3639146693 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f409006ac90 msgr2=0x7f409006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.975+0000 7f40b2b25f40 1 --2- 172.21.15.88:0/3639146693 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f409006ac90 0x7f409006d150 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f40a8005400 tx=0x7f40a800c040 comp rx=0 tx=0).stop 2024-09-16T01:40:11.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.975+0000 7f40b2b25f40 1 -- 172.21.15.88:0/3639146693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5588431e2540 msgr2=0x5588431d1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:11.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.975+0000 7f40b2b25f40 1 --2- 172.21.15.88:0/3639146693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5588431e2540 0x5588431d1a00 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f4098000c00 tx=0x7f409800e6a0 comp rx=0 tx=0).stop 2024-09-16T01:40:11.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.975+0000 7f40b0504640 1 -- 172.21.15.88:0/3639146693 reap_dead start 2024-09-16T01:40:11.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.975+0000 7f40b2b25f40 1 -- 172.21.15.88:0/3639146693 shutdown_connections 2024-09-16T01:40:11.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.975+0000 7f40b2b25f40 1 -- 172.21.15.88:0/3639146693 >> 172.21.15.88:0/3639146693 conn(0x558843158750 msgr2=0x5588431e2d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:11.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.975+0000 7f40b2b25f40 1 -- 172.21.15.88:0/3639146693 shutdown_connections 2024-09-16T01:40:11.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:11.975+0000 7f40b2b25f40 1 -- 172.21.15.88:0/3639146693 wait complete. 2024-09-16T01:40:11.983 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 1.00000000.inode 2024-09-16T01:40:12.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.064+0000 7fc28176ef40 1 Processor -- start 2024-09-16T01:40:12.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.064+0000 7fc28176ef40 1 -- start start 2024-09-16T01:40:12.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.065+0000 7fc28176ef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55709cfb4940 0x55709d083ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.065+0000 7fc28176ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55709cfb4d20 0x55709d081690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.065+0000 7fc28176ef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55709d088f30 0x55709cfdef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.065+0000 7fc28176ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55709cfb52c0 con 0x55709cfb4d20 2024-09-16T01:40:12.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.065+0000 7fc28176ef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55709cf05510 con 0x55709d088f30 2024-09-16T01:40:12.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.065+0000 7fc28176ef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55709cfb2c80 con 0x55709cfb4940 2024-09-16T01:40:12.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.065+0000 7fc27ea63640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55709cfb4d20 0x55709d081690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.065+0000 7fc27ea63640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55709cfb4d20 0x55709d081690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46128/0 (socket says 172.21.15.88:46128) 2024-09-16T01:40:12.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.065+0000 7fc27ea63640 1 -- 172.21.15.88:0/1072322936 learned_addr learned my addr 172.21.15.88:0/1072322936 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:12.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.065+0000 7fc27f264640 1 --2- 172.21.15.88:0/1072322936 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55709cfb4940 0x55709d083ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.065+0000 7fc27fa65640 1 --2- 172.21.15.88:0/1072322936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55709d088f30 0x55709cfdef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.065+0000 7fc27ea63640 1 -- 172.21.15.88:0/1072322936 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55709cfb4940 msgr2=0x55709d083ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:12.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.065+0000 7fc27ea63640 1 --2- 172.21.15.88:0/1072322936 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55709cfb4940 0x55709d083ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.065+0000 7fc27ea63640 1 -- 172.21.15.88:0/1072322936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55709d088f30 msgr2=0x55709cfdef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:12.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.065+0000 7fc27ea63640 1 --2- 172.21.15.88:0/1072322936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55709d088f30 0x55709cfdef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.065+0000 7fc27ea63640 1 -- 172.21.15.88:0/1072322936 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55709cfe3780 con 0x55709cfb4d20 2024-09-16T01:40:12.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.065+0000 7fc27fa65640 1 --2- 172.21.15.88:0/1072322936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55709d088f30 0x55709cfdef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:12.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.066+0000 7fc27ea63640 1 --2- 172.21.15.88:0/1072322936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55709cfb4d20 0x55709d081690 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fc26800a7a0 tx=0x7fc26800a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1640954dfb126b06 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.066+0000 7fc27da61640 1 -- 172.21.15.88:0/1072322936 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc268019070 con 0x55709cfb4d20 2024-09-16T01:40:12.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.066+0000 7fc27da61640 1 -- 172.21.15.88:0/1072322936 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc268007600 con 0x55709cfb4d20 2024-09-16T01:40:12.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.066+0000 7fc27da61640 1 -- 172.21.15.88:0/1072322936 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2680079d0 con 0x55709cfb4d20 2024-09-16T01:40:12.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.066+0000 7fc28176ef40 1 -- 172.21.15.88:0/1072322936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55709cfb4d20 msgr2=0x55709d081690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.066+0000 7fc28176ef40 1 --2- 172.21.15.88:0/1072322936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55709cfb4d20 0x55709d081690 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fc26800a7a0 tx=0x7fc26800a7d0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.066+0000 7fc28176ef40 1 -- 172.21.15.88:0/1072322936 shutdown_connections 2024-09-16T01:40:12.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.066+0000 7fc28176ef40 1 --2- 172.21.15.88:0/1072322936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55709d088f30 0x55709cfdef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.066+0000 7fc28176ef40 1 --2- 172.21.15.88:0/1072322936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55709cfb4d20 0x55709d081690 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.066+0000 7fc28176ef40 1 --2- 172.21.15.88:0/1072322936 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55709cfb4940 0x55709d083ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.066+0000 7fc28176ef40 1 -- 172.21.15.88:0/1072322936 >> 172.21.15.88:0/1072322936 conn(0x55709cffe750 msgr2=0x55709cffeb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:12.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.066+0000 7fc28176ef40 1 -- 172.21.15.88:0/1072322936 shutdown_connections 2024-09-16T01:40:12.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.066+0000 7fc28176ef40 1 -- 172.21.15.88:0/1072322936 wait complete. 2024-09-16T01:40:12.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.067+0000 7fc28176ef40 1 Processor -- start 2024-09-16T01:40:12.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.067+0000 7fc28176ef40 1 -- start start 2024-09-16T01:40:12.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.068+0000 7fc28176ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55709cfb4940 0x55709cfdf530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.068+0000 7fc28176ef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55709cfb4d20 0x55709d0872f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.068+0000 7fc28176ef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55709d088f30 0x55709d087830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.068+0000 7fc28176ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55709d0886c0 con 0x55709cfb4940 2024-09-16T01:40:12.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.068+0000 7fc28176ef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55709d088400 con 0x55709d088f30 2024-09-16T01:40:12.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.068+0000 7fc28176ef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55709cfde410 con 0x55709cfb4d20 2024-09-16T01:40:12.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.068+0000 7fc27f264640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55709cfb4940 0x55709cfdf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.068+0000 7fc27f264640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55709cfb4940 0x55709cfdf530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46136/0 (socket says 172.21.15.88:46136) 2024-09-16T01:40:12.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.068+0000 7fc27f264640 1 -- 172.21.15.88:0/4289278137 learned_addr learned my addr 172.21.15.88:0/4289278137 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:12.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.068+0000 7fc27fa65640 1 --2- 172.21.15.88:0/4289278137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55709d088f30 0x55709d087830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.068+0000 7fc27ea63640 1 --2- 172.21.15.88:0/4289278137 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55709cfb4d20 0x55709d0872f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.068+0000 7fc27f264640 1 -- 172.21.15.88:0/4289278137 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55709cfb4d20 msgr2=0x55709d0872f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.068+0000 7fc27f264640 1 --2- 172.21.15.88:0/4289278137 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55709cfb4d20 0x55709d0872f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.068+0000 7fc27f264640 1 -- 172.21.15.88:0/4289278137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55709d088f30 msgr2=0x55709d087830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.068+0000 7fc27f264640 1 --2- 172.21.15.88:0/4289278137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55709d088f30 0x55709d087830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.068+0000 7fc27f264640 1 -- 172.21.15.88:0/4289278137 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55709d07a120 con 0x55709cfb4940 2024-09-16T01:40:12.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.068+0000 7fc27ea63640 1 --2- 172.21.15.88:0/4289278137 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55709cfb4d20 0x55709d0872f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:12.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.068+0000 7fc27f264640 1 --2- 172.21.15.88:0/4289278137 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55709cfb4940 0x55709cfdf530 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fc278046050 tx=0x7fc278002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.068+0000 7fc27ca5f640 1 -- 172.21.15.88:0/4289278137 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc278055070 con 0x55709cfb4940 2024-09-16T01:40:12.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.068+0000 7fc27ca5f640 1 -- 172.21.15.88:0/4289278137 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc278051820 con 0x55709cfb4940 2024-09-16T01:40:12.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.068+0000 7fc28176ef40 1 -- 172.21.15.88:0/4289278137 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55709d07ce90 con 0x55709cfb4940 2024-09-16T01:40:12.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.068+0000 7fc27ca5f640 1 -- 172.21.15.88:0/4289278137 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc278051bd0 con 0x55709cfb4940 2024-09-16T01:40:12.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.068+0000 7fc28176ef40 1 -- 172.21.15.88:0/4289278137 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55709d07d450 con 0x55709cfb4940 2024-09-16T01:40:12.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.070+0000 7fc27ca5f640 1 -- 172.21.15.88:0/4289278137 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc278058070 con 0x55709cfb4940 2024-09-16T01:40:12.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.070+0000 7fc27ca5f640 1 --2- 172.21.15.88:0/4289278137 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc26006ac90 0x7fc26006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.070+0000 7fc27ca5f640 1 -- 172.21.15.88:0/4289278137 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7fc2780cdb40 con 0x55709cfb4940 2024-09-16T01:40:12.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.070+0000 7fc27ea63640 1 --2- 172.21.15.88:0/4289278137 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc26006ac90 0x7fc26006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.070+0000 7fc28176ef40 1 --2- 172.21.15.88:0/4289278137 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55709d07e9e0 0x55709d07edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.070+0000 7fc28176ef40 1 -- 172.21.15.88:0/4289278137 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:1 12.14 12:287034c4:::1.00000000.inode:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55709d07f300 con 0x55709d07e9e0 2024-09-16T01:40:12.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.071+0000 7fc27fa65640 1 --2- 172.21.15.88:0/4289278137 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55709d07e9e0 0x55709d07edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.071+0000 7fc27ea63640 1 --2- 172.21.15.88:0/4289278137 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc26006ac90 0x7fc26006d150 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x55709cfdeb50 tx=0x7fc26800c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.071+0000 7fc27fa65640 1 --2- 172.21.15.88:0/4289278137 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55709d07e9e0 0x55709d07edc0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.071+0000 7fc27fa65640 1 -- 172.21.15.88:0/4289278137 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(1 1.00000000.inode [omap-get-header] v0'0 uv16 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc270002650 con 0x55709d07e9e0 2024-09-16T01:40:12.077 INFO:teuthology.orchestra.run.smithi088.stdout:header (0 bytes) : 2024-09-16T01:40:12.077 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:12.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.071+0000 7fc28176ef40 1 -- 172.21.15.88:0/4289278137 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55709d07e9e0 msgr2=0x55709d07edc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.071+0000 7fc28176ef40 1 --2- 172.21.15.88:0/4289278137 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55709d07e9e0 0x55709d07edc0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.071+0000 7fc28176ef40 1 -- 172.21.15.88:0/4289278137 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc26006ac90 msgr2=0x7fc26006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.071+0000 7fc28176ef40 1 --2- 172.21.15.88:0/4289278137 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc26006ac90 0x7fc26006d150 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x55709cfdeb50 tx=0x7fc26800c000 comp rx=0 tx=0).stop 2024-09-16T01:40:12.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.071+0000 7fc28176ef40 1 -- 172.21.15.88:0/4289278137 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55709cfb4940 msgr2=0x55709cfdf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.071+0000 7fc28176ef40 1 --2- 172.21.15.88:0/4289278137 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55709cfb4940 0x55709cfdf530 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fc278046050 tx=0x7fc278002fb0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.072+0000 7fc27fa65640 1 -- 172.21.15.88:0/4289278137 reap_dead start 2024-09-16T01:40:12.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.072+0000 7fc28176ef40 1 -- 172.21.15.88:0/4289278137 shutdown_connections 2024-09-16T01:40:12.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.072+0000 7fc28176ef40 1 -- 172.21.15.88:0/4289278137 >> 172.21.15.88:0/4289278137 conn(0x55709cffe750 msgr2=0x55709cffeb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:12.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.072+0000 7fc28176ef40 1 -- 172.21.15.88:0/4289278137 shutdown_connections 2024-09-16T01:40:12.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.072+0000 7fc28176ef40 1 -- 172.21.15.88:0/4289278137 wait complete. 2024-09-16T01:40:12.080 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 200.00000000 2024-09-16T01:40:12.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.160+0000 7fda987dcf40 1 Processor -- start 2024-09-16T01:40:12.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.160+0000 7fda987dcf40 1 -- start start 2024-09-16T01:40:12.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.161+0000 7fda987dcf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cacd8b4940 0x55cacd980c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.161+0000 7fda987dcf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacd8b4d20 0x55cacda17900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.161+0000 7fda987dcf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55cacd988f30 0x55cacd8e2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.161+0000 7fda987dcf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55cacd8b52c0 con 0x55cacd8b4940 2024-09-16T01:40:12.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.161+0000 7fda987dcf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cacd805510 con 0x55cacd8b4d20 2024-09-16T01:40:12.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.161+0000 7fda987dcf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55cacd8b2c80 con 0x55cacd988f30 2024-09-16T01:40:12.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.161+0000 7fda962d4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cacd8b4940 0x55cacd980c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.161+0000 7fda962d4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cacd8b4940 0x55cacd980c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46152/0 (socket says 172.21.15.88:46152) 2024-09-16T01:40:12.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.161+0000 7fda962d4640 1 -- 172.21.15.88:0/3660886314 learned_addr learned my addr 172.21.15.88:0/3660886314 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:12.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.161+0000 7fda8ffff640 1 --2- 172.21.15.88:0/3660886314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacd8b4d20 0x55cacda17900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.161+0000 7fda96ad5640 1 --2- 172.21.15.88:0/3660886314 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55cacd988f30 0x55cacd8e2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.161+0000 7fda962d4640 1 -- 172.21.15.88:0/3660886314 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55cacd988f30 msgr2=0x55cacd8e2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:12.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.161+0000 7fda962d4640 1 --2- 172.21.15.88:0/3660886314 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55cacd988f30 0x55cacd8e2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.161+0000 7fda962d4640 1 -- 172.21.15.88:0/3660886314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacd8b4d20 msgr2=0x55cacda17900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:12.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.161+0000 7fda962d4640 1 --2- 172.21.15.88:0/3660886314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacd8b4d20 0x55cacda17900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.161+0000 7fda962d4640 1 -- 172.21.15.88:0/3660886314 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cacd8f9a10 con 0x55cacd8b4940 2024-09-16T01:40:12.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.161+0000 7fda8ffff640 1 --2- 172.21.15.88:0/3660886314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacd8b4d20 0x55cacda17900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:12.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.161+0000 7fda96ad5640 1 --2- 172.21.15.88:0/3660886314 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55cacd988f30 0x55cacd8e2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:12.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.161+0000 7fda962d4640 1 --2- 172.21.15.88:0/3660886314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cacd8b4940 0x55cacd980c10 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fda88000c00 tx=0x7fda88051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4754bd10fb6934c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.161+0000 7fda952d2640 1 -- 172.21.15.88:0/3660886314 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda880566f0 con 0x55cacd8b4940 2024-09-16T01:40:12.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.161+0000 7fda952d2640 1 -- 172.21.15.88:0/3660886314 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda880568b0 con 0x55cacd8b4940 2024-09-16T01:40:12.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.161+0000 7fda952d2640 1 -- 172.21.15.88:0/3660886314 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda88056ca0 con 0x55cacd8b4940 2024-09-16T01:40:12.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.162+0000 7fda987dcf40 1 -- 172.21.15.88:0/3660886314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cacd8b4940 msgr2=0x55cacd980c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.162+0000 7fda987dcf40 1 --2- 172.21.15.88:0/3660886314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cacd8b4940 0x55cacd980c10 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fda88000c00 tx=0x7fda88051000 comp rx=0 tx=0).stop 2024-09-16T01:40:12.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.162+0000 7fda987dcf40 1 -- 172.21.15.88:0/3660886314 shutdown_connections 2024-09-16T01:40:12.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.162+0000 7fda987dcf40 1 --2- 172.21.15.88:0/3660886314 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55cacd988f30 0x55cacd8e2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.162+0000 7fda987dcf40 1 --2- 172.21.15.88:0/3660886314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacd8b4d20 0x55cacda17900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.162+0000 7fda987dcf40 1 --2- 172.21.15.88:0/3660886314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cacd8b4940 0x55cacd980c10 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.162+0000 7fda987dcf40 1 -- 172.21.15.88:0/3660886314 >> 172.21.15.88:0/3660886314 conn(0x55cacd8fe750 msgr2=0x55cacd8e2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:12.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.162+0000 7fda987dcf40 1 -- 172.21.15.88:0/3660886314 shutdown_connections 2024-09-16T01:40:12.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.162+0000 7fda987dcf40 1 -- 172.21.15.88:0/3660886314 wait complete. 2024-09-16T01:40:12.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.163+0000 7fda987dcf40 1 Processor -- start 2024-09-16T01:40:12.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.163+0000 7fda987dcf40 1 -- start start 2024-09-16T01:40:12.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.163+0000 7fda987dcf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cacd8b4940 0x55cacda17db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.163+0000 7fda987dcf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacd8b4d20 0x55cacd8f6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.163+0000 7fda987dcf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55cacd988f30 0x55cacda21b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.163+0000 7fda987dcf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55cacd9846e0 con 0x55cacd8b4940 2024-09-16T01:40:12.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.163+0000 7fda987dcf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cacd8f93e0 con 0x55cacd8b4d20 2024-09-16T01:40:12.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.163+0000 7fda987dcf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55cacd9815f0 con 0x55cacd988f30 2024-09-16T01:40:12.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.163+0000 7fda962d4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cacd8b4940 0x55cacda17db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.163+0000 7fda962d4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cacd8b4940 0x55cacda17db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46158/0 (socket says 172.21.15.88:46158) 2024-09-16T01:40:12.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.163+0000 7fda962d4640 1 -- 172.21.15.88:0/3168134853 learned_addr learned my addr 172.21.15.88:0/3168134853 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:12.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.163+0000 7fda8ffff640 1 --2- 172.21.15.88:0/3168134853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacd8b4d20 0x55cacd8f6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.163+0000 7fda96ad5640 1 --2- 172.21.15.88:0/3168134853 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55cacd988f30 0x55cacda21b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.164+0000 7fda962d4640 1 -- 172.21.15.88:0/3168134853 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55cacd988f30 msgr2=0x55cacda21b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.164+0000 7fda962d4640 1 --2- 172.21.15.88:0/3168134853 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55cacd988f30 0x55cacda21b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.164+0000 7fda962d4640 1 -- 172.21.15.88:0/3168134853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacd8b4d20 msgr2=0x55cacd8f6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.164+0000 7fda962d4640 1 --2- 172.21.15.88:0/3168134853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cacd8b4d20 0x55cacd8f6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.164+0000 7fda962d4640 1 -- 172.21.15.88:0/3168134853 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cacd8f91e0 con 0x55cacd8b4940 2024-09-16T01:40:12.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.164+0000 7fda962d4640 1 --2- 172.21.15.88:0/3168134853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cacd8b4940 0x55cacda17db0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fda88025b90 tx=0x7fda88002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.164+0000 7fda8f7fe640 1 -- 172.21.15.88:0/3168134853 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda8806b070 con 0x55cacd8b4940 2024-09-16T01:40:12.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.164+0000 7fda8f7fe640 1 -- 172.21.15.88:0/3168134853 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda88056c60 con 0x55cacd8b4940 2024-09-16T01:40:12.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.164+0000 7fda987dcf40 1 -- 172.21.15.88:0/3168134853 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55cacd8f8140 con 0x55cacd8b4940 2024-09-16T01:40:12.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.164+0000 7fda8f7fe640 1 -- 172.21.15.88:0/3168134853 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda88056270 con 0x55cacd8b4940 2024-09-16T01:40:12.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.164+0000 7fda987dcf40 1 -- 172.21.15.88:0/3168134853 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cacd8f8700 con 0x55cacd8b4940 2024-09-16T01:40:12.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.165+0000 7fda8f7fe640 1 -- 172.21.15.88:0/3168134853 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fda88054440 con 0x55cacd8b4940 2024-09-16T01:40:12.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.166+0000 7fda8f7fe640 1 --2- 172.21.15.88:0/3168134853 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fda7806ac90 0x7fda7806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.166+0000 7fda8f7fe640 1 -- 172.21.15.88:0/3168134853 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7fda880cf510 con 0x55cacd8b4940 2024-09-16T01:40:12.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.166+0000 7fda8ffff640 1 --2- 172.21.15.88:0/3168134853 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fda7806ac90 0x7fda7806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.166+0000 7fda987dcf40 1 --2- 172.21.15.88:0/3168134853 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55cacd984ce0 0x55cacda22050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.166+0000 7fda987dcf40 1 -- 172.21.15.88:0/3168134853 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:1 12.14 12:292cf221:::200.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55cacd8b6ba0 con 0x55cacd984ce0 2024-09-16T01:40:12.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.166+0000 7fda8ffff640 1 --2- 172.21.15.88:0/3168134853 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fda7806ac90 0x7fda7806d150 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x55cacd985a00 tx=0x7fda80005a00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.166+0000 7fda96ad5640 1 --2- 172.21.15.88:0/3168134853 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55cacd984ce0 0x55cacda22050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.167+0000 7fda96ad5640 1 --2- 172.21.15.88:0/3168134853 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55cacd984ce0 0x55cacda22050 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.167+0000 7fda96ad5640 1 -- 172.21.15.88:0/3168134853 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(1 200.00000000 [omap-get-header] v0'0 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fda90001920 con 0x55cacd984ce0 2024-09-16T01:40:12.173 INFO:teuthology.orchestra.run.smithi088.stdout:header (0 bytes) : 2024-09-16T01:40:12.173 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:12.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.167+0000 7fda987dcf40 1 -- 172.21.15.88:0/3168134853 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55cacd984ce0 msgr2=0x55cacda22050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.167+0000 7fda987dcf40 1 --2- 172.21.15.88:0/3168134853 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55cacd984ce0 0x55cacda22050 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.167+0000 7fda987dcf40 1 -- 172.21.15.88:0/3168134853 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fda7806ac90 msgr2=0x7fda7806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.167+0000 7fda987dcf40 1 --2- 172.21.15.88:0/3168134853 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fda7806ac90 0x7fda7806d150 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x55cacd985a00 tx=0x7fda80005a00 comp rx=0 tx=0).stop 2024-09-16T01:40:12.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.167+0000 7fda987dcf40 1 -- 172.21.15.88:0/3168134853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cacd8b4940 msgr2=0x55cacda17db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.167+0000 7fda987dcf40 1 --2- 172.21.15.88:0/3168134853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cacd8b4940 0x55cacda17db0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fda88025b90 tx=0x7fda88002fb0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.167+0000 7fda96ad5640 1 -- 172.21.15.88:0/3168134853 reap_dead start 2024-09-16T01:40:12.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.167+0000 7fda987dcf40 1 -- 172.21.15.88:0/3168134853 shutdown_connections 2024-09-16T01:40:12.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.167+0000 7fda987dcf40 1 -- 172.21.15.88:0/3168134853 >> 172.21.15.88:0/3168134853 conn(0x55cacd8fe750 msgr2=0x55cacd8e2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:12.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.167+0000 7fda987dcf40 1 -- 172.21.15.88:0/3168134853 shutdown_connections 2024-09-16T01:40:12.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.167+0000 7fda987dcf40 1 -- 172.21.15.88:0/3168134853 wait complete. 2024-09-16T01:40:12.175 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 200.00000001 2024-09-16T01:40:12.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.256+0000 7f119f229f40 1 Processor -- start 2024-09-16T01:40:12.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.256+0000 7f119f229f40 1 -- start start 2024-09-16T01:40:12.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.257+0000 7f119f229f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c27c5e3940 0x55c27c6b4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.257+0000 7f119f229f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c27c5e3d50 0x55c27c6b6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.257+0000 7f119f229f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c27c6b7f30 0x55c27c6b6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.257+0000 7f119f229f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55c27c5e42c0 con 0x55c27c5e3940 2024-09-16T01:40:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.257+0000 7f119f229f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c27c534510 con 0x55c27c6b7f30 2024-09-16T01:40:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.257+0000 7f119f229f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55c27c5e1c80 con 0x55c27c5e3d50 2024-09-16T01:40:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.257+0000 7f119cd1f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c27c5e3940 0x55c27c6b4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.257+0000 7f119cd1f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c27c5e3940 0x55c27c6b4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46164/0 (socket says 172.21.15.88:46164) 2024-09-16T01:40:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.257+0000 7f119cd1f640 1 -- 172.21.15.88:0/2929603327 learned_addr learned my addr 172.21.15.88:0/2929603327 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.257+0000 7f1197fff640 1 --2- 172.21.15.88:0/2929603327 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c27c5e3d50 0x55c27c6b6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.257+0000 7f119d520640 1 --2- 172.21.15.88:0/2929603327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c27c6b7f30 0x55c27c6b6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.257+0000 7f119cd1f640 1 -- 172.21.15.88:0/2929603327 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c27c5e3d50 msgr2=0x55c27c6b6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.257+0000 7f119cd1f640 1 --2- 172.21.15.88:0/2929603327 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c27c5e3d50 0x55c27c6b6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.257+0000 7f119cd1f640 1 -- 172.21.15.88:0/2929603327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c27c6b7f30 msgr2=0x55c27c6b6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.257+0000 7f119cd1f640 1 --2- 172.21.15.88:0/2929603327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c27c6b7f30 0x55c27c6b6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.257+0000 7f119cd1f640 1 -- 172.21.15.88:0/2929603327 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c27c6b5070 con 0x55c27c5e3940 2024-09-16T01:40:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.257+0000 7f119cd1f640 1 --2- 172.21.15.88:0/2929603327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c27c5e3940 0x55c27c6b4b30 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f1190001ce0 tx=0x7f119000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f55aee66f44b6c95 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.257+0000 7f1196ffd640 1 -- 172.21.15.88:0/2929603327 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f119000bd80 con 0x55c27c5e3940 2024-09-16T01:40:12.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.257+0000 7f1196ffd640 1 -- 172.21.15.88:0/2929603327 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f119000f400 con 0x55c27c5e3940 2024-09-16T01:40:12.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.257+0000 7f1196ffd640 1 -- 172.21.15.88:0/2929603327 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f119000f7f0 con 0x55c27c5e3940 2024-09-16T01:40:12.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.258+0000 7f119f229f40 1 -- 172.21.15.88:0/2929603327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c27c5e3940 msgr2=0x55c27c6b4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.258+0000 7f119f229f40 1 --2- 172.21.15.88:0/2929603327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c27c5e3940 0x55c27c6b4b30 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f1190001ce0 tx=0x7f119000b5b0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.258+0000 7f119f229f40 1 -- 172.21.15.88:0/2929603327 shutdown_connections 2024-09-16T01:40:12.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.258+0000 7f119f229f40 1 --2- 172.21.15.88:0/2929603327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c27c6b7f30 0x55c27c6b6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.258+0000 7f119f229f40 1 --2- 172.21.15.88:0/2929603327 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c27c5e3d50 0x55c27c6b6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.258+0000 7f119f229f40 1 --2- 172.21.15.88:0/2929603327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c27c5e3940 0x55c27c6b4b30 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.258+0000 7f119f229f40 1 -- 172.21.15.88:0/2929603327 >> 172.21.15.88:0/2929603327 conn(0x55c27c62d750 msgr2=0x55c27c6b5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:12.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.258+0000 7f119f229f40 1 -- 172.21.15.88:0/2929603327 shutdown_connections 2024-09-16T01:40:12.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.258+0000 7f119f229f40 1 -- 172.21.15.88:0/2929603327 wait complete. 2024-09-16T01:40:12.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.258+0000 7f119f229f40 1 Processor -- start 2024-09-16T01:40:12.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.258+0000 7f119f229f40 1 -- start start 2024-09-16T01:40:12.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.259+0000 7f119f229f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c27c5e3940 0x55c27c7501d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.259+0000 7f119f229f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c27c5e3d50 0x55c27c5e5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.259+0000 7f119f229f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c27c6b7f30 0x55c27c5e60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.259+0000 7f119f229f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55c27c6bb490 con 0x55c27c5e3d50 2024-09-16T01:40:12.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.259+0000 7f119f229f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c27c6bb1b0 con 0x55c27c5e3940 2024-09-16T01:40:12.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.259+0000 7f119f229f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55c27c62a130 con 0x55c27c6b7f30 2024-09-16T01:40:12.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.259+0000 7f1197fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c27c5e3d50 0x55c27c5e5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.259+0000 7f1197fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c27c5e3d50 0x55c27c5e5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46166/0 (socket says 172.21.15.88:46166) 2024-09-16T01:40:12.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.259+0000 7f1197fff640 1 -- 172.21.15.88:0/2379886208 learned_addr learned my addr 172.21.15.88:0/2379886208 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:12.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.259+0000 7f119cd1f640 1 --2- 172.21.15.88:0/2379886208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c27c5e3940 0x55c27c7501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.259+0000 7f119d520640 1 --2- 172.21.15.88:0/2379886208 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c27c6b7f30 0x55c27c5e60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.260+0000 7f1197fff640 1 -- 172.21.15.88:0/2379886208 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c27c6b7f30 msgr2=0x55c27c5e60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.260+0000 7f1197fff640 1 --2- 172.21.15.88:0/2379886208 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c27c6b7f30 0x55c27c5e60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.260+0000 7f1197fff640 1 -- 172.21.15.88:0/2379886208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c27c5e3940 msgr2=0x55c27c7501d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.260+0000 7f1197fff640 1 --2- 172.21.15.88:0/2379886208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c27c5e3940 0x55c27c7501d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.260+0000 7f1197fff640 1 -- 172.21.15.88:0/2379886208 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c27c62af10 con 0x55c27c5e3d50 2024-09-16T01:40:12.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.260+0000 7f1197fff640 1 --2- 172.21.15.88:0/2379886208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c27c5e3d50 0x55c27c5e5ba0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f1188005400 tx=0x7f11880087d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.260+0000 7f1195ffb640 1 -- 172.21.15.88:0/2379886208 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11880148a0 con 0x55c27c5e3d50 2024-09-16T01:40:12.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.260+0000 7f1195ffb640 1 -- 172.21.15.88:0/2379886208 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f118801a070 con 0x55c27c5e3d50 2024-09-16T01:40:12.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.260+0000 7f1195ffb640 1 -- 172.21.15.88:0/2379886208 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1188014c60 con 0x55c27c5e3d50 2024-09-16T01:40:12.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.260+0000 7f119f229f40 1 -- 172.21.15.88:0/2379886208 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c27c62b1f0 con 0x55c27c5e3d50 2024-09-16T01:40:12.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.260+0000 7f119f229f40 1 -- 172.21.15.88:0/2379886208 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c27c62b7b0 con 0x55c27c5e3d50 2024-09-16T01:40:12.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.261+0000 7f1195ffb640 1 -- 172.21.15.88:0/2379886208 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1188015470 con 0x55c27c5e3d50 2024-09-16T01:40:12.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.262+0000 7f1195ffb640 1 --2- 172.21.15.88:0/2379886208 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f118006ac90 0x7f118006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.262+0000 7f119cd1f640 1 --2- 172.21.15.88:0/2379886208 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f118006ac90 0x7f118006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.262+0000 7f1195ffb640 1 -- 172.21.15.88:0/2379886208 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f118808a9d0 con 0x55c27c5e3d50 2024-09-16T01:40:12.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.262+0000 7f119f229f40 1 --2- 172.21.15.88:0/2379886208 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55c27c6b9ec0 0x55c27c6ba2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.262+0000 7f119f229f40 1 -- 172.21.15.88:0/2379886208 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:1 12.14 12:2e2fa760:::200.00000001:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55c27c6ba7e0 con 0x55c27c6b9ec0 2024-09-16T01:40:12.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.262+0000 7f119cd1f640 1 --2- 172.21.15.88:0/2379886208 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f118006ac90 0x7f118006d150 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1190003040 tx=0x7f1190009000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.262+0000 7f119d520640 1 --2- 172.21.15.88:0/2379886208 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55c27c6b9ec0 0x55c27c6ba2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.263+0000 7f119d520640 1 --2- 172.21.15.88:0/2379886208 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55c27c6b9ec0 0x55c27c6ba2a0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.263+0000 7f119d520640 1 -- 172.21.15.88:0/2379886208 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(1 200.00000001 [omap-get-header] v0'0 uv17 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f119804f990 con 0x55c27c6b9ec0 2024-09-16T01:40:12.268 INFO:teuthology.orchestra.run.smithi088.stdout:header (0 bytes) : 2024-09-16T01:40:12.268 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:12.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.263+0000 7f119f229f40 1 -- 172.21.15.88:0/2379886208 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55c27c6b9ec0 msgr2=0x55c27c6ba2a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.263+0000 7f119f229f40 1 --2- 172.21.15.88:0/2379886208 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55c27c6b9ec0 0x55c27c6ba2a0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.263+0000 7f119f229f40 1 -- 172.21.15.88:0/2379886208 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f118006ac90 msgr2=0x7f118006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.263+0000 7f119f229f40 1 --2- 172.21.15.88:0/2379886208 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f118006ac90 0x7f118006d150 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1190003040 tx=0x7f1190009000 comp rx=0 tx=0).stop 2024-09-16T01:40:12.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.263+0000 7f119f229f40 1 -- 172.21.15.88:0/2379886208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c27c5e3d50 msgr2=0x55c27c5e5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.263+0000 7f119f229f40 1 --2- 172.21.15.88:0/2379886208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c27c5e3d50 0x55c27c5e5ba0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f1188005400 tx=0x7f11880087d0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.263+0000 7f119d520640 1 -- 172.21.15.88:0/2379886208 reap_dead start 2024-09-16T01:40:12.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.263+0000 7f119f229f40 1 -- 172.21.15.88:0/2379886208 shutdown_connections 2024-09-16T01:40:12.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.263+0000 7f119f229f40 1 -- 172.21.15.88:0/2379886208 >> 172.21.15.88:0/2379886208 conn(0x55c27c62d750 msgr2=0x55c27c628e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:12.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.263+0000 7f119f229f40 1 -- 172.21.15.88:0/2379886208 shutdown_connections 2024-09-16T01:40:12.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.263+0000 7f119f229f40 1 -- 172.21.15.88:0/2379886208 wait complete. 2024-09-16T01:40:12.271 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader mds0_openfiles.0 2024-09-16T01:40:12.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.352+0000 7fc7dc4eff40 1 Processor -- start 2024-09-16T01:40:12.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.352+0000 7fc7dc4eff40 1 -- start start 2024-09-16T01:40:12.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.352+0000 7fc7dc4eff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f5ae7b650 0x563f5ae825d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.352+0000 7fc7dc4eff40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563f5addf500 0x563f5ae82b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.352+0000 7fc7dc4eff40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563f5addfcc0 0x563f5adb2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.352+0000 7fc7dc4eff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x563f5ad05510 con 0x563f5ae7b650 2024-09-16T01:40:12.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.352+0000 7fc7dc4eff40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563f5adfbbd0 con 0x563f5addfcc0 2024-09-16T01:40:12.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.352+0000 7fc7dc4eff40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x563f5ae88a30 con 0x563f5addf500 2024-09-16T01:40:12.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.352+0000 7fc7d9fe5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f5ae7b650 0x563f5ae825d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.352+0000 7fc7d9fe5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f5ae7b650 0x563f5ae825d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46180/0 (socket says 172.21.15.88:46180) 2024-09-16T01:40:12.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.352+0000 7fc7d9fe5640 1 -- 172.21.15.88:0/917815295 learned_addr learned my addr 172.21.15.88:0/917815295 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:12.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.353+0000 7fc7d9fe5640 1 -- 172.21.15.88:0/917815295 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563f5addf500 msgr2=0x563f5ae82b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:12.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.353+0000 7fc7d9fe5640 1 --2- 172.21.15.88:0/917815295 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563f5addf500 0x563f5ae82b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.353+0000 7fc7d9fe5640 1 -- 172.21.15.88:0/917815295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563f5addfcc0 msgr2=0x563f5adb2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:12.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.353+0000 7fc7da7e6640 1 --2- 172.21.15.88:0/917815295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563f5addfcc0 0x563f5adb2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.353+0000 7fc7d9fe5640 1 --2- 172.21.15.88:0/917815295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563f5addfcc0 0x563f5adb2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.353+0000 7fc7d9fe5640 1 -- 172.21.15.88:0/917815295 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f5ae88750 con 0x563f5ae7b650 2024-09-16T01:40:12.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.353+0000 7fc7d9fe5640 1 --2- 172.21.15.88:0/917815295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f5ae7b650 0x563f5ae825d0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc009760 tx=0x7fc7cc005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9daabe33f2b2460 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.353+0000 7fc7d8fe3640 1 -- 172.21.15.88:0/917815295 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc008930 con 0x563f5ae7b650 2024-09-16T01:40:12.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.353+0000 7fc7d8fe3640 1 -- 172.21.15.88:0/917815295 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7cc01a070 con 0x563f5ae7b650 2024-09-16T01:40:12.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.353+0000 7fc7d8fe3640 1 -- 172.21.15.88:0/917815295 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc008cf0 con 0x563f5ae7b650 2024-09-16T01:40:12.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.353+0000 7fc7dc4eff40 1 -- 172.21.15.88:0/917815295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f5ae7b650 msgr2=0x563f5ae825d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.353+0000 7fc7dc4eff40 1 --2- 172.21.15.88:0/917815295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f5ae7b650 0x563f5ae825d0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc009760 tx=0x7fc7cc005a00 comp rx=0 tx=0).stop 2024-09-16T01:40:12.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.353+0000 7fc7dc4eff40 1 -- 172.21.15.88:0/917815295 shutdown_connections 2024-09-16T01:40:12.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.353+0000 7fc7dc4eff40 1 --2- 172.21.15.88:0/917815295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563f5addfcc0 0x563f5adb2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.353+0000 7fc7dc4eff40 1 --2- 172.21.15.88:0/917815295 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563f5addf500 0x563f5ae82b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.353+0000 7fc7dc4eff40 1 --2- 172.21.15.88:0/917815295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f5ae7b650 0x563f5ae825d0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.353+0000 7fc7dc4eff40 1 -- 172.21.15.88:0/917815295 >> 172.21.15.88:0/917815295 conn(0x563f5adfe750 msgr2=0x563f5ae8ae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:12.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.353+0000 7fc7dc4eff40 1 -- 172.21.15.88:0/917815295 shutdown_connections 2024-09-16T01:40:12.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.354+0000 7fc7dc4eff40 1 -- 172.21.15.88:0/917815295 wait complete. 2024-09-16T01:40:12.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.354+0000 7fc7dc4eff40 1 Processor -- start 2024-09-16T01:40:12.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.354+0000 7fc7dc4eff40 1 -- start start 2024-09-16T01:40:12.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.355+0000 7fc7dc4eff40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563f5addf500 0x563f5adfd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.355+0000 7fc7dc4eff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f5addfcc0 0x563f5ae79e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.355+0000 7fc7dc4eff40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563f5ae7b650 0x563f5ae76fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.355+0000 7fc7dc4eff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x563f5adb71b0 con 0x563f5addfcc0 2024-09-16T01:40:12.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.355+0000 7fc7dc4eff40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563f5adb6ed0 con 0x563f5addf500 2024-09-16T01:40:12.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.355+0000 7fc7dc4eff40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x563f5adfa5e0 con 0x563f5ae7b650 2024-09-16T01:40:12.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.355+0000 7fc7d3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f5addfcc0 0x563f5ae79e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.355+0000 7fc7d3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f5addfcc0 0x563f5ae79e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46192/0 (socket says 172.21.15.88:46192) 2024-09-16T01:40:12.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.355+0000 7fc7d3fff640 1 -- 172.21.15.88:0/4066381771 learned_addr learned my addr 172.21.15.88:0/4066381771 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:12.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.355+0000 7fc7d9fe5640 1 --2- 172.21.15.88:0/4066381771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563f5addf500 0x563f5adfd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.355+0000 7fc7da7e6640 1 --2- 172.21.15.88:0/4066381771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563f5ae7b650 0x563f5ae76fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.355+0000 7fc7d3fff640 1 -- 172.21.15.88:0/4066381771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563f5ae7b650 msgr2=0x563f5ae76fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.355+0000 7fc7d3fff640 1 --2- 172.21.15.88:0/4066381771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563f5ae7b650 0x563f5ae76fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.355+0000 7fc7d3fff640 1 -- 172.21.15.88:0/4066381771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563f5addf500 msgr2=0x563f5adfd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.355+0000 7fc7d3fff640 1 --2- 172.21.15.88:0/4066381771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563f5addf500 0x563f5adfd2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.355+0000 7fc7d3fff640 1 -- 172.21.15.88:0/4066381771 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f5ae7acd0 con 0x563f5addfcc0 2024-09-16T01:40:12.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.355+0000 7fc7da7e6640 1 --2- 172.21.15.88:0/4066381771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563f5ae7b650 0x563f5ae76fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:12.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.355+0000 7fc7d3fff640 1 --2- 172.21.15.88:0/4066381771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f5addfcc0 0x563f5ae79e30 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4009df0 tx=0x7fc7c4009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.355+0000 7fc7d2ffd640 1 -- 172.21.15.88:0/4066381771 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c4014400 con 0x563f5addfcc0 2024-09-16T01:40:12.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.355+0000 7fc7d2ffd640 1 -- 172.21.15.88:0/4066381771 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7c4019070 con 0x563f5addfcc0 2024-09-16T01:40:12.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.355+0000 7fc7d2ffd640 1 -- 172.21.15.88:0/4066381771 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c40147e0 con 0x563f5addfcc0 2024-09-16T01:40:12.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.355+0000 7fc7dc4eff40 1 -- 172.21.15.88:0/4066381771 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563f5ae7afb0 con 0x563f5addfcc0 2024-09-16T01:40:12.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.355+0000 7fc7dc4eff40 1 -- 172.21.15.88:0/4066381771 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f5ae86ad0 con 0x563f5addfcc0 2024-09-16T01:40:12.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.357+0000 7fc7d2ffd640 1 -- 172.21.15.88:0/4066381771 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc7c4013070 con 0x563f5addfcc0 2024-09-16T01:40:12.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.357+0000 7fc7d2ffd640 1 --2- 172.21.15.88:0/4066381771 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc7bc06ac90 0x7fc7bc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.357+0000 7fc7d2ffd640 1 -- 172.21.15.88:0/4066381771 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7fc7c4089d30 con 0x563f5addfcc0 2024-09-16T01:40:12.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.357+0000 7fc7d9fe5640 1 --2- 172.21.15.88:0/4066381771 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc7bc06ac90 0x7fc7bc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.358+0000 7fc7dc4eff40 1 --2- 172.21.15.88:0/4066381771 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x563f5ae7a370 0x563f5ae85540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.358+0000 7fc7dc4eff40 1 -- 172.21.15.88:0/4066381771 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:1 12.6 12:654134d2:::mds0_openfiles.0:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x563f5ae85a80 con 0x563f5ae7a370 2024-09-16T01:40:12.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.358+0000 7fc7da7e6640 1 --2- 172.21.15.88:0/4066381771 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x563f5ae7a370 0x563f5ae85540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.358+0000 7fc7d9fe5640 1 --2- 172.21.15.88:0/4066381771 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc7bc06ac90 0x7fc7bc06d150 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc017040 tx=0x7fc7cc007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.358+0000 7fc7da7e6640 1 --2- 172.21.15.88:0/4066381771 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x563f5ae7a370 0x563f5ae85540 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.358+0000 7fc7da7e6640 1 -- 172.21.15.88:0/4066381771 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(1 mds0_openfiles.0 [omap-get-header out=42b] v0'0 uv3 ondisk = 0) ==== 160+0+42 (crc 0 0 0) 0x7fc7d40017b0 con 0x563f5ae7a370 2024-09-16T01:40:12.364 INFO:teuthology.orchestra.run.smithi088.stdout:header (42 bytes) : 2024-09-16T01:40:12.364 INFO:teuthology.orchestra.run.smithi088.stdout:00000000 13 00 00 00 63 65 70 68 20 66 73 20 76 6f 6c 75 |....ceph fs volu| 2024-09-16T01:40:12.364 INFO:teuthology.orchestra.run.smithi088.stdout:00000010 6d 65 20 76 30 31 31 01 01 0d 00 00 00 03 00 00 |me v011.........| 2024-09-16T01:40:12.364 INFO:teuthology.orchestra.run.smithi088.stdout:00000020 00 00 00 00 00 01 00 00 00 00 |..........| 2024-09-16T01:40:12.364 INFO:teuthology.orchestra.run.smithi088.stdout:0000002a 2024-09-16T01:40:12.364 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:12.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.358+0000 7fc7dc4eff40 1 -- 172.21.15.88:0/4066381771 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x563f5ae7a370 msgr2=0x563f5ae85540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.358+0000 7fc7dc4eff40 1 --2- 172.21.15.88:0/4066381771 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x563f5ae7a370 0x563f5ae85540 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.358+0000 7fc7dc4eff40 1 -- 172.21.15.88:0/4066381771 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc7bc06ac90 msgr2=0x7fc7bc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.358+0000 7fc7dc4eff40 1 --2- 172.21.15.88:0/4066381771 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc7bc06ac90 0x7fc7bc06d150 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc017040 tx=0x7fc7cc007000 comp rx=0 tx=0).stop 2024-09-16T01:40:12.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.358+0000 7fc7dc4eff40 1 -- 172.21.15.88:0/4066381771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f5addfcc0 msgr2=0x563f5ae79e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.358+0000 7fc7dc4eff40 1 --2- 172.21.15.88:0/4066381771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f5addfcc0 0x563f5ae79e30 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4009df0 tx=0x7fc7c4009ee0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.358+0000 7fc7da7e6640 1 -- 172.21.15.88:0/4066381771 reap_dead start 2024-09-16T01:40:12.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.358+0000 7fc7dc4eff40 1 -- 172.21.15.88:0/4066381771 shutdown_connections 2024-09-16T01:40:12.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.358+0000 7fc7dc4eff40 1 -- 172.21.15.88:0/4066381771 >> 172.21.15.88:0/4066381771 conn(0x563f5adfe750 msgr2=0x563f5adde9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:12.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.359+0000 7fc7dc4eff40 1 -- 172.21.15.88:0/4066381771 shutdown_connections 2024-09-16T01:40:12.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.359+0000 7fc7dc4eff40 1 -- 172.21.15.88:0/4066381771 wait complete. 2024-09-16T01:40:12.367 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 500.00000000 2024-09-16T01:40:12.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.449+0000 7f1eb1d22f40 1 Processor -- start 2024-09-16T01:40:12.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.449+0000 7f1eb1d22f40 1 -- start start 2024-09-16T01:40:12.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.449+0000 7f1eb1d22f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56018a64fdd0 0x56018a64e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.449+0000 7f1eb1d22f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56018a652230 0x56018a716280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.449+0000 7f1eb1d22f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56018a71e540 0x56018a725f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.449+0000 7f1eb1d22f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56018a6995b0 con 0x56018a64fdd0 2024-09-16T01:40:12.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.449+0000 7f1eb1d22f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56018a5a0510 con 0x56018a652230 2024-09-16T01:40:12.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.449+0000 7f1eb1d22f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56018a6501e0 con 0x56018a71e540 2024-09-16T01:40:12.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.449+0000 7f1eaf81a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56018a64fdd0 0x56018a64e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.449+0000 7f1eaf81a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56018a64fdd0 0x56018a64e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46198/0 (socket says 172.21.15.88:46198) 2024-09-16T01:40:12.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.449+0000 7f1eaf81a640 1 -- 172.21.15.88:0/1778098114 learned_addr learned my addr 172.21.15.88:0/1778098114 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:12.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.450+0000 7f1eb001b640 1 --2- 172.21.15.88:0/1778098114 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56018a71e540 0x56018a725f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.450+0000 7f1ea7fff640 1 --2- 172.21.15.88:0/1778098114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56018a652230 0x56018a716280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.450+0000 7f1eaf81a640 1 -- 172.21.15.88:0/1778098114 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56018a71e540 msgr2=0x56018a725f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:12.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.450+0000 7f1eaf81a640 1 --2- 172.21.15.88:0/1778098114 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56018a71e540 0x56018a725f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.450+0000 7f1eaf81a640 1 -- 172.21.15.88:0/1778098114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56018a652230 msgr2=0x56018a716280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:12.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.450+0000 7f1eaf81a640 1 --2- 172.21.15.88:0/1778098114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56018a652230 0x56018a716280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.450+0000 7f1eaf81a640 1 -- 172.21.15.88:0/1778098114 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56018a726ed0 con 0x56018a64fdd0 2024-09-16T01:40:12.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.450+0000 7f1ea7fff640 1 --2- 172.21.15.88:0/1778098114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56018a652230 0x56018a716280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:12.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.450+0000 7f1eaf81a640 1 --2- 172.21.15.88:0/1778098114 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56018a64fdd0 0x56018a64e7c0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f1ea000a460 tx=0x7f1ea000a920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4825b843a8331e9d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.450+0000 7f1eae818640 1 -- 172.21.15.88:0/1778098114 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea000b920 con 0x56018a64fdd0 2024-09-16T01:40:12.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.450+0000 7f1eae818640 1 -- 172.21.15.88:0/1778098114 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ea000f070 con 0x56018a64fdd0 2024-09-16T01:40:12.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.450+0000 7f1eae818640 1 -- 172.21.15.88:0/1778098114 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea000bd20 con 0x56018a64fdd0 2024-09-16T01:40:12.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.450+0000 7f1eb1d22f40 1 -- 172.21.15.88:0/1778098114 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56018a64fdd0 msgr2=0x56018a64e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.450+0000 7f1eb1d22f40 1 --2- 172.21.15.88:0/1778098114 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56018a64fdd0 0x56018a64e7c0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f1ea000a460 tx=0x7f1ea000a920 comp rx=0 tx=0).stop 2024-09-16T01:40:12.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.450+0000 7f1eb1d22f40 1 -- 172.21.15.88:0/1778098114 shutdown_connections 2024-09-16T01:40:12.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.450+0000 7f1eb1d22f40 1 --2- 172.21.15.88:0/1778098114 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56018a71e540 0x56018a725f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.450+0000 7f1eb1d22f40 1 --2- 172.21.15.88:0/1778098114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56018a652230 0x56018a716280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.450+0000 7f1eb1d22f40 1 --2- 172.21.15.88:0/1778098114 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56018a64fdd0 0x56018a64e7c0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.450+0000 7f1eb1d22f40 1 -- 172.21.15.88:0/1778098114 >> 172.21.15.88:0/1778098114 conn(0x56018a699750 msgr2=0x56018a64d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:12.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.451+0000 7f1eb1d22f40 1 -- 172.21.15.88:0/1778098114 shutdown_connections 2024-09-16T01:40:12.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.451+0000 7f1eb1d22f40 1 -- 172.21.15.88:0/1778098114 wait complete. 2024-09-16T01:40:12.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.451+0000 7f1eb1d22f40 1 Processor -- start 2024-09-16T01:40:12.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.451+0000 7f1eb1d22f40 1 -- start start 2024-09-16T01:40:12.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.452+0000 7f1eb1d22f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56018a64fdd0 0x56018a70d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.452+0000 7f1eb1d22f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56018a652230 0x56018a696e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.452+0000 7f1eb1d22f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56018a71e540 0x56018a697370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.452+0000 7f1eb1d22f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56018a727420 con 0x56018a71e540 2024-09-16T01:40:12.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.452+0000 7f1eb1d22f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56018a64e330 con 0x56018a64fdd0 2024-09-16T01:40:12.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.452+0000 7f1eb1d22f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56018a727140 con 0x56018a652230 2024-09-16T01:40:12.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.452+0000 7f1eb001b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56018a71e540 0x56018a697370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.452+0000 7f1eb001b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56018a71e540 0x56018a697370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46200/0 (socket says 172.21.15.88:46200) 2024-09-16T01:40:12.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.452+0000 7f1eb001b640 1 -- 172.21.15.88:0/4271927140 learned_addr learned my addr 172.21.15.88:0/4271927140 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:12.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.452+0000 7f1eaf81a640 1 --2- 172.21.15.88:0/4271927140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56018a64fdd0 0x56018a70d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.452+0000 7f1ea7fff640 1 --2- 172.21.15.88:0/4271927140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56018a652230 0x56018a696e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.452+0000 7f1eb001b640 1 -- 172.21.15.88:0/4271927140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56018a652230 msgr2=0x56018a696e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.452+0000 7f1eb001b640 1 --2- 172.21.15.88:0/4271927140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56018a652230 0x56018a696e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.452+0000 7f1eb001b640 1 -- 172.21.15.88:0/4271927140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56018a64fdd0 msgr2=0x56018a70d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.452+0000 7f1eb001b640 1 --2- 172.21.15.88:0/4271927140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56018a64fdd0 0x56018a70d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.452+0000 7f1eb001b640 1 -- 172.21.15.88:0/4271927140 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56018a7142c0 con 0x56018a71e540 2024-09-16T01:40:12.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.452+0000 7f1eb001b640 1 --2- 172.21.15.88:0/4271927140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56018a71e540 0x56018a697370 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8025b20 tx=0x7f1ea8003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.452+0000 7f1ead816640 1 -- 172.21.15.88:0/4271927140 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea8066020 con 0x56018a71e540 2024-09-16T01:40:12.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.453+0000 7f1ead816640 1 -- 172.21.15.88:0/4271927140 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ea805c040 con 0x56018a71e540 2024-09-16T01:40:12.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.453+0000 7f1ead816640 1 -- 172.21.15.88:0/4271927140 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea805b040 con 0x56018a71e540 2024-09-16T01:40:12.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.453+0000 7f1eb1d22f40 1 -- 172.21.15.88:0/4271927140 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56018a7145a0 con 0x56018a71e540 2024-09-16T01:40:12.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.453+0000 7f1eb1d22f40 1 -- 172.21.15.88:0/4271927140 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56018a714b60 con 0x56018a71e540 2024-09-16T01:40:12.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.454+0000 7f1ead816640 1 -- 172.21.15.88:0/4271927140 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1ea8069020 con 0x56018a71e540 2024-09-16T01:40:12.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.454+0000 7f1ead816640 1 --2- 172.21.15.88:0/4271927140 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1e9006ac90 0x7f1e9006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.454+0000 7f1ead816640 1 -- 172.21.15.88:0/4271927140 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f1ea80ce5c0 con 0x56018a71e540 2024-09-16T01:40:12.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.454+0000 7f1eaf81a640 1 --2- 172.21.15.88:0/4271927140 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1e9006ac90 0x7f1e9006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.455+0000 7f1eb1d22f40 1 --2- 172.21.15.88:0/4271927140 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56018a722f40 0x56018a723320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.455+0000 7f1eb1d22f40 1 -- 172.21.15.88:0/4271927140 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:1 12.15 12:aa448500:::500.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x56018a723860 con 0x56018a722f40 2024-09-16T01:40:12.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.455+0000 7f1eaf81a640 1 --2- 172.21.15.88:0/4271927140 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1e9006ac90 0x7f1e9006d150 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1ea000a460 tx=0x7f1ea0002000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.455+0000 7f1ea7fff640 1 --2- 172.21.15.88:0/4271927140 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56018a722f40 0x56018a723320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.455+0000 7f1ea7fff640 1 --2- 172.21.15.88:0/4271927140 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56018a722f40 0x56018a723320 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.456+0000 7f1ea7fff640 1 -- 172.21.15.88:0/4271927140 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(1 500.00000000 [omap-get-header] v0'0 uv3 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1e980078d0 con 0x56018a722f40 2024-09-16T01:40:12.461 INFO:teuthology.orchestra.run.smithi088.stdout:header (0 bytes) : 2024-09-16T01:40:12.461 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:12.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.456+0000 7f1eb1d22f40 1 -- 172.21.15.88:0/4271927140 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56018a722f40 msgr2=0x56018a723320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.456+0000 7f1eb1d22f40 1 --2- 172.21.15.88:0/4271927140 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56018a722f40 0x56018a723320 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.456+0000 7f1eb1d22f40 1 -- 172.21.15.88:0/4271927140 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1e9006ac90 msgr2=0x7f1e9006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.456+0000 7f1eb1d22f40 1 --2- 172.21.15.88:0/4271927140 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1e9006ac90 0x7f1e9006d150 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1ea000a460 tx=0x7f1ea0002000 comp rx=0 tx=0).stop 2024-09-16T01:40:12.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.456+0000 7f1eb1d22f40 1 -- 172.21.15.88:0/4271927140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56018a71e540 msgr2=0x56018a697370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.456+0000 7f1eb1d22f40 1 --2- 172.21.15.88:0/4271927140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56018a71e540 0x56018a697370 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8025b20 tx=0x7f1ea8003330 comp rx=0 tx=0).stop 2024-09-16T01:40:12.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.456+0000 7f1eb001b640 1 -- 172.21.15.88:0/4271927140 reap_dead start 2024-09-16T01:40:12.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.456+0000 7f1eb1d22f40 1 -- 172.21.15.88:0/4271927140 shutdown_connections 2024-09-16T01:40:12.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.456+0000 7f1eb1d22f40 1 -- 172.21.15.88:0/4271927140 >> 172.21.15.88:0/4271927140 conn(0x56018a699750 msgr2=0x56018a64d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:12.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.456+0000 7f1eb1d22f40 1 -- 172.21.15.88:0/4271927140 shutdown_connections 2024-09-16T01:40:12.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.456+0000 7f1eb1d22f40 1 -- 172.21.15.88:0/4271927140 wait complete. 2024-09-16T01:40:12.464 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader mds_snaptable 2024-09-16T01:40:12.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.545+0000 7f5569185f40 1 Processor -- start 2024-09-16T01:40:12.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.545+0000 7f5569185f40 1 -- start start 2024-09-16T01:40:12.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.546+0000 7f5569185f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a2614aeee0 0x55a26157e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.546+0000 7f5569185f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2614df590 0x55a26157a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.546+0000 7f5569185f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55a261582c90 0x55a261580c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.546+0000 7f5569185f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55a2614af840 con 0x55a2614aeee0 2024-09-16T01:40:12.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.546+0000 7f5569185f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a261400510 con 0x55a2614df590 2024-09-16T01:40:12.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.546+0000 7f5569185f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55a2615836b0 con 0x55a261582c90 2024-09-16T01:40:12.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.546+0000 7f5566c7b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a2614aeee0 0x55a26157e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.546+0000 7f5566c7b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a2614aeee0 0x55a26157e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46206/0 (socket says 172.21.15.88:46206) 2024-09-16T01:40:12.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.546+0000 7f5566c7b640 1 -- 172.21.15.88:0/1645998516 learned_addr learned my addr 172.21.15.88:0/1645998516 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:12.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.546+0000 7f555e47a640 1 --2- 172.21.15.88:0/1645998516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2614df590 0x55a26157a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.546+0000 7f556747c640 1 --2- 172.21.15.88:0/1645998516 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55a261582c90 0x55a261580c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.546+0000 7f5566c7b640 1 -- 172.21.15.88:0/1645998516 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55a261582c90 msgr2=0x55a261580c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:12.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.546+0000 7f5566c7b640 1 --2- 172.21.15.88:0/1645998516 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55a261582c90 0x55a261580c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.546+0000 7f5566c7b640 1 -- 172.21.15.88:0/1645998516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2614df590 msgr2=0x55a26157a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:12.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.546+0000 7f5566c7b640 1 --2- 172.21.15.88:0/1645998516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2614df590 0x55a26157a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.546+0000 7f5566c7b640 1 -- 172.21.15.88:0/1645998516 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2614f8700 con 0x55a2614aeee0 2024-09-16T01:40:12.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.546+0000 7f556747c640 1 --2- 172.21.15.88:0/1645998516 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55a261582c90 0x55a261580c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:12.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.547+0000 7f5566c7b640 1 --2- 172.21.15.88:0/1645998516 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a2614aeee0 0x55a26157e800 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f555800b2c0 tx=0x7f555800b780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54efde0ecbf93b0f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.547+0000 7f5565c79640 1 -- 172.21.15.88:0/1645998516 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5558008070 con 0x55a2614aeee0 2024-09-16T01:40:12.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.547+0000 7f5565c79640 1 -- 172.21.15.88:0/1645998516 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f555800d500 con 0x55a2614aeee0 2024-09-16T01:40:12.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.547+0000 7f5565c79640 1 -- 172.21.15.88:0/1645998516 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555800d8d0 con 0x55a2614aeee0 2024-09-16T01:40:12.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.547+0000 7f5569185f40 1 -- 172.21.15.88:0/1645998516 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a2614aeee0 msgr2=0x55a26157e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.547+0000 7f5569185f40 1 --2- 172.21.15.88:0/1645998516 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a2614aeee0 0x55a26157e800 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f555800b2c0 tx=0x7f555800b780 comp rx=0 tx=0).stop 2024-09-16T01:40:12.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.547+0000 7f5569185f40 1 -- 172.21.15.88:0/1645998516 shutdown_connections 2024-09-16T01:40:12.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.547+0000 7f5569185f40 1 --2- 172.21.15.88:0/1645998516 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55a261582c90 0x55a261580c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.547+0000 7f5569185f40 1 --2- 172.21.15.88:0/1645998516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2614df590 0x55a26157a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.547+0000 7f5569185f40 1 --2- 172.21.15.88:0/1645998516 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a2614aeee0 0x55a26157e800 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.547+0000 7f5569185f40 1 -- 172.21.15.88:0/1645998516 >> 172.21.15.88:0/1645998516 conn(0x55a2614d9680 msgr2=0x55a2614ad940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:12.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.547+0000 7f5569185f40 1 -- 172.21.15.88:0/1645998516 shutdown_connections 2024-09-16T01:40:12.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.547+0000 7f5569185f40 1 -- 172.21.15.88:0/1645998516 wait complete. 2024-09-16T01:40:12.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.548+0000 7f5569185f40 1 Processor -- start 2024-09-16T01:40:12.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.548+0000 7f5569185f40 1 -- start start 2024-09-16T01:40:12.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.549+0000 7f5569185f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a2614aeee0 0x55a26156f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.549+0000 7f5569185f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2614df590 0x55a2614f2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.549+0000 7f5569185f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55a261582c90 0x55a261577010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.549+0000 7f5569185f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55a261575a50 con 0x55a2614aeee0 2024-09-16T01:40:12.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.549+0000 7f5569185f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a2614f8a10 con 0x55a2614df590 2024-09-16T01:40:12.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.549+0000 7f5569185f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55a2614f4e70 con 0x55a261582c90 2024-09-16T01:40:12.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.549+0000 7f5566c7b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a2614aeee0 0x55a26156f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.549+0000 7f5566c7b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a2614aeee0 0x55a26156f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46218/0 (socket says 172.21.15.88:46218) 2024-09-16T01:40:12.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.549+0000 7f5566c7b640 1 -- 172.21.15.88:0/3228775842 learned_addr learned my addr 172.21.15.88:0/3228775842 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:12.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.549+0000 7f555e47a640 1 --2- 172.21.15.88:0/3228775842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2614df590 0x55a2614f2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.549+0000 7f556747c640 1 --2- 172.21.15.88:0/3228775842 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55a261582c90 0x55a261577010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.549+0000 7f5566c7b640 1 -- 172.21.15.88:0/3228775842 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55a261582c90 msgr2=0x55a261577010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.549+0000 7f5566c7b640 1 --2- 172.21.15.88:0/3228775842 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55a261582c90 0x55a261577010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.549+0000 7f5566c7b640 1 -- 172.21.15.88:0/3228775842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2614df590 msgr2=0x55a2614f2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.549+0000 7f5566c7b640 1 --2- 172.21.15.88:0/3228775842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a2614df590 0x55a2614f2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.549+0000 7f5566c7b640 1 -- 172.21.15.88:0/3228775842 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2614f4590 con 0x55a2614aeee0 2024-09-16T01:40:12.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.549+0000 7f556747c640 1 --2- 172.21.15.88:0/3228775842 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55a261582c90 0x55a261577010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:12.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.549+0000 7f5566c7b640 1 --2- 172.21.15.88:0/3228775842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a2614aeee0 0x55a26156f130 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f5558001ed0 tx=0x7f5558000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.549+0000 7f5564c77640 1 -- 172.21.15.88:0/3228775842 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555801c440 con 0x55a2614aeee0 2024-09-16T01:40:12.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.549+0000 7f5564c77640 1 -- 172.21.15.88:0/3228775842 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5558021020 con 0x55a2614aeee0 2024-09-16T01:40:12.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.549+0000 7f5564c77640 1 -- 172.21.15.88:0/3228775842 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555801c600 con 0x55a2614aeee0 2024-09-16T01:40:12.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.549+0000 7f5569185f40 1 -- 172.21.15.88:0/3228775842 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a2614f34f0 con 0x55a2614aeee0 2024-09-16T01:40:12.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.549+0000 7f5569185f40 1 -- 172.21.15.88:0/3228775842 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2614f3ab0 con 0x55a2614aeee0 2024-09-16T01:40:12.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.551+0000 7f5564c77640 1 -- 172.21.15.88:0/3228775842 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5558017070 con 0x55a2614aeee0 2024-09-16T01:40:12.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.551+0000 7f5564c77640 1 --2- 172.21.15.88:0/3228775842 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f554806acc0 0x7f554806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.551+0000 7f5564c77640 1 -- 172.21.15.88:0/3228775842 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f555808bb10 con 0x55a2614aeee0 2024-09-16T01:40:12.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.551+0000 7f555e47a640 1 --2- 172.21.15.88:0/3228775842 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f554806acc0 0x7f554806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.551+0000 7f5569185f40 1 --2- 172.21.15.88:0/3228775842 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55a2614f6e90 0x55a26157ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.551+0000 7f5569185f40 1 -- 172.21.15.88:0/3228775842 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:1 12.d 12:b50e409b:::mds_snaptable:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55a261577f10 con 0x55a2614f6e90 2024-09-16T01:40:12.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.551+0000 7f555e47a640 1 --2- 172.21.15.88:0/3228775842 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f554806acc0 0x7f554806d180 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5550059000 tx=0x7f5550053000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.552+0000 7f556747c640 1 --2- 172.21.15.88:0/3228775842 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55a2614f6e90 0x55a26157ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.552+0000 7f556747c640 1 --2- 172.21.15.88:0/3228775842 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55a2614f6e90 0x55a26157ff10 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.552+0000 7f556747c640 1 -- 172.21.15.88:0/3228775842 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(1 mds_snaptable [omap-get-header] v0'0 uv1 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f5560011040 con 0x55a2614f6e90 2024-09-16T01:40:12.558 INFO:teuthology.orchestra.run.smithi088.stdout:header (0 bytes) : 2024-09-16T01:40:12.558 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:12.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.552+0000 7f5569185f40 1 -- 172.21.15.88:0/3228775842 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55a2614f6e90 msgr2=0x55a26157ff10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.552+0000 7f5569185f40 1 --2- 172.21.15.88:0/3228775842 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55a2614f6e90 0x55a26157ff10 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.552+0000 7f5569185f40 1 -- 172.21.15.88:0/3228775842 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f554806acc0 msgr2=0x7f554806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.552+0000 7f5569185f40 1 --2- 172.21.15.88:0/3228775842 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f554806acc0 0x7f554806d180 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5550059000 tx=0x7f5550053000 comp rx=0 tx=0).stop 2024-09-16T01:40:12.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.552+0000 7f5569185f40 1 -- 172.21.15.88:0/3228775842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a2614aeee0 msgr2=0x55a26156f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.552+0000 7f5569185f40 1 --2- 172.21.15.88:0/3228775842 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a2614aeee0 0x55a26156f130 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f5558001ed0 tx=0x7f5558000ed0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.552+0000 7f556747c640 1 -- 172.21.15.88:0/3228775842 reap_dead start 2024-09-16T01:40:12.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.553+0000 7f5569185f40 1 -- 172.21.15.88:0/3228775842 shutdown_connections 2024-09-16T01:40:12.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.553+0000 7f5569185f40 1 -- 172.21.15.88:0/3228775842 >> 172.21.15.88:0/3228775842 conn(0x55a2614d9680 msgr2=0x55a26157a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:12.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.553+0000 7f5569185f40 1 -- 172.21.15.88:0/3228775842 shutdown_connections 2024-09-16T01:40:12.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.553+0000 7f5569185f40 1 -- 172.21.15.88:0/3228775842 wait complete. 2024-09-16T01:40:12.561 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader mds0_inotable 2024-09-16T01:40:12.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.641+0000 7f197d01af40 1 Processor -- start 2024-09-16T01:40:12.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.641+0000 7f197d01af40 1 -- start start 2024-09-16T01:40:12.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.642+0000 7f197d01af40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601d70cafe0 0x5601d709d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.642+0000 7f197d01af40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601d716fd20 0x5601d70c6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.642+0000 7f197d01af40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601d7170100 0x5601d71639e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.642+0000 7f197d01af40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5601d6fed510 con 0x5601d70cafe0 2024-09-16T01:40:12.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.642+0000 7f197d01af40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5601d7046c50 con 0x5601d716fd20 2024-09-16T01:40:12.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.642+0000 7f197d01af40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5601d7170e50 con 0x5601d7170100 2024-09-16T01:40:12.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.642+0000 7f1979ea3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601d70cafe0 0x5601d709d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.642+0000 7f1979ea3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601d70cafe0 0x5601d709d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46234/0 (socket says 172.21.15.88:46234) 2024-09-16T01:40:12.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.642+0000 7f1979ea3640 1 -- 172.21.15.88:0/4230343788 learned_addr learned my addr 172.21.15.88:0/4230343788 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:12.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.642+0000 7f19716a2640 1 --2- 172.21.15.88:0/4230343788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601d716fd20 0x5601d70c6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.642+0000 7f197a6a4640 1 --2- 172.21.15.88:0/4230343788 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601d7170100 0x5601d71639e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.642+0000 7f1979ea3640 1 -- 172.21.15.88:0/4230343788 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601d7170100 msgr2=0x5601d71639e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:12.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.642+0000 7f1979ea3640 1 --2- 172.21.15.88:0/4230343788 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601d7170100 0x5601d71639e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.642+0000 7f1979ea3640 1 -- 172.21.15.88:0/4230343788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601d716fd20 msgr2=0x5601d70c6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:12.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.642+0000 7f1979ea3640 1 --2- 172.21.15.88:0/4230343788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601d716fd20 0x5601d70c6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.642+0000 7f1979ea3640 1 -- 172.21.15.88:0/4230343788 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601d70c82b0 con 0x5601d70cafe0 2024-09-16T01:40:12.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.642+0000 7f197a6a4640 1 --2- 172.21.15.88:0/4230343788 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601d7170100 0x5601d71639e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:12.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.642+0000 7f1979ea3640 1 --2- 172.21.15.88:0/4230343788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601d70cafe0 0x5601d709d9f0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f196c048200 tx=0x7f196c053000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e5ca7f6b0bd59ca server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.642+0000 7f1978ea1640 1 -- 172.21.15.88:0/4230343788 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c00a3c0 con 0x5601d70cafe0 2024-09-16T01:40:12.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.642+0000 7f1978ea1640 1 -- 172.21.15.88:0/4230343788 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f196c008740 con 0x5601d70cafe0 2024-09-16T01:40:12.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.642+0000 7f1978ea1640 1 -- 172.21.15.88:0/4230343788 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c008b10 con 0x5601d70cafe0 2024-09-16T01:40:12.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.643+0000 7f197d01af40 1 -- 172.21.15.88:0/4230343788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601d70cafe0 msgr2=0x5601d709d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.643+0000 7f197d01af40 1 --2- 172.21.15.88:0/4230343788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601d70cafe0 0x5601d709d9f0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f196c048200 tx=0x7f196c053000 comp rx=0 tx=0).stop 2024-09-16T01:40:12.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.643+0000 7f197d01af40 1 -- 172.21.15.88:0/4230343788 shutdown_connections 2024-09-16T01:40:12.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.643+0000 7f197d01af40 1 --2- 172.21.15.88:0/4230343788 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601d7170100 0x5601d71639e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.643+0000 7f197d01af40 1 --2- 172.21.15.88:0/4230343788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601d716fd20 0x5601d70c6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.643+0000 7f197d01af40 1 --2- 172.21.15.88:0/4230343788 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601d70cafe0 0x5601d709d9f0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.643+0000 7f197d01af40 1 -- 172.21.15.88:0/4230343788 >> 172.21.15.88:0/4230343788 conn(0x5601d70e6750 msgr2=0x5601d70c5e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:12.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.643+0000 7f197d01af40 1 -- 172.21.15.88:0/4230343788 shutdown_connections 2024-09-16T01:40:12.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.643+0000 7f197d01af40 1 -- 172.21.15.88:0/4230343788 wait complete. 2024-09-16T01:40:12.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.643+0000 7f197d01af40 1 Processor -- start 2024-09-16T01:40:12.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.643+0000 7f197d01af40 1 -- start start 2024-09-16T01:40:12.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.644+0000 7f197d01af40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601d70cafe0 0x5601d716b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.644+0000 7f197d01af40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601d716fd20 0x5601d715f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.644+0000 7f197d01af40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601d7170100 0x5601d7161120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.644+0000 7f197d01af40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5601d7164730 con 0x5601d70cafe0 2024-09-16T01:40:12.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.644+0000 7f197d01af40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5601d70c6610 con 0x5601d716fd20 2024-09-16T01:40:12.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.644+0000 7f197d01af40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5601d70c7cf0 con 0x5601d7170100 2024-09-16T01:40:12.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.644+0000 7f1979ea3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601d70cafe0 0x5601d716b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.644+0000 7f1979ea3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601d70cafe0 0x5601d716b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46236/0 (socket says 172.21.15.88:46236) 2024-09-16T01:40:12.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.644+0000 7f1979ea3640 1 -- 172.21.15.88:0/3676954702 learned_addr learned my addr 172.21.15.88:0/3676954702 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:12.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.644+0000 7f19716a2640 1 --2- 172.21.15.88:0/3676954702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601d716fd20 0x5601d715f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.644+0000 7f197a6a4640 1 --2- 172.21.15.88:0/3676954702 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601d7170100 0x5601d7161120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.645+0000 7f1979ea3640 1 -- 172.21.15.88:0/3676954702 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601d7170100 msgr2=0x5601d7161120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.645+0000 7f1979ea3640 1 --2- 172.21.15.88:0/3676954702 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601d7170100 0x5601d7161120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.645+0000 7f1979ea3640 1 -- 172.21.15.88:0/3676954702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601d716fd20 msgr2=0x5601d715f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.645+0000 7f1979ea3640 1 --2- 172.21.15.88:0/3676954702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601d716fd20 0x5601d715f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.645+0000 7f1979ea3640 1 -- 172.21.15.88:0/3676954702 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601d716b730 con 0x5601d70cafe0 2024-09-16T01:40:12.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.645+0000 7f19716a2640 1 --2- 172.21.15.88:0/3676954702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601d716fd20 0x5601d715f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:12.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.645+0000 7f1979ea3640 1 --2- 172.21.15.88:0/3676954702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601d70cafe0 0x5601d716b1f0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f196c00a600 tx=0x7f196c002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.645+0000 7f19737fe640 1 -- 172.21.15.88:0/3676954702 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c071020 con 0x5601d70cafe0 2024-09-16T01:40:12.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.645+0000 7f19737fe640 1 -- 172.21.15.88:0/3676954702 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f196c0089f0 con 0x5601d70cafe0 2024-09-16T01:40:12.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.645+0000 7f197d01af40 1 -- 172.21.15.88:0/3676954702 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5601d709db20 con 0x5601d70cafe0 2024-09-16T01:40:12.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.645+0000 7f19737fe640 1 -- 172.21.15.88:0/3676954702 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c008e00 con 0x5601d70cafe0 2024-09-16T01:40:12.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.645+0000 7f197d01af40 1 -- 172.21.15.88:0/3676954702 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601d715fd60 con 0x5601d70cafe0 2024-09-16T01:40:12.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.646+0000 7f19737fe640 1 -- 172.21.15.88:0/3676954702 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f196c074020 con 0x5601d70cafe0 2024-09-16T01:40:12.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.647+0000 7f19737fe640 1 --2- 172.21.15.88:0/3676954702 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f195c06acc0 0x7f195c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.647+0000 7f19737fe640 1 -- 172.21.15.88:0/3676954702 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f196c0d0ba0 con 0x5601d70cafe0 2024-09-16T01:40:12.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.647+0000 7f19716a2640 1 --2- 172.21.15.88:0/3676954702 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f195c06acc0 0x7f195c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.647+0000 7f197d01af40 1 --2- 172.21.15.88:0/3676954702 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5601d709ad30 0x5601d7164b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.647+0000 7f197d01af40 1 -- 172.21.15.88:0/3676954702 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:1 12.13 12:c91d4a1d:::mds0_inotable:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5601d7161660 con 0x5601d709ad30 2024-09-16T01:40:12.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.647+0000 7f197a6a4640 1 --2- 172.21.15.88:0/3676954702 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5601d709ad30 0x5601d7164b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.647+0000 7f19716a2640 1 --2- 172.21.15.88:0/3676954702 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f195c06acc0 0x7f195c06d180 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x5601d709b950 tx=0x7f19640016e0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.647+0000 7f197a6a4640 1 --2- 172.21.15.88:0/3676954702 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5601d709ad30 0x5601d7164b80 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.648+0000 7f197a6a4640 1 -- 172.21.15.88:0/3676954702 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(1 mds0_inotable [omap-get-header] v0'0 uv3 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1974002650 con 0x5601d709ad30 2024-09-16T01:40:12.656 INFO:teuthology.orchestra.run.smithi088.stdout:header (0 bytes) : 2024-09-16T01:40:12.656 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:12.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.648+0000 7f197d01af40 1 -- 172.21.15.88:0/3676954702 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5601d709ad30 msgr2=0x5601d7164b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.648+0000 7f197d01af40 1 --2- 172.21.15.88:0/3676954702 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5601d709ad30 0x5601d7164b80 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.648+0000 7f197d01af40 1 -- 172.21.15.88:0/3676954702 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f195c06acc0 msgr2=0x7f195c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.648+0000 7f197d01af40 1 --2- 172.21.15.88:0/3676954702 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f195c06acc0 0x7f195c06d180 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x5601d709b950 tx=0x7f19640016e0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.648+0000 7f197d01af40 1 -- 172.21.15.88:0/3676954702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601d70cafe0 msgr2=0x5601d716b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.648+0000 7f197d01af40 1 --2- 172.21.15.88:0/3676954702 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601d70cafe0 0x5601d716b1f0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f196c00a600 tx=0x7f196c002fb0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.648+0000 7f197a6a4640 1 -- 172.21.15.88:0/3676954702 reap_dead start 2024-09-16T01:40:12.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.648+0000 7f197d01af40 1 -- 172.21.15.88:0/3676954702 shutdown_connections 2024-09-16T01:40:12.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.648+0000 7f197d01af40 1 -- 172.21.15.88:0/3676954702 >> 172.21.15.88:0/3676954702 conn(0x5601d70e6750 msgr2=0x5601d70c5e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:12.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.648+0000 7f197d01af40 1 -- 172.21.15.88:0/3676954702 shutdown_connections 2024-09-16T01:40:12.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.648+0000 7f197d01af40 1 -- 172.21.15.88:0/3676954702 wait complete. 2024-09-16T01:40:12.660 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 100.00000000 2024-09-16T01:40:12.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.740+0000 7f1e66f30f40 1 Processor -- start 2024-09-16T01:40:12.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.740+0000 7f1e66f30f40 1 -- start start 2024-09-16T01:40:12.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.741+0000 7f1e66f30f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56286095d480 0x5628609f6900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.741+0000 7f1e66f30f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56286097ae60 0x562860a053a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.741+0000 7f1e66f30f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562860a06f00 0x562860a058e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.741+0000 7f1e66f30f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x562860a07970 con 0x56286097ae60 2024-09-16T01:40:12.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.741+0000 7f1e66f30f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56286097b270 con 0x562860a06f00 2024-09-16T01:40:12.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.741+0000 7f1e66f30f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x562860a07c10 con 0x56286095d480 2024-09-16T01:40:12.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.741+0000 7f1e5ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56286097ae60 0x562860a053a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.741+0000 7f1e5ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56286097ae60 0x562860a053a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46256/0 (socket says 172.21.15.88:46256) 2024-09-16T01:40:12.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.741+0000 7f1e5ffff640 1 -- 172.21.15.88:0/3166906656 learned_addr learned my addr 172.21.15.88:0/3166906656 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:12.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.741+0000 7f1e64a28640 1 --2- 172.21.15.88:0/3166906656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56286095d480 0x5628609f6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.741+0000 7f1e65229640 1 --2- 172.21.15.88:0/3166906656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562860a06f00 0x562860a058e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.741+0000 7f1e5ffff640 1 -- 172.21.15.88:0/3166906656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56286095d480 msgr2=0x5628609f6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:12.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.741+0000 7f1e5ffff640 1 --2- 172.21.15.88:0/3166906656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56286095d480 0x5628609f6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.741+0000 7f1e5ffff640 1 -- 172.21.15.88:0/3166906656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562860a06f00 msgr2=0x562860a058e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:12.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.741+0000 7f1e5ffff640 1 --2- 172.21.15.88:0/3166906656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562860a06f00 0x562860a058e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.741+0000 7f1e5ffff640 1 -- 172.21.15.88:0/3166906656 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56286085ccb0 con 0x56286097ae60 2024-09-16T01:40:12.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.741+0000 7f1e5ffff640 1 --2- 172.21.15.88:0/3166906656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56286097ae60 0x562860a053a0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f1e5000a770 tx=0x7f1e5000a860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2afb7d201dba3aa3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.742+0000 7f1e5effd640 1 -- 172.21.15.88:0/3166906656 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e50019070 con 0x56286097ae60 2024-09-16T01:40:12.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.742+0000 7f1e5effd640 1 -- 172.21.15.88:0/3166906656 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e50007600 con 0x56286097ae60 2024-09-16T01:40:12.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.742+0000 7f1e5effd640 1 -- 172.21.15.88:0/3166906656 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e500079d0 con 0x56286097ae60 2024-09-16T01:40:12.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.742+0000 7f1e66f30f40 1 -- 172.21.15.88:0/3166906656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56286097ae60 msgr2=0x562860a053a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.742+0000 7f1e66f30f40 1 --2- 172.21.15.88:0/3166906656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56286097ae60 0x562860a053a0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f1e5000a770 tx=0x7f1e5000a860 comp rx=0 tx=0).stop 2024-09-16T01:40:12.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.742+0000 7f1e66f30f40 1 -- 172.21.15.88:0/3166906656 shutdown_connections 2024-09-16T01:40:12.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.742+0000 7f1e66f30f40 1 --2- 172.21.15.88:0/3166906656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562860a06f00 0x562860a058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.742+0000 7f1e66f30f40 1 --2- 172.21.15.88:0/3166906656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56286097ae60 0x562860a053a0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.742+0000 7f1e66f30f40 1 --2- 172.21.15.88:0/3166906656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56286095d480 0x5628609f6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.742+0000 7f1e66f30f40 1 -- 172.21.15.88:0/3166906656 >> 172.21.15.88:0/3166906656 conn(0x56286095c960 msgr2=0x56286095cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:12.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.742+0000 7f1e66f30f40 1 -- 172.21.15.88:0/3166906656 shutdown_connections 2024-09-16T01:40:12.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.742+0000 7f1e66f30f40 1 -- 172.21.15.88:0/3166906656 wait complete. 2024-09-16T01:40:12.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.743+0000 7f1e66f30f40 1 Processor -- start 2024-09-16T01:40:12.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.743+0000 7f1e66f30f40 1 -- start start 2024-09-16T01:40:12.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.743+0000 7f1e66f30f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56286095d480 0x5628609ea3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.743+0000 7f1e66f30f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56286097ae60 0x562860a94c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.743+0000 7f1e66f30f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562860a06f00 0x562860a951c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.743+0000 7f1e66f30f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5628609782b0 con 0x56286097ae60 2024-09-16T01:40:12.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.743+0000 7f1e66f30f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562860933d40 con 0x562860a06f00 2024-09-16T01:40:12.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.743+0000 7f1e66f30f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x562860978590 con 0x56286095d480 2024-09-16T01:40:12.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.743+0000 7f1e5ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56286097ae60 0x562860a94c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.743+0000 7f1e5ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56286097ae60 0x562860a94c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46262/0 (socket says 172.21.15.88:46262) 2024-09-16T01:40:12.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.743+0000 7f1e5ffff640 1 -- 172.21.15.88:0/3552123162 learned_addr learned my addr 172.21.15.88:0/3552123162 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:12.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.744+0000 7f1e64a28640 1 --2- 172.21.15.88:0/3552123162 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56286095d480 0x5628609ea3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.744+0000 7f1e65229640 1 --2- 172.21.15.88:0/3552123162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562860a06f00 0x562860a951c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.744+0000 7f1e5ffff640 1 -- 172.21.15.88:0/3552123162 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56286095d480 msgr2=0x5628609ea3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.744+0000 7f1e5ffff640 1 --2- 172.21.15.88:0/3552123162 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56286095d480 0x5628609ea3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.744+0000 7f1e5ffff640 1 -- 172.21.15.88:0/3552123162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562860a06f00 msgr2=0x562860a951c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.744+0000 7f1e5ffff640 1 --2- 172.21.15.88:0/3552123162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562860a06f00 0x562860a951c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.744+0000 7f1e5ffff640 1 -- 172.21.15.88:0/3552123162 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562860a05330 con 0x56286097ae60 2024-09-16T01:40:12.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.744+0000 7f1e64a28640 1 --2- 172.21.15.88:0/3552123162 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56286095d480 0x5628609ea3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:12.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.744+0000 7f1e5ffff640 1 --2- 172.21.15.88:0/3552123162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56286097ae60 0x562860a94c80 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f1e5000a740 tx=0x7f1e50003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.744+0000 7f1e5dffb640 1 -- 172.21.15.88:0/3552123162 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e50019040 con 0x56286097ae60 2024-09-16T01:40:12.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.744+0000 7f1e5dffb640 1 -- 172.21.15.88:0/3552123162 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e50014470 con 0x56286097ae60 2024-09-16T01:40:12.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.744+0000 7f1e5dffb640 1 -- 172.21.15.88:0/3552123162 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e50014860 con 0x56286097ae60 2024-09-16T01:40:12.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.744+0000 7f1e66f30f40 1 -- 172.21.15.88:0/3552123162 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x562860930d30 con 0x56286097ae60 2024-09-16T01:40:12.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.744+0000 7f1e66f30f40 1 -- 172.21.15.88:0/3552123162 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628609ea8e0 con 0x56286097ae60 2024-09-16T01:40:12.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.745+0000 7f1e5dffb640 1 -- 172.21.15.88:0/3552123162 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1e50014a00 con 0x56286097ae60 2024-09-16T01:40:12.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.746+0000 7f1e5dffb640 1 --2- 172.21.15.88:0/3552123162 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1e4806ac90 0x7f1e4806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.746+0000 7f1e5dffb640 1 -- 172.21.15.88:0/3552123162 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f1e5008b060 con 0x56286097ae60 2024-09-16T01:40:12.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.746+0000 7f1e64a28640 1 --2- 172.21.15.88:0/3552123162 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1e4806ac90 0x7f1e4806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.746+0000 7f1e66f30f40 1 --2- 172.21.15.88:0/3552123162 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x562860977dd0 0x5628609f1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.746+0000 7f1e66f30f40 1 -- 172.21.15.88:0/3552123162 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:1 12.13 12:cd5a64a3:::100.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x562860a0b6c0 con 0x562860977dd0 2024-09-16T01:40:12.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.746+0000 7f1e65229640 1 --2- 172.21.15.88:0/3552123162 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x562860977dd0 0x5628609f1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.746+0000 7f1e64a28640 1 --2- 172.21.15.88:0/3552123162 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1e4806ac90 0x7f1e4806d150 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f1e58057000 tx=0x7f1e58052000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.746+0000 7f1e65229640 1 --2- 172.21.15.88:0/3552123162 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x562860977dd0 0x5628609f1fd0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.747+0000 7f1e65229640 1 -- 172.21.15.88:0/3552123162 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(1 100.00000000 [omap-get-header out=274b] v0'0 uv1 ondisk = 0) ==== 156+0+274 (crc 0 0 0) 0x7f1e60011040 con 0x562860977dd0 2024-09-16T01:40:12.753 INFO:teuthology.orchestra.run.smithi088.stdout:header (274 bytes) : 2024-09-16T01:40:12.753 INFO:teuthology.orchestra.run.smithi088.stdout:00000000 04 03 0c 01 00 00 01 00 00 00 00 00 00 00 00 00 |................| 2024-09-16T01:40:12.753 INFO:teuthology.orchestra.run.smithi088.stdout:00000010 00 00 00 00 00 00 03 02 28 00 00 00 00 00 00 00 |........(.......| 2024-09-16T01:40:12.753 INFO:teuthology.orchestra.run.smithi088.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-16T01:40:12.753 INFO:teuthology.orchestra.run.smithi088.stdout:00000030 00 00 00 00 0a 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-16T01:40:12.753 INFO:teuthology.orchestra.run.smithi088.stdout:00000040 00 00 00 00 03 02 28 00 00 00 00 00 00 00 00 00 |......(.........| 2024-09-16T01:40:12.753 INFO:teuthology.orchestra.run.smithi088.stdout:00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-16T01:40:12.753 INFO:teuthology.orchestra.run.smithi088.stdout:00000060 00 00 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-16T01:40:12.754 INFO:teuthology.orchestra.run.smithi088.stdout:00000070 00 00 03 02 38 00 00 00 00 00 00 00 00 00 00 00 |....8...........| 2024-09-16T01:40:12.754 INFO:teuthology.orchestra.run.smithi088.stdout:00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-16T01:40:12.754 INFO:teuthology.orchestra.run.smithi088.stdout:00000090 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-16T01:40:12.754 INFO:teuthology.orchestra.run.smithi088.stdout:000000a0 00 00 00 00 00 00 00 00 74 8c e7 66 93 3a 20 0f |........t..f.: .| 2024-09-16T01:40:12.754 INFO:teuthology.orchestra.run.smithi088.stdout:000000b0 03 02 38 00 00 00 00 00 00 00 00 00 00 00 00 00 |..8.............| 2024-09-16T01:40:12.754 INFO:teuthology.orchestra.run.smithi088.stdout:000000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0a 00 |................| 2024-09-16T01:40:12.754 INFO:teuthology.orchestra.run.smithi088.stdout:000000d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-16T01:40:12.754 INFO:teuthology.orchestra.run.smithi088.stdout:000000e0 00 00 00 00 00 00 74 8c e7 66 93 3a 20 0f 00 00 |......t..f.: ...| 2024-09-16T01:40:12.755 INFO:teuthology.orchestra.run.smithi088.stdout:000000f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-16T01:40:12.755 INFO:teuthology.orchestra.run.smithi088.stdout:* 2024-09-16T01:40:12.755 INFO:teuthology.orchestra.run.smithi088.stdout:00000110 00 00 |..| 2024-09-16T01:40:12.755 INFO:teuthology.orchestra.run.smithi088.stdout:00000112 2024-09-16T01:40:12.755 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:12.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.747+0000 7f1e66f30f40 1 -- 172.21.15.88:0/3552123162 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x562860977dd0 msgr2=0x5628609f1fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.747+0000 7f1e66f30f40 1 --2- 172.21.15.88:0/3552123162 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x562860977dd0 0x5628609f1fd0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.747+0000 7f1e66f30f40 1 -- 172.21.15.88:0/3552123162 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1e4806ac90 msgr2=0x7f1e4806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.747+0000 7f1e66f30f40 1 --2- 172.21.15.88:0/3552123162 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1e4806ac90 0x7f1e4806d150 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f1e58057000 tx=0x7f1e58052000 comp rx=0 tx=0).stop 2024-09-16T01:40:12.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.747+0000 7f1e66f30f40 1 -- 172.21.15.88:0/3552123162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56286097ae60 msgr2=0x562860a94c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.747+0000 7f1e66f30f40 1 --2- 172.21.15.88:0/3552123162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56286097ae60 0x562860a94c80 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f1e5000a740 tx=0x7f1e50003020 comp rx=0 tx=0).stop 2024-09-16T01:40:12.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.747+0000 7f1e65229640 1 -- 172.21.15.88:0/3552123162 reap_dead start 2024-09-16T01:40:12.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.747+0000 7f1e66f30f40 1 -- 172.21.15.88:0/3552123162 shutdown_connections 2024-09-16T01:40:12.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.747+0000 7f1e66f30f40 1 -- 172.21.15.88:0/3552123162 >> 172.21.15.88:0/3552123162 conn(0x56286095c960 msgr2=0x562860933ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:12.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.747+0000 7f1e66f30f40 1 -- 172.21.15.88:0/3552123162 shutdown_connections 2024-09-16T01:40:12.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.747+0000 7f1e66f30f40 1 -- 172.21.15.88:0/3552123162 wait complete. 2024-09-16T01:40:12.758 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader mds0_sessionmap 2024-09-16T01:40:12.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.838+0000 7fc99625df40 1 Processor -- start 2024-09-16T01:40:12.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.838+0000 7fc99625df40 1 -- start start 2024-09-16T01:40:12.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.839+0000 7fc99625df40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f81cb7ced0 0x55f81cbe87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.839+0000 7fc99625df40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f81cbf2e00 0x55f81cc0af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.839+0000 7fc99625df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f81cc0ab70 0x55f81cbf1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.839+0000 7fc99625df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f81ca86510 con 0x55f81cc0ab70 2024-09-16T01:40:12.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.839+0000 7fc99625df40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f81cb7d4a0 con 0x55f81cbf2e00 2024-09-16T01:40:12.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.839+0000 7fc99625df40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55f81cbec1a0 con 0x55f81cb7ced0 2024-09-16T01:40:12.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.839+0000 7fc994554640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f81cc0ab70 0x55f81cbf1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.839+0000 7fc994554640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f81cc0ab70 0x55f81cbf1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46272/0 (socket says 172.21.15.88:46272) 2024-09-16T01:40:12.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.839+0000 7fc994554640 1 -- 172.21.15.88:0/1576226514 learned_addr learned my addr 172.21.15.88:0/1576226514 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:12.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.839+0000 7fc994554640 1 -- 172.21.15.88:0/1576226514 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f81cb7ced0 msgr2=0x55f81cbe87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:12.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.839+0000 7fc994554640 1 --2- 172.21.15.88:0/1576226514 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f81cb7ced0 0x55f81cbe87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.839+0000 7fc994554640 1 -- 172.21.15.88:0/1576226514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f81cbf2e00 msgr2=0x55f81cc0af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:12.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.839+0000 7fc994554640 1 --2- 172.21.15.88:0/1576226514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f81cbf2e00 0x55f81cc0af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.839+0000 7fc994554640 1 -- 172.21.15.88:0/1576226514 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f81cbe8140 con 0x55f81cc0ab70 2024-09-16T01:40:12.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.839+0000 7fc994554640 1 --2- 172.21.15.88:0/1576226514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f81cc0ab70 0x55f81cbf1e20 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fc98c00a490 tx=0x7fc98c00a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=929f95380b7f2c21 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.840+0000 7fc992550640 1 -- 172.21.15.88:0/1576226514 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98c00f7b0 con 0x55f81cc0ab70 2024-09-16T01:40:12.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.840+0000 7fc992550640 1 -- 172.21.15.88:0/1576226514 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc98c015070 con 0x55f81cc0ab70 2024-09-16T01:40:12.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.840+0000 7fc992550640 1 -- 172.21.15.88:0/1576226514 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98c00fb90 con 0x55f81cc0ab70 2024-09-16T01:40:12.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.840+0000 7fc99625df40 1 -- 172.21.15.88:0/1576226514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f81cc0ab70 msgr2=0x55f81cbf1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.840+0000 7fc99625df40 1 --2- 172.21.15.88:0/1576226514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f81cc0ab70 0x55f81cbf1e20 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fc98c00a490 tx=0x7fc98c00a950 comp rx=0 tx=0).stop 2024-09-16T01:40:12.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.840+0000 7fc99625df40 1 -- 172.21.15.88:0/1576226514 shutdown_connections 2024-09-16T01:40:12.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.840+0000 7fc99625df40 1 --2- 172.21.15.88:0/1576226514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f81cc0ab70 0x55f81cbf1e20 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.840+0000 7fc99625df40 1 --2- 172.21.15.88:0/1576226514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f81cbf2e00 0x55f81cc0af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.840+0000 7fc99625df40 1 --2- 172.21.15.88:0/1576226514 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f81cb7ced0 0x55f81cbe87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.840+0000 7fc99625df40 1 -- 172.21.15.88:0/1576226514 >> 172.21.15.88:0/1576226514 conn(0x55f81cb5f960 msgr2=0x55f81cc0ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:12.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.840+0000 7fc99625df40 1 -- 172.21.15.88:0/1576226514 shutdown_connections 2024-09-16T01:40:12.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.840+0000 7fc99625df40 1 -- 172.21.15.88:0/1576226514 wait complete. 2024-09-16T01:40:12.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.841+0000 7fc99625df40 1 Processor -- start 2024-09-16T01:40:12.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.841+0000 7fc99625df40 1 -- start start 2024-09-16T01:40:12.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.841+0000 7fc99625df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f81cb7ced0 0x55f81cbeb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.841+0000 7fc99625df40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f81cbf2e00 0x55f81cc09db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.841+0000 7fc99625df40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f81cc0ab70 0x55f81cc0af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.841+0000 7fc99625df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f81cbebec0 con 0x55f81cb7ced0 2024-09-16T01:40:12.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.841+0000 7fc99625df40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f81cbf2360 con 0x55f81cbf2e00 2024-09-16T01:40:12.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.841+0000 7fc99625df40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55f81cbe83c0 con 0x55f81cc0ab70 2024-09-16T01:40:12.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.841+0000 7fc993d53640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f81cb7ced0 0x55f81cbeb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.841+0000 7fc993d53640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f81cb7ced0 0x55f81cbeb0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46278/0 (socket says 172.21.15.88:46278) 2024-09-16T01:40:12.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.842+0000 7fc993d53640 1 -- 172.21.15.88:0/826458106 learned_addr learned my addr 172.21.15.88:0/826458106 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:12.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.842+0000 7fc994554640 1 --2- 172.21.15.88:0/826458106 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f81cc0ab70 0x55f81cc0af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.842+0000 7fc993552640 1 --2- 172.21.15.88:0/826458106 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f81cbf2e00 0x55f81cc09db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.842+0000 7fc993d53640 1 -- 172.21.15.88:0/826458106 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f81cc0ab70 msgr2=0x55f81cc0af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.842+0000 7fc993d53640 1 --2- 172.21.15.88:0/826458106 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f81cc0ab70 0x55f81cc0af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.842+0000 7fc993d53640 1 -- 172.21.15.88:0/826458106 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f81cbf2e00 msgr2=0x55f81cc09db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.842+0000 7fc993d53640 1 --2- 172.21.15.88:0/826458106 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f81cbf2e00 0x55f81cc09db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.842+0000 7fc993d53640 1 -- 172.21.15.88:0/826458106 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f81cbeb680 con 0x55f81cb7ced0 2024-09-16T01:40:12.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.842+0000 7fc993552640 1 --2- 172.21.15.88:0/826458106 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f81cbf2e00 0x55f81cc09db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:12.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.842+0000 7fc993d53640 1 --2- 172.21.15.88:0/826458106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f81cb7ced0 0x55f81cbeb0a0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fc984008460 tx=0x7fc984008920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.842+0000 7fc99154e640 1 -- 172.21.15.88:0/826458106 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc984016070 con 0x55f81cb7ced0 2024-09-16T01:40:12.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.842+0000 7fc99154e640 1 -- 172.21.15.88:0/826458106 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9840067c0 con 0x55f81cb7ced0 2024-09-16T01:40:12.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.842+0000 7fc99625df40 1 -- 172.21.15.88:0/826458106 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f81cb757d0 con 0x55f81cb7ced0 2024-09-16T01:40:12.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.842+0000 7fc99154e640 1 -- 172.21.15.88:0/826458106 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc984006bb0 con 0x55f81cb7ced0 2024-09-16T01:40:12.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.842+0000 7fc99625df40 1 -- 172.21.15.88:0/826458106 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f81cc0a2f0 con 0x55f81cb7ced0 2024-09-16T01:40:12.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.843+0000 7fc99154e640 1 -- 172.21.15.88:0/826458106 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc984019070 con 0x55f81cb7ced0 2024-09-16T01:40:12.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.844+0000 7fc99154e640 1 --2- 172.21.15.88:0/826458106 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc98006ac90 0x7fc98006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.844+0000 7fc993552640 1 --2- 172.21.15.88:0/826458106 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc98006ac90 0x7fc98006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.844+0000 7fc99154e640 1 -- 172.21.15.88:0/826458106 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7fc984089a60 con 0x55f81cb7ced0 2024-09-16T01:40:12.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.844+0000 7fc99625df40 1 --2- 172.21.15.88:0/826458106 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55f81cbe9680 0x55f81cbe9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.844+0000 7fc99625df40 1 -- 172.21.15.88:0/826458106 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:1 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55f81cbe9fa0 con 0x55f81cbe9680 2024-09-16T01:40:12.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.844+0000 7fc994554640 1 --2- 172.21.15.88:0/826458106 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55f81cbe9680 0x55f81cbe9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.844+0000 7fc993552640 1 --2- 172.21.15.88:0/826458106 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc98006ac90 0x7fc98006d150 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc988057000 tx=0x7fc988051000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.845+0000 7fc994554640 1 --2- 172.21.15.88:0/826458106 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55f81cbe9680 0x55f81cbe9a60 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.845+0000 7fc994554640 1 -- 172.21.15.88:0/826458106 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(1 mds0_sessionmap [omap-get-header out=14b] v0'0 uv3 ondisk = 0) ==== 159+0+14 (crc 0 0 0) 0x7fc98c00e070 con 0x55f81cbe9680 2024-09-16T01:40:12.851 INFO:teuthology.orchestra.run.smithi088.stdout:header (14 bytes) : 2024-09-16T01:40:12.851 INFO:teuthology.orchestra.run.smithi088.stdout:00000000 01 01 08 00 00 00 04 00 00 00 00 00 00 00 |..............| 2024-09-16T01:40:12.852 INFO:teuthology.orchestra.run.smithi088.stdout:0000000e 2024-09-16T01:40:12.852 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:12.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.845+0000 7fc99625df40 1 -- 172.21.15.88:0/826458106 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55f81cbe9680 msgr2=0x55f81cbe9a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.845+0000 7fc99625df40 1 --2- 172.21.15.88:0/826458106 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55f81cbe9680 0x55f81cbe9a60 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.845+0000 7fc99625df40 1 -- 172.21.15.88:0/826458106 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc98006ac90 msgr2=0x7fc98006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.845+0000 7fc99625df40 1 --2- 172.21.15.88:0/826458106 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc98006ac90 0x7fc98006d150 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc988057000 tx=0x7fc988051000 comp rx=0 tx=0).stop 2024-09-16T01:40:12.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.845+0000 7fc99625df40 1 -- 172.21.15.88:0/826458106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f81cb7ced0 msgr2=0x55f81cbeb0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.845+0000 7fc99625df40 1 --2- 172.21.15.88:0/826458106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f81cb7ced0 0x55f81cbeb0a0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fc984008460 tx=0x7fc984008920 comp rx=0 tx=0).stop 2024-09-16T01:40:12.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.845+0000 7fc994554640 1 -- 172.21.15.88:0/826458106 reap_dead start 2024-09-16T01:40:12.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.845+0000 7fc99625df40 1 -- 172.21.15.88:0/826458106 shutdown_connections 2024-09-16T01:40:12.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.845+0000 7fc99625df40 1 -- 172.21.15.88:0/826458106 >> 172.21.15.88:0/826458106 conn(0x55f81cb5f960 msgr2=0x55f81cbf06d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:12.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.845+0000 7fc99625df40 1 -- 172.21.15.88:0/826458106 shutdown_connections 2024-09-16T01:40:12.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.845+0000 7fc99625df40 1 -- 172.21.15.88:0/826458106 wait complete. 2024-09-16T01:40:12.854 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 400.00000000 2024-09-16T01:40:12.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.936+0000 7fabceb92f40 1 Processor -- start 2024-09-16T01:40:12.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.936+0000 7fabceb92f40 1 -- start start 2024-09-16T01:40:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.937+0000 7fabceb92f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabb40030f0 0x55ffe28558f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.937+0000 7fabceb92f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffe27ac4d0 0x55ffe27aa970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.937+0000 7fabceb92f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ffe2855e30 0x55ffe27aaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.937+0000 7fabceb92f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55ffe27c8640 con 0x55ffe2855e30 2024-09-16T01:40:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.937+0000 7fabceb92f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ffe2643510 con 0x55ffe27ac4d0 2024-09-16T01:40:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.937+0000 7fabceb92f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55ffe27ad720 con 0x7fabb40030f0 2024-09-16T01:40:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.937+0000 7fabcce89640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ffe2855e30 0x55ffe27aaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.937+0000 7fabcce89640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ffe2855e30 0x55ffe27aaeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46302/0 (socket says 172.21.15.88:46302) 2024-09-16T01:40:12.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.937+0000 7fabcce89640 1 -- 172.21.15.88:0/3465700757 learned_addr learned my addr 172.21.15.88:0/3465700757 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:12.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.937+0000 7fabcc688640 1 --2- 172.21.15.88:0/3465700757 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabb40030f0 0x55ffe28558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.937+0000 7fabcce89640 1 -- 172.21.15.88:0/3465700757 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabb40030f0 msgr2=0x55ffe28558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:12.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.937+0000 7fabcbe87640 1 --2- 172.21.15.88:0/3465700757 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffe27ac4d0 0x55ffe27aa970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.937+0000 7fabcce89640 1 --2- 172.21.15.88:0/3465700757 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabb40030f0 0x55ffe28558f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.937+0000 7fabcce89640 1 -- 172.21.15.88:0/3465700757 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffe27ac4d0 msgr2=0x55ffe27aa970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:12.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.937+0000 7fabcce89640 1 --2- 172.21.15.88:0/3465700757 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffe27ac4d0 0x55ffe27aa970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.937+0000 7fabcce89640 1 -- 172.21.15.88:0/3465700757 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffe27a5270 con 0x55ffe2855e30 2024-09-16T01:40:12.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.937+0000 7fabcce89640 1 --2- 172.21.15.88:0/3465700757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ffe2855e30 0x55ffe27aaeb0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x55ffe26f5840 tx=0x7fabc4005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=534874f926ce9700 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.937+0000 7fabcae85640 1 -- 172.21.15.88:0/3465700757 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc4012670 con 0x55ffe2855e30 2024-09-16T01:40:12.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.937+0000 7fabcae85640 1 -- 172.21.15.88:0/3465700757 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabc401a070 con 0x55ffe2855e30 2024-09-16T01:40:12.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.937+0000 7fabcae85640 1 -- 172.21.15.88:0/3465700757 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc4012a10 con 0x55ffe2855e30 2024-09-16T01:40:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.938+0000 7fabceb92f40 1 -- 172.21.15.88:0/3465700757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ffe2855e30 msgr2=0x55ffe27aaeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.938+0000 7fabceb92f40 1 --2- 172.21.15.88:0/3465700757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ffe2855e30 0x55ffe27aaeb0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x55ffe26f5840 tx=0x7fabc4005d80 comp rx=0 tx=0).stop 2024-09-16T01:40:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.938+0000 7fabceb92f40 1 -- 172.21.15.88:0/3465700757 shutdown_connections 2024-09-16T01:40:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.938+0000 7fabceb92f40 1 --2- 172.21.15.88:0/3465700757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ffe2855e30 0x55ffe27aaeb0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.938+0000 7fabceb92f40 1 --2- 172.21.15.88:0/3465700757 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ffe27ac4d0 0x55ffe27aa970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.938+0000 7fabceb92f40 1 --2- 172.21.15.88:0/3465700757 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabb40030f0 0x55ffe28558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.938+0000 7fabceb92f40 1 -- 172.21.15.88:0/3465700757 >> 172.21.15.88:0/3465700757 conn(0x55ffe271c960 msgr2=0x55ffe27c87c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.938+0000 7fabceb92f40 1 -- 172.21.15.88:0/3465700757 shutdown_connections 2024-09-16T01:40:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.938+0000 7fabceb92f40 1 -- 172.21.15.88:0/3465700757 wait complete. 2024-09-16T01:40:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.938+0000 7fabceb92f40 1 Processor -- start 2024-09-16T01:40:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.939+0000 7fabceb92f40 1 -- start start 2024-09-16T01:40:12.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.939+0000 7fabceb92f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ffe27ac4d0 0x55ffe2856c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.939+0000 7fabceb92f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ffe2855e30 0x55ffe27c2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.939+0000 7fabceb92f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabb40030f0 0x55ffe27c2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.939+0000 7fabceb92f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55ffe27ca0f0 con 0x55ffe27ac4d0 2024-09-16T01:40:12.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.939+0000 7fabceb92f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ffe27ada00 con 0x7fabb40030f0 2024-09-16T01:40:12.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.939+0000 7fabceb92f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55ffe27a5620 con 0x55ffe2855e30 2024-09-16T01:40:12.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.939+0000 7fabcc688640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ffe27ac4d0 0x55ffe2856c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.939+0000 7fabcc688640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ffe27ac4d0 0x55ffe2856c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46314/0 (socket says 172.21.15.88:46314) 2024-09-16T01:40:12.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.939+0000 7fabcc688640 1 -- 172.21.15.88:0/1793832407 learned_addr learned my addr 172.21.15.88:0/1793832407 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:12.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.939+0000 7fabcce89640 1 --2- 172.21.15.88:0/1793832407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabb40030f0 0x55ffe27c2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.940+0000 7fabcbe87640 1 --2- 172.21.15.88:0/1793832407 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ffe2855e30 0x55ffe27c2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.940+0000 7fabcc688640 1 -- 172.21.15.88:0/1793832407 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ffe2855e30 msgr2=0x55ffe27c2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.940+0000 7fabcc688640 1 --2- 172.21.15.88:0/1793832407 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ffe2855e30 0x55ffe27c2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.940+0000 7fabcc688640 1 -- 172.21.15.88:0/1793832407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabb40030f0 msgr2=0x55ffe27c2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.940+0000 7fabcc688640 1 --2- 172.21.15.88:0/1793832407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabb40030f0 0x55ffe27c2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.940+0000 7fabcc688640 1 -- 172.21.15.88:0/1793832407 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffe2858350 con 0x55ffe27ac4d0 2024-09-16T01:40:12.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.940+0000 7fabcc688640 1 --2- 172.21.15.88:0/1793832407 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ffe27ac4d0 0x55ffe2856c20 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fabbc00f470 tx=0x7fabbc00f930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.940+0000 7fabc9e83640 1 -- 172.21.15.88:0/1793832407 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabbc00d8a0 con 0x55ffe27ac4d0 2024-09-16T01:40:12.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.940+0000 7fabc9e83640 1 -- 172.21.15.88:0/1793832407 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabbc00da40 con 0x55ffe27ac4d0 2024-09-16T01:40:12.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.940+0000 7fabceb92f40 1 -- 172.21.15.88:0/1793832407 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ffe28572b0 con 0x55ffe27ac4d0 2024-09-16T01:40:12.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.940+0000 7fabc9e83640 1 -- 172.21.15.88:0/1793832407 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabbc00ddf0 con 0x55ffe27ac4d0 2024-09-16T01:40:12.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.940+0000 7fabceb92f40 1 -- 172.21.15.88:0/1793832407 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffe2857870 con 0x55ffe27ac4d0 2024-09-16T01:40:12.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.941+0000 7fabc9e83640 1 -- 172.21.15.88:0/1793832407 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fabbc021440 con 0x55ffe27ac4d0 2024-09-16T01:40:12.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.942+0000 7fabc9e83640 1 --2- 172.21.15.88:0/1793832407 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fabb806acc0 0x7fabb806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.942+0000 7fabc9e83640 1 -- 172.21.15.88:0/1793832407 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7fabbc08ad40 con 0x55ffe27ac4d0 2024-09-16T01:40:12.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.942+0000 7fabcbe87640 1 --2- 172.21.15.88:0/1793832407 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fabb806acc0 0x7fabb806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.942+0000 7fabceb92f40 1 --2- 172.21.15.88:0/1793832407 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55ffe285e4b0 0x55ffe285e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:12.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.942+0000 7fabceb92f40 1 -- 172.21.15.88:0/1793832407 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:1 12.f 12:f1f69726:::400.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55ffe285edd0 con 0x55ffe285e4b0 2024-09-16T01:40:12.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.942+0000 7fabcbe87640 1 --2- 172.21.15.88:0/1793832407 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fabb806acc0 0x7fabb806d180 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fabc0009810 tx=0x7fabc005a040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.942+0000 7fabcce89640 1 --2- 172.21.15.88:0/1793832407 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55ffe285e4b0 0x55ffe285e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:12.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.943+0000 7fabcce89640 1 --2- 172.21.15.88:0/1793832407 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55ffe285e4b0 0x55ffe285e890 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:12.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.943+0000 7fabcce89640 1 -- 172.21.15.88:0/1793832407 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(1 400.00000000 [omap-get-header] v0'0 uv1 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fabc400e040 con 0x55ffe285e4b0 2024-09-16T01:40:12.948 INFO:teuthology.orchestra.run.smithi088.stdout:header (0 bytes) : 2024-09-16T01:40:12.948 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:12.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.943+0000 7fabceb92f40 1 -- 172.21.15.88:0/1793832407 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55ffe285e4b0 msgr2=0x55ffe285e890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.943+0000 7fabceb92f40 1 --2- 172.21.15.88:0/1793832407 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55ffe285e4b0 0x55ffe285e890 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:12.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.943+0000 7fabceb92f40 1 -- 172.21.15.88:0/1793832407 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fabb806acc0 msgr2=0x7fabb806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.943+0000 7fabceb92f40 1 --2- 172.21.15.88:0/1793832407 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fabb806acc0 0x7fabb806d180 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fabc0009810 tx=0x7fabc005a040 comp rx=0 tx=0).stop 2024-09-16T01:40:12.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.943+0000 7fabceb92f40 1 -- 172.21.15.88:0/1793832407 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ffe27ac4d0 msgr2=0x55ffe2856c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:12.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.943+0000 7fabceb92f40 1 --2- 172.21.15.88:0/1793832407 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ffe27ac4d0 0x55ffe2856c20 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fabbc00f470 tx=0x7fabbc00f930 comp rx=0 tx=0).stop 2024-09-16T01:40:12.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.943+0000 7fabcce89640 1 -- 172.21.15.88:0/1793832407 reap_dead start 2024-09-16T01:40:12.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.943+0000 7fabceb92f40 1 -- 172.21.15.88:0/1793832407 shutdown_connections 2024-09-16T01:40:12.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.943+0000 7fabceb92f40 1 -- 172.21.15.88:0/1793832407 >> 172.21.15.88:0/1793832407 conn(0x55ffe271c960 msgr2=0x55ffe27c7800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:12.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.943+0000 7fabceb92f40 1 -- 172.21.15.88:0/1793832407 shutdown_connections 2024-09-16T01:40:12.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:12.943+0000 7fabceb92f40 1 -- 172.21.15.88:0/1793832407 wait complete. 2024-09-16T01:40:12.951 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 100.00000000.inode 2024-09-16T01:40:13.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.031+0000 7fb71033df40 1 Processor -- start 2024-09-16T01:40:13.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.031+0000 7fb71033df40 1 -- start start 2024-09-16T01:40:13.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.032+0000 7fb71033df40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b6fddd9f00 0x55b6fdea7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.032+0000 7fb71033df40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6fde8fed0 0x55b6fde883b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.032+0000 7fb71033df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6fde92190 0x55b6fde22610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.032+0000 7fb71033df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b6fddda850 con 0x55b6fde92190 2024-09-16T01:40:13.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.032+0000 7fb71033df40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b6fde91210 con 0x55b6fde8fed0 2024-09-16T01:40:13.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.032+0000 7fb71033df40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55b6fde914b0 con 0x55b6fddd9f00 2024-09-16T01:40:13.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.032+0000 7fb70e634640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6fde92190 0x55b6fde22610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.032+0000 7fb70e634640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6fde92190 0x55b6fde22610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46326/0 (socket says 172.21.15.88:46326) 2024-09-16T01:40:13.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.032+0000 7fb70e634640 1 -- 172.21.15.88:0/1151490911 learned_addr learned my addr 172.21.15.88:0/1151490911 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:13.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.032+0000 7fb705632640 1 --2- 172.21.15.88:0/1151490911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6fde8fed0 0x55b6fde883b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.032+0000 7fb70e634640 1 -- 172.21.15.88:0/1151490911 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b6fddd9f00 msgr2=0x55b6fdea7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:13.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.032+0000 7fb70de33640 1 --2- 172.21.15.88:0/1151490911 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b6fddd9f00 0x55b6fdea7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.032+0000 7fb70e634640 1 --2- 172.21.15.88:0/1151490911 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b6fddd9f00 0x55b6fdea7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.032+0000 7fb70e634640 1 -- 172.21.15.88:0/1151490911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6fde8fed0 msgr2=0x55b6fde883b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:13.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.032+0000 7fb70e634640 1 --2- 172.21.15.88:0/1151490911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6fde8fed0 0x55b6fde883b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.032+0000 7fb70e634640 1 -- 172.21.15.88:0/1151490911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6fdd02cb0 con 0x55b6fde92190 2024-09-16T01:40:13.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.032+0000 7fb705632640 1 --2- 172.21.15.88:0/1151490911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6fde8fed0 0x55b6fde883b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:13.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.032+0000 7fb70e634640 1 --2- 172.21.15.88:0/1151490911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6fde92190 0x55b6fde22610 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fb70800a4e0 tx=0x7fb70800a9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7646b635dfb00f2d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.033+0000 7fb70ce31640 1 -- 172.21.15.88:0/1151490911 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70800f7b0 con 0x55b6fde92190 2024-09-16T01:40:13.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.033+0000 7fb70ce31640 1 -- 172.21.15.88:0/1151490911 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb708015070 con 0x55b6fde92190 2024-09-16T01:40:13.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.033+0000 7fb70ce31640 1 -- 172.21.15.88:0/1151490911 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70800fb70 con 0x55b6fde92190 2024-09-16T01:40:13.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.033+0000 7fb71033df40 1 -- 172.21.15.88:0/1151490911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6fde92190 msgr2=0x55b6fde22610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.033+0000 7fb71033df40 1 --2- 172.21.15.88:0/1151490911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6fde92190 0x55b6fde22610 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fb70800a4e0 tx=0x7fb70800a9a0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.033+0000 7fb71033df40 1 -- 172.21.15.88:0/1151490911 shutdown_connections 2024-09-16T01:40:13.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.033+0000 7fb71033df40 1 --2- 172.21.15.88:0/1151490911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6fde92190 0x55b6fde22610 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.033+0000 7fb71033df40 1 --2- 172.21.15.88:0/1151490911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6fde8fed0 0x55b6fde883b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.033+0000 7fb71033df40 1 --2- 172.21.15.88:0/1151490911 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b6fddd9f00 0x55b6fdea7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.033+0000 7fb71033df40 1 -- 172.21.15.88:0/1151490911 >> 172.21.15.88:0/1151490911 conn(0x55b6fddd6be0 msgr2=0x55b6fde20b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:13.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.033+0000 7fb71033df40 1 -- 172.21.15.88:0/1151490911 shutdown_connections 2024-09-16T01:40:13.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.033+0000 7fb71033df40 1 -- 172.21.15.88:0/1151490911 wait complete. 2024-09-16T01:40:13.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.034+0000 7fb71033df40 1 Processor -- start 2024-09-16T01:40:13.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.034+0000 7fb71033df40 1 -- start start 2024-09-16T01:40:13.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.034+0000 7fb71033df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6fddd9f00 0x55b6fde89750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.034+0000 7fb71033df40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6fde8fed0 0x55b6fdea16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.034+0000 7fb71033df40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b6fde92190 0x55b6fde90550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.034+0000 7fb71033df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b6fdeb0720 con 0x55b6fddd9f00 2024-09-16T01:40:13.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.034+0000 7fb71033df40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b6fde8ecd0 con 0x55b6fde8fed0 2024-09-16T01:40:13.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.034+0000 7fb71033df40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55b6fde8e9f0 con 0x55b6fde92190 2024-09-16T01:40:13.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.034+0000 7fb70de33640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6fddd9f00 0x55b6fde89750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.034+0000 7fb70de33640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6fddd9f00 0x55b6fde89750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46330/0 (socket says 172.21.15.88:46330) 2024-09-16T01:40:13.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.034+0000 7fb70de33640 1 -- 172.21.15.88:0/595229646 learned_addr learned my addr 172.21.15.88:0/595229646 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:13.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.035+0000 7fb705632640 1 --2- 172.21.15.88:0/595229646 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6fde8fed0 0x55b6fdea16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.035+0000 7fb70e634640 1 --2- 172.21.15.88:0/595229646 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b6fde92190 0x55b6fde90550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.035+0000 7fb70de33640 1 -- 172.21.15.88:0/595229646 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b6fde92190 msgr2=0x55b6fde90550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.035+0000 7fb70de33640 1 --2- 172.21.15.88:0/595229646 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b6fde92190 0x55b6fde90550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.035+0000 7fb70de33640 1 -- 172.21.15.88:0/595229646 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6fde8fed0 msgr2=0x55b6fdea16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.035+0000 7fb70de33640 1 --2- 172.21.15.88:0/595229646 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6fde8fed0 0x55b6fdea16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.035+0000 7fb70de33640 1 -- 172.21.15.88:0/595229646 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6fdf3e460 con 0x55b6fddd9f00 2024-09-16T01:40:13.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.035+0000 7fb705632640 1 --2- 172.21.15.88:0/595229646 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6fde8fed0 0x55b6fdea16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:13.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.035+0000 7fb70de33640 1 --2- 172.21.15.88:0/595229646 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6fddd9f00 0x55b6fde89750 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fb700000c00 tx=0x7fb700002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.035+0000 7fb7077fe640 1 -- 172.21.15.88:0/595229646 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb700055070 con 0x55b6fddd9f00 2024-09-16T01:40:13.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.035+0000 7fb7077fe640 1 -- 172.21.15.88:0/595229646 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7000517f0 con 0x55b6fddd9f00 2024-09-16T01:40:13.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.035+0000 7fb71033df40 1 -- 172.21.15.88:0/595229646 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b6fdf3e740 con 0x55b6fddd9f00 2024-09-16T01:40:13.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.035+0000 7fb7077fe640 1 -- 172.21.15.88:0/595229646 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb700051ba0 con 0x55b6fddd9f00 2024-09-16T01:40:13.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.035+0000 7fb71033df40 1 -- 172.21.15.88:0/595229646 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6fde8d630 con 0x55b6fddd9f00 2024-09-16T01:40:13.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.036+0000 7fb7077fe640 1 -- 172.21.15.88:0/595229646 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb700058070 con 0x55b6fddd9f00 2024-09-16T01:40:13.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.037+0000 7fb7077fe640 1 --2- 172.21.15.88:0/595229646 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb6f006ac90 0x7fb6f006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.037+0000 7fb705632640 1 --2- 172.21.15.88:0/595229646 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb6f006ac90 0x7fb6f006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.037+0000 7fb7077fe640 1 -- 172.21.15.88:0/595229646 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7fb7000cdbe0 con 0x55b6fddd9f00 2024-09-16T01:40:13.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.037+0000 7fb71033df40 1 --2- 172.21.15.88:0/595229646 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b6fde22a60 0x55b6fde90a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.037+0000 7fb71033df40 1 -- 172.21.15.88:0/595229646 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:1 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55b6fde8de30 con 0x55b6fde22a60 2024-09-16T01:40:13.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.037+0000 7fb70e634640 1 --2- 172.21.15.88:0/595229646 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b6fde22a60 0x55b6fde90a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.037+0000 7fb705632640 1 --2- 172.21.15.88:0/595229646 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb6f006ac90 0x7fb6f006d150 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8005fd0 tx=0x7fb6f8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.038+0000 7fb70e634640 1 --2- 172.21.15.88:0/595229646 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b6fde22a60 0x55b6fde90a90 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.038+0000 7fb70e634640 1 -- 172.21.15.88:0/595229646 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(1 100.00000000.inode [omap-get-header] v0'0 uv3 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fb70800e070 con 0x55b6fde22a60 2024-09-16T01:40:13.043 INFO:teuthology.orchestra.run.smithi088.stdout:header (0 bytes) : 2024-09-16T01:40:13.043 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:13.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.038+0000 7fb71033df40 1 -- 172.21.15.88:0/595229646 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b6fde22a60 msgr2=0x55b6fde90a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.038+0000 7fb71033df40 1 --2- 172.21.15.88:0/595229646 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b6fde22a60 0x55b6fde90a90 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.038+0000 7fb71033df40 1 -- 172.21.15.88:0/595229646 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb6f006ac90 msgr2=0x7fb6f006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.038+0000 7fb71033df40 1 --2- 172.21.15.88:0/595229646 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb6f006ac90 0x7fb6f006d150 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8005fd0 tx=0x7fb6f8009040 comp rx=0 tx=0).stop 2024-09-16T01:40:13.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.038+0000 7fb71033df40 1 -- 172.21.15.88:0/595229646 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6fddd9f00 msgr2=0x55b6fde89750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.038+0000 7fb71033df40 1 --2- 172.21.15.88:0/595229646 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b6fddd9f00 0x55b6fde89750 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fb700000c00 tx=0x7fb700002fb0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.038+0000 7fb70e634640 1 -- 172.21.15.88:0/595229646 reap_dead start 2024-09-16T01:40:13.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.038+0000 7fb71033df40 1 -- 172.21.15.88:0/595229646 shutdown_connections 2024-09-16T01:40:13.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.038+0000 7fb71033df40 1 -- 172.21.15.88:0/595229646 >> 172.21.15.88:0/595229646 conn(0x55b6fddd6be0 msgr2=0x55b6fdea7470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:13.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.038+0000 7fb71033df40 1 -- 172.21.15.88:0/595229646 shutdown_connections 2024-09-16T01:40:13.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.038+0000 7fb71033df40 1 -- 172.21.15.88:0/595229646 wait complete. 2024-09-16T01:40:13.046 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 1.00000000 2024-09-16T01:40:13.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.126+0000 7f8a80893f40 1 Processor -- start 2024-09-16T01:40:13.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.126+0000 7f8a80893f40 1 -- start start 2024-09-16T01:40:13.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.126+0000 7f8a80893f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1ffff2dd0 0x55f2000b9240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.127+0000 7f8a80893f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f1ffff5230 0x55f2000c9680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.127+0000 7f8a80893f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f2000c1540 0x55f1ffff03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.127+0000 7f8a80893f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f1ffff31e0 con 0x55f2000c1540 2024-09-16T01:40:13.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.127+0000 7f8a80893f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f20003c5b0 con 0x55f1ffff2dd0 2024-09-16T01:40:13.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.127+0000 7f8a80893f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55f20003b610 con 0x55f1ffff5230 2024-09-16T01:40:13.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.127+0000 7f8a7eb8c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f2000c1540 0x55f1ffff03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.127+0000 7f8a7e38b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1ffff2dd0 0x55f2000b9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.127+0000 7f8a7e38b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1ffff2dd0 0x55f2000b9240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:42944/0 (socket says 172.21.15.88:42944) 2024-09-16T01:40:13.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.127+0000 7f8a7eb8c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f2000c1540 0x55f1ffff03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46356/0 (socket says 172.21.15.88:46356) 2024-09-16T01:40:13.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.127+0000 7f8a7e38b640 1 -- 172.21.15.88:0/1112495193 learned_addr learned my addr 172.21.15.88:0/1112495193 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:13.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.127+0000 7f8a75b8a640 1 --2- 172.21.15.88:0/1112495193 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f1ffff5230 0x55f2000c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.127+0000 7f8a7eb8c640 1 -- 172.21.15.88:0/1112495193 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f1ffff5230 msgr2=0x55f2000c9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:13.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.127+0000 7f8a7eb8c640 1 --2- 172.21.15.88:0/1112495193 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f1ffff5230 0x55f2000c9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.127+0000 7f8a7eb8c640 1 -- 172.21.15.88:0/1112495193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1ffff2dd0 msgr2=0x55f2000b9240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:13.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.127+0000 7f8a7eb8c640 1 --2- 172.21.15.88:0/1112495193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1ffff2dd0 0x55f2000b9240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.127+0000 7f8a7eb8c640 1 -- 172.21.15.88:0/1112495193 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1fff1ccb0 con 0x55f2000c1540 2024-09-16T01:40:13.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.127+0000 7f8a75b8a640 1 --2- 172.21.15.88:0/1112495193 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f1ffff5230 0x55f2000c9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:13.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.127+0000 7f8a7eb8c640 1 --2- 172.21.15.88:0/1112495193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f2000c1540 0x55f1ffff03a0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f8a7800b520 tx=0x7f8a7800b9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f93c2032cfdf3988 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.127+0000 7f8a7d389640 1 -- 172.21.15.88:0/1112495193 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a78014070 con 0x55f2000c1540 2024-09-16T01:40:13.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.127+0000 7f8a7d389640 1 -- 172.21.15.88:0/1112495193 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a78010800 con 0x55f2000c1540 2024-09-16T01:40:13.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.127+0000 7f8a7d389640 1 -- 172.21.15.88:0/1112495193 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a78010bf0 con 0x55f2000c1540 2024-09-16T01:40:13.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.128+0000 7f8a80893f40 1 -- 172.21.15.88:0/1112495193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f2000c1540 msgr2=0x55f1ffff03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.128+0000 7f8a80893f40 1 --2- 172.21.15.88:0/1112495193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f2000c1540 0x55f1ffff03a0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f8a7800b520 tx=0x7f8a7800b9e0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.128+0000 7f8a80893f40 1 -- 172.21.15.88:0/1112495193 shutdown_connections 2024-09-16T01:40:13.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.128+0000 7f8a80893f40 1 --2- 172.21.15.88:0/1112495193 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f2000c1540 0x55f1ffff03a0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.128+0000 7f8a80893f40 1 --2- 172.21.15.88:0/1112495193 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f1ffff5230 0x55f2000c9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.128+0000 7f8a80893f40 1 --2- 172.21.15.88:0/1112495193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1ffff2dd0 0x55f2000b9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.128+0000 7f8a80893f40 1 -- 172.21.15.88:0/1112495193 >> 172.21.15.88:0/1112495193 conn(0x55f20003c750 msgr2=0x55f2000c8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:13.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.128+0000 7f8a80893f40 1 -- 172.21.15.88:0/1112495193 shutdown_connections 2024-09-16T01:40:13.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.128+0000 7f8a80893f40 1 -- 172.21.15.88:0/1112495193 wait complete. 2024-09-16T01:40:13.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.128+0000 7f8a80893f40 1 Processor -- start 2024-09-16T01:40:13.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.128+0000 7f8a80893f40 1 -- start start 2024-09-16T01:40:13.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.129+0000 7f8a80893f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f1ffff2dd0 0x55f1ffff1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.129+0000 7f8a80893f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f1ffff5230 0x55f2000aa8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.129+0000 7f8a80893f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2000c1540 0x55f2000aadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.129+0000 7f8a80893f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f2000c6ff0 con 0x55f1ffff5230 2024-09-16T01:40:13.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.129+0000 7f8a80893f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f2000bc6d0 con 0x55f2000c1540 2024-09-16T01:40:13.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.129+0000 7f8a80893f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55f2000ca2c0 con 0x55f1ffff2dd0 2024-09-16T01:40:13.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.129+0000 7f8a75b8a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f1ffff5230 0x55f2000aa8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.129+0000 7f8a75b8a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f1ffff5230 0x55f2000aa8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46358/0 (socket says 172.21.15.88:46358) 2024-09-16T01:40:13.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.129+0000 7f8a75b8a640 1 -- 172.21.15.88:0/3082781789 learned_addr learned my addr 172.21.15.88:0/3082781789 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:13.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.129+0000 7f8a7e38b640 1 --2- 172.21.15.88:0/3082781789 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f1ffff2dd0 0x55f1ffff1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.129+0000 7f8a7eb8c640 1 --2- 172.21.15.88:0/3082781789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2000c1540 0x55f2000aadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.129+0000 7f8a75b8a640 1 -- 172.21.15.88:0/3082781789 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f1ffff2dd0 msgr2=0x55f1ffff1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.129+0000 7f8a75b8a640 1 --2- 172.21.15.88:0/3082781789 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f1ffff2dd0 0x55f1ffff1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.129+0000 7f8a75b8a640 1 -- 172.21.15.88:0/3082781789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2000c1540 msgr2=0x55f2000aadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.129+0000 7f8a75b8a640 1 --2- 172.21.15.88:0/3082781789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f2000c1540 0x55f2000aadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.129+0000 7f8a75b8a640 1 -- 172.21.15.88:0/3082781789 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1ffff1fb0 con 0x55f1ffff5230 2024-09-16T01:40:13.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.130+0000 7f8a75b8a640 1 --2- 172.21.15.88:0/3082781789 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f1ffff5230 0x55f2000aa8b0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f8a680460d0 tx=0x7f8a68002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.130+0000 7f8a77fff640 1 -- 172.21.15.88:0/3082781789 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68055040 con 0x55f1ffff5230 2024-09-16T01:40:13.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.130+0000 7f8a77fff640 1 -- 172.21.15.88:0/3082781789 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a68051970 con 0x55f1ffff5230 2024-09-16T01:40:13.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.130+0000 7f8a80893f40 1 -- 172.21.15.88:0/3082781789 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f2000a9af0 con 0x55f1ffff5230 2024-09-16T01:40:13.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.130+0000 7f8a77fff640 1 -- 172.21.15.88:0/3082781789 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68051d60 con 0x55f1ffff5230 2024-09-16T01:40:13.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.130+0000 7f8a80893f40 1 -- 172.21.15.88:0/3082781789 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2000aa0b0 con 0x55f1ffff5230 2024-09-16T01:40:13.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.131+0000 7f8a77fff640 1 -- 172.21.15.88:0/3082781789 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8a68058070 con 0x55f1ffff5230 2024-09-16T01:40:13.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.132+0000 7f8a77fff640 1 --2- 172.21.15.88:0/3082781789 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8a6006acc0 0x7f8a6006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.132+0000 7f8a77fff640 1 -- 172.21.15.88:0/3082781789 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f8a680cdce0 con 0x55f1ffff5230 2024-09-16T01:40:13.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.132+0000 7f8a7e38b640 1 --2- 172.21.15.88:0/3082781789 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8a6006acc0 0x7f8a6006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.132+0000 7f8a80893f40 1 --2- 172.21.15.88:0/3082781789 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f2000ba620 0x55f2000baa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.132+0000 7f8a80893f40 1 -- 172.21.15.88:0/3082781789 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:1 12.1f 12:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55f2000baf40 con 0x55f2000ba620 2024-09-16T01:40:13.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.132+0000 7f8a7e38b640 1 --2- 172.21.15.88:0/3082781789 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8a6006acc0 0x7f8a6006d180 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8a70009700 tx=0x7f8a7000d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.132+0000 7f8a7eb8c640 1 --2- 172.21.15.88:0/3082781789 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f2000ba620 0x55f2000baa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.133+0000 7f8a7eb8c640 1 --2- 172.21.15.88:0/3082781789 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f2000ba620 0x55f2000baa00 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.133+0000 7f8a7eb8c640 1 -- 172.21.15.88:0/3082781789 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(1 1.00000000 [omap-get-header out=274b] v0'0 uv6 ondisk = 0) ==== 154+0+274 (crc 0 0 0) 0x7f8a78010a30 con 0x55f2000ba620 2024-09-16T01:40:13.139 INFO:teuthology.orchestra.run.smithi088.stdout:header (274 bytes) : 2024-09-16T01:40:13.139 INFO:teuthology.orchestra.run.smithi088.stdout:00000000 04 03 0c 01 00 00 0d 00 00 00 00 00 00 00 00 00 |................| 2024-09-16T01:40:13.139 INFO:teuthology.orchestra.run.smithi088.stdout:00000010 00 00 00 00 00 00 03 02 28 00 00 00 00 00 00 00 |........(.......| 2024-09-16T01:40:13.139 INFO:teuthology.orchestra.run.smithi088.stdout:00000020 00 00 00 00 79 8c e7 66 2c 9f ed 37 00 00 00 00 |....y..f,..7....| 2024-09-16T01:40:13.139 INFO:teuthology.orchestra.run.smithi088.stdout:00000030 00 00 00 00 01 00 00 00 00 00 00 00 01 00 00 00 |................| 2024-09-16T01:40:13.139 INFO:teuthology.orchestra.run.smithi088.stdout:00000040 00 00 00 00 03 02 28 00 00 00 00 00 00 00 00 00 |......(.........| 2024-09-16T01:40:13.139 INFO:teuthology.orchestra.run.smithi088.stdout:00000050 00 00 79 8c e7 66 2c 9f ed 37 00 00 00 00 00 00 |..y..f,..7......| 2024-09-16T01:40:13.139 INFO:teuthology.orchestra.run.smithi088.stdout:00000060 00 00 01 00 00 00 00 00 00 00 01 00 00 00 00 00 |................| 2024-09-16T01:40:13.139 INFO:teuthology.orchestra.run.smithi088.stdout:00000070 00 00 03 02 38 00 00 00 00 00 00 00 00 00 00 00 |....8...........| 2024-09-16T01:40:13.139 INFO:teuthology.orchestra.run.smithi088.stdout:00000080 00 00 60 00 00 00 00 00 01 00 00 00 00 00 00 00 |..`.............| 2024-09-16T01:40:13.139 INFO:teuthology.orchestra.run.smithi088.stdout:00000090 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-16T01:40:13.139 INFO:teuthology.orchestra.run.smithi088.stdout:000000a0 00 00 00 00 00 00 00 00 7a 8c e7 66 77 ba 5e 04 |........z..fw.^.| 2024-09-16T01:40:13.139 INFO:teuthology.orchestra.run.smithi088.stdout:000000b0 03 02 38 00 00 00 00 00 00 00 00 00 00 00 00 00 |..8.............| 2024-09-16T01:40:13.140 INFO:teuthology.orchestra.run.smithi088.stdout:000000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 |................| 2024-09-16T01:40:13.140 INFO:teuthology.orchestra.run.smithi088.stdout:000000d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-16T01:40:13.140 INFO:teuthology.orchestra.run.smithi088.stdout:000000e0 00 00 00 00 00 00 79 8c e7 66 2c 9f ed 37 00 00 |......y..f,..7..| 2024-09-16T01:40:13.140 INFO:teuthology.orchestra.run.smithi088.stdout:000000f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-16T01:40:13.140 INFO:teuthology.orchestra.run.smithi088.stdout:* 2024-09-16T01:40:13.140 INFO:teuthology.orchestra.run.smithi088.stdout:00000110 00 00 |..| 2024-09-16T01:40:13.140 INFO:teuthology.orchestra.run.smithi088.stdout:00000112 2024-09-16T01:40:13.140 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:13.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.133+0000 7f8a80893f40 1 -- 172.21.15.88:0/3082781789 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f2000ba620 msgr2=0x55f2000baa00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.133+0000 7f8a80893f40 1 --2- 172.21.15.88:0/3082781789 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f2000ba620 0x55f2000baa00 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.133+0000 7f8a80893f40 1 -- 172.21.15.88:0/3082781789 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8a6006acc0 msgr2=0x7f8a6006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.133+0000 7f8a80893f40 1 --2- 172.21.15.88:0/3082781789 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8a6006acc0 0x7f8a6006d180 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8a70009700 tx=0x7f8a7000d040 comp rx=0 tx=0).stop 2024-09-16T01:40:13.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.133+0000 7f8a80893f40 1 -- 172.21.15.88:0/3082781789 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f1ffff5230 msgr2=0x55f2000aa8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.133+0000 7f8a80893f40 1 --2- 172.21.15.88:0/3082781789 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f1ffff5230 0x55f2000aa8b0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f8a680460d0 tx=0x7f8a68002fb0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.133+0000 7f8a7eb8c640 1 -- 172.21.15.88:0/3082781789 reap_dead start 2024-09-16T01:40:13.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.133+0000 7f8a80893f40 1 -- 172.21.15.88:0/3082781789 shutdown_connections 2024-09-16T01:40:13.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.133+0000 7f8a80893f40 1 -- 172.21.15.88:0/3082781789 >> 172.21.15.88:0/3082781789 conn(0x55f20003c750 msgr2=0x55f2000c9910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:13.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.133+0000 7f8a80893f40 1 -- 172.21.15.88:0/3082781789 shutdown_connections 2024-09-16T01:40:13.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.133+0000 7f8a80893f40 1 -- 172.21.15.88:0/3082781789 wait complete. 2024-09-16T01:40:13.142 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 10000000000.00000000 2024-09-16T01:40:13.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.223+0000 7f24a3f25f40 1 Processor -- start 2024-09-16T01:40:13.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.223+0000 7f24a3f25f40 1 -- start start 2024-09-16T01:40:13.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.224+0000 7f24a3f25f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594ad1eee10 0x5594ad299b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.224+0000 7f24a3f25f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5594ad1efa60 0x5594ad28a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.224+0000 7f24a3f25f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5594ad209250 0x5594ad289020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.224+0000 7f24a3f25f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5594ad111510 con 0x5594ad1efa60 2024-09-16T01:40:13.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.224+0000 7f24a3f25f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5594ad16ac50 con 0x5594ad1eee10 2024-09-16T01:40:13.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.224+0000 7f24a3f25f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5594ad1ebf20 con 0x5594ad209250 2024-09-16T01:40:13.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.224+0000 7f249bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5594ad1efa60 0x5594ad28a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.224+0000 7f249bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5594ad1efa60 0x5594ad28a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46368/0 (socket says 172.21.15.88:46368) 2024-09-16T01:40:13.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.224+0000 7f249bfff640 1 -- 172.21.15.88:0/4268909474 learned_addr learned my addr 172.21.15.88:0/4268909474 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:13.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.224+0000 7f24a0f4e640 1 --2- 172.21.15.88:0/4268909474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594ad1eee10 0x5594ad299b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.224+0000 7f24a174f640 1 --2- 172.21.15.88:0/4268909474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5594ad209250 0x5594ad289020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.224+0000 7f249bfff640 1 -- 172.21.15.88:0/4268909474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5594ad209250 msgr2=0x5594ad289020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:13.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.224+0000 7f249bfff640 1 --2- 172.21.15.88:0/4268909474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5594ad209250 0x5594ad289020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.224+0000 7f249bfff640 1 -- 172.21.15.88:0/4268909474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594ad1eee10 msgr2=0x5594ad299b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:13.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.224+0000 7f249bfff640 1 --2- 172.21.15.88:0/4268909474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594ad1eee10 0x5594ad299b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.224+0000 7f249bfff640 1 -- 172.21.15.88:0/4268909474 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594ad291680 con 0x5594ad1efa60 2024-09-16T01:40:13.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.224+0000 7f24a0f4e640 1 --2- 172.21.15.88:0/4268909474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594ad1eee10 0x5594ad299b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:13.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.224+0000 7f249bfff640 1 --2- 172.21.15.88:0/4268909474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5594ad1efa60 0x5594ad28a120 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f248c00a520 tx=0x7f248c051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4992da1d6a117718 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.225+0000 7f249affd640 1 -- 172.21.15.88:0/4268909474 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248c0537a0 con 0x5594ad1efa60 2024-09-16T01:40:13.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.225+0000 7f249affd640 1 -- 172.21.15.88:0/4268909474 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f248c053940 con 0x5594ad1efa60 2024-09-16T01:40:13.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.225+0000 7f249affd640 1 -- 172.21.15.88:0/4268909474 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248c053d10 con 0x5594ad1efa60 2024-09-16T01:40:13.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.225+0000 7f24a3f25f40 1 -- 172.21.15.88:0/4268909474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5594ad1efa60 msgr2=0x5594ad28a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.225+0000 7f24a3f25f40 1 --2- 172.21.15.88:0/4268909474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5594ad1efa60 0x5594ad28a120 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f248c00a520 tx=0x7f248c051000 comp rx=0 tx=0).stop 2024-09-16T01:40:13.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.225+0000 7f24a3f25f40 1 -- 172.21.15.88:0/4268909474 shutdown_connections 2024-09-16T01:40:13.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.225+0000 7f24a3f25f40 1 --2- 172.21.15.88:0/4268909474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5594ad209250 0x5594ad289020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.225+0000 7f24a3f25f40 1 --2- 172.21.15.88:0/4268909474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5594ad1efa60 0x5594ad28a120 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.225+0000 7f24a3f25f40 1 --2- 172.21.15.88:0/4268909474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594ad1eee10 0x5594ad299b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.225+0000 7f24a3f25f40 1 -- 172.21.15.88:0/4268909474 >> 172.21.15.88:0/4268909474 conn(0x5594ad20a750 msgr2=0x5594ad293e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:13.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.225+0000 7f24a3f25f40 1 -- 172.21.15.88:0/4268909474 shutdown_connections 2024-09-16T01:40:13.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.225+0000 7f24a3f25f40 1 -- 172.21.15.88:0/4268909474 wait complete. 2024-09-16T01:40:13.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.226+0000 7f24a3f25f40 1 Processor -- start 2024-09-16T01:40:13.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.226+0000 7f24a3f25f40 1 -- start start 2024-09-16T01:40:13.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.226+0000 7f24a3f25f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5594ad1eee10 0x5594ad2834f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.226+0000 7f24a3f25f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5594ad1efa60 0x5594ad28ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.226+0000 7f24a3f25f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594ad209250 0x5594ad28b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.226+0000 7f24a3f25f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5594ad1ec1c0 con 0x5594ad1efa60 2024-09-16T01:40:13.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.226+0000 7f24a3f25f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5594ad29a9e0 con 0x5594ad209250 2024-09-16T01:40:13.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.226+0000 7f24a3f25f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5594ad28d2a0 con 0x5594ad1eee10 2024-09-16T01:40:13.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.227+0000 7f249bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5594ad1efa60 0x5594ad28ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.227+0000 7f249bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5594ad1efa60 0x5594ad28ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46374/0 (socket says 172.21.15.88:46374) 2024-09-16T01:40:13.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.227+0000 7f249bfff640 1 -- 172.21.15.88:0/1027264365 learned_addr learned my addr 172.21.15.88:0/1027264365 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:13.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.227+0000 7f24a0f4e640 1 --2- 172.21.15.88:0/1027264365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5594ad1eee10 0x5594ad2834f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.227+0000 7f24a174f640 1 --2- 172.21.15.88:0/1027264365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594ad209250 0x5594ad28b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.227+0000 7f249bfff640 1 -- 172.21.15.88:0/1027264365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5594ad1eee10 msgr2=0x5594ad2834f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.227+0000 7f249bfff640 1 --2- 172.21.15.88:0/1027264365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5594ad1eee10 0x5594ad2834f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.227+0000 7f249bfff640 1 -- 172.21.15.88:0/1027264365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594ad209250 msgr2=0x5594ad28b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.227+0000 7f249bfff640 1 --2- 172.21.15.88:0/1027264365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594ad209250 0x5594ad28b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.227+0000 7f249bfff640 1 -- 172.21.15.88:0/1027264365 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594ad291ea0 con 0x5594ad1efa60 2024-09-16T01:40:13.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.227+0000 7f24a174f640 1 --2- 172.21.15.88:0/1027264365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594ad209250 0x5594ad28b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:13.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.227+0000 7f249bfff640 1 --2- 172.21.15.88:0/1027264365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5594ad1efa60 0x5594ad28ace0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f248c025c40 tx=0x7f248c002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.227+0000 7f2499ffb640 1 -- 172.21.15.88:0/1027264365 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248c075070 con 0x5594ad1efa60 2024-09-16T01:40:13.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.227+0000 7f2499ffb640 1 -- 172.21.15.88:0/1027264365 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f248c053af0 con 0x5594ad1efa60 2024-09-16T01:40:13.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.227+0000 7f24a3f25f40 1 -- 172.21.15.88:0/1027264365 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5594ad28f090 con 0x5594ad1efa60 2024-09-16T01:40:13.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.227+0000 7f2499ffb640 1 -- 172.21.15.88:0/1027264365 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248c070480 con 0x5594ad1efa60 2024-09-16T01:40:13.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.227+0000 7f24a3f25f40 1 -- 172.21.15.88:0/1027264365 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594ad29a420 con 0x5594ad1efa60 2024-09-16T01:40:13.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.229+0000 7f2499ffb640 1 -- 172.21.15.88:0/1027264365 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f248c078070 con 0x5594ad1efa60 2024-09-16T01:40:13.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.229+0000 7f2499ffb640 1 --2- 172.21.15.88:0/1027264365 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f248406acc0 0x7f248406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.229+0000 7f2499ffb640 1 -- 172.21.15.88:0/1027264365 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f248c0cf580 con 0x5594ad1efa60 2024-09-16T01:40:13.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.229+0000 7f24a0f4e640 1 --2- 172.21.15.88:0/1027264365 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f248406acc0 0x7f248406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.229+0000 7f24a3f25f40 1 --2- 172.21.15.88:0/1027264365 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5594ad28da20 0x5594ad28de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.229+0000 7f24a3f25f40 1 -- 172.21.15.88:0/1027264365 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5594ad283a30 con 0x5594ad28da20 2024-09-16T01:40:13.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.229+0000 7f24a174f640 1 --2- 172.21.15.88:0/1027264365 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5594ad28da20 0x5594ad28de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.229+0000 7f24a0f4e640 1 --2- 172.21.15.88:0/1027264365 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f248406acc0 0x7f248406d180 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2494000c00 tx=0x7f2494003820 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.230+0000 7f24a174f640 1 --2- 172.21.15.88:0/1027264365 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5594ad28da20 0x5594ad28de30 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.230+0000 7f24a174f640 1 -- 172.21.15.88:0/1027264365 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 10000000000.00000000 [omap-get-keys out=21b] v0'0 uv10 ondisk = 0) ==== 164+0+21 (crc 0 0 0) 0x7f249c0044c0 con 0x5594ad28da20 2024-09-16T01:40:13.238 INFO:teuthology.orchestra.run.smithi088.stdout:sixmegs_head 2024-09-16T01:40:13.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.230+0000 7f24a3f25f40 1 -- 172.21.15.88:0/1027264365 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5594ad28da20 msgr2=0x5594ad28de30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.230+0000 7f24a3f25f40 1 --2- 172.21.15.88:0/1027264365 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5594ad28da20 0x5594ad28de30 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.230+0000 7f24a3f25f40 1 -- 172.21.15.88:0/1027264365 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f248406acc0 msgr2=0x7f248406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.230+0000 7f24a3f25f40 1 --2- 172.21.15.88:0/1027264365 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f248406acc0 0x7f248406d180 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2494000c00 tx=0x7f2494003820 comp rx=0 tx=0).stop 2024-09-16T01:40:13.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.230+0000 7f24a3f25f40 1 -- 172.21.15.88:0/1027264365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5594ad1efa60 msgr2=0x5594ad28ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.230+0000 7f24a3f25f40 1 --2- 172.21.15.88:0/1027264365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5594ad1efa60 0x5594ad28ace0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f248c025c40 tx=0x7f248c002fb0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.230+0000 7f24a174f640 1 -- 172.21.15.88:0/1027264365 reap_dead start 2024-09-16T01:40:13.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.230+0000 7f24a3f25f40 1 -- 172.21.15.88:0/1027264365 shutdown_connections 2024-09-16T01:40:13.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.230+0000 7f24a3f25f40 1 -- 172.21.15.88:0/1027264365 >> 172.21.15.88:0/1027264365 conn(0x5594ad20a750 msgr2=0x5594ad296300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:13.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.230+0000 7f24a3f25f40 1 -- 172.21.15.88:0/1027264365 shutdown_connections 2024-09-16T01:40:13.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.230+0000 7f24a3f25f40 1 -- 172.21.15.88:0/1027264365 wait complete. 2024-09-16T01:40:13.241 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 600.00000000 2024-09-16T01:40:13.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.322+0000 7f0d46565f40 1 Processor -- start 2024-09-16T01:40:13.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.322+0000 7f0d46565f40 1 -- start start 2024-09-16T01:40:13.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.322+0000 7f0d46565f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d300022a0 0x55ed572e0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.322+0000 7f0d46565f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ed572c76f0 0x55ed57368320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.322+0000 7f0d46565f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed572c47a0 0x55ed572c4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.322+0000 7f0d46565f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55ed571ea510 con 0x55ed572c47a0 2024-09-16T01:40:13.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.322+0000 7f0d46565f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ed57243c50 con 0x7f0d300022a0 2024-09-16T01:40:13.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.322+0000 7f0d46565f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55ed57368be0 con 0x55ed572c76f0 2024-09-16T01:40:13.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.323+0000 7f0d4485c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed572c47a0 0x55ed572c4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.323+0000 7f0d4485c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed572c47a0 0x55ed572c4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46394/0 (socket says 172.21.15.88:46394) 2024-09-16T01:40:13.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.323+0000 7f0d4485c640 1 -- 172.21.15.88:0/1678169922 learned_addr learned my addr 172.21.15.88:0/1678169922 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:13.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.323+0000 7f0d3b85a640 1 --2- 172.21.15.88:0/1678169922 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ed572c76f0 0x55ed57368320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.323+0000 7f0d4405b640 1 --2- 172.21.15.88:0/1678169922 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d300022a0 0x55ed572e0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.323+0000 7f0d4485c640 1 -- 172.21.15.88:0/1678169922 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ed572c76f0 msgr2=0x55ed57368320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:13.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.323+0000 7f0d4485c640 1 --2- 172.21.15.88:0/1678169922 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ed572c76f0 0x55ed57368320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.323+0000 7f0d4485c640 1 -- 172.21.15.88:0/1678169922 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d300022a0 msgr2=0x55ed572e0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:13.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.323+0000 7f0d4485c640 1 --2- 172.21.15.88:0/1678169922 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d300022a0 0x55ed572e0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.323+0000 7f0d4485c640 1 -- 172.21.15.88:0/1678169922 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed57368860 con 0x55ed572c47a0 2024-09-16T01:40:13.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.323+0000 7f0d4485c640 1 --2- 172.21.15.88:0/1678169922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed572c47a0 0x55ed572c4c50 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x55ed5729c260 tx=0x7f0d3c004830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab0ccf70ffdd864f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.323+0000 7f0d43059640 1 -- 172.21.15.88:0/1678169922 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c0019d0 con 0x55ed572c47a0 2024-09-16T01:40:13.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.323+0000 7f0d43059640 1 -- 172.21.15.88:0/1678169922 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d3c00f450 con 0x55ed572c47a0 2024-09-16T01:40:13.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.323+0000 7f0d43059640 1 -- 172.21.15.88:0/1678169922 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c00f820 con 0x55ed572c47a0 2024-09-16T01:40:13.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.324+0000 7f0d46565f40 1 -- 172.21.15.88:0/1678169922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed572c47a0 msgr2=0x55ed572c4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.324+0000 7f0d46565f40 1 --2- 172.21.15.88:0/1678169922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed572c47a0 0x55ed572c4c50 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x55ed5729c260 tx=0x7f0d3c004830 comp rx=0 tx=0).stop 2024-09-16T01:40:13.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.324+0000 7f0d46565f40 1 -- 172.21.15.88:0/1678169922 shutdown_connections 2024-09-16T01:40:13.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.324+0000 7f0d46565f40 1 --2- 172.21.15.88:0/1678169922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed572c47a0 0x55ed572c4c50 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.324+0000 7f0d46565f40 1 --2- 172.21.15.88:0/1678169922 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ed572c76f0 0x55ed57368320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.324+0000 7f0d46565f40 1 --2- 172.21.15.88:0/1678169922 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d300022a0 0x55ed572e0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.324+0000 7f0d46565f40 1 -- 172.21.15.88:0/1678169922 >> 172.21.15.88:0/1678169922 conn(0x55ed572e3750 msgr2=0x55ed572c6fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:13.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.324+0000 7f0d46565f40 1 -- 172.21.15.88:0/1678169922 shutdown_connections 2024-09-16T01:40:13.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.324+0000 7f0d46565f40 1 -- 172.21.15.88:0/1678169922 wait complete. 2024-09-16T01:40:13.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.324+0000 7f0d46565f40 1 Processor -- start 2024-09-16T01:40:13.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.324+0000 7f0d46565f40 1 -- start start 2024-09-16T01:40:13.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.325+0000 7f0d46565f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ed572c47a0 0x55ed573623c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.325+0000 7f0d46565f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed572c76f0 0x55ed572c3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.325+0000 7f0d46565f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d300022a0 0x55ed572c3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.325+0000 7f0d46565f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55ed5736dcb0 con 0x55ed572c76f0 2024-09-16T01:40:13.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.325+0000 7f0d46565f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ed572c3fb0 con 0x7f0d300022a0 2024-09-16T01:40:13.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.325+0000 7f0d46565f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55ed572e18e0 con 0x55ed572c47a0 2024-09-16T01:40:13.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.325+0000 7f0d3b85a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed572c76f0 0x55ed572c3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.325+0000 7f0d3b85a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed572c76f0 0x55ed572c3400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46400/0 (socket says 172.21.15.88:46400) 2024-09-16T01:40:13.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.325+0000 7f0d3b85a640 1 -- 172.21.15.88:0/981894740 learned_addr learned my addr 172.21.15.88:0/981894740 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:13.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.325+0000 7f0d4405b640 1 --2- 172.21.15.88:0/981894740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ed572c47a0 0x55ed573623c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.325+0000 7f0d4485c640 1 --2- 172.21.15.88:0/981894740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d300022a0 0x55ed572c3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.325+0000 7f0d3b85a640 1 -- 172.21.15.88:0/981894740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ed572c47a0 msgr2=0x55ed573623c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.325+0000 7f0d3b85a640 1 --2- 172.21.15.88:0/981894740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ed572c47a0 0x55ed573623c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.325+0000 7f0d3b85a640 1 -- 172.21.15.88:0/981894740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d300022a0 msgr2=0x55ed572c3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.325+0000 7f0d3b85a640 1 --2- 172.21.15.88:0/981894740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d300022a0 0x55ed572c3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.325+0000 7f0d3b85a640 1 -- 172.21.15.88:0/981894740 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed5736eb60 con 0x55ed572c76f0 2024-09-16T01:40:13.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.325+0000 7f0d4485c640 1 --2- 172.21.15.88:0/981894740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d300022a0 0x55ed572c3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:13.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.326+0000 7f0d3b85a640 1 --2- 172.21.15.88:0/981894740 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed572c76f0 0x55ed572c3400 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x55ed57286300 tx=0x7f0d2c009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.326+0000 7f0d42057640 1 -- 172.21.15.88:0/981894740 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c01a020 con 0x55ed572c76f0 2024-09-16T01:40:13.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.326+0000 7f0d42057640 1 -- 172.21.15.88:0/981894740 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d2c016450 con 0x55ed572c76f0 2024-09-16T01:40:13.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.326+0000 7f0d46565f40 1 -- 172.21.15.88:0/981894740 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ed574072c0 con 0x55ed572c76f0 2024-09-16T01:40:13.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.326+0000 7f0d42057640 1 -- 172.21.15.88:0/981894740 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c016820 con 0x55ed572c76f0 2024-09-16T01:40:13.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.326+0000 7f0d46565f40 1 -- 172.21.15.88:0/981894740 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed57362cd0 con 0x55ed572c76f0 2024-09-16T01:40:13.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.327+0000 7f0d42057640 1 -- 172.21.15.88:0/981894740 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0d2c01d020 con 0x55ed572c76f0 2024-09-16T01:40:13.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.327+0000 7f0d42057640 1 --2- 172.21.15.88:0/981894740 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0d2406ac90 0x7f0d2406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.328+0000 7f0d42057640 1 -- 172.21.15.88:0/981894740 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f0d2c08a700 con 0x55ed572c76f0 2024-09-16T01:40:13.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.328+0000 7f0d4405b640 1 --2- 172.21.15.88:0/981894740 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0d2406ac90 0x7f0d2406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.328+0000 7f0d46565f40 1 --2- 172.21.15.88:0/981894740 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55ed5736f040 0x55ed5736f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.328+0000 7f0d46565f40 1 -- 172.21.15.88:0/981894740 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:0d82a743:::600.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55ed5736f960 con 0x55ed5736f040 2024-09-16T01:40:13.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.328+0000 7f0d4485c640 1 --2- 172.21.15.88:0/981894740 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55ed5736f040 0x55ed5736f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.328+0000 7f0d4405b640 1 --2- 172.21.15.88:0/981894740 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0d2406ac90 0x7f0d2406d150 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f0d34054000 tx=0x7f0d34050000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.328+0000 7f0d4485c640 1 --2- 172.21.15.88:0/981894740 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55ed5736f040 0x55ed5736f420 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.328+0000 7f0d4485c640 1 -- 172.21.15.88:0/981894740 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 600.00000000 [omap-get-keys out=5b] v0'0 uv2 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f0d3c00f700 con 0x55ed5736f040 2024-09-16T01:40:13.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.328+0000 7f0d46565f40 1 -- 172.21.15.88:0/981894740 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55ed5736f040 msgr2=0x55ed5736f420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.328+0000 7f0d46565f40 1 --2- 172.21.15.88:0/981894740 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55ed5736f040 0x55ed5736f420 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.329+0000 7f0d46565f40 1 -- 172.21.15.88:0/981894740 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0d2406ac90 msgr2=0x7f0d2406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.329+0000 7f0d46565f40 1 --2- 172.21.15.88:0/981894740 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0d2406ac90 0x7f0d2406d150 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f0d34054000 tx=0x7f0d34050000 comp rx=0 tx=0).stop 2024-09-16T01:40:13.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.329+0000 7f0d46565f40 1 -- 172.21.15.88:0/981894740 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed572c76f0 msgr2=0x55ed572c3400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.329+0000 7f0d46565f40 1 --2- 172.21.15.88:0/981894740 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ed572c76f0 0x55ed572c3400 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x55ed57286300 tx=0x7f0d2c009e90 comp rx=0 tx=0).stop 2024-09-16T01:40:13.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.329+0000 7f0d4485c640 1 -- 172.21.15.88:0/981894740 reap_dead start 2024-09-16T01:40:13.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.329+0000 7f0d46565f40 1 -- 172.21.15.88:0/981894740 shutdown_connections 2024-09-16T01:40:13.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.329+0000 7f0d46565f40 1 -- 172.21.15.88:0/981894740 >> 172.21.15.88:0/981894740 conn(0x55ed572e3750 msgr2=0x55ed572c6590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:13.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.329+0000 7f0d46565f40 1 -- 172.21.15.88:0/981894740 shutdown_connections 2024-09-16T01:40:13.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.329+0000 7f0d46565f40 1 -- 172.21.15.88:0/981894740 wait complete. 2024-09-16T01:40:13.336 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 1.00000000.inode 2024-09-16T01:40:13.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.415+0000 7fa551bb4f40 1 Processor -- start 2024-09-16T01:40:13.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.415+0000 7fa551bb4f40 1 -- start start 2024-09-16T01:40:13.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.416+0000 7fa551bb4f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556922d0c700 0x556922d8abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.416+0000 7fa551bb4f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556922d0cae0 0x556922d95340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.416+0000 7fa551bb4f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556922d9b740 0x556922d9a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.416+0000 7fa551bb4f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x556922cc4a70 con 0x556922d0cae0 2024-09-16T01:40:13.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.416+0000 7fa551bb4f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556922c14510 con 0x556922d0c700 2024-09-16T01:40:13.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.416+0000 7fa551bb4f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x556922d0a0a0 con 0x556922d9b740 2024-09-16T01:40:13.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.416+0000 7fa546ea9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556922d0cae0 0x556922d95340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.416+0000 7fa546ea9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556922d0cae0 0x556922d95340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46420/0 (socket says 172.21.15.88:46420) 2024-09-16T01:40:13.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.416+0000 7fa546ea9640 1 -- 172.21.15.88:0/823392523 learned_addr learned my addr 172.21.15.88:0/823392523 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:13.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.416+0000 7fa54f6aa640 1 --2- 172.21.15.88:0/823392523 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556922d0c700 0x556922d8abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.416+0000 7fa54feab640 1 --2- 172.21.15.88:0/823392523 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556922d9b740 0x556922d9a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.416+0000 7fa546ea9640 1 -- 172.21.15.88:0/823392523 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556922d9b740 msgr2=0x556922d9a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:13.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.416+0000 7fa546ea9640 1 --2- 172.21.15.88:0/823392523 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556922d9b740 0x556922d9a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.416+0000 7fa546ea9640 1 -- 172.21.15.88:0/823392523 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556922d0c700 msgr2=0x556922d8abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:13.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.416+0000 7fa546ea9640 1 --2- 172.21.15.88:0/823392523 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556922d0c700 0x556922d8abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.416+0000 7fa546ea9640 1 -- 172.21.15.88:0/823392523 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556922d95880 con 0x556922d0cae0 2024-09-16T01:40:13.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.416+0000 7fa546ea9640 1 --2- 172.21.15.88:0/823392523 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556922d0cae0 0x556922d95340 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fa538001da0 tx=0x7fa53800b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4068845019ec77ca server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.416+0000 7fa54e6a8640 1 -- 172.21.15.88:0/823392523 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa53800bde0 con 0x556922d0cae0 2024-09-16T01:40:13.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.417+0000 7fa54e6a8640 1 -- 172.21.15.88:0/823392523 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa53800f400 con 0x556922d0cae0 2024-09-16T01:40:13.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.417+0000 7fa54e6a8640 1 -- 172.21.15.88:0/823392523 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa53800f830 con 0x556922d0cae0 2024-09-16T01:40:13.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.417+0000 7fa551bb4f40 1 -- 172.21.15.88:0/823392523 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556922d0cae0 msgr2=0x556922d95340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.417+0000 7fa551bb4f40 1 --2- 172.21.15.88:0/823392523 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556922d0cae0 0x556922d95340 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fa538001da0 tx=0x7fa53800b5b0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.417+0000 7fa551bb4f40 1 -- 172.21.15.88:0/823392523 shutdown_connections 2024-09-16T01:40:13.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.417+0000 7fa551bb4f40 1 --2- 172.21.15.88:0/823392523 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556922d9b740 0x556922d9a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.417+0000 7fa551bb4f40 1 --2- 172.21.15.88:0/823392523 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556922d0cae0 0x556922d95340 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.417+0000 7fa551bb4f40 1 --2- 172.21.15.88:0/823392523 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556922d0c700 0x556922d8abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.417+0000 7fa551bb4f40 1 -- 172.21.15.88:0/823392523 >> 172.21.15.88:0/823392523 conn(0x556922ced680 msgr2=0x556922cf2930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:13.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.417+0000 7fa551bb4f40 1 -- 172.21.15.88:0/823392523 shutdown_connections 2024-09-16T01:40:13.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.417+0000 7fa551bb4f40 1 -- 172.21.15.88:0/823392523 wait complete. 2024-09-16T01:40:13.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.418+0000 7fa551bb4f40 1 Processor -- start 2024-09-16T01:40:13.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.418+0000 7fa551bb4f40 1 -- start start 2024-09-16T01:40:13.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.418+0000 7fa551bb4f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556922d0c700 0x556922e31500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.418+0000 7fa551bb4f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556922d0cae0 0x556922e2e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.418+0000 7fa551bb4f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556922d9b740 0x556922d891c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.418+0000 7fa551bb4f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x556922cc11c0 con 0x556922d0cae0 2024-09-16T01:40:13.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.418+0000 7fa551bb4f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556922d98230 con 0x556922d9b740 2024-09-16T01:40:13.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.418+0000 7fa551bb4f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x556922cc14a0 con 0x556922d0c700 2024-09-16T01:40:13.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.418+0000 7fa546ea9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556922d0cae0 0x556922e2e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.418+0000 7fa546ea9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556922d0cae0 0x556922e2e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46436/0 (socket says 172.21.15.88:46436) 2024-09-16T01:40:13.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.418+0000 7fa546ea9640 1 -- 172.21.15.88:0/2267574805 learned_addr learned my addr 172.21.15.88:0/2267574805 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:13.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.419+0000 7fa54f6aa640 1 --2- 172.21.15.88:0/2267574805 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556922d0c700 0x556922e31500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.419+0000 7fa54feab640 1 --2- 172.21.15.88:0/2267574805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556922d9b740 0x556922d891c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.419+0000 7fa546ea9640 1 -- 172.21.15.88:0/2267574805 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556922d0c700 msgr2=0x556922e31500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.419+0000 7fa546ea9640 1 --2- 172.21.15.88:0/2267574805 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556922d0c700 0x556922e31500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.419+0000 7fa546ea9640 1 -- 172.21.15.88:0/2267574805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556922d9b740 msgr2=0x556922d891c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.419+0000 7fa546ea9640 1 --2- 172.21.15.88:0/2267574805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556922d9b740 0x556922d891c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.419+0000 7fa546ea9640 1 -- 172.21.15.88:0/2267574805 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556922d8a510 con 0x556922d0cae0 2024-09-16T01:40:13.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.419+0000 7fa546ea9640 1 --2- 172.21.15.88:0/2267574805 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556922d0cae0 0x556922e2e560 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x556922cc2420 tx=0x7fa538000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.419+0000 7fa54d6a6640 1 -- 172.21.15.88:0/2267574805 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa538020070 con 0x556922d0cae0 2024-09-16T01:40:13.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.419+0000 7fa54d6a6640 1 -- 172.21.15.88:0/2267574805 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa538001200 con 0x556922d0cae0 2024-09-16T01:40:13.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.419+0000 7fa551bb4f40 1 -- 172.21.15.88:0/2267574805 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556922cf1460 con 0x556922d0cae0 2024-09-16T01:40:13.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.419+0000 7fa54d6a6640 1 -- 172.21.15.88:0/2267574805 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa53801c680 con 0x556922d0cae0 2024-09-16T01:40:13.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.419+0000 7fa551bb4f40 1 -- 172.21.15.88:0/2267574805 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556922cf1a20 con 0x556922d0cae0 2024-09-16T01:40:13.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.421+0000 7fa54d6a6640 1 -- 172.21.15.88:0/2267574805 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa538023070 con 0x556922d0cae0 2024-09-16T01:40:13.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.421+0000 7fa54d6a6640 1 --2- 172.21.15.88:0/2267574805 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa53006acc0 0x7fa53006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.421+0000 7fa54d6a6640 1 -- 172.21.15.88:0/2267574805 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7fa53808b800 con 0x556922d0cae0 2024-09-16T01:40:13.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.421+0000 7fa54f6aa640 1 --2- 172.21.15.88:0/2267574805 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa53006acc0 0x7fa53006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.421+0000 7fa551bb4f40 1 --2- 172.21.15.88:0/2267574805 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x556922d93710 0x556922cf2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.421+0000 7fa551bb4f40 1 -- 172.21.15.88:0/2267574805 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:1 12.14 12:287034c4:::1.00000000.inode:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x556922cf2760 con 0x556922d93710 2024-09-16T01:40:13.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.421+0000 7fa54f6aa640 1 --2- 172.21.15.88:0/2267574805 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa53006acc0 0x7fa53006d180 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fa540005fd0 tx=0x7fa540005b30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.421+0000 7fa54feab640 1 --2- 172.21.15.88:0/2267574805 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x556922d93710 0x556922cf2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.422+0000 7fa54feab640 1 --2- 172.21.15.88:0/2267574805 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x556922d93710 0x556922cf2220 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.422+0000 7fa54feab640 1 -- 172.21.15.88:0/2267574805 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(1 1.00000000.inode [omap-get-keys out=5b] v0'0 uv16 ondisk = 0) ==== 160+0+5 (crc 0 0 0) 0x7fa54804f990 con 0x556922d93710 2024-09-16T01:40:13.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.422+0000 7fa551bb4f40 1 -- 172.21.15.88:0/2267574805 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x556922d93710 msgr2=0x556922cf2220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.422+0000 7fa551bb4f40 1 --2- 172.21.15.88:0/2267574805 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x556922d93710 0x556922cf2220 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.422+0000 7fa551bb4f40 1 -- 172.21.15.88:0/2267574805 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa53006acc0 msgr2=0x7fa53006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.422+0000 7fa551bb4f40 1 --2- 172.21.15.88:0/2267574805 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa53006acc0 0x7fa53006d180 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fa540005fd0 tx=0x7fa540005b30 comp rx=0 tx=0).stop 2024-09-16T01:40:13.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.422+0000 7fa551bb4f40 1 -- 172.21.15.88:0/2267574805 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556922d0cae0 msgr2=0x556922e2e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.422+0000 7fa551bb4f40 1 --2- 172.21.15.88:0/2267574805 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556922d0cae0 0x556922e2e560 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x556922cc2420 tx=0x7fa538000ed0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.422+0000 7fa54feab640 1 -- 172.21.15.88:0/2267574805 reap_dead start 2024-09-16T01:40:13.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.422+0000 7fa551bb4f40 1 -- 172.21.15.88:0/2267574805 shutdown_connections 2024-09-16T01:40:13.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.422+0000 7fa551bb4f40 1 -- 172.21.15.88:0/2267574805 >> 172.21.15.88:0/2267574805 conn(0x556922ced680 msgr2=0x556922d9a550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:13.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.422+0000 7fa551bb4f40 1 -- 172.21.15.88:0/2267574805 shutdown_connections 2024-09-16T01:40:13.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.422+0000 7fa551bb4f40 1 -- 172.21.15.88:0/2267574805 wait complete. 2024-09-16T01:40:13.430 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 200.00000000 2024-09-16T01:40:13.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.509+0000 7f0f4f31ff40 1 Processor -- start 2024-09-16T01:40:13.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.509+0000 7f0f4f31ff40 1 -- start start 2024-09-16T01:40:13.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.510+0000 7f0f4f31ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5565c22c8fe0 0x5565c236d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.510+0000 7f0f4f31ff40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c22c9a60 0x5565c236dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.510+0000 7f0f4f31ff40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5565c22e3250 0x5565c236f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.510+0000 7f0f4f31ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5565c21eb510 con 0x5565c22c8fe0 2024-09-16T01:40:13.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.510+0000 7f0f4f31ff40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5565c2244c50 con 0x5565c22c9a60 2024-09-16T01:40:13.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.510+0000 7f0f4f31ff40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5565c236b1b0 con 0x5565c22e3250 2024-09-16T01:40:13.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.510+0000 7f0f4c390640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5565c22c8fe0 0x5565c236d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.510+0000 7f0f4c390640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5565c22c8fe0 0x5565c236d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46446/0 (socket says 172.21.15.88:46446) 2024-09-16T01:40:13.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.510+0000 7f0f4c390640 1 -- 172.21.15.88:0/1732978212 learned_addr learned my addr 172.21.15.88:0/1732978212 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:13.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.510+0000 7f0f4bb8f640 1 --2- 172.21.15.88:0/1732978212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c22c9a60 0x5565c236dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.510+0000 7f0f4cb91640 1 --2- 172.21.15.88:0/1732978212 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5565c22e3250 0x5565c236f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.510+0000 7f0f4c390640 1 -- 172.21.15.88:0/1732978212 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5565c22e3250 msgr2=0x5565c236f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:13.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.510+0000 7f0f4c390640 1 --2- 172.21.15.88:0/1732978212 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5565c22e3250 0x5565c236f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.510+0000 7f0f4c390640 1 -- 172.21.15.88:0/1732978212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c22c9a60 msgr2=0x5565c236dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:13.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.510+0000 7f0f4c390640 1 --2- 172.21.15.88:0/1732978212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c22c9a60 0x5565c236dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.510+0000 7f0f4c390640 1 -- 172.21.15.88:0/1732978212 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565c22c53d0 con 0x5565c22c8fe0 2024-09-16T01:40:13.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.510+0000 7f0f4c390640 1 --2- 172.21.15.88:0/1732978212 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5565c22c8fe0 0x5565c236d790 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c001d30 tx=0x7f0f3c00b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65a499111be9a26b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.511+0000 7f0f4ab8d640 1 -- 172.21.15.88:0/1732978212 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c00bd20 con 0x5565c22c8fe0 2024-09-16T01:40:13.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.511+0000 7f0f4ab8d640 1 -- 172.21.15.88:0/1732978212 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f3c00f400 con 0x5565c22c8fe0 2024-09-16T01:40:13.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.511+0000 7f0f4ab8d640 1 -- 172.21.15.88:0/1732978212 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c00f830 con 0x5565c22c8fe0 2024-09-16T01:40:13.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.511+0000 7f0f4f31ff40 1 -- 172.21.15.88:0/1732978212 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5565c22c8fe0 msgr2=0x5565c236d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.511+0000 7f0f4f31ff40 1 --2- 172.21.15.88:0/1732978212 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5565c22c8fe0 0x5565c236d790 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c001d30 tx=0x7f0f3c00b5b0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.511+0000 7f0f4f31ff40 1 -- 172.21.15.88:0/1732978212 shutdown_connections 2024-09-16T01:40:13.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.511+0000 7f0f4f31ff40 1 --2- 172.21.15.88:0/1732978212 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5565c22e3250 0x5565c236f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.511+0000 7f0f4f31ff40 1 --2- 172.21.15.88:0/1732978212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c22c9a60 0x5565c236dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.511+0000 7f0f4f31ff40 1 --2- 172.21.15.88:0/1732978212 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5565c22c8fe0 0x5565c236d790 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.511+0000 7f0f4f31ff40 1 -- 172.21.15.88:0/1732978212 >> 172.21.15.88:0/1732978212 conn(0x5565c22e4750 msgr2=0x5565c22c4c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:13.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.511+0000 7f0f4f31ff40 1 -- 172.21.15.88:0/1732978212 shutdown_connections 2024-09-16T01:40:13.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.511+0000 7f0f4f31ff40 1 -- 172.21.15.88:0/1732978212 wait complete. 2024-09-16T01:40:13.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.512+0000 7f0f4f31ff40 1 Processor -- start 2024-09-16T01:40:13.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.512+0000 7f0f4f31ff40 1 -- start start 2024-09-16T01:40:13.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.512+0000 7f0f4f31ff40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5565c22c8fe0 0x5565c236ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.512+0000 7f0f4f31ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5565c22c9a60 0x5565c22c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.512+0000 7f0f4f31ff40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c22e3250 0x5565c2365410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.512+0000 7f0f4f31ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5565c22e4b30 con 0x5565c22c9a60 2024-09-16T01:40:13.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.512+0000 7f0f4f31ff40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5565c236ae00 con 0x5565c22e3250 2024-09-16T01:40:13.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.512+0000 7f0f4f31ff40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5565c2407a50 con 0x5565c22c8fe0 2024-09-16T01:40:13.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.512+0000 7f0f4bb8f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5565c22c9a60 0x5565c22c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.513+0000 7f0f4bb8f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5565c22c9a60 0x5565c22c45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46452/0 (socket says 172.21.15.88:46452) 2024-09-16T01:40:13.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.513+0000 7f0f4bb8f640 1 -- 172.21.15.88:0/2082958833 learned_addr learned my addr 172.21.15.88:0/2082958833 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:13.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.513+0000 7f0f4c390640 1 --2- 172.21.15.88:0/2082958833 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5565c22c8fe0 0x5565c236ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.513+0000 7f0f4cb91640 1 --2- 172.21.15.88:0/2082958833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c22e3250 0x5565c2365410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.513+0000 7f0f4bb8f640 1 -- 172.21.15.88:0/2082958833 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5565c22c8fe0 msgr2=0x5565c236ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.513+0000 7f0f4bb8f640 1 --2- 172.21.15.88:0/2082958833 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5565c22c8fe0 0x5565c236ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.513+0000 7f0f4bb8f640 1 -- 172.21.15.88:0/2082958833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c22e3250 msgr2=0x5565c2365410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.513+0000 7f0f4bb8f640 1 --2- 172.21.15.88:0/2082958833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5565c22e3250 0x5565c2365410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.513+0000 7f0f4bb8f640 1 -- 172.21.15.88:0/2082958833 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565c2407cc0 con 0x5565c22c9a60 2024-09-16T01:40:13.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.513+0000 7f0f4bb8f640 1 --2- 172.21.15.88:0/2082958833 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5565c22c9a60 0x5565c22c45f0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f0f4000df30 tx=0x7f0f4000b6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.513+0000 7f0f49b8b640 1 -- 172.21.15.88:0/2082958833 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f40014020 con 0x5565c22c9a60 2024-09-16T01:40:13.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.513+0000 7f0f49b8b640 1 -- 172.21.15.88:0/2082958833 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f40010620 con 0x5565c22c9a60 2024-09-16T01:40:13.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.513+0000 7f0f49b8b640 1 -- 172.21.15.88:0/2082958833 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f40008030 con 0x5565c22c9a60 2024-09-16T01:40:13.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.513+0000 7f0f4f31ff40 1 -- 172.21.15.88:0/2082958833 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5565c2407fa0 con 0x5565c22c9a60 2024-09-16T01:40:13.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.513+0000 7f0f4f31ff40 1 -- 172.21.15.88:0/2082958833 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565c2408560 con 0x5565c22c9a60 2024-09-16T01:40:13.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.514+0000 7f0f49b8b640 1 -- 172.21.15.88:0/2082958833 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0f40017020 con 0x5565c22c9a60 2024-09-16T01:40:13.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.515+0000 7f0f49b8b640 1 --2- 172.21.15.88:0/2082958833 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0f3806ac90 0x7f0f3806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.515+0000 7f0f49b8b640 1 -- 172.21.15.88:0/2082958833 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f0f40089ff0 con 0x5565c22c9a60 2024-09-16T01:40:13.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.515+0000 7f0f4c390640 1 --2- 172.21.15.88:0/2082958833 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0f3806ac90 0x7f0f3806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.515+0000 7f0f4f31ff40 1 --2- 172.21.15.88:0/2082958833 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5565c22c4b30 0x5565c2408f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.515+0000 7f0f4f31ff40 1 -- 172.21.15.88:0/2082958833 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:1 12.14 12:292cf221:::200.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5565c2409480 con 0x5565c22c4b30 2024-09-16T01:40:13.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.515+0000 7f0f4c390640 1 --2- 172.21.15.88:0/2082958833 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0f3806ac90 0x7f0f3806d150 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c000c00 tx=0x7f0f3c009000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.515+0000 7f0f4cb91640 1 --2- 172.21.15.88:0/2082958833 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5565c22c4b30 0x5565c2408f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.516+0000 7f0f4cb91640 1 --2- 172.21.15.88:0/2082958833 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5565c22c4b30 0x5565c2408f40 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.516+0000 7f0f4cb91640 1 -- 172.21.15.88:0/2082958833 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(1 200.00000000 [omap-get-keys out=5b] v0'0 uv20 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f0f4405d040 con 0x5565c22c4b30 2024-09-16T01:40:13.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.516+0000 7f0f4f31ff40 1 -- 172.21.15.88:0/2082958833 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5565c22c4b30 msgr2=0x5565c2408f40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.516+0000 7f0f4f31ff40 1 --2- 172.21.15.88:0/2082958833 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5565c22c4b30 0x5565c2408f40 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.516+0000 7f0f4f31ff40 1 -- 172.21.15.88:0/2082958833 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0f3806ac90 msgr2=0x7f0f3806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.516+0000 7f0f4f31ff40 1 --2- 172.21.15.88:0/2082958833 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0f3806ac90 0x7f0f3806d150 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c000c00 tx=0x7f0f3c009000 comp rx=0 tx=0).stop 2024-09-16T01:40:13.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.516+0000 7f0f4f31ff40 1 -- 172.21.15.88:0/2082958833 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5565c22c9a60 msgr2=0x5565c22c45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.516+0000 7f0f4f31ff40 1 --2- 172.21.15.88:0/2082958833 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5565c22c9a60 0x5565c22c45f0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f0f4000df30 tx=0x7f0f4000b6a0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.516+0000 7f0f4cb91640 1 -- 172.21.15.88:0/2082958833 reap_dead start 2024-09-16T01:40:13.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.516+0000 7f0f4f31ff40 1 -- 172.21.15.88:0/2082958833 shutdown_connections 2024-09-16T01:40:13.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.516+0000 7f0f4f31ff40 1 -- 172.21.15.88:0/2082958833 >> 172.21.15.88:0/2082958833 conn(0x5565c22e4750 msgr2=0x5565c22e0050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:13.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.516+0000 7f0f4f31ff40 1 -- 172.21.15.88:0/2082958833 shutdown_connections 2024-09-16T01:40:13.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.516+0000 7f0f4f31ff40 1 -- 172.21.15.88:0/2082958833 wait complete. 2024-09-16T01:40:13.525 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 200.00000001 2024-09-16T01:40:13.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.606+0000 7f003140bf40 1 Processor -- start 2024-09-16T01:40:13.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.606+0000 7f003140bf40 1 -- start start 2024-09-16T01:40:13.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.607+0000 7f003140bf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f001c0022a0 0x56070be93e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.607+0000 7f003140bf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56070be9c2f0 0x56070be9df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.607+0000 7f003140bf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56070be9c750 0x56070be9e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.607+0000 7f003140bf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56070bdcba70 con 0x7f001c0022a0 2024-09-16T01:40:13.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.607+0000 7f003140bf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56070bd1b510 con 0x56070be9c750 2024-09-16T01:40:13.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.607+0000 7f003140bf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56070be110a0 con 0x56070be9c2f0 2024-09-16T01:40:13.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.607+0000 7f002e224640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f001c0022a0 0x56070be93e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.607+0000 7f002e224640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f001c0022a0 0x56070be93e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46474/0 (socket says 172.21.15.88:46474) 2024-09-16T01:40:13.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.607+0000 7f002e224640 1 -- 172.21.15.88:0/1644270406 learned_addr learned my addr 172.21.15.88:0/1644270406 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:13.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.607+0000 7f002ea25640 1 --2- 172.21.15.88:0/1644270406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56070be9c750 0x56070be9e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.607+0000 7f0027fff640 1 --2- 172.21.15.88:0/1644270406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56070be9c2f0 0x56070be9df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.607+0000 7f002e224640 1 -- 172.21.15.88:0/1644270406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56070be9c2f0 msgr2=0x56070be9df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:13.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.607+0000 7f002e224640 1 --2- 172.21.15.88:0/1644270406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56070be9c2f0 0x56070be9df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.607+0000 7f002e224640 1 -- 172.21.15.88:0/1644270406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56070be9c750 msgr2=0x56070be9e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:13.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.607+0000 7f002e224640 1 --2- 172.21.15.88:0/1644270406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56070be9c750 0x56070be9e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.607+0000 7f002e224640 1 -- 172.21.15.88:0/1644270406 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56070be9e9c0 con 0x7f001c0022a0 2024-09-16T01:40:13.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.607+0000 7f0027fff640 1 --2- 172.21.15.88:0/1644270406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56070be9c2f0 0x56070be9df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:13.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.607+0000 7f002e224640 1 --2- 172.21.15.88:0/1644270406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f001c0022a0 0x56070be93e70 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f0020045cb0 tx=0x7f0020051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62f65699298700fb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.607+0000 7f002d222640 1 -- 172.21.15.88:0/1644270406 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002004f8a0 con 0x7f001c0022a0 2024-09-16T01:40:13.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.607+0000 7f002d222640 1 -- 172.21.15.88:0/1644270406 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f002004fa60 con 0x7f001c0022a0 2024-09-16T01:40:13.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.607+0000 7f002d222640 1 -- 172.21.15.88:0/1644270406 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002004fe50 con 0x7f001c0022a0 2024-09-16T01:40:13.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.608+0000 7f003140bf40 1 -- 172.21.15.88:0/1644270406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f001c0022a0 msgr2=0x56070be93e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.608+0000 7f003140bf40 1 --2- 172.21.15.88:0/1644270406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f001c0022a0 0x56070be93e70 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f0020045cb0 tx=0x7f0020051000 comp rx=0 tx=0).stop 2024-09-16T01:40:13.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.608+0000 7f003140bf40 1 -- 172.21.15.88:0/1644270406 shutdown_connections 2024-09-16T01:40:13.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.608+0000 7f003140bf40 1 --2- 172.21.15.88:0/1644270406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56070be9c750 0x56070be9e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.608+0000 7f003140bf40 1 --2- 172.21.15.88:0/1644270406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56070be9c2f0 0x56070be9df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.608+0000 7f003140bf40 1 --2- 172.21.15.88:0/1644270406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f001c0022a0 0x56070be93e70 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.608+0000 7f003140bf40 1 -- 172.21.15.88:0/1644270406 >> 172.21.15.88:0/1644270406 conn(0x56070bdf4680 msgr2=0x56070bdf9640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:13.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.608+0000 7f003140bf40 1 -- 172.21.15.88:0/1644270406 shutdown_connections 2024-09-16T01:40:13.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.608+0000 7f003140bf40 1 -- 172.21.15.88:0/1644270406 wait complete. 2024-09-16T01:40:13.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.608+0000 7f003140bf40 1 Processor -- start 2024-09-16T01:40:13.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.608+0000 7f003140bf40 1 -- start start 2024-09-16T01:40:13.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.609+0000 7f003140bf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56070be9c2f0 0x56070be8e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.609+0000 7f003140bf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56070be9c750 0x56070be0b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.609+0000 7f003140bf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f001c0022a0 0x56070bdf9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.609+0000 7f003140bf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56070be934b0 con 0x56070be9c750 2024-09-16T01:40:13.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.609+0000 7f003140bf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56070bdc82c0 con 0x7f001c0022a0 2024-09-16T01:40:13.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.609+0000 7f003140bf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56070be98cc0 con 0x56070be9c2f0 2024-09-16T01:40:13.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.609+0000 7f0027fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56070be9c750 0x56070be0b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.609+0000 7f0027fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56070be9c750 0x56070be0b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46490/0 (socket says 172.21.15.88:46490) 2024-09-16T01:40:13.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.609+0000 7f0027fff640 1 -- 172.21.15.88:0/4086159444 learned_addr learned my addr 172.21.15.88:0/4086159444 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:13.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.609+0000 7f002ea25640 1 --2- 172.21.15.88:0/4086159444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f001c0022a0 0x56070bdf9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.609+0000 7f002e224640 1 --2- 172.21.15.88:0/4086159444 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56070be9c2f0 0x56070be8e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.609+0000 7f0027fff640 1 -- 172.21.15.88:0/4086159444 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56070be9c2f0 msgr2=0x56070be8e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.609+0000 7f0027fff640 1 --2- 172.21.15.88:0/4086159444 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56070be9c2f0 0x56070be8e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.609+0000 7f0027fff640 1 -- 172.21.15.88:0/4086159444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f001c0022a0 msgr2=0x56070bdf9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.609+0000 7f0027fff640 1 --2- 172.21.15.88:0/4086159444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f001c0022a0 0x56070bdf9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.609+0000 7f0027fff640 1 -- 172.21.15.88:0/4086159444 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56070be0a700 con 0x56070be9c750 2024-09-16T01:40:13.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.610+0000 7f0027fff640 1 --2- 172.21.15.88:0/4086159444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56070be9c750 0x56070be0b7a0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f0018009ec0 tx=0x7f0018008400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.610+0000 7f00277fe640 1 -- 172.21.15.88:0/4086159444 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0018016680 con 0x56070be9c750 2024-09-16T01:40:13.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.610+0000 7f00277fe640 1 -- 172.21.15.88:0/4086159444 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f001801b070 con 0x56070be9c750 2024-09-16T01:40:13.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.610+0000 7f00277fe640 1 -- 172.21.15.88:0/4086159444 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0018016a60 con 0x56070be9c750 2024-09-16T01:40:13.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.610+0000 7f003140bf40 1 -- 172.21.15.88:0/4086159444 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56070be0a9e0 con 0x56070be9c750 2024-09-16T01:40:13.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.610+0000 7f003140bf40 1 -- 172.21.15.88:0/4086159444 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56070be0afa0 con 0x56070be9c750 2024-09-16T01:40:13.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.611+0000 7f00277fe640 1 -- 172.21.15.88:0/4086159444 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0018016c00 con 0x56070be9c750 2024-09-16T01:40:13.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.611+0000 7f00277fe640 1 --2- 172.21.15.88:0/4086159444 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f001006ac90 0x7f001006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.612+0000 7f00277fe640 1 -- 172.21.15.88:0/4086159444 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f001808a870 con 0x56070be9c750 2024-09-16T01:40:13.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.612+0000 7f002e224640 1 --2- 172.21.15.88:0/4086159444 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f001006ac90 0x7f001006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.612+0000 7f003140bf40 1 --2- 172.21.15.88:0/4086159444 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x56070bdca730 0x56070be8e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.612+0000 7f003140bf40 1 -- 172.21.15.88:0/4086159444 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:1 12.14 12:2e2fa760:::200.00000001:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x56070be8ede0 con 0x56070bdca730 2024-09-16T01:40:13.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.612+0000 7f002e224640 1 --2- 172.21.15.88:0/4086159444 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f001006ac90 0x7f001006d150 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x56070be933c0 tx=0x7f0020053000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.612+0000 7f002ea25640 1 --2- 172.21.15.88:0/4086159444 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x56070bdca730 0x56070be8e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.612+0000 7f002ea25640 1 --2- 172.21.15.88:0/4086159444 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x56070bdca730 0x56070be8e8a0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.613+0000 7f002ea25640 1 -- 172.21.15.88:0/4086159444 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(1 200.00000001 [omap-get-keys out=5b] v0'0 uv17 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f0028006260 con 0x56070bdca730 2024-09-16T01:40:13.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.613+0000 7f003140bf40 1 -- 172.21.15.88:0/4086159444 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x56070bdca730 msgr2=0x56070be8e8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.613+0000 7f003140bf40 1 --2- 172.21.15.88:0/4086159444 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x56070bdca730 0x56070be8e8a0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.613+0000 7f003140bf40 1 -- 172.21.15.88:0/4086159444 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f001006ac90 msgr2=0x7f001006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.613+0000 7f003140bf40 1 --2- 172.21.15.88:0/4086159444 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f001006ac90 0x7f001006d150 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x56070be933c0 tx=0x7f0020053000 comp rx=0 tx=0).stop 2024-09-16T01:40:13.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.613+0000 7f003140bf40 1 -- 172.21.15.88:0/4086159444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56070be9c750 msgr2=0x56070be0b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.613+0000 7f003140bf40 1 --2- 172.21.15.88:0/4086159444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56070be9c750 0x56070be0b7a0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f0018009ec0 tx=0x7f0018008400 comp rx=0 tx=0).stop 2024-09-16T01:40:13.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.613+0000 7f002ea25640 1 -- 172.21.15.88:0/4086159444 reap_dead start 2024-09-16T01:40:13.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.613+0000 7f003140bf40 1 -- 172.21.15.88:0/4086159444 shutdown_connections 2024-09-16T01:40:13.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.613+0000 7f003140bf40 1 -- 172.21.15.88:0/4086159444 >> 172.21.15.88:0/4086159444 conn(0x56070bdf4680 msgr2=0x56070be9e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:13.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.613+0000 7f003140bf40 1 -- 172.21.15.88:0/4086159444 shutdown_connections 2024-09-16T01:40:13.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.613+0000 7f003140bf40 1 -- 172.21.15.88:0/4086159444 wait complete. 2024-09-16T01:40:13.620 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys mds0_openfiles.0 2024-09-16T01:40:13.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.660+0000 7ff531068f40 1 Processor -- start 2024-09-16T01:40:13.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.660+0000 7ff531068f40 1 -- start start 2024-09-16T01:40:13.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.661+0000 7ff531068f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d5be682940 0x55d5be6b1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.661+0000 7ff531068f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5be682d50 0x55d5be759a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.661+0000 7ff531068f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d5be756f30 0x55d5be759fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.661+0000 7ff531068f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d5be5d3510 con 0x55d5be756f30 2024-09-16T01:40:13.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.661+0000 7ff531068f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d5be62cc50 con 0x55d5be682d50 2024-09-16T01:40:13.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.661+0000 7ff531068f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55d5be680c80 con 0x55d5be682940 2024-09-16T01:40:13.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.661+0000 7ff52f361640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d5be756f30 0x55d5be759fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.661+0000 7ff52f361640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d5be756f30 0x55d5be759fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46496/0 (socket says 172.21.15.88:46496) 2024-09-16T01:40:13.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.661+0000 7ff52f361640 1 -- 172.21.15.88:0/239032178 learned_addr learned my addr 172.21.15.88:0/239032178 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:13.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.661+0000 7ff52e35f640 1 --2- 172.21.15.88:0/239032178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5be682d50 0x55d5be759a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.661+0000 7ff52eb60640 1 --2- 172.21.15.88:0/239032178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d5be682940 0x55d5be6b1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.661+0000 7ff52f361640 1 -- 172.21.15.88:0/239032178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d5be682940 msgr2=0x55d5be6b1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:13.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.661+0000 7ff52f361640 1 --2- 172.21.15.88:0/239032178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d5be682940 0x55d5be6b1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.661+0000 7ff52f361640 1 -- 172.21.15.88:0/239032178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5be682d50 msgr2=0x55d5be759a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:13.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.661+0000 7ff52f361640 1 --2- 172.21.15.88:0/239032178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5be682d50 0x55d5be759a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.661+0000 7ff52f361640 1 -- 172.21.15.88:0/239032178 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5be754620 con 0x55d5be756f30 2024-09-16T01:40:13.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.661+0000 7ff52e35f640 1 --2- 172.21.15.88:0/239032178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5be682d50 0x55d5be759a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:13.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.662+0000 7ff52f361640 1 --2- 172.21.15.88:0/239032178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d5be756f30 0x55d5be759fd0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7ff5280065e0 tx=0x7ff528006aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0f422ea9473e0a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.662+0000 7ff52d35d640 1 -- 172.21.15.88:0/239032178 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff528016070 con 0x55d5be756f30 2024-09-16T01:40:13.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.662+0000 7ff52d35d640 1 -- 172.21.15.88:0/239032178 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff52800b790 con 0x55d5be756f30 2024-09-16T01:40:13.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.662+0000 7ff52d35d640 1 -- 172.21.15.88:0/239032178 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff52800bb80 con 0x55d5be756f30 2024-09-16T01:40:13.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.662+0000 7ff531068f40 1 -- 172.21.15.88:0/239032178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d5be756f30 msgr2=0x55d5be759fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.662+0000 7ff531068f40 1 --2- 172.21.15.88:0/239032178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d5be756f30 0x55d5be759fd0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7ff5280065e0 tx=0x7ff528006aa0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.662+0000 7ff531068f40 1 -- 172.21.15.88:0/239032178 shutdown_connections 2024-09-16T01:40:13.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.662+0000 7ff531068f40 1 --2- 172.21.15.88:0/239032178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d5be756f30 0x55d5be759fd0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.662+0000 7ff531068f40 1 --2- 172.21.15.88:0/239032178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5be682d50 0x55d5be759a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.662+0000 7ff531068f40 1 --2- 172.21.15.88:0/239032178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d5be682940 0x55d5be6b1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.662+0000 7ff531068f40 1 -- 172.21.15.88:0/239032178 >> 172.21.15.88:0/239032178 conn(0x55d5be6cc750 msgr2=0x55d5be6ccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:13.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.662+0000 7ff531068f40 1 -- 172.21.15.88:0/239032178 shutdown_connections 2024-09-16T01:40:13.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.662+0000 7ff531068f40 1 -- 172.21.15.88:0/239032178 wait complete. 2024-09-16T01:40:13.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.663+0000 7ff531068f40 1 Processor -- start 2024-09-16T01:40:13.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.663+0000 7ff531068f40 1 -- start start 2024-09-16T01:40:13.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.663+0000 7ff531068f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5be682940 0x55d5be753ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.663+0000 7ff531068f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d5be682d50 0x55d5be685150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.664+0000 7ff531068f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d5be756f30 0x55d5be759cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.664+0000 7ff531068f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d5be6b0b70 con 0x55d5be756f30 2024-09-16T01:40:13.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.664+0000 7ff531068f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d5be756330 con 0x55d5be682940 2024-09-16T01:40:13.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.664+0000 7ff531068f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55d5be756050 con 0x55d5be682d50 2024-09-16T01:40:13.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.664+0000 7ff52f361640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d5be756f30 0x55d5be759cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.664+0000 7ff52f361640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d5be756f30 0x55d5be759cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46510/0 (socket says 172.21.15.88:46510) 2024-09-16T01:40:13.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.664+0000 7ff52f361640 1 -- 172.21.15.88:0/2254431230 learned_addr learned my addr 172.21.15.88:0/2254431230 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:13.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.664+0000 7ff52e35f640 1 --2- 172.21.15.88:0/2254431230 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d5be682d50 0x55d5be685150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.664+0000 7ff52eb60640 1 --2- 172.21.15.88:0/2254431230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5be682940 0x55d5be753ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.664+0000 7ff52f361640 1 -- 172.21.15.88:0/2254431230 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d5be682d50 msgr2=0x55d5be685150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.664+0000 7ff52f361640 1 --2- 172.21.15.88:0/2254431230 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d5be682d50 0x55d5be685150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.664+0000 7ff52f361640 1 -- 172.21.15.88:0/2254431230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5be682940 msgr2=0x55d5be753ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.664+0000 7ff52f361640 1 --2- 172.21.15.88:0/2254431230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d5be682940 0x55d5be753ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.664+0000 7ff52f361640 1 -- 172.21.15.88:0/2254431230 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5be74e700 con 0x55d5be756f30 2024-09-16T01:40:13.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.664+0000 7ff52f361640 1 --2- 172.21.15.88:0/2254431230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d5be756f30 0x55d5be759cf0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7ff528006ae0 tx=0x7ff528001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.664+0000 7ff517fff640 1 -- 172.21.15.88:0/2254431230 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff528020030 con 0x55d5be756f30 2024-09-16T01:40:13.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.664+0000 7ff517fff640 1 -- 172.21.15.88:0/2254431230 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff528011580 con 0x55d5be756f30 2024-09-16T01:40:13.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.664+0000 7ff531068f40 1 -- 172.21.15.88:0/2254431230 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d5be6c8c00 con 0x55d5be756f30 2024-09-16T01:40:13.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.664+0000 7ff517fff640 1 -- 172.21.15.88:0/2254431230 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff528016040 con 0x55d5be756f30 2024-09-16T01:40:13.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.664+0000 7ff531068f40 1 -- 172.21.15.88:0/2254431230 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5be6ad7a0 con 0x55d5be756f30 2024-09-16T01:40:13.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.666+0000 7ff517fff640 1 -- 172.21.15.88:0/2254431230 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff528023030 con 0x55d5be756f30 2024-09-16T01:40:13.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.666+0000 7ff517fff640 1 --2- 172.21.15.88:0/2254431230 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff51c06acc0 0x7ff51c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.666+0000 7ff517fff640 1 -- 172.21.15.88:0/2254431230 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7ff52808b390 con 0x55d5be756f30 2024-09-16T01:40:13.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.666+0000 7ff52eb60640 1 --2- 172.21.15.88:0/2254431230 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff51c06acc0 0x7ff51c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.666+0000 7ff531068f40 1 --2- 172.21.15.88:0/2254431230 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55d5be6aba50 0x55d5be74d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.666+0000 7ff531068f40 1 -- 172.21.15.88:0/2254431230 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:1 12.6 12:654134d2:::mds0_openfiles.0:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55d5be751bc0 con 0x55d5be6aba50 2024-09-16T01:40:13.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.666+0000 7ff52e35f640 1 --2- 172.21.15.88:0/2254431230 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55d5be6aba50 0x55d5be74d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.666+0000 7ff52eb60640 1 --2- 172.21.15.88:0/2254431230 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff51c06acc0 0x7ff51c06d180 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff520010650 tx=0x7ff520001210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.667+0000 7ff52e35f640 1 --2- 172.21.15.88:0/2254431230 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55d5be6aba50 0x55d5be74d460 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.667+0000 7ff52e35f640 1 -- 172.21.15.88:0/2254431230 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(1 mds0_openfiles.0 [omap-get-keys out=5b] v0'0 uv3 ondisk = 0) ==== 160+0+5 (crc 0 0 0) 0x7ff5240548d0 con 0x55d5be6aba50 2024-09-16T01:40:13.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.667+0000 7ff531068f40 1 -- 172.21.15.88:0/2254431230 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55d5be6aba50 msgr2=0x55d5be74d460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.667+0000 7ff531068f40 1 --2- 172.21.15.88:0/2254431230 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55d5be6aba50 0x55d5be74d460 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.667+0000 7ff531068f40 1 -- 172.21.15.88:0/2254431230 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff51c06acc0 msgr2=0x7ff51c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.667+0000 7ff531068f40 1 --2- 172.21.15.88:0/2254431230 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff51c06acc0 0x7ff51c06d180 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff520010650 tx=0x7ff520001210 comp rx=0 tx=0).stop 2024-09-16T01:40:13.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.667+0000 7ff531068f40 1 -- 172.21.15.88:0/2254431230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d5be756f30 msgr2=0x55d5be759cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.667+0000 7ff531068f40 1 --2- 172.21.15.88:0/2254431230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d5be756f30 0x55d5be759cf0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7ff528006ae0 tx=0x7ff528001250 comp rx=0 tx=0).stop 2024-09-16T01:40:13.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.667+0000 7ff52f361640 1 -- 172.21.15.88:0/2254431230 reap_dead start 2024-09-16T01:40:13.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.667+0000 7ff531068f40 1 -- 172.21.15.88:0/2254431230 shutdown_connections 2024-09-16T01:40:13.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.667+0000 7ff531068f40 1 -- 172.21.15.88:0/2254431230 >> 172.21.15.88:0/2254431230 conn(0x55d5be6cc750 msgr2=0x55d5be6ccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:13.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.667+0000 7ff531068f40 1 -- 172.21.15.88:0/2254431230 shutdown_connections 2024-09-16T01:40:13.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.667+0000 7ff531068f40 1 -- 172.21.15.88:0/2254431230 wait complete. 2024-09-16T01:40:13.679 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 500.00000000 2024-09-16T01:40:13.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.761+0000 7f60d4e1af40 1 Processor -- start 2024-09-16T01:40:13.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.761+0000 7f60d4e1af40 1 -- start start 2024-09-16T01:40:13.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.762+0000 7f60d4e1af40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56280da78fe0 0x56280db163b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.762+0000 7f60d4e1af40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56280db1dd20 0x56280da755f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.762+0000 7f60d4e1af40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56280db1e100 0x56280da75b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.762+0000 7f60d4e1af40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56280d99b510 con 0x56280db1e100 2024-09-16T01:40:13.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.762+0000 7f60d4e1af40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56280da91bd0 con 0x56280da78fe0 2024-09-16T01:40:13.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.762+0000 7f60d4e1af40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56280db173b0 con 0x56280db1dd20 2024-09-16T01:40:13.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.762+0000 7f60d3111640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56280db1e100 0x56280da75b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.762+0000 7f60d3111640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56280db1e100 0x56280da75b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46516/0 (socket says 172.21.15.88:46516) 2024-09-16T01:40:13.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.762+0000 7f60d3111640 1 -- 172.21.15.88:0/2973123361 learned_addr learned my addr 172.21.15.88:0/2973123361 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:13.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.762+0000 7f60d2910640 1 --2- 172.21.15.88:0/2973123361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56280da78fe0 0x56280db163b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.762+0000 7f60cbfff640 1 --2- 172.21.15.88:0/2973123361 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56280db1dd20 0x56280da755f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.762+0000 7f60d3111640 1 -- 172.21.15.88:0/2973123361 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56280db1dd20 msgr2=0x56280da755f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:13.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.762+0000 7f60d3111640 1 --2- 172.21.15.88:0/2973123361 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56280db1dd20 0x56280da755f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.762+0000 7f60d3111640 1 -- 172.21.15.88:0/2973123361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56280da78fe0 msgr2=0x56280db163b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:13.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.762+0000 7f60d3111640 1 --2- 172.21.15.88:0/2973123361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56280da78fe0 0x56280db163b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.762+0000 7f60d3111640 1 -- 172.21.15.88:0/2973123361 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56280da48d10 con 0x56280db1e100 2024-09-16T01:40:13.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.762+0000 7f60cbfff640 1 --2- 172.21.15.88:0/2973123361 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56280db1dd20 0x56280da755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:13.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.762+0000 7f60d3111640 1 --2- 172.21.15.88:0/2973123361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56280db1e100 0x56280da75b30 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f60cc025930 tx=0x7f60cc051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2579150fcafcc40 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.762+0000 7f60d190e640 1 -- 172.21.15.88:0/2973123361 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc069070 con 0x56280db1e100 2024-09-16T01:40:13.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.762+0000 7f60d190e640 1 -- 172.21.15.88:0/2973123361 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60cc0517c0 con 0x56280db1e100 2024-09-16T01:40:13.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.762+0000 7f60d190e640 1 -- 172.21.15.88:0/2973123361 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc055040 con 0x56280db1e100 2024-09-16T01:40:13.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.763+0000 7f60d4e1af40 1 -- 172.21.15.88:0/2973123361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56280db1e100 msgr2=0x56280da75b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.763+0000 7f60d4e1af40 1 --2- 172.21.15.88:0/2973123361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56280db1e100 0x56280da75b30 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f60cc025930 tx=0x7f60cc051000 comp rx=0 tx=0).stop 2024-09-16T01:40:13.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.763+0000 7f60d4e1af40 1 -- 172.21.15.88:0/2973123361 shutdown_connections 2024-09-16T01:40:13.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.763+0000 7f60d4e1af40 1 --2- 172.21.15.88:0/2973123361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56280db1e100 0x56280da75b30 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.763+0000 7f60d4e1af40 1 --2- 172.21.15.88:0/2973123361 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56280db1dd20 0x56280da755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.763+0000 7f60d4e1af40 1 --2- 172.21.15.88:0/2973123361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56280da78fe0 0x56280db163b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.763+0000 7f60d4e1af40 1 -- 172.21.15.88:0/2973123361 >> 172.21.15.88:0/2973123361 conn(0x56280da94750 msgr2=0x56280da93200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:13.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.763+0000 7f60d4e1af40 1 -- 172.21.15.88:0/2973123361 shutdown_connections 2024-09-16T01:40:13.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.763+0000 7f60d4e1af40 1 -- 172.21.15.88:0/2973123361 wait complete. 2024-09-16T01:40:13.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.763+0000 7f60d4e1af40 1 Processor -- start 2024-09-16T01:40:13.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.764+0000 7f60d4e1af40 1 -- start start 2024-09-16T01:40:13.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.764+0000 7f60d4e1af40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56280da78fe0 0x56280db121b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.764+0000 7f60d4e1af40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56280db1dd20 0x56280db19920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.764+0000 7f60d4e1af40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56280db1e100 0x56280da74460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.764+0000 7f60d4e1af40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56280da49290 con 0x56280db1dd20 2024-09-16T01:40:13.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.764+0000 7f60d4e1af40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56280da48fb0 con 0x56280db1e100 2024-09-16T01:40:13.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.764+0000 7f60d4e1af40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56280da497b0 con 0x56280da78fe0 2024-09-16T01:40:13.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.764+0000 7f60cbfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56280db1dd20 0x56280db19920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.764+0000 7f60cbfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56280db1dd20 0x56280db19920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46526/0 (socket says 172.21.15.88:46526) 2024-09-16T01:40:13.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.764+0000 7f60cbfff640 1 -- 172.21.15.88:0/3870466778 learned_addr learned my addr 172.21.15.88:0/3870466778 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:13.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.765+0000 7f60d3111640 1 --2- 172.21.15.88:0/3870466778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56280db1e100 0x56280da74460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.765+0000 7f60d2910640 1 --2- 172.21.15.88:0/3870466778 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56280da78fe0 0x56280db121b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.765+0000 7f60cbfff640 1 -- 172.21.15.88:0/3870466778 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56280da78fe0 msgr2=0x56280db121b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.765+0000 7f60cbfff640 1 --2- 172.21.15.88:0/3870466778 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56280da78fe0 0x56280db121b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.765+0000 7f60cbfff640 1 -- 172.21.15.88:0/3870466778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56280db1e100 msgr2=0x56280da74460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.765+0000 7f60cbfff640 1 --2- 172.21.15.88:0/3870466778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56280db1e100 0x56280da74460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.765+0000 7f60cbfff640 1 -- 172.21.15.88:0/3870466778 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56280da71ed0 con 0x56280db1dd20 2024-09-16T01:40:13.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.765+0000 7f60d3111640 1 --2- 172.21.15.88:0/3870466778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56280db1e100 0x56280da74460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:13.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.765+0000 7f60cbfff640 1 --2- 172.21.15.88:0/3870466778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56280db1dd20 0x56280db19920 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f60bc00dc60 tx=0x7f60bc00b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.765+0000 7f60d090c640 1 -- 172.21.15.88:0/3870466778 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60bc00f650 con 0x56280db1dd20 2024-09-16T01:40:13.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.765+0000 7f60d090c640 1 -- 172.21.15.88:0/3870466778 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60bc00f7f0 con 0x56280db1dd20 2024-09-16T01:40:13.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.765+0000 7f60d090c640 1 -- 172.21.15.88:0/3870466778 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60bc00fbc0 con 0x56280db1dd20 2024-09-16T01:40:13.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.765+0000 7f60d4e1af40 1 -- 172.21.15.88:0/3870466778 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56280da721b0 con 0x56280db1dd20 2024-09-16T01:40:13.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.765+0000 7f60d4e1af40 1 -- 172.21.15.88:0/3870466778 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56280da72770 con 0x56280db1dd20 2024-09-16T01:40:13.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.766+0000 7f60d090c640 1 -- 172.21.15.88:0/3870466778 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f60bc017400 con 0x56280db1dd20 2024-09-16T01:40:13.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.767+0000 7f60d090c640 1 --2- 172.21.15.88:0/3870466778 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f60b406ac90 0x7f60b406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.767+0000 7f60d090c640 1 -- 172.21.15.88:0/3870466778 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f60bc08a9b0 con 0x56280db1dd20 2024-09-16T01:40:13.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.767+0000 7f60d2910640 1 --2- 172.21.15.88:0/3870466778 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f60b406ac90 0x7f60b406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.767+0000 7f60d4e1af40 1 --2- 172.21.15.88:0/3870466778 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56280db16d60 0x56280db1ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.767+0000 7f60d4e1af40 1 -- 172.21.15.88:0/3870466778 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:1 12.15 12:aa448500:::500.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x56280db1f3b0 con 0x56280db16d60 2024-09-16T01:40:13.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.767+0000 7f60d2910640 1 --2- 172.21.15.88:0/3870466778 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f60b406ac90 0x7f60b406d150 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f60c4005fd0 tx=0x7f60c400c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.767+0000 7f60d3111640 1 --2- 172.21.15.88:0/3870466778 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56280db16d60 0x56280db1ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.768+0000 7f60d3111640 1 --2- 172.21.15.88:0/3870466778 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56280db16d60 0x56280db1ee70 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.768+0000 7f60d3111640 1 -- 172.21.15.88:0/3870466778 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(1 500.00000000 [omap-get-keys out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f60cc06d040 con 0x56280db16d60 2024-09-16T01:40:13.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.768+0000 7f60d4e1af40 1 -- 172.21.15.88:0/3870466778 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56280db16d60 msgr2=0x56280db1ee70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.768+0000 7f60d4e1af40 1 --2- 172.21.15.88:0/3870466778 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56280db16d60 0x56280db1ee70 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.768+0000 7f60d4e1af40 1 -- 172.21.15.88:0/3870466778 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f60b406ac90 msgr2=0x7f60b406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.768+0000 7f60d4e1af40 1 --2- 172.21.15.88:0/3870466778 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f60b406ac90 0x7f60b406d150 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f60c4005fd0 tx=0x7f60c400c040 comp rx=0 tx=0).stop 2024-09-16T01:40:13.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.768+0000 7f60d4e1af40 1 -- 172.21.15.88:0/3870466778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56280db1dd20 msgr2=0x56280db19920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.768+0000 7f60d4e1af40 1 --2- 172.21.15.88:0/3870466778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56280db1dd20 0x56280db19920 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f60bc00dc60 tx=0x7f60bc00b6c0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.768+0000 7f60d3111640 1 -- 172.21.15.88:0/3870466778 reap_dead start 2024-09-16T01:40:13.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.768+0000 7f60d4e1af40 1 -- 172.21.15.88:0/3870466778 shutdown_connections 2024-09-16T01:40:13.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.768+0000 7f60d4e1af40 1 -- 172.21.15.88:0/3870466778 >> 172.21.15.88:0/3870466778 conn(0x56280da94750 msgr2=0x56280da93200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:13.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.768+0000 7f60d4e1af40 1 -- 172.21.15.88:0/3870466778 shutdown_connections 2024-09-16T01:40:13.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.768+0000 7f60d4e1af40 1 -- 172.21.15.88:0/3870466778 wait complete. 2024-09-16T01:40:13.776 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys mds_snaptable 2024-09-16T01:40:13.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.857+0000 7fa09084af40 1 Processor -- start 2024-09-16T01:40:13.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.857+0000 7fa09084af40 1 -- start start 2024-09-16T01:40:13.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.857+0000 7fa09084af40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa07c0022d0 0x56457df81c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.857+0000 7fa09084af40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56457defcc60 0x56457df8b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.857+0000 7fa09084af40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56457df82c80 0x56457df8bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.857+0000 7fa09084af40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56457de03510 con 0x7fa07c0022d0 2024-09-16T01:40:13.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.857+0000 7fa09084af40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56457de5cc50 con 0x56457defcc60 2024-09-16T01:40:13.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.857+0000 7fa09084af40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56457df82620 con 0x56457df82c80 2024-09-16T01:40:13.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.857+0000 7fa08e342640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa07c0022d0 0x56457df81c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.857+0000 7fa08e342640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa07c0022d0 0x56457df81c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46554/0 (socket says 172.21.15.88:46554) 2024-09-16T01:40:13.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.857+0000 7fa08e342640 1 -- 172.21.15.88:0/3789445575 learned_addr learned my addr 172.21.15.88:0/3789445575 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:13.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.858+0000 7fa08e342640 1 -- 172.21.15.88:0/3789445575 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56457df82c80 msgr2=0x56457df8bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:13.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.858+0000 7fa08e342640 1 --2- 172.21.15.88:0/3789445575 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56457df82c80 0x56457df8bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.858+0000 7fa08e342640 1 -- 172.21.15.88:0/3789445575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56457defcc60 msgr2=0x56457df8b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:13.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.858+0000 7fa08e342640 1 --2- 172.21.15.88:0/3789445575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56457defcc60 0x56457df8b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.858+0000 7fa08e342640 1 -- 172.21.15.88:0/3789445575 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56457df82a90 con 0x7fa07c0022d0 2024-09-16T01:40:13.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.858+0000 7fa08e342640 1 --2- 172.21.15.88:0/3789445575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa07c0022d0 0x56457df81c00 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fa080025df0 tx=0x7fa080051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32798cccca305155 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.858+0000 7fa08d340640 1 -- 172.21.15.88:0/3789445575 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0800504f0 con 0x7fa07c0022d0 2024-09-16T01:40:13.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.858+0000 7fa08d340640 1 -- 172.21.15.88:0/3789445575 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0800508a0 con 0x7fa07c0022d0 2024-09-16T01:40:13.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.858+0000 7fa08d340640 1 -- 172.21.15.88:0/3789445575 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa080050c90 con 0x7fa07c0022d0 2024-09-16T01:40:13.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.858+0000 7fa09084af40 1 -- 172.21.15.88:0/3789445575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa07c0022d0 msgr2=0x56457df81c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.858+0000 7fa09084af40 1 --2- 172.21.15.88:0/3789445575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa07c0022d0 0x56457df81c00 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fa080025df0 tx=0x7fa080051000 comp rx=0 tx=0).stop 2024-09-16T01:40:13.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.858+0000 7fa09084af40 1 -- 172.21.15.88:0/3789445575 shutdown_connections 2024-09-16T01:40:13.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.858+0000 7fa09084af40 1 --2- 172.21.15.88:0/3789445575 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56457df82c80 0x56457df8bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.858+0000 7fa09084af40 1 --2- 172.21.15.88:0/3789445575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56457defcc60 0x56457df8b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.858+0000 7fa09084af40 1 --2- 172.21.15.88:0/3789445575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa07c0022d0 0x56457df81c00 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.858+0000 7fa09084af40 1 -- 172.21.15.88:0/3789445575 >> 172.21.15.88:0/3789445575 conn(0x56457defc750 msgr2=0x56457df83e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:13.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.858+0000 7fa09084af40 1 -- 172.21.15.88:0/3789445575 shutdown_connections 2024-09-16T01:40:13.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.858+0000 7fa09084af40 1 -- 172.21.15.88:0/3789445575 wait complete. 2024-09-16T01:40:13.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.859+0000 7fa09084af40 1 Processor -- start 2024-09-16T01:40:13.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.859+0000 7fa09084af40 1 -- start start 2024-09-16T01:40:13.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.860+0000 7fa09084af40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56457df82c80 0x56457df761e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.860+0000 7fa09084af40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa07c0022d0 0x56457df84910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.860+0000 7fa09084af40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56457def81c0 0x56457df84e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.860+0000 7fa09084af40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56457e017bd0 con 0x56457def81c0 2024-09-16T01:40:13.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.860+0000 7fa09084af40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56457df846d0 con 0x56457df82c80 2024-09-16T01:40:13.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.860+0000 7fa09084af40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56457defacb0 con 0x7fa07c0022d0 2024-09-16T01:40:13.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.860+0000 7fa08eb43640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56457def81c0 0x56457df84e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.860+0000 7fa08eb43640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56457def81c0 0x56457df84e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46568/0 (socket says 172.21.15.88:46568) 2024-09-16T01:40:13.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.860+0000 7fa08eb43640 1 -- 172.21.15.88:0/1322842762 learned_addr learned my addr 172.21.15.88:0/1322842762 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:13.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.860+0000 7fa08eb43640 1 -- 172.21.15.88:0/1322842762 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa07c0022d0 msgr2=0x56457df84910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.860+0000 7fa087fff640 1 --2- 172.21.15.88:0/1322842762 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa07c0022d0 0x56457df84910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.860+0000 7fa08e342640 1 --2- 172.21.15.88:0/1322842762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56457df82c80 0x56457df761e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.860+0000 7fa08eb43640 1 --2- 172.21.15.88:0/1322842762 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa07c0022d0 0x56457df84910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.860+0000 7fa08eb43640 1 -- 172.21.15.88:0/1322842762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56457df82c80 msgr2=0x56457df761e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.860+0000 7fa08eb43640 1 --2- 172.21.15.88:0/1322842762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56457df82c80 0x56457df761e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.860+0000 7fa08eb43640 1 -- 172.21.15.88:0/1322842762 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56457df76720 con 0x56457def81c0 2024-09-16T01:40:13.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.860+0000 7fa08eb43640 1 --2- 172.21.15.88:0/1322842762 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56457def81c0 0x56457df84e50 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x56457de9f300 tx=0x7fa08800d5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.860+0000 7fa0877fe640 1 -- 172.21.15.88:0/1322842762 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa088018070 con 0x56457def81c0 2024-09-16T01:40:13.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.860+0000 7fa0877fe640 1 -- 172.21.15.88:0/1322842762 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa088013480 con 0x56457def81c0 2024-09-16T01:40:13.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.860+0000 7fa09084af40 1 -- 172.21.15.88:0/1322842762 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56457df86010 con 0x56457def81c0 2024-09-16T01:40:13.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.860+0000 7fa0877fe640 1 -- 172.21.15.88:0/1322842762 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa088013870 con 0x56457def81c0 2024-09-16T01:40:13.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.861+0000 7fa09084af40 1 -- 172.21.15.88:0/1322842762 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56457df865d0 con 0x56457def81c0 2024-09-16T01:40:13.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.862+0000 7fa0877fe640 1 -- 172.21.15.88:0/1322842762 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa08801b070 con 0x56457def81c0 2024-09-16T01:40:13.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.862+0000 7fa0877fe640 1 --2- 172.21.15.88:0/1322842762 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa07006acc0 0x7fa07006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.862+0000 7fa0877fe640 1 -- 172.21.15.88:0/1322842762 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7fa088089430 con 0x56457def81c0 2024-09-16T01:40:13.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.862+0000 7fa08e342640 1 --2- 172.21.15.88:0/1322842762 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa07006acc0 0x7fa07006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.863+0000 7fa09084af40 1 --2- 172.21.15.88:0/1322842762 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x56457e019f90 0x56457e01a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.863+0000 7fa09084af40 1 -- 172.21.15.88:0/1322842762 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:1 12.d 12:b50e409b:::mds_snaptable:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x56457e01a8b0 con 0x56457e019f90 2024-09-16T01:40:13.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.863+0000 7fa08e342640 1 --2- 172.21.15.88:0/1322842762 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa07006acc0 0x7fa07006d180 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa080045c30 tx=0x7fa080053000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.863+0000 7fa087fff640 1 --2- 172.21.15.88:0/1322842762 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x56457e019f90 0x56457e01a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.863+0000 7fa087fff640 1 --2- 172.21.15.88:0/1322842762 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x56457e019f90 0x56457e01a370 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.864+0000 7fa087fff640 1 -- 172.21.15.88:0/1322842762 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(1 mds_snaptable [omap-get-keys out=5b] v0'0 uv1 ondisk = 0) ==== 157+0+5 (crc 0 0 0) 0x7fa078010350 con 0x56457e019f90 2024-09-16T01:40:13.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.864+0000 7fa09084af40 1 -- 172.21.15.88:0/1322842762 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x56457e019f90 msgr2=0x56457e01a370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.864+0000 7fa09084af40 1 --2- 172.21.15.88:0/1322842762 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x56457e019f90 0x56457e01a370 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.864+0000 7fa09084af40 1 -- 172.21.15.88:0/1322842762 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa07006acc0 msgr2=0x7fa07006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.864+0000 7fa09084af40 1 --2- 172.21.15.88:0/1322842762 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa07006acc0 0x7fa07006d180 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa080045c30 tx=0x7fa080053000 comp rx=0 tx=0).stop 2024-09-16T01:40:13.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.864+0000 7fa09084af40 1 -- 172.21.15.88:0/1322842762 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56457def81c0 msgr2=0x56457df84e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.864+0000 7fa09084af40 1 --2- 172.21.15.88:0/1322842762 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56457def81c0 0x56457df84e50 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x56457de9f300 tx=0x7fa08800d5f0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.864+0000 7fa08eb43640 1 -- 172.21.15.88:0/1322842762 reap_dead start 2024-09-16T01:40:13.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.864+0000 7fa09084af40 1 -- 172.21.15.88:0/1322842762 shutdown_connections 2024-09-16T01:40:13.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.864+0000 7fa09084af40 1 -- 172.21.15.88:0/1322842762 >> 172.21.15.88:0/1322842762 conn(0x56457defc750 msgr2=0x56457df83e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:13.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.864+0000 7fa09084af40 1 -- 172.21.15.88:0/1322842762 shutdown_connections 2024-09-16T01:40:13.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.864+0000 7fa09084af40 1 -- 172.21.15.88:0/1322842762 wait complete. 2024-09-16T01:40:13.870 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys mds0_inotable 2024-09-16T01:40:13.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.952+0000 7fa433e1af40 1 Processor -- start 2024-09-16T01:40:13.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.952+0000 7fa433e1af40 1 -- start start 2024-09-16T01:40:13.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.953+0000 7fa433e1af40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5563653e5fe0 0x556365400340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.953+0000 7fa433e1af40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55636548ad20 0x5563654840b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.953+0000 7fa433e1af40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55636548b100 0x5563653b6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.953+0000 7fa433e1af40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x556365308510 con 0x55636548ad20 2024-09-16T01:40:13.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.953+0000 7fa433e1af40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5563653febd0 con 0x55636548b100 2024-09-16T01:40:13.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.953+0000 7fa433e1af40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x556365486f70 con 0x5563653e5fe0 2024-09-16T01:40:13.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.953+0000 7fa4284a2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55636548ad20 0x5563654840b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.953+0000 7fa4284a2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55636548ad20 0x5563654840b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46578/0 (socket says 172.21.15.88:46578) 2024-09-16T01:40:13.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.953+0000 7fa4284a2640 1 -- 172.21.15.88:0/3446991385 learned_addr learned my addr 172.21.15.88:0/3446991385 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:13.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.953+0000 7fa4284a2640 1 -- 172.21.15.88:0/3446991385 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5563653e5fe0 msgr2=0x556365400340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:13.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.953+0000 7fa4284a2640 1 --2- 172.21.15.88:0/3446991385 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5563653e5fe0 0x556365400340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.953+0000 7fa4284a2640 1 -- 172.21.15.88:0/3446991385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55636548b100 msgr2=0x5563653b6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:13.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.953+0000 7fa4284a2640 1 --2- 172.21.15.88:0/3446991385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55636548b100 0x5563653b6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.953+0000 7fa4284a2640 1 -- 172.21.15.88:0/3446991385 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563653a57c0 con 0x55636548ad20 2024-09-16T01:40:13.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.953+0000 7fa4284a2640 1 --2- 172.21.15.88:0/3446991385 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55636548ad20 0x5563654840b0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fa41800b240 tx=0x7fa41800b700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=388799e6baf46f0f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.953+0000 7fa42b7fe640 1 -- 172.21.15.88:0/3446991385 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa418008070 con 0x55636548ad20 2024-09-16T01:40:13.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.953+0000 7fa42b7fe640 1 -- 172.21.15.88:0/3446991385 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa41800d500 con 0x55636548ad20 2024-09-16T01:40:13.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.953+0000 7fa42b7fe640 1 -- 172.21.15.88:0/3446991385 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41800d910 con 0x55636548ad20 2024-09-16T01:40:13.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.954+0000 7fa433e1af40 1 -- 172.21.15.88:0/3446991385 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55636548ad20 msgr2=0x5563654840b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.954+0000 7fa433e1af40 1 --2- 172.21.15.88:0/3446991385 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55636548ad20 0x5563654840b0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fa41800b240 tx=0x7fa41800b700 comp rx=0 tx=0).stop 2024-09-16T01:40:13.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.954+0000 7fa433e1af40 1 -- 172.21.15.88:0/3446991385 shutdown_connections 2024-09-16T01:40:13.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.954+0000 7fa433e1af40 1 --2- 172.21.15.88:0/3446991385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55636548b100 0x5563653b6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.954+0000 7fa433e1af40 1 --2- 172.21.15.88:0/3446991385 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55636548ad20 0x5563654840b0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.954+0000 7fa433e1af40 1 --2- 172.21.15.88:0/3446991385 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5563653e5fe0 0x556365400340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.954+0000 7fa433e1af40 1 -- 172.21.15.88:0/3446991385 >> 172.21.15.88:0/3446991385 conn(0x556365401750 msgr2=0x556365401b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:13.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.954+0000 7fa433e1af40 1 -- 172.21.15.88:0/3446991385 shutdown_connections 2024-09-16T01:40:13.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.954+0000 7fa433e1af40 1 -- 172.21.15.88:0/3446991385 wait complete. 2024-09-16T01:40:13.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.954+0000 7fa433e1af40 1 Processor -- start 2024-09-16T01:40:13.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.954+0000 7fa433e1af40 1 -- start start 2024-09-16T01:40:13.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.955+0000 7fa433e1af40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5563653e5fe0 0x556365483890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.955+0000 7fa433e1af40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55636548ad20 0x55636547a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.955+0000 7fa433e1af40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55636548b100 0x5563654826a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.955+0000 7fa433e1af40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x556365486980 con 0x55636548ad20 2024-09-16T01:40:13.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.955+0000 7fa433e1af40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5563653b68b0 con 0x55636548b100 2024-09-16T01:40:13.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.955+0000 7fa433e1af40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x556365486c90 con 0x5563653e5fe0 2024-09-16T01:40:13.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.955+0000 7fa4284a2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55636548ad20 0x55636547a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.955+0000 7fa4284a2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55636548ad20 0x55636547a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46584/0 (socket says 172.21.15.88:46584) 2024-09-16T01:40:13.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.955+0000 7fa4284a2640 1 -- 172.21.15.88:0/46356120 learned_addr learned my addr 172.21.15.88:0/46356120 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:13.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.955+0000 7fa4314a4640 1 --2- 172.21.15.88:0/46356120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55636548b100 0x5563654826a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.955+0000 7fa430ca3640 1 --2- 172.21.15.88:0/46356120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5563653e5fe0 0x556365483890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.956+0000 7fa4284a2640 1 -- 172.21.15.88:0/46356120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5563653e5fe0 msgr2=0x556365483890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.956+0000 7fa4284a2640 1 --2- 172.21.15.88:0/46356120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5563653e5fe0 0x556365483890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.956+0000 7fa4284a2640 1 -- 172.21.15.88:0/46356120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55636548b100 msgr2=0x5563654826a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.956+0000 7fa4284a2640 1 --2- 172.21.15.88:0/46356120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55636548b100 0x5563654826a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.956+0000 7fa4284a2640 1 -- 172.21.15.88:0/46356120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55636547ad40 con 0x55636548ad20 2024-09-16T01:40:13.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.956+0000 7fa4314a4640 1 --2- 172.21.15.88:0/46356120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55636548b100 0x5563654826a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:13.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.956+0000 7fa4284a2640 1 --2- 172.21.15.88:0/46356120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55636548ad20 0x55636547a260 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fa418000c00 tx=0x7fa418000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.956+0000 7fa42a7fc640 1 -- 172.21.15.88:0/46356120 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa418019470 con 0x55636548ad20 2024-09-16T01:40:13.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.956+0000 7fa42a7fc640 1 -- 172.21.15.88:0/46356120 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa418019610 con 0x55636548ad20 2024-09-16T01:40:13.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.956+0000 7fa433e1af40 1 -- 172.21.15.88:0/46356120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55636547aaa0 con 0x55636548ad20 2024-09-16T01:40:13.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.956+0000 7fa42a7fc640 1 -- 172.21.15.88:0/46356120 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4180199e0 con 0x55636548ad20 2024-09-16T01:40:13.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.956+0000 7fa433e1af40 1 -- 172.21.15.88:0/46356120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563653e5c70 con 0x55636548ad20 2024-09-16T01:40:13.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.957+0000 7fa42a7fc640 1 -- 172.21.15.88:0/46356120 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa418017040 con 0x55636548ad20 2024-09-16T01:40:13.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.958+0000 7fa42a7fc640 1 --2- 172.21.15.88:0/46356120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa41006ac90 0x7fa41006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.958+0000 7fa42a7fc640 1 -- 172.21.15.88:0/46356120 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7fa41808c3e0 con 0x55636548ad20 2024-09-16T01:40:13.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.958+0000 7fa430ca3640 1 --2- 172.21.15.88:0/46356120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa41006ac90 0x7fa41006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.958+0000 7fa433e1af40 1 --2- 172.21.15.88:0/46356120 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x556365485290 0x556365485670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:13.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.958+0000 7fa433e1af40 1 -- 172.21.15.88:0/46356120 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:1 12.13 12:c91d4a1d:::mds0_inotable:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55636547b4a0 con 0x556365485290 2024-09-16T01:40:13.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.958+0000 7fa4314a4640 1 --2- 172.21.15.88:0/46356120 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x556365485290 0x556365485670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:13.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.958+0000 7fa430ca3640 1 --2- 172.21.15.88:0/46356120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa41006ac90 0x7fa41006d150 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa4200030a0 tx=0x7fa4200059a0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.958+0000 7fa4314a4640 1 --2- 172.21.15.88:0/46356120 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x556365485290 0x556365485670 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:13.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.959+0000 7fa4314a4640 1 -- 172.21.15.88:0/46356120 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(1 mds0_inotable [omap-get-keys out=5b] v0'0 uv3 ondisk = 0) ==== 157+0+5 (crc 0 0 0) 0x7fa42c04fba0 con 0x556365485290 2024-09-16T01:40:13.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.959+0000 7fa433e1af40 1 -- 172.21.15.88:0/46356120 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x556365485290 msgr2=0x556365485670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.959+0000 7fa433e1af40 1 --2- 172.21.15.88:0/46356120 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x556365485290 0x556365485670 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.959+0000 7fa433e1af40 1 -- 172.21.15.88:0/46356120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa41006ac90 msgr2=0x7fa41006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.959+0000 7fa433e1af40 1 --2- 172.21.15.88:0/46356120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa41006ac90 0x7fa41006d150 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa4200030a0 tx=0x7fa4200059a0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.959+0000 7fa433e1af40 1 -- 172.21.15.88:0/46356120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55636548ad20 msgr2=0x55636547a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:13.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.959+0000 7fa433e1af40 1 --2- 172.21.15.88:0/46356120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55636548ad20 0x55636547a260 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fa418000c00 tx=0x7fa418000ed0 comp rx=0 tx=0).stop 2024-09-16T01:40:13.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.959+0000 7fa4314a4640 1 -- 172.21.15.88:0/46356120 reap_dead start 2024-09-16T01:40:13.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.959+0000 7fa433e1af40 1 -- 172.21.15.88:0/46356120 shutdown_connections 2024-09-16T01:40:13.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.959+0000 7fa433e1af40 1 -- 172.21.15.88:0/46356120 >> 172.21.15.88:0/46356120 conn(0x556365401750 msgr2=0x556365401b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:13.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.959+0000 7fa433e1af40 1 -- 172.21.15.88:0/46356120 shutdown_connections 2024-09-16T01:40:13.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:13.959+0000 7fa433e1af40 1 -- 172.21.15.88:0/46356120 wait complete. 2024-09-16T01:40:13.966 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 100.00000000 2024-09-16T01:40:14.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.047+0000 7f6f1fa73f40 1 Processor -- start 2024-09-16T01:40:14.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.047+0000 7f6f1fa73f40 1 -- start start 2024-09-16T01:40:14.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.048+0000 7f6f1fa73f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559526fd4480 0x55952706dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.048+0000 7f6f1fa73f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559526ff1e60 0x559526fd77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.048+0000 7f6f1fa73f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55952707df00 0x559526faa710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.048+0000 7f6f1fa73f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55952707e970 con 0x559526fd4480 2024-09-16T01:40:14.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.048+0000 7f6f1fa73f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559526ff2270 con 0x559526ff1e60 2024-09-16T01:40:14.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.048+0000 7f6f1fa73f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55952707ec10 con 0x55952707df00 2024-09-16T01:40:14.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.048+0000 7f6f1d569640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559526fd4480 0x55952706dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.048+0000 7f6f1d569640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559526fd4480 0x55952706dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46602/0 (socket says 172.21.15.88:46602) 2024-09-16T01:40:14.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.048+0000 7f6f1d569640 1 -- 172.21.15.88:0/2568727759 learned_addr learned my addr 172.21.15.88:0/2568727759 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:14.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.048+0000 7f6f1d569640 1 -- 172.21.15.88:0/2568727759 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55952707df00 msgr2=0x559526faa710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:14.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.048+0000 7f6f1d569640 1 --2- 172.21.15.88:0/2568727759 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55952707df00 0x559526faa710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.048+0000 7f6f1d569640 1 -- 172.21.15.88:0/2568727759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559526ff1e60 msgr2=0x559526fd77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:14.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.048+0000 7f6f1d569640 1 --2- 172.21.15.88:0/2568727759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559526ff1e60 0x559526fd77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.048+0000 7f6f1d569640 1 -- 172.21.15.88:0/2568727759 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559526ed3cb0 con 0x559526fd4480 2024-09-16T01:40:14.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.048+0000 7f6f1d569640 1 --2- 172.21.15.88:0/2568727759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559526fd4480 0x55952706dab0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f6f10045ae0 tx=0x7f6f10051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da25b8e35c874c84 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.049+0000 7f6f0f7fe640 1 -- 172.21.15.88:0/2568727759 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f10056050 con 0x559526fd4480 2024-09-16T01:40:14.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.049+0000 7f6f0f7fe640 1 -- 172.21.15.88:0/2568727759 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f10052ad0 con 0x559526fd4480 2024-09-16T01:40:14.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.049+0000 7f6f0f7fe640 1 -- 172.21.15.88:0/2568727759 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f10065440 con 0x559526fd4480 2024-09-16T01:40:14.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.049+0000 7f6f1fa73f40 1 -- 172.21.15.88:0/2568727759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559526fd4480 msgr2=0x55952706dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.049+0000 7f6f1fa73f40 1 --2- 172.21.15.88:0/2568727759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559526fd4480 0x55952706dab0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f6f10045ae0 tx=0x7f6f10051000 comp rx=0 tx=0).stop 2024-09-16T01:40:14.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.049+0000 7f6f1fa73f40 1 -- 172.21.15.88:0/2568727759 shutdown_connections 2024-09-16T01:40:14.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.049+0000 7f6f1fa73f40 1 --2- 172.21.15.88:0/2568727759 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55952707df00 0x559526faa710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.049+0000 7f6f1fa73f40 1 --2- 172.21.15.88:0/2568727759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559526ff1e60 0x559526fd77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.049+0000 7f6f1fa73f40 1 --2- 172.21.15.88:0/2568727759 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559526fd4480 0x55952706dab0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.049+0000 7f6f1fa73f40 1 -- 172.21.15.88:0/2568727759 >> 172.21.15.88:0/2568727759 conn(0x559526fd3960 msgr2=0x559526fd3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:14.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.049+0000 7f6f1fa73f40 1 -- 172.21.15.88:0/2568727759 shutdown_connections 2024-09-16T01:40:14.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.049+0000 7f6f1fa73f40 1 -- 172.21.15.88:0/2568727759 wait complete. 2024-09-16T01:40:14.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.050+0000 7f6f1fa73f40 1 Processor -- start 2024-09-16T01:40:14.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.050+0000 7f6f1fa73f40 1 -- start start 2024-09-16T01:40:14.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.050+0000 7f6f1fa73f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559526fd4480 0x559527070440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.050+0000 7f6f1fa73f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559526ff1e60 0x55952707b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.050+0000 7f6f1fa73f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55952707df00 0x55952707b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.050+0000 7f6f1fa73f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x559526fa9df0 con 0x55952707df00 2024-09-16T01:40:14.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.050+0000 7f6f1fa73f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55952707e970 con 0x559526ff1e60 2024-09-16T01:40:14.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.050+0000 7f6f1fa73f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x559526fef210 con 0x559526fd4480 2024-09-16T01:40:14.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.050+0000 7f6f1dd6a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55952707df00 0x55952707b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.051+0000 7f6f1dd6a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55952707df00 0x55952707b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46604/0 (socket says 172.21.15.88:46604) 2024-09-16T01:40:14.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.051+0000 7f6f1dd6a640 1 -- 172.21.15.88:0/3658677990 learned_addr learned my addr 172.21.15.88:0/3658677990 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:14.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.051+0000 7f6f1cd68640 1 --2- 172.21.15.88:0/3658677990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559526ff1e60 0x55952707b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.051+0000 7f6f1d569640 1 --2- 172.21.15.88:0/3658677990 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559526fd4480 0x559527070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.051+0000 7f6f1dd6a640 1 -- 172.21.15.88:0/3658677990 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559526fd4480 msgr2=0x559527070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.051+0000 7f6f1dd6a640 1 --2- 172.21.15.88:0/3658677990 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559526fd4480 0x559527070440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.051+0000 7f6f1dd6a640 1 -- 172.21.15.88:0/3658677990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559526ff1e60 msgr2=0x55952707b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.051+0000 7f6f1dd6a640 1 --2- 172.21.15.88:0/3658677990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559526ff1e60 0x55952707b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.051+0000 7f6f1dd6a640 1 -- 172.21.15.88:0/3658677990 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595271144d0 con 0x55952707df00 2024-09-16T01:40:14.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.051+0000 7f6f1dd6a640 1 --2- 172.21.15.88:0/3658677990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55952707df00 0x55952707b7f0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800a140 tx=0x7f6f1800a600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.051+0000 7f6f0e7fc640 1 -- 172.21.15.88:0/3658677990 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1800ae30 con 0x55952707df00 2024-09-16T01:40:14.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.051+0000 7f6f0e7fc640 1 -- 172.21.15.88:0/3658677990 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f180115f0 con 0x55952707df00 2024-09-16T01:40:14.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.051+0000 7f6f1fa73f40 1 -- 172.21.15.88:0/3658677990 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5595271147a0 con 0x55952707df00 2024-09-16T01:40:14.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.051+0000 7f6f0e7fc640 1 -- 172.21.15.88:0/3658677990 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f18011a00 con 0x55952707df00 2024-09-16T01:40:14.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.051+0000 7f6f1fa73f40 1 -- 172.21.15.88:0/3658677990 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55952706f960 con 0x55952707df00 2024-09-16T01:40:14.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.053+0000 7f6f0e7fc640 1 -- 172.21.15.88:0/3658677990 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6f18011ba0 con 0x55952707df00 2024-09-16T01:40:14.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.053+0000 7f6f0e7fc640 1 --2- 172.21.15.88:0/3658677990 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6f0006ac90 0x7f6f0006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.053+0000 7f6f0e7fc640 1 -- 172.21.15.88:0/3658677990 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f6f1808a380 con 0x55952707df00 2024-09-16T01:40:14.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.053+0000 7f6f1d569640 1 --2- 172.21.15.88:0/3658677990 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6f0006ac90 0x7f6f0006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.053+0000 7f6f1fa73f40 1 --2- 172.21.15.88:0/3658677990 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55952706a400 0x55952706a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.053+0000 7f6f1fa73f40 1 -- 172.21.15.88:0/3658677990 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:1 12.13 12:cd5a64a3:::100.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55952706dff0 con 0x55952706a400 2024-09-16T01:40:14.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.053+0000 7f6f1cd68640 1 --2- 172.21.15.88:0/3658677990 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55952706a400 0x55952706a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.053+0000 7f6f1d569640 1 --2- 172.21.15.88:0/3658677990 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6f0006ac90 0x7f6f0006d150 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6f10055040 tx=0x7f6f10053000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.054+0000 7f6f1cd68640 1 --2- 172.21.15.88:0/3658677990 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55952706a400 0x55952706a7e0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.054+0000 7f6f1cd68640 1 -- 172.21.15.88:0/3658677990 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(1 100.00000000 [omap-get-keys out=155b] v0'0 uv1 ondisk = 0) ==== 156+0+155 (crc 0 0 0) 0x7f6f14009890 con 0x55952706a400 2024-09-16T01:40:14.059 INFO:teuthology.orchestra.run.smithi088.stdout:stray0_head 2024-09-16T01:40:14.059 INFO:teuthology.orchestra.run.smithi088.stdout:stray1_head 2024-09-16T01:40:14.059 INFO:teuthology.orchestra.run.smithi088.stdout:stray2_head 2024-09-16T01:40:14.059 INFO:teuthology.orchestra.run.smithi088.stdout:stray3_head 2024-09-16T01:40:14.059 INFO:teuthology.orchestra.run.smithi088.stdout:stray4_head 2024-09-16T01:40:14.059 INFO:teuthology.orchestra.run.smithi088.stdout:stray5_head 2024-09-16T01:40:14.059 INFO:teuthology.orchestra.run.smithi088.stdout:stray6_head 2024-09-16T01:40:14.059 INFO:teuthology.orchestra.run.smithi088.stdout:stray7_head 2024-09-16T01:40:14.060 INFO:teuthology.orchestra.run.smithi088.stdout:stray8_head 2024-09-16T01:40:14.060 INFO:teuthology.orchestra.run.smithi088.stdout:stray9_head 2024-09-16T01:40:14.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.054+0000 7f6f1fa73f40 1 -- 172.21.15.88:0/3658677990 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55952706a400 msgr2=0x55952706a7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.054+0000 7f6f1fa73f40 1 --2- 172.21.15.88:0/3658677990 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55952706a400 0x55952706a7e0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.054+0000 7f6f1fa73f40 1 -- 172.21.15.88:0/3658677990 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6f0006ac90 msgr2=0x7f6f0006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.054+0000 7f6f1fa73f40 1 --2- 172.21.15.88:0/3658677990 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6f0006ac90 0x7f6f0006d150 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6f10055040 tx=0x7f6f10053000 comp rx=0 tx=0).stop 2024-09-16T01:40:14.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.054+0000 7f6f1fa73f40 1 -- 172.21.15.88:0/3658677990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55952707df00 msgr2=0x55952707b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.054+0000 7f6f1fa73f40 1 --2- 172.21.15.88:0/3658677990 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55952707df00 0x55952707b7f0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800a140 tx=0x7f6f1800a600 comp rx=0 tx=0).stop 2024-09-16T01:40:14.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.054+0000 7f6f1dd6a640 1 -- 172.21.15.88:0/3658677990 reap_dead start 2024-09-16T01:40:14.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.054+0000 7f6f1fa73f40 1 -- 172.21.15.88:0/3658677990 shutdown_connections 2024-09-16T01:40:14.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.054+0000 7f6f1fa73f40 1 -- 172.21.15.88:0/3658677990 >> 172.21.15.88:0/3658677990 conn(0x559526fd3960 msgr2=0x55952706cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:14.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.054+0000 7f6f1fa73f40 1 -- 172.21.15.88:0/3658677990 shutdown_connections 2024-09-16T01:40:14.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.054+0000 7f6f1fa73f40 1 -- 172.21.15.88:0/3658677990 wait complete. 2024-09-16T01:40:14.062 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys mds0_sessionmap 2024-09-16T01:40:14.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.142+0000 7f87657cef40 1 Processor -- start 2024-09-16T01:40:14.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.142+0000 7f87657cef40 1 -- start start 2024-09-16T01:40:14.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.142+0000 7f87657cef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f34a68efe0 0x55f34a731300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.142+0000 7f87657cef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f34a733d20 0x55f34a68a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.142+0000 7f87657cef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f34a734100 0x55f34a68ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.142+0000 7f87657cef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f34a5b1510 con 0x55f34a733d20 2024-09-16T01:40:14.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.142+0000 7f87657cef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f34a6a7bd0 con 0x55f34a68efe0 2024-09-16T01:40:14.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.142+0000 7f87657cef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55f34a72dbe0 con 0x55f34a734100 2024-09-16T01:40:14.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.143+0000 7f875aac3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f34a733d20 0x55f34a68a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.143+0000 7f875aac3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f34a733d20 0x55f34a68a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46632/0 (socket says 172.21.15.88:46632) 2024-09-16T01:40:14.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.143+0000 7f875aac3640 1 -- 172.21.15.88:0/586398304 learned_addr learned my addr 172.21.15.88:0/586398304 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:14.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.143+0000 7f87632c4640 1 --2- 172.21.15.88:0/586398304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f34a68efe0 0x55f34a731300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.143+0000 7f8763ac5640 1 --2- 172.21.15.88:0/586398304 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f34a734100 0x55f34a68ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.143+0000 7f875aac3640 1 -- 172.21.15.88:0/586398304 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f34a734100 msgr2=0x55f34a68ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:14.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.143+0000 7f875aac3640 1 --2- 172.21.15.88:0/586398304 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f34a734100 0x55f34a68ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.143+0000 7f875aac3640 1 -- 172.21.15.88:0/586398304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f34a68efe0 msgr2=0x55f34a731300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:14.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.143+0000 7f875aac3640 1 --2- 172.21.15.88:0/586398304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f34a68efe0 0x55f34a731300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.143+0000 7f875aac3640 1 -- 172.21.15.88:0/586398304 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f34a6aad80 con 0x55f34a733d20 2024-09-16T01:40:14.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.143+0000 7f875aac3640 1 --2- 172.21.15.88:0/586398304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f34a733d20 0x55f34a68a8d0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f874c00a7a0 tx=0x7f874c00a890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6eb7e80c1d10b0d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.143+0000 7f87622c2640 1 -- 172.21.15.88:0/586398304 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874c018070 con 0x55f34a733d20 2024-09-16T01:40:14.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.143+0000 7f87622c2640 1 -- 172.21.15.88:0/586398304 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f874c007600 con 0x55f34a733d20 2024-09-16T01:40:14.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.143+0000 7f87622c2640 1 -- 172.21.15.88:0/586398304 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874c0079b0 con 0x55f34a733d20 2024-09-16T01:40:14.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.144+0000 7f87657cef40 1 -- 172.21.15.88:0/586398304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f34a733d20 msgr2=0x55f34a68a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.144+0000 7f87657cef40 1 --2- 172.21.15.88:0/586398304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f34a733d20 0x55f34a68a8d0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f874c00a7a0 tx=0x7f874c00a890 comp rx=0 tx=0).stop 2024-09-16T01:40:14.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.144+0000 7f87657cef40 1 -- 172.21.15.88:0/586398304 shutdown_connections 2024-09-16T01:40:14.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.144+0000 7f87657cef40 1 --2- 172.21.15.88:0/586398304 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f34a734100 0x55f34a68ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.144+0000 7f87657cef40 1 --2- 172.21.15.88:0/586398304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f34a733d20 0x55f34a68a8d0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.144+0000 7f87657cef40 1 --2- 172.21.15.88:0/586398304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f34a68efe0 0x55f34a731300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.144+0000 7f87657cef40 1 -- 172.21.15.88:0/586398304 >> 172.21.15.88:0/586398304 conn(0x55f34a6aa750 msgr2=0x55f34a65e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:14.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.144+0000 7f87657cef40 1 -- 172.21.15.88:0/586398304 shutdown_connections 2024-09-16T01:40:14.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.144+0000 7f87657cef40 1 -- 172.21.15.88:0/586398304 wait complete. 2024-09-16T01:40:14.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.144+0000 7f87657cef40 1 Processor -- start 2024-09-16T01:40:14.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.144+0000 7f87657cef40 1 -- start start 2024-09-16T01:40:14.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.145+0000 7f87657cef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f34a68efe0 0x55f34a662a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.145+0000 7f87657cef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f34a733d20 0x55f34a73ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.145+0000 7f87657cef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f34a734100 0x55f34a73fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.145+0000 7f87657cef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f34a72d900 con 0x55f34a734100 2024-09-16T01:40:14.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.145+0000 7f87657cef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f34a6aab30 con 0x55f34a733d20 2024-09-16T01:40:14.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.145+0000 7f87657cef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55f34a72d610 con 0x55f34a68efe0 2024-09-16T01:40:14.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.145+0000 7f8763ac5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f34a734100 0x55f34a73fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.145+0000 7f8763ac5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f34a734100 0x55f34a73fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46644/0 (socket says 172.21.15.88:46644) 2024-09-16T01:40:14.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.145+0000 7f8763ac5640 1 -- 172.21.15.88:0/1969952037 learned_addr learned my addr 172.21.15.88:0/1969952037 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:14.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.145+0000 7f875aac3640 1 --2- 172.21.15.88:0/1969952037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f34a733d20 0x55f34a73ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.145+0000 7f87632c4640 1 --2- 172.21.15.88:0/1969952037 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f34a68efe0 0x55f34a662a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.145+0000 7f8763ac5640 1 -- 172.21.15.88:0/1969952037 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f34a68efe0 msgr2=0x55f34a662a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.145+0000 7f8763ac5640 1 --2- 172.21.15.88:0/1969952037 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f34a68efe0 0x55f34a662a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.145+0000 7f8763ac5640 1 -- 172.21.15.88:0/1969952037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f34a733d20 msgr2=0x55f34a73ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.145+0000 7f8763ac5640 1 --2- 172.21.15.88:0/1969952037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f34a733d20 0x55f34a73ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.145+0000 7f8763ac5640 1 -- 172.21.15.88:0/1969952037 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f34a68e320 con 0x55f34a734100 2024-09-16T01:40:14.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.145+0000 7f875aac3640 1 --2- 172.21.15.88:0/1969952037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f34a733d20 0x55f34a73ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:14.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.146+0000 7f8763ac5640 1 --2- 172.21.15.88:0/1969952037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f34a734100 0x55f34a73fdb0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f875c046060 tx=0x7f875c003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.146+0000 7f87612c0640 1 -- 172.21.15.88:0/1969952037 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875c068070 con 0x55f34a734100 2024-09-16T01:40:14.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.146+0000 7f87612c0640 1 -- 172.21.15.88:0/1969952037 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f875c051410 con 0x55f34a734100 2024-09-16T01:40:14.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.146+0000 7f87612c0640 1 -- 172.21.15.88:0/1969952037 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875c055040 con 0x55f34a734100 2024-09-16T01:40:14.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.146+0000 7f87657cef40 1 -- 172.21.15.88:0/1969952037 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f34a72c870 con 0x55f34a734100 2024-09-16T01:40:14.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.146+0000 7f87657cef40 1 -- 172.21.15.88:0/1969952037 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f34a72cb10 con 0x55f34a734100 2024-09-16T01:40:14.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.147+0000 7f87612c0640 1 -- 172.21.15.88:0/1969952037 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f875c06b070 con 0x55f34a734100 2024-09-16T01:40:14.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.147+0000 7f87612c0640 1 --2- 172.21.15.88:0/1969952037 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f874406ac90 0x7f874406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.148+0000 7f87612c0640 1 -- 172.21.15.88:0/1969952037 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f875c0cdad0 con 0x55f34a734100 2024-09-16T01:40:14.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.148+0000 7f87632c4640 1 --2- 172.21.15.88:0/1969952037 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f874406ac90 0x7f874406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.148+0000 7f87657cef40 1 --2- 172.21.15.88:0/1969952037 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55f34a73f2c0 0x55f34a740800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.148+0000 7f87657cef40 1 -- 172.21.15.88:0/1969952037 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:1 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55f34a732b90 con 0x55f34a73f2c0 2024-09-16T01:40:14.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.148+0000 7f875aac3640 1 --2- 172.21.15.88:0/1969952037 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55f34a73f2c0 0x55f34a740800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.148+0000 7f87632c4640 1 --2- 172.21.15.88:0/1969952037 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f874406ac90 0x7f874406d150 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8754001ed0 tx=0x7f8754001670 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.148+0000 7f875aac3640 1 --2- 172.21.15.88:0/1969952037 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55f34a73f2c0 0x55f34a740800 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.148+0000 7f875aac3640 1 -- 172.21.15.88:0/1969952037 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(1 mds0_sessionmap [omap-get-keys out=5b] v0'0 uv3 ondisk = 0) ==== 159+0+5 (crc 0 0 0) 0x7f874c0077b0 con 0x55f34a73f2c0 2024-09-16T01:40:14.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.148+0000 7f87657cef40 1 -- 172.21.15.88:0/1969952037 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55f34a73f2c0 msgr2=0x55f34a740800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.148+0000 7f87657cef40 1 --2- 172.21.15.88:0/1969952037 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55f34a73f2c0 0x55f34a740800 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.149+0000 7f87657cef40 1 -- 172.21.15.88:0/1969952037 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f874406ac90 msgr2=0x7f874406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.149+0000 7f87657cef40 1 --2- 172.21.15.88:0/1969952037 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f874406ac90 0x7f874406d150 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8754001ed0 tx=0x7f8754001670 comp rx=0 tx=0).stop 2024-09-16T01:40:14.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.149+0000 7f87657cef40 1 -- 172.21.15.88:0/1969952037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f34a734100 msgr2=0x55f34a73fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.149+0000 7f87657cef40 1 --2- 172.21.15.88:0/1969952037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f34a734100 0x55f34a73fdb0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f875c046060 tx=0x7f875c003330 comp rx=0 tx=0).stop 2024-09-16T01:40:14.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.149+0000 7f8763ac5640 1 -- 172.21.15.88:0/1969952037 reap_dead start 2024-09-16T01:40:14.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.149+0000 7f87657cef40 1 -- 172.21.15.88:0/1969952037 shutdown_connections 2024-09-16T01:40:14.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.149+0000 7f87657cef40 1 -- 172.21.15.88:0/1969952037 >> 172.21.15.88:0/1969952037 conn(0x55f34a6aa750 msgr2=0x55f34a65e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:14.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.149+0000 7f87657cef40 1 -- 172.21.15.88:0/1969952037 shutdown_connections 2024-09-16T01:40:14.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.149+0000 7f87657cef40 1 -- 172.21.15.88:0/1969952037 wait complete. 2024-09-16T01:40:14.156 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 400.00000000 2024-09-16T01:40:14.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.236+0000 7f9646cd0f40 1 Processor -- start 2024-09-16T01:40:14.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.236+0000 7f9646cd0f40 1 -- start start 2024-09-16T01:40:14.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.237+0000 7f9646cd0f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558b86fdcf00 0x558b870913e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.237+0000 7f9646cd0f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b87092ed0 0x558b87022fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.237+0000 7f9646cd0f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558b87095190 0x558b87025610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.237+0000 7f9646cd0f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558b86f2c510 con 0x558b87095190 2024-09-16T01:40:14.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.237+0000 7f9646cd0f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558b87094210 con 0x558b87092ed0 2024-09-16T01:40:14.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.237+0000 7f9646cd0f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x558b870aa380 con 0x558b86fdcf00 2024-09-16T01:40:14.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.237+0000 7f9644fc7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558b87095190 0x558b87025610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.237+0000 7f9644fc7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558b87095190 0x558b87025610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46652/0 (socket says 172.21.15.88:46652) 2024-09-16T01:40:14.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.237+0000 7f9644fc7640 1 -- 172.21.15.88:0/3615576964 learned_addr learned my addr 172.21.15.88:0/3615576964 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:14.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.237+0000 7f963bfc5640 1 --2- 172.21.15.88:0/3615576964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b87092ed0 0x558b87022fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.237+0000 7f96447c6640 1 --2- 172.21.15.88:0/3615576964 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558b86fdcf00 0x558b870913e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.237+0000 7f9644fc7640 1 -- 172.21.15.88:0/3615576964 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558b86fdcf00 msgr2=0x558b870913e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:14.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.237+0000 7f9644fc7640 1 --2- 172.21.15.88:0/3615576964 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558b86fdcf00 0x558b870913e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.237+0000 7f9644fc7640 1 -- 172.21.15.88:0/3615576964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b87092ed0 msgr2=0x558b87022fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:14.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.237+0000 7f9644fc7640 1 --2- 172.21.15.88:0/3615576964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b87092ed0 0x558b87022fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.237+0000 7f9644fc7640 1 -- 172.21.15.88:0/3615576964 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b870aa0c0 con 0x558b87095190 2024-09-16T01:40:14.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.237+0000 7f9644fc7640 1 --2- 172.21.15.88:0/3615576964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558b87095190 0x558b87025610 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f963c001690 tx=0x7f963c001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddedfaa0ee0f505 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.237+0000 7f96437c4640 1 -- 172.21.15.88:0/3615576964 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f963c004520 con 0x558b87095190 2024-09-16T01:40:14.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.237+0000 7f96437c4640 1 -- 172.21.15.88:0/3615576964 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f963c0046c0 con 0x558b87095190 2024-09-16T01:40:14.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.238+0000 7f96437c4640 1 -- 172.21.15.88:0/3615576964 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f963c004a70 con 0x558b87095190 2024-09-16T01:40:14.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.238+0000 7f9646cd0f40 1 -- 172.21.15.88:0/3615576964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558b87095190 msgr2=0x558b87025610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.238+0000 7f9646cd0f40 1 --2- 172.21.15.88:0/3615576964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558b87095190 0x558b87025610 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f963c001690 tx=0x7f963c001b50 comp rx=0 tx=0).stop 2024-09-16T01:40:14.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.238+0000 7f9646cd0f40 1 -- 172.21.15.88:0/3615576964 shutdown_connections 2024-09-16T01:40:14.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.238+0000 7f9646cd0f40 1 --2- 172.21.15.88:0/3615576964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558b87095190 0x558b87025610 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.238+0000 7f9646cd0f40 1 --2- 172.21.15.88:0/3615576964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b87092ed0 0x558b87022fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.238+0000 7f9646cd0f40 1 --2- 172.21.15.88:0/3615576964 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558b86fdcf00 0x558b870913e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.238+0000 7f9646cd0f40 1 -- 172.21.15.88:0/3615576964 >> 172.21.15.88:0/3615576964 conn(0x558b86fd9be0 msgr2=0x558b870239d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:14.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.238+0000 7f9646cd0f40 1 -- 172.21.15.88:0/3615576964 shutdown_connections 2024-09-16T01:40:14.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.238+0000 7f9646cd0f40 1 -- 172.21.15.88:0/3615576964 wait complete. 2024-09-16T01:40:14.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.239+0000 7f9646cd0f40 1 Processor -- start 2024-09-16T01:40:14.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.239+0000 7f9646cd0f40 1 -- start start 2024-09-16T01:40:14.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.239+0000 7f9646cd0f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558b86fdcf00 0x558b870989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.239+0000 7f9646cd0f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b87092ed0 0x558b87025610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.239+0000 7f9646cd0f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558b87095190 0x558b87025b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.239+0000 7f9646cd0f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558b87022ab0 con 0x558b87095190 2024-09-16T01:40:14.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.239+0000 7f9646cd0f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558b870b35b0 con 0x558b87092ed0 2024-09-16T01:40:14.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.239+0000 7f9646cd0f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x558b87091920 con 0x558b86fdcf00 2024-09-16T01:40:14.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.239+0000 7f9644fc7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558b87095190 0x558b87025b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.239+0000 7f9644fc7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558b87095190 0x558b87025b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46654/0 (socket says 172.21.15.88:46654) 2024-09-16T01:40:14.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.239+0000 7f9644fc7640 1 -- 172.21.15.88:0/1009489592 learned_addr learned my addr 172.21.15.88:0/1009489592 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:14.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.240+0000 7f963bfc5640 1 --2- 172.21.15.88:0/1009489592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b87092ed0 0x558b87025610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.240+0000 7f96447c6640 1 --2- 172.21.15.88:0/1009489592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558b86fdcf00 0x558b870989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.240+0000 7f9644fc7640 1 -- 172.21.15.88:0/1009489592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558b86fdcf00 msgr2=0x558b870989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.240+0000 7f9644fc7640 1 --2- 172.21.15.88:0/1009489592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558b86fdcf00 0x558b870989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.240+0000 7f9644fc7640 1 -- 172.21.15.88:0/1009489592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b87092ed0 msgr2=0x558b87025610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.240+0000 7f9644fc7640 1 --2- 172.21.15.88:0/1009489592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558b87092ed0 0x558b87025610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.240+0000 7f9644fc7640 1 -- 172.21.15.88:0/1009489592 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b8701f370 con 0x558b87095190 2024-09-16T01:40:14.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.240+0000 7f9644fc7640 1 --2- 172.21.15.88:0/1009489592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558b87095190 0x558b87025b50 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f963c001330 tx=0x7f963c001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.240+0000 7f96427c2640 1 -- 172.21.15.88:0/1009489592 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f963c024070 con 0x558b87095190 2024-09-16T01:40:14.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.240+0000 7f96427c2640 1 -- 172.21.15.88:0/1009489592 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f963c013620 con 0x558b87095190 2024-09-16T01:40:14.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.240+0000 7f96427c2640 1 -- 172.21.15.88:0/1009489592 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f963c018040 con 0x558b87095190 2024-09-16T01:40:14.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.240+0000 7f9646cd0f40 1 -- 172.21.15.88:0/1009489592 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x558b8708e240 con 0x558b87095190 2024-09-16T01:40:14.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.240+0000 7f9646cd0f40 1 -- 172.21.15.88:0/1009489592 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b87138a20 con 0x558b87095190 2024-09-16T01:40:14.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.242+0000 7f96427c2640 1 -- 172.21.15.88:0/1009489592 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f963c010070 con 0x558b87095190 2024-09-16T01:40:14.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.242+0000 7f96427c2640 1 --2- 172.21.15.88:0/1009489592 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f962406ac90 0x7f962406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.242+0000 7f96427c2640 1 -- 172.21.15.88:0/1009489592 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f963c08b0d0 con 0x558b87095190 2024-09-16T01:40:14.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.242+0000 7f96447c6640 1 --2- 172.21.15.88:0/1009489592 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f962406ac90 0x7f962406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.242+0000 7f9646cd0f40 1 --2- 172.21.15.88:0/1009489592 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x558b870a9310 0x558b8701db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.242+0000 7f9646cd0f40 1 -- 172.21.15.88:0/1009489592 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:1 12.f 12:f1f69726:::400.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x558b8701e0b0 con 0x558b870a9310 2024-09-16T01:40:14.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.242+0000 7f96447c6640 1 --2- 172.21.15.88:0/1009489592 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f962406ac90 0x7f962406d150 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f9634005400 tx=0x7f963400c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.243+0000 7f963bfc5640 1 --2- 172.21.15.88:0/1009489592 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x558b870a9310 0x558b8701db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.243+0000 7f963bfc5640 1 --2- 172.21.15.88:0/1009489592 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x558b870a9310 0x558b8701db70 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.243+0000 7f963bfc5640 1 -- 172.21.15.88:0/1009489592 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(1 400.00000000 [omap-get-keys out=5b] v0'0 uv1 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x7f962c054ad0 con 0x558b870a9310 2024-09-16T01:40:14.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.243+0000 7f9646cd0f40 1 -- 172.21.15.88:0/1009489592 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x558b870a9310 msgr2=0x558b8701db70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.243+0000 7f9646cd0f40 1 --2- 172.21.15.88:0/1009489592 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x558b870a9310 0x558b8701db70 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.243+0000 7f9646cd0f40 1 -- 172.21.15.88:0/1009489592 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f962406ac90 msgr2=0x7f962406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.243+0000 7f9646cd0f40 1 --2- 172.21.15.88:0/1009489592 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f962406ac90 0x7f962406d150 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f9634005400 tx=0x7f963400c040 comp rx=0 tx=0).stop 2024-09-16T01:40:14.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.243+0000 7f9646cd0f40 1 -- 172.21.15.88:0/1009489592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558b87095190 msgr2=0x558b87025b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.243+0000 7f9646cd0f40 1 --2- 172.21.15.88:0/1009489592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558b87095190 0x558b87025b50 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f963c001330 tx=0x7f963c001250 comp rx=0 tx=0).stop 2024-09-16T01:40:14.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.243+0000 7f9644fc7640 1 -- 172.21.15.88:0/1009489592 reap_dead start 2024-09-16T01:40:14.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.243+0000 7f9646cd0f40 1 -- 172.21.15.88:0/1009489592 shutdown_connections 2024-09-16T01:40:14.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.243+0000 7f9646cd0f40 1 -- 172.21.15.88:0/1009489592 >> 172.21.15.88:0/1009489592 conn(0x558b86fd9be0 msgr2=0x558b87093550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:14.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.243+0000 7f9646cd0f40 1 -- 172.21.15.88:0/1009489592 shutdown_connections 2024-09-16T01:40:14.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.243+0000 7f9646cd0f40 1 -- 172.21.15.88:0/1009489592 wait complete. 2024-09-16T01:40:14.252 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 100.00000000.inode 2024-09-16T01:40:14.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.333+0000 7f38ba334f40 1 Processor -- start 2024-09-16T01:40:14.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.333+0000 7f38ba334f40 1 -- start start 2024-09-16T01:40:14.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.334+0000 7f38ba334f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be1a7ba940 0x55be1a7ffde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.334+0000 7f38ba334f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55be1a803b30 0x55be1a889ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.334+0000 7f38ba334f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55be1a88ef30 0x55be1a88dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.334+0000 7f38ba334f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55be1a7bb2c0 con 0x55be1a803b30 2024-09-16T01:40:14.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.334+0000 7f38ba334f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55be1a70b510 con 0x55be1a7ba940 2024-09-16T01:40:14.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.334+0000 7f38ba334f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55be1a7b8c80 con 0x55be1a88ef30 2024-09-16T01:40:14.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.334+0000 7f38b7629640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55be1a803b30 0x55be1a889ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.334+0000 7f38b7629640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55be1a803b30 0x55be1a889ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46660/0 (socket says 172.21.15.88:46660) 2024-09-16T01:40:14.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.334+0000 7f38b7629640 1 -- 172.21.15.88:0/1200579812 learned_addr learned my addr 172.21.15.88:0/1200579812 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:14.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.334+0000 7f38b7629640 1 -- 172.21.15.88:0/1200579812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55be1a88ef30 msgr2=0x55be1a88dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:14.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.334+0000 7f38b7629640 1 --2- 172.21.15.88:0/1200579812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55be1a88ef30 0x55be1a88dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.334+0000 7f38b7629640 1 -- 172.21.15.88:0/1200579812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be1a7ba940 msgr2=0x55be1a7ffde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:14.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.334+0000 7f38b7e2a640 1 --2- 172.21.15.88:0/1200579812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be1a7ba940 0x55be1a7ffde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.334+0000 7f38b7629640 1 --2- 172.21.15.88:0/1200579812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be1a7ba940 0x55be1a7ffde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.334+0000 7f38b7629640 1 -- 172.21.15.88:0/1200579812 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be1a7e62d0 con 0x55be1a803b30 2024-09-16T01:40:14.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.334+0000 7f38b7e2a640 1 --2- 172.21.15.88:0/1200579812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be1a7ba940 0x55be1a7ffde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:14.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.334+0000 7f38b7629640 1 --2- 172.21.15.88:0/1200579812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55be1a803b30 0x55be1a889ac0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f38a000a5c0 tx=0x7f38a000aa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6998fe11d03e1a1e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.335+0000 7f38b6627640 1 -- 172.21.15.88:0/1200579812 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a00089f0 con 0x55be1a803b30 2024-09-16T01:40:14.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.335+0000 7f38b6627640 1 -- 172.21.15.88:0/1200579812 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38a000f070 con 0x55be1a803b30 2024-09-16T01:40:14.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.335+0000 7f38b6627640 1 -- 172.21.15.88:0/1200579812 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a0008df0 con 0x55be1a803b30 2024-09-16T01:40:14.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.335+0000 7f38ba334f40 1 -- 172.21.15.88:0/1200579812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55be1a803b30 msgr2=0x55be1a889ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.335+0000 7f38ba334f40 1 --2- 172.21.15.88:0/1200579812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55be1a803b30 0x55be1a889ac0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f38a000a5c0 tx=0x7f38a000aa80 comp rx=0 tx=0).stop 2024-09-16T01:40:14.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.335+0000 7f38ba334f40 1 -- 172.21.15.88:0/1200579812 shutdown_connections 2024-09-16T01:40:14.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.335+0000 7f38ba334f40 1 --2- 172.21.15.88:0/1200579812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55be1a88ef30 0x55be1a88dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.335+0000 7f38ba334f40 1 --2- 172.21.15.88:0/1200579812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55be1a803b30 0x55be1a889ac0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.335+0000 7f38ba334f40 1 --2- 172.21.15.88:0/1200579812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be1a7ba940 0x55be1a7ffde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.335+0000 7f38ba334f40 1 -- 172.21.15.88:0/1200579812 >> 172.21.15.88:0/1200579812 conn(0x55be1a804750 msgr2=0x55be1a7e5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:14.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.335+0000 7f38ba334f40 1 -- 172.21.15.88:0/1200579812 shutdown_connections 2024-09-16T01:40:14.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.335+0000 7f38ba334f40 1 -- 172.21.15.88:0/1200579812 wait complete. 2024-09-16T01:40:14.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.336+0000 7f38ba334f40 1 Processor -- start 2024-09-16T01:40:14.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.336+0000 7f38ba334f40 1 -- start start 2024-09-16T01:40:14.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.336+0000 7f38ba334f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55be1a7ba940 0x55be1a7dc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.336+0000 7f38ba334f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be1a803b30 0x55be1a8856a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.336+0000 7f38ba334f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55be1a88ef30 0x55be1a7fd850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.336+0000 7f38ba334f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55be1a7ffae0 con 0x55be1a88ef30 2024-09-16T01:40:14.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.336+0000 7f38ba334f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55be1a88c0c0 con 0x55be1a803b30 2024-09-16T01:40:14.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.336+0000 7f38ba334f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55be1a800320 con 0x55be1a7ba940 2024-09-16T01:40:14.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.337+0000 7f38b862b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55be1a88ef30 0x55be1a7fd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.337+0000 7f38b862b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55be1a88ef30 0x55be1a7fd850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46662/0 (socket says 172.21.15.88:46662) 2024-09-16T01:40:14.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.337+0000 7f38b862b640 1 -- 172.21.15.88:0/238497823 learned_addr learned my addr 172.21.15.88:0/238497823 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:14.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.337+0000 7f38b7e2a640 1 --2- 172.21.15.88:0/238497823 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55be1a7ba940 0x55be1a7dc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.337+0000 7f38b7629640 1 --2- 172.21.15.88:0/238497823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be1a803b30 0x55be1a8856a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.337+0000 7f38b862b640 1 -- 172.21.15.88:0/238497823 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55be1a7ba940 msgr2=0x55be1a7dc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.337+0000 7f38b862b640 1 --2- 172.21.15.88:0/238497823 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55be1a7ba940 0x55be1a7dc8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.337+0000 7f38b862b640 1 -- 172.21.15.88:0/238497823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be1a803b30 msgr2=0x55be1a8856a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.337+0000 7f38b862b640 1 --2- 172.21.15.88:0/238497823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be1a803b30 0x55be1a8856a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.337+0000 7f38b862b640 1 -- 172.21.15.88:0/238497823 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be1a886f80 con 0x55be1a88ef30 2024-09-16T01:40:14.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.337+0000 7f38b862b640 1 --2- 172.21.15.88:0/238497823 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55be1a88ef30 0x55be1a7fd850 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f38b0025da0 tx=0x7f38b00514a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.337+0000 7f38b5625640 1 -- 172.21.15.88:0/238497823 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b0055070 con 0x55be1a88ef30 2024-09-16T01:40:14.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.337+0000 7f38b5625640 1 -- 172.21.15.88:0/238497823 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38b0051830 con 0x55be1a88ef30 2024-09-16T01:40:14.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.337+0000 7f38ba334f40 1 -- 172.21.15.88:0/238497823 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55be1a886400 con 0x55be1a88ef30 2024-09-16T01:40:14.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.337+0000 7f38b5625640 1 -- 172.21.15.88:0/238497823 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b0051c20 con 0x55be1a88ef30 2024-09-16T01:40:14.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.337+0000 7f38ba334f40 1 -- 172.21.15.88:0/238497823 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be1a804160 con 0x55be1a88ef30 2024-09-16T01:40:14.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.339+0000 7f38b5625640 1 -- 172.21.15.88:0/238497823 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f38b0058070 con 0x55be1a88ef30 2024-09-16T01:40:14.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.339+0000 7f38b5625640 1 --2- 172.21.15.88:0/238497823 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f389806ac90 0x7f389806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.339+0000 7f38b5625640 1 -- 172.21.15.88:0/238497823 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f38b00cdcb0 con 0x55be1a88ef30 2024-09-16T01:40:14.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.339+0000 7f38b7e2a640 1 --2- 172.21.15.88:0/238497823 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f389806ac90 0x7f389806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.339+0000 7f38ba334f40 1 --2- 172.21.15.88:0/238497823 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55be1a8867b0 0x55be1a7fc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.339+0000 7f38ba334f40 1 -- 172.21.15.88:0/238497823 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:1 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55be1a7fcb50 con 0x55be1a8867b0 2024-09-16T01:40:14.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.340+0000 7f38b7e2a640 1 --2- 172.21.15.88:0/238497823 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f389806ac90 0x7f389806d150 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x55be1a7fc300 tx=0x7f38a8007040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.340+0000 7f38b7629640 1 --2- 172.21.15.88:0/238497823 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55be1a8867b0 0x55be1a7fc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.340+0000 7f38b7629640 1 --2- 172.21.15.88:0/238497823 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55be1a8867b0 0x55be1a7fc610 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.340+0000 7f38b7629640 1 -- 172.21.15.88:0/238497823 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(1 100.00000000.inode [omap-get-keys out=5b] v0'0 uv3 ondisk = 0) ==== 162+0+5 (crc 0 0 0) 0x7f38a0008970 con 0x55be1a8867b0 2024-09-16T01:40:14.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.340+0000 7f38ba334f40 1 -- 172.21.15.88:0/238497823 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55be1a8867b0 msgr2=0x55be1a7fc610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.340+0000 7f38ba334f40 1 --2- 172.21.15.88:0/238497823 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55be1a8867b0 0x55be1a7fc610 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.340+0000 7f38ba334f40 1 -- 172.21.15.88:0/238497823 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f389806ac90 msgr2=0x7f389806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.340+0000 7f38ba334f40 1 --2- 172.21.15.88:0/238497823 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f389806ac90 0x7f389806d150 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x55be1a7fc300 tx=0x7f38a8007040 comp rx=0 tx=0).stop 2024-09-16T01:40:14.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.340+0000 7f38ba334f40 1 -- 172.21.15.88:0/238497823 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55be1a88ef30 msgr2=0x55be1a7fd850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.340+0000 7f38ba334f40 1 --2- 172.21.15.88:0/238497823 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55be1a88ef30 0x55be1a7fd850 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f38b0025da0 tx=0x7f38b00514a0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.340+0000 7f38b862b640 1 -- 172.21.15.88:0/238497823 reap_dead start 2024-09-16T01:40:14.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.341+0000 7f38ba334f40 1 -- 172.21.15.88:0/238497823 shutdown_connections 2024-09-16T01:40:14.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.341+0000 7f38ba334f40 1 -- 172.21.15.88:0/238497823 >> 172.21.15.88:0/238497823 conn(0x55be1a804750 msgr2=0x55be1a88cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:14.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.341+0000 7f38ba334f40 1 -- 172.21.15.88:0/238497823 shutdown_connections 2024-09-16T01:40:14.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.341+0000 7f38ba334f40 1 -- 172.21.15.88:0/238497823 wait complete. 2024-09-16T01:40:14.348 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 1.00000000 2024-09-16T01:40:14.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.428+0000 7fb82b8eef40 1 Processor -- start 2024-09-16T01:40:14.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.428+0000 7fb82b8eef40 1 -- start start 2024-09-16T01:40:14.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.429+0000 7fb82b8eef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5623f0a0eee0 0x5623f0acbb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.429+0000 7fb82b8eef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5623f0ae2c90 0x5623f0a3ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.429+0000 7fb82b8eef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623f0ae3070 0x5623f0a56350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.429+0000 7fb82b8eef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5623f0960510 con 0x5623f0ae3070 2024-09-16T01:40:14.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.429+0000 7fb82b8eef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5623f0ae36b0 con 0x5623f0a0eee0 2024-09-16T01:40:14.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.429+0000 7fb82b8eef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5623f0ae24d0 con 0x5623f0ae2c90 2024-09-16T01:40:14.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.429+0000 7fb829be5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623f0ae3070 0x5623f0a56350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.429+0000 7fb829be5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623f0ae3070 0x5623f0a56350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46678/0 (socket says 172.21.15.88:46678) 2024-09-16T01:40:14.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.429+0000 7fb829be5640 1 -- 172.21.15.88:0/3364743800 learned_addr learned my addr 172.21.15.88:0/3364743800 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:14.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.429+0000 7fb8293e4640 1 --2- 172.21.15.88:0/3364743800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5623f0a0eee0 0x5623f0acbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.429+0000 7fb823fff640 1 --2- 172.21.15.88:0/3364743800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5623f0ae2c90 0x5623f0a3ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.429+0000 7fb829be5640 1 -- 172.21.15.88:0/3364743800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5623f0ae2c90 msgr2=0x5623f0a3ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:14.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.429+0000 7fb829be5640 1 --2- 172.21.15.88:0/3364743800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5623f0ae2c90 0x5623f0a3ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.429+0000 7fb829be5640 1 -- 172.21.15.88:0/3364743800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5623f0a0eee0 msgr2=0x5623f0acbb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:14.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.429+0000 7fb829be5640 1 --2- 172.21.15.88:0/3364743800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5623f0a0eee0 0x5623f0acbb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.429+0000 7fb829be5640 1 -- 172.21.15.88:0/3364743800 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623f0a3e2a0 con 0x5623f0ae3070 2024-09-16T01:40:14.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.429+0000 7fb823fff640 1 --2- 172.21.15.88:0/3364743800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5623f0ae2c90 0x5623f0a3ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:14.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.429+0000 7fb829be5640 1 --2- 172.21.15.88:0/3364743800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623f0ae3070 0x5623f0a56350 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fb824001660 tx=0x7fb824001b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e70622d8eb9edcd6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.430+0000 7fb8237fe640 1 -- 172.21.15.88:0/3364743800 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb824016070 con 0x5623f0ae3070 2024-09-16T01:40:14.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.430+0000 7fb8237fe640 1 -- 172.21.15.88:0/3364743800 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb82400d690 con 0x5623f0ae3070 2024-09-16T01:40:14.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.430+0000 7fb8237fe640 1 -- 172.21.15.88:0/3364743800 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82400da60 con 0x5623f0ae3070 2024-09-16T01:40:14.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.430+0000 7fb82b8eef40 1 -- 172.21.15.88:0/3364743800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623f0ae3070 msgr2=0x5623f0a56350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.430+0000 7fb82b8eef40 1 --2- 172.21.15.88:0/3364743800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623f0ae3070 0x5623f0a56350 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fb824001660 tx=0x7fb824001b20 comp rx=0 tx=0).stop 2024-09-16T01:40:14.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.430+0000 7fb82b8eef40 1 -- 172.21.15.88:0/3364743800 shutdown_connections 2024-09-16T01:40:14.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.430+0000 7fb82b8eef40 1 --2- 172.21.15.88:0/3364743800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623f0ae3070 0x5623f0a56350 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.430+0000 7fb82b8eef40 1 --2- 172.21.15.88:0/3364743800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5623f0ae2c90 0x5623f0a3ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.430+0000 7fb82b8eef40 1 --2- 172.21.15.88:0/3364743800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5623f0a0eee0 0x5623f0acbb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.430+0000 7fb82b8eef40 1 -- 172.21.15.88:0/3364743800 >> 172.21.15.88:0/3364743800 conn(0x5623f0a39680 msgr2=0x5623f0adf6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:14.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.430+0000 7fb82b8eef40 1 -- 172.21.15.88:0/3364743800 shutdown_connections 2024-09-16T01:40:14.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.430+0000 7fb82b8eef40 1 -- 172.21.15.88:0/3364743800 wait complete. 2024-09-16T01:40:14.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.431+0000 7fb82b8eef40 1 Processor -- start 2024-09-16T01:40:14.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.431+0000 7fb82b8eef40 1 -- start start 2024-09-16T01:40:14.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.431+0000 7fb82b8eef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5623f0a0eee0 0x5623f0ad65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.431+0000 7fb82b8eef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5623f0ae2c90 0x5623f0a56540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.431+0000 7fb82b8eef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623f0ae3070 0x5623f0add5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.431+0000 7fb82b8eef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5623f0a3e5d0 con 0x5623f0ae3070 2024-09-16T01:40:14.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.431+0000 7fb82b8eef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5623f0ada0a0 con 0x5623f0a0eee0 2024-09-16T01:40:14.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.431+0000 7fb82b8eef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5623f0ae21f0 con 0x5623f0ae2c90 2024-09-16T01:40:14.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.431+0000 7fb829be5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623f0ae3070 0x5623f0add5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.432+0000 7fb829be5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623f0ae3070 0x5623f0add5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46688/0 (socket says 172.21.15.88:46688) 2024-09-16T01:40:14.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.432+0000 7fb829be5640 1 -- 172.21.15.88:0/1706509650 learned_addr learned my addr 172.21.15.88:0/1706509650 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:14.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.432+0000 7fb8293e4640 1 --2- 172.21.15.88:0/1706509650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5623f0a0eee0 0x5623f0ad65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.432+0000 7fb823fff640 1 --2- 172.21.15.88:0/1706509650 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5623f0ae2c90 0x5623f0a56540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.432+0000 7fb829be5640 1 -- 172.21.15.88:0/1706509650 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5623f0ae2c90 msgr2=0x5623f0a56540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.432+0000 7fb829be5640 1 --2- 172.21.15.88:0/1706509650 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5623f0ae2c90 0x5623f0a56540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.432+0000 7fb829be5640 1 -- 172.21.15.88:0/1706509650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5623f0a0eee0 msgr2=0x5623f0ad65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.432+0000 7fb829be5640 1 --2- 172.21.15.88:0/1706509650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5623f0a0eee0 0x5623f0ad65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.432+0000 7fb829be5640 1 -- 172.21.15.88:0/1706509650 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623f0b6c790 con 0x5623f0ae3070 2024-09-16T01:40:14.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.432+0000 7fb829be5640 1 --2- 172.21.15.88:0/1706509650 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623f0ae3070 0x5623f0add5e0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fb824001b60 tx=0x7fb824000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.432+0000 7fb8227fc640 1 -- 172.21.15.88:0/1706509650 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb824020030 con 0x5623f0ae3070 2024-09-16T01:40:14.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.432+0000 7fb8227fc640 1 -- 172.21.15.88:0/1706509650 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb824006c30 con 0x5623f0ae3070 2024-09-16T01:40:14.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.432+0000 7fb82b8eef40 1 -- 172.21.15.88:0/1706509650 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5623f0b6ca30 con 0x5623f0ae3070 2024-09-16T01:40:14.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.432+0000 7fb8227fc640 1 -- 172.21.15.88:0/1706509650 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb824016040 con 0x5623f0ae3070 2024-09-16T01:40:14.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.432+0000 7fb82b8eef40 1 -- 172.21.15.88:0/1706509650 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623f0b6cff0 con 0x5623f0ae3070 2024-09-16T01:40:14.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.434+0000 7fb8227fc640 1 -- 172.21.15.88:0/1706509650 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb824023030 con 0x5623f0ae3070 2024-09-16T01:40:14.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.434+0000 7fb8227fc640 1 --2- 172.21.15.88:0/1706509650 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb80c06acc0 0x7fb80c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.434+0000 7fb8227fc640 1 -- 172.21.15.88:0/1706509650 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7fb82408af80 con 0x5623f0ae3070 2024-09-16T01:40:14.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.434+0000 7fb8293e4640 1 --2- 172.21.15.88:0/1706509650 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb80c06acc0 0x7fb80c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.434+0000 7fb82b8eef40 1 --2- 172.21.15.88:0/1706509650 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5623f0a56a80 0x5623f0b6c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.434+0000 7fb82b8eef40 1 -- 172.21.15.88:0/1706509650 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:1 12.1f 12:ff5b34d6:::1.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5623f0addb20 con 0x5623f0a56a80 2024-09-16T01:40:14.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.434+0000 7fb8293e4640 1 --2- 172.21.15.88:0/1706509650 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb80c06acc0 0x7fb80c06d180 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb81c057000 tx=0x7fb81c052000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.434+0000 7fb823fff640 1 --2- 172.21.15.88:0/1706509650 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5623f0a56a80 0x5623f0b6c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.435+0000 7fb823fff640 1 --2- 172.21.15.88:0/1706509650 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5623f0a56a80 0x5623f0b6c040 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.435+0000 7fb823fff640 1 -- 172.21.15.88:0/1706509650 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(1 1.00000000 [omap-get-keys out=20b] v0'0 uv6 ondisk = 0) ==== 154+0+20 (crc 0 0 0) 0x7fb814007850 con 0x5623f0a56a80 2024-09-16T01:40:14.441 INFO:teuthology.orchestra.run.smithi088.stdout:subdir_head 2024-09-16T01:40:14.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.435+0000 7fb82b8eef40 1 -- 172.21.15.88:0/1706509650 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5623f0a56a80 msgr2=0x5623f0b6c040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.435+0000 7fb82b8eef40 1 --2- 172.21.15.88:0/1706509650 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5623f0a56a80 0x5623f0b6c040 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.435+0000 7fb82b8eef40 1 -- 172.21.15.88:0/1706509650 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb80c06acc0 msgr2=0x7fb80c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.435+0000 7fb82b8eef40 1 --2- 172.21.15.88:0/1706509650 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb80c06acc0 0x7fb80c06d180 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb81c057000 tx=0x7fb81c052000 comp rx=0 tx=0).stop 2024-09-16T01:40:14.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.435+0000 7fb82b8eef40 1 -- 172.21.15.88:0/1706509650 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623f0ae3070 msgr2=0x5623f0add5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.435+0000 7fb82b8eef40 1 --2- 172.21.15.88:0/1706509650 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623f0ae3070 0x5623f0add5e0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fb824001b60 tx=0x7fb824000f40 comp rx=0 tx=0).stop 2024-09-16T01:40:14.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.435+0000 7fb829be5640 1 -- 172.21.15.88:0/1706509650 reap_dead start 2024-09-16T01:40:14.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.435+0000 7fb82b8eef40 1 -- 172.21.15.88:0/1706509650 shutdown_connections 2024-09-16T01:40:14.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.435+0000 7fb82b8eef40 1 -- 172.21.15.88:0/1706509650 >> 172.21.15.88:0/1706509650 conn(0x5623f0a39680 msgr2=0x5623f0ade5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:14.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.435+0000 7fb82b8eef40 1 -- 172.21.15.88:0/1706509650 shutdown_connections 2024-09-16T01:40:14.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.435+0000 7fb82b8eef40 1 -- 172.21.15.88:0/1706509650 wait complete. 2024-09-16T01:40:14.443 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-09-16T01:40:14.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.524+0000 7fae8cae5f40 1 Processor -- start 2024-09-16T01:40:14.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.524+0000 7fae8cae5f40 1 -- start start 2024-09-16T01:40:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.525+0000 7fae8cae5f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f042b68fe0 0x55f042c094d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.525+0000 7fae8cae5f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f042b69a60 0x55f042b80b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.525+0000 7fae8cae5f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f042b83250 0x55f042c0e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.525+0000 7fae8cae5f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f042a8b510 con 0x55f042b68fe0 2024-09-16T01:40:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.525+0000 7fae8cae5f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f042c12120 con 0x55f042b69a60 2024-09-16T01:40:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.525+0000 7fae8cae5f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55f042b82c30 con 0x55f042b83250 2024-09-16T01:40:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.525+0000 7fae8a5db640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f042b68fe0 0x55f042c094d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.525+0000 7fae8a5db640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f042b68fe0 0x55f042c094d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46714/0 (socket says 172.21.15.88:46714) 2024-09-16T01:40:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.525+0000 7fae8a5db640 1 -- 172.21.15.88:0/2735181636 learned_addr learned my addr 172.21.15.88:0/2735181636 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.525+0000 7fae81dda640 1 --2- 172.21.15.88:0/2735181636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f042b69a60 0x55f042b80b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.525+0000 7fae8addc640 1 --2- 172.21.15.88:0/2735181636 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f042b83250 0x55f042c0e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.525+0000 7fae8a5db640 1 -- 172.21.15.88:0/2735181636 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f042b83250 msgr2=0x55f042c0e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.525+0000 7fae8a5db640 1 --2- 172.21.15.88:0/2735181636 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f042b83250 0x55f042c0e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.525+0000 7fae8a5db640 1 -- 172.21.15.88:0/2735181636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f042b69a60 msgr2=0x55f042b80b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:14.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.525+0000 7fae8a5db640 1 --2- 172.21.15.88:0/2735181636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f042b69a60 0x55f042b80b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.525+0000 7fae8a5db640 1 -- 172.21.15.88:0/2735181636 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f042b84d40 con 0x55f042b68fe0 2024-09-16T01:40:14.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.525+0000 7fae8addc640 1 --2- 172.21.15.88:0/2735181636 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f042b83250 0x55f042c0e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:14.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.525+0000 7fae8a5db640 1 --2- 172.21.15.88:0/2735181636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f042b68fe0 0x55f042c094d0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fae7c055040 tx=0x7fae7c051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fef4a37ee27e7a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.525+0000 7fae895d9640 1 -- 172.21.15.88:0/2735181636 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7c056030 con 0x55f042b68fe0 2024-09-16T01:40:14.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.525+0000 7fae895d9640 1 -- 172.21.15.88:0/2735181636 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae7c060070 con 0x55f042b68fe0 2024-09-16T01:40:14.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.525+0000 7fae895d9640 1 -- 172.21.15.88:0/2735181636 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7c065470 con 0x55f042b68fe0 2024-09-16T01:40:14.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.526+0000 7fae8cae5f40 1 -- 172.21.15.88:0/2735181636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f042b68fe0 msgr2=0x55f042c094d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.526+0000 7fae8cae5f40 1 --2- 172.21.15.88:0/2735181636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f042b68fe0 0x55f042c094d0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fae7c055040 tx=0x7fae7c051000 comp rx=0 tx=0).stop 2024-09-16T01:40:14.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.526+0000 7fae8cae5f40 1 -- 172.21.15.88:0/2735181636 shutdown_connections 2024-09-16T01:40:14.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.526+0000 7fae8cae5f40 1 --2- 172.21.15.88:0/2735181636 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f042b83250 0x55f042c0e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.526+0000 7fae8cae5f40 1 --2- 172.21.15.88:0/2735181636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f042b69a60 0x55f042b80b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.526+0000 7fae8cae5f40 1 --2- 172.21.15.88:0/2735181636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f042b68fe0 0x55f042c094d0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.526+0000 7fae8cae5f40 1 -- 172.21.15.88:0/2735181636 >> 172.21.15.88:0/2735181636 conn(0x55f042b84750 msgr2=0x55f042b64400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:14.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.526+0000 7fae8cae5f40 1 -- 172.21.15.88:0/2735181636 shutdown_connections 2024-09-16T01:40:14.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.526+0000 7fae8cae5f40 1 -- 172.21.15.88:0/2735181636 wait complete. 2024-09-16T01:40:14.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.526+0000 7fae8cae5f40 1 Processor -- start 2024-09-16T01:40:14.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.526+0000 7fae8cae5f40 1 -- start start 2024-09-16T01:40:14.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.527+0000 7fae8cae5f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f042b68fe0 0x55f042c0e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.527+0000 7fae8cae5f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f042b69a60 0x55f042c0e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.527+0000 7fae8cae5f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f042b83250 0x55f042cab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.527+0000 7fae8cae5f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f042c0af50 con 0x55f042b69a60 2024-09-16T01:40:14.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.527+0000 7fae8cae5f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f042c112c0 con 0x55f042b68fe0 2024-09-16T01:40:14.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.527+0000 7fae8cae5f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55f042c0f400 con 0x55f042b83250 2024-09-16T01:40:14.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.527+0000 7fae81dda640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f042b69a60 0x55f042c0e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.527+0000 7fae81dda640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f042b69a60 0x55f042c0e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46722/0 (socket says 172.21.15.88:46722) 2024-09-16T01:40:14.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.527+0000 7fae81dda640 1 -- 172.21.15.88:0/43077564 learned_addr learned my addr 172.21.15.88:0/43077564 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:14.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.527+0000 7fae8addc640 1 --2- 172.21.15.88:0/43077564 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f042b83250 0x55f042cab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.527+0000 7fae8a5db640 1 --2- 172.21.15.88:0/43077564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f042b68fe0 0x55f042c0e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.527+0000 7fae81dda640 1 -- 172.21.15.88:0/43077564 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f042b83250 msgr2=0x55f042cab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.528+0000 7fae81dda640 1 --2- 172.21.15.88:0/43077564 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f042b83250 0x55f042cab1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.528+0000 7fae81dda640 1 -- 172.21.15.88:0/43077564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f042b68fe0 msgr2=0x55f042c0e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.528+0000 7fae81dda640 1 --2- 172.21.15.88:0/43077564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f042b68fe0 0x55f042c0e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.528+0000 7fae81dda640 1 -- 172.21.15.88:0/43077564 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f042c07780 con 0x55f042b69a60 2024-09-16T01:40:14.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.528+0000 7fae8addc640 1 --2- 172.21.15.88:0/43077564 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f042b83250 0x55f042cab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:14.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.528+0000 7fae81dda640 1 --2- 172.21.15.88:0/43077564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f042b69a60 0x55f042c0e570 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fae74000c00 tx=0x7fae7400df90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.528+0000 7fae83fff640 1 -- 172.21.15.88:0/43077564 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae74014020 con 0x55f042b69a60 2024-09-16T01:40:14.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.528+0000 7fae83fff640 1 -- 172.21.15.88:0/43077564 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae74010410 con 0x55f042b69a60 2024-09-16T01:40:14.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.528+0000 7fae8cae5f40 1 -- 172.21.15.88:0/43077564 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f042c08ff0 con 0x55f042b69a60 2024-09-16T01:40:14.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.528+0000 7fae83fff640 1 -- 172.21.15.88:0/43077564 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae74008030 con 0x55f042b69a60 2024-09-16T01:40:14.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.528+0000 7fae8cae5f40 1 -- 172.21.15.88:0/43077564 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f042c096e0 con 0x55f042b69a60 2024-09-16T01:40:14.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.529+0000 7fae83fff640 1 -- 172.21.15.88:0/43077564 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fae74017020 con 0x55f042b69a60 2024-09-16T01:40:14.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.530+0000 7fae83fff640 1 --2- 172.21.15.88:0/43077564 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fae6c06ac90 0x7fae6c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.530+0000 7fae83fff640 1 -- 172.21.15.88:0/43077564 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7fae7408a080 con 0x55f042b69a60 2024-09-16T01:40:14.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.530+0000 7fae8a5db640 1 --2- 172.21.15.88:0/43077564 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fae6c06ac90 0x7fae6c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.530+0000 7fae8cae5f40 1 --2- 172.21.15.88:0/43077564 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55f042b68b40 0x55f042c07aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.530+0000 7fae8cae5f40 1 -- 172.21.15.88:0/43077564 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55f042c07fe0 con 0x55f042b68b40 2024-09-16T01:40:14.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.530+0000 7fae8addc640 1 --2- 172.21.15.88:0/43077564 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55f042b68b40 0x55f042c07aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.530+0000 7fae8a5db640 1 --2- 172.21.15.88:0/43077564 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fae6c06ac90 0x7fae6c06d150 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fae7c061000 tx=0x7fae7c053000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.530+0000 7fae8addc640 1 --2- 172.21.15.88:0/43077564 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55f042b68b40 0x55f042c07aa0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.531+0000 7fae8addc640 1 -- 172.21.15.88:0/43077564 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 10000000000.00000000 [stat out=16b] v0'0 uv10 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x7fae8400c7f0 con 0x55f042b68b40 2024-09-16T01:40:14.536 INFO:teuthology.orchestra.run.smithi088.stdout:cephfs_metadata/10000000000.00000000 mtime 2024-09-16T01:40:11.000000+0000, size 0 2024-09-16T01:40:14.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.531+0000 7fae8cae5f40 1 -- 172.21.15.88:0/43077564 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55f042b68b40 msgr2=0x55f042c07aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.531+0000 7fae8cae5f40 1 --2- 172.21.15.88:0/43077564 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55f042b68b40 0x55f042c07aa0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.531+0000 7fae8cae5f40 1 -- 172.21.15.88:0/43077564 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fae6c06ac90 msgr2=0x7fae6c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.531+0000 7fae8cae5f40 1 --2- 172.21.15.88:0/43077564 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fae6c06ac90 0x7fae6c06d150 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fae7c061000 tx=0x7fae7c053000 comp rx=0 tx=0).stop 2024-09-16T01:40:14.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.531+0000 7fae8cae5f40 1 -- 172.21.15.88:0/43077564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f042b69a60 msgr2=0x55f042c0e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.531+0000 7fae8cae5f40 1 --2- 172.21.15.88:0/43077564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f042b69a60 0x55f042c0e570 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fae74000c00 tx=0x7fae7400df90 comp rx=0 tx=0).stop 2024-09-16T01:40:14.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.531+0000 7fae8addc640 1 -- 172.21.15.88:0/43077564 reap_dead start 2024-09-16T01:40:14.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.531+0000 7fae8cae5f40 1 -- 172.21.15.88:0/43077564 shutdown_connections 2024-09-16T01:40:14.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.531+0000 7fae8cae5f40 1 -- 172.21.15.88:0/43077564 >> 172.21.15.88:0/43077564 conn(0x55f042b84750 msgr2=0x55f042b643d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:14.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.531+0000 7fae8cae5f40 1 -- 172.21.15.88:0/43077564 shutdown_connections 2024-09-16T01:40:14.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.531+0000 7fae8cae5f40 1 -- 172.21.15.88:0/43077564 wait complete. 2024-09-16T01:40:14.538 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 600.00000000 2024-09-16T01:40:14.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.619+0000 7f109ab5af40 1 Processor -- start 2024-09-16T01:40:14.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.619+0000 7f109ab5af40 1 -- start start 2024-09-16T01:40:14.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.620+0000 7f109ab5af40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555b5b767940 0x555b5b8398a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.620+0000 7f109ab5af40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555b5b767d50 0x555b5b8cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.620+0000 7f109ab5af40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b5b83bf30 0x555b5b8ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.620+0000 7f109ab5af40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x555b5b7682c0 con 0x555b5b767d50 2024-09-16T01:40:14.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.620+0000 7f109ab5af40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x555b5b6b8510 con 0x555b5b83bf30 2024-09-16T01:40:14.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.620+0000 7f109ab5af40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x555b5b765c80 con 0x555b5b767940 2024-09-16T01:40:14.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.620+0000 7f1097e4f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555b5b767d50 0x555b5b8cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.620+0000 7f1097e4f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555b5b767d50 0x555b5b8cdd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46736/0 (socket says 172.21.15.88:46736) 2024-09-16T01:40:14.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.620+0000 7f1097e4f640 1 -- 172.21.15.88:0/137944499 learned_addr learned my addr 172.21.15.88:0/137944499 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:14.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.620+0000 7f1098e51640 1 --2- 172.21.15.88:0/137944499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b5b83bf30 0x555b5b8ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.620+0000 7f1098650640 1 --2- 172.21.15.88:0/137944499 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555b5b767940 0x555b5b8398a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.620+0000 7f1097e4f640 1 -- 172.21.15.88:0/137944499 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555b5b767940 msgr2=0x555b5b8398a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:14.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.620+0000 7f1097e4f640 1 --2- 172.21.15.88:0/137944499 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555b5b767940 0x555b5b8398a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.620+0000 7f1097e4f640 1 -- 172.21.15.88:0/137944499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b5b83bf30 msgr2=0x555b5b8ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:14.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.620+0000 7f1097e4f640 1 --2- 172.21.15.88:0/137944499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b5b83bf30 0x555b5b8ce2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.620+0000 7f1097e4f640 1 -- 172.21.15.88:0/137944499 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b5b7b1cd0 con 0x555b5b767d50 2024-09-16T01:40:14.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.620+0000 7f1098650640 1 --2- 172.21.15.88:0/137944499 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555b5b767940 0x555b5b8398a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:14.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.620+0000 7f1097e4f640 1 --2- 172.21.15.88:0/137944499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555b5b767d50 0x555b5b8cdd60 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f108c009760 tx=0x7f108c009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51e560108c7790df server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.621+0000 7f1096e4d640 1 -- 172.21.15.88:0/137944499 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108c017070 con 0x555b5b767d50 2024-09-16T01:40:14.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.621+0000 7f1096e4d640 1 -- 172.21.15.88:0/137944499 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f108c0090d0 con 0x555b5b767d50 2024-09-16T01:40:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.621+0000 7f1096e4d640 1 -- 172.21.15.88:0/137944499 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108c006b60 con 0x555b5b767d50 2024-09-16T01:40:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.621+0000 7f109ab5af40 1 -- 172.21.15.88:0/137944499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555b5b767d50 msgr2=0x555b5b8cdd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.621+0000 7f109ab5af40 1 --2- 172.21.15.88:0/137944499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555b5b767d50 0x555b5b8cdd60 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f108c009760 tx=0x7f108c009c20 comp rx=0 tx=0).stop 2024-09-16T01:40:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.621+0000 7f109ab5af40 1 -- 172.21.15.88:0/137944499 shutdown_connections 2024-09-16T01:40:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.621+0000 7f109ab5af40 1 --2- 172.21.15.88:0/137944499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b5b83bf30 0x555b5b8ce2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.621+0000 7f109ab5af40 1 --2- 172.21.15.88:0/137944499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555b5b767d50 0x555b5b8cdd60 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.621+0000 7f109ab5af40 1 --2- 172.21.15.88:0/137944499 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555b5b767940 0x555b5b8398a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.621+0000 7f109ab5af40 1 -- 172.21.15.88:0/137944499 >> 172.21.15.88:0/137944499 conn(0x555b5b7b1750 msgr2=0x555b5b7acdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.621+0000 7f109ab5af40 1 -- 172.21.15.88:0/137944499 shutdown_connections 2024-09-16T01:40:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.621+0000 7f109ab5af40 1 -- 172.21.15.88:0/137944499 wait complete. 2024-09-16T01:40:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.622+0000 7f109ab5af40 1 Processor -- start 2024-09-16T01:40:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.622+0000 7f109ab5af40 1 -- start start 2024-09-16T01:40:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.622+0000 7f109ab5af40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b5b767940 0x555b5b8232f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.622+0000 7f109ab5af40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555b5b767d50 0x555b5b836960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.622+0000 7f109ab5af40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555b5b83bf30 0x555b5b836ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.622+0000 7f109ab5af40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x555b5b834610 con 0x555b5b767d50 2024-09-16T01:40:14.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.622+0000 7f109ab5af40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x555b5b83b1c0 con 0x555b5b767940 2024-09-16T01:40:14.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.622+0000 7f109ab5af40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x555b5b83aee0 con 0x555b5b83bf30 2024-09-16T01:40:14.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.623+0000 7f1097e4f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555b5b767d50 0x555b5b836960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.623+0000 7f1097e4f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555b5b767d50 0x555b5b836960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46740/0 (socket says 172.21.15.88:46740) 2024-09-16T01:40:14.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.623+0000 7f1097e4f640 1 -- 172.21.15.88:0/3986421400 learned_addr learned my addr 172.21.15.88:0/3986421400 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:14.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.623+0000 7f1098650640 1 --2- 172.21.15.88:0/3986421400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b5b767940 0x555b5b8232f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.623+0000 7f1098e51640 1 --2- 172.21.15.88:0/3986421400 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555b5b83bf30 0x555b5b836ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.623+0000 7f1097e4f640 1 -- 172.21.15.88:0/3986421400 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555b5b83bf30 msgr2=0x555b5b836ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.623+0000 7f1097e4f640 1 --2- 172.21.15.88:0/3986421400 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555b5b83bf30 0x555b5b836ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.623+0000 7f1097e4f640 1 -- 172.21.15.88:0/3986421400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b5b767940 msgr2=0x555b5b8232f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.623+0000 7f1097e4f640 1 --2- 172.21.15.88:0/3986421400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b5b767940 0x555b5b8232f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.623+0000 7f1097e4f640 1 -- 172.21.15.88:0/3986421400 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b5b82c430 con 0x555b5b767d50 2024-09-16T01:40:14.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.623+0000 7f1098e51640 1 --2- 172.21.15.88:0/3986421400 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555b5b83bf30 0x555b5b836ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:14.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.623+0000 7f1097e4f640 1 --2- 172.21.15.88:0/3986421400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555b5b767d50 0x555b5b836960 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f108c015040 tx=0x7f108c003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.623+0000 7f1095e4b640 1 -- 172.21.15.88:0/3986421400 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108c017040 con 0x555b5b767d50 2024-09-16T01:40:14.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.623+0000 7f1095e4b640 1 -- 172.21.15.88:0/3986421400 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f108c020070 con 0x555b5b767d50 2024-09-16T01:40:14.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.623+0000 7f1095e4b640 1 -- 172.21.15.88:0/3986421400 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108c01b890 con 0x555b5b767d50 2024-09-16T01:40:14.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.623+0000 7f109ab5af40 1 -- 172.21.15.88:0/3986421400 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x555b5b833fe0 con 0x555b5b767d50 2024-09-16T01:40:14.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.623+0000 7f109ab5af40 1 -- 172.21.15.88:0/3986421400 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b5b82c9b0 con 0x555b5b767d50 2024-09-16T01:40:14.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.624+0000 7f1095e4b640 1 -- 172.21.15.88:0/3986421400 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f108c01ba30 con 0x555b5b767d50 2024-09-16T01:40:14.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.625+0000 7f1095e4b640 1 --2- 172.21.15.88:0/3986421400 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f108406ac90 0x7f108406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.625+0000 7f1098650640 1 --2- 172.21.15.88:0/3986421400 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f108406ac90 0x7f108406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.625+0000 7f1095e4b640 1 -- 172.21.15.88:0/3986421400 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f108c08ad50 con 0x555b5b767d50 2024-09-16T01:40:14.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.625+0000 7f109ab5af40 1 --2- 172.21.15.88:0/3986421400 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x555b5b82d6c0 0x555b5b82daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.625+0000 7f109ab5af40 1 -- 172.21.15.88:0/3986421400 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:0d82a743:::600.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x555b5b82dfe0 con 0x555b5b82d6c0 2024-09-16T01:40:14.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.625+0000 7f1098e51640 1 --2- 172.21.15.88:0/3986421400 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x555b5b82d6c0 0x555b5b82daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.625+0000 7f1098650640 1 --2- 172.21.15.88:0/3986421400 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f108406ac90 0x7f108406d150 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1088001ed0 tx=0x7f10880017d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.626+0000 7f1098e51640 1 --2- 172.21.15.88:0/3986421400 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x555b5b82d6c0 0x555b5b82daa0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.626+0000 7f1098e51640 1 -- 172.21.15.88:0/3986421400 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 600.00000000 [stat out=16b] v0'0 uv2 ondisk = 0) ==== 156+0+16 (crc 0 0 0) 0x7f109004d7a0 con 0x555b5b82d6c0 2024-09-16T01:40:14.631 INFO:teuthology.orchestra.run.smithi088.stdout:cephfs_metadata/600.00000000 mtime 2024-09-16T01:40:04.000000+0000, size 0 2024-09-16T01:40:14.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.626+0000 7f109ab5af40 1 -- 172.21.15.88:0/3986421400 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x555b5b82d6c0 msgr2=0x555b5b82daa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.626+0000 7f109ab5af40 1 --2- 172.21.15.88:0/3986421400 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x555b5b82d6c0 0x555b5b82daa0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.626+0000 7f109ab5af40 1 -- 172.21.15.88:0/3986421400 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f108406ac90 msgr2=0x7f108406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.626+0000 7f109ab5af40 1 --2- 172.21.15.88:0/3986421400 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f108406ac90 0x7f108406d150 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1088001ed0 tx=0x7f10880017d0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.626+0000 7f109ab5af40 1 -- 172.21.15.88:0/3986421400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555b5b767d50 msgr2=0x555b5b836960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.626+0000 7f109ab5af40 1 --2- 172.21.15.88:0/3986421400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555b5b767d50 0x555b5b836960 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f108c015040 tx=0x7f108c003020 comp rx=0 tx=0).stop 2024-09-16T01:40:14.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.626+0000 7f1098e51640 1 -- 172.21.15.88:0/3986421400 reap_dead start 2024-09-16T01:40:14.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.626+0000 7f109ab5af40 1 -- 172.21.15.88:0/3986421400 shutdown_connections 2024-09-16T01:40:14.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.626+0000 7f109ab5af40 1 -- 172.21.15.88:0/3986421400 >> 172.21.15.88:0/3986421400 conn(0x555b5b7b1750 msgr2=0x555b5b83a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:14.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.626+0000 7f109ab5af40 1 -- 172.21.15.88:0/3986421400 shutdown_connections 2024-09-16T01:40:14.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.626+0000 7f109ab5af40 1 -- 172.21.15.88:0/3986421400 wait complete. 2024-09-16T01:40:14.634 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 1.00000000.inode 2024-09-16T01:40:14.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.714+0000 7f442b62ff40 1 Processor -- start 2024-09-16T01:40:14.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.714+0000 7f442b62ff40 1 -- start start 2024-09-16T01:40:14.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.714+0000 7f442b62ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d27f175ee0 0x55d27f1a49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.714+0000 7f442b62ff40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d27f1a6590 0x55d27f247270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.714+0000 7f442b62ff40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d27f249c90 0x55d27f174570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.714+0000 7f442b62ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d27f176840 con 0x55d27f175ee0 2024-09-16T01:40:14.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.714+0000 7f442b62ff40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d27f0c7510 con 0x55d27f1a6590 2024-09-16T01:40:14.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.714+0000 7f442b62ff40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55d27f24a6b0 con 0x55d27f249c90 2024-09-16T01:40:14.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.714+0000 7f4429125640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d27f175ee0 0x55d27f1a49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.715+0000 7f4429125640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d27f175ee0 0x55d27f1a49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46754/0 (socket says 172.21.15.88:46754) 2024-09-16T01:40:14.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.715+0000 7f4429125640 1 -- 172.21.15.88:0/2147273358 learned_addr learned my addr 172.21.15.88:0/2147273358 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:14.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.715+0000 7f4429926640 1 --2- 172.21.15.88:0/2147273358 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d27f249c90 0x55d27f174570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.715+0000 7f4420924640 1 --2- 172.21.15.88:0/2147273358 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d27f1a6590 0x55d27f247270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.715+0000 7f4429125640 1 -- 172.21.15.88:0/2147273358 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d27f249c90 msgr2=0x55d27f174570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:14.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.715+0000 7f4429125640 1 --2- 172.21.15.88:0/2147273358 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d27f249c90 0x55d27f174570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.715+0000 7f4429125640 1 -- 172.21.15.88:0/2147273358 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d27f1a6590 msgr2=0x55d27f247270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:14.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.715+0000 7f4429125640 1 --2- 172.21.15.88:0/2147273358 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d27f1a6590 0x55d27f247270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.715+0000 7f4429125640 1 -- 172.21.15.88:0/2147273358 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d27f245ae0 con 0x55d27f175ee0 2024-09-16T01:40:14.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.715+0000 7f4429926640 1 --2- 172.21.15.88:0/2147273358 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d27f249c90 0x55d27f174570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:14.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.715+0000 7f4429125640 1 --2- 172.21.15.88:0/2147273358 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d27f175ee0 0x55d27f1a49a0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f441c001d10 tx=0x7f441c00b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54abc56d5faf779a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.715+0000 7f4423fff640 1 -- 172.21.15.88:0/2147273358 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441c001e60 con 0x55d27f175ee0 2024-09-16T01:40:14.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.715+0000 7f4423fff640 1 -- 172.21.15.88:0/2147273358 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f441c00bd80 con 0x55d27f175ee0 2024-09-16T01:40:14.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.715+0000 7f4423fff640 1 -- 172.21.15.88:0/2147273358 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441c00f620 con 0x55d27f175ee0 2024-09-16T01:40:14.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.715+0000 7f442b62ff40 1 -- 172.21.15.88:0/2147273358 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d27f175ee0 msgr2=0x55d27f1a49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.715+0000 7f442b62ff40 1 --2- 172.21.15.88:0/2147273358 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d27f175ee0 0x55d27f1a49a0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f441c001d10 tx=0x7f441c00b5b0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.715+0000 7f442b62ff40 1 -- 172.21.15.88:0/2147273358 shutdown_connections 2024-09-16T01:40:14.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.715+0000 7f442b62ff40 1 --2- 172.21.15.88:0/2147273358 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d27f249c90 0x55d27f174570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.715+0000 7f442b62ff40 1 --2- 172.21.15.88:0/2147273358 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d27f1a6590 0x55d27f247270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.715+0000 7f442b62ff40 1 --2- 172.21.15.88:0/2147273358 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d27f175ee0 0x55d27f1a49a0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.715+0000 7f442b62ff40 1 -- 172.21.15.88:0/2147273358 >> 172.21.15.88:0/2147273358 conn(0x55d27f1a0680 msgr2=0x55d27f1a57c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:14.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.716+0000 7f442b62ff40 1 -- 172.21.15.88:0/2147273358 shutdown_connections 2024-09-16T01:40:14.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.716+0000 7f442b62ff40 1 -- 172.21.15.88:0/2147273358 wait complete. 2024-09-16T01:40:14.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.716+0000 7f442b62ff40 1 Processor -- start 2024-09-16T01:40:14.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.716+0000 7f442b62ff40 1 -- start start 2024-09-16T01:40:14.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.717+0000 7f442b62ff40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d27f175ee0 0x55d27f2421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.717+0000 7f442b62ff40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d27f1a6590 0x55d27f174760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.717+0000 7f442b62ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d27f249c90 0x55d27f234490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.717+0000 7f442b62ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d27f176680 con 0x55d27f249c90 2024-09-16T01:40:14.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.717+0000 7f442b62ff40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d27f245d50 con 0x55d27f1a6590 2024-09-16T01:40:14.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.717+0000 7f442b62ff40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55d27f248c50 con 0x55d27f175ee0 2024-09-16T01:40:14.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.717+0000 7f4429926640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d27f249c90 0x55d27f234490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.717+0000 7f4429926640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d27f249c90 0x55d27f234490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46758/0 (socket says 172.21.15.88:46758) 2024-09-16T01:40:14.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.717+0000 7f4429926640 1 -- 172.21.15.88:0/4249374075 learned_addr learned my addr 172.21.15.88:0/4249374075 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:14.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.717+0000 7f4429926640 1 -- 172.21.15.88:0/4249374075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d27f175ee0 msgr2=0x55d27f2421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.717+0000 7f4429926640 1 --2- 172.21.15.88:0/4249374075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d27f175ee0 0x55d27f2421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.717+0000 7f4429926640 1 -- 172.21.15.88:0/4249374075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d27f1a6590 msgr2=0x55d27f174760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.717+0000 7f4429926640 1 --2- 172.21.15.88:0/4249374075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d27f1a6590 0x55d27f174760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.717+0000 7f4429926640 1 -- 172.21.15.88:0/4249374075 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d27f2430e0 con 0x55d27f249c90 2024-09-16T01:40:14.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.717+0000 7f4429926640 1 --2- 172.21.15.88:0/4249374075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d27f249c90 0x55d27f234490 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f442400e2e0 tx=0x7f442400e7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.717+0000 7f4422ffd640 1 -- 172.21.15.88:0/4249374075 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4424013870 con 0x55d27f249c90 2024-09-16T01:40:14.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.717+0000 7f4422ffd640 1 -- 172.21.15.88:0/4249374075 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4424013a10 con 0x55d27f249c90 2024-09-16T01:40:14.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.717+0000 7f4422ffd640 1 -- 172.21.15.88:0/4249374075 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4424013e00 con 0x55d27f249c90 2024-09-16T01:40:14.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.717+0000 7f442b62ff40 1 -- 172.21.15.88:0/4249374075 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d27f242b20 con 0x55d27f249c90 2024-09-16T01:40:14.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.718+0000 7f442b62ff40 1 -- 172.21.15.88:0/4249374075 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d27f22ecd0 con 0x55d27f249c90 2024-09-16T01:40:14.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.719+0000 7f4422ffd640 1 -- 172.21.15.88:0/4249374075 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4424011070 con 0x55d27f249c90 2024-09-16T01:40:14.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.719+0000 7f4422ffd640 1 --2- 172.21.15.88:0/4249374075 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f440c06ac90 0x7f440c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.719+0000 7f4422ffd640 1 -- 172.21.15.88:0/4249374075 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f442408ab90 con 0x55d27f249c90 2024-09-16T01:40:14.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.719+0000 7f4429125640 1 --2- 172.21.15.88:0/4249374075 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f440c06ac90 0x7f440c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.719+0000 7f442b62ff40 1 --2- 172.21.15.88:0/4249374075 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55d27f174ca0 0x55d27f23a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.719+0000 7f442b62ff40 1 -- 172.21.15.88:0/4249374075 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:1 12.14 12:287034c4:::1.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55d27f23ab10 con 0x55d27f174ca0 2024-09-16T01:40:14.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.720+0000 7f4429125640 1 --2- 172.21.15.88:0/4249374075 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f440c06ac90 0x7f440c06d150 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f441c00b5f0 tx=0x7f441c006000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.720+0000 7f4420924640 1 --2- 172.21.15.88:0/4249374075 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55d27f174ca0 0x55d27f23a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.720+0000 7f4420924640 1 --2- 172.21.15.88:0/4249374075 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55d27f174ca0 0x55d27f23a5d0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.720+0000 7f4420924640 1 -- 172.21.15.88:0/4249374075 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(1 1.00000000.inode [stat out=16b] v0'0 uv16 ondisk = 0) ==== 160+0+16 (crc 0 0 0) 0x7f4414057030 con 0x55d27f174ca0 2024-09-16T01:40:14.726 INFO:teuthology.orchestra.run.smithi088.stdout:cephfs_metadata/1.00000000.inode mtime 2024-09-16T01:40:11.000000+0000, size 582 2024-09-16T01:40:14.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.721+0000 7f442b62ff40 1 -- 172.21.15.88:0/4249374075 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55d27f174ca0 msgr2=0x55d27f23a5d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.721+0000 7f442b62ff40 1 --2- 172.21.15.88:0/4249374075 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55d27f174ca0 0x55d27f23a5d0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.721+0000 7f442b62ff40 1 -- 172.21.15.88:0/4249374075 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f440c06ac90 msgr2=0x7f440c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.721+0000 7f442b62ff40 1 --2- 172.21.15.88:0/4249374075 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f440c06ac90 0x7f440c06d150 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f441c00b5f0 tx=0x7f441c006000 comp rx=0 tx=0).stop 2024-09-16T01:40:14.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.721+0000 7f442b62ff40 1 -- 172.21.15.88:0/4249374075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d27f249c90 msgr2=0x55d27f234490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.721+0000 7f442b62ff40 1 --2- 172.21.15.88:0/4249374075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d27f249c90 0x55d27f234490 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f442400e2e0 tx=0x7f442400e7a0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.721+0000 7f4429926640 1 -- 172.21.15.88:0/4249374075 reap_dead start 2024-09-16T01:40:14.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.721+0000 7f442b62ff40 1 -- 172.21.15.88:0/4249374075 shutdown_connections 2024-09-16T01:40:14.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.721+0000 7f442b62ff40 1 -- 172.21.15.88:0/4249374075 >> 172.21.15.88:0/4249374075 conn(0x55d27f1a0680 msgr2=0x55d27f1a49a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:14.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.721+0000 7f442b62ff40 1 -- 172.21.15.88:0/4249374075 shutdown_connections 2024-09-16T01:40:14.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.721+0000 7f442b62ff40 1 -- 172.21.15.88:0/4249374075 wait complete. 2024-09-16T01:40:14.729 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 200.00000000 2024-09-16T01:40:14.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.809+0000 7f1b0e1c5f40 1 Processor -- start 2024-09-16T01:40:14.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.809+0000 7f1b0e1c5f40 1 -- start start 2024-09-16T01:40:14.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.810+0000 7f1b0e1c5f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563e8cf2fa0 0x5563e8dbcfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.810+0000 7f1b0e1c5f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e8cf5230 0x5563e8cf1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.810+0000 7f1b0e1c5f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5563e8dc1540 0x5563e8db83c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.810+0000 7f1b0e1c5f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5563e8d3c5b0 con 0x5563e8cf5230 2024-09-16T01:40:14.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.810+0000 7f1b0e1c5f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5563e8c43510 con 0x5563e8cf2fa0 2024-09-16T01:40:14.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.810+0000 7f1b0e1c5f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5563e8d3b610 con 0x5563e8dc1540 2024-09-16T01:40:14.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.810+0000 7f1b03fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e8cf5230 0x5563e8cf1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.810+0000 7f1b03fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e8cf5230 0x5563e8cf1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46762/0 (socket says 172.21.15.88:46762) 2024-09-16T01:40:14.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.810+0000 7f1b03fff640 1 -- 172.21.15.88:0/3423786653 learned_addr learned my addr 172.21.15.88:0/3423786653 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:14.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.810+0000 7f1b0bcbb640 1 --2- 172.21.15.88:0/3423786653 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563e8cf2fa0 0x5563e8dbcfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.810+0000 7f1b0c4bc640 1 --2- 172.21.15.88:0/3423786653 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5563e8dc1540 0x5563e8db83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.810+0000 7f1b03fff640 1 -- 172.21.15.88:0/3423786653 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5563e8dc1540 msgr2=0x5563e8db83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:14.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.810+0000 7f1b03fff640 1 --2- 172.21.15.88:0/3423786653 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5563e8dc1540 0x5563e8db83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.810+0000 7f1b03fff640 1 -- 172.21.15.88:0/3423786653 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563e8cf2fa0 msgr2=0x5563e8dbcfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:14.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.810+0000 7f1b03fff640 1 --2- 172.21.15.88:0/3423786653 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563e8cf2fa0 0x5563e8dbcfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.810+0000 7f1b03fff640 1 -- 172.21.15.88:0/3423786653 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563e8cf2690 con 0x5563e8cf5230 2024-09-16T01:40:14.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.810+0000 7f1b0c4bc640 1 --2- 172.21.15.88:0/3423786653 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5563e8dc1540 0x5563e8db83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:14.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.810+0000 7f1b03fff640 1 --2- 172.21.15.88:0/3423786653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e8cf5230 0x5563e8cf1c90 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f1af400b310 tx=0x7f1af400b7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3198884b63d6be7f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.810+0000 7f1b0acb9640 1 -- 172.21.15.88:0/3423786653 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af4008070 con 0x5563e8cf5230 2024-09-16T01:40:14.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.810+0000 7f1b0acb9640 1 -- 172.21.15.88:0/3423786653 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1af400d500 con 0x5563e8cf5230 2024-09-16T01:40:14.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.810+0000 7f1b0acb9640 1 -- 172.21.15.88:0/3423786653 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af400d8b0 con 0x5563e8cf5230 2024-09-16T01:40:14.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.811+0000 7f1b0e1c5f40 1 -- 172.21.15.88:0/3423786653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e8cf5230 msgr2=0x5563e8cf1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.811+0000 7f1b0e1c5f40 1 --2- 172.21.15.88:0/3423786653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e8cf5230 0x5563e8cf1c90 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f1af400b310 tx=0x7f1af400b7d0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.811+0000 7f1b0e1c5f40 1 -- 172.21.15.88:0/3423786653 shutdown_connections 2024-09-16T01:40:14.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.811+0000 7f1b0e1c5f40 1 --2- 172.21.15.88:0/3423786653 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5563e8dc1540 0x5563e8db83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.811+0000 7f1b0e1c5f40 1 --2- 172.21.15.88:0/3423786653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e8cf5230 0x5563e8cf1c90 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.811+0000 7f1b0e1c5f40 1 --2- 172.21.15.88:0/3423786653 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563e8cf2fa0 0x5563e8dbcfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.811+0000 7f1b0e1c5f40 1 -- 172.21.15.88:0/3423786653 >> 172.21.15.88:0/3423786653 conn(0x5563e8d3c750 msgr2=0x5563e8cf05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:14.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.811+0000 7f1b0e1c5f40 1 -- 172.21.15.88:0/3423786653 shutdown_connections 2024-09-16T01:40:14.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.811+0000 7f1b0e1c5f40 1 -- 172.21.15.88:0/3423786653 wait complete. 2024-09-16T01:40:14.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.811+0000 7f1b0e1c5f40 1 Processor -- start 2024-09-16T01:40:14.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.811+0000 7f1b0e1c5f40 1 -- start start 2024-09-16T01:40:14.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.812+0000 7f1b0e1c5f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e8cf2fa0 0x5563e8db4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.812+0000 7f1b0e1c5f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563e8cf5230 0x5563e8d1d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.812+0000 7f1b0e1c5f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5563e8dc1540 0x5563e8d1d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.812+0000 7f1b0e1c5f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5563e8cf3e00 con 0x5563e8cf2fa0 2024-09-16T01:40:14.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.812+0000 7f1b0e1c5f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5563e8dc0330 con 0x5563e8cf5230 2024-09-16T01:40:14.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.812+0000 7f1b0e1c5f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5563e8db9290 con 0x5563e8dc1540 2024-09-16T01:40:14.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.812+0000 7f1b0bcbb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e8cf2fa0 0x5563e8db4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.812+0000 7f1b0bcbb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e8cf2fa0 0x5563e8db4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46778/0 (socket says 172.21.15.88:46778) 2024-09-16T01:40:14.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.812+0000 7f1b0bcbb640 1 -- 172.21.15.88:0/3196840619 learned_addr learned my addr 172.21.15.88:0/3196840619 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:14.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.812+0000 7f1b03fff640 1 --2- 172.21.15.88:0/3196840619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563e8cf5230 0x5563e8d1d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.812+0000 7f1b0c4bc640 1 --2- 172.21.15.88:0/3196840619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5563e8dc1540 0x5563e8d1d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.812+0000 7f1b0bcbb640 1 -- 172.21.15.88:0/3196840619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5563e8dc1540 msgr2=0x5563e8d1d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.812+0000 7f1b0bcbb640 1 --2- 172.21.15.88:0/3196840619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5563e8dc1540 0x5563e8d1d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.812+0000 7f1b0bcbb640 1 -- 172.21.15.88:0/3196840619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563e8cf5230 msgr2=0x5563e8d1d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.812+0000 7f1b0bcbb640 1 --2- 172.21.15.88:0/3196840619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563e8cf5230 0x5563e8d1d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.812+0000 7f1b0bcbb640 1 -- 172.21.15.88:0/3196840619 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563e8e5d600 con 0x5563e8cf2fa0 2024-09-16T01:40:14.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.812+0000 7f1b0c4bc640 1 --2- 172.21.15.88:0/3196840619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5563e8dc1540 0x5563e8d1d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:14.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.813+0000 7f1b0bcbb640 1 --2- 172.21.15.88:0/3196840619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e8cf2fa0 0x5563e8db4550 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f1afc055040 tx=0x7f1afc002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.813+0000 7f1b09cb7640 1 -- 172.21.15.88:0/3196840619 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1afc06a070 con 0x5563e8cf2fa0 2024-09-16T01:40:14.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.813+0000 7f1b09cb7640 1 -- 172.21.15.88:0/3196840619 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1afc052b20 con 0x5563e8cf2fa0 2024-09-16T01:40:14.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.813+0000 7f1b0e1c5f40 1 -- 172.21.15.88:0/3196840619 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5563e8e5e380 con 0x5563e8cf2fa0 2024-09-16T01:40:14.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.813+0000 7f1b09cb7640 1 -- 172.21.15.88:0/3196840619 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1afc056040 con 0x5563e8cf2fa0 2024-09-16T01:40:14.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.813+0000 7f1b0e1c5f40 1 -- 172.21.15.88:0/3196840619 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563e8e5db80 con 0x5563e8cf2fa0 2024-09-16T01:40:14.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.814+0000 7f1b09cb7640 1 -- 172.21.15.88:0/3196840619 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1afc06d070 con 0x5563e8cf2fa0 2024-09-16T01:40:14.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.815+0000 7f1b09cb7640 1 --2- 172.21.15.88:0/3196840619 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1aec06ac90 0x7f1aec06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.815+0000 7f1b09cb7640 1 -- 172.21.15.88:0/3196840619 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f1afc0cdb80 con 0x5563e8cf2fa0 2024-09-16T01:40:14.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.815+0000 7f1b03fff640 1 --2- 172.21.15.88:0/3196840619 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1aec06ac90 0x7f1aec06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.815+0000 7f1b0e1c5f40 1 --2- 172.21.15.88:0/3196840619 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5563e8dc5190 0x5563e8e60ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.815+0000 7f1b0e1c5f40 1 -- 172.21.15.88:0/3196840619 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:1 12.14 12:292cf221:::200.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5563e8db89c0 con 0x5563e8dc5190 2024-09-16T01:40:14.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.815+0000 7f1b03fff640 1 --2- 172.21.15.88:0/3196840619 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1aec06ac90 0x7f1aec06d150 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1af400b310 tx=0x7f1af4008000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.815+0000 7f1b0c4bc640 1 --2- 172.21.15.88:0/3196840619 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5563e8dc5190 0x5563e8e60ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.816+0000 7f1b0c4bc640 1 --2- 172.21.15.88:0/3196840619 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5563e8dc5190 0x5563e8e60ef0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.816+0000 7f1b0c4bc640 1 -- 172.21.15.88:0/3196840619 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(1 200.00000000 [stat out=16b] v0'0 uv20 ondisk = 0) ==== 156+0+16 (crc 0 0 0) 0x7f1b0400c860 con 0x5563e8dc5190 2024-09-16T01:40:14.821 INFO:teuthology.orchestra.run.smithi088.stdout:cephfs_metadata/200.00000000 mtime 2024-09-16T01:40:11.000000+0000, size 90 2024-09-16T01:40:14.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.816+0000 7f1b0e1c5f40 1 -- 172.21.15.88:0/3196840619 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5563e8dc5190 msgr2=0x5563e8e60ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.816+0000 7f1b0e1c5f40 1 --2- 172.21.15.88:0/3196840619 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5563e8dc5190 0x5563e8e60ef0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.816+0000 7f1b0e1c5f40 1 -- 172.21.15.88:0/3196840619 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1aec06ac90 msgr2=0x7f1aec06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.816+0000 7f1b0e1c5f40 1 --2- 172.21.15.88:0/3196840619 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1aec06ac90 0x7f1aec06d150 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1af400b310 tx=0x7f1af4008000 comp rx=0 tx=0).stop 2024-09-16T01:40:14.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.816+0000 7f1b0e1c5f40 1 -- 172.21.15.88:0/3196840619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e8cf2fa0 msgr2=0x5563e8db4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.816+0000 7f1b0e1c5f40 1 --2- 172.21.15.88:0/3196840619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563e8cf2fa0 0x5563e8db4550 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f1afc055040 tx=0x7f1afc002fb0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.816+0000 7f1b0c4bc640 1 -- 172.21.15.88:0/3196840619 reap_dead start 2024-09-16T01:40:14.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.816+0000 7f1b0e1c5f40 1 -- 172.21.15.88:0/3196840619 shutdown_connections 2024-09-16T01:40:14.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.816+0000 7f1b0e1c5f40 1 -- 172.21.15.88:0/3196840619 >> 172.21.15.88:0/3196840619 conn(0x5563e8d3c750 msgr2=0x5563e8cf05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:14.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.816+0000 7f1b0e1c5f40 1 -- 172.21.15.88:0/3196840619 shutdown_connections 2024-09-16T01:40:14.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.816+0000 7f1b0e1c5f40 1 -- 172.21.15.88:0/3196840619 wait complete. 2024-09-16T01:40:14.824 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 200.00000001 2024-09-16T01:40:14.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.904+0000 7fba60ffff40 1 Processor -- start 2024-09-16T01:40:14.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.904+0000 7fba60ffff40 1 -- start start 2024-09-16T01:40:14.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.904+0000 7fba60ffff40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ef1c3b6720 0x55ef1c3ab3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.904+0000 7fba60ffff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef1c3be2f0 0x55ef1c3b00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.904+0000 7fba60ffff40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ef1c3c0020 0x55ef1c3b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.904+0000 7fba60ffff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55ef1c3bea00 con 0x55ef1c3be2f0 2024-09-16T01:40:14.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.904+0000 7fba60ffff40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ef1c236510 con 0x55ef1c3c0020 2024-09-16T01:40:14.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.904+0000 7fba60ffff40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55ef1c3beca0 con 0x55ef1c3b6720 2024-09-16T01:40:14.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.904+0000 7fba562f4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef1c3be2f0 0x55ef1c3b00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.904+0000 7fba562f4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef1c3be2f0 0x55ef1c3b00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46798/0 (socket says 172.21.15.88:46798) 2024-09-16T01:40:14.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.904+0000 7fba562f4640 1 -- 172.21.15.88:0/2733723791 learned_addr learned my addr 172.21.15.88:0/2733723791 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:14.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.905+0000 7fba5f2f6640 1 --2- 172.21.15.88:0/2733723791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ef1c3c0020 0x55ef1c3b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.905+0000 7fba562f4640 1 -- 172.21.15.88:0/2733723791 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ef1c3b6720 msgr2=0x55ef1c3ab3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:14.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.905+0000 7fba5eaf5640 1 --2- 172.21.15.88:0/2733723791 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ef1c3b6720 0x55ef1c3ab3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.905+0000 7fba562f4640 1 --2- 172.21.15.88:0/2733723791 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ef1c3b6720 0x55ef1c3ab3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.905+0000 7fba562f4640 1 -- 172.21.15.88:0/2733723791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ef1c3c0020 msgr2=0x55ef1c3b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:14.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.905+0000 7fba562f4640 1 --2- 172.21.15.88:0/2733723791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ef1c3c0020 0x55ef1c3b05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.905+0000 7fba562f4640 1 -- 172.21.15.88:0/2733723791 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef1c32bfb0 con 0x55ef1c3be2f0 2024-09-16T01:40:14.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.905+0000 7fba5f2f6640 1 --2- 172.21.15.88:0/2733723791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ef1c3c0020 0x55ef1c3b05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:14.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.905+0000 7fba5eaf5640 1 --2- 172.21.15.88:0/2733723791 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ef1c3b6720 0x55ef1c3ab3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:14.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.905+0000 7fba562f4640 1 --2- 172.21.15.88:0/2733723791 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef1c3be2f0 0x55ef1c3b00a0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fba4800a7a0 tx=0x7fba4800a890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2b6ef32254b7524 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.905+0000 7fba5daf3640 1 -- 172.21.15.88:0/2733723791 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba48018070 con 0x55ef1c3be2f0 2024-09-16T01:40:14.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.905+0000 7fba5daf3640 1 -- 172.21.15.88:0/2733723791 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba48007600 con 0x55ef1c3be2f0 2024-09-16T01:40:14.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.905+0000 7fba5daf3640 1 -- 172.21.15.88:0/2733723791 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba480079b0 con 0x55ef1c3be2f0 2024-09-16T01:40:14.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.905+0000 7fba60ffff40 1 -- 172.21.15.88:0/2733723791 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef1c3be2f0 msgr2=0x55ef1c3b00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.905+0000 7fba60ffff40 1 --2- 172.21.15.88:0/2733723791 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef1c3be2f0 0x55ef1c3b00a0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fba4800a7a0 tx=0x7fba4800a890 comp rx=0 tx=0).stop 2024-09-16T01:40:14.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.905+0000 7fba60ffff40 1 -- 172.21.15.88:0/2733723791 shutdown_connections 2024-09-16T01:40:14.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.905+0000 7fba60ffff40 1 --2- 172.21.15.88:0/2733723791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ef1c3c0020 0x55ef1c3b05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.905+0000 7fba60ffff40 1 --2- 172.21.15.88:0/2733723791 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef1c3be2f0 0x55ef1c3b00a0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.905+0000 7fba60ffff40 1 --2- 172.21.15.88:0/2733723791 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ef1c3b6720 0x55ef1c3ab3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.905+0000 7fba60ffff40 1 -- 172.21.15.88:0/2733723791 >> 172.21.15.88:0/2733723791 conn(0x55ef1c32f750 msgr2=0x55ef1c3b31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:14.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.906+0000 7fba60ffff40 1 -- 172.21.15.88:0/2733723791 shutdown_connections 2024-09-16T01:40:14.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.906+0000 7fba60ffff40 1 -- 172.21.15.88:0/2733723791 wait complete. 2024-09-16T01:40:14.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.906+0000 7fba60ffff40 1 Processor -- start 2024-09-16T01:40:14.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.906+0000 7fba60ffff40 1 -- start start 2024-09-16T01:40:14.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.907+0000 7fba60ffff40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ef1c3b6720 0x55ef1c310290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.907+0000 7fba60ffff40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ef1c3be2f0 0x55ef1c326970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.907+0000 7fba60ffff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef1c3c0020 0x55ef1c3acee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.907+0000 7fba60ffff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55ef1c32c870 con 0x55ef1c3c0020 2024-09-16T01:40:14.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.907+0000 7fba60ffff40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ef1c32c2f0 con 0x55ef1c3be2f0 2024-09-16T01:40:14.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.907+0000 7fba60ffff40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55ef1c32bcd0 con 0x55ef1c3b6720 2024-09-16T01:40:14.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.907+0000 7fba5f2f6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef1c3c0020 0x55ef1c3acee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.907+0000 7fba5f2f6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef1c3c0020 0x55ef1c3acee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46808/0 (socket says 172.21.15.88:46808) 2024-09-16T01:40:14.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.907+0000 7fba5f2f6640 1 -- 172.21.15.88:0/4109119334 learned_addr learned my addr 172.21.15.88:0/4109119334 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:14.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.907+0000 7fba5eaf5640 1 --2- 172.21.15.88:0/4109119334 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ef1c3b6720 0x55ef1c310290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.907+0000 7fba562f4640 1 --2- 172.21.15.88:0/4109119334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ef1c3be2f0 0x55ef1c326970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.907+0000 7fba5f2f6640 1 -- 172.21.15.88:0/4109119334 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ef1c3b6720 msgr2=0x55ef1c310290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.907+0000 7fba5f2f6640 1 --2- 172.21.15.88:0/4109119334 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ef1c3b6720 0x55ef1c310290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.907+0000 7fba5f2f6640 1 -- 172.21.15.88:0/4109119334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ef1c3be2f0 msgr2=0x55ef1c326970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.907+0000 7fba5f2f6640 1 --2- 172.21.15.88:0/4109119334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ef1c3be2f0 0x55ef1c326970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.907+0000 7fba5f2f6640 1 -- 172.21.15.88:0/4109119334 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef1c3ac420 con 0x55ef1c3c0020 2024-09-16T01:40:14.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.907+0000 7fba5f2f6640 1 --2- 172.21.15.88:0/4109119334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef1c3c0020 0x55ef1c3acee0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fba58046140 tx=0x7fba58003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.907+0000 7fba5caf1640 1 -- 172.21.15.88:0/4109119334 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba58069070 con 0x55ef1c3c0020 2024-09-16T01:40:14.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.907+0000 7fba5caf1640 1 -- 172.21.15.88:0/4109119334 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba58059450 con 0x55ef1c3c0020 2024-09-16T01:40:14.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.907+0000 7fba60ffff40 1 -- 172.21.15.88:0/4109119334 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ef1c3ac700 con 0x55ef1c3c0020 2024-09-16T01:40:14.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.908+0000 7fba5caf1640 1 -- 172.21.15.88:0/4109119334 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba58054040 con 0x55ef1c3c0020 2024-09-16T01:40:14.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.908+0000 7fba60ffff40 1 -- 172.21.15.88:0/4109119334 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef1c3ab600 con 0x55ef1c3c0020 2024-09-16T01:40:14.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.909+0000 7fba5caf1640 1 -- 172.21.15.88:0/4109119334 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fba5806c070 con 0x55ef1c3c0020 2024-09-16T01:40:14.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.909+0000 7fba5caf1640 1 --2- 172.21.15.88:0/4109119334 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba4006ac90 0x7fba4006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.909+0000 7fba5caf1640 1 -- 172.21.15.88:0/4109119334 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7fba580cdc90 con 0x55ef1c3c0020 2024-09-16T01:40:14.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.909+0000 7fba5eaf5640 1 --2- 172.21.15.88:0/4109119334 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba4006ac90 0x7fba4006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.909+0000 7fba60ffff40 1 --2- 172.21.15.88:0/4109119334 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55ef1c3a9e20 0x55ef1c3ae070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:14.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.909+0000 7fba60ffff40 1 -- 172.21.15.88:0/4109119334 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:1 12.14 12:2e2fa760:::200.00000001:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55ef1c3aa8e0 con 0x55ef1c3a9e20 2024-09-16T01:40:14.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.910+0000 7fba562f4640 1 --2- 172.21.15.88:0/4109119334 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55ef1c3a9e20 0x55ef1c3ae070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:14.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.910+0000 7fba5eaf5640 1 --2- 172.21.15.88:0/4109119334 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba4006ac90 0x7fba4006d150 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fba50000c00 tx=0x7fba50001720 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.910+0000 7fba562f4640 1 --2- 172.21.15.88:0/4109119334 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55ef1c3a9e20 0x55ef1c3ae070 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:14.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.910+0000 7fba562f4640 1 -- 172.21.15.88:0/4109119334 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(1 200.00000001 [stat out=16b] v0'0 uv17 ondisk = 0) ==== 156+0+16 (crc 0 0 0) 0x7fba48007c10 con 0x55ef1c3a9e20 2024-09-16T01:40:14.915 INFO:teuthology.orchestra.run.smithi088.stdout:cephfs_metadata/200.00000001 mtime 2024-09-16T01:40:11.000000+0000, size 17490 2024-09-16T01:40:14.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.910+0000 7fba60ffff40 1 -- 172.21.15.88:0/4109119334 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55ef1c3a9e20 msgr2=0x55ef1c3ae070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.910+0000 7fba60ffff40 1 --2- 172.21.15.88:0/4109119334 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55ef1c3a9e20 0x55ef1c3ae070 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:14.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.911+0000 7fba60ffff40 1 -- 172.21.15.88:0/4109119334 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba4006ac90 msgr2=0x7fba4006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.911+0000 7fba60ffff40 1 --2- 172.21.15.88:0/4109119334 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba4006ac90 0x7fba4006d150 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fba50000c00 tx=0x7fba50001720 comp rx=0 tx=0).stop 2024-09-16T01:40:14.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.911+0000 7fba60ffff40 1 -- 172.21.15.88:0/4109119334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef1c3c0020 msgr2=0x55ef1c3acee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:14.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.911+0000 7fba60ffff40 1 --2- 172.21.15.88:0/4109119334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef1c3c0020 0x55ef1c3acee0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fba58046140 tx=0x7fba58003330 comp rx=0 tx=0).stop 2024-09-16T01:40:14.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.911+0000 7fba5f2f6640 1 -- 172.21.15.88:0/4109119334 reap_dead start 2024-09-16T01:40:14.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.911+0000 7fba60ffff40 1 -- 172.21.15.88:0/4109119334 shutdown_connections 2024-09-16T01:40:14.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.911+0000 7fba60ffff40 1 -- 172.21.15.88:0/4109119334 >> 172.21.15.88:0/4109119334 conn(0x55ef1c32f750 msgr2=0x55ef1c32b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:14.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.911+0000 7fba60ffff40 1 -- 172.21.15.88:0/4109119334 shutdown_connections 2024-09-16T01:40:14.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.911+0000 7fba60ffff40 1 -- 172.21.15.88:0/4109119334 wait complete. 2024-09-16T01:40:14.918 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat mds0_openfiles.0 2024-09-16T01:40:14.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.998+0000 7ff779bf6f40 1 Processor -- start 2024-09-16T01:40:15.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.998+0000 7ff779bf6f40 1 -- start start 2024-09-16T01:40:15.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.999+0000 7ff779bf6f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559770b32fe0 0x559770b4c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.999+0000 7ff779bf6f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559770b33a60 0x559770bd8b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.999+0000 7ff779bf6f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559770b4d250 0x559770bcff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.999+0000 7ff779bf6f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x559770a55510 con 0x559770b32fe0 2024-09-16T01:40:15.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.999+0000 7ff779bf6f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559770aaec50 con 0x559770b33a60 2024-09-16T01:40:15.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.999+0000 7ff779bf6f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x559770bd8670 con 0x559770b4d250 2024-09-16T01:40:15.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.999+0000 7ff7776ec640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559770b32fe0 0x559770b4c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.999+0000 7ff7776ec640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559770b32fe0 0x559770b4c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46830/0 (socket says 172.21.15.88:46830) 2024-09-16T01:40:15.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.999+0000 7ff7776ec640 1 -- 172.21.15.88:0/2719551113 learned_addr learned my addr 172.21.15.88:0/2719551113 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:15.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.999+0000 7ff76ffff640 1 --2- 172.21.15.88:0/2719551113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559770b33a60 0x559770bd8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.999+0000 7ff777eed640 1 --2- 172.21.15.88:0/2719551113 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559770b4d250 0x559770bcff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.999+0000 7ff7776ec640 1 -- 172.21.15.88:0/2719551113 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559770b4d250 msgr2=0x559770bcff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:15.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.999+0000 7ff7776ec640 1 --2- 172.21.15.88:0/2719551113 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559770b4d250 0x559770bcff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.999+0000 7ff7776ec640 1 -- 172.21.15.88:0/2719551113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559770b33a60 msgr2=0x559770bd8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:15.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.999+0000 7ff7776ec640 1 --2- 172.21.15.88:0/2719551113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559770b33a60 0x559770bd8b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.999+0000 7ff7776ec640 1 -- 172.21.15.88:0/2719551113 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559770bcfab0 con 0x559770b32fe0 2024-09-16T01:40:15.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.999+0000 7ff76ffff640 1 --2- 172.21.15.88:0/2719551113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559770b33a60 0x559770bd8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:15.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:14.999+0000 7ff7776ec640 1 --2- 172.21.15.88:0/2719551113 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559770b32fe0 0x559770b4c370 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7ff76800b240 tx=0x7ff76800b700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e97575f49778e53 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.000+0000 7ff7766ea640 1 -- 172.21.15.88:0/2719551113 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff768008070 con 0x559770b32fe0 2024-09-16T01:40:15.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.000+0000 7ff7766ea640 1 -- 172.21.15.88:0/2719551113 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff76800d500 con 0x559770b32fe0 2024-09-16T01:40:15.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.000+0000 7ff7766ea640 1 -- 172.21.15.88:0/2719551113 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76800d910 con 0x559770b32fe0 2024-09-16T01:40:15.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.000+0000 7ff779bf6f40 1 -- 172.21.15.88:0/2719551113 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559770b32fe0 msgr2=0x559770b4c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.000+0000 7ff779bf6f40 1 --2- 172.21.15.88:0/2719551113 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559770b32fe0 0x559770b4c370 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7ff76800b240 tx=0x7ff76800b700 comp rx=0 tx=0).stop 2024-09-16T01:40:15.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.000+0000 7ff779bf6f40 1 -- 172.21.15.88:0/2719551113 shutdown_connections 2024-09-16T01:40:15.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.000+0000 7ff779bf6f40 1 --2- 172.21.15.88:0/2719551113 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559770b4d250 0x559770bcff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.000+0000 7ff779bf6f40 1 --2- 172.21.15.88:0/2719551113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559770b33a60 0x559770bd8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.000+0000 7ff779bf6f40 1 --2- 172.21.15.88:0/2719551113 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559770b32fe0 0x559770b4c370 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.000+0000 7ff779bf6f40 1 -- 172.21.15.88:0/2719551113 >> 172.21.15.88:0/2719551113 conn(0x559770b4e750 msgr2=0x559770bda900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:15.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.000+0000 7ff779bf6f40 1 -- 172.21.15.88:0/2719551113 shutdown_connections 2024-09-16T01:40:15.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.000+0000 7ff779bf6f40 1 -- 172.21.15.88:0/2719551113 wait complete. 2024-09-16T01:40:15.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.001+0000 7ff779bf6f40 1 Processor -- start 2024-09-16T01:40:15.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.001+0000 7ff779bf6f40 1 -- start start 2024-09-16T01:40:15.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.001+0000 7ff779bf6f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559770b32fe0 0x559770bce460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.001+0000 7ff779bf6f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559770b33a60 0x559770bd9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.001+0000 7ff779bf6f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559770b4d250 0x559770bd9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.001+0000 7ff779bf6f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x559770bd57a0 con 0x559770b33a60 2024-09-16T01:40:15.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.001+0000 7ff779bf6f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559770bd7a50 con 0x559770b32fe0 2024-09-16T01:40:15.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.001+0000 7ff779bf6f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x559770bd7440 con 0x559770b4d250 2024-09-16T01:40:15.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.001+0000 7ff76ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559770b33a60 0x559770bd9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.002+0000 7ff76ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559770b33a60 0x559770bd9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46842/0 (socket says 172.21.15.88:46842) 2024-09-16T01:40:15.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.002+0000 7ff76ffff640 1 -- 172.21.15.88:0/1449527762 learned_addr learned my addr 172.21.15.88:0/1449527762 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:15.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.002+0000 7ff7776ec640 1 --2- 172.21.15.88:0/1449527762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559770b32fe0 0x559770bce460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.002+0000 7ff777eed640 1 --2- 172.21.15.88:0/1449527762 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559770b4d250 0x559770bd9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.002+0000 7ff76ffff640 1 -- 172.21.15.88:0/1449527762 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559770b4d250 msgr2=0x559770bd9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.002+0000 7ff76ffff640 1 --2- 172.21.15.88:0/1449527762 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559770b4d250 0x559770bd9da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.002+0000 7ff76ffff640 1 -- 172.21.15.88:0/1449527762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559770b32fe0 msgr2=0x559770bce460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.002+0000 7ff76ffff640 1 --2- 172.21.15.88:0/1449527762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559770b32fe0 0x559770bce460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.002+0000 7ff76ffff640 1 -- 172.21.15.88:0/1449527762 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559770bd4df0 con 0x559770b33a60 2024-09-16T01:40:15.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.002+0000 7ff76ffff640 1 --2- 172.21.15.88:0/1449527762 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559770b33a60 0x559770bd9860 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7ff76000dfa0 tx=0x7ff76000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.002+0000 7ff7756e8640 1 -- 172.21.15.88:0/1449527762 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7600032d0 con 0x559770b33a60 2024-09-16T01:40:15.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.002+0000 7ff7756e8640 1 -- 172.21.15.88:0/1449527762 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff760016070 con 0x559770b33a60 2024-09-16T01:40:15.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.002+0000 7ff779bf6f40 1 -- 172.21.15.88:0/1449527762 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559770bd50d0 con 0x559770b33a60 2024-09-16T01:40:15.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.002+0000 7ff7756e8640 1 -- 172.21.15.88:0/1449527762 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7600016f0 con 0x559770b33a60 2024-09-16T01:40:15.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.002+0000 7ff779bf6f40 1 -- 172.21.15.88:0/1449527762 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559770c631e0 con 0x559770b33a60 2024-09-16T01:40:15.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.003+0000 7ff7756e8640 1 -- 172.21.15.88:0/1449527762 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff760001890 con 0x559770b33a60 2024-09-16T01:40:15.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.004+0000 7ff7756e8640 1 --2- 172.21.15.88:0/1449527762 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff75806ac90 0x7ff75806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.004+0000 7ff7756e8640 1 -- 172.21.15.88:0/1449527762 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7ff760089fe0 con 0x559770b33a60 2024-09-16T01:40:15.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.004+0000 7ff7776ec640 1 --2- 172.21.15.88:0/1449527762 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff75806ac90 0x7ff75806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.004+0000 7ff779bf6f40 1 --2- 172.21.15.88:0/1449527762 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x559770c639e0 0x559770c63de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.004+0000 7ff779bf6f40 1 -- 172.21.15.88:0/1449527762 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:1 12.6 12:654134d2:::mds0_openfiles.0:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x559770c65f60 con 0x559770c639e0 2024-09-16T01:40:15.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.004+0000 7ff777eed640 1 --2- 172.21.15.88:0/1449527762 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x559770c639e0 0x559770c63de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.004+0000 7ff7776ec640 1 --2- 172.21.15.88:0/1449527762 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff75806ac90 0x7ff75806d150 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7ff76800b240 tx=0x7ff768009000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.004+0000 7ff777eed640 1 --2- 172.21.15.88:0/1449527762 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x559770c639e0 0x559770c63de0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.005+0000 7ff777eed640 1 -- 172.21.15.88:0/1449527762 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(1 mds0_openfiles.0 [stat out=16b] v0'0 uv3 ondisk = 0) ==== 160+0+16 (crc 0 0 0) 0x7ff770055970 con 0x559770c639e0 2024-09-16T01:40:15.010 INFO:teuthology.orchestra.run.smithi088.stdout:cephfs_metadata/mds0_openfiles.0 mtime 2024-09-16T01:40:11.000000+0000, size 0 2024-09-16T01:40:15.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.005+0000 7ff779bf6f40 1 -- 172.21.15.88:0/1449527762 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x559770c639e0 msgr2=0x559770c63de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.005+0000 7ff779bf6f40 1 --2- 172.21.15.88:0/1449527762 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x559770c639e0 0x559770c63de0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.005+0000 7ff779bf6f40 1 -- 172.21.15.88:0/1449527762 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff75806ac90 msgr2=0x7ff75806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.005+0000 7ff779bf6f40 1 --2- 172.21.15.88:0/1449527762 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff75806ac90 0x7ff75806d150 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7ff76800b240 tx=0x7ff768009000 comp rx=0 tx=0).stop 2024-09-16T01:40:15.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.005+0000 7ff779bf6f40 1 -- 172.21.15.88:0/1449527762 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559770b33a60 msgr2=0x559770bd9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.005+0000 7ff779bf6f40 1 --2- 172.21.15.88:0/1449527762 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559770b33a60 0x559770bd9860 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7ff76000dfa0 tx=0x7ff76000b5b0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.005+0000 7ff777eed640 1 -- 172.21.15.88:0/1449527762 reap_dead start 2024-09-16T01:40:15.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.005+0000 7ff779bf6f40 1 -- 172.21.15.88:0/1449527762 shutdown_connections 2024-09-16T01:40:15.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.005+0000 7ff779bf6f40 1 -- 172.21.15.88:0/1449527762 >> 172.21.15.88:0/1449527762 conn(0x559770b4e750 msgr2=0x559770bcf0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:15.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.005+0000 7ff779bf6f40 1 -- 172.21.15.88:0/1449527762 shutdown_connections 2024-09-16T01:40:15.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.005+0000 7ff779bf6f40 1 -- 172.21.15.88:0/1449527762 wait complete. 2024-09-16T01:40:15.013 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 500.00000000 2024-09-16T01:40:15.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.093+0000 7ff11b62ff40 1 Processor -- start 2024-09-16T01:40:15.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.093+0000 7ff11b62ff40 1 -- start start 2024-09-16T01:40:15.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.094+0000 7ff11b62ff40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560f88754ee0 0x560f88827f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.094+0000 7ff11b62ff40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f88828c90 0x560f88826ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.094+0000 7ff11b62ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f88829070 0x560f888273e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.094+0000 7ff11b62ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x560f88829450 con 0x560f88829070 2024-09-16T01:40:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.094+0000 7ff11b62ff40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560f886a6510 con 0x560f88828c90 2024-09-16T01:40:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.094+0000 7ff11b62ff40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x560f888296b0 con 0x560f88754ee0 2024-09-16T01:40:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.094+0000 7ff119926640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f88829070 0x560f888273e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.094+0000 7ff119926640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f88829070 0x560f888273e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46848/0 (socket says 172.21.15.88:46848) 2024-09-16T01:40:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.094+0000 7ff119926640 1 -- 172.21.15.88:0/876966286 learned_addr learned my addr 172.21.15.88:0/876966286 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.094+0000 7ff119125640 1 --2- 172.21.15.88:0/876966286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560f88754ee0 0x560f88827f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.094+0000 7ff119926640 1 -- 172.21.15.88:0/876966286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560f88754ee0 msgr2=0x560f88827f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.095+0000 7ff119926640 1 --2- 172.21.15.88:0/876966286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560f88754ee0 0x560f88827f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.095+0000 7ff119926640 1 -- 172.21.15.88:0/876966286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f88828c90 msgr2=0x560f88826ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.095+0000 7ff110924640 1 --2- 172.21.15.88:0/876966286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f88828c90 0x560f88826ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.095+0000 7ff119926640 1 --2- 172.21.15.88:0/876966286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f88828c90 0x560f88826ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.095+0000 7ff119926640 1 -- 172.21.15.88:0/876966286 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f88817510 con 0x560f88829070 2024-09-16T01:40:15.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.095+0000 7ff110924640 1 --2- 172.21.15.88:0/876966286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f88828c90 0x560f88826ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:15.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.095+0000 7ff119926640 1 --2- 172.21.15.88:0/876966286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f88829070 0x560f888273e0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7ff114001670 tx=0x7ff114001b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8daa1604fcad8cb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.095+0000 7ff113fff640 1 -- 172.21.15.88:0/876966286 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff114006320 con 0x560f88829070 2024-09-16T01:40:15.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.095+0000 7ff113fff640 1 -- 172.21.15.88:0/876966286 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff114013620 con 0x560f88829070 2024-09-16T01:40:15.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.095+0000 7ff113fff640 1 -- 172.21.15.88:0/876966286 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1140139d0 con 0x560f88829070 2024-09-16T01:40:15.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.095+0000 7ff11b62ff40 1 -- 172.21.15.88:0/876966286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f88829070 msgr2=0x560f888273e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.095+0000 7ff11b62ff40 1 --2- 172.21.15.88:0/876966286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f88829070 0x560f888273e0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7ff114001670 tx=0x7ff114001b30 comp rx=0 tx=0).stop 2024-09-16T01:40:15.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.095+0000 7ff11b62ff40 1 -- 172.21.15.88:0/876966286 shutdown_connections 2024-09-16T01:40:15.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.095+0000 7ff11b62ff40 1 --2- 172.21.15.88:0/876966286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f88829070 0x560f888273e0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.095+0000 7ff11b62ff40 1 --2- 172.21.15.88:0/876966286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f88828c90 0x560f88826ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.095+0000 7ff11b62ff40 1 --2- 172.21.15.88:0/876966286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560f88754ee0 0x560f88827f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.095+0000 7ff11b62ff40 1 -- 172.21.15.88:0/876966286 >> 172.21.15.88:0/876966286 conn(0x560f8877f680 msgr2=0x560f8879e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:15.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.095+0000 7ff11b62ff40 1 -- 172.21.15.88:0/876966286 shutdown_connections 2024-09-16T01:40:15.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.095+0000 7ff11b62ff40 1 -- 172.21.15.88:0/876966286 wait complete. 2024-09-16T01:40:15.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.096+0000 7ff11b62ff40 1 Processor -- start 2024-09-16T01:40:15.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.096+0000 7ff11b62ff40 1 -- start start 2024-09-16T01:40:15.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.097+0000 7ff11b62ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f88754ee0 0x560f88822dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.097+0000 7ff11b62ff40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560f88828c90 0x560f88799f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.097+0000 7ff11b62ff40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f88829070 0x560f8879ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.097+0000 7ff11b62ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x560f88817a90 con 0x560f88754ee0 2024-09-16T01:40:15.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.097+0000 7ff11b62ff40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560f888284d0 con 0x560f88829070 2024-09-16T01:40:15.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.097+0000 7ff11b62ff40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x560f888262d0 con 0x560f88828c90 2024-09-16T01:40:15.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.097+0000 7ff119125640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f88754ee0 0x560f88822dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.097+0000 7ff119125640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f88754ee0 0x560f88822dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46850/0 (socket says 172.21.15.88:46850) 2024-09-16T01:40:15.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.097+0000 7ff119125640 1 -- 172.21.15.88:0/1092504708 learned_addr learned my addr 172.21.15.88:0/1092504708 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:15.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.097+0000 7ff119926640 1 --2- 172.21.15.88:0/1092504708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f88829070 0x560f8879ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.097+0000 7ff110924640 1 --2- 172.21.15.88:0/1092504708 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560f88828c90 0x560f88799f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.097+0000 7ff119125640 1 -- 172.21.15.88:0/1092504708 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560f88828c90 msgr2=0x560f88799f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.097+0000 7ff119125640 1 --2- 172.21.15.88:0/1092504708 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560f88828c90 0x560f88799f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.097+0000 7ff119125640 1 -- 172.21.15.88:0/1092504708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f88829070 msgr2=0x560f8879ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.097+0000 7ff119125640 1 --2- 172.21.15.88:0/1092504708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f88829070 0x560f8879ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.097+0000 7ff119125640 1 -- 172.21.15.88:0/1092504708 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f88825540 con 0x560f88754ee0 2024-09-16T01:40:15.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.097+0000 7ff110924640 1 --2- 172.21.15.88:0/1092504708 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560f88828c90 0x560f88799f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:15.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.097+0000 7ff119125640 1 --2- 172.21.15.88:0/1092504708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f88754ee0 0x560f88822dc0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7ff10c045b60 tx=0x7ff10c002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.097+0000 7ff112ffd640 1 -- 172.21.15.88:0/1092504708 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10c055070 con 0x560f88754ee0 2024-09-16T01:40:15.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.097+0000 7ff112ffd640 1 -- 172.21.15.88:0/1092504708 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff10c0516b0 con 0x560f88754ee0 2024-09-16T01:40:15.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.097+0000 7ff11b62ff40 1 -- 172.21.15.88:0/1092504708 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x560f8879a600 con 0x560f88754ee0 2024-09-16T01:40:15.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.097+0000 7ff112ffd640 1 -- 172.21.15.88:0/1092504708 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10c051a60 con 0x560f88754ee0 2024-09-16T01:40:15.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.097+0000 7ff11b62ff40 1 -- 172.21.15.88:0/1092504708 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f88824d40 con 0x560f88754ee0 2024-09-16T01:40:15.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.099+0000 7ff112ffd640 1 -- 172.21.15.88:0/1092504708 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff10c058070 con 0x560f88754ee0 2024-09-16T01:40:15.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.099+0000 7ff112ffd640 1 --2- 172.21.15.88:0/1092504708 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff0fc06ac90 0x7ff0fc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.099+0000 7ff112ffd640 1 -- 172.21.15.88:0/1092504708 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7ff10c0cdc00 con 0x560f88754ee0 2024-09-16T01:40:15.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.099+0000 7ff110924640 1 --2- 172.21.15.88:0/1092504708 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff0fc06ac90 0x7ff0fc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.099+0000 7ff11b62ff40 1 --2- 172.21.15.88:0/1092504708 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x560f8879ef80 0x560f88813410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.099+0000 7ff11b62ff40 1 -- 172.21.15.88:0/1092504708 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:1 12.15 12:aa448500:::500.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x560f888c3eb0 con 0x560f8879ef80 2024-09-16T01:40:15.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.100+0000 7ff110924640 1 --2- 172.21.15.88:0/1092504708 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff0fc06ac90 0x7ff0fc06d150 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff104005770 tx=0x7ff104005a00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.100+0000 7ff119926640 1 --2- 172.21.15.88:0/1092504708 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x560f8879ef80 0x560f88813410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.100+0000 7ff119926640 1 --2- 172.21.15.88:0/1092504708 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x560f8879ef80 0x560f88813410 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.100+0000 7ff119926640 1 -- 172.21.15.88:0/1092504708 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(1 500.00000000 [stat out=16b] v0'0 uv3 ondisk = 0) ==== 156+0+16 (crc 0 0 0) 0x7ff11400f030 con 0x560f8879ef80 2024-09-16T01:40:15.106 INFO:teuthology.orchestra.run.smithi088.stdout:cephfs_metadata/500.00000000 mtime 2024-09-16T01:40:04.000000+0000, size 90 2024-09-16T01:40:15.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.100+0000 7ff11b62ff40 1 -- 172.21.15.88:0/1092504708 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x560f8879ef80 msgr2=0x560f88813410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.100+0000 7ff11b62ff40 1 --2- 172.21.15.88:0/1092504708 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x560f8879ef80 0x560f88813410 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.100+0000 7ff11b62ff40 1 -- 172.21.15.88:0/1092504708 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff0fc06ac90 msgr2=0x7ff0fc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.100+0000 7ff11b62ff40 1 --2- 172.21.15.88:0/1092504708 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff0fc06ac90 0x7ff0fc06d150 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff104005770 tx=0x7ff104005a00 comp rx=0 tx=0).stop 2024-09-16T01:40:15.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.100+0000 7ff11b62ff40 1 -- 172.21.15.88:0/1092504708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f88754ee0 msgr2=0x560f88822dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.100+0000 7ff11b62ff40 1 --2- 172.21.15.88:0/1092504708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f88754ee0 0x560f88822dc0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7ff10c045b60 tx=0x7ff10c002fb0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.100+0000 7ff119926640 1 -- 172.21.15.88:0/1092504708 reap_dead start 2024-09-16T01:40:15.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.101+0000 7ff11b62ff40 1 -- 172.21.15.88:0/1092504708 shutdown_connections 2024-09-16T01:40:15.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.101+0000 7ff11b62ff40 1 -- 172.21.15.88:0/1092504708 >> 172.21.15.88:0/1092504708 conn(0x560f8877f680 msgr2=0x560f888259b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:15.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.101+0000 7ff11b62ff40 1 -- 172.21.15.88:0/1092504708 shutdown_connections 2024-09-16T01:40:15.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.101+0000 7ff11b62ff40 1 -- 172.21.15.88:0/1092504708 wait complete. 2024-09-16T01:40:15.108 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat mds_snaptable 2024-09-16T01:40:15.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.188+0000 7f51cccd2f40 1 Processor -- start 2024-09-16T01:40:15.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.188+0000 7f51cccd2f40 1 -- start start 2024-09-16T01:40:15.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.188+0000 7f51cccd2f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56332e3c2fe0 0x56332e46b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.188+0000 7f51cccd2f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56332e467d20 0x56332e3c20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.188+0000 7f51cccd2f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56332e468100 0x56332e3c2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.189+0000 7f51cccd2f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56332e2e5510 con 0x56332e468100 2024-09-16T01:40:15.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.189+0000 7f51cccd2f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56332e33ec50 con 0x56332e467d20 2024-09-16T01:40:15.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.189+0000 7f51cccd2f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56332e3deb30 con 0x56332e3c2fe0 2024-09-16T01:40:15.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.189+0000 7f51cafc9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56332e468100 0x56332e3c2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.189+0000 7f51cafc9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56332e468100 0x56332e3c2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46870/0 (socket says 172.21.15.88:46870) 2024-09-16T01:40:15.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.189+0000 7f51cafc9640 1 -- 172.21.15.88:0/2791459175 learned_addr learned my addr 172.21.15.88:0/2791459175 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:15.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.189+0000 7f51cafc9640 1 -- 172.21.15.88:0/2791459175 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56332e3c2fe0 msgr2=0x56332e46b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:15.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.189+0000 7f51cafc9640 1 --2- 172.21.15.88:0/2791459175 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56332e3c2fe0 0x56332e46b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.189+0000 7f51c3fff640 1 --2- 172.21.15.88:0/2791459175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56332e467d20 0x56332e3c20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.189+0000 7f51cafc9640 1 -- 172.21.15.88:0/2791459175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56332e467d20 msgr2=0x56332e3c20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:15.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.189+0000 7f51cafc9640 1 --2- 172.21.15.88:0/2791459175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56332e467d20 0x56332e3c20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.189+0000 7f51cafc9640 1 -- 172.21.15.88:0/2791459175 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56332e46b670 con 0x56332e468100 2024-09-16T01:40:15.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.189+0000 7f51c3fff640 1 --2- 172.21.15.88:0/2791459175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56332e467d20 0x56332e3c20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:15.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.189+0000 7f51cafc9640 1 --2- 172.21.15.88:0/2791459175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56332e468100 0x56332e3c2600 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f51c4001660 tx=0x7f51c4001b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3d6327bb7c32391 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.189+0000 7f51c97c6640 1 -- 172.21.15.88:0/2791459175 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c4004520 con 0x56332e468100 2024-09-16T01:40:15.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.189+0000 7f51c97c6640 1 -- 172.21.15.88:0/2791459175 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51c40046c0 con 0x56332e468100 2024-09-16T01:40:15.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.189+0000 7f51c97c6640 1 -- 172.21.15.88:0/2791459175 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c4004a90 con 0x56332e468100 2024-09-16T01:40:15.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.190+0000 7f51cccd2f40 1 -- 172.21.15.88:0/2791459175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56332e468100 msgr2=0x56332e3c2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.190+0000 7f51cccd2f40 1 --2- 172.21.15.88:0/2791459175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56332e468100 0x56332e3c2600 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f51c4001660 tx=0x7f51c4001b20 comp rx=0 tx=0).stop 2024-09-16T01:40:15.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.190+0000 7f51cccd2f40 1 -- 172.21.15.88:0/2791459175 shutdown_connections 2024-09-16T01:40:15.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.190+0000 7f51cccd2f40 1 --2- 172.21.15.88:0/2791459175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56332e468100 0x56332e3c2600 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.190+0000 7f51cccd2f40 1 --2- 172.21.15.88:0/2791459175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56332e467d20 0x56332e3c20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.190+0000 7f51cccd2f40 1 --2- 172.21.15.88:0/2791459175 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56332e3c2fe0 0x56332e46b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.190+0000 7f51cccd2f40 1 -- 172.21.15.88:0/2791459175 >> 172.21.15.88:0/2791459175 conn(0x56332e3de750 msgr2=0x56332e396da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:15.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.190+0000 7f51cccd2f40 1 -- 172.21.15.88:0/2791459175 shutdown_connections 2024-09-16T01:40:15.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.190+0000 7f51cccd2f40 1 -- 172.21.15.88:0/2791459175 wait complete. 2024-09-16T01:40:15.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.190+0000 7f51cccd2f40 1 Processor -- start 2024-09-16T01:40:15.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.190+0000 7f51cccd2f40 1 -- start start 2024-09-16T01:40:15.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.191+0000 7f51cccd2f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56332e3c2fe0 0x56332e45afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.191+0000 7f51cccd2f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56332e467d20 0x56332e5028f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.191+0000 7f51cccd2f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56332e468100 0x56332e457d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.191+0000 7f51cccd2f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56332e3ded80 con 0x56332e467d20 2024-09-16T01:40:15.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.191+0000 7f51cccd2f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56332e4687d0 con 0x56332e3c2fe0 2024-09-16T01:40:15.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.191+0000 7f51cccd2f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56332e4671a0 con 0x56332e468100 2024-09-16T01:40:15.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.191+0000 7f51c3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56332e467d20 0x56332e5028f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.191+0000 7f51c3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56332e467d20 0x56332e5028f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46872/0 (socket says 172.21.15.88:46872) 2024-09-16T01:40:15.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.191+0000 7f51c3fff640 1 -- 172.21.15.88:0/1737198305 learned_addr learned my addr 172.21.15.88:0/1737198305 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:15.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.191+0000 7f51ca7c8640 1 --2- 172.21.15.88:0/1737198305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56332e3c2fe0 0x56332e45afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.191+0000 7f51cafc9640 1 --2- 172.21.15.88:0/1737198305 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56332e468100 0x56332e457d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.191+0000 7f51c3fff640 1 -- 172.21.15.88:0/1737198305 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56332e468100 msgr2=0x56332e457d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.191+0000 7f51c3fff640 1 --2- 172.21.15.88:0/1737198305 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56332e468100 0x56332e457d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.191+0000 7f51c3fff640 1 -- 172.21.15.88:0/1737198305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56332e3c2fe0 msgr2=0x56332e45afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.191+0000 7f51c3fff640 1 --2- 172.21.15.88:0/1737198305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56332e3c2fe0 0x56332e45afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.191+0000 7f51c3fff640 1 -- 172.21.15.88:0/1737198305 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56332e502e30 con 0x56332e467d20 2024-09-16T01:40:15.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.191+0000 7f51ca7c8640 1 --2- 172.21.15.88:0/1737198305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56332e3c2fe0 0x56332e45afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:15.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.192+0000 7f51c3fff640 1 --2- 172.21.15.88:0/1737198305 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56332e467d20 0x56332e5028f0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f51b4009360 tx=0x7f51b4009820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.192+0000 7f51c37fe640 1 -- 172.21.15.88:0/1737198305 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b400c8f0 con 0x56332e467d20 2024-09-16T01:40:15.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.192+0000 7f51c37fe640 1 -- 172.21.15.88:0/1737198305 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51b4010070 con 0x56332e467d20 2024-09-16T01:40:15.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.192+0000 7f51cccd2f40 1 -- 172.21.15.88:0/1737198305 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56332e459f10 con 0x56332e467d20 2024-09-16T01:40:15.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.192+0000 7f51c37fe640 1 -- 172.21.15.88:0/1737198305 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b400ccb0 con 0x56332e467d20 2024-09-16T01:40:15.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.192+0000 7f51cccd2f40 1 -- 172.21.15.88:0/1737198305 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56332e45a4d0 con 0x56332e467d20 2024-09-16T01:40:15.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.193+0000 7f51c37fe640 1 -- 172.21.15.88:0/1737198305 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f51b401e3f0 con 0x56332e467d20 2024-09-16T01:40:15.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.194+0000 7f51c37fe640 1 --2- 172.21.15.88:0/1737198305 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f51ac06ac90 0x7f51ac06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.194+0000 7f51c37fe640 1 -- 172.21.15.88:0/1737198305 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f51b408a410 con 0x56332e467d20 2024-09-16T01:40:15.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.194+0000 7f51ca7c8640 1 --2- 172.21.15.88:0/1737198305 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f51ac06ac90 0x7f51ac06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.194+0000 7f51cccd2f40 1 --2- 172.21.15.88:0/1737198305 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x56332e3961f0 0x56332e464730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.194+0000 7f51cccd2f40 1 -- 172.21.15.88:0/1737198305 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:1 12.d 12:b50e409b:::mds_snaptable:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x56332e464c70 con 0x56332e3961f0 2024-09-16T01:40:15.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.194+0000 7f51ca7c8640 1 --2- 172.21.15.88:0/1737198305 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f51ac06ac90 0x7f51ac06d150 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f51bc057000 tx=0x7f51bc051000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.194+0000 7f51cafc9640 1 --2- 172.21.15.88:0/1737198305 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x56332e3961f0 0x56332e464730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.195+0000 7f51cafc9640 1 --2- 172.21.15.88:0/1737198305 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x56332e3961f0 0x56332e464730 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.195+0000 7f51cafc9640 1 -- 172.21.15.88:0/1737198305 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(1 mds_snaptable [stat out=16b] v0'0 uv1 ondisk = 0) ==== 157+0+16 (crc 0 0 0) 0x7f51c400f030 con 0x56332e3961f0 2024-09-16T01:40:15.199 INFO:teuthology.orchestra.run.smithi088.stdout:cephfs_metadata/mds_snaptable mtime 2024-09-16T01:40:04.000000+0000, size 70 2024-09-16T01:40:15.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.195+0000 7f51cccd2f40 1 -- 172.21.15.88:0/1737198305 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x56332e3961f0 msgr2=0x56332e464730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.195+0000 7f51cccd2f40 1 --2- 172.21.15.88:0/1737198305 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x56332e3961f0 0x56332e464730 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.195+0000 7f51cccd2f40 1 -- 172.21.15.88:0/1737198305 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f51ac06ac90 msgr2=0x7f51ac06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.195+0000 7f51cccd2f40 1 --2- 172.21.15.88:0/1737198305 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f51ac06ac90 0x7f51ac06d150 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f51bc057000 tx=0x7f51bc051000 comp rx=0 tx=0).stop 2024-09-16T01:40:15.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.195+0000 7f51cccd2f40 1 -- 172.21.15.88:0/1737198305 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56332e467d20 msgr2=0x56332e5028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.195+0000 7f51cccd2f40 1 --2- 172.21.15.88:0/1737198305 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56332e467d20 0x56332e5028f0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f51b4009360 tx=0x7f51b4009820 comp rx=0 tx=0).stop 2024-09-16T01:40:15.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.195+0000 7f51cafc9640 1 -- 172.21.15.88:0/1737198305 reap_dead start 2024-09-16T01:40:15.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.195+0000 7f51cccd2f40 1 -- 172.21.15.88:0/1737198305 shutdown_connections 2024-09-16T01:40:15.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.195+0000 7f51cccd2f40 1 -- 172.21.15.88:0/1737198305 >> 172.21.15.88:0/1737198305 conn(0x56332e3de750 msgr2=0x56332e3be400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:15.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.195+0000 7f51cccd2f40 1 -- 172.21.15.88:0/1737198305 shutdown_connections 2024-09-16T01:40:15.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.195+0000 7f51cccd2f40 1 -- 172.21.15.88:0/1737198305 wait complete. 2024-09-16T01:40:15.202 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat mds0_inotable 2024-09-16T01:40:15.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.282+0000 7faac84d2f40 1 Processor -- start 2024-09-16T01:40:15.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.282+0000 7faac84d2f40 1 -- start start 2024-09-16T01:40:15.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.283+0000 7faac84d2f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcca8a08d0 0x55fcca93e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.283+0000 7faac84d2f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcca945d20 0x55fcca8b8c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.283+0000 7faac84d2f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fcca946100 0x55fcca8b9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.283+0000 7faac84d2f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55fcca7c3510 con 0x55fcca945d20 2024-09-16T01:40:15.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.283+0000 7faac84d2f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fcca8b9bd0 con 0x55fcca8a08d0 2024-09-16T01:40:15.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.283+0000 7faac84d2f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55fcca8bd040 con 0x55fcca946100 2024-09-16T01:40:15.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.283+0000 7faac57c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcca945d20 0x55fcca8b8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.283+0000 7faac57c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcca945d20 0x55fcca8b8c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46888/0 (socket says 172.21.15.88:46888) 2024-09-16T01:40:15.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.283+0000 7faac57c9640 1 -- 172.21.15.88:0/821271231 learned_addr learned my addr 172.21.15.88:0/821271231 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.283+0000 7faac57c9640 1 -- 172.21.15.88:0/821271231 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fcca946100 msgr2=0x55fcca8b9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.283+0000 7faac57c9640 1 --2- 172.21.15.88:0/821271231 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fcca946100 0x55fcca8b9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.283+0000 7faac57c9640 1 -- 172.21.15.88:0/821271231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcca8a08d0 msgr2=0x55fcca93e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.283+0000 7faac57c9640 1 --2- 172.21.15.88:0/821271231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcca8a08d0 0x55fcca93e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.283+0000 7faac57c9640 1 -- 172.21.15.88:0/821271231 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcca874c30 con 0x55fcca945d20 2024-09-16T01:40:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.283+0000 7faac57c9640 1 --2- 172.21.15.88:0/821271231 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcca945d20 0x55fcca8b8c20 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7faabc045b00 tx=0x7faabc051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf70916270f23f88 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.283+0000 7faab7fff640 1 -- 172.21.15.88:0/821271231 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faabc056030 con 0x55fcca945d20 2024-09-16T01:40:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.283+0000 7faab7fff640 1 -- 172.21.15.88:0/821271231 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faabc060070 con 0x55fcca945d20 2024-09-16T01:40:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.283+0000 7faab7fff640 1 -- 172.21.15.88:0/821271231 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faabc052d10 con 0x55fcca945d20 2024-09-16T01:40:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.284+0000 7faac84d2f40 1 -- 172.21.15.88:0/821271231 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcca945d20 msgr2=0x55fcca8b8c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.284+0000 7faac84d2f40 1 --2- 172.21.15.88:0/821271231 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcca945d20 0x55fcca8b8c20 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7faabc045b00 tx=0x7faabc051000 comp rx=0 tx=0).stop 2024-09-16T01:40:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.284+0000 7faac84d2f40 1 -- 172.21.15.88:0/821271231 shutdown_connections 2024-09-16T01:40:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.284+0000 7faac84d2f40 1 --2- 172.21.15.88:0/821271231 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fcca946100 0x55fcca8b9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.284+0000 7faac84d2f40 1 --2- 172.21.15.88:0/821271231 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcca945d20 0x55fcca8b8c20 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.284+0000 7faac84d2f40 1 --2- 172.21.15.88:0/821271231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcca8a08d0 0x55fcca93e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.284+0000 7faac84d2f40 1 -- 172.21.15.88:0/821271231 >> 172.21.15.88:0/821271231 conn(0x55fcca8bc750 msgr2=0x55fcca89dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:15.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.284+0000 7faac84d2f40 1 -- 172.21.15.88:0/821271231 shutdown_connections 2024-09-16T01:40:15.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.284+0000 7faac84d2f40 1 -- 172.21.15.88:0/821271231 wait complete. 2024-09-16T01:40:15.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.284+0000 7faac84d2f40 1 Processor -- start 2024-09-16T01:40:15.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.285+0000 7faac84d2f40 1 -- start start 2024-09-16T01:40:15.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.285+0000 7faac84d2f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fcca8a08d0 0x55fcca932ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.285+0000 7faac84d2f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcca945d20 0x55fcca933da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.285+0000 7faac84d2f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcca946100 0x55fcca948b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.285+0000 7faac84d2f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55fcca8bccd0 con 0x55fcca945d20 2024-09-16T01:40:15.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.285+0000 7faac84d2f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fcca936d90 con 0x55fcca946100 2024-09-16T01:40:15.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.285+0000 7faac84d2f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55fcca937240 con 0x55fcca8a08d0 2024-09-16T01:40:15.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.285+0000 7faac57c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcca945d20 0x55fcca933da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.285+0000 7faac57c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcca945d20 0x55fcca933da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46904/0 (socket says 172.21.15.88:46904) 2024-09-16T01:40:15.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.285+0000 7faac57c9640 1 -- 172.21.15.88:0/2906965383 learned_addr learned my addr 172.21.15.88:0/2906965383 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:15.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.285+0000 7faac67cb640 1 --2- 172.21.15.88:0/2906965383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcca946100 0x55fcca948b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.285+0000 7faac5fca640 1 --2- 172.21.15.88:0/2906965383 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fcca8a08d0 0x55fcca932ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.286+0000 7faac57c9640 1 -- 172.21.15.88:0/2906965383 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fcca8a08d0 msgr2=0x55fcca932ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.286+0000 7faac57c9640 1 --2- 172.21.15.88:0/2906965383 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fcca8a08d0 0x55fcca932ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.286+0000 7faac57c9640 1 -- 172.21.15.88:0/2906965383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcca946100 msgr2=0x55fcca948b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.286+0000 7faac57c9640 1 --2- 172.21.15.88:0/2906965383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcca946100 0x55fcca948b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.286+0000 7faac57c9640 1 -- 172.21.15.88:0/2906965383 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcca946af0 con 0x55fcca945d20 2024-09-16T01:40:15.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.286+0000 7faac5fca640 1 --2- 172.21.15.88:0/2906965383 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fcca8a08d0 0x55fcca932ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:15.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.286+0000 7faac57c9640 1 --2- 172.21.15.88:0/2906965383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcca945d20 0x55fcca933da0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7faabc025cc0 tx=0x7faabc002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.286+0000 7faab6ffd640 1 -- 172.21.15.88:0/2906965383 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faabc075070 con 0x55fcca945d20 2024-09-16T01:40:15.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.286+0000 7faab6ffd640 1 -- 172.21.15.88:0/2906965383 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faabc070470 con 0x55fcca945d20 2024-09-16T01:40:15.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.286+0000 7faac84d2f40 1 -- 172.21.15.88:0/2906965383 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55fcca935f40 con 0x55fcca945d20 2024-09-16T01:40:15.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.286+0000 7faab6ffd640 1 -- 172.21.15.88:0/2906965383 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faabc070840 con 0x55fcca945d20 2024-09-16T01:40:15.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.286+0000 7faac84d2f40 1 -- 172.21.15.88:0/2906965383 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcca93ed40 con 0x55fcca945d20 2024-09-16T01:40:15.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.287+0000 7faab6ffd640 1 -- 172.21.15.88:0/2906965383 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7faabc059020 con 0x55fcca945d20 2024-09-16T01:40:15.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.288+0000 7faab6ffd640 1 --2- 172.21.15.88:0/2906965383 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faaa806ac90 0x7faaa806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.288+0000 7faac5fca640 1 --2- 172.21.15.88:0/2906965383 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faaa806ac90 0x7faaa806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.288+0000 7faab6ffd640 1 -- 172.21.15.88:0/2906965383 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7faabc0cf3c0 con 0x55fcca945d20 2024-09-16T01:40:15.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.288+0000 7faac84d2f40 1 --2- 172.21.15.88:0/2906965383 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55fcca933570 0x55fcca8bb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.288+0000 7faac84d2f40 1 -- 172.21.15.88:0/2906965383 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:1 12.13 12:c91d4a1d:::mds0_inotable:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fcca860820 con 0x55fcca933570 2024-09-16T01:40:15.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.288+0000 7faac67cb640 1 --2- 172.21.15.88:0/2906965383 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55fcca933570 0x55fcca8bb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.288+0000 7faac5fca640 1 --2- 172.21.15.88:0/2906965383 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faaa806ac90 0x7faaa806d150 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7faab80030a0 tx=0x7faab80058c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.288+0000 7faac67cb640 1 --2- 172.21.15.88:0/2906965383 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55fcca933570 0x55fcca8bb290 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.289+0000 7faac67cb640 1 -- 172.21.15.88:0/2906965383 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(1 mds0_inotable [stat out=16b] v0'0 uv3 ondisk = 0) ==== 157+0+16 (crc 0 0 0) 0x7faac0003450 con 0x55fcca933570 2024-09-16T01:40:15.293 INFO:teuthology.orchestra.run.smithi088.stdout:cephfs_metadata/mds0_inotable mtime 2024-09-16T01:40:11.000000+0000, size 34 2024-09-16T01:40:15.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.289+0000 7faac84d2f40 1 -- 172.21.15.88:0/2906965383 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55fcca933570 msgr2=0x55fcca8bb290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.289+0000 7faac84d2f40 1 --2- 172.21.15.88:0/2906965383 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55fcca933570 0x55fcca8bb290 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.289+0000 7faac84d2f40 1 -- 172.21.15.88:0/2906965383 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faaa806ac90 msgr2=0x7faaa806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.289+0000 7faac84d2f40 1 --2- 172.21.15.88:0/2906965383 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faaa806ac90 0x7faaa806d150 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7faab80030a0 tx=0x7faab80058c0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.289+0000 7faac84d2f40 1 -- 172.21.15.88:0/2906965383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcca945d20 msgr2=0x55fcca933da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.289+0000 7faac84d2f40 1 --2- 172.21.15.88:0/2906965383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcca945d20 0x55fcca933da0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7faabc025cc0 tx=0x7faabc002fb0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.289+0000 7faac67cb640 1 -- 172.21.15.88:0/2906965383 reap_dead start 2024-09-16T01:40:15.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.289+0000 7faac84d2f40 1 -- 172.21.15.88:0/2906965383 shutdown_connections 2024-09-16T01:40:15.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.289+0000 7faac84d2f40 1 -- 172.21.15.88:0/2906965383 >> 172.21.15.88:0/2906965383 conn(0x55fcca8bc750 msgr2=0x55fcca89ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:15.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.289+0000 7faac84d2f40 1 -- 172.21.15.88:0/2906965383 shutdown_connections 2024-09-16T01:40:15.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.289+0000 7faac84d2f40 1 -- 172.21.15.88:0/2906965383 wait complete. 2024-09-16T01:40:15.296 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 100.00000000 2024-09-16T01:40:15.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.377+0000 7fd4b1f1ef40 1 Processor -- start 2024-09-16T01:40:15.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.377+0000 7fd4b1f1ef40 1 -- start start 2024-09-16T01:40:15.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.377+0000 7fd4b1f1ef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592b2ba7940 0x5592b2bef6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.377+0000 7fd4b1f1ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5592b2bf0b30 0x5592b2c73db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.377+0000 7fd4b1f1ef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5592b2c7bf30 0x5592b2c7a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.377+0000 7fd4b1f1ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5592b2ba82c0 con 0x5592b2bf0b30 2024-09-16T01:40:15.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.377+0000 7fd4b1f1ef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5592b2af8510 con 0x5592b2ba7940 2024-09-16T01:40:15.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.377+0000 7fd4b1f1ef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5592b2ba5c80 con 0x5592b2c7bf30 2024-09-16T01:40:15.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.377+0000 7fd4ae7ae640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5592b2bf0b30 0x5592b2c73db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.377+0000 7fd4ae7ae640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5592b2bf0b30 0x5592b2c73db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46908/0 (socket says 172.21.15.88:46908) 2024-09-16T01:40:15.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.377+0000 7fd4ae7ae640 1 -- 172.21.15.88:0/3972963685 learned_addr learned my addr 172.21.15.88:0/3972963685 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:15.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.378+0000 7fd4aefaf640 1 --2- 172.21.15.88:0/3972963685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592b2ba7940 0x5592b2bef6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.378+0000 7fd4af7b0640 1 --2- 172.21.15.88:0/3972963685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5592b2c7bf30 0x5592b2c7a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.378+0000 7fd4ae7ae640 1 -- 172.21.15.88:0/3972963685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5592b2c7bf30 msgr2=0x5592b2c7a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:15.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.378+0000 7fd4ae7ae640 1 --2- 172.21.15.88:0/3972963685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5592b2c7bf30 0x5592b2c7a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.378+0000 7fd4ae7ae640 1 -- 172.21.15.88:0/3972963685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592b2ba7940 msgr2=0x5592b2bef6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:15.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.378+0000 7fd4ae7ae640 1 --2- 172.21.15.88:0/3972963685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592b2ba7940 0x5592b2bef6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.378+0000 7fd4ae7ae640 1 -- 172.21.15.88:0/3972963685 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592b2c78400 con 0x5592b2bf0b30 2024-09-16T01:40:15.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.378+0000 7fd4aefaf640 1 --2- 172.21.15.88:0/3972963685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592b2ba7940 0x5592b2bef6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:15.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.378+0000 7fd4af7b0640 1 --2- 172.21.15.88:0/3972963685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5592b2c7bf30 0x5592b2c7a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:15.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.378+0000 7fd4ae7ae640 1 --2- 172.21.15.88:0/3972963685 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5592b2bf0b30 0x5592b2c73db0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fd49800b420 tx=0x7fd49800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2034dfa85dbc42e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.378+0000 7fd4ad7ac640 1 -- 172.21.15.88:0/3972963685 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd498012020 con 0x5592b2bf0b30 2024-09-16T01:40:15.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.378+0000 7fd4ad7ac640 1 -- 172.21.15.88:0/3972963685 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd49800d6b0 con 0x5592b2bf0b30 2024-09-16T01:40:15.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.378+0000 7fd4ad7ac640 1 -- 172.21.15.88:0/3972963685 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49800dae0 con 0x5592b2bf0b30 2024-09-16T01:40:15.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.378+0000 7fd4b1f1ef40 1 -- 172.21.15.88:0/3972963685 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5592b2bf0b30 msgr2=0x5592b2c73db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.378+0000 7fd4b1f1ef40 1 --2- 172.21.15.88:0/3972963685 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5592b2bf0b30 0x5592b2c73db0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fd49800b420 tx=0x7fd49800b8e0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.379+0000 7fd4b1f1ef40 1 -- 172.21.15.88:0/3972963685 shutdown_connections 2024-09-16T01:40:15.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.379+0000 7fd4b1f1ef40 1 --2- 172.21.15.88:0/3972963685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5592b2c7bf30 0x5592b2c7a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.379+0000 7fd4b1f1ef40 1 --2- 172.21.15.88:0/3972963685 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5592b2bf0b30 0x5592b2c73db0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.379+0000 7fd4b1f1ef40 1 --2- 172.21.15.88:0/3972963685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592b2ba7940 0x5592b2bef6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.379+0000 7fd4b1f1ef40 1 -- 172.21.15.88:0/3972963685 >> 172.21.15.88:0/3972963685 conn(0x5592b2bf1750 msgr2=0x5592b2bedb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:15.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.379+0000 7fd4b1f1ef40 1 -- 172.21.15.88:0/3972963685 shutdown_connections 2024-09-16T01:40:15.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.379+0000 7fd4b1f1ef40 1 -- 172.21.15.88:0/3972963685 wait complete. 2024-09-16T01:40:15.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.379+0000 7fd4b1f1ef40 1 Processor -- start 2024-09-16T01:40:15.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.379+0000 7fd4b1f1ef40 1 -- start start 2024-09-16T01:40:15.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.380+0000 7fd4b1f1ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5592b2ba7940 0x5592b2c7a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.380+0000 7fd4b1f1ef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5592b2bf0b30 0x5592b2c75580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.380+0000 7fd4b1f1ef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592b2c7bf30 0x5592b2c75ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.380+0000 7fd4b1f1ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5592b2b95570 con 0x5592b2ba7940 2024-09-16T01:40:15.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.380+0000 7fd4b1f1ef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5592b2c781b0 con 0x5592b2c7bf30 2024-09-16T01:40:15.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.380+0000 7fd4b1f1ef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5592b2b95850 con 0x5592b2bf0b30 2024-09-16T01:40:15.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.380+0000 7fd4aefaf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5592b2ba7940 0x5592b2c7a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.380+0000 7fd4aefaf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5592b2ba7940 0x5592b2c7a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46922/0 (socket says 172.21.15.88:46922) 2024-09-16T01:40:15.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.380+0000 7fd4aefaf640 1 -- 172.21.15.88:0/2017585945 learned_addr learned my addr 172.21.15.88:0/2017585945 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:15.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.380+0000 7fd4af7b0640 1 --2- 172.21.15.88:0/2017585945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592b2c7bf30 0x5592b2c75ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.380+0000 7fd4aefaf640 1 -- 172.21.15.88:0/2017585945 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5592b2bf0b30 msgr2=0x5592b2c75580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.380+0000 7fd4aefaf640 1 --2- 172.21.15.88:0/2017585945 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5592b2bf0b30 0x5592b2c75580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.380+0000 7fd4aefaf640 1 -- 172.21.15.88:0/2017585945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592b2c7bf30 msgr2=0x5592b2c75ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.380+0000 7fd4aefaf640 1 --2- 172.21.15.88:0/2017585945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5592b2c7bf30 0x5592b2c75ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.380+0000 7fd4aefaf640 1 -- 172.21.15.88:0/2017585945 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592b2c73f50 con 0x5592b2ba7940 2024-09-16T01:40:15.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.380+0000 7fd4aefaf640 1 --2- 172.21.15.88:0/2017585945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5592b2ba7940 0x5592b2c7a410 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8009f30 tx=0x7fd4a80084c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.380+0000 7fd4a7fff640 1 -- 172.21.15.88:0/2017585945 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a80143f0 con 0x5592b2ba7940 2024-09-16T01:40:15.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.381+0000 7fd4a7fff640 1 -- 172.21.15.88:0/2017585945 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4a8014590 con 0x5592b2ba7940 2024-09-16T01:40:15.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.381+0000 7fd4a7fff640 1 -- 172.21.15.88:0/2017585945 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a8014940 con 0x5592b2ba7940 2024-09-16T01:40:15.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.381+0000 7fd4b1f1ef40 1 -- 172.21.15.88:0/2017585945 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5592b2c74230 con 0x5592b2ba7940 2024-09-16T01:40:15.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.381+0000 7fd4b1f1ef40 1 -- 172.21.15.88:0/2017585945 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592b2c70610 con 0x5592b2ba7940 2024-09-16T01:40:15.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.382+0000 7fd4a7fff640 1 -- 172.21.15.88:0/2017585945 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd4a801b020 con 0x5592b2ba7940 2024-09-16T01:40:15.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.382+0000 7fd4a7fff640 1 --2- 172.21.15.88:0/2017585945 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd49006ac90 0x7fd49006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.382+0000 7fd4a7fff640 1 -- 172.21.15.88:0/2017585945 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7fd4a8089b60 con 0x5592b2ba7940 2024-09-16T01:40:15.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.382+0000 7fd4ae7ae640 1 --2- 172.21.15.88:0/2017585945 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd49006ac90 0x7fd49006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.383+0000 7fd4b1f1ef40 1 --2- 172.21.15.88:0/2017585945 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5592b2c77dd0 0x5592b2d189b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.383+0000 7fd4b1f1ef40 1 -- 172.21.15.88:0/2017585945 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:1 12.13 12:cd5a64a3:::100.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5592b2c73630 con 0x5592b2c77dd0 2024-09-16T01:40:15.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.383+0000 7fd4af7b0640 1 --2- 172.21.15.88:0/2017585945 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5592b2c77dd0 0x5592b2d189b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.383+0000 7fd4ae7ae640 1 --2- 172.21.15.88:0/2017585945 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd49006ac90 0x7fd49006d150 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x5592b2bf2570 tx=0x7fd498006000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.383+0000 7fd4af7b0640 1 --2- 172.21.15.88:0/2017585945 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5592b2c77dd0 0x5592b2d189b0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.383+0000 7fd4af7b0640 1 -- 172.21.15.88:0/2017585945 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(1 100.00000000 [stat out=16b] v0'0 uv1 ondisk = 0) ==== 156+0+16 (crc 0 0 0) 0x7fd4a0052690 con 0x5592b2c77dd0 2024-09-16T01:40:15.389 INFO:teuthology.orchestra.run.smithi088.stdout:cephfs_metadata/100.00000000 mtime 2024-09-16T01:40:04.000000+0000, size 0 2024-09-16T01:40:15.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.383+0000 7fd4b1f1ef40 1 -- 172.21.15.88:0/2017585945 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5592b2c77dd0 msgr2=0x5592b2d189b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.383+0000 7fd4b1f1ef40 1 --2- 172.21.15.88:0/2017585945 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5592b2c77dd0 0x5592b2d189b0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.383+0000 7fd4b1f1ef40 1 -- 172.21.15.88:0/2017585945 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd49006ac90 msgr2=0x7fd49006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.383+0000 7fd4b1f1ef40 1 --2- 172.21.15.88:0/2017585945 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd49006ac90 0x7fd49006d150 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x5592b2bf2570 tx=0x7fd498006000 comp rx=0 tx=0).stop 2024-09-16T01:40:15.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.383+0000 7fd4b1f1ef40 1 -- 172.21.15.88:0/2017585945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5592b2ba7940 msgr2=0x5592b2c7a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.383+0000 7fd4b1f1ef40 1 --2- 172.21.15.88:0/2017585945 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5592b2ba7940 0x5592b2c7a410 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8009f30 tx=0x7fd4a80084c0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.383+0000 7fd4af7b0640 1 -- 172.21.15.88:0/2017585945 reap_dead start 2024-09-16T01:40:15.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.383+0000 7fd4b1f1ef40 1 -- 172.21.15.88:0/2017585945 shutdown_connections 2024-09-16T01:40:15.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.383+0000 7fd4b1f1ef40 1 -- 172.21.15.88:0/2017585945 >> 172.21.15.88:0/2017585945 conn(0x5592b2bf1750 msgr2=0x5592b2bef620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:15.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.383+0000 7fd4b1f1ef40 1 -- 172.21.15.88:0/2017585945 shutdown_connections 2024-09-16T01:40:15.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.383+0000 7fd4b1f1ef40 1 -- 172.21.15.88:0/2017585945 wait complete. 2024-09-16T01:40:15.391 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat mds0_sessionmap 2024-09-16T01:40:15.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.472+0000 7f699851bf40 1 Processor -- start 2024-09-16T01:40:15.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.472+0000 7f699851bf40 1 -- start start 2024-09-16T01:40:15.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.472+0000 7f699851bf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561a80d7bf0 0x5561a816a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.472+0000 7f699851bf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5561a80da700 0x5561a8168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.472+0000 7f699851bf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561a80daae0 0x5561a8169190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.472+0000 7f699851bf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5561a7fe2510 con 0x5561a80d7bf0 2024-09-16T01:40:15.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.472+0000 7f699851bf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5561a808f3c0 con 0x5561a80daae0 2024-09-16T01:40:15.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.472+0000 7f699851bf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5561a815f020 con 0x5561a80da700 2024-09-16T01:40:15.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.473+0000 7f6995563640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561a80d7bf0 0x5561a816a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.473+0000 7f6995563640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561a80d7bf0 0x5561a816a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46926/0 (socket says 172.21.15.88:46926) 2024-09-16T01:40:15.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.473+0000 7f6995563640 1 -- 172.21.15.88:0/1176219406 learned_addr learned my addr 172.21.15.88:0/1176219406 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:15.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.473+0000 7f698cd62640 1 --2- 172.21.15.88:0/1176219406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5561a80da700 0x5561a8168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.473+0000 7f6995d64640 1 --2- 172.21.15.88:0/1176219406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561a80daae0 0x5561a8169190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.473+0000 7f6995563640 1 -- 172.21.15.88:0/1176219406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5561a80da700 msgr2=0x5561a8168c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:15.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.473+0000 7f6995563640 1 --2- 172.21.15.88:0/1176219406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5561a80da700 0x5561a8168c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.473+0000 7f6995563640 1 -- 172.21.15.88:0/1176219406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561a80daae0 msgr2=0x5561a8169190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:15.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.473+0000 7f6995563640 1 --2- 172.21.15.88:0/1176219406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561a80daae0 0x5561a8169190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.473+0000 7f6995563640 1 -- 172.21.15.88:0/1176219406 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561a8161350 con 0x5561a80d7bf0 2024-09-16T01:40:15.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.473+0000 7f698cd62640 1 --2- 172.21.15.88:0/1176219406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5561a80da700 0x5561a8168c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:15.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.473+0000 7f6995563640 1 --2- 172.21.15.88:0/1176219406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561a80d7bf0 0x5561a816a230 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f698800b2e0 tx=0x7f698800b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2af511f3d3077a3f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.473+0000 7f698ffff640 1 -- 172.21.15.88:0/1176219406 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6988008070 con 0x5561a80d7bf0 2024-09-16T01:40:15.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.473+0000 7f698ffff640 1 -- 172.21.15.88:0/1176219406 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f698800d500 con 0x5561a80d7bf0 2024-09-16T01:40:15.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.473+0000 7f698ffff640 1 -- 172.21.15.88:0/1176219406 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698800d8f0 con 0x5561a80d7bf0 2024-09-16T01:40:15.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.474+0000 7f699851bf40 1 -- 172.21.15.88:0/1176219406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561a80d7bf0 msgr2=0x5561a816a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.474+0000 7f699851bf40 1 --2- 172.21.15.88:0/1176219406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561a80d7bf0 0x5561a816a230 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f698800b2e0 tx=0x7f698800b7a0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.474+0000 7f699851bf40 1 -- 172.21.15.88:0/1176219406 shutdown_connections 2024-09-16T01:40:15.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.474+0000 7f699851bf40 1 --2- 172.21.15.88:0/1176219406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561a80daae0 0x5561a8169190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.474+0000 7f699851bf40 1 --2- 172.21.15.88:0/1176219406 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5561a80da700 0x5561a8168c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.474+0000 7f699851bf40 1 --2- 172.21.15.88:0/1176219406 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561a80d7bf0 0x5561a816a230 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.474+0000 7f699851bf40 1 -- 172.21.15.88:0/1176219406 >> 172.21.15.88:0/1176219406 conn(0x5561a80bb680 msgr2=0x5561a8161940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:15.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.474+0000 7f699851bf40 1 -- 172.21.15.88:0/1176219406 shutdown_connections 2024-09-16T01:40:15.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.474+0000 7f699851bf40 1 -- 172.21.15.88:0/1176219406 wait complete. 2024-09-16T01:40:15.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.474+0000 7f699851bf40 1 Processor -- start 2024-09-16T01:40:15.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.474+0000 7f699851bf40 1 -- start start 2024-09-16T01:40:15.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.475+0000 7f699851bf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561a80d7bf0 0x5561a8159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.475+0000 7f699851bf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561a80da700 0x5561a81fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.475+0000 7f699851bf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5561a80daae0 0x5561a8166d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.475+0000 7f699851bf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5561a8163a70 con 0x5561a80d7bf0 2024-09-16T01:40:15.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.475+0000 7f699851bf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5561a815a060 con 0x5561a80da700 2024-09-16T01:40:15.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.475+0000 7f699851bf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5561a815a8f0 con 0x5561a80daae0 2024-09-16T01:40:15.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.475+0000 7f6995563640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561a80d7bf0 0x5561a8159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.475+0000 7f6995563640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561a80d7bf0 0x5561a8159600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46934/0 (socket says 172.21.15.88:46934) 2024-09-16T01:40:15.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.475+0000 7f6995563640 1 -- 172.21.15.88:0/3718303416 learned_addr learned my addr 172.21.15.88:0/3718303416 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:15.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.475+0000 7f698cd62640 1 --2- 172.21.15.88:0/3718303416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561a80da700 0x5561a81fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.475+0000 7f6995d64640 1 --2- 172.21.15.88:0/3718303416 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5561a80daae0 0x5561a8166d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.475+0000 7f6995563640 1 -- 172.21.15.88:0/3718303416 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5561a80daae0 msgr2=0x5561a8166d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.475+0000 7f6995563640 1 --2- 172.21.15.88:0/3718303416 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5561a80daae0 0x5561a8166d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.475+0000 7f6995563640 1 -- 172.21.15.88:0/3718303416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561a80da700 msgr2=0x5561a81fc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.475+0000 7f6995563640 1 --2- 172.21.15.88:0/3718303416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561a80da700 0x5561a81fc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.475+0000 7f6995563640 1 -- 172.21.15.88:0/3718303416 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561a8169070 con 0x5561a80d7bf0 2024-09-16T01:40:15.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.475+0000 7f698cd62640 1 --2- 172.21.15.88:0/3718303416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561a80da700 0x5561a81fc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:15.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.475+0000 7f6995563640 1 --2- 172.21.15.88:0/3718303416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561a80d7bf0 0x5561a8159600 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f6988000c00 tx=0x7f6988000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.476+0000 7f698effd640 1 -- 172.21.15.88:0/3718303416 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6988021030 con 0x5561a80d7bf0 2024-09-16T01:40:15.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.476+0000 7f698effd640 1 -- 172.21.15.88:0/3718303416 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f698801c440 con 0x5561a80d7bf0 2024-09-16T01:40:15.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.476+0000 7f698effd640 1 -- 172.21.15.88:0/3718303416 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698801c630 con 0x5561a80d7bf0 2024-09-16T01:40:15.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.476+0000 7f699851bf40 1 -- 172.21.15.88:0/3718303416 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5561a8169350 con 0x5561a80d7bf0 2024-09-16T01:40:15.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.476+0000 7f699851bf40 1 -- 172.21.15.88:0/3718303416 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561a8159b40 con 0x5561a80d7bf0 2024-09-16T01:40:15.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.477+0000 7f698effd640 1 -- 172.21.15.88:0/3718303416 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6988024070 con 0x5561a80d7bf0 2024-09-16T01:40:15.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.477+0000 7f698effd640 1 --2- 172.21.15.88:0/3718303416 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f697806ac90 0x7f697806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.478+0000 7f698effd640 1 -- 172.21.15.88:0/3718303416 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f698808b1a0 con 0x5561a80d7bf0 2024-09-16T01:40:15.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.478+0000 7f698cd62640 1 --2- 172.21.15.88:0/3718303416 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f697806ac90 0x7f697806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.478+0000 7f699851bf40 1 --2- 172.21.15.88:0/3718303416 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5561a81fcbc0 0x5561a815bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.478+0000 7f699851bf40 1 -- 172.21.15.88:0/3718303416 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:1 12.b 12:d0630e4c:::mds0_sessionmap:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x5561a80c0ec0 con 0x5561a81fcbc0 2024-09-16T01:40:15.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.478+0000 7f6995d64640 1 --2- 172.21.15.88:0/3718303416 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5561a81fcbc0 0x5561a815bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.478+0000 7f698cd62640 1 --2- 172.21.15.88:0/3718303416 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f697806ac90 0x7f697806d150 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x5561a80923c0 tx=0x7f6980005d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.478+0000 7f6995d64640 1 --2- 172.21.15.88:0/3718303416 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5561a81fcbc0 0x5561a815bfb0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.478+0000 7f6995d64640 1 -- 172.21.15.88:0/3718303416 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(1 mds0_sessionmap [stat out=16b] v0'0 uv3 ondisk = 0) ==== 159+0+16 (crc 0 0 0) 0x7f6990056030 con 0x5561a81fcbc0 2024-09-16T01:40:15.484 INFO:teuthology.orchestra.run.smithi088.stdout:cephfs_metadata/mds0_sessionmap mtime 2024-09-16T01:40:11.000000+0000, size 0 2024-09-16T01:40:15.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.478+0000 7f699851bf40 1 -- 172.21.15.88:0/3718303416 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5561a81fcbc0 msgr2=0x5561a815bfb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.478+0000 7f699851bf40 1 --2- 172.21.15.88:0/3718303416 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5561a81fcbc0 0x5561a815bfb0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.479+0000 7f699851bf40 1 -- 172.21.15.88:0/3718303416 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f697806ac90 msgr2=0x7f697806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.479+0000 7f699851bf40 1 --2- 172.21.15.88:0/3718303416 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f697806ac90 0x7f697806d150 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x5561a80923c0 tx=0x7f6980005d50 comp rx=0 tx=0).stop 2024-09-16T01:40:15.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.479+0000 7f699851bf40 1 -- 172.21.15.88:0/3718303416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561a80d7bf0 msgr2=0x5561a8159600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.479+0000 7f699851bf40 1 --2- 172.21.15.88:0/3718303416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5561a80d7bf0 0x5561a8159600 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f6988000c00 tx=0x7f6988000ed0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.479+0000 7f6995d64640 1 -- 172.21.15.88:0/3718303416 reap_dead start 2024-09-16T01:40:15.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.479+0000 7f699851bf40 1 -- 172.21.15.88:0/3718303416 shutdown_connections 2024-09-16T01:40:15.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.479+0000 7f699851bf40 1 -- 172.21.15.88:0/3718303416 >> 172.21.15.88:0/3718303416 conn(0x5561a80bb680 msgr2=0x5561a8161790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:15.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.479+0000 7f699851bf40 1 -- 172.21.15.88:0/3718303416 shutdown_connections 2024-09-16T01:40:15.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.479+0000 7f699851bf40 1 -- 172.21.15.88:0/3718303416 wait complete. 2024-09-16T01:40:15.486 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 400.00000000 2024-09-16T01:40:15.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.567+0000 7f5ea5ed2f40 1 Processor -- start 2024-09-16T01:40:15.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.567+0000 7f5ea5ed2f40 1 -- start start 2024-09-16T01:40:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.568+0000 7f5ea5ed2f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560a19c78d50 0x560a19c49d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.568+0000 7f5ea5ed2f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a19c94ed0 0x560a19d18db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.568+0000 7f5ea5ed2f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560a19d1bac0 0x560a19d192f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.568+0000 7f5ea5ed2f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x560a19b9a510 con 0x560a19d1bac0 2024-09-16T01:40:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.568+0000 7f5ea5ed2f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560a19d1f560 con 0x560a19c94ed0 2024-09-16T01:40:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.568+0000 7f5ea5ed2f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x560a19d1d740 con 0x560a19c78d50 2024-09-16T01:40:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.568+0000 7f5ea41cb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560a19d1bac0 0x560a19d192f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.568+0000 7f5ea41cb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560a19d1bac0 0x560a19d192f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46958/0 (socket says 172.21.15.88:46958) 2024-09-16T01:40:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.568+0000 7f5ea41cb640 1 -- 172.21.15.88:0/4029865111 learned_addr learned my addr 172.21.15.88:0/4029865111 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.568+0000 7f5ea41cb640 1 -- 172.21.15.88:0/4029865111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560a19c78d50 msgr2=0x560a19c49d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.568+0000 7f5ea39ca640 1 --2- 172.21.15.88:0/4029865111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560a19c78d50 0x560a19c49d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.568+0000 7f5e9b1c9640 1 --2- 172.21.15.88:0/4029865111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a19c94ed0 0x560a19d18db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.568+0000 7f5ea41cb640 1 --2- 172.21.15.88:0/4029865111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560a19c78d50 0x560a19c49d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.568+0000 7f5ea41cb640 1 -- 172.21.15.88:0/4029865111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a19c94ed0 msgr2=0x560a19d18db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.568+0000 7f5ea41cb640 1 --2- 172.21.15.88:0/4029865111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a19c94ed0 0x560a19d18db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.568+0000 7f5ea41cb640 1 -- 172.21.15.88:0/4029865111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a19c48f80 con 0x560a19d1bac0 2024-09-16T01:40:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.568+0000 7f5ea41cb640 1 --2- 172.21.15.88:0/4029865111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560a19d1bac0 0x560a19d192f0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c00c480 tx=0x7f5e9c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7886d631d6974454 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.568+0000 7f5ea29c8640 1 -- 172.21.15.88:0/4029865111 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c0116b0 con 0x560a19d1bac0 2024-09-16T01:40:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.568+0000 7f5ea29c8640 1 -- 172.21.15.88:0/4029865111 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e9c017070 con 0x560a19d1bac0 2024-09-16T01:40:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.568+0000 7f5ea29c8640 1 -- 172.21.15.88:0/4029865111 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c011a90 con 0x560a19d1bac0 2024-09-16T01:40:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.569+0000 7f5ea5ed2f40 1 -- 172.21.15.88:0/4029865111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560a19d1bac0 msgr2=0x560a19d192f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.569+0000 7f5ea5ed2f40 1 --2- 172.21.15.88:0/4029865111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560a19d1bac0 0x560a19d192f0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c00c480 tx=0x7f5e9c00c940 comp rx=0 tx=0).stop 2024-09-16T01:40:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.569+0000 7f5ea5ed2f40 1 -- 172.21.15.88:0/4029865111 shutdown_connections 2024-09-16T01:40:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.569+0000 7f5ea5ed2f40 1 --2- 172.21.15.88:0/4029865111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560a19d1bac0 0x560a19d192f0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.569+0000 7f5ea5ed2f40 1 --2- 172.21.15.88:0/4029865111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a19c94ed0 0x560a19d18db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.569+0000 7f5ea5ed2f40 1 --2- 172.21.15.88:0/4029865111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560a19c78d50 0x560a19c49d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.569+0000 7f5ea5ed2f40 1 -- 172.21.15.88:0/4029865111 >> 172.21.15.88:0/4029865111 conn(0x560a19c473c0 msgr2=0x560a19c477a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.569+0000 7f5ea5ed2f40 1 -- 172.21.15.88:0/4029865111 shutdown_connections 2024-09-16T01:40:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.569+0000 7f5ea5ed2f40 1 -- 172.21.15.88:0/4029865111 wait complete. 2024-09-16T01:40:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.569+0000 7f5ea5ed2f40 1 Processor -- start 2024-09-16T01:40:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.570+0000 7f5ea5ed2f40 1 -- start start 2024-09-16T01:40:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.570+0000 7f5ea5ed2f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560a19c78d50 0x560a19dab7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.570+0000 7f5ea5ed2f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a19c94ed0 0x560a19c76c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.570+0000 7f5ea5ed2f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560a19d1bac0 0x560a19d15110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.570+0000 7f5ea5ed2f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x560a19c49540 con 0x560a19c78d50 2024-09-16T01:40:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.570+0000 7f5ea5ed2f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560a19c49260 con 0x560a19c94ed0 2024-09-16T01:40:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.570+0000 7f5ea5ed2f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x560a19c49820 con 0x560a19d1bac0 2024-09-16T01:40:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.570+0000 7f5ea39ca640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560a19c78d50 0x560a19dab7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.570+0000 7f5ea39ca640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560a19c78d50 0x560a19dab7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46968/0 (socket says 172.21.15.88:46968) 2024-09-16T01:40:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.570+0000 7f5ea39ca640 1 -- 172.21.15.88:0/1191500733 learned_addr learned my addr 172.21.15.88:0/1191500733 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.570+0000 7f5e9b1c9640 1 --2- 172.21.15.88:0/1191500733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a19c94ed0 0x560a19c76c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.570+0000 7f5ea41cb640 1 --2- 172.21.15.88:0/1191500733 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560a19d1bac0 0x560a19d15110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.571+0000 7f5ea39ca640 1 -- 172.21.15.88:0/1191500733 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560a19d1bac0 msgr2=0x560a19d15110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.571+0000 7f5ea39ca640 1 --2- 172.21.15.88:0/1191500733 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560a19d1bac0 0x560a19d15110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.571+0000 7f5ea39ca640 1 -- 172.21.15.88:0/1191500733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a19c94ed0 msgr2=0x560a19c76c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.571+0000 7f5ea39ca640 1 --2- 172.21.15.88:0/1191500733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a19c94ed0 0x560a19c76c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.571+0000 7f5ea39ca640 1 -- 172.21.15.88:0/1191500733 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a19d1a440 con 0x560a19c78d50 2024-09-16T01:40:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.571+0000 7f5ea39ca640 1 --2- 172.21.15.88:0/1191500733 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560a19c78d50 0x560a19dab7e0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f5e94045e50 tx=0x7f5e94002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.571+0000 7f5ea19c6640 1 -- 172.21.15.88:0/1191500733 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e94062020 con 0x560a19c78d50 2024-09-16T01:40:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.571+0000 7f5ea19c6640 1 -- 172.21.15.88:0/1191500733 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e94054bb0 con 0x560a19c78d50 2024-09-16T01:40:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.571+0000 7f5ea5ed2f40 1 -- 172.21.15.88:0/1191500733 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x560a19d1a720 con 0x560a19c78d50 2024-09-16T01:40:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.571+0000 7f5ea19c6640 1 -- 172.21.15.88:0/1191500733 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e94059040 con 0x560a19c78d50 2024-09-16T01:40:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.571+0000 7f5ea5ed2f40 1 -- 172.21.15.88:0/1191500733 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a19d1ace0 con 0x560a19c78d50 2024-09-16T01:40:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.572+0000 7f5ea19c6640 1 -- 172.21.15.88:0/1191500733 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5e94065020 con 0x560a19c78d50 2024-09-16T01:40:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.573+0000 7f5ea19c6640 1 --2- 172.21.15.88:0/1191500733 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e8406ac90 0x7f5e8406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.573+0000 7f5e9b1c9640 1 --2- 172.21.15.88:0/1191500733 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e8406ac90 0x7f5e8406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.573+0000 7f5ea19c6640 1 -- 172.21.15.88:0/1191500733 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f5e940cd910 con 0x560a19c78d50 2024-09-16T01:40:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.573+0000 7f5ea5ed2f40 1 --2- 172.21.15.88:0/1191500733 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x560a19c77150 0x560a19d172c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.573+0000 7f5ea5ed2f40 1 -- 172.21.15.88:0/1191500733 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:1 12.f 12:f1f69726:::400.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x560a19d15650 con 0x560a19c77150 2024-09-16T01:40:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.573+0000 7f5e9b1c9640 1 --2- 172.21.15.88:0/1191500733 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e8406ac90 0x7f5e8406d150 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x560a19c94280 tx=0x7f5e8c001720 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.573+0000 7f5ea41cb640 1 --2- 172.21.15.88:0/1191500733 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x560a19c77150 0x560a19d172c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.574+0000 7f5ea41cb640 1 --2- 172.21.15.88:0/1191500733 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x560a19c77150 0x560a19d172c0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.574+0000 7f5ea41cb640 1 -- 172.21.15.88:0/1191500733 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(1 400.00000000 [stat out=16b] v0'0 uv1 ondisk = 0) ==== 156+0+16 (crc 0 0 0) 0x7f5e9c0118d0 con 0x560a19c77150 2024-09-16T01:40:15.617 INFO:teuthology.orchestra.run.smithi088.stdout:cephfs_metadata/400.00000000 mtime 2024-09-16T01:40:04.000000+0000, size 22 2024-09-16T01:40:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.574+0000 7f5ea5ed2f40 1 -- 172.21.15.88:0/1191500733 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x560a19c77150 msgr2=0x560a19d172c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.574+0000 7f5ea5ed2f40 1 --2- 172.21.15.88:0/1191500733 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x560a19c77150 0x560a19d172c0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.574+0000 7f5ea5ed2f40 1 -- 172.21.15.88:0/1191500733 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e8406ac90 msgr2=0x7f5e8406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.574+0000 7f5ea5ed2f40 1 --2- 172.21.15.88:0/1191500733 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e8406ac90 0x7f5e8406d150 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x560a19c94280 tx=0x7f5e8c001720 comp rx=0 tx=0).stop 2024-09-16T01:40:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.574+0000 7f5ea5ed2f40 1 -- 172.21.15.88:0/1191500733 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560a19c78d50 msgr2=0x560a19dab7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.574+0000 7f5ea5ed2f40 1 --2- 172.21.15.88:0/1191500733 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560a19c78d50 0x560a19dab7e0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f5e94045e50 tx=0x7f5e94002fb0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.574+0000 7f5ea41cb640 1 -- 172.21.15.88:0/1191500733 reap_dead start 2024-09-16T01:40:15.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.574+0000 7f5ea5ed2f40 1 -- 172.21.15.88:0/1191500733 shutdown_connections 2024-09-16T01:40:15.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.574+0000 7f5ea5ed2f40 1 -- 172.21.15.88:0/1191500733 >> 172.21.15.88:0/1191500733 conn(0x560a19c473c0 msgr2=0x560a19c77b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:15.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.574+0000 7f5ea5ed2f40 1 -- 172.21.15.88:0/1191500733 shutdown_connections 2024-09-16T01:40:15.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.574+0000 7f5ea5ed2f40 1 -- 172.21.15.88:0/1191500733 wait complete. 2024-09-16T01:40:15.619 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 100.00000000.inode 2024-09-16T01:40:15.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.700+0000 7f1d7781cf40 1 Processor -- start 2024-09-16T01:40:15.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.700+0000 7f1d7781cf40 1 -- start start 2024-09-16T01:40:15.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.700+0000 7f1d7781cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d640022d0 0x55c3a675e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.700+0000 7f1d7781cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c3a67896f0 0x55c3a675f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.700+0000 7f1d7781cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c3a675f5c0 0x55c3a6835300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.700+0000 7f1d7781cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55c3a66b1510 con 0x55c3a67896f0 2024-09-16T01:40:15.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.700+0000 7f1d7781cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c3a670ac50 con 0x7f1d640022d0 2024-09-16T01:40:15.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.700+0000 7f1d7781cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55c3a68377b0 con 0x55c3a675f5c0 2024-09-16T01:40:15.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.700+0000 7f1d6cb11640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c3a67896f0 0x55c3a675f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.700+0000 7f1d6cb11640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c3a67896f0 0x55c3a675f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46988/0 (socket says 172.21.15.88:46988) 2024-09-16T01:40:15.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.701+0000 7f1d6cb11640 1 -- 172.21.15.88:0/1518820290 learned_addr learned my addr 172.21.15.88:0/1518820290 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:15.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.701+0000 7f1d75312640 1 --2- 172.21.15.88:0/1518820290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d640022d0 0x55c3a675e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.701+0000 7f1d75b13640 1 --2- 172.21.15.88:0/1518820290 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c3a675f5c0 0x55c3a6835300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.701+0000 7f1d6cb11640 1 -- 172.21.15.88:0/1518820290 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c3a675f5c0 msgr2=0x55c3a6835300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:15.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.701+0000 7f1d6cb11640 1 --2- 172.21.15.88:0/1518820290 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c3a675f5c0 0x55c3a6835300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.701+0000 7f1d6cb11640 1 -- 172.21.15.88:0/1518820290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d640022d0 msgr2=0x55c3a675e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:15.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.701+0000 7f1d6cb11640 1 --2- 172.21.15.88:0/1518820290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d640022d0 0x55c3a675e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.701+0000 7f1d6cb11640 1 -- 172.21.15.88:0/1518820290 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3a682e960 con 0x55c3a67896f0 2024-09-16T01:40:15.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.701+0000 7f1d6cb11640 1 --2- 172.21.15.88:0/1518820290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c3a67896f0 0x55c3a675f080 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f1d60009760 tx=0x7f1d60009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92f43ef676d4b909 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.701+0000 7f1d6ffff640 1 -- 172.21.15.88:0/1518820290 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d60007800 con 0x55c3a67896f0 2024-09-16T01:40:15.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.701+0000 7f1d6ffff640 1 -- 172.21.15.88:0/1518820290 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d60019070 con 0x55c3a67896f0 2024-09-16T01:40:15.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.701+0000 7f1d6ffff640 1 -- 172.21.15.88:0/1518820290 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d60007ba0 con 0x55c3a67896f0 2024-09-16T01:40:15.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.701+0000 7f1d7781cf40 1 -- 172.21.15.88:0/1518820290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c3a67896f0 msgr2=0x55c3a675f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.701+0000 7f1d7781cf40 1 --2- 172.21.15.88:0/1518820290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c3a67896f0 0x55c3a675f080 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f1d60009760 tx=0x7f1d60009850 comp rx=0 tx=0).stop 2024-09-16T01:40:15.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.701+0000 7f1d7781cf40 1 -- 172.21.15.88:0/1518820290 shutdown_connections 2024-09-16T01:40:15.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.701+0000 7f1d7781cf40 1 --2- 172.21.15.88:0/1518820290 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c3a675f5c0 0x55c3a6835300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.701+0000 7f1d7781cf40 1 --2- 172.21.15.88:0/1518820290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c3a67896f0 0x55c3a675f080 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.701+0000 7f1d7781cf40 1 --2- 172.21.15.88:0/1518820290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d640022d0 0x55c3a675e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.701+0000 7f1d7781cf40 1 -- 172.21.15.88:0/1518820290 >> 172.21.15.88:0/1518820290 conn(0x55c3a678a960 msgr2=0x55c3a67a3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:15.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.702+0000 7f1d7781cf40 1 -- 172.21.15.88:0/1518820290 shutdown_connections 2024-09-16T01:40:15.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.702+0000 7f1d7781cf40 1 -- 172.21.15.88:0/1518820290 wait complete. 2024-09-16T01:40:15.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.702+0000 7f1d7781cf40 1 Processor -- start 2024-09-16T01:40:15.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.702+0000 7f1d7781cf40 1 -- start start 2024-09-16T01:40:15.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.703+0000 7f1d7781cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3a675f5c0 0x55c3a6784c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.703+0000 7f1d7781cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c3a67896f0 0x55c3a682cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.703+0000 7f1d7781cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d640022d0 0x55c3a682d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.703+0000 7f1d7781cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55c3a6836790 con 0x7f1d640022d0 2024-09-16T01:40:15.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.703+0000 7f1d7781cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c3a682ede0 con 0x55c3a675f5c0 2024-09-16T01:40:15.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.703+0000 7f1d7781cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55c3a682e610 con 0x55c3a67896f0 2024-09-16T01:40:15.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.703+0000 7f1d75b13640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d640022d0 0x55c3a682d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.703+0000 7f1d75b13640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d640022d0 0x55c3a682d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46994/0 (socket says 172.21.15.88:46994) 2024-09-16T01:40:15.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.703+0000 7f1d75b13640 1 -- 172.21.15.88:0/2299081303 learned_addr learned my addr 172.21.15.88:0/2299081303 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:15.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.703+0000 7f1d6cb11640 1 --2- 172.21.15.88:0/2299081303 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c3a67896f0 0x55c3a682cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.703+0000 7f1d75312640 1 --2- 172.21.15.88:0/2299081303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3a675f5c0 0x55c3a6784c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.703+0000 7f1d75b13640 1 -- 172.21.15.88:0/2299081303 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c3a67896f0 msgr2=0x55c3a682cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.703+0000 7f1d75b13640 1 --2- 172.21.15.88:0/2299081303 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c3a67896f0 0x55c3a682cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.703+0000 7f1d75b13640 1 -- 172.21.15.88:0/2299081303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3a675f5c0 msgr2=0x55c3a6784c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.703+0000 7f1d75b13640 1 --2- 172.21.15.88:0/2299081303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3a675f5c0 0x55c3a6784c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.703+0000 7f1d75b13640 1 -- 172.21.15.88:0/2299081303 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3a675e3c0 con 0x7f1d640022d0 2024-09-16T01:40:15.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.703+0000 7f1d75b13640 1 --2- 172.21.15.88:0/2299081303 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d640022d0 0x55c3a682d2f0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f1d70000f80 tx=0x7f1d7000d530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.703+0000 7f1d6effd640 1 -- 172.21.15.88:0/2299081303 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d70006240 con 0x7f1d640022d0 2024-09-16T01:40:15.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.704+0000 7f1d6effd640 1 -- 172.21.15.88:0/2299081303 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d70017070 con 0x7f1d640022d0 2024-09-16T01:40:15.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.704+0000 7f1d7781cf40 1 -- 172.21.15.88:0/2299081303 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c3a682d930 con 0x7f1d640022d0 2024-09-16T01:40:15.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.704+0000 7f1d6effd640 1 -- 172.21.15.88:0/2299081303 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7000ddd0 con 0x7f1d640022d0 2024-09-16T01:40:15.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.704+0000 7f1d7781cf40 1 -- 172.21.15.88:0/2299081303 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3a678d6e0 con 0x7f1d640022d0 2024-09-16T01:40:15.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.705+0000 7f1d6effd640 1 -- 172.21.15.88:0/2299081303 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1d7001c460 con 0x7f1d640022d0 2024-09-16T01:40:15.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.705+0000 7f1d6effd640 1 --2- 172.21.15.88:0/2299081303 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1d5806ac90 0x7f1d5806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.705+0000 7f1d6effd640 1 -- 172.21.15.88:0/2299081303 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f1d70089930 con 0x7f1d640022d0 2024-09-16T01:40:15.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.705+0000 7f1d75312640 1 --2- 172.21.15.88:0/2299081303 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1d5806ac90 0x7f1d5806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.706+0000 7f1d7781cf40 1 --2- 172.21.15.88:0/2299081303 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55c3a67a3fa0 0x55c3a6782f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.706+0000 7f1d7781cf40 1 -- 172.21.15.88:0/2299081303 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:1 12.1f 12:fe07bba1:::100.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55c3a68322d0 con 0x55c3a67a3fa0 2024-09-16T01:40:15.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.706+0000 7f1d75312640 1 --2- 172.21.15.88:0/2299081303 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1d5806ac90 0x7f1d5806d150 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1d68056000 tx=0x7f1d68051000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.706+0000 7f1d6cb11640 1 --2- 172.21.15.88:0/2299081303 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55c3a67a3fa0 0x55c3a6782f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.706+0000 7f1d6cb11640 1 --2- 172.21.15.88:0/2299081303 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55c3a67a3fa0 0x55c3a6782f70 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.707+0000 7f1d6cb11640 1 -- 172.21.15.88:0/2299081303 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(1 100.00000000.inode [stat out=16b] v0'0 uv3 ondisk = 0) ==== 162+0+16 (crc 0 0 0) 0x7f1d600078b0 con 0x55c3a67a3fa0 2024-09-16T01:40:15.711 INFO:teuthology.orchestra.run.smithi088.stdout:cephfs_metadata/100.00000000.inode mtime 2024-09-16T01:40:04.000000+0000, size 582 2024-09-16T01:40:15.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.707+0000 7f1d7781cf40 1 -- 172.21.15.88:0/2299081303 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55c3a67a3fa0 msgr2=0x55c3a6782f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.707+0000 7f1d7781cf40 1 --2- 172.21.15.88:0/2299081303 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55c3a67a3fa0 0x55c3a6782f70 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.707+0000 7f1d7781cf40 1 -- 172.21.15.88:0/2299081303 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1d5806ac90 msgr2=0x7f1d5806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.707+0000 7f1d7781cf40 1 --2- 172.21.15.88:0/2299081303 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1d5806ac90 0x7f1d5806d150 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1d68056000 tx=0x7f1d68051000 comp rx=0 tx=0).stop 2024-09-16T01:40:15.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.707+0000 7f1d7781cf40 1 -- 172.21.15.88:0/2299081303 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d640022d0 msgr2=0x55c3a682d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.707+0000 7f1d7781cf40 1 --2- 172.21.15.88:0/2299081303 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d640022d0 0x55c3a682d2f0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f1d70000f80 tx=0x7f1d7000d530 comp rx=0 tx=0).stop 2024-09-16T01:40:15.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.707+0000 7f1d75b13640 1 -- 172.21.15.88:0/2299081303 reap_dead start 2024-09-16T01:40:15.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.707+0000 7f1d7781cf40 1 -- 172.21.15.88:0/2299081303 shutdown_connections 2024-09-16T01:40:15.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.707+0000 7f1d7781cf40 1 -- 172.21.15.88:0/2299081303 >> 172.21.15.88:0/2299081303 conn(0x55c3a678a960 msgr2=0x55c3a68331d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:15.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.707+0000 7f1d7781cf40 1 -- 172.21.15.88:0/2299081303 shutdown_connections 2024-09-16T01:40:15.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.707+0000 7f1d7781cf40 1 -- 172.21.15.88:0/2299081303 wait complete. 2024-09-16T01:40:15.713 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 1.00000000 2024-09-16T01:40:15.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.795+0000 7fd27b309f40 1 Processor -- start 2024-09-16T01:40:15.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.795+0000 7fd27b309f40 1 -- start start 2024-09-16T01:40:15.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.795+0000 7fd27b309f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56221019cfe0 0x5622102410c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.795+0000 7fd27b309f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562210241d20 0x5622101b6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.795+0000 7fd27b309f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562210242100 0x562210198ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.795+0000 7fd27b309f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5622100bf510 con 0x562210241d20 2024-09-16T01:40:15.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.795+0000 7fd27b309f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5622101b5bd0 con 0x562210242100 2024-09-16T01:40:15.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.795+0000 7fd27b309f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56221015cb10 con 0x56221019cfe0 2024-09-16T01:40:15.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.795+0000 7fd2785fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562210241d20 0x5622101b6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.795+0000 7fd2785fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562210241d20 0x5622101b6ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47012/0 (socket says 172.21.15.88:47012) 2024-09-16T01:40:15.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.795+0000 7fd2785fe640 1 -- 172.21.15.88:0/1859401558 learned_addr learned my addr 172.21.15.88:0/1859401558 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:15.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.796+0000 7fd278dff640 1 --2- 172.21.15.88:0/1859401558 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56221019cfe0 0x5622102410c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.796+0000 7fd279600640 1 --2- 172.21.15.88:0/1859401558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562210242100 0x562210198ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.796+0000 7fd2785fe640 1 -- 172.21.15.88:0/1859401558 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56221019cfe0 msgr2=0x5622102410c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:15.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.796+0000 7fd2785fe640 1 --2- 172.21.15.88:0/1859401558 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56221019cfe0 0x5622102410c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.796+0000 7fd2785fe640 1 -- 172.21.15.88:0/1859401558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562210242100 msgr2=0x562210198ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:15.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.796+0000 7fd2785fe640 1 --2- 172.21.15.88:0/1859401558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562210242100 0x562210198ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.796+0000 7fd2785fe640 1 -- 172.21.15.88:0/1859401558 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562210232980 con 0x562210241d20 2024-09-16T01:40:15.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.796+0000 7fd279600640 1 --2- 172.21.15.88:0/1859401558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562210242100 0x562210198ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:15.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.796+0000 7fd278dff640 1 --2- 172.21.15.88:0/1859401558 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56221019cfe0 0x5622102410c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:15.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.796+0000 7fd2785fe640 1 --2- 172.21.15.88:0/1859401558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562210241d20 0x5622101b6ed0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fd26c00a4f0 tx=0x7fd26c051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1694d552b059cbd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.796+0000 7fd2775fc640 1 -- 172.21.15.88:0/1859401558 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26c06d070 con 0x562210241d20 2024-09-16T01:40:15.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.796+0000 7fd2775fc640 1 -- 172.21.15.88:0/1859401558 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd26c05d980 con 0x562210241d20 2024-09-16T01:40:15.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.796+0000 7fd2775fc640 1 -- 172.21.15.88:0/1859401558 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26c05dd50 con 0x562210241d20 2024-09-16T01:40:15.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.796+0000 7fd27b309f40 1 -- 172.21.15.88:0/1859401558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562210241d20 msgr2=0x5622101b6ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.796+0000 7fd27b309f40 1 --2- 172.21.15.88:0/1859401558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562210241d20 0x5622101b6ed0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fd26c00a4f0 tx=0x7fd26c051000 comp rx=0 tx=0).stop 2024-09-16T01:40:15.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.796+0000 7fd27b309f40 1 -- 172.21.15.88:0/1859401558 shutdown_connections 2024-09-16T01:40:15.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.796+0000 7fd27b309f40 1 --2- 172.21.15.88:0/1859401558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562210242100 0x562210198ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.796+0000 7fd27b309f40 1 --2- 172.21.15.88:0/1859401558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562210241d20 0x5622101b6ed0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.796+0000 7fd27b309f40 1 --2- 172.21.15.88:0/1859401558 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56221019cfe0 0x5622102410c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.796+0000 7fd27b309f40 1 -- 172.21.15.88:0/1859401558 >> 172.21.15.88:0/1859401558 conn(0x5622101b8750 msgr2=0x5622101b8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:15.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.796+0000 7fd27b309f40 1 -- 172.21.15.88:0/1859401558 shutdown_connections 2024-09-16T01:40:15.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.796+0000 7fd27b309f40 1 -- 172.21.15.88:0/1859401558 wait complete. 2024-09-16T01:40:15.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.797+0000 7fd27b309f40 1 Processor -- start 2024-09-16T01:40:15.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.797+0000 7fd27b309f40 1 -- start start 2024-09-16T01:40:15.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.798+0000 7fd27b309f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56221019cfe0 0x56221023b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.798+0000 7fd27b309f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562210241d20 0x5622102312f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.798+0000 7fd27b309f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562210242100 0x56221023a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.798+0000 7fd27b309f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5622101984d0 con 0x562210241d20 2024-09-16T01:40:15.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.798+0000 7fd27b309f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5622101987b0 con 0x562210242100 2024-09-16T01:40:15.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.798+0000 7fd27b309f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56221015c7a0 con 0x56221019cfe0 2024-09-16T01:40:15.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.798+0000 7fd2785fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562210241d20 0x5622102312f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.798+0000 7fd2785fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562210241d20 0x5622102312f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47018/0 (socket says 172.21.15.88:47018) 2024-09-16T01:40:15.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.798+0000 7fd2785fe640 1 -- 172.21.15.88:0/1742309573 learned_addr learned my addr 172.21.15.88:0/1742309573 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:15.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.798+0000 7fd278dff640 1 --2- 172.21.15.88:0/1742309573 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56221019cfe0 0x56221023b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.798+0000 7fd279600640 1 --2- 172.21.15.88:0/1742309573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562210242100 0x56221023a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.798+0000 7fd2785fe640 1 -- 172.21.15.88:0/1742309573 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56221019cfe0 msgr2=0x56221023b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.798+0000 7fd2785fe640 1 --2- 172.21.15.88:0/1742309573 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56221019cfe0 0x56221023b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.798+0000 7fd2785fe640 1 -- 172.21.15.88:0/1742309573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562210242100 msgr2=0x56221023a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.798+0000 7fd2785fe640 1 --2- 172.21.15.88:0/1742309573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562210242100 0x56221023a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.798+0000 7fd2785fe640 1 -- 172.21.15.88:0/1742309573 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622101712a0 con 0x562210241d20 2024-09-16T01:40:15.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.798+0000 7fd2785fe640 1 --2- 172.21.15.88:0/1742309573 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562210241d20 0x5622102312f0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fd26c025c20 tx=0x7fd26c002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.798+0000 7fd2765fa640 1 -- 172.21.15.88:0/1742309573 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26c076070 con 0x562210241d20 2024-09-16T01:40:15.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.798+0000 7fd2765fa640 1 -- 172.21.15.88:0/1742309573 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd26c05ed80 con 0x562210241d20 2024-09-16T01:40:15.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.798+0000 7fd27b309f40 1 -- 172.21.15.88:0/1742309573 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5622102447d0 con 0x562210241d20 2024-09-16T01:40:15.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.798+0000 7fd2765fa640 1 -- 172.21.15.88:0/1742309573 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26c06d040 con 0x562210241d20 2024-09-16T01:40:15.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.798+0000 7fd27b309f40 1 -- 172.21.15.88:0/1742309573 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562210232340 con 0x562210241d20 2024-09-16T01:40:15.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.800+0000 7fd2765fa640 1 -- 172.21.15.88:0/1742309573 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd26c05d240 con 0x562210241d20 2024-09-16T01:40:15.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.800+0000 7fd2765fa640 1 --2- 172.21.15.88:0/1742309573 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd26406ac90 0x7fd26406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.800+0000 7fd2765fa640 1 -- 172.21.15.88:0/1742309573 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7fd26c0d0c60 con 0x562210241d20 2024-09-16T01:40:15.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.800+0000 7fd278dff640 1 --2- 172.21.15.88:0/1742309573 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd26406ac90 0x7fd26406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.800+0000 7fd27b309f40 1 --2- 172.21.15.88:0/1742309573 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56221023dd90 0x562210242510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:15.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.800+0000 7fd27b309f40 1 -- 172.21.15.88:0/1742309573 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:1 12.1f 12:ff5b34d6:::1.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x562210199790 con 0x56221023dd90 2024-09-16T01:40:15.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.801+0000 7fd278dff640 1 --2- 172.21.15.88:0/1742309573 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd26406ac90 0x7fd26406d150 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd268000f80 tx=0x7fd268008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.801+0000 7fd279600640 1 --2- 172.21.15.88:0/1742309573 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56221023dd90 0x562210242510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:15.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.801+0000 7fd279600640 1 --2- 172.21.15.88:0/1742309573 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56221023dd90 0x562210242510 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:15.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.801+0000 7fd279600640 1 -- 172.21.15.88:0/1742309573 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(1 1.00000000 [stat out=16b] v0'0 uv6 ondisk = 0) ==== 154+0+16 (crc 0 0 0) 0x7fd2700075a0 con 0x56221023dd90 2024-09-16T01:40:15.807 INFO:teuthology.orchestra.run.smithi088.stdout:cephfs_metadata/1.00000000 mtime 2024-09-16T01:40:11.000000+0000, size 0 2024-09-16T01:40:15.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.801+0000 7fd27b309f40 1 -- 172.21.15.88:0/1742309573 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56221023dd90 msgr2=0x562210242510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.801+0000 7fd27b309f40 1 --2- 172.21.15.88:0/1742309573 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56221023dd90 0x562210242510 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.801+0000 7fd27b309f40 1 -- 172.21.15.88:0/1742309573 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd26406ac90 msgr2=0x7fd26406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.801+0000 7fd27b309f40 1 --2- 172.21.15.88:0/1742309573 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd26406ac90 0x7fd26406d150 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd268000f80 tx=0x7fd268008040 comp rx=0 tx=0).stop 2024-09-16T01:40:15.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.801+0000 7fd27b309f40 1 -- 172.21.15.88:0/1742309573 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562210241d20 msgr2=0x5622102312f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:15.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.801+0000 7fd27b309f40 1 --2- 172.21.15.88:0/1742309573 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562210241d20 0x5622102312f0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fd26c025c20 tx=0x7fd26c002fb0 comp rx=0 tx=0).stop 2024-09-16T01:40:15.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.801+0000 7fd279600640 1 -- 172.21.15.88:0/1742309573 reap_dead start 2024-09-16T01:40:15.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.802+0000 7fd27b309f40 1 -- 172.21.15.88:0/1742309573 shutdown_connections 2024-09-16T01:40:15.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.802+0000 7fd27b309f40 1 -- 172.21.15.88:0/1742309573 >> 172.21.15.88:0/1742309573 conn(0x5622101b8750 msgr2=0x56221019bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:15.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.802+0000 7fd27b309f40 1 -- 172.21.15.88:0/1742309573 shutdown_connections 2024-09-16T01:40:15.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:15.802+0000 7fd27b309f40 1 -- 172.21.15.88:0/1742309573 wait complete. 2024-09-16T01:40:15.809 INFO:tasks.cephfs.test_damage:Expectation on rm '10000000000.00000000' will be 'no damage' 2024-09-16T01:40:15.809 INFO:tasks.cephfs.test_damage:Expectation on rm '600.00000000' will be 'damaged_on_start' 2024-09-16T01:40:15.809 INFO:tasks.cephfs.test_damage:Expectation on rm '1.00000000.inode' will be 'damaged_on_start' 2024-09-16T01:40:15.809 INFO:tasks.cephfs.test_damage:Expectation on rm '200.00000000' will be 'damaged_on_start' 2024-09-16T01:40:15.809 INFO:tasks.cephfs.test_damage:Expectation on rm '200.00000001' will be 'damaged_on_start' 2024-09-16T01:40:15.809 INFO:tasks.cephfs.test_damage:Expectation on rm 'mds0_openfiles.0' will be 'no damage' 2024-09-16T01:40:15.809 INFO:tasks.cephfs.test_damage:Expectation on rm '500.00000000' will be 'no damage' 2024-09-16T01:40:15.809 INFO:tasks.cephfs.test_damage:Expectation on rm 'mds_snaptable' will be 'damaged_on_start' 2024-09-16T01:40:15.810 INFO:tasks.cephfs.test_damage:Expectation on rm 'mds0_inotable' will be 'damaged_on_start' 2024-09-16T01:40:15.810 INFO:tasks.cephfs.test_damage:Expectation on rm '100.00000000' will be 'damaged_on_start' 2024-09-16T01:40:15.810 INFO:tasks.cephfs.test_damage:Expectation on rm 'mds0_sessionmap' will be 'damaged_on_start' 2024-09-16T01:40:15.810 INFO:tasks.cephfs.test_damage:Expectation on rm '400.00000000' will be 'no damage' 2024-09-16T01:40:15.810 INFO:tasks.cephfs.test_damage:Expectation on rm '100.00000000.inode' will be 'damaged_on_start' 2024-09-16T01:40:15.810 INFO:tasks.cephfs.test_damage:Expectation on rm '1.00000000' will be 'damaged_on_start' 2024-09-16T01:40:15.810 INFO:tasks.cephfs.test_damage:Expectation on corrupt header '10000000000.00000000' will be 'no damage' 2024-09-16T01:40:15.810 INFO:tasks.cephfs.test_damage:Expectation on corrupt header '600.00000000' will be 'damaged_on_start' 2024-09-16T01:40:15.810 INFO:tasks.cephfs.test_damage:Expectation on corrupt header 'mds0_openfiles.0' will be 'no damage' 2024-09-16T01:40:15.810 INFO:tasks.cephfs.test_damage:Expectation on corrupt header '100.00000000' will be 'damaged_on_start' 2024-09-16T01:40:15.810 INFO:tasks.cephfs.test_damage:Expectation on corrupt header 'mds0_sessionmap' will be 'damaged_on_start' 2024-09-16T01:40:15.810 INFO:tasks.cephfs.test_damage:Expectation on corrupt header '1.00000000' will be 'damaged_on_start' 2024-09-16T01:40:15.810 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 10000000000.00000000' 2024-09-16T01:40:15.811 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:40:15.811 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:40:15.864 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:40:15.864 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:40:15.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:15.864 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:40:15.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:40:15.919 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:40:15.919 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:40:16.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.057+0000 7f9fed0dd640 1 Processor -- start 2024-09-16T01:40:16.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.057+0000 7f9fed0dd640 1 -- start start 2024-09-16T01:40:16.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.058+0000 7f9fed0dd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9fe8158a00 0x7f9fe8071f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:16.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.058+0000 7f9fed0dd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9fe8072470 0x7f9fe80b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:16.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.058+0000 7f9fed0dd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fe8158de0 0x7f9fe80b53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:16.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.058+0000 7f9fed0dd640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f9fe80728f0 con 0x7f9fe8072470 2024-09-16T01:40:16.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.058+0000 7f9fed0dd640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9fe8072bd0 con 0x7f9fe8158a00 2024-09-16T01:40:16.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.058+0000 7f9fed0dd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9fe8072eb0 con 0x7f9fe8158de0 2024-09-16T01:40:16.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.058+0000 7f9fdffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9fe8072470 0x7f9fe80b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:16.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.058+0000 7f9fdffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9fe8072470 0x7f9fe80b4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47026/0 (socket says 172.21.15.88:47026) 2024-09-16T01:40:16.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.058+0000 7f9fdffff640 1 -- 172.21.15.88:0/649848149 learned_addr learned my addr 172.21.15.88:0/649848149 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:16.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.058+0000 7f9fe6d9d640 1 --2- 172.21.15.88:0/649848149 >> v2:172.21.15.154:3300/0 conn(0x7f9fe8158a00 0x7f9fe8071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:16.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.058+0000 7f9fe759e640 1 --2- 172.21.15.88:0/649848149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fe8158de0 0x7f9fe80b53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:16.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.058+0000 7f9fdffff640 1 -- 172.21.15.88:0/649848149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fe8158de0 msgr2=0x7f9fe80b53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:16.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.058+0000 7f9fdffff640 1 --2- 172.21.15.88:0/649848149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fe8158de0 0x7f9fe80b53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:16.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.058+0000 7f9fdffff640 1 -- 172.21.15.88:0/649848149 >> v2:172.21.15.154:3300/0 conn(0x7f9fe8158a00 msgr2=0x7f9fe8071f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:16.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.058+0000 7f9fdffff640 1 --2- 172.21.15.88:0/649848149 >> v2:172.21.15.154:3300/0 conn(0x7f9fe8158a00 0x7f9fe8071f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:16.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.058+0000 7f9fdffff640 1 -- 172.21.15.88:0/649848149 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe8073190 con 0x7f9fe8072470 2024-09-16T01:40:16.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.058+0000 7f9fe759e640 1 --2- 172.21.15.88:0/649848149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fe8158de0 0x7f9fe80b53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:16.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.059+0000 7f9fdffff640 1 --2- 172.21.15.88:0/649848149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fe8072470 0x7f9fe80b4e60 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8069f90 tx=0x7f9fd002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8e62c681b45cf18 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:16.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.059+0000 7f9fe659c640 1 -- 172.21.15.88:0/649848149 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd0037070 con 0x7f9fe8072470 2024-09-16T01:40:16.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.059+0000 7f9fe659c640 1 -- 172.21.15.88:0/649848149 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fd0032440 con 0x7f9fe8072470 2024-09-16T01:40:16.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.059+0000 7f9fe659c640 1 -- 172.21.15.88:0/649848149 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd0032810 con 0x7f9fe8072470 2024-09-16T01:40:16.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.060+0000 7f9fed0dd640 1 -- 172.21.15.88:0/649848149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fe8072470 msgr2=0x7f9fe80b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:16.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.060+0000 7f9fed0dd640 1 --2- 172.21.15.88:0/649848149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fe8072470 0x7f9fe80b4e60 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f9fe8069f90 tx=0x7f9fd002a1e0 comp rx=0 tx=0).stop 2024-09-16T01:40:16.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.060+0000 7f9fed0dd640 1 -- 172.21.15.88:0/649848149 shutdown_connections 2024-09-16T01:40:16.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.060+0000 7f9fed0dd640 1 --2- 172.21.15.88:0/649848149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fe8158de0 0x7f9fe80b53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:16.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.060+0000 7f9fed0dd640 1 --2- 172.21.15.88:0/649848149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fe8072470 0x7f9fe80b4e60 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:16.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.060+0000 7f9fed0dd640 1 --2- 172.21.15.88:0/649848149 >> v2:172.21.15.154:3300/0 conn(0x7f9fe8158a00 0x7f9fe8071f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:16.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.060+0000 7f9fed0dd640 1 -- 172.21.15.88:0/649848149 >> 172.21.15.88:0/649848149 conn(0x7f9fe806cb50 msgr2=0x7f9fe806cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:16.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.060+0000 7f9fed0dd640 1 -- 172.21.15.88:0/649848149 shutdown_connections 2024-09-16T01:40:16.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.060+0000 7f9fed0dd640 1 -- 172.21.15.88:0/649848149 wait complete. 2024-09-16T01:40:16.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.061+0000 7f9fed0dd640 1 Processor -- start 2024-09-16T01:40:16.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.061+0000 7f9fed0dd640 1 -- start start 2024-09-16T01:40:16.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.062+0000 7f9fed0dd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fe8072470 0x7f9fe80d0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:16.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.062+0000 7f9fed0dd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fe8158a00 0x7f9fe8157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:16.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.062+0000 7f9fed0dd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fe8158de0 0x7f9fe8157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:16.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.062+0000 7f9fed0dd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe80b5ca0 con 0x7f9fe8158de0 2024-09-16T01:40:16.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.062+0000 7f9fed0dd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe80b5b20 con 0x7f9fe8158a00 2024-09-16T01:40:16.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.062+0000 7f9fed0dd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9fe8162110 con 0x7f9fe8072470 2024-09-16T01:40:16.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.062+0000 7f9fe759e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fe8158de0 0x7f9fe8157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:16.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.062+0000 7f9fe759e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fe8158de0 0x7f9fe8157580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47028/0 (socket says 172.21.15.88:47028) 2024-09-16T01:40:16.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.062+0000 7f9fe759e640 1 -- 172.21.15.88:0/434957920 learned_addr learned my addr 172.21.15.88:0/434957920 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:16.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.062+0000 7f9fdffff640 1 --2- 172.21.15.88:0/434957920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fe8158a00 0x7f9fe8157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:16.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.062+0000 7f9fe6d9d640 1 --2- 172.21.15.88:0/434957920 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fe8072470 0x7f9fe80d0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:16.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.062+0000 7f9fe759e640 1 -- 172.21.15.88:0/434957920 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fe8072470 msgr2=0x7f9fe80d0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:16.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.062+0000 7f9fe759e640 1 --2- 172.21.15.88:0/434957920 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fe8072470 0x7f9fe80d0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:16.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.062+0000 7f9fe759e640 1 -- 172.21.15.88:0/434957920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fe8158a00 msgr2=0x7f9fe8157040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:16.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.062+0000 7f9fe759e640 1 --2- 172.21.15.88:0/434957920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fe8158a00 0x7f9fe8157040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:16.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.062+0000 7f9fe759e640 1 -- 172.21.15.88:0/434957920 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe8160850 con 0x7f9fe8158de0 2024-09-16T01:40:16.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.062+0000 7f9fe6d9d640 1 --2- 172.21.15.88:0/434957920 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fe8072470 0x7f9fe80d0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:16.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.062+0000 7f9fe759e640 1 --2- 172.21.15.88:0/434957920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fe8158de0 0x7f9fe8157580 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f9fe000cd80 tx=0x7f9fe000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:16.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.062+0000 7f9fdf7fe640 1 -- 172.21.15.88:0/434957920 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe000f680 con 0x7f9fe8158de0 2024-09-16T01:40:16.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.062+0000 7f9fdf7fe640 1 -- 172.21.15.88:0/434957920 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fe000f840 con 0x7f9fe8158de0 2024-09-16T01:40:16.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.062+0000 7f9fed0dd640 1 -- 172.21.15.88:0/434957920 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fe8160b30 con 0x7f9fe8158de0 2024-09-16T01:40:16.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.063+0000 7f9fdf7fe640 1 -- 172.21.15.88:0/434957920 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe000fc30 con 0x7f9fe8158de0 2024-09-16T01:40:16.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.063+0000 7f9fed0dd640 1 -- 172.21.15.88:0/434957920 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fe81610f0 con 0x7f9fe8158de0 2024-09-16T01:40:16.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.064+0000 7f9fed0dd640 1 -- 172.21.15.88:0/434957920 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fe8071f30 con 0x7f9fe8158de0 2024-09-16T01:40:16.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.064+0000 7f9fdf7fe640 1 -- 172.21.15.88:0/434957920 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9fe00213f0 con 0x7f9fe8158de0 2024-09-16T01:40:16.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.064+0000 7f9fdf7fe640 1 --2- 172.21.15.88:0/434957920 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9fc806c2b0 0x7f9fc806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:16.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.065+0000 7f9fdf7fe640 1 -- 172.21.15.88:0/434957920 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f9fe008c120 con 0x7f9fe8158de0 2024-09-16T01:40:16.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.065+0000 7f9fe6d9d640 1 --2- 172.21.15.88:0/434957920 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9fc806c2b0 0x7f9fc806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:16.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.065+0000 7f9fe6d9d640 1 --2- 172.21.15.88:0/434957920 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9fc806c2b0 0x7f9fc806e770 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8006fd0 tx=0x7f9fd8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:16.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.069+0000 7f9fdf7fe640 1 -- 172.21.15.88:0/434957920 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9fe0014030 con 0x7f9fe8158de0 2024-09-16T01:40:16.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.271+0000 7f9fed0dd640 1 -- 172.21.15.88:0/434957920 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f9fe80b53d0 con 0x7f9fe8158de0 2024-09-16T01:40:16.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.655+0000 7f9fdf7fe640 1 -- 172.21.15.88:0/434957920 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v73) ==== 161+0+0 (secure 0 0 0) 0x7f9fe005a8c0 con 0x7f9fe8158de0 2024-09-16T01:40:16.656 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:40:16.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.657+0000 7f9fed0dd640 1 -- 172.21.15.88:0/434957920 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9fc806c2b0 msgr2=0x7f9fc806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:16.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.657+0000 7f9fed0dd640 1 --2- 172.21.15.88:0/434957920 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9fc806c2b0 0x7f9fc806e770 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8006fd0 tx=0x7f9fd8008040 comp rx=0 tx=0).stop 2024-09-16T01:40:16.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.657+0000 7f9fed0dd640 1 -- 172.21.15.88:0/434957920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fe8158de0 msgr2=0x7f9fe8157580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:16.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.657+0000 7f9fed0dd640 1 --2- 172.21.15.88:0/434957920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fe8158de0 0x7f9fe8157580 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f9fe000cd80 tx=0x7f9fe000a650 comp rx=0 tx=0).stop 2024-09-16T01:40:16.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.657+0000 7f9fed0dd640 1 -- 172.21.15.88:0/434957920 shutdown_connections 2024-09-16T01:40:16.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.657+0000 7f9fed0dd640 1 --2- 172.21.15.88:0/434957920 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9fc806c2b0 0x7f9fc806e770 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:16.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.657+0000 7f9fed0dd640 1 --2- 172.21.15.88:0/434957920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9fe8158de0 0x7f9fe8157580 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:16.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.657+0000 7f9fed0dd640 1 --2- 172.21.15.88:0/434957920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fe8158a00 0x7f9fe8157040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:16.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.657+0000 7f9fed0dd640 1 --2- 172.21.15.88:0/434957920 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fe8072470 0x7f9fe80d0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:16.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.657+0000 7f9fed0dd640 1 -- 172.21.15.88:0/434957920 >> 172.21.15.88:0/434957920 conn(0x7f9fe806cb50 msgr2=0x7f9fe815e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:16.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.657+0000 7f9fed0dd640 1 -- 172.21.15.88:0/434957920 shutdown_connections 2024-09-16T01:40:16.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.657+0000 7f9fed0dd640 1 -- 172.21.15.88:0/434957920 wait complete. 2024-09-16T01:40:16.670 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:40:16.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.770+0000 7f6a6f36d640 1 Processor -- start 2024-09-16T01:40:16.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.770+0000 7f6a6f36d640 1 -- start start 2024-09-16T01:40:16.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.771+0000 7f6a6f36d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6a6815b480 0x7f6a680b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:16.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.771+0000 7f6a6f36d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6a680b60c0 0x7f6a68071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:16.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.771+0000 7f6a6f36d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a680ccff0 con 0x7f6a68072530 2024-09-16T01:40:16.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.771+0000 7f6a6f36d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6a68072980 con 0x7f6a6815b480 2024-09-16T01:40:16.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.771+0000 7f6a6f36d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6a680cd2d0 con 0x7f6a680b60c0 2024-09-16T01:40:16.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.771+0000 7f6a6d109640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6a6815b480 0x7f6a680b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:16.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.771+0000 7f6a6d109640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6a6815b480 0x7f6a680b5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47036/0 (socket says 172.21.15.88:47036) 2024-09-16T01:40:16.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.771+0000 7f6a6d109640 1 -- 172.21.15.88:0/1211976215 learned_addr learned my addr 172.21.15.88:0/1211976215 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:16.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.771+0000 7f6a6c908640 1 --2- 172.21.15.88:0/1211976215 >> v2:172.21.15.154:3300/0 conn(0x7f6a680b60c0 0x7f6a68071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:16.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.771+0000 7f6a6d109640 1 -- 172.21.15.88:0/1211976215 >> v2:172.21.15.154:3300/0 conn(0x7f6a680b60c0 msgr2=0x7f6a68071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:16.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.771+0000 7f6a6d109640 1 --2- 172.21.15.88:0/1211976215 >> v2:172.21.15.154:3300/0 conn(0x7f6a680b60c0 0x7f6a68071ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:16.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.771+0000 7f6a6d109640 1 -- 172.21.15.88:0/1211976215 >> v1:172.21.15.154:6789/0 conn(0x7f6a68072530 legacy=0x7f6a68070fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:16.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.771+0000 7f6a6d109640 1 -- 172.21.15.88:0/1211976215 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a680cd5b0 con 0x7f6a6815b480 2024-09-16T01:40:16.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.771+0000 7f6a6c908640 1 --2- 172.21.15.88:0/1211976215 >> v2:172.21.15.154:3300/0 conn(0x7f6a680b60c0 0x7f6a68071ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:16.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.772+0000 7f6a6d109640 1 --2- 172.21.15.88:0/1211976215 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a6815b480 0x7f6a680b5b80 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f6a680b2ee0 tx=0x7f6a6002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a798f3d9b71e7cc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:16.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.772+0000 7f6a5ffff640 1 -- 172.21.15.88:0/1211976215 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a60036070 con 0x7f6a6815b480 2024-09-16T01:40:16.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.772+0000 7f6a5ffff640 1 -- 172.21.15.88:0/1211976215 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a60031600 con 0x7f6a6815b480 2024-09-16T01:40:16.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.772+0000 7f6a5ffff640 1 -- 172.21.15.88:0/1211976215 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a600319d0 con 0x7f6a6815b480 2024-09-16T01:40:16.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.773+0000 7f6a6f36d640 1 -- 172.21.15.88:0/1211976215 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a6815b480 msgr2=0x7f6a680b5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:16.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.773+0000 7f6a6f36d640 1 --2- 172.21.15.88:0/1211976215 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a6815b480 0x7f6a680b5b80 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f6a680b2ee0 tx=0x7f6a6002a6f0 comp rx=0 tx=0).stop 2024-09-16T01:40:16.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.773+0000 7f6a6f36d640 1 -- 172.21.15.88:0/1211976215 shutdown_connections 2024-09-16T01:40:16.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.773+0000 7f6a6f36d640 1 --2- 172.21.15.88:0/1211976215 >> v2:172.21.15.154:3300/0 conn(0x7f6a680b60c0 0x7f6a68071ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:16.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.773+0000 7f6a6f36d640 1 --2- 172.21.15.88:0/1211976215 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a6815b480 0x7f6a680b5b80 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:16.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.773+0000 7f6a6f36d640 1 -- 172.21.15.88:0/1211976215 >> 172.21.15.88:0/1211976215 conn(0x7f6a6806d0f0 msgr2=0x7f6a6806d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:16.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.773+0000 7f6a6f36d640 1 -- 172.21.15.88:0/1211976215 shutdown_connections 2024-09-16T01:40:16.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.773+0000 7f6a6f36d640 1 -- 172.21.15.88:0/1211976215 wait complete. 2024-09-16T01:40:16.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.774+0000 7f6a6f36d640 1 Processor -- start 2024-09-16T01:40:16.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.774+0000 7f6a6f36d640 1 -- start start 2024-09-16T01:40:16.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.775+0000 7f6a6f36d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a68072530 0x7f6a68153300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:16.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.775+0000 7f6a6f36d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a680b60c0 0x7f6a681e5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:16.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.775+0000 7f6a6f36d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a6815b480 0x7f6a6815c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:16.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.775+0000 7f6a6f36d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6815b860 con 0x7f6a680b60c0 2024-09-16T01:40:16.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.775+0000 7f6a6f36d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6a680cdb70 con 0x7f6a6815b480 2024-09-16T01:40:16.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.775+0000 7f6a6f36d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6a6815b9e0 con 0x7f6a68072530 2024-09-16T01:40:16.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.775+0000 7f6a6c908640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a680b60c0 0x7f6a681e5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:16.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.775+0000 7f6a6c908640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a680b60c0 0x7f6a681e5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47046/0 (socket says 172.21.15.88:47046) 2024-09-16T01:40:16.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.775+0000 7f6a6c908640 1 -- 172.21.15.88:0/2178337911 learned_addr learned my addr 172.21.15.88:0/2178337911 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:16.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.775+0000 7f6a6d90a640 1 --2- 172.21.15.88:0/2178337911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a6815b480 0x7f6a6815c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:16.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.775+0000 7f6a6d109640 1 --2- 172.21.15.88:0/2178337911 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a68072530 0x7f6a68153300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:16.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.775+0000 7f6a6c908640 1 -- 172.21.15.88:0/2178337911 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a68072530 msgr2=0x7f6a68153300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:16.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.775+0000 7f6a6c908640 1 --2- 172.21.15.88:0/2178337911 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a68072530 0x7f6a68153300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:16.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.775+0000 7f6a6c908640 1 -- 172.21.15.88:0/2178337911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a6815b480 msgr2=0x7f6a6815c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:16.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.775+0000 7f6a6c908640 1 --2- 172.21.15.88:0/2178337911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a6815b480 0x7f6a6815c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:16.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.775+0000 7f6a6c908640 1 -- 172.21.15.88:0/2178337911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a68153840 con 0x7f6a680b60c0 2024-09-16T01:40:16.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.775+0000 7f6a6d109640 1 --2- 172.21.15.88:0/2178337911 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a68072530 0x7f6a68153300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:16.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.775+0000 7f6a6c908640 1 --2- 172.21.15.88:0/2178337911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a680b60c0 0x7f6a681e5b70 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800c7f0 tx=0x7f6a5800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:16.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.776+0000 7f6a5dffb640 1 -- 172.21.15.88:0/2178337911 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a58015070 con 0x7f6a680b60c0 2024-09-16T01:40:16.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.776+0000 7f6a5dffb640 1 -- 172.21.15.88:0/2178337911 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a58007a30 con 0x7f6a680b60c0 2024-09-16T01:40:16.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.776+0000 7f6a6f36d640 1 -- 172.21.15.88:0/2178337911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a681e64a0 con 0x7f6a680b60c0 2024-09-16T01:40:16.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.776+0000 7f6a5dffb640 1 -- 172.21.15.88:0/2178337911 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a58007e00 con 0x7f6a680b60c0 2024-09-16T01:40:16.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.776+0000 7f6a6f36d640 1 -- 172.21.15.88:0/2178337911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a681e6d40 con 0x7f6a680b60c0 2024-09-16T01:40:16.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.777+0000 7f6a6f36d640 1 -- 172.21.15.88:0/2178337911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a68071ff0 con 0x7f6a680b60c0 2024-09-16T01:40:16.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.777+0000 7f6a5dffb640 1 -- 172.21.15.88:0/2178337911 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6a58018070 con 0x7f6a680b60c0 2024-09-16T01:40:16.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.778+0000 7f6a5dffb640 1 --2- 172.21.15.88:0/2178337911 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6a4c06c2b0 0x7f6a4c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:16.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.778+0000 7f6a5dffb640 1 -- 172.21.15.88:0/2178337911 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f6a5808b450 con 0x7f6a680b60c0 2024-09-16T01:40:16.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.778+0000 7f6a6d109640 1 --2- 172.21.15.88:0/2178337911 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6a4c06c2b0 0x7f6a4c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:16.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.778+0000 7f6a6d109640 1 --2- 172.21.15.88:0/2178337911 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6a4c06c2b0 0x7f6a4c06e770 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6a60033000 tx=0x7f6a60009000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:16.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:16.782+0000 7f6a5dffb640 1 -- 172.21.15.88:0/2178337911 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6a58059b60 con 0x7f6a680b60c0 2024-09-16T01:40:17.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.012+0000 7f6a6f36d640 1 -- 172.21.15.88:0/2178337911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f6a680d0fb0 con 0x7f6a680b60c0 2024-09-16T01:40:17.334 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:40:17.337 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:40:17.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.661+0000 7f6a5dffb640 1 -- 172.21.15.88:0/2178337911 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v74) ==== 107+0+0 (secure 0 0 0) 0x7f6a58059460 con 0x7f6a680b60c0 2024-09-16T01:40:17.663 INFO:teuthology.orchestra.run.smithi088.stderr:nothing to do: rank is not damaged 2024-09-16T01:40:17.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.664+0000 7f6a6f36d640 1 -- 172.21.15.88:0/2178337911 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6a4c06c2b0 msgr2=0x7f6a4c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:17.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.664+0000 7f6a6f36d640 1 --2- 172.21.15.88:0/2178337911 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6a4c06c2b0 0x7f6a4c06e770 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6a60033000 tx=0x7f6a60009000 comp rx=0 tx=0).stop 2024-09-16T01:40:17.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.664+0000 7f6a6f36d640 1 -- 172.21.15.88:0/2178337911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a680b60c0 msgr2=0x7f6a681e5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:17.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.664+0000 7f6a6f36d640 1 --2- 172.21.15.88:0/2178337911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a680b60c0 0x7f6a681e5b70 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800c7f0 tx=0x7f6a5800cbf0 comp rx=0 tx=0).stop 2024-09-16T01:40:17.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.664+0000 7f6a6f36d640 1 -- 172.21.15.88:0/2178337911 shutdown_connections 2024-09-16T01:40:17.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.664+0000 7f6a6f36d640 1 --2- 172.21.15.88:0/2178337911 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6a4c06c2b0 0x7f6a4c06e770 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:17.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.664+0000 7f6a6f36d640 1 --2- 172.21.15.88:0/2178337911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a6815b480 0x7f6a6815c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:17.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.664+0000 7f6a6f36d640 1 --2- 172.21.15.88:0/2178337911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a680b60c0 0x7f6a681e5b70 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:17.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.664+0000 7f6a6f36d640 1 --2- 172.21.15.88:0/2178337911 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a68072530 0x7f6a68153300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:17.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.664+0000 7f6a6f36d640 1 -- 172.21.15.88:0/2178337911 >> 172.21.15.88:0/2178337911 conn(0x7f6a6806d0f0 msgr2=0x7f6a68074c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:17.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.664+0000 7f6a6f36d640 1 -- 172.21.15.88:0/2178337911 shutdown_connections 2024-09-16T01:40:17.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.664+0000 7f6a6f36d640 1 -- 172.21.15.88:0/2178337911 wait complete. 2024-09-16T01:40:17.677 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:40:17.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.718+0000 7f02abfe7f40 1 Processor -- start 2024-09-16T01:40:17.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.718+0000 7f02abfe7f40 1 -- start start 2024-09-16T01:40:17.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.719+0000 7f02abfe7f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d80009510 0x559d800801b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:17.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.719+0000 7f02abfe7f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d8007d1b0 0x559d80093910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:17.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.719+0000 7f02abfe7f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559d8009b540 0x559d8009d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:17.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.719+0000 7f02abfe7f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x559d8007d590 con 0x559d8007d1b0 2024-09-16T01:40:17.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.719+0000 7f02abfe7f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559d7ff14510 con 0x559d80009510 2024-09-16T01:40:17.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.719+0000 7f02abfe7f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x559d80008550 con 0x559d8009b540 2024-09-16T01:40:17.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.719+0000 7f02a92dc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d8007d1b0 0x559d80093910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:17.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.719+0000 7f02a92dc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d8007d1b0 0x559d80093910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47056/0 (socket says 172.21.15.88:47056) 2024-09-16T01:40:17.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.719+0000 7f02a92dc640 1 -- 172.21.15.88:0/4003505861 learned_addr learned my addr 172.21.15.88:0/4003505861 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:17.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.719+0000 7f02a9add640 1 --2- 172.21.15.88:0/4003505861 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d80009510 0x559d800801b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:17.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.719+0000 7f02aa2de640 1 --2- 172.21.15.88:0/4003505861 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559d8009b540 0x559d8009d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:17.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.719+0000 7f02a92dc640 1 -- 172.21.15.88:0/4003505861 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559d8009b540 msgr2=0x559d8009d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:17.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.719+0000 7f02a92dc640 1 --2- 172.21.15.88:0/4003505861 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559d8009b540 0x559d8009d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:17.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.719+0000 7f02a92dc640 1 -- 172.21.15.88:0/4003505861 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d80009510 msgr2=0x559d800801b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:17.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.719+0000 7f02a92dc640 1 --2- 172.21.15.88:0/4003505861 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d80009510 0x559d800801b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:17.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.719+0000 7f02a92dc640 1 -- 172.21.15.88:0/4003505861 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d8007c1b0 con 0x559d8007d1b0 2024-09-16T01:40:17.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.719+0000 7f02a9add640 1 --2- 172.21.15.88:0/4003505861 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d80009510 0x559d800801b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:17.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.719+0000 7f02a92dc640 1 --2- 172.21.15.88:0/4003505861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d8007d1b0 0x559d80093910 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f02a0055040 tx=0x7f02a0051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f6fd90c52668503 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:17.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.720+0000 7f029bfff640 1 -- 172.21.15.88:0/4003505861 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a0053800 con 0x559d8007d1b0 2024-09-16T01:40:17.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.720+0000 7f029bfff640 1 -- 172.21.15.88:0/4003505861 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02a00539a0 con 0x559d8007d1b0 2024-09-16T01:40:17.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.720+0000 7f029bfff640 1 -- 172.21.15.88:0/4003505861 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a0053d90 con 0x559d8007d1b0 2024-09-16T01:40:17.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.720+0000 7f02abfe7f40 1 -- 172.21.15.88:0/4003505861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d8007d1b0 msgr2=0x559d80093910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:17.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.720+0000 7f02abfe7f40 1 --2- 172.21.15.88:0/4003505861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d8007d1b0 0x559d80093910 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f02a0055040 tx=0x7f02a0051000 comp rx=0 tx=0).stop 2024-09-16T01:40:17.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.720+0000 7f02abfe7f40 1 -- 172.21.15.88:0/4003505861 shutdown_connections 2024-09-16T01:40:17.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.720+0000 7f02abfe7f40 1 --2- 172.21.15.88:0/4003505861 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559d8009b540 0x559d8009d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:17.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.720+0000 7f02abfe7f40 1 --2- 172.21.15.88:0/4003505861 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d8007d1b0 0x559d80093910 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:17.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.720+0000 7f02abfe7f40 1 --2- 172.21.15.88:0/4003505861 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d80009510 0x559d800801b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:17.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.720+0000 7f02abfe7f40 1 -- 172.21.15.88:0/4003505861 >> 172.21.15.88:0/4003505861 conn(0x559d7ffc1be0 msgr2=0x559d8007f780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:17.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.720+0000 7f02abfe7f40 1 -- 172.21.15.88:0/4003505861 shutdown_connections 2024-09-16T01:40:17.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.720+0000 7f02abfe7f40 1 -- 172.21.15.88:0/4003505861 wait complete. 2024-09-16T01:40:17.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.721+0000 7f02abfe7f40 1 Processor -- start 2024-09-16T01:40:17.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.721+0000 7f02abfe7f40 1 -- start start 2024-09-16T01:40:17.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.721+0000 7f02abfe7f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559d80009510 0x559d80094ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:17.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.721+0000 7f02abfe7f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d8007d1b0 0x559d80093b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:17.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.721+0000 7f02abfe7f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d8009b540 0x559d80095840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:17.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.721+0000 7f02abfe7f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x559d8009a920 con 0x559d8009b540 2024-09-16T01:40:17.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.721+0000 7f02abfe7f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559d800806f0 con 0x559d8007d1b0 2024-09-16T01:40:17.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.721+0000 7f02abfe7f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x559d8009ab70 con 0x559d80009510 2024-09-16T01:40:17.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.721+0000 7f02aa2de640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d8009b540 0x559d80095840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:17.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.722+0000 7f02aa2de640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d8009b540 0x559d80095840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47066/0 (socket says 172.21.15.88:47066) 2024-09-16T01:40:17.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.722+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 learned_addr learned my addr 172.21.15.88:0/863976071 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:17.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.722+0000 7f02a9add640 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559d80009510 0x559d80094ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:17.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.722+0000 7f02a92dc640 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d8007d1b0 0x559d80093b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:17.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.722+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559d80009510 msgr2=0x559d80094ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:17.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.722+0000 7f02aa2de640 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559d80009510 0x559d80094ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:17.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.722+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d8007d1b0 msgr2=0x559d80093b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:17.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.722+0000 7f02aa2de640 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d8007d1b0 0x559d80093b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:17.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.722+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d800931c0 con 0x559d8009b540 2024-09-16T01:40:17.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.722+0000 7f02aa2de640 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d8009b540 0x559d80095840 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f02a400f610 tx=0x7f02a400fad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:17.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.722+0000 7f029affd640 1 -- 172.21.15.88:0/863976071 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a4019020 con 0x559d8009b540 2024-09-16T01:40:17.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.722+0000 7f029affd640 1 -- 172.21.15.88:0/863976071 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02a4015690 con 0x559d8009b540 2024-09-16T01:40:17.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.722+0000 7f029affd640 1 -- 172.21.15.88:0/863976071 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a4015a60 con 0x559d8009b540 2024-09-16T01:40:17.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.722+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559d8000dba0 con 0x559d8009b540 2024-09-16T01:40:17.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.722+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d80095400 con 0x559d8009b540 2024-09-16T01:40:17.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.723+0000 7f029affd640 1 -- 172.21.15.88:0/863976071 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f02a401c020 con 0x559d8009b540 2024-09-16T01:40:17.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.724+0000 7f029affd640 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f028c06acc0 0x7f028c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:17.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.724+0000 7f029affd640 1 -- 172.21.15.88:0/863976071 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f02a408a6b0 con 0x559d8009b540 2024-09-16T01:40:17.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.724+0000 7f02a9add640 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f028c06acc0 0x7f028c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:17.728 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:40:17.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.724+0000 7f02abfe7f40 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x559d80097fc0 0x559d8007dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:17.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.724+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8007b410 con 0x559d80097fc0 2024-09-16T01:40:17.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.724+0000 7f02a92dc640 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x559d80097fc0 0x559d8007dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:17.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.724+0000 7f02a9add640 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f028c06acc0 0x7f028c06d180 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f029c000c00 tx=0x7f029c0018b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:17.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.725+0000 7f02a92dc640 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x559d80097fc0 0x559d8007dc80 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:17.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.726+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v86'11 uv4 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.726+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e86) -- 0x559d80094040 con 0x559d80097fc0 2024-09-16T01:40:17.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.728+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v86'12 uv12 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.728+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8007ae30 con 0x559d80097fc0 2024-09-16T01:40:17.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.729+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v86'13 uv13 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.730 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:40:17.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.729+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8007e1c0 con 0x559d80097fc0 2024-09-16T01:40:17.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.732+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v86'14 uv14 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.732+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8013e290 con 0x559d80097fc0 2024-09-16T01:40:17.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.734+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v86'15 uv15 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.734+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8013d280 con 0x559d80097fc0 2024-09-16T01:40:17.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.736+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v86'16 uv16 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.736+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8013d620 con 0x559d80097fc0 2024-09-16T01:40:17.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.737+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v86'17 uv6 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.737+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e86) -- 0x559d8007e560 con 0x559d80097fc0 2024-09-16T01:40:17.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.738+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v86'18 uv18 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.738+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8013d9c0 con 0x559d80097fc0 2024-09-16T01:40:17.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.739+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v86'19 uv19 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.741 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:40:17.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.739+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8013d9c0 con 0x559d80097fc0 2024-09-16T01:40:17.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.741+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v86'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.741+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8013d9c0 con 0x559d80097fc0 2024-09-16T01:40:17.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.741+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v86'21 uv21 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.741+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8013d9c0 con 0x559d80097fc0 2024-09-16T01:40:17.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.742+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v86'22 uv22 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.742+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8013d9c0 con 0x559d80097fc0 2024-09-16T01:40:17.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.743+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v86'23 uv10 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.744+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e86) -- 0x559d8013d9c0 con 0x559d80097fc0 2024-09-16T01:40:17.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.745+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v86'24 uv24 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.745+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8013d9c0 con 0x559d80097fc0 2024-09-16T01:40:17.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.746+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v86'25 uv25 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.747 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:40:17.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.746+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8013d9c0 con 0x559d80097fc0 2024-09-16T01:40:17.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.747+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v86'26 uv26 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.747+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8013d9c0 con 0x559d80097fc0 2024-09-16T01:40:17.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.748+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v86'27 uv27 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.748+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8013d9c0 con 0x559d80097fc0 2024-09-16T01:40:17.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.749+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v86'28 uv28 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.749+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8013e630 con 0x559d80097fc0 2024-09-16T01:40:17.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.751+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v86'29 uv29 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.751+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8013e630 con 0x559d80097fc0 2024-09-16T01:40:17.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.751+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v86'30 uv2 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.752+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e86) -- 0x559d8013e630 con 0x559d80097fc0 2024-09-16T01:40:17.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.754+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v86'31 uv31 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.754+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8013e630 con 0x559d80097fc0 2024-09-16T01:40:17.756 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:40:17.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.755+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v86'32 uv32 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.756+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8013f640 con 0x559d80097fc0 2024-09-16T01:40:17.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.757+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v86'33 uv33 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.757+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8013f640 con 0x559d80097fc0 2024-09-16T01:40:17.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.758+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v86'34 uv34 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.758+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8013e630 con 0x559d80097fc0 2024-09-16T01:40:17.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.760+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v86'35 uv35 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.760+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8013e630 con 0x559d80097fc0 2024-09-16T01:40:17.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.762+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v86'36 uv8 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.762+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e86) -- 0x559d8013e630 con 0x559d80097fc0 2024-09-16T01:40:17.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.764+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v86'37 uv37 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.764+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8013e630 con 0x559d80097fc0 2024-09-16T01:40:17.766 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:40:17.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.765+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v86'38 uv38 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.765+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8013f640 con 0x559d80097fc0 2024-09-16T01:40:17.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.767+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v86'39 uv39 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.767+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8013f640 con 0x559d80097fc0 2024-09-16T01:40:17.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.769+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v86'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.769+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8013e630 con 0x559d80097fc0 2024-09-16T01:40:17.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.771+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v86'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.771+0000 7f02abfe7f40 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x559d8013eb40 0x559d80140f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:17.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.771+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80141470 con 0x559d8013eb40 2024-09-16T01:40:17.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.771+0000 7f02a9add640 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x559d8013eb40 0x559d80140f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:17.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.772+0000 7f02a9add640 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x559d8013eb40 0x559d80140f30 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:17.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.773+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v86'21 uv16 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8013eb40 2024-09-16T01:40:17.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.773+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e86) -- 0x559d80141810 con 0x559d8013eb40 2024-09-16T01:40:17.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.774+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v86'22 uv22 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8013eb40 2024-09-16T01:40:17.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.774+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80141bb0 con 0x559d8013eb40 2024-09-16T01:40:17.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.775+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v86'23 uv23 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8013eb40 2024-09-16T01:40:17.776 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:40:17.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.775+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80142400 con 0x559d8013eb40 2024-09-16T01:40:17.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.776+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v86'24 uv24 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8013eb40 2024-09-16T01:40:17.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.776+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d801427a0 con 0x559d8013eb40 2024-09-16T01:40:17.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.777+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v86'25 uv25 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8013eb40 2024-09-16T01:40:17.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.777+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80142b40 con 0x559d8013eb40 2024-09-16T01:40:17.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.779+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v86'26 uv20 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8013eb40 2024-09-16T01:40:17.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.779+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e86) -- 0x559d80142ee0 con 0x559d8013eb40 2024-09-16T01:40:17.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.781+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v86'27 uv27 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8013eb40 2024-09-16T01:40:17.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.781+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80143280 con 0x559d8013eb40 2024-09-16T01:40:17.785 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:40:17.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.784+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v86'28 uv28 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8013eb40 2024-09-16T01:40:17.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.784+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80143280 con 0x559d8013eb40 2024-09-16T01:40:17.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.786+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v86'29 uv29 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8013eb40 2024-09-16T01:40:17.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.786+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80143280 con 0x559d8013eb40 2024-09-16T01:40:17.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.787+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v86'30 uv30 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8013eb40 2024-09-16T01:40:17.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.787+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80143280 con 0x559d8013eb40 2024-09-16T01:40:17.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.789+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v86'31 uv17 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8013eb40 2024-09-16T01:40:17.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.789+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e86) -- 0x559d80143280 con 0x559d8013eb40 2024-09-16T01:40:17.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.791+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v86'32 uv32 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8013eb40 2024-09-16T01:40:17.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.791+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80143280 con 0x559d8013eb40 2024-09-16T01:40:17.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.793+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v86'33 uv33 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8013eb40 2024-09-16T01:40:17.794 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:40:17.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.793+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80147700 con 0x559d8013eb40 2024-09-16T01:40:17.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.795+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v86'34 uv34 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8013eb40 2024-09-16T01:40:17.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.795+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80147700 con 0x559d8013eb40 2024-09-16T01:40:17.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.796+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v86'35 uv35 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8013eb40 2024-09-16T01:40:17.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.796+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80143280 con 0x559d8013eb40 2024-09-16T01:40:17.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.798+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v86'36 uv2 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8013eb40 2024-09-16T01:40:17.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.798+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e86) -- 0x559d80143280 con 0x559d8013eb40 2024-09-16T01:40:17.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.800+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v86'37 uv37 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8013eb40 2024-09-16T01:40:17.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.800+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80143280 con 0x559d8013eb40 2024-09-16T01:40:17.803 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:40:17.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.802+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v86'38 uv38 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8013eb40 2024-09-16T01:40:17.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.802+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80144290 con 0x559d8013eb40 2024-09-16T01:40:17.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.804+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v86'39 uv39 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8013eb40 2024-09-16T01:40:17.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.804+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80144290 con 0x559d8013eb40 2024-09-16T01:40:17.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.807+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v86'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8013eb40 2024-09-16T01:40:17.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.807+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80143280 con 0x559d8013eb40 2024-09-16T01:40:17.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.809+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v86'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8013eb40 2024-09-16T01:40:17.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.809+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80143280 con 0x559d80097fc0 2024-09-16T01:40:17.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.810+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v86'3 uv2 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.810+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e86) -- 0x559d80143280 con 0x559d80097fc0 2024-09-16T01:40:17.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.811+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v86'4 uv4 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.811+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80143280 con 0x559d80097fc0 2024-09-16T01:40:17.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.811+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v86'5 uv5 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.813 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:40:17.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.811+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80144290 con 0x559d80097fc0 2024-09-16T01:40:17.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.812+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v86'6 uv6 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.812+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80144290 con 0x559d80097fc0 2024-09-16T01:40:17.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.813+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v86'7 uv7 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.813+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80143280 con 0x559d80097fc0 2024-09-16T01:40:17.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.814+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v86'8 uv8 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.814+0000 7f02abfe7f40 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x559d80143900 0x559d80145cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:17.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.814+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80146300 con 0x559d80143900 2024-09-16T01:40:17.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.814+0000 7f02a92dc640 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x559d80143900 0x559d80145cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:17.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.815+0000 7f02a92dc640 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x559d80143900 0x559d80145cf0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:17.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.815+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v86'4 uv3 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80143900 2024-09-16T01:40:17.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.815+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e86) -- 0x559d801466f0 con 0x559d80143900 2024-09-16T01:40:17.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.817+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v86'5 uv5 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80143900 2024-09-16T01:40:17.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.817+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d801466f0 con 0x559d80143900 2024-09-16T01:40:17.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.817+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v86'6 uv6 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80143900 2024-09-16T01:40:17.819 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:40:17.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.817+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d801466f0 con 0x559d80143900 2024-09-16T01:40:17.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.818+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v86'7 uv7 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80143900 2024-09-16T01:40:17.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.818+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d801466a0 con 0x559d80097fc0 2024-09-16T01:40:17.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.820+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v86'3 uv2 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.820+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e86) -- 0x559d801466a0 con 0x559d80097fc0 2024-09-16T01:40:17.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.821+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v86'4 uv4 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.821+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d801465a0 con 0x559d80097fc0 2024-09-16T01:40:17.823 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:40:17.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.822+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v86'5 uv5 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.822+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d801479a0 con 0x559d80097fc0 2024-09-16T01:40:17.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.823+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v86'6 uv6 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.823+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d801479a0 con 0x559d80097fc0 2024-09-16T01:40:17.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.824+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v86'7 uv7 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.824+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80146670 con 0x559d80097fc0 2024-09-16T01:40:17.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.825+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v86'8 uv8 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.825+0000 7f02abfe7f40 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x559d80146ca0 0x559d80149090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:17.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.825+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80149740 con 0x559d80146ca0 2024-09-16T01:40:17.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.825+0000 7f02aa2de640 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x559d80146ca0 0x559d80149090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:17.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.825+0000 7f02aa2de640 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x559d80146ca0 0x559d80149090 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:17.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.826+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v86'4 uv2 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.826+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e86) -- 0x559d80149c20 con 0x559d80146ca0 2024-09-16T01:40:17.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.827+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v86'5 uv5 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.827+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8014a100 con 0x559d80146ca0 2024-09-16T01:40:17.830 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:40:17.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.829+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v86'6 uv6 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.829+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8014b9b0 con 0x559d80146ca0 2024-09-16T01:40:17.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.830+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v86'7 uv7 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.830+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8014bd50 con 0x559d80146ca0 2024-09-16T01:40:17.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.831+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v86'8 uv8 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.831+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8014a9a0 con 0x559d80146ca0 2024-09-16T01:40:17.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.833+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v86'9 uv9 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.833+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8014ad90 con 0x559d80146ca0 2024-09-16T01:40:17.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.834+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v86'10 uv3 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.834+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e86) -- 0x559d8014b180 con 0x559d80146ca0 2024-09-16T01:40:17.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.836+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v86'11 uv11 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.836+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8014b180 con 0x559d80146ca0 2024-09-16T01:40:17.839 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:40:17.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.838+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v86'12 uv12 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.838+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8014b130 con 0x559d80146ca0 2024-09-16T01:40:17.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.839+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v86'13 uv13 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.839+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8014b130 con 0x559d80146ca0 2024-09-16T01:40:17.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.841+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v86'14 uv14 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.841+0000 7f02abfe7f40 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x559d8014b520 0x559d8014e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:17.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.841+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8014eb50 con 0x559d8014b520 2024-09-16T01:40:17.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.841+0000 7f02a9add640 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x559d8014b520 0x559d8014e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:17.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.842+0000 7f02a9add640 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x559d8014b520 0x559d8014e610 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:17.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.843+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v86'2 uv1 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8014b520 2024-09-16T01:40:17.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.843+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e86) -- 0x559d8014eb50 con 0x559d8014b520 2024-09-16T01:40:17.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.844+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v86'3 uv3 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8014b520 2024-09-16T01:40:17.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.844+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8014eb50 con 0x559d8014b520 2024-09-16T01:40:17.848 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:40:17.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.847+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v86'4 uv4 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8014b520 2024-09-16T01:40:17.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.847+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8014a5d0 con 0x559d8014b520 2024-09-16T01:40:17.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.848+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v86'5 uv5 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8014b520 2024-09-16T01:40:17.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.848+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8014a590 con 0x559d8014b520 2024-09-16T01:40:17.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.850+0000 7f02a9add640 1 -- 172.21.15.88:0/863976071 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v86'6 uv6 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f029c00e980 con 0x559d8014b520 2024-09-16T01:40:17.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.850+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8014a590 con 0x559d80097fc0 2024-09-16T01:40:17.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.851+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v86'3 uv2 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.851+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e86) -- 0x559d8014a590 con 0x559d80097fc0 2024-09-16T01:40:17.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.852+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v86'4 uv4 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.852+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8014a590 con 0x559d80097fc0 2024-09-16T01:40:17.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.855+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v86'5 uv5 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.856 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:40:17.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.855+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8014a5f0 con 0x559d80097fc0 2024-09-16T01:40:17.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.857+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v86'6 uv6 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.857+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d801501a0 con 0x559d80097fc0 2024-09-16T01:40:17.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.858+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v86'7 uv7 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.858+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8014f0d0 con 0x559d80097fc0 2024-09-16T01:40:17.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.859+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v86'8 uv8 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80097fc0 2024-09-16T01:40:17.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.859+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8014f0d0 con 0x559d80143900 2024-09-16T01:40:17.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.860+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v86'4 uv3 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80143900 2024-09-16T01:40:17.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.860+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e86) -- 0x559d8014f0d0 con 0x559d80143900 2024-09-16T01:40:17.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.861+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v86'5 uv5 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80143900 2024-09-16T01:40:17.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.861+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8014f0d0 con 0x559d80143900 2024-09-16T01:40:17.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.862+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v86'6 uv6 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80143900 2024-09-16T01:40:17.863 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:40:17.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.862+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8014f1b0 con 0x559d80143900 2024-09-16T01:40:17.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.863+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v86'7 uv7 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80143900 2024-09-16T01:40:17.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.863+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8014f1b0 con 0x559d80143900 2024-09-16T01:40:17.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.864+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v86'8 uv8 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80143900 2024-09-16T01:40:17.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.864+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8014f190 con 0x559d80143900 2024-09-16T01:40:17.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.865+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v86'9 uv1 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80143900 2024-09-16T01:40:17.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.865+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e86) -- 0x559d8014f190 con 0x559d80143900 2024-09-16T01:40:17.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.867+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v86'10 uv10 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80143900 2024-09-16T01:40:17.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.867+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8014f190 con 0x559d80143900 2024-09-16T01:40:17.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.867+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v86'11 uv11 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80143900 2024-09-16T01:40:17.869 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:40:17.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.868+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8014f580 con 0x559d80143900 2024-09-16T01:40:17.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.868+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v86'12 uv12 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80143900 2024-09-16T01:40:17.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.868+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d8014f970 con 0x559d80143900 2024-09-16T01:40:17.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.870+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v86'13 uv13 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80143900 2024-09-16T01:40:17.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.870+0000 7f02abfe7f40 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x559d80150000 0x559d801523f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:17.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.870+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80152930 con 0x559d80150000 2024-09-16T01:40:17.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.870+0000 7f02a92dc640 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x559d80150000 0x559d801523f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:17.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.870+0000 7f02a92dc640 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x559d80150000 0x559d801523f0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:17.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.871+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v86'4 uv3 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80150000 2024-09-16T01:40:17.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.871+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e86) -- 0x559d80152930 con 0x559d80150000 2024-09-16T01:40:17.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.872+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v86'5 uv5 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80150000 2024-09-16T01:40:17.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.872+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80152930 con 0x559d80150000 2024-09-16T01:40:17.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.875+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v86'6 uv6 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80150000 2024-09-16T01:40:17.876 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:40:17.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.875+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80152930 con 0x559d80150000 2024-09-16T01:40:17.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.877+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v86'7 uv7 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80150000 2024-09-16T01:40:17.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.877+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80152930 con 0x559d80143900 2024-09-16T01:40:17.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.878+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v86'3 uv2 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80143900 2024-09-16T01:40:17.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.878+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e86) -- 0x559d80152930 con 0x559d80143900 2024-09-16T01:40:17.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.882+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v86'4 uv4 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80143900 2024-09-16T01:40:17.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.882+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80152930 con 0x559d80143900 2024-09-16T01:40:17.884 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:40:17.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.883+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v86'5 uv5 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80143900 2024-09-16T01:40:17.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.883+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80153940 con 0x559d80143900 2024-09-16T01:40:17.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.885+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v86'6 uv6 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80143900 2024-09-16T01:40:17.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.885+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80153940 con 0x559d80143900 2024-09-16T01:40:17.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.886+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v86'7 uv7 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80143900 2024-09-16T01:40:17.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.886+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80152930 con 0x559d80143900 2024-09-16T01:40:17.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.890+0000 7f02a92dc640 1 -- 172.21.15.88:0/863976071 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v86'8 uv8 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a0053b50 con 0x559d80143900 2024-09-16T01:40:17.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.890+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80152930 con 0x559d80146ca0 2024-09-16T01:40:17.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.891+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v86'2 uv1 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.891+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e86) -- 0x559d80152930 con 0x559d80146ca0 2024-09-16T01:40:17.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.892+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v86'3 uv3 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.892+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80152930 con 0x559d80146ca0 2024-09-16T01:40:17.895 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:40:17.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.894+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v86'4 uv4 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.894+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80152930 con 0x559d80146ca0 2024-09-16T01:40:17.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.896+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v86'5 uv5 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.896+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80152930 con 0x559d80146ca0 2024-09-16T01:40:17.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.898+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v86'6 uv6 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.898+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80152930 con 0x559d80146ca0 2024-09-16T01:40:17.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.900+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v86'7 uv3 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.900+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e86) -- 0x559d80152930 con 0x559d80146ca0 2024-09-16T01:40:17.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.901+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v86'8 uv8 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.901+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80152930 con 0x559d80146ca0 2024-09-16T01:40:17.904 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:40:17.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.903+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v86'9 uv9 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.903+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80152930 con 0x559d80146ca0 2024-09-16T01:40:17.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.905+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v86'10 uv10 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.905+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80152930 con 0x559d80146ca0 2024-09-16T01:40:17.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.907+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v86'11 uv11 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.907+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80152930 con 0x559d80146ca0 2024-09-16T01:40:17.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.909+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v86'12 uv6 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.909+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e86) -- 0x559d80152930 con 0x559d80146ca0 2024-09-16T01:40:17.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.910+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v86'13 uv13 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.910+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80152930 con 0x559d80146ca0 2024-09-16T01:40:17.912 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:40:17.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.911+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v86'14 uv14 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.911+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80153940 con 0x559d80146ca0 2024-09-16T01:40:17.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.914+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v86'15 uv15 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.914+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80153940 con 0x559d80146ca0 2024-09-16T01:40:17.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.917+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v86'16 uv16 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.917+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80152930 con 0x559d80146ca0 2024-09-16T01:40:17.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.919+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v86'17 uv17 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.919+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x559d80152de0 con 0x559d80146ca0 2024-09-16T01:40:17.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v86'18 uv18 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f02a4054e70 con 0x559d80146ca0 2024-09-16T01:40:17.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x559d80097fc0 msgr2=0x559d8007dc80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:17.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02abfe7f40 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x559d80097fc0 0x559d8007dc80 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:17.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x559d80143900 msgr2=0x559d80145cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:17.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02abfe7f40 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x559d80143900 0x559d80145cf0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:17.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x559d80150000 msgr2=0x559d801523f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:17.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02abfe7f40 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x559d80150000 0x559d801523f0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:17.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x559d8013eb40 msgr2=0x559d80140f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:17.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02abfe7f40 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x559d8013eb40 0x559d80140f30 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:17.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 reap_dead start 2024-09-16T01:40:17.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x559d80146ca0 msgr2=0x559d80149090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:17.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02abfe7f40 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x559d80146ca0 0x559d80149090 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:17.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02aa2de640 1 -- 172.21.15.88:0/863976071 reap_dead start 2024-09-16T01:40:17.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x559d8014b520 msgr2=0x559d8014e610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:17.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02abfe7f40 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x559d8014b520 0x559d8014e610 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:17.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f028c06acc0 msgr2=0x7f028c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:17.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02abfe7f40 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f028c06acc0 0x7f028c06d180 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f029c000c00 tx=0x7f029c0018b0 comp rx=0 tx=0).stop 2024-09-16T01:40:17.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d8009b540 msgr2=0x559d80095840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:17.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02abfe7f40 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d8009b540 0x559d80095840 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f02a400f610 tx=0x7f02a400fad0 comp rx=0 tx=0).stop 2024-09-16T01:40:17.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 shutdown_connections 2024-09-16T01:40:17.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02abfe7f40 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f028c06acc0 0x7f028c06d180 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:17.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02abfe7f40 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d8009b540 0x559d80095840 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:17.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02abfe7f40 1 --2- 172.21.15.88:0/863976071 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x559d8014b520 0x559d8014e610 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:17.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 >> 172.21.15.88:0/863976071 conn(0x559d7ffc1be0 msgr2=0x559d8009d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:17.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 shutdown_connections 2024-09-16T01:40:17.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:17.922+0000 7f02abfe7f40 1 -- 172.21.15.88:0/863976071 wait complete. 2024-09-16T01:40:17.927 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 10000000000.00000000 2024-09-16T01:40:18.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.008+0000 7ff751b1bf40 1 Processor -- start 2024-09-16T01:40:18.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.008+0000 7ff751b1bf40 1 -- start start 2024-09-16T01:40:18.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.008+0000 7ff751b1bf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f6080b8d0 0x563f60825c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:18.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.008+0000 7ff751b1bf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563f608b0d20 0x563f60826190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:18.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.008+0000 7ff751b1bf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563f608b1100 0x563f608a3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:18.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.008+0000 7ff751b1bf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x563f6072e510 con 0x563f6080b8d0 2024-09-16T01:40:18.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.008+0000 7ff751b1bf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563f60824bd0 con 0x563f608b0d20 2024-09-16T01:40:18.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.008+0000 7ff751b1bf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x563f60808880 con 0x563f608b1100 2024-09-16T01:40:18.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.009+0000 7ff74ece9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f6080b8d0 0x563f60825c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:18.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.009+0000 7ff74ece9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f6080b8d0 0x563f60825c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47092/0 (socket says 172.21.15.88:47092) 2024-09-16T01:40:18.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.009+0000 7ff74ece9640 1 -- 172.21.15.88:0/3077106551 learned_addr learned my addr 172.21.15.88:0/3077106551 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:18.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.009+0000 7ff7464e8640 1 --2- 172.21.15.88:0/3077106551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563f608b0d20 0x563f60826190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:18.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.009+0000 7ff74f4ea640 1 --2- 172.21.15.88:0/3077106551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563f608b1100 0x563f608a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:18.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.009+0000 7ff74ece9640 1 -- 172.21.15.88:0/3077106551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563f608b1100 msgr2=0x563f608a3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:18.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.009+0000 7ff74ece9640 1 --2- 172.21.15.88:0/3077106551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563f608b1100 0x563f608a3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:18.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.009+0000 7ff74ece9640 1 -- 172.21.15.88:0/3077106551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563f608b0d20 msgr2=0x563f60826190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:18.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.009+0000 7ff74ece9640 1 --2- 172.21.15.88:0/3077106551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563f608b0d20 0x563f60826190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:18.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.009+0000 7ff74ece9640 1 -- 172.21.15.88:0/3077106551 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f608b1d10 con 0x563f6080b8d0 2024-09-16T01:40:18.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.009+0000 7ff74ece9640 1 --2- 172.21.15.88:0/3077106551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f6080b8d0 0x563f60825c50 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7ff740025b40 tx=0x7ff740051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22bbdef82d6b99c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:18.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.009+0000 7ff74dce7640 1 -- 172.21.15.88:0/3077106551 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74005a070 con 0x563f6080b8d0 2024-09-16T01:40:18.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.009+0000 7ff74dce7640 1 -- 172.21.15.88:0/3077106551 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff740064070 con 0x563f6080b8d0 2024-09-16T01:40:18.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.009+0000 7ff74dce7640 1 -- 172.21.15.88:0/3077106551 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff740055d00 con 0x563f6080b8d0 2024-09-16T01:40:18.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.010+0000 7ff751b1bf40 1 -- 172.21.15.88:0/3077106551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f6080b8d0 msgr2=0x563f60825c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:18.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.010+0000 7ff751b1bf40 1 --2- 172.21.15.88:0/3077106551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f6080b8d0 0x563f60825c50 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7ff740025b40 tx=0x7ff740051000 comp rx=0 tx=0).stop 2024-09-16T01:40:18.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.010+0000 7ff751b1bf40 1 -- 172.21.15.88:0/3077106551 shutdown_connections 2024-09-16T01:40:18.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.010+0000 7ff751b1bf40 1 --2- 172.21.15.88:0/3077106551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563f608b1100 0x563f608a3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:18.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.010+0000 7ff751b1bf40 1 --2- 172.21.15.88:0/3077106551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563f608b0d20 0x563f60826190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:18.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.010+0000 7ff751b1bf40 1 --2- 172.21.15.88:0/3077106551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f6080b8d0 0x563f60825c50 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:18.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.010+0000 7ff751b1bf40 1 -- 172.21.15.88:0/3077106551 >> 172.21.15.88:0/3077106551 conn(0x563f60827750 msgr2=0x563f608b40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:18.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.010+0000 7ff751b1bf40 1 -- 172.21.15.88:0/3077106551 shutdown_connections 2024-09-16T01:40:18.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.010+0000 7ff751b1bf40 1 -- 172.21.15.88:0/3077106551 wait complete. 2024-09-16T01:40:18.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.010+0000 7ff751b1bf40 1 Processor -- start 2024-09-16T01:40:18.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.010+0000 7ff751b1bf40 1 -- start start 2024-09-16T01:40:18.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.011+0000 7ff751b1bf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f6080b8d0 0x563f60884a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:18.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.011+0000 7ff751b1bf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563f608b0d20 0x563f6089e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:18.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.011+0000 7ff751b1bf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563f608b1100 0x563f6094b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:18.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.011+0000 7ff751b1bf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x563f608b1ac0 con 0x563f6080b8d0 2024-09-16T01:40:18.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.011+0000 7ff751b1bf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563f608b1ff0 con 0x563f608b0d20 2024-09-16T01:40:18.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.011+0000 7ff751b1bf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x563f60898f80 con 0x563f608b1100 2024-09-16T01:40:18.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.011+0000 7ff74ece9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f6080b8d0 0x563f60884a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:18.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.011+0000 7ff74ece9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f6080b8d0 0x563f60884a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47102/0 (socket says 172.21.15.88:47102) 2024-09-16T01:40:18.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.011+0000 7ff74ece9640 1 -- 172.21.15.88:0/1304736306 learned_addr learned my addr 172.21.15.88:0/1304736306 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:18.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.011+0000 7ff7464e8640 1 --2- 172.21.15.88:0/1304736306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563f608b0d20 0x563f6089e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:18.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.011+0000 7ff74f4ea640 1 --2- 172.21.15.88:0/1304736306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563f608b1100 0x563f6094b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:18.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.011+0000 7ff74ece9640 1 -- 172.21.15.88:0/1304736306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563f608b1100 msgr2=0x563f6094b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:18.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.011+0000 7ff74ece9640 1 --2- 172.21.15.88:0/1304736306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563f608b1100 0x563f6094b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:18.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.011+0000 7ff74ece9640 1 -- 172.21.15.88:0/1304736306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563f608b0d20 msgr2=0x563f6089e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:18.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.011+0000 7ff74ece9640 1 --2- 172.21.15.88:0/1304736306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563f608b0d20 0x563f6089e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:18.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.011+0000 7ff74ece9640 1 -- 172.21.15.88:0/1304736306 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f6089e940 con 0x563f6080b8d0 2024-09-16T01:40:18.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.012+0000 7ff74ece9640 1 --2- 172.21.15.88:0/1304736306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f6080b8d0 0x563f60884a70 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7ff740045980 tx=0x7ff740002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:18.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.012+0000 7ff74cce5640 1 -- 172.21.15.88:0/1304736306 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74005a040 con 0x563f6080b8d0 2024-09-16T01:40:18.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.012+0000 7ff74cce5640 1 -- 172.21.15.88:0/1304736306 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff74005b040 con 0x563f6080b8d0 2024-09-16T01:40:18.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.012+0000 7ff751b1bf40 1 -- 172.21.15.88:0/1304736306 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563f608a13d0 con 0x563f6080b8d0 2024-09-16T01:40:18.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.012+0000 7ff74cce5640 1 -- 172.21.15.88:0/1304736306 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff740055cc0 con 0x563f6080b8d0 2024-09-16T01:40:18.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.012+0000 7ff751b1bf40 1 -- 172.21.15.88:0/1304736306 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f608a1990 con 0x563f6080b8d0 2024-09-16T01:40:18.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.013+0000 7ff74cce5640 1 -- 172.21.15.88:0/1304736306 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff740071020 con 0x563f6080b8d0 2024-09-16T01:40:18.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.013+0000 7ff74cce5640 1 --2- 172.21.15.88:0/1304736306 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff73006ac90 0x7ff73006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:18.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.013+0000 7ff74cce5640 1 -- 172.21.15.88:0/1304736306 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7ff7400cf210 con 0x563f6080b8d0 2024-09-16T01:40:18.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.014+0000 7ff7464e8640 1 --2- 172.21.15.88:0/1304736306 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff73006ac90 0x7ff73006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:18.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.014+0000 7ff751b1bf40 1 --2- 172.21.15.88:0/1304736306 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x563f6094ba80 0x563f608a2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:18.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.014+0000 7ff751b1bf40 1 -- 172.21.15.88:0/1304736306 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e86) -- 0x563f608a0ed0 con 0x563f6094ba80 2024-09-16T01:40:18.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.014+0000 7ff74f4ea640 1 --2- 172.21.15.88:0/1304736306 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x563f6094ba80 0x563f608a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:18.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.014+0000 7ff7464e8640 1 --2- 172.21.15.88:0/1304736306 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff73006ac90 0x7ff73006d150 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x563f608b06b0 tx=0x7ff73800e040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:18.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.014+0000 7ff74f4ea640 1 --2- 172.21.15.88:0/1304736306 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x563f6094ba80 0x563f608a2980 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:18.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.015+0000 7ff74f4ea640 1 -- 172.21.15.88:0/1304736306 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 10000000000.00000000 [delete] v86'42 uv42 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff748001630 con 0x563f6094ba80 2024-09-16T01:40:18.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.015+0000 7ff751b1bf40 1 -- 172.21.15.88:0/1304736306 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x563f6094ba80 msgr2=0x563f608a2980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:18.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.015+0000 7ff751b1bf40 1 --2- 172.21.15.88:0/1304736306 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x563f6094ba80 0x563f608a2980 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:18.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.016+0000 7ff751b1bf40 1 -- 172.21.15.88:0/1304736306 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff73006ac90 msgr2=0x7ff73006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:18.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.016+0000 7ff751b1bf40 1 --2- 172.21.15.88:0/1304736306 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff73006ac90 0x7ff73006d150 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x563f608b06b0 tx=0x7ff73800e040 comp rx=0 tx=0).stop 2024-09-16T01:40:18.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.016+0000 7ff751b1bf40 1 -- 172.21.15.88:0/1304736306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f6080b8d0 msgr2=0x563f60884a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:18.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.016+0000 7ff751b1bf40 1 --2- 172.21.15.88:0/1304736306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563f6080b8d0 0x563f60884a70 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7ff740045980 tx=0x7ff740002fb0 comp rx=0 tx=0).stop 2024-09-16T01:40:18.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.016+0000 7ff74f4ea640 1 -- 172.21.15.88:0/1304736306 reap_dead start 2024-09-16T01:40:18.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.016+0000 7ff751b1bf40 1 -- 172.21.15.88:0/1304736306 shutdown_connections 2024-09-16T01:40:18.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.016+0000 7ff751b1bf40 1 -- 172.21.15.88:0/1304736306 >> 172.21.15.88:0/1304736306 conn(0x563f60827750 msgr2=0x563f60807f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:18.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.016+0000 7ff751b1bf40 1 -- 172.21.15.88:0/1304736306 shutdown_connections 2024-09-16T01:40:18.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.016+0000 7ff751b1bf40 1 -- 172.21.15.88:0/1304736306 wait complete. 2024-09-16T01:40:18.023 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:40:18.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.161+0000 7f3a8c55b640 1 Processor -- start 2024-09-16T01:40:18.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.161+0000 7f3a8c55b640 1 -- start start 2024-09-16T01:40:18.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.162+0000 7f3a8c55b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a84158a00 0x7f3a84071f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:18.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.162+0000 7f3a8c55b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3a84158de0 0x7f3a8406eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:18.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.162+0000 7f3a8c55b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a840728f0 con 0x7f3a84072470 2024-09-16T01:40:18.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.162+0000 7f3a8c55b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f3a84072bd0 con 0x7f3a84158de0 2024-09-16T01:40:18.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.163+0000 7f3a8c55b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3a84072eb0 con 0x7f3a84158a00 2024-09-16T01:40:18.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.163+0000 7f3a8aaf8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3a84158de0 0x7f3a8406eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:18.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.163+0000 7f3a8aaf8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3a84158de0 0x7f3a8406eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47114/0 (socket says 172.21.15.88:47114) 2024-09-16T01:40:18.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.163+0000 7f3a8aaf8640 1 -- 172.21.15.88:0/3498552530 learned_addr learned my addr 172.21.15.88:0/3498552530 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:18.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.163+0000 7f3a8a2f7640 1 --2- 172.21.15.88:0/3498552530 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a84158a00 0x7f3a84071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:18.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.163+0000 7f3a8aaf8640 1 -- 172.21.15.88:0/3498552530 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a84158a00 msgr2=0x7f3a84071f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:18.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.163+0000 7f3a8aaf8640 1 --2- 172.21.15.88:0/3498552530 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a84158a00 0x7f3a84071f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:18.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.163+0000 7f3a8aaf8640 1 -- 172.21.15.88:0/3498552530 >> v1:172.21.15.154:6789/0 conn(0x7f3a84072470 legacy=0x7f3a840b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:18.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.163+0000 7f3a8aaf8640 1 -- 172.21.15.88:0/3498552530 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a84073190 con 0x7f3a84158de0 2024-09-16T01:40:18.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.164+0000 7f3a8aaf8640 1 --2- 172.21.15.88:0/3498552530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a84158de0 0x7f3a8406eae0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c009e50 tx=0x7f3a7c02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83fec8ff0882e9a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:18.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.164+0000 7f3a892f5640 1 -- 172.21.15.88:0/3498552530 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c036070 con 0x7f3a84158de0 2024-09-16T01:40:18.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.164+0000 7f3a892f5640 1 -- 172.21.15.88:0/3498552530 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a7c031600 con 0x7f3a84158de0 2024-09-16T01:40:18.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.164+0000 7f3a892f5640 1 -- 172.21.15.88:0/3498552530 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c0319d0 con 0x7f3a84158de0 2024-09-16T01:40:18.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.164+0000 7f3a8c55b640 1 -- 172.21.15.88:0/3498552530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a84158de0 msgr2=0x7f3a8406eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:18.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.164+0000 7f3a8c55b640 1 --2- 172.21.15.88:0/3498552530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a84158de0 0x7f3a8406eae0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c009e50 tx=0x7f3a7c02a710 comp rx=0 tx=0).stop 2024-09-16T01:40:18.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.164+0000 7f3a8c55b640 1 -- 172.21.15.88:0/3498552530 shutdown_connections 2024-09-16T01:40:18.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.164+0000 7f3a8c55b640 1 --2- 172.21.15.88:0/3498552530 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a84158de0 0x7f3a8406eae0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:18.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.164+0000 7f3a8c55b640 1 --2- 172.21.15.88:0/3498552530 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a84158a00 0x7f3a84071f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:18.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.164+0000 7f3a8c55b640 1 -- 172.21.15.88:0/3498552530 >> 172.21.15.88:0/3498552530 conn(0x7f3a8406cb50 msgr2=0x7f3a8406cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:18.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.164+0000 7f3a8c55b640 1 -- 172.21.15.88:0/3498552530 shutdown_connections 2024-09-16T01:40:18.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.164+0000 7f3a8c55b640 1 -- 172.21.15.88:0/3498552530 wait complete. 2024-09-16T01:40:18.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.166+0000 7f3a8c55b640 1 Processor -- start 2024-09-16T01:40:18.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.166+0000 7f3a8c55b640 1 -- start start 2024-09-16T01:40:18.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.166+0000 7f3a8c55b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a84072470 0x7f3a841f53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:18.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.166+0000 7f3a8c55b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a84158a00 0x7f3a8414b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:18.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.166+0000 7f3a8c55b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a84158de0 0x7f3a8414cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:18.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.166+0000 7f3a8c55b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3a840b5ca0 con 0x7f3a84158de0 2024-09-16T01:40:18.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.166+0000 7f3a8c55b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3a840b5b20 con 0x7f3a84158a00 2024-09-16T01:40:18.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.166+0000 7f3a8c55b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3a840b5570 con 0x7f3a84072470 2024-09-16T01:40:18.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.166+0000 7f3a8aaf8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a84158de0 0x7f3a8414cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:18.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.166+0000 7f3a8aaf8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a84158de0 0x7f3a8414cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47128/0 (socket says 172.21.15.88:47128) 2024-09-16T01:40:18.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.166+0000 7f3a8aaf8640 1 -- 172.21.15.88:0/768072011 learned_addr learned my addr 172.21.15.88:0/768072011 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:18.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.166+0000 7f3a8a2f7640 1 --2- 172.21.15.88:0/768072011 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a84072470 0x7f3a841f53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:18.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.167+0000 7f3a89af6640 1 --2- 172.21.15.88:0/768072011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a84158a00 0x7f3a8414b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:18.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.167+0000 7f3a8aaf8640 1 -- 172.21.15.88:0/768072011 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a84072470 msgr2=0x7f3a841f53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:18.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.167+0000 7f3a8aaf8640 1 --2- 172.21.15.88:0/768072011 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a84072470 0x7f3a841f53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:18.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.167+0000 7f3a8aaf8640 1 -- 172.21.15.88:0/768072011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a84158a00 msgr2=0x7f3a8414b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:18.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.167+0000 7f3a8aaf8640 1 --2- 172.21.15.88:0/768072011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a84158a00 0x7f3a8414b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:18.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.167+0000 7f3a8aaf8640 1 -- 172.21.15.88:0/768072011 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a841f5900 con 0x7f3a84158de0 2024-09-16T01:40:18.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.167+0000 7f3a8aaf8640 1 --2- 172.21.15.88:0/768072011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a84158de0 0x7f3a8414cd90 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c009f70 tx=0x7f3a7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:18.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.167+0000 7f3a76ffd640 1 -- 172.21.15.88:0/768072011 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c03f020 con 0x7f3a84158de0 2024-09-16T01:40:18.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.167+0000 7f3a76ffd640 1 -- 172.21.15.88:0/768072011 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a7c037040 con 0x7f3a84158de0 2024-09-16T01:40:18.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.167+0000 7f3a8c55b640 1 -- 172.21.15.88:0/768072011 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a8414bb70 con 0x7f3a84158de0 2024-09-16T01:40:18.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.167+0000 7f3a76ffd640 1 -- 172.21.15.88:0/768072011 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c036070 con 0x7f3a84158de0 2024-09-16T01:40:18.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.167+0000 7f3a8c55b640 1 -- 172.21.15.88:0/768072011 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a840cec70 con 0x7f3a84158de0 2024-09-16T01:40:18.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.168+0000 7f3a8c55b640 1 -- 172.21.15.88:0/768072011 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a84068f10 con 0x7f3a84158de0 2024-09-16T01:40:18.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.169+0000 7f3a76ffd640 1 -- 172.21.15.88:0/768072011 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3a7c042020 con 0x7f3a84158de0 2024-09-16T01:40:18.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.169+0000 7f3a76ffd640 1 --2- 172.21.15.88:0/768072011 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a6806c2b0 0x7f3a6806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:18.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.169+0000 7f3a76ffd640 1 -- 172.21.15.88:0/768072011 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f3a7c0ab810 con 0x7f3a84158de0 2024-09-16T01:40:18.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.169+0000 7f3a8a2f7640 1 --2- 172.21.15.88:0/768072011 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a6806c2b0 0x7f3a6806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:18.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.170+0000 7f3a8a2f7640 1 --2- 172.21.15.88:0/768072011 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a6806c2b0 0x7f3a6806e770 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f3a80006fd0 tx=0x7f3a80008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:18.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.173+0000 7f3a76ffd640 1 -- 172.21.15.88:0/768072011 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3a84068f10 con 0x7f3a84158de0 2024-09-16T01:40:18.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.372+0000 7f3a8c55b640 1 -- 172.21.15.88:0/768072011 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f3a8406f490 con 0x7f3a84158de0 2024-09-16T01:40:18.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.666+0000 7f3a76ffd640 1 -- 172.21.15.88:0/768072011 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v75) ==== 162+0+0 (secure 0 0 0) 0x7f3a7c049030 con 0x7f3a84158de0 2024-09-16T01:40:18.668 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:40:18.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.668+0000 7f3a8c55b640 1 -- 172.21.15.88:0/768072011 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a6806c2b0 msgr2=0x7f3a6806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:18.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.668+0000 7f3a8c55b640 1 --2- 172.21.15.88:0/768072011 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a6806c2b0 0x7f3a6806e770 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f3a80006fd0 tx=0x7f3a80008040 comp rx=0 tx=0).stop 2024-09-16T01:40:18.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.668+0000 7f3a8c55b640 1 -- 172.21.15.88:0/768072011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a84158de0 msgr2=0x7f3a8414cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:18.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.668+0000 7f3a8c55b640 1 --2- 172.21.15.88:0/768072011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a84158de0 0x7f3a8414cd90 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c009f70 tx=0x7f3a7c004a40 comp rx=0 tx=0).stop 2024-09-16T01:40:18.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.669+0000 7f3a8c55b640 1 -- 172.21.15.88:0/768072011 shutdown_connections 2024-09-16T01:40:18.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.669+0000 7f3a8c55b640 1 --2- 172.21.15.88:0/768072011 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a6806c2b0 0x7f3a6806e770 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:18.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.669+0000 7f3a8c55b640 1 --2- 172.21.15.88:0/768072011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a84158de0 0x7f3a8414cd90 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:18.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.669+0000 7f3a8c55b640 1 --2- 172.21.15.88:0/768072011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a84158a00 0x7f3a8414b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:18.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.669+0000 7f3a8c55b640 1 --2- 172.21.15.88:0/768072011 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a84072470 0x7f3a841f53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:18.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.669+0000 7f3a8c55b640 1 -- 172.21.15.88:0/768072011 >> 172.21.15.88:0/768072011 conn(0x7f3a8406cb50 msgr2=0x7f3a84074c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:18.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.669+0000 7f3a8c55b640 1 -- 172.21.15.88:0/768072011 shutdown_connections 2024-09-16T01:40:18.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:18.669+0000 7f3a8c55b640 1 -- 172.21.15.88:0/768072011 wait complete. 2024-09-16T01:40:19.085 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:40:19.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.184+0000 7fc380aa5640 1 Processor -- start 2024-09-16T01:40:19.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.184+0000 7fc380aa5640 1 -- start start 2024-09-16T01:40:19.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.185+0000 7fc380aa5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc37c0b4d00 0x7fc37c0b5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:19.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.185+0000 7fc380aa5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc37c0ce4f0 con 0x7fc37c0cd780 2024-09-16T01:40:19.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.185+0000 7fc380aa5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc37c0ce7d0 con 0x7fc37c0b4920 2024-09-16T01:40:19.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.185+0000 7fc380aa5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc37c0cf100 con 0x7fc37c0b4d00 2024-09-16T01:40:19.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.185+0000 7fc37ad9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc37c0cd780 0x7fc37c0cdbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:58114/0 (socket says 172.21.15.88:58114) 2024-09-16T01:40:19.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.185+0000 7fc37ad9d640 1 -- 172.21.15.88:0/1632835522 learned_addr learned my addr 172.21.15.88:0/1632835522 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:19.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.185+0000 7fc379d9b640 1 --2- 172.21.15.88:0/1632835522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc37c0b4d00 0x7fc37c0b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:19.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.186+0000 7fc37959a640 1 -- 172.21.15.88:0/1632835522 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3933966205 0 0) 0x7fc37c0ce4f0 con 0x7fc37c0cd780 2024-09-16T01:40:19.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.186+0000 7fc37959a640 1 -- 172.21.15.88:0/1632835522 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc364003520 con 0x7fc37c0cd780 2024-09-16T01:40:19.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.186+0000 7fc37959a640 1 -- 172.21.15.88:0/1632835522 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1664504726 0 0) 0x7fc37c0ce7d0 con 0x7fc37c0b4920 2024-09-16T01:40:19.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.186+0000 7fc37959a640 1 -- 172.21.15.88:0/1632835522 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc37c0ce4f0 con 0x7fc37c0b4920 2024-09-16T01:40:19.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.186+0000 7fc379d9b640 1 -- 172.21.15.88:0/1632835522 >> v1:172.21.15.154:6789/0 conn(0x7fc37c0b4920 legacy=0x7fc37c15b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:19.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.186+0000 7fc379d9b640 1 -- 172.21.15.88:0/1632835522 >> v1:172.21.15.88:6789/0 conn(0x7fc37c0cd780 legacy=0x7fc37c0cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:19.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.186+0000 7fc379d9b640 1 -- 172.21.15.88:0/1632835522 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc37c0ceab0 con 0x7fc37c0b4d00 2024-09-16T01:40:19.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.187+0000 7fc379d9b640 1 --2- 172.21.15.88:0/1632835522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc37c0b4d00 0x7fc37c0b5140 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc37c0cf280 tx=0x7fc37002a200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb9f9e4b0a278409 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:19.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.187+0000 7fc37959a640 1 -- 172.21.15.88:0/1632835522 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc370037070 con 0x7fc37c0b4d00 2024-09-16T01:40:19.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.187+0000 7fc37959a640 1 -- 172.21.15.88:0/1632835522 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc370032440 con 0x7fc37c0b4d00 2024-09-16T01:40:19.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.187+0000 7fc37959a640 1 -- 172.21.15.88:0/1632835522 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc370032810 con 0x7fc37c0b4d00 2024-09-16T01:40:19.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.187+0000 7fc380aa5640 1 -- 172.21.15.88:0/1632835522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc37c0b4d00 msgr2=0x7fc37c0b5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:19.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.187+0000 7fc380aa5640 1 --2- 172.21.15.88:0/1632835522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc37c0b4d00 0x7fc37c0b5140 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc37c0cf280 tx=0x7fc37002a200 comp rx=0 tx=0).stop 2024-09-16T01:40:19.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.187+0000 7fc380aa5640 1 -- 172.21.15.88:0/1632835522 shutdown_connections 2024-09-16T01:40:19.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.187+0000 7fc380aa5640 1 --2- 172.21.15.88:0/1632835522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc37c0b4d00 0x7fc37c0b5140 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:19.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.187+0000 7fc380aa5640 1 -- 172.21.15.88:0/1632835522 >> 172.21.15.88:0/1632835522 conn(0x7fc37c06bc10 msgr2=0x7fc37c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:19.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.187+0000 7fc380aa5640 1 -- 172.21.15.88:0/1632835522 shutdown_connections 2024-09-16T01:40:19.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.187+0000 7fc380aa5640 1 -- 172.21.15.88:0/1632835522 wait complete. 2024-09-16T01:40:19.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.189+0000 7fc380aa5640 1 Processor -- start 2024-09-16T01:40:19.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.189+0000 7fc380aa5640 1 -- start start 2024-09-16T01:40:19.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.189+0000 7fc380aa5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc37c0b4920 0x7fc37c0c71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:19.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.189+0000 7fc380aa5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc37c0b4d00 0x7fc37c15c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:19.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.189+0000 7fc380aa5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc37c0cd780 0x7fc37c147e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:19.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.189+0000 7fc380aa5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc37c0cf590 con 0x7fc37c0b4d00 2024-09-16T01:40:19.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.189+0000 7fc380aa5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc37c0b5680 con 0x7fc37c0b4920 2024-09-16T01:40:19.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.189+0000 7fc380aa5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc37c0cf710 con 0x7fc37c0cd780 2024-09-16T01:40:19.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.190+0000 7fc379d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc37c0b4d00 0x7fc37c15c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:19.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.190+0000 7fc379d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc37c0b4d00 0x7fc37c15c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47138/0 (socket says 172.21.15.88:47138) 2024-09-16T01:40:19.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.190+0000 7fc379d9b640 1 -- 172.21.15.88:0/340748395 learned_addr learned my addr 172.21.15.88:0/340748395 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:19.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.190+0000 7fc37a59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc37c0b4920 0x7fc37c0c71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:19.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.190+0000 7fc37ad9d640 1 --2- 172.21.15.88:0/340748395 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc37c0cd780 0x7fc37c147e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.190+0000 7fc379d9b640 1 -- 172.21.15.88:0/340748395 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc37c0cd780 msgr2=0x7fc37c147e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.190+0000 7fc379d9b640 1 --2- 172.21.15.88:0/340748395 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc37c0cd780 0x7fc37c147e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.190+0000 7fc379d9b640 1 -- 172.21.15.88:0/340748395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc37c0b4920 msgr2=0x7fc37c0c71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.190+0000 7fc379d9b640 1 --2- 172.21.15.88:0/340748395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc37c0b4920 0x7fc37c0c71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.190+0000 7fc379d9b640 1 -- 172.21.15.88:0/340748395 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc37c0c78a0 con 0x7fc37c0b4d00 2024-09-16T01:40:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.190+0000 7fc379d9b640 1 --2- 172.21.15.88:0/340748395 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc37c0b4d00 0x7fc37c15c270 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fc370035040 tx=0x7fc3700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.190+0000 7fc36affd640 1 -- 172.21.15.88:0/340748395 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc370041030 con 0x7fc37c0b4d00 2024-09-16T01:40:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.190+0000 7fc36affd640 1 -- 172.21.15.88:0/340748395 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc37003d4e0 con 0x7fc37c0b4d00 2024-09-16T01:40:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.190+0000 7fc380aa5640 1 -- 172.21.15.88:0/340748395 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc37c0c7b80 con 0x7fc37c0b4d00 2024-09-16T01:40:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.190+0000 7fc36affd640 1 -- 172.21.15.88:0/340748395 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc370037040 con 0x7fc37c0b4d00 2024-09-16T01:40:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.190+0000 7fc380aa5640 1 -- 172.21.15.88:0/340748395 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc37c15d2e0 con 0x7fc37c0b4d00 2024-09-16T01:40:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.191+0000 7fc380aa5640 1 -- 172.21.15.88:0/340748395 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc37c0b5140 con 0x7fc37c0b4d00 2024-09-16T01:40:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.192+0000 7fc36affd640 1 -- 172.21.15.88:0/340748395 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc370044070 con 0x7fc37c0b4d00 2024-09-16T01:40:19.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.192+0000 7fc36affd640 1 --2- 172.21.15.88:0/340748395 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc35c06c2b0 0x7fc35c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:19.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.192+0000 7fc36affd640 1 -- 172.21.15.88:0/340748395 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7fc3700ab9b0 con 0x7fc37c0b4d00 2024-09-16T01:40:19.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.192+0000 7fc37a59c640 1 --2- 172.21.15.88:0/340748395 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc35c06c2b0 0x7fc35c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:19.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.193+0000 7fc37a59c640 1 --2- 172.21.15.88:0/340748395 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc35c06c2b0 0x7fc35c06e770 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc37c0b2b40 tx=0x7fc36c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:19.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.196+0000 7fc36affd640 1 -- 172.21.15.88:0/340748395 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc370079be0 con 0x7fc37c0b4d00 2024-09-16T01:40:19.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.396+0000 7fc380aa5640 1 -- 172.21.15.88:0/340748395 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc37c15a700 con 0x7fc37c0b4d00 2024-09-16T01:40:19.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.397+0000 7fc36affd640 1 -- 172.21.15.88:0/340748395 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+5222 (secure 0 0 0) 0x7fc37007dee0 con 0x7fc37c0b4d00 2024-09-16T01:40:19.398 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:19.398 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":76,"btime":"2024-09-16T01:40:18:669752+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6849,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/1289932173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1289932173},{"type":"v1","addr":"172.21.15.88:6836","nonce":1289932173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:40:18.669739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7026},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7026":{"gid":7026,"name":"c","rank":0,"incarnation":76,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6837/2733458327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2733458327},{"type":"v1","addr":"172.21.15.88:6837","nonce":2733458327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:40:19.398 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 76 2024-09-16T01:40:19.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.398+0000 7fc380aa5640 1 -- 172.21.15.88:0/340748395 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc35c06c2b0 msgr2=0x7fc35c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:19.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.398+0000 7fc380aa5640 1 --2- 172.21.15.88:0/340748395 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc35c06c2b0 0x7fc35c06e770 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc37c0b2b40 tx=0x7fc36c009040 comp rx=0 tx=0).stop 2024-09-16T01:40:19.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.398+0000 7fc380aa5640 1 -- 172.21.15.88:0/340748395 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc37c0b4d00 msgr2=0x7fc37c15c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:19.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.398+0000 7fc380aa5640 1 --2- 172.21.15.88:0/340748395 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc37c0b4d00 0x7fc37c15c270 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fc370035040 tx=0x7fc3700046c0 comp rx=0 tx=0).stop 2024-09-16T01:40:19.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.398+0000 7fc380aa5640 1 -- 172.21.15.88:0/340748395 shutdown_connections 2024-09-16T01:40:19.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.398+0000 7fc380aa5640 1 --2- 172.21.15.88:0/340748395 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc35c06c2b0 0x7fc35c06e770 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:19.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.398+0000 7fc380aa5640 1 --2- 172.21.15.88:0/340748395 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc37c0cd780 0x7fc37c147e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:19.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.398+0000 7fc380aa5640 1 --2- 172.21.15.88:0/340748395 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc37c0b4d00 0x7fc37c15c270 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:19.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.398+0000 7fc380aa5640 1 --2- 172.21.15.88:0/340748395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc37c0b4920 0x7fc37c0c71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:19.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.398+0000 7fc380aa5640 1 -- 172.21.15.88:0/340748395 >> 172.21.15.88:0/340748395 conn(0x7fc37c06bc10 msgr2=0x7fc37c157d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:19.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.399+0000 7fc380aa5640 1 -- 172.21.15.88:0/340748395 shutdown_connections 2024-09-16T01:40:19.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:19.399+0000 7fc380aa5640 1 -- 172.21.15.88:0/340748395 wait complete. 2024-09-16T01:40:19.410 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:40:03.597796+0000', 'modified': '2024-09-16T01:40:18.669739+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 86, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7026}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7026': {'gid': 7026, 'name': 'c', 'rank': 0, 'incarnation': 76, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.88:6837/2733458327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6835', 'nonce': 2733458327}, {'type': 'v1', 'addr': '172.21.15.88:6837', 'nonce': 2733458327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-16T01:40:19.411 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7026:up:replay 2024-09-16T01:40:19.411 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-09-16T01:40:22.533 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:40:22.532+0000 7f9849fdc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:40:24.814 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:40:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.924+0000 7ff460eaf640 1 Processor -- start 2024-09-16T01:40:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.924+0000 7ff460eaf640 1 -- start start 2024-09-16T01:40:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.925+0000 7ff460eaf640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff45c15c710 0x7ff45c072050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.925+0000 7ff460eaf640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff45c072590 0x7ff45c15cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.925+0000 7ff460eaf640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff45c072a30 con 0x7ff45c06bc10 2024-09-16T01:40:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.925+0000 7ff460eaf640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff45c072d10 con 0x7ff45c15c710 2024-09-16T01:40:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.925+0000 7ff460eaf640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff45c072ff0 con 0x7ff45c072590 2024-09-16T01:40:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.926+0000 7ff45ad9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff45c072590 0x7ff45c15cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.926+0000 7ff453fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff45c15c710 0x7ff45c072050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:24.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.926+0000 7ff45a59c640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7ff45c06bc10 0x7ff45c06bff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:42600/0 (socket says 172.21.15.88:42600) 2024-09-16T01:40:24.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.926+0000 7ff45ad9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff45c072590 0x7ff45c15cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:56162/0 (socket says 172.21.15.88:56162) 2024-09-16T01:40:24.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.926+0000 7ff453fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff45c15c710 0x7ff45c072050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:51562/0 (socket says 172.21.15.88:51562) 2024-09-16T01:40:24.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.926+0000 7ff45a59c640 1 -- 172.21.15.88:0/4119355999 learned_addr learned my addr 172.21.15.88:0/4119355999 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:24.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.926+0000 7ff453fff640 1 -- 172.21.15.88:0/4119355999 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff45c072590 msgr2=0x7ff45c15cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:24.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.926+0000 7ff453fff640 1 --2- 172.21.15.88:0/4119355999 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff45c072590 0x7ff45c15cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:24.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.926+0000 7ff453fff640 1 -- 172.21.15.88:0/4119355999 >> v1:172.21.15.154:6789/0 conn(0x7ff45c06bc10 legacy=0x7ff45c06bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:24.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.926+0000 7ff459d9b640 1 -- 172.21.15.88:0/4119355999 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3516103774 0 0) 0x7ff45c072a30 con 0x7ff45c06bc10 2024-09-16T01:40:24.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.926+0000 7ff453fff640 1 -- 172.21.15.88:0/4119355999 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff45c0732d0 con 0x7ff45c15c710 2024-09-16T01:40:24.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.926+0000 7ff45ad9d640 1 --2- 172.21.15.88:0/4119355999 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff45c072590 0x7ff45c15cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:40:24.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.927+0000 7ff453fff640 1 --2- 172.21.15.88:0/4119355999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff45c15c710 0x7ff45c072050 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff4440097a0 tx=0x7ff44402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fe7243ce05dce3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:24.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.927+0000 7ff459d9b640 1 -- 172.21.15.88:0/4119355999 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff444037070 con 0x7ff45c15c710 2024-09-16T01:40:24.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.927+0000 7ff459d9b640 1 -- 172.21.15.88:0/4119355999 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff444032440 con 0x7ff45c15c710 2024-09-16T01:40:24.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.927+0000 7ff459d9b640 1 -- 172.21.15.88:0/4119355999 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff444032810 con 0x7ff45c15c710 2024-09-16T01:40:24.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.928+0000 7ff460eaf640 1 -- 172.21.15.88:0/4119355999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff45c15c710 msgr2=0x7ff45c072050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:24.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.928+0000 7ff460eaf640 1 --2- 172.21.15.88:0/4119355999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff45c15c710 0x7ff45c072050 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff4440097a0 tx=0x7ff44402a180 comp rx=0 tx=0).stop 2024-09-16T01:40:24.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.928+0000 7ff460eaf640 1 -- 172.21.15.88:0/4119355999 shutdown_connections 2024-09-16T01:40:24.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.928+0000 7ff460eaf640 1 --2- 172.21.15.88:0/4119355999 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff45c072590 0x7ff45c15cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:24.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.928+0000 7ff460eaf640 1 --2- 172.21.15.88:0/4119355999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff45c15c710 0x7ff45c072050 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:24.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.928+0000 7ff460eaf640 1 -- 172.21.15.88:0/4119355999 >> 172.21.15.88:0/4119355999 conn(0x7ff45c0d14d0 msgr2=0x7ff45c0d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:24.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.928+0000 7ff460eaf640 1 -- 172.21.15.88:0/4119355999 shutdown_connections 2024-09-16T01:40:24.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.928+0000 7ff460eaf640 1 -- 172.21.15.88:0/4119355999 wait complete. 2024-09-16T01:40:24.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.929+0000 7ff460eaf640 1 Processor -- start 2024-09-16T01:40:24.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.929+0000 7ff460eaf640 1 -- start start 2024-09-16T01:40:24.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.930+0000 7ff460eaf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff45c06bc10 0x7ff45c145e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:24.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.930+0000 7ff460eaf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff45c072590 0x7ff45c158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:24.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.930+0000 7ff460eaf640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff45c15c710 0x7ff45c1585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:24.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.930+0000 7ff460eaf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff45c06f9d0 con 0x7ff45c06bc10 2024-09-16T01:40:24.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.930+0000 7ff460eaf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff45c06f850 con 0x7ff45c072590 2024-09-16T01:40:24.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.930+0000 7ff460eaf640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff45c15d2f0 con 0x7ff45c15c710 2024-09-16T01:40:24.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.930+0000 7ff45a59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff45c06bc10 0x7ff45c145e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:24.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.930+0000 7ff45a59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff45c06bc10 0x7ff45c145e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38498/0 (socket says 172.21.15.88:38498) 2024-09-16T01:40:24.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.930+0000 7ff45a59c640 1 -- 172.21.15.88:0/3850812077 learned_addr learned my addr 172.21.15.88:0/3850812077 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:24.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.930+0000 7ff453fff640 1 --2- 172.21.15.88:0/3850812077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff45c072590 0x7ff45c158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:24.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.930+0000 7ff45ad9d640 1 --2- 172.21.15.88:0/3850812077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff45c15c710 0x7ff45c1585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:24.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.930+0000 7ff45a59c640 1 -- 172.21.15.88:0/3850812077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff45c15c710 msgr2=0x7ff45c1585b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:24.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.930+0000 7ff45a59c640 1 --2- 172.21.15.88:0/3850812077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff45c15c710 0x7ff45c1585b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:24.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.930+0000 7ff45a59c640 1 -- 172.21.15.88:0/3850812077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff45c072590 msgr2=0x7ff45c158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:24.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.930+0000 7ff45a59c640 1 --2- 172.21.15.88:0/3850812077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff45c072590 0x7ff45c158070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:24.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.930+0000 7ff45a59c640 1 -- 172.21.15.88:0/3850812077 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff45c1e4e00 con 0x7ff45c06bc10 2024-09-16T01:40:24.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.930+0000 7ff45a59c640 1 --2- 172.21.15.88:0/3850812077 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff45c06bc10 0x7ff45c145e50 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7ff44c00c4d0 tx=0x7ff44c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:24.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.931+0000 7ff452ffd640 1 -- 172.21.15.88:0/3850812077 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44c018070 con 0x7ff45c06bc10 2024-09-16T01:40:24.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.931+0000 7ff452ffd640 1 -- 172.21.15.88:0/3850812077 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff44c007500 con 0x7ff45c06bc10 2024-09-16T01:40:24.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.931+0000 7ff460eaf640 1 -- 172.21.15.88:0/3850812077 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff45c1e50e0 con 0x7ff45c06bc10 2024-09-16T01:40:24.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.931+0000 7ff452ffd640 1 -- 172.21.15.88:0/3850812077 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44c004a20 con 0x7ff45c06bc10 2024-09-16T01:40:24.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.931+0000 7ff460eaf640 1 -- 172.21.15.88:0/3850812077 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff45c1e56a0 con 0x7ff45c06bc10 2024-09-16T01:40:24.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.932+0000 7ff460eaf640 1 -- 172.21.15.88:0/3850812077 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff45c15cdb0 con 0x7ff45c06bc10 2024-09-16T01:40:24.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.932+0000 7ff452ffd640 1 -- 172.21.15.88:0/3850812077 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff44c004bc0 con 0x7ff45c06bc10 2024-09-16T01:40:24.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.933+0000 7ff452ffd640 1 --2- 172.21.15.88:0/3850812077 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff43c06c2b0 0x7ff43c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:24.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.933+0000 7ff452ffd640 1 -- 172.21.15.88:0/3850812077 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7ff44c08a750 con 0x7ff45c06bc10 2024-09-16T01:40:24.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.933+0000 7ff453fff640 1 --2- 172.21.15.88:0/3850812077 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff43c06c2b0 0x7ff43c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:24.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.933+0000 7ff453fff640 1 --2- 172.21.15.88:0/3850812077 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff43c06c2b0 0x7ff43c06e770 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff444034000 tx=0x7ff44402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:24.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:24.937+0000 7ff452ffd640 1 -- 172.21.15.88:0/3850812077 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff44c058e70 con 0x7ff45c06bc10 2024-09-16T01:40:25.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.137+0000 7ff460eaf640 1 -- 172.21.15.88:0/3850812077 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff45c0b4700 con 0x7ff45c06bc10 2024-09-16T01:40:25.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.137+0000 7ff452ffd640 1 -- 172.21.15.88:0/3850812077 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+5229 (secure 0 0 0) 0x7ff44c058770 con 0x7ff45c06bc10 2024-09-16T01:40:25.139 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:25.139 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":79,"btime":"2024-09-16T01:40:21:679207+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6849,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/1289932173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1289932173},{"type":"v1","addr":"172.21.15.88:6836","nonce":1289932173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:40:21.679195+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7026},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7026":{"gid":7026,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":5,"addr":"172.21.15.88:6837/2733458327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2733458327},{"type":"v1","addr":"172.21.15.88:6837","nonce":2733458327}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7026,"qdb_cluster":[7026]},"id":6}]} 2024-09-16T01:40:25.139 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 79 2024-09-16T01:40:25.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.139+0000 7ff460eaf640 1 -- 172.21.15.88:0/3850812077 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff43c06c2b0 msgr2=0x7ff43c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:25.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.139+0000 7ff460eaf640 1 --2- 172.21.15.88:0/3850812077 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff43c06c2b0 0x7ff43c06e770 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff444034000 tx=0x7ff44402e000 comp rx=0 tx=0).stop 2024-09-16T01:40:25.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.139+0000 7ff460eaf640 1 -- 172.21.15.88:0/3850812077 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff45c06bc10 msgr2=0x7ff45c145e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:25.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.139+0000 7ff460eaf640 1 --2- 172.21.15.88:0/3850812077 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff45c06bc10 0x7ff45c145e50 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7ff44c00c4d0 tx=0x7ff44c00c990 comp rx=0 tx=0).stop 2024-09-16T01:40:25.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.139+0000 7ff460eaf640 1 -- 172.21.15.88:0/3850812077 shutdown_connections 2024-09-16T01:40:25.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.139+0000 7ff460eaf640 1 --2- 172.21.15.88:0/3850812077 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff43c06c2b0 0x7ff43c06e770 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.139+0000 7ff460eaf640 1 --2- 172.21.15.88:0/3850812077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff45c15c710 0x7ff45c1585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.139+0000 7ff460eaf640 1 --2- 172.21.15.88:0/3850812077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff45c072590 0x7ff45c158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.139+0000 7ff460eaf640 1 --2- 172.21.15.88:0/3850812077 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff45c06bc10 0x7ff45c145e50 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.139+0000 7ff460eaf640 1 -- 172.21.15.88:0/3850812077 >> 172.21.15.88:0/3850812077 conn(0x7ff45c0d14d0 msgr2=0x7ff45c15a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:25.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.139+0000 7ff460eaf640 1 -- 172.21.15.88:0/3850812077 shutdown_connections 2024-09-16T01:40:25.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.139+0000 7ff460eaf640 1 -- 172.21.15.88:0/3850812077 wait complete. 2024-09-16T01:40:25.152 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:40:03.597796+0000', 'modified': '2024-09-16T01:40:21.679195+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 86, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7026}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7026': {'gid': 7026, 'name': 'c', 'rank': 0, 'incarnation': 76, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.88:6837/2733458327', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6835', 'nonce': 2733458327}, {'type': 'v1', 'addr': '172.21.15.88:6837', 'nonce': 2733458327}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7026, 'qdb_cluster': [7026]} 2024-09-16T01:40:25.152 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T01:40:25.152 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-16T01:40:25.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.250+0000 7f7827627640 1 Processor -- start 2024-09-16T01:40:25.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.250+0000 7f7827627640 1 -- start start 2024-09-16T01:40:25.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.251+0000 7f7827627640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f782015ad10 0x7f78200719c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.251+0000 7f7827627640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f78200cca40 0x7f7820071f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.251+0000 7f7827627640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f78200cc360 0x7f782015e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.251+0000 7f7827627640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7820070fb0 con 0x7f78200cc360 2024-09-16T01:40:25.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.251+0000 7f7827627640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7820071720 con 0x7f78200cca40 2024-09-16T01:40:25.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.251+0000 7f7827627640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7820073920 con 0x7f782015ad10 2024-09-16T01:40:25.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.251+0000 7f7825bc4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f78200cc360 0x7f782015e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:25.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.251+0000 7f7825bc4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f78200cc360 0x7f782015e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38500/0 (socket says 172.21.15.88:38500) 2024-09-16T01:40:25.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.251+0000 7f7825bc4640 1 -- 172.21.15.88:0/1040764625 learned_addr learned my addr 172.21.15.88:0/1040764625 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:25.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.252+0000 7f7824bc2640 1 --2- 172.21.15.88:0/1040764625 >> v2:172.21.15.154:3300/0 conn(0x7f78200cca40 0x7f7820071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:25.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.252+0000 7f78253c3640 1 --2- 172.21.15.88:0/1040764625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f782015ad10 0x7f78200719c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:25.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.252+0000 7f7825bc4640 1 -- 172.21.15.88:0/1040764625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f782015ad10 msgr2=0x7f78200719c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:25.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.252+0000 7f7825bc4640 1 --2- 172.21.15.88:0/1040764625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f782015ad10 0x7f78200719c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.252+0000 7f7825bc4640 1 -- 172.21.15.88:0/1040764625 >> v2:172.21.15.154:3300/0 conn(0x7f78200cca40 msgr2=0x7f7820071f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:25.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.252+0000 7f7825bc4640 1 --2- 172.21.15.88:0/1040764625 >> v2:172.21.15.154:3300/0 conn(0x7f78200cca40 0x7f7820071f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.252+0000 7f7825bc4640 1 -- 172.21.15.88:0/1040764625 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7820073c00 con 0x7f78200cc360 2024-09-16T01:40:25.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.252+0000 7f7825bc4640 1 --2- 172.21.15.88:0/1040764625 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78200cc360 0x7f782015e710 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f7820075800 tx=0x7f781802c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d35ef6de31ca5f79 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:25.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.253+0000 7f7817fff640 1 -- 172.21.15.88:0/1040764625 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781803b070 con 0x7f78200cc360 2024-09-16T01:40:25.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.253+0000 7f7817fff640 1 -- 172.21.15.88:0/1040764625 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7818033480 con 0x7f78200cc360 2024-09-16T01:40:25.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.253+0000 7f7817fff640 1 -- 172.21.15.88:0/1040764625 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7818033870 con 0x7f78200cc360 2024-09-16T01:40:25.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.253+0000 7f7827627640 1 -- 172.21.15.88:0/1040764625 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78200cc360 msgr2=0x7f782015e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:25.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.253+0000 7f7827627640 1 --2- 172.21.15.88:0/1040764625 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78200cc360 0x7f782015e710 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f7820075800 tx=0x7f781802c810 comp rx=0 tx=0).stop 2024-09-16T01:40:25.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.253+0000 7f7827627640 1 -- 172.21.15.88:0/1040764625 shutdown_connections 2024-09-16T01:40:25.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.253+0000 7f7827627640 1 --2- 172.21.15.88:0/1040764625 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78200cc360 0x7f782015e710 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.253+0000 7f7827627640 1 --2- 172.21.15.88:0/1040764625 >> v2:172.21.15.154:3300/0 conn(0x7f78200cca40 0x7f7820071f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.253+0000 7f7827627640 1 --2- 172.21.15.88:0/1040764625 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f782015ad10 0x7f78200719c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.253+0000 7f7827627640 1 -- 172.21.15.88:0/1040764625 >> 172.21.15.88:0/1040764625 conn(0x7f782006d0f0 msgr2=0x7f782006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:25.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.253+0000 7f7827627640 1 -- 172.21.15.88:0/1040764625 shutdown_connections 2024-09-16T01:40:25.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.253+0000 7f7827627640 1 -- 172.21.15.88:0/1040764625 wait complete. 2024-09-16T01:40:25.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.254+0000 7f7827627640 1 Processor -- start 2024-09-16T01:40:25.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.254+0000 7f7827627640 1 -- start start 2024-09-16T01:40:25.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.255+0000 7f7827627640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78200cc360 0x7f782014a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.255+0000 7f7827627640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78200cca40 0x7f7820153310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.255+0000 7f7827627640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f782015ad10 0x7f78201f5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.255+0000 7f7827627640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f782015ec50 con 0x7f78200cc360 2024-09-16T01:40:25.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.255+0000 7f7827627640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7820071200 con 0x7f78200cca40 2024-09-16T01:40:25.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.255+0000 7f7827627640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f782015edd0 con 0x7f782015ad10 2024-09-16T01:40:25.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.255+0000 7f78253c3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78200cc360 0x7f782014a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.255+0000 7f78253c3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78200cc360 0x7f782014a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38516/0 (socket says 172.21.15.88:38516) 2024-09-16T01:40:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.255+0000 7f78253c3640 1 -- 172.21.15.88:0/2772631796 learned_addr learned my addr 172.21.15.88:0/2772631796 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.255+0000 7f7824bc2640 1 --2- 172.21.15.88:0/2772631796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78200cca40 0x7f7820153310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.255+0000 7f7825bc4640 1 --2- 172.21.15.88:0/2772631796 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f782015ad10 0x7f78201f5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.256+0000 7f78253c3640 1 -- 172.21.15.88:0/2772631796 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f782015ad10 msgr2=0x7f78201f5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.256+0000 7f78253c3640 1 --2- 172.21.15.88:0/2772631796 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f782015ad10 0x7f78201f5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.256+0000 7f78253c3640 1 -- 172.21.15.88:0/2772631796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78200cca40 msgr2=0x7f7820153310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.256+0000 7f78253c3640 1 --2- 172.21.15.88:0/2772631796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78200cca40 0x7f7820153310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.256+0000 7f78253c3640 1 -- 172.21.15.88:0/2772631796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7820152270 con 0x7f78200cc360 2024-09-16T01:40:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.256+0000 7f7825bc4640 1 --2- 172.21.15.88:0/2772631796 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f782015ad10 0x7f78201f5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.256+0000 7f78253c3640 1 --2- 172.21.15.88:0/2772631796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78200cc360 0x7f782014a150 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f781c00c7f0 tx=0x7f781c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.256+0000 7f7815ffb640 1 -- 172.21.15.88:0/2772631796 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781c015070 con 0x7f78200cc360 2024-09-16T01:40:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.256+0000 7f7815ffb640 1 -- 172.21.15.88:0/2772631796 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f781c004220 con 0x7f78200cc360 2024-09-16T01:40:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.256+0000 7f7827627640 1 -- 172.21.15.88:0/2772631796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7820152550 con 0x7f78200cc360 2024-09-16T01:40:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.256+0000 7f7815ffb640 1 -- 172.21.15.88:0/2772631796 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781c007620 con 0x7f78200cc360 2024-09-16T01:40:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.256+0000 7f7827627640 1 -- 172.21.15.88:0/2772631796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7820152b10 con 0x7f78200cc360 2024-09-16T01:40:25.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.258+0000 7f7815ffb640 1 -- 172.21.15.88:0/2772631796 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f781c0079d0 con 0x7f78200cc360 2024-09-16T01:40:25.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.258+0000 7f7815ffb640 1 --2- 172.21.15.88:0/2772631796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f780406c2b0 0x7f780406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.258+0000 7f7815ffb640 1 -- 172.21.15.88:0/2772631796 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f781c08b510 con 0x7f78200cc360 2024-09-16T01:40:25.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.258+0000 7f7824bc2640 1 --2- 172.21.15.88:0/2772631796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f780406c2b0 0x7f780406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:25.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.259+0000 7f7824bc2640 1 --2- 172.21.15.88:0/2772631796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f780406c2b0 0x7f780406e770 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f78200b31b0 tx=0x7f7810008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:25.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.259+0000 7f782a3c3740 1 Processor -- start 2024-09-16T01:40:25.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.259+0000 7f782a3c3740 1 -- start start 2024-09-16T01:40:25.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.260+0000 7f782a3c3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621923292c0 0x56219236ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.260+0000 7f782a3c3740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x562192326a80 0x5621923296a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.260+0000 7f782a3c3740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56219232aa50 0x5621923238b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.260+0000 7f782a3c3740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56219228b520 con 0x56219232aa50 2024-09-16T01:40:25.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.260+0000 7f782a3c3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5621922333e0 con 0x5621923292c0 2024-09-16T01:40:25.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.260+0000 7f782a3c3740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5621922874e0 con 0x562192326a80 2024-09-16T01:40:25.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.260+0000 7f7825bc4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56219232aa50 0x5621923238b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:25.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.260+0000 7f7825bc4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56219232aa50 0x5621923238b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38518/0 (socket says 172.21.15.88:38518) 2024-09-16T01:40:25.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.260+0000 7f7825bc4640 1 -- 172.21.15.88:0/2282280186 learned_addr learned my addr 172.21.15.88:0/2282280186 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:25.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.260+0000 7f78253c3640 1 --2- 172.21.15.88:0/2282280186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621923292c0 0x56219236ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:25.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.260+0000 7f7825bc4640 1 -- 172.21.15.88:0/2282280186 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x562192326a80 msgr2=0x5621923296a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:25.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.260+0000 7f7825bc4640 1 --2- 172.21.15.88:0/2282280186 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x562192326a80 0x5621923296a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.260+0000 7f7825bc4640 1 -- 172.21.15.88:0/2282280186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621923292c0 msgr2=0x56219236ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:25.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.260+0000 7f7825bc4640 1 --2- 172.21.15.88:0/2282280186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621923292c0 0x56219236ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.260+0000 7f7825bc4640 1 -- 172.21.15.88:0/2282280186 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56219224e2f0 con 0x56219232aa50 2024-09-16T01:40:25.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.260+0000 7f7825bc4640 1 --2- 172.21.15.88:0/2282280186 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56219232aa50 0x5621923238b0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f781800d240 tx=0x7f7818033000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c368f08faf7250e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:25.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.260+0000 7f780affd640 1 -- 172.21.15.88:0/2282280186 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7818043070 con 0x56219232aa50 2024-09-16T01:40:25.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.260+0000 7f780affd640 1 -- 172.21.15.88:0/2282280186 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f781800b2b0 con 0x56219232aa50 2024-09-16T01:40:25.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.261+0000 7f780affd640 1 -- 172.21.15.88:0/2282280186 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781803b070 con 0x56219232aa50 2024-09-16T01:40:25.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.261+0000 7f782a3c3740 1 -- 172.21.15.88:0/2282280186 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56219232aa50 msgr2=0x5621923238b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:25.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.261+0000 7f782a3c3740 1 --2- 172.21.15.88:0/2282280186 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56219232aa50 0x5621923238b0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f781800d240 tx=0x7f7818033000 comp rx=0 tx=0).stop 2024-09-16T01:40:25.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.261+0000 7f782a3c3740 1 -- 172.21.15.88:0/2282280186 shutdown_connections 2024-09-16T01:40:25.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.261+0000 7f782a3c3740 1 --2- 172.21.15.88:0/2282280186 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56219232aa50 0x5621923238b0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.261+0000 7f782a3c3740 1 --2- 172.21.15.88:0/2282280186 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x562192326a80 0x5621923296a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.261+0000 7f782a3c3740 1 --2- 172.21.15.88:0/2282280186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621923292c0 0x56219236ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.261+0000 7f782a3c3740 1 -- 172.21.15.88:0/2282280186 >> 172.21.15.88:0/2282280186 conn(0x562192325a80 msgr2=0x56219236df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:25.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.261+0000 7f782a3c3740 1 -- 172.21.15.88:0/2282280186 shutdown_connections 2024-09-16T01:40:25.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.261+0000 7f782a3c3740 1 -- 172.21.15.88:0/2282280186 wait complete. 2024-09-16T01:40:25.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.262+0000 7f782a3c3740 1 -- start start 2024-09-16T01:40:25.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.262+0000 7f782a3c3740 1 Processor -- start 2024-09-16T01:40:25.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.263+0000 7f782a3c3740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621923292c0 0x562192345ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.263+0000 7f782a3c3740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56219232aa50 0x5621923496b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.263+0000 7f782a3c3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562192349bf0 0x562192353080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.263+0000 7f782a3c3740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x562192324f20 con 0x5621923292c0 2024-09-16T01:40:25.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.263+0000 7f782a3c3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562192214450 con 0x562192349bf0 2024-09-16T01:40:25.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.263+0000 7f782a3c3740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56219236f290 con 0x56219232aa50 2024-09-16T01:40:25.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.263+0000 7f782a3c3740 10 client.0 _ll_register_callbacks cb 0x562191f9dac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:40:25.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.263+0000 7f78253c3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621923292c0 0x562192345ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:25.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.263+0000 7f78253c3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621923292c0 0x562192345ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38520/0 (socket says 172.21.15.88:38520) 2024-09-16T01:40:25.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.263+0000 7f78253c3640 1 -- 172.21.15.88:0/32329077 learned_addr learned my addr 172.21.15.88:0/32329077 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:25.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.263+0000 7f7824bc2640 1 --2- 172.21.15.88:0/32329077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56219232aa50 0x5621923496b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:25.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.263+0000 7f7825bc4640 1 --2- 172.21.15.88:0/32329077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562192349bf0 0x562192353080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:25.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.263+0000 7f78253c3640 1 -- 172.21.15.88:0/32329077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56219232aa50 msgr2=0x5621923496b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:25.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.263+0000 7f78253c3640 1 --2- 172.21.15.88:0/32329077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56219232aa50 0x5621923496b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.263+0000 7f78253c3640 1 -- 172.21.15.88:0/32329077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562192349bf0 msgr2=0x562192353080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:25.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.263+0000 7f78253c3640 1 --2- 172.21.15.88:0/32329077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562192349bf0 0x562192353080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.263+0000 7f78253c3640 1 -- 172.21.15.88:0/32329077 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562191f9e2d0 con 0x5621923292c0 2024-09-16T01:40:25.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.263+0000 7f7825bc4640 1 --2- 172.21.15.88:0/32329077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562192349bf0 0x562192353080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:25.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.263+0000 7f78253c3640 1 --2- 172.21.15.88:0/32329077 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621923292c0 0x562192345ae0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f781c00a500 tx=0x7f781c059d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb17b36c35ae368f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:25.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.263+0000 7f77fffff640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:40:25.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.263+0000 7f77fffff640 1 -- 172.21.15.88:0/32329077 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781c01e040 con 0x5621923292c0 2024-09-16T01:40:25.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.263+0000 7f77fffff640 1 -- 172.21.15.88:0/32329077 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f781c08b0c0 con 0x5621923292c0 2024-09-16T01:40:25.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.263+0000 7f782a3c3740 1 -- 172.21.15.88:0/32329077 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f781800b2b0 con 0x5621923292c0 2024-09-16T01:40:25.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.264+0000 7f77fffff640 1 -- 172.21.15.88:0/32329077 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f781c08b2c0 con 0x5621923292c0 2024-09-16T01:40:25.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.264+0000 7f782a3c3740 10 client.7524 fetch_fsmap learned FSMap version 79 2024-09-16T01:40:25.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.264+0000 7f782a3c3740 1 -- 172.21.15.88:0/32329077 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=2+}) -- 0x5621920686b0 con 0x5621923292c0 2024-09-16T01:40:25.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.264+0000 7f77fffff640 1 -- 172.21.15.88:0/32329077 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f781800b2b0 con 0x5621923292c0 2024-09-16T01:40:25.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.264+0000 7f77fffff640 1 -- 172.21.15.88:0/32329077 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 79) ==== 3502+0+0 (secure 0 0 0) 0x7f781c058960 con 0x5621923292c0 2024-09-16T01:40:25.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.264+0000 7f782a3c3740 10 client.7524 fetch_fsmap finished waiting for FSMap version 79 2024-09-16T01:40:25.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.264+0000 7f782a3c3740 10 client.7524 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T01:40:25.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.264+0000 7f782a3c3740 20 client.7524 populate_metadata read hostname 'smithi088' 2024-09-16T01:40:25.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.264+0000 7f782a3c3740 1 --2- 172.21.15.88:0/32329077 >> [v2:172.21.15.88:6835/2733458327,v1:172.21.15.88:6837/2733458327] conn(0x56219234c750 0x562192351620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.264+0000 7f782a3c3740 4 client.7524 mds_command: new command op to 7026 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:40:25.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.264+0000 7f782a3c3740 1 -- 172.21.15.88:0/32329077 --> [v2:172.21.15.88:6835/2733458327,v1:172.21.15.88:6837/2733458327] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562192298c50 con 0x56219234c750 2024-09-16T01:40:25.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.264+0000 7f7824bc2640 1 --2- 172.21.15.88:0/32329077 >> [v2:172.21.15.88:6835/2733458327,v1:172.21.15.88:6837/2733458327] conn(0x56219234c750 0x562192351620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:25.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.264+0000 7f7824bc2640 1 --2- 172.21.15.88:0/32329077 >> [v2:172.21.15.88:6835/2733458327,v1:172.21.15.88:6837/2733458327] conn(0x56219234c750 0x562192351620 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a0244ee2df061f6 server_cookie=475bf11bacf6143e in_seq=0 out_seq=0 2024-09-16T01:40:25.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.264+0000 7f77fffff640 10 client.7524 ms_handle_connect on v2:172.21.15.88:6835/2733458327 2024-09-16T01:40:25.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.265+0000 7f77fffff640 1 -- 172.21.15.88:0/32329077 <== mds.0 v2:172.21.15.88:6835/2733458327 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562192298c50 con 0x56219234c750 2024-09-16T01:40:25.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.265+0000 7f77fffff640 10 client.7524 handle_command_reply: tid=0 2024-09-16T01:40:25.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.265+0000 7f782a3c3740 1 client.7524 shutdown 2024-09-16T01:40:25.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.265+0000 7f782a3c3740 1 -- 172.21.15.88:0/32329077 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621923292c0 msgr2=0x562192345ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:25.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.265+0000 7f782a3c3740 1 --2- 172.21.15.88:0/32329077 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621923292c0 0x562192345ae0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f781c00a500 tx=0x7f781c059d80 comp rx=0 tx=0).stop 2024-09-16T01:40:25.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.265+0000 7f782a3c3740 1 -- 172.21.15.88:0/32329077 shutdown_connections 2024-09-16T01:40:25.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.265+0000 7f782a3c3740 1 --2- 172.21.15.88:0/32329077 >> [v2:172.21.15.88:6835/2733458327,v1:172.21.15.88:6837/2733458327] conn(0x56219234c750 0x562192351620 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.265+0000 7f782a3c3740 1 --2- 172.21.15.88:0/32329077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562192349bf0 0x562192353080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.265+0000 7f782a3c3740 1 --2- 172.21.15.88:0/32329077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56219232aa50 0x5621923496b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.265+0000 7f77fffff640 0 client.7524 ms_handle_reset on v2:172.21.15.88:6835/2733458327 2024-09-16T01:40:25.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.265+0000 7f782a3c3740 1 --2- 172.21.15.88:0/32329077 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5621923292c0 0x562192345ae0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.265+0000 7f782a3c3740 1 -- 172.21.15.88:0/32329077 >> 172.21.15.88:0/32329077 conn(0x562192326a80 msgr2=0x56219236d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:25.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.265+0000 7f782a3c3740 1 -- 172.21.15.88:0/32329077 shutdown_connections 2024-09-16T01:40:25.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.265+0000 7f782a3c3740 1 -- 172.21.15.88:0/32329077 wait complete. 2024-09-16T01:40:25.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.265+0000 7f782a3c3740 20 client.7524 trim_cache size 0 max 16384 2024-09-16T01:40:25.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.283+0000 7f782a3c3740 1 Processor -- start 2024-09-16T01:40:25.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.283+0000 7f782a3c3740 1 -- start start 2024-09-16T01:40:25.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.284+0000 7f782a3c3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562192326a80 0x56219234b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.284+0000 7f782a3c3740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5621923292c0 0x56219236afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.284+0000 7f782a3c3740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56219232aa50 0x56219236c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.284+0000 7f782a3c3740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f78201ffd50 con 0x56219232aa50 2024-09-16T01:40:25.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.284+0000 7f782a3c3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7820157770 con 0x562192326a80 2024-09-16T01:40:25.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.284+0000 7f782a3c3740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f78200ca5c0 con 0x5621923292c0 2024-09-16T01:40:25.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.284+0000 7f7825bc4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56219232aa50 0x56219236c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:25.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.284+0000 7f7825bc4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56219232aa50 0x56219236c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38534/0 (socket says 172.21.15.88:38534) 2024-09-16T01:40:25.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.284+0000 7f7825bc4640 1 -- 172.21.15.88:0/1809051261 learned_addr learned my addr 172.21.15.88:0/1809051261 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:25.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.284+0000 7f7825bc4640 1 -- 172.21.15.88:0/1809051261 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5621923292c0 msgr2=0x56219236afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:25.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.284+0000 7f7825bc4640 1 --2- 172.21.15.88:0/1809051261 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5621923292c0 0x56219236afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.284+0000 7f7825bc4640 1 -- 172.21.15.88:0/1809051261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562192326a80 msgr2=0x56219234b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:25.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.284+0000 7f7825bc4640 1 --2- 172.21.15.88:0/1809051261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562192326a80 0x56219234b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.284+0000 7f7825bc4640 1 -- 172.21.15.88:0/1809051261 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562192329a10 con 0x56219232aa50 2024-09-16T01:40:25.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.284+0000 7f7825bc4640 1 --2- 172.21.15.88:0/1809051261 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56219232aa50 0x56219236c1a0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f781800d210 tx=0x7f7818034000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f1296781e60a0dd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:25.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.284+0000 7f780a7fc640 1 -- 172.21.15.88:0/1809051261 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7818043040 con 0x56219232aa50 2024-09-16T01:40:25.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.284+0000 7f780a7fc640 1 -- 172.21.15.88:0/1809051261 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f781804c070 con 0x56219232aa50 2024-09-16T01:40:25.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.284+0000 7f780a7fc640 1 -- 172.21.15.88:0/1809051261 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781803b040 con 0x56219232aa50 2024-09-16T01:40:25.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.285+0000 7f782a3c3740 1 -- 172.21.15.88:0/1809051261 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56219232aa50 msgr2=0x56219236c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:25.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.285+0000 7f782a3c3740 1 --2- 172.21.15.88:0/1809051261 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56219232aa50 0x56219236c1a0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f781800d210 tx=0x7f7818034000 comp rx=0 tx=0).stop 2024-09-16T01:40:25.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.285+0000 7f782a3c3740 1 -- 172.21.15.88:0/1809051261 shutdown_connections 2024-09-16T01:40:25.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.285+0000 7f782a3c3740 1 --2- 172.21.15.88:0/1809051261 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56219232aa50 0x56219236c1a0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.285+0000 7f782a3c3740 1 --2- 172.21.15.88:0/1809051261 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5621923292c0 0x56219236afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.285+0000 7f782a3c3740 1 --2- 172.21.15.88:0/1809051261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562192326a80 0x56219234b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.285+0000 7f782a3c3740 1 -- 172.21.15.88:0/1809051261 >> 172.21.15.88:0/1809051261 conn(0x56219236d870 msgr2=0x56219236de60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:25.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.285+0000 7f782a3c3740 1 -- 172.21.15.88:0/1809051261 shutdown_connections 2024-09-16T01:40:25.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.285+0000 7f782a3c3740 1 -- 172.21.15.88:0/1809051261 wait complete. 2024-09-16T01:40:25.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.285+0000 7f782a3c3740 1 -- start start 2024-09-16T01:40:25.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.286+0000 7f782a3c3740 1 Processor -- start 2024-09-16T01:40:25.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.286+0000 7f782a3c3740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562192326a80 0x5621923ef900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.286+0000 7f782a3c3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621923292c0 0x56219236afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.286+0000 7f782a3c3740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56219232aa50 0x56219236b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.286+0000 7f782a3c3740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56219234c5d0 con 0x562192326a80 2024-09-16T01:40:25.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.286+0000 7f782a3c3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56219234a7a0 con 0x5621923292c0 2024-09-16T01:40:25.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.286+0000 7f782a3c3740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56219234cbf0 con 0x56219232aa50 2024-09-16T01:40:25.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.286+0000 7f782a3c3740 10 client.0 _ll_register_callbacks cb 0x7f781c00a430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:40:25.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.287+0000 7f78253c3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562192326a80 0x5621923ef900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:25.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.287+0000 7f78253c3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562192326a80 0x5621923ef900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38550/0 (socket says 172.21.15.88:38550) 2024-09-16T01:40:25.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.287+0000 7f78253c3640 1 -- 172.21.15.88:0/910587836 learned_addr learned my addr 172.21.15.88:0/910587836 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:25.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.287+0000 7f7824bc2640 1 --2- 172.21.15.88:0/910587836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621923292c0 0x56219236afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:25.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.287+0000 7f7825bc4640 1 --2- 172.21.15.88:0/910587836 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56219232aa50 0x56219236b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:25.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.287+0000 7f78253c3640 1 -- 172.21.15.88:0/910587836 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56219232aa50 msgr2=0x56219236b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:25.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.287+0000 7f78253c3640 1 --2- 172.21.15.88:0/910587836 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56219232aa50 0x56219236b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.287+0000 7f78253c3640 1 -- 172.21.15.88:0/910587836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621923292c0 msgr2=0x56219236afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:25.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.287+0000 7f78253c3640 1 --2- 172.21.15.88:0/910587836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621923292c0 0x56219236afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.287+0000 7f78253c3640 1 -- 172.21.15.88:0/910587836 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56219236a150 con 0x562192326a80 2024-09-16T01:40:25.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.287+0000 7f7824bc2640 1 --2- 172.21.15.88:0/910587836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621923292c0 0x56219236afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:25.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.287+0000 7f78253c3640 1 --2- 172.21.15.88:0/910587836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562192326a80 0x5621923ef900 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f781c0577c0 tx=0x7f781c0045f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcf9e695013db51a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:25.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.287+0000 7f77fffff640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:40:25.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.287+0000 7f77fffff640 1 -- 172.21.15.88:0/910587836 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781c01e040 con 0x562192326a80 2024-09-16T01:40:25.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.287+0000 7f77fffff640 1 -- 172.21.15.88:0/910587836 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f781c08b0c0 con 0x562192326a80 2024-09-16T01:40:25.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.287+0000 7f782a3c3740 1 -- 172.21.15.88:0/910587836 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f781804c070 con 0x562192326a80 2024-09-16T01:40:25.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.287+0000 7f77fffff640 1 -- 172.21.15.88:0/910587836 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) ==== 24+0+0 (secure 0 0 0) 0x7f781c07d050 con 0x562192326a80 2024-09-16T01:40:25.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.287+0000 7f782a3c3740 10 client.7530 fetch_fsmap learned FSMap version 79 2024-09-16T01:40:25.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.287+0000 7f782a3c3740 1 -- 172.21.15.88:0/910587836 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=2+}) -- 0x5621923efea0 con 0x562192326a80 2024-09-16T01:40:25.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.287+0000 7f77fffff640 1 -- 172.21.15.88:0/910587836 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f781804c070 con 0x562192326a80 2024-09-16T01:40:25.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.287+0000 7f77fffff640 1 -- 172.21.15.88:0/910587836 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 79) ==== 3502+0+0 (secure 0 0 0) 0x7f781c078310 con 0x562192326a80 2024-09-16T01:40:25.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.287+0000 7f782a3c3740 10 client.7530 fetch_fsmap finished waiting for FSMap version 79 2024-09-16T01:40:25.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.288+0000 7f782a3c3740 10 client.7530 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T01:40:25.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.288+0000 7f782a3c3740 20 client.7530 populate_metadata read hostname 'smithi088' 2024-09-16T01:40:25.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.288+0000 7f782a3c3740 1 --2- 172.21.15.88:0/910587836 >> [v2:172.21.15.88:6835/2733458327,v1:172.21.15.88:6837/2733458327] conn(0x562192349bf0 0x5621923807f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.288+0000 7f782a3c3740 4 client.7530 mds_command: new command op to 7026 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:40:25.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.288+0000 7f782a3c3740 1 -- 172.21.15.88:0/910587836 --> [v2:172.21.15.88:6835/2733458327,v1:172.21.15.88:6837/2733458327] -- command(tid 0: {"prefix": "status"}) -- 0x562192292fa0 con 0x562192349bf0 2024-09-16T01:40:25.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.288+0000 7f7824bc2640 1 --2- 172.21.15.88:0/910587836 >> [v2:172.21.15.88:6835/2733458327,v1:172.21.15.88:6837/2733458327] conn(0x562192349bf0 0x5621923807f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:25.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.288+0000 7f77fffff640 10 client.7530 ms_handle_connect on v2:172.21.15.88:6835/2733458327 2024-09-16T01:40:25.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.288+0000 7f7824bc2640 1 --2- 172.21.15.88:0/910587836 >> [v2:172.21.15.88:6835/2733458327,v1:172.21.15.88:6837/2733458327] conn(0x562192349bf0 0x5621923807f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f9bbc0cb440195e server_cookie=e4f10ea0082dd739 in_seq=0 out_seq=0 2024-09-16T01:40:25.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.288+0000 7f77fffff640 1 -- 172.21.15.88:0/910587836 <== mds.0 v2:172.21.15.88:6835/2733458327 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x562192292fa0 con 0x562192349bf0 2024-09-16T01:40:25.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.288+0000 7f77fffff640 10 client.7530 handle_command_reply: tid=0 2024-09-16T01:40:25.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.288+0000 7f782a3c3740 1 client.7530 shutdown 2024-09-16T01:40:25.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.288+0000 7f782a3c3740 1 -- 172.21.15.88:0/910587836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562192326a80 msgr2=0x5621923ef900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:25.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.288+0000 7f782a3c3740 1 --2- 172.21.15.88:0/910587836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562192326a80 0x5621923ef900 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f781c0577c0 tx=0x7f781c0045f0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.288+0000 7f782a3c3740 1 -- 172.21.15.88:0/910587836 shutdown_connections 2024-09-16T01:40:25.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.288+0000 7f782a3c3740 1 --2- 172.21.15.88:0/910587836 >> [v2:172.21.15.88:6835/2733458327,v1:172.21.15.88:6837/2733458327] conn(0x562192349bf0 0x5621923807f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.288+0000 7f782a3c3740 1 --2- 172.21.15.88:0/910587836 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56219232aa50 0x56219236b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.288+0000 7f77fffff640 0 client.7530 ms_handle_reset on v2:172.21.15.88:6835/2733458327 2024-09-16T01:40:25.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.288+0000 7f782a3c3740 1 --2- 172.21.15.88:0/910587836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5621923292c0 0x56219236afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.288+0000 7f782a3c3740 1 --2- 172.21.15.88:0/910587836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562192326a80 0x5621923ef900 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.288+0000 7f782a3c3740 1 -- 172.21.15.88:0/910587836 >> 172.21.15.88:0/910587836 conn(0x56219236d870 msgr2=0x562192349160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:25.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.288+0000 7f782a3c3740 1 -- 172.21.15.88:0/910587836 shutdown_connections 2024-09-16T01:40:25.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.288+0000 7f782a3c3740 1 -- 172.21.15.88:0/910587836 wait complete. 2024-09-16T01:40:25.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.289+0000 7f782a3c3740 20 client.7530 trim_cache size 0 max 16384 2024-09-16T01:40:25.297 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:40:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "5ace2c91-46f9-482b-8a89-731fbdc171bf", 2024-09-16T01:40:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2024-09-16T01:40:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 7026, 2024-09-16T01:40:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2024-09-16T01:40:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2024-09-16T01:40:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2024-09-16T01:40:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 6.6175906400000004, 2024-09-16T01:40:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 79, 2024-09-16T01:40:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 86, 2024-09-16T01:40:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 86, 2024-09-16T01:40:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 13.724806487 2024-09-16T01:40:25.298 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:40:25.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.289+0000 7f7827627640 1 -- 172.21.15.88:0/2772631796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f780406c2b0 msgr2=0x7f780406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:25.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.289+0000 7f7827627640 1 --2- 172.21.15.88:0/2772631796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f780406c2b0 0x7f780406e770 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f78200b31b0 tx=0x7f7810008040 comp rx=0 tx=0).stop 2024-09-16T01:40:25.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.289+0000 7f7827627640 1 -- 172.21.15.88:0/2772631796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78200cc360 msgr2=0x7f782014a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:25.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.289+0000 7f7827627640 1 --2- 172.21.15.88:0/2772631796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78200cc360 0x7f782014a150 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f781c00c7f0 tx=0x7f781c00ccb0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.289+0000 7f7827627640 1 -- 172.21.15.88:0/2772631796 shutdown_connections 2024-09-16T01:40:25.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.289+0000 7f7827627640 1 --2- 172.21.15.88:0/2772631796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f780406c2b0 0x7f780406e770 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.289+0000 7f7827627640 1 --2- 172.21.15.88:0/2772631796 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f782015ad10 0x7f78201f5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.289+0000 7f7827627640 1 --2- 172.21.15.88:0/2772631796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78200cca40 0x7f7820153310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.289+0000 7f7827627640 1 --2- 172.21.15.88:0/2772631796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78200cc360 0x7f782014a150 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.289+0000 7f7827627640 1 -- 172.21.15.88:0/2772631796 >> 172.21.15.88:0/2772631796 conn(0x7f782006d0f0 msgr2=0x7f78200b3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:25.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.289+0000 7f7827627640 1 -- 172.21.15.88:0/2772631796 shutdown_connections 2024-09-16T01:40:25.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.289+0000 7f7827627640 1 -- 172.21.15.88:0/2772631796 wait complete. 2024-09-16T01:40:25.301 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-09-16T01:40:25.301 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Delete 10000000000.00000000', proceeding to ls 2024-09-16T01:40:25.301 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:40:25.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.397+0000 7eff6825e640 1 Processor -- start 2024-09-16T01:40:25.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.398+0000 7eff6825e640 1 -- start start 2024-09-16T01:40:25.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.398+0000 7eff6825e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7eff6015a9c0 0x7eff6015e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.398+0000 7eff6825e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7eff600b29a0 0x7eff600cde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.399+0000 7eff6825e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff600b6830 con 0x7eff600b5be0 2024-09-16T01:40:25.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.399+0000 7eff6825e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7eff6015f410 con 0x7eff600b29a0 2024-09-16T01:40:25.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.399+0000 7eff6825e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7eff600cd0b0 con 0x7eff6015a9c0 2024-09-16T01:40:25.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.399+0000 7eff667fb640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7eff600b29a0 0x7eff600cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:25.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.399+0000 7eff667fb640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7eff600b29a0 0x7eff600cde70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38552/0 (socket says 172.21.15.88:38552) 2024-09-16T01:40:25.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.399+0000 7eff667fb640 1 -- 172.21.15.88:0/1783235719 learned_addr learned my addr 172.21.15.88:0/1783235719 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:25.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.399+0000 7eff65ffa640 1 --2- 172.21.15.88:0/1783235719 >> v2:172.21.15.154:3300/0 conn(0x7eff6015a9c0 0x7eff6015e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:25.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.399+0000 7eff667fb640 1 -- 172.21.15.88:0/1783235719 >> v2:172.21.15.154:3300/0 conn(0x7eff6015a9c0 msgr2=0x7eff6015e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:25.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.399+0000 7eff667fb640 1 --2- 172.21.15.88:0/1783235719 >> v2:172.21.15.154:3300/0 conn(0x7eff6015a9c0 0x7eff6015e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.399+0000 7eff667fb640 1 -- 172.21.15.88:0/1783235719 >> v1:172.21.15.154:6789/0 conn(0x7eff600b5be0 legacy=0x7eff6015eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:25.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.399+0000 7eff667fb640 1 -- 172.21.15.88:0/1783235719 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff600cd390 con 0x7eff600b29a0 2024-09-16T01:40:25.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.400+0000 7eff667fb640 1 --2- 172.21.15.88:0/1783235719 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff600b29a0 0x7eff600cde70 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7eff6006f1a0 tx=0x7eff5c02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4199cb80a9c2ab4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:25.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.400+0000 7eff64ff8640 1 -- 172.21.15.88:0/1783235719 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c036070 con 0x7eff600b29a0 2024-09-16T01:40:25.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.400+0000 7eff64ff8640 1 -- 172.21.15.88:0/1783235719 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff5c031600 con 0x7eff600b29a0 2024-09-16T01:40:25.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.400+0000 7eff64ff8640 1 -- 172.21.15.88:0/1783235719 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c0319d0 con 0x7eff600b29a0 2024-09-16T01:40:25.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.400+0000 7eff6825e640 1 -- 172.21.15.88:0/1783235719 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff600b29a0 msgr2=0x7eff600cde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:25.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.400+0000 7eff6825e640 1 --2- 172.21.15.88:0/1783235719 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff600b29a0 0x7eff600cde70 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7eff6006f1a0 tx=0x7eff5c02a6f0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.400+0000 7eff6825e640 1 -- 172.21.15.88:0/1783235719 shutdown_connections 2024-09-16T01:40:25.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.400+0000 7eff6825e640 1 --2- 172.21.15.88:0/1783235719 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff600b29a0 0x7eff600cde70 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.400+0000 7eff6825e640 1 --2- 172.21.15.88:0/1783235719 >> v2:172.21.15.154:3300/0 conn(0x7eff6015a9c0 0x7eff6015e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.400+0000 7eff6825e640 1 -- 172.21.15.88:0/1783235719 >> 172.21.15.88:0/1783235719 conn(0x7eff6006bc10 msgr2=0x7eff6006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:25.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.401+0000 7eff6825e640 1 -- 172.21.15.88:0/1783235719 shutdown_connections 2024-09-16T01:40:25.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.401+0000 7eff6825e640 1 -- 172.21.15.88:0/1783235719 wait complete. 2024-09-16T01:40:25.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.402+0000 7eff6825e640 1 Processor -- start 2024-09-16T01:40:25.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.402+0000 7eff6825e640 1 -- start start 2024-09-16T01:40:25.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.402+0000 7eff6825e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff600b29a0 0x7eff6014f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.402+0000 7eff6825e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff600b5be0 0x7eff60152150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.402+0000 7eff6825e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff6015a9c0 0x7eff600cb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.403+0000 7eff6825e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7eff6015ada0 con 0x7eff600b29a0 2024-09-16T01:40:25.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.403+0000 7eff6825e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7eff600cd950 con 0x7eff6015a9c0 2024-09-16T01:40:25.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.403+0000 7eff6825e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7eff6006de40 con 0x7eff600b5be0 2024-09-16T01:40:25.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.403+0000 7eff65ffa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff600b29a0 0x7eff6014f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:25.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.403+0000 7eff65ffa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff600b29a0 0x7eff6014f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38564/0 (socket says 172.21.15.88:38564) 2024-09-16T01:40:25.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.403+0000 7eff65ffa640 1 -- 172.21.15.88:0/1536416065 learned_addr learned my addr 172.21.15.88:0/1536416065 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:25.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.403+0000 7eff667fb640 1 --2- 172.21.15.88:0/1536416065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff6015a9c0 0x7eff600cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:25.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.403+0000 7eff657f9640 1 --2- 172.21.15.88:0/1536416065 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff600b5be0 0x7eff60152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:25.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.403+0000 7eff65ffa640 1 -- 172.21.15.88:0/1536416065 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff600b5be0 msgr2=0x7eff60152150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:25.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.403+0000 7eff65ffa640 1 --2- 172.21.15.88:0/1536416065 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff600b5be0 0x7eff60152150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.403+0000 7eff65ffa640 1 -- 172.21.15.88:0/1536416065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff6015a9c0 msgr2=0x7eff600cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:25.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.403+0000 7eff65ffa640 1 --2- 172.21.15.88:0/1536416065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff6015a9c0 0x7eff600cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.403+0000 7eff65ffa640 1 -- 172.21.15.88:0/1536416065 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff6014b640 con 0x7eff600b29a0 2024-09-16T01:40:25.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.403+0000 7eff657f9640 1 --2- 172.21.15.88:0/1536416065 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff600b5be0 0x7eff60152150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:25.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.403+0000 7eff65ffa640 1 --2- 172.21.15.88:0/1536416065 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff600b29a0 0x7eff6014f4b0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7eff5800c7f0 tx=0x7eff5800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:25.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.403+0000 7eff567fc640 1 -- 172.21.15.88:0/1536416065 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff58015070 con 0x7eff600b29a0 2024-09-16T01:40:25.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.403+0000 7eff567fc640 1 -- 172.21.15.88:0/1536416065 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff58004220 con 0x7eff600b29a0 2024-09-16T01:40:25.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.403+0000 7eff6825e640 1 -- 172.21.15.88:0/1536416065 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff600ca4d0 con 0x7eff600b29a0 2024-09-16T01:40:25.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.403+0000 7eff567fc640 1 -- 172.21.15.88:0/1536416065 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff58007620 con 0x7eff600b29a0 2024-09-16T01:40:25.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.403+0000 7eff6825e640 1 -- 172.21.15.88:0/1536416065 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff600caa90 con 0x7eff600b29a0 2024-09-16T01:40:25.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.404+0000 7eff6825e640 1 -- 172.21.15.88:0/1536416065 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff6006e1d0 con 0x7eff600b29a0 2024-09-16T01:40:25.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.405+0000 7eff567fc640 1 -- 172.21.15.88:0/1536416065 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7eff580077c0 con 0x7eff600b29a0 2024-09-16T01:40:25.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.405+0000 7eff567fc640 1 --2- 172.21.15.88:0/1536416065 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff4406c2b0 0x7eff4406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:25.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.405+0000 7eff567fc640 1 -- 172.21.15.88:0/1536416065 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7eff5808b540 con 0x7eff600b29a0 2024-09-16T01:40:25.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.405+0000 7eff657f9640 1 --2- 172.21.15.88:0/1536416065 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff4406c2b0 0x7eff4406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:25.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.406+0000 7eff657f9640 1 --2- 172.21.15.88:0/1536416065 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff4406c2b0 0x7eff4406e770 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7eff60153d20 tx=0x7eff50009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:25.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.410+0000 7eff567fc640 1 -- 172.21.15.88:0/1536416065 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7eff58059600 con 0x7eff600b29a0 2024-09-16T01:40:25.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.611+0000 7eff6825e640 1 -- 172.21.15.88:0/1536416065 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7eff600752d0 con 0x7eff600b29a0 2024-09-16T01:40:25.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.612+0000 7eff567fc640 1 -- 172.21.15.88:0/1536416065 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v79) ==== 53+0+73 (secure 0 0 0) 0x7eff5805d900 con 0x7eff600b29a0 2024-09-16T01:40:25.613 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:40:25.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.613+0000 7eff6825e640 1 -- 172.21.15.88:0/1536416065 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff4406c2b0 msgr2=0x7eff4406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:25.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.613+0000 7eff6825e640 1 --2- 172.21.15.88:0/1536416065 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff4406c2b0 0x7eff4406e770 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7eff60153d20 tx=0x7eff50009040 comp rx=0 tx=0).stop 2024-09-16T01:40:25.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.614+0000 7eff6825e640 1 -- 172.21.15.88:0/1536416065 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff600b29a0 msgr2=0x7eff6014f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:25.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.614+0000 7eff6825e640 1 --2- 172.21.15.88:0/1536416065 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff600b29a0 0x7eff6014f4b0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7eff5800c7f0 tx=0x7eff5800ccb0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.614+0000 7eff6825e640 1 -- 172.21.15.88:0/1536416065 shutdown_connections 2024-09-16T01:40:25.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.614+0000 7eff6825e640 1 --2- 172.21.15.88:0/1536416065 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff4406c2b0 0x7eff4406e770 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.614+0000 7eff6825e640 1 --2- 172.21.15.88:0/1536416065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff6015a9c0 0x7eff600cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.614+0000 7eff6825e640 1 --2- 172.21.15.88:0/1536416065 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff600b5be0 0x7eff60152150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.614+0000 7eff6825e640 1 --2- 172.21.15.88:0/1536416065 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff600b29a0 0x7eff6014f4b0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:25.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.614+0000 7eff6825e640 1 -- 172.21.15.88:0/1536416065 >> 172.21.15.88:0/1536416065 conn(0x7eff6006bc10 msgr2=0x7eff60071090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:25.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.614+0000 7eff6825e640 1 -- 172.21.15.88:0/1536416065 shutdown_connections 2024-09-16T01:40:25.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25.614+0000 7eff6825e640 1 -- 172.21.15.88:0/1536416065 wait complete. 2024-09-16T01:40:25.626 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-16T01:40:25.626 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:40:25.626 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2024-09-16T01:40:25.626 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T01:40:25.626 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T01:40:25.626 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-16T01:40:25.626 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-16T01:40:25.626 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:40:25.626 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:40:25.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:25.627 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2024-09-16T01:40:25.642 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:40:25.643 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:40:25.643 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:40:25.643 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:40:25.643 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-09-16T01:40:25.643 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:40:25.643 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:40:25.643 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:40:25.643 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2024-09-16T01:40:25.643 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:40:25.643 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:40:25.644 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2024-09-16T01:40:25.644 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:40:25.644 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2024-09-16T01:40:25.644 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:40:25.644 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2024-09-16T01:40:25.644 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:40:25.644 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42398sec preferred_lft 42398sec 2024-09-16T01:40:25.644 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link noprefixroute 2024-09-16T01:40:25.644 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:40:25.644 INFO:teuthology.orchestra.run.smithi088.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:40:25.644 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff 2024-09-16T01:40:25.644 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T01:40:25.644 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:40:25.645 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::e486:6eff:fe81:a1c/64 scope link 2024-09-16T01:40:25.645 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:40:25.645 INFO:teuthology.orchestra.run.smithi088.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:40:25.645 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 12:6f:29:be:95:48 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:40:25.645 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::106f:29ff:febe:9548/64 scope link 2024-09-16T01:40:25.645 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:40:25.645 INFO:teuthology.orchestra.run.smithi088.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:40:25.645 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T01:40:25.645 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::c63:b4ff:feb3:ad9e/64 scope link 2024-09-16T01:40:25.645 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:40:25.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:25.646 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:40:25.707 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-09-16T01:40:25.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:25.708 DEBUG:teuthology.orchestra.run.smithi088:> route 2024-09-16T01:40:25.765 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2024-09-16T01:40:25.765 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:40:25.765 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:40:25.765 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:40:25.765 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:40:25.765 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:40:25.765 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:40:25.765 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:40:25.766 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:40:25.766 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:40:25.766 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:40:25.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:40:25.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:40:25.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:25.890 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:40:25.905 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T01:40:25.905 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T01:40:25.905 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:40:25.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:25.905 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2024-09-16T01:40:25.970 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:40:25.970 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T01:40:26.025 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:40:26.025 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:40:26.026 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:40:26.081 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-16T01:40:26.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:26.081 DEBUG:teuthology.orchestra.run.smithi088:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:40:26.164 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:40:26.164 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:40:26.164 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:40:26.173 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:40:26.173 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:40:26.173 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:40:26.173 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:40:26.173 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:40:26.178 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:40:26.178 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:40:26.178 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:40:26.179 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:40:26.179 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:40:26.179 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790" 2024-09-16T01:40:26.179 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2024-09-16T01:40:26.179 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5ace2c91-46f9-482b-8a89-731fbdc171bf" will pass to kernel 2024-09-16T01:40:26.179 INFO:teuthology.orchestra.run.smithi088.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-16T01:40:26.180 INFO:teuthology.orchestra.run.smithi088.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:40:26.180 INFO:teuthology.orchestra.run.smithi088.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:40:26.180 INFO:teuthology.orchestra.run.smithi088.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:40:26.180 INFO:teuthology.orchestra.run.smithi088.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:40:26.181 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:40:26.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:26.182 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:40:26.247 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:40:26.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:40:26.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:40:26.342 INFO:teuthology.orchestra.run.smithi088.stdout:client6002 2024-09-16T01:40:26.343 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:40:26.343 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:40:26.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:26.359 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R .) 2024-09-16T01:40:26.401 INFO:teuthology.orchestra.run:waiting for 20 2024-09-16T01:40:26.415 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:40:26.413+0000 7f9845fd4640 -1 log_channel(cluster) log [ERR] : dir 0x10000000000 object missing on disk; some files may be lost (/subdir) 2024-09-16T01:40:26.416 INFO:teuthology.orchestra.run.smithi088.stdout:.: 2024-09-16T01:40:26.416 INFO:teuthology.orchestra.run.smithi088.stdout:subdir 2024-09-16T01:40:26.416 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:26.416 INFO:teuthology.orchestra.run.smithi088.stdout:./subdir: 2024-09-16T01:40:26.416 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 10000000000.00000000' did not caused DAMAGED state 2024-09-16T01:40:26.416 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 600.00000000' 2024-09-16T01:40:26.416 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:40:26.417 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:40:26.473 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:40:26.473 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:40:26.529 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:40:26.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.665+0000 7f913b152640 1 Processor -- start 2024-09-16T01:40:26.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.666+0000 7f913b152640 1 -- start start 2024-09-16T01:40:26.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.666+0000 7f913b152640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f913415e7a0 0x7f913406caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:26.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.666+0000 7f913b152640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f91340cf1f0 0x7f91340b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:26.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.666+0000 7f913b152640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91340cf860 con 0x7f91340b6720 2024-09-16T01:40:26.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.666+0000 7f913b152640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f91340d2ca0 con 0x7f91340cf1f0 2024-09-16T01:40:26.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.666+0000 7f913b152640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f91340cfb40 con 0x7f913415e7a0 2024-09-16T01:40:26.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.666+0000 7f9133fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f91340cf1f0 0x7f91340b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:26.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.666+0000 7f9133fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f91340cf1f0 0x7f91340b61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38574/0 (socket says 172.21.15.88:38574) 2024-09-16T01:40:26.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.666+0000 7f9133fff640 1 -- 172.21.15.88:0/291879812 learned_addr learned my addr 172.21.15.88:0/291879812 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:26.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.667+0000 7f9138eee640 1 --2- 172.21.15.88:0/291879812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f913415e7a0 0x7f913406caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:26.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.667+0000 7f9133fff640 1 -- 172.21.15.88:0/291879812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f913415e7a0 msgr2=0x7f913406caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:26.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.667+0000 7f9133fff640 1 --2- 172.21.15.88:0/291879812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f913415e7a0 0x7f913406caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:26.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.667+0000 7f9133fff640 1 -- 172.21.15.88:0/291879812 >> v1:172.21.15.154:6789/0 conn(0x7f91340b6720 legacy=0x7f91341623c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:26.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.667+0000 7f9133fff640 1 -- 172.21.15.88:0/291879812 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91340cfe20 con 0x7f91340cf1f0 2024-09-16T01:40:26.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.667+0000 7f9133fff640 1 --2- 172.21.15.88:0/291879812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91340cf1f0 0x7f91340b61e0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f9128009e50 tx=0x7f912802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb7e04452ea0a0b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:26.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.668+0000 7f91337fe640 1 -- 172.21.15.88:0/291879812 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9128036070 con 0x7f91340cf1f0 2024-09-16T01:40:26.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.668+0000 7f91337fe640 1 -- 172.21.15.88:0/291879812 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9128031600 con 0x7f91340cf1f0 2024-09-16T01:40:26.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.668+0000 7f91337fe640 1 -- 172.21.15.88:0/291879812 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91280319d0 con 0x7f91340cf1f0 2024-09-16T01:40:26.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.668+0000 7f913b152640 1 -- 172.21.15.88:0/291879812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91340cf1f0 msgr2=0x7f91340b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:26.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.668+0000 7f913b152640 1 --2- 172.21.15.88:0/291879812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91340cf1f0 0x7f91340b61e0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f9128009e50 tx=0x7f912802a680 comp rx=0 tx=0).stop 2024-09-16T01:40:26.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.668+0000 7f913b152640 1 -- 172.21.15.88:0/291879812 shutdown_connections 2024-09-16T01:40:26.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.668+0000 7f913b152640 1 --2- 172.21.15.88:0/291879812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91340cf1f0 0x7f91340b61e0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:26.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.668+0000 7f913b152640 1 --2- 172.21.15.88:0/291879812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f913415e7a0 0x7f913406caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:26.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.668+0000 7f913b152640 1 -- 172.21.15.88:0/291879812 >> 172.21.15.88:0/291879812 conn(0x7f91340d11e0 msgr2=0x7f91340d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:26.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.668+0000 7f913b152640 1 -- 172.21.15.88:0/291879812 shutdown_connections 2024-09-16T01:40:26.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.668+0000 7f913b152640 1 -- 172.21.15.88:0/291879812 wait complete. 2024-09-16T01:40:26.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.669+0000 7f913b152640 1 Processor -- start 2024-09-16T01:40:26.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.669+0000 7f913b152640 1 -- start start 2024-09-16T01:40:26.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.670+0000 7f913b152640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91340b6720 0x7f913414caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:26.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.670+0000 7f913b152640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91340cf1f0 0x7f913415c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:26.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.670+0000 7f913b152640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f913415e7a0 0x7f913415c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:26.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.670+0000 7f913b152640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f913406cfe0 con 0x7f91340cf1f0 2024-09-16T01:40:26.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.670+0000 7f913b152640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f91340b7a00 con 0x7f913415e7a0 2024-09-16T01:40:26.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.670+0000 7f913b152640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f91340d4420 con 0x7f91340b6720 2024-09-16T01:40:26.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.671+0000 7f9133fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91340cf1f0 0x7f913415c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:26.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.671+0000 7f9133fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91340cf1f0 0x7f913415c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38582/0 (socket says 172.21.15.88:38582) 2024-09-16T01:40:26.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.671+0000 7f9133fff640 1 -- 172.21.15.88:0/1214652273 learned_addr learned my addr 172.21.15.88:0/1214652273 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:26.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.671+0000 7f9138eee640 1 --2- 172.21.15.88:0/1214652273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91340b6720 0x7f913414caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:26.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.671+0000 7f91396ef640 1 --2- 172.21.15.88:0/1214652273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f913415e7a0 0x7f913415c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:26.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.671+0000 7f9133fff640 1 -- 172.21.15.88:0/1214652273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91340b6720 msgr2=0x7f913414caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:26.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.671+0000 7f9133fff640 1 --2- 172.21.15.88:0/1214652273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91340b6720 0x7f913414caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:26.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.671+0000 7f9133fff640 1 -- 172.21.15.88:0/1214652273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f913415e7a0 msgr2=0x7f913415c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:26.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.671+0000 7f9133fff640 1 --2- 172.21.15.88:0/1214652273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f913415e7a0 0x7f913415c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:26.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.671+0000 7f9133fff640 1 -- 172.21.15.88:0/1214652273 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f913414d5d0 con 0x7f91340cf1f0 2024-09-16T01:40:26.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.671+0000 7f9133fff640 1 --2- 172.21.15.88:0/1214652273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91340cf1f0 0x7f913415c400 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f9128009e50 tx=0x7f9128004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:26.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.671+0000 7f91317fa640 1 -- 172.21.15.88:0/1214652273 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9128041020 con 0x7f91340cf1f0 2024-09-16T01:40:26.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.671+0000 7f91317fa640 1 -- 172.21.15.88:0/1214652273 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9128007820 con 0x7f91340cf1f0 2024-09-16T01:40:26.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.671+0000 7f913b152640 1 -- 172.21.15.88:0/1214652273 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91340ab5c0 con 0x7f91340cf1f0 2024-09-16T01:40:26.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.671+0000 7f91317fa640 1 -- 172.21.15.88:0/1214652273 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9128036040 con 0x7f91340cf1f0 2024-09-16T01:40:26.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.671+0000 7f913b152640 1 -- 172.21.15.88:0/1214652273 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9134161440 con 0x7f91340cf1f0 2024-09-16T01:40:26.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.672+0000 7f913b152640 1 -- 172.21.15.88:0/1214652273 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91340d2a80 con 0x7f91340cf1f0 2024-09-16T01:40:26.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.673+0000 7f91317fa640 1 -- 172.21.15.88:0/1214652273 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9128044020 con 0x7f91340cf1f0 2024-09-16T01:40:26.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.673+0000 7f91317fa640 1 --2- 172.21.15.88:0/1214652273 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f912006c2b0 0x7f912006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:26.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.674+0000 7f9138eee640 1 --2- 172.21.15.88:0/1214652273 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f912006c2b0 0x7f912006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:26.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.674+0000 7f91317fa640 1 -- 172.21.15.88:0/1214652273 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f91280ac0e0 con 0x7f91340cf1f0 2024-09-16T01:40:26.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.674+0000 7f9138eee640 1 --2- 172.21.15.88:0/1214652273 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f912006c2b0 0x7f912006e770 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9124006fd0 tx=0x7f9124008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:26.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.678+0000 7f91317fa640 1 -- 172.21.15.88:0/1214652273 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f912804b030 con 0x7f91340cf1f0 2024-09-16T01:40:26.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.855+0000 7f913b152640 1 -- 172.21.15.88:0/1214652273 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f91340b8800 con 0x7f91340cf1f0 2024-09-16T01:40:26.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.856+0000 7f91317fa640 1 -- 172.21.15.88:0/1214652273 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v86) ==== 80+0+122 (secure 0 0 0) 0x7f912807a230 con 0x7f91340cf1f0 2024-09-16T01:40:26.858 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/4162274767 2024-09-17T01:40:11.522091+0000 2024-09-16T01:40:26.858 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/4162274767 2024-09-17T01:40:11.522091+0000 2024-09-16T01:40:26.859 INFO:teuthology.orchestra.run.smithi088.stderr:listed 2 entries 2024-09-16T01:40:26.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.858+0000 7f913b152640 1 -- 172.21.15.88:0/1214652273 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f912006c2b0 msgr2=0x7f912006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:26.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.858+0000 7f913b152640 1 --2- 172.21.15.88:0/1214652273 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f912006c2b0 0x7f912006e770 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9124006fd0 tx=0x7f9124008040 comp rx=0 tx=0).stop 2024-09-16T01:40:26.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.858+0000 7f913b152640 1 -- 172.21.15.88:0/1214652273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91340cf1f0 msgr2=0x7f913415c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:26.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.858+0000 7f913b152640 1 --2- 172.21.15.88:0/1214652273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91340cf1f0 0x7f913415c400 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f9128009e50 tx=0x7f9128004a40 comp rx=0 tx=0).stop 2024-09-16T01:40:26.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.858+0000 7f913b152640 1 -- 172.21.15.88:0/1214652273 shutdown_connections 2024-09-16T01:40:26.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.858+0000 7f913b152640 1 --2- 172.21.15.88:0/1214652273 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f912006c2b0 0x7f912006e770 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:26.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.858+0000 7f913b152640 1 --2- 172.21.15.88:0/1214652273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f913415e7a0 0x7f913415c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:26.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.858+0000 7f913b152640 1 --2- 172.21.15.88:0/1214652273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91340cf1f0 0x7f913415c400 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:26.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.858+0000 7f913b152640 1 --2- 172.21.15.88:0/1214652273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91340b6720 0x7f913414caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:26.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.858+0000 7f913b152640 1 -- 172.21.15.88:0/1214652273 >> 172.21.15.88:0/1214652273 conn(0x7f91340d11e0 msgr2=0x7f91340d4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:26.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.858+0000 7f913b152640 1 -- 172.21.15.88:0/1214652273 shutdown_connections 2024-09-16T01:40:26.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:26.858+0000 7f913b152640 1 -- 172.21.15.88:0/1214652273 wait complete. 2024-09-16T01:40:26.871 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-16T01:40:26.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:26.872 DEBUG:teuthology.orchestra.run.smithi088:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-09-16T01:40:26.905 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:40:26.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:26.905 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:40:26.961 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:40:27.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.099+0000 7f120d311640 1 Processor -- start 2024-09-16T01:40:27.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.099+0000 7f120d311640 1 -- start start 2024-09-16T01:40:27.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.100+0000 7f120d311640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f120815e7a0 0x7f120806caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:27.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.100+0000 7f120d311640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12080cf1f0 0x7f12080b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:27.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.100+0000 7f120d311640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12080cf860 con 0x7f12080b6720 2024-09-16T01:40:27.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.100+0000 7f120d311640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f12080d2ca0 con 0x7f120815e7a0 2024-09-16T01:40:27.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.100+0000 7f120d311640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f12080cfb40 con 0x7f12080cf1f0 2024-09-16T01:40:27.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.100+0000 7f1206ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f120815e7a0 0x7f120806caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:27.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.100+0000 7f1206ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f120815e7a0 0x7f120806caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38590/0 (socket says 172.21.15.88:38590) 2024-09-16T01:40:27.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.100+0000 7f1206ffd640 1 -- 172.21.15.88:0/2294651023 learned_addr learned my addr 172.21.15.88:0/2294651023 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:27.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.100+0000 7f11fffff640 1 --2- 172.21.15.88:0/2294651023 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12080cf1f0 0x7f12080b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:27.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.100+0000 7f1206ffd640 1 -- 172.21.15.88:0/2294651023 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12080cf1f0 msgr2=0x7f12080b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:27.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.100+0000 7f1206ffd640 1 --2- 172.21.15.88:0/2294651023 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12080cf1f0 0x7f12080b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:27.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.100+0000 7f1206ffd640 1 -- 172.21.15.88:0/2294651023 >> v1:172.21.15.154:6789/0 conn(0x7f12080b6720 legacy=0x7f12081623c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:27.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.100+0000 7f1206ffd640 1 -- 172.21.15.88:0/2294651023 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12080cfe20 con 0x7f120815e7a0 2024-09-16T01:40:27.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.100+0000 7f11fffff640 1 --2- 172.21.15.88:0/2294651023 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12080cf1f0 0x7f12080b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:27.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.101+0000 7f1206ffd640 1 --2- 172.21.15.88:0/2294651023 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f120815e7a0 0x7f120806caa0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f11f8009870 tx=0x7f11f802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2c98a02afe3e064 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:27.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.101+0000 7f12067fc640 1 -- 172.21.15.88:0/2294651023 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f8037070 con 0x7f120815e7a0 2024-09-16T01:40:27.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.101+0000 7f12067fc640 1 -- 172.21.15.88:0/2294651023 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11f8032440 con 0x7f120815e7a0 2024-09-16T01:40:27.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.101+0000 7f12067fc640 1 -- 172.21.15.88:0/2294651023 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11f8032810 con 0x7f120815e7a0 2024-09-16T01:40:27.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.102+0000 7f120d311640 1 -- 172.21.15.88:0/2294651023 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f120815e7a0 msgr2=0x7f120806caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:27.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.102+0000 7f120d311640 1 --2- 172.21.15.88:0/2294651023 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f120815e7a0 0x7f120806caa0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f11f8009870 tx=0x7f11f802a180 comp rx=0 tx=0).stop 2024-09-16T01:40:27.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.102+0000 7f120d311640 1 -- 172.21.15.88:0/2294651023 shutdown_connections 2024-09-16T01:40:27.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.102+0000 7f120d311640 1 --2- 172.21.15.88:0/2294651023 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12080cf1f0 0x7f12080b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:27.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.102+0000 7f120d311640 1 --2- 172.21.15.88:0/2294651023 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f120815e7a0 0x7f120806caa0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:27.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.102+0000 7f120d311640 1 -- 172.21.15.88:0/2294651023 >> 172.21.15.88:0/2294651023 conn(0x7f12080d11e0 msgr2=0x7f12080d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:27.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.102+0000 7f120d311640 1 -- 172.21.15.88:0/2294651023 shutdown_connections 2024-09-16T01:40:27.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.102+0000 7f120d311640 1 -- 172.21.15.88:0/2294651023 wait complete. 2024-09-16T01:40:27.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.103+0000 7f120d311640 1 Processor -- start 2024-09-16T01:40:27.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.103+0000 7f120d311640 1 -- start start 2024-09-16T01:40:27.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.104+0000 7f120d311640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12080b6720 0x7f1208158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:27.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.104+0000 7f120d311640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12080cf1f0 0x7f120815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:27.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.104+0000 7f120d311640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f120815e7a0 0x7f120815f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:27.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.104+0000 7f120d311640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f120806cfe0 con 0x7f120815e7a0 2024-09-16T01:40:27.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.104+0000 7f120d311640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f12080b7a00 con 0x7f12080b6720 2024-09-16T01:40:27.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.104+0000 7f120d311640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f12080d4420 con 0x7f12080cf1f0 2024-09-16T01:40:27.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.104+0000 7f12077fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f120815e7a0 0x7f120815f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:27.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.104+0000 7f12077fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f120815e7a0 0x7f120815f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38596/0 (socket says 172.21.15.88:38596) 2024-09-16T01:40:27.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.104+0000 7f12077fe640 1 -- 172.21.15.88:0/531328378 learned_addr learned my addr 172.21.15.88:0/531328378 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:27.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.104+0000 7f1206ffd640 1 --2- 172.21.15.88:0/531328378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12080b6720 0x7f1208158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:27.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.104+0000 7f11fffff640 1 --2- 172.21.15.88:0/531328378 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12080cf1f0 0x7f120815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:27.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.104+0000 7f12077fe640 1 -- 172.21.15.88:0/531328378 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12080cf1f0 msgr2=0x7f120815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:27.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.104+0000 7f12077fe640 1 --2- 172.21.15.88:0/531328378 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12080cf1f0 0x7f120815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:27.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.104+0000 7f12077fe640 1 -- 172.21.15.88:0/531328378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12080b6720 msgr2=0x7f1208158e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:27.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.104+0000 7f12077fe640 1 --2- 172.21.15.88:0/531328378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12080b6720 0x7f1208158e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:27.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.104+0000 7f12077fe640 1 -- 172.21.15.88:0/531328378 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12080c7d70 con 0x7f120815e7a0 2024-09-16T01:40:27.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.104+0000 7f12077fe640 1 --2- 172.21.15.88:0/531328378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f120815e7a0 0x7f120815f2d0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f120000c7b0 tx=0x7f120000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:27.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.104+0000 7f11ff7fe640 1 -- 172.21.15.88:0/531328378 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1200004b60 con 0x7f120815e7a0 2024-09-16T01:40:27.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.104+0000 7f11ff7fe640 1 -- 172.21.15.88:0/531328378 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1200004d00 con 0x7f120815e7a0 2024-09-16T01:40:27.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.104+0000 7f120d311640 1 -- 172.21.15.88:0/531328378 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12080cace0 con 0x7f120815e7a0 2024-09-16T01:40:27.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.104+0000 7f11ff7fe640 1 -- 172.21.15.88:0/531328378 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120000aa90 con 0x7f120815e7a0 2024-09-16T01:40:27.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.104+0000 7f120d311640 1 -- 172.21.15.88:0/531328378 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1208148e40 con 0x7f120815e7a0 2024-09-16T01:40:27.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.105+0000 7f120d311640 1 -- 172.21.15.88:0/531328378 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f120815eb80 con 0x7f120815e7a0 2024-09-16T01:40:27.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.106+0000 7f11ff7fe640 1 -- 172.21.15.88:0/531328378 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f120000ac30 con 0x7f120815e7a0 2024-09-16T01:40:27.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.106+0000 7f11ff7fe640 1 --2- 172.21.15.88:0/531328378 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f11e806c2b0 0x7f11e806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:27.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.106+0000 7f1206ffd640 1 --2- 172.21.15.88:0/531328378 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f11e806c2b0 0x7f11e806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:27.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.106+0000 7f11ff7fe640 1 -- 172.21.15.88:0/531328378 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5795+0+0 (secure 0 0 0) 0x7f120008b500 con 0x7f120815e7a0 2024-09-16T01:40:27.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.107+0000 7f1206ffd640 1 --2- 172.21.15.88:0/531328378 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f11e806c2b0 0x7f11e806e770 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f11f8034000 tx=0x7f11f802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:27.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.110+0000 7f11ff7fe640 1 -- 172.21.15.88:0/531328378 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1200020070 con 0x7f120815e7a0 2024-09-16T01:40:27.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.312+0000 7f120d311640 1 -- 172.21.15.88:0/531328378 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f12080d2a80 con 0x7f120815e7a0 2024-09-16T01:40:27.317 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:40:27.313+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:40:27.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.314+0000 7f11ff7fe640 1 -- 172.21.15.88:0/531328378 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v80) ==== 161+0+0 (secure 0 0 0) 0x7f1200059650 con 0x7f120815e7a0 2024-09-16T01:40:27.318 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:40:27.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.316+0000 7f120d311640 1 -- 172.21.15.88:0/531328378 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f11e806c2b0 msgr2=0x7f11e806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:27.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.316+0000 7f120d311640 1 --2- 172.21.15.88:0/531328378 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f11e806c2b0 0x7f11e806e770 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f11f8034000 tx=0x7f11f802e000 comp rx=0 tx=0).stop 2024-09-16T01:40:27.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.316+0000 7f120d311640 1 -- 172.21.15.88:0/531328378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f120815e7a0 msgr2=0x7f120815f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:27.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.316+0000 7f120d311640 1 --2- 172.21.15.88:0/531328378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f120815e7a0 0x7f120815f2d0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f120000c7b0 tx=0x7f120000cc70 comp rx=0 tx=0).stop 2024-09-16T01:40:27.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.316+0000 7f120d311640 1 -- 172.21.15.88:0/531328378 shutdown_connections 2024-09-16T01:40:27.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.316+0000 7f120d311640 1 --2- 172.21.15.88:0/531328378 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f11e806c2b0 0x7f11e806e770 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:27.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.316+0000 7f120d311640 1 --2- 172.21.15.88:0/531328378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f120815e7a0 0x7f120815f2d0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:27.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.316+0000 7f120d311640 1 --2- 172.21.15.88:0/531328378 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12080cf1f0 0x7f120815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:27.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.316+0000 7f120d311640 1 --2- 172.21.15.88:0/531328378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12080b6720 0x7f1208158e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:27.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.316+0000 7f120d311640 1 -- 172.21.15.88:0/531328378 >> 172.21.15.88:0/531328378 conn(0x7f12080d11e0 msgr2=0x7f120815c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:27.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.316+0000 7f120d311640 1 -- 172.21.15.88:0/531328378 shutdown_connections 2024-09-16T01:40:27.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.316+0000 7f120d311640 1 -- 172.21.15.88:0/531328378 wait complete. 2024-09-16T01:40:27.320 INFO:tasks.ceph.mds.c.smithi088.stderr: -639> 2024-09-16T01:40:22.532+0000 7f9849fdc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:40:27.321 INFO:tasks.ceph.mds.c.smithi088.stderr: -145> 2024-09-16T01:40:26.413+0000 7f9845fd4640 -1 log_channel(cluster) log [ERR] : dir 0x10000000000 object missing on disk; some files may be lost (/subdir) 2024-09-16T01:40:27.331 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:40:27.345 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:40:27.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.430+0000 7f023a391640 1 Processor -- start 2024-09-16T01:40:27.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.430+0000 7f023a391640 1 -- start start 2024-09-16T01:40:27.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.431+0000 7f023a391640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f023415b480 0x7f02340b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:27.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.431+0000 7f023a391640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0234072700 0x7f023415bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:27.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.431+0000 7f023a391640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02340ccff0 con 0x7f02340b60c0 2024-09-16T01:40:27.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.431+0000 7f023a391640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f02340b6500 con 0x7f0234072700 2024-09-16T01:40:27.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.431+0000 7f023a391640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f02340cd2d0 con 0x7f023415b480 2024-09-16T01:40:27.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.431+0000 7f023892e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0234072700 0x7f023415bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:27.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.431+0000 7f023892e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0234072700 0x7f023415bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38608/0 (socket says 172.21.15.88:38608) 2024-09-16T01:40:27.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.431+0000 7f023892e640 1 -- 172.21.15.88:0/1384244450 learned_addr learned my addr 172.21.15.88:0/1384244450 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:27.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.431+0000 7f0233fff640 1 --2- 172.21.15.88:0/1384244450 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f023415b480 0x7f02340b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:27.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.431+0000 7f023892e640 1 -- 172.21.15.88:0/1384244450 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f023415b480 msgr2=0x7f02340b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:27.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.431+0000 7f023892e640 1 --2- 172.21.15.88:0/1384244450 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f023415b480 0x7f02340b5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:27.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.431+0000 7f023892e640 1 -- 172.21.15.88:0/1384244450 >> v1:172.21.15.154:6789/0 conn(0x7f02340b60c0 legacy=0x7f0234071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:27.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.431+0000 7f023892e640 1 -- 172.21.15.88:0/1384244450 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02340cd5b0 con 0x7f0234072700 2024-09-16T01:40:27.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.432+0000 7f023892e640 1 --2- 172.21.15.88:0/1384244450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0234072700 0x7f023415bc60 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f022c009e50 tx=0x7f022c02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=621ed611247640ce server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:27.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.432+0000 7f02337fe640 1 -- 172.21.15.88:0/1384244450 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022c036070 con 0x7f0234072700 2024-09-16T01:40:27.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.432+0000 7f02337fe640 1 -- 172.21.15.88:0/1384244450 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f022c031600 con 0x7f0234072700 2024-09-16T01:40:27.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.432+0000 7f02337fe640 1 -- 172.21.15.88:0/1384244450 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022c0319d0 con 0x7f0234072700 2024-09-16T01:40:27.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.433+0000 7f023a391640 1 -- 172.21.15.88:0/1384244450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0234072700 msgr2=0x7f023415bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:27.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.433+0000 7f023a391640 1 --2- 172.21.15.88:0/1384244450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0234072700 0x7f023415bc60 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f022c009e50 tx=0x7f022c02a680 comp rx=0 tx=0).stop 2024-09-16T01:40:27.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.433+0000 7f023a391640 1 -- 172.21.15.88:0/1384244450 shutdown_connections 2024-09-16T01:40:27.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.433+0000 7f023a391640 1 --2- 172.21.15.88:0/1384244450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0234072700 0x7f023415bc60 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:27.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.433+0000 7f023a391640 1 --2- 172.21.15.88:0/1384244450 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f023415b480 0x7f02340b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:27.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.433+0000 7f023a391640 1 -- 172.21.15.88:0/1384244450 >> 172.21.15.88:0/1384244450 conn(0x7f023406d0f0 msgr2=0x7f023406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:27.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.433+0000 7f023a391640 1 -- 172.21.15.88:0/1384244450 shutdown_connections 2024-09-16T01:40:27.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.433+0000 7f023a391640 1 -- 172.21.15.88:0/1384244450 wait complete. 2024-09-16T01:40:27.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.434+0000 7f023a391640 1 Processor -- start 2024-09-16T01:40:27.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.434+0000 7f023a391640 1 -- start start 2024-09-16T01:40:27.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.435+0000 7f023a391640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0234072700 0x7f0234145120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:27.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.435+0000 7f023a391640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02340b60c0 0x7f023414faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:27.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.435+0000 7f023a391640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f023415b480 0x7f02340cf970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:27.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.435+0000 7f023a391640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f023415b860 con 0x7f0234072700 2024-09-16T01:40:27.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.435+0000 7f023a391640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f02340cdb70 con 0x7f023415b480 2024-09-16T01:40:27.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.435+0000 7f023a391640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f023415b9e0 con 0x7f02340b60c0 2024-09-16T01:40:27.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.435+0000 7f0233fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0234072700 0x7f0234145120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:27.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.435+0000 7f0233fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0234072700 0x7f0234145120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38614/0 (socket says 172.21.15.88:38614) 2024-09-16T01:40:27.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.435+0000 7f0233fff640 1 -- 172.21.15.88:0/702824197 learned_addr learned my addr 172.21.15.88:0/702824197 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:27.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.435+0000 7f022bfff640 1 --2- 172.21.15.88:0/702824197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02340b60c0 0x7f023414faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:27.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.435+0000 7f023892e640 1 --2- 172.21.15.88:0/702824197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f023415b480 0x7f02340cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:27.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.435+0000 7f0233fff640 1 -- 172.21.15.88:0/702824197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02340b60c0 msgr2=0x7f023414faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:27.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.435+0000 7f0233fff640 1 --2- 172.21.15.88:0/702824197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02340b60c0 0x7f023414faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:27.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.435+0000 7f0233fff640 1 -- 172.21.15.88:0/702824197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f023415b480 msgr2=0x7f02340cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:27.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.435+0000 7f0233fff640 1 --2- 172.21.15.88:0/702824197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f023415b480 0x7f02340cf970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:27.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.435+0000 7f0233fff640 1 -- 172.21.15.88:0/702824197 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02340ca630 con 0x7f0234072700 2024-09-16T01:40:27.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.435+0000 7f022bfff640 1 --2- 172.21.15.88:0/702824197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02340b60c0 0x7f023414faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:27.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.435+0000 7f0233fff640 1 --2- 172.21.15.88:0/702824197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0234072700 0x7f0234145120 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f022400d620 tx=0x7f022400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:27.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.435+0000 7f02317fa640 1 -- 172.21.15.88:0/702824197 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022400b620 con 0x7f0234072700 2024-09-16T01:40:27.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.435+0000 7f02317fa640 1 -- 172.21.15.88:0/702824197 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0224019070 con 0x7f0234072700 2024-09-16T01:40:27.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.435+0000 7f02317fa640 1 -- 172.21.15.88:0/702824197 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022400ba00 con 0x7f0234072700 2024-09-16T01:40:27.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.435+0000 7f023a391640 1 -- 172.21.15.88:0/702824197 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02340ca8a0 con 0x7f0234072700 2024-09-16T01:40:27.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.436+0000 7f023a391640 1 -- 172.21.15.88:0/702824197 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02340cae60 con 0x7f0234072700 2024-09-16T01:40:27.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.437+0000 7f023a391640 1 -- 172.21.15.88:0/702824197 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02340b5b80 con 0x7f0234072700 2024-09-16T01:40:27.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.437+0000 7f02317fa640 1 -- 172.21.15.88:0/702824197 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f022400bba0 con 0x7f0234072700 2024-09-16T01:40:27.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.437+0000 7f02317fa640 1 --2- 172.21.15.88:0/702824197 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f021406c2b0 0x7f021406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:27.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.437+0000 7f022bfff640 1 --2- 172.21.15.88:0/702824197 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f021406c2b0 0x7f021406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:27.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.437+0000 7f02317fa640 1 -- 172.21.15.88:0/702824197 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 5881+0+0 (secure 0 0 0) 0x7f022408b860 con 0x7f0234072700 2024-09-16T01:40:27.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.438+0000 7f022bfff640 1 --2- 172.21.15.88:0/702824197 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f021406c2b0 0x7f021406e770 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f02341f7ba0 tx=0x7f021c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:27.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.441+0000 7f02317fa640 1 -- 172.21.15.88:0/702824197 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0224059fb0 con 0x7f0234072700 2024-09-16T01:40:27.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:27.674+0000 7f023a391640 1 -- 172.21.15.88:0/702824197 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f02340d0f90 con 0x7f0234072700 2024-09-16T01:40:28.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.316+0000 7f02317fa640 1 -- 172.21.15.88:0/702824197 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v81) ==== 107+0+0 (secure 0 0 0) 0x7f02240598b0 con 0x7f0234072700 2024-09-16T01:40:28.318 INFO:teuthology.orchestra.run.smithi088.stderr:nothing to do: rank is not damaged 2024-09-16T01:40:28.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.318+0000 7f023a391640 1 -- 172.21.15.88:0/702824197 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f021406c2b0 msgr2=0x7f021406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:28.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.318+0000 7f023a391640 1 --2- 172.21.15.88:0/702824197 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f021406c2b0 0x7f021406e770 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f02341f7ba0 tx=0x7f021c009040 comp rx=0 tx=0).stop 2024-09-16T01:40:28.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.318+0000 7f023a391640 1 -- 172.21.15.88:0/702824197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0234072700 msgr2=0x7f0234145120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:28.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.318+0000 7f023a391640 1 --2- 172.21.15.88:0/702824197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0234072700 0x7f0234145120 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f022400d620 tx=0x7f022400dae0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.318+0000 7f023a391640 1 -- 172.21.15.88:0/702824197 shutdown_connections 2024-09-16T01:40:28.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.318+0000 7f023a391640 1 --2- 172.21.15.88:0/702824197 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f021406c2b0 0x7f021406e770 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.318+0000 7f023a391640 1 --2- 172.21.15.88:0/702824197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f023415b480 0x7f02340cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.318+0000 7f023a391640 1 --2- 172.21.15.88:0/702824197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02340b60c0 0x7f023414faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.318+0000 7f023a391640 1 --2- 172.21.15.88:0/702824197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0234072700 0x7f0234145120 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.318+0000 7f023a391640 1 -- 172.21.15.88:0/702824197 >> 172.21.15.88:0/702824197 conn(0x7f023406d0f0 msgr2=0x7f023415a130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:28.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.318+0000 7f023a391640 1 -- 172.21.15.88:0/702824197 shutdown_connections 2024-09-16T01:40:28.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.318+0000 7f023a391640 1 -- 172.21.15.88:0/702824197 wait complete. 2024-09-16T01:40:28.334 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:40:28.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.416+0000 7f8cd8cb0f40 1 Processor -- start 2024-09-16T01:40:28.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.416+0000 7f8cd8cb0f40 1 -- start start 2024-09-16T01:40:28.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.417+0000 7f8cd8cb0f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f09777fdd0 0x55f097856fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.417+0000 7f8cd8cb0f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f097782230 0x55f09777ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.417+0000 7f8cd8cb0f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f09784e540 0x55f09777f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.417+0000 7f8cd8cb0f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f0977801e0 con 0x55f097782230 2024-09-16T01:40:28.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.417+0000 7f8cd8cb0f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f0977c95b0 con 0x55f09777fdd0 2024-09-16T01:40:28.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.417+0000 7f8cd8cb0f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55f0977c8610 con 0x55f09784e540 2024-09-16T01:40:28.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.417+0000 7f8cd5fa5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f097782230 0x55f09777ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:28.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.417+0000 7f8cd5fa5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f097782230 0x55f09777ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38638/0 (socket says 172.21.15.88:38638) 2024-09-16T01:40:28.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.417+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1042004630 learned_addr learned my addr 172.21.15.88:0/1042004630 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:28.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.417+0000 7f8cd67a6640 1 --2- 172.21.15.88:0/1042004630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f09777fdd0 0x55f097856fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:28.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.417+0000 7f8cd6fa7640 1 --2- 172.21.15.88:0/1042004630 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f09784e540 0x55f09777f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:28.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.417+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1042004630 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f09784e540 msgr2=0x55f09777f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:28.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.417+0000 7f8cd5fa5640 1 --2- 172.21.15.88:0/1042004630 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f09784e540 0x55f09777f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.417+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1042004630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f09777fdd0 msgr2=0x55f097856fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:28.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.417+0000 7f8cd5fa5640 1 --2- 172.21.15.88:0/1042004630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f09777fdd0 0x55f097856fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.417+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1042004630 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0976a9cb0 con 0x55f097782230 2024-09-16T01:40:28.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.418+0000 7f8cd5fa5640 1 --2- 172.21.15.88:0/1042004630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f097782230 0x55f09777ee60 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc00a770 tx=0x7f8ccc00a860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f91ceb4e4ec901db server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:28.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.418+0000 7f8cd4fa3640 1 -- 172.21.15.88:0/1042004630 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc019070 con 0x55f097782230 2024-09-16T01:40:28.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.418+0000 7f8cd4fa3640 1 -- 172.21.15.88:0/1042004630 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ccc007600 con 0x55f097782230 2024-09-16T01:40:28.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.418+0000 7f8cd4fa3640 1 -- 172.21.15.88:0/1042004630 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc0079d0 con 0x55f097782230 2024-09-16T01:40:28.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.418+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1042004630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f097782230 msgr2=0x55f09777ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:28.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.418+0000 7f8cd8cb0f40 1 --2- 172.21.15.88:0/1042004630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f097782230 0x55f09777ee60 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc00a770 tx=0x7f8ccc00a860 comp rx=0 tx=0).stop 2024-09-16T01:40:28.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.418+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1042004630 shutdown_connections 2024-09-16T01:40:28.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.418+0000 7f8cd8cb0f40 1 --2- 172.21.15.88:0/1042004630 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f09784e540 0x55f09777f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.418+0000 7f8cd8cb0f40 1 --2- 172.21.15.88:0/1042004630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f097782230 0x55f09777ee60 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.418+0000 7f8cd8cb0f40 1 --2- 172.21.15.88:0/1042004630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f09777fdd0 0x55f097856fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.418+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1042004630 >> 172.21.15.88:0/1042004630 conn(0x55f0977c9750 msgr2=0x55f097848f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:28.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.418+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1042004630 shutdown_connections 2024-09-16T01:40:28.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.418+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1042004630 wait complete. 2024-09-16T01:40:28.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.419+0000 7f8cd8cb0f40 1 Processor -- start 2024-09-16T01:40:28.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.419+0000 7f8cd8cb0f40 1 -- start start 2024-09-16T01:40:28.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.420+0000 7f8cd8cb0f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f09777fdd0 0x55f09777ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.420+0000 7f8cd8cb0f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f097782230 0x55f0978525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.420+0000 7f8cd8cb0f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f09784e540 0x55f0978532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.420+0000 7f8cd8cb0f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f097854c90 con 0x55f097782230 2024-09-16T01:40:28.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.420+0000 7f8cd8cb0f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f09777f8e0 con 0x55f09784e540 2024-09-16T01:40:28.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.420+0000 7f8cd8cb0f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55f097849fd0 con 0x55f09777fdd0 2024-09-16T01:40:28.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.420+0000 7f8cd5fa5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f097782230 0x55f0978525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:28.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.420+0000 7f8cd5fa5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f097782230 0x55f0978525d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38640/0 (socket says 172.21.15.88:38640) 2024-09-16T01:40:28.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.420+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 learned_addr learned my addr 172.21.15.88:0/1339628582 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:28.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.420+0000 7f8cd6fa7640 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f09784e540 0x55f0978532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:28.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.420+0000 7f8cd67a6640 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f09777fdd0 0x55f09777ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:28.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.420+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f09777fdd0 msgr2=0x55f09777ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:28.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.420+0000 7f8cd5fa5640 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f09777fdd0 0x55f09777ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.420+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f09784e540 msgr2=0x55f0978532c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:28.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.420+0000 7f8cd5fa5640 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f09784e540 0x55f0978532c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.420+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f09777f4f0 con 0x55f097782230 2024-09-16T01:40:28.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.420+0000 7f8cd5fa5640 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f097782230 0x55f0978525d0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc017040 tx=0x7f8ccc003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:28.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.421+0000 7f8cc77fe640 1 -- 172.21.15.88:0/1339628582 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc019040 con 0x55f097782230 2024-09-16T01:40:28.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.421+0000 7f8cc77fe640 1 -- 172.21.15.88:0/1339628582 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ccc024070 con 0x55f097782230 2024-09-16T01:40:28.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.421+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f097851af0 con 0x55f097782230 2024-09-16T01:40:28.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.421+0000 7f8cc77fe640 1 -- 172.21.15.88:0/1339628582 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc007e60 con 0x55f097782230 2024-09-16T01:40:28.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.421+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0978520b0 con 0x55f097782230 2024-09-16T01:40:28.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.422+0000 7f8cc77fe640 1 -- 172.21.15.88:0/1339628582 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8ccc014690 con 0x55f097782230 2024-09-16T01:40:28.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.423+0000 7f8cc77fe640 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8cb806acc0 0x7f8cb806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.423+0000 7f8cc77fe640 1 -- 172.21.15.88:0/1339628582 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 5881+0+0 (secure 0 0 0) 0x7f8ccc08b3a0 con 0x55f097782230 2024-09-16T01:40:28.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.423+0000 7f8cd67a6640 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8cb806acc0 0x7f8cb806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:28.428 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:40:28.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.423+0000 7f8cd8cb0f40 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55f0978463c0 0x55f09784b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.423+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09784b9d0 con 0x55f0978463c0 2024-09-16T01:40:28.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.423+0000 7f8cd6fa7640 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55f0978463c0 0x55f09784b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:28.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.423+0000 7f8cd67a6640 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8cb806acc0 0x7f8cb806d180 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8005fd0 tx=0x7f8cc8005a90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:28.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.423+0000 7f8cd6fa7640 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55f0978463c0 0x55f09784b490 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:28.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.425+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v87'43 uv16 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.425+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e87) -- 0x55f09784bda0 con 0x55f0978463c0 2024-09-16T01:40:28.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.426+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v87'44 uv44 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.426+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f8dd0 con 0x55f0978463c0 2024-09-16T01:40:28.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.427+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v87'45 uv45 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.431 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:40:28.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.427+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978fa180 con 0x55f0978463c0 2024-09-16T01:40:28.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.428+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v87'46 uv46 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.428+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978fa520 con 0x55f0978463c0 2024-09-16T01:40:28.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.429+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v87'47 uv47 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.429+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f9170 con 0x55f0978463c0 2024-09-16T01:40:28.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.430+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v87'48 uv48 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.430+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f9510 con 0x55f0978463c0 2024-09-16T01:40:28.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.431+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v87'49 uv22 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.431+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f9900 con 0x55f0978463c0 2024-09-16T01:40:28.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.432+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v87'50 uv50 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.432+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f9900 con 0x55f0978463c0 2024-09-16T01:40:28.435 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:40:28.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.433+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v87'51 uv51 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.433+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f98b0 con 0x55f0978463c0 2024-09-16T01:40:28.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.434+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v87'52 uv52 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.434+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f98b0 con 0x55f0978463c0 2024-09-16T01:40:28.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.435+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v87'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.435+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f98b0 con 0x55f0978463c0 2024-09-16T01:40:28.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.437+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v87'54 uv54 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.437+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f98b0 con 0x55f0978463c0 2024-09-16T01:40:28.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.438+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v87'55 uv55 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.440 INFO:teuthology.orchestra.run.smithi088.stdout:Write #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:40:28.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.438+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f98b0 con 0x55f0978463c0 2024-09-16T01:40:28.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.439+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [setxattr (30)] v87'56 uv56 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.439+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f98b0 con 0x55f0978463c0 2024-09-16T01:40:28.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.440+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [setxattr (62)] v87'57 uv57 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.440+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f98b0 con 0x55f0978463c0 2024-09-16T01:40:28.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.441+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [omap-set-header] v87'58 uv58 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.441+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f9c90 con 0x55f0978463c0 2024-09-16T01:40:28.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.442+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [omap-set-vals] v87'59 uv59 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.442+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f9b60 con 0x55f0978463c0 2024-09-16T01:40:28.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.443+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 600.00000000 [create] v87'60 uv35 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.443+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f9b60 con 0x55f0978463c0 2024-09-16T01:40:28.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.444+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 600.00000000 [delete] v87'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.444+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f9b60 con 0x55f0978463c0 2024-09-16T01:40:28.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.445+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v87'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.447 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:40:28.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.445+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f9b60 con 0x55f0978463c0 2024-09-16T01:40:28.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.446+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [setxattr (30)] v87'63 uv63 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.446+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f9b60 con 0x55f0978463c0 2024-09-16T01:40:28.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.448+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [setxattr (62)] v87'64 uv64 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.448+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f9b60 con 0x55f0978463c0 2024-09-16T01:40:28.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.449+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [omap-set-header] v87'65 uv65 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.449+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f9b60 con 0x55f0978463c0 2024-09-16T01:40:28.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.451+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 603.00000000 [create] v87'66 uv41 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.451+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f9b60 con 0x55f0978463c0 2024-09-16T01:40:28.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.453+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 603.00000000 [delete] v87'67 uv67 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.453+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f9b60 con 0x55f0978463c0 2024-09-16T01:40:28.456 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:40:28.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.454+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v87'68 uv68 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.454+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f9b60 con 0x55f0978463c0 2024-09-16T01:40:28.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.457+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [setxattr (30)] v87'69 uv69 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.457+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f9b60 con 0x55f0978463c0 2024-09-16T01:40:28.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.460+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [setxattr (62)] v87'70 uv70 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.460+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f9b60 con 0x55f0978463c0 2024-09-16T01:40:28.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.464+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [omap-set-header] v87'71 uv71 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.464+0000 7f8cd8cb0f40 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55f0978fadd0 0x55f0978fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.464+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:30 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978f9b60 con 0x55f0978fadd0 2024-09-16T01:40:28.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.465+0000 7f8cd67a6640 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55f0978fadd0 0x55f0978fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:28.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.465+0000 7f8cd67a6640 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55f0978fadd0 0x55f0978fd1c0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:28.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.466+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(30 1.00000000.inode [create] v87'46 uv25 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f0978fadd0 2024-09-16T01:40:28.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.466+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:31 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e87) -- 0x55f0978fd700 con 0x55f0978fadd0 2024-09-16T01:40:28.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.467+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(31 1.00000000.inode [delete] v87'47 uv47 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f0978fadd0 2024-09-16T01:40:28.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.467+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978fdaa0 con 0x55f0978fadd0 2024-09-16T01:40:28.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.468+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(32 1.00000000.inode [create] v87'48 uv48 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f0978fadd0 2024-09-16T01:40:28.471 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:40:28.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.468+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978fe2f0 con 0x55f0978fadd0 2024-09-16T01:40:28.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.469+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(33 1.00000000.inode [write 0~582] v87'49 uv49 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f0978fadd0 2024-09-16T01:40:28.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.469+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978fe690 con 0x55f0978fadd0 2024-09-16T01:40:28.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.470+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(34 1.00000000.inode [omap-set-header] v87'50 uv50 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f0978fadd0 2024-09-16T01:40:28.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.470+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978fea30 con 0x55f0978fadd0 2024-09-16T01:40:28.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.473+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(35 200.00000000 [create] v87'51 uv44 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f0978fadd0 2024-09-16T01:40:28.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.473+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e87) -- 0x55f0978fedd0 con 0x55f0978fadd0 2024-09-16T01:40:28.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.476+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(36 200.00000000 [delete] v87'52 uv52 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f0978fadd0 2024-09-16T01:40:28.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.476+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978ff170 con 0x55f0978fadd0 2024-09-16T01:40:28.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.477+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(37 200.00000000 [create] v87'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f0978fadd0 2024-09-16T01:40:28.479 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:40:28.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.477+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978ff170 con 0x55f0978fadd0 2024-09-16T01:40:28.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.480+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(38 200.00000000 [write 0~90] v87'54 uv54 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f0978fadd0 2024-09-16T01:40:28.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.480+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978ff170 con 0x55f0978fadd0 2024-09-16T01:40:28.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.482+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(39 200.00000000 [omap-set-header] v87'55 uv55 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f0978fadd0 2024-09-16T01:40:28.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.483+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978ff170 con 0x55f0978fadd0 2024-09-16T01:40:28.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.485+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(40 200.00000001 [create] v87'56 uv45 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f0978fadd0 2024-09-16T01:40:28.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.485+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e87) -- 0x55f0978ff170 con 0x55f0978fadd0 2024-09-16T01:40:28.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.486+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(41 200.00000001 [delete] v87'57 uv57 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f0978fadd0 2024-09-16T01:40:28.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.486+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978ff170 con 0x55f0978fadd0 2024-09-16T01:40:28.491 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:40:28.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.489+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(42 200.00000001 [create] v87'58 uv58 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f0978fadd0 2024-09-16T01:40:28.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.489+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0979035f0 con 0x55f0978fadd0 2024-09-16T01:40:28.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.492+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(43 200.00000001 [write 0~17490] v87'59 uv59 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f0978fadd0 2024-09-16T01:40:28.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.492+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0979035f0 con 0x55f0978fadd0 2024-09-16T01:40:28.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.497+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(44 200.00000001 [omap-set-header] v87'60 uv60 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f0978fadd0 2024-09-16T01:40:28.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.497+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978ff170 con 0x55f0978fadd0 2024-09-16T01:40:28.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.499+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(45 606.00000000 [create] v87'61 uv41 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f0978fadd0 2024-09-16T01:40:28.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.499+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e87) -- 0x55f0978ff170 con 0x55f0978fadd0 2024-09-16T01:40:28.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.501+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(46 606.00000000 [delete] v87'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f0978fadd0 2024-09-16T01:40:28.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.501+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978ff170 con 0x55f0978fadd0 2024-09-16T01:40:28.506 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:40:28.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.504+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(47 606.00000000 [create] v87'63 uv63 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f0978fadd0 2024-09-16T01:40:28.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.504+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f097900180 con 0x55f0978fadd0 2024-09-16T01:40:28.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.507+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(48 606.00000000 [setxattr (30)] v87'64 uv64 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f0978fadd0 2024-09-16T01:40:28.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.507+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f097900180 con 0x55f0978fadd0 2024-09-16T01:40:28.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.509+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(49 606.00000000 [setxattr (62)] v87'65 uv65 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f0978fadd0 2024-09-16T01:40:28.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.509+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978ff170 con 0x55f0978fadd0 2024-09-16T01:40:28.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.512+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(50 606.00000000 [omap-set-header] v87'66 uv66 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f0978fadd0 2024-09-16T01:40:28.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.512+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:51 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978ff170 con 0x55f0978463c0 2024-09-16T01:40:28.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.513+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(51 607.00000000 [create] v87'9 uv8 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.513+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:52 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e87) -- 0x55f0978ff170 con 0x55f0978463c0 2024-09-16T01:40:28.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.514+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(52 607.00000000 [delete] v87'10 uv10 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.514+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978ff170 con 0x55f0978463c0 2024-09-16T01:40:28.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.517+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v87'11 uv11 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.519 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:40:28.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.517+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f097900180 con 0x55f0978463c0 2024-09-16T01:40:28.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.519+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [setxattr (30)] v87'12 uv12 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.519+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f097900180 con 0x55f0978463c0 2024-09-16T01:40:28.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.521+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [setxattr (62)] v87'13 uv13 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.521+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978ff170 con 0x55f0978463c0 2024-09-16T01:40:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.526+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [omap-set-header] v87'14 uv14 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.526+0000 7f8cd8cb0f40 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55f0978ff830 0x55f097901c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.526+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:57 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f097902300 con 0x55f0978ff830 2024-09-16T01:40:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.526+0000 7f8cd5fa5640 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55f0978ff830 0x55f097901c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.527+0000 7f8cd5fa5640 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55f0978ff830 0x55f097901c80 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:28.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.528+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(57 mds0_openfiles.0 [create] v87'10 uv9 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f8ccc055b10 con 0x55f0978ff830 2024-09-16T01:40:28.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.528+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:58 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e87) -- 0x55f097902810 con 0x55f0978ff830 2024-09-16T01:40:28.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.529+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(58 mds0_openfiles.0 [delete] v87'11 uv11 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8ccc055b10 con 0x55f0978ff830 2024-09-16T01:40:28.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.529+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f097902d10 con 0x55f0978ff830 2024-09-16T01:40:28.533 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:40:28.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.531+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(59 mds0_openfiles.0 [create] v87'12 uv12 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8ccc055b10 con 0x55f0978ff830 2024-09-16T01:40:28.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.531+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f0978fdf20 con 0x55f0978ff830 2024-09-16T01:40:28.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.536+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(60 mds0_openfiles.0 [omap-set-header] v87'13 uv13 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8ccc055b10 con 0x55f0978ff830 2024-09-16T01:40:28.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.536+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:61 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f097903640 con 0x55f0978463c0 2024-09-16T01:40:28.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.537+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(61 608.00000000 [create] v87'9 uv8 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.537+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:62 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e87) -- 0x55f097903640 con 0x55f0978463c0 2024-09-16T01:40:28.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.538+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(62 608.00000000 [delete] v87'10 uv10 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.538+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f097903640 con 0x55f0978463c0 2024-09-16T01:40:28.543 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:40:28.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.541+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v87'11 uv11 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.541+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f097903640 con 0x55f0978463c0 2024-09-16T01:40:28.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.542+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [setxattr (30)] v87'12 uv12 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.542+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f097903640 con 0x55f0978463c0 2024-09-16T01:40:28.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.544+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [setxattr (62)] v87'13 uv13 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.544+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f097903640 con 0x55f0978463c0 2024-09-16T01:40:28.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.545+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [omap-set-header] v87'14 uv14 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.545+0000 7f8cd8cb0f40 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f097903f40 0x55f097906330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.545+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:67 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f097906870 con 0x55f097903f40 2024-09-16T01:40:28.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.545+0000 7f8cd6fa7640 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f097903f40 0x55f097906330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:28.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.546+0000 7f8cd6fa7640 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f097903f40 0x55f097906330 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:28.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.547+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(67 604.00000000 [create] v87'15 uv9 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.547+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:68 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e87) -- 0x55f097906870 con 0x55f097903f40 2024-09-16T01:40:28.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.550+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(68 604.00000000 [delete] v87'16 uv16 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.550+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f097906c60 con 0x55f097903f40 2024-09-16T01:40:28.553 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:40:28.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.551+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(69 604.00000000 [create] v87'17 uv17 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.551+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f097907c70 con 0x55f097903f40 2024-09-16T01:40:28.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.554+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(70 604.00000000 [setxattr (30)] v87'18 uv18 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.554+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f097907c70 con 0x55f097903f40 2024-09-16T01:40:28.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.556+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(71 604.00000000 [setxattr (62)] v87'19 uv19 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.556+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f097906c60 con 0x55f097903f40 2024-09-16T01:40:28.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.558+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(72 604.00000000 [omap-set-header] v87'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.558+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f097906c60 con 0x55f097903f40 2024-09-16T01:40:28.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.560+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(73 500.00000000 [create] v87'21 uv14 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.560+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e87) -- 0x55f097906c60 con 0x55f097903f40 2024-09-16T01:40:28.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.563+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(74 500.00000000 [delete] v87'22 uv22 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.563+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f097906c60 con 0x55f097903f40 2024-09-16T01:40:28.567 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:40:28.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.565+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(75 500.00000000 [create] v87'23 uv23 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.565+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f097906c60 con 0x55f097903f40 2024-09-16T01:40:28.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.568+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(76 500.00000000 [write 0~90] v87'24 uv24 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.568+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f097906c60 con 0x55f097903f40 2024-09-16T01:40:28.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.569+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(77 500.00000000 [omap-set-header] v87'25 uv25 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.569+0000 7f8cd8cb0f40 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55f097907170 0x55f097909560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.569+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:78 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f097909c90 con 0x55f097907170 2024-09-16T01:40:28.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.569+0000 7f8cd67a6640 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55f097907170 0x55f097909560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:28.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.570+0000 7f8cd67a6640 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55f097907170 0x55f097909560 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:28.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.571+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(78 mds_snaptable [create] v87'7 uv6 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f097907170 2024-09-16T01:40:28.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.571+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:79 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e87) -- 0x55f09790a060 con 0x55f097907170 2024-09-16T01:40:28.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.572+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(79 mds_snaptable [delete] v87'8 uv8 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f097907170 2024-09-16T01:40:28.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.572+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790a060 con 0x55f097907170 2024-09-16T01:40:28.578 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:40:28.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.576+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(80 mds_snaptable [create] v87'9 uv9 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f097907170 2024-09-16T01:40:28.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.576+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790a060 con 0x55f097907170 2024-09-16T01:40:28.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.579+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(81 mds_snaptable [write 0~70] v87'10 uv10 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f097907170 2024-09-16T01:40:28.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.579+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790a060 con 0x55f097907170 2024-09-16T01:40:28.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.583+0000 7f8cd67a6640 1 -- 172.21.15.88:0/1339628582 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(82 mds_snaptable [omap-set-header] v87'11 uv11 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8cc8016860 con 0x55f097907170 2024-09-16T01:40:28.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.583+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:83 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790a060 con 0x55f0978463c0 2024-09-16T01:40:28.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.584+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(83 605.00000000 [create] v87'9 uv8 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.584+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:84 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e87) -- 0x55f09790a060 con 0x55f0978463c0 2024-09-16T01:40:28.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.585+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(84 605.00000000 [delete] v87'10 uv10 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.585+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790a060 con 0x55f0978463c0 2024-09-16T01:40:28.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.586+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v87'11 uv11 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.588 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:40:28.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.586+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790b070 con 0x55f0978463c0 2024-09-16T01:40:28.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.587+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [setxattr (30)] v87'12 uv12 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.587+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790b070 con 0x55f0978463c0 2024-09-16T01:40:28.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.588+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [setxattr (62)] v87'13 uv13 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.588+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790a060 con 0x55f0978463c0 2024-09-16T01:40:28.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.588+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [omap-set-header] v87'14 uv14 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f0978463c0 2024-09-16T01:40:28.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.589+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:89 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790a060 con 0x55f0978ff830 2024-09-16T01:40:28.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.589+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(89 mds0_inotable [create] v87'14 uv8 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f8ccc055b10 con 0x55f0978ff830 2024-09-16T01:40:28.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.589+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:90 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e87) -- 0x55f09790a420 con 0x55f0978ff830 2024-09-16T01:40:28.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.590+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(90 mds0_inotable [delete] v87'15 uv15 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8ccc055b10 con 0x55f0978ff830 2024-09-16T01:40:28.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.590+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790a810 con 0x55f0978ff830 2024-09-16T01:40:28.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.591+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(91 mds0_inotable [create] v87'16 uv16 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8ccc055b10 con 0x55f0978ff830 2024-09-16T01:40:28.594 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:40:28.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.591+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790ac00 con 0x55f0978ff830 2024-09-16T01:40:28.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.592+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(92 mds0_inotable [write 0~34] v87'17 uv17 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8ccc055b10 con 0x55f0978ff830 2024-09-16T01:40:28.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.592+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790ac00 con 0x55f0978ff830 2024-09-16T01:40:28.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.594+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(93 mds0_inotable [omap-set-header] v87'18 uv18 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8ccc055b10 con 0x55f0978ff830 2024-09-16T01:40:28.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.594+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790aff0 con 0x55f0978ff830 2024-09-16T01:40:28.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.594+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(94 100.00000000 [create] v87'19 uv13 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8ccc055b10 con 0x55f0978ff830 2024-09-16T01:40:28.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.595+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e87) -- 0x55f09790a8b0 con 0x55f0978ff830 2024-09-16T01:40:28.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.595+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(95 100.00000000 [delete] v87'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8ccc055b10 con 0x55f0978ff830 2024-09-16T01:40:28.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.595+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790a8b0 con 0x55f0978ff830 2024-09-16T01:40:28.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.596+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(96 100.00000000 [create] v87'21 uv21 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8ccc055b10 con 0x55f0978ff830 2024-09-16T01:40:28.599 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:40:28.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.596+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790a8b0 con 0x55f0978ff830 2024-09-16T01:40:28.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.597+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(97 100.00000000 [omap-set-header] v87'22 uv22 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8ccc055b10 con 0x55f0978ff830 2024-09-16T01:40:28.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.597+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f097903230 con 0x55f0978ff830 2024-09-16T01:40:28.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.598+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v87'23 uv23 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8ccc055b10 con 0x55f0978ff830 2024-09-16T01:40:28.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.598+0000 7f8cd8cb0f40 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55f09790b6b0 0x55f09790de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.598+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:99 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790e340 con 0x55f09790b6b0 2024-09-16T01:40:28.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.599+0000 7f8cd5fa5640 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55f09790b6b0 0x55f09790de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:28.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.599+0000 7f8cd5fa5640 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55f09790b6b0 0x55f09790de00 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:28.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.600+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(99 mds0_sessionmap [create] v87'9 uv7 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f8ccc055b10 con 0x55f09790b6b0 2024-09-16T01:40:28.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.600+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:100 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e87) -- 0x55f09790e340 con 0x55f09790b6b0 2024-09-16T01:40:28.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.601+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v87'10 uv10 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f8ccc055b10 con 0x55f09790b6b0 2024-09-16T01:40:28.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.601+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790e340 con 0x55f09790b6b0 2024-09-16T01:40:28.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.602+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(101 mds0_sessionmap [create] v87'11 uv11 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f8ccc055b10 con 0x55f09790b6b0 2024-09-16T01:40:28.604 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:40:28.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.602+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790e340 con 0x55f09790b6b0 2024-09-16T01:40:28.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.603+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v87'12 uv12 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f8ccc055b10 con 0x55f09790b6b0 2024-09-16T01:40:28.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.603+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790e340 con 0x55f0978ff830 2024-09-16T01:40:28.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.604+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(103 609.00000000 [create] v87'9 uv8 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8ccc055b10 con 0x55f0978ff830 2024-09-16T01:40:28.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.604+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e87) -- 0x55f09790e340 con 0x55f0978ff830 2024-09-16T01:40:28.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.608+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(104 609.00000000 [delete] v87'10 uv10 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8ccc055b10 con 0x55f0978ff830 2024-09-16T01:40:28.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.608+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790e340 con 0x55f0978ff830 2024-09-16T01:40:28.611 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:40:28.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.609+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(105 609.00000000 [create] v87'11 uv11 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8ccc055b10 con 0x55f0978ff830 2024-09-16T01:40:28.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.609+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790f350 con 0x55f0978ff830 2024-09-16T01:40:28.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.610+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v87'12 uv12 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8ccc055b10 con 0x55f0978ff830 2024-09-16T01:40:28.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.610+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790f350 con 0x55f0978ff830 2024-09-16T01:40:28.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.611+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v87'13 uv13 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8ccc055b10 con 0x55f0978ff830 2024-09-16T01:40:28.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.611+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790e340 con 0x55f0978ff830 2024-09-16T01:40:28.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.612+0000 7f8cd5fa5640 1 -- 172.21.15.88:0/1339628582 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(108 609.00000000 [omap-set-header] v87'14 uv14 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8ccc055b10 con 0x55f0978ff830 2024-09-16T01:40:28.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.612+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790e340 con 0x55f097903f40 2024-09-16T01:40:28.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.613+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(109 400.00000000 [create] v87'7 uv6 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.613+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e87) -- 0x55f09790e340 con 0x55f097903f40 2024-09-16T01:40:28.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.616+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(110 400.00000000 [delete] v87'8 uv8 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.616+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790e340 con 0x55f097903f40 2024-09-16T01:40:28.620 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:40:28.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.618+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(111 400.00000000 [create] v87'9 uv9 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.618+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790e340 con 0x55f097903f40 2024-09-16T01:40:28.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.622+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v87'10 uv10 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.623+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790e340 con 0x55f097903f40 2024-09-16T01:40:28.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.624+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v87'11 uv11 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.624+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790e340 con 0x55f097903f40 2024-09-16T01:40:28.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.625+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(114 100.00000000.inode [create] v87'20 uv11 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.625+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e87) -- 0x55f09790e340 con 0x55f097903f40 2024-09-16T01:40:28.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.626+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(115 100.00000000.inode [delete] v87'21 uv21 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.626+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790e340 con 0x55f097903f40 2024-09-16T01:40:28.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.627+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(116 100.00000000.inode [create] v87'22 uv22 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.629 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:40:28.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.627+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790e340 con 0x55f097903f40 2024-09-16T01:40:28.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.628+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v87'23 uv23 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.628+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790e340 con 0x55f097903f40 2024-09-16T01:40:28.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.629+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v87'24 uv24 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.629+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790e340 con 0x55f097903f40 2024-09-16T01:40:28.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.630+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(119 1.00000000 [create] v87'25 uv18 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.631+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e87) -- 0x55f09790e340 con 0x55f097903f40 2024-09-16T01:40:28.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.632+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(120 1.00000000 [delete] v87'26 uv26 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.632+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790e340 con 0x55f097903f40 2024-09-16T01:40:28.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.633+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(121 1.00000000 [create] v87'27 uv27 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.636 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:40:28.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.633+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790f350 con 0x55f097903f40 2024-09-16T01:40:28.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.636+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v87'28 uv28 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.636+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790f350 con 0x55f097903f40 2024-09-16T01:40:28.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.637+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v87'29 uv29 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.637+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790e340 con 0x55f097903f40 2024-09-16T01:40:28.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.639+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v87'30 uv30 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.639+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) -- 0x55f09790e340 con 0x55f097903f40 2024-09-16T01:40:28.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.641+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v87'31 uv31 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f8cd0056030 con 0x55f097903f40 2024-09-16T01:40:28.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.641+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55f0978463c0 msgr2=0x55f09784b490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:28.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.641+0000 7f8cd8cb0f40 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55f0978463c0 0x55f09784b490 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.641+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55f0978ff830 msgr2=0x55f097901c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:28.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.641+0000 7f8cd8cb0f40 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55f0978ff830 0x55f097901c80 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.641+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55f09790b6b0 msgr2=0x55f09790de00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:28.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.641+0000 7f8cd8cb0f40 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55f09790b6b0 0x55f09790de00 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.641+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55f0978fadd0 msgr2=0x55f0978fd1c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:28.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.641+0000 7f8cd8cb0f40 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55f0978fadd0 0x55f0978fd1c0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.641+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f097903f40 msgr2=0x55f097906330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:28.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.641+0000 7f8cd8cb0f40 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f097903f40 0x55f097906330 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.641+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55f097907170 msgr2=0x55f097909560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:28.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.641+0000 7f8cd8cb0f40 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55f097907170 0x55f097909560 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.641+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 reap_dead start 2024-09-16T01:40:28.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.641+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 reap_dead start 2024-09-16T01:40:28.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.641+0000 7f8cd6fa7640 1 -- 172.21.15.88:0/1339628582 reap_dead start 2024-09-16T01:40:28.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.641+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8cb806acc0 msgr2=0x7f8cb806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:28.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.641+0000 7f8cd8cb0f40 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8cb806acc0 0x7f8cb806d180 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8005fd0 tx=0x7f8cc8005a90 comp rx=0 tx=0).stop 2024-09-16T01:40:28.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.641+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f097782230 msgr2=0x55f0978525d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:28.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.641+0000 7f8cd8cb0f40 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f097782230 0x55f0978525d0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc017040 tx=0x7f8ccc003020 comp rx=0 tx=0).stop 2024-09-16T01:40:28.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.641+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 shutdown_connections 2024-09-16T01:40:28.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.641+0000 7f8cd8cb0f40 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8cb806acc0 0x7f8cb806d180 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.641+0000 7f8cd8cb0f40 1 --2- 172.21.15.88:0/1339628582 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f097782230 0x55f0978525d0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.641+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 >> 172.21.15.88:0/1339628582 conn(0x55f0977c9750 msgr2=0x55f09777de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:28.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.642+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 shutdown_connections 2024-09-16T01:40:28.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.642+0000 7f8cd8cb0f40 1 -- 172.21.15.88:0/1339628582 wait complete. 2024-09-16T01:40:28.647 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 600.00000000 2024-09-16T01:40:28.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.728+0000 7f7702c35f40 1 Processor -- start 2024-09-16T01:40:28.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.728+0000 7f7702c35f40 1 -- start start 2024-09-16T01:40:28.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.728+0000 7f7702c35f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56471dd05e10 0x56471ddaae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.728+0000 7f7702c35f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56471dd06a60 0x56471dda9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.728+0000 7f7702c35f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56471dd20250 0x56471de3d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.728+0000 7f7702c35f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56471dc28510 con 0x56471dd05e10 2024-09-16T01:40:28.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.728+0000 7f7702c35f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56471dd06220 con 0x56471dd20250 2024-09-16T01:40:28.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.728+0000 7f7702c35f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56471dd1d5f0 con 0x56471dd06a60 2024-09-16T01:40:28.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.729+0000 7f770072d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56471dd05e10 0x56471ddaae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:28.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.729+0000 7f770072d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56471dd05e10 0x56471ddaae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38656/0 (socket says 172.21.15.88:38656) 2024-09-16T01:40:28.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.729+0000 7f770072d640 1 -- 172.21.15.88:0/1937447152 learned_addr learned my addr 172.21.15.88:0/1937447152 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:28.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.729+0000 7f770072d640 1 -- 172.21.15.88:0/1937447152 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56471dd06a60 msgr2=0x56471dda9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:28.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.729+0000 7f770072d640 1 --2- 172.21.15.88:0/1937447152 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56471dd06a60 0x56471dda9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.729+0000 7f770072d640 1 -- 172.21.15.88:0/1937447152 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56471dd20250 msgr2=0x56471de3d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:28.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.729+0000 7f770072d640 1 --2- 172.21.15.88:0/1937447152 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56471dd20250 0x56471de3d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.729+0000 7f770072d640 1 -- 172.21.15.88:0/1937447152 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56471ddab390 con 0x56471dd05e10 2024-09-16T01:40:28.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.729+0000 7f770072d640 1 --2- 172.21.15.88:0/1937447152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56471dd05e10 0x56471ddaae50 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f76f0005cf0 tx=0x7f76f0051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cbb8350e2671999 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:28.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.729+0000 7f76ff72b640 1 -- 172.21.15.88:0/1937447152 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f0056070 con 0x56471dd05e10 2024-09-16T01:40:28.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.729+0000 7f76ff72b640 1 -- 172.21.15.88:0/1937447152 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76f0052bb0 con 0x56471dd05e10 2024-09-16T01:40:28.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.729+0000 7f76ff72b640 1 -- 172.21.15.88:0/1937447152 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f0065440 con 0x56471dd05e10 2024-09-16T01:40:28.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.730+0000 7f7702c35f40 1 -- 172.21.15.88:0/1937447152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56471dd05e10 msgr2=0x56471ddaae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:28.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.730+0000 7f7702c35f40 1 --2- 172.21.15.88:0/1937447152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56471dd05e10 0x56471ddaae50 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f76f0005cf0 tx=0x7f76f0051000 comp rx=0 tx=0).stop 2024-09-16T01:40:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.730+0000 7f7702c35f40 1 -- 172.21.15.88:0/1937447152 shutdown_connections 2024-09-16T01:40:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.730+0000 7f7702c35f40 1 --2- 172.21.15.88:0/1937447152 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56471dd20250 0x56471de3d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.730+0000 7f7702c35f40 1 --2- 172.21.15.88:0/1937447152 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56471dd06a60 0x56471dda9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.730+0000 7f7702c35f40 1 --2- 172.21.15.88:0/1937447152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56471dd05e10 0x56471ddaae50 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.730+0000 7f7702c35f40 1 -- 172.21.15.88:0/1937447152 >> 172.21.15.88:0/1937447152 conn(0x56471dd21750 msgr2=0x56471dd05650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.730+0000 7f7702c35f40 1 -- 172.21.15.88:0/1937447152 shutdown_connections 2024-09-16T01:40:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.730+0000 7f7702c35f40 1 -- 172.21.15.88:0/1937447152 wait complete. 2024-09-16T01:40:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.730+0000 7f7702c35f40 1 Processor -- start 2024-09-16T01:40:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.730+0000 7f7702c35f40 1 -- start start 2024-09-16T01:40:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.731+0000 7f7702c35f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56471dd05e10 0x56471de3f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.731+0000 7f7702c35f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56471dd06a60 0x56471dda65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.731+0000 7f7702c35f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56471dd20250 0x56471dda6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.731+0000 7f7702c35f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56471dd1d280 con 0x56471dd05e10 2024-09-16T01:40:28.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.731+0000 7f7702c35f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56471dda92f0 con 0x56471dd20250 2024-09-16T01:40:28.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.731+0000 7f7702c35f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56471dda8f10 con 0x56471dd06a60 2024-09-16T01:40:28.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.731+0000 7f770072d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56471dd05e10 0x56471de3f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:28.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.731+0000 7f770072d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56471dd05e10 0x56471de3f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38670/0 (socket says 172.21.15.88:38670) 2024-09-16T01:40:28.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.731+0000 7f770072d640 1 -- 172.21.15.88:0/2076093332 learned_addr learned my addr 172.21.15.88:0/2076093332 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:28.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.731+0000 7f76f7f2c640 1 --2- 172.21.15.88:0/2076093332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56471dd06a60 0x56471dda65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:28.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.731+0000 7f7700f2e640 1 --2- 172.21.15.88:0/2076093332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56471dd20250 0x56471dda6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:28.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.731+0000 7f770072d640 1 -- 172.21.15.88:0/2076093332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56471dd06a60 msgr2=0x56471dda65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:28.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.731+0000 7f770072d640 1 --2- 172.21.15.88:0/2076093332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56471dd06a60 0x56471dda65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.731+0000 7f770072d640 1 -- 172.21.15.88:0/2076093332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56471dd20250 msgr2=0x56471dda6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:28.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.731+0000 7f770072d640 1 --2- 172.21.15.88:0/2076093332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56471dd20250 0x56471dda6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.731+0000 7f770072d640 1 -- 172.21.15.88:0/2076093332 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56471dd9e580 con 0x56471dd05e10 2024-09-16T01:40:28.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.732+0000 7f770072d640 1 --2- 172.21.15.88:0/2076093332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56471dd05e10 0x56471de3f430 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f76f0045ff0 tx=0x7f76f0002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:28.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.732+0000 7f76fe729640 1 -- 172.21.15.88:0/2076093332 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f0056070 con 0x56471dd05e10 2024-09-16T01:40:28.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.732+0000 7f76fe729640 1 -- 172.21.15.88:0/2076093332 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76f0065640 con 0x56471dd05e10 2024-09-16T01:40:28.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.732+0000 7f76fe729640 1 -- 172.21.15.88:0/2076093332 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f006a040 con 0x56471dd05e10 2024-09-16T01:40:28.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.732+0000 7f7702c35f40 1 -- 172.21.15.88:0/2076093332 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56471dd9de90 con 0x56471dd05e10 2024-09-16T01:40:28.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.732+0000 7f7702c35f40 1 -- 172.21.15.88:0/2076093332 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56471ddab560 con 0x56471dd05e10 2024-09-16T01:40:28.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.733+0000 7f76fe729640 1 -- 172.21.15.88:0/2076093332 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f76f0059020 con 0x56471dd05e10 2024-09-16T01:40:28.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.733+0000 7f76fe729640 1 --2- 172.21.15.88:0/2076093332 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f76e006ac90 0x7f76e006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.734+0000 7f76fe729640 1 -- 172.21.15.88:0/2076093332 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 5881+0+0 (secure 0 0 0) 0x7f76f00cf790 con 0x56471dd05e10 2024-09-16T01:40:28.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.734+0000 7f76f7f2c640 1 --2- 172.21.15.88:0/2076093332 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f76e006ac90 0x7f76e006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:28.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.734+0000 7f7702c35f40 1 --2- 172.21.15.88:0/2076093332 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x56471de3f970 0x56471dda4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.734+0000 7f7702c35f40 1 -- 172.21.15.88:0/2076093332 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e87) -- 0x56471dd9e890 con 0x56471de3f970 2024-09-16T01:40:28.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.734+0000 7f7700f2e640 1 --2- 172.21.15.88:0/2076093332 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x56471de3f970 0x56471dda4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:28.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.734+0000 7f76f7f2c640 1 --2- 172.21.15.88:0/2076093332 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f76e006ac90 0x7f76e006d150 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x56471dd1c7d0 tx=0x7f76e8005810 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:28.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.734+0000 7f7700f2e640 1 --2- 172.21.15.88:0/2076093332 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x56471de3f970 0x56471dda4760 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:28.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.735+0000 7f7700f2e640 1 -- 172.21.15.88:0/2076093332 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 600.00000000 [delete] v87'72 uv72 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f76f8001460 con 0x56471de3f970 2024-09-16T01:40:28.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.735+0000 7f7702c35f40 1 -- 172.21.15.88:0/2076093332 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x56471de3f970 msgr2=0x56471dda4760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:28.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.735+0000 7f7702c35f40 1 --2- 172.21.15.88:0/2076093332 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x56471de3f970 0x56471dda4760 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.735+0000 7f7702c35f40 1 -- 172.21.15.88:0/2076093332 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f76e006ac90 msgr2=0x7f76e006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:28.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.735+0000 7f7702c35f40 1 --2- 172.21.15.88:0/2076093332 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f76e006ac90 0x7f76e006d150 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x56471dd1c7d0 tx=0x7f76e8005810 comp rx=0 tx=0).stop 2024-09-16T01:40:28.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.735+0000 7f7702c35f40 1 -- 172.21.15.88:0/2076093332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56471dd05e10 msgr2=0x56471de3f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:28.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.735+0000 7f7702c35f40 1 --2- 172.21.15.88:0/2076093332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56471dd05e10 0x56471de3f430 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f76f0045ff0 tx=0x7f76f0002fb0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.735+0000 7f7700f2e640 1 -- 172.21.15.88:0/2076093332 reap_dead start 2024-09-16T01:40:28.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.735+0000 7f7702c35f40 1 -- 172.21.15.88:0/2076093332 shutdown_connections 2024-09-16T01:40:28.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.735+0000 7f7702c35f40 1 -- 172.21.15.88:0/2076093332 >> 172.21.15.88:0/2076093332 conn(0x56471dd21750 msgr2=0x56471dd05650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:28.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.735+0000 7f7702c35f40 1 -- 172.21.15.88:0/2076093332 shutdown_connections 2024-09-16T01:40:28.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.735+0000 7f7702c35f40 1 -- 172.21.15.88:0/2076093332 wait complete. 2024-09-16T01:40:28.745 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:40:28.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.879+0000 7fcaa3fff640 1 Processor -- start 2024-09-16T01:40:28.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.879+0000 7fcaa3fff640 1 -- start start 2024-09-16T01:40:28.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.880+0000 7fcaa3fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcaa415f890 0x7fcaa40b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.880+0000 7fcaa3fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcaa40b40f0 0x7fcaa40b6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.880+0000 7fcaa3fff640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcaa406d150 con 0x7fcaa40ce720 2024-09-16T01:40:28.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.880+0000 7fcaa3fff640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fcaa40b5360 con 0x7fcaa415f890 2024-09-16T01:40:28.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.880+0000 7fcaa3fff640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcaa40b5640 con 0x7fcaa40b40f0 2024-09-16T01:40:28.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.881+0000 7fcaa37fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcaa40b40f0 0x7fcaa40b6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:28.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.881+0000 7fcaa2ffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcaa415f890 0x7fcaa40b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:28.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.881+0000 7fcaa27fc640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fcaa40ce720 0x7fcaa415ffd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:42658/0 (socket says 172.21.15.88:42658) 2024-09-16T01:40:28.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.881+0000 7fcaa2ffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcaa415f890 0x7fcaa40b5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:51778/0 (socket says 172.21.15.88:51778) 2024-09-16T01:40:28.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.881+0000 7fcaa27fc640 1 -- 172.21.15.88:0/4185535236 learned_addr learned my addr 172.21.15.88:0/4185535236 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:28.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.881+0000 7fcaa37fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcaa40b40f0 0x7fcaa40b6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:56390/0 (socket says 172.21.15.88:56390) 2024-09-16T01:40:28.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.881+0000 7fcaa2ffd640 1 -- 172.21.15.88:0/4185535236 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcaa40b40f0 msgr2=0x7fcaa40b6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:28.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.881+0000 7fcaa2ffd640 1 --2- 172.21.15.88:0/4185535236 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcaa40b40f0 0x7fcaa40b6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.881+0000 7fcaa2ffd640 1 -- 172.21.15.88:0/4185535236 >> v1:172.21.15.154:6789/0 conn(0x7fcaa40ce720 legacy=0x7fcaa415ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:28.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.881+0000 7fcaa2ffd640 1 -- 172.21.15.88:0/4185535236 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaa40cf220 con 0x7fcaa415f890 2024-09-16T01:40:28.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.881+0000 7fcaa37fe640 1 --2- 172.21.15.88:0/4185535236 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcaa40b40f0 0x7fcaa40b6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:40:28.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.882+0000 7fcaa2ffd640 1 --2- 172.21.15.88:0/4185535236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcaa415f890 0x7fcaa40b5b30 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fca940097a0 tx=0x7fca9402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f4a948135912ad server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:28.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.882+0000 7fcaa1ffb640 1 -- 172.21.15.88:0/4185535236 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca94037070 con 0x7fcaa415f890 2024-09-16T01:40:28.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.882+0000 7fcaa1ffb640 1 -- 172.21.15.88:0/4185535236 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca94032440 con 0x7fcaa415f890 2024-09-16T01:40:28.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.882+0000 7fcaa1ffb640 1 -- 172.21.15.88:0/4185535236 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca94032810 con 0x7fcaa415f890 2024-09-16T01:40:28.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.882+0000 7fcaa3fff640 1 -- 172.21.15.88:0/4185535236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcaa415f890 msgr2=0x7fcaa40b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:28.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.882+0000 7fcaa3fff640 1 --2- 172.21.15.88:0/4185535236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcaa415f890 0x7fcaa40b5b30 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fca940097a0 tx=0x7fca9402a180 comp rx=0 tx=0).stop 2024-09-16T01:40:28.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.882+0000 7fcaa3fff640 1 -- 172.21.15.88:0/4185535236 shutdown_connections 2024-09-16T01:40:28.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.882+0000 7fcaa3fff640 1 --2- 172.21.15.88:0/4185535236 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcaa40b40f0 0x7fcaa40b6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.882+0000 7fcaa3fff640 1 --2- 172.21.15.88:0/4185535236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcaa415f890 0x7fcaa40b5b30 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.882+0000 7fcaa3fff640 1 -- 172.21.15.88:0/4185535236 >> 172.21.15.88:0/4185535236 conn(0x7fcaa40cdc00 msgr2=0x7fcaa40ce010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:28.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.882+0000 7fcaa3fff640 1 -- 172.21.15.88:0/4185535236 shutdown_connections 2024-09-16T01:40:28.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.882+0000 7fcaa3fff640 1 -- 172.21.15.88:0/4185535236 wait complete. 2024-09-16T01:40:28.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.884+0000 7fcaa3fff640 1 Processor -- start 2024-09-16T01:40:28.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.884+0000 7fcaa3fff640 1 -- start start 2024-09-16T01:40:28.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.885+0000 7fcaa3fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcaa40b40f0 0x7fcaa4143d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.885+0000 7fcaa3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcaa40ce720 0x7fcaa415e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.885+0000 7fcaa3fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcaa415f890 0x7fcaa406dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.885+0000 7fcaa3fff640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcaa415fc70 con 0x7fcaa40ce720 2024-09-16T01:40:28.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.885+0000 7fcaa3fff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcaa40b4fe0 con 0x7fcaa40b40f0 2024-09-16T01:40:28.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.885+0000 7fcaa3fff640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcaa40d0a30 con 0x7fcaa415f890 2024-09-16T01:40:28.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.885+0000 7fcaa27fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcaa40ce720 0x7fcaa415e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:28.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.885+0000 7fcaa27fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcaa40ce720 0x7fcaa415e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38674/0 (socket says 172.21.15.88:38674) 2024-09-16T01:40:28.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.885+0000 7fcaa27fc640 1 -- 172.21.15.88:0/2769456056 learned_addr learned my addr 172.21.15.88:0/2769456056 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:28.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.885+0000 7fcaa37fe640 1 --2- 172.21.15.88:0/2769456056 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcaa415f890 0x7fcaa406dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:28.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.885+0000 7fcaa2ffd640 1 --2- 172.21.15.88:0/2769456056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcaa40b40f0 0x7fcaa4143d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:28.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.885+0000 7fcaa27fc640 1 -- 172.21.15.88:0/2769456056 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcaa415f890 msgr2=0x7fcaa406dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:28.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.885+0000 7fcaa27fc640 1 --2- 172.21.15.88:0/2769456056 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcaa415f890 0x7fcaa406dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.885+0000 7fcaa27fc640 1 -- 172.21.15.88:0/2769456056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcaa40b40f0 msgr2=0x7fcaa4143d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:28.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.885+0000 7fcaa27fc640 1 --2- 172.21.15.88:0/2769456056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcaa40b40f0 0x7fcaa4143d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:28.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.885+0000 7fcaa27fc640 1 -- 172.21.15.88:0/2769456056 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaa4153670 con 0x7fcaa40ce720 2024-09-16T01:40:28.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.885+0000 7fcaa2ffd640 1 --2- 172.21.15.88:0/2769456056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcaa40b40f0 0x7fcaa4143d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:40:28.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.885+0000 7fcaa37fe640 1 --2- 172.21.15.88:0/2769456056 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcaa415f890 0x7fcaa406dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:40:28.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.885+0000 7fcaa27fc640 1 --2- 172.21.15.88:0/2769456056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcaa40ce720 0x7fcaa415e980 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fca9800c430 tx=0x7fca9800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:28.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.885+0000 7fca8bfff640 1 -- 172.21.15.88:0/2769456056 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca98017070 con 0x7fcaa40ce720 2024-09-16T01:40:28.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.886+0000 7fca8bfff640 1 -- 172.21.15.88:0/2769456056 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca98005ce0 con 0x7fcaa40ce720 2024-09-16T01:40:28.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.886+0000 7fcaa3fff640 1 -- 172.21.15.88:0/2769456056 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaa4153950 con 0x7fcaa40ce720 2024-09-16T01:40:28.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.886+0000 7fca8bfff640 1 -- 172.21.15.88:0/2769456056 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca980072f0 con 0x7fcaa40ce720 2024-09-16T01:40:28.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.886+0000 7fcaa3fff640 1 -- 172.21.15.88:0/2769456056 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaa415c670 con 0x7fcaa40ce720 2024-09-16T01:40:28.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.887+0000 7fcaa3fff640 1 -- 172.21.15.88:0/2769456056 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaa40b4800 con 0x7fcaa40ce720 2024-09-16T01:40:28.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.887+0000 7fca8bfff640 1 -- 172.21.15.88:0/2769456056 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fca9801a070 con 0x7fcaa40ce720 2024-09-16T01:40:28.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.888+0000 7fca8bfff640 1 --2- 172.21.15.88:0/2769456056 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fca9006c2b0 0x7fca9006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:28.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.888+0000 7fcaa2ffd640 1 --2- 172.21.15.88:0/2769456056 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fca9006c2b0 0x7fca9006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:28.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.888+0000 7fca8bfff640 1 -- 172.21.15.88:0/2769456056 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 5881+0+0 (secure 0 0 0) 0x7fca9808a7d0 con 0x7fcaa40ce720 2024-09-16T01:40:28.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.888+0000 7fcaa2ffd640 1 --2- 172.21.15.88:0/2769456056 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fca9006c2b0 0x7fca9006e770 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fca94034000 tx=0x7fca9402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:28.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:28.892+0000 7fca8bfff640 1 -- 172.21.15.88:0/2769456056 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fcaa40b4800 con 0x7fcaa40ce720 2024-09-16T01:40:29.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.092+0000 7fcaa3fff640 1 -- 172.21.15.88:0/2769456056 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fcaa4159f10 con 0x7fcaa40ce720 2024-09-16T01:40:29.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.319+0000 7fca8bfff640 1 -- 172.21.15.88:0/2769456056 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v82) ==== 162+0+0 (secure 0 0 0) 0x7fca980587c0 con 0x7fcaa40ce720 2024-09-16T01:40:29.322 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:40:29.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.322+0000 7fcaa3fff640 1 -- 172.21.15.88:0/2769456056 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fca9006c2b0 msgr2=0x7fca9006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:29.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.322+0000 7fcaa3fff640 1 --2- 172.21.15.88:0/2769456056 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fca9006c2b0 0x7fca9006e770 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fca94034000 tx=0x7fca9402e000 comp rx=0 tx=0).stop 2024-09-16T01:40:29.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.322+0000 7fcaa3fff640 1 -- 172.21.15.88:0/2769456056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcaa40ce720 msgr2=0x7fcaa415e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:29.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.322+0000 7fcaa3fff640 1 --2- 172.21.15.88:0/2769456056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcaa40ce720 0x7fcaa415e980 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fca9800c430 tx=0x7fca9800c830 comp rx=0 tx=0).stop 2024-09-16T01:40:29.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.322+0000 7fcaa3fff640 1 -- 172.21.15.88:0/2769456056 shutdown_connections 2024-09-16T01:40:29.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.322+0000 7fcaa3fff640 1 --2- 172.21.15.88:0/2769456056 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fca9006c2b0 0x7fca9006e770 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:29.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.322+0000 7fcaa3fff640 1 --2- 172.21.15.88:0/2769456056 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcaa415f890 0x7fcaa406dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:29.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.322+0000 7fcaa3fff640 1 --2- 172.21.15.88:0/2769456056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcaa40ce720 0x7fcaa415e980 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:29.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.322+0000 7fcaa3fff640 1 --2- 172.21.15.88:0/2769456056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcaa40b40f0 0x7fcaa4143d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:29.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.322+0000 7fcaa3fff640 1 -- 172.21.15.88:0/2769456056 >> 172.21.15.88:0/2769456056 conn(0x7fcaa40cdc00 msgr2=0x7fcaa40cd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:29.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.322+0000 7fcaa3fff640 1 -- 172.21.15.88:0/2769456056 shutdown_connections 2024-09-16T01:40:29.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.322+0000 7fcaa3fff640 1 -- 172.21.15.88:0/2769456056 wait complete. 2024-09-16T01:40:29.337 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:40:29.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.433+0000 7f577a1e8640 1 Processor -- start 2024-09-16T01:40:29.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.433+0000 7f577a1e8640 1 -- start start 2024-09-16T01:40:29.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.434+0000 7f577a1e8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f577406d540 0x7f57740cf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:29.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.434+0000 7f577a1e8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f57740735b0 0x7f57740cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:29.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.434+0000 7f577a1e8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f577406faf0 con 0x7f577415e950 2024-09-16T01:40:29.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.434+0000 7f577a1e8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f577406fdd0 con 0x7f57740735b0 2024-09-16T01:40:29.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.434+0000 7f577a1e8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f57740700b0 con 0x7f577406d540 2024-09-16T01:40:29.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.434+0000 7f5773fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f57740735b0 0x7f57740cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:29.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.434+0000 7f5773fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f57740735b0 0x7f57740cf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38688/0 (socket says 172.21.15.88:38688) 2024-09-16T01:40:29.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.434+0000 7f5773fff640 1 -- 172.21.15.88:0/63991262 learned_addr learned my addr 172.21.15.88:0/63991262 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:29.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.434+0000 7f5772ffd640 1 --2- 172.21.15.88:0/63991262 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f577406d540 0x7f57740cf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:29.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.435+0000 7f5773fff640 1 -- 172.21.15.88:0/63991262 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f577406d540 msgr2=0x7f57740cf0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:29.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.435+0000 7f5773fff640 1 --2- 172.21.15.88:0/63991262 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f577406d540 0x7f57740cf0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:29.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.435+0000 7f5773fff640 1 -- 172.21.15.88:0/63991262 >> v1:172.21.15.154:6789/0 conn(0x7f577415e950 legacy=0x7f577406ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:29.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.435+0000 7f5773fff640 1 -- 172.21.15.88:0/63991262 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5774070390 con 0x7f57740735b0 2024-09-16T01:40:29.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.435+0000 7f5773fff640 1 --2- 172.21.15.88:0/63991262 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57740735b0 0x7f57740cf5e0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f576c009e50 tx=0x7f576c02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f0d0dc427d2f066 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:29.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.435+0000 7f57727fc640 1 -- 172.21.15.88:0/63991262 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576c036070 con 0x7f57740735b0 2024-09-16T01:40:29.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.436+0000 7f57727fc640 1 -- 172.21.15.88:0/63991262 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f576c031600 con 0x7f57740735b0 2024-09-16T01:40:29.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.436+0000 7f57727fc640 1 -- 172.21.15.88:0/63991262 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576c0319d0 con 0x7f57740735b0 2024-09-16T01:40:29.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.436+0000 7f577a1e8640 1 -- 172.21.15.88:0/63991262 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57740735b0 msgr2=0x7f57740cf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:29.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.436+0000 7f577a1e8640 1 --2- 172.21.15.88:0/63991262 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57740735b0 0x7f57740cf5e0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f576c009e50 tx=0x7f576c02a710 comp rx=0 tx=0).stop 2024-09-16T01:40:29.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.436+0000 7f577a1e8640 1 -- 172.21.15.88:0/63991262 shutdown_connections 2024-09-16T01:40:29.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.436+0000 7f577a1e8640 1 --2- 172.21.15.88:0/63991262 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57740735b0 0x7f57740cf5e0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:29.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.436+0000 7f577a1e8640 1 --2- 172.21.15.88:0/63991262 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f577406d540 0x7f57740cf0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:29.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.436+0000 7f577a1e8640 1 -- 172.21.15.88:0/63991262 >> 172.21.15.88:0/63991262 conn(0x7f57740d14d0 msgr2=0x7f57740d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:29.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.436+0000 7f577a1e8640 1 -- 172.21.15.88:0/63991262 shutdown_connections 2024-09-16T01:40:29.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.436+0000 7f577a1e8640 1 -- 172.21.15.88:0/63991262 wait complete. 2024-09-16T01:40:29.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.437+0000 7f577a1e8640 1 Processor -- start 2024-09-16T01:40:29.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.437+0000 7f577a1e8640 1 -- start start 2024-09-16T01:40:29.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.438+0000 7f577a1e8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f577406d540 0x7f5774155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:29.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.438+0000 7f577a1e8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57740735b0 0x7f577415a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:29.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.438+0000 7f577a1e8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f577415e950 0x7f577415faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:29.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.438+0000 7f577a1e8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f57740721d0 con 0x7f57740735b0 2024-09-16T01:40:29.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.438+0000 7f577a1e8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5774072050 con 0x7f577415e950 2024-09-16T01:40:29.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.438+0000 7f577a1e8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5774072350 con 0x7f577406d540 2024-09-16T01:40:29.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.438+0000 7f5772ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57740735b0 0x7f577415a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:29.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.438+0000 7f5772ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57740735b0 0x7f577415a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38690/0 (socket says 172.21.15.88:38690) 2024-09-16T01:40:29.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.438+0000 7f5772ffd640 1 -- 172.21.15.88:0/779859073 learned_addr learned my addr 172.21.15.88:0/779859073 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:29.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.438+0000 7f5773fff640 1 --2- 172.21.15.88:0/779859073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f577415e950 0x7f577415faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:29.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.438+0000 7f57737fe640 1 --2- 172.21.15.88:0/779859073 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f577406d540 0x7f5774155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:29.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.439+0000 7f5772ffd640 1 -- 172.21.15.88:0/779859073 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f577406d540 msgr2=0x7f5774155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:29.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.439+0000 7f5772ffd640 1 --2- 172.21.15.88:0/779859073 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f577406d540 0x7f5774155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:29.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.439+0000 7f5772ffd640 1 -- 172.21.15.88:0/779859073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f577415e950 msgr2=0x7f577415faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:29.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.439+0000 7f5772ffd640 1 --2- 172.21.15.88:0/779859073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f577415e950 0x7f577415faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:29.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.439+0000 7f5772ffd640 1 -- 172.21.15.88:0/779859073 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f577415b150 con 0x7f57740735b0 2024-09-16T01:40:29.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.439+0000 7f57737fe640 1 --2- 172.21.15.88:0/779859073 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f577406d540 0x7f5774155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:29.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.439+0000 7f5772ffd640 1 --2- 172.21.15.88:0/779859073 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57740735b0 0x7f577415a6d0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f576800d530 tx=0x7f576800d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:29.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.439+0000 7f575bfff640 1 -- 172.21.15.88:0/779859073 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57680125f0 con 0x7f57740735b0 2024-09-16T01:40:29.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.439+0000 7f575bfff640 1 -- 172.21.15.88:0/779859073 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5768019070 con 0x7f57740735b0 2024-09-16T01:40:29.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.439+0000 7f575bfff640 1 -- 172.21.15.88:0/779859073 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57680129f0 con 0x7f57740735b0 2024-09-16T01:40:29.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.439+0000 7f577a1e8640 1 -- 172.21.15.88:0/779859073 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f577415ed30 con 0x7f57740735b0 2024-09-16T01:40:29.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.439+0000 7f577a1e8640 1 -- 172.21.15.88:0/779859073 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f577415f2f0 con 0x7f57740735b0 2024-09-16T01:40:29.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.440+0000 7f577a1e8640 1 -- 172.21.15.88:0/779859073 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f577406ce30 con 0x7f57740735b0 2024-09-16T01:40:29.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.440+0000 7f575bfff640 1 -- 172.21.15.88:0/779859073 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5768012b90 con 0x7f57740735b0 2024-09-16T01:40:29.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.441+0000 7f575bfff640 1 --2- 172.21.15.88:0/779859073 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f576006c2b0 0x7f576006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:29.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.441+0000 7f575bfff640 1 -- 172.21.15.88:0/779859073 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 5881+0+0 (secure 0 0 0) 0x7f576808b8a0 con 0x7f57740735b0 2024-09-16T01:40:29.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.441+0000 7f57737fe640 1 --2- 172.21.15.88:0/779859073 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f576006c2b0 0x7f576006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:29.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.441+0000 7f57737fe640 1 --2- 172.21.15.88:0/779859073 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f576006c2b0 0x7f576006e770 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f5774154780 tx=0x7f5764009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:29.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.445+0000 7f575bfff640 1 -- 172.21.15.88:0/779859073 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5768059f60 con 0x7f57740735b0 2024-09-16T01:40:29.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.645+0000 7f577a1e8640 1 -- 172.21.15.88:0/779859073 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f57741419b0 con 0x7f57740735b0 2024-09-16T01:40:29.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.646+0000 7f575bfff640 1 -- 172.21.15.88:0/779859073 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) ==== 94+0+5222 (secure 0 0 0) 0x7f5768059860 con 0x7f57740735b0 2024-09-16T01:40:29.649 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:29.649 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":83,"btime":"2024-09-16T01:40:29:323903+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6849,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/1289932173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1289932173},{"type":"v1","addr":"172.21.15.88:6836","nonce":1289932173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:40:29.323889+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7551},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7551":{"gid":7551,"name":"c","rank":0,"incarnation":83,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6837/3356432718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":3356432718},{"type":"v1","addr":"172.21.15.88:6837","nonce":3356432718}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:40:29.650 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 83 2024-09-16T01:40:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.648+0000 7f577a1e8640 1 -- 172.21.15.88:0/779859073 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f576006c2b0 msgr2=0x7f576006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.648+0000 7f577a1e8640 1 --2- 172.21.15.88:0/779859073 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f576006c2b0 0x7f576006e770 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f5774154780 tx=0x7f5764009040 comp rx=0 tx=0).stop 2024-09-16T01:40:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.648+0000 7f577a1e8640 1 -- 172.21.15.88:0/779859073 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57740735b0 msgr2=0x7f577415a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.648+0000 7f577a1e8640 1 --2- 172.21.15.88:0/779859073 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57740735b0 0x7f577415a6d0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f576800d530 tx=0x7f576800d9f0 comp rx=0 tx=0).stop 2024-09-16T01:40:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.648+0000 7f577a1e8640 1 -- 172.21.15.88:0/779859073 shutdown_connections 2024-09-16T01:40:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.648+0000 7f577a1e8640 1 --2- 172.21.15.88:0/779859073 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f576006c2b0 0x7f576006e770 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.648+0000 7f577a1e8640 1 --2- 172.21.15.88:0/779859073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f577415e950 0x7f577415faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.648+0000 7f577a1e8640 1 --2- 172.21.15.88:0/779859073 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57740735b0 0x7f577415a6d0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.648+0000 7f577a1e8640 1 --2- 172.21.15.88:0/779859073 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f577406d540 0x7f5774155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.648+0000 7f577a1e8640 1 -- 172.21.15.88:0/779859073 >> 172.21.15.88:0/779859073 conn(0x7f57740d14d0 msgr2=0x7f5774141ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.648+0000 7f577a1e8640 1 -- 172.21.15.88:0/779859073 shutdown_connections 2024-09-16T01:40:29.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:29.648+0000 7f577a1e8640 1 -- 172.21.15.88:0/779859073 wait complete. 2024-09-16T01:40:29.662 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-09-16T01:40:32.344 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:40:32.342+0000 7f88b6f27640 -1 log_channel(cluster) log [ERR] : dir 0x600 object missing on disk; some files may be lost (~mds0/stray0) 2024-09-16T01:40:32.344 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:40:32.342+0000 7f88b6f27640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x600 is fatal because it is a system directory for this rank 2024-09-16T01:40:32.345 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:40:32.342+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:40:32.345 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:40:32.342+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:40:33.324 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:40:33.322+0000 7f88baf2f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:40:34.662 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:40:34.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.771+0000 7f547af9a640 1 Processor -- start 2024-09-16T01:40:34.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.771+0000 7f547af9a640 1 -- start start 2024-09-16T01:40:34.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.772+0000 7f547af9a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f54740b4d00 0x7f54740b5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:34.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.772+0000 7f547af9a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54740cd780 0x7f54740cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:34.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.772+0000 7f547af9a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54740ce4f0 con 0x7f54740b4920 2024-09-16T01:40:34.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.772+0000 7f547af9a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f54740cf100 con 0x7f54740b4d00 2024-09-16T01:40:34.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.772+0000 7f547af9a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f54740ce7d0 con 0x7f54740cd780 2024-09-16T01:40:34.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.772+0000 7f5478d36640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f54740b4920 0x7f547415b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46594/0 (socket says 172.21.15.88:46594) 2024-09-16T01:40:34.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.772+0000 7f5478d36640 1 -- 172.21.15.88:0/3320998540 learned_addr learned my addr 172.21.15.88:0/3320998540 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:34.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.772+0000 7f5473fff640 1 --2- 172.21.15.88:0/3320998540 >> v2:172.21.15.154:3300/0 conn(0x7f54740b4d00 0x7f54740b5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:34.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.772+0000 7f5479537640 1 --2- 172.21.15.88:0/3320998540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54740cd780 0x7f54740cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:34.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.772+0000 7f54737fe640 1 -- 172.21.15.88:0/3320998540 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1239151809 0 0) 0x7f54740ce4f0 con 0x7f54740b4920 2024-09-16T01:40:34.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.773+0000 7f54737fe640 1 -- 172.21.15.88:0/3320998540 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f545c003520 con 0x7f54740b4920 2024-09-16T01:40:34.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.773+0000 7f54737fe640 1 -- 172.21.15.88:0/3320998540 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4046194746 0 0) 0x7f545c003520 con 0x7f54740b4920 2024-09-16T01:40:34.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.773+0000 7f5473fff640 1 -- 172.21.15.88:0/3320998540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54740cd780 msgr2=0x7f54740cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:34.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.773+0000 7f5473fff640 1 --2- 172.21.15.88:0/3320998540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54740cd780 0x7f54740cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:34.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.773+0000 7f5473fff640 1 -- 172.21.15.88:0/3320998540 >> v1:172.21.15.88:6789/0 conn(0x7f54740b4920 legacy=0x7f547415b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:34.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.773+0000 7f5473fff640 1 -- 172.21.15.88:0/3320998540 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54740ceab0 con 0x7f54740b4d00 2024-09-16T01:40:34.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.774+0000 7f5473fff640 1 --2- 172.21.15.88:0/3320998540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54740b4d00 0x7f54740b5170 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f54680097a0 tx=0x7f546802a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ac5bbb9ec38cee1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:34.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.774+0000 7f54737fe640 1 -- 172.21.15.88:0/3320998540 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5468037070 con 0x7f54740b4d00 2024-09-16T01:40:34.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.774+0000 7f54737fe640 1 -- 172.21.15.88:0/3320998540 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f546802aba0 con 0x7f54740b4d00 2024-09-16T01:40:34.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.774+0000 7f54737fe640 1 -- 172.21.15.88:0/3320998540 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5464001670 con 0x7f54740b4d00 2024-09-16T01:40:34.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.774+0000 7f547af9a640 1 -- 172.21.15.88:0/3320998540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54740b4d00 msgr2=0x7f54740b5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:34.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.774+0000 7f547af9a640 1 --2- 172.21.15.88:0/3320998540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54740b4d00 0x7f54740b5170 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f54680097a0 tx=0x7f546802a110 comp rx=0 tx=0).stop 2024-09-16T01:40:34.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.774+0000 7f547af9a640 1 -- 172.21.15.88:0/3320998540 shutdown_connections 2024-09-16T01:40:34.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.774+0000 7f547af9a640 1 --2- 172.21.15.88:0/3320998540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54740cd780 0x7f54740cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:34.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.774+0000 7f547af9a640 1 --2- 172.21.15.88:0/3320998540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54740b4d00 0x7f54740b5170 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:34.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.774+0000 7f547af9a640 1 -- 172.21.15.88:0/3320998540 >> 172.21.15.88:0/3320998540 conn(0x7f547406bc10 msgr2=0x7f547406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:34.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.774+0000 7f547af9a640 1 -- 172.21.15.88:0/3320998540 shutdown_connections 2024-09-16T01:40:34.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.774+0000 7f547af9a640 1 -- 172.21.15.88:0/3320998540 wait complete. 2024-09-16T01:40:34.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.776+0000 7f547af9a640 1 Processor -- start 2024-09-16T01:40:34.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.776+0000 7f547af9a640 1 -- start start 2024-09-16T01:40:34.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.777+0000 7f547af9a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54740b4920 0x7f54741f0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:34.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.777+0000 7f547af9a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54740b4d00 0x7f5474153200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:34.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.777+0000 7f547af9a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54740cd780 0x7f54741f3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:34.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.777+0000 7f547af9a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f54740ce110 con 0x7f54740b4920 2024-09-16T01:40:34.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.777+0000 7f547af9a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f54740b56b0 con 0x7f54740cd780 2024-09-16T01:40:34.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.777+0000 7f547af9a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f54740ce290 con 0x7f54740b4d00 2024-09-16T01:40:34.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.777+0000 7f5478d36640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54740b4920 0x7f54741f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:34.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.777+0000 7f5478d36640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54740b4920 0x7f54741f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44038/0 (socket says 172.21.15.88:44038) 2024-09-16T01:40:34.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.777+0000 7f5478d36640 1 -- 172.21.15.88:0/3873016094 learned_addr learned my addr 172.21.15.88:0/3873016094 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:34.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.777+0000 7f5478d36640 1 -- 172.21.15.88:0/3873016094 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54740b4d00 msgr2=0x7f5474153200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:34.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.777+0000 7f5478d36640 1 --2- 172.21.15.88:0/3873016094 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54740b4d00 0x7f5474153200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:34.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.777+0000 7f5478d36640 1 -- 172.21.15.88:0/3873016094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54740cd780 msgr2=0x7f54741f3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:34.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.777+0000 7f5478d36640 1 --2- 172.21.15.88:0/3873016094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54740cd780 0x7f54741f3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:34.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.777+0000 7f5478d36640 1 -- 172.21.15.88:0/3873016094 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54741f0970 con 0x7f54740b4920 2024-09-16T01:40:34.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.777+0000 7f5478d36640 1 --2- 172.21.15.88:0/3873016094 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54740b4920 0x7f54741f0430 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f546400c7b0 tx=0x7f546400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:34.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.777+0000 7f54717fa640 1 -- 172.21.15.88:0/3873016094 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5464017070 con 0x7f54740b4920 2024-09-16T01:40:34.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.777+0000 7f54717fa640 1 -- 172.21.15.88:0/3873016094 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5464005ad0 con 0x7f54740b4920 2024-09-16T01:40:34.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.777+0000 7f54717fa640 1 -- 172.21.15.88:0/3873016094 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5464007070 con 0x7f54740b4920 2024-09-16T01:40:34.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.777+0000 7f547af9a640 1 -- 172.21.15.88:0/3873016094 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54741f39f0 con 0x7f54740b4920 2024-09-16T01:40:34.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.777+0000 7f547af9a640 1 -- 172.21.15.88:0/3873016094 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54741efc30 con 0x7f54740b4920 2024-09-16T01:40:34.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.778+0000 7f547af9a640 1 -- 172.21.15.88:0/3873016094 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54740b5170 con 0x7f54740b4920 2024-09-16T01:40:34.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.779+0000 7f54717fa640 1 -- 172.21.15.88:0/3873016094 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f546401a070 con 0x7f54740b4920 2024-09-16T01:40:34.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.779+0000 7f54717fa640 1 --2- 172.21.15.88:0/3873016094 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f546006c2b0 0x7f546006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:34.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.779+0000 7f54717fa640 1 -- 172.21.15.88:0/3873016094 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5967+0+0 (secure 0 0 0) 0x7f546408a9a0 con 0x7f54740b4920 2024-09-16T01:40:34.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.779+0000 7f5473fff640 1 --2- 172.21.15.88:0/3873016094 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f546006c2b0 0x7f546006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:34.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.780+0000 7f5473fff640 1 --2- 172.21.15.88:0/3873016094 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f546006c2b0 0x7f546006e770 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5468034000 tx=0x7f546802c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:34.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.784+0000 7f54717fa640 1 -- 172.21.15.88:0/3873016094 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f54741f0970 con 0x7f54740b4920 2024-09-16T01:40:34.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.982+0000 7f547af9a640 1 -- 172.21.15.88:0/3873016094 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5474157070 con 0x7f54740b4920 2024-09-16T01:40:34.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.982+0000 7f54717fa640 1 -- 172.21.15.88:0/3873016094 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+4388 (secure 0 0 0) 0x7f54640590a0 con 0x7f54740b4920 2024-09-16T01:40:34.985 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:34.985 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":87,"btime":"2024-09-16T01:40:32:344463+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6849,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/1289932173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1289932173},{"type":"v1","addr":"172.21.15.88:6836","nonce":1289932173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:40:32.344458+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:40:34.985 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 87 2024-09-16T01:40:34.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.985+0000 7f547af9a640 1 -- 172.21.15.88:0/3873016094 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f546006c2b0 msgr2=0x7f546006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:34.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.985+0000 7f547af9a640 1 --2- 172.21.15.88:0/3873016094 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f546006c2b0 0x7f546006e770 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5468034000 tx=0x7f546802c000 comp rx=0 tx=0).stop 2024-09-16T01:40:34.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.985+0000 7f547af9a640 1 -- 172.21.15.88:0/3873016094 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54740b4920 msgr2=0x7f54741f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:34.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.985+0000 7f547af9a640 1 --2- 172.21.15.88:0/3873016094 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54740b4920 0x7f54741f0430 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f546400c7b0 tx=0x7f546400cc70 comp rx=0 tx=0).stop 2024-09-16T01:40:34.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.985+0000 7f547af9a640 1 -- 172.21.15.88:0/3873016094 shutdown_connections 2024-09-16T01:40:34.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.985+0000 7f547af9a640 1 --2- 172.21.15.88:0/3873016094 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f546006c2b0 0x7f546006e770 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:34.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.985+0000 7f547af9a640 1 --2- 172.21.15.88:0/3873016094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54740cd780 0x7f54741f3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:34.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.985+0000 7f547af9a640 1 --2- 172.21.15.88:0/3873016094 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54740b4d00 0x7f5474153200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:34.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.985+0000 7f547af9a640 1 --2- 172.21.15.88:0/3873016094 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54740b4920 0x7f54741f0430 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:34.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.985+0000 7f547af9a640 1 -- 172.21.15.88:0/3873016094 >> 172.21.15.88:0/3873016094 conn(0x7f547406bc10 msgr2=0x7f54740b5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:34.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.985+0000 7f547af9a640 1 -- 172.21.15.88:0/3873016094 shutdown_connections 2024-09-16T01:40:34.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:34.985+0000 7f547af9a640 1 -- 172.21.15.88:0/3873016094 wait complete. 2024-09-16T01:40:34.998 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-09-16T01:40:34.998 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:40:35.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.094+0000 7fd62e4d7640 1 Processor -- start 2024-09-16T01:40:35.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.094+0000 7fd62e4d7640 1 -- start start 2024-09-16T01:40:35.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.095+0000 7fd62e4d7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd62815c550 0x7fd6280b5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:35.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.095+0000 7fd62e4d7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6280b4cf0 con 0x7fd6280b6310 2024-09-16T01:40:35.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.095+0000 7fd62e4d7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6280b4fd0 con 0x7fd62815c170 2024-09-16T01:40:35.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.095+0000 7fd62e4d7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd6280b52b0 con 0x7fd62815c550 2024-09-16T01:40:35.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.095+0000 7fd6277fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd62815c550 0x7fd6280b5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:35.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.095+0000 7fd6277fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd62815c550 0x7fd6280b5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44050/0 (socket says 172.21.15.88:44050) 2024-09-16T01:40:35.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.095+0000 7fd62ca74640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd6280b6310 0x7fd6280b39b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46602/0 (socket says 172.21.15.88:46602) 2024-09-16T01:40:35.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.095+0000 7fd6277fe640 1 -- 172.21.15.88:0/4071858797 learned_addr learned my addr 172.21.15.88:0/4071858797 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:35.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.095+0000 7fd6277fe640 1 -- 172.21.15.88:0/4071858797 >> v1:172.21.15.154:6789/0 conn(0x7fd62815c170 legacy=0x7fd6280cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:35.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.095+0000 7fd6277fe640 1 -- 172.21.15.88:0/4071858797 >> v1:172.21.15.88:6789/0 conn(0x7fd6280b6310 legacy=0x7fd6280b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:35.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.095+0000 7fd6277fe640 1 -- 172.21.15.88:0/4071858797 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6280b5590 con 0x7fd62815c550 2024-09-16T01:40:35.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.095+0000 7fd626ffd640 1 -- 172.21.15.88:0/4071858797 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2955201732 0 0) 0x7fd6280b4cf0 con 0x7fd6280b6310 2024-09-16T01:40:35.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.096+0000 7fd6277fe640 1 --2- 172.21.15.88:0/4071858797 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd62815c550 0x7fd6280b5dd0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fd61c0098a0 tx=0x7fd61c02a2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19ff62c0335b6977 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:35.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.096+0000 7fd626ffd640 1 -- 172.21.15.88:0/4071858797 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd61c037070 con 0x7fd62815c550 2024-09-16T01:40:35.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.096+0000 7fd626ffd640 1 -- 172.21.15.88:0/4071858797 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd61c032440 con 0x7fd62815c550 2024-09-16T01:40:35.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.096+0000 7fd626ffd640 1 -- 172.21.15.88:0/4071858797 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd61c032810 con 0x7fd62815c550 2024-09-16T01:40:35.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.097+0000 7fd62e4d7640 1 -- 172.21.15.88:0/4071858797 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd62815c550 msgr2=0x7fd6280b5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:35.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.097+0000 7fd62e4d7640 1 --2- 172.21.15.88:0/4071858797 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd62815c550 0x7fd6280b5dd0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fd61c0098a0 tx=0x7fd61c02a2e0 comp rx=0 tx=0).stop 2024-09-16T01:40:35.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.097+0000 7fd62e4d7640 1 -- 172.21.15.88:0/4071858797 shutdown_connections 2024-09-16T01:40:35.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.097+0000 7fd62e4d7640 1 --2- 172.21.15.88:0/4071858797 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd62815c550 0x7fd6280b5dd0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:35.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.097+0000 7fd62e4d7640 1 -- 172.21.15.88:0/4071858797 >> 172.21.15.88:0/4071858797 conn(0x7fd62806bc10 msgr2=0x7fd62806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:35.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.097+0000 7fd62e4d7640 1 -- 172.21.15.88:0/4071858797 shutdown_connections 2024-09-16T01:40:35.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.097+0000 7fd62e4d7640 1 -- 172.21.15.88:0/4071858797 wait complete. 2024-09-16T01:40:35.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.098+0000 7fd62e4d7640 1 Processor -- start 2024-09-16T01:40:35.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.098+0000 7fd62e4d7640 1 -- start start 2024-09-16T01:40:35.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.099+0000 7fd62e4d7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6280b6310 0x7fd62814b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:35.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.099+0000 7fd62e4d7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd62815c170 0x7fd62815db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:35.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.099+0000 7fd62e4d7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd62815c550 0x7fd62815a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:35.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.099+0000 7fd62e4d7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd62815c9c0 con 0x7fd6280b6310 2024-09-16T01:40:35.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.099+0000 7fd62e4d7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd6280b5b50 con 0x7fd62815c550 2024-09-16T01:40:35.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.099+0000 7fd62e4d7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd6280cfdd0 con 0x7fd62815c170 2024-09-16T01:40:35.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.099+0000 7fd627fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6280b6310 0x7fd62814b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:35.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.099+0000 7fd627fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6280b6310 0x7fd62814b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44060/0 (socket says 172.21.15.88:44060) 2024-09-16T01:40:35.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.099+0000 7fd627fff640 1 -- 172.21.15.88:0/4140792689 learned_addr learned my addr 172.21.15.88:0/4140792689 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:35.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.099+0000 7fd62ca74640 1 --2- 172.21.15.88:0/4140792689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd62815c550 0x7fd62815a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:35.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.099+0000 7fd6277fe640 1 --2- 172.21.15.88:0/4140792689 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd62815c170 0x7fd62815db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:35.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.099+0000 7fd627fff640 1 -- 172.21.15.88:0/4140792689 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd62815c170 msgr2=0x7fd62815db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:35.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.099+0000 7fd627fff640 1 --2- 172.21.15.88:0/4140792689 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd62815c170 0x7fd62815db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:35.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.099+0000 7fd627fff640 1 -- 172.21.15.88:0/4140792689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd62815c550 msgr2=0x7fd62815a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:35.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.099+0000 7fd627fff640 1 --2- 172.21.15.88:0/4140792689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd62815c550 0x7fd62815a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:35.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.099+0000 7fd627fff640 1 -- 172.21.15.88:0/4140792689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd628158370 con 0x7fd6280b6310 2024-09-16T01:40:35.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.099+0000 7fd627fff640 1 --2- 172.21.15.88:0/4140792689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6280b6310 0x7fd62814b0f0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fd61800c460 tx=0x7fd61800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:35.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.099+0000 7fd624ff9640 1 -- 172.21.15.88:0/4140792689 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd618016020 con 0x7fd6280b6310 2024-09-16T01:40:35.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.100+0000 7fd624ff9640 1 -- 172.21.15.88:0/4140792689 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd618012870 con 0x7fd6280b6310 2024-09-16T01:40:35.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.100+0000 7fd624ff9640 1 -- 172.21.15.88:0/4140792689 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd618012c60 con 0x7fd6280b6310 2024-09-16T01:40:35.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.100+0000 7fd62e4d7640 1 -- 172.21.15.88:0/4140792689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd62815cd90 con 0x7fd6280b6310 2024-09-16T01:40:35.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.100+0000 7fd62e4d7640 1 -- 172.21.15.88:0/4140792689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd62815d350 con 0x7fd6280b6310 2024-09-16T01:40:35.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.100+0000 7fd62e4d7640 1 -- 172.21.15.88:0/4140792689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6280b5dd0 con 0x7fd6280b6310 2024-09-16T01:40:35.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.101+0000 7fd624ff9640 1 -- 172.21.15.88:0/4140792689 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd618019020 con 0x7fd6280b6310 2024-09-16T01:40:35.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.101+0000 7fd624ff9640 1 --2- 172.21.15.88:0/4140792689 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd61406c2b0 0x7fd61406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:35.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.102+0000 7fd624ff9640 1 -- 172.21.15.88:0/4140792689 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5967+0+0 (secure 0 0 0) 0x7fd61808b1d0 con 0x7fd6280b6310 2024-09-16T01:40:35.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.102+0000 7fd6277fe640 1 --2- 172.21.15.88:0/4140792689 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd61406c2b0 0x7fd61406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:35.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.102+0000 7fd6277fe640 1 --2- 172.21.15.88:0/4140792689 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd61406c2b0 0x7fd61406e770 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd61c034000 tx=0x7fd61c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:35.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.107+0000 7fd624ff9640 1 -- 172.21.15.88:0/4140792689 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd618020030 con 0x7fd6280b6310 2024-09-16T01:40:35.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.307+0000 7fd62e4d7640 1 -- 172.21.15.88:0/4140792689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6281522d0 con 0x7fd6280b6310 2024-09-16T01:40:35.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.308+0000 7fd624ff9640 1 -- 172.21.15.88:0/4140792689 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+4388 (secure 0 0 0) 0x7fd628158370 con 0x7fd6280b6310 2024-09-16T01:40:35.311 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:35.311 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":87,"btime":"2024-09-16T01:40:32:344463+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6849,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/1289932173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1289932173},{"type":"v1","addr":"172.21.15.88:6836","nonce":1289932173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:40:32.344458+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:40:35.311 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 87 2024-09-16T01:40:35.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.310+0000 7fd62e4d7640 1 -- 172.21.15.88:0/4140792689 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd61406c2b0 msgr2=0x7fd61406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:35.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.310+0000 7fd62e4d7640 1 --2- 172.21.15.88:0/4140792689 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd61406c2b0 0x7fd61406e770 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd61c034000 tx=0x7fd61c02e000 comp rx=0 tx=0).stop 2024-09-16T01:40:35.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.310+0000 7fd62e4d7640 1 -- 172.21.15.88:0/4140792689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6280b6310 msgr2=0x7fd62814b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:35.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.310+0000 7fd62e4d7640 1 --2- 172.21.15.88:0/4140792689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6280b6310 0x7fd62814b0f0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fd61800c460 tx=0x7fd61800c920 comp rx=0 tx=0).stop 2024-09-16T01:40:35.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.310+0000 7fd62e4d7640 1 -- 172.21.15.88:0/4140792689 shutdown_connections 2024-09-16T01:40:35.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.310+0000 7fd62e4d7640 1 --2- 172.21.15.88:0/4140792689 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd61406c2b0 0x7fd61406e770 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:35.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.310+0000 7fd62e4d7640 1 --2- 172.21.15.88:0/4140792689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd62815c550 0x7fd62815a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:35.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.310+0000 7fd62e4d7640 1 --2- 172.21.15.88:0/4140792689 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd62815c170 0x7fd62815db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:35.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.310+0000 7fd62e4d7640 1 --2- 172.21.15.88:0/4140792689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd6280b6310 0x7fd62814b0f0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:35.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.310+0000 7fd62e4d7640 1 -- 172.21.15.88:0/4140792689 >> 172.21.15.88:0/4140792689 conn(0x7fd62806bc10 msgr2=0x7fd62806d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:35.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.310+0000 7fd62e4d7640 1 -- 172.21.15.88:0/4140792689 shutdown_connections 2024-09-16T01:40:35.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.310+0000 7fd62e4d7640 1 -- 172.21.15.88:0/4140792689 wait complete. 2024-09-16T01:40:35.324 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:40:35.324 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:40:35.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.420+0000 7fe53d5e4640 1 Processor -- start 2024-09-16T01:40:35.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.420+0000 7fe53d5e4640 1 -- start start 2024-09-16T01:40:35.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.421+0000 7fe53d5e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe53815c550 0x7fe5380715e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:35.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.421+0000 7fe53d5e4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe5380b6e10 0x7fe5380b5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:35.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.421+0000 7fe53d5e4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe538073160 0x7fe5380735d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:35.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.421+0000 7fe53d5e4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe538073b10 con 0x7fe538073160 2024-09-16T01:40:35.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.421+0000 7fe53d5e4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe53815c930 con 0x7fe5380b6e10 2024-09-16T01:40:35.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.421+0000 7fe53d5e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe5380d0270 con 0x7fe53815c550 2024-09-16T01:40:35.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.421+0000 7fe5377fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe538073160 0x7fe5380735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:35.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.421+0000 7fe5377fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe538073160 0x7fe5380735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44062/0 (socket says 172.21.15.88:44062) 2024-09-16T01:40:35.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.421+0000 7fe5377fe640 1 -- 172.21.15.88:0/3293769785 learned_addr learned my addr 172.21.15.88:0/3293769785 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:35.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.421+0000 7fe52e7fc640 1 --2- 172.21.15.88:0/3293769785 >> v2:172.21.15.154:3300/0 conn(0x7fe5380b6e10 0x7fe5380b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:35.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.421+0000 7fe536ffd640 1 --2- 172.21.15.88:0/3293769785 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe53815c550 0x7fe5380715e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:35.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.422+0000 7fe5377fe640 1 -- 172.21.15.88:0/3293769785 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe53815c550 msgr2=0x7fe5380715e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:35.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.422+0000 7fe5377fe640 1 --2- 172.21.15.88:0/3293769785 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe53815c550 0x7fe5380715e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:35.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.422+0000 7fe5377fe640 1 -- 172.21.15.88:0/3293769785 >> v2:172.21.15.154:3300/0 conn(0x7fe5380b6e10 msgr2=0x7fe5380b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:35.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.422+0000 7fe5377fe640 1 --2- 172.21.15.88:0/3293769785 >> v2:172.21.15.154:3300/0 conn(0x7fe5380b6e10 0x7fe5380b5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:35.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.422+0000 7fe5377fe640 1 -- 172.21.15.88:0/3293769785 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5380d04f0 con 0x7fe538073160 2024-09-16T01:40:35.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.422+0000 7fe5377fe640 1 --2- 172.21.15.88:0/3293769785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe538073160 0x7fe5380735d0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fe5380d0420 tx=0x7fe53002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf6d7bb2fbfae10 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:35.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.422+0000 7fe5367fc640 1 -- 172.21.15.88:0/3293769785 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe530036070 con 0x7fe538073160 2024-09-16T01:40:35.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.422+0000 7fe5367fc640 1 -- 172.21.15.88:0/3293769785 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe530031600 con 0x7fe538073160 2024-09-16T01:40:35.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.422+0000 7fe5367fc640 1 -- 172.21.15.88:0/3293769785 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5300319d0 con 0x7fe538073160 2024-09-16T01:40:35.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.423+0000 7fe53d5e4640 1 -- 172.21.15.88:0/3293769785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe538073160 msgr2=0x7fe5380735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:35.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.423+0000 7fe53d5e4640 1 --2- 172.21.15.88:0/3293769785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe538073160 0x7fe5380735d0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fe5380d0420 tx=0x7fe53002a680 comp rx=0 tx=0).stop 2024-09-16T01:40:35.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.423+0000 7fe53d5e4640 1 -- 172.21.15.88:0/3293769785 shutdown_connections 2024-09-16T01:40:35.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.423+0000 7fe53d5e4640 1 --2- 172.21.15.88:0/3293769785 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe538073160 0x7fe5380735d0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:35.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.423+0000 7fe53d5e4640 1 --2- 172.21.15.88:0/3293769785 >> v2:172.21.15.154:3300/0 conn(0x7fe5380b6e10 0x7fe5380b5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:35.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.423+0000 7fe53d5e4640 1 --2- 172.21.15.88:0/3293769785 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe53815c550 0x7fe5380715e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:35.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.423+0000 7fe53d5e4640 1 -- 172.21.15.88:0/3293769785 >> 172.21.15.88:0/3293769785 conn(0x7fe5380b4240 msgr2=0x7fe5380b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:35.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.423+0000 7fe53d5e4640 1 -- 172.21.15.88:0/3293769785 shutdown_connections 2024-09-16T01:40:35.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.423+0000 7fe53d5e4640 1 -- 172.21.15.88:0/3293769785 wait complete. 2024-09-16T01:40:35.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.424+0000 7fe53d5e4640 1 Processor -- start 2024-09-16T01:40:35.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.424+0000 7fe53d5e4640 1 -- start start 2024-09-16T01:40:35.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.425+0000 7fe53d5e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe538073160 0x7fe53806d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:35.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.425+0000 7fe53d5e4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5380b6e10 0x7fe538149f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:35.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.425+0000 7fe53d5e4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe53815c550 0x7fe53815a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:35.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.425+0000 7fe53d5e4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe5380d07b0 con 0x7fe53815c550 2024-09-16T01:40:35.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.425+0000 7fe53d5e4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe5380b7860 con 0x7fe5380b6e10 2024-09-16T01:40:35.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.425+0000 7fe53d5e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe5380b7250 con 0x7fe538073160 2024-09-16T01:40:35.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.425+0000 7fe5377fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe53815c550 0x7fe53815a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:35.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.425+0000 7fe5377fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe53815c550 0x7fe53815a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44074/0 (socket says 172.21.15.88:44074) 2024-09-16T01:40:35.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.425+0000 7fe5377fe640 1 -- 172.21.15.88:0/2785523576 learned_addr learned my addr 172.21.15.88:0/2785523576 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:35.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.425+0000 7fe52e7fc640 1 --2- 172.21.15.88:0/2785523576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5380b6e10 0x7fe538149f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:35.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.425+0000 7fe536ffd640 1 --2- 172.21.15.88:0/2785523576 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe538073160 0x7fe53806d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:35.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.426+0000 7fe5377fe640 1 -- 172.21.15.88:0/2785523576 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe538073160 msgr2=0x7fe53806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:35.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.426+0000 7fe5377fe640 1 --2- 172.21.15.88:0/2785523576 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe538073160 0x7fe53806d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:35.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.426+0000 7fe5377fe640 1 -- 172.21.15.88:0/2785523576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5380b6e10 msgr2=0x7fe538149f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:35.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.426+0000 7fe5377fe640 1 --2- 172.21.15.88:0/2785523576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5380b6e10 0x7fe538149f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:35.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.426+0000 7fe5377fe640 1 -- 172.21.15.88:0/2785523576 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe53806dc40 con 0x7fe53815c550 2024-09-16T01:40:35.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.426+0000 7fe5377fe640 1 --2- 172.21.15.88:0/2785523576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe53815c550 0x7fe53815a9c0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fe530034040 tx=0x7fe530004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:35.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.426+0000 7fe52ffff640 1 -- 172.21.15.88:0/2785523576 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe530036050 con 0x7fe53815c550 2024-09-16T01:40:35.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.426+0000 7fe52ffff640 1 -- 172.21.15.88:0/2785523576 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe530042070 con 0x7fe53815c550 2024-09-16T01:40:35.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.426+0000 7fe53d5e4640 1 -- 172.21.15.88:0/2785523576 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe53806e410 con 0x7fe53815c550 2024-09-16T01:40:35.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.426+0000 7fe52ffff640 1 -- 172.21.15.88:0/2785523576 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe530009510 con 0x7fe53815c550 2024-09-16T01:40:35.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.426+0000 7fe53d5e4640 1 -- 172.21.15.88:0/2785523576 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5381f7a60 con 0x7fe53815c550 2024-09-16T01:40:35.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.427+0000 7fe53d5e4640 1 -- 172.21.15.88:0/2785523576 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5380735d0 con 0x7fe53815c550 2024-09-16T01:40:35.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.427+0000 7fe52ffff640 1 -- 172.21.15.88:0/2785523576 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe530007bd0 con 0x7fe53815c550 2024-09-16T01:40:35.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.428+0000 7fe52ffff640 1 --2- 172.21.15.88:0/2785523576 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe51806c2b0 0x7fe51806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:35.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.428+0000 7fe52ffff640 1 -- 172.21.15.88:0/2785523576 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5967+0+0 (secure 0 0 0) 0x7fe5300ac1c0 con 0x7fe53815c550 2024-09-16T01:40:35.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.428+0000 7fe536ffd640 1 --2- 172.21.15.88:0/2785523576 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe51806c2b0 0x7fe51806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:35.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.428+0000 7fe536ffd640 1 --2- 172.21.15.88:0/2785523576 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe51806c2b0 0x7fe51806e770 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fe5280079a0 tx=0x7fe528008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:35.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.432+0000 7fe52ffff640 1 -- 172.21.15.88:0/2785523576 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe53007a270 con 0x7fe53815c550 2024-09-16T01:40:35.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.632+0000 7fe53d5e4640 1 -- 172.21.15.88:0/2785523576 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5380d20b0 con 0x7fe53815c550 2024-09-16T01:40:35.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.632+0000 7fe52ffff640 1 -- 172.21.15.88:0/2785523576 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+4388 (secure 0 0 0) 0x7fe53007e570 con 0x7fe53815c550 2024-09-16T01:40:35.634 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:35.634 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":87,"btime":"2024-09-16T01:40:32:344463+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6849,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/1289932173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1289932173},{"type":"v1","addr":"172.21.15.88:6836","nonce":1289932173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:40:32.344458+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:40:35.634 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 87 2024-09-16T01:40:35.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.634+0000 7fe53d5e4640 1 -- 172.21.15.88:0/2785523576 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe51806c2b0 msgr2=0x7fe51806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:35.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.634+0000 7fe53d5e4640 1 --2- 172.21.15.88:0/2785523576 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe51806c2b0 0x7fe51806e770 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fe5280079a0 tx=0x7fe528008040 comp rx=0 tx=0).stop 2024-09-16T01:40:35.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.634+0000 7fe53d5e4640 1 -- 172.21.15.88:0/2785523576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe53815c550 msgr2=0x7fe53815a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:35.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.634+0000 7fe53d5e4640 1 --2- 172.21.15.88:0/2785523576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe53815c550 0x7fe53815a9c0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fe530034040 tx=0x7fe530004a40 comp rx=0 tx=0).stop 2024-09-16T01:40:35.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.634+0000 7fe53d5e4640 1 -- 172.21.15.88:0/2785523576 shutdown_connections 2024-09-16T01:40:35.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.634+0000 7fe53d5e4640 1 --2- 172.21.15.88:0/2785523576 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe51806c2b0 0x7fe51806e770 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:35.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.634+0000 7fe53d5e4640 1 --2- 172.21.15.88:0/2785523576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe53815c550 0x7fe53815a9c0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:35.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.634+0000 7fe53d5e4640 1 --2- 172.21.15.88:0/2785523576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5380b6e10 0x7fe538149f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:35.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.634+0000 7fe53d5e4640 1 --2- 172.21.15.88:0/2785523576 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe538073160 0x7fe53806d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:35.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.634+0000 7fe53d5e4640 1 -- 172.21.15.88:0/2785523576 >> 172.21.15.88:0/2785523576 conn(0x7fe5380b4240 msgr2=0x7fe5380b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:35.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.634+0000 7fe53d5e4640 1 -- 172.21.15.88:0/2785523576 shutdown_connections 2024-09-16T01:40:35.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.634+0000 7fe53d5e4640 1 -- 172.21.15.88:0/2785523576 wait complete. 2024-09-16T01:40:35.647 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 600.00000000' led to DAMAGED state 2024-09-16T01:40:35.647 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 1.00000000.inode' 2024-09-16T01:40:35.647 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:40:35.647 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:40:35.663 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:40:35.663 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:40:35.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:35.663 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:40:35.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:40:35.719 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:40:35.719 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:40:35.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.856+0000 7f16c7931640 1 Processor -- start 2024-09-16T01:40:35.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.857+0000 7f16c7931640 1 -- start start 2024-09-16T01:40:35.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.857+0000 7f16c7931640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f16c015e7a0 0x7f16c006caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:35.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.857+0000 7f16c7931640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f16c00b68f0 0x7f16c00d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:35.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.857+0000 7f16c7931640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16c00cf860 con 0x7f16c00cf1f0 2024-09-16T01:40:35.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.857+0000 7f16c7931640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f16c00d2ca0 con 0x7f16c00b68f0 2024-09-16T01:40:35.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.857+0000 7f16c7931640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f16c00cfb40 con 0x7f16c015e7a0 2024-09-16T01:40:35.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.858+0000 7f16c5ece640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f16c00b68f0 0x7f16c00d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:35.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.858+0000 7f16c5ece640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f16c00b68f0 0x7f16c00d3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44082/0 (socket says 172.21.15.88:44082) 2024-09-16T01:40:35.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.858+0000 7f16c4ecc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f16c00cf1f0 0x7f16c00b61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46606/0 (socket says 172.21.15.88:46606) 2024-09-16T01:40:35.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.858+0000 7f16c5ece640 1 -- 172.21.15.88:0/3709709649 learned_addr learned my addr 172.21.15.88:0/3709709649 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:35.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.858+0000 7f16c56cd640 1 --2- 172.21.15.88:0/3709709649 >> v2:172.21.15.154:3300/0 conn(0x7f16c015e7a0 0x7f16c006caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:35.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.858+0000 7f16c5ece640 1 -- 172.21.15.88:0/3709709649 >> v2:172.21.15.154:3300/0 conn(0x7f16c015e7a0 msgr2=0x7f16c006caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:35.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.858+0000 7f16c5ece640 1 --2- 172.21.15.88:0/3709709649 >> v2:172.21.15.154:3300/0 conn(0x7f16c015e7a0 0x7f16c006caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:35.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.858+0000 7f16c5ece640 1 -- 172.21.15.88:0/3709709649 >> v1:172.21.15.88:6789/0 conn(0x7f16c00cf1f0 legacy=0x7f16c00b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:35.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.858+0000 7f16c5ece640 1 -- 172.21.15.88:0/3709709649 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c00cfe20 con 0x7f16c00b68f0 2024-09-16T01:40:35.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.858+0000 7f16bffff640 1 -- 172.21.15.88:0/3709709649 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1676400343 0 0) 0x7f16c00cf860 con 0x7f16c00cf1f0 2024-09-16T01:40:35.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.859+0000 7f16c5ece640 1 --2- 172.21.15.88:0/3709709649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16c00b68f0 0x7f16c00d3410 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f16c00d5820 tx=0x7f16b802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c0bad4f35103650 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:35.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.859+0000 7f16bffff640 1 -- 172.21.15.88:0/3709709649 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b8037070 con 0x7f16c00b68f0 2024-09-16T01:40:35.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.859+0000 7f16bffff640 1 -- 172.21.15.88:0/3709709649 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16b8032440 con 0x7f16c00b68f0 2024-09-16T01:40:35.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.859+0000 7f16bffff640 1 -- 172.21.15.88:0/3709709649 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b8032810 con 0x7f16c00b68f0 2024-09-16T01:40:35.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.859+0000 7f16c7931640 1 -- 172.21.15.88:0/3709709649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16c00b68f0 msgr2=0x7f16c00d3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:35.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.859+0000 7f16c7931640 1 --2- 172.21.15.88:0/3709709649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16c00b68f0 0x7f16c00d3410 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f16c00d5820 tx=0x7f16b802a1e0 comp rx=0 tx=0).stop 2024-09-16T01:40:35.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.859+0000 7f16c7931640 1 -- 172.21.15.88:0/3709709649 shutdown_connections 2024-09-16T01:40:35.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.859+0000 7f16c7931640 1 --2- 172.21.15.88:0/3709709649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16c00b68f0 0x7f16c00d3410 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:35.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.859+0000 7f16c7931640 1 --2- 172.21.15.88:0/3709709649 >> v2:172.21.15.154:3300/0 conn(0x7f16c015e7a0 0x7f16c006caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:35.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.859+0000 7f16c7931640 1 -- 172.21.15.88:0/3709709649 >> 172.21.15.88:0/3709709649 conn(0x7f16c00d11e0 msgr2=0x7f16c00d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:35.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.859+0000 7f16c7931640 1 -- 172.21.15.88:0/3709709649 shutdown_connections 2024-09-16T01:40:35.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.859+0000 7f16c7931640 1 -- 172.21.15.88:0/3709709649 wait complete. 2024-09-16T01:40:35.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.861+0000 7f16c7931640 1 Processor -- start 2024-09-16T01:40:35.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.861+0000 7f16c7931640 1 -- start start 2024-09-16T01:40:35.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.861+0000 7f16c7931640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16c00b68f0 0x7f16c0157880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:35.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.861+0000 7f16c7931640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16c00cf1f0 0x7f16c01f3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:35.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.861+0000 7f16c7931640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16c015e7a0 0x7f16c01f3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:35.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.861+0000 7f16c7931640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f16c006cfe0 con 0x7f16c00b68f0 2024-09-16T01:40:35.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.861+0000 7f16c7931640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f16c00b7a00 con 0x7f16c015e7a0 2024-09-16T01:40:35.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.861+0000 7f16c7931640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f16c015eb80 con 0x7f16c00cf1f0 2024-09-16T01:40:35.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.861+0000 7f16c56cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16c00b68f0 0x7f16c0157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:35.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.861+0000 7f16c56cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16c00b68f0 0x7f16c0157880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44098/0 (socket says 172.21.15.88:44098) 2024-09-16T01:40:35.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.861+0000 7f16c56cd640 1 -- 172.21.15.88:0/678288427 learned_addr learned my addr 172.21.15.88:0/678288427 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:35.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.862+0000 7f16c5ece640 1 --2- 172.21.15.88:0/678288427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16c015e7a0 0x7f16c01f3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:35.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.862+0000 7f16c4ecc640 1 --2- 172.21.15.88:0/678288427 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16c00cf1f0 0x7f16c01f3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:35.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.862+0000 7f16c56cd640 1 -- 172.21.15.88:0/678288427 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16c00cf1f0 msgr2=0x7f16c01f3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:35.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.862+0000 7f16c56cd640 1 --2- 172.21.15.88:0/678288427 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16c00cf1f0 0x7f16c01f3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:35.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.862+0000 7f16c56cd640 1 -- 172.21.15.88:0/678288427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16c015e7a0 msgr2=0x7f16c01f3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:35.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.862+0000 7f16c56cd640 1 --2- 172.21.15.88:0/678288427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16c015e7a0 0x7f16c01f3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:35.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.862+0000 7f16c56cd640 1 -- 172.21.15.88:0/678288427 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c00ca2f0 con 0x7f16c00b68f0 2024-09-16T01:40:35.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.862+0000 7f16c56cd640 1 --2- 172.21.15.88:0/678288427 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16c00b68f0 0x7f16c0157880 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f16b000efc0 tx=0x7f16b000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:35.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.862+0000 7f16bdffb640 1 -- 172.21.15.88:0/678288427 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b0018020 con 0x7f16c00b68f0 2024-09-16T01:40:35.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.862+0000 7f16bdffb640 1 -- 172.21.15.88:0/678288427 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16b0014480 con 0x7f16c00b68f0 2024-09-16T01:40:35.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.862+0000 7f16bdffb640 1 -- 172.21.15.88:0/678288427 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b0014850 con 0x7f16c00b68f0 2024-09-16T01:40:35.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.862+0000 7f16c7931640 1 -- 172.21.15.88:0/678288427 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16c00ced30 con 0x7f16c00b68f0 2024-09-16T01:40:35.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.862+0000 7f16c7931640 1 -- 172.21.15.88:0/678288427 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16c01f7840 con 0x7f16c00b68f0 2024-09-16T01:40:35.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.863+0000 7f16c7931640 1 -- 172.21.15.88:0/678288427 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16c006caa0 con 0x7f16c00b68f0 2024-09-16T01:40:35.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.864+0000 7f16bdffb640 1 -- 172.21.15.88:0/678288427 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f16b0005d10 con 0x7f16c00b68f0 2024-09-16T01:40:35.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.864+0000 7f16bdffb640 1 --2- 172.21.15.88:0/678288427 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f16ac06c2e0 0x7f16ac06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:35.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.864+0000 7f16bdffb640 1 -- 172.21.15.88:0/678288427 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5967+0+0 (secure 0 0 0) 0x7f16b008c2c0 con 0x7f16c00b68f0 2024-09-16T01:40:35.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.864+0000 7f16c4ecc640 1 --2- 172.21.15.88:0/678288427 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f16ac06c2e0 0x7f16ac06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:35.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.865+0000 7f16c4ecc640 1 --2- 172.21.15.88:0/678288427 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f16ac06c2e0 0x7f16ac06e7a0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f16c0159870 tx=0x7f16b4007b90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:35.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:35.869+0000 7f16bdffb640 1 -- 172.21.15.88:0/678288427 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f16b0022030 con 0x7f16c00b68f0 2024-09-16T01:40:36.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.067+0000 7f16c7931640 1 -- 172.21.15.88:0/678288427 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f16c00d2a80 con 0x7f16c00b68f0 2024-09-16T01:40:36.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.353+0000 7f16bdffb640 1 -- 172.21.15.88:0/678288427 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v88) ==== 161+0+0 (secure 0 0 0) 0x7f16c00ca2f0 con 0x7f16c00b68f0 2024-09-16T01:40:36.355 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:40:36.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.355+0000 7f16c7931640 1 -- 172.21.15.88:0/678288427 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f16ac06c2e0 msgr2=0x7f16ac06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:36.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.355+0000 7f16c7931640 1 --2- 172.21.15.88:0/678288427 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f16ac06c2e0 0x7f16ac06e7a0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f16c0159870 tx=0x7f16b4007b90 comp rx=0 tx=0).stop 2024-09-16T01:40:36.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.355+0000 7f16c7931640 1 -- 172.21.15.88:0/678288427 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16c00b68f0 msgr2=0x7f16c0157880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:36.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.355+0000 7f16c7931640 1 --2- 172.21.15.88:0/678288427 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16c00b68f0 0x7f16c0157880 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f16b000efc0 tx=0x7f16b000c490 comp rx=0 tx=0).stop 2024-09-16T01:40:36.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.355+0000 7f16c7931640 1 -- 172.21.15.88:0/678288427 shutdown_connections 2024-09-16T01:40:36.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.355+0000 7f16c7931640 1 --2- 172.21.15.88:0/678288427 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f16ac06c2e0 0x7f16ac06e7a0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:36.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.355+0000 7f16c7931640 1 --2- 172.21.15.88:0/678288427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16c015e7a0 0x7f16c01f3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:36.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.355+0000 7f16c7931640 1 --2- 172.21.15.88:0/678288427 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16c00cf1f0 0x7f16c01f3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:36.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.355+0000 7f16c7931640 1 --2- 172.21.15.88:0/678288427 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16c00b68f0 0x7f16c0157880 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:36.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.355+0000 7f16c7931640 1 -- 172.21.15.88:0/678288427 >> 172.21.15.88:0/678288427 conn(0x7f16c00d11e0 msgr2=0x7f16c0158810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:36.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.355+0000 7f16c7931640 1 -- 172.21.15.88:0/678288427 shutdown_connections 2024-09-16T01:40:36.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.355+0000 7f16c7931640 1 -- 172.21.15.88:0/678288427 wait complete. 2024-09-16T01:40:36.368 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:40:36.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.466+0000 7f3710fc4640 1 Processor -- start 2024-09-16T01:40:36.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.466+0000 7f3710fc4640 1 -- start start 2024-09-16T01:40:36.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.467+0000 7f3710fc4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f370c0b6290 0x7f370c071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:36.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.467+0000 7f3710fc4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f370c0ccff0 con 0x7f370c15b480 2024-09-16T01:40:36.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.467+0000 7f3710fc4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f370c0cd2d0 con 0x7f370c072530 2024-09-16T01:40:36.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.467+0000 7f3710fc4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f370c0cd5b0 con 0x7f370c0b6290 2024-09-16T01:40:36.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.467+0000 7f370a59c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f370c15b480 0x7f370c0b5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46608/0 (socket says 172.21.15.88:46608) 2024-09-16T01:40:36.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.467+0000 7f370a59c640 1 -- 172.21.15.88:0/1755230992 learned_addr learned my addr 172.21.15.88:0/1755230992 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:36.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.467+0000 7f3709d9b640 1 --2- 172.21.15.88:0/1755230992 >> v2:172.21.15.154:3300/0 conn(0x7f370c0b6290 0x7f370c071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:36.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.467+0000 7f370959a640 1 -- 172.21.15.88:0/1755230992 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3372334605 0 0) 0x7f370c0ccff0 con 0x7f370c15b480 2024-09-16T01:40:36.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.467+0000 7f370959a640 1 -- 172.21.15.88:0/1755230992 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f36f4003520 con 0x7f370c15b480 2024-09-16T01:40:36.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.468+0000 7f370959a640 1 -- 172.21.15.88:0/1755230992 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2612640812 0 0) 0x7f36f4003520 con 0x7f370c15b480 2024-09-16T01:40:36.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.468+0000 7f370959a640 1 -- 172.21.15.88:0/1755230992 >> v2:172.21.15.154:3300/0 conn(0x7f370c0b6290 msgr2=0x7f370c071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:36.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.468+0000 7f370959a640 1 --2- 172.21.15.88:0/1755230992 >> v2:172.21.15.154:3300/0 conn(0x7f370c0b6290 0x7f370c071ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:36.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.468+0000 7f370959a640 1 -- 172.21.15.88:0/1755230992 >> v1:172.21.15.154:6789/0 conn(0x7f370c072530 legacy=0x7f370c15bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:36.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.468+0000 7f370959a640 1 -- 172.21.15.88:0/1755230992 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f370c0cd890 con 0x7f370c15b480 2024-09-16T01:40:36.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.468+0000 7f370959a640 1 -- 172.21.15.88:0/1755230992 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f36fc001670 con 0x7f370c15b480 2024-09-16T01:40:36.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.468+0000 7f370959a640 1 -- 172.21.15.88:0/1755230992 >> v1:172.21.15.88:6789/0 conn(0x7f370c15b480 legacy=0x7f370c0b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:36.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.468+0000 7f370959a640 1 --2- 172.21.15.88:0/1755230992 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36f40039b0 0x7f36f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:36.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.468+0000 7f370959a640 1 --2- 172.21.15.88:0/1755230992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36f4004800 0x7f36f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:36.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.468+0000 7f370959a640 1 --2- 172.21.15.88:0/1755230992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f36f40051c0 0x7f36f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:36.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.468+0000 7f370959a640 1 -- 172.21.15.88:0/1755230992 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f370c0cd5b0 con 0x7f36f40039b0 2024-09-16T01:40:36.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.468+0000 7f370959a640 1 -- 172.21.15.88:0/1755230992 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f36f400c160 con 0x7f36f4004800 2024-09-16T01:40:36.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.468+0000 7f370959a640 1 -- 172.21.15.88:0/1755230992 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f36f400c360 con 0x7f36f40051c0 2024-09-16T01:40:36.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.468+0000 7f370a59c640 1 --2- 172.21.15.88:0/1755230992 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36f40039b0 0x7f36f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:36.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.468+0000 7f3709d9b640 1 --2- 172.21.15.88:0/1755230992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36f4004800 0x7f36f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:36.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.468+0000 7f370ad9d640 1 --2- 172.21.15.88:0/1755230992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f36f40051c0 0x7f36f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:36.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.468+0000 7f370a59c640 1 -- 172.21.15.88:0/1755230992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f36f40051c0 msgr2=0x7f36f400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:36.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.468+0000 7f370a59c640 1 --2- 172.21.15.88:0/1755230992 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f36f40051c0 0x7f36f400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:36.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.468+0000 7f370a59c640 1 -- 172.21.15.88:0/1755230992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36f4004800 msgr2=0x7f36f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:36.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.468+0000 7f370a59c640 1 --2- 172.21.15.88:0/1755230992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36f4004800 0x7f36f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:36.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.468+0000 7f370a59c640 1 -- 172.21.15.88:0/1755230992 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f4003520 con 0x7f36f40039b0 2024-09-16T01:40:36.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.468+0000 7f370ad9d640 1 -- 172.21.15.88:0/1755230992 reap_dead start 2024-09-16T01:40:36.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.469+0000 7f370a59c640 1 --2- 172.21.15.88:0/1755230992 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36f40039b0 0x7f36f4003dd0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f36fc00c240 tx=0x7f36f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e85d31ccdb94b88 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:36.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.469+0000 7f370959a640 1 -- 172.21.15.88:0/1755230992 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36fc007d10 con 0x7f36f40039b0 2024-09-16T01:40:36.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.469+0000 7f370959a640 1 -- 172.21.15.88:0/1755230992 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36fc005ad0 con 0x7f36f40039b0 2024-09-16T01:40:36.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.469+0000 7f370959a640 1 -- 172.21.15.88:0/1755230992 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36fc007070 con 0x7f36f40039b0 2024-09-16T01:40:36.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.469+0000 7f3710fc4640 1 -- 172.21.15.88:0/1755230992 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36f40039b0 msgr2=0x7f36f4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:36.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.469+0000 7f3710fc4640 1 --2- 172.21.15.88:0/1755230992 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36f40039b0 0x7f36f4003dd0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f36fc00c240 tx=0x7f36f4005830 comp rx=0 tx=0).stop 2024-09-16T01:40:36.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.469+0000 7f3710fc4640 1 -- 172.21.15.88:0/1755230992 shutdown_connections 2024-09-16T01:40:36.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.469+0000 7f3710fc4640 1 --2- 172.21.15.88:0/1755230992 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36f40039b0 0x7f36f4003dd0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:36.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.469+0000 7f3710fc4640 1 -- 172.21.15.88:0/1755230992 >> 172.21.15.88:0/1755230992 conn(0x7f370c06d0f0 msgr2=0x7f370c06d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:36.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.469+0000 7f3710fc4640 1 -- 172.21.15.88:0/1755230992 shutdown_connections 2024-09-16T01:40:36.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.469+0000 7f3710fc4640 1 -- 172.21.15.88:0/1755230992 wait complete. 2024-09-16T01:40:36.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.471+0000 7f3710fc4640 1 Processor -- start 2024-09-16T01:40:36.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.471+0000 7f3710fc4640 1 -- start start 2024-09-16T01:40:36.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.471+0000 7f3710fc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f36f40039b0 0x7f370c14a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:36.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.471+0000 7f3710fc4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f370c0cfcf0 0x7f370c1f38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:36.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.471+0000 7f3710fc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f370c1f3e30 0x7f370c1f42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:36.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.471+0000 7f3710fc4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f370c15b9e0 con 0x7f370c0cfcf0 2024-09-16T01:40:36.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.471+0000 7f3710fc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f370c15b860 con 0x7f370c1f3e30 2024-09-16T01:40:36.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.471+0000 7f3710fc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f370c071ff0 con 0x7f36f40039b0 2024-09-16T01:40:36.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.472+0000 7f3709d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f370c0cfcf0 0x7f370c1f38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:36.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.472+0000 7f3709d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f370c0cfcf0 0x7f370c1f38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44116/0 (socket says 172.21.15.88:44116) 2024-09-16T01:40:36.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.472+0000 7f3709d9b640 1 -- 172.21.15.88:0/1938722382 learned_addr learned my addr 172.21.15.88:0/1938722382 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:36.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.472+0000 7f370a59c640 1 --2- 172.21.15.88:0/1938722382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f36f40039b0 0x7f370c14a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:36.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.472+0000 7f370ad9d640 1 --2- 172.21.15.88:0/1938722382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f370c1f3e30 0x7f370c1f42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:36.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.472+0000 7f3709d9b640 1 -- 172.21.15.88:0/1938722382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f36f40039b0 msgr2=0x7f370c14a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:36.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.472+0000 7f3709d9b640 1 --2- 172.21.15.88:0/1938722382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f36f40039b0 0x7f370c14a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:36.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.472+0000 7f3709d9b640 1 -- 172.21.15.88:0/1938722382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f370c1f3e30 msgr2=0x7f370c1f42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:36.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.472+0000 7f3709d9b640 1 --2- 172.21.15.88:0/1938722382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f370c1f3e30 0x7f370c1f42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:36.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.472+0000 7f3709d9b640 1 -- 172.21.15.88:0/1938722382 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f370c14a590 con 0x7f370c0cfcf0 2024-09-16T01:40:36.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.472+0000 7f3709d9b640 1 --2- 172.21.15.88:0/1938722382 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f370c0cfcf0 0x7f370c1f38f0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f37000119e0 tx=0x7f3700011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:36.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.472+0000 7f36faffd640 1 -- 172.21.15.88:0/1938722382 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370000fb20 con 0x7f370c0cfcf0 2024-09-16T01:40:36.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.472+0000 7f36faffd640 1 -- 172.21.15.88:0/1938722382 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f370000fcc0 con 0x7f370c0cfcf0 2024-09-16T01:40:36.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.472+0000 7f3710fc4640 1 -- 172.21.15.88:0/1938722382 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f370c0d0500 con 0x7f370c0cfcf0 2024-09-16T01:40:36.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.472+0000 7f36faffd640 1 -- 172.21.15.88:0/1938722382 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3700014660 con 0x7f370c0cfcf0 2024-09-16T01:40:36.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.472+0000 7f3710fc4640 1 -- 172.21.15.88:0/1938722382 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f370c0d0ac0 con 0x7f370c0cfcf0 2024-09-16T01:40:36.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.473+0000 7f3710fc4640 1 -- 172.21.15.88:0/1938722382 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f370c0b5b80 con 0x7f370c0cfcf0 2024-09-16T01:40:36.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.474+0000 7f36faffd640 1 -- 172.21.15.88:0/1938722382 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3700012040 con 0x7f370c0cfcf0 2024-09-16T01:40:36.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.474+0000 7f36faffd640 1 --2- 172.21.15.88:0/1938722382 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f36ec06c2e0 0x7f36ec06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:36.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.474+0000 7f36faffd640 1 -- 172.21.15.88:0/1938722382 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5967+0+0 (secure 0 0 0) 0x7f370008b9f0 con 0x7f370c0cfcf0 2024-09-16T01:40:36.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.474+0000 7f370a59c640 1 --2- 172.21.15.88:0/1938722382 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f36ec06c2e0 0x7f36ec06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:36.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.475+0000 7f370a59c640 1 --2- 172.21.15.88:0/1938722382 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f36ec06c2e0 0x7f36ec06e7a0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f36fc033000 tx=0x7f36fc02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:36.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.478+0000 7f36faffd640 1 -- 172.21.15.88:0/1938722382 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f370c14a590 con 0x7f370c0cfcf0 2024-09-16T01:40:36.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:36.712+0000 7f3710fc4640 1 -- 172.21.15.88:0/1938722382 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f370c0b6040 con 0x7f370c0cfcf0 2024-09-16T01:40:37.096 INFO:tasks.ceph.mds.c.smithi088.stderr: -48> 2024-09-16T01:40:32.342+0000 7f88b6f27640 -1 log_channel(cluster) log [ERR] : dir 0x600 object missing on disk; some files may be lost (~mds0/stray0) 2024-09-16T01:40:37.096 INFO:tasks.ceph.mds.c.smithi088.stderr: -46> 2024-09-16T01:40:32.342+0000 7f88b6f27640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x600 is fatal because it is a system directory for this rank 2024-09-16T01:40:37.096 INFO:tasks.ceph.mds.c.smithi088.stderr: -31> 2024-09-16T01:40:33.322+0000 7f88baf2f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:40:37.120 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:40:37.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.367+0000 7f36faffd640 1 -- 172.21.15.88:0/1938722382 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v89) ==== 101+0+0 (secure 0 0 0) 0x7f3700019030 con 0x7f370c0cfcf0 2024-09-16T01:40:37.368 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:40:37.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.369+0000 7f3710fc4640 1 -- 172.21.15.88:0/1938722382 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f36ec06c2e0 msgr2=0x7f36ec06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:37.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.369+0000 7f3710fc4640 1 --2- 172.21.15.88:0/1938722382 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f36ec06c2e0 0x7f36ec06e7a0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f36fc033000 tx=0x7f36fc02e000 comp rx=0 tx=0).stop 2024-09-16T01:40:37.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.369+0000 7f3710fc4640 1 -- 172.21.15.88:0/1938722382 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f370c0cfcf0 msgr2=0x7f370c1f38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:37.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.369+0000 7f3710fc4640 1 --2- 172.21.15.88:0/1938722382 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f370c0cfcf0 0x7f370c1f38f0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f37000119e0 tx=0x7f3700011ea0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.369+0000 7f3710fc4640 1 -- 172.21.15.88:0/1938722382 shutdown_connections 2024-09-16T01:40:37.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.369+0000 7f3710fc4640 1 --2- 172.21.15.88:0/1938722382 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f36ec06c2e0 0x7f36ec06e7a0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.369+0000 7f3710fc4640 1 --2- 172.21.15.88:0/1938722382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f370c1f3e30 0x7f370c1f42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.369+0000 7f3710fc4640 1 --2- 172.21.15.88:0/1938722382 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f370c0cfcf0 0x7f370c1f38f0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.369+0000 7f3710fc4640 1 --2- 172.21.15.88:0/1938722382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f36f40039b0 0x7f370c14a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.369+0000 7f3710fc4640 1 -- 172.21.15.88:0/1938722382 >> 172.21.15.88:0/1938722382 conn(0x7f370c06d0f0 msgr2=0x7f370c1588a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:37.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.369+0000 7f3710fc4640 1 -- 172.21.15.88:0/1938722382 shutdown_connections 2024-09-16T01:40:37.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.369+0000 7f3710fc4640 1 -- 172.21.15.88:0/1938722382 wait complete. 2024-09-16T01:40:37.382 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:40:37.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.424+0000 7f8d5fd1ff40 1 Processor -- start 2024-09-16T01:40:37.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.424+0000 7f8d5fd1ff40 1 -- start start 2024-09-16T01:40:37.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.425+0000 7f8d5fd1ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d440029e0 0x55971408d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.425+0000 7f8d5fd1ff40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971411d330 0x55971411a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.425+0000 7f8d5fd1ff40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559714118e30 0x559714117d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.425+0000 7f8d5fd1ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x559713f98510 con 0x7f8d440029e0 2024-09-16T01:40:37.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.425+0000 7f8d5fd1ff40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559713ff1c50 con 0x55971411d330 2024-09-16T01:40:37.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.425+0000 7f8d5fd1ff40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55971408f9c0 con 0x559714118e30 2024-09-16T01:40:37.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.425+0000 7f8d5cdcf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d440029e0 0x55971408d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:37.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.425+0000 7f8d5cdcf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d440029e0 0x55971408d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44164/0 (socket says 172.21.15.88:44164) 2024-09-16T01:40:37.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.425+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/1603954585 learned_addr learned my addr 172.21.15.88:0/1603954585 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:37.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.425+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/1603954585 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559714118e30 msgr2=0x559714117d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:37.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.425+0000 7f8d5cdcf640 1 --2- 172.21.15.88:0/1603954585 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559714118e30 0x559714117d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.425+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/1603954585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971411d330 msgr2=0x55971411a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:37.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.425+0000 7f8d5cdcf640 1 --2- 172.21.15.88:0/1603954585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971411d330 0x55971411a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.425+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/1603954585 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55971408fca0 con 0x7f8d440029e0 2024-09-16T01:40:37.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.425+0000 7f8d5cdcf640 1 --2- 172.21.15.88:0/1603954585 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d440029e0 0x55971408d1f0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c006630 tx=0x7f8d4c006af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1b9973e98106a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:37.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.425+0000 7f8d5b5cc640 1 -- 172.21.15.88:0/1603954585 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c018070 con 0x7f8d440029e0 2024-09-16T01:40:37.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.425+0000 7f8d5b5cc640 1 -- 172.21.15.88:0/1603954585 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d4c006320 con 0x7f8d440029e0 2024-09-16T01:40:37.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.425+0000 7f8d5b5cc640 1 -- 172.21.15.88:0/1603954585 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c013680 con 0x7f8d440029e0 2024-09-16T01:40:37.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.426+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/1603954585 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d440029e0 msgr2=0x55971408d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:37.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.426+0000 7f8d5fd1ff40 1 --2- 172.21.15.88:0/1603954585 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d440029e0 0x55971408d1f0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c006630 tx=0x7f8d4c006af0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.426+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/1603954585 shutdown_connections 2024-09-16T01:40:37.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.426+0000 7f8d5fd1ff40 1 --2- 172.21.15.88:0/1603954585 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559714118e30 0x559714117d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.426+0000 7f8d5fd1ff40 1 --2- 172.21.15.88:0/1603954585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55971411d330 0x55971411a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.426+0000 7f8d5fd1ff40 1 --2- 172.21.15.88:0/1603954585 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d440029e0 0x55971408d1f0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.426+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/1603954585 >> 172.21.15.88:0/1603954585 conn(0x559714091750 msgr2=0x55971411cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:37.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.426+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/1603954585 shutdown_connections 2024-09-16T01:40:37.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.426+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/1603954585 wait complete. 2024-09-16T01:40:37.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.426+0000 7f8d5fd1ff40 1 Processor -- start 2024-09-16T01:40:37.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.427+0000 7f8d5fd1ff40 1 -- start start 2024-09-16T01:40:37.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.427+0000 7f8d5fd1ff40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559714118e30 0x55971410d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.427+0000 7f8d5fd1ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55971411d330 0x55971411b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.427+0000 7f8d5fd1ff40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d440029e0 0x55971411c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.427+0000 7f8d5fd1ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x559714112c90 con 0x55971411d330 2024-09-16T01:40:37.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.427+0000 7f8d5fd1ff40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5597141129b0 con 0x559714118e30 2024-09-16T01:40:37.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.427+0000 7f8d5fd1ff40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55971411de00 con 0x7f8d440029e0 2024-09-16T01:40:37.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.427+0000 7f8d5c5ce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55971411d330 0x55971411b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:37.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.427+0000 7f8d5c5ce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55971411d330 0x55971411b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44178/0 (socket says 172.21.15.88:44178) 2024-09-16T01:40:37.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.427+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 learned_addr learned my addr 172.21.15.88:0/3020101189 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:37.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.427+0000 7f8d5d5d0640 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d440029e0 0x55971411c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:37.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.427+0000 7f8d5cdcf640 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559714118e30 0x55971410d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:37.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.428+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d440029e0 msgr2=0x55971411c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:37.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.428+0000 7f8d5c5ce640 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d440029e0 0x55971411c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.428+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559714118e30 msgr2=0x55971410d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:37.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.428+0000 7f8d5c5ce640 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559714118e30 0x55971410d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.428+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55971411bbd0 con 0x55971411d330 2024-09-16T01:40:37.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.428+0000 7f8d5cdcf640 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559714118e30 0x55971410d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:37.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.428+0000 7f8d5d5d0640 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d440029e0 0x55971411c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:37.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.428+0000 7f8d5c5ce640 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55971411d330 0x55971411b2d0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f8d50000c00 tx=0x7f8d500558f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:37.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.428+0000 7f8d5a5ca640 1 -- 172.21.15.88:0/3020101189 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d50065020 con 0x55971411d330 2024-09-16T01:40:37.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.428+0000 7f8d5a5ca640 1 -- 172.21.15.88:0/3020101189 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d50055ae0 con 0x55971411d330 2024-09-16T01:40:37.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.428+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559714115a10 con 0x55971411d330 2024-09-16T01:40:37.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.428+0000 7f8d5a5ca640 1 -- 172.21.15.88:0/3020101189 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d500096f0 con 0x55971411d330 2024-09-16T01:40:37.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.428+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559714115f90 con 0x55971411d330 2024-09-16T01:40:37.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.429+0000 7f8d5a5ca640 1 -- 172.21.15.88:0/3020101189 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8d50068020 con 0x55971411d330 2024-09-16T01:40:37.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.430+0000 7f8d5a5ca640 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8d4806ac90 0x7f8d4806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.430+0000 7f8d5a5ca640 1 -- 172.21.15.88:0/3020101189 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5967+0+0 (secure 0 0 0) 0x7f8d500ced40 con 0x55971411d330 2024-09-16T01:40:37.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.430+0000 7f8d5cdcf640 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8d4806ac90 0x7f8d4806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:37.435 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:40:37.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.430+0000 7f8d5fd1ff40 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55971411bde0 0x5597141a6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.430+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141a6c50 con 0x55971411bde0 2024-09-16T01:40:37.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.430+0000 7f8d5d5d0640 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55971411bde0 0x5597141a6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:37.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.430+0000 7f8d5cdcf640 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8d4806ac90 0x7f8d4806d150 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c006630 tx=0x7f8d4c008070 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:37.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.430+0000 7f8d5d5d0640 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55971411bde0 0x5597141a6710 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:37.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.432+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v88'73 uv48 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.432+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e88) -- 0x5597141a7020 con 0x55971411bde0 2024-09-16T01:40:37.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.433+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v88'74 uv74 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.433+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c1b30 con 0x55971411bde0 2024-09-16T01:40:37.437 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:40:37.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.435+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v88'75 uv75 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.435+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c2ee0 con 0x55971411bde0 2024-09-16T01:40:37.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.438+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v88'76 uv76 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.438+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c3280 con 0x55971411bde0 2024-09-16T01:40:37.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.441+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v88'77 uv77 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.441+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c1ed0 con 0x55971411bde0 2024-09-16T01:40:37.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.442+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v88'78 uv78 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.443+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c2270 con 0x55971411bde0 2024-09-16T01:40:37.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.444+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v88'79 uv54 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.444+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e88) -- 0x5597141c2610 con 0x55971411bde0 2024-09-16T01:40:37.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.447+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v88'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.447+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c2610 con 0x55971411bde0 2024-09-16T01:40:37.450 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:40:37.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.449+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v88'81 uv81 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.449+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c2610 con 0x55971411bde0 2024-09-16T01:40:37.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.451+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v88'82 uv82 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.451+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c2610 con 0x55971411bde0 2024-09-16T01:40:37.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.453+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v88'83 uv83 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.453+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c2610 con 0x55971411bde0 2024-09-16T01:40:37.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.455+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v88'84 uv84 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.455+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c2610 con 0x55971411bde0 2024-09-16T01:40:37.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.457+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v88'85 uv59 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.457+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e88) -- 0x5597141c2610 con 0x55971411bde0 2024-09-16T01:40:37.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.459+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v88'86 uv86 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.459+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c2610 con 0x55971411bde0 2024-09-16T01:40:37.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.461+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v88'87 uv87 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.463 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:40:37.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.461+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c2610 con 0x55971411bde0 2024-09-16T01:40:37.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.464+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v88'88 uv88 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.464+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c2610 con 0x55971411bde0 2024-09-16T01:40:37.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.466+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v88'89 uv89 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.466+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c2610 con 0x55971411bde0 2024-09-16T01:40:37.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.469+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v88'90 uv90 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.469+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c2610 con 0x55971411bde0 2024-09-16T01:40:37.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.471+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v88'91 uv91 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.471+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c2610 con 0x55971411bde0 2024-09-16T01:40:37.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.474+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v88'92 uv92 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.475 INFO:teuthology.orchestra.run.smithi088.stdout:Write #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:40:37.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.474+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c2610 con 0x55971411bde0 2024-09-16T01:40:37.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.475+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [setxattr (30)] v88'93 uv93 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.475+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c2610 con 0x55971411bde0 2024-09-16T01:40:37.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.476+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [setxattr (62)] v88'94 uv94 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.476+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c2610 con 0x55971411bde0 2024-09-16T01:40:37.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.478+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [omap-set-header] v88'95 uv95 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.478+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c2610 con 0x55971411bde0 2024-09-16T01:40:37.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.479+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 603.00000000 [create] v88'96 uv71 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.479+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e88) -- 0x5597141c2610 con 0x55971411bde0 2024-09-16T01:40:37.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.480+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 603.00000000 [delete] v88'97 uv97 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.480+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c2610 con 0x55971411bde0 2024-09-16T01:40:37.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.480+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v88'98 uv98 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.482 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:40:37.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.480+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c2610 con 0x55971411bde0 2024-09-16T01:40:37.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.481+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [setxattr (30)] v88'99 uv99 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.481+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c2610 con 0x55971411bde0 2024-09-16T01:40:37.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.483+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [setxattr (62)] v88'100 uv100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.483+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c2610 con 0x55971411bde0 2024-09-16T01:40:37.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.484+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [omap-set-header] v88'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.484+0000 7f8d5fd1ff40 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5597141c29f0 0x5597141c5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.484+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:30 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c2610 con 0x5597141c29f0 2024-09-16T01:40:37.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.484+0000 7f8d5cdcf640 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5597141c29f0 0x5597141c5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:37.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.484+0000 7f8d5cdcf640 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5597141c29f0 0x5597141c5b40 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:37.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.485+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(30 1.00000000.inode [create] v88'68 uv50 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141c29f0 2024-09-16T01:40:37.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.485+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:31 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e88) -- 0x5597141c6080 con 0x5597141c29f0 2024-09-16T01:40:37.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.487+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(31 1.00000000.inode [delete] v88'69 uv69 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141c29f0 2024-09-16T01:40:37.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.487+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c6420 con 0x5597141c29f0 2024-09-16T01:40:37.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.488+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(32 1.00000000.inode [create] v88'70 uv70 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141c29f0 2024-09-16T01:40:37.490 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:40:37.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.488+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c6c70 con 0x5597141c29f0 2024-09-16T01:40:37.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.490+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(33 1.00000000.inode [write 0~582] v88'71 uv71 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141c29f0 2024-09-16T01:40:37.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.490+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c7010 con 0x5597141c29f0 2024-09-16T01:40:37.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.492+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(34 1.00000000.inode [omap-set-header] v88'72 uv72 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141c29f0 2024-09-16T01:40:37.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.493+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c73b0 con 0x5597141c29f0 2024-09-16T01:40:37.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.494+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(35 200.00000000 [create] v88'73 uv55 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141c29f0 2024-09-16T01:40:37.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.494+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e88) -- 0x5597141c7750 con 0x5597141c29f0 2024-09-16T01:40:37.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.496+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(36 200.00000000 [delete] v88'74 uv74 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141c29f0 2024-09-16T01:40:37.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.496+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c7750 con 0x5597141c29f0 2024-09-16T01:40:37.499 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:40:37.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.498+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(37 200.00000000 [create] v88'75 uv75 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141c29f0 2024-09-16T01:40:37.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.499+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c7750 con 0x5597141c29f0 2024-09-16T01:40:37.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.500+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(38 200.00000000 [write 0~90] v88'76 uv76 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141c29f0 2024-09-16T01:40:37.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.500+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c7750 con 0x5597141c29f0 2024-09-16T01:40:37.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.501+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(39 200.00000000 [omap-set-header] v88'77 uv77 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141c29f0 2024-09-16T01:40:37.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.501+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c7750 con 0x5597141c29f0 2024-09-16T01:40:37.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.502+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(40 200.00000001 [create] v88'78 uv60 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141c29f0 2024-09-16T01:40:37.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.502+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e88) -- 0x5597141c7750 con 0x5597141c29f0 2024-09-16T01:40:37.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.504+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(41 200.00000001 [delete] v88'79 uv79 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141c29f0 2024-09-16T01:40:37.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.504+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c7750 con 0x5597141c29f0 2024-09-16T01:40:37.506 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:40:37.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.505+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(42 200.00000001 [create] v88'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141c29f0 2024-09-16T01:40:37.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.505+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141cbbd0 con 0x5597141c29f0 2024-09-16T01:40:37.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.506+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(43 200.00000001 [write 0~17490] v88'81 uv81 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141c29f0 2024-09-16T01:40:37.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.506+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141cbbd0 con 0x5597141c29f0 2024-09-16T01:40:37.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.507+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(44 200.00000001 [omap-set-header] v88'82 uv82 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141c29f0 2024-09-16T01:40:37.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.507+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c7750 con 0x5597141c29f0 2024-09-16T01:40:37.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.508+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(45 606.00000000 [create] v88'83 uv66 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141c29f0 2024-09-16T01:40:37.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.508+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e88) -- 0x5597141c7750 con 0x5597141c29f0 2024-09-16T01:40:37.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.509+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(46 606.00000000 [delete] v88'84 uv84 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141c29f0 2024-09-16T01:40:37.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.509+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c7750 con 0x5597141c29f0 2024-09-16T01:40:37.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.511+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(47 606.00000000 [create] v88'85 uv85 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141c29f0 2024-09-16T01:40:37.512 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:40:37.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.511+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c8760 con 0x5597141c29f0 2024-09-16T01:40:37.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.513+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(48 606.00000000 [setxattr (30)] v88'86 uv86 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141c29f0 2024-09-16T01:40:37.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.513+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c8760 con 0x5597141c29f0 2024-09-16T01:40:37.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.515+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(49 606.00000000 [setxattr (62)] v88'87 uv87 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141c29f0 2024-09-16T01:40:37.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.515+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c7750 con 0x5597141c29f0 2024-09-16T01:40:37.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.516+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(50 606.00000000 [omap-set-header] v88'88 uv88 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141c29f0 2024-09-16T01:40:37.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.516+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:51 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c7750 con 0x55971411bde0 2024-09-16T01:40:37.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.517+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(51 607.00000000 [create] v88'15 uv14 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.517+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:52 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e88) -- 0x5597141c7750 con 0x55971411bde0 2024-09-16T01:40:37.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.518+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(52 607.00000000 [delete] v88'16 uv16 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.518+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c7750 con 0x55971411bde0 2024-09-16T01:40:37.522 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:40:37.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.521+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v88'17 uv17 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.521+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c8760 con 0x55971411bde0 2024-09-16T01:40:37.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.523+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [setxattr (30)] v88'18 uv18 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.523+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c8760 con 0x55971411bde0 2024-09-16T01:40:37.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.528+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [setxattr (62)] v88'19 uv19 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.528+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c7750 con 0x55971411bde0 2024-09-16T01:40:37.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.529+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [omap-set-header] v88'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.529+0000 7f8d5fd1ff40 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5597141c7dd0 0x5597141ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.529+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:57 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141ca840 con 0x5597141c7dd0 2024-09-16T01:40:37.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.529+0000 7f8d5c5ce640 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5597141c7dd0 0x5597141ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:37.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.529+0000 7f8d5c5ce640 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5597141c7dd0 0x5597141ca1c0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:37.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.531+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(57 mds0_openfiles.0 [create] v88'14 uv13 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f8d500994d0 con 0x5597141c7dd0 2024-09-16T01:40:37.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.532+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:58 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e88) -- 0x5597141cad50 con 0x5597141c7dd0 2024-09-16T01:40:37.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.532+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(58 mds0_openfiles.0 [delete] v88'15 uv15 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8d500994d0 con 0x5597141c7dd0 2024-09-16T01:40:37.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.532+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141cb250 con 0x5597141c7dd0 2024-09-16T01:40:37.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.533+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(59 mds0_openfiles.0 [create] v88'16 uv16 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8d500994d0 con 0x5597141c7dd0 2024-09-16T01:40:37.535 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:40:37.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.533+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141c68a0 con 0x5597141c7dd0 2024-09-16T01:40:37.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.534+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(60 mds0_openfiles.0 [omap-set-header] v88'17 uv17 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f8d500994d0 con 0x5597141c7dd0 2024-09-16T01:40:37.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.534+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:61 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141cbb80 con 0x55971411bde0 2024-09-16T01:40:37.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.536+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(61 608.00000000 [create] v88'15 uv14 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.536+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:62 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e88) -- 0x5597141cbb80 con 0x55971411bde0 2024-09-16T01:40:37.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.537+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(62 608.00000000 [delete] v88'16 uv16 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.537+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141cbb80 con 0x55971411bde0 2024-09-16T01:40:37.539 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:40:37.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.538+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v88'17 uv17 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.538+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141cbbc0 con 0x55971411bde0 2024-09-16T01:40:37.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.539+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [setxattr (30)] v88'18 uv18 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.539+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141cd060 con 0x55971411bde0 2024-09-16T01:40:37.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.540+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [setxattr (62)] v88'19 uv19 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.540+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141cbf70 con 0x55971411bde0 2024-09-16T01:40:37.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.540+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [omap-set-header] v88'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.540+0000 7f8d5fd1ff40 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5597141cc850 0x5597141cf020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.540+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:67 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141cf560 con 0x5597141cc850 2024-09-16T01:40:37.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.541+0000 7f8d5d5d0640 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5597141cc850 0x5597141cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:37.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.541+0000 7f8d5d5d0640 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5597141cc850 0x5597141cf020 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:37.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.542+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(67 604.00000000 [create] v88'26 uv20 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.542+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:68 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e88) -- 0x5597141cf560 con 0x5597141cc850 2024-09-16T01:40:37.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.545+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(68 604.00000000 [delete] v88'27 uv27 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.545+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141cf560 con 0x5597141cc850 2024-09-16T01:40:37.547 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:40:37.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.546+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(69 604.00000000 [create] v88'28 uv28 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.546+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d0570 con 0x5597141cc850 2024-09-16T01:40:37.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.549+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(70 604.00000000 [setxattr (30)] v88'29 uv29 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.549+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d0570 con 0x5597141cc850 2024-09-16T01:40:37.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.550+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(71 604.00000000 [setxattr (62)] v88'30 uv30 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.550+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141cf560 con 0x5597141cc850 2024-09-16T01:40:37.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.552+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(72 604.00000000 [omap-set-header] v88'31 uv31 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.552+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141cf560 con 0x5597141cc850 2024-09-16T01:40:37.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.555+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(73 500.00000000 [create] v88'32 uv25 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.555+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e88) -- 0x5597141cf560 con 0x5597141cc850 2024-09-16T01:40:37.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.558+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(74 500.00000000 [delete] v88'33 uv33 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.558+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141cf560 con 0x5597141cc850 2024-09-16T01:40:37.562 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:40:37.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.561+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(75 500.00000000 [create] v88'34 uv34 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.561+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141cf560 con 0x5597141cc850 2024-09-16T01:40:37.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.564+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(76 500.00000000 [write 0~90] v88'35 uv35 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.564+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141cf560 con 0x5597141cc850 2024-09-16T01:40:37.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.565+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(77 500.00000000 [omap-set-header] v88'36 uv36 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.565+0000 7f8d5fd1ff40 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5597141cfbe0 0x5597141d1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.565+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:78 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d2650 con 0x5597141cfbe0 2024-09-16T01:40:37.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.565+0000 7f8d5cdcf640 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5597141cfbe0 0x5597141d1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:37.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.566+0000 7f8d5cdcf640 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5597141cfbe0 0x5597141d1fd0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:37.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.567+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(78 mds_snaptable [create] v88'12 uv11 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141cfbe0 2024-09-16T01:40:37.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.567+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:79 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e88) -- 0x5597141d2a20 con 0x5597141cfbe0 2024-09-16T01:40:37.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.569+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(79 mds_snaptable [delete] v88'13 uv13 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141cfbe0 2024-09-16T01:40:37.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.569+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d2a20 con 0x5597141cfbe0 2024-09-16T01:40:37.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.570+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(80 mds_snaptable [create] v88'14 uv14 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141cfbe0 2024-09-16T01:40:37.571 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:40:37.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.570+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d2a20 con 0x5597141cfbe0 2024-09-16T01:40:37.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.571+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(81 mds_snaptable [write 0~70] v88'15 uv15 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141cfbe0 2024-09-16T01:40:37.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.571+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d2a20 con 0x5597141cfbe0 2024-09-16T01:40:37.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.572+0000 7f8d5cdcf640 1 -- 172.21.15.88:0/3020101189 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(82 mds_snaptable [omap-set-header] v88'16 uv16 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8d4c005770 con 0x5597141cfbe0 2024-09-16T01:40:37.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.572+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:83 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d2a20 con 0x55971411bde0 2024-09-16T01:40:37.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.573+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(83 605.00000000 [create] v88'15 uv14 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.573+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:84 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e88) -- 0x5597141d2a20 con 0x55971411bde0 2024-09-16T01:40:37.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.574+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(84 605.00000000 [delete] v88'16 uv16 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.574+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d2a20 con 0x55971411bde0 2024-09-16T01:40:37.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.575+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v88'17 uv17 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.576 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:40:37.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.575+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d3a30 con 0x55971411bde0 2024-09-16T01:40:37.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.576+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [setxattr (30)] v88'18 uv18 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.576+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d3a30 con 0x55971411bde0 2024-09-16T01:40:37.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.577+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [setxattr (62)] v88'19 uv19 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.577+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d2a20 con 0x55971411bde0 2024-09-16T01:40:37.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.578+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [omap-set-header] v88'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x55971411bde0 2024-09-16T01:40:37.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.578+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:89 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d2a20 con 0x5597141c7dd0 2024-09-16T01:40:37.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.579+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(89 mds0_inotable [create] v88'24 uv18 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f8d500994d0 con 0x5597141c7dd0 2024-09-16T01:40:37.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.579+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:90 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e88) -- 0x5597141d2e10 con 0x5597141c7dd0 2024-09-16T01:40:37.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.580+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(90 mds0_inotable [delete] v88'25 uv25 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8d500994d0 con 0x5597141c7dd0 2024-09-16T01:40:37.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.580+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d3200 con 0x5597141c7dd0 2024-09-16T01:40:37.581 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:40:37.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.580+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(91 mds0_inotable [create] v88'26 uv26 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8d500994d0 con 0x5597141c7dd0 2024-09-16T01:40:37.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.580+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141cb700 con 0x5597141c7dd0 2024-09-16T01:40:37.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.581+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(92 mds0_inotable [write 0~34] v88'27 uv27 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8d500994d0 con 0x5597141c7dd0 2024-09-16T01:40:37.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.581+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141cb700 con 0x5597141c7dd0 2024-09-16T01:40:37.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.582+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(93 mds0_inotable [omap-set-header] v88'28 uv28 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f8d500994d0 con 0x5597141c7dd0 2024-09-16T01:40:37.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.582+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141cb700 con 0x5597141c7dd0 2024-09-16T01:40:37.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.583+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(94 100.00000000 [create] v88'29 uv23 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8d500994d0 con 0x5597141c7dd0 2024-09-16T01:40:37.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.583+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e88) -- 0x5597141cb700 con 0x5597141c7dd0 2024-09-16T01:40:37.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.584+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(95 100.00000000 [delete] v88'30 uv30 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d500994d0 con 0x5597141c7dd0 2024-09-16T01:40:37.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.584+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141cb700 con 0x5597141c7dd0 2024-09-16T01:40:37.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.585+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(96 100.00000000 [create] v88'31 uv31 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d500994d0 con 0x5597141c7dd0 2024-09-16T01:40:37.586 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:40:37.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.585+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141cb700 con 0x5597141c7dd0 2024-09-16T01:40:37.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.586+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(97 100.00000000 [omap-set-header] v88'32 uv32 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d500994d0 con 0x5597141c7dd0 2024-09-16T01:40:37.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.586+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6750 con 0x5597141c7dd0 2024-09-16T01:40:37.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.587+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v88'33 uv33 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d500994d0 con 0x5597141c7dd0 2024-09-16T01:40:37.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.587+0000 7f8d5fd1ff40 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5597141d41c0 0x5597141d65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.587+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:99 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6af0 con 0x5597141d41c0 2024-09-16T01:40:37.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.587+0000 7f8d5c5ce640 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5597141d41c0 0x5597141d65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:37.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.587+0000 7f8d5c5ce640 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5597141d41c0 0x5597141d65b0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:37.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.588+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(99 mds0_sessionmap [create] v88'14 uv12 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f8d500994d0 con 0x5597141d41c0 2024-09-16T01:40:37.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.588+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:100 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6af0 con 0x5597141d41c0 2024-09-16T01:40:37.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.591+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v88'15 uv15 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f8d500994d0 con 0x5597141d41c0 2024-09-16T01:40:37.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.591+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6af0 con 0x5597141d41c0 2024-09-16T01:40:37.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.592+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(101 mds0_sessionmap [create] v88'16 uv16 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f8d500994d0 con 0x5597141d41c0 2024-09-16T01:40:37.593 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:40:37.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.592+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6b70 con 0x5597141d41c0 2024-09-16T01:40:37.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.593+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v88'17 uv17 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f8d500994d0 con 0x5597141d41c0 2024-09-16T01:40:37.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.593+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6b70 con 0x5597141c7dd0 2024-09-16T01:40:37.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.594+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(103 609.00000000 [create] v88'15 uv14 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8d500994d0 con 0x5597141c7dd0 2024-09-16T01:40:37.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.594+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6b70 con 0x5597141c7dd0 2024-09-16T01:40:37.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.595+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(104 609.00000000 [delete] v88'16 uv16 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d500994d0 con 0x5597141c7dd0 2024-09-16T01:40:37.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.595+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6b70 con 0x5597141c7dd0 2024-09-16T01:40:37.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.596+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(105 609.00000000 [create] v88'17 uv17 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d500994d0 con 0x5597141c7dd0 2024-09-16T01:40:37.597 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:40:37.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.596+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6b70 con 0x5597141c7dd0 2024-09-16T01:40:37.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.597+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v88'18 uv18 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d500994d0 con 0x5597141c7dd0 2024-09-16T01:40:37.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.597+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6b70 con 0x5597141c7dd0 2024-09-16T01:40:37.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.598+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v88'19 uv19 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d500994d0 con 0x5597141c7dd0 2024-09-16T01:40:37.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.598+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6b70 con 0x5597141c7dd0 2024-09-16T01:40:37.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.599+0000 7f8d5c5ce640 1 -- 172.21.15.88:0/3020101189 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(108 609.00000000 [omap-set-header] v88'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d500994d0 con 0x5597141c7dd0 2024-09-16T01:40:37.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.599+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6b70 con 0x5597141cc850 2024-09-16T01:40:37.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.600+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(109 400.00000000 [create] v88'12 uv11 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.600+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6b70 con 0x5597141cc850 2024-09-16T01:40:37.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.601+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(110 400.00000000 [delete] v88'13 uv13 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.601+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6b70 con 0x5597141cc850 2024-09-16T01:40:37.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.604+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(111 400.00000000 [create] v88'14 uv14 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.605 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:40:37.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.604+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6b70 con 0x5597141cc850 2024-09-16T01:40:37.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.607+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v88'15 uv15 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.607+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6b70 con 0x5597141cc850 2024-09-16T01:40:37.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.610+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v88'16 uv16 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.610+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6b70 con 0x5597141cc850 2024-09-16T01:40:37.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.612+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(114 100.00000000.inode [create] v88'33 uv24 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.612+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6b70 con 0x5597141cc850 2024-09-16T01:40:37.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.613+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(115 100.00000000.inode [delete] v88'34 uv34 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.613+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6b70 con 0x5597141cc850 2024-09-16T01:40:37.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.617+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(116 100.00000000.inode [create] v88'35 uv35 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.618 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:40:37.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.617+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6b70 con 0x5597141cc850 2024-09-16T01:40:37.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.618+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v88'36 uv36 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.618+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6b70 con 0x5597141cc850 2024-09-16T01:40:37.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.621+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v88'37 uv37 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.621+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6b70 con 0x5597141cc850 2024-09-16T01:40:37.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.624+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(119 1.00000000 [create] v88'38 uv31 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.624+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6b70 con 0x5597141cc850 2024-09-16T01:40:37.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.629+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(120 1.00000000 [delete] v88'39 uv39 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.629+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6b70 con 0x5597141cc850 2024-09-16T01:40:37.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.630+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(121 1.00000000 [create] v88'40 uv40 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.631 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:40:37.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.630+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6b70 con 0x5597141cc850 2024-09-16T01:40:37.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.632+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v88'41 uv41 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.633+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6b70 con 0x5597141cc850 2024-09-16T01:40:37.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.636+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v88'42 uv42 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.636+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6b70 con 0x5597141cc850 2024-09-16T01:40:37.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.641+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v88'43 uv43 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.641+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x5597141d6fd0 con 0x5597141cc850 2024-09-16T01:40:37.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.642+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v88'44 uv44 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f8d540095b0 con 0x5597141cc850 2024-09-16T01:40:37.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.642+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55971411bde0 msgr2=0x5597141a6710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:37.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.642+0000 7f8d5fd1ff40 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55971411bde0 0x5597141a6710 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.642+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5597141c7dd0 msgr2=0x5597141ca1c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:37.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.642+0000 7f8d5fd1ff40 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5597141c7dd0 0x5597141ca1c0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.642+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5597141d41c0 msgr2=0x5597141d65b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:37.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.642+0000 7f8d5fd1ff40 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5597141d41c0 0x5597141d65b0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.642+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5597141c29f0 msgr2=0x5597141c5b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:37.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.642+0000 7f8d5fd1ff40 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5597141c29f0 0x5597141c5b40 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.642+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5597141cc850 msgr2=0x5597141cf020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:37.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.642+0000 7f8d5fd1ff40 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5597141cc850 0x5597141cf020 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.642+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 reap_dead start 2024-09-16T01:40:37.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.642+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5597141cfbe0 msgr2=0x5597141d1fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:37.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.642+0000 7f8d5fd1ff40 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5597141cfbe0 0x5597141d1fd0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.642+0000 7f8d5d5d0640 1 -- 172.21.15.88:0/3020101189 reap_dead start 2024-09-16T01:40:37.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.642+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8d4806ac90 msgr2=0x7f8d4806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:37.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.642+0000 7f8d5fd1ff40 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8d4806ac90 0x7f8d4806d150 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c006630 tx=0x7f8d4c008070 comp rx=0 tx=0).stop 2024-09-16T01:40:37.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.642+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55971411d330 msgr2=0x55971411b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:37.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.642+0000 7f8d5fd1ff40 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55971411d330 0x55971411b2d0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f8d50000c00 tx=0x7f8d500558f0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.642+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 shutdown_connections 2024-09-16T01:40:37.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.642+0000 7f8d5fd1ff40 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8d4806ac90 0x7f8d4806d150 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.642+0000 7f8d5fd1ff40 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55971411d330 0x55971411b2d0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.642+0000 7f8d5fd1ff40 1 --2- 172.21.15.88:0/3020101189 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5597141cfbe0 0x5597141d1fd0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.642+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 >> 172.21.15.88:0/3020101189 conn(0x559714091750 msgr2=0x55971408ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:37.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.642+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 shutdown_connections 2024-09-16T01:40:37.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.643+0000 7f8d5fd1ff40 1 -- 172.21.15.88:0/3020101189 wait complete. 2024-09-16T01:40:37.647 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 1.00000000.inode 2024-09-16T01:40:37.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.728+0000 7fbc87ecef40 1 Processor -- start 2024-09-16T01:40:37.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.728+0000 7fbc87ecef40 1 -- start start 2024-09-16T01:40:37.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.729+0000 7fbc87ecef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5651543a0fe0 0x56515443b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.729+0000 7fbc87ecef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5651543a1a60 0x565154448760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.729+0000 7fbc87ecef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5651543bb250 0x565154445ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.729+0000 7fbc87ecef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5651542c3510 con 0x5651543a1a60 2024-09-16T01:40:37.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.729+0000 7fbc87ecef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56515431cc50 con 0x5651543bb250 2024-09-16T01:40:37.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.729+0000 7fbc87ecef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x565154448ff0 con 0x5651543a0fe0 2024-09-16T01:40:37.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.729+0000 7fbc7ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5651543a1a60 0x565154448760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:37.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.729+0000 7fbc7ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5651543a1a60 0x565154448760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44206/0 (socket says 172.21.15.88:44206) 2024-09-16T01:40:37.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.729+0000 7fbc7ffff640 1 -- 172.21.15.88:0/4176569616 learned_addr learned my addr 172.21.15.88:0/4176569616 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:37.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.729+0000 7fbc7ffff640 1 -- 172.21.15.88:0/4176569616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5651543a0fe0 msgr2=0x56515443b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:37.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.729+0000 7fbc7ffff640 1 --2- 172.21.15.88:0/4176569616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5651543a0fe0 0x56515443b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.729+0000 7fbc861c5640 1 --2- 172.21.15.88:0/4176569616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5651543bb250 0x565154445ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:37.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.729+0000 7fbc7ffff640 1 -- 172.21.15.88:0/4176569616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5651543bb250 msgr2=0x565154445ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:37.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.729+0000 7fbc7ffff640 1 --2- 172.21.15.88:0/4176569616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5651543bb250 0x565154445ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.729+0000 7fbc7ffff640 1 -- 172.21.15.88:0/4176569616 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56515443b920 con 0x5651543a1a60 2024-09-16T01:40:37.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.729+0000 7fbc7ffff640 1 --2- 172.21.15.88:0/4176569616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5651543a1a60 0x565154448760 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fbc70001dd0 tx=0x7fbc7000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a68fd26fbe5d7ab server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:37.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.729+0000 7fbc849c2640 1 -- 172.21.15.88:0/4176569616 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc7000bde0 con 0x5651543a1a60 2024-09-16T01:40:37.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.729+0000 7fbc849c2640 1 -- 172.21.15.88:0/4176569616 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc7000f400 con 0x5651543a1a60 2024-09-16T01:40:37.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.729+0000 7fbc849c2640 1 -- 172.21.15.88:0/4176569616 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc7000f7f0 con 0x5651543a1a60 2024-09-16T01:40:37.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.730+0000 7fbc87ecef40 1 -- 172.21.15.88:0/4176569616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5651543a1a60 msgr2=0x565154448760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:37.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.730+0000 7fbc87ecef40 1 --2- 172.21.15.88:0/4176569616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5651543a1a60 0x565154448760 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fbc70001dd0 tx=0x7fbc7000b5b0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.730+0000 7fbc87ecef40 1 -- 172.21.15.88:0/4176569616 shutdown_connections 2024-09-16T01:40:37.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.730+0000 7fbc87ecef40 1 --2- 172.21.15.88:0/4176569616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5651543bb250 0x565154445ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.730+0000 7fbc87ecef40 1 --2- 172.21.15.88:0/4176569616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5651543a1a60 0x565154448760 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.730+0000 7fbc87ecef40 1 --2- 172.21.15.88:0/4176569616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5651543a0fe0 0x56515443b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.730+0000 7fbc87ecef40 1 -- 172.21.15.88:0/4176569616 >> 172.21.15.88:0/4176569616 conn(0x5651543bc750 msgr2=0x56515444be80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:37.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.730+0000 7fbc87ecef40 1 -- 172.21.15.88:0/4176569616 shutdown_connections 2024-09-16T01:40:37.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.730+0000 7fbc87ecef40 1 -- 172.21.15.88:0/4176569616 wait complete. 2024-09-16T01:40:37.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.730+0000 7fbc87ecef40 1 Processor -- start 2024-09-16T01:40:37.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.731+0000 7fbc87ecef40 1 -- start start 2024-09-16T01:40:37.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.731+0000 7fbc87ecef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5651543a0fe0 0x56515443b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.731+0000 7fbc87ecef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5651543a1a60 0x56515443e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.731+0000 7fbc87ecef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5651543bb250 0x56515443a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.731+0000 7fbc87ecef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x565154449240 con 0x5651543a1a60 2024-09-16T01:40:37.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.731+0000 7fbc87ecef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5651543bd080 con 0x5651543a0fe0 2024-09-16T01:40:37.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.731+0000 7fbc87ecef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5651543bad40 con 0x5651543bb250 2024-09-16T01:40:37.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.731+0000 7fbc7ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5651543a1a60 0x56515443e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:37.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.731+0000 7fbc7ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5651543a1a60 0x56515443e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44220/0 (socket says 172.21.15.88:44220) 2024-09-16T01:40:37.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.731+0000 7fbc7ffff640 1 -- 172.21.15.88:0/2840742118 learned_addr learned my addr 172.21.15.88:0/2840742118 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:37.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.732+0000 7fbc7ffff640 1 -- 172.21.15.88:0/2840742118 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5651543bb250 msgr2=0x56515443a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:37.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.732+0000 7fbc7ffff640 1 --2- 172.21.15.88:0/2840742118 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5651543bb250 0x56515443a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.732+0000 7fbc7ffff640 1 -- 172.21.15.88:0/2840742118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5651543a0fe0 msgr2=0x56515443b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:37.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.732+0000 7fbc859c4640 1 --2- 172.21.15.88:0/2840742118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5651543a0fe0 0x56515443b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:37.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.732+0000 7fbc7ffff640 1 --2- 172.21.15.88:0/2840742118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5651543a0fe0 0x56515443b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.732+0000 7fbc7ffff640 1 -- 172.21.15.88:0/2840742118 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56515443a9f0 con 0x5651543a1a60 2024-09-16T01:40:37.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.732+0000 7fbc7ffff640 1 --2- 172.21.15.88:0/2840742118 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5651543a1a60 0x56515443e520 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fbc70002000 tx=0x7fbc70000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:37.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.732+0000 7fbc7effd640 1 -- 172.21.15.88:0/2840742118 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc7001c440 con 0x5651543a1a60 2024-09-16T01:40:37.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.732+0000 7fbc7effd640 1 -- 172.21.15.88:0/2840742118 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc70021020 con 0x5651543a1a60 2024-09-16T01:40:37.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.732+0000 7fbc7effd640 1 -- 172.21.15.88:0/2840742118 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc7001c620 con 0x5651543a1a60 2024-09-16T01:40:37.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.732+0000 7fbc87ecef40 1 -- 172.21.15.88:0/2840742118 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5651544414c0 con 0x5651543a1a60 2024-09-16T01:40:37.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.732+0000 7fbc87ecef40 1 -- 172.21.15.88:0/2840742118 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565154441a80 con 0x5651543a1a60 2024-09-16T01:40:37.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.733+0000 7fbc7effd640 1 -- 172.21.15.88:0/2840742118 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fbc70017070 con 0x5651543a1a60 2024-09-16T01:40:37.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.734+0000 7fbc7effd640 1 --2- 172.21.15.88:0/2840742118 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc6806ac90 0x7fbc6806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.734+0000 7fbc7effd640 1 -- 172.21.15.88:0/2840742118 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5967+0+0 (secure 0 0 0) 0x7fbc7008ba20 con 0x5651543a1a60 2024-09-16T01:40:37.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.734+0000 7fbc859c4640 1 --2- 172.21.15.88:0/2840742118 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc6806ac90 0x7fbc6806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:37.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.734+0000 7fbc87ecef40 1 --2- 172.21.15.88:0/2840742118 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x56515443bee0 0x56515443c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.734+0000 7fbc87ecef40 1 -- 172.21.15.88:0/2840742118 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:1 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e88) -- 0x56515443c830 con 0x56515443bee0 2024-09-16T01:40:37.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.734+0000 7fbc859c4640 1 --2- 172.21.15.88:0/2840742118 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc6806ac90 0x7fbc6806d150 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fbc78057000 tx=0x7fbc7805a040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:37.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.734+0000 7fbc861c5640 1 --2- 172.21.15.88:0/2840742118 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x56515443bee0 0x56515443c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:37.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.735+0000 7fbc861c5640 1 --2- 172.21.15.88:0/2840742118 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x56515443bee0 0x56515443c2f0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:37.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.736+0000 7fbc861c5640 1 -- 172.21.15.88:0/2840742118 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(1 1.00000000.inode [delete] v88'89 uv89 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbc80001690 con 0x56515443bee0 2024-09-16T01:40:37.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.736+0000 7fbc87ecef40 1 -- 172.21.15.88:0/2840742118 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x56515443bee0 msgr2=0x56515443c2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:37.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.736+0000 7fbc87ecef40 1 --2- 172.21.15.88:0/2840742118 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x56515443bee0 0x56515443c2f0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.736+0000 7fbc87ecef40 1 -- 172.21.15.88:0/2840742118 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc6806ac90 msgr2=0x7fbc6806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:37.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.736+0000 7fbc87ecef40 1 --2- 172.21.15.88:0/2840742118 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc6806ac90 0x7fbc6806d150 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fbc78057000 tx=0x7fbc7805a040 comp rx=0 tx=0).stop 2024-09-16T01:40:37.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.736+0000 7fbc87ecef40 1 -- 172.21.15.88:0/2840742118 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5651543a1a60 msgr2=0x56515443e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:37.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.736+0000 7fbc87ecef40 1 --2- 172.21.15.88:0/2840742118 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5651543a1a60 0x56515443e520 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fbc70002000 tx=0x7fbc70000ed0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.736+0000 7fbc861c5640 1 -- 172.21.15.88:0/2840742118 reap_dead start 2024-09-16T01:40:37.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.736+0000 7fbc87ecef40 1 -- 172.21.15.88:0/2840742118 shutdown_connections 2024-09-16T01:40:37.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.736+0000 7fbc87ecef40 1 -- 172.21.15.88:0/2840742118 >> 172.21.15.88:0/2840742118 conn(0x5651543bc750 msgr2=0x565154442b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:37.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.736+0000 7fbc87ecef40 1 -- 172.21.15.88:0/2840742118 shutdown_connections 2024-09-16T01:40:37.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.736+0000 7fbc87ecef40 1 -- 172.21.15.88:0/2840742118 wait complete. 2024-09-16T01:40:37.745 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:40:37.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.885+0000 7f9844a77640 1 Processor -- start 2024-09-16T01:40:37.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.885+0000 7f9844a77640 1 -- start start 2024-09-16T01:40:37.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.886+0000 7f9844a77640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f984015f9f0 0x7f9840062840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.886+0000 7f9844a77640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f984015fdd0 con 0x7f98400d0970 2024-09-16T01:40:37.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.886+0000 7f9844a77640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98400ce540 con 0x7f98400b6850 2024-09-16T01:40:37.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.886+0000 7f9844a77640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f98400b0180 con 0x7f984015f9f0 2024-09-16T01:40:37.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.886+0000 7f983e59c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f984015f9f0 0x7f9840062840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:37.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.886+0000 7f983e59c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f984015f9f0 0x7f9840062840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44236/0 (socket says 172.21.15.88:44236) 2024-09-16T01:40:37.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.886+0000 7f983dd9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f98400d0970 0x7f98400b6140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46622/0 (socket says 172.21.15.88:46622) 2024-09-16T01:40:37.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.886+0000 7f983e59c640 1 -- 172.21.15.88:0/1164669753 learned_addr learned my addr 172.21.15.88:0/1164669753 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:37.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.887+0000 7f983e59c640 1 -- 172.21.15.88:0/1164669753 >> v1:172.21.15.154:6789/0 conn(0x7f98400b6850 legacy=0x7f984015cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:37.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.887+0000 7f983e59c640 1 -- 172.21.15.88:0/1164669753 >> v1:172.21.15.88:6789/0 conn(0x7f98400d0970 legacy=0x7f98400b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:37.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.887+0000 7f983d59a640 1 -- 172.21.15.88:0/1164669753 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 789213174 0 0) 0x7f984015fdd0 con 0x7f98400d0970 2024-09-16T01:40:37.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.887+0000 7f983e59c640 1 -- 172.21.15.88:0/1164669753 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98400b0460 con 0x7f984015f9f0 2024-09-16T01:40:37.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.887+0000 7f983e59c640 1 --2- 172.21.15.88:0/1164669753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f984015f9f0 0x7f9840062840 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f9830009e50 tx=0x7f983002a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5599b102e4031245 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:37.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.888+0000 7f983d59a640 1 -- 172.21.15.88:0/1164669753 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9830036070 con 0x7f984015f9f0 2024-09-16T01:40:37.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.888+0000 7f983d59a640 1 -- 172.21.15.88:0/1164669753 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9830031600 con 0x7f984015f9f0 2024-09-16T01:40:37.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.888+0000 7f983d59a640 1 -- 172.21.15.88:0/1164669753 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98300319f0 con 0x7f984015f9f0 2024-09-16T01:40:37.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.888+0000 7f9844a77640 1 -- 172.21.15.88:0/1164669753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f984015f9f0 msgr2=0x7f9840062840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:37.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.888+0000 7f9844a77640 1 --2- 172.21.15.88:0/1164669753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f984015f9f0 0x7f9840062840 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f9830009e50 tx=0x7f983002a7c0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.888+0000 7f9844a77640 1 -- 172.21.15.88:0/1164669753 shutdown_connections 2024-09-16T01:40:37.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.888+0000 7f9844a77640 1 --2- 172.21.15.88:0/1164669753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f984015f9f0 0x7f9840062840 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.888+0000 7f9844a77640 1 -- 172.21.15.88:0/1164669753 >> 172.21.15.88:0/1164669753 conn(0x7f98400b0aa0 msgr2=0x7f98400b0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:37.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.888+0000 7f9844a77640 1 -- 172.21.15.88:0/1164669753 shutdown_connections 2024-09-16T01:40:37.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.888+0000 7f9844a77640 1 -- 172.21.15.88:0/1164669753 wait complete. 2024-09-16T01:40:37.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.889+0000 7f9844a77640 1 Processor -- start 2024-09-16T01:40:37.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.890+0000 7f9844a77640 1 -- start start 2024-09-16T01:40:37.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.890+0000 7f9844a77640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98400b6850 0x7f984015bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.890+0000 7f9844a77640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98400d0970 0x7f9840159d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.890+0000 7f9844a77640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f984015f9f0 0x7f984015a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.890+0000 7f9844a77640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f984015d390 con 0x7f984015f9f0 2024-09-16T01:40:37.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.890+0000 7f9844a77640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f98400ce820 con 0x7f98400d0970 2024-09-16T01:40:37.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.890+0000 7f9844a77640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f984015d510 con 0x7f98400b6850 2024-09-16T01:40:37.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.890+0000 7f983ed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f984015f9f0 0x7f984015a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:37.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.890+0000 7f983ed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f984015f9f0 0x7f984015a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44250/0 (socket says 172.21.15.88:44250) 2024-09-16T01:40:37.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.890+0000 7f983ed9d640 1 -- 172.21.15.88:0/881158818 learned_addr learned my addr 172.21.15.88:0/881158818 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:37.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.890+0000 7f983dd9b640 1 --2- 172.21.15.88:0/881158818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98400d0970 0x7f9840159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:37.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.890+0000 7f983e59c640 1 --2- 172.21.15.88:0/881158818 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98400b6850 0x7f984015bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:37.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.891+0000 7f983ed9d640 1 -- 172.21.15.88:0/881158818 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98400b6850 msgr2=0x7f984015bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:37.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.891+0000 7f983ed9d640 1 --2- 172.21.15.88:0/881158818 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98400b6850 0x7f984015bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.891+0000 7f983ed9d640 1 -- 172.21.15.88:0/881158818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98400d0970 msgr2=0x7f9840159d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:37.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.891+0000 7f983ed9d640 1 --2- 172.21.15.88:0/881158818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98400d0970 0x7f9840159d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:37.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.891+0000 7f983ed9d640 1 -- 172.21.15.88:0/881158818 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98400ae260 con 0x7f984015f9f0 2024-09-16T01:40:37.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.891+0000 7f983ed9d640 1 --2- 172.21.15.88:0/881158818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f984015f9f0 0x7f984015a260 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f983800c3a0 tx=0x7f983800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:37.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.891+0000 7f982effd640 1 -- 172.21.15.88:0/881158818 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9838017070 con 0x7f984015f9f0 2024-09-16T01:40:37.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.891+0000 7f982effd640 1 -- 172.21.15.88:0/881158818 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98380052d0 con 0x7f984015f9f0 2024-09-16T01:40:37.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.891+0000 7f9844a77640 1 -- 172.21.15.88:0/881158818 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98400ae540 con 0x7f984015f9f0 2024-09-16T01:40:37.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.891+0000 7f982effd640 1 -- 172.21.15.88:0/881158818 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9838003830 con 0x7f984015f9f0 2024-09-16T01:40:37.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.891+0000 7f9844a77640 1 -- 172.21.15.88:0/881158818 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98400aeb00 con 0x7f984015f9f0 2024-09-16T01:40:37.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.892+0000 7f9844a77640 1 -- 172.21.15.88:0/881158818 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9840062840 con 0x7f984015f9f0 2024-09-16T01:40:37.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.893+0000 7f982effd640 1 -- 172.21.15.88:0/881158818 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f983801a070 con 0x7f984015f9f0 2024-09-16T01:40:37.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.893+0000 7f982effd640 1 --2- 172.21.15.88:0/881158818 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f982006c2b0 0x7f982006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:37.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.893+0000 7f982effd640 1 -- 172.21.15.88:0/881158818 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5967+0+0 (secure 0 0 0) 0x7f983808a8e0 con 0x7f984015f9f0 2024-09-16T01:40:37.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.893+0000 7f983e59c640 1 --2- 172.21.15.88:0/881158818 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f982006c2b0 0x7f982006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:37.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.893+0000 7f983e59c640 1 --2- 172.21.15.88:0/881158818 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f982006c2b0 0x7f982006e770 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f9830033000 tx=0x7f9830009000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:37.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:37.897+0000 7f982effd640 1 -- 172.21.15.88:0/881158818 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9838058930 con 0x7f984015f9f0 2024-09-16T01:40:38.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.096+0000 7f9844a77640 1 -- 172.21.15.88:0/881158818 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f984006ddc0 con 0x7f984015f9f0 2024-09-16T01:40:38.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.247+0000 7f982effd640 1 -- 172.21.15.88:0/881158818 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v90) ==== 162+0+0 (secure 0 0 0) 0x7f983805cc30 con 0x7f984015f9f0 2024-09-16T01:40:38.249 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:40:38.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.250+0000 7f9844a77640 1 -- 172.21.15.88:0/881158818 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f982006c2b0 msgr2=0x7f982006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:38.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.250+0000 7f9844a77640 1 --2- 172.21.15.88:0/881158818 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f982006c2b0 0x7f982006e770 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f9830033000 tx=0x7f9830009000 comp rx=0 tx=0).stop 2024-09-16T01:40:38.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.250+0000 7f9844a77640 1 -- 172.21.15.88:0/881158818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f984015f9f0 msgr2=0x7f984015a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:38.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.250+0000 7f9844a77640 1 --2- 172.21.15.88:0/881158818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f984015f9f0 0x7f984015a260 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f983800c3a0 tx=0x7f983800c860 comp rx=0 tx=0).stop 2024-09-16T01:40:38.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.250+0000 7f9844a77640 1 -- 172.21.15.88:0/881158818 shutdown_connections 2024-09-16T01:40:38.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.250+0000 7f9844a77640 1 --2- 172.21.15.88:0/881158818 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f982006c2b0 0x7f982006e770 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.250+0000 7f9844a77640 1 --2- 172.21.15.88:0/881158818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f984015f9f0 0x7f984015a260 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.250+0000 7f9844a77640 1 --2- 172.21.15.88:0/881158818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98400d0970 0x7f9840159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.250+0000 7f9844a77640 1 --2- 172.21.15.88:0/881158818 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98400b6850 0x7f984015bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.250+0000 7f9844a77640 1 -- 172.21.15.88:0/881158818 >> 172.21.15.88:0/881158818 conn(0x7f98400b0aa0 msgr2=0x7f984006f550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:38.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.250+0000 7f9844a77640 1 -- 172.21.15.88:0/881158818 shutdown_connections 2024-09-16T01:40:38.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.250+0000 7f9844a77640 1 -- 172.21.15.88:0/881158818 wait complete. 2024-09-16T01:40:38.255 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:40:38.254+0000 7f54a2d5a640 -1 mds.0.cache.ino(0x1) No data while reading inode 0x1 2024-09-16T01:40:38.255 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:40:38.254+0000 7f54a2d5a640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-09-16T01:40:38.256 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:40:38.255+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:40:38.256 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:40:38.255+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:40:38.264 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:40:38.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.360+0000 7f8d2228b640 1 Processor -- start 2024-09-16T01:40:38.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.360+0000 7f8d2228b640 1 -- start start 2024-09-16T01:40:38.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.361+0000 7f8d2228b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d1c0d3310 0x7f8d1c0d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:38.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.361+0000 7f8d2228b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8d1c0d1650 0x7f8d1c0b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:38.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.361+0000 7f8d2228b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d1c15ea40 con 0x7f8d1c15e660 2024-09-16T01:40:38.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.361+0000 7f8d2228b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f8d1c0d1ac0 con 0x7f8d1c0d1650 2024-09-16T01:40:38.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.361+0000 7f8d2228b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8d1c0d3870 con 0x7f8d1c0d3310 2024-09-16T01:40:38.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.361+0000 7f8d1bfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8d1c15e660 0x7f8d1c0cf730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46624/0 (socket says 172.21.15.88:46624) 2024-09-16T01:40:38.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.361+0000 7f8d1bfff640 1 -- 172.21.15.88:0/1376426871 learned_addr learned my addr 172.21.15.88:0/1376426871 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:38.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d1affd640 1 -- 172.21.15.88:0/1376426871 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1744189342 0 0) 0x7f8d1c15ea40 con 0x7f8d1c15e660 2024-09-16T01:40:38.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d1affd640 1 -- 172.21.15.88:0/1376426871 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d04003520 con 0x7f8d1c15e660 2024-09-16T01:40:38.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d1b7fe640 1 --2- 172.21.15.88:0/1376426871 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d1c0d3310 0x7f8d1c0d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:38.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d20828640 1 --2- 172.21.15.88:0/1376426871 >> v2:172.21.15.154:3300/0 conn(0x7f8d1c0d1650 0x7f8d1c0b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:38.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d1affd640 1 -- 172.21.15.88:0/1376426871 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 579738945 0 0) 0x7f8d04003520 con 0x7f8d1c15e660 2024-09-16T01:40:38.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d1affd640 1 -- 172.21.15.88:0/1376426871 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d1c0d3310 msgr2=0x7f8d1c0d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:38.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d1affd640 1 --2- 172.21.15.88:0/1376426871 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d1c0d3310 0x7f8d1c0d1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d1affd640 1 -- 172.21.15.88:0/1376426871 >> v2:172.21.15.154:3300/0 conn(0x7f8d1c0d1650 msgr2=0x7f8d1c0b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:38.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d1affd640 1 --2- 172.21.15.88:0/1376426871 >> v2:172.21.15.154:3300/0 conn(0x7f8d1c0d1650 0x7f8d1c0b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d1affd640 1 -- 172.21.15.88:0/1376426871 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d1c0d3b50 con 0x7f8d1c15e660 2024-09-16T01:40:38.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d1affd640 1 -- 172.21.15.88:0/1376426871 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f8d0c001670 con 0x7f8d1c15e660 2024-09-16T01:40:38.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d1affd640 1 -- 172.21.15.88:0/1376426871 >> v1:172.21.15.88:6789/0 conn(0x7f8d1c15e660 legacy=0x7f8d1c0cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:38.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d1affd640 1 --2- 172.21.15.88:0/1376426871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d040039b0 0x7f8d04003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:38.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d1affd640 1 --2- 172.21.15.88:0/1376426871 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d04004800 0x7f8d04004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:38.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d1affd640 1 --2- 172.21.15.88:0/1376426871 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d040051c0 0x7f8d0400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:38.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d1affd640 1 -- 172.21.15.88:0/1376426871 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c0d1ac0 con 0x7f8d04004800 2024-09-16T01:40:38.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d1affd640 1 -- 172.21.15.88:0/1376426871 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c0d3870 con 0x7f8d040039b0 2024-09-16T01:40:38.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d1affd640 1 -- 172.21.15.88:0/1376426871 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8d0400bf30 con 0x7f8d040051c0 2024-09-16T01:40:38.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d1b7fe640 1 --2- 172.21.15.88:0/1376426871 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d1c0d3310 0x7f8d1c0d1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:38.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d1bfff640 1 --2- 172.21.15.88:0/1376426871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d040039b0 0x7f8d04003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:38.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d1b7fe640 1 --2- 172.21.15.88:0/1376426871 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d04004800 0x7f8d04004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:38.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d20828640 1 --2- 172.21.15.88:0/1376426871 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d040051c0 0x7f8d0400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:38.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d1b7fe640 1 -- 172.21.15.88:0/1376426871 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d040051c0 msgr2=0x7f8d0400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:38.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d1b7fe640 1 --2- 172.21.15.88:0/1376426871 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d040051c0 0x7f8d0400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d1b7fe640 1 -- 172.21.15.88:0/1376426871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d040039b0 msgr2=0x7f8d04003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:38.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d1b7fe640 1 --2- 172.21.15.88:0/1376426871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d040039b0 0x7f8d04003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.362+0000 7f8d1b7fe640 1 -- 172.21.15.88:0/1376426871 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d04003520 con 0x7f8d04004800 2024-09-16T01:40:38.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.363+0000 7f8d1b7fe640 1 --2- 172.21.15.88:0/1376426871 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d04004800 0x7f8d04004c80 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f8d10006fd0 tx=0x7f8d04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a342a246ca4fb088 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:38.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.363+0000 7f8d1affd640 1 -- 172.21.15.88:0/1376426871 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d10037070 con 0x7f8d04004800 2024-09-16T01:40:38.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.364+0000 7f8d1affd640 1 -- 172.21.15.88:0/1376426871 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d1002ab30 con 0x7f8d04004800 2024-09-16T01:40:38.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.364+0000 7f8d1affd640 1 -- 172.21.15.88:0/1376426871 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d1000f040 con 0x7f8d04004800 2024-09-16T01:40:38.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.364+0000 7f8d2228b640 1 -- 172.21.15.88:0/1376426871 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d04004800 msgr2=0x7f8d04004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:38.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.364+0000 7f8d2228b640 1 --2- 172.21.15.88:0/1376426871 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d04004800 0x7f8d04004c80 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f8d10006fd0 tx=0x7f8d04005830 comp rx=0 tx=0).stop 2024-09-16T01:40:38.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.364+0000 7f8d20828640 1 -- 172.21.15.88:0/1376426871 reap_dead start 2024-09-16T01:40:38.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.364+0000 7f8d2228b640 1 -- 172.21.15.88:0/1376426871 shutdown_connections 2024-09-16T01:40:38.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.364+0000 7f8d2228b640 1 -- 172.21.15.88:0/1376426871 >> 172.21.15.88:0/1376426871 conn(0x7f8d1c0b6410 msgr2=0x7f8d1c0b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:38.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.364+0000 7f8d2228b640 1 -- 172.21.15.88:0/1376426871 shutdown_connections 2024-09-16T01:40:38.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.364+0000 7f8d2228b640 1 -- 172.21.15.88:0/1376426871 wait complete. 2024-09-16T01:40:38.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.365+0000 7f8d2228b640 1 Processor -- start 2024-09-16T01:40:38.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.366+0000 7f8d2228b640 1 -- start start 2024-09-16T01:40:38.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.366+0000 7f8d2228b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d04004800 0x7f8d1c0ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:38.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.366+0000 7f8d2228b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d1c155c60 0x7f8d1c151e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:38.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.366+0000 7f8d2228b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d1c153670 0x7f8d1c153af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:38.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.366+0000 7f8d2228b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c0d43f0 con 0x7f8d1c153670 2024-09-16T01:40:38.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.366+0000 7f8d2228b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c0d4110 con 0x7f8d1c155c60 2024-09-16T01:40:38.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.366+0000 7f8d2228b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8d1c0d1110 con 0x7f8d04004800 2024-09-16T01:40:38.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.366+0000 7f8d20828640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d1c153670 0x7f8d1c153af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:38.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.366+0000 7f8d20828640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d1c153670 0x7f8d1c153af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44268/0 (socket says 172.21.15.88:44268) 2024-09-16T01:40:38.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.366+0000 7f8d20828640 1 -- 172.21.15.88:0/156853225 learned_addr learned my addr 172.21.15.88:0/156853225 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:38.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.367+0000 7f8d1bfff640 1 --2- 172.21.15.88:0/156853225 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d04004800 0x7f8d1c0ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:38.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.367+0000 7f8d1b7fe640 1 --2- 172.21.15.88:0/156853225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d1c155c60 0x7f8d1c151e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:38.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.367+0000 7f8d20828640 1 -- 172.21.15.88:0/156853225 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d04004800 msgr2=0x7f8d1c0ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:38.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.367+0000 7f8d20828640 1 --2- 172.21.15.88:0/156853225 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d04004800 0x7f8d1c0ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.367+0000 7f8d20828640 1 -- 172.21.15.88:0/156853225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d1c155c60 msgr2=0x7f8d1c151e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:38.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.367+0000 7f8d20828640 1 --2- 172.21.15.88:0/156853225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d1c155c60 0x7f8d1c151e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.367+0000 7f8d20828640 1 -- 172.21.15.88:0/156853225 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d1c160980 con 0x7f8d1c153670 2024-09-16T01:40:38.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.367+0000 7f8d1bfff640 1 --2- 172.21.15.88:0/156853225 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d04004800 0x7f8d1c0ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.367+0000 7f8d1b7fe640 1 --2- 172.21.15.88:0/156853225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d1c155c60 0x7f8d1c151e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.367+0000 7f8d20828640 1 --2- 172.21.15.88:0/156853225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d1c153670 0x7f8d1c153af0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c0d5800 tx=0x7f8d1400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.367+0000 7f8d18ff9640 1 -- 172.21.15.88:0/156853225 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d1400fd80 con 0x7f8d1c153670 2024-09-16T01:40:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.367+0000 7f8d18ff9640 1 -- 172.21.15.88:0/156853225 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d14015430 con 0x7f8d1c153670 2024-09-16T01:40:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.367+0000 7f8d18ff9640 1 -- 172.21.15.88:0/156853225 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d14015820 con 0x7f8d1c153670 2024-09-16T01:40:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.367+0000 7f8d2228b640 1 -- 172.21.15.88:0/156853225 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d1c158fa0 con 0x7f8d1c153670 2024-09-16T01:40:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.367+0000 7f8d2228b640 1 -- 172.21.15.88:0/156853225 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d1c156a10 con 0x7f8d1c153670 2024-09-16T01:40:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.368+0000 7f8d2228b640 1 -- 172.21.15.88:0/156853225 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d1c0cf730 con 0x7f8d1c153670 2024-09-16T01:40:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.369+0000 7f8d18ff9640 1 -- 172.21.15.88:0/156853225 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8d140159c0 con 0x7f8d1c153670 2024-09-16T01:40:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.369+0000 7f8d18ff9640 1 --2- 172.21.15.88:0/156853225 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8d0806c2b0 0x7f8d0806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.369+0000 7f8d1bfff640 1 --2- 172.21.15.88:0/156853225 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8d0806c2b0 0x7f8d0806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.369+0000 7f8d18ff9640 1 -- 172.21.15.88:0/156853225 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6053+0+0 (secure 0 0 0) 0x7f8d1408c400 con 0x7f8d1c153670 2024-09-16T01:40:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.369+0000 7f8d1bfff640 1 --2- 172.21.15.88:0/156853225 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8d0806c2b0 0x7f8d0806e770 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c007fd0 tx=0x7f8d0c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:38.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.373+0000 7f8d18ff9640 1 -- 172.21.15.88:0/156853225 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8d1c160980 con 0x7f8d1c153670 2024-09-16T01:40:38.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.571+0000 7f8d2228b640 1 -- 172.21.15.88:0/156853225 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d1c15d2c0 con 0x7f8d1c153670 2024-09-16T01:40:38.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.572+0000 7f8d18ff9640 1 -- 172.21.15.88:0/156853225 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+4388 (secure 0 0 0) 0x7f8d14019030 con 0x7f8d1c153670 2024-09-16T01:40:38.573 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:38.574 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":91,"btime":"2024-09-16T01:40:38:255962+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6849,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/1289932173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1289932173},{"type":"v1","addr":"172.21.15.88:6836","nonce":1289932173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:40:38.255911+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:40:38.574 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 91 2024-09-16T01:40:38.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.574+0000 7f8d2228b640 1 -- 172.21.15.88:0/156853225 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8d0806c2b0 msgr2=0x7f8d0806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:38.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.574+0000 7f8d2228b640 1 --2- 172.21.15.88:0/156853225 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8d0806c2b0 0x7f8d0806e770 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c007fd0 tx=0x7f8d0c009040 comp rx=0 tx=0).stop 2024-09-16T01:40:38.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.574+0000 7f8d2228b640 1 -- 172.21.15.88:0/156853225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d1c153670 msgr2=0x7f8d1c153af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:38.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.574+0000 7f8d2228b640 1 --2- 172.21.15.88:0/156853225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d1c153670 0x7f8d1c153af0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c0d5800 tx=0x7f8d1400f580 comp rx=0 tx=0).stop 2024-09-16T01:40:38.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.574+0000 7f8d2228b640 1 -- 172.21.15.88:0/156853225 shutdown_connections 2024-09-16T01:40:38.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.574+0000 7f8d2228b640 1 --2- 172.21.15.88:0/156853225 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8d0806c2b0 0x7f8d0806e770 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.574+0000 7f8d2228b640 1 --2- 172.21.15.88:0/156853225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d1c153670 0x7f8d1c153af0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.574+0000 7f8d2228b640 1 --2- 172.21.15.88:0/156853225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d1c155c60 0x7f8d1c151e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.574+0000 7f8d2228b640 1 --2- 172.21.15.88:0/156853225 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8d04004800 0x7f8d1c0ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.574+0000 7f8d2228b640 1 -- 172.21.15.88:0/156853225 >> 172.21.15.88:0/156853225 conn(0x7f8d1c0b6410 msgr2=0x7f8d1c0ba140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:38.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.574+0000 7f8d2228b640 1 -- 172.21.15.88:0/156853225 shutdown_connections 2024-09-16T01:40:38.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.574+0000 7f8d2228b640 1 -- 172.21.15.88:0/156853225 wait complete. 2024-09-16T01:40:38.586 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:40:38.586 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:40:38.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.683+0000 7faa8da50640 1 Processor -- start 2024-09-16T01:40:38.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.683+0000 7faa8da50640 1 -- start start 2024-09-16T01:40:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.684+0000 7faa8da50640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7faa880b63d0 0x7faa880d39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.684+0000 7faa8da50640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faa880cf6b0 0x7faa880b7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.684+0000 7faa8da50640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa880d1c80 0x7faa880b7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.684+0000 7faa8da50640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7faa880b8300 con 0x7faa880cf6b0 2024-09-16T01:40:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.684+0000 7faa8da50640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7faa880cfc20 con 0x7faa880b63d0 2024-09-16T01:40:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.684+0000 7faa8da50640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7faa880d2ed0 con 0x7faa880d1c80 2024-09-16T01:40:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.684+0000 7faa7e7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faa880cf6b0 0x7faa880b7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.684+0000 7faa7e7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faa880cf6b0 0x7faa880b7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44282/0 (socket says 172.21.15.88:44282) 2024-09-16T01:40:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.684+0000 7faa7e7fc640 1 -- 172.21.15.88:0/1367517904 learned_addr learned my addr 172.21.15.88:0/1367517904 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.684+0000 7faa7e7fc640 1 -- 172.21.15.88:0/1367517904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa880d1c80 msgr2=0x7faa880b7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:38.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.684+0000 7faa7e7fc640 1 --2- 172.21.15.88:0/1367517904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa880d1c80 0x7faa880b7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.684+0000 7faa7e7fc640 1 -- 172.21.15.88:0/1367517904 >> v2:172.21.15.154:3300/0 conn(0x7faa880b63d0 msgr2=0x7faa880d39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:38.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.684+0000 7faa7e7fc640 1 --2- 172.21.15.88:0/1367517904 >> v2:172.21.15.154:3300/0 conn(0x7faa880b63d0 0x7faa880d39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.684+0000 7faa7e7fc640 1 -- 172.21.15.88:0/1367517904 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa880d31b0 con 0x7faa880cf6b0 2024-09-16T01:40:38.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.685+0000 7faa7e7fc640 1 --2- 172.21.15.88:0/1367517904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa880cf6b0 0x7faa880b7880 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7faa88050260 tx=0x7faa7002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1945c5a60281711 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:38.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.685+0000 7faa867fc640 1 -- 172.21.15.88:0/1367517904 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa70037070 con 0x7faa880cf6b0 2024-09-16T01:40:38.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.685+0000 7faa867fc640 1 -- 172.21.15.88:0/1367517904 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa70032440 con 0x7faa880cf6b0 2024-09-16T01:40:38.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.685+0000 7faa867fc640 1 -- 172.21.15.88:0/1367517904 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa70032810 con 0x7faa880cf6b0 2024-09-16T01:40:38.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.686+0000 7faa8da50640 1 -- 172.21.15.88:0/1367517904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa880cf6b0 msgr2=0x7faa880b7880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:38.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.686+0000 7faa8da50640 1 --2- 172.21.15.88:0/1367517904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa880cf6b0 0x7faa880b7880 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7faa88050260 tx=0x7faa7002a140 comp rx=0 tx=0).stop 2024-09-16T01:40:38.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.686+0000 7faa8da50640 1 -- 172.21.15.88:0/1367517904 shutdown_connections 2024-09-16T01:40:38.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.686+0000 7faa8da50640 1 --2- 172.21.15.88:0/1367517904 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa880d1c80 0x7faa880b7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.686+0000 7faa8da50640 1 --2- 172.21.15.88:0/1367517904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa880cf6b0 0x7faa880b7880 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.686+0000 7faa8da50640 1 --2- 172.21.15.88:0/1367517904 >> v2:172.21.15.154:3300/0 conn(0x7faa880b63d0 0x7faa880d39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.686+0000 7faa8da50640 1 -- 172.21.15.88:0/1367517904 >> 172.21.15.88:0/1367517904 conn(0x7faa880d1160 msgr2=0x7faa880d1570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:38.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.686+0000 7faa8da50640 1 -- 172.21.15.88:0/1367517904 shutdown_connections 2024-09-16T01:40:38.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.686+0000 7faa8da50640 1 -- 172.21.15.88:0/1367517904 wait complete. 2024-09-16T01:40:38.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.687+0000 7faa8da50640 1 Processor -- start 2024-09-16T01:40:38.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.687+0000 7faa8da50640 1 -- start start 2024-09-16T01:40:38.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.688+0000 7faa8da50640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa880b63d0 0x7faa8815b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:38.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.688+0000 7faa8da50640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa880cf6b0 0x7faa8814a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:38.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.688+0000 7faa8da50640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa880d1c80 0x7faa8814ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:38.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.688+0000 7faa8da50640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7faa880cb5c0 con 0x7faa880cf6b0 2024-09-16T01:40:38.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.688+0000 7faa8da50640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faa880cb440 con 0x7faa880d1c80 2024-09-16T01:40:38.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.688+0000 7faa8da50640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7faa880d3ef0 con 0x7faa880b63d0 2024-09-16T01:40:38.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.688+0000 7faa7e7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa880cf6b0 0x7faa8814a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:38.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.688+0000 7faa7e7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa880cf6b0 0x7faa8814a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44290/0 (socket says 172.21.15.88:44290) 2024-09-16T01:40:38.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.688+0000 7faa7e7fc640 1 -- 172.21.15.88:0/365414249 learned_addr learned my addr 172.21.15.88:0/365414249 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:38.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.688+0000 7faa877fe640 1 --2- 172.21.15.88:0/365414249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa880d1c80 0x7faa8814ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:38.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.688+0000 7faa7e7fc640 1 -- 172.21.15.88:0/365414249 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa880b63d0 msgr2=0x7faa8815b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:38.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.688+0000 7faa7e7fc640 1 --2- 172.21.15.88:0/365414249 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa880b63d0 0x7faa8815b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.688+0000 7faa7e7fc640 1 -- 172.21.15.88:0/365414249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa880d1c80 msgr2=0x7faa8814ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:38.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.688+0000 7faa7e7fc640 1 --2- 172.21.15.88:0/365414249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa880d1c80 0x7faa8814ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.688+0000 7faa7e7fc640 1 -- 172.21.15.88:0/365414249 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa881594f0 con 0x7faa880cf6b0 2024-09-16T01:40:38.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.689+0000 7faa877fe640 1 --2- 172.21.15.88:0/365414249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa880d1c80 0x7faa8814ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:38.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.689+0000 7faa7e7fc640 1 --2- 172.21.15.88:0/365414249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa880cf6b0 0x7faa8814a6d0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7faa70000c00 tx=0x7faa700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:38.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.689+0000 7faa84ff9640 1 -- 172.21.15.88:0/365414249 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa70041070 con 0x7faa880cf6b0 2024-09-16T01:40:38.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.689+0000 7faa84ff9640 1 -- 172.21.15.88:0/365414249 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa7003c870 con 0x7faa880cf6b0 2024-09-16T01:40:38.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.689+0000 7faa8da50640 1 -- 172.21.15.88:0/365414249 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa881597d0 con 0x7faa880cf6b0 2024-09-16T01:40:38.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.689+0000 7faa84ff9640 1 -- 172.21.15.88:0/365414249 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa70037040 con 0x7faa880cf6b0 2024-09-16T01:40:38.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.689+0000 7faa8da50640 1 -- 172.21.15.88:0/365414249 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa88159110 con 0x7faa880cf6b0 2024-09-16T01:40:38.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.690+0000 7faa8da50640 1 -- 172.21.15.88:0/365414249 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa881599a0 con 0x7faa880cf6b0 2024-09-16T01:40:38.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.690+0000 7faa84ff9640 1 -- 172.21.15.88:0/365414249 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7faa70044070 con 0x7faa880cf6b0 2024-09-16T01:40:38.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.691+0000 7faa84ff9640 1 --2- 172.21.15.88:0/365414249 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faa6806c2b0 0x7faa6806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:38.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.691+0000 7faa84ff9640 1 -- 172.21.15.88:0/365414249 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6053+0+0 (secure 0 0 0) 0x7faa700ac0f0 con 0x7faa880cf6b0 2024-09-16T01:40:38.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.691+0000 7faa86ffd640 1 --2- 172.21.15.88:0/365414249 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faa6806c2b0 0x7faa6806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:38.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.691+0000 7faa86ffd640 1 --2- 172.21.15.88:0/365414249 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faa6806c2b0 0x7faa6806e770 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7faa78009670 tx=0x7faa78006bf0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:38.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.695+0000 7faa84ff9640 1 -- 172.21.15.88:0/365414249 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7faa7007a790 con 0x7faa880cf6b0 2024-09-16T01:40:38.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.891+0000 7faa8da50640 1 -- 172.21.15.88:0/365414249 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa880cf280 con 0x7faa880cf6b0 2024-09-16T01:40:38.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.892+0000 7faa84ff9640 1 -- 172.21.15.88:0/365414249 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+4388 (secure 0 0 0) 0x7faa7007a090 con 0x7faa880cf6b0 2024-09-16T01:40:38.894 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:38.894 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":91,"btime":"2024-09-16T01:40:38:255962+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6849,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/1289932173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1289932173},{"type":"v1","addr":"172.21.15.88:6836","nonce":1289932173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:40:38.255911+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:40:38.894 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 91 2024-09-16T01:40:38.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.894+0000 7faa8da50640 1 -- 172.21.15.88:0/365414249 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faa6806c2b0 msgr2=0x7faa6806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:38.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.894+0000 7faa8da50640 1 --2- 172.21.15.88:0/365414249 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faa6806c2b0 0x7faa6806e770 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7faa78009670 tx=0x7faa78006bf0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.894+0000 7faa8da50640 1 -- 172.21.15.88:0/365414249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa880cf6b0 msgr2=0x7faa8814a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:38.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.894+0000 7faa8da50640 1 --2- 172.21.15.88:0/365414249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa880cf6b0 0x7faa8814a6d0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7faa70000c00 tx=0x7faa700046c0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.894+0000 7faa8da50640 1 -- 172.21.15.88:0/365414249 shutdown_connections 2024-09-16T01:40:38.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.894+0000 7faa8da50640 1 --2- 172.21.15.88:0/365414249 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faa6806c2b0 0x7faa6806e770 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.894+0000 7faa8da50640 1 --2- 172.21.15.88:0/365414249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa880d1c80 0x7faa8814ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.894+0000 7faa8da50640 1 --2- 172.21.15.88:0/365414249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa880cf6b0 0x7faa8814a6d0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.894+0000 7faa8da50640 1 --2- 172.21.15.88:0/365414249 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa880b63d0 0x7faa8815b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:38.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.894+0000 7faa8da50640 1 -- 172.21.15.88:0/365414249 >> 172.21.15.88:0/365414249 conn(0x7faa880d1160 msgr2=0x7faa880d1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:38.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.894+0000 7faa8da50640 1 -- 172.21.15.88:0/365414249 shutdown_connections 2024-09-16T01:40:38.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:38.894+0000 7faa8da50640 1 -- 172.21.15.88:0/365414249 wait complete. 2024-09-16T01:40:38.906 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:40:38.906 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:40:39.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.002+0000 7f8080b5f640 1 Processor -- start 2024-09-16T01:40:39.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.002+0000 7f8080b5f640 1 -- start start 2024-09-16T01:40:39.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.003+0000 7f8080b5f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f807c15e630 0x7f807c0b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:39.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.003+0000 7f8080b5f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f807c073160 0x7f807c0735f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:39.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.003+0000 7f8080b5f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f807c06d2d0 con 0x7f807c06cc40 2024-09-16T01:40:39.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.003+0000 7f8080b5f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f807c06d080 con 0x7f807c15e630 2024-09-16T01:40:39.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.003+0000 7f8080b5f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f807c06d5b0 con 0x7f807c073160 2024-09-16T01:40:39.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.003+0000 7f8079d9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f807c06cc40 0x7f807c0ceef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46626/0 (socket says 172.21.15.88:46626) 2024-09-16T01:40:39.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.003+0000 7f8079d9b640 1 -- 172.21.15.88:0/732085176 learned_addr learned my addr 172.21.15.88:0/732085176 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:39.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f807959a640 1 -- 172.21.15.88:0/732085176 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1947298926 0 0) 0x7f807c06d2d0 con 0x7f807c06cc40 2024-09-16T01:40:39.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f807959a640 1 -- 172.21.15.88:0/732085176 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8064003520 con 0x7f807c06cc40 2024-09-16T01:40:39.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f807a59c640 1 --2- 172.21.15.88:0/732085176 >> v2:172.21.15.154:3300/0 conn(0x7f807c15e630 0x7f807c0b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:39.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f807ad9d640 1 --2- 172.21.15.88:0/732085176 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f807c073160 0x7f807c0735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:39.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f807959a640 1 -- 172.21.15.88:0/732085176 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1696253578 0 0) 0x7f8064003520 con 0x7f807c06cc40 2024-09-16T01:40:39.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f807959a640 1 -- 172.21.15.88:0/732085176 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f807c073160 msgr2=0x7f807c0735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:39.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f807959a640 1 --2- 172.21.15.88:0/732085176 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f807c073160 0x7f807c0735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:39.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f807959a640 1 -- 172.21.15.88:0/732085176 >> v2:172.21.15.154:3300/0 conn(0x7f807c15e630 msgr2=0x7f807c0b4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:39.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f807959a640 1 --2- 172.21.15.88:0/732085176 >> v2:172.21.15.154:3300/0 conn(0x7f807c15e630 0x7f807c0b4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:39.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f807959a640 1 -- 172.21.15.88:0/732085176 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f807c06d890 con 0x7f807c06cc40 2024-09-16T01:40:39.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f807959a640 1 -- 172.21.15.88:0/732085176 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f8070001670 con 0x7f807c06cc40 2024-09-16T01:40:39.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f807959a640 1 -- 172.21.15.88:0/732085176 >> v1:172.21.15.88:6789/0 conn(0x7f807c06cc40 legacy=0x7f807c0ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:39.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f807959a640 1 --2- 172.21.15.88:0/732085176 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f807c073160 0x7f80640039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:39.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f807959a640 1 --2- 172.21.15.88:0/732085176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8064003f30 0x7f8064004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:39.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f807959a640 1 --2- 172.21.15.88:0/732085176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8064004d40 0x7f80640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:39.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f807959a640 1 -- 172.21.15.88:0/732085176 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f807c06d080 con 0x7f8064004d40 2024-09-16T01:40:39.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f807959a640 1 -- 172.21.15.88:0/732085176 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f807c06d5b0 con 0x7f8064003f30 2024-09-16T01:40:39.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f807959a640 1 -- 172.21.15.88:0/732085176 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f806400b9c0 con 0x7f807c073160 2024-09-16T01:40:39.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f807ad9d640 1 --2- 172.21.15.88:0/732085176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8064004d40 0x7f80640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:39.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f8079d9b640 1 --2- 172.21.15.88:0/732085176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8064003f30 0x7f8064004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:39.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f807a59c640 1 --2- 172.21.15.88:0/732085176 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f807c073160 0x7f80640039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:39.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f807ad9d640 1 -- 172.21.15.88:0/732085176 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f807c073160 msgr2=0x7f80640039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:39.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f807ad9d640 1 --2- 172.21.15.88:0/732085176 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f807c073160 0x7f80640039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:39.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f807ad9d640 1 -- 172.21.15.88:0/732085176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8064003f30 msgr2=0x7f8064004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:39.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f807ad9d640 1 --2- 172.21.15.88:0/732085176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8064003f30 0x7f8064004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:39.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.004+0000 7f807ad9d640 1 -- 172.21.15.88:0/732085176 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8064003520 con 0x7f8064004d40 2024-09-16T01:40:39.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.005+0000 7f807ad9d640 1 --2- 172.21.15.88:0/732085176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8064004d40 0x7f80640051c0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f8074009fc0 tx=0x7f8064005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45d501af2f7057d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:39.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.005+0000 7f807959a640 1 -- 172.21.15.88:0/732085176 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8074009c10 con 0x7f8064004d40 2024-09-16T01:40:39.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.005+0000 7f807959a640 1 -- 172.21.15.88:0/732085176 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8074036070 con 0x7f8064004d40 2024-09-16T01:40:39.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.005+0000 7f807959a640 1 -- 172.21.15.88:0/732085176 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8074031670 con 0x7f8064004d40 2024-09-16T01:40:39.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.006+0000 7f8080b5f640 1 -- 172.21.15.88:0/732085176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8064004d40 msgr2=0x7f80640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:39.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.006+0000 7f8080b5f640 1 --2- 172.21.15.88:0/732085176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8064004d40 0x7f80640051c0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f8074009fc0 tx=0x7f8064005830 comp rx=0 tx=0).stop 2024-09-16T01:40:39.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.006+0000 7f807ad9d640 1 -- 172.21.15.88:0/732085176 reap_dead start 2024-09-16T01:40:39.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.006+0000 7f8080b5f640 1 -- 172.21.15.88:0/732085176 shutdown_connections 2024-09-16T01:40:39.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.006+0000 7f8080b5f640 1 -- 172.21.15.88:0/732085176 >> 172.21.15.88:0/732085176 conn(0x7f807c0b4240 msgr2=0x7f807c0b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:39.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.006+0000 7f8080b5f640 1 -- 172.21.15.88:0/732085176 shutdown_connections 2024-09-16T01:40:39.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.006+0000 7f8080b5f640 1 -- 172.21.15.88:0/732085176 wait complete. 2024-09-16T01:40:39.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.007+0000 7f8080b5f640 1 Processor -- start 2024-09-16T01:40:39.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.007+0000 7f8080b5f640 1 -- start start 2024-09-16T01:40:39.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.008+0000 7f8080b5f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8064004d40 0x7f807c14cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:39.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.008+0000 7f8080b5f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f807c159810 0x7f807c15c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:39.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.008+0000 7f8080b5f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f807c15cd30 0x7f807c157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:39.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.008+0000 7f8080b5f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f807c15ea10 con 0x7f807c159810 2024-09-16T01:40:39.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.008+0000 7f8080b5f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f807c06de50 con 0x7f8064004d40 2024-09-16T01:40:39.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.008+0000 7f8080b5f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f807c0735f0 con 0x7f807c15cd30 2024-09-16T01:40:39.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.008+0000 7f8079d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f807c159810 0x7f807c15c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:39.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.008+0000 7f8079d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f807c159810 0x7f807c15c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44314/0 (socket says 172.21.15.88:44314) 2024-09-16T01:40:39.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.008+0000 7f8079d9b640 1 -- 172.21.15.88:0/1226896642 learned_addr learned my addr 172.21.15.88:0/1226896642 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:39.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.008+0000 7f807a59c640 1 --2- 172.21.15.88:0/1226896642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8064004d40 0x7f807c14cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:39.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.008+0000 7f807ad9d640 1 --2- 172.21.15.88:0/1226896642 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f807c15cd30 0x7f807c157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:39.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.008+0000 7f8079d9b640 1 -- 172.21.15.88:0/1226896642 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f807c15cd30 msgr2=0x7f807c157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:39.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.008+0000 7f8079d9b640 1 --2- 172.21.15.88:0/1226896642 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f807c15cd30 0x7f807c157a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:39.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.008+0000 7f8079d9b640 1 -- 172.21.15.88:0/1226896642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8064004d40 msgr2=0x7f807c14cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:39.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.008+0000 7f8079d9b640 1 --2- 172.21.15.88:0/1226896642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8064004d40 0x7f807c14cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:39.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.008+0000 7f8079d9b640 1 -- 172.21.15.88:0/1226896642 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f807c158770 con 0x7f807c159810 2024-09-16T01:40:39.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.008+0000 7f8079d9b640 1 --2- 172.21.15.88:0/1226896642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f807c159810 0x7f807c15c7f0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f8070007340 tx=0x7f8070007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:39.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.009+0000 7f806affd640 1 -- 172.21.15.88:0/1226896642 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f807000f8e0 con 0x7f807c159810 2024-09-16T01:40:39.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.009+0000 7f806affd640 1 -- 172.21.15.88:0/1226896642 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8070018070 con 0x7f807c159810 2024-09-16T01:40:39.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.009+0000 7f8080b5f640 1 -- 172.21.15.88:0/1226896642 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f807c158a50 con 0x7f807c159810 2024-09-16T01:40:39.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.009+0000 7f806affd640 1 -- 172.21.15.88:0/1226896642 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f807000fce0 con 0x7f807c159810 2024-09-16T01:40:39.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.009+0000 7f8080b5f640 1 -- 172.21.15.88:0/1226896642 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f807c159010 con 0x7f807c159810 2024-09-16T01:40:39.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.010+0000 7f8080b5f640 1 -- 172.21.15.88:0/1226896642 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f807c073980 con 0x7f807c159810 2024-09-16T01:40:39.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.010+0000 7f806affd640 1 -- 172.21.15.88:0/1226896642 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f807001d460 con 0x7f807c159810 2024-09-16T01:40:39.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.011+0000 7f806affd640 1 --2- 172.21.15.88:0/1226896642 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f805c06c2b0 0x7f805c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:39.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.011+0000 7f806affd640 1 -- 172.21.15.88:0/1226896642 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6053+0+0 (secure 0 0 0) 0x7f807008b690 con 0x7f807c159810 2024-09-16T01:40:39.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.011+0000 7f807a59c640 1 --2- 172.21.15.88:0/1226896642 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f805c06c2b0 0x7f805c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:39.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.011+0000 7f807a59c640 1 --2- 172.21.15.88:0/1226896642 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f805c06c2b0 0x7f805c06e770 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f806c009b80 tx=0x7f806c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:39.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.015+0000 7f806affd640 1 -- 172.21.15.88:0/1226896642 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8070059ca0 con 0x7f807c159810 2024-09-16T01:40:39.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.212+0000 7f8080b5f640 1 -- 172.21.15.88:0/1226896642 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f807c15d110 con 0x7f807c159810 2024-09-16T01:40:39.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.213+0000 7f806affd640 1 -- 172.21.15.88:0/1226896642 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+4388 (secure 0 0 0) 0x7f80700595a0 con 0x7f807c159810 2024-09-16T01:40:39.214 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:39.214 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":91,"btime":"2024-09-16T01:40:38:255962+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6849,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/1289932173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1289932173},{"type":"v1","addr":"172.21.15.88:6836","nonce":1289932173}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:40:38.255911+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":89,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:40:39.214 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 91 2024-09-16T01:40:39.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.214+0000 7f8080b5f640 1 -- 172.21.15.88:0/1226896642 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f805c06c2b0 msgr2=0x7f805c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:39.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.214+0000 7f8080b5f640 1 --2- 172.21.15.88:0/1226896642 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f805c06c2b0 0x7f805c06e770 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f806c009b80 tx=0x7f806c008040 comp rx=0 tx=0).stop 2024-09-16T01:40:39.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.214+0000 7f8080b5f640 1 -- 172.21.15.88:0/1226896642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f807c159810 msgr2=0x7f807c15c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:39.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.214+0000 7f8080b5f640 1 --2- 172.21.15.88:0/1226896642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f807c159810 0x7f807c15c7f0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f8070007340 tx=0x7f8070007800 comp rx=0 tx=0).stop 2024-09-16T01:40:39.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.214+0000 7f8080b5f640 1 -- 172.21.15.88:0/1226896642 shutdown_connections 2024-09-16T01:40:39.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.214+0000 7f8080b5f640 1 --2- 172.21.15.88:0/1226896642 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f805c06c2b0 0x7f805c06e770 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:39.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.214+0000 7f8080b5f640 1 --2- 172.21.15.88:0/1226896642 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f807c15cd30 0x7f807c157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:39.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.214+0000 7f8080b5f640 1 --2- 172.21.15.88:0/1226896642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f807c159810 0x7f807c15c7f0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:39.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.214+0000 7f8080b5f640 1 --2- 172.21.15.88:0/1226896642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8064004d40 0x7f807c14cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:39.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.214+0000 7f8080b5f640 1 -- 172.21.15.88:0/1226896642 >> 172.21.15.88:0/1226896642 conn(0x7f807c0b4240 msgr2=0x7f807c0d1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:39.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.215+0000 7f8080b5f640 1 -- 172.21.15.88:0/1226896642 shutdown_connections 2024-09-16T01:40:39.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.215+0000 7f8080b5f640 1 -- 172.21.15.88:0/1226896642 wait complete. 2024-09-16T01:40:39.227 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 1.00000000.inode' led to DAMAGED state 2024-09-16T01:40:39.227 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 200.00000000' 2024-09-16T01:40:39.227 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:40:39.228 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:40:39.243 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:40:39.243 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:40:39.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:39.244 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:40:39.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:40:39.299 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:40:39.299 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:40:39.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.436+0000 7f54a76e6640 1 Processor -- start 2024-09-16T01:40:39.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.437+0000 7f54a76e6640 1 -- start start 2024-09-16T01:40:39.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.437+0000 7f54a76e6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f54a00cf1f0 0x7f54a00b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:39.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.437+0000 7f54a76e6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54a00b6720 0x7f54a00d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:39.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.437+0000 7f54a76e6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54a00cf860 con 0x7f54a015e7a0 2024-09-16T01:40:39.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.437+0000 7f54a76e6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f54a00d2ca0 con 0x7f54a00cf1f0 2024-09-16T01:40:39.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.437+0000 7f54a76e6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f54a00cfb40 con 0x7f54a00b6720 2024-09-16T01:40:39.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.437+0000 7f54a4c81640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f54a00cf1f0 0x7f54a00b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:39.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.437+0000 7f54a4c81640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f54a00cf1f0 0x7f54a00b61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44324/0 (socket says 172.21.15.88:44324) 2024-09-16T01:40:39.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.437+0000 7f54a4c81640 1 -- 172.21.15.88:0/1878644964 learned_addr learned my addr 172.21.15.88:0/1878644964 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:39.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.438+0000 7f54a4c81640 1 -- 172.21.15.88:0/1878644964 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54a00b6720 msgr2=0x7f54a00d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:39.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.438+0000 7f54a5c83640 1 --2- 172.21.15.88:0/1878644964 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54a00b6720 0x7f54a00d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:39.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.438+0000 7f54a4c81640 1 --2- 172.21.15.88:0/1878644964 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54a00b6720 0x7f54a00d3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:39.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.438+0000 7f54a4c81640 1 -- 172.21.15.88:0/1878644964 >> v1:172.21.15.154:6789/0 conn(0x7f54a015e7a0 legacy=0x7f54a006caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:39.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.438+0000 7f54a4c81640 1 -- 172.21.15.88:0/1878644964 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54a00cfe20 con 0x7f54a00cf1f0 2024-09-16T01:40:39.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.438+0000 7f54a4c81640 1 --2- 172.21.15.88:0/1878644964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54a00cf1f0 0x7f54a00b61e0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f5494009870 tx=0x7f549402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4be1d05bfce6a307 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:39.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.439+0000 7f549ffff640 1 -- 172.21.15.88:0/1878644964 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5494037070 con 0x7f54a00cf1f0 2024-09-16T01:40:39.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.439+0000 7f549ffff640 1 -- 172.21.15.88:0/1878644964 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5494032440 con 0x7f54a00cf1f0 2024-09-16T01:40:39.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.439+0000 7f549ffff640 1 -- 172.21.15.88:0/1878644964 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5494032810 con 0x7f54a00cf1f0 2024-09-16T01:40:39.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.439+0000 7f54a76e6640 1 -- 172.21.15.88:0/1878644964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54a00cf1f0 msgr2=0x7f54a00b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:39.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.439+0000 7f54a76e6640 1 --2- 172.21.15.88:0/1878644964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54a00cf1f0 0x7f54a00b61e0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f5494009870 tx=0x7f549402a180 comp rx=0 tx=0).stop 2024-09-16T01:40:39.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.439+0000 7f54a76e6640 1 -- 172.21.15.88:0/1878644964 shutdown_connections 2024-09-16T01:40:39.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.439+0000 7f54a76e6640 1 --2- 172.21.15.88:0/1878644964 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54a00b6720 0x7f54a00d3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:39.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.439+0000 7f54a76e6640 1 --2- 172.21.15.88:0/1878644964 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54a00cf1f0 0x7f54a00b61e0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:39.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.439+0000 7f54a76e6640 1 -- 172.21.15.88:0/1878644964 >> 172.21.15.88:0/1878644964 conn(0x7f54a00d11e0 msgr2=0x7f54a00d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:39.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.439+0000 7f54a76e6640 1 -- 172.21.15.88:0/1878644964 shutdown_connections 2024-09-16T01:40:39.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.439+0000 7f54a76e6640 1 -- 172.21.15.88:0/1878644964 wait complete. 2024-09-16T01:40:39.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.440+0000 7f54a76e6640 1 Processor -- start 2024-09-16T01:40:39.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.440+0000 7f54a76e6640 1 -- start start 2024-09-16T01:40:39.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.441+0000 7f54a76e6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54a00b6720 0x7f54a00d45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:39.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.441+0000 7f54a76e6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54a00cf1f0 0x7f54a01f48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:39.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.441+0000 7f54a76e6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54a015e7a0 0x7f54a00ce8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:39.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.441+0000 7f54a76e6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f54a015eb80 con 0x7f54a00b6720 2024-09-16T01:40:39.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.441+0000 7f54a76e6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f54a00b7a00 con 0x7f54a00cf1f0 2024-09-16T01:40:39.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.441+0000 7f54a76e6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f54a015ed00 con 0x7f54a015e7a0 2024-09-16T01:40:39.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.441+0000 7f54a5482640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54a00b6720 0x7f54a00d45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:39.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.441+0000 7f54a5482640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54a00b6720 0x7f54a00d45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44336/0 (socket says 172.21.15.88:44336) 2024-09-16T01:40:39.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.441+0000 7f54a5482640 1 -- 172.21.15.88:0/1722466483 learned_addr learned my addr 172.21.15.88:0/1722466483 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:39.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.441+0000 7f54a4c81640 1 --2- 172.21.15.88:0/1722466483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54a00cf1f0 0x7f54a01f48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:39.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.441+0000 7f54a5c83640 1 --2- 172.21.15.88:0/1722466483 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54a015e7a0 0x7f54a00ce8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:39.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.442+0000 7f54a5482640 1 -- 172.21.15.88:0/1722466483 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54a015e7a0 msgr2=0x7f54a00ce8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:39.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.442+0000 7f54a5482640 1 --2- 172.21.15.88:0/1722466483 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54a015e7a0 0x7f54a00ce8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:39.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.442+0000 7f54a5482640 1 -- 172.21.15.88:0/1722466483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54a00cf1f0 msgr2=0x7f54a01f48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:39.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.442+0000 7f54a5482640 1 --2- 172.21.15.88:0/1722466483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54a00cf1f0 0x7f54a01f48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:39.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.442+0000 7f54a5482640 1 -- 172.21.15.88:0/1722466483 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54a00c8790 con 0x7f54a00b6720 2024-09-16T01:40:39.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.442+0000 7f54a5482640 1 --2- 172.21.15.88:0/1722466483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54a00b6720 0x7f54a00d45b0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f549000b3f0 tx=0x7f549000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:39.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.442+0000 7f549dffb640 1 -- 172.21.15.88:0/1722466483 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5490015020 con 0x7f54a00b6720 2024-09-16T01:40:39.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.442+0000 7f549dffb640 1 -- 172.21.15.88:0/1722466483 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5490011830 con 0x7f54a00b6720 2024-09-16T01:40:39.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.442+0000 7f54a76e6640 1 -- 172.21.15.88:0/1722466483 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54a00c8a70 con 0x7f54a00b6720 2024-09-16T01:40:39.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.442+0000 7f549dffb640 1 -- 172.21.15.88:0/1722466483 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5490011c20 con 0x7f54a00b6720 2024-09-16T01:40:39.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.442+0000 7f54a76e6640 1 -- 172.21.15.88:0/1722466483 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54a00c9030 con 0x7f54a00b6720 2024-09-16T01:40:39.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.443+0000 7f54a76e6640 1 -- 172.21.15.88:0/1722466483 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54a00b61e0 con 0x7f54a00b6720 2024-09-16T01:40:39.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.444+0000 7f549dffb640 1 -- 172.21.15.88:0/1722466483 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5490018020 con 0x7f54a00b6720 2024-09-16T01:40:39.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.444+0000 7f549dffb640 1 --2- 172.21.15.88:0/1722466483 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f548c06c2b0 0x7f548c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:39.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.444+0000 7f549dffb640 1 -- 172.21.15.88:0/1722466483 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6053+0+0 (secure 0 0 0) 0x7f549008b070 con 0x7f54a00b6720 2024-09-16T01:40:39.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.444+0000 7f54a4c81640 1 --2- 172.21.15.88:0/1722466483 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f548c06c2b0 0x7f548c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:39.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.445+0000 7f54a4c81640 1 --2- 172.21.15.88:0/1722466483 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f548c06c2b0 0x7f548c06e770 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f5494034000 tx=0x7f549402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:39.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.448+0000 7f549dffb640 1 -- 172.21.15.88:0/1722466483 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f549001f030 con 0x7f54a00b6720 2024-09-16T01:40:39.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:39.651+0000 7f54a76e6640 1 -- 172.21.15.88:0/1722466483 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f54a00d2a80 con 0x7f54a00b6720 2024-09-16T01:40:40.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.372+0000 7f549dffb640 1 -- 172.21.15.88:0/1722466483 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v92) ==== 161+0+0 (secure 0 0 0) 0x7f54900590c0 con 0x7f54a00b6720 2024-09-16T01:40:40.373 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:40:40.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.375+0000 7f54a76e6640 1 -- 172.21.15.88:0/1722466483 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f548c06c2b0 msgr2=0x7f548c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:40.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.375+0000 7f54a76e6640 1 --2- 172.21.15.88:0/1722466483 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f548c06c2b0 0x7f548c06e770 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f5494034000 tx=0x7f549402e000 comp rx=0 tx=0).stop 2024-09-16T01:40:40.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.375+0000 7f54a76e6640 1 -- 172.21.15.88:0/1722466483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54a00b6720 msgr2=0x7f54a00d45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:40.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.375+0000 7f54a76e6640 1 --2- 172.21.15.88:0/1722466483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54a00b6720 0x7f54a00d45b0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f549000b3f0 tx=0x7f549000b8b0 comp rx=0 tx=0).stop 2024-09-16T01:40:40.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.375+0000 7f54a76e6640 1 -- 172.21.15.88:0/1722466483 shutdown_connections 2024-09-16T01:40:40.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.375+0000 7f54a76e6640 1 --2- 172.21.15.88:0/1722466483 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f548c06c2b0 0x7f548c06e770 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:40.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.375+0000 7f54a76e6640 1 --2- 172.21.15.88:0/1722466483 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f54a015e7a0 0x7f54a00ce8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:40.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.375+0000 7f54a76e6640 1 --2- 172.21.15.88:0/1722466483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54a00cf1f0 0x7f54a01f48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:40.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.375+0000 7f54a76e6640 1 --2- 172.21.15.88:0/1722466483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54a00b6720 0x7f54a00d45b0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:40.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.375+0000 7f54a76e6640 1 -- 172.21.15.88:0/1722466483 >> 172.21.15.88:0/1722466483 conn(0x7f54a00d11e0 msgr2=0x7f54a00d15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:40.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.375+0000 7f54a76e6640 1 -- 172.21.15.88:0/1722466483 shutdown_connections 2024-09-16T01:40:40.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.375+0000 7f54a76e6640 1 -- 172.21.15.88:0/1722466483 wait complete. 2024-09-16T01:40:40.389 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:40:40.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.490+0000 7f7190731640 1 Processor -- start 2024-09-16T01:40:40.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.490+0000 7f7190731640 1 -- start start 2024-09-16T01:40:40.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f7190731640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f718815e7a0 0x7f718806caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:40.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f7190731640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71880b68f0 0x7f71880d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:40.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f7190731640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71880cf860 con 0x7f71880cf1f0 2024-09-16T01:40:40.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f7190731640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f71880d2ca0 con 0x7f718815e7a0 2024-09-16T01:40:40.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f7190731640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f71880cfb40 con 0x7f71880b68f0 2024-09-16T01:40:40.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f718dccc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f71880cf1f0 0x7f71880b61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59626/0 (socket says 172.21.15.88:59626) 2024-09-16T01:40:40.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f718dccc640 1 -- 172.21.15.88:0/4025198326 learned_addr learned my addr 172.21.15.88:0/4025198326 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:40.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f718d4cb640 1 -- 172.21.15.88:0/4025198326 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4233393145 0 0) 0x7f71880cf860 con 0x7f71880cf1f0 2024-09-16T01:40:40.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f718d4cb640 1 -- 172.21.15.88:0/4025198326 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7174003520 con 0x7f71880cf1f0 2024-09-16T01:40:40.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f718e4cd640 1 --2- 172.21.15.88:0/4025198326 >> v2:172.21.15.154:3300/0 conn(0x7f718815e7a0 0x7f718806caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:40.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f718ecce640 1 --2- 172.21.15.88:0/4025198326 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71880b68f0 0x7f71880d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:40.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f718d4cb640 1 -- 172.21.15.88:0/4025198326 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3071396381 0 0) 0x7f7174003520 con 0x7f71880cf1f0 2024-09-16T01:40:40.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f718d4cb640 1 -- 172.21.15.88:0/4025198326 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71880b68f0 msgr2=0x7f71880d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:40.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f718d4cb640 1 --2- 172.21.15.88:0/4025198326 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71880b68f0 0x7f71880d3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:40.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f718d4cb640 1 -- 172.21.15.88:0/4025198326 >> v2:172.21.15.154:3300/0 conn(0x7f718815e7a0 msgr2=0x7f718806caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:40.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f718d4cb640 1 --2- 172.21.15.88:0/4025198326 >> v2:172.21.15.154:3300/0 conn(0x7f718815e7a0 0x7f718806caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:40.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f718d4cb640 1 -- 172.21.15.88:0/4025198326 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71880cfe20 con 0x7f71880cf1f0 2024-09-16T01:40:40.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f718d4cb640 1 -- 172.21.15.88:0/4025198326 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f7178001670 con 0x7f71880cf1f0 2024-09-16T01:40:40.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f718e4cd640 1 --2- 172.21.15.88:0/4025198326 >> v2:172.21.15.154:3300/0 conn(0x7f718815e7a0 0x7f718806caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:40.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f718d4cb640 1 -- 172.21.15.88:0/4025198326 >> v1:172.21.15.88:6789/0 conn(0x7f71880cf1f0 legacy=0x7f71880b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:40.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f718d4cb640 1 --2- 172.21.15.88:0/4025198326 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71880b68f0 0x7f71740039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:40.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f718d4cb640 1 --2- 172.21.15.88:0/4025198326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7174003fe0 0x7f7174004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:40.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f718d4cb640 1 --2- 172.21.15.88:0/4025198326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7174004d40 0x7f71740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:40.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f718d4cb640 1 -- 172.21.15.88:0/4025198326 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f71880d2ca0 con 0x7f7174003fe0 2024-09-16T01:40:40.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f718d4cb640 1 -- 172.21.15.88:0/4025198326 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f71880cfb40 con 0x7f7174004d40 2024-09-16T01:40:40.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.491+0000 7f718d4cb640 1 -- 172.21.15.88:0/4025198326 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f717400bb80 con 0x7f71880b68f0 2024-09-16T01:40:40.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.492+0000 7f718dccc640 1 --2- 172.21.15.88:0/4025198326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7174003fe0 0x7f7174004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:40.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.492+0000 7f718ecce640 1 --2- 172.21.15.88:0/4025198326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7174004d40 0x7f71740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:40.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.492+0000 7f718dccc640 1 -- 172.21.15.88:0/4025198326 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71880b68f0 msgr2=0x7f71740039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:40.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.492+0000 7f718dccc640 1 --2- 172.21.15.88:0/4025198326 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71880b68f0 0x7f71740039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:40.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.492+0000 7f718dccc640 1 -- 172.21.15.88:0/4025198326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7174004d40 msgr2=0x7f71740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:40.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.492+0000 7f718dccc640 1 --2- 172.21.15.88:0/4025198326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7174004d40 0x7f71740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:40.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.492+0000 7f718dccc640 1 -- 172.21.15.88:0/4025198326 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7174003520 con 0x7f7174003fe0 2024-09-16T01:40:40.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.493+0000 7f718dccc640 1 --2- 172.21.15.88:0/4025198326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7174003fe0 0x7f7174004800 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f7178001050 tx=0x7f7174005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b905ab936293f2d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:40.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.493+0000 7f718d4cb640 1 -- 172.21.15.88:0/4025198326 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f717802e070 con 0x7f7174003fe0 2024-09-16T01:40:40.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.493+0000 7f718d4cb640 1 -- 172.21.15.88:0/4025198326 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7178005af0 con 0x7f7174003fe0 2024-09-16T01:40:40.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.493+0000 7f718d4cb640 1 -- 172.21.15.88:0/4025198326 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7178007d70 con 0x7f7174003fe0 2024-09-16T01:40:40.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.493+0000 7f7190731640 1 -- 172.21.15.88:0/4025198326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7174003fe0 msgr2=0x7f7174004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:40.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.493+0000 7f7190731640 1 --2- 172.21.15.88:0/4025198326 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7174003fe0 0x7f7174004800 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f7178001050 tx=0x7f7174005830 comp rx=0 tx=0).stop 2024-09-16T01:40:40.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.493+0000 7f718ecce640 1 -- 172.21.15.88:0/4025198326 reap_dead start 2024-09-16T01:40:40.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.493+0000 7f7190731640 1 -- 172.21.15.88:0/4025198326 shutdown_connections 2024-09-16T01:40:40.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.493+0000 7f7190731640 1 -- 172.21.15.88:0/4025198326 >> 172.21.15.88:0/4025198326 conn(0x7f71880d11e0 msgr2=0x7f71880d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:40.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.493+0000 7f7190731640 1 -- 172.21.15.88:0/4025198326 shutdown_connections 2024-09-16T01:40:40.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.493+0000 7f7190731640 1 -- 172.21.15.88:0/4025198326 wait complete. 2024-09-16T01:40:40.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.494+0000 7f7190731640 1 Processor -- start 2024-09-16T01:40:40.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.495+0000 7f7190731640 1 -- start start 2024-09-16T01:40:40.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.495+0000 7f7190731640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7174003fe0 0x7f71880ca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:40.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.495+0000 7f7190731640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71881437e0 0x7f7188154430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:40.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.495+0000 7f7190731640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7188154970 0x7f71881566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:40.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.495+0000 7f7190731640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f718806cfe0 con 0x7f7174003fe0 2024-09-16T01:40:40.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.495+0000 7f7190731640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f71880b7a00 con 0x7f71881437e0 2024-09-16T01:40:40.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.495+0000 7f7190731640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f718815eb80 con 0x7f7188154970 2024-09-16T01:40:40.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.495+0000 7f718e4cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7174003fe0 0x7f71880ca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:40.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.495+0000 7f718e4cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7174003fe0 0x7f71880ca9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55276/0 (socket says 172.21.15.88:55276) 2024-09-16T01:40:40.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.495+0000 7f718e4cd640 1 -- 172.21.15.88:0/1609157664 learned_addr learned my addr 172.21.15.88:0/1609157664 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:40.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.496+0000 7f718dccc640 1 --2- 172.21.15.88:0/1609157664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71881437e0 0x7f7188154430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:40.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.496+0000 7f718e4cd640 1 -- 172.21.15.88:0/1609157664 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7188154970 msgr2=0x7f71881566b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T01:40:40.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.496+0000 7f718e4cd640 1 --2- 172.21.15.88:0/1609157664 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7188154970 0x7f71881566b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:40.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.496+0000 7f718e4cd640 1 -- 172.21.15.88:0/1609157664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71881437e0 msgr2=0x7f7188154430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:40.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.496+0000 7f718e4cd640 1 --2- 172.21.15.88:0/1609157664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71881437e0 0x7f7188154430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:40.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.496+0000 7f718e4cd640 1 -- 172.21.15.88:0/1609157664 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71881573e0 con 0x7f7174003fe0 2024-09-16T01:40:40.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.496+0000 7f718e4cd640 1 --2- 172.21.15.88:0/1609157664 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7174003fe0 0x7f71880ca9b0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f718400e840 tx=0x7f718400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:40.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.496+0000 7f717effd640 1 -- 172.21.15.88:0/1609157664 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718400f040 con 0x7f7174003fe0 2024-09-16T01:40:40.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.496+0000 7f717effd640 1 -- 172.21.15.88:0/1609157664 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7184004230 con 0x7f7174003fe0 2024-09-16T01:40:40.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.496+0000 7f717effd640 1 -- 172.21.15.88:0/1609157664 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7184003610 con 0x7f7174003fe0 2024-09-16T01:40:40.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.496+0000 7f7190731640 1 -- 172.21.15.88:0/1609157664 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71880b4710 con 0x7f7174003fe0 2024-09-16T01:40:40.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.496+0000 7f7190731640 1 -- 172.21.15.88:0/1609157664 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71880b4cd0 con 0x7f7174003fe0 2024-09-16T01:40:40.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.497+0000 7f7190731640 1 -- 172.21.15.88:0/1609157664 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7188154d70 con 0x7f7174003fe0 2024-09-16T01:40:40.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.497+0000 7f717effd640 1 -- 172.21.15.88:0/1609157664 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f71840037b0 con 0x7f7174003fe0 2024-09-16T01:40:40.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.498+0000 7f717effd640 1 --2- 172.21.15.88:0/1609157664 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f716c06c2b0 0x7f716c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:40.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.498+0000 7f718dccc640 1 --2- 172.21.15.88:0/1609157664 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f716c06c2b0 0x7f716c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:40.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.498+0000 7f717effd640 1 -- 172.21.15.88:0/1609157664 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6053+0+0 (secure 0 0 0) 0x7f718408b1d0 con 0x7f7174003fe0 2024-09-16T01:40:40.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.498+0000 7f718dccc640 1 --2- 172.21.15.88:0/1609157664 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f716c06c2b0 0x7f716c06e770 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f7178033000 tx=0x7f7178009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:40.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.502+0000 7f717effd640 1 -- 172.21.15.88:0/1609157664 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7184017020 con 0x7f7174003fe0 2024-09-16T01:40:40.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:40.735+0000 7f7190731640 1 -- 172.21.15.88:0/1609157664 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f718806caa0 con 0x7f7174003fe0 2024-09-16T01:40:41.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.385+0000 7f717effd640 1 -- 172.21.15.88:0/1609157664 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v93) ==== 101+0+0 (secure 0 0 0) 0x7f7184059870 con 0x7f7174003fe0 2024-09-16T01:40:41.386 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:40:41.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.388+0000 7f7190731640 1 -- 172.21.15.88:0/1609157664 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f716c06c2b0 msgr2=0x7f716c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:41.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.388+0000 7f7190731640 1 --2- 172.21.15.88:0/1609157664 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f716c06c2b0 0x7f716c06e770 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f7178033000 tx=0x7f7178009030 comp rx=0 tx=0).stop 2024-09-16T01:40:41.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.388+0000 7f7190731640 1 -- 172.21.15.88:0/1609157664 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7174003fe0 msgr2=0x7f71880ca9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:41.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.388+0000 7f7190731640 1 --2- 172.21.15.88:0/1609157664 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7174003fe0 0x7f71880ca9b0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f718400e840 tx=0x7f718400ed00 comp rx=0 tx=0).stop 2024-09-16T01:40:41.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.388+0000 7f7190731640 1 -- 172.21.15.88:0/1609157664 shutdown_connections 2024-09-16T01:40:41.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.388+0000 7f7190731640 1 --2- 172.21.15.88:0/1609157664 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f716c06c2b0 0x7f716c06e770 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.388+0000 7f7190731640 1 --2- 172.21.15.88:0/1609157664 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7188154970 0x7f71881566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.388+0000 7f7190731640 1 --2- 172.21.15.88:0/1609157664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71881437e0 0x7f7188154430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.388+0000 7f7190731640 1 --2- 172.21.15.88:0/1609157664 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7174003fe0 0x7f71880ca9b0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.388+0000 7f7190731640 1 -- 172.21.15.88:0/1609157664 >> 172.21.15.88:0/1609157664 conn(0x7f71880d11e0 msgr2=0x7f718806c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:41.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.388+0000 7f7190731640 1 -- 172.21.15.88:0/1609157664 shutdown_connections 2024-09-16T01:40:41.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.388+0000 7f7190731640 1 -- 172.21.15.88:0/1609157664 wait complete. 2024-09-16T01:40:41.402 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:40:41.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.446+0000 7fb93974cf40 1 Processor -- start 2024-09-16T01:40:41.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.446+0000 7fb93974cf40 1 -- start start 2024-09-16T01:40:41.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.447+0000 7fb93974cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560fd4ff5390 0x560fd4ff4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.447+0000 7fb93974cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560fd4ff77c0 0x560fd4feafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.447+0000 7fb93974cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560fd4ff82b0 0x560fd4ff3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.447+0000 7fb93974cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x560fd4e71510 con 0x560fd4ff77c0 2024-09-16T01:40:41.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.447+0000 7fb93974cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560fd4ecac50 con 0x560fd4ff5390 2024-09-16T01:40:41.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.447+0000 7fb93974cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x560fd4ff9230 con 0x560fd4ff82b0 2024-09-16T01:40:41.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.447+0000 7fb92ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560fd4ff77c0 0x560fd4feafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.447+0000 7fb92ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560fd4ff77c0 0x560fd4feafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55292/0 (socket says 172.21.15.88:55292) 2024-09-16T01:40:41.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.447+0000 7fb92ffff640 1 -- 172.21.15.88:0/853738117 learned_addr learned my addr 172.21.15.88:0/853738117 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:41.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.447+0000 7fb937242640 1 --2- 172.21.15.88:0/853738117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560fd4ff5390 0x560fd4ff4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.447+0000 7fb937a43640 1 --2- 172.21.15.88:0/853738117 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560fd4ff82b0 0x560fd4ff3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.448+0000 7fb92ffff640 1 -- 172.21.15.88:0/853738117 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560fd4ff82b0 msgr2=0x560fd4ff3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:41.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.448+0000 7fb92ffff640 1 --2- 172.21.15.88:0/853738117 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560fd4ff82b0 0x560fd4ff3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.448+0000 7fb92ffff640 1 -- 172.21.15.88:0/853738117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560fd4ff5390 msgr2=0x560fd4ff4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:41.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.448+0000 7fb92ffff640 1 --2- 172.21.15.88:0/853738117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560fd4ff5390 0x560fd4ff4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.448+0000 7fb92ffff640 1 -- 172.21.15.88:0/853738117 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fd4f69dd0 con 0x560fd4ff77c0 2024-09-16T01:40:41.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.448+0000 7fb92ffff640 1 --2- 172.21.15.88:0/853738117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560fd4ff77c0 0x560fd4feafb0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fb920002de0 tx=0x7fb92000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7a9bcbeb2a9b9eb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:41.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.448+0000 7fb936240640 1 -- 172.21.15.88:0/853738117 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb92000f400 con 0x560fd4ff77c0 2024-09-16T01:40:41.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.448+0000 7fb936240640 1 -- 172.21.15.88:0/853738117 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb920014070 con 0x560fd4ff77c0 2024-09-16T01:40:41.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.448+0000 7fb936240640 1 -- 172.21.15.88:0/853738117 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb92000f7e0 con 0x560fd4ff77c0 2024-09-16T01:40:41.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.448+0000 7fb93974cf40 1 -- 172.21.15.88:0/853738117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560fd4ff77c0 msgr2=0x560fd4feafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:41.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.448+0000 7fb93974cf40 1 --2- 172.21.15.88:0/853738117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560fd4ff77c0 0x560fd4feafb0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fb920002de0 tx=0x7fb92000b5b0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.448+0000 7fb93974cf40 1 -- 172.21.15.88:0/853738117 shutdown_connections 2024-09-16T01:40:41.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.448+0000 7fb93974cf40 1 --2- 172.21.15.88:0/853738117 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560fd4ff82b0 0x560fd4ff3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.448+0000 7fb93974cf40 1 --2- 172.21.15.88:0/853738117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560fd4ff77c0 0x560fd4feafb0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.448+0000 7fb93974cf40 1 --2- 172.21.15.88:0/853738117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560fd4ff5390 0x560fd4ff4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.448+0000 7fb93974cf40 1 -- 172.21.15.88:0/853738117 >> 172.21.15.88:0/853738117 conn(0x560fd4f4a680 msgr2=0x560fd4ff7ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:41.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.448+0000 7fb93974cf40 1 -- 172.21.15.88:0/853738117 shutdown_connections 2024-09-16T01:40:41.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.448+0000 7fb93974cf40 1 -- 172.21.15.88:0/853738117 wait complete. 2024-09-16T01:40:41.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.449+0000 7fb93974cf40 1 Processor -- start 2024-09-16T01:40:41.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.449+0000 7fb93974cf40 1 -- start start 2024-09-16T01:40:41.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.450+0000 7fb93974cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560fd4ff5390 0x560fd5085e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.450+0000 7fb93974cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560fd4ff77c0 0x560fd4fe95f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.450+0000 7fb93974cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560fd4ff82b0 0x560fd4fe9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.450+0000 7fb93974cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x560fd4f6a0b0 con 0x560fd4ff82b0 2024-09-16T01:40:41.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.450+0000 7fb93974cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560fd4ff0940 con 0x560fd4ff5390 2024-09-16T01:40:41.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.450+0000 7fb93974cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x560fd4ff05d0 con 0x560fd4ff77c0 2024-09-16T01:40:41.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.450+0000 7fb937a43640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560fd4ff82b0 0x560fd4fe9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.450+0000 7fb937a43640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560fd4ff82b0 0x560fd4fe9b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55296/0 (socket says 172.21.15.88:55296) 2024-09-16T01:40:41.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.450+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 learned_addr learned my addr 172.21.15.88:0/3075913932 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:41.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.450+0000 7fb937242640 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560fd4ff5390 0x560fd5085e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.450+0000 7fb92ffff640 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560fd4ff77c0 0x560fd4fe95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.450+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560fd4ff77c0 msgr2=0x560fd4fe95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:41.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.450+0000 7fb937a43640 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560fd4ff77c0 0x560fd4fe95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.450+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560fd4ff5390 msgr2=0x560fd5085e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:41.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.450+0000 7fb937a43640 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560fd4ff5390 0x560fd5085e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.450+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fd4ff4210 con 0x560fd4ff82b0 2024-09-16T01:40:41.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.450+0000 7fb937a43640 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560fd4ff82b0 0x560fd4fe9b30 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fb930025b00 tx=0x7fb930003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:41.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.450+0000 7fb93523e640 1 -- 172.21.15.88:0/3075913932 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb930056cb0 con 0x560fd4ff82b0 2024-09-16T01:40:41.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.450+0000 7fb93523e640 1 -- 172.21.15.88:0/3075913932 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb93006a410 con 0x560fd4ff82b0 2024-09-16T01:40:41.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.450+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x560fd4ff44f0 con 0x560fd4ff82b0 2024-09-16T01:40:41.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.450+0000 7fb93523e640 1 -- 172.21.15.88:0/3075913932 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93006a800 con 0x560fd4ff82b0 2024-09-16T01:40:41.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.451+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fd5088c60 con 0x560fd4ff82b0 2024-09-16T01:40:41.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.452+0000 7fb93523e640 1 -- 172.21.15.88:0/3075913932 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb930068020 con 0x560fd4ff82b0 2024-09-16T01:40:41.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.452+0000 7fb93523e640 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb91806ac90 0x7fb91806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.452+0000 7fb93523e640 1 -- 172.21.15.88:0/3075913932 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6053+0+0 (secure 0 0 0) 0x7fb9300cea70 con 0x560fd4ff82b0 2024-09-16T01:40:41.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.452+0000 7fb937242640 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb91806ac90 0x7fb91806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.458 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:40:41.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.453+0000 7fb93974cf40 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x560fd50863a0 0x560fd4f67050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.453+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd4fe7d30 con 0x560fd50863a0 2024-09-16T01:40:41.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.453+0000 7fb92ffff640 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x560fd50863a0 0x560fd4f67050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.453+0000 7fb937242640 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb91806ac90 0x7fb91806d150 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x560fd4feb430 tx=0x7fb9280018b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:41.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.453+0000 7fb92ffff640 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x560fd50863a0 0x560fd4f67050 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:41.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.454+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v89'102 uv78 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.455+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e89) -- 0x560fd4f67590 con 0x560fd50863a0 2024-09-16T01:40:41.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.456+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v89'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.456+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd4f67960 con 0x560fd50863a0 2024-09-16T01:40:41.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.457+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v89'104 uv104 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.460 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:40:41.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.457+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd4f67d00 con 0x560fd50863a0 2024-09-16T01:40:41.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.458+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v89'105 uv105 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.458+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd4f68100 con 0x560fd50863a0 2024-09-16T01:40:41.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.459+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v89'106 uv106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.459+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509a6a0 con 0x560fd50863a0 2024-09-16T01:40:41.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.460+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v89'107 uv107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.460+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509aa40 con 0x560fd50863a0 2024-09-16T01:40:41.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.461+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v89'108 uv84 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.461+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e89) -- 0x560fd509ade0 con 0x560fd50863a0 2024-09-16T01:40:41.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.462+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v89'109 uv109 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.462+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509ade0 con 0x560fd50863a0 2024-09-16T01:40:41.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.463+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v89'110 uv110 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.464 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:40:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.463+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509bdf0 con 0x560fd50863a0 2024-09-16T01:40:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.464+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v89'111 uv111 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.464+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509bdf0 con 0x560fd50863a0 2024-09-16T01:40:41.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.465+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v89'112 uv112 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.465+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509ade0 con 0x560fd50863a0 2024-09-16T01:40:41.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.466+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v89'113 uv113 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.466+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509ade0 con 0x560fd50863a0 2024-09-16T01:40:41.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.467+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v89'114 uv91 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.467+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e89) -- 0x560fd509ade0 con 0x560fd50863a0 2024-09-16T01:40:41.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.468+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v89'115 uv115 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.468+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509ade0 con 0x560fd50863a0 2024-09-16T01:40:41.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.469+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v89'116 uv116 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.471 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:40:41.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.469+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509bdf0 con 0x560fd50863a0 2024-09-16T01:40:41.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.470+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v89'117 uv117 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.470+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509bdf0 con 0x560fd50863a0 2024-09-16T01:40:41.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.471+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v89'118 uv118 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.471+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509ade0 con 0x560fd50863a0 2024-09-16T01:40:41.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.472+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v89'119 uv119 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.473+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509b290 con 0x560fd50863a0 2024-09-16T01:40:41.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.474+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v89'120 uv120 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.474+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509b290 con 0x560fd50863a0 2024-09-16T01:40:41.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.475+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v89'121 uv95 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.475+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e89) -- 0x560fd509b290 con 0x560fd50863a0 2024-09-16T01:40:41.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.476+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v89'122 uv122 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.476+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509b290 con 0x560fd50863a0 2024-09-16T01:40:41.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.477+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v89'123 uv123 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.478 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:40:41.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.477+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509c2a0 con 0x560fd50863a0 2024-09-16T01:40:41.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.478+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v89'124 uv124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.478+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509c2a0 con 0x560fd50863a0 2024-09-16T01:40:41.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.479+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v89'125 uv125 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.479+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509b290 con 0x560fd50863a0 2024-09-16T01:40:41.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.481+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v89'126 uv126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.481+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509b290 con 0x560fd50863a0 2024-09-16T01:40:41.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.482+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v89'127 uv101 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.482+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e89) -- 0x560fd509b290 con 0x560fd50863a0 2024-09-16T01:40:41.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.483+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v89'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.483+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509b290 con 0x560fd50863a0 2024-09-16T01:40:41.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.484+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v89'129 uv129 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.485 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:40:41.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.484+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509c2a0 con 0x560fd50863a0 2024-09-16T01:40:41.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.485+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v89'130 uv130 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.485+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509c2a0 con 0x560fd50863a0 2024-09-16T01:40:41.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.486+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v89'131 uv131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.486+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509b290 con 0x560fd50863a0 2024-09-16T01:40:41.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.487+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v89'132 uv132 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.487+0000 7fb93974cf40 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x560fd509b7a0 0x560fd509db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.487+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509e0d0 con 0x560fd509b7a0 2024-09-16T01:40:41.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.487+0000 7fb937242640 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x560fd509b7a0 0x560fd509db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.488+0000 7fb937242640 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x560fd509b7a0 0x560fd509db90 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:41.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.489+0000 7fb937242640 1 -- 172.21.15.88:0/3075913932 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v89'90 uv90 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fb92800e980 con 0x560fd509b7a0 2024-09-16T01:40:41.490 INFO:teuthology.orchestra.run.smithi088.stdout:Write #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:40:41.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.489+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509e920 con 0x560fd509b7a0 2024-09-16T01:40:41.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.490+0000 7fb937242640 1 -- 172.21.15.88:0/3075913932 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [write 0~582] v89'91 uv91 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fb92800e980 con 0x560fd509b7a0 2024-09-16T01:40:41.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.490+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509ecc0 con 0x560fd509b7a0 2024-09-16T01:40:41.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.493+0000 7fb937242640 1 -- 172.21.15.88:0/3075913932 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [omap-set-header] v89'92 uv92 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fb92800e980 con 0x560fd509b7a0 2024-09-16T01:40:41.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.493+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509f060 con 0x560fd509b7a0 2024-09-16T01:40:41.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.496+0000 7fb937242640 1 -- 172.21.15.88:0/3075913932 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 200.00000000 [create] v89'93 uv77 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb92800e980 con 0x560fd509b7a0 2024-09-16T01:40:41.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.496+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e89) -- 0x560fd509f400 con 0x560fd509b7a0 2024-09-16T01:40:41.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.499+0000 7fb937242640 1 -- 172.21.15.88:0/3075913932 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 200.00000000 [delete] v89'94 uv94 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92800e980 con 0x560fd509b7a0 2024-09-16T01:40:41.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.499+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509f7a0 con 0x560fd509b7a0 2024-09-16T01:40:41.502 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:40:41.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.501+0000 7fb937242640 1 -- 172.21.15.88:0/3075913932 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v89'95 uv95 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92800e980 con 0x560fd509b7a0 2024-09-16T01:40:41.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.501+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509fb40 con 0x560fd509b7a0 2024-09-16T01:40:41.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.504+0000 7fb937242640 1 -- 172.21.15.88:0/3075913932 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [write 0~90] v89'96 uv96 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92800e980 con 0x560fd509b7a0 2024-09-16T01:40:41.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.504+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509fee0 con 0x560fd509b7a0 2024-09-16T01:40:41.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.507+0000 7fb937242640 1 -- 172.21.15.88:0/3075913932 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [omap-set-header] v89'97 uv97 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92800e980 con 0x560fd509b7a0 2024-09-16T01:40:41.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.507+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509fee0 con 0x560fd509b7a0 2024-09-16T01:40:41.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.510+0000 7fb937242640 1 -- 172.21.15.88:0/3075913932 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000001 [create] v89'98 uv82 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb92800e980 con 0x560fd509b7a0 2024-09-16T01:40:41.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.510+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e89) -- 0x560fd509fee0 con 0x560fd509b7a0 2024-09-16T01:40:41.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.512+0000 7fb937242640 1 -- 172.21.15.88:0/3075913932 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000001 [delete] v89'99 uv99 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92800e980 con 0x560fd509b7a0 2024-09-16T01:40:41.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.512+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509fee0 con 0x560fd509b7a0 2024-09-16T01:40:41.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.515+0000 7fb937242640 1 -- 172.21.15.88:0/3075913932 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v89'100 uv100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92800e980 con 0x560fd509b7a0 2024-09-16T01:40:41.516 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:40:41.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.515+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a4360 con 0x560fd509b7a0 2024-09-16T01:40:41.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.518+0000 7fb937242640 1 -- 172.21.15.88:0/3075913932 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [write 0~17490] v89'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92800e980 con 0x560fd509b7a0 2024-09-16T01:40:41.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.518+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a4360 con 0x560fd509b7a0 2024-09-16T01:40:41.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.521+0000 7fb937242640 1 -- 172.21.15.88:0/3075913932 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [omap-set-header] v89'102 uv102 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92800e980 con 0x560fd509b7a0 2024-09-16T01:40:41.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.521+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509fee0 con 0x560fd509b7a0 2024-09-16T01:40:41.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.522+0000 7fb937242640 1 -- 172.21.15.88:0/3075913932 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 606.00000000 [create] v89'103 uv88 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb92800e980 con 0x560fd509b7a0 2024-09-16T01:40:41.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.522+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e89) -- 0x560fd509fee0 con 0x560fd509b7a0 2024-09-16T01:40:41.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.523+0000 7fb937242640 1 -- 172.21.15.88:0/3075913932 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 606.00000000 [delete] v89'104 uv104 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92800e980 con 0x560fd509b7a0 2024-09-16T01:40:41.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.523+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509fee0 con 0x560fd509b7a0 2024-09-16T01:40:41.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.524+0000 7fb937242640 1 -- 172.21.15.88:0/3075913932 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v89'105 uv105 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92800e980 con 0x560fd509b7a0 2024-09-16T01:40:41.526 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:40:41.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.524+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a0ef0 con 0x560fd509b7a0 2024-09-16T01:40:41.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.526+0000 7fb937242640 1 -- 172.21.15.88:0/3075913932 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [setxattr (30)] v89'106 uv106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92800e980 con 0x560fd509b7a0 2024-09-16T01:40:41.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.526+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a0ef0 con 0x560fd509b7a0 2024-09-16T01:40:41.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.527+0000 7fb937242640 1 -- 172.21.15.88:0/3075913932 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [setxattr (62)] v89'107 uv107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92800e980 con 0x560fd509b7a0 2024-09-16T01:40:41.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.527+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509fee0 con 0x560fd509b7a0 2024-09-16T01:40:41.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.529+0000 7fb937242640 1 -- 172.21.15.88:0/3075913932 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [omap-set-header] v89'108 uv108 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92800e980 con 0x560fd509b7a0 2024-09-16T01:40:41.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.529+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:51 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509fee0 con 0x560fd50863a0 2024-09-16T01:40:41.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.530+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(51 607.00000000 [create] v89'21 uv20 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.530+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:52 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e89) -- 0x560fd509fee0 con 0x560fd50863a0 2024-09-16T01:40:41.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.531+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(52 607.00000000 [delete] v89'22 uv22 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.531+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509fee0 con 0x560fd50863a0 2024-09-16T01:40:41.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.534+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(53 607.00000000 [create] v89'23 uv23 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.535 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:40:41.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.534+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a0ef0 con 0x560fd50863a0 2024-09-16T01:40:41.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.536+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(54 607.00000000 [setxattr (30)] v89'24 uv24 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.537+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a0ef0 con 0x560fd50863a0 2024-09-16T01:40:41.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.539+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(55 607.00000000 [setxattr (62)] v89'25 uv25 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.539+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd509fee0 con 0x560fd50863a0 2024-09-16T01:40:41.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.540+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(56 607.00000000 [omap-set-header] v89'26 uv26 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.540+0000 7fb93974cf40 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x560fd50a0560 0x560fd50a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.540+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:57 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a32f0 con 0x560fd50a0560 2024-09-16T01:40:41.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.540+0000 7fb92ffff640 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x560fd50a0560 0x560fd50a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.541+0000 7fb92ffff640 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x560fd50a0560 0x560fd50a0980 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:41.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.543+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(57 mds0_openfiles.0 [create] v89'18 uv17 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50a0560 2024-09-16T01:40:41.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.543+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:58 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a32f0 con 0x560fd50a0560 2024-09-16T01:40:41.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.545+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(58 mds0_openfiles.0 [delete] v89'19 uv19 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50a0560 2024-09-16T01:40:41.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.545+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a32f0 con 0x560fd50a0560 2024-09-16T01:40:41.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.548+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(59 mds0_openfiles.0 [create] v89'20 uv20 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50a0560 2024-09-16T01:40:41.549 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:40:41.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.548+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a36e0 con 0x560fd50a0560 2024-09-16T01:40:41.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.550+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(60 mds0_openfiles.0 [omap-set-header] v89'21 uv21 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50a0560 2024-09-16T01:40:41.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.551+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:61 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a36e0 con 0x560fd50863a0 2024-09-16T01:40:41.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.553+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(61 608.00000000 [create] v89'21 uv20 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.553+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:62 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a36e0 con 0x560fd50863a0 2024-09-16T01:40:41.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.554+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(62 608.00000000 [delete] v89'22 uv22 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.554+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a36e0 con 0x560fd50863a0 2024-09-16T01:40:41.558 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:40:41.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.557+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(63 608.00000000 [create] v89'23 uv23 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.557+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a46f0 con 0x560fd50863a0 2024-09-16T01:40:41.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.560+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(64 608.00000000 [setxattr (30)] v89'24 uv24 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.560+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a46f0 con 0x560fd50863a0 2024-09-16T01:40:41.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.562+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(65 608.00000000 [setxattr (62)] v89'25 uv25 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.562+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a36e0 con 0x560fd50863a0 2024-09-16T01:40:41.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.565+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(66 608.00000000 [omap-set-header] v89'26 uv26 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.565+0000 7fb93974cf40 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x560fd50a3bf0 0x560fd50a5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.565+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:67 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a6520 con 0x560fd50a3bf0 2024-09-16T01:40:41.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.566+0000 7fb937a43640 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x560fd50a3bf0 0x560fd50a5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.566+0000 7fb937a43640 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x560fd50a3bf0 0x560fd50a5fe0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:41.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.567+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(67 604.00000000 [create] v89'37 uv31 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.567+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:68 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a6920 con 0x560fd50a3bf0 2024-09-16T01:40:41.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.568+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(68 604.00000000 [delete] v89'38 uv38 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.568+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a6e00 con 0x560fd50a3bf0 2024-09-16T01:40:41.572 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:40:41.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.571+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(69 604.00000000 [create] v89'39 uv39 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.571+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a86b0 con 0x560fd50a3bf0 2024-09-16T01:40:41.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.574+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(70 604.00000000 [setxattr (30)] v89'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.574+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a8a50 con 0x560fd50a3bf0 2024-09-16T01:40:41.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.575+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(71 604.00000000 [setxattr (62)] v89'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.575+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a76a0 con 0x560fd50a3bf0 2024-09-16T01:40:41.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.576+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(72 604.00000000 [omap-set-header] v89'42 uv42 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.576+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a7a90 con 0x560fd50a3bf0 2024-09-16T01:40:41.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.577+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(73 500.00000000 [create] v89'43 uv36 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.577+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a7a90 con 0x560fd50a3bf0 2024-09-16T01:40:41.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.578+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(74 500.00000000 [delete] v89'44 uv44 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.578+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a7a90 con 0x560fd50a3bf0 2024-09-16T01:40:41.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.579+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(75 500.00000000 [create] v89'45 uv45 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.580 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:40:41.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.579+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a7a90 con 0x560fd50a3bf0 2024-09-16T01:40:41.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.580+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(76 500.00000000 [write 0~90] v89'46 uv46 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.580+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a7a90 con 0x560fd50a3bf0 2024-09-16T01:40:41.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.581+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(77 500.00000000 [omap-set-header] v89'47 uv47 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.581+0000 7fb93974cf40 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x560fd50a8df0 0x560fd50ab1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.581+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:78 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a7a90 con 0x560fd50a8df0 2024-09-16T01:40:41.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.581+0000 7fb937242640 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x560fd50a8df0 0x560fd50ab1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.582+0000 7fb937242640 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x560fd50a8df0 0x560fd50ab1e0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:41.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.584+0000 7fb937242640 1 -- 172.21.15.88:0/3075913932 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(78 mds_snaptable [create] v89'17 uv16 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fb92800e980 con 0x560fd50a8df0 2024-09-16T01:40:41.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.584+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:79 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a7a90 con 0x560fd50a8df0 2024-09-16T01:40:41.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.585+0000 7fb937242640 1 -- 172.21.15.88:0/3075913932 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(79 mds_snaptable [delete] v89'18 uv18 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fb92800e980 con 0x560fd50a8df0 2024-09-16T01:40:41.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.585+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ab7a0 con 0x560fd50a8df0 2024-09-16T01:40:41.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.586+0000 7fb937242640 1 -- 172.21.15.88:0/3075913932 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(80 mds_snaptable [create] v89'19 uv19 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fb92800e980 con 0x560fd50a8df0 2024-09-16T01:40:41.587 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:40:41.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.586+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ab7a0 con 0x560fd50a8df0 2024-09-16T01:40:41.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.587+0000 7fb937242640 1 -- 172.21.15.88:0/3075913932 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(81 mds_snaptable [write 0~70] v89'20 uv20 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fb92800e980 con 0x560fd50a8df0 2024-09-16T01:40:41.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.587+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ab7a0 con 0x560fd50a8df0 2024-09-16T01:40:41.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.588+0000 7fb937242640 1 -- 172.21.15.88:0/3075913932 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(82 mds_snaptable [omap-set-header] v89'21 uv21 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fb92800e980 con 0x560fd50a8df0 2024-09-16T01:40:41.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.588+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:83 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ab7a0 con 0x560fd50863a0 2024-09-16T01:40:41.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.589+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(83 605.00000000 [create] v89'21 uv20 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.589+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:84 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ab7a0 con 0x560fd50863a0 2024-09-16T01:40:41.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.590+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(84 605.00000000 [delete] v89'22 uv22 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.590+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ab7a0 con 0x560fd50863a0 2024-09-16T01:40:41.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.593+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(85 605.00000000 [create] v89'23 uv23 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.594 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:40:41.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.593+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ac7b0 con 0x560fd50863a0 2024-09-16T01:40:41.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.593+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(86 605.00000000 [setxattr (30)] v89'24 uv24 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.594+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ac7b0 con 0x560fd50863a0 2024-09-16T01:40:41.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.596+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(87 605.00000000 [setxattr (62)] v89'25 uv25 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.596+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ab7a0 con 0x560fd50863a0 2024-09-16T01:40:41.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.599+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(88 605.00000000 [omap-set-header] v89'26 uv26 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50863a0 2024-09-16T01:40:41.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.599+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:89 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ab7a0 con 0x560fd50a0560 2024-09-16T01:40:41.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.599+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(89 mds0_inotable [create] v89'34 uv28 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50a0560 2024-09-16T01:40:41.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.599+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:90 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ab7a0 con 0x560fd50a0560 2024-09-16T01:40:41.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.601+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(90 mds0_inotable [delete] v89'35 uv35 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50a0560 2024-09-16T01:40:41.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.601+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ab7a0 con 0x560fd50a0560 2024-09-16T01:40:41.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.602+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(91 mds0_inotable [create] v89'36 uv36 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50a0560 2024-09-16T01:40:41.603 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:40:41.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.602+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a72b0 con 0x560fd50a0560 2024-09-16T01:40:41.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.603+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(92 mds0_inotable [write 0~34] v89'37 uv37 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50a0560 2024-09-16T01:40:41.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.603+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a72b0 con 0x560fd50a0560 2024-09-16T01:40:41.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.604+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(93 mds0_inotable [omap-set-header] v89'38 uv38 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50a0560 2024-09-16T01:40:41.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.604+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a72b0 con 0x560fd50a0560 2024-09-16T01:40:41.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.604+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(94 100.00000000 [create] v89'39 uv33 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50a0560 2024-09-16T01:40:41.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.604+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a72b0 con 0x560fd50a0560 2024-09-16T01:40:41.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.606+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(95 100.00000000 [delete] v89'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50a0560 2024-09-16T01:40:41.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.606+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a72b0 con 0x560fd50a0560 2024-09-16T01:40:41.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.607+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(96 100.00000000 [create] v89'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50a0560 2024-09-16T01:40:41.608 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:40:41.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.607+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50a72b0 con 0x560fd50a0560 2024-09-16T01:40:41.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.608+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(97 100.00000000 [omap-set-header] v89'42 uv42 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50a0560 2024-09-16T01:40:41.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.608+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae630 con 0x560fd50a0560 2024-09-16T01:40:41.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.609+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v89'43 uv43 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50a0560 2024-09-16T01:40:41.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.609+0000 7fb93974cf40 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x560fd50abcb0 0x560fd50aecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.609+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:99 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae0a0 con 0x560fd50abcb0 2024-09-16T01:40:41.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.609+0000 7fb92ffff640 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x560fd50abcb0 0x560fd50aecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.609+0000 7fb92ffff640 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x560fd50abcb0 0x560fd50aecf0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:41.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.610+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(99 mds0_sessionmap [create] v89'18 uv17 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50abcb0 2024-09-16T01:40:41.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.610+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:100 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae0a0 con 0x560fd50abcb0 2024-09-16T01:40:41.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.611+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v89'19 uv19 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50abcb0 2024-09-16T01:40:41.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.611+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae0a0 con 0x560fd50abcb0 2024-09-16T01:40:41.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.612+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(101 mds0_sessionmap [create] v89'20 uv20 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50abcb0 2024-09-16T01:40:41.614 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:40:41.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.613+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae690 con 0x560fd50abcb0 2024-09-16T01:40:41.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.613+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v89'21 uv21 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50abcb0 2024-09-16T01:40:41.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.613+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae690 con 0x560fd50a0560 2024-09-16T01:40:41.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.615+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(103 609.00000000 [create] v89'21 uv20 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50a0560 2024-09-16T01:40:41.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.615+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae670 con 0x560fd50a0560 2024-09-16T01:40:41.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.616+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(104 609.00000000 [delete] v89'22 uv22 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50a0560 2024-09-16T01:40:41.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.616+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae5f0 con 0x560fd50a0560 2024-09-16T01:40:41.618 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:40:41.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.617+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(105 609.00000000 [create] v89'23 uv23 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50a0560 2024-09-16T01:40:41.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.617+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae5f0 con 0x560fd50a0560 2024-09-16T01:40:41.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.618+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v89'24 uv24 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50a0560 2024-09-16T01:40:41.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.618+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae5f0 con 0x560fd50a0560 2024-09-16T01:40:41.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.619+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v89'25 uv25 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50a0560 2024-09-16T01:40:41.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.619+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae5f0 con 0x560fd50a0560 2024-09-16T01:40:41.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.620+0000 7fb92ffff640 1 -- 172.21.15.88:0/3075913932 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(108 609.00000000 [omap-set-header] v89'26 uv26 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb92001a490 con 0x560fd50a0560 2024-09-16T01:40:41.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.620+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae5f0 con 0x560fd50a3bf0 2024-09-16T01:40:41.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.621+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(109 400.00000000 [create] v89'17 uv16 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.621+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae5f0 con 0x560fd50a3bf0 2024-09-16T01:40:41.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.622+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(110 400.00000000 [delete] v89'18 uv18 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.622+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae5f0 con 0x560fd50a3bf0 2024-09-16T01:40:41.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.623+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(111 400.00000000 [create] v89'19 uv19 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.625 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:40:41.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.623+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae5f0 con 0x560fd50a3bf0 2024-09-16T01:40:41.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.625+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v89'20 uv20 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.625+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae5f0 con 0x560fd50a3bf0 2024-09-16T01:40:41.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.626+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v89'21 uv21 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.626+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae5f0 con 0x560fd50a3bf0 2024-09-16T01:40:41.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.627+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(114 100.00000000.inode [create] v89'45 uv37 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.627+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae5f0 con 0x560fd50a3bf0 2024-09-16T01:40:41.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.628+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(115 100.00000000.inode [delete] v89'46 uv46 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.628+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae5f0 con 0x560fd50a3bf0 2024-09-16T01:40:41.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.629+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(116 100.00000000.inode [create] v89'47 uv47 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.631 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:40:41.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.630+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae5f0 con 0x560fd50a3bf0 2024-09-16T01:40:41.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.631+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v89'48 uv48 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.631+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae5f0 con 0x560fd50a3bf0 2024-09-16T01:40:41.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.632+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v89'49 uv49 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.632+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae5f0 con 0x560fd50a3bf0 2024-09-16T01:40:41.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.633+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(119 1.00000000 [create] v89'50 uv44 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.633+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae5f0 con 0x560fd50a3bf0 2024-09-16T01:40:41.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.634+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(120 1.00000000 [delete] v89'51 uv51 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.634+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae5f0 con 0x560fd50a3bf0 2024-09-16T01:40:41.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.637+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(121 1.00000000 [create] v89'52 uv52 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.638 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:40:41.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.637+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae5f0 con 0x560fd50a3bf0 2024-09-16T01:40:41.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.639+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v89'53 uv53 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.639+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae5f0 con 0x560fd50a3bf0 2024-09-16T01:40:41.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.640+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v89'54 uv54 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.640+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50ae5f0 con 0x560fd50a3bf0 2024-09-16T01:40:41.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.643+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v89'55 uv55 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.643+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) -- 0x560fd50af230 con 0x560fd50a3bf0 2024-09-16T01:40:41.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.648+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v89'56 uv56 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fb9300ce040 con 0x560fd50a3bf0 2024-09-16T01:40:41.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.648+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x560fd50863a0 msgr2=0x560fd4f67050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:41.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.648+0000 7fb93974cf40 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x560fd50863a0 0x560fd4f67050 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.648+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x560fd50a0560 msgr2=0x560fd50a0980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:41.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.648+0000 7fb93974cf40 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x560fd50a0560 0x560fd50a0980 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.648+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x560fd50abcb0 msgr2=0x560fd50aecf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:41.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.648+0000 7fb93974cf40 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x560fd50abcb0 0x560fd50aecf0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.648+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x560fd509b7a0 msgr2=0x560fd509db90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:41.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.648+0000 7fb93974cf40 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x560fd509b7a0 0x560fd509db90 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.648+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x560fd50a3bf0 msgr2=0x560fd50a5fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:41.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.648+0000 7fb93974cf40 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x560fd50a3bf0 0x560fd50a5fe0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.648+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x560fd50a8df0 msgr2=0x560fd50ab1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:41.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.648+0000 7fb93974cf40 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x560fd50a8df0 0x560fd50ab1e0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.648+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 reap_dead start 2024-09-16T01:40:41.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.648+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 reap_dead start 2024-09-16T01:40:41.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.648+0000 7fb937a43640 1 -- 172.21.15.88:0/3075913932 reap_dead start 2024-09-16T01:40:41.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.648+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb91806ac90 msgr2=0x7fb91806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:41.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.648+0000 7fb93974cf40 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb91806ac90 0x7fb91806d150 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x560fd4feb430 tx=0x7fb9280018b0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.648+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560fd4ff82b0 msgr2=0x560fd4fe9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:41.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.648+0000 7fb93974cf40 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560fd4ff82b0 0x560fd4fe9b30 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fb930025b00 tx=0x7fb930003330 comp rx=0 tx=0).stop 2024-09-16T01:40:41.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.649+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 shutdown_connections 2024-09-16T01:40:41.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.649+0000 7fb93974cf40 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb91806ac90 0x7fb91806d150 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.649+0000 7fb93974cf40 1 --2- 172.21.15.88:0/3075913932 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560fd4ff82b0 0x560fd4fe9b30 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.649+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 >> 172.21.15.88:0/3075913932 conn(0x560fd4f4a680 msgr2=0x560fd4ff7ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:41.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.649+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 shutdown_connections 2024-09-16T01:40:41.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.649+0000 7fb93974cf40 1 -- 172.21.15.88:0/3075913932 wait complete. 2024-09-16T01:40:41.653 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 200.00000000 2024-09-16T01:40:41.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.734+0000 7f3613970f40 1 Processor -- start 2024-09-16T01:40:41.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.734+0000 7f3613970f40 1 -- start start 2024-09-16T01:40:41.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.734+0000 7f3613970f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bfe80a8510 0x55bfe81cab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.734+0000 7f3613970f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfe80a9820 0x55bfe81c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.734+0000 7f3613970f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfe811c1b0 0x55bfe80aced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.734+0000 7f3613970f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55bfe811c590 con 0x55bfe80a9820 2024-09-16T01:40:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.734+0000 7f3613970f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bfe7fb3510 con 0x55bfe811c1b0 2024-09-16T01:40:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.734+0000 7f3613970f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55bfe80a7550 con 0x55bfe80a8510 2024-09-16T01:40:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.735+0000 7f3608c65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfe80a9820 0x55bfe81c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.735+0000 7f3608c65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfe80a9820 0x55bfe81c7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55304/0 (socket says 172.21.15.88:55304) 2024-09-16T01:40:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.735+0000 7f3608c65640 1 -- 172.21.15.88:0/1314668177 learned_addr learned my addr 172.21.15.88:0/1314668177 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.735+0000 7f3608c65640 1 -- 172.21.15.88:0/1314668177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bfe80a8510 msgr2=0x55bfe81cab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.735+0000 7f3611466640 1 --2- 172.21.15.88:0/1314668177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bfe80a8510 0x55bfe81cab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.735+0000 7f3611c67640 1 --2- 172.21.15.88:0/1314668177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfe811c1b0 0x55bfe80aced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.735+0000 7f3608c65640 1 --2- 172.21.15.88:0/1314668177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bfe80a8510 0x55bfe81cab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.735+0000 7f3608c65640 1 -- 172.21.15.88:0/1314668177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfe811c1b0 msgr2=0x55bfe80aced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.735+0000 7f3608c65640 1 --2- 172.21.15.88:0/1314668177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfe811c1b0 0x55bfe80aced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.735+0000 7f3608c65640 1 -- 172.21.15.88:0/1314668177 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfe813a3b0 con 0x55bfe80a9820 2024-09-16T01:40:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.735+0000 7f3608c65640 1 --2- 172.21.15.88:0/1314668177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfe80a9820 0x55bfe81c7b80 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00b240 tx=0x7f35fc00b700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb9e534a406600ad server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.735+0000 7f360bfff640 1 -- 172.21.15.88:0/1314668177 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc008070 con 0x55bfe80a9820 2024-09-16T01:40:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.735+0000 7f360bfff640 1 -- 172.21.15.88:0/1314668177 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35fc00d500 con 0x55bfe80a9820 2024-09-16T01:40:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.735+0000 7f360bfff640 1 -- 172.21.15.88:0/1314668177 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc00d910 con 0x55bfe80a9820 2024-09-16T01:40:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.736+0000 7f3613970f40 1 -- 172.21.15.88:0/1314668177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfe80a9820 msgr2=0x55bfe81c7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.736+0000 7f3613970f40 1 --2- 172.21.15.88:0/1314668177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfe80a9820 0x55bfe81c7b80 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00b240 tx=0x7f35fc00b700 comp rx=0 tx=0).stop 2024-09-16T01:40:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.736+0000 7f3613970f40 1 -- 172.21.15.88:0/1314668177 shutdown_connections 2024-09-16T01:40:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.736+0000 7f3613970f40 1 --2- 172.21.15.88:0/1314668177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfe811c1b0 0x55bfe80aced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.736+0000 7f3613970f40 1 --2- 172.21.15.88:0/1314668177 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfe80a9820 0x55bfe81c7b80 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.736+0000 7f3613970f40 1 --2- 172.21.15.88:0/1314668177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bfe80a8510 0x55bfe81cab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.736+0000 7f3613970f40 1 -- 172.21.15.88:0/1314668177 >> 172.21.15.88:0/1314668177 conn(0x55bfe8060be0 msgr2=0x55bfe8138210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.736+0000 7f3613970f40 1 -- 172.21.15.88:0/1314668177 shutdown_connections 2024-09-16T01:40:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.736+0000 7f3613970f40 1 -- 172.21.15.88:0/1314668177 wait complete. 2024-09-16T01:40:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.736+0000 7f3613970f40 1 Processor -- start 2024-09-16T01:40:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.736+0000 7f3613970f40 1 -- start start 2024-09-16T01:40:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.737+0000 7f3613970f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bfe80a8510 0x55bfe8132a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.737+0000 7f3613970f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfe80a9820 0x55bfe8112250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.737+0000 7f3613970f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfe811c1b0 0x55bfe8112790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.737+0000 7f3613970f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55bfe813a970 con 0x55bfe80a9820 2024-09-16T01:40:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.737+0000 7f3613970f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bfe813a690 con 0x55bfe811c1b0 2024-09-16T01:40:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.737+0000 7f3613970f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55bfe813ac50 con 0x55bfe80a8510 2024-09-16T01:40:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.737+0000 7f3608c65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfe80a9820 0x55bfe8112250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.737+0000 7f3608c65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfe80a9820 0x55bfe8112250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55306/0 (socket says 172.21.15.88:55306) 2024-09-16T01:40:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.737+0000 7f3608c65640 1 -- 172.21.15.88:0/2517719962 learned_addr learned my addr 172.21.15.88:0/2517719962 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.737+0000 7f3611466640 1 --2- 172.21.15.88:0/2517719962 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bfe80a8510 0x55bfe8132a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.737+0000 7f3611c67640 1 --2- 172.21.15.88:0/2517719962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfe811c1b0 0x55bfe8112790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.737+0000 7f3608c65640 1 -- 172.21.15.88:0/2517719962 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bfe80a8510 msgr2=0x55bfe8132a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.737+0000 7f3608c65640 1 --2- 172.21.15.88:0/2517719962 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bfe80a8510 0x55bfe8132a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.737+0000 7f3608c65640 1 -- 172.21.15.88:0/2517719962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfe811c1b0 msgr2=0x55bfe8112790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.737+0000 7f3608c65640 1 --2- 172.21.15.88:0/2517719962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfe811c1b0 0x55bfe8112790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.737+0000 7f3608c65640 1 -- 172.21.15.88:0/2517719962 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfe8118b60 con 0x55bfe80a9820 2024-09-16T01:40:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.738+0000 7f3608c65640 1 --2- 172.21.15.88:0/2517719962 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfe80a9820 0x55bfe8112250 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00c010 tx=0x7f35fc000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.738+0000 7f360affd640 1 -- 172.21.15.88:0/2517719962 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc019740 con 0x55bfe80a9820 2024-09-16T01:40:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.738+0000 7f360affd640 1 -- 172.21.15.88:0/2517719962 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35fc0198e0 con 0x55bfe80a9820 2024-09-16T01:40:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.738+0000 7f3613970f40 1 -- 172.21.15.88:0/2517719962 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55bfe8117d60 con 0x55bfe80a9820 2024-09-16T01:40:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.738+0000 7f360affd640 1 -- 172.21.15.88:0/2517719962 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc019cd0 con 0x55bfe80a9820 2024-09-16T01:40:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.738+0000 7f3613970f40 1 -- 172.21.15.88:0/2517719962 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfe8118320 con 0x55bfe80a9820 2024-09-16T01:40:41.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.739+0000 7f360affd640 1 -- 172.21.15.88:0/2517719962 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f35fc017040 con 0x55bfe80a9820 2024-09-16T01:40:41.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.739+0000 7f360affd640 1 --2- 172.21.15.88:0/2517719962 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f35f406ac90 0x7f35f406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.740+0000 7f360affd640 1 -- 172.21.15.88:0/2517719962 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6053+0+0 (secure 0 0 0) 0x7f35fc08c600 con 0x55bfe80a9820 2024-09-16T01:40:41.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.740+0000 7f3611466640 1 --2- 172.21.15.88:0/2517719962 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f35f406ac90 0x7f35f406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.740+0000 7f3613970f40 1 --2- 172.21.15.88:0/2517719962 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55bfe81d3ac0 0x55bfe8114370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.740+0000 7f3613970f40 1 -- 172.21.15.88:0/2517719962 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:1 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e89) -- 0x55bfe81148b0 con 0x55bfe81d3ac0 2024-09-16T01:40:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.740+0000 7f3611466640 1 --2- 172.21.15.88:0/2517719962 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f35f406ac90 0x7f35f406d150 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3604057000 tx=0x7f3604052000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.740+0000 7f3611c67640 1 --2- 172.21.15.88:0/2517719962 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55bfe81d3ac0 0x55bfe8114370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.741+0000 7f3611c67640 1 --2- 172.21.15.88:0/2517719962 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55bfe81d3ac0 0x55bfe8114370 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.742+0000 7f3611c67640 1 -- 172.21.15.88:0/2517719962 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(1 200.00000000 [delete] v89'109 uv109 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f360c00c730 con 0x55bfe81d3ac0 2024-09-16T01:40:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.742+0000 7f3613970f40 1 -- 172.21.15.88:0/2517719962 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55bfe81d3ac0 msgr2=0x55bfe8114370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.742+0000 7f3613970f40 1 --2- 172.21.15.88:0/2517719962 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55bfe81d3ac0 0x55bfe8114370 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.742+0000 7f3613970f40 1 -- 172.21.15.88:0/2517719962 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f35f406ac90 msgr2=0x7f35f406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.742+0000 7f3613970f40 1 --2- 172.21.15.88:0/2517719962 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f35f406ac90 0x7f35f406d150 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3604057000 tx=0x7f3604052000 comp rx=0 tx=0).stop 2024-09-16T01:40:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.742+0000 7f3613970f40 1 -- 172.21.15.88:0/2517719962 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfe80a9820 msgr2=0x55bfe8112250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.742+0000 7f3613970f40 1 --2- 172.21.15.88:0/2517719962 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfe80a9820 0x55bfe8112250 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00c010 tx=0x7f35fc000ed0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.742+0000 7f3611c67640 1 -- 172.21.15.88:0/2517719962 reap_dead start 2024-09-16T01:40:41.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.742+0000 7f3613970f40 1 -- 172.21.15.88:0/2517719962 shutdown_connections 2024-09-16T01:40:41.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.742+0000 7f3613970f40 1 -- 172.21.15.88:0/2517719962 >> 172.21.15.88:0/2517719962 conn(0x55bfe8060be0 msgr2=0x55bfe8138210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:41.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.742+0000 7f3613970f40 1 -- 172.21.15.88:0/2517719962 shutdown_connections 2024-09-16T01:40:41.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.742+0000 7f3613970f40 1 -- 172.21.15.88:0/2517719962 wait complete. 2024-09-16T01:40:41.748 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:40:41.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.888+0000 7fe8a70af640 1 Processor -- start 2024-09-16T01:40:41.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.888+0000 7fe8a70af640 1 -- start start 2024-09-16T01:40:41.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.889+0000 7fe8a70af640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe8a015e6f0 0x7fe8a006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.889+0000 7fe8a70af640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8a006e990 con 0x7fe8a00b54d0 2024-09-16T01:40:41.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.889+0000 7fe8a70af640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8a006ec70 con 0x7fe8a00b7450 2024-09-16T01:40:41.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.889+0000 7fe8a70af640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe8a015ec70 con 0x7fe8a015e6f0 2024-09-16T01:40:41.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.889+0000 7fe89ffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe8a00b54d0 0x7fe8a00b6d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59660/0 (socket says 172.21.15.88:59660) 2024-09-16T01:40:41.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.889+0000 7fe89ffff640 1 -- 172.21.15.88:0/4206488713 learned_addr learned my addr 172.21.15.88:0/4206488713 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:41.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.889+0000 7fe8a4e4b640 1 --2- 172.21.15.88:0/4206488713 >> v2:172.21.15.154:3300/0 conn(0x7fe8a015e6f0 0x7fe8a006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.889+0000 7fe89f7fe640 1 -- 172.21.15.88:0/4206488713 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 614801838 0 0) 0x7fe8a006e990 con 0x7fe8a00b54d0 2024-09-16T01:40:41.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.889+0000 7fe89f7fe640 1 -- 172.21.15.88:0/4206488713 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe888003520 con 0x7fe8a00b54d0 2024-09-16T01:40:41.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.890+0000 7fe8a4e4b640 1 -- 172.21.15.88:0/4206488713 >> v1:172.21.15.154:6789/0 conn(0x7fe8a00b7450 legacy=0x7fe8a0070830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:41.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.890+0000 7fe8a4e4b640 1 -- 172.21.15.88:0/4206488713 >> v1:172.21.15.88:6789/0 conn(0x7fe8a00b54d0 legacy=0x7fe8a00b6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:41.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.890+0000 7fe8a4e4b640 1 -- 172.21.15.88:0/4206488713 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a015ef50 con 0x7fe8a015e6f0 2024-09-16T01:40:41.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.890+0000 7fe8a4e4b640 1 --2- 172.21.15.88:0/4206488713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8a015e6f0 0x7fe8a006fd00 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe8900097a0 tx=0x7fe89002a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4288dfb1463e5aa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:41.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.891+0000 7fe89f7fe640 1 -- 172.21.15.88:0/4206488713 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe890037070 con 0x7fe8a015e6f0 2024-09-16T01:40:41.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.891+0000 7fe89f7fe640 1 -- 172.21.15.88:0/4206488713 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe890032440 con 0x7fe8a015e6f0 2024-09-16T01:40:41.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.891+0000 7fe89f7fe640 1 -- 172.21.15.88:0/4206488713 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8900327f0 con 0x7fe8a015e6f0 2024-09-16T01:40:41.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.891+0000 7fe8a70af640 1 -- 172.21.15.88:0/4206488713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8a015e6f0 msgr2=0x7fe8a006fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:41.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.891+0000 7fe8a70af640 1 --2- 172.21.15.88:0/4206488713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8a015e6f0 0x7fe8a006fd00 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe8900097a0 tx=0x7fe89002a230 comp rx=0 tx=0).stop 2024-09-16T01:40:41.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.891+0000 7fe8a70af640 1 -- 172.21.15.88:0/4206488713 shutdown_connections 2024-09-16T01:40:41.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.891+0000 7fe8a70af640 1 --2- 172.21.15.88:0/4206488713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8a015e6f0 0x7fe8a006fd00 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.891+0000 7fe8a70af640 1 -- 172.21.15.88:0/4206488713 >> 172.21.15.88:0/4206488713 conn(0x7fe8a00d2430 msgr2=0x7fe8a00d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:41.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.891+0000 7fe8a70af640 1 -- 172.21.15.88:0/4206488713 shutdown_connections 2024-09-16T01:40:41.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.891+0000 7fe8a70af640 1 -- 172.21.15.88:0/4206488713 wait complete. 2024-09-16T01:40:41.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.892+0000 7fe8a70af640 1 Processor -- start 2024-09-16T01:40:41.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.892+0000 7fe8a70af640 1 -- start start 2024-09-16T01:40:41.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.893+0000 7fe8a70af640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8a00b54d0 0x7fe8a0161450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.893+0000 7fe8a70af640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8a00b7450 0x7fe8a0162ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.893+0000 7fe8a70af640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8a015e6f0 0x7fe8a015c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.893+0000 7fe8a70af640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a006e730 con 0x7fe8a00b54d0 2024-09-16T01:40:41.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.893+0000 7fe8a70af640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a006e5b0 con 0x7fe8a015e6f0 2024-09-16T01:40:41.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.893+0000 7fe8a70af640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe8a015f750 con 0x7fe8a00b7450 2024-09-16T01:40:41.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.893+0000 7fe8a4e4b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8a00b54d0 0x7fe8a0161450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.893+0000 7fe8a4e4b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8a00b54d0 0x7fe8a0161450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55322/0 (socket says 172.21.15.88:55322) 2024-09-16T01:40:41.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.893+0000 7fe8a4e4b640 1 -- 172.21.15.88:0/539647775 learned_addr learned my addr 172.21.15.88:0/539647775 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:41.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.893+0000 7fe8a564c640 1 --2- 172.21.15.88:0/539647775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8a015e6f0 0x7fe8a015c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.894+0000 7fe89ffff640 1 --2- 172.21.15.88:0/539647775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8a00b7450 0x7fe8a0162ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.894+0000 7fe8a4e4b640 1 -- 172.21.15.88:0/539647775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8a00b7450 msgr2=0x7fe8a0162ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:41.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.894+0000 7fe8a4e4b640 1 --2- 172.21.15.88:0/539647775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8a00b7450 0x7fe8a0162ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.894+0000 7fe8a4e4b640 1 -- 172.21.15.88:0/539647775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8a015e6f0 msgr2=0x7fe8a015c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:41.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.894+0000 7fe8a4e4b640 1 --2- 172.21.15.88:0/539647775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8a015e6f0 0x7fe8a015c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:41.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.894+0000 7fe8a4e4b640 1 -- 172.21.15.88:0/539647775 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a0161a30 con 0x7fe8a00b54d0 2024-09-16T01:40:41.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.894+0000 7fe89ffff640 1 --2- 172.21.15.88:0/539647775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8a00b7450 0x7fe8a0162ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:41.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.894+0000 7fe8a4e4b640 1 --2- 172.21.15.88:0/539647775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8a00b54d0 0x7fe8a0161450 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fe8900097a0 tx=0x7fe8900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:41.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.894+0000 7fe89d7fa640 1 -- 172.21.15.88:0/539647775 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe890037050 con 0x7fe8a00b54d0 2024-09-16T01:40:41.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.894+0000 7fe89d7fa640 1 -- 172.21.15.88:0/539647775 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe890038040 con 0x7fe8a00b54d0 2024-09-16T01:40:41.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.894+0000 7fe8a70af640 1 -- 172.21.15.88:0/539647775 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8a0161d10 con 0x7fe8a00b54d0 2024-09-16T01:40:41.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.894+0000 7fe89d7fa640 1 -- 172.21.15.88:0/539647775 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe890004470 con 0x7fe8a00b54d0 2024-09-16T01:40:41.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.894+0000 7fe8a70af640 1 -- 172.21.15.88:0/539647775 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8a01622d0 con 0x7fe8a00b54d0 2024-09-16T01:40:41.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.895+0000 7fe8a70af640 1 -- 172.21.15.88:0/539647775 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8a0068e10 con 0x7fe8a00b54d0 2024-09-16T01:40:41.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.896+0000 7fe89d7fa640 1 -- 172.21.15.88:0/539647775 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe89003c6d0 con 0x7fe8a00b54d0 2024-09-16T01:40:41.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.896+0000 7fe89d7fa640 1 --2- 172.21.15.88:0/539647775 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe88c06c2b0 0x7fe88c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:41.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.896+0000 7fe89d7fa640 1 -- 172.21.15.88:0/539647775 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 6053+0+0 (secure 0 0 0) 0x7fe8900abec0 con 0x7fe8a00b54d0 2024-09-16T01:40:41.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.896+0000 7fe89ffff640 1 --2- 172.21.15.88:0/539647775 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe88c06c2b0 0x7fe88c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:41.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.897+0000 7fe89ffff640 1 --2- 172.21.15.88:0/539647775 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe88c06c2b0 0x7fe88c06e770 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe8a01548b0 tx=0x7fe894009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:41.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:41.900+0000 7fe89d7fa640 1 -- 172.21.15.88:0/539647775 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe89007a560 con 0x7fe8a00b54d0 2024-09-16T01:40:42.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.100+0000 7fe8a70af640 1 -- 172.21.15.88:0/539647775 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fe8a015a160 con 0x7fe8a00b54d0 2024-09-16T01:40:42.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.386+0000 7fe89d7fa640 1 -- 172.21.15.88:0/539647775 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v94) ==== 162+0+0 (secure 0 0 0) 0x7fe890079e60 con 0x7fe8a00b54d0 2024-09-16T01:40:42.388 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:40:42.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.388+0000 7fe8a70af640 1 -- 172.21.15.88:0/539647775 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe88c06c2b0 msgr2=0x7fe88c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:42.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.388+0000 7fe8a70af640 1 --2- 172.21.15.88:0/539647775 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe88c06c2b0 0x7fe88c06e770 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe8a01548b0 tx=0x7fe894009040 comp rx=0 tx=0).stop 2024-09-16T01:40:42.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.388+0000 7fe8a70af640 1 -- 172.21.15.88:0/539647775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8a00b54d0 msgr2=0x7fe8a0161450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:42.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.388+0000 7fe8a70af640 1 --2- 172.21.15.88:0/539647775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8a00b54d0 0x7fe8a0161450 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fe8900097a0 tx=0x7fe8900046c0 comp rx=0 tx=0).stop 2024-09-16T01:40:42.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.389+0000 7fe8a70af640 1 -- 172.21.15.88:0/539647775 shutdown_connections 2024-09-16T01:40:42.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.389+0000 7fe8a70af640 1 --2- 172.21.15.88:0/539647775 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe88c06c2b0 0x7fe88c06e770 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:42.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.389+0000 7fe8a70af640 1 --2- 172.21.15.88:0/539647775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8a015e6f0 0x7fe8a015c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:42.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.389+0000 7fe8a70af640 1 --2- 172.21.15.88:0/539647775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe8a00b7450 0x7fe8a0162ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:42.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.389+0000 7fe8a70af640 1 --2- 172.21.15.88:0/539647775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe8a00b54d0 0x7fe8a0161450 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:42.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.389+0000 7fe8a70af640 1 -- 172.21.15.88:0/539647775 >> 172.21.15.88:0/539647775 conn(0x7fe8a00d2430 msgr2=0x7fe8a00b2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:42.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.389+0000 7fe8a70af640 1 -- 172.21.15.88:0/539647775 shutdown_connections 2024-09-16T01:40:42.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.389+0000 7fe8a70af640 1 -- 172.21.15.88:0/539647775 wait complete. 2024-09-16T01:40:42.399 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:40:42.398+0000 7ff36aa6b640 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x200: (2) No such file or directory 2024-09-16T01:40:42.399 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:40:42.398+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:40:42.400 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:40:42.398+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:40:42.403 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:40:42.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.499+0000 7fc8140af640 1 Processor -- start 2024-09-16T01:40:42.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.499+0000 7fc8140af640 1 -- start start 2024-09-16T01:40:42.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.500+0000 7fc8140af640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc80c0b4920 0x7fc80c15b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:42.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.500+0000 7fc8140af640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc80c0ce4f0 con 0x7fc80c0b4d00 2024-09-16T01:40:42.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.500+0000 7fc8140af640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc80c0ce7d0 con 0x7fc80c0cd780 2024-09-16T01:40:42.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.500+0000 7fc8140af640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc80c0cf100 con 0x7fc80c0b4920 2024-09-16T01:40:42.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.500+0000 7fc81164a640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc80c0b4d00 0x7fc80c0b5140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59672/0 (socket says 172.21.15.88:59672) 2024-09-16T01:40:42.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.500+0000 7fc81164a640 1 -- 172.21.15.88:0/3862351070 learned_addr learned my addr 172.21.15.88:0/3862351070 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:42.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.501+0000 7fc811e4b640 1 --2- 172.21.15.88:0/3862351070 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc80c0b4920 0x7fc80c15b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:42.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.501+0000 7fc810e49640 1 -- 172.21.15.88:0/3862351070 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2048933342 0 0) 0x7fc80c0ce4f0 con 0x7fc80c0b4d00 2024-09-16T01:40:42.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.501+0000 7fc810e49640 1 -- 172.21.15.88:0/3862351070 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7f4003520 con 0x7fc80c0b4d00 2024-09-16T01:40:42.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.501+0000 7fc810e49640 1 -- 172.21.15.88:0/3862351070 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2138840399 0 0) 0x7fc7f4003520 con 0x7fc80c0b4d00 2024-09-16T01:40:42.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.501+0000 7fc810e49640 1 -- 172.21.15.88:0/3862351070 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc80c0b4920 msgr2=0x7fc80c15b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:42.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.501+0000 7fc810e49640 1 --2- 172.21.15.88:0/3862351070 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc80c0b4920 0x7fc80c15b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:42.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.501+0000 7fc810e49640 1 -- 172.21.15.88:0/3862351070 >> v1:172.21.15.154:6789/0 conn(0x7fc80c0cd780 legacy=0x7fc80c0cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:42.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.501+0000 7fc810e49640 1 -- 172.21.15.88:0/3862351070 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc80c0ceab0 con 0x7fc80c0b4d00 2024-09-16T01:40:42.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.501+0000 7fc810e49640 1 -- 172.21.15.88:0/3862351070 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fc800001670 con 0x7fc80c0b4d00 2024-09-16T01:40:42.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.501+0000 7fc810e49640 1 -- 172.21.15.88:0/3862351070 >> v1:172.21.15.88:6789/0 conn(0x7fc80c0b4d00 legacy=0x7fc80c0b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:42.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.501+0000 7fc810e49640 1 --2- 172.21.15.88:0/3862351070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7f40039e0 0x7fc7f4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:42.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.501+0000 7fc810e49640 1 --2- 172.21.15.88:0/3862351070 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc80c0b4920 0x7fc7f4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:42.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.501+0000 7fc810e49640 1 --2- 172.21.15.88:0/3862351070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7f4004d40 0x7fc7f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:42.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.501+0000 7fc810e49640 1 -- 172.21.15.88:0/3862351070 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc80c0cf100 con 0x7fc7f4004d40 2024-09-16T01:40:42.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.501+0000 7fc810e49640 1 -- 172.21.15.88:0/3862351070 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f400ba70 con 0x7fc7f40039e0 2024-09-16T01:40:42.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.501+0000 7fc810e49640 1 -- 172.21.15.88:0/3862351070 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc7f400bc70 con 0x7fc80c0b4920 2024-09-16T01:40:42.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.501+0000 7fc81264c640 1 --2- 172.21.15.88:0/3862351070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7f4004d40 0x7fc7f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:42.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.501+0000 7fc81164a640 1 --2- 172.21.15.88:0/3862351070 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc80c0b4920 0x7fc7f4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:42.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.501+0000 7fc811e4b640 1 --2- 172.21.15.88:0/3862351070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7f40039e0 0x7fc7f4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:42.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.501+0000 7fc81264c640 1 -- 172.21.15.88:0/3862351070 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc80c0b4920 msgr2=0x7fc7f4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:42.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.501+0000 7fc81264c640 1 --2- 172.21.15.88:0/3862351070 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc80c0b4920 0x7fc7f4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:42.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.501+0000 7fc81264c640 1 -- 172.21.15.88:0/3862351070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7f40039e0 msgr2=0x7fc7f4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:42.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.501+0000 7fc81264c640 1 --2- 172.21.15.88:0/3862351070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7f40039e0 0x7fc7f4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:42.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.501+0000 7fc81264c640 1 -- 172.21.15.88:0/3862351070 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f4003520 con 0x7fc7f4004d40 2024-09-16T01:40:42.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.502+0000 7fc81264c640 1 --2- 172.21.15.88:0/3862351070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7f4004d40 0x7fc7f40051c0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fc804001720 tx=0x7fc7f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=789453d1c1c674a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:42.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.502+0000 7fc810e49640 1 -- 172.21.15.88:0/3862351070 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc804001370 con 0x7fc7f4004d40 2024-09-16T01:40:42.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.502+0000 7fc810e49640 1 -- 172.21.15.88:0/3862351070 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8040052d0 con 0x7fc7f4004d40 2024-09-16T01:40:42.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.502+0000 7fc810e49640 1 -- 172.21.15.88:0/3862351070 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80402abc0 con 0x7fc7f4004d40 2024-09-16T01:40:42.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.503+0000 7fc8140af640 1 -- 172.21.15.88:0/3862351070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7f4004d40 msgr2=0x7fc7f40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:42.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.503+0000 7fc8140af640 1 --2- 172.21.15.88:0/3862351070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc7f4004d40 0x7fc7f40051c0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fc804001720 tx=0x7fc7f4005830 comp rx=0 tx=0).stop 2024-09-16T01:40:42.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.503+0000 7fc81264c640 1 -- 172.21.15.88:0/3862351070 reap_dead start 2024-09-16T01:40:42.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.503+0000 7fc8140af640 1 -- 172.21.15.88:0/3862351070 shutdown_connections 2024-09-16T01:40:42.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.503+0000 7fc8140af640 1 -- 172.21.15.88:0/3862351070 >> 172.21.15.88:0/3862351070 conn(0x7fc80c06bc10 msgr2=0x7fc80c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:42.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.503+0000 7fc8140af640 1 -- 172.21.15.88:0/3862351070 shutdown_connections 2024-09-16T01:40:42.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.503+0000 7fc8140af640 1 -- 172.21.15.88:0/3862351070 wait complete. 2024-09-16T01:40:42.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.504+0000 7fc8140af640 1 Processor -- start 2024-09-16T01:40:42.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.504+0000 7fc8140af640 1 -- start start 2024-09-16T01:40:42.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.505+0000 7fc8140af640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7f4004d40 0x7fc80c1f7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:42.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.505+0000 7fc8140af640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc80c14e150 0x7fc80c147b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:42.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.505+0000 7fc8140af640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc80c0cf490 0x7fc80c0cf910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:42.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.505+0000 7fc8140af640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc80c15b950 con 0x7fc80c14e150 2024-09-16T01:40:42.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.505+0000 7fc8140af640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc80c15b7d0 con 0x7fc7f4004d40 2024-09-16T01:40:42.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.505+0000 7fc8140af640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc80c15bad0 con 0x7fc80c0cf490 2024-09-16T01:40:42.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.505+0000 7fc81164a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc80c14e150 0x7fc80c147b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:42.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.505+0000 7fc81164a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc80c14e150 0x7fc80c147b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55338/0 (socket says 172.21.15.88:55338) 2024-09-16T01:40:42.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.505+0000 7fc81164a640 1 -- 172.21.15.88:0/3029810307 learned_addr learned my addr 172.21.15.88:0/3029810307 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:42.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.505+0000 7fc811e4b640 1 --2- 172.21.15.88:0/3029810307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7f4004d40 0x7fc80c1f7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:42.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.505+0000 7fc81264c640 1 --2- 172.21.15.88:0/3029810307 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc80c0cf490 0x7fc80c0cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:42.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.505+0000 7fc81164a640 1 -- 172.21.15.88:0/3029810307 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc80c0cf490 msgr2=0x7fc80c0cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:42.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.505+0000 7fc81164a640 1 --2- 172.21.15.88:0/3029810307 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc80c0cf490 0x7fc80c0cf910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:42.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.505+0000 7fc81164a640 1 -- 172.21.15.88:0/3029810307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7f4004d40 msgr2=0x7fc80c1f7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:42.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.505+0000 7fc81164a640 1 --2- 172.21.15.88:0/3029810307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7f4004d40 0x7fc80c1f7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:42.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.505+0000 7fc81164a640 1 -- 172.21.15.88:0/3029810307 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc80c151cd0 con 0x7fc80c14e150 2024-09-16T01:40:42.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.505+0000 7fc81264c640 1 --2- 172.21.15.88:0/3029810307 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc80c0cf490 0x7fc80c0cf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:42.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.505+0000 7fc81164a640 1 --2- 172.21.15.88:0/3029810307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc80c14e150 0x7fc80c147b70 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fc8000072f0 tx=0x7fc8000076f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:42.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.505+0000 7fc80a7fc640 1 -- 172.21.15.88:0/3029810307 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80000f960 con 0x7fc80c14e150 2024-09-16T01:40:42.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.505+0000 7fc80a7fc640 1 -- 172.21.15.88:0/3029810307 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc800018070 con 0x7fc80c14e150 2024-09-16T01:40:42.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.505+0000 7fc8140af640 1 -- 172.21.15.88:0/3029810307 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc80c1f7f60 con 0x7fc80c14e150 2024-09-16T01:40:42.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.505+0000 7fc80a7fc640 1 -- 172.21.15.88:0/3029810307 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80000fd40 con 0x7fc80c14e150 2024-09-16T01:40:42.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.505+0000 7fc8140af640 1 -- 172.21.15.88:0/3029810307 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc80c1f8520 con 0x7fc80c14e150 2024-09-16T01:40:42.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.506+0000 7fc8140af640 1 -- 172.21.15.88:0/3029810307 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc80c0b5140 con 0x7fc80c14e150 2024-09-16T01:40:42.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.507+0000 7fc80a7fc640 1 -- 172.21.15.88:0/3029810307 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc80001d460 con 0x7fc80c14e150 2024-09-16T01:40:42.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.507+0000 7fc80a7fc640 1 --2- 172.21.15.88:0/3029810307 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc7f806c2e0 0x7fc7f806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:42.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.507+0000 7fc811e4b640 1 --2- 172.21.15.88:0/3029810307 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc7f806c2e0 0x7fc7f806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:42.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.507+0000 7fc80a7fc640 1 -- 172.21.15.88:0/3029810307 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6139+0+0 (secure 0 0 0) 0x7fc80008b570 con 0x7fc80c14e150 2024-09-16T01:40:42.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.508+0000 7fc811e4b640 1 --2- 172.21.15.88:0/3029810307 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc7f806c2e0 0x7fc7f806e7a0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc014510 tx=0x7fc7fc00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:42.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.511+0000 7fc80a7fc640 1 -- 172.21.15.88:0/3029810307 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc800059bc0 con 0x7fc80c14e150 2024-09-16T01:40:42.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.714+0000 7fc8140af640 1 -- 172.21.15.88:0/3029810307 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc80c15dc10 con 0x7fc80c14e150 2024-09-16T01:40:42.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.714+0000 7fc80a7fc640 1 -- 172.21.15.88:0/3029810307 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+3563 (secure 0 0 0) 0x7fc8000594c0 con 0x7fc80c14e150 2024-09-16T01:40:42.716 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:42.716 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":96,"btime":"2024-09-16T01:40:42:399593+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:40:42.399539+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:40:42.716 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 96 2024-09-16T01:40:42.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.716+0000 7fc8140af640 1 -- 172.21.15.88:0/3029810307 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc7f806c2e0 msgr2=0x7fc7f806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:42.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.716+0000 7fc8140af640 1 --2- 172.21.15.88:0/3029810307 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc7f806c2e0 0x7fc7f806e7a0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc014510 tx=0x7fc7fc00b040 comp rx=0 tx=0).stop 2024-09-16T01:40:42.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.716+0000 7fc8140af640 1 -- 172.21.15.88:0/3029810307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc80c14e150 msgr2=0x7fc80c147b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:42.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.716+0000 7fc8140af640 1 --2- 172.21.15.88:0/3029810307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc80c14e150 0x7fc80c147b70 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fc8000072f0 tx=0x7fc8000076f0 comp rx=0 tx=0).stop 2024-09-16T01:40:42.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.716+0000 7fc8140af640 1 -- 172.21.15.88:0/3029810307 shutdown_connections 2024-09-16T01:40:42.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.716+0000 7fc8140af640 1 --2- 172.21.15.88:0/3029810307 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc7f806c2e0 0x7fc7f806e7a0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:42.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.716+0000 7fc8140af640 1 --2- 172.21.15.88:0/3029810307 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc80c0cf490 0x7fc80c0cf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:42.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.716+0000 7fc8140af640 1 --2- 172.21.15.88:0/3029810307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc80c14e150 0x7fc80c147b70 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:42.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.716+0000 7fc8140af640 1 --2- 172.21.15.88:0/3029810307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7f4004d40 0x7fc80c1f7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:42.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.716+0000 7fc8140af640 1 -- 172.21.15.88:0/3029810307 >> 172.21.15.88:0/3029810307 conn(0x7fc80c06bc10 msgr2=0x7fc80c0cdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:42.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.716+0000 7fc8140af640 1 -- 172.21.15.88:0/3029810307 shutdown_connections 2024-09-16T01:40:42.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.716+0000 7fc8140af640 1 -- 172.21.15.88:0/3029810307 wait complete. 2024-09-16T01:40:42.729 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:40:42.729 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:40:42.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.825+0000 7f8c23d86640 1 Processor -- start 2024-09-16T01:40:42.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.826+0000 7f8c23d86640 1 -- start start 2024-09-16T01:40:42.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.826+0000 7f8c23d86640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8c1c0d3310 0x7f8c1c0d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:42.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.826+0000 7f8c23d86640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c1c0d1650 0x7f8c1c0b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:42.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.826+0000 7f8c23d86640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c1c15ea40 con 0x7f8c1c15e660 2024-09-16T01:40:42.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.826+0000 7f8c23d86640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f8c1c0d1a90 con 0x7f8c1c0d3310 2024-09-16T01:40:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.826+0000 7f8c23d86640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8c1c0d3870 con 0x7f8c1c0d1650 2024-09-16T01:40:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.826+0000 7f8c21b22640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8c1c15e660 0x7f8c1c0cf730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59686/0 (socket says 172.21.15.88:59686) 2024-09-16T01:40:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.826+0000 7f8c21b22640 1 -- 172.21.15.88:0/2414839096 learned_addr learned my addr 172.21.15.88:0/2414839096 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c21321640 1 --2- 172.21.15.88:0/2414839096 >> v2:172.21.15.154:3300/0 conn(0x7f8c1c0d3310 0x7f8c1c0d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c22323640 1 --2- 172.21.15.88:0/2414839096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c1c0d1650 0x7f8c1c0b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c20b20640 1 -- 172.21.15.88:0/2414839096 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2760284027 0 0) 0x7f8c1c15ea40 con 0x7f8c1c15e660 2024-09-16T01:40:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c20b20640 1 -- 172.21.15.88:0/2414839096 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c08003520 con 0x7f8c1c15e660 2024-09-16T01:40:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c20b20640 1 -- 172.21.15.88:0/2414839096 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2263760617 0 0) 0x7f8c08003520 con 0x7f8c1c15e660 2024-09-16T01:40:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c20b20640 1 -- 172.21.15.88:0/2414839096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c1c0d1650 msgr2=0x7f8c1c0b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c20b20640 1 --2- 172.21.15.88:0/2414839096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c1c0d1650 0x7f8c1c0b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c20b20640 1 -- 172.21.15.88:0/2414839096 >> v2:172.21.15.154:3300/0 conn(0x7f8c1c0d3310 msgr2=0x7f8c1c0d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c20b20640 1 --2- 172.21.15.88:0/2414839096 >> v2:172.21.15.154:3300/0 conn(0x7f8c1c0d3310 0x7f8c1c0d1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c20b20640 1 -- 172.21.15.88:0/2414839096 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c1c0d3b50 con 0x7f8c1c15e660 2024-09-16T01:40:42.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c20b20640 1 -- 172.21.15.88:0/2414839096 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f8c180019f0 con 0x7f8c1c15e660 2024-09-16T01:40:42.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c20b20640 1 -- 172.21.15.88:0/2414839096 >> v1:172.21.15.88:6789/0 conn(0x7f8c1c15e660 legacy=0x7f8c1c0cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:42.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c20b20640 1 --2- 172.21.15.88:0/2414839096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c080039b0 0x7f8c08003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:42.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c20b20640 1 --2- 172.21.15.88:0/2414839096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c08004800 0x7f8c08004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:42.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c20b20640 1 --2- 172.21.15.88:0/2414839096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c1c0d1650 0x7f8c080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:42.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c20b20640 1 -- 172.21.15.88:0/2414839096 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8c1c0d1a90 con 0x7f8c080039b0 2024-09-16T01:40:42.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c20b20640 1 -- 172.21.15.88:0/2414839096 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8c1c0d3870 con 0x7f8c08004800 2024-09-16T01:40:42.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c20b20640 1 -- 172.21.15.88:0/2414839096 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8c0800b9c0 con 0x7f8c1c0d1650 2024-09-16T01:40:42.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c21b22640 1 --2- 172.21.15.88:0/2414839096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c080039b0 0x7f8c08003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:42.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c21321640 1 --2- 172.21.15.88:0/2414839096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c08004800 0x7f8c08004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:42.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c22323640 1 --2- 172.21.15.88:0/2414839096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c1c0d1650 0x7f8c080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:42.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c21b22640 1 -- 172.21.15.88:0/2414839096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c1c0d1650 msgr2=0x7f8c080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:42.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c21b22640 1 --2- 172.21.15.88:0/2414839096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c1c0d1650 0x7f8c080051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:42.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c21b22640 1 -- 172.21.15.88:0/2414839096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c08004800 msgr2=0x7f8c08004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:42.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c21b22640 1 --2- 172.21.15.88:0/2414839096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c08004800 0x7f8c08004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:42.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.827+0000 7f8c21b22640 1 -- 172.21.15.88:0/2414839096 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c08003520 con 0x7f8c080039b0 2024-09-16T01:40:42.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.828+0000 7f8c21b22640 1 --2- 172.21.15.88:0/2414839096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c080039b0 0x7f8c08003dd0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f8c1800a880 tx=0x7f8c08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c693e4c914f520 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:42.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.828+0000 7f8c20b20640 1 -- 172.21.15.88:0/2414839096 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c1800a4a0 con 0x7f8c080039b0 2024-09-16T01:40:42.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.828+0000 7f8c20b20640 1 -- 172.21.15.88:0/2414839096 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c1802f070 con 0x7f8c080039b0 2024-09-16T01:40:42.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.828+0000 7f8c20b20640 1 -- 172.21.15.88:0/2414839096 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c1802bd00 con 0x7f8c080039b0 2024-09-16T01:40:42.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.829+0000 7f8c23d86640 1 -- 172.21.15.88:0/2414839096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c080039b0 msgr2=0x7f8c08003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:42.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.829+0000 7f8c23d86640 1 --2- 172.21.15.88:0/2414839096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c080039b0 0x7f8c08003dd0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f8c1800a880 tx=0x7f8c08005830 comp rx=0 tx=0).stop 2024-09-16T01:40:42.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.829+0000 7f8c22323640 1 -- 172.21.15.88:0/2414839096 reap_dead start 2024-09-16T01:40:42.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.829+0000 7f8c23d86640 1 -- 172.21.15.88:0/2414839096 shutdown_connections 2024-09-16T01:40:42.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.829+0000 7f8c23d86640 1 -- 172.21.15.88:0/2414839096 >> 172.21.15.88:0/2414839096 conn(0x7f8c1c0b6410 msgr2=0x7f8c1c0b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:42.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.829+0000 7f8c23d86640 1 -- 172.21.15.88:0/2414839096 shutdown_connections 2024-09-16T01:40:42.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.829+0000 7f8c23d86640 1 -- 172.21.15.88:0/2414839096 wait complete. 2024-09-16T01:40:42.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.830+0000 7f8c23d86640 1 Processor -- start 2024-09-16T01:40:42.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.830+0000 7f8c23d86640 1 -- start start 2024-09-16T01:40:42.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.831+0000 7f8c23d86640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c080039b0 0x7f8c1c1f4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:42.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.831+0000 7f8c23d86640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c1c0cd6e0 0x7f8c1c0ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:42.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.831+0000 7f8c23d86640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c1c0cc0f0 0x7f8c1c0cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:42.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.831+0000 7f8c23d86640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8c1c0d43f0 con 0x7f8c1c0cd6e0 2024-09-16T01:40:42.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.831+0000 7f8c23d86640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8c1c0d4110 con 0x7f8c1c0cc0f0 2024-09-16T01:40:42.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.831+0000 7f8c23d86640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8c1c0b7b70 con 0x7f8c080039b0 2024-09-16T01:40:42.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.831+0000 7f8c21321640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c1c0cd6e0 0x7f8c1c0ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:42.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.831+0000 7f8c21321640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c1c0cd6e0 0x7f8c1c0ceaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55368/0 (socket says 172.21.15.88:55368) 2024-09-16T01:40:42.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.831+0000 7f8c21321640 1 -- 172.21.15.88:0/1917714751 learned_addr learned my addr 172.21.15.88:0/1917714751 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:42.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.831+0000 7f8c21b22640 1 --2- 172.21.15.88:0/1917714751 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c080039b0 0x7f8c1c1f4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:42.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.831+0000 7f8c22323640 1 --2- 172.21.15.88:0/1917714751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c1c0cc0f0 0x7f8c1c0cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:42.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.831+0000 7f8c21321640 1 -- 172.21.15.88:0/1917714751 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c080039b0 msgr2=0x7f8c1c1f4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:42.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.831+0000 7f8c21321640 1 --2- 172.21.15.88:0/1917714751 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c080039b0 0x7f8c1c1f4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:42.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.831+0000 7f8c21321640 1 -- 172.21.15.88:0/1917714751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c1c0cc0f0 msgr2=0x7f8c1c0cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:42.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.831+0000 7f8c21321640 1 --2- 172.21.15.88:0/1917714751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c1c0cc0f0 0x7f8c1c0cefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:42.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.831+0000 7f8c21321640 1 -- 172.21.15.88:0/1917714751 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c1c0cc640 con 0x7f8c1c0cd6e0 2024-09-16T01:40:42.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.831+0000 7f8c21b22640 1 --2- 172.21.15.88:0/1917714751 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c080039b0 0x7f8c1c1f4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:42.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.831+0000 7f8c21321640 1 --2- 172.21.15.88:0/1917714751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c1c0cd6e0 0x7f8c1c0ceaa0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c011960 tx=0x7f8c0c011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:42.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.831+0000 7f8c127fc640 1 -- 172.21.15.88:0/1917714751 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c00faf0 con 0x7f8c1c0cd6e0 2024-09-16T01:40:42.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.832+0000 7f8c127fc640 1 -- 172.21.15.88:0/1917714751 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c0c00fc90 con 0x7f8c1c0cd6e0 2024-09-16T01:40:42.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.832+0000 7f8c23d86640 1 -- 172.21.15.88:0/1917714751 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c1c0cc920 con 0x7f8c1c0cd6e0 2024-09-16T01:40:42.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.832+0000 7f8c127fc640 1 -- 172.21.15.88:0/1917714751 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c014440 con 0x7f8c1c0cd6e0 2024-09-16T01:40:42.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.832+0000 7f8c23d86640 1 -- 172.21.15.88:0/1917714751 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c1c0ccee0 con 0x7f8c1c0cd6e0 2024-09-16T01:40:42.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.833+0000 7f8c23d86640 1 -- 172.21.15.88:0/1917714751 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c1c0d1110 con 0x7f8c1c0cd6e0 2024-09-16T01:40:42.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.833+0000 7f8c127fc640 1 -- 172.21.15.88:0/1917714751 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8c0c012040 con 0x7f8c1c0cd6e0 2024-09-16T01:40:42.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.834+0000 7f8c127fc640 1 --2- 172.21.15.88:0/1917714751 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8c0006c2b0 0x7f8c0006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:42.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.834+0000 7f8c21b22640 1 --2- 172.21.15.88:0/1917714751 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8c0006c2b0 0x7f8c0006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:42.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.834+0000 7f8c127fc640 1 -- 172.21.15.88:0/1917714751 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6139+0+0 (secure 0 0 0) 0x7f8c0c08bbc0 con 0x7f8c1c0cd6e0 2024-09-16T01:40:42.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.834+0000 7f8c21b22640 1 --2- 172.21.15.88:0/1917714751 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8c0006c2b0 0x7f8c0006e770 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f8c18033000 tx=0x7f8c18007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:42.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:42.838+0000 7f8c127fc640 1 -- 172.21.15.88:0/1917714751 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8c0c019030 con 0x7f8c1c0cd6e0 2024-09-16T01:40:43.007 INFO:tasks.ceph.mds.c.smithi088.stderr: -32> 2024-09-16T01:40:38.254+0000 7f54a2d5a640 -1 mds.0.cache.ino(0x1) No data while reading inode 0x1 2024-09-16T01:40:43.007 INFO:tasks.ceph.mds.c.smithi088.stderr: -30> 2024-09-16T01:40:38.254+0000 7f54a2d5a640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-09-16T01:40:43.030 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:40:43.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.039+0000 7f8c23d86640 1 -- 172.21.15.88:0/1917714751 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c1c0ce880 con 0x7f8c1c0cd6e0 2024-09-16T01:40:43.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.039+0000 7f8c127fc640 1 -- 172.21.15.88:0/1917714751 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+3563 (secure 0 0 0) 0x7f8c0c059bc0 con 0x7f8c1c0cd6e0 2024-09-16T01:40:43.041 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:43.041 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":96,"btime":"2024-09-16T01:40:42:399593+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:40:42.399539+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:40:43.041 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 96 2024-09-16T01:40:43.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.041+0000 7f8c23d86640 1 -- 172.21.15.88:0/1917714751 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8c0006c2b0 msgr2=0x7f8c0006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:43.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.041+0000 7f8c23d86640 1 --2- 172.21.15.88:0/1917714751 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8c0006c2b0 0x7f8c0006e770 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f8c18033000 tx=0x7f8c18007000 comp rx=0 tx=0).stop 2024-09-16T01:40:43.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.041+0000 7f8c23d86640 1 -- 172.21.15.88:0/1917714751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c1c0cd6e0 msgr2=0x7f8c1c0ceaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:43.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.041+0000 7f8c23d86640 1 --2- 172.21.15.88:0/1917714751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c1c0cd6e0 0x7f8c1c0ceaa0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c011960 tx=0x7f8c0c011e20 comp rx=0 tx=0).stop 2024-09-16T01:40:43.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.041+0000 7f8c23d86640 1 -- 172.21.15.88:0/1917714751 shutdown_connections 2024-09-16T01:40:43.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.041+0000 7f8c23d86640 1 --2- 172.21.15.88:0/1917714751 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8c0006c2b0 0x7f8c0006e770 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:43.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.041+0000 7f8c23d86640 1 --2- 172.21.15.88:0/1917714751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c1c0cc0f0 0x7f8c1c0cefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:43.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.041+0000 7f8c23d86640 1 --2- 172.21.15.88:0/1917714751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c1c0cd6e0 0x7f8c1c0ceaa0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:43.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.041+0000 7f8c23d86640 1 --2- 172.21.15.88:0/1917714751 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c080039b0 0x7f8c1c1f4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:43.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.041+0000 7f8c23d86640 1 -- 172.21.15.88:0/1917714751 >> 172.21.15.88:0/1917714751 conn(0x7f8c1c0b6410 msgr2=0x7f8c1c15c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:43.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.041+0000 7f8c23d86640 1 -- 172.21.15.88:0/1917714751 shutdown_connections 2024-09-16T01:40:43.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.041+0000 7f8c23d86640 1 -- 172.21.15.88:0/1917714751 wait complete. 2024-09-16T01:40:43.053 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:40:43.054 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:40:43.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.150+0000 7f40e7e65640 1 Processor -- start 2024-09-16T01:40:43.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.151+0000 7f40e7e65640 1 -- start start 2024-09-16T01:40:43.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.152+0000 7f40e7e65640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f40e015fad0 0x7f40e006c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:43.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.152+0000 7f40e7e65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40e006e910 0x7f40e00d0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:43.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.152+0000 7f40e7e65640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40e00cf450 con 0x7f40e006cb60 2024-09-16T01:40:43.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.152+0000 7f40e7e65640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f40e006d070 con 0x7f40e015fad0 2024-09-16T01:40:43.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.152+0000 7f40e7e65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f40e00cfd80 con 0x7f40e006e910 2024-09-16T01:40:43.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.152+0000 7f40e5400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f40e006cb60 0x7f40e006e200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:54024/0 (socket says 172.21.15.88:54024) 2024-09-16T01:40:43.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.152+0000 7f40e5400640 1 -- 172.21.15.88:0/687162911 learned_addr learned my addr 172.21.15.88:0/687162911 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:43.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.152+0000 7f40e5c01640 1 --2- 172.21.15.88:0/687162911 >> v2:172.21.15.88:3300/0 conn(0x7f40e015fad0 0x7f40e006c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:43.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.152+0000 7f40e6402640 1 --2- 172.21.15.88:0/687162911 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40e006e910 0x7f40e00d0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:43.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.152+0000 7f40e4bff640 1 -- 172.21.15.88:0/687162911 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 323232810 0 0) 0x7f40e00cf450 con 0x7f40e006cb60 2024-09-16T01:40:43.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.152+0000 7f40e4bff640 1 -- 172.21.15.88:0/687162911 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40c8003520 con 0x7f40e006cb60 2024-09-16T01:40:43.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.152+0000 7f40e5c01640 1 -- 172.21.15.88:0/687162911 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40e006e910 msgr2=0x7f40e00d0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:43.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.152+0000 7f40e5c01640 1 --2- 172.21.15.88:0/687162911 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40e006e910 0x7f40e00d0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:43.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.152+0000 7f40e5c01640 1 -- 172.21.15.88:0/687162911 >> v1:172.21.15.154:6789/0 conn(0x7f40e006cb60 legacy=0x7f40e006e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:43.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.152+0000 7f40e5c01640 1 -- 172.21.15.88:0/687162911 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40e00cf730 con 0x7f40e015fad0 2024-09-16T01:40:43.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.153+0000 7f40e5c01640 1 --2- 172.21.15.88:0/687162911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40e015fad0 0x7f40e006c620 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f40d0009d90 tx=0x7f40d002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6888f1f84ba3c36 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:43.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.153+0000 7f40e4bff640 1 -- 172.21.15.88:0/687162911 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d0007740 con 0x7f40e015fad0 2024-09-16T01:40:43.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.153+0000 7f40e4bff640 1 -- 172.21.15.88:0/687162911 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40d00078e0 con 0x7f40e015fad0 2024-09-16T01:40:43.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.153+0000 7f40e4bff640 1 -- 172.21.15.88:0/687162911 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d0007cb0 con 0x7f40e015fad0 2024-09-16T01:40:43.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.154+0000 7f40e7e65640 1 -- 172.21.15.88:0/687162911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40e015fad0 msgr2=0x7f40e006c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:43.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.154+0000 7f40e7e65640 1 --2- 172.21.15.88:0/687162911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40e015fad0 0x7f40e006c620 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f40d0009d90 tx=0x7f40d002a660 comp rx=0 tx=0).stop 2024-09-16T01:40:43.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.154+0000 7f40e7e65640 1 -- 172.21.15.88:0/687162911 shutdown_connections 2024-09-16T01:40:43.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.154+0000 7f40e7e65640 1 --2- 172.21.15.88:0/687162911 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40e006e910 0x7f40e00d0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:43.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.154+0000 7f40e7e65640 1 --2- 172.21.15.88:0/687162911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40e015fad0 0x7f40e006c620 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:43.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.154+0000 7f40e7e65640 1 -- 172.21.15.88:0/687162911 >> 172.21.15.88:0/687162911 conn(0x7f40e00b6ec0 msgr2=0x7f40e00b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:43.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.154+0000 7f40e7e65640 1 -- 172.21.15.88:0/687162911 shutdown_connections 2024-09-16T01:40:43.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.154+0000 7f40e7e65640 1 -- 172.21.15.88:0/687162911 wait complete. 2024-09-16T01:40:43.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.155+0000 7f40e7e65640 1 Processor -- start 2024-09-16T01:40:43.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.155+0000 7f40e7e65640 1 -- start start 2024-09-16T01:40:43.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.156+0000 7f40e7e65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40e006cb60 0x7f40e0155990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:43.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.156+0000 7f40e7e65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40e006e910 0x7f40e015e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:43.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.156+0000 7f40e7e65640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40e015fad0 0x7f40e015bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:43.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.156+0000 7f40e7e65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f40e006d7a0 con 0x7f40e006e910 2024-09-16T01:40:43.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.156+0000 7f40e7e65640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f40e006d620 con 0x7f40e015fad0 2024-09-16T01:40:43.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.156+0000 7f40e7e65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f40e0071f80 con 0x7f40e006cb60 2024-09-16T01:40:43.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.156+0000 7f40e5400640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40e006e910 0x7f40e015e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:43.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.156+0000 7f40e5400640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40e006e910 0x7f40e015e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55396/0 (socket says 172.21.15.88:55396) 2024-09-16T01:40:43.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.156+0000 7f40e5400640 1 -- 172.21.15.88:0/928898591 learned_addr learned my addr 172.21.15.88:0/928898591 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:43.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.156+0000 7f40e6402640 1 --2- 172.21.15.88:0/928898591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40e015fad0 0x7f40e015bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:43.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.156+0000 7f40e5400640 1 -- 172.21.15.88:0/928898591 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40e006cb60 msgr2=0x7f40e0155990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:43.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.156+0000 7f40e5400640 1 --2- 172.21.15.88:0/928898591 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40e006cb60 0x7f40e0155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:43.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.156+0000 7f40e5400640 1 -- 172.21.15.88:0/928898591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40e015fad0 msgr2=0x7f40e015bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:43.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.156+0000 7f40e5400640 1 --2- 172.21.15.88:0/928898591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40e015fad0 0x7f40e015bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:43.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.156+0000 7f40e5400640 1 -- 172.21.15.88:0/928898591 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40e0152db0 con 0x7f40e006e910 2024-09-16T01:40:43.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.156+0000 7f40e5400640 1 --2- 172.21.15.88:0/928898591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40e006e910 0x7f40e015e7a0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f40d400c400 tx=0x7f40d400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:43.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.157+0000 7f40de7fc640 1 -- 172.21.15.88:0/928898591 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d4017070 con 0x7f40e006e910 2024-09-16T01:40:43.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.157+0000 7f40de7fc640 1 -- 172.21.15.88:0/928898591 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40d4005d00 con 0x7f40e006e910 2024-09-16T01:40:43.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.157+0000 7f40de7fc640 1 -- 172.21.15.88:0/928898591 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d4007310 con 0x7f40e006e910 2024-09-16T01:40:43.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.157+0000 7f40e7e65640 1 -- 172.21.15.88:0/928898591 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40e015efd0 con 0x7f40e006e910 2024-09-16T01:40:43.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.157+0000 7f40e7e65640 1 -- 172.21.15.88:0/928898591 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40e0153330 con 0x7f40e006e910 2024-09-16T01:40:43.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.158+0000 7f40e7e65640 1 -- 172.21.15.88:0/928898591 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40e0153530 con 0x7f40e006e910 2024-09-16T01:40:43.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.158+0000 7f40de7fc640 1 -- 172.21.15.88:0/928898591 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f40d401a070 con 0x7f40e006e910 2024-09-16T01:40:43.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.158+0000 7f40de7fc640 1 --2- 172.21.15.88:0/928898591 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f40cc06c2b0 0x7f40cc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:43.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.159+0000 7f40de7fc640 1 -- 172.21.15.88:0/928898591 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6139+0+0 (secure 0 0 0) 0x7f40d408aa00 con 0x7f40e006e910 2024-09-16T01:40:43.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.159+0000 7f40e5c01640 1 --2- 172.21.15.88:0/928898591 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f40cc06c2b0 0x7f40cc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:43.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.159+0000 7f40e5c01640 1 --2- 172.21.15.88:0/928898591 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f40cc06c2b0 0x7f40cc06e770 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f40d0034000 tx=0x7f40d002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:43.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.163+0000 7f40de7fc640 1 -- 172.21.15.88:0/928898591 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f40d4059050 con 0x7f40e006e910 2024-09-16T01:40:43.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.363+0000 7f40e7e65640 1 -- 172.21.15.88:0/928898591 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40e00738b0 con 0x7f40e006e910 2024-09-16T01:40:43.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.364+0000 7f40de7fc640 1 -- 172.21.15.88:0/928898591 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+3563 (secure 0 0 0) 0x7f40d4058950 con 0x7f40e006e910 2024-09-16T01:40:43.365 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:43.365 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":96,"btime":"2024-09-16T01:40:42:399593+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:40:42.399539+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":90,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:40:43.366 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 96 2024-09-16T01:40:43.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.366+0000 7f40e7e65640 1 -- 172.21.15.88:0/928898591 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f40cc06c2b0 msgr2=0x7f40cc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:43.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.366+0000 7f40e7e65640 1 --2- 172.21.15.88:0/928898591 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f40cc06c2b0 0x7f40cc06e770 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f40d0034000 tx=0x7f40d002e000 comp rx=0 tx=0).stop 2024-09-16T01:40:43.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.366+0000 7f40e7e65640 1 -- 172.21.15.88:0/928898591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40e006e910 msgr2=0x7f40e015e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:43.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.366+0000 7f40e7e65640 1 --2- 172.21.15.88:0/928898591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40e006e910 0x7f40e015e7a0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f40d400c400 tx=0x7f40d400c8c0 comp rx=0 tx=0).stop 2024-09-16T01:40:43.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.366+0000 7f40e7e65640 1 -- 172.21.15.88:0/928898591 shutdown_connections 2024-09-16T01:40:43.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.366+0000 7f40e7e65640 1 --2- 172.21.15.88:0/928898591 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f40cc06c2b0 0x7f40cc06e770 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:43.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.366+0000 7f40e7e65640 1 --2- 172.21.15.88:0/928898591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40e015fad0 0x7f40e015bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:43.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.366+0000 7f40e7e65640 1 --2- 172.21.15.88:0/928898591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40e006e910 0x7f40e015e7a0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:43.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.366+0000 7f40e7e65640 1 --2- 172.21.15.88:0/928898591 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40e006cb60 0x7f40e0155990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:43.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.366+0000 7f40e7e65640 1 -- 172.21.15.88:0/928898591 >> 172.21.15.88:0/928898591 conn(0x7f40e00b6ec0 msgr2=0x7f40e0072e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:43.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.366+0000 7f40e7e65640 1 -- 172.21.15.88:0/928898591 shutdown_connections 2024-09-16T01:40:43.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.366+0000 7f40e7e65640 1 -- 172.21.15.88:0/928898591 wait complete. 2024-09-16T01:40:43.378 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 200.00000000' led to DAMAGED state 2024-09-16T01:40:43.378 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 200.00000001' 2024-09-16T01:40:43.378 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:40:43.378 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:40:43.394 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:40:43.394 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:40:43.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:43.394 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:40:43.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:40:43.450 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:40:43.450 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:40:43.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.589+0000 7f4ca9cac640 1 Processor -- start 2024-09-16T01:40:43.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.589+0000 7f4ca9cac640 1 -- start start 2024-09-16T01:40:43.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.590+0000 7f4ca9cac640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4ca406fba0 0x7f4ca406c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:43.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.590+0000 7f4ca9cac640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ca4070010 con 0x7f4ca415c2a0 2024-09-16T01:40:43.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.590+0000 7f4ca9cac640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ca4074120 con 0x7f4ca406f710 2024-09-16T01:40:43.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.590+0000 7f4ca9cac640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f4ca415c680 con 0x7f4ca406fba0 2024-09-16T01:40:43.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.590+0000 7f4ca37fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4ca415c2a0 0x7f4ca40735d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59704/0 (socket says 172.21.15.88:59704) 2024-09-16T01:40:43.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.590+0000 7f4ca37fe640 1 -- 172.21.15.88:0/1308763867 learned_addr learned my addr 172.21.15.88:0/1308763867 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:43.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.590+0000 7f4ca27fc640 1 -- 172.21.15.88:0/1308763867 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 731124551 0 0) 0x7f4ca4070010 con 0x7f4ca415c2a0 2024-09-16T01:40:43.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.590+0000 7f4ca27fc640 1 -- 172.21.15.88:0/1308763867 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c8c003520 con 0x7f4ca415c2a0 2024-09-16T01:40:43.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.590+0000 7f4ca3fff640 1 --2- 172.21.15.88:0/1308763867 >> v2:172.21.15.154:3300/0 conn(0x7f4ca406fba0 0x7f4ca406c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:43.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.591+0000 7f4ca27fc640 1 -- 172.21.15.88:0/1308763867 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 607109037 0 0) 0x7f4c8c003520 con 0x7f4ca415c2a0 2024-09-16T01:40:43.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.591+0000 7f4ca27fc640 1 -- 172.21.15.88:0/1308763867 >> v2:172.21.15.154:3300/0 conn(0x7f4ca406fba0 msgr2=0x7f4ca406c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:43.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.591+0000 7f4ca27fc640 1 --2- 172.21.15.88:0/1308763867 >> v2:172.21.15.154:3300/0 conn(0x7f4ca406fba0 0x7f4ca406c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:43.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.591+0000 7f4ca27fc640 1 -- 172.21.15.88:0/1308763867 >> v1:172.21.15.154:6789/0 conn(0x7f4ca406f710 legacy=0x7f4ca406bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:43.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.591+0000 7f4ca27fc640 1 -- 172.21.15.88:0/1308763867 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca415ef00 con 0x7f4ca415c2a0 2024-09-16T01:40:43.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.591+0000 7f4ca27fc640 1 -- 172.21.15.88:0/1308763867 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f4c94001670 con 0x7f4ca415c2a0 2024-09-16T01:40:43.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.591+0000 7f4ca27fc640 1 -- 172.21.15.88:0/1308763867 >> v1:172.21.15.88:6789/0 conn(0x7f4ca415c2a0 legacy=0x7f4ca40735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:43.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.591+0000 7f4ca27fc640 1 --2- 172.21.15.88:0/1308763867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c8c0039b0 0x7f4c8c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:43.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.591+0000 7f4ca27fc640 1 --2- 172.21.15.88:0/1308763867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c8c004800 0x7f4c8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.591+0000 7f4ca27fc640 1 --2- 172.21.15.88:0/1308763867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c8c0051c0 0x7f4c8c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.591+0000 7f4ca27fc640 1 -- 172.21.15.88:0/1308763867 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4c8c00c0e0 con 0x7f4c8c0051c0 2024-09-16T01:40:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.591+0000 7f4ca27fc640 1 -- 172.21.15.88:0/1308763867 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4c8c00c290 con 0x7f4c8c004800 2024-09-16T01:40:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.591+0000 7f4ca27fc640 1 -- 172.21.15.88:0/1308763867 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4c8c00c490 con 0x7f4c8c0039b0 2024-09-16T01:40:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.591+0000 7f4ca3fff640 1 --2- 172.21.15.88:0/1308763867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c8c0051c0 0x7f4c8c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.591+0000 7f4ca2ffd640 1 --2- 172.21.15.88:0/1308763867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c8c004800 0x7f4c8c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.591+0000 7f4ca37fe640 1 --2- 172.21.15.88:0/1308763867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c8c0039b0 0x7f4c8c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.591+0000 7f4ca3fff640 1 -- 172.21.15.88:0/1308763867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c8c0039b0 msgr2=0x7f4c8c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.591+0000 7f4ca3fff640 1 --2- 172.21.15.88:0/1308763867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c8c0039b0 0x7f4c8c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.591+0000 7f4ca3fff640 1 -- 172.21.15.88:0/1308763867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c8c004800 msgr2=0x7f4c8c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.591+0000 7f4ca3fff640 1 --2- 172.21.15.88:0/1308763867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c8c004800 0x7f4c8c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.591+0000 7f4ca3fff640 1 -- 172.21.15.88:0/1308763867 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c8c003520 con 0x7f4c8c0051c0 2024-09-16T01:40:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.592+0000 7f4ca3fff640 1 -- 172.21.15.88:0/1308763867 reap_dead start 2024-09-16T01:40:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.592+0000 7f4ca3fff640 1 --2- 172.21.15.88:0/1308763867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c8c0051c0 0x7f4c8c00b9c0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c007360 tx=0x7f4c8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abd4c0b38a1e4bd5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.592+0000 7f4ca27fc640 1 -- 172.21.15.88:0/1308763867 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c009be0 con 0x7f4c8c0051c0 2024-09-16T01:40:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.592+0000 7f4ca27fc640 1 -- 172.21.15.88:0/1308763867 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c9c003bd0 con 0x7f4c8c0051c0 2024-09-16T01:40:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.592+0000 7f4ca27fc640 1 -- 172.21.15.88:0/1308763867 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c004020 con 0x7f4c8c0051c0 2024-09-16T01:40:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.592+0000 7f4ca9cac640 1 -- 172.21.15.88:0/1308763867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c8c0051c0 msgr2=0x7f4c8c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.592+0000 7f4ca9cac640 1 --2- 172.21.15.88:0/1308763867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c8c0051c0 0x7f4c8c00b9c0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c007360 tx=0x7f4c8c005830 comp rx=0 tx=0).stop 2024-09-16T01:40:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.592+0000 7f4ca9cac640 1 -- 172.21.15.88:0/1308763867 shutdown_connections 2024-09-16T01:40:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.592+0000 7f4ca9cac640 1 --2- 172.21.15.88:0/1308763867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c8c0051c0 0x7f4c8c00b9c0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.592+0000 7f4ca9cac640 1 -- 172.21.15.88:0/1308763867 >> 172.21.15.88:0/1308763867 conn(0x7f4ca40b3070 msgr2=0x7f4ca40b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:43.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.593+0000 7f4ca9cac640 1 -- 172.21.15.88:0/1308763867 shutdown_connections 2024-09-16T01:40:43.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.593+0000 7f4ca9cac640 1 -- 172.21.15.88:0/1308763867 wait complete. 2024-09-16T01:40:43.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.594+0000 7f4ca9cac640 1 Processor -- start 2024-09-16T01:40:43.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.594+0000 7f4ca9cac640 1 -- start start 2024-09-16T01:40:43.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.595+0000 7f4ca9cac640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c8c0051c0 0x7f4ca415fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:43.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.595+0000 7f4ca9cac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ca415e440 0x7f4ca40cf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:43.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.595+0000 7f4ca9cac640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ca414d2e0 0x7f4ca406cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:43.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.595+0000 7f4ca9cac640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca415ebf0 con 0x7f4c8c0051c0 2024-09-16T01:40:43.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.595+0000 7f4ca9cac640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca415f4c0 con 0x7f4ca415e440 2024-09-16T01:40:43.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.595+0000 7f4ca9cac640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4ca40b5f20 con 0x7f4ca414d2e0 2024-09-16T01:40:43.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.595+0000 7f4ca37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c8c0051c0 0x7f4ca415fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:43.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.595+0000 7f4ca37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c8c0051c0 0x7f4ca415fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55414/0 (socket says 172.21.15.88:55414) 2024-09-16T01:40:43.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.595+0000 7f4ca37fe640 1 -- 172.21.15.88:0/1243142696 learned_addr learned my addr 172.21.15.88:0/1243142696 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:43.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.595+0000 7f4ca2ffd640 1 --2- 172.21.15.88:0/1243142696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ca415e440 0x7f4ca40cf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:43.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.595+0000 7f4ca3fff640 1 --2- 172.21.15.88:0/1243142696 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ca414d2e0 0x7f4ca406cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:43.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.595+0000 7f4ca37fe640 1 -- 172.21.15.88:0/1243142696 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ca414d2e0 msgr2=0x7f4ca406cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:43.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.595+0000 7f4ca37fe640 1 --2- 172.21.15.88:0/1243142696 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ca414d2e0 0x7f4ca406cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:43.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.595+0000 7f4ca37fe640 1 -- 172.21.15.88:0/1243142696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ca415e440 msgr2=0x7f4ca40cf6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:43.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.595+0000 7f4ca37fe640 1 --2- 172.21.15.88:0/1243142696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ca415e440 0x7f4ca40cf6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:43.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.595+0000 7f4ca37fe640 1 -- 172.21.15.88:0/1243142696 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca40cfbf0 con 0x7f4c8c0051c0 2024-09-16T01:40:43.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.595+0000 7f4ca2ffd640 1 --2- 172.21.15.88:0/1243142696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ca415e440 0x7f4ca40cf6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:43.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.595+0000 7f4ca3fff640 1 --2- 172.21.15.88:0/1243142696 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ca414d2e0 0x7f4ca406cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:43.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.595+0000 7f4ca37fe640 1 --2- 172.21.15.88:0/1243142696 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c8c0051c0 0x7f4ca415fbf0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f4c94001ba0 tx=0x7f4c940077a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:43.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.595+0000 7f4c8bfff640 1 -- 172.21.15.88:0/1243142696 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9400f960 con 0x7f4c8c0051c0 2024-09-16T01:40:43.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.595+0000 7f4c8bfff640 1 -- 172.21.15.88:0/1243142696 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c94018070 con 0x7f4c8c0051c0 2024-09-16T01:40:43.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.595+0000 7f4ca9cac640 1 -- 172.21.15.88:0/1243142696 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ca41601d0 con 0x7f4c8c0051c0 2024-09-16T01:40:43.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.595+0000 7f4c8bfff640 1 -- 172.21.15.88:0/1243142696 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9400fd20 con 0x7f4c8c0051c0 2024-09-16T01:40:43.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.595+0000 7f4ca9cac640 1 -- 172.21.15.88:0/1243142696 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ca4160790 con 0x7f4c8c0051c0 2024-09-16T01:40:43.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.596+0000 7f4ca9cac640 1 -- 172.21.15.88:0/1243142696 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ca415f9e0 con 0x7f4c8c0051c0 2024-09-16T01:40:43.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.597+0000 7f4c8bfff640 1 -- 172.21.15.88:0/1243142696 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4c9401d460 con 0x7f4c8c0051c0 2024-09-16T01:40:43.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.597+0000 7f4c8bfff640 1 --2- 172.21.15.88:0/1243142696 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c9006c2b0 0x7f4c9006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:43.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.597+0000 7f4ca2ffd640 1 --2- 172.21.15.88:0/1243142696 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c9006c2b0 0x7f4c9006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:43.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.597+0000 7f4c8bfff640 1 -- 172.21.15.88:0/1243142696 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6139+0+0 (secure 0 0 0) 0x7f4c9408b640 con 0x7f4c8c0051c0 2024-09-16T01:40:43.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.598+0000 7f4ca2ffd640 1 --2- 172.21.15.88:0/1243142696 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c9006c2b0 0x7f4c9006e770 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f4c98000c00 tx=0x7f4c98009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:43.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.602+0000 7f4c8bfff640 1 -- 172.21.15.88:0/1243142696 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4c94059c90 con 0x7f4c8c0051c0 2024-09-16T01:40:43.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:43.805+0000 7f4ca9cac640 1 -- 172.21.15.88:0/1243142696 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f4ca4073a10 con 0x7f4c8c0051c0 2024-09-16T01:40:44.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.411+0000 7f4c8bfff640 1 -- 172.21.15.88:0/1243142696 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v98) ==== 161+0+0 (secure 0 0 0) 0x7f4c94059590 con 0x7f4c8c0051c0 2024-09-16T01:40:44.413 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:40:44.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.413+0000 7f4ca9cac640 1 -- 172.21.15.88:0/1243142696 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c9006c2b0 msgr2=0x7f4c9006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:44.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.413+0000 7f4ca9cac640 1 --2- 172.21.15.88:0/1243142696 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c9006c2b0 0x7f4c9006e770 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f4c98000c00 tx=0x7f4c98009040 comp rx=0 tx=0).stop 2024-09-16T01:40:44.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.413+0000 7f4ca9cac640 1 -- 172.21.15.88:0/1243142696 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c8c0051c0 msgr2=0x7f4ca415fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:44.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.413+0000 7f4ca9cac640 1 --2- 172.21.15.88:0/1243142696 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c8c0051c0 0x7f4ca415fbf0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f4c94001ba0 tx=0x7f4c940077a0 comp rx=0 tx=0).stop 2024-09-16T01:40:44.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.413+0000 7f4ca9cac640 1 -- 172.21.15.88:0/1243142696 shutdown_connections 2024-09-16T01:40:44.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.413+0000 7f4ca9cac640 1 --2- 172.21.15.88:0/1243142696 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c9006c2b0 0x7f4c9006e770 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:44.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.413+0000 7f4ca9cac640 1 --2- 172.21.15.88:0/1243142696 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ca414d2e0 0x7f4ca406cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:44.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.413+0000 7f4ca9cac640 1 --2- 172.21.15.88:0/1243142696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ca415e440 0x7f4ca40cf6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:44.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.413+0000 7f4ca9cac640 1 --2- 172.21.15.88:0/1243142696 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4c8c0051c0 0x7f4ca415fbf0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:44.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.413+0000 7f4ca9cac640 1 -- 172.21.15.88:0/1243142696 >> 172.21.15.88:0/1243142696 conn(0x7f4ca40b3070 msgr2=0x7f4ca40b5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:44.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.414+0000 7f4ca9cac640 1 -- 172.21.15.88:0/1243142696 shutdown_connections 2024-09-16T01:40:44.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.414+0000 7f4ca9cac640 1 -- 172.21.15.88:0/1243142696 wait complete. 2024-09-16T01:40:44.427 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:40:44.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.525+0000 7ff29c52a640 1 Processor -- start 2024-09-16T01:40:44.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.525+0000 7ff29c52a640 1 -- start start 2024-09-16T01:40:44.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.526+0000 7ff29c52a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff294072700 0x7ff294166120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:44.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.526+0000 7ff29c52a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2940ccff0 con 0x7ff29415b480 2024-09-16T01:40:44.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.526+0000 7ff29c52a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2940cd2d0 con 0x7ff2940b6290 2024-09-16T01:40:44.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.526+0000 7ff29c52a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff2940cd5b0 con 0x7ff294072700 2024-09-16T01:40:44.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.526+0000 7ff29aac7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff294072700 0x7ff294166120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:44.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.526+0000 7ff29a2c6640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff29415b480 0x7ff2940b5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59718/0 (socket says 172.21.15.88:59718) 2024-09-16T01:40:44.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.526+0000 7ff29aac7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff294072700 0x7ff294166120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55424/0 (socket says 172.21.15.88:55424) 2024-09-16T01:40:44.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.526+0000 7ff29a2c6640 1 -- 172.21.15.88:0/1988235121 learned_addr learned my addr 172.21.15.88:0/1988235121 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:44.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.527+0000 7ff29aac7640 1 -- 172.21.15.88:0/1988235121 >> v1:172.21.15.154:6789/0 conn(0x7ff2940b6290 legacy=0x7ff294071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:44.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.527+0000 7ff29aac7640 1 -- 172.21.15.88:0/1988235121 >> v1:172.21.15.88:6789/0 conn(0x7ff29415b480 legacy=0x7ff2940b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:44.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.527+0000 7ff29aac7640 1 -- 172.21.15.88:0/1988235121 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2940cd890 con 0x7ff294072700 2024-09-16T01:40:44.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.527+0000 7ff2992c4640 1 -- 172.21.15.88:0/1988235121 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1257090184 0 0) 0x7ff2940ccff0 con 0x7ff29415b480 2024-09-16T01:40:44.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.527+0000 7ff29aac7640 1 --2- 172.21.15.88:0/1988235121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff294072700 0x7ff294166120 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7ff28c009d90 tx=0x7ff28c02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=232f9fb80a7ffa94 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:44.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.527+0000 7ff2992c4640 1 -- 172.21.15.88:0/1988235121 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28c031600 con 0x7ff294072700 2024-09-16T01:40:44.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.527+0000 7ff2992c4640 1 -- 172.21.15.88:0/1988235121 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff28c0317a0 con 0x7ff294072700 2024-09-16T01:40:44.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.527+0000 7ff2992c4640 1 -- 172.21.15.88:0/1988235121 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28c031b70 con 0x7ff294072700 2024-09-16T01:40:44.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.528+0000 7ff29c52a640 1 -- 172.21.15.88:0/1988235121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff294072700 msgr2=0x7ff294166120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:44.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.528+0000 7ff29c52a640 1 --2- 172.21.15.88:0/1988235121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff294072700 0x7ff294166120 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7ff28c009d90 tx=0x7ff28c02a610 comp rx=0 tx=0).stop 2024-09-16T01:40:44.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.528+0000 7ff29c52a640 1 -- 172.21.15.88:0/1988235121 shutdown_connections 2024-09-16T01:40:44.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.528+0000 7ff29c52a640 1 --2- 172.21.15.88:0/1988235121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff294072700 0x7ff294166120 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:44.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.528+0000 7ff29c52a640 1 -- 172.21.15.88:0/1988235121 >> 172.21.15.88:0/1988235121 conn(0x7ff29406d0f0 msgr2=0x7ff29406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:44.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.528+0000 7ff29c52a640 1 -- 172.21.15.88:0/1988235121 shutdown_connections 2024-09-16T01:40:44.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.528+0000 7ff29c52a640 1 -- 172.21.15.88:0/1988235121 wait complete. 2024-09-16T01:40:44.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.529+0000 7ff29c52a640 1 Processor -- start 2024-09-16T01:40:44.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.529+0000 7ff29c52a640 1 -- start start 2024-09-16T01:40:44.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.530+0000 7ff29c52a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff294072700 0x7ff294146080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:44.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.530+0000 7ff29c52a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff29415b480 0x7ff2941537b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:44.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.530+0000 7ff29c52a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff29414e660 0x7ff294150b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:44.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.530+0000 7ff29c52a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff29415b9e0 con 0x7ff29415b480 2024-09-16T01:40:44.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.530+0000 7ff29c52a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff29415b860 con 0x7ff294072700 2024-09-16T01:40:44.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.530+0000 7ff29c52a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff29415cc70 con 0x7ff29414e660 2024-09-16T01:40:44.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.530+0000 7ff299ac5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff29415b480 0x7ff2941537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:44.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.530+0000 7ff299ac5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff29415b480 0x7ff2941537b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55434/0 (socket says 172.21.15.88:55434) 2024-09-16T01:40:44.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.530+0000 7ff299ac5640 1 -- 172.21.15.88:0/1865317271 learned_addr learned my addr 172.21.15.88:0/1865317271 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:44.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.530+0000 7ff299ac5640 1 -- 172.21.15.88:0/1865317271 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff29414e660 msgr2=0x7ff294150b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:44.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.530+0000 7ff299ac5640 1 --2- 172.21.15.88:0/1865317271 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff29414e660 0x7ff294150b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:44.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.530+0000 7ff299ac5640 1 -- 172.21.15.88:0/1865317271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff294072700 msgr2=0x7ff294146080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:44.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.530+0000 7ff299ac5640 1 --2- 172.21.15.88:0/1865317271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff294072700 0x7ff294146080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:44.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.530+0000 7ff299ac5640 1 -- 172.21.15.88:0/1865317271 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff294155150 con 0x7ff29415b480 2024-09-16T01:40:44.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.531+0000 7ff299ac5640 1 --2- 172.21.15.88:0/1865317271 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff29415b480 0x7ff2941537b0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7ff28400c400 tx=0x7ff28400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:44.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.531+0000 7ff28affd640 1 -- 172.21.15.88:0/1865317271 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff284011790 con 0x7ff29415b480 2024-09-16T01:40:44.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.531+0000 7ff28affd640 1 -- 172.21.15.88:0/1865317271 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff284017070 con 0x7ff29415b480 2024-09-16T01:40:44.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.531+0000 7ff28affd640 1 -- 172.21.15.88:0/1865317271 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff284011bb0 con 0x7ff29415b480 2024-09-16T01:40:44.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.531+0000 7ff29c52a640 1 -- 172.21.15.88:0/1865317271 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2941540b0 con 0x7ff29415b480 2024-09-16T01:40:44.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.531+0000 7ff29c52a640 1 -- 172.21.15.88:0/1865317271 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff294154670 con 0x7ff29415b480 2024-09-16T01:40:44.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.532+0000 7ff29c52a640 1 -- 172.21.15.88:0/1865317271 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff294071ff0 con 0x7ff29415b480 2024-09-16T01:40:44.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.532+0000 7ff28affd640 1 -- 172.21.15.88:0/1865317271 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff28401c460 con 0x7ff29415b480 2024-09-16T01:40:44.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.533+0000 7ff28affd640 1 --2- 172.21.15.88:0/1865317271 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff27806c2b0 0x7ff27806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:44.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.533+0000 7ff28affd640 1 -- 172.21.15.88:0/1865317271 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6139+0+0 (secure 0 0 0) 0x7ff28408b4b0 con 0x7ff29415b480 2024-09-16T01:40:44.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.533+0000 7ff29a2c6640 1 --2- 172.21.15.88:0/1865317271 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff27806c2b0 0x7ff27806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:44.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.533+0000 7ff29a2c6640 1 --2- 172.21.15.88:0/1865317271 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff27806c2b0 0x7ff27806e770 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff290007fd0 tx=0x7ff290007960 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:44.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.537+0000 7ff28affd640 1 -- 172.21.15.88:0/1865317271 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff284059a70 con 0x7ff29415b480 2024-09-16T01:40:44.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:44.769+0000 7ff29c52a640 1 -- 172.21.15.88:0/1865317271 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7ff2940b3bf0 con 0x7ff29415b480 2024-09-16T01:40:45.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.413+0000 7ff28affd640 1 -- 172.21.15.88:0/1865317271 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v99) ==== 101+0+0 (secure 0 0 0) 0x7ff284059370 con 0x7ff29415b480 2024-09-16T01:40:45.414 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:40:45.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.415+0000 7ff29c52a640 1 -- 172.21.15.88:0/1865317271 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff27806c2b0 msgr2=0x7ff27806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:45.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.415+0000 7ff29c52a640 1 --2- 172.21.15.88:0/1865317271 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff27806c2b0 0x7ff27806e770 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff290007fd0 tx=0x7ff290007960 comp rx=0 tx=0).stop 2024-09-16T01:40:45.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.415+0000 7ff29c52a640 1 -- 172.21.15.88:0/1865317271 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff29415b480 msgr2=0x7ff2941537b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:45.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.415+0000 7ff29c52a640 1 --2- 172.21.15.88:0/1865317271 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff29415b480 0x7ff2941537b0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7ff28400c400 tx=0x7ff28400c8c0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.415+0000 7ff29c52a640 1 -- 172.21.15.88:0/1865317271 shutdown_connections 2024-09-16T01:40:45.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.415+0000 7ff29c52a640 1 --2- 172.21.15.88:0/1865317271 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff27806c2b0 0x7ff27806e770 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.415+0000 7ff29c52a640 1 --2- 172.21.15.88:0/1865317271 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff29414e660 0x7ff294150b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.415+0000 7ff29c52a640 1 --2- 172.21.15.88:0/1865317271 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff29415b480 0x7ff2941537b0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.415+0000 7ff29c52a640 1 --2- 172.21.15.88:0/1865317271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff294072700 0x7ff294146080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.415+0000 7ff29c52a640 1 -- 172.21.15.88:0/1865317271 >> 172.21.15.88:0/1865317271 conn(0x7ff29406d0f0 msgr2=0x7ff29415bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:45.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.415+0000 7ff29c52a640 1 -- 172.21.15.88:0/1865317271 shutdown_connections 2024-09-16T01:40:45.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.415+0000 7ff29c52a640 1 -- 172.21.15.88:0/1865317271 wait complete. 2024-09-16T01:40:45.429 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:40:45.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.472+0000 7fd42031df40 1 Processor -- start 2024-09-16T01:40:45.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.472+0000 7fd42031df40 1 -- start start 2024-09-16T01:40:45.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.473+0000 7fd42031df40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55aca886cf00 0x55aca8939bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.473+0000 7fd42031df40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aca8922ed0 0x55aca893f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.473+0000 7fd42031df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca8925190 0x55aca893f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.473+0000 7fd42031df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55aca87bc510 con 0x55aca8925190 2024-09-16T01:40:45.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.473+0000 7fd42031df40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55aca8924210 con 0x55aca8922ed0 2024-09-16T01:40:45.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.473+0000 7fd42031df40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55aca893fce0 con 0x55aca886cf00 2024-09-16T01:40:45.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.473+0000 7fd41db66640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca8925190 0x55aca893f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.473+0000 7fd41db66640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca8925190 0x55aca893f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55458/0 (socket says 172.21.15.88:55458) 2024-09-16T01:40:45.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.473+0000 7fd41db66640 1 -- 172.21.15.88:0/990399647 learned_addr learned my addr 172.21.15.88:0/990399647 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:45.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.473+0000 7fd41cb64640 1 --2- 172.21.15.88:0/990399647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aca8922ed0 0x55aca893f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.473+0000 7fd41d365640 1 --2- 172.21.15.88:0/990399647 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55aca886cf00 0x55aca8939bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.473+0000 7fd41db66640 1 -- 172.21.15.88:0/990399647 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55aca886cf00 msgr2=0x55aca8939bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:45.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.473+0000 7fd41db66640 1 --2- 172.21.15.88:0/990399647 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55aca886cf00 0x55aca8939bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.474+0000 7fd41db66640 1 -- 172.21.15.88:0/990399647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aca8922ed0 msgr2=0x55aca893f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:45.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.474+0000 7fd41db66640 1 --2- 172.21.15.88:0/990399647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aca8922ed0 0x55aca893f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.474+0000 7fd41db66640 1 -- 172.21.15.88:0/990399647 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aca8927fb0 con 0x55aca8925190 2024-09-16T01:40:45.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.474+0000 7fd41d365640 1 --2- 172.21.15.88:0/990399647 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55aca886cf00 0x55aca8939bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:45.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.474+0000 7fd41db66640 1 --2- 172.21.15.88:0/990399647 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca8925190 0x55aca893f5f0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fd418001580 tx=0x7fd418001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4fb62a2d427677c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:45.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.474+0000 7fd40f7fe640 1 -- 172.21.15.88:0/990399647 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41800f710 con 0x55aca8925190 2024-09-16T01:40:45.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.474+0000 7fd40f7fe640 1 -- 172.21.15.88:0/990399647 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd41800f8b0 con 0x55aca8925190 2024-09-16T01:40:45.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.474+0000 7fd40f7fe640 1 -- 172.21.15.88:0/990399647 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41800fca0 con 0x55aca8925190 2024-09-16T01:40:45.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.474+0000 7fd42031df40 1 -- 172.21.15.88:0/990399647 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca8925190 msgr2=0x55aca893f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:45.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.474+0000 7fd42031df40 1 --2- 172.21.15.88:0/990399647 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca8925190 0x55aca893f5f0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fd418001580 tx=0x7fd418001a40 comp rx=0 tx=0).stop 2024-09-16T01:40:45.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.474+0000 7fd42031df40 1 -- 172.21.15.88:0/990399647 shutdown_connections 2024-09-16T01:40:45.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.474+0000 7fd42031df40 1 --2- 172.21.15.88:0/990399647 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca8925190 0x55aca893f5f0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.474+0000 7fd42031df40 1 --2- 172.21.15.88:0/990399647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aca8922ed0 0x55aca893f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.474+0000 7fd42031df40 1 --2- 172.21.15.88:0/990399647 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55aca886cf00 0x55aca8939bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.474+0000 7fd42031df40 1 -- 172.21.15.88:0/990399647 >> 172.21.15.88:0/990399647 conn(0x55aca8869be0 msgr2=0x55aca88b5610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:45.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.474+0000 7fd42031df40 1 -- 172.21.15.88:0/990399647 shutdown_connections 2024-09-16T01:40:45.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.474+0000 7fd42031df40 1 -- 172.21.15.88:0/990399647 wait complete. 2024-09-16T01:40:45.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.475+0000 7fd42031df40 1 Processor -- start 2024-09-16T01:40:45.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.475+0000 7fd42031df40 1 -- start start 2024-09-16T01:40:45.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.476+0000 7fd42031df40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aca886cf00 0x55aca89d85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.476+0000 7fd42031df40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55aca8922ed0 0x55aca891f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.476+0000 7fd42031df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca8925190 0x55aca89381c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.476+0000 7fd42031df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55aca89226c0 con 0x55aca8925190 2024-09-16T01:40:45.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.476+0000 7fd42031df40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55aca8928c60 con 0x55aca886cf00 2024-09-16T01:40:45.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.476+0000 7fd42031df40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55aca893a610 con 0x55aca8922ed0 2024-09-16T01:40:45.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.476+0000 7fd41db66640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca8925190 0x55aca89381c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.476+0000 7fd41db66640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca8925190 0x55aca89381c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55460/0 (socket says 172.21.15.88:55460) 2024-09-16T01:40:45.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.476+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 learned_addr learned my addr 172.21.15.88:0/1182002894 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:45.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.476+0000 7fd41d365640 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aca886cf00 0x55aca89d85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.476+0000 7fd41cb64640 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55aca8922ed0 0x55aca891f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.476+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55aca8922ed0 msgr2=0x55aca891f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:45.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.476+0000 7fd41db66640 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55aca8922ed0 0x55aca891f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.476+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aca886cf00 msgr2=0x55aca89d85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:45.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.476+0000 7fd41db66640 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aca886cf00 0x55aca89d85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.476+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aca893ff30 con 0x55aca8925190 2024-09-16T01:40:45.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.476+0000 7fd41db66640 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca8925190 0x55aca89381c0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fd418001f40 tx=0x7fd418001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:45.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.477+0000 7fd40e7fc640 1 -- 172.21.15.88:0/1182002894 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd418022020 con 0x55aca8925190 2024-09-16T01:40:45.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.477+0000 7fd40e7fc640 1 -- 172.21.15.88:0/1182002894 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd41801e880 con 0x55aca8925190 2024-09-16T01:40:45.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.477+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55aca8927940 con 0x55aca8925190 2024-09-16T01:40:45.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.477+0000 7fd40e7fc640 1 -- 172.21.15.88:0/1182002894 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41801ec70 con 0x55aca8925190 2024-09-16T01:40:45.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.477+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aca88b1fe0 con 0x55aca8925190 2024-09-16T01:40:45.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.478+0000 7fd40e7fc640 1 -- 172.21.15.88:0/1182002894 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd418025020 con 0x55aca8925190 2024-09-16T01:40:45.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.478+0000 7fd40e7fc640 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd40006ac90 0x7fd40006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.478+0000 7fd40e7fc640 1 -- 172.21.15.88:0/1182002894 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6139+0+0 (secure 0 0 0) 0x7fd41808c290 con 0x55aca8925190 2024-09-16T01:40:45.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.478+0000 7fd41d365640 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd40006ac90 0x7fd40006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.485 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:40:45.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.479+0000 7fd42031df40 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55aca88b3fd0 0x55aca88b1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.479+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca88b15a0 con 0x55aca88b3fd0 2024-09-16T01:40:45.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.479+0000 7fd41cb64640 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55aca88b3fd0 0x55aca88b1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.479+0000 7fd41d365640 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd40006ac90 0x7fd40006d150 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd4100057f0 tx=0x7fd41000d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:45.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.479+0000 7fd41cb64640 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55aca88b3fd0 0x55aca88b1060 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:45.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.481+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v90'133 uv107 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.481+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e90) -- 0x55aca88b19a0 con 0x55aca88b3fd0 2024-09-16T01:40:45.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.482+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v90'134 uv134 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.482+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e5190 con 0x55aca88b3fd0 2024-09-16T01:40:45.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.483+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v90'135 uv135 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.487 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:40:45.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.483+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e6540 con 0x55aca88b3fd0 2024-09-16T01:40:45.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.484+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v90'136 uv136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.484+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e68e0 con 0x55aca88b3fd0 2024-09-16T01:40:45.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.485+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v90'137 uv137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.485+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e5530 con 0x55aca88b3fd0 2024-09-16T01:40:45.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.486+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v90'138 uv138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.486+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e58d0 con 0x55aca88b3fd0 2024-09-16T01:40:45.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.487+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v90'139 uv113 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.487+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e5cc0 con 0x55aca88b3fd0 2024-09-16T01:40:45.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.488+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v90'140 uv140 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.488+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e5cc0 con 0x55aca88b3fd0 2024-09-16T01:40:45.490 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:40:45.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.489+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v90'141 uv141 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.489+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e5c70 con 0x55aca88b3fd0 2024-09-16T01:40:45.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.490+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v90'142 uv142 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.490+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e5c70 con 0x55aca88b3fd0 2024-09-16T01:40:45.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.491+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v90'143 uv143 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.491+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e5c70 con 0x55aca88b3fd0 2024-09-16T01:40:45.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.492+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v90'144 uv144 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.492+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e5c70 con 0x55aca88b3fd0 2024-09-16T01:40:45.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.493+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v90'145 uv120 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.493+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e5c70 con 0x55aca88b3fd0 2024-09-16T01:40:45.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.494+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v90'146 uv146 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.494+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e5c70 con 0x55aca88b3fd0 2024-09-16T01:40:45.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.496+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v90'147 uv147 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.497 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:40:45.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.496+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e5c70 con 0x55aca88b3fd0 2024-09-16T01:40:45.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.497+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v90'148 uv148 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.497+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e5c70 con 0x55aca88b3fd0 2024-09-16T01:40:45.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.498+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v90'149 uv149 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.498+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e5c70 con 0x55aca88b3fd0 2024-09-16T01:40:45.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.499+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v90'150 uv150 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.499+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e6c80 con 0x55aca88b3fd0 2024-09-16T01:40:45.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.500+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v90'151 uv151 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.500+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e6c80 con 0x55aca88b3fd0 2024-09-16T01:40:45.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.501+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v90'152 uv126 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.501+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e6c80 con 0x55aca88b3fd0 2024-09-16T01:40:45.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.502+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v90'153 uv153 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.502+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e6c80 con 0x55aca88b3fd0 2024-09-16T01:40:45.504 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:40:45.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.503+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v90'154 uv154 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.503+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e7c90 con 0x55aca88b3fd0 2024-09-16T01:40:45.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.504+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v90'155 uv155 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.504+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e7c90 con 0x55aca88b3fd0 2024-09-16T01:40:45.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.505+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v90'156 uv156 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.505+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e6c80 con 0x55aca88b3fd0 2024-09-16T01:40:45.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.506+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v90'157 uv157 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.506+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e6c80 con 0x55aca88b3fd0 2024-09-16T01:40:45.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.507+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v90'158 uv132 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.507+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e6c80 con 0x55aca88b3fd0 2024-09-16T01:40:45.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.508+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v90'159 uv159 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.508+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e6c80 con 0x55aca88b3fd0 2024-09-16T01:40:45.510 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:40:45.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.508+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v90'160 uv160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.509+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e7c90 con 0x55aca88b3fd0 2024-09-16T01:40:45.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.509+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v90'161 uv161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.509+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e7c90 con 0x55aca88b3fd0 2024-09-16T01:40:45.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.510+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v90'162 uv162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.511+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e6c80 con 0x55aca88b3fd0 2024-09-16T01:40:45.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.512+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v90'163 uv163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.512+0000 7fd42031df40 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55aca89e7190 0x55aca89e9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.512+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e9ac0 con 0x55aca89e7190 2024-09-16T01:40:45.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.512+0000 7fd41d365640 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55aca89e7190 0x55aca89e9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.512+0000 7fd41d365640 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55aca89e7190 0x55aca89e9580 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:45.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.514+0000 7fd41d365640 1 -- 172.21.15.88:0/1182002894 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v90'110 uv92 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fd410007930 con 0x55aca89e7190 2024-09-16T01:40:45.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.514+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e90) -- 0x55aca89e9e60 con 0x55aca89e7190 2024-09-16T01:40:45.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.515+0000 7fd41d365640 1 -- 172.21.15.88:0/1182002894 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v90'111 uv111 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd410007930 con 0x55aca89e7190 2024-09-16T01:40:45.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.515+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89ea200 con 0x55aca89e7190 2024-09-16T01:40:45.517 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:40:45.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.516+0000 7fd41d365640 1 -- 172.21.15.88:0/1182002894 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v90'112 uv112 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd410007930 con 0x55aca89e7190 2024-09-16T01:40:45.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.516+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89eaa50 con 0x55aca89e7190 2024-09-16T01:40:45.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.519+0000 7fd41d365640 1 -- 172.21.15.88:0/1182002894 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v90'113 uv113 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd410007930 con 0x55aca89e7190 2024-09-16T01:40:45.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.519+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89eadf0 con 0x55aca89e7190 2024-09-16T01:40:45.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.520+0000 7fd41d365640 1 -- 172.21.15.88:0/1182002894 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v90'114 uv114 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd410007930 con 0x55aca89e7190 2024-09-16T01:40:45.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.520+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89eb190 con 0x55aca89e7190 2024-09-16T01:40:45.522 INFO:teuthology.orchestra.run.smithi088.stdout:Write #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:40:45.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.521+0000 7fd41d365640 1 -- 172.21.15.88:0/1182002894 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v90'115 uv115 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd410007930 con 0x55aca89e7190 2024-09-16T01:40:45.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.521+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89eb530 con 0x55aca89e7190 2024-09-16T01:40:45.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.523+0000 7fd41d365640 1 -- 172.21.15.88:0/1182002894 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [write 0~90] v90'116 uv116 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd410007930 con 0x55aca89e7190 2024-09-16T01:40:45.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.523+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89eb8d0 con 0x55aca89e7190 2024-09-16T01:40:45.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.524+0000 7fd41d365640 1 -- 172.21.15.88:0/1182002894 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [omap-set-header] v90'117 uv117 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd410007930 con 0x55aca89e7190 2024-09-16T01:40:45.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.524+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89eb8d0 con 0x55aca89e7190 2024-09-16T01:40:45.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.525+0000 7fd41d365640 1 -- 172.21.15.88:0/1182002894 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000001 [create] v90'118 uv102 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd410007930 con 0x55aca89e7190 2024-09-16T01:40:45.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.525+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e90) -- 0x55aca89eb8d0 con 0x55aca89e7190 2024-09-16T01:40:45.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.526+0000 7fd41d365640 1 -- 172.21.15.88:0/1182002894 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000001 [delete] v90'119 uv119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd410007930 con 0x55aca89e7190 2024-09-16T01:40:45.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.526+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89eb8d0 con 0x55aca89e7190 2024-09-16T01:40:45.528 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:40:45.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.527+0000 7fd41d365640 1 -- 172.21.15.88:0/1182002894 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v90'120 uv120 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd410007930 con 0x55aca89e7190 2024-09-16T01:40:45.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.527+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89efd50 con 0x55aca89e7190 2024-09-16T01:40:45.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.528+0000 7fd41d365640 1 -- 172.21.15.88:0/1182002894 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [write 0~17490] v90'121 uv121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd410007930 con 0x55aca89e7190 2024-09-16T01:40:45.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.528+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89efd50 con 0x55aca89e7190 2024-09-16T01:40:45.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.530+0000 7fd41d365640 1 -- 172.21.15.88:0/1182002894 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [omap-set-header] v90'122 uv122 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd410007930 con 0x55aca89e7190 2024-09-16T01:40:45.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.530+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89eb8d0 con 0x55aca89e7190 2024-09-16T01:40:45.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.531+0000 7fd41d365640 1 -- 172.21.15.88:0/1182002894 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 606.00000000 [create] v90'123 uv108 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd410007930 con 0x55aca89e7190 2024-09-16T01:40:45.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.531+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e90) -- 0x55aca89eb8d0 con 0x55aca89e7190 2024-09-16T01:40:45.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.532+0000 7fd41d365640 1 -- 172.21.15.88:0/1182002894 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 606.00000000 [delete] v90'124 uv124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd410007930 con 0x55aca89e7190 2024-09-16T01:40:45.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.532+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89eb8d0 con 0x55aca89e7190 2024-09-16T01:40:45.535 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:40:45.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.533+0000 7fd41d365640 1 -- 172.21.15.88:0/1182002894 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v90'125 uv125 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd410007930 con 0x55aca89e7190 2024-09-16T01:40:45.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.534+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89ec8e0 con 0x55aca89e7190 2024-09-16T01:40:45.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.536+0000 7fd41d365640 1 -- 172.21.15.88:0/1182002894 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [setxattr (30)] v90'126 uv126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd410007930 con 0x55aca89e7190 2024-09-16T01:40:45.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.536+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89ec8e0 con 0x55aca89e7190 2024-09-16T01:40:45.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.539+0000 7fd41d365640 1 -- 172.21.15.88:0/1182002894 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [setxattr (62)] v90'127 uv127 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd410007930 con 0x55aca89e7190 2024-09-16T01:40:45.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.539+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89eb8d0 con 0x55aca89e7190 2024-09-16T01:40:45.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.542+0000 7fd41d365640 1 -- 172.21.15.88:0/1182002894 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [omap-set-header] v90'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd410007930 con 0x55aca89e7190 2024-09-16T01:40:45.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.542+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:51 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89eb8d0 con 0x55aca88b3fd0 2024-09-16T01:40:45.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.543+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(51 607.00000000 [create] v90'27 uv26 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.543+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:52 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e90) -- 0x55aca89eb8d0 con 0x55aca88b3fd0 2024-09-16T01:40:45.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.544+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(52 607.00000000 [delete] v90'28 uv28 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.544+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89eb8d0 con 0x55aca88b3fd0 2024-09-16T01:40:45.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.545+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(53 607.00000000 [create] v90'29 uv29 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.546 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:40:45.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.545+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89ec8e0 con 0x55aca88b3fd0 2024-09-16T01:40:45.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.546+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(54 607.00000000 [setxattr (30)] v90'30 uv30 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.546+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89ec8e0 con 0x55aca88b3fd0 2024-09-16T01:40:45.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.546+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(55 607.00000000 [setxattr (62)] v90'31 uv31 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.546+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89eb8d0 con 0x55aca88b3fd0 2024-09-16T01:40:45.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.547+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(56 607.00000000 [omap-set-header] v90'32 uv32 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.547+0000 7fd42031df40 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55aca89ebde0 0x55aca89ee1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.547+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:57 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89ee710 con 0x55aca89ebde0 2024-09-16T01:40:45.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.547+0000 7fd41cb64640 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55aca89ebde0 0x55aca89ee1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.548+0000 7fd41cb64640 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55aca89ebde0 0x55aca89ee1d0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:45.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.549+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(57 mds0_openfiles.0 [create] v90'22 uv21 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca89ebde0 2024-09-16T01:40:45.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.549+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:58 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e90) -- 0x55aca89ee710 con 0x55aca89ebde0 2024-09-16T01:40:45.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.550+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(58 mds0_openfiles.0 [delete] v90'23 uv23 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca89ebde0 2024-09-16T01:40:45.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.550+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89ee710 con 0x55aca89ebde0 2024-09-16T01:40:45.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.550+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(59 mds0_openfiles.0 [create] v90'24 uv24 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca89ebde0 2024-09-16T01:40:45.552 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:40:45.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.550+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89ee710 con 0x55aca89ebde0 2024-09-16T01:40:45.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.551+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(60 mds0_openfiles.0 [omap-set-header] v90'25 uv25 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca89ebde0 2024-09-16T01:40:45.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.551+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:61 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89ee710 con 0x55aca88b3fd0 2024-09-16T01:40:45.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.553+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(61 608.00000000 [create] v90'27 uv26 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.553+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:62 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e90) -- 0x55aca89ee710 con 0x55aca88b3fd0 2024-09-16T01:40:45.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.554+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(62 608.00000000 [delete] v90'28 uv28 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.554+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89ee710 con 0x55aca88b3fd0 2024-09-16T01:40:45.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.555+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(63 608.00000000 [create] v90'29 uv29 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.556 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:40:45.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.555+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89ef720 con 0x55aca88b3fd0 2024-09-16T01:40:45.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.558+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(64 608.00000000 [setxattr (30)] v90'30 uv30 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.558+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89ef720 con 0x55aca88b3fd0 2024-09-16T01:40:45.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.560+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(65 608.00000000 [setxattr (62)] v90'31 uv31 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.560+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89ee710 con 0x55aca88b3fd0 2024-09-16T01:40:45.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.563+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(66 608.00000000 [omap-set-header] v90'32 uv32 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.563+0000 7fd42031df40 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55aca89eefe0 0x55aca89ef3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.563+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:67 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f1b10 con 0x55aca89eefe0 2024-09-16T01:40:45.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.563+0000 7fd41db66640 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55aca89eefe0 0x55aca89ef3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.564+0000 7fd41db66640 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55aca89eefe0 0x55aca89ef3c0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:45.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.565+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(67 604.00000000 [create] v90'48 uv42 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.565+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:68 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f1eb0 con 0x55aca89eefe0 2024-09-16T01:40:45.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.566+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(68 604.00000000 [delete] v90'49 uv49 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.566+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f2250 con 0x55aca89eefe0 2024-09-16T01:40:45.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.567+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(69 604.00000000 [create] v90'50 uv50 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.568 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:40:45.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.567+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f3ab0 con 0x55aca89eefe0 2024-09-16T01:40:45.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.568+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(70 604.00000000 [setxattr (30)] v90'51 uv51 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.568+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f3e50 con 0x55aca89eefe0 2024-09-16T01:40:45.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.569+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(71 604.00000000 [setxattr (62)] v90'52 uv52 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.569+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f2aa0 con 0x55aca89eefe0 2024-09-16T01:40:45.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.570+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(72 604.00000000 [omap-set-header] v90'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.570+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f2e90 con 0x55aca89eefe0 2024-09-16T01:40:45.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.571+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(73 500.00000000 [create] v90'54 uv47 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.571+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f3280 con 0x55aca89eefe0 2024-09-16T01:40:45.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.572+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(74 500.00000000 [delete] v90'55 uv55 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.572+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f3280 con 0x55aca89eefe0 2024-09-16T01:40:45.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.575+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(75 500.00000000 [create] v90'56 uv56 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.576 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:40:45.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.575+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f3230 con 0x55aca89eefe0 2024-09-16T01:40:45.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.577+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(76 500.00000000 [write 0~90] v90'57 uv57 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.577+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f3230 con 0x55aca89eefe0 2024-09-16T01:40:45.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.580+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(77 500.00000000 [omap-set-header] v90'58 uv58 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.580+0000 7fd42031df40 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55aca89f35f0 0x55aca89f6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.580+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:78 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f6c50 con 0x55aca89f35f0 2024-09-16T01:40:45.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.580+0000 7fd41d365640 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55aca89f35f0 0x55aca89f6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.581+0000 7fd41d365640 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55aca89f35f0 0x55aca89f6710 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:45.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.582+0000 7fd41d365640 1 -- 172.21.15.88:0/1182002894 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(78 mds_snaptable [create] v90'22 uv21 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fd410007930 con 0x55aca89f35f0 2024-09-16T01:40:45.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.582+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:79 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f6c50 con 0x55aca89f35f0 2024-09-16T01:40:45.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.583+0000 7fd41d365640 1 -- 172.21.15.88:0/1182002894 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(79 mds_snaptable [delete] v90'23 uv23 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd410007930 con 0x55aca89f35f0 2024-09-16T01:40:45.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.583+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f6c50 con 0x55aca89f35f0 2024-09-16T01:40:45.585 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:40:45.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.584+0000 7fd41d365640 1 -- 172.21.15.88:0/1182002894 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(80 mds_snaptable [create] v90'24 uv24 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd410007930 con 0x55aca89f35f0 2024-09-16T01:40:45.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.584+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f6c50 con 0x55aca89f35f0 2024-09-16T01:40:45.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.585+0000 7fd41d365640 1 -- 172.21.15.88:0/1182002894 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(81 mds_snaptable [write 0~70] v90'25 uv25 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd410007930 con 0x55aca89f35f0 2024-09-16T01:40:45.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.585+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f6c50 con 0x55aca89f35f0 2024-09-16T01:40:45.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.586+0000 7fd41d365640 1 -- 172.21.15.88:0/1182002894 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(82 mds_snaptable [omap-set-header] v90'26 uv26 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd410007930 con 0x55aca89f35f0 2024-09-16T01:40:45.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.586+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:83 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f6c50 con 0x55aca88b3fd0 2024-09-16T01:40:45.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.587+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(83 605.00000000 [create] v90'27 uv26 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.587+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:84 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f6c50 con 0x55aca88b3fd0 2024-09-16T01:40:45.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.588+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(84 605.00000000 [delete] v90'28 uv28 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.588+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f6c50 con 0x55aca88b3fd0 2024-09-16T01:40:45.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.591+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(85 605.00000000 [create] v90'29 uv29 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.592 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:40:45.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.591+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f80a0 con 0x55aca88b3fd0 2024-09-16T01:40:45.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.592+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(86 605.00000000 [setxattr (30)] v90'30 uv30 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.592+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f80a0 con 0x55aca88b3fd0 2024-09-16T01:40:45.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.597+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(87 605.00000000 [setxattr (62)] v90'31 uv31 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.597+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca88b3fd0 2024-09-16T01:40:45.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.597+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(88 605.00000000 [omap-set-header] v90'32 uv32 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca88b3fd0 2024-09-16T01:40:45.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.598+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:89 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89ebde0 2024-09-16T01:40:45.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.600+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(89 mds0_inotable [create] v90'44 uv38 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca89ebde0 2024-09-16T01:40:45.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.600+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:90 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89ebde0 2024-09-16T01:40:45.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.601+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(90 mds0_inotable [delete] v90'45 uv45 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca89ebde0 2024-09-16T01:40:45.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.601+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89ebde0 2024-09-16T01:40:45.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.602+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(91 mds0_inotable [create] v90'46 uv46 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca89ebde0 2024-09-16T01:40:45.603 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:40:45.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.602+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89ebde0 2024-09-16T01:40:45.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.603+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(92 mds0_inotable [write 0~34] v90'47 uv47 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca89ebde0 2024-09-16T01:40:45.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.603+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89ebde0 2024-09-16T01:40:45.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.604+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(93 mds0_inotable [omap-set-header] v90'48 uv48 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca89ebde0 2024-09-16T01:40:45.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.604+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89ebde0 2024-09-16T01:40:45.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.605+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(94 100.00000000 [create] v90'49 uv43 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca89ebde0 2024-09-16T01:40:45.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.605+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89ebde0 2024-09-16T01:40:45.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.606+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(95 100.00000000 [delete] v90'50 uv50 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca89ebde0 2024-09-16T01:40:45.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.606+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89ebde0 2024-09-16T01:40:45.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.607+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(96 100.00000000 [create] v90'51 uv51 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca89ebde0 2024-09-16T01:40:45.608 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:40:45.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.607+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89ebde0 2024-09-16T01:40:45.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.608+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(97 100.00000000 [omap-set-header] v90'52 uv52 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca89ebde0 2024-09-16T01:40:45.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.608+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f9e50 con 0x55aca89ebde0 2024-09-16T01:40:45.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.609+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v90'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca89ebde0 2024-09-16T01:40:45.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.609+0000 7fd42031df40 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55aca89f7a00 0x55aca89fa2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.609+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:99 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f9df0 con 0x55aca89f7a00 2024-09-16T01:40:45.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.609+0000 7fd41cb64640 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55aca89f7a00 0x55aca89fa2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.609+0000 7fd41cb64640 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55aca89f7a00 0x55aca89fa2f0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:45.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.610+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(99 mds0_sessionmap [create] v90'22 uv21 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca89f7a00 2024-09-16T01:40:45.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.610+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:100 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89f7a00 2024-09-16T01:40:45.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.611+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v90'23 uv23 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca89f7a00 2024-09-16T01:40:45.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.611+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89f7a00 2024-09-16T01:40:45.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.614+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(101 mds0_sessionmap [create] v90'24 uv24 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca89f7a00 2024-09-16T01:40:45.615 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:40:45.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.614+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89f7a00 2024-09-16T01:40:45.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.616+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v90'25 uv25 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca89f7a00 2024-09-16T01:40:45.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.616+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89ebde0 2024-09-16T01:40:45.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.618+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(103 609.00000000 [create] v90'27 uv26 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca89ebde0 2024-09-16T01:40:45.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.618+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89ebde0 2024-09-16T01:40:45.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.619+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(104 609.00000000 [delete] v90'28 uv28 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca89ebde0 2024-09-16T01:40:45.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.619+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89ebde0 2024-09-16T01:40:45.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.620+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(105 609.00000000 [create] v90'29 uv29 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca89ebde0 2024-09-16T01:40:45.621 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:40:45.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.620+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89ebde0 2024-09-16T01:40:45.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.621+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v90'30 uv30 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca89ebde0 2024-09-16T01:40:45.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.621+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89ebde0 2024-09-16T01:40:45.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.622+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v90'31 uv31 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca89ebde0 2024-09-16T01:40:45.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.622+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89ebde0 2024-09-16T01:40:45.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.623+0000 7fd41cb64640 1 -- 172.21.15.88:0/1182002894 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(108 609.00000000 [omap-set-header] v90'32 uv32 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd414050050 con 0x55aca89ebde0 2024-09-16T01:40:45.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.623+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89eefe0 2024-09-16T01:40:45.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.624+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(109 400.00000000 [create] v90'22 uv21 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.624+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89eefe0 2024-09-16T01:40:45.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.626+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(110 400.00000000 [delete] v90'23 uv23 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.626+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89eefe0 2024-09-16T01:40:45.629 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:40:45.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.628+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(111 400.00000000 [create] v90'24 uv24 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.628+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89eefe0 2024-09-16T01:40:45.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.631+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v90'25 uv25 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.631+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89eefe0 2024-09-16T01:40:45.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.633+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v90'26 uv26 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.634+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89eefe0 2024-09-16T01:40:45.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.636+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(114 100.00000000.inode [create] v90'57 uv49 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.636+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89eefe0 2024-09-16T01:40:45.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.637+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(115 100.00000000.inode [delete] v90'58 uv58 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.637+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89eefe0 2024-09-16T01:40:45.639 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:40:45.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.638+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(116 100.00000000.inode [create] v90'59 uv59 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.638+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89eefe0 2024-09-16T01:40:45.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.639+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v90'60 uv60 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.639+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89eefe0 2024-09-16T01:40:45.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.640+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v90'61 uv61 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.640+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89eefe0 2024-09-16T01:40:45.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.641+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(119 1.00000000 [create] v90'62 uv56 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.641+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89eefe0 2024-09-16T01:40:45.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.642+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(120 1.00000000 [delete] v90'63 uv63 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.642+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89eefe0 2024-09-16T01:40:45.646 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:40:45.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.645+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(121 1.00000000 [create] v90'64 uv64 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.645+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89eefe0 2024-09-16T01:40:45.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.646+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v90'65 uv65 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.646+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89eefe0 2024-09-16T01:40:45.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.649+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v90'66 uv66 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.649+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89f7040 con 0x55aca89eefe0 2024-09-16T01:40:45.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.651+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v90'67 uv67 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.652+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90) -- 0x55aca89fa830 con 0x55aca89eefe0 2024-09-16T01:40:45.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.654+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v90'68 uv68 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd41800f070 con 0x55aca89eefe0 2024-09-16T01:40:45.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.654+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55aca88b3fd0 msgr2=0x55aca88b1060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:45.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.654+0000 7fd42031df40 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55aca88b3fd0 0x55aca88b1060 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.654+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55aca89ebde0 msgr2=0x55aca89ee1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:45.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.654+0000 7fd42031df40 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55aca89ebde0 0x55aca89ee1d0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.654+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55aca89f7a00 msgr2=0x55aca89fa2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:45.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.654+0000 7fd42031df40 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55aca89f7a00 0x55aca89fa2f0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.654+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55aca89e7190 msgr2=0x55aca89e9580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:45.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.654+0000 7fd42031df40 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55aca89e7190 0x55aca89e9580 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.654+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 reap_dead start 2024-09-16T01:40:45.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.654+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55aca89eefe0 msgr2=0x55aca89ef3c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:45.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.654+0000 7fd42031df40 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55aca89eefe0 0x55aca89ef3c0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.654+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55aca89f35f0 msgr2=0x55aca89f6710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:45.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.654+0000 7fd41db66640 1 -- 172.21.15.88:0/1182002894 reap_dead start 2024-09-16T01:40:45.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.654+0000 7fd42031df40 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55aca89f35f0 0x55aca89f6710 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.654+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd40006ac90 msgr2=0x7fd40006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:45.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.654+0000 7fd42031df40 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd40006ac90 0x7fd40006d150 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd4100057f0 tx=0x7fd41000d040 comp rx=0 tx=0).stop 2024-09-16T01:40:45.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.654+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca8925190 msgr2=0x55aca89381c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:45.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.654+0000 7fd42031df40 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca8925190 0x55aca89381c0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fd418001f40 tx=0x7fd418001250 comp rx=0 tx=0).stop 2024-09-16T01:40:45.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.654+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 shutdown_connections 2024-09-16T01:40:45.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.654+0000 7fd42031df40 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd40006ac90 0x7fd40006d150 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.655+0000 7fd42031df40 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca8925190 0x55aca89381c0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.655+0000 7fd42031df40 1 --2- 172.21.15.88:0/1182002894 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55aca89f35f0 0x55aca89f6710 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.655+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 >> 172.21.15.88:0/1182002894 conn(0x55aca8869be0 msgr2=0x55aca8939e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:45.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.655+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 shutdown_connections 2024-09-16T01:40:45.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.655+0000 7fd42031df40 1 -- 172.21.15.88:0/1182002894 wait complete. 2024-09-16T01:40:45.659 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 200.00000001 2024-09-16T01:40:45.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.699+0000 7f146296cf40 1 Processor -- start 2024-09-16T01:40:45.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.699+0000 7f146296cf40 1 -- start start 2024-09-16T01:40:45.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.700+0000 7f146296cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55684233cee0 0x55684240a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.700+0000 7f146296cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55684236d590 0x55684236b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.700+0000 7f146296cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556842410c90 0x55684233b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.700+0000 7f146296cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55684233d840 con 0x556842410c90 2024-09-16T01:40:45.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.700+0000 7f146296cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55684228e510 con 0x55684236d590 2024-09-16T01:40:45.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.700+0000 7f146296cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5568424116b0 con 0x55684233cee0 2024-09-16T01:40:45.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.700+0000 7f1460c63640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556842410c90 0x55684233b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.700+0000 7f1460c63640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556842410c90 0x55684233b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55474/0 (socket says 172.21.15.88:55474) 2024-09-16T01:40:45.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.700+0000 7f1460c63640 1 -- 172.21.15.88:0/4153407538 learned_addr learned my addr 172.21.15.88:0/4153407538 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:45.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.700+0000 7f1457fff640 1 --2- 172.21.15.88:0/4153407538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55684236d590 0x55684236b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.700+0000 7f1460462640 1 --2- 172.21.15.88:0/4153407538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55684233cee0 0x55684240a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.700+0000 7f1460c63640 1 -- 172.21.15.88:0/4153407538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55684233cee0 msgr2=0x55684240a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:45.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.700+0000 7f1460c63640 1 --2- 172.21.15.88:0/4153407538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55684233cee0 0x55684240a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.700+0000 7f1460c63640 1 -- 172.21.15.88:0/4153407538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55684236d590 msgr2=0x55684236b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:45.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.701+0000 7f1460c63640 1 --2- 172.21.15.88:0/4153407538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55684236d590 0x55684236b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.701+0000 7f1460c63640 1 -- 172.21.15.88:0/4153407538 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55684240c8d0 con 0x556842410c90 2024-09-16T01:40:45.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.701+0000 7f1460c63640 1 --2- 172.21.15.88:0/4153407538 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556842410c90 0x55684233b3c0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f1458025ad0 tx=0x7f1458050000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f60c4775683bbf2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:45.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.701+0000 7f145f460640 1 -- 172.21.15.88:0/4153407538 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145804fbb0 con 0x556842410c90 2024-09-16T01:40:45.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.701+0000 7f145f460640 1 -- 172.21.15.88:0/4153407538 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f145804fd50 con 0x556842410c90 2024-09-16T01:40:45.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.701+0000 7f145f460640 1 -- 172.21.15.88:0/4153407538 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1458057600 con 0x556842410c90 2024-09-16T01:40:45.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.701+0000 7f146296cf40 1 -- 172.21.15.88:0/4153407538 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556842410c90 msgr2=0x55684233b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:45.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.701+0000 7f146296cf40 1 --2- 172.21.15.88:0/4153407538 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556842410c90 0x55684233b3c0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f1458025ad0 tx=0x7f1458050000 comp rx=0 tx=0).stop 2024-09-16T01:40:45.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.701+0000 7f146296cf40 1 -- 172.21.15.88:0/4153407538 shutdown_connections 2024-09-16T01:40:45.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.701+0000 7f146296cf40 1 --2- 172.21.15.88:0/4153407538 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556842410c90 0x55684233b3c0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.701+0000 7f146296cf40 1 --2- 172.21.15.88:0/4153407538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55684236d590 0x55684236b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.701+0000 7f146296cf40 1 --2- 172.21.15.88:0/4153407538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55684233cee0 0x55684240a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.701+0000 7f146296cf40 1 -- 172.21.15.88:0/4153407538 >> 172.21.15.88:0/4153407538 conn(0x556842367680 msgr2=0x55684236c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:45.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.701+0000 7f146296cf40 1 -- 172.21.15.88:0/4153407538 shutdown_connections 2024-09-16T01:40:45.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.701+0000 7f146296cf40 1 -- 172.21.15.88:0/4153407538 wait complete. 2024-09-16T01:40:45.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.702+0000 7f146296cf40 1 Processor -- start 2024-09-16T01:40:45.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.702+0000 7f146296cf40 1 -- start start 2024-09-16T01:40:45.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.703+0000 7f146296cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55684233cee0 0x556842403950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.703+0000 7f146296cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55684236d590 0x55684240b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.703+0000 7f146296cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556842410c90 0x556842401400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.703+0000 7f146296cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x556842382f20 con 0x556842410c90 2024-09-16T01:40:45.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.703+0000 7f146296cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556842382cd0 con 0x55684236d590 2024-09-16T01:40:45.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.703+0000 7f146296cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55684240c540 con 0x55684233cee0 2024-09-16T01:40:45.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.703+0000 7f1460c63640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556842410c90 0x556842401400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.703+0000 7f1460c63640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556842410c90 0x556842401400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55484/0 (socket says 172.21.15.88:55484) 2024-09-16T01:40:45.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.703+0000 7f1460c63640 1 -- 172.21.15.88:0/1387513169 learned_addr learned my addr 172.21.15.88:0/1387513169 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:45.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.703+0000 7f1460462640 1 --2- 172.21.15.88:0/1387513169 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55684233cee0 0x556842403950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.703+0000 7f1457fff640 1 --2- 172.21.15.88:0/1387513169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55684236d590 0x55684240b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.703+0000 7f1460c63640 1 -- 172.21.15.88:0/1387513169 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55684233cee0 msgr2=0x556842403950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:45.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.703+0000 7f1460c63640 1 --2- 172.21.15.88:0/1387513169 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55684233cee0 0x556842403950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.703+0000 7f1460c63640 1 -- 172.21.15.88:0/1387513169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55684236d590 msgr2=0x55684240b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:45.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.703+0000 7f1460c63640 1 --2- 172.21.15.88:0/1387513169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55684236d590 0x55684240b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.703+0000 7f1460c63640 1 -- 172.21.15.88:0/1387513169 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556842383ad0 con 0x556842410c90 2024-09-16T01:40:45.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.703+0000 7f1460c63640 1 --2- 172.21.15.88:0/1387513169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556842410c90 0x556842401400 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f1458025ad0 tx=0x7f1458003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:45.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.703+0000 7f145e45e640 1 -- 172.21.15.88:0/1387513169 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1458071020 con 0x556842410c90 2024-09-16T01:40:45.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.703+0000 7f145e45e640 1 -- 172.21.15.88:0/1387513169 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f145804f9b0 con 0x556842410c90 2024-09-16T01:40:45.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.703+0000 7f146296cf40 1 -- 172.21.15.88:0/1387513169 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556842402e70 con 0x556842410c90 2024-09-16T01:40:45.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.703+0000 7f145e45e640 1 -- 172.21.15.88:0/1387513169 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1458057c40 con 0x556842410c90 2024-09-16T01:40:45.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.703+0000 7f146296cf40 1 -- 172.21.15.88:0/1387513169 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556842404700 con 0x556842410c90 2024-09-16T01:40:45.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.705+0000 7f145e45e640 1 -- 172.21.15.88:0/1387513169 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1458074020 con 0x556842410c90 2024-09-16T01:40:45.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.705+0000 7f145e45e640 1 --2- 172.21.15.88:0/1387513169 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f144006ac90 0x7f144006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.705+0000 7f145e45e640 1 -- 172.21.15.88:0/1387513169 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6139+0+0 (secure 0 0 0) 0x7f14580cf5b0 con 0x556842410c90 2024-09-16T01:40:45.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.705+0000 7f1460462640 1 --2- 172.21.15.88:0/1387513169 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f144006ac90 0x7f144006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.705+0000 7f146296cf40 1 --2- 172.21.15.88:0/1387513169 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x556842383eb0 0x5568423842c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.705+0000 7f146296cf40 1 -- 172.21.15.88:0/1387513169 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:1 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e90) -- 0x556842384800 con 0x556842383eb0 2024-09-16T01:40:45.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.706+0000 7f1457fff640 1 --2- 172.21.15.88:0/1387513169 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x556842383eb0 0x5568423842c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.706+0000 7f1460462640 1 --2- 172.21.15.88:0/1387513169 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f144006ac90 0x7f144006d150 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f1450011480 tx=0x7f14500016f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:45.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.706+0000 7f1457fff640 1 --2- 172.21.15.88:0/1387513169 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x556842383eb0 0x5568423842c0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:45.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.707+0000 7f1457fff640 1 -- 172.21.15.88:0/1387513169 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(1 200.00000001 [delete] v90'129 uv129 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1448009930 con 0x556842383eb0 2024-09-16T01:40:45.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.707+0000 7f146296cf40 1 -- 172.21.15.88:0/1387513169 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x556842383eb0 msgr2=0x5568423842c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:45.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.707+0000 7f146296cf40 1 --2- 172.21.15.88:0/1387513169 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x556842383eb0 0x5568423842c0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.707+0000 7f146296cf40 1 -- 172.21.15.88:0/1387513169 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f144006ac90 msgr2=0x7f144006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:45.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.707+0000 7f146296cf40 1 --2- 172.21.15.88:0/1387513169 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f144006ac90 0x7f144006d150 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f1450011480 tx=0x7f14500016f0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.707+0000 7f146296cf40 1 -- 172.21.15.88:0/1387513169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556842410c90 msgr2=0x556842401400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:45.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.707+0000 7f146296cf40 1 --2- 172.21.15.88:0/1387513169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556842410c90 0x556842401400 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f1458025ad0 tx=0x7f1458003330 comp rx=0 tx=0).stop 2024-09-16T01:40:45.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.707+0000 7f1460c63640 1 -- 172.21.15.88:0/1387513169 reap_dead start 2024-09-16T01:40:45.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.707+0000 7f146296cf40 1 -- 172.21.15.88:0/1387513169 shutdown_connections 2024-09-16T01:40:45.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.707+0000 7f146296cf40 1 -- 172.21.15.88:0/1387513169 >> 172.21.15.88:0/1387513169 conn(0x556842367680 msgr2=0x55684236c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:45.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.707+0000 7f146296cf40 1 -- 172.21.15.88:0/1387513169 shutdown_connections 2024-09-16T01:40:45.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.707+0000 7f146296cf40 1 -- 172.21.15.88:0/1387513169 wait complete. 2024-09-16T01:40:45.715 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:40:45.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.855+0000 7eff9ea35640 1 Processor -- start 2024-09-16T01:40:45.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.855+0000 7eff9ea35640 1 -- start start 2024-09-16T01:40:45.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.856+0000 7eff9ea35640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7eff9815e6f0 0x7eff9806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.856+0000 7eff9ea35640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7eff980b54d0 0x7eff9815f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.856+0000 7eff9ea35640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff9806e990 con 0x7eff980b6d40 2024-09-16T01:40:45.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.856+0000 7eff9ea35640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7eff9806ec70 con 0x7eff9815e6f0 2024-09-16T01:40:45.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.856+0000 7eff9ea35640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7eff9815ec70 con 0x7eff980b54d0 2024-09-16T01:40:45.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.856+0000 7eff97fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7eff9815e6f0 0x7eff9806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.856+0000 7eff9cfd2640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7eff980b6d40 0x7eff980b71f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59734/0 (socket says 172.21.15.88:59734) 2024-09-16T01:40:45.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.856+0000 7eff9cfd2640 1 -- 172.21.15.88:0/3989353175 learned_addr learned my addr 172.21.15.88:0/3989353175 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:45.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.856+0000 7eff977fe640 1 --2- 172.21.15.88:0/3989353175 >> v2:172.21.15.154:3300/0 conn(0x7eff980b54d0 0x7eff9815f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.856+0000 7eff97fff640 1 -- 172.21.15.88:0/3989353175 >> v2:172.21.15.154:3300/0 conn(0x7eff980b54d0 msgr2=0x7eff9815f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:45.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.856+0000 7eff97fff640 1 --2- 172.21.15.88:0/3989353175 >> v2:172.21.15.154:3300/0 conn(0x7eff980b54d0 0x7eff9815f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.856+0000 7eff97fff640 1 -- 172.21.15.88:0/3989353175 >> v1:172.21.15.88:6789/0 conn(0x7eff980b6d40 legacy=0x7eff980b71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:45.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.856+0000 7eff97fff640 1 -- 172.21.15.88:0/3989353175 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff9815ef50 con 0x7eff9815e6f0 2024-09-16T01:40:45.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.856+0000 7eff96ffd640 1 -- 172.21.15.88:0/3989353175 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1810176779 0 0) 0x7eff9806e990 con 0x7eff980b6d40 2024-09-16T01:40:45.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.857+0000 7eff97fff640 1 --2- 172.21.15.88:0/3989353175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff9815e6f0 0x7eff9806fd00 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7eff980705f0 tx=0x7eff8802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=303ada720ecc2fb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:45.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.857+0000 7eff96ffd640 1 -- 172.21.15.88:0/3989353175 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff88037070 con 0x7eff9815e6f0 2024-09-16T01:40:45.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.857+0000 7eff96ffd640 1 -- 172.21.15.88:0/3989353175 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff88032440 con 0x7eff9815e6f0 2024-09-16T01:40:45.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.857+0000 7eff96ffd640 1 -- 172.21.15.88:0/3989353175 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff88032810 con 0x7eff9815e6f0 2024-09-16T01:40:45.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.858+0000 7eff9ea35640 1 -- 172.21.15.88:0/3989353175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff9815e6f0 msgr2=0x7eff9806fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:45.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.858+0000 7eff9ea35640 1 --2- 172.21.15.88:0/3989353175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff9815e6f0 0x7eff9806fd00 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7eff980705f0 tx=0x7eff8802a1a0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.858+0000 7eff9ea35640 1 -- 172.21.15.88:0/3989353175 shutdown_connections 2024-09-16T01:40:45.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.858+0000 7eff9ea35640 1 --2- 172.21.15.88:0/3989353175 >> v2:172.21.15.154:3300/0 conn(0x7eff980b54d0 0x7eff9815f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.858+0000 7eff9ea35640 1 --2- 172.21.15.88:0/3989353175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff9815e6f0 0x7eff9806fd00 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.858+0000 7eff9ea35640 1 -- 172.21.15.88:0/3989353175 >> 172.21.15.88:0/3989353175 conn(0x7eff980d2430 msgr2=0x7eff980d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:45.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.858+0000 7eff9ea35640 1 -- 172.21.15.88:0/3989353175 shutdown_connections 2024-09-16T01:40:45.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.858+0000 7eff9ea35640 1 -- 172.21.15.88:0/3989353175 wait complete. 2024-09-16T01:40:45.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.859+0000 7eff9ea35640 1 Processor -- start 2024-09-16T01:40:45.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.859+0000 7eff9ea35640 1 -- start start 2024-09-16T01:40:45.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.860+0000 7eff9ea35640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff980b54d0 0x7eff98153710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.860+0000 7eff9ea35640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff980b6d40 0x7eff9815f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.860+0000 7eff9ea35640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff9815e6f0 0x7eff9815ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.860+0000 7eff9ea35640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7eff9806e730 con 0x7eff9815e6f0 2024-09-16T01:40:45.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.860+0000 7eff9ea35640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7eff9806e5b0 con 0x7eff980b6d40 2024-09-16T01:40:45.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.860+0000 7eff9ea35640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7eff98070830 con 0x7eff980b54d0 2024-09-16T01:40:45.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.860+0000 7eff9cfd2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff9815e6f0 0x7eff9815ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.860+0000 7eff9cfd2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff9815e6f0 0x7eff9815ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55498/0 (socket says 172.21.15.88:55498) 2024-09-16T01:40:45.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.860+0000 7eff9cfd2640 1 -- 172.21.15.88:0/3890325418 learned_addr learned my addr 172.21.15.88:0/3890325418 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:45.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.860+0000 7eff97fff640 1 --2- 172.21.15.88:0/3890325418 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff980b54d0 0x7eff98153710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.860+0000 7eff977fe640 1 --2- 172.21.15.88:0/3890325418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff980b6d40 0x7eff9815f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.860+0000 7eff9cfd2640 1 -- 172.21.15.88:0/3890325418 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff980b54d0 msgr2=0x7eff98153710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:45.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.860+0000 7eff9cfd2640 1 --2- 172.21.15.88:0/3890325418 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff980b54d0 0x7eff98153710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.860+0000 7eff9cfd2640 1 -- 172.21.15.88:0/3890325418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff980b6d40 msgr2=0x7eff9815f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:45.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.860+0000 7eff9cfd2640 1 --2- 172.21.15.88:0/3890325418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff980b6d40 0x7eff9815f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:45.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.860+0000 7eff9cfd2640 1 -- 172.21.15.88:0/3890325418 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff981600d0 con 0x7eff9815e6f0 2024-09-16T01:40:45.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.860+0000 7eff9cfd2640 1 --2- 172.21.15.88:0/3890325418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff9815e6f0 0x7eff9815ca70 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7eff9000c7a0 tx=0x7eff9000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:45.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.861+0000 7eff94ff9640 1 -- 172.21.15.88:0/3890325418 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff90018070 con 0x7eff9815e6f0 2024-09-16T01:40:45.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.861+0000 7eff94ff9640 1 -- 172.21.15.88:0/3890325418 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff9000a640 con 0x7eff9815e6f0 2024-09-16T01:40:45.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.861+0000 7eff94ff9640 1 -- 172.21.15.88:0/3890325418 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff9000aa70 con 0x7eff9815e6f0 2024-09-16T01:40:45.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.861+0000 7eff9ea35640 1 -- 172.21.15.88:0/3890325418 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff981603b0 con 0x7eff9815e6f0 2024-09-16T01:40:45.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.861+0000 7eff9ea35640 1 -- 172.21.15.88:0/3890325418 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff98161370 con 0x7eff9815e6f0 2024-09-16T01:40:45.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.862+0000 7eff9ea35640 1 -- 172.21.15.88:0/3890325418 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff9806c6c0 con 0x7eff9815e6f0 2024-09-16T01:40:45.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.862+0000 7eff94ff9640 1 -- 172.21.15.88:0/3890325418 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7eff9001b070 con 0x7eff9815e6f0 2024-09-16T01:40:45.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.863+0000 7eff94ff9640 1 --2- 172.21.15.88:0/3890325418 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff8406c2b0 0x7eff8406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:45.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.863+0000 7eff94ff9640 1 -- 172.21.15.88:0/3890325418 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 6139+0+0 (secure 0 0 0) 0x7eff9008ab20 con 0x7eff9815e6f0 2024-09-16T01:40:45.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.863+0000 7eff97fff640 1 --2- 172.21.15.88:0/3890325418 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff8406c2b0 0x7eff8406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:45.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.863+0000 7eff97fff640 1 --2- 172.21.15.88:0/3890325418 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff8406c2b0 0x7eff8406e770 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7eff88034000 tx=0x7eff8802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:45.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:45.867+0000 7eff94ff9640 1 -- 172.21.15.88:0/3890325418 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7eff90058b00 con 0x7eff9815e6f0 2024-09-16T01:40:46.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.072+0000 7eff9ea35640 1 -- 172.21.15.88:0/3890325418 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7eff9815f750 con 0x7eff9815e6f0 2024-09-16T01:40:46.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.422+0000 7eff94ff9640 1 -- 172.21.15.88:0/3890325418 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v100) ==== 162+0+0 (secure 0 0 0) 0x7eff9005ce00 con 0x7eff9815e6f0 2024-09-16T01:40:46.423 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:40:46.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.425+0000 7eff9ea35640 1 -- 172.21.15.88:0/3890325418 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff8406c2b0 msgr2=0x7eff8406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:46.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.425+0000 7eff9ea35640 1 --2- 172.21.15.88:0/3890325418 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff8406c2b0 0x7eff8406e770 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7eff88034000 tx=0x7eff8802e000 comp rx=0 tx=0).stop 2024-09-16T01:40:46.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.425+0000 7eff9ea35640 1 -- 172.21.15.88:0/3890325418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff9815e6f0 msgr2=0x7eff9815ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:46.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.425+0000 7eff9ea35640 1 --2- 172.21.15.88:0/3890325418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff9815e6f0 0x7eff9815ca70 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7eff9000c7a0 tx=0x7eff9000cc60 comp rx=0 tx=0).stop 2024-09-16T01:40:46.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.425+0000 7eff9ea35640 1 -- 172.21.15.88:0/3890325418 shutdown_connections 2024-09-16T01:40:46.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.425+0000 7eff9ea35640 1 --2- 172.21.15.88:0/3890325418 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7eff8406c2b0 0x7eff8406e770 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:46.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.425+0000 7eff9ea35640 1 --2- 172.21.15.88:0/3890325418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff9815e6f0 0x7eff9815ca70 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:46.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.425+0000 7eff9ea35640 1 --2- 172.21.15.88:0/3890325418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff980b6d40 0x7eff9815f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:46.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.425+0000 7eff9ea35640 1 --2- 172.21.15.88:0/3890325418 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff980b54d0 0x7eff98153710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:46.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.425+0000 7eff9ea35640 1 -- 172.21.15.88:0/3890325418 >> 172.21.15.88:0/3890325418 conn(0x7eff980d2430 msgr2=0x7eff980b33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:46.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.425+0000 7eff9ea35640 1 -- 172.21.15.88:0/3890325418 shutdown_connections 2024-09-16T01:40:46.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.425+0000 7eff9ea35640 1 -- 172.21.15.88:0/3890325418 wait complete. 2024-09-16T01:40:46.434 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:40:46.433+0000 7f9a08fb3640 -1 log_channel(cluster) log [ERR] : missing journal object 2024-09-16T01:40:46.434 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:40:46.433+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:40:46.435 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:40:46.433+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:40:46.439 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:40:46.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.536+0000 7f55ed627640 1 Processor -- start 2024-09-16T01:40:46.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.536+0000 7f55ed627640 1 -- start start 2024-09-16T01:40:46.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.537+0000 7f55ed627640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f55e815e950 0x7f55e806ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:46.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.537+0000 7f55ed627640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f55e8072050 0x7f55e806d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:46.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.537+0000 7f55ed627640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55e806faf0 con 0x7f55e80735b0 2024-09-16T01:40:46.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.537+0000 7f55ed627640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f55e806fdd0 con 0x7f55e815e950 2024-09-16T01:40:46.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.537+0000 7f55ed627640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f55e80700b0 con 0x7f55e8072050 2024-09-16T01:40:46.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.537+0000 7f55e6ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f55e815e950 0x7f55e806ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:46.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.537+0000 7f55e67fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f55e80735b0 0x7f55e806bc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59742/0 (socket says 172.21.15.88:59742) 2024-09-16T01:40:46.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.537+0000 7f55e67fc640 1 -- 172.21.15.88:0/2984732166 learned_addr learned my addr 172.21.15.88:0/2984732166 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:46.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.537+0000 7f55e77fe640 1 --2- 172.21.15.88:0/2984732166 >> v2:172.21.15.154:3300/0 conn(0x7f55e8072050 0x7f55e806d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:46.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.537+0000 7f55e6ffd640 1 -- 172.21.15.88:0/2984732166 >> v2:172.21.15.154:3300/0 conn(0x7f55e8072050 msgr2=0x7f55e806d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:46.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.537+0000 7f55e6ffd640 1 --2- 172.21.15.88:0/2984732166 >> v2:172.21.15.154:3300/0 conn(0x7f55e8072050 0x7f55e806d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:46.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.537+0000 7f55e6ffd640 1 -- 172.21.15.88:0/2984732166 >> v1:172.21.15.88:6789/0 conn(0x7f55e80735b0 legacy=0x7f55e806bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:46.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.537+0000 7f55e6ffd640 1 -- 172.21.15.88:0/2984732166 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55e8070390 con 0x7f55e815e950 2024-09-16T01:40:46.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.537+0000 7f55e5ffb640 1 -- 172.21.15.88:0/2984732166 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2646417742 0 0) 0x7f55e806faf0 con 0x7f55e80735b0 2024-09-16T01:40:46.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.538+0000 7f55e6ffd640 1 --2- 172.21.15.88:0/2984732166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55e815e950 0x7f55e806ce30 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f55e80724e0 tx=0x7f55d802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99694e78bb9b02e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:46.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.538+0000 7f55e5ffb640 1 -- 172.21.15.88:0/2984732166 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d8007740 con 0x7f55e815e950 2024-09-16T01:40:46.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.538+0000 7f55e5ffb640 1 -- 172.21.15.88:0/2984732166 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55d80078e0 con 0x7f55e815e950 2024-09-16T01:40:46.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.538+0000 7f55e5ffb640 1 -- 172.21.15.88:0/2984732166 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d8007cb0 con 0x7f55e815e950 2024-09-16T01:40:46.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.539+0000 7f55ed627640 1 -- 172.21.15.88:0/2984732166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55e815e950 msgr2=0x7f55e806ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:46.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.539+0000 7f55ed627640 1 --2- 172.21.15.88:0/2984732166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55e815e950 0x7f55e806ce30 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f55e80724e0 tx=0x7f55d802a680 comp rx=0 tx=0).stop 2024-09-16T01:40:46.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.539+0000 7f55ed627640 1 -- 172.21.15.88:0/2984732166 shutdown_connections 2024-09-16T01:40:46.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.539+0000 7f55ed627640 1 --2- 172.21.15.88:0/2984732166 >> v2:172.21.15.154:3300/0 conn(0x7f55e8072050 0x7f55e806d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:46.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.539+0000 7f55ed627640 1 --2- 172.21.15.88:0/2984732166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55e815e950 0x7f55e806ce30 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:46.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.539+0000 7f55ed627640 1 -- 172.21.15.88:0/2984732166 >> 172.21.15.88:0/2984732166 conn(0x7f55e80d14d0 msgr2=0x7f55e80d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:46.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.539+0000 7f55ed627640 1 -- 172.21.15.88:0/2984732166 shutdown_connections 2024-09-16T01:40:46.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.539+0000 7f55ed627640 1 -- 172.21.15.88:0/2984732166 wait complete. 2024-09-16T01:40:46.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.540+0000 7f55ed627640 1 Processor -- start 2024-09-16T01:40:46.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.540+0000 7f55ed627640 1 -- start start 2024-09-16T01:40:46.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.541+0000 7f55ed627640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55e8072050 0x7f55e8148a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:46.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.541+0000 7f55ed627640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55e80735b0 0x7f55e80cb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:46.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.541+0000 7f55ed627640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f55e815e950 0x7f55e80cbe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:46.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.541+0000 7f55ed627640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f55e80d0230 con 0x7f55e80735b0 2024-09-16T01:40:46.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.541+0000 7f55ed627640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f55e80d00b0 con 0x7f55e8072050 2024-09-16T01:40:46.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.541+0000 7f55ed627640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f55e80d03b0 con 0x7f55e815e950 2024-09-16T01:40:46.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.541+0000 7f55e67fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55e80735b0 0x7f55e80cb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:46.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.541+0000 7f55e67fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55e80735b0 0x7f55e80cb950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55512/0 (socket says 172.21.15.88:55512) 2024-09-16T01:40:46.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.541+0000 7f55e67fc640 1 -- 172.21.15.88:0/955842310 learned_addr learned my addr 172.21.15.88:0/955842310 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:46.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.541+0000 7f55e6ffd640 1 --2- 172.21.15.88:0/955842310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55e8072050 0x7f55e8148a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:46.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.541+0000 7f55e77fe640 1 --2- 172.21.15.88:0/955842310 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f55e815e950 0x7f55e80cbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:46.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.541+0000 7f55e67fc640 1 -- 172.21.15.88:0/955842310 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f55e815e950 msgr2=0x7f55e80cbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:46.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.541+0000 7f55e67fc640 1 --2- 172.21.15.88:0/955842310 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f55e815e950 0x7f55e80cbe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:46.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.541+0000 7f55e67fc640 1 -- 172.21.15.88:0/955842310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55e8072050 msgr2=0x7f55e8148a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:46.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.541+0000 7f55e67fc640 1 --2- 172.21.15.88:0/955842310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55e8072050 0x7f55e8148a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:46.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.541+0000 7f55e67fc640 1 -- 172.21.15.88:0/955842310 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55e815bd10 con 0x7f55e80735b0 2024-09-16T01:40:46.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.541+0000 7f55e6ffd640 1 --2- 172.21.15.88:0/955842310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55e8072050 0x7f55e8148a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:46.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.541+0000 7f55e67fc640 1 --2- 172.21.15.88:0/955842310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55e80735b0 0x7f55e80cb950 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f55dc00c4e0 tx=0x7f55dc00c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:46.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.541+0000 7f55cf7fe640 1 -- 172.21.15.88:0/955842310 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55dc018070 con 0x7f55e80735b0 2024-09-16T01:40:46.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.542+0000 7f55cf7fe640 1 -- 172.21.15.88:0/955842310 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55dc007500 con 0x7f55e80735b0 2024-09-16T01:40:46.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.542+0000 7f55ed627640 1 -- 172.21.15.88:0/955842310 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55e815bff0 con 0x7f55e80735b0 2024-09-16T01:40:46.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.542+0000 7f55cf7fe640 1 -- 172.21.15.88:0/955842310 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55dc004a20 con 0x7f55e80735b0 2024-09-16T01:40:46.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.542+0000 7f55ed627640 1 -- 172.21.15.88:0/955842310 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55e8149540 con 0x7f55e80735b0 2024-09-16T01:40:46.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.543+0000 7f55ed627640 1 -- 172.21.15.88:0/955842310 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55e806ce30 con 0x7f55e80735b0 2024-09-16T01:40:46.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.543+0000 7f55cf7fe640 1 -- 172.21.15.88:0/955842310 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f55dc01b070 con 0x7f55e80735b0 2024-09-16T01:40:46.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.544+0000 7f55cf7fe640 1 --2- 172.21.15.88:0/955842310 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f55d406c2b0 0x7f55d406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:46.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.544+0000 7f55cf7fe640 1 -- 172.21.15.88:0/955842310 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6225+0+0 (secure 0 0 0) 0x7f55dc08a8e0 con 0x7f55e80735b0 2024-09-16T01:40:46.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.544+0000 7f55e6ffd640 1 --2- 172.21.15.88:0/955842310 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f55d406c2b0 0x7f55d406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:46.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.544+0000 7f55e6ffd640 1 --2- 172.21.15.88:0/955842310 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f55d406c2b0 0x7f55d406e770 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f55d8034000 tx=0x7f55d802f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:46.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.548+0000 7f55cf7fe640 1 -- 172.21.15.88:0/955842310 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f55dc058f90 con 0x7f55e80735b0 2024-09-16T01:40:46.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.749+0000 7f55ed627640 1 -- 172.21.15.88:0/955842310 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f55e80cde00 con 0x7f55e80735b0 2024-09-16T01:40:46.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.750+0000 7f55cf7fe640 1 -- 172.21.15.88:0/955842310 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+3565 (secure 0 0 0) 0x7f55dc058890 con 0x7f55e80735b0 2024-09-16T01:40:46.751 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:46.751 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":102,"btime":"2024-09-16T01:40:46:434694+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:40:46.434636+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:40:46.752 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 102 2024-09-16T01:40:46.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.752+0000 7f55ed627640 1 -- 172.21.15.88:0/955842310 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f55d406c2b0 msgr2=0x7f55d406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:46.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.752+0000 7f55ed627640 1 --2- 172.21.15.88:0/955842310 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f55d406c2b0 0x7f55d406e770 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f55d8034000 tx=0x7f55d802f000 comp rx=0 tx=0).stop 2024-09-16T01:40:46.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.752+0000 7f55ed627640 1 -- 172.21.15.88:0/955842310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55e80735b0 msgr2=0x7f55e80cb950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:46.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.752+0000 7f55ed627640 1 --2- 172.21.15.88:0/955842310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55e80735b0 0x7f55e80cb950 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f55dc00c4e0 tx=0x7f55dc00c9a0 comp rx=0 tx=0).stop 2024-09-16T01:40:46.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.752+0000 7f55ed627640 1 -- 172.21.15.88:0/955842310 shutdown_connections 2024-09-16T01:40:46.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.752+0000 7f55ed627640 1 --2- 172.21.15.88:0/955842310 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f55d406c2b0 0x7f55d406e770 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:46.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.752+0000 7f55ed627640 1 --2- 172.21.15.88:0/955842310 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f55e815e950 0x7f55e80cbe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:46.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.752+0000 7f55ed627640 1 --2- 172.21.15.88:0/955842310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55e80735b0 0x7f55e80cb950 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:46.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.752+0000 7f55ed627640 1 --2- 172.21.15.88:0/955842310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55e8072050 0x7f55e8148a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:46.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.752+0000 7f55ed627640 1 -- 172.21.15.88:0/955842310 >> 172.21.15.88:0/955842310 conn(0x7f55e80d14d0 msgr2=0x7f55e80b29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:46.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.752+0000 7f55ed627640 1 -- 172.21.15.88:0/955842310 shutdown_connections 2024-09-16T01:40:46.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.752+0000 7f55ed627640 1 -- 172.21.15.88:0/955842310 wait complete. 2024-09-16T01:40:46.764 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:40:46.764 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:40:46.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.860+0000 7f213eb8d640 1 Processor -- start 2024-09-16T01:40:46.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.860+0000 7f213eb8d640 1 -- start start 2024-09-16T01:40:46.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.861+0000 7f213eb8d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f213815c4c0 0x7f213806c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:46.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.861+0000 7f213eb8d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f213806e200 0x7f213806e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:46.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.861+0000 7f213eb8d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f213806eb60 con 0x7f213815ffc0 2024-09-16T01:40:46.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.861+0000 7f213eb8d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f21380cfc50 con 0x7f213815c4c0 2024-09-16T01:40:46.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.861+0000 7f213eb8d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f213815c8a0 con 0x7f213806e200 2024-09-16T01:40:46.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.861+0000 7f213c929640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f213815c4c0 0x7f213806c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:46.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.861+0000 7f213c929640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f213815c4c0 0x7f213806c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55522/0 (socket says 172.21.15.88:55522) 2024-09-16T01:40:46.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.861+0000 7f213c929640 1 -- 172.21.15.88:0/2484987292 learned_addr learned my addr 172.21.15.88:0/2484987292 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:46.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.861+0000 7f2137fff640 1 --2- 172.21.15.88:0/2484987292 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f213806e200 0x7f213806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:46.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.861+0000 7f213c929640 1 -- 172.21.15.88:0/2484987292 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f213806e200 msgr2=0x7f213806e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:46.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.861+0000 7f213c929640 1 --2- 172.21.15.88:0/2484987292 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f213806e200 0x7f213806e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:46.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.861+0000 7f213c929640 1 -- 172.21.15.88:0/2484987292 >> v1:172.21.15.154:6789/0 conn(0x7f213815ffc0 legacy=0x7f2138160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:46.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.861+0000 7f213c929640 1 -- 172.21.15.88:0/2484987292 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f213806cf90 con 0x7f213815c4c0 2024-09-16T01:40:46.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.862+0000 7f213c929640 1 --2- 172.21.15.88:0/2484987292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f213815c4c0 0x7f213806c620 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f212c009870 tx=0x7f212c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e648589d9645f460 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:46.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.862+0000 7f21377fe640 1 -- 172.21.15.88:0/2484987292 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f212c037070 con 0x7f213815c4c0 2024-09-16T01:40:46.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.862+0000 7f21377fe640 1 -- 172.21.15.88:0/2484987292 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f212c032440 con 0x7f213815c4c0 2024-09-16T01:40:46.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.862+0000 7f21377fe640 1 -- 172.21.15.88:0/2484987292 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f212c032810 con 0x7f213815c4c0 2024-09-16T01:40:46.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.863+0000 7f213eb8d640 1 -- 172.21.15.88:0/2484987292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f213815c4c0 msgr2=0x7f213806c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:46.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.863+0000 7f213eb8d640 1 --2- 172.21.15.88:0/2484987292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f213815c4c0 0x7f213806c620 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f212c009870 tx=0x7f212c02a270 comp rx=0 tx=0).stop 2024-09-16T01:40:46.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.863+0000 7f213eb8d640 1 -- 172.21.15.88:0/2484987292 shutdown_connections 2024-09-16T01:40:46.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.863+0000 7f213eb8d640 1 --2- 172.21.15.88:0/2484987292 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f213806e200 0x7f213806e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:46.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.863+0000 7f213eb8d640 1 --2- 172.21.15.88:0/2484987292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f213815c4c0 0x7f213806c620 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:46.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.863+0000 7f213eb8d640 1 -- 172.21.15.88:0/2484987292 >> 172.21.15.88:0/2484987292 conn(0x7f21380b6ec0 msgr2=0x7f21380b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:46.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.863+0000 7f213eb8d640 1 -- 172.21.15.88:0/2484987292 shutdown_connections 2024-09-16T01:40:46.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.863+0000 7f213eb8d640 1 -- 172.21.15.88:0/2484987292 wait complete. 2024-09-16T01:40:46.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.864+0000 7f213eb8d640 1 Processor -- start 2024-09-16T01:40:46.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.864+0000 7f213eb8d640 1 -- start start 2024-09-16T01:40:46.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.865+0000 7f213eb8d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f213806e200 0x7f213815f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:46.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.865+0000 7f213eb8d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f213815c4c0 0x7f21381f4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:46.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.865+0000 7f213eb8d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f213815ffc0 0x7f21381f4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:46.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.865+0000 7f213eb8d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f21380cf240 con 0x7f213806e200 2024-09-16T01:40:46.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.865+0000 7f213eb8d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f21380cf8e0 con 0x7f213815ffc0 2024-09-16T01:40:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.865+0000 7f213eb8d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f21380cf3c0 con 0x7f213815c4c0 2024-09-16T01:40:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.865+0000 7f213c929640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f213806e200 0x7f213815f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.865+0000 7f213c929640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f213806e200 0x7f213815f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55530/0 (socket says 172.21.15.88:55530) 2024-09-16T01:40:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.865+0000 7f213c929640 1 -- 172.21.15.88:0/2336769911 learned_addr learned my addr 172.21.15.88:0/2336769911 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.865+0000 7f2137fff640 1 --2- 172.21.15.88:0/2336769911 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f213815c4c0 0x7f21381f4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.865+0000 7f213d12a640 1 --2- 172.21.15.88:0/2336769911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f213815ffc0 0x7f21381f4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.865+0000 7f213c929640 1 -- 172.21.15.88:0/2336769911 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f213815c4c0 msgr2=0x7f21381f4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.865+0000 7f213c929640 1 --2- 172.21.15.88:0/2336769911 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f213815c4c0 0x7f21381f4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.865+0000 7f213c929640 1 -- 172.21.15.88:0/2336769911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f213815ffc0 msgr2=0x7f21381f4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.865+0000 7f213c929640 1 --2- 172.21.15.88:0/2336769911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f213815ffc0 0x7f21381f4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.865+0000 7f213c929640 1 -- 172.21.15.88:0/2336769911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21381e60f0 con 0x7f213806e200 2024-09-16T01:40:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.865+0000 7f213c929640 1 --2- 172.21.15.88:0/2336769911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f213806e200 0x7f213815f120 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f212c02a2b0 tx=0x7f212c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.866+0000 7f21357fa640 1 -- 172.21.15.88:0/2336769911 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f212c041020 con 0x7f213806e200 2024-09-16T01:40:46.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.866+0000 7f21357fa640 1 -- 172.21.15.88:0/2336769911 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f212c0306c0 con 0x7f213806e200 2024-09-16T01:40:46.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.866+0000 7f21357fa640 1 -- 172.21.15.88:0/2336769911 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f212c037040 con 0x7f213806e200 2024-09-16T01:40:46.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.866+0000 7f213eb8d640 1 -- 172.21.15.88:0/2336769911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21380b5080 con 0x7f213806e200 2024-09-16T01:40:46.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.866+0000 7f213eb8d640 1 -- 172.21.15.88:0/2336769911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21380b5640 con 0x7f213806e200 2024-09-16T01:40:46.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.867+0000 7f213eb8d640 1 -- 172.21.15.88:0/2336769911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f213806da10 con 0x7f213806e200 2024-09-16T01:40:46.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.867+0000 7f21357fa640 1 -- 172.21.15.88:0/2336769911 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f212c044020 con 0x7f213806e200 2024-09-16T01:40:46.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.868+0000 7f21357fa640 1 --2- 172.21.15.88:0/2336769911 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f211c06c2b0 0x7f211c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:46.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.868+0000 7f2137fff640 1 --2- 172.21.15.88:0/2336769911 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f211c06c2b0 0x7f211c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:46.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.868+0000 7f21357fa640 1 -- 172.21.15.88:0/2336769911 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6225+0+0 (secure 0 0 0) 0x7f212c0abe40 con 0x7f213806e200 2024-09-16T01:40:46.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.868+0000 7f2137fff640 1 --2- 172.21.15.88:0/2336769911 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f211c06c2b0 0x7f211c06e770 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2138156110 tx=0x7f2124008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:46.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:46.872+0000 7f21357fa640 1 -- 172.21.15.88:0/2336769911 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f212c04b030 con 0x7f213806e200 2024-09-16T01:40:47.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.072+0000 7f213eb8d640 1 -- 172.21.15.88:0/2336769911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2138158220 con 0x7f213806e200 2024-09-16T01:40:47.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.072+0000 7f21357fa640 1 -- 172.21.15.88:0/2336769911 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+3565 (secure 0 0 0) 0x7f212c07a3a0 con 0x7f213806e200 2024-09-16T01:40:47.074 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:47.074 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":102,"btime":"2024-09-16T01:40:46:434694+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:40:46.434636+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:40:47.074 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 102 2024-09-16T01:40:47.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.074+0000 7f213eb8d640 1 -- 172.21.15.88:0/2336769911 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f211c06c2b0 msgr2=0x7f211c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:47.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.074+0000 7f213eb8d640 1 --2- 172.21.15.88:0/2336769911 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f211c06c2b0 0x7f211c06e770 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2138156110 tx=0x7f2124008040 comp rx=0 tx=0).stop 2024-09-16T01:40:47.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.074+0000 7f213eb8d640 1 -- 172.21.15.88:0/2336769911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f213806e200 msgr2=0x7f213815f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:47.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.074+0000 7f213eb8d640 1 --2- 172.21.15.88:0/2336769911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f213806e200 0x7f213815f120 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f212c02a2b0 tx=0x7f212c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:40:47.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.074+0000 7f213eb8d640 1 -- 172.21.15.88:0/2336769911 shutdown_connections 2024-09-16T01:40:47.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.074+0000 7f213eb8d640 1 --2- 172.21.15.88:0/2336769911 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f211c06c2b0 0x7f211c06e770 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:47.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.074+0000 7f213eb8d640 1 --2- 172.21.15.88:0/2336769911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f213815ffc0 0x7f21381f4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:47.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.074+0000 7f213eb8d640 1 --2- 172.21.15.88:0/2336769911 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f213815c4c0 0x7f21381f4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:47.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.074+0000 7f213eb8d640 1 --2- 172.21.15.88:0/2336769911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f213806e200 0x7f213815f120 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:47.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.074+0000 7f213eb8d640 1 -- 172.21.15.88:0/2336769911 >> 172.21.15.88:0/2336769911 conn(0x7f21380b6ec0 msgr2=0x7f21380b1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:47.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.074+0000 7f213eb8d640 1 -- 172.21.15.88:0/2336769911 shutdown_connections 2024-09-16T01:40:47.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.074+0000 7f213eb8d640 1 -- 172.21.15.88:0/2336769911 wait complete. 2024-09-16T01:40:47.087 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:40:47.088 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:40:47.151 INFO:tasks.ceph.mds.a.smithi088.stderr: -36> 2024-09-16T01:40:42.398+0000 7ff36aa6b640 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x200: (2) No such file or directory 2024-09-16T01:40:47.174 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:40:47.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.187+0000 7fbc610b7640 1 Processor -- start 2024-09-16T01:40:47.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.187+0000 7fbc610b7640 1 -- start start 2024-09-16T01:40:47.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.187+0000 7fbc610b7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbc5c0ba240 0x7fbc5c0d3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:47.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.187+0000 7fbc610b7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc5c0cf2f0 0x7fbc5c0cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:47.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.187+0000 7fbc610b7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc5c0d2f40 con 0x7fbc5c15e7a0 2024-09-16T01:40:47.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.187+0000 7fbc610b7640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbc5c0ba690 con 0x7fbc5c0ba240 2024-09-16T01:40:47.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.187+0000 7fbc610b7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbc5c0d3220 con 0x7fbc5c0cf2f0 2024-09-16T01:40:47.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.188+0000 7fbc53fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbc5c0ba240 0x7fbc5c0d3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:47.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.188+0000 7fbc5b59e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc5c0cf2f0 0x7fbc5c0cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:47.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.188+0000 7fbc5ad9d640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fbc5c15e7a0 0x7fbc5c0d2780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:54108/0 (socket says 172.21.15.88:54108) 2024-09-16T01:40:47.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.188+0000 7fbc5ad9d640 1 -- 172.21.15.88:0/1744676611 learned_addr learned my addr 172.21.15.88:0/1744676611 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:47.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.188+0000 7fbc53fff640 1 -- 172.21.15.88:0/1744676611 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc5c0cf2f0 msgr2=0x7fbc5c0cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:47.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.188+0000 7fbc53fff640 1 --2- 172.21.15.88:0/1744676611 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc5c0cf2f0 0x7fbc5c0cf780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:47.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.188+0000 7fbc53fff640 1 -- 172.21.15.88:0/1744676611 >> v1:172.21.15.154:6789/0 conn(0x7fbc5c15e7a0 legacy=0x7fbc5c0d2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:47.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.188+0000 7fbc53fff640 1 -- 172.21.15.88:0/1744676611 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc5c0d3500 con 0x7fbc5c0ba240 2024-09-16T01:40:47.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.188+0000 7fbc5a59c640 1 -- 172.21.15.88:0/1744676611 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2728550741 0 0) 0x7fbc5c0d2f40 con 0x7fbc5c15e7a0 2024-09-16T01:40:47.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.188+0000 7fbc5b59e640 1 --2- 172.21.15.88:0/1744676611 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc5c0cf2f0 0x7fbc5c0cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:40:47.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.189+0000 7fbc53fff640 1 --2- 172.21.15.88:0/1744676611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc5c0ba240 0x7fbc5c0d3fe0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fbc440097a0 tx=0x7fbc4402a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6be9d9fb5c266c23 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:47.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.189+0000 7fbc5a59c640 1 -- 172.21.15.88:0/1744676611 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc44037070 con 0x7fbc5c0ba240 2024-09-16T01:40:47.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.189+0000 7fbc5a59c640 1 -- 172.21.15.88:0/1744676611 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc44032440 con 0x7fbc5c0ba240 2024-09-16T01:40:47.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.189+0000 7fbc5a59c640 1 -- 172.21.15.88:0/1744676611 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc44032810 con 0x7fbc5c0ba240 2024-09-16T01:40:47.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.190+0000 7fbc610b7640 1 -- 172.21.15.88:0/1744676611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc5c0ba240 msgr2=0x7fbc5c0d3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:47.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.190+0000 7fbc610b7640 1 --2- 172.21.15.88:0/1744676611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc5c0ba240 0x7fbc5c0d3fe0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fbc440097a0 tx=0x7fbc4402a210 comp rx=0 tx=0).stop 2024-09-16T01:40:47.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.190+0000 7fbc610b7640 1 -- 172.21.15.88:0/1744676611 shutdown_connections 2024-09-16T01:40:47.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.190+0000 7fbc610b7640 1 --2- 172.21.15.88:0/1744676611 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc5c0cf2f0 0x7fbc5c0cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:47.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.190+0000 7fbc610b7640 1 --2- 172.21.15.88:0/1744676611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc5c0ba240 0x7fbc5c0d3fe0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:47.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.190+0000 7fbc610b7640 1 -- 172.21.15.88:0/1744676611 >> 172.21.15.88:0/1744676611 conn(0x7fbc5c0d1550 msgr2=0x7fbc5c0d1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:47.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.190+0000 7fbc610b7640 1 -- 172.21.15.88:0/1744676611 shutdown_connections 2024-09-16T01:40:47.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.190+0000 7fbc610b7640 1 -- 172.21.15.88:0/1744676611 wait complete. 2024-09-16T01:40:47.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.191+0000 7fbc610b7640 1 Processor -- start 2024-09-16T01:40:47.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.191+0000 7fbc610b7640 1 -- start start 2024-09-16T01:40:47.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.192+0000 7fbc610b7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc5c0ba240 0x7fbc5c160920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:47.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.192+0000 7fbc610b7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc5c0cf2f0 0x7fbc5c1f15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:47.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.192+0000 7fbc610b7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc5c15e7a0 0x7fbc5c1f41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:47.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.192+0000 7fbc610b7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c15eb80 con 0x7fbc5c0ba240 2024-09-16T01:40:47.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.192+0000 7fbc610b7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbc5c0d3ac0 con 0x7fbc5c0cf2f0 2024-09-16T01:40:47.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.192+0000 7fbc610b7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbc5c15ed00 con 0x7fbc5c15e7a0 2024-09-16T01:40:47.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.192+0000 7fbc5ad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc5c0ba240 0x7fbc5c160920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:47.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.192+0000 7fbc5ad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc5c0ba240 0x7fbc5c160920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55564/0 (socket says 172.21.15.88:55564) 2024-09-16T01:40:47.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.192+0000 7fbc5ad9d640 1 -- 172.21.15.88:0/3106794282 learned_addr learned my addr 172.21.15.88:0/3106794282 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:47.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.192+0000 7fbc53fff640 1 --2- 172.21.15.88:0/3106794282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc5c0cf2f0 0x7fbc5c1f15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:47.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.192+0000 7fbc5b59e640 1 --2- 172.21.15.88:0/3106794282 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc5c15e7a0 0x7fbc5c1f41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:47.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.192+0000 7fbc5ad9d640 1 -- 172.21.15.88:0/3106794282 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc5c15e7a0 msgr2=0x7fbc5c1f41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:47.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.192+0000 7fbc5ad9d640 1 --2- 172.21.15.88:0/3106794282 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc5c15e7a0 0x7fbc5c1f41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:47.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.192+0000 7fbc5ad9d640 1 -- 172.21.15.88:0/3106794282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc5c0cf2f0 msgr2=0x7fbc5c1f15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:47.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.192+0000 7fbc5ad9d640 1 --2- 172.21.15.88:0/3106794282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc5c0cf2f0 0x7fbc5c1f15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:47.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.192+0000 7fbc5ad9d640 1 -- 172.21.15.88:0/3106794282 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc5c160e60 con 0x7fbc5c0ba240 2024-09-16T01:40:47.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.192+0000 7fbc5ad9d640 1 --2- 172.21.15.88:0/3106794282 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc5c0ba240 0x7fbc5c160920 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c00c430 tx=0x7fbc4c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:47.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.192+0000 7fbc537fe640 1 -- 172.21.15.88:0/3106794282 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c0074e0 con 0x7fbc5c0ba240 2024-09-16T01:40:47.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.192+0000 7fbc537fe640 1 -- 172.21.15.88:0/3106794282 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc4c0047e0 con 0x7fbc5c0ba240 2024-09-16T01:40:47.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.192+0000 7fbc537fe640 1 -- 172.21.15.88:0/3106794282 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c004bd0 con 0x7fbc5c0ba240 2024-09-16T01:40:47.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.192+0000 7fbc610b7640 1 -- 172.21.15.88:0/3106794282 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc5c1f4c90 con 0x7fbc5c0ba240 2024-09-16T01:40:47.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.193+0000 7fbc610b7640 1 -- 172.21.15.88:0/3106794282 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc5c160120 con 0x7fbc5c0ba240 2024-09-16T01:40:47.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.193+0000 7fbc610b7640 1 -- 172.21.15.88:0/3106794282 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc5c06cb50 con 0x7fbc5c0ba240 2024-09-16T01:40:47.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.194+0000 7fbc537fe640 1 -- 172.21.15.88:0/3106794282 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fbc4c011070 con 0x7fbc5c0ba240 2024-09-16T01:40:47.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.194+0000 7fbc537fe640 1 --2- 172.21.15.88:0/3106794282 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc3c06c2b0 0x7fbc3c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:47.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.194+0000 7fbc537fe640 1 -- 172.21.15.88:0/3106794282 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6225+0+0 (secure 0 0 0) 0x7fbc4c08b120 con 0x7fbc5c0ba240 2024-09-16T01:40:47.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.194+0000 7fbc53fff640 1 --2- 172.21.15.88:0/3106794282 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc3c06c2b0 0x7fbc3c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:47.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.195+0000 7fbc53fff640 1 --2- 172.21.15.88:0/3106794282 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc3c06c2b0 0x7fbc3c06e770 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fbc44034000 tx=0x7fbc4402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:47.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.198+0000 7fbc537fe640 1 -- 172.21.15.88:0/3106794282 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fbc5c06cb50 con 0x7fbc5c0ba240 2024-09-16T01:40:47.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.398+0000 7fbc610b7640 1 -- 172.21.15.88:0/3106794282 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc5c15ff30 con 0x7fbc5c0ba240 2024-09-16T01:40:47.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.399+0000 7fbc537fe640 1 -- 172.21.15.88:0/3106794282 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+3565 (secure 0 0 0) 0x7fbc4c059680 con 0x7fbc5c0ba240 2024-09-16T01:40:47.401 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:47.401 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":102,"btime":"2024-09-16T01:40:46:434694+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:40:46.434636+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:40:47.403 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 102 2024-09-16T01:40:47.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.401+0000 7fbc610b7640 1 -- 172.21.15.88:0/3106794282 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc3c06c2b0 msgr2=0x7fbc3c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:47.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.401+0000 7fbc610b7640 1 --2- 172.21.15.88:0/3106794282 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc3c06c2b0 0x7fbc3c06e770 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fbc44034000 tx=0x7fbc4402e000 comp rx=0 tx=0).stop 2024-09-16T01:40:47.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.401+0000 7fbc610b7640 1 -- 172.21.15.88:0/3106794282 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc5c0ba240 msgr2=0x7fbc5c160920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:47.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.401+0000 7fbc610b7640 1 --2- 172.21.15.88:0/3106794282 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc5c0ba240 0x7fbc5c160920 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c00c430 tx=0x7fbc4c00c830 comp rx=0 tx=0).stop 2024-09-16T01:40:47.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.401+0000 7fbc610b7640 1 -- 172.21.15.88:0/3106794282 shutdown_connections 2024-09-16T01:40:47.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.401+0000 7fbc610b7640 1 --2- 172.21.15.88:0/3106794282 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc3c06c2b0 0x7fbc3c06e770 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:47.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.401+0000 7fbc610b7640 1 --2- 172.21.15.88:0/3106794282 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc5c15e7a0 0x7fbc5c1f41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:47.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.401+0000 7fbc610b7640 1 --2- 172.21.15.88:0/3106794282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc5c0cf2f0 0x7fbc5c1f15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:47.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.401+0000 7fbc610b7640 1 --2- 172.21.15.88:0/3106794282 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc5c0ba240 0x7fbc5c160920 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:47.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.401+0000 7fbc610b7640 1 -- 172.21.15.88:0/3106794282 >> 172.21.15.88:0/3106794282 conn(0x7fbc5c0d1550 msgr2=0x7fbc5c06c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:47.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.401+0000 7fbc610b7640 1 -- 172.21.15.88:0/3106794282 shutdown_connections 2024-09-16T01:40:47.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.401+0000 7fbc610b7640 1 -- 172.21.15.88:0/3106794282 wait complete. 2024-09-16T01:40:47.414 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 200.00000001' led to DAMAGED state 2024-09-16T01:40:47.414 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete mds0_openfiles.0' 2024-09-16T01:40:47.414 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:40:47.415 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:40:47.417 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:40:47.419 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:40:47.433 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:40:47.433 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:40:47.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:47.434 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:40:47.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:40:47.524 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:40:47.524 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:40:47.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.661+0000 7f96fb9c4640 1 Processor -- start 2024-09-16T01:40:47.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.661+0000 7f96fb9c4640 1 -- start start 2024-09-16T01:40:47.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.662+0000 7f96fb9c4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f96f40731c0 0x7f96f4073600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:47.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.662+0000 7f96fb9c4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f96f415de80 0x7f96f415e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:47.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.662+0000 7f96fb9c4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96f415a430 con 0x7f96f415a050 2024-09-16T01:40:47.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.662+0000 7f96fb9c4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f96f4073b40 con 0x7f96f415de80 2024-09-16T01:40:47.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.662+0000 7f96fb9c4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f96f40cd0a0 con 0x7f96f40731c0 2024-09-16T01:40:47.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.662+0000 7f96f9f61640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f96f415de80 0x7f96f415e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:47.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.662+0000 7f96f9760640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f96f415a050 0x7f96f415d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59746/0 (socket says 172.21.15.88:59746) 2024-09-16T01:40:47.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.662+0000 7f96f9f61640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f96f415de80 0x7f96f415e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55574/0 (socket says 172.21.15.88:55574) 2024-09-16T01:40:47.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.662+0000 7f96f9760640 1 -- 172.21.15.88:0/1955092537 learned_addr learned my addr 172.21.15.88:0/1955092537 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:47.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.662+0000 7f96f8f5f640 1 --2- 172.21.15.88:0/1955092537 >> v2:172.21.15.154:3300/0 conn(0x7f96f40731c0 0x7f96f4073600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:47.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.662+0000 7f96f9f61640 1 -- 172.21.15.88:0/1955092537 >> v2:172.21.15.154:3300/0 conn(0x7f96f40731c0 msgr2=0x7f96f4073600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:47.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.662+0000 7f96f9f61640 1 --2- 172.21.15.88:0/1955092537 >> v2:172.21.15.154:3300/0 conn(0x7f96f40731c0 0x7f96f4073600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:47.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.662+0000 7f96f9f61640 1 -- 172.21.15.88:0/1955092537 >> v1:172.21.15.88:6789/0 conn(0x7f96f415a050 legacy=0x7f96f415d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:47.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.662+0000 7f96ebfff640 1 -- 172.21.15.88:0/1955092537 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2437566657 0 0) 0x7f96f415a430 con 0x7f96f415a050 2024-09-16T01:40:47.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.662+0000 7f96f9f61640 1 -- 172.21.15.88:0/1955092537 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96f40cd380 con 0x7f96f415de80 2024-09-16T01:40:47.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.662+0000 7f96f8f5f640 1 --2- 172.21.15.88:0/1955092537 >> v2:172.21.15.154:3300/0 conn(0x7f96f40731c0 0x7f96f4073600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:47.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.663+0000 7f96f9f61640 1 --2- 172.21.15.88:0/1955092537 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96f415de80 0x7f96f415e310 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f96f415db70 tx=0x7f96ec02a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70525fef9ee90530 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:47.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.663+0000 7f96ebfff640 1 -- 172.21.15.88:0/1955092537 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ec007740 con 0x7f96f415de80 2024-09-16T01:40:47.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.663+0000 7f96ebfff640 1 -- 172.21.15.88:0/1955092537 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96ec0078e0 con 0x7f96f415de80 2024-09-16T01:40:47.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.663+0000 7f96ebfff640 1 -- 172.21.15.88:0/1955092537 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ec007cb0 con 0x7f96f415de80 2024-09-16T01:40:47.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.664+0000 7f96fb9c4640 1 -- 172.21.15.88:0/1955092537 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96f415de80 msgr2=0x7f96f415e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:47.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.664+0000 7f96fb9c4640 1 --2- 172.21.15.88:0/1955092537 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96f415de80 0x7f96f415e310 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f96f415db70 tx=0x7f96ec02a5f0 comp rx=0 tx=0).stop 2024-09-16T01:40:47.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.664+0000 7f96fb9c4640 1 -- 172.21.15.88:0/1955092537 shutdown_connections 2024-09-16T01:40:47.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.664+0000 7f96fb9c4640 1 --2- 172.21.15.88:0/1955092537 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96f415de80 0x7f96f415e310 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:47.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.664+0000 7f96fb9c4640 1 --2- 172.21.15.88:0/1955092537 >> v2:172.21.15.154:3300/0 conn(0x7f96f40731c0 0x7f96f4073600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:47.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.664+0000 7f96fb9c4640 1 -- 172.21.15.88:0/1955092537 >> 172.21.15.88:0/1955092537 conn(0x7f96f406d0f0 msgr2=0x7f96f406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:47.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.664+0000 7f96fb9c4640 1 -- 172.21.15.88:0/1955092537 shutdown_connections 2024-09-16T01:40:47.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.664+0000 7f96fb9c4640 1 -- 172.21.15.88:0/1955092537 wait complete. 2024-09-16T01:40:47.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.665+0000 7f96fb9c4640 1 Processor -- start 2024-09-16T01:40:47.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.665+0000 7f96fb9c4640 1 -- start start 2024-09-16T01:40:47.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.666+0000 7f96fb9c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96f40731c0 0x7f96f4152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:47.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.666+0000 7f96fb9c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96f415a050 0x7f96f4158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:47.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.666+0000 7f96fb9c4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96f415de80 0x7f96f4158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:47.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.666+0000 7f96fb9c4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f96f40cdc20 con 0x7f96f415de80 2024-09-16T01:40:47.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.666+0000 7f96fb9c4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f96f40cd940 con 0x7f96f40731c0 2024-09-16T01:40:47.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.666+0000 7f96fb9c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f96f40b36e0 con 0x7f96f415a050 2024-09-16T01:40:47.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.666+0000 7f96f9f61640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96f415de80 0x7f96f4158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:47.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.666+0000 7f96f9f61640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96f415de80 0x7f96f4158990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55580/0 (socket says 172.21.15.88:55580) 2024-09-16T01:40:47.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.666+0000 7f96f9f61640 1 -- 172.21.15.88:0/3929421078 learned_addr learned my addr 172.21.15.88:0/3929421078 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:47.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.666+0000 7f96f9760640 1 --2- 172.21.15.88:0/3929421078 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96f40731c0 0x7f96f4152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:47.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.666+0000 7f96f8f5f640 1 --2- 172.21.15.88:0/3929421078 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96f415a050 0x7f96f4158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:47.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.666+0000 7f96f9f61640 1 -- 172.21.15.88:0/3929421078 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96f415a050 msgr2=0x7f96f4158450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:47.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.666+0000 7f96f9f61640 1 --2- 172.21.15.88:0/3929421078 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96f415a050 0x7f96f4158450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:47.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.666+0000 7f96f9f61640 1 -- 172.21.15.88:0/3929421078 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96f40731c0 msgr2=0x7f96f4152be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:47.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.666+0000 7f96f9f61640 1 --2- 172.21.15.88:0/3929421078 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96f40731c0 0x7f96f4152be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:47.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.666+0000 7f96f9f61640 1 -- 172.21.15.88:0/3929421078 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96f41540f0 con 0x7f96f415de80 2024-09-16T01:40:47.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.666+0000 7f96f9f61640 1 --2- 172.21.15.88:0/3929421078 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96f415de80 0x7f96f4158990 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f96ec009e80 tx=0x7f96ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:47.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.666+0000 7f96e9ffb640 1 -- 172.21.15.88:0/3929421078 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ec03f070 con 0x7f96f415de80 2024-09-16T01:40:47.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.667+0000 7f96e9ffb640 1 -- 172.21.15.88:0/3929421078 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96ec031c40 con 0x7f96f415de80 2024-09-16T01:40:47.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.667+0000 7f96e9ffb640 1 -- 172.21.15.88:0/3929421078 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ec003b90 con 0x7f96f415de80 2024-09-16T01:40:47.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.667+0000 7f96fb9c4640 1 -- 172.21.15.88:0/3929421078 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96f4153120 con 0x7f96f415de80 2024-09-16T01:40:47.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.667+0000 7f96fb9c4640 1 -- 172.21.15.88:0/3929421078 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96f4156db0 con 0x7f96f415de80 2024-09-16T01:40:47.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.668+0000 7f96fb9c4640 1 -- 172.21.15.88:0/3929421078 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96f40ccd00 con 0x7f96f415de80 2024-09-16T01:40:47.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.668+0000 7f96e9ffb640 1 -- 172.21.15.88:0/3929421078 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f96ec004030 con 0x7f96f415de80 2024-09-16T01:40:47.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.669+0000 7f96e9ffb640 1 --2- 172.21.15.88:0/3929421078 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f96d806c2b0 0x7f96d806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:47.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.669+0000 7f96e9ffb640 1 -- 172.21.15.88:0/3929421078 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6225+0+0 (secure 0 0 0) 0x7f96ec0ac7a0 con 0x7f96f415de80 2024-09-16T01:40:47.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.669+0000 7f96f9760640 1 --2- 172.21.15.88:0/3929421078 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f96d806c2b0 0x7f96d806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:47.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.669+0000 7f96f9760640 1 --2- 172.21.15.88:0/3929421078 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f96d806c2b0 0x7f96d806e770 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f96f0000f60 tx=0x7f96f0007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:47.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.673+0000 7f96e9ffb640 1 -- 172.21.15.88:0/3929421078 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f96ec07a740 con 0x7f96f415de80 2024-09-16T01:40:47.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:47.876+0000 7f96fb9c4640 1 -- 172.21.15.88:0/3929421078 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f96f40ce6f0 con 0x7f96f415de80 2024-09-16T01:40:48.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.440+0000 7f96e9ffb640 1 -- 172.21.15.88:0/3929421078 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v104) ==== 161+0+0 (secure 0 0 0) 0x7f96ec07ea40 con 0x7f96f415de80 2024-09-16T01:40:48.442 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:40:48.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.444+0000 7f96fb9c4640 1 -- 172.21.15.88:0/3929421078 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f96d806c2b0 msgr2=0x7f96d806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:48.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.444+0000 7f96fb9c4640 1 --2- 172.21.15.88:0/3929421078 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f96d806c2b0 0x7f96d806e770 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f96f0000f60 tx=0x7f96f0007d50 comp rx=0 tx=0).stop 2024-09-16T01:40:48.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.444+0000 7f96fb9c4640 1 -- 172.21.15.88:0/3929421078 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96f415de80 msgr2=0x7f96f4158990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:48.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.444+0000 7f96fb9c4640 1 --2- 172.21.15.88:0/3929421078 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96f415de80 0x7f96f4158990 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f96ec009e80 tx=0x7f96ec004a40 comp rx=0 tx=0).stop 2024-09-16T01:40:48.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.444+0000 7f96fb9c4640 1 -- 172.21.15.88:0/3929421078 shutdown_connections 2024-09-16T01:40:48.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.444+0000 7f96fb9c4640 1 --2- 172.21.15.88:0/3929421078 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f96d806c2b0 0x7f96d806e770 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:48.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.444+0000 7f96fb9c4640 1 --2- 172.21.15.88:0/3929421078 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96f415de80 0x7f96f4158990 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:48.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.444+0000 7f96fb9c4640 1 --2- 172.21.15.88:0/3929421078 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96f415a050 0x7f96f4158450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:48.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.444+0000 7f96fb9c4640 1 --2- 172.21.15.88:0/3929421078 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f96f40731c0 0x7f96f4152be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:48.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.444+0000 7f96fb9c4640 1 -- 172.21.15.88:0/3929421078 >> 172.21.15.88:0/3929421078 conn(0x7f96f406d0f0 msgr2=0x7f96f415ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:48.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.444+0000 7f96fb9c4640 1 -- 172.21.15.88:0/3929421078 shutdown_connections 2024-09-16T01:40:48.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.444+0000 7f96fb9c4640 1 -- 172.21.15.88:0/3929421078 wait complete. 2024-09-16T01:40:48.458 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:40:48.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.560+0000 7f16afa7b640 1 Processor -- start 2024-09-16T01:40:48.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.560+0000 7f16afa7b640 1 -- start start 2024-09-16T01:40:48.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.561+0000 7f16afa7b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f16a80cca40 0x7f16a80b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:48.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.561+0000 7f16afa7b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16a8071720 con 0x7f16a815ad10 2024-09-16T01:40:48.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.561+0000 7f16afa7b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16a8073920 con 0x7f16a80720d0 2024-09-16T01:40:48.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.561+0000 7f16afa7b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f16a8070fb0 con 0x7f16a80cca40 2024-09-16T01:40:48.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.561+0000 7f16ae018640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f16a80cca40 0x7f16a80b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:48.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.561+0000 7f16ae018640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f16a80cca40 0x7f16a80b45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55584/0 (socket says 172.21.15.88:55584) 2024-09-16T01:40:48.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.561+0000 7f16ae018640 1 -- 172.21.15.88:0/592353832 learned_addr learned my addr 172.21.15.88:0/592353832 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:48.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.561+0000 7f16ad817640 1 --1- 172.21.15.88:0/592353832 >> v1:172.21.15.88:6789/0 conn(0x7f16a815ad10 0x7f16a80719c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59756/0 (socket says 172.21.15.88:59756) 2024-09-16T01:40:48.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.561+0000 7f16ae018640 1 -- 172.21.15.88:0/592353832 >> v1:172.21.15.154:6789/0 conn(0x7f16a80720d0 legacy=0x7f16a80b3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:48.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.561+0000 7f16ae018640 1 -- 172.21.15.88:0/592353832 >> v1:172.21.15.88:6789/0 conn(0x7f16a815ad10 legacy=0x7f16a80719c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:48.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.561+0000 7f16ae018640 1 -- 172.21.15.88:0/592353832 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a8073c00 con 0x7f16a80cca40 2024-09-16T01:40:48.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.561+0000 7f16ac815640 1 -- 172.21.15.88:0/592353832 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3094723013 0 0) 0x7f16a8071720 con 0x7f16a815ad10 2024-09-16T01:40:48.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.562+0000 7f16ae018640 1 --2- 172.21.15.88:0/592353832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16a80cca40 0x7f16a80b45b0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009e50 tx=0x7f16a002a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6deb261f9a7cdc99 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:48.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.562+0000 7f16ac815640 1 -- 172.21.15.88:0/592353832 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a0036070 con 0x7f16a80cca40 2024-09-16T01:40:48.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.562+0000 7f16ac815640 1 -- 172.21.15.88:0/592353832 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16a0031600 con 0x7f16a80cca40 2024-09-16T01:40:48.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.562+0000 7f16ac815640 1 -- 172.21.15.88:0/592353832 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a00319d0 con 0x7f16a80cca40 2024-09-16T01:40:48.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.562+0000 7f16afa7b640 1 -- 172.21.15.88:0/592353832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16a80cca40 msgr2=0x7f16a80b45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:48.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.562+0000 7f16afa7b640 1 --2- 172.21.15.88:0/592353832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16a80cca40 0x7f16a80b45b0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009e50 tx=0x7f16a002a6d0 comp rx=0 tx=0).stop 2024-09-16T01:40:48.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.563+0000 7f16afa7b640 1 -- 172.21.15.88:0/592353832 shutdown_connections 2024-09-16T01:40:48.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.563+0000 7f16afa7b640 1 --2- 172.21.15.88:0/592353832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16a80cca40 0x7f16a80b45b0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:48.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.563+0000 7f16afa7b640 1 -- 172.21.15.88:0/592353832 >> 172.21.15.88:0/592353832 conn(0x7f16a806d0f0 msgr2=0x7f16a806d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:48.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.563+0000 7f16afa7b640 1 -- 172.21.15.88:0/592353832 shutdown_connections 2024-09-16T01:40:48.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.563+0000 7f16afa7b640 1 -- 172.21.15.88:0/592353832 wait complete. 2024-09-16T01:40:48.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.564+0000 7f16afa7b640 1 Processor -- start 2024-09-16T01:40:48.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.564+0000 7f16afa7b640 1 -- start start 2024-09-16T01:40:48.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.564+0000 7f16afa7b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16a80720d0 0x7f16a81f13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:48.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.564+0000 7f16afa7b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16a80cca40 0x7f16a8155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:48.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.564+0000 7f16afa7b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16a815ad10 0x7f16a80c5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:48.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.564+0000 7f16afa7b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f16a80cd770 con 0x7f16a80720d0 2024-09-16T01:40:48.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.564+0000 7f16afa7b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f16a8071200 con 0x7f16a815ad10 2024-09-16T01:40:48.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.564+0000 7f16afa7b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f16a80cd8f0 con 0x7f16a80cca40 2024-09-16T01:40:48.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.565+0000 7f16ad817640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16a80720d0 0x7f16a81f13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:48.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.565+0000 7f16ad817640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16a80720d0 0x7f16a81f13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55588/0 (socket says 172.21.15.88:55588) 2024-09-16T01:40:48.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.565+0000 7f16ad817640 1 -- 172.21.15.88:0/556098190 learned_addr learned my addr 172.21.15.88:0/556098190 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:48.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.565+0000 7f16ad016640 1 --2- 172.21.15.88:0/556098190 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16a80cca40 0x7f16a8155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:48.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.565+0000 7f16ad817640 1 -- 172.21.15.88:0/556098190 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16a80cca40 msgr2=0x7f16a8155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.565+0000 7f16ae018640 1 --2- 172.21.15.88:0/556098190 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16a815ad10 0x7f16a80c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.565+0000 7f16ad817640 1 --2- 172.21.15.88:0/556098190 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16a80cca40 0x7f16a8155190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.565+0000 7f16ad817640 1 -- 172.21.15.88:0/556098190 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16a815ad10 msgr2=0x7f16a80c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.565+0000 7f16ad817640 1 --2- 172.21.15.88:0/556098190 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16a815ad10 0x7f16a80c5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.565+0000 7f16ad817640 1 -- 172.21.15.88:0/556098190 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a80c5c50 con 0x7f16a80720d0 2024-09-16T01:40:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.565+0000 7f16ad817640 1 --2- 172.21.15.88:0/556098190 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16a80720d0 0x7f16a81f13f0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f16a400c430 tx=0x7f16a400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.565+0000 7f169e7fc640 1 -- 172.21.15.88:0/556098190 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a40074e0 con 0x7f16a80720d0 2024-09-16T01:40:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.565+0000 7f169e7fc640 1 -- 172.21.15.88:0/556098190 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16a40047e0 con 0x7f16a80720d0 2024-09-16T01:40:48.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.565+0000 7f16afa7b640 1 -- 172.21.15.88:0/556098190 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16a80c5f30 con 0x7f16a80720d0 2024-09-16T01:40:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.565+0000 7f169e7fc640 1 -- 172.21.15.88:0/556098190 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a4004bd0 con 0x7f16a80720d0 2024-09-16T01:40:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.565+0000 7f16afa7b640 1 -- 172.21.15.88:0/556098190 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16a80c64f0 con 0x7f16a80720d0 2024-09-16T01:40:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.566+0000 7f16afa7b640 1 -- 172.21.15.88:0/556098190 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16a80d00b0 con 0x7f16a80720d0 2024-09-16T01:40:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.567+0000 7f169e7fc640 1 -- 172.21.15.88:0/556098190 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f16a4011070 con 0x7f16a80720d0 2024-09-16T01:40:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.567+0000 7f169e7fc640 1 --2- 172.21.15.88:0/556098190 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f168c06c2b0 0x7f168c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.567+0000 7f169e7fc640 1 -- 172.21.15.88:0/556098190 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6225+0+0 (secure 0 0 0) 0x7f16a408b250 con 0x7f16a80720d0 2024-09-16T01:40:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.567+0000 7f16ad016640 1 --2- 172.21.15.88:0/556098190 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f168c06c2b0 0x7f168c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:48.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.567+0000 7f16ad016640 1 --2- 172.21.15.88:0/556098190 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f168c06c2b0 0x7f168c06e770 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f16a80c75c0 tx=0x7f1698008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:48.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.571+0000 7f169e7fc640 1 -- 172.21.15.88:0/556098190 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f16a4059840 con 0x7f16a80720d0 2024-09-16T01:40:48.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:48.803+0000 7f16afa7b640 1 -- 172.21.15.88:0/556098190 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f16a80cbc00 con 0x7f16a80720d0 2024-09-16T01:40:49.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.450+0000 7f169e7fc640 1 -- 172.21.15.88:0/556098190 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v105) ==== 101+0+0 (secure 0 0 0) 0x7f16a4059140 con 0x7f16a80720d0 2024-09-16T01:40:49.451 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:40:49.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.452+0000 7f16afa7b640 1 -- 172.21.15.88:0/556098190 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f168c06c2b0 msgr2=0x7f168c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:49.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.452+0000 7f16afa7b640 1 --2- 172.21.15.88:0/556098190 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f168c06c2b0 0x7f168c06e770 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f16a80c75c0 tx=0x7f1698008040 comp rx=0 tx=0).stop 2024-09-16T01:40:49.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.452+0000 7f16afa7b640 1 -- 172.21.15.88:0/556098190 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16a80720d0 msgr2=0x7f16a81f13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:49.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.452+0000 7f16afa7b640 1 --2- 172.21.15.88:0/556098190 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16a80720d0 0x7f16a81f13f0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f16a400c430 tx=0x7f16a400c8f0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.452+0000 7f16afa7b640 1 -- 172.21.15.88:0/556098190 shutdown_connections 2024-09-16T01:40:49.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.452+0000 7f16afa7b640 1 --2- 172.21.15.88:0/556098190 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f168c06c2b0 0x7f168c06e770 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.452+0000 7f16afa7b640 1 --2- 172.21.15.88:0/556098190 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16a815ad10 0x7f16a80c5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.452+0000 7f16afa7b640 1 --2- 172.21.15.88:0/556098190 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f16a80cca40 0x7f16a8155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.452+0000 7f16afa7b640 1 --2- 172.21.15.88:0/556098190 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16a80720d0 0x7f16a81f13f0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.452+0000 7f16afa7b640 1 -- 172.21.15.88:0/556098190 >> 172.21.15.88:0/556098190 conn(0x7f16a806d0f0 msgr2=0x7f16a80cf3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:49.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.453+0000 7f16afa7b640 1 -- 172.21.15.88:0/556098190 shutdown_connections 2024-09-16T01:40:49.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.453+0000 7f16afa7b640 1 -- 172.21.15.88:0/556098190 wait complete. 2024-09-16T01:40:49.466 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:40:49.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.509+0000 7ff233520f40 1 Processor -- start 2024-09-16T01:40:49.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.509+0000 7ff233520f40 1 -- start start 2024-09-16T01:40:49.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.510+0000 7ff233520f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b14c570e10 0x55b14c60ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.510+0000 7ff233520f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b14c571a60 0x55b14c611280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.510+0000 7ff233520f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b14c58b250 0x55b14c616400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.510+0000 7ff233520f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b14c571220 con 0x55b14c570e10 2024-09-16T01:40:49.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.510+0000 7ff233520f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b14c61a120 con 0x55b14c58b250 2024-09-16T01:40:49.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.510+0000 7ff233520f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55b14c61a3c0 con 0x55b14c571a60 2024-09-16T01:40:49.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.510+0000 7ff2305af640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b14c570e10 0x55b14c60ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.510+0000 7ff2305af640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b14c570e10 0x55b14c60ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55596/0 (socket says 172.21.15.88:55596) 2024-09-16T01:40:49.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.510+0000 7ff2305af640 1 -- 172.21.15.88:0/1933895897 learned_addr learned my addr 172.21.15.88:0/1933895897 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:49.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.510+0000 7ff230db0640 1 --2- 172.21.15.88:0/1933895897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b14c58b250 0x55b14c616400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.510+0000 7ff22fdae640 1 --2- 172.21.15.88:0/1933895897 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b14c571a60 0x55b14c611280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.510+0000 7ff2305af640 1 -- 172.21.15.88:0/1933895897 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b14c571a60 msgr2=0x55b14c611280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:49.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.510+0000 7ff2305af640 1 --2- 172.21.15.88:0/1933895897 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b14c571a60 0x55b14c611280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.510+0000 7ff2305af640 1 -- 172.21.15.88:0/1933895897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b14c58b250 msgr2=0x55b14c616400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:49.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.510+0000 7ff2305af640 1 --2- 172.21.15.88:0/1933895897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b14c58b250 0x55b14c616400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.510+0000 7ff2305af640 1 -- 172.21.15.88:0/1933895897 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b14c46ccb0 con 0x55b14c570e10 2024-09-16T01:40:49.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.510+0000 7ff230db0640 1 --2- 172.21.15.88:0/1933895897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b14c58b250 0x55b14c616400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:49.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.510+0000 7ff2305af640 1 --2- 172.21.15.88:0/1933895897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b14c570e10 0x55b14c60ef80 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7ff22000a470 tx=0x7ff22000a930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f99877012afbfd18 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:49.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.510+0000 7ff22edac640 1 -- 172.21.15.88:0/1933895897 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22000b930 con 0x55b14c570e10 2024-09-16T01:40:49.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.510+0000 7ff22edac640 1 -- 172.21.15.88:0/1933895897 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff22000f070 con 0x55b14c570e10 2024-09-16T01:40:49.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.510+0000 7ff22edac640 1 -- 172.21.15.88:0/1933895897 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22000bd30 con 0x55b14c570e10 2024-09-16T01:40:49.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.511+0000 7ff233520f40 1 -- 172.21.15.88:0/1933895897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b14c570e10 msgr2=0x55b14c60ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:49.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.511+0000 7ff233520f40 1 --2- 172.21.15.88:0/1933895897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b14c570e10 0x55b14c60ef80 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7ff22000a470 tx=0x7ff22000a930 comp rx=0 tx=0).stop 2024-09-16T01:40:49.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.511+0000 7ff233520f40 1 -- 172.21.15.88:0/1933895897 shutdown_connections 2024-09-16T01:40:49.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.511+0000 7ff233520f40 1 --2- 172.21.15.88:0/1933895897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b14c58b250 0x55b14c616400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.511+0000 7ff233520f40 1 --2- 172.21.15.88:0/1933895897 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b14c571a60 0x55b14c611280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.511+0000 7ff233520f40 1 --2- 172.21.15.88:0/1933895897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b14c570e10 0x55b14c60ef80 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.511+0000 7ff233520f40 1 -- 172.21.15.88:0/1933895897 >> 172.21.15.88:0/1933895897 conn(0x55b14c58c750 msgr2=0x55b14c58bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:49.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.511+0000 7ff233520f40 1 -- 172.21.15.88:0/1933895897 shutdown_connections 2024-09-16T01:40:49.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.511+0000 7ff233520f40 1 -- 172.21.15.88:0/1933895897 wait complete. 2024-09-16T01:40:49.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.511+0000 7ff233520f40 1 Processor -- start 2024-09-16T01:40:49.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.511+0000 7ff233520f40 1 -- start start 2024-09-16T01:40:49.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.512+0000 7ff233520f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b14c570e10 0x55b14c614540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.512+0000 7ff233520f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b14c571a60 0x55b14c605640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.512+0000 7ff233520f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b14c58b250 0x55b14c609eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.512+0000 7ff233520f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b14c58ac10 con 0x55b14c570e10 2024-09-16T01:40:49.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.512+0000 7ff233520f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b14c60f4c0 con 0x55b14c571a60 2024-09-16T01:40:49.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.512+0000 7ff233520f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55b14c60f870 con 0x55b14c58b250 2024-09-16T01:40:49.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.512+0000 7ff2305af640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b14c570e10 0x55b14c614540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.512+0000 7ff2305af640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b14c570e10 0x55b14c614540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55610/0 (socket says 172.21.15.88:55610) 2024-09-16T01:40:49.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.512+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 learned_addr learned my addr 172.21.15.88:0/1740977581 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:49.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.512+0000 7ff22fdae640 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b14c571a60 0x55b14c605640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.512+0000 7ff230db0640 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b14c58b250 0x55b14c609eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.512+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b14c58b250 msgr2=0x55b14c609eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:49.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.512+0000 7ff2305af640 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b14c58b250 0x55b14c609eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.512+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b14c571a60 msgr2=0x55b14c605640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:49.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.512+0000 7ff2305af640 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b14c571a60 0x55b14c605640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.512+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b14c605b80 con 0x55b14c570e10 2024-09-16T01:40:49.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.513+0000 7ff2305af640 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b14c570e10 0x55b14c614540 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x55b14c610800 tx=0x7ff220002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:49.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.513+0000 7ff22ddaa640 1 -- 172.21.15.88:0/1740977581 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff220024070 con 0x55b14c570e10 2024-09-16T01:40:49.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.513+0000 7ff22ddaa640 1 -- 172.21.15.88:0/1740977581 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2200148a0 con 0x55b14c570e10 2024-09-16T01:40:49.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.513+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b14c6b4040 con 0x55b14c570e10 2024-09-16T01:40:49.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.513+0000 7ff22ddaa640 1 -- 172.21.15.88:0/1740977581 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff220014c90 con 0x55b14c570e10 2024-09-16T01:40:49.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.513+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b14c6093d0 con 0x55b14c570e10 2024-09-16T01:40:49.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.514+0000 7ff22ddaa640 1 -- 172.21.15.88:0/1740977581 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff220027070 con 0x55b14c570e10 2024-09-16T01:40:49.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.515+0000 7ff22ddaa640 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff21c06ac90 0x7ff21c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.515+0000 7ff22ddaa640 1 -- 172.21.15.88:0/1740977581 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6225+0+0 (secure 0 0 0) 0x7ff22008b310 con 0x55b14c570e10 2024-09-16T01:40:49.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.515+0000 7ff22fdae640 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff21c06ac90 0x7ff21c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.522 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:40:49.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.515+0000 7ff233520f40 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55b14c60a3f0 0x55b14c617450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.515+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c58a5a0 con 0x55b14c60a3f0 2024-09-16T01:40:49.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.515+0000 7ff230db0640 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55b14c60a3f0 0x55b14c617450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.515+0000 7ff22fdae640 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff21c06ac90 0x7ff21c06d150 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff224005fd0 tx=0x7ff22400c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:49.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.515+0000 7ff230db0640 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55b14c60a3f0 0x55b14c617450 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:49.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.517+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v91'164 uv138 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.517+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e91) -- 0x55b14c617990 con 0x55b14c60a3f0 2024-09-16T01:40:49.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.518+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v91'165 uv165 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.518+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bc2d0 con 0x55b14c60a3f0 2024-09-16T01:40:49.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.520+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v91'166 uv166 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.524 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:40:49.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.520+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bd680 con 0x55b14c60a3f0 2024-09-16T01:40:49.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.521+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v91'167 uv167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.521+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bda20 con 0x55b14c60a3f0 2024-09-16T01:40:49.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.522+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v91'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.522+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bc670 con 0x55b14c60a3f0 2024-09-16T01:40:49.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.523+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v91'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.523+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bca10 con 0x55b14c60a3f0 2024-09-16T01:40:49.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.524+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v91'170 uv144 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.524+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bcdb0 con 0x55b14c60a3f0 2024-09-16T01:40:49.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.526+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v91'171 uv171 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.526+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bcdb0 con 0x55b14c60a3f0 2024-09-16T01:40:49.529 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:40:49.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.527+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v91'172 uv172 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.527+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bcdb0 con 0x55b14c60a3f0 2024-09-16T01:40:49.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.529+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v91'173 uv173 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.529+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bcdb0 con 0x55b14c60a3f0 2024-09-16T01:40:49.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.530+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v91'174 uv174 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.530+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bcdb0 con 0x55b14c60a3f0 2024-09-16T01:40:49.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.531+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v91'175 uv175 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.531+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bcdb0 con 0x55b14c60a3f0 2024-09-16T01:40:49.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.533+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v91'176 uv151 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.533+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bcdb0 con 0x55b14c60a3f0 2024-09-16T01:40:49.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.534+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v91'177 uv177 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.534+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bcdb0 con 0x55b14c60a3f0 2024-09-16T01:40:49.536 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:40:49.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.535+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v91'178 uv178 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.535+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bcdb0 con 0x55b14c60a3f0 2024-09-16T01:40:49.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.536+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v91'179 uv179 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.537+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bcdb0 con 0x55b14c60a3f0 2024-09-16T01:40:49.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.538+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v91'180 uv180 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.538+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bcdb0 con 0x55b14c60a3f0 2024-09-16T01:40:49.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.539+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v91'181 uv181 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.539+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bd200 con 0x55b14c60a3f0 2024-09-16T01:40:49.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.540+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v91'182 uv182 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.540+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bd190 con 0x55b14c60a3f0 2024-09-16T01:40:49.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.542+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v91'183 uv157 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.542+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bd190 con 0x55b14c60a3f0 2024-09-16T01:40:49.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.543+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v91'184 uv184 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.543+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bd190 con 0x55b14c60a3f0 2024-09-16T01:40:49.545 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:40:49.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.544+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v91'185 uv185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.544+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bd190 con 0x55b14c60a3f0 2024-09-16T01:40:49.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.546+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v91'186 uv186 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.546+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bd190 con 0x55b14c60a3f0 2024-09-16T01:40:49.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.547+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v91'187 uv187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.547+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bd060 con 0x55b14c60a3f0 2024-09-16T01:40:49.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.548+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v91'188 uv188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.548+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bd060 con 0x55b14c60a3f0 2024-09-16T01:40:49.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.550+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v91'189 uv163 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.550+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bd060 con 0x55b14c60a3f0 2024-09-16T01:40:49.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.551+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v91'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.551+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bd060 con 0x55b14c60a3f0 2024-09-16T01:40:49.553 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:40:49.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.552+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v91'191 uv191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.552+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bd060 con 0x55b14c60a3f0 2024-09-16T01:40:49.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.553+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v91'192 uv192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.553+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bd060 con 0x55b14c60a3f0 2024-09-16T01:40:49.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.554+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v91'193 uv193 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.554+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6bd060 con 0x55b14c60a3f0 2024-09-16T01:40:49.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.556+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v91'194 uv194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.556+0000 7ff233520f40 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55b14c6bddc0 0x55b14c6c01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.556+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c06f0 con 0x55b14c6bddc0 2024-09-16T01:40:49.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.556+0000 7ff2305af640 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55b14c6bddc0 0x55b14c6c01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.556+0000 7ff2305af640 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55b14c6bddc0 0x55b14c6c01b0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:49.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.558+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v91'130 uv114 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7ff220026030 con 0x55b14c6bddc0 2024-09-16T01:40:49.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.558+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c0a90 con 0x55b14c6bddc0 2024-09-16T01:40:49.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.559+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v91'131 uv131 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff220026030 con 0x55b14c6bddc0 2024-09-16T01:40:49.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.559+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c0e30 con 0x55b14c6bddc0 2024-09-16T01:40:49.561 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:40:49.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.560+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v91'132 uv132 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff220026030 con 0x55b14c6bddc0 2024-09-16T01:40:49.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.560+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c11d0 con 0x55b14c6bddc0 2024-09-16T01:40:49.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.562+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v91'133 uv133 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff220026030 con 0x55b14c6bddc0 2024-09-16T01:40:49.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.562+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c1570 con 0x55b14c6bddc0 2024-09-16T01:40:49.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.563+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v91'134 uv134 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff220026030 con 0x55b14c6bddc0 2024-09-16T01:40:49.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.563+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c1910 con 0x55b14c6bddc0 2024-09-16T01:40:49.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.564+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v91'135 uv117 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff220026030 con 0x55b14c6bddc0 2024-09-16T01:40:49.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.564+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c1cb0 con 0x55b14c6bddc0 2024-09-16T01:40:49.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.565+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v91'136 uv136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff220026030 con 0x55b14c6bddc0 2024-09-16T01:40:49.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.565+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c1cb0 con 0x55b14c6bddc0 2024-09-16T01:40:49.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.566+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v91'137 uv137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff220026030 con 0x55b14c6bddc0 2024-09-16T01:40:49.568 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:40:49.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.566+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c1cb0 con 0x55b14c6bddc0 2024-09-16T01:40:49.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.567+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v91'138 uv138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff220026030 con 0x55b14c6bddc0 2024-09-16T01:40:49.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.567+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c1cb0 con 0x55b14c6bddc0 2024-09-16T01:40:49.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.569+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v91'139 uv139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff220026030 con 0x55b14c6bddc0 2024-09-16T01:40:49.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.569+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c1cb0 con 0x55b14c6bddc0 2024-09-16T01:40:49.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.569+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v91'140 uv140 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff220026030 con 0x55b14c6bddc0 2024-09-16T01:40:49.572 INFO:teuthology.orchestra.run.smithi088.stdout:Write #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:40:49.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.570+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c6130 con 0x55b14c6bddc0 2024-09-16T01:40:49.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.571+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [write 0~17490] v91'141 uv141 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff220026030 con 0x55b14c6bddc0 2024-09-16T01:40:49.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.571+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c6130 con 0x55b14c6bddc0 2024-09-16T01:40:49.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.572+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [omap-set-header] v91'142 uv142 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff220026030 con 0x55b14c6bddc0 2024-09-16T01:40:49.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.572+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c1cb0 con 0x55b14c6bddc0 2024-09-16T01:40:49.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.573+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 606.00000000 [create] v91'143 uv128 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff220026030 con 0x55b14c6bddc0 2024-09-16T01:40:49.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.573+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c1cb0 con 0x55b14c6bddc0 2024-09-16T01:40:49.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.574+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 606.00000000 [delete] v91'144 uv144 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff220026030 con 0x55b14c6bddc0 2024-09-16T01:40:49.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.574+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c1cb0 con 0x55b14c6bddc0 2024-09-16T01:40:49.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.575+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v91'145 uv145 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff220026030 con 0x55b14c6bddc0 2024-09-16T01:40:49.577 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:40:49.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.575+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c60c6b0 con 0x55b14c6bddc0 2024-09-16T01:40:49.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.576+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [setxattr (30)] v91'146 uv146 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff220026030 con 0x55b14c6bddc0 2024-09-16T01:40:49.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.576+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c60c6b0 con 0x55b14c6bddc0 2024-09-16T01:40:49.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.577+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [setxattr (62)] v91'147 uv147 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff220026030 con 0x55b14c6bddc0 2024-09-16T01:40:49.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.577+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c60c6b0 con 0x55b14c6bddc0 2024-09-16T01:40:49.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.578+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [omap-set-header] v91'148 uv148 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff220026030 con 0x55b14c6bddc0 2024-09-16T01:40:49.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.578+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:51 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c60c6b0 con 0x55b14c60a3f0 2024-09-16T01:40:49.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.579+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(51 607.00000000 [create] v91'33 uv32 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.579+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:52 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e91) -- 0x55b14c60c6b0 con 0x55b14c60a3f0 2024-09-16T01:40:49.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.580+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(52 607.00000000 [delete] v91'34 uv34 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.580+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c60c6b0 con 0x55b14c60a3f0 2024-09-16T01:40:49.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.581+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(53 607.00000000 [create] v91'35 uv35 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.583 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:40:49.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.581+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c60c6b0 con 0x55b14c60a3f0 2024-09-16T01:40:49.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.582+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(54 607.00000000 [setxattr (30)] v91'36 uv36 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.582+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c60c6b0 con 0x55b14c60a3f0 2024-09-16T01:40:49.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.583+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(55 607.00000000 [setxattr (62)] v91'37 uv37 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.583+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c60c6b0 con 0x55b14c60a3f0 2024-09-16T01:40:49.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.584+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(56 607.00000000 [omap-set-header] v91'38 uv38 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.584+0000 7ff233520f40 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55b14c6c2670 0x55b14c6c4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.584+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:57 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c4fa0 con 0x55b14c6c2670 2024-09-16T01:40:49.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.584+0000 7ff22fdae640 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55b14c6c2670 0x55b14c6c4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.585+0000 7ff22fdae640 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55b14c6c2670 0x55b14c6c4a60 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:49.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.586+0000 7ff22fdae640 1 -- 172.21.15.88:0/1740977581 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(57 mds0_openfiles.0 [create] v91'26 uv25 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7ff224008830 con 0x55b14c6c2670 2024-09-16T01:40:49.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.586+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:58 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c5440 con 0x55b14c6c2670 2024-09-16T01:40:49.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.587+0000 7ff22fdae640 1 -- 172.21.15.88:0/1740977581 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(58 mds0_openfiles.0 [delete] v91'27 uv27 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff224008830 con 0x55b14c6c2670 2024-09-16T01:40:49.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.587+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c5940 con 0x55b14c6c2670 2024-09-16T01:40:49.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.587+0000 7ff22fdae640 1 -- 172.21.15.88:0/1740977581 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(59 mds0_openfiles.0 [create] v91'28 uv28 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff224008830 con 0x55b14c6c2670 2024-09-16T01:40:49.590 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:40:49.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.587+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c1d90 con 0x55b14c6c2670 2024-09-16T01:40:49.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.588+0000 7ff22fdae640 1 -- 172.21.15.88:0/1740977581 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(60 mds0_openfiles.0 [omap-set-header] v91'29 uv29 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff224008830 con 0x55b14c6c2670 2024-09-16T01:40:49.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.588+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:61 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c6270 con 0x55b14c60a3f0 2024-09-16T01:40:49.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.590+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(61 608.00000000 [create] v91'33 uv32 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.590+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:62 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c6270 con 0x55b14c60a3f0 2024-09-16T01:40:49.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.591+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(62 608.00000000 [delete] v91'34 uv34 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.591+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c6270 con 0x55b14c60a3f0 2024-09-16T01:40:49.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.592+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(63 608.00000000 [create] v91'35 uv35 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.593 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:40:49.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.592+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c7280 con 0x55b14c60a3f0 2024-09-16T01:40:49.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.593+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(64 608.00000000 [setxattr (30)] v91'36 uv36 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.593+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c7280 con 0x55b14c60a3f0 2024-09-16T01:40:49.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.594+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(65 608.00000000 [setxattr (62)] v91'37 uv37 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.594+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c6270 con 0x55b14c60a3f0 2024-09-16T01:40:49.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.595+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(66 608.00000000 [omap-set-header] v91'38 uv38 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.595+0000 7ff233520f40 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b14c6c6b40 0x55b14c6c6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.595+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:67 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c9650 con 0x55b14c6c6b40 2024-09-16T01:40:49.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.595+0000 7ff230db0640 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b14c6c6b40 0x55b14c6c6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.596+0000 7ff230db0640 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b14c6c6b40 0x55b14c6c6f20 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:49.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.597+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(67 604.00000000 [create] v91'59 uv53 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.597+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:68 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c9650 con 0x55b14c6c6b40 2024-09-16T01:40:49.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.598+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(68 604.00000000 [delete] v91'60 uv60 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.598+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c9650 con 0x55b14c6c6b40 2024-09-16T01:40:49.600 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:40:49.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.599+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(69 604.00000000 [create] v91'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.599+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6ca660 con 0x55b14c6c6b40 2024-09-16T01:40:49.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.600+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(70 604.00000000 [setxattr (30)] v91'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.600+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6ca660 con 0x55b14c6c6b40 2024-09-16T01:40:49.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.603+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(71 604.00000000 [setxattr (62)] v91'63 uv63 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.603+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c9650 con 0x55b14c6c6b40 2024-09-16T01:40:49.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.605+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(72 604.00000000 [omap-set-header] v91'64 uv64 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.606+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c9650 con 0x55b14c6c6b40 2024-09-16T01:40:49.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.608+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(73 500.00000000 [create] v91'65 uv58 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.608+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c9650 con 0x55b14c6c6b40 2024-09-16T01:40:49.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.611+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(74 500.00000000 [delete] v91'66 uv66 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.611+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c9650 con 0x55b14c6c6b40 2024-09-16T01:40:49.615 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:40:49.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.614+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(75 500.00000000 [create] v91'67 uv67 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.614+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c9650 con 0x55b14c6c6b40 2024-09-16T01:40:49.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.617+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(76 500.00000000 [write 0~90] v91'68 uv68 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.617+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6c9650 con 0x55b14c6c6b40 2024-09-16T01:40:49.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.619+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(77 500.00000000 [omap-set-header] v91'69 uv69 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.620+0000 7ff233520f40 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55b14c6c9f70 0x55b14c6ca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.620+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:78 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6cca50 con 0x55b14c6c9f70 2024-09-16T01:40:49.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.620+0000 7ff2305af640 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55b14c6c9f70 0x55b14c6ca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.620+0000 7ff2305af640 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55b14c6c9f70 0x55b14c6ca3d0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:49.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.621+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(78 mds_snaptable [create] v91'27 uv26 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7ff220026030 con 0x55b14c6c9f70 2024-09-16T01:40:49.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.621+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:79 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6cca50 con 0x55b14c6c9f70 2024-09-16T01:40:49.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.622+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(79 mds_snaptable [delete] v91'28 uv28 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff220026030 con 0x55b14c6c9f70 2024-09-16T01:40:49.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.622+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6ccad0 con 0x55b14c6c9f70 2024-09-16T01:40:49.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.623+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(80 mds_snaptable [create] v91'29 uv29 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff220026030 con 0x55b14c6c9f70 2024-09-16T01:40:49.624 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:40:49.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.623+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6ccec0 con 0x55b14c6c9f70 2024-09-16T01:40:49.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.624+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(81 mds_snaptable [write 0~70] v91'30 uv30 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff220026030 con 0x55b14c6c9f70 2024-09-16T01:40:49.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.624+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6ccec0 con 0x55b14c6c9f70 2024-09-16T01:40:49.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.625+0000 7ff2305af640 1 -- 172.21.15.88:0/1740977581 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(82 mds_snaptable [omap-set-header] v91'31 uv31 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff220026030 con 0x55b14c6c9f70 2024-09-16T01:40:49.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.625+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:83 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6ccec0 con 0x55b14c60a3f0 2024-09-16T01:40:49.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.626+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(83 605.00000000 [create] v91'33 uv32 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.626+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:84 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6ccec0 con 0x55b14c60a3f0 2024-09-16T01:40:49.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.627+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(84 605.00000000 [delete] v91'34 uv34 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.627+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6ccec0 con 0x55b14c60a3f0 2024-09-16T01:40:49.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.629+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(85 605.00000000 [create] v91'35 uv35 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.631 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:40:49.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.629+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6cded0 con 0x55b14c60a3f0 2024-09-16T01:40:49.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.632+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(86 605.00000000 [setxattr (30)] v91'36 uv36 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.632+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6cded0 con 0x55b14c60a3f0 2024-09-16T01:40:49.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.634+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(87 605.00000000 [setxattr (62)] v91'37 uv37 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.634+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6ccec0 con 0x55b14c60a3f0 2024-09-16T01:40:49.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.637+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(88 605.00000000 [omap-set-header] v91'38 uv38 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c60a3f0 2024-09-16T01:40:49.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.637+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:89 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6ccec0 con 0x55b14c6c2670 2024-09-16T01:40:49.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.640+0000 7ff22fdae640 1 -- 172.21.15.88:0/1740977581 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(89 mds0_inotable [create] v91'54 uv48 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7ff224008830 con 0x55b14c6c2670 2024-09-16T01:40:49.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.640+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:90 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6cd260 con 0x55b14c6c2670 2024-09-16T01:40:49.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.641+0000 7ff22fdae640 1 -- 172.21.15.88:0/1740977581 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(90 mds0_inotable [delete] v91'55 uv55 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff224008830 con 0x55b14c6c2670 2024-09-16T01:40:49.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.641+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6cd600 con 0x55b14c6c2670 2024-09-16T01:40:49.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.642+0000 7ff22fdae640 1 -- 172.21.15.88:0/1740977581 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(91 mds0_inotable [create] v91'56 uv56 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff224008830 con 0x55b14c6c2670 2024-09-16T01:40:49.643 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:40:49.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.642+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6cd600 con 0x55b14c6c2670 2024-09-16T01:40:49.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.642+0000 7ff22fdae640 1 -- 172.21.15.88:0/1740977581 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(92 mds0_inotable [write 0~34] v91'57 uv57 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff224008830 con 0x55b14c6c2670 2024-09-16T01:40:49.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.642+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6cd600 con 0x55b14c6c2670 2024-09-16T01:40:49.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.643+0000 7ff22fdae640 1 -- 172.21.15.88:0/1740977581 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(93 mds0_inotable [omap-set-header] v91'58 uv58 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff224008830 con 0x55b14c6c2670 2024-09-16T01:40:49.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.643+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6cd600 con 0x55b14c6c2670 2024-09-16T01:40:49.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.644+0000 7ff22fdae640 1 -- 172.21.15.88:0/1740977581 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(94 100.00000000 [create] v91'59 uv53 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff224008830 con 0x55b14c6c2670 2024-09-16T01:40:49.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.644+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6cd600 con 0x55b14c6c2670 2024-09-16T01:40:49.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.645+0000 7ff22fdae640 1 -- 172.21.15.88:0/1740977581 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(95 100.00000000 [delete] v91'60 uv60 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff224008830 con 0x55b14c6c2670 2024-09-16T01:40:49.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.645+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6cd600 con 0x55b14c6c2670 2024-09-16T01:40:49.647 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:40:49.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.646+0000 7ff22fdae640 1 -- 172.21.15.88:0/1740977581 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(96 100.00000000 [create] v91'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff224008830 con 0x55b14c6c2670 2024-09-16T01:40:49.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.646+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6cd600 con 0x55b14c6c2670 2024-09-16T01:40:49.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.647+0000 7ff22fdae640 1 -- 172.21.15.88:0/1740977581 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(97 100.00000000 [omap-set-header] v91'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff224008830 con 0x55b14c6c2670 2024-09-16T01:40:49.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.647+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0410 con 0x55b14c6c2670 2024-09-16T01:40:49.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.648+0000 7ff22fdae640 1 -- 172.21.15.88:0/1740977581 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v91'63 uv63 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff224008830 con 0x55b14c6c2670 2024-09-16T01:40:49.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.648+0000 7ff233520f40 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55b14c6ce130 0x55b14c6d0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.648+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:99 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6ce130 2024-09-16T01:40:49.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.648+0000 7ff22fdae640 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55b14c6ce130 0x55b14c6d0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.648+0000 7ff22fdae640 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55b14c6ce130 0x55b14c6d0a10 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:49.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.649+0000 7ff22fdae640 1 -- 172.21.15.88:0/1740977581 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(99 mds0_sessionmap [create] v91'26 uv25 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7ff224008830 con 0x55b14c6ce130 2024-09-16T01:40:49.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.649+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:100 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6ce130 2024-09-16T01:40:49.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.650+0000 7ff22fdae640 1 -- 172.21.15.88:0/1740977581 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v91'27 uv27 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7ff224008830 con 0x55b14c6ce130 2024-09-16T01:40:49.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.650+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6ce130 2024-09-16T01:40:49.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.651+0000 7ff22fdae640 1 -- 172.21.15.88:0/1740977581 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(101 mds0_sessionmap [create] v91'28 uv28 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7ff224008830 con 0x55b14c6ce130 2024-09-16T01:40:49.652 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:40:49.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.651+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6ce130 2024-09-16T01:40:49.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.654+0000 7ff22fdae640 1 -- 172.21.15.88:0/1740977581 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v91'29 uv29 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7ff224008830 con 0x55b14c6ce130 2024-09-16T01:40:49.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.654+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6c2670 2024-09-16T01:40:49.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.655+0000 7ff22fdae640 1 -- 172.21.15.88:0/1740977581 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(103 609.00000000 [create] v91'33 uv32 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff224008830 con 0x55b14c6c2670 2024-09-16T01:40:49.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.655+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6c2670 2024-09-16T01:40:49.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.656+0000 7ff22fdae640 1 -- 172.21.15.88:0/1740977581 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(104 609.00000000 [delete] v91'34 uv34 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff224008830 con 0x55b14c6c2670 2024-09-16T01:40:49.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.656+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6c2670 2024-09-16T01:40:49.658 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:40:49.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.657+0000 7ff22fdae640 1 -- 172.21.15.88:0/1740977581 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(105 609.00000000 [create] v91'35 uv35 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff224008830 con 0x55b14c6c2670 2024-09-16T01:40:49.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.657+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6c2670 2024-09-16T01:40:49.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.658+0000 7ff22fdae640 1 -- 172.21.15.88:0/1740977581 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v91'36 uv36 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff224008830 con 0x55b14c6c2670 2024-09-16T01:40:49.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.658+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6c2670 2024-09-16T01:40:49.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.659+0000 7ff22fdae640 1 -- 172.21.15.88:0/1740977581 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v91'37 uv37 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff224008830 con 0x55b14c6c2670 2024-09-16T01:40:49.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.659+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6c2670 2024-09-16T01:40:49.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.660+0000 7ff22fdae640 1 -- 172.21.15.88:0/1740977581 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(108 609.00000000 [omap-set-header] v91'38 uv38 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff224008830 con 0x55b14c6c2670 2024-09-16T01:40:49.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.660+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6c6b40 2024-09-16T01:40:49.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.661+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(109 400.00000000 [create] v91'27 uv26 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.661+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6c6b40 2024-09-16T01:40:49.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.662+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(110 400.00000000 [delete] v91'28 uv28 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.662+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6c6b40 2024-09-16T01:40:49.666 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:40:49.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.665+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(111 400.00000000 [create] v91'29 uv29 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.665+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6c6b40 2024-09-16T01:40:49.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.668+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v91'30 uv30 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.668+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6c6b40 2024-09-16T01:40:49.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.670+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v91'31 uv31 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.670+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6c6b40 2024-09-16T01:40:49.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.673+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(114 100.00000000.inode [create] v91'69 uv61 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.673+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6c6b40 2024-09-16T01:40:49.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.674+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(115 100.00000000.inode [delete] v91'70 uv70 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.674+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6c6b40 2024-09-16T01:40:49.676 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:40:49.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.675+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(116 100.00000000.inode [create] v91'71 uv71 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.675+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6c6b40 2024-09-16T01:40:49.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.676+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v91'72 uv72 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.676+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6c6b40 2024-09-16T01:40:49.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.677+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v91'73 uv73 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.678+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6c6b40 2024-09-16T01:40:49.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.678+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(119 1.00000000 [create] v91'74 uv68 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.678+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6c6b40 2024-09-16T01:40:49.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.679+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(120 1.00000000 [delete] v91'75 uv75 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.679+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6c6b40 2024-09-16T01:40:49.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.680+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(121 1.00000000 [create] v91'76 uv76 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.682 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:40:49.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.680+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6c6b40 2024-09-16T01:40:49.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.681+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v91'77 uv77 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.681+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6c6b40 2024-09-16T01:40:49.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.684+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v91'78 uv78 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.684+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d0520 con 0x55b14c6c6b40 2024-09-16T01:40:49.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.689+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v91'79 uv79 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.689+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91) -- 0x55b14c6d1400 con 0x55b14c6c6b40 2024-09-16T01:40:49.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.690+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v91'80 uv80 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff22804e960 con 0x55b14c6c6b40 2024-09-16T01:40:49.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.690+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55b14c60a3f0 msgr2=0x55b14c617450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:49.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.690+0000 7ff233520f40 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55b14c60a3f0 0x55b14c617450 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.690+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55b14c6c2670 msgr2=0x55b14c6c4a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:49.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.690+0000 7ff233520f40 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55b14c6c2670 0x55b14c6c4a60 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.690+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55b14c6ce130 msgr2=0x55b14c6d0a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:49.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.690+0000 7ff233520f40 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55b14c6ce130 0x55b14c6d0a10 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.690+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55b14c6bddc0 msgr2=0x55b14c6c01b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:49.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.690+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 reap_dead start 2024-09-16T01:40:49.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.690+0000 7ff233520f40 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55b14c6bddc0 0x55b14c6c01b0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.690+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b14c6c6b40 msgr2=0x55b14c6c6f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:49.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.690+0000 7ff233520f40 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b14c6c6b40 0x55b14c6c6f20 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.690+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55b14c6c9f70 msgr2=0x55b14c6ca3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:49.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.690+0000 7ff233520f40 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55b14c6c9f70 0x55b14c6ca3d0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.691+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff21c06ac90 msgr2=0x7ff21c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:49.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.691+0000 7ff233520f40 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff21c06ac90 0x7ff21c06d150 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff224005fd0 tx=0x7ff22400c040 comp rx=0 tx=0).stop 2024-09-16T01:40:49.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.691+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b14c570e10 msgr2=0x55b14c614540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:49.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.691+0000 7ff233520f40 1 --2- 172.21.15.88:0/1740977581 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b14c570e10 0x55b14c614540 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x55b14c610800 tx=0x7ff220002fb0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.691+0000 7ff230db0640 1 -- 172.21.15.88:0/1740977581 reap_dead start 2024-09-16T01:40:49.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.691+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 shutdown_connections 2024-09-16T01:40:49.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.691+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 >> 172.21.15.88:0/1740977581 conn(0x55b14c58c750 msgr2=0x55b14c613e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:49.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.691+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 shutdown_connections 2024-09-16T01:40:49.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.691+0000 7ff233520f40 1 -- 172.21.15.88:0/1740977581 wait complete. 2024-09-16T01:40:49.695 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm mds0_openfiles.0 2024-09-16T01:40:49.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.777+0000 7f2485d5cf40 1 Processor -- start 2024-09-16T01:40:49.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.777+0000 7f2485d5cf40 1 -- start start 2024-09-16T01:40:49.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.777+0000 7f2485d5cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605d9b50e10 0x5605d9b4deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.777+0000 7f2485d5cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605d9b51a60 0x5605d9b62ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.777+0000 7f2485d5cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5605d9b6b250 0x5605d9bf75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.778+0000 7f2485d5cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5605d9bfa120 con 0x5605d9b51a60 2024-09-16T01:40:49.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.778+0000 7f2485d5cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5605d9bfa3c0 con 0x5605d9b50e10 2024-09-16T01:40:49.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.778+0000 7f2485d5cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5605d9bfa660 con 0x5605d9b6b250 2024-09-16T01:40:49.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.778+0000 7f247bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605d9b51a60 0x5605d9b62ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.778+0000 7f247bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605d9b51a60 0x5605d9b62ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55630/0 (socket says 172.21.15.88:55630) 2024-09-16T01:40:49.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.778+0000 7f247bfff640 1 -- 172.21.15.88:0/3862299156 learned_addr learned my addr 172.21.15.88:0/3862299156 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:49.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.778+0000 7f2483852640 1 --2- 172.21.15.88:0/3862299156 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605d9b50e10 0x5605d9b4deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.778+0000 7f2484053640 1 --2- 172.21.15.88:0/3862299156 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5605d9b6b250 0x5605d9bf75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.778+0000 7f247bfff640 1 -- 172.21.15.88:0/3862299156 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5605d9b6b250 msgr2=0x5605d9bf75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:49.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.778+0000 7f247bfff640 1 --2- 172.21.15.88:0/3862299156 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5605d9b6b250 0x5605d9bf75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.778+0000 7f247bfff640 1 -- 172.21.15.88:0/3862299156 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605d9b50e10 msgr2=0x5605d9b4deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:49.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.778+0000 7f247bfff640 1 --2- 172.21.15.88:0/3862299156 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605d9b50e10 0x5605d9b4deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.778+0000 7f247bfff640 1 -- 172.21.15.88:0/3862299156 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605d9a4ccb0 con 0x5605d9b51a60 2024-09-16T01:40:49.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.778+0000 7f2484053640 1 --2- 172.21.15.88:0/3862299156 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5605d9b6b250 0x5605d9bf75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:49.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.778+0000 7f247bfff640 1 --2- 172.21.15.88:0/3862299156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605d9b51a60 0x5605d9b62ed0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f246c0098b0 tx=0x7f246c005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7265eae238a98a90 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:49.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.778+0000 7f2482850640 1 -- 172.21.15.88:0/3862299156 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246c008930 con 0x5605d9b51a60 2024-09-16T01:40:49.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.778+0000 7f2482850640 1 -- 172.21.15.88:0/3862299156 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f246c018070 con 0x5605d9b51a60 2024-09-16T01:40:49.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.778+0000 7f2482850640 1 -- 172.21.15.88:0/3862299156 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246c008cd0 con 0x5605d9b51a60 2024-09-16T01:40:49.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.779+0000 7f2485d5cf40 1 -- 172.21.15.88:0/3862299156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605d9b51a60 msgr2=0x5605d9b62ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:49.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.779+0000 7f2485d5cf40 1 --2- 172.21.15.88:0/3862299156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605d9b51a60 0x5605d9b62ed0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f246c0098b0 tx=0x7f246c005a00 comp rx=0 tx=0).stop 2024-09-16T01:40:49.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.779+0000 7f2485d5cf40 1 -- 172.21.15.88:0/3862299156 shutdown_connections 2024-09-16T01:40:49.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.779+0000 7f2485d5cf40 1 --2- 172.21.15.88:0/3862299156 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5605d9b6b250 0x5605d9bf75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.779+0000 7f2485d5cf40 1 --2- 172.21.15.88:0/3862299156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605d9b51a60 0x5605d9b62ed0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.779+0000 7f2485d5cf40 1 --2- 172.21.15.88:0/3862299156 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605d9b50e10 0x5605d9b4deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.779+0000 7f2485d5cf40 1 -- 172.21.15.88:0/3862299156 >> 172.21.15.88:0/3862299156 conn(0x5605d9b6c750 msgr2=0x5605d9bf4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:49.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.779+0000 7f2485d5cf40 1 -- 172.21.15.88:0/3862299156 shutdown_connections 2024-09-16T01:40:49.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.779+0000 7f2485d5cf40 1 -- 172.21.15.88:0/3862299156 wait complete. 2024-09-16T01:40:49.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.779+0000 7f2485d5cf40 1 Processor -- start 2024-09-16T01:40:49.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.779+0000 7f2485d5cf40 1 -- start start 2024-09-16T01:40:49.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.780+0000 7f2485d5cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605d9b50e10 0x5605d9bf22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.780+0000 7f2485d5cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605d9b51a60 0x5605d9bf1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.780+0000 7f2485d5cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5605d9b6b250 0x5605d9b671a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.780+0000 7f2485d5cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5605d9b63410 con 0x5605d9b50e10 2024-09-16T01:40:49.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.780+0000 7f2485d5cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5605d9bf70d0 con 0x5605d9b51a60 2024-09-16T01:40:49.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.780+0000 7f2485d5cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5605d9b647f0 con 0x5605d9b6b250 2024-09-16T01:40:49.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.780+0000 7f2483852640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605d9b50e10 0x5605d9bf22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.780+0000 7f2483852640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605d9b50e10 0x5605d9bf22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55634/0 (socket says 172.21.15.88:55634) 2024-09-16T01:40:49.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.780+0000 7f2483852640 1 -- 172.21.15.88:0/2492291381 learned_addr learned my addr 172.21.15.88:0/2492291381 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:49.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.780+0000 7f247bfff640 1 --2- 172.21.15.88:0/2492291381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605d9b51a60 0x5605d9bf1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.780+0000 7f2484053640 1 --2- 172.21.15.88:0/2492291381 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5605d9b6b250 0x5605d9b671a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.780+0000 7f2483852640 1 -- 172.21.15.88:0/2492291381 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5605d9b6b250 msgr2=0x5605d9b671a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:49.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.780+0000 7f2483852640 1 --2- 172.21.15.88:0/2492291381 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5605d9b6b250 0x5605d9b671a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.780+0000 7f2483852640 1 -- 172.21.15.88:0/2492291381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605d9b51a60 msgr2=0x5605d9bf1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:49.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.780+0000 7f2483852640 1 --2- 172.21.15.88:0/2492291381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605d9b51a60 0x5605d9bf1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.780+0000 7f2483852640 1 -- 172.21.15.88:0/2492291381 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605d9bec100 con 0x5605d9b50e10 2024-09-16T01:40:49.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.780+0000 7f2484053640 1 --2- 172.21.15.88:0/2492291381 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5605d9b6b250 0x5605d9b671a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:49.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.781+0000 7f2483852640 1 --2- 172.21.15.88:0/2492291381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605d9b50e10 0x5605d9bf22f0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f247400a520 tx=0x7f2474002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:49.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.781+0000 7f248184e640 1 -- 172.21.15.88:0/2492291381 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f247406a070 con 0x5605d9b50e10 2024-09-16T01:40:49.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.781+0000 7f248184e640 1 -- 172.21.15.88:0/2492291381 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2474052890 con 0x5605d9b50e10 2024-09-16T01:40:49.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.781+0000 7f2485d5cf40 1 -- 172.21.15.88:0/2492291381 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5605d9bf8ce0 con 0x5605d9b50e10 2024-09-16T01:40:49.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.781+0000 7f248184e640 1 -- 172.21.15.88:0/2492291381 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2474056040 con 0x5605d9b50e10 2024-09-16T01:40:49.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.781+0000 7f2485d5cf40 1 -- 172.21.15.88:0/2492291381 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605d9bebe00 con 0x5605d9b50e10 2024-09-16T01:40:49.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.782+0000 7f248184e640 1 -- 172.21.15.88:0/2492291381 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f247406d070 con 0x5605d9b50e10 2024-09-16T01:40:49.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.782+0000 7f248184e640 1 --2- 172.21.15.88:0/2492291381 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f246406ac90 0x7f246406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.783+0000 7f248184e640 1 -- 172.21.15.88:0/2492291381 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6225+0+0 (secure 0 0 0) 0x7f24740cd8b0 con 0x5605d9b50e10 2024-09-16T01:40:49.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.783+0000 7f247bfff640 1 --2- 172.21.15.88:0/2492291381 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f246406ac90 0x7f246406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.783+0000 7f2485d5cf40 1 --2- 172.21.15.88:0/2492291381 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5605d9bf1e60 0x5605d9b66190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.783+0000 7f2485d5cf40 1 -- 172.21.15.88:0/2492291381 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:1 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e91) -- 0x5605d9bed830 con 0x5605d9bf1e60 2024-09-16T01:40:49.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.783+0000 7f2484053640 1 --2- 172.21.15.88:0/2492291381 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5605d9bf1e60 0x5605d9b66190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.783+0000 7f247bfff640 1 --2- 172.21.15.88:0/2492291381 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f246406ac90 0x7f246406d150 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f246c0098b0 tx=0x7f246c012000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:49.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.783+0000 7f2484053640 1 --2- 172.21.15.88:0/2492291381 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5605d9bf1e60 0x5605d9b66190 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:49.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.784+0000 7f2484053640 1 -- 172.21.15.88:0/2492291381 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(1 mds0_openfiles.0 [delete] v91'30 uv30 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f247c002ac0 con 0x5605d9bf1e60 2024-09-16T01:40:49.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.784+0000 7f2485d5cf40 1 -- 172.21.15.88:0/2492291381 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5605d9bf1e60 msgr2=0x5605d9b66190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:49.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.784+0000 7f2485d5cf40 1 --2- 172.21.15.88:0/2492291381 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5605d9bf1e60 0x5605d9b66190 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.784+0000 7f2485d5cf40 1 -- 172.21.15.88:0/2492291381 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f246406ac90 msgr2=0x7f246406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:49.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.784+0000 7f2485d5cf40 1 --2- 172.21.15.88:0/2492291381 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f246406ac90 0x7f246406d150 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f246c0098b0 tx=0x7f246c012000 comp rx=0 tx=0).stop 2024-09-16T01:40:49.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.784+0000 7f2485d5cf40 1 -- 172.21.15.88:0/2492291381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605d9b50e10 msgr2=0x5605d9bf22f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:49.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.784+0000 7f2485d5cf40 1 --2- 172.21.15.88:0/2492291381 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5605d9b50e10 0x5605d9bf22f0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f247400a520 tx=0x7f2474002fb0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.784+0000 7f2484053640 1 -- 172.21.15.88:0/2492291381 reap_dead start 2024-09-16T01:40:49.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.784+0000 7f2485d5cf40 1 -- 172.21.15.88:0/2492291381 shutdown_connections 2024-09-16T01:40:49.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.784+0000 7f2485d5cf40 1 -- 172.21.15.88:0/2492291381 >> 172.21.15.88:0/2492291381 conn(0x5605d9b6c750 msgr2=0x5605d9bf6940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:49.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.784+0000 7f2485d5cf40 1 -- 172.21.15.88:0/2492291381 shutdown_connections 2024-09-16T01:40:49.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.784+0000 7f2485d5cf40 1 -- 172.21.15.88:0/2492291381 wait complete. 2024-09-16T01:40:49.791 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:40:49.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.931+0000 7f3bf46ae640 1 Processor -- start 2024-09-16T01:40:49.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.931+0000 7f3bf46ae640 1 -- start start 2024-09-16T01:40:49.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.932+0000 7f3bf46ae640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3bec15e6f0 0x7f3bec06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.932+0000 7f3bf46ae640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bec0b7450 0x7f3bec15f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.932+0000 7f3bf46ae640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bec06e990 con 0x7f3bec0b54d0 2024-09-16T01:40:49.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.932+0000 7f3bf46ae640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f3bec06ec70 con 0x7f3bec15e6f0 2024-09-16T01:40:49.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.932+0000 7f3bf46ae640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3bec15ec70 con 0x7f3bec0b7450 2024-09-16T01:40:49.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.932+0000 7f3bf244a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3bec15e6f0 0x7f3bec06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.932+0000 7f3bf244a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3bec15e6f0 0x7f3bec06fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55636/0 (socket says 172.21.15.88:55636) 2024-09-16T01:40:49.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.932+0000 7f3bf1c49640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3bec0b54d0 0x7f3bec0b6d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59772/0 (socket says 172.21.15.88:59772) 2024-09-16T01:40:49.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.932+0000 7f3bf244a640 1 -- 172.21.15.88:0/47513003 learned_addr learned my addr 172.21.15.88:0/47513003 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:49.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.933+0000 7f3bf2c4b640 1 --2- 172.21.15.88:0/47513003 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bec0b7450 0x7f3bec15f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.933+0000 7f3bf244a640 1 -- 172.21.15.88:0/47513003 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bec0b7450 msgr2=0x7f3bec15f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:49.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.933+0000 7f3bf244a640 1 --2- 172.21.15.88:0/47513003 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bec0b7450 0x7f3bec15f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.933+0000 7f3bf244a640 1 -- 172.21.15.88:0/47513003 >> v1:172.21.15.88:6789/0 conn(0x7f3bec0b54d0 legacy=0x7f3bec0b6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:49.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.933+0000 7f3bf244a640 1 -- 172.21.15.88:0/47513003 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bec15ef50 con 0x7f3bec15e6f0 2024-09-16T01:40:49.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.933+0000 7f3bf1448640 1 -- 172.21.15.88:0/47513003 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 635174569 0 0) 0x7f3bec06e990 con 0x7f3bec0b54d0 2024-09-16T01:40:49.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.933+0000 7f3bf244a640 1 --2- 172.21.15.88:0/47513003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bec15e6f0 0x7f3bec06fd00 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f3be8009e50 tx=0x7f3be802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=737e6e0394d95639 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:49.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.933+0000 7f3bf1448640 1 -- 172.21.15.88:0/47513003 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be8036070 con 0x7f3bec15e6f0 2024-09-16T01:40:49.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.934+0000 7f3bf1448640 1 -- 172.21.15.88:0/47513003 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3be8031600 con 0x7f3bec15e6f0 2024-09-16T01:40:49.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.934+0000 7f3bf1448640 1 -- 172.21.15.88:0/47513003 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be80319d0 con 0x7f3bec15e6f0 2024-09-16T01:40:49.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.934+0000 7f3bf46ae640 1 -- 172.21.15.88:0/47513003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bec15e6f0 msgr2=0x7f3bec06fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:49.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.934+0000 7f3bf46ae640 1 --2- 172.21.15.88:0/47513003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bec15e6f0 0x7f3bec06fd00 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f3be8009e50 tx=0x7f3be802a710 comp rx=0 tx=0).stop 2024-09-16T01:40:49.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.934+0000 7f3bf46ae640 1 -- 172.21.15.88:0/47513003 shutdown_connections 2024-09-16T01:40:49.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.934+0000 7f3bf46ae640 1 --2- 172.21.15.88:0/47513003 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bec0b7450 0x7f3bec15f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.934+0000 7f3bf46ae640 1 --2- 172.21.15.88:0/47513003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bec15e6f0 0x7f3bec06fd00 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.934+0000 7f3bf46ae640 1 -- 172.21.15.88:0/47513003 >> 172.21.15.88:0/47513003 conn(0x7f3bec0d2430 msgr2=0x7f3bec0d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:49.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.934+0000 7f3bf46ae640 1 -- 172.21.15.88:0/47513003 shutdown_connections 2024-09-16T01:40:49.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.934+0000 7f3bf46ae640 1 -- 172.21.15.88:0/47513003 wait complete. 2024-09-16T01:40:49.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.935+0000 7f3bf46ae640 1 Processor -- start 2024-09-16T01:40:49.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.935+0000 7f3bf46ae640 1 -- start start 2024-09-16T01:40:49.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.936+0000 7f3bf46ae640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bec0b54d0 0x7f3bec161820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.936+0000 7f3bf46ae640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bec0b7450 0x7f3bec1580e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.936+0000 7f3bf46ae640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bec15e6f0 0x7f3bec158620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.936+0000 7f3bf46ae640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3bec06e730 con 0x7f3bec15e6f0 2024-09-16T01:40:49.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.936+0000 7f3bf46ae640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3bec06e5b0 con 0x7f3bec0b54d0 2024-09-16T01:40:49.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.936+0000 7f3bf46ae640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3bec070830 con 0x7f3bec0b7450 2024-09-16T01:40:49.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.936+0000 7f3bf2c4b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bec15e6f0 0x7f3bec158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.936+0000 7f3bf2c4b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bec15e6f0 0x7f3bec158620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55648/0 (socket says 172.21.15.88:55648) 2024-09-16T01:40:49.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.936+0000 7f3bf2c4b640 1 -- 172.21.15.88:0/825096200 learned_addr learned my addr 172.21.15.88:0/825096200 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:49.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.936+0000 7f3bf244a640 1 --2- 172.21.15.88:0/825096200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bec0b54d0 0x7f3bec161820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.936+0000 7f3bf1c49640 1 --2- 172.21.15.88:0/825096200 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bec0b7450 0x7f3bec1580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.936+0000 7f3bf2c4b640 1 -- 172.21.15.88:0/825096200 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bec0b7450 msgr2=0x7f3bec1580e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:49.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.936+0000 7f3bf2c4b640 1 --2- 172.21.15.88:0/825096200 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bec0b7450 0x7f3bec1580e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.936+0000 7f3bf2c4b640 1 -- 172.21.15.88:0/825096200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bec0b54d0 msgr2=0x7f3bec161820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:49.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.936+0000 7f3bf2c4b640 1 --2- 172.21.15.88:0/825096200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bec0b54d0 0x7f3bec161820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:49.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.936+0000 7f3bf2c4b640 1 -- 172.21.15.88:0/825096200 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bec1622e0 con 0x7f3bec15e6f0 2024-09-16T01:40:49.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.937+0000 7f3bf2c4b640 1 --2- 172.21.15.88:0/825096200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bec15e6f0 0x7f3bec158620 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f3be400c7f0 tx=0x7f3be400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:49.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.937+0000 7f3be2ffd640 1 -- 172.21.15.88:0/825096200 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be4015070 con 0x7f3bec15e6f0 2024-09-16T01:40:49.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.937+0000 7f3be2ffd640 1 -- 172.21.15.88:0/825096200 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3be4004220 con 0x7f3bec15e6f0 2024-09-16T01:40:49.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.937+0000 7f3bf46ae640 1 -- 172.21.15.88:0/825096200 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bec1f4520 con 0x7f3bec15e6f0 2024-09-16T01:40:49.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.937+0000 7f3be2ffd640 1 -- 172.21.15.88:0/825096200 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be4003610 con 0x7f3bec15e6f0 2024-09-16T01:40:49.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.937+0000 7f3bf46ae640 1 -- 172.21.15.88:0/825096200 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bec15d390 con 0x7f3bec15e6f0 2024-09-16T01:40:49.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.938+0000 7f3bf46ae640 1 -- 172.21.15.88:0/825096200 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bec06c6c0 con 0x7f3bec15e6f0 2024-09-16T01:40:49.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.938+0000 7f3be2ffd640 1 -- 172.21.15.88:0/825096200 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3be40039c0 con 0x7f3bec15e6f0 2024-09-16T01:40:49.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.939+0000 7f3be2ffd640 1 --2- 172.21.15.88:0/825096200 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3bd006c2b0 0x7f3bd006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:49.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.939+0000 7f3be2ffd640 1 -- 172.21.15.88:0/825096200 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6225+0+0 (secure 0 0 0) 0x7f3be408b560 con 0x7f3bec15e6f0 2024-09-16T01:40:49.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.939+0000 7f3bf244a640 1 --2- 172.21.15.88:0/825096200 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3bd006c2b0 0x7f3bd006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:49.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.939+0000 7f3bf244a640 1 --2- 172.21.15.88:0/825096200 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3bd006c2b0 0x7f3bd006e770 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3be8033000 tx=0x7f3be8009000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:49.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:49.943+0000 7f3be2ffd640 1 -- 172.21.15.88:0/825096200 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3be4059500 con 0x7f3bec15e6f0 2024-09-16T01:40:50.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.144+0000 7f3bf46ae640 1 -- 172.21.15.88:0/825096200 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f3bec0b5910 con 0x7f3bec15e6f0 2024-09-16T01:40:50.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.451+0000 7f3be2ffd640 1 -- 172.21.15.88:0/825096200 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v106) ==== 162+0+0 (secure 0 0 0) 0x7f3be405d800 con 0x7f3bec15e6f0 2024-09-16T01:40:50.453 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:40:50.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.454+0000 7f3bf46ae640 1 -- 172.21.15.88:0/825096200 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3bd006c2b0 msgr2=0x7f3bd006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:50.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.454+0000 7f3bf46ae640 1 --2- 172.21.15.88:0/825096200 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3bd006c2b0 0x7f3bd006e770 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3be8033000 tx=0x7f3be8009000 comp rx=0 tx=0).stop 2024-09-16T01:40:50.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.454+0000 7f3bf46ae640 1 -- 172.21.15.88:0/825096200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bec15e6f0 msgr2=0x7f3bec158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:50.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.454+0000 7f3bf46ae640 1 --2- 172.21.15.88:0/825096200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bec15e6f0 0x7f3bec158620 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f3be400c7f0 tx=0x7f3be400ccb0 comp rx=0 tx=0).stop 2024-09-16T01:40:50.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.454+0000 7f3bf46ae640 1 -- 172.21.15.88:0/825096200 shutdown_connections 2024-09-16T01:40:50.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.454+0000 7f3bf46ae640 1 --2- 172.21.15.88:0/825096200 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3bd006c2b0 0x7f3bd006e770 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:50.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.454+0000 7f3bf46ae640 1 --2- 172.21.15.88:0/825096200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bec15e6f0 0x7f3bec158620 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:50.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.454+0000 7f3bf46ae640 1 --2- 172.21.15.88:0/825096200 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bec0b7450 0x7f3bec1580e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:50.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.454+0000 7f3bf46ae640 1 --2- 172.21.15.88:0/825096200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bec0b54d0 0x7f3bec161820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:50.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.454+0000 7f3bf46ae640 1 -- 172.21.15.88:0/825096200 >> 172.21.15.88:0/825096200 conn(0x7f3bec0d2430 msgr2=0x7f3bec0b2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:50.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.454+0000 7f3bf46ae640 1 -- 172.21.15.88:0/825096200 shutdown_connections 2024-09-16T01:40:50.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.454+0000 7f3bf46ae640 1 -- 172.21.15.88:0/825096200 wait complete. 2024-09-16T01:40:50.460 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:40:50.459+0000 7f745d73a640 -1 mds.0.openfiles _load_finish got (2) No such file or directory 2024-09-16T01:40:50.871 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:40:50.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.973+0000 7fa0dab0a640 1 Processor -- start 2024-09-16T01:40:50.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.973+0000 7fa0dab0a640 1 -- start start 2024-09-16T01:40:50.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.973+0000 7fa0dab0a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa0d415e660 0x7fa0d40cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:50.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.973+0000 7fa0dab0a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0d40d1820 0x7fa0d40b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:50.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.973+0000 7fa0dab0a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0d415ea40 con 0x7fa0d40d3310 2024-09-16T01:40:50.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.973+0000 7fa0dab0a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa0d40d3870 con 0x7fa0d415e660 2024-09-16T01:40:50.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.973+0000 7fa0dab0a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa0d40d3b50 con 0x7fa0d40d1820 2024-09-16T01:40:50.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.974+0000 7fa0d90a7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0d40d1820 0x7fa0d40b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:50.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.974+0000 7fa0d88a6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa0d415e660 0x7fa0d40cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:50.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.974+0000 7fa0d90a7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0d40d1820 0x7fa0d40b7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:52050/0 (socket says 172.21.15.88:52050) 2024-09-16T01:40:50.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.974+0000 7fa0d90a7640 1 -- 172.21.15.88:0/3595172439 learned_addr learned my addr 172.21.15.88:0/3595172439 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:50.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.974+0000 7fa0d3fff640 1 --1- 172.21.15.88:0/3595172439 >> v1:172.21.15.154:6789/0 conn(0x7fa0d40d3310 0x7fa0d40d1110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:55518/0 (socket says 172.21.15.88:55518) 2024-09-16T01:40:50.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.974+0000 7fa0d90a7640 1 -- 172.21.15.88:0/3595172439 >> v2:172.21.15.154:3300/0 conn(0x7fa0d415e660 msgr2=0x7fa0d40cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:50.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.974+0000 7fa0d37fe640 1 -- 172.21.15.88:0/3595172439 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1761553193 0 0) 0x7fa0d415ea40 con 0x7fa0d40d3310 2024-09-16T01:40:50.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.974+0000 7fa0d90a7640 1 --2- 172.21.15.88:0/3595172439 >> v2:172.21.15.154:3300/0 conn(0x7fa0d415e660 0x7fa0d40cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:50.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.974+0000 7fa0d90a7640 1 -- 172.21.15.88:0/3595172439 >> v1:172.21.15.154:6789/0 conn(0x7fa0d40d3310 legacy=0x7fa0d40d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:50.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.974+0000 7fa0d90a7640 1 -- 172.21.15.88:0/3595172439 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d40d3e30 con 0x7fa0d40d1820 2024-09-16T01:40:50.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.975+0000 7fa0d90a7640 1 --2- 172.21.15.88:0/3595172439 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0d40d1820 0x7fa0d40b7b70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc009e50 tx=0x7fa0cc02a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8550ba33d8a081ad server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:50.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.975+0000 7fa0d37fe640 1 -- 172.21.15.88:0/3595172439 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc036070 con 0x7fa0d40d1820 2024-09-16T01:40:50.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.975+0000 7fa0d37fe640 1 -- 172.21.15.88:0/3595172439 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0cc031600 con 0x7fa0d40d1820 2024-09-16T01:40:50.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.975+0000 7fa0d37fe640 1 -- 172.21.15.88:0/3595172439 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc0319d0 con 0x7fa0d40d1820 2024-09-16T01:40:50.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.975+0000 7fa0dab0a640 1 -- 172.21.15.88:0/3595172439 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0d40d1820 msgr2=0x7fa0d40b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:50.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.975+0000 7fa0dab0a640 1 --2- 172.21.15.88:0/3595172439 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0d40d1820 0x7fa0d40b7b70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc009e50 tx=0x7fa0cc02a710 comp rx=0 tx=0).stop 2024-09-16T01:40:50.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.976+0000 7fa0dab0a640 1 -- 172.21.15.88:0/3595172439 shutdown_connections 2024-09-16T01:40:50.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.976+0000 7fa0dab0a640 1 --2- 172.21.15.88:0/3595172439 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0d40d1820 0x7fa0d40b7b70 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:50.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.976+0000 7fa0dab0a640 1 --2- 172.21.15.88:0/3595172439 >> v2:172.21.15.154:3300/0 conn(0x7fa0d415e660 0x7fa0d40cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:50.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.976+0000 7fa0dab0a640 1 -- 172.21.15.88:0/3595172439 >> 172.21.15.88:0/3595172439 conn(0x7fa0d40b6410 msgr2=0x7fa0d40b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:50.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.976+0000 7fa0dab0a640 1 -- 172.21.15.88:0/3595172439 shutdown_connections 2024-09-16T01:40:50.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.976+0000 7fa0dab0a640 1 -- 172.21.15.88:0/3595172439 wait complete. 2024-09-16T01:40:50.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.977+0000 7fa0dab0a640 1 Processor -- start 2024-09-16T01:40:50.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.977+0000 7fa0dab0a640 1 -- start start 2024-09-16T01:40:50.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.978+0000 7fa0dab0a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0d40d1820 0x7fa0d41614f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:50.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.978+0000 7fa0dab0a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0d40d3310 0x7fa0d41498c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:50.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.978+0000 7fa0dab0a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0d415e660 0x7fa0d40ca130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:50.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.978+0000 7fa0dab0a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d40cfc70 con 0x7fa0d40d1820 2024-09-16T01:40:50.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.978+0000 7fa0dab0a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d40d43f0 con 0x7fa0d40d3310 2024-09-16T01:40:50.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.978+0000 7fa0dab0a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa0d40b80b0 con 0x7fa0d415e660 2024-09-16T01:40:50.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.978+0000 7fa0d88a6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0d40d1820 0x7fa0d41614f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:50.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.978+0000 7fa0d88a6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0d40d1820 0x7fa0d41614f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55958/0 (socket says 172.21.15.88:55958) 2024-09-16T01:40:50.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.978+0000 7fa0d88a6640 1 -- 172.21.15.88:0/59299093 learned_addr learned my addr 172.21.15.88:0/59299093 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:50.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.978+0000 7fa0d3fff640 1 --2- 172.21.15.88:0/59299093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0d40d3310 0x7fa0d41498c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:50.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.978+0000 7fa0d90a7640 1 --2- 172.21.15.88:0/59299093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0d415e660 0x7fa0d40ca130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:50.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.978+0000 7fa0d88a6640 1 -- 172.21.15.88:0/59299093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0d415e660 msgr2=0x7fa0d40ca130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:50.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.978+0000 7fa0d88a6640 1 --2- 172.21.15.88:0/59299093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0d415e660 0x7fa0d40ca130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:50.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.978+0000 7fa0d88a6640 1 -- 172.21.15.88:0/59299093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0d40d3310 msgr2=0x7fa0d41498c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:50.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.978+0000 7fa0d88a6640 1 --2- 172.21.15.88:0/59299093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0d40d3310 0x7fa0d41498c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:50.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.978+0000 7fa0d88a6640 1 -- 172.21.15.88:0/59299093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d415ef90 con 0x7fa0d40d1820 2024-09-16T01:40:50.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.978+0000 7fa0d88a6640 1 --2- 172.21.15.88:0/59299093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0d40d1820 0x7fa0d41614f0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fa0c800e870 tx=0x7fa0c800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:50.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.978+0000 7fa0d17fa640 1 -- 172.21.15.88:0/59299093 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8017070 con 0x7fa0d40d1820 2024-09-16T01:40:50.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.978+0000 7fa0d17fa640 1 -- 172.21.15.88:0/59299093 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0c800c940 con 0x7fa0d40d1820 2024-09-16T01:40:50.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.978+0000 7fa0d17fa640 1 -- 172.21.15.88:0/59299093 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c800cd30 con 0x7fa0d40d1820 2024-09-16T01:40:50.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.978+0000 7fa0dab0a640 1 -- 172.21.15.88:0/59299093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0d40c9370 con 0x7fa0d40d1820 2024-09-16T01:40:50.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.979+0000 7fa0dab0a640 1 -- 172.21.15.88:0/59299093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0d40c9930 con 0x7fa0d40d1820 2024-09-16T01:40:50.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.979+0000 7fa0dab0a640 1 -- 172.21.15.88:0/59299093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0d40d2c50 con 0x7fa0d40d1820 2024-09-16T01:40:50.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.980+0000 7fa0d17fa640 1 -- 172.21.15.88:0/59299093 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa0c801a070 con 0x7fa0d40d1820 2024-09-16T01:40:50.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.980+0000 7fa0d17fa640 1 --2- 172.21.15.88:0/59299093 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa0b806c2b0 0x7fa0b806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:50.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.980+0000 7fa0d17fa640 1 -- 172.21.15.88:0/59299093 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6225+0+0 (secure 0 0 0) 0x7fa0c808b800 con 0x7fa0d40d1820 2024-09-16T01:40:50.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.980+0000 7fa0d3fff640 1 --2- 172.21.15.88:0/59299093 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa0b806c2b0 0x7fa0b806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:50.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.981+0000 7fa0d3fff640 1 --2- 172.21.15.88:0/59299093 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa0b806c2b0 0x7fa0b806e770 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fa0d40cf850 tx=0x7fa0c0007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:50.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:50.985+0000 7fa0d17fa640 1 -- 172.21.15.88:0/59299093 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa0d40d2c50 con 0x7fa0d40d1820 2024-09-16T01:40:51.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:51.183+0000 7fa0dab0a640 1 -- 172.21.15.88:0/59299093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0d40b7b70 con 0x7fa0d40d1820 2024-09-16T01:40:51.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:51.183+0000 7fa0d17fa640 1 -- 172.21.15.88:0/59299093 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+4400 (secure 0 0 0) 0x7fa0c8059710 con 0x7fa0d40d1820 2024-09-16T01:40:51.185 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:51.185 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":107,"btime":"2024-09-16T01:40:50:454761+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:40:50.454748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7806},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7806":{"gid":7806,"name":"a","rank":0,"incarnation":107,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6836/1042715824","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1042715824},{"type":"v1","addr":"172.21.15.88:6836","nonce":1042715824}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:40:51.185 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 107 2024-09-16T01:40:51.186 INFO:tasks.ceph.mds.c.smithi088.stderr: -38> 2024-09-16T01:40:46.433+0000 7f9a08fb3640 -1 log_channel(cluster) log [ERR] : missing journal object 2024-09-16T01:40:51.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:51.185+0000 7fa0dab0a640 1 -- 172.21.15.88:0/59299093 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa0b806c2b0 msgr2=0x7fa0b806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:51.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:51.185+0000 7fa0dab0a640 1 --2- 172.21.15.88:0/59299093 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa0b806c2b0 0x7fa0b806e770 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fa0d40cf850 tx=0x7fa0c0007d50 comp rx=0 tx=0).stop 2024-09-16T01:40:51.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:51.185+0000 7fa0dab0a640 1 -- 172.21.15.88:0/59299093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0d40d1820 msgr2=0x7fa0d41614f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:51.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:51.185+0000 7fa0dab0a640 1 --2- 172.21.15.88:0/59299093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0d40d1820 0x7fa0d41614f0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fa0c800e870 tx=0x7fa0c800ed30 comp rx=0 tx=0).stop 2024-09-16T01:40:51.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:51.186+0000 7fa0dab0a640 1 -- 172.21.15.88:0/59299093 shutdown_connections 2024-09-16T01:40:51.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:51.186+0000 7fa0dab0a640 1 --2- 172.21.15.88:0/59299093 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa0b806c2b0 0x7fa0b806e770 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:51.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:51.186+0000 7fa0dab0a640 1 --2- 172.21.15.88:0/59299093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0d415e660 0x7fa0d40ca130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:51.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:51.186+0000 7fa0dab0a640 1 --2- 172.21.15.88:0/59299093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0d40d3310 0x7fa0d41498c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:51.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:51.186+0000 7fa0dab0a640 1 --2- 172.21.15.88:0/59299093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa0d40d1820 0x7fa0d41614f0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:51.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:51.186+0000 7fa0dab0a640 1 -- 172.21.15.88:0/59299093 >> 172.21.15.88:0/59299093 conn(0x7fa0d40b6410 msgr2=0x7fa0d415b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:51.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:51.186+0000 7fa0dab0a640 1 -- 172.21.15.88:0/59299093 shutdown_connections 2024-09-16T01:40:51.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:51.186+0000 7fa0dab0a640 1 -- 172.21.15.88:0/59299093 wait complete. 2024-09-16T01:40:51.202 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:40:03.597796+0000', 'modified': '2024-09-16T01:40:50.454748+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 91, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7806}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7806': {'gid': 7806, 'name': 'a', 'rank': 0, 'incarnation': 107, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.88:6836/1042715824', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 1042715824}, {'type': 'v1', 'addr': '172.21.15.88:6836', 'nonce': 1042715824}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-16T01:40:51.202 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7806:up:replay 2024-09-16T01:40:51.203 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-09-16T01:40:51.213 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:40:56.607 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:40:56.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.718+0000 7fc1afeae640 1 Processor -- start 2024-09-16T01:40:56.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.718+0000 7fc1afeae640 1 -- start start 2024-09-16T01:40:56.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.719+0000 7fc1afeae640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc1a815a9c0 0x7fc1a815e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:56.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.719+0000 7fc1afeae640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1a80b29a0 0x7fc1a80cde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:56.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.719+0000 7fc1afeae640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1a80b6830 con 0x7fc1a80b5be0 2024-09-16T01:40:56.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.719+0000 7fc1afeae640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc1a815f410 con 0x7fc1a815a9c0 2024-09-16T01:40:56.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.719+0000 7fc1afeae640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc1a80cd0b0 con 0x7fc1a80b29a0 2024-09-16T01:40:56.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.719+0000 7fc1ad449640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc1a80b5be0 0x7fc1a815eb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:32876/0 (socket says 172.21.15.88:32876) 2024-09-16T01:40:56.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.719+0000 7fc1ad449640 1 -- 172.21.15.88:0/2297623584 learned_addr learned my addr 172.21.15.88:0/2297623584 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:56.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.719+0000 7fc1adc4a640 1 --2- 172.21.15.88:0/2297623584 >> v2:172.21.15.154:3300/0 conn(0x7fc1a815a9c0 0x7fc1a815e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:56.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.719+0000 7fc1ae44b640 1 --2- 172.21.15.88:0/2297623584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1a80b29a0 0x7fc1a80cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:56.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.719+0000 7fc1acc48640 1 -- 172.21.15.88:0/2297623584 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4232887922 0 0) 0x7fc1a80b6830 con 0x7fc1a80b5be0 2024-09-16T01:40:56.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.719+0000 7fc1acc48640 1 -- 172.21.15.88:0/2297623584 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc190003520 con 0x7fc1a80b5be0 2024-09-16T01:40:56.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.719+0000 7fc1acc48640 1 -- 172.21.15.88:0/2297623584 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2200536555 0 0) 0x7fc190003520 con 0x7fc1a80b5be0 2024-09-16T01:40:56.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.719+0000 7fc1acc48640 1 -- 172.21.15.88:0/2297623584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1a80b29a0 msgr2=0x7fc1a80cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:56.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.719+0000 7fc1acc48640 1 --2- 172.21.15.88:0/2297623584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1a80b29a0 0x7fc1a80cde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:56.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.719+0000 7fc1acc48640 1 -- 172.21.15.88:0/2297623584 >> v2:172.21.15.154:3300/0 conn(0x7fc1a815a9c0 msgr2=0x7fc1a815e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:56.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.719+0000 7fc1acc48640 1 --2- 172.21.15.88:0/2297623584 >> v2:172.21.15.154:3300/0 conn(0x7fc1a815a9c0 0x7fc1a815e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:56.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.719+0000 7fc1acc48640 1 -- 172.21.15.88:0/2297623584 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1a80cd390 con 0x7fc1a80b5be0 2024-09-16T01:40:56.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.719+0000 7fc1acc48640 1 -- 172.21.15.88:0/2297623584 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fc19c001670 con 0x7fc1a80b5be0 2024-09-16T01:40:56.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.719+0000 7fc1acc48640 1 -- 172.21.15.88:0/2297623584 >> v1:172.21.15.88:6789/0 conn(0x7fc1a80b5be0 legacy=0x7fc1a815eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:56.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.719+0000 7fc1acc48640 1 --2- 172.21.15.88:0/2297623584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1a80b29a0 0x7fc1900039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:56.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.719+0000 7fc1acc48640 1 --2- 172.21.15.88:0/2297623584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc190003f30 0x7fc190004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:56.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.719+0000 7fc1acc48640 1 --2- 172.21.15.88:0/2297623584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc190004d40 0x7fc1900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:56.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.719+0000 7fc1acc48640 1 -- 172.21.15.88:0/2297623584 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a815f410 con 0x7fc190003f30 2024-09-16T01:40:56.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.719+0000 7fc1acc48640 1 -- 172.21.15.88:0/2297623584 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a80cd0b0 con 0x7fc190004d40 2024-09-16T01:40:56.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.719+0000 7fc1acc48640 1 -- 172.21.15.88:0/2297623584 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc19000b9c0 con 0x7fc1a80b29a0 2024-09-16T01:40:56.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.720+0000 7fc1ad449640 1 --2- 172.21.15.88:0/2297623584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc190003f30 0x7fc190004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:56.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.720+0000 7fc1adc4a640 1 --2- 172.21.15.88:0/2297623584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1a80b29a0 0x7fc1900039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:56.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.720+0000 7fc1ae44b640 1 --2- 172.21.15.88:0/2297623584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc190004d40 0x7fc1900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:56.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.720+0000 7fc1ad449640 1 -- 172.21.15.88:0/2297623584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1a80b29a0 msgr2=0x7fc1900039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:56.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.720+0000 7fc1ad449640 1 --2- 172.21.15.88:0/2297623584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1a80b29a0 0x7fc1900039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:56.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.720+0000 7fc1ad449640 1 -- 172.21.15.88:0/2297623584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc190004d40 msgr2=0x7fc1900051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:56.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.720+0000 7fc1ad449640 1 --2- 172.21.15.88:0/2297623584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc190004d40 0x7fc1900051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:56.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.720+0000 7fc1ad449640 1 -- 172.21.15.88:0/2297623584 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc190003520 con 0x7fc190003f30 2024-09-16T01:40:56.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.720+0000 7fc1ae44b640 1 --2- 172.21.15.88:0/2297623584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc190004d40 0x7fc1900051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:56.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.721+0000 7fc1ad449640 1 --2- 172.21.15.88:0/2297623584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc190003f30 0x7fc190004800 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fc19c007fd0 tx=0x7fc190005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9406ba41bb440076 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:56.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.721+0000 7fc1acc48640 1 -- 172.21.15.88:0/2297623584 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19c007d50 con 0x7fc190003f30 2024-09-16T01:40:56.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.721+0000 7fc1acc48640 1 -- 172.21.15.88:0/2297623584 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc19c005af0 con 0x7fc190003f30 2024-09-16T01:40:56.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.721+0000 7fc1acc48640 1 -- 172.21.15.88:0/2297623584 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19c007070 con 0x7fc190003f30 2024-09-16T01:40:56.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.721+0000 7fc1afeae640 1 -- 172.21.15.88:0/2297623584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc190003f30 msgr2=0x7fc190004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:56.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.721+0000 7fc1afeae640 1 --2- 172.21.15.88:0/2297623584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc190003f30 0x7fc190004800 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fc19c007fd0 tx=0x7fc190005830 comp rx=0 tx=0).stop 2024-09-16T01:40:56.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.721+0000 7fc1ae44b640 1 -- 172.21.15.88:0/2297623584 reap_dead start 2024-09-16T01:40:56.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.721+0000 7fc1afeae640 1 -- 172.21.15.88:0/2297623584 shutdown_connections 2024-09-16T01:40:56.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.721+0000 7fc1afeae640 1 -- 172.21.15.88:0/2297623584 >> 172.21.15.88:0/2297623584 conn(0x7fc1a806bc10 msgr2=0x7fc1a806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:56.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.721+0000 7fc1afeae640 1 -- 172.21.15.88:0/2297623584 shutdown_connections 2024-09-16T01:40:56.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.721+0000 7fc1afeae640 1 -- 172.21.15.88:0/2297623584 wait complete. 2024-09-16T01:40:56.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.723+0000 7fc1afeae640 1 Processor -- start 2024-09-16T01:40:56.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.723+0000 7fc1afeae640 1 -- start start 2024-09-16T01:40:56.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.723+0000 7fc1afeae640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc190003f30 0x7fc1a8150e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:56.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.723+0000 7fc1afeae640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1a814e7b0 0x7fc1a815d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:56.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.723+0000 7fc1afeae640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1a8151db0 0x7fc1a8152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:56.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.723+0000 7fc1afeae640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a815ada0 con 0x7fc1a8151db0 2024-09-16T01:40:56.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.723+0000 7fc1afeae640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a80cd950 con 0x7fc190003f30 2024-09-16T01:40:56.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.723+0000 7fc1afeae640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc1a806de40 con 0x7fc1a814e7b0 2024-09-16T01:40:56.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.723+0000 7fc1ae44b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1a8151db0 0x7fc1a8152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:56.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.723+0000 7fc1ae44b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1a8151db0 0x7fc1a8152230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56010/0 (socket says 172.21.15.88:56010) 2024-09-16T01:40:56.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.723+0000 7fc1ae44b640 1 -- 172.21.15.88:0/4209103063 learned_addr learned my addr 172.21.15.88:0/4209103063 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:56.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.724+0000 7fc1adc4a640 1 --2- 172.21.15.88:0/4209103063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc190003f30 0x7fc1a8150e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:56.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.724+0000 7fc1ad449640 1 --2- 172.21.15.88:0/4209103063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1a814e7b0 0x7fc1a815d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:56.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.724+0000 7fc1ae44b640 1 -- 172.21.15.88:0/4209103063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1a814e7b0 msgr2=0x7fc1a815d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:56.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.724+0000 7fc1ae44b640 1 --2- 172.21.15.88:0/4209103063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1a814e7b0 0x7fc1a815d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:56.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.724+0000 7fc1ae44b640 1 -- 172.21.15.88:0/4209103063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc190003f30 msgr2=0x7fc1a8150e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:56.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.724+0000 7fc1ae44b640 1 --2- 172.21.15.88:0/4209103063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc190003f30 0x7fc1a8150e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:56.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.724+0000 7fc1ae44b640 1 -- 172.21.15.88:0/4209103063 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1a8144850 con 0x7fc1a8151db0 2024-09-16T01:40:56.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.724+0000 7fc1ae44b640 1 --2- 172.21.15.88:0/4209103063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1a8151db0 0x7fc1a8152230 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fc1a806f250 tx=0x7fc1a0011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:56.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.724+0000 7fc1a67fc640 1 -- 172.21.15.88:0/4209103063 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a0019020 con 0x7fc1a8151db0 2024-09-16T01:40:56.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.724+0000 7fc1a67fc640 1 -- 172.21.15.88:0/4209103063 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1a0004440 con 0x7fc1a8151db0 2024-09-16T01:40:56.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.724+0000 7fc1afeae640 1 -- 172.21.15.88:0/4209103063 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1a815d630 con 0x7fc1a8151db0 2024-09-16T01:40:56.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.724+0000 7fc1a67fc640 1 -- 172.21.15.88:0/4209103063 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a0003830 con 0x7fc1a8151db0 2024-09-16T01:40:56.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.724+0000 7fc1afeae640 1 -- 172.21.15.88:0/4209103063 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1a814a680 con 0x7fc1a8151db0 2024-09-16T01:40:56.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.725+0000 7fc1afeae640 1 -- 172.21.15.88:0/4209103063 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1a806e1d0 con 0x7fc1a8151db0 2024-09-16T01:40:56.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.726+0000 7fc1a67fc640 1 -- 172.21.15.88:0/4209103063 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc1a001c020 con 0x7fc1a8151db0 2024-09-16T01:40:56.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.726+0000 7fc1a67fc640 1 --2- 172.21.15.88:0/4209103063 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc19406c2b0 0x7fc19406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:56.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.726+0000 7fc1a67fc640 1 -- 172.21.15.88:0/4209103063 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6225+0+0 (secure 0 0 0) 0x7fc1a008b4c0 con 0x7fc1a8151db0 2024-09-16T01:40:56.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.726+0000 7fc1adc4a640 1 --2- 172.21.15.88:0/4209103063 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc19406c2b0 0x7fc19406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:56.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.727+0000 7fc1adc4a640 1 --2- 172.21.15.88:0/4209103063 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc19406c2b0 0x7fc19406e770 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fc19800afd0 tx=0x7fc19800c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:56.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.730+0000 7fc1a67fc640 1 -- 172.21.15.88:0/4209103063 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc1a0023030 con 0x7fc1a8151db0 2024-09-16T01:40:56.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.927+0000 7fc1afeae640 1 -- 172.21.15.88:0/4209103063 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc1a8072750 con 0x7fc1a8151db0 2024-09-16T01:40:56.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.928+0000 7fc1a67fc640 1 -- 172.21.15.88:0/4209103063 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+5230 (secure 0 0 0) 0x7fc1a0059460 con 0x7fc1a8151db0 2024-09-16T01:40:56.929 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:56.929 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":110,"btime":"2024-09-16T01:40:53:467731+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":7866,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/293764774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":293764774},{"type":"v1","addr":"172.21.15.88:6837","nonce":293764774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:40:53.467726+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7806},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7806":{"gid":7806,"name":"a","rank":0,"incarnation":107,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6836/1042715824","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1042715824},{"type":"v1","addr":"172.21.15.88:6836","nonce":1042715824}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7806,"qdb_cluster":[7806]},"id":6}]} 2024-09-16T01:40:56.930 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 110 2024-09-16T01:40:56.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.930+0000 7fc1afeae640 1 -- 172.21.15.88:0/4209103063 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc19406c2b0 msgr2=0x7fc19406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:56.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.930+0000 7fc1afeae640 1 --2- 172.21.15.88:0/4209103063 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc19406c2b0 0x7fc19406e770 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fc19800afd0 tx=0x7fc19800c040 comp rx=0 tx=0).stop 2024-09-16T01:40:56.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.930+0000 7fc1afeae640 1 -- 172.21.15.88:0/4209103063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1a8151db0 msgr2=0x7fc1a8152230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:56.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.930+0000 7fc1afeae640 1 --2- 172.21.15.88:0/4209103063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1a8151db0 0x7fc1a8152230 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fc1a806f250 tx=0x7fc1a0011920 comp rx=0 tx=0).stop 2024-09-16T01:40:56.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.930+0000 7fc1afeae640 1 -- 172.21.15.88:0/4209103063 shutdown_connections 2024-09-16T01:40:56.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.930+0000 7fc1afeae640 1 --2- 172.21.15.88:0/4209103063 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc19406c2b0 0x7fc19406e770 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:56.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.930+0000 7fc1afeae640 1 --2- 172.21.15.88:0/4209103063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1a8151db0 0x7fc1a8152230 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:56.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.930+0000 7fc1afeae640 1 --2- 172.21.15.88:0/4209103063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1a814e7b0 0x7fc1a815d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:56.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.930+0000 7fc1afeae640 1 --2- 172.21.15.88:0/4209103063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc190003f30 0x7fc1a8150e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:56.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.930+0000 7fc1afeae640 1 -- 172.21.15.88:0/4209103063 >> 172.21.15.88:0/4209103063 conn(0x7fc1a806bc10 msgr2=0x7fc1a806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:56.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.930+0000 7fc1afeae640 1 -- 172.21.15.88:0/4209103063 shutdown_connections 2024-09-16T01:40:56.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:56.930+0000 7fc1afeae640 1 -- 172.21.15.88:0/4209103063 wait complete. 2024-09-16T01:40:56.942 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 110, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:40:03.597796+0000', 'modified': '2024-09-16T01:40:53.467726+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 91, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7806}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7806': {'gid': 7806, 'name': 'a', 'rank': 0, 'incarnation': 107, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.88:6836/1042715824', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 1042715824}, {'type': 'v1', 'addr': '172.21.15.88:6836', 'nonce': 1042715824}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7806, 'qdb_cluster': [7806]} 2024-09-16T01:40:56.942 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T01:40:56.942 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-16T01:40:57.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.040+0000 7f81f1b9f640 1 Processor -- start 2024-09-16T01:40:57.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.040+0000 7f81f1b9f640 1 -- start start 2024-09-16T01:40:57.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.041+0000 7f81f1b9f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81ec15e570 0x7f81ec0b7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.041+0000 7f81f1b9f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f81ec062d40 0x7f81ec1629f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.041+0000 7f81f1b9f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81ec0d2330 con 0x7f81ec0cf3a0 2024-09-16T01:40:57.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.041+0000 7f81f1b9f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f81ec15e950 con 0x7f81ec062d40 2024-09-16T01:40:57.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.041+0000 7f81f1b9f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f81ec162f30 con 0x7f81ec15e570 2024-09-16T01:40:57.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.041+0000 7f81f139e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f81ec062d40 0x7f81ec1629f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.041+0000 7f81f139e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f81ec062d40 0x7f81ec1629f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56012/0 (socket says 172.21.15.88:56012) 2024-09-16T01:40:57.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.041+0000 7f81ebfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f81ec0cf3a0 0x7f81ec062630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:32892/0 (socket says 172.21.15.88:32892) 2024-09-16T01:40:57.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.041+0000 7f81f139e640 1 -- 172.21.15.88:0/438431619 learned_addr learned my addr 172.21.15.88:0/438431619 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:57.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.042+0000 7f81f139e640 1 -- 172.21.15.88:0/438431619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81ec15e570 msgr2=0x7f81ec0b7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:57.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.042+0000 7f81f0b9d640 1 --2- 172.21.15.88:0/438431619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81ec15e570 0x7f81ec0b7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.042+0000 7f81f139e640 1 --2- 172.21.15.88:0/438431619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81ec15e570 0x7f81ec0b7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.042+0000 7f81f139e640 1 -- 172.21.15.88:0/438431619 >> v1:172.21.15.88:6789/0 conn(0x7f81ec0cf3a0 legacy=0x7f81ec062630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:57.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.042+0000 7f81eb7fe640 1 -- 172.21.15.88:0/438431619 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2915247409 0 0) 0x7f81ec0d2330 con 0x7f81ec0cf3a0 2024-09-16T01:40:57.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.042+0000 7f81f139e640 1 -- 172.21.15.88:0/438431619 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81ec0cf830 con 0x7f81ec062d40 2024-09-16T01:40:57.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.042+0000 7f81f0b9d640 1 --2- 172.21.15.88:0/438431619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81ec15e570 0x7f81ec0b7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:57.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.042+0000 7f81f139e640 1 --2- 172.21.15.88:0/438431619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81ec062d40 0x7f81ec1629f0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f81e4009e50 tx=0x7f81e402a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7e4521c7e082226 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:57.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.043+0000 7f81eb7fe640 1 -- 172.21.15.88:0/438431619 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e4036070 con 0x7f81ec062d40 2024-09-16T01:40:57.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.043+0000 7f81eb7fe640 1 -- 172.21.15.88:0/438431619 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81e4031600 con 0x7f81ec062d40 2024-09-16T01:40:57.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.043+0000 7f81eb7fe640 1 -- 172.21.15.88:0/438431619 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e40319d0 con 0x7f81ec062d40 2024-09-16T01:40:57.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.043+0000 7f81f1b9f640 1 -- 172.21.15.88:0/438431619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81ec062d40 msgr2=0x7f81ec1629f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:57.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.043+0000 7f81f1b9f640 1 --2- 172.21.15.88:0/438431619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81ec062d40 0x7f81ec1629f0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f81e4009e50 tx=0x7f81e402a7a0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.043+0000 7f81f1b9f640 1 -- 172.21.15.88:0/438431619 shutdown_connections 2024-09-16T01:40:57.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.043+0000 7f81f1b9f640 1 --2- 172.21.15.88:0/438431619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81ec062d40 0x7f81ec1629f0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.043+0000 7f81f1b9f640 1 --2- 172.21.15.88:0/438431619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81ec15e570 0x7f81ec0b7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.043+0000 7f81f1b9f640 1 -- 172.21.15.88:0/438431619 >> 172.21.15.88:0/438431619 conn(0x7f81ec0d4480 msgr2=0x7f81ec0d4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:57.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.043+0000 7f81f1b9f640 1 -- 172.21.15.88:0/438431619 shutdown_connections 2024-09-16T01:40:57.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.043+0000 7f81f1b9f640 1 -- 172.21.15.88:0/438431619 wait complete. 2024-09-16T01:40:57.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.044+0000 7f81f1b9f640 1 Processor -- start 2024-09-16T01:40:57.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.044+0000 7f81f1b9f640 1 -- start start 2024-09-16T01:40:57.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.045+0000 7f81f1b9f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81ec062d40 0x7f81ec1ea2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.045+0000 7f81f1b9f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81ec15e570 0x7f81ec1572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.045+0000 7f81f1b9f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81ec0ce010 0x7f81ec0cb550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.045+0000 7f81f1b9f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f81ec0d2c00 con 0x7f81ec062d40 2024-09-16T01:40:57.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.045+0000 7f81f1b9f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f81ec0ba3e0 con 0x7f81ec0ce010 2024-09-16T01:40:57.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.045+0000 7f81f1b9f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f81ec0d2ee0 con 0x7f81ec15e570 2024-09-16T01:40:57.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.045+0000 7f81f0b9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81ec062d40 0x7f81ec1ea2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.045+0000 7f81f0b9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81ec062d40 0x7f81ec1ea2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56014/0 (socket says 172.21.15.88:56014) 2024-09-16T01:40:57.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.045+0000 7f81f0b9d640 1 -- 172.21.15.88:0/4154386906 learned_addr learned my addr 172.21.15.88:0/4154386906 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:57.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.046+0000 7f81f139e640 1 --2- 172.21.15.88:0/4154386906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81ec0ce010 0x7f81ec0cb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.046+0000 7f81ebfff640 1 --2- 172.21.15.88:0/4154386906 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81ec15e570 0x7f81ec1572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.046+0000 7f81f0b9d640 1 -- 172.21.15.88:0/4154386906 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81ec15e570 msgr2=0x7f81ec1572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:57.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.046+0000 7f81f0b9d640 1 --2- 172.21.15.88:0/4154386906 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81ec15e570 0x7f81ec1572a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.046+0000 7f81f0b9d640 1 -- 172.21.15.88:0/4154386906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81ec0ce010 msgr2=0x7f81ec0cb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:57.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.046+0000 7f81f0b9d640 1 --2- 172.21.15.88:0/4154386906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81ec0ce010 0x7f81ec0cb550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.046+0000 7f81f0b9d640 1 -- 172.21.15.88:0/4154386906 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81ec1e9230 con 0x7f81ec062d40 2024-09-16T01:40:57.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.046+0000 7f81f0b9d640 1 --2- 172.21.15.88:0/4154386906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81ec062d40 0x7f81ec1ea2d0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f81dc00c7f0 tx=0x7f81dc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:57.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.046+0000 7f81e97fa640 1 -- 172.21.15.88:0/4154386906 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81dc015070 con 0x7f81ec062d40 2024-09-16T01:40:57.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.046+0000 7f81e97fa640 1 -- 172.21.15.88:0/4154386906 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81dc007820 con 0x7f81ec062d40 2024-09-16T01:40:57.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.046+0000 7f81f1b9f640 1 -- 172.21.15.88:0/4154386906 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81ec1e9510 con 0x7f81ec062d40 2024-09-16T01:40:57.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.046+0000 7f81e97fa640 1 -- 172.21.15.88:0/4154386906 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81dc007bd0 con 0x7f81ec062d40 2024-09-16T01:40:57.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.046+0000 7f81f1b9f640 1 -- 172.21.15.88:0/4154386906 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81ec1e9ad0 con 0x7f81ec062d40 2024-09-16T01:40:57.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.048+0000 7f81e97fa640 1 -- 172.21.15.88:0/4154386906 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f81dc018070 con 0x7f81ec062d40 2024-09-16T01:40:57.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.048+0000 7f81e97fa640 1 --2- 172.21.15.88:0/4154386906 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f81d806c2e0 0x7f81d806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.048+0000 7f81e97fa640 1 -- 172.21.15.88:0/4154386906 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6225+0+0 (secure 0 0 0) 0x7f81dc08b700 con 0x7f81ec062d40 2024-09-16T01:40:57.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.049+0000 7f81ebfff640 1 --2- 172.21.15.88:0/4154386906 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f81d806c2e0 0x7f81d806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.049+0000 7f81ebfff640 1 --2- 172.21.15.88:0/4154386906 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f81d806c2e0 0x7f81d806e7a0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f81ec0cb490 tx=0x7f81e0007b50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:57.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.049+0000 7f81f5c70740 1 Processor -- start 2024-09-16T01:40:57.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.049+0000 7f81f5c70740 1 -- start start 2024-09-16T01:40:57.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.050+0000 7f81f5c70740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bae84b0 0x557f9bb2dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.050+0000 7f81f5c70740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9bae5c70 0x557f9bae8890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.050+0000 7f81f5c70740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557f9bae9c40 0x557f9bae2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.050+0000 7f81f5c70740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x557f9ba4a520 con 0x557f9bae84b0 2024-09-16T01:40:57.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.050+0000 7f81f5c70740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557f9b9f23e0 con 0x557f9bae5c70 2024-09-16T01:40:57.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.050+0000 7f81f5c70740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x557f9ba464e0 con 0x557f9bae9c40 2024-09-16T01:40:57.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.050+0000 7f81f0b9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bae84b0 0x557f9bb2dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.050+0000 7f81f0b9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bae84b0 0x557f9bb2dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56024/0 (socket says 172.21.15.88:56024) 2024-09-16T01:40:57.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.050+0000 7f81f0b9d640 1 -- 172.21.15.88:0/2463765119 learned_addr learned my addr 172.21.15.88:0/2463765119 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:57.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.050+0000 7f81ebfff640 1 --2- 172.21.15.88:0/2463765119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9bae5c70 0x557f9bae8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.050+0000 7f81f139e640 1 --2- 172.21.15.88:0/2463765119 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557f9bae9c40 0x557f9bae2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.050+0000 7f81f0b9d640 1 -- 172.21.15.88:0/2463765119 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557f9bae9c40 msgr2=0x557f9bae2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:57.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.050+0000 7f81f0b9d640 1 --2- 172.21.15.88:0/2463765119 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557f9bae9c40 0x557f9bae2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.050+0000 7f81f0b9d640 1 -- 172.21.15.88:0/2463765119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9bae5c70 msgr2=0x557f9bae8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:57.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.050+0000 7f81f0b9d640 1 --2- 172.21.15.88:0/2463765119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9bae5c70 0x557f9bae8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.050+0000 7f81f0b9d640 1 -- 172.21.15.88:0/2463765119 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f9ba0d2f0 con 0x557f9bae84b0 2024-09-16T01:40:57.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.050+0000 7f81ebfff640 1 --2- 172.21.15.88:0/2463765119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9bae5c70 0x557f9bae8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:57.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.050+0000 7f81f139e640 1 --2- 172.21.15.88:0/2463765119 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557f9bae9c40 0x557f9bae2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:57.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.050+0000 7f81f0b9d640 1 --2- 172.21.15.88:0/2463765119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bae84b0 0x557f9bb2dc50 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f81dc089990 tx=0x7f81dc059620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55848fdc0df087e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:57.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.051+0000 7f81d27fc640 1 -- 172.21.15.88:0/2463765119 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81dc01f040 con 0x557f9bae84b0 2024-09-16T01:40:57.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.051+0000 7f81d27fc640 1 -- 172.21.15.88:0/2463765119 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81dc057860 con 0x557f9bae84b0 2024-09-16T01:40:57.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.051+0000 7f81d27fc640 1 -- 172.21.15.88:0/2463765119 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81dc0589b0 con 0x557f9bae84b0 2024-09-16T01:40:57.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.051+0000 7f81f5c70740 1 -- 172.21.15.88:0/2463765119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bae84b0 msgr2=0x557f9bb2dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:57.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.051+0000 7f81f5c70740 1 --2- 172.21.15.88:0/2463765119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bae84b0 0x557f9bb2dc50 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f81dc089990 tx=0x7f81dc059620 comp rx=0 tx=0).stop 2024-09-16T01:40:57.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.051+0000 7f81f5c70740 1 -- 172.21.15.88:0/2463765119 shutdown_connections 2024-09-16T01:40:57.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.051+0000 7f81f5c70740 1 --2- 172.21.15.88:0/2463765119 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557f9bae9c40 0x557f9bae2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.051+0000 7f81f5c70740 1 --2- 172.21.15.88:0/2463765119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9bae5c70 0x557f9bae8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.051+0000 7f81f5c70740 1 --2- 172.21.15.88:0/2463765119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bae84b0 0x557f9bb2dc50 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.051+0000 7f81f5c70740 1 -- 172.21.15.88:0/2463765119 >> 172.21.15.88:0/2463765119 conn(0x557f9bae4c70 msgr2=0x557f9bb2d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:57.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.051+0000 7f81f5c70740 1 -- 172.21.15.88:0/2463765119 shutdown_connections 2024-09-16T01:40:57.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.051+0000 7f81f5c70740 1 -- 172.21.15.88:0/2463765119 wait complete. 2024-09-16T01:40:57.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.052+0000 7f81f5c70740 1 -- start start 2024-09-16T01:40:57.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.052+0000 7f81f5c70740 1 Processor -- start 2024-09-16T01:40:57.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.053+0000 7f81f5c70740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557f9bae84b0 0x557f9bb41140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.053+0000 7f81f5c70740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9bae9c40 0x557f9bb01690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.053+0000 7f81f5c70740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bb12410 0x557f9bb12870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.053+0000 7f81f5c70740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x557f9bae4110 con 0x557f9bb12410 2024-09-16T01:40:57.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.053+0000 7f81f5c70740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557f9b9d3450 con 0x557f9bae9c40 2024-09-16T01:40:57.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.053+0000 7f81f5c70740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x557f9bb2e480 con 0x557f9bae84b0 2024-09-16T01:40:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.053+0000 7f81f5c70740 10 client.0 _ll_register_callbacks cb 0x557f9b75cac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:40:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.053+0000 7f81f139e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bb12410 0x557f9bb12870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.053+0000 7f81f139e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bb12410 0x557f9bb12870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56028/0 (socket says 172.21.15.88:56028) 2024-09-16T01:40:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.053+0000 7f81f139e640 1 -- 172.21.15.88:0/2241261550 learned_addr learned my addr 172.21.15.88:0/2241261550 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.053+0000 7f81f0b9d640 1 --2- 172.21.15.88:0/2241261550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557f9bae84b0 0x557f9bb41140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.053+0000 7f81ebfff640 1 --2- 172.21.15.88:0/2241261550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9bae9c40 0x557f9bb01690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.053+0000 7f81f139e640 1 -- 172.21.15.88:0/2241261550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557f9bae84b0 msgr2=0x557f9bb41140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.053+0000 7f81f139e640 1 --2- 172.21.15.88:0/2241261550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557f9bae84b0 0x557f9bb41140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.053+0000 7f81f139e640 1 -- 172.21.15.88:0/2241261550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9bae9c40 msgr2=0x557f9bb01690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.053+0000 7f81f139e640 1 --2- 172.21.15.88:0/2241261550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9bae9c40 0x557f9bb01690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.053+0000 7f81f139e640 1 -- 172.21.15.88:0/2241261550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f9b75d2d0 con 0x557f9bb12410 2024-09-16T01:40:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.053+0000 7f81ebfff640 1 --2- 172.21.15.88:0/2241261550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9bae9c40 0x557f9bb01690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:57.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.053+0000 7f81f139e640 1 --2- 172.21.15.88:0/2241261550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bb12410 0x557f9bb12870 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f81e402ad90 tx=0x7f81e4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b36456ea337de47d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:57.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.053+0000 7f81cb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:40:57.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.054+0000 7f81cb7fe640 1 -- 172.21.15.88:0/2241261550 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e4040030 con 0x557f9bb12410 2024-09-16T01:40:57.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.054+0000 7f81f5c70740 1 -- 172.21.15.88:0/2241261550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81dc057860 con 0x557f9bb12410 2024-09-16T01:40:57.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.054+0000 7f81cb7fe640 1 -- 172.21.15.88:0/2241261550 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 6225+0+0 (secure 0 0 0) 0x7f81e40043f0 con 0x557f9bb12410 2024-09-16T01:40:57.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.054+0000 7f81cb7fe640 1 -- 172.21.15.88:0/2241261550 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7f81e404c050 con 0x557f9bb12410 2024-09-16T01:40:57.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.054+0000 7f81f5c70740 10 client.7887 fetch_fsmap learned FSMap version 110 2024-09-16T01:40:57.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.054+0000 7f81f5c70740 1 -- 172.21.15.88:0/2241261550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=110,monmap=2+}) -- 0x557f9b8276b0 con 0x557f9bb12410 2024-09-16T01:40:57.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.054+0000 7f81cb7fe640 1 -- 172.21.15.88:0/2241261550 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81dc057860 con 0x557f9bb12410 2024-09-16T01:40:57.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.054+0000 7f81cb7fe640 1 -- 172.21.15.88:0/2241261550 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 110) ==== 3502+0+0 (secure 0 0 0) 0x7f81e404f020 con 0x557f9bb12410 2024-09-16T01:40:57.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.054+0000 7f81f5c70740 10 client.7887 fetch_fsmap finished waiting for FSMap version 110 2024-09-16T01:40:57.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.054+0000 7f81f5c70740 10 client.7887 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:40:57.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.054+0000 7f81f5c70740 20 client.7887 populate_metadata read hostname 'smithi088' 2024-09-16T01:40:57.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.054+0000 7f81f5c70740 1 --2- 172.21.15.88:0/2241261550 >> [v2:172.21.15.88:6834/1042715824,v1:172.21.15.88:6836/1042715824] conn(0x557f9bb28fe0 0x557f9bb293c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.054+0000 7f81f5c70740 4 client.7887 mds_command: new command op to 7806 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:40:57.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.054+0000 7f81f5c70740 1 -- 172.21.15.88:0/2241261550 --> [v2:172.21.15.88:6834/1042715824,v1:172.21.15.88:6836/1042715824] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f9ba57c50 con 0x557f9bb28fe0 2024-09-16T01:40:57.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.054+0000 7f81f0b9d640 1 --2- 172.21.15.88:0/2241261550 >> [v2:172.21.15.88:6834/1042715824,v1:172.21.15.88:6836/1042715824] conn(0x557f9bb28fe0 0x557f9bb293c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.055+0000 7f81cb7fe640 10 client.7887 ms_handle_connect on v2:172.21.15.88:6834/1042715824 2024-09-16T01:40:57.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.055+0000 7f81f0b9d640 1 --2- 172.21.15.88:0/2241261550 >> [v2:172.21.15.88:6834/1042715824,v1:172.21.15.88:6836/1042715824] conn(0x557f9bb28fe0 0x557f9bb293c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7745903b92ebe649 server_cookie=45d2d8a993d2abb7 in_seq=0 out_seq=0 2024-09-16T01:40:57.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.055+0000 7f81cb7fe640 1 -- 172.21.15.88:0/2241261550 <== mds.0 v2:172.21.15.88:6834/1042715824 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557f9ba57c50 con 0x557f9bb28fe0 2024-09-16T01:40:57.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.055+0000 7f81cb7fe640 10 client.7887 handle_command_reply: tid=0 2024-09-16T01:40:57.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.055+0000 7f81f5c70740 1 client.7887 shutdown 2024-09-16T01:40:57.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.055+0000 7f81f5c70740 1 -- 172.21.15.88:0/2241261550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bb12410 msgr2=0x557f9bb12870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:57.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.055+0000 7f81f5c70740 1 --2- 172.21.15.88:0/2241261550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bb12410 0x557f9bb12870 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f81e402ad90 tx=0x7f81e4004ab0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.055+0000 7f81f5c70740 1 -- 172.21.15.88:0/2241261550 shutdown_connections 2024-09-16T01:40:57.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.055+0000 7f81f5c70740 1 --2- 172.21.15.88:0/2241261550 >> [v2:172.21.15.88:6834/1042715824,v1:172.21.15.88:6836/1042715824] conn(0x557f9bb28fe0 0x557f9bb293c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.055+0000 7f81f5c70740 1 --2- 172.21.15.88:0/2241261550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bb12410 0x557f9bb12870 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.055+0000 7f81f5c70740 1 --2- 172.21.15.88:0/2241261550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9bae9c40 0x557f9bb01690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.055+0000 7f81cb7fe640 0 client.7887 ms_handle_reset on v2:172.21.15.88:6834/1042715824 2024-09-16T01:40:57.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.055+0000 7f81f5c70740 1 --2- 172.21.15.88:0/2241261550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557f9bae84b0 0x557f9bb41140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.055+0000 7f81f5c70740 1 -- 172.21.15.88:0/2241261550 >> 172.21.15.88:0/2241261550 conn(0x557f9bae5c70 msgr2=0x557f9bb2b1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:57.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.056+0000 7f81f5c70740 1 -- 172.21.15.88:0/2241261550 shutdown_connections 2024-09-16T01:40:57.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.056+0000 7f81f5c70740 1 -- 172.21.15.88:0/2241261550 wait complete. 2024-09-16T01:40:57.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.056+0000 7f81f5c70740 20 client.7887 trim_cache size 0 max 16384 2024-09-16T01:40:57.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.073+0000 7f81f5c70740 1 Processor -- start 2024-09-16T01:40:57.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.073+0000 7f81f5c70740 1 -- start start 2024-09-16T01:40:57.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.074+0000 7f81f5c70740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557f9bae84b0 0x557f9bb11c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.074+0000 7f81f5c70740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9bae9c40 0x557f9bba6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.074+0000 7f81f5c70740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bb12410 0x557f9bb2d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.074+0000 7f81f5c70740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f81ec200c60 con 0x557f9bb12410 2024-09-16T01:40:57.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.074+0000 7f81f5c70740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f81ec1ffce0 con 0x557f9bae9c40 2024-09-16T01:40:57.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.074+0000 7f81f5c70740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f81ec146e30 con 0x557f9bae84b0 2024-09-16T01:40:57.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.074+0000 7f81f139e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bb12410 0x557f9bb2d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.074+0000 7f81f139e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bb12410 0x557f9bb2d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56030/0 (socket says 172.21.15.88:56030) 2024-09-16T01:40:57.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.074+0000 7f81f139e640 1 -- 172.21.15.88:0/2838656239 learned_addr learned my addr 172.21.15.88:0/2838656239 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:57.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.074+0000 7f81f139e640 1 -- 172.21.15.88:0/2838656239 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557f9bae84b0 msgr2=0x557f9bb11c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:57.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.074+0000 7f81f139e640 1 --2- 172.21.15.88:0/2838656239 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557f9bae84b0 0x557f9bb11c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.074+0000 7f81ebfff640 1 --2- 172.21.15.88:0/2838656239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9bae9c40 0x557f9bba6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.074+0000 7f81f139e640 1 -- 172.21.15.88:0/2838656239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9bae9c40 msgr2=0x557f9bba6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:57.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.074+0000 7f81f139e640 1 --2- 172.21.15.88:0/2838656239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9bae9c40 0x557f9bba6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.074+0000 7f81f139e640 1 -- 172.21.15.88:0/2838656239 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f9bb0eec0 con 0x557f9bb12410 2024-09-16T01:40:57.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.074+0000 7f81f139e640 1 --2- 172.21.15.88:0/2838656239 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bb12410 0x557f9bb2d050 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f81e4009e50 tx=0x7f81e403c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df4991a4b0cefe39 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:57.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.074+0000 7f81d1ffb640 1 -- 172.21.15.88:0/2838656239 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e4040070 con 0x557f9bb12410 2024-09-16T01:40:57.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.074+0000 7f81d1ffb640 1 -- 172.21.15.88:0/2838656239 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81e4009af0 con 0x557f9bb12410 2024-09-16T01:40:57.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.074+0000 7f81d1ffb640 1 -- 172.21.15.88:0/2838656239 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e4036040 con 0x557f9bb12410 2024-09-16T01:40:57.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.075+0000 7f81f5c70740 1 -- 172.21.15.88:0/2838656239 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bb12410 msgr2=0x557f9bb2d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:57.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.075+0000 7f81f5c70740 1 --2- 172.21.15.88:0/2838656239 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bb12410 0x557f9bb2d050 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f81e4009e50 tx=0x7f81e403c000 comp rx=0 tx=0).stop 2024-09-16T01:40:57.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.075+0000 7f81f5c70740 1 -- 172.21.15.88:0/2838656239 shutdown_connections 2024-09-16T01:40:57.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.075+0000 7f81f5c70740 1 --2- 172.21.15.88:0/2838656239 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bb12410 0x557f9bb2d050 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.075+0000 7f81f5c70740 1 --2- 172.21.15.88:0/2838656239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9bae9c40 0x557f9bba6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.075+0000 7f81f5c70740 1 --2- 172.21.15.88:0/2838656239 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557f9bae84b0 0x557f9bb11c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.075+0000 7f81f5c70740 1 -- 172.21.15.88:0/2838656239 >> 172.21.15.88:0/2838656239 conn(0x557f9bae5c70 msgr2=0x557f9bb127f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:57.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.075+0000 7f81f5c70740 1 -- 172.21.15.88:0/2838656239 shutdown_connections 2024-09-16T01:40:57.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.075+0000 7f81f5c70740 1 -- 172.21.15.88:0/2838656239 wait complete. 2024-09-16T01:40:57.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.076+0000 7f81f5c70740 1 -- start start 2024-09-16T01:40:57.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.076+0000 7f81f5c70740 1 Processor -- start 2024-09-16T01:40:57.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.077+0000 7f81f5c70740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9bae84b0 0x557f9bb01430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.077+0000 7f81f5c70740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bae9c40 0x557f9bba4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.077+0000 7f81f5c70740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557f9bb12410 0x557f9bb50e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.077+0000 7f81f5c70740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x557f9bb282b0 con 0x557f9bae9c40 2024-09-16T01:40:57.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.077+0000 7f81f5c70740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f81ec1ff690 con 0x557f9bae84b0 2024-09-16T01:40:57.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.077+0000 7f81f5c70740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x557f9bb113a0 con 0x557f9bb12410 2024-09-16T01:40:57.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.077+0000 7f81f5c70740 10 client.0 _ll_register_callbacks cb 0x7f81e4034070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:40:57.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.077+0000 7f81ebfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bae9c40 0x557f9bba4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.077+0000 7f81ebfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bae9c40 0x557f9bba4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56044/0 (socket says 172.21.15.88:56044) 2024-09-16T01:40:57.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.077+0000 7f81ebfff640 1 -- 172.21.15.88:0/1725976166 learned_addr learned my addr 172.21.15.88:0/1725976166 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:57.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.077+0000 7f81f0b9d640 1 --2- 172.21.15.88:0/1725976166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9bae84b0 0x557f9bb01430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.077+0000 7f81f139e640 1 --2- 172.21.15.88:0/1725976166 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557f9bb12410 0x557f9bb50e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.077+0000 7f81ebfff640 1 -- 172.21.15.88:0/1725976166 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557f9bb12410 msgr2=0x557f9bb50e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:57.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.077+0000 7f81ebfff640 1 --2- 172.21.15.88:0/1725976166 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557f9bb12410 0x557f9bb50e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.077+0000 7f81ebfff640 1 -- 172.21.15.88:0/1725976166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9bae84b0 msgr2=0x557f9bb01430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:57.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.077+0000 7f81ebfff640 1 --2- 172.21.15.88:0/1725976166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9bae84b0 0x557f9bb01430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.077+0000 7f81ebfff640 1 -- 172.21.15.88:0/1725976166 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f9bb01c60 con 0x557f9bae9c40 2024-09-16T01:40:57.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.077+0000 7f81ebfff640 1 --2- 172.21.15.88:0/1725976166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bae9c40 0x557f9bba4fe0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f81e0000f60 tx=0x7f81e0000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=307f65b3b5def56e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:57.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.077+0000 7f81cb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:40:57.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.077+0000 7f81cb7fe640 1 -- 172.21.15.88:0/1725976166 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e0028060 con 0x557f9bae9c40 2024-09-16T01:40:57.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.077+0000 7f81cb7fe640 1 -- 172.21.15.88:0/1725976166 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 6225+0+0 (secure 0 0 0) 0x7f81e0010040 con 0x557f9bae9c40 2024-09-16T01:40:57.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.077+0000 7f81f5c70740 1 -- 172.21.15.88:0/1725976166 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81e4009af0 con 0x557f9bae9c40 2024-09-16T01:40:57.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.078+0000 7f81cb7fe640 1 -- 172.21.15.88:0/1725976166 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7f81e002e050 con 0x557f9bae9c40 2024-09-16T01:40:57.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.078+0000 7f81f5c70740 10 client.7893 fetch_fsmap learned FSMap version 110 2024-09-16T01:40:57.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.078+0000 7f81f5c70740 1 -- 172.21.15.88:0/1725976166 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=110,monmap=2+}) -- 0x557f9bbad8d0 con 0x557f9bae9c40 2024-09-16T01:40:57.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.078+0000 7f81cb7fe640 1 -- 172.21.15.88:0/1725976166 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81e4009af0 con 0x557f9bae9c40 2024-09-16T01:40:57.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.078+0000 7f81cb7fe640 1 -- 172.21.15.88:0/1725976166 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 110) ==== 3502+0+0 (secure 0 0 0) 0x7f81e0031020 con 0x557f9bae9c40 2024-09-16T01:40:57.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.078+0000 7f81f5c70740 10 client.7893 fetch_fsmap finished waiting for FSMap version 110 2024-09-16T01:40:57.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.078+0000 7f81f5c70740 10 client.7893 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:40:57.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.078+0000 7f81f5c70740 20 client.7893 populate_metadata read hostname 'smithi088' 2024-09-16T01:40:57.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.078+0000 7f81f5c70740 1 --2- 172.21.15.88:0/1725976166 >> [v2:172.21.15.88:6834/1042715824,v1:172.21.15.88:6836/1042715824] conn(0x557f9bbacfc0 0x557f9bb293c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.078+0000 7f81f5c70740 4 client.7893 mds_command: new command op to 7806 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:40:57.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.078+0000 7f81f5c70740 1 -- 172.21.15.88:0/1725976166 --> [v2:172.21.15.88:6834/1042715824,v1:172.21.15.88:6836/1042715824] -- command(tid 0: {"prefix": "status"}) -- 0x557f9bae3cb0 con 0x557f9bbacfc0 2024-09-16T01:40:57.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.078+0000 7f81f0b9d640 1 --2- 172.21.15.88:0/1725976166 >> [v2:172.21.15.88:6834/1042715824,v1:172.21.15.88:6836/1042715824] conn(0x557f9bbacfc0 0x557f9bb293c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.078+0000 7f81cb7fe640 10 client.7893 ms_handle_connect on v2:172.21.15.88:6834/1042715824 2024-09-16T01:40:57.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.078+0000 7f81f0b9d640 1 --2- 172.21.15.88:0/1725976166 >> [v2:172.21.15.88:6834/1042715824,v1:172.21.15.88:6836/1042715824] conn(0x557f9bbacfc0 0x557f9bb293c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aebf1717cc10b560 server_cookie=80a7b797836fe7ce in_seq=0 out_seq=0 2024-09-16T01:40:57.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.079+0000 7f81cb7fe640 1 -- 172.21.15.88:0/1725976166 <== mds.0 v2:172.21.15.88:6834/1042715824 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x557f9bae3cb0 con 0x557f9bbacfc0 2024-09-16T01:40:57.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.079+0000 7f81cb7fe640 10 client.7893 handle_command_reply: tid=0 2024-09-16T01:40:57.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.079+0000 7f81f5c70740 1 client.7893 shutdown 2024-09-16T01:40:57.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.079+0000 7f81f5c70740 1 -- 172.21.15.88:0/1725976166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bae9c40 msgr2=0x557f9bba4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:57.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.079+0000 7f81f5c70740 1 --2- 172.21.15.88:0/1725976166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bae9c40 0x557f9bba4fe0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f81e0000f60 tx=0x7f81e0000ed0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.079+0000 7f81f5c70740 1 -- 172.21.15.88:0/1725976166 shutdown_connections 2024-09-16T01:40:57.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.079+0000 7f81f5c70740 1 --2- 172.21.15.88:0/1725976166 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557f9bb12410 0x557f9bb50e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.079+0000 7f81f5c70740 1 --2- 172.21.15.88:0/1725976166 >> [v2:172.21.15.88:6834/1042715824,v1:172.21.15.88:6836/1042715824] conn(0x557f9bbacfc0 0x557f9bb293c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.079+0000 7f81f5c70740 1 --2- 172.21.15.88:0/1725976166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557f9bae9c40 0x557f9bba4fe0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.079+0000 7f81f5c70740 1 --2- 172.21.15.88:0/1725976166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9bae84b0 0x557f9bb01430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.079+0000 7f81f5c70740 1 -- 172.21.15.88:0/1725976166 >> 172.21.15.88:0/1725976166 conn(0x557f9bae5c70 msgr2=0x557f9bb2b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:57.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.079+0000 7f81cb7fe640 0 client.7893 ms_handle_reset on v2:172.21.15.88:6834/1042715824 2024-09-16T01:40:57.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.079+0000 7f81f5c70740 1 -- 172.21.15.88:0/1725976166 shutdown_connections 2024-09-16T01:40:57.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.079+0000 7f81f5c70740 1 -- 172.21.15.88:0/1725976166 wait complete. 2024-09-16T01:40:57.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.079+0000 7f81f5c70740 20 client.7893 trim_cache size 0 max 16384 2024-09-16T01:40:57.088 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:40:57.088 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "5ace2c91-46f9-482b-8a89-731fbdc171bf", 2024-09-16T01:40:57.088 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2024-09-16T01:40:57.088 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 7806, 2024-09-16T01:40:57.088 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2024-09-16T01:40:57.089 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2024-09-16T01:40:57.089 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2024-09-16T01:40:57.089 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 6.6233607670000003, 2024-09-16T01:40:57.089 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 110, 2024-09-16T01:40:57.089 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 91, 2024-09-16T01:40:57.089 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 91, 2024-09-16T01:40:57.089 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 9.9044278830000003 2024-09-16T01:40:57.089 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:40:57.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.079+0000 7f81f1b9f640 1 -- 172.21.15.88:0/4154386906 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f81d806c2e0 msgr2=0x7f81d806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:57.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.079+0000 7f81f1b9f640 1 --2- 172.21.15.88:0/4154386906 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f81d806c2e0 0x7f81d806e7a0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f81ec0cb490 tx=0x7f81e0007b50 comp rx=0 tx=0).stop 2024-09-16T01:40:57.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.079+0000 7f81f1b9f640 1 -- 172.21.15.88:0/4154386906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81ec062d40 msgr2=0x7f81ec1ea2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:57.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.079+0000 7f81f1b9f640 1 --2- 172.21.15.88:0/4154386906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81ec062d40 0x7f81ec1ea2d0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f81dc00c7f0 tx=0x7f81dc00ccb0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.079+0000 7f81f1b9f640 1 -- 172.21.15.88:0/4154386906 shutdown_connections 2024-09-16T01:40:57.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.079+0000 7f81f1b9f640 1 --2- 172.21.15.88:0/4154386906 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f81d806c2e0 0x7f81d806e7a0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.079+0000 7f81f1b9f640 1 --2- 172.21.15.88:0/4154386906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81ec0ce010 0x7f81ec0cb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.079+0000 7f81f1b9f640 1 --2- 172.21.15.88:0/4154386906 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81ec15e570 0x7f81ec1572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.079+0000 7f81f1b9f640 1 --2- 172.21.15.88:0/4154386906 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81ec062d40 0x7f81ec1ea2d0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.079+0000 7f81f1b9f640 1 -- 172.21.15.88:0/4154386906 >> 172.21.15.88:0/4154386906 conn(0x7f81ec0d4480 msgr2=0x7f81ec15f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:57.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.080+0000 7f81f1b9f640 1 -- 172.21.15.88:0/4154386906 shutdown_connections 2024-09-16T01:40:57.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.080+0000 7f81f1b9f640 1 -- 172.21.15.88:0/4154386906 wait complete. 2024-09-16T01:40:57.092 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-09-16T01:40:57.092 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Delete mds0_openfiles.0', proceeding to ls 2024-09-16T01:40:57.092 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:40:57.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.188+0000 7ff236f0a640 1 Processor -- start 2024-09-16T01:40:57.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.188+0000 7ff236f0a640 1 -- start start 2024-09-16T01:40:57.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.189+0000 7ff236f0a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff23015e800 0x7ff2300b5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.189+0000 7ff236f0a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff23006ea60 con 0x7ff23015ebe0 2024-09-16T01:40:57.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.189+0000 7ff236f0a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff23006cde0 con 0x7ff2300b7560 2024-09-16T01:40:57.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.189+0000 7ff236f0a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff23006ed40 con 0x7ff23015e800 2024-09-16T01:40:57.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.189+0000 7ff227fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff23015ebe0 0x7ff2300b6e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:32894/0 (socket says 172.21.15.88:32894) 2024-09-16T01:40:57.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.189+0000 7ff227fff640 1 -- 172.21.15.88:0/1421003893 learned_addr learned my addr 172.21.15.88:0/1421003893 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:57.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff234ca6640 1 --2- 172.21.15.88:0/1421003893 >> v2:172.21.15.154:3300/0 conn(0x7ff23015e800 0x7ff2300b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff2277fe640 1 -- 172.21.15.88:0/1421003893 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2240477890 0 0) 0x7ff23006ea60 con 0x7ff23015ebe0 2024-09-16T01:40:57.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff2277fe640 1 -- 172.21.15.88:0/1421003893 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff218003520 con 0x7ff23015ebe0 2024-09-16T01:40:57.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff2277fe640 1 -- 172.21.15.88:0/1421003893 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 695690882 0 0) 0x7ff218003520 con 0x7ff23015ebe0 2024-09-16T01:40:57.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff2277fe640 1 -- 172.21.15.88:0/1421003893 >> v2:172.21.15.154:3300/0 conn(0x7ff23015e800 msgr2=0x7ff2300b5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:57.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff2277fe640 1 --2- 172.21.15.88:0/1421003893 >> v2:172.21.15.154:3300/0 conn(0x7ff23015e800 0x7ff2300b5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff2277fe640 1 -- 172.21.15.88:0/1421003893 >> v1:172.21.15.154:6789/0 conn(0x7ff2300b7560 legacy=0x7ff23006d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:57.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff2277fe640 1 -- 172.21.15.88:0/1421003893 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff23006d0c0 con 0x7ff23015ebe0 2024-09-16T01:40:57.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff2277fe640 1 -- 172.21.15.88:0/1421003893 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7ff220001670 con 0x7ff23015ebe0 2024-09-16T01:40:57.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff234ca6640 1 --2- 172.21.15.88:0/1421003893 >> v2:172.21.15.154:3300/0 conn(0x7ff23015e800 0x7ff2300b5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:40:57.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff2277fe640 1 -- 172.21.15.88:0/1421003893 >> v1:172.21.15.88:6789/0 conn(0x7ff23015ebe0 legacy=0x7ff2300b6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:57.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff2277fe640 1 --2- 172.21.15.88:0/1421003893 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2180039b0 0x7ff218003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff2277fe640 1 --2- 172.21.15.88:0/1421003893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2180043b0 0x7ff218004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff2277fe640 1 --2- 172.21.15.88:0/1421003893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff218005080 0x7ff21800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff2277fe640 1 -- 172.21.15.88:0/1421003893 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff23006ed40 con 0x7ff2180039b0 2024-09-16T01:40:57.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff2277fe640 1 -- 172.21.15.88:0/1421003893 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff21800bfe0 con 0x7ff2180043b0 2024-09-16T01:40:57.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff2277fe640 1 -- 172.21.15.88:0/1421003893 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff21800c1e0 con 0x7ff218005080 2024-09-16T01:40:57.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff234ca6640 1 --2- 172.21.15.88:0/1421003893 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2180039b0 0x7ff218003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff227fff640 1 --2- 172.21.15.88:0/1421003893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2180043b0 0x7ff218004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff2354a7640 1 --2- 172.21.15.88:0/1421003893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff218005080 0x7ff21800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff234ca6640 1 -- 172.21.15.88:0/1421003893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff218005080 msgr2=0x7ff21800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:57.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff234ca6640 1 --2- 172.21.15.88:0/1421003893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff218005080 0x7ff21800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff234ca6640 1 -- 172.21.15.88:0/1421003893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2180043b0 msgr2=0x7ff218004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:57.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff234ca6640 1 --2- 172.21.15.88:0/1421003893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2180043b0 0x7ff218004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff234ca6640 1 -- 172.21.15.88:0/1421003893 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff218003520 con 0x7ff2180039b0 2024-09-16T01:40:57.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff2354a7640 1 -- 172.21.15.88:0/1421003893 reap_dead start 2024-09-16T01:40:57.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.190+0000 7ff227fff640 1 --2- 172.21.15.88:0/1421003893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2180043b0 0x7ff218004880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:57.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.191+0000 7ff234ca6640 1 --2- 172.21.15.88:0/1421003893 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2180039b0 0x7ff218003d90 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7ff22c00e350 tx=0x7ff218005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=579ac73e06b3b261 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:57.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.191+0000 7ff2277fe640 1 -- 172.21.15.88:0/1421003893 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c033070 con 0x7ff2180039b0 2024-09-16T01:40:57.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.191+0000 7ff2277fe640 1 -- 172.21.15.88:0/1421003893 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff22c02f660 con 0x7ff2180039b0 2024-09-16T01:40:57.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.191+0000 7ff2277fe640 1 -- 172.21.15.88:0/1421003893 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c009de0 con 0x7ff2180039b0 2024-09-16T01:40:57.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.192+0000 7ff236f0a640 1 -- 172.21.15.88:0/1421003893 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2180039b0 msgr2=0x7ff218003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:57.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.192+0000 7ff236f0a640 1 --2- 172.21.15.88:0/1421003893 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2180039b0 0x7ff218003d90 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7ff22c00e350 tx=0x7ff218005830 comp rx=0 tx=0).stop 2024-09-16T01:40:57.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.192+0000 7ff236f0a640 1 -- 172.21.15.88:0/1421003893 shutdown_connections 2024-09-16T01:40:57.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.192+0000 7ff236f0a640 1 --2- 172.21.15.88:0/1421003893 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2180039b0 0x7ff218003d90 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.192+0000 7ff236f0a640 1 -- 172.21.15.88:0/1421003893 >> 172.21.15.88:0/1421003893 conn(0x7ff2300d24d0 msgr2=0x7ff2300d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:57.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.192+0000 7ff236f0a640 1 -- 172.21.15.88:0/1421003893 shutdown_connections 2024-09-16T01:40:57.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.192+0000 7ff236f0a640 1 -- 172.21.15.88:0/1421003893 wait complete. 2024-09-16T01:40:57.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.193+0000 7ff236f0a640 1 Processor -- start 2024-09-16T01:40:57.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.193+0000 7ff236f0a640 1 -- start start 2024-09-16T01:40:57.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.194+0000 7ff236f0a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2180039b0 0x7ff230153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.194+0000 7ff236f0a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2301535d0 0x7ff2301614a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.194+0000 7ff236f0a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2301619e0 0x7ff230156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.194+0000 7ff236f0a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff23006e680 con 0x7ff2180039b0 2024-09-16T01:40:57.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.194+0000 7ff236f0a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff2300d31e0 con 0x7ff2301619e0 2024-09-16T01:40:57.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.194+0000 7ff236f0a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff23006e800 con 0x7ff2301535d0 2024-09-16T01:40:57.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.194+0000 7ff234ca6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2180039b0 0x7ff230153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.194+0000 7ff234ca6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2180039b0 0x7ff230153090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56062/0 (socket says 172.21.15.88:56062) 2024-09-16T01:40:57.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.194+0000 7ff234ca6640 1 -- 172.21.15.88:0/3501778263 learned_addr learned my addr 172.21.15.88:0/3501778263 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:57.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.194+0000 7ff227fff640 1 --2- 172.21.15.88:0/3501778263 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2301535d0 0x7ff2301614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.194+0000 7ff2354a7640 1 --2- 172.21.15.88:0/3501778263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2301619e0 0x7ff230156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.194+0000 7ff234ca6640 1 -- 172.21.15.88:0/3501778263 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2301535d0 msgr2=0x7ff2301614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:57.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.194+0000 7ff234ca6640 1 --2- 172.21.15.88:0/3501778263 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2301535d0 0x7ff2301614a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.194+0000 7ff234ca6640 1 -- 172.21.15.88:0/3501778263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2301619e0 msgr2=0x7ff230156960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:57.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.194+0000 7ff234ca6640 1 --2- 172.21.15.88:0/3501778263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2301619e0 0x7ff230156960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.194+0000 7ff234ca6640 1 -- 172.21.15.88:0/3501778263 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff230151630 con 0x7ff2180039b0 2024-09-16T01:40:57.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.194+0000 7ff2354a7640 1 --2- 172.21.15.88:0/3501778263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2301619e0 0x7ff230156960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:57.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.194+0000 7ff234ca6640 1 --2- 172.21.15.88:0/3501778263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2180039b0 0x7ff230153090 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7ff22c00e470 tx=0x7ff22c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:57.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.195+0000 7ff2257fa640 1 -- 172.21.15.88:0/3501778263 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c033040 con 0x7ff2180039b0 2024-09-16T01:40:57.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.195+0000 7ff2257fa640 1 -- 172.21.15.88:0/3501778263 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff22c02f460 con 0x7ff2180039b0 2024-09-16T01:40:57.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.195+0000 7ff236f0a640 1 -- 172.21.15.88:0/3501778263 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff230160400 con 0x7ff2180039b0 2024-09-16T01:40:57.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.195+0000 7ff2257fa640 1 -- 172.21.15.88:0/3501778263 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22c009de0 con 0x7ff2180039b0 2024-09-16T01:40:57.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.195+0000 7ff236f0a640 1 -- 172.21.15.88:0/3501778263 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2301609c0 con 0x7ff2180039b0 2024-09-16T01:40:57.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.196+0000 7ff236f0a640 1 -- 172.21.15.88:0/3501778263 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff23015f070 con 0x7ff2180039b0 2024-09-16T01:40:57.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.196+0000 7ff2257fa640 1 -- 172.21.15.88:0/3501778263 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff22c048060 con 0x7ff2180039b0 2024-09-16T01:40:57.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.197+0000 7ff2257fa640 1 --2- 172.21.15.88:0/3501778263 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff21c06c2e0 0x7ff21c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:57.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.197+0000 7ff2257fa640 1 -- 172.21.15.88:0/3501778263 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6225+0+0 (secure 0 0 0) 0x7ff22c0abb60 con 0x7ff2180039b0 2024-09-16T01:40:57.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.197+0000 7ff227fff640 1 --2- 172.21.15.88:0/3501778263 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff21c06c2e0 0x7ff21c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:57.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.197+0000 7ff227fff640 1 --2- 172.21.15.88:0/3501778263 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff21c06c2e0 0x7ff21c06e7a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7ff220000fd0 tx=0x7ff220007040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:57.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.201+0000 7ff2257fa640 1 -- 172.21.15.88:0/3501778263 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff22c07a150 con 0x7ff2180039b0 2024-09-16T01:40:57.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.398+0000 7ff236f0a640 1 -- 172.21.15.88:0/3501778263 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff2300693c0 con 0x7ff2180039b0 2024-09-16T01:40:57.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.398+0000 7ff2257fa640 1 -- 172.21.15.88:0/3501778263 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v110) ==== 53+0+73 (secure 0 0 0) 0x7ff22c079a50 con 0x7ff2180039b0 2024-09-16T01:40:57.400 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:40:57.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.400+0000 7ff236f0a640 1 -- 172.21.15.88:0/3501778263 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff21c06c2e0 msgr2=0x7ff21c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:57.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.400+0000 7ff236f0a640 1 --2- 172.21.15.88:0/3501778263 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff21c06c2e0 0x7ff21c06e7a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7ff220000fd0 tx=0x7ff220007040 comp rx=0 tx=0).stop 2024-09-16T01:40:57.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.400+0000 7ff236f0a640 1 -- 172.21.15.88:0/3501778263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2180039b0 msgr2=0x7ff230153090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:57.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.400+0000 7ff236f0a640 1 --2- 172.21.15.88:0/3501778263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2180039b0 0x7ff230153090 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7ff22c00e470 tx=0x7ff22c009750 comp rx=0 tx=0).stop 2024-09-16T01:40:57.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.400+0000 7ff236f0a640 1 -- 172.21.15.88:0/3501778263 shutdown_connections 2024-09-16T01:40:57.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.400+0000 7ff236f0a640 1 --2- 172.21.15.88:0/3501778263 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff21c06c2e0 0x7ff21c06e7a0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.400+0000 7ff236f0a640 1 --2- 172.21.15.88:0/3501778263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2301619e0 0x7ff230156960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.400+0000 7ff236f0a640 1 --2- 172.21.15.88:0/3501778263 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2301535d0 0x7ff2301614a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.400+0000 7ff236f0a640 1 --2- 172.21.15.88:0/3501778263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2180039b0 0x7ff230153090 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:57.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.400+0000 7ff236f0a640 1 -- 172.21.15.88:0/3501778263 >> 172.21.15.88:0/3501778263 conn(0x7ff2300d24d0 msgr2=0x7ff23015bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:57.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.400+0000 7ff236f0a640 1 -- 172.21.15.88:0/3501778263 shutdown_connections 2024-09-16T01:40:57.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57.400+0000 7ff236f0a640 1 -- 172.21.15.88:0/3501778263 wait complete. 2024-09-16T01:40:57.413 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-16T01:40:57.413 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:40:57.413 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2024-09-16T01:40:57.413 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T01:40:57.413 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T01:40:57.413 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-16T01:40:57.413 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-16T01:40:57.413 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:40:57.413 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:40:57.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:57.414 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2024-09-16T01:40:57.429 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:40:57.429 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:40:57.429 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:40:57.429 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:40:57.430 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-09-16T01:40:57.430 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:40:57.430 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:40:57.430 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:40:57.430 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2024-09-16T01:40:57.430 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:40:57.430 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:40:57.430 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2024-09-16T01:40:57.430 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:40:57.430 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2024-09-16T01:40:57.430 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:40:57.430 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2024-09-16T01:40:57.431 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:40:57.431 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42367sec preferred_lft 42367sec 2024-09-16T01:40:57.431 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link noprefixroute 2024-09-16T01:40:57.431 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:40:57.431 INFO:teuthology.orchestra.run.smithi088.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:40:57.431 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff 2024-09-16T01:40:57.431 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T01:40:57.431 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:40:57.431 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::e486:6eff:fe81:a1c/64 scope link 2024-09-16T01:40:57.431 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:40:57.431 INFO:teuthology.orchestra.run.smithi088.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:40:57.431 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 12:6f:29:be:95:48 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:40:57.431 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::106f:29ff:febe:9548/64 scope link 2024-09-16T01:40:57.431 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:40:57.432 INFO:teuthology.orchestra.run.smithi088.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:40:57.432 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T01:40:57.432 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::c63:b4ff:feb3:ad9e/64 scope link 2024-09-16T01:40:57.432 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:40:57.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:57.432 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:40:57.445 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:40:57.444+0000 7f7461742640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:40:57.468 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-09-16T01:40:57.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:57.469 DEBUG:teuthology.orchestra.run.smithi088:> route 2024-09-16T01:40:57.526 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2024-09-16T01:40:57.526 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:40:57.526 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:40:57.526 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:40:57.526 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:40:57.527 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:40:57.527 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:40:57.527 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:40:57.527 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:40:57.527 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:40:57.527 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:40:57.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:40:57.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:40:57.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:57.652 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:40:57.667 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T01:40:57.667 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T01:40:57.668 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:40:57.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:57.668 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2024-09-16T01:40:57.733 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:40:57.733 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T01:40:57.788 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:40:57.788 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:40:57.789 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:40:57.844 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-16T01:40:57.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:57.844 DEBUG:teuthology.orchestra.run.smithi088:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:40:57.926 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:40:57.926 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:40:57.927 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:40:57.935 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:40:57.935 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:40:57.935 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:40:57.935 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:40:57.936 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:40:57.941 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:40:57.941 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:40:57.941 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:40:57.941 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:40:57.941 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:40:57.941 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790" 2024-09-16T01:40:57.941 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2024-09-16T01:40:57.942 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5ace2c91-46f9-482b-8a89-731fbdc171bf" will pass to kernel 2024-09-16T01:40:57.942 INFO:teuthology.orchestra.run.smithi088.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-16T01:40:57.942 INFO:teuthology.orchestra.run.smithi088.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:40:57.942 INFO:teuthology.orchestra.run.smithi088.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:40:57.942 INFO:teuthology.orchestra.run.smithi088.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:40:57.942 INFO:teuthology.orchestra.run.smithi088.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:40:57.943 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:40:57.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:57.943 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:40:58.008 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:40:58.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:40:58.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:40:58.104 INFO:teuthology.orchestra.run.smithi088.stdout:client6724 2024-09-16T01:40:58.104 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:40:58.104 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:40:58.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:58.120 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R .) 2024-09-16T01:40:58.162 INFO:teuthology.orchestra.run:waiting for 20 2024-09-16T01:40:58.176 INFO:teuthology.orchestra.run.smithi088.stdout:.: 2024-09-16T01:40:58.176 INFO:teuthology.orchestra.run.smithi088.stdout:subdir 2024-09-16T01:40:58.176 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:40:58.176 INFO:teuthology.orchestra.run.smithi088.stdout:./subdir: 2024-09-16T01:40:58.176 INFO:teuthology.orchestra.run.smithi088.stdout:sixmegs 2024-09-16T01:40:58.177 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete mds0_openfiles.0' did not caused DAMAGED state 2024-09-16T01:40:58.177 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 500.00000000' 2024-09-16T01:40:58.177 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:40:58.177 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:40:58.233 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:40:58.233 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:40:58.288 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:40:58.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.425+0000 7f7eec5e4640 1 Processor -- start 2024-09-16T01:40:58.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.425+0000 7f7eec5e4640 1 -- start start 2024-09-16T01:40:58.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.426+0000 7f7eec5e4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7ee415b360 0x7f7ee40b2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:58.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.426+0000 7f7eec5e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ee406d430 0x7f7ee40b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:58.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.426+0000 7f7eec5e4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ee40b1bf0 con 0x7f7ee40b11a0 2024-09-16T01:40:58.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.426+0000 7f7eec5e4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7ee40b1ed0 con 0x7f7ee415b360 2024-09-16T01:40:58.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.426+0000 7f7eec5e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7ee40b21b0 con 0x7f7ee406d430 2024-09-16T01:40:58.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.427+0000 7f7ee9b7f640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7ee40b11a0 0x7f7ee406cd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:32902/0 (socket says 172.21.15.88:32902) 2024-09-16T01:40:58.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.427+0000 7f7ee9b7f640 1 -- 172.21.15.88:0/1537606014 learned_addr learned my addr 172.21.15.88:0/1537606014 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.427+0000 7f7eeab81640 1 --2- 172.21.15.88:0/1537606014 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ee406d430 0x7f7ee40b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.427+0000 7f7ee937e640 1 -- 172.21.15.88:0/1537606014 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2694993160 0 0) 0x7f7ee40b1bf0 con 0x7f7ee40b11a0 2024-09-16T01:40:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.427+0000 7f7eea380640 1 --2- 172.21.15.88:0/1537606014 >> v2:172.21.15.154:3300/0 conn(0x7f7ee415b360 0x7f7ee40b2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.427+0000 7f7ee937e640 1 -- 172.21.15.88:0/1537606014 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ed0003520 con 0x7f7ee40b11a0 2024-09-16T01:40:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.427+0000 7f7ee937e640 1 -- 172.21.15.88:0/1537606014 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1450267041 0 0) 0x7f7ed0003520 con 0x7f7ee40b11a0 2024-09-16T01:40:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.427+0000 7f7ee937e640 1 -- 172.21.15.88:0/1537606014 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ee406d430 msgr2=0x7f7ee40b15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.427+0000 7f7ee937e640 1 --2- 172.21.15.88:0/1537606014 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ee406d430 0x7f7ee40b15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.427+0000 7f7ee937e640 1 -- 172.21.15.88:0/1537606014 >> v2:172.21.15.154:3300/0 conn(0x7f7ee415b360 msgr2=0x7f7ee40b2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:58.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.427+0000 7f7ee937e640 1 --2- 172.21.15.88:0/1537606014 >> v2:172.21.15.154:3300/0 conn(0x7f7ee415b360 0x7f7ee40b2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.427+0000 7f7ee937e640 1 -- 172.21.15.88:0/1537606014 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ee40b2490 con 0x7f7ee40b11a0 2024-09-16T01:40:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.427+0000 7f7ee937e640 1 -- 172.21.15.88:0/1537606014 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f7ed4001670 con 0x7f7ee40b11a0 2024-09-16T01:40:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.427+0000 7f7eea380640 1 --2- 172.21.15.88:0/1537606014 >> v2:172.21.15.154:3300/0 conn(0x7f7ee415b360 0x7f7ee40b2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.427+0000 7f7ee937e640 1 -- 172.21.15.88:0/1537606014 >> v1:172.21.15.88:6789/0 conn(0x7f7ee40b11a0 legacy=0x7f7ee406cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.427+0000 7f7ee937e640 1 --2- 172.21.15.88:0/1537606014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ed00039b0 0x7f7ed0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.427+0000 7f7ee937e640 1 --2- 172.21.15.88:0/1537606014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ed0004800 0x7f7ed0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.427+0000 7f7ee937e640 1 --2- 172.21.15.88:0/1537606014 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ee406d430 0x7f7ed00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.427+0000 7f7ee937e640 1 -- 172.21.15.88:0/1537606014 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee40b1ed0 con 0x7f7ed00039b0 2024-09-16T01:40:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.427+0000 7f7ee937e640 1 -- 172.21.15.88:0/1537606014 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee40b21b0 con 0x7f7ed0004800 2024-09-16T01:40:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.427+0000 7f7ee937e640 1 -- 172.21.15.88:0/1537606014 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7ed000baf0 con 0x7f7ee406d430 2024-09-16T01:40:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.427+0000 7f7eea380640 1 --2- 172.21.15.88:0/1537606014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ed00039b0 0x7f7ed0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.427+0000 7f7ee9b7f640 1 --2- 172.21.15.88:0/1537606014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ed0004800 0x7f7ed0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.427+0000 7f7eeab81640 1 --2- 172.21.15.88:0/1537606014 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ee406d430 0x7f7ed00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.427+0000 7f7eea380640 1 -- 172.21.15.88:0/1537606014 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ee406d430 msgr2=0x7f7ed00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.428+0000 7f7eea380640 1 --2- 172.21.15.88:0/1537606014 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ee406d430 0x7f7ed00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.428+0000 7f7eea380640 1 -- 172.21.15.88:0/1537606014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ed0004800 msgr2=0x7f7ed0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:58.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.428+0000 7f7eea380640 1 --2- 172.21.15.88:0/1537606014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ed0004800 0x7f7ed0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:58.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.428+0000 7f7eea380640 1 -- 172.21.15.88:0/1537606014 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed0003520 con 0x7f7ed00039b0 2024-09-16T01:40:58.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.428+0000 7f7eea380640 1 --2- 172.21.15.88:0/1537606014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ed00039b0 0x7f7ed0003dd0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0006fd0 tx=0x7f7ed0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96ab94dddac90844 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:58.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.428+0000 7f7ee937e640 1 -- 172.21.15.88:0/1537606014 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee000f040 con 0x7f7ed00039b0 2024-09-16T01:40:58.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.428+0000 7f7ee937e640 1 -- 172.21.15.88:0/1537606014 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ee002ad00 con 0x7f7ed00039b0 2024-09-16T01:40:58.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.428+0000 7f7ee937e640 1 -- 172.21.15.88:0/1537606014 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0004250 con 0x7f7ed00039b0 2024-09-16T01:40:58.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.429+0000 7f7eec5e4640 1 -- 172.21.15.88:0/1537606014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ed00039b0 msgr2=0x7f7ed0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:58.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.429+0000 7f7eec5e4640 1 --2- 172.21.15.88:0/1537606014 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ed00039b0 0x7f7ed0003dd0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0006fd0 tx=0x7f7ed0005830 comp rx=0 tx=0).stop 2024-09-16T01:40:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.429+0000 7f7eeab81640 1 -- 172.21.15.88:0/1537606014 reap_dead start 2024-09-16T01:40:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.429+0000 7f7eec5e4640 1 -- 172.21.15.88:0/1537606014 shutdown_connections 2024-09-16T01:40:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.429+0000 7f7eec5e4640 1 -- 172.21.15.88:0/1537606014 >> 172.21.15.88:0/1537606014 conn(0x7f7ee4062a50 msgr2=0x7f7ee406e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.429+0000 7f7eec5e4640 1 -- 172.21.15.88:0/1537606014 shutdown_connections 2024-09-16T01:40:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.429+0000 7f7eec5e4640 1 -- 172.21.15.88:0/1537606014 wait complete. 2024-09-16T01:40:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.430+0000 7f7eec5e4640 1 Processor -- start 2024-09-16T01:40:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.430+0000 7f7eec5e4640 1 -- start start 2024-09-16T01:40:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.431+0000 7f7eec5e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ed00039b0 0x7f7ee41f73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.431+0000 7f7eec5e4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ee40af1d0 0x7f7ee41555f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.431+0000 7f7eec5e4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ee406bc40 0x7f7ee406c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.431+0000 7f7eec5e4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee40d11c0 con 0x7f7ee40af1d0 2024-09-16T01:40:58.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.431+0000 7f7eec5e4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee415b740 con 0x7f7ee406bc40 2024-09-16T01:40:58.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.431+0000 7f7eec5e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7ee40b15d0 con 0x7f7ed00039b0 2024-09-16T01:40:58.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.431+0000 7f7ee9b7f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ee40af1d0 0x7f7ee41555f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:58.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.431+0000 7f7ee9b7f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ee40af1d0 0x7f7ee41555f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56076/0 (socket says 172.21.15.88:56076) 2024-09-16T01:40:58.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.431+0000 7f7ee9b7f640 1 -- 172.21.15.88:0/2231270443 learned_addr learned my addr 172.21.15.88:0/2231270443 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:58.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.431+0000 7f7ee9b7f640 1 -- 172.21.15.88:0/2231270443 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ed00039b0 msgr2=0x7f7ee41f73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:58.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.431+0000 7f7ee9b7f640 1 --2- 172.21.15.88:0/2231270443 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ed00039b0 0x7f7ee41f73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:58.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.431+0000 7f7ee9b7f640 1 -- 172.21.15.88:0/2231270443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ee406bc40 msgr2=0x7f7ee406c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:58.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.431+0000 7f7ee9b7f640 1 --2- 172.21.15.88:0/2231270443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ee406bc40 0x7f7ee406c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.431+0000 7f7ee9b7f640 1 -- 172.21.15.88:0/2231270443 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ee40af650 con 0x7f7ee40af1d0 2024-09-16T01:40:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.431+0000 7f7ee9b7f640 1 --2- 172.21.15.88:0/2231270443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ee40af1d0 0x7f7ee41555f0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4007fd0 tx=0x7f7ed4007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.431+0000 7f7edaffd640 1 -- 172.21.15.88:0/2231270443 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed400f730 con 0x7f7ee40af1d0 2024-09-16T01:40:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.431+0000 7f7edaffd640 1 -- 172.21.15.88:0/2231270443 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ed4018070 con 0x7f7ee40af1d0 2024-09-16T01:40:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.432+0000 7f7eec5e4640 1 -- 172.21.15.88:0/2231270443 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ee4155d20 con 0x7f7ee40af1d0 2024-09-16T01:40:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.432+0000 7f7edaffd640 1 -- 172.21.15.88:0/2231270443 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed400fb50 con 0x7f7ee40af1d0 2024-09-16T01:40:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.432+0000 7f7eec5e4640 1 -- 172.21.15.88:0/2231270443 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ee41562e0 con 0x7f7ee40af1d0 2024-09-16T01:40:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.432+0000 7f7eec5e4640 1 -- 172.21.15.88:0/2231270443 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ee40b2c90 con 0x7f7ee40af1d0 2024-09-16T01:40:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.433+0000 7f7edaffd640 1 -- 172.21.15.88:0/2231270443 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7ed400fcf0 con 0x7f7ee40af1d0 2024-09-16T01:40:58.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.433+0000 7f7edaffd640 1 --2- 172.21.15.88:0/2231270443 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7ec806c2b0 0x7f7ec806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:58.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.434+0000 7f7edaffd640 1 -- 172.21.15.88:0/2231270443 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6225+0+0 (secure 0 0 0) 0x7f7ed408b550 con 0x7f7ee40af1d0 2024-09-16T01:40:58.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.434+0000 7f7eea380640 1 --2- 172.21.15.88:0/2231270443 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7ec806c2b0 0x7f7ec806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:58.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.434+0000 7f7eea380640 1 --2- 172.21.15.88:0/2231270443 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7ec806c2b0 0x7f7ec806e770 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0033000 tx=0x7f7ee002c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:58.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.438+0000 7f7edaffd640 1 -- 172.21.15.88:0/2231270443 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7ed4059b40 con 0x7f7ee40af1d0 2024-09-16T01:40:58.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.618+0000 7f7eec5e4640 1 -- 172.21.15.88:0/2231270443 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7ee40d03f0 con 0x7f7ee40af1d0 2024-09-16T01:40:58.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.618+0000 7f7edaffd640 1 -- 172.21.15.88:0/2231270443 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 12 entries v91) ==== 81+0+732 (secure 0 0 0) 0x7f7ed4059440 con 0x7f7ee40af1d0 2024-09-16T01:40:58.620 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2408579939 2024-09-17T01:40:46.434492+0000 2024-09-16T01:40:58.620 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/1289932173 2024-09-17T01:40:42.399399+0000 2024-09-16T01:40:58.620 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2408579939 2024-09-17T01:40:46.434492+0000 2024-09-16T01:40:58.620 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2970185561 2024-09-17T01:40:38.255774+0000 2024-09-16T01:40:58.620 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/3356432718 2024-09-17T01:40:32.344252+0000 2024-09-16T01:40:58.620 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/3356432718 2024-09-17T01:40:32.344252+0000 2024-09-16T01:40:58.620 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/1289932173 2024-09-17T01:40:42.399399+0000 2024-09-16T01:40:58.620 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2970185561 2024-09-17T01:40:38.255774+0000 2024-09-16T01:40:58.620 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2733458327 2024-09-17T01:40:27.314712+0000 2024-09-16T01:40:58.621 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2733458327 2024-09-17T01:40:27.314712+0000 2024-09-16T01:40:58.621 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/4162274767 2024-09-17T01:40:11.522091+0000 2024-09-16T01:40:58.621 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/4162274767 2024-09-17T01:40:11.522091+0000 2024-09-16T01:40:58.622 INFO:teuthology.orchestra.run.smithi088.stderr:listed 12 entries 2024-09-16T01:40:58.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.620+0000 7f7eec5e4640 1 -- 172.21.15.88:0/2231270443 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7ec806c2b0 msgr2=0x7f7ec806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:58.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.620+0000 7f7eec5e4640 1 --2- 172.21.15.88:0/2231270443 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7ec806c2b0 0x7f7ec806e770 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0033000 tx=0x7f7ee002c000 comp rx=0 tx=0).stop 2024-09-16T01:40:58.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.620+0000 7f7eec5e4640 1 -- 172.21.15.88:0/2231270443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ee40af1d0 msgr2=0x7f7ee41555f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:58.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.620+0000 7f7eec5e4640 1 --2- 172.21.15.88:0/2231270443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ee40af1d0 0x7f7ee41555f0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4007fd0 tx=0x7f7ed4007430 comp rx=0 tx=0).stop 2024-09-16T01:40:58.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.620+0000 7f7eec5e4640 1 -- 172.21.15.88:0/2231270443 shutdown_connections 2024-09-16T01:40:58.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.620+0000 7f7eec5e4640 1 --2- 172.21.15.88:0/2231270443 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7ec806c2b0 0x7f7ec806e770 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:58.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.620+0000 7f7eec5e4640 1 --2- 172.21.15.88:0/2231270443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ee406bc40 0x7f7ee406c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:58.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.620+0000 7f7eec5e4640 1 --2- 172.21.15.88:0/2231270443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ee40af1d0 0x7f7ee41555f0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:58.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.620+0000 7f7eec5e4640 1 --2- 172.21.15.88:0/2231270443 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ed00039b0 0x7f7ee41f73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:58.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.620+0000 7f7eec5e4640 1 -- 172.21.15.88:0/2231270443 >> 172.21.15.88:0/2231270443 conn(0x7f7ee4062a50 msgr2=0x7f7ee40d14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:58.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.620+0000 7f7eec5e4640 1 -- 172.21.15.88:0/2231270443 shutdown_connections 2024-09-16T01:40:58.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.620+0000 7f7eec5e4640 1 -- 172.21.15.88:0/2231270443 wait complete. 2024-09-16T01:40:58.633 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-16T01:40:58.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:58.633 DEBUG:teuthology.orchestra.run.smithi088:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-09-16T01:40:58.674 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:40:58.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:40:58.674 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:40:58.729 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:40:58.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.866+0000 7f57cd52a640 1 Processor -- start 2024-09-16T01:40:58.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.866+0000 7f57cd52a640 1 -- start start 2024-09-16T01:40:58.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.867+0000 7f57cd52a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f57c80cf1f0 0x7f57c80b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:58.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.867+0000 7f57cd52a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f57c80b6720 0x7f57c80d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:58.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.867+0000 7f57cd52a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57c80cf860 con 0x7f57c815e7a0 2024-09-16T01:40:58.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.867+0000 7f57cd52a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f57c80b6b90 con 0x7f57c80b6720 2024-09-16T01:40:58.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.867+0000 7f57cd52a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f57c80d2ca0 con 0x7f57c80cf1f0 2024-09-16T01:40:58.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.867+0000 7f57c77fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f57c80b6720 0x7f57c80d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:58.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.867+0000 7f57c77fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f57c80b6720 0x7f57c80d3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56086/0 (socket says 172.21.15.88:56086) 2024-09-16T01:40:58.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.867+0000 7f57c77fe640 1 -- 172.21.15.88:0/2017032659 learned_addr learned my addr 172.21.15.88:0/2017032659 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:58.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.867+0000 7f57c67fc640 1 --2- 172.21.15.88:0/2017032659 >> v2:172.21.15.154:3300/0 conn(0x7f57c80cf1f0 0x7f57c80b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:58.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.867+0000 7f57c77fe640 1 -- 172.21.15.88:0/2017032659 >> v2:172.21.15.154:3300/0 conn(0x7f57c80cf1f0 msgr2=0x7f57c80b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:58.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.867+0000 7f57c77fe640 1 --2- 172.21.15.88:0/2017032659 >> v2:172.21.15.154:3300/0 conn(0x7f57c80cf1f0 0x7f57c80b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:58.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.867+0000 7f57c77fe640 1 -- 172.21.15.88:0/2017032659 >> v1:172.21.15.154:6789/0 conn(0x7f57c815e7a0 legacy=0x7f57c806caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:58.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.867+0000 7f57c77fe640 1 -- 172.21.15.88:0/2017032659 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c80cfb40 con 0x7f57c80b6720 2024-09-16T01:40:58.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.867+0000 7f57c67fc640 1 --2- 172.21.15.88:0/2017032659 >> v2:172.21.15.154:3300/0 conn(0x7f57c80cf1f0 0x7f57c80b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:40:58.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.868+0000 7f57c77fe640 1 --2- 172.21.15.88:0/2017032659 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57c80b6720 0x7f57c80d3410 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f57c80d2110 tx=0x7f57c002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e4767362a75ce80 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:58.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.868+0000 7f57c5ffb640 1 -- 172.21.15.88:0/2017032659 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c0036070 con 0x7f57c80b6720 2024-09-16T01:40:58.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.868+0000 7f57c5ffb640 1 -- 172.21.15.88:0/2017032659 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57c0031600 con 0x7f57c80b6720 2024-09-16T01:40:58.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.868+0000 7f57c5ffb640 1 -- 172.21.15.88:0/2017032659 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c00319d0 con 0x7f57c80b6720 2024-09-16T01:40:58.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.869+0000 7f57cd52a640 1 -- 172.21.15.88:0/2017032659 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57c80b6720 msgr2=0x7f57c80d3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:58.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.869+0000 7f57cd52a640 1 --2- 172.21.15.88:0/2017032659 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57c80b6720 0x7f57c80d3410 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f57c80d2110 tx=0x7f57c002a660 comp rx=0 tx=0).stop 2024-09-16T01:40:58.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.869+0000 7f57cd52a640 1 -- 172.21.15.88:0/2017032659 shutdown_connections 2024-09-16T01:40:58.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.869+0000 7f57cd52a640 1 --2- 172.21.15.88:0/2017032659 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57c80b6720 0x7f57c80d3410 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:58.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.869+0000 7f57cd52a640 1 --2- 172.21.15.88:0/2017032659 >> v2:172.21.15.154:3300/0 conn(0x7f57c80cf1f0 0x7f57c80b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:58.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.869+0000 7f57cd52a640 1 -- 172.21.15.88:0/2017032659 >> 172.21.15.88:0/2017032659 conn(0x7f57c80d11e0 msgr2=0x7f57c80d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:58.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.869+0000 7f57cd52a640 1 -- 172.21.15.88:0/2017032659 shutdown_connections 2024-09-16T01:40:58.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.869+0000 7f57cd52a640 1 -- 172.21.15.88:0/2017032659 wait complete. 2024-09-16T01:40:58.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.870+0000 7f57cd52a640 1 Processor -- start 2024-09-16T01:40:58.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.870+0000 7f57cd52a640 1 -- start start 2024-09-16T01:40:58.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.871+0000 7f57cd52a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57c80b6720 0x7f57c80cae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:58.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.871+0000 7f57cd52a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57c80cf1f0 0x7f57c81458e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:58.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.871+0000 7f57cd52a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57c815e7a0 0x7f57c80ce250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:58.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.871+0000 7f57cd52a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f57c80b7a00 con 0x7f57c80b6720 2024-09-16T01:40:58.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.871+0000 7f57cd52a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f57c80d2ef0 con 0x7f57c80cf1f0 2024-09-16T01:40:58.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.871+0000 7f57cd52a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f57c815eb80 con 0x7f57c815e7a0 2024-09-16T01:40:58.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.871+0000 7f57c6ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57c80b6720 0x7f57c80cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:58.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.871+0000 7f57c6ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57c80b6720 0x7f57c80cae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56096/0 (socket says 172.21.15.88:56096) 2024-09-16T01:40:58.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.871+0000 7f57c6ffd640 1 -- 172.21.15.88:0/100525082 learned_addr learned my addr 172.21.15.88:0/100525082 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:58.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.871+0000 7f57c67fc640 1 --2- 172.21.15.88:0/100525082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57c80cf1f0 0x7f57c81458e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:58.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.871+0000 7f57c77fe640 1 --2- 172.21.15.88:0/100525082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57c815e7a0 0x7f57c80ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:58.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.871+0000 7f57c6ffd640 1 -- 172.21.15.88:0/100525082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57c815e7a0 msgr2=0x7f57c80ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:58.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.871+0000 7f57c6ffd640 1 --2- 172.21.15.88:0/100525082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57c815e7a0 0x7f57c80ce250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:58.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.871+0000 7f57c6ffd640 1 -- 172.21.15.88:0/100525082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57c80cf1f0 msgr2=0x7f57c81458e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:58.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.871+0000 7f57c6ffd640 1 --2- 172.21.15.88:0/100525082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57c80cf1f0 0x7f57c81458e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:58.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.871+0000 7f57c6ffd640 1 -- 172.21.15.88:0/100525082 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c80d3310 con 0x7f57c80b6720 2024-09-16T01:40:58.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.871+0000 7f57c67fc640 1 --2- 172.21.15.88:0/100525082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57c80cf1f0 0x7f57c81458e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:58.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.871+0000 7f57c6ffd640 1 --2- 172.21.15.88:0/100525082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57c80b6720 0x7f57c80cae20 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f57b800c460 tx=0x7f57b800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:58.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.871+0000 7f57af7fe640 1 -- 172.21.15.88:0/100525082 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b8016020 con 0x7f57c80b6720 2024-09-16T01:40:58.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.872+0000 7f57af7fe640 1 -- 172.21.15.88:0/100525082 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57b8012870 con 0x7f57c80b6720 2024-09-16T01:40:58.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.872+0000 7f57cd52a640 1 -- 172.21.15.88:0/100525082 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57c80d35b0 con 0x7f57c80b6720 2024-09-16T01:40:58.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.872+0000 7f57af7fe640 1 -- 172.21.15.88:0/100525082 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b8012c60 con 0x7f57c80b6720 2024-09-16T01:40:58.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.872+0000 7f57cd52a640 1 -- 172.21.15.88:0/100525082 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57c80d3b70 con 0x7f57c80b6720 2024-09-16T01:40:58.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.873+0000 7f57cd52a640 1 -- 172.21.15.88:0/100525082 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57c80b61e0 con 0x7f57c80b6720 2024-09-16T01:40:58.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.873+0000 7f57af7fe640 1 -- 172.21.15.88:0/100525082 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f57b8019020 con 0x7f57c80b6720 2024-09-16T01:40:58.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.874+0000 7f57af7fe640 1 --2- 172.21.15.88:0/100525082 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f57b406c2b0 0x7f57b406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:58.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.874+0000 7f57af7fe640 1 -- 172.21.15.88:0/100525082 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 6225+0+0 (secure 0 0 0) 0x7f57b808b400 con 0x7f57c80b6720 2024-09-16T01:40:58.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.874+0000 7f57c67fc640 1 --2- 172.21.15.88:0/100525082 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f57b406c2b0 0x7f57b406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:58.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.874+0000 7f57c67fc640 1 --2- 172.21.15.88:0/100525082 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f57b406c2b0 0x7f57b406e770 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f57c81e9320 tx=0x7f57bc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:58.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:58.878+0000 7f57af7fe640 1 -- 172.21.15.88:0/100525082 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f57b8020030 con 0x7f57c80b6720 2024-09-16T01:40:59.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.079+0000 7f57cd52a640 1 -- 172.21.15.88:0/100525082 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f57c81e9350 con 0x7f57c80b6720 2024-09-16T01:40:59.081 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:40:59.079+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:40:59.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.081+0000 7f57af7fe640 1 -- 172.21.15.88:0/100525082 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v111) ==== 161+0+0 (secure 0 0 0) 0x7f57b80598d0 con 0x7f57c80b6720 2024-09-16T01:40:59.082 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:40:59.084 INFO:tasks.ceph.mds.a.smithi088.stderr: -1787> 2024-09-16T01:40:50.459+0000 7f745d73a640 -1 mds.0.openfiles _load_finish got (2) No such file or directory 2024-09-16T01:40:59.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.083+0000 7f57cd52a640 1 -- 172.21.15.88:0/100525082 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f57b406c2b0 msgr2=0x7f57b406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:59.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.083+0000 7f57cd52a640 1 --2- 172.21.15.88:0/100525082 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f57b406c2b0 0x7f57b406e770 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f57c81e9320 tx=0x7f57bc008040 comp rx=0 tx=0).stop 2024-09-16T01:40:59.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.083+0000 7f57cd52a640 1 -- 172.21.15.88:0/100525082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57c80b6720 msgr2=0x7f57c80cae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:59.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.083+0000 7f57cd52a640 1 --2- 172.21.15.88:0/100525082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57c80b6720 0x7f57c80cae20 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f57b800c460 tx=0x7f57b800c920 comp rx=0 tx=0).stop 2024-09-16T01:40:59.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.083+0000 7f57cd52a640 1 -- 172.21.15.88:0/100525082 shutdown_connections 2024-09-16T01:40:59.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.083+0000 7f57cd52a640 1 --2- 172.21.15.88:0/100525082 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f57b406c2b0 0x7f57b406e770 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:59.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.083+0000 7f57cd52a640 1 --2- 172.21.15.88:0/100525082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57c815e7a0 0x7f57c80ce250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:59.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.083+0000 7f57cd52a640 1 --2- 172.21.15.88:0/100525082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57c80cf1f0 0x7f57c81458e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:59.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.083+0000 7f57cd52a640 1 --2- 172.21.15.88:0/100525082 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57c80b6720 0x7f57c80cae20 secure :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f57b800c460 tx=0x7f57b800c920 comp rx=0 tx=0).stop 2024-09-16T01:40:59.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.083+0000 7f57cd52a640 1 -- 172.21.15.88:0/100525082 >> 172.21.15.88:0/100525082 conn(0x7f57c80d11e0 msgr2=0x7f57c81e9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:59.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.084+0000 7f57cd52a640 1 -- 172.21.15.88:0/100525082 shutdown_connections 2024-09-16T01:40:59.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.084+0000 7f57cd52a640 1 -- 172.21.15.88:0/100525082 wait complete. 2024-09-16T01:40:59.086 INFO:tasks.ceph.mds.a.smithi088.stderr: -525> 2024-09-16T01:40:57.444+0000 7f7461742640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:40:59.096 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:40:59.111 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:40:59.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.197+0000 7f9693fff640 1 Processor -- start 2024-09-16T01:40:59.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.197+0000 7f9693fff640 1 -- start start 2024-09-16T01:40:59.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f9693fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96940d2e30 0x7f96940b5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:59.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f9693fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f96940b6bf0 0x7f96940b7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:59.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f9693fff640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f969406cd20 con 0x7f969415e700 2024-09-16T01:40:59.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f9693fff640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f969406d000 con 0x7f96940b6bf0 2024-09-16T01:40:59.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f9693fff640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f969406d2e0 con 0x7f96940d2e30 2024-09-16T01:40:59.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f9692ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f969415e700 0x7f969415eae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:32922/0 (socket says 172.21.15.88:32922) 2024-09-16T01:40:59.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f9692ffd640 1 -- 172.21.15.88:0/1434537727 learned_addr learned my addr 172.21.15.88:0/1434537727 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:40:59.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f96937fe640 1 --2- 172.21.15.88:0/1434537727 >> v2:172.21.15.154:3300/0 conn(0x7f96940b6bf0 0x7f96940b7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:59.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f96927fc640 1 --2- 172.21.15.88:0/1434537727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96940d2e30 0x7f96940b5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:59.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f9691ffb640 1 -- 172.21.15.88:0/1434537727 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 64525376 0 0) 0x7f969406cd20 con 0x7f969415e700 2024-09-16T01:40:59.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f9691ffb640 1 -- 172.21.15.88:0/1434537727 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f967c003520 con 0x7f969415e700 2024-09-16T01:40:59.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f9691ffb640 1 -- 172.21.15.88:0/1434537727 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2586446816 0 0) 0x7f967c003520 con 0x7f969415e700 2024-09-16T01:40:59.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f9691ffb640 1 -- 172.21.15.88:0/1434537727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96940d2e30 msgr2=0x7f96940b5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:59.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f9691ffb640 1 --2- 172.21.15.88:0/1434537727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96940d2e30 0x7f96940b5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:59.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f9691ffb640 1 -- 172.21.15.88:0/1434537727 >> v2:172.21.15.154:3300/0 conn(0x7f96940b6bf0 msgr2=0x7f96940b7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:59.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f9691ffb640 1 --2- 172.21.15.88:0/1434537727 >> v2:172.21.15.154:3300/0 conn(0x7f96940b6bf0 0x7f96940b7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:59.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f9691ffb640 1 -- 172.21.15.88:0/1434537727 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f969406d5c0 con 0x7f969415e700 2024-09-16T01:40:59.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f9691ffb640 1 -- 172.21.15.88:0/1434537727 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f9684001670 con 0x7f969415e700 2024-09-16T01:40:59.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f9691ffb640 1 -- 172.21.15.88:0/1434537727 >> v1:172.21.15.88:6789/0 conn(0x7f969415e700 legacy=0x7f969415eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:59.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f9691ffb640 1 --2- 172.21.15.88:0/1434537727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f967c0039b0 0x7f967c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:59.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f9691ffb640 1 --2- 172.21.15.88:0/1434537727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96940d2e30 0x7f967c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:59.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f9691ffb640 1 --2- 172.21.15.88:0/1434537727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f967c004d40 0x7f967c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:59.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f9691ffb640 1 -- 172.21.15.88:0/1434537727 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f969406d000 con 0x7f967c0039b0 2024-09-16T01:40:59.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f9691ffb640 1 -- 172.21.15.88:0/1434537727 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f969406d2e0 con 0x7f967c004d40 2024-09-16T01:40:59.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f9691ffb640 1 -- 172.21.15.88:0/1434537727 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f967c00b9f0 con 0x7f96940d2e30 2024-09-16T01:40:59.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.198+0000 7f9692ffd640 1 --2- 172.21.15.88:0/1434537727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f967c0039b0 0x7f967c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:59.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.199+0000 7f96927fc640 1 --2- 172.21.15.88:0/1434537727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96940d2e30 0x7f967c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:59.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.199+0000 7f96937fe640 1 --2- 172.21.15.88:0/1434537727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f967c004d40 0x7f967c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:59.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.199+0000 7f9692ffd640 1 -- 172.21.15.88:0/1434537727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96940d2e30 msgr2=0x7f967c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:59.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.199+0000 7f9692ffd640 1 --2- 172.21.15.88:0/1434537727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96940d2e30 0x7f967c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:59.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.199+0000 7f9692ffd640 1 -- 172.21.15.88:0/1434537727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f967c004d40 msgr2=0x7f967c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:40:59.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.199+0000 7f9692ffd640 1 --2- 172.21.15.88:0/1434537727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f967c004d40 0x7f967c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:59.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.199+0000 7f9692ffd640 1 -- 172.21.15.88:0/1434537727 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f967c003520 con 0x7f967c0039b0 2024-09-16T01:40:59.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.199+0000 7f96927fc640 1 --2- 172.21.15.88:0/1434537727 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96940d2e30 0x7f967c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:40:59.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.199+0000 7f9692ffd640 1 --2- 172.21.15.88:0/1434537727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f967c0039b0 0x7f967c003dd0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f9684001050 tx=0x7f967c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a25ed2777522064 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:59.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.200+0000 7f9691ffb640 1 -- 172.21.15.88:0/1434537727 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f968402e070 con 0x7f967c0039b0 2024-09-16T01:40:59.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.200+0000 7f9691ffb640 1 -- 172.21.15.88:0/1434537727 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9684005af0 con 0x7f967c0039b0 2024-09-16T01:40:59.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.200+0000 7f9691ffb640 1 -- 172.21.15.88:0/1434537727 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9684007d70 con 0x7f967c0039b0 2024-09-16T01:40:59.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.200+0000 7f9693fff640 1 -- 172.21.15.88:0/1434537727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f967c0039b0 msgr2=0x7f967c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:59.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.200+0000 7f9693fff640 1 --2- 172.21.15.88:0/1434537727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f967c0039b0 0x7f967c003dd0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f9684001050 tx=0x7f967c005830 comp rx=0 tx=0).stop 2024-09-16T01:40:59.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.200+0000 7f96937fe640 1 -- 172.21.15.88:0/1434537727 reap_dead start 2024-09-16T01:40:59.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.200+0000 7f9693fff640 1 -- 172.21.15.88:0/1434537727 shutdown_connections 2024-09-16T01:40:59.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.200+0000 7f9693fff640 1 -- 172.21.15.88:0/1434537727 >> 172.21.15.88:0/1434537727 conn(0x7f96940d2340 msgr2=0x7f96940d2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:40:59.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.200+0000 7f9693fff640 1 -- 172.21.15.88:0/1434537727 shutdown_connections 2024-09-16T01:40:59.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.200+0000 7f9693fff640 1 -- 172.21.15.88:0/1434537727 wait complete. 2024-09-16T01:40:59.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.201+0000 7f9693fff640 1 Processor -- start 2024-09-16T01:40:59.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.201+0000 7f9693fff640 1 -- start start 2024-09-16T01:40:59.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.202+0000 7f9693fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f967c0039b0 0x7f96940c7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:59.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.202+0000 7f9693fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96941f80c0 0x7f9694154d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:59.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.202+0000 7f9693fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96941474d0 0x7f9694147980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:59.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.202+0000 7f9693fff640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f96940b7580 con 0x7f96941474d0 2024-09-16T01:40:59.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.202+0000 7f9693fff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f969406db80 con 0x7f967c0039b0 2024-09-16T01:40:59.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.202+0000 7f9693fff640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f96940b5310 con 0x7f96941f80c0 2024-09-16T01:40:59.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.202+0000 7f96937fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96941474d0 0x7f9694147980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:59.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.202+0000 7f96937fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96941474d0 0x7f9694147980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56116/0 (socket says 172.21.15.88:56116) 2024-09-16T01:40:59.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.202+0000 7f96937fe640 1 -- 172.21.15.88:0/1883630247 learned_addr learned my addr 172.21.15.88:0/1883630247 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:40:59.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.202+0000 7f9692ffd640 1 --2- 172.21.15.88:0/1883630247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f967c0039b0 0x7f96940c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:59.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.202+0000 7f96927fc640 1 --2- 172.21.15.88:0/1883630247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96941f80c0 0x7f9694154d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:59.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.203+0000 7f96937fe640 1 -- 172.21.15.88:0/1883630247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96941f80c0 msgr2=0x7f9694154d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:59.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.203+0000 7f96937fe640 1 --2- 172.21.15.88:0/1883630247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96941f80c0 0x7f9694154d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:59.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.203+0000 7f96937fe640 1 -- 172.21.15.88:0/1883630247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f967c0039b0 msgr2=0x7f96940c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:40:59.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.203+0000 7f96937fe640 1 --2- 172.21.15.88:0/1883630247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f967c0039b0 0x7f96940c7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:40:59.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.203+0000 7f96937fe640 1 -- 172.21.15.88:0/1883630247 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f969414a5f0 con 0x7f96941474d0 2024-09-16T01:40:59.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.203+0000 7f96937fe640 1 --2- 172.21.15.88:0/1883630247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96941474d0 0x7f9694147980 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f96940b69e0 tx=0x7f968c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:59.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.203+0000 7f967b7fe640 1 -- 172.21.15.88:0/1883630247 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f968c00fde0 con 0x7f96941474d0 2024-09-16T01:40:59.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.203+0000 7f967b7fe640 1 -- 172.21.15.88:0/1883630247 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f968c01d070 con 0x7f96941474d0 2024-09-16T01:40:59.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.203+0000 7f9693fff640 1 -- 172.21.15.88:0/1883630247 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f969414b370 con 0x7f96941474d0 2024-09-16T01:40:59.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.203+0000 7f967b7fe640 1 -- 172.21.15.88:0/1883630247 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f968c015670 con 0x7f96941474d0 2024-09-16T01:40:59.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.203+0000 7f9693fff640 1 -- 172.21.15.88:0/1883630247 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96940c78d0 con 0x7f96941474d0 2024-09-16T01:40:59.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.204+0000 7f9693fff640 1 -- 172.21.15.88:0/1883630247 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96940b7040 con 0x7f96941474d0 2024-09-16T01:40:59.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.204+0000 7f967b7fe640 1 -- 172.21.15.88:0/1883630247 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f968c015810 con 0x7f96941474d0 2024-09-16T01:40:59.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.205+0000 7f967b7fe640 1 --2- 172.21.15.88:0/1883630247 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f968006c2e0 0x7f968006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:40:59.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.205+0000 7f9692ffd640 1 --2- 172.21.15.88:0/1883630247 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f968006c2e0 0x7f968006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:40:59.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.205+0000 7f967b7fe640 1 -- 172.21.15.88:0/1883630247 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6311+0+0 (secure 0 0 0) 0x7f968c08bec0 con 0x7f96941474d0 2024-09-16T01:40:59.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.205+0000 7f9692ffd640 1 --2- 172.21.15.88:0/1883630247 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f968006c2e0 0x7f968006e7a0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f9684033000 tx=0x7f9684009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:40:59.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.209+0000 7f967b7fe640 1 -- 172.21.15.88:0/1883630247 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f968c010040 con 0x7f96941474d0 2024-09-16T01:40:59.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:40:59.444+0000 7f9693fff640 1 -- 172.21.15.88:0/1883630247 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f969415d200 con 0x7f96941474d0 2024-09-16T01:41:00.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.082+0000 7f967b7fe640 1 -- 172.21.15.88:0/1883630247 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v112) ==== 107+0+0 (secure 0 0 0) 0x7f968c05a3d0 con 0x7f96941474d0 2024-09-16T01:41:00.084 INFO:teuthology.orchestra.run.smithi088.stderr:nothing to do: rank is not damaged 2024-09-16T01:41:00.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.085+0000 7f9693fff640 1 -- 172.21.15.88:0/1883630247 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f968006c2e0 msgr2=0x7f968006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:00.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.085+0000 7f9693fff640 1 --2- 172.21.15.88:0/1883630247 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f968006c2e0 0x7f968006e7a0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f9684033000 tx=0x7f9684009030 comp rx=0 tx=0).stop 2024-09-16T01:41:00.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.085+0000 7f9693fff640 1 -- 172.21.15.88:0/1883630247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96941474d0 msgr2=0x7f9694147980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:00.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.085+0000 7f9693fff640 1 --2- 172.21.15.88:0/1883630247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96941474d0 0x7f9694147980 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f96940b69e0 tx=0x7f968c00f580 comp rx=0 tx=0).stop 2024-09-16T01:41:00.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.085+0000 7f9693fff640 1 -- 172.21.15.88:0/1883630247 shutdown_connections 2024-09-16T01:41:00.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.085+0000 7f9693fff640 1 --2- 172.21.15.88:0/1883630247 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f968006c2e0 0x7f968006e7a0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.085+0000 7f9693fff640 1 --2- 172.21.15.88:0/1883630247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f96941474d0 0x7f9694147980 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.085+0000 7f9693fff640 1 --2- 172.21.15.88:0/1883630247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f96941f80c0 0x7f9694154d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.085+0000 7f9693fff640 1 --2- 172.21.15.88:0/1883630247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f967c0039b0 0x7f96940c7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.085+0000 7f9693fff640 1 -- 172.21.15.88:0/1883630247 >> 172.21.15.88:0/1883630247 conn(0x7f96940d2340 msgr2=0x7f96940d2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:00.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.085+0000 7f9693fff640 1 -- 172.21.15.88:0/1883630247 shutdown_connections 2024-09-16T01:41:00.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.085+0000 7f9693fff640 1 -- 172.21.15.88:0/1883630247 wait complete. 2024-09-16T01:41:00.099 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:41:00.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.181+0000 7f74be329f40 1 Processor -- start 2024-09-16T01:41:00.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.181+0000 7f74be329f40 1 -- start start 2024-09-16T01:41:00.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.182+0000 7f74be329f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564a17088700 0x564a17106b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.182+0000 7f74be329f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a17088ae0 0x564a17113080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.182+0000 7f74be329f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564a171167b0 0x564a171135c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.182+0000 7f74be329f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564a17040a70 con 0x564a171167b0 2024-09-16T01:41:00.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.182+0000 7f74be329f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564a16f90510 con 0x564a17088ae0 2024-09-16T01:41:00.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.182+0000 7f74be329f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x564a171149d0 con 0x564a17088700 2024-09-16T01:41:00.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.182+0000 7f74bc622640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564a171167b0 0x564a171135c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:00.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.182+0000 7f74bc622640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564a171167b0 0x564a171135c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56128/0 (socket says 172.21.15.88:56128) 2024-09-16T01:41:00.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.182+0000 7f74bc622640 1 -- 172.21.15.88:0/1957134304 learned_addr learned my addr 172.21.15.88:0/1957134304 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:00.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.183+0000 7f74bb620640 1 --2- 172.21.15.88:0/1957134304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a17088ae0 0x564a17113080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:00.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.183+0000 7f74bc622640 1 -- 172.21.15.88:0/1957134304 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564a17088700 msgr2=0x564a17106b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:00.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.183+0000 7f74bbe21640 1 --2- 172.21.15.88:0/1957134304 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564a17088700 0x564a17106b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:00.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.183+0000 7f74bc622640 1 --2- 172.21.15.88:0/1957134304 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564a17088700 0x564a17106b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.183+0000 7f74bc622640 1 -- 172.21.15.88:0/1957134304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a17088ae0 msgr2=0x564a17113080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:00.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.183+0000 7f74bc622640 1 --2- 172.21.15.88:0/1957134304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a17088ae0 0x564a17113080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.183+0000 7f74bc622640 1 -- 172.21.15.88:0/1957134304 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a171070a0 con 0x564a171167b0 2024-09-16T01:41:00.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.183+0000 7f74bb620640 1 --2- 172.21.15.88:0/1957134304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a17088ae0 0x564a17113080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:00.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.183+0000 7f74bbe21640 1 --2- 172.21.15.88:0/1957134304 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564a17088700 0x564a17106b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:00.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.183+0000 7f74bc622640 1 --2- 172.21.15.88:0/1957134304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564a171167b0 0x564a171135c0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f74b400b650 tx=0x7f74b400bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6c4bf0e02deda server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:00.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.183+0000 7f74ba61e640 1 -- 172.21.15.88:0/1957134304 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b4014020 con 0x564a171167b0 2024-09-16T01:41:00.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.183+0000 7f74ba61e640 1 -- 172.21.15.88:0/1957134304 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74b4009930 con 0x564a171167b0 2024-09-16T01:41:00.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.183+0000 7f74ba61e640 1 -- 172.21.15.88:0/1957134304 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b4009d20 con 0x564a171167b0 2024-09-16T01:41:00.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.183+0000 7f74be329f40 1 -- 172.21.15.88:0/1957134304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564a171167b0 msgr2=0x564a171135c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:00.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.183+0000 7f74be329f40 1 --2- 172.21.15.88:0/1957134304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564a171167b0 0x564a171135c0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f74b400b650 tx=0x7f74b400bb10 comp rx=0 tx=0).stop 2024-09-16T01:41:00.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.183+0000 7f74be329f40 1 -- 172.21.15.88:0/1957134304 shutdown_connections 2024-09-16T01:41:00.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.183+0000 7f74be329f40 1 --2- 172.21.15.88:0/1957134304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564a171167b0 0x564a171135c0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.183+0000 7f74be329f40 1 --2- 172.21.15.88:0/1957134304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a17088ae0 0x564a17113080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.183+0000 7f74be329f40 1 --2- 172.21.15.88:0/1957134304 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564a17088700 0x564a17106b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.183+0000 7f74be329f40 1 -- 172.21.15.88:0/1957134304 >> 172.21.15.88:0/1957134304 conn(0x564a17069680 msgr2=0x564a1706e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:00.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.183+0000 7f74be329f40 1 -- 172.21.15.88:0/1957134304 shutdown_connections 2024-09-16T01:41:00.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.183+0000 7f74be329f40 1 -- 172.21.15.88:0/1957134304 wait complete. 2024-09-16T01:41:00.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.184+0000 7f74be329f40 1 Processor -- start 2024-09-16T01:41:00.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.184+0000 7f74be329f40 1 -- start start 2024-09-16T01:41:00.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.185+0000 7f74be329f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564a17088700 0x564a17101280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.185+0000 7f74be329f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a17088ae0 0x564a170800e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.185+0000 7f74be329f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564a171167b0 0x564a17104210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.185+0000 7f74be329f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564a1710f8e0 con 0x564a17088700 2024-09-16T01:41:00.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.185+0000 7f74be329f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564a1710f600 con 0x564a17088ae0 2024-09-16T01:41:00.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.185+0000 7f74be329f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x564a1710fbc0 con 0x564a171167b0 2024-09-16T01:41:00.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.185+0000 7f74bbe21640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564a17088700 0x564a17101280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:00.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.185+0000 7f74bbe21640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564a17088700 0x564a17101280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:56130/0 (socket says 172.21.15.88:56130) 2024-09-16T01:41:00.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.185+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 learned_addr learned my addr 172.21.15.88:0/3954937878 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:00.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.185+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564a171167b0 msgr2=0x564a17104210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:00.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.185+0000 7f74bbe21640 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564a171167b0 0x564a17104210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.185+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a17088ae0 msgr2=0x564a170800e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:00.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.185+0000 7f74bbe21640 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a17088ae0 0x564a170800e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.185+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a1703ff60 con 0x564a17088700 2024-09-16T01:41:00.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.185+0000 7f74bbe21640 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564a17088700 0x564a17101280 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00cf70 tx=0x7f74ac00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:00.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.185+0000 7f74b961c640 1 -- 172.21.15.88:0/3954937878 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ac014070 con 0x564a17088700 2024-09-16T01:41:00.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.185+0000 7f74b961c640 1 -- 172.21.15.88:0/3954937878 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74ac00ae30 con 0x564a17088700 2024-09-16T01:41:00.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.185+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564a1706d330 con 0x564a17088700 2024-09-16T01:41:00.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.185+0000 7f74b961c640 1 -- 172.21.15.88:0/3954937878 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ac010880 con 0x564a17088700 2024-09-16T01:41:00.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.186+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a1706d8f0 con 0x564a17088700 2024-09-16T01:41:00.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.187+0000 7f74b961c640 1 -- 172.21.15.88:0/3954937878 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f74ac00e040 con 0x564a17088700 2024-09-16T01:41:00.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.187+0000 7f74b961c640 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f74a806ac90 0x7f74a806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.187+0000 7f74b961c640 1 -- 172.21.15.88:0/3954937878 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6311+0+0 (secure 0 0 0) 0x7f74ac089f50 con 0x564a17088700 2024-09-16T01:41:00.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.187+0000 7f74bb620640 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f74a806ac90 0x7f74a806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:00.193 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:41:00.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.188+0000 7f74be329f40 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x564a17104750 0x564a1706e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.188+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a17104ce0 con 0x564a17104750 2024-09-16T01:41:00.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.188+0000 7f74bc622640 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x564a17104750 0x564a1706e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:00.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.188+0000 7f74bb620640 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f74a806ac90 0x7f74a806d150 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f74b0057000 tx=0x7f74b0051000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:00.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.188+0000 7f74bc622640 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x564a17104750 0x564a1706e0f0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:00.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.189+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v92'195 uv169 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.189+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e92) -- 0x564a1706e630 con 0x564a17104750 2024-09-16T01:41:00.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.191+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v92'196 uv196 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.191+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a1706ea60 con 0x564a17104750 2024-09-16T01:41:00.195 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:41:00.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.192+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v92'197 uv197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.192+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7fb0 con 0x564a17104750 2024-09-16T01:41:00.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.193+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v92'198 uv198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.193+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171b8350 con 0x564a17104750 2024-09-16T01:41:00.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.194+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v92'199 uv199 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.194+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171b6fa0 con 0x564a17104750 2024-09-16T01:41:00.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.195+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v92'200 uv200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.195+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7390 con 0x564a17104750 2024-09-16T01:41:00.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.196+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v92'201 uv175 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.196+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7390 con 0x564a17104750 2024-09-16T01:41:00.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.197+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v92'202 uv202 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.197+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7340 con 0x564a17104750 2024-09-16T01:41:00.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.198+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v92'203 uv203 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.199 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:41:00.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.198+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7340 con 0x564a17104750 2024-09-16T01:41:00.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.199+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v92'204 uv204 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.199+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7340 con 0x564a17104750 2024-09-16T01:41:00.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.200+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v92'205 uv205 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.200+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7340 con 0x564a17104750 2024-09-16T01:41:00.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.201+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v92'206 uv206 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.202+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7340 con 0x564a17104750 2024-09-16T01:41:00.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.203+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v92'207 uv182 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.203+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7340 con 0x564a17104750 2024-09-16T01:41:00.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.205+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v92'208 uv208 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.205+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7340 con 0x564a17104750 2024-09-16T01:41:00.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.206+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v92'209 uv209 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.207 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:41:00.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.206+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7340 con 0x564a17104750 2024-09-16T01:41:00.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.207+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v92'210 uv210 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.207+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7340 con 0x564a17104750 2024-09-16T01:41:00.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.208+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v92'211 uv211 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.208+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7340 con 0x564a17104750 2024-09-16T01:41:00.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.209+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v92'212 uv212 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.209+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7340 con 0x564a17104750 2024-09-16T01:41:00.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.210+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v92'213 uv213 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.211+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7340 con 0x564a17104750 2024-09-16T01:41:00.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.212+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v92'214 uv188 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.212+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7340 con 0x564a17104750 2024-09-16T01:41:00.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.213+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v92'215 uv215 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.213+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7340 con 0x564a17104750 2024-09-16T01:41:00.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.214+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v92'216 uv216 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.215 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:41:00.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.214+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7340 con 0x564a17104750 2024-09-16T01:41:00.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.215+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v92'217 uv217 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.215+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7340 con 0x564a17104750 2024-09-16T01:41:00.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.216+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v92'218 uv218 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.216+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7340 con 0x564a17104750 2024-09-16T01:41:00.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.217+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v92'219 uv219 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.217+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7340 con 0x564a17104750 2024-09-16T01:41:00.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.218+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v92'220 uv194 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.218+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7340 con 0x564a17104750 2024-09-16T01:41:00.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.219+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v92'221 uv221 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.219+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7340 con 0x564a17104750 2024-09-16T01:41:00.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.220+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v92'222 uv222 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.221 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:41:00.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.220+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7340 con 0x564a17104750 2024-09-16T01:41:00.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.221+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v92'223 uv223 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.221+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7340 con 0x564a17104750 2024-09-16T01:41:00.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.222+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v92'224 uv224 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.222+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7340 con 0x564a17104750 2024-09-16T01:41:00.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.223+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v92'225 uv225 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.223+0000 7f74be329f40 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x564a171b86f0 0x564a171baae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.223+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171b7340 con 0x564a171b86f0 2024-09-16T01:41:00.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.224+0000 7f74bbe21640 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x564a171b86f0 0x564a171baae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:00.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.224+0000 7f74bbe21640 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x564a171b86f0 0x564a171baae0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:00.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.225+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v92'153 uv134 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171b86f0 2024-09-16T01:41:00.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.225+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e92) -- 0x564a171bb020 con 0x564a171b86f0 2024-09-16T01:41:00.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.227+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v92'154 uv154 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171b86f0 2024-09-16T01:41:00.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.227+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171bb3c0 con 0x564a171b86f0 2024-09-16T01:41:00.232 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:41:00.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.230+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v92'155 uv155 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171b86f0 2024-09-16T01:41:00.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.230+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171bbc10 con 0x564a171b86f0 2024-09-16T01:41:00.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.232+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v92'156 uv156 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171b86f0 2024-09-16T01:41:00.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.232+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171bbfb0 con 0x564a171b86f0 2024-09-16T01:41:00.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.235+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v92'157 uv157 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171b86f0 2024-09-16T01:41:00.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.235+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171bc350 con 0x564a171b86f0 2024-09-16T01:41:00.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.237+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v92'158 uv151 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171b86f0 2024-09-16T01:41:00.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.237+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e92) -- 0x564a171bc6f0 con 0x564a171b86f0 2024-09-16T01:41:00.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.239+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v92'159 uv159 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171b86f0 2024-09-16T01:41:00.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.239+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171bca90 con 0x564a171b86f0 2024-09-16T01:41:00.243 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:41:00.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.242+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v92'160 uv160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171b86f0 2024-09-16T01:41:00.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.242+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171bca90 con 0x564a171b86f0 2024-09-16T01:41:00.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.245+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v92'161 uv161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171b86f0 2024-09-16T01:41:00.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.245+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171bca90 con 0x564a171b86f0 2024-09-16T01:41:00.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.247+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v92'162 uv162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171b86f0 2024-09-16T01:41:00.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.247+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171bca90 con 0x564a171b86f0 2024-09-16T01:41:00.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.250+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v92'163 uv152 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171b86f0 2024-09-16T01:41:00.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.250+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e92) -- 0x564a171bca90 con 0x564a171b86f0 2024-09-16T01:41:00.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.252+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v92'164 uv164 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171b86f0 2024-09-16T01:41:00.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.252+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171bca90 con 0x564a171b86f0 2024-09-16T01:41:00.254 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:41:00.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.253+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v92'165 uv165 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171b86f0 2024-09-16T01:41:00.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.253+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c0f10 con 0x564a171b86f0 2024-09-16T01:41:00.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.255+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v92'166 uv166 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171b86f0 2024-09-16T01:41:00.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.255+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c0f10 con 0x564a171b86f0 2024-09-16T01:41:00.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.257+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v92'167 uv167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171b86f0 2024-09-16T01:41:00.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.257+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171bca90 con 0x564a171b86f0 2024-09-16T01:41:00.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.258+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v92'168 uv148 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171b86f0 2024-09-16T01:41:00.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.258+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e92) -- 0x564a171bca90 con 0x564a171b86f0 2024-09-16T01:41:00.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.259+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v92'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171b86f0 2024-09-16T01:41:00.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.259+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171bca90 con 0x564a171b86f0 2024-09-16T01:41:00.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.260+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v92'170 uv170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171b86f0 2024-09-16T01:41:00.261 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:41:00.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.260+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171bdaa0 con 0x564a171b86f0 2024-09-16T01:41:00.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.261+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v92'171 uv171 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171b86f0 2024-09-16T01:41:00.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.261+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171bdaa0 con 0x564a171b86f0 2024-09-16T01:41:00.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.262+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v92'172 uv172 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171b86f0 2024-09-16T01:41:00.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.262+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171bca90 con 0x564a171b86f0 2024-09-16T01:41:00.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.264+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v92'173 uv173 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171b86f0 2024-09-16T01:41:00.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.264+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171bca90 con 0x564a17104750 2024-09-16T01:41:00.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.265+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v92'39 uv38 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.265+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e92) -- 0x564a171bca90 con 0x564a17104750 2024-09-16T01:41:00.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.266+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v92'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.266+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171bca90 con 0x564a17104750 2024-09-16T01:41:00.268 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:41:00.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.267+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v92'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.267+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171bdaa0 con 0x564a17104750 2024-09-16T01:41:00.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.268+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v92'42 uv42 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.268+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171bdaa0 con 0x564a17104750 2024-09-16T01:41:00.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.269+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v92'43 uv43 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.269+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171bca90 con 0x564a17104750 2024-09-16T01:41:00.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.270+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v92'44 uv44 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.270+0000 7f74be329f40 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x564a171bcfa0 0x564a171bf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.270+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171bfa50 con 0x564a171bcfa0 2024-09-16T01:41:00.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.270+0000 7f74bb620640 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x564a171bcfa0 0x564a171bf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:00.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.271+0000 7f74bb620640 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x564a171bcfa0 0x564a171bf3f0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:00.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.271+0000 7f74bb620640 1 -- 172.21.15.88:0/3954937878 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v92'33 uv32 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f74b0057030 con 0x564a171bcfa0 2024-09-16T01:41:00.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.271+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e92) -- 0x564a171bff70 con 0x564a171bcfa0 2024-09-16T01:41:00.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.273+0000 7f74bb620640 1 -- 172.21.15.88:0/3954937878 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v92'34 uv34 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f74b0057030 con 0x564a171bcfa0 2024-09-16T01:41:00.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.273+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c0470 con 0x564a171bcfa0 2024-09-16T01:41:00.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.274+0000 7f74bb620640 1 -- 172.21.15.88:0/3954937878 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v92'35 uv35 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f74b0057030 con 0x564a171bcfa0 2024-09-16T01:41:00.275 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:41:00.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.274+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171bb840 con 0x564a171bcfa0 2024-09-16T01:41:00.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.275+0000 7f74bb620640 1 -- 172.21.15.88:0/3954937878 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v92'36 uv36 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f74b0057030 con 0x564a171bcfa0 2024-09-16T01:41:00.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.275+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c0da0 con 0x564a17104750 2024-09-16T01:41:00.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.276+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v92'39 uv38 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.276+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e92) -- 0x564a171c0da0 con 0x564a17104750 2024-09-16T01:41:00.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.277+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v92'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.277+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c0da0 con 0x564a17104750 2024-09-16T01:41:00.279 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:41:00.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.278+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v92'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.278+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c0da0 con 0x564a17104750 2024-09-16T01:41:00.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.280+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v92'42 uv42 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.280+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c0da0 con 0x564a17104750 2024-09-16T01:41:00.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.281+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v92'43 uv43 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.281+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c0da0 con 0x564a17104750 2024-09-16T01:41:00.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.282+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v92'44 uv44 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.282+0000 7f74be329f40 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x564a171c16b0 0x564a171c3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.282+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c3fe0 con 0x564a171c16b0 2024-09-16T01:41:00.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.282+0000 7f74bc622640 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x564a171c16b0 0x564a171c3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:00.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.282+0000 7f74bc622640 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x564a171c16b0 0x564a171c3aa0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:00.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.284+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v92'70 uv64 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.284+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e92) -- 0x564a171c3fe0 con 0x564a171c16b0 2024-09-16T01:41:00.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.285+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v92'71 uv71 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.285+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c43d0 con 0x564a171c16b0 2024-09-16T01:41:00.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.287+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v92'72 uv72 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.289 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:41:00.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.287+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c53e0 con 0x564a171c16b0 2024-09-16T01:41:00.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.290+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v92'73 uv73 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.290+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c53e0 con 0x564a171c16b0 2024-09-16T01:41:00.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.293+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v92'74 uv74 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.293+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c43d0 con 0x564a171c16b0 2024-09-16T01:41:00.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.296+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v92'75 uv75 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.296+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c43d0 con 0x564a171c16b0 2024-09-16T01:41:00.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.299+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v92'76 uv69 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.299+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e92) -- 0x564a171c43d0 con 0x564a171c16b0 2024-09-16T01:41:00.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.300+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v92'77 uv77 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.300+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c43d0 con 0x564a171c16b0 2024-09-16T01:41:00.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.302+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v92'78 uv78 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.304 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:41:00.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.302+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c43d0 con 0x564a171c16b0 2024-09-16T01:41:00.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.305+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v92'79 uv79 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.305+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c43d0 con 0x564a171c16b0 2024-09-16T01:41:00.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.311+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v92'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.311+0000 7f74be329f40 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x564a171c4a50 0x564a171c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.311+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c7410 con 0x564a171c4a50 2024-09-16T01:41:00.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.311+0000 7f74bbe21640 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x564a171c4a50 0x564a171c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:00.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.311+0000 7f74bbe21640 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x564a171c4a50 0x564a171c6e40 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:00.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.312+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v92'32 uv31 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171c4a50 2024-09-16T01:41:00.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.312+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e92) -- 0x564a171c74d0 con 0x564a171c4a50 2024-09-16T01:41:00.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.313+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v92'33 uv33 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171c4a50 2024-09-16T01:41:00.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.313+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c78f0 con 0x564a171c4a50 2024-09-16T01:41:00.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.314+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v92'34 uv34 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171c4a50 2024-09-16T01:41:00.315 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:41:00.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.314+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c78f0 con 0x564a171c4a50 2024-09-16T01:41:00.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.315+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v92'35 uv35 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171c4a50 2024-09-16T01:41:00.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.315+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c78f0 con 0x564a171c4a50 2024-09-16T01:41:00.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.316+0000 7f74bbe21640 1 -- 172.21.15.88:0/3954937878 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v92'36 uv36 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f74ac01f070 con 0x564a171c4a50 2024-09-16T01:41:00.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.316+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c78f0 con 0x564a17104750 2024-09-16T01:41:00.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.317+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v92'39 uv38 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.317+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e92) -- 0x564a171c78f0 con 0x564a17104750 2024-09-16T01:41:00.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.319+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v92'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.319+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c78f0 con 0x564a17104750 2024-09-16T01:41:00.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.319+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v92'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.321 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:41:00.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.320+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c8900 con 0x564a17104750 2024-09-16T01:41:00.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.320+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v92'42 uv42 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.320+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c8900 con 0x564a17104750 2024-09-16T01:41:00.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.321+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v92'43 uv43 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.321+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c78f0 con 0x564a17104750 2024-09-16T01:41:00.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.323+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v92'44 uv44 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a17104750 2024-09-16T01:41:00.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.323+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c78f0 con 0x564a171bcfa0 2024-09-16T01:41:00.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.324+0000 7f74bb620640 1 -- 172.21.15.88:0/3954937878 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v92'64 uv58 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f74b0057030 con 0x564a171bcfa0 2024-09-16T01:41:00.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.324+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e92) -- 0x564a171c7c90 con 0x564a171bcfa0 2024-09-16T01:41:00.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.325+0000 7f74bb620640 1 -- 172.21.15.88:0/3954937878 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v92'65 uv65 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f74b0057030 con 0x564a171bcfa0 2024-09-16T01:41:00.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.325+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c8030 con 0x564a171bcfa0 2024-09-16T01:41:00.326 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:41:00.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.325+0000 7f74bb620640 1 -- 172.21.15.88:0/3954937878 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v92'66 uv66 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f74b0057030 con 0x564a171bcfa0 2024-09-16T01:41:00.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.325+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c0920 con 0x564a171bcfa0 2024-09-16T01:41:00.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.326+0000 7f74bb620640 1 -- 172.21.15.88:0/3954937878 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v92'67 uv67 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f74b0057030 con 0x564a171bcfa0 2024-09-16T01:41:00.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.326+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c0920 con 0x564a171bcfa0 2024-09-16T01:41:00.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.327+0000 7f74bb620640 1 -- 172.21.15.88:0/3954937878 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v92'68 uv68 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f74b0057030 con 0x564a171bcfa0 2024-09-16T01:41:00.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.327+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c0920 con 0x564a171bcfa0 2024-09-16T01:41:00.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.328+0000 7f74bb620640 1 -- 172.21.15.88:0/3954937878 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v92'69 uv63 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f74b0057030 con 0x564a171bcfa0 2024-09-16T01:41:00.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.328+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e92) -- 0x564a171c0920 con 0x564a171bcfa0 2024-09-16T01:41:00.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.329+0000 7f74bb620640 1 -- 172.21.15.88:0/3954937878 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v92'70 uv70 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b0057030 con 0x564a171bcfa0 2024-09-16T01:41:00.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.329+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c0920 con 0x564a171bcfa0 2024-09-16T01:41:00.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.330+0000 7f74bb620640 1 -- 172.21.15.88:0/3954937878 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v92'71 uv71 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b0057030 con 0x564a171bcfa0 2024-09-16T01:41:00.331 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:41:00.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.330+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171c0920 con 0x564a171bcfa0 2024-09-16T01:41:00.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.331+0000 7f74bb620640 1 -- 172.21.15.88:0/3954937878 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v92'72 uv72 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b0057030 con 0x564a171bcfa0 2024-09-16T01:41:00.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.331+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171cb270 con 0x564a171bcfa0 2024-09-16T01:41:00.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.332+0000 7f74bb620640 1 -- 172.21.15.88:0/3954937878 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v92'73 uv73 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b0057030 con 0x564a171bcfa0 2024-09-16T01:41:00.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.332+0000 7f74be329f40 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x564a171cb3e0 0x564a171cb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.332+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171cb3e0 2024-09-16T01:41:00.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.332+0000 7f74bb620640 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x564a171cb3e0 0x564a171cb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:00.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.332+0000 7f74bb620640 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x564a171cb3e0 0x564a171cb930 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:00.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.333+0000 7f74bb620640 1 -- 172.21.15.88:0/3954937878 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v92'31 uv29 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f74b0057030 con 0x564a171cb3e0 2024-09-16T01:41:00.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.333+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171cb3e0 2024-09-16T01:41:00.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.334+0000 7f74bb620640 1 -- 172.21.15.88:0/3954937878 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v92'32 uv32 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f74b0057030 con 0x564a171cb3e0 2024-09-16T01:41:00.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.334+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171cb3e0 2024-09-16T01:41:00.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.335+0000 7f74bb620640 1 -- 172.21.15.88:0/3954937878 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v92'33 uv33 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f74b0057030 con 0x564a171cb3e0 2024-09-16T01:41:00.336 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:41:00.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.335+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171cb3e0 2024-09-16T01:41:00.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.336+0000 7f74bb620640 1 -- 172.21.15.88:0/3954937878 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v92'34 uv34 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f74b0057030 con 0x564a171cb3e0 2024-09-16T01:41:00.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.336+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171bcfa0 2024-09-16T01:41:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.337+0000 7f74bb620640 1 -- 172.21.15.88:0/3954937878 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v92'39 uv38 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f74b0057030 con 0x564a171bcfa0 2024-09-16T01:41:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.337+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171bcfa0 2024-09-16T01:41:00.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.338+0000 7f74bb620640 1 -- 172.21.15.88:0/3954937878 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v92'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b0057030 con 0x564a171bcfa0 2024-09-16T01:41:00.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.338+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171bcfa0 2024-09-16T01:41:00.340 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:41:00.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.339+0000 7f74bb620640 1 -- 172.21.15.88:0/3954937878 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v92'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b0057030 con 0x564a171bcfa0 2024-09-16T01:41:00.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.339+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171bcfa0 2024-09-16T01:41:00.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.340+0000 7f74bb620640 1 -- 172.21.15.88:0/3954937878 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v92'42 uv42 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b0057030 con 0x564a171bcfa0 2024-09-16T01:41:00.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.340+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171bcfa0 2024-09-16T01:41:00.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.341+0000 7f74bb620640 1 -- 172.21.15.88:0/3954937878 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v92'43 uv43 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b0057030 con 0x564a171bcfa0 2024-09-16T01:41:00.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.341+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171bcfa0 2024-09-16T01:41:00.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.342+0000 7f74bb620640 1 -- 172.21.15.88:0/3954937878 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v92'44 uv44 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b0057030 con 0x564a171bcfa0 2024-09-16T01:41:00.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.342+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171c16b0 2024-09-16T01:41:00.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.343+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v92'32 uv31 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.343+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171c16b0 2024-09-16T01:41:00.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.345+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v92'33 uv33 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.345+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171c16b0 2024-09-16T01:41:00.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.346+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v92'34 uv34 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.347 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:41:00.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.346+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171c16b0 2024-09-16T01:41:00.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.347+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v92'35 uv35 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.347+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171c16b0 2024-09-16T01:41:00.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.348+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v92'36 uv36 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.348+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171c16b0 2024-09-16T01:41:00.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.349+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v92'82 uv73 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.349+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171c16b0 2024-09-16T01:41:00.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.350+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v92'83 uv83 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.350+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171c16b0 2024-09-16T01:41:00.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.351+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v92'84 uv84 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.352 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:41:00.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.351+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171c16b0 2024-09-16T01:41:00.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.352+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v92'85 uv85 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.352+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171c16b0 2024-09-16T01:41:00.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.353+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v92'86 uv86 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.353+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171c16b0 2024-09-16T01:41:00.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.354+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v92'87 uv80 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.354+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171c16b0 2024-09-16T01:41:00.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.355+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v92'88 uv88 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.355+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171c16b0 2024-09-16T01:41:00.357 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:41:00.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.356+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v92'89 uv89 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.356+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171c16b0 2024-09-16T01:41:00.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.357+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v92'90 uv90 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.357+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171c16b0 2024-09-16T01:41:00.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.358+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v92'91 uv91 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.358+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171cadf0 con 0x564a171c16b0 2024-09-16T01:41:00.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.359+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v92'92 uv92 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.359+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x564a171cbe70 con 0x564a171c16b0 2024-09-16T01:41:00.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v92'93 uv93 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f74b4009ae0 con 0x564a171c16b0 2024-09-16T01:41:00.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x564a17104750 msgr2=0x564a1706e0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:00.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74be329f40 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x564a17104750 0x564a1706e0f0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x564a171bcfa0 msgr2=0x564a171bf3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:00.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74be329f40 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x564a171bcfa0 0x564a171bf3f0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x564a171cb3e0 msgr2=0x564a171cb930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:00.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74be329f40 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x564a171cb3e0 0x564a171cb930 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x564a171b86f0 msgr2=0x564a171baae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:00.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74be329f40 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x564a171b86f0 0x564a171baae0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x564a171c16b0 msgr2=0x564a171c3aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:00.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74bc622640 1 -- 172.21.15.88:0/3954937878 reap_dead start 2024-09-16T01:41:00.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74be329f40 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x564a171c16b0 0x564a171c3aa0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x564a171c4a50 msgr2=0x564a171c6e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:00.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74be329f40 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x564a171c4a50 0x564a171c6e40 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f74a806ac90 msgr2=0x7f74a806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:00.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74be329f40 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f74a806ac90 0x7f74a806d150 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f74b0057000 tx=0x7f74b0051000 comp rx=0 tx=0).stop 2024-09-16T01:41:00.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564a17088700 msgr2=0x564a17101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:00.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74be329f40 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564a17088700 0x564a17101280 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00cf70 tx=0x7f74ac00a660 comp rx=0 tx=0).stop 2024-09-16T01:41:00.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 shutdown_connections 2024-09-16T01:41:00.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74be329f40 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f74a806ac90 0x7f74a806d150 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74be329f40 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x564a171c4a50 0x564a171c6e40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74be329f40 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x564a171c16b0 0x564a171c3aa0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74be329f40 1 --2- 172.21.15.88:0/3954937878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564a17088700 0x564a17101280 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 >> 172.21.15.88:0/3954937878 conn(0x564a17069680 msgr2=0x564a1706f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:00.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 shutdown_connections 2024-09-16T01:41:00.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.360+0000 7f74be329f40 1 -- 172.21.15.88:0/3954937878 wait complete. 2024-09-16T01:41:00.368 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 500.00000000 2024-09-16T01:41:00.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.451+0000 7f4c83107f40 1 Processor -- start 2024-09-16T01:41:00.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.451+0000 7f4c83107f40 1 -- start start 2024-09-16T01:41:00.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.451+0000 7f4c83107f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f29f580fe0 0x55f29f61f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.451+0000 7f4c83107f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f29f581a60 0x55f29f62c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.451+0000 7f4c83107f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f29f59b250 0x55f29f625e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.451+0000 7f4c83107f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f29f4a3510 con 0x55f29f59b250 2024-09-16T01:41:00.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.451+0000 7f4c83107f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f29f62a120 con 0x55f29f580fe0 2024-09-16T01:41:00.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.451+0000 7f4c83107f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55f29f599e60 con 0x55f29f581a60 2024-09-16T01:41:00.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.451+0000 7f4c813fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f29f59b250 0x55f29f625e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:00.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.451+0000 7f4c813fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f29f59b250 0x55f29f625e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55766/0 (socket says 172.21.15.88:55766) 2024-09-16T01:41:00.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.451+0000 7f4c813fe640 1 -- 172.21.15.88:0/2564050504 learned_addr learned my addr 172.21.15.88:0/2564050504 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:00.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.452+0000 7f4c7bfff640 1 --2- 172.21.15.88:0/2564050504 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f29f581a60 0x55f29f62c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:00.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.452+0000 7f4c80bfd640 1 --2- 172.21.15.88:0/2564050504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f29f580fe0 0x55f29f61f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:00.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.452+0000 7f4c813fe640 1 -- 172.21.15.88:0/2564050504 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f29f581a60 msgr2=0x55f29f62c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:00.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.452+0000 7f4c813fe640 1 --2- 172.21.15.88:0/2564050504 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f29f581a60 0x55f29f62c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.452+0000 7f4c813fe640 1 -- 172.21.15.88:0/2564050504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f29f580fe0 msgr2=0x55f29f61f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:00.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.452+0000 7f4c813fe640 1 --2- 172.21.15.88:0/2564050504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f29f580fe0 0x55f29f61f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.452+0000 7f4c813fe640 1 -- 172.21.15.88:0/2564050504 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f29f61bd20 con 0x55f29f59b250 2024-09-16T01:41:00.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.452+0000 7f4c7bfff640 1 --2- 172.21.15.88:0/2564050504 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f29f581a60 0x55f29f62c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:00.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.452+0000 7f4c813fe640 1 --2- 172.21.15.88:0/2564050504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f29f59b250 0x55f29f625e30 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c005fd0 tx=0x7f4c7c050000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22fe8db06016ac11 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:00.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.452+0000 7f4c7affd640 1 -- 172.21.15.88:0/2564050504 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c064070 con 0x55f29f59b250 2024-09-16T01:41:00.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.452+0000 7f4c7affd640 1 -- 172.21.15.88:0/2564050504 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c7c04e5d0 con 0x55f29f59b250 2024-09-16T01:41:00.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.452+0000 7f4c7affd640 1 -- 172.21.15.88:0/2564050504 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c04eb80 con 0x55f29f59b250 2024-09-16T01:41:00.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.453+0000 7f4c83107f40 1 -- 172.21.15.88:0/2564050504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f29f59b250 msgr2=0x55f29f625e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:00.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.453+0000 7f4c83107f40 1 --2- 172.21.15.88:0/2564050504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f29f59b250 0x55f29f625e30 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c005fd0 tx=0x7f4c7c050000 comp rx=0 tx=0).stop 2024-09-16T01:41:00.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.453+0000 7f4c83107f40 1 -- 172.21.15.88:0/2564050504 shutdown_connections 2024-09-16T01:41:00.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.453+0000 7f4c83107f40 1 --2- 172.21.15.88:0/2564050504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f29f59b250 0x55f29f625e30 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.453+0000 7f4c83107f40 1 --2- 172.21.15.88:0/2564050504 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f29f581a60 0x55f29f62c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.453+0000 7f4c83107f40 1 --2- 172.21.15.88:0/2564050504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f29f580fe0 0x55f29f61f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.453+0000 7f4c83107f40 1 -- 172.21.15.88:0/2564050504 >> 172.21.15.88:0/2564050504 conn(0x55f29f59c750 msgr2=0x55f29f624fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:00.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.453+0000 7f4c83107f40 1 -- 172.21.15.88:0/2564050504 shutdown_connections 2024-09-16T01:41:00.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.453+0000 7f4c83107f40 1 -- 172.21.15.88:0/2564050504 wait complete. 2024-09-16T01:41:00.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.453+0000 7f4c83107f40 1 Processor -- start 2024-09-16T01:41:00.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.453+0000 7f4c83107f40 1 -- start start 2024-09-16T01:41:00.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.454+0000 7f4c83107f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f29f580fe0 0x55f29f6224b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.454+0000 7f4c83107f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f29f581a60 0x55f29f626020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.454+0000 7f4c83107f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f29f59b250 0x55f29f61af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.454+0000 7f4c83107f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f29f629130 con 0x55f29f59b250 2024-09-16T01:41:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.454+0000 7f4c83107f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f29f628ee0 con 0x55f29f580fe0 2024-09-16T01:41:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.454+0000 7f4c83107f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55f29f59a9e0 con 0x55f29f581a60 2024-09-16T01:41:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.454+0000 7f4c813fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f29f59b250 0x55f29f61af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.454+0000 7f4c813fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f29f59b250 0x55f29f61af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55774/0 (socket says 172.21.15.88:55774) 2024-09-16T01:41:00.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.454+0000 7f4c813fe640 1 -- 172.21.15.88:0/1857954543 learned_addr learned my addr 172.21.15.88:0/1857954543 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:00.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.454+0000 7f4c80bfd640 1 --2- 172.21.15.88:0/1857954543 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f29f580fe0 0x55f29f6224b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:00.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.454+0000 7f4c7bfff640 1 --2- 172.21.15.88:0/1857954543 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f29f581a60 0x55f29f626020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:00.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.454+0000 7f4c813fe640 1 -- 172.21.15.88:0/1857954543 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f29f581a60 msgr2=0x55f29f626020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:00.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.454+0000 7f4c813fe640 1 --2- 172.21.15.88:0/1857954543 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f29f581a60 0x55f29f626020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.454+0000 7f4c813fe640 1 -- 172.21.15.88:0/1857954543 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f29f580fe0 msgr2=0x55f29f6224b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:00.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.454+0000 7f4c813fe640 1 --2- 172.21.15.88:0/1857954543 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f29f580fe0 0x55f29f6224b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.454+0000 7f4c813fe640 1 -- 172.21.15.88:0/1857954543 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f29f6207b0 con 0x55f29f59b250 2024-09-16T01:41:00.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.455+0000 7f4c813fe640 1 --2- 172.21.15.88:0/1857954543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f29f59b250 0x55f29f61af40 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c059040 tx=0x7f4c7c003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:00.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.455+0000 7f4c79ffb640 1 -- 172.21.15.88:0/1857954543 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c064040 con 0x55f29f59b250 2024-09-16T01:41:00.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.455+0000 7f4c79ffb640 1 -- 172.21.15.88:0/1857954543 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c7c06a670 con 0x55f29f59b250 2024-09-16T01:41:00.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.455+0000 7f4c83107f40 1 -- 172.21.15.88:0/1857954543 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f29f620a90 con 0x55f29f59b250 2024-09-16T01:41:00.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.455+0000 7f4c79ffb640 1 -- 172.21.15.88:0/1857954543 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c06aa20 con 0x55f29f59b250 2024-09-16T01:41:00.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.455+0000 7f4c83107f40 1 -- 172.21.15.88:0/1857954543 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f29f621050 con 0x55f29f59b250 2024-09-16T01:41:00.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.456+0000 7f4c79ffb640 1 -- 172.21.15.88:0/1857954543 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4c7c05e070 con 0x55f29f59b250 2024-09-16T01:41:00.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.456+0000 7f4c79ffb640 1 --2- 172.21.15.88:0/1857954543 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c6406ac90 0x7f4c6406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.457+0000 7f4c79ffb640 1 -- 172.21.15.88:0/1857954543 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6311+0+0 (secure 0 0 0) 0x7f4c7c05b020 con 0x55f29f59b250 2024-09-16T01:41:00.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.457+0000 7f4c80bfd640 1 --2- 172.21.15.88:0/1857954543 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c6406ac90 0x7f4c6406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:00.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.457+0000 7f4c83107f40 1 --2- 172.21.15.88:0/1857954543 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f29f626560 0x55f29f61f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.457+0000 7f4c83107f40 1 -- 172.21.15.88:0/1857954543 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:1 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e92) -- 0x55f29f59a580 con 0x55f29f626560 2024-09-16T01:41:00.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.457+0000 7f4c80bfd640 1 --2- 172.21.15.88:0/1857954543 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c6406ac90 0x7f4c6406d150 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f4c74005fd0 tx=0x7f4c7400c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:00.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.457+0000 7f4c7bfff640 1 --2- 172.21.15.88:0/1857954543 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f29f626560 0x55f29f61f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:00.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.457+0000 7f4c7bfff640 1 --2- 172.21.15.88:0/1857954543 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f29f626560 0x55f29f61f7c0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:00.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.458+0000 7f4c7bfff640 1 -- 172.21.15.88:0/1857954543 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(1 500.00000000 [delete] v92'81 uv81 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4c6c00e040 con 0x55f29f626560 2024-09-16T01:41:00.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.458+0000 7f4c83107f40 1 -- 172.21.15.88:0/1857954543 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f29f626560 msgr2=0x55f29f61f7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:00.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.458+0000 7f4c83107f40 1 --2- 172.21.15.88:0/1857954543 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f29f626560 0x55f29f61f7c0 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.459+0000 7f4c83107f40 1 -- 172.21.15.88:0/1857954543 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c6406ac90 msgr2=0x7f4c6406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:00.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.459+0000 7f4c83107f40 1 --2- 172.21.15.88:0/1857954543 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4c6406ac90 0x7f4c6406d150 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f4c74005fd0 tx=0x7f4c7400c040 comp rx=0 tx=0).stop 2024-09-16T01:41:00.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.459+0000 7f4c83107f40 1 -- 172.21.15.88:0/1857954543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f29f59b250 msgr2=0x55f29f61af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:00.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.459+0000 7f4c83107f40 1 --2- 172.21.15.88:0/1857954543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f29f59b250 0x55f29f61af40 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c059040 tx=0x7f4c7c003330 comp rx=0 tx=0).stop 2024-09-16T01:41:00.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.459+0000 7f4c813fe640 1 -- 172.21.15.88:0/1857954543 reap_dead start 2024-09-16T01:41:00.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.459+0000 7f4c83107f40 1 -- 172.21.15.88:0/1857954543 shutdown_connections 2024-09-16T01:41:00.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.459+0000 7f4c83107f40 1 -- 172.21.15.88:0/1857954543 >> 172.21.15.88:0/1857954543 conn(0x55f29f59c750 msgr2=0x55f29f624fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:00.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.459+0000 7f4c83107f40 1 -- 172.21.15.88:0/1857954543 shutdown_connections 2024-09-16T01:41:00.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.459+0000 7f4c83107f40 1 -- 172.21.15.88:0/1857954543 wait complete. 2024-09-16T01:41:00.469 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:41:00.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.569+0000 7fe2eb72a640 1 Processor -- start 2024-09-16T01:41:00.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.569+0000 7fe2eb72a640 1 -- start start 2024-09-16T01:41:00.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.570+0000 7fe2eb72a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe2e40b6290 0x7fe2e4071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.570+0000 7fe2eb72a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2e40ccff0 con 0x7fe2e4072530 2024-09-16T01:41:00.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.570+0000 7fe2eb72a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2e40cd2d0 con 0x7fe2e415b480 2024-09-16T01:41:00.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.570+0000 7fe2eb72a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe2e40cd5b0 con 0x7fe2e40b6290 2024-09-16T01:41:00.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.570+0000 7fe2e8cc5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe2e40b6290 0x7fe2e4071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:00.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.570+0000 7fe2e9cc7640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe2e4072530 0x7fe2e415bc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:47926/0 (socket says 172.21.15.88:47926) 2024-09-16T01:41:00.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.570+0000 7fe2e8cc5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe2e40b6290 0x7fe2e4071ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55776/0 (socket says 172.21.15.88:55776) 2024-09-16T01:41:00.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.570+0000 7fe2e9cc7640 1 -- 172.21.15.88:0/319266282 learned_addr learned my addr 172.21.15.88:0/319266282 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:00.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.571+0000 7fe2e8cc5640 1 -- 172.21.15.88:0/319266282 >> v1:172.21.15.154:6789/0 conn(0x7fe2e415b480 legacy=0x7fe2e40b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:00.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.571+0000 7fe2e8cc5640 1 -- 172.21.15.88:0/319266282 >> v1:172.21.15.88:6789/0 conn(0x7fe2e4072530 legacy=0x7fe2e415bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:00.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.571+0000 7fe2e3fff640 1 -- 172.21.15.88:0/319266282 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1013148738 0 0) 0x7fe2e40ccff0 con 0x7fe2e4072530 2024-09-16T01:41:00.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.571+0000 7fe2e8cc5640 1 -- 172.21.15.88:0/319266282 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2e40cd890 con 0x7fe2e40b6290 2024-09-16T01:41:00.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.571+0000 7fe2e8cc5640 1 --2- 172.21.15.88:0/319266282 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2e40b6290 0x7fe2e4071ff0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fe2d80097a0 tx=0x7fe2d802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69603e05b590726c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:00.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.571+0000 7fe2e3fff640 1 -- 172.21.15.88:0/319266282 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8037070 con 0x7fe2e40b6290 2024-09-16T01:41:00.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.571+0000 7fe2e3fff640 1 -- 172.21.15.88:0/319266282 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2d802ae50 con 0x7fe2e40b6290 2024-09-16T01:41:00.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.571+0000 7fe2e3fff640 1 -- 172.21.15.88:0/319266282 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8032660 con 0x7fe2e40b6290 2024-09-16T01:41:00.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.572+0000 7fe2eb72a640 1 -- 172.21.15.88:0/319266282 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2e40b6290 msgr2=0x7fe2e4071ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:00.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.572+0000 7fe2eb72a640 1 --2- 172.21.15.88:0/319266282 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2e40b6290 0x7fe2e4071ff0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fe2d80097a0 tx=0x7fe2d802a140 comp rx=0 tx=0).stop 2024-09-16T01:41:00.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.572+0000 7fe2eb72a640 1 -- 172.21.15.88:0/319266282 shutdown_connections 2024-09-16T01:41:00.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.572+0000 7fe2eb72a640 1 --2- 172.21.15.88:0/319266282 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2e40b6290 0x7fe2e4071ff0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.572+0000 7fe2eb72a640 1 -- 172.21.15.88:0/319266282 >> 172.21.15.88:0/319266282 conn(0x7fe2e406d0f0 msgr2=0x7fe2e406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:00.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.572+0000 7fe2eb72a640 1 -- 172.21.15.88:0/319266282 shutdown_connections 2024-09-16T01:41:00.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.572+0000 7fe2eb72a640 1 -- 172.21.15.88:0/319266282 wait complete. 2024-09-16T01:41:00.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.573+0000 7fe2eb72a640 1 Processor -- start 2024-09-16T01:41:00.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.573+0000 7fe2eb72a640 1 -- start start 2024-09-16T01:41:00.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.574+0000 7fe2eb72a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2e4072530 0x7fe2e41510d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.574+0000 7fe2eb72a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2e415b480 0x7fe2e414b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.574+0000 7fe2eb72a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2e40d0cf0 0x7fe2e415a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.574+0000 7fe2eb72a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e415b9e0 con 0x7fe2e4072530 2024-09-16T01:41:00.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.574+0000 7fe2eb72a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e415b860 con 0x7fe2e40d0cf0 2024-09-16T01:41:00.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.574+0000 7fe2eb72a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe2e415c370 con 0x7fe2e415b480 2024-09-16T01:41:00.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.574+0000 7fe2e94c6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2e4072530 0x7fe2e41510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:00.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.574+0000 7fe2e94c6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2e4072530 0x7fe2e41510d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55780/0 (socket says 172.21.15.88:55780) 2024-09-16T01:41:00.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.574+0000 7fe2e94c6640 1 -- 172.21.15.88:0/736561999 learned_addr learned my addr 172.21.15.88:0/736561999 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:00.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.574+0000 7fe2e9cc7640 1 --2- 172.21.15.88:0/736561999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2e40d0cf0 0x7fe2e415a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:00.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.574+0000 7fe2e94c6640 1 -- 172.21.15.88:0/736561999 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2e415b480 msgr2=0x7fe2e414b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:00.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.574+0000 7fe2e94c6640 1 --2- 172.21.15.88:0/736561999 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2e415b480 0x7fe2e414b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.574+0000 7fe2e94c6640 1 -- 172.21.15.88:0/736561999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2e40d0cf0 msgr2=0x7fe2e415a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:00.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.574+0000 7fe2e94c6640 1 --2- 172.21.15.88:0/736561999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2e40d0cf0 0x7fe2e415a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:00.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.574+0000 7fe2e94c6640 1 -- 172.21.15.88:0/736561999 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2e40c99b0 con 0x7fe2e4072530 2024-09-16T01:41:00.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.575+0000 7fe2e94c6640 1 --2- 172.21.15.88:0/736561999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2e4072530 0x7fe2e41510d0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fe2d400ba50 tx=0x7fe2d400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:00.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.575+0000 7fe2e1ffb640 1 -- 172.21.15.88:0/736561999 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d4019070 con 0x7fe2e4072530 2024-09-16T01:41:00.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.575+0000 7fe2e1ffb640 1 -- 172.21.15.88:0/736561999 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2d400cc20 con 0x7fe2e4072530 2024-09-16T01:41:00.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.575+0000 7fe2e1ffb640 1 -- 172.21.15.88:0/736561999 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d4014680 con 0x7fe2e4072530 2024-09-16T01:41:00.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.575+0000 7fe2eb72a640 1 -- 172.21.15.88:0/736561999 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2e40d02f0 con 0x7fe2e4072530 2024-09-16T01:41:00.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.575+0000 7fe2eb72a640 1 -- 172.21.15.88:0/736561999 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2e4156880 con 0x7fe2e4072530 2024-09-16T01:41:00.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.576+0000 7fe2eb72a640 1 -- 172.21.15.88:0/736561999 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2e4071ff0 con 0x7fe2e4072530 2024-09-16T01:41:00.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.576+0000 7fe2e1ffb640 1 -- 172.21.15.88:0/736561999 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe2d401c070 con 0x7fe2e4072530 2024-09-16T01:41:00.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.577+0000 7fe2e1ffb640 1 --2- 172.21.15.88:0/736561999 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe2d006c2b0 0x7fe2d006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:00.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.577+0000 7fe2e1ffb640 1 -- 172.21.15.88:0/736561999 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6311+0+0 (secure 0 0 0) 0x7fe2d408b2b0 con 0x7fe2e4072530 2024-09-16T01:41:00.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.577+0000 7fe2e8cc5640 1 --2- 172.21.15.88:0/736561999 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe2d006c2b0 0x7fe2d006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:00.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.577+0000 7fe2e8cc5640 1 --2- 172.21.15.88:0/736561999 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe2d006c2b0 0x7fe2d006e770 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fe2d8034000 tx=0x7fe2d802c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:00.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.581+0000 7fe2e1ffb640 1 -- 172.21.15.88:0/736561999 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe2d4059200 con 0x7fe2e4072530 2024-09-16T01:41:00.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:00.783+0000 7fe2eb72a640 1 -- 172.21.15.88:0/736561999 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fe2e415e540 con 0x7fe2e4072530 2024-09-16T01:41:01.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:01.487+0000 7fe2e1ffb640 1 -- 172.21.15.88:0/736561999 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v113) ==== 162+0+0 (secure 0 0 0) 0x7fe2d405d500 con 0x7fe2e4072530 2024-09-16T01:41:01.489 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:41:01.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:01.491+0000 7fe2eb72a640 1 -- 172.21.15.88:0/736561999 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe2d006c2b0 msgr2=0x7fe2d006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:01.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:01.491+0000 7fe2eb72a640 1 --2- 172.21.15.88:0/736561999 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe2d006c2b0 0x7fe2d006e770 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fe2d8034000 tx=0x7fe2d802c000 comp rx=0 tx=0).stop 2024-09-16T01:41:01.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:01.491+0000 7fe2eb72a640 1 -- 172.21.15.88:0/736561999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2e4072530 msgr2=0x7fe2e41510d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:01.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:01.491+0000 7fe2eb72a640 1 --2- 172.21.15.88:0/736561999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2e4072530 0x7fe2e41510d0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fe2d400ba50 tx=0x7fe2d400bf10 comp rx=0 tx=0).stop 2024-09-16T01:41:01.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:01.491+0000 7fe2eb72a640 1 -- 172.21.15.88:0/736561999 shutdown_connections 2024-09-16T01:41:01.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:01.491+0000 7fe2eb72a640 1 --2- 172.21.15.88:0/736561999 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe2d006c2b0 0x7fe2d006e770 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:01.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:01.491+0000 7fe2eb72a640 1 --2- 172.21.15.88:0/736561999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2e40d0cf0 0x7fe2e415a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:01.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:01.491+0000 7fe2eb72a640 1 --2- 172.21.15.88:0/736561999 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2e415b480 0x7fe2e414b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:01.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:01.491+0000 7fe2eb72a640 1 --2- 172.21.15.88:0/736561999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2e4072530 0x7fe2e41510d0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:01.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:01.491+0000 7fe2eb72a640 1 -- 172.21.15.88:0/736561999 >> 172.21.15.88:0/736561999 conn(0x7fe2e406d0f0 msgr2=0x7fe2e4157010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:01.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:01.491+0000 7fe2eb72a640 1 -- 172.21.15.88:0/736561999 shutdown_connections 2024-09-16T01:41:01.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:01.491+0000 7fe2eb72a640 1 -- 172.21.15.88:0/736561999 wait complete. 2024-09-16T01:41:01.908 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:02.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.017+0000 7f5cfb23c640 1 Processor -- start 2024-09-16T01:41:02.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.017+0000 7f5cfb23c640 1 -- start start 2024-09-16T01:41:02.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.018+0000 7f5cfb23c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5cf415e660 0x7f5cf40cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:02.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.018+0000 7f5cfb23c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cf40d3310 0x7f5cf40d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:02.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.018+0000 7f5cfb23c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cf415ea40 con 0x7f5cf40d1650 2024-09-16T01:41:02.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.018+0000 7f5cfb23c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5cf40d1ac0 con 0x7f5cf415e660 2024-09-16T01:41:02.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.018+0000 7f5cfb23c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5cf40d3870 con 0x7f5cf40d3310 2024-09-16T01:41:02.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.019+0000 7f5cf97d9640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5cf40d1650 0x7f5cf406bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:47940/0 (socket says 172.21.15.88:47940) 2024-09-16T01:41:02.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.019+0000 7f5cf97d9640 1 -- 172.21.15.88:0/4088825205 learned_addr learned my addr 172.21.15.88:0/4088825205 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:02.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.019+0000 7f5cebfff640 1 --2- 172.21.15.88:0/4088825205 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cf40d3310 0x7f5cf40d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:02.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.019+0000 7f5cf8fd8640 1 --2- 172.21.15.88:0/4088825205 >> v2:172.21.15.154:3300/0 conn(0x7f5cf415e660 0x7f5cf40cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:02.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.019+0000 7f5ceb7fe640 1 -- 172.21.15.88:0/4088825205 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 474680442 0 0) 0x7f5cf415ea40 con 0x7f5cf40d1650 2024-09-16T01:41:02.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.019+0000 7f5ceb7fe640 1 -- 172.21.15.88:0/4088825205 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cdc003520 con 0x7f5cf40d1650 2024-09-16T01:41:02.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.019+0000 7f5ceb7fe640 1 -- 172.21.15.88:0/4088825205 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3099432957 0 0) 0x7f5cdc003520 con 0x7f5cf40d1650 2024-09-16T01:41:02.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.019+0000 7f5ceb7fe640 1 -- 172.21.15.88:0/4088825205 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cf40d3310 msgr2=0x7f5cf40d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:02.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.019+0000 7f5ceb7fe640 1 --2- 172.21.15.88:0/4088825205 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cf40d3310 0x7f5cf40d1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:02.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.019+0000 7f5ceb7fe640 1 -- 172.21.15.88:0/4088825205 >> v2:172.21.15.154:3300/0 conn(0x7f5cf415e660 msgr2=0x7f5cf40cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:02.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.019+0000 7f5ceb7fe640 1 --2- 172.21.15.88:0/4088825205 >> v2:172.21.15.154:3300/0 conn(0x7f5cf415e660 0x7f5cf40cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:02.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.019+0000 7f5ceb7fe640 1 -- 172.21.15.88:0/4088825205 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf40d3b50 con 0x7f5cf40d1650 2024-09-16T01:41:02.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.019+0000 7f5ceb7fe640 1 -- 172.21.15.88:0/4088825205 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f5cec0019f0 con 0x7f5cf40d1650 2024-09-16T01:41:02.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.019+0000 7f5ceb7fe640 1 -- 172.21.15.88:0/4088825205 >> v1:172.21.15.88:6789/0 conn(0x7f5cf40d1650 legacy=0x7f5cf406bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:02.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.019+0000 7f5ceb7fe640 1 --2- 172.21.15.88:0/4088825205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc0039b0 0x7f5cdc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:02.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.019+0000 7f5ceb7fe640 1 --2- 172.21.15.88:0/4088825205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5cdc004800 0x7f5cdc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:02.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.019+0000 7f5ceb7fe640 1 --2- 172.21.15.88:0/4088825205 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cdc0051c0 0x7f5cdc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:02.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.019+0000 7f5ceb7fe640 1 -- 172.21.15.88:0/4088825205 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf40d1ac0 con 0x7f5cdc0039b0 2024-09-16T01:41:02.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.019+0000 7f5ceb7fe640 1 -- 172.21.15.88:0/4088825205 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf40d3870 con 0x7f5cdc004800 2024-09-16T01:41:02.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.019+0000 7f5ceb7fe640 1 -- 172.21.15.88:0/4088825205 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5cdc00bf00 con 0x7f5cdc0051c0 2024-09-16T01:41:02.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.019+0000 7f5cebfff640 1 --2- 172.21.15.88:0/4088825205 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cf40d3310 0x7f5cf40d1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:41:02.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.019+0000 7f5cf8fd8640 1 --2- 172.21.15.88:0/4088825205 >> v2:172.21.15.154:3300/0 conn(0x7f5cf415e660 0x7f5cf40cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:41:02.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.019+0000 7f5cf8fd8640 1 --2- 172.21.15.88:0/4088825205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc0039b0 0x7f5cdc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:02.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.019+0000 7f5cf97d9640 1 --2- 172.21.15.88:0/4088825205 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cdc0051c0 0x7f5cdc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:02.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.019+0000 7f5cebfff640 1 --2- 172.21.15.88:0/4088825205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5cdc004800 0x7f5cdc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:02.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.020+0000 7f5cf8fd8640 1 -- 172.21.15.88:0/4088825205 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cdc0051c0 msgr2=0x7f5cdc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:02.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.020+0000 7f5cf8fd8640 1 --2- 172.21.15.88:0/4088825205 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cdc0051c0 0x7f5cdc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:02.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.020+0000 7f5cf8fd8640 1 -- 172.21.15.88:0/4088825205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5cdc004800 msgr2=0x7f5cdc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:02.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.020+0000 7f5cf8fd8640 1 --2- 172.21.15.88:0/4088825205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5cdc004800 0x7f5cdc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:02.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.020+0000 7f5cf8fd8640 1 -- 172.21.15.88:0/4088825205 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cdc003520 con 0x7f5cdc0039b0 2024-09-16T01:41:02.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.020+0000 7f5cf8fd8640 1 --2- 172.21.15.88:0/4088825205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc0039b0 0x7f5cdc003dd0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0009ca0 tx=0x7f5cdc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43ab82715354d3ac server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:02.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.021+0000 7f5ceb7fe640 1 -- 172.21.15.88:0/4088825205 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0009840 con 0x7f5cdc0039b0 2024-09-16T01:41:02.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.021+0000 7f5ceb7fe640 1 -- 172.21.15.88:0/4088825205 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cf0004020 con 0x7f5cdc0039b0 2024-09-16T01:41:02.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.021+0000 7f5ceb7fe640 1 -- 172.21.15.88:0/4088825205 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf00043d0 con 0x7f5cdc0039b0 2024-09-16T01:41:02.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.021+0000 7f5cfb23c640 1 -- 172.21.15.88:0/4088825205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc0039b0 msgr2=0x7f5cdc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:02.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.021+0000 7f5cfb23c640 1 --2- 172.21.15.88:0/4088825205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc0039b0 0x7f5cdc003dd0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0009ca0 tx=0x7f5cdc005830 comp rx=0 tx=0).stop 2024-09-16T01:41:02.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.021+0000 7f5cf97d9640 1 -- 172.21.15.88:0/4088825205 reap_dead start 2024-09-16T01:41:02.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.021+0000 7f5cfb23c640 1 -- 172.21.15.88:0/4088825205 shutdown_connections 2024-09-16T01:41:02.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.021+0000 7f5cfb23c640 1 -- 172.21.15.88:0/4088825205 >> 172.21.15.88:0/4088825205 conn(0x7f5cf40b6410 msgr2=0x7f5cf40b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:02.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.021+0000 7f5cfb23c640 1 -- 172.21.15.88:0/4088825205 shutdown_connections 2024-09-16T01:41:02.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.021+0000 7f5cfb23c640 1 -- 172.21.15.88:0/4088825205 wait complete. 2024-09-16T01:41:02.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.022+0000 7f5cfb23c640 1 Processor -- start 2024-09-16T01:41:02.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.022+0000 7f5cfb23c640 1 -- start start 2024-09-16T01:41:02.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.023+0000 7f5cfb23c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc0039b0 0x7f5cf4155bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:02.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.023+0000 7f5cfb23c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5cf4156130 0x7f5cf41f3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:02.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.023+0000 7f5cfb23c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cf41f4510 0x7f5cf41f49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:02.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.023+0000 7f5cfb23c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf40d43f0 con 0x7f5cdc0039b0 2024-09-16T01:41:02.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.023+0000 7f5cfb23c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf40d4110 con 0x7f5cf4156130 2024-09-16T01:41:02.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.023+0000 7f5cfb23c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5cf40cfc70 con 0x7f5cf41f4510 2024-09-16T01:41:02.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.023+0000 7f5cf8fd8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc0039b0 0x7f5cf4155bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:02.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.023+0000 7f5cf8fd8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc0039b0 0x7f5cf4155bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55810/0 (socket says 172.21.15.88:55810) 2024-09-16T01:41:02.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.023+0000 7f5cf8fd8640 1 -- 172.21.15.88:0/239528705 learned_addr learned my addr 172.21.15.88:0/239528705 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:02.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.024+0000 7f5cebfff640 1 --2- 172.21.15.88:0/239528705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5cf4156130 0x7f5cf41f3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:02.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.024+0000 7f5cf97d9640 1 --2- 172.21.15.88:0/239528705 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cf41f4510 0x7f5cf41f49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:02.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.024+0000 7f5cf8fd8640 1 -- 172.21.15.88:0/239528705 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cf41f4510 msgr2=0x7f5cf41f49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:02.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.024+0000 7f5cf8fd8640 1 --2- 172.21.15.88:0/239528705 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cf41f4510 0x7f5cf41f49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:02.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.024+0000 7f5cf8fd8640 1 -- 172.21.15.88:0/239528705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5cf4156130 msgr2=0x7f5cf41f3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:02.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.024+0000 7f5cf8fd8640 1 --2- 172.21.15.88:0/239528705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5cf4156130 0x7f5cf41f3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:02.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.024+0000 7f5cf8fd8640 1 -- 172.21.15.88:0/239528705 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf4158550 con 0x7f5cdc0039b0 2024-09-16T01:41:02.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.024+0000 7f5cf97d9640 1 --2- 172.21.15.88:0/239528705 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cf41f4510 0x7f5cf41f49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:02.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.024+0000 7f5cf8fd8640 1 --2- 172.21.15.88:0/239528705 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc0039b0 0x7f5cf4155bf0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0009dc0 tx=0x7f5cf0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:02.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.024+0000 7f5ce97fa640 1 -- 172.21.15.88:0/239528705 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0040070 con 0x7f5cdc0039b0 2024-09-16T01:41:02.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.024+0000 7f5ce97fa640 1 -- 172.21.15.88:0/239528705 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cf0003820 con 0x7f5cdc0039b0 2024-09-16T01:41:02.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.024+0000 7f5cfb23c640 1 -- 172.21.15.88:0/239528705 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cf4158830 con 0x7f5cdc0039b0 2024-09-16T01:41:02.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.024+0000 7f5ce97fa640 1 -- 172.21.15.88:0/239528705 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0006e10 con 0x7f5cdc0039b0 2024-09-16T01:41:02.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.024+0000 7f5cfb23c640 1 -- 172.21.15.88:0/239528705 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cf4158df0 con 0x7f5cdc0039b0 2024-09-16T01:41:02.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.025+0000 7f5cfb23c640 1 -- 172.21.15.88:0/239528705 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cf40cf730 con 0x7f5cdc0039b0 2024-09-16T01:41:02.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.026+0000 7f5ce97fa640 1 -- 172.21.15.88:0/239528705 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5cf0043070 con 0x7f5cdc0039b0 2024-09-16T01:41:02.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.026+0000 7f5ce97fa640 1 --2- 172.21.15.88:0/239528705 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5ce006c2b0 0x7f5ce006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:02.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.026+0000 7f5ce97fa640 1 -- 172.21.15.88:0/239528705 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 6311+0+0 (secure 0 0 0) 0x7f5cf00abc70 con 0x7f5cdc0039b0 2024-09-16T01:41:02.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.026+0000 7f5cebfff640 1 --2- 172.21.15.88:0/239528705 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5ce006c2b0 0x7f5ce006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:02.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.026+0000 7f5cebfff640 1 --2- 172.21.15.88:0/239528705 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5ce006c2b0 0x7f5ce006e770 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5cf40cf430 tx=0x7f5ce4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:02.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.030+0000 7f5ce97fa640 1 -- 172.21.15.88:0/239528705 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5cf0079bc0 con 0x7f5cdc0039b0 2024-09-16T01:41:02.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.230+0000 7f5cfb23c640 1 -- 172.21.15.88:0/239528705 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5cf4159540 con 0x7f5cdc0039b0 2024-09-16T01:41:02.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.231+0000 7f5ce97fa640 1 -- 172.21.15.88:0/239528705 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) ==== 95+0+5220 (secure 0 0 0) 0x7f5cf007dec0 con 0x7f5cdc0039b0 2024-09-16T01:41:02.232 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:02.233 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":114,"btime":"2024-09-16T01:41:01:491330+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":7866,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/293764774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":293764774},{"type":"v1","addr":"172.21.15.88:6837","nonce":293764774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:01.491313+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":114,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6836/128058689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":128058689},{"type":"v1","addr":"172.21.15.88:6836","nonce":128058689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:41:02.233 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 114 2024-09-16T01:41:02.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.233+0000 7f5cfb23c640 1 -- 172.21.15.88:0/239528705 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5ce006c2b0 msgr2=0x7f5ce006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:02.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.233+0000 7f5cfb23c640 1 --2- 172.21.15.88:0/239528705 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5ce006c2b0 0x7f5ce006e770 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5cf40cf430 tx=0x7f5ce4009190 comp rx=0 tx=0).stop 2024-09-16T01:41:02.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.233+0000 7f5cfb23c640 1 -- 172.21.15.88:0/239528705 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc0039b0 msgr2=0x7f5cf4155bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:02.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.233+0000 7f5cfb23c640 1 --2- 172.21.15.88:0/239528705 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc0039b0 0x7f5cf4155bf0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0009dc0 tx=0x7f5cf0004730 comp rx=0 tx=0).stop 2024-09-16T01:41:02.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.233+0000 7f5cfb23c640 1 -- 172.21.15.88:0/239528705 shutdown_connections 2024-09-16T01:41:02.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.233+0000 7f5cfb23c640 1 --2- 172.21.15.88:0/239528705 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5ce006c2b0 0x7f5ce006e770 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:02.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.233+0000 7f5cfb23c640 1 --2- 172.21.15.88:0/239528705 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5cf41f4510 0x7f5cf41f49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:02.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.233+0000 7f5cfb23c640 1 --2- 172.21.15.88:0/239528705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5cf4156130 0x7f5cf41f3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:02.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.233+0000 7f5cfb23c640 1 --2- 172.21.15.88:0/239528705 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc0039b0 0x7f5cf4155bf0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:02.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.233+0000 7f5cfb23c640 1 -- 172.21.15.88:0/239528705 >> 172.21.15.88:0/239528705 conn(0x7f5cf40b6410 msgr2=0x7f5cf415d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:02.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.233+0000 7f5cfb23c640 1 -- 172.21.15.88:0/239528705 shutdown_connections 2024-09-16T01:41:02.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:02.233+0000 7f5cfb23c640 1 -- 172.21.15.88:0/239528705 wait complete. 2024-09-16T01:41:02.245 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 114, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:40:03.597796+0000', 'modified': '2024-09-16T01:41:01.491313+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 92, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7920}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7920': {'gid': 7920, 'name': 'a', 'rank': 0, 'incarnation': 114, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.88:6836/128058689', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 128058689}, {'type': 'v1', 'addr': '172.21.15.88:6836', 'nonce': 128058689}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-16T01:41:02.245 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7920:up:replay 2024-09-16T01:41:02.245 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-09-16T01:41:05.090 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:41:05.089+0000 7f7610823640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:41:07.649 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:07.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.758+0000 7fa8eb03c640 1 Processor -- start 2024-09-16T01:41:07.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.758+0000 7fa8eb03c640 1 -- start start 2024-09-16T01:41:07.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.759+0000 7fa8eb03c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa8e40b5be0 0x7fa8e415a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:07.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.759+0000 7fa8eb03c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa8e415e600 0x7fa8e415eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:07.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.759+0000 7fa8eb03c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8e40b6830 con 0x7fa8e40b29a0 2024-09-16T01:41:07.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.759+0000 7fa8eb03c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa8e406e550 con 0x7fa8e40b5be0 2024-09-16T01:41:07.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.759+0000 7fa8eb03c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa8e415f410 con 0x7fa8e415e600 2024-09-16T01:41:07.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.759+0000 7fa8e8dd8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa8e40b5be0 0x7fa8e415a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:07.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.759+0000 7fa8e8dd8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa8e40b5be0 0x7fa8e415a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55816/0 (socket says 172.21.15.88:55816) 2024-09-16T01:41:07.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.759+0000 7fa8e8dd8640 1 -- 172.21.15.88:0/2092925290 learned_addr learned my addr 172.21.15.88:0/2092925290 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:07.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.760+0000 7fa8e8dd8640 1 -- 172.21.15.88:0/2092925290 >> v2:172.21.15.154:3300/0 conn(0x7fa8e415e600 msgr2=0x7fa8e415eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:07.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.760+0000 7fa8e8dd8640 1 --2- 172.21.15.88:0/2092925290 >> v2:172.21.15.154:3300/0 conn(0x7fa8e415e600 0x7fa8e415eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:07.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.760+0000 7fa8e8dd8640 1 -- 172.21.15.88:0/2092925290 >> v1:172.21.15.154:6789/0 conn(0x7fa8e40b29a0 legacy=0x7fa8e406de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:07.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.760+0000 7fa8e8dd8640 1 -- 172.21.15.88:0/2092925290 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8e40cd0b0 con 0x7fa8e40b5be0 2024-09-16T01:41:07.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.760+0000 7fa8e8dd8640 1 --2- 172.21.15.88:0/2092925290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8e40b5be0 0x7fa8e415a9c0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fa8e406f3b0 tx=0x7fa8e002c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c201dfa309fe659c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:07.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.761+0000 7fa8db7fe640 1 -- 172.21.15.88:0/2092925290 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0032710 con 0x7fa8e40b5be0 2024-09-16T01:41:07.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.761+0000 7fa8db7fe640 1 -- 172.21.15.88:0/2092925290 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8e00328b0 con 0x7fa8e40b5be0 2024-09-16T01:41:07.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.761+0000 7fa8db7fe640 1 -- 172.21.15.88:0/2092925290 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0032ca0 con 0x7fa8e40b5be0 2024-09-16T01:41:07.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.761+0000 7fa8eb03c640 1 -- 172.21.15.88:0/2092925290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8e40b5be0 msgr2=0x7fa8e415a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:07.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.761+0000 7fa8eb03c640 1 --2- 172.21.15.88:0/2092925290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8e40b5be0 0x7fa8e415a9c0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fa8e406f3b0 tx=0x7fa8e002c850 comp rx=0 tx=0).stop 2024-09-16T01:41:07.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.761+0000 7fa8eb03c640 1 -- 172.21.15.88:0/2092925290 shutdown_connections 2024-09-16T01:41:07.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.761+0000 7fa8eb03c640 1 --2- 172.21.15.88:0/2092925290 >> v2:172.21.15.154:3300/0 conn(0x7fa8e415e600 0x7fa8e415eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:07.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.761+0000 7fa8eb03c640 1 --2- 172.21.15.88:0/2092925290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8e40b5be0 0x7fa8e415a9c0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:07.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.761+0000 7fa8eb03c640 1 -- 172.21.15.88:0/2092925290 >> 172.21.15.88:0/2092925290 conn(0x7fa8e406bc10 msgr2=0x7fa8e406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:07.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.761+0000 7fa8eb03c640 1 -- 172.21.15.88:0/2092925290 shutdown_connections 2024-09-16T01:41:07.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.761+0000 7fa8eb03c640 1 -- 172.21.15.88:0/2092925290 wait complete. 2024-09-16T01:41:07.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.762+0000 7fa8eb03c640 1 Processor -- start 2024-09-16T01:41:07.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.763+0000 7fa8eb03c640 1 -- start start 2024-09-16T01:41:07.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.763+0000 7fa8eb03c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8e40b29a0 0x7fa8e415c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:07.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.763+0000 7fa8eb03c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8e40b5be0 0x7fa8e4141ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:07.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.763+0000 7fa8eb03c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8e415e600 0x7fa8e415a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:07.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.763+0000 7fa8eb03c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e40cd950 con 0x7fa8e40b29a0 2024-09-16T01:41:07.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.763+0000 7fa8eb03c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e40cd670 con 0x7fa8e415e600 2024-09-16T01:41:07.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.763+0000 7fa8eb03c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa8e415eff0 con 0x7fa8e40b5be0 2024-09-16T01:41:07.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.764+0000 7fa8e8dd8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8e40b29a0 0x7fa8e415c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:07.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.764+0000 7fa8e8dd8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8e40b29a0 0x7fa8e415c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55818/0 (socket says 172.21.15.88:55818) 2024-09-16T01:41:07.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.764+0000 7fa8e8dd8640 1 -- 172.21.15.88:0/3273903529 learned_addr learned my addr 172.21.15.88:0/3273903529 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:07.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.764+0000 7fa8e95d9640 1 --2- 172.21.15.88:0/3273903529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8e415e600 0x7fa8e415a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:07.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.764+0000 7fa8dbfff640 1 --2- 172.21.15.88:0/3273903529 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8e40b5be0 0x7fa8e4141ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:07.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.764+0000 7fa8e8dd8640 1 -- 172.21.15.88:0/3273903529 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8e40b5be0 msgr2=0x7fa8e4141ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:07.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.764+0000 7fa8e8dd8640 1 --2- 172.21.15.88:0/3273903529 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8e40b5be0 0x7fa8e4141ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:07.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.764+0000 7fa8e8dd8640 1 -- 172.21.15.88:0/3273903529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8e415e600 msgr2=0x7fa8e415a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:07.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.764+0000 7fa8e8dd8640 1 --2- 172.21.15.88:0/3273903529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8e415e600 0x7fa8e415a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:07.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.764+0000 7fa8e8dd8640 1 -- 172.21.15.88:0/3273903529 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8e415b2d0 con 0x7fa8e40b29a0 2024-09-16T01:41:07.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.764+0000 7fa8e8dd8640 1 --2- 172.21.15.88:0/3273903529 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8e40b29a0 0x7fa8e415c370 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fa8e002c890 tx=0x7fa8e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:07.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.764+0000 7fa8d97fa640 1 -- 172.21.15.88:0/3273903529 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0044070 con 0x7fa8e40b29a0 2024-09-16T01:41:07.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.764+0000 7fa8d97fa640 1 -- 172.21.15.88:0/3273903529 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8e0003810 con 0x7fa8e40b29a0 2024-09-16T01:41:07.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.764+0000 7fa8eb03c640 1 -- 172.21.15.88:0/3273903529 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8e415b5b0 con 0x7fa8e40b29a0 2024-09-16T01:41:07.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.764+0000 7fa8d97fa640 1 -- 172.21.15.88:0/3273903529 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0003c00 con 0x7fa8e40b29a0 2024-09-16T01:41:07.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.764+0000 7fa8eb03c640 1 -- 172.21.15.88:0/3273903529 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8e415bb70 con 0x7fa8e40b29a0 2024-09-16T01:41:07.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.765+0000 7fa8eb03c640 1 -- 172.21.15.88:0/3273903529 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8e415eab0 con 0x7fa8e40b29a0 2024-09-16T01:41:07.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.766+0000 7fa8d97fa640 1 -- 172.21.15.88:0/3273903529 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa8e0047070 con 0x7fa8e40b29a0 2024-09-16T01:41:07.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.766+0000 7fa8d97fa640 1 --2- 172.21.15.88:0/3273903529 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa8d006c2b0 0x7fa8d006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:07.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.767+0000 7fa8d97fa640 1 -- 172.21.15.88:0/3273903529 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6340+0+0 (secure 0 0 0) 0x7fa8e00aea30 con 0x7fa8e40b29a0 2024-09-16T01:41:07.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.767+0000 7fa8dbfff640 1 --2- 172.21.15.88:0/3273903529 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa8d006c2b0 0x7fa8d006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:07.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.767+0000 7fa8dbfff640 1 --2- 172.21.15.88:0/3273903529 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa8d006c2b0 0x7fa8d006e770 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa8e41526e0 tx=0x7fa8d4007e30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:07.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.771+0000 7fa8d97fa640 1 -- 172.21.15.88:0/3273903529 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa8e007ca80 con 0x7fa8e40b29a0 2024-09-16T01:41:07.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.971+0000 7fa8eb03c640 1 -- 172.21.15.88:0/3273903529 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8e4157550 con 0x7fa8e40b29a0 2024-09-16T01:41:07.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.971+0000 7fa8d97fa640 1 -- 172.21.15.88:0/3273903529 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+5227 (secure 0 0 0) 0x7fa8e0080d80 con 0x7fa8e40b29a0 2024-09-16T01:41:07.973 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:07.973 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":117,"btime":"2024-09-16T01:41:04:503938+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":7866,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/293764774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":293764774},{"type":"v1","addr":"172.21.15.88:6837","nonce":293764774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:04.503933+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7920},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7920":{"gid":7920,"name":"a","rank":0,"incarnation":114,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6836/128058689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":128058689},{"type":"v1","addr":"172.21.15.88:6836","nonce":128058689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7920,"qdb_cluster":[7920]},"id":6}]} 2024-09-16T01:41:07.973 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 117 2024-09-16T01:41:07.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.973+0000 7fa8eb03c640 1 -- 172.21.15.88:0/3273903529 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa8d006c2b0 msgr2=0x7fa8d006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:07.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.973+0000 7fa8eb03c640 1 --2- 172.21.15.88:0/3273903529 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa8d006c2b0 0x7fa8d006e770 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa8e41526e0 tx=0x7fa8d4007e30 comp rx=0 tx=0).stop 2024-09-16T01:41:07.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.973+0000 7fa8eb03c640 1 -- 172.21.15.88:0/3273903529 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8e40b29a0 msgr2=0x7fa8e415c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:07.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.973+0000 7fa8eb03c640 1 --2- 172.21.15.88:0/3273903529 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8e40b29a0 0x7fa8e415c370 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fa8e002c890 tx=0x7fa8e00046c0 comp rx=0 tx=0).stop 2024-09-16T01:41:07.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.973+0000 7fa8eb03c640 1 -- 172.21.15.88:0/3273903529 shutdown_connections 2024-09-16T01:41:07.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.973+0000 7fa8eb03c640 1 --2- 172.21.15.88:0/3273903529 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa8d006c2b0 0x7fa8d006e770 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:07.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.973+0000 7fa8eb03c640 1 --2- 172.21.15.88:0/3273903529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8e415e600 0x7fa8e415a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:07.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.973+0000 7fa8eb03c640 1 --2- 172.21.15.88:0/3273903529 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8e40b5be0 0x7fa8e4141ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:07.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.973+0000 7fa8eb03c640 1 --2- 172.21.15.88:0/3273903529 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8e40b29a0 0x7fa8e415c370 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:07.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.973+0000 7fa8eb03c640 1 -- 172.21.15.88:0/3273903529 >> 172.21.15.88:0/3273903529 conn(0x7fa8e406bc10 msgr2=0x7fa8e4156930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:07.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.973+0000 7fa8eb03c640 1 -- 172.21.15.88:0/3273903529 shutdown_connections 2024-09-16T01:41:07.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:07.973+0000 7fa8eb03c640 1 -- 172.21.15.88:0/3273903529 wait complete. 2024-09-16T01:41:07.986 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:40:03.597796+0000', 'modified': '2024-09-16T01:41:04.503933+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 92, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7920}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7920': {'gid': 7920, 'name': 'a', 'rank': 0, 'incarnation': 114, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.88:6836/128058689', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 128058689}, {'type': 'v1', 'addr': '172.21.15.88:6836', 'nonce': 128058689}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7920, 'qdb_cluster': [7920]} 2024-09-16T01:41:07.986 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T01:41:07.986 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-16T01:41:08.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.084+0000 7f7793b4b640 1 Processor -- start 2024-09-16T01:41:08.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.085+0000 7f7793b4b640 1 -- start start 2024-09-16T01:41:08.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.085+0000 7f7793b4b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f778c0d2820 0x7f778c0ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.085+0000 7f7793b4b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f778c0cdf50 con 0x7f778c15c710 2024-09-16T01:41:08.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.085+0000 7f7793b4b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f778c0ce230 con 0x7f778c0b53f0 2024-09-16T01:41:08.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.085+0000 7f7793b4b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f778c0d2c30 con 0x7f778c0d2820 2024-09-16T01:41:08.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.085+0000 7f77918e7640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f778c15c710 0x7f778c073970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:47952/0 (socket says 172.21.15.88:47952) 2024-09-16T01:41:08.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.085+0000 7f77918e7640 1 -- 172.21.15.88:0/2300531471 learned_addr learned my addr 172.21.15.88:0/2300531471 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:08.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.086+0000 7f77920e8640 1 --2- 172.21.15.88:0/2300531471 >> v2:172.21.15.154:3300/0 conn(0x7f778c0d2820 0x7f778c0ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.086+0000 7f77908e5640 1 -- 172.21.15.88:0/2300531471 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2402388708 0 0) 0x7f778c0cdf50 con 0x7f778c15c710 2024-09-16T01:41:08.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.086+0000 7f77908e5640 1 -- 172.21.15.88:0/2300531471 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7778003520 con 0x7f778c15c710 2024-09-16T01:41:08.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.086+0000 7f77908e5640 1 -- 172.21.15.88:0/2300531471 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 24497852 0 0) 0x7f7778003520 con 0x7f778c15c710 2024-09-16T01:41:08.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.086+0000 7f77908e5640 1 -- 172.21.15.88:0/2300531471 >> v2:172.21.15.154:3300/0 conn(0x7f778c0d2820 msgr2=0x7f778c0ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:08.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.086+0000 7f77908e5640 1 --2- 172.21.15.88:0/2300531471 >> v2:172.21.15.154:3300/0 conn(0x7f778c0d2820 0x7f778c0ceff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.086+0000 7f77908e5640 1 -- 172.21.15.88:0/2300531471 >> v1:172.21.15.154:6789/0 conn(0x7f778c0b53f0 legacy=0x7f778c0d2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:08.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.086+0000 7f77908e5640 1 -- 172.21.15.88:0/2300531471 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f778c0ce510 con 0x7f778c15c710 2024-09-16T01:41:08.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.086+0000 7f77908e5640 1 -- 172.21.15.88:0/2300531471 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f77880019f0 con 0x7f778c15c710 2024-09-16T01:41:08.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.086+0000 7f77908e5640 1 -- 172.21.15.88:0/2300531471 >> v1:172.21.15.88:6789/0 conn(0x7f778c15c710 legacy=0x7f778c073970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:08.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.086+0000 7f77908e5640 1 --2- 172.21.15.88:0/2300531471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77780039b0 0x7f7778003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.086+0000 7f77908e5640 1 --2- 172.21.15.88:0/2300531471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7778004800 0x7f7778004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.086+0000 7f77908e5640 1 --2- 172.21.15.88:0/2300531471 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f77780051c0 0x7f777800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.086+0000 7f77908e5640 1 -- 172.21.15.88:0/2300531471 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f778c0d2c30 con 0x7f7778004800 2024-09-16T01:41:08.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.086+0000 7f77908e5640 1 -- 172.21.15.88:0/2300531471 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f777800c160 con 0x7f77780039b0 2024-09-16T01:41:08.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.086+0000 7f77908e5640 1 -- 172.21.15.88:0/2300531471 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f777800c360 con 0x7f77780051c0 2024-09-16T01:41:08.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.086+0000 7f77910e6640 1 --2- 172.21.15.88:0/2300531471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7778004800 0x7f7778004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.086+0000 7f77918e7640 1 --2- 172.21.15.88:0/2300531471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77780039b0 0x7f7778003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.086+0000 7f77920e8640 1 --2- 172.21.15.88:0/2300531471 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f77780051c0 0x7f777800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.086+0000 7f77910e6640 1 -- 172.21.15.88:0/2300531471 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f77780051c0 msgr2=0x7f777800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:08.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.086+0000 7f77910e6640 1 --2- 172.21.15.88:0/2300531471 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f77780051c0 0x7f777800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.086+0000 7f77910e6640 1 -- 172.21.15.88:0/2300531471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77780039b0 msgr2=0x7f7778003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:08.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.086+0000 7f77910e6640 1 --2- 172.21.15.88:0/2300531471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77780039b0 0x7f7778003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.086+0000 7f77910e6640 1 -- 172.21.15.88:0/2300531471 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7778003520 con 0x7f7778004800 2024-09-16T01:41:08.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.086+0000 7f77920e8640 1 -- 172.21.15.88:0/2300531471 reap_dead start 2024-09-16T01:41:08.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.087+0000 7f77910e6640 1 --2- 172.21.15.88:0/2300531471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7778004800 0x7f7778004c80 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f777c001750 tx=0x7f7778005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bedb99f48b45b0c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:08.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.087+0000 7f77908e5640 1 -- 172.21.15.88:0/2300531471 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f777c001370 con 0x7f7778004800 2024-09-16T01:41:08.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.087+0000 7f77908e5640 1 -- 172.21.15.88:0/2300531471 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f777c0052d0 con 0x7f7778004800 2024-09-16T01:41:08.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.087+0000 7f77908e5640 1 -- 172.21.15.88:0/2300531471 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f777c02abc0 con 0x7f7778004800 2024-09-16T01:41:08.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.088+0000 7f7793b4b640 1 -- 172.21.15.88:0/2300531471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7778004800 msgr2=0x7f7778004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:08.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.088+0000 7f7793b4b640 1 --2- 172.21.15.88:0/2300531471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7778004800 0x7f7778004c80 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f777c001750 tx=0x7f7778005830 comp rx=0 tx=0).stop 2024-09-16T01:41:08.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.088+0000 7f7793b4b640 1 -- 172.21.15.88:0/2300531471 shutdown_connections 2024-09-16T01:41:08.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.088+0000 7f7793b4b640 1 --2- 172.21.15.88:0/2300531471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7778004800 0x7f7778004c80 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.088+0000 7f7793b4b640 1 -- 172.21.15.88:0/2300531471 >> 172.21.15.88:0/2300531471 conn(0x7f778c062a20 msgr2=0x7f778c062e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:08.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.088+0000 7f7793b4b640 1 -- 172.21.15.88:0/2300531471 shutdown_connections 2024-09-16T01:41:08.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.088+0000 7f7793b4b640 1 -- 172.21.15.88:0/2300531471 wait complete. 2024-09-16T01:41:08.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.089+0000 7f7793b4b640 1 Processor -- start 2024-09-16T01:41:08.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.089+0000 7f7793b4b640 1 -- start start 2024-09-16T01:41:08.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.090+0000 7f7793b4b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7778004800 0x7f778c148800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.090+0000 7f7793b4b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f778c15ed90 0x7f778c15f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.090+0000 7f7793b4b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f778c0cb960 0x7f778c0cbde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.090+0000 7f7793b4b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f778c0cdc00 con 0x7f778c0cb960 2024-09-16T01:41:08.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.090+0000 7f7793b4b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f778c0cead0 con 0x7f778c15ed90 2024-09-16T01:41:08.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.090+0000 7f7793b4b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f778c0d2110 con 0x7f7778004800 2024-09-16T01:41:08.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.090+0000 7f77920e8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f778c0cb960 0x7f778c0cbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.090+0000 7f77920e8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f778c0cb960 0x7f778c0cbde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55834/0 (socket says 172.21.15.88:55834) 2024-09-16T01:41:08.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.090+0000 7f77920e8640 1 -- 172.21.15.88:0/2357835638 learned_addr learned my addr 172.21.15.88:0/2357835638 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:08.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.090+0000 7f77910e6640 1 --2- 172.21.15.88:0/2357835638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f778c15ed90 0x7f778c15f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.090+0000 7f77918e7640 1 --2- 172.21.15.88:0/2357835638 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7778004800 0x7f778c148800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.090+0000 7f77920e8640 1 -- 172.21.15.88:0/2357835638 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7778004800 msgr2=0x7f778c148800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:08.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.090+0000 7f77920e8640 1 --2- 172.21.15.88:0/2357835638 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7778004800 0x7f778c148800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.090+0000 7f77920e8640 1 -- 172.21.15.88:0/2357835638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f778c15ed90 msgr2=0x7f778c15f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:08.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.090+0000 7f77920e8640 1 --2- 172.21.15.88:0/2357835638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f778c15ed90 0x7f778c15f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.090+0000 7f77920e8640 1 -- 172.21.15.88:0/2357835638 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f778c148d40 con 0x7f778c0cb960 2024-09-16T01:41:08.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.090+0000 7f77920e8640 1 --2- 172.21.15.88:0/2357835638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f778c0cb960 0x7f778c0cbde0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f778c0b2010 tx=0x7f7784009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:08.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.090+0000 7f77827fc640 1 -- 172.21.15.88:0/2357835638 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7784003610 con 0x7f778c0cb960 2024-09-16T01:41:08.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.090+0000 7f77827fc640 1 -- 172.21.15.88:0/2357835638 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f778401b070 con 0x7f778c0cb960 2024-09-16T01:41:08.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.090+0000 7f7793b4b640 1 -- 172.21.15.88:0/2357835638 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f778c0ca3a0 con 0x7f778c0cb960 2024-09-16T01:41:08.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.091+0000 7f77827fc640 1 -- 172.21.15.88:0/2357835638 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7784003a30 con 0x7f778c0cb960 2024-09-16T01:41:08.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.091+0000 7f7793b4b640 1 -- 172.21.15.88:0/2357835638 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f778c0ca960 con 0x7f778c0cb960 2024-09-16T01:41:08.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.092+0000 7f77827fc640 1 -- 172.21.15.88:0/2357835638 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7784003bd0 con 0x7f778c0cb960 2024-09-16T01:41:08.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.092+0000 7f77827fc640 1 --2- 172.21.15.88:0/2357835638 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f777006c2b0 0x7f777006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.093+0000 7f77827fc640 1 -- 172.21.15.88:0/2357835638 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6340+0+0 (secure 0 0 0) 0x7f778408b9e0 con 0x7f778c0cb960 2024-09-16T01:41:08.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.093+0000 7f77918e7640 1 --2- 172.21.15.88:0/2357835638 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f777006c2b0 0x7f777006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.093+0000 7f77918e7640 1 --2- 172.21.15.88:0/2357835638 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f777006c2b0 0x7f777006e770 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f7788001330 tx=0x7f7788006040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:08.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.094+0000 7f7796522740 1 Processor -- start 2024-09-16T01:41:08.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.094+0000 7f7796522740 1 -- start start 2024-09-16T01:41:08.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.094+0000 7f7796522740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a4b2c0 0x558191a90a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.094+0000 7f7796522740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191a48a80 0x558191a4b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.094+0000 7f7796522740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558191a4ca50 0x558191a458b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.094+0000 7f7796522740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5581919ad520 con 0x558191a4b2c0 2024-09-16T01:41:08.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.094+0000 7f7796522740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5581919553e0 con 0x558191a48a80 2024-09-16T01:41:08.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.094+0000 7f7796522740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5581919a94e0 con 0x558191a4ca50 2024-09-16T01:41:08.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.094+0000 7f77918e7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a4b2c0 0x558191a90a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.095+0000 7f77918e7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a4b2c0 0x558191a90a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55836/0 (socket says 172.21.15.88:55836) 2024-09-16T01:41:08.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.095+0000 7f77918e7640 1 -- 172.21.15.88:0/3077804778 learned_addr learned my addr 172.21.15.88:0/3077804778 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:08.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.095+0000 7f77920e8640 1 --2- 172.21.15.88:0/3077804778 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558191a4ca50 0x558191a458b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.095+0000 7f77910e6640 1 --2- 172.21.15.88:0/3077804778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191a48a80 0x558191a4b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.095+0000 7f77918e7640 1 -- 172.21.15.88:0/3077804778 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558191a4ca50 msgr2=0x558191a458b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:08.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.095+0000 7f77918e7640 1 --2- 172.21.15.88:0/3077804778 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558191a4ca50 0x558191a458b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.095+0000 7f77918e7640 1 -- 172.21.15.88:0/3077804778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191a48a80 msgr2=0x558191a4b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:08.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.095+0000 7f77918e7640 1 --2- 172.21.15.88:0/3077804778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191a48a80 0x558191a4b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.095+0000 7f77918e7640 1 -- 172.21.15.88:0/3077804778 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581919702f0 con 0x558191a4b2c0 2024-09-16T01:41:08.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.095+0000 7f77920e8640 1 --2- 172.21.15.88:0/3077804778 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558191a4ca50 0x558191a458b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:08.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.095+0000 7f77910e6640 1 --2- 172.21.15.88:0/3077804778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191a48a80 0x558191a4b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:08.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.095+0000 7f77918e7640 1 --2- 172.21.15.88:0/3077804778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a4b2c0 0x558191a90a60 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f77880155d0 tx=0x7f7788015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15e63586a1efa47e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:08.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.095+0000 7f776f7fe640 1 -- 172.21.15.88:0/3077804778 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7788019070 con 0x558191a4b2c0 2024-09-16T01:41:08.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.095+0000 7f776f7fe640 1 -- 172.21.15.88:0/3077804778 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7788026070 con 0x558191a4b2c0 2024-09-16T01:41:08.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.095+0000 7f776f7fe640 1 -- 172.21.15.88:0/3077804778 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7788021610 con 0x558191a4b2c0 2024-09-16T01:41:08.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.096+0000 7f7796522740 1 -- 172.21.15.88:0/3077804778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a4b2c0 msgr2=0x558191a90a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:08.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.096+0000 7f7796522740 1 --2- 172.21.15.88:0/3077804778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a4b2c0 0x558191a90a60 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f77880155d0 tx=0x7f7788015000 comp rx=0 tx=0).stop 2024-09-16T01:41:08.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.096+0000 7f7796522740 1 -- 172.21.15.88:0/3077804778 shutdown_connections 2024-09-16T01:41:08.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.096+0000 7f7796522740 1 --2- 172.21.15.88:0/3077804778 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558191a4ca50 0x558191a458b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.096+0000 7f7796522740 1 --2- 172.21.15.88:0/3077804778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191a48a80 0x558191a4b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.096+0000 7f7796522740 1 --2- 172.21.15.88:0/3077804778 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a4b2c0 0x558191a90a60 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.096+0000 7f7796522740 1 -- 172.21.15.88:0/3077804778 >> 172.21.15.88:0/3077804778 conn(0x558191a47a80 msgr2=0x558191a8ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:08.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.096+0000 7f7796522740 1 -- 172.21.15.88:0/3077804778 shutdown_connections 2024-09-16T01:41:08.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.096+0000 7f7796522740 1 -- 172.21.15.88:0/3077804778 wait complete. 2024-09-16T01:41:08.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.096+0000 7f7796522740 1 -- start start 2024-09-16T01:41:08.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.096+0000 7f7796522740 1 Processor -- start 2024-09-16T01:41:08.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.097+0000 7f7796522740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191a4b2c0 0x558191a674d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.097+0000 7f7796522740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558191a4ca50 0x558191a74e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.097+0000 7f7796522740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a753b0 0x558191a73640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.097+0000 7f7796522740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558191a46f20 con 0x558191a753b0 2024-09-16T01:41:08.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.097+0000 7f7796522740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558191936450 con 0x558191a4b2c0 2024-09-16T01:41:08.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.097+0000 7f7796522740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x558191a91290 con 0x558191a4ca50 2024-09-16T01:41:08.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.097+0000 7f7796522740 10 client.0 _ll_register_callbacks cb 0x5581916bfac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:41:08.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.097+0000 7f77920e8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a753b0 0x558191a73640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.097+0000 7f77920e8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a753b0 0x558191a73640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55852/0 (socket says 172.21.15.88:55852) 2024-09-16T01:41:08.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.098+0000 7f77920e8640 1 -- 172.21.15.88:0/3436022985 learned_addr learned my addr 172.21.15.88:0/3436022985 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:08.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.098+0000 7f77910e6640 1 --2- 172.21.15.88:0/3436022985 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558191a4ca50 0x558191a74e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.098+0000 7f77918e7640 1 --2- 172.21.15.88:0/3436022985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191a4b2c0 0x558191a674d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.098+0000 7f77920e8640 1 -- 172.21.15.88:0/3436022985 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558191a4ca50 msgr2=0x558191a74e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:08.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.098+0000 7f77920e8640 1 --2- 172.21.15.88:0/3436022985 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558191a4ca50 0x558191a74e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.098+0000 7f77920e8640 1 -- 172.21.15.88:0/3436022985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191a4b2c0 msgr2=0x558191a674d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:08.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.098+0000 7f77920e8640 1 --2- 172.21.15.88:0/3436022985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191a4b2c0 0x558191a674d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.098+0000 7f77920e8640 1 -- 172.21.15.88:0/3436022985 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581916c02d0 con 0x558191a753b0 2024-09-16T01:41:08.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.098+0000 7f77920e8640 1 --2- 172.21.15.88:0/3436022985 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a753b0 0x558191a73640 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f7784055c50 tx=0x7f77840598b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6267508f50182384 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:08.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.098+0000 7f776cff9640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:41:08.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.098+0000 7f776cff9640 1 -- 172.21.15.88:0/3436022985 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7784058e40 con 0x558191a753b0 2024-09-16T01:41:08.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.098+0000 7f776cff9640 1 -- 172.21.15.88:0/3436022985 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 6340+0+0 (secure 0 0 0) 0x7f7784055540 con 0x558191a753b0 2024-09-16T01:41:08.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.098+0000 7f7796522740 1 -- 172.21.15.88:0/3436022985 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7788026070 con 0x558191a753b0 2024-09-16T01:41:08.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.098+0000 7f776cff9640 1 -- 172.21.15.88:0/3436022985 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=117) ==== 24+0+0 (secure 0 0 0) 0x7f7784055740 con 0x558191a753b0 2024-09-16T01:41:08.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.098+0000 7f7796522740 10 client.7977 fetch_fsmap learned FSMap version 117 2024-09-16T01:41:08.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.098+0000 7f7796522740 1 -- 172.21.15.88:0/3436022985 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=117,monmap=2+}) -- 0x55819178a6b0 con 0x558191a753b0 2024-09-16T01:41:08.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.098+0000 7f776cff9640 1 -- 172.21.15.88:0/3436022985 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7788026070 con 0x558191a753b0 2024-09-16T01:41:08.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.098+0000 7f776cff9640 1 -- 172.21.15.88:0/3436022985 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 117) ==== 3502+0+0 (secure 0 0 0) 0x7f77840586b0 con 0x558191a753b0 2024-09-16T01:41:08.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.099+0000 7f7796522740 10 client.7977 fetch_fsmap finished waiting for FSMap version 117 2024-09-16T01:41:08.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.099+0000 7f7796522740 10 client.7977 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:41:08.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.099+0000 7f7796522740 20 client.7977 populate_metadata read hostname 'smithi088' 2024-09-16T01:41:08.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.099+0000 7f7796522740 1 --2- 172.21.15.88:0/3436022985 >> [v2:172.21.15.88:6834/128058689,v1:172.21.15.88:6836/128058689] conn(0x558191a8b260 0x558191a8b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.099+0000 7f7796522740 4 client.7977 mds_command: new command op to 7920 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:41:08.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.099+0000 7f7796522740 1 -- 172.21.15.88:0/3436022985 --> [v2:172.21.15.88:6834/128058689,v1:172.21.15.88:6836/128058689] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581919bac50 con 0x558191a8b260 2024-09-16T01:41:08.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.099+0000 7f77918e7640 1 --2- 172.21.15.88:0/3436022985 >> [v2:172.21.15.88:6834/128058689,v1:172.21.15.88:6836/128058689] conn(0x558191a8b260 0x558191a8b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.099+0000 7f77918e7640 1 --2- 172.21.15.88:0/3436022985 >> [v2:172.21.15.88:6834/128058689,v1:172.21.15.88:6836/128058689] conn(0x558191a8b260 0x558191a8b640 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cebb6757a834af8 server_cookie=a48afd692fdcb152 in_seq=0 out_seq=0 2024-09-16T01:41:08.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.099+0000 7f776cff9640 10 client.7977 ms_handle_connect on v2:172.21.15.88:6834/128058689 2024-09-16T01:41:08.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.100+0000 7f776cff9640 1 -- 172.21.15.88:0/3436022985 <== mds.0 v2:172.21.15.88:6834/128058689 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5581919bac50 con 0x558191a8b260 2024-09-16T01:41:08.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.100+0000 7f776cff9640 10 client.7977 handle_command_reply: tid=0 2024-09-16T01:41:08.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.100+0000 7f7796522740 1 client.7977 shutdown 2024-09-16T01:41:08.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.100+0000 7f7796522740 1 -- 172.21.15.88:0/3436022985 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a753b0 msgr2=0x558191a73640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:08.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.100+0000 7f7796522740 1 --2- 172.21.15.88:0/3436022985 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a753b0 0x558191a73640 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f7784055c50 tx=0x7f77840598b0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.100+0000 7f7796522740 1 -- 172.21.15.88:0/3436022985 shutdown_connections 2024-09-16T01:41:08.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.100+0000 7f7796522740 1 --2- 172.21.15.88:0/3436022985 >> [v2:172.21.15.88:6834/128058689,v1:172.21.15.88:6836/128058689] conn(0x558191a8b260 0x558191a8b640 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.100+0000 7f7796522740 1 --2- 172.21.15.88:0/3436022985 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a753b0 0x558191a73640 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.100+0000 7f7796522740 1 --2- 172.21.15.88:0/3436022985 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558191a4ca50 0x558191a74e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.100+0000 7f776cff9640 0 client.7977 ms_handle_reset on v2:172.21.15.88:6834/128058689 2024-09-16T01:41:08.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.100+0000 7f7796522740 1 --2- 172.21.15.88:0/3436022985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191a4b2c0 0x558191a674d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.100+0000 7f7796522740 1 -- 172.21.15.88:0/3436022985 >> 172.21.15.88:0/3436022985 conn(0x558191a48a80 msgr2=0x558191a8f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:08.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.100+0000 7f7796522740 1 -- 172.21.15.88:0/3436022985 shutdown_connections 2024-09-16T01:41:08.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.100+0000 7f7796522740 1 -- 172.21.15.88:0/3436022985 wait complete. 2024-09-16T01:41:08.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.100+0000 7f7796522740 20 client.7977 trim_cache size 0 max 16384 2024-09-16T01:41:08.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.118+0000 7f7796522740 1 Processor -- start 2024-09-16T01:41:08.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.118+0000 7f7796522740 1 -- start start 2024-09-16T01:41:08.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.119+0000 7f7796522740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191a4b2c0 0x558191a71b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.119+0000 7f7796522740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558191a4ca50 0x558191ab4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.119+0000 7f7796522740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a753b0 0x558191a40ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.119+0000 7f7796522740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f778c1fe560 con 0x558191a753b0 2024-09-16T01:41:08.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.119+0000 7f7796522740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f778c1fd9a0 con 0x558191a4b2c0 2024-09-16T01:41:08.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.119+0000 7f7796522740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f778c1fdf20 con 0x558191a4ca50 2024-09-16T01:41:08.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.119+0000 7f77920e8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a753b0 0x558191a40ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.119+0000 7f77920e8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a753b0 0x558191a40ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55858/0 (socket says 172.21.15.88:55858) 2024-09-16T01:41:08.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.119+0000 7f77920e8640 1 -- 172.21.15.88:0/1252414525 learned_addr learned my addr 172.21.15.88:0/1252414525 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:08.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.119+0000 7f77918e7640 1 --2- 172.21.15.88:0/1252414525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191a4b2c0 0x558191a71b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.119+0000 7f77910e6640 1 --2- 172.21.15.88:0/1252414525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558191a4ca50 0x558191ab4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.119+0000 7f77920e8640 1 -- 172.21.15.88:0/1252414525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558191a4ca50 msgr2=0x558191ab4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:08.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.119+0000 7f77920e8640 1 --2- 172.21.15.88:0/1252414525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558191a4ca50 0x558191ab4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.119+0000 7f77920e8640 1 -- 172.21.15.88:0/1252414525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191a4b2c0 msgr2=0x558191a71b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:08.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.119+0000 7f77920e8640 1 --2- 172.21.15.88:0/1252414525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191a4b2c0 0x558191a71b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.119+0000 7f77920e8640 1 -- 172.21.15.88:0/1252414525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558191a41430 con 0x558191a753b0 2024-09-16T01:41:08.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.119+0000 7f77920e8640 1 --2- 172.21.15.88:0/1252414525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a753b0 0x558191a40ef0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f778408bd00 tx=0x7f778406e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e47b751b38bbd7b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:08.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.119+0000 7f776effd640 1 -- 172.21.15.88:0/1252414525 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7784055a90 con 0x558191a753b0 2024-09-16T01:41:08.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.119+0000 7f776effd640 1 -- 172.21.15.88:0/1252414525 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f778405daf0 con 0x558191a753b0 2024-09-16T01:41:08.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.119+0000 7f776effd640 1 -- 172.21.15.88:0/1252414525 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7784058c40 con 0x558191a753b0 2024-09-16T01:41:08.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.120+0000 7f7796522740 1 -- 172.21.15.88:0/1252414525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a753b0 msgr2=0x558191a40ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:08.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.120+0000 7f7796522740 1 --2- 172.21.15.88:0/1252414525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a753b0 0x558191a40ef0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f778408bd00 tx=0x7f778406e000 comp rx=0 tx=0).stop 2024-09-16T01:41:08.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.120+0000 7f7796522740 1 -- 172.21.15.88:0/1252414525 shutdown_connections 2024-09-16T01:41:08.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.120+0000 7f7796522740 1 --2- 172.21.15.88:0/1252414525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a753b0 0x558191a40ef0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.120+0000 7f7796522740 1 --2- 172.21.15.88:0/1252414525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558191a4ca50 0x558191ab4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.120+0000 7f7796522740 1 --2- 172.21.15.88:0/1252414525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191a4b2c0 0x558191a71b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.120+0000 7f7796522740 1 -- 172.21.15.88:0/1252414525 >> 172.21.15.88:0/1252414525 conn(0x558191a48a80 msgr2=0x558191a900f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:08.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.120+0000 7f7796522740 1 -- 172.21.15.88:0/1252414525 shutdown_connections 2024-09-16T01:41:08.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.120+0000 7f7796522740 1 -- 172.21.15.88:0/1252414525 wait complete. 2024-09-16T01:41:08.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.120+0000 7f7796522740 1 -- start start 2024-09-16T01:41:08.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.121+0000 7f7796522740 1 Processor -- start 2024-09-16T01:41:08.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.121+0000 7f7796522740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a4b2c0 0x558191a8b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.121+0000 7f7796522740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558191a4ca50 0x558191b0f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.121+0000 7f7796522740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191a753b0 0x558191af41a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.121+0000 7f7796522740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558191a8c300 con 0x558191a4b2c0 2024-09-16T01:41:08.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.121+0000 7f7796522740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558191a8ad10 con 0x558191a753b0 2024-09-16T01:41:08.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.121+0000 7f7796522740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x558191aa3770 con 0x558191a4ca50 2024-09-16T01:41:08.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.121+0000 7f7796522740 10 client.0 _ll_register_callbacks cb 0x7f7784004500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:41:08.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.122+0000 7f77918e7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a4b2c0 0x558191a8b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.122+0000 7f77918e7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a4b2c0 0x558191a8b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55868/0 (socket says 172.21.15.88:55868) 2024-09-16T01:41:08.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.122+0000 7f77918e7640 1 -- 172.21.15.88:0/800379952 learned_addr learned my addr 172.21.15.88:0/800379952 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:08.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.122+0000 7f77920e8640 1 --2- 172.21.15.88:0/800379952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191a753b0 0x558191af41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.122+0000 7f77910e6640 1 --2- 172.21.15.88:0/800379952 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558191a4ca50 0x558191b0f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.122+0000 7f77918e7640 1 -- 172.21.15.88:0/800379952 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558191a4ca50 msgr2=0x558191b0f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:08.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.122+0000 7f77918e7640 1 --2- 172.21.15.88:0/800379952 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558191a4ca50 0x558191b0f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.122+0000 7f77918e7640 1 -- 172.21.15.88:0/800379952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191a753b0 msgr2=0x558191af41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:08.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.122+0000 7f77918e7640 1 --2- 172.21.15.88:0/800379952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191a753b0 0x558191af41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.122+0000 7f77918e7640 1 -- 172.21.15.88:0/800379952 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558191ab40e0 con 0x558191a4b2c0 2024-09-16T01:41:08.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.122+0000 7f77910e6640 1 --2- 172.21.15.88:0/800379952 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558191a4ca50 0x558191b0f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:08.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.122+0000 7f77918e7640 1 --2- 172.21.15.88:0/800379952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a4b2c0 0x558191a8b6f0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f7788004fd0 tx=0x7f7788001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1da478b13b4e17de server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:08.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.122+0000 7f776cff9640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:41:08.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.122+0000 7f776cff9640 1 -- 172.21.15.88:0/800379952 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7788019040 con 0x558191a4b2c0 2024-09-16T01:41:08.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.122+0000 7f776cff9640 1 -- 172.21.15.88:0/800379952 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 6340+0+0 (secure 0 0 0) 0x7f77880396f0 con 0x558191a4b2c0 2024-09-16T01:41:08.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.122+0000 7f7796522740 1 -- 172.21.15.88:0/800379952 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f778405daf0 con 0x558191a4b2c0 2024-09-16T01:41:08.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.122+0000 7f776cff9640 1 -- 172.21.15.88:0/800379952 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=117) ==== 24+0+0 (secure 0 0 0) 0x7f7788049050 con 0x558191a4b2c0 2024-09-16T01:41:08.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.122+0000 7f7796522740 10 client.7983 fetch_fsmap learned FSMap version 117 2024-09-16T01:41:08.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.122+0000 7f7796522740 1 -- 172.21.15.88:0/800379952 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=117,monmap=2+}) -- 0x558191ab43c0 con 0x558191a4b2c0 2024-09-16T01:41:08.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.122+0000 7f776cff9640 1 -- 172.21.15.88:0/800379952 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f778405daf0 con 0x558191a4b2c0 2024-09-16T01:41:08.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.122+0000 7f776cff9640 1 -- 172.21.15.88:0/800379952 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 117) ==== 3502+0+0 (secure 0 0 0) 0x7f778804c020 con 0x558191a4b2c0 2024-09-16T01:41:08.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.123+0000 7f7796522740 10 client.7983 fetch_fsmap finished waiting for FSMap version 117 2024-09-16T01:41:08.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.123+0000 7f7796522740 10 client.7983 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:41:08.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.123+0000 7f7796522740 20 client.7983 populate_metadata read hostname 'smithi088' 2024-09-16T01:41:08.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.123+0000 7f7796522740 1 --2- 172.21.15.88:0/800379952 >> [v2:172.21.15.88:6834/128058689,v1:172.21.15.88:6836/128058689] conn(0x558191aa2250 0x558191a6ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.123+0000 7f7796522740 4 client.7983 mds_command: new command op to 7920 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:41:08.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.123+0000 7f7796522740 1 -- 172.21.15.88:0/800379952 --> [v2:172.21.15.88:6834/128058689,v1:172.21.15.88:6836/128058689] -- command(tid 0: {"prefix": "status"}) -- 0x5581919b4fa0 con 0x558191aa2250 2024-09-16T01:41:08.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.123+0000 7f77910e6640 1 --2- 172.21.15.88:0/800379952 >> [v2:172.21.15.88:6834/128058689,v1:172.21.15.88:6836/128058689] conn(0x558191aa2250 0x558191a6ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.123+0000 7f77910e6640 1 --2- 172.21.15.88:0/800379952 >> [v2:172.21.15.88:6834/128058689,v1:172.21.15.88:6836/128058689] conn(0x558191aa2250 0x558191a6ded0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e367060873e1d35a server_cookie=a155685ad7192c56 in_seq=0 out_seq=0 2024-09-16T01:41:08.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.123+0000 7f776cff9640 10 client.7983 ms_handle_connect on v2:172.21.15.88:6834/128058689 2024-09-16T01:41:08.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.123+0000 7f776cff9640 1 -- 172.21.15.88:0/800379952 <== mds.0 v2:172.21.15.88:6834/128058689 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x5581919b4fa0 con 0x558191aa2250 2024-09-16T01:41:08.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.123+0000 7f776cff9640 10 client.7983 handle_command_reply: tid=0 2024-09-16T01:41:08.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.123+0000 7f7796522740 1 client.7983 shutdown 2024-09-16T01:41:08.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.123+0000 7f7796522740 1 -- 172.21.15.88:0/800379952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a4b2c0 msgr2=0x558191a8b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:08.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.123+0000 7f7796522740 1 --2- 172.21.15.88:0/800379952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a4b2c0 0x558191a8b6f0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f7788004fd0 tx=0x7f7788001360 comp rx=0 tx=0).stop 2024-09-16T01:41:08.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.123+0000 7f7796522740 1 -- 172.21.15.88:0/800379952 shutdown_connections 2024-09-16T01:41:08.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.123+0000 7f7796522740 1 --2- 172.21.15.88:0/800379952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191a753b0 0x558191af41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.123+0000 7f7796522740 1 --2- 172.21.15.88:0/800379952 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558191a4ca50 0x558191b0f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.123+0000 7f7796522740 1 --2- 172.21.15.88:0/800379952 >> [v2:172.21.15.88:6834/128058689,v1:172.21.15.88:6836/128058689] conn(0x558191aa2250 0x558191a6ded0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.123+0000 7f7796522740 1 --2- 172.21.15.88:0/800379952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558191a4b2c0 0x558191a8b6f0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.123+0000 7f776cff9640 0 client.7983 ms_handle_reset on v2:172.21.15.88:6834/128058689 2024-09-16T01:41:08.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.123+0000 7f7796522740 1 -- 172.21.15.88:0/800379952 >> 172.21.15.88:0/800379952 conn(0x558191a48a80 msgr2=0x558191a900f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:08.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.123+0000 7f7796522740 1 -- 172.21.15.88:0/800379952 shutdown_connections 2024-09-16T01:41:08.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.123+0000 7f7796522740 1 -- 172.21.15.88:0/800379952 wait complete. 2024-09-16T01:41:08.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.124+0000 7f7796522740 20 client.7983 trim_cache size 0 max 16384 2024-09-16T01:41:08.132 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:41:08.132 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "5ace2c91-46f9-482b-8a89-731fbdc171bf", 2024-09-16T01:41:08.132 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2024-09-16T01:41:08.132 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 7920, 2024-09-16T01:41:08.132 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2024-09-16T01:41:08.132 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2024-09-16T01:41:08.132 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2024-09-16T01:41:08.132 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 6.6290870530000001, 2024-09-16T01:41:08.133 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 117, 2024-09-16T01:41:08.133 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 94, 2024-09-16T01:41:08.133 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 93, 2024-09-16T01:41:08.133 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 9.011890223 2024-09-16T01:41:08.133 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:41:08.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.124+0000 7f7793b4b640 1 -- 172.21.15.88:0/2357835638 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f777006c2b0 msgr2=0x7f777006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:08.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.124+0000 7f7793b4b640 1 --2- 172.21.15.88:0/2357835638 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f777006c2b0 0x7f777006e770 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f7788001330 tx=0x7f7788006040 comp rx=0 tx=0).stop 2024-09-16T01:41:08.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.124+0000 7f7793b4b640 1 -- 172.21.15.88:0/2357835638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f778c0cb960 msgr2=0x7f778c0cbde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:08.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.124+0000 7f7793b4b640 1 --2- 172.21.15.88:0/2357835638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f778c0cb960 0x7f778c0cbde0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f778c0b2010 tx=0x7f7784009f90 comp rx=0 tx=0).stop 2024-09-16T01:41:08.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.124+0000 7f7793b4b640 1 -- 172.21.15.88:0/2357835638 shutdown_connections 2024-09-16T01:41:08.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.124+0000 7f7793b4b640 1 --2- 172.21.15.88:0/2357835638 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f777006c2b0 0x7f777006e770 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.124+0000 7f7793b4b640 1 --2- 172.21.15.88:0/2357835638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f778c0cb960 0x7f778c0cbde0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.124+0000 7f7793b4b640 1 --2- 172.21.15.88:0/2357835638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f778c15ed90 0x7f778c15f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.124+0000 7f7793b4b640 1 --2- 172.21.15.88:0/2357835638 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7778004800 0x7f778c148800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.124+0000 7f7793b4b640 1 -- 172.21.15.88:0/2357835638 >> 172.21.15.88:0/2357835638 conn(0x7f778c062a20 msgr2=0x7f778c0b2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:08.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.124+0000 7f7793b4b640 1 -- 172.21.15.88:0/2357835638 shutdown_connections 2024-09-16T01:41:08.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.124+0000 7f7793b4b640 1 -- 172.21.15.88:0/2357835638 wait complete. 2024-09-16T01:41:08.136 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-09-16T01:41:08.136 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Delete 500.00000000', proceeding to ls 2024-09-16T01:41:08.136 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:41:08.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.233+0000 7f6d09598640 1 Processor -- start 2024-09-16T01:41:08.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.233+0000 7f6d09598640 1 -- start start 2024-09-16T01:41:08.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.234+0000 7f6d09598640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6d0415e660 0x7f6d040cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.234+0000 7f6d09598640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6d040d1820 0x7f6d040b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.234+0000 7f6d09598640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d0415ea40 con 0x7f6d040d3310 2024-09-16T01:41:08.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.234+0000 7f6d09598640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6d040d3870 con 0x7f6d0415e660 2024-09-16T01:41:08.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.234+0000 7f6d09598640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6d040d3b50 con 0x7f6d040d1820 2024-09-16T01:41:08.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.234+0000 7f6d02ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6d0415e660 0x7f6d040cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.234+0000 7f6cfa7fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6d040d3310 0x7f6d040d1110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:47954/0 (socket says 172.21.15.88:47954) 2024-09-16T01:41:08.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.234+0000 7f6cfa7fc640 1 -- 172.21.15.88:0/4287853823 learned_addr learned my addr 172.21.15.88:0/4287853823 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:08.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.234+0000 7f6d02ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6d0415e660 0x7f6d040cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55876/0 (socket says 172.21.15.88:55876) 2024-09-16T01:41:08.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.234+0000 7f6d037fe640 1 --2- 172.21.15.88:0/4287853823 >> v2:172.21.15.154:3300/0 conn(0x7f6d040d1820 0x7f6d040b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.234+0000 7f6d02ffd640 1 -- 172.21.15.88:0/4287853823 >> v2:172.21.15.154:3300/0 conn(0x7f6d040d1820 msgr2=0x7f6d040b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:08.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.234+0000 7f6d02ffd640 1 --2- 172.21.15.88:0/4287853823 >> v2:172.21.15.154:3300/0 conn(0x7f6d040d1820 0x7f6d040b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.234+0000 7f6d02ffd640 1 -- 172.21.15.88:0/4287853823 >> v1:172.21.15.88:6789/0 conn(0x7f6d040d3310 legacy=0x7f6d040d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:08.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.234+0000 7f6d02ffd640 1 -- 172.21.15.88:0/4287853823 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d040d3e30 con 0x7f6d0415e660 2024-09-16T01:41:08.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.234+0000 7f6d027fc640 1 -- 172.21.15.88:0/4287853823 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3901093872 0 0) 0x7f6d0415ea40 con 0x7f6d040d3310 2024-09-16T01:41:08.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.235+0000 7f6d02ffd640 1 --2- 172.21.15.88:0/4287853823 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d0415e660 0x7f6d040cf730 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f6d040d26b0 tx=0x7f6cf402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e268c8d591130897 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:08.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.235+0000 7f6d027fc640 1 -- 172.21.15.88:0/4287853823 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4037070 con 0x7f6d0415e660 2024-09-16T01:41:08.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.235+0000 7f6d027fc640 1 -- 172.21.15.88:0/4287853823 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cf4032440 con 0x7f6d0415e660 2024-09-16T01:41:08.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.235+0000 7f6d027fc640 1 -- 172.21.15.88:0/4287853823 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4032810 con 0x7f6d0415e660 2024-09-16T01:41:08.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.236+0000 7f6d09598640 1 -- 172.21.15.88:0/4287853823 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d0415e660 msgr2=0x7f6d040cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:08.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.236+0000 7f6d09598640 1 --2- 172.21.15.88:0/4287853823 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d0415e660 0x7f6d040cf730 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f6d040d26b0 tx=0x7f6cf402a1a0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.236+0000 7f6d09598640 1 -- 172.21.15.88:0/4287853823 shutdown_connections 2024-09-16T01:41:08.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.236+0000 7f6d09598640 1 --2- 172.21.15.88:0/4287853823 >> v2:172.21.15.154:3300/0 conn(0x7f6d040d1820 0x7f6d040b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.236+0000 7f6d09598640 1 --2- 172.21.15.88:0/4287853823 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d0415e660 0x7f6d040cf730 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.236+0000 7f6d09598640 1 -- 172.21.15.88:0/4287853823 >> 172.21.15.88:0/4287853823 conn(0x7f6d040b6410 msgr2=0x7f6d040b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:08.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.236+0000 7f6d09598640 1 -- 172.21.15.88:0/4287853823 shutdown_connections 2024-09-16T01:41:08.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.236+0000 7f6d09598640 1 -- 172.21.15.88:0/4287853823 wait complete. 2024-09-16T01:41:08.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.237+0000 7f6d09598640 1 Processor -- start 2024-09-16T01:41:08.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.237+0000 7f6d09598640 1 -- start start 2024-09-16T01:41:08.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.238+0000 7f6d09598640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d040d1820 0x7f6d040c89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.238+0000 7f6d09598640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d040d3310 0x7f6d040b8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.238+0000 7f6d09598640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d0415e660 0x7f6d041f3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.238+0000 7f6d09598640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6d040cfc70 con 0x7f6d040d1820 2024-09-16T01:41:08.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.238+0000 7f6d09598640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6d040d43f0 con 0x7f6d040d3310 2024-09-16T01:41:08.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.238+0000 7f6d09598640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6d040b80b0 con 0x7f6d0415e660 2024-09-16T01:41:08.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.238+0000 7f6d02ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d040d1820 0x7f6d040c89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.238+0000 7f6d02ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d040d1820 0x7f6d040c89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55884/0 (socket says 172.21.15.88:55884) 2024-09-16T01:41:08.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.238+0000 7f6d02ffd640 1 -- 172.21.15.88:0/924918400 learned_addr learned my addr 172.21.15.88:0/924918400 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:08.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.238+0000 7f6d02ffd640 1 -- 172.21.15.88:0/924918400 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d0415e660 msgr2=0x7f6d041f3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:08.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.238+0000 7f6d02ffd640 1 --2- 172.21.15.88:0/924918400 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d0415e660 0x7f6d041f3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.238+0000 7f6d02ffd640 1 -- 172.21.15.88:0/924918400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d040d3310 msgr2=0x7f6d040b8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:08.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.238+0000 7f6d02ffd640 1 --2- 172.21.15.88:0/924918400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d040d3310 0x7f6d040b8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.238+0000 7f6d02ffd640 1 -- 172.21.15.88:0/924918400 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d041f4910 con 0x7f6d040d1820 2024-09-16T01:41:08.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.238+0000 7f6d02ffd640 1 --2- 172.21.15.88:0/924918400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d040d1820 0x7f6d040c89f0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4000c00 tx=0x7f6cf40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:08.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.238+0000 7f6cfbfff640 1 -- 172.21.15.88:0/924918400 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4037040 con 0x7f6d040d1820 2024-09-16T01:41:08.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.238+0000 7f6cfbfff640 1 -- 172.21.15.88:0/924918400 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cf4042070 con 0x7f6d040d1820 2024-09-16T01:41:08.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.238+0000 7f6d09598640 1 -- 172.21.15.88:0/924918400 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d041f4bf0 con 0x7f6d040d1820 2024-09-16T01:41:08.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.238+0000 7f6cfbfff640 1 -- 172.21.15.88:0/924918400 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf403b950 con 0x7f6d040d1820 2024-09-16T01:41:08.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.238+0000 7f6d09598640 1 -- 172.21.15.88:0/924918400 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d040c84d0 con 0x7f6d040d1820 2024-09-16T01:41:08.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.239+0000 7f6d09598640 1 -- 172.21.15.88:0/924918400 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d041f4dc0 con 0x7f6d040d1820 2024-09-16T01:41:08.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.240+0000 7f6cfbfff640 1 -- 172.21.15.88:0/924918400 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6cf403baf0 con 0x7f6d040d1820 2024-09-16T01:41:08.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.240+0000 7f6cfbfff640 1 --2- 172.21.15.88:0/924918400 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ce406c2b0 0x7f6ce406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:08.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.241+0000 7f6cfbfff640 1 -- 172.21.15.88:0/924918400 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6340+0+0 (secure 0 0 0) 0x7f6cf40abfb0 con 0x7f6d040d1820 2024-09-16T01:41:08.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.241+0000 7f6cfa7fc640 1 --2- 172.21.15.88:0/924918400 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ce406c2b0 0x7f6ce406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:08.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.241+0000 7f6cfa7fc640 1 --2- 172.21.15.88:0/924918400 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ce406c2b0 0x7f6ce406e770 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f6cec007fd0 tx=0x7f6cec007920 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:08.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.245+0000 7f6cfbfff640 1 -- 172.21.15.88:0/924918400 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6cf407a4a0 con 0x7f6d040d1820 2024-09-16T01:41:08.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.449+0000 7f6d09598640 1 -- 172.21.15.88:0/924918400 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f6d040c86a0 con 0x7f6d040d1820 2024-09-16T01:41:08.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.449+0000 7f6cfbfff640 1 -- 172.21.15.88:0/924918400 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v117) ==== 53+0+73 (secure 0 0 0) 0x7f6cf4079da0 con 0x7f6d040d1820 2024-09-16T01:41:08.451 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:41:08.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.451+0000 7f6d09598640 1 -- 172.21.15.88:0/924918400 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ce406c2b0 msgr2=0x7f6ce406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:08.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.451+0000 7f6d09598640 1 --2- 172.21.15.88:0/924918400 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ce406c2b0 0x7f6ce406e770 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f6cec007fd0 tx=0x7f6cec007920 comp rx=0 tx=0).stop 2024-09-16T01:41:08.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.451+0000 7f6d09598640 1 -- 172.21.15.88:0/924918400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d040d1820 msgr2=0x7f6d040c89f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:08.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.451+0000 7f6d09598640 1 --2- 172.21.15.88:0/924918400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d040d1820 0x7f6d040c89f0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4000c00 tx=0x7f6cf40046c0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.451+0000 7f6d09598640 1 -- 172.21.15.88:0/924918400 shutdown_connections 2024-09-16T01:41:08.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.451+0000 7f6d09598640 1 --2- 172.21.15.88:0/924918400 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ce406c2b0 0x7f6ce406e770 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.451+0000 7f6d09598640 1 --2- 172.21.15.88:0/924918400 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d0415e660 0x7f6d041f3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.451+0000 7f6d09598640 1 --2- 172.21.15.88:0/924918400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d040d3310 0x7f6d040b8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.451+0000 7f6d09598640 1 --2- 172.21.15.88:0/924918400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d040d1820 0x7f6d040c89f0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:08.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.451+0000 7f6d09598640 1 -- 172.21.15.88:0/924918400 >> 172.21.15.88:0/924918400 conn(0x7f6d040b6410 msgr2=0x7f6d040b4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:08.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.451+0000 7f6d09598640 1 -- 172.21.15.88:0/924918400 shutdown_connections 2024-09-16T01:41:08.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08.451+0000 7f6d09598640 1 -- 172.21.15.88:0/924918400 wait complete. 2024-09-16T01:41:08.464 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-16T01:41:08.464 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:41:08.464 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2024-09-16T01:41:08.464 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T01:41:08.464 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T01:41:08.464 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-16T01:41:08.464 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-16T01:41:08.465 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:41:08.465 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:41:08.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:08.467 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2024-09-16T01:41:08.482 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:41:08.483 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:41:08.483 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:41:08.483 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:41:08.483 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-09-16T01:41:08.483 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:41:08.483 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:41:08.483 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:41:08.483 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2024-09-16T01:41:08.483 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:41:08.483 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:41:08.483 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2024-09-16T01:41:08.484 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:41:08.484 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2024-09-16T01:41:08.484 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:41:08.484 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2024-09-16T01:41:08.484 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:41:08.484 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42356sec preferred_lft 42356sec 2024-09-16T01:41:08.484 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link noprefixroute 2024-09-16T01:41:08.484 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:41:08.484 INFO:teuthology.orchestra.run.smithi088.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:41:08.484 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff 2024-09-16T01:41:08.484 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T01:41:08.484 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:41:08.484 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::e486:6eff:fe81:a1c/64 scope link 2024-09-16T01:41:08.485 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:41:08.485 INFO:teuthology.orchestra.run.smithi088.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:41:08.485 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 12:6f:29:be:95:48 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:41:08.485 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::106f:29ff:febe:9548/64 scope link 2024-09-16T01:41:08.485 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:41:08.485 INFO:teuthology.orchestra.run.smithi088.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:41:08.485 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T01:41:08.485 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::c63:b4ff:feb3:ad9e/64 scope link 2024-09-16T01:41:08.485 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:41:08.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:08.486 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:41:08.546 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-09-16T01:41:08.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:08.548 DEBUG:teuthology.orchestra.run.smithi088:> route 2024-09-16T01:41:08.604 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2024-09-16T01:41:08.605 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:41:08.605 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:41:08.605 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:41:08.605 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:41:08.605 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:41:08.605 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:41:08.605 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:41:08.605 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:41:08.605 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:41:08.606 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:41:08.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:41:08.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:41:08.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:08.732 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:41:08.747 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T01:41:08.748 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T01:41:08.748 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:41:08.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:08.748 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2024-09-16T01:41:08.813 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:41:08.813 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T01:41:08.868 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:41:08.869 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:41:08.869 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:41:08.924 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-16T01:41:08.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:08.924 DEBUG:teuthology.orchestra.run.smithi088:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:41:09.006 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:41:09.007 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:41:09.007 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:41:09.015 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:41:09.015 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:41:09.015 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:41:09.015 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:41:09.015 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:41:09.020 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:41:09.020 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:41:09.021 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:41:09.021 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:41:09.021 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:41:09.021 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790" 2024-09-16T01:41:09.021 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2024-09-16T01:41:09.021 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5ace2c91-46f9-482b-8a89-731fbdc171bf" will pass to kernel 2024-09-16T01:41:09.021 INFO:teuthology.orchestra.run.smithi088.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-16T01:41:09.021 INFO:teuthology.orchestra.run.smithi088.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:41:09.022 INFO:teuthology.orchestra.run.smithi088.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:41:09.022 INFO:teuthology.orchestra.run.smithi088.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:41:09.022 INFO:teuthology.orchestra.run.smithi088.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:41:09.026 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:41:09.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:09.027 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:41:09.088 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:41:09.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:41:09.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:41:09.183 INFO:teuthology.orchestra.run.smithi088.stdout:client7995 2024-09-16T01:41:09.184 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:41:09.184 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:41:09.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:09.199 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R .) 2024-09-16T01:41:09.242 INFO:teuthology.orchestra.run:waiting for 20 2024-09-16T01:41:09.257 INFO:teuthology.orchestra.run.smithi088.stdout:.: 2024-09-16T01:41:09.257 INFO:teuthology.orchestra.run.smithi088.stdout:subdir 2024-09-16T01:41:09.257 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:09.257 INFO:teuthology.orchestra.run.smithi088.stdout:./subdir: 2024-09-16T01:41:09.257 INFO:teuthology.orchestra.run.smithi088.stdout:sixmegs 2024-09-16T01:41:09.258 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 500.00000000' did not caused DAMAGED state 2024-09-16T01:41:09.258 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete mds_snaptable' 2024-09-16T01:41:09.258 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:41:09.258 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:41:09.312 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:41:09.312 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:41:09.367 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:41:09.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.506+0000 7fde1c817640 1 Processor -- start 2024-09-16T01:41:09.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.506+0000 7fde1c817640 1 -- start start 2024-09-16T01:41:09.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.507+0000 7fde1c817640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fde1815c2a0 0x7fde180735d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:09.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.507+0000 7fde1c817640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde18074120 con 0x7fde18073b10 2024-09-16T01:41:09.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.507+0000 7fde1c817640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde1815ef00 con 0x7fde1806fe20 2024-09-16T01:41:09.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.507+0000 7fde1c817640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fde1815c680 con 0x7fde1815c2a0 2024-09-16T01:41:09.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.507+0000 7fde0dd9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fde18073b10 0x7fde1806f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:47966/0 (socket says 172.21.15.88:47966) 2024-09-16T01:41:09.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.507+0000 7fde0dd9b640 1 -- 172.21.15.88:0/3300289140 learned_addr learned my addr 172.21.15.88:0/3300289140 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:09.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.507+0000 7fde15d9b640 1 -- 172.21.15.88:0/3300289140 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2942096014 0 0) 0x7fde18074120 con 0x7fde18073b10 2024-09-16T01:41:09.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.507+0000 7fde15d9b640 1 -- 172.21.15.88:0/3300289140 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde04003520 con 0x7fde18073b10 2024-09-16T01:41:09.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.507+0000 7fde1659c640 1 --2- 172.21.15.88:0/3300289140 >> v2:172.21.15.154:3300/0 conn(0x7fde1815c2a0 0x7fde180735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:09.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.507+0000 7fde15d9b640 1 -- 172.21.15.88:0/3300289140 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1639188034 0 0) 0x7fde04003520 con 0x7fde18073b10 2024-09-16T01:41:09.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.507+0000 7fde15d9b640 1 -- 172.21.15.88:0/3300289140 >> v2:172.21.15.154:3300/0 conn(0x7fde1815c2a0 msgr2=0x7fde180735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:09.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.507+0000 7fde15d9b640 1 --2- 172.21.15.88:0/3300289140 >> v2:172.21.15.154:3300/0 conn(0x7fde1815c2a0 0x7fde180735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:09.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.507+0000 7fde15d9b640 1 -- 172.21.15.88:0/3300289140 >> v1:172.21.15.154:6789/0 conn(0x7fde1806fe20 legacy=0x7fde1806bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:09.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.507+0000 7fde15d9b640 1 -- 172.21.15.88:0/3300289140 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde1815f1e0 con 0x7fde18073b10 2024-09-16T01:41:09.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.507+0000 7fde15d9b640 1 -- 172.21.15.88:0/3300289140 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fde00001670 con 0x7fde18073b10 2024-09-16T01:41:09.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.508+0000 7fde15d9b640 1 -- 172.21.15.88:0/3300289140 >> v1:172.21.15.88:6789/0 conn(0x7fde18073b10 legacy=0x7fde1806f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:09.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.508+0000 7fde15d9b640 1 --2- 172.21.15.88:0/3300289140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde040039b0 0x7fde04003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:09.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.508+0000 7fde15d9b640 1 --2- 172.21.15.88:0/3300289140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde04004800 0x7fde04004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:09.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.508+0000 7fde15d9b640 1 --2- 172.21.15.88:0/3300289140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde040051c0 0x7fde0400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:09.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.508+0000 7fde15d9b640 1 -- 172.21.15.88:0/3300289140 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fde0400c0c0 con 0x7fde040039b0 2024-09-16T01:41:09.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.508+0000 7fde15d9b640 1 -- 172.21.15.88:0/3300289140 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fde0400c270 con 0x7fde04004800 2024-09-16T01:41:09.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.508+0000 7fde15d9b640 1 -- 172.21.15.88:0/3300289140 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fde0400c470 con 0x7fde040051c0 2024-09-16T01:41:09.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.508+0000 7fde1659c640 1 --2- 172.21.15.88:0/3300289140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde040039b0 0x7fde04003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:09.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.508+0000 7fde16d9d640 1 --2- 172.21.15.88:0/3300289140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde040051c0 0x7fde0400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:09.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.508+0000 7fde1659c640 1 -- 172.21.15.88:0/3300289140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde040051c0 msgr2=0x7fde0400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:09.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.508+0000 7fde1659c640 1 --2- 172.21.15.88:0/3300289140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde040051c0 0x7fde0400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:09.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.508+0000 7fde1659c640 1 -- 172.21.15.88:0/3300289140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde04004800 msgr2=0x7fde04004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:09.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.508+0000 7fde1659c640 1 --2- 172.21.15.88:0/3300289140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde04004800 0x7fde04004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:09.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.508+0000 7fde1659c640 1 -- 172.21.15.88:0/3300289140 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde04003520 con 0x7fde040039b0 2024-09-16T01:41:09.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.508+0000 7fde16d9d640 1 -- 172.21.15.88:0/3300289140 reap_dead start 2024-09-16T01:41:09.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.509+0000 7fde1659c640 1 --2- 172.21.15.88:0/3300289140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde040039b0 0x7fde04003db0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fde0800ba00 tx=0x7fde04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a67014f47721e417 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:09.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.509+0000 7fde15d9b640 1 -- 172.21.15.88:0/3300289140 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde08030070 con 0x7fde040039b0 2024-09-16T01:41:09.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.509+0000 7fde15d9b640 1 -- 172.21.15.88:0/3300289140 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde0802ada0 con 0x7fde040039b0 2024-09-16T01:41:09.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.509+0000 7fde15d9b640 1 -- 172.21.15.88:0/3300289140 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0800b600 con 0x7fde040039b0 2024-09-16T01:41:09.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.509+0000 7fde1c817640 1 -- 172.21.15.88:0/3300289140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde040039b0 msgr2=0x7fde04003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:09.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.509+0000 7fde1c817640 1 --2- 172.21.15.88:0/3300289140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde040039b0 0x7fde04003db0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fde0800ba00 tx=0x7fde04005830 comp rx=0 tx=0).stop 2024-09-16T01:41:09.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.509+0000 7fde1c817640 1 -- 172.21.15.88:0/3300289140 shutdown_connections 2024-09-16T01:41:09.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.509+0000 7fde1c817640 1 --2- 172.21.15.88:0/3300289140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde040039b0 0x7fde04003db0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:09.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.509+0000 7fde1c817640 1 -- 172.21.15.88:0/3300289140 >> 172.21.15.88:0/3300289140 conn(0x7fde180b3070 msgr2=0x7fde180b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:09.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.509+0000 7fde1c817640 1 -- 172.21.15.88:0/3300289140 shutdown_connections 2024-09-16T01:41:09.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.509+0000 7fde1c817640 1 -- 172.21.15.88:0/3300289140 wait complete. 2024-09-16T01:41:09.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.511+0000 7fde1c817640 1 Processor -- start 2024-09-16T01:41:09.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.511+0000 7fde1c817640 1 -- start start 2024-09-16T01:41:09.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.512+0000 7fde1c817640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde040039b0 0x7fde18158040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:09.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.512+0000 7fde1c817640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde180ce800 0x7fde18159650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:09.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.512+0000 7fde1c817640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde1815dc70 0x7fde1815e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:09.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.512+0000 7fde1c817640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fde180b5f20 con 0x7fde1815dc70 2024-09-16T01:41:09.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.512+0000 7fde1c817640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fde1815ebf0 con 0x7fde040039b0 2024-09-16T01:41:09.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.512+0000 7fde1c817640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fde180b60a0 con 0x7fde180ce800 2024-09-16T01:41:09.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.512+0000 7fde16d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde1815dc70 0x7fde1815e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:09.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.512+0000 7fde16d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde1815dc70 0x7fde1815e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55898/0 (socket says 172.21.15.88:55898) 2024-09-16T01:41:09.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.512+0000 7fde16d9d640 1 -- 172.21.15.88:0/3649977367 learned_addr learned my addr 172.21.15.88:0/3649977367 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:09.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.512+0000 7fde0dd9b640 1 --2- 172.21.15.88:0/3649977367 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde180ce800 0x7fde18159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:09.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.512+0000 7fde1659c640 1 --2- 172.21.15.88:0/3649977367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde040039b0 0x7fde18158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:09.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.512+0000 7fde16d9d640 1 -- 172.21.15.88:0/3649977367 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde180ce800 msgr2=0x7fde18159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:09.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.512+0000 7fde16d9d640 1 --2- 172.21.15.88:0/3649977367 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde180ce800 0x7fde18159650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:09.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.512+0000 7fde16d9d640 1 -- 172.21.15.88:0/3649977367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde040039b0 msgr2=0x7fde18158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:09.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.512+0000 7fde16d9d640 1 --2- 172.21.15.88:0/3649977367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde040039b0 0x7fde18158040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:09.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.512+0000 7fde16d9d640 1 -- 172.21.15.88:0/3649977367 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde18159b90 con 0x7fde1815dc70 2024-09-16T01:41:09.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.512+0000 7fde16d9d640 1 --2- 172.21.15.88:0/3649977367 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde1815dc70 0x7fde1815e0f0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fde180b6840 tx=0x7fde1000e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:09.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.512+0000 7fde0f7fe640 1 -- 172.21.15.88:0/3649977367 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde100135f0 con 0x7fde1815dc70 2024-09-16T01:41:09.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.512+0000 7fde0f7fe640 1 -- 172.21.15.88:0/3649977367 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde10013790 con 0x7fde1815dc70 2024-09-16T01:41:09.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.512+0000 7fde1c817640 1 -- 172.21.15.88:0/3649977367 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde18159e60 con 0x7fde1815dc70 2024-09-16T01:41:09.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.512+0000 7fde0f7fe640 1 -- 172.21.15.88:0/3649977367 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde10013980 con 0x7fde1815dc70 2024-09-16T01:41:09.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.512+0000 7fde1c817640 1 -- 172.21.15.88:0/3649977367 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde1815a420 con 0x7fde1815dc70 2024-09-16T01:41:09.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.513+0000 7fde1c817640 1 -- 172.21.15.88:0/3649977367 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde1815f9e0 con 0x7fde1815dc70 2024-09-16T01:41:09.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.514+0000 7fde0f7fe640 1 -- 172.21.15.88:0/3649977367 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fde1001c020 con 0x7fde1815dc70 2024-09-16T01:41:09.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.514+0000 7fde0f7fe640 1 --2- 172.21.15.88:0/3649977367 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fddf806c2b0 0x7fddf806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:09.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.514+0000 7fde0f7fe640 1 -- 172.21.15.88:0/3649977367 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6340+0+0 (secure 0 0 0) 0x7fde1008b440 con 0x7fde1815dc70 2024-09-16T01:41:09.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.514+0000 7fde1659c640 1 --2- 172.21.15.88:0/3649977367 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fddf806c2b0 0x7fddf806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:09.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.515+0000 7fde1659c640 1 --2- 172.21.15.88:0/3649977367 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fddf806c2b0 0x7fddf806e770 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fde040057e0 tx=0x7fde08030000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:09.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.519+0000 7fde0f7fe640 1 -- 172.21.15.88:0/3649977367 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fde10059370 con 0x7fde1815dc70 2024-09-16T01:41:09.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.697+0000 7fde1c817640 1 -- 172.21.15.88:0/3649977367 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fde180b29d0 con 0x7fde1815dc70 2024-09-16T01:41:09.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.698+0000 7fde0f7fe640 1 -- 172.21.15.88:0/3649977367 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 14 entries v94) ==== 81+0+854 (secure 0 0 0) 0x7fde1005d670 con 0x7fde1815dc70 2024-09-16T01:41:09.700 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/1042715824 2024-09-17T01:40:59.080515+0000 2024-09-16T01:41:09.700 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/4162274767 2024-09-17T01:40:11.522091+0000 2024-09-16T01:41:09.700 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2733458327 2024-09-17T01:40:27.314712+0000 2024-09-16T01:41:09.701 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2970185561 2024-09-17T01:40:38.255774+0000 2024-09-16T01:41:09.701 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/1289932173 2024-09-17T01:40:42.399399+0000 2024-09-16T01:41:09.701 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/3356432718 2024-09-17T01:40:32.344252+0000 2024-09-16T01:41:09.701 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/1042715824 2024-09-17T01:40:59.080515+0000 2024-09-16T01:41:09.701 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/3356432718 2024-09-17T01:40:32.344252+0000 2024-09-16T01:41:09.701 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2970185561 2024-09-17T01:40:38.255774+0000 2024-09-16T01:41:09.702 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2408579939 2024-09-17T01:40:46.434492+0000 2024-09-16T01:41:09.702 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/4162274767 2024-09-17T01:40:11.522091+0000 2024-09-16T01:41:09.702 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/1289932173 2024-09-17T01:40:42.399399+0000 2024-09-16T01:41:09.702 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2733458327 2024-09-17T01:40:27.314712+0000 2024-09-16T01:41:09.702 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2408579939 2024-09-17T01:40:46.434492+0000 2024-09-16T01:41:09.703 INFO:teuthology.orchestra.run.smithi088.stderr:listed 14 entries 2024-09-16T01:41:09.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.700+0000 7fde1c817640 1 -- 172.21.15.88:0/3649977367 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fddf806c2b0 msgr2=0x7fddf806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:09.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.700+0000 7fde1c817640 1 --2- 172.21.15.88:0/3649977367 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fddf806c2b0 0x7fddf806e770 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fde040057e0 tx=0x7fde08030000 comp rx=0 tx=0).stop 2024-09-16T01:41:09.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.700+0000 7fde1c817640 1 -- 172.21.15.88:0/3649977367 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde1815dc70 msgr2=0x7fde1815e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:09.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.700+0000 7fde1c817640 1 --2- 172.21.15.88:0/3649977367 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde1815dc70 0x7fde1815e0f0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fde180b6840 tx=0x7fde1000e9e0 comp rx=0 tx=0).stop 2024-09-16T01:41:09.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.700+0000 7fde1c817640 1 -- 172.21.15.88:0/3649977367 shutdown_connections 2024-09-16T01:41:09.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.700+0000 7fde1c817640 1 --2- 172.21.15.88:0/3649977367 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fddf806c2b0 0x7fddf806e770 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:09.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.700+0000 7fde1c817640 1 --2- 172.21.15.88:0/3649977367 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde1815dc70 0x7fde1815e0f0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:09.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.700+0000 7fde1c817640 1 --2- 172.21.15.88:0/3649977367 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde180ce800 0x7fde18159650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:09.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.700+0000 7fde1c817640 1 --2- 172.21.15.88:0/3649977367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde040039b0 0x7fde18158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:09.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.700+0000 7fde1c817640 1 -- 172.21.15.88:0/3649977367 >> 172.21.15.88:0/3649977367 conn(0x7fde180b3070 msgr2=0x7fde180ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:09.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.700+0000 7fde1c817640 1 -- 172.21.15.88:0/3649977367 shutdown_connections 2024-09-16T01:41:09.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.700+0000 7fde1c817640 1 -- 172.21.15.88:0/3649977367 wait complete. 2024-09-16T01:41:09.713 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-16T01:41:09.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:09.713 DEBUG:teuthology.orchestra.run.smithi088:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-09-16T01:41:09.755 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:41:09.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:09.755 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:41:09.811 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:41:09.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.948+0000 7f44fc05e640 1 Processor -- start 2024-09-16T01:41:09.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.949+0000 7f44fc05e640 1 -- start start 2024-09-16T01:41:09.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.949+0000 7f44fc05e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f44f415e7a0 0x7f44f406caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:09.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.949+0000 7f44fc05e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44f40cf1f0 0x7f44f40b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:09.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.949+0000 7f44fc05e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44f40cf860 con 0x7f44f40b6720 2024-09-16T01:41:09.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.949+0000 7f44fc05e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f44f40d2ca0 con 0x7f44f415e7a0 2024-09-16T01:41:09.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.949+0000 7f44fc05e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f44f40cfb40 con 0x7f44f40cf1f0 2024-09-16T01:41:09.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.950+0000 7f44fa5fb640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f44f40b6720 0x7f44f41623c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:47980/0 (socket says 172.21.15.88:47980) 2024-09-16T01:41:09.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.950+0000 7f44fa5fb640 1 -- 172.21.15.88:0/4054660902 learned_addr learned my addr 172.21.15.88:0/4054660902 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:09.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.950+0000 7f44f9dfa640 1 --2- 172.21.15.88:0/4054660902 >> v2:172.21.15.154:3300/0 conn(0x7f44f415e7a0 0x7f44f406caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:09.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.950+0000 7f44f95f9640 1 --2- 172.21.15.88:0/4054660902 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44f40cf1f0 0x7f44f40b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:09.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.950+0000 7f44f8df8640 1 -- 172.21.15.88:0/4054660902 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3765322082 0 0) 0x7f44f40cf860 con 0x7f44f40b6720 2024-09-16T01:41:09.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.950+0000 7f44f8df8640 1 -- 172.21.15.88:0/4054660902 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44dc003520 con 0x7f44f40b6720 2024-09-16T01:41:09.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.950+0000 7f44f8df8640 1 -- 172.21.15.88:0/4054660902 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2079706448 0 0) 0x7f44dc003520 con 0x7f44f40b6720 2024-09-16T01:41:09.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.950+0000 7f44f8df8640 1 -- 172.21.15.88:0/4054660902 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44f40cf1f0 msgr2=0x7f44f40b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:09.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.950+0000 7f44f8df8640 1 --2- 172.21.15.88:0/4054660902 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44f40cf1f0 0x7f44f40b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:09.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.950+0000 7f44f8df8640 1 -- 172.21.15.88:0/4054660902 >> v2:172.21.15.154:3300/0 conn(0x7f44f415e7a0 msgr2=0x7f44f406caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:09.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.950+0000 7f44f8df8640 1 --2- 172.21.15.88:0/4054660902 >> v2:172.21.15.154:3300/0 conn(0x7f44f415e7a0 0x7f44f406caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:09.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.950+0000 7f44f8df8640 1 -- 172.21.15.88:0/4054660902 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f40cfe20 con 0x7f44f40b6720 2024-09-16T01:41:09.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.950+0000 7f44f8df8640 1 -- 172.21.15.88:0/4054660902 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f44ec001670 con 0x7f44f40b6720 2024-09-16T01:41:09.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.950+0000 7f44f8df8640 1 -- 172.21.15.88:0/4054660902 >> v1:172.21.15.88:6789/0 conn(0x7f44f40b6720 legacy=0x7f44f41623c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:09.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.950+0000 7f44f8df8640 1 --2- 172.21.15.88:0/4054660902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44dc0039b0 0x7f44dc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:09.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.950+0000 7f44f8df8640 1 --2- 172.21.15.88:0/4054660902 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44f40cf1f0 0x7f44dc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:09.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.950+0000 7f44f8df8640 1 --2- 172.21.15.88:0/4054660902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44dc004d40 0x7f44dc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:09.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.950+0000 7f44f8df8640 1 -- 172.21.15.88:0/4054660902 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f44f40d2ca0 con 0x7f44dc0039b0 2024-09-16T01:41:09.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.950+0000 7f44f8df8640 1 -- 172.21.15.88:0/4054660902 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f44f40cfb40 con 0x7f44dc004d40 2024-09-16T01:41:09.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.950+0000 7f44f8df8640 1 -- 172.21.15.88:0/4054660902 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f44dc00b9c0 con 0x7f44f40cf1f0 2024-09-16T01:41:09.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.950+0000 7f44f9dfa640 1 --2- 172.21.15.88:0/4054660902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44dc0039b0 0x7f44dc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:09.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.950+0000 7f44fa5fb640 1 --2- 172.21.15.88:0/4054660902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44dc004d40 0x7f44dc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:09.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.950+0000 7f44f95f9640 1 --2- 172.21.15.88:0/4054660902 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44f40cf1f0 0x7f44dc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:09.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.951+0000 7f44f9dfa640 1 -- 172.21.15.88:0/4054660902 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44f40cf1f0 msgr2=0x7f44dc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:09.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.951+0000 7f44f9dfa640 1 --2- 172.21.15.88:0/4054660902 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44f40cf1f0 0x7f44dc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:09.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.951+0000 7f44f9dfa640 1 -- 172.21.15.88:0/4054660902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44dc004d40 msgr2=0x7f44dc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:09.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.951+0000 7f44f9dfa640 1 --2- 172.21.15.88:0/4054660902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44dc004d40 0x7f44dc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:09.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.951+0000 7f44f9dfa640 1 -- 172.21.15.88:0/4054660902 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44dc003520 con 0x7f44dc0039b0 2024-09-16T01:41:09.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.951+0000 7f44f9dfa640 1 --2- 172.21.15.88:0/4054660902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44dc0039b0 0x7f44dc003dd0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f44e4009fc0 tx=0x7f44dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a544057d947b857d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:09.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.951+0000 7f44f8df8640 1 -- 172.21.15.88:0/4054660902 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e4009c10 con 0x7f44dc0039b0 2024-09-16T01:41:09.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.951+0000 7f44f8df8640 1 -- 172.21.15.88:0/4054660902 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44e4036070 con 0x7f44dc0039b0 2024-09-16T01:41:09.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.951+0000 7f44f8df8640 1 -- 172.21.15.88:0/4054660902 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e402d070 con 0x7f44dc0039b0 2024-09-16T01:41:09.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.952+0000 7f44fc05e640 1 -- 172.21.15.88:0/4054660902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44dc0039b0 msgr2=0x7f44dc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:09.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.952+0000 7f44fc05e640 1 --2- 172.21.15.88:0/4054660902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44dc0039b0 0x7f44dc003dd0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f44e4009fc0 tx=0x7f44dc005830 comp rx=0 tx=0).stop 2024-09-16T01:41:09.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.952+0000 7f44fa5fb640 1 -- 172.21.15.88:0/4054660902 reap_dead start 2024-09-16T01:41:09.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.952+0000 7f44fc05e640 1 -- 172.21.15.88:0/4054660902 shutdown_connections 2024-09-16T01:41:09.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.952+0000 7f44fc05e640 1 -- 172.21.15.88:0/4054660902 >> 172.21.15.88:0/4054660902 conn(0x7f44f40d11e0 msgr2=0x7f44f40d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:09.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.952+0000 7f44fc05e640 1 -- 172.21.15.88:0/4054660902 shutdown_connections 2024-09-16T01:41:09.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.952+0000 7f44fc05e640 1 -- 172.21.15.88:0/4054660902 wait complete. 2024-09-16T01:41:09.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.953+0000 7f44fc05e640 1 Processor -- start 2024-09-16T01:41:09.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.953+0000 7f44fc05e640 1 -- start start 2024-09-16T01:41:09.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.954+0000 7f44fc05e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44dc0039b0 0x7f44f414f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:09.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.954+0000 7f44fc05e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44f40cd530 0x7f44f40cd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:09.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.954+0000 7f44fc05e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44f415abc0 0x7f44f415b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:09.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.954+0000 7f44fc05e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f44f406cfe0 con 0x7f44dc0039b0 2024-09-16T01:41:09.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.954+0000 7f44fc05e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f44f40b7a00 con 0x7f44f415abc0 2024-09-16T01:41:09.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.954+0000 7f44fc05e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f44f40d4420 con 0x7f44f40cd530 2024-09-16T01:41:09.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.954+0000 7f44f9dfa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44dc0039b0 0x7f44f414f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:09.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.954+0000 7f44f9dfa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44dc0039b0 0x7f44f414f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55916/0 (socket says 172.21.15.88:55916) 2024-09-16T01:41:09.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.954+0000 7f44f9dfa640 1 -- 172.21.15.88:0/386654828 learned_addr learned my addr 172.21.15.88:0/386654828 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:09.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.954+0000 7f44fa5fb640 1 --2- 172.21.15.88:0/386654828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44f415abc0 0x7f44f415b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:09.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.954+0000 7f44f95f9640 1 --2- 172.21.15.88:0/386654828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44f40cd530 0x7f44f40cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:09.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.954+0000 7f44f9dfa640 1 -- 172.21.15.88:0/386654828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44f40cd530 msgr2=0x7f44f40cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:09.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.954+0000 7f44f9dfa640 1 --2- 172.21.15.88:0/386654828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44f40cd530 0x7f44f40cd990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:09.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.954+0000 7f44f9dfa640 1 -- 172.21.15.88:0/386654828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44f415abc0 msgr2=0x7f44f415b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:09.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.954+0000 7f44f9dfa640 1 --2- 172.21.15.88:0/386654828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44f415abc0 0x7f44f415b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:09.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.954+0000 7f44f9dfa640 1 -- 172.21.15.88:0/386654828 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f415ef30 con 0x7f44dc0039b0 2024-09-16T01:41:09.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.954+0000 7f44f9dfa640 1 --2- 172.21.15.88:0/386654828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44dc0039b0 0x7f44f414f910 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f44e402af10 tx=0x7f44e4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:09.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.955+0000 7f44f27fc640 1 -- 172.21.15.88:0/386654828 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e4009c10 con 0x7f44dc0039b0 2024-09-16T01:41:09.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.955+0000 7f44f27fc640 1 -- 172.21.15.88:0/386654828 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44e4003bb0 con 0x7f44dc0039b0 2024-09-16T01:41:09.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.955+0000 7f44f27fc640 1 -- 172.21.15.88:0/386654828 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e4004020 con 0x7f44dc0039b0 2024-09-16T01:41:09.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.955+0000 7f44fc05e640 1 -- 172.21.15.88:0/386654828 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44f415f1d0 con 0x7f44dc0039b0 2024-09-16T01:41:09.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.955+0000 7f44fc05e640 1 -- 172.21.15.88:0/386654828 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44f415f790 con 0x7f44dc0039b0 2024-09-16T01:41:09.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.956+0000 7f44fc05e640 1 -- 172.21.15.88:0/386654828 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44f415b580 con 0x7f44dc0039b0 2024-09-16T01:41:09.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.956+0000 7f44f27fc640 1 -- 172.21.15.88:0/386654828 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f44e4005d10 con 0x7f44dc0039b0 2024-09-16T01:41:09.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.956+0000 7f44f27fc640 1 --2- 172.21.15.88:0/386654828 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f44e006c2e0 0x7f44e006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:09.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.957+0000 7f44f95f9640 1 --2- 172.21.15.88:0/386654828 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f44e006c2e0 0x7f44e006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:09.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.957+0000 7f44f27fc640 1 -- 172.21.15.88:0/386654828 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 6340+0+0 (secure 0 0 0) 0x7f44e40acd20 con 0x7f44dc0039b0 2024-09-16T01:41:09.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.957+0000 7f44f95f9640 1 --2- 172.21.15.88:0/386654828 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f44e006c2e0 0x7f44e006e7a0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f44f40cecf0 tx=0x7f44e8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:09.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:09.961+0000 7f44f27fc640 1 -- 172.21.15.88:0/386654828 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f44e407b330 con 0x7f44dc0039b0 2024-09-16T01:41:10.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.162+0000 7f44fc05e640 1 -- 172.21.15.88:0/386654828 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f44f40d2a80 con 0x7f44dc0039b0 2024-09-16T01:41:10.164 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:41:10.163+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:41:10.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.164+0000 7f44f27fc640 1 -- 172.21.15.88:0/386654828 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v118) ==== 161+0+0 (secure 0 0 0) 0x7f44e4043e40 con 0x7f44dc0039b0 2024-09-16T01:41:10.166 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:41:10.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.167+0000 7f44fc05e640 1 -- 172.21.15.88:0/386654828 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f44e006c2e0 msgr2=0x7f44e006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:10.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.167+0000 7f44fc05e640 1 --2- 172.21.15.88:0/386654828 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f44e006c2e0 0x7f44e006e7a0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f44f40cecf0 tx=0x7f44e8008040 comp rx=0 tx=0).stop 2024-09-16T01:41:10.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.167+0000 7f44fc05e640 1 -- 172.21.15.88:0/386654828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44dc0039b0 msgr2=0x7f44f414f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:10.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.167+0000 7f44fc05e640 1 --2- 172.21.15.88:0/386654828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44dc0039b0 0x7f44f414f910 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f44e402af10 tx=0x7f44e4009750 comp rx=0 tx=0).stop 2024-09-16T01:41:10.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.167+0000 7f44fc05e640 1 -- 172.21.15.88:0/386654828 shutdown_connections 2024-09-16T01:41:10.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.167+0000 7f44fc05e640 1 --2- 172.21.15.88:0/386654828 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f44e006c2e0 0x7f44e006e7a0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:10.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.167+0000 7f44fc05e640 1 --2- 172.21.15.88:0/386654828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44f415abc0 0x7f44f415b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:10.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.167+0000 7f44fc05e640 1 --2- 172.21.15.88:0/386654828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44f40cd530 0x7f44f40cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:10.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.167+0000 7f44fc05e640 1 --2- 172.21.15.88:0/386654828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44dc0039b0 0x7f44f414f910 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:10.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.167+0000 7f44fc05e640 1 -- 172.21.15.88:0/386654828 >> 172.21.15.88:0/386654828 conn(0x7f44f40d11e0 msgr2=0x7f44f4159d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:10.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.167+0000 7f44fc05e640 1 -- 172.21.15.88:0/386654828 shutdown_connections 2024-09-16T01:41:10.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.167+0000 7f44fc05e640 1 -- 172.21.15.88:0/386654828 wait complete. 2024-09-16T01:41:10.170 INFO:tasks.ceph.mds.a.smithi088.stderr: -667> 2024-09-16T01:41:05.089+0000 7f7610823640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:41:10.180 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:41:10.195 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:41:10.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.280+0000 7fc4775f8640 1 Processor -- start 2024-09-16T01:41:10.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.280+0000 7fc4775f8640 1 -- start start 2024-09-16T01:41:10.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.281+0000 7fc4775f8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc47015b480 0x7fc4700b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:10.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.281+0000 7fc4775f8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc470072700 0x7fc47015bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:10.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.281+0000 7fc4775f8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4700ccff0 con 0x7fc4700b60c0 2024-09-16T01:41:10.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.281+0000 7fc4775f8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc4700cd2d0 con 0x7fc470072700 2024-09-16T01:41:10.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.281+0000 7fc4775f8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc4700cd5b0 con 0x7fc47015b480 2024-09-16T01:41:10.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.281+0000 7fc475b95640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc470072700 0x7fc47015bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:10.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.281+0000 7fc475b95640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc470072700 0x7fc47015bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55936/0 (socket says 172.21.15.88:55936) 2024-09-16T01:41:10.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.281+0000 7fc475b95640 1 -- 172.21.15.88:0/3069066877 learned_addr learned my addr 172.21.15.88:0/3069066877 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:10.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.281+0000 7fc475b95640 1 -- 172.21.15.88:0/3069066877 >> v2:172.21.15.154:3300/0 conn(0x7fc47015b480 msgr2=0x7fc4700b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:10.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.281+0000 7fc475394640 1 --2- 172.21.15.88:0/3069066877 >> v2:172.21.15.154:3300/0 conn(0x7fc47015b480 0x7fc4700b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:10.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.281+0000 7fc475b95640 1 --2- 172.21.15.88:0/3069066877 >> v2:172.21.15.154:3300/0 conn(0x7fc47015b480 0x7fc4700b5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:10.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.281+0000 7fc475b95640 1 -- 172.21.15.88:0/3069066877 >> v1:172.21.15.154:6789/0 conn(0x7fc4700b60c0 legacy=0x7fc470071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:10.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.281+0000 7fc475b95640 1 -- 172.21.15.88:0/3069066877 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4700cd890 con 0x7fc470072700 2024-09-16T01:41:10.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.282+0000 7fc475b95640 1 --2- 172.21.15.88:0/3069066877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc470072700 0x7fc47015bc60 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fc4700694d0 tx=0x7fc46802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=575cb3a4b0babce8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.282+0000 7fc467fff640 1 -- 172.21.15.88:0/3069066877 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc468036070 con 0x7fc470072700 2024-09-16T01:41:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.282+0000 7fc467fff640 1 -- 172.21.15.88:0/3069066877 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc468031600 con 0x7fc470072700 2024-09-16T01:41:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.282+0000 7fc467fff640 1 -- 172.21.15.88:0/3069066877 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4680319d0 con 0x7fc470072700 2024-09-16T01:41:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.282+0000 7fc4775f8640 1 -- 172.21.15.88:0/3069066877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc470072700 msgr2=0x7fc47015bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.282+0000 7fc4775f8640 1 --2- 172.21.15.88:0/3069066877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc470072700 0x7fc47015bc60 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fc4700694d0 tx=0x7fc46802a660 comp rx=0 tx=0).stop 2024-09-16T01:41:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.283+0000 7fc4775f8640 1 -- 172.21.15.88:0/3069066877 shutdown_connections 2024-09-16T01:41:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.283+0000 7fc4775f8640 1 --2- 172.21.15.88:0/3069066877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc470072700 0x7fc47015bc60 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.283+0000 7fc4775f8640 1 --2- 172.21.15.88:0/3069066877 >> v2:172.21.15.154:3300/0 conn(0x7fc47015b480 0x7fc4700b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.283+0000 7fc4775f8640 1 -- 172.21.15.88:0/3069066877 >> 172.21.15.88:0/3069066877 conn(0x7fc47006d0f0 msgr2=0x7fc47006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.283+0000 7fc4775f8640 1 -- 172.21.15.88:0/3069066877 shutdown_connections 2024-09-16T01:41:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.283+0000 7fc4775f8640 1 -- 172.21.15.88:0/3069066877 wait complete. 2024-09-16T01:41:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.284+0000 7fc4775f8640 1 Processor -- start 2024-09-16T01:41:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.284+0000 7fc4775f8640 1 -- start start 2024-09-16T01:41:10.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.284+0000 7fc4775f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc470072700 0x7fc47014d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:10.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.284+0000 7fc4775f8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4700b60c0 0x7fc4701e52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:10.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.284+0000 7fc4775f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc47015b480 0x7fc4701e57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.284+0000 7fc4775f8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc47015b9e0 con 0x7fc4700b60c0 2024-09-16T01:41:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.284+0000 7fc4775f8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc47015b860 con 0x7fc47015b480 2024-09-16T01:41:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.284+0000 7fc4775f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc47015c1a0 con 0x7fc470072700 2024-09-16T01:41:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.285+0000 7fc474b93640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4700b60c0 0x7fc4701e52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.285+0000 7fc474b93640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4700b60c0 0x7fc4701e52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55940/0 (socket says 172.21.15.88:55940) 2024-09-16T01:41:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.285+0000 7fc474b93640 1 -- 172.21.15.88:0/1517050902 learned_addr learned my addr 172.21.15.88:0/1517050902 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.285+0000 7fc475394640 1 --2- 172.21.15.88:0/1517050902 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc470072700 0x7fc47014d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.285+0000 7fc475b95640 1 --2- 172.21.15.88:0/1517050902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc47015b480 0x7fc4701e57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.285+0000 7fc474b93640 1 -- 172.21.15.88:0/1517050902 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc470072700 msgr2=0x7fc47014d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.285+0000 7fc474b93640 1 --2- 172.21.15.88:0/1517050902 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc470072700 0x7fc47014d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.285+0000 7fc474b93640 1 -- 172.21.15.88:0/1517050902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc47015b480 msgr2=0x7fc4701e57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.285+0000 7fc474b93640 1 --2- 172.21.15.88:0/1517050902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc47015b480 0x7fc4701e57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.285+0000 7fc474b93640 1 -- 172.21.15.88:0/1517050902 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc47014d5c0 con 0x7fc4700b60c0 2024-09-16T01:41:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.285+0000 7fc475394640 1 --2- 172.21.15.88:0/1517050902 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc470072700 0x7fc47014d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.285+0000 7fc474b93640 1 --2- 172.21.15.88:0/1517050902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4700b60c0 0x7fc4701e52b0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fc46000c410 tx=0x7fc46000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.285+0000 7fc465ffb640 1 -- 172.21.15.88:0/1517050902 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc460017070 con 0x7fc4700b60c0 2024-09-16T01:41:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.285+0000 7fc465ffb640 1 -- 172.21.15.88:0/1517050902 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4600035f0 con 0x7fc4700b60c0 2024-09-16T01:41:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.285+0000 7fc4775f8640 1 -- 172.21.15.88:0/1517050902 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4701e4210 con 0x7fc4700b60c0 2024-09-16T01:41:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.285+0000 7fc465ffb640 1 -- 172.21.15.88:0/1517050902 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc460003a20 con 0x7fc4700b60c0 2024-09-16T01:41:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.285+0000 7fc4775f8640 1 -- 172.21.15.88:0/1517050902 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4701e47d0 con 0x7fc4700b60c0 2024-09-16T01:41:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.286+0000 7fc4775f8640 1 -- 172.21.15.88:0/1517050902 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc47015bc60 con 0x7fc4700b60c0 2024-09-16T01:41:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.287+0000 7fc465ffb640 1 -- 172.21.15.88:0/1517050902 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc46001a070 con 0x7fc4700b60c0 2024-09-16T01:41:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.287+0000 7fc465ffb640 1 --2- 172.21.15.88:0/1517050902 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc45406c2b0 0x7fc45406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.287+0000 7fc465ffb640 1 -- 172.21.15.88:0/1517050902 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6426+0+0 (secure 0 0 0) 0x7fc46008aaf0 con 0x7fc4700b60c0 2024-09-16T01:41:10.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.287+0000 7fc475394640 1 --2- 172.21.15.88:0/1517050902 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc45406c2b0 0x7fc45406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:10.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.288+0000 7fc475394640 1 --2- 172.21.15.88:0/1517050902 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc45406c2b0 0x7fc45406e770 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fc46c006fd0 tx=0x7fc46c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:10.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.291+0000 7fc465ffb640 1 -- 172.21.15.88:0/1517050902 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc47014d5c0 con 0x7fc4700b60c0 2024-09-16T01:41:10.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:10.520+0000 7fc4775f8640 1 -- 172.21.15.88:0/1517050902 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fc470063360 con 0x7fc4700b60c0 2024-09-16T01:41:11.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.169+0000 7fc465ffb640 1 -- 172.21.15.88:0/1517050902 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v119) ==== 107+0+0 (secure 0 0 0) 0x7fc460058940 con 0x7fc4700b60c0 2024-09-16T01:41:11.171 INFO:teuthology.orchestra.run.smithi088.stderr:nothing to do: rank is not damaged 2024-09-16T01:41:11.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.172+0000 7fc4775f8640 1 -- 172.21.15.88:0/1517050902 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc45406c2b0 msgr2=0x7fc45406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:11.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.172+0000 7fc4775f8640 1 --2- 172.21.15.88:0/1517050902 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc45406c2b0 0x7fc45406e770 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fc46c006fd0 tx=0x7fc46c008040 comp rx=0 tx=0).stop 2024-09-16T01:41:11.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.172+0000 7fc4775f8640 1 -- 172.21.15.88:0/1517050902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4700b60c0 msgr2=0x7fc4701e52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:11.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.172+0000 7fc4775f8640 1 --2- 172.21.15.88:0/1517050902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4700b60c0 0x7fc4701e52b0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fc46000c410 tx=0x7fc46000c8d0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.172+0000 7fc4775f8640 1 -- 172.21.15.88:0/1517050902 shutdown_connections 2024-09-16T01:41:11.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.172+0000 7fc4775f8640 1 --2- 172.21.15.88:0/1517050902 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc45406c2b0 0x7fc45406e770 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.172+0000 7fc4775f8640 1 --2- 172.21.15.88:0/1517050902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc47015b480 0x7fc4701e57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.172+0000 7fc4775f8640 1 --2- 172.21.15.88:0/1517050902 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4700b60c0 0x7fc4701e52b0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.172+0000 7fc4775f8640 1 --2- 172.21.15.88:0/1517050902 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc470072700 0x7fc47014d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.172+0000 7fc4775f8640 1 -- 172.21.15.88:0/1517050902 >> 172.21.15.88:0/1517050902 conn(0x7fc47006d0f0 msgr2=0x7fc4701588c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:11.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.172+0000 7fc4775f8640 1 -- 172.21.15.88:0/1517050902 shutdown_connections 2024-09-16T01:41:11.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.172+0000 7fc4775f8640 1 -- 172.21.15.88:0/1517050902 wait complete. 2024-09-16T01:41:11.187 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:41:11.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.273+0000 7f07808fef40 1 Processor -- start 2024-09-16T01:41:11.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.273+0000 7f07808fef40 1 -- start start 2024-09-16T01:41:11.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.274+0000 7f07808fef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ceb8f3dd0 0x564ceb93b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.274+0000 7f07808fef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564ceb8f6230 0x564ceb93a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.274+0000 7f07808fef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564ceb9c2540 0x564ceb93a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.274+0000 7f07808fef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564ceb93d5b0 con 0x564ceb8f6230 2024-09-16T01:41:11.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.274+0000 7f07808fef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564ceb844510 con 0x564ceb8f3dd0 2024-09-16T01:41:11.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.274+0000 7f07808fef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x564ceb8f41e0 con 0x564ceb9c2540 2024-09-16T01:41:11.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.274+0000 7f0775bf3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564ceb8f6230 0x564ceb93a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.274+0000 7f0775bf3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564ceb8f6230 0x564ceb93a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41278/0 (socket says 172.21.15.88:41278) 2024-09-16T01:41:11.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.274+0000 7f0775bf3640 1 -- 172.21.15.88:0/4163159584 learned_addr learned my addr 172.21.15.88:0/4163159584 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:11.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.274+0000 7f077e3f4640 1 --2- 172.21.15.88:0/4163159584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ceb8f3dd0 0x564ceb93b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.274+0000 7f077ebf5640 1 --2- 172.21.15.88:0/4163159584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564ceb9c2540 0x564ceb93a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.275+0000 7f0775bf3640 1 -- 172.21.15.88:0/4163159584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564ceb9c2540 msgr2=0x564ceb93a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:11.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.275+0000 7f0775bf3640 1 --2- 172.21.15.88:0/4163159584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564ceb9c2540 0x564ceb93a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.275+0000 7f0775bf3640 1 -- 172.21.15.88:0/4163159584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ceb8f3dd0 msgr2=0x564ceb93b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:11.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.275+0000 7f0775bf3640 1 --2- 172.21.15.88:0/4163159584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ceb8f3dd0 0x564ceb93b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.275+0000 7f0775bf3640 1 -- 172.21.15.88:0/4163159584 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ceb8f20a0 con 0x564ceb8f6230 2024-09-16T01:41:11.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.275+0000 7f077ebf5640 1 --2- 172.21.15.88:0/4163159584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564ceb9c2540 0x564ceb93a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:11.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.275+0000 7f0775bf3640 1 --2- 172.21.15.88:0/4163159584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564ceb8f6230 0x564ceb93a210 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f0768000c00 tx=0x7f0768051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec7a32fee3919e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:11.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.275+0000 7f077d3f2640 1 -- 172.21.15.88:0/4163159584 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0768063070 con 0x564ceb8f6230 2024-09-16T01:41:11.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.275+0000 7f077d3f2640 1 -- 172.21.15.88:0/4163159584 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0768055b20 con 0x564ceb8f6230 2024-09-16T01:41:11.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.275+0000 7f077d3f2640 1 -- 172.21.15.88:0/4163159584 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0768068460 con 0x564ceb8f6230 2024-09-16T01:41:11.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.275+0000 7f07808fef40 1 -- 172.21.15.88:0/4163159584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564ceb8f6230 msgr2=0x564ceb93a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:11.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.275+0000 7f07808fef40 1 --2- 172.21.15.88:0/4163159584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564ceb8f6230 0x564ceb93a210 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f0768000c00 tx=0x7f0768051000 comp rx=0 tx=0).stop 2024-09-16T01:41:11.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.276+0000 7f07808fef40 1 -- 172.21.15.88:0/4163159584 shutdown_connections 2024-09-16T01:41:11.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.276+0000 7f07808fef40 1 --2- 172.21.15.88:0/4163159584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564ceb9c2540 0x564ceb93a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.276+0000 7f07808fef40 1 --2- 172.21.15.88:0/4163159584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564ceb8f6230 0x564ceb93a210 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.276+0000 7f07808fef40 1 --2- 172.21.15.88:0/4163159584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ceb8f3dd0 0x564ceb93b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.276+0000 7f07808fef40 1 -- 172.21.15.88:0/4163159584 >> 172.21.15.88:0/4163159584 conn(0x564ceb93d750 msgr2=0x564ceb9caea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:11.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.276+0000 7f07808fef40 1 -- 172.21.15.88:0/4163159584 shutdown_connections 2024-09-16T01:41:11.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.276+0000 7f07808fef40 1 -- 172.21.15.88:0/4163159584 wait complete. 2024-09-16T01:41:11.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.276+0000 7f07808fef40 1 Processor -- start 2024-09-16T01:41:11.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.276+0000 7f07808fef40 1 -- start start 2024-09-16T01:41:11.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.277+0000 7f07808fef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564ceb8f3dd0 0x564ceb9ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.277+0000 7f07808fef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564ceb8f6230 0x564ceb9cab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.277+0000 7f07808fef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ceb9c2540 0x564ceb9c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.277+0000 7f07808fef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564ceb8f28c0 con 0x564ceb8f3dd0 2024-09-16T01:41:11.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.277+0000 7f07808fef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564ceb8f1e50 con 0x564ceb9c2540 2024-09-16T01:41:11.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.277+0000 7f07808fef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x564ceb8f2b10 con 0x564ceb8f6230 2024-09-16T01:41:11.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.277+0000 7f077e3f4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564ceb8f3dd0 0x564ceb9ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.277+0000 7f077e3f4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564ceb8f3dd0 0x564ceb9ca5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41288/0 (socket says 172.21.15.88:41288) 2024-09-16T01:41:11.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.277+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 learned_addr learned my addr 172.21.15.88:0/3710798792 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:11.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.277+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564ceb8f6230 msgr2=0x564ceb9cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:11.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.277+0000 7f077e3f4640 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564ceb8f6230 0x564ceb9cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.277+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ceb9c2540 msgr2=0x564ceb9c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:11.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.277+0000 7f077e3f4640 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ceb9c2540 0x564ceb9c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.277+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ceb8f3b30 con 0x564ceb8f3dd0 2024-09-16T01:41:11.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.277+0000 7f077e3f4640 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564ceb8f3dd0 0x564ceb9ca5c0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f077000ee40 tx=0x7f077000c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:11.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.278+0000 7f0777fff640 1 -- 172.21.15.88:0/3710798792 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077000f860 con 0x564ceb8f3dd0 2024-09-16T01:41:11.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.278+0000 7f0777fff640 1 -- 172.21.15.88:0/3710798792 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0770013030 con 0x564ceb8f3dd0 2024-09-16T01:41:11.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.278+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564ceb93c010 con 0x564ceb8f3dd0 2024-09-16T01:41:11.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.278+0000 7f0777fff640 1 -- 172.21.15.88:0/3710798792 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077000fc40 con 0x564ceb8f3dd0 2024-09-16T01:41:11.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.278+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ceb9ba940 con 0x564ceb8f3dd0 2024-09-16T01:41:11.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.279+0000 7f0777fff640 1 -- 172.21.15.88:0/3710798792 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f07700183e0 con 0x564ceb8f3dd0 2024-09-16T01:41:11.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.280+0000 7f0777fff640 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f076006acc0 0x7f076006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.280+0000 7f0777fff640 1 -- 172.21.15.88:0/3710798792 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6426+0+0 (secure 0 0 0) 0x7f077008ad70 con 0x564ceb8f3dd0 2024-09-16T01:41:11.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.280+0000 7f0775bf3640 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f076006acc0 0x7f076006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.285 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:41:11.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.280+0000 7f07808fef40 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x564ceb9cb040 0x564ceb9c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.280+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceb9c0550 con 0x564ceb9cb040 2024-09-16T01:41:11.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.280+0000 7f077ebf5640 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x564ceb9cb040 0x564ceb9c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.280+0000 7f0775bf3640 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f076006acc0 0x7f076006d180 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0768005cf0 tx=0x7f0768005a00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:11.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.280+0000 7f077ebf5640 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x564ceb9cb040 0x564ceb9c98e0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:11.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.282+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v95'226 uv200 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.282+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e95) -- 0x564ceb9bafb0 con 0x564ceb9cb040 2024-09-16T01:41:11.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.284+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v95'227 uv227 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.284+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceb9bb410 con 0x564ceb9cb040 2024-09-16T01:41:11.288 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:41:11.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.285+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v95'228 uv228 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.285+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceb9bb7b0 con 0x564ceb9cb040 2024-09-16T01:41:11.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.286+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v95'229 uv229 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.286+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceb9bbc30 con 0x564ceb9cb040 2024-09-16T01:41:11.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.287+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v95'230 uv230 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.287+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6d490 con 0x564ceb9cb040 2024-09-16T01:41:11.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.288+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v95'231 uv231 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.288+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6d830 con 0x564ceb9cb040 2024-09-16T01:41:11.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.289+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v95'232 uv206 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.289+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6dbd0 con 0x564ceb9cb040 2024-09-16T01:41:11.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.290+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v95'233 uv233 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.290+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6dbd0 con 0x564ceb9cb040 2024-09-16T01:41:11.292 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:41:11.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.291+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v95'234 uv234 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.291+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6ebe0 con 0x564ceb9cb040 2024-09-16T01:41:11.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.293+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v95'235 uv235 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.293+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6ebe0 con 0x564ceb9cb040 2024-09-16T01:41:11.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.294+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v95'236 uv236 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.294+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6dbd0 con 0x564ceb9cb040 2024-09-16T01:41:11.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.295+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v95'237 uv237 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.295+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6dbd0 con 0x564ceb9cb040 2024-09-16T01:41:11.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.297+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v95'238 uv213 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.297+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6dbd0 con 0x564ceb9cb040 2024-09-16T01:41:11.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.298+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v95'239 uv239 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.298+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6dbd0 con 0x564ceb9cb040 2024-09-16T01:41:11.300 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:41:11.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.299+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v95'240 uv240 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.299+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6ebe0 con 0x564ceb9cb040 2024-09-16T01:41:11.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.301+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v95'241 uv241 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.301+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6ebe0 con 0x564ceb9cb040 2024-09-16T01:41:11.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.302+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v95'242 uv242 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.302+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6dbd0 con 0x564ceb9cb040 2024-09-16T01:41:11.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.303+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v95'243 uv243 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.303+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6e330 con 0x564ceb9cb040 2024-09-16T01:41:11.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.304+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v95'244 uv244 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.304+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6e330 con 0x564ceb9cb040 2024-09-16T01:41:11.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.305+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v95'245 uv219 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.305+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6e330 con 0x564ceb9cb040 2024-09-16T01:41:11.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.307+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v95'246 uv246 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.307+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6e330 con 0x564ceb9cb040 2024-09-16T01:41:11.309 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:41:11.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.308+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v95'247 uv247 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.308+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6f340 con 0x564ceb9cb040 2024-09-16T01:41:11.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.309+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v95'248 uv248 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.309+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6f340 con 0x564ceb9cb040 2024-09-16T01:41:11.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.310+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v95'249 uv249 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.310+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6e330 con 0x564ceb9cb040 2024-09-16T01:41:11.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.311+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v95'250 uv250 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.311+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6e330 con 0x564ceb9cb040 2024-09-16T01:41:11.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.313+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v95'251 uv225 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.313+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6e330 con 0x564ceb9cb040 2024-09-16T01:41:11.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.314+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v95'252 uv252 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.314+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6e330 con 0x564ceb9cb040 2024-09-16T01:41:11.316 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:41:11.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.315+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v95'253 uv253 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.315+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6f340 con 0x564ceb9cb040 2024-09-16T01:41:11.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.316+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v95'254 uv254 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.316+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6f340 con 0x564ceb9cb040 2024-09-16T01:41:11.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.317+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v95'255 uv255 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.317+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba6e330 con 0x564ceb9cb040 2024-09-16T01:41:11.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.318+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v95'256 uv256 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.318+0000 7f07808fef40 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x564ceba6e840 0x564ceba70c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.318+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba71170 con 0x564ceba6e840 2024-09-16T01:41:11.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.319+0000 7f077e3f4640 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x564ceba6e840 0x564ceba70c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.319+0000 7f077e3f4640 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x564ceba6e840 0x564ceba70c30 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:11.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.321+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v95'178 uv157 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba6e840 2024-09-16T01:41:11.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.321+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e95) -- 0x564ceba71510 con 0x564ceba6e840 2024-09-16T01:41:11.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.322+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v95'179 uv179 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba6e840 2024-09-16T01:41:11.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.322+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba71950 con 0x564ceba6e840 2024-09-16T01:41:11.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.323+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v95'180 uv180 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba6e840 2024-09-16T01:41:11.324 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:41:11.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.323+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba72210 con 0x564ceba6e840 2024-09-16T01:41:11.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.324+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v95'181 uv181 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba6e840 2024-09-16T01:41:11.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.324+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba71de0 con 0x564ceba6e840 2024-09-16T01:41:11.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.325+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v95'182 uv182 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba6e840 2024-09-16T01:41:11.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.325+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba72a60 con 0x564ceba6e840 2024-09-16T01:41:11.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.327+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v95'183 uv176 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba6e840 2024-09-16T01:41:11.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.327+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e95) -- 0x564ceba72e00 con 0x564ceba6e840 2024-09-16T01:41:11.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.328+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v95'184 uv184 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba6e840 2024-09-16T01:41:11.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.328+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba731a0 con 0x564ceba6e840 2024-09-16T01:41:11.331 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:41:11.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.330+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v95'185 uv185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba6e840 2024-09-16T01:41:11.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.330+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba731a0 con 0x564ceba6e840 2024-09-16T01:41:11.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.331+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v95'186 uv186 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba6e840 2024-09-16T01:41:11.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.331+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba731a0 con 0x564ceba6e840 2024-09-16T01:41:11.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.333+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v95'187 uv187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba6e840 2024-09-16T01:41:11.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.333+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba731a0 con 0x564ceba6e840 2024-09-16T01:41:11.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.334+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v95'188 uv177 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba6e840 2024-09-16T01:41:11.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.334+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e95) -- 0x564ceba731a0 con 0x564ceba6e840 2024-09-16T01:41:11.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.335+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v95'189 uv189 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba6e840 2024-09-16T01:41:11.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.335+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba731a0 con 0x564ceba6e840 2024-09-16T01:41:11.338 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:41:11.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.337+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v95'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba6e840 2024-09-16T01:41:11.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.337+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba77620 con 0x564ceba6e840 2024-09-16T01:41:11.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.338+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v95'191 uv191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba6e840 2024-09-16T01:41:11.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.338+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba77620 con 0x564ceba6e840 2024-09-16T01:41:11.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.339+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v95'192 uv192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba6e840 2024-09-16T01:41:11.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.339+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba731a0 con 0x564ceba6e840 2024-09-16T01:41:11.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.343+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v95'193 uv173 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba6e840 2024-09-16T01:41:11.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.343+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e95) -- 0x564ceba731a0 con 0x564ceba6e840 2024-09-16T01:41:11.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.345+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v95'194 uv194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba6e840 2024-09-16T01:41:11.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.345+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba731a0 con 0x564ceba6e840 2024-09-16T01:41:11.349 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:41:11.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.348+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v95'195 uv195 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba6e840 2024-09-16T01:41:11.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.348+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba741b0 con 0x564ceba6e840 2024-09-16T01:41:11.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.350+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v95'196 uv196 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba6e840 2024-09-16T01:41:11.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.350+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba741b0 con 0x564ceba6e840 2024-09-16T01:41:11.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.353+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v95'197 uv197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba6e840 2024-09-16T01:41:11.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.353+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba731a0 con 0x564ceba6e840 2024-09-16T01:41:11.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.354+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v95'198 uv198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba6e840 2024-09-16T01:41:11.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.354+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba731a0 con 0x564ceb9cb040 2024-09-16T01:41:11.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.355+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v95'45 uv44 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.355+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e95) -- 0x564ceba731a0 con 0x564ceb9cb040 2024-09-16T01:41:11.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.359+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v95'46 uv46 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.359+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba731a0 con 0x564ceb9cb040 2024-09-16T01:41:11.362 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:41:11.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.361+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v95'47 uv47 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.361+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba731a0 con 0x564ceb9cb040 2024-09-16T01:41:11.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.364+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v95'48 uv48 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.364+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba731a0 con 0x564ceb9cb040 2024-09-16T01:41:11.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.365+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v95'49 uv49 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.365+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba731a0 con 0x564ceb9cb040 2024-09-16T01:41:11.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.366+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v95'50 uv50 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.366+0000 7f07808fef40 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x564ceba73870 0x564ceba73ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.366+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba76580 con 0x564ceba73870 2024-09-16T01:41:11.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.366+0000 7f0775bf3640 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x564ceba73870 0x564ceba73ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.366+0000 7f0775bf3640 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x564ceba73870 0x564ceba73ca0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:11.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.369+0000 7f0775bf3640 1 -- 172.21.15.88:0/3710798792 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v95'39 uv38 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f0768076430 con 0x564ceba73870 2024-09-16T01:41:11.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.369+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e95) -- 0x564ceba76920 con 0x564ceba73870 2024-09-16T01:41:11.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.370+0000 7f0775bf3640 1 -- 172.21.15.88:0/3710798792 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v95'40 uv40 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0768076430 con 0x564ceba73870 2024-09-16T01:41:11.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.370+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba76d90 con 0x564ceba73870 2024-09-16T01:41:11.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.371+0000 7f0775bf3640 1 -- 172.21.15.88:0/3710798792 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v95'41 uv41 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0768076430 con 0x564ceba73870 2024-09-16T01:41:11.372 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:41:11.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.371+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba72690 con 0x564ceba73870 2024-09-16T01:41:11.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.372+0000 7f0775bf3640 1 -- 172.21.15.88:0/3710798792 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v95'42 uv42 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0768076430 con 0x564ceba73870 2024-09-16T01:41:11.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.372+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba776c0 con 0x564ceb9cb040 2024-09-16T01:41:11.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.373+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v95'45 uv44 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.373+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e95) -- 0x564ceba776c0 con 0x564ceb9cb040 2024-09-16T01:41:11.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.374+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v95'46 uv46 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.374+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba776c0 con 0x564ceb9cb040 2024-09-16T01:41:11.377 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:41:11.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.376+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v95'47 uv47 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.376+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba786d0 con 0x564ceb9cb040 2024-09-16T01:41:11.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.377+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v95'48 uv48 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.377+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba786d0 con 0x564ceb9cb040 2024-09-16T01:41:11.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.378+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v95'49 uv49 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.378+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba776c0 con 0x564ceb9cb040 2024-09-16T01:41:11.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.379+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v95'50 uv50 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.379+0000 7f07808fef40 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x564ceba77d90 0x564ceba7a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.379+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7a800 con 0x564ceba77d90 2024-09-16T01:41:11.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.379+0000 7f077ebf5640 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x564ceba77d90 0x564ceba7a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.379+0000 7f077ebf5640 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x564ceba77d90 0x564ceba7a1c0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:11.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.381+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v95'83 uv75 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.381+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7a8c0 con 0x564ceba77d90 2024-09-16T01:41:11.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.382+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v95'84 uv84 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.382+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7acb0 con 0x564ceba77d90 2024-09-16T01:41:11.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.382+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v95'85 uv85 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.384 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:41:11.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.383+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7bcc0 con 0x564ceba77d90 2024-09-16T01:41:11.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.383+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v95'86 uv86 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.383+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7bcc0 con 0x564ceba77d90 2024-09-16T01:41:11.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.384+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v95'87 uv87 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.384+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7acb0 con 0x564ceba77d90 2024-09-16T01:41:11.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.386+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v95'88 uv88 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.386+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7acb0 con 0x564ceba77d90 2024-09-16T01:41:11.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.388+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v95'89 uv82 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.388+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7acb0 con 0x564ceba77d90 2024-09-16T01:41:11.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.391+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v95'90 uv90 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.391+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7acb0 con 0x564ceba77d90 2024-09-16T01:41:11.393 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:41:11.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.392+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v95'91 uv91 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.392+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7acb0 con 0x564ceba77d90 2024-09-16T01:41:11.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.395+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v95'92 uv92 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.395+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7acb0 con 0x564ceba77d90 2024-09-16T01:41:11.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.397+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v95'93 uv93 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.398+0000 7f07808fef40 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x564ceba7b1c0 0x564ceba7d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.398+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7dce0 con 0x564ceba7b1c0 2024-09-16T01:41:11.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.398+0000 7f077e3f4640 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x564ceba7b1c0 0x564ceba7d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.398+0000 7f077e3f4640 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x564ceba7b1c0 0x564ceba7d5b0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:11.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.399+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v95'37 uv36 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba7b1c0 2024-09-16T01:41:11.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.399+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7e0b0 con 0x564ceba7b1c0 2024-09-16T01:41:11.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.400+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v95'38 uv38 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba7b1c0 2024-09-16T01:41:11.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.400+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7e0b0 con 0x564ceba7b1c0 2024-09-16T01:41:11.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.401+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v95'39 uv39 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba7b1c0 2024-09-16T01:41:11.403 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:41:11.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.401+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7e0b0 con 0x564ceba7b1c0 2024-09-16T01:41:11.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.402+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v95'40 uv40 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba7b1c0 2024-09-16T01:41:11.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.402+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7e0b0 con 0x564ceba7b1c0 2024-09-16T01:41:11.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.403+0000 7f077e3f4640 1 -- 172.21.15.88:0/3710798792 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v95'41 uv41 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0770020070 con 0x564ceba7b1c0 2024-09-16T01:41:11.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.403+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7e0b0 con 0x564ceb9cb040 2024-09-16T01:41:11.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.404+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v95'45 uv44 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.404+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7e0b0 con 0x564ceb9cb040 2024-09-16T01:41:11.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.405+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v95'46 uv46 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.405+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7e0b0 con 0x564ceb9cb040 2024-09-16T01:41:11.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.406+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v95'47 uv47 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.407 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:41:11.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.406+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7f0c0 con 0x564ceb9cb040 2024-09-16T01:41:11.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.409+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v95'48 uv48 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.409+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7f0c0 con 0x564ceb9cb040 2024-09-16T01:41:11.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.411+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v95'49 uv49 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.411+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7e0b0 con 0x564ceb9cb040 2024-09-16T01:41:11.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.414+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v95'50 uv50 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceb9cb040 2024-09-16T01:41:11.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.414+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7e0b0 con 0x564ceba73870 2024-09-16T01:41:11.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.415+0000 7f0775bf3640 1 -- 172.21.15.88:0/3710798792 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v95'74 uv68 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f0768076430 con 0x564ceba73870 2024-09-16T01:41:11.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.415+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7e4a0 con 0x564ceba73870 2024-09-16T01:41:11.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.416+0000 7f0775bf3640 1 -- 172.21.15.88:0/3710798792 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v95'75 uv75 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0768076430 con 0x564ceba73870 2024-09-16T01:41:11.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.416+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7e890 con 0x564ceba73870 2024-09-16T01:41:11.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.417+0000 7f0775bf3640 1 -- 172.21.15.88:0/3710798792 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v95'76 uv76 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0768076430 con 0x564ceba73870 2024-09-16T01:41:11.418 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:41:11.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.417+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7f0e0 con 0x564ceba73870 2024-09-16T01:41:11.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.418+0000 7f0775bf3640 1 -- 172.21.15.88:0/3710798792 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v95'77 uv77 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0768076430 con 0x564ceba73870 2024-09-16T01:41:11.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.418+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7f0e0 con 0x564ceba73870 2024-09-16T01:41:11.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.419+0000 7f0775bf3640 1 -- 172.21.15.88:0/3710798792 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v95'78 uv78 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0768076430 con 0x564ceba73870 2024-09-16T01:41:11.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.419+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7f0e0 con 0x564ceba73870 2024-09-16T01:41:11.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.420+0000 7f0775bf3640 1 -- 172.21.15.88:0/3710798792 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v95'79 uv73 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0768076430 con 0x564ceba73870 2024-09-16T01:41:11.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.420+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7f0e0 con 0x564ceba73870 2024-09-16T01:41:11.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.421+0000 7f0775bf3640 1 -- 172.21.15.88:0/3710798792 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v95'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0768076430 con 0x564ceba73870 2024-09-16T01:41:11.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.421+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7f0e0 con 0x564ceba73870 2024-09-16T01:41:11.423 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:41:11.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.422+0000 7f0775bf3640 1 -- 172.21.15.88:0/3710798792 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v95'81 uv81 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0768076430 con 0x564ceba73870 2024-09-16T01:41:11.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.422+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7f0e0 con 0x564ceba73870 2024-09-16T01:41:11.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.422+0000 7f0775bf3640 1 -- 172.21.15.88:0/3710798792 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v95'82 uv82 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0768076430 con 0x564ceba73870 2024-09-16T01:41:11.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.423+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba7f0e0 con 0x564ceba73870 2024-09-16T01:41:11.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.423+0000 7f0775bf3640 1 -- 172.21.15.88:0/3710798792 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v95'83 uv83 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0768076430 con 0x564ceba73870 2024-09-16T01:41:11.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.423+0000 7f07808fef40 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x564ceba7fa30 0x564ceba821a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.423+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba826e0 con 0x564ceba7fa30 2024-09-16T01:41:11.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.424+0000 7f0775bf3640 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x564ceba7fa30 0x564ceba821a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.424+0000 7f0775bf3640 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x564ceba7fa30 0x564ceba821a0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:11.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.425+0000 7f0775bf3640 1 -- 172.21.15.88:0/3710798792 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v95'36 uv34 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f0768076430 con 0x564ceba7fa30 2024-09-16T01:41:11.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.425+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e95) -- 0x564ceba826e0 con 0x564ceba7fa30 2024-09-16T01:41:11.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.426+0000 7f0775bf3640 1 -- 172.21.15.88:0/3710798792 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v95'37 uv37 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f0768076430 con 0x564ceba7fa30 2024-09-16T01:41:11.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.426+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba826e0 con 0x564ceba7fa30 2024-09-16T01:41:11.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.429+0000 7f0775bf3640 1 -- 172.21.15.88:0/3710798792 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v95'38 uv38 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f0768076430 con 0x564ceba7fa30 2024-09-16T01:41:11.430 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:41:11.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.429+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba826e0 con 0x564ceba7fa30 2024-09-16T01:41:11.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.431+0000 7f0775bf3640 1 -- 172.21.15.88:0/3710798792 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v95'39 uv39 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f0768076430 con 0x564ceba7fa30 2024-09-16T01:41:11.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.431+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba826e0 con 0x564ceba73870 2024-09-16T01:41:11.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.432+0000 7f0775bf3640 1 -- 172.21.15.88:0/3710798792 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v95'45 uv44 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0768076430 con 0x564ceba73870 2024-09-16T01:41:11.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.432+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e95) -- 0x564ceba826e0 con 0x564ceba73870 2024-09-16T01:41:11.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.434+0000 7f0775bf3640 1 -- 172.21.15.88:0/3710798792 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v95'46 uv46 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0768076430 con 0x564ceba73870 2024-09-16T01:41:11.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.434+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba826e0 con 0x564ceba73870 2024-09-16T01:41:11.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.435+0000 7f0775bf3640 1 -- 172.21.15.88:0/3710798792 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v95'47 uv47 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0768076430 con 0x564ceba73870 2024-09-16T01:41:11.436 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:41:11.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.435+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba836f0 con 0x564ceba73870 2024-09-16T01:41:11.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.435+0000 7f0775bf3640 1 -- 172.21.15.88:0/3710798792 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v95'48 uv48 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0768076430 con 0x564ceba73870 2024-09-16T01:41:11.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.435+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba836f0 con 0x564ceba73870 2024-09-16T01:41:11.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.436+0000 7f0775bf3640 1 -- 172.21.15.88:0/3710798792 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v95'49 uv49 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0768076430 con 0x564ceba73870 2024-09-16T01:41:11.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.436+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba826e0 con 0x564ceba73870 2024-09-16T01:41:11.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.437+0000 7f0775bf3640 1 -- 172.21.15.88:0/3710798792 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v95'50 uv50 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0768076430 con 0x564ceba73870 2024-09-16T01:41:11.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.437+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba826e0 con 0x564ceba77d90 2024-09-16T01:41:11.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.438+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v95'37 uv36 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.438+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e95) -- 0x564ceba826e0 con 0x564ceba77d90 2024-09-16T01:41:11.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.440+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v95'38 uv38 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.440+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba826e0 con 0x564ceba77d90 2024-09-16T01:41:11.443 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:41:11.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.443+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v95'39 uv39 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.443+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba826e0 con 0x564ceba77d90 2024-09-16T01:41:11.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.445+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v95'40 uv40 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.445+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba826e0 con 0x564ceba77d90 2024-09-16T01:41:11.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.446+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v95'41 uv41 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.446+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba826e0 con 0x564ceba77d90 2024-09-16T01:41:11.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.449+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v95'95 uv86 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.449+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e95) -- 0x564ceba826e0 con 0x564ceba77d90 2024-09-16T01:41:11.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.451+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v95'96 uv96 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.451+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba826e0 con 0x564ceba77d90 2024-09-16T01:41:11.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.453+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v95'97 uv97 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.454 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:41:11.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.453+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba826e0 con 0x564ceba77d90 2024-09-16T01:41:11.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.455+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v95'98 uv98 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.455+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba826e0 con 0x564ceba77d90 2024-09-16T01:41:11.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.458+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v95'99 uv99 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.458+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba826e0 con 0x564ceba77d90 2024-09-16T01:41:11.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.459+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v95'100 uv93 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.459+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e95) -- 0x564ceba826e0 con 0x564ceba77d90 2024-09-16T01:41:11.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.462+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v95'101 uv101 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.462+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba826e0 con 0x564ceba77d90 2024-09-16T01:41:11.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.464+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v95'102 uv102 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.465 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:41:11.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.464+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba836f0 con 0x564ceba77d90 2024-09-16T01:41:11.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.465+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v95'103 uv103 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.465+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba836f0 con 0x564ceba77d90 2024-09-16T01:41:11.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.470+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v95'104 uv104 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.470+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba826e0 con 0x564ceba77d90 2024-09-16T01:41:11.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.471+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v95'105 uv105 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.471+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95) -- 0x564ceba826e0 con 0x564ceba77d90 2024-09-16T01:41:11.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v95'106 uv106 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f077800c720 con 0x564ceba77d90 2024-09-16T01:41:11.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x564ceb9cb040 msgr2=0x564ceb9c98e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:11.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f07808fef40 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x564ceb9cb040 0x564ceb9c98e0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x564ceba73870 msgr2=0x564ceba73ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:11.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f07808fef40 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x564ceba73870 0x564ceba73ca0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x564ceba7fa30 msgr2=0x564ceba821a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:11.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f07808fef40 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x564ceba7fa30 0x564ceba821a0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x564ceba6e840 msgr2=0x564ceba70c30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:11.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f07808fef40 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x564ceba6e840 0x564ceba70c30 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x564ceba77d90 msgr2=0x564ceba7a1c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:11.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f07808fef40 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x564ceba77d90 0x564ceba7a1c0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f077ebf5640 1 -- 172.21.15.88:0/3710798792 reap_dead start 2024-09-16T01:41:11.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x564ceba7b1c0 msgr2=0x564ceba7d5b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:11.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f07808fef40 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x564ceba7b1c0 0x564ceba7d5b0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f076006acc0 msgr2=0x7f076006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:11.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f07808fef40 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f076006acc0 0x7f076006d180 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0768005cf0 tx=0x7f0768005a00 comp rx=0 tx=0).stop 2024-09-16T01:41:11.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564ceb8f3dd0 msgr2=0x564ceb9ca5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:11.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f07808fef40 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564ceb8f3dd0 0x564ceb9ca5c0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f077000ee40 tx=0x7f077000c730 comp rx=0 tx=0).stop 2024-09-16T01:41:11.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 shutdown_connections 2024-09-16T01:41:11.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f07808fef40 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f076006acc0 0x7f076006d180 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f07808fef40 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x564ceba7b1c0 0x564ceba7d5b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f07808fef40 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x564ceba77d90 0x564ceba7a1c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f07808fef40 1 --2- 172.21.15.88:0/3710798792 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564ceb8f3dd0 0x564ceb9ca5c0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 >> 172.21.15.88:0/3710798792 conn(0x564ceb93d750 msgr2=0x564ceb8f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:11.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 shutdown_connections 2024-09-16T01:41:11.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.474+0000 7f07808fef40 1 -- 172.21.15.88:0/3710798792 wait complete. 2024-09-16T01:41:11.479 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm mds_snaptable 2024-09-16T01:41:11.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.520+0000 7f15e4c2ff40 1 Processor -- start 2024-09-16T01:41:11.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.520+0000 7f15e4c2ff40 1 -- start start 2024-09-16T01:41:11.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.521+0000 7f15e4c2ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572f81abfe0 0x5572f817be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.521+0000 7f15e4c2ff40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5572f8250d20 0x5572f817e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.521+0000 7f15e4c2ff40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572f8251100 0x5572f817ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.521+0000 7f15e4c2ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5572f80ce510 con 0x5572f81abfe0 2024-09-16T01:41:11.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.521+0000 7f15e4c2ff40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5572f8127c50 con 0x5572f8251100 2024-09-16T01:41:11.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.521+0000 7f15e4c2ff40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5572f81a7ea0 con 0x5572f8250d20 2024-09-16T01:41:11.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.521+0000 7f15e2725640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572f81abfe0 0x5572f817be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.521+0000 7f15e2725640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572f81abfe0 0x5572f817be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41300/0 (socket says 172.21.15.88:41300) 2024-09-16T01:41:11.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.521+0000 7f15e2725640 1 -- 172.21.15.88:0/1840976223 learned_addr learned my addr 172.21.15.88:0/1840976223 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:11.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.521+0000 7f15d9f24640 1 --2- 172.21.15.88:0/1840976223 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5572f8250d20 0x5572f817e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.521+0000 7f15e2f26640 1 --2- 172.21.15.88:0/1840976223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572f8251100 0x5572f817ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.521+0000 7f15e2725640 1 -- 172.21.15.88:0/1840976223 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5572f8250d20 msgr2=0x5572f817e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:11.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.521+0000 7f15e2725640 1 --2- 172.21.15.88:0/1840976223 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5572f8250d20 0x5572f817e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.521+0000 7f15e2725640 1 -- 172.21.15.88:0/1840976223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572f8251100 msgr2=0x5572f817ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:11.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.521+0000 7f15e2725640 1 --2- 172.21.15.88:0/1840976223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572f8251100 0x5572f817ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.521+0000 7f15e2725640 1 -- 172.21.15.88:0/1840976223 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572f817fc90 con 0x5572f81abfe0 2024-09-16T01:41:11.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.521+0000 7f15e2725640 1 --2- 172.21.15.88:0/1840976223 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572f81abfe0 0x5572f817be20 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f15d4057040 tx=0x7f15d404e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c244fbc1f4c82231 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:11.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.521+0000 7f15e1723640 1 -- 172.21.15.88:0/1840976223 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d4059040 con 0x5572f81abfe0 2024-09-16T01:41:11.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.521+0000 7f15e1723640 1 -- 172.21.15.88:0/1840976223 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15d40546b0 con 0x5572f81abfe0 2024-09-16T01:41:11.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.521+0000 7f15e1723640 1 -- 172.21.15.88:0/1840976223 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d4054a80 con 0x5572f81abfe0 2024-09-16T01:41:11.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.522+0000 7f15e4c2ff40 1 -- 172.21.15.88:0/1840976223 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572f81abfe0 msgr2=0x5572f817be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:11.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.522+0000 7f15e4c2ff40 1 --2- 172.21.15.88:0/1840976223 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572f81abfe0 0x5572f817be20 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f15d4057040 tx=0x7f15d404e000 comp rx=0 tx=0).stop 2024-09-16T01:41:11.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.522+0000 7f15e4c2ff40 1 -- 172.21.15.88:0/1840976223 shutdown_connections 2024-09-16T01:41:11.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.522+0000 7f15e4c2ff40 1 --2- 172.21.15.88:0/1840976223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572f8251100 0x5572f817ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.522+0000 7f15e4c2ff40 1 --2- 172.21.15.88:0/1840976223 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5572f8250d20 0x5572f817e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.522+0000 7f15e4c2ff40 1 --2- 172.21.15.88:0/1840976223 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572f81abfe0 0x5572f817be20 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.522+0000 7f15e4c2ff40 1 -- 172.21.15.88:0/1840976223 >> 172.21.15.88:0/1840976223 conn(0x5572f81c7750 msgr2=0x5572f82551e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:11.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.522+0000 7f15e4c2ff40 1 -- 172.21.15.88:0/1840976223 shutdown_connections 2024-09-16T01:41:11.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.522+0000 7f15e4c2ff40 1 -- 172.21.15.88:0/1840976223 wait complete. 2024-09-16T01:41:11.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.522+0000 7f15e4c2ff40 1 Processor -- start 2024-09-16T01:41:11.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.522+0000 7f15e4c2ff40 1 -- start start 2024-09-16T01:41:11.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.523+0000 7f15e4c2ff40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5572f81abfe0 0x5572f81a5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.523+0000 7f15e4c2ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572f8250d20 0x5572f81bc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.523+0000 7f15e4c2ff40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572f8251100 0x5572f81bcc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.523+0000 7f15e4c2ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5572f81a68c0 con 0x5572f8250d20 2024-09-16T01:41:11.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.523+0000 7f15e4c2ff40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5572f82504d0 con 0x5572f8251100 2024-09-16T01:41:11.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.523+0000 7f15e4c2ff40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5572f8252450 con 0x5572f81abfe0 2024-09-16T01:41:11.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.523+0000 7f15d9f24640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572f8250d20 0x5572f81bc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.523+0000 7f15d9f24640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572f8250d20 0x5572f81bc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41314/0 (socket says 172.21.15.88:41314) 2024-09-16T01:41:11.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.523+0000 7f15d9f24640 1 -- 172.21.15.88:0/3435283230 learned_addr learned my addr 172.21.15.88:0/3435283230 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:11.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.523+0000 7f15e2725640 1 --2- 172.21.15.88:0/3435283230 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5572f81abfe0 0x5572f81a5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.523+0000 7f15e2f26640 1 --2- 172.21.15.88:0/3435283230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572f8251100 0x5572f81bcc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.524+0000 7f15d9f24640 1 -- 172.21.15.88:0/3435283230 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5572f81abfe0 msgr2=0x5572f81a5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:11.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.524+0000 7f15d9f24640 1 --2- 172.21.15.88:0/3435283230 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5572f81abfe0 0x5572f81a5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.524+0000 7f15d9f24640 1 -- 172.21.15.88:0/3435283230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572f8251100 msgr2=0x5572f81bcc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:11.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.524+0000 7f15d9f24640 1 --2- 172.21.15.88:0/3435283230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572f8251100 0x5572f81bcc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.524+0000 7f15d9f24640 1 -- 172.21.15.88:0/3435283230 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572f817ed40 con 0x5572f8250d20 2024-09-16T01:41:11.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.524+0000 7f15d9f24640 1 --2- 172.21.15.88:0/3435283230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572f8250d20 0x5572f81bc730 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f15cc0010e0 tx=0x7f15cc00b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:11.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.524+0000 7f15dbfff640 1 -- 172.21.15.88:0/3435283230 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15cc00f650 con 0x5572f8250d20 2024-09-16T01:41:11.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.524+0000 7f15dbfff640 1 -- 172.21.15.88:0/3435283230 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15cc00f810 con 0x5572f8250d20 2024-09-16T01:41:11.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.524+0000 7f15e4c2ff40 1 -- 172.21.15.88:0/3435283230 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5572f817f020 con 0x5572f8250d20 2024-09-16T01:41:11.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.524+0000 7f15dbfff640 1 -- 172.21.15.88:0/3435283230 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15cc00fc00 con 0x5572f8250d20 2024-09-16T01:41:11.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.524+0000 7f15e4c2ff40 1 -- 172.21.15.88:0/3435283230 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572f81a5e40 con 0x5572f8250d20 2024-09-16T01:41:11.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.525+0000 7f15dbfff640 1 -- 172.21.15.88:0/3435283230 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f15cc0203f0 con 0x5572f8250d20 2024-09-16T01:41:11.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.526+0000 7f15dbfff640 1 --2- 172.21.15.88:0/3435283230 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f15c406ac90 0x7f15c406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.526+0000 7f15dbfff640 1 -- 172.21.15.88:0/3435283230 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6426+0+0 (secure 0 0 0) 0x7f15cc08aaa0 con 0x5572f8250d20 2024-09-16T01:41:11.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.526+0000 7f15e2725640 1 --2- 172.21.15.88:0/3435283230 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f15c406ac90 0x7f15c406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.526+0000 7f15e4c2ff40 1 --2- 172.21.15.88:0/3435283230 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5572f8253f10 0x5572f81a4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.526+0000 7f15e4c2ff40 1 -- 172.21.15.88:0/3435283230 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:1 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e95) -- 0x5572f81a4f60 con 0x5572f8253f10 2024-09-16T01:41:11.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.526+0000 7f15e2725640 1 --2- 172.21.15.88:0/3435283230 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f15c406ac90 0x7f15c406d150 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f15d4061000 tx=0x7f15d4056040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:11.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.526+0000 7f15e2f26640 1 --2- 172.21.15.88:0/3435283230 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5572f8253f10 0x5572f81a4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.527+0000 7f15e2f26640 1 --2- 172.21.15.88:0/3435283230 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5572f8253f10 0x5572f81a4a20 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:11.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.528+0000 7f15e2f26640 1 -- 172.21.15.88:0/3435283230 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(1 mds_snaptable [delete] v95'42 uv42 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f15dc0038f0 con 0x5572f8253f10 2024-09-16T01:41:11.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.528+0000 7f15e4c2ff40 1 -- 172.21.15.88:0/3435283230 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5572f8253f10 msgr2=0x5572f81a4a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:11.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.528+0000 7f15e4c2ff40 1 --2- 172.21.15.88:0/3435283230 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5572f8253f10 0x5572f81a4a20 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.528+0000 7f15e4c2ff40 1 -- 172.21.15.88:0/3435283230 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f15c406ac90 msgr2=0x7f15c406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:11.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.528+0000 7f15e4c2ff40 1 --2- 172.21.15.88:0/3435283230 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f15c406ac90 0x7f15c406d150 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f15d4061000 tx=0x7f15d4056040 comp rx=0 tx=0).stop 2024-09-16T01:41:11.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.528+0000 7f15e4c2ff40 1 -- 172.21.15.88:0/3435283230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572f8250d20 msgr2=0x5572f81bc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:11.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.528+0000 7f15e4c2ff40 1 --2- 172.21.15.88:0/3435283230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572f8250d20 0x5572f81bc730 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f15cc0010e0 tx=0x7f15cc00b6c0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.528+0000 7f15e2f26640 1 -- 172.21.15.88:0/3435283230 reap_dead start 2024-09-16T01:41:11.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.528+0000 7f15e4c2ff40 1 -- 172.21.15.88:0/3435283230 shutdown_connections 2024-09-16T01:41:11.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.528+0000 7f15e4c2ff40 1 -- 172.21.15.88:0/3435283230 >> 172.21.15.88:0/3435283230 conn(0x5572f81c7750 msgr2=0x5572f81ab670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:11.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.528+0000 7f15e4c2ff40 1 -- 172.21.15.88:0/3435283230 shutdown_connections 2024-09-16T01:41:11.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.528+0000 7f15e4c2ff40 1 -- 172.21.15.88:0/3435283230 wait complete. 2024-09-16T01:41:11.534 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:41:11.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.674+0000 7f795320e640 1 Processor -- start 2024-09-16T01:41:11.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.674+0000 7f795320e640 1 -- start start 2024-09-16T01:41:11.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.675+0000 7f795320e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f794c15c2a0 0x7f794c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.675+0000 7f795320e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f794c073b10 0x7f794c06f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.675+0000 7f795320e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f794c06fc50 0x7f794c06cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.675+0000 7f795320e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f794c15c680 con 0x7f794c15c2a0 2024-09-16T01:41:11.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.675+0000 7f795320e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f794c074120 con 0x7f794c073b10 2024-09-16T01:41:11.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.675+0000 7f795320e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f794c15ef00 con 0x7f794c06fc50 2024-09-16T01:41:11.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.675+0000 7f7950faa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f794c15c2a0 0x7f794c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.675+0000 7f7950faa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f794c15c2a0 0x7f794c0735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41328/0 (socket says 172.21.15.88:41328) 2024-09-16T01:41:11.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.675+0000 7f7950faa640 1 -- 172.21.15.88:0/2930024389 learned_addr learned my addr 172.21.15.88:0/2930024389 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:11.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.675+0000 7f794bfff640 1 --2- 172.21.15.88:0/2930024389 >> v2:172.21.15.154:3300/0 conn(0x7f794c073b10 0x7f794c06f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.675+0000 7f79517ab640 1 --2- 172.21.15.88:0/2930024389 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f794c06fc50 0x7f794c06cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.675+0000 7f7950faa640 1 -- 172.21.15.88:0/2930024389 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f794c06fc50 msgr2=0x7f794c06cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:11.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.675+0000 7f7950faa640 1 --2- 172.21.15.88:0/2930024389 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f794c06fc50 0x7f794c06cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.675+0000 7f7950faa640 1 -- 172.21.15.88:0/2930024389 >> v2:172.21.15.154:3300/0 conn(0x7f794c073b10 msgr2=0x7f794c06f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:11.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.675+0000 7f7950faa640 1 --2- 172.21.15.88:0/2930024389 >> v2:172.21.15.154:3300/0 conn(0x7f794c073b10 0x7f794c06f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.675+0000 7f7950faa640 1 -- 172.21.15.88:0/2930024389 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f794c15f1e0 con 0x7f794c15c2a0 2024-09-16T01:41:11.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.675+0000 7f794bfff640 1 --2- 172.21.15.88:0/2930024389 >> v2:172.21.15.154:3300/0 conn(0x7f794c073b10 0x7f794c06f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:11.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.675+0000 7f79517ab640 1 --2- 172.21.15.88:0/2930024389 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f794c06fc50 0x7f794c06cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:11.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.676+0000 7f7950faa640 1 --2- 172.21.15.88:0/2930024389 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f794c15c2a0 0x7f794c0735d0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f794c072d80 tx=0x7f794002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46cd784f9bf6c4b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:11.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.676+0000 7f794b7fe640 1 -- 172.21.15.88:0/2930024389 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7940037070 con 0x7f794c15c2a0 2024-09-16T01:41:11.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.676+0000 7f794b7fe640 1 -- 172.21.15.88:0/2930024389 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7940032440 con 0x7f794c15c2a0 2024-09-16T01:41:11.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.676+0000 7f794b7fe640 1 -- 172.21.15.88:0/2930024389 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7940032810 con 0x7f794c15c2a0 2024-09-16T01:41:11.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.676+0000 7f795320e640 1 -- 172.21.15.88:0/2930024389 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f794c15c2a0 msgr2=0x7f794c0735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:11.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.676+0000 7f795320e640 1 --2- 172.21.15.88:0/2930024389 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f794c15c2a0 0x7f794c0735d0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f794c072d80 tx=0x7f794002a1e0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.676+0000 7f795320e640 1 -- 172.21.15.88:0/2930024389 shutdown_connections 2024-09-16T01:41:11.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.676+0000 7f795320e640 1 --2- 172.21.15.88:0/2930024389 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f794c06fc50 0x7f794c06cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.676+0000 7f795320e640 1 --2- 172.21.15.88:0/2930024389 >> v2:172.21.15.154:3300/0 conn(0x7f794c073b10 0x7f794c06f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.676+0000 7f795320e640 1 --2- 172.21.15.88:0/2930024389 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f794c15c2a0 0x7f794c0735d0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.676+0000 7f795320e640 1 -- 172.21.15.88:0/2930024389 >> 172.21.15.88:0/2930024389 conn(0x7f794c0b3070 msgr2=0x7f794c0b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:11.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.676+0000 7f795320e640 1 -- 172.21.15.88:0/2930024389 shutdown_connections 2024-09-16T01:41:11.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.676+0000 7f795320e640 1 -- 172.21.15.88:0/2930024389 wait complete. 2024-09-16T01:41:11.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.678+0000 7f795320e640 1 Processor -- start 2024-09-16T01:41:11.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.678+0000 7f795320e640 1 -- start start 2024-09-16T01:41:11.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.678+0000 7f795320e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f794c06fc50 0x7f794c152e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.678+0000 7f795320e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f794c073b10 0x7f794c15b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.678+0000 7f795320e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f794c15c2a0 0x7f794c160490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.678+0000 7f795320e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f794c0b5f20 con 0x7f794c06fc50 2024-09-16T01:41:11.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.678+0000 7f795320e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f794c15ebf0 con 0x7f794c15c2a0 2024-09-16T01:41:11.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.678+0000 7f795320e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f794c0b60a0 con 0x7f794c073b10 2024-09-16T01:41:11.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.679+0000 7f7950faa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f794c06fc50 0x7f794c152e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.679+0000 7f7950faa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f794c06fc50 0x7f794c152e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41340/0 (socket says 172.21.15.88:41340) 2024-09-16T01:41:11.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.679+0000 7f7950faa640 1 -- 172.21.15.88:0/4104218913 learned_addr learned my addr 172.21.15.88:0/4104218913 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:11.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.679+0000 7f79517ab640 1 --2- 172.21.15.88:0/4104218913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f794c15c2a0 0x7f794c160490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.679+0000 7f794bfff640 1 --2- 172.21.15.88:0/4104218913 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f794c073b10 0x7f794c15b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.679+0000 7f7950faa640 1 -- 172.21.15.88:0/4104218913 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f794c073b10 msgr2=0x7f794c15b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:11.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.679+0000 7f7950faa640 1 --2- 172.21.15.88:0/4104218913 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f794c073b10 0x7f794c15b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.679+0000 7f7950faa640 1 -- 172.21.15.88:0/4104218913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f794c15c2a0 msgr2=0x7f794c160490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:11.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.679+0000 7f7950faa640 1 --2- 172.21.15.88:0/4104218913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f794c15c2a0 0x7f794c160490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:11.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.679+0000 7f7950faa640 1 -- 172.21.15.88:0/4104218913 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f794c1618c0 con 0x7f794c06fc50 2024-09-16T01:41:11.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.679+0000 7f7950faa640 1 --2- 172.21.15.88:0/4104218913 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f794c06fc50 0x7f794c152e80 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f794002a220 tx=0x7f79400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:11.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.679+0000 7f79497fa640 1 -- 172.21.15.88:0/4104218913 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7940037040 con 0x7f794c06fc50 2024-09-16T01:41:11.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.679+0000 7f79497fa640 1 -- 172.21.15.88:0/4104218913 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7940042070 con 0x7f794c06fc50 2024-09-16T01:41:11.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.679+0000 7f79497fa640 1 -- 172.21.15.88:0/4104218913 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7940030bb0 con 0x7f794c06fc50 2024-09-16T01:41:11.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.679+0000 7f795320e640 1 -- 172.21.15.88:0/4104218913 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f794c161ba0 con 0x7f794c06fc50 2024-09-16T01:41:11.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.679+0000 7f795320e640 1 -- 172.21.15.88:0/4104218913 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f794c15fc90 con 0x7f794c06fc50 2024-09-16T01:41:11.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.680+0000 7f795320e640 1 -- 172.21.15.88:0/4104218913 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f794c0b12f0 con 0x7f794c06fc50 2024-09-16T01:41:11.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.681+0000 7f79497fa640 1 -- 172.21.15.88:0/4104218913 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7940047460 con 0x7f794c06fc50 2024-09-16T01:41:11.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.681+0000 7f79497fa640 1 --2- 172.21.15.88:0/4104218913 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f793006c2b0 0x7f793006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:11.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.681+0000 7f79497fa640 1 -- 172.21.15.88:0/4104218913 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 6426+0+0 (secure 0 0 0) 0x7f79400ac240 con 0x7f794c06fc50 2024-09-16T01:41:11.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.681+0000 7f794bfff640 1 --2- 172.21.15.88:0/4104218913 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f793006c2b0 0x7f793006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:11.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.681+0000 7f794bfff640 1 --2- 172.21.15.88:0/4104218913 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f793006c2b0 0x7f793006e770 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f794c15abf0 tx=0x7f7938008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:11.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.685+0000 7f79497fa640 1 -- 172.21.15.88:0/4104218913 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f794007a090 con 0x7f794c06fc50 2024-09-16T01:41:11.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:11.885+0000 7f795320e640 1 -- 172.21.15.88:0/4104218913 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f794c063280 con 0x7f794c06fc50 2024-09-16T01:41:12.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.171+0000 7f79497fa640 1 -- 172.21.15.88:0/4104218913 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v120) ==== 162+0+0 (secure 0 0 0) 0x7f794007e390 con 0x7f794c06fc50 2024-09-16T01:41:12.172 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:41:12.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.173+0000 7f795320e640 1 -- 172.21.15.88:0/4104218913 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f793006c2b0 msgr2=0x7f793006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:12.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.173+0000 7f795320e640 1 --2- 172.21.15.88:0/4104218913 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f793006c2b0 0x7f793006e770 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f794c15abf0 tx=0x7f7938008040 comp rx=0 tx=0).stop 2024-09-16T01:41:12.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.173+0000 7f795320e640 1 -- 172.21.15.88:0/4104218913 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f794c06fc50 msgr2=0x7f794c152e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:12.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.173+0000 7f795320e640 1 --2- 172.21.15.88:0/4104218913 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f794c06fc50 0x7f794c152e80 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f794002a220 tx=0x7f79400046c0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.173+0000 7f795320e640 1 -- 172.21.15.88:0/4104218913 shutdown_connections 2024-09-16T01:41:12.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.173+0000 7f795320e640 1 --2- 172.21.15.88:0/4104218913 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f793006c2b0 0x7f793006e770 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.173+0000 7f795320e640 1 --2- 172.21.15.88:0/4104218913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f794c15c2a0 0x7f794c160490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.173+0000 7f795320e640 1 --2- 172.21.15.88:0/4104218913 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f794c073b10 0x7f794c15b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.173+0000 7f795320e640 1 --2- 172.21.15.88:0/4104218913 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f794c06fc50 0x7f794c152e80 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.173+0000 7f795320e640 1 -- 172.21.15.88:0/4104218913 >> 172.21.15.88:0/4104218913 conn(0x7f794c0b3070 msgr2=0x7f794c0b15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:12.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.173+0000 7f795320e640 1 -- 172.21.15.88:0/4104218913 shutdown_connections 2024-09-16T01:41:12.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.173+0000 7f795320e640 1 -- 172.21.15.88:0/4104218913 wait complete. 2024-09-16T01:41:12.182 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:41:12.180+0000 7f2ab8e16640 -1 mds.0.snaptable: load_2 could not read table: -2 2024-09-16T01:41:12.183 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:41:12.180+0000 7f2ab8e16640 -1 log_channel(cluster) log [ERR] : error reading table object 'mds_snaptable' -2 ((2) No such file or directory) 2024-09-16T01:41:12.184 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:41:12.181+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:41:12.184 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:41:12.181+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:41:12.191 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:12.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.289+0000 7fe91e991640 1 Processor -- start 2024-09-16T01:41:12.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.289+0000 7fe91e991640 1 -- start start 2024-09-16T01:41:12.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.289+0000 7fe91e991640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe9180d3310 0x7fe9180d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:12.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.289+0000 7fe91e991640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe9180d1650 0x7fe9180b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:12.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.289+0000 7fe91e991640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe91815ea40 con 0x7fe91815e660 2024-09-16T01:41:12.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.289+0000 7fe91e991640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe9180d1ac0 con 0x7fe9180d3310 2024-09-16T01:41:12.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.289+0000 7fe91e991640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe9180d3870 con 0x7fe9180d1650 2024-09-16T01:41:12.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.290+0000 7fe9177fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe9180d3310 0x7fe9180d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:12.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.290+0000 7fe9177fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe9180d3310 0x7fe9180d1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41354/0 (socket says 172.21.15.88:41354) 2024-09-16T01:41:12.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.290+0000 7fe9177fe640 1 -- 172.21.15.88:0/3393195908 learned_addr learned my addr 172.21.15.88:0/3393195908 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:12.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.290+0000 7fe91cf2e640 1 --2- 172.21.15.88:0/3393195908 >> v2:172.21.15.154:3300/0 conn(0x7fe9180d1650 0x7fe9180b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:12.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.290+0000 7fe9177fe640 1 -- 172.21.15.88:0/3393195908 >> v2:172.21.15.154:3300/0 conn(0x7fe9180d1650 msgr2=0x7fe9180b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:12.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.290+0000 7fe9177fe640 1 --2- 172.21.15.88:0/3393195908 >> v2:172.21.15.154:3300/0 conn(0x7fe9180d1650 0x7fe9180b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.290+0000 7fe9177fe640 1 -- 172.21.15.88:0/3393195908 >> v1:172.21.15.154:6789/0 conn(0x7fe91815e660 legacy=0x7fe9180cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:12.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.290+0000 7fe9177fe640 1 -- 172.21.15.88:0/3393195908 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9180d3b50 con 0x7fe9180d3310 2024-09-16T01:41:12.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.291+0000 7fe9177fe640 1 --2- 172.21.15.88:0/3393195908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9180d3310 0x7fe9180d1110 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fe9180d26b0 tx=0x7fe90c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c1532d75729e49 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:12.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.291+0000 7fe916ffd640 1 -- 172.21.15.88:0/3393195908 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90c037070 con 0x7fe9180d3310 2024-09-16T01:41:12.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.291+0000 7fe916ffd640 1 -- 172.21.15.88:0/3393195908 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe90c032440 con 0x7fe9180d3310 2024-09-16T01:41:12.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.291+0000 7fe916ffd640 1 -- 172.21.15.88:0/3393195908 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90c032810 con 0x7fe9180d3310 2024-09-16T01:41:12.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.291+0000 7fe91e991640 1 -- 172.21.15.88:0/3393195908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9180d3310 msgr2=0x7fe9180d1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:12.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.291+0000 7fe91e991640 1 --2- 172.21.15.88:0/3393195908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9180d3310 0x7fe9180d1110 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fe9180d26b0 tx=0x7fe90c02a230 comp rx=0 tx=0).stop 2024-09-16T01:41:12.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.291+0000 7fe91e991640 1 -- 172.21.15.88:0/3393195908 shutdown_connections 2024-09-16T01:41:12.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.291+0000 7fe91e991640 1 --2- 172.21.15.88:0/3393195908 >> v2:172.21.15.154:3300/0 conn(0x7fe9180d1650 0x7fe9180b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.291+0000 7fe91e991640 1 --2- 172.21.15.88:0/3393195908 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9180d3310 0x7fe9180d1110 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.291+0000 7fe91e991640 1 -- 172.21.15.88:0/3393195908 >> 172.21.15.88:0/3393195908 conn(0x7fe9180b6410 msgr2=0x7fe9180b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:12.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.291+0000 7fe91e991640 1 -- 172.21.15.88:0/3393195908 shutdown_connections 2024-09-16T01:41:12.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.291+0000 7fe91e991640 1 -- 172.21.15.88:0/3393195908 wait complete. 2024-09-16T01:41:12.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.292+0000 7fe91e991640 1 Processor -- start 2024-09-16T01:41:12.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.293+0000 7fe91e991640 1 -- start start 2024-09-16T01:41:12.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.293+0000 7fe91e991640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe9180d1650 0x7fe9180cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:12.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.293+0000 7fe91e991640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9180d3310 0x7fe9180c8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:12.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.293+0000 7fe91e991640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe91815e660 0x7fe9180cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:12.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.293+0000 7fe91e991640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe9180d43f0 con 0x7fe9180d3310 2024-09-16T01:41:12.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.293+0000 7fe91e991640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe9180d4110 con 0x7fe91815e660 2024-09-16T01:41:12.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.293+0000 7fe91e991640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe9180b80b0 con 0x7fe9180d1650 2024-09-16T01:41:12.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.293+0000 7fe9177fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9180d3310 0x7fe9180c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:12.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.294+0000 7fe9177fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9180d3310 0x7fe9180c8420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41364/0 (socket says 172.21.15.88:41364) 2024-09-16T01:41:12.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.294+0000 7fe9177fe640 1 -- 172.21.15.88:0/146871471 learned_addr learned my addr 172.21.15.88:0/146871471 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:12.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.294+0000 7fe91cf2e640 1 --2- 172.21.15.88:0/146871471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe91815e660 0x7fe9180cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:12.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.294+0000 7fe917fff640 1 --2- 172.21.15.88:0/146871471 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe9180d1650 0x7fe9180cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:12.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.294+0000 7fe9177fe640 1 -- 172.21.15.88:0/146871471 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe9180d1650 msgr2=0x7fe9180cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:12.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.294+0000 7fe9177fe640 1 --2- 172.21.15.88:0/146871471 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe9180d1650 0x7fe9180cd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.294+0000 7fe9177fe640 1 -- 172.21.15.88:0/146871471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe91815e660 msgr2=0x7fe9180cb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:12.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.294+0000 7fe9177fe640 1 --2- 172.21.15.88:0/146871471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe91815e660 0x7fe9180cb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.294+0000 7fe9177fe640 1 -- 172.21.15.88:0/146871471 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9180cbd10 con 0x7fe9180d3310 2024-09-16T01:41:12.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.294+0000 7fe917fff640 1 --2- 172.21.15.88:0/146871471 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe9180d1650 0x7fe9180cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:12.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.294+0000 7fe9177fe640 1 --2- 172.21.15.88:0/146871471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9180d3310 0x7fe9180c8420 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fe90c000c00 tx=0x7fe90c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:12.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.294+0000 7fe914ff9640 1 -- 172.21.15.88:0/146871471 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90c041020 con 0x7fe9180d3310 2024-09-16T01:41:12.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.294+0000 7fe914ff9640 1 -- 172.21.15.88:0/146871471 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe90c0306c0 con 0x7fe9180d3310 2024-09-16T01:41:12.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.294+0000 7fe91e991640 1 -- 172.21.15.88:0/146871471 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe91814bff0 con 0x7fe9180d3310 2024-09-16T01:41:12.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.294+0000 7fe914ff9640 1 -- 172.21.15.88:0/146871471 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90c037040 con 0x7fe9180d3310 2024-09-16T01:41:12.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.294+0000 7fe91e991640 1 -- 172.21.15.88:0/146871471 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9180cdbf0 con 0x7fe9180d3310 2024-09-16T01:41:12.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.295+0000 7fe91e991640 1 -- 172.21.15.88:0/146871471 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9180b7b70 con 0x7fe9180d3310 2024-09-16T01:41:12.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.296+0000 7fe914ff9640 1 -- 172.21.15.88:0/146871471 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe90c044020 con 0x7fe9180d3310 2024-09-16T01:41:12.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.296+0000 7fe914ff9640 1 --2- 172.21.15.88:0/146871471 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe90406c2b0 0x7fe90406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:12.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.296+0000 7fe914ff9640 1 -- 172.21.15.88:0/146871471 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6512+0+0 (secure 0 0 0) 0x7fe90c0abb20 con 0x7fe9180d3310 2024-09-16T01:41:12.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.296+0000 7fe917fff640 1 --2- 172.21.15.88:0/146871471 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe90406c2b0 0x7fe90406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:12.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.296+0000 7fe917fff640 1 --2- 172.21.15.88:0/146871471 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe90406c2b0 0x7fe90406e770 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe908006fd0 tx=0x7fe908008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:12.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.300+0000 7fe914ff9640 1 -- 172.21.15.88:0/146871471 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe90c04b070 con 0x7fe9180d3310 2024-09-16T01:41:12.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.499+0000 7fe91e991640 1 -- 172.21.15.88:0/146871471 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe91806ad90 con 0x7fe9180d3310 2024-09-16T01:41:12.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.499+0000 7fe914ff9640 1 -- 172.21.15.88:0/146871471 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+4388 (secure 0 0 0) 0x7fe90c07a000 con 0x7fe9180d3310 2024-09-16T01:41:12.501 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:12.501 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":122,"btime":"2024-09-16T01:41:12:181725+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":7866,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/293764774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":293764774},{"type":"v1","addr":"172.21.15.88:6837","nonce":293764774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:12.181665+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:41:12.501 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 122 2024-09-16T01:41:12.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.501+0000 7fe91e991640 1 -- 172.21.15.88:0/146871471 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe90406c2b0 msgr2=0x7fe90406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:12.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.501+0000 7fe91e991640 1 --2- 172.21.15.88:0/146871471 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe90406c2b0 0x7fe90406e770 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe908006fd0 tx=0x7fe908008040 comp rx=0 tx=0).stop 2024-09-16T01:41:12.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.501+0000 7fe91e991640 1 -- 172.21.15.88:0/146871471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9180d3310 msgr2=0x7fe9180c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:12.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.501+0000 7fe91e991640 1 --2- 172.21.15.88:0/146871471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9180d3310 0x7fe9180c8420 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fe90c000c00 tx=0x7fe90c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.501+0000 7fe91e991640 1 -- 172.21.15.88:0/146871471 shutdown_connections 2024-09-16T01:41:12.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.501+0000 7fe91e991640 1 --2- 172.21.15.88:0/146871471 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe90406c2b0 0x7fe90406e770 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.501+0000 7fe91e991640 1 --2- 172.21.15.88:0/146871471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe91815e660 0x7fe9180cb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.501+0000 7fe91e991640 1 --2- 172.21.15.88:0/146871471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9180d3310 0x7fe9180c8420 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.501+0000 7fe91e991640 1 --2- 172.21.15.88:0/146871471 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe9180d1650 0x7fe9180cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.501+0000 7fe91e991640 1 -- 172.21.15.88:0/146871471 >> 172.21.15.88:0/146871471 conn(0x7fe9180b6410 msgr2=0x7fe918158940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:12.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.501+0000 7fe91e991640 1 -- 172.21.15.88:0/146871471 shutdown_connections 2024-09-16T01:41:12.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.501+0000 7fe91e991640 1 -- 172.21.15.88:0/146871471 wait complete. 2024-09-16T01:41:12.514 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:41:12.514 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:12.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.611+0000 7f74f9453640 1 Processor -- start 2024-09-16T01:41:12.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.611+0000 7f74f9453640 1 -- start start 2024-09-16T01:41:12.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.612+0000 7f74f9453640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f74f4073080 0x7f74f40b5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:12.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.612+0000 7f74f9453640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f74f415c670 0x7f74f4073460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:12.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.612+0000 7f74f9453640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74f4071750 con 0x7f74f40b1990 2024-09-16T01:41:12.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.612+0000 7f74f9453640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f74f40d0c60 con 0x7f74f4073080 2024-09-16T01:41:12.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.612+0000 7f74f9453640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f74f40d0370 con 0x7f74f415c670 2024-09-16T01:41:12.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.612+0000 7f74f2ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f74f4073080 0x7f74f40b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:12.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.612+0000 7f74f2ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f74f4073080 0x7f74f40b5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41380/0 (socket says 172.21.15.88:41380) 2024-09-16T01:41:12.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.612+0000 7f74f2ffd640 1 -- 172.21.15.88:0/1326539741 learned_addr learned my addr 172.21.15.88:0/1326539741 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:12.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.612+0000 7f74f2ffd640 1 -- 172.21.15.88:0/1326539741 >> v2:172.21.15.154:3300/0 conn(0x7f74f415c670 msgr2=0x7f74f4073460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:12.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.612+0000 7f74f2ffd640 1 --2- 172.21.15.88:0/1326539741 >> v2:172.21.15.154:3300/0 conn(0x7f74f415c670 0x7f74f4073460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.612+0000 7f74f2ffd640 1 -- 172.21.15.88:0/1326539741 >> v1:172.21.15.154:6789/0 conn(0x7f74f40b1990 legacy=0x7f74f40d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:12.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.613+0000 7f74f2ffd640 1 -- 172.21.15.88:0/1326539741 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74f40b59a0 con 0x7f74f4073080 2024-09-16T01:41:12.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.613+0000 7f74f2ffd640 1 --2- 172.21.15.88:0/1326539741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74f4073080 0x7f74f40b5460 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f74f40cf840 tx=0x7f74e402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50237e9e540f0e1e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:12.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.613+0000 7f74f27fc640 1 -- 172.21.15.88:0/1326539741 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e4037070 con 0x7f74f4073080 2024-09-16T01:41:12.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.613+0000 7f74f27fc640 1 -- 172.21.15.88:0/1326539741 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74e4032440 con 0x7f74f4073080 2024-09-16T01:41:12.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.613+0000 7f74f27fc640 1 -- 172.21.15.88:0/1326539741 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e4032810 con 0x7f74f4073080 2024-09-16T01:41:12.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.614+0000 7f74f9453640 1 -- 172.21.15.88:0/1326539741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74f4073080 msgr2=0x7f74f40b5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:12.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.614+0000 7f74f9453640 1 --2- 172.21.15.88:0/1326539741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74f4073080 0x7f74f40b5460 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f74f40cf840 tx=0x7f74e402a230 comp rx=0 tx=0).stop 2024-09-16T01:41:12.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.614+0000 7f74f9453640 1 -- 172.21.15.88:0/1326539741 shutdown_connections 2024-09-16T01:41:12.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.614+0000 7f74f9453640 1 --2- 172.21.15.88:0/1326539741 >> v2:172.21.15.154:3300/0 conn(0x7f74f415c670 0x7f74f4073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.614+0000 7f74f9453640 1 --2- 172.21.15.88:0/1326539741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74f4073080 0x7f74f40b5460 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.614+0000 7f74f9453640 1 -- 172.21.15.88:0/1326539741 >> 172.21.15.88:0/1326539741 conn(0x7f74f40d24d0 msgr2=0x7f74f40d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:12.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.614+0000 7f74f9453640 1 -- 172.21.15.88:0/1326539741 shutdown_connections 2024-09-16T01:41:12.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.614+0000 7f74f9453640 1 -- 172.21.15.88:0/1326539741 wait complete. 2024-09-16T01:41:12.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.615+0000 7f74f9453640 1 Processor -- start 2024-09-16T01:41:12.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.615+0000 7f74f9453640 1 -- start start 2024-09-16T01:41:12.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.616+0000 7f74f9453640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74f4073080 0x7f74f40b0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:12.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.616+0000 7f74f9453640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74f40b1990 0x7f74f415ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:12.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.616+0000 7f74f9453640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74f415c670 0x7f74f40b6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:12.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.616+0000 7f74f9453640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f74f4071470 con 0x7f74f415c670 2024-09-16T01:41:12.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.616+0000 7f74f9453640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f74f4071190 con 0x7f74f4073080 2024-09-16T01:41:12.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.616+0000 7f74f9453640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f74f40d0610 con 0x7f74f40b1990 2024-09-16T01:41:12.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.616+0000 7f74f37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74f415c670 0x7f74f40b6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:12.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.616+0000 7f74f37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74f415c670 0x7f74f40b6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41390/0 (socket says 172.21.15.88:41390) 2024-09-16T01:41:12.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.616+0000 7f74f37fe640 1 -- 172.21.15.88:0/243919086 learned_addr learned my addr 172.21.15.88:0/243919086 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:12.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.616+0000 7f74f2ffd640 1 --2- 172.21.15.88:0/243919086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74f4073080 0x7f74f40b0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:12.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.616+0000 7f74ea7fc640 1 --2- 172.21.15.88:0/243919086 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74f40b1990 0x7f74f415ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:12.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.616+0000 7f74f37fe640 1 -- 172.21.15.88:0/243919086 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74f40b1990 msgr2=0x7f74f415ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:12.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.616+0000 7f74f37fe640 1 --2- 172.21.15.88:0/243919086 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74f40b1990 0x7f74f415ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.616+0000 7f74f37fe640 1 -- 172.21.15.88:0/243919086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74f4073080 msgr2=0x7f74f40b0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:12.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.616+0000 7f74f37fe640 1 --2- 172.21.15.88:0/243919086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74f4073080 0x7f74f40b0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.616+0000 7f74f37fe640 1 -- 172.21.15.88:0/243919086 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74f415bd20 con 0x7f74f415c670 2024-09-16T01:41:12.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.616+0000 7f74f2ffd640 1 --2- 172.21.15.88:0/243919086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74f4073080 0x7f74f40b0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:12.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.617+0000 7f74f37fe640 1 --2- 172.21.15.88:0/243919086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74f415c670 0x7f74f40b6bc0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f74ec00c7b0 tx=0x7f74ec00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:12.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.617+0000 7f74ebfff640 1 -- 172.21.15.88:0/243919086 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ec017070 con 0x7f74f415c670 2024-09-16T01:41:12.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.617+0000 7f74ebfff640 1 -- 172.21.15.88:0/243919086 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74ec003da0 con 0x7f74f415c670 2024-09-16T01:41:12.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.617+0000 7f74f9453640 1 -- 172.21.15.88:0/243919086 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74f415b480 con 0x7f74f415c670 2024-09-16T01:41:12.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.617+0000 7f74ebfff640 1 -- 172.21.15.88:0/243919086 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ec00ce60 con 0x7f74f415c670 2024-09-16T01:41:12.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.617+0000 7f74f9453640 1 -- 172.21.15.88:0/243919086 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74f415ba40 con 0x7f74f415c670 2024-09-16T01:41:12.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.618+0000 7f74f9453640 1 -- 172.21.15.88:0/243919086 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74f40d08f0 con 0x7f74f415c670 2024-09-16T01:41:12.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.618+0000 7f74ebfff640 1 -- 172.21.15.88:0/243919086 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f74ec01a070 con 0x7f74f415c670 2024-09-16T01:41:12.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.619+0000 7f74ebfff640 1 --2- 172.21.15.88:0/243919086 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f74d406c2b0 0x7f74d406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:12.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.619+0000 7f74ebfff640 1 -- 172.21.15.88:0/243919086 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6512+0+0 (secure 0 0 0) 0x7f74ec08ad60 con 0x7f74f415c670 2024-09-16T01:41:12.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.619+0000 7f74f2ffd640 1 --2- 172.21.15.88:0/243919086 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f74d406c2b0 0x7f74d406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:12.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.619+0000 7f74f2ffd640 1 --2- 172.21.15.88:0/243919086 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f74d406c2b0 0x7f74d406e770 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f74e4034000 tx=0x7f74e402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:12.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.623+0000 7f74ebfff640 1 -- 172.21.15.88:0/243919086 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f74ec059240 con 0x7f74f415c670 2024-09-16T01:41:12.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.820+0000 7f74f9453640 1 -- 172.21.15.88:0/243919086 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74f40d3050 con 0x7f74f415c670 2024-09-16T01:41:12.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.821+0000 7f74ebfff640 1 -- 172.21.15.88:0/243919086 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+4388 (secure 0 0 0) 0x7f74ec058b40 con 0x7f74f415c670 2024-09-16T01:41:12.822 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:12.823 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":122,"btime":"2024-09-16T01:41:12:181725+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":7866,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/293764774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":293764774},{"type":"v1","addr":"172.21.15.88:6837","nonce":293764774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:12.181665+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:41:12.823 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 122 2024-09-16T01:41:12.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.823+0000 7f74f9453640 1 -- 172.21.15.88:0/243919086 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f74d406c2b0 msgr2=0x7f74d406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:12.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.823+0000 7f74f9453640 1 --2- 172.21.15.88:0/243919086 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f74d406c2b0 0x7f74d406e770 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f74e4034000 tx=0x7f74e402e000 comp rx=0 tx=0).stop 2024-09-16T01:41:12.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.823+0000 7f74f9453640 1 -- 172.21.15.88:0/243919086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74f415c670 msgr2=0x7f74f40b6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:12.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.823+0000 7f74f9453640 1 --2- 172.21.15.88:0/243919086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74f415c670 0x7f74f40b6bc0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f74ec00c7b0 tx=0x7f74ec00cc70 comp rx=0 tx=0).stop 2024-09-16T01:41:12.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.823+0000 7f74f9453640 1 -- 172.21.15.88:0/243919086 shutdown_connections 2024-09-16T01:41:12.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.823+0000 7f74f9453640 1 --2- 172.21.15.88:0/243919086 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f74d406c2b0 0x7f74d406e770 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.823+0000 7f74f9453640 1 --2- 172.21.15.88:0/243919086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74f415c670 0x7f74f40b6bc0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.823+0000 7f74f9453640 1 --2- 172.21.15.88:0/243919086 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74f40b1990 0x7f74f415ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.823+0000 7f74f9453640 1 --2- 172.21.15.88:0/243919086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74f4073080 0x7f74f40b0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.823+0000 7f74f9453640 1 -- 172.21.15.88:0/243919086 >> 172.21.15.88:0/243919086 conn(0x7f74f40d24d0 msgr2=0x7f74f40d10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:12.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.823+0000 7f74f9453640 1 -- 172.21.15.88:0/243919086 shutdown_connections 2024-09-16T01:41:12.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.823+0000 7f74f9453640 1 -- 172.21.15.88:0/243919086 wait complete. 2024-09-16T01:41:12.835 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:41:12.835 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:12.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.932+0000 7f67d7cba640 1 Processor -- start 2024-09-16T01:41:12.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.932+0000 7f67d7cba640 1 -- start start 2024-09-16T01:41:12.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.933+0000 7f67d7cba640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f67d015c2b0 0x7f67d00b6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:12.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.933+0000 7f67d7cba640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67d00b72f0 0x7f67d015eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:12.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.933+0000 7f67d7cba640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67d015fb10 con 0x7f67d015f030 2024-09-16T01:41:12.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.933+0000 7f67d7cba640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f67d00b5900 con 0x7f67d015c2b0 2024-09-16T01:41:12.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.933+0000 7f67d7cba640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f67d015f830 con 0x7f67d00b72f0 2024-09-16T01:41:12.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.933+0000 7f67d6257640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f67d015f030 0x7f67d006f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49520/0 (socket says 172.21.15.88:49520) 2024-09-16T01:41:12.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.933+0000 7f67d6257640 1 -- 172.21.15.88:0/559288634 learned_addr learned my addr 172.21.15.88:0/559288634 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:12.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d5a56640 1 --2- 172.21.15.88:0/559288634 >> v2:172.21.15.154:3300/0 conn(0x7f67d015c2b0 0x7f67d00b6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:12.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d5255640 1 --2- 172.21.15.88:0/559288634 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67d00b72f0 0x7f67d015eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:12.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d4a54640 1 -- 172.21.15.88:0/559288634 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1840702297 0 0) 0x7f67d015fb10 con 0x7f67d015f030 2024-09-16T01:41:12.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d4a54640 1 -- 172.21.15.88:0/559288634 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67bc003520 con 0x7f67d015f030 2024-09-16T01:41:12.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d4a54640 1 -- 172.21.15.88:0/559288634 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1546089253 0 0) 0x7f67bc003520 con 0x7f67d015f030 2024-09-16T01:41:12.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d4a54640 1 -- 172.21.15.88:0/559288634 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67d00b72f0 msgr2=0x7f67d015eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:12.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d4a54640 1 --2- 172.21.15.88:0/559288634 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67d00b72f0 0x7f67d015eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d4a54640 1 -- 172.21.15.88:0/559288634 >> v2:172.21.15.154:3300/0 conn(0x7f67d015c2b0 msgr2=0x7f67d00b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:12.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d4a54640 1 --2- 172.21.15.88:0/559288634 >> v2:172.21.15.154:3300/0 conn(0x7f67d015c2b0 0x7f67d00b6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d4a54640 1 -- 172.21.15.88:0/559288634 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d006ce30 con 0x7f67d015f030 2024-09-16T01:41:12.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d4a54640 1 -- 172.21.15.88:0/559288634 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f67c80019f0 con 0x7f67d015f030 2024-09-16T01:41:12.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d4a54640 1 -- 172.21.15.88:0/559288634 >> v1:172.21.15.88:6789/0 conn(0x7f67d015f030 legacy=0x7f67d006f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d4a54640 1 --2- 172.21.15.88:0/559288634 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67d00b72f0 0x7f67bc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d4a54640 1 --2- 172.21.15.88:0/559288634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67bc003f30 0x7f67bc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d4a54640 1 --2- 172.21.15.88:0/559288634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67bc004d40 0x7f67bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d4a54640 1 -- 172.21.15.88:0/559288634 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f67d00b5900 con 0x7f67bc004d40 2024-09-16T01:41:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d4a54640 1 -- 172.21.15.88:0/559288634 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f67d015f830 con 0x7f67bc003f30 2024-09-16T01:41:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d4a54640 1 -- 172.21.15.88:0/559288634 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f67bc00b9c0 con 0x7f67d00b72f0 2024-09-16T01:41:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d6257640 1 --2- 172.21.15.88:0/559288634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67bc004d40 0x7f67bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d5a56640 1 --2- 172.21.15.88:0/559288634 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67d00b72f0 0x7f67bc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d5255640 1 --2- 172.21.15.88:0/559288634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67bc003f30 0x7f67bc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:12.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d6257640 1 -- 172.21.15.88:0/559288634 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67d00b72f0 msgr2=0x7f67bc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:12.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d6257640 1 --2- 172.21.15.88:0/559288634 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67d00b72f0 0x7f67bc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d6257640 1 -- 172.21.15.88:0/559288634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67bc003f30 msgr2=0x7f67bc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:12.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d6257640 1 --2- 172.21.15.88:0/559288634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67bc003f30 0x7f67bc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.934+0000 7f67d6257640 1 -- 172.21.15.88:0/559288634 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67bc003520 con 0x7f67bc004d40 2024-09-16T01:41:12.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.935+0000 7f67d6257640 1 --2- 172.21.15.88:0/559288634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67bc004d40 0x7f67bc0051c0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f67c800a870 tx=0x7f67bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c82168e0f256d1d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:12.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.935+0000 7f67d4a54640 1 -- 172.21.15.88:0/559288634 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c800a4c0 con 0x7f67bc004d40 2024-09-16T01:41:12.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.935+0000 7f67d4a54640 1 -- 172.21.15.88:0/559288634 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67c802f070 con 0x7f67bc004d40 2024-09-16T01:41:12.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.935+0000 7f67d4a54640 1 -- 172.21.15.88:0/559288634 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c802bcb0 con 0x7f67bc004d40 2024-09-16T01:41:12.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.936+0000 7f67d7cba640 1 -- 172.21.15.88:0/559288634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67bc004d40 msgr2=0x7f67bc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:12.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.936+0000 7f67d7cba640 1 --2- 172.21.15.88:0/559288634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67bc004d40 0x7f67bc0051c0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f67c800a870 tx=0x7f67bc005830 comp rx=0 tx=0).stop 2024-09-16T01:41:12.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.936+0000 7f67d6257640 1 -- 172.21.15.88:0/559288634 reap_dead start 2024-09-16T01:41:12.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.936+0000 7f67d7cba640 1 -- 172.21.15.88:0/559288634 shutdown_connections 2024-09-16T01:41:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.936+0000 7f67d7cba640 1 -- 172.21.15.88:0/559288634 >> 172.21.15.88:0/559288634 conn(0x7f67d00b4240 msgr2=0x7f67d00b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.936+0000 7f67d7cba640 1 -- 172.21.15.88:0/559288634 shutdown_connections 2024-09-16T01:41:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.936+0000 7f67d7cba640 1 -- 172.21.15.88:0/559288634 wait complete. 2024-09-16T01:41:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.937+0000 7f67d7cba640 1 Processor -- start 2024-09-16T01:41:12.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.937+0000 7f67d7cba640 1 -- start start 2024-09-16T01:41:12.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.938+0000 7f67d7cba640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67bc004d40 0x7f67d015ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:12.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.938+0000 7f67d7cba640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67d01f3250 0x7f67d01f36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:12.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.938+0000 7f67d7cba640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67d00d1d50 0x7f67d00d21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:12.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.938+0000 7f67d7cba640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f67d00b7730 con 0x7f67bc004d40 2024-09-16T01:41:12.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.938+0000 7f67d7cba640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f67d006d3f0 con 0x7f67d01f3250 2024-09-16T01:41:12.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.938+0000 7f67d7cba640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f67d015f4c0 con 0x7f67d00d1d50 2024-09-16T01:41:12.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.938+0000 7f67d5a56640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67bc004d40 0x7f67d015ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:12.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.938+0000 7f67d5a56640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67bc004d40 0x7f67d015ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41404/0 (socket says 172.21.15.88:41404) 2024-09-16T01:41:12.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.938+0000 7f67d5a56640 1 -- 172.21.15.88:0/3587845657 learned_addr learned my addr 172.21.15.88:0/3587845657 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:12.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.938+0000 7f67d5255640 1 --2- 172.21.15.88:0/3587845657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67d01f3250 0x7f67d01f36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:12.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.938+0000 7f67d6257640 1 --2- 172.21.15.88:0/3587845657 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67d00d1d50 0x7f67d00d21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:12.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.938+0000 7f67d5a56640 1 -- 172.21.15.88:0/3587845657 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67d00d1d50 msgr2=0x7f67d00d21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:12.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.938+0000 7f67d5a56640 1 --2- 172.21.15.88:0/3587845657 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67d00d1d50 0x7f67d00d21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.938+0000 7f67d5a56640 1 -- 172.21.15.88:0/3587845657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67d01f3250 msgr2=0x7f67d01f36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:12.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.938+0000 7f67d5a56640 1 --2- 172.21.15.88:0/3587845657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67d01f3250 0x7f67d01f36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:12.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.938+0000 7f67d5a56640 1 -- 172.21.15.88:0/3587845657 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d01f3bf0 con 0x7f67bc004d40 2024-09-16T01:41:12.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.938+0000 7f67d5a56640 1 --2- 172.21.15.88:0/3587845657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67bc004d40 0x7f67d015ff10 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f67cc011a70 tx=0x7f67cc011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:12.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.939+0000 7f67c67fc640 1 -- 172.21.15.88:0/3587845657 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67cc00fa90 con 0x7f67bc004d40 2024-09-16T01:41:12.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.939+0000 7f67c67fc640 1 -- 172.21.15.88:0/3587845657 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67cc00fc30 con 0x7f67bc004d40 2024-09-16T01:41:12.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.939+0000 7f67d7cba640 1 -- 172.21.15.88:0/3587845657 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67d00aa600 con 0x7f67bc004d40 2024-09-16T01:41:12.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.939+0000 7f67c67fc640 1 -- 172.21.15.88:0/3587845657 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67cc014440 con 0x7f67bc004d40 2024-09-16T01:41:12.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.939+0000 7f67d7cba640 1 -- 172.21.15.88:0/3587845657 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67d00aabc0 con 0x7f67bc004d40 2024-09-16T01:41:12.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.940+0000 7f67d7cba640 1 -- 172.21.15.88:0/3587845657 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67d00b6db0 con 0x7f67bc004d40 2024-09-16T01:41:12.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.940+0000 7f67c67fc640 1 -- 172.21.15.88:0/3587845657 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f67cc012040 con 0x7f67bc004d40 2024-09-16T01:41:12.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.941+0000 7f67c67fc640 1 --2- 172.21.15.88:0/3587845657 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f67b406c300 0x7f67b406e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:12.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.941+0000 7f67c67fc640 1 -- 172.21.15.88:0/3587845657 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6512+0+0 (secure 0 0 0) 0x7f67cc08bcd0 con 0x7f67bc004d40 2024-09-16T01:41:12.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.941+0000 7f67d5255640 1 --2- 172.21.15.88:0/3587845657 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f67b406c300 0x7f67b406e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:12.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.941+0000 7f67d5255640 1 --2- 172.21.15.88:0/3587845657 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f67b406c300 0x7f67b406e7c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f67d006dc40 tx=0x7f67c0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:12.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:12.945+0000 7f67c67fc640 1 -- 172.21.15.88:0/3587845657 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f67cc019030 con 0x7f67bc004d40 2024-09-16T01:41:13.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.141+0000 7f67d7cba640 1 -- 172.21.15.88:0/3587845657 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67d0161d20 con 0x7f67bc004d40 2024-09-16T01:41:13.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.142+0000 7f67c67fc640 1 -- 172.21.15.88:0/3587845657 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+4388 (secure 0 0 0) 0x7f67cc059bd0 con 0x7f67bc004d40 2024-09-16T01:41:13.143 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:13.143 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":122,"btime":"2024-09-16T01:41:12:181725+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":7866,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/293764774","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":293764774},{"type":"v1","addr":"172.21.15.88:6837","nonce":293764774}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":108}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:12.181665+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:41:13.143 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 122 2024-09-16T01:41:13.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.144+0000 7f67d7cba640 1 -- 172.21.15.88:0/3587845657 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f67b406c300 msgr2=0x7f67b406e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:13.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.144+0000 7f67d7cba640 1 --2- 172.21.15.88:0/3587845657 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f67b406c300 0x7f67b406e7c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f67d006dc40 tx=0x7f67c0009190 comp rx=0 tx=0).stop 2024-09-16T01:41:13.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.144+0000 7f67d7cba640 1 -- 172.21.15.88:0/3587845657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67bc004d40 msgr2=0x7f67d015ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:13.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.144+0000 7f67d7cba640 1 --2- 172.21.15.88:0/3587845657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67bc004d40 0x7f67d015ff10 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f67cc011a70 tx=0x7f67cc011e70 comp rx=0 tx=0).stop 2024-09-16T01:41:13.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.144+0000 7f67d7cba640 1 -- 172.21.15.88:0/3587845657 shutdown_connections 2024-09-16T01:41:13.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.144+0000 7f67d7cba640 1 --2- 172.21.15.88:0/3587845657 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f67b406c300 0x7f67b406e7c0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:13.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.144+0000 7f67d7cba640 1 --2- 172.21.15.88:0/3587845657 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67d00d1d50 0x7f67d00d21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:13.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.144+0000 7f67d7cba640 1 --2- 172.21.15.88:0/3587845657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67d01f3250 0x7f67d01f36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:13.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.144+0000 7f67d7cba640 1 --2- 172.21.15.88:0/3587845657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67bc004d40 0x7f67d015ff10 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:13.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.144+0000 7f67d7cba640 1 -- 172.21.15.88:0/3587845657 >> 172.21.15.88:0/3587845657 conn(0x7f67d00b4240 msgr2=0x7f67d01612d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:13.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.144+0000 7f67d7cba640 1 -- 172.21.15.88:0/3587845657 shutdown_connections 2024-09-16T01:41:13.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.144+0000 7f67d7cba640 1 -- 172.21.15.88:0/3587845657 wait complete. 2024-09-16T01:41:13.156 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete mds_snaptable' led to DAMAGED state 2024-09-16T01:41:13.156 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete mds0_inotable' 2024-09-16T01:41:13.156 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:41:13.156 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:41:13.173 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:41:13.173 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:41:13.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:13.173 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:41:13.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:41:13.229 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:41:13.229 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:41:13.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.367+0000 7f292d477640 1 Processor -- start 2024-09-16T01:41:13.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.367+0000 7f292d477640 1 -- start start 2024-09-16T01:41:13.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.368+0000 7f292d477640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29280cf1f0 0x7f29280b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:13.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.368+0000 7f292d477640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29280cf860 con 0x7f292815e7a0 2024-09-16T01:41:13.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.368+0000 7f292d477640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29280cfb40 con 0x7f29280b6720 2024-09-16T01:41:13.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.368+0000 7f292d477640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f29280d2ca0 con 0x7f29280cf1f0 2024-09-16T01:41:13.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.368+0000 7f2926ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f292815e7a0 0x7f292806caa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49532/0 (socket says 172.21.15.88:49532) 2024-09-16T01:41:13.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.368+0000 7f2926ffd640 1 -- 172.21.15.88:0/1567462845 learned_addr learned my addr 172.21.15.88:0/1567462845 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:13.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.368+0000 7f291e7fc640 1 --2- 172.21.15.88:0/1567462845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29280cf1f0 0x7f29280b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:13.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.368+0000 7f29267fc640 1 -- 172.21.15.88:0/1567462845 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2981973387 0 0) 0x7f29280cf860 con 0x7f292815e7a0 2024-09-16T01:41:13.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.368+0000 7f29267fc640 1 -- 172.21.15.88:0/1567462845 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2914003520 con 0x7f292815e7a0 2024-09-16T01:41:13.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.368+0000 7f29267fc640 1 -- 172.21.15.88:0/1567462845 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1695285508 0 0) 0x7f2914003520 con 0x7f292815e7a0 2024-09-16T01:41:13.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.368+0000 7f29267fc640 1 -- 172.21.15.88:0/1567462845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29280cf1f0 msgr2=0x7f29280b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:13.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.368+0000 7f29267fc640 1 --2- 172.21.15.88:0/1567462845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29280cf1f0 0x7f29280b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:13.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.368+0000 7f29267fc640 1 -- 172.21.15.88:0/1567462845 >> v1:172.21.15.154:6789/0 conn(0x7f29280b6720 legacy=0x7f29280d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:13.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.368+0000 7f29267fc640 1 -- 172.21.15.88:0/1567462845 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29280cfe20 con 0x7f292815e7a0 2024-09-16T01:41:13.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.368+0000 7f29267fc640 1 -- 172.21.15.88:0/1567462845 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f2918001670 con 0x7f292815e7a0 2024-09-16T01:41:13.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.369+0000 7f29267fc640 1 -- 172.21.15.88:0/1567462845 >> v1:172.21.15.88:6789/0 conn(0x7f292815e7a0 legacy=0x7f292806caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:13.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.369+0000 7f29267fc640 1 --2- 172.21.15.88:0/1567462845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29280cf1f0 0x7f29140039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:13.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.369+0000 7f29267fc640 1 --2- 172.21.15.88:0/1567462845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2914003f20 0x7f2914004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:13.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.369+0000 7f29267fc640 1 --2- 172.21.15.88:0/1567462845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2914004d40 0x7f29140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:13.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.369+0000 7f29267fc640 1 -- 172.21.15.88:0/1567462845 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f29280d2ca0 con 0x7f2914004d40 2024-09-16T01:41:13.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.369+0000 7f29267fc640 1 -- 172.21.15.88:0/1567462845 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f291400bad0 con 0x7f2914003f20 2024-09-16T01:41:13.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.369+0000 7f29267fc640 1 -- 172.21.15.88:0/1567462845 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f291400bcd0 con 0x7f29280cf1f0 2024-09-16T01:41:13.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.369+0000 7f29277fe640 1 --2- 172.21.15.88:0/1567462845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2914004d40 0x7f29140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:13.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.369+0000 7f291e7fc640 1 --2- 172.21.15.88:0/1567462845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2914003f20 0x7f2914004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:13.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.369+0000 7f2926ffd640 1 --2- 172.21.15.88:0/1567462845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29280cf1f0 0x7f29140039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:13.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.369+0000 7f29277fe640 1 -- 172.21.15.88:0/1567462845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29280cf1f0 msgr2=0x7f29140039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:13.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.369+0000 7f29277fe640 1 --2- 172.21.15.88:0/1567462845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29280cf1f0 0x7f29140039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:13.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.369+0000 7f29277fe640 1 -- 172.21.15.88:0/1567462845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2914003f20 msgr2=0x7f2914004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:13.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.369+0000 7f29277fe640 1 --2- 172.21.15.88:0/1567462845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2914003f20 0x7f2914004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:13.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.369+0000 7f29277fe640 1 -- 172.21.15.88:0/1567462845 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2914003520 con 0x7f2914004d40 2024-09-16T01:41:13.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.370+0000 7f29277fe640 1 --2- 172.21.15.88:0/1567462845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2914004d40 0x7f29140051c0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f2920002010 tx=0x7f2914005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d507d72e1764284d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:13.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.370+0000 7f29267fc640 1 -- 172.21.15.88:0/1567462845 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2920036070 con 0x7f2914004d40 2024-09-16T01:41:13.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.370+0000 7f29267fc640 1 -- 172.21.15.88:0/1567462845 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2920031600 con 0x7f2914004d40 2024-09-16T01:41:13.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.370+0000 7f29267fc640 1 -- 172.21.15.88:0/1567462845 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2920001be0 con 0x7f2914004d40 2024-09-16T01:41:13.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.370+0000 7f292d477640 1 -- 172.21.15.88:0/1567462845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2914004d40 msgr2=0x7f29140051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:13.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.370+0000 7f292d477640 1 --2- 172.21.15.88:0/1567462845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2914004d40 0x7f29140051c0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f2920002010 tx=0x7f2914005830 comp rx=0 tx=0).stop 2024-09-16T01:41:13.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.370+0000 7f29277fe640 1 -- 172.21.15.88:0/1567462845 reap_dead start 2024-09-16T01:41:13.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.370+0000 7f292d477640 1 -- 172.21.15.88:0/1567462845 shutdown_connections 2024-09-16T01:41:13.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.370+0000 7f292d477640 1 -- 172.21.15.88:0/1567462845 >> 172.21.15.88:0/1567462845 conn(0x7f29280d11e0 msgr2=0x7f29280d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:13.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.370+0000 7f292d477640 1 -- 172.21.15.88:0/1567462845 shutdown_connections 2024-09-16T01:41:13.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.370+0000 7f292d477640 1 -- 172.21.15.88:0/1567462845 wait complete. 2024-09-16T01:41:13.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.372+0000 7f292d477640 1 Processor -- start 2024-09-16T01:41:13.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.372+0000 7f292d477640 1 -- start start 2024-09-16T01:41:13.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.372+0000 7f292d477640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2914004d40 0x7f2928152a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:13.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.372+0000 7f292d477640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2928158620 0x7f29280d4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:13.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.372+0000 7f292d477640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2928158aa0 0x7f29280d48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:13.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.372+0000 7f292d477640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f292815eb80 con 0x7f2928158aa0 2024-09-16T01:41:13.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.372+0000 7f292d477640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f29280b7a00 con 0x7f2914004d40 2024-09-16T01:41:13.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.372+0000 7f292d477640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f292815ed00 con 0x7f2928158620 2024-09-16T01:41:13.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.373+0000 7f29277fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2928158aa0 0x7f29280d48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:13.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.373+0000 7f29277fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2928158aa0 0x7f29280d48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41422/0 (socket says 172.21.15.88:41422) 2024-09-16T01:41:13.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.373+0000 7f29277fe640 1 -- 172.21.15.88:0/3250770095 learned_addr learned my addr 172.21.15.88:0/3250770095 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:13.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.373+0000 7f2926ffd640 1 --2- 172.21.15.88:0/3250770095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2914004d40 0x7f2928152a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:13.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.373+0000 7f291e7fc640 1 --2- 172.21.15.88:0/3250770095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2928158620 0x7f29280d4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:13.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.373+0000 7f29277fe640 1 -- 172.21.15.88:0/3250770095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2928158620 msgr2=0x7f29280d4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:13.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.373+0000 7f29277fe640 1 --2- 172.21.15.88:0/3250770095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2928158620 0x7f29280d4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:13.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.373+0000 7f29277fe640 1 -- 172.21.15.88:0/3250770095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2914004d40 msgr2=0x7f2928152a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:13.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.373+0000 7f29277fe640 1 --2- 172.21.15.88:0/3250770095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2914004d40 0x7f2928152a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:13.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.373+0000 7f29277fe640 1 -- 172.21.15.88:0/3250770095 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f292815f1d0 con 0x7f2928158aa0 2024-09-16T01:41:13.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.373+0000 7f291e7fc640 1 --2- 172.21.15.88:0/3250770095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2928158620 0x7f29280d4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:13.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.373+0000 7f29277fe640 1 --2- 172.21.15.88:0/3250770095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2928158aa0 0x7f29280d48a0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f292002aab0 tx=0x7f2920001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:13.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.373+0000 7f291ffff640 1 -- 172.21.15.88:0/3250770095 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2920036040 con 0x7f2928158aa0 2024-09-16T01:41:13.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.373+0000 7f291ffff640 1 -- 172.21.15.88:0/3250770095 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2920003970 con 0x7f2928158aa0 2024-09-16T01:41:13.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.373+0000 7f291ffff640 1 -- 172.21.15.88:0/3250770095 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2920007df0 con 0x7f2928158aa0 2024-09-16T01:41:13.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.373+0000 7f292d477640 1 -- 172.21.15.88:0/3250770095 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f292815f4b0 con 0x7f2928158aa0 2024-09-16T01:41:13.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.373+0000 7f292d477640 1 -- 172.21.15.88:0/3250770095 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f292815fa70 con 0x7f2928158aa0 2024-09-16T01:41:13.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.374+0000 7f292d477640 1 -- 172.21.15.88:0/3250770095 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29280b61e0 con 0x7f2928158aa0 2024-09-16T01:41:13.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.375+0000 7f291ffff640 1 -- 172.21.15.88:0/3250770095 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2920005070 con 0x7f2928158aa0 2024-09-16T01:41:13.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.375+0000 7f291ffff640 1 --2- 172.21.15.88:0/3250770095 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f290806c2b0 0x7f290806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:13.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.375+0000 7f291ffff640 1 -- 172.21.15.88:0/3250770095 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6512+0+0 (secure 0 0 0) 0x7f29200ab990 con 0x7f2928158aa0 2024-09-16T01:41:13.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.375+0000 7f2926ffd640 1 --2- 172.21.15.88:0/3250770095 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f290806c2b0 0x7f290806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:13.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.376+0000 7f2926ffd640 1 --2- 172.21.15.88:0/3250770095 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f290806c2b0 0x7f290806e770 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f2918000c00 tx=0x7f2918001bb0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:13.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.380+0000 7f291ffff640 1 -- 172.21.15.88:0/3250770095 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2920079820 con 0x7f2928158aa0 2024-09-16T01:41:13.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:13.581+0000 7f292d477640 1 -- 172.21.15.88:0/3250770095 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f29280d2a80 con 0x7f2928158aa0 2024-09-16T01:41:14.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.184+0000 7f291ffff640 1 -- 172.21.15.88:0/3250770095 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v123) ==== 161+0+0 (secure 0 0 0) 0x7f292007db20 con 0x7f2928158aa0 2024-09-16T01:41:14.186 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:41:14.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.188+0000 7f292d477640 1 -- 172.21.15.88:0/3250770095 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f290806c2b0 msgr2=0x7f290806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:14.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.188+0000 7f292d477640 1 --2- 172.21.15.88:0/3250770095 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f290806c2b0 0x7f290806e770 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f2918000c00 tx=0x7f2918001bb0 comp rx=0 tx=0).stop 2024-09-16T01:41:14.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.188+0000 7f292d477640 1 -- 172.21.15.88:0/3250770095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2928158aa0 msgr2=0x7f29280d48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:14.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.188+0000 7f292d477640 1 --2- 172.21.15.88:0/3250770095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2928158aa0 0x7f29280d48a0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f292002aab0 tx=0x7f2920001630 comp rx=0 tx=0).stop 2024-09-16T01:41:14.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.188+0000 7f292d477640 1 -- 172.21.15.88:0/3250770095 shutdown_connections 2024-09-16T01:41:14.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.188+0000 7f292d477640 1 --2- 172.21.15.88:0/3250770095 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f290806c2b0 0x7f290806e770 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:14.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.188+0000 7f292d477640 1 --2- 172.21.15.88:0/3250770095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2928158aa0 0x7f29280d48a0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:14.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.188+0000 7f292d477640 1 --2- 172.21.15.88:0/3250770095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2928158620 0x7f29280d4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:14.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.188+0000 7f292d477640 1 --2- 172.21.15.88:0/3250770095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2914004d40 0x7f2928152a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:14.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.188+0000 7f292d477640 1 -- 172.21.15.88:0/3250770095 >> 172.21.15.88:0/3250770095 conn(0x7f29280d11e0 msgr2=0x7f292815c920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:14.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.188+0000 7f292d477640 1 -- 172.21.15.88:0/3250770095 shutdown_connections 2024-09-16T01:41:14.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.188+0000 7f292d477640 1 -- 172.21.15.88:0/3250770095 wait complete. 2024-09-16T01:41:14.203 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:41:14.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.305+0000 7fe3569d6640 1 Processor -- start 2024-09-16T01:41:14.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.305+0000 7fe3569d6640 1 -- start start 2024-09-16T01:41:14.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.306+0000 7fe3569d6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe35015e610 0x7fe35006f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:14.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.306+0000 7fe3569d6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe35006bd40 con 0x7fe35006fc50 2024-09-16T01:41:14.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.306+0000 7fe3569d6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe35006c020 con 0x7fe350073f30 2024-09-16T01:41:14.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.306+0000 7fe3569d6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe3500700c0 con 0x7fe35015e610 2024-09-16T01:41:14.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.306+0000 7fe34ffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe35015e610 0x7fe35006f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:14.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.306+0000 7fe34ffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe35015e610 0x7fe35006f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41434/0 (socket says 172.21.15.88:41434) 2024-09-16T01:41:14.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.306+0000 7fe34ffff640 1 -- 172.21.15.88:0/1863783873 learned_addr learned my addr 172.21.15.88:0/1863783873 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:14.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.306+0000 7fe34f7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe35006fc50 0x7fe350073820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49548/0 (socket says 172.21.15.88:49548) 2024-09-16T01:41:14.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.306+0000 7fe34ffff640 1 -- 172.21.15.88:0/1863783873 >> v1:172.21.15.154:6789/0 conn(0x7fe350073f30 legacy=0x7fe3500ae990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:14.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.306+0000 7fe34ffff640 1 -- 172.21.15.88:0/1863783873 >> v1:172.21.15.88:6789/0 conn(0x7fe35006fc50 legacy=0x7fe350073820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:14.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.307+0000 7fe34effd640 1 -- 172.21.15.88:0/1863783873 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3016663874 0 0) 0x7fe35006bd40 con 0x7fe35006fc50 2024-09-16T01:41:14.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.307+0000 7fe34ffff640 1 -- 172.21.15.88:0/1863783873 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe35006c300 con 0x7fe35015e610 2024-09-16T01:41:14.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.307+0000 7fe34ffff640 1 --2- 172.21.15.88:0/1863783873 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe35015e610 0x7fe35006f710 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fe340009870 tx=0x7fe34002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9843d8fe3ba95ccc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:14.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.307+0000 7fe34effd640 1 -- 172.21.15.88:0/1863783873 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe340037070 con 0x7fe35015e610 2024-09-16T01:41:14.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.307+0000 7fe34effd640 1 -- 172.21.15.88:0/1863783873 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe340032440 con 0x7fe35015e610 2024-09-16T01:41:14.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.307+0000 7fe34effd640 1 -- 172.21.15.88:0/1863783873 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe340032810 con 0x7fe35015e610 2024-09-16T01:41:14.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.308+0000 7fe3569d6640 1 -- 172.21.15.88:0/1863783873 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe35015e610 msgr2=0x7fe35006f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:14.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.308+0000 7fe3569d6640 1 --2- 172.21.15.88:0/1863783873 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe35015e610 0x7fe35006f710 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fe340009870 tx=0x7fe34002a210 comp rx=0 tx=0).stop 2024-09-16T01:41:14.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.308+0000 7fe3569d6640 1 -- 172.21.15.88:0/1863783873 shutdown_connections 2024-09-16T01:41:14.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.308+0000 7fe3569d6640 1 --2- 172.21.15.88:0/1863783873 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe35015e610 0x7fe35006f710 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:14.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.308+0000 7fe3569d6640 1 -- 172.21.15.88:0/1863783873 >> 172.21.15.88:0/1863783873 conn(0x7fe3500b3070 msgr2=0x7fe3500b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:14.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.308+0000 7fe3569d6640 1 -- 172.21.15.88:0/1863783873 shutdown_connections 2024-09-16T01:41:14.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.308+0000 7fe3569d6640 1 -- 172.21.15.88:0/1863783873 wait complete. 2024-09-16T01:41:14.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.309+0000 7fe3569d6640 1 Processor -- start 2024-09-16T01:41:14.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.309+0000 7fe3569d6640 1 -- start start 2024-09-16T01:41:14.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.310+0000 7fe3569d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe35006fc50 0x7fe350149df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:14.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.310+0000 7fe3569d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe350073f30 0x7fe3501598f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:14.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.310+0000 7fe3569d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe35015e610 0x7fe350158f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:14.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.310+0000 7fe3569d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe3500af0a0 con 0x7fe35015e610 2024-09-16T01:41:14.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.310+0000 7fe3569d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe35006c8c0 con 0x7fe350073f30 2024-09-16T01:41:14.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.310+0000 7fe3569d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe3500af220 con 0x7fe35006fc50 2024-09-16T01:41:14.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.310+0000 7fe354f73640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe35015e610 0x7fe350158f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:14.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.310+0000 7fe354f73640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe35015e610 0x7fe350158f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41444/0 (socket says 172.21.15.88:41444) 2024-09-16T01:41:14.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.310+0000 7fe354f73640 1 -- 172.21.15.88:0/2227375850 learned_addr learned my addr 172.21.15.88:0/2227375850 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:14.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.310+0000 7fe34f7fe640 1 --2- 172.21.15.88:0/2227375850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe350073f30 0x7fe3501598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:14.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.310+0000 7fe34ffff640 1 --2- 172.21.15.88:0/2227375850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe35006fc50 0x7fe350149df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:14.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.310+0000 7fe354f73640 1 -- 172.21.15.88:0/2227375850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe35006fc50 msgr2=0x7fe350149df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:14.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.310+0000 7fe354f73640 1 --2- 172.21.15.88:0/2227375850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe35006fc50 0x7fe350149df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:14.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.310+0000 7fe354f73640 1 -- 172.21.15.88:0/2227375850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe350073f30 msgr2=0x7fe3501598f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:14.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.310+0000 7fe354f73640 1 --2- 172.21.15.88:0/2227375850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe350073f30 0x7fe3501598f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:14.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.310+0000 7fe354f73640 1 -- 172.21.15.88:0/2227375850 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe35014a330 con 0x7fe35015e610 2024-09-16T01:41:14.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.310+0000 7fe34ffff640 1 --2- 172.21.15.88:0/2227375850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe35006fc50 0x7fe350149df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:14.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.311+0000 7fe354f73640 1 --2- 172.21.15.88:0/2227375850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe35015e610 0x7fe350158f40 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fe34800c7b0 tx=0x7fe34800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:14.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.311+0000 7fe34cff9640 1 -- 172.21.15.88:0/2227375850 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe348017070 con 0x7fe35015e610 2024-09-16T01:41:14.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.311+0000 7fe34cff9640 1 -- 172.21.15.88:0/2227375850 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe348003da0 con 0x7fe35015e610 2024-09-16T01:41:14.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.311+0000 7fe34cff9640 1 -- 172.21.15.88:0/2227375850 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3480052b0 con 0x7fe35015e610 2024-09-16T01:41:14.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.311+0000 7fe3569d6640 1 -- 172.21.15.88:0/2227375850 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe350159e30 con 0x7fe35015e610 2024-09-16T01:41:14.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.311+0000 7fe3569d6640 1 -- 172.21.15.88:0/2227375850 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe350157ea0 con 0x7fe35015e610 2024-09-16T01:41:14.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.312+0000 7fe3569d6640 1 -- 172.21.15.88:0/2227375850 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe350070460 con 0x7fe35015e610 2024-09-16T01:41:14.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.312+0000 7fe34cff9640 1 -- 172.21.15.88:0/2227375850 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe34801a070 con 0x7fe35015e610 2024-09-16T01:41:14.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.313+0000 7fe34cff9640 1 --2- 172.21.15.88:0/2227375850 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe33c06c2b0 0x7fe33c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:14.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.313+0000 7fe34cff9640 1 -- 172.21.15.88:0/2227375850 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6512+0+0 (secure 0 0 0) 0x7fe34808ade0 con 0x7fe35015e610 2024-09-16T01:41:14.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.313+0000 7fe34ffff640 1 --2- 172.21.15.88:0/2227375850 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe33c06c2b0 0x7fe33c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:14.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.313+0000 7fe34ffff640 1 --2- 172.21.15.88:0/2227375850 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe33c06c2b0 0x7fe33c06e770 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe340034000 tx=0x7fe34002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:14.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.317+0000 7fe34cff9640 1 -- 172.21.15.88:0/2227375850 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe350070460 con 0x7fe35015e610 2024-09-16T01:41:14.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:14.548+0000 7fe3569d6640 1 -- 172.21.15.88:0/2227375850 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fe3500b2d40 con 0x7fe35015e610 2024-09-16T01:41:15.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.194+0000 7fe34cff9640 1 -- 172.21.15.88:0/2227375850 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v124) ==== 101+0+0 (secure 0 0 0) 0x7fe35014a330 con 0x7fe35015e610 2024-09-16T01:41:15.195 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:41:15.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.196+0000 7fe3569d6640 1 -- 172.21.15.88:0/2227375850 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe33c06c2b0 msgr2=0x7fe33c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:15.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.196+0000 7fe3569d6640 1 --2- 172.21.15.88:0/2227375850 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe33c06c2b0 0x7fe33c06e770 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe340034000 tx=0x7fe34002e000 comp rx=0 tx=0).stop 2024-09-16T01:41:15.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.196+0000 7fe3569d6640 1 -- 172.21.15.88:0/2227375850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe35015e610 msgr2=0x7fe350158f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:15.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.196+0000 7fe3569d6640 1 --2- 172.21.15.88:0/2227375850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe35015e610 0x7fe350158f40 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fe34800c7b0 tx=0x7fe34800cc70 comp rx=0 tx=0).stop 2024-09-16T01:41:15.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.196+0000 7fe3569d6640 1 -- 172.21.15.88:0/2227375850 shutdown_connections 2024-09-16T01:41:15.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.196+0000 7fe3569d6640 1 --2- 172.21.15.88:0/2227375850 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe33c06c2b0 0x7fe33c06e770 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.196+0000 7fe3569d6640 1 --2- 172.21.15.88:0/2227375850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe35015e610 0x7fe350158f40 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.196+0000 7fe3569d6640 1 --2- 172.21.15.88:0/2227375850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe350073f30 0x7fe3501598f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.196+0000 7fe3569d6640 1 --2- 172.21.15.88:0/2227375850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe35006fc50 0x7fe350149df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.196+0000 7fe3569d6640 1 -- 172.21.15.88:0/2227375850 >> 172.21.15.88:0/2227375850 conn(0x7fe3500b3070 msgr2=0x7fe3500b13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:15.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.197+0000 7fe3569d6640 1 -- 172.21.15.88:0/2227375850 shutdown_connections 2024-09-16T01:41:15.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.197+0000 7fe3569d6640 1 -- 172.21.15.88:0/2227375850 wait complete. 2024-09-16T01:41:15.211 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:41:15.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.256+0000 7f5078348f40 1 Processor -- start 2024-09-16T01:41:15.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.256+0000 7f5078348f40 1 -- start start 2024-09-16T01:41:15.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.257+0000 7f5078348f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50640022d0 0x55c6609a6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.257+0000 7f5078348f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6608bf710 0x55c6609972a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.257+0000 7f5078348f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c6608fb400 0x55c6608d2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.257+0000 7f5078348f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55c660822510 con 0x55c6608fb400 2024-09-16T01:41:15.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.257+0000 7f5078348f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c66087bc50 con 0x55c6608bf710 2024-09-16T01:41:15.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.257+0000 7f5078348f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55c660996ac0 con 0x7f50640022d0 2024-09-16T01:41:15.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.257+0000 7f507663f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c6608fb400 0x55c6608d2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.257+0000 7f507663f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c6608fb400 0x55c6608d2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41468/0 (socket says 172.21.15.88:41468) 2024-09-16T01:41:15.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.257+0000 7f507663f640 1 -- 172.21.15.88:0/800882658 learned_addr learned my addr 172.21.15.88:0/800882658 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:15.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.257+0000 7f506ffff640 1 --2- 172.21.15.88:0/800882658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6608bf710 0x55c6609972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.257+0000 7f507663f640 1 -- 172.21.15.88:0/800882658 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50640022d0 msgr2=0x55c6609a6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:15.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.257+0000 7f5075e3e640 1 --2- 172.21.15.88:0/800882658 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50640022d0 0x55c6609a6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.257+0000 7f507663f640 1 --2- 172.21.15.88:0/800882658 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50640022d0 0x55c6609a6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.257+0000 7f507663f640 1 -- 172.21.15.88:0/800882658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6608bf710 msgr2=0x55c6609972a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:15.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.257+0000 7f507663f640 1 --2- 172.21.15.88:0/800882658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6608bf710 0x55c6609972a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.257+0000 7f507663f640 1 -- 172.21.15.88:0/800882658 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6608d38b0 con 0x55c6608fb400 2024-09-16T01:41:15.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.257+0000 7f506ffff640 1 --2- 172.21.15.88:0/800882658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6608bf710 0x55c6609972a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:15.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.258+0000 7f507663f640 1 --2- 172.21.15.88:0/800882658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c6608fb400 0x55c6608d2c70 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x55c66091c690 tx=0x7f507000e670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28c07fa6c8f7c6ba server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:15.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.258+0000 7f5074e3c640 1 -- 172.21.15.88:0/800882658 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507001a070 con 0x55c6608fb400 2024-09-16T01:41:15.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.258+0000 7f5074e3c640 1 -- 172.21.15.88:0/800882658 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5070013460 con 0x55c6608fb400 2024-09-16T01:41:15.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.258+0000 7f5074e3c640 1 -- 172.21.15.88:0/800882658 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5070013850 con 0x55c6608fb400 2024-09-16T01:41:15.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.258+0000 7f5078348f40 1 -- 172.21.15.88:0/800882658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c6608fb400 msgr2=0x55c6608d2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:15.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.258+0000 7f5078348f40 1 --2- 172.21.15.88:0/800882658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c6608fb400 0x55c6608d2c70 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x55c66091c690 tx=0x7f507000e670 comp rx=0 tx=0).stop 2024-09-16T01:41:15.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.258+0000 7f5078348f40 1 -- 172.21.15.88:0/800882658 shutdown_connections 2024-09-16T01:41:15.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.258+0000 7f5078348f40 1 --2- 172.21.15.88:0/800882658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c6608fb400 0x55c6608d2c70 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.258+0000 7f5078348f40 1 --2- 172.21.15.88:0/800882658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6608bf710 0x55c6609972a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.258+0000 7f5078348f40 1 --2- 172.21.15.88:0/800882658 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50640022d0 0x55c6609a6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.258+0000 7f5078348f40 1 -- 172.21.15.88:0/800882658 >> 172.21.15.88:0/800882658 conn(0x55c66091b750 msgr2=0x55c6608ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:15.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.258+0000 7f5078348f40 1 -- 172.21.15.88:0/800882658 shutdown_connections 2024-09-16T01:41:15.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.258+0000 7f5078348f40 1 -- 172.21.15.88:0/800882658 wait complete. 2024-09-16T01:41:15.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.259+0000 7f5078348f40 1 Processor -- start 2024-09-16T01:41:15.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.259+0000 7f5078348f40 1 -- start start 2024-09-16T01:41:15.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.260+0000 7f5078348f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6608fb400 0x55c6609a3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.260+0000 7f5078348f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50640022d0 0x55c660a3e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.260+0000 7f5078348f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c660a3c650 0x55c6608cf120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.260+0000 7f5078348f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55c6608fc760 con 0x55c660a3c650 2024-09-16T01:41:15.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.260+0000 7f5078348f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c66091a320 con 0x55c6608fb400 2024-09-16T01:41:15.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.260+0000 7f5078348f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55c6609a6670 con 0x7f50640022d0 2024-09-16T01:41:15.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.260+0000 7f507663f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c660a3c650 0x55c6608cf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.260+0000 7f507663f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c660a3c650 0x55c6608cf120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41478/0 (socket says 172.21.15.88:41478) 2024-09-16T01:41:15.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.260+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 learned_addr learned my addr 172.21.15.88:0/33576807 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:15.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.260+0000 7f506ffff640 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50640022d0 0x55c660a3e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.260+0000 7f5075e3e640 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6608fb400 0x55c6609a3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.260+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50640022d0 msgr2=0x55c660a3e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:15.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.260+0000 7f507663f640 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f50640022d0 0x55c660a3e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.260+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6608fb400 msgr2=0x55c6609a3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:15.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.260+0000 7f507663f640 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c6608fb400 0x55c6609a3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.260+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6609a0bb0 con 0x55c660a3c650 2024-09-16T01:41:15.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.260+0000 7f507663f640 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c660a3c650 0x55c6608cf120 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x55c6608ffbc0 tx=0x7f5070003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:15.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.260+0000 7f506effd640 1 -- 172.21.15.88:0/33576807 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507001a040 con 0x55c660a3c650 2024-09-16T01:41:15.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.260+0000 7f506effd640 1 -- 172.21.15.88:0/33576807 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5070026070 con 0x55c660a3c650 2024-09-16T01:41:15.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.261+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c660a3e820 con 0x55c660a3c650 2024-09-16T01:41:15.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.261+0000 7f506effd640 1 -- 172.21.15.88:0/33576807 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507001f410 con 0x55c660a3c650 2024-09-16T01:41:15.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.261+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6609a3dd0 con 0x55c660a3c650 2024-09-16T01:41:15.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.262+0000 7f506effd640 1 -- 172.21.15.88:0/33576807 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5070011070 con 0x55c660a3c650 2024-09-16T01:41:15.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.262+0000 7f506effd640 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f505806ac90 0x7f505806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.262+0000 7f506effd640 1 -- 172.21.15.88:0/33576807 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6512+0+0 (secure 0 0 0) 0x7f507008b120 con 0x55c660a3c650 2024-09-16T01:41:15.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.262+0000 7f5075e3e640 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f505806ac90 0x7f505806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.270 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:41:15.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.263+0000 7f5078348f40 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55c660a44ab0 0x55c660a46ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.263+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a473e0 con 0x55c660a44ab0 2024-09-16T01:41:15.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.263+0000 7f506ffff640 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55c660a44ab0 0x55c660a46ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.263+0000 7f5075e3e640 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f505806ac90 0x7f505806d150 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5068055000 tx=0x7f5068052000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:15.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.263+0000 7f506ffff640 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55c660a44ab0 0x55c660a46ea0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:15.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.265+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v96'257 uv231 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.265+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) -- 0x55c660a47780 con 0x55c660a44ab0 2024-09-16T01:41:15.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.266+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v96'258 uv258 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.266+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a47b20 con 0x55c660a44ab0 2024-09-16T01:41:15.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.267+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v96'259 uv259 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.273 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:41:15.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.268+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48ed0 con 0x55c660a44ab0 2024-09-16T01:41:15.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.269+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v96'260 uv260 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.269+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a49270 con 0x55c660a44ab0 2024-09-16T01:41:15.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.270+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v96'261 uv261 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.270+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48370 con 0x55c660a44ab0 2024-09-16T01:41:15.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.271+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v96'262 uv262 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.271+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48780 con 0x55c660a44ab0 2024-09-16T01:41:15.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.272+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v96'263 uv237 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.272+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48780 con 0x55c660a44ab0 2024-09-16T01:41:15.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.274+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v96'264 uv264 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.274+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48780 con 0x55c660a44ab0 2024-09-16T01:41:15.276 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:41:15.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.275+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v96'265 uv265 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.275+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48780 con 0x55c660a44ab0 2024-09-16T01:41:15.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.276+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v96'266 uv266 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.276+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48780 con 0x55c660a44ab0 2024-09-16T01:41:15.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.277+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v96'267 uv267 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.277+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48780 con 0x55c660a44ab0 2024-09-16T01:41:15.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.278+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v96'268 uv268 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.278+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48780 con 0x55c660a44ab0 2024-09-16T01:41:15.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.280+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v96'269 uv244 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.280+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48780 con 0x55c660a44ab0 2024-09-16T01:41:15.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.281+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v96'270 uv270 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.281+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48780 con 0x55c660a44ab0 2024-09-16T01:41:15.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.282+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v96'271 uv271 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.283 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:41:15.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.282+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48780 con 0x55c660a44ab0 2024-09-16T01:41:15.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.283+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v96'272 uv272 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.283+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48780 con 0x55c660a44ab0 2024-09-16T01:41:15.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.284+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v96'273 uv273 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.284+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48780 con 0x55c660a44ab0 2024-09-16T01:41:15.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.285+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v96'274 uv274 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.285+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48780 con 0x55c660a44ab0 2024-09-16T01:41:15.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.286+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v96'275 uv275 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.286+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48780 con 0x55c660a44ab0 2024-09-16T01:41:15.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.288+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v96'276 uv250 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.288+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48780 con 0x55c660a44ab0 2024-09-16T01:41:15.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.289+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v96'277 uv277 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.289+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48780 con 0x55c660a44ab0 2024-09-16T01:41:15.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.290+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v96'278 uv278 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.291 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:41:15.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.290+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48780 con 0x55c660a44ab0 2024-09-16T01:41:15.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.291+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v96'279 uv279 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.291+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48780 con 0x55c660a44ab0 2024-09-16T01:41:15.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.292+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v96'280 uv280 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.292+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48780 con 0x55c660a44ab0 2024-09-16T01:41:15.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.294+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v96'281 uv281 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.294+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48780 con 0x55c660a44ab0 2024-09-16T01:41:15.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.295+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v96'282 uv256 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.295+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48780 con 0x55c660a44ab0 2024-09-16T01:41:15.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.296+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v96'283 uv283 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.296+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48780 con 0x55c660a44ab0 2024-09-16T01:41:15.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.297+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v96'284 uv284 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.299 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:41:15.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.298+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48780 con 0x55c660a44ab0 2024-09-16T01:41:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.299+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v96'285 uv285 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.299+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48780 con 0x55c660a44ab0 2024-09-16T01:41:15.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.300+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v96'286 uv286 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.300+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a48780 con 0x55c660a44ab0 2024-09-16T01:41:15.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.301+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v96'287 uv287 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.301+0000 7f5078348f40 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55c660a49b20 0x55c660a4bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.301+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a4c450 con 0x55c660a49b20 2024-09-16T01:41:15.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.301+0000 7f5075e3e640 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55c660a49b20 0x55c660a4bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.302+0000 7f5075e3e640 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55c660a49b20 0x55c660a4bf10 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:15.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.303+0000 7f5075e3e640 1 -- 172.21.15.88:0/33576807 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v96'199 uv182 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f5068055810 con 0x55c660a49b20 2024-09-16T01:41:15.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.303+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) -- 0x55c660a4c7f0 con 0x55c660a49b20 2024-09-16T01:41:15.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.304+0000 7f5075e3e640 1 -- 172.21.15.88:0/33576807 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v96'200 uv200 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f5068055810 con 0x55c660a49b20 2024-09-16T01:41:15.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.304+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a4cb90 con 0x55c660a49b20 2024-09-16T01:41:15.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.307+0000 7f5075e3e640 1 -- 172.21.15.88:0/33576807 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v96'201 uv201 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f5068055810 con 0x55c660a49b20 2024-09-16T01:41:15.308 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:41:15.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.307+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a4d3e0 con 0x55c660a49b20 2024-09-16T01:41:15.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.308+0000 7f5075e3e640 1 -- 172.21.15.88:0/33576807 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v96'202 uv202 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f5068055810 con 0x55c660a49b20 2024-09-16T01:41:15.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.308+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a4d780 con 0x55c660a49b20 2024-09-16T01:41:15.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.311+0000 7f5075e3e640 1 -- 172.21.15.88:0/33576807 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v96'203 uv203 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f5068055810 con 0x55c660a49b20 2024-09-16T01:41:15.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.311+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a4db20 con 0x55c660a49b20 2024-09-16T01:41:15.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.314+0000 7f5075e3e640 1 -- 172.21.15.88:0/33576807 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v96'204 uv187 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5068055810 con 0x55c660a49b20 2024-09-16T01:41:15.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.314+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) -- 0x55c660a4dec0 con 0x55c660a49b20 2024-09-16T01:41:15.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.316+0000 7f5075e3e640 1 -- 172.21.15.88:0/33576807 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v96'205 uv205 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5068055810 con 0x55c660a49b20 2024-09-16T01:41:15.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.316+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a4dec0 con 0x55c660a49b20 2024-09-16T01:41:15.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.319+0000 7f5075e3e640 1 -- 172.21.15.88:0/33576807 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v96'206 uv206 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5068055810 con 0x55c660a49b20 2024-09-16T01:41:15.320 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:41:15.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.319+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a4dec0 con 0x55c660a49b20 2024-09-16T01:41:15.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.321+0000 7f5075e3e640 1 -- 172.21.15.88:0/33576807 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v96'207 uv207 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5068055810 con 0x55c660a49b20 2024-09-16T01:41:15.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.321+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a4dec0 con 0x55c660a49b20 2024-09-16T01:41:15.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.324+0000 7f5075e3e640 1 -- 172.21.15.88:0/33576807 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v96'208 uv208 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5068055810 con 0x55c660a49b20 2024-09-16T01:41:15.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.324+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a4dec0 con 0x55c660a49b20 2024-09-16T01:41:15.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.327+0000 7f5075e3e640 1 -- 172.21.15.88:0/33576807 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v96'209 uv192 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5068055810 con 0x55c660a49b20 2024-09-16T01:41:15.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.327+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) -- 0x55c660a4dec0 con 0x55c660a49b20 2024-09-16T01:41:15.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.329+0000 7f5075e3e640 1 -- 172.21.15.88:0/33576807 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v96'210 uv210 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5068055810 con 0x55c660a49b20 2024-09-16T01:41:15.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.329+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a4dec0 con 0x55c660a49b20 2024-09-16T01:41:15.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.332+0000 7f5075e3e640 1 -- 172.21.15.88:0/33576807 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v96'211 uv211 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5068055810 con 0x55c660a49b20 2024-09-16T01:41:15.333 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:41:15.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.332+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a52340 con 0x55c660a49b20 2024-09-16T01:41:15.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.334+0000 7f5075e3e640 1 -- 172.21.15.88:0/33576807 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v96'212 uv212 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5068055810 con 0x55c660a49b20 2024-09-16T01:41:15.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.334+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a52340 con 0x55c660a49b20 2024-09-16T01:41:15.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.337+0000 7f5075e3e640 1 -- 172.21.15.88:0/33576807 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v96'213 uv213 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5068055810 con 0x55c660a49b20 2024-09-16T01:41:15.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.337+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a4dec0 con 0x55c660a49b20 2024-09-16T01:41:15.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.338+0000 7f5075e3e640 1 -- 172.21.15.88:0/33576807 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v96'214 uv198 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5068055810 con 0x55c660a49b20 2024-09-16T01:41:15.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.338+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) -- 0x55c660a4dec0 con 0x55c660a49b20 2024-09-16T01:41:15.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.340+0000 7f5075e3e640 1 -- 172.21.15.88:0/33576807 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v96'215 uv215 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5068055810 con 0x55c660a49b20 2024-09-16T01:41:15.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.340+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a4dec0 con 0x55c660a49b20 2024-09-16T01:41:15.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.341+0000 7f5075e3e640 1 -- 172.21.15.88:0/33576807 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v96'216 uv216 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5068055810 con 0x55c660a49b20 2024-09-16T01:41:15.342 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:41:15.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.341+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a4eed0 con 0x55c660a49b20 2024-09-16T01:41:15.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.342+0000 7f5075e3e640 1 -- 172.21.15.88:0/33576807 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v96'217 uv217 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5068055810 con 0x55c660a49b20 2024-09-16T01:41:15.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.342+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a4eed0 con 0x55c660a49b20 2024-09-16T01:41:15.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.343+0000 7f5075e3e640 1 -- 172.21.15.88:0/33576807 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v96'218 uv218 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5068055810 con 0x55c660a49b20 2024-09-16T01:41:15.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.343+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a4dec0 con 0x55c660a49b20 2024-09-16T01:41:15.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.344+0000 7f5075e3e640 1 -- 172.21.15.88:0/33576807 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v96'219 uv219 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5068055810 con 0x55c660a49b20 2024-09-16T01:41:15.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.344+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a4dec0 con 0x55c660a44ab0 2024-09-16T01:41:15.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.345+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v96'51 uv50 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.345+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) -- 0x55c660a4dec0 con 0x55c660a44ab0 2024-09-16T01:41:15.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.346+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v96'52 uv52 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.346+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a4dec0 con 0x55c660a44ab0 2024-09-16T01:41:15.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.347+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v96'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.348 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:41:15.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.347+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a4eed0 con 0x55c660a44ab0 2024-09-16T01:41:15.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.348+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v96'54 uv54 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.348+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a4eed0 con 0x55c660a44ab0 2024-09-16T01:41:15.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.349+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v96'55 uv55 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.349+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a4dec0 con 0x55c660a44ab0 2024-09-16T01:41:15.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.350+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v96'56 uv56 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.350+0000 7f5078348f40 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55c660a4e540 0x55c660a50930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.350+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a50f00 con 0x55c660a4e540 2024-09-16T01:41:15.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.350+0000 7f506ffff640 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55c660a4e540 0x55c660a50930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.350+0000 7f506ffff640 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55c660a4e540 0x55c660a50930 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:15.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.351+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v96'43 uv42 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a4e540 2024-09-16T01:41:15.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.351+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) -- 0x55c660a51000 con 0x55c660a4e540 2024-09-16T01:41:15.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.352+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v96'44 uv44 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a4e540 2024-09-16T01:41:15.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.352+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a51440 con 0x55c660a4e540 2024-09-16T01:41:15.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.353+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v96'45 uv45 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a4e540 2024-09-16T01:41:15.355 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:41:15.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.353+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a51440 con 0x55c660a4e540 2024-09-16T01:41:15.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.354+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v96'46 uv46 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a4e540 2024-09-16T01:41:15.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.354+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a51440 con 0x55c660a44ab0 2024-09-16T01:41:15.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.356+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v96'51 uv50 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.356+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) -- 0x55c660a51440 con 0x55c660a44ab0 2024-09-16T01:41:15.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.357+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v96'52 uv52 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.357+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a51440 con 0x55c660a44ab0 2024-09-16T01:41:15.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.358+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v96'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.359 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:41:15.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.358+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a52450 con 0x55c660a44ab0 2024-09-16T01:41:15.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.359+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v96'54 uv54 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.359+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a52450 con 0x55c660a44ab0 2024-09-16T01:41:15.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.360+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v96'55 uv55 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.360+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a51440 con 0x55c660a44ab0 2024-09-16T01:41:15.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.361+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v96'56 uv56 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.361+0000 7f5078348f40 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55c660a51950 0x55c660a53d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.361+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a543a0 con 0x55c660a51950 2024-09-16T01:41:15.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.361+0000 7f507663f640 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55c660a51950 0x55c660a53d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.362+0000 7f507663f640 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55c660a51950 0x55c660a53d40 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:15.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.363+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v96'94 uv88 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.363+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) -- 0x55c660a54880 con 0x55c660a51950 2024-09-16T01:41:15.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.364+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v96'95 uv95 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.364+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a54d60 con 0x55c660a51950 2024-09-16T01:41:15.366 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:41:15.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.365+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v96'96 uv96 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.365+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a56610 con 0x55c660a51950 2024-09-16T01:41:15.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.366+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v96'97 uv97 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.366+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a569b0 con 0x55c660a51950 2024-09-16T01:41:15.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.367+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v96'98 uv98 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.367+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a55600 con 0x55c660a51950 2024-09-16T01:41:15.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.368+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v96'99 uv99 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.368+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a559f0 con 0x55c660a51950 2024-09-16T01:41:15.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.369+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v96'100 uv93 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.369+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) -- 0x55c660a559f0 con 0x55c660a51950 2024-09-16T01:41:15.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.370+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v96'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.370+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a559f0 con 0x55c660a51950 2024-09-16T01:41:15.372 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:41:15.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.371+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v96'102 uv102 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.371+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a559f0 con 0x55c660a51950 2024-09-16T01:41:15.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.372+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v96'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.372+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a559f0 con 0x55c660a51950 2024-09-16T01:41:15.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.373+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v96'104 uv104 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.373+0000 7f5078348f40 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55c660a56d50 0x55c660a59140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.373+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a559f0 con 0x55c660a56d50 2024-09-16T01:41:15.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.373+0000 7f5075e3e640 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55c660a56d50 0x55c660a59140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.373+0000 7f5075e3e640 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55c660a56d50 0x55c660a59140 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:15.375 INFO:teuthology.orchestra.run.smithi088.stdout:Write #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:41:15.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.374+0000 7f5075e3e640 1 -- 172.21.15.88:0/33576807 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v96'43 uv43 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f5068055810 con 0x55c660a56d50 2024-09-16T01:41:15.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.374+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a559f0 con 0x55c660a56d50 2024-09-16T01:41:15.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.375+0000 7f5075e3e640 1 -- 172.21.15.88:0/33576807 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [write 0~70] v96'44 uv44 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f5068055810 con 0x55c660a56d50 2024-09-16T01:41:15.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.375+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a559f0 con 0x55c660a56d50 2024-09-16T01:41:15.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.376+0000 7f5075e3e640 1 -- 172.21.15.88:0/33576807 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [omap-set-header] v96'45 uv45 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f5068055810 con 0x55c660a56d50 2024-09-16T01:41:15.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.376+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:83 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a596d0 con 0x55c660a44ab0 2024-09-16T01:41:15.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.377+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(83 605.00000000 [create] v96'51 uv50 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.377+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:84 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) -- 0x55c660a596d0 con 0x55c660a44ab0 2024-09-16T01:41:15.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.378+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(84 605.00000000 [delete] v96'52 uv52 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.378+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a596d0 con 0x55c660a44ab0 2024-09-16T01:41:15.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.379+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(85 605.00000000 [create] v96'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.380 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:41:15.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.379+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5a6e0 con 0x55c660a44ab0 2024-09-16T01:41:15.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.380+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(86 605.00000000 [setxattr (30)] v96'54 uv54 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.380+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5a6e0 con 0x55c660a44ab0 2024-09-16T01:41:15.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.381+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(87 605.00000000 [setxattr (62)] v96'55 uv55 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.381+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a596d0 con 0x55c660a44ab0 2024-09-16T01:41:15.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.382+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(88 605.00000000 [omap-set-header] v96'56 uv56 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a44ab0 2024-09-16T01:41:15.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.382+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:89 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a596d0 con 0x55c660a4e540 2024-09-16T01:41:15.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.383+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(89 mds0_inotable [create] v96'84 uv78 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a4e540 2024-09-16T01:41:15.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.383+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:90 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) -- 0x55c660a596d0 con 0x55c660a4e540 2024-09-16T01:41:15.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.383+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(90 mds0_inotable [delete] v96'85 uv85 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a4e540 2024-09-16T01:41:15.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.383+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a596d0 con 0x55c660a4e540 2024-09-16T01:41:15.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.384+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(91 mds0_inotable [create] v96'86 uv86 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a4e540 2024-09-16T01:41:15.386 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:41:15.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.384+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a596d0 con 0x55c660a4e540 2024-09-16T01:41:15.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.385+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(92 mds0_inotable [write 0~34] v96'87 uv87 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a4e540 2024-09-16T01:41:15.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.385+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a596d0 con 0x55c660a4e540 2024-09-16T01:41:15.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.386+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(93 mds0_inotable [omap-set-header] v96'88 uv88 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a4e540 2024-09-16T01:41:15.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.386+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a596d0 con 0x55c660a4e540 2024-09-16T01:41:15.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.387+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(94 100.00000000 [create] v96'89 uv83 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a4e540 2024-09-16T01:41:15.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.387+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) -- 0x55c660a596d0 con 0x55c660a4e540 2024-09-16T01:41:15.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.388+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(95 100.00000000 [delete] v96'90 uv90 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a4e540 2024-09-16T01:41:15.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.388+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a596d0 con 0x55c660a4e540 2024-09-16T01:41:15.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.389+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(96 100.00000000 [create] v96'91 uv91 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a4e540 2024-09-16T01:41:15.391 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:41:15.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.389+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a596d0 con 0x55c660a4e540 2024-09-16T01:41:15.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.390+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(97 100.00000000 [omap-set-header] v96'92 uv92 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a4e540 2024-09-16T01:41:15.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.390+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c4e0 con 0x55c660a4e540 2024-09-16T01:41:15.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.391+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v96'93 uv93 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a4e540 2024-09-16T01:41:15.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.391+0000 7f5078348f40 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55c660a5a090 0x55c660a5c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.391+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:99 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a5a090 2024-09-16T01:41:15.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.391+0000 7f506ffff640 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55c660a5a090 0x55c660a5c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.392+0000 7f506ffff640 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55c660a5a090 0x55c660a5c9e0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:15.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.393+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(99 mds0_sessionmap [create] v96'40 uv39 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a5a090 2024-09-16T01:41:15.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.393+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:100 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a5a090 2024-09-16T01:41:15.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.394+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v96'41 uv41 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a5a090 2024-09-16T01:41:15.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.394+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a5a090 2024-09-16T01:41:15.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.396+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(101 mds0_sessionmap [create] v96'42 uv42 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a5a090 2024-09-16T01:41:15.397 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:41:15.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.396+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a5a090 2024-09-16T01:41:15.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.399+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v96'43 uv43 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a5a090 2024-09-16T01:41:15.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.399+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a4e540 2024-09-16T01:41:15.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.400+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(103 609.00000000 [create] v96'51 uv50 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a4e540 2024-09-16T01:41:15.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.400+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a4e540 2024-09-16T01:41:15.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.401+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(104 609.00000000 [delete] v96'52 uv52 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a4e540 2024-09-16T01:41:15.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.401+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a4e540 2024-09-16T01:41:15.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.402+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(105 609.00000000 [create] v96'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a4e540 2024-09-16T01:41:15.403 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:41:15.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.402+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a4e540 2024-09-16T01:41:15.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.403+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v96'54 uv54 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a4e540 2024-09-16T01:41:15.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.403+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a4e540 2024-09-16T01:41:15.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.404+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v96'55 uv55 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a4e540 2024-09-16T01:41:15.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.404+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a4e540 2024-09-16T01:41:15.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.405+0000 7f506ffff640 1 -- 172.21.15.88:0/33576807 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(108 609.00000000 [omap-set-header] v96'56 uv56 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5060009040 con 0x55c660a4e540 2024-09-16T01:41:15.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.405+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a51950 2024-09-16T01:41:15.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.406+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(109 400.00000000 [create] v96'42 uv41 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.406+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a51950 2024-09-16T01:41:15.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.407+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(110 400.00000000 [delete] v96'43 uv43 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.407+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a51950 2024-09-16T01:41:15.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.408+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(111 400.00000000 [create] v96'44 uv44 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.410 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:41:15.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.408+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a51950 2024-09-16T01:41:15.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.410+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v96'45 uv45 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.410+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a51950 2024-09-16T01:41:15.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.411+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v96'46 uv46 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.411+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a51950 2024-09-16T01:41:15.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.412+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(114 100.00000000.inode [create] v96'107 uv99 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.412+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a51950 2024-09-16T01:41:15.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.413+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(115 100.00000000.inode [delete] v96'108 uv108 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.413+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a51950 2024-09-16T01:41:15.415 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:41:15.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.414+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(116 100.00000000.inode [create] v96'109 uv109 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.414+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a51950 2024-09-16T01:41:15.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.415+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v96'110 uv110 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.415+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a51950 2024-09-16T01:41:15.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.417+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v96'111 uv111 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.417+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a51950 2024-09-16T01:41:15.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.420+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(119 1.00000000 [create] v96'112 uv106 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.420+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a51950 2024-09-16T01:41:15.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.421+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(120 1.00000000 [delete] v96'113 uv113 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.421+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a51950 2024-09-16T01:41:15.425 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:41:15.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.424+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(121 1.00000000 [create] v96'114 uv114 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.424+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a51950 2024-09-16T01:41:15.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.426+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v96'115 uv115 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.427+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a51950 2024-09-16T01:41:15.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.429+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v96'116 uv116 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.429+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5c480 con 0x55c660a51950 2024-09-16T01:41:15.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.432+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v96'117 uv117 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.432+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) -- 0x55c660a5cf20 con 0x55c660a51950 2024-09-16T01:41:15.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.434+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v96'118 uv118 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f507001d040 con 0x55c660a51950 2024-09-16T01:41:15.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.434+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55c660a44ab0 msgr2=0x55c660a46ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:15.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.434+0000 7f5078348f40 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55c660a44ab0 0x55c660a46ea0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.434+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55c660a4e540 msgr2=0x55c660a50930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:15.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.434+0000 7f5078348f40 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55c660a4e540 0x55c660a50930 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.434+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55c660a5a090 msgr2=0x55c660a5c9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:15.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.434+0000 7f5078348f40 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55c660a5a090 0x55c660a5c9e0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.434+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55c660a49b20 msgr2=0x55c660a4bf10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:15.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.434+0000 7f5078348f40 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55c660a49b20 0x55c660a4bf10 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.434+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55c660a51950 msgr2=0x55c660a53d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:15.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.434+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 reap_dead start 2024-09-16T01:41:15.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.434+0000 7f5078348f40 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55c660a51950 0x55c660a53d40 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.434+0000 7f507663f640 1 -- 172.21.15.88:0/33576807 reap_dead start 2024-09-16T01:41:15.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.434+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55c660a56d50 msgr2=0x55c660a59140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:15.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.434+0000 7f5078348f40 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55c660a56d50 0x55c660a59140 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.434+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f505806ac90 msgr2=0x7f505806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:15.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.434+0000 7f5078348f40 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f505806ac90 0x7f505806d150 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5068055000 tx=0x7f5068052000 comp rx=0 tx=0).stop 2024-09-16T01:41:15.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.434+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c660a3c650 msgr2=0x55c6608cf120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:15.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.434+0000 7f5078348f40 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c660a3c650 0x55c6608cf120 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x55c6608ffbc0 tx=0x7f5070003330 comp rx=0 tx=0).stop 2024-09-16T01:41:15.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.434+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 shutdown_connections 2024-09-16T01:41:15.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.434+0000 7f5078348f40 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f505806ac90 0x7f505806d150 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.434+0000 7f5078348f40 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c660a3c650 0x55c6608cf120 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.434+0000 7f5078348f40 1 --2- 172.21.15.88:0/33576807 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55c660a56d50 0x55c660a59140 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.434+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 >> 172.21.15.88:0/33576807 conn(0x55c66091b750 msgr2=0x55c6608d3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:15.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.435+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 shutdown_connections 2024-09-16T01:41:15.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.435+0000 7f5078348f40 1 -- 172.21.15.88:0/33576807 wait complete. 2024-09-16T01:41:15.439 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm mds0_inotable 2024-09-16T01:41:15.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.520+0000 7f0be5818f40 1 Processor -- start 2024-09-16T01:41:15.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.520+0000 7f0be5818f40 1 -- start start 2024-09-16T01:41:15.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.521+0000 7f0be5818f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561458048c10 0x561458048ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.521+0000 7f0be5818f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614580d3e10 0x561458047370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.521+0000 7f0be5818f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56145802f720 0x56145803ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.521+0000 7f0be5818f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x561457f52510 con 0x561458048c10 2024-09-16T01:41:15.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.521+0000 7f0be5818f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561457fabc50 con 0x5614580d3e10 2024-09-16T01:41:15.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.521+0000 7f0be5818f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56145804ab80 con 0x56145802f720 2024-09-16T01:41:15.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.521+0000 7f0be26a0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561458048c10 0x561458048ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.521+0000 7f0be26a0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561458048c10 0x561458048ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41500/0 (socket says 172.21.15.88:41500) 2024-09-16T01:41:15.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.521+0000 7f0be26a0640 1 -- 172.21.15.88:0/1522333401 learned_addr learned my addr 172.21.15.88:0/1522333401 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:15.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.521+0000 7f0bdbfff640 1 --2- 172.21.15.88:0/1522333401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614580d3e10 0x561458047370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.521+0000 7f0be2ea1640 1 --2- 172.21.15.88:0/1522333401 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56145802f720 0x56145803ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.521+0000 7f0be26a0640 1 -- 172.21.15.88:0/1522333401 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56145802f720 msgr2=0x56145803ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:15.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.521+0000 7f0be26a0640 1 --2- 172.21.15.88:0/1522333401 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56145802f720 0x56145803ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.521+0000 7f0be26a0640 1 -- 172.21.15.88:0/1522333401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614580d3e10 msgr2=0x561458047370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:15.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.521+0000 7f0be26a0640 1 --2- 172.21.15.88:0/1522333401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614580d3e10 0x561458047370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.521+0000 7f0be26a0640 1 -- 172.21.15.88:0/1522333401 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614580d06a0 con 0x561458048c10 2024-09-16T01:41:15.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.521+0000 7f0be26a0640 1 --2- 172.21.15.88:0/1522333401 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561458048c10 0x561458048ff0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f0bd40095b0 tx=0x7f0bd4009a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1e25fe842877cf5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:15.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.522+0000 7f0be169e640 1 -- 172.21.15.88:0/1522333401 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd400e800 con 0x561458048c10 2024-09-16T01:41:15.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.522+0000 7f0be169e640 1 -- 172.21.15.88:0/1522333401 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bd4013030 con 0x561458048c10 2024-09-16T01:41:15.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.522+0000 7f0be169e640 1 -- 172.21.15.88:0/1522333401 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd400ebe0 con 0x561458048c10 2024-09-16T01:41:15.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.522+0000 7f0be5818f40 1 -- 172.21.15.88:0/1522333401 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561458048c10 msgr2=0x561458048ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:15.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.522+0000 7f0be5818f40 1 --2- 172.21.15.88:0/1522333401 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561458048c10 0x561458048ff0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f0bd40095b0 tx=0x7f0bd4009a70 comp rx=0 tx=0).stop 2024-09-16T01:41:15.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.522+0000 7f0be5818f40 1 -- 172.21.15.88:0/1522333401 shutdown_connections 2024-09-16T01:41:15.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.522+0000 7f0be5818f40 1 --2- 172.21.15.88:0/1522333401 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56145802f720 0x56145803ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.522+0000 7f0be5818f40 1 --2- 172.21.15.88:0/1522333401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614580d3e10 0x561458047370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.522+0000 7f0be5818f40 1 --2- 172.21.15.88:0/1522333401 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561458048c10 0x561458048ff0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.522+0000 7f0be5818f40 1 -- 172.21.15.88:0/1522333401 >> 172.21.15.88:0/1522333401 conn(0x56145802b960 msgr2=0x561457fff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:15.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.522+0000 7f0be5818f40 1 -- 172.21.15.88:0/1522333401 shutdown_connections 2024-09-16T01:41:15.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.522+0000 7f0be5818f40 1 -- 172.21.15.88:0/1522333401 wait complete. 2024-09-16T01:41:15.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.523+0000 7f0be5818f40 1 Processor -- start 2024-09-16T01:41:15.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.523+0000 7f0be5818f40 1 -- start start 2024-09-16T01:41:15.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.523+0000 7f0be5818f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145802f720 0x561458041c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.523+0000 7f0be5818f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561458048c10 0x561458044660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.523+0000 7f0be5818f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614580d3e10 0x5614580d26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.523+0000 7f0be5818f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5614580d8530 con 0x56145802f720 2024-09-16T01:41:15.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.523+0000 7f0be5818f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56145802f270 con 0x5614580d3e10 2024-09-16T01:41:15.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.523+0000 7f0be5818f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x561458049660 con 0x561458048c10 2024-09-16T01:41:15.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.523+0000 7f0be26a0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145802f720 0x561458041c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.523+0000 7f0be26a0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145802f720 0x561458041c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41506/0 (socket says 172.21.15.88:41506) 2024-09-16T01:41:15.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.524+0000 7f0be26a0640 1 -- 172.21.15.88:0/2470503455 learned_addr learned my addr 172.21.15.88:0/2470503455 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:15.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.524+0000 7f0be2ea1640 1 --2- 172.21.15.88:0/2470503455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614580d3e10 0x5614580d26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.524+0000 7f0bdbfff640 1 --2- 172.21.15.88:0/2470503455 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561458048c10 0x561458044660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.524+0000 7f0be26a0640 1 -- 172.21.15.88:0/2470503455 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561458048c10 msgr2=0x561458044660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:15.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.524+0000 7f0be26a0640 1 --2- 172.21.15.88:0/2470503455 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561458048c10 0x561458044660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.524+0000 7f0be26a0640 1 -- 172.21.15.88:0/2470503455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614580d3e10 msgr2=0x5614580d26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:15.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.524+0000 7f0be26a0640 1 --2- 172.21.15.88:0/2470503455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614580d3e10 0x5614580d26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.524+0000 7f0be26a0640 1 -- 172.21.15.88:0/2470503455 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56145802f510 con 0x56145802f720 2024-09-16T01:41:15.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.524+0000 7f0be26a0640 1 --2- 172.21.15.88:0/2470503455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145802f720 0x561458041c60 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f0bd4009ab0 tx=0x7f0bd4001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:15.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.524+0000 7f0bdb7fe640 1 -- 172.21.15.88:0/2470503455 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd400f860 con 0x56145802f720 2024-09-16T01:41:15.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.524+0000 7f0bdb7fe640 1 -- 172.21.15.88:0/2470503455 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bd400fa20 con 0x56145802f720 2024-09-16T01:41:15.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.524+0000 7f0be5818f40 1 -- 172.21.15.88:0/2470503455 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x561457fffa10 con 0x56145802f720 2024-09-16T01:41:15.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.524+0000 7f0bdb7fe640 1 -- 172.21.15.88:0/2470503455 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd400fe10 con 0x56145802f720 2024-09-16T01:41:15.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.524+0000 7f0be5818f40 1 -- 172.21.15.88:0/2470503455 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614580d3430 con 0x56145802f720 2024-09-16T01:41:15.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.525+0000 7f0bdb7fe640 1 -- 172.21.15.88:0/2470503455 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0bd4019430 con 0x56145802f720 2024-09-16T01:41:15.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.526+0000 7f0bdb7fe640 1 --2- 172.21.15.88:0/2470503455 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0bc406ac90 0x7f0bc406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.526+0000 7f0bdb7fe640 1 -- 172.21.15.88:0/2470503455 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6512+0+0 (secure 0 0 0) 0x7f0bd408c7e0 con 0x56145802f720 2024-09-16T01:41:15.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.526+0000 7f0bdbfff640 1 --2- 172.21.15.88:0/2470503455 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0bc406ac90 0x7f0bc406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.526+0000 7f0be5818f40 1 --2- 172.21.15.88:0/2470503455 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5614580421a0 0x561458040850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.526+0000 7f0be5818f40 1 -- 172.21.15.88:0/2470503455 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:1 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) -- 0x561458044ba0 con 0x5614580421a0 2024-09-16T01:41:15.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.526+0000 7f0be2ea1640 1 --2- 172.21.15.88:0/2470503455 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5614580421a0 0x561458040850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.526+0000 7f0bdbfff640 1 --2- 172.21.15.88:0/2470503455 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0bc406ac90 0x7f0bc406d150 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x561458046e90 tx=0x7f0bcc00c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:15.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.526+0000 7f0be2ea1640 1 --2- 172.21.15.88:0/2470503455 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5614580421a0 0x561458040850 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:15.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.528+0000 7f0be2ea1640 1 -- 172.21.15.88:0/2470503455 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(1 mds0_inotable [delete] v96'94 uv94 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0bdc003030 con 0x5614580421a0 2024-09-16T01:41:15.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.528+0000 7f0be5818f40 1 -- 172.21.15.88:0/2470503455 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5614580421a0 msgr2=0x561458040850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:15.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.528+0000 7f0be5818f40 1 --2- 172.21.15.88:0/2470503455 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5614580421a0 0x561458040850 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.528+0000 7f0be5818f40 1 -- 172.21.15.88:0/2470503455 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0bc406ac90 msgr2=0x7f0bc406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:15.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.528+0000 7f0be5818f40 1 --2- 172.21.15.88:0/2470503455 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0bc406ac90 0x7f0bc406d150 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x561458046e90 tx=0x7f0bcc00c040 comp rx=0 tx=0).stop 2024-09-16T01:41:15.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.528+0000 7f0be5818f40 1 -- 172.21.15.88:0/2470503455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145802f720 msgr2=0x561458041c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:15.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.528+0000 7f0be5818f40 1 --2- 172.21.15.88:0/2470503455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56145802f720 0x561458041c60 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f0bd4009ab0 tx=0x7f0bd4001970 comp rx=0 tx=0).stop 2024-09-16T01:41:15.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.528+0000 7f0be2ea1640 1 -- 172.21.15.88:0/2470503455 reap_dead start 2024-09-16T01:41:15.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.528+0000 7f0be5818f40 1 -- 172.21.15.88:0/2470503455 shutdown_connections 2024-09-16T01:41:15.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.528+0000 7f0be5818f40 1 -- 172.21.15.88:0/2470503455 >> 172.21.15.88:0/2470503455 conn(0x56145802b960 msgr2=0x561457fff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:15.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.528+0000 7f0be5818f40 1 -- 172.21.15.88:0/2470503455 shutdown_connections 2024-09-16T01:41:15.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.528+0000 7f0be5818f40 1 -- 172.21.15.88:0/2470503455 wait complete. 2024-09-16T01:41:15.539 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:41:15.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.679+0000 7f4f97b8d640 1 Processor -- start 2024-09-16T01:41:15.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.679+0000 7f4f97b8d640 1 -- start start 2024-09-16T01:41:15.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.680+0000 7f4f97b8d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f9015b480 0x7f4f900b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.680+0000 7f4f97b8d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4f90072700 0x7f4f9015bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.680+0000 7f4f97b8d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f900ccff0 con 0x7f4f900b60c0 2024-09-16T01:41:15.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.680+0000 7f4f97b8d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f4f900b6500 con 0x7f4f90072700 2024-09-16T01:41:15.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.680+0000 7f4f97b8d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4f900cd2d0 con 0x7f4f9015b480 2024-09-16T01:41:15.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.680+0000 7f4f95929640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f9015b480 0x7f4f900b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.680+0000 7f4f9612a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4f90072700 0x7f4f9015bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.680+0000 7f4f95128640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f4f900b60c0 0x7f4f90071ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:56140/0 (socket says 172.21.15.88:56140) 2024-09-16T01:41:15.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.680+0000 7f4f95128640 1 -- 172.21.15.88:0/3411372701 learned_addr learned my addr 172.21.15.88:0/3411372701 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:15.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.680+0000 7f4f9612a640 1 -- 172.21.15.88:0/3411372701 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f9015b480 msgr2=0x7f4f900b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:15.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.680+0000 7f4f9612a640 1 --2- 172.21.15.88:0/3411372701 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f9015b480 0x7f4f900b5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.680+0000 7f4f9612a640 1 -- 172.21.15.88:0/3411372701 >> v1:172.21.15.154:6789/0 conn(0x7f4f900b60c0 legacy=0x7f4f90071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:15.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.680+0000 7f4f9612a640 1 -- 172.21.15.88:0/3411372701 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f900cd5b0 con 0x7f4f90072700 2024-09-16T01:41:15.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.680+0000 7f4f94927640 1 -- 172.21.15.88:0/3411372701 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 246297610 0 0) 0x7f4f900ccff0 con 0x7f4f900b60c0 2024-09-16T01:41:15.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.681+0000 7f4f9612a640 1 --2- 172.21.15.88:0/3411372701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f90072700 0x7f4f9015bc60 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4f88009d90 tx=0x7f4f8802a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccf34aefbdaf9a72 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:15.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.681+0000 7f4f94927640 1 -- 172.21.15.88:0/3411372701 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f88031600 con 0x7f4f90072700 2024-09-16T01:41:15.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.681+0000 7f4f94927640 1 -- 172.21.15.88:0/3411372701 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f880317a0 con 0x7f4f90072700 2024-09-16T01:41:15.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.681+0000 7f4f94927640 1 -- 172.21.15.88:0/3411372701 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f88031b70 con 0x7f4f90072700 2024-09-16T01:41:15.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.682+0000 7f4f97b8d640 1 -- 172.21.15.88:0/3411372701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f90072700 msgr2=0x7f4f9015bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:15.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.682+0000 7f4f97b8d640 1 --2- 172.21.15.88:0/3411372701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f90072700 0x7f4f9015bc60 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4f88009d90 tx=0x7f4f8802a5d0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.682+0000 7f4f97b8d640 1 -- 172.21.15.88:0/3411372701 shutdown_connections 2024-09-16T01:41:15.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.682+0000 7f4f97b8d640 1 --2- 172.21.15.88:0/3411372701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f90072700 0x7f4f9015bc60 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.682+0000 7f4f97b8d640 1 --2- 172.21.15.88:0/3411372701 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f9015b480 0x7f4f900b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.682+0000 7f4f97b8d640 1 -- 172.21.15.88:0/3411372701 >> 172.21.15.88:0/3411372701 conn(0x7f4f9006d0f0 msgr2=0x7f4f9006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:15.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.682+0000 7f4f97b8d640 1 -- 172.21.15.88:0/3411372701 shutdown_connections 2024-09-16T01:41:15.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.682+0000 7f4f97b8d640 1 -- 172.21.15.88:0/3411372701 wait complete. 2024-09-16T01:41:15.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.683+0000 7f4f97b8d640 1 Processor -- start 2024-09-16T01:41:15.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.683+0000 7f4f97b8d640 1 -- start start 2024-09-16T01:41:15.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.684+0000 7f4f97b8d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f90072700 0x7f4f901491d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.684+0000 7f4f97b8d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f900b60c0 0x7f4f900c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.684+0000 7f4f97b8d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f9015b480 0x7f4f900ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.684+0000 7f4f97b8d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9015b860 con 0x7f4f900b60c0 2024-09-16T01:41:15.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.684+0000 7f4f97b8d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4f900cdb70 con 0x7f4f90072700 2024-09-16T01:41:15.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.684+0000 7f4f97b8d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4f9015b9e0 con 0x7f4f9015b480 2024-09-16T01:41:15.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.684+0000 7f4f95128640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f900b60c0 0x7f4f900c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.684+0000 7f4f95128640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f900b60c0 0x7f4f900c9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41522/0 (socket says 172.21.15.88:41522) 2024-09-16T01:41:15.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.684+0000 7f4f95128640 1 -- 172.21.15.88:0/3338880105 learned_addr learned my addr 172.21.15.88:0/3338880105 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:15.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.684+0000 7f4f95929640 1 --2- 172.21.15.88:0/3338880105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f90072700 0x7f4f901491d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.684+0000 7f4f9612a640 1 --2- 172.21.15.88:0/3338880105 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f9015b480 0x7f4f900ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.684+0000 7f4f95128640 1 -- 172.21.15.88:0/3338880105 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f9015b480 msgr2=0x7f4f900ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:15.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.684+0000 7f4f95128640 1 --2- 172.21.15.88:0/3338880105 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f9015b480 0x7f4f900ca7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.684+0000 7f4f95128640 1 -- 172.21.15.88:0/3338880105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f90072700 msgr2=0x7f4f901491d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:15.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.684+0000 7f4f95128640 1 --2- 172.21.15.88:0/3338880105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f90072700 0x7f4f901491d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:15.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.684+0000 7f4f95128640 1 -- 172.21.15.88:0/3338880105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f90152940 con 0x7f4f900b60c0 2024-09-16T01:41:15.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.684+0000 7f4f95929640 1 --2- 172.21.15.88:0/3338880105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f90072700 0x7f4f901491d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:15.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.684+0000 7f4f95128640 1 --2- 172.21.15.88:0/3338880105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f900b60c0 0x7f4f900c9810 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000c4b0 tx=0x7f4f8000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:15.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.684+0000 7f4f867fc640 1 -- 172.21.15.88:0/3338880105 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f80018070 con 0x7f4f900b60c0 2024-09-16T01:41:15.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.685+0000 7f4f867fc640 1 -- 172.21.15.88:0/3338880105 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f80007500 con 0x7f4f900b60c0 2024-09-16T01:41:15.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.685+0000 7f4f867fc640 1 -- 172.21.15.88:0/3338880105 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f80004a20 con 0x7f4f900b60c0 2024-09-16T01:41:15.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.685+0000 7f4f97b8d640 1 -- 172.21.15.88:0/3338880105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f90152c20 con 0x7f4f900b60c0 2024-09-16T01:41:15.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.685+0000 7f4f97b8d640 1 -- 172.21.15.88:0/3338880105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f901531e0 con 0x7f4f900b60c0 2024-09-16T01:41:15.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.685+0000 7f4f97b8d640 1 -- 172.21.15.88:0/3338880105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f900b5b80 con 0x7f4f900b60c0 2024-09-16T01:41:15.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.686+0000 7f4f867fc640 1 -- 172.21.15.88:0/3338880105 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4f80004bc0 con 0x7f4f900b60c0 2024-09-16T01:41:15.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.687+0000 7f4f867fc640 1 --2- 172.21.15.88:0/3338880105 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f7406c2e0 0x7f4f7406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:15.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.687+0000 7f4f867fc640 1 -- 172.21.15.88:0/3338880105 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 6512+0+0 (secure 0 0 0) 0x7f4f8008a990 con 0x7f4f900b60c0 2024-09-16T01:41:15.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.687+0000 7f4f95929640 1 --2- 172.21.15.88:0/3338880105 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f7406c2e0 0x7f4f7406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:15.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.687+0000 7f4f95929640 1 --2- 172.21.15.88:0/3338880105 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f7406c2e0 0x7f4f7406e7a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c009af0 tx=0x7f4f8c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:15.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.691+0000 7f4f867fc640 1 -- 172.21.15.88:0/3338880105 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4f80058ea0 con 0x7f4f900b60c0 2024-09-16T01:41:15.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:15.890+0000 7f4f97b8d640 1 -- 172.21.15.88:0/3338880105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f4f900713e0 con 0x7f4f900b60c0 2024-09-16T01:41:16.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.195+0000 7f4f867fc640 1 -- 172.21.15.88:0/3338880105 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v125) ==== 162+0+0 (secure 0 0 0) 0x7f4f800587a0 con 0x7f4f900b60c0 2024-09-16T01:41:16.196 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:41:16.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.197+0000 7f4f97b8d640 1 -- 172.21.15.88:0/3338880105 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f7406c2e0 msgr2=0x7f4f7406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:16.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.197+0000 7f4f97b8d640 1 --2- 172.21.15.88:0/3338880105 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f7406c2e0 0x7f4f7406e7a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c009af0 tx=0x7f4f8c008040 comp rx=0 tx=0).stop 2024-09-16T01:41:16.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.197+0000 7f4f97b8d640 1 -- 172.21.15.88:0/3338880105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f900b60c0 msgr2=0x7f4f900c9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:16.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.197+0000 7f4f97b8d640 1 --2- 172.21.15.88:0/3338880105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f900b60c0 0x7f4f900c9810 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f4f8000c4b0 tx=0x7f4f8000c970 comp rx=0 tx=0).stop 2024-09-16T01:41:16.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.197+0000 7f4f97b8d640 1 -- 172.21.15.88:0/3338880105 shutdown_connections 2024-09-16T01:41:16.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.197+0000 7f4f97b8d640 1 --2- 172.21.15.88:0/3338880105 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f7406c2e0 0x7f4f7406e7a0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.197+0000 7f4f97b8d640 1 --2- 172.21.15.88:0/3338880105 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f9015b480 0x7f4f900ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.197+0000 7f4f97b8d640 1 --2- 172.21.15.88:0/3338880105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f900b60c0 0x7f4f900c9810 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.197+0000 7f4f97b8d640 1 --2- 172.21.15.88:0/3338880105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f90072700 0x7f4f901491d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.197+0000 7f4f97b8d640 1 -- 172.21.15.88:0/3338880105 >> 172.21.15.88:0/3338880105 conn(0x7f4f9006d0f0 msgr2=0x7f4f90158810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:16.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.197+0000 7f4f97b8d640 1 -- 172.21.15.88:0/3338880105 shutdown_connections 2024-09-16T01:41:16.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.197+0000 7f4f97b8d640 1 -- 172.21.15.88:0/3338880105 wait complete. 2024-09-16T01:41:16.203 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:41:16.202+0000 7f383e4a3640 -1 mds.0.inotable: load_2 could not read table: -2 2024-09-16T01:41:16.203 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:41:16.202+0000 7f383e4a3640 -1 log_channel(cluster) log [ERR] : error reading table object 'mds0_inotable' -2 ((2) No such file or directory) 2024-09-16T01:41:16.204 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:41:16.203+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:41:16.204 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:41:16.203+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:41:16.212 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:16.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.312+0000 7ff333d5f640 1 Processor -- start 2024-09-16T01:41:16.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.312+0000 7ff333d5f640 1 -- start start 2024-09-16T01:41:16.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.313+0000 7ff333d5f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff32c15c550 0x7ff32c0715e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:16.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.313+0000 7ff333d5f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff32c0b6e10 0x7ff32c0b5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:16.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.313+0000 7ff333d5f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff32c0d0270 con 0x7ff32c073160 2024-09-16T01:41:16.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.313+0000 7ff333d5f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff32c15c930 con 0x7ff32c15c550 2024-09-16T01:41:16.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.313+0000 7ff333d5f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff32c0d04f0 con 0x7ff32c0b6e10 2024-09-16T01:41:16.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.313+0000 7ff331afb640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff32c15c550 0x7ff32c0715e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:16.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.313+0000 7ff331afb640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff32c15c550 0x7ff32c0715e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41526/0 (socket says 172.21.15.88:41526) 2024-09-16T01:41:16.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.313+0000 7ff331afb640 1 -- 172.21.15.88:0/424168140 learned_addr learned my addr 172.21.15.88:0/424168140 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:16.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.313+0000 7ff331afb640 1 -- 172.21.15.88:0/424168140 >> v2:172.21.15.154:3300/0 conn(0x7ff32c0b6e10 msgr2=0x7ff32c0b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:16.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.313+0000 7ff331afb640 1 --2- 172.21.15.88:0/424168140 >> v2:172.21.15.154:3300/0 conn(0x7ff32c0b6e10 0x7ff32c0b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.313+0000 7ff331afb640 1 -- 172.21.15.88:0/424168140 >> v1:172.21.15.154:6789/0 conn(0x7ff32c073160 legacy=0x7ff32c0d0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:16.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.313+0000 7ff331afb640 1 -- 172.21.15.88:0/424168140 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff32c0b7580 con 0x7ff32c15c550 2024-09-16T01:41:16.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.314+0000 7ff331afb640 1 --2- 172.21.15.88:0/424168140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff32c15c550 0x7ff32c0715e0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7ff32c0b2160 tx=0x7ff32802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ac3197c12723c22 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:16.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.314+0000 7ff330af9640 1 -- 172.21.15.88:0/424168140 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff328037070 con 0x7ff32c15c550 2024-09-16T01:41:16.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.314+0000 7ff330af9640 1 -- 172.21.15.88:0/424168140 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff328032440 con 0x7ff32c15c550 2024-09-16T01:41:16.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.314+0000 7ff330af9640 1 -- 172.21.15.88:0/424168140 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff328032810 con 0x7ff32c15c550 2024-09-16T01:41:16.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.314+0000 7ff333d5f640 1 -- 172.21.15.88:0/424168140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff32c15c550 msgr2=0x7ff32c0715e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:16.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.314+0000 7ff333d5f640 1 --2- 172.21.15.88:0/424168140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff32c15c550 0x7ff32c0715e0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7ff32c0b2160 tx=0x7ff32802a230 comp rx=0 tx=0).stop 2024-09-16T01:41:16.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.314+0000 7ff333d5f640 1 -- 172.21.15.88:0/424168140 shutdown_connections 2024-09-16T01:41:16.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.314+0000 7ff333d5f640 1 --2- 172.21.15.88:0/424168140 >> v2:172.21.15.154:3300/0 conn(0x7ff32c0b6e10 0x7ff32c0b5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.314+0000 7ff333d5f640 1 --2- 172.21.15.88:0/424168140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff32c15c550 0x7ff32c0715e0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.315+0000 7ff333d5f640 1 -- 172.21.15.88:0/424168140 >> 172.21.15.88:0/424168140 conn(0x7ff32c0b4240 msgr2=0x7ff32c0b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:16.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.315+0000 7ff333d5f640 1 -- 172.21.15.88:0/424168140 shutdown_connections 2024-09-16T01:41:16.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.315+0000 7ff333d5f640 1 -- 172.21.15.88:0/424168140 wait complete. 2024-09-16T01:41:16.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.316+0000 7ff333d5f640 1 Processor -- start 2024-09-16T01:41:16.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.316+0000 7ff333d5f640 1 -- start start 2024-09-16T01:41:16.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.316+0000 7ff333d5f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff32c073160 0x7ff32c14a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:16.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.316+0000 7ff333d5f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff32c0b6e10 0x7ff32c15cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:16.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.316+0000 7ff333d5f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff32c15c550 0x7ff32c158460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:16.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.316+0000 7ff333d5f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c0b7280 con 0x7ff32c0b6e10 2024-09-16T01:41:16.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.316+0000 7ff333d5f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c0d07b0 con 0x7ff32c073160 2024-09-16T01:41:16.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.316+0000 7ff333d5f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff32c0b5a80 con 0x7ff32c15c550 2024-09-16T01:41:16.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.317+0000 7ff3312fa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff32c0b6e10 0x7ff32c15cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:16.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.317+0000 7ff3312fa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff32c0b6e10 0x7ff32c15cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41536/0 (socket says 172.21.15.88:41536) 2024-09-16T01:41:16.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.317+0000 7ff3312fa640 1 -- 172.21.15.88:0/1242012891 learned_addr learned my addr 172.21.15.88:0/1242012891 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:16.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.317+0000 7ff331afb640 1 --2- 172.21.15.88:0/1242012891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff32c073160 0x7ff32c14a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:16.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.317+0000 7ff3322fc640 1 --2- 172.21.15.88:0/1242012891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff32c15c550 0x7ff32c158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:16.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.317+0000 7ff3312fa640 1 -- 172.21.15.88:0/1242012891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff32c15c550 msgr2=0x7ff32c158460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:16.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.317+0000 7ff3312fa640 1 --2- 172.21.15.88:0/1242012891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff32c15c550 0x7ff32c158460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.317+0000 7ff3312fa640 1 -- 172.21.15.88:0/1242012891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff32c073160 msgr2=0x7ff32c14a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:16.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.317+0000 7ff3312fa640 1 --2- 172.21.15.88:0/1242012891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff32c073160 0x7ff32c14a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.317+0000 7ff3312fa640 1 -- 172.21.15.88:0/1242012891 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff32c15d7d0 con 0x7ff32c0b6e10 2024-09-16T01:41:16.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.317+0000 7ff3312fa640 1 --2- 172.21.15.88:0/1242012891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff32c0b6e10 0x7ff32c15cdf0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7ff31c00b4a0 tx=0x7ff31c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:16.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.317+0000 7ff3227fc640 1 -- 172.21.15.88:0/1242012891 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31c014070 con 0x7ff32c0b6e10 2024-09-16T01:41:16.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.317+0000 7ff3227fc640 1 -- 172.21.15.88:0/1242012891 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff31c010610 con 0x7ff32c0b6e10 2024-09-16T01:41:16.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.317+0000 7ff3227fc640 1 -- 172.21.15.88:0/1242012891 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31c010a00 con 0x7ff32c0b6e10 2024-09-16T01:41:16.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.317+0000 7ff333d5f640 1 -- 172.21.15.88:0/1242012891 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff32c15dab0 con 0x7ff32c0b6e10 2024-09-16T01:41:16.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.317+0000 7ff333d5f640 1 -- 172.21.15.88:0/1242012891 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff32c15e070 con 0x7ff32c0b6e10 2024-09-16T01:41:16.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.318+0000 7ff333d5f640 1 -- 172.21.15.88:0/1242012891 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff32c0b5540 con 0x7ff32c0b6e10 2024-09-16T01:41:16.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.319+0000 7ff3227fc640 1 -- 172.21.15.88:0/1242012891 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff31c017070 con 0x7ff32c0b6e10 2024-09-16T01:41:16.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.319+0000 7ff3227fc640 1 --2- 172.21.15.88:0/1242012891 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff31006c2b0 0x7ff31006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:16.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.319+0000 7ff3227fc640 1 -- 172.21.15.88:0/1242012891 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6598+0+0 (secure 0 0 0) 0x7ff31c08b100 con 0x7ff32c0b6e10 2024-09-16T01:41:16.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.319+0000 7ff331afb640 1 --2- 172.21.15.88:0/1242012891 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff31006c2b0 0x7ff31006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:16.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.320+0000 7ff331afb640 1 --2- 172.21.15.88:0/1242012891 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff31006c2b0 0x7ff31006e770 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff328034000 tx=0x7ff32802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:16.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.324+0000 7ff3227fc640 1 -- 172.21.15.88:0/1242012891 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff31c058f30 con 0x7ff32c0b6e10 2024-09-16T01:41:16.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.527+0000 7ff333d5f640 1 -- 172.21.15.88:0/1242012891 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff32c0b1d60 con 0x7ff32c0b6e10 2024-09-16T01:41:16.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.528+0000 7ff3227fc640 1 -- 172.21.15.88:0/1242012891 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+3565 (secure 0 0 0) 0x7ff31c05d230 con 0x7ff32c0b6e10 2024-09-16T01:41:16.530 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:16.530 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":127,"btime":"2024-09-16T01:41:16:203965+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:16.203917+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:41:16.531 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 127 2024-09-16T01:41:16.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.529+0000 7ff333d5f640 1 -- 172.21.15.88:0/1242012891 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff31006c2b0 msgr2=0x7ff31006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:16.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.529+0000 7ff333d5f640 1 --2- 172.21.15.88:0/1242012891 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff31006c2b0 0x7ff31006e770 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff328034000 tx=0x7ff32802e000 comp rx=0 tx=0).stop 2024-09-16T01:41:16.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.529+0000 7ff333d5f640 1 -- 172.21.15.88:0/1242012891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff32c0b6e10 msgr2=0x7ff32c15cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:16.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.529+0000 7ff333d5f640 1 --2- 172.21.15.88:0/1242012891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff32c0b6e10 0x7ff32c15cdf0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7ff31c00b4a0 tx=0x7ff31c00b960 comp rx=0 tx=0).stop 2024-09-16T01:41:16.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.530+0000 7ff333d5f640 1 -- 172.21.15.88:0/1242012891 shutdown_connections 2024-09-16T01:41:16.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.530+0000 7ff333d5f640 1 --2- 172.21.15.88:0/1242012891 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff31006c2b0 0x7ff31006e770 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.530+0000 7ff333d5f640 1 --2- 172.21.15.88:0/1242012891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff32c15c550 0x7ff32c158460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.530+0000 7ff333d5f640 1 --2- 172.21.15.88:0/1242012891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff32c0b6e10 0x7ff32c15cdf0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.530+0000 7ff333d5f640 1 --2- 172.21.15.88:0/1242012891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff32c073160 0x7ff32c14a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.530+0000 7ff333d5f640 1 -- 172.21.15.88:0/1242012891 >> 172.21.15.88:0/1242012891 conn(0x7ff32c0b4240 msgr2=0x7ff32c0b2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:16.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.530+0000 7ff333d5f640 1 -- 172.21.15.88:0/1242012891 shutdown_connections 2024-09-16T01:41:16.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.530+0000 7ff333d5f640 1 -- 172.21.15.88:0/1242012891 wait complete. 2024-09-16T01:41:16.543 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:41:16.543 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:16.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.639+0000 7fb50cb49640 1 Processor -- start 2024-09-16T01:41:16.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.639+0000 7fb50cb49640 1 -- start start 2024-09-16T01:41:16.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.640+0000 7fb50cb49640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb5080d1820 0x7fb5081691c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:16.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.640+0000 7fb50cb49640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb50815ea40 con 0x7fb5080d3310 2024-09-16T01:41:16.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.640+0000 7fb50cb49640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5080d3870 con 0x7fb50815e660 2024-09-16T01:41:16.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.640+0000 7fb50cb49640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb5080d3b50 con 0x7fb5080d1820 2024-09-16T01:41:16.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.640+0000 7fb506d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb5080d1820 0x7fb5081691c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:16.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.641+0000 7fb505d9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb5080d3310 0x7fb5080d1110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49566/0 (socket says 172.21.15.88:49566) 2024-09-16T01:41:16.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.641+0000 7fb505d9b640 1 -- 172.21.15.88:0/723108387 learned_addr learned my addr 172.21.15.88:0/723108387 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:16.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.641+0000 7fb506d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb5080d1820 0x7fb5081691c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41540/0 (socket says 172.21.15.88:41540) 2024-09-16T01:41:16.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.641+0000 7fb506d9d640 1 -- 172.21.15.88:0/723108387 >> v1:172.21.15.154:6789/0 conn(0x7fb50815e660 legacy=0x7fb5080cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:16.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.641+0000 7fb506d9d640 1 -- 172.21.15.88:0/723108387 >> v1:172.21.15.88:6789/0 conn(0x7fb5080d3310 legacy=0x7fb5080d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:16.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.641+0000 7fb506d9d640 1 -- 172.21.15.88:0/723108387 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5080d3e30 con 0x7fb5080d1820 2024-09-16T01:41:16.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.641+0000 7fb50559a640 1 -- 172.21.15.88:0/723108387 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3544249880 0 0) 0x7fb50815ea40 con 0x7fb5080d3310 2024-09-16T01:41:16.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.641+0000 7fb506d9d640 1 --2- 172.21.15.88:0/723108387 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5080d1820 0x7fb5081691c0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fb500009d90 tx=0x7fb50002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=721894e60f4e79c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:16.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.642+0000 7fb50559a640 1 -- 172.21.15.88:0/723108387 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb500007740 con 0x7fb5080d1820 2024-09-16T01:41:16.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.642+0000 7fb50559a640 1 -- 172.21.15.88:0/723108387 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5000078e0 con 0x7fb5080d1820 2024-09-16T01:41:16.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.642+0000 7fb50559a640 1 -- 172.21.15.88:0/723108387 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb500007cb0 con 0x7fb5080d1820 2024-09-16T01:41:16.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.642+0000 7fb50cb49640 1 -- 172.21.15.88:0/723108387 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5080d1820 msgr2=0x7fb5081691c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:16.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.642+0000 7fb50cb49640 1 --2- 172.21.15.88:0/723108387 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5080d1820 0x7fb5081691c0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fb500009d90 tx=0x7fb50002a6a0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.642+0000 7fb50cb49640 1 -- 172.21.15.88:0/723108387 shutdown_connections 2024-09-16T01:41:16.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.642+0000 7fb50cb49640 1 --2- 172.21.15.88:0/723108387 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5080d1820 0x7fb5081691c0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.642+0000 7fb50cb49640 1 -- 172.21.15.88:0/723108387 >> 172.21.15.88:0/723108387 conn(0x7fb5080b6410 msgr2=0x7fb5080b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:16.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.642+0000 7fb50cb49640 1 -- 172.21.15.88:0/723108387 shutdown_connections 2024-09-16T01:41:16.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.642+0000 7fb50cb49640 1 -- 172.21.15.88:0/723108387 wait complete. 2024-09-16T01:41:16.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.644+0000 7fb50cb49640 1 Processor -- start 2024-09-16T01:41:16.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.644+0000 7fb50cb49640 1 -- start start 2024-09-16T01:41:16.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.644+0000 7fb50cb49640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5080d1820 0x7fb508157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:16.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.644+0000 7fb50cb49640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5080d3310 0x7fb5080ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:16.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.644+0000 7fb50cb49640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb50815e660 0x7fb5080b53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:16.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.644+0000 7fb50cb49640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb5080b8b80 con 0x7fb5080d1820 2024-09-16T01:41:16.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.644+0000 7fb50cb49640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb5080d43f0 con 0x7fb50815e660 2024-09-16T01:41:16.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.644+0000 7fb50cb49640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb5080b8d00 con 0x7fb5080d3310 2024-09-16T01:41:16.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.644+0000 7fb50659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5080d1820 0x7fb508157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:16.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.645+0000 7fb50659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5080d1820 0x7fb508157420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41542/0 (socket says 172.21.15.88:41542) 2024-09-16T01:41:16.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.645+0000 7fb50659c640 1 -- 172.21.15.88:0/873995671 learned_addr learned my addr 172.21.15.88:0/873995671 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:16.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.645+0000 7fb506d9d640 1 --2- 172.21.15.88:0/873995671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb50815e660 0x7fb5080b53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:16.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.645+0000 7fb505d9b640 1 --2- 172.21.15.88:0/873995671 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5080d3310 0x7fb5080ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:16.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.645+0000 7fb50659c640 1 -- 172.21.15.88:0/873995671 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5080d3310 msgr2=0x7fb5080ca0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:16.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.645+0000 7fb50659c640 1 --2- 172.21.15.88:0/873995671 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5080d3310 0x7fb5080ca0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.645+0000 7fb50659c640 1 -- 172.21.15.88:0/873995671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb50815e660 msgr2=0x7fb5080b53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:16.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.645+0000 7fb50659c640 1 --2- 172.21.15.88:0/873995671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb50815e660 0x7fb5080b53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.645+0000 7fb50659c640 1 -- 172.21.15.88:0/873995671 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5080cab70 con 0x7fb5080d1820 2024-09-16T01:41:16.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.645+0000 7fb505d9b640 1 --2- 172.21.15.88:0/873995671 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5080d3310 0x7fb5080ca0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:16.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.645+0000 7fb50659c640 1 --2- 172.21.15.88:0/873995671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5080d1820 0x7fb508157420 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fb4f800c430 tx=0x7fb4f800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:16.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.645+0000 7fb4f6ffd640 1 -- 172.21.15.88:0/873995671 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8011790 con 0x7fb5080d1820 2024-09-16T01:41:16.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.645+0000 7fb4f6ffd640 1 -- 172.21.15.88:0/873995671 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4f8017070 con 0x7fb5080d1820 2024-09-16T01:41:16.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.645+0000 7fb50cb49640 1 -- 172.21.15.88:0/873995671 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5081549a0 con 0x7fb5080d1820 2024-09-16T01:41:16.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.645+0000 7fb4f6ffd640 1 -- 172.21.15.88:0/873995671 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8011bb0 con 0x7fb5080d1820 2024-09-16T01:41:16.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.645+0000 7fb50cb49640 1 -- 172.21.15.88:0/873995671 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb508154f60 con 0x7fb5080d1820 2024-09-16T01:41:16.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.646+0000 7fb50cb49640 1 -- 172.21.15.88:0/873995671 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5080d1110 con 0x7fb5080d1820 2024-09-16T01:41:16.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.647+0000 7fb4f6ffd640 1 -- 172.21.15.88:0/873995671 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb4f801c460 con 0x7fb5080d1820 2024-09-16T01:41:16.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.647+0000 7fb4f6ffd640 1 --2- 172.21.15.88:0/873995671 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb4e806c2b0 0x7fb4e806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:16.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.647+0000 7fb4f6ffd640 1 -- 172.21.15.88:0/873995671 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6598+0+0 (secure 0 0 0) 0x7fb4f808b610 con 0x7fb5080d1820 2024-09-16T01:41:16.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.647+0000 7fb505d9b640 1 --2- 172.21.15.88:0/873995671 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb4e806c2b0 0x7fb4e806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:16.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.648+0000 7fb505d9b640 1 --2- 172.21.15.88:0/873995671 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb4e806c2b0 0x7fb4e806e770 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb508161730 tx=0x7fb4fc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:16.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.651+0000 7fb4f6ffd640 1 -- 172.21.15.88:0/873995671 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb4f8059a00 con 0x7fb5080d1820 2024-09-16T01:41:16.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.850+0000 7fb50cb49640 1 -- 172.21.15.88:0/873995671 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb50815cef0 con 0x7fb5080d1820 2024-09-16T01:41:16.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.851+0000 7fb4f6ffd640 1 -- 172.21.15.88:0/873995671 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+3565 (secure 0 0 0) 0x7fb4f8059300 con 0x7fb5080d1820 2024-09-16T01:41:16.853 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:16.854 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":127,"btime":"2024-09-16T01:41:16:203965+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:16.203917+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:41:16.854 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 127 2024-09-16T01:41:16.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.853+0000 7fb50cb49640 1 -- 172.21.15.88:0/873995671 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb4e806c2b0 msgr2=0x7fb4e806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:16.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.853+0000 7fb50cb49640 1 --2- 172.21.15.88:0/873995671 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb4e806c2b0 0x7fb4e806e770 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb508161730 tx=0x7fb4fc009040 comp rx=0 tx=0).stop 2024-09-16T01:41:16.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.853+0000 7fb50cb49640 1 -- 172.21.15.88:0/873995671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5080d1820 msgr2=0x7fb508157420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:16.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.853+0000 7fb50cb49640 1 --2- 172.21.15.88:0/873995671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5080d1820 0x7fb508157420 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fb4f800c430 tx=0x7fb4f800c8f0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.853+0000 7fb50cb49640 1 -- 172.21.15.88:0/873995671 shutdown_connections 2024-09-16T01:41:16.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.853+0000 7fb50cb49640 1 --2- 172.21.15.88:0/873995671 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb4e806c2b0 0x7fb4e806e770 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.853+0000 7fb50cb49640 1 --2- 172.21.15.88:0/873995671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb50815e660 0x7fb5080b53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.853+0000 7fb50cb49640 1 --2- 172.21.15.88:0/873995671 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5080d3310 0x7fb5080ca0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.853+0000 7fb50cb49640 1 --2- 172.21.15.88:0/873995671 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5080d1820 0x7fb508157420 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.853+0000 7fb50cb49640 1 -- 172.21.15.88:0/873995671 >> 172.21.15.88:0/873995671 conn(0x7fb5080b6410 msgr2=0x7fb50806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:16.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.853+0000 7fb50cb49640 1 -- 172.21.15.88:0/873995671 shutdown_connections 2024-09-16T01:41:16.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.853+0000 7fb50cb49640 1 -- 172.21.15.88:0/873995671 wait complete. 2024-09-16T01:41:16.866 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:41:16.866 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:16.933 INFO:tasks.ceph.mds.a.smithi088.stderr: -43> 2024-09-16T01:41:12.180+0000 7f2ab8e16640 -1 mds.0.snaptable: load_2 could not read table: -2 2024-09-16T01:41:16.933 INFO:tasks.ceph.mds.a.smithi088.stderr: -42> 2024-09-16T01:41:12.180+0000 7f2ab8e16640 -1 log_channel(cluster) log [ERR] : error reading table object 'mds_snaptable' -2 ((2) No such file or directory) 2024-09-16T01:41:16.956 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:41:16.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.964+0000 7fb511465640 1 Processor -- start 2024-09-16T01:41:16.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.964+0000 7fb511465640 1 -- start start 2024-09-16T01:41:16.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.964+0000 7fb511465640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb50c0cfa00 0x7fb50c0d3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:16.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.964+0000 7fb511465640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb50c0d2f40 con 0x7fb50c0ba240 2024-09-16T01:41:16.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.964+0000 7fb511465640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb50c0d3220 con 0x7fb50c15e7a0 2024-09-16T01:41:16.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.964+0000 7fb511465640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb50c0ba690 con 0x7fb50c0cfa00 2024-09-16T01:41:16.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb50a7fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb50c0ba240 0x7fb50c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49570/0 (socket says 172.21.15.88:49570) 2024-09-16T01:41:16.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb50a7fc640 1 -- 172.21.15.88:0/1217158013 learned_addr learned my addr 172.21.15.88:0/1217158013 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:16.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb509ffb640 1 -- 172.21.15.88:0/1217158013 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2176776370 0 0) 0x7fb50c0d2f40 con 0x7fb50c0ba240 2024-09-16T01:41:16.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb50b7fe640 1 --2- 172.21.15.88:0/1217158013 >> v2:172.21.15.154:3300/0 conn(0x7fb50c0cfa00 0x7fb50c0d3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:16.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb509ffb640 1 -- 172.21.15.88:0/1217158013 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4f4003520 con 0x7fb50c0ba240 2024-09-16T01:41:16.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb509ffb640 1 -- 172.21.15.88:0/1217158013 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2021063460 0 0) 0x7fb4f4003520 con 0x7fb50c0ba240 2024-09-16T01:41:16.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb509ffb640 1 -- 172.21.15.88:0/1217158013 >> v2:172.21.15.154:3300/0 conn(0x7fb50c0cfa00 msgr2=0x7fb50c0d3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:16.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb509ffb640 1 --2- 172.21.15.88:0/1217158013 >> v2:172.21.15.154:3300/0 conn(0x7fb50c0cfa00 0x7fb50c0d3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb509ffb640 1 -- 172.21.15.88:0/1217158013 >> v1:172.21.15.154:6789/0 conn(0x7fb50c15e7a0 legacy=0x7fb50c0d2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:16.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb509ffb640 1 -- 172.21.15.88:0/1217158013 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb50c0d3500 con 0x7fb50c0ba240 2024-09-16T01:41:16.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb509ffb640 1 -- 172.21.15.88:0/1217158013 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fb500001670 con 0x7fb50c0ba240 2024-09-16T01:41:16.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb509ffb640 1 -- 172.21.15.88:0/1217158013 >> v1:172.21.15.88:6789/0 conn(0x7fb50c0ba240 legacy=0x7fb50c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:16.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb509ffb640 1 --2- 172.21.15.88:0/1217158013 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4f40039b0 0x7fb4f4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:16.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb509ffb640 1 --2- 172.21.15.88:0/1217158013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4f40043b0 0x7fb4f4004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:16.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb509ffb640 1 --2- 172.21.15.88:0/1217158013 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb4f4005080 0x7fb4f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:16.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb509ffb640 1 -- 172.21.15.88:0/1217158013 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb50c0ba690 con 0x7fb4f40039b0 2024-09-16T01:41:16.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb509ffb640 1 -- 172.21.15.88:0/1217158013 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f400bf30 con 0x7fb4f40043b0 2024-09-16T01:41:16.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb509ffb640 1 -- 172.21.15.88:0/1217158013 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb4f400c130 con 0x7fb4f4005080 2024-09-16T01:41:16.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb50affd640 1 --2- 172.21.15.88:0/1217158013 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4f40039b0 0x7fb4f4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:16.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb50a7fc640 1 --2- 172.21.15.88:0/1217158013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4f40043b0 0x7fb4f4004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:16.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb50b7fe640 1 --2- 172.21.15.88:0/1217158013 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb4f4005080 0x7fb4f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:16.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb50affd640 1 -- 172.21.15.88:0/1217158013 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb4f4005080 msgr2=0x7fb4f400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:16.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb50affd640 1 --2- 172.21.15.88:0/1217158013 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb4f4005080 0x7fb4f400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb50affd640 1 -- 172.21.15.88:0/1217158013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4f40043b0 msgr2=0x7fb4f4004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:16.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb50affd640 1 --2- 172.21.15.88:0/1217158013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4f40043b0 0x7fb4f4004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.965+0000 7fb50affd640 1 -- 172.21.15.88:0/1217158013 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4f4003520 con 0x7fb4f40039b0 2024-09-16T01:41:16.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.966+0000 7fb50b7fe640 1 -- 172.21.15.88:0/1217158013 reap_dead start 2024-09-16T01:41:16.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.966+0000 7fb50affd640 1 --2- 172.21.15.88:0/1217158013 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4f40039b0 0x7fb4f4003d90 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc001860 tx=0x7fb4f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e0ceaf4abf85ed5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:16.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.966+0000 7fb509ffb640 1 -- 172.21.15.88:0/1217158013 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc037070 con 0x7fb4f40039b0 2024-09-16T01:41:16.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.966+0000 7fb509ffb640 1 -- 172.21.15.88:0/1217158013 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4fc02ac30 con 0x7fb4f40039b0 2024-09-16T01:41:16.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.966+0000 7fb509ffb640 1 -- 172.21.15.88:0/1217158013 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc001430 con 0x7fb4f40039b0 2024-09-16T01:41:16.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.967+0000 7fb511465640 1 -- 172.21.15.88:0/1217158013 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4f40039b0 msgr2=0x7fb4f4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:16.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.967+0000 7fb511465640 1 --2- 172.21.15.88:0/1217158013 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4f40039b0 0x7fb4f4003d90 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc001860 tx=0x7fb4f4005830 comp rx=0 tx=0).stop 2024-09-16T01:41:16.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.967+0000 7fb511465640 1 -- 172.21.15.88:0/1217158013 shutdown_connections 2024-09-16T01:41:16.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.967+0000 7fb511465640 1 --2- 172.21.15.88:0/1217158013 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb4f40039b0 0x7fb4f4003d90 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.967+0000 7fb511465640 1 -- 172.21.15.88:0/1217158013 >> 172.21.15.88:0/1217158013 conn(0x7fb50c0d1550 msgr2=0x7fb50c0d1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:16.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.967+0000 7fb511465640 1 -- 172.21.15.88:0/1217158013 shutdown_connections 2024-09-16T01:41:16.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.967+0000 7fb511465640 1 -- 172.21.15.88:0/1217158013 wait complete. 2024-09-16T01:41:16.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.968+0000 7fb511465640 1 Processor -- start 2024-09-16T01:41:16.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.968+0000 7fb511465640 1 -- start start 2024-09-16T01:41:16.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.969+0000 7fb511465640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4f40039b0 0x7fb50c0cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:16.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.969+0000 7fb511465640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb50c161a10 0x7fb50c1f3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:16.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.969+0000 7fb511465640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb50c0b5160 0x7fb50c0b55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:16.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.969+0000 7fb511465640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb50c15eb80 con 0x7fb50c161a10 2024-09-16T01:41:16.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.969+0000 7fb511465640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb50c0d3ac0 con 0x7fb4f40039b0 2024-09-16T01:41:16.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.969+0000 7fb511465640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb50c15ed00 con 0x7fb50c0b5160 2024-09-16T01:41:16.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.969+0000 7fb50a7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb50c161a10 0x7fb50c1f3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:16.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.969+0000 7fb50a7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb50c161a10 0x7fb50c1f3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41582/0 (socket says 172.21.15.88:41582) 2024-09-16T01:41:16.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.969+0000 7fb50a7fc640 1 -- 172.21.15.88:0/2713287388 learned_addr learned my addr 172.21.15.88:0/2713287388 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:16.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.969+0000 7fb50a7fc640 1 -- 172.21.15.88:0/2713287388 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb50c0b5160 msgr2=0x7fb50c0b55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:16.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.969+0000 7fb50a7fc640 1 --2- 172.21.15.88:0/2713287388 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb50c0b5160 0x7fb50c0b55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.969+0000 7fb50a7fc640 1 -- 172.21.15.88:0/2713287388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4f40039b0 msgr2=0x7fb50c0cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:16.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.969+0000 7fb50a7fc640 1 --2- 172.21.15.88:0/2713287388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4f40039b0 0x7fb50c0cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:16.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.969+0000 7fb50a7fc640 1 -- 172.21.15.88:0/2713287388 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb50c0cc1e0 con 0x7fb50c161a10 2024-09-16T01:41:16.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.969+0000 7fb50a7fc640 1 --2- 172.21.15.88:0/2713287388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb50c161a10 0x7fb50c1f3840 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fb50000e700 tx=0x7fb50000ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:16.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.970+0000 7fb4f37fe640 1 -- 172.21.15.88:0/2713287388 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb50000c640 con 0x7fb50c161a10 2024-09-16T01:41:16.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.970+0000 7fb4f37fe640 1 -- 172.21.15.88:0/2713287388 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb500018070 con 0x7fb50c161a10 2024-09-16T01:41:16.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.970+0000 7fb511465640 1 -- 172.21.15.88:0/2713287388 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb50c0cc4c0 con 0x7fb50c161a10 2024-09-16T01:41:16.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.970+0000 7fb4f37fe640 1 -- 172.21.15.88:0/2713287388 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb50000ca00 con 0x7fb50c161a10 2024-09-16T01:41:16.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.970+0000 7fb511465640 1 -- 172.21.15.88:0/2713287388 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb50c15f200 con 0x7fb50c161a10 2024-09-16T01:41:16.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.971+0000 7fb511465640 1 -- 172.21.15.88:0/2713287388 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb50c0d4070 con 0x7fb50c161a10 2024-09-16T01:41:16.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.971+0000 7fb4f37fe640 1 -- 172.21.15.88:0/2713287388 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb50000cba0 con 0x7fb50c161a10 2024-09-16T01:41:16.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.972+0000 7fb4f37fe640 1 --2- 172.21.15.88:0/2713287388 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb4f806c2b0 0x7fb4f806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:16.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.972+0000 7fb50affd640 1 --2- 172.21.15.88:0/2713287388 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb4f806c2b0 0x7fb4f806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:16.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.972+0000 7fb4f37fe640 1 -- 172.21.15.88:0/2713287388 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6598+0+0 (secure 0 0 0) 0x7fb50008b510 con 0x7fb50c161a10 2024-09-16T01:41:16.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.972+0000 7fb50affd640 1 --2- 172.21.15.88:0/2713287388 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb4f806c2b0 0x7fb4f806e770 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc034000 tx=0x7fb4fc02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:16.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:16.976+0000 7fb4f37fe640 1 -- 172.21.15.88:0/2713287388 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb500059990 con 0x7fb50c161a10 2024-09-16T01:41:17.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.175+0000 7fb511465640 1 -- 172.21.15.88:0/2713287388 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb50c0b3900 con 0x7fb50c161a10 2024-09-16T01:41:17.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.176+0000 7fb4f37fe640 1 -- 172.21.15.88:0/2713287388 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+3565 (secure 0 0 0) 0x7fb500059290 con 0x7fb50c161a10 2024-09-16T01:41:17.178 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:17.178 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":127,"btime":"2024-09-16T01:41:16:203965+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:16.203917+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:41:17.178 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 127 2024-09-16T01:41:17.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.177+0000 7fb511465640 1 -- 172.21.15.88:0/2713287388 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb4f806c2b0 msgr2=0x7fb4f806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:17.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.177+0000 7fb511465640 1 --2- 172.21.15.88:0/2713287388 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb4f806c2b0 0x7fb4f806e770 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc034000 tx=0x7fb4fc02e000 comp rx=0 tx=0).stop 2024-09-16T01:41:17.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.177+0000 7fb511465640 1 -- 172.21.15.88:0/2713287388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb50c161a10 msgr2=0x7fb50c1f3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:17.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.177+0000 7fb511465640 1 --2- 172.21.15.88:0/2713287388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb50c161a10 0x7fb50c1f3840 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fb50000e700 tx=0x7fb50000ebc0 comp rx=0 tx=0).stop 2024-09-16T01:41:17.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.177+0000 7fb511465640 1 -- 172.21.15.88:0/2713287388 shutdown_connections 2024-09-16T01:41:17.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.178+0000 7fb511465640 1 --2- 172.21.15.88:0/2713287388 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb4f806c2b0 0x7fb4f806e770 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:17.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.178+0000 7fb511465640 1 --2- 172.21.15.88:0/2713287388 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb50c0b5160 0x7fb50c0b55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:17.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.178+0000 7fb511465640 1 --2- 172.21.15.88:0/2713287388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb50c161a10 0x7fb50c1f3840 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:17.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.178+0000 7fb511465640 1 --2- 172.21.15.88:0/2713287388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4f40039b0 0x7fb50c0cbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:17.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.178+0000 7fb511465640 1 -- 172.21.15.88:0/2713287388 >> 172.21.15.88:0/2713287388 conn(0x7fb50c0d1550 msgr2=0x7fb50c06c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:17.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.178+0000 7fb511465640 1 -- 172.21.15.88:0/2713287388 shutdown_connections 2024-09-16T01:41:17.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.178+0000 7fb511465640 1 -- 172.21.15.88:0/2713287388 wait complete. 2024-09-16T01:41:17.191 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete mds0_inotable' led to DAMAGED state 2024-09-16T01:41:17.191 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 100.00000000' 2024-09-16T01:41:17.191 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:41:17.191 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:41:17.208 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:41:17.208 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:41:17.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:17.208 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:41:17.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:41:17.263 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:41:17.264 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:41:17.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.400+0000 7f52c76c9640 1 Processor -- start 2024-09-16T01:41:17.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.401+0000 7f52c76c9640 1 -- start start 2024-09-16T01:41:17.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.401+0000 7f52c76c9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f52c00cf1f0 0x7f52c00b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:17.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.401+0000 7f52c76c9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f52c00b6720 0x7f52c00d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:17.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.401+0000 7f52c76c9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52c00cf860 con 0x7f52c015e7a0 2024-09-16T01:41:17.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.401+0000 7f52c76c9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f52c00b6b90 con 0x7f52c00cf1f0 2024-09-16T01:41:17.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.401+0000 7f52c76c9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f52c00d2ca0 con 0x7f52c00b6720 2024-09-16T01:41:17.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.402+0000 7f52c4c64640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f52c00cf1f0 0x7f52c00b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:17.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.402+0000 7f52c4c64640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f52c00cf1f0 0x7f52c00b61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41584/0 (socket says 172.21.15.88:41584) 2024-09-16T01:41:17.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.402+0000 7f52c4c64640 1 -- 172.21.15.88:0/3301827760 learned_addr learned my addr 172.21.15.88:0/3301827760 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:17.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.402+0000 7f52c5c66640 1 --2- 172.21.15.88:0/3301827760 >> v2:172.21.15.154:3300/0 conn(0x7f52c00b6720 0x7f52c00d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:17.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.402+0000 7f52c4c64640 1 -- 172.21.15.88:0/3301827760 >> v2:172.21.15.154:3300/0 conn(0x7f52c00b6720 msgr2=0x7f52c00d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:17.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.402+0000 7f52c4c64640 1 --2- 172.21.15.88:0/3301827760 >> v2:172.21.15.154:3300/0 conn(0x7f52c00b6720 0x7f52c00d3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:17.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.402+0000 7f52c4c64640 1 -- 172.21.15.88:0/3301827760 >> v1:172.21.15.154:6789/0 conn(0x7f52c015e7a0 legacy=0x7f52c006caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:17.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.402+0000 7f52c4c64640 1 -- 172.21.15.88:0/3301827760 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52c00cfb40 con 0x7f52c00cf1f0 2024-09-16T01:41:17.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.402+0000 7f52c5c66640 1 --2- 172.21.15.88:0/3301827760 >> v2:172.21.15.154:3300/0 conn(0x7f52c00b6720 0x7f52c00d3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:17.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.403+0000 7f52c4c64640 1 --2- 172.21.15.88:0/3301827760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52c00cf1f0 0x7f52c00b61e0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f52c008d930 tx=0x7f52b002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f4edb86e721ebca server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:17.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.403+0000 7f52b7fff640 1 -- 172.21.15.88:0/3301827760 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b0037070 con 0x7f52c00cf1f0 2024-09-16T01:41:17.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.403+0000 7f52b7fff640 1 -- 172.21.15.88:0/3301827760 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52b0032440 con 0x7f52c00cf1f0 2024-09-16T01:41:17.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.403+0000 7f52b7fff640 1 -- 172.21.15.88:0/3301827760 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b0032810 con 0x7f52c00cf1f0 2024-09-16T01:41:17.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.403+0000 7f52c76c9640 1 -- 172.21.15.88:0/3301827760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52c00cf1f0 msgr2=0x7f52c00b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:17.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.403+0000 7f52c76c9640 1 --2- 172.21.15.88:0/3301827760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52c00cf1f0 0x7f52c00b61e0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f52c008d930 tx=0x7f52b002a1a0 comp rx=0 tx=0).stop 2024-09-16T01:41:17.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.403+0000 7f52c76c9640 1 -- 172.21.15.88:0/3301827760 shutdown_connections 2024-09-16T01:41:17.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.403+0000 7f52c76c9640 1 --2- 172.21.15.88:0/3301827760 >> v2:172.21.15.154:3300/0 conn(0x7f52c00b6720 0x7f52c00d3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:17.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.403+0000 7f52c76c9640 1 --2- 172.21.15.88:0/3301827760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52c00cf1f0 0x7f52c00b61e0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:17.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.403+0000 7f52c76c9640 1 -- 172.21.15.88:0/3301827760 >> 172.21.15.88:0/3301827760 conn(0x7f52c00d11e0 msgr2=0x7f52c00d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:17.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.403+0000 7f52c76c9640 1 -- 172.21.15.88:0/3301827760 shutdown_connections 2024-09-16T01:41:17.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.403+0000 7f52c76c9640 1 -- 172.21.15.88:0/3301827760 wait complete. 2024-09-16T01:41:17.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.405+0000 7f52c76c9640 1 Processor -- start 2024-09-16T01:41:17.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.405+0000 7f52c76c9640 1 -- start start 2024-09-16T01:41:17.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.405+0000 7f52c76c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52c00b6720 0x7f52c014dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:17.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.405+0000 7f52c76c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52c00cf1f0 0x7f52c00c8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:17.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.405+0000 7f52c76c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52c015e7a0 0x7f52c00c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:17.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.405+0000 7f52c76c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f52c00b7a00 con 0x7f52c00b6720 2024-09-16T01:41:17.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.405+0000 7f52c76c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f52c00d2ef0 con 0x7f52c00cf1f0 2024-09-16T01:41:17.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.405+0000 7f52c76c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f52c015eb80 con 0x7f52c015e7a0 2024-09-16T01:41:17.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.405+0000 7f52c5465640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52c00b6720 0x7f52c014dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:17.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.405+0000 7f52c5465640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52c00b6720 0x7f52c014dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41600/0 (socket says 172.21.15.88:41600) 2024-09-16T01:41:17.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.405+0000 7f52c5465640 1 -- 172.21.15.88:0/4097388567 learned_addr learned my addr 172.21.15.88:0/4097388567 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:17.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.405+0000 7f52c4c64640 1 --2- 172.21.15.88:0/4097388567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52c00cf1f0 0x7f52c00c8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:17.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.406+0000 7f52c5c66640 1 --2- 172.21.15.88:0/4097388567 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52c015e7a0 0x7f52c00c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:17.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.406+0000 7f52c5465640 1 -- 172.21.15.88:0/4097388567 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52c015e7a0 msgr2=0x7f52c00c87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:17.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.406+0000 7f52c5465640 1 --2- 172.21.15.88:0/4097388567 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52c015e7a0 0x7f52c00c87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:17.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.406+0000 7f52c5465640 1 -- 172.21.15.88:0/4097388567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52c00cf1f0 msgr2=0x7f52c00c8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:17.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.406+0000 7f52c5465640 1 --2- 172.21.15.88:0/4097388567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52c00cf1f0 0x7f52c00c8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:17.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.406+0000 7f52c5465640 1 -- 172.21.15.88:0/4097388567 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52c00b9450 con 0x7f52c00b6720 2024-09-16T01:41:17.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.406+0000 7f52c5465640 1 --2- 172.21.15.88:0/4097388567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52c00b6720 0x7f52c014dc20 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f52bc00c7c0 tx=0x7f52bc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:17.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.406+0000 7f52b5ffb640 1 -- 172.21.15.88:0/4097388567 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52bc016020 con 0x7f52c00b6720 2024-09-16T01:41:17.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.406+0000 7f52b5ffb640 1 -- 172.21.15.88:0/4097388567 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52bc00aa50 con 0x7f52c00b6720 2024-09-16T01:41:17.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.406+0000 7f52b5ffb640 1 -- 172.21.15.88:0/4097388567 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52bc012420 con 0x7f52c00b6720 2024-09-16T01:41:17.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.406+0000 7f52c76c9640 1 -- 172.21.15.88:0/4097388567 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52c00d3710 con 0x7f52c00b6720 2024-09-16T01:41:17.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.406+0000 7f52c76c9640 1 -- 172.21.15.88:0/4097388567 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52c00d3cd0 con 0x7f52c00b6720 2024-09-16T01:41:17.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.407+0000 7f52c76c9640 1 -- 172.21.15.88:0/4097388567 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52c00d2a80 con 0x7f52c00b6720 2024-09-16T01:41:17.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.408+0000 7f52b5ffb640 1 -- 172.21.15.88:0/4097388567 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f52bc019020 con 0x7f52c00b6720 2024-09-16T01:41:17.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.408+0000 7f52b5ffb640 1 --2- 172.21.15.88:0/4097388567 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f52a406c2b0 0x7f52a406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:17.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.408+0000 7f52b5ffb640 1 -- 172.21.15.88:0/4097388567 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6598+0+0 (secure 0 0 0) 0x7f52bc08b3d0 con 0x7f52c00b6720 2024-09-16T01:41:17.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.408+0000 7f52c4c64640 1 --2- 172.21.15.88:0/4097388567 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f52a406c2b0 0x7f52a406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:17.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.408+0000 7f52c4c64640 1 --2- 172.21.15.88:0/4097388567 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f52a406c2b0 0x7f52a406e770 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f52b0034000 tx=0x7f52b002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:17.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.412+0000 7f52b5ffb640 1 -- 172.21.15.88:0/4097388567 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f52bc020070 con 0x7f52c00b6720 2024-09-16T01:41:17.555 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:41:17.559 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:41:17.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:17.619+0000 7f52c76c9640 1 -- 172.21.15.88:0/4097388567 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f52c015dab0 con 0x7f52c00b6720 2024-09-16T01:41:18.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.215+0000 7f52b5ffb640 1 -- 172.21.15.88:0/4097388567 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v129) ==== 161+0+0 (secure 0 0 0) 0x7f52bc059200 con 0x7f52c00b6720 2024-09-16T01:41:18.217 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:41:18.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.218+0000 7f52c76c9640 1 -- 172.21.15.88:0/4097388567 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f52a406c2b0 msgr2=0x7f52a406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:18.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.218+0000 7f52c76c9640 1 --2- 172.21.15.88:0/4097388567 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f52a406c2b0 0x7f52a406e770 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f52b0034000 tx=0x7f52b002e000 comp rx=0 tx=0).stop 2024-09-16T01:41:18.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.218+0000 7f52c76c9640 1 -- 172.21.15.88:0/4097388567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52c00b6720 msgr2=0x7f52c014dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:18.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.218+0000 7f52c76c9640 1 --2- 172.21.15.88:0/4097388567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52c00b6720 0x7f52c014dc20 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f52bc00c7c0 tx=0x7f52bc00cc80 comp rx=0 tx=0).stop 2024-09-16T01:41:18.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.218+0000 7f52c76c9640 1 -- 172.21.15.88:0/4097388567 shutdown_connections 2024-09-16T01:41:18.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.218+0000 7f52c76c9640 1 --2- 172.21.15.88:0/4097388567 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f52a406c2b0 0x7f52a406e770 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:18.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.218+0000 7f52c76c9640 1 --2- 172.21.15.88:0/4097388567 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52c015e7a0 0x7f52c00c87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:18.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.218+0000 7f52c76c9640 1 --2- 172.21.15.88:0/4097388567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52c00cf1f0 0x7f52c00c8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:18.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.218+0000 7f52c76c9640 1 --2- 172.21.15.88:0/4097388567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52c00b6720 0x7f52c014dc20 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:18.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.218+0000 7f52c76c9640 1 -- 172.21.15.88:0/4097388567 >> 172.21.15.88:0/4097388567 conn(0x7f52c00d11e0 msgr2=0x7f52c0157490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:18.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.218+0000 7f52c76c9640 1 -- 172.21.15.88:0/4097388567 shutdown_connections 2024-09-16T01:41:18.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.218+0000 7f52c76c9640 1 -- 172.21.15.88:0/4097388567 wait complete. 2024-09-16T01:41:18.234 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:41:18.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.380+0000 7f3cb65c4640 1 Processor -- start 2024-09-16T01:41:18.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.380+0000 7f3cb65c4640 1 -- start start 2024-09-16T01:41:18.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.380+0000 7f3cb65c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cb00cca40 0x7f3cb00b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:18.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.380+0000 7f3cb65c4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cb0071720 con 0x7f3cb00720d0 2024-09-16T01:41:18.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.380+0000 7f3cb65c4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cb0073920 con 0x7f3cb015ad10 2024-09-16T01:41:18.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.380+0000 7f3cb65c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3cb0070fb0 con 0x7f3cb00cca40 2024-09-16T01:41:18.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.381+0000 7f3caf7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3cb00720d0 0x7f3cb00b3ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49582/0 (socket says 172.21.15.88:49582) 2024-09-16T01:41:18.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.381+0000 7f3caf7fe640 1 -- 172.21.15.88:0/3852508731 learned_addr learned my addr 172.21.15.88:0/3852508731 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:18.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.381+0000 7f3caeffd640 1 -- 172.21.15.88:0/3852508731 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3912966376 0 0) 0x7f3cb0071720 con 0x7f3cb00720d0 2024-09-16T01:41:18.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.381+0000 7f3cb4b61640 1 --2- 172.21.15.88:0/3852508731 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cb00cca40 0x7f3cb00b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:18.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.381+0000 7f3caeffd640 1 -- 172.21.15.88:0/3852508731 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c98003440 con 0x7f3cb00720d0 2024-09-16T01:41:18.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.381+0000 7f3caeffd640 1 -- 172.21.15.88:0/3852508731 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1632267894 0 0) 0x7f3c98003440 con 0x7f3cb00720d0 2024-09-16T01:41:18.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.381+0000 7f3caeffd640 1 -- 172.21.15.88:0/3852508731 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cb00cca40 msgr2=0x7f3cb00b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:18.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.381+0000 7f3caeffd640 1 --2- 172.21.15.88:0/3852508731 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cb00cca40 0x7f3cb00b45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:18.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.381+0000 7f3caeffd640 1 -- 172.21.15.88:0/3852508731 >> v1:172.21.15.154:6789/0 conn(0x7f3cb015ad10 legacy=0x7f3cb00719c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:18.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.381+0000 7f3caeffd640 1 -- 172.21.15.88:0/3852508731 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cb0073c00 con 0x7f3cb00720d0 2024-09-16T01:41:18.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.381+0000 7f3caeffd640 1 -- 172.21.15.88:0/3852508731 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f3ca4001670 con 0x7f3cb00720d0 2024-09-16T01:41:18.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.381+0000 7f3caeffd640 1 -- 172.21.15.88:0/3852508731 >> v1:172.21.15.88:6789/0 conn(0x7f3cb00720d0 legacy=0x7f3cb00b3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:18.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.381+0000 7f3caeffd640 1 --2- 172.21.15.88:0/3852508731 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cb00cca40 0x7f3c98003ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:18.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.381+0000 7f3caeffd640 1 --2- 172.21.15.88:0/3852508731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c98004010 0x7f3c9800a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:18.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.381+0000 7f3caeffd640 1 --2- 172.21.15.88:0/3852508731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c9800af80 0x7f3c9800d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:18.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.381+0000 7f3caeffd640 1 -- 172.21.15.88:0/3852508731 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb0070fb0 con 0x7f3c98004010 2024-09-16T01:41:18.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.381+0000 7f3caeffd640 1 -- 172.21.15.88:0/3852508731 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9800dcd0 con 0x7f3c9800af80 2024-09-16T01:41:18.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.381+0000 7f3caeffd640 1 -- 172.21.15.88:0/3852508731 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3c9800ded0 con 0x7f3cb00cca40 2024-09-16T01:41:18.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.381+0000 7f3caf7fe640 1 --2- 172.21.15.88:0/3852508731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c98004010 0x7f3c9800a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:18.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.381+0000 7f3caffff640 1 --2- 172.21.15.88:0/3852508731 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cb00cca40 0x7f3c98003ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:18.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.381+0000 7f3cb4b61640 1 --2- 172.21.15.88:0/3852508731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c9800af80 0x7f3c9800d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:18.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.382+0000 7f3caf7fe640 1 -- 172.21.15.88:0/3852508731 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cb00cca40 msgr2=0x7f3c98003ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:18.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.382+0000 7f3caf7fe640 1 --2- 172.21.15.88:0/3852508731 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cb00cca40 0x7f3c98003ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:18.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.382+0000 7f3caf7fe640 1 -- 172.21.15.88:0/3852508731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c9800af80 msgr2=0x7f3c9800d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:18.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.382+0000 7f3caf7fe640 1 --2- 172.21.15.88:0/3852508731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c9800af80 0x7f3c9800d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:18.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.382+0000 7f3caf7fe640 1 -- 172.21.15.88:0/3852508731 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c98003440 con 0x7f3c98004010 2024-09-16T01:41:18.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.382+0000 7f3cb4b61640 1 --2- 172.21.15.88:0/3852508731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c9800af80 0x7f3c9800d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:18.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.382+0000 7f3caf7fe640 1 --2- 172.21.15.88:0/3852508731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c98004010 0x7f3c9800a860 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f3ca400c290 tx=0x7f3c98003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e2ff5e5c4b4344a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:18.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.382+0000 7f3caeffd640 1 -- 172.21.15.88:0/3852508731 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca4007d50 con 0x7f3c98004010 2024-09-16T01:41:18.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.382+0000 7f3caeffd640 1 -- 172.21.15.88:0/3852508731 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ca4005af0 con 0x7f3c98004010 2024-09-16T01:41:18.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.382+0000 7f3caeffd640 1 -- 172.21.15.88:0/3852508731 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca4007070 con 0x7f3c98004010 2024-09-16T01:41:18.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.383+0000 7f3cb65c4640 1 -- 172.21.15.88:0/3852508731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c98004010 msgr2=0x7f3c9800a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:18.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.383+0000 7f3cb65c4640 1 --2- 172.21.15.88:0/3852508731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3c98004010 0x7f3c9800a860 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f3ca400c290 tx=0x7f3c98003730 comp rx=0 tx=0).stop 2024-09-16T01:41:18.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.383+0000 7f3cb4b61640 1 -- 172.21.15.88:0/3852508731 reap_dead start 2024-09-16T01:41:18.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.383+0000 7f3cb65c4640 1 -- 172.21.15.88:0/3852508731 shutdown_connections 2024-09-16T01:41:18.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.383+0000 7f3cb65c4640 1 -- 172.21.15.88:0/3852508731 >> 172.21.15.88:0/3852508731 conn(0x7f3cb006d0f0 msgr2=0x7f3cb006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:18.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.383+0000 7f3cb65c4640 1 -- 172.21.15.88:0/3852508731 shutdown_connections 2024-09-16T01:41:18.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.383+0000 7f3cb65c4640 1 -- 172.21.15.88:0/3852508731 wait complete. 2024-09-16T01:41:18.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.384+0000 7f3cb65c4640 1 Processor -- start 2024-09-16T01:41:18.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.384+0000 7f3cb65c4640 1 -- start start 2024-09-16T01:41:18.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.385+0000 7f3cb65c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c98004010 0x7f3cb015d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:18.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.385+0000 7f3cb65c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cb0152060 0x7f3cb01e7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:18.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.385+0000 7f3cb65c4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3cb00cf380 0x7f3cb00cf800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:18.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.385+0000 7f3cb65c4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb00cc360 con 0x7f3cb00cf380 2024-09-16T01:41:18.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.385+0000 7f3cb65c4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb0071200 con 0x7f3cb0152060 2024-09-16T01:41:18.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.385+0000 7f3cb65c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3cb00cc4e0 con 0x7f3c98004010 2024-09-16T01:41:18.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.385+0000 7f3cb4b61640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3cb00cf380 0x7f3cb00cf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:18.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.385+0000 7f3cb4b61640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3cb00cf380 0x7f3cb00cf800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41630/0 (socket says 172.21.15.88:41630) 2024-09-16T01:41:18.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.385+0000 7f3cb4b61640 1 -- 172.21.15.88:0/4131784971 learned_addr learned my addr 172.21.15.88:0/4131784971 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:18.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.385+0000 7f3cb4b61640 1 -- 172.21.15.88:0/4131784971 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c98004010 msgr2=0x7f3cb015d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:18.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.385+0000 7f3cb4b61640 1 --2- 172.21.15.88:0/4131784971 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c98004010 0x7f3cb015d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:18.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.385+0000 7f3cb4b61640 1 -- 172.21.15.88:0/4131784971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cb0152060 msgr2=0x7f3cb01e7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:18.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.385+0000 7f3cb4b61640 1 --2- 172.21.15.88:0/4131784971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cb0152060 0x7f3cb01e7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:18.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.385+0000 7f3cb4b61640 1 -- 172.21.15.88:0/4131784971 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cb01524e0 con 0x7f3cb00cf380 2024-09-16T01:41:18.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.386+0000 7f3cb4b61640 1 --2- 172.21.15.88:0/4131784971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3cb00cf380 0x7f3cb00cf800 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f3c98004520 tx=0x7f3ca8010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:18.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.386+0000 7f3cacff9640 1 -- 172.21.15.88:0/4131784971 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8015440 con 0x7f3cb00cf380 2024-09-16T01:41:18.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.386+0000 7f3cacff9640 1 -- 172.21.15.88:0/4131784971 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ca801c070 con 0x7f3cb00cf380 2024-09-16T01:41:18.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.386+0000 7f3cb65c4640 1 -- 172.21.15.88:0/4131784971 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cb01e7640 con 0x7f3cb00cf380 2024-09-16T01:41:18.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.386+0000 7f3cacff9640 1 -- 172.21.15.88:0/4131784971 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8015820 con 0x7f3cb00cf380 2024-09-16T01:41:18.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.386+0000 7f3cb65c4640 1 -- 172.21.15.88:0/4131784971 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cb01e7bd0 con 0x7f3cb00cf380 2024-09-16T01:41:18.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.387+0000 7f3cb65c4640 1 -- 172.21.15.88:0/4131784971 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cb0073920 con 0x7f3cb00cf380 2024-09-16T01:41:18.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.387+0000 7f3cacff9640 1 -- 172.21.15.88:0/4131784971 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3ca8005d10 con 0x7f3cb00cf380 2024-09-16T01:41:18.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.388+0000 7f3cacff9640 1 --2- 172.21.15.88:0/4131784971 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3c9c06c2e0 0x7f3c9c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:18.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.388+0000 7f3cacff9640 1 -- 172.21.15.88:0/4131784971 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6598+0+0 (secure 0 0 0) 0x7f3ca808c130 con 0x7f3cb00cf380 2024-09-16T01:41:18.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.388+0000 7f3caffff640 1 --2- 172.21.15.88:0/4131784971 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3c9c06c2e0 0x7f3c9c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:18.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.388+0000 7f3caffff640 1 --2- 172.21.15.88:0/4131784971 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3c9c06c2e0 0x7f3c9c06e7a0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3cb01f75c0 tx=0x7f3ca0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:18.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.392+0000 7f3cacff9640 1 -- 172.21.15.88:0/4131784971 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3ca805a520 con 0x7f3cb00cf380 2024-09-16T01:41:18.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:18.626+0000 7f3cb65c4640 1 -- 172.21.15.88:0/4131784971 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f3cb00ca370 con 0x7f3cb00cf380 2024-09-16T01:41:19.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.223+0000 7f3cacff9640 1 -- 172.21.15.88:0/4131784971 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v130) ==== 101+0+0 (secure 0 0 0) 0x7f3ca8059e20 con 0x7f3cb00cf380 2024-09-16T01:41:19.225 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:41:19.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.225+0000 7f3cb65c4640 1 -- 172.21.15.88:0/4131784971 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3c9c06c2e0 msgr2=0x7f3c9c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:19.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.225+0000 7f3cb65c4640 1 --2- 172.21.15.88:0/4131784971 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3c9c06c2e0 0x7f3c9c06e7a0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3cb01f75c0 tx=0x7f3ca0009040 comp rx=0 tx=0).stop 2024-09-16T01:41:19.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.225+0000 7f3cb65c4640 1 -- 172.21.15.88:0/4131784971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3cb00cf380 msgr2=0x7f3cb00cf800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:19.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.225+0000 7f3cb65c4640 1 --2- 172.21.15.88:0/4131784971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3cb00cf380 0x7f3cb00cf800 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f3c98004520 tx=0x7f3ca8010590 comp rx=0 tx=0).stop 2024-09-16T01:41:19.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.226+0000 7f3cb65c4640 1 -- 172.21.15.88:0/4131784971 shutdown_connections 2024-09-16T01:41:19.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.226+0000 7f3cb65c4640 1 --2- 172.21.15.88:0/4131784971 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3c9c06c2e0 0x7f3c9c06e7a0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.226+0000 7f3cb65c4640 1 --2- 172.21.15.88:0/4131784971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3cb00cf380 0x7f3cb00cf800 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.226+0000 7f3cb65c4640 1 --2- 172.21.15.88:0/4131784971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cb0152060 0x7f3cb01e7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.226+0000 7f3cb65c4640 1 --2- 172.21.15.88:0/4131784971 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c98004010 0x7f3cb015d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.226+0000 7f3cb65c4640 1 -- 172.21.15.88:0/4131784971 >> 172.21.15.88:0/4131784971 conn(0x7f3cb006d0f0 msgr2=0x7f3cb00b3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:19.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.226+0000 7f3cb65c4640 1 -- 172.21.15.88:0/4131784971 shutdown_connections 2024-09-16T01:41:19.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.226+0000 7f3cb65c4640 1 -- 172.21.15.88:0/4131784971 wait complete. 2024-09-16T01:41:19.241 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:41:19.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.283+0000 7f5e21dcef40 1 Processor -- start 2024-09-16T01:41:19.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.283+0000 7f5e21dcef40 1 -- start start 2024-09-16T01:41:19.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.283+0000 7f5e21dcef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e0c0022d0 0x56410ecfd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.283+0000 7f5e21dcef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56410edb6060 0x56410ed47890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.283+0000 7f5e21dcef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56410edcc090 0x56410edcc520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.283+0000 7f5e21dcef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56410ec4e510 con 0x56410edb6060 2024-09-16T01:41:19.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.283+0000 7f5e21dcef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56410eca7c50 con 0x56410edcc090 2024-09-16T01:41:19.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.283+0000 7f5e21dcef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56410ed42550 con 0x7f5e0c0022d0 2024-09-16T01:41:19.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.284+0000 7f5e17fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56410edb6060 0x56410ed47890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.284+0000 7f5e17fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56410edb6060 0x56410ed47890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41646/0 (socket says 172.21.15.88:41646) 2024-09-16T01:41:19.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.284+0000 7f5e17fff640 1 -- 172.21.15.88:0/855994811 learned_addr learned my addr 172.21.15.88:0/855994811 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:19.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.284+0000 7f5e1f8c6640 1 --2- 172.21.15.88:0/855994811 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e0c0022d0 0x56410ecfd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.284+0000 7f5e200c7640 1 --2- 172.21.15.88:0/855994811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56410edcc090 0x56410edcc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.284+0000 7f5e17fff640 1 -- 172.21.15.88:0/855994811 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e0c0022d0 msgr2=0x56410ecfd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:19.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.284+0000 7f5e17fff640 1 --2- 172.21.15.88:0/855994811 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e0c0022d0 0x56410ecfd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.284+0000 7f5e17fff640 1 -- 172.21.15.88:0/855994811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56410edcc090 msgr2=0x56410edcc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:19.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.284+0000 7f5e17fff640 1 --2- 172.21.15.88:0/855994811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56410edcc090 0x56410edcc520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.284+0000 7f5e17fff640 1 -- 172.21.15.88:0/855994811 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56410edba450 con 0x56410edb6060 2024-09-16T01:41:19.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.284+0000 7f5e17fff640 1 --2- 172.21.15.88:0/855994811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56410edb6060 0x56410ed47890 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f5e08003ed0 tx=0x7f5e0800d6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eb01684da6de8c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:19.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.284+0000 7f5e1e8c4640 1 -- 172.21.15.88:0/855994811 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e08003110 con 0x56410edb6060 2024-09-16T01:41:19.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.284+0000 7f5e1e8c4640 1 -- 172.21.15.88:0/855994811 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e080032b0 con 0x56410edb6060 2024-09-16T01:41:19.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.284+0000 7f5e1e8c4640 1 -- 172.21.15.88:0/855994811 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e08001600 con 0x56410edb6060 2024-09-16T01:41:19.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.284+0000 7f5e21dcef40 1 -- 172.21.15.88:0/855994811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56410edb6060 msgr2=0x56410ed47890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:19.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.284+0000 7f5e21dcef40 1 --2- 172.21.15.88:0/855994811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56410edb6060 0x56410ed47890 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f5e08003ed0 tx=0x7f5e0800d6f0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.284+0000 7f5e21dcef40 1 -- 172.21.15.88:0/855994811 shutdown_connections 2024-09-16T01:41:19.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.284+0000 7f5e21dcef40 1 --2- 172.21.15.88:0/855994811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56410edcc090 0x56410edcc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.285+0000 7f5e21dcef40 1 --2- 172.21.15.88:0/855994811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56410edb6060 0x56410ed47890 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.285+0000 7f5e21dcef40 1 --2- 172.21.15.88:0/855994811 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e0c0022d0 0x56410ecfd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.285+0000 7f5e21dcef40 1 -- 172.21.15.88:0/855994811 >> 172.21.15.88:0/855994811 conn(0x56410ecfbbe0 msgr2=0x56410edb9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:19.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.285+0000 7f5e21dcef40 1 -- 172.21.15.88:0/855994811 shutdown_connections 2024-09-16T01:41:19.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.285+0000 7f5e21dcef40 1 -- 172.21.15.88:0/855994811 wait complete. 2024-09-16T01:41:19.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.285+0000 7f5e21dcef40 1 Processor -- start 2024-09-16T01:41:19.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.285+0000 7f5e21dcef40 1 -- start start 2024-09-16T01:41:19.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.286+0000 7f5e21dcef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56410edb6060 0x56410edae170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.286+0000 7f5e21dcef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56410edcc090 0x56410edd1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.286+0000 7f5e21dcef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e0c0022d0 0x56410ee6b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.286+0000 7f5e21dcef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56410edcca60 con 0x56410edcc090 2024-09-16T01:41:19.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.286+0000 7f5e21dcef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56410edaef20 con 0x56410edb6060 2024-09-16T01:41:19.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.286+0000 7f5e21dcef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56410edcb830 con 0x7f5e0c0022d0 2024-09-16T01:41:19.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.286+0000 7f5e17fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56410edcc090 0x56410edd1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.286+0000 7f5e17fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56410edcc090 0x56410edd1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41658/0 (socket says 172.21.15.88:41658) 2024-09-16T01:41:19.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.286+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 learned_addr learned my addr 172.21.15.88:0/1398598180 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:19.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.286+0000 7f5e1f8c6640 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56410edb6060 0x56410edae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.286+0000 7f5e200c7640 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e0c0022d0 0x56410ee6b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.286+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e0c0022d0 msgr2=0x56410ee6b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:19.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.286+0000 7f5e17fff640 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e0c0022d0 0x56410ee6b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.286+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56410edb6060 msgr2=0x56410edae170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:19.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.286+0000 7f5e17fff640 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56410edb6060 0x56410edae170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.286+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56410edcb5c0 con 0x56410edcc090 2024-09-16T01:41:19.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.286+0000 7f5e17fff640 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56410edcc090 0x56410edd1980 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x56410edb5dd0 tx=0x7f5e08001190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:19.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.287+0000 7f5e1d8c2640 1 -- 172.21.15.88:0/1398598180 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e08003460 con 0x56410edcc090 2024-09-16T01:41:19.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.287+0000 7f5e1d8c2640 1 -- 172.21.15.88:0/1398598180 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e08023020 con 0x56410edcc090 2024-09-16T01:41:19.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.287+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56410ecfc9b0 con 0x56410edcc090 2024-09-16T01:41:19.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.287+0000 7f5e1d8c2640 1 -- 172.21.15.88:0/1398598180 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e08011440 con 0x56410edcc090 2024-09-16T01:41:19.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.287+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56410edd1180 con 0x56410edcc090 2024-09-16T01:41:19.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.288+0000 7f5e1d8c2640 1 -- 172.21.15.88:0/1398598180 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5e08019070 con 0x56410edcc090 2024-09-16T01:41:19.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.288+0000 7f5e1d8c2640 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e0006ac90 0x7f5e0006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.289+0000 7f5e1d8c2640 1 -- 172.21.15.88:0/1398598180 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6598+0+0 (secure 0 0 0) 0x7f5e0808bc70 con 0x56410edcc090 2024-09-16T01:41:19.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.289+0000 7f5e1f8c6640 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e0006ac90 0x7f5e0006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.296 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:41:19.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.289+0000 7f5e21dcef40 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x56410ed44f00 0x56410edd2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.289+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410edae6b0 con 0x56410ed44f00 2024-09-16T01:41:19.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.289+0000 7f5e200c7640 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x56410ed44f00 0x56410edd2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.289+0000 7f5e1f8c6640 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e0006ac90 0x7f5e0006d150 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f5e10056000 tx=0x7f5e10051000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:19.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.289+0000 7f5e200c7640 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x56410ed44f00 0x56410edd2430 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:19.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.291+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v97'288 uv262 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.291+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) -- 0x56410edb9ea0 con 0x56410ed44f00 2024-09-16T01:41:19.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.292+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v97'289 uv289 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.292+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410edd2970 con 0x56410ed44f00 2024-09-16T01:41:19.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.293+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v97'290 uv290 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.298 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:41:19.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.293+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee78830 con 0x56410ed44f00 2024-09-16T01:41:19.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.294+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v97'291 uv291 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.294+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee78bd0 con 0x56410ed44f00 2024-09-16T01:41:19.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.295+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v97'292 uv292 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.295+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee77820 con 0x56410ed44f00 2024-09-16T01:41:19.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.296+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v97'293 uv293 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.296+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee77bc0 con 0x56410ed44f00 2024-09-16T01:41:19.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.297+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v97'294 uv268 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.297+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) -- 0x56410ee77f60 con 0x56410ed44f00 2024-09-16T01:41:19.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.298+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v97'295 uv295 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.298+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee77f60 con 0x56410ed44f00 2024-09-16T01:41:19.301 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:41:19.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.299+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v97'296 uv296 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.299+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee77f60 con 0x56410ed44f00 2024-09-16T01:41:19.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.300+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v97'297 uv297 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.300+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee77f60 con 0x56410ed44f00 2024-09-16T01:41:19.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.301+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v97'298 uv298 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.301+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee77f60 con 0x56410ed44f00 2024-09-16T01:41:19.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.302+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v97'299 uv299 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.302+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee77f60 con 0x56410ed44f00 2024-09-16T01:41:19.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.303+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v97'300 uv275 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.303+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) -- 0x56410ee77f60 con 0x56410ed44f00 2024-09-16T01:41:19.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.304+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v97'301 uv301 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.304+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee77f60 con 0x56410ed44f00 2024-09-16T01:41:19.308 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:41:19.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.306+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v97'302 uv302 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.306+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee77f60 con 0x56410ed44f00 2024-09-16T01:41:19.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.307+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v97'303 uv303 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.307+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee77f60 con 0x56410ed44f00 2024-09-16T01:41:19.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.308+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v97'304 uv304 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.308+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee77f60 con 0x56410ed44f00 2024-09-16T01:41:19.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.309+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v97'305 uv305 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.309+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee78320 con 0x56410ed44f00 2024-09-16T01:41:19.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.310+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v97'306 uv306 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.310+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee78210 con 0x56410ed44f00 2024-09-16T01:41:19.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.311+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v97'307 uv281 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.311+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) -- 0x56410ee78210 con 0x56410ed44f00 2024-09-16T01:41:19.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.312+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v97'308 uv308 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.312+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee78210 con 0x56410ed44f00 2024-09-16T01:41:19.315 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:41:19.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.313+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v97'309 uv309 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.313+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee78210 con 0x56410ed44f00 2024-09-16T01:41:19.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.314+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v97'310 uv310 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.314+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee78210 con 0x56410ed44f00 2024-09-16T01:41:19.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.315+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v97'311 uv311 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.315+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee78210 con 0x56410ed44f00 2024-09-16T01:41:19.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.316+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v97'312 uv312 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.316+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee78210 con 0x56410ed44f00 2024-09-16T01:41:19.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.317+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v97'313 uv287 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.317+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) -- 0x56410ee78210 con 0x56410ed44f00 2024-09-16T01:41:19.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.318+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v97'314 uv314 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.318+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee78210 con 0x56410ed44f00 2024-09-16T01:41:19.320 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:41:19.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.318+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v97'315 uv315 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.319+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee78210 con 0x56410ed44f00 2024-09-16T01:41:19.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.320+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v97'316 uv316 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.320+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee78210 con 0x56410ed44f00 2024-09-16T01:41:19.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.321+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v97'317 uv317 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.321+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee78210 con 0x56410ed44f00 2024-09-16T01:41:19.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.322+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v97'318 uv318 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.322+0000 7f5e21dcef40 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x56410ee79480 0x56410ee7b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.322+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee78210 con 0x56410ee79480 2024-09-16T01:41:19.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.322+0000 7f5e1f8c6640 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x56410ee79480 0x56410ee7b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.322+0000 7f5e1f8c6640 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x56410ee79480 0x56410ee7b870 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:19.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.324+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v97'220 uv203 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee79480 2024-09-16T01:41:19.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.324+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7bdb0 con 0x56410ee79480 2024-09-16T01:41:19.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.328+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v97'221 uv221 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee79480 2024-09-16T01:41:19.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.328+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7c1d0 con 0x56410ee79480 2024-09-16T01:41:19.331 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:41:19.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.329+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v97'222 uv222 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee79480 2024-09-16T01:41:19.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.329+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7ca70 con 0x56410ee79480 2024-09-16T01:41:19.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.332+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v97'223 uv223 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee79480 2024-09-16T01:41:19.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.332+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7ce10 con 0x56410ee79480 2024-09-16T01:41:19.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.335+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v97'224 uv224 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee79480 2024-09-16T01:41:19.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.335+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7d1b0 con 0x56410ee79480 2024-09-16T01:41:19.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.337+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v97'225 uv208 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee79480 2024-09-16T01:41:19.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.338+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7d5a0 con 0x56410ee79480 2024-09-16T01:41:19.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.340+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v97'226 uv226 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee79480 2024-09-16T01:41:19.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.340+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7d990 con 0x56410ee79480 2024-09-16T01:41:19.345 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:41:19.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.343+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v97'227 uv227 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee79480 2024-09-16T01:41:19.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.343+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7d990 con 0x56410ee79480 2024-09-16T01:41:19.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.344+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v97'228 uv228 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee79480 2024-09-16T01:41:19.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.344+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7d940 con 0x56410ee79480 2024-09-16T01:41:19.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.347+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v97'229 uv229 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee79480 2024-09-16T01:41:19.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.347+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7d940 con 0x56410ee79480 2024-09-16T01:41:19.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.349+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v97'230 uv213 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee79480 2024-09-16T01:41:19.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.349+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7d940 con 0x56410ee79480 2024-09-16T01:41:19.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.352+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v97'231 uv231 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee79480 2024-09-16T01:41:19.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.352+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7d940 con 0x56410ee79480 2024-09-16T01:41:19.357 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:41:19.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.355+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v97'232 uv232 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee79480 2024-09-16T01:41:19.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.355+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7d940 con 0x56410ee79480 2024-09-16T01:41:19.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.356+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v97'233 uv233 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee79480 2024-09-16T01:41:19.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.356+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7d940 con 0x56410ee79480 2024-09-16T01:41:19.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.357+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v97'234 uv234 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee79480 2024-09-16T01:41:19.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.357+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7d940 con 0x56410ee79480 2024-09-16T01:41:19.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.358+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v97'235 uv219 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee79480 2024-09-16T01:41:19.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.358+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7d940 con 0x56410ee79480 2024-09-16T01:41:19.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.359+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v97'236 uv236 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee79480 2024-09-16T01:41:19.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.359+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7d940 con 0x56410ee79480 2024-09-16T01:41:19.362 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:41:19.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.360+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v97'237 uv237 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee79480 2024-09-16T01:41:19.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.360+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7d940 con 0x56410ee79480 2024-09-16T01:41:19.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.361+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v97'238 uv238 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee79480 2024-09-16T01:41:19.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.361+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7d940 con 0x56410ee79480 2024-09-16T01:41:19.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.362+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v97'239 uv239 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee79480 2024-09-16T01:41:19.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.362+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7d940 con 0x56410ee79480 2024-09-16T01:41:19.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.363+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v97'240 uv240 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee79480 2024-09-16T01:41:19.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.363+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7d940 con 0x56410ed44f00 2024-09-16T01:41:19.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.364+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v97'57 uv56 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.364+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7d940 con 0x56410ed44f00 2024-09-16T01:41:19.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.365+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v97'58 uv58 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.365+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7d940 con 0x56410ed44f00 2024-09-16T01:41:19.368 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:41:19.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.366+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v97'59 uv59 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.366+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7d940 con 0x56410ed44f00 2024-09-16T01:41:19.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.367+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v97'60 uv60 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.367+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7d940 con 0x56410ed44f00 2024-09-16T01:41:19.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.368+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v97'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.368+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7d940 con 0x56410ed44f00 2024-09-16T01:41:19.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.369+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v97'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.369+0000 7f5e21dcef40 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x56410ee7e2b0 0x56410ee7e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.369+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee80ed0 con 0x56410ee7e2b0 2024-09-16T01:41:19.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.369+0000 7f5e17fff640 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x56410ee7e2b0 0x56410ee7e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.369+0000 7f5e17fff640 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x56410ee7e2b0 0x56410ee7e710 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:19.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.370+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v97'47 uv46 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f5e0800e040 con 0x56410ee7e2b0 2024-09-16T01:41:19.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.370+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) -- 0x56410ee81270 con 0x56410ee7e2b0 2024-09-16T01:41:19.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.371+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v97'48 uv48 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f5e0800e040 con 0x56410ee7e2b0 2024-09-16T01:41:19.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.371+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee81610 con 0x56410ee7e2b0 2024-09-16T01:41:19.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.372+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v97'49 uv49 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f5e0800e040 con 0x56410ee7e2b0 2024-09-16T01:41:19.374 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:41:19.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.372+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee7c6c0 con 0x56410ee7e2b0 2024-09-16T01:41:19.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.373+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v97'50 uv50 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f5e0800e040 con 0x56410ee7e2b0 2024-09-16T01:41:19.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.373+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee81f40 con 0x56410ed44f00 2024-09-16T01:41:19.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.374+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v97'57 uv56 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.374+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) -- 0x56410ee81f40 con 0x56410ed44f00 2024-09-16T01:41:19.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.375+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v97'58 uv58 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.375+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee81f40 con 0x56410ed44f00 2024-09-16T01:41:19.378 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:41:19.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.376+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v97'59 uv59 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.376+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee82f50 con 0x56410ed44f00 2024-09-16T01:41:19.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.377+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v97'60 uv60 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.377+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee82f50 con 0x56410ed44f00 2024-09-16T01:41:19.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.378+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v97'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.379+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee81f40 con 0x56410ed44f00 2024-09-16T01:41:19.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.380+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v97'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.380+0000 7f5e21dcef40 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56410ee82610 0x56410ee84a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.380+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee85020 con 0x56410ee82610 2024-09-16T01:41:19.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.380+0000 7f5e200c7640 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56410ee82610 0x56410ee84a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.380+0000 7f5e200c7640 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56410ee82610 0x56410ee84a00 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:19.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.381+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v97'105 uv99 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.381+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) -- 0x56410ee850e0 con 0x56410ee82610 2024-09-16T01:41:19.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.382+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v97'106 uv106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.382+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee854d0 con 0x56410ee82610 2024-09-16T01:41:19.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.383+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v97'107 uv107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.385 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:41:19.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.383+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee864e0 con 0x56410ee82610 2024-09-16T01:41:19.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.384+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v97'108 uv108 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.384+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee864e0 con 0x56410ee82610 2024-09-16T01:41:19.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.385+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v97'109 uv109 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.385+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee854d0 con 0x56410ee82610 2024-09-16T01:41:19.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.386+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v97'110 uv110 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.386+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee854d0 con 0x56410ee82610 2024-09-16T01:41:19.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.387+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v97'111 uv104 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.387+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) -- 0x56410ee854d0 con 0x56410ee82610 2024-09-16T01:41:19.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.389+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v97'112 uv112 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.389+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee854d0 con 0x56410ee82610 2024-09-16T01:41:19.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.390+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v97'113 uv113 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.392 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:41:19.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.390+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee854d0 con 0x56410ee82610 2024-09-16T01:41:19.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.391+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v97'114 uv114 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.391+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee854d0 con 0x56410ee82610 2024-09-16T01:41:19.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.392+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v97'115 uv115 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.392+0000 7f5e21dcef40 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x56410ee859e0 0x56410ee87df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.392+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee88510 con 0x56410ee859e0 2024-09-16T01:41:19.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.393+0000 7f5e1f8c6640 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x56410ee859e0 0x56410ee87df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.393+0000 7f5e1f8c6640 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x56410ee859e0 0x56410ee87df0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:19.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.394+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v97'46 uv45 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee859e0 2024-09-16T01:41:19.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.394+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) -- 0x56410ee888e0 con 0x56410ee859e0 2024-09-16T01:41:19.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.395+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v97'47 uv47 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee859e0 2024-09-16T01:41:19.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.395+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee888e0 con 0x56410ee859e0 2024-09-16T01:41:19.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.396+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v97'48 uv48 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee859e0 2024-09-16T01:41:19.398 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:41:19.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.396+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee888e0 con 0x56410ee859e0 2024-09-16T01:41:19.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.397+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v97'49 uv49 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee859e0 2024-09-16T01:41:19.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.397+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee888e0 con 0x56410ee859e0 2024-09-16T01:41:19.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.398+0000 7f5e1f8c6640 1 -- 172.21.15.88:0/1398598180 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v97'50 uv50 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f5e1004dac0 con 0x56410ee859e0 2024-09-16T01:41:19.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.398+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee888e0 con 0x56410ed44f00 2024-09-16T01:41:19.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.399+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v97'57 uv56 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.399+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) -- 0x56410ee888e0 con 0x56410ed44f00 2024-09-16T01:41:19.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.400+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v97'58 uv58 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.400+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee888e0 con 0x56410ed44f00 2024-09-16T01:41:19.402 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:41:19.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.401+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v97'59 uv59 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.401+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee888e0 con 0x56410ed44f00 2024-09-16T01:41:19.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.401+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v97'60 uv60 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.401+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee888e0 con 0x56410ed44f00 2024-09-16T01:41:19.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.402+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v97'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.402+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee888e0 con 0x56410ed44f00 2024-09-16T01:41:19.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.403+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v97'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ed44f00 2024-09-16T01:41:19.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.403+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee888e0 con 0x56410ee7e2b0 2024-09-16T01:41:19.406 INFO:teuthology.orchestra.run.smithi088.stdout:Write #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:41:19.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.404+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v97'95 uv95 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f5e0800e040 con 0x56410ee7e2b0 2024-09-16T01:41:19.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.404+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee81aa0 con 0x56410ee7e2b0 2024-09-16T01:41:19.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.405+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [write 0~34] v97'96 uv96 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f5e0800e040 con 0x56410ee7e2b0 2024-09-16T01:41:19.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.405+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee89180 con 0x56410ee7e2b0 2024-09-16T01:41:19.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.406+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [omap-set-header] v97'97 uv97 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f5e0800e040 con 0x56410ee7e2b0 2024-09-16T01:41:19.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.406+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee89570 con 0x56410ee7e2b0 2024-09-16T01:41:19.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.407+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 100.00000000 [create] v97'98 uv93 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5e0800e040 con 0x56410ee7e2b0 2024-09-16T01:41:19.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.407+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) -- 0x56410ee89960 con 0x56410ee7e2b0 2024-09-16T01:41:19.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.408+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 100.00000000 [delete] v97'99 uv99 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e0800e040 con 0x56410ee7e2b0 2024-09-16T01:41:19.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.408+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee89960 con 0x56410ee7e2b0 2024-09-16T01:41:19.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.409+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v97'100 uv100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e0800e040 con 0x56410ee7e2b0 2024-09-16T01:41:19.411 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:41:19.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.409+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee89910 con 0x56410ee7e2b0 2024-09-16T01:41:19.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.410+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [omap-set-header] v97'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e0800e040 con 0x56410ee7e2b0 2024-09-16T01:41:19.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.410+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8c7e0 con 0x56410ee7e2b0 2024-09-16T01:41:19.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.411+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v97'102 uv102 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e0800e040 con 0x56410ee7e2b0 2024-09-16T01:41:19.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.411+0000 7f5e21dcef40 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x56410ee89de0 0x56410ee8c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.411+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:99 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8cea0 con 0x56410ee89de0 2024-09-16T01:41:19.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.411+0000 7f5e17fff640 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x56410ee89de0 0x56410ee8c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.411+0000 7f5e17fff640 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x56410ee89de0 0x56410ee8c1d0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:19.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.412+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(99 mds0_sessionmap [create] v97'44 uv43 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f5e0800e040 con 0x56410ee89de0 2024-09-16T01:41:19.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.412+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:100 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8cea0 con 0x56410ee89de0 2024-09-16T01:41:19.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.413+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v97'45 uv45 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f5e0800e040 con 0x56410ee89de0 2024-09-16T01:41:19.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.413+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8cea0 con 0x56410ee89de0 2024-09-16T01:41:19.416 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:41:19.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.414+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(101 mds0_sessionmap [create] v97'46 uv46 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f5e0800e040 con 0x56410ee89de0 2024-09-16T01:41:19.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.414+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8cf20 con 0x56410ee89de0 2024-09-16T01:41:19.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.415+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v97'47 uv47 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f5e0800e040 con 0x56410ee89de0 2024-09-16T01:41:19.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.415+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8cf20 con 0x56410ee7e2b0 2024-09-16T01:41:19.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.416+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(103 609.00000000 [create] v97'57 uv56 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5e0800e040 con 0x56410ee7e2b0 2024-09-16T01:41:19.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.416+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8cf20 con 0x56410ee7e2b0 2024-09-16T01:41:19.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.417+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(104 609.00000000 [delete] v97'58 uv58 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e0800e040 con 0x56410ee7e2b0 2024-09-16T01:41:19.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.417+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8cea0 con 0x56410ee7e2b0 2024-09-16T01:41:19.421 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:41:19.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.419+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v97'59 uv59 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e0800e040 con 0x56410ee7e2b0 2024-09-16T01:41:19.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.419+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8deb0 con 0x56410ee7e2b0 2024-09-16T01:41:19.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.420+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v97'60 uv60 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e0800e040 con 0x56410ee7e2b0 2024-09-16T01:41:19.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.420+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8de60 con 0x56410ee7e2b0 2024-09-16T01:41:19.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.421+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v97'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e0800e040 con 0x56410ee7e2b0 2024-09-16T01:41:19.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.421+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8ce50 con 0x56410ee7e2b0 2024-09-16T01:41:19.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.422+0000 7f5e17fff640 1 -- 172.21.15.88:0/1398598180 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [omap-set-header] v97'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e0800e040 con 0x56410ee7e2b0 2024-09-16T01:41:19.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.422+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8ce50 con 0x56410ee82610 2024-09-16T01:41:19.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.423+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(109 400.00000000 [create] v97'47 uv46 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.423+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8ce50 con 0x56410ee82610 2024-09-16T01:41:19.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.424+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(110 400.00000000 [delete] v97'48 uv48 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.424+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8ce50 con 0x56410ee82610 2024-09-16T01:41:19.427 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:41:19.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.424+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(111 400.00000000 [create] v97'49 uv49 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.425+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8ce50 con 0x56410ee82610 2024-09-16T01:41:19.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.426+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v97'50 uv50 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.426+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8ce50 con 0x56410ee82610 2024-09-16T01:41:19.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.428+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v97'51 uv51 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.429+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8ce50 con 0x56410ee82610 2024-09-16T01:41:19.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.436+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(114 100.00000000.inode [create] v97'119 uv111 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.436+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8ce50 con 0x56410ee82610 2024-09-16T01:41:19.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.437+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(115 100.00000000.inode [delete] v97'120 uv120 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.437+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8ce50 con 0x56410ee82610 2024-09-16T01:41:19.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.439+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(116 100.00000000.inode [create] v97'121 uv121 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.442 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:41:19.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.439+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8ce50 con 0x56410ee82610 2024-09-16T01:41:19.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.442+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v97'122 uv122 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.442+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8ce50 con 0x56410ee82610 2024-09-16T01:41:19.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.445+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v97'123 uv123 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.445+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8ce50 con 0x56410ee82610 2024-09-16T01:41:19.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.448+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(119 1.00000000 [create] v97'124 uv118 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.448+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8ce50 con 0x56410ee82610 2024-09-16T01:41:19.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.450+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(120 1.00000000 [delete] v97'125 uv125 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.450+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8ce50 con 0x56410ee82610 2024-09-16T01:41:19.456 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:41:19.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.453+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(121 1.00000000 [create] v97'126 uv126 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.453+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8ce50 con 0x56410ee82610 2024-09-16T01:41:19.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.456+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v97'127 uv127 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.456+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8ce50 con 0x56410ee82610 2024-09-16T01:41:19.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.459+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v97'128 uv128 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.459+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8ce50 con 0x56410ee82610 2024-09-16T01:41:19.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.462+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v97'129 uv129 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.462+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) -- 0x56410ee8d2d0 con 0x56410ee82610 2024-09-16T01:41:19.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.464+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v97'130 uv130 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f5e18011040 con 0x56410ee82610 2024-09-16T01:41:19.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.464+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x56410ed44f00 msgr2=0x56410edd2430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:19.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.464+0000 7f5e21dcef40 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x56410ed44f00 0x56410edd2430 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.465+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x56410ee7e2b0 msgr2=0x56410ee7e710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:19.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.465+0000 7f5e21dcef40 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x56410ee7e2b0 0x56410ee7e710 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.465+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x56410ee89de0 msgr2=0x56410ee8c1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:19.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.465+0000 7f5e21dcef40 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x56410ee89de0 0x56410ee8c1d0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.465+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x56410ee79480 msgr2=0x56410ee7b870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:19.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.465+0000 7f5e21dcef40 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x56410ee79480 0x56410ee7b870 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.465+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56410ee82610 msgr2=0x56410ee84a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:19.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.465+0000 7f5e200c7640 1 -- 172.21.15.88:0/1398598180 reap_dead start 2024-09-16T01:41:19.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.465+0000 7f5e21dcef40 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56410ee82610 0x56410ee84a00 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.465+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x56410ee859e0 msgr2=0x56410ee87df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:19.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.465+0000 7f5e21dcef40 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x56410ee859e0 0x56410ee87df0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.465+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e0006ac90 msgr2=0x7f5e0006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:19.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.465+0000 7f5e21dcef40 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e0006ac90 0x7f5e0006d150 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f5e10056000 tx=0x7f5e10051000 comp rx=0 tx=0).stop 2024-09-16T01:41:19.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.465+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56410edcc090 msgr2=0x56410edd1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:19.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.465+0000 7f5e21dcef40 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56410edcc090 0x56410edd1980 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x56410edb5dd0 tx=0x7f5e08001190 comp rx=0 tx=0).stop 2024-09-16T01:41:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.465+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 shutdown_connections 2024-09-16T01:41:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.465+0000 7f5e21dcef40 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e0006ac90 0x7f5e0006d150 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.465+0000 7f5e21dcef40 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56410edcc090 0x56410edd1980 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.465+0000 7f5e21dcef40 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x56410ee859e0 0x56410ee87df0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.465+0000 7f5e21dcef40 1 --2- 172.21.15.88:0/1398598180 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56410ee82610 0x56410ee84a00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.465+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 >> 172.21.15.88:0/1398598180 conn(0x56410ecfbbe0 msgr2=0x56410edd7170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.465+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 shutdown_connections 2024-09-16T01:41:19.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.465+0000 7f5e21dcef40 1 -- 172.21.15.88:0/1398598180 wait complete. 2024-09-16T01:41:19.471 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 100.00000000 2024-09-16T01:41:19.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.551+0000 7f073ff2ef40 1 Processor -- start 2024-09-16T01:41:19.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.551+0000 7f073ff2ef40 1 -- start start 2024-09-16T01:41:19.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.552+0000 7f073ff2ef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dfb776dfe0 0x55dfb78079a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.552+0000 7f073ff2ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dfb776ea60 0x55dfb7815250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.552+0000 7f073ff2ef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfb7788250 0x55dfb78a84e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.552+0000 7f073ff2ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55dfb7690510 con 0x55dfb776ea60 2024-09-16T01:41:19.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.552+0000 7f073ff2ef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dfb7817120 con 0x55dfb7788250 2024-09-16T01:41:19.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.552+0000 7f073ff2ef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55dfb789cdc0 con 0x55dfb776dfe0 2024-09-16T01:41:19.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.552+0000 7f073c7bc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dfb776ea60 0x55dfb7815250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.552+0000 7f073c7bc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dfb776ea60 0x55dfb7815250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41682/0 (socket says 172.21.15.88:41682) 2024-09-16T01:41:19.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.552+0000 7f073c7bc640 1 -- 172.21.15.88:0/3425915562 learned_addr learned my addr 172.21.15.88:0/3425915562 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:19.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.552+0000 7f073cfbd640 1 --2- 172.21.15.88:0/3425915562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dfb776dfe0 0x55dfb78079a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.552+0000 7f073d7be640 1 --2- 172.21.15.88:0/3425915562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfb7788250 0x55dfb78a84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.552+0000 7f073c7bc640 1 -- 172.21.15.88:0/3425915562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dfb776dfe0 msgr2=0x55dfb78079a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:19.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.552+0000 7f073c7bc640 1 --2- 172.21.15.88:0/3425915562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dfb776dfe0 0x55dfb78079a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.552+0000 7f073c7bc640 1 -- 172.21.15.88:0/3425915562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfb7788250 msgr2=0x55dfb78a84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:19.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.552+0000 7f073c7bc640 1 --2- 172.21.15.88:0/3425915562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfb7788250 0x55dfb78a84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.553+0000 7f073c7bc640 1 -- 172.21.15.88:0/3425915562 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfb7814740 con 0x55dfb776ea60 2024-09-16T01:41:19.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.553+0000 7f073c7bc640 1 --2- 172.21.15.88:0/3425915562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dfb776ea60 0x55dfb7815250 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f0724002de0 tx=0x7f072400b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=846f756b515900 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:19.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.553+0000 7f073b7ba640 1 -- 172.21.15.88:0/3425915562 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072400f400 con 0x55dfb776ea60 2024-09-16T01:41:19.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.553+0000 7f073b7ba640 1 -- 172.21.15.88:0/3425915562 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0724014070 con 0x55dfb776ea60 2024-09-16T01:41:19.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.553+0000 7f073b7ba640 1 -- 172.21.15.88:0/3425915562 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072400f7e0 con 0x55dfb776ea60 2024-09-16T01:41:19.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.553+0000 7f073ff2ef40 1 -- 172.21.15.88:0/3425915562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dfb776ea60 msgr2=0x55dfb7815250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:19.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.553+0000 7f073ff2ef40 1 --2- 172.21.15.88:0/3425915562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dfb776ea60 0x55dfb7815250 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f0724002de0 tx=0x7f072400b5b0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.553+0000 7f073ff2ef40 1 -- 172.21.15.88:0/3425915562 shutdown_connections 2024-09-16T01:41:19.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.553+0000 7f073ff2ef40 1 --2- 172.21.15.88:0/3425915562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfb7788250 0x55dfb78a84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.553+0000 7f073ff2ef40 1 --2- 172.21.15.88:0/3425915562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dfb776ea60 0x55dfb7815250 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.553+0000 7f073ff2ef40 1 --2- 172.21.15.88:0/3425915562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dfb776dfe0 0x55dfb78079a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.553+0000 7f073ff2ef40 1 -- 172.21.15.88:0/3425915562 >> 172.21.15.88:0/3425915562 conn(0x55dfb7789750 msgr2=0x55dfb776a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:19.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.553+0000 7f073ff2ef40 1 -- 172.21.15.88:0/3425915562 shutdown_connections 2024-09-16T01:41:19.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.553+0000 7f073ff2ef40 1 -- 172.21.15.88:0/3425915562 wait complete. 2024-09-16T01:41:19.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.554+0000 7f073ff2ef40 1 Processor -- start 2024-09-16T01:41:19.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.554+0000 7f073ff2ef40 1 -- start start 2024-09-16T01:41:19.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.555+0000 7f073ff2ef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dfb776dfe0 0x55dfb7809d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.555+0000 7f073ff2ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dfb776ea60 0x55dfb78ac930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.555+0000 7f073ff2ef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfb7788250 0x55dfb7805dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.555+0000 7f073ff2ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55dfb78104b0 con 0x55dfb776ea60 2024-09-16T01:41:19.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.555+0000 7f073ff2ef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dfb776aa60 con 0x55dfb7788250 2024-09-16T01:41:19.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.555+0000 7f073ff2ef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55dfb7815be0 con 0x55dfb776dfe0 2024-09-16T01:41:19.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.555+0000 7f073c7bc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dfb776ea60 0x55dfb78ac930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.555+0000 7f073c7bc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dfb776ea60 0x55dfb78ac930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41686/0 (socket says 172.21.15.88:41686) 2024-09-16T01:41:19.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.555+0000 7f073c7bc640 1 -- 172.21.15.88:0/3278730120 learned_addr learned my addr 172.21.15.88:0/3278730120 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:19.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.555+0000 7f073d7be640 1 --2- 172.21.15.88:0/3278730120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfb7788250 0x55dfb7805dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.555+0000 7f073cfbd640 1 --2- 172.21.15.88:0/3278730120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dfb776dfe0 0x55dfb7809d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.555+0000 7f073c7bc640 1 -- 172.21.15.88:0/3278730120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dfb776dfe0 msgr2=0x55dfb7809d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:19.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.555+0000 7f073c7bc640 1 --2- 172.21.15.88:0/3278730120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dfb776dfe0 0x55dfb7809d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.555+0000 7f073c7bc640 1 -- 172.21.15.88:0/3278730120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfb7788250 msgr2=0x55dfb7805dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:19.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.555+0000 7f073c7bc640 1 --2- 172.21.15.88:0/3278730120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfb7788250 0x55dfb7805dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.555+0000 7f073c7bc640 1 -- 172.21.15.88:0/3278730120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfb780ff10 con 0x55dfb776ea60 2024-09-16T01:41:19.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.555+0000 7f073d7be640 1 --2- 172.21.15.88:0/3278730120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dfb7788250 0x55dfb7805dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:41:19.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.555+0000 7f073c7bc640 1 --2- 172.21.15.88:0/3278730120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dfb776ea60 0x55dfb78ac930 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f0724010000 tx=0x7f0724001fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:19.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.555+0000 7f073a7b8640 1 -- 172.21.15.88:0/3278730120 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07240021d0 con 0x55dfb776ea60 2024-09-16T01:41:19.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.556+0000 7f073a7b8640 1 -- 172.21.15.88:0/3278730120 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0724019480 con 0x55dfb776ea60 2024-09-16T01:41:19.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.556+0000 7f073ff2ef40 1 -- 172.21.15.88:0/3278730120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55dfb7806ec0 con 0x55dfb776ea60 2024-09-16T01:41:19.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.556+0000 7f073a7b8640 1 -- 172.21.15.88:0/3278730120 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07240198b0 con 0x55dfb776ea60 2024-09-16T01:41:19.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.556+0000 7f073ff2ef40 1 -- 172.21.15.88:0/3278730120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfb7807480 con 0x55dfb776ea60 2024-09-16T01:41:19.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.557+0000 7f073a7b8640 1 -- 172.21.15.88:0/3278730120 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f072400e070 con 0x55dfb776ea60 2024-09-16T01:41:19.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.557+0000 7f073a7b8640 1 --2- 172.21.15.88:0/3278730120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f071c06acc0 0x7f071c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.557+0000 7f073a7b8640 1 -- 172.21.15.88:0/3278730120 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6598+0+0 (secure 0 0 0) 0x7f072408be10 con 0x55dfb776ea60 2024-09-16T01:41:19.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.557+0000 7f073cfbd640 1 --2- 172.21.15.88:0/3278730120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f071c06acc0 0x7f071c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.557+0000 7f073ff2ef40 1 --2- 172.21.15.88:0/3278730120 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55dfb7813fd0 0x55dfb780e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.557+0000 7f073ff2ef40 1 -- 172.21.15.88:0/3278730120 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:1 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) -- 0x55dfb7806300 con 0x55dfb7813fd0 2024-09-16T01:41:19.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.558+0000 7f073d7be640 1 --2- 172.21.15.88:0/3278730120 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55dfb7813fd0 0x55dfb780e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.558+0000 7f073cfbd640 1 --2- 172.21.15.88:0/3278730120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f071c06acc0 0x7f071c06d180 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f072c057000 tx=0x7f072c052000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:19.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.558+0000 7f073d7be640 1 --2- 172.21.15.88:0/3278730120 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55dfb7813fd0 0x55dfb780e110 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:19.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.559+0000 7f073d7be640 1 -- 172.21.15.88:0/3278730120 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(1 100.00000000 [delete] v97'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0734014420 con 0x55dfb7813fd0 2024-09-16T01:41:19.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.559+0000 7f073ff2ef40 1 -- 172.21.15.88:0/3278730120 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55dfb7813fd0 msgr2=0x55dfb780e110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:19.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.559+0000 7f073ff2ef40 1 --2- 172.21.15.88:0/3278730120 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55dfb7813fd0 0x55dfb780e110 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.559+0000 7f073ff2ef40 1 -- 172.21.15.88:0/3278730120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f071c06acc0 msgr2=0x7f071c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:19.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.559+0000 7f073ff2ef40 1 --2- 172.21.15.88:0/3278730120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f071c06acc0 0x7f071c06d180 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f072c057000 tx=0x7f072c052000 comp rx=0 tx=0).stop 2024-09-16T01:41:19.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.559+0000 7f073ff2ef40 1 -- 172.21.15.88:0/3278730120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dfb776ea60 msgr2=0x55dfb78ac930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:19.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.559+0000 7f073ff2ef40 1 --2- 172.21.15.88:0/3278730120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dfb776ea60 0x55dfb78ac930 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f0724010000 tx=0x7f0724001fe0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.559+0000 7f073d7be640 1 -- 172.21.15.88:0/3278730120 reap_dead start 2024-09-16T01:41:19.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.559+0000 7f073ff2ef40 1 -- 172.21.15.88:0/3278730120 shutdown_connections 2024-09-16T01:41:19.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.559+0000 7f073ff2ef40 1 -- 172.21.15.88:0/3278730120 >> 172.21.15.88:0/3278730120 conn(0x55dfb7789750 msgr2=0x55dfb7815250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:19.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.559+0000 7f073ff2ef40 1 -- 172.21.15.88:0/3278730120 shutdown_connections 2024-09-16T01:41:19.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.559+0000 7f073ff2ef40 1 -- 172.21.15.88:0/3278730120 wait complete. 2024-09-16T01:41:19.567 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:41:19.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.703+0000 7f7267add640 1 Processor -- start 2024-09-16T01:41:19.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.704+0000 7f7267add640 1 -- start start 2024-09-16T01:41:19.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.704+0000 7f7267add640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f72600b53f0 0x7f72600d2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.704+0000 7f7267add640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72600cdf50 con 0x7f72600d2650 2024-09-16T01:41:19.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.704+0000 7f7267add640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72600ce230 con 0x7f726015c710 2024-09-16T01:41:19.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.704+0000 7f7267add640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f72600ce510 con 0x7f72600b53f0 2024-09-16T01:41:19.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.705+0000 7f726607a640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f72600d2650 0x7f72600ceff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49604/0 (socket says 172.21.15.88:49604) 2024-09-16T01:41:19.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.705+0000 7f726607a640 1 -- 172.21.15.88:0/3149487364 learned_addr learned my addr 172.21.15.88:0/3149487364 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:19.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.705+0000 7f7265078640 1 --2- 172.21.15.88:0/3149487364 >> v2:172.21.15.154:3300/0 conn(0x7f72600b53f0 0x7f72600d2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.705+0000 7f7264877640 1 -- 172.21.15.88:0/3149487364 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1367389074 0 0) 0x7f72600cdf50 con 0x7f72600d2650 2024-09-16T01:41:19.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.705+0000 7f7264877640 1 -- 172.21.15.88:0/3149487364 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7248003520 con 0x7f72600d2650 2024-09-16T01:41:19.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.705+0000 7f7265078640 1 -- 172.21.15.88:0/3149487364 >> v1:172.21.15.154:6789/0 conn(0x7f726015c710 legacy=0x7f7260073970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:19.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.705+0000 7f7265078640 1 -- 172.21.15.88:0/3149487364 >> v1:172.21.15.88:6789/0 conn(0x7f72600d2650 legacy=0x7f72600ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:19.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.705+0000 7f7265078640 1 -- 172.21.15.88:0/3149487364 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72600ce7f0 con 0x7f72600b53f0 2024-09-16T01:41:19.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.706+0000 7f7265078640 1 --2- 172.21.15.88:0/3149487364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72600b53f0 0x7f72600d2110 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f72540097a0 tx=0x7f725402a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7c4673f7f9e1022 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:19.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.706+0000 7f7264877640 1 -- 172.21.15.88:0/3149487364 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7254037070 con 0x7f72600b53f0 2024-09-16T01:41:19.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.706+0000 7f7264877640 1 -- 172.21.15.88:0/3149487364 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7254032440 con 0x7f72600b53f0 2024-09-16T01:41:19.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.706+0000 7f7264877640 1 -- 172.21.15.88:0/3149487364 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72540327f0 con 0x7f72600b53f0 2024-09-16T01:41:19.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.706+0000 7f7267add640 1 -- 172.21.15.88:0/3149487364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72600b53f0 msgr2=0x7f72600d2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:19.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.706+0000 7f7267add640 1 --2- 172.21.15.88:0/3149487364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72600b53f0 0x7f72600d2110 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f72540097a0 tx=0x7f725402a1a0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.706+0000 7f7267add640 1 -- 172.21.15.88:0/3149487364 shutdown_connections 2024-09-16T01:41:19.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.706+0000 7f7267add640 1 --2- 172.21.15.88:0/3149487364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72600b53f0 0x7f72600d2110 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.706+0000 7f7267add640 1 -- 172.21.15.88:0/3149487364 >> 172.21.15.88:0/3149487364 conn(0x7f7260062a20 msgr2=0x7f7260062e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:19.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.706+0000 7f7267add640 1 -- 172.21.15.88:0/3149487364 shutdown_connections 2024-09-16T01:41:19.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.706+0000 7f7267add640 1 -- 172.21.15.88:0/3149487364 wait complete. 2024-09-16T01:41:19.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.708+0000 7f7267add640 1 Processor -- start 2024-09-16T01:41:19.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.708+0000 7f7267add640 1 -- start start 2024-09-16T01:41:19.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.708+0000 7f7267add640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72600b53f0 0x7f72600d1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.708+0000 7f7267add640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72600d2650 0x7f72600d09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.708+0000 7f7267add640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f726015c710 0x7f726006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.708+0000 7f7267add640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7260074080 con 0x7f72600b53f0 2024-09-16T01:41:19.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.708+0000 7f7267add640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f72600cdc00 con 0x7f72600d2650 2024-09-16T01:41:19.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.708+0000 7f7267add640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7260074200 con 0x7f726015c710 2024-09-16T01:41:19.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.708+0000 7f7265879640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72600b53f0 0x7f72600d1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.708+0000 7f7265879640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72600b53f0 0x7f72600d1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41694/0 (socket says 172.21.15.88:41694) 2024-09-16T01:41:19.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.708+0000 7f7265879640 1 -- 172.21.15.88:0/410827498 learned_addr learned my addr 172.21.15.88:0/410827498 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:19.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.709+0000 7f726607a640 1 --2- 172.21.15.88:0/410827498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f726015c710 0x7f726006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.709+0000 7f7265078640 1 --2- 172.21.15.88:0/410827498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72600d2650 0x7f72600d09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.709+0000 7f7265879640 1 -- 172.21.15.88:0/410827498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f726015c710 msgr2=0x7f726006e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:19.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.709+0000 7f7265879640 1 --2- 172.21.15.88:0/410827498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f726015c710 0x7f726006e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.709+0000 7f7265879640 1 -- 172.21.15.88:0/410827498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72600d2650 msgr2=0x7f72600d09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:19.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.709+0000 7f7265879640 1 --2- 172.21.15.88:0/410827498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72600d2650 0x7f72600d09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:19.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.709+0000 7f7265879640 1 -- 172.21.15.88:0/410827498 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72601f4670 con 0x7f72600b53f0 2024-09-16T01:41:19.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.709+0000 7f7265879640 1 --2- 172.21.15.88:0/410827498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72600b53f0 0x7f72600d1480 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f725800c3c0 tx=0x7f725800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:19.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.709+0000 7f725e7fc640 1 -- 172.21.15.88:0/410827498 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72580052d0 con 0x7f72600b53f0 2024-09-16T01:41:19.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.709+0000 7f725e7fc640 1 -- 172.21.15.88:0/410827498 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7258005490 con 0x7f72600b53f0 2024-09-16T01:41:19.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.709+0000 7f725e7fc640 1 -- 172.21.15.88:0/410827498 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7258007260 con 0x7f72600b53f0 2024-09-16T01:41:19.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.709+0000 7f7267add640 1 -- 172.21.15.88:0/410827498 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72601f4950 con 0x7f72600b53f0 2024-09-16T01:41:19.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.709+0000 7f7267add640 1 -- 172.21.15.88:0/410827498 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72601f4f10 con 0x7f72600b53f0 2024-09-16T01:41:19.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.710+0000 7f7267add640 1 -- 172.21.15.88:0/410827498 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72600d2a60 con 0x7f72600b53f0 2024-09-16T01:41:19.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.710+0000 7f725e7fc640 1 -- 172.21.15.88:0/410827498 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7258007400 con 0x7f72600b53f0 2024-09-16T01:41:19.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.711+0000 7f725e7fc640 1 --2- 172.21.15.88:0/410827498 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f724c06c2b0 0x7f724c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:19.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.711+0000 7f725e7fc640 1 -- 172.21.15.88:0/410827498 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6598+0+0 (secure 0 0 0) 0x7f725808ab50 con 0x7f72600b53f0 2024-09-16T01:41:19.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.711+0000 7f7265078640 1 --2- 172.21.15.88:0/410827498 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f724c06c2b0 0x7f724c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:19.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.711+0000 7f7265078640 1 --2- 172.21.15.88:0/410827498 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f724c06c2b0 0x7f724c06e770 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f7254034000 tx=0x7f725402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:19.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.715+0000 7f725e7fc640 1 -- 172.21.15.88:0/410827498 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7258058980 con 0x7f72600b53f0 2024-09-16T01:41:19.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:19.914+0000 7f7267add640 1 -- 172.21.15.88:0/410827498 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f72600d0ee0 con 0x7f72600b53f0 2024-09-16T01:41:20.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.224+0000 7f725e7fc640 1 -- 172.21.15.88:0/410827498 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v131) ==== 162+0+0 (secure 0 0 0) 0x7f725805cc80 con 0x7f72600b53f0 2024-09-16T01:41:20.227 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:41:20.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.227+0000 7f7267add640 1 -- 172.21.15.88:0/410827498 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f724c06c2b0 msgr2=0x7f724c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:20.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.227+0000 7f7267add640 1 --2- 172.21.15.88:0/410827498 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f724c06c2b0 0x7f724c06e770 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f7254034000 tx=0x7f725402e000 comp rx=0 tx=0).stop 2024-09-16T01:41:20.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.227+0000 7f7267add640 1 -- 172.21.15.88:0/410827498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72600b53f0 msgr2=0x7f72600d1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:20.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.227+0000 7f7267add640 1 --2- 172.21.15.88:0/410827498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72600b53f0 0x7f72600d1480 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f725800c3c0 tx=0x7f725800c880 comp rx=0 tx=0).stop 2024-09-16T01:41:20.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.227+0000 7f7267add640 1 -- 172.21.15.88:0/410827498 shutdown_connections 2024-09-16T01:41:20.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.227+0000 7f7267add640 1 --2- 172.21.15.88:0/410827498 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f724c06c2b0 0x7f724c06e770 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:20.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.227+0000 7f7267add640 1 --2- 172.21.15.88:0/410827498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f726015c710 0x7f726006e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:20.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.227+0000 7f7267add640 1 --2- 172.21.15.88:0/410827498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72600d2650 0x7f72600d09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:20.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.227+0000 7f7267add640 1 --2- 172.21.15.88:0/410827498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72600b53f0 0x7f72600d1480 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:20.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.227+0000 7f7267add640 1 -- 172.21.15.88:0/410827498 >> 172.21.15.88:0/410827498 conn(0x7f7260062a20 msgr2=0x7f726015ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:20.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.227+0000 7f7267add640 1 -- 172.21.15.88:0/410827498 shutdown_connections 2024-09-16T01:41:20.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.227+0000 7f7267add640 1 -- 172.21.15.88:0/410827498 wait complete. 2024-09-16T01:41:20.242 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:20.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.338+0000 7f2022d31640 1 Processor -- start 2024-09-16T01:41:20.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.339+0000 7f2022d31640 1 -- start start 2024-09-16T01:41:20.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.339+0000 7f2022d31640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f201c15e660 0x7f201c0cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:20.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.339+0000 7f2022d31640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f201c0d3310 0x7f201c0d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:20.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.339+0000 7f2022d31640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f201c0d1650 0x7f201c06bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:20.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.339+0000 7f2022d31640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f201c0d1ac0 con 0x7f201c0d1650 2024-09-16T01:41:20.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.339+0000 7f2022d31640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f201c15ea40 con 0x7f201c15e660 2024-09-16T01:41:20.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.339+0000 7f2022d31640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f201c0d3870 con 0x7f201c0d3310 2024-09-16T01:41:20.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.339+0000 7f20212ce640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f201c0d1650 0x7f201c06bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:20.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.340+0000 7f20212ce640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f201c0d1650 0x7f201c06bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60284/0 (socket says 172.21.15.88:60284) 2024-09-16T01:41:20.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.340+0000 7f20212ce640 1 -- 172.21.15.88:0/994022243 learned_addr learned my addr 172.21.15.88:0/994022243 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:20.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.340+0000 7f2020acd640 1 --2- 172.21.15.88:0/994022243 >> v2:172.21.15.154:3300/0 conn(0x7f201c15e660 0x7f201c0cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:20.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.340+0000 7f201bfff640 1 --2- 172.21.15.88:0/994022243 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f201c0d3310 0x7f201c0d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:20.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.340+0000 7f20212ce640 1 -- 172.21.15.88:0/994022243 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f201c0d3310 msgr2=0x7f201c0d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:20.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.340+0000 7f20212ce640 1 --2- 172.21.15.88:0/994022243 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f201c0d3310 0x7f201c0d1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:20.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.340+0000 7f20212ce640 1 -- 172.21.15.88:0/994022243 >> v2:172.21.15.154:3300/0 conn(0x7f201c15e660 msgr2=0x7f201c0cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:20.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.340+0000 7f20212ce640 1 --2- 172.21.15.88:0/994022243 >> v2:172.21.15.154:3300/0 conn(0x7f201c15e660 0x7f201c0cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:20.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.340+0000 7f20212ce640 1 -- 172.21.15.88:0/994022243 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f201c0d3b50 con 0x7f201c0d1650 2024-09-16T01:41:20.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.341+0000 7f20212ce640 1 --2- 172.21.15.88:0/994022243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f201c0d1650 0x7f201c06bfd0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f201c0d10e0 tx=0x7f201402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bab35723bd87fa3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:20.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.341+0000 7f201b7fe640 1 -- 172.21.15.88:0/994022243 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2014037070 con 0x7f201c0d1650 2024-09-16T01:41:20.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.341+0000 7f201b7fe640 1 -- 172.21.15.88:0/994022243 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2014032440 con 0x7f201c0d1650 2024-09-16T01:41:20.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.341+0000 7f201b7fe640 1 -- 172.21.15.88:0/994022243 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2014032810 con 0x7f201c0d1650 2024-09-16T01:41:20.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.341+0000 7f2022d31640 1 -- 172.21.15.88:0/994022243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f201c0d1650 msgr2=0x7f201c06bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:20.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.341+0000 7f2022d31640 1 --2- 172.21.15.88:0/994022243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f201c0d1650 0x7f201c06bfd0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f201c0d10e0 tx=0x7f201402a1e0 comp rx=0 tx=0).stop 2024-09-16T01:41:20.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.341+0000 7f2022d31640 1 -- 172.21.15.88:0/994022243 shutdown_connections 2024-09-16T01:41:20.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.341+0000 7f2022d31640 1 --2- 172.21.15.88:0/994022243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f201c0d1650 0x7f201c06bfd0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:20.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.341+0000 7f2022d31640 1 --2- 172.21.15.88:0/994022243 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f201c0d3310 0x7f201c0d1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:20.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.341+0000 7f2022d31640 1 --2- 172.21.15.88:0/994022243 >> v2:172.21.15.154:3300/0 conn(0x7f201c15e660 0x7f201c0cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:20.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.341+0000 7f2022d31640 1 -- 172.21.15.88:0/994022243 >> 172.21.15.88:0/994022243 conn(0x7f201c0b6410 msgr2=0x7f201c0b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:20.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.341+0000 7f2022d31640 1 -- 172.21.15.88:0/994022243 shutdown_connections 2024-09-16T01:41:20.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.341+0000 7f2022d31640 1 -- 172.21.15.88:0/994022243 wait complete. 2024-09-16T01:41:20.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.342+0000 7f2022d31640 1 Processor -- start 2024-09-16T01:41:20.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.342+0000 7f2022d31640 1 -- start start 2024-09-16T01:41:20.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.343+0000 7f2022d31640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f201c0d1650 0x7f201c160bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:20.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.343+0000 7f2022d31640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f201c0d3310 0x7f201c150910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:20.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.343+0000 7f2022d31640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f201c15e660 0x7f201c0b5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:20.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.343+0000 7f2022d31640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f201c0d43f0 con 0x7f201c0d3310 2024-09-16T01:41:20.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.343+0000 7f2022d31640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f201c0d4110 con 0x7f201c0d1650 2024-09-16T01:41:20.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.343+0000 7f2022d31640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f201c0cfc70 con 0x7f201c15e660 2024-09-16T01:41:20.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.343+0000 7f201bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f201c0d3310 0x7f201c150910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:20.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.343+0000 7f201bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f201c0d3310 0x7f201c150910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60300/0 (socket says 172.21.15.88:60300) 2024-09-16T01:41:20.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.343+0000 7f201bfff640 1 -- 172.21.15.88:0/3462967933 learned_addr learned my addr 172.21.15.88:0/3462967933 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:20.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.344+0000 7f2020acd640 1 --2- 172.21.15.88:0/3462967933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f201c0d1650 0x7f201c160bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:20.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.344+0000 7f20212ce640 1 --2- 172.21.15.88:0/3462967933 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f201c15e660 0x7f201c0b5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:20.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.344+0000 7f201bfff640 1 -- 172.21.15.88:0/3462967933 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f201c15e660 msgr2=0x7f201c0b5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:20.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.344+0000 7f201bfff640 1 --2- 172.21.15.88:0/3462967933 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f201c15e660 0x7f201c0b5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:20.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.344+0000 7f201bfff640 1 -- 172.21.15.88:0/3462967933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f201c0d1650 msgr2=0x7f201c160bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:20.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.344+0000 7f201bfff640 1 --2- 172.21.15.88:0/3462967933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f201c0d1650 0x7f201c160bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:20.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.344+0000 7f201bfff640 1 -- 172.21.15.88:0/3462967933 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f201c15fb10 con 0x7f201c0d3310 2024-09-16T01:41:20.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.344+0000 7f201bfff640 1 --2- 172.21.15.88:0/3462967933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f201c0d3310 0x7f201c150910 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f200400e8a0 tx=0x7f200400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:20.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.344+0000 7f20197fa640 1 -- 172.21.15.88:0/3462967933 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2004017070 con 0x7f201c0d3310 2024-09-16T01:41:20.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.344+0000 7f20197fa640 1 -- 172.21.15.88:0/3462967933 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f200400ee90 con 0x7f201c0d3310 2024-09-16T01:41:20.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.344+0000 7f2022d31640 1 -- 172.21.15.88:0/3462967933 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f201c15fdf0 con 0x7f201c0d3310 2024-09-16T01:41:20.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.344+0000 7f20197fa640 1 -- 172.21.15.88:0/3462967933 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200400cb80 con 0x7f201c0d3310 2024-09-16T01:41:20.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.344+0000 7f2022d31640 1 -- 172.21.15.88:0/3462967933 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f201c1603b0 con 0x7f201c0d3310 2024-09-16T01:41:20.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.345+0000 7f2022d31640 1 -- 172.21.15.88:0/3462967933 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f201c0d1110 con 0x7f201c0d3310 2024-09-16T01:41:20.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.346+0000 7f20197fa640 1 -- 172.21.15.88:0/3462967933 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f200401a070 con 0x7f201c0d3310 2024-09-16T01:41:20.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.346+0000 7f20197fa640 1 --2- 172.21.15.88:0/3462967933 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ffc06c2b0 0x7f1ffc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:20.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.346+0000 7f20197fa640 1 -- 172.21.15.88:0/3462967933 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6598+0+0 (secure 0 0 0) 0x7f200408b5b0 con 0x7f201c0d3310 2024-09-16T01:41:20.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.346+0000 7f2020acd640 1 --2- 172.21.15.88:0/3462967933 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ffc06c2b0 0x7f1ffc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:20.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.346+0000 7f2020acd640 1 --2- 172.21.15.88:0/3462967933 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ffc06c2b0 0x7f1ffc06e770 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f201c1f77d0 tx=0x7f200c00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:20.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.350+0000 7f20197fa640 1 -- 172.21.15.88:0/3462967933 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2004059350 con 0x7f201c0d3310 2024-09-16T01:41:20.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.549+0000 7f2022d31640 1 -- 172.21.15.88:0/3462967933 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f201c161a90 con 0x7f201c0d3310 2024-09-16T01:41:20.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.550+0000 7f20197fa640 1 -- 172.21.15.88:0/3462967933 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) ==== 95+0+4400 (secure 0 0 0) 0x7f200405d650 con 0x7f201c0d3310 2024-09-16T01:41:20.552 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:20.552 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":132,"btime":"2024-09-16T01:41:20:228775+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:20.228761+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8121},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8121":{"gid":8121,"name":"a","rank":0,"incarnation":132,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6836/1162318664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1162318664},{"type":"v1","addr":"172.21.15.88:6836","nonce":1162318664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:41:20.553 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 132 2024-09-16T01:41:20.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.552+0000 7f2022d31640 1 -- 172.21.15.88:0/3462967933 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ffc06c2b0 msgr2=0x7f1ffc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:20.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.552+0000 7f2022d31640 1 --2- 172.21.15.88:0/3462967933 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ffc06c2b0 0x7f1ffc06e770 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f201c1f77d0 tx=0x7f200c00b040 comp rx=0 tx=0).stop 2024-09-16T01:41:20.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.552+0000 7f2022d31640 1 -- 172.21.15.88:0/3462967933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f201c0d3310 msgr2=0x7f201c150910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:20.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.552+0000 7f2022d31640 1 --2- 172.21.15.88:0/3462967933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f201c0d3310 0x7f201c150910 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f200400e8a0 tx=0x7f200400ed60 comp rx=0 tx=0).stop 2024-09-16T01:41:20.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.552+0000 7f2022d31640 1 -- 172.21.15.88:0/3462967933 shutdown_connections 2024-09-16T01:41:20.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.552+0000 7f2022d31640 1 --2- 172.21.15.88:0/3462967933 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ffc06c2b0 0x7f1ffc06e770 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:20.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.552+0000 7f2022d31640 1 --2- 172.21.15.88:0/3462967933 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f201c15e660 0x7f201c0b5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:20.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.552+0000 7f2022d31640 1 --2- 172.21.15.88:0/3462967933 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f201c0d3310 0x7f201c150910 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:20.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.552+0000 7f2022d31640 1 --2- 172.21.15.88:0/3462967933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f201c0d1650 0x7f201c160bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:20.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.552+0000 7f2022d31640 1 -- 172.21.15.88:0/3462967933 >> 172.21.15.88:0/3462967933 conn(0x7f201c0b6410 msgr2=0x7f201c1597e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:20.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.552+0000 7f2022d31640 1 -- 172.21.15.88:0/3462967933 shutdown_connections 2024-09-16T01:41:20.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:20.552+0000 7f2022d31640 1 -- 172.21.15.88:0/3462967933 wait complete. 2024-09-16T01:41:20.565 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-09-16T01:41:20.955 INFO:tasks.ceph.mds.c.smithi088.stderr: -51> 2024-09-16T01:41:16.202+0000 7f383e4a3640 -1 mds.0.inotable: load_2 could not read table: -2 2024-09-16T01:41:20.955 INFO:tasks.ceph.mds.c.smithi088.stderr: -50> 2024-09-16T01:41:16.202+0000 7f383e4a3640 -1 log_channel(cluster) log [ERR] : error reading table object 'mds0_inotable' -2 ((2) No such file or directory) 2024-09-16T01:41:20.979 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:41:23.256 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:41:23.254+0000 7ff92f416640 -1 log_channel(cluster) log [ERR] : dir 0x100 object missing on disk; some files may be lost (~mds0) 2024-09-16T01:41:23.258 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:41:23.256+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:41:23.258 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:41:23.256+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:41:25.566 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:25.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.676+0000 7f6ab759e640 1 Processor -- start 2024-09-16T01:41:25.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.676+0000 7f6ab759e640 1 -- start start 2024-09-16T01:41:25.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.677+0000 7f6ab759e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6ab815e750 0x7f6ab80b6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:25.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.677+0000 7f6ab759e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ab8070660 0x7f6ab80b6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:25.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.677+0000 7f6ab759e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6ab80b73b0 0x7f6ab81620b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:25.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.677+0000 7f6ab759e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6ab8072f40 con 0x7f6ab815e750 2024-09-16T01:41:25.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.677+0000 7f6ab759e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6ab8073220 con 0x7f6ab80b73b0 2024-09-16T01:41:25.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.677+0000 7f6ab759e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6ab8073500 con 0x7f6ab8070660 2024-09-16T01:41:25.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.677+0000 7f6ab659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6ab815e750 0x7f6ab80b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:25.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.677+0000 7f6ab659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6ab815e750 0x7f6ab80b6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60324/0 (socket says 172.21.15.88:60324) 2024-09-16T01:41:25.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.677+0000 7f6ab659c640 1 -- 172.21.15.88:0/2882790071 learned_addr learned my addr 172.21.15.88:0/2882790071 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:25.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.677+0000 7f6ab6d9d640 1 --2- 172.21.15.88:0/2882790071 >> v2:172.21.15.154:3300/0 conn(0x7f6ab80b73b0 0x7f6ab81620b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:25.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.677+0000 7f6ab5d9b640 1 --2- 172.21.15.88:0/2882790071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ab8070660 0x7f6ab80b6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:25.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.677+0000 7f6ab659c640 1 -- 172.21.15.88:0/2882790071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ab8070660 msgr2=0x7f6ab80b6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:25.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.677+0000 7f6ab659c640 1 --2- 172.21.15.88:0/2882790071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ab8070660 0x7f6ab80b6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:25.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.677+0000 7f6ab659c640 1 -- 172.21.15.88:0/2882790071 >> v2:172.21.15.154:3300/0 conn(0x7f6ab80b73b0 msgr2=0x7f6ab81620b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:25.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.677+0000 7f6ab659c640 1 --2- 172.21.15.88:0/2882790071 >> v2:172.21.15.154:3300/0 conn(0x7f6ab80b73b0 0x7f6ab81620b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:25.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.677+0000 7f6ab659c640 1 -- 172.21.15.88:0/2882790071 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ab80737e0 con 0x7f6ab815e750 2024-09-16T01:41:25.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.677+0000 7f6ab6d9d640 1 --2- 172.21.15.88:0/2882790071 >> v2:172.21.15.154:3300/0 conn(0x7f6ab80b73b0 0x7f6ab81620b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:25.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.677+0000 7f6ab5d9b640 1 --2- 172.21.15.88:0/2882790071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ab8070660 0x7f6ab80b6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:25.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.678+0000 7f6ab659c640 1 --2- 172.21.15.88:0/2882790071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ab815e750 0x7f6ab80b6930 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8072ed0 tx=0x7f6aa802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f72461df0eaf6c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:25.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.678+0000 7f6ab559a640 1 -- 172.21.15.88:0/2882790071 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa8037070 con 0x7f6ab815e750 2024-09-16T01:41:25.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.678+0000 7f6ab559a640 1 -- 172.21.15.88:0/2882790071 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6aa8032440 con 0x7f6ab815e750 2024-09-16T01:41:25.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.678+0000 7f6ab559a640 1 -- 172.21.15.88:0/2882790071 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa8032810 con 0x7f6ab815e750 2024-09-16T01:41:25.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.679+0000 7f6ab759e640 1 -- 172.21.15.88:0/2882790071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ab815e750 msgr2=0x7f6ab80b6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:25.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.679+0000 7f6ab759e640 1 --2- 172.21.15.88:0/2882790071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ab815e750 0x7f6ab80b6930 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8072ed0 tx=0x7f6aa802a1e0 comp rx=0 tx=0).stop 2024-09-16T01:41:25.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.679+0000 7f6ab759e640 1 -- 172.21.15.88:0/2882790071 shutdown_connections 2024-09-16T01:41:25.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.679+0000 7f6ab759e640 1 --2- 172.21.15.88:0/2882790071 >> v2:172.21.15.154:3300/0 conn(0x7f6ab80b73b0 0x7f6ab81620b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:25.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.679+0000 7f6ab759e640 1 --2- 172.21.15.88:0/2882790071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ab8070660 0x7f6ab80b6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:25.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.679+0000 7f6ab759e640 1 --2- 172.21.15.88:0/2882790071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ab815e750 0x7f6ab80b6930 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:25.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.679+0000 7f6ab759e640 1 -- 172.21.15.88:0/2882790071 >> 172.21.15.88:0/2882790071 conn(0x7f6ab80d2300 msgr2=0x7f6ab80d26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:25.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.679+0000 7f6ab759e640 1 -- 172.21.15.88:0/2882790071 shutdown_connections 2024-09-16T01:41:25.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.679+0000 7f6ab759e640 1 -- 172.21.15.88:0/2882790071 wait complete. 2024-09-16T01:41:25.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.680+0000 7f6ab759e640 1 Processor -- start 2024-09-16T01:41:25.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.680+0000 7f6ab759e640 1 -- start start 2024-09-16T01:41:25.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.681+0000 7f6ab759e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ab8070660 0x7f6ab815b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:25.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.681+0000 7f6ab759e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ab80b73b0 0x7f6ab8158e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:25.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.681+0000 7f6ab759e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ab80b32e0 0x7f6ab80b3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:25.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.681+0000 7f6ab759e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab80b5310 con 0x7f6ab80b32e0 2024-09-16T01:41:25.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.681+0000 7f6ab759e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab8073da0 con 0x7f6ab8070660 2024-09-16T01:41:25.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.681+0000 7f6ab759e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6ab80b5490 con 0x7f6ab80b73b0 2024-09-16T01:41:25.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.681+0000 7f6ab6d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ab80b32e0 0x7f6ab80b3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:25.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.681+0000 7f6ab6d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ab80b32e0 0x7f6ab80b3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60330/0 (socket says 172.21.15.88:60330) 2024-09-16T01:41:25.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.681+0000 7f6ab6d9d640 1 -- 172.21.15.88:0/181500779 learned_addr learned my addr 172.21.15.88:0/181500779 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:25.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.681+0000 7f6ab659c640 1 --2- 172.21.15.88:0/181500779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ab8070660 0x7f6ab815b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:25.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.681+0000 7f6ab5d9b640 1 --2- 172.21.15.88:0/181500779 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ab80b73b0 0x7f6ab8158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:25.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.682+0000 7f6ab6d9d640 1 -- 172.21.15.88:0/181500779 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ab80b73b0 msgr2=0x7f6ab8158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:25.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.682+0000 7f6ab6d9d640 1 --2- 172.21.15.88:0/181500779 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ab80b73b0 0x7f6ab8158e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:25.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.682+0000 7f6ab6d9d640 1 -- 172.21.15.88:0/181500779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ab8070660 msgr2=0x7f6ab815b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:25.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.682+0000 7f6ab6d9d640 1 --2- 172.21.15.88:0/181500779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ab8070660 0x7f6ab815b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:25.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.682+0000 7f6ab6d9d640 1 -- 172.21.15.88:0/181500779 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ab8159340 con 0x7f6ab80b32e0 2024-09-16T01:41:25.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.682+0000 7f6ab6d9d640 1 --2- 172.21.15.88:0/181500779 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ab80b32e0 0x7f6ab80b3770 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000ceb0 tx=0x7f6ab000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:25.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.682+0000 7f6aa6ffd640 1 -- 172.21.15.88:0/181500779 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab001a070 con 0x7f6ab80b32e0 2024-09-16T01:41:25.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.682+0000 7f6aa6ffd640 1 -- 172.21.15.88:0/181500779 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ab000d840 con 0x7f6ab80b32e0 2024-09-16T01:41:25.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.682+0000 7f6ab759e640 1 -- 172.21.15.88:0/181500779 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ab80b1a00 con 0x7f6ab80b32e0 2024-09-16T01:41:25.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.682+0000 7f6aa6ffd640 1 -- 172.21.15.88:0/181500779 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab000dc30 con 0x7f6ab80b32e0 2024-09-16T01:41:25.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.682+0000 7f6ab759e640 1 -- 172.21.15.88:0/181500779 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ab80ae940 con 0x7f6ab80b32e0 2024-09-16T01:41:25.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.683+0000 7f6ab759e640 1 -- 172.21.15.88:0/181500779 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ab80b6930 con 0x7f6ab80b32e0 2024-09-16T01:41:25.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.683+0000 7f6aa6ffd640 1 -- 172.21.15.88:0/181500779 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6ab001d070 con 0x7f6ab80b32e0 2024-09-16T01:41:25.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.684+0000 7f6aa6ffd640 1 --2- 172.21.15.88:0/181500779 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6a9806c2b0 0x7f6a9806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:25.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.684+0000 7f6aa6ffd640 1 -- 172.21.15.88:0/181500779 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 6684+0+0 (secure 0 0 0) 0x7f6ab008bf30 con 0x7f6ab80b32e0 2024-09-16T01:41:25.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.684+0000 7f6ab659c640 1 --2- 172.21.15.88:0/181500779 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6a9806c2b0 0x7f6a9806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:25.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.684+0000 7f6ab659c640 1 --2- 172.21.15.88:0/181500779 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6a9806c2b0 0x7f6a9806e770 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8034000 tx=0x7f6aa802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:25.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.688+0000 7f6aa6ffd640 1 -- 172.21.15.88:0/181500779 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6ab0059d10 con 0x7f6ab80b32e0 2024-09-16T01:41:25.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.888+0000 7f6ab759e640 1 -- 172.21.15.88:0/181500779 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ab8162970 con 0x7f6ab80b32e0 2024-09-16T01:41:25.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.889+0000 7f6aa6ffd640 1 -- 172.21.15.88:0/181500779 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) ==== 95+0+4391 (secure 0 0 0) 0x7f6ab005e010 con 0x7f6ab80b32e0 2024-09-16T01:41:25.890 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:25.890 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":136,"btime":"2024-09-16T01:41:23:258090+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8175,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1255135754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1255135754},{"type":"v1","addr":"172.21.15.88:6837","nonce":1255135754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:23.258086+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:41:25.890 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 136 2024-09-16T01:41:25.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.891+0000 7f6ab759e640 1 -- 172.21.15.88:0/181500779 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6a9806c2b0 msgr2=0x7f6a9806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:25.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.891+0000 7f6ab759e640 1 --2- 172.21.15.88:0/181500779 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6a9806c2b0 0x7f6a9806e770 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8034000 tx=0x7f6aa802e000 comp rx=0 tx=0).stop 2024-09-16T01:41:25.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.891+0000 7f6ab759e640 1 -- 172.21.15.88:0/181500779 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ab80b32e0 msgr2=0x7f6ab80b3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:25.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.891+0000 7f6ab759e640 1 --2- 172.21.15.88:0/181500779 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ab80b32e0 0x7f6ab80b3770 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000ceb0 tx=0x7f6ab000a710 comp rx=0 tx=0).stop 2024-09-16T01:41:25.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.891+0000 7f6ab759e640 1 -- 172.21.15.88:0/181500779 shutdown_connections 2024-09-16T01:41:25.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.891+0000 7f6ab759e640 1 --2- 172.21.15.88:0/181500779 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6a9806c2b0 0x7f6a9806e770 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:25.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.891+0000 7f6ab759e640 1 --2- 172.21.15.88:0/181500779 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6ab80b32e0 0x7f6ab80b3770 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:25.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.891+0000 7f6ab759e640 1 --2- 172.21.15.88:0/181500779 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ab80b73b0 0x7f6ab8158e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:25.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.891+0000 7f6ab759e640 1 --2- 172.21.15.88:0/181500779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ab8070660 0x7f6ab815b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:25.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.891+0000 7f6ab759e640 1 -- 172.21.15.88:0/181500779 >> 172.21.15.88:0/181500779 conn(0x7f6ab80d2300 msgr2=0x7f6ab8070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:25.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.891+0000 7f6ab759e640 1 -- 172.21.15.88:0/181500779 shutdown_connections 2024-09-16T01:41:25.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:25.891+0000 7f6ab759e640 1 -- 172.21.15.88:0/181500779 wait complete. 2024-09-16T01:41:25.903 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-09-16T01:41:25.903 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:26.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.000+0000 7f5e298dd640 1 Processor -- start 2024-09-16T01:41:26.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.000+0000 7f5e298dd640 1 -- start start 2024-09-16T01:41:26.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.001+0000 7f5e298dd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5e2415e660 0x7f5e240cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:26.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.001+0000 7f5e298dd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5e240d1820 0x7f5e240b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:26.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.001+0000 7f5e298dd640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e2415ea40 con 0x7f5e240d3310 2024-09-16T01:41:26.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.001+0000 7f5e298dd640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f5e240d3870 con 0x7f5e2415e660 2024-09-16T01:41:26.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.001+0000 7f5e298dd640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5e240d3b50 con 0x7f5e240d1820 2024-09-16T01:41:26.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.001+0000 7f5e22ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5e2415e660 0x7f5e240cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:26.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.001+0000 7f5e22ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5e2415e660 0x7f5e240cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60340/0 (socket says 172.21.15.88:60340) 2024-09-16T01:41:26.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.001+0000 7f5e22ffd640 1 -- 172.21.15.88:0/3875518254 learned_addr learned my addr 172.21.15.88:0/3875518254 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:26.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.001+0000 7f5e237fe640 1 --2- 172.21.15.88:0/3875518254 >> v2:172.21.15.154:3300/0 conn(0x7f5e240d1820 0x7f5e240b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:26.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.001+0000 7f5e22ffd640 1 -- 172.21.15.88:0/3875518254 >> v2:172.21.15.154:3300/0 conn(0x7f5e240d1820 msgr2=0x7f5e240b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:26.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.001+0000 7f5e22ffd640 1 --2- 172.21.15.88:0/3875518254 >> v2:172.21.15.154:3300/0 conn(0x7f5e240d1820 0x7f5e240b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:26.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.001+0000 7f5e22ffd640 1 -- 172.21.15.88:0/3875518254 >> v1:172.21.15.154:6789/0 conn(0x7f5e240d3310 legacy=0x7f5e240d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:26.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.001+0000 7f5e22ffd640 1 -- 172.21.15.88:0/3875518254 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e240d3e30 con 0x7f5e2415e660 2024-09-16T01:41:26.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.001+0000 7f5e237fe640 1 --2- 172.21.15.88:0/3875518254 >> v2:172.21.15.154:3300/0 conn(0x7f5e240d1820 0x7f5e240b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:26.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.002+0000 7f5e22ffd640 1 --2- 172.21.15.88:0/3875518254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e2415e660 0x7f5e240cf730 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f5e240d26b0 tx=0x7f5e1402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc4af60f09983fca server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:26.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.002+0000 7f5e21ffb640 1 -- 172.21.15.88:0/3875518254 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e14037070 con 0x7f5e2415e660 2024-09-16T01:41:26.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.002+0000 7f5e21ffb640 1 -- 172.21.15.88:0/3875518254 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e14032440 con 0x7f5e2415e660 2024-09-16T01:41:26.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.002+0000 7f5e21ffb640 1 -- 172.21.15.88:0/3875518254 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e14032810 con 0x7f5e2415e660 2024-09-16T01:41:26.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.003+0000 7f5e298dd640 1 -- 172.21.15.88:0/3875518254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e2415e660 msgr2=0x7f5e240cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:26.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.003+0000 7f5e298dd640 1 --2- 172.21.15.88:0/3875518254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e2415e660 0x7f5e240cf730 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f5e240d26b0 tx=0x7f5e1402a270 comp rx=0 tx=0).stop 2024-09-16T01:41:26.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.003+0000 7f5e298dd640 1 -- 172.21.15.88:0/3875518254 shutdown_connections 2024-09-16T01:41:26.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.003+0000 7f5e298dd640 1 --2- 172.21.15.88:0/3875518254 >> v2:172.21.15.154:3300/0 conn(0x7f5e240d1820 0x7f5e240b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:26.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.003+0000 7f5e298dd640 1 --2- 172.21.15.88:0/3875518254 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e2415e660 0x7f5e240cf730 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:26.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.003+0000 7f5e298dd640 1 -- 172.21.15.88:0/3875518254 >> 172.21.15.88:0/3875518254 conn(0x7f5e240b6410 msgr2=0x7f5e240b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:26.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.003+0000 7f5e298dd640 1 -- 172.21.15.88:0/3875518254 shutdown_connections 2024-09-16T01:41:26.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.003+0000 7f5e298dd640 1 -- 172.21.15.88:0/3875518254 wait complete. 2024-09-16T01:41:26.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.004+0000 7f5e298dd640 1 Processor -- start 2024-09-16T01:41:26.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.004+0000 7f5e298dd640 1 -- start start 2024-09-16T01:41:26.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.005+0000 7f5e298dd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e240d1820 0x7f5e24161190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:26.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.005+0000 7f5e298dd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e240d3310 0x7f5e24151040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:26.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.005+0000 7f5e298dd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e2415e660 0x7f5e240b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:26.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.005+0000 7f5e298dd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5e240cfc70 con 0x7f5e240d3310 2024-09-16T01:41:26.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.005+0000 7f5e298dd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5e240d43f0 con 0x7f5e240d1820 2024-09-16T01:41:26.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.005+0000 7f5e298dd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5e240b80b0 con 0x7f5e2415e660 2024-09-16T01:41:26.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.005+0000 7f5e227fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e240d3310 0x7f5e24151040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:26.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.005+0000 7f5e227fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e240d3310 0x7f5e24151040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60352/0 (socket says 172.21.15.88:60352) 2024-09-16T01:41:26.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.005+0000 7f5e227fc640 1 -- 172.21.15.88:0/3644634870 learned_addr learned my addr 172.21.15.88:0/3644634870 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:26.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.005+0000 7f5e22ffd640 1 --2- 172.21.15.88:0/3644634870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e240d1820 0x7f5e24161190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:26.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.005+0000 7f5e237fe640 1 --2- 172.21.15.88:0/3644634870 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e2415e660 0x7f5e240b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:26.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.005+0000 7f5e227fc640 1 -- 172.21.15.88:0/3644634870 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e2415e660 msgr2=0x7f5e240b48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:26.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.005+0000 7f5e227fc640 1 --2- 172.21.15.88:0/3644634870 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e2415e660 0x7f5e240b48f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:26.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.005+0000 7f5e227fc640 1 -- 172.21.15.88:0/3644634870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e240d1820 msgr2=0x7f5e24161190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:26.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.005+0000 7f5e227fc640 1 --2- 172.21.15.88:0/3644634870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e240d1820 0x7f5e24161190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:26.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.005+0000 7f5e227fc640 1 -- 172.21.15.88:0/3644634870 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e2415ed90 con 0x7f5e240d3310 2024-09-16T01:41:26.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.006+0000 7f5e227fc640 1 --2- 172.21.15.88:0/3644634870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e240d3310 0x7f5e24151040 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f5e1800c400 tx=0x7f5e1800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:26.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.006+0000 7f5e0bfff640 1 -- 172.21.15.88:0/3644634870 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e180052d0 con 0x7f5e240d3310 2024-09-16T01:41:26.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.006+0000 7f5e0bfff640 1 -- 172.21.15.88:0/3644634870 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e18005490 con 0x7f5e240d3310 2024-09-16T01:41:26.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.006+0000 7f5e298dd640 1 -- 172.21.15.88:0/3644634870 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e2415f070 con 0x7f5e240d3310 2024-09-16T01:41:26.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.006+0000 7f5e0bfff640 1 -- 172.21.15.88:0/3644634870 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e18007260 con 0x7f5e240d3310 2024-09-16T01:41:26.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.006+0000 7f5e298dd640 1 -- 172.21.15.88:0/3644634870 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e2415f630 con 0x7f5e240d3310 2024-09-16T01:41:26.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.007+0000 7f5e298dd640 1 -- 172.21.15.88:0/3644634870 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e240b7b70 con 0x7f5e240d3310 2024-09-16T01:41:26.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.007+0000 7f5e0bfff640 1 -- 172.21.15.88:0/3644634870 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5e18007400 con 0x7f5e240d3310 2024-09-16T01:41:26.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.008+0000 7f5e0bfff640 1 --2- 172.21.15.88:0/3644634870 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e1006c2b0 0x7f5e1006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:26.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.008+0000 7f5e0bfff640 1 -- 172.21.15.88:0/3644634870 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 6684+0+0 (secure 0 0 0) 0x7f5e1808acb0 con 0x7f5e240d3310 2024-09-16T01:41:26.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.008+0000 7f5e22ffd640 1 --2- 172.21.15.88:0/3644634870 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e1006c2b0 0x7f5e1006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:26.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.008+0000 7f5e22ffd640 1 --2- 172.21.15.88:0/3644634870 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e1006c2b0 0x7f5e1006e770 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f5e14034000 tx=0x7f5e1402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:26.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.012+0000 7f5e0bfff640 1 -- 172.21.15.88:0/3644634870 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5e180590e0 con 0x7f5e240d3310 2024-09-16T01:41:26.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.212+0000 7f5e298dd640 1 -- 172.21.15.88:0/3644634870 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5e240ba620 con 0x7f5e240d3310 2024-09-16T01:41:26.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.213+0000 7f5e0bfff640 1 -- 172.21.15.88:0/3644634870 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) ==== 95+0+4391 (secure 0 0 0) 0x7f5e180589e0 con 0x7f5e240d3310 2024-09-16T01:41:26.215 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:26.215 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":136,"btime":"2024-09-16T01:41:23:258090+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8175,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1255135754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1255135754},{"type":"v1","addr":"172.21.15.88:6837","nonce":1255135754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:23.258086+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:41:26.215 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 136 2024-09-16T01:41:26.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.215+0000 7f5e298dd640 1 -- 172.21.15.88:0/3644634870 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e1006c2b0 msgr2=0x7f5e1006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:26.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.215+0000 7f5e298dd640 1 --2- 172.21.15.88:0/3644634870 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e1006c2b0 0x7f5e1006e770 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f5e14034000 tx=0x7f5e1402e000 comp rx=0 tx=0).stop 2024-09-16T01:41:26.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.215+0000 7f5e298dd640 1 -- 172.21.15.88:0/3644634870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e240d3310 msgr2=0x7f5e24151040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:26.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.215+0000 7f5e298dd640 1 --2- 172.21.15.88:0/3644634870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e240d3310 0x7f5e24151040 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f5e1800c400 tx=0x7f5e1800c8c0 comp rx=0 tx=0).stop 2024-09-16T01:41:26.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.215+0000 7f5e298dd640 1 -- 172.21.15.88:0/3644634870 shutdown_connections 2024-09-16T01:41:26.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.215+0000 7f5e298dd640 1 --2- 172.21.15.88:0/3644634870 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e1006c2b0 0x7f5e1006e770 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:26.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.215+0000 7f5e298dd640 1 --2- 172.21.15.88:0/3644634870 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e2415e660 0x7f5e240b48f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:26.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.215+0000 7f5e298dd640 1 --2- 172.21.15.88:0/3644634870 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e240d3310 0x7f5e24151040 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:26.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.215+0000 7f5e298dd640 1 --2- 172.21.15.88:0/3644634870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e240d1820 0x7f5e24161190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:26.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.215+0000 7f5e298dd640 1 -- 172.21.15.88:0/3644634870 >> 172.21.15.88:0/3644634870 conn(0x7f5e240b6410 msgr2=0x7f5e240b8470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:26.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.215+0000 7f5e298dd640 1 -- 172.21.15.88:0/3644634870 shutdown_connections 2024-09-16T01:41:26.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.215+0000 7f5e298dd640 1 -- 172.21.15.88:0/3644634870 wait complete. 2024-09-16T01:41:26.227 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:41:26.228 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:26.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.325+0000 7f4489331640 1 Processor -- start 2024-09-16T01:41:26.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.325+0000 7f4489331640 1 -- start start 2024-09-16T01:41:26.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.326+0000 7f4489331640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44840d1820 0x7f44841691c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:26.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.326+0000 7f4489331640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f448415ea40 con 0x7f448415e660 2024-09-16T01:41:26.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.326+0000 7f4489331640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44840d3870 con 0x7f44840d3310 2024-09-16T01:41:26.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.326+0000 7f4489331640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f44840d3b50 con 0x7f44840d1820 2024-09-16T01:41:26.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.326+0000 7f4482ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f448415e660 0x7f44840cf730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59472/0 (socket says 172.21.15.88:59472) 2024-09-16T01:41:26.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.326+0000 7f4482ffd640 1 -- 172.21.15.88:0/2461612792 learned_addr learned my addr 172.21.15.88:0/2461612792 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:26.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.326+0000 7f4481ffb640 1 -- 172.21.15.88:0/2461612792 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 512344081 0 0) 0x7f448415ea40 con 0x7f448415e660 2024-09-16T01:41:26.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.326+0000 7f44837fe640 1 --2- 172.21.15.88:0/2461612792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44840d1820 0x7f44841691c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:26.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.326+0000 7f4481ffb640 1 -- 172.21.15.88:0/2461612792 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f446c003520 con 0x7f448415e660 2024-09-16T01:41:26.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.326+0000 7f4481ffb640 1 -- 172.21.15.88:0/2461612792 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1116312329 0 0) 0x7f446c003520 con 0x7f448415e660 2024-09-16T01:41:26.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.326+0000 7f4481ffb640 1 -- 172.21.15.88:0/2461612792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44840d1820 msgr2=0x7f44841691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:26.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.326+0000 7f4481ffb640 1 --2- 172.21.15.88:0/2461612792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44840d1820 0x7f44841691c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:26.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.326+0000 7f4481ffb640 1 -- 172.21.15.88:0/2461612792 >> v1:172.21.15.154:6789/0 conn(0x7f44840d3310 legacy=0x7f44840d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:26.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.326+0000 7f4481ffb640 1 -- 172.21.15.88:0/2461612792 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44840d3e30 con 0x7f448415e660 2024-09-16T01:41:26.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.326+0000 7f4481ffb640 1 -- 172.21.15.88:0/2461612792 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f44740019f0 con 0x7f448415e660 2024-09-16T01:41:26.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.326+0000 7f4481ffb640 1 -- 172.21.15.88:0/2461612792 >> v1:172.21.15.88:6789/0 conn(0x7f448415e660 legacy=0x7f44840cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:26.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.326+0000 7f4481ffb640 1 --2- 172.21.15.88:0/2461612792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f446c0039e0 0x7f446c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:26.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.326+0000 7f4481ffb640 1 --2- 172.21.15.88:0/2461612792 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f446c004800 0x7f446c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:26.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.327+0000 7f4481ffb640 1 --2- 172.21.15.88:0/2461612792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44840d1820 0x7f446c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:26.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.327+0000 7f4481ffb640 1 -- 172.21.15.88:0/2461612792 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f44840d3b50 con 0x7f446c004800 2024-09-16T01:41:26.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.327+0000 7f4481ffb640 1 -- 172.21.15.88:0/2461612792 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f446c00ba70 con 0x7f446c0039e0 2024-09-16T01:41:26.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.327+0000 7f4481ffb640 1 -- 172.21.15.88:0/2461612792 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f446c00bc70 con 0x7f44840d1820 2024-09-16T01:41:26.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.327+0000 7f44827fc640 1 --2- 172.21.15.88:0/2461612792 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f446c004800 0x7f446c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:26.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.327+0000 7f4482ffd640 1 --2- 172.21.15.88:0/2461612792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f446c0039e0 0x7f446c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:26.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.327+0000 7f44837fe640 1 --2- 172.21.15.88:0/2461612792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44840d1820 0x7f446c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:26.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.327+0000 7f44827fc640 1 -- 172.21.15.88:0/2461612792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44840d1820 msgr2=0x7f446c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:26.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.327+0000 7f44827fc640 1 --2- 172.21.15.88:0/2461612792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44840d1820 0x7f446c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:26.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.327+0000 7f44827fc640 1 -- 172.21.15.88:0/2461612792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f446c0039e0 msgr2=0x7f446c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:26.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.327+0000 7f44827fc640 1 --2- 172.21.15.88:0/2461612792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f446c0039e0 0x7f446c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:26.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.327+0000 7f44827fc640 1 -- 172.21.15.88:0/2461612792 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f446c003520 con 0x7f446c004800 2024-09-16T01:41:26.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.328+0000 7f44827fc640 1 --2- 172.21.15.88:0/2461612792 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f446c004800 0x7f446c004c80 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f4478001720 tx=0x7f446c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45c4d5bb6445c737 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:26.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.328+0000 7f4481ffb640 1 -- 172.21.15.88:0/2461612792 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4478001370 con 0x7f446c004800 2024-09-16T01:41:26.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.328+0000 7f4481ffb640 1 -- 172.21.15.88:0/2461612792 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44780052d0 con 0x7f446c004800 2024-09-16T01:41:26.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.328+0000 7f4481ffb640 1 -- 172.21.15.88:0/2461612792 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447802ab90 con 0x7f446c004800 2024-09-16T01:41:26.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.328+0000 7f4489331640 1 -- 172.21.15.88:0/2461612792 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f446c004800 msgr2=0x7f446c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:26.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.328+0000 7f4489331640 1 --2- 172.21.15.88:0/2461612792 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f446c004800 0x7f446c004c80 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f4478001720 tx=0x7f446c005830 comp rx=0 tx=0).stop 2024-09-16T01:41:26.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.328+0000 7f44837fe640 1 -- 172.21.15.88:0/2461612792 reap_dead start 2024-09-16T01:41:26.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.328+0000 7f4489331640 1 -- 172.21.15.88:0/2461612792 shutdown_connections 2024-09-16T01:41:26.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.328+0000 7f4489331640 1 -- 172.21.15.88:0/2461612792 >> 172.21.15.88:0/2461612792 conn(0x7f44840b6410 msgr2=0x7f44840b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:26.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.328+0000 7f4489331640 1 -- 172.21.15.88:0/2461612792 shutdown_connections 2024-09-16T01:41:26.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.328+0000 7f4489331640 1 -- 172.21.15.88:0/2461612792 wait complete. 2024-09-16T01:41:26.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.330+0000 7f4489331640 1 Processor -- start 2024-09-16T01:41:26.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.330+0000 7f4489331640 1 -- start start 2024-09-16T01:41:26.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.330+0000 7f4489331640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f446c004800 0x7f4484150010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:26.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.330+0000 7f4489331640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44840cfa60 0x7f44840cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:26.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.330+0000 7f4489331640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f448415d080 0x7f448415d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:26.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.330+0000 7f4489331640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f44840b8b80 con 0x7f44840cfa60 2024-09-16T01:41:26.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.330+0000 7f4489331640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f44840d43f0 con 0x7f448415d080 2024-09-16T01:41:26.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.330+0000 7f4489331640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f44840b8d00 con 0x7f446c004800 2024-09-16T01:41:26.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.330+0000 7f44827fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44840cfa60 0x7f44840cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:26.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.330+0000 7f44827fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44840cfa60 0x7f44840cdb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60358/0 (socket says 172.21.15.88:60358) 2024-09-16T01:41:26.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.330+0000 7f44827fc640 1 -- 172.21.15.88:0/1184894174 learned_addr learned my addr 172.21.15.88:0/1184894174 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:26.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.331+0000 7f44837fe640 1 --2- 172.21.15.88:0/1184894174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f448415d080 0x7f448415d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:26.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.331+0000 7f4482ffd640 1 --2- 172.21.15.88:0/1184894174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f446c004800 0x7f4484150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:26.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.331+0000 7f44827fc640 1 -- 172.21.15.88:0/1184894174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f446c004800 msgr2=0x7f4484150010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:26.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.331+0000 7f44827fc640 1 --2- 172.21.15.88:0/1184894174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f446c004800 0x7f4484150010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:26.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.331+0000 7f44827fc640 1 -- 172.21.15.88:0/1184894174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f448415d080 msgr2=0x7f448415d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:26.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.331+0000 7f44827fc640 1 --2- 172.21.15.88:0/1184894174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f448415d080 0x7f448415d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:26.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.331+0000 7f44827fc640 1 -- 172.21.15.88:0/1184894174 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4484150550 con 0x7f44840cfa60 2024-09-16T01:41:26.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.331+0000 7f44827fc640 1 --2- 172.21.15.88:0/1184894174 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44840cfa60 0x7f44840cdb30 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f4478008fd0 tx=0x7f4478001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:26.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.331+0000 7f446b7fe640 1 -- 172.21.15.88:0/1184894174 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4478040020 con 0x7f44840cfa60 2024-09-16T01:41:26.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.331+0000 7f446b7fe640 1 -- 172.21.15.88:0/1184894174 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4478007680 con 0x7f44840cfa60 2024-09-16T01:41:26.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.331+0000 7f4489331640 1 -- 172.21.15.88:0/1184894174 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44840ce4a0 con 0x7f44840cfa60 2024-09-16T01:41:26.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.331+0000 7f446b7fe640 1 -- 172.21.15.88:0/1184894174 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4478007a30 con 0x7f44840cfa60 2024-09-16T01:41:26.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.331+0000 7f4489331640 1 -- 172.21.15.88:0/1184894174 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44840cea60 con 0x7f44840cfa60 2024-09-16T01:41:26.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.332+0000 7f4489331640 1 -- 172.21.15.88:0/1184894174 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f448415ca60 con 0x7f44840cfa60 2024-09-16T01:41:26.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.333+0000 7f446b7fe640 1 -- 172.21.15.88:0/1184894174 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4478043020 con 0x7f44840cfa60 2024-09-16T01:41:26.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.333+0000 7f446b7fe640 1 --2- 172.21.15.88:0/1184894174 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f447006c2e0 0x7f447006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:26.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.333+0000 7f4482ffd640 1 --2- 172.21.15.88:0/1184894174 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f447006c2e0 0x7f447006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:26.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.333+0000 7f446b7fe640 1 -- 172.21.15.88:0/1184894174 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 6684+0+0 (secure 0 0 0) 0x7f44780ab5a0 con 0x7f44840cfa60 2024-09-16T01:41:26.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.334+0000 7f4482ffd640 1 --2- 172.21.15.88:0/1184894174 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f447006c2e0 0x7f447006e7a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f448415dc60 tx=0x7f4474006040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:26.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.337+0000 7f446b7fe640 1 -- 172.21.15.88:0/1184894174 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f448415ca60 con 0x7f44840cfa60 2024-09-16T01:41:26.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.536+0000 7f4489331640 1 -- 172.21.15.88:0/1184894174 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44840d1110 con 0x7f44840cfa60 2024-09-16T01:41:26.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.537+0000 7f446b7fe640 1 -- 172.21.15.88:0/1184894174 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) ==== 95+0+4391 (secure 0 0 0) 0x7f447804a030 con 0x7f44840cfa60 2024-09-16T01:41:26.538 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:26.538 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":136,"btime":"2024-09-16T01:41:23:258090+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8175,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1255135754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1255135754},{"type":"v1","addr":"172.21.15.88:6837","nonce":1255135754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:23.258086+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:41:26.539 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 136 2024-09-16T01:41:26.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.539+0000 7f4489331640 1 -- 172.21.15.88:0/1184894174 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f447006c2e0 msgr2=0x7f447006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:26.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.539+0000 7f4489331640 1 --2- 172.21.15.88:0/1184894174 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f447006c2e0 0x7f447006e7a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f448415dc60 tx=0x7f4474006040 comp rx=0 tx=0).stop 2024-09-16T01:41:26.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.539+0000 7f4489331640 1 -- 172.21.15.88:0/1184894174 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44840cfa60 msgr2=0x7f44840cdb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:26.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.539+0000 7f4489331640 1 --2- 172.21.15.88:0/1184894174 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44840cfa60 0x7f44840cdb30 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f4478008fd0 tx=0x7f4478001120 comp rx=0 tx=0).stop 2024-09-16T01:41:26.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.539+0000 7f4489331640 1 -- 172.21.15.88:0/1184894174 shutdown_connections 2024-09-16T01:41:26.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.539+0000 7f4489331640 1 --2- 172.21.15.88:0/1184894174 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f447006c2e0 0x7f447006e7a0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:26.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.539+0000 7f4489331640 1 --2- 172.21.15.88:0/1184894174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f448415d080 0x7f448415d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:26.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.539+0000 7f4489331640 1 --2- 172.21.15.88:0/1184894174 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44840cfa60 0x7f44840cdb30 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:26.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.539+0000 7f4489331640 1 --2- 172.21.15.88:0/1184894174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f446c004800 0x7f4484150010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:26.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.539+0000 7f4489331640 1 -- 172.21.15.88:0/1184894174 >> 172.21.15.88:0/1184894174 conn(0x7f44840b6410 msgr2=0x7f44840b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:26.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.539+0000 7f4489331640 1 -- 172.21.15.88:0/1184894174 shutdown_connections 2024-09-16T01:41:26.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.539+0000 7f4489331640 1 -- 172.21.15.88:0/1184894174 wait complete. 2024-09-16T01:41:26.551 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 100.00000000' led to DAMAGED state 2024-09-16T01:41:26.552 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete mds0_sessionmap' 2024-09-16T01:41:26.552 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:41:26.552 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:41:26.568 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:41:26.568 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:41:26.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:26.568 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:41:26.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:41:26.625 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:41:26.626 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:41:26.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.764+0000 7f0774bc4640 1 Processor -- start 2024-09-16T01:41:26.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.764+0000 7f0774bc4640 1 -- start start 2024-09-16T01:41:26.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.765+0000 7f0774bc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f077015e350 0x7f077006f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:26.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.765+0000 7f0774bc4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0770073f30 0x7f077006fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:26.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.765+0000 7f0774bc4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f077006bd40 con 0x7f077015e730 2024-09-16T01:41:26.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.765+0000 7f0774bc4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f077006c020 con 0x7f0770073f30 2024-09-16T01:41:26.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.765+0000 7f0774bc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f077006c300 con 0x7f077015e350 2024-09-16T01:41:26.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.765+0000 7f0765d9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f077015e730 0x7f0770073820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59482/0 (socket says 172.21.15.88:59482) 2024-09-16T01:41:26.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.765+0000 7f0765d9b640 1 -- 172.21.15.88:0/2527870306 learned_addr learned my addr 172.21.15.88:0/2527870306 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:26.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.765+0000 7f076ed9d640 1 --2- 172.21.15.88:0/2527870306 >> v2:172.21.15.154:3300/0 conn(0x7f0770073f30 0x7f077006fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:26.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.765+0000 7f076e59c640 1 --2- 172.21.15.88:0/2527870306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f077015e350 0x7f077006f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:26.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.765+0000 7f076dd9b640 1 -- 172.21.15.88:0/2527870306 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1327628774 0 0) 0x7f077006bd40 con 0x7f077015e730 2024-09-16T01:41:26.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.765+0000 7f076dd9b640 1 -- 172.21.15.88:0/2527870306 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f075c003520 con 0x7f077015e730 2024-09-16T01:41:26.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.765+0000 7f076dd9b640 1 -- 172.21.15.88:0/2527870306 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1031228432 0 0) 0x7f075c003520 con 0x7f077015e730 2024-09-16T01:41:26.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.765+0000 7f076dd9b640 1 -- 172.21.15.88:0/2527870306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f077015e350 msgr2=0x7f077006f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:26.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.765+0000 7f076dd9b640 1 --2- 172.21.15.88:0/2527870306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f077015e350 0x7f077006f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:26.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.765+0000 7f076dd9b640 1 -- 172.21.15.88:0/2527870306 >> v2:172.21.15.154:3300/0 conn(0x7f0770073f30 msgr2=0x7f077006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:26.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.765+0000 7f076dd9b640 1 --2- 172.21.15.88:0/2527870306 >> v2:172.21.15.154:3300/0 conn(0x7f0770073f30 0x7f077006fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:26.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.765+0000 7f076dd9b640 1 -- 172.21.15.88:0/2527870306 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f077006c5e0 con 0x7f077015e730 2024-09-16T01:41:26.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.765+0000 7f076dd9b640 1 -- 172.21.15.88:0/2527870306 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f0758001670 con 0x7f077015e730 2024-09-16T01:41:26.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.765+0000 7f076dd9b640 1 -- 172.21.15.88:0/2527870306 >> v1:172.21.15.88:6789/0 conn(0x7f077015e730 legacy=0x7f0770073820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:26.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.765+0000 7f076dd9b640 1 --2- 172.21.15.88:0/2527870306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f075c0039b0 0x7f075c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:26.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.765+0000 7f076dd9b640 1 --2- 172.21.15.88:0/2527870306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f075c004800 0x7f075c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:26.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.765+0000 7f076dd9b640 1 --2- 172.21.15.88:0/2527870306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f077015e350 0x7f075c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:26.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.765+0000 7f076dd9b640 1 -- 172.21.15.88:0/2527870306 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f077006c020 con 0x7f075c004800 2024-09-16T01:41:26.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.765+0000 7f076dd9b640 1 -- 172.21.15.88:0/2527870306 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f077006c300 con 0x7f075c0039b0 2024-09-16T01:41:26.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.765+0000 7f076dd9b640 1 -- 172.21.15.88:0/2527870306 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f075c00b9c0 con 0x7f077015e350 2024-09-16T01:41:26.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.766+0000 7f0765d9b640 1 --2- 172.21.15.88:0/2527870306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f075c004800 0x7f075c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:26.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.766+0000 7f076e59c640 1 --2- 172.21.15.88:0/2527870306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f075c0039b0 0x7f075c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:26.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.766+0000 7f076ed9d640 1 --2- 172.21.15.88:0/2527870306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f077015e350 0x7f075c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:26.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.766+0000 7f0765d9b640 1 -- 172.21.15.88:0/2527870306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f077015e350 msgr2=0x7f075c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:26.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.766+0000 7f0765d9b640 1 --2- 172.21.15.88:0/2527870306 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f077015e350 0x7f075c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:26.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.766+0000 7f0765d9b640 1 -- 172.21.15.88:0/2527870306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f075c0039b0 msgr2=0x7f075c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:26.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.766+0000 7f0765d9b640 1 --2- 172.21.15.88:0/2527870306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f075c0039b0 0x7f075c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:26.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.766+0000 7f0765d9b640 1 -- 172.21.15.88:0/2527870306 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f075c003520 con 0x7f075c004800 2024-09-16T01:41:26.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.766+0000 7f0765d9b640 1 --2- 172.21.15.88:0/2527870306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f075c004800 0x7f075c004c80 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f0758007fd0 tx=0x7f075c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98a1e0e874e464d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:26.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.767+0000 7f076dd9b640 1 -- 172.21.15.88:0/2527870306 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0758007d50 con 0x7f075c004800 2024-09-16T01:41:26.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.767+0000 7f076dd9b640 1 -- 172.21.15.88:0/2527870306 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0758005ad0 con 0x7f075c004800 2024-09-16T01:41:26.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.767+0000 7f076dd9b640 1 -- 172.21.15.88:0/2527870306 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0758007070 con 0x7f075c004800 2024-09-16T01:41:26.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.767+0000 7f0774bc4640 1 -- 172.21.15.88:0/2527870306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f075c004800 msgr2=0x7f075c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:26.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.767+0000 7f0774bc4640 1 --2- 172.21.15.88:0/2527870306 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f075c004800 0x7f075c004c80 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f0758007fd0 tx=0x7f075c005830 comp rx=0 tx=0).stop 2024-09-16T01:41:26.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.767+0000 7f076ed9d640 1 -- 172.21.15.88:0/2527870306 reap_dead start 2024-09-16T01:41:26.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.767+0000 7f0774bc4640 1 -- 172.21.15.88:0/2527870306 shutdown_connections 2024-09-16T01:41:26.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.767+0000 7f0774bc4640 1 -- 172.21.15.88:0/2527870306 >> 172.21.15.88:0/2527870306 conn(0x7f07700b3070 msgr2=0x7f07700b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:26.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.767+0000 7f0774bc4640 1 -- 172.21.15.88:0/2527870306 shutdown_connections 2024-09-16T01:41:26.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.767+0000 7f0774bc4640 1 -- 172.21.15.88:0/2527870306 wait complete. 2024-09-16T01:41:26.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.769+0000 7f0774bc4640 1 Processor -- start 2024-09-16T01:41:26.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.769+0000 7f0774bc4640 1 -- start start 2024-09-16T01:41:26.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.769+0000 7f0774bc4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f075c004800 0x7f0770147dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:26.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.769+0000 7f0774bc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f077015eb60 0x7f077015efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:26.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.769+0000 7f0774bc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f077015ca50 0x7f077015cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:26.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.769+0000 7f0774bc4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f077006f890 con 0x7f075c004800 2024-09-16T01:41:26.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.769+0000 7f0774bc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f077006f710 con 0x7f077015eb60 2024-09-16T01:41:26.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.769+0000 7f0774bc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f077006fa10 con 0x7f077015ca50 2024-09-16T01:41:26.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.769+0000 7f076e59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f075c004800 0x7f0770147dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:26.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.769+0000 7f076e59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f075c004800 0x7f0770147dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60372/0 (socket says 172.21.15.88:60372) 2024-09-16T01:41:26.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.769+0000 7f076e59c640 1 -- 172.21.15.88:0/3266701862 learned_addr learned my addr 172.21.15.88:0/3266701862 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:26.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.770+0000 7f0765d9b640 1 --2- 172.21.15.88:0/3266701862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f077015eb60 0x7f077015efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:26.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.770+0000 7f076ed9d640 1 --2- 172.21.15.88:0/3266701862 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f077015ca50 0x7f077015cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:26.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.770+0000 7f076e59c640 1 -- 172.21.15.88:0/3266701862 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f077015ca50 msgr2=0x7f077015cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:26.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.770+0000 7f076e59c640 1 --2- 172.21.15.88:0/3266701862 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f077015ca50 0x7f077015cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:26.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.770+0000 7f076e59c640 1 -- 172.21.15.88:0/3266701862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f077015eb60 msgr2=0x7f077015efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:26.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.770+0000 7f076e59c640 1 --2- 172.21.15.88:0/3266701862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f077015eb60 0x7f077015efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:26.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.770+0000 7f076e59c640 1 -- 172.21.15.88:0/3266701862 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0770148310 con 0x7f075c004800 2024-09-16T01:41:26.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.770+0000 7f076e59c640 1 --2- 172.21.15.88:0/3266701862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f075c004800 0x7f0770147dd0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f0760011940 tx=0x7f0760011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:26.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.770+0000 7f07677fe640 1 -- 172.21.15.88:0/3266701862 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f076000faf0 con 0x7f075c004800 2024-09-16T01:41:26.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.770+0000 7f07677fe640 1 -- 172.21.15.88:0/3266701862 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f076000fc90 con 0x7f075c004800 2024-09-16T01:41:26.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.770+0000 7f07677fe640 1 -- 172.21.15.88:0/3266701862 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0760014440 con 0x7f075c004800 2024-09-16T01:41:26.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.770+0000 7f0774bc4640 1 -- 172.21.15.88:0/3266701862 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0770160930 con 0x7f075c004800 2024-09-16T01:41:26.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.770+0000 7f0774bc4640 1 -- 172.21.15.88:0/3266701862 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f077015f990 con 0x7f075c004800 2024-09-16T01:41:26.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.771+0000 7f0774bc4640 1 -- 172.21.15.88:0/3266701862 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f077006fc30 con 0x7f075c004800 2024-09-16T01:41:26.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.772+0000 7f07677fe640 1 -- 172.21.15.88:0/3266701862 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0760012040 con 0x7f075c004800 2024-09-16T01:41:26.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.772+0000 7f07677fe640 1 --2- 172.21.15.88:0/3266701862 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f075006c2e0 0x7f075006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:26.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.772+0000 7f07677fe640 1 -- 172.21.15.88:0/3266701862 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 6684+0+0 (secure 0 0 0) 0x7f076008bcf0 con 0x7f075c004800 2024-09-16T01:41:26.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.772+0000 7f0765d9b640 1 --2- 172.21.15.88:0/3266701862 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f075006c2e0 0x7f075006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:26.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.772+0000 7f0765d9b640 1 --2- 172.21.15.88:0/3266701862 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f075006c2e0 0x7f075006e7a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0758033000 tx=0x7f0758009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:26.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.776+0000 7f07677fe640 1 -- 172.21.15.88:0/3266701862 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0760090070 con 0x7f075c004800 2024-09-16T01:41:26.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:26.977+0000 7f0774bc4640 1 -- 172.21.15.88:0/3266701862 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f07700b5620 con 0x7f075c004800 2024-09-16T01:41:27.217 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:41:27.216+0000 7ff93341e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:41:27.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.254+0000 7f07677fe640 1 -- 172.21.15.88:0/3266701862 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v137) ==== 161+0+0 (secure 0 0 0) 0x7f0770148310 con 0x7f075c004800 2024-09-16T01:41:27.255 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:41:27.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.256+0000 7f0774bc4640 1 -- 172.21.15.88:0/3266701862 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f075006c2e0 msgr2=0x7f075006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:27.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.256+0000 7f0774bc4640 1 --2- 172.21.15.88:0/3266701862 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f075006c2e0 0x7f075006e7a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0758033000 tx=0x7f0758009030 comp rx=0 tx=0).stop 2024-09-16T01:41:27.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.256+0000 7f0774bc4640 1 -- 172.21.15.88:0/3266701862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f075c004800 msgr2=0x7f0770147dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:27.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.256+0000 7f0774bc4640 1 --2- 172.21.15.88:0/3266701862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f075c004800 0x7f0770147dd0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f0760011940 tx=0x7f0760011e00 comp rx=0 tx=0).stop 2024-09-16T01:41:27.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.256+0000 7f0774bc4640 1 -- 172.21.15.88:0/3266701862 shutdown_connections 2024-09-16T01:41:27.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.256+0000 7f0774bc4640 1 --2- 172.21.15.88:0/3266701862 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f075006c2e0 0x7f075006e7a0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:27.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.256+0000 7f0774bc4640 1 --2- 172.21.15.88:0/3266701862 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f077015ca50 0x7f077015cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:27.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.256+0000 7f0774bc4640 1 --2- 172.21.15.88:0/3266701862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f077015eb60 0x7f077015efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:27.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.256+0000 7f0774bc4640 1 --2- 172.21.15.88:0/3266701862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f075c004800 0x7f0770147dd0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:27.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.256+0000 7f0774bc4640 1 -- 172.21.15.88:0/3266701862 >> 172.21.15.88:0/3266701862 conn(0x7f07700b3070 msgr2=0x7f07700b1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:27.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.256+0000 7f0774bc4640 1 -- 172.21.15.88:0/3266701862 shutdown_connections 2024-09-16T01:41:27.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.256+0000 7f0774bc4640 1 -- 172.21.15.88:0/3266701862 wait complete. 2024-09-16T01:41:27.269 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:41:27.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.367+0000 7faf235cf640 1 Processor -- start 2024-09-16T01:41:27.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.367+0000 7faf235cf640 1 -- start start 2024-09-16T01:41:27.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.368+0000 7faf235cf640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7faf1c15b480 0x7faf1c0b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:27.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.368+0000 7faf235cf640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faf1c072700 0x7faf1c15bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:27.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.368+0000 7faf235cf640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf1c0ccff0 con 0x7faf1c0b60c0 2024-09-16T01:41:27.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.368+0000 7faf235cf640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7faf1c0cd2d0 con 0x7faf1c072700 2024-09-16T01:41:27.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.368+0000 7faf235cf640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7faf1c0cd5b0 con 0x7faf1c15b480 2024-09-16T01:41:27.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.368+0000 7faf21b6c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faf1c072700 0x7faf1c15bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:27.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.368+0000 7faf21b6c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faf1c072700 0x7faf1c15bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60374/0 (socket says 172.21.15.88:60374) 2024-09-16T01:41:27.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.368+0000 7faf21b6c640 1 -- 172.21.15.88:0/1980522390 learned_addr learned my addr 172.21.15.88:0/1980522390 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:27.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.368+0000 7faf21b6c640 1 -- 172.21.15.88:0/1980522390 >> v2:172.21.15.154:3300/0 conn(0x7faf1c15b480 msgr2=0x7faf1c0b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:27.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.368+0000 7faf21b6c640 1 --2- 172.21.15.88:0/1980522390 >> v2:172.21.15.154:3300/0 conn(0x7faf1c15b480 0x7faf1c0b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:27.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.368+0000 7faf21b6c640 1 -- 172.21.15.88:0/1980522390 >> v1:172.21.15.154:6789/0 conn(0x7faf1c0b60c0 legacy=0x7faf1c071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:27.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.368+0000 7faf21b6c640 1 -- 172.21.15.88:0/1980522390 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf1c0cd890 con 0x7faf1c072700 2024-09-16T01:41:27.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.369+0000 7faf21b6c640 1 --2- 172.21.15.88:0/1980522390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf1c072700 0x7faf1c15bc60 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7faf1c0694d0 tx=0x7faf1402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0a0bad22e1228a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:27.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.369+0000 7faf13fff640 1 -- 172.21.15.88:0/1980522390 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf14036070 con 0x7faf1c072700 2024-09-16T01:41:27.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.369+0000 7faf13fff640 1 -- 172.21.15.88:0/1980522390 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf14031600 con 0x7faf1c072700 2024-09-16T01:41:27.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.369+0000 7faf13fff640 1 -- 172.21.15.88:0/1980522390 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf140319d0 con 0x7faf1c072700 2024-09-16T01:41:27.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.370+0000 7faf235cf640 1 -- 172.21.15.88:0/1980522390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf1c072700 msgr2=0x7faf1c15bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:27.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.370+0000 7faf235cf640 1 --2- 172.21.15.88:0/1980522390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf1c072700 0x7faf1c15bc60 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7faf1c0694d0 tx=0x7faf1402a660 comp rx=0 tx=0).stop 2024-09-16T01:41:27.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.370+0000 7faf235cf640 1 -- 172.21.15.88:0/1980522390 shutdown_connections 2024-09-16T01:41:27.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.370+0000 7faf235cf640 1 --2- 172.21.15.88:0/1980522390 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf1c072700 0x7faf1c15bc60 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:27.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.370+0000 7faf235cf640 1 --2- 172.21.15.88:0/1980522390 >> v2:172.21.15.154:3300/0 conn(0x7faf1c15b480 0x7faf1c0b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:27.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.370+0000 7faf235cf640 1 -- 172.21.15.88:0/1980522390 >> 172.21.15.88:0/1980522390 conn(0x7faf1c06d0f0 msgr2=0x7faf1c06d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:27.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.370+0000 7faf235cf640 1 -- 172.21.15.88:0/1980522390 shutdown_connections 2024-09-16T01:41:27.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.370+0000 7faf235cf640 1 -- 172.21.15.88:0/1980522390 wait complete. 2024-09-16T01:41:27.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.371+0000 7faf235cf640 1 Processor -- start 2024-09-16T01:41:27.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.371+0000 7faf235cf640 1 -- start start 2024-09-16T01:41:27.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.372+0000 7faf235cf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf1c072700 0x7faf1c0cb040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:27.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.372+0000 7faf235cf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf1c0b60c0 0x7faf1c148ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:27.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.372+0000 7faf235cf640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faf1c15b480 0x7faf1c144f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:27.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.372+0000 7faf235cf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7faf1c15b9e0 con 0x7faf1c0b60c0 2024-09-16T01:41:27.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.372+0000 7faf235cf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faf1c15b860 con 0x7faf1c072700 2024-09-16T01:41:27.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.372+0000 7faf235cf640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7faf1c15c1a0 con 0x7faf1c15b480 2024-09-16T01:41:27.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.372+0000 7faf20b6a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf1c0b60c0 0x7faf1c148ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:27.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.372+0000 7faf20b6a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf1c0b60c0 0x7faf1c148ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60390/0 (socket says 172.21.15.88:60390) 2024-09-16T01:41:27.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.372+0000 7faf20b6a640 1 -- 172.21.15.88:0/3768715885 learned_addr learned my addr 172.21.15.88:0/3768715885 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:27.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.372+0000 7faf2136b640 1 --2- 172.21.15.88:0/3768715885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf1c072700 0x7faf1c0cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:27.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.372+0000 7faf21b6c640 1 --2- 172.21.15.88:0/3768715885 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faf1c15b480 0x7faf1c144f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:27.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.372+0000 7faf20b6a640 1 -- 172.21.15.88:0/3768715885 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faf1c15b480 msgr2=0x7faf1c144f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:27.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.372+0000 7faf20b6a640 1 --2- 172.21.15.88:0/3768715885 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faf1c15b480 0x7faf1c144f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:27.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.372+0000 7faf20b6a640 1 -- 172.21.15.88:0/3768715885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf1c072700 msgr2=0x7faf1c0cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:27.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.372+0000 7faf20b6a640 1 --2- 172.21.15.88:0/3768715885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf1c072700 0x7faf1c0cb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:27.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.372+0000 7faf20b6a640 1 -- 172.21.15.88:0/3768715885 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf1c1491e0 con 0x7faf1c0b60c0 2024-09-16T01:41:27.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.372+0000 7faf2136b640 1 --2- 172.21.15.88:0/3768715885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf1c072700 0x7faf1c0cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:27.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.372+0000 7faf20b6a640 1 --2- 172.21.15.88:0/3768715885 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf1c0b60c0 0x7faf1c148ca0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7faf0c00c410 tx=0x7faf0c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:27.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.373+0000 7faf11ffb640 1 -- 172.21.15.88:0/3768715885 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf0c017070 con 0x7faf1c0b60c0 2024-09-16T01:41:27.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.373+0000 7faf11ffb640 1 -- 172.21.15.88:0/3768715885 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf0c0035f0 con 0x7faf1c0b60c0 2024-09-16T01:41:27.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.373+0000 7faf235cf640 1 -- 172.21.15.88:0/3768715885 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf1c1e4120 con 0x7faf1c0b60c0 2024-09-16T01:41:27.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.373+0000 7faf11ffb640 1 -- 172.21.15.88:0/3768715885 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf0c003a20 con 0x7faf1c0b60c0 2024-09-16T01:41:27.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.373+0000 7faf235cf640 1 -- 172.21.15.88:0/3768715885 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf1c1e46e0 con 0x7faf1c0b60c0 2024-09-16T01:41:27.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.374+0000 7faf235cf640 1 -- 172.21.15.88:0/3768715885 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf1c15bc60 con 0x7faf1c0b60c0 2024-09-16T01:41:27.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.374+0000 7faf11ffb640 1 -- 172.21.15.88:0/3768715885 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7faf0c01a070 con 0x7faf1c0b60c0 2024-09-16T01:41:27.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.375+0000 7faf11ffb640 1 --2- 172.21.15.88:0/3768715885 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faf0006c2b0 0x7faf0006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:27.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.375+0000 7faf11ffb640 1 -- 172.21.15.88:0/3768715885 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 6684+0+0 (secure 0 0 0) 0x7faf0c08ac20 con 0x7faf1c0b60c0 2024-09-16T01:41:27.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.375+0000 7faf2136b640 1 --2- 172.21.15.88:0/3768715885 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faf0006c2b0 0x7faf0006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:27.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.375+0000 7faf2136b640 1 --2- 172.21.15.88:0/3768715885 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faf0006c2b0 0x7faf0006e770 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7faf18009890 tx=0x7faf18008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:27.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.379+0000 7faf11ffb640 1 -- 172.21.15.88:0/3768715885 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7faf0c059050 con 0x7faf1c0b60c0 2024-09-16T01:41:27.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:27.613+0000 7faf235cf640 1 -- 172.21.15.88:0/3768715885 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7faf1c0cc370 con 0x7faf1c0b60c0 2024-09-16T01:41:28.011 INFO:tasks.ceph.mds.a.smithi088.stderr: -52> 2024-09-16T01:41:23.254+0000 7ff92f416640 -1 log_channel(cluster) log [ERR] : dir 0x100 object missing on disk; some files may be lost (~mds0) 2024-09-16T01:41:28.011 INFO:tasks.ceph.mds.a.smithi088.stderr: -17> 2024-09-16T01:41:27.216+0000 7ff93341e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:41:28.032 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:41:28.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.255+0000 7faf11ffb640 1 -- 172.21.15.88:0/3768715885 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v138) ==== 101+0+0 (secure 0 0 0) 0x7faf0c0010c0 con 0x7faf1c0b60c0 2024-09-16T01:41:28.257 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:41:28.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.258+0000 7faf235cf640 1 -- 172.21.15.88:0/3768715885 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faf0006c2b0 msgr2=0x7faf0006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:28.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.258+0000 7faf235cf640 1 --2- 172.21.15.88:0/3768715885 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faf0006c2b0 0x7faf0006e770 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7faf18009890 tx=0x7faf18008040 comp rx=0 tx=0).stop 2024-09-16T01:41:28.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.258+0000 7faf235cf640 1 -- 172.21.15.88:0/3768715885 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf1c0b60c0 msgr2=0x7faf1c148ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:28.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.258+0000 7faf235cf640 1 --2- 172.21.15.88:0/3768715885 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf1c0b60c0 0x7faf1c148ca0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7faf0c00c410 tx=0x7faf0c00c8d0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.258+0000 7faf235cf640 1 -- 172.21.15.88:0/3768715885 shutdown_connections 2024-09-16T01:41:28.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.258+0000 7faf235cf640 1 --2- 172.21.15.88:0/3768715885 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faf0006c2b0 0x7faf0006e770 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.258+0000 7faf235cf640 1 --2- 172.21.15.88:0/3768715885 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faf1c15b480 0x7faf1c144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.258+0000 7faf235cf640 1 --2- 172.21.15.88:0/3768715885 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faf1c0b60c0 0x7faf1c148ca0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.258+0000 7faf235cf640 1 --2- 172.21.15.88:0/3768715885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf1c072700 0x7faf1c0cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.258+0000 7faf235cf640 1 -- 172.21.15.88:0/3768715885 >> 172.21.15.88:0/3768715885 conn(0x7faf1c06d0f0 msgr2=0x7faf1c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:28.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.258+0000 7faf235cf640 1 -- 172.21.15.88:0/3768715885 shutdown_connections 2024-09-16T01:41:28.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.258+0000 7faf235cf640 1 -- 172.21.15.88:0/3768715885 wait complete. 2024-09-16T01:41:28.272 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:41:28.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.314+0000 7f93b6b3ff40 1 Processor -- start 2024-09-16T01:41:28.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.314+0000 7f93b6b3ff40 1 -- start start 2024-09-16T01:41:28.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.315+0000 7f93b6b3ff40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f939c0030f0 0x555a56a2f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.315+0000 7f93b6b3ff40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555a569bc020 0x555a56a50e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.315+0000 7f93b6b3ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555a56a51390 0x555a56ae4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.315+0000 7f93b6b3ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x555a568ca510 con 0x555a56a51390 2024-09-16T01:41:28.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.315+0000 7f93b6b3ff40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x555a56923c50 con 0x7f939c0030f0 2024-09-16T01:41:28.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.315+0000 7f93b6b3ff40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x555a56a4def0 con 0x555a569bc020 2024-09-16T01:41:28.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.315+0000 7f93b4e36640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555a56a51390 0x555a56ae4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.315+0000 7f93b4e36640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555a56a51390 0x555a56ae4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60432/0 (socket says 172.21.15.88:60432) 2024-09-16T01:41:28.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.315+0000 7f93b4e36640 1 -- 172.21.15.88:0/3121175372 learned_addr learned my addr 172.21.15.88:0/3121175372 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:28.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.315+0000 7f93b4635640 1 --2- 172.21.15.88:0/3121175372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f939c0030f0 0x555a56a2f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.315+0000 7f93b3e34640 1 --2- 172.21.15.88:0/3121175372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555a569bc020 0x555a56a50e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.315+0000 7f93b4e36640 1 -- 172.21.15.88:0/3121175372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555a569bc020 msgr2=0x555a56a50e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:28.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.315+0000 7f93b4e36640 1 --2- 172.21.15.88:0/3121175372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555a569bc020 0x555a56a50e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.315+0000 7f93b4e36640 1 -- 172.21.15.88:0/3121175372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f939c0030f0 msgr2=0x555a56a2f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:28.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.315+0000 7f93b4e36640 1 --2- 172.21.15.88:0/3121175372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f939c0030f0 0x555a56a2f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.315+0000 7f93b4e36640 1 -- 172.21.15.88:0/3121175372 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a569b98a0 con 0x555a56a51390 2024-09-16T01:41:28.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.315+0000 7f93b4e36640 1 --2- 172.21.15.88:0/3121175372 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555a56a51390 0x555a56ae4e90 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x555a5697e120 tx=0x7f93ac0029c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6202a9c9b1e3556 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:28.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.315+0000 7f93b2e32640 1 -- 172.21.15.88:0/3121175372 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ac013640 con 0x555a56a51390 2024-09-16T01:41:28.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.315+0000 7f93b2e32640 1 -- 172.21.15.88:0/3121175372 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93ac01a070 con 0x555a56a51390 2024-09-16T01:41:28.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.315+0000 7f93b2e32640 1 -- 172.21.15.88:0/3121175372 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ac013a40 con 0x555a56a51390 2024-09-16T01:41:28.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.316+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3121175372 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555a56a51390 msgr2=0x555a56ae4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:28.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.316+0000 7f93b6b3ff40 1 --2- 172.21.15.88:0/3121175372 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555a56a51390 0x555a56ae4e90 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x555a5697e120 tx=0x7f93ac0029c0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.316+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3121175372 shutdown_connections 2024-09-16T01:41:28.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.316+0000 7f93b6b3ff40 1 --2- 172.21.15.88:0/3121175372 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555a56a51390 0x555a56ae4e90 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.316+0000 7f93b6b3ff40 1 --2- 172.21.15.88:0/3121175372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555a569bc020 0x555a56a50e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.316+0000 7f93b6b3ff40 1 --2- 172.21.15.88:0/3121175372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f939c0030f0 0x555a56a2f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.316+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3121175372 >> 172.21.15.88:0/3121175372 conn(0x555a56977be0 msgr2=0x555a569c3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:28.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.316+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3121175372 shutdown_connections 2024-09-16T01:41:28.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.316+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3121175372 wait complete. 2024-09-16T01:41:28.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.317+0000 7f93b6b3ff40 1 Processor -- start 2024-09-16T01:41:28.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.317+0000 7f93b6b3ff40 1 -- start start 2024-09-16T01:41:28.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.317+0000 7f93b6b3ff40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a569bc020 0x555a56a33bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.317+0000 7f93b6b3ff40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555a56a51390 0x555a569a9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.317+0000 7f93b6b3ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f939c0030f0 0x555a56a2e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.317+0000 7f93b6b3ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x555a56a2f820 con 0x7f939c0030f0 2024-09-16T01:41:28.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.317+0000 7f93b6b3ff40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x555a569c4610 con 0x555a569bc020 2024-09-16T01:41:28.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.317+0000 7f93b6b3ff40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x555a569c3ec0 con 0x555a56a51390 2024-09-16T01:41:28.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.317+0000 7f93b4e36640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f939c0030f0 0x555a56a2e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.317+0000 7f93b4e36640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f939c0030f0 0x555a56a2e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60434/0 (socket says 172.21.15.88:60434) 2024-09-16T01:41:28.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.317+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 learned_addr learned my addr 172.21.15.88:0/3264172619 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:28.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.317+0000 7f93b4635640 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a569bc020 0x555a56a33bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.317+0000 7f93b3e34640 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555a56a51390 0x555a569a9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.318+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555a56a51390 msgr2=0x555a569a9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:28.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.318+0000 7f93b4e36640 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555a56a51390 0x555a569a9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.318+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a569bc020 msgr2=0x555a56a33bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:28.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.318+0000 7f93b4e36640 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555a569bc020 0x555a56a33bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.318+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a56a33710 con 0x7f939c0030f0 2024-09-16T01:41:28.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.318+0000 7f93b4e36640 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f939c0030f0 0x555a56a2e140 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x555a56a48c40 tx=0x7f93ac003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:28.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.318+0000 7f93b1e30640 1 -- 172.21.15.88:0/3264172619 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ac024030 con 0x7f939c0030f0 2024-09-16T01:41:28.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.318+0000 7f93b1e30640 1 -- 172.21.15.88:0/3264172619 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93ac01f610 con 0x7f939c0030f0 2024-09-16T01:41:28.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.318+0000 7f93b1e30640 1 -- 172.21.15.88:0/3264172619 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ac01f7e0 con 0x7f939c0030f0 2024-09-16T01:41:28.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.318+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x555a569a9d70 con 0x7f939c0030f0 2024-09-16T01:41:28.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.318+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a56a30360 con 0x7f939c0030f0 2024-09-16T01:41:28.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.319+0000 7f93b1e30640 1 -- 172.21.15.88:0/3264172619 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f93ac027030 con 0x7f939c0030f0 2024-09-16T01:41:28.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.320+0000 7f93b1e30640 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f93a006ac90 0x7f93a006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.320+0000 7f93b1e30640 1 -- 172.21.15.88:0/3264172619 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 6684+0+0 (secure 0 0 0) 0x7f93ac08bb60 con 0x7f939c0030f0 2024-09-16T01:41:28.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.320+0000 7f93b4635640 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f93a006ac90 0x7f93a006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.326 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:41:28.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.320+0000 7f93b6b3ff40 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x555a56a31550 0x555a56a4e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.320+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56a50fc0 con 0x555a56a31550 2024-09-16T01:41:28.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.320+0000 7f93b3e34640 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x555a56a31550 0x555a56a4e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.320+0000 7f93b4635640 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f93a006ac90 0x7f93a006d150 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f93a40017e0 tx=0x7f93a4001430 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:28.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.321+0000 7f93b3e34640 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x555a56a31550 0x555a56a4e610 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:28.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.322+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v98'319 uv293 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.322+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) -- 0x555a56a4eb50 con 0x555a56a31550 2024-09-16T01:41:28.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.323+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v98'320 uv320 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.323+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56a4ef20 con 0x555a56a31550 2024-09-16T01:41:28.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.324+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v98'321 uv321 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.328 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:41:28.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.325+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af24f0 con 0x555a56a31550 2024-09-16T01:41:28.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.326+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v98'322 uv322 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.326+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af2890 con 0x555a56a31550 2024-09-16T01:41:28.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.327+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v98'323 uv323 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.327+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af14e0 con 0x555a56a31550 2024-09-16T01:41:28.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.328+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v98'324 uv324 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.328+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af18f0 con 0x555a56a31550 2024-09-16T01:41:28.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.329+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v98'325 uv299 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.329+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) -- 0x555a56af18f0 con 0x555a56a31550 2024-09-16T01:41:28.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.330+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v98'326 uv326 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.330+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af18f0 con 0x555a56a31550 2024-09-16T01:41:28.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.331+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v98'327 uv327 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.332 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:41:28.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.331+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af18f0 con 0x555a56a31550 2024-09-16T01:41:28.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.332+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v98'328 uv328 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.332+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af18f0 con 0x555a56a31550 2024-09-16T01:41:28.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.334+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v98'329 uv329 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.334+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af18f0 con 0x555a56a31550 2024-09-16T01:41:28.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.335+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v98'330 uv330 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.335+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af18f0 con 0x555a56a31550 2024-09-16T01:41:28.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.337+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v98'331 uv306 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.337+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) -- 0x555a56af18f0 con 0x555a56a31550 2024-09-16T01:41:28.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.338+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v98'332 uv332 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.338+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af18f0 con 0x555a56a31550 2024-09-16T01:41:28.340 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:41:28.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.339+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v98'333 uv333 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.339+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af18f0 con 0x555a56a31550 2024-09-16T01:41:28.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.340+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v98'334 uv334 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.340+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af18f0 con 0x555a56a31550 2024-09-16T01:41:28.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.341+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v98'335 uv335 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.341+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af18f0 con 0x555a56a31550 2024-09-16T01:41:28.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.342+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v98'336 uv336 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.342+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af1cf0 con 0x555a56a31550 2024-09-16T01:41:28.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.343+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v98'337 uv337 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.343+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af1ba0 con 0x555a56a31550 2024-09-16T01:41:28.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.344+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v98'338 uv312 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.344+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) -- 0x555a56af1ba0 con 0x555a56a31550 2024-09-16T01:41:28.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.345+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v98'339 uv339 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.345+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af1ba0 con 0x555a56a31550 2024-09-16T01:41:28.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.346+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v98'340 uv340 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.348 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:41:28.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.346+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af1ba0 con 0x555a56a31550 2024-09-16T01:41:28.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.347+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v98'341 uv341 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.347+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af1ba0 con 0x555a56a31550 2024-09-16T01:41:28.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.348+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v98'342 uv342 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.348+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af1ba0 con 0x555a56a31550 2024-09-16T01:41:28.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.349+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v98'343 uv343 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.349+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af1ba0 con 0x555a56a31550 2024-09-16T01:41:28.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.350+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v98'344 uv318 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.350+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) -- 0x555a56af1ba0 con 0x555a56a31550 2024-09-16T01:41:28.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.352+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v98'345 uv345 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.352+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af1ba0 con 0x555a56a31550 2024-09-16T01:41:28.354 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:41:28.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.353+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v98'346 uv346 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.353+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af1ba0 con 0x555a56a31550 2024-09-16T01:41:28.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.354+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v98'347 uv347 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.354+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af1ba0 con 0x555a56a31550 2024-09-16T01:41:28.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.355+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v98'348 uv348 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.355+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af1ba0 con 0x555a56a31550 2024-09-16T01:41:28.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.356+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v98'349 uv349 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.356+0000 7f93b6b3ff40 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x555a56af2c30 0x555a56af5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.356+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af5560 con 0x555a56af2c30 2024-09-16T01:41:28.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.356+0000 7f93b4635640 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x555a56af2c30 0x555a56af5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.356+0000 7f93b4635640 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x555a56af2c30 0x555a56af5020 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:28.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.358+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v98'242 uv224 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56af2c30 2024-09-16T01:41:28.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.358+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) -- 0x555a56af5900 con 0x555a56af2c30 2024-09-16T01:41:28.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.359+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v98'243 uv243 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56af2c30 2024-09-16T01:41:28.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.359+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af5ca0 con 0x555a56af2c30 2024-09-16T01:41:28.363 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:41:28.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.362+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v98'244 uv244 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56af2c30 2024-09-16T01:41:28.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.362+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af64f0 con 0x555a56af2c30 2024-09-16T01:41:28.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.364+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v98'245 uv245 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56af2c30 2024-09-16T01:41:28.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.364+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af6890 con 0x555a56af2c30 2024-09-16T01:41:28.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.367+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v98'246 uv246 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56af2c30 2024-09-16T01:41:28.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.367+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af6c30 con 0x555a56af2c30 2024-09-16T01:41:28.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.368+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v98'247 uv229 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56af2c30 2024-09-16T01:41:28.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.368+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) -- 0x555a56af6fd0 con 0x555a56af2c30 2024-09-16T01:41:28.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.369+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v98'248 uv248 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56af2c30 2024-09-16T01:41:28.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.369+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af7370 con 0x555a56af2c30 2024-09-16T01:41:28.371 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:41:28.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.370+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v98'249 uv249 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56af2c30 2024-09-16T01:41:28.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.370+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af7370 con 0x555a56af2c30 2024-09-16T01:41:28.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.371+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v98'250 uv250 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56af2c30 2024-09-16T01:41:28.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.371+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af7370 con 0x555a56af2c30 2024-09-16T01:41:28.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.372+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v98'251 uv251 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56af2c30 2024-09-16T01:41:28.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.372+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af7370 con 0x555a56af2c30 2024-09-16T01:41:28.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.373+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v98'252 uv234 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56af2c30 2024-09-16T01:41:28.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.373+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) -- 0x555a56af7370 con 0x555a56af2c30 2024-09-16T01:41:28.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.374+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v98'253 uv253 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56af2c30 2024-09-16T01:41:28.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.374+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af7370 con 0x555a56af2c30 2024-09-16T01:41:28.378 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:41:28.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.375+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v98'254 uv254 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56af2c30 2024-09-16T01:41:28.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.375+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56afb7f0 con 0x555a56af2c30 2024-09-16T01:41:28.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.376+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v98'255 uv255 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56af2c30 2024-09-16T01:41:28.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.376+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56afb7f0 con 0x555a56af2c30 2024-09-16T01:41:28.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.377+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v98'256 uv256 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56af2c30 2024-09-16T01:41:28.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.377+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af7370 con 0x555a56af2c30 2024-09-16T01:41:28.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.378+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v98'257 uv240 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56af2c30 2024-09-16T01:41:28.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.378+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) -- 0x555a56af7370 con 0x555a56af2c30 2024-09-16T01:41:28.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.380+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v98'258 uv258 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56af2c30 2024-09-16T01:41:28.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.380+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af7370 con 0x555a56af2c30 2024-09-16T01:41:28.382 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:41:28.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.381+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v98'259 uv259 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56af2c30 2024-09-16T01:41:28.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.381+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af8380 con 0x555a56af2c30 2024-09-16T01:41:28.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.383+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v98'260 uv260 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56af2c30 2024-09-16T01:41:28.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.383+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af8380 con 0x555a56af2c30 2024-09-16T01:41:28.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.384+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v98'261 uv261 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56af2c30 2024-09-16T01:41:28.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.384+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af7370 con 0x555a56af2c30 2024-09-16T01:41:28.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.385+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v98'262 uv262 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56af2c30 2024-09-16T01:41:28.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.385+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af7370 con 0x555a56a31550 2024-09-16T01:41:28.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.386+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v98'63 uv62 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.386+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) -- 0x555a56af7370 con 0x555a56a31550 2024-09-16T01:41:28.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.387+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v98'64 uv64 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.387+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af7370 con 0x555a56a31550 2024-09-16T01:41:28.389 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:41:28.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.388+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v98'65 uv65 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.388+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af8380 con 0x555a56a31550 2024-09-16T01:41:28.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.389+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v98'66 uv66 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.389+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af8380 con 0x555a56a31550 2024-09-16T01:41:28.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.389+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v98'67 uv67 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.390+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af7370 con 0x555a56a31550 2024-09-16T01:41:28.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.390+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v98'68 uv68 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.390+0000 7f93b6b3ff40 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x555a56af7880 0x555a56af9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.390+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56afa370 con 0x555a56af7880 2024-09-16T01:41:28.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.391+0000 7f93b3e34640 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x555a56af7880 0x555a56af9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.391+0000 7f93b3e34640 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x555a56af7880 0x555a56af9ca0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:28.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.392+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v98'51 uv50 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56af7880 2024-09-16T01:41:28.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.392+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) -- 0x555a56afa760 con 0x555a56af7880 2024-09-16T01:41:28.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.393+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v98'52 uv52 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56af7880 2024-09-16T01:41:28.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.393+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56afa760 con 0x555a56af7880 2024-09-16T01:41:28.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.394+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v98'53 uv53 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56af7880 2024-09-16T01:41:28.395 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:41:28.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.394+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56afa760 con 0x555a56af7880 2024-09-16T01:41:28.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.395+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v98'54 uv54 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56af7880 2024-09-16T01:41:28.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.395+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56afa730 con 0x555a56a31550 2024-09-16T01:41:28.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.396+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v98'63 uv62 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.396+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) -- 0x555a56afa670 con 0x555a56a31550 2024-09-16T01:41:28.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.397+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v98'64 uv64 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.397+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56afa670 con 0x555a56a31550 2024-09-16T01:41:28.399 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:41:28.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.398+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v98'65 uv65 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.398+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56afa670 con 0x555a56a31550 2024-09-16T01:41:28.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.399+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v98'66 uv66 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.399+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56afa670 con 0x555a56a31550 2024-09-16T01:41:28.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.401+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v98'67 uv67 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.401+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56afa5f0 con 0x555a56a31550 2024-09-16T01:41:28.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.402+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v98'68 uv68 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.402+0000 7f93b6b3ff40 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x555a56afaf10 0x555a56afb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.402+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56afdbd0 con 0x555a56afaf10 2024-09-16T01:41:28.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.402+0000 7f93b4e36640 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x555a56afaf10 0x555a56afb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.402+0000 7f93b4e36640 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x555a56afaf10 0x555a56afb2f0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:28.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.403+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v98'116 uv110 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.403+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) -- 0x555a56afdf70 con 0x555a56afaf10 2024-09-16T01:41:28.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.404+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v98'117 uv117 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.404+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56afe310 con 0x555a56afaf10 2024-09-16T01:41:28.406 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:41:28.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.405+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v98'118 uv118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.405+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56af60e0 con 0x555a56afaf10 2024-09-16T01:41:28.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.406+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v98'119 uv119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.406+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56affbc0 con 0x555a56afaf10 2024-09-16T01:41:28.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.407+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v98'120 uv120 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.407+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56afeb60 con 0x555a56afaf10 2024-09-16T01:41:28.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.408+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v98'121 uv121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.408+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56afef50 con 0x555a56afaf10 2024-09-16T01:41:28.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.409+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v98'122 uv115 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.409+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) -- 0x555a56aff2f0 con 0x555a56afaf10 2024-09-16T01:41:28.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.410+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v98'123 uv123 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.410+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56aff2f0 con 0x555a56afaf10 2024-09-16T01:41:28.412 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:41:28.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.411+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v98'124 uv124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.411+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56aff2f0 con 0x555a56afaf10 2024-09-16T01:41:28.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.412+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v98'125 uv125 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.412+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56aff2f0 con 0x555a56afaf10 2024-09-16T01:41:28.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.413+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v98'126 uv126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.413+0000 7f93b6b3ff40 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x555a56aff6e0 0x555a56b024d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.413+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b02a10 con 0x555a56aff6e0 2024-09-16T01:41:28.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.413+0000 7f93b4635640 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x555a56aff6e0 0x555a56b024d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.414+0000 7f93b4635640 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x555a56aff6e0 0x555a56b024d0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:28.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.415+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v98'51 uv50 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56aff6e0 2024-09-16T01:41:28.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.415+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) -- 0x555a56b02a10 con 0x555a56aff6e0 2024-09-16T01:41:28.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.416+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v98'52 uv52 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56aff6e0 2024-09-16T01:41:28.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.416+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b02e30 con 0x555a56aff6e0 2024-09-16T01:41:28.421 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:41:28.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.417+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v98'53 uv53 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56aff6e0 2024-09-16T01:41:28.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.417+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b02e30 con 0x555a56aff6e0 2024-09-16T01:41:28.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.418+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v98'54 uv54 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56aff6e0 2024-09-16T01:41:28.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.418+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b02e30 con 0x555a56aff6e0 2024-09-16T01:41:28.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.419+0000 7f93b4635640 1 -- 172.21.15.88:0/3264172619 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v98'55 uv55 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f93a4011b90 con 0x555a56aff6e0 2024-09-16T01:41:28.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.419+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b02e30 con 0x555a56a31550 2024-09-16T01:41:28.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.420+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v98'63 uv62 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.420+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) -- 0x555a56b02e30 con 0x555a56a31550 2024-09-16T01:41:28.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.421+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v98'64 uv64 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.421+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b02e30 con 0x555a56a31550 2024-09-16T01:41:28.424 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:41:28.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.422+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v98'65 uv65 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.422+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b03e40 con 0x555a56a31550 2024-09-16T01:41:28.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.422+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v98'66 uv66 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.423+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b03e40 con 0x555a56a31550 2024-09-16T01:41:28.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.423+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v98'67 uv67 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.423+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b02e30 con 0x555a56a31550 2024-09-16T01:41:28.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.424+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v98'68 uv68 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56a31550 2024-09-16T01:41:28.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.424+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b02e30 con 0x555a56af7880 2024-09-16T01:41:28.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.427+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v98'104 uv97 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56af7880 2024-09-16T01:41:28.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.427+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) -- 0x555a56b02e30 con 0x555a56af7880 2024-09-16T01:41:28.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.428+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v98'105 uv105 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56af7880 2024-09-16T01:41:28.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.428+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b02e30 con 0x555a56af7880 2024-09-16T01:41:28.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.428+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v98'106 uv106 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56af7880 2024-09-16T01:41:28.430 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:41:28.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.429+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b031f0 con 0x555a56af7880 2024-09-16T01:41:28.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.430+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v98'107 uv107 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56af7880 2024-09-16T01:41:28.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.430+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b031f0 con 0x555a56af7880 2024-09-16T01:41:28.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.430+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v98'108 uv108 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56af7880 2024-09-16T01:41:28.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.430+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b031f0 con 0x555a56af7880 2024-09-16T01:41:28.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.431+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v98'109 uv109 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56af7880 2024-09-16T01:41:28.433 INFO:teuthology.orchestra.run.smithi088.stdout:Write #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:41:28.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.431+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b031f0 con 0x555a56af7880 2024-09-16T01:41:28.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.433+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [omap-set-header] v98'110 uv110 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56af7880 2024-09-16T01:41:28.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.433+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b06000 con 0x555a56af7880 2024-09-16T01:41:28.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.434+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v98'111 uv111 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56af7880 2024-09-16T01:41:28.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.434+0000 7f93b6b3ff40 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x555a56b03bb0 0x555a56b05fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.434+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:99 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b064e0 con 0x555a56b03bb0 2024-09-16T01:41:28.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.434+0000 7f93b3e34640 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x555a56b03bb0 0x555a56b05fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.434+0000 7f93b3e34640 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x555a56b03bb0 0x555a56b05fa0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:28.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.435+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(99 mds0_sessionmap [create] v98'49 uv47 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56b03bb0 2024-09-16T01:41:28.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.435+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:100 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) -- 0x555a56b064e0 con 0x555a56b03bb0 2024-09-16T01:41:28.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.436+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v98'50 uv50 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56b03bb0 2024-09-16T01:41:28.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.436+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b064e0 con 0x555a56b03bb0 2024-09-16T01:41:28.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.437+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(101 mds0_sessionmap [create] v98'51 uv51 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56b03bb0 2024-09-16T01:41:28.439 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:41:28.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.437+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b064e0 con 0x555a56b03bb0 2024-09-16T01:41:28.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.438+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v98'52 uv52 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56b03bb0 2024-09-16T01:41:28.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.438+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b064e0 con 0x555a56af7880 2024-09-16T01:41:28.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.440+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(103 609.00000000 [create] v98'63 uv62 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56af7880 2024-09-16T01:41:28.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.440+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) -- 0x555a56b064e0 con 0x555a56af7880 2024-09-16T01:41:28.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.441+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(104 609.00000000 [delete] v98'64 uv64 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56af7880 2024-09-16T01:41:28.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.441+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b064e0 con 0x555a56af7880 2024-09-16T01:41:28.443 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:41:28.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.442+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v98'65 uv65 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56af7880 2024-09-16T01:41:28.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.442+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b074f0 con 0x555a56af7880 2024-09-16T01:41:28.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.443+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v98'66 uv66 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56af7880 2024-09-16T01:41:28.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.443+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b074f0 con 0x555a56af7880 2024-09-16T01:41:28.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.444+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v98'67 uv67 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56af7880 2024-09-16T01:41:28.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.444+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b064e0 con 0x555a56af7880 2024-09-16T01:41:28.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.445+0000 7f93b3e34640 1 -- 172.21.15.88:0/3264172619 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [omap-set-header] v98'68 uv68 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93a80547a0 con 0x555a56af7880 2024-09-16T01:41:28.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.445+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b064e0 con 0x555a56afaf10 2024-09-16T01:41:28.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.446+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(109 400.00000000 [create] v98'52 uv51 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.446+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) -- 0x555a56b064e0 con 0x555a56afaf10 2024-09-16T01:41:28.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.447+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(110 400.00000000 [delete] v98'53 uv53 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.448+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b064e0 con 0x555a56afaf10 2024-09-16T01:41:28.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.449+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(111 400.00000000 [create] v98'54 uv54 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.450 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:41:28.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.449+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b064e0 con 0x555a56afaf10 2024-09-16T01:41:28.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.450+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v98'55 uv55 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.450+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b064e0 con 0x555a56afaf10 2024-09-16T01:41:28.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.451+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v98'56 uv56 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.451+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b064e0 con 0x555a56afaf10 2024-09-16T01:41:28.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.452+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(114 100.00000000.inode [create] v98'132 uv123 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.452+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) -- 0x555a56b064e0 con 0x555a56afaf10 2024-09-16T01:41:28.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.453+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(115 100.00000000.inode [delete] v98'133 uv133 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.454+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b064e0 con 0x555a56afaf10 2024-09-16T01:41:28.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.455+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(116 100.00000000.inode [create] v98'134 uv134 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.456 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:41:28.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.455+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b064e0 con 0x555a56afaf10 2024-09-16T01:41:28.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.456+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v98'135 uv135 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.456+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b064e0 con 0x555a56afaf10 2024-09-16T01:41:28.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.457+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v98'136 uv136 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.457+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b064e0 con 0x555a56afaf10 2024-09-16T01:41:28.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.458+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(119 1.00000000 [create] v98'137 uv130 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.458+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) -- 0x555a56b064e0 con 0x555a56afaf10 2024-09-16T01:41:28.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.459+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(120 1.00000000 [delete] v98'138 uv138 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.459+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b064e0 con 0x555a56afaf10 2024-09-16T01:41:28.461 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:41:28.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.460+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(121 1.00000000 [create] v98'139 uv139 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.460+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b074f0 con 0x555a56afaf10 2024-09-16T01:41:28.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.461+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v98'140 uv140 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.461+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b074f0 con 0x555a56afaf10 2024-09-16T01:41:28.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.462+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v98'141 uv141 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.462+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b064e0 con 0x555a56afaf10 2024-09-16T01:41:28.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.463+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v98'142 uv142 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.463+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x555a56b068a0 con 0x555a56afaf10 2024-09-16T01:41:28.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.464+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v98'143 uv143 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f93ac055fc0 con 0x555a56afaf10 2024-09-16T01:41:28.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.464+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x555a56a31550 msgr2=0x555a56a4e610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:28.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.464+0000 7f93b6b3ff40 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x555a56a31550 0x555a56a4e610 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.464+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x555a56af7880 msgr2=0x555a56af9ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:28.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.464+0000 7f93b6b3ff40 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x555a56af7880 0x555a56af9ca0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.464+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x555a56b03bb0 msgr2=0x555a56b05fa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:28.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.464+0000 7f93b6b3ff40 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x555a56b03bb0 0x555a56b05fa0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.464+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x555a56af2c30 msgr2=0x555a56af5020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:28.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.464+0000 7f93b6b3ff40 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x555a56af2c30 0x555a56af5020 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.464+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 reap_dead start 2024-09-16T01:41:28.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.464+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x555a56afaf10 msgr2=0x555a56afb2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:28.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.464+0000 7f93b6b3ff40 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x555a56afaf10 0x555a56afb2f0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.464+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x555a56aff6e0 msgr2=0x555a56b024d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:28.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.464+0000 7f93b6b3ff40 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x555a56aff6e0 0x555a56b024d0 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.464+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f93a006ac90 msgr2=0x7f93a006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:28.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.464+0000 7f93b6b3ff40 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f93a006ac90 0x7f93a006d150 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f93a40017e0 tx=0x7f93a4001430 comp rx=0 tx=0).stop 2024-09-16T01:41:28.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.464+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f939c0030f0 msgr2=0x555a56a2e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:28.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.464+0000 7f93b6b3ff40 1 --2- 172.21.15.88:0/3264172619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f939c0030f0 0x555a56a2e140 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x555a56a48c40 tx=0x7f93ac003330 comp rx=0 tx=0).stop 2024-09-16T01:41:28.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.464+0000 7f93b4e36640 1 -- 172.21.15.88:0/3264172619 reap_dead start 2024-09-16T01:41:28.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.464+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 shutdown_connections 2024-09-16T01:41:28.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.464+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 >> 172.21.15.88:0/3264172619 conn(0x555a56977be0 msgr2=0x555a56a37440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:28.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.464+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 shutdown_connections 2024-09-16T01:41:28.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.464+0000 7f93b6b3ff40 1 -- 172.21.15.88:0/3264172619 wait complete. 2024-09-16T01:41:28.468 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm mds0_sessionmap 2024-09-16T01:41:28.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.550+0000 7faf9bf63f40 1 Processor -- start 2024-09-16T01:41:28.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.550+0000 7faf9bf63f40 1 -- start start 2024-09-16T01:41:28.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.550+0000 7faf9bf63f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555611cfdfe0 0x555611d963c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.550+0000 7faf9bf63f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555611da2d20 0x555611d9f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.550+0000 7faf9bf63f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555611da3100 0x555611cfd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.550+0000 7faf9bf63f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x555611c20510 con 0x555611cfdfe0 2024-09-16T01:41:28.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.550+0000 7faf9bf63f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x555611c79c50 con 0x555611da3100 2024-09-16T01:41:28.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.550+0000 7faf9bf63f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x555611d96900 con 0x555611da2d20 2024-09-16T01:41:28.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.550+0000 7faf99a59640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555611cfdfe0 0x555611d963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.550+0000 7faf99a59640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555611cfdfe0 0x555611d963c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60450/0 (socket says 172.21.15.88:60450) 2024-09-16T01:41:28.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.550+0000 7faf99a59640 1 -- 172.21.15.88:0/1588557152 learned_addr learned my addr 172.21.15.88:0/1588557152 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:28.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.551+0000 7faf9a25a640 1 --2- 172.21.15.88:0/1588557152 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555611da3100 0x555611cfd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.551+0000 7faf99258640 1 --2- 172.21.15.88:0/1588557152 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555611da2d20 0x555611d9f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.551+0000 7faf99a59640 1 -- 172.21.15.88:0/1588557152 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555611da2d20 msgr2=0x555611d9f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:28.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.551+0000 7faf99a59640 1 --2- 172.21.15.88:0/1588557152 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555611da2d20 0x555611d9f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.551+0000 7faf99a59640 1 -- 172.21.15.88:0/1588557152 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555611da3100 msgr2=0x555611cfd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:28.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.551+0000 7faf99a59640 1 --2- 172.21.15.88:0/1588557152 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555611da3100 0x555611cfd0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.551+0000 7faf99a59640 1 -- 172.21.15.88:0/1588557152 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555611da39d0 con 0x555611cfdfe0 2024-09-16T01:41:28.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.551+0000 7faf99a59640 1 --2- 172.21.15.88:0/1588557152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555611cfdfe0 0x555611d963c0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7faf8c025da0 tx=0x7faf8c051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e72451ade64f6587 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:28.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.551+0000 7faf8bfff640 1 -- 172.21.15.88:0/1588557152 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8c069070 con 0x555611cfdfe0 2024-09-16T01:41:28.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.551+0000 7faf8bfff640 1 -- 172.21.15.88:0/1588557152 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf8c051830 con 0x555611cfdfe0 2024-09-16T01:41:28.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.551+0000 7faf8bfff640 1 -- 172.21.15.88:0/1588557152 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8c055070 con 0x555611cfdfe0 2024-09-16T01:41:28.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.551+0000 7faf9bf63f40 1 -- 172.21.15.88:0/1588557152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555611cfdfe0 msgr2=0x555611d963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:28.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.551+0000 7faf9bf63f40 1 --2- 172.21.15.88:0/1588557152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555611cfdfe0 0x555611d963c0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7faf8c025da0 tx=0x7faf8c051000 comp rx=0 tx=0).stop 2024-09-16T01:41:28.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.551+0000 7faf9bf63f40 1 -- 172.21.15.88:0/1588557152 shutdown_connections 2024-09-16T01:41:28.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.551+0000 7faf9bf63f40 1 --2- 172.21.15.88:0/1588557152 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555611da3100 0x555611cfd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.551+0000 7faf9bf63f40 1 --2- 172.21.15.88:0/1588557152 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555611da2d20 0x555611d9f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.551+0000 7faf9bf63f40 1 --2- 172.21.15.88:0/1588557152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555611cfdfe0 0x555611d963c0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.551+0000 7faf9bf63f40 1 -- 172.21.15.88:0/1588557152 >> 172.21.15.88:0/1588557152 conn(0x555611d19750 msgr2=0x555611da6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:28.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.551+0000 7faf9bf63f40 1 -- 172.21.15.88:0/1588557152 shutdown_connections 2024-09-16T01:41:28.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.551+0000 7faf9bf63f40 1 -- 172.21.15.88:0/1588557152 wait complete. 2024-09-16T01:41:28.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.552+0000 7faf9bf63f40 1 Processor -- start 2024-09-16T01:41:28.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.552+0000 7faf9bf63f40 1 -- start start 2024-09-16T01:41:28.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.553+0000 7faf9bf63f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555611cfdfe0 0x555611cf9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.553+0000 7faf9bf63f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555611da2d20 0x555611d9f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.553+0000 7faf9bf63f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555611da3100 0x555611da1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.553+0000 7faf9bf63f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x555611cd1d40 con 0x555611da2d20 2024-09-16T01:41:28.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.553+0000 7faf9bf63f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x555611da3cb0 con 0x555611cfdfe0 2024-09-16T01:41:28.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.553+0000 7faf9bf63f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x555611cd2020 con 0x555611da3100 2024-09-16T01:41:28.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.553+0000 7faf99258640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555611da2d20 0x555611d9f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.553+0000 7faf99258640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555611da2d20 0x555611d9f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60458/0 (socket says 172.21.15.88:60458) 2024-09-16T01:41:28.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.553+0000 7faf99258640 1 -- 172.21.15.88:0/1802415638 learned_addr learned my addr 172.21.15.88:0/1802415638 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:28.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.553+0000 7faf9a25a640 1 --2- 172.21.15.88:0/1802415638 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555611da3100 0x555611da1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.553+0000 7faf99a59640 1 --2- 172.21.15.88:0/1802415638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555611cfdfe0 0x555611cf9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.553+0000 7faf99258640 1 -- 172.21.15.88:0/1802415638 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555611da3100 msgr2=0x555611da1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:28.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.553+0000 7faf99258640 1 --2- 172.21.15.88:0/1802415638 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x555611da3100 0x555611da1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.553+0000 7faf99258640 1 -- 172.21.15.88:0/1802415638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555611cfdfe0 msgr2=0x555611cf9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:28.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.553+0000 7faf99258640 1 --2- 172.21.15.88:0/1802415638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555611cfdfe0 0x555611cf9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.553+0000 7faf99258640 1 -- 172.21.15.88:0/1802415638 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555611d94d50 con 0x555611da2d20 2024-09-16T01:41:28.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.553+0000 7faf99a59640 1 --2- 172.21.15.88:0/1802415638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555611cfdfe0 0x555611cf9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:28.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.553+0000 7faf99258640 1 --2- 172.21.15.88:0/1802415638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555611da2d20 0x555611d9f8d0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7faf90003ef0 tx=0x7faf9000b590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:28.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.553+0000 7faf8affd640 1 -- 172.21.15.88:0/1802415638 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf90015020 con 0x555611da2d20 2024-09-16T01:41:28.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.553+0000 7faf8affd640 1 -- 172.21.15.88:0/1802415638 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf90011660 con 0x555611da2d20 2024-09-16T01:41:28.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.553+0000 7faf9bf63f40 1 -- 172.21.15.88:0/1802415638 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x555611cf9fa0 con 0x555611da2d20 2024-09-16T01:41:28.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.553+0000 7faf8affd640 1 -- 172.21.15.88:0/1802415638 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf90011a50 con 0x555611da2d20 2024-09-16T01:41:28.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.554+0000 7faf9bf63f40 1 -- 172.21.15.88:0/1802415638 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555611cf9980 con 0x555611da2d20 2024-09-16T01:41:28.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.555+0000 7faf8affd640 1 -- 172.21.15.88:0/1802415638 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7faf90018020 con 0x555611da2d20 2024-09-16T01:41:28.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.555+0000 7faf8affd640 1 --2- 172.21.15.88:0/1802415638 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faf7c06ac90 0x7faf7c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.555+0000 7faf99a59640 1 --2- 172.21.15.88:0/1802415638 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faf7c06ac90 0x7faf7c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.555+0000 7faf8affd640 1 -- 172.21.15.88:0/1802415638 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 6684+0+0 (secure 0 0 0) 0x7faf9008a900 con 0x555611da2d20 2024-09-16T01:41:28.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.555+0000 7faf9bf63f40 1 --2- 172.21.15.88:0/1802415638 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x555611cfa440 0x555611d13b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.555+0000 7faf9bf63f40 1 -- 172.21.15.88:0/1802415638 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:1 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) -- 0x555611d9d640 con 0x555611cfa440 2024-09-16T01:41:28.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.556+0000 7faf9a25a640 1 --2- 172.21.15.88:0/1802415638 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x555611cfa440 0x555611d13b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.556+0000 7faf99a59640 1 --2- 172.21.15.88:0/1802415638 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faf7c06ac90 0x7faf7c06d150 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7faf8c000c00 tx=0x7faf8c053000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:28.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.556+0000 7faf9a25a640 1 --2- 172.21.15.88:0/1802415638 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x555611cfa440 0x555611d13b90 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:28.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.557+0000 7faf9a25a640 1 -- 172.21.15.88:0/1802415638 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(1 mds0_sessionmap [delete] v98'53 uv53 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7faf94010630 con 0x555611cfa440 2024-09-16T01:41:28.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.557+0000 7faf9bf63f40 1 -- 172.21.15.88:0/1802415638 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x555611cfa440 msgr2=0x555611d13b90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:28.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.557+0000 7faf9bf63f40 1 --2- 172.21.15.88:0/1802415638 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x555611cfa440 0x555611d13b90 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.557+0000 7faf9bf63f40 1 -- 172.21.15.88:0/1802415638 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faf7c06ac90 msgr2=0x7faf7c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:28.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.557+0000 7faf9bf63f40 1 --2- 172.21.15.88:0/1802415638 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faf7c06ac90 0x7faf7c06d150 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7faf8c000c00 tx=0x7faf8c053000 comp rx=0 tx=0).stop 2024-09-16T01:41:28.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.557+0000 7faf9bf63f40 1 -- 172.21.15.88:0/1802415638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555611da2d20 msgr2=0x555611d9f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:28.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.557+0000 7faf9bf63f40 1 --2- 172.21.15.88:0/1802415638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555611da2d20 0x555611d9f8d0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7faf90003ef0 tx=0x7faf9000b590 comp rx=0 tx=0).stop 2024-09-16T01:41:28.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.557+0000 7faf9a25a640 1 -- 172.21.15.88:0/1802415638 reap_dead start 2024-09-16T01:41:28.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.557+0000 7faf9bf63f40 1 -- 172.21.15.88:0/1802415638 shutdown_connections 2024-09-16T01:41:28.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.557+0000 7faf9bf63f40 1 -- 172.21.15.88:0/1802415638 >> 172.21.15.88:0/1802415638 conn(0x555611d19750 msgr2=0x555611d19b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:28.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.557+0000 7faf9bf63f40 1 -- 172.21.15.88:0/1802415638 shutdown_connections 2024-09-16T01:41:28.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.557+0000 7faf9bf63f40 1 -- 172.21.15.88:0/1802415638 wait complete. 2024-09-16T01:41:28.563 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:41:28.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.702+0000 7f4f4f2c3640 1 Processor -- start 2024-09-16T01:41:28.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.703+0000 7f4f4f2c3640 1 -- start start 2024-09-16T01:41:28.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.703+0000 7f4f4f2c3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4f480b6290 0x7f4f48071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.704+0000 7f4f4f2c3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f48072530 0x7f4f4815bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.704+0000 7f4f4f2c3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f480ccff0 con 0x7f4f4815b480 2024-09-16T01:41:28.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.704+0000 7f4f4f2c3640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f4f480cd2d0 con 0x7f4f480b6290 2024-09-16T01:41:28.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.704+0000 7f4f4f2c3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4f480cd5b0 con 0x7f4f48072530 2024-09-16T01:41:28.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.704+0000 7f4f4c85e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4f480b6290 0x7f4f48071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.704+0000 7f4f4d05f640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4f4815b480 0x7f4f480b5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59510/0 (socket says 172.21.15.88:59510) 2024-09-16T01:41:28.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.704+0000 7f4f4d05f640 1 -- 172.21.15.88:0/1097301640 learned_addr learned my addr 172.21.15.88:0/1097301640 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:28.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.704+0000 7f4f4d860640 1 --2- 172.21.15.88:0/1097301640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f48072530 0x7f4f4815bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.704+0000 7f4f4c85e640 1 -- 172.21.15.88:0/1097301640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f48072530 msgr2=0x7f4f4815bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:28.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.704+0000 7f4f4c85e640 1 --2- 172.21.15.88:0/1097301640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f48072530 0x7f4f4815bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.704+0000 7f4f4c85e640 1 -- 172.21.15.88:0/1097301640 >> v1:172.21.15.88:6789/0 conn(0x7f4f4815b480 legacy=0x7f4f480b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:28.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.704+0000 7f4f4c85e640 1 -- 172.21.15.88:0/1097301640 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f480cd890 con 0x7f4f480b6290 2024-09-16T01:41:28.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.705+0000 7f4f4c85e640 1 --2- 172.21.15.88:0/1097301640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f480b6290 0x7f4f48071ff0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f4f380097a0 tx=0x7f4f3802a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57785ce26054214b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:28.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.705+0000 7f4f3ffff640 1 -- 172.21.15.88:0/1097301640 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f38037070 con 0x7f4f480b6290 2024-09-16T01:41:28.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.705+0000 7f4f3ffff640 1 -- 172.21.15.88:0/1097301640 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f3802ac90 con 0x7f4f480b6290 2024-09-16T01:41:28.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.705+0000 7f4f3ffff640 1 -- 172.21.15.88:0/1097301640 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f38032660 con 0x7f4f480b6290 2024-09-16T01:41:28.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.705+0000 7f4f4f2c3640 1 -- 172.21.15.88:0/1097301640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f480b6290 msgr2=0x7f4f48071ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:28.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.705+0000 7f4f4f2c3640 1 --2- 172.21.15.88:0/1097301640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f480b6290 0x7f4f48071ff0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f4f380097a0 tx=0x7f4f3802a000 comp rx=0 tx=0).stop 2024-09-16T01:41:28.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.705+0000 7f4f4f2c3640 1 -- 172.21.15.88:0/1097301640 shutdown_connections 2024-09-16T01:41:28.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.705+0000 7f4f4f2c3640 1 --2- 172.21.15.88:0/1097301640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f48072530 0x7f4f4815bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.705+0000 7f4f4f2c3640 1 --2- 172.21.15.88:0/1097301640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f480b6290 0x7f4f48071ff0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.705+0000 7f4f4f2c3640 1 -- 172.21.15.88:0/1097301640 >> 172.21.15.88:0/1097301640 conn(0x7f4f4806d0f0 msgr2=0x7f4f4806d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:28.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.706+0000 7f4f4f2c3640 1 -- 172.21.15.88:0/1097301640 shutdown_connections 2024-09-16T01:41:28.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.706+0000 7f4f4f2c3640 1 -- 172.21.15.88:0/1097301640 wait complete. 2024-09-16T01:41:28.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.707+0000 7f4f4f2c3640 1 Processor -- start 2024-09-16T01:41:28.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.707+0000 7f4f4f2c3640 1 -- start start 2024-09-16T01:41:28.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.707+0000 7f4f4f2c3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f48072530 0x7f4f481480c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.707+0000 7f4f4f2c3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f4815b480 0x7f4f48156950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.707+0000 7f4f4f2c3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f48157430 0x7f4f4815c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.707+0000 7f4f4f2c3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4f4815b9e0 con 0x7f4f4815b480 2024-09-16T01:41:28.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.707+0000 7f4f4f2c3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4f4815b860 con 0x7f4f48072530 2024-09-16T01:41:28.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.707+0000 7f4f4f2c3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4f4815c1a0 con 0x7f4f48157430 2024-09-16T01:41:28.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.707+0000 7f4f4c85e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f4815b480 0x7f4f48156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.708+0000 7f4f4c85e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f4815b480 0x7f4f48156950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60486/0 (socket says 172.21.15.88:60486) 2024-09-16T01:41:28.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.708+0000 7f4f4c85e640 1 -- 172.21.15.88:0/3133848120 learned_addr learned my addr 172.21.15.88:0/3133848120 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:28.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.708+0000 7f4f4d860640 1 --2- 172.21.15.88:0/3133848120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f48157430 0x7f4f4815c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.708+0000 7f4f4d05f640 1 --2- 172.21.15.88:0/3133848120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f48072530 0x7f4f481480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.708+0000 7f4f4c85e640 1 -- 172.21.15.88:0/3133848120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f48157430 msgr2=0x7f4f4815c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:28.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.708+0000 7f4f4c85e640 1 --2- 172.21.15.88:0/3133848120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f48157430 0x7f4f4815c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.708+0000 7f4f4c85e640 1 -- 172.21.15.88:0/3133848120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f48072530 msgr2=0x7f4f481480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:28.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.708+0000 7f4f4c85e640 1 --2- 172.21.15.88:0/3133848120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f48072530 0x7f4f481480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:28.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.708+0000 7f4f4c85e640 1 -- 172.21.15.88:0/3133848120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f48155b90 con 0x7f4f4815b480 2024-09-16T01:41:28.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.708+0000 7f4f4d860640 1 --2- 172.21.15.88:0/3133848120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f48157430 0x7f4f4815c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:28.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.708+0000 7f4f4c85e640 1 --2- 172.21.15.88:0/3133848120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f4815b480 0x7f4f48156950 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f4f380098c0 tx=0x7f4f380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:28.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.708+0000 7f4f3dffb640 1 -- 172.21.15.88:0/3133848120 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f38037040 con 0x7f4f4815b480 2024-09-16T01:41:28.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.708+0000 7f4f3dffb640 1 -- 172.21.15.88:0/3133848120 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f3803b430 con 0x7f4f4815b480 2024-09-16T01:41:28.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.708+0000 7f4f3dffb640 1 -- 172.21.15.88:0/3133848120 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3803b800 con 0x7f4f4815b480 2024-09-16T01:41:28.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.708+0000 7f4f4f2c3640 1 -- 172.21.15.88:0/3133848120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f48155e70 con 0x7f4f4815b480 2024-09-16T01:41:28.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.708+0000 7f4f4f2c3640 1 -- 172.21.15.88:0/3133848120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f48156430 con 0x7f4f4815b480 2024-09-16T01:41:28.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.709+0000 7f4f4f2c3640 1 -- 172.21.15.88:0/3133848120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f4815bc60 con 0x7f4f4815b480 2024-09-16T01:41:28.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.710+0000 7f4f3dffb640 1 -- 172.21.15.88:0/3133848120 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4f38044020 con 0x7f4f4815b480 2024-09-16T01:41:28.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.710+0000 7f4f3dffb640 1 --2- 172.21.15.88:0/3133848120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f2c06c2b0 0x7f4f2c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:28.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.710+0000 7f4f3dffb640 1 -- 172.21.15.88:0/3133848120 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 6684+0+0 (secure 0 0 0) 0x7f4f380ab430 con 0x7f4f4815b480 2024-09-16T01:41:28.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.710+0000 7f4f4d05f640 1 --2- 172.21.15.88:0/3133848120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f2c06c2b0 0x7f4f2c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:28.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.711+0000 7f4f4d05f640 1 --2- 172.21.15.88:0/3133848120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f2c06c2b0 0x7f4f2c06e770 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f4f44000f60 tx=0x7f4f44007c60 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:28.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.714+0000 7f4f3dffb640 1 -- 172.21.15.88:0/3133848120 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4f3804b020 con 0x7f4f4815b480 2024-09-16T01:41:28.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:28.918+0000 7f4f4f2c3640 1 -- 172.21.15.88:0/3133848120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f4f48156600 con 0x7f4f4815b480 2024-09-16T01:41:29.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.267+0000 7f4f3dffb640 1 -- 172.21.15.88:0/3133848120 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v139) ==== 162+0+0 (secure 0 0 0) 0x7f4f38079860 con 0x7f4f4815b480 2024-09-16T01:41:29.269 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:41:29.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.270+0000 7f4f4f2c3640 1 -- 172.21.15.88:0/3133848120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f2c06c2b0 msgr2=0x7f4f2c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:29.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.270+0000 7f4f4f2c3640 1 --2- 172.21.15.88:0/3133848120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f2c06c2b0 0x7f4f2c06e770 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f4f44000f60 tx=0x7f4f44007c60 comp rx=0 tx=0).stop 2024-09-16T01:41:29.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.270+0000 7f4f4f2c3640 1 -- 172.21.15.88:0/3133848120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f4815b480 msgr2=0x7f4f48156950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:29.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.270+0000 7f4f4f2c3640 1 --2- 172.21.15.88:0/3133848120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f4815b480 0x7f4f48156950 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f4f380098c0 tx=0x7f4f380046c0 comp rx=0 tx=0).stop 2024-09-16T01:41:29.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.270+0000 7f4f4f2c3640 1 -- 172.21.15.88:0/3133848120 shutdown_connections 2024-09-16T01:41:29.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.270+0000 7f4f4f2c3640 1 --2- 172.21.15.88:0/3133848120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4f2c06c2b0 0x7f4f2c06e770 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:29.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.270+0000 7f4f4f2c3640 1 --2- 172.21.15.88:0/3133848120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4f48157430 0x7f4f4815c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:29.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.270+0000 7f4f4f2c3640 1 --2- 172.21.15.88:0/3133848120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f4815b480 0x7f4f48156950 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:29.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.270+0000 7f4f4f2c3640 1 --2- 172.21.15.88:0/3133848120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f48072530 0x7f4f481480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:29.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.270+0000 7f4f4f2c3640 1 -- 172.21.15.88:0/3133848120 >> 172.21.15.88:0/3133848120 conn(0x7f4f4806d0f0 msgr2=0x7f4f480cc590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:29.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.270+0000 7f4f4f2c3640 1 -- 172.21.15.88:0/3133848120 shutdown_connections 2024-09-16T01:41:29.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.270+0000 7f4f4f2c3640 1 -- 172.21.15.88:0/3133848120 wait complete. 2024-09-16T01:41:29.275 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:41:29.274+0000 7f8fff31e640 -1 mds.0.sessionmap _load_finish got (2) No such file or directory 2024-09-16T01:41:29.275 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:41:29.274+0000 7f8fff31e640 -1 log_channel(cluster) log [ERR] : error reading sessionmap 'mds0_sessionmap' -2 ((2) No such file or directory) 2024-09-16T01:41:29.276 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:41:29.275+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:41:29.276 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:41:29.275+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:41:29.286 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:29.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.384+0000 7f3f70366640 1 Processor -- start 2024-09-16T01:41:29.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.384+0000 7f3f70366640 1 -- start start 2024-09-16T01:41:29.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.385+0000 7f3f70366640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3f680d3310 0x7f3f680d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:29.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.385+0000 7f3f70366640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f680d1650 0x7f3f680b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:29.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.385+0000 7f3f70366640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f6815ea40 con 0x7f3f6815e660 2024-09-16T01:41:29.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.385+0000 7f3f70366640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3f680d1a90 con 0x7f3f680d3310 2024-09-16T01:41:29.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.385+0000 7f3f70366640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3f680d3870 con 0x7f3f680d1650 2024-09-16T01:41:29.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.385+0000 7f3f6e102640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3f6815e660 0x7f3f680cf730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59516/0 (socket says 172.21.15.88:59516) 2024-09-16T01:41:29.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.385+0000 7f3f6e102640 1 -- 172.21.15.88:0/1401813203 learned_addr learned my addr 172.21.15.88:0/1401813203 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:29.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6e903640 1 --2- 172.21.15.88:0/1401813203 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f680d1650 0x7f3f680b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:29.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6d901640 1 --2- 172.21.15.88:0/1401813203 >> v2:172.21.15.154:3300/0 conn(0x7f3f680d3310 0x7f3f680d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:29.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6d100640 1 -- 172.21.15.88:0/1401813203 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1711357427 0 0) 0x7f3f6815ea40 con 0x7f3f6815e660 2024-09-16T01:41:29.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6d100640 1 -- 172.21.15.88:0/1401813203 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f50003520 con 0x7f3f6815e660 2024-09-16T01:41:29.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6d100640 1 -- 172.21.15.88:0/1401813203 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1745344413 0 0) 0x7f3f50003520 con 0x7f3f6815e660 2024-09-16T01:41:29.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6d100640 1 -- 172.21.15.88:0/1401813203 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f680d1650 msgr2=0x7f3f680b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:29.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6d100640 1 --2- 172.21.15.88:0/1401813203 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f680d1650 0x7f3f680b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:29.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6d100640 1 -- 172.21.15.88:0/1401813203 >> v2:172.21.15.154:3300/0 conn(0x7f3f680d3310 msgr2=0x7f3f680d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:29.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6d100640 1 --2- 172.21.15.88:0/1401813203 >> v2:172.21.15.154:3300/0 conn(0x7f3f680d3310 0x7f3f680d1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:29.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6d100640 1 -- 172.21.15.88:0/1401813203 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f680d3b50 con 0x7f3f6815e660 2024-09-16T01:41:29.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6d100640 1 -- 172.21.15.88:0/1401813203 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f3f58001670 con 0x7f3f6815e660 2024-09-16T01:41:29.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6d100640 1 -- 172.21.15.88:0/1401813203 >> v1:172.21.15.88:6789/0 conn(0x7f3f6815e660 legacy=0x7f3f680cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:29.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6d100640 1 --2- 172.21.15.88:0/1401813203 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f680d1650 0x7f3f500039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:29.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6d100640 1 --2- 172.21.15.88:0/1401813203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f50003f30 0x7f3f50004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:29.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6d100640 1 --2- 172.21.15.88:0/1401813203 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f50004d40 0x7f3f500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:29.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6d100640 1 -- 172.21.15.88:0/1401813203 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3f680d1a90 con 0x7f3f50004d40 2024-09-16T01:41:29.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6d100640 1 -- 172.21.15.88:0/1401813203 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3f680d3870 con 0x7f3f50003f30 2024-09-16T01:41:29.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6d100640 1 -- 172.21.15.88:0/1401813203 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3f5000b9c0 con 0x7f3f680d1650 2024-09-16T01:41:29.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6e903640 1 --2- 172.21.15.88:0/1401813203 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f50004d40 0x7f3f500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:29.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6e102640 1 --2- 172.21.15.88:0/1401813203 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f680d1650 0x7f3f500039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:29.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6d901640 1 --2- 172.21.15.88:0/1401813203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f50003f30 0x7f3f50004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:29.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6e903640 1 -- 172.21.15.88:0/1401813203 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f680d1650 msgr2=0x7f3f500039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:29.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6e903640 1 --2- 172.21.15.88:0/1401813203 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f680d1650 0x7f3f500039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:29.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6e903640 1 -- 172.21.15.88:0/1401813203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f50003f30 msgr2=0x7f3f50004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:29.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6e903640 1 --2- 172.21.15.88:0/1401813203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f50003f30 0x7f3f50004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:29.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6e903640 1 -- 172.21.15.88:0/1401813203 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f50003520 con 0x7f3f50004d40 2024-09-16T01:41:29.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.386+0000 7f3f6d901640 1 --2- 172.21.15.88:0/1401813203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f50003f30 0x7f3f50004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:29.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.387+0000 7f3f6e903640 1 --2- 172.21.15.88:0/1401813203 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f50004d40 0x7f3f500051c0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f3f60009ac0 tx=0x7f3f50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48121a6018bc1046 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:29.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.387+0000 7f3f6d100640 1 -- 172.21.15.88:0/1401813203 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f60036070 con 0x7f3f50004d40 2024-09-16T01:41:29.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.387+0000 7f3f6d100640 1 -- 172.21.15.88:0/1401813203 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f60031400 con 0x7f3f50004d40 2024-09-16T01:41:29.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.387+0000 7f3f6d100640 1 -- 172.21.15.88:0/1401813203 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f60009690 con 0x7f3f50004d40 2024-09-16T01:41:29.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.388+0000 7f3f70366640 1 -- 172.21.15.88:0/1401813203 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f50004d40 msgr2=0x7f3f500051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:29.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.388+0000 7f3f70366640 1 --2- 172.21.15.88:0/1401813203 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f50004d40 0x7f3f500051c0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f3f60009ac0 tx=0x7f3f50005830 comp rx=0 tx=0).stop 2024-09-16T01:41:29.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.388+0000 7f3f6e903640 1 -- 172.21.15.88:0/1401813203 reap_dead start 2024-09-16T01:41:29.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.388+0000 7f3f70366640 1 -- 172.21.15.88:0/1401813203 shutdown_connections 2024-09-16T01:41:29.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.388+0000 7f3f70366640 1 -- 172.21.15.88:0/1401813203 >> 172.21.15.88:0/1401813203 conn(0x7f3f680b6410 msgr2=0x7f3f680b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:29.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.388+0000 7f3f70366640 1 -- 172.21.15.88:0/1401813203 shutdown_connections 2024-09-16T01:41:29.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.388+0000 7f3f70366640 1 -- 172.21.15.88:0/1401813203 wait complete. 2024-09-16T01:41:29.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.389+0000 7f3f70366640 1 Processor -- start 2024-09-16T01:41:29.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.389+0000 7f3f70366640 1 -- start start 2024-09-16T01:41:29.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.390+0000 7f3f70366640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f50004d40 0x7f3f681551d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:29.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.390+0000 7f3f70366640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f681610c0 0x7f3f68158550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:29.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.390+0000 7f3f70366640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f68155b30 0x7f3f68155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:29.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.390+0000 7f3f70366640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3f680d43f0 con 0x7f3f50004d40 2024-09-16T01:41:29.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.390+0000 7f3f70366640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3f680d4110 con 0x7f3f681610c0 2024-09-16T01:41:29.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.390+0000 7f3f70366640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3f680b7b70 con 0x7f3f68155b30 2024-09-16T01:41:29.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.390+0000 7f3f6e102640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f50004d40 0x7f3f681551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:29.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.390+0000 7f3f6e102640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f50004d40 0x7f3f681551d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60512/0 (socket says 172.21.15.88:60512) 2024-09-16T01:41:29.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.390+0000 7f3f6e102640 1 -- 172.21.15.88:0/2955870779 learned_addr learned my addr 172.21.15.88:0/2955870779 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:29.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.390+0000 7f3f6e903640 1 --2- 172.21.15.88:0/2955870779 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f68155b30 0x7f3f68155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:29.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.390+0000 7f3f6e102640 1 -- 172.21.15.88:0/2955870779 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f68155b30 msgr2=0x7f3f68155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:29.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.390+0000 7f3f6e102640 1 --2- 172.21.15.88:0/2955870779 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f68155b30 0x7f3f68155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:29.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.390+0000 7f3f6e102640 1 -- 172.21.15.88:0/2955870779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f681610c0 msgr2=0x7f3f68158550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:29.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.390+0000 7f3f6e102640 1 --2- 172.21.15.88:0/2955870779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f681610c0 0x7f3f68158550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:29.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.390+0000 7f3f6e102640 1 -- 172.21.15.88:0/2955870779 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f68160020 con 0x7f3f50004d40 2024-09-16T01:41:29.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.390+0000 7f3f6e102640 1 --2- 172.21.15.88:0/2955870779 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f50004d40 0x7f3f681551d0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f3f58007340 tx=0x7f3f58007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:29.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.390+0000 7f3f66ffd640 1 -- 172.21.15.88:0/2955870779 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5800f9a0 con 0x7f3f50004d40 2024-09-16T01:41:29.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.390+0000 7f3f66ffd640 1 -- 172.21.15.88:0/2955870779 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f58018070 con 0x7f3f50004d40 2024-09-16T01:41:29.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.390+0000 7f3f70366640 1 -- 172.21.15.88:0/2955870779 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f68160300 con 0x7f3f50004d40 2024-09-16T01:41:29.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.390+0000 7f3f66ffd640 1 -- 172.21.15.88:0/2955870779 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5800fd80 con 0x7f3f50004d40 2024-09-16T01:41:29.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.390+0000 7f3f70366640 1 -- 172.21.15.88:0/2955870779 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f681608c0 con 0x7f3f50004d40 2024-09-16T01:41:29.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.391+0000 7f3f70366640 1 -- 172.21.15.88:0/2955870779 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f680d1110 con 0x7f3f50004d40 2024-09-16T01:41:29.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.392+0000 7f3f66ffd640 1 -- 172.21.15.88:0/2955870779 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3f5801d460 con 0x7f3f50004d40 2024-09-16T01:41:29.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.392+0000 7f3f66ffd640 1 --2- 172.21.15.88:0/2955870779 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3f5406c2e0 0x7f3f5406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:29.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.392+0000 7f3f66ffd640 1 -- 172.21.15.88:0/2955870779 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 6770+0+0 (secure 0 0 0) 0x7f3f5808b9b0 con 0x7f3f50004d40 2024-09-16T01:41:29.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.392+0000 7f3f6d901640 1 --2- 172.21.15.88:0/2955870779 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3f5406c2e0 0x7f3f5406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:29.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.393+0000 7f3f6d901640 1 --2- 172.21.15.88:0/2955870779 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3f5406c2e0 0x7f3f5406e7a0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f3f680d1440 tx=0x7f3f5c009750 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:29.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.397+0000 7f3f66ffd640 1 -- 172.21.15.88:0/2955870779 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3f58059cf0 con 0x7f3f50004d40 2024-09-16T01:41:29.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.597+0000 7f3f70366640 1 -- 172.21.15.88:0/2955870779 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f680b4340 con 0x7f3f50004d40 2024-09-16T01:41:29.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.598+0000 7f3f66ffd640 1 -- 172.21.15.88:0/2955870779 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) ==== 95+0+4391 (secure 0 0 0) 0x7f3f580595f0 con 0x7f3f50004d40 2024-09-16T01:41:29.599 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:29.600 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":141,"btime":"2024-09-16T01:41:29:275955+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8175,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1255135754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1255135754},{"type":"v1","addr":"172.21.15.88:6837","nonce":1255135754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:29.275881+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:41:29.600 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 141 2024-09-16T01:41:29.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.600+0000 7f3f70366640 1 -- 172.21.15.88:0/2955870779 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3f5406c2e0 msgr2=0x7f3f5406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:29.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.600+0000 7f3f70366640 1 --2- 172.21.15.88:0/2955870779 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3f5406c2e0 0x7f3f5406e7a0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f3f680d1440 tx=0x7f3f5c009750 comp rx=0 tx=0).stop 2024-09-16T01:41:29.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.600+0000 7f3f70366640 1 -- 172.21.15.88:0/2955870779 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f50004d40 msgr2=0x7f3f681551d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:29.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.600+0000 7f3f70366640 1 --2- 172.21.15.88:0/2955870779 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f50004d40 0x7f3f681551d0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f3f58007340 tx=0x7f3f58007800 comp rx=0 tx=0).stop 2024-09-16T01:41:29.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.600+0000 7f3f70366640 1 -- 172.21.15.88:0/2955870779 shutdown_connections 2024-09-16T01:41:29.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.600+0000 7f3f70366640 1 --2- 172.21.15.88:0/2955870779 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3f5406c2e0 0x7f3f5406e7a0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:29.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.600+0000 7f3f70366640 1 --2- 172.21.15.88:0/2955870779 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3f68155b30 0x7f3f68155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:29.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.600+0000 7f3f70366640 1 --2- 172.21.15.88:0/2955870779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f681610c0 0x7f3f68158550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:29.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.600+0000 7f3f70366640 1 --2- 172.21.15.88:0/2955870779 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f50004d40 0x7f3f681551d0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:29.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.600+0000 7f3f70366640 1 -- 172.21.15.88:0/2955870779 >> 172.21.15.88:0/2955870779 conn(0x7f3f680b6410 msgr2=0x7f3f680b9fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:29.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.600+0000 7f3f70366640 1 -- 172.21.15.88:0/2955870779 shutdown_connections 2024-09-16T01:41:29.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.600+0000 7f3f70366640 1 -- 172.21.15.88:0/2955870779 wait complete. 2024-09-16T01:41:29.612 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:41:29.613 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:29.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.710+0000 7f3a795fd640 1 Processor -- start 2024-09-16T01:41:29.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.710+0000 7f3a795fd640 1 -- start start 2024-09-16T01:41:29.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.711+0000 7f3a795fd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a7415c2b0 0x7f3a740b6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:29.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.711+0000 7f3a795fd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3a740b72f0 0x7f3a7415eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:29.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.711+0000 7f3a795fd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a7415fb10 con 0x7f3a7415f030 2024-09-16T01:41:29.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.711+0000 7f3a795fd640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f3a740b5900 con 0x7f3a740b72f0 2024-09-16T01:41:29.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.711+0000 7f3a795fd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3a7415f830 con 0x7f3a7415c2b0 2024-09-16T01:41:29.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.711+0000 7f3a727fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3a740b72f0 0x7f3a7415eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:29.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.711+0000 7f3a727fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3a740b72f0 0x7f3a7415eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60514/0 (socket says 172.21.15.88:60514) 2024-09-16T01:41:29.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.711+0000 7f3a737fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3a7415f030 0x7f3a7406f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59520/0 (socket says 172.21.15.88:59520) 2024-09-16T01:41:29.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.711+0000 7f3a727fc640 1 -- 172.21.15.88:0/867012922 learned_addr learned my addr 172.21.15.88:0/867012922 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:29.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.711+0000 7f3a72ffd640 1 --2- 172.21.15.88:0/867012922 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a7415c2b0 0x7f3a740b6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:29.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.711+0000 7f3a727fc640 1 -- 172.21.15.88:0/867012922 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a7415c2b0 msgr2=0x7f3a740b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:29.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.711+0000 7f3a727fc640 1 --2- 172.21.15.88:0/867012922 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a7415c2b0 0x7f3a740b6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:29.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.711+0000 7f3a727fc640 1 -- 172.21.15.88:0/867012922 >> v1:172.21.15.88:6789/0 conn(0x7f3a7415f030 legacy=0x7f3a7406f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:29.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.711+0000 7f3a727fc640 1 -- 172.21.15.88:0/867012922 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a7406ce30 con 0x7f3a740b72f0 2024-09-16T01:41:29.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.711+0000 7f3a71ffb640 1 -- 172.21.15.88:0/867012922 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 322945800 0 0) 0x7f3a7415fb10 con 0x7f3a7415f030 2024-09-16T01:41:29.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.711+0000 7f3a72ffd640 1 --2- 172.21.15.88:0/867012922 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a7415c2b0 0x7f3a740b6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:29.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.712+0000 7f3a727fc640 1 --2- 172.21.15.88:0/867012922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a740b72f0 0x7f3a7415eaf0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f3a68009870 tx=0x7f3a6802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e4046fde2e94293 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:29.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.712+0000 7f3a71ffb640 1 -- 172.21.15.88:0/867012922 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a68037070 con 0x7f3a740b72f0 2024-09-16T01:41:29.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.712+0000 7f3a71ffb640 1 -- 172.21.15.88:0/867012922 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a68032440 con 0x7f3a740b72f0 2024-09-16T01:41:29.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.712+0000 7f3a71ffb640 1 -- 172.21.15.88:0/867012922 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a68032810 con 0x7f3a740b72f0 2024-09-16T01:41:29.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.713+0000 7f3a795fd640 1 -- 172.21.15.88:0/867012922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a740b72f0 msgr2=0x7f3a7415eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:29.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.713+0000 7f3a795fd640 1 --2- 172.21.15.88:0/867012922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a740b72f0 0x7f3a7415eaf0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f3a68009870 tx=0x7f3a6802a270 comp rx=0 tx=0).stop 2024-09-16T01:41:29.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.713+0000 7f3a795fd640 1 -- 172.21.15.88:0/867012922 shutdown_connections 2024-09-16T01:41:29.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.713+0000 7f3a795fd640 1 --2- 172.21.15.88:0/867012922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a740b72f0 0x7f3a7415eaf0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:29.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.713+0000 7f3a795fd640 1 --2- 172.21.15.88:0/867012922 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a7415c2b0 0x7f3a740b6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:29.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.713+0000 7f3a795fd640 1 -- 172.21.15.88:0/867012922 >> 172.21.15.88:0/867012922 conn(0x7f3a740b4240 msgr2=0x7f3a740b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:29.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.713+0000 7f3a795fd640 1 -- 172.21.15.88:0/867012922 shutdown_connections 2024-09-16T01:41:29.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.713+0000 7f3a795fd640 1 -- 172.21.15.88:0/867012922 wait complete. 2024-09-16T01:41:29.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.714+0000 7f3a795fd640 1 Processor -- start 2024-09-16T01:41:29.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.714+0000 7f3a795fd640 1 -- start start 2024-09-16T01:41:29.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.715+0000 7f3a795fd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a740b72f0 0x7f3a741551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:29.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.715+0000 7f3a795fd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a7415c2b0 0x7f3a740ce320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:29.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.715+0000 7f3a795fd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a7415f030 0x7f3a741f5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:29.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.715+0000 7f3a795fd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3a740b7730 con 0x7f3a7415c2b0 2024-09-16T01:41:29.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.715+0000 7f3a795fd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7406d3f0 con 0x7f3a740b72f0 2024-09-16T01:41:29.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.715+0000 7f3a795fd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3a7415f4e0 con 0x7f3a7415f030 2024-09-16T01:41:29.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.715+0000 7f3a727fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a7415c2b0 0x7f3a740ce320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:29.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.715+0000 7f3a727fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a7415c2b0 0x7f3a740ce320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60528/0 (socket says 172.21.15.88:60528) 2024-09-16T01:41:29.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.715+0000 7f3a727fc640 1 -- 172.21.15.88:0/2553482796 learned_addr learned my addr 172.21.15.88:0/2553482796 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:29.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.715+0000 7f3a72ffd640 1 --2- 172.21.15.88:0/2553482796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a740b72f0 0x7f3a741551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:29.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.715+0000 7f3a727fc640 1 -- 172.21.15.88:0/2553482796 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a7415f030 msgr2=0x7f3a741f5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:29.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.715+0000 7f3a737fe640 1 --2- 172.21.15.88:0/2553482796 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a7415f030 0x7f3a741f5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:29.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.715+0000 7f3a727fc640 1 --2- 172.21.15.88:0/2553482796 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a7415f030 0x7f3a741f5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:29.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.715+0000 7f3a727fc640 1 -- 172.21.15.88:0/2553482796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a740b72f0 msgr2=0x7f3a741551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:29.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.715+0000 7f3a727fc640 1 --2- 172.21.15.88:0/2553482796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a740b72f0 0x7f3a741551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:29.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.715+0000 7f3a727fc640 1 -- 172.21.15.88:0/2553482796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a740ce860 con 0x7f3a7415c2b0 2024-09-16T01:41:29.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.715+0000 7f3a72ffd640 1 --2- 172.21.15.88:0/2553482796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a740b72f0 0x7f3a741551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:29.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.715+0000 7f3a727fc640 1 --2- 172.21.15.88:0/2553482796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a7415c2b0 0x7f3a740ce320 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f3a68000c00 tx=0x7f3a680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:29.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.716+0000 7f3a5b7fe640 1 -- 172.21.15.88:0/2553482796 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a68037040 con 0x7f3a7415c2b0 2024-09-16T01:41:29.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.716+0000 7f3a5b7fe640 1 -- 172.21.15.88:0/2553482796 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a68042070 con 0x7f3a7415c2b0 2024-09-16T01:41:29.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.716+0000 7f3a795fd640 1 -- 172.21.15.88:0/2553482796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a741f5640 con 0x7f3a7415c2b0 2024-09-16T01:41:29.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.716+0000 7f3a5b7fe640 1 -- 172.21.15.88:0/2553482796 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a68030bd0 con 0x7f3a7415c2b0 2024-09-16T01:41:29.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.716+0000 7f3a795fd640 1 -- 172.21.15.88:0/2553482796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a740cd1b0 con 0x7f3a7415c2b0 2024-09-16T01:41:29.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.716+0000 7f3a795fd640 1 -- 172.21.15.88:0/2553482796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a7415e0d0 con 0x7f3a7415c2b0 2024-09-16T01:41:29.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.717+0000 7f3a5b7fe640 1 -- 172.21.15.88:0/2553482796 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3a68047460 con 0x7f3a7415c2b0 2024-09-16T01:41:29.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.717+0000 7f3a5b7fe640 1 --2- 172.21.15.88:0/2553482796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a6006c2e0 0x7f3a6006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:29.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.717+0000 7f3a5b7fe640 1 -- 172.21.15.88:0/2553482796 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 6770+0+0 (secure 0 0 0) 0x7f3a680ac420 con 0x7f3a7415c2b0 2024-09-16T01:41:29.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.718+0000 7f3a72ffd640 1 --2- 172.21.15.88:0/2553482796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a6006c2e0 0x7f3a6006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:29.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.718+0000 7f3a72ffd640 1 --2- 172.21.15.88:0/2553482796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a6006c2e0 0x7f3a6006e7a0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f3a6400d430 tx=0x7f3a64007530 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:29.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.722+0000 7f3a5b7fe640 1 -- 172.21.15.88:0/2553482796 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3a7415e0d0 con 0x7f3a7415c2b0 2024-09-16T01:41:29.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.918+0000 7f3a795fd640 1 -- 172.21.15.88:0/2553482796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a7415eaf0 con 0x7f3a7415c2b0 2024-09-16T01:41:29.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.919+0000 7f3a5b7fe640 1 -- 172.21.15.88:0/2553482796 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) ==== 95+0+4391 (secure 0 0 0) 0x7f3a68045360 con 0x7f3a7415c2b0 2024-09-16T01:41:29.920 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:29.920 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":141,"btime":"2024-09-16T01:41:29:275955+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8175,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1255135754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1255135754},{"type":"v1","addr":"172.21.15.88:6837","nonce":1255135754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:29.275881+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:41:29.921 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 141 2024-09-16T01:41:29.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.921+0000 7f3a795fd640 1 -- 172.21.15.88:0/2553482796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a6006c2e0 msgr2=0x7f3a6006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:29.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.921+0000 7f3a795fd640 1 --2- 172.21.15.88:0/2553482796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a6006c2e0 0x7f3a6006e7a0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f3a6400d430 tx=0x7f3a64007530 comp rx=0 tx=0).stop 2024-09-16T01:41:29.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.921+0000 7f3a795fd640 1 -- 172.21.15.88:0/2553482796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a7415c2b0 msgr2=0x7f3a740ce320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:29.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.921+0000 7f3a795fd640 1 --2- 172.21.15.88:0/2553482796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a7415c2b0 0x7f3a740ce320 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f3a68000c00 tx=0x7f3a680046c0 comp rx=0 tx=0).stop 2024-09-16T01:41:29.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.921+0000 7f3a795fd640 1 -- 172.21.15.88:0/2553482796 shutdown_connections 2024-09-16T01:41:29.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.921+0000 7f3a795fd640 1 --2- 172.21.15.88:0/2553482796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3a6006c2e0 0x7f3a6006e7a0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:29.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.921+0000 7f3a795fd640 1 --2- 172.21.15.88:0/2553482796 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a7415f030 0x7f3a741f5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:29.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.921+0000 7f3a795fd640 1 --2- 172.21.15.88:0/2553482796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a7415c2b0 0x7f3a740ce320 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:29.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.921+0000 7f3a795fd640 1 --2- 172.21.15.88:0/2553482796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a740b72f0 0x7f3a741551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:29.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.921+0000 7f3a795fd640 1 -- 172.21.15.88:0/2553482796 >> 172.21.15.88:0/2553482796 conn(0x7f3a740b4240 msgr2=0x7f3a740b2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:29.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.921+0000 7f3a795fd640 1 -- 172.21.15.88:0/2553482796 shutdown_connections 2024-09-16T01:41:29.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:29.921+0000 7f3a795fd640 1 -- 172.21.15.88:0/2553482796 wait complete. 2024-09-16T01:41:29.933 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:41:29.933 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:30.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.031+0000 7f9193b8d640 1 Processor -- start 2024-09-16T01:41:30.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.031+0000 7f9193b8d640 1 -- start start 2024-09-16T01:41:30.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.032+0000 7f9193b8d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f918c0cf370 0x7f918c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:30.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.032+0000 7f9193b8d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f918c0b52b0 0x7f918c0b5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:30.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.032+0000 7f9193b8d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f918c15c850 con 0x7f918c15c470 2024-09-16T01:41:30.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.032+0000 7f9193b8d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f918c0cfd10 con 0x7f918c0b52b0 2024-09-16T01:41:30.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.032+0000 7f9193b8d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f918c0b4160 con 0x7f918c0cf370 2024-09-16T01:41:30.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.032+0000 7f9191929640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f918c15c470 0x7f918c0b3710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59528/0 (socket says 172.21.15.88:59528) 2024-09-16T01:41:30.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.032+0000 7f9191929640 1 -- 172.21.15.88:0/2649895171 learned_addr learned my addr 172.21.15.88:0/2649895171 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:30.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.032+0000 7f919212a640 1 --2- 172.21.15.88:0/2649895171 >> v2:172.21.15.154:3300/0 conn(0x7f918c0b52b0 0x7f918c0b5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:30.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.032+0000 7f9190927640 1 -- 172.21.15.88:0/2649895171 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4196258410 0 0) 0x7f918c15c850 con 0x7f918c15c470 2024-09-16T01:41:30.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.032+0000 7f9190927640 1 -- 172.21.15.88:0/2649895171 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9178003520 con 0x7f918c15c470 2024-09-16T01:41:30.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.032+0000 7f9191128640 1 --2- 172.21.15.88:0/2649895171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f918c0cf370 0x7f918c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:30.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.032+0000 7f9190927640 1 -- 172.21.15.88:0/2649895171 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2328750021 0 0) 0x7f9178003520 con 0x7f918c15c470 2024-09-16T01:41:30.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.032+0000 7f9190927640 1 -- 172.21.15.88:0/2649895171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f918c0cf370 msgr2=0x7f918c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:30.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.032+0000 7f9190927640 1 --2- 172.21.15.88:0/2649895171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f918c0cf370 0x7f918c0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:30.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.032+0000 7f9190927640 1 -- 172.21.15.88:0/2649895171 >> v2:172.21.15.154:3300/0 conn(0x7f918c0b52b0 msgr2=0x7f918c0b5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:30.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.032+0000 7f9190927640 1 --2- 172.21.15.88:0/2649895171 >> v2:172.21.15.154:3300/0 conn(0x7f918c0b52b0 0x7f918c0b5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:30.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.032+0000 7f9190927640 1 -- 172.21.15.88:0/2649895171 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f918c0b4440 con 0x7f918c15c470 2024-09-16T01:41:30.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.032+0000 7f9190927640 1 -- 172.21.15.88:0/2649895171 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f9188001670 con 0x7f918c15c470 2024-09-16T01:41:30.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.032+0000 7f9190927640 1 -- 172.21.15.88:0/2649895171 >> v1:172.21.15.88:6789/0 conn(0x7f918c15c470 legacy=0x7f918c0b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:30.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.032+0000 7f9190927640 1 --2- 172.21.15.88:0/2649895171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91780039b0 0x7f9178003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:30.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.032+0000 7f9190927640 1 --2- 172.21.15.88:0/2649895171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9178004800 0x7f9178004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:30.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.032+0000 7f9190927640 1 --2- 172.21.15.88:0/2649895171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f918c0cf370 0x7f91780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:30.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.032+0000 7f9190927640 1 -- 172.21.15.88:0/2649895171 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f918c0b4160 con 0x7f9178004800 2024-09-16T01:41:30.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.032+0000 7f9190927640 1 -- 172.21.15.88:0/2649895171 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f917800b9c0 con 0x7f91780039b0 2024-09-16T01:41:30.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.032+0000 7f9190927640 1 -- 172.21.15.88:0/2649895171 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f917800bb40 con 0x7f918c0cf370 2024-09-16T01:41:30.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.033+0000 7f9191128640 1 --2- 172.21.15.88:0/2649895171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9178004800 0x7f9178004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:30.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.033+0000 7f9191929640 1 --2- 172.21.15.88:0/2649895171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91780039b0 0x7f9178003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:30.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.033+0000 7f919212a640 1 --2- 172.21.15.88:0/2649895171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f918c0cf370 0x7f91780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:30.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.033+0000 7f9191128640 1 -- 172.21.15.88:0/2649895171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f918c0cf370 msgr2=0x7f91780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:30.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.033+0000 7f9191128640 1 --2- 172.21.15.88:0/2649895171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f918c0cf370 0x7f91780051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:30.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.033+0000 7f9191128640 1 -- 172.21.15.88:0/2649895171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91780039b0 msgr2=0x7f9178003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:30.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.033+0000 7f9191128640 1 --2- 172.21.15.88:0/2649895171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91780039b0 0x7f9178003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:30.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.033+0000 7f9191128640 1 -- 172.21.15.88:0/2649895171 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9178003520 con 0x7f9178004800 2024-09-16T01:41:30.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.033+0000 7f919212a640 1 --2- 172.21.15.88:0/2649895171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f918c0cf370 0x7f91780051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:30.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.033+0000 7f9191128640 1 --2- 172.21.15.88:0/2649895171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9178004800 0x7f9178004c80 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f917c009ac0 tx=0x7f9178005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=870bba2a09aaec8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:30.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.034+0000 7f9190927640 1 -- 172.21.15.88:0/2649895171 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917c036070 con 0x7f9178004800 2024-09-16T01:41:30.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.034+0000 7f9190927640 1 -- 172.21.15.88:0/2649895171 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f917c031400 con 0x7f9178004800 2024-09-16T01:41:30.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.034+0000 7f9190927640 1 -- 172.21.15.88:0/2649895171 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917c009690 con 0x7f9178004800 2024-09-16T01:41:30.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.034+0000 7f9193b8d640 1 -- 172.21.15.88:0/2649895171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9178004800 msgr2=0x7f9178004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:30.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.034+0000 7f9193b8d640 1 --2- 172.21.15.88:0/2649895171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9178004800 0x7f9178004c80 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f917c009ac0 tx=0x7f9178005830 comp rx=0 tx=0).stop 2024-09-16T01:41:30.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.034+0000 7f919212a640 1 -- 172.21.15.88:0/2649895171 reap_dead start 2024-09-16T01:41:30.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.034+0000 7f9193b8d640 1 -- 172.21.15.88:0/2649895171 shutdown_connections 2024-09-16T01:41:30.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.034+0000 7f9193b8d640 1 -- 172.21.15.88:0/2649895171 >> 172.21.15.88:0/2649895171 conn(0x7f918c06bc10 msgr2=0x7f918c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:30.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.034+0000 7f9193b8d640 1 -- 172.21.15.88:0/2649895171 shutdown_connections 2024-09-16T01:41:30.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.034+0000 7f9193b8d640 1 -- 172.21.15.88:0/2649895171 wait complete. 2024-09-16T01:41:30.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.035+0000 7f9193b8d640 1 Processor -- start 2024-09-16T01:41:30.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.035+0000 7f9193b8d640 1 -- start start 2024-09-16T01:41:30.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.036+0000 7f9193b8d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9178004800 0x7f918c14ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:30.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.036+0000 7f9193b8d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f918c13c620 0x7f918c0cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:30.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.036+0000 7f9193b8d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f918c13ca80 0x7f918c0cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:30.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.036+0000 7f9193b8d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f918c0b4d90 con 0x7f918c13ca80 2024-09-16T01:41:30.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.036+0000 7f9193b8d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f918c0b4a00 con 0x7f9178004800 2024-09-16T01:41:30.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.036+0000 7f9193b8d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f918c0b3e20 con 0x7f918c13c620 2024-09-16T01:41:30.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.036+0000 7f919212a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f918c13ca80 0x7f918c0cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:30.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.036+0000 7f919212a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f918c13ca80 0x7f918c0cd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60546/0 (socket says 172.21.15.88:60546) 2024-09-16T01:41:30.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.036+0000 7f919212a640 1 -- 172.21.15.88:0/2334778686 learned_addr learned my addr 172.21.15.88:0/2334778686 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:30.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.036+0000 7f9191929640 1 --2- 172.21.15.88:0/2334778686 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9178004800 0x7f918c14ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:30.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.036+0000 7f9191128640 1 --2- 172.21.15.88:0/2334778686 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f918c13c620 0x7f918c0cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:30.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.036+0000 7f919212a640 1 -- 172.21.15.88:0/2334778686 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f918c13c620 msgr2=0x7f918c0cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:30.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.037+0000 7f919212a640 1 --2- 172.21.15.88:0/2334778686 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f918c13c620 0x7f918c0cd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:30.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.037+0000 7f919212a640 1 -- 172.21.15.88:0/2334778686 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9178004800 msgr2=0x7f918c14ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:30.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.037+0000 7f919212a640 1 --2- 172.21.15.88:0/2334778686 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9178004800 0x7f918c14ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:30.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.037+0000 7f919212a640 1 -- 172.21.15.88:0/2334778686 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f918c152f60 con 0x7f918c13ca80 2024-09-16T01:41:30.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.037+0000 7f9191929640 1 --2- 172.21.15.88:0/2334778686 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9178004800 0x7f918c14ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:30.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.037+0000 7f9191128640 1 --2- 172.21.15.88:0/2334778686 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f918c13c620 0x7f918c0cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:30.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.037+0000 7f919212a640 1 --2- 172.21.15.88:0/2334778686 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f918c13ca80 0x7f918c0cd6a0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f918c0c6b00 tx=0x7f9184010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:30.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.037+0000 7f91827fc640 1 -- 172.21.15.88:0/2334778686 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918401a030 con 0x7f918c13ca80 2024-09-16T01:41:30.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.037+0000 7f91827fc640 1 -- 172.21.15.88:0/2334778686 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9184010da0 con 0x7f918c13ca80 2024-09-16T01:41:30.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.037+0000 7f9193b8d640 1 -- 172.21.15.88:0/2334778686 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f918c153220 con 0x7f918c13ca80 2024-09-16T01:41:30.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.037+0000 7f91827fc640 1 -- 172.21.15.88:0/2334778686 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9184003bd0 con 0x7f918c13ca80 2024-09-16T01:41:30.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.037+0000 7f9193b8d640 1 -- 172.21.15.88:0/2334778686 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f918c14b7f0 con 0x7f918c13ca80 2024-09-16T01:41:30.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.038+0000 7f9193b8d640 1 -- 172.21.15.88:0/2334778686 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f918c06e1b0 con 0x7f918c13ca80 2024-09-16T01:41:30.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.038+0000 7f91827fc640 1 -- 172.21.15.88:0/2334778686 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f918401d070 con 0x7f918c13ca80 2024-09-16T01:41:30.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.039+0000 7f91827fc640 1 --2- 172.21.15.88:0/2334778686 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f917006c2b0 0x7f917006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:30.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.039+0000 7f91827fc640 1 -- 172.21.15.88:0/2334778686 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 6770+0+0 (secure 0 0 0) 0x7f918408bee0 con 0x7f918c13ca80 2024-09-16T01:41:30.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.039+0000 7f9191929640 1 --2- 172.21.15.88:0/2334778686 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f917006c2b0 0x7f917006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:30.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.039+0000 7f9191929640 1 --2- 172.21.15.88:0/2334778686 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f917006c2b0 0x7f917006e770 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f9188001050 tx=0x7f9188009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:30.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.043+0000 7f91827fc640 1 -- 172.21.15.88:0/2334778686 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f918405a200 con 0x7f918c13ca80 2024-09-16T01:41:30.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.244+0000 7f9193b8d640 1 -- 172.21.15.88:0/2334778686 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f918c06df10 con 0x7f918c13ca80 2024-09-16T01:41:30.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.244+0000 7f91827fc640 1 -- 172.21.15.88:0/2334778686 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) ==== 95+0+4391 (secure 0 0 0) 0x7f9184059b00 con 0x7f918c13ca80 2024-09-16T01:41:30.246 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:30.246 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":141,"btime":"2024-09-16T01:41:29:275955+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8175,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1255135754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1255135754},{"type":"v1","addr":"172.21.15.88:6837","nonce":1255135754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:29.275881+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:41:30.246 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 141 2024-09-16T01:41:30.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.246+0000 7f9193b8d640 1 -- 172.21.15.88:0/2334778686 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f917006c2b0 msgr2=0x7f917006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:30.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.247+0000 7f9193b8d640 1 --2- 172.21.15.88:0/2334778686 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f917006c2b0 0x7f917006e770 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f9188001050 tx=0x7f9188009040 comp rx=0 tx=0).stop 2024-09-16T01:41:30.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.247+0000 7f9193b8d640 1 -- 172.21.15.88:0/2334778686 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f918c13ca80 msgr2=0x7f918c0cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:30.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.247+0000 7f9193b8d640 1 --2- 172.21.15.88:0/2334778686 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f918c13ca80 0x7f918c0cd6a0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f918c0c6b00 tx=0x7f9184010590 comp rx=0 tx=0).stop 2024-09-16T01:41:30.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.247+0000 7f9193b8d640 1 -- 172.21.15.88:0/2334778686 shutdown_connections 2024-09-16T01:41:30.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.247+0000 7f9193b8d640 1 --2- 172.21.15.88:0/2334778686 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f917006c2b0 0x7f917006e770 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:30.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.247+0000 7f9193b8d640 1 --2- 172.21.15.88:0/2334778686 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f918c13ca80 0x7f918c0cd6a0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:30.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.247+0000 7f9193b8d640 1 --2- 172.21.15.88:0/2334778686 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f918c13c620 0x7f918c0cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:30.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.247+0000 7f9193b8d640 1 --2- 172.21.15.88:0/2334778686 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9178004800 0x7f918c14ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:30.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.247+0000 7f9193b8d640 1 -- 172.21.15.88:0/2334778686 >> 172.21.15.88:0/2334778686 conn(0x7f918c06bc10 msgr2=0x7f918c0c5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:30.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.247+0000 7f9193b8d640 1 -- 172.21.15.88:0/2334778686 shutdown_connections 2024-09-16T01:41:30.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.247+0000 7f9193b8d640 1 -- 172.21.15.88:0/2334778686 wait complete. 2024-09-16T01:41:30.259 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete mds0_sessionmap' led to DAMAGED state 2024-09-16T01:41:30.259 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 400.00000000' 2024-09-16T01:41:30.259 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:41:30.259 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:41:30.275 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:41:30.275 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:41:30.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:30.276 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:41:30.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:41:30.331 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:41:30.332 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:41:30.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.469+0000 7ff800f6d640 1 Processor -- start 2024-09-16T01:41:30.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.469+0000 7ff800f6d640 1 -- start start 2024-09-16T01:41:30.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.470+0000 7ff800f6d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff7fc072700 0x7ff7fc166120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:30.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.470+0000 7ff800f6d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7fc0ccff0 con 0x7ff7fc15b480 2024-09-16T01:41:30.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.470+0000 7ff800f6d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7fc0cd2d0 con 0x7ff7fc0b6290 2024-09-16T01:41:30.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.470+0000 7ff800f6d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff7fc0cd5b0 con 0x7ff7fc072700 2024-09-16T01:41:30.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.470+0000 7ff7fa59c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff7fc15b480 0x7ff7fc0b5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60100/0 (socket says 172.21.15.88:60100) 2024-09-16T01:41:30.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.470+0000 7ff7fa59c640 1 -- 172.21.15.88:0/369494952 learned_addr learned my addr 172.21.15.88:0/369494952 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:30.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7fad9d640 1 --2- 172.21.15.88:0/369494952 >> v2:172.21.15.154:3300/0 conn(0x7ff7fc072700 0x7ff7fc166120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:30.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7f959a640 1 -- 172.21.15.88:0/369494952 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3674880562 0 0) 0x7ff7fc0ccff0 con 0x7ff7fc15b480 2024-09-16T01:41:30.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7f959a640 1 -- 172.21.15.88:0/369494952 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7e8003520 con 0x7ff7fc15b480 2024-09-16T01:41:30.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7f959a640 1 -- 172.21.15.88:0/369494952 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1470743685 0 0) 0x7ff7e8003520 con 0x7ff7fc15b480 2024-09-16T01:41:30.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7f959a640 1 -- 172.21.15.88:0/369494952 >> v2:172.21.15.154:3300/0 conn(0x7ff7fc072700 msgr2=0x7ff7fc166120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:30.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7f959a640 1 --2- 172.21.15.88:0/369494952 >> v2:172.21.15.154:3300/0 conn(0x7ff7fc072700 0x7ff7fc166120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:30.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7f959a640 1 -- 172.21.15.88:0/369494952 >> v1:172.21.15.154:6789/0 conn(0x7ff7fc0b6290 legacy=0x7ff7fc071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:30.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7f959a640 1 -- 172.21.15.88:0/369494952 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7fc0cd890 con 0x7ff7fc15b480 2024-09-16T01:41:30.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7f959a640 1 -- 172.21.15.88:0/369494952 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7ff7ec0019f0 con 0x7ff7fc15b480 2024-09-16T01:41:30.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7f959a640 1 -- 172.21.15.88:0/369494952 >> v1:172.21.15.88:6789/0 conn(0x7ff7fc15b480 legacy=0x7ff7fc0b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:30.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7f959a640 1 --2- 172.21.15.88:0/369494952 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7e80039b0 0x7ff7e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:30.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7f959a640 1 --2- 172.21.15.88:0/369494952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7e8004800 0x7ff7e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:30.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7f959a640 1 --2- 172.21.15.88:0/369494952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7e80051c0 0x7ff7e800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:30.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7f959a640 1 -- 172.21.15.88:0/369494952 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc0cd5b0 con 0x7ff7e8004800 2024-09-16T01:41:30.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7f959a640 1 -- 172.21.15.88:0/369494952 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff7e800c160 con 0x7ff7e80051c0 2024-09-16T01:41:30.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7f959a640 1 -- 172.21.15.88:0/369494952 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff7e800c360 con 0x7ff7e80039b0 2024-09-16T01:41:30.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7f9d9b640 1 --2- 172.21.15.88:0/369494952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7e8004800 0x7ff7e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:30.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7fa59c640 1 --2- 172.21.15.88:0/369494952 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7e80039b0 0x7ff7e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:30.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7fad9d640 1 --2- 172.21.15.88:0/369494952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7e80051c0 0x7ff7e800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:30.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7f9d9b640 1 -- 172.21.15.88:0/369494952 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7e80039b0 msgr2=0x7ff7e8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:30.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7f9d9b640 1 --2- 172.21.15.88:0/369494952 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7e80039b0 0x7ff7e8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:30.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7f9d9b640 1 -- 172.21.15.88:0/369494952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7e80051c0 msgr2=0x7ff7e800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:30.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7f9d9b640 1 --2- 172.21.15.88:0/369494952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7e80051c0 0x7ff7e800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:30.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7f9d9b640 1 -- 172.21.15.88:0/369494952 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7e8003520 con 0x7ff7e8004800 2024-09-16T01:41:30.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7fad9d640 1 --2- 172.21.15.88:0/369494952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7e80051c0 0x7ff7e800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:30.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.471+0000 7ff7fad9d640 1 -- 172.21.15.88:0/369494952 reap_dead start 2024-09-16T01:41:30.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.472+0000 7ff7f9d9b640 1 --2- 172.21.15.88:0/369494952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7e8004800 0x7ff7e8004c80 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4001720 tx=0x7ff7e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53cd5a1e4e7b04e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:30.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.472+0000 7ff7f959a640 1 -- 172.21.15.88:0/369494952 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e4001370 con 0x7ff7e8004800 2024-09-16T01:41:30.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.472+0000 7ff7f959a640 1 -- 172.21.15.88:0/369494952 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7e40052d0 con 0x7ff7e8004800 2024-09-16T01:41:30.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.472+0000 7ff7f959a640 1 -- 172.21.15.88:0/369494952 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e402ab90 con 0x7ff7e8004800 2024-09-16T01:41:30.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.473+0000 7ff800f6d640 1 -- 172.21.15.88:0/369494952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7e8004800 msgr2=0x7ff7e8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:30.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.473+0000 7ff800f6d640 1 --2- 172.21.15.88:0/369494952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7e8004800 0x7ff7e8004c80 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4001720 tx=0x7ff7e8005830 comp rx=0 tx=0).stop 2024-09-16T01:41:30.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.473+0000 7ff800f6d640 1 -- 172.21.15.88:0/369494952 shutdown_connections 2024-09-16T01:41:30.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.473+0000 7ff800f6d640 1 --2- 172.21.15.88:0/369494952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7e8004800 0x7ff7e8004c80 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:30.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.473+0000 7ff800f6d640 1 -- 172.21.15.88:0/369494952 >> 172.21.15.88:0/369494952 conn(0x7ff7fc06d0f0 msgr2=0x7ff7fc06d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:30.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.473+0000 7ff800f6d640 1 -- 172.21.15.88:0/369494952 shutdown_connections 2024-09-16T01:41:30.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.473+0000 7ff800f6d640 1 -- 172.21.15.88:0/369494952 wait complete. 2024-09-16T01:41:30.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.474+0000 7ff800f6d640 1 Processor -- start 2024-09-16T01:41:30.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.474+0000 7ff800f6d640 1 -- start start 2024-09-16T01:41:30.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.475+0000 7ff800f6d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7e8004800 0x7ff7fc0b3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:30.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.475+0000 7ff800f6d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7fc0c55f0 0x7ff7fc0b3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:30.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.475+0000 7ff800f6d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7fc0cc8c0 0x7ff7fc1461a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:30.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.475+0000 7ff800f6d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc15b9e0 con 0x7ff7fc0cc8c0 2024-09-16T01:41:30.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.475+0000 7ff800f6d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc15b860 con 0x7ff7fc0c55f0 2024-09-16T01:41:30.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.475+0000 7ff800f6d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff7fc071ff0 con 0x7ff7e8004800 2024-09-16T01:41:30.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.475+0000 7ff7fad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7fc0cc8c0 0x7ff7fc1461a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:30.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.475+0000 7ff7fad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7fc0cc8c0 0x7ff7fc1461a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42714/0 (socket says 172.21.15.88:42714) 2024-09-16T01:41:30.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.475+0000 7ff7fad9d640 1 -- 172.21.15.88:0/1608996965 learned_addr learned my addr 172.21.15.88:0/1608996965 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:30.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.475+0000 7ff7f9d9b640 1 --2- 172.21.15.88:0/1608996965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7fc0c55f0 0x7ff7fc0b3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:30.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.475+0000 7ff7fa59c640 1 --2- 172.21.15.88:0/1608996965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7e8004800 0x7ff7fc0b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:30.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.475+0000 7ff7fad9d640 1 -- 172.21.15.88:0/1608996965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7e8004800 msgr2=0x7ff7fc0b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:30.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.475+0000 7ff7fad9d640 1 --2- 172.21.15.88:0/1608996965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7e8004800 0x7ff7fc0b3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:30.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.475+0000 7ff7fad9d640 1 -- 172.21.15.88:0/1608996965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7fc0c55f0 msgr2=0x7ff7fc0b3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:30.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.475+0000 7ff7fad9d640 1 --2- 172.21.15.88:0/1608996965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7fc0c55f0 0x7ff7fc0b3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:30.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.475+0000 7ff7fad9d640 1 -- 172.21.15.88:0/1608996965 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7fc144030 con 0x7ff7fc0cc8c0 2024-09-16T01:41:30.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.475+0000 7ff7fad9d640 1 --2- 172.21.15.88:0/1608996965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7fc0cc8c0 0x7ff7fc1461a0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc0cdf50 tx=0x7ff7f4011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:30.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.475+0000 7ff7f2ffd640 1 -- 172.21.15.88:0/1608996965 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4004440 con 0x7ff7fc0cc8c0 2024-09-16T01:41:30.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.475+0000 7ff7f2ffd640 1 -- 172.21.15.88:0/1608996965 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7f401b070 con 0x7ff7fc0cc8c0 2024-09-16T01:41:30.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.475+0000 7ff7f2ffd640 1 -- 172.21.15.88:0/1608996965 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4003850 con 0x7ff7fc0cc8c0 2024-09-16T01:41:30.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.475+0000 7ff800f6d640 1 -- 172.21.15.88:0/1608996965 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7fc144310 con 0x7ff7fc0cc8c0 2024-09-16T01:41:30.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.476+0000 7ff800f6d640 1 -- 172.21.15.88:0/1608996965 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7fc1448d0 con 0x7ff7fc0cc8c0 2024-09-16T01:41:30.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.476+0000 7ff800f6d640 1 -- 172.21.15.88:0/1608996965 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7fc06e240 con 0x7ff7fc0cc8c0 2024-09-16T01:41:30.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.477+0000 7ff7f2ffd640 1 -- 172.21.15.88:0/1608996965 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff7f40039f0 con 0x7ff7fc0cc8c0 2024-09-16T01:41:30.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.477+0000 7ff7f2ffd640 1 --2- 172.21.15.88:0/1608996965 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff7dc06c2e0 0x7ff7dc06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:30.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.477+0000 7ff7f2ffd640 1 -- 172.21.15.88:0/1608996965 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 6770+0+0 (secure 0 0 0) 0x7ff7f408bc70 con 0x7ff7fc0cc8c0 2024-09-16T01:41:30.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.477+0000 7ff7fa59c640 1 --2- 172.21.15.88:0/1608996965 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff7dc06c2e0 0x7ff7dc06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:30.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.478+0000 7ff7fa59c640 1 --2- 172.21.15.88:0/1608996965 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff7dc06c2e0 0x7ff7dc06e7a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec004c20 tx=0x7ff7ec006040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:30.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.482+0000 7ff7f2ffd640 1 -- 172.21.15.88:0/1608996965 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff7fc06e240 con 0x7ff7fc0cc8c0 2024-09-16T01:41:30.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:30.683+0000 7ff800f6d640 1 -- 172.21.15.88:0/1608996965 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ff7fc144ad0 con 0x7ff7fc0cc8c0 2024-09-16T01:41:31.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.273+0000 7ff7f2ffd640 1 -- 172.21.15.88:0/1608996965 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v142) ==== 161+0+0 (secure 0 0 0) 0x7ff7fc144ad0 con 0x7ff7fc0cc8c0 2024-09-16T01:41:31.274 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:41:31.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.275+0000 7ff800f6d640 1 -- 172.21.15.88:0/1608996965 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff7dc06c2e0 msgr2=0x7ff7dc06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:31.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.275+0000 7ff800f6d640 1 --2- 172.21.15.88:0/1608996965 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff7dc06c2e0 0x7ff7dc06e7a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec004c20 tx=0x7ff7ec006040 comp rx=0 tx=0).stop 2024-09-16T01:41:31.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.275+0000 7ff800f6d640 1 -- 172.21.15.88:0/1608996965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7fc0cc8c0 msgr2=0x7ff7fc1461a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:31.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.275+0000 7ff800f6d640 1 --2- 172.21.15.88:0/1608996965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7fc0cc8c0 0x7ff7fc1461a0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc0cdf50 tx=0x7ff7f4011920 comp rx=0 tx=0).stop 2024-09-16T01:41:31.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.276+0000 7ff800f6d640 1 -- 172.21.15.88:0/1608996965 shutdown_connections 2024-09-16T01:41:31.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.276+0000 7ff800f6d640 1 --2- 172.21.15.88:0/1608996965 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff7dc06c2e0 0x7ff7dc06e7a0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:31.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.276+0000 7ff800f6d640 1 --2- 172.21.15.88:0/1608996965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7fc0cc8c0 0x7ff7fc1461a0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:31.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.276+0000 7ff800f6d640 1 --2- 172.21.15.88:0/1608996965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7fc0c55f0 0x7ff7fc0b3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:31.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.276+0000 7ff800f6d640 1 --2- 172.21.15.88:0/1608996965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7e8004800 0x7ff7fc0b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:31.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.276+0000 7ff800f6d640 1 -- 172.21.15.88:0/1608996965 >> 172.21.15.88:0/1608996965 conn(0x7ff7fc06d0f0 msgr2=0x7ff7fc1588c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:31.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.276+0000 7ff800f6d640 1 -- 172.21.15.88:0/1608996965 shutdown_connections 2024-09-16T01:41:31.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.276+0000 7ff800f6d640 1 -- 172.21.15.88:0/1608996965 wait complete. 2024-09-16T01:41:31.289 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:41:31.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.390+0000 7fe16628b640 1 Processor -- start 2024-09-16T01:41:31.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.390+0000 7fe16628b640 1 -- start start 2024-09-16T01:41:31.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.391+0000 7fe16628b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe160158a00 0x7fe160071f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:31.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.391+0000 7fe16628b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe160072470 0x7fe1600b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:31.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.391+0000 7fe16628b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe160158de0 0x7fe1600b53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:31.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.391+0000 7fe16628b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe1600728f0 con 0x7fe160072470 2024-09-16T01:41:31.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.391+0000 7fe16628b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe160072bd0 con 0x7fe160158a00 2024-09-16T01:41:31.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.391+0000 7fe16628b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe160072eb0 con 0x7fe160158de0 2024-09-16T01:41:31.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.391+0000 7fe15f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe160072470 0x7fe1600b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:31.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.391+0000 7fe15f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe160072470 0x7fe1600b4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42722/0 (socket says 172.21.15.88:42722) 2024-09-16T01:41:31.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.391+0000 7fe15f7fe640 1 -- 172.21.15.88:0/3546197292 learned_addr learned my addr 172.21.15.88:0/3546197292 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:31.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.391+0000 7fe15ffff640 1 --2- 172.21.15.88:0/3546197292 >> v2:172.21.15.154:3300/0 conn(0x7fe160158a00 0x7fe160071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:31.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.391+0000 7fe164828640 1 --2- 172.21.15.88:0/3546197292 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe160158de0 0x7fe1600b53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:31.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.391+0000 7fe15f7fe640 1 -- 172.21.15.88:0/3546197292 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe160158de0 msgr2=0x7fe1600b53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:31.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.391+0000 7fe15f7fe640 1 --2- 172.21.15.88:0/3546197292 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe160158de0 0x7fe1600b53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:31.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.391+0000 7fe15f7fe640 1 -- 172.21.15.88:0/3546197292 >> v2:172.21.15.154:3300/0 conn(0x7fe160158a00 msgr2=0x7fe160071f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:31.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.391+0000 7fe15f7fe640 1 --2- 172.21.15.88:0/3546197292 >> v2:172.21.15.154:3300/0 conn(0x7fe160158a00 0x7fe160071f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:31.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.391+0000 7fe15f7fe640 1 -- 172.21.15.88:0/3546197292 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe160073190 con 0x7fe160072470 2024-09-16T01:41:31.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.391+0000 7fe15ffff640 1 --2- 172.21.15.88:0/3546197292 >> v2:172.21.15.154:3300/0 conn(0x7fe160158a00 0x7fe160071f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:31.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.392+0000 7fe15f7fe640 1 --2- 172.21.15.88:0/3546197292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe160072470 0x7fe1600b4e60 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fe160069f90 tx=0x7fe15402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=120931f9eb3bb46b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:31.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.392+0000 7fe15effd640 1 -- 172.21.15.88:0/3546197292 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe154037070 con 0x7fe160072470 2024-09-16T01:41:31.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.392+0000 7fe15effd640 1 -- 172.21.15.88:0/3546197292 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe154032440 con 0x7fe160072470 2024-09-16T01:41:31.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.392+0000 7fe15effd640 1 -- 172.21.15.88:0/3546197292 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe154032810 con 0x7fe160072470 2024-09-16T01:41:31.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.393+0000 7fe16628b640 1 -- 172.21.15.88:0/3546197292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe160072470 msgr2=0x7fe1600b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:31.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.393+0000 7fe16628b640 1 --2- 172.21.15.88:0/3546197292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe160072470 0x7fe1600b4e60 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fe160069f90 tx=0x7fe15402a1e0 comp rx=0 tx=0).stop 2024-09-16T01:41:31.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.393+0000 7fe16628b640 1 -- 172.21.15.88:0/3546197292 shutdown_connections 2024-09-16T01:41:31.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.393+0000 7fe16628b640 1 --2- 172.21.15.88:0/3546197292 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe160158de0 0x7fe1600b53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:31.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.393+0000 7fe16628b640 1 --2- 172.21.15.88:0/3546197292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe160072470 0x7fe1600b4e60 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:31.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.393+0000 7fe16628b640 1 --2- 172.21.15.88:0/3546197292 >> v2:172.21.15.154:3300/0 conn(0x7fe160158a00 0x7fe160071f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:31.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.393+0000 7fe16628b640 1 -- 172.21.15.88:0/3546197292 >> 172.21.15.88:0/3546197292 conn(0x7fe16006cb50 msgr2=0x7fe16006cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:31.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.393+0000 7fe16628b640 1 -- 172.21.15.88:0/3546197292 shutdown_connections 2024-09-16T01:41:31.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.393+0000 7fe16628b640 1 -- 172.21.15.88:0/3546197292 wait complete. 2024-09-16T01:41:31.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.394+0000 7fe16628b640 1 Processor -- start 2024-09-16T01:41:31.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.394+0000 7fe16628b640 1 -- start start 2024-09-16T01:41:31.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.395+0000 7fe16628b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe160072470 0x7fe16014e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:31.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.395+0000 7fe16628b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe160158a00 0x7fe16013dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:31.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.395+0000 7fe16628b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe160158de0 0x7fe160140f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:31.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.395+0000 7fe16628b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe1600b5ca0 con 0x7fe160158de0 2024-09-16T01:41:31.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.395+0000 7fe16628b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe1600b5b20 con 0x7fe160158a00 2024-09-16T01:41:31.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.395+0000 7fe16628b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe160162110 con 0x7fe160072470 2024-09-16T01:41:31.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.395+0000 7fe164828640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe160158de0 0x7fe160140f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:31.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.395+0000 7fe164828640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe160158de0 0x7fe160140f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42734/0 (socket says 172.21.15.88:42734) 2024-09-16T01:41:31.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.395+0000 7fe164828640 1 -- 172.21.15.88:0/2361323790 learned_addr learned my addr 172.21.15.88:0/2361323790 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:31.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.395+0000 7fe15ffff640 1 --2- 172.21.15.88:0/2361323790 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe160072470 0x7fe16014e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:31.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.395+0000 7fe15f7fe640 1 --2- 172.21.15.88:0/2361323790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe160158a00 0x7fe16013dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:31.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.395+0000 7fe164828640 1 -- 172.21.15.88:0/2361323790 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe160072470 msgr2=0x7fe16014e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:31.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.395+0000 7fe164828640 1 --2- 172.21.15.88:0/2361323790 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe160072470 0x7fe16014e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:31.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.395+0000 7fe164828640 1 -- 172.21.15.88:0/2361323790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe160158a00 msgr2=0x7fe16013dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:31.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.395+0000 7fe164828640 1 --2- 172.21.15.88:0/2361323790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe160158a00 0x7fe16013dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:31.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.395+0000 7fe164828640 1 -- 172.21.15.88:0/2361323790 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe16013e1b0 con 0x7fe160158de0 2024-09-16T01:41:31.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.396+0000 7fe164828640 1 --2- 172.21.15.88:0/2361323790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe160158de0 0x7fe160140f20 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fe158009fd0 tx=0x7fe15800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:31.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.396+0000 7fe15cff9640 1 -- 172.21.15.88:0/2361323790 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe158014420 con 0x7fe160158de0 2024-09-16T01:41:31.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.396+0000 7fe15cff9640 1 -- 172.21.15.88:0/2361323790 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe158019070 con 0x7fe160158de0 2024-09-16T01:41:31.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.396+0000 7fe15cff9640 1 -- 172.21.15.88:0/2361323790 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe158014800 con 0x7fe160158de0 2024-09-16T01:41:31.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.396+0000 7fe16628b640 1 -- 172.21.15.88:0/2361323790 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe160141460 con 0x7fe160158de0 2024-09-16T01:41:31.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.396+0000 7fe16628b640 1 -- 172.21.15.88:0/2361323790 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe16013e5b0 con 0x7fe160158de0 2024-09-16T01:41:31.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.397+0000 7fe16628b640 1 -- 172.21.15.88:0/2361323790 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe160071f30 con 0x7fe160158de0 2024-09-16T01:41:31.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.397+0000 7fe15cff9640 1 -- 172.21.15.88:0/2361323790 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe1580149a0 con 0x7fe160158de0 2024-09-16T01:41:31.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.398+0000 7fe15cff9640 1 --2- 172.21.15.88:0/2361323790 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe14c06c2b0 0x7fe14c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:31.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.398+0000 7fe15cff9640 1 -- 172.21.15.88:0/2361323790 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 6770+0+0 (secure 0 0 0) 0x7fe15808be30 con 0x7fe160158de0 2024-09-16T01:41:31.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.398+0000 7fe15ffff640 1 --2- 172.21.15.88:0/2361323790 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe14c06c2b0 0x7fe14c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:31.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.398+0000 7fe15ffff640 1 --2- 172.21.15.88:0/2361323790 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe14c06c2b0 0x7fe14c06e770 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fe1600b55c0 tx=0x7fe150008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:31.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.402+0000 7fe15cff9640 1 -- 172.21.15.88:0/2361323790 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe160141460 con 0x7fe160158de0 2024-09-16T01:41:31.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:31.635+0000 7fe16628b640 1 -- 172.21.15.88:0/2361323790 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fe1600af340 con 0x7fe160158de0 2024-09-16T01:41:32.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.274+0000 7fe15cff9640 1 -- 172.21.15.88:0/2361323790 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v143) ==== 101+0+0 (secure 0 0 0) 0x7fe158059bb0 con 0x7fe160158de0 2024-09-16T01:41:32.276 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:41:32.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.277+0000 7fe16628b640 1 -- 172.21.15.88:0/2361323790 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe14c06c2b0 msgr2=0x7fe14c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:32.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.277+0000 7fe16628b640 1 --2- 172.21.15.88:0/2361323790 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe14c06c2b0 0x7fe14c06e770 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fe1600b55c0 tx=0x7fe150008040 comp rx=0 tx=0).stop 2024-09-16T01:41:32.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.277+0000 7fe16628b640 1 -- 172.21.15.88:0/2361323790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe160158de0 msgr2=0x7fe160140f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:32.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.277+0000 7fe16628b640 1 --2- 172.21.15.88:0/2361323790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe160158de0 0x7fe160140f20 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fe158009fd0 tx=0x7fe15800e4b0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.277+0000 7fe16628b640 1 -- 172.21.15.88:0/2361323790 shutdown_connections 2024-09-16T01:41:32.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.277+0000 7fe16628b640 1 --2- 172.21.15.88:0/2361323790 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe14c06c2b0 0x7fe14c06e770 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.277+0000 7fe16628b640 1 --2- 172.21.15.88:0/2361323790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe160158de0 0x7fe160140f20 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.277+0000 7fe16628b640 1 --2- 172.21.15.88:0/2361323790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe160158a00 0x7fe16013dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.277+0000 7fe16628b640 1 --2- 172.21.15.88:0/2361323790 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe160072470 0x7fe16014e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.277+0000 7fe16628b640 1 -- 172.21.15.88:0/2361323790 >> 172.21.15.88:0/2361323790 conn(0x7fe16006cb50 msgr2=0x7fe16006e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:32.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.277+0000 7fe16628b640 1 -- 172.21.15.88:0/2361323790 shutdown_connections 2024-09-16T01:41:32.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.277+0000 7fe16628b640 1 -- 172.21.15.88:0/2361323790 wait complete. 2024-09-16T01:41:32.291 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:41:32.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.333+0000 7f1da100af40 1 Processor -- start 2024-09-16T01:41:32.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.333+0000 7f1da100af40 1 -- start start 2024-09-16T01:41:32.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.334+0000 7f1da100af40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565128bb2fe0 0x565128c38fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.334+0000 7f1da100af40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x565128c57d20 0x565128bccc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.334+0000 7f1da100af40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565128c58100 0x565128bcd190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.334+0000 7f1da100af40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x565128ad5510 con 0x565128c58100 2024-09-16T01:41:32.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.334+0000 7f1da100af40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x565128bcbbd0 con 0x565128bb2fe0 2024-09-16T01:41:32.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.334+0000 7f1da100af40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x565128c39b40 con 0x565128c57d20 2024-09-16T01:41:32.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.334+0000 7f1d9e6b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565128c58100 0x565128bcd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:32.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.334+0000 7f1d9e6b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565128c58100 0x565128bcd190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42750/0 (socket says 172.21.15.88:42750) 2024-09-16T01:41:32.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.334+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/3807463616 learned_addr learned my addr 172.21.15.88:0/3807463616 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:32.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.334+0000 7f1d9deb3640 1 --2- 172.21.15.88:0/3807463616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565128bb2fe0 0x565128c38fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:32.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.334+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/3807463616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x565128c57d20 msgr2=0x565128bccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:32.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.334+0000 7f1d9e6b4640 1 --2- 172.21.15.88:0/3807463616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x565128c57d20 0x565128bccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.334+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/3807463616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565128bb2fe0 msgr2=0x565128c38fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:32.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.334+0000 7f1d9e6b4640 1 --2- 172.21.15.88:0/3807463616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565128bb2fe0 0x565128c38fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.334+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/3807463616 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565128c4d920 con 0x565128c58100 2024-09-16T01:41:32.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.334+0000 7f1d9deb3640 1 --2- 172.21.15.88:0/3807463616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565128bb2fe0 0x565128c38fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:32.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.335+0000 7f1d9e6b4640 1 --2- 172.21.15.88:0/3807463616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565128c58100 0x565128bcd190 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f1d98046080 tx=0x7f1d98051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a805236132a24ea server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:32.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.335+0000 7f1d9ceb1640 1 -- 172.21.15.88:0/3807463616 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d98068070 con 0x565128c58100 2024-09-16T01:41:32.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.335+0000 7f1d9ceb1640 1 -- 172.21.15.88:0/3807463616 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d980517e0 con 0x565128c58100 2024-09-16T01:41:32.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.335+0000 7f1d9ceb1640 1 -- 172.21.15.88:0/3807463616 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d98055040 con 0x565128c58100 2024-09-16T01:41:32.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.335+0000 7f1da100af40 1 -- 172.21.15.88:0/3807463616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565128c58100 msgr2=0x565128bcd190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:32.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.335+0000 7f1da100af40 1 --2- 172.21.15.88:0/3807463616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565128c58100 0x565128bcd190 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f1d98046080 tx=0x7f1d98051000 comp rx=0 tx=0).stop 2024-09-16T01:41:32.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.335+0000 7f1da100af40 1 -- 172.21.15.88:0/3807463616 shutdown_connections 2024-09-16T01:41:32.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.335+0000 7f1da100af40 1 --2- 172.21.15.88:0/3807463616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565128c58100 0x565128bcd190 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.335+0000 7f1da100af40 1 --2- 172.21.15.88:0/3807463616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x565128c57d20 0x565128bccc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.335+0000 7f1da100af40 1 --2- 172.21.15.88:0/3807463616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565128bb2fe0 0x565128c38fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.335+0000 7f1da100af40 1 -- 172.21.15.88:0/3807463616 >> 172.21.15.88:0/3807463616 conn(0x565128bce750 msgr2=0x565128bb2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:32.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.335+0000 7f1da100af40 1 -- 172.21.15.88:0/3807463616 shutdown_connections 2024-09-16T01:41:32.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.335+0000 7f1da100af40 1 -- 172.21.15.88:0/3807463616 wait complete. 2024-09-16T01:41:32.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.336+0000 7f1da100af40 1 Processor -- start 2024-09-16T01:41:32.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.336+0000 7f1da100af40 1 -- start start 2024-09-16T01:41:32.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.336+0000 7f1da100af40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565128bb2fe0 0x565128c4ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.336+0000 7f1da100af40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565128c57d20 0x565128c5ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.336+0000 7f1da100af40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x565128c58100 0x565128c58580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.336+0000 7f1da100af40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x565128bae6d0 con 0x565128bb2fe0 2024-09-16T01:41:32.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.336+0000 7f1da100af40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x565128baf020 con 0x565128c57d20 2024-09-16T01:41:32.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.337+0000 7f1da100af40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x565128bae9b0 con 0x565128c58100 2024-09-16T01:41:32.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.337+0000 7f1d9deb3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565128bb2fe0 0x565128c4ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:32.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.337+0000 7f1d9deb3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565128bb2fe0 0x565128c4ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42756/0 (socket says 172.21.15.88:42756) 2024-09-16T01:41:32.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.337+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 learned_addr learned my addr 172.21.15.88:0/577036796 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:32.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.337+0000 7f1d9e6b4640 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x565128c58100 0x565128c58580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.337+0000 7f1d956b2640 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565128c57d20 0x565128c5ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.337+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x565128c58100 msgr2=0x565128c58580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.337+0000 7f1d9deb3640 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x565128c58100 0x565128c58580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.337+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565128c57d20 msgr2=0x565128c5ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.337+0000 7f1d9deb3640 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565128c57d20 0x565128c5ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.337+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565128c504b0 con 0x565128bb2fe0 2024-09-16T01:41:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.337+0000 7f1d9deb3640 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565128bb2fe0 0x565128c4ff70 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f1d90001ea0 tx=0x7f1d9000a690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.337+0000 7f1d977fe640 1 -- 172.21.15.88:0/577036796 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d90008040 con 0x565128bb2fe0 2024-09-16T01:41:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.337+0000 7f1d977fe640 1 -- 172.21.15.88:0/577036796 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d90014020 con 0x565128bb2fe0 2024-09-16T01:41:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.337+0000 7f1d977fe640 1 -- 172.21.15.88:0/577036796 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9000ae60 con 0x565128bb2fe0 2024-09-16T01:41:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.337+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x565128c4d000 con 0x565128bb2fe0 2024-09-16T01:41:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.337+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565128c50d10 con 0x565128bb2fe0 2024-09-16T01:41:32.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.339+0000 7f1d977fe640 1 -- 172.21.15.88:0/577036796 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1d9000f600 con 0x565128bb2fe0 2024-09-16T01:41:32.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.339+0000 7f1d977fe640 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1d8006acc0 0x7f1d8006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.339+0000 7f1d977fe640 1 -- 172.21.15.88:0/577036796 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 6770+0+0 (secure 0 0 0) 0x7f1d9008a160 con 0x565128bb2fe0 2024-09-16T01:41:32.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.339+0000 7f1d956b2640 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1d8006acc0 0x7f1d8006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:32.344 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:41:32.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.339+0000 7f1da100af40 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x565128c5b350 0x565128c4bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.339+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128bb27d0 con 0x565128c5b350 2024-09-16T01:41:32.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.340+0000 7f1d9e6b4640 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x565128c5b350 0x565128c4bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:32.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.340+0000 7f1d956b2640 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1d8006acc0 0x7f1d8006d180 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x565128c57600 tx=0x7f1d880059f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:32.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.340+0000 7f1d9e6b4640 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x565128c5b350 0x565128c4bec0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:32.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.341+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v99'350 uv324 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.341+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) -- 0x565128c4c400 con 0x565128c5b350 2024-09-16T01:41:32.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.343+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v99'351 uv351 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.343+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128c4c7d0 con 0x565128c5b350 2024-09-16T01:41:32.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.344+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v99'352 uv352 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.346 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:41:32.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.344+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128c4cb70 con 0x565128c5b350 2024-09-16T01:41:32.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.345+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v99'353 uv353 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.345+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128cfd4c0 con 0x565128c5b350 2024-09-16T01:41:32.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.346+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v99'354 uv354 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.346+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128cfc4b0 con 0x565128c5b350 2024-09-16T01:41:32.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.347+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v99'355 uv355 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.347+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128cfc8c0 con 0x565128c5b350 2024-09-16T01:41:32.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.348+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v99'356 uv330 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.348+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) -- 0x565128cfc8c0 con 0x565128c5b350 2024-09-16T01:41:32.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.349+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v99'357 uv357 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.349+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128cfc850 con 0x565128c5b350 2024-09-16T01:41:32.351 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:41:32.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.350+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v99'358 uv358 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.350+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128cfc850 con 0x565128c5b350 2024-09-16T01:41:32.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.351+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v99'359 uv359 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.352+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128cfc850 con 0x565128c5b350 2024-09-16T01:41:32.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.353+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v99'360 uv360 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.353+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128cfc850 con 0x565128c5b350 2024-09-16T01:41:32.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.354+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v99'361 uv361 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.354+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128cfc850 con 0x565128c5b350 2024-09-16T01:41:32.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.356+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v99'362 uv337 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.356+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) -- 0x565128cfc850 con 0x565128c5b350 2024-09-16T01:41:32.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.357+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v99'363 uv363 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.357+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128cfc850 con 0x565128c5b350 2024-09-16T01:41:32.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.358+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v99'364 uv364 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.359 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:41:32.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.358+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128cfc850 con 0x565128c5b350 2024-09-16T01:41:32.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.359+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v99'365 uv365 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.359+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128cfc850 con 0x565128c5b350 2024-09-16T01:41:32.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.360+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v99'366 uv366 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.360+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128cfc850 con 0x565128c5b350 2024-09-16T01:41:32.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.361+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v99'367 uv367 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.361+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128cfcca0 con 0x565128c5b350 2024-09-16T01:41:32.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.362+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v99'368 uv368 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.362+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128cfd110 con 0x565128c5b350 2024-09-16T01:41:32.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.363+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v99'369 uv343 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.363+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) -- 0x565128cfcb00 con 0x565128c5b350 2024-09-16T01:41:32.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.364+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v99'370 uv370 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.364+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128cfcb00 con 0x565128c5b350 2024-09-16T01:41:32.366 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:41:32.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.365+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v99'371 uv371 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.365+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128cfcb00 con 0x565128c5b350 2024-09-16T01:41:32.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.367+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v99'372 uv372 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.367+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128cfcb00 con 0x565128c5b350 2024-09-16T01:41:32.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.368+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v99'373 uv373 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.368+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128cfcb00 con 0x565128c5b350 2024-09-16T01:41:32.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.369+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v99'374 uv374 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.369+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128cfcb00 con 0x565128c5b350 2024-09-16T01:41:32.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.370+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v99'375 uv349 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.370+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) -- 0x565128cfcb00 con 0x565128c5b350 2024-09-16T01:41:32.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.372+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v99'376 uv376 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.372+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128cfcb00 con 0x565128c5b350 2024-09-16T01:41:32.375 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:41:32.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.374+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v99'377 uv377 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.374+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128cfcb00 con 0x565128c5b350 2024-09-16T01:41:32.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.375+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v99'378 uv378 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.375+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128cfcb00 con 0x565128c5b350 2024-09-16T01:41:32.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.376+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v99'379 uv379 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.376+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128cfcb00 con 0x565128c5b350 2024-09-16T01:41:32.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.378+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v99'380 uv380 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.378+0000 7f1da100af40 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x565128cfdd70 0x565128d00160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.378+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128cfcb00 con 0x565128cfdd70 2024-09-16T01:41:32.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.378+0000 7f1d9deb3640 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x565128cfdd70 0x565128d00160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:32.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.378+0000 7f1d9deb3640 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x565128cfdd70 0x565128d00160 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:32.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.380+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v99'263 uv246 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128cfdd70 2024-09-16T01:41:32.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.380+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) -- 0x565128d006a0 con 0x565128cfdd70 2024-09-16T01:41:32.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.381+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v99'264 uv264 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128cfdd70 2024-09-16T01:41:32.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.381+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d00a40 con 0x565128cfdd70 2024-09-16T01:41:32.383 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:41:32.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.382+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v99'265 uv265 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128cfdd70 2024-09-16T01:41:32.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.382+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d01290 con 0x565128cfdd70 2024-09-16T01:41:32.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.383+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v99'266 uv266 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128cfdd70 2024-09-16T01:41:32.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.383+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d01630 con 0x565128cfdd70 2024-09-16T01:41:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.384+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v99'267 uv267 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128cfdd70 2024-09-16T01:41:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.384+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d019d0 con 0x565128cfdd70 2024-09-16T01:41:32.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.386+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v99'268 uv251 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128cfdd70 2024-09-16T01:41:32.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.386+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) -- 0x565128d01d70 con 0x565128cfdd70 2024-09-16T01:41:32.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.387+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v99'269 uv269 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128cfdd70 2024-09-16T01:41:32.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.387+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d01d70 con 0x565128cfdd70 2024-09-16T01:41:32.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.388+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v99'270 uv270 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128cfdd70 2024-09-16T01:41:32.389 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:41:32.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.388+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d01d70 con 0x565128cfdd70 2024-09-16T01:41:32.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.389+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v99'271 uv271 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128cfdd70 2024-09-16T01:41:32.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.389+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d01d70 con 0x565128cfdd70 2024-09-16T01:41:32.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.390+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v99'272 uv272 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128cfdd70 2024-09-16T01:41:32.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.390+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d01d70 con 0x565128cfdd70 2024-09-16T01:41:32.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.391+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v99'273 uv256 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128cfdd70 2024-09-16T01:41:32.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.391+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) -- 0x565128d01d70 con 0x565128cfdd70 2024-09-16T01:41:32.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.392+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v99'274 uv274 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128cfdd70 2024-09-16T01:41:32.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.392+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d01d70 con 0x565128cfdd70 2024-09-16T01:41:32.394 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:41:32.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.393+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v99'275 uv275 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128cfdd70 2024-09-16T01:41:32.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.393+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d061f0 con 0x565128cfdd70 2024-09-16T01:41:32.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.395+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v99'276 uv276 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128cfdd70 2024-09-16T01:41:32.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.395+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d061f0 con 0x565128cfdd70 2024-09-16T01:41:32.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.396+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v99'277 uv277 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128cfdd70 2024-09-16T01:41:32.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.396+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d01d70 con 0x565128cfdd70 2024-09-16T01:41:32.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.397+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v99'278 uv262 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128cfdd70 2024-09-16T01:41:32.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.397+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) -- 0x565128d01d70 con 0x565128cfdd70 2024-09-16T01:41:32.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.398+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v99'279 uv279 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128cfdd70 2024-09-16T01:41:32.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.398+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d01d70 con 0x565128cfdd70 2024-09-16T01:41:32.400 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:41:32.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.399+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v99'280 uv280 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128cfdd70 2024-09-16T01:41:32.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.399+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d02d80 con 0x565128cfdd70 2024-09-16T01:41:32.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.400+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v99'281 uv281 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128cfdd70 2024-09-16T01:41:32.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.400+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d02d80 con 0x565128cfdd70 2024-09-16T01:41:32.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.401+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v99'282 uv282 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128cfdd70 2024-09-16T01:41:32.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.401+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d01d70 con 0x565128cfdd70 2024-09-16T01:41:32.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.402+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v99'283 uv283 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128cfdd70 2024-09-16T01:41:32.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.402+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d01d70 con 0x565128c5b350 2024-09-16T01:41:32.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.403+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v99'69 uv68 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.403+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) -- 0x565128d01d70 con 0x565128c5b350 2024-09-16T01:41:32.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.404+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v99'70 uv70 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.404+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d01d70 con 0x565128c5b350 2024-09-16T01:41:32.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.405+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v99'71 uv71 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.406 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:41:32.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.405+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d02d80 con 0x565128c5b350 2024-09-16T01:41:32.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.406+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v99'72 uv72 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.406+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d02d80 con 0x565128c5b350 2024-09-16T01:41:32.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.407+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v99'73 uv73 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.407+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d01d70 con 0x565128c5b350 2024-09-16T01:41:32.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.408+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v99'74 uv74 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.408+0000 7f1da100af40 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x565128d023f0 0x565128d047e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.408+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d04e10 con 0x565128d023f0 2024-09-16T01:41:32.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.408+0000 7f1d956b2640 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x565128d023f0 0x565128d047e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:32.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.408+0000 7f1d956b2640 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x565128d023f0 0x565128d047e0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:32.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.409+0000 7f1d956b2640 1 -- 172.21.15.88:0/577036796 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v99'55 uv54 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f1d88018140 con 0x565128d023f0 2024-09-16T01:41:32.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.409+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) -- 0x565128d05320 con 0x565128d023f0 2024-09-16T01:41:32.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.410+0000 7f1d956b2640 1 -- 172.21.15.88:0/577036796 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v99'56 uv56 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1d88018140 con 0x565128d023f0 2024-09-16T01:41:32.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.410+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d05820 con 0x565128d023f0 2024-09-16T01:41:32.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.411+0000 7f1d956b2640 1 -- 172.21.15.88:0/577036796 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v99'57 uv57 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1d88018140 con 0x565128d023f0 2024-09-16T01:41:32.413 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:41:32.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.411+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d00ec0 con 0x565128d023f0 2024-09-16T01:41:32.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.412+0000 7f1d956b2640 1 -- 172.21.15.88:0/577036796 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v99'58 uv58 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1d88018140 con 0x565128d023f0 2024-09-16T01:41:32.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.412+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d06150 con 0x565128c5b350 2024-09-16T01:41:32.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.413+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v99'69 uv68 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.413+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) -- 0x565128d06150 con 0x565128c5b350 2024-09-16T01:41:32.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.415+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v99'70 uv70 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.415+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d06150 con 0x565128c5b350 2024-09-16T01:41:32.417 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:41:32.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.416+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v99'71 uv71 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.416+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d06150 con 0x565128c5b350 2024-09-16T01:41:32.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.417+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v99'72 uv72 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.417+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d06150 con 0x565128c5b350 2024-09-16T01:41:32.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.418+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v99'73 uv73 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.418+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d06150 con 0x565128c5b350 2024-09-16T01:41:32.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.419+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v99'74 uv74 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.419+0000 7f1da100af40 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x565128d06a70 0x565128d08e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.419+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d093a0 con 0x565128d06a70 2024-09-16T01:41:32.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.419+0000 7f1d9e6b4640 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x565128d06a70 0x565128d08e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:32.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.419+0000 7f1d9e6b4640 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x565128d06a70 0x565128d08e60 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:32.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.420+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v99'127 uv121 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.420+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) -- 0x565128d093a0 con 0x565128d06a70 2024-09-16T01:41:32.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.421+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v99'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.421+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d09790 con 0x565128d06a70 2024-09-16T01:41:32.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.422+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v99'129 uv129 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.423 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:41:32.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.422+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d0a7a0 con 0x565128d06a70 2024-09-16T01:41:32.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.423+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v99'130 uv130 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.423+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d0a7a0 con 0x565128d06a70 2024-09-16T01:41:32.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.424+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v99'131 uv131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.424+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d09790 con 0x565128d06a70 2024-09-16T01:41:32.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.425+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v99'132 uv132 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.425+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d09790 con 0x565128d06a70 2024-09-16T01:41:32.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.426+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v99'133 uv126 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.426+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) -- 0x565128d09790 con 0x565128d06a70 2024-09-16T01:41:32.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.427+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v99'134 uv134 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.427+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d09790 con 0x565128d06a70 2024-09-16T01:41:32.429 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:41:32.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.428+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v99'135 uv135 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.428+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d09790 con 0x565128d06a70 2024-09-16T01:41:32.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.429+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v99'136 uv136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.429+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d09790 con 0x565128d06a70 2024-09-16T01:41:32.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.430+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v99'137 uv137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.430+0000 7f1da100af40 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x565128d09e10 0x565128d0c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.430+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d0c780 con 0x565128d09e10 2024-09-16T01:41:32.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.430+0000 7f1d9deb3640 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x565128d09e10 0x565128d0c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:32.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.431+0000 7f1d9deb3640 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x565128d09e10 0x565128d0c200 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:32.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.432+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v99'56 uv55 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128d09e10 2024-09-16T01:41:32.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.432+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) -- 0x565128d0cb50 con 0x565128d09e10 2024-09-16T01:41:32.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.432+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v99'57 uv57 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128d09e10 2024-09-16T01:41:32.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.432+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d0cb50 con 0x565128d09e10 2024-09-16T01:41:32.435 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:41:32.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.434+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v99'58 uv58 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128d09e10 2024-09-16T01:41:32.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.434+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d0cb50 con 0x565128d09e10 2024-09-16T01:41:32.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.435+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v99'59 uv59 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128d09e10 2024-09-16T01:41:32.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.435+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d0cb50 con 0x565128d09e10 2024-09-16T01:41:32.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.436+0000 7f1d9deb3640 1 -- 172.21.15.88:0/577036796 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v99'60 uv60 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1d90054550 con 0x565128d09e10 2024-09-16T01:41:32.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.436+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d0cb50 con 0x565128c5b350 2024-09-16T01:41:32.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.437+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v99'69 uv68 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.437+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) -- 0x565128d0cb50 con 0x565128c5b350 2024-09-16T01:41:32.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.437+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v99'70 uv70 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.437+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d0cb50 con 0x565128c5b350 2024-09-16T01:41:32.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.438+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v99'71 uv71 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.440 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:41:32.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.438+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d0cb50 con 0x565128c5b350 2024-09-16T01:41:32.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.439+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v99'72 uv72 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.439+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d0db60 con 0x565128c5b350 2024-09-16T01:41:32.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.440+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v99'73 uv73 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.440+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d0cb50 con 0x565128c5b350 2024-09-16T01:41:32.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.441+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v99'74 uv74 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128c5b350 2024-09-16T01:41:32.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.441+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d0cb50 con 0x565128d023f0 2024-09-16T01:41:32.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.442+0000 7f1d956b2640 1 -- 172.21.15.88:0/577036796 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v99'112 uv108 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f1d88018140 con 0x565128d023f0 2024-09-16T01:41:32.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.442+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) -- 0x565128d0cf10 con 0x565128d023f0 2024-09-16T01:41:32.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.443+0000 7f1d956b2640 1 -- 172.21.15.88:0/577036796 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v99'113 uv113 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1d88018140 con 0x565128d023f0 2024-09-16T01:41:32.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.443+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d0d300 con 0x565128d023f0 2024-09-16T01:41:32.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.444+0000 7f1d956b2640 1 -- 172.21.15.88:0/577036796 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v99'114 uv114 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1d88018140 con 0x565128d023f0 2024-09-16T01:41:32.445 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:41:32.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.444+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d0d300 con 0x565128d023f0 2024-09-16T01:41:32.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.445+0000 7f1d956b2640 1 -- 172.21.15.88:0/577036796 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v99'115 uv115 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1d88018140 con 0x565128d023f0 2024-09-16T01:41:32.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.445+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d0d300 con 0x565128d023f0 2024-09-16T01:41:32.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.446+0000 7f1d956b2640 1 -- 172.21.15.88:0/577036796 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v99'116 uv116 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1d88018140 con 0x565128d023f0 2024-09-16T01:41:32.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.446+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d0d300 con 0x565128d023f0 2024-09-16T01:41:32.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.447+0000 7f1d956b2640 1 -- 172.21.15.88:0/577036796 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v99'117 uv111 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d88018140 con 0x565128d023f0 2024-09-16T01:41:32.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.447+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) -- 0x565128d0d300 con 0x565128d023f0 2024-09-16T01:41:32.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.448+0000 7f1d956b2640 1 -- 172.21.15.88:0/577036796 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v99'118 uv118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d88018140 con 0x565128d023f0 2024-09-16T01:41:32.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.448+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d0d300 con 0x565128d023f0 2024-09-16T01:41:32.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.448+0000 7f1d956b2640 1 -- 172.21.15.88:0/577036796 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v99'119 uv119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d88018140 con 0x565128d023f0 2024-09-16T01:41:32.450 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:41:32.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.449+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d0d300 con 0x565128d023f0 2024-09-16T01:41:32.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.449+0000 7f1d956b2640 1 -- 172.21.15.88:0/577036796 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v99'120 uv120 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d88018140 con 0x565128d023f0 2024-09-16T01:41:32.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.449+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d0d840 con 0x565128d023f0 2024-09-16T01:41:32.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.450+0000 7f1d956b2640 1 -- 172.21.15.88:0/577036796 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v99'121 uv121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d88018140 con 0x565128d023f0 2024-09-16T01:41:32.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.451+0000 7f1da100af40 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x565128d0e040 0x565128d10430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.451+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d10970 con 0x565128d0e040 2024-09-16T01:41:32.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.451+0000 7f1d956b2640 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x565128d0e040 0x565128d10430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:32.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.451+0000 7f1d956b2640 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x565128d0e040 0x565128d10430 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:32.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.452+0000 7f1d956b2640 1 -- 172.21.15.88:0/577036796 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v99'54 uv54 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f1d88018140 con 0x565128d0e040 2024-09-16T01:41:32.453 INFO:teuthology.orchestra.run.smithi088.stdout:Write #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:41:32.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.452+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d10970 con 0x565128d0e040 2024-09-16T01:41:32.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.453+0000 7f1d956b2640 1 -- 172.21.15.88:0/577036796 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v99'55 uv55 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f1d88018140 con 0x565128d0e040 2024-09-16T01:41:32.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.453+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:103 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d10970 con 0x565128d023f0 2024-09-16T01:41:32.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.454+0000 7f1d956b2640 1 -- 172.21.15.88:0/577036796 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(103 609.00000000 [create] v99'69 uv68 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d88018140 con 0x565128d023f0 2024-09-16T01:41:32.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.454+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:104 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) -- 0x565128d10970 con 0x565128d023f0 2024-09-16T01:41:32.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.455+0000 7f1d956b2640 1 -- 172.21.15.88:0/577036796 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(104 609.00000000 [delete] v99'70 uv70 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d88018140 con 0x565128d023f0 2024-09-16T01:41:32.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.455+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d10970 con 0x565128d023f0 2024-09-16T01:41:32.457 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:41:32.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.456+0000 7f1d956b2640 1 -- 172.21.15.88:0/577036796 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(105 609.00000000 [create] v99'71 uv71 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d88018140 con 0x565128d023f0 2024-09-16T01:41:32.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.456+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d11d70 con 0x565128d023f0 2024-09-16T01:41:32.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.457+0000 7f1d956b2640 1 -- 172.21.15.88:0/577036796 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v99'72 uv72 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d88018140 con 0x565128d023f0 2024-09-16T01:41:32.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.457+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d11d70 con 0x565128d023f0 2024-09-16T01:41:32.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.458+0000 7f1d956b2640 1 -- 172.21.15.88:0/577036796 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v99'73 uv73 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d88018140 con 0x565128d023f0 2024-09-16T01:41:32.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.458+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d10d60 con 0x565128d023f0 2024-09-16T01:41:32.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.459+0000 7f1d956b2640 1 -- 172.21.15.88:0/577036796 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(108 609.00000000 [omap-set-header] v99'74 uv74 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d88018140 con 0x565128d023f0 2024-09-16T01:41:32.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.459+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:109 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d10d60 con 0x565128d06a70 2024-09-16T01:41:32.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.460+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(109 400.00000000 [create] v99'57 uv56 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.460+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:110 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) -- 0x565128d10d60 con 0x565128d06a70 2024-09-16T01:41:32.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.461+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(110 400.00000000 [delete] v99'58 uv58 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.461+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d10d60 con 0x565128d06a70 2024-09-16T01:41:32.464 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:41:32.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.462+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(111 400.00000000 [create] v99'59 uv59 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.463+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d10d60 con 0x565128d06a70 2024-09-16T01:41:32.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.464+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(112 400.00000000 [write 0~22] v99'60 uv60 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.464+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d10d60 con 0x565128d06a70 2024-09-16T01:41:32.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.465+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(113 400.00000000 [omap-set-header] v99'61 uv61 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.465+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d10d60 con 0x565128d06a70 2024-09-16T01:41:32.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.466+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(114 100.00000000.inode [create] v99'144 uv136 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.466+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) -- 0x565128d10d60 con 0x565128d06a70 2024-09-16T01:41:32.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.467+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(115 100.00000000.inode [delete] v99'145 uv145 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.468+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d10d60 con 0x565128d06a70 2024-09-16T01:41:32.469 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:41:32.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.469+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(116 100.00000000.inode [create] v99'146 uv146 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.469+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d10d60 con 0x565128d06a70 2024-09-16T01:41:32.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.470+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v99'147 uv147 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.470+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d10d60 con 0x565128d06a70 2024-09-16T01:41:32.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.471+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v99'148 uv148 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.471+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d10d60 con 0x565128d06a70 2024-09-16T01:41:32.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.472+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(119 1.00000000 [create] v99'149 uv143 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.472+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) -- 0x565128d10d60 con 0x565128d06a70 2024-09-16T01:41:32.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.473+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(120 1.00000000 [delete] v99'150 uv150 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.473+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d10d60 con 0x565128d06a70 2024-09-16T01:41:32.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.474+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(121 1.00000000 [create] v99'151 uv151 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.475 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:41:32.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.474+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d11d70 con 0x565128d06a70 2024-09-16T01:41:32.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.475+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v99'152 uv152 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.475+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d11d70 con 0x565128d06a70 2024-09-16T01:41:32.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.476+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v99'153 uv153 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.476+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d10d60 con 0x565128d06a70 2024-09-16T01:41:32.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.477+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(124 1.00000000 [omap-set-header] v99'154 uv154 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.477+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) -- 0x565128d11210 con 0x565128d06a70 2024-09-16T01:41:32.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.478+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v99'155 uv155 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1d98057030 con 0x565128d06a70 2024-09-16T01:41:32.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.478+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x565128c5b350 msgr2=0x565128c4bec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:32.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.478+0000 7f1da100af40 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x565128c5b350 0x565128c4bec0 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.478+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x565128d023f0 msgr2=0x565128d047e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:32.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.478+0000 7f1da100af40 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x565128d023f0 0x565128d047e0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.478+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x565128d0e040 msgr2=0x565128d10430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:32.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.478+0000 7f1da100af40 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x565128d0e040 0x565128d10430 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.478+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x565128cfdd70 msgr2=0x565128d00160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:32.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.478+0000 7f1da100af40 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x565128cfdd70 0x565128d00160 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.478+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 reap_dead start 2024-09-16T01:41:32.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.478+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x565128d06a70 msgr2=0x565128d08e60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:32.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.478+0000 7f1da100af40 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x565128d06a70 0x565128d08e60 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.478+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x565128d09e10 msgr2=0x565128d0c200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:32.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.478+0000 7f1da100af40 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x565128d09e10 0x565128d0c200 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.478+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1d8006acc0 msgr2=0x7f1d8006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:32.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.478+0000 7f1da100af40 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1d8006acc0 0x7f1d8006d180 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x565128c57600 tx=0x7f1d880059f0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.478+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565128bb2fe0 msgr2=0x565128c4ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:32.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.478+0000 7f1da100af40 1 --2- 172.21.15.88:0/577036796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x565128bb2fe0 0x565128c4ff70 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f1d90001ea0 tx=0x7f1d9000a690 comp rx=0 tx=0).stop 2024-09-16T01:41:32.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.478+0000 7f1d9e6b4640 1 -- 172.21.15.88:0/577036796 reap_dead start 2024-09-16T01:41:32.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.478+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 shutdown_connections 2024-09-16T01:41:32.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.478+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 >> 172.21.15.88:0/577036796 conn(0x565128bce750 msgr2=0x565128b83320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:32.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.478+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 shutdown_connections 2024-09-16T01:41:32.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.478+0000 7f1da100af40 1 -- 172.21.15.88:0/577036796 wait complete. 2024-09-16T01:41:32.483 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 400.00000000 2024-09-16T01:41:32.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.564+0000 7f411af79f40 1 Processor -- start 2024-09-16T01:41:32.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.564+0000 7f411af79f40 1 -- start start 2024-09-16T01:41:32.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.565+0000 7f411af79f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aca1ea1fa0 0x55aca1f655b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.565+0000 7f411af79f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca1ea4230 0x55aca1f67a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.565+0000 7f411af79f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55aca1f70540 0x55aca1ea0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.565+0000 7f411af79f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55aca1eeb5b0 con 0x55aca1ea4230 2024-09-16T01:41:32.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.565+0000 7f411af79f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55aca1df2510 con 0x55aca1ea1fa0 2024-09-16T01:41:32.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.565+0000 7f411af79f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55aca1eea610 con 0x55aca1f70540 2024-09-16T01:41:32.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.565+0000 7f4110270640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca1ea4230 0x55aca1f67a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:32.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.565+0000 7f4110270640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca1ea4230 0x55aca1f67a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42758/0 (socket says 172.21.15.88:42758) 2024-09-16T01:41:32.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.565+0000 7f4110270640 1 -- 172.21.15.88:0/484927430 learned_addr learned my addr 172.21.15.88:0/484927430 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:32.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.565+0000 7f4118a71640 1 --2- 172.21.15.88:0/484927430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aca1ea1fa0 0x55aca1f655b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:32.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.565+0000 7f4119272640 1 --2- 172.21.15.88:0/484927430 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55aca1f70540 0x55aca1ea0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:32.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.565+0000 7f4110270640 1 -- 172.21.15.88:0/484927430 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55aca1f70540 msgr2=0x55aca1ea0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:32.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.565+0000 7f4110270640 1 --2- 172.21.15.88:0/484927430 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55aca1f70540 0x55aca1ea0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.565+0000 7f4110270640 1 -- 172.21.15.88:0/484927430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aca1ea1fa0 msgr2=0x55aca1f655b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:32.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.565+0000 7f4110270640 1 --2- 172.21.15.88:0/484927430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aca1ea1fa0 0x55aca1f655b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.565+0000 7f4110270640 1 -- 172.21.15.88:0/484927430 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aca1f68580 con 0x55aca1ea4230 2024-09-16T01:41:32.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.566+0000 7f4110270640 1 --2- 172.21.15.88:0/484927430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca1ea4230 0x55aca1f67a40 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f410000b310 tx=0x7f410000b7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b3f5c9a4cb431c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:32.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.566+0000 7f41137fe640 1 -- 172.21.15.88:0/484927430 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4100008070 con 0x55aca1ea4230 2024-09-16T01:41:32.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.566+0000 7f41137fe640 1 -- 172.21.15.88:0/484927430 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f410000d500 con 0x55aca1ea4230 2024-09-16T01:41:32.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.566+0000 7f41137fe640 1 -- 172.21.15.88:0/484927430 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410000d8b0 con 0x55aca1ea4230 2024-09-16T01:41:32.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.566+0000 7f411af79f40 1 -- 172.21.15.88:0/484927430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca1ea4230 msgr2=0x55aca1f67a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:32.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.566+0000 7f411af79f40 1 --2- 172.21.15.88:0/484927430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca1ea4230 0x55aca1f67a40 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f410000b310 tx=0x7f410000b7d0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.566+0000 7f411af79f40 1 -- 172.21.15.88:0/484927430 shutdown_connections 2024-09-16T01:41:32.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.566+0000 7f411af79f40 1 --2- 172.21.15.88:0/484927430 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55aca1f70540 0x55aca1ea0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.566+0000 7f411af79f40 1 --2- 172.21.15.88:0/484927430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca1ea4230 0x55aca1f67a40 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.566+0000 7f411af79f40 1 --2- 172.21.15.88:0/484927430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aca1ea1fa0 0x55aca1f655b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.566+0000 7f411af79f40 1 -- 172.21.15.88:0/484927430 >> 172.21.15.88:0/484927430 conn(0x55aca1eeb750 msgr2=0x55aca1e9f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:32.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.566+0000 7f411af79f40 1 -- 172.21.15.88:0/484927430 shutdown_connections 2024-09-16T01:41:32.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.566+0000 7f411af79f40 1 -- 172.21.15.88:0/484927430 wait complete. 2024-09-16T01:41:32.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.567+0000 7f411af79f40 1 Processor -- start 2024-09-16T01:41:32.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.567+0000 7f411af79f40 1 -- start start 2024-09-16T01:41:32.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.568+0000 7f411af79f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aca1ea1fa0 0x55aca1f69eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.568+0000 7f411af79f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55aca1ea4230 0x55aca1f61180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.568+0000 7f411af79f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca1f70540 0x55aca1ea0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.568+0000 7f411af79f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55aca1ea1c20 con 0x55aca1f70540 2024-09-16T01:41:32.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.568+0000 7f411af79f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55aca1f65af0 con 0x55aca1ea1fa0 2024-09-16T01:41:32.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.568+0000 7f411af79f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55aca1f682e0 con 0x55aca1ea4230 2024-09-16T01:41:32.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.568+0000 7f4119272640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca1f70540 0x55aca1ea0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:32.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.568+0000 7f4119272640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca1f70540 0x55aca1ea0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42768/0 (socket says 172.21.15.88:42768) 2024-09-16T01:41:32.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.568+0000 7f4119272640 1 -- 172.21.15.88:0/399241963 learned_addr learned my addr 172.21.15.88:0/399241963 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:32.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.568+0000 7f4110270640 1 --2- 172.21.15.88:0/399241963 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55aca1ea4230 0x55aca1f61180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:32.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.568+0000 7f4118a71640 1 --2- 172.21.15.88:0/399241963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aca1ea1fa0 0x55aca1f69eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:32.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.568+0000 7f4119272640 1 -- 172.21.15.88:0/399241963 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55aca1ea4230 msgr2=0x55aca1f61180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:32.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.568+0000 7f4119272640 1 --2- 172.21.15.88:0/399241963 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55aca1ea4230 0x55aca1f61180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.568+0000 7f4119272640 1 -- 172.21.15.88:0/399241963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aca1ea1fa0 msgr2=0x55aca1f69eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:32.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.568+0000 7f4119272640 1 --2- 172.21.15.88:0/399241963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aca1ea1fa0 0x55aca1f69eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.568+0000 7f4119272640 1 -- 172.21.15.88:0/399241963 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aca1eccaa0 con 0x55aca1f70540 2024-09-16T01:41:32.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.568+0000 7f4119272640 1 --2- 172.21.15.88:0/399241963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca1f70540 0x55aca1ea0a50 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f4114010910 tx=0x7f4114010dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:32.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.568+0000 7f41127fc640 1 -- 172.21.15.88:0/399241963 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4114019070 con 0x55aca1f70540 2024-09-16T01:41:32.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.568+0000 7f41127fc640 1 -- 172.21.15.88:0/399241963 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f411400e9b0 con 0x55aca1f70540 2024-09-16T01:41:32.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.568+0000 7f41127fc640 1 -- 172.21.15.88:0/399241963 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411400edc0 con 0x55aca1f70540 2024-09-16T01:41:32.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.569+0000 7f411af79f40 1 -- 172.21.15.88:0/399241963 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55aca1eccd40 con 0x55aca1f70540 2024-09-16T01:41:32.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.569+0000 7f411af79f40 1 -- 172.21.15.88:0/399241963 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aca1f616c0 con 0x55aca1f70540 2024-09-16T01:41:32.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.570+0000 7f41127fc640 1 -- 172.21.15.88:0/399241963 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4114011040 con 0x55aca1f70540 2024-09-16T01:41:32.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.570+0000 7f41127fc640 1 --2- 172.21.15.88:0/399241963 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f40f806ac90 0x7f40f806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.570+0000 7f41127fc640 1 -- 172.21.15.88:0/399241963 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 6770+0+0 (secure 0 0 0) 0x7f411408aa50 con 0x55aca1f70540 2024-09-16T01:41:32.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.570+0000 7f4118a71640 1 --2- 172.21.15.88:0/399241963 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f40f806ac90 0x7f40f806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:32.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.570+0000 7f411af79f40 1 --2- 172.21.15.88:0/399241963 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55aca1f619d0 0x55aca1f62a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.570+0000 7f411af79f40 1 -- 172.21.15.88:0/399241963 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:1 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) -- 0x55aca1f76010 con 0x55aca1f619d0 2024-09-16T01:41:32.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.571+0000 7f4110270640 1 --2- 172.21.15.88:0/399241963 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55aca1f619d0 0x55aca1f62a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:32.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.571+0000 7f4118a71640 1 --2- 172.21.15.88:0/399241963 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f40f806ac90 0x7f40f806d150 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f4108057000 tx=0x7f4108052000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:32.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.571+0000 7f4110270640 1 --2- 172.21.15.88:0/399241963 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55aca1f619d0 0x55aca1f62a50 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:32.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.572+0000 7f4110270640 1 -- 172.21.15.88:0/399241963 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(1 400.00000000 [delete] v99'62 uv62 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f410000d830 con 0x55aca1f619d0 2024-09-16T01:41:32.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.572+0000 7f411af79f40 1 -- 172.21.15.88:0/399241963 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55aca1f619d0 msgr2=0x55aca1f62a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:32.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.572+0000 7f411af79f40 1 --2- 172.21.15.88:0/399241963 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55aca1f619d0 0x55aca1f62a50 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.572+0000 7f411af79f40 1 -- 172.21.15.88:0/399241963 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f40f806ac90 msgr2=0x7f40f806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:32.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.572+0000 7f411af79f40 1 --2- 172.21.15.88:0/399241963 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f40f806ac90 0x7f40f806d150 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f4108057000 tx=0x7f4108052000 comp rx=0 tx=0).stop 2024-09-16T01:41:32.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.572+0000 7f411af79f40 1 -- 172.21.15.88:0/399241963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca1f70540 msgr2=0x55aca1ea0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:32.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.572+0000 7f411af79f40 1 --2- 172.21.15.88:0/399241963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55aca1f70540 0x55aca1ea0a50 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f4114010910 tx=0x7f4114010dd0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.572+0000 7f4119272640 1 -- 172.21.15.88:0/399241963 reap_dead start 2024-09-16T01:41:32.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.572+0000 7f411af79f40 1 -- 172.21.15.88:0/399241963 shutdown_connections 2024-09-16T01:41:32.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.572+0000 7f411af79f40 1 -- 172.21.15.88:0/399241963 >> 172.21.15.88:0/399241963 conn(0x55aca1eeb750 msgr2=0x55aca1f78d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:32.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.572+0000 7f411af79f40 1 -- 172.21.15.88:0/399241963 shutdown_connections 2024-09-16T01:41:32.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.572+0000 7f411af79f40 1 -- 172.21.15.88:0/399241963 wait complete. 2024-09-16T01:41:32.578 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:41:32.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.720+0000 7f6a37d11640 1 Processor -- start 2024-09-16T01:41:32.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.720+0000 7f6a37d11640 1 -- start start 2024-09-16T01:41:32.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.721+0000 7f6a37d11640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6a30158a00 0x7f6a30071f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.721+0000 7f6a37d11640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a30072470 0x7f6a300b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.721+0000 7f6a37d11640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a300728f0 con 0x7f6a30158de0 2024-09-16T01:41:32.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.721+0000 7f6a37d11640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6a30072bd0 con 0x7f6a30158a00 2024-09-16T01:41:32.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.721+0000 7f6a37d11640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6a30072eb0 con 0x7f6a30072470 2024-09-16T01:41:32.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.721+0000 7f6a35aad640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6a30158a00 0x7f6a30071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:32.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.721+0000 7f6a35aad640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6a30158a00 0x7f6a30071f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42782/0 (socket says 172.21.15.88:42782) 2024-09-16T01:41:32.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.721+0000 7f6a35aad640 1 -- 172.21.15.88:0/3415945828 learned_addr learned my addr 172.21.15.88:0/3415945828 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:32.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.721+0000 7f6a352ac640 1 --2- 172.21.15.88:0/3415945828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a30072470 0x7f6a300b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:32.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.721+0000 7f6a35aad640 1 -- 172.21.15.88:0/3415945828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a30072470 msgr2=0x7f6a300b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:32.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.721+0000 7f6a35aad640 1 --2- 172.21.15.88:0/3415945828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a30072470 0x7f6a300b4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.721+0000 7f6a35aad640 1 -- 172.21.15.88:0/3415945828 >> v1:172.21.15.154:6789/0 conn(0x7f6a30158de0 legacy=0x7f6a30162110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:32.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.721+0000 7f6a35aad640 1 -- 172.21.15.88:0/3415945828 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a30073190 con 0x7f6a30158a00 2024-09-16T01:41:32.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.722+0000 7f6a352ac640 1 --2- 172.21.15.88:0/3415945828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a30072470 0x7f6a300b4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:32.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.722+0000 7f6a35aad640 1 --2- 172.21.15.88:0/3415945828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a30158a00 0x7f6a30071f30 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c009e50 tx=0x7f6a2c02a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e507da1db8e5e217 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:32.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.722+0000 7f6a34aab640 1 -- 172.21.15.88:0/3415945828 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c036070 con 0x7f6a30158a00 2024-09-16T01:41:32.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.722+0000 7f6a34aab640 1 -- 172.21.15.88:0/3415945828 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a2c031600 con 0x7f6a30158a00 2024-09-16T01:41:32.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.722+0000 7f6a34aab640 1 -- 172.21.15.88:0/3415945828 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c0319d0 con 0x7f6a30158a00 2024-09-16T01:41:32.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.723+0000 7f6a37d11640 1 -- 172.21.15.88:0/3415945828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a30158a00 msgr2=0x7f6a30071f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:32.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.723+0000 7f6a37d11640 1 --2- 172.21.15.88:0/3415945828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a30158a00 0x7f6a30071f30 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c009e50 tx=0x7f6a2c02a6c0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.723+0000 7f6a37d11640 1 -- 172.21.15.88:0/3415945828 shutdown_connections 2024-09-16T01:41:32.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.723+0000 7f6a37d11640 1 --2- 172.21.15.88:0/3415945828 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a30072470 0x7f6a300b4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.723+0000 7f6a37d11640 1 --2- 172.21.15.88:0/3415945828 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a30158a00 0x7f6a30071f30 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.723+0000 7f6a37d11640 1 -- 172.21.15.88:0/3415945828 >> 172.21.15.88:0/3415945828 conn(0x7f6a3006cb50 msgr2=0x7f6a3006cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:32.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.723+0000 7f6a37d11640 1 -- 172.21.15.88:0/3415945828 shutdown_connections 2024-09-16T01:41:32.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.723+0000 7f6a37d11640 1 -- 172.21.15.88:0/3415945828 wait complete. 2024-09-16T01:41:32.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.724+0000 7f6a37d11640 1 Processor -- start 2024-09-16T01:41:32.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.724+0000 7f6a37d11640 1 -- start start 2024-09-16T01:41:32.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.725+0000 7f6a37d11640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a30072470 0x7f6a3014e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.725+0000 7f6a37d11640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a30158a00 0x7f6a3015f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.725+0000 7f6a37d11640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a30158de0 0x7f6a301ea120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.725+0000 7f6a37d11640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6a300b5ca0 con 0x7f6a30158a00 2024-09-16T01:41:32.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.725+0000 7f6a37d11640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6a300b5b20 con 0x7f6a30158de0 2024-09-16T01:41:32.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.725+0000 7f6a37d11640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6a3006faf0 con 0x7f6a30072470 2024-09-16T01:41:32.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.725+0000 7f6a352ac640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a30158a00 0x7f6a3015f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:32.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.725+0000 7f6a352ac640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a30158a00 0x7f6a3015f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42790/0 (socket says 172.21.15.88:42790) 2024-09-16T01:41:32.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.725+0000 7f6a352ac640 1 -- 172.21.15.88:0/3554216466 learned_addr learned my addr 172.21.15.88:0/3554216466 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:32.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.725+0000 7f6a362ae640 1 --2- 172.21.15.88:0/3554216466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a30158de0 0x7f6a301ea120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:32.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.725+0000 7f6a352ac640 1 -- 172.21.15.88:0/3554216466 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a30072470 msgr2=0x7f6a3014e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:32.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.725+0000 7f6a352ac640 1 --2- 172.21.15.88:0/3554216466 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a30072470 0x7f6a3014e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.725+0000 7f6a352ac640 1 -- 172.21.15.88:0/3554216466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a30158de0 msgr2=0x7f6a301ea120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:32.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.725+0000 7f6a352ac640 1 --2- 172.21.15.88:0/3554216466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a30158de0 0x7f6a301ea120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:32.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.725+0000 7f6a352ac640 1 -- 172.21.15.88:0/3554216466 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a301570a0 con 0x7f6a30158a00 2024-09-16T01:41:32.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.725+0000 7f6a362ae640 1 --2- 172.21.15.88:0/3554216466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a30158de0 0x7f6a301ea120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:32.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.725+0000 7f6a352ac640 1 --2- 172.21.15.88:0/3554216466 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a30158a00 0x7f6a3015f570 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f6a2000c820 tx=0x7f6a2000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:32.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.725+0000 7f6a267fc640 1 -- 172.21.15.88:0/3554216466 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a20015070 con 0x7f6a30158a00 2024-09-16T01:41:32.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.726+0000 7f6a267fc640 1 -- 172.21.15.88:0/3554216466 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a20007a30 con 0x7f6a30158a00 2024-09-16T01:41:32.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.726+0000 7f6a267fc640 1 -- 172.21.15.88:0/3554216466 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a20007e00 con 0x7f6a30158a00 2024-09-16T01:41:32.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.726+0000 7f6a37d11640 1 -- 172.21.15.88:0/3554216466 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a30157380 con 0x7f6a30158a00 2024-09-16T01:41:32.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.726+0000 7f6a37d11640 1 -- 172.21.15.88:0/3554216466 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a30157940 con 0x7f6a30158a00 2024-09-16T01:41:32.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.727+0000 7f6a37d11640 1 -- 172.21.15.88:0/3554216466 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a30071f30 con 0x7f6a30158a00 2024-09-16T01:41:32.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.727+0000 7f6a267fc640 1 -- 172.21.15.88:0/3554216466 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6a20018070 con 0x7f6a30158a00 2024-09-16T01:41:32.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.728+0000 7f6a267fc640 1 --2- 172.21.15.88:0/3554216466 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6a1406c2b0 0x7f6a1406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:32.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.728+0000 7f6a267fc640 1 -- 172.21.15.88:0/3554216466 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 6770+0+0 (secure 0 0 0) 0x7f6a2008b800 con 0x7f6a30158a00 2024-09-16T01:41:32.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.728+0000 7f6a35aad640 1 --2- 172.21.15.88:0/3554216466 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6a1406c2b0 0x7f6a1406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:32.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.728+0000 7f6a35aad640 1 --2- 172.21.15.88:0/3554216466 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6a1406c2b0 0x7f6a1406e770 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c033000 tx=0x7f6a2c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:32.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.732+0000 7f6a267fc640 1 -- 172.21.15.88:0/3554216466 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6a20059b40 con 0x7f6a30158a00 2024-09-16T01:41:32.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:32.932+0000 7f6a37d11640 1 -- 172.21.15.88:0/3554216466 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f6a300b4690 con 0x7f6a30158a00 2024-09-16T01:41:33.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.257+0000 7f6a267fc640 1 -- 172.21.15.88:0/3554216466 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v144) ==== 162+0+0 (secure 0 0 0) 0x7f6a20059440 con 0x7f6a30158a00 2024-09-16T01:41:33.264 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:41:33.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.259+0000 7f6a37d11640 1 -- 172.21.15.88:0/3554216466 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6a1406c2b0 msgr2=0x7f6a1406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:33.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.259+0000 7f6a37d11640 1 --2- 172.21.15.88:0/3554216466 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6a1406c2b0 0x7f6a1406e770 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c033000 tx=0x7f6a2c02e000 comp rx=0 tx=0).stop 2024-09-16T01:41:33.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.259+0000 7f6a37d11640 1 -- 172.21.15.88:0/3554216466 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a30158a00 msgr2=0x7f6a3015f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:33.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.259+0000 7f6a37d11640 1 --2- 172.21.15.88:0/3554216466 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a30158a00 0x7f6a3015f570 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f6a2000c820 tx=0x7f6a2000cc20 comp rx=0 tx=0).stop 2024-09-16T01:41:33.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.259+0000 7f6a37d11640 1 -- 172.21.15.88:0/3554216466 shutdown_connections 2024-09-16T01:41:33.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.259+0000 7f6a37d11640 1 --2- 172.21.15.88:0/3554216466 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6a1406c2b0 0x7f6a1406e770 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:33.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.259+0000 7f6a37d11640 1 --2- 172.21.15.88:0/3554216466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a30158de0 0x7f6a301ea120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:33.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.259+0000 7f6a37d11640 1 --2- 172.21.15.88:0/3554216466 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6a30158a00 0x7f6a3015f570 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:33.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.259+0000 7f6a37d11640 1 --2- 172.21.15.88:0/3554216466 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6a30072470 0x7f6a3014e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:33.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.259+0000 7f6a37d11640 1 -- 172.21.15.88:0/3554216466 >> 172.21.15.88:0/3554216466 conn(0x7f6a3006cb50 msgr2=0x7f6a300aec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:33.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.259+0000 7f6a37d11640 1 -- 172.21.15.88:0/3554216466 shutdown_connections 2024-09-16T01:41:33.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.259+0000 7f6a37d11640 1 -- 172.21.15.88:0/3554216466 wait complete. 2024-09-16T01:41:33.677 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:33.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.777+0000 7f84dceba640 1 Processor -- start 2024-09-16T01:41:33.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.777+0000 7f84dceba640 1 -- start start 2024-09-16T01:41:33.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.778+0000 7f84dceba640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84d815ddc0 0x7f84d8167100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:33.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.778+0000 7f84dceba640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84d8072760 con 0x7f84d815c710 2024-09-16T01:41:33.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.778+0000 7f84dceba640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84d8072a30 con 0x7f84d806bc10 2024-09-16T01:41:33.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.778+0000 7f84dceba640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f84d8072d10 con 0x7f84d815ddc0 2024-09-16T01:41:33.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.778+0000 7f84cdd9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f84d815c710 0x7f84d8072050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60128/0 (socket says 172.21.15.88:60128) 2024-09-16T01:41:33.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.778+0000 7f84cdd9b640 1 -- 172.21.15.88:0/3207087752 learned_addr learned my addr 172.21.15.88:0/3207087752 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:33.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.778+0000 7f84d5d9b640 1 -- 172.21.15.88:0/3207087752 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 623521239 0 0) 0x7f84d8072760 con 0x7f84d815c710 2024-09-16T01:41:33.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.778+0000 7f84d5d9b640 1 -- 172.21.15.88:0/3207087752 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84c4003520 con 0x7f84d815c710 2024-09-16T01:41:33.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.778+0000 7f84d6d9d640 1 --2- 172.21.15.88:0/3207087752 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84d815ddc0 0x7f84d8167100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:33.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.778+0000 7f84d5d9b640 1 -- 172.21.15.88:0/3207087752 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 481142621 0 0) 0x7f84c4003520 con 0x7f84d815c710 2024-09-16T01:41:33.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.778+0000 7f84d5d9b640 1 -- 172.21.15.88:0/3207087752 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84d815ddc0 msgr2=0x7f84d8167100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:33.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.778+0000 7f84d5d9b640 1 --2- 172.21.15.88:0/3207087752 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84d815ddc0 0x7f84d8167100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:33.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.778+0000 7f84d5d9b640 1 -- 172.21.15.88:0/3207087752 >> v1:172.21.15.154:6789/0 conn(0x7f84d806bc10 legacy=0x7f84d806bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:33.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.778+0000 7f84d5d9b640 1 -- 172.21.15.88:0/3207087752 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d8072ff0 con 0x7f84d815c710 2024-09-16T01:41:33.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.778+0000 7f84d5d9b640 1 -- 172.21.15.88:0/3207087752 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f84c0001670 con 0x7f84d815c710 2024-09-16T01:41:33.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.778+0000 7f84d6d9d640 1 --2- 172.21.15.88:0/3207087752 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84d815ddc0 0x7f84d8167100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:33.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.778+0000 7f84d5d9b640 1 -- 172.21.15.88:0/3207087752 >> v1:172.21.15.88:6789/0 conn(0x7f84d815c710 legacy=0x7f84d8072050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:33.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.778+0000 7f84d5d9b640 1 --2- 172.21.15.88:0/3207087752 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84d815ddc0 0x7f84c40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:33.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.778+0000 7f84d5d9b640 1 --2- 172.21.15.88:0/3207087752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84c4003fd0 0x7f84c4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:33.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.778+0000 7f84d5d9b640 1 --2- 172.21.15.88:0/3207087752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84c4004d40 0x7f84c40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:33.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.778+0000 7f84d5d9b640 1 -- 172.21.15.88:0/3207087752 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f84d8072d10 con 0x7f84c4004d40 2024-09-16T01:41:33.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.778+0000 7f84d5d9b640 1 -- 172.21.15.88:0/3207087752 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f84c400bb20 con 0x7f84c4003fd0 2024-09-16T01:41:33.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.778+0000 7f84d5d9b640 1 -- 172.21.15.88:0/3207087752 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f84c4010130 con 0x7f84d815ddc0 2024-09-16T01:41:33.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.779+0000 7f84d6d9d640 1 --2- 172.21.15.88:0/3207087752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84c4004d40 0x7f84c40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:33.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.779+0000 7f84cdd9b640 1 --2- 172.21.15.88:0/3207087752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84c4003fd0 0x7f84c4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:33.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.779+0000 7f84d659c640 1 --2- 172.21.15.88:0/3207087752 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84d815ddc0 0x7f84c40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:33.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.779+0000 7f84d6d9d640 1 -- 172.21.15.88:0/3207087752 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84d815ddc0 msgr2=0x7f84c40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:33.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.779+0000 7f84d6d9d640 1 --2- 172.21.15.88:0/3207087752 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84d815ddc0 0x7f84c40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:33.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.779+0000 7f84d6d9d640 1 -- 172.21.15.88:0/3207087752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84c4003fd0 msgr2=0x7f84c4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:33.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.779+0000 7f84d6d9d640 1 --2- 172.21.15.88:0/3207087752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84c4003fd0 0x7f84c4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:33.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.779+0000 7f84d6d9d640 1 -- 172.21.15.88:0/3207087752 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84c4003520 con 0x7f84c4004d40 2024-09-16T01:41:33.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.779+0000 7f84d659c640 1 --2- 172.21.15.88:0/3207087752 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84d815ddc0 0x7f84c40039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:33.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.779+0000 7f84d6d9d640 1 --2- 172.21.15.88:0/3207087752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84c4004d40 0x7f84c40051c0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f84d00073b0 tx=0x7f84c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=355b2dcf14cce4dd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:33.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.780+0000 7f84d5d9b640 1 -- 172.21.15.88:0/3207087752 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d0009c80 con 0x7f84c4004d40 2024-09-16T01:41:33.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.780+0000 7f84d5d9b640 1 -- 172.21.15.88:0/3207087752 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84d002e620 con 0x7f84c4004d40 2024-09-16T01:41:33.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.780+0000 7f84d5d9b640 1 -- 172.21.15.88:0/3207087752 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d002e9d0 con 0x7f84c4004d40 2024-09-16T01:41:33.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.780+0000 7f84dceba640 1 -- 172.21.15.88:0/3207087752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84c4004d40 msgr2=0x7f84c40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:33.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.780+0000 7f84dceba640 1 --2- 172.21.15.88:0/3207087752 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84c4004d40 0x7f84c40051c0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f84d00073b0 tx=0x7f84c4005830 comp rx=0 tx=0).stop 2024-09-16T01:41:33.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.780+0000 7f84d6d9d640 1 -- 172.21.15.88:0/3207087752 reap_dead start 2024-09-16T01:41:33.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.780+0000 7f84dceba640 1 -- 172.21.15.88:0/3207087752 shutdown_connections 2024-09-16T01:41:33.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.780+0000 7f84dceba640 1 -- 172.21.15.88:0/3207087752 >> 172.21.15.88:0/3207087752 conn(0x7f84d80d14d0 msgr2=0x7f84d80d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:33.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.780+0000 7f84dceba640 1 -- 172.21.15.88:0/3207087752 shutdown_connections 2024-09-16T01:41:33.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.780+0000 7f84dceba640 1 -- 172.21.15.88:0/3207087752 wait complete. 2024-09-16T01:41:33.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.782+0000 7f84dceba640 1 Processor -- start 2024-09-16T01:41:33.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.782+0000 7f84dceba640 1 -- start start 2024-09-16T01:41:33.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.782+0000 7f84dceba640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84c4004d40 0x7f84d814d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:33.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.782+0000 7f84dceba640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84d814db60 0x7f84d814f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:33.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.782+0000 7f84dceba640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84d814d5a0 0x7f84d815ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:33.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.782+0000 7f84dceba640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f84d806f850 con 0x7f84d814db60 2024-09-16T01:41:33.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.782+0000 7f84dceba640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f84d80735b0 con 0x7f84d814d5a0 2024-09-16T01:41:33.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.782+0000 7f84dceba640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f84d806f9d0 con 0x7f84c4004d40 2024-09-16T01:41:33.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.782+0000 7f84cdd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84d814db60 0x7f84d814f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:33.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.782+0000 7f84cdd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84d814db60 0x7f84d814f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42802/0 (socket says 172.21.15.88:42802) 2024-09-16T01:41:33.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.782+0000 7f84cdd9b640 1 -- 172.21.15.88:0/1816870552 learned_addr learned my addr 172.21.15.88:0/1816870552 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:33.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.783+0000 7f84d659c640 1 --2- 172.21.15.88:0/1816870552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84c4004d40 0x7f84d814d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:33.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.783+0000 7f84d6d9d640 1 --2- 172.21.15.88:0/1816870552 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84d814d5a0 0x7f84d815ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:33.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.783+0000 7f84cdd9b640 1 -- 172.21.15.88:0/1816870552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84c4004d40 msgr2=0x7f84d814d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:33.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.783+0000 7f84cdd9b640 1 --2- 172.21.15.88:0/1816870552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84c4004d40 0x7f84d814d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:33.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.783+0000 7f84cdd9b640 1 -- 172.21.15.88:0/1816870552 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84d814d5a0 msgr2=0x7f84d815ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:33.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.783+0000 7f84cdd9b640 1 --2- 172.21.15.88:0/1816870552 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84d814d5a0 0x7f84d815ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:33.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.783+0000 7f84cdd9b640 1 -- 172.21.15.88:0/1816870552 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d814e770 con 0x7f84d814db60 2024-09-16T01:41:33.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.783+0000 7f84d6d9d640 1 --2- 172.21.15.88:0/1816870552 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84d814d5a0 0x7f84d815ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:33.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.783+0000 7f84cdd9b640 1 --2- 172.21.15.88:0/1816870552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84d814db60 0x7f84d814f190 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f84c0007290 tx=0x7f84c0007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:33.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.783+0000 7f84cf7fe640 1 -- 172.21.15.88:0/1816870552 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c000f960 con 0x7f84d814db60 2024-09-16T01:41:33.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.783+0000 7f84cf7fe640 1 -- 172.21.15.88:0/1816870552 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84c0018070 con 0x7f84d814db60 2024-09-16T01:41:33.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.783+0000 7f84dceba640 1 -- 172.21.15.88:0/1816870552 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84d80cf1b0 con 0x7f84d814db60 2024-09-16T01:41:33.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.783+0000 7f84cf7fe640 1 -- 172.21.15.88:0/1816870552 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c000fd60 con 0x7f84d814db60 2024-09-16T01:41:33.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.783+0000 7f84dceba640 1 -- 172.21.15.88:0/1816870552 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84d80cf770 con 0x7f84d814db60 2024-09-16T01:41:33.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.784+0000 7f84dceba640 1 -- 172.21.15.88:0/1816870552 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84d806bff0 con 0x7f84d814db60 2024-09-16T01:41:33.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.784+0000 7f84cf7fe640 1 -- 172.21.15.88:0/1816870552 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f84c001d460 con 0x7f84d814db60 2024-09-16T01:41:33.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.785+0000 7f84cf7fe640 1 --2- 172.21.15.88:0/1816870552 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f84b806c2e0 0x7f84b806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:33.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.785+0000 7f84d659c640 1 --2- 172.21.15.88:0/1816870552 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f84b806c2e0 0x7f84b806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:33.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.785+0000 7f84cf7fe640 1 -- 172.21.15.88:0/1816870552 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 6770+0+0 (secure 0 0 0) 0x7f84c008b880 con 0x7f84d814db60 2024-09-16T01:41:33.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.786+0000 7f84d659c640 1 --2- 172.21.15.88:0/1816870552 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f84b806c2e0 0x7f84b806e7a0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f84c8001750 tx=0x7f84c8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:33.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.789+0000 7f84cf7fe640 1 -- 172.21.15.88:0/1816870552 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f84c0059c50 con 0x7f84d814db60 2024-09-16T01:41:33.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.990+0000 7f84dceba640 1 -- 172.21.15.88:0/1816870552 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f84d8157660 con 0x7f84d814db60 2024-09-16T01:41:33.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.991+0000 7f84cf7fe640 1 -- 172.21.15.88:0/1816870552 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) ==== 95+0+4400 (secure 0 0 0) 0x7f84c0059550 con 0x7f84d814db60 2024-09-16T01:41:33.992 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:33.993 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":144,"btime":"2024-09-16T01:41:33:256524+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:33.256494+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8175},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8175":{"gid":8175,"name":"c","rank":0,"incarnation":144,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6837/1255135754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1255135754},{"type":"v1","addr":"172.21.15.88:6837","nonce":1255135754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:41:33.993 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 144 2024-09-16T01:41:33.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.993+0000 7f84dceba640 1 -- 172.21.15.88:0/1816870552 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f84b806c2e0 msgr2=0x7f84b806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:33.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.993+0000 7f84dceba640 1 --2- 172.21.15.88:0/1816870552 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f84b806c2e0 0x7f84b806e7a0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f84c8001750 tx=0x7f84c8009040 comp rx=0 tx=0).stop 2024-09-16T01:41:33.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.993+0000 7f84dceba640 1 -- 172.21.15.88:0/1816870552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84d814db60 msgr2=0x7f84d814f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:33.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.993+0000 7f84dceba640 1 --2- 172.21.15.88:0/1816870552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84d814db60 0x7f84d814f190 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f84c0007290 tx=0x7f84c0007750 comp rx=0 tx=0).stop 2024-09-16T01:41:33.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.993+0000 7f84dceba640 1 -- 172.21.15.88:0/1816870552 shutdown_connections 2024-09-16T01:41:33.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.993+0000 7f84dceba640 1 --2- 172.21.15.88:0/1816870552 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f84b806c2e0 0x7f84b806e7a0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:33.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.993+0000 7f84dceba640 1 --2- 172.21.15.88:0/1816870552 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84d814d5a0 0x7f84d815ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:33.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.993+0000 7f84dceba640 1 --2- 172.21.15.88:0/1816870552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84d814db60 0x7f84d814f190 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:33.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.993+0000 7f84dceba640 1 --2- 172.21.15.88:0/1816870552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f84c4004d40 0x7f84d814d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:33.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.993+0000 7f84dceba640 1 -- 172.21.15.88:0/1816870552 >> 172.21.15.88:0/1816870552 conn(0x7f84d80d14d0 msgr2=0x7f84d8157b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:33.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.993+0000 7f84dceba640 1 -- 172.21.15.88:0/1816870552 shutdown_connections 2024-09-16T01:41:33.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:33.993+0000 7f84dceba640 1 -- 172.21.15.88:0/1816870552 wait complete. 2024-09-16T01:41:34.005 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 144, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:40:03.597796+0000', 'modified': '2024-09-16T01:41:33.256494+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 99, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8175}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8175': {'gid': 8175, 'name': 'c', 'rank': 0, 'incarnation': 144, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.88:6837/1255135754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6835', 'nonce': 1255135754}, {'type': 'v1', 'addr': '172.21.15.88:6837', 'nonce': 1255135754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-16T01:41:34.006 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8175:up:replay 2024-09-16T01:41:34.006 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-09-16T01:41:34.027 INFO:tasks.ceph.mds.a.smithi088.stderr: -45> 2024-09-16T01:41:29.274+0000 7f8fff31e640 -1 mds.0.sessionmap _load_finish got (2) No such file or directory 2024-09-16T01:41:34.027 INFO:tasks.ceph.mds.a.smithi088.stderr: -44> 2024-09-16T01:41:29.274+0000 7f8fff31e640 -1 log_channel(cluster) log [ERR] : error reading sessionmap 'mds0_sessionmap' -2 ((2) No such file or directory) 2024-09-16T01:41:34.049 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:41:39.410 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:39.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.520+0000 7f2b46fc0640 1 Processor -- start 2024-09-16T01:41:39.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.520+0000 7f2b46fc0640 1 -- start start 2024-09-16T01:41:39.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.521+0000 7f2b46fc0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2b4015c550 0x7f2b400715e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.521+0000 7f2b46fc0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b400b6e10 0x7f2b400b5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.521+0000 7f2b46fc0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2b40073160 0x7f2b400735b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.521+0000 7f2b46fc0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2b40073af0 con 0x7f2b40073160 2024-09-16T01:41:39.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.521+0000 7f2b46fc0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2b4015c930 con 0x7f2b4015c550 2024-09-16T01:41:39.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.521+0000 7f2b46fc0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2b400d0270 con 0x7f2b400b6e10 2024-09-16T01:41:39.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.521+0000 7f2b4555d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2b40073160 0x7f2b400735b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.522+0000 7f2b4555d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2b40073160 0x7f2b400735b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42820/0 (socket says 172.21.15.88:42820) 2024-09-16T01:41:39.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.522+0000 7f2b4555d640 1 -- 172.21.15.88:0/3125147541 learned_addr learned my addr 172.21.15.88:0/3125147541 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:39.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.522+0000 7f2b3ffff640 1 --2- 172.21.15.88:0/3125147541 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b400b6e10 0x7f2b400b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.522+0000 7f2b44d5c640 1 --2- 172.21.15.88:0/3125147541 >> v2:172.21.15.154:3300/0 conn(0x7f2b4015c550 0x7f2b400715e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.522+0000 7f2b4555d640 1 -- 172.21.15.88:0/3125147541 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b400b6e10 msgr2=0x7f2b400b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:39.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.522+0000 7f2b4555d640 1 --2- 172.21.15.88:0/3125147541 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b400b6e10 0x7f2b400b5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.522+0000 7f2b4555d640 1 -- 172.21.15.88:0/3125147541 >> v2:172.21.15.154:3300/0 conn(0x7f2b4015c550 msgr2=0x7f2b400715e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:39.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.522+0000 7f2b4555d640 1 --2- 172.21.15.88:0/3125147541 >> v2:172.21.15.154:3300/0 conn(0x7f2b4015c550 0x7f2b400715e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.522+0000 7f2b4555d640 1 -- 172.21.15.88:0/3125147541 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b400d04f0 con 0x7f2b40073160 2024-09-16T01:41:39.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.523+0000 7f2b4555d640 1 --2- 172.21.15.88:0/3125147541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b40073160 0x7f2b400735b0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f2b4006f010 tx=0x7f2b3802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9158749c477688e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:39.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.523+0000 7f2b3f7fe640 1 -- 172.21.15.88:0/3125147541 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b38037070 con 0x7f2b40073160 2024-09-16T01:41:39.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.523+0000 7f2b3f7fe640 1 -- 172.21.15.88:0/3125147541 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b38032440 con 0x7f2b40073160 2024-09-16T01:41:39.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.523+0000 7f2b3f7fe640 1 -- 172.21.15.88:0/3125147541 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b38032810 con 0x7f2b40073160 2024-09-16T01:41:39.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.523+0000 7f2b46fc0640 1 -- 172.21.15.88:0/3125147541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b40073160 msgr2=0x7f2b400735b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:39.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.523+0000 7f2b46fc0640 1 --2- 172.21.15.88:0/3125147541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b40073160 0x7f2b400735b0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f2b4006f010 tx=0x7f2b3802a1e0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.523+0000 7f2b46fc0640 1 -- 172.21.15.88:0/3125147541 shutdown_connections 2024-09-16T01:41:39.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.523+0000 7f2b46fc0640 1 --2- 172.21.15.88:0/3125147541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b40073160 0x7f2b400735b0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.523+0000 7f2b46fc0640 1 --2- 172.21.15.88:0/3125147541 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b400b6e10 0x7f2b400b5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.523+0000 7f2b46fc0640 1 --2- 172.21.15.88:0/3125147541 >> v2:172.21.15.154:3300/0 conn(0x7f2b4015c550 0x7f2b400715e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.523+0000 7f2b46fc0640 1 -- 172.21.15.88:0/3125147541 >> 172.21.15.88:0/3125147541 conn(0x7f2b400b4240 msgr2=0x7f2b400b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:39.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.523+0000 7f2b46fc0640 1 -- 172.21.15.88:0/3125147541 shutdown_connections 2024-09-16T01:41:39.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.523+0000 7f2b46fc0640 1 -- 172.21.15.88:0/3125147541 wait complete. 2024-09-16T01:41:39.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.525+0000 7f2b46fc0640 1 Processor -- start 2024-09-16T01:41:39.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.525+0000 7f2b46fc0640 1 -- start start 2024-09-16T01:41:39.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.526+0000 7f2b46fc0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b40073160 0x7f2b40149d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.526+0000 7f2b46fc0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b400b6e10 0x7f2b40159710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.526+0000 7f2b46fc0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4015c550 0x7f2b401529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.526+0000 7f2b46fc0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2b400d07b0 con 0x7f2b400b6e10 2024-09-16T01:41:39.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.526+0000 7f2b46fc0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2b400b7860 con 0x7f2b40073160 2024-09-16T01:41:39.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.526+0000 7f2b46fc0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2b400b7250 con 0x7f2b4015c550 2024-09-16T01:41:39.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.526+0000 7f2b3ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b400b6e10 0x7f2b40159710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.526+0000 7f2b3ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b400b6e10 0x7f2b40159710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42836/0 (socket says 172.21.15.88:42836) 2024-09-16T01:41:39.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.526+0000 7f2b3ffff640 1 -- 172.21.15.88:0/671437121 learned_addr learned my addr 172.21.15.88:0/671437121 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:39.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.526+0000 7f2b44d5c640 1 --2- 172.21.15.88:0/671437121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b40073160 0x7f2b40149d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.526+0000 7f2b4555d640 1 --2- 172.21.15.88:0/671437121 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4015c550 0x7f2b401529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.526+0000 7f2b3ffff640 1 -- 172.21.15.88:0/671437121 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4015c550 msgr2=0x7f2b401529a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:39.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.526+0000 7f2b3ffff640 1 --2- 172.21.15.88:0/671437121 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4015c550 0x7f2b401529a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.526+0000 7f2b3ffff640 1 -- 172.21.15.88:0/671437121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b40073160 msgr2=0x7f2b40149d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:39.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.526+0000 7f2b3ffff640 1 --2- 172.21.15.88:0/671437121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b40073160 0x7f2b40149d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.526+0000 7f2b3ffff640 1 -- 172.21.15.88:0/671437121 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b40156b70 con 0x7f2b400b6e10 2024-09-16T01:41:39.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.526+0000 7f2b3ffff640 1 --2- 172.21.15.88:0/671437121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b400b6e10 0x7f2b40159710 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f2b2800d800 tx=0x7f2b2800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:39.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.526+0000 7f2b3d7fa640 1 -- 172.21.15.88:0/671437121 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b28018020 con 0x7f2b400b6e10 2024-09-16T01:41:39.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.526+0000 7f2b3d7fa640 1 -- 172.21.15.88:0/671437121 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b280096a0 con 0x7f2b400b6e10 2024-09-16T01:41:39.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.526+0000 7f2b3d7fa640 1 -- 172.21.15.88:0/671437121 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b28009a70 con 0x7f2b400b6e10 2024-09-16T01:41:39.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.526+0000 7f2b46fc0640 1 -- 172.21.15.88:0/671437121 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b40156e50 con 0x7f2b400b6e10 2024-09-16T01:41:39.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.527+0000 7f2b46fc0640 1 -- 172.21.15.88:0/671437121 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b40158f10 con 0x7f2b400b6e10 2024-09-16T01:41:39.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.528+0000 7f2b46fc0640 1 -- 172.21.15.88:0/671437121 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b400735b0 con 0x7f2b400b6e10 2024-09-16T01:41:39.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.528+0000 7f2b3d7fa640 1 -- 172.21.15.88:0/671437121 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2b2801b020 con 0x7f2b400b6e10 2024-09-16T01:41:39.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.528+0000 7f2b3d7fa640 1 --2- 172.21.15.88:0/671437121 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b2006c2b0 0x7f2b2006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.528+0000 7f2b3d7fa640 1 -- 172.21.15.88:0/671437121 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 6770+0+0 (secure 0 0 0) 0x7f2b2808b960 con 0x7f2b400b6e10 2024-09-16T01:41:39.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.529+0000 7f2b44d5c640 1 --2- 172.21.15.88:0/671437121 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b2006c2b0 0x7f2b2006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.529+0000 7f2b44d5c640 1 --2- 172.21.15.88:0/671437121 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b2006c2b0 0x7f2b2006e770 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f2b30003aa0 tx=0x7f2b30009830 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:39.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.533+0000 7f2b3d7fa640 1 -- 172.21.15.88:0/671437121 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2b28022030 con 0x7f2b400b6e10 2024-09-16T01:41:39.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.732+0000 7f2b46fc0640 1 -- 172.21.15.88:0/671437121 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b400b2720 con 0x7f2b400b6e10 2024-09-16T01:41:39.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.733+0000 7f2b3d7fa640 1 -- 172.21.15.88:0/671437121 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) ==== 95+0+5230 (secure 0 0 0) 0x7f2b28059650 con 0x7f2b400b6e10 2024-09-16T01:41:39.735 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:39.735 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":147,"btime":"2024-09-16T01:41:36:261618+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/981209627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":981209627},{"type":"v1","addr":"172.21.15.88:6836","nonce":981209627}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:36.261612+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8175},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8175":{"gid":8175,"name":"c","rank":0,"incarnation":144,"state":"up:active","state_seq":7,"addr":"172.21.15.88:6837/1255135754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1255135754},{"type":"v1","addr":"172.21.15.88:6837","nonce":1255135754}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8175,"qdb_cluster":[8175]},"id":6}]} 2024-09-16T01:41:39.735 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 147 2024-09-16T01:41:39.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.735+0000 7f2b46fc0640 1 -- 172.21.15.88:0/671437121 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b2006c2b0 msgr2=0x7f2b2006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:39.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.735+0000 7f2b46fc0640 1 --2- 172.21.15.88:0/671437121 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b2006c2b0 0x7f2b2006e770 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f2b30003aa0 tx=0x7f2b30009830 comp rx=0 tx=0).stop 2024-09-16T01:41:39.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.735+0000 7f2b46fc0640 1 -- 172.21.15.88:0/671437121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b400b6e10 msgr2=0x7f2b40159710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:39.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.735+0000 7f2b46fc0640 1 --2- 172.21.15.88:0/671437121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b400b6e10 0x7f2b40159710 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f2b2800d800 tx=0x7f2b2800dcc0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.735+0000 7f2b46fc0640 1 -- 172.21.15.88:0/671437121 shutdown_connections 2024-09-16T01:41:39.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.735+0000 7f2b46fc0640 1 --2- 172.21.15.88:0/671437121 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b2006c2b0 0x7f2b2006e770 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.735+0000 7f2b46fc0640 1 --2- 172.21.15.88:0/671437121 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4015c550 0x7f2b401529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.735+0000 7f2b46fc0640 1 --2- 172.21.15.88:0/671437121 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b400b6e10 0x7f2b40159710 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.735+0000 7f2b46fc0640 1 --2- 172.21.15.88:0/671437121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b40073160 0x7f2b40149d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.735+0000 7f2b46fc0640 1 -- 172.21.15.88:0/671437121 >> 172.21.15.88:0/671437121 conn(0x7f2b400b4240 msgr2=0x7f2b400b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:39.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.735+0000 7f2b46fc0640 1 -- 172.21.15.88:0/671437121 shutdown_connections 2024-09-16T01:41:39.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.735+0000 7f2b46fc0640 1 -- 172.21.15.88:0/671437121 wait complete. 2024-09-16T01:41:39.748 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 147, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:40:03.597796+0000', 'modified': '2024-09-16T01:41:36.261612+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 99, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8175}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8175': {'gid': 8175, 'name': 'c', 'rank': 0, 'incarnation': 144, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.88:6837/1255135754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6835', 'nonce': 1255135754}, {'type': 'v1', 'addr': '172.21.15.88:6837', 'nonce': 1255135754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8175, 'qdb_cluster': [8175]} 2024-09-16T01:41:39.748 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T01:41:39.748 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-16T01:41:39.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.845+0000 7f0db9711640 1 Processor -- start 2024-09-16T01:41:39.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.845+0000 7f0db9711640 1 -- start start 2024-09-16T01:41:39.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.846+0000 7f0db9711640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0db40d1c10 0x7f0db40cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.846+0000 7f0db9711640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0db40b7ad0 con 0x7f0db40cf2f0 2024-09-16T01:41:39.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.846+0000 7f0db9711640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0db40cf760 con 0x7f0db415e600 2024-09-16T01:41:39.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.846+0000 7f0db9711640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0db40d2ee0 con 0x7f0db40d1c10 2024-09-16T01:41:39.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.846+0000 7f0db37fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0db40cf2f0 0x7f0db40d3370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60136/0 (socket says 172.21.15.88:60136) 2024-09-16T01:41:39.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.846+0000 7f0db37fe640 1 -- 172.21.15.88:0/2664735949 learned_addr learned my addr 172.21.15.88:0/2664735949 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:39.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.846+0000 7f0db27fc640 1 --2- 172.21.15.88:0/2664735949 >> v2:172.21.15.154:3300/0 conn(0x7f0db40d1c10 0x7f0db40cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.846+0000 7f0db1ffb640 1 -- 172.21.15.88:0/2664735949 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3579463673 0 0) 0x7f0dac001490 con 0x7f0db40cf2f0 2024-09-16T01:41:39.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.847+0000 7f0db1ffb640 1 -- 172.21.15.88:0/2664735949 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d9c003520 con 0x7f0db40cf2f0 2024-09-16T01:41:39.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.847+0000 7f0db1ffb640 1 -- 172.21.15.88:0/2664735949 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 756635736 0 0) 0x7f0d9c003520 con 0x7f0db40cf2f0 2024-09-16T01:41:39.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.847+0000 7f0db1ffb640 1 -- 172.21.15.88:0/2664735949 >> v2:172.21.15.154:3300/0 conn(0x7f0db40d1c10 msgr2=0x7f0db40cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:39.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.847+0000 7f0db1ffb640 1 --2- 172.21.15.88:0/2664735949 >> v2:172.21.15.154:3300/0 conn(0x7f0db40d1c10 0x7f0db40cedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.847+0000 7f0db1ffb640 1 -- 172.21.15.88:0/2664735949 >> v1:172.21.15.154:6789/0 conn(0x7f0db415e600 legacy=0x7f0db40d1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:39.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.847+0000 7f0db1ffb640 1 -- 172.21.15.88:0/2664735949 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0db40cf9e0 con 0x7f0db40cf2f0 2024-09-16T01:41:39.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.847+0000 7f0db1ffb640 1 -- 172.21.15.88:0/2664735949 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f0dac001bc0 con 0x7f0db40cf2f0 2024-09-16T01:41:39.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.847+0000 7f0db1ffb640 1 -- 172.21.15.88:0/2664735949 >> v1:172.21.15.88:6789/0 conn(0x7f0db40cf2f0 legacy=0x7f0db40d3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:39.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.847+0000 7f0db1ffb640 1 --2- 172.21.15.88:0/2664735949 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d9c0039b0 0x7f0d9c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.847+0000 7f0db1ffb640 1 --2- 172.21.15.88:0/2664735949 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d9c004800 0x7f0d9c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.847+0000 7f0db1ffb640 1 --2- 172.21.15.88:0/2664735949 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d9c0051c0 0x7f0d9c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.847+0000 7f0db1ffb640 1 -- 172.21.15.88:0/2664735949 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0db40d2ee0 con 0x7f0d9c0039b0 2024-09-16T01:41:39.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.847+0000 7f0db1ffb640 1 -- 172.21.15.88:0/2664735949 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9c00c090 con 0x7f0d9c004800 2024-09-16T01:41:39.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.847+0000 7f0db1ffb640 1 -- 172.21.15.88:0/2664735949 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0d9c00c290 con 0x7f0d9c0051c0 2024-09-16T01:41:39.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.847+0000 7f0db2ffd640 1 --2- 172.21.15.88:0/2664735949 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d9c0039b0 0x7f0d9c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.847+0000 7f0db27fc640 1 --2- 172.21.15.88:0/2664735949 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d9c004800 0x7f0d9c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.847+0000 7f0db37fe640 1 --2- 172.21.15.88:0/2664735949 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d9c0051c0 0x7f0d9c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.847+0000 7f0db2ffd640 1 -- 172.21.15.88:0/2664735949 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d9c0051c0 msgr2=0x7f0d9c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:39.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.847+0000 7f0db2ffd640 1 --2- 172.21.15.88:0/2664735949 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d9c0051c0 0x7f0d9c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.847+0000 7f0db2ffd640 1 -- 172.21.15.88:0/2664735949 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d9c004800 msgr2=0x7f0d9c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:39.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.847+0000 7f0db2ffd640 1 --2- 172.21.15.88:0/2664735949 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d9c004800 0x7f0d9c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.847+0000 7f0db37fe640 1 -- 172.21.15.88:0/2664735949 reap_dead start 2024-09-16T01:41:39.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.847+0000 7f0db2ffd640 1 -- 172.21.15.88:0/2664735949 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d9c003520 con 0x7f0d9c0039b0 2024-09-16T01:41:39.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.848+0000 7f0db2ffd640 1 --2- 172.21.15.88:0/2664735949 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d9c0039b0 0x7f0d9c003df0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f0da4001750 tx=0x7f0d9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28703b019ea37fb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:39.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.848+0000 7f0db1ffb640 1 -- 172.21.15.88:0/2664735949 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da4001370 con 0x7f0d9c0039b0 2024-09-16T01:41:39.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.848+0000 7f0db1ffb640 1 -- 172.21.15.88:0/2664735949 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0da40052d0 con 0x7f0d9c0039b0 2024-09-16T01:41:39.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.848+0000 7f0db1ffb640 1 -- 172.21.15.88:0/2664735949 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da402abe0 con 0x7f0d9c0039b0 2024-09-16T01:41:39.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.849+0000 7f0db9711640 1 -- 172.21.15.88:0/2664735949 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d9c0039b0 msgr2=0x7f0d9c003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:39.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.849+0000 7f0db9711640 1 --2- 172.21.15.88:0/2664735949 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d9c0039b0 0x7f0d9c003df0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f0da4001750 tx=0x7f0d9c005830 comp rx=0 tx=0).stop 2024-09-16T01:41:39.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.849+0000 7f0db9711640 1 -- 172.21.15.88:0/2664735949 shutdown_connections 2024-09-16T01:41:39.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.849+0000 7f0db9711640 1 --2- 172.21.15.88:0/2664735949 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d9c0039b0 0x7f0d9c003df0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.849+0000 7f0db9711640 1 -- 172.21.15.88:0/2664735949 >> 172.21.15.88:0/2664735949 conn(0x7f0db406c2c0 msgr2=0x7f0db406c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:39.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.849+0000 7f0db9711640 1 -- 172.21.15.88:0/2664735949 shutdown_connections 2024-09-16T01:41:39.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.849+0000 7f0db9711640 1 -- 172.21.15.88:0/2664735949 wait complete. 2024-09-16T01:41:39.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.850+0000 7f0db9711640 1 Processor -- start 2024-09-16T01:41:39.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.850+0000 7f0db9711640 1 -- start start 2024-09-16T01:41:39.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.851+0000 7f0db9711640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d9c0039b0 0x7f0db40cb830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.851+0000 7f0db9711640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0db40ccfd0 0x7f0db4161850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.851+0000 7f0db9711640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0db40cdcb0 0x7f0db414e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.851+0000 7f0db9711640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0db40d2b70 con 0x7f0db40cdcb0 2024-09-16T01:41:39.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.851+0000 7f0db9711640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0db40cffa0 con 0x7f0db40ccfd0 2024-09-16T01:41:39.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.851+0000 7f0db9711640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0db415e9e0 con 0x7f0d9c0039b0 2024-09-16T01:41:39.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.851+0000 7f0db37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0db40cdcb0 0x7f0db414e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.851+0000 7f0db37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0db40cdcb0 0x7f0db414e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42862/0 (socket says 172.21.15.88:42862) 2024-09-16T01:41:39.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.851+0000 7f0db37fe640 1 -- 172.21.15.88:0/2666248631 learned_addr learned my addr 172.21.15.88:0/2666248631 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:39.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.851+0000 7f0db27fc640 1 --2- 172.21.15.88:0/2666248631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0db40ccfd0 0x7f0db4161850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.851+0000 7f0db2ffd640 1 --2- 172.21.15.88:0/2666248631 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d9c0039b0 0x7f0db40cb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.851+0000 7f0db37fe640 1 -- 172.21.15.88:0/2666248631 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d9c0039b0 msgr2=0x7f0db40cb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:39.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.851+0000 7f0db37fe640 1 --2- 172.21.15.88:0/2666248631 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d9c0039b0 0x7f0db40cb830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.851+0000 7f0db37fe640 1 -- 172.21.15.88:0/2666248631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0db40ccfd0 msgr2=0x7f0db4161850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:39.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.851+0000 7f0db37fe640 1 --2- 172.21.15.88:0/2666248631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0db40ccfd0 0x7f0db4161850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.851+0000 7f0db37fe640 1 -- 172.21.15.88:0/2666248631 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0db40cbf30 con 0x7f0db40cdcb0 2024-09-16T01:41:39.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.851+0000 7f0db2ffd640 1 --2- 172.21.15.88:0/2666248631 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d9c0039b0 0x7f0db40cb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:39.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.851+0000 7f0db37fe640 1 --2- 172.21.15.88:0/2666248631 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0db40cdcb0 0x7f0db414e930 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f0db40b7350 tx=0x7f0dac007b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:39.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.851+0000 7f0d9b7fe640 1 -- 172.21.15.88:0/2666248631 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dac00ea40 con 0x7f0db40cdcb0 2024-09-16T01:41:39.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.851+0000 7f0d9b7fe640 1 -- 172.21.15.88:0/2666248631 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dac019070 con 0x7f0db40cdcb0 2024-09-16T01:41:39.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.851+0000 7f0db9711640 1 -- 172.21.15.88:0/2666248631 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0db40cc210 con 0x7f0db40cdcb0 2024-09-16T01:41:39.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.851+0000 7f0d9b7fe640 1 -- 172.21.15.88:0/2666248631 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dac00ee20 con 0x7f0db40cdcb0 2024-09-16T01:41:39.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.852+0000 7f0db9711640 1 -- 172.21.15.88:0/2666248631 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0db40cc7d0 con 0x7f0db40cdcb0 2024-09-16T01:41:39.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.853+0000 7f0d9b7fe640 1 -- 172.21.15.88:0/2666248631 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0dac01e460 con 0x7f0db40cdcb0 2024-09-16T01:41:39.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.853+0000 7f0d9b7fe640 1 --2- 172.21.15.88:0/2666248631 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0da006c2b0 0x7f0da006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.854+0000 7f0db2ffd640 1 --2- 172.21.15.88:0/2666248631 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0da006c2b0 0x7f0da006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.854+0000 7f0d9b7fe640 1 -- 172.21.15.88:0/2666248631 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 6770+0+0 (secure 0 0 0) 0x7f0dac08bc10 con 0x7f0db40cdcb0 2024-09-16T01:41:39.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.854+0000 7f0db2ffd640 1 --2- 172.21.15.88:0/2666248631 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0da006c2b0 0x7f0da006e770 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f0da4001750 tx=0x7f0da402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:39.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.854+0000 7f0dbc4a6740 1 Processor -- start 2024-09-16T01:41:39.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.855+0000 7f0dbc4a6740 1 -- start start 2024-09-16T01:41:39.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.855+0000 7f0dbc4a6740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56116f0562c0 0x56116f09ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.855+0000 7f0dbc4a6740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f053a80 0x56116f0566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.855+0000 7f0dbc4a6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56116f057a50 0x56116f0508b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.855+0000 7f0dbc4a6740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56116efb8520 con 0x56116f053a80 2024-09-16T01:41:39.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.855+0000 7f0dbc4a6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56116ef603e0 con 0x56116f057a50 2024-09-16T01:41:39.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.855+0000 7f0dbc4a6740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56116efb44e0 con 0x56116f0562c0 2024-09-16T01:41:39.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.855+0000 7f0db27fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f053a80 0x56116f0566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.855+0000 7f0db27fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f053a80 0x56116f0566a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42876/0 (socket says 172.21.15.88:42876) 2024-09-16T01:41:39.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.855+0000 7f0db27fc640 1 -- 172.21.15.88:0/1909684002 learned_addr learned my addr 172.21.15.88:0/1909684002 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:39.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.856+0000 7f0db37fe640 1 --2- 172.21.15.88:0/1909684002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56116f057a50 0x56116f0508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.856+0000 7f0db2ffd640 1 --2- 172.21.15.88:0/1909684002 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56116f0562c0 0x56116f09ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.856+0000 7f0db27fc640 1 -- 172.21.15.88:0/1909684002 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56116f0562c0 msgr2=0x56116f09ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:39.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.856+0000 7f0db27fc640 1 --2- 172.21.15.88:0/1909684002 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56116f0562c0 0x56116f09ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.856+0000 7f0db27fc640 1 -- 172.21.15.88:0/1909684002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56116f057a50 msgr2=0x56116f0508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:39.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.856+0000 7f0db27fc640 1 --2- 172.21.15.88:0/1909684002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56116f057a50 0x56116f0508b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.856+0000 7f0db27fc640 1 -- 172.21.15.88:0/1909684002 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56116ef7b2f0 con 0x56116f053a80 2024-09-16T01:41:39.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.856+0000 7f0db27fc640 1 --2- 172.21.15.88:0/1909684002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f053a80 0x56116f0566a0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f0da8009b40 tx=0x7f0da80143b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c77fc7cb8e304073 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:39.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.856+0000 7f0d98ff9640 1 -- 172.21.15.88:0/1909684002 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da8014df0 con 0x56116f053a80 2024-09-16T01:41:39.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.856+0000 7f0d98ff9640 1 -- 172.21.15.88:0/1909684002 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0da8019070 con 0x56116f053a80 2024-09-16T01:41:39.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.856+0000 7f0d98ff9640 1 -- 172.21.15.88:0/1909684002 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da8015640 con 0x56116f053a80 2024-09-16T01:41:39.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.857+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/1909684002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f053a80 msgr2=0x56116f0566a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:39.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.857+0000 7f0dbc4a6740 1 --2- 172.21.15.88:0/1909684002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f053a80 0x56116f0566a0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f0da8009b40 tx=0x7f0da80143b0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.857+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/1909684002 shutdown_connections 2024-09-16T01:41:39.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.857+0000 7f0dbc4a6740 1 --2- 172.21.15.88:0/1909684002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56116f057a50 0x56116f0508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.857+0000 7f0dbc4a6740 1 --2- 172.21.15.88:0/1909684002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f053a80 0x56116f0566a0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.857+0000 7f0dbc4a6740 1 --2- 172.21.15.88:0/1909684002 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56116f0562c0 0x56116f09ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.857+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/1909684002 >> 172.21.15.88:0/1909684002 conn(0x56116f052a80 msgr2=0x56116f09af30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:39.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.857+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/1909684002 shutdown_connections 2024-09-16T01:41:39.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.857+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/1909684002 wait complete. 2024-09-16T01:41:39.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.857+0000 7f0dbc4a6740 1 -- start start 2024-09-16T01:41:39.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.857+0000 7f0dbc4a6740 1 Processor -- start 2024-09-16T01:41:39.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.858+0000 7f0dbc4a6740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f0562c0 0x56116f0512e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.858+0000 7f0dbc4a6740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56116f057a50 0x56116f077230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.858+0000 7f0dbc4a6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56116f077770 0x56116f074530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.858+0000 7f0dbc4a6740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56116f051f20 con 0x56116f0562c0 2024-09-16T01:41:39.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.858+0000 7f0dbc4a6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56116ef41450 con 0x56116f077770 2024-09-16T01:41:39.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.858+0000 7f0dbc4a6740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56116f09c290 con 0x56116f057a50 2024-09-16T01:41:39.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.858+0000 7f0dbc4a6740 10 client.0 _ll_register_callbacks cb 0x56116eccaac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:41:39.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.858+0000 7f0db2ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f0562c0 0x56116f0512e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.858+0000 7f0db2ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f0562c0 0x56116f0512e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42880/0 (socket says 172.21.15.88:42880) 2024-09-16T01:41:39.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.858+0000 7f0db2ffd640 1 -- 172.21.15.88:0/2240761678 learned_addr learned my addr 172.21.15.88:0/2240761678 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:39.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.858+0000 7f0db37fe640 1 --2- 172.21.15.88:0/2240761678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56116f077770 0x56116f074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.858+0000 7f0db27fc640 1 --2- 172.21.15.88:0/2240761678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56116f057a50 0x56116f077230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.859+0000 7f0db2ffd640 1 -- 172.21.15.88:0/2240761678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56116f057a50 msgr2=0x56116f077230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:39.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.859+0000 7f0db2ffd640 1 --2- 172.21.15.88:0/2240761678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56116f057a50 0x56116f077230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.859+0000 7f0db2ffd640 1 -- 172.21.15.88:0/2240761678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56116f077770 msgr2=0x56116f074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:39.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.859+0000 7f0db2ffd640 1 --2- 172.21.15.88:0/2240761678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56116f077770 0x56116f074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.859+0000 7f0db2ffd640 1 -- 172.21.15.88:0/2240761678 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56116eccb2d0 con 0x56116f0562c0 2024-09-16T01:41:39.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.859+0000 7f0db27fc640 1 --2- 172.21.15.88:0/2240761678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56116f057a50 0x56116f077230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:39.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.859+0000 7f0db2ffd640 1 --2- 172.21.15.88:0/2240761678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f0562c0 0x56116f0512e0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f0da402a590 tx=0x7f0da402c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad7f23ac36199d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:39.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.859+0000 7f0d91ffb640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:41:39.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.859+0000 7f0d91ffb640 1 -- 172.21.15.88:0/2240761678 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da4037040 con 0x56116f0562c0 2024-09-16T01:41:39.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.859+0000 7f0d91ffb640 1 -- 172.21.15.88:0/2240761678 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 6770+0+0 (secure 0 0 0) 0x7f0da40069e0 con 0x56116f0562c0 2024-09-16T01:41:39.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.859+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/2240761678 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0da8019070 con 0x56116f0562c0 2024-09-16T01:41:39.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.859+0000 7f0d91ffb640 1 -- 172.21.15.88:0/2240761678 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7f0da4057050 con 0x56116f0562c0 2024-09-16T01:41:39.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.859+0000 7f0dbc4a6740 10 client.8328 fetch_fsmap learned FSMap version 147 2024-09-16T01:41:39.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.859+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/2240761678 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=2+}) -- 0x56116ed956b0 con 0x56116f0562c0 2024-09-16T01:41:39.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.859+0000 7f0d91ffb640 1 -- 172.21.15.88:0/2240761678 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0da8019070 con 0x56116f0562c0 2024-09-16T01:41:39.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.859+0000 7f0d91ffb640 1 -- 172.21.15.88:0/2240761678 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 147) ==== 3502+0+0 (secure 0 0 0) 0x7f0da405a020 con 0x56116f0562c0 2024-09-16T01:41:39.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.859+0000 7f0dbc4a6740 10 client.8328 fetch_fsmap finished waiting for FSMap version 147 2024-09-16T01:41:39.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.859+0000 7f0dbc4a6740 10 client.8328 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T01:41:39.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.859+0000 7f0dbc4a6740 20 client.8328 populate_metadata read hostname 'smithi088' 2024-09-16T01:41:39.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.859+0000 7f0dbc4a6740 1 --2- 172.21.15.88:0/2240761678 >> [v2:172.21.15.88:6835/1255135754,v1:172.21.15.88:6837/1255135754] conn(0x56116f0ae960 0x56116f0aed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.859+0000 7f0dbc4a6740 4 client.8328 mds_command: new command op to 8175 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:41:39.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.859+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/2240761678 --> [v2:172.21.15.88:6835/1255135754,v1:172.21.15.88:6837/1255135754] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56116efc5c50 con 0x56116f0ae960 2024-09-16T01:41:39.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.859+0000 7f0db27fc640 1 --2- 172.21.15.88:0/2240761678 >> [v2:172.21.15.88:6835/1255135754,v1:172.21.15.88:6837/1255135754] conn(0x56116f0ae960 0x56116f0aed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.860+0000 7f0d91ffb640 10 client.8328 ms_handle_connect on v2:172.21.15.88:6835/1255135754 2024-09-16T01:41:39.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.860+0000 7f0db27fc640 1 --2- 172.21.15.88:0/2240761678 >> [v2:172.21.15.88:6835/1255135754,v1:172.21.15.88:6837/1255135754] conn(0x56116f0ae960 0x56116f0aed40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d25a11a8e515fe1c server_cookie=ed6809f7dd308f58 in_seq=0 out_seq=0 2024-09-16T01:41:39.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.860+0000 7f0d91ffb640 1 -- 172.21.15.88:0/2240761678 <== mds.0 v2:172.21.15.88:6835/1255135754 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56116efc5c50 con 0x56116f0ae960 2024-09-16T01:41:39.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.860+0000 7f0d91ffb640 10 client.8328 handle_command_reply: tid=0 2024-09-16T01:41:39.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.860+0000 7f0dbc4a6740 1 client.8328 shutdown 2024-09-16T01:41:39.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.861+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/2240761678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f0562c0 msgr2=0x56116f0512e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:39.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.861+0000 7f0dbc4a6740 1 --2- 172.21.15.88:0/2240761678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f0562c0 0x56116f0512e0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f0da402a590 tx=0x7f0da402c070 comp rx=0 tx=0).stop 2024-09-16T01:41:39.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.861+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/2240761678 shutdown_connections 2024-09-16T01:41:39.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.861+0000 7f0dbc4a6740 1 --2- 172.21.15.88:0/2240761678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56116f077770 0x56116f074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.861+0000 7f0dbc4a6740 1 --2- 172.21.15.88:0/2240761678 >> [v2:172.21.15.88:6835/1255135754,v1:172.21.15.88:6837/1255135754] conn(0x56116f0ae960 0x56116f0aed40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.861+0000 7f0dbc4a6740 1 --2- 172.21.15.88:0/2240761678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56116f057a50 0x56116f077230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.861+0000 7f0dbc4a6740 1 --2- 172.21.15.88:0/2240761678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f0562c0 0x56116f0512e0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.861+0000 7f0d91ffb640 0 client.8328 ms_handle_reset on v2:172.21.15.88:6835/1255135754 2024-09-16T01:41:39.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.861+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/2240761678 >> 172.21.15.88:0/2240761678 conn(0x56116f053a80 msgr2=0x56116f09e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:39.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.861+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/2240761678 shutdown_connections 2024-09-16T01:41:39.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.861+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/2240761678 wait complete. 2024-09-16T01:41:39.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.861+0000 7f0dbc4a6740 20 client.8328 trim_cache size 0 max 16384 2024-09-16T01:41:39.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.878+0000 7f0dbc4a6740 1 Processor -- start 2024-09-16T01:41:39.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.878+0000 7f0dbc4a6740 1 -- start start 2024-09-16T01:41:39.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.879+0000 7f0dbc4a6740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f0562c0 0x56116f09e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.879+0000 7f0dbc4a6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56116f057a50 0x56116f0510b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.879+0000 7f0dbc4a6740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56116f077770 0x56116f076ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.879+0000 7f0dbc4a6740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0db4200d00 con 0x56116f0562c0 2024-09-16T01:41:39.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.879+0000 7f0dbc4a6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0db4200060 con 0x56116f057a50 2024-09-16T01:41:39.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.879+0000 7f0dbc4a6740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0db42006b0 con 0x56116f077770 2024-09-16T01:41:39.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.879+0000 7f0db2ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f0562c0 0x56116f09e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.879+0000 7f0db2ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f0562c0 0x56116f09e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42888/0 (socket says 172.21.15.88:42888) 2024-09-16T01:41:39.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.879+0000 7f0db2ffd640 1 -- 172.21.15.88:0/2483055153 learned_addr learned my addr 172.21.15.88:0/2483055153 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:39.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.879+0000 7f0db2ffd640 1 -- 172.21.15.88:0/2483055153 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56116f077770 msgr2=0x56116f076ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:39.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.879+0000 7f0db2ffd640 1 --2- 172.21.15.88:0/2483055153 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56116f077770 0x56116f076ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.879+0000 7f0db2ffd640 1 -- 172.21.15.88:0/2483055153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56116f057a50 msgr2=0x56116f0510b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:39.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.879+0000 7f0db2ffd640 1 --2- 172.21.15.88:0/2483055153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56116f057a50 0x56116f0510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.879+0000 7f0db2ffd640 1 -- 172.21.15.88:0/2483055153 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56116f0750c0 con 0x56116f0562c0 2024-09-16T01:41:39.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.879+0000 7f0db2ffd640 1 --2- 172.21.15.88:0/2483055153 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f0562c0 0x56116f09e1d0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f0da4007410 tx=0x7f0da403b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4f4c0a0ed5651e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:39.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.879+0000 7f0d93fff640 1 -- 172.21.15.88:0/2483055153 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da40069e0 con 0x56116f0562c0 2024-09-16T01:41:39.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.879+0000 7f0d93fff640 1 -- 172.21.15.88:0/2483055153 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0da404f070 con 0x56116f0562c0 2024-09-16T01:41:39.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.880+0000 7f0d93fff640 1 -- 172.21.15.88:0/2483055153 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da40035f0 con 0x56116f0562c0 2024-09-16T01:41:39.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.880+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/2483055153 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f0562c0 msgr2=0x56116f09e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:39.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.880+0000 7f0dbc4a6740 1 --2- 172.21.15.88:0/2483055153 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f0562c0 0x56116f09e1d0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f0da4007410 tx=0x7f0da403b000 comp rx=0 tx=0).stop 2024-09-16T01:41:39.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.880+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/2483055153 shutdown_connections 2024-09-16T01:41:39.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.880+0000 7f0dbc4a6740 1 --2- 172.21.15.88:0/2483055153 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56116f077770 0x56116f076ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.880+0000 7f0dbc4a6740 1 --2- 172.21.15.88:0/2483055153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56116f057a50 0x56116f0510b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.880+0000 7f0dbc4a6740 1 --2- 172.21.15.88:0/2483055153 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f0562c0 0x56116f09e1d0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.880+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/2483055153 >> 172.21.15.88:0/2483055153 conn(0x56116f053a80 msgr2=0x56116f09ae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:39.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.880+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/2483055153 shutdown_connections 2024-09-16T01:41:39.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.880+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/2483055153 wait complete. 2024-09-16T01:41:39.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.881+0000 7f0dbc4a6740 1 -- start start 2024-09-16T01:41:39.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.881+0000 7f0dbc4a6740 1 Processor -- start 2024-09-16T01:41:39.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.881+0000 7f0dbc4a6740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f0562c0 0x56116f097810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.881+0000 7f0dbc4a6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56116f057a50 0x56116f07c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.881+0000 7f0dbc4a6740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56116f077770 0x56116f0b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.881+0000 7f0dbc4a6740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56116f07e040 con 0x56116f0562c0 2024-09-16T01:41:39.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.881+0000 7f0dbc4a6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56116f076030 con 0x56116f057a50 2024-09-16T01:41:39.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.881+0000 7f0dbc4a6740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56116f0500a0 con 0x56116f077770 2024-09-16T01:41:39.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.881+0000 7f0dbc4a6740 10 client.0 _ll_register_callbacks cb 0x7f0d9c0036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:41:39.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.882+0000 7f0db2ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f0562c0 0x56116f097810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.882+0000 7f0db2ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f0562c0 0x56116f097810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42892/0 (socket says 172.21.15.88:42892) 2024-09-16T01:41:39.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.882+0000 7f0db2ffd640 1 -- 172.21.15.88:0/2860723143 learned_addr learned my addr 172.21.15.88:0/2860723143 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:39.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.882+0000 7f0db27fc640 1 --2- 172.21.15.88:0/2860723143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56116f057a50 0x56116f07c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.882+0000 7f0db37fe640 1 --2- 172.21.15.88:0/2860723143 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56116f077770 0x56116f0b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.882+0000 7f0db2ffd640 1 -- 172.21.15.88:0/2860723143 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56116f077770 msgr2=0x56116f0b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:39.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.882+0000 7f0db2ffd640 1 --2- 172.21.15.88:0/2860723143 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56116f077770 0x56116f0b3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.882+0000 7f0db2ffd640 1 -- 172.21.15.88:0/2860723143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56116f057a50 msgr2=0x56116f07c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:39.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.882+0000 7f0db2ffd640 1 --2- 172.21.15.88:0/2860723143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56116f057a50 0x56116f07c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.882+0000 7f0db2ffd640 1 -- 172.21.15.88:0/2860723143 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56116f07c860 con 0x56116f0562c0 2024-09-16T01:41:39.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.882+0000 7f0db27fc640 1 --2- 172.21.15.88:0/2860723143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56116f057a50 0x56116f07c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:39.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.882+0000 7f0db2ffd640 1 --2- 172.21.15.88:0/2860723143 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f0562c0 0x56116f097810 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f0da4000c00 tx=0x7f0da4007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5353f2cc0b84f5b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:39.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.882+0000 7f0d91ffb640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:41:39.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.882+0000 7f0d91ffb640 1 -- 172.21.15.88:0/2860723143 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da4003a00 con 0x56116f0562c0 2024-09-16T01:41:39.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.882+0000 7f0d91ffb640 1 -- 172.21.15.88:0/2860723143 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 6770+0+0 (secure 0 0 0) 0x7f0da4003ba0 con 0x56116f0562c0 2024-09-16T01:41:39.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.882+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/2860723143 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0da404f070 con 0x56116f0562c0 2024-09-16T01:41:39.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.882+0000 7f0d91ffb640 1 -- 172.21.15.88:0/2860723143 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=147) ==== 24+0+0 (secure 0 0 0) 0x7f0da4003da0 con 0x56116f0562c0 2024-09-16T01:41:39.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.882+0000 7f0dbc4a6740 10 client.8334 fetch_fsmap learned FSMap version 147 2024-09-16T01:41:39.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.882+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/2860723143 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=147,monmap=2+}) -- 0x56116f11a280 con 0x56116f0562c0 2024-09-16T01:41:39.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.882+0000 7f0d91ffb640 1 -- 172.21.15.88:0/2860723143 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0da404f070 con 0x56116f0562c0 2024-09-16T01:41:39.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.883+0000 7f0d91ffb640 1 -- 172.21.15.88:0/2860723143 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 147) ==== 3502+0+0 (secure 0 0 0) 0x7f0da405a040 con 0x56116f0562c0 2024-09-16T01:41:39.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.883+0000 7f0dbc4a6740 10 client.8334 fetch_fsmap finished waiting for FSMap version 147 2024-09-16T01:41:39.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.883+0000 7f0dbc4a6740 10 client.8334 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T01:41:39.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.883+0000 7f0dbc4a6740 20 client.8334 populate_metadata read hostname 'smithi088' 2024-09-16T01:41:39.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.883+0000 7f0dbc4a6740 1 --2- 172.21.15.88:0/2860723143 >> [v2:172.21.15.88:6835/1255135754,v1:172.21.15.88:6837/1255135754] conn(0x56116f0aa620 0x56116f0aaa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.883+0000 7f0dbc4a6740 4 client.8334 mds_command: new command op to 8175 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:41:39.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.883+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/2860723143 --> [v2:172.21.15.88:6835/1255135754,v1:172.21.15.88:6837/1255135754] -- command(tid 0: {"prefix": "status"}) -- 0x56116efbffa0 con 0x56116f0aa620 2024-09-16T01:41:39.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.883+0000 7f0db27fc640 1 --2- 172.21.15.88:0/2860723143 >> [v2:172.21.15.88:6835/1255135754,v1:172.21.15.88:6837/1255135754] conn(0x56116f0aa620 0x56116f0aaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.883+0000 7f0d91ffb640 10 client.8334 ms_handle_connect on v2:172.21.15.88:6835/1255135754 2024-09-16T01:41:39.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.883+0000 7f0db27fc640 1 --2- 172.21.15.88:0/2860723143 >> [v2:172.21.15.88:6835/1255135754,v1:172.21.15.88:6837/1255135754] conn(0x56116f0aa620 0x56116f0aaa00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d48d4a8dbe01a190 server_cookie=8b7ffcb1514ff8cc in_seq=0 out_seq=0 2024-09-16T01:41:39.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.883+0000 7f0d91ffb640 1 -- 172.21.15.88:0/2860723143 <== mds.0 v2:172.21.15.88:6835/1255135754 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x56116efbffa0 con 0x56116f0aa620 2024-09-16T01:41:39.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.883+0000 7f0d91ffb640 10 client.8334 handle_command_reply: tid=0 2024-09-16T01:41:39.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.883+0000 7f0dbc4a6740 1 client.8334 shutdown 2024-09-16T01:41:39.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.883+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/2860723143 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f0562c0 msgr2=0x56116f097810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:39.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.883+0000 7f0dbc4a6740 1 --2- 172.21.15.88:0/2860723143 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f0562c0 0x56116f097810 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f0da4000c00 tx=0x7f0da4007c80 comp rx=0 tx=0).stop 2024-09-16T01:41:39.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.883+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/2860723143 shutdown_connections 2024-09-16T01:41:39.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.883+0000 7f0dbc4a6740 1 --2- 172.21.15.88:0/2860723143 >> [v2:172.21.15.88:6835/1255135754,v1:172.21.15.88:6837/1255135754] conn(0x56116f0aa620 0x56116f0aaa00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.883+0000 7f0dbc4a6740 1 --2- 172.21.15.88:0/2860723143 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56116f077770 0x56116f0b3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.883+0000 7f0dbc4a6740 1 --2- 172.21.15.88:0/2860723143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56116f057a50 0x56116f07c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.883+0000 7f0dbc4a6740 1 --2- 172.21.15.88:0/2860723143 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56116f0562c0 0x56116f097810 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.883+0000 7f0d91ffb640 0 client.8334 ms_handle_reset on v2:172.21.15.88:6835/1255135754 2024-09-16T01:41:39.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.883+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/2860723143 >> 172.21.15.88:0/2860723143 conn(0x56116f053a80 msgr2=0x56116f0b4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:39.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.883+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/2860723143 shutdown_connections 2024-09-16T01:41:39.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.883+0000 7f0dbc4a6740 1 -- 172.21.15.88:0/2860723143 wait complete. 2024-09-16T01:41:39.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.883+0000 7f0dbc4a6740 20 client.8334 trim_cache size 0 max 16384 2024-09-16T01:41:39.891 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:41:39.891 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "5ace2c91-46f9-482b-8a89-731fbdc171bf", 2024-09-16T01:41:39.891 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2024-09-16T01:41:39.891 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 8175, 2024-09-16T01:41:39.891 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2024-09-16T01:41:39.892 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2024-09-16T01:41:39.892 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2024-09-16T01:41:39.892 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 6.6259210849999999, 2024-09-16T01:41:39.892 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 147, 2024-09-16T01:41:39.892 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 99, 2024-09-16T01:41:39.892 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 99, 2024-09-16T01:41:39.892 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 18.903733532 2024-09-16T01:41:39.892 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:41:39.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.884+0000 7f0db9711640 1 -- 172.21.15.88:0/2666248631 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0da006c2b0 msgr2=0x7f0da006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:39.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.884+0000 7f0db9711640 1 --2- 172.21.15.88:0/2666248631 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0da006c2b0 0x7f0da006e770 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f0da4001750 tx=0x7f0da402e000 comp rx=0 tx=0).stop 2024-09-16T01:41:39.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.884+0000 7f0db9711640 1 -- 172.21.15.88:0/2666248631 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0db40cdcb0 msgr2=0x7f0db414e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:39.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.884+0000 7f0db9711640 1 --2- 172.21.15.88:0/2666248631 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0db40cdcb0 0x7f0db414e930 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f0db40b7350 tx=0x7f0dac007b00 comp rx=0 tx=0).stop 2024-09-16T01:41:39.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.884+0000 7f0db9711640 1 -- 172.21.15.88:0/2666248631 shutdown_connections 2024-09-16T01:41:39.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.884+0000 7f0db9711640 1 --2- 172.21.15.88:0/2666248631 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0da006c2b0 0x7f0da006e770 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.884+0000 7f0db9711640 1 --2- 172.21.15.88:0/2666248631 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0db40cdcb0 0x7f0db414e930 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.884+0000 7f0db9711640 1 --2- 172.21.15.88:0/2666248631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0db40ccfd0 0x7f0db4161850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.884+0000 7f0db9711640 1 --2- 172.21.15.88:0/2666248631 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d9c0039b0 0x7f0db40cb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.884+0000 7f0db9711640 1 -- 172.21.15.88:0/2666248631 >> 172.21.15.88:0/2666248631 conn(0x7f0db406c2c0 msgr2=0x7f0db40b5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:39.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.884+0000 7f0db9711640 1 -- 172.21.15.88:0/2666248631 shutdown_connections 2024-09-16T01:41:39.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.884+0000 7f0db9711640 1 -- 172.21.15.88:0/2666248631 wait complete. 2024-09-16T01:41:39.896 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-09-16T01:41:39.896 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Delete 400.00000000', proceeding to ls 2024-09-16T01:41:39.896 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:41:39.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.993+0000 7ff1902e4640 1 Processor -- start 2024-09-16T01:41:39.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.993+0000 7ff1902e4640 1 -- start start 2024-09-16T01:41:39.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.994+0000 7ff1902e4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff18815e950 0x7ff18806ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.994+0000 7ff1902e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1880735b0 0x7ff18806d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.994+0000 7ff1902e4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff188072050 0x7ff18806bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:39.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.994+0000 7ff1902e4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff18806faf0 con 0x7ff188072050 2024-09-16T01:41:39.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.994+0000 7ff1902e4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff18806fdd0 con 0x7ff18815e950 2024-09-16T01:41:39.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.994+0000 7ff1902e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff1880700b0 con 0x7ff1880735b0 2024-09-16T01:41:39.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.994+0000 7ff18e881640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff188072050 0x7ff18806bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.994+0000 7ff18e881640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff188072050 0x7ff18806bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42900/0 (socket says 172.21.15.88:42900) 2024-09-16T01:41:39.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.994+0000 7ff18e881640 1 -- 172.21.15.88:0/3069188481 learned_addr learned my addr 172.21.15.88:0/3069188481 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:39.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.995+0000 7ff18d87f640 1 --2- 172.21.15.88:0/3069188481 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1880735b0 0x7ff18806d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.995+0000 7ff18e080640 1 --2- 172.21.15.88:0/3069188481 >> v2:172.21.15.154:3300/0 conn(0x7ff18815e950 0x7ff18806ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:39.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.995+0000 7ff18e881640 1 -- 172.21.15.88:0/3069188481 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1880735b0 msgr2=0x7ff18806d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:39.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.995+0000 7ff18e881640 1 --2- 172.21.15.88:0/3069188481 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1880735b0 0x7ff18806d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.995+0000 7ff18e881640 1 -- 172.21.15.88:0/3069188481 >> v2:172.21.15.154:3300/0 conn(0x7ff18815e950 msgr2=0x7ff18806ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:39.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.995+0000 7ff18e881640 1 --2- 172.21.15.88:0/3069188481 >> v2:172.21.15.154:3300/0 conn(0x7ff18815e950 0x7ff18806ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:39.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.995+0000 7ff18e881640 1 -- 172.21.15.88:0/3069188481 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff188070390 con 0x7ff188072050 2024-09-16T01:41:39.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.995+0000 7ff18e080640 1 --2- 172.21.15.88:0/3069188481 >> v2:172.21.15.154:3300/0 conn(0x7ff18815e950 0x7ff18806ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:40.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.995+0000 7ff18e881640 1 --2- 172.21.15.88:0/3069188481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff188072050 0x7ff18806bc10 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7ff188069190 tx=0x7ff18002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bdf991a19fc4669 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:40.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.995+0000 7ff18d07e640 1 -- 172.21.15.88:0/3069188481 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff180036070 con 0x7ff188072050 2024-09-16T01:41:40.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.996+0000 7ff18d07e640 1 -- 172.21.15.88:0/3069188481 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff180031600 con 0x7ff188072050 2024-09-16T01:41:40.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.996+0000 7ff18d07e640 1 -- 172.21.15.88:0/3069188481 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1800319d0 con 0x7ff188072050 2024-09-16T01:41:40.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.996+0000 7ff1902e4640 1 -- 172.21.15.88:0/3069188481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff188072050 msgr2=0x7ff18806bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:40.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.996+0000 7ff1902e4640 1 --2- 172.21.15.88:0/3069188481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff188072050 0x7ff18806bc10 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7ff188069190 tx=0x7ff18002a6a0 comp rx=0 tx=0).stop 2024-09-16T01:41:40.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.996+0000 7ff1902e4640 1 -- 172.21.15.88:0/3069188481 shutdown_connections 2024-09-16T01:41:40.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.996+0000 7ff1902e4640 1 --2- 172.21.15.88:0/3069188481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff188072050 0x7ff18806bc10 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:40.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.996+0000 7ff1902e4640 1 --2- 172.21.15.88:0/3069188481 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1880735b0 0x7ff18806d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:40.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.996+0000 7ff1902e4640 1 --2- 172.21.15.88:0/3069188481 >> v2:172.21.15.154:3300/0 conn(0x7ff18815e950 0x7ff18806ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:40.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.996+0000 7ff1902e4640 1 -- 172.21.15.88:0/3069188481 >> 172.21.15.88:0/3069188481 conn(0x7ff1880d14d0 msgr2=0x7ff1880d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:40.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.996+0000 7ff1902e4640 1 -- 172.21.15.88:0/3069188481 shutdown_connections 2024-09-16T01:41:40.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.996+0000 7ff1902e4640 1 -- 172.21.15.88:0/3069188481 wait complete. 2024-09-16T01:41:40.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.997+0000 7ff1902e4640 1 Processor -- start 2024-09-16T01:41:40.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.997+0000 7ff1902e4640 1 -- start start 2024-09-16T01:41:40.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.998+0000 7ff1902e4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff188072050 0x7ff188154340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:40.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.998+0000 7ff1902e4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1880735b0 0x7ff188155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:40.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.998+0000 7ff1902e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff18815e950 0x7ff188159be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:40.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.998+0000 7ff1902e4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff18806c2d0 con 0x7ff1880735b0 2024-09-16T01:41:40.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.998+0000 7ff1902e4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff18806c150 con 0x7ff188072050 2024-09-16T01:41:40.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.998+0000 7ff1902e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff18806c450 con 0x7ff18815e950 2024-09-16T01:41:40.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.998+0000 7ff18d87f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1880735b0 0x7ff188155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:40.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.998+0000 7ff18d87f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1880735b0 0x7ff188155670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42902/0 (socket says 172.21.15.88:42902) 2024-09-16T01:41:40.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.998+0000 7ff18d87f640 1 -- 172.21.15.88:0/3333064221 learned_addr learned my addr 172.21.15.88:0/3333064221 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:40.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.998+0000 7ff18e080640 1 --2- 172.21.15.88:0/3333064221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff188072050 0x7ff188154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:40.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.998+0000 7ff18e881640 1 --2- 172.21.15.88:0/3333064221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff18815e950 0x7ff188159be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:40.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.998+0000 7ff18d87f640 1 -- 172.21.15.88:0/3333064221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff18815e950 msgr2=0x7ff188159be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:40.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.998+0000 7ff18d87f640 1 --2- 172.21.15.88:0/3333064221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff18815e950 0x7ff188159be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:40.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.998+0000 7ff18d87f640 1 -- 172.21.15.88:0/3333064221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff188072050 msgr2=0x7ff188154340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:40.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.998+0000 7ff18d87f640 1 --2- 172.21.15.88:0/3333064221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff188072050 0x7ff188154340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:40.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.998+0000 7ff18d87f640 1 -- 172.21.15.88:0/3333064221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff188157f70 con 0x7ff1880735b0 2024-09-16T01:41:40.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.999+0000 7ff18e881640 1 --2- 172.21.15.88:0/3333064221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff18815e950 0x7ff188159be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:40.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.999+0000 7ff18d87f640 1 --2- 172.21.15.88:0/3333064221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1880735b0 0x7ff188155670 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7ff17800d5f0 tx=0x7ff17800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:40.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.999+0000 7ff17effd640 1 -- 172.21.15.88:0/3333064221 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff178004240 con 0x7ff1880735b0 2024-09-16T01:41:40.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.999+0000 7ff17effd640 1 -- 172.21.15.88:0/3333064221 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1780043e0 con 0x7ff1880735b0 2024-09-16T01:41:40.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.999+0000 7ff1902e4640 1 -- 172.21.15.88:0/3333064221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff188155d70 con 0x7ff1880735b0 2024-09-16T01:41:40.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.999+0000 7ff17effd640 1 -- 172.21.15.88:0/3333064221 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff178003610 con 0x7ff1880735b0 2024-09-16T01:41:40.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:39.999+0000 7ff1902e4640 1 -- 172.21.15.88:0/3333064221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff188156330 con 0x7ff1880735b0 2024-09-16T01:41:40.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:40.000+0000 7ff1902e4640 1 -- 172.21.15.88:0/3333064221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff18806c800 con 0x7ff1880735b0 2024-09-16T01:41:40.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:40.001+0000 7ff17effd640 1 -- 172.21.15.88:0/3333064221 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff17801b020 con 0x7ff1880735b0 2024-09-16T01:41:40.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:40.001+0000 7ff17effd640 1 --2- 172.21.15.88:0/3333064221 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff16c06c2b0 0x7ff16c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:40.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:40.001+0000 7ff17effd640 1 -- 172.21.15.88:0/3333064221 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 6770+0+0 (secure 0 0 0) 0x7ff17808b460 con 0x7ff1880735b0 2024-09-16T01:41:40.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:40.001+0000 7ff18e080640 1 --2- 172.21.15.88:0/3333064221 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff16c06c2b0 0x7ff16c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:40.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:40.001+0000 7ff18e080640 1 --2- 172.21.15.88:0/3333064221 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff16c06c2b0 0x7ff16c06e770 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff18815b710 tx=0x7ff184008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:40.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:40.005+0000 7ff17effd640 1 -- 172.21.15.88:0/3333064221 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff178059830 con 0x7ff1880735b0 2024-09-16T01:41:40.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:40.204+0000 7ff1902e4640 1 -- 172.21.15.88:0/3333064221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff18815d360 con 0x7ff1880735b0 2024-09-16T01:41:40.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:40.205+0000 7ff17effd640 1 -- 172.21.15.88:0/3333064221 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v147) ==== 53+0+73 (secure 0 0 0) 0x7ff178059130 con 0x7ff1880735b0 2024-09-16T01:41:40.206 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:41:40.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:40.206+0000 7ff1902e4640 1 -- 172.21.15.88:0/3333064221 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff16c06c2b0 msgr2=0x7ff16c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:40.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:40.206+0000 7ff1902e4640 1 --2- 172.21.15.88:0/3333064221 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff16c06c2b0 0x7ff16c06e770 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff18815b710 tx=0x7ff184008040 comp rx=0 tx=0).stop 2024-09-16T01:41:40.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:40.207+0000 7ff1902e4640 1 -- 172.21.15.88:0/3333064221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1880735b0 msgr2=0x7ff188155670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:40.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:40.207+0000 7ff1902e4640 1 --2- 172.21.15.88:0/3333064221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1880735b0 0x7ff188155670 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7ff17800d5f0 tx=0x7ff17800dab0 comp rx=0 tx=0).stop 2024-09-16T01:41:40.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:40.207+0000 7ff1902e4640 1 -- 172.21.15.88:0/3333064221 shutdown_connections 2024-09-16T01:41:40.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:40.207+0000 7ff1902e4640 1 --2- 172.21.15.88:0/3333064221 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff16c06c2b0 0x7ff16c06e770 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:40.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:40.207+0000 7ff1902e4640 1 --2- 172.21.15.88:0/3333064221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff18815e950 0x7ff188159be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:40.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:40.207+0000 7ff1902e4640 1 --2- 172.21.15.88:0/3333064221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1880735b0 0x7ff188155670 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:40.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:40.207+0000 7ff1902e4640 1 --2- 172.21.15.88:0/3333064221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff188072050 0x7ff188154340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:40.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:40.207+0000 7ff1902e4640 1 -- 172.21.15.88:0/3333064221 >> 172.21.15.88:0/3333064221 conn(0x7ff1880d14d0 msgr2=0x7ff188072480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:40.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:40.207+0000 7ff1902e4640 1 -- 172.21.15.88:0/3333064221 shutdown_connections 2024-09-16T01:41:40.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:40.207+0000 7ff1902e4640 1 -- 172.21.15.88:0/3333064221 wait complete. 2024-09-16T01:41:40.219 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-16T01:41:40.219 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:41:40.219 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2024-09-16T01:41:40.219 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T01:41:40.219 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T01:41:40.219 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-16T01:41:40.219 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-16T01:41:40.219 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:41:40.219 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:41:40.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:40.220 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2024-09-16T01:41:40.235 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:41:40.235 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:41:40.235 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:41:40.236 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:41:40.236 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-09-16T01:41:40.236 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:41:40.236 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:41:40.236 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:41:40.236 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2024-09-16T01:41:40.236 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:41:40.236 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:41:40.236 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2024-09-16T01:41:40.236 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:41:40.237 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2024-09-16T01:41:40.237 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:41:40.237 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2024-09-16T01:41:40.237 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:41:40.237 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42324sec preferred_lft 42324sec 2024-09-16T01:41:40.237 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link noprefixroute 2024-09-16T01:41:40.237 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:41:40.237 INFO:teuthology.orchestra.run.smithi088.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:41:40.237 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff 2024-09-16T01:41:40.237 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T01:41:40.237 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:41:40.237 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::e486:6eff:fe81:a1c/64 scope link 2024-09-16T01:41:40.237 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:41:40.238 INFO:teuthology.orchestra.run.smithi088.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:41:40.238 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 12:6f:29:be:95:48 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:41:40.238 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::106f:29ff:febe:9548/64 scope link 2024-09-16T01:41:40.238 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:41:40.238 INFO:teuthology.orchestra.run.smithi088.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:41:40.238 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T01:41:40.238 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::c63:b4ff:feb3:ad9e/64 scope link 2024-09-16T01:41:40.238 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:41:40.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:40.238 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:41:40.300 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-09-16T01:41:40.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:40.301 DEBUG:teuthology.orchestra.run.smithi088:> route 2024-09-16T01:41:40.359 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2024-09-16T01:41:40.359 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:41:40.359 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:41:40.359 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:41:40.359 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:41:40.360 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:41:40.360 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:41:40.360 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:41:40.360 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:41:40.361 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:41:40.361 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:41:40.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:41:40.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:41:40.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:40.485 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:41:40.500 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T01:41:40.500 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T01:41:40.500 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:41:40.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:40.501 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2024-09-16T01:41:40.565 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:41:40.565 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T01:41:40.620 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:41:40.621 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:41:40.621 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:41:40.676 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-16T01:41:40.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:40.676 DEBUG:teuthology.orchestra.run.smithi088:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:41:40.759 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:41:40.760 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:41:40.760 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:41:40.768 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:41:40.768 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:41:40.768 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:41:40.769 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:41:40.769 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:41:40.775 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:41:40.775 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:41:40.775 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:41:40.775 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:41:40.775 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:41:40.775 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790" 2024-09-16T01:41:40.776 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2024-09-16T01:41:40.776 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5ace2c91-46f9-482b-8a89-731fbdc171bf" will pass to kernel 2024-09-16T01:41:40.776 INFO:teuthology.orchestra.run.smithi088.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-16T01:41:40.776 INFO:teuthology.orchestra.run.smithi088.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:41:40.776 INFO:teuthology.orchestra.run.smithi088.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:41:40.776 INFO:teuthology.orchestra.run.smithi088.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:41:40.776 INFO:teuthology.orchestra.run.smithi088.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:41:40.777 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:41:40.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:40.778 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:41:40.843 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:41:40.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:40 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:41:40.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:41:40.940 INFO:teuthology.orchestra.run.smithi088.stdout:client6449 2024-09-16T01:41:40.940 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:41:40.941 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:41:40.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:40.957 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R .) 2024-09-16T01:41:40.999 INFO:teuthology.orchestra.run:waiting for 20 2024-09-16T01:41:41.013 INFO:teuthology.orchestra.run.smithi088.stdout:.: 2024-09-16T01:41:41.013 INFO:teuthology.orchestra.run.smithi088.stdout:subdir 2024-09-16T01:41:41.013 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:41.013 INFO:teuthology.orchestra.run.smithi088.stdout:./subdir: 2024-09-16T01:41:41.013 INFO:teuthology.orchestra.run.smithi088.stdout:sixmegs 2024-09-16T01:41:41.014 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 400.00000000' did not caused DAMAGED state 2024-09-16T01:41:41.014 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 100.00000000.inode' 2024-09-16T01:41:41.014 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:41:41.014 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:41:41.069 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:41:41.069 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:41:41.126 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:41:41.246 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:41:41.245+0000 7f9f1c79a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:41:41.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.264+0000 7efebeffd640 1 Processor -- start 2024-09-16T01:41:41.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.264+0000 7efebeffd640 1 -- start start 2024-09-16T01:41:41.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.265+0000 7efebeffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efeb80b6290 0x7efeb8071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:41.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.265+0000 7efebeffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efeb8072530 0x7efeb815bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:41.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.265+0000 7efebeffd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efeb80ccff0 con 0x7efeb815b480 2024-09-16T01:41:41.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.265+0000 7efebeffd640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7efeb80cd2d0 con 0x7efeb80b6290 2024-09-16T01:41:41.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.265+0000 7efebeffd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efeb80cd5b0 con 0x7efeb8072530 2024-09-16T01:41:41.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.265+0000 7efebcd99640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7efeb815b480 0x7efeb80b5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34660/0 (socket says 172.21.15.88:34660) 2024-09-16T01:41:41.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.265+0000 7efebcd99640 1 -- 172.21.15.88:0/4203765347 learned_addr learned my addr 172.21.15.88:0/4203765347 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:41.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.265+0000 7efeb7fff640 1 --2- 172.21.15.88:0/4203765347 >> v2:172.21.15.88:3300/0 conn(0x7efeb80b6290 0x7efeb8071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:41.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.266+0000 7efeb7fff640 1 -- 172.21.15.88:0/4203765347 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efeb8072530 msgr2=0x7efeb815bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:41.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.266+0000 7efeb7fff640 1 --2- 172.21.15.88:0/4203765347 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efeb8072530 0x7efeb815bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:41.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.266+0000 7efeb7fff640 1 -- 172.21.15.88:0/4203765347 >> v1:172.21.15.88:6789/0 conn(0x7efeb815b480 legacy=0x7efeb80b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:41.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.266+0000 7efeb7fff640 1 -- 172.21.15.88:0/4203765347 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeb80cd890 con 0x7efeb80b6290 2024-09-16T01:41:41.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.266+0000 7efeb77fe640 1 -- 172.21.15.88:0/4203765347 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3332744598 0 0) 0x7efeb80ccff0 con 0x7efeb815b480 2024-09-16T01:41:41.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.266+0000 7efeb7fff640 1 --2- 172.21.15.88:0/4203765347 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efeb80b6290 0x7efeb8071ff0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7efeac009d90 tx=0x7efeac02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9362e69236be01c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:41.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.266+0000 7efeb77fe640 1 -- 172.21.15.88:0/4203765347 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac031600 con 0x7efeb80b6290 2024-09-16T01:41:41.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.266+0000 7efeb77fe640 1 -- 172.21.15.88:0/4203765347 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeac0317a0 con 0x7efeb80b6290 2024-09-16T01:41:41.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.266+0000 7efeb77fe640 1 -- 172.21.15.88:0/4203765347 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac031b70 con 0x7efeb80b6290 2024-09-16T01:41:41.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.267+0000 7efebeffd640 1 -- 172.21.15.88:0/4203765347 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efeb80b6290 msgr2=0x7efeb8071ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:41.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.267+0000 7efebeffd640 1 --2- 172.21.15.88:0/4203765347 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efeb80b6290 0x7efeb8071ff0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7efeac009d90 tx=0x7efeac02a5d0 comp rx=0 tx=0).stop 2024-09-16T01:41:41.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.267+0000 7efebeffd640 1 -- 172.21.15.88:0/4203765347 shutdown_connections 2024-09-16T01:41:41.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.267+0000 7efebeffd640 1 --2- 172.21.15.88:0/4203765347 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efeb8072530 0x7efeb815bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:41.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.267+0000 7efebeffd640 1 --2- 172.21.15.88:0/4203765347 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efeb80b6290 0x7efeb8071ff0 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:41.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.267+0000 7efebeffd640 1 -- 172.21.15.88:0/4203765347 >> 172.21.15.88:0/4203765347 conn(0x7efeb806d0f0 msgr2=0x7efeb806d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:41.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.267+0000 7efebeffd640 1 -- 172.21.15.88:0/4203765347 shutdown_connections 2024-09-16T01:41:41.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.267+0000 7efebeffd640 1 -- 172.21.15.88:0/4203765347 wait complete. 2024-09-16T01:41:41.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.268+0000 7efebeffd640 1 Processor -- start 2024-09-16T01:41:41.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.268+0000 7efebeffd640 1 -- start start 2024-09-16T01:41:41.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.269+0000 7efebeffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efeb8072530 0x7efeb8149af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:41.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.269+0000 7efebeffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efeb815b480 0x7efeb814a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:41.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.269+0000 7efebeffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efeb8149150 0x7efeb81470d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:41.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.269+0000 7efebeffd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7efeb815b860 con 0x7efeb815b480 2024-09-16T01:41:41.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.269+0000 7efebeffd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efeb80729a0 con 0x7efeb8149150 2024-09-16T01:41:41.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.269+0000 7efebeffd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efeb815b9e0 con 0x7efeb8072530 2024-09-16T01:41:41.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.269+0000 7efeb7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efeb815b480 0x7efeb814a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:41.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.269+0000 7efeb7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efeb815b480 0x7efeb814a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47310/0 (socket says 172.21.15.88:47310) 2024-09-16T01:41:41.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.269+0000 7efeb7fff640 1 -- 172.21.15.88:0/1977463641 learned_addr learned my addr 172.21.15.88:0/1977463641 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:41.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.270+0000 7efebcd99640 1 --2- 172.21.15.88:0/1977463641 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efeb8072530 0x7efeb8149af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:41.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.270+0000 7efebd59a640 1 --2- 172.21.15.88:0/1977463641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efeb8149150 0x7efeb81470d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:41.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.270+0000 7efeb7fff640 1 -- 172.21.15.88:0/1977463641 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efeb8072530 msgr2=0x7efeb8149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:41.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.270+0000 7efeb7fff640 1 --2- 172.21.15.88:0/1977463641 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efeb8072530 0x7efeb8149af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:41.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.270+0000 7efeb7fff640 1 -- 172.21.15.88:0/1977463641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efeb8149150 msgr2=0x7efeb81470d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:41.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.270+0000 7efeb7fff640 1 --2- 172.21.15.88:0/1977463641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efeb8149150 0x7efeb81470d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:41.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.270+0000 7efeb7fff640 1 -- 172.21.15.88:0/1977463641 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeb8145d20 con 0x7efeb815b480 2024-09-16T01:41:41.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.270+0000 7efebd59a640 1 --2- 172.21.15.88:0/1977463641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efeb8149150 0x7efeb81470d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:41.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.270+0000 7efeb7fff640 1 --2- 172.21.15.88:0/1977463641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efeb815b480 0x7efeb814a030 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7efeac009d90 tx=0x7efeac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:41.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.270+0000 7efeb57fa640 1 -- 172.21.15.88:0/1977463641 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac040070 con 0x7efeb815b480 2024-09-16T01:41:41.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.270+0000 7efeb57fa640 1 -- 172.21.15.88:0/1977463641 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeac030be0 con 0x7efeb815b480 2024-09-16T01:41:41.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.270+0000 7efebeffd640 1 -- 172.21.15.88:0/1977463641 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efeb8146000 con 0x7efeb815b480 2024-09-16T01:41:41.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.270+0000 7efeb57fa640 1 -- 172.21.15.88:0/1977463641 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac03c420 con 0x7efeb815b480 2024-09-16T01:41:41.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.270+0000 7efebeffd640 1 -- 172.21.15.88:0/1977463641 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efeb81465c0 con 0x7efeb815b480 2024-09-16T01:41:41.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.271+0000 7efebeffd640 1 -- 172.21.15.88:0/1977463641 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efeb8071ff0 con 0x7efeb815b480 2024-09-16T01:41:41.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.271+0000 7efeb57fa640 1 -- 172.21.15.88:0/1977463641 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7efeac039040 con 0x7efeb815b480 2024-09-16T01:41:41.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.272+0000 7efeb57fa640 1 --2- 172.21.15.88:0/1977463641 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efea406c2e0 0x7efea406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:41.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.272+0000 7efeb57fa640 1 -- 172.21.15.88:0/1977463641 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 6770+0+0 (secure 0 0 0) 0x7efeac0acce0 con 0x7efeb815b480 2024-09-16T01:41:41.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.272+0000 7efebcd99640 1 --2- 172.21.15.88:0/1977463641 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efea406c2e0 0x7efea406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:41.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.272+0000 7efebcd99640 1 --2- 172.21.15.88:0/1977463641 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efea406c2e0 0x7efea406e7a0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7efea8000f60 tx=0x7efea8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:41.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.276+0000 7efeb57fa640 1 -- 172.21.15.88:0/1977463641 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7efeac07b0e0 con 0x7efeb815b480 2024-09-16T01:41:41.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.455+0000 7efebeffd640 1 -- 172.21.15.88:0/1977463641 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7efeb80ccb50 con 0x7efeb815b480 2024-09-16T01:41:41.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.455+0000 7efeb57fa640 1 -- 172.21.15.88:0/1977463641 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 24 entries v99) ==== 81+0+1460 (secure 0 0 0) 0x7efeac07a9e0 con 0x7efeb815b480 2024-09-16T01:41:41.457 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/1162318664 2024-09-17T01:41:23.257870+0000 2024-09-16T01:41:41.457 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/293764774 2024-09-17T01:41:16.203758+0000 2024-09-16T01:41:41.457 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2918194823 2024-09-17T01:41:29.275696+0000 2024-09-16T01:41:41.457 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/293764774 2024-09-17T01:41:16.203758+0000 2024-09-16T01:41:41.457 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2957965650 2024-09-17T01:41:12.181485+0000 2024-09-16T01:41:41.457 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/1162318664 2024-09-17T01:41:23.257870+0000 2024-09-16T01:41:41.457 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/128058689 2024-09-17T01:41:10.163743+0000 2024-09-16T01:41:41.457 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2918194823 2024-09-17T01:41:29.275696+0000 2024-09-16T01:41:41.457 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/128058689 2024-09-17T01:41:10.163743+0000 2024-09-16T01:41:41.457 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/1042715824 2024-09-17T01:40:59.080515+0000 2024-09-16T01:41:41.457 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2957965650 2024-09-17T01:41:12.181485+0000 2024-09-16T01:41:41.457 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/4162274767 2024-09-17T01:40:11.522091+0000 2024-09-16T01:41:41.458 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2733458327 2024-09-17T01:40:27.314712+0000 2024-09-16T01:41:41.458 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2970185561 2024-09-17T01:40:38.255774+0000 2024-09-16T01:41:41.458 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/1289932173 2024-09-17T01:40:42.399399+0000 2024-09-16T01:41:41.458 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/3356432718 2024-09-17T01:40:32.344252+0000 2024-09-16T01:41:41.458 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/1042715824 2024-09-17T01:40:59.080515+0000 2024-09-16T01:41:41.458 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/3356432718 2024-09-17T01:40:32.344252+0000 2024-09-16T01:41:41.458 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2970185561 2024-09-17T01:40:38.255774+0000 2024-09-16T01:41:41.458 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2408579939 2024-09-17T01:40:46.434492+0000 2024-09-16T01:41:41.458 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/4162274767 2024-09-17T01:40:11.522091+0000 2024-09-16T01:41:41.459 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/1289932173 2024-09-17T01:40:42.399399+0000 2024-09-16T01:41:41.459 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2733458327 2024-09-17T01:40:27.314712+0000 2024-09-16T01:41:41.459 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2408579939 2024-09-17T01:40:46.434492+0000 2024-09-16T01:41:41.460 INFO:teuthology.orchestra.run.smithi088.stderr:listed 24 entries 2024-09-16T01:41:41.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.457+0000 7efebeffd640 1 -- 172.21.15.88:0/1977463641 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efea406c2e0 msgr2=0x7efea406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:41.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.457+0000 7efebeffd640 1 --2- 172.21.15.88:0/1977463641 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efea406c2e0 0x7efea406e7a0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7efea8000f60 tx=0x7efea8009040 comp rx=0 tx=0).stop 2024-09-16T01:41:41.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.457+0000 7efebeffd640 1 -- 172.21.15.88:0/1977463641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efeb815b480 msgr2=0x7efeb814a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:41.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.457+0000 7efebeffd640 1 --2- 172.21.15.88:0/1977463641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efeb815b480 0x7efeb814a030 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7efeac009d90 tx=0x7efeac004a40 comp rx=0 tx=0).stop 2024-09-16T01:41:41.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.457+0000 7efebeffd640 1 -- 172.21.15.88:0/1977463641 shutdown_connections 2024-09-16T01:41:41.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.457+0000 7efebeffd640 1 --2- 172.21.15.88:0/1977463641 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efea406c2e0 0x7efea406e7a0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:41.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.457+0000 7efebeffd640 1 --2- 172.21.15.88:0/1977463641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efeb8149150 0x7efeb81470d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:41.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.457+0000 7efebeffd640 1 --2- 172.21.15.88:0/1977463641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efeb815b480 0x7efeb814a030 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:41.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.457+0000 7efebeffd640 1 --2- 172.21.15.88:0/1977463641 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efeb8072530 0x7efeb8149af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:41.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.457+0000 7efebeffd640 1 -- 172.21.15.88:0/1977463641 >> 172.21.15.88:0/1977463641 conn(0x7efeb806d0f0 msgr2=0x7efeb8153c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:41.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.457+0000 7efebeffd640 1 -- 172.21.15.88:0/1977463641 shutdown_connections 2024-09-16T01:41:41.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.457+0000 7efebeffd640 1 -- 172.21.15.88:0/1977463641 wait complete. 2024-09-16T01:41:41.469 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-16T01:41:41.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:41.470 DEBUG:teuthology.orchestra.run.smithi088:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-09-16T01:41:41.502 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:41:41.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:41.502 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:41:41.558 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:41:41.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.695+0000 7f718fcdd640 1 Processor -- start 2024-09-16T01:41:41.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.695+0000 7f718fcdd640 1 -- start start 2024-09-16T01:41:41.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.696+0000 7f718fcdd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f718815b480 0x7f71880b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:41.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.696+0000 7f718fcdd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7188072700 0x7f718815bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:41.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.696+0000 7f718fcdd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71880ccff0 con 0x7f71880b60c0 2024-09-16T01:41:41.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.696+0000 7f718fcdd640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f71880b6500 con 0x7f7188072700 2024-09-16T01:41:41.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.696+0000 7f718fcdd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f71880cd2d0 con 0x7f718815b480 2024-09-16T01:41:41.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.696+0000 7f718d278640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f71880b60c0 0x7f7188071ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34668/0 (socket says 172.21.15.88:34668) 2024-09-16T01:41:41.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.696+0000 7f718d278640 1 -- 172.21.15.88:0/1982627285 learned_addr learned my addr 172.21.15.88:0/1982627285 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:41.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.696+0000 7f718e27a640 1 --2- 172.21.15.88:0/1982627285 >> v2:172.21.15.154:3300/0 conn(0x7f7188072700 0x7f718815bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:41.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.696+0000 7f718da79640 1 --2- 172.21.15.88:0/1982627285 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f718815b480 0x7f71880b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:41.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.696+0000 7f718ca77640 1 -- 172.21.15.88:0/1982627285 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3941593869 0 0) 0x7f71880ccff0 con 0x7f71880b60c0 2024-09-16T01:41:41.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.696+0000 7f718ca77640 1 -- 172.21.15.88:0/1982627285 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7174005730 con 0x7f71880b60c0 2024-09-16T01:41:41.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.696+0000 7f718ca77640 1 -- 172.21.15.88:0/1982627285 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2598955550 0 0) 0x7f7174005730 con 0x7f71880b60c0 2024-09-16T01:41:41.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.696+0000 7f718ca77640 1 -- 172.21.15.88:0/1982627285 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f718815b480 msgr2=0x7f71880b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:41.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.696+0000 7f718ca77640 1 --2- 172.21.15.88:0/1982627285 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f718815b480 0x7f71880b5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:41.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.696+0000 7f718ca77640 1 -- 172.21.15.88:0/1982627285 >> v2:172.21.15.154:3300/0 conn(0x7f7188072700 msgr2=0x7f718815bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:41.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.696+0000 7f718ca77640 1 --2- 172.21.15.88:0/1982627285 >> v2:172.21.15.154:3300/0 conn(0x7f7188072700 0x7f718815bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:41.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.696+0000 7f718ca77640 1 -- 172.21.15.88:0/1982627285 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71880cd5b0 con 0x7f71880b60c0 2024-09-16T01:41:41.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.696+0000 7f718ca77640 1 -- 172.21.15.88:0/1982627285 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f7178001670 con 0x7f71880b60c0 2024-09-16T01:41:41.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.696+0000 7f718ca77640 1 -- 172.21.15.88:0/1982627285 >> v1:172.21.15.88:6789/0 conn(0x7f71880b60c0 legacy=0x7f7188071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:41.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.696+0000 7f718ca77640 1 --2- 172.21.15.88:0/1982627285 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f718815b480 0x7f7174005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:41.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.696+0000 7f718ca77640 1 --2- 172.21.15.88:0/1982627285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7174006140 0x7f7174006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:41.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.696+0000 7f718ca77640 1 --2- 172.21.15.88:0/1982627285 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7174006f50 0x7f71740073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:41.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.696+0000 7f718ca77640 1 -- 172.21.15.88:0/1982627285 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f71880b6500 con 0x7f7174006f50 2024-09-16T01:41:41.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.696+0000 7f718ca77640 1 -- 172.21.15.88:0/1982627285 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f71880cd2d0 con 0x7f7174006140 2024-09-16T01:41:41.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.696+0000 7f718ca77640 1 -- 172.21.15.88:0/1982627285 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f717400dc20 con 0x7f718815b480 2024-09-16T01:41:41.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.697+0000 7f718e27a640 1 --2- 172.21.15.88:0/1982627285 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7174006f50 0x7f71740073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:41.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.697+0000 7f718d278640 1 --2- 172.21.15.88:0/1982627285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7174006140 0x7f7174006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:41.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.697+0000 7f718da79640 1 --2- 172.21.15.88:0/1982627285 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f718815b480 0x7f7174005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:41.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.697+0000 7f718e27a640 1 -- 172.21.15.88:0/1982627285 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f718815b480 msgr2=0x7f7174005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:41.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.697+0000 7f718e27a640 1 --2- 172.21.15.88:0/1982627285 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f718815b480 0x7f7174005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:41.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.697+0000 7f718e27a640 1 -- 172.21.15.88:0/1982627285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7174006140 msgr2=0x7f7174006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:41.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.697+0000 7f718e27a640 1 --2- 172.21.15.88:0/1982627285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7174006140 0x7f7174006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:41.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.697+0000 7f718e27a640 1 -- 172.21.15.88:0/1982627285 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7174005730 con 0x7f7174006f50 2024-09-16T01:41:41.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.697+0000 7f718da79640 1 --2- 172.21.15.88:0/1982627285 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f718815b480 0x7f7174005bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:41.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.698+0000 7f718e27a640 1 --2- 172.21.15.88:0/1982627285 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7174006f50 0x7f71740073d0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f7180009af0 tx=0x7f7174007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a9139065e2a2277 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:41.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.698+0000 7f718ca77640 1 -- 172.21.15.88:0/1982627285 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7180036070 con 0x7f7174006f50 2024-09-16T01:41:41.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.698+0000 7f718ca77640 1 -- 172.21.15.88:0/1982627285 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7180031400 con 0x7f7174006f50 2024-09-16T01:41:41.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.698+0000 7f718ca77640 1 -- 172.21.15.88:0/1982627285 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7180009690 con 0x7f7174006f50 2024-09-16T01:41:41.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.698+0000 7f718fcdd640 1 -- 172.21.15.88:0/1982627285 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7174006f50 msgr2=0x7f71740073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:41.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.698+0000 7f718fcdd640 1 --2- 172.21.15.88:0/1982627285 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7174006f50 0x7f71740073d0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f7180009af0 tx=0x7f7174007a40 comp rx=0 tx=0).stop 2024-09-16T01:41:41.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.698+0000 7f718e27a640 1 -- 172.21.15.88:0/1982627285 reap_dead start 2024-09-16T01:41:41.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.698+0000 7f718fcdd640 1 -- 172.21.15.88:0/1982627285 shutdown_connections 2024-09-16T01:41:41.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.698+0000 7f718fcdd640 1 -- 172.21.15.88:0/1982627285 >> 172.21.15.88:0/1982627285 conn(0x7f718806d0f0 msgr2=0x7f718806d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:41.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.698+0000 7f718fcdd640 1 -- 172.21.15.88:0/1982627285 shutdown_connections 2024-09-16T01:41:41.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.698+0000 7f718fcdd640 1 -- 172.21.15.88:0/1982627285 wait complete. 2024-09-16T01:41:41.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.699+0000 7f718fcdd640 1 Processor -- start 2024-09-16T01:41:41.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.699+0000 7f718fcdd640 1 -- start start 2024-09-16T01:41:41.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.700+0000 7f718fcdd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71880cae60 0x7f71880ab100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:41.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.700+0000 7f718fcdd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71880abd00 0x7f71880ac180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:41.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.700+0000 7f718fcdd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71881f7df0 0x7f71881f82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:41.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.700+0000 7f718fcdd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f718815b860 con 0x7f71880cae60 2024-09-16T01:41:41.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.700+0000 7f718fcdd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f71880cdb70 con 0x7f71880abd00 2024-09-16T01:41:41.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.700+0000 7f718fcdd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f718815b9e0 con 0x7f71881f7df0 2024-09-16T01:41:41.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.700+0000 7f718da79640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71880cae60 0x7f71880ab100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:41.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.700+0000 7f718da79640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71880cae60 0x7f71880ab100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47324/0 (socket says 172.21.15.88:47324) 2024-09-16T01:41:41.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.700+0000 7f718da79640 1 -- 172.21.15.88:0/4270696802 learned_addr learned my addr 172.21.15.88:0/4270696802 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:41.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.700+0000 7f718d278640 1 --2- 172.21.15.88:0/4270696802 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71880abd00 0x7f71880ac180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:41.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.701+0000 7f718e27a640 1 --2- 172.21.15.88:0/4270696802 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71881f7df0 0x7f71881f82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:41.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.701+0000 7f718da79640 1 -- 172.21.15.88:0/4270696802 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71881f7df0 msgr2=0x7f71881f82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:41.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.701+0000 7f718da79640 1 --2- 172.21.15.88:0/4270696802 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71881f7df0 0x7f71881f82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:41.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.701+0000 7f718da79640 1 -- 172.21.15.88:0/4270696802 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71880abd00 msgr2=0x7f71880ac180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:41.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.701+0000 7f718da79640 1 --2- 172.21.15.88:0/4270696802 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71880abd00 0x7f71880ac180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:41.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.701+0000 7f718da79640 1 -- 172.21.15.88:0/4270696802 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71880cc030 con 0x7f71880cae60 2024-09-16T01:41:41.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.701+0000 7f718d278640 1 --2- 172.21.15.88:0/4270696802 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71880abd00 0x7f71880ac180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:41.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.701+0000 7f718e27a640 1 --2- 172.21.15.88:0/4270696802 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71881f7df0 0x7f71881f82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:41.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.701+0000 7f718da79640 1 --2- 172.21.15.88:0/4270696802 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71880cae60 0x7f71880ab100 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f71840101c0 tx=0x7f7184010680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:41.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.701+0000 7f717e7fc640 1 -- 172.21.15.88:0/4270696802 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718401a030 con 0x7f71880cae60 2024-09-16T01:41:41.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.701+0000 7f717e7fc640 1 -- 172.21.15.88:0/4270696802 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7184003990 con 0x7f71880cae60 2024-09-16T01:41:41.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.701+0000 7f717e7fc640 1 -- 172.21.15.88:0/4270696802 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7184003d60 con 0x7f71880cae60 2024-09-16T01:41:41.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.701+0000 7f718fcdd640 1 -- 172.21.15.88:0/4270696802 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71880cb240 con 0x7f71880cae60 2024-09-16T01:41:41.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.701+0000 7f718fcdd640 1 -- 172.21.15.88:0/4270696802 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71880aa620 con 0x7f71880cae60 2024-09-16T01:41:41.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.702+0000 7f718fcdd640 1 -- 172.21.15.88:0/4270696802 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71880b5b80 con 0x7f71880cae60 2024-09-16T01:41:41.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.702+0000 7f717e7fc640 1 -- 172.21.15.88:0/4270696802 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f718401d070 con 0x7f71880cae60 2024-09-16T01:41:41.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.703+0000 7f717e7fc640 1 --2- 172.21.15.88:0/4270696802 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f716c06c2e0 0x7f716c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:41.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.703+0000 7f717e7fc640 1 -- 172.21.15.88:0/4270696802 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 6770+0+0 (secure 0 0 0) 0x7f718408c710 con 0x7f71880cae60 2024-09-16T01:41:41.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.703+0000 7f718d278640 1 --2- 172.21.15.88:0/4270696802 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f716c06c2e0 0x7f716c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:41.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.703+0000 7f718d278640 1 --2- 172.21.15.88:0/4270696802 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f716c06c2e0 0x7f716c06e7a0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f718815a6a0 tx=0x7f7178009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:41.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.707+0000 7f717e7fc640 1 -- 172.21.15.88:0/4270696802 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f71880cc030 con 0x7f71880cae60 2024-09-16T01:41:41.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.904+0000 7f718fcdd640 1 -- 172.21.15.88:0/4270696802 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f718815a3f0 con 0x7f71880cae60 2024-09-16T01:41:41.905 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:41:41.904+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:41:41.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.906+0000 7f717e7fc640 1 -- 172.21.15.88:0/4270696802 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v148) ==== 161+0+0 (secure 0 0 0) 0x7f718401de40 con 0x7f71880cae60 2024-09-16T01:41:41.907 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:41:41.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.908+0000 7f718fcdd640 1 -- 172.21.15.88:0/4270696802 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f716c06c2e0 msgr2=0x7f716c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:41.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.908+0000 7f718fcdd640 1 --2- 172.21.15.88:0/4270696802 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f716c06c2e0 0x7f716c06e7a0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f718815a6a0 tx=0x7f7178009040 comp rx=0 tx=0).stop 2024-09-16T01:41:41.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.908+0000 7f718fcdd640 1 -- 172.21.15.88:0/4270696802 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71880cae60 msgr2=0x7f71880ab100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:41.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.908+0000 7f718fcdd640 1 --2- 172.21.15.88:0/4270696802 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71880cae60 0x7f71880ab100 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f71840101c0 tx=0x7f7184010680 comp rx=0 tx=0).stop 2024-09-16T01:41:41.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.908+0000 7f718fcdd640 1 -- 172.21.15.88:0/4270696802 shutdown_connections 2024-09-16T01:41:41.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.908+0000 7f718fcdd640 1 --2- 172.21.15.88:0/4270696802 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f716c06c2e0 0x7f716c06e7a0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:41.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.908+0000 7f718fcdd640 1 --2- 172.21.15.88:0/4270696802 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f71881f7df0 0x7f71881f82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:41.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.908+0000 7f718fcdd640 1 --2- 172.21.15.88:0/4270696802 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71880abd00 0x7f71880ac180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:41.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.908+0000 7f718fcdd640 1 --2- 172.21.15.88:0/4270696802 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71880cae60 0x7f71880ab100 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:41.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.908+0000 7f718fcdd640 1 -- 172.21.15.88:0/4270696802 >> 172.21.15.88:0/4270696802 conn(0x7f718806d0f0 msgr2=0x7f718806d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:41.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.908+0000 7f718fcdd640 1 -- 172.21.15.88:0/4270696802 shutdown_connections 2024-09-16T01:41:41.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:41.908+0000 7f718fcdd640 1 -- 172.21.15.88:0/4270696802 wait complete. 2024-09-16T01:41:41.911 INFO:tasks.ceph.mds.c.smithi088.stderr: -75> 2024-09-16T01:41:41.245+0000 7f9f1c79a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:41:41.921 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:41:41.936 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:41:42.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.018+0000 7fd75a0c3640 1 Processor -- start 2024-09-16T01:41:42.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.018+0000 7fd75a0c3640 1 -- start start 2024-09-16T01:41:42.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.019+0000 7fd75a0c3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7540b68f0 0x7fd7541696f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:42.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.019+0000 7fd75a0c3640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7540cf860 con 0x7fd7540cf1f0 2024-09-16T01:41:42.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.019+0000 7fd75a0c3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7540cfb40 con 0x7fd75415e7a0 2024-09-16T01:41:42.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.019+0000 7fd75a0c3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd7540d2ca0 con 0x7fd7540b68f0 2024-09-16T01:41:42.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.019+0000 7fd752ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd7540cf1f0 0x7fd7540b61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34680/0 (socket says 172.21.15.88:34680) 2024-09-16T01:41:42.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.019+0000 7fd752ffd640 1 -- 172.21.15.88:0/182571748 learned_addr learned my addr 172.21.15.88:0/182571748 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:42.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.019+0000 7fd753fff640 1 --2- 172.21.15.88:0/182571748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7540b68f0 0x7fd7541696f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:42.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.019+0000 7fd7527fc640 1 -- 172.21.15.88:0/182571748 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 115736937 0 0) 0x7fd7540cf860 con 0x7fd7540cf1f0 2024-09-16T01:41:42.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.019+0000 7fd7527fc640 1 -- 172.21.15.88:0/182571748 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd73c003520 con 0x7fd7540cf1f0 2024-09-16T01:41:42.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.019+0000 7fd7527fc640 1 -- 172.21.15.88:0/182571748 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3181626484 0 0) 0x7fd73c003520 con 0x7fd7540cf1f0 2024-09-16T01:41:42.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.019+0000 7fd7527fc640 1 -- 172.21.15.88:0/182571748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7540b68f0 msgr2=0x7fd7541696f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:42.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.019+0000 7fd7527fc640 1 --2- 172.21.15.88:0/182571748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7540b68f0 0x7fd7541696f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:42.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.019+0000 7fd7527fc640 1 -- 172.21.15.88:0/182571748 >> v1:172.21.15.154:6789/0 conn(0x7fd75415e7a0 legacy=0x7fd75406caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:42.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.019+0000 7fd7527fc640 1 -- 172.21.15.88:0/182571748 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7540cfe20 con 0x7fd7540cf1f0 2024-09-16T01:41:42.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.019+0000 7fd7527fc640 1 -- 172.21.15.88:0/182571748 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fd748001670 con 0x7fd7540cf1f0 2024-09-16T01:41:42.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.019+0000 7fd7527fc640 1 -- 172.21.15.88:0/182571748 >> v1:172.21.15.88:6789/0 conn(0x7fd7540cf1f0 legacy=0x7fd7540b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:42.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.019+0000 7fd7527fc640 1 --2- 172.21.15.88:0/182571748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd73c0039e0 0x7fd73c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:42.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.019+0000 7fd7527fc640 1 --2- 172.21.15.88:0/182571748 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd73c004800 0x7fd73c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:42.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.019+0000 7fd7527fc640 1 --2- 172.21.15.88:0/182571748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7540b68f0 0x7fd73c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:42.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.019+0000 7fd7527fc640 1 -- 172.21.15.88:0/182571748 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd7540d2ca0 con 0x7fd73c004800 2024-09-16T01:41:42.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.019+0000 7fd7527fc640 1 -- 172.21.15.88:0/182571748 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c00bb20 con 0x7fd73c0039e0 2024-09-16T01:41:42.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.019+0000 7fd7527fc640 1 -- 172.21.15.88:0/182571748 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd73c00bd20 con 0x7fd7540b68f0 2024-09-16T01:41:42.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.019+0000 7fd752ffd640 1 --2- 172.21.15.88:0/182571748 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd73c004800 0x7fd73c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:42.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.019+0000 7fd7537fe640 1 --2- 172.21.15.88:0/182571748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd73c0039e0 0x7fd73c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:42.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.020+0000 7fd753fff640 1 --2- 172.21.15.88:0/182571748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7540b68f0 0x7fd73c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:42.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.020+0000 7fd752ffd640 1 -- 172.21.15.88:0/182571748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7540b68f0 msgr2=0x7fd73c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:42.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.020+0000 7fd752ffd640 1 --2- 172.21.15.88:0/182571748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7540b68f0 0x7fd73c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:42.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.020+0000 7fd752ffd640 1 -- 172.21.15.88:0/182571748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd73c0039e0 msgr2=0x7fd73c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:42.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.020+0000 7fd752ffd640 1 --2- 172.21.15.88:0/182571748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd73c0039e0 0x7fd73c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:42.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.020+0000 7fd752ffd640 1 -- 172.21.15.88:0/182571748 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd73c003520 con 0x7fd73c004800 2024-09-16T01:41:42.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.020+0000 7fd752ffd640 1 --2- 172.21.15.88:0/182571748 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd73c004800 0x7fd73c004c80 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fd74800c210 tx=0x7fd73c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea7584bde8fc13f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:42.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.020+0000 7fd7527fc640 1 -- 172.21.15.88:0/182571748 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd748007d30 con 0x7fd73c004800 2024-09-16T01:41:42.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.020+0000 7fd7527fc640 1 -- 172.21.15.88:0/182571748 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd748005ad0 con 0x7fd73c004800 2024-09-16T01:41:42.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.020+0000 7fd7527fc640 1 -- 172.21.15.88:0/182571748 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd748007070 con 0x7fd73c004800 2024-09-16T01:41:42.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.021+0000 7fd75a0c3640 1 -- 172.21.15.88:0/182571748 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd73c004800 msgr2=0x7fd73c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:42.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.021+0000 7fd75a0c3640 1 --2- 172.21.15.88:0/182571748 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd73c004800 0x7fd73c004c80 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fd74800c210 tx=0x7fd73c005830 comp rx=0 tx=0).stop 2024-09-16T01:41:42.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.021+0000 7fd753fff640 1 -- 172.21.15.88:0/182571748 reap_dead start 2024-09-16T01:41:42.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.021+0000 7fd75a0c3640 1 -- 172.21.15.88:0/182571748 shutdown_connections 2024-09-16T01:41:42.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.021+0000 7fd75a0c3640 1 -- 172.21.15.88:0/182571748 >> 172.21.15.88:0/182571748 conn(0x7fd7540d11e0 msgr2=0x7fd7540d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:42.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.021+0000 7fd75a0c3640 1 -- 172.21.15.88:0/182571748 shutdown_connections 2024-09-16T01:41:42.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.021+0000 7fd75a0c3640 1 -- 172.21.15.88:0/182571748 wait complete. 2024-09-16T01:41:42.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.022+0000 7fd75a0c3640 1 Processor -- start 2024-09-16T01:41:42.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.022+0000 7fd75a0c3640 1 -- start start 2024-09-16T01:41:42.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.023+0000 7fd75a0c3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd73c004800 0x7fd7540cab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:42.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.023+0000 7fd75a0c3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7541e9300 0x7fd7541e9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:42.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.023+0000 7fd75a0c3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd754160de0 0x7fd754161260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:42.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.023+0000 7fd75a0c3640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd7540d4420 con 0x7fd73c004800 2024-09-16T01:41:42.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.023+0000 7fd75a0c3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd7540b7a00 con 0x7fd754160de0 2024-09-16T01:41:42.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.023+0000 7fd75a0c3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd7541e8e70 con 0x7fd7541e9300 2024-09-16T01:41:42.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.023+0000 7fd7537fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd73c004800 0x7fd7540cab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:42.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.023+0000 7fd7537fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd73c004800 0x7fd7540cab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47348/0 (socket says 172.21.15.88:47348) 2024-09-16T01:41:42.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.023+0000 7fd7537fe640 1 -- 172.21.15.88:0/1320417499 learned_addr learned my addr 172.21.15.88:0/1320417499 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:42.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.023+0000 7fd7537fe640 1 -- 172.21.15.88:0/1320417499 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7541e9300 msgr2=0x7fd7541e9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:42.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.023+0000 7fd7537fe640 1 --2- 172.21.15.88:0/1320417499 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7541e9300 0x7fd7541e9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:42.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.023+0000 7fd7537fe640 1 -- 172.21.15.88:0/1320417499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd754160de0 msgr2=0x7fd754161260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:42.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.023+0000 7fd753fff640 1 --2- 172.21.15.88:0/1320417499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd754160de0 0x7fd754161260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:42.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.023+0000 7fd7537fe640 1 --2- 172.21.15.88:0/1320417499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd754160de0 0x7fd754161260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:42.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.024+0000 7fd7537fe640 1 -- 172.21.15.88:0/1320417499 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7540cb080 con 0x7fd73c004800 2024-09-16T01:41:42.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.024+0000 7fd753fff640 1 --2- 172.21.15.88:0/1320417499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd754160de0 0x7fd754161260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:42.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.024+0000 7fd7537fe640 1 --2- 172.21.15.88:0/1320417499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd73c004800 0x7fd7540cab40 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fd74400f5f0 tx=0x7fd74400fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:42.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.024+0000 7fd73bfff640 1 -- 172.21.15.88:0/1320417499 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7440052e0 con 0x7fd73c004800 2024-09-16T01:41:42.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.024+0000 7fd73bfff640 1 -- 172.21.15.88:0/1320417499 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd744018070 con 0x7fd73c004800 2024-09-16T01:41:42.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.024+0000 7fd75a0c3640 1 -- 172.21.15.88:0/1320417499 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7541e8260 con 0x7fd73c004800 2024-09-16T01:41:42.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.024+0000 7fd73bfff640 1 -- 172.21.15.88:0/1320417499 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd744003810 con 0x7fd73c004800 2024-09-16T01:41:42.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.024+0000 7fd75a0c3640 1 -- 172.21.15.88:0/1320417499 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7541e8820 con 0x7fd73c004800 2024-09-16T01:41:42.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.025+0000 7fd75a0c3640 1 -- 172.21.15.88:0/1320417499 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd75415eb80 con 0x7fd73c004800 2024-09-16T01:41:42.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.025+0000 7fd73bfff640 1 -- 172.21.15.88:0/1320417499 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd7440039b0 con 0x7fd73c004800 2024-09-16T01:41:42.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.026+0000 7fd73bfff640 1 --2- 172.21.15.88:0/1320417499 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd74006c2b0 0x7fd74006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:42.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.026+0000 7fd73bfff640 1 -- 172.21.15.88:0/1320417499 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 6856+0+0 (secure 0 0 0) 0x7fd74408b700 con 0x7fd73c004800 2024-09-16T01:41:42.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.026+0000 7fd752ffd640 1 --2- 172.21.15.88:0/1320417499 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd74006c2b0 0x7fd74006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:42.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.026+0000 7fd752ffd640 1 --2- 172.21.15.88:0/1320417499 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd74006c2b0 0x7fd74006e770 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd748033000 tx=0x7fd748009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:42.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.030+0000 7fd73bfff640 1 -- 172.21.15.88:0/1320417499 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd7440599f0 con 0x7fd73c004800 2024-09-16T01:41:42.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.264+0000 7fd75a0c3640 1 -- 172.21.15.88:0/1320417499 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fd7540d2a80 con 0x7fd73c004800 2024-09-16T01:41:42.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.907+0000 7fd73bfff640 1 -- 172.21.15.88:0/1320417499 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v149) ==== 107+0+0 (secure 0 0 0) 0x7fd7440592f0 con 0x7fd73c004800 2024-09-16T01:41:42.909 INFO:teuthology.orchestra.run.smithi088.stderr:nothing to do: rank is not damaged 2024-09-16T01:41:42.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.910+0000 7fd75a0c3640 1 -- 172.21.15.88:0/1320417499 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd74006c2b0 msgr2=0x7fd74006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:42.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.910+0000 7fd75a0c3640 1 --2- 172.21.15.88:0/1320417499 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd74006c2b0 0x7fd74006e770 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd748033000 tx=0x7fd748009030 comp rx=0 tx=0).stop 2024-09-16T01:41:42.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.910+0000 7fd75a0c3640 1 -- 172.21.15.88:0/1320417499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd73c004800 msgr2=0x7fd7540cab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:42.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.910+0000 7fd75a0c3640 1 --2- 172.21.15.88:0/1320417499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd73c004800 0x7fd7540cab40 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fd74400f5f0 tx=0x7fd74400fab0 comp rx=0 tx=0).stop 2024-09-16T01:41:42.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.910+0000 7fd75a0c3640 1 -- 172.21.15.88:0/1320417499 shutdown_connections 2024-09-16T01:41:42.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.910+0000 7fd75a0c3640 1 --2- 172.21.15.88:0/1320417499 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd74006c2b0 0x7fd74006e770 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:42.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.910+0000 7fd75a0c3640 1 --2- 172.21.15.88:0/1320417499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd754160de0 0x7fd754161260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:42.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.910+0000 7fd75a0c3640 1 --2- 172.21.15.88:0/1320417499 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7541e9300 0x7fd7541e9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:42.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.910+0000 7fd75a0c3640 1 --2- 172.21.15.88:0/1320417499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd73c004800 0x7fd7540cab40 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:42.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.910+0000 7fd75a0c3640 1 -- 172.21.15.88:0/1320417499 >> 172.21.15.88:0/1320417499 conn(0x7fd7540d11e0 msgr2=0x7fd75415d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:42.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.910+0000 7fd75a0c3640 1 -- 172.21.15.88:0/1320417499 shutdown_connections 2024-09-16T01:41:42.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:42.910+0000 7fd75a0c3640 1 -- 172.21.15.88:0/1320417499 wait complete. 2024-09-16T01:41:42.924 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:41:43.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.008+0000 7f41ee1b8f40 1 Processor -- start 2024-09-16T01:41:43.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.008+0000 7f41ee1b8f40 1 -- start start 2024-09-16T01:41:43.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.009+0000 7f41ee1b8f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561caa963e10 0x561caa9fe710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.009+0000 7f41ee1b8f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561caa964a60 0x561caaa0bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.009+0000 7f41ee1b8f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561caa97e250 0x561caaa02ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.009+0000 7f41ee1b8f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x561caa886510 con 0x561caa97e250 2024-09-16T01:41:43.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.009+0000 7f41ee1b8f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561caaa0fac0 con 0x561caa964a60 2024-09-16T01:41:43.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.009+0000 7f41ee1b8f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x561caaa0f110 con 0x561caa963e10 2024-09-16T01:41:43.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.009+0000 7f41ec4b1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561caa97e250 0x561caaa02ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:43.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.009+0000 7f41ec4b1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561caa97e250 0x561caaa02ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47376/0 (socket says 172.21.15.88:47376) 2024-09-16T01:41:43.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.009+0000 7f41ec4b1640 1 -- 172.21.15.88:0/585014693 learned_addr learned my addr 172.21.15.88:0/585014693 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:43.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.009+0000 7f41ec4b1640 1 -- 172.21.15.88:0/585014693 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561caa963e10 msgr2=0x561caa9fe710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:43.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.009+0000 7f41ec4b1640 1 --2- 172.21.15.88:0/585014693 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561caa963e10 0x561caa9fe710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.010+0000 7f41ec4b1640 1 -- 172.21.15.88:0/585014693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561caa964a60 msgr2=0x561caaa0bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:43.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.010+0000 7f41ec4b1640 1 --2- 172.21.15.88:0/585014693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561caa964a60 0x561caaa0bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.010+0000 7f41ec4b1640 1 -- 172.21.15.88:0/585014693 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561caa9fee50 con 0x561caa97e250 2024-09-16T01:41:43.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.010+0000 7f41ec4b1640 1 --2- 172.21.15.88:0/585014693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561caa97e250 0x561caaa02ef0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f41e400a4b0 tx=0x7f41e400a970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b975c856e1933993 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:43.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.010+0000 7f41eacae640 1 -- 172.21.15.88:0/585014693 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e400f7b0 con 0x561caa97e250 2024-09-16T01:41:43.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.010+0000 7f41eacae640 1 -- 172.21.15.88:0/585014693 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41e4015070 con 0x561caa97e250 2024-09-16T01:41:43.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.010+0000 7f41eacae640 1 -- 172.21.15.88:0/585014693 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e400fb70 con 0x561caa97e250 2024-09-16T01:41:43.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.010+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/585014693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561caa97e250 msgr2=0x561caaa02ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:43.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.010+0000 7f41ee1b8f40 1 --2- 172.21.15.88:0/585014693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561caa97e250 0x561caaa02ef0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f41e400a4b0 tx=0x7f41e400a970 comp rx=0 tx=0).stop 2024-09-16T01:41:43.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.010+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/585014693 shutdown_connections 2024-09-16T01:41:43.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.010+0000 7f41ee1b8f40 1 --2- 172.21.15.88:0/585014693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561caa97e250 0x561caaa02ef0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.010+0000 7f41ee1b8f40 1 --2- 172.21.15.88:0/585014693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561caa964a60 0x561caaa0bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.010+0000 7f41ee1b8f40 1 --2- 172.21.15.88:0/585014693 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561caa963e10 0x561caa9fe710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.010+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/585014693 >> 172.21.15.88:0/585014693 conn(0x561caa97f750 msgr2=0x561caaa0e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:43.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.010+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/585014693 shutdown_connections 2024-09-16T01:41:43.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.010+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/585014693 wait complete. 2024-09-16T01:41:43.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.011+0000 7f41ee1b8f40 1 Processor -- start 2024-09-16T01:41:43.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.011+0000 7f41ee1b8f40 1 -- start start 2024-09-16T01:41:43.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.012+0000 7f41ee1b8f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561caa963e10 0x561caa9faf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.012+0000 7f41ee1b8f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561caa964a60 0x561caa97d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.012+0000 7f41ee1b8f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561caa97e250 0x561caaa0e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.012+0000 7f41ee1b8f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x561caaa0bff0 con 0x561caa963e10 2024-09-16T01:41:43.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.012+0000 7f41ee1b8f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561caa963690 con 0x561caa97e250 2024-09-16T01:41:43.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.012+0000 7f41ee1b8f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x561caaa0c170 con 0x561caa964a60 2024-09-16T01:41:43.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.012+0000 7f41ebcb0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561caa963e10 0x561caa9faf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:43.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.012+0000 7f41ebcb0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561caa963e10 0x561caa9faf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47386/0 (socket says 172.21.15.88:47386) 2024-09-16T01:41:43.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.012+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 learned_addr learned my addr 172.21.15.88:0/175841998 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:43.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.012+0000 7f41ec4b1640 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561caa97e250 0x561caaa0e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:43.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.012+0000 7f41e3fff640 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561caa964a60 0x561caa97d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:43.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.012+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561caa964a60 msgr2=0x561caa97d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:43.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.012+0000 7f41ebcb0640 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561caa964a60 0x561caa97d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.012+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561caa97e250 msgr2=0x561caaa0e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:43.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.012+0000 7f41ebcb0640 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561caa97e250 0x561caaa0e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.012+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561caa9ff3c0 con 0x561caa963e10 2024-09-16T01:41:43.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.012+0000 7f41e3fff640 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561caa964a60 0x561caa97d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:43.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.012+0000 7f41ec4b1640 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561caa97e250 0x561caaa0e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:43.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.012+0000 7f41ebcb0640 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561caa963e10 0x561caa9faf70 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f41dc00bd10 tx=0x7f41dc009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:43.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.012+0000 7f41e9cac640 1 -- 172.21.15.88:0/175841998 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41dc008070 con 0x561caa963e10 2024-09-16T01:41:43.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.012+0000 7f41e9cac640 1 -- 172.21.15.88:0/175841998 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41dc00f650 con 0x561caa963e10 2024-09-16T01:41:43.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.012+0000 7f41e9cac640 1 -- 172.21.15.88:0/175841998 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41dc00fa40 con 0x561caa963e10 2024-09-16T01:41:43.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.012+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x561caa97cc40 con 0x561caa963e10 2024-09-16T01:41:43.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.012+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561caa97d200 con 0x561caa963e10 2024-09-16T01:41:43.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.014+0000 7f41e9cac640 1 -- 172.21.15.88:0/175841998 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f41dc00fbe0 con 0x561caa963e10 2024-09-16T01:41:43.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.014+0000 7f41e9cac640 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f41cc06ac90 0x7f41cc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.014+0000 7f41e3fff640 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f41cc06ac90 0x7f41cc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:43.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.014+0000 7f41e9cac640 1 -- 172.21.15.88:0/175841998 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 6856+0+0 (secure 0 0 0) 0x7f41dc08a590 con 0x561caa963e10 2024-09-16T01:41:43.019 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:41:43.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.015+0000 7f41ee1b8f40 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x561caaa0e7c0 0x561caaa00f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.015+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaa014b0 con 0x561caaa0e7c0 2024-09-16T01:41:43.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.015+0000 7f41ec4b1640 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x561caaa0e7c0 0x561caaa00f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:43.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.015+0000 7f41e3fff640 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f41cc06ac90 0x7f41cc06d150 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f41d4057000 tx=0x7f41d4051000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:43.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.015+0000 7f41ec4b1640 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x561caaa0e7c0 0x561caaa00f70 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:43.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.017+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v100'381 uv355 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.017+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e100) -- 0x561caaaa6280 con 0x561caaa0e7c0 2024-09-16T01:41:43.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.018+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v100'382 uv382 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.018+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaaa66b0 con 0x561caaa0e7c0 2024-09-16T01:41:43.021 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:41:43.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.019+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v100'383 uv383 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.019+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaaa6a50 con 0x561caaa0e7c0 2024-09-16T01:41:43.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.021+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v100'384 uv384 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.021+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaaa6e50 con 0x561caaa0e7c0 2024-09-16T01:41:43.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.022+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v100'385 uv385 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.022+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaaaf550 con 0x561caaa0e7c0 2024-09-16T01:41:43.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.023+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v100'386 uv386 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.023+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaaaf8f0 con 0x561caaa0e7c0 2024-09-16T01:41:43.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.024+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v100'387 uv361 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.024+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e100) -- 0x561caaaafc90 con 0x561caaa0e7c0 2024-09-16T01:41:43.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.025+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v100'388 uv388 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.025+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaaafc90 con 0x561caaa0e7c0 2024-09-16T01:41:43.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.026+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v100'389 uv389 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.028 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:41:43.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.026+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab0ca0 con 0x561caaa0e7c0 2024-09-16T01:41:43.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.028+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v100'390 uv390 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.028+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab0ca0 con 0x561caaa0e7c0 2024-09-16T01:41:43.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.029+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v100'391 uv391 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.029+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaaafc90 con 0x561caaa0e7c0 2024-09-16T01:41:43.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.030+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v100'392 uv392 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.030+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaaafc90 con 0x561caaa0e7c0 2024-09-16T01:41:43.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.031+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v100'393 uv368 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.031+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e100) -- 0x561caaaafc90 con 0x561caaa0e7c0 2024-09-16T01:41:43.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.032+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v100'394 uv394 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.032+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaaafc90 con 0x561caaa0e7c0 2024-09-16T01:41:43.035 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:41:43.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.034+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v100'395 uv395 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.034+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab0ca0 con 0x561caaa0e7c0 2024-09-16T01:41:43.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.035+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v100'396 uv396 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.035+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab0ca0 con 0x561caaa0e7c0 2024-09-16T01:41:43.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.036+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v100'397 uv397 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.036+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaaafc90 con 0x561caaa0e7c0 2024-09-16T01:41:43.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.037+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v100'398 uv398 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.037+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab03f0 con 0x561caaa0e7c0 2024-09-16T01:41:43.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.039+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v100'399 uv399 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.039+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab03f0 con 0x561caaa0e7c0 2024-09-16T01:41:43.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.040+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v100'400 uv374 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.040+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e100) -- 0x561caaab03f0 con 0x561caaa0e7c0 2024-09-16T01:41:43.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.041+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v100'401 uv401 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.041+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab03f0 con 0x561caaa0e7c0 2024-09-16T01:41:43.043 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:41:43.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.042+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v100'402 uv402 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.042+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab1400 con 0x561caaa0e7c0 2024-09-16T01:41:43.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.043+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v100'403 uv403 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.043+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab1400 con 0x561caaa0e7c0 2024-09-16T01:41:43.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.044+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v100'404 uv404 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.044+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab03f0 con 0x561caaa0e7c0 2024-09-16T01:41:43.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.045+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v100'405 uv405 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.045+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab03f0 con 0x561caaa0e7c0 2024-09-16T01:41:43.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.047+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v100'406 uv380 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.047+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e100) -- 0x561caaab03f0 con 0x561caaa0e7c0 2024-09-16T01:41:43.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.048+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v100'407 uv407 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.048+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab03f0 con 0x561caaa0e7c0 2024-09-16T01:41:43.050 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:41:43.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.049+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v100'408 uv408 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.049+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab1400 con 0x561caaa0e7c0 2024-09-16T01:41:43.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.050+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v100'409 uv409 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.050+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab1400 con 0x561caaa0e7c0 2024-09-16T01:41:43.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.052+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v100'410 uv410 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.052+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab03f0 con 0x561caaa0e7c0 2024-09-16T01:41:43.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.053+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v100'411 uv411 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.053+0000 7f41ee1b8f40 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x561caaab0900 0x561caaab2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.053+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab3230 con 0x561caaab0900 2024-09-16T01:41:43.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.053+0000 7f41ebcb0640 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x561caaab0900 0x561caaab2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:43.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.054+0000 7f41ebcb0640 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x561caaab0900 0x561caaab2cf0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:43.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.055+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v100'288 uv267 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f41dc054930 con 0x561caaab0900 2024-09-16T01:41:43.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.055+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e100) -- 0x561caaab3610 con 0x561caaab0900 2024-09-16T01:41:43.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.056+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v100'289 uv289 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f41dc054930 con 0x561caaab0900 2024-09-16T01:41:43.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.056+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab3b10 con 0x561caaab0900 2024-09-16T01:41:43.058 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:41:43.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.057+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v100'290 uv290 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f41dc054930 con 0x561caaab0900 2024-09-16T01:41:43.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.057+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab43d0 con 0x561caaab0900 2024-09-16T01:41:43.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.059+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v100'291 uv291 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f41dc054930 con 0x561caaab0900 2024-09-16T01:41:43.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.059+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab4770 con 0x561caaab0900 2024-09-16T01:41:43.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.060+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v100'292 uv292 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f41dc054930 con 0x561caaab0900 2024-09-16T01:41:43.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.060+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab4b10 con 0x561caaab0900 2024-09-16T01:41:43.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.061+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v100'293 uv286 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f41dc054930 con 0x561caaab0900 2024-09-16T01:41:43.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.061+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e100) -- 0x561caaab4eb0 con 0x561caaab0900 2024-09-16T01:41:43.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.062+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v100'294 uv294 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41dc054930 con 0x561caaab0900 2024-09-16T01:41:43.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.062+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab4eb0 con 0x561caaab0900 2024-09-16T01:41:43.065 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:41:43.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.064+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v100'295 uv295 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41dc054930 con 0x561caaab0900 2024-09-16T01:41:43.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.064+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab4eb0 con 0x561caaab0900 2024-09-16T01:41:43.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.065+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v100'296 uv296 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41dc054930 con 0x561caaab0900 2024-09-16T01:41:43.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.065+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab4eb0 con 0x561caaab0900 2024-09-16T01:41:43.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.066+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v100'297 uv297 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41dc054930 con 0x561caaab0900 2024-09-16T01:41:43.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.066+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab4eb0 con 0x561caaab0900 2024-09-16T01:41:43.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.067+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v100'298 uv287 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f41dc088040 con 0x561caaab0900 2024-09-16T01:41:43.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.067+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e100) -- 0x561caaab4eb0 con 0x561caaab0900 2024-09-16T01:41:43.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.068+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v100'299 uv299 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41dc088040 con 0x561caaab0900 2024-09-16T01:41:43.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.068+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab4eb0 con 0x561caaab0900 2024-09-16T01:41:43.070 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:41:43.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.069+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v100'300 uv300 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41dc088040 con 0x561caaab0900 2024-09-16T01:41:43.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.069+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab9330 con 0x561caaab0900 2024-09-16T01:41:43.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.070+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v100'301 uv301 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41dc088040 con 0x561caaab0900 2024-09-16T01:41:43.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.070+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab9330 con 0x561caaab0900 2024-09-16T01:41:43.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.071+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v100'302 uv302 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41dc088040 con 0x561caaab0900 2024-09-16T01:41:43.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.071+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab4eb0 con 0x561caaab0900 2024-09-16T01:41:43.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.073+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v100'303 uv283 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f41dc088040 con 0x561caaab0900 2024-09-16T01:41:43.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.073+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e100) -- 0x561caaab4eb0 con 0x561caaab0900 2024-09-16T01:41:43.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.074+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v100'304 uv304 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41dc088040 con 0x561caaab0900 2024-09-16T01:41:43.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.074+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab4eb0 con 0x561caaab0900 2024-09-16T01:41:43.076 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:41:43.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.075+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v100'305 uv305 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41dc088040 con 0x561caaab0900 2024-09-16T01:41:43.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.075+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab5ec0 con 0x561caaab0900 2024-09-16T01:41:43.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.078+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v100'306 uv306 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41dc088040 con 0x561caaab0900 2024-09-16T01:41:43.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.078+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab5ec0 con 0x561caaab0900 2024-09-16T01:41:43.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.079+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v100'307 uv307 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41dc088040 con 0x561caaab0900 2024-09-16T01:41:43.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.079+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab4eb0 con 0x561caaab0900 2024-09-16T01:41:43.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.080+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v100'308 uv308 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41dc088040 con 0x561caaab0900 2024-09-16T01:41:43.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.080+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab4eb0 con 0x561caaa0e7c0 2024-09-16T01:41:43.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.081+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v100'75 uv74 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.081+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e100) -- 0x561caaab4eb0 con 0x561caaa0e7c0 2024-09-16T01:41:43.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.082+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v100'76 uv76 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.082+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab4eb0 con 0x561caaa0e7c0 2024-09-16T01:41:43.084 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:41:43.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.083+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v100'77 uv77 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.083+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab4eb0 con 0x561caaa0e7c0 2024-09-16T01:41:43.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.084+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v100'78 uv78 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.084+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab5ec0 con 0x561caaa0e7c0 2024-09-16T01:41:43.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.085+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v100'79 uv79 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.085+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab4eb0 con 0x561caaa0e7c0 2024-09-16T01:41:43.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.086+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v100'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.086+0000 7f41ee1b8f40 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x561caaab5530 0x561caaab59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.086+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab82d0 con 0x561caaab5530 2024-09-16T01:41:43.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.086+0000 7f41e3fff640 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x561caaab5530 0x561caaab59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:43.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.086+0000 7f41e3fff640 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x561caaab5530 0x561caaab59e0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:43.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.087+0000 7f41e3fff640 1 -- 172.21.15.88:0/175841998 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v100'61 uv60 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f41d4064070 con 0x561caaab5530 2024-09-16T01:41:43.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.087+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e100) -- 0x561caaab8670 con 0x561caaab5530 2024-09-16T01:41:43.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.088+0000 7f41e3fff640 1 -- 172.21.15.88:0/175841998 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v100'62 uv62 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f41d4064070 con 0x561caaab5530 2024-09-16T01:41:43.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.088+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab8a30 con 0x561caaab5530 2024-09-16T01:41:43.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.089+0000 7f41e3fff640 1 -- 172.21.15.88:0/175841998 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v100'63 uv63 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f41d4064070 con 0x561caaab5530 2024-09-16T01:41:43.090 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:41:43.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.089+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab4000 con 0x561caaab5530 2024-09-16T01:41:43.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.092+0000 7f41e3fff640 1 -- 172.21.15.88:0/175841998 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v100'64 uv64 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f41d4064070 con 0x561caaab5530 2024-09-16T01:41:43.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.092+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab9360 con 0x561caaa0e7c0 2024-09-16T01:41:43.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.093+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v100'75 uv74 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.093+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e100) -- 0x561caaab9360 con 0x561caaa0e7c0 2024-09-16T01:41:43.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.094+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v100'76 uv76 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.094+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab9360 con 0x561caaa0e7c0 2024-09-16T01:41:43.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.095+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v100'77 uv77 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.096 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:41:43.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.095+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab9360 con 0x561caaa0e7c0 2024-09-16T01:41:43.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.098+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v100'78 uv78 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.098+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab9360 con 0x561caaa0e7c0 2024-09-16T01:41:43.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.101+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v100'79 uv79 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.101+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab9360 con 0x561caaa0e7c0 2024-09-16T01:41:43.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.107+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v100'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.107+0000 7f41ee1b8f40 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x561caaab9c60 0x561caaabc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.107+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaabc590 con 0x561caaab9c60 2024-09-16T01:41:43.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.107+0000 7f41ec4b1640 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x561caaab9c60 0x561caaabc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:43.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.108+0000 7f41ec4b1640 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x561caaab9c60 0x561caaabc050 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:43.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.109+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v100'138 uv132 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.109+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e100) -- 0x561caaabc590 con 0x561caaab9c60 2024-09-16T01:41:43.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.110+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v100'139 uv139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.110+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaabc9b0 con 0x561caaab9c60 2024-09-16T01:41:43.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.111+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v100'140 uv140 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.113 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:41:43.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.111+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaabd9c0 con 0x561caaab9c60 2024-09-16T01:41:43.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.112+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v100'141 uv141 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.112+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaabd9c0 con 0x561caaab9c60 2024-09-16T01:41:43.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.113+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v100'142 uv142 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.113+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaabc9b0 con 0x561caaab9c60 2024-09-16T01:41:43.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.114+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v100'143 uv143 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.114+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaabc9b0 con 0x561caaab9c60 2024-09-16T01:41:43.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.115+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v100'144 uv137 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.115+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e100) -- 0x561caaabc9b0 con 0x561caaab9c60 2024-09-16T01:41:43.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.116+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v100'145 uv145 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.116+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaabc9b0 con 0x561caaab9c60 2024-09-16T01:41:43.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.117+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v100'146 uv146 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.119 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:41:43.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.117+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaabc9b0 con 0x561caaab9c60 2024-09-16T01:41:43.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.118+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v100'147 uv147 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.118+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaabc9b0 con 0x561caaab9c60 2024-09-16T01:41:43.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.119+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v100'148 uv148 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.120+0000 7f41ee1b8f40 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x561caaabd030 0x561caaabf420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.120+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaabfa30 con 0x561caaabd030 2024-09-16T01:41:43.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.120+0000 7f41ebcb0640 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x561caaabd030 0x561caaabf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:43.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.120+0000 7f41ebcb0640 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x561caaabd030 0x561caaabf420 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:43.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.121+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v100'61 uv60 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f41dc088040 con 0x561caaabd030 2024-09-16T01:41:43.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.121+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e100) -- 0x561caaabfe20 con 0x561caaabd030 2024-09-16T01:41:43.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.122+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v100'62 uv62 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f41dc088040 con 0x561caaabd030 2024-09-16T01:41:43.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.122+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaabfe20 con 0x561caaabd030 2024-09-16T01:41:43.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.123+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v100'63 uv63 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f41dc088040 con 0x561caaabd030 2024-09-16T01:41:43.127 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:41:43.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.123+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaabfe20 con 0x561caaabd030 2024-09-16T01:41:43.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.124+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v100'64 uv64 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f41dc088040 con 0x561caaabd030 2024-09-16T01:41:43.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.124+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaabfe20 con 0x561caaabd030 2024-09-16T01:41:43.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.125+0000 7f41ebcb0640 1 -- 172.21.15.88:0/175841998 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v100'65 uv65 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f41dc088040 con 0x561caaabd030 2024-09-16T01:41:43.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.125+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaabfe20 con 0x561caaa0e7c0 2024-09-16T01:41:43.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.126+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v100'75 uv74 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.126+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e100) -- 0x561caaabfe20 con 0x561caaa0e7c0 2024-09-16T01:41:43.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.127+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v100'76 uv76 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.127+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaabfe20 con 0x561caaa0e7c0 2024-09-16T01:41:43.132 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:41:43.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.128+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v100'77 uv77 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.128+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac0e30 con 0x561caaa0e7c0 2024-09-16T01:41:43.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.129+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v100'78 uv78 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.129+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac0e30 con 0x561caaa0e7c0 2024-09-16T01:41:43.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.130+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v100'79 uv79 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.130+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaabfe20 con 0x561caaa0e7c0 2024-09-16T01:41:43.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.131+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v100'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaa0e7c0 2024-09-16T01:41:43.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.131+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaabfe20 con 0x561caaab5530 2024-09-16T01:41:43.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.132+0000 7f41e3fff640 1 -- 172.21.15.88:0/175841998 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v100'122 uv116 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f41d4064070 con 0x561caaab5530 2024-09-16T01:41:43.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.132+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e100) -- 0x561caaac01c0 con 0x561caaab5530 2024-09-16T01:41:43.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.132+0000 7f41e3fff640 1 -- 172.21.15.88:0/175841998 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v100'123 uv123 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f41d4064070 con 0x561caaab5530 2024-09-16T01:41:43.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.133+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac0560 con 0x561caaab5530 2024-09-16T01:41:43.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.133+0000 7f41e3fff640 1 -- 172.21.15.88:0/175841998 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v100'124 uv124 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f41d4064070 con 0x561caaab5530 2024-09-16T01:41:43.139 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:41:43.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.133+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab8ee0 con 0x561caaab5530 2024-09-16T01:41:43.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.134+0000 7f41e3fff640 1 -- 172.21.15.88:0/175841998 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v100'125 uv125 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f41d4064070 con 0x561caaab5530 2024-09-16T01:41:43.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.134+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab8ee0 con 0x561caaab5530 2024-09-16T01:41:43.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.135+0000 7f41e3fff640 1 -- 172.21.15.88:0/175841998 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v100'126 uv126 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f41d4064070 con 0x561caaab5530 2024-09-16T01:41:43.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.135+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab8ee0 con 0x561caaab5530 2024-09-16T01:41:43.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.136+0000 7f41e3fff640 1 -- 172.21.15.88:0/175841998 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v100'127 uv121 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f41d4064070 con 0x561caaab5530 2024-09-16T01:41:43.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.136+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e100) -- 0x561caaab8ee0 con 0x561caaab5530 2024-09-16T01:41:43.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.137+0000 7f41e3fff640 1 -- 172.21.15.88:0/175841998 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v100'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41d4064070 con 0x561caaab5530 2024-09-16T01:41:43.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.137+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab8ee0 con 0x561caaab5530 2024-09-16T01:41:43.145 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:41:43.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.138+0000 7f41e3fff640 1 -- 172.21.15.88:0/175841998 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v100'129 uv129 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41d4064070 con 0x561caaab5530 2024-09-16T01:41:43.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.138+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaab8ee0 con 0x561caaab5530 2024-09-16T01:41:43.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.139+0000 7f41e3fff640 1 -- 172.21.15.88:0/175841998 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v100'130 uv130 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41d4064070 con 0x561caaab5530 2024-09-16T01:41:43.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.139+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac3860 con 0x561caaab5530 2024-09-16T01:41:43.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.140+0000 7f41e3fff640 1 -- 172.21.15.88:0/175841998 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v100'131 uv131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41d4064070 con 0x561caaab5530 2024-09-16T01:41:43.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.140+0000 7f41ee1b8f40 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x561caaac1520 0x561caaac3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.140+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac3f20 con 0x561caaac1520 2024-09-16T01:41:43.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.140+0000 7f41e3fff640 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x561caaac1520 0x561caaac3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:43.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.140+0000 7f41e3fff640 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x561caaac1520 0x561caaac3910 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:43.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.141+0000 7f41e3fff640 1 -- 172.21.15.88:0/175841998 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v100'57 uv55 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f41d4064070 con 0x561caaac1520 2024-09-16T01:41:43.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.141+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e100) -- 0x561caaac3f20 con 0x561caaac1520 2024-09-16T01:41:43.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.142+0000 7f41e3fff640 1 -- 172.21.15.88:0/175841998 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v100'58 uv58 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f41d4064070 con 0x561caaac1520 2024-09-16T01:41:43.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.142+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac3fa0 con 0x561caaac1520 2024-09-16T01:41:43.150 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:41:43.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.143+0000 7f41e3fff640 1 -- 172.21.15.88:0/175841998 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v100'59 uv59 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f41d4064070 con 0x561caaac1520 2024-09-16T01:41:43.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.143+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac0a00 con 0x561caaac1520 2024-09-16T01:41:43.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.144+0000 7f41e3fff640 1 -- 172.21.15.88:0/175841998 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v100'60 uv60 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f41d4064070 con 0x561caaac1520 2024-09-16T01:41:43.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.144+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac4450 con 0x561caaab5530 2024-09-16T01:41:43.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.145+0000 7f41e3fff640 1 -- 172.21.15.88:0/175841998 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v100'75 uv74 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f41d4064070 con 0x561caaab5530 2024-09-16T01:41:43.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.145+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e100) -- 0x561caaac4450 con 0x561caaab5530 2024-09-16T01:41:43.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.146+0000 7f41e3fff640 1 -- 172.21.15.88:0/175841998 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v100'76 uv76 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41d4064070 con 0x561caaab5530 2024-09-16T01:41:43.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.146+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac4450 con 0x561caaab5530 2024-09-16T01:41:43.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.147+0000 7f41e3fff640 1 -- 172.21.15.88:0/175841998 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v100'77 uv77 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41d4064070 con 0x561caaab5530 2024-09-16T01:41:43.154 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:41:43.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.147+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac5460 con 0x561caaab5530 2024-09-16T01:41:43.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.148+0000 7f41e3fff640 1 -- 172.21.15.88:0/175841998 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v100'78 uv78 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41d4064070 con 0x561caaab5530 2024-09-16T01:41:43.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.148+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac5460 con 0x561caaab5530 2024-09-16T01:41:43.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.149+0000 7f41e3fff640 1 -- 172.21.15.88:0/175841998 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v100'79 uv79 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41d4064070 con 0x561caaab5530 2024-09-16T01:41:43.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.149+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac4450 con 0x561caaab5530 2024-09-16T01:41:43.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.150+0000 7f41e3fff640 1 -- 172.21.15.88:0/175841998 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v100'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41d4064070 con 0x561caaab5530 2024-09-16T01:41:43.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.150+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac4450 con 0x561caaab9c60 2024-09-16T01:41:43.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.151+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v100'64 uv63 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.151+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e100) -- 0x561caaac4450 con 0x561caaab9c60 2024-09-16T01:41:43.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.152+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v100'65 uv65 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.152+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac4450 con 0x561caaab9c60 2024-09-16T01:41:43.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.153+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v100'66 uv66 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.159 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:41:43.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.153+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac4450 con 0x561caaab9c60 2024-09-16T01:41:43.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.154+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v100'67 uv67 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.154+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac4450 con 0x561caaab9c60 2024-09-16T01:41:43.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.155+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v100'68 uv68 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.155+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac4450 con 0x561caaab9c60 2024-09-16T01:41:43.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.157+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v100'157 uv148 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.157+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e100) -- 0x561caaac4450 con 0x561caaab9c60 2024-09-16T01:41:43.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.158+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v100'158 uv158 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.158+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac4450 con 0x561caaab9c60 2024-09-16T01:41:43.161 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:41:43.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.159+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v100'159 uv159 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.159+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac4450 con 0x561caaab9c60 2024-09-16T01:41:43.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.160+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v100'160 uv160 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.160+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac4450 con 0x561caaab9c60 2024-09-16T01:41:43.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.161+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v100'161 uv161 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.161+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac4450 con 0x561caaab9c60 2024-09-16T01:41:43.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.162+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v100'162 uv155 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.162+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e100) -- 0x561caaac4450 con 0x561caaab9c60 2024-09-16T01:41:43.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.163+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v100'163 uv163 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.163+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac4450 con 0x561caaab9c60 2024-09-16T01:41:43.165 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:41:43.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.164+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v100'164 uv164 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.164+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac4450 con 0x561caaab9c60 2024-09-16T01:41:43.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.165+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v100'165 uv165 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.165+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac4450 con 0x561caaab9c60 2024-09-16T01:41:43.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.166+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v100'166 uv166 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.166+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac4450 con 0x561caaab9c60 2024-09-16T01:41:43.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.167+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v100'167 uv167 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.167+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100) -- 0x561caaac4820 con 0x561caaab9c60 2024-09-16T01:41:43.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.168+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v100'168 uv168 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f41e400e070 con 0x561caaab9c60 2024-09-16T01:41:43.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.168+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x561caaa0e7c0 msgr2=0x561caaa00f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:43.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.168+0000 7f41ee1b8f40 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x561caaa0e7c0 0x561caaa00f70 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.168+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x561caaab5530 msgr2=0x561caaab59e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:43.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.168+0000 7f41ee1b8f40 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x561caaab5530 0x561caaab59e0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.168+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x561caaac1520 msgr2=0x561caaac3910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:43.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.168+0000 7f41ee1b8f40 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x561caaac1520 0x561caaac3910 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.168+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x561caaab0900 msgr2=0x561caaab2cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:43.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.168+0000 7f41ee1b8f40 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x561caaab0900 0x561caaab2cf0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.168+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x561caaab9c60 msgr2=0x561caaabc050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:43.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.168+0000 7f41ee1b8f40 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x561caaab9c60 0x561caaabc050 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.168+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x561caaabd030 msgr2=0x561caaabf420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:43.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.168+0000 7f41ee1b8f40 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x561caaabd030 0x561caaabf420 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.168+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 reap_dead start 2024-09-16T01:41:43.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.168+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 reap_dead start 2024-09-16T01:41:43.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.168+0000 7f41ec4b1640 1 -- 172.21.15.88:0/175841998 reap_dead start 2024-09-16T01:41:43.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.168+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f41cc06ac90 msgr2=0x7f41cc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:43.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.168+0000 7f41ee1b8f40 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f41cc06ac90 0x7f41cc06d150 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f41d4057000 tx=0x7f41d4051000 comp rx=0 tx=0).stop 2024-09-16T01:41:43.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.168+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561caa963e10 msgr2=0x561caa9faf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:43.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.168+0000 7f41ee1b8f40 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561caa963e10 0x561caa9faf70 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f41dc00bd10 tx=0x7f41dc009560 comp rx=0 tx=0).stop 2024-09-16T01:41:43.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.168+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 shutdown_connections 2024-09-16T01:41:43.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.168+0000 7f41ee1b8f40 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f41cc06ac90 0x7f41cc06d150 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.168+0000 7f41ee1b8f40 1 --2- 172.21.15.88:0/175841998 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561caa963e10 0x561caa9faf70 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.169+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 >> 172.21.15.88:0/175841998 conn(0x561caa97f750 msgr2=0x561caaa0b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:43.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.169+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 shutdown_connections 2024-09-16T01:41:43.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.169+0000 7f41ee1b8f40 1 -- 172.21.15.88:0/175841998 wait complete. 2024-09-16T01:41:43.173 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 100.00000000.inode 2024-09-16T01:41:43.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.255+0000 7fdb49411f40 1 Processor -- start 2024-09-16T01:41:43.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.255+0000 7fdb49411f40 1 -- start start 2024-09-16T01:41:43.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.256+0000 7fdb49411f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcb4311400 0x55fcb439f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.256+0000 7fdb49411f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fcb43125d0 0x55fcb42cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.256+0000 7fdb49411f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcb43a5140 0x55fcb43a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.256+0000 7fdb49411f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55fcb4316640 con 0x55fcb4311400 2024-09-16T01:41:43.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.256+0000 7fdb49411f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fcb421f510 con 0x55fcb43a5140 2024-09-16T01:41:43.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.256+0000 7fdb49411f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55fcb43167c0 con 0x55fcb43125d0 2024-09-16T01:41:43.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.256+0000 7fdb46295640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcb4311400 0x55fcb439f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:43.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.256+0000 7fdb46295640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcb4311400 0x55fcb439f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47412/0 (socket says 172.21.15.88:47412) 2024-09-16T01:41:43.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.256+0000 7fdb46295640 1 -- 172.21.15.88:0/377132579 learned_addr learned my addr 172.21.15.88:0/377132579 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:43.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.256+0000 7fdb46a96640 1 --2- 172.21.15.88:0/377132579 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcb43a5140 0x55fcb43a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:43.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.256+0000 7fdb45a94640 1 --2- 172.21.15.88:0/377132579 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fcb43125d0 0x55fcb42cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:43.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.256+0000 7fdb46295640 1 -- 172.21.15.88:0/377132579 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fcb43125d0 msgr2=0x55fcb42cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:43.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.256+0000 7fdb46295640 1 --2- 172.21.15.88:0/377132579 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fcb43125d0 0x55fcb42cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.256+0000 7fdb46295640 1 -- 172.21.15.88:0/377132579 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcb43a5140 msgr2=0x55fcb43a15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:43.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.256+0000 7fdb46295640 1 --2- 172.21.15.88:0/377132579 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcb43a5140 0x55fcb43a15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.256+0000 7fdb46295640 1 -- 172.21.15.88:0/377132579 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcb42cfef0 con 0x55fcb4311400 2024-09-16T01:41:43.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.256+0000 7fdb45a94640 1 --2- 172.21.15.88:0/377132579 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fcb43125d0 0x55fcb42cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:43.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.256+0000 7fdb46295640 1 --2- 172.21.15.88:0/377132579 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcb4311400 0x55fcb439f3f0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fdb38045b60 tx=0x7fdb38051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e995fb9f4528cdd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:43.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.257+0000 7fdb44a92640 1 -- 172.21.15.88:0/377132579 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb38055070 con 0x55fcb4311400 2024-09-16T01:41:43.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.257+0000 7fdb44a92640 1 -- 172.21.15.88:0/377132579 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb380517e0 con 0x55fcb4311400 2024-09-16T01:41:43.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.257+0000 7fdb44a92640 1 -- 172.21.15.88:0/377132579 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb38051b90 con 0x55fcb4311400 2024-09-16T01:41:43.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.257+0000 7fdb49411f40 1 -- 172.21.15.88:0/377132579 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcb4311400 msgr2=0x55fcb439f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:43.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.257+0000 7fdb49411f40 1 --2- 172.21.15.88:0/377132579 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcb4311400 0x55fcb439f3f0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fdb38045b60 tx=0x7fdb38051000 comp rx=0 tx=0).stop 2024-09-16T01:41:43.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.257+0000 7fdb49411f40 1 -- 172.21.15.88:0/377132579 shutdown_connections 2024-09-16T01:41:43.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.257+0000 7fdb49411f40 1 --2- 172.21.15.88:0/377132579 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcb43a5140 0x55fcb43a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.257+0000 7fdb49411f40 1 --2- 172.21.15.88:0/377132579 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fcb43125d0 0x55fcb42cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.257+0000 7fdb49411f40 1 --2- 172.21.15.88:0/377132579 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcb4311400 0x55fcb439f3f0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.257+0000 7fdb49411f40 1 -- 172.21.15.88:0/377132579 >> 172.21.15.88:0/377132579 conn(0x55fcb42f8960 msgr2=0x55fcb4311c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:43.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.257+0000 7fdb49411f40 1 -- 172.21.15.88:0/377132579 shutdown_connections 2024-09-16T01:41:43.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.257+0000 7fdb49411f40 1 -- 172.21.15.88:0/377132579 wait complete. 2024-09-16T01:41:43.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.258+0000 7fdb49411f40 1 Processor -- start 2024-09-16T01:41:43.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.258+0000 7fdb49411f40 1 -- start start 2024-09-16T01:41:43.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.258+0000 7fdb49411f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcb4311400 0x55fcb439d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.258+0000 7fdb49411f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fcb43125d0 0x55fcb42f4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.258+0000 7fdb49411f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcb43a5140 0x55fcb42f23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.258+0000 7fdb49411f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55fcb439fc70 con 0x55fcb43a5140 2024-09-16T01:41:43.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.258+0000 7fdb49411f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fcb43a0260 con 0x55fcb4311400 2024-09-16T01:41:43.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.258+0000 7fdb49411f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55fcb439f990 con 0x55fcb43125d0 2024-09-16T01:41:43.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.259+0000 7fdb46a96640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcb43a5140 0x55fcb42f23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:43.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.259+0000 7fdb46a96640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcb43a5140 0x55fcb42f23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47424/0 (socket says 172.21.15.88:47424) 2024-09-16T01:41:43.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.259+0000 7fdb46a96640 1 -- 172.21.15.88:0/236080394 learned_addr learned my addr 172.21.15.88:0/236080394 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:43.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.259+0000 7fdb46295640 1 --2- 172.21.15.88:0/236080394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcb4311400 0x55fcb439d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:43.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.259+0000 7fdb45a94640 1 --2- 172.21.15.88:0/236080394 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fcb43125d0 0x55fcb42f4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:43.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.259+0000 7fdb46a96640 1 -- 172.21.15.88:0/236080394 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fcb43125d0 msgr2=0x55fcb42f4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:43.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.259+0000 7fdb46a96640 1 --2- 172.21.15.88:0/236080394 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fcb43125d0 0x55fcb42f4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.259+0000 7fdb46a96640 1 -- 172.21.15.88:0/236080394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcb4311400 msgr2=0x55fcb439d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:43.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.259+0000 7fdb46a96640 1 --2- 172.21.15.88:0/236080394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fcb4311400 0x55fcb439d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.259+0000 7fdb46a96640 1 -- 172.21.15.88:0/236080394 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcb42cc250 con 0x55fcb43a5140 2024-09-16T01:41:43.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.259+0000 7fdb46a96640 1 --2- 172.21.15.88:0/236080394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcb43a5140 0x55fcb42f23f0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fdb4000c1f0 tx=0x7fdb4000c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:43.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.259+0000 7fdb377fe640 1 -- 172.21.15.88:0/236080394 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb40017020 con 0x55fcb43a5140 2024-09-16T01:41:43.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.259+0000 7fdb377fe640 1 -- 172.21.15.88:0/236080394 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb40013670 con 0x55fcb43a5140 2024-09-16T01:41:43.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.259+0000 7fdb377fe640 1 -- 172.21.15.88:0/236080394 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb40013a40 con 0x55fcb43a5140 2024-09-16T01:41:43.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.259+0000 7fdb49411f40 1 -- 172.21.15.88:0/236080394 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55fcb43a2530 con 0x55fcb43a5140 2024-09-16T01:41:43.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.259+0000 7fdb49411f40 1 -- 172.21.15.88:0/236080394 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcb43a1150 con 0x55fcb43a5140 2024-09-16T01:41:43.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.261+0000 7fdb377fe640 1 -- 172.21.15.88:0/236080394 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fdb4001a020 con 0x55fcb43a5140 2024-09-16T01:41:43.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.261+0000 7fdb377fe640 1 --2- 172.21.15.88:0/236080394 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdb2806ac90 0x7fdb2806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.261+0000 7fdb377fe640 1 -- 172.21.15.88:0/236080394 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 6856+0+0 (secure 0 0 0) 0x7fdb4008a6f0 con 0x55fcb43a5140 2024-09-16T01:41:43.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.261+0000 7fdb46295640 1 --2- 172.21.15.88:0/236080394 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdb2806ac90 0x7fdb2806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:43.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.261+0000 7fdb49411f40 1 --2- 172.21.15.88:0/236080394 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55fcb42f46c0 0x55fcb4315810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.261+0000 7fdb49411f40 1 -- 172.21.15.88:0/236080394 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:1 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e100) -- 0x55fcb439e350 con 0x55fcb42f46c0 2024-09-16T01:41:43.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.261+0000 7fdb46295640 1 --2- 172.21.15.88:0/236080394 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdb2806ac90 0x7fdb2806d150 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fdb38064000 tx=0x7fdb38053000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:43.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.262+0000 7fdb45a94640 1 --2- 172.21.15.88:0/236080394 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55fcb42f46c0 0x55fcb4315810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:43.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.262+0000 7fdb45a94640 1 --2- 172.21.15.88:0/236080394 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55fcb42f46c0 0x55fcb4315810 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:43.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.263+0000 7fdb45a94640 1 -- 172.21.15.88:0/236080394 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(1 100.00000000.inode [delete] v100'169 uv169 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fdb3c00e040 con 0x55fcb42f46c0 2024-09-16T01:41:43.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.263+0000 7fdb49411f40 1 -- 172.21.15.88:0/236080394 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55fcb42f46c0 msgr2=0x55fcb4315810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:43.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.263+0000 7fdb49411f40 1 --2- 172.21.15.88:0/236080394 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55fcb42f46c0 0x55fcb4315810 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.263+0000 7fdb49411f40 1 -- 172.21.15.88:0/236080394 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdb2806ac90 msgr2=0x7fdb2806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:43.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.263+0000 7fdb49411f40 1 --2- 172.21.15.88:0/236080394 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdb2806ac90 0x7fdb2806d150 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fdb38064000 tx=0x7fdb38053000 comp rx=0 tx=0).stop 2024-09-16T01:41:43.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.263+0000 7fdb49411f40 1 -- 172.21.15.88:0/236080394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcb43a5140 msgr2=0x55fcb42f23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:43.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.263+0000 7fdb49411f40 1 --2- 172.21.15.88:0/236080394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fcb43a5140 0x55fcb42f23f0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fdb4000c1f0 tx=0x7fdb4000c6b0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.263+0000 7fdb46a96640 1 -- 172.21.15.88:0/236080394 reap_dead start 2024-09-16T01:41:43.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.263+0000 7fdb49411f40 1 -- 172.21.15.88:0/236080394 shutdown_connections 2024-09-16T01:41:43.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.263+0000 7fdb49411f40 1 -- 172.21.15.88:0/236080394 >> 172.21.15.88:0/236080394 conn(0x55fcb42f8960 msgr2=0x55fcb42f7270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:43.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.263+0000 7fdb49411f40 1 -- 172.21.15.88:0/236080394 shutdown_connections 2024-09-16T01:41:43.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.263+0000 7fdb49411f40 1 -- 172.21.15.88:0/236080394 wait complete. 2024-09-16T01:41:43.273 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:41:43.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.413+0000 7feaa6b52640 1 Processor -- start 2024-09-16T01:41:43.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.413+0000 7feaa6b52640 1 -- start start 2024-09-16T01:41:43.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.413+0000 7feaa6b52640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7feaa015e7a0 0x7feaa006caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.413+0000 7feaa6b52640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaa00cf1f0 0x7feaa00b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.413+0000 7feaa6b52640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feaa00cf860 con 0x7feaa00b6720 2024-09-16T01:41:43.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.413+0000 7feaa6b52640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7feaa00d2ca0 con 0x7feaa015e7a0 2024-09-16T01:41:43.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.413+0000 7feaa6b52640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7feaa00cfb40 con 0x7feaa00cf1f0 2024-09-16T01:41:43.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.414+0000 7feaa48ee640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7feaa015e7a0 0x7feaa006caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:43.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.414+0000 7feaa50ef640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7feaa00b6720 0x7feaa01623c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34686/0 (socket says 172.21.15.88:34686) 2024-09-16T01:41:43.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.414+0000 7feaa50ef640 1 -- 172.21.15.88:0/3302470197 learned_addr learned my addr 172.21.15.88:0/3302470197 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:43.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.414+0000 7fea9ffff640 1 --2- 172.21.15.88:0/3302470197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaa00cf1f0 0x7feaa00b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:43.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.414+0000 7feaa48ee640 1 -- 172.21.15.88:0/3302470197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaa00cf1f0 msgr2=0x7feaa00b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:43.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.414+0000 7feaa48ee640 1 --2- 172.21.15.88:0/3302470197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaa00cf1f0 0x7feaa00b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.414+0000 7feaa48ee640 1 -- 172.21.15.88:0/3302470197 >> v1:172.21.15.88:6789/0 conn(0x7feaa00b6720 legacy=0x7feaa01623c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:43.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.414+0000 7fea9f7fe640 1 -- 172.21.15.88:0/3302470197 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2057336193 0 0) 0x7feaa00cf860 con 0x7feaa00b6720 2024-09-16T01:41:43.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.414+0000 7feaa48ee640 1 -- 172.21.15.88:0/3302470197 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaa00cfe20 con 0x7feaa015e7a0 2024-09-16T01:41:43.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.414+0000 7fea9ffff640 1 --2- 172.21.15.88:0/3302470197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaa00cf1f0 0x7feaa00b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:43.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.415+0000 7feaa48ee640 1 --2- 172.21.15.88:0/3302470197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feaa015e7a0 0x7feaa006caa0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fea940097a0 tx=0x7fea9402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f293cc04c0235368 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:43.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.415+0000 7fea9f7fe640 1 -- 172.21.15.88:0/3302470197 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea94037070 con 0x7feaa015e7a0 2024-09-16T01:41:43.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.415+0000 7fea9f7fe640 1 -- 172.21.15.88:0/3302470197 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea9402ae50 con 0x7feaa015e7a0 2024-09-16T01:41:43.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.415+0000 7fea9f7fe640 1 -- 172.21.15.88:0/3302470197 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea94032660 con 0x7feaa015e7a0 2024-09-16T01:41:43.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.415+0000 7feaa6b52640 1 -- 172.21.15.88:0/3302470197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feaa015e7a0 msgr2=0x7feaa006caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:43.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.415+0000 7feaa6b52640 1 --2- 172.21.15.88:0/3302470197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feaa015e7a0 0x7feaa006caa0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fea940097a0 tx=0x7fea9402a100 comp rx=0 tx=0).stop 2024-09-16T01:41:43.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.415+0000 7feaa6b52640 1 -- 172.21.15.88:0/3302470197 shutdown_connections 2024-09-16T01:41:43.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.415+0000 7feaa6b52640 1 --2- 172.21.15.88:0/3302470197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaa00cf1f0 0x7feaa00b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.415+0000 7feaa6b52640 1 --2- 172.21.15.88:0/3302470197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feaa015e7a0 0x7feaa006caa0 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.415+0000 7feaa6b52640 1 -- 172.21.15.88:0/3302470197 >> 172.21.15.88:0/3302470197 conn(0x7feaa00d11e0 msgr2=0x7feaa00d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:43.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.415+0000 7feaa6b52640 1 -- 172.21.15.88:0/3302470197 shutdown_connections 2024-09-16T01:41:43.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.415+0000 7feaa6b52640 1 -- 172.21.15.88:0/3302470197 wait complete. 2024-09-16T01:41:43.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.416+0000 7feaa6b52640 1 Processor -- start 2024-09-16T01:41:43.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.417+0000 7feaa6b52640 1 -- start start 2024-09-16T01:41:43.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.417+0000 7feaa6b52640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feaa00b6720 0x7feaa014f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.417+0000 7feaa6b52640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feaa00cf1f0 0x7feaa00cdf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.417+0000 7feaa6b52640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaa015e7a0 0x7feaa00ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.417+0000 7feaa6b52640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7feaa006cfe0 con 0x7feaa00cf1f0 2024-09-16T01:41:43.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.417+0000 7feaa6b52640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7feaa00b7a00 con 0x7feaa00b6720 2024-09-16T01:41:43.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.417+0000 7feaa6b52640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7feaa00d4420 con 0x7feaa015e7a0 2024-09-16T01:41:43.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.417+0000 7fea9ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feaa00cf1f0 0x7feaa00cdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:43.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.418+0000 7fea9ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feaa00cf1f0 0x7feaa00cdf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47442/0 (socket says 172.21.15.88:47442) 2024-09-16T01:41:43.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.418+0000 7fea9ffff640 1 -- 172.21.15.88:0/1953351182 learned_addr learned my addr 172.21.15.88:0/1953351182 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:43.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.418+0000 7feaa48ee640 1 --2- 172.21.15.88:0/1953351182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feaa00b6720 0x7feaa014f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:43.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.418+0000 7feaa50ef640 1 --2- 172.21.15.88:0/1953351182 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaa015e7a0 0x7feaa00ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:43.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.418+0000 7fea9ffff640 1 -- 172.21.15.88:0/1953351182 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaa015e7a0 msgr2=0x7feaa00ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.418+0000 7fea9ffff640 1 --2- 172.21.15.88:0/1953351182 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaa015e7a0 0x7feaa00ccbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.418+0000 7fea9ffff640 1 -- 172.21.15.88:0/1953351182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feaa00b6720 msgr2=0x7feaa014f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.418+0000 7fea9ffff640 1 --2- 172.21.15.88:0/1953351182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feaa00b6720 0x7feaa014f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.418+0000 7fea9ffff640 1 -- 172.21.15.88:0/1953351182 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaa00cadf0 con 0x7feaa00cf1f0 2024-09-16T01:41:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.418+0000 7fea9ffff640 1 --2- 172.21.15.88:0/1953351182 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feaa00cf1f0 0x7feaa00cdf60 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fea8c00c7f0 tx=0x7fea8c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.418+0000 7fea9d7fa640 1 -- 172.21.15.88:0/1953351182 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c004230 con 0x7feaa00cf1f0 2024-09-16T01:41:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.418+0000 7fea9d7fa640 1 -- 172.21.15.88:0/1953351182 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea8c0043d0 con 0x7feaa00cf1f0 2024-09-16T01:41:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.418+0000 7fea9d7fa640 1 -- 172.21.15.88:0/1953351182 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c007a70 con 0x7feaa00cf1f0 2024-09-16T01:41:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.418+0000 7feaa6b52640 1 -- 172.21.15.88:0/1953351182 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaa00cb0d0 con 0x7feaa00cf1f0 2024-09-16T01:41:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.418+0000 7feaa6b52640 1 -- 172.21.15.88:0/1953351182 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaa014fec0 con 0x7feaa00cf1f0 2024-09-16T01:41:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.419+0000 7feaa6b52640 1 -- 172.21.15.88:0/1953351182 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feaa015eb80 con 0x7feaa00cf1f0 2024-09-16T01:41:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.420+0000 7fea9d7fa640 1 -- 172.21.15.88:0/1953351182 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fea8c007c10 con 0x7feaa00cf1f0 2024-09-16T01:41:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.420+0000 7fea9d7fa640 1 --2- 172.21.15.88:0/1953351182 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fea8406c2b0 0x7fea8406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:43.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.420+0000 7fea9d7fa640 1 -- 172.21.15.88:0/1953351182 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 6856+0+0 (secure 0 0 0) 0x7fea8c08b920 con 0x7feaa00cf1f0 2024-09-16T01:41:43.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.420+0000 7feaa48ee640 1 --2- 172.21.15.88:0/1953351182 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fea8406c2b0 0x7fea8406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:43.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.421+0000 7feaa48ee640 1 --2- 172.21.15.88:0/1953351182 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fea8406c2b0 0x7fea8406e770 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fea94034000 tx=0x7fea9402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:43.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.425+0000 7fea9d7fa640 1 -- 172.21.15.88:0/1953351182 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fea8c015020 con 0x7feaa00cf1f0 2024-09-16T01:41:43.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:43.628+0000 7feaa6b52640 1 -- 172.21.15.88:0/1953351182 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7feaa00d2a80 con 0x7feaa00cf1f0 2024-09-16T01:41:44.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.282+0000 7fea9d7fa640 1 -- 172.21.15.88:0/1953351182 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v150) ==== 162+0+0 (secure 0 0 0) 0x7fea8c059ca0 con 0x7feaa00cf1f0 2024-09-16T01:41:44.283 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:41:44.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.284+0000 7feaa6b52640 1 -- 172.21.15.88:0/1953351182 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fea8406c2b0 msgr2=0x7fea8406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:44.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.284+0000 7feaa6b52640 1 --2- 172.21.15.88:0/1953351182 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fea8406c2b0 0x7fea8406e770 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fea94034000 tx=0x7fea9402e000 comp rx=0 tx=0).stop 2024-09-16T01:41:44.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.284+0000 7feaa6b52640 1 -- 172.21.15.88:0/1953351182 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feaa00cf1f0 msgr2=0x7feaa00cdf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:44.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.284+0000 7feaa6b52640 1 --2- 172.21.15.88:0/1953351182 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feaa00cf1f0 0x7feaa00cdf60 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fea8c00c7f0 tx=0x7fea8c00ccb0 comp rx=0 tx=0).stop 2024-09-16T01:41:44.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.284+0000 7feaa6b52640 1 -- 172.21.15.88:0/1953351182 shutdown_connections 2024-09-16T01:41:44.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.284+0000 7feaa6b52640 1 --2- 172.21.15.88:0/1953351182 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fea8406c2b0 0x7fea8406e770 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:44.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.284+0000 7feaa6b52640 1 --2- 172.21.15.88:0/1953351182 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feaa015e7a0 0x7feaa00ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:44.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.284+0000 7feaa6b52640 1 --2- 172.21.15.88:0/1953351182 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feaa00cf1f0 0x7feaa00cdf60 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:44.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.284+0000 7feaa6b52640 1 --2- 172.21.15.88:0/1953351182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feaa00b6720 0x7feaa014f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:44.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.284+0000 7feaa6b52640 1 -- 172.21.15.88:0/1953351182 >> 172.21.15.88:0/1953351182 conn(0x7feaa00d11e0 msgr2=0x7feaa00d0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:44.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.285+0000 7feaa6b52640 1 -- 172.21.15.88:0/1953351182 shutdown_connections 2024-09-16T01:41:44.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.285+0000 7feaa6b52640 1 -- 172.21.15.88:0/1953351182 wait complete. 2024-09-16T01:41:44.294 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:41:44.293+0000 7f323d588640 -1 mds.0.cache.ino(0x100) No data while reading inode 0x100 2024-09-16T01:41:44.295 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:41:44.293+0000 7f323d588640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-09-16T01:41:44.295 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:41:44.294+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:41:44.296 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:41:44.294+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:41:44.300 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:44.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.401+0000 7ff029d9f640 1 Processor -- start 2024-09-16T01:41:44.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.401+0000 7ff029d9f640 1 -- start start 2024-09-16T01:41:44.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.401+0000 7ff029d9f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff024155800 0x7ff0240b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:44.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.401+0000 7ff029d9f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0240cfc80 0x7ff0240b4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:44.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.402+0000 7ff029d9f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0240b50a0 con 0x7ff02415c330 2024-09-16T01:41:44.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.402+0000 7ff029d9f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff0240b4e60 con 0x7ff024155800 2024-09-16T01:41:44.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.402+0000 7ff029d9f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff02415c7a0 con 0x7ff0240cfc80 2024-09-16T01:41:44.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.402+0000 7ff0237fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff024155800 0x7ff0240b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:44.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.402+0000 7ff0237fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff024155800 0x7ff0240b6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47448/0 (socket says 172.21.15.88:47448) 2024-09-16T01:41:44.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.402+0000 7ff0237fe640 1 -- 172.21.15.88:0/1785257069 learned_addr learned my addr 172.21.15.88:0/1785257069 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:44.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.402+0000 7ff023fff640 1 --2- 172.21.15.88:0/1785257069 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0240cfc80 0x7ff0240b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:44.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.402+0000 7ff0237fe640 1 -- 172.21.15.88:0/1785257069 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0240cfc80 msgr2=0x7ff0240b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:44.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.402+0000 7ff0237fe640 1 --2- 172.21.15.88:0/1785257069 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0240cfc80 0x7ff0240b4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:44.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.402+0000 7ff0237fe640 1 -- 172.21.15.88:0/1785257069 >> v1:172.21.15.154:6789/0 conn(0x7ff02415c330 legacy=0x7ff0240cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:44.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.402+0000 7ff0237fe640 1 -- 172.21.15.88:0/1785257069 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0240b5380 con 0x7ff024155800 2024-09-16T01:41:44.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.403+0000 7ff0237fe640 1 --2- 172.21.15.88:0/1785257069 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff024155800 0x7ff0240b6140 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7ff014009870 tx=0x7ff01402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a2f9b192e6306 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:44.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.403+0000 7ff0227fc640 1 -- 172.21.15.88:0/1785257069 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff014037070 con 0x7ff024155800 2024-09-16T01:41:44.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.403+0000 7ff0227fc640 1 -- 172.21.15.88:0/1785257069 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff014032440 con 0x7ff024155800 2024-09-16T01:41:44.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.403+0000 7ff0227fc640 1 -- 172.21.15.88:0/1785257069 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff014032810 con 0x7ff024155800 2024-09-16T01:41:44.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.403+0000 7ff029d9f640 1 -- 172.21.15.88:0/1785257069 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff024155800 msgr2=0x7ff0240b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:44.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.403+0000 7ff029d9f640 1 --2- 172.21.15.88:0/1785257069 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff024155800 0x7ff0240b6140 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7ff014009870 tx=0x7ff01402a210 comp rx=0 tx=0).stop 2024-09-16T01:41:44.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.403+0000 7ff029d9f640 1 -- 172.21.15.88:0/1785257069 shutdown_connections 2024-09-16T01:41:44.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.403+0000 7ff029d9f640 1 --2- 172.21.15.88:0/1785257069 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff0240cfc80 0x7ff0240b4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:44.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.403+0000 7ff029d9f640 1 --2- 172.21.15.88:0/1785257069 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff024155800 0x7ff0240b6140 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:44.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.403+0000 7ff029d9f640 1 -- 172.21.15.88:0/1785257069 >> 172.21.15.88:0/1785257069 conn(0x7ff02406bc10 msgr2=0x7ff02406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:44.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.403+0000 7ff029d9f640 1 -- 172.21.15.88:0/1785257069 shutdown_connections 2024-09-16T01:41:44.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.403+0000 7ff029d9f640 1 -- 172.21.15.88:0/1785257069 wait complete. 2024-09-16T01:41:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.404+0000 7ff029d9f640 1 Processor -- start 2024-09-16T01:41:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.405+0000 7ff029d9f640 1 -- start start 2024-09-16T01:41:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.405+0000 7ff029d9f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0240cfc80 0x7ff0241e63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.405+0000 7ff029d9f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff024155800 0x7ff0241e4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.405+0000 7ff029d9f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff02415c330 0x7ff024156c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.405+0000 7ff029d9f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff0240b5c20 con 0x7ff02415c330 2024-09-16T01:41:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.405+0000 7ff029d9f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff0240b5940 con 0x7ff0240cfc80 2024-09-16T01:41:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.405+0000 7ff029d9f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff024155be0 con 0x7ff024155800 2024-09-16T01:41:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.405+0000 7ff023fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff02415c330 0x7ff024156c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.406+0000 7ff0237fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0240cfc80 0x7ff0241e63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.406+0000 7ff023fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff02415c330 0x7ff024156c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47462/0 (socket says 172.21.15.88:47462) 2024-09-16T01:41:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.406+0000 7ff023fff640 1 -- 172.21.15.88:0/3196799644 learned_addr learned my addr 172.21.15.88:0/3196799644 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.406+0000 7ff022ffd640 1 --2- 172.21.15.88:0/3196799644 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff024155800 0x7ff0241e4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.406+0000 7ff023fff640 1 -- 172.21.15.88:0/3196799644 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff024155800 msgr2=0x7ff0241e4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:44.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.406+0000 7ff023fff640 1 --2- 172.21.15.88:0/3196799644 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff024155800 0x7ff0241e4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:44.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.406+0000 7ff023fff640 1 -- 172.21.15.88:0/3196799644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0240cfc80 msgr2=0x7ff0241e63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:44.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.406+0000 7ff023fff640 1 --2- 172.21.15.88:0/3196799644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0240cfc80 0x7ff0241e63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:44.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.406+0000 7ff023fff640 1 -- 172.21.15.88:0/3196799644 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0241e5350 con 0x7ff02415c330 2024-09-16T01:41:44.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.406+0000 7ff023fff640 1 --2- 172.21.15.88:0/3196799644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff02415c330 0x7ff024156c50 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7ff01c007d90 tx=0x7ff01c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:44.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.406+0000 7ff00bfff640 1 -- 172.21.15.88:0/3196799644 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff01c014420 con 0x7ff02415c330 2024-09-16T01:41:44.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.406+0000 7ff00bfff640 1 -- 172.21.15.88:0/3196799644 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff01c019070 con 0x7ff02415c330 2024-09-16T01:41:44.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.406+0000 7ff00bfff640 1 -- 172.21.15.88:0/3196799644 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff01c014820 con 0x7ff02415c330 2024-09-16T01:41:44.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.406+0000 7ff029d9f640 1 -- 172.21.15.88:0/3196799644 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0241e5630 con 0x7ff02415c330 2024-09-16T01:41:44.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.406+0000 7ff029d9f640 1 -- 172.21.15.88:0/3196799644 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0241e5bf0 con 0x7ff02415c330 2024-09-16T01:41:44.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.407+0000 7ff029d9f640 1 -- 172.21.15.88:0/3196799644 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0240b6140 con 0x7ff02415c330 2024-09-16T01:41:44.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.408+0000 7ff00bfff640 1 -- 172.21.15.88:0/3196799644 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff01c0149c0 con 0x7ff02415c330 2024-09-16T01:41:44.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.408+0000 7ff00bfff640 1 --2- 172.21.15.88:0/3196799644 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff01006c2b0 0x7ff01006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:44.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.408+0000 7ff00bfff640 1 -- 172.21.15.88:0/3196799644 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6942+0+0 (secure 0 0 0) 0x7ff01c08bf70 con 0x7ff02415c330 2024-09-16T01:41:44.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.408+0000 7ff0237fe640 1 --2- 172.21.15.88:0/3196799644 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff01006c2b0 0x7ff01006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:44.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.409+0000 7ff0237fe640 1 --2- 172.21.15.88:0/3196799644 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff01006c2b0 0x7ff01006e770 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7ff014034000 tx=0x7ff01402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:44.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.412+0000 7ff00bfff640 1 -- 172.21.15.88:0/3196799644 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff01c059c50 con 0x7ff02415c330 2024-09-16T01:41:44.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.613+0000 7ff029d9f640 1 -- 172.21.15.88:0/3196799644 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0240633d0 con 0x7ff02415c330 2024-09-16T01:41:44.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.614+0000 7ff00bfff640 1 -- 172.21.15.88:0/3196799644 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) ==== 95+0+4389 (secure 0 0 0) 0x7ff01c05df50 con 0x7ff02415c330 2024-09-16T01:41:44.615 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:44.615 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":152,"btime":"2024-09-16T01:41:44:295259+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/981209627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":981209627},{"type":"v1","addr":"172.21.15.88:6836","nonce":981209627}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:44.295197+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:41:44.616 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 152 2024-09-16T01:41:44.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.616+0000 7ff029d9f640 1 -- 172.21.15.88:0/3196799644 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff01006c2b0 msgr2=0x7ff01006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:44.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.616+0000 7ff029d9f640 1 --2- 172.21.15.88:0/3196799644 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff01006c2b0 0x7ff01006e770 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7ff014034000 tx=0x7ff01402e000 comp rx=0 tx=0).stop 2024-09-16T01:41:44.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.616+0000 7ff029d9f640 1 -- 172.21.15.88:0/3196799644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff02415c330 msgr2=0x7ff024156c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:44.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.616+0000 7ff029d9f640 1 --2- 172.21.15.88:0/3196799644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff02415c330 0x7ff024156c50 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7ff01c007d90 tx=0x7ff01c00e4b0 comp rx=0 tx=0).stop 2024-09-16T01:41:44.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.616+0000 7ff029d9f640 1 -- 172.21.15.88:0/3196799644 shutdown_connections 2024-09-16T01:41:44.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.616+0000 7ff029d9f640 1 --2- 172.21.15.88:0/3196799644 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff01006c2b0 0x7ff01006e770 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:44.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.616+0000 7ff029d9f640 1 --2- 172.21.15.88:0/3196799644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff02415c330 0x7ff024156c50 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:44.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.616+0000 7ff029d9f640 1 --2- 172.21.15.88:0/3196799644 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff024155800 0x7ff0241e4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:44.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.616+0000 7ff029d9f640 1 --2- 172.21.15.88:0/3196799644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0240cfc80 0x7ff0241e63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:44.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.616+0000 7ff029d9f640 1 -- 172.21.15.88:0/3196799644 >> 172.21.15.88:0/3196799644 conn(0x7ff02406bc10 msgr2=0x7ff02406ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:44.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.616+0000 7ff029d9f640 1 -- 172.21.15.88:0/3196799644 shutdown_connections 2024-09-16T01:41:44.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.616+0000 7ff029d9f640 1 -- 172.21.15.88:0/3196799644 wait complete. 2024-09-16T01:41:44.629 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:41:44.629 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:44.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.726+0000 7f0299f11640 1 Processor -- start 2024-09-16T01:41:44.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.726+0000 7f0299f11640 1 -- start start 2024-09-16T01:41:44.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.727+0000 7f0299f11640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f029415a9c0 0x7f029415e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.727+0000 7f0299f11640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f02940b5be0 0x7f029415eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.727+0000 7f0299f11640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02940b6830 con 0x7f02940b29a0 2024-09-16T01:41:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.727+0000 7f0299f11640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f029415f410 con 0x7f02940b5be0 2024-09-16T01:41:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.727+0000 7f0299f11640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f02940cd0b0 con 0x7f029415a9c0 2024-09-16T01:41:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.727+0000 7f028bfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f02940b5be0 0x7f029415eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.727+0000 7f0293fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f02940b29a0 0x7f02940b3710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34700/0 (socket says 172.21.15.88:34700) 2024-09-16T01:41:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.727+0000 7f0293fff640 1 -- 172.21.15.88:0/3055903735 learned_addr learned my addr 172.21.15.88:0/3055903735 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.727+0000 7f02937fe640 1 --2- 172.21.15.88:0/3055903735 >> v2:172.21.15.154:3300/0 conn(0x7f029415a9c0 0x7f029415e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.727+0000 7f028bfff640 1 -- 172.21.15.88:0/3055903735 >> v2:172.21.15.154:3300/0 conn(0x7f029415a9c0 msgr2=0x7f029415e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.727+0000 7f028bfff640 1 --2- 172.21.15.88:0/3055903735 >> v2:172.21.15.154:3300/0 conn(0x7f029415a9c0 0x7f029415e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.727+0000 7f028bfff640 1 -- 172.21.15.88:0/3055903735 >> v1:172.21.15.88:6789/0 conn(0x7f02940b29a0 legacy=0x7f02940b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.727+0000 7f028bfff640 1 -- 172.21.15.88:0/3055903735 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02940cd390 con 0x7f02940b5be0 2024-09-16T01:41:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.727+0000 7f0292ffd640 1 -- 172.21.15.88:0/3055903735 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3626881926 0 0) 0x7f02940b6830 con 0x7f02940b29a0 2024-09-16T01:41:44.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.727+0000 7f02937fe640 1 --2- 172.21.15.88:0/3055903735 >> v2:172.21.15.154:3300/0 conn(0x7f029415a9c0 0x7f029415e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:44.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.728+0000 7f028bfff640 1 --2- 172.21.15.88:0/3055903735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02940b5be0 0x7f029415eb40 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f029406f1a0 tx=0x7f027c02b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d215c757df7472f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:44.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.728+0000 7f0292ffd640 1 -- 172.21.15.88:0/3055903735 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027c00b040 con 0x7f02940b5be0 2024-09-16T01:41:44.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.728+0000 7f0292ffd640 1 -- 172.21.15.88:0/3055903735 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f027c030710 con 0x7f02940b5be0 2024-09-16T01:41:44.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.728+0000 7f0292ffd640 1 -- 172.21.15.88:0/3055903735 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027c030ae0 con 0x7f02940b5be0 2024-09-16T01:41:44.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.728+0000 7f0299f11640 1 -- 172.21.15.88:0/3055903735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02940b5be0 msgr2=0x7f029415eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:44.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.728+0000 7f0299f11640 1 --2- 172.21.15.88:0/3055903735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02940b5be0 0x7f029415eb40 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f029406f1a0 tx=0x7f027c02b560 comp rx=0 tx=0).stop 2024-09-16T01:41:44.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.729+0000 7f0299f11640 1 -- 172.21.15.88:0/3055903735 shutdown_connections 2024-09-16T01:41:44.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.729+0000 7f0299f11640 1 --2- 172.21.15.88:0/3055903735 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02940b5be0 0x7f029415eb40 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:44.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.729+0000 7f0299f11640 1 --2- 172.21.15.88:0/3055903735 >> v2:172.21.15.154:3300/0 conn(0x7f029415a9c0 0x7f029415e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:44.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.729+0000 7f0299f11640 1 -- 172.21.15.88:0/3055903735 >> 172.21.15.88:0/3055903735 conn(0x7f029406bc10 msgr2=0x7f029406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:44.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.729+0000 7f0299f11640 1 -- 172.21.15.88:0/3055903735 shutdown_connections 2024-09-16T01:41:44.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.729+0000 7f0299f11640 1 -- 172.21.15.88:0/3055903735 wait complete. 2024-09-16T01:41:44.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.730+0000 7f0299f11640 1 Processor -- start 2024-09-16T01:41:44.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.730+0000 7f0299f11640 1 -- start start 2024-09-16T01:41:44.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.730+0000 7f0299f11640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02940b29a0 0x7f029415d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:44.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.730+0000 7f0299f11640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02940b5be0 0x7f02941f2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:44.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.730+0000 7f0299f11640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f029415a9c0 0x7f02940c8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:44.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.730+0000 7f0299f11640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f029415ada0 con 0x7f02940b29a0 2024-09-16T01:41:44.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.730+0000 7f0299f11640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f02940cd950 con 0x7f02940b5be0 2024-09-16T01:41:44.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.730+0000 7f0299f11640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f029415f080 con 0x7f029415a9c0 2024-09-16T01:41:44.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.731+0000 7f02937fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02940b29a0 0x7f029415d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:44.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.731+0000 7f02937fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02940b29a0 0x7f029415d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47480/0 (socket says 172.21.15.88:47480) 2024-09-16T01:41:44.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.731+0000 7f02937fe640 1 -- 172.21.15.88:0/3802054942 learned_addr learned my addr 172.21.15.88:0/3802054942 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:44.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.731+0000 7f028bfff640 1 --2- 172.21.15.88:0/3802054942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02940b5be0 0x7f02941f2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:44.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.731+0000 7f0293fff640 1 --2- 172.21.15.88:0/3802054942 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f029415a9c0 0x7f02940c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:44.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.731+0000 7f02937fe640 1 -- 172.21.15.88:0/3802054942 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f029415a9c0 msgr2=0x7f02940c8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:44.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.731+0000 7f02937fe640 1 --2- 172.21.15.88:0/3802054942 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f029415a9c0 0x7f02940c8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:44.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.731+0000 7f02937fe640 1 -- 172.21.15.88:0/3802054942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02940b5be0 msgr2=0x7f02941f2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:44.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.731+0000 7f02937fe640 1 --2- 172.21.15.88:0/3802054942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02940b5be0 0x7f02941f2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:44.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.731+0000 7f02937fe640 1 -- 172.21.15.88:0/3802054942 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02940c61e0 con 0x7f02940b29a0 2024-09-16T01:41:44.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.731+0000 7f028bfff640 1 --2- 172.21.15.88:0/3802054942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02940b5be0 0x7f02941f2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:44.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.731+0000 7f02937fe640 1 --2- 172.21.15.88:0/3802054942 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02940b29a0 0x7f029415d870 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f028400c7f0 tx=0x7f028400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:44.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.731+0000 7f0290ff9640 1 -- 172.21.15.88:0/3802054942 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0284015070 con 0x7f02940b29a0 2024-09-16T01:41:44.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.731+0000 7f0290ff9640 1 -- 172.21.15.88:0/3802054942 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0284007820 con 0x7f02940b29a0 2024-09-16T01:41:44.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.731+0000 7f0290ff9640 1 -- 172.21.15.88:0/3802054942 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0284007bd0 con 0x7f02940b29a0 2024-09-16T01:41:44.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.731+0000 7f0299f11640 1 -- 172.21.15.88:0/3802054942 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02940c64c0 con 0x7f02940b29a0 2024-09-16T01:41:44.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.731+0000 7f0299f11640 1 -- 172.21.15.88:0/3802054942 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02940c6a80 con 0x7f02940b29a0 2024-09-16T01:41:44.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.732+0000 7f0299f11640 1 -- 172.21.15.88:0/3802054942 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f029406de40 con 0x7f02940b29a0 2024-09-16T01:41:44.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.733+0000 7f0290ff9640 1 -- 172.21.15.88:0/3802054942 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0284018070 con 0x7f02940b29a0 2024-09-16T01:41:44.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.733+0000 7f0290ff9640 1 --2- 172.21.15.88:0/3802054942 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f027406c2b0 0x7f027406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:44.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.733+0000 7f0290ff9640 1 -- 172.21.15.88:0/3802054942 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6942+0+0 (secure 0 0 0) 0x7f028408b9e0 con 0x7f02940b29a0 2024-09-16T01:41:44.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.733+0000 7f028bfff640 1 --2- 172.21.15.88:0/3802054942 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f027406c2b0 0x7f027406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:44.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.734+0000 7f028bfff640 1 --2- 172.21.15.88:0/3802054942 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f027406c2b0 0x7f027406e770 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f027c035000 tx=0x7f027c00c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:44.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.737+0000 7f0290ff9640 1 -- 172.21.15.88:0/3802054942 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0284059d10 con 0x7f02940b29a0 2024-09-16T01:41:44.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.935+0000 7f0299f11640 1 -- 172.21.15.88:0/3802054942 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02940b4300 con 0x7f02940b29a0 2024-09-16T01:41:44.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.935+0000 7f0290ff9640 1 -- 172.21.15.88:0/3802054942 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) ==== 95+0+4389 (secure 0 0 0) 0x7f0284059610 con 0x7f02940b29a0 2024-09-16T01:41:44.937 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:44.937 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":152,"btime":"2024-09-16T01:41:44:295259+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/981209627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":981209627},{"type":"v1","addr":"172.21.15.88:6836","nonce":981209627}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:44.295197+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:41:44.937 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 152 2024-09-16T01:41:44.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.937+0000 7f0299f11640 1 -- 172.21.15.88:0/3802054942 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f027406c2b0 msgr2=0x7f027406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:44.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.937+0000 7f0299f11640 1 --2- 172.21.15.88:0/3802054942 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f027406c2b0 0x7f027406e770 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f027c035000 tx=0x7f027c00c040 comp rx=0 tx=0).stop 2024-09-16T01:41:44.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.937+0000 7f0299f11640 1 -- 172.21.15.88:0/3802054942 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02940b29a0 msgr2=0x7f029415d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:44.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.937+0000 7f0299f11640 1 --2- 172.21.15.88:0/3802054942 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02940b29a0 0x7f029415d870 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f028400c7f0 tx=0x7f028400ccb0 comp rx=0 tx=0).stop 2024-09-16T01:41:44.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.937+0000 7f0299f11640 1 -- 172.21.15.88:0/3802054942 shutdown_connections 2024-09-16T01:41:44.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.937+0000 7f0299f11640 1 --2- 172.21.15.88:0/3802054942 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f027406c2b0 0x7f027406e770 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:44.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.937+0000 7f0299f11640 1 --2- 172.21.15.88:0/3802054942 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f029415a9c0 0x7f02940c8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:44.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.937+0000 7f0299f11640 1 --2- 172.21.15.88:0/3802054942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02940b5be0 0x7f02941f2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:44.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.937+0000 7f0299f11640 1 --2- 172.21.15.88:0/3802054942 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f02940b29a0 0x7f029415d870 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:44.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.937+0000 7f0299f11640 1 -- 172.21.15.88:0/3802054942 >> 172.21.15.88:0/3802054942 conn(0x7f029406bc10 msgr2=0x7f0294070f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:44.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.938+0000 7f0299f11640 1 -- 172.21.15.88:0/3802054942 shutdown_connections 2024-09-16T01:41:44.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:44.938+0000 7f0299f11640 1 -- 172.21.15.88:0/3802054942 wait complete. 2024-09-16T01:41:44.950 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:41:44.950 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:45.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.049+0000 7f3ef60c9640 1 Processor -- start 2024-09-16T01:41:45.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.049+0000 7f3ef60c9640 1 -- start start 2024-09-16T01:41:45.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.050+0000 7f3ef60c9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3ef00b4920 0x7f3ef015b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:45.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.050+0000 7f3ef60c9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ef00ce4f0 con 0x7f3ef00b4d00 2024-09-16T01:41:45.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.050+0000 7f3ef60c9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ef00ce7d0 con 0x7f3ef00cd780 2024-09-16T01:41:45.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.050+0000 7f3ef60c9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3ef00cf100 con 0x7f3ef00b4920 2024-09-16T01:41:45.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.050+0000 7f3eeeffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3ef00b4d00 0x7f3ef00b5170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34710/0 (socket says 172.21.15.88:34710) 2024-09-16T01:41:45.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.050+0000 7f3eeeffd640 1 -- 172.21.15.88:0/3341695391 learned_addr learned my addr 172.21.15.88:0/3341695391 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:45.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.050+0000 7f3eef7fe640 1 --2- 172.21.15.88:0/3341695391 >> v2:172.21.15.154:3300/0 conn(0x7f3ef00b4920 0x7f3ef015b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:45.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.050+0000 7f3eee7fc640 1 -- 172.21.15.88:0/3341695391 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2808793519 0 0) 0x7f3ef00ce4f0 con 0x7f3ef00b4d00 2024-09-16T01:41:45.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.050+0000 7f3eee7fc640 1 -- 172.21.15.88:0/3341695391 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ed8003520 con 0x7f3ef00b4d00 2024-09-16T01:41:45.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.050+0000 7f3eee7fc640 1 -- 172.21.15.88:0/3341695391 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2937372828 0 0) 0x7f3ed8003520 con 0x7f3ef00b4d00 2024-09-16T01:41:45.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.050+0000 7f3eee7fc640 1 -- 172.21.15.88:0/3341695391 >> v2:172.21.15.154:3300/0 conn(0x7f3ef00b4920 msgr2=0x7f3ef015b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:45.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.050+0000 7f3eee7fc640 1 --2- 172.21.15.88:0/3341695391 >> v2:172.21.15.154:3300/0 conn(0x7f3ef00b4920 0x7f3ef015b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:45.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.050+0000 7f3eee7fc640 1 -- 172.21.15.88:0/3341695391 >> v1:172.21.15.154:6789/0 conn(0x7f3ef00cd780 legacy=0x7f3ef00cdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:45.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.050+0000 7f3eee7fc640 1 -- 172.21.15.88:0/3341695391 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ef00ceab0 con 0x7f3ef00b4d00 2024-09-16T01:41:45.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.050+0000 7f3eee7fc640 1 -- 172.21.15.88:0/3341695391 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f3ee4001670 con 0x7f3ef00b4d00 2024-09-16T01:41:45.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.051+0000 7f3eee7fc640 1 -- 172.21.15.88:0/3341695391 >> v1:172.21.15.88:6789/0 conn(0x7f3ef00b4d00 legacy=0x7f3ef00b5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:45.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.051+0000 7f3eee7fc640 1 --2- 172.21.15.88:0/3341695391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ed80039b0 0x7f3ed8003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:45.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.051+0000 7f3eee7fc640 1 --2- 172.21.15.88:0/3341695391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ed8004800 0x7f3ed8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:45.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.051+0000 7f3eee7fc640 1 --2- 172.21.15.88:0/3341695391 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed80051c0 0x7f3ed800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:45.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.051+0000 7f3eee7fc640 1 -- 172.21.15.88:0/3341695391 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef00cf100 con 0x7f3ed8004800 2024-09-16T01:41:45.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.051+0000 7f3eee7fc640 1 -- 172.21.15.88:0/3341695391 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed800c090 con 0x7f3ed80039b0 2024-09-16T01:41:45.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.051+0000 7f3eee7fc640 1 -- 172.21.15.88:0/3341695391 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3ed800c290 con 0x7f3ed80051c0 2024-09-16T01:41:45.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.051+0000 7f3eeeffd640 1 --2- 172.21.15.88:0/3341695391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ed8004800 0x7f3ed8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:45.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.051+0000 7f3eef7fe640 1 --2- 172.21.15.88:0/3341695391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ed80039b0 0x7f3ed8003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:45.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.051+0000 7f3eeffff640 1 --2- 172.21.15.88:0/3341695391 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed80051c0 0x7f3ed800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:45.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.051+0000 7f3eeeffd640 1 -- 172.21.15.88:0/3341695391 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed80051c0 msgr2=0x7f3ed800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:45.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.051+0000 7f3eeeffd640 1 --2- 172.21.15.88:0/3341695391 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed80051c0 0x7f3ed800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:45.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.051+0000 7f3eeeffd640 1 -- 172.21.15.88:0/3341695391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ed80039b0 msgr2=0x7f3ed8003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:45.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.051+0000 7f3eeeffd640 1 --2- 172.21.15.88:0/3341695391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ed80039b0 0x7f3ed8003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:45.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.051+0000 7f3eeeffd640 1 -- 172.21.15.88:0/3341695391 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed8003520 con 0x7f3ed8004800 2024-09-16T01:41:45.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.051+0000 7f3eeffff640 1 --2- 172.21.15.88:0/3341695391 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed80051c0 0x7f3ed800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:45.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.051+0000 7f3eeffff640 1 -- 172.21.15.88:0/3341695391 reap_dead start 2024-09-16T01:41:45.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.052+0000 7f3eeeffd640 1 --2- 172.21.15.88:0/3341695391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ed8004800 0x7f3ed8004c80 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f3ee400c240 tx=0x7f3ed8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8364c7ea8da35ec server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:45.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.052+0000 7f3eee7fc640 1 -- 172.21.15.88:0/3341695391 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee4007d10 con 0x7f3ed8004800 2024-09-16T01:41:45.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.052+0000 7f3eee7fc640 1 -- 172.21.15.88:0/3341695391 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ee4005ad0 con 0x7f3ed8004800 2024-09-16T01:41:45.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.052+0000 7f3eee7fc640 1 -- 172.21.15.88:0/3341695391 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee4007070 con 0x7f3ed8004800 2024-09-16T01:41:45.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.052+0000 7f3ef60c9640 1 -- 172.21.15.88:0/3341695391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ed8004800 msgr2=0x7f3ed8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:45.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.052+0000 7f3ef60c9640 1 --2- 172.21.15.88:0/3341695391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ed8004800 0x7f3ed8004c80 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f3ee400c240 tx=0x7f3ed8005830 comp rx=0 tx=0).stop 2024-09-16T01:41:45.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.052+0000 7f3ef60c9640 1 -- 172.21.15.88:0/3341695391 shutdown_connections 2024-09-16T01:41:45.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.052+0000 7f3ef60c9640 1 --2- 172.21.15.88:0/3341695391 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ed8004800 0x7f3ed8004c80 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:45.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.052+0000 7f3ef60c9640 1 -- 172.21.15.88:0/3341695391 >> 172.21.15.88:0/3341695391 conn(0x7f3ef006bc10 msgr2=0x7f3ef006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:45.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.052+0000 7f3ef60c9640 1 -- 172.21.15.88:0/3341695391 shutdown_connections 2024-09-16T01:41:45.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.052+0000 7f3ef60c9640 1 -- 172.21.15.88:0/3341695391 wait complete. 2024-09-16T01:41:45.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.054+0000 7f3ef60c9640 1 Processor -- start 2024-09-16T01:41:45.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.054+0000 7f3ef60c9640 1 -- start start 2024-09-16T01:41:45.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.054+0000 7f3ef60c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ed8004800 0x7f3ef01f3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:45.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.054+0000 7f3ef60c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ef0147040 0x7f3ef0155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:45.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.054+0000 7f3ef60c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ef00b5800 0x7f3ef00b5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.054+0000 7f3ef60c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef00b52f0 con 0x7f3ed8004800 2024-09-16T01:41:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.054+0000 7f3ef60c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef00b5170 con 0x7f3ef00b5800 2024-09-16T01:41:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.054+0000 7f3ef60c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3ef00b5470 con 0x7f3ef0147040 2024-09-16T01:41:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.055+0000 7f3eef7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ed8004800 0x7f3ef01f3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.055+0000 7f3eef7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ed8004800 0x7f3ef01f3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47504/0 (socket says 172.21.15.88:47504) 2024-09-16T01:41:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.055+0000 7f3eef7fe640 1 -- 172.21.15.88:0/2802413083 learned_addr learned my addr 172.21.15.88:0/2802413083 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.055+0000 7f3eeeffd640 1 --2- 172.21.15.88:0/2802413083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ef0147040 0x7f3ef0155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.055+0000 7f3eeffff640 1 --2- 172.21.15.88:0/2802413083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ef00b5800 0x7f3ef00b5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.055+0000 7f3eef7fe640 1 -- 172.21.15.88:0/2802413083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ef0147040 msgr2=0x7f3ef0155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.055+0000 7f3eef7fe640 1 --2- 172.21.15.88:0/2802413083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ef0147040 0x7f3ef0155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.055+0000 7f3eef7fe640 1 -- 172.21.15.88:0/2802413083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ef00b5800 msgr2=0x7f3ef00b5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.055+0000 7f3eef7fe640 1 --2- 172.21.15.88:0/2802413083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ef00b5800 0x7f3ef00b5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.055+0000 7f3eef7fe640 1 -- 172.21.15.88:0/2802413083 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ef01474c0 con 0x7f3ed8004800 2024-09-16T01:41:45.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.055+0000 7f3eeffff640 1 --2- 172.21.15.88:0/2802413083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ef00b5800 0x7f3ef00b5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:45.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.055+0000 7f3eef7fe640 1 --2- 172.21.15.88:0/2802413083 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ed8004800 0x7f3ef01f3100 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0011e60 tx=0x7f3ee000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:45.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.055+0000 7f3ed7fff640 1 -- 172.21.15.88:0/2802413083 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0015630 con 0x7f3ed8004800 2024-09-16T01:41:45.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.055+0000 7f3ed7fff640 1 -- 172.21.15.88:0/2802413083 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ee00157d0 con 0x7f3ed8004800 2024-09-16T01:41:45.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.055+0000 7f3ed7fff640 1 -- 172.21.15.88:0/2802413083 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0015ba0 con 0x7f3ed8004800 2024-09-16T01:41:45.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.055+0000 7f3ef60c9640 1 -- 172.21.15.88:0/2802413083 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ef0153040 con 0x7f3ed8004800 2024-09-16T01:41:45.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.055+0000 7f3ef60c9640 1 -- 172.21.15.88:0/2802413083 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ef0153600 con 0x7f3ed8004800 2024-09-16T01:41:45.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.056+0000 7f3ef60c9640 1 -- 172.21.15.88:0/2802413083 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ef015b7d0 con 0x7f3ed8004800 2024-09-16T01:41:45.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.057+0000 7f3ed7fff640 1 -- 172.21.15.88:0/2802413083 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3ee001d400 con 0x7f3ed8004800 2024-09-16T01:41:45.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.057+0000 7f3ed7fff640 1 --2- 172.21.15.88:0/2802413083 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3edc06c2e0 0x7f3edc06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:45.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.057+0000 7f3ed7fff640 1 -- 172.21.15.88:0/2802413083 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6942+0+0 (secure 0 0 0) 0x7f3ee008caa0 con 0x7f3ed8004800 2024-09-16T01:41:45.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.057+0000 7f3eeeffd640 1 --2- 172.21.15.88:0/2802413083 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3edc06c2e0 0x7f3edc06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:45.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.058+0000 7f3eeeffd640 1 --2- 172.21.15.88:0/2802413083 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3edc06c2e0 0x7f3edc06e7a0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f3ee400c240 tx=0x7f3ee402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:45.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.061+0000 7f3ed7fff640 1 -- 172.21.15.88:0/2802413083 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3ee0019030 con 0x7f3ed8004800 2024-09-16T01:41:45.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.260+0000 7f3ef60c9640 1 -- 172.21.15.88:0/2802413083 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ef0156780 con 0x7f3ed8004800 2024-09-16T01:41:45.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.260+0000 7f3ed7fff640 1 -- 172.21.15.88:0/2802413083 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) ==== 95+0+4389 (secure 0 0 0) 0x7f3ee0010040 con 0x7f3ed8004800 2024-09-16T01:41:45.262 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:45.262 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":152,"btime":"2024-09-16T01:41:44:295259+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/981209627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":981209627},{"type":"v1","addr":"172.21.15.88:6836","nonce":981209627}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:44.295197+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:41:45.262 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 152 2024-09-16T01:41:45.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.262+0000 7f3ef60c9640 1 -- 172.21.15.88:0/2802413083 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3edc06c2e0 msgr2=0x7f3edc06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:45.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.262+0000 7f3ef60c9640 1 --2- 172.21.15.88:0/2802413083 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3edc06c2e0 0x7f3edc06e7a0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f3ee400c240 tx=0x7f3ee402e000 comp rx=0 tx=0).stop 2024-09-16T01:41:45.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.262+0000 7f3ef60c9640 1 -- 172.21.15.88:0/2802413083 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ed8004800 msgr2=0x7f3ef01f3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:45.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.262+0000 7f3ef60c9640 1 --2- 172.21.15.88:0/2802413083 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ed8004800 0x7f3ef01f3100 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0011e60 tx=0x7f3ee000f580 comp rx=0 tx=0).stop 2024-09-16T01:41:45.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.262+0000 7f3ef60c9640 1 -- 172.21.15.88:0/2802413083 shutdown_connections 2024-09-16T01:41:45.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.262+0000 7f3ef60c9640 1 --2- 172.21.15.88:0/2802413083 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3edc06c2e0 0x7f3edc06e7a0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:45.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.262+0000 7f3ef60c9640 1 --2- 172.21.15.88:0/2802413083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ef00b5800 0x7f3ef00b5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:45.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.262+0000 7f3ef60c9640 1 --2- 172.21.15.88:0/2802413083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ef0147040 0x7f3ef0155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:45.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.262+0000 7f3ef60c9640 1 --2- 172.21.15.88:0/2802413083 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ed8004800 0x7f3ef01f3100 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:45.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.262+0000 7f3ef60c9640 1 -- 172.21.15.88:0/2802413083 >> 172.21.15.88:0/2802413083 conn(0x7f3ef006bc10 msgr2=0x7f3ef0156f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:45.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.263+0000 7f3ef60c9640 1 -- 172.21.15.88:0/2802413083 shutdown_connections 2024-09-16T01:41:45.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.263+0000 7f3ef60c9640 1 -- 172.21.15.88:0/2802413083 wait complete. 2024-09-16T01:41:45.275 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 100.00000000.inode' led to DAMAGED state 2024-09-16T01:41:45.275 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 1.00000000' 2024-09-16T01:41:45.275 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:41:45.275 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:41:45.291 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:41:45.291 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:41:45.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:45.292 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:41:45.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:41:45.346 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:41:45.347 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:41:45.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.484+0000 7fb0a2791640 1 Processor -- start 2024-09-16T01:41:45.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.485+0000 7fb0a2791640 1 -- start start 2024-09-16T01:41:45.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.485+0000 7fb0a2791640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb09c0720d0 0x7fb09c0b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:45.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.485+0000 7fb0a2791640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb09c0cca40 0x7fb09c0b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:45.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.485+0000 7fb0a2791640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb09c071720 con 0x7fb09c15ad10 2024-09-16T01:41:45.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.485+0000 7fb0a2791640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb09c070fb0 con 0x7fb09c0cca40 2024-09-16T01:41:45.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.485+0000 7fb0a2791640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb09c073920 con 0x7fb09c0720d0 2024-09-16T01:41:45.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.485+0000 7fb0a0d2e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb09c0cca40 0x7fb09c0b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:45.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.486+0000 7fb0a0d2e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb09c0cca40 0x7fb09c0b43e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47510/0 (socket says 172.21.15.88:47510) 2024-09-16T01:41:45.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.486+0000 7fb0a0d2e640 1 -- 172.21.15.88:0/2225645050 learned_addr learned my addr 172.21.15.88:0/2225645050 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:45.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.486+0000 7fb09bfff640 1 --1- 172.21.15.88:0/2225645050 >> v1:172.21.15.88:6789/0 conn(0x7fb09c15ad10 0x7fb09c0719c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34714/0 (socket says 172.21.15.88:34714) 2024-09-16T01:41:45.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.486+0000 7fb09b7fe640 1 --2- 172.21.15.88:0/2225645050 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb09c0720d0 0x7fb09c0b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:45.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.486+0000 7fb0a0d2e640 1 -- 172.21.15.88:0/2225645050 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb09c0720d0 msgr2=0x7fb09c0b3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:45.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.486+0000 7fb0a0d2e640 1 --2- 172.21.15.88:0/2225645050 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb09c0720d0 0x7fb09c0b3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:45.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.486+0000 7fb0a0d2e640 1 -- 172.21.15.88:0/2225645050 >> v1:172.21.15.88:6789/0 conn(0x7fb09c15ad10 legacy=0x7fb09c0719c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:45.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.486+0000 7fb0a0d2e640 1 -- 172.21.15.88:0/2225645050 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb09c073c00 con 0x7fb09c0cca40 2024-09-16T01:41:45.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.486+0000 7fb09affd640 1 -- 172.21.15.88:0/2225645050 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2582892710 0 0) 0x7fb09c071720 con 0x7fb09c15ad10 2024-09-16T01:41:45.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.486+0000 7fb0a0d2e640 1 --2- 172.21.15.88:0/2225645050 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb09c0cca40 0x7fb09c0b43e0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fb094009e50 tx=0x7fb09402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbf470896a41b23b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:45.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.487+0000 7fb09affd640 1 -- 172.21.15.88:0/2225645050 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb094036070 con 0x7fb09c0cca40 2024-09-16T01:41:45.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.487+0000 7fb09affd640 1 -- 172.21.15.88:0/2225645050 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb094031600 con 0x7fb09c0cca40 2024-09-16T01:41:45.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.487+0000 7fb09affd640 1 -- 172.21.15.88:0/2225645050 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0940319d0 con 0x7fb09c0cca40 2024-09-16T01:41:45.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.487+0000 7fb0a2791640 1 -- 172.21.15.88:0/2225645050 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb09c0cca40 msgr2=0x7fb09c0b43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:45.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.487+0000 7fb0a2791640 1 --2- 172.21.15.88:0/2225645050 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb09c0cca40 0x7fb09c0b43e0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fb094009e50 tx=0x7fb09402a680 comp rx=0 tx=0).stop 2024-09-16T01:41:45.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.487+0000 7fb0a2791640 1 -- 172.21.15.88:0/2225645050 shutdown_connections 2024-09-16T01:41:45.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.487+0000 7fb0a2791640 1 --2- 172.21.15.88:0/2225645050 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb09c0cca40 0x7fb09c0b43e0 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:45.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.487+0000 7fb0a2791640 1 --2- 172.21.15.88:0/2225645050 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb09c0720d0 0x7fb09c0b3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:45.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.487+0000 7fb0a2791640 1 -- 172.21.15.88:0/2225645050 >> 172.21.15.88:0/2225645050 conn(0x7fb09c06d0f0 msgr2=0x7fb09c06d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:45.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.487+0000 7fb0a2791640 1 -- 172.21.15.88:0/2225645050 shutdown_connections 2024-09-16T01:41:45.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.487+0000 7fb0a2791640 1 -- 172.21.15.88:0/2225645050 wait complete. 2024-09-16T01:41:45.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.488+0000 7fb0a2791640 1 Processor -- start 2024-09-16T01:41:45.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.489+0000 7fb0a2791640 1 -- start start 2024-09-16T01:41:45.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.489+0000 7fb0a2791640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb09c0720d0 0x7fb09c14b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:45.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.489+0000 7fb0a2791640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb09c0cca40 0x7fb09c15db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:45.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.489+0000 7fb0a2791640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb09c15ad10 0x7fb09c15cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:45.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.489+0000 7fb0a2791640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb09c0cc360 con 0x7fb09c0720d0 2024-09-16T01:41:45.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.489+0000 7fb0a2791640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb09c071200 con 0x7fb09c15ad10 2024-09-16T01:41:45.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.489+0000 7fb0a2791640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb09c0cc4e0 con 0x7fb09c0cca40 2024-09-16T01:41:45.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.489+0000 7fb09bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb09c0720d0 0x7fb09c14b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:45.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.489+0000 7fb09bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb09c0720d0 0x7fb09c14b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47526/0 (socket says 172.21.15.88:47526) 2024-09-16T01:41:45.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.489+0000 7fb09bfff640 1 -- 172.21.15.88:0/3930356519 learned_addr learned my addr 172.21.15.88:0/3930356519 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:45.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.489+0000 7fb09b7fe640 1 --2- 172.21.15.88:0/3930356519 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb09c0cca40 0x7fb09c15db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:45.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.489+0000 7fb0a0d2e640 1 --2- 172.21.15.88:0/3930356519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb09c15ad10 0x7fb09c15cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:45.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.490+0000 7fb09bfff640 1 -- 172.21.15.88:0/3930356519 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb09c0cca40 msgr2=0x7fb09c15db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:45.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.490+0000 7fb09bfff640 1 --2- 172.21.15.88:0/3930356519 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb09c0cca40 0x7fb09c15db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:45.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.490+0000 7fb09bfff640 1 -- 172.21.15.88:0/3930356519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb09c15ad10 msgr2=0x7fb09c15cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:45.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.490+0000 7fb09bfff640 1 --2- 172.21.15.88:0/3930356519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb09c15ad10 0x7fb09c15cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:45.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.490+0000 7fb09bfff640 1 -- 172.21.15.88:0/3930356519 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb09c15bc90 con 0x7fb09c0720d0 2024-09-16T01:41:45.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.490+0000 7fb09bfff640 1 --2- 172.21.15.88:0/3930356519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb09c0720d0 0x7fb09c14b0f0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fb08400c430 tx=0x7fb08400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:45.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.490+0000 7fb098ff9640 1 -- 172.21.15.88:0/3930356519 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0840074e0 con 0x7fb09c0720d0 2024-09-16T01:41:45.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.490+0000 7fb098ff9640 1 -- 172.21.15.88:0/3930356519 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0840047e0 con 0x7fb09c0720d0 2024-09-16T01:41:45.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.490+0000 7fb0a2791640 1 -- 172.21.15.88:0/3930356519 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb09c15bf70 con 0x7fb09c0720d0 2024-09-16T01:41:45.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.490+0000 7fb098ff9640 1 -- 172.21.15.88:0/3930356519 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb084004bd0 con 0x7fb09c0720d0 2024-09-16T01:41:45.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.490+0000 7fb0a2791640 1 -- 172.21.15.88:0/3930356519 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb09c15c530 con 0x7fb09c0720d0 2024-09-16T01:41:45.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.491+0000 7fb0a2791640 1 -- 172.21.15.88:0/3930356519 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb09c15e040 con 0x7fb09c0720d0 2024-09-16T01:41:45.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.491+0000 7fb098ff9640 1 -- 172.21.15.88:0/3930356519 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb084011070 con 0x7fb09c0720d0 2024-09-16T01:41:45.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.492+0000 7fb098ff9640 1 --2- 172.21.15.88:0/3930356519 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb07c06c2b0 0x7fb07c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:45.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.492+0000 7fb098ff9640 1 -- 172.21.15.88:0/3930356519 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6942+0+0 (secure 0 0 0) 0x7fb08408b440 con 0x7fb09c0720d0 2024-09-16T01:41:45.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.492+0000 7fb09b7fe640 1 --2- 172.21.15.88:0/3930356519 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb07c06c2b0 0x7fb07c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:45.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.492+0000 7fb09b7fe640 1 --2- 172.21.15.88:0/3930356519 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb07c06c2b0 0x7fb07c06e770 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fb09c159930 tx=0x7fb08c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:45.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.496+0000 7fb098ff9640 1 -- 172.21.15.88:0/3930356519 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb084059090 con 0x7fb09c0720d0 2024-09-16T01:41:45.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:45.695+0000 7fb0a2791640 1 -- 172.21.15.88:0/3930356519 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fb09c071940 con 0x7fb09c0720d0 2024-09-16T01:41:46.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.298+0000 7fb098ff9640 1 -- 172.21.15.88:0/3930356519 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v153) ==== 161+0+0 (secure 0 0 0) 0x7fb08405d390 con 0x7fb09c0720d0 2024-09-16T01:41:46.300 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:41:46.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.301+0000 7fb0a2791640 1 -- 172.21.15.88:0/3930356519 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb07c06c2b0 msgr2=0x7fb07c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:46.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.301+0000 7fb0a2791640 1 --2- 172.21.15.88:0/3930356519 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb07c06c2b0 0x7fb07c06e770 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fb09c159930 tx=0x7fb08c008040 comp rx=0 tx=0).stop 2024-09-16T01:41:46.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.301+0000 7fb0a2791640 1 -- 172.21.15.88:0/3930356519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb09c0720d0 msgr2=0x7fb09c14b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:46.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.301+0000 7fb0a2791640 1 --2- 172.21.15.88:0/3930356519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb09c0720d0 0x7fb09c14b0f0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fb08400c430 tx=0x7fb08400c8f0 comp rx=0 tx=0).stop 2024-09-16T01:41:46.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.301+0000 7fb0a2791640 1 -- 172.21.15.88:0/3930356519 shutdown_connections 2024-09-16T01:41:46.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.301+0000 7fb0a2791640 1 --2- 172.21.15.88:0/3930356519 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb07c06c2b0 0x7fb07c06e770 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:46.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.301+0000 7fb0a2791640 1 --2- 172.21.15.88:0/3930356519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb09c15ad10 0x7fb09c15cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:46.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.301+0000 7fb0a2791640 1 --2- 172.21.15.88:0/3930356519 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb09c0cca40 0x7fb09c15db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:46.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.301+0000 7fb0a2791640 1 --2- 172.21.15.88:0/3930356519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb09c0720d0 0x7fb09c14b0f0 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:46.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.301+0000 7fb0a2791640 1 -- 172.21.15.88:0/3930356519 >> 172.21.15.88:0/3930356519 conn(0x7fb09c06d0f0 msgr2=0x7fb09c0b5080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:46.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.301+0000 7fb0a2791640 1 -- 172.21.15.88:0/3930356519 shutdown_connections 2024-09-16T01:41:46.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.301+0000 7fb0a2791640 1 -- 172.21.15.88:0/3930356519 wait complete. 2024-09-16T01:41:46.316 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:41:46.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.418+0000 7fcb51a7b640 1 Processor -- start 2024-09-16T01:41:46.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.418+0000 7fcb51a7b640 1 -- start start 2024-09-16T01:41:46.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.419+0000 7fcb51a7b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb4c0b53f0 0x7fcb4c0d2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:46.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.419+0000 7fcb51a7b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcb4c0d2650 0x7fcb4c0ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:46.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.419+0000 7fcb51a7b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb4c0cdf50 con 0x7fcb4c15c710 2024-09-16T01:41:46.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.419+0000 7fcb51a7b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fcb4c0ce230 con 0x7fcb4c0d2650 2024-09-16T01:41:46.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.419+0000 7fcb51a7b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcb4c0ce510 con 0x7fcb4c0b53f0 2024-09-16T01:41:46.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.420+0000 7fcb42ffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb4c0b53f0 0x7fcb4c0d2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:46.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.420+0000 7fcb4bfff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcb4c0d2650 0x7fcb4c0ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:46.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.420+0000 7fcb4b7fe640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fcb4c15c710 0x7fcb4c073970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:49508/0 (socket says 172.21.15.88:49508) 2024-09-16T01:41:46.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.420+0000 7fcb4b7fe640 1 -- 172.21.15.88:0/192228129 learned_addr learned my addr 172.21.15.88:0/192228129 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:46.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.420+0000 7fcb4bfff640 1 --2- 172.21.15.88:0/192228129 >> v2:172.21.15.154:3300/0 conn(0x7fcb4c0d2650 0x7fcb4c0ceff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:43624/0 (socket says 172.21.15.88:43624) 2024-09-16T01:41:46.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.420+0000 7fcb4bfff640 1 -- 172.21.15.88:0/192228129 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb4c0b53f0 msgr2=0x7fcb4c0d2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:46.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.420+0000 7fcb4bfff640 1 --2- 172.21.15.88:0/192228129 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb4c0b53f0 0x7fcb4c0d2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:46.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.420+0000 7fcb4bfff640 1 -- 172.21.15.88:0/192228129 >> v1:172.21.15.154:6789/0 conn(0x7fcb4c15c710 legacy=0x7fcb4c073970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:46.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.420+0000 7fcb4bfff640 1 -- 172.21.15.88:0/192228129 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb4c0ce7f0 con 0x7fcb4c0d2650 2024-09-16T01:41:46.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.420+0000 7fcb4affd640 1 -- 172.21.15.88:0/192228129 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3883300584 0 0) 0x7fcb4c0cdf50 con 0x7fcb4c15c710 2024-09-16T01:41:46.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.420+0000 7fcb42ffd640 1 --2- 172.21.15.88:0/192228129 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb4c0b53f0 0x7fcb4c0d2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:41:46.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.421+0000 7fcb4bfff640 1 --2- 172.21.15.88:0/192228129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb4c0d2650 0x7fcb4c0ceff0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fcb44009d90 tx=0x7fcb4402a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3a8e11a3a051b2c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:46.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.421+0000 7fcb4affd640 1 -- 172.21.15.88:0/192228129 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb44031600 con 0x7fcb4c0d2650 2024-09-16T01:41:46.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.421+0000 7fcb4affd640 1 -- 172.21.15.88:0/192228129 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb440317a0 con 0x7fcb4c0d2650 2024-09-16T01:41:46.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.421+0000 7fcb4affd640 1 -- 172.21.15.88:0/192228129 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb44031b70 con 0x7fcb4c0d2650 2024-09-16T01:41:46.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.422+0000 7fcb51a7b640 1 -- 172.21.15.88:0/192228129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb4c0d2650 msgr2=0x7fcb4c0ceff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:46.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.422+0000 7fcb51a7b640 1 --2- 172.21.15.88:0/192228129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb4c0d2650 0x7fcb4c0ceff0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fcb44009d90 tx=0x7fcb4402a5d0 comp rx=0 tx=0).stop 2024-09-16T01:41:46.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.422+0000 7fcb51a7b640 1 -- 172.21.15.88:0/192228129 shutdown_connections 2024-09-16T01:41:46.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.422+0000 7fcb51a7b640 1 --2- 172.21.15.88:0/192228129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb4c0d2650 0x7fcb4c0ceff0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:46.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.422+0000 7fcb51a7b640 1 --2- 172.21.15.88:0/192228129 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb4c0b53f0 0x7fcb4c0d2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:46.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.422+0000 7fcb51a7b640 1 -- 172.21.15.88:0/192228129 >> 172.21.15.88:0/192228129 conn(0x7fcb4c062a20 msgr2=0x7fcb4c062e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:46.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.422+0000 7fcb51a7b640 1 -- 172.21.15.88:0/192228129 shutdown_connections 2024-09-16T01:41:46.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.422+0000 7fcb51a7b640 1 -- 172.21.15.88:0/192228129 wait complete. 2024-09-16T01:41:46.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.423+0000 7fcb51a7b640 1 Processor -- start 2024-09-16T01:41:46.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.423+0000 7fcb51a7b640 1 -- start start 2024-09-16T01:41:46.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.424+0000 7fcb51a7b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb4c0b53f0 0x7fcb4c1f57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:46.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.424+0000 7fcb51a7b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb4c0d2650 0x7fcb4c15e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:46.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.424+0000 7fcb51a7b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb4c15c710 0x7fcb4c152e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:46.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.424+0000 7fcb51a7b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4c0d2ac0 con 0x7fcb4c0b53f0 2024-09-16T01:41:46.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.424+0000 7fcb51a7b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4c0cdc00 con 0x7fcb4c0d2650 2024-09-16T01:41:46.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.424+0000 7fcb51a7b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcb4c0d2c40 con 0x7fcb4c15c710 2024-09-16T01:41:46.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.424+0000 7fcb4b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb4c0b53f0 0x7fcb4c1f57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:46.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.424+0000 7fcb4b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb4c0b53f0 0x7fcb4c1f57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47530/0 (socket says 172.21.15.88:47530) 2024-09-16T01:41:46.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.424+0000 7fcb4b7fe640 1 -- 172.21.15.88:0/93709037 learned_addr learned my addr 172.21.15.88:0/93709037 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:46.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.424+0000 7fcb42ffd640 1 --2- 172.21.15.88:0/93709037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb4c0d2650 0x7fcb4c15e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:46.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.424+0000 7fcb4bfff640 1 --2- 172.21.15.88:0/93709037 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb4c15c710 0x7fcb4c152e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:46.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.424+0000 7fcb4b7fe640 1 -- 172.21.15.88:0/93709037 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb4c15c710 msgr2=0x7fcb4c152e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:46.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.424+0000 7fcb4b7fe640 1 --2- 172.21.15.88:0/93709037 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb4c15c710 0x7fcb4c152e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:46.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.424+0000 7fcb4b7fe640 1 -- 172.21.15.88:0/93709037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb4c0d2650 msgr2=0x7fcb4c15e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:46.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.424+0000 7fcb4b7fe640 1 --2- 172.21.15.88:0/93709037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb4c0d2650 0x7fcb4c15e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:46.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.424+0000 7fcb4b7fe640 1 -- 172.21.15.88:0/93709037 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb4c15a8c0 con 0x7fcb4c0b53f0 2024-09-16T01:41:46.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.424+0000 7fcb42ffd640 1 --2- 172.21.15.88:0/93709037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb4c0d2650 0x7fcb4c15e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:46.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.424+0000 7fcb4bfff640 1 --2- 172.21.15.88:0/93709037 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb4c15c710 0x7fcb4c152e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:46.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.424+0000 7fcb4b7fe640 1 --2- 172.21.15.88:0/93709037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb4c0b53f0 0x7fcb4c1f57d0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c00c4d0 tx=0x7fcb3c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:46.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.425+0000 7fcb48ff9640 1 -- 172.21.15.88:0/93709037 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c018070 con 0x7fcb4c0b53f0 2024-09-16T01:41:46.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.425+0000 7fcb48ff9640 1 -- 172.21.15.88:0/93709037 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb3c007500 con 0x7fcb4c0b53f0 2024-09-16T01:41:46.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.425+0000 7fcb51a7b640 1 -- 172.21.15.88:0/93709037 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb4c0cc200 con 0x7fcb4c0b53f0 2024-09-16T01:41:46.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.425+0000 7fcb48ff9640 1 -- 172.21.15.88:0/93709037 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c004a20 con 0x7fcb4c0b53f0 2024-09-16T01:41:46.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.425+0000 7fcb51a7b640 1 -- 172.21.15.88:0/93709037 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb4c0cbb50 con 0x7fcb4c0b53f0 2024-09-16T01:41:46.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.426+0000 7fcb51a7b640 1 -- 172.21.15.88:0/93709037 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb4c0d2110 con 0x7fcb4c0b53f0 2024-09-16T01:41:46.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.426+0000 7fcb48ff9640 1 -- 172.21.15.88:0/93709037 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fcb3c01b070 con 0x7fcb4c0b53f0 2024-09-16T01:41:46.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.427+0000 7fcb48ff9640 1 --2- 172.21.15.88:0/93709037 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcb2c06c2b0 0x7fcb2c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:46.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.427+0000 7fcb48ff9640 1 -- 172.21.15.88:0/93709037 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6942+0+0 (secure 0 0 0) 0x7fcb3c08ac30 con 0x7fcb4c0b53f0 2024-09-16T01:41:46.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.427+0000 7fcb42ffd640 1 --2- 172.21.15.88:0/93709037 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcb2c06c2b0 0x7fcb2c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:46.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.427+0000 7fcb42ffd640 1 --2- 172.21.15.88:0/93709037 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcb2c06c2b0 0x7fcb2c06e770 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c0b1660 tx=0x7fcb34008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:46.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.431+0000 7fcb48ff9640 1 -- 172.21.15.88:0/93709037 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fcb3c058ed0 con 0x7fcb4c0b53f0 2024-09-16T01:41:46.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:46.662+0000 7fcb51a7b640 1 -- 172.21.15.88:0/93709037 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fcb4c0b2060 con 0x7fcb4c0b53f0 2024-09-16T01:41:47.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.307+0000 7fcb48ff9640 1 -- 172.21.15.88:0/93709037 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v154) ==== 101+0+0 (secure 0 0 0) 0x7fcb3c0587d0 con 0x7fcb4c0b53f0 2024-09-16T01:41:47.308 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:41:47.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.309+0000 7fcb51a7b640 1 -- 172.21.15.88:0/93709037 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcb2c06c2b0 msgr2=0x7fcb2c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:47.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.309+0000 7fcb51a7b640 1 --2- 172.21.15.88:0/93709037 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcb2c06c2b0 0x7fcb2c06e770 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c0b1660 tx=0x7fcb34008040 comp rx=0 tx=0).stop 2024-09-16T01:41:47.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.309+0000 7fcb51a7b640 1 -- 172.21.15.88:0/93709037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb4c0b53f0 msgr2=0x7fcb4c1f57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:47.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.309+0000 7fcb51a7b640 1 --2- 172.21.15.88:0/93709037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb4c0b53f0 0x7fcb4c1f57d0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c00c4d0 tx=0x7fcb3c00c990 comp rx=0 tx=0).stop 2024-09-16T01:41:47.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.309+0000 7fcb51a7b640 1 -- 172.21.15.88:0/93709037 shutdown_connections 2024-09-16T01:41:47.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.309+0000 7fcb51a7b640 1 --2- 172.21.15.88:0/93709037 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcb2c06c2b0 0x7fcb2c06e770 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.309+0000 7fcb51a7b640 1 --2- 172.21.15.88:0/93709037 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb4c15c710 0x7fcb4c152e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.309+0000 7fcb51a7b640 1 --2- 172.21.15.88:0/93709037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb4c0d2650 0x7fcb4c15e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.309+0000 7fcb51a7b640 1 --2- 172.21.15.88:0/93709037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb4c0b53f0 0x7fcb4c1f57d0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.309+0000 7fcb51a7b640 1 -- 172.21.15.88:0/93709037 >> 172.21.15.88:0/93709037 conn(0x7fcb4c062a20 msgr2=0x7fcb4c062e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:47.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.309+0000 7fcb51a7b640 1 -- 172.21.15.88:0/93709037 shutdown_connections 2024-09-16T01:41:47.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.309+0000 7fcb51a7b640 1 -- 172.21.15.88:0/93709037 wait complete. 2024-09-16T01:41:47.322 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:41:47.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.364+0000 7f0efc61af40 1 Processor -- start 2024-09-16T01:41:47.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.364+0000 7f0efc61af40 1 -- start start 2024-09-16T01:41:47.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.364+0000 7f0efc61af40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564dd4655ed0 0x564dd46c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.364+0000 7f0efc61af40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564dd46cbe00 0x564dd46c9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.364+0000 7f0efc61af40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dd46e3d40 0x564dd46e5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.364+0000 7f0efc61af40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564dd455f510 con 0x564dd46cbe00 2024-09-16T01:41:47.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.364+0000 7f0efc61af40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564dd46564a0 con 0x564dd46e3d40 2024-09-16T01:41:47.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.364+0000 7f0efc61af40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x564dd46c1590 con 0x564dd4655ed0 2024-09-16T01:41:47.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.365+0000 7f0ef0ca2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564dd46cbe00 0x564dd46c9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:47.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.365+0000 7f0ef0ca2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564dd46cbe00 0x564dd46c9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47554/0 (socket says 172.21.15.88:47554) 2024-09-16T01:41:47.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.365+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/4194011374 learned_addr learned my addr 172.21.15.88:0/4194011374 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:47.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.365+0000 7f0ef9ca4640 1 --2- 172.21.15.88:0/4194011374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dd46e3d40 0x564dd46e5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:47.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.365+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/4194011374 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564dd4655ed0 msgr2=0x564dd46c7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:47.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.365+0000 7f0ef0ca2640 1 --2- 172.21.15.88:0/4194011374 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564dd4655ed0 0x564dd46c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.365+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/4194011374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dd46e3d40 msgr2=0x564dd46e5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:47.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.365+0000 7f0ef0ca2640 1 --2- 172.21.15.88:0/4194011374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dd46e3d40 0x564dd46e5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.365+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/4194011374 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564dd46c10b0 con 0x564dd46cbe00 2024-09-16T01:41:47.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.365+0000 7f0ef0ca2640 1 --2- 172.21.15.88:0/4194011374 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564dd46cbe00 0x564dd46c9f40 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f0ee400b2f0 tx=0x7f0ee400b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa202a92c3195a32 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:47.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.365+0000 7f0ef3fff640 1 -- 172.21.15.88:0/4194011374 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4008070 con 0x564dd46cbe00 2024-09-16T01:41:47.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.365+0000 7f0ef3fff640 1 -- 172.21.15.88:0/4194011374 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ee400d500 con 0x564dd46cbe00 2024-09-16T01:41:47.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.365+0000 7f0ef3fff640 1 -- 172.21.15.88:0/4194011374 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee400d8d0 con 0x564dd46cbe00 2024-09-16T01:41:47.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.366+0000 7f0efc61af40 1 -- 172.21.15.88:0/4194011374 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564dd46cbe00 msgr2=0x564dd46c9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:47.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.366+0000 7f0efc61af40 1 --2- 172.21.15.88:0/4194011374 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564dd46cbe00 0x564dd46c9f40 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f0ee400b2f0 tx=0x7f0ee400b7b0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.366+0000 7f0efc61af40 1 -- 172.21.15.88:0/4194011374 shutdown_connections 2024-09-16T01:41:47.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.366+0000 7f0efc61af40 1 --2- 172.21.15.88:0/4194011374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dd46e3d40 0x564dd46e5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.366+0000 7f0efc61af40 1 --2- 172.21.15.88:0/4194011374 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564dd46cbe00 0x564dd46c9f40 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.366+0000 7f0efc61af40 1 --2- 172.21.15.88:0/4194011374 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564dd4655ed0 0x564dd46c7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.366+0000 7f0efc61af40 1 -- 172.21.15.88:0/4194011374 >> 172.21.15.88:0/4194011374 conn(0x564dd4638960 msgr2=0x564dd46103e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:47.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.366+0000 7f0efc61af40 1 -- 172.21.15.88:0/4194011374 shutdown_connections 2024-09-16T01:41:47.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.366+0000 7f0efc61af40 1 -- 172.21.15.88:0/4194011374 wait complete. 2024-09-16T01:41:47.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.366+0000 7f0efc61af40 1 Processor -- start 2024-09-16T01:41:47.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.366+0000 7f0efc61af40 1 -- start start 2024-09-16T01:41:47.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.367+0000 7f0efc61af40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564dd4655ed0 0x564dd46e2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.367+0000 7f0efc61af40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dd46cbe00 0x564dd460de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.367+0000 7f0efc61af40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564dd46e3d40 0x564dd460e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.367+0000 7f0efc61af40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564dd4655b20 con 0x564dd46e3d40 2024-09-16T01:41:47.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.367+0000 7f0efc61af40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564dd46559a0 con 0x564dd46cbe00 2024-09-16T01:41:47.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.367+0000 7f0efc61af40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x564dd46c7a90 con 0x564dd4655ed0 2024-09-16T01:41:47.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.367+0000 7f0ef9ca4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564dd46e3d40 0x564dd460e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:47.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.367+0000 7f0ef9ca4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564dd46e3d40 0x564dd460e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47566/0 (socket says 172.21.15.88:47566) 2024-09-16T01:41:47.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.367+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 learned_addr learned my addr 172.21.15.88:0/1017134453 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:47.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.367+0000 7f0ef0ca2640 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dd46cbe00 0x564dd460de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:47.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.367+0000 7f0ef94a3640 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564dd4655ed0 0x564dd46e2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:47.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.367+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564dd4655ed0 msgr2=0x564dd46e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:47.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.367+0000 7f0ef9ca4640 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564dd4655ed0 0x564dd46e2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.367+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dd46cbe00 msgr2=0x564dd460de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:47.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.367+0000 7f0ef9ca4640 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564dd46cbe00 0x564dd460de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.367+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564dd46cb1f0 con 0x564dd46e3d40 2024-09-16T01:41:47.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.368+0000 7f0ef9ca4640 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564dd46e3d40 0x564dd460e340 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f0ef400bfc0 tx=0x7f0ef4009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:47.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.368+0000 7f0ef2ffd640 1 -- 172.21.15.88:0/1017134453 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4014020 con 0x564dd46e3d40 2024-09-16T01:41:47.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.368+0000 7f0ef2ffd640 1 -- 172.21.15.88:0/1017134453 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ef4010630 con 0x564dd46e3d40 2024-09-16T01:41:47.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.368+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564dd46c3a60 con 0x564dd46e3d40 2024-09-16T01:41:47.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.368+0000 7f0ef2ffd640 1 -- 172.21.15.88:0/1017134453 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4010a20 con 0x564dd46e3d40 2024-09-16T01:41:47.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.368+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564dd46e3210 con 0x564dd46e3d40 2024-09-16T01:41:47.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.369+0000 7f0ef2ffd640 1 -- 172.21.15.88:0/1017134453 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0ef4017020 con 0x564dd46e3d40 2024-09-16T01:41:47.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.369+0000 7f0ef2ffd640 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0edc06acc0 0x7f0edc06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.370+0000 7f0ef94a3640 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0edc06acc0 0x7f0edc06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:47.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.370+0000 7f0ef2ffd640 1 -- 172.21.15.88:0/1017134453 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6942+0+0 (secure 0 0 0) 0x7f0ef408a640 con 0x564dd46e3d40 2024-09-16T01:41:47.375 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:41:47.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.370+0000 7f0efc61af40 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x564dd476fcc0 0x564dd47700a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.370+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4654450 con 0x564dd476fcc0 2024-09-16T01:41:47.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.370+0000 7f0ef0ca2640 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x564dd476fcc0 0x564dd47700a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:47.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.370+0000 7f0ef94a3640 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0edc06acc0 0x7f0edc06d180 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f0eec057000 tx=0x7f0eec051000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:47.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.370+0000 7f0ef0ca2640 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x564dd476fcc0 0x564dd47700a0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:47.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.372+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v101'412 uv386 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.372+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e101) -- 0x564dd47705e0 con 0x564dd476fcc0 2024-09-16T01:41:47.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.373+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v101'413 uv413 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.373+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4770a40 con 0x564dd476fcc0 2024-09-16T01:41:47.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.374+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v101'414 uv414 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.377 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:41:47.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.374+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4770de0 con 0x564dd476fcc0 2024-09-16T01:41:47.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.375+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v101'415 uv415 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.375+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4788ef0 con 0x564dd476fcc0 2024-09-16T01:41:47.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.377+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v101'416 uv416 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.377+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4788390 con 0x564dd476fcc0 2024-09-16T01:41:47.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.378+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v101'417 uv417 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.378+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4788780 con 0x564dd476fcc0 2024-09-16T01:41:47.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.379+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v101'418 uv392 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.379+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e101) -- 0x564dd4788b20 con 0x564dd476fcc0 2024-09-16T01:41:47.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.380+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v101'419 uv419 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.380+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4789290 con 0x564dd476fcc0 2024-09-16T01:41:47.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.381+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v101'420 uv420 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.382 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:41:47.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.381+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478a2a0 con 0x564dd476fcc0 2024-09-16T01:41:47.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.382+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v101'421 uv421 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.382+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478a2a0 con 0x564dd476fcc0 2024-09-16T01:41:47.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.383+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v101'422 uv422 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.383+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4789290 con 0x564dd476fcc0 2024-09-16T01:41:47.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.385+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v101'423 uv423 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.385+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4789290 con 0x564dd476fcc0 2024-09-16T01:41:47.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.386+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v101'424 uv399 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.386+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e101) -- 0x564dd4789290 con 0x564dd476fcc0 2024-09-16T01:41:47.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.387+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v101'425 uv425 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.387+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4789290 con 0x564dd476fcc0 2024-09-16T01:41:47.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.388+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v101'426 uv426 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.389 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:41:47.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.388+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478a2a0 con 0x564dd476fcc0 2024-09-16T01:41:47.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.389+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v101'427 uv427 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.389+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478a2a0 con 0x564dd476fcc0 2024-09-16T01:41:47.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.390+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v101'428 uv428 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.390+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4789290 con 0x564dd476fcc0 2024-09-16T01:41:47.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.391+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v101'429 uv429 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.391+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4789740 con 0x564dd476fcc0 2024-09-16T01:41:47.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.392+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v101'430 uv430 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.392+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4789740 con 0x564dd476fcc0 2024-09-16T01:41:47.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.393+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v101'431 uv405 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.393+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e101) -- 0x564dd4789740 con 0x564dd476fcc0 2024-09-16T01:41:47.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.394+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v101'432 uv432 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.394+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4789740 con 0x564dd476fcc0 2024-09-16T01:41:47.396 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:41:47.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.395+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v101'433 uv433 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.395+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478a750 con 0x564dd476fcc0 2024-09-16T01:41:47.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.396+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v101'434 uv434 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.396+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478a750 con 0x564dd476fcc0 2024-09-16T01:41:47.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.397+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v101'435 uv435 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.398+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4789740 con 0x564dd476fcc0 2024-09-16T01:41:47.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.399+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v101'436 uv436 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.399+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4789740 con 0x564dd476fcc0 2024-09-16T01:41:47.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.400+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v101'437 uv411 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.400+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e101) -- 0x564dd4789740 con 0x564dd476fcc0 2024-09-16T01:41:47.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.401+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v101'438 uv438 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.401+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4789740 con 0x564dd476fcc0 2024-09-16T01:41:47.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.402+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v101'439 uv439 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.404 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:41:47.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.402+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478a750 con 0x564dd476fcc0 2024-09-16T01:41:47.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.403+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v101'440 uv440 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.403+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478a750 con 0x564dd476fcc0 2024-09-16T01:41:47.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.404+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v101'441 uv441 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.404+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4789740 con 0x564dd476fcc0 2024-09-16T01:41:47.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.405+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v101'442 uv442 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.405+0000 7f0efc61af40 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x564dd4789c50 0x564dd478c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.405+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478c580 con 0x564dd4789c50 2024-09-16T01:41:47.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.406+0000 7f0ef94a3640 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x564dd4789c50 0x564dd478c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:47.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.406+0000 7f0ef94a3640 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x564dd4789c50 0x564dd478c040 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:47.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.408+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v101'309 uv292 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4789c50 2024-09-16T01:41:47.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.408+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e101) -- 0x564dd478c920 con 0x564dd4789c50 2024-09-16T01:41:47.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.409+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v101'310 uv310 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4789c50 2024-09-16T01:41:47.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.409+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478ccc0 con 0x564dd4789c50 2024-09-16T01:41:47.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.410+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v101'311 uv311 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4789c50 2024-09-16T01:41:47.411 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:41:47.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.410+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478d510 con 0x564dd4789c50 2024-09-16T01:41:47.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.411+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v101'312 uv312 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4789c50 2024-09-16T01:41:47.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.411+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478d8b0 con 0x564dd4789c50 2024-09-16T01:41:47.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.412+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v101'313 uv313 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4789c50 2024-09-16T01:41:47.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.412+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478dc50 con 0x564dd4789c50 2024-09-16T01:41:47.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.413+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v101'314 uv297 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4789c50 2024-09-16T01:41:47.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.413+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e101) -- 0x564dd478dff0 con 0x564dd4789c50 2024-09-16T01:41:47.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.414+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v101'315 uv315 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4789c50 2024-09-16T01:41:47.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.414+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478e390 con 0x564dd4789c50 2024-09-16T01:41:47.416 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:41:47.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.415+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v101'316 uv316 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4789c50 2024-09-16T01:41:47.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.415+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478e390 con 0x564dd4789c50 2024-09-16T01:41:47.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.416+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v101'317 uv317 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4789c50 2024-09-16T01:41:47.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.416+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478e390 con 0x564dd4789c50 2024-09-16T01:41:47.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.417+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v101'318 uv318 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4789c50 2024-09-16T01:41:47.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.417+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478e390 con 0x564dd4789c50 2024-09-16T01:41:47.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.418+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v101'319 uv302 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4789c50 2024-09-16T01:41:47.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.418+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e101) -- 0x564dd478e390 con 0x564dd4789c50 2024-09-16T01:41:47.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.419+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v101'320 uv320 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4789c50 2024-09-16T01:41:47.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.419+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478e390 con 0x564dd4789c50 2024-09-16T01:41:47.421 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:41:47.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.420+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v101'321 uv321 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4789c50 2024-09-16T01:41:47.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.420+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4792810 con 0x564dd4789c50 2024-09-16T01:41:47.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.421+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v101'322 uv322 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4789c50 2024-09-16T01:41:47.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.421+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4792810 con 0x564dd4789c50 2024-09-16T01:41:47.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.422+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v101'323 uv323 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4789c50 2024-09-16T01:41:47.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.422+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478e390 con 0x564dd4789c50 2024-09-16T01:41:47.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.423+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v101'324 uv308 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4789c50 2024-09-16T01:41:47.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.423+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e101) -- 0x564dd478e390 con 0x564dd4789c50 2024-09-16T01:41:47.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.424+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v101'325 uv325 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4789c50 2024-09-16T01:41:47.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.424+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478e390 con 0x564dd4789c50 2024-09-16T01:41:47.426 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:41:47.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.425+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v101'326 uv326 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4789c50 2024-09-16T01:41:47.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.425+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478f3a0 con 0x564dd4789c50 2024-09-16T01:41:47.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.426+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v101'327 uv327 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4789c50 2024-09-16T01:41:47.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.426+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478f3a0 con 0x564dd4789c50 2024-09-16T01:41:47.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.427+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v101'328 uv328 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4789c50 2024-09-16T01:41:47.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.427+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478e390 con 0x564dd4789c50 2024-09-16T01:41:47.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.428+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v101'329 uv329 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4789c50 2024-09-16T01:41:47.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.428+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478e390 con 0x564dd476fcc0 2024-09-16T01:41:47.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.429+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v101'81 uv80 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.429+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e101) -- 0x564dd478e390 con 0x564dd476fcc0 2024-09-16T01:41:47.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.430+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v101'82 uv82 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.430+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478e390 con 0x564dd476fcc0 2024-09-16T01:41:47.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.431+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v101'83 uv83 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.433 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:41:47.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.431+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478f3a0 con 0x564dd476fcc0 2024-09-16T01:41:47.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.432+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v101'84 uv84 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.432+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478f3a0 con 0x564dd476fcc0 2024-09-16T01:41:47.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.433+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v101'85 uv85 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.433+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478e390 con 0x564dd476fcc0 2024-09-16T01:41:47.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.434+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v101'86 uv86 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.434+0000 7f0efc61af40 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x564dd478ea10 0x564dd4790e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.434+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd47914a0 con 0x564dd478ea10 2024-09-16T01:41:47.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.434+0000 7f0ef0ca2640 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x564dd478ea10 0x564dd4790e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:47.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.434+0000 7f0ef0ca2640 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x564dd478ea10 0x564dd4790e20 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:47.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.435+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v101'65 uv64 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd478ea10 2024-09-16T01:41:47.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.435+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e101) -- 0x564dd4791890 con 0x564dd478ea10 2024-09-16T01:41:47.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.436+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v101'66 uv66 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd478ea10 2024-09-16T01:41:47.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.436+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4791890 con 0x564dd478ea10 2024-09-16T01:41:47.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.437+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v101'67 uv67 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd478ea10 2024-09-16T01:41:47.438 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:41:47.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.437+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4791890 con 0x564dd478ea10 2024-09-16T01:41:47.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.438+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v101'68 uv68 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd478ea10 2024-09-16T01:41:47.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.438+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd47917b0 con 0x564dd476fcc0 2024-09-16T01:41:47.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.439+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v101'81 uv80 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.439+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e101) -- 0x564dd47917b0 con 0x564dd476fcc0 2024-09-16T01:41:47.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.441+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v101'82 uv82 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.441+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4791ba0 con 0x564dd476fcc0 2024-09-16T01:41:47.443 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:41:47.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.442+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v101'83 uv83 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.442+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4792800 con 0x564dd476fcc0 2024-09-16T01:41:47.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.445+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v101'84 uv84 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.445+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd47927c0 con 0x564dd476fcc0 2024-09-16T01:41:47.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.448+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v101'85 uv85 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.448+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd47917b0 con 0x564dd476fcc0 2024-09-16T01:41:47.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.451+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v101'86 uv86 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.451+0000 7f0efc61af40 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x564dd4791e30 0x564dd4792250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.451+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4794cc0 con 0x564dd4791e30 2024-09-16T01:41:47.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.451+0000 7f0ef9ca4640 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x564dd4791e30 0x564dd4792250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:47.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.451+0000 7f0ef9ca4640 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x564dd4791e30 0x564dd4792250 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:47.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.452+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v101'149 uv143 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.452+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e101) -- 0x564dd4795060 con 0x564dd4791e30 2024-09-16T01:41:47.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.454+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v101'150 uv150 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.454+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4795400 con 0x564dd4791e30 2024-09-16T01:41:47.456 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:41:47.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.455+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v101'151 uv151 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.455+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd478d160 con 0x564dd4791e30 2024-09-16T01:41:47.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.456+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v101'152 uv152 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.456+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4796d40 con 0x564dd4791e30 2024-09-16T01:41:47.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.457+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v101'153 uv153 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.457+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4795c50 con 0x564dd4791e30 2024-09-16T01:41:47.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.458+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v101'154 uv154 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.458+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4796040 con 0x564dd4791e30 2024-09-16T01:41:47.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.459+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v101'155 uv148 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.459+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e101) -- 0x564dd4796430 con 0x564dd4791e30 2024-09-16T01:41:47.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.460+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v101'156 uv156 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.460+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4796430 con 0x564dd4791e30 2024-09-16T01:41:47.462 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:41:47.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.461+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v101'157 uv157 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.461+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd47963e0 con 0x564dd4791e30 2024-09-16T01:41:47.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.462+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v101'158 uv158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.462+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd47963e0 con 0x564dd4791e30 2024-09-16T01:41:47.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.463+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v101'159 uv159 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.463+0000 7f0efc61af40 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x564dd4796820 0x564dd47996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.463+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4799bf0 con 0x564dd4796820 2024-09-16T01:41:47.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.463+0000 7f0ef94a3640 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x564dd4796820 0x564dd47996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:47.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.464+0000 7f0ef94a3640 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x564dd4796820 0x564dd47996b0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.465+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v101'66 uv65 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4796820 2024-09-16T01:41:47.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.465+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e101) -- 0x564dd4799c30 con 0x564dd4796820 2024-09-16T01:41:47.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.466+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v101'67 uv67 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4796820 2024-09-16T01:41:47.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.466+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479a050 con 0x564dd4796820 2024-09-16T01:41:47.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.467+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v101'68 uv68 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4796820 2024-09-16T01:41:47.468 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:41:47.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.467+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479a050 con 0x564dd4796820 2024-09-16T01:41:47.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.468+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v101'69 uv69 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4796820 2024-09-16T01:41:47.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.468+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479a050 con 0x564dd4796820 2024-09-16T01:41:47.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.469+0000 7f0ef94a3640 1 -- 172.21.15.88:0/1017134453 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v101'70 uv70 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0eec052b50 con 0x564dd4796820 2024-09-16T01:41:47.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.469+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479a050 con 0x564dd476fcc0 2024-09-16T01:41:47.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.470+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v101'81 uv80 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.470+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e101) -- 0x564dd479a050 con 0x564dd476fcc0 2024-09-16T01:41:47.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.471+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v101'82 uv82 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.471+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479a050 con 0x564dd476fcc0 2024-09-16T01:41:47.473 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:41:47.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.472+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v101'83 uv83 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.472+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479b060 con 0x564dd476fcc0 2024-09-16T01:41:47.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.473+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v101'84 uv84 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.473+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479b060 con 0x564dd476fcc0 2024-09-16T01:41:47.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.473+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v101'85 uv85 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.474+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479a050 con 0x564dd476fcc0 2024-09-16T01:41:47.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.474+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v101'86 uv86 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd476fcc0 2024-09-16T01:41:47.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.474+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479a050 con 0x564dd478ea10 2024-09-16T01:41:47.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.475+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v101'132 uv126 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd478ea10 2024-09-16T01:41:47.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.475+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e101) -- 0x564dd479a050 con 0x564dd478ea10 2024-09-16T01:41:47.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.476+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v101'133 uv133 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd478ea10 2024-09-16T01:41:47.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.476+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479a050 con 0x564dd478ea10 2024-09-16T01:41:47.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.477+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v101'134 uv134 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd478ea10 2024-09-16T01:41:47.479 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:41:47.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.477+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4795840 con 0x564dd478ea10 2024-09-16T01:41:47.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.478+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v101'135 uv135 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd478ea10 2024-09-16T01:41:47.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.478+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4795840 con 0x564dd478ea10 2024-09-16T01:41:47.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.479+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v101'136 uv136 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd478ea10 2024-09-16T01:41:47.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.479+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4795840 con 0x564dd478ea10 2024-09-16T01:41:47.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.480+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v101'137 uv131 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd478ea10 2024-09-16T01:41:47.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.480+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e101) -- 0x564dd4795840 con 0x564dd478ea10 2024-09-16T01:41:47.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.481+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v101'138 uv138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd478ea10 2024-09-16T01:41:47.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.481+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4795840 con 0x564dd478ea10 2024-09-16T01:41:47.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.482+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v101'139 uv139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd478ea10 2024-09-16T01:41:47.483 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:41:47.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.482+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd4795840 con 0x564dd478ea10 2024-09-16T01:41:47.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.483+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v101'140 uv140 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd478ea10 2024-09-16T01:41:47.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.483+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479ce60 con 0x564dd478ea10 2024-09-16T01:41:47.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.484+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v101'141 uv141 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee400d730 con 0x564dd478ea10 2024-09-16T01:41:47.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.484+0000 7f0efc61af40 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x564dd479ab80 0x564dd479d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.484+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479cf70 con 0x564dd479ab80 2024-09-16T01:41:47.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.484+0000 7f0ef0ca2640 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x564dd479ab80 0x564dd479d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:47.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.485+0000 7f0ef0ca2640 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x564dd479ab80 0x564dd479d4c0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:47.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.486+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v101'61 uv60 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f0ee4029040 con 0x564dd479ab80 2024-09-16T01:41:47.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.486+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e101) -- 0x564dd479cf70 con 0x564dd479ab80 2024-09-16T01:41:47.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.486+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v101'62 uv62 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f0ee4029040 con 0x564dd479ab80 2024-09-16T01:41:47.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.486+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479cf70 con 0x564dd479ab80 2024-09-16T01:41:47.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.487+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v101'63 uv63 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f0ee4029040 con 0x564dd479ab80 2024-09-16T01:41:47.489 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:41:47.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.487+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479cf70 con 0x564dd479ab80 2024-09-16T01:41:47.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.488+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v101'64 uv64 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f0ee4029040 con 0x564dd479ab80 2024-09-16T01:41:47.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.488+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479cf70 con 0x564dd478ea10 2024-09-16T01:41:47.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.489+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v101'81 uv80 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0ee4029040 con 0x564dd478ea10 2024-09-16T01:41:47.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.489+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e101) -- 0x564dd479cf70 con 0x564dd478ea10 2024-09-16T01:41:47.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.490+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v101'82 uv82 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee4029040 con 0x564dd478ea10 2024-09-16T01:41:47.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.490+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479cf70 con 0x564dd478ea10 2024-09-16T01:41:47.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.492+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v101'83 uv83 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee4029040 con 0x564dd478ea10 2024-09-16T01:41:47.493 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:41:47.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.492+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479cf70 con 0x564dd478ea10 2024-09-16T01:41:47.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.493+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v101'84 uv84 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee4029040 con 0x564dd478ea10 2024-09-16T01:41:47.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.493+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479cf70 con 0x564dd478ea10 2024-09-16T01:41:47.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.494+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v101'85 uv85 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee4029040 con 0x564dd478ea10 2024-09-16T01:41:47.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.494+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479cf70 con 0x564dd478ea10 2024-09-16T01:41:47.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.494+0000 7f0ef0ca2640 1 -- 172.21.15.88:0/1017134453 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v101'86 uv86 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ee4029040 con 0x564dd478ea10 2024-09-16T01:41:47.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.495+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479cf70 con 0x564dd4791e30 2024-09-16T01:41:47.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.496+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v101'69 uv68 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.496+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e101) -- 0x564dd479cf70 con 0x564dd4791e30 2024-09-16T01:41:47.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.497+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v101'70 uv70 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.497+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479cf70 con 0x564dd4791e30 2024-09-16T01:41:47.501 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:41:47.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.499+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v101'71 uv71 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.500+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479cf70 con 0x564dd4791e30 2024-09-16T01:41:47.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.501+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v101'72 uv72 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.501+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479cf70 con 0x564dd4791e30 2024-09-16T01:41:47.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.502+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v101'73 uv73 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.502+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479cf70 con 0x564dd4791e30 2024-09-16T01:41:47.504 INFO:teuthology.orchestra.run.smithi088.stdout:Write #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:41:47.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.503+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v101'170 uv170 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.503+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479cf70 con 0x564dd4791e30 2024-09-16T01:41:47.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.504+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v101'171 uv171 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.504+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479cf70 con 0x564dd4791e30 2024-09-16T01:41:47.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.505+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v101'172 uv172 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.505+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479cf70 con 0x564dd4791e30 2024-09-16T01:41:47.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.507+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 1.00000000 [create] v101'173 uv168 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.507+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e101) -- 0x564dd479cf70 con 0x564dd4791e30 2024-09-16T01:41:47.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.508+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 1.00000000 [delete] v101'174 uv174 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.508+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479cf70 con 0x564dd4791e30 2024-09-16T01:41:47.509 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:41:47.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.508+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v101'175 uv175 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.508+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479cf70 con 0x564dd4791e30 2024-09-16T01:41:47.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.509+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v101'176 uv176 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.509+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479cf70 con 0x564dd4791e30 2024-09-16T01:41:47.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.510+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v101'177 uv177 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.510+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479cf70 con 0x564dd4791e30 2024-09-16T01:41:47.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.511+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [omap-set-header] v101'178 uv178 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.511+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101) -- 0x564dd479da00 con 0x564dd4791e30 2024-09-16T01:41:47.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v101'179 uv179 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f0ef4020070 con 0x564dd4791e30 2024-09-16T01:41:47.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x564dd476fcc0 msgr2=0x564dd47700a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:47.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0efc61af40 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x564dd476fcc0 0x564dd47700a0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x564dd478ea10 msgr2=0x564dd4790e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:47.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0efc61af40 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x564dd478ea10 0x564dd4790e20 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x564dd479ab80 msgr2=0x564dd479d4c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:47.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0efc61af40 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x564dd479ab80 0x564dd479d4c0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x564dd4789c50 msgr2=0x564dd478c040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:47.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0efc61af40 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x564dd4789c50 0x564dd478c040 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 reap_dead start 2024-09-16T01:41:47.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x564dd4791e30 msgr2=0x564dd4792250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:47.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0efc61af40 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x564dd4791e30 0x564dd4792250 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0ef9ca4640 1 -- 172.21.15.88:0/1017134453 reap_dead start 2024-09-16T01:41:47.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x564dd4796820 msgr2=0x564dd47996b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:47.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0efc61af40 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x564dd4796820 0x564dd47996b0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0edc06acc0 msgr2=0x7f0edc06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:47.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0efc61af40 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0edc06acc0 0x7f0edc06d180 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f0eec057000 tx=0x7f0eec051000 comp rx=0 tx=0).stop 2024-09-16T01:41:47.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564dd46e3d40 msgr2=0x564dd460e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:47.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0efc61af40 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564dd46e3d40 0x564dd460e340 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f0ef400bfc0 tx=0x7f0ef4009560 comp rx=0 tx=0).stop 2024-09-16T01:41:47.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 shutdown_connections 2024-09-16T01:41:47.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0efc61af40 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0edc06acc0 0x7f0edc06d180 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0efc61af40 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564dd46e3d40 0x564dd460e340 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0efc61af40 1 --2- 172.21.15.88:0/1017134453 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x564dd4796820 0x564dd47996b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 >> 172.21.15.88:0/1017134453 conn(0x564dd4638960 msgr2=0x564dd46c9f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:47.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 shutdown_connections 2024-09-16T01:41:47.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.512+0000 7f0efc61af40 1 -- 172.21.15.88:0/1017134453 wait complete. 2024-09-16T01:41:47.517 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 1.00000000 2024-09-16T01:41:47.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.598+0000 7fa050336f40 1 Processor -- start 2024-09-16T01:41:47.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.598+0000 7fa050336f40 1 -- start start 2024-09-16T01:41:47.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.599+0000 7fa050336f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572a7151510 0x5572a71e1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.599+0000 7fa050336f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5572a7152820 0x5572a71c7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.599+0000 7fa050336f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572a71c51b0 0x5572a71c3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.599+0000 7fa050336f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5572a71c5590 con 0x5572a7151510 2024-09-16T01:41:47.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.599+0000 7fa050336f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5572a705c510 con 0x5572a71c51b0 2024-09-16T01:41:47.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.599+0000 7fa050336f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5572a7150550 con 0x5572a7152820 2024-09-16T01:41:47.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.599+0000 7fa04de2c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572a7151510 0x5572a71e1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:47.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.599+0000 7fa04de2c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572a7151510 0x5572a71e1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47578/0 (socket says 172.21.15.88:47578) 2024-09-16T01:41:47.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.599+0000 7fa04de2c640 1 -- 172.21.15.88:0/4285761597 learned_addr learned my addr 172.21.15.88:0/4285761597 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:47.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.599+0000 7fa04de2c640 1 -- 172.21.15.88:0/4285761597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5572a7152820 msgr2=0x5572a71c7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:47.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.599+0000 7fa04de2c640 1 --2- 172.21.15.88:0/4285761597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5572a7152820 0x5572a71c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.599+0000 7fa04de2c640 1 -- 172.21.15.88:0/4285761597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572a71c51b0 msgr2=0x5572a71c3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:47.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.599+0000 7fa04de2c640 1 --2- 172.21.15.88:0/4285761597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572a71c51b0 0x5572a71c3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.599+0000 7fa04de2c640 1 -- 172.21.15.88:0/4285761597 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572a71dc550 con 0x5572a7151510 2024-09-16T01:41:47.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.599+0000 7fa04de2c640 1 --2- 172.21.15.88:0/4285761597 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572a7151510 0x5572a71e1190 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fa04000a770 tx=0x7fa04000a860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bafeec7f80b8685b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:47.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.600+0000 7fa03ffff640 1 -- 172.21.15.88:0/4285761597 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa040018070 con 0x5572a7151510 2024-09-16T01:41:47.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.600+0000 7fa03ffff640 1 -- 172.21.15.88:0/4285761597 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa040007600 con 0x5572a7151510 2024-09-16T01:41:47.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.600+0000 7fa03ffff640 1 -- 172.21.15.88:0/4285761597 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0400079b0 con 0x5572a7151510 2024-09-16T01:41:47.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.600+0000 7fa050336f40 1 -- 172.21.15.88:0/4285761597 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572a7151510 msgr2=0x5572a71e1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:47.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.600+0000 7fa050336f40 1 --2- 172.21.15.88:0/4285761597 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572a7151510 0x5572a71e1190 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fa04000a770 tx=0x7fa04000a860 comp rx=0 tx=0).stop 2024-09-16T01:41:47.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.600+0000 7fa050336f40 1 -- 172.21.15.88:0/4285761597 shutdown_connections 2024-09-16T01:41:47.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.600+0000 7fa050336f40 1 --2- 172.21.15.88:0/4285761597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572a71c51b0 0x5572a71c3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.600+0000 7fa050336f40 1 --2- 172.21.15.88:0/4285761597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5572a7152820 0x5572a71c7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.600+0000 7fa050336f40 1 --2- 172.21.15.88:0/4285761597 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572a7151510 0x5572a71e1190 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.600+0000 7fa050336f40 1 -- 172.21.15.88:0/4285761597 >> 172.21.15.88:0/4285761597 conn(0x5572a7109be0 msgr2=0x5572a71c6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:47.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.600+0000 7fa050336f40 1 -- 172.21.15.88:0/4285761597 shutdown_connections 2024-09-16T01:41:47.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.600+0000 7fa050336f40 1 -- 172.21.15.88:0/4285761597 wait complete. 2024-09-16T01:41:47.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.601+0000 7fa050336f40 1 Processor -- start 2024-09-16T01:41:47.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.601+0000 7fa050336f40 1 -- start start 2024-09-16T01:41:47.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.601+0000 7fa050336f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572a7151510 0x5572a71da310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.601+0000 7fa050336f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5572a7152820 0x5572a71c3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.601+0000 7fa050336f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572a71c51b0 0x5572a71bf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.601+0000 7fa050336f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5572a71c7eb0 con 0x5572a71c51b0 2024-09-16T01:41:47.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.601+0000 7fa050336f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5572a71c5910 con 0x5572a7151510 2024-09-16T01:41:47.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.601+0000 7fa050336f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5572a71c5bf0 con 0x5572a7152820 2024-09-16T01:41:47.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.601+0000 7fa04e62d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572a71c51b0 0x5572a71bf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:47.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.602+0000 7fa04e62d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572a71c51b0 0x5572a71bf3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47594/0 (socket says 172.21.15.88:47594) 2024-09-16T01:41:47.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.602+0000 7fa04e62d640 1 -- 172.21.15.88:0/816939071 learned_addr learned my addr 172.21.15.88:0/816939071 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:47.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.602+0000 7fa04d62b640 1 --2- 172.21.15.88:0/816939071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5572a7152820 0x5572a71c3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:47.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.602+0000 7fa04de2c640 1 --2- 172.21.15.88:0/816939071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572a7151510 0x5572a71da310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:47.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.602+0000 7fa04e62d640 1 -- 172.21.15.88:0/816939071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5572a7152820 msgr2=0x5572a71c3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:47.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.602+0000 7fa04e62d640 1 --2- 172.21.15.88:0/816939071 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5572a7152820 0x5572a71c3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.602+0000 7fa04e62d640 1 -- 172.21.15.88:0/816939071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572a7151510 msgr2=0x5572a71da310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:47.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.602+0000 7fa04e62d640 1 --2- 172.21.15.88:0/816939071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5572a7151510 0x5572a71da310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.602+0000 7fa04e62d640 1 -- 172.21.15.88:0/816939071 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572a71dab90 con 0x5572a71c51b0 2024-09-16T01:41:47.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.602+0000 7fa04e62d640 1 --2- 172.21.15.88:0/816939071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572a71c51b0 0x5572a71bf3e0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fa048005dc0 tx=0x7fa048005a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:47.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.602+0000 7fa03effd640 1 -- 172.21.15.88:0/816939071 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa048058400 con 0x5572a71c51b0 2024-09-16T01:41:47.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.602+0000 7fa03effd640 1 -- 172.21.15.88:0/816939071 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0480585a0 con 0x5572a71c51b0 2024-09-16T01:41:47.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.602+0000 7fa050336f40 1 -- 172.21.15.88:0/816939071 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5572a7278da0 con 0x5572a71c51b0 2024-09-16T01:41:47.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.602+0000 7fa03effd640 1 -- 172.21.15.88:0/816939071 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa048062610 con 0x5572a71c51b0 2024-09-16T01:41:47.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.602+0000 7fa050336f40 1 -- 172.21.15.88:0/816939071 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572a7279320 con 0x5572a71c51b0 2024-09-16T01:41:47.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.604+0000 7fa03effd640 1 -- 172.21.15.88:0/816939071 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa0480627b0 con 0x5572a71c51b0 2024-09-16T01:41:47.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.604+0000 7fa03effd640 1 --2- 172.21.15.88:0/816939071 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa03006ac90 0x7fa03006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.604+0000 7fa04de2c640 1 --2- 172.21.15.88:0/816939071 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa03006ac90 0x7fa03006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:47.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.604+0000 7fa03effd640 1 -- 172.21.15.88:0/816939071 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6942+0+0 (secure 0 0 0) 0x7fa0480ce340 con 0x5572a71c51b0 2024-09-16T01:41:47.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.605+0000 7fa050336f40 1 --2- 172.21.15.88:0/816939071 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5572a71c2410 0x5572a71c2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.605+0000 7fa050336f40 1 -- 172.21.15.88:0/816939071 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:1 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e101) -- 0x5572a71c2d60 con 0x5572a71c2410 2024-09-16T01:41:47.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.605+0000 7fa04de2c640 1 --2- 172.21.15.88:0/816939071 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa03006ac90 0x7fa03006d150 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fa040016040 tx=0x7fa04000c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:47.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.605+0000 7fa04d62b640 1 --2- 172.21.15.88:0/816939071 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5572a71c2410 0x5572a71c2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:47.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.605+0000 7fa04d62b640 1 --2- 172.21.15.88:0/816939071 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5572a71c2410 0x5572a71c2820 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:47.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.606+0000 7fa04d62b640 1 -- 172.21.15.88:0/816939071 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(1 1.00000000 [delete] v101'180 uv180 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa04400e580 con 0x5572a71c2410 2024-09-16T01:41:47.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.607+0000 7fa050336f40 1 -- 172.21.15.88:0/816939071 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5572a71c2410 msgr2=0x5572a71c2820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:47.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.607+0000 7fa050336f40 1 --2- 172.21.15.88:0/816939071 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5572a71c2410 0x5572a71c2820 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.607+0000 7fa050336f40 1 -- 172.21.15.88:0/816939071 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa03006ac90 msgr2=0x7fa03006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:47.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.607+0000 7fa050336f40 1 --2- 172.21.15.88:0/816939071 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa03006ac90 0x7fa03006d150 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fa040016040 tx=0x7fa04000c000 comp rx=0 tx=0).stop 2024-09-16T01:41:47.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.607+0000 7fa050336f40 1 -- 172.21.15.88:0/816939071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572a71c51b0 msgr2=0x5572a71bf3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:47.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.607+0000 7fa050336f40 1 --2- 172.21.15.88:0/816939071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5572a71c51b0 0x5572a71bf3e0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fa048005dc0 tx=0x7fa048005a80 comp rx=0 tx=0).stop 2024-09-16T01:41:47.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.607+0000 7fa04e62d640 1 -- 172.21.15.88:0/816939071 reap_dead start 2024-09-16T01:41:47.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.607+0000 7fa050336f40 1 -- 172.21.15.88:0/816939071 shutdown_connections 2024-09-16T01:41:47.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.607+0000 7fa050336f40 1 -- 172.21.15.88:0/816939071 >> 172.21.15.88:0/816939071 conn(0x5572a7109be0 msgr2=0x5572a7109120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:47.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.607+0000 7fa050336f40 1 -- 172.21.15.88:0/816939071 shutdown_connections 2024-09-16T01:41:47.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.607+0000 7fa050336f40 1 -- 172.21.15.88:0/816939071 wait complete. 2024-09-16T01:41:47.613 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:41:47.618 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:41:47.655 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:41:47.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.755+0000 7f3cc9c27640 1 Processor -- start 2024-09-16T01:41:47.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.755+0000 7f3cc9c27640 1 -- start start 2024-09-16T01:41:47.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.756+0000 7f3cc9c27640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3cc415ad10 0x7f3cc40719c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.756+0000 7f3cc9c27640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cc4071720 con 0x7f3cc40cc360 2024-09-16T01:41:47.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.756+0000 7f3cc9c27640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cc415f530 con 0x7f3cc40cca40 2024-09-16T01:41:47.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.756+0000 7f3cc9c27640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3cc4070fb0 con 0x7f3cc415ad10 2024-09-16T01:41:47.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.756+0000 7f3cc3fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3cc40cc360 0x7f3cc415ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34738/0 (socket says 172.21.15.88:34738) 2024-09-16T01:41:47.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.756+0000 7f3cc3fff640 1 -- 172.21.15.88:0/2682442978 learned_addr learned my addr 172.21.15.88:0/2682442978 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:47.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.756+0000 7f3cc37fe640 1 --2- 172.21.15.88:0/2682442978 >> v2:172.21.15.154:3300/0 conn(0x7f3cc415ad10 0x7f3cc40719c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:47.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.756+0000 7f3cc27fc640 1 -- 172.21.15.88:0/2682442978 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2430924386 0 0) 0x7f3cc4071720 con 0x7f3cc40cc360 2024-09-16T01:41:47.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.756+0000 7f3cc27fc640 1 -- 172.21.15.88:0/2682442978 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cac003520 con 0x7f3cc40cc360 2024-09-16T01:41:47.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.757+0000 7f3cc27fc640 1 -- 172.21.15.88:0/2682442978 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 482735024 0 0) 0x7f3cac003520 con 0x7f3cc40cc360 2024-09-16T01:41:47.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.757+0000 7f3cc37fe640 1 -- 172.21.15.88:0/2682442978 >> v1:172.21.15.154:6789/0 conn(0x7f3cc40cca40 legacy=0x7f3cc415e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:47.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.757+0000 7f3cc37fe640 1 -- 172.21.15.88:0/2682442978 >> v1:172.21.15.88:6789/0 conn(0x7f3cc40cc360 legacy=0x7f3cc415ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:47.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.757+0000 7f3cc37fe640 1 -- 172.21.15.88:0/2682442978 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cc4073920 con 0x7f3cc415ad10 2024-09-16T01:41:47.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.757+0000 7f3cc37fe640 1 --2- 172.21.15.88:0/2682442978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cc415ad10 0x7f3cc40719c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3cb40097a0 tx=0x7f3cb402a130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=560ec8a9f0f65e03 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:47.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.758+0000 7f3cc27fc640 1 -- 172.21.15.88:0/2682442978 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb4037070 con 0x7f3cc415ad10 2024-09-16T01:41:47.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.758+0000 7f3cc27fc640 1 -- 172.21.15.88:0/2682442978 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cb402abc0 con 0x7f3cc415ad10 2024-09-16T01:41:47.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.758+0000 7f3cc27fc640 1 -- 172.21.15.88:0/2682442978 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc0019f0 con 0x7f3cc415ad10 2024-09-16T01:41:47.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.758+0000 7f3cc9c27640 1 -- 172.21.15.88:0/2682442978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cc415ad10 msgr2=0x7f3cc40719c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:47.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.758+0000 7f3cc9c27640 1 --2- 172.21.15.88:0/2682442978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cc415ad10 0x7f3cc40719c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3cb40097a0 tx=0x7f3cb402a130 comp rx=0 tx=0).stop 2024-09-16T01:41:47.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.758+0000 7f3cc9c27640 1 -- 172.21.15.88:0/2682442978 shutdown_connections 2024-09-16T01:41:47.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.758+0000 7f3cc9c27640 1 --2- 172.21.15.88:0/2682442978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cc415ad10 0x7f3cc40719c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.758+0000 7f3cc9c27640 1 -- 172.21.15.88:0/2682442978 >> 172.21.15.88:0/2682442978 conn(0x7f3cc406d0f0 msgr2=0x7f3cc406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:47.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.758+0000 7f3cc9c27640 1 -- 172.21.15.88:0/2682442978 shutdown_connections 2024-09-16T01:41:47.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.758+0000 7f3cc9c27640 1 -- 172.21.15.88:0/2682442978 wait complete. 2024-09-16T01:41:47.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.759+0000 7f3cc9c27640 1 Processor -- start 2024-09-16T01:41:47.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.760+0000 7f3cc9c27640 1 -- start start 2024-09-16T01:41:47.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.760+0000 7f3cc9c27640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3cc40cc360 0x7f3cc41483f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.760+0000 7f3cc9c27640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cc40cca40 0x7f3cc414eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.760+0000 7f3cc9c27640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cc415ad10 0x7f3cc415d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.760+0000 7f3cc9c27640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc4071200 con 0x7f3cc40cc360 2024-09-16T01:41:47.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.760+0000 7f3cc9c27640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc4073ee0 con 0x7f3cc415ad10 2024-09-16T01:41:47.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.760+0000 7f3cc9c27640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3cc40b4eb0 con 0x7f3cc40cca40 2024-09-16T01:41:47.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.760+0000 7f3cc37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3cc40cc360 0x7f3cc41483f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:47.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.760+0000 7f3cc37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3cc40cc360 0x7f3cc41483f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47602/0 (socket says 172.21.15.88:47602) 2024-09-16T01:41:47.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.760+0000 7f3cc37fe640 1 -- 172.21.15.88:0/2995049200 learned_addr learned my addr 172.21.15.88:0/2995049200 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:47.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.761+0000 7f3cc3fff640 1 --2- 172.21.15.88:0/2995049200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cc415ad10 0x7f3cc415d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:47.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.761+0000 7f3cc2ffd640 1 --2- 172.21.15.88:0/2995049200 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cc40cca40 0x7f3cc414eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:47.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.761+0000 7f3cc37fe640 1 -- 172.21.15.88:0/2995049200 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cc40cca40 msgr2=0x7f3cc414eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:47.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.761+0000 7f3cc37fe640 1 --2- 172.21.15.88:0/2995049200 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cc40cca40 0x7f3cc414eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.761+0000 7f3cc37fe640 1 -- 172.21.15.88:0/2995049200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cc415ad10 msgr2=0x7f3cc415d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:47.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.761+0000 7f3cc37fe640 1 --2- 172.21.15.88:0/2995049200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cc415ad10 0x7f3cc415d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:47.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.761+0000 7f3cc37fe640 1 -- 172.21.15.88:0/2995049200 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cc4145e40 con 0x7f3cc40cc360 2024-09-16T01:41:47.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.761+0000 7f3cc37fe640 1 --2- 172.21.15.88:0/2995049200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3cc40cc360 0x7f3cc41483f0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f3cb40098c0 tx=0x7f3cb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:47.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.761+0000 7f3cabfff640 1 -- 172.21.15.88:0/2995049200 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb4037050 con 0x7f3cc40cc360 2024-09-16T01:41:47.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.761+0000 7f3cabfff640 1 -- 172.21.15.88:0/2995049200 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cb4043070 con 0x7f3cc40cc360 2024-09-16T01:41:47.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.761+0000 7f3cc9c27640 1 -- 172.21.15.88:0/2995049200 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cc4146120 con 0x7f3cc40cc360 2024-09-16T01:41:47.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.761+0000 7f3cabfff640 1 -- 172.21.15.88:0/2995049200 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb4006e10 con 0x7f3cc40cc360 2024-09-16T01:41:47.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.761+0000 7f3cc9c27640 1 -- 172.21.15.88:0/2995049200 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cc4146400 con 0x7f3cc40cc360 2024-09-16T01:41:47.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.762+0000 7f3cc9c27640 1 -- 172.21.15.88:0/2995049200 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cc4152c30 con 0x7f3cc40cc360 2024-09-16T01:41:47.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.762+0000 7f3cabfff640 1 -- 172.21.15.88:0/2995049200 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3cb403ca10 con 0x7f3cc40cc360 2024-09-16T01:41:47.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.763+0000 7f3cabfff640 1 --2- 172.21.15.88:0/2995049200 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3cb006c2b0 0x7f3cb006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:47.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.763+0000 7f3cabfff640 1 -- 172.21.15.88:0/2995049200 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6942+0+0 (secure 0 0 0) 0x7f3cb40ac2b0 con 0x7f3cc40cc360 2024-09-16T01:41:47.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.763+0000 7f3cc2ffd640 1 --2- 172.21.15.88:0/2995049200 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3cb006c2b0 0x7f3cb006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:47.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.763+0000 7f3cc2ffd640 1 --2- 172.21.15.88:0/2995049200 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3cb006c2b0 0x7f3cb006e770 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f3cc415aa70 tx=0x7f3cb8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:47.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.767+0000 7f3cabfff640 1 -- 172.21.15.88:0/2995049200 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3cb4079f90 con 0x7f3cc40cc360 2024-09-16T01:41:47.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:47.972+0000 7f3cc9c27640 1 -- 172.21.15.88:0/2995049200 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f3cc4063360 con 0x7f3cc40cc360 2024-09-16T01:41:48.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.259+0000 7f3cabfff640 1 -- 172.21.15.88:0/2995049200 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v155) ==== 162+0+0 (secure 0 0 0) 0x7f3cb407e290 con 0x7f3cc40cc360 2024-09-16T01:41:48.261 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:41:48.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.262+0000 7f3cc9c27640 1 -- 172.21.15.88:0/2995049200 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3cb006c2b0 msgr2=0x7f3cb006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:48.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.262+0000 7f3cc9c27640 1 --2- 172.21.15.88:0/2995049200 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3cb006c2b0 0x7f3cb006e770 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f3cc415aa70 tx=0x7f3cb8009040 comp rx=0 tx=0).stop 2024-09-16T01:41:48.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.262+0000 7f3cc9c27640 1 -- 172.21.15.88:0/2995049200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3cc40cc360 msgr2=0x7f3cc41483f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:48.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.262+0000 7f3cc9c27640 1 --2- 172.21.15.88:0/2995049200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3cc40cc360 0x7f3cc41483f0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f3cb40098c0 tx=0x7f3cb40046c0 comp rx=0 tx=0).stop 2024-09-16T01:41:48.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.262+0000 7f3cc9c27640 1 -- 172.21.15.88:0/2995049200 shutdown_connections 2024-09-16T01:41:48.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.262+0000 7f3cc9c27640 1 --2- 172.21.15.88:0/2995049200 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3cb006c2b0 0x7f3cb006e770 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:48.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.262+0000 7f3cc9c27640 1 --2- 172.21.15.88:0/2995049200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cc415ad10 0x7f3cc415d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:48.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.262+0000 7f3cc9c27640 1 --2- 172.21.15.88:0/2995049200 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cc40cca40 0x7f3cc414eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:48.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.262+0000 7f3cc9c27640 1 --2- 172.21.15.88:0/2995049200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3cc40cc360 0x7f3cc41483f0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:48.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.262+0000 7f3cc9c27640 1 -- 172.21.15.88:0/2995049200 >> 172.21.15.88:0/2995049200 conn(0x7f3cc406d0f0 msgr2=0x7f3cc40b46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:48.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.262+0000 7f3cc9c27640 1 -- 172.21.15.88:0/2995049200 shutdown_connections 2024-09-16T01:41:48.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.262+0000 7f3cc9c27640 1 -- 172.21.15.88:0/2995049200 wait complete. 2024-09-16T01:41:48.277 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:48.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.376+0000 7fdf1b378640 1 Processor -- start 2024-09-16T01:41:48.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.377+0000 7fdf1b378640 1 -- start start 2024-09-16T01:41:48.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.377+0000 7fdf1b378640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf140b5be0 0x7fdf1415ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:48.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.377+0000 7fdf1b378640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdf140b29a0 0x7fdf140cde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:48.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.377+0000 7fdf1b378640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf140b6830 con 0x7fdf1415a9c0 2024-09-16T01:41:48.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.377+0000 7fdf1b378640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fdf1415f410 con 0x7fdf140b29a0 2024-09-16T01:41:48.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.377+0000 7fdf1b378640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdf140cd0b0 con 0x7fdf140b5be0 2024-09-16T01:41:48.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.378+0000 7fdf19114640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fdf1415a9c0 0x7fdf1415e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:34752/0 (socket says 172.21.15.88:34752) 2024-09-16T01:41:48.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.378+0000 7fdf19114640 1 -- 172.21.15.88:0/612102568 learned_addr learned my addr 172.21.15.88:0/612102568 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:48.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.378+0000 7fdf0bfff640 1 -- 172.21.15.88:0/612102568 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2958580581 0 0) 0x7fdf140b6830 con 0x7fdf1415a9c0 2024-09-16T01:41:48.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.378+0000 7fdf0bfff640 1 -- 172.21.15.88:0/612102568 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf00003520 con 0x7fdf1415a9c0 2024-09-16T01:41:48.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.378+0000 7fdf19915640 1 --2- 172.21.15.88:0/612102568 >> v2:172.21.15.154:3300/0 conn(0x7fdf140b29a0 0x7fdf140cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:48.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.378+0000 7fdf18913640 1 --2- 172.21.15.88:0/612102568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf140b5be0 0x7fdf1415ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:48.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.378+0000 7fdf0bfff640 1 -- 172.21.15.88:0/612102568 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2659658884 0 0) 0x7fdf00003520 con 0x7fdf1415a9c0 2024-09-16T01:41:48.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.378+0000 7fdf0bfff640 1 -- 172.21.15.88:0/612102568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf140b5be0 msgr2=0x7fdf1415ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:48.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.378+0000 7fdf0bfff640 1 --2- 172.21.15.88:0/612102568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf140b5be0 0x7fdf1415ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:48.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.378+0000 7fdf0bfff640 1 -- 172.21.15.88:0/612102568 >> v2:172.21.15.154:3300/0 conn(0x7fdf140b29a0 msgr2=0x7fdf140cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:48.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.378+0000 7fdf0bfff640 1 --2- 172.21.15.88:0/612102568 >> v2:172.21.15.154:3300/0 conn(0x7fdf140b29a0 0x7fdf140cde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:48.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.378+0000 7fdf0bfff640 1 -- 172.21.15.88:0/612102568 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf140cd390 con 0x7fdf1415a9c0 2024-09-16T01:41:48.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.378+0000 7fdf0bfff640 1 -- 172.21.15.88:0/612102568 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fdf10001670 con 0x7fdf1415a9c0 2024-09-16T01:41:48.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.378+0000 7fdf18913640 1 --2- 172.21.15.88:0/612102568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf140b5be0 0x7fdf1415ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:48.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.378+0000 7fdf0bfff640 1 -- 172.21.15.88:0/612102568 >> v1:172.21.15.88:6789/0 conn(0x7fdf1415a9c0 legacy=0x7fdf1415e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:48.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.378+0000 7fdf0bfff640 1 --2- 172.21.15.88:0/612102568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf000039b0 0x7fdf00003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:48.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.378+0000 7fdf0bfff640 1 --2- 172.21.15.88:0/612102568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf00004800 0x7fdf00004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:48.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.378+0000 7fdf0bfff640 1 --2- 172.21.15.88:0/612102568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf140b5be0 0x7fdf000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:48.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.378+0000 7fdf0bfff640 1 -- 172.21.15.88:0/612102568 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1415f410 con 0x7fdf000039b0 2024-09-16T01:41:48.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.378+0000 7fdf0bfff640 1 -- 172.21.15.88:0/612102568 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdf140cd0b0 con 0x7fdf00004800 2024-09-16T01:41:48.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.378+0000 7fdf0bfff640 1 -- 172.21.15.88:0/612102568 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdf0000b9c0 con 0x7fdf140b5be0 2024-09-16T01:41:48.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.378+0000 7fdf19114640 1 --2- 172.21.15.88:0/612102568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf000039b0 0x7fdf00003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:48.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.378+0000 7fdf18913640 1 --2- 172.21.15.88:0/612102568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf00004800 0x7fdf00004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:48.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.378+0000 7fdf19915640 1 --2- 172.21.15.88:0/612102568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf140b5be0 0x7fdf000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:48.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.379+0000 7fdf19114640 1 -- 172.21.15.88:0/612102568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf140b5be0 msgr2=0x7fdf000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:48.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.379+0000 7fdf19114640 1 --2- 172.21.15.88:0/612102568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf140b5be0 0x7fdf000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:48.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.379+0000 7fdf19114640 1 -- 172.21.15.88:0/612102568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf00004800 msgr2=0x7fdf00004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:48.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.379+0000 7fdf19114640 1 --2- 172.21.15.88:0/612102568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf00004800 0x7fdf00004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:48.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.379+0000 7fdf19114640 1 -- 172.21.15.88:0/612102568 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf00003520 con 0x7fdf000039b0 2024-09-16T01:41:48.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.379+0000 7fdf19114640 1 --2- 172.21.15.88:0/612102568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf000039b0 0x7fdf00003dd0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fdf1000c2c0 tx=0x7fdf00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a387f30e56af20fa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:48.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.379+0000 7fdf0bfff640 1 -- 172.21.15.88:0/612102568 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf10007d50 con 0x7fdf000039b0 2024-09-16T01:41:48.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.379+0000 7fdf0bfff640 1 -- 172.21.15.88:0/612102568 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf10033070 con 0x7fdf000039b0 2024-09-16T01:41:48.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.379+0000 7fdf0bfff640 1 -- 172.21.15.88:0/612102568 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf10005d30 con 0x7fdf000039b0 2024-09-16T01:41:48.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.380+0000 7fdf1b378640 1 -- 172.21.15.88:0/612102568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf000039b0 msgr2=0x7fdf00003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:48.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.380+0000 7fdf1b378640 1 --2- 172.21.15.88:0/612102568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf000039b0 0x7fdf00003dd0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fdf1000c2c0 tx=0x7fdf00005830 comp rx=0 tx=0).stop 2024-09-16T01:41:48.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.380+0000 7fdf19915640 1 -- 172.21.15.88:0/612102568 reap_dead start 2024-09-16T01:41:48.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.380+0000 7fdf1b378640 1 -- 172.21.15.88:0/612102568 shutdown_connections 2024-09-16T01:41:48.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.380+0000 7fdf1b378640 1 -- 172.21.15.88:0/612102568 >> 172.21.15.88:0/612102568 conn(0x7fdf1406bc10 msgr2=0x7fdf1406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:48.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.380+0000 7fdf1b378640 1 -- 172.21.15.88:0/612102568 shutdown_connections 2024-09-16T01:41:48.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.380+0000 7fdf1b378640 1 -- 172.21.15.88:0/612102568 wait complete. 2024-09-16T01:41:48.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.381+0000 7fdf1b378640 1 Processor -- start 2024-09-16T01:41:48.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.381+0000 7fdf1b378640 1 -- start start 2024-09-16T01:41:48.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.382+0000 7fdf1b378640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf000039b0 0x7fdf1414e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:48.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.382+0000 7fdf1b378640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf14152470 0x7fdf140c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:48.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.382+0000 7fdf1b378640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf1415c0a0 0x7fdf1415c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:48.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.382+0000 7fdf1b378640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdf1415ada0 con 0x7fdf1415c0a0 2024-09-16T01:41:48.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.382+0000 7fdf1b378640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdf140cd950 con 0x7fdf000039b0 2024-09-16T01:41:48.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.382+0000 7fdf1b378640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdf1415ed10 con 0x7fdf14152470 2024-09-16T01:41:48.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.382+0000 7fdf19915640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf1415c0a0 0x7fdf1415c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:48.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.382+0000 7fdf19915640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf1415c0a0 0x7fdf1415c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47626/0 (socket says 172.21.15.88:47626) 2024-09-16T01:41:48.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.382+0000 7fdf19915640 1 -- 172.21.15.88:0/1189693095 learned_addr learned my addr 172.21.15.88:0/1189693095 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:48.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.382+0000 7fdf19114640 1 --2- 172.21.15.88:0/1189693095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf000039b0 0x7fdf1414e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:48.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.382+0000 7fdf18913640 1 --2- 172.21.15.88:0/1189693095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf14152470 0x7fdf140c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:48.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.382+0000 7fdf19915640 1 -- 172.21.15.88:0/1189693095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf14152470 msgr2=0x7fdf140c5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:48.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.382+0000 7fdf19915640 1 --2- 172.21.15.88:0/1189693095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf14152470 0x7fdf140c5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:48.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.382+0000 7fdf19915640 1 -- 172.21.15.88:0/1189693095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf000039b0 msgr2=0x7fdf1414e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:48.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.382+0000 7fdf19915640 1 --2- 172.21.15.88:0/1189693095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf000039b0 0x7fdf1414e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:48.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.382+0000 7fdf19915640 1 -- 172.21.15.88:0/1189693095 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf141f0110 con 0x7fdf1415c0a0 2024-09-16T01:41:48.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.382+0000 7fdf18913640 1 --2- 172.21.15.88:0/1189693095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf14152470 0x7fdf140c5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:48.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.382+0000 7fdf19915640 1 --2- 172.21.15.88:0/1189693095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf1415c0a0 0x7fdf1415c530 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fdf140b2e10 tx=0x7fdf0c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:48.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.383+0000 7fdf09ffb640 1 -- 172.21.15.88:0/1189693095 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c011e70 con 0x7fdf1415c0a0 2024-09-16T01:41:48.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.383+0000 7fdf09ffb640 1 -- 172.21.15.88:0/1189693095 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf0c01b070 con 0x7fdf1415c0a0 2024-09-16T01:41:48.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.383+0000 7fdf09ffb640 1 -- 172.21.15.88:0/1189693095 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c003990 con 0x7fdf1415c0a0 2024-09-16T01:41:48.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.383+0000 7fdf1b378640 1 -- 172.21.15.88:0/1189693095 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf141f03b0 con 0x7fdf1415c0a0 2024-09-16T01:41:48.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.383+0000 7fdf1b378640 1 -- 172.21.15.88:0/1189693095 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf141f0970 con 0x7fdf1415c0a0 2024-09-16T01:41:48.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.384+0000 7fdf1b378640 1 -- 172.21.15.88:0/1189693095 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf14071920 con 0x7fdf1415c0a0 2024-09-16T01:41:48.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.384+0000 7fdf09ffb640 1 -- 172.21.15.88:0/1189693095 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fdf0c005d10 con 0x7fdf1415c0a0 2024-09-16T01:41:48.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.385+0000 7fdf09ffb640 1 --2- 172.21.15.88:0/1189693095 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdef806c2b0 0x7fdef806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:48.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.385+0000 7fdf09ffb640 1 -- 172.21.15.88:0/1189693095 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6942+0+0 (secure 0 0 0) 0x7fdf0c08bc20 con 0x7fdf1415c0a0 2024-09-16T01:41:48.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.385+0000 7fdf19114640 1 --2- 172.21.15.88:0/1189693095 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdef806c2b0 0x7fdef806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:48.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.385+0000 7fdf19114640 1 --2- 172.21.15.88:0/1189693095 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdef806c2b0 0x7fdef806e770 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fdf10037000 tx=0x7fdf10007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:48.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.389+0000 7fdf09ffb640 1 -- 172.21.15.88:0/1189693095 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fdf14071920 con 0x7fdf1415c0a0 2024-09-16T01:41:48.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.590+0000 7fdf1b378640 1 -- 172.21.15.88:0/1189693095 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf1415f0f0 con 0x7fdf1415c0a0 2024-09-16T01:41:48.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.590+0000 7fdf09ffb640 1 -- 172.21.15.88:0/1189693095 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) ==== 95+0+4398 (secure 0 0 0) 0x7fdf0c059ec0 con 0x7fdf1415c0a0 2024-09-16T01:41:48.592 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:48.592 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":155,"btime":"2024-09-16T01:41:48:259175+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:48.259161+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8310},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8310":{"gid":8310,"name":"a","rank":0,"incarnation":155,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6836/981209627","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":981209627},{"type":"v1","addr":"172.21.15.88:6836","nonce":981209627}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:41:48.592 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 155 2024-09-16T01:41:48.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.592+0000 7fdf1b378640 1 -- 172.21.15.88:0/1189693095 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdef806c2b0 msgr2=0x7fdef806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:48.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.592+0000 7fdf1b378640 1 --2- 172.21.15.88:0/1189693095 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdef806c2b0 0x7fdef806e770 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fdf10037000 tx=0x7fdf10007000 comp rx=0 tx=0).stop 2024-09-16T01:41:48.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.592+0000 7fdf1b378640 1 -- 172.21.15.88:0/1189693095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf1415c0a0 msgr2=0x7fdf1415c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:48.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.592+0000 7fdf1b378640 1 --2- 172.21.15.88:0/1189693095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf1415c0a0 0x7fdf1415c530 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fdf140b2e10 tx=0x7fdf0c0115a0 comp rx=0 tx=0).stop 2024-09-16T01:41:48.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.592+0000 7fdf1b378640 1 -- 172.21.15.88:0/1189693095 shutdown_connections 2024-09-16T01:41:48.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.592+0000 7fdf1b378640 1 --2- 172.21.15.88:0/1189693095 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdef806c2b0 0x7fdef806e770 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:48.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.592+0000 7fdf1b378640 1 --2- 172.21.15.88:0/1189693095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdf1415c0a0 0x7fdf1415c530 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:48.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.592+0000 7fdf1b378640 1 --2- 172.21.15.88:0/1189693095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf14152470 0x7fdf140c5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:48.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.592+0000 7fdf1b378640 1 --2- 172.21.15.88:0/1189693095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf000039b0 0x7fdf1414e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:48.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.592+0000 7fdf1b378640 1 -- 172.21.15.88:0/1189693095 >> 172.21.15.88:0/1189693095 conn(0x7fdf1406bc10 msgr2=0x7fdf140ce250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:48.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.592+0000 7fdf1b378640 1 -- 172.21.15.88:0/1189693095 shutdown_connections 2024-09-16T01:41:48.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:48.592+0000 7fdf1b378640 1 -- 172.21.15.88:0/1189693095 wait complete. 2024-09-16T01:41:48.605 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-09-16T01:41:49.047 INFO:tasks.ceph.mds.c.smithi088.stderr: -46> 2024-09-16T01:41:44.293+0000 7f323d588640 -1 mds.0.cache.ino(0x100) No data while reading inode 0x100 2024-09-16T01:41:49.047 INFO:tasks.ceph.mds.c.smithi088.stderr: -36> 2024-09-16T01:41:44.293+0000 7f323d588640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-09-16T01:41:49.071 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:41:51.267 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:41:51.266+0000 7f3c370db640 -1 log_channel(cluster) log [ERR] : dir 0x1 object missing on disk; some files may be lost () 2024-09-16T01:41:51.267 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:41:51.266+0000 7f3c370db640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x1 is fatal because it is a system directory for this rank 2024-09-16T01:41:51.269 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:41:51.268+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:41:51.269 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:41:51.268+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:41:53.606 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:53.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.714+0000 7ff837991640 1 Processor -- start 2024-09-16T01:41:53.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.715+0000 7ff837991640 1 -- start start 2024-09-16T01:41:53.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.715+0000 7ff837991640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff8300b4920 0x7ff83015b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:53.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.715+0000 7ff837991640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff8300b4d00 0x7ff8300b5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:53.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.715+0000 7ff837991640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8300cd780 0x7ff8300cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:53.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.715+0000 7ff837991640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff8300cf100 con 0x7ff8300b4920 2024-09-16T01:41:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.715+0000 7ff837991640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff8300ce4f0 con 0x7ff8300b4d00 2024-09-16T01:41:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.715+0000 7ff837991640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff8300ce7d0 con 0x7ff8300cd780 2024-09-16T01:41:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.716+0000 7ff83572d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff8300b4920 0x7ff83015b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.716+0000 7ff83572d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff8300b4920 0x7ff83015b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53758/0 (socket says 172.21.15.88:53758) 2024-09-16T01:41:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.716+0000 7ff83572d640 1 -- 172.21.15.88:0/2680418959 learned_addr learned my addr 172.21.15.88:0/2680418959 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.716+0000 7ff835f2e640 1 --2- 172.21.15.88:0/2680418959 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8300cd780 0x7ff8300cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.716+0000 7ff834f2c640 1 --2- 172.21.15.88:0/2680418959 >> v2:172.21.15.154:3300/0 conn(0x7ff8300b4d00 0x7ff8300b5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.716+0000 7ff83572d640 1 -- 172.21.15.88:0/2680418959 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8300cd780 msgr2=0x7ff8300cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.716+0000 7ff83572d640 1 --2- 172.21.15.88:0/2680418959 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8300cd780 0x7ff8300cdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.716+0000 7ff83572d640 1 -- 172.21.15.88:0/2680418959 >> v2:172.21.15.154:3300/0 conn(0x7ff8300b4d00 msgr2=0x7ff8300b5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.716+0000 7ff83572d640 1 --2- 172.21.15.88:0/2680418959 >> v2:172.21.15.154:3300/0 conn(0x7ff8300b4d00 0x7ff8300b5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.716+0000 7ff83572d640 1 -- 172.21.15.88:0/2680418959 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8300ceab0 con 0x7ff8300b4920 2024-09-16T01:41:53.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.717+0000 7ff83572d640 1 --2- 172.21.15.88:0/2680418959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8300b4920 0x7ff83015b7d0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7ff8300b29e0 tx=0x7ff82802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e4c43fda8e340e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:53.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.717+0000 7ff82ffff640 1 -- 172.21.15.88:0/2680418959 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff828037070 con 0x7ff8300b4920 2024-09-16T01:41:53.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.717+0000 7ff82ffff640 1 -- 172.21.15.88:0/2680418959 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff828032440 con 0x7ff8300b4920 2024-09-16T01:41:53.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.717+0000 7ff82ffff640 1 -- 172.21.15.88:0/2680418959 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff828032810 con 0x7ff8300b4920 2024-09-16T01:41:53.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.717+0000 7ff837991640 1 -- 172.21.15.88:0/2680418959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8300b4920 msgr2=0x7ff83015b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:53.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.717+0000 7ff837991640 1 --2- 172.21.15.88:0/2680418959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8300b4920 0x7ff83015b7d0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7ff8300b29e0 tx=0x7ff82802a1e0 comp rx=0 tx=0).stop 2024-09-16T01:41:53.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.717+0000 7ff837991640 1 -- 172.21.15.88:0/2680418959 shutdown_connections 2024-09-16T01:41:53.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.717+0000 7ff837991640 1 --2- 172.21.15.88:0/2680418959 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8300cd780 0x7ff8300cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:53.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.717+0000 7ff837991640 1 --2- 172.21.15.88:0/2680418959 >> v2:172.21.15.154:3300/0 conn(0x7ff8300b4d00 0x7ff8300b5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:53.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.717+0000 7ff837991640 1 --2- 172.21.15.88:0/2680418959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8300b4920 0x7ff83015b7d0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:53.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.717+0000 7ff837991640 1 -- 172.21.15.88:0/2680418959 >> 172.21.15.88:0/2680418959 conn(0x7ff83006bc10 msgr2=0x7ff83006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:53.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.718+0000 7ff837991640 1 -- 172.21.15.88:0/2680418959 shutdown_connections 2024-09-16T01:41:53.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.718+0000 7ff837991640 1 -- 172.21.15.88:0/2680418959 wait complete. 2024-09-16T01:41:53.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.719+0000 7ff837991640 1 Processor -- start 2024-09-16T01:41:53.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.719+0000 7ff837991640 1 -- start start 2024-09-16T01:41:53.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.719+0000 7ff837991640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8300b4920 0x7ff830142f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:53.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.719+0000 7ff837991640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8300b4d00 0x7ff830150b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:53.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.719+0000 7ff837991640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8300cd780 0x7ff830151070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:53.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.719+0000 7ff837991640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff83015bd10 con 0x7ff8300cd780 2024-09-16T01:41:53.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.719+0000 7ff837991640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff8300b56b0 con 0x7ff8300b4d00 2024-09-16T01:41:53.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.719+0000 7ff837991640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff8300ce110 con 0x7ff8300b4920 2024-09-16T01:41:53.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.720+0000 7ff835f2e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8300cd780 0x7ff830151070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:53.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.720+0000 7ff835f2e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8300cd780 0x7ff830151070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53772/0 (socket says 172.21.15.88:53772) 2024-09-16T01:41:53.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.720+0000 7ff835f2e640 1 -- 172.21.15.88:0/3757540751 learned_addr learned my addr 172.21.15.88:0/3757540751 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:53.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.720+0000 7ff83572d640 1 --2- 172.21.15.88:0/3757540751 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8300b4920 0x7ff830142f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:53.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.720+0000 7ff834f2c640 1 --2- 172.21.15.88:0/3757540751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8300b4d00 0x7ff830150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:53.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.720+0000 7ff835f2e640 1 -- 172.21.15.88:0/3757540751 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8300b4920 msgr2=0x7ff830142f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:53.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.720+0000 7ff835f2e640 1 --2- 172.21.15.88:0/3757540751 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8300b4920 0x7ff830142f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:53.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.720+0000 7ff835f2e640 1 -- 172.21.15.88:0/3757540751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8300b4d00 msgr2=0x7ff830150b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:53.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.720+0000 7ff835f2e640 1 --2- 172.21.15.88:0/3757540751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8300b4d00 0x7ff830150b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:53.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.720+0000 7ff835f2e640 1 -- 172.21.15.88:0/3757540751 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8301581d0 con 0x7ff8300cd780 2024-09-16T01:41:53.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.720+0000 7ff835f2e640 1 --2- 172.21.15.88:0/3757540751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8300cd780 0x7ff830151070 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7ff82000cc80 tx=0x7ff82000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:53.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.720+0000 7ff82dffb640 1 -- 172.21.15.88:0/3757540751 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82000ce30 con 0x7ff8300cd780 2024-09-16T01:41:53.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.720+0000 7ff82dffb640 1 -- 172.21.15.88:0/3757540751 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff820003990 con 0x7ff8300cd780 2024-09-16T01:41:53.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.720+0000 7ff837991640 1 -- 172.21.15.88:0/3757540751 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8301584b0 con 0x7ff8300cd780 2024-09-16T01:41:53.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.720+0000 7ff82dffb640 1 -- 172.21.15.88:0/3757540751 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff820003d60 con 0x7ff8300cd780 2024-09-16T01:41:53.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.721+0000 7ff837991640 1 -- 172.21.15.88:0/3757540751 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff830158a70 con 0x7ff8300cd780 2024-09-16T01:41:53.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.721+0000 7ff837991640 1 -- 172.21.15.88:0/3757540751 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8300b5170 con 0x7ff8300cd780 2024-09-16T01:41:53.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.722+0000 7ff82dffb640 1 -- 172.21.15.88:0/3757540751 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff820003f00 con 0x7ff8300cd780 2024-09-16T01:41:53.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.722+0000 7ff82dffb640 1 --2- 172.21.15.88:0/3757540751 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff81c06c2e0 0x7ff81c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:53.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.723+0000 7ff82dffb640 1 -- 172.21.15.88:0/3757540751 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7028+0+0 (secure 0 0 0) 0x7ff82008bc60 con 0x7ff8300cd780 2024-09-16T01:41:53.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.723+0000 7ff83572d640 1 --2- 172.21.15.88:0/3757540751 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff81c06c2e0 0x7ff81c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:53.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.723+0000 7ff83572d640 1 --2- 172.21.15.88:0/3757540751 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff81c06c2e0 0x7ff81c06e7a0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff828034000 tx=0x7ff82802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:53.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.727+0000 7ff82dffb640 1 -- 172.21.15.88:0/3757540751 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff820014030 con 0x7ff8300cd780 2024-09-16T01:41:53.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.925+0000 7ff837991640 1 -- 172.21.15.88:0/3757540751 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff830072620 con 0x7ff8300cd780 2024-09-16T01:41:53.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.926+0000 7ff82dffb640 1 -- 172.21.15.88:0/3757540751 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) ==== 95+0+4392 (secure 0 0 0) 0x7ff820059850 con 0x7ff8300cd780 2024-09-16T01:41:53.927 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:53.927 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":159,"btime":"2024-09-16T01:41:51:268733+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/3163343083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":3163343083},{"type":"v1","addr":"172.21.15.88:6837","nonce":3163343083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:51.268729+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:41:53.928 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 159 2024-09-16T01:41:53.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.928+0000 7ff837991640 1 -- 172.21.15.88:0/3757540751 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff81c06c2e0 msgr2=0x7ff81c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:53.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.928+0000 7ff837991640 1 --2- 172.21.15.88:0/3757540751 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff81c06c2e0 0x7ff81c06e7a0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff828034000 tx=0x7ff82802e000 comp rx=0 tx=0).stop 2024-09-16T01:41:53.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.928+0000 7ff837991640 1 -- 172.21.15.88:0/3757540751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8300cd780 msgr2=0x7ff830151070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:53.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.928+0000 7ff837991640 1 --2- 172.21.15.88:0/3757540751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8300cd780 0x7ff830151070 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7ff82000cc80 tx=0x7ff82000a5c0 comp rx=0 tx=0).stop 2024-09-16T01:41:53.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.928+0000 7ff837991640 1 -- 172.21.15.88:0/3757540751 shutdown_connections 2024-09-16T01:41:53.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.928+0000 7ff837991640 1 --2- 172.21.15.88:0/3757540751 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff81c06c2e0 0x7ff81c06e7a0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:53.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.928+0000 7ff837991640 1 --2- 172.21.15.88:0/3757540751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8300cd780 0x7ff830151070 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:53.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.928+0000 7ff837991640 1 --2- 172.21.15.88:0/3757540751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8300b4d00 0x7ff830150b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:53.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.928+0000 7ff837991640 1 --2- 172.21.15.88:0/3757540751 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8300b4920 0x7ff830142f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:53.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.928+0000 7ff837991640 1 -- 172.21.15.88:0/3757540751 >> 172.21.15.88:0/3757540751 conn(0x7ff83006bc10 msgr2=0x7ff830071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:53.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.928+0000 7ff837991640 1 -- 172.21.15.88:0/3757540751 shutdown_connections 2024-09-16T01:41:53.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:53.928+0000 7ff837991640 1 -- 172.21.15.88:0/3757540751 wait complete. 2024-09-16T01:41:53.940 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-09-16T01:41:53.940 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:54.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.038+0000 7fdde6959640 1 Processor -- start 2024-09-16T01:41:54.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.038+0000 7fdde6959640 1 -- start start 2024-09-16T01:41:54.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.038+0000 7fdde6959640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdde015c170 0x7fdde015c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:54.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.038+0000 7fdde6959640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdde00b4cf0 con 0x7fdde00b64e0 2024-09-16T01:41:54.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.038+0000 7fdde6959640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdde00b4fd0 con 0x7fdde00cf6c0 2024-09-16T01:41:54.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.038+0000 7fdde6959640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fdde015ca90 con 0x7fdde015c170 2024-09-16T01:41:54.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.039+0000 7fdde4ef6640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fdde00b64e0 0x7fdde00b39b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:56858/0 (socket says 172.21.15.88:56858) 2024-09-16T01:41:54.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.039+0000 7fdde4ef6640 1 -- 172.21.15.88:0/1786594541 learned_addr learned my addr 172.21.15.88:0/1786594541 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:54.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.039+0000 7fdddffff640 1 --2- 172.21.15.88:0/1786594541 >> v2:172.21.15.154:3300/0 conn(0x7fdde015c170 0x7fdde015c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:54.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.039+0000 7fdddeffd640 1 -- 172.21.15.88:0/1786594541 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4097588718 0 0) 0x7fdde00b4cf0 con 0x7fdde00b64e0 2024-09-16T01:41:54.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.039+0000 7fdddeffd640 1 -- 172.21.15.88:0/1786594541 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddc8003520 con 0x7fdde00b64e0 2024-09-16T01:41:54.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.039+0000 7fdddeffd640 1 -- 172.21.15.88:0/1786594541 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 704849615 0 0) 0x7fddc8003520 con 0x7fdde00b64e0 2024-09-16T01:41:54.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.039+0000 7fdddeffd640 1 -- 172.21.15.88:0/1786594541 >> v2:172.21.15.154:3300/0 conn(0x7fdde015c170 msgr2=0x7fdde015c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:54.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.039+0000 7fdddeffd640 1 --2- 172.21.15.88:0/1786594541 >> v2:172.21.15.154:3300/0 conn(0x7fdde015c170 0x7fdde015c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:54.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.039+0000 7fdddeffd640 1 -- 172.21.15.88:0/1786594541 >> v1:172.21.15.154:6789/0 conn(0x7fdde00cf6c0 legacy=0x7fdde00b5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:54.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.039+0000 7fdddeffd640 1 -- 172.21.15.88:0/1786594541 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdde00b52b0 con 0x7fdde00b64e0 2024-09-16T01:41:54.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.039+0000 7fdddeffd640 1 -- 172.21.15.88:0/1786594541 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fddd80019f0 con 0x7fdde00b64e0 2024-09-16T01:41:54.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.039+0000 7fdddeffd640 1 -- 172.21.15.88:0/1786594541 >> v1:172.21.15.88:6789/0 conn(0x7fdde00b64e0 legacy=0x7fdde00b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:54.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.039+0000 7fdddeffd640 1 --2- 172.21.15.88:0/1786594541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddc8003a00 0x7fddc8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:54.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.039+0000 7fdddeffd640 1 --2- 172.21.15.88:0/1786594541 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fddc8004800 0x7fddc8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:54.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.039+0000 7fdddeffd640 1 --2- 172.21.15.88:0/1786594541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddc8005210 0x7fddc800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:54.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.039+0000 7fdddeffd640 1 -- 172.21.15.88:0/1786594541 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdde015ca90 con 0x7fddc8003a00 2024-09-16T01:41:54.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.039+0000 7fdddeffd640 1 -- 172.21.15.88:0/1786594541 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fddc800c160 con 0x7fddc8005210 2024-09-16T01:41:54.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.039+0000 7fdddeffd640 1 -- 172.21.15.88:0/1786594541 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fddc800c360 con 0x7fddc8004800 2024-09-16T01:41:54.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.039+0000 7fdddffff640 1 --2- 172.21.15.88:0/1786594541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddc8003a00 0x7fddc8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:54.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.039+0000 7fdde4ef6640 1 --2- 172.21.15.88:0/1786594541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddc8005210 0x7fddc800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:54.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.039+0000 7fdddf7fe640 1 --2- 172.21.15.88:0/1786594541 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fddc8004800 0x7fddc8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:54.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.040+0000 7fdddffff640 1 -- 172.21.15.88:0/1786594541 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fddc8004800 msgr2=0x7fddc8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:54.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.040+0000 7fdddffff640 1 --2- 172.21.15.88:0/1786594541 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fddc8004800 0x7fddc8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:54.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.040+0000 7fdddffff640 1 -- 172.21.15.88:0/1786594541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddc8005210 msgr2=0x7fddc800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:54.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.040+0000 7fdddffff640 1 --2- 172.21.15.88:0/1786594541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddc8005210 0x7fddc800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:54.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.040+0000 7fdde4ef6640 1 -- 172.21.15.88:0/1786594541 reap_dead start 2024-09-16T01:41:54.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.040+0000 7fdddffff640 1 -- 172.21.15.88:0/1786594541 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddc8003520 con 0x7fddc8003a00 2024-09-16T01:41:54.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.040+0000 7fdddf7fe640 1 --2- 172.21.15.88:0/1786594541 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fddc8004800 0x7fddc8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:54.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.040+0000 7fdddffff640 1 --2- 172.21.15.88:0/1786594541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddc8003a00 0x7fddc8003e20 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fddd0009a90 tx=0x7fddc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f8703a4b3e4deb0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:54.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.040+0000 7fdddeffd640 1 -- 172.21.15.88:0/1786594541 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd0036070 con 0x7fddc8003a00 2024-09-16T01:41:54.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.040+0000 7fdddeffd640 1 -- 172.21.15.88:0/1786594541 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddd0031400 con 0x7fddc8003a00 2024-09-16T01:41:54.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.040+0000 7fdddeffd640 1 -- 172.21.15.88:0/1786594541 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd0009690 con 0x7fddc8003a00 2024-09-16T01:41:54.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.041+0000 7fdde6959640 1 -- 172.21.15.88:0/1786594541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddc8003a00 msgr2=0x7fddc8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:54.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.041+0000 7fdde6959640 1 --2- 172.21.15.88:0/1786594541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddc8003a00 0x7fddc8003e20 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fddd0009a90 tx=0x7fddc8005830 comp rx=0 tx=0).stop 2024-09-16T01:41:54.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.041+0000 7fdde6959640 1 -- 172.21.15.88:0/1786594541 shutdown_connections 2024-09-16T01:41:54.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.041+0000 7fdde6959640 1 --2- 172.21.15.88:0/1786594541 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddc8003a00 0x7fddc8003e20 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:54.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.041+0000 7fdde6959640 1 -- 172.21.15.88:0/1786594541 >> 172.21.15.88:0/1786594541 conn(0x7fdde006bc10 msgr2=0x7fdde006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:54.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.041+0000 7fdde6959640 1 -- 172.21.15.88:0/1786594541 shutdown_connections 2024-09-16T01:41:54.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.041+0000 7fdde6959640 1 -- 172.21.15.88:0/1786594541 wait complete. 2024-09-16T01:41:54.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.042+0000 7fdde6959640 1 Processor -- start 2024-09-16T01:41:54.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.042+0000 7fdde6959640 1 -- start start 2024-09-16T01:41:54.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.043+0000 7fdde6959640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fddc8003a00 0x7fdde014c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:54.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.043+0000 7fdde6959640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdde00cadf0 0x7fdde00cb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:54.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.043+0000 7fdde6959640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdde01597e0 0x7fdde0159c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:54.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.043+0000 7fdde6959640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdde00b5b50 con 0x7fdde01597e0 2024-09-16T01:41:54.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.043+0000 7fdde6959640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdde00b5870 con 0x7fdde00cadf0 2024-09-16T01:41:54.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.043+0000 7fdde6959640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdde015c550 con 0x7fddc8003a00 2024-09-16T01:41:54.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.043+0000 7fdde4ef6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdde01597e0 0x7fdde0159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:54.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.043+0000 7fdde4ef6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdde01597e0 0x7fdde0159c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53794/0 (socket says 172.21.15.88:53794) 2024-09-16T01:41:54.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.043+0000 7fdde4ef6640 1 -- 172.21.15.88:0/489925384 learned_addr learned my addr 172.21.15.88:0/489925384 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:54.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.043+0000 7fdddf7fe640 1 --2- 172.21.15.88:0/489925384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdde00cadf0 0x7fdde00cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:54.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.043+0000 7fdddffff640 1 --2- 172.21.15.88:0/489925384 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fddc8003a00 0x7fdde014c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:54.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.043+0000 7fdde4ef6640 1 -- 172.21.15.88:0/489925384 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fddc8003a00 msgr2=0x7fdde014c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:54.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.043+0000 7fdde4ef6640 1 --2- 172.21.15.88:0/489925384 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fddc8003a00 0x7fdde014c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:54.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.043+0000 7fdde4ef6640 1 -- 172.21.15.88:0/489925384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdde00cadf0 msgr2=0x7fdde00cb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:54.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.043+0000 7fdde4ef6640 1 --2- 172.21.15.88:0/489925384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdde00cadf0 0x7fdde00cb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:54.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.043+0000 7fdde4ef6640 1 -- 172.21.15.88:0/489925384 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdde015a510 con 0x7fdde01597e0 2024-09-16T01:41:54.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.043+0000 7fdddf7fe640 1 --2- 172.21.15.88:0/489925384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdde00cadf0 0x7fdde00cb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:54.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.044+0000 7fdde4ef6640 1 --2- 172.21.15.88:0/489925384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdde01597e0 0x7fdde0159c90 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fdde015ff10 tx=0x7fddd8007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:54.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.044+0000 7fdddcff9640 1 -- 172.21.15.88:0/489925384 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd800e8c0 con 0x7fdde01597e0 2024-09-16T01:41:54.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.044+0000 7fdddcff9640 1 -- 172.21.15.88:0/489925384 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddd8018070 con 0x7fdde01597e0 2024-09-16T01:41:54.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.044+0000 7fdde6959640 1 -- 172.21.15.88:0/489925384 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdde015a7f0 con 0x7fdde01597e0 2024-09-16T01:41:54.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.044+0000 7fdddcff9640 1 -- 172.21.15.88:0/489925384 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd800ecc0 con 0x7fdde01597e0 2024-09-16T01:41:54.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.044+0000 7fdde6959640 1 -- 172.21.15.88:0/489925384 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdde015adb0 con 0x7fdde01597e0 2024-09-16T01:41:54.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.045+0000 7fdde6959640 1 -- 172.21.15.88:0/489925384 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdde00cfb30 con 0x7fdde01597e0 2024-09-16T01:41:54.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.045+0000 7fdddcff9640 1 -- 172.21.15.88:0/489925384 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fddd801d460 con 0x7fdde01597e0 2024-09-16T01:41:54.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.046+0000 7fdddcff9640 1 --2- 172.21.15.88:0/489925384 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fddcc06c2e0 0x7fddcc06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:54.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.046+0000 7fdddcff9640 1 -- 172.21.15.88:0/489925384 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7028+0+0 (secure 0 0 0) 0x7fddd808bb10 con 0x7fdde01597e0 2024-09-16T01:41:54.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.046+0000 7fdddffff640 1 --2- 172.21.15.88:0/489925384 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fddcc06c2e0 0x7fddcc06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:54.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.046+0000 7fdddffff640 1 --2- 172.21.15.88:0/489925384 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fddcc06c2e0 0x7fddcc06e7a0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fddd0033000 tx=0x7fddd0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:54.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.050+0000 7fdddcff9640 1 -- 172.21.15.88:0/489925384 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fddd800ee60 con 0x7fdde01597e0 2024-09-16T01:41:54.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.248+0000 7fdde6959640 1 -- 172.21.15.88:0/489925384 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdde0071e60 con 0x7fdde01597e0 2024-09-16T01:41:54.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.249+0000 7fdddcff9640 1 -- 172.21.15.88:0/489925384 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) ==== 95+0+4392 (secure 0 0 0) 0x7fddd8059d30 con 0x7fdde01597e0 2024-09-16T01:41:54.250 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:54.251 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":159,"btime":"2024-09-16T01:41:51:268733+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/3163343083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":3163343083},{"type":"v1","addr":"172.21.15.88:6837","nonce":3163343083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:51.268729+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:41:54.251 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 159 2024-09-16T01:41:54.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.251+0000 7fdde6959640 1 -- 172.21.15.88:0/489925384 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fddcc06c2e0 msgr2=0x7fddcc06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:54.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.251+0000 7fdde6959640 1 --2- 172.21.15.88:0/489925384 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fddcc06c2e0 0x7fddcc06e7a0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fddd0033000 tx=0x7fddd0008030 comp rx=0 tx=0).stop 2024-09-16T01:41:54.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.251+0000 7fdde6959640 1 -- 172.21.15.88:0/489925384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdde01597e0 msgr2=0x7fdde0159c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:54.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.251+0000 7fdde6959640 1 --2- 172.21.15.88:0/489925384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdde01597e0 0x7fdde0159c90 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fdde015ff10 tx=0x7fddd8007a50 comp rx=0 tx=0).stop 2024-09-16T01:41:54.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.251+0000 7fdde6959640 1 -- 172.21.15.88:0/489925384 shutdown_connections 2024-09-16T01:41:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.251+0000 7fdde6959640 1 --2- 172.21.15.88:0/489925384 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fddcc06c2e0 0x7fddcc06e7a0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.251+0000 7fdde6959640 1 --2- 172.21.15.88:0/489925384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdde01597e0 0x7fdde0159c90 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.251+0000 7fdde6959640 1 --2- 172.21.15.88:0/489925384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdde00cadf0 0x7fdde00cb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.251+0000 7fdde6959640 1 --2- 172.21.15.88:0/489925384 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fddc8003a00 0x7fdde014c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.251+0000 7fdde6959640 1 -- 172.21.15.88:0/489925384 >> 172.21.15.88:0/489925384 conn(0x7fdde006bc10 msgr2=0x7fdde015f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.251+0000 7fdde6959640 1 -- 172.21.15.88:0/489925384 shutdown_connections 2024-09-16T01:41:54.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.251+0000 7fdde6959640 1 -- 172.21.15.88:0/489925384 wait complete. 2024-09-16T01:41:54.264 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:41:54.264 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:54.265 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:41:54.264+0000 7f3c3b0e3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:41:54.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.364+0000 7f9aa3391640 1 Processor -- start 2024-09-16T01:41:54.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.364+0000 7f9aa3391640 1 -- start start 2024-09-16T01:41:54.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.365+0000 7f9aa3391640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9a9c15c4c0 0x7f9a9c06c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:54.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.365+0000 7f9aa3391640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9a9c06e200 0x7f9a9c06e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:54.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.365+0000 7f9aa3391640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a9c06eb60 con 0x7f9a9c15ffc0 2024-09-16T01:41:54.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.365+0000 7f9aa3391640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f9a9c0cfc50 con 0x7f9a9c15c4c0 2024-09-16T01:41:54.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.365+0000 7f9aa3391640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9a9c15c8a0 con 0x7f9a9c06e200 2024-09-16T01:41:54.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.365+0000 7f9aa112d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9a9c15c4c0 0x7f9a9c06c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:54.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.366+0000 7f9aa112d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9a9c15c4c0 0x7f9a9c06c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53796/0 (socket says 172.21.15.88:53796) 2024-09-16T01:41:54.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.366+0000 7f9aa112d640 1 -- 172.21.15.88:0/3473939333 learned_addr learned my addr 172.21.15.88:0/3473939333 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:54.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.366+0000 7f9aa092c640 1 --2- 172.21.15.88:0/3473939333 >> v2:172.21.15.154:3300/0 conn(0x7f9a9c06e200 0x7f9a9c06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:54.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.366+0000 7f9aa112d640 1 -- 172.21.15.88:0/3473939333 >> v2:172.21.15.154:3300/0 conn(0x7f9a9c06e200 msgr2=0x7f9a9c06e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:54.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.366+0000 7f9aa112d640 1 --2- 172.21.15.88:0/3473939333 >> v2:172.21.15.154:3300/0 conn(0x7f9a9c06e200 0x7f9a9c06e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:54.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.366+0000 7f9aa112d640 1 -- 172.21.15.88:0/3473939333 >> v1:172.21.15.154:6789/0 conn(0x7f9a9c15ffc0 legacy=0x7f9a9c1603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:54.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.366+0000 7f9aa112d640 1 -- 172.21.15.88:0/3473939333 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a9c06cf90 con 0x7f9a9c15c4c0 2024-09-16T01:41:54.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.367+0000 7f9aa112d640 1 --2- 172.21.15.88:0/3473939333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a9c15c4c0 0x7f9a9c06c620 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c15ff20 tx=0x7f9a8c02a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67acb86c5a4e45ce server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:54.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.367+0000 7f9a9bfff640 1 -- 172.21.15.88:0/3473939333 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c036070 con 0x7f9a9c15c4c0 2024-09-16T01:41:54.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.367+0000 7f9a9bfff640 1 -- 172.21.15.88:0/3473939333 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a8c031600 con 0x7f9a9c15c4c0 2024-09-16T01:41:54.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.367+0000 7f9a9bfff640 1 -- 172.21.15.88:0/3473939333 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c0319d0 con 0x7f9a9c15c4c0 2024-09-16T01:41:54.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.367+0000 7f9aa3391640 1 -- 172.21.15.88:0/3473939333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a9c15c4c0 msgr2=0x7f9a9c06c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:54.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.367+0000 7f9aa3391640 1 --2- 172.21.15.88:0/3473939333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a9c15c4c0 0x7f9a9c06c620 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c15ff20 tx=0x7f9a8c02a730 comp rx=0 tx=0).stop 2024-09-16T01:41:54.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.367+0000 7f9aa3391640 1 -- 172.21.15.88:0/3473939333 shutdown_connections 2024-09-16T01:41:54.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.367+0000 7f9aa3391640 1 --2- 172.21.15.88:0/3473939333 >> v2:172.21.15.154:3300/0 conn(0x7f9a9c06e200 0x7f9a9c06e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:54.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.367+0000 7f9aa3391640 1 --2- 172.21.15.88:0/3473939333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a9c15c4c0 0x7f9a9c06c620 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:54.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.367+0000 7f9aa3391640 1 -- 172.21.15.88:0/3473939333 >> 172.21.15.88:0/3473939333 conn(0x7f9a9c0b6ec0 msgr2=0x7f9a9c0b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:54.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.367+0000 7f9aa3391640 1 -- 172.21.15.88:0/3473939333 shutdown_connections 2024-09-16T01:41:54.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.367+0000 7f9aa3391640 1 -- 172.21.15.88:0/3473939333 wait complete. 2024-09-16T01:41:54.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.368+0000 7f9aa3391640 1 Processor -- start 2024-09-16T01:41:54.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.369+0000 7f9aa3391640 1 -- start start 2024-09-16T01:41:54.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.369+0000 7f9aa3391640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a9c06e200 0x7f9a9c151790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:54.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.369+0000 7f9aa3391640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9a9c15c4c0 0x7f9a9c1f1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:54.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.369+0000 7f9aa3391640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a9c15ffc0 0x7f9a9c142c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:54.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.369+0000 7f9aa3391640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c0cf240 con 0x7f9a9c06e200 2024-09-16T01:41:54.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.369+0000 7f9aa3391640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c0cf8e0 con 0x7f9a9c15ffc0 2024-09-16T01:41:54.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.369+0000 7f9aa3391640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9a9c0cf3c0 con 0x7f9a9c15c4c0 2024-09-16T01:41:54.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.369+0000 7f9aa112d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a9c06e200 0x7f9a9c151790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:54.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.369+0000 7f9aa112d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a9c06e200 0x7f9a9c151790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53802/0 (socket says 172.21.15.88:53802) 2024-09-16T01:41:54.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.369+0000 7f9aa112d640 1 -- 172.21.15.88:0/4096923499 learned_addr learned my addr 172.21.15.88:0/4096923499 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:54.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.370+0000 7f9aa092c640 1 --2- 172.21.15.88:0/4096923499 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9a9c15c4c0 0x7f9a9c1f1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:54.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.370+0000 7f9aa192e640 1 --2- 172.21.15.88:0/4096923499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a9c15ffc0 0x7f9a9c142c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:54.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.370+0000 7f9aa112d640 1 -- 172.21.15.88:0/4096923499 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9a9c15c4c0 msgr2=0x7f9a9c1f1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:54.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.370+0000 7f9aa112d640 1 --2- 172.21.15.88:0/4096923499 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9a9c15c4c0 0x7f9a9c1f1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:54.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.370+0000 7f9aa112d640 1 -- 172.21.15.88:0/4096923499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a9c15ffc0 msgr2=0x7f9a9c142c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:54.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.370+0000 7f9aa112d640 1 --2- 172.21.15.88:0/4096923499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a9c15ffc0 0x7f9a9c142c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:54.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.370+0000 7f9aa112d640 1 -- 172.21.15.88:0/4096923499 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a9c152140 con 0x7f9a9c06e200 2024-09-16T01:41:54.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.370+0000 7f9aa112d640 1 --2- 172.21.15.88:0/4096923499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a9c06e200 0x7f9a9c151790 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c034040 tx=0x7f9a8c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:54.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.370+0000 7f9a99ffb640 1 -- 172.21.15.88:0/4096923499 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c036050 con 0x7f9a9c06e200 2024-09-16T01:41:54.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.370+0000 7f9a99ffb640 1 -- 172.21.15.88:0/4096923499 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a8c042070 con 0x7f9a9c06e200 2024-09-16T01:41:54.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.370+0000 7f9aa3391640 1 -- 172.21.15.88:0/4096923499 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a9c152420 con 0x7f9a9c06e200 2024-09-16T01:41:54.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.370+0000 7f9a99ffb640 1 -- 172.21.15.88:0/4096923499 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c009510 con 0x7f9a9c06e200 2024-09-16T01:41:54.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.370+0000 7f9aa3391640 1 -- 172.21.15.88:0/4096923499 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a9c155b90 con 0x7f9a9c06e200 2024-09-16T01:41:54.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.371+0000 7f9aa3391640 1 -- 172.21.15.88:0/4096923499 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a9c06da10 con 0x7f9a9c06e200 2024-09-16T01:41:54.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.372+0000 7f9a99ffb640 1 -- 172.21.15.88:0/4096923499 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9a8c007c50 con 0x7f9a9c06e200 2024-09-16T01:41:54.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.372+0000 7f9a99ffb640 1 --2- 172.21.15.88:0/4096923499 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9a8806c2b0 0x7f9a8806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:54.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.372+0000 7f9a99ffb640 1 -- 172.21.15.88:0/4096923499 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7028+0+0 (secure 0 0 0) 0x7f9a8c0ac5e0 con 0x7f9a9c06e200 2024-09-16T01:41:54.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.372+0000 7f9aa092c640 1 --2- 172.21.15.88:0/4096923499 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9a8806c2b0 0x7f9a8806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:54.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.373+0000 7f9aa092c640 1 --2- 172.21.15.88:0/4096923499 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9a8806c2b0 0x7f9a8806e770 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c157340 tx=0x7f9a90009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:54.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.377+0000 7f9a99ffb640 1 -- 172.21.15.88:0/4096923499 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9a8c07a1d0 con 0x7f9a9c06e200 2024-09-16T01:41:54.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.575+0000 7f9aa3391640 1 -- 172.21.15.88:0/4096923499 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a9c0b32a0 con 0x7f9a9c06e200 2024-09-16T01:41:54.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.575+0000 7f9a99ffb640 1 -- 172.21.15.88:0/4096923499 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) ==== 95+0+4392 (secure 0 0 0) 0x7f9a8c07e4d0 con 0x7f9a9c06e200 2024-09-16T01:41:54.577 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:54.577 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":159,"btime":"2024-09-16T01:41:51:268733+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/3163343083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":3163343083},{"type":"v1","addr":"172.21.15.88:6837","nonce":3163343083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:51.268729+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:41:54.577 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 159 2024-09-16T01:41:54.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.577+0000 7f9aa3391640 1 -- 172.21.15.88:0/4096923499 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9a8806c2b0 msgr2=0x7f9a8806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:54.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.577+0000 7f9aa3391640 1 --2- 172.21.15.88:0/4096923499 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9a8806c2b0 0x7f9a8806e770 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c157340 tx=0x7f9a90009190 comp rx=0 tx=0).stop 2024-09-16T01:41:54.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.577+0000 7f9aa3391640 1 -- 172.21.15.88:0/4096923499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a9c06e200 msgr2=0x7f9a9c151790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:54.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.577+0000 7f9aa3391640 1 --2- 172.21.15.88:0/4096923499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a9c06e200 0x7f9a9c151790 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c034040 tx=0x7f9a8c004a40 comp rx=0 tx=0).stop 2024-09-16T01:41:54.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.577+0000 7f9aa3391640 1 -- 172.21.15.88:0/4096923499 shutdown_connections 2024-09-16T01:41:54.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.577+0000 7f9aa3391640 1 --2- 172.21.15.88:0/4096923499 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9a8806c2b0 0x7f9a8806e770 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:54.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.577+0000 7f9aa3391640 1 --2- 172.21.15.88:0/4096923499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a9c15ffc0 0x7f9a9c142c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:54.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.577+0000 7f9aa3391640 1 --2- 172.21.15.88:0/4096923499 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9a9c15c4c0 0x7f9a9c1f1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:54.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.577+0000 7f9aa3391640 1 --2- 172.21.15.88:0/4096923499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9a9c06e200 0x7f9a9c151790 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:54.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.577+0000 7f9aa3391640 1 -- 172.21.15.88:0/4096923499 >> 172.21.15.88:0/4096923499 conn(0x7f9a9c0b6ec0 msgr2=0x7f9a9c0b1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:54.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.577+0000 7f9aa3391640 1 -- 172.21.15.88:0/4096923499 shutdown_connections 2024-09-16T01:41:54.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.577+0000 7f9aa3391640 1 -- 172.21.15.88:0/4096923499 wait complete. 2024-09-16T01:41:54.589 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 1.00000000' led to DAMAGED state 2024-09-16T01:41:54.590 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt 1.00000000.inode' 2024-09-16T01:41:54.590 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:41:54.590 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:41:54.605 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:41:54.606 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:41:54.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:54.606 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:41:54.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:41:54.662 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:41:54.662 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:41:54.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.799+0000 7f73b37f6640 1 Processor -- start 2024-09-16T01:41:54.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.799+0000 7f73b37f6640 1 -- start start 2024-09-16T01:41:54.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.800+0000 7f73b37f6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f73ac0b68f0 0x7f73ac1696f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:54.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.800+0000 7f73b37f6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73ac0cf860 con 0x7f73ac15e7a0 2024-09-16T01:41:54.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.800+0000 7f73b37f6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73ac0cfb40 con 0x7f73ac0cf1f0 2024-09-16T01:41:54.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.800+0000 7f73b37f6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f73ac0d2ca0 con 0x7f73ac0b68f0 2024-09-16T01:41:54.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.800+0000 7f73b1d93640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f73ac0b68f0 0x7f73ac1696f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:54.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.800+0000 7f73b1592640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f73ac15e7a0 0x7f73ac06caa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:56872/0 (socket says 172.21.15.88:56872) 2024-09-16T01:41:54.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.800+0000 7f73b1592640 1 -- 172.21.15.88:0/4110877298 learned_addr learned my addr 172.21.15.88:0/4110877298 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:54.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.801+0000 7f73b1d93640 1 -- 172.21.15.88:0/4110877298 >> v1:172.21.15.154:6789/0 conn(0x7f73ac0cf1f0 legacy=0x7f73ac0b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:54.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.801+0000 7f73b1d93640 1 -- 172.21.15.88:0/4110877298 >> v1:172.21.15.88:6789/0 conn(0x7f73ac15e7a0 legacy=0x7f73ac06caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:54.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.801+0000 7f73abfff640 1 -- 172.21.15.88:0/4110877298 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3735235482 0 0) 0x7f73ac0cf860 con 0x7f73ac15e7a0 2024-09-16T01:41:54.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.801+0000 7f73b1d93640 1 -- 172.21.15.88:0/4110877298 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73ac0cfe20 con 0x7f73ac0b68f0 2024-09-16T01:41:54.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.801+0000 7f73b1d93640 1 --2- 172.21.15.88:0/4110877298 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73ac0b68f0 0x7f73ac1696f0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f73a4009d90 tx=0x7f73a402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5516ade3338b0c24 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:54.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.801+0000 7f73abfff640 1 -- 172.21.15.88:0/4110877298 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a4031600 con 0x7f73ac0b68f0 2024-09-16T01:41:54.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.802+0000 7f73abfff640 1 -- 172.21.15.88:0/4110877298 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73a40317a0 con 0x7f73ac0b68f0 2024-09-16T01:41:54.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.802+0000 7f73abfff640 1 -- 172.21.15.88:0/4110877298 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a4031b70 con 0x7f73ac0b68f0 2024-09-16T01:41:54.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.802+0000 7f73b37f6640 1 -- 172.21.15.88:0/4110877298 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73ac0b68f0 msgr2=0x7f73ac1696f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:54.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.802+0000 7f73b37f6640 1 --2- 172.21.15.88:0/4110877298 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73ac0b68f0 0x7f73ac1696f0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f73a4009d90 tx=0x7f73a402a610 comp rx=0 tx=0).stop 2024-09-16T01:41:54.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.802+0000 7f73b37f6640 1 -- 172.21.15.88:0/4110877298 shutdown_connections 2024-09-16T01:41:54.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.802+0000 7f73b37f6640 1 --2- 172.21.15.88:0/4110877298 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73ac0b68f0 0x7f73ac1696f0 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:54.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.802+0000 7f73b37f6640 1 -- 172.21.15.88:0/4110877298 >> 172.21.15.88:0/4110877298 conn(0x7f73ac0d11e0 msgr2=0x7f73ac0d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:54.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.802+0000 7f73b37f6640 1 -- 172.21.15.88:0/4110877298 shutdown_connections 2024-09-16T01:41:54.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.802+0000 7f73b37f6640 1 -- 172.21.15.88:0/4110877298 wait complete. 2024-09-16T01:41:54.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.803+0000 7f73b37f6640 1 Processor -- start 2024-09-16T01:41:54.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.803+0000 7f73b37f6640 1 -- start start 2024-09-16T01:41:54.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.804+0000 7f73b37f6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73ac0b68f0 0x7f73ac160f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:54.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.804+0000 7f73b37f6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73ac0cf1f0 0x7f73ac151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:54.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.804+0000 7f73b37f6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73ac15e7a0 0x7f73ac15f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:54.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.804+0000 7f73b37f6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac0d4420 con 0x7f73ac0cf1f0 2024-09-16T01:41:54.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.804+0000 7f73b37f6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f73ac0b7a00 con 0x7f73ac0b68f0 2024-09-16T01:41:54.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.804+0000 7f73b37f6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f73ac15fe80 con 0x7f73ac15e7a0 2024-09-16T01:41:54.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.804+0000 7f73b0d91640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73ac0cf1f0 0x7f73ac151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:54.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.804+0000 7f73b0d91640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73ac0cf1f0 0x7f73ac151cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53806/0 (socket says 172.21.15.88:53806) 2024-09-16T01:41:54.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.804+0000 7f73b0d91640 1 -- 172.21.15.88:0/2153187085 learned_addr learned my addr 172.21.15.88:0/2153187085 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:54.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.804+0000 7f73b1592640 1 --2- 172.21.15.88:0/2153187085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73ac0b68f0 0x7f73ac160f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:54.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.804+0000 7f73b1d93640 1 --2- 172.21.15.88:0/2153187085 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73ac15e7a0 0x7f73ac15f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:54.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.804+0000 7f73b0d91640 1 -- 172.21.15.88:0/2153187085 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73ac15e7a0 msgr2=0x7f73ac15f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:54.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.804+0000 7f73b0d91640 1 --2- 172.21.15.88:0/2153187085 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73ac15e7a0 0x7f73ac15f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:54.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.804+0000 7f73b0d91640 1 -- 172.21.15.88:0/2153187085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73ac0b68f0 msgr2=0x7f73ac160f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:54.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.804+0000 7f73b0d91640 1 --2- 172.21.15.88:0/2153187085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73ac0b68f0 0x7f73ac160f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:54.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.804+0000 7f73b0d91640 1 -- 172.21.15.88:0/2153187085 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73ac160160 con 0x7f73ac0cf1f0 2024-09-16T01:41:54.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.804+0000 7f73b1592640 1 --2- 172.21.15.88:0/2153187085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73ac0b68f0 0x7f73ac160f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:54.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.805+0000 7f73b0d91640 1 --2- 172.21.15.88:0/2153187085 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73ac0cf1f0 0x7f73ac151cb0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f73a000c430 tx=0x7f73a000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:54.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.805+0000 7f73a9ffb640 1 -- 172.21.15.88:0/2153187085 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a0011790 con 0x7f73ac0cf1f0 2024-09-16T01:41:54.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.805+0000 7f73a9ffb640 1 -- 172.21.15.88:0/2153187085 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73a0017070 con 0x7f73ac0cf1f0 2024-09-16T01:41:54.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.805+0000 7f73b37f6640 1 -- 172.21.15.88:0/2153187085 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73ac160440 con 0x7f73ac0cf1f0 2024-09-16T01:41:54.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.805+0000 7f73a9ffb640 1 -- 172.21.15.88:0/2153187085 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a0011bb0 con 0x7f73ac0cf1f0 2024-09-16T01:41:54.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.805+0000 7f73b37f6640 1 -- 172.21.15.88:0/2153187085 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73ac160a00 con 0x7f73ac0cf1f0 2024-09-16T01:41:54.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.806+0000 7f73b37f6640 1 -- 172.21.15.88:0/2153187085 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73ac160c00 con 0x7f73ac0cf1f0 2024-09-16T01:41:54.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.806+0000 7f73a9ffb640 1 -- 172.21.15.88:0/2153187085 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f73a001c460 con 0x7f73ac0cf1f0 2024-09-16T01:41:54.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.807+0000 7f73a9ffb640 1 --2- 172.21.15.88:0/2153187085 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f739806c2b0 0x7f739806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:54.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.807+0000 7f73a9ffb640 1 -- 172.21.15.88:0/2153187085 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7028+0+0 (secure 0 0 0) 0x7f73a008b850 con 0x7f73ac0cf1f0 2024-09-16T01:41:54.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.807+0000 7f73b1592640 1 --2- 172.21.15.88:0/2153187085 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f739806c2b0 0x7f739806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:54.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.807+0000 7f73b1592640 1 --2- 172.21.15.88:0/2153187085 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f739806c2b0 0x7f739806e770 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f739c000f60 tx=0x7f739c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:54.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:54.811+0000 7f73a9ffb640 1 -- 172.21.15.88:0/2153187085 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f73ac160c00 con 0x7f73ac0cf1f0 2024-09-16T01:41:55.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.013+0000 7f73b37f6640 1 -- 172.21.15.88:0/2153187085 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f73ac15eb80 con 0x7f73ac0cf1f0 2024-09-16T01:41:55.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.327+0000 7f73a9ffb640 1 -- 172.21.15.88:0/2153187085 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v160) ==== 161+0+0 (secure 0 0 0) 0x7f73a0059a90 con 0x7f73ac0cf1f0 2024-09-16T01:41:55.329 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:41:55.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.331+0000 7f73b37f6640 1 -- 172.21.15.88:0/2153187085 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f739806c2b0 msgr2=0x7f739806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:55.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.331+0000 7f73b37f6640 1 --2- 172.21.15.88:0/2153187085 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f739806c2b0 0x7f739806e770 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f739c000f60 tx=0x7f739c009040 comp rx=0 tx=0).stop 2024-09-16T01:41:55.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.331+0000 7f73b37f6640 1 -- 172.21.15.88:0/2153187085 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73ac0cf1f0 msgr2=0x7f73ac151cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:55.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.331+0000 7f73b37f6640 1 --2- 172.21.15.88:0/2153187085 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73ac0cf1f0 0x7f73ac151cb0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f73a000c430 tx=0x7f73a000c8f0 comp rx=0 tx=0).stop 2024-09-16T01:41:55.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.331+0000 7f73b37f6640 1 -- 172.21.15.88:0/2153187085 shutdown_connections 2024-09-16T01:41:55.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.331+0000 7f73b37f6640 1 --2- 172.21.15.88:0/2153187085 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f739806c2b0 0x7f739806e770 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:55.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.331+0000 7f73b37f6640 1 --2- 172.21.15.88:0/2153187085 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73ac15e7a0 0x7f73ac15f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:55.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.331+0000 7f73b37f6640 1 --2- 172.21.15.88:0/2153187085 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73ac0cf1f0 0x7f73ac151cb0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:55.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.331+0000 7f73b37f6640 1 --2- 172.21.15.88:0/2153187085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73ac0b68f0 0x7f73ac160f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:55.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.331+0000 7f73b37f6640 1 -- 172.21.15.88:0/2153187085 >> 172.21.15.88:0/2153187085 conn(0x7f73ac0d11e0 msgr2=0x7f73ac1588a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:55.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.331+0000 7f73b37f6640 1 -- 172.21.15.88:0/2153187085 shutdown_connections 2024-09-16T01:41:55.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.331+0000 7f73b37f6640 1 -- 172.21.15.88:0/2153187085 wait complete. 2024-09-16T01:41:55.344 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:41:55.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.442+0000 7f2cb6718640 1 Processor -- start 2024-09-16T01:41:55.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.442+0000 7f2cb6718640 1 -- start start 2024-09-16T01:41:55.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.443+0000 7f2cb6718640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2cb00720d0 0x7f2cb00b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:55.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.443+0000 7f2cb6718640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cb0071720 con 0x7f2cb015ad10 2024-09-16T01:41:55.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.443+0000 7f2cb6718640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cb0073920 con 0x7f2cb00cca40 2024-09-16T01:41:55.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.443+0000 7f2cb6718640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2cb0070fb0 con 0x7f2cb00720d0 2024-09-16T01:41:55.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.443+0000 7f2caffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f2cb015ad10 0x7f2cb00719c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:56884/0 (socket says 172.21.15.88:56884) 2024-09-16T01:41:55.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.443+0000 7f2caffff640 1 -- 172.21.15.88:0/3011898763 learned_addr learned my addr 172.21.15.88:0/3011898763 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:55.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.443+0000 7f2caf7fe640 1 --2- 172.21.15.88:0/3011898763 >> v2:172.21.15.154:3300/0 conn(0x7f2cb00720d0 0x7f2cb00b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:55.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.443+0000 7f2caeffd640 1 -- 172.21.15.88:0/3011898763 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2227347799 0 0) 0x7f2cb0071720 con 0x7f2cb015ad10 2024-09-16T01:41:55.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.444+0000 7f2caeffd640 1 -- 172.21.15.88:0/3011898763 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c98003520 con 0x7f2cb015ad10 2024-09-16T01:41:55.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.444+0000 7f2caeffd640 1 -- 172.21.15.88:0/3011898763 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 55949378 0 0) 0x7f2c98003520 con 0x7f2cb015ad10 2024-09-16T01:41:55.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.444+0000 7f2caeffd640 1 -- 172.21.15.88:0/3011898763 >> v2:172.21.15.154:3300/0 conn(0x7f2cb00720d0 msgr2=0x7f2cb00b3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:55.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.444+0000 7f2caeffd640 1 --2- 172.21.15.88:0/3011898763 >> v2:172.21.15.154:3300/0 conn(0x7f2cb00720d0 0x7f2cb00b3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:55.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.444+0000 7f2caeffd640 1 -- 172.21.15.88:0/3011898763 >> v1:172.21.15.154:6789/0 conn(0x7f2cb00cca40 legacy=0x7f2cb00b43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:55.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.444+0000 7f2caeffd640 1 -- 172.21.15.88:0/3011898763 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cb0073c00 con 0x7f2cb015ad10 2024-09-16T01:41:55.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.444+0000 7f2caeffd640 1 -- 172.21.15.88:0/3011898763 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f2ca0001670 con 0x7f2cb015ad10 2024-09-16T01:41:55.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.444+0000 7f2caeffd640 1 -- 172.21.15.88:0/3011898763 >> v1:172.21.15.88:6789/0 conn(0x7f2cb015ad10 legacy=0x7f2cb00719c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:55.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.444+0000 7f2caeffd640 1 --2- 172.21.15.88:0/3011898763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c980039b0 0x7f2c98003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:55.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.444+0000 7f2caeffd640 1 --2- 172.21.15.88:0/3011898763 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c98004800 0x7f2c98004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:55.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.444+0000 7f2caeffd640 1 --2- 172.21.15.88:0/3011898763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c980051c0 0x7f2c9800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:55.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.444+0000 7f2caeffd640 1 -- 172.21.15.88:0/3011898763 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb0070fb0 con 0x7f2c980051c0 2024-09-16T01:41:55.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.444+0000 7f2caeffd640 1 -- 172.21.15.88:0/3011898763 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9800c160 con 0x7f2c980039b0 2024-09-16T01:41:55.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.444+0000 7f2caeffd640 1 -- 172.21.15.88:0/3011898763 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2c9800c360 con 0x7f2c98004800 2024-09-16T01:41:55.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.444+0000 7f2cb4cb5640 1 --2- 172.21.15.88:0/3011898763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c980051c0 0x7f2c9800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:55.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.444+0000 7f2caffff640 1 --2- 172.21.15.88:0/3011898763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c980039b0 0x7f2c98003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:55.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.444+0000 7f2caf7fe640 1 --2- 172.21.15.88:0/3011898763 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c98004800 0x7f2c98004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:55.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.444+0000 7f2cb4cb5640 1 -- 172.21.15.88:0/3011898763 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c98004800 msgr2=0x7f2c98004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:55.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.444+0000 7f2cb4cb5640 1 --2- 172.21.15.88:0/3011898763 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c98004800 0x7f2c98004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:55.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.444+0000 7f2cb4cb5640 1 -- 172.21.15.88:0/3011898763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c980039b0 msgr2=0x7f2c98003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:55.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.444+0000 7f2cb4cb5640 1 --2- 172.21.15.88:0/3011898763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c980039b0 0x7f2c98003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:55.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.444+0000 7f2cb4cb5640 1 -- 172.21.15.88:0/3011898763 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c98003520 con 0x7f2c980051c0 2024-09-16T01:41:55.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.445+0000 7f2cb4cb5640 1 -- 172.21.15.88:0/3011898763 reap_dead start 2024-09-16T01:41:55.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.445+0000 7f2cb4cb5640 1 --2- 172.21.15.88:0/3011898763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c980051c0 0x7f2c9800b9c0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8001e50 tx=0x7f2c98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb04217bac56eede server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:55.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.445+0000 7f2caeffd640 1 -- 172.21.15.88:0/3011898763 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca8001a70 con 0x7f2c980051c0 2024-09-16T01:41:55.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.445+0000 7f2caeffd640 1 -- 172.21.15.88:0/3011898763 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ca8003970 con 0x7f2c980051c0 2024-09-16T01:41:55.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.445+0000 7f2caeffd640 1 -- 172.21.15.88:0/3011898763 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca8003d60 con 0x7f2c980051c0 2024-09-16T01:41:55.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.446+0000 7f2cb6718640 1 -- 172.21.15.88:0/3011898763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c980051c0 msgr2=0x7f2c9800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:55.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.446+0000 7f2cb6718640 1 --2- 172.21.15.88:0/3011898763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c980051c0 0x7f2c9800b9c0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8001e50 tx=0x7f2c98005830 comp rx=0 tx=0).stop 2024-09-16T01:41:55.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.446+0000 7f2cb6718640 1 -- 172.21.15.88:0/3011898763 shutdown_connections 2024-09-16T01:41:55.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.446+0000 7f2cb6718640 1 --2- 172.21.15.88:0/3011898763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c980051c0 0x7f2c9800b9c0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:55.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.446+0000 7f2cb6718640 1 -- 172.21.15.88:0/3011898763 >> 172.21.15.88:0/3011898763 conn(0x7f2cb006d0f0 msgr2=0x7f2cb006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:55.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.446+0000 7f2cb6718640 1 -- 172.21.15.88:0/3011898763 shutdown_connections 2024-09-16T01:41:55.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.446+0000 7f2cb6718640 1 -- 172.21.15.88:0/3011898763 wait complete. 2024-09-16T01:41:55.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.447+0000 7f2cb6718640 1 Processor -- start 2024-09-16T01:41:55.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.447+0000 7f2cb6718640 1 -- start start 2024-09-16T01:41:55.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.448+0000 7f2cb6718640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c980051c0 0x7f2cb0152e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:55.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.448+0000 7f2cb6718640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cb0154700 0x7f2cb00b4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:55.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.448+0000 7f2cb6718640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cb0155770 0x7f2cb0155bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:55.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.448+0000 7f2cb6718640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb0071200 con 0x7f2c980051c0 2024-09-16T01:41:55.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.448+0000 7f2cb6718640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb00b4fb0 con 0x7f2cb0155770 2024-09-16T01:41:55.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.448+0000 7f2cb6718640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2cb00cc360 con 0x7f2cb0154700 2024-09-16T01:41:55.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.448+0000 7f2caffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c980051c0 0x7f2cb0152e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:55.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.448+0000 7f2caffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c980051c0 0x7f2cb0152e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53820/0 (socket says 172.21.15.88:53820) 2024-09-16T01:41:55.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.448+0000 7f2caffff640 1 -- 172.21.15.88:0/1410816251 learned_addr learned my addr 172.21.15.88:0/1410816251 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:55.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.448+0000 7f2cb4cb5640 1 --2- 172.21.15.88:0/1410816251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cb0155770 0x7f2cb0155bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:55.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.448+0000 7f2caf7fe640 1 --2- 172.21.15.88:0/1410816251 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cb0154700 0x7f2cb00b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:55.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.448+0000 7f2caffff640 1 -- 172.21.15.88:0/1410816251 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cb0154700 msgr2=0x7f2cb00b4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:55.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.448+0000 7f2caffff640 1 --2- 172.21.15.88:0/1410816251 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cb0154700 0x7f2cb00b4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:55.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.448+0000 7f2caffff640 1 -- 172.21.15.88:0/1410816251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cb0155770 msgr2=0x7f2cb0155bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:55.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.448+0000 7f2caffff640 1 --2- 172.21.15.88:0/1410816251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cb0155770 0x7f2cb0155bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:55.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.448+0000 7f2caffff640 1 -- 172.21.15.88:0/1410816251 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cb01f4370 con 0x7f2c980051c0 2024-09-16T01:41:55.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.448+0000 7f2caffff640 1 --2- 172.21.15.88:0/1410816251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c980051c0 0x7f2cb0152e20 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0001050 tx=0x7f2ca0007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:55.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.448+0000 7f2cacff9640 1 -- 172.21.15.88:0/1410816251 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca000f960 con 0x7f2c980051c0 2024-09-16T01:41:55.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.448+0000 7f2cacff9640 1 -- 172.21.15.88:0/1410816251 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ca0018070 con 0x7f2c980051c0 2024-09-16T01:41:55.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.448+0000 7f2cacff9640 1 -- 172.21.15.88:0/1410816251 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca000fd80 con 0x7f2c980051c0 2024-09-16T01:41:55.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.448+0000 7f2cb6718640 1 -- 172.21.15.88:0/1410816251 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cb01f4650 con 0x7f2c980051c0 2024-09-16T01:41:55.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.449+0000 7f2cb6718640 1 -- 172.21.15.88:0/1410816251 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cb01f4c10 con 0x7f2c980051c0 2024-09-16T01:41:55.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.450+0000 7f2cb6718640 1 -- 172.21.15.88:0/1410816251 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cb01575d0 con 0x7f2c980051c0 2024-09-16T01:41:55.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.450+0000 7f2cacff9640 1 -- 172.21.15.88:0/1410816251 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2ca001d460 con 0x7f2c980051c0 2024-09-16T01:41:55.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.451+0000 7f2cacff9640 1 --2- 172.21.15.88:0/1410816251 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2c9c06c2b0 0x7f2c9c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:55.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.451+0000 7f2cacff9640 1 -- 172.21.15.88:0/1410816251 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7028+0+0 (secure 0 0 0) 0x7f2ca008ba80 con 0x7f2c980051c0 2024-09-16T01:41:55.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.451+0000 7f2caf7fe640 1 --2- 172.21.15.88:0/1410816251 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2c9c06c2b0 0x7f2c9c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:55.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.451+0000 7f2caf7fe640 1 --2- 172.21.15.88:0/1410816251 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2c9c06c2b0 0x7f2c9c06e770 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0158750 tx=0x7f2ca4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:55.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.455+0000 7f2cacff9640 1 -- 172.21.15.88:0/1410816251 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2ca0059cc0 con 0x7f2c980051c0 2024-09-16T01:41:55.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:55.690+0000 7f2cb6718640 1 -- 172.21.15.88:0/1410816251 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f2cb00cc5d0 con 0x7f2c980051c0 2024-09-16T01:41:56.019 INFO:tasks.ceph.mds.a.smithi088.stderr: -47> 2024-09-16T01:41:51.266+0000 7f3c370db640 -1 log_channel(cluster) log [ERR] : dir 0x1 object missing on disk; some files may be lost () 2024-09-16T01:41:56.019 INFO:tasks.ceph.mds.a.smithi088.stderr: -45> 2024-09-16T01:41:51.266+0000 7f3c370db640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x1 is fatal because it is a system directory for this rank 2024-09-16T01:41:56.020 INFO:tasks.ceph.mds.a.smithi088.stderr: -17> 2024-09-16T01:41:54.264+0000 7f3c3b0e3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:41:56.043 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:41:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.333+0000 7f2cacff9640 1 -- 172.21.15.88:0/1410816251 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v161) ==== 101+0+0 (secure 0 0 0) 0x7f2ca00595c0 con 0x7f2c980051c0 2024-09-16T01:41:56.335 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:41:56.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.335+0000 7f2cb6718640 1 -- 172.21.15.88:0/1410816251 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2c9c06c2b0 msgr2=0x7f2c9c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:56.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.335+0000 7f2cb6718640 1 --2- 172.21.15.88:0/1410816251 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2c9c06c2b0 0x7f2c9c06e770 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0158750 tx=0x7f2ca4008040 comp rx=0 tx=0).stop 2024-09-16T01:41:56.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.335+0000 7f2cb6718640 1 -- 172.21.15.88:0/1410816251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c980051c0 msgr2=0x7f2cb0152e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:56.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.335+0000 7f2cb6718640 1 --2- 172.21.15.88:0/1410816251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c980051c0 0x7f2cb0152e20 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0001050 tx=0x7f2ca0007720 comp rx=0 tx=0).stop 2024-09-16T01:41:56.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.335+0000 7f2cb6718640 1 -- 172.21.15.88:0/1410816251 shutdown_connections 2024-09-16T01:41:56.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.335+0000 7f2cb6718640 1 --2- 172.21.15.88:0/1410816251 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2c9c06c2b0 0x7f2c9c06e770 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.335+0000 7f2cb6718640 1 --2- 172.21.15.88:0/1410816251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cb0155770 0x7f2cb0155bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.335+0000 7f2cb6718640 1 --2- 172.21.15.88:0/1410816251 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cb0154700 0x7f2cb00b4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.335+0000 7f2cb6718640 1 --2- 172.21.15.88:0/1410816251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c980051c0 0x7f2cb0152e20 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.335+0000 7f2cb6718640 1 -- 172.21.15.88:0/1410816251 >> 172.21.15.88:0/1410816251 conn(0x7f2cb006d0f0 msgr2=0x7f2cb015b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:56.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.336+0000 7f2cb6718640 1 -- 172.21.15.88:0/1410816251 shutdown_connections 2024-09-16T01:41:56.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.336+0000 7f2cb6718640 1 -- 172.21.15.88:0/1410816251 wait complete. 2024-09-16T01:41:56.349 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:41:56.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.391+0000 7fd4e47cef40 1 Processor -- start 2024-09-16T01:41:56.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.391+0000 7fd4e47cef40 1 -- start start 2024-09-16T01:41:56.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.391+0000 7fd4e47cef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ff32775f0 0x563ff3293df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.391+0000 7fd4e47cef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563ff3277be0 0x563ff3294330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.391+0000 7fd4e47cef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ff3298bf0 0x563ff3295e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.391+0000 7fd4e47cef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x563ff3201b30 con 0x563ff3298bf0 2024-09-16T01:41:56.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.391+0000 7fd4e47cef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563ff3113510 con 0x563ff32775f0 2024-09-16T01:41:56.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.391+0000 7fd4e47cef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x563ff329ac50 con 0x563ff3277be0 2024-09-16T01:41:56.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.391+0000 7fd4e2ac7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ff3298bf0 0x563ff3295e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.391+0000 7fd4e2ac7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ff3298bf0 0x563ff3295e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53850/0 (socket says 172.21.15.88:53850) 2024-09-16T01:41:56.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.391+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/754719520 learned_addr learned my addr 172.21.15.88:0/754719520 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:56.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.391+0000 7fd4dbfff640 1 --2- 172.21.15.88:0/754719520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563ff3277be0 0x563ff3294330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.391+0000 7fd4e22c6640 1 --2- 172.21.15.88:0/754719520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ff32775f0 0x563ff3293df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.392+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/754719520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563ff3277be0 msgr2=0x563ff3294330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:56.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.392+0000 7fd4e2ac7640 1 --2- 172.21.15.88:0/754719520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563ff3277be0 0x563ff3294330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.392+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/754719520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ff32775f0 msgr2=0x563ff3293df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:56.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.392+0000 7fd4e2ac7640 1 --2- 172.21.15.88:0/754719520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ff32775f0 0x563ff3293df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.392+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/754719520 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ff32779d0 con 0x563ff3298bf0 2024-09-16T01:41:56.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.392+0000 7fd4e22c6640 1 --2- 172.21.15.88:0/754719520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ff32775f0 0x563ff3293df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:56.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.392+0000 7fd4e2ac7640 1 --2- 172.21.15.88:0/754719520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ff3298bf0 0x563ff3295e50 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc0460c0 tx=0x7fd4dc053000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8759deca1cfe1ac5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:56.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.392+0000 7fd4e12c4640 1 -- 172.21.15.88:0/754719520 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc00c200 con 0x563ff3298bf0 2024-09-16T01:41:56.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.392+0000 7fd4e12c4640 1 -- 172.21.15.88:0/754719520 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4dc064070 con 0x563ff3298bf0 2024-09-16T01:41:56.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.392+0000 7fd4e12c4640 1 -- 172.21.15.88:0/754719520 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc050aa0 con 0x563ff3298bf0 2024-09-16T01:41:56.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.392+0000 7fd4e47cef40 1 -- 172.21.15.88:0/754719520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ff3298bf0 msgr2=0x563ff3295e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:56.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.392+0000 7fd4e47cef40 1 --2- 172.21.15.88:0/754719520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ff3298bf0 0x563ff3295e50 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc0460c0 tx=0x7fd4dc053000 comp rx=0 tx=0).stop 2024-09-16T01:41:56.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.392+0000 7fd4e47cef40 1 -- 172.21.15.88:0/754719520 shutdown_connections 2024-09-16T01:41:56.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.392+0000 7fd4e47cef40 1 --2- 172.21.15.88:0/754719520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ff3298bf0 0x563ff3295e50 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.392+0000 7fd4e47cef40 1 --2- 172.21.15.88:0/754719520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563ff3277be0 0x563ff3294330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.392+0000 7fd4e47cef40 1 --2- 172.21.15.88:0/754719520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ff32775f0 0x563ff3293df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.392+0000 7fd4e47cef40 1 -- 172.21.15.88:0/754719520 >> 172.21.15.88:0/754719520 conn(0x563ff31ec680 msgr2=0x563ff3298030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:56.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.392+0000 7fd4e47cef40 1 -- 172.21.15.88:0/754719520 shutdown_connections 2024-09-16T01:41:56.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.393+0000 7fd4e47cef40 1 -- 172.21.15.88:0/754719520 wait complete. 2024-09-16T01:41:56.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.393+0000 7fd4e47cef40 1 Processor -- start 2024-09-16T01:41:56.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.393+0000 7fd4e47cef40 1 -- start start 2024-09-16T01:41:56.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.394+0000 7fd4e47cef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ff32775f0 0x563ff3292940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.394+0000 7fd4e47cef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ff3277be0 0x563ff3279830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.394+0000 7fd4e47cef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563ff3298bf0 0x563ff3279d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.394+0000 7fd4e47cef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x563ff3296fe0 con 0x563ff32775f0 2024-09-16T01:41:56.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.394+0000 7fd4e47cef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x563ff31ef920 con 0x563ff3277be0 2024-09-16T01:41:56.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.394+0000 7fd4e47cef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x563ff31efc00 con 0x563ff3298bf0 2024-09-16T01:41:56.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.394+0000 7fd4e22c6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ff32775f0 0x563ff3292940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.394+0000 7fd4e22c6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ff32775f0 0x563ff3292940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53856/0 (socket says 172.21.15.88:53856) 2024-09-16T01:41:56.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.394+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 learned_addr learned my addr 172.21.15.88:0/59275294 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:56.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.394+0000 7fd4dbfff640 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ff3277be0 0x563ff3279830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.394+0000 7fd4e2ac7640 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563ff3298bf0 0x563ff3279d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.394+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563ff3298bf0 msgr2=0x563ff3279d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:56.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.394+0000 7fd4e22c6640 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x563ff3298bf0 0x563ff3279d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.394+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ff3277be0 msgr2=0x563ff3279830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:56.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.394+0000 7fd4e22c6640 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ff3277be0 0x563ff3279830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.394+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ff327c610 con 0x563ff32775f0 2024-09-16T01:41:56.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.394+0000 7fd4dbfff640 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563ff3277be0 0x563ff3279830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:56.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.394+0000 7fd4e22c6640 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ff32775f0 0x563ff3292940 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4009ef0 tx=0x7fd4d4008400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:56.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.394+0000 7fd4db7fe640 1 -- 172.21.15.88:0/59275294 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d401a070 con 0x563ff32775f0 2024-09-16T01:41:56.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.394+0000 7fd4db7fe640 1 -- 172.21.15.88:0/59275294 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4d4016610 con 0x563ff32775f0 2024-09-16T01:41:56.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.394+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563ff3292e80 con 0x563ff32775f0 2024-09-16T01:41:56.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.394+0000 7fd4db7fe640 1 -- 172.21.15.88:0/59275294 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d40169c0 con 0x563ff32775f0 2024-09-16T01:41:56.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.395+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ff3278e30 con 0x563ff32775f0 2024-09-16T01:41:56.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.396+0000 7fd4db7fe640 1 -- 172.21.15.88:0/59275294 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd4d401d070 con 0x563ff32775f0 2024-09-16T01:41:56.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.396+0000 7fd4db7fe640 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd4c406acc0 0x7fd4c406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.396+0000 7fd4db7fe640 1 -- 172.21.15.88:0/59275294 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7028+0+0 (secure 0 0 0) 0x7fd4d408ad00 con 0x563ff32775f0 2024-09-16T01:41:56.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.396+0000 7fd4dbfff640 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd4c406acc0 0x7fd4c406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.403 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:41:56.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.397+0000 7fd4e47cef40 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x563ff32947d0 0x563ff327d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.397+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff32745c0 con 0x563ff32947d0 2024-09-16T01:41:56.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.397+0000 7fd4e2ac7640 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x563ff32947d0 0x563ff327d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.397+0000 7fd4dbfff640 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd4c406acc0 0x7fd4c406d180 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x563ff3295a90 tx=0x7fd4cc0013b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:56.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.397+0000 7fd4e2ac7640 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x563ff32947d0 0x563ff327d790 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:56.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.399+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v102'443 uv417 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.399+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e102) -- 0x563ff327dcd0 con 0x563ff32947d0 2024-09-16T01:41:56.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.400+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v102'444 uv444 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.400+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff333ca90 con 0x563ff32947d0 2024-09-16T01:41:56.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.401+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v102'445 uv445 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.405 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:41:56.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.401+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff333de40 con 0x563ff32947d0 2024-09-16T01:41:56.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.402+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v102'446 uv446 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.402+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff333e1e0 con 0x563ff32947d0 2024-09-16T01:41:56.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.403+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v102'447 uv447 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.403+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff333ce30 con 0x563ff32947d0 2024-09-16T01:41:56.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.404+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v102'448 uv448 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.404+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff333d1d0 con 0x563ff32947d0 2024-09-16T01:41:56.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.405+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v102'449 uv423 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.405+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e102) -- 0x563ff333d570 con 0x563ff32947d0 2024-09-16T01:41:56.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.407+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v102'450 uv450 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.407+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff333d570 con 0x563ff32947d0 2024-09-16T01:41:56.409 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:41:56.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.408+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v102'451 uv451 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.408+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff333d570 con 0x563ff32947d0 2024-09-16T01:41:56.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.409+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v102'452 uv452 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.409+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff333d570 con 0x563ff32947d0 2024-09-16T01:41:56.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.410+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v102'453 uv453 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.410+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff333d570 con 0x563ff32947d0 2024-09-16T01:41:56.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.411+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v102'454 uv454 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.411+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff333d570 con 0x563ff32947d0 2024-09-16T01:41:56.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.412+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v102'455 uv430 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.412+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e102) -- 0x563ff333d570 con 0x563ff32947d0 2024-09-16T01:41:56.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.413+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v102'456 uv456 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.413+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff333d570 con 0x563ff32947d0 2024-09-16T01:41:56.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.414+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v102'457 uv457 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.416 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:41:56.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.414+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff333d570 con 0x563ff32947d0 2024-09-16T01:41:56.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.416+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v102'458 uv458 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.416+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff333d570 con 0x563ff32947d0 2024-09-16T01:41:56.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.417+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v102'459 uv459 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.417+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff333d570 con 0x563ff32947d0 2024-09-16T01:41:56.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.418+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v102'460 uv460 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.418+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff333e580 con 0x563ff32947d0 2024-09-16T01:41:56.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.419+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v102'461 uv461 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.419+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff333e580 con 0x563ff32947d0 2024-09-16T01:41:56.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.420+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v102'462 uv436 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.420+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e102) -- 0x563ff333e580 con 0x563ff32947d0 2024-09-16T01:41:56.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.421+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v102'463 uv463 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.421+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff333e580 con 0x563ff32947d0 2024-09-16T01:41:56.423 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:41:56.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.422+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v102'464 uv464 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.422+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff333f590 con 0x563ff32947d0 2024-09-16T01:41:56.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.423+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v102'465 uv465 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.423+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff333f590 con 0x563ff32947d0 2024-09-16T01:41:56.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.424+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v102'466 uv466 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.424+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff333e580 con 0x563ff32947d0 2024-09-16T01:41:56.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.426+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v102'467 uv467 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.426+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff333e580 con 0x563ff32947d0 2024-09-16T01:41:56.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.427+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v102'468 uv442 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.427+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e102) -- 0x563ff333e580 con 0x563ff32947d0 2024-09-16T01:41:56.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.428+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v102'469 uv469 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.428+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff333e580 con 0x563ff32947d0 2024-09-16T01:41:56.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.429+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v102'470 uv470 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.430 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:41:56.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.429+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff333f590 con 0x563ff32947d0 2024-09-16T01:41:56.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.430+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v102'471 uv471 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.430+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff333f590 con 0x563ff32947d0 2024-09-16T01:41:56.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.431+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v102'472 uv472 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.431+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff333e580 con 0x563ff32947d0 2024-09-16T01:41:56.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.433+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v102'473 uv473 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.433+0000 7fd4e47cef40 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x563ff333ea90 0x563ff3340e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.433+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff33413c0 con 0x563ff333ea90 2024-09-16T01:41:56.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.433+0000 7fd4e22c6640 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x563ff333ea90 0x563ff3340e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.433+0000 7fd4e22c6640 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x563ff333ea90 0x563ff3340e80 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:56.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.434+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v102'331 uv313 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fd4d4026070 con 0x563ff333ea90 2024-09-16T01:41:56.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.435+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e102) -- 0x563ff3341760 con 0x563ff333ea90 2024-09-16T01:41:56.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.436+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v102'332 uv332 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd4d4026070 con 0x563ff333ea90 2024-09-16T01:41:56.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.436+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3341b00 con 0x563ff333ea90 2024-09-16T01:41:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.437+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v102'333 uv333 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd4d4026070 con 0x563ff333ea90 2024-09-16T01:41:56.438 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:41:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.437+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3342350 con 0x563ff333ea90 2024-09-16T01:41:56.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.438+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v102'334 uv334 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd4d4026070 con 0x563ff333ea90 2024-09-16T01:41:56.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.438+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff33426f0 con 0x563ff333ea90 2024-09-16T01:41:56.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.439+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v102'335 uv335 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd4d4026070 con 0x563ff333ea90 2024-09-16T01:41:56.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.439+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3342a90 con 0x563ff333ea90 2024-09-16T01:41:56.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.440+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v102'336 uv318 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd4d4026070 con 0x563ff333ea90 2024-09-16T01:41:56.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.440+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e102) -- 0x563ff3342e30 con 0x563ff333ea90 2024-09-16T01:41:56.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.441+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v102'337 uv337 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4d4026070 con 0x563ff333ea90 2024-09-16T01:41:56.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.441+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff33431d0 con 0x563ff333ea90 2024-09-16T01:41:56.443 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:41:56.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.442+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v102'338 uv338 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4d4026070 con 0x563ff333ea90 2024-09-16T01:41:56.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.442+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff33431d0 con 0x563ff333ea90 2024-09-16T01:41:56.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.443+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v102'339 uv339 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4d4026070 con 0x563ff333ea90 2024-09-16T01:41:56.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.443+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff33431d0 con 0x563ff333ea90 2024-09-16T01:41:56.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.444+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v102'340 uv340 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4d4026070 con 0x563ff333ea90 2024-09-16T01:41:56.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.444+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff33431d0 con 0x563ff333ea90 2024-09-16T01:41:56.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.446+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v102'341 uv323 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd4d4026070 con 0x563ff333ea90 2024-09-16T01:41:56.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.446+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e102) -- 0x563ff33431d0 con 0x563ff333ea90 2024-09-16T01:41:56.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.447+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v102'342 uv342 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4d4026070 con 0x563ff333ea90 2024-09-16T01:41:56.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.447+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff33431d0 con 0x563ff333ea90 2024-09-16T01:41:56.449 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:41:56.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.448+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v102'343 uv343 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4d4026070 con 0x563ff333ea90 2024-09-16T01:41:56.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.448+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3347650 con 0x563ff333ea90 2024-09-16T01:41:56.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.449+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v102'344 uv344 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4d4026070 con 0x563ff333ea90 2024-09-16T01:41:56.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.449+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3347650 con 0x563ff333ea90 2024-09-16T01:41:56.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.450+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v102'345 uv345 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4d4026070 con 0x563ff333ea90 2024-09-16T01:41:56.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.450+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff33431d0 con 0x563ff333ea90 2024-09-16T01:41:56.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.451+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v102'346 uv329 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd4d4026070 con 0x563ff333ea90 2024-09-16T01:41:56.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.451+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e102) -- 0x563ff33431d0 con 0x563ff333ea90 2024-09-16T01:41:56.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.452+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v102'347 uv347 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4d4026070 con 0x563ff333ea90 2024-09-16T01:41:56.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.452+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff33431d0 con 0x563ff333ea90 2024-09-16T01:41:56.454 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:41:56.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.453+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v102'348 uv348 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4d4026070 con 0x563ff333ea90 2024-09-16T01:41:56.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.453+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff33441e0 con 0x563ff333ea90 2024-09-16T01:41:56.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.455+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v102'349 uv349 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4d4026070 con 0x563ff333ea90 2024-09-16T01:41:56.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.455+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff33441e0 con 0x563ff333ea90 2024-09-16T01:41:56.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.456+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v102'350 uv350 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4d4026070 con 0x563ff333ea90 2024-09-16T01:41:56.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.456+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff33431d0 con 0x563ff333ea90 2024-09-16T01:41:56.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.457+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v102'351 uv351 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4d4026070 con 0x563ff333ea90 2024-09-16T01:41:56.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.457+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff33431d0 con 0x563ff32947d0 2024-09-16T01:41:56.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.458+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v102'87 uv86 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.458+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e102) -- 0x563ff33431d0 con 0x563ff32947d0 2024-09-16T01:41:56.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.459+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v102'88 uv88 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.459+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff33431d0 con 0x563ff32947d0 2024-09-16T01:41:56.461 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:41:56.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.460+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v102'89 uv89 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.460+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff33441e0 con 0x563ff32947d0 2024-09-16T01:41:56.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.461+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v102'90 uv90 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.461+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff33441e0 con 0x563ff32947d0 2024-09-16T01:41:56.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.461+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v102'91 uv91 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.461+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff33431d0 con 0x563ff32947d0 2024-09-16T01:41:56.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.462+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v102'92 uv92 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.462+0000 7fd4e47cef40 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x563ff33436e0 0x563ff3345ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.462+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3346010 con 0x563ff33436e0 2024-09-16T01:41:56.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.462+0000 7fd4dbfff640 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x563ff33436e0 0x563ff3345ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.463+0000 7fd4dbfff640 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x563ff33436e0 0x563ff3345ad0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:56.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.464+0000 7fd4dbfff640 1 -- 172.21.15.88:0/59275294 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v102'69 uv68 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fd4cc00ec70 con 0x563ff33436e0 2024-09-16T01:41:56.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.464+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e102) -- 0x563ff33464e0 con 0x563ff33436e0 2024-09-16T01:41:56.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.465+0000 7fd4dbfff640 1 -- 172.21.15.88:0/59275294 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v102'70 uv70 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd4cc00ec70 con 0x563ff33436e0 2024-09-16T01:41:56.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.465+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff33469e0 con 0x563ff33436e0 2024-09-16T01:41:56.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.465+0000 7fd4dbfff640 1 -- 172.21.15.88:0/59275294 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v102'71 uv71 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd4cc00ec70 con 0x563ff33436e0 2024-09-16T01:41:56.467 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:41:56.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.466+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3341f80 con 0x563ff33436e0 2024-09-16T01:41:56.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.466+0000 7fd4dbfff640 1 -- 172.21.15.88:0/59275294 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v102'72 uv72 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd4cc00ec70 con 0x563ff33436e0 2024-09-16T01:41:56.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.466+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3347310 con 0x563ff32947d0 2024-09-16T01:41:56.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.468+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v102'87 uv86 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.468+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e102) -- 0x563ff3347310 con 0x563ff32947d0 2024-09-16T01:41:56.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.469+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v102'88 uv88 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.469+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3347310 con 0x563ff32947d0 2024-09-16T01:41:56.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.470+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v102'89 uv89 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.471 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:41:56.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.470+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3347310 con 0x563ff32947d0 2024-09-16T01:41:56.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.471+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v102'90 uv90 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.471+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3347310 con 0x563ff32947d0 2024-09-16T01:41:56.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.472+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v102'91 uv91 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.472+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3347310 con 0x563ff32947d0 2024-09-16T01:41:56.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.473+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v102'92 uv92 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.473+0000 7fd4e47cef40 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x563ff3347c10 0x563ff334a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.473+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff334a540 con 0x563ff3347c10 2024-09-16T01:41:56.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.473+0000 7fd4e2ac7640 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x563ff3347c10 0x563ff334a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.474+0000 7fd4e2ac7640 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x563ff3347c10 0x563ff334a000 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:56.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.475+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v102'160 uv154 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.475+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e102) -- 0x563ff334a540 con 0x563ff3347c10 2024-09-16T01:41:56.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.476+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v102'161 uv161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.476+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff334a930 con 0x563ff3347c10 2024-09-16T01:41:56.478 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:41:56.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.477+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v102'162 uv162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.477+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff334b940 con 0x563ff3347c10 2024-09-16T01:41:56.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.478+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v102'163 uv163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.478+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff334b940 con 0x563ff3347c10 2024-09-16T01:41:56.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.479+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v102'164 uv164 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.479+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff334a930 con 0x563ff3347c10 2024-09-16T01:41:56.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.480+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v102'165 uv165 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.480+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff334a930 con 0x563ff3347c10 2024-09-16T01:41:56.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.481+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v102'166 uv159 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.481+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e102) -- 0x563ff334a930 con 0x563ff3347c10 2024-09-16T01:41:56.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.481+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v102'167 uv167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.482+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff334a930 con 0x563ff3347c10 2024-09-16T01:41:56.484 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:41:56.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.483+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v102'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.483+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff334a930 con 0x563ff3347c10 2024-09-16T01:41:56.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.484+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v102'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.484+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff334a930 con 0x563ff3347c10 2024-09-16T01:41:56.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.485+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v102'170 uv170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.485+0000 7fd4e47cef40 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x563ff334afb0 0x563ff334d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.485+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff334da00 con 0x563ff334afb0 2024-09-16T01:41:56.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.485+0000 7fd4e22c6640 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x563ff334afb0 0x563ff334d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.485+0000 7fd4e22c6640 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x563ff334afb0 0x563ff334d3a0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:56.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.486+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v102'71 uv70 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fd4d406dd70 con 0x563ff334afb0 2024-09-16T01:41:56.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.486+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e102) -- 0x563ff334dae0 con 0x563ff334afb0 2024-09-16T01:41:56.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.487+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v102'72 uv72 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd4d406dd70 con 0x563ff334afb0 2024-09-16T01:41:56.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.487+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff334df00 con 0x563ff334afb0 2024-09-16T01:41:56.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.488+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v102'73 uv73 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd4d406dd70 con 0x563ff334afb0 2024-09-16T01:41:56.489 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:41:56.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.488+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff334df00 con 0x563ff334afb0 2024-09-16T01:41:56.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.489+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v102'74 uv74 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd4d406dd70 con 0x563ff334afb0 2024-09-16T01:41:56.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.489+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff334df00 con 0x563ff334afb0 2024-09-16T01:41:56.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.490+0000 7fd4e22c6640 1 -- 172.21.15.88:0/59275294 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v102'75 uv75 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd4d406dd70 con 0x563ff334afb0 2024-09-16T01:41:56.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.490+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff334df00 con 0x563ff32947d0 2024-09-16T01:41:56.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.491+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v102'87 uv86 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.491+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e102) -- 0x563ff334df00 con 0x563ff32947d0 2024-09-16T01:41:56.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.492+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v102'88 uv88 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.492+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff334df00 con 0x563ff32947d0 2024-09-16T01:41:56.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.493+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v102'89 uv89 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.494 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:41:56.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.493+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff334ef10 con 0x563ff32947d0 2024-09-16T01:41:56.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.494+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v102'90 uv90 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.494+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff334ef10 con 0x563ff32947d0 2024-09-16T01:41:56.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.495+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v102'91 uv91 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.495+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff334df00 con 0x563ff32947d0 2024-09-16T01:41:56.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.496+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v102'92 uv92 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff32947d0 2024-09-16T01:41:56.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.496+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff334df00 con 0x563ff33436e0 2024-09-16T01:41:56.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.497+0000 7fd4dbfff640 1 -- 172.21.15.88:0/59275294 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v102'142 uv136 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fd4cc00ec70 con 0x563ff33436e0 2024-09-16T01:41:56.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.497+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e102) -- 0x563ff334e2a0 con 0x563ff33436e0 2024-09-16T01:41:56.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.498+0000 7fd4dbfff640 1 -- 172.21.15.88:0/59275294 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v102'143 uv143 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd4cc00ec70 con 0x563ff33436e0 2024-09-16T01:41:56.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.498+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff334e640 con 0x563ff33436e0 2024-09-16T01:41:56.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.499+0000 7fd4dbfff640 1 -- 172.21.15.88:0/59275294 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v102'144 uv144 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd4cc00ec70 con 0x563ff33436e0 2024-09-16T01:41:56.500 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:41:56.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.499+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff334e640 con 0x563ff33436e0 2024-09-16T01:41:56.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.500+0000 7fd4dbfff640 1 -- 172.21.15.88:0/59275294 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v102'145 uv145 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd4cc00ec70 con 0x563ff33436e0 2024-09-16T01:41:56.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.500+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff334e640 con 0x563ff33436e0 2024-09-16T01:41:56.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.501+0000 7fd4dbfff640 1 -- 172.21.15.88:0/59275294 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v102'146 uv146 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd4cc00ec70 con 0x563ff33436e0 2024-09-16T01:41:56.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.501+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff334e640 con 0x563ff33436e0 2024-09-16T01:41:56.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.501+0000 7fd4dbfff640 1 -- 172.21.15.88:0/59275294 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v102'147 uv141 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd4cc00ec70 con 0x563ff33436e0 2024-09-16T01:41:56.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.501+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e102) -- 0x563ff334e640 con 0x563ff33436e0 2024-09-16T01:41:56.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.502+0000 7fd4dbfff640 1 -- 172.21.15.88:0/59275294 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v102'148 uv148 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4cc00ec70 con 0x563ff33436e0 2024-09-16T01:41:56.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.502+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff334e640 con 0x563ff33436e0 2024-09-16T01:41:56.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.503+0000 7fd4dbfff640 1 -- 172.21.15.88:0/59275294 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v102'149 uv149 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4cc00ec70 con 0x563ff33436e0 2024-09-16T01:41:56.504 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:41:56.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.503+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff334e640 con 0x563ff33436e0 2024-09-16T01:41:56.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.504+0000 7fd4dbfff640 1 -- 172.21.15.88:0/59275294 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v102'150 uv150 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4cc00ec70 con 0x563ff33436e0 2024-09-16T01:41:56.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.504+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3346ee0 con 0x563ff33436e0 2024-09-16T01:41:56.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.505+0000 7fd4dbfff640 1 -- 172.21.15.88:0/59275294 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v102'151 uv151 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4cc00ec70 con 0x563ff33436e0 2024-09-16T01:41:56.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.505+0000 7fd4e47cef40 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x563ff334f420 0x563ff3351bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.505+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3352100 con 0x563ff334f420 2024-09-16T01:41:56.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.505+0000 7fd4dbfff640 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x563ff334f420 0x563ff3351bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.506+0000 7fd4dbfff640 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x563ff334f420 0x563ff3351bc0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:56.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.507+0000 7fd4dbfff640 1 -- 172.21.15.88:0/59275294 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v102'66 uv64 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fd4cc00ec70 con 0x563ff334f420 2024-09-16T01:41:56.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.507+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e102) -- 0x563ff3352100 con 0x563ff334f420 2024-09-16T01:41:56.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.507+0000 7fd4dbfff640 1 -- 172.21.15.88:0/59275294 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v102'67 uv67 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fd4cc00ec70 con 0x563ff334f420 2024-09-16T01:41:56.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.508+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3352100 con 0x563ff334f420 2024-09-16T01:41:56.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.508+0000 7fd4dbfff640 1 -- 172.21.15.88:0/59275294 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v102'68 uv68 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fd4cc00ec70 con 0x563ff334f420 2024-09-16T01:41:56.510 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:41:56.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.508+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3352100 con 0x563ff334f420 2024-09-16T01:41:56.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.509+0000 7fd4dbfff640 1 -- 172.21.15.88:0/59275294 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v102'69 uv69 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fd4cc00ec70 con 0x563ff334f420 2024-09-16T01:41:56.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.509+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3352100 con 0x563ff33436e0 2024-09-16T01:41:56.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.510+0000 7fd4dbfff640 1 -- 172.21.15.88:0/59275294 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v102'87 uv86 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd4cc00ec70 con 0x563ff33436e0 2024-09-16T01:41:56.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.510+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e102) -- 0x563ff3352100 con 0x563ff33436e0 2024-09-16T01:41:56.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.512+0000 7fd4dbfff640 1 -- 172.21.15.88:0/59275294 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v102'88 uv88 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4cc00ec70 con 0x563ff33436e0 2024-09-16T01:41:56.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.512+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3352100 con 0x563ff33436e0 2024-09-16T01:41:56.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.512+0000 7fd4dbfff640 1 -- 172.21.15.88:0/59275294 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v102'89 uv89 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4cc00ec70 con 0x563ff33436e0 2024-09-16T01:41:56.514 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:41:56.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.512+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3353110 con 0x563ff33436e0 2024-09-16T01:41:56.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.513+0000 7fd4dbfff640 1 -- 172.21.15.88:0/59275294 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v102'90 uv90 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4cc00ec70 con 0x563ff33436e0 2024-09-16T01:41:56.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.513+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3353110 con 0x563ff33436e0 2024-09-16T01:41:56.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.514+0000 7fd4dbfff640 1 -- 172.21.15.88:0/59275294 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v102'91 uv91 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4cc00ec70 con 0x563ff33436e0 2024-09-16T01:41:56.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.514+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3352100 con 0x563ff33436e0 2024-09-16T01:41:56.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.515+0000 7fd4dbfff640 1 -- 172.21.15.88:0/59275294 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v102'92 uv92 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4cc00ec70 con 0x563ff33436e0 2024-09-16T01:41:56.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.515+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3352100 con 0x563ff3347c10 2024-09-16T01:41:56.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.516+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v102'74 uv73 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.516+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e102) -- 0x563ff3352100 con 0x563ff3347c10 2024-09-16T01:41:56.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.518+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v102'75 uv75 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.518+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3352100 con 0x563ff3347c10 2024-09-16T01:41:56.520 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:41:56.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.519+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v102'76 uv76 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.519+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3352100 con 0x563ff3347c10 2024-09-16T01:41:56.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.520+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v102'77 uv77 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.520+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3352100 con 0x563ff3347c10 2024-09-16T01:41:56.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.521+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v102'78 uv78 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.521+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3352100 con 0x563ff3347c10 2024-09-16T01:41:56.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.522+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v102'182 uv172 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.522+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e102) -- 0x563ff3352100 con 0x563ff3347c10 2024-09-16T01:41:56.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.523+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v102'183 uv183 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.523+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3352100 con 0x563ff3347c10 2024-09-16T01:41:56.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.524+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v102'184 uv184 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.526 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:41:56.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.524+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3352100 con 0x563ff3347c10 2024-09-16T01:41:56.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.525+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v102'185 uv185 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.526+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3352100 con 0x563ff3347c10 2024-09-16T01:41:56.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.526+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v102'186 uv186 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.526+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3352100 con 0x563ff3347c10 2024-09-16T01:41:56.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.527+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v102'187 uv187 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.529 INFO:teuthology.orchestra.run.smithi088.stdout:Write #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:41:56.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.527+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3353110 con 0x563ff3347c10 2024-09-16T01:41:56.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.529+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v102'188 uv188 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.529+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3353110 con 0x563ff3347c10 2024-09-16T01:41:56.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.530+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v102'189 uv189 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.530+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3352100 con 0x563ff3347c10 2024-09-16T01:41:56.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.531+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [omap-set-header] v102'190 uv190 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.531+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x563ff3352100 con 0x563ff3347c10 2024-09-16T01:41:56.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v102'191 uv191 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd4dc050860 con 0x563ff3347c10 2024-09-16T01:41:56.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x563ff32947d0 msgr2=0x563ff327d790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:56.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e47cef40 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x563ff32947d0 0x563ff327d790 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x563ff33436e0 msgr2=0x563ff3345ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:56.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e47cef40 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x563ff33436e0 0x563ff3345ad0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x563ff334f420 msgr2=0x563ff3351bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:56.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e47cef40 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x563ff334f420 0x563ff3351bc0 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x563ff333ea90 msgr2=0x563ff3340e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e47cef40 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x563ff333ea90 0x563ff3340e80 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x563ff3347c10 msgr2=0x563ff334a000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e47cef40 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x563ff3347c10 0x563ff334a000 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x563ff334afb0 msgr2=0x563ff334d3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e47cef40 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x563ff334afb0 0x563ff334d3a0 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 reap_dead start 2024-09-16T01:41:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 reap_dead start 2024-09-16T01:41:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e2ac7640 1 -- 172.21.15.88:0/59275294 reap_dead start 2024-09-16T01:41:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd4c406acc0 msgr2=0x7fd4c406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e47cef40 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd4c406acc0 0x7fd4c406d180 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x563ff3295a90 tx=0x7fd4cc0013b0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ff32775f0 msgr2=0x563ff3292940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e47cef40 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ff32775f0 0x563ff3292940 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4009ef0 tx=0x7fd4d4008400 comp rx=0 tx=0).stop 2024-09-16T01:41:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 shutdown_connections 2024-09-16T01:41:56.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e47cef40 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd4c406acc0 0x7fd4c406d180 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e47cef40 1 --2- 172.21.15.88:0/59275294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x563ff32775f0 0x563ff3292940 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 >> 172.21.15.88:0/59275294 conn(0x563ff31ec680 msgr2=0x563ff3295ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:56.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 shutdown_connections 2024-09-16T01:41:56.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.532+0000 7fd4e47cef40 1 -- 172.21.15.88:0/59275294 wait complete. 2024-09-16T01:41:56.537 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put 1.00000000.inode - 2024-09-16T01:41:56.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.619+0000 7f04f216df40 1 Processor -- start 2024-09-16T01:41:56.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.619+0000 7f04f216df40 1 -- start start 2024-09-16T01:41:56.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.620+0000 7f04f216df40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5555645ffee0 0x5555645fe3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.620+0000 7f04f216df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555564630590 0x5555646cf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.620+0000 7f04f216df40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555646d3c90 0x555564765050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.620+0000 7f04f216df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x555564551510 con 0x555564630590 2024-09-16T01:41:56.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.620+0000 7f04f216df40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5555645aac50 con 0x5555646d3c90 2024-09-16T01:41:56.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.620+0000 7f04f216df40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5555646d4c30 con 0x5555645ffee0 2024-09-16T01:41:56.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.620+0000 7f04e7462640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555564630590 0x5555646cf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.620+0000 7f04e7462640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555564630590 0x5555646cf650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53866/0 (socket says 172.21.15.88:53866) 2024-09-16T01:41:56.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.620+0000 7f04e7462640 1 -- 172.21.15.88:0/2249167709 learned_addr learned my addr 172.21.15.88:0/2249167709 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:56.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.620+0000 7f04efc63640 1 --2- 172.21.15.88:0/2249167709 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5555645ffee0 0x5555645fe3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.620+0000 7f04f0464640 1 --2- 172.21.15.88:0/2249167709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555646d3c90 0x555564765050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.620+0000 7f04e7462640 1 -- 172.21.15.88:0/2249167709 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5555645ffee0 msgr2=0x5555645fe3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:56.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.620+0000 7f04e7462640 1 --2- 172.21.15.88:0/2249167709 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5555645ffee0 0x5555645fe3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.620+0000 7f04e7462640 1 -- 172.21.15.88:0/2249167709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555646d3c90 msgr2=0x555564765050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:56.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.620+0000 7f04e7462640 1 --2- 172.21.15.88:0/2249167709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555646d3c90 0x555564765050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.620+0000 7f04e7462640 1 -- 172.21.15.88:0/2249167709 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555646c72e0 con 0x555564630590 2024-09-16T01:41:56.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.620+0000 7f04f0464640 1 --2- 172.21.15.88:0/2249167709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555646d3c90 0x555564765050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:56.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.620+0000 7f04e7462640 1 --2- 172.21.15.88:0/2249167709 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555564630590 0x5555646cf650 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f04d8000c00 tx=0x7f04d8051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebedba8cd931d7e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:56.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.620+0000 7f04eec61640 1 -- 172.21.15.88:0/2249167709 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d8052ca0 con 0x555564630590 2024-09-16T01:41:56.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.620+0000 7f04eec61640 1 -- 172.21.15.88:0/2249167709 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04d8066430 con 0x555564630590 2024-09-16T01:41:56.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.620+0000 7f04eec61640 1 -- 172.21.15.88:0/2249167709 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d8052e60 con 0x555564630590 2024-09-16T01:41:56.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.621+0000 7f04f216df40 1 -- 172.21.15.88:0/2249167709 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555564630590 msgr2=0x5555646cf650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:56.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.621+0000 7f04f216df40 1 --2- 172.21.15.88:0/2249167709 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555564630590 0x5555646cf650 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f04d8000c00 tx=0x7f04d8051000 comp rx=0 tx=0).stop 2024-09-16T01:41:56.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.621+0000 7f04f216df40 1 -- 172.21.15.88:0/2249167709 shutdown_connections 2024-09-16T01:41:56.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.621+0000 7f04f216df40 1 --2- 172.21.15.88:0/2249167709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555646d3c90 0x555564765050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.621+0000 7f04f216df40 1 --2- 172.21.15.88:0/2249167709 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555564630590 0x5555646cf650 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.621+0000 7f04f216df40 1 --2- 172.21.15.88:0/2249167709 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5555645ffee0 0x5555645fe3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.621+0000 7f04f216df40 1 -- 172.21.15.88:0/2249167709 >> 172.21.15.88:0/2249167709 conn(0x55556462a680 msgr2=0x5555645ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:56.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.621+0000 7f04f216df40 1 -- 172.21.15.88:0/2249167709 shutdown_connections 2024-09-16T01:41:56.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.621+0000 7f04f216df40 1 -- 172.21.15.88:0/2249167709 wait complete. 2024-09-16T01:41:56.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.621+0000 7f04f216df40 1 Processor -- start 2024-09-16T01:41:56.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.621+0000 7f04f216df40 1 -- start start 2024-09-16T01:41:56.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.622+0000 7f04f216df40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555645ffee0 0x5555646c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.622+0000 7f04f216df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555564630590 0x555564646c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.622+0000 7f04f216df40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5555646d3c90 0x555564647160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.622+0000 7f04f216df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5555646c6c90 con 0x555564630590 2024-09-16T01:41:56.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.622+0000 7f04f216df40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x555564649b40 con 0x5555645ffee0 2024-09-16T01:41:56.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.622+0000 7f04f216df40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55556462a1e0 con 0x5555646d3c90 2024-09-16T01:41:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.622+0000 7f04e7462640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555564630590 0x555564646c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.622+0000 7f04e7462640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555564630590 0x555564646c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53870/0 (socket says 172.21.15.88:53870) 2024-09-16T01:41:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.622+0000 7f04e7462640 1 -- 172.21.15.88:0/2352632555 learned_addr learned my addr 172.21.15.88:0/2352632555 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.622+0000 7f04efc63640 1 --2- 172.21.15.88:0/2352632555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555645ffee0 0x5555646c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.622+0000 7f04f0464640 1 --2- 172.21.15.88:0/2352632555 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5555646d3c90 0x555564647160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.622+0000 7f04e7462640 1 -- 172.21.15.88:0/2352632555 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5555646d3c90 msgr2=0x555564647160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.622+0000 7f04e7462640 1 --2- 172.21.15.88:0/2352632555 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5555646d3c90 0x555564647160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.622+0000 7f04e7462640 1 -- 172.21.15.88:0/2352632555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555645ffee0 msgr2=0x5555646c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.622+0000 7f04e7462640 1 --2- 172.21.15.88:0/2352632555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555645ffee0 0x5555646c6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.622+0000 7f04e7462640 1 -- 172.21.15.88:0/2352632555 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555646c6a90 con 0x555564630590 2024-09-16T01:41:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.623+0000 7f04e7462640 1 --2- 172.21.15.88:0/2352632555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555564630590 0x555564646c20 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f04d8005870 tx=0x7f04d8002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.623+0000 7f04edc5f640 1 -- 172.21.15.88:0/2352632555 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d8052e40 con 0x555564630590 2024-09-16T01:41:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.623+0000 7f04edc5f640 1 -- 172.21.15.88:0/2352632555 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04d80665e0 con 0x555564630590 2024-09-16T01:41:56.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.623+0000 7f04f216df40 1 -- 172.21.15.88:0/2352632555 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x555564769ef0 con 0x555564630590 2024-09-16T01:41:56.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.623+0000 7f04edc5f640 1 -- 172.21.15.88:0/2352632555 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d8066990 con 0x555564630590 2024-09-16T01:41:56.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.623+0000 7f04f216df40 1 -- 172.21.15.88:0/2352632555 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55556476a4b0 con 0x555564630590 2024-09-16T01:41:56.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.624+0000 7f04edc5f640 1 -- 172.21.15.88:0/2352632555 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f04d8059020 con 0x555564630590 2024-09-16T01:41:56.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.625+0000 7f04edc5f640 1 --2- 172.21.15.88:0/2352632555 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f04d006acc0 0x7f04d006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.625+0000 7f04edc5f640 1 -- 172.21.15.88:0/2352632555 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7028+0+0 (secure 0 0 0) 0x7f04d80cfb60 con 0x555564630590 2024-09-16T01:41:56.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.625+0000 7f04efc63640 1 --2- 172.21.15.88:0/2352632555 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f04d006acc0 0x7f04d006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.625+0000 7f04f216df40 1 --2- 172.21.15.88:0/2352632555 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x555564761ae0 0x555564761ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.625+0000 7f04f216df40 1 -- 172.21.15.88:0/2352632555 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:1 12.14 12:287034c4:::1.00000000.inode:head [writefull 0~80 in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102) -- 0x555564762430 con 0x555564761ae0 2024-09-16T01:41:56.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.625+0000 7f04efc63640 1 --2- 172.21.15.88:0/2352632555 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f04d006acc0 0x7f04d006d180 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x555564602540 tx=0x7f04e00017e0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:56.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.625+0000 7f04f0464640 1 --2- 172.21.15.88:0/2352632555 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x555564761ae0 0x555564761ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.625+0000 7f04f0464640 1 --2- 172.21.15.88:0/2352632555 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x555564761ae0 0x555564761ef0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:56.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.627+0000 7f04f0464640 1 -- 172.21.15.88:0/2352632555 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(1 1.00000000.inode [writefull 0~80] v102'352 uv352 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f04e80106e0 con 0x555564761ae0 2024-09-16T01:41:56.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.627+0000 7f04f216df40 1 -- 172.21.15.88:0/2352632555 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x555564761ae0 msgr2=0x555564761ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:56.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.627+0000 7f04f216df40 1 --2- 172.21.15.88:0/2352632555 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x555564761ae0 0x555564761ef0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.627+0000 7f04f216df40 1 -- 172.21.15.88:0/2352632555 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f04d006acc0 msgr2=0x7f04d006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:56.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.627+0000 7f04f216df40 1 --2- 172.21.15.88:0/2352632555 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f04d006acc0 0x7f04d006d180 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x555564602540 tx=0x7f04e00017e0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.627+0000 7f04f216df40 1 -- 172.21.15.88:0/2352632555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555564630590 msgr2=0x555564646c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:56.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.627+0000 7f04f216df40 1 --2- 172.21.15.88:0/2352632555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x555564630590 0x555564646c20 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f04d8005870 tx=0x7f04d8002fb0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.627+0000 7f04f0464640 1 -- 172.21.15.88:0/2352632555 reap_dead start 2024-09-16T01:41:56.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.627+0000 7f04f216df40 1 -- 172.21.15.88:0/2352632555 shutdown_connections 2024-09-16T01:41:56.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.627+0000 7f04f216df40 1 -- 172.21.15.88:0/2352632555 >> 172.21.15.88:0/2352632555 conn(0x55556462a680 msgr2=0x5555646cfa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:56.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.627+0000 7f04f216df40 1 -- 172.21.15.88:0/2352632555 shutdown_connections 2024-09-16T01:41:56.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.627+0000 7f04f216df40 1 -- 172.21.15.88:0/2352632555 wait complete. 2024-09-16T01:41:56.634 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:41:56.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.773+0000 7f726c65e640 1 Processor -- start 2024-09-16T01:41:56.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.773+0000 7f726c65e640 1 -- start start 2024-09-16T01:41:56.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.774+0000 7f726c65e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f72640ceda0 0x7f7264073490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.774+0000 7f726c65e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f72640cf5c0 0x7f726406e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.774+0000 7f726c65e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72640722d0 con 0x7f72640cf180 2024-09-16T01:41:56.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.774+0000 7f726c65e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f72640cfa10 con 0x7f72640ceda0 2024-09-16T01:41:56.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.774+0000 7f726c65e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7264072590 con 0x7f72640cf5c0 2024-09-16T01:41:56.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.774+0000 7f726a3fa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f72640ceda0 0x7f7264073490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.774+0000 7f726a3fa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f72640ceda0 0x7f7264073490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53872/0 (socket says 172.21.15.88:53872) 2024-09-16T01:41:56.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.774+0000 7f726a3fa640 1 -- 172.21.15.88:0/4029250151 learned_addr learned my addr 172.21.15.88:0/4029250151 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:56.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.775+0000 7f726a3fa640 1 -- 172.21.15.88:0/4029250151 >> v2:172.21.15.154:3300/0 conn(0x7f72640cf5c0 msgr2=0x7f726406e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:56.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.775+0000 7f726a3fa640 1 --2- 172.21.15.88:0/4029250151 >> v2:172.21.15.154:3300/0 conn(0x7f72640cf5c0 0x7f726406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.775+0000 7f726a3fa640 1 -- 172.21.15.88:0/4029250151 >> v1:172.21.15.154:6789/0 conn(0x7f72640cf180 legacy=0x7f726406dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:56.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.775+0000 7f726a3fa640 1 -- 172.21.15.88:0/4029250151 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7264072870 con 0x7f72640ceda0 2024-09-16T01:41:56.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.775+0000 7f726a3fa640 1 --2- 172.21.15.88:0/4029250151 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72640ceda0 0x7f7264073490 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f72640b29e0 tx=0x7f726002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5212034ef0e7bc9e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:56.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.775+0000 7f72693f8640 1 -- 172.21.15.88:0/4029250151 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7260037070 con 0x7f72640ceda0 2024-09-16T01:41:56.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.775+0000 7f72693f8640 1 -- 172.21.15.88:0/4029250151 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7260032440 con 0x7f72640ceda0 2024-09-16T01:41:56.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.775+0000 7f72693f8640 1 -- 172.21.15.88:0/4029250151 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7260032810 con 0x7f72640ceda0 2024-09-16T01:41:56.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.776+0000 7f726c65e640 1 -- 172.21.15.88:0/4029250151 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72640ceda0 msgr2=0x7f7264073490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:56.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.776+0000 7f726c65e640 1 --2- 172.21.15.88:0/4029250151 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72640ceda0 0x7f7264073490 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f72640b29e0 tx=0x7f726002a230 comp rx=0 tx=0).stop 2024-09-16T01:41:56.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.776+0000 7f726c65e640 1 -- 172.21.15.88:0/4029250151 shutdown_connections 2024-09-16T01:41:56.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.776+0000 7f726c65e640 1 --2- 172.21.15.88:0/4029250151 >> v2:172.21.15.154:3300/0 conn(0x7f72640cf5c0 0x7f726406e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.776+0000 7f726c65e640 1 --2- 172.21.15.88:0/4029250151 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72640ceda0 0x7f7264073490 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.776+0000 7f726c65e640 1 -- 172.21.15.88:0/4029250151 >> 172.21.15.88:0/4029250151 conn(0x7f726406cb50 msgr2=0x7f726406cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:56.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.776+0000 7f726c65e640 1 -- 172.21.15.88:0/4029250151 shutdown_connections 2024-09-16T01:41:56.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.776+0000 7f726c65e640 1 -- 172.21.15.88:0/4029250151 wait complete. 2024-09-16T01:41:56.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.777+0000 7f726c65e640 1 Processor -- start 2024-09-16T01:41:56.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.777+0000 7f726c65e640 1 -- start start 2024-09-16T01:41:56.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.778+0000 7f726c65e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72640ceda0 0x7f726414f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.778+0000 7f726c65e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72640cf180 0x7f7264153410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.778+0000 7f726c65e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72640cf5c0 0x7f72641e35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.778+0000 7f726c65e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f72640d0510 con 0x7f72640cf5c0 2024-09-16T01:41:56.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.778+0000 7f726c65e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7264072e30 con 0x7f72640ceda0 2024-09-16T01:41:56.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.778+0000 7f726c65e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f72640739d0 con 0x7f72640cf180 2024-09-16T01:41:56.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.778+0000 7f726abfb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72640cf5c0 0x7f72641e35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.778+0000 7f726abfb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72640cf5c0 0x7f72641e35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53880/0 (socket says 172.21.15.88:53880) 2024-09-16T01:41:56.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.778+0000 7f726abfb640 1 -- 172.21.15.88:0/1416838274 learned_addr learned my addr 172.21.15.88:0/1416838274 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:56.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.778+0000 7f726a3fa640 1 --2- 172.21.15.88:0/1416838274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72640ceda0 0x7f726414f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.778+0000 7f726abfb640 1 -- 172.21.15.88:0/1416838274 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72640cf180 msgr2=0x7f7264153410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:56.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.778+0000 7f7269bf9640 1 --2- 172.21.15.88:0/1416838274 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72640cf180 0x7f7264153410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.778+0000 7f726abfb640 1 --2- 172.21.15.88:0/1416838274 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72640cf180 0x7f7264153410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.778+0000 7f726abfb640 1 -- 172.21.15.88:0/1416838274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72640ceda0 msgr2=0x7f726414f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:56.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.778+0000 7f726abfb640 1 --2- 172.21.15.88:0/1416838274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72640ceda0 0x7f726414f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:56.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.778+0000 7f726abfb640 1 -- 172.21.15.88:0/1416838274 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7264153bb0 con 0x7f72640cf5c0 2024-09-16T01:41:56.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.779+0000 7f726a3fa640 1 --2- 172.21.15.88:0/1416838274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72640ceda0 0x7f726414f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:56.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.779+0000 7f7269bf9640 1 --2- 172.21.15.88:0/1416838274 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72640cf180 0x7f7264153410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:56.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.779+0000 7f726abfb640 1 --2- 172.21.15.88:0/1416838274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72640cf5c0 0x7f72641e35c0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f725c00b970 tx=0x7f725c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:56.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.779+0000 7f725affd640 1 -- 172.21.15.88:0/1416838274 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725c019070 con 0x7f72640cf5c0 2024-09-16T01:41:56.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.779+0000 7f725affd640 1 -- 172.21.15.88:0/1416838274 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f725c00cb60 con 0x7f72640cf5c0 2024-09-16T01:41:56.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.779+0000 7f726c65e640 1 -- 172.21.15.88:0/1416838274 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7264153e90 con 0x7f72640cf5c0 2024-09-16T01:41:56.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.779+0000 7f725affd640 1 -- 172.21.15.88:0/1416838274 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725c014440 con 0x7f72640cf5c0 2024-09-16T01:41:56.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.779+0000 7f726c65e640 1 -- 172.21.15.88:0/1416838274 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7264154450 con 0x7f72640cf5c0 2024-09-16T01:41:56.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.780+0000 7f726c65e640 1 -- 172.21.15.88:0/1416838274 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7264073490 con 0x7f72640cf5c0 2024-09-16T01:41:56.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.780+0000 7f725affd640 1 -- 172.21.15.88:0/1416838274 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f725c01c070 con 0x7f72640cf5c0 2024-09-16T01:41:56.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.781+0000 7f725affd640 1 --2- 172.21.15.88:0/1416838274 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f724806c2b0 0x7f724806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:56.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.781+0000 7f726a3fa640 1 --2- 172.21.15.88:0/1416838274 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f724806c2b0 0x7f724806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:56.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.781+0000 7f725affd640 1 -- 172.21.15.88:0/1416838274 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 7028+0+0 (secure 0 0 0) 0x7f725c08b6d0 con 0x7f72640cf5c0 2024-09-16T01:41:56.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.781+0000 7f726a3fa640 1 --2- 172.21.15.88:0/1416838274 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f724806c2b0 0x7f724806e770 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f7260034000 tx=0x7f726002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:56.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.785+0000 7f725affd640 1 -- 172.21.15.88:0/1416838274 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f725c059910 con 0x7f72640cf5c0 2024-09-16T01:41:56.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:56.985+0000 7f726c65e640 1 -- 172.21.15.88:0/1416838274 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f726415af00 con 0x7f72640cf5c0 2024-09-16T01:41:57.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.344+0000 7f725affd640 1 -- 172.21.15.88:0/1416838274 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v162) ==== 162+0+0 (secure 0 0 0) 0x7f725c059210 con 0x7f72640cf5c0 2024-09-16T01:41:57.345 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:41:57.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.346+0000 7f726c65e640 1 -- 172.21.15.88:0/1416838274 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f724806c2b0 msgr2=0x7f724806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:57.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.346+0000 7f726c65e640 1 --2- 172.21.15.88:0/1416838274 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f724806c2b0 0x7f724806e770 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f7260034000 tx=0x7f726002e000 comp rx=0 tx=0).stop 2024-09-16T01:41:57.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.346+0000 7f726c65e640 1 -- 172.21.15.88:0/1416838274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72640cf5c0 msgr2=0x7f72641e35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:57.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.346+0000 7f726c65e640 1 --2- 172.21.15.88:0/1416838274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72640cf5c0 0x7f72641e35c0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f725c00b970 tx=0x7f725c00be30 comp rx=0 tx=0).stop 2024-09-16T01:41:57.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.346+0000 7f726c65e640 1 -- 172.21.15.88:0/1416838274 shutdown_connections 2024-09-16T01:41:57.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.346+0000 7f726c65e640 1 --2- 172.21.15.88:0/1416838274 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f724806c2b0 0x7f724806e770 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:57.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.346+0000 7f726c65e640 1 --2- 172.21.15.88:0/1416838274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72640cf5c0 0x7f72641e35c0 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:57.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.346+0000 7f726c65e640 1 --2- 172.21.15.88:0/1416838274 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72640cf180 0x7f7264153410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:57.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.346+0000 7f726c65e640 1 --2- 172.21.15.88:0/1416838274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72640ceda0 0x7f726414f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:57.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.346+0000 7f726c65e640 1 -- 172.21.15.88:0/1416838274 >> 172.21.15.88:0/1416838274 conn(0x7f726406cb50 msgr2=0x7f72640b4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:57.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.346+0000 7f726c65e640 1 -- 172.21.15.88:0/1416838274 shutdown_connections 2024-09-16T01:41:57.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.346+0000 7f726c65e640 1 -- 172.21.15.88:0/1416838274 wait complete. 2024-09-16T01:41:57.358 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:41:57.357+0000 7fba44961640 -1 mds.0.cache.ino(0x1) Corrupt inode 0x1: End of buffer [buffer:2] 2024-09-16T01:41:57.358 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:41:57.357+0000 7fba44961640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (22) Invalid argument 2024-09-16T01:41:57.358 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:41:57.357+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:41:57.358 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:41:57.357+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:41:57.360 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:57.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.457+0000 7f39f518d640 1 Processor -- start 2024-09-16T01:41:57.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.457+0000 7f39f518d640 1 -- start start 2024-09-16T01:41:57.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.457+0000 7f39f518d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f39f0062a70 0x7f39f00b52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:57.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.458+0000 7f39f518d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39f015e9b0 con 0x7f39f00b5800 2024-09-16T01:41:57.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.458+0000 7f39f518d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39f0162a50 con 0x7f39f015e5d0 2024-09-16T01:41:57.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.458+0000 7f39f518d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f39f0162d30 con 0x7f39f0062a70 2024-09-16T01:41:57.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.458+0000 7f39ef59e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f39f00b5800 0x7f39f0072e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:56918/0 (socket says 172.21.15.88:56918) 2024-09-16T01:41:57.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.458+0000 7f39ef59e640 1 -- 172.21.15.88:0/2462080524 learned_addr learned my addr 172.21.15.88:0/2462080524 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:57.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.458+0000 7f39ee59c640 1 -- 172.21.15.88:0/2462080524 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 317996324 0 0) 0x7f39f015e9b0 con 0x7f39f00b5800 2024-09-16T01:41:57.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.458+0000 7f39e659c640 1 --2- 172.21.15.88:0/2462080524 >> v2:172.21.15.154:3300/0 conn(0x7f39f0062a70 0x7f39f00b52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:57.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.458+0000 7f39ee59c640 1 -- 172.21.15.88:0/2462080524 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39dc003520 con 0x7f39f00b5800 2024-09-16T01:41:57.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.458+0000 7f39ee59c640 1 -- 172.21.15.88:0/2462080524 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3742410286 0 0) 0x7f39dc003520 con 0x7f39f00b5800 2024-09-16T01:41:57.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.458+0000 7f39ee59c640 1 -- 172.21.15.88:0/2462080524 >> v2:172.21.15.154:3300/0 conn(0x7f39f0062a70 msgr2=0x7f39f00b52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:57.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.458+0000 7f39ee59c640 1 --2- 172.21.15.88:0/2462080524 >> v2:172.21.15.154:3300/0 conn(0x7f39f0062a70 0x7f39f00b52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:57.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.458+0000 7f39ee59c640 1 -- 172.21.15.88:0/2462080524 >> v1:172.21.15.154:6789/0 conn(0x7f39f015e5d0 legacy=0x7f39f0162170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:57.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.458+0000 7f39ee59c640 1 -- 172.21.15.88:0/2462080524 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39f0163010 con 0x7f39f00b5800 2024-09-16T01:41:57.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.458+0000 7f39ee59c640 1 -- 172.21.15.88:0/2462080524 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f39e80019f0 con 0x7f39f00b5800 2024-09-16T01:41:57.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.458+0000 7f39ee59c640 1 -- 172.21.15.88:0/2462080524 >> v1:172.21.15.88:6789/0 conn(0x7f39f00b5800 legacy=0x7f39f0072e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:57.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.458+0000 7f39ee59c640 1 --2- 172.21.15.88:0/2462080524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39dc0039b0 0x7f39dc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:57.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.458+0000 7f39ee59c640 1 --2- 172.21.15.88:0/2462080524 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39dc004800 0x7f39dc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:57.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.458+0000 7f39ee59c640 1 --2- 172.21.15.88:0/2462080524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39dc0051c0 0x7f39dc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:57.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.458+0000 7f39ee59c640 1 -- 172.21.15.88:0/2462080524 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f39f0162d30 con 0x7f39dc0039b0 2024-09-16T01:41:57.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.458+0000 7f39ee59c640 1 -- 172.21.15.88:0/2462080524 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f39dc00c160 con 0x7f39dc0051c0 2024-09-16T01:41:57.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.458+0000 7f39ee59c640 1 -- 172.21.15.88:0/2462080524 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f39dc00c360 con 0x7f39dc004800 2024-09-16T01:41:57.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.458+0000 7f39eed9d640 1 --2- 172.21.15.88:0/2462080524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39dc0039b0 0x7f39dc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:57.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.459+0000 7f39ef59e640 1 --2- 172.21.15.88:0/2462080524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39dc0051c0 0x7f39dc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:57.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.459+0000 7f39e659c640 1 --2- 172.21.15.88:0/2462080524 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39dc004800 0x7f39dc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:57.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.459+0000 7f39eed9d640 1 -- 172.21.15.88:0/2462080524 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39dc004800 msgr2=0x7f39dc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:57.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.459+0000 7f39eed9d640 1 --2- 172.21.15.88:0/2462080524 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39dc004800 0x7f39dc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:57.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.459+0000 7f39eed9d640 1 -- 172.21.15.88:0/2462080524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39dc0051c0 msgr2=0x7f39dc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:57.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.459+0000 7f39eed9d640 1 --2- 172.21.15.88:0/2462080524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39dc0051c0 0x7f39dc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:57.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.459+0000 7f39eed9d640 1 -- 172.21.15.88:0/2462080524 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39dc003520 con 0x7f39dc0039b0 2024-09-16T01:41:57.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.459+0000 7f39ef59e640 1 --2- 172.21.15.88:0/2462080524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39dc0051c0 0x7f39dc00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:57.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.459+0000 7f39ef59e640 1 -- 172.21.15.88:0/2462080524 reap_dead start 2024-09-16T01:41:57.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.459+0000 7f39eed9d640 1 --2- 172.21.15.88:0/2462080524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39dc0039b0 0x7f39dc003dd0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f39e0001750 tx=0x7f39dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a13dc08b35637bc3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:57.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.459+0000 7f39ee59c640 1 -- 172.21.15.88:0/2462080524 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e0001370 con 0x7f39dc0039b0 2024-09-16T01:41:57.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.459+0000 7f39ee59c640 1 -- 172.21.15.88:0/2462080524 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39e00052d0 con 0x7f39dc0039b0 2024-09-16T01:41:57.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.459+0000 7f39ee59c640 1 -- 172.21.15.88:0/2462080524 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e002abc0 con 0x7f39dc0039b0 2024-09-16T01:41:57.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.460+0000 7f39f518d640 1 -- 172.21.15.88:0/2462080524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39dc0039b0 msgr2=0x7f39dc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:57.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.460+0000 7f39f518d640 1 --2- 172.21.15.88:0/2462080524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39dc0039b0 0x7f39dc003dd0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f39e0001750 tx=0x7f39dc005830 comp rx=0 tx=0).stop 2024-09-16T01:41:57.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.460+0000 7f39f518d640 1 -- 172.21.15.88:0/2462080524 shutdown_connections 2024-09-16T01:41:57.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.460+0000 7f39f518d640 1 --2- 172.21.15.88:0/2462080524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39dc0039b0 0x7f39dc003dd0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:57.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.460+0000 7f39f518d640 1 -- 172.21.15.88:0/2462080524 >> 172.21.15.88:0/2462080524 conn(0x7f39f006bc10 msgr2=0x7f39f006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:57.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.460+0000 7f39f518d640 1 -- 172.21.15.88:0/2462080524 shutdown_connections 2024-09-16T01:41:57.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.460+0000 7f39f518d640 1 -- 172.21.15.88:0/2462080524 wait complete. 2024-09-16T01:41:57.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.461+0000 7f39f518d640 1 Processor -- start 2024-09-16T01:41:57.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.462+0000 7f39f518d640 1 -- start start 2024-09-16T01:41:57.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.462+0000 7f39f518d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39dc0039b0 0x7f39f0159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:57.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.462+0000 7f39f518d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39f0145330 0x7f39f01589d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:57.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.462+0000 7f39f518d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39f0158f10 0x7f39f015fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:57.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.462+0000 7f39f518d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f39f00b52c0 con 0x7f39dc0039b0 2024-09-16T01:41:57.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.462+0000 7f39f518d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f39f0071b90 con 0x7f39f0145330 2024-09-16T01:41:57.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.462+0000 7f39f518d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f39f00b5440 con 0x7f39f0158f10 2024-09-16T01:41:57.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.462+0000 7f39eed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39dc0039b0 0x7f39f0159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:57.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.462+0000 7f39eed9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39dc0039b0 0x7f39f0159610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53886/0 (socket says 172.21.15.88:53886) 2024-09-16T01:41:57.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.462+0000 7f39eed9d640 1 -- 172.21.15.88:0/2733867818 learned_addr learned my addr 172.21.15.88:0/2733867818 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:57.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.463+0000 7f39e659c640 1 --2- 172.21.15.88:0/2733867818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39f0145330 0x7f39f01589d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:57.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.463+0000 7f39ef59e640 1 --2- 172.21.15.88:0/2733867818 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39f0158f10 0x7f39f015fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:57.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.463+0000 7f39eed9d640 1 -- 172.21.15.88:0/2733867818 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39f0158f10 msgr2=0x7f39f015fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:57.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.463+0000 7f39eed9d640 1 --2- 172.21.15.88:0/2733867818 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39f0158f10 0x7f39f015fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:57.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.463+0000 7f39eed9d640 1 -- 172.21.15.88:0/2733867818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39f0145330 msgr2=0x7f39f01589d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:57.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.463+0000 7f39eed9d640 1 --2- 172.21.15.88:0/2733867818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39f0145330 0x7f39f01589d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:57.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.463+0000 7f39eed9d640 1 -- 172.21.15.88:0/2733867818 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39f015ed60 con 0x7f39dc0039b0 2024-09-16T01:41:57.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.463+0000 7f39eed9d640 1 --2- 172.21.15.88:0/2733867818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39dc0039b0 0x7f39f0159610 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f39e0000c00 tx=0x7f39e0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:57.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.463+0000 7f39e7fff640 1 -- 172.21.15.88:0/2733867818 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e003f030 con 0x7f39dc0039b0 2024-09-16T01:41:57.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.463+0000 7f39e7fff640 1 -- 172.21.15.88:0/2733867818 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39e0005480 con 0x7f39dc0039b0 2024-09-16T01:41:57.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.463+0000 7f39f518d640 1 -- 172.21.15.88:0/2733867818 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39f015f040 con 0x7f39dc0039b0 2024-09-16T01:41:57.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.463+0000 7f39e7fff640 1 -- 172.21.15.88:0/2733867818 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e0037040 con 0x7f39dc0039b0 2024-09-16T01:41:57.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.463+0000 7f39f518d640 1 -- 172.21.15.88:0/2733867818 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39f015f600 con 0x7f39dc0039b0 2024-09-16T01:41:57.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.464+0000 7f39f518d640 1 -- 172.21.15.88:0/2733867818 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39f0162170 con 0x7f39dc0039b0 2024-09-16T01:41:57.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.465+0000 7f39e7fff640 1 -- 172.21.15.88:0/2733867818 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f39e0042070 con 0x7f39dc0039b0 2024-09-16T01:41:57.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.465+0000 7f39e7fff640 1 --2- 172.21.15.88:0/2733867818 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f39d006c2e0 0x7f39d006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:57.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.465+0000 7f39e7fff640 1 -- 172.21.15.88:0/2733867818 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7114+0+0 (secure 0 0 0) 0x7f39e00aaf10 con 0x7f39dc0039b0 2024-09-16T01:41:57.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.465+0000 7f39e659c640 1 --2- 172.21.15.88:0/2733867818 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f39d006c2e0 0x7f39d006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:57.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.465+0000 7f39e659c640 1 --2- 172.21.15.88:0/2733867818 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f39d006c2e0 0x7f39d006e7a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f39f00d06a0 tx=0x7f39d8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:57.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.469+0000 7f39e7fff640 1 -- 172.21.15.88:0/2733867818 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f39e0079070 con 0x7f39dc0039b0 2024-09-16T01:41:57.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.670+0000 7f39f518d640 1 -- 172.21.15.88:0/2733867818 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39f006f2f0 con 0x7f39dc0039b0 2024-09-16T01:41:57.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.670+0000 7f39e7fff640 1 -- 172.21.15.88:0/2733867818 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) ==== 95+0+4392 (secure 0 0 0) 0x7f39e0078970 con 0x7f39dc0039b0 2024-09-16T01:41:57.672 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:57.672 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":164,"btime":"2024-09-16T01:41:57:358504+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/3163343083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":3163343083},{"type":"v1","addr":"172.21.15.88:6837","nonce":3163343083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:57.358449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":103,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:41:57.673 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 164 2024-09-16T01:41:57.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.672+0000 7f39f518d640 1 -- 172.21.15.88:0/2733867818 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f39d006c2e0 msgr2=0x7f39d006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:57.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.672+0000 7f39f518d640 1 --2- 172.21.15.88:0/2733867818 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f39d006c2e0 0x7f39d006e7a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f39f00d06a0 tx=0x7f39d8008040 comp rx=0 tx=0).stop 2024-09-16T01:41:57.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.672+0000 7f39f518d640 1 -- 172.21.15.88:0/2733867818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39dc0039b0 msgr2=0x7f39f0159610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:57.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.672+0000 7f39f518d640 1 --2- 172.21.15.88:0/2733867818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39dc0039b0 0x7f39f0159610 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f39e0000c00 tx=0x7f39e0001120 comp rx=0 tx=0).stop 2024-09-16T01:41:57.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.672+0000 7f39f518d640 1 -- 172.21.15.88:0/2733867818 shutdown_connections 2024-09-16T01:41:57.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.672+0000 7f39f518d640 1 --2- 172.21.15.88:0/2733867818 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f39d006c2e0 0x7f39d006e7a0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:57.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.672+0000 7f39f518d640 1 --2- 172.21.15.88:0/2733867818 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39f0158f10 0x7f39f015fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:57.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.672+0000 7f39f518d640 1 --2- 172.21.15.88:0/2733867818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39f0145330 0x7f39f01589d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:57.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.672+0000 7f39f518d640 1 --2- 172.21.15.88:0/2733867818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39dc0039b0 0x7f39f0159610 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:57.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.672+0000 7f39f518d640 1 -- 172.21.15.88:0/2733867818 >> 172.21.15.88:0/2733867818 conn(0x7f39f006bc10 msgr2=0x7f39f015d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:57.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.672+0000 7f39f518d640 1 -- 172.21.15.88:0/2733867818 shutdown_connections 2024-09-16T01:41:57.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.672+0000 7f39f518d640 1 -- 172.21.15.88:0/2733867818 wait complete. 2024-09-16T01:41:57.685 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:41:57.685 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:57.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.782+0000 7faf03ec9640 1 Processor -- start 2024-09-16T01:41:57.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.782+0000 7faf03ec9640 1 -- start start 2024-09-16T01:41:57.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.783+0000 7faf03ec9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faefc15e560 0x7faefc06e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:57.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.783+0000 7faf03ec9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faefc073130 0x7faefc0735c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:57.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.783+0000 7faf03ec9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faefc0b5220 con 0x7faefc072220 2024-09-16T01:41:57.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.783+0000 7faf03ec9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7faefc0aefa0 con 0x7faefc073130 2024-09-16T01:41:57.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.783+0000 7faf03ec9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7faefc0b54c0 con 0x7faefc15e560 2024-09-16T01:41:57.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.783+0000 7faf02466640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faefc073130 0x7faefc0735c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:57.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.783+0000 7faf02466640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7faefc073130 0x7faefc0735c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53898/0 (socket says 172.21.15.88:53898) 2024-09-16T01:41:57.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.783+0000 7faf01464640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7faefc072220 0x7faefc0ae890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:56932/0 (socket says 172.21.15.88:56932) 2024-09-16T01:41:57.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.783+0000 7faf02466640 1 -- 172.21.15.88:0/3744850674 learned_addr learned my addr 172.21.15.88:0/3744850674 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:57.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.783+0000 7faf01c65640 1 --2- 172.21.15.88:0/3744850674 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faefc15e560 0x7faefc06e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:57.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.783+0000 7faf02466640 1 -- 172.21.15.88:0/3744850674 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faefc15e560 msgr2=0x7faefc06e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:57.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.783+0000 7faf02466640 1 --2- 172.21.15.88:0/3744850674 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faefc15e560 0x7faefc06e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:57.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.783+0000 7faf02466640 1 -- 172.21.15.88:0/3744850674 >> v1:172.21.15.88:6789/0 conn(0x7faefc072220 legacy=0x7faefc0ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:57.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.783+0000 7faf02466640 1 -- 172.21.15.88:0/3744850674 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faefc0b57a0 con 0x7faefc073130 2024-09-16T01:41:57.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.783+0000 7faf00c63640 1 -- 172.21.15.88:0/3744850674 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2368802674 0 0) 0x7faefc0b5220 con 0x7faefc072220 2024-09-16T01:41:57.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.784+0000 7faf02466640 1 --2- 172.21.15.88:0/3744850674 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faefc073130 0x7faefc0735c0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7faef4009e50 tx=0x7faef402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d26b21d9a55eac3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:57.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.784+0000 7faf00c63640 1 -- 172.21.15.88:0/3744850674 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef4036070 con 0x7faefc073130 2024-09-16T01:41:57.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.784+0000 7faf00c63640 1 -- 172.21.15.88:0/3744850674 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faef4031600 con 0x7faefc073130 2024-09-16T01:41:57.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.784+0000 7faf00c63640 1 -- 172.21.15.88:0/3744850674 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef40319d0 con 0x7faefc073130 2024-09-16T01:41:57.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.785+0000 7faf03ec9640 1 -- 172.21.15.88:0/3744850674 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faefc073130 msgr2=0x7faefc0735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:57.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.785+0000 7faf03ec9640 1 --2- 172.21.15.88:0/3744850674 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faefc073130 0x7faefc0735c0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7faef4009e50 tx=0x7faef402a710 comp rx=0 tx=0).stop 2024-09-16T01:41:57.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.785+0000 7faf03ec9640 1 -- 172.21.15.88:0/3744850674 shutdown_connections 2024-09-16T01:41:57.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.785+0000 7faf03ec9640 1 --2- 172.21.15.88:0/3744850674 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faefc073130 0x7faefc0735c0 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:57.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.785+0000 7faf03ec9640 1 --2- 172.21.15.88:0/3744850674 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faefc15e560 0x7faefc06e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:57.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.785+0000 7faf03ec9640 1 -- 172.21.15.88:0/3744850674 >> 172.21.15.88:0/3744850674 conn(0x7faefc06bc10 msgr2=0x7faefc06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:57.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.785+0000 7faf03ec9640 1 -- 172.21.15.88:0/3744850674 shutdown_connections 2024-09-16T01:41:57.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.785+0000 7faf03ec9640 1 -- 172.21.15.88:0/3744850674 wait complete. 2024-09-16T01:41:57.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.786+0000 7faf03ec9640 1 Processor -- start 2024-09-16T01:41:57.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.786+0000 7faf03ec9640 1 -- start start 2024-09-16T01:41:57.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.787+0000 7faf03ec9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faefc072220 0x7faefc157cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:57.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.787+0000 7faf03ec9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faefc073130 0x7faefc150c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:57.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.787+0000 7faf03ec9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faefc15e560 0x7faefc151170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:57.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.787+0000 7faf03ec9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7faefc072660 con 0x7faefc072220 2024-09-16T01:41:57.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.787+0000 7faf03ec9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faefc0b5d60 con 0x7faefc15e560 2024-09-16T01:41:57.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.787+0000 7faf03ec9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7faefc15e940 con 0x7faefc073130 2024-09-16T01:41:57.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.787+0000 7faf01c65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faefc072220 0x7faefc157cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:57.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.787+0000 7faf01c65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faefc072220 0x7faefc157cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53906/0 (socket says 172.21.15.88:53906) 2024-09-16T01:41:57.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.787+0000 7faf01c65640 1 -- 172.21.15.88:0/166722157 learned_addr learned my addr 172.21.15.88:0/166722157 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:57.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.787+0000 7faf02466640 1 --2- 172.21.15.88:0/166722157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faefc15e560 0x7faefc151170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:57.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.787+0000 7faf01c65640 1 -- 172.21.15.88:0/166722157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faefc073130 msgr2=0x7faefc150c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:57.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.787+0000 7faf01c65640 1 --2- 172.21.15.88:0/166722157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faefc073130 0x7faefc150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:57.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.787+0000 7faf01c65640 1 -- 172.21.15.88:0/166722157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faefc15e560 msgr2=0x7faefc151170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:57.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.787+0000 7faf01c65640 1 --2- 172.21.15.88:0/166722157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faefc15e560 0x7faefc151170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:57.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.787+0000 7faf01c65640 1 -- 172.21.15.88:0/166722157 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faefc0cf300 con 0x7faefc072220 2024-09-16T01:41:57.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.787+0000 7faf02466640 1 --2- 172.21.15.88:0/166722157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faefc15e560 0x7faefc151170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:41:57.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.787+0000 7faf01c65640 1 --2- 172.21.15.88:0/166722157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faefc072220 0x7faefc157cf0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7faef800d5f0 tx=0x7faef800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:57.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.787+0000 7faef27fc640 1 -- 172.21.15.88:0/166722157 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef80125f0 con 0x7faefc072220 2024-09-16T01:41:57.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.788+0000 7faef27fc640 1 -- 172.21.15.88:0/166722157 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faef8019070 con 0x7faefc072220 2024-09-16T01:41:57.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.788+0000 7faf03ec9640 1 -- 172.21.15.88:0/166722157 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faefc0cf5e0 con 0x7faefc072220 2024-09-16T01:41:57.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.788+0000 7faef27fc640 1 -- 172.21.15.88:0/166722157 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef80129d0 con 0x7faefc072220 2024-09-16T01:41:57.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.788+0000 7faf03ec9640 1 -- 172.21.15.88:0/166722157 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faefc0cf000 con 0x7faefc072220 2024-09-16T01:41:57.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.789+0000 7faf03ec9640 1 -- 172.21.15.88:0/166722157 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faefc06e780 con 0x7faefc072220 2024-09-16T01:41:57.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.789+0000 7faef27fc640 1 -- 172.21.15.88:0/166722157 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7faef8012b70 con 0x7faefc072220 2024-09-16T01:41:57.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.790+0000 7faef27fc640 1 --2- 172.21.15.88:0/166722157 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faee006c2b0 0x7faee006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:57.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.790+0000 7faf01464640 1 --2- 172.21.15.88:0/166722157 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faee006c2b0 0x7faee006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:57.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.790+0000 7faef27fc640 1 -- 172.21.15.88:0/166722157 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7114+0+0 (secure 0 0 0) 0x7faef808bd30 con 0x7faefc072220 2024-09-16T01:41:57.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.790+0000 7faf01464640 1 --2- 172.21.15.88:0/166722157 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faee006c2b0 0x7faee006e770 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7faeec007fd0 tx=0x7faeec007920 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:57.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.794+0000 7faef27fc640 1 -- 172.21.15.88:0/166722157 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7faef8059fb0 con 0x7faefc072220 2024-09-16T01:41:57.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.996+0000 7faf03ec9640 1 -- 172.21.15.88:0/166722157 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faefc0b1d90 con 0x7faefc072220 2024-09-16T01:41:57.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.996+0000 7faef27fc640 1 -- 172.21.15.88:0/166722157 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) ==== 95+0+4392 (secure 0 0 0) 0x7faef80598b0 con 0x7faefc072220 2024-09-16T01:41:57.998 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:57.998 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":164,"btime":"2024-09-16T01:41:57:358504+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/3163343083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":3163343083},{"type":"v1","addr":"172.21.15.88:6837","nonce":3163343083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:57.358449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":103,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:41:57.998 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 164 2024-09-16T01:41:58.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.998+0000 7faf03ec9640 1 -- 172.21.15.88:0/166722157 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faee006c2b0 msgr2=0x7faee006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:58.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.998+0000 7faf03ec9640 1 --2- 172.21.15.88:0/166722157 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faee006c2b0 0x7faee006e770 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7faeec007fd0 tx=0x7faeec007920 comp rx=0 tx=0).stop 2024-09-16T01:41:58.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.998+0000 7faf03ec9640 1 -- 172.21.15.88:0/166722157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faefc072220 msgr2=0x7faefc157cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:58.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.998+0000 7faf03ec9640 1 --2- 172.21.15.88:0/166722157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faefc072220 0x7faefc157cf0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7faef800d5f0 tx=0x7faef800dab0 comp rx=0 tx=0).stop 2024-09-16T01:41:58.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.998+0000 7faf03ec9640 1 -- 172.21.15.88:0/166722157 shutdown_connections 2024-09-16T01:41:58.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.998+0000 7faf03ec9640 1 --2- 172.21.15.88:0/166722157 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faee006c2b0 0x7faee006e770 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:58.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.998+0000 7faf03ec9640 1 --2- 172.21.15.88:0/166722157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faefc15e560 0x7faefc151170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:58.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.998+0000 7faf03ec9640 1 --2- 172.21.15.88:0/166722157 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faefc073130 0x7faefc150c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:58.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.998+0000 7faf03ec9640 1 --2- 172.21.15.88:0/166722157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faefc072220 0x7faefc157cf0 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:58.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.998+0000 7faf03ec9640 1 -- 172.21.15.88:0/166722157 >> 172.21.15.88:0/166722157 conn(0x7faefc06bc10 msgr2=0x7faefc15d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:58.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.998+0000 7faf03ec9640 1 -- 172.21.15.88:0/166722157 shutdown_connections 2024-09-16T01:41:58.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:57.998+0000 7faf03ec9640 1 -- 172.21.15.88:0/166722157 wait complete. 2024-09-16T01:41:58.010 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:41:58.010 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:41:58.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.106+0000 7f2ff884c640 1 Processor -- start 2024-09-16T01:41:58.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.106+0000 7f2ff884c640 1 -- start start 2024-09-16T01:41:58.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.107+0000 7f2ff884c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2ff415e660 0x7f2ff40cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:58.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.107+0000 7f2ff884c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2ff40d3310 0x7f2ff40d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:58.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.107+0000 7f2ff884c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ff415ea40 con 0x7f2ff40d1650 2024-09-16T01:41:58.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.107+0000 7f2ff884c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2ff40d1ac0 con 0x7f2ff415e660 2024-09-16T01:41:58.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.107+0000 7f2ff884c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2ff40d3870 con 0x7f2ff40d3310 2024-09-16T01:41:58.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.108+0000 7f2ff259c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2ff415e660 0x7f2ff40cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:58.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.108+0000 7f2ff2d9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f2ff40d1650 0x7f2ff406bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:56946/0 (socket says 172.21.15.88:56946) 2024-09-16T01:41:58.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.108+0000 7f2ff259c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2ff415e660 0x7f2ff40cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53908/0 (socket says 172.21.15.88:53908) 2024-09-16T01:41:58.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.108+0000 7f2ff2d9d640 1 -- 172.21.15.88:0/2171314905 learned_addr learned my addr 172.21.15.88:0/2171314905 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:58.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.108+0000 7f2ff1d9b640 1 --2- 172.21.15.88:0/2171314905 >> v2:172.21.15.154:3300/0 conn(0x7f2ff40d3310 0x7f2ff40d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:58.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.108+0000 7f2ff259c640 1 -- 172.21.15.88:0/2171314905 >> v2:172.21.15.154:3300/0 conn(0x7f2ff40d3310 msgr2=0x7f2ff40d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:58.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.108+0000 7f2ff259c640 1 --2- 172.21.15.88:0/2171314905 >> v2:172.21.15.154:3300/0 conn(0x7f2ff40d3310 0x7f2ff40d1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:58.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.108+0000 7f2ff259c640 1 -- 172.21.15.88:0/2171314905 >> v1:172.21.15.88:6789/0 conn(0x7f2ff40d1650 legacy=0x7f2ff406bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:58.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.108+0000 7f2ff259c640 1 -- 172.21.15.88:0/2171314905 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ff40d3b50 con 0x7f2ff415e660 2024-09-16T01:41:58.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.108+0000 7f2ff159a640 1 -- 172.21.15.88:0/2171314905 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2003159102 0 0) 0x7f2ff415ea40 con 0x7f2ff40d1650 2024-09-16T01:41:58.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.109+0000 7f2ff259c640 1 --2- 172.21.15.88:0/2171314905 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ff415e660 0x7f2ff40cf730 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40c8020 tx=0x7f2fe402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad26cecf690eb548 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:58.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.109+0000 7f2ff159a640 1 -- 172.21.15.88:0/2171314905 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe4037070 con 0x7f2ff415e660 2024-09-16T01:41:58.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.109+0000 7f2ff159a640 1 -- 172.21.15.88:0/2171314905 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fe4032440 con 0x7f2ff415e660 2024-09-16T01:41:58.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.109+0000 7f2ff159a640 1 -- 172.21.15.88:0/2171314905 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe4032810 con 0x7f2ff415e660 2024-09-16T01:41:58.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.109+0000 7f2ff884c640 1 -- 172.21.15.88:0/2171314905 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ff415e660 msgr2=0x7f2ff40cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:58.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.109+0000 7f2ff884c640 1 --2- 172.21.15.88:0/2171314905 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ff415e660 0x7f2ff40cf730 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40c8020 tx=0x7f2fe402a1a0 comp rx=0 tx=0).stop 2024-09-16T01:41:58.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.109+0000 7f2ff884c640 1 -- 172.21.15.88:0/2171314905 shutdown_connections 2024-09-16T01:41:58.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.109+0000 7f2ff884c640 1 --2- 172.21.15.88:0/2171314905 >> v2:172.21.15.154:3300/0 conn(0x7f2ff40d3310 0x7f2ff40d1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:58.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.109+0000 7f2ff884c640 1 --2- 172.21.15.88:0/2171314905 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ff415e660 0x7f2ff40cf730 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:58.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.109+0000 7f2ff884c640 1 -- 172.21.15.88:0/2171314905 >> 172.21.15.88:0/2171314905 conn(0x7f2ff40b6410 msgr2=0x7f2ff40b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:58.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.109+0000 7f2ff884c640 1 -- 172.21.15.88:0/2171314905 shutdown_connections 2024-09-16T01:41:58.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.109+0000 7f2ff884c640 1 -- 172.21.15.88:0/2171314905 wait complete. 2024-09-16T01:41:58.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.111+0000 7f2ff884c640 1 Processor -- start 2024-09-16T01:41:58.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.111+0000 7f2ff884c640 1 -- start start 2024-09-16T01:41:58.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.111+0000 7f2ff884c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ff40d1650 0x7f2ff41f4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:58.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.111+0000 7f2ff884c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ff40d3310 0x7f2ff41f4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:58.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.111+0000 7f2ff884c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ff415e660 0x7f2ff41515b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:58.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.111+0000 7f2ff884c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff40d43f0 con 0x7f2ff40d1650 2024-09-16T01:41:58.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.111+0000 7f2ff884c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff40d4110 con 0x7f2ff40d3310 2024-09-16T01:41:58.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.111+0000 7f2ff884c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2ff40cfc70 con 0x7f2ff415e660 2024-09-16T01:41:58.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.112+0000 7f2ff259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ff40d1650 0x7f2ff41f4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:58.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.112+0000 7f2ff259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ff40d1650 0x7f2ff41f4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53924/0 (socket says 172.21.15.88:53924) 2024-09-16T01:41:58.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.112+0000 7f2ff259c640 1 -- 172.21.15.88:0/1768130111 learned_addr learned my addr 172.21.15.88:0/1768130111 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:58.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.112+0000 7f2ff1d9b640 1 --2- 172.21.15.88:0/1768130111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ff40d3310 0x7f2ff41f4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:58.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.112+0000 7f2ff2d9d640 1 --2- 172.21.15.88:0/1768130111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ff415e660 0x7f2ff41515b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:58.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.112+0000 7f2ff259c640 1 -- 172.21.15.88:0/1768130111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ff415e660 msgr2=0x7f2ff41515b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:58.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.112+0000 7f2ff259c640 1 --2- 172.21.15.88:0/1768130111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ff415e660 0x7f2ff41515b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:58.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.112+0000 7f2ff259c640 1 -- 172.21.15.88:0/1768130111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ff40d3310 msgr2=0x7f2ff41f4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:58.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.112+0000 7f2ff259c640 1 --2- 172.21.15.88:0/1768130111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ff40d3310 0x7f2ff41f4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:58.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.112+0000 7f2ff259c640 1 -- 172.21.15.88:0/1768130111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ff4158990 con 0x7f2ff40d1650 2024-09-16T01:41:58.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.112+0000 7f2ff259c640 1 --2- 172.21.15.88:0/1768130111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ff40d1650 0x7f2ff41f4510 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f2fe4000c00 tx=0x7f2fe40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:58.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.112+0000 7f2fe2ffd640 1 -- 172.21.15.88:0/1768130111 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe4037050 con 0x7f2ff40d1650 2024-09-16T01:41:58.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.112+0000 7f2fe2ffd640 1 -- 172.21.15.88:0/1768130111 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fe4038040 con 0x7f2ff40d1650 2024-09-16T01:41:58.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.112+0000 7f2ff884c640 1 -- 172.21.15.88:0/1768130111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ff4158c70 con 0x7f2ff40d1650 2024-09-16T01:41:58.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.112+0000 7f2fe2ffd640 1 -- 172.21.15.88:0/1768130111 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe4006e10 con 0x7f2ff40d1650 2024-09-16T01:41:58.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.112+0000 7f2ff884c640 1 -- 172.21.15.88:0/1768130111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ff415ff80 con 0x7f2ff40d1650 2024-09-16T01:41:58.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.113+0000 7f2ff884c640 1 -- 172.21.15.88:0/1768130111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ff415fa90 con 0x7f2ff40d1650 2024-09-16T01:41:58.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.114+0000 7f2fe2ffd640 1 -- 172.21.15.88:0/1768130111 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2fe4030980 con 0x7f2ff40d1650 2024-09-16T01:41:58.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.114+0000 7f2fe2ffd640 1 --2- 172.21.15.88:0/1768130111 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2fd406c2b0 0x7f2fd406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:58.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.114+0000 7f2fe2ffd640 1 -- 172.21.15.88:0/1768130111 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7114+0+0 (secure 0 0 0) 0x7f2fe40ac330 con 0x7f2ff40d1650 2024-09-16T01:41:58.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.114+0000 7f2ff1d9b640 1 --2- 172.21.15.88:0/1768130111 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2fd406c2b0 0x7f2fd406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:58.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.115+0000 7f2ff1d9b640 1 --2- 172.21.15.88:0/1768130111 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2fd406c2b0 0x7f2fd406e770 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40cfb80 tx=0x7f2fe8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:58.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.118+0000 7f2fe2ffd640 1 -- 172.21.15.88:0/1768130111 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2fe4079f60 con 0x7f2ff40d1650 2024-09-16T01:41:58.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.321+0000 7f2ff884c640 1 -- 172.21.15.88:0/1768130111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ff40d1110 con 0x7f2ff40d1650 2024-09-16T01:41:58.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.322+0000 7f2fe2ffd640 1 -- 172.21.15.88:0/1768130111 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) ==== 95+0+4392 (secure 0 0 0) 0x7f2fe407e260 con 0x7f2ff40d1650 2024-09-16T01:41:58.323 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:41:58.324 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":164,"btime":"2024-09-16T01:41:57:358504+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/3163343083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":3163343083},{"type":"v1","addr":"172.21.15.88:6837","nonce":3163343083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":156}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:41:57.358449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":103,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:41:58.324 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 164 2024-09-16T01:41:58.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.324+0000 7f2ff884c640 1 -- 172.21.15.88:0/1768130111 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2fd406c2b0 msgr2=0x7f2fd406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:58.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.324+0000 7f2ff884c640 1 --2- 172.21.15.88:0/1768130111 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2fd406c2b0 0x7f2fd406e770 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40cfb80 tx=0x7f2fe8008040 comp rx=0 tx=0).stop 2024-09-16T01:41:58.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.324+0000 7f2ff884c640 1 -- 172.21.15.88:0/1768130111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ff40d1650 msgr2=0x7f2ff41f4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:58.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.324+0000 7f2ff884c640 1 --2- 172.21.15.88:0/1768130111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ff40d1650 0x7f2ff41f4510 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f2fe4000c00 tx=0x7f2fe40046c0 comp rx=0 tx=0).stop 2024-09-16T01:41:58.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.324+0000 7f2ff884c640 1 -- 172.21.15.88:0/1768130111 shutdown_connections 2024-09-16T01:41:58.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.324+0000 7f2ff884c640 1 --2- 172.21.15.88:0/1768130111 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2fd406c2b0 0x7f2fd406e770 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:58.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.324+0000 7f2ff884c640 1 --2- 172.21.15.88:0/1768130111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ff415e660 0x7f2ff41515b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:58.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.324+0000 7f2ff884c640 1 --2- 172.21.15.88:0/1768130111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ff40d3310 0x7f2ff41f4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:58.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.324+0000 7f2ff884c640 1 --2- 172.21.15.88:0/1768130111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2ff40d1650 0x7f2ff41f4510 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:58.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.324+0000 7f2ff884c640 1 -- 172.21.15.88:0/1768130111 >> 172.21.15.88:0/1768130111 conn(0x7f2ff40b6410 msgr2=0x7f2ff415b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:58.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.324+0000 7f2ff884c640 1 -- 172.21.15.88:0/1768130111 shutdown_connections 2024-09-16T01:41:58.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.324+0000 7f2ff884c640 1 -- 172.21.15.88:0/1768130111 wait complete. 2024-09-16T01:41:58.336 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt 1.00000000.inode' led to DAMAGED state 2024-09-16T01:41:58.336 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt 200.00000000' 2024-09-16T01:41:58.337 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:41:58.337 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:41:58.352 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:41:58.352 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:41:58.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:41:58.353 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:41:58.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:41:58.408 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:41:58.409 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:41:58.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.548+0000 7f6628d9d640 1 Processor -- start 2024-09-16T01:41:58.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.548+0000 7f6628d9d640 1 -- start start 2024-09-16T01:41:58.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.549+0000 7f6628d9d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f66240b5750 0x7f66240ae890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:58.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.549+0000 7f6628d9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6624071d60 0x7f66240aedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:58.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.549+0000 7f6628d9d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66240b02e0 con 0x7f662415e570 2024-09-16T01:41:58.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.549+0000 7f6628d9d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f66240b0b80 con 0x7f66240b5750 2024-09-16T01:41:58.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.549+0000 7f6628d9d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f66240b42b0 con 0x7f6624071d60 2024-09-16T01:41:58.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.549+0000 7f6623fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6624071d60 0x7f66240aedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:58.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.549+0000 7f6622ffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f66240b5750 0x7f66240ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:58.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.549+0000 7f6622ffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f66240b5750 0x7f66240ae890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:42428/0 (socket says 172.21.15.88:42428) 2024-09-16T01:41:58.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.549+0000 7f6622ffd640 1 -- 172.21.15.88:0/1708084047 learned_addr learned my addr 172.21.15.88:0/1708084047 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:58.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.549+0000 7f6623fff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6624071d60 0x7f66240aedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:37514/0 (socket says 172.21.15.88:37514) 2024-09-16T01:41:58.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.549+0000 7f66237fe640 1 --1- 172.21.15.88:0/1708084047 >> v1:172.21.15.154:6789/0 conn(0x7f662415e570 0x7f66240afab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:59180/0 (socket says 172.21.15.88:59180) 2024-09-16T01:41:58.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.550+0000 7f6622ffd640 1 -- 172.21.15.88:0/1708084047 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6624071d60 msgr2=0x7f66240aedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:58.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.550+0000 7f6622ffd640 1 --2- 172.21.15.88:0/1708084047 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6624071d60 0x7f66240aedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:58.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.550+0000 7f6622ffd640 1 -- 172.21.15.88:0/1708084047 >> v1:172.21.15.154:6789/0 conn(0x7f662415e570 legacy=0x7f66240afab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:58.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.550+0000 7f6622ffd640 1 -- 172.21.15.88:0/1708084047 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66240b05c0 con 0x7f66240b5750 2024-09-16T01:41:58.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.550+0000 7f6622ffd640 1 --2- 172.21.15.88:0/1708084047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66240b5750 0x7f66240ae890 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6618009870 tx=0x7f661802a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3cfab17de97d4a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:58.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.551+0000 7f66227fc640 1 -- 172.21.15.88:0/1708084047 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6618037070 con 0x7f66240b5750 2024-09-16T01:41:58.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.551+0000 7f66227fc640 1 -- 172.21.15.88:0/1708084047 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6618032440 con 0x7f66240b5750 2024-09-16T01:41:58.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.551+0000 7f66227fc640 1 -- 172.21.15.88:0/1708084047 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6618032810 con 0x7f66240b5750 2024-09-16T01:41:58.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.551+0000 7f6628d9d640 1 -- 172.21.15.88:0/1708084047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66240b5750 msgr2=0x7f66240ae890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:58.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.551+0000 7f6628d9d640 1 --2- 172.21.15.88:0/1708084047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66240b5750 0x7f66240ae890 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6618009870 tx=0x7f661802a1c0 comp rx=0 tx=0).stop 2024-09-16T01:41:58.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.551+0000 7f6628d9d640 1 -- 172.21.15.88:0/1708084047 shutdown_connections 2024-09-16T01:41:58.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.551+0000 7f6628d9d640 1 --2- 172.21.15.88:0/1708084047 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6624071d60 0x7f66240aedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:58.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.551+0000 7f6628d9d640 1 --2- 172.21.15.88:0/1708084047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66240b5750 0x7f66240ae890 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:58.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.551+0000 7f6628d9d640 1 -- 172.21.15.88:0/1708084047 >> 172.21.15.88:0/1708084047 conn(0x7f66240cdc00 msgr2=0x7f66240ce010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:58.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.551+0000 7f6628d9d640 1 -- 172.21.15.88:0/1708084047 shutdown_connections 2024-09-16T01:41:58.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.551+0000 7f6628d9d640 1 -- 172.21.15.88:0/1708084047 wait complete. 2024-09-16T01:41:58.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.552+0000 7f6628d9d640 1 Processor -- start 2024-09-16T01:41:58.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.552+0000 7f6628d9d640 1 -- start start 2024-09-16T01:41:58.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.553+0000 7f6628d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6624071d60 0x7f662414e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:58.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.553+0000 7f6628d9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66240b5750 0x7f66241510a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:58.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.553+0000 7f6628d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f662415e570 0x7f662406cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:58.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.553+0000 7f6628d9d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6624168150 con 0x7f6624071d60 2024-09-16T01:41:58.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.553+0000 7f6628d9d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f662415e950 con 0x7f662415e570 2024-09-16T01:41:58.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.553+0000 7f6628d9d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f66241682d0 con 0x7f66240b5750 2024-09-16T01:41:58.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.553+0000 7f66237fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6624071d60 0x7f662414e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:58.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.553+0000 7f66237fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6624071d60 0x7f662414e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53926/0 (socket says 172.21.15.88:53926) 2024-09-16T01:41:58.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.553+0000 7f66237fe640 1 -- 172.21.15.88:0/3012522944 learned_addr learned my addr 172.21.15.88:0/3012522944 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:58.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.553+0000 7f66237fe640 1 -- 172.21.15.88:0/3012522944 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66240b5750 msgr2=0x7f66241510a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:58.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.553+0000 7f66237fe640 1 --2- 172.21.15.88:0/3012522944 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66240b5750 0x7f66241510a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:58.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.553+0000 7f66237fe640 1 -- 172.21.15.88:0/3012522944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f662415e570 msgr2=0x7f662406cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:58.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.553+0000 7f66237fe640 1 --2- 172.21.15.88:0/3012522944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f662415e570 0x7f662406cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:58.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.553+0000 7f66237fe640 1 -- 172.21.15.88:0/3012522944 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6624155590 con 0x7f6624071d60 2024-09-16T01:41:58.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.554+0000 7f66237fe640 1 --2- 172.21.15.88:0/3012522944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6624071d60 0x7f662414e590 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f661400c3c0 tx=0x7f661400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:58.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.554+0000 7f660bfff640 1 -- 172.21.15.88:0/3012522944 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66140074e0 con 0x7f6624071d60 2024-09-16T01:41:58.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.554+0000 7f660bfff640 1 -- 172.21.15.88:0/3012522944 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6614007680 con 0x7f6624071d60 2024-09-16T01:41:58.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.554+0000 7f660bfff640 1 -- 172.21.15.88:0/3012522944 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6614004a00 con 0x7f6624071d60 2024-09-16T01:41:58.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.554+0000 7f6628d9d640 1 -- 172.21.15.88:0/3012522944 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6624155870 con 0x7f6624071d60 2024-09-16T01:41:58.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.554+0000 7f6628d9d640 1 -- 172.21.15.88:0/3012522944 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6624155e30 con 0x7f6624071d60 2024-09-16T01:41:58.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.555+0000 7f6628d9d640 1 -- 172.21.15.88:0/3012522944 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66240afab0 con 0x7f6624071d60 2024-09-16T01:41:58.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.555+0000 7f660bfff640 1 -- 172.21.15.88:0/3012522944 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f661401a020 con 0x7f6624071d60 2024-09-16T01:41:58.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.556+0000 7f660bfff640 1 --2- 172.21.15.88:0/3012522944 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f661006c2b0 0x7f661006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:58.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.556+0000 7f660bfff640 1 -- 172.21.15.88:0/3012522944 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7114+0+0 (secure 0 0 0) 0x7f661408b0f0 con 0x7f6624071d60 2024-09-16T01:41:58.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.556+0000 7f6622ffd640 1 --2- 172.21.15.88:0/3012522944 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f661006c2b0 0x7f661006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:58.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.556+0000 7f6622ffd640 1 --2- 172.21.15.88:0/3012522944 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f661006c2b0 0x7f661006e770 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f6618034000 tx=0x7f661802c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:58.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.560+0000 7f660bfff640 1 -- 172.21.15.88:0/3012522944 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f66140592e0 con 0x7f6624071d60 2024-09-16T01:41:58.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:58.760+0000 7f6628d9d640 1 -- 172.21.15.88:0/3012522944 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f6624157f90 con 0x7f6624071d60 2024-09-16T01:41:59.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.353+0000 7f660bfff640 1 -- 172.21.15.88:0/3012522944 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v165) ==== 161+0+0 (secure 0 0 0) 0x7f6614058be0 con 0x7f6624071d60 2024-09-16T01:41:59.354 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:41:59.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.355+0000 7f6628d9d640 1 -- 172.21.15.88:0/3012522944 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f661006c2b0 msgr2=0x7f661006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:59.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.355+0000 7f6628d9d640 1 --2- 172.21.15.88:0/3012522944 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f661006c2b0 0x7f661006e770 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f6618034000 tx=0x7f661802c000 comp rx=0 tx=0).stop 2024-09-16T01:41:59.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.355+0000 7f6628d9d640 1 -- 172.21.15.88:0/3012522944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6624071d60 msgr2=0x7f662414e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:59.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.355+0000 7f6628d9d640 1 --2- 172.21.15.88:0/3012522944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6624071d60 0x7f662414e590 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f661400c3c0 tx=0x7f661400c880 comp rx=0 tx=0).stop 2024-09-16T01:41:59.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.355+0000 7f6628d9d640 1 -- 172.21.15.88:0/3012522944 shutdown_connections 2024-09-16T01:41:59.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.355+0000 7f6628d9d640 1 --2- 172.21.15.88:0/3012522944 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f661006c2b0 0x7f661006e770 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:59.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.355+0000 7f6628d9d640 1 --2- 172.21.15.88:0/3012522944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f662415e570 0x7f662406cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:59.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.355+0000 7f6628d9d640 1 --2- 172.21.15.88:0/3012522944 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66240b5750 0x7f66241510a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:59.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.355+0000 7f6628d9d640 1 --2- 172.21.15.88:0/3012522944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6624071d60 0x7f662414e590 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:59.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.355+0000 7f6628d9d640 1 -- 172.21.15.88:0/3012522944 >> 172.21.15.88:0/3012522944 conn(0x7f66240cdc00 msgr2=0x7f66240cdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:59.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.355+0000 7f6628d9d640 1 -- 172.21.15.88:0/3012522944 shutdown_connections 2024-09-16T01:41:59.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.355+0000 7f6628d9d640 1 -- 172.21.15.88:0/3012522944 wait complete. 2024-09-16T01:41:59.369 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:41:59.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.470+0000 7f909a682640 1 Processor -- start 2024-09-16T01:41:59.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.470+0000 7f909a682640 1 -- start start 2024-09-16T01:41:59.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f909a682640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f90940b54d0 0x7f909406fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:59.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f909a682640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90940b7450 0x7f909415f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:59.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f909a682640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f909406e990 con 0x7f909415e6f0 2024-09-16T01:41:59.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f909a682640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f909406ec70 con 0x7f90940b54d0 2024-09-16T01:41:59.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f909a682640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f909415ec70 con 0x7f90940b7450 2024-09-16T01:41:59.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f9093fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f909415e6f0 0x7f90940b6d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:56950/0 (socket says 172.21.15.88:56950) 2024-09-16T01:41:59.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f9093fff640 1 -- 172.21.15.88:0/4286751417 learned_addr learned my addr 172.21.15.88:0/4286751417 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:41:59.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f9092ffd640 1 -- 172.21.15.88:0/4286751417 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3894193287 0 0) 0x7f909406e990 con 0x7f909415e6f0 2024-09-16T01:41:59.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f9092ffd640 1 -- 172.21.15.88:0/4286751417 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9080003520 con 0x7f909415e6f0 2024-09-16T01:41:59.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f90937fe640 1 --2- 172.21.15.88:0/4286751417 >> v2:172.21.15.154:3300/0 conn(0x7f90940b54d0 0x7f909406fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:59.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f9098c1f640 1 --2- 172.21.15.88:0/4286751417 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90940b7450 0x7f909415f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:59.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f9092ffd640 1 -- 172.21.15.88:0/4286751417 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1855483529 0 0) 0x7f9080003520 con 0x7f909415e6f0 2024-09-16T01:41:59.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f9092ffd640 1 -- 172.21.15.88:0/4286751417 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90940b7450 msgr2=0x7f909415f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:59.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f9092ffd640 1 --2- 172.21.15.88:0/4286751417 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90940b7450 0x7f909415f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:59.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f9092ffd640 1 -- 172.21.15.88:0/4286751417 >> v2:172.21.15.154:3300/0 conn(0x7f90940b54d0 msgr2=0x7f909406fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:59.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f9092ffd640 1 --2- 172.21.15.88:0/4286751417 >> v2:172.21.15.154:3300/0 conn(0x7f90940b54d0 0x7f909406fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:59.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f9092ffd640 1 -- 172.21.15.88:0/4286751417 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f909415ef50 con 0x7f909415e6f0 2024-09-16T01:41:59.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f9092ffd640 1 -- 172.21.15.88:0/4286751417 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f9084001670 con 0x7f909415e6f0 2024-09-16T01:41:59.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f9098c1f640 1 --2- 172.21.15.88:0/4286751417 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90940b7450 0x7f909415f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:59.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f9092ffd640 1 -- 172.21.15.88:0/4286751417 >> v1:172.21.15.88:6789/0 conn(0x7f909415e6f0 legacy=0x7f90940b6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:59.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f9092ffd640 1 --2- 172.21.15.88:0/4286751417 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90940b7450 0x7f90800039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:59.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f9092ffd640 1 --2- 172.21.15.88:0/4286751417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9080003ef0 0x7f9080004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:59.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f9092ffd640 1 --2- 172.21.15.88:0/4286751417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9080004d40 0x7f90800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:59.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f9092ffd640 1 -- 172.21.15.88:0/4286751417 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f909406ec70 con 0x7f9080003ef0 2024-09-16T01:41:59.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f9092ffd640 1 -- 172.21.15.88:0/4286751417 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f909415ec70 con 0x7f9080004d40 2024-09-16T01:41:59.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.471+0000 7f9092ffd640 1 -- 172.21.15.88:0/4286751417 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f908000b9f0 con 0x7f90940b7450 2024-09-16T01:41:59.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.472+0000 7f90937fe640 1 --2- 172.21.15.88:0/4286751417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9080003ef0 0x7f9080004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:59.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.472+0000 7f9093fff640 1 --2- 172.21.15.88:0/4286751417 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90940b7450 0x7f90800039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:59.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.472+0000 7f9098c1f640 1 --2- 172.21.15.88:0/4286751417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9080004d40 0x7f90800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:59.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.472+0000 7f90937fe640 1 -- 172.21.15.88:0/4286751417 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90940b7450 msgr2=0x7f90800039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:59.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.472+0000 7f90937fe640 1 --2- 172.21.15.88:0/4286751417 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90940b7450 0x7f90800039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:59.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.472+0000 7f90937fe640 1 -- 172.21.15.88:0/4286751417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9080004d40 msgr2=0x7f90800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:41:59.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.472+0000 7f90937fe640 1 --2- 172.21.15.88:0/4286751417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9080004d40 0x7f90800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:59.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.472+0000 7f90937fe640 1 -- 172.21.15.88:0/4286751417 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9080003520 con 0x7f9080003ef0 2024-09-16T01:41:59.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.472+0000 7f9098c1f640 1 --2- 172.21.15.88:0/4286751417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9080004d40 0x7f90800051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:41:59.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.472+0000 7f90937fe640 1 --2- 172.21.15.88:0/4286751417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9080003ef0 0x7f9080004800 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f907c009a90 tx=0x7f9080005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44b6dc6b6fa806a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:59.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.473+0000 7f9092ffd640 1 -- 172.21.15.88:0/4286751417 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c036070 con 0x7f9080003ef0 2024-09-16T01:41:59.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.473+0000 7f9092ffd640 1 -- 172.21.15.88:0/4286751417 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f907c031400 con 0x7f9080003ef0 2024-09-16T01:41:59.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.473+0000 7f9092ffd640 1 -- 172.21.15.88:0/4286751417 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f907c009690 con 0x7f9080003ef0 2024-09-16T01:41:59.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.473+0000 7f909a682640 1 -- 172.21.15.88:0/4286751417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9080003ef0 msgr2=0x7f9080004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:59.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.473+0000 7f909a682640 1 --2- 172.21.15.88:0/4286751417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9080003ef0 0x7f9080004800 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f907c009a90 tx=0x7f9080005830 comp rx=0 tx=0).stop 2024-09-16T01:41:59.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.473+0000 7f9098c1f640 1 -- 172.21.15.88:0/4286751417 reap_dead start 2024-09-16T01:41:59.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.473+0000 7f909a682640 1 -- 172.21.15.88:0/4286751417 shutdown_connections 2024-09-16T01:41:59.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.473+0000 7f909a682640 1 -- 172.21.15.88:0/4286751417 >> 172.21.15.88:0/4286751417 conn(0x7f90940d2430 msgr2=0x7f90940d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:41:59.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.473+0000 7f909a682640 1 -- 172.21.15.88:0/4286751417 shutdown_connections 2024-09-16T01:41:59.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.473+0000 7f909a682640 1 -- 172.21.15.88:0/4286751417 wait complete. 2024-09-16T01:41:59.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.474+0000 7f909a682640 1 Processor -- start 2024-09-16T01:41:59.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.474+0000 7f909a682640 1 -- start start 2024-09-16T01:41:59.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.475+0000 7f909a682640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9080003ef0 0x7f90941612d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:59.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.475+0000 7f909a682640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f909415fd00 0x7f9094149dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:59.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.475+0000 7f909a682640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90941625c0 0x7f9094162a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:59.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.475+0000 7f909a682640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f909406e730 con 0x7f90941625c0 2024-09-16T01:41:59.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.475+0000 7f909a682640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f909406e5b0 con 0x7f909415fd00 2024-09-16T01:41:59.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.475+0000 7f909a682640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f909415f750 con 0x7f9080003ef0 2024-09-16T01:41:59.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.475+0000 7f9098c1f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90941625c0 0x7f9094162a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:59.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.475+0000 7f9098c1f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90941625c0 0x7f9094162a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:53940/0 (socket says 172.21.15.88:53940) 2024-09-16T01:41:59.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.475+0000 7f9098c1f640 1 -- 172.21.15.88:0/2889405471 learned_addr learned my addr 172.21.15.88:0/2889405471 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:41:59.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.475+0000 7f90937fe640 1 --2- 172.21.15.88:0/2889405471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f909415fd00 0x7f9094149dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:59.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.475+0000 7f9093fff640 1 --2- 172.21.15.88:0/2889405471 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9080003ef0 0x7f90941612d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:59.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.476+0000 7f9098c1f640 1 -- 172.21.15.88:0/2889405471 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9080003ef0 msgr2=0x7f90941612d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:59.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.476+0000 7f9098c1f640 1 --2- 172.21.15.88:0/2889405471 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9080003ef0 0x7f90941612d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:59.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.476+0000 7f9098c1f640 1 -- 172.21.15.88:0/2889405471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f909415fd00 msgr2=0x7f9094149dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:41:59.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.476+0000 7f9098c1f640 1 --2- 172.21.15.88:0/2889405471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f909415fd00 0x7f9094149dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:41:59.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.476+0000 7f9098c1f640 1 -- 172.21.15.88:0/2889405471 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9094160230 con 0x7f90941625c0 2024-09-16T01:41:59.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.476+0000 7f9098c1f640 1 --2- 172.21.15.88:0/2889405471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90941625c0 0x7f9094162a40 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f9094074440 tx=0x7f908c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:59.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.476+0000 7f9090ff9640 1 -- 172.21.15.88:0/2889405471 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908c007c10 con 0x7f90941625c0 2024-09-16T01:41:59.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.476+0000 7f9090ff9640 1 -- 172.21.15.88:0/2889405471 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f908c019070 con 0x7f90941625c0 2024-09-16T01:41:59.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.476+0000 7f9090ff9640 1 -- 172.21.15.88:0/2889405471 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908c014410 con 0x7f90941625c0 2024-09-16T01:41:59.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.476+0000 7f909a682640 1 -- 172.21.15.88:0/2889405471 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9094160510 con 0x7f90941625c0 2024-09-16T01:41:59.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.476+0000 7f909a682640 1 -- 172.21.15.88:0/2889405471 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9094160ad0 con 0x7f90941625c0 2024-09-16T01:41:59.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.477+0000 7f909a682640 1 -- 172.21.15.88:0/2889405471 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f909406c6c0 con 0x7f90941625c0 2024-09-16T01:41:59.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.477+0000 7f9090ff9640 1 -- 172.21.15.88:0/2889405471 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f908c0145b0 con 0x7f90941625c0 2024-09-16T01:41:59.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.478+0000 7f9090ff9640 1 --2- 172.21.15.88:0/2889405471 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f907406c2e0 0x7f907406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:41:59.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.478+0000 7f9090ff9640 1 -- 172.21.15.88:0/2889405471 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7114+0+0 (secure 0 0 0) 0x7f908c08bbd0 con 0x7f90941625c0 2024-09-16T01:41:59.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.478+0000 7f9093fff640 1 --2- 172.21.15.88:0/2889405471 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f907406c2e0 0x7f907406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:41:59.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.478+0000 7f9093fff640 1 --2- 172.21.15.88:0/2889405471 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f907406c2e0 0x7f907406e7a0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f90940713f0 tx=0x7f90840078f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:41:59.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.482+0000 7f9090ff9640 1 -- 172.21.15.88:0/2889405471 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f908c059e50 con 0x7f90941625c0 2024-09-16T01:41:59.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:41:59.715+0000 7f909a682640 1 -- 172.21.15.88:0/2889405471 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f9094071140 con 0x7f90941625c0 2024-09-16T01:42:00.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.354+0000 7f9090ff9640 1 -- 172.21.15.88:0/2889405471 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v166) ==== 101+0+0 (secure 0 0 0) 0x7f908c059750 con 0x7f90941625c0 2024-09-16T01:42:00.355 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:42:00.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.356+0000 7f909a682640 1 -- 172.21.15.88:0/2889405471 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f907406c2e0 msgr2=0x7f907406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:00.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.356+0000 7f909a682640 1 --2- 172.21.15.88:0/2889405471 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f907406c2e0 0x7f907406e7a0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f90940713f0 tx=0x7f90840078f0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.356+0000 7f909a682640 1 -- 172.21.15.88:0/2889405471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90941625c0 msgr2=0x7f9094162a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:00.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.356+0000 7f909a682640 1 --2- 172.21.15.88:0/2889405471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90941625c0 0x7f9094162a40 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f9094074440 tx=0x7f908c0046b0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.356+0000 7f909a682640 1 -- 172.21.15.88:0/2889405471 shutdown_connections 2024-09-16T01:42:00.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.356+0000 7f909a682640 1 --2- 172.21.15.88:0/2889405471 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f907406c2e0 0x7f907406e7a0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.356+0000 7f909a682640 1 --2- 172.21.15.88:0/2889405471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90941625c0 0x7f9094162a40 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.356+0000 7f909a682640 1 --2- 172.21.15.88:0/2889405471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f909415fd00 0x7f9094149dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.356+0000 7f909a682640 1 --2- 172.21.15.88:0/2889405471 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9080003ef0 0x7f90941612d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.356+0000 7f909a682640 1 -- 172.21.15.88:0/2889405471 >> 172.21.15.88:0/2889405471 conn(0x7f90940d2430 msgr2=0x7f90940702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:00.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.356+0000 7f909a682640 1 -- 172.21.15.88:0/2889405471 shutdown_connections 2024-09-16T01:42:00.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.356+0000 7f909a682640 1 -- 172.21.15.88:0/2889405471 wait complete. 2024-09-16T01:42:00.372 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:42:00.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.416+0000 7f51c814cf40 1 Processor -- start 2024-09-16T01:42:00.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.416+0000 7f51c814cf40 1 -- start start 2024-09-16T01:42:00.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.417+0000 7f51c814cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b40030f0 0x556ea2be1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.417+0000 7f51c814cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556ea2bdce90 0x556ea2be2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.417+0000 7f51c814cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ea2b39b90 0x556ea2b4fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.417+0000 7f51c814cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x556ea2a5d510 con 0x7f51b40030f0 2024-09-16T01:42:00.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.417+0000 7f51c814cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556ea2ab6c50 con 0x556ea2b39b90 2024-09-16T01:42:00.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.417+0000 7f51c814cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x556ea2bd5f50 con 0x556ea2bdce90 2024-09-16T01:42:00.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.417+0000 7f51c5c42640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b40030f0 0x556ea2be1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.417+0000 7f51c5c42640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b40030f0 0x556ea2be1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44510/0 (socket says 172.21.15.88:44510) 2024-09-16T01:42:00.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.417+0000 7f51c5c42640 1 -- 172.21.15.88:0/3380504961 learned_addr learned my addr 172.21.15.88:0/3380504961 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:00.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.417+0000 7f51c6443640 1 --2- 172.21.15.88:0/3380504961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ea2b39b90 0x556ea2b4fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.417+0000 7f51c5441640 1 --2- 172.21.15.88:0/3380504961 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556ea2bdce90 0x556ea2be2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.417+0000 7f51c5c42640 1 -- 172.21.15.88:0/3380504961 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556ea2bdce90 msgr2=0x556ea2be2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:00.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.417+0000 7f51c5c42640 1 --2- 172.21.15.88:0/3380504961 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556ea2bdce90 0x556ea2be2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.417+0000 7f51c5c42640 1 -- 172.21.15.88:0/3380504961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ea2b39b90 msgr2=0x556ea2b4fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:00.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.417+0000 7f51c5c42640 1 --2- 172.21.15.88:0/3380504961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ea2b39b90 0x556ea2b4fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.417+0000 7f51c5c42640 1 -- 172.21.15.88:0/3380504961 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ea2bd5ce0 con 0x7f51b40030f0 2024-09-16T01:42:00.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.417+0000 7f51c6443640 1 --2- 172.21.15.88:0/3380504961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ea2b39b90 0x556ea2b4fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:00.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.417+0000 7f51c5c42640 1 --2- 172.21.15.88:0/3380504961 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b40030f0 0x556ea2be1fc0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f51c000d2c0 tx=0x7f51c000d780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f20a926e6000a9da server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:00.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.417+0000 7f51bffff640 1 -- 172.21.15.88:0/3380504961 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c0003110 con 0x7f51b40030f0 2024-09-16T01:42:00.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.417+0000 7f51bffff640 1 -- 172.21.15.88:0/3380504961 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51c00032b0 con 0x7f51b40030f0 2024-09-16T01:42:00.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.417+0000 7f51bffff640 1 -- 172.21.15.88:0/3380504961 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c0003460 con 0x7f51b40030f0 2024-09-16T01:42:00.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.418+0000 7f51c814cf40 1 -- 172.21.15.88:0/3380504961 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b40030f0 msgr2=0x556ea2be1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:00.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.418+0000 7f51c814cf40 1 --2- 172.21.15.88:0/3380504961 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b40030f0 0x556ea2be1fc0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f51c000d2c0 tx=0x7f51c000d780 comp rx=0 tx=0).stop 2024-09-16T01:42:00.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.418+0000 7f51c814cf40 1 -- 172.21.15.88:0/3380504961 shutdown_connections 2024-09-16T01:42:00.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.418+0000 7f51c814cf40 1 --2- 172.21.15.88:0/3380504961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556ea2b39b90 0x556ea2b4fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.418+0000 7f51c814cf40 1 --2- 172.21.15.88:0/3380504961 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556ea2bdce90 0x556ea2be2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.418+0000 7f51c814cf40 1 --2- 172.21.15.88:0/3380504961 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51b40030f0 0x556ea2be1fc0 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.418+0000 7f51c814cf40 1 -- 172.21.15.88:0/3380504961 >> 172.21.15.88:0/3380504961 conn(0x556ea2b36680 msgr2=0x556ea2bde4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:00.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.418+0000 7f51c814cf40 1 -- 172.21.15.88:0/3380504961 shutdown_connections 2024-09-16T01:42:00.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.418+0000 7f51c814cf40 1 -- 172.21.15.88:0/3380504961 wait complete. 2024-09-16T01:42:00.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.419+0000 7f51c814cf40 1 Processor -- start 2024-09-16T01:42:00.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.419+0000 7f51c814cf40 1 -- start start 2024-09-16T01:42:00.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.419+0000 7f51c814cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556ea2b39b90 0x556ea2bd05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.419+0000 7f51c814cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556ea2bdce90 0x556ea2bd0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.419+0000 7f51c814cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51b40030f0 0x556ea2bd2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.419+0000 7f51c814cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x556ea2bdfc40 con 0x556ea2bdce90 2024-09-16T01:42:00.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.419+0000 7f51c814cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556ea2be3b80 con 0x7f51b40030f0 2024-09-16T01:42:00.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.419+0000 7f51c814cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x556ea2b3c490 con 0x556ea2b39b90 2024-09-16T01:42:00.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.419+0000 7f51c5441640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556ea2bdce90 0x556ea2bd0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.419+0000 7f51c5441640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556ea2bdce90 0x556ea2bd0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44526/0 (socket says 172.21.15.88:44526) 2024-09-16T01:42:00.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.419+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 learned_addr learned my addr 172.21.15.88:0/4233758145 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:00.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.419+0000 7f51c6443640 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51b40030f0 0x556ea2bd2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.419+0000 7f51c5c42640 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556ea2b39b90 0x556ea2bd05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.420+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556ea2b39b90 msgr2=0x556ea2bd05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:00.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.420+0000 7f51c5441640 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556ea2b39b90 0x556ea2bd05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.420+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51b40030f0 msgr2=0x556ea2bd2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:00.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.420+0000 7f51c5441640 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51b40030f0 0x556ea2bd2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.420+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ea2be2840 con 0x556ea2bdce90 2024-09-16T01:42:00.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.420+0000 7f51c5c42640 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556ea2b39b90 0x556ea2bd05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:00.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.420+0000 7f51c5441640 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556ea2bdce90 0x556ea2bd0cf0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f51b0046920 tx=0x7f51b0001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:00.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.420+0000 7f51beffd640 1 -- 172.21.15.88:0/4233758145 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b0065020 con 0x556ea2bdce90 2024-09-16T01:42:00.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.420+0000 7f51beffd640 1 -- 172.21.15.88:0/4233758145 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51b004dc80 con 0x556ea2bdce90 2024-09-16T01:42:00.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.420+0000 7f51beffd640 1 -- 172.21.15.88:0/4233758145 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b0061670 con 0x556ea2bdce90 2024-09-16T01:42:00.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.420+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556ea2bcf7e0 con 0x556ea2bdce90 2024-09-16T01:42:00.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.420+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ea2bcfda0 con 0x556ea2bdce90 2024-09-16T01:42:00.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.422+0000 7f51beffd640 1 -- 172.21.15.88:0/4233758145 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f51b0068020 con 0x556ea2bdce90 2024-09-16T01:42:00.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.422+0000 7f51beffd640 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f51a806ac90 0x7f51a806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.422+0000 7f51beffd640 1 -- 172.21.15.88:0/4233758145 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7114+0+0 (secure 0 0 0) 0x7f51b00ceae0 con 0x556ea2bdce90 2024-09-16T01:42:00.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.422+0000 7f51c5c42640 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f51a806ac90 0x7f51a806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.429 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:42:00.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.422+0000 7f51c814cf40 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x556ea2be1fe0 0x556ea2bd3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.422+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2be23c0 con 0x556ea2be1fe0 2024-09-16T01:42:00.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.422+0000 7f51c6443640 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x556ea2be1fe0 0x556ea2bd3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.422+0000 7f51c5c42640 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f51a806ac90 0x7f51a806d150 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f51c000d260 tx=0x7f51c0002000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:00.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.423+0000 7f51c6443640 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x556ea2be1fe0 0x556ea2bd3d00 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:00.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.425+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v103'474 uv448 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.425+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103) -- 0x556ea2bd4240 con 0x556ea2be1fe0 2024-09-16T01:42:00.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.426+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v103'475 uv475 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.426+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2bd4610 con 0x556ea2be1fe0 2024-09-16T01:42:00.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.427+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v103'476 uv476 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.432 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:42:00.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.427+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c879b0 con 0x556ea2be1fe0 2024-09-16T01:42:00.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.428+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v103'477 uv477 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.428+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c87d50 con 0x556ea2be1fe0 2024-09-16T01:42:00.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.429+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v103'478 uv478 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.429+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c869a0 con 0x556ea2be1fe0 2024-09-16T01:42:00.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.430+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v103'479 uv479 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.430+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c86d60 con 0x556ea2be1fe0 2024-09-16T01:42:00.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.432+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v103'480 uv454 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.432+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c87100 con 0x556ea2be1fe0 2024-09-16T01:42:00.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.433+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v103'481 uv481 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.433+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c87100 con 0x556ea2be1fe0 2024-09-16T01:42:00.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.435+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v103'482 uv482 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.436 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:42:00.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.435+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c87100 con 0x556ea2be1fe0 2024-09-16T01:42:00.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.436+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v103'483 uv483 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.436+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c87100 con 0x556ea2be1fe0 2024-09-16T01:42:00.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.437+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v103'484 uv484 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.437+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c87100 con 0x556ea2be1fe0 2024-09-16T01:42:00.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.438+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v103'485 uv485 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.438+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c87100 con 0x556ea2be1fe0 2024-09-16T01:42:00.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.440+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v103'486 uv461 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.440+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c87100 con 0x556ea2be1fe0 2024-09-16T01:42:00.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.441+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v103'487 uv487 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.441+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c87100 con 0x556ea2be1fe0 2024-09-16T01:42:00.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.443+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v103'488 uv488 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.444 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:42:00.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.443+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c87100 con 0x556ea2be1fe0 2024-09-16T01:42:00.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.444+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v103'489 uv489 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.444+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c87100 con 0x556ea2be1fe0 2024-09-16T01:42:00.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.446+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v103'490 uv490 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.446+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c87100 con 0x556ea2be1fe0 2024-09-16T01:42:00.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.447+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v103'491 uv491 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.447+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2bd3110 con 0x556ea2be1fe0 2024-09-16T01:42:00.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.448+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v103'492 uv492 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.449+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2bd3110 con 0x556ea2be1fe0 2024-09-16T01:42:00.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.450+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v103'493 uv467 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.450+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103) -- 0x556ea2bd3110 con 0x556ea2be1fe0 2024-09-16T01:42:00.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.451+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v103'494 uv494 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.451+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2bd3110 con 0x556ea2be1fe0 2024-09-16T01:42:00.453 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:42:00.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.452+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v103'495 uv495 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.452+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2bd3110 con 0x556ea2be1fe0 2024-09-16T01:42:00.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.454+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v103'496 uv496 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.454+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2bd3110 con 0x556ea2be1fe0 2024-09-16T01:42:00.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.455+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v103'497 uv497 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.455+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2bd3110 con 0x556ea2be1fe0 2024-09-16T01:42:00.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.456+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v103'498 uv498 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.456+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2bd3110 con 0x556ea2be1fe0 2024-09-16T01:42:00.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.458+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v103'499 uv473 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.458+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103) -- 0x556ea2bd3110 con 0x556ea2be1fe0 2024-09-16T01:42:00.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.459+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v103'500 uv500 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.459+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2bd3110 con 0x556ea2be1fe0 2024-09-16T01:42:00.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.461+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v103'501 uv501 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.462 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:42:00.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.461+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2bd3110 con 0x556ea2be1fe0 2024-09-16T01:42:00.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.462+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v103'502 uv502 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.462+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2bd3110 con 0x556ea2be1fe0 2024-09-16T01:42:00.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.463+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v103'503 uv503 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.463+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2bd3110 con 0x556ea2be1fe0 2024-09-16T01:42:00.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.464+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v103'504 uv504 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.464+0000 7f51c814cf40 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x556ea2c88600 0x556ea2c8a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.464+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2bd3110 con 0x556ea2c88600 2024-09-16T01:42:00.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.464+0000 7f51c5c42640 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x556ea2c88600 0x556ea2c8a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.465+0000 7f51c5c42640 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x556ea2c88600 0x556ea2c8a9f0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:00.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.466+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v103'353 uv352 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c88600 2024-09-16T01:42:00.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.466+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8af30 con 0x556ea2c88600 2024-09-16T01:42:00.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.467+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v103'354 uv354 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c88600 2024-09-16T01:42:00.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.467+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8b2d0 con 0x556ea2c88600 2024-09-16T01:42:00.470 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:42:00.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.468+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v103'355 uv355 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c88600 2024-09-16T01:42:00.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.468+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8bb20 con 0x556ea2c88600 2024-09-16T01:42:00.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.469+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v103'356 uv356 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c88600 2024-09-16T01:42:00.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.469+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8bec0 con 0x556ea2c88600 2024-09-16T01:42:00.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.470+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v103'357 uv357 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c88600 2024-09-16T01:42:00.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.470+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8c260 con 0x556ea2c88600 2024-09-16T01:42:00.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.471+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v103'358 uv340 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c88600 2024-09-16T01:42:00.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.471+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8c600 con 0x556ea2c88600 2024-09-16T01:42:00.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.473+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v103'359 uv359 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c88600 2024-09-16T01:42:00.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.473+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8c9a0 con 0x556ea2c88600 2024-09-16T01:42:00.475 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:42:00.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.474+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v103'360 uv360 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c88600 2024-09-16T01:42:00.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.474+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8c9a0 con 0x556ea2c88600 2024-09-16T01:42:00.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.475+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v103'361 uv361 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c88600 2024-09-16T01:42:00.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.475+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8c9a0 con 0x556ea2c88600 2024-09-16T01:42:00.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.476+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v103'362 uv362 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c88600 2024-09-16T01:42:00.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.476+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8c9a0 con 0x556ea2c88600 2024-09-16T01:42:00.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.477+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v103'363 uv345 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c88600 2024-09-16T01:42:00.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.477+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8c9a0 con 0x556ea2c88600 2024-09-16T01:42:00.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.478+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v103'364 uv364 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c88600 2024-09-16T01:42:00.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.478+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8c9a0 con 0x556ea2c88600 2024-09-16T01:42:00.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.479+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v103'365 uv365 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c88600 2024-09-16T01:42:00.481 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:42:00.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.479+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c90e20 con 0x556ea2c88600 2024-09-16T01:42:00.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.480+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v103'366 uv366 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c88600 2024-09-16T01:42:00.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.480+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c90e20 con 0x556ea2c88600 2024-09-16T01:42:00.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.481+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v103'367 uv367 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c88600 2024-09-16T01:42:00.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.481+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8c9a0 con 0x556ea2c88600 2024-09-16T01:42:00.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.482+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v103'368 uv351 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c88600 2024-09-16T01:42:00.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.482+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8c9a0 con 0x556ea2c88600 2024-09-16T01:42:00.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.483+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v103'369 uv369 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c88600 2024-09-16T01:42:00.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.483+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8c9a0 con 0x556ea2c88600 2024-09-16T01:42:00.486 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:42:00.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.485+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v103'370 uv370 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c88600 2024-09-16T01:42:00.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.485+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8d9b0 con 0x556ea2c88600 2024-09-16T01:42:00.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.486+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v103'371 uv371 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c88600 2024-09-16T01:42:00.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.486+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8d9b0 con 0x556ea2c88600 2024-09-16T01:42:00.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.487+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v103'372 uv372 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c88600 2024-09-16T01:42:00.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.487+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8c9a0 con 0x556ea2c88600 2024-09-16T01:42:00.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.488+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v103'373 uv373 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c88600 2024-09-16T01:42:00.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.488+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8c9a0 con 0x556ea2be1fe0 2024-09-16T01:42:00.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.489+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v103'93 uv92 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.489+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8c9a0 con 0x556ea2be1fe0 2024-09-16T01:42:00.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.489+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v103'94 uv94 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.489+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8c9a0 con 0x556ea2be1fe0 2024-09-16T01:42:00.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.490+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v103'95 uv95 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.493 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:42:00.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.490+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8d9b0 con 0x556ea2be1fe0 2024-09-16T01:42:00.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.491+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v103'96 uv96 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.491+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8d9b0 con 0x556ea2be1fe0 2024-09-16T01:42:00.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.492+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v103'97 uv97 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.492+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8c9a0 con 0x556ea2be1fe0 2024-09-16T01:42:00.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.493+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v103'98 uv98 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.493+0000 7f51c814cf40 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x556ea2c8d020 0x556ea2c8f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.493+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8f9c0 con 0x556ea2c8d020 2024-09-16T01:42:00.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.493+0000 7f51c5441640 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x556ea2c8d020 0x556ea2c8f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.494+0000 7f51c5441640 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x556ea2c8d020 0x556ea2c8f410 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:00.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.495+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v103'73 uv72 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f51b0098df0 con 0x556ea2c8d020 2024-09-16T01:42:00.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.495+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8fec0 con 0x556ea2c8d020 2024-09-16T01:42:00.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.496+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v103'74 uv74 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f51b0098df0 con 0x556ea2c8d020 2024-09-16T01:42:00.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.496+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c903c0 con 0x556ea2c8d020 2024-09-16T01:42:00.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.496+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v103'75 uv75 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f51b0098df0 con 0x556ea2c8d020 2024-09-16T01:42:00.499 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:42:00.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.496+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c8b750 con 0x556ea2c8d020 2024-09-16T01:42:00.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.497+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v103'76 uv76 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f51b0098df0 con 0x556ea2c8d020 2024-09-16T01:42:00.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.497+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c90cf0 con 0x556ea2be1fe0 2024-09-16T01:42:00.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.499+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v103'93 uv92 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.499+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c90cf0 con 0x556ea2be1fe0 2024-09-16T01:42:00.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.500+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v103'94 uv94 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.500+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c90cf0 con 0x556ea2be1fe0 2024-09-16T01:42:00.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.503+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v103'95 uv95 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.504 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:42:00.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.503+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c91d00 con 0x556ea2be1fe0 2024-09-16T01:42:00.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.505+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v103'96 uv96 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.505+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c91d00 con 0x556ea2be1fe0 2024-09-16T01:42:00.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.506+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v103'97 uv97 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.506+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c90cf0 con 0x556ea2be1fe0 2024-09-16T01:42:00.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.507+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v103'98 uv98 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.507+0000 7f51c814cf40 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x556ea2c915c0 0x556ea2c939b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.507+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c93ef0 con 0x556ea2c915c0 2024-09-16T01:42:00.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.507+0000 7f51c6443640 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x556ea2c915c0 0x556ea2c939b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.508+0000 7f51c6443640 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x556ea2c915c0 0x556ea2c939b0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:00.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.509+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v103'171 uv165 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.509+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c93ef0 con 0x556ea2c915c0 2024-09-16T01:42:00.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.510+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v103'172 uv172 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.510+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c94310 con 0x556ea2c915c0 2024-09-16T01:42:00.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.511+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v103'173 uv173 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.512 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:42:00.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.511+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c95320 con 0x556ea2c915c0 2024-09-16T01:42:00.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.512+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v103'174 uv174 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.512+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c95320 con 0x556ea2c915c0 2024-09-16T01:42:00.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.513+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v103'175 uv175 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.513+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c94310 con 0x556ea2c915c0 2024-09-16T01:42:00.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.514+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v103'176 uv176 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.514+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c94310 con 0x556ea2c915c0 2024-09-16T01:42:00.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.515+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v103'177 uv170 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.515+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c94310 con 0x556ea2c915c0 2024-09-16T01:42:00.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.516+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v103'178 uv178 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.516+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c94310 con 0x556ea2c915c0 2024-09-16T01:42:00.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.517+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v103'179 uv179 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.518 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:42:00.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.517+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c94310 con 0x556ea2c915c0 2024-09-16T01:42:00.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.518+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v103'180 uv180 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.518+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c94310 con 0x556ea2c915c0 2024-09-16T01:42:00.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.519+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v103'181 uv181 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.519+0000 7f51c814cf40 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x556ea2c94990 0x556ea2c96d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.519+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c97350 con 0x556ea2c94990 2024-09-16T01:42:00.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.519+0000 7f51c5c42640 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x556ea2c94990 0x556ea2c96d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.520+0000 7f51c5c42640 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x556ea2c94990 0x556ea2c96d80 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:00.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.521+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v103'76 uv75 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c94990 2024-09-16T01:42:00.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.521+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c97720 con 0x556ea2c94990 2024-09-16T01:42:00.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.522+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v103'77 uv77 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c94990 2024-09-16T01:42:00.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.522+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c97720 con 0x556ea2c94990 2024-09-16T01:42:00.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.523+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v103'78 uv78 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c94990 2024-09-16T01:42:00.525 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:42:00.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.523+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c97720 con 0x556ea2c94990 2024-09-16T01:42:00.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.524+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v103'79 uv79 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c94990 2024-09-16T01:42:00.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.524+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c97720 con 0x556ea2c94990 2024-09-16T01:42:00.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.525+0000 7f51c5c42640 1 -- 172.21.15.88:0/4233758145 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v103'80 uv80 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f51c000e040 con 0x556ea2c94990 2024-09-16T01:42:00.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.525+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c97720 con 0x556ea2be1fe0 2024-09-16T01:42:00.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.526+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v103'93 uv92 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.527+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c97720 con 0x556ea2be1fe0 2024-09-16T01:42:00.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.527+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v103'94 uv94 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.528+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c97720 con 0x556ea2be1fe0 2024-09-16T01:42:00.529 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:42:00.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.528+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v103'95 uv95 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.528+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c98730 con 0x556ea2be1fe0 2024-09-16T01:42:00.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.529+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v103'96 uv96 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.529+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c98730 con 0x556ea2be1fe0 2024-09-16T01:42:00.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.530+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v103'97 uv97 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.530+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c97720 con 0x556ea2be1fe0 2024-09-16T01:42:00.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.531+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v103'98 uv98 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2be1fe0 2024-09-16T01:42:00.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.531+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c97720 con 0x556ea2c8d020 2024-09-16T01:42:00.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.532+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v103'152 uv146 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f51b0098df0 con 0x556ea2c8d020 2024-09-16T01:42:00.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.532+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c97ae0 con 0x556ea2c8d020 2024-09-16T01:42:00.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.533+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v103'153 uv153 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f51b0098df0 con 0x556ea2c8d020 2024-09-16T01:42:00.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.533+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c97ed0 con 0x556ea2c8d020 2024-09-16T01:42:00.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.534+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v103'154 uv154 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f51b0098df0 con 0x556ea2c8d020 2024-09-16T01:42:00.536 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:42:00.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.534+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c98770 con 0x556ea2c8d020 2024-09-16T01:42:00.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.535+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v103'155 uv155 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f51b0098df0 con 0x556ea2c8d020 2024-09-16T01:42:00.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.535+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c98770 con 0x556ea2c8d020 2024-09-16T01:42:00.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.536+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v103'156 uv156 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f51b0098df0 con 0x556ea2c8d020 2024-09-16T01:42:00.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.536+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c98720 con 0x556ea2c8d020 2024-09-16T01:42:00.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.537+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v103'157 uv151 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f51b0098df0 con 0x556ea2c8d020 2024-09-16T01:42:00.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.537+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c98720 con 0x556ea2c8d020 2024-09-16T01:42:00.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.538+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v103'158 uv158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b0098df0 con 0x556ea2c8d020 2024-09-16T01:42:00.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.538+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c98720 con 0x556ea2c8d020 2024-09-16T01:42:00.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.539+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v103'159 uv159 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b0098df0 con 0x556ea2c8d020 2024-09-16T01:42:00.540 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:42:00.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.539+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c98720 con 0x556ea2c8d020 2024-09-16T01:42:00.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.540+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v103'160 uv160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b0098df0 con 0x556ea2c8d020 2024-09-16T01:42:00.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.540+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9b500 con 0x556ea2c8d020 2024-09-16T01:42:00.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.541+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v103'161 uv161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b0098df0 con 0x556ea2c8d020 2024-09-16T01:42:00.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.541+0000 7f51c814cf40 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x556ea2c990b0 0x556ea2c9b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.541+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9b9e0 con 0x556ea2c990b0 2024-09-16T01:42:00.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.541+0000 7f51c5441640 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x556ea2c990b0 0x556ea2c9b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.541+0000 7f51c5441640 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x556ea2c990b0 0x556ea2c9b4a0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:00.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.543+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v103'70 uv69 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f51b0098df0 con 0x556ea2c990b0 2024-09-16T01:42:00.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.543+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9b9e0 con 0x556ea2c990b0 2024-09-16T01:42:00.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.544+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v103'71 uv71 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f51b0098df0 con 0x556ea2c990b0 2024-09-16T01:42:00.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.544+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9b9e0 con 0x556ea2c990b0 2024-09-16T01:42:00.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.544+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v103'72 uv72 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f51b0098df0 con 0x556ea2c990b0 2024-09-16T01:42:00.546 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:42:00.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.544+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9b9e0 con 0x556ea2c990b0 2024-09-16T01:42:00.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.545+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v103'73 uv73 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f51b0098df0 con 0x556ea2c990b0 2024-09-16T01:42:00.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.545+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9b9e0 con 0x556ea2c8d020 2024-09-16T01:42:00.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.547+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v103'93 uv92 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f51b0098df0 con 0x556ea2c8d020 2024-09-16T01:42:00.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.547+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9b9e0 con 0x556ea2c8d020 2024-09-16T01:42:00.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.548+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v103'94 uv94 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b0098df0 con 0x556ea2c8d020 2024-09-16T01:42:00.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.548+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9b9e0 con 0x556ea2c8d020 2024-09-16T01:42:00.549 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:42:00.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.548+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v103'95 uv95 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b0098df0 con 0x556ea2c8d020 2024-09-16T01:42:00.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.548+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9c9f0 con 0x556ea2c8d020 2024-09-16T01:42:00.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.549+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v103'96 uv96 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b0098df0 con 0x556ea2c8d020 2024-09-16T01:42:00.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.550+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9c9f0 con 0x556ea2c8d020 2024-09-16T01:42:00.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.551+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v103'97 uv97 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b0098df0 con 0x556ea2c8d020 2024-09-16T01:42:00.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.551+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9b9e0 con 0x556ea2c8d020 2024-09-16T01:42:00.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.552+0000 7f51c5441640 1 -- 172.21.15.88:0/4233758145 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v103'98 uv98 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b0098df0 con 0x556ea2c8d020 2024-09-16T01:42:00.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.552+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9b9e0 con 0x556ea2c915c0 2024-09-16T01:42:00.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.553+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v103'79 uv78 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.553+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9b9e0 con 0x556ea2c915c0 2024-09-16T01:42:00.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.554+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v103'80 uv80 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.554+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9b9e0 con 0x556ea2c915c0 2024-09-16T01:42:00.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.555+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v103'81 uv81 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.556 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:42:00.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.555+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9b9e0 con 0x556ea2c915c0 2024-09-16T01:42:00.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.556+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v103'82 uv82 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.556+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9b9e0 con 0x556ea2c915c0 2024-09-16T01:42:00.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.558+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v103'83 uv83 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.558+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9b9e0 con 0x556ea2c915c0 2024-09-16T01:42:00.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.559+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v103'192 uv186 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.559+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9b9e0 con 0x556ea2c915c0 2024-09-16T01:42:00.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.560+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v103'193 uv193 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.560+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9b9e0 con 0x556ea2c915c0 2024-09-16T01:42:00.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.561+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v103'194 uv194 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.562 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:42:00.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.561+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9b9e0 con 0x556ea2c915c0 2024-09-16T01:42:00.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.562+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v103'195 uv195 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.562+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9b9e0 con 0x556ea2c915c0 2024-09-16T01:42:00.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.563+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v103'196 uv196 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.563+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9b9e0 con 0x556ea2c915c0 2024-09-16T01:42:00.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.563+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v103'197 uv191 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.563+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9b9e0 con 0x556ea2c915c0 2024-09-16T01:42:00.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.564+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v103'198 uv198 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.564+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9b9e0 con 0x556ea2c915c0 2024-09-16T01:42:00.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.565+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v103'199 uv199 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.566 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:42:00.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.565+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9b9e0 con 0x556ea2c915c0 2024-09-16T01:42:00.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.566+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v103'200 uv200 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.566+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9b9e0 con 0x556ea2c915c0 2024-09-16T01:42:00.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.567+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v103'201 uv201 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.567+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9b9e0 con 0x556ea2c915c0 2024-09-16T01:42:00.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.568+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v103'202 uv202 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.568+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x556ea2c9bda0 con 0x556ea2c915c0 2024-09-16T01:42:00.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.569+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v103'203 uv203 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f51b8002870 con 0x556ea2c915c0 2024-09-16T01:42:00.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.569+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x556ea2be1fe0 msgr2=0x556ea2bd3d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:00.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.569+0000 7f51c814cf40 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x556ea2be1fe0 0x556ea2bd3d00 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.569+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x556ea2c8d020 msgr2=0x556ea2c8f410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:00.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.569+0000 7f51c814cf40 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x556ea2c8d020 0x556ea2c8f410 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.569+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x556ea2c990b0 msgr2=0x556ea2c9b4a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:00.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.569+0000 7f51c814cf40 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x556ea2c990b0 0x556ea2c9b4a0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.569+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x556ea2c88600 msgr2=0x556ea2c8a9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:00.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.569+0000 7f51c814cf40 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x556ea2c88600 0x556ea2c8a9f0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.569+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 reap_dead start 2024-09-16T01:42:00.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.569+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x556ea2c915c0 msgr2=0x556ea2c939b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:00.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.569+0000 7f51c814cf40 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x556ea2c915c0 0x556ea2c939b0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.569+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x556ea2c94990 msgr2=0x556ea2c96d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:00.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.569+0000 7f51c814cf40 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x556ea2c94990 0x556ea2c96d80 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.569+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f51a806ac90 msgr2=0x7f51a806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:00.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.569+0000 7f51c814cf40 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f51a806ac90 0x7f51a806d150 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f51c000d260 tx=0x7f51c0002000 comp rx=0 tx=0).stop 2024-09-16T01:42:00.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.569+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556ea2bdce90 msgr2=0x556ea2bd0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:00.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.569+0000 7f51c814cf40 1 --2- 172.21.15.88:0/4233758145 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556ea2bdce90 0x556ea2bd0cf0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f51b0046920 tx=0x7f51b0001a40 comp rx=0 tx=0).stop 2024-09-16T01:42:00.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.569+0000 7f51c6443640 1 -- 172.21.15.88:0/4233758145 reap_dead start 2024-09-16T01:42:00.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.569+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 shutdown_connections 2024-09-16T01:42:00.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.569+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 >> 172.21.15.88:0/4233758145 conn(0x556ea2b36680 msgr2=0x556ea2b513d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:00.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.569+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 shutdown_connections 2024-09-16T01:42:00.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.569+0000 7f51c814cf40 1 -- 172.21.15.88:0/4233758145 wait complete. 2024-09-16T01:42:00.574 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put 200.00000000 - 2024-09-16T01:42:00.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.656+0000 7f4764016f40 1 Processor -- start 2024-09-16T01:42:00.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.656+0000 7f4764016f40 1 -- start start 2024-09-16T01:42:00.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.657+0000 7f4764016f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5571c9b15fe0 0x5571c9b11ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.657+0000 7f4764016f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5571c9b16a60 0x5571c9bbba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.657+0000 7f4764016f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571c9b30250 0x5571c9bbbf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.657+0000 7f4764016f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5571c9a38510 con 0x5571c9b15fe0 2024-09-16T01:42:00.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.657+0000 7f4764016f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5571c9a91c50 con 0x5571c9b30250 2024-09-16T01:42:00.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.657+0000 7f4764016f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5571c9bb2ae0 con 0x5571c9b16a60 2024-09-16T01:42:00.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.657+0000 7f4760e9c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5571c9b15fe0 0x5571c9b11ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.657+0000 7f4760e9c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5571c9b15fe0 0x5571c9b11ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44530/0 (socket says 172.21.15.88:44530) 2024-09-16T01:42:00.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.657+0000 7f4760e9c640 1 -- 172.21.15.88:0/853496266 learned_addr learned my addr 172.21.15.88:0/853496266 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:00.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.657+0000 7f476169d640 1 --2- 172.21.15.88:0/853496266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571c9b30250 0x5571c9bbbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.657+0000 7f475869b640 1 --2- 172.21.15.88:0/853496266 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5571c9b16a60 0x5571c9bbba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.657+0000 7f4760e9c640 1 -- 172.21.15.88:0/853496266 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5571c9b16a60 msgr2=0x5571c9bbba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:00.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.657+0000 7f4760e9c640 1 --2- 172.21.15.88:0/853496266 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5571c9b16a60 0x5571c9bbba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.657+0000 7f4760e9c640 1 -- 172.21.15.88:0/853496266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571c9b30250 msgr2=0x5571c9bbbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:00.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.657+0000 7f4760e9c640 1 --2- 172.21.15.88:0/853496266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571c9b30250 0x5571c9bbbf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.657+0000 7f4760e9c640 1 -- 172.21.15.88:0/853496266 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571c9bb2d90 con 0x5571c9b15fe0 2024-09-16T01:42:00.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.657+0000 7f476169d640 1 --2- 172.21.15.88:0/853496266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571c9b30250 0x5571c9bbbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:00.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.658+0000 7f4760e9c640 1 --2- 172.21.15.88:0/853496266 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5571c9b15fe0 0x5571c9b11ea0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f4750001d30 tx=0x7f475000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edb36d5906d5ecaf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:00.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.658+0000 7f475b7fe640 1 -- 172.21.15.88:0/853496266 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475000bd20 con 0x5571c9b15fe0 2024-09-16T01:42:00.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.658+0000 7f475b7fe640 1 -- 172.21.15.88:0/853496266 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f475000f400 con 0x5571c9b15fe0 2024-09-16T01:42:00.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.658+0000 7f475b7fe640 1 -- 172.21.15.88:0/853496266 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475000f830 con 0x5571c9b15fe0 2024-09-16T01:42:00.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.658+0000 7f4764016f40 1 -- 172.21.15.88:0/853496266 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5571c9b15fe0 msgr2=0x5571c9b11ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:00.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.658+0000 7f4764016f40 1 --2- 172.21.15.88:0/853496266 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5571c9b15fe0 0x5571c9b11ea0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f4750001d30 tx=0x7f475000b5b0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.658+0000 7f4764016f40 1 -- 172.21.15.88:0/853496266 shutdown_connections 2024-09-16T01:42:00.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.658+0000 7f4764016f40 1 --2- 172.21.15.88:0/853496266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571c9b30250 0x5571c9bbbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.658+0000 7f4764016f40 1 --2- 172.21.15.88:0/853496266 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5571c9b16a60 0x5571c9bbba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.658+0000 7f4764016f40 1 --2- 172.21.15.88:0/853496266 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5571c9b15fe0 0x5571c9b11ea0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.658+0000 7f4764016f40 1 -- 172.21.15.88:0/853496266 >> 172.21.15.88:0/853496266 conn(0x5571c9b31750 msgr2=0x5571c9b12c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:00.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.658+0000 7f4764016f40 1 -- 172.21.15.88:0/853496266 shutdown_connections 2024-09-16T01:42:00.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.658+0000 7f4764016f40 1 -- 172.21.15.88:0/853496266 wait complete. 2024-09-16T01:42:00.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.659+0000 7f4764016f40 1 Processor -- start 2024-09-16T01:42:00.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.659+0000 7f4764016f40 1 -- start start 2024-09-16T01:42:00.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.660+0000 7f4764016f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5571c9b15fe0 0x5571c9bad540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.660+0000 7f4764016f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571c9b16a60 0x5571c9bb0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.660+0000 7f4764016f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5571c9b30250 0x5571c9c549a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.660+0000 7f4764016f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5571c9bbe200 con 0x5571c9b30250 2024-09-16T01:42:00.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.660+0000 7f4764016f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5571c9b10e90 con 0x5571c9b16a60 2024-09-16T01:42:00.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.660+0000 7f4764016f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5571c9bbd970 con 0x5571c9b15fe0 2024-09-16T01:42:00.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.660+0000 7f476169d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5571c9b30250 0x5571c9c549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.660+0000 7f476169d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5571c9b30250 0x5571c9c549a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44540/0 (socket says 172.21.15.88:44540) 2024-09-16T01:42:00.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.660+0000 7f476169d640 1 -- 172.21.15.88:0/1089063187 learned_addr learned my addr 172.21.15.88:0/1089063187 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:00.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.660+0000 7f4760e9c640 1 --2- 172.21.15.88:0/1089063187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5571c9b15fe0 0x5571c9bad540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.660+0000 7f475869b640 1 --2- 172.21.15.88:0/1089063187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571c9b16a60 0x5571c9bb0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.660+0000 7f476169d640 1 -- 172.21.15.88:0/1089063187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5571c9b15fe0 msgr2=0x5571c9bad540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:00.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.660+0000 7f476169d640 1 --2- 172.21.15.88:0/1089063187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5571c9b15fe0 0x5571c9bad540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.660+0000 7f476169d640 1 -- 172.21.15.88:0/1089063187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571c9b16a60 msgr2=0x5571c9bb0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:00.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.660+0000 7f476169d640 1 --2- 172.21.15.88:0/1089063187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571c9b16a60 0x5571c9bb0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.660+0000 7f476169d640 1 -- 172.21.15.88:0/1089063187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571c9bafb60 con 0x5571c9b30250 2024-09-16T01:42:00.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.660+0000 7f476169d640 1 --2- 172.21.15.88:0/1089063187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5571c9b30250 0x5571c9c549a0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f475c025960 tx=0x7f475c003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:00.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.660+0000 7f475a7fc640 1 -- 172.21.15.88:0/1089063187 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c06a070 con 0x5571c9b30250 2024-09-16T01:42:00.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.660+0000 7f475a7fc640 1 -- 172.21.15.88:0/1089063187 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f475c05a450 con 0x5571c9b30250 2024-09-16T01:42:00.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.660+0000 7f4764016f40 1 -- 172.21.15.88:0/1089063187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5571c9bafe40 con 0x5571c9b30250 2024-09-16T01:42:00.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.660+0000 7f475a7fc640 1 -- 172.21.15.88:0/1089063187 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c055040 con 0x5571c9b30250 2024-09-16T01:42:00.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.660+0000 7f4764016f40 1 -- 172.21.15.88:0/1089063187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571c9bada80 con 0x5571c9b30250 2024-09-16T01:42:00.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.662+0000 7f475a7fc640 1 -- 172.21.15.88:0/1089063187 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f475c06d070 con 0x5571c9b30250 2024-09-16T01:42:00.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.662+0000 7f475a7fc640 1 --2- 172.21.15.88:0/1089063187 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f474006ac90 0x7f474006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.662+0000 7f475a7fc640 1 -- 172.21.15.88:0/1089063187 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7114+0+0 (secure 0 0 0) 0x7f475c0ce2e0 con 0x5571c9b30250 2024-09-16T01:42:00.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.662+0000 7f4760e9c640 1 --2- 172.21.15.88:0/1089063187 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f474006ac90 0x7f474006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.662+0000 7f4764016f40 1 --2- 172.21.15.88:0/1089063187 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5571c9bb0e60 0x5571c9badfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.662+0000 7f4764016f40 1 -- 172.21.15.88:0/1089063187 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:1 12.14 12:292cf221:::200.00000000:head [writefull 0~80 in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103) -- 0x5571c9bae4e0 con 0x5571c9bb0e60 2024-09-16T01:42:00.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.663+0000 7f475869b640 1 --2- 172.21.15.88:0/1089063187 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5571c9bb0e60 0x5571c9badfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.663+0000 7f4760e9c640 1 --2- 172.21.15.88:0/1089063187 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f474006ac90 0x7f474006d150 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f4750000c00 tx=0x7f4750009000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:00.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.663+0000 7f475869b640 1 --2- 172.21.15.88:0/1089063187 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5571c9bb0e60 0x5571c9badfa0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:00.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.664+0000 7f475869b640 1 -- 172.21.15.88:0/1089063187 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(1 200.00000000 [writefull 0~80] v103'374 uv374 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f474800e3e0 con 0x5571c9bb0e60 2024-09-16T01:42:00.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.664+0000 7f4764016f40 1 -- 172.21.15.88:0/1089063187 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5571c9bb0e60 msgr2=0x5571c9badfa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:00.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.664+0000 7f4764016f40 1 --2- 172.21.15.88:0/1089063187 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5571c9bb0e60 0x5571c9badfa0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.664+0000 7f4764016f40 1 -- 172.21.15.88:0/1089063187 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f474006ac90 msgr2=0x7f474006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:00.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.664+0000 7f4764016f40 1 --2- 172.21.15.88:0/1089063187 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f474006ac90 0x7f474006d150 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f4750000c00 tx=0x7f4750009000 comp rx=0 tx=0).stop 2024-09-16T01:42:00.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.664+0000 7f4764016f40 1 -- 172.21.15.88:0/1089063187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5571c9b30250 msgr2=0x5571c9c549a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:00.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.664+0000 7f4764016f40 1 --2- 172.21.15.88:0/1089063187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5571c9b30250 0x5571c9c549a0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f475c025960 tx=0x7f475c003330 comp rx=0 tx=0).stop 2024-09-16T01:42:00.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.664+0000 7f476169d640 1 -- 172.21.15.88:0/1089063187 reap_dead start 2024-09-16T01:42:00.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.665+0000 7f4764016f40 1 -- 172.21.15.88:0/1089063187 shutdown_connections 2024-09-16T01:42:00.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.665+0000 7f4764016f40 1 -- 172.21.15.88:0/1089063187 >> 172.21.15.88:0/1089063187 conn(0x5571c9b31750 msgr2=0x5571c9bb1950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:00.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.665+0000 7f4764016f40 1 -- 172.21.15.88:0/1089063187 shutdown_connections 2024-09-16T01:42:00.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.665+0000 7f4764016f40 1 -- 172.21.15.88:0/1089063187 wait complete. 2024-09-16T01:42:00.671 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:42:00.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.811+0000 7f9f25017640 1 Processor -- start 2024-09-16T01:42:00.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.811+0000 7f9f25017640 1 -- start start 2024-09-16T01:42:00.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.812+0000 7f9f25017640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f2015b480 0x7f9f200b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.812+0000 7f9f25017640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9f200b60c0 0x7f9f20071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.812+0000 7f9f25017640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9f20072530 0x7f9f20070fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.812+0000 7f9f25017640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f9f200b6500 con 0x7f9f200b60c0 2024-09-16T01:42:00.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.812+0000 7f9f25017640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9f20072990 con 0x7f9f20072530 2024-09-16T01:42:00.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.812+0000 7f9f25017640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9f200ccff0 con 0x7f9f2015b480 2024-09-16T01:42:00.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.812+0000 7f9f1659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9f200b60c0 0x7f9f20071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.812+0000 7f9f1659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9f200b60c0 0x7f9f20071ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44550/0 (socket says 172.21.15.88:44550) 2024-09-16T01:42:00.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.812+0000 7f9f1659c640 1 -- 172.21.15.88:0/401540182 learned_addr learned my addr 172.21.15.88:0/401540182 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:00.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.812+0000 7f9f1659c640 1 -- 172.21.15.88:0/401540182 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f2015b480 msgr2=0x7f9f200b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:00.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.812+0000 7f9f1659c640 1 --2- 172.21.15.88:0/401540182 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f2015b480 0x7f9f200b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.812+0000 7f9f1659c640 1 -- 172.21.15.88:0/401540182 >> v2:172.21.15.154:3300/0 conn(0x7f9f20072530 msgr2=0x7f9f20070fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:00.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.812+0000 7f9f1659c640 1 --2- 172.21.15.88:0/401540182 >> v2:172.21.15.154:3300/0 conn(0x7f9f20072530 0x7f9f20070fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.813+0000 7f9f1659c640 1 -- 172.21.15.88:0/401540182 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f200cd2d0 con 0x7f9f200b60c0 2024-09-16T01:42:00.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.813+0000 7f9f1659c640 1 --2- 172.21.15.88:0/401540182 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f200b60c0 0x7f9f20071ff0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f9f200b2eb0 tx=0x7f9f0802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d034b0c325b2476c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:00.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.813+0000 7f9f1e59c640 1 -- 172.21.15.88:0/401540182 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f08037070 con 0x7f9f200b60c0 2024-09-16T01:42:00.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.813+0000 7f9f1e59c640 1 -- 172.21.15.88:0/401540182 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f08032440 con 0x7f9f200b60c0 2024-09-16T01:42:00.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.813+0000 7f9f1e59c640 1 -- 172.21.15.88:0/401540182 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f08032810 con 0x7f9f200b60c0 2024-09-16T01:42:00.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.814+0000 7f9f25017640 1 -- 172.21.15.88:0/401540182 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f200b60c0 msgr2=0x7f9f20071ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:00.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.814+0000 7f9f25017640 1 --2- 172.21.15.88:0/401540182 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f200b60c0 0x7f9f20071ff0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f9f200b2eb0 tx=0x7f9f0802a180 comp rx=0 tx=0).stop 2024-09-16T01:42:00.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.814+0000 7f9f25017640 1 -- 172.21.15.88:0/401540182 shutdown_connections 2024-09-16T01:42:00.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.814+0000 7f9f25017640 1 --2- 172.21.15.88:0/401540182 >> v2:172.21.15.154:3300/0 conn(0x7f9f20072530 0x7f9f20070fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.814+0000 7f9f25017640 1 --2- 172.21.15.88:0/401540182 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f200b60c0 0x7f9f20071ff0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.814+0000 7f9f25017640 1 --2- 172.21.15.88:0/401540182 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f2015b480 0x7f9f200b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.814+0000 7f9f25017640 1 -- 172.21.15.88:0/401540182 >> 172.21.15.88:0/401540182 conn(0x7f9f2006d0f0 msgr2=0x7f9f2006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:00.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.814+0000 7f9f25017640 1 -- 172.21.15.88:0/401540182 shutdown_connections 2024-09-16T01:42:00.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.814+0000 7f9f25017640 1 -- 172.21.15.88:0/401540182 wait complete. 2024-09-16T01:42:00.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.815+0000 7f9f25017640 1 Processor -- start 2024-09-16T01:42:00.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.815+0000 7f9f25017640 1 -- start start 2024-09-16T01:42:00.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.816+0000 7f9f25017640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f20072530 0x7f9f201537e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.816+0000 7f9f25017640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f200b60c0 0x7f9f201557d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.816+0000 7f9f25017640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f2015b480 0x7f9f2014ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.816+0000 7f9f25017640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9f200cdb70 con 0x7f9f200b60c0 2024-09-16T01:42:00.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.816+0000 7f9f25017640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9f200cd890 con 0x7f9f2015b480 2024-09-16T01:42:00.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.816+0000 7f9f25017640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9f2015b860 con 0x7f9f20072530 2024-09-16T01:42:00.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.816+0000 7f9f1659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f200b60c0 0x7f9f201557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.816+0000 7f9f1659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f200b60c0 0x7f9f201557d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44566/0 (socket says 172.21.15.88:44566) 2024-09-16T01:42:00.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.816+0000 7f9f1659c640 1 -- 172.21.15.88:0/1704918162 learned_addr learned my addr 172.21.15.88:0/1704918162 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:00.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.816+0000 7f9f1ed9d640 1 --2- 172.21.15.88:0/1704918162 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f20072530 0x7f9f201537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.816+0000 7f9f1f59e640 1 --2- 172.21.15.88:0/1704918162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f2015b480 0x7f9f2014ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.817+0000 7f9f1659c640 1 -- 172.21.15.88:0/1704918162 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f20072530 msgr2=0x7f9f201537e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:00.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.817+0000 7f9f1659c640 1 --2- 172.21.15.88:0/1704918162 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f20072530 0x7f9f201537e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.817+0000 7f9f1659c640 1 -- 172.21.15.88:0/1704918162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f2015b480 msgr2=0x7f9f2014ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:00.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.817+0000 7f9f1659c640 1 --2- 172.21.15.88:0/1704918162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f2015b480 0x7f9f2014ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:00.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.817+0000 7f9f1659c640 1 -- 172.21.15.88:0/1704918162 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f2014f190 con 0x7f9f200b60c0 2024-09-16T01:42:00.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.817+0000 7f9f1659c640 1 --2- 172.21.15.88:0/1704918162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f200b60c0 0x7f9f201557d0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f9f08009870 tx=0x7f9f080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:00.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.817+0000 7f9f17fff640 1 -- 172.21.15.88:0/1704918162 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f08041070 con 0x7f9f200b60c0 2024-09-16T01:42:00.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.817+0000 7f9f17fff640 1 -- 172.21.15.88:0/1704918162 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f0803c870 con 0x7f9f200b60c0 2024-09-16T01:42:00.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.817+0000 7f9f17fff640 1 -- 172.21.15.88:0/1704918162 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f08037040 con 0x7f9f200b60c0 2024-09-16T01:42:00.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.817+0000 7f9f25017640 1 -- 172.21.15.88:0/1704918162 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f20153ea0 con 0x7f9f200b60c0 2024-09-16T01:42:00.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.817+0000 7f9f25017640 1 -- 172.21.15.88:0/1704918162 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f20154460 con 0x7f9f200b60c0 2024-09-16T01:42:00.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.818+0000 7f9f25017640 1 -- 172.21.15.88:0/1704918162 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f20071ff0 con 0x7f9f200b60c0 2024-09-16T01:42:00.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.819+0000 7f9f17fff640 1 -- 172.21.15.88:0/1704918162 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9f08044070 con 0x7f9f200b60c0 2024-09-16T01:42:00.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.819+0000 7f9f17fff640 1 --2- 172.21.15.88:0/1704918162 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9f0006c2e0 0x7f9f0006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:00.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.819+0000 7f9f17fff640 1 -- 172.21.15.88:0/1704918162 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 7114+0+0 (secure 0 0 0) 0x7f9f080ac5e0 con 0x7f9f200b60c0 2024-09-16T01:42:00.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.819+0000 7f9f1ed9d640 1 --2- 172.21.15.88:0/1704918162 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9f0006c2e0 0x7f9f0006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:00.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.819+0000 7f9f1ed9d640 1 --2- 172.21.15.88:0/1704918162 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9f0006c2e0 0x7f9f0006e7a0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f9f200ccad0 tx=0x7f9f10009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:00.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:00.823+0000 7f9f17fff640 1 -- 172.21.15.88:0/1704918162 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9f0807a2a0 con 0x7f9f200b60c0 2024-09-16T01:42:01.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.024+0000 7f9f25017640 1 -- 172.21.15.88:0/1704918162 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f9f20155d10 con 0x7f9f200b60c0 2024-09-16T01:42:01.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.364+0000 7f9f17fff640 1 -- 172.21.15.88:0/1704918162 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v167) ==== 162+0+0 (secure 0 0 0) 0x7f9f0807e5a0 con 0x7f9f200b60c0 2024-09-16T01:42:01.365 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:42:01.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.366+0000 7f9f25017640 1 -- 172.21.15.88:0/1704918162 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9f0006c2e0 msgr2=0x7f9f0006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:01.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.366+0000 7f9f25017640 1 --2- 172.21.15.88:0/1704918162 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9f0006c2e0 0x7f9f0006e7a0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f9f200ccad0 tx=0x7f9f10009060 comp rx=0 tx=0).stop 2024-09-16T01:42:01.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.366+0000 7f9f25017640 1 -- 172.21.15.88:0/1704918162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f200b60c0 msgr2=0x7f9f201557d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:01.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.366+0000 7f9f25017640 1 --2- 172.21.15.88:0/1704918162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f200b60c0 0x7f9f201557d0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f9f08009870 tx=0x7f9f080046c0 comp rx=0 tx=0).stop 2024-09-16T01:42:01.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.366+0000 7f9f25017640 1 -- 172.21.15.88:0/1704918162 shutdown_connections 2024-09-16T01:42:01.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.366+0000 7f9f25017640 1 --2- 172.21.15.88:0/1704918162 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9f0006c2e0 0x7f9f0006e7a0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:01.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.366+0000 7f9f25017640 1 --2- 172.21.15.88:0/1704918162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f2015b480 0x7f9f2014ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:01.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.366+0000 7f9f25017640 1 --2- 172.21.15.88:0/1704918162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f200b60c0 0x7f9f201557d0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:01.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.366+0000 7f9f25017640 1 --2- 172.21.15.88:0/1704918162 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f20072530 0x7f9f201537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:01.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.366+0000 7f9f25017640 1 -- 172.21.15.88:0/1704918162 >> 172.21.15.88:0/1704918162 conn(0x7f9f2006d0f0 msgr2=0x7f9f200d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:01.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.366+0000 7f9f25017640 1 -- 172.21.15.88:0/1704918162 shutdown_connections 2024-09-16T01:42:01.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.366+0000 7f9f25017640 1 -- 172.21.15.88:0/1704918162 wait complete. 2024-09-16T01:42:01.375 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:42:01.374+0000 7f35a9dca640 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x200: (22) Invalid argument 2024-09-16T01:42:01.375 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:42:01.374+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:42:01.376 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:42:01.374+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:42:01.382 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:01.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.480+0000 7fc8743c4640 1 Processor -- start 2024-09-16T01:42:01.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.480+0000 7fc8743c4640 1 -- start start 2024-09-16T01:42:01.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.481+0000 7fc8743c4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc86c073080 0x7fc86c0b5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:01.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.481+0000 7fc8743c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc86c15c670 0x7fc86c073460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:01.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.481+0000 7fc8743c4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc86c071750 con 0x7fc86c0b1990 2024-09-16T01:42:01.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.481+0000 7fc8743c4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc86c0d0c60 con 0x7fc86c073080 2024-09-16T01:42:01.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.481+0000 7fc8743c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc86c0d0370 con 0x7fc86c15c670 2024-09-16T01:42:01.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.482+0000 7fc872961640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc86c0b1990 0x7fc86c0d10f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:40006/0 (socket says 172.21.15.88:40006) 2024-09-16T01:42:01.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.482+0000 7fc872961640 1 -- 172.21.15.88:0/3429257103 learned_addr learned my addr 172.21.15.88:0/3429257103 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:01.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.482+0000 7fc872160640 1 --2- 172.21.15.88:0/3429257103 >> v2:172.21.15.154:3300/0 conn(0x7fc86c073080 0x7fc86c0b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:01.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.482+0000 7fc87195f640 1 --2- 172.21.15.88:0/3429257103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc86c15c670 0x7fc86c073460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:01.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.482+0000 7fc87115e640 1 -- 172.21.15.88:0/3429257103 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2766241244 0 0) 0x7fc86c071750 con 0x7fc86c0b1990 2024-09-16T01:42:01.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.482+0000 7fc87115e640 1 -- 172.21.15.88:0/3429257103 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc854003520 con 0x7fc86c0b1990 2024-09-16T01:42:01.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.482+0000 7fc87115e640 1 -- 172.21.15.88:0/3429257103 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 99418176 0 0) 0x7fc854003520 con 0x7fc86c0b1990 2024-09-16T01:42:01.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.482+0000 7fc872160640 1 -- 172.21.15.88:0/3429257103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc86c15c670 msgr2=0x7fc86c073460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:01.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.482+0000 7fc872160640 1 --2- 172.21.15.88:0/3429257103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc86c15c670 0x7fc86c073460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:01.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.482+0000 7fc872160640 1 -- 172.21.15.88:0/3429257103 >> v1:172.21.15.88:6789/0 conn(0x7fc86c0b1990 legacy=0x7fc86c0d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:01.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.482+0000 7fc872160640 1 -- 172.21.15.88:0/3429257103 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc86c0b59a0 con 0x7fc86c073080 2024-09-16T01:42:01.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.482+0000 7fc87195f640 1 --2- 172.21.15.88:0/3429257103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc86c15c670 0x7fc86c073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:42:01.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.483+0000 7fc872160640 1 --2- 172.21.15.88:0/3429257103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc86c073080 0x7fc86c0b5460 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc85c009d90 tx=0x7fc85c02a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1851636cad9021f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:01.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.483+0000 7fc87115e640 1 -- 172.21.15.88:0/3429257103 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc864001670 con 0x7fc86c073080 2024-09-16T01:42:01.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.483+0000 7fc87115e640 1 -- 172.21.15.88:0/3429257103 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc85c007540 con 0x7fc86c073080 2024-09-16T01:42:01.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.483+0000 7fc87115e640 1 -- 172.21.15.88:0/3429257103 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c0078f0 con 0x7fc86c073080 2024-09-16T01:42:01.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.483+0000 7fc8743c4640 1 -- 172.21.15.88:0/3429257103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc86c073080 msgr2=0x7fc86c0b5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:01.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.483+0000 7fc8743c4640 1 --2- 172.21.15.88:0/3429257103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc86c073080 0x7fc86c0b5460 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc85c009d90 tx=0x7fc85c02a5f0 comp rx=0 tx=0).stop 2024-09-16T01:42:01.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.484+0000 7fc8743c4640 1 -- 172.21.15.88:0/3429257103 shutdown_connections 2024-09-16T01:42:01.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.484+0000 7fc8743c4640 1 --2- 172.21.15.88:0/3429257103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc86c15c670 0x7fc86c073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:01.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.484+0000 7fc8743c4640 1 --2- 172.21.15.88:0/3429257103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc86c073080 0x7fc86c0b5460 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:01.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.484+0000 7fc8743c4640 1 -- 172.21.15.88:0/3429257103 >> 172.21.15.88:0/3429257103 conn(0x7fc86c0d24d0 msgr2=0x7fc86c0d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:01.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.484+0000 7fc8743c4640 1 -- 172.21.15.88:0/3429257103 shutdown_connections 2024-09-16T01:42:01.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.484+0000 7fc8743c4640 1 -- 172.21.15.88:0/3429257103 wait complete. 2024-09-16T01:42:01.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.485+0000 7fc8743c4640 1 Processor -- start 2024-09-16T01:42:01.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.485+0000 7fc8743c4640 1 -- start start 2024-09-16T01:42:01.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.486+0000 7fc8743c4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc86c073080 0x7fc86c15f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:01.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.486+0000 7fc8743c4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc86c0b1990 0x7fc86c147ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:01.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.486+0000 7fc8743c4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc86c15c670 0x7fc86c1f22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:01.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.486+0000 7fc8743c4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc86c071470 con 0x7fc86c073080 2024-09-16T01:42:01.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.486+0000 7fc8743c4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc86c071190 con 0x7fc86c0b1990 2024-09-16T01:42:01.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.486+0000 7fc8743c4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc86c0d0610 con 0x7fc86c15c670 2024-09-16T01:42:01.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.486+0000 7fc872160640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc86c073080 0x7fc86c15f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:01.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.486+0000 7fc872160640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc86c073080 0x7fc86c15f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44580/0 (socket says 172.21.15.88:44580) 2024-09-16T01:42:01.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.486+0000 7fc872160640 1 -- 172.21.15.88:0/3582624021 learned_addr learned my addr 172.21.15.88:0/3582624021 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:01.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.486+0000 7fc87195f640 1 --2- 172.21.15.88:0/3582624021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc86c0b1990 0x7fc86c147ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:01.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.486+0000 7fc872961640 1 --2- 172.21.15.88:0/3582624021 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc86c15c670 0x7fc86c1f22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:01.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.486+0000 7fc872160640 1 -- 172.21.15.88:0/3582624021 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc86c15c670 msgr2=0x7fc86c1f22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:01.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.486+0000 7fc872160640 1 --2- 172.21.15.88:0/3582624021 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc86c15c670 0x7fc86c1f22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:01.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.486+0000 7fc872160640 1 -- 172.21.15.88:0/3582624021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc86c0b1990 msgr2=0x7fc86c147ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:01.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.486+0000 7fc872160640 1 --2- 172.21.15.88:0/3582624021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc86c0b1990 0x7fc86c147ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:01.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.486+0000 7fc872160640 1 -- 172.21.15.88:0/3582624021 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc86c15f970 con 0x7fc86c073080 2024-09-16T01:42:01.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.486+0000 7fc872160640 1 --2- 172.21.15.88:0/3582624021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc86c073080 0x7fc86c15f430 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fc85c009eb0 tx=0x7fc85c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:01.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.486+0000 7fc86affd640 1 -- 172.21.15.88:0/3582624021 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c009450 con 0x7fc86c073080 2024-09-16T01:42:01.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.486+0000 7fc86affd640 1 -- 172.21.15.88:0/3582624021 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc85c004240 con 0x7fc86c073080 2024-09-16T01:42:01.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.486+0000 7fc86affd640 1 -- 172.21.15.88:0/3582624021 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c0045f0 con 0x7fc86c073080 2024-09-16T01:42:01.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.486+0000 7fc8743c4640 1 -- 172.21.15.88:0/3582624021 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc86c155640 con 0x7fc86c073080 2024-09-16T01:42:01.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.486+0000 7fc8743c4640 1 -- 172.21.15.88:0/3582624021 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc86c155c00 con 0x7fc86c073080 2024-09-16T01:42:01.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.487+0000 7fc8743c4640 1 -- 172.21.15.88:0/3582624021 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc86c0d08f0 con 0x7fc86c073080 2024-09-16T01:42:01.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.488+0000 7fc86affd640 1 -- 172.21.15.88:0/3582624021 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc85c039040 con 0x7fc86c073080 2024-09-16T01:42:01.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.488+0000 7fc86affd640 1 --2- 172.21.15.88:0/3582624021 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc85806c2b0 0x7fc85806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:01.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.488+0000 7fc86affd640 1 -- 172.21.15.88:0/3582624021 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 7200+0+0 (secure 0 0 0) 0x7fc85c0acea0 con 0x7fc86c073080 2024-09-16T01:42:01.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.488+0000 7fc87195f640 1 --2- 172.21.15.88:0/3582624021 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc85806c2b0 0x7fc85806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:01.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.489+0000 7fc87195f640 1 --2- 172.21.15.88:0/3582624021 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc85806c2b0 0x7fc85806e770 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fc86c0d2470 tx=0x7fc860009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:01.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.493+0000 7fc86affd640 1 -- 172.21.15.88:0/3582624021 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc85c040030 con 0x7fc86c073080 2024-09-16T01:42:01.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.692+0000 7fc8743c4640 1 -- 172.21.15.88:0/3582624021 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc86c0d3050 con 0x7fc86c073080 2024-09-16T01:42:01.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.693+0000 7fc86affd640 1 -- 172.21.15.88:0/3582624021 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) ==== 95+0+3566 (secure 0 0 0) 0x7fc85c07a9f0 con 0x7fc86c073080 2024-09-16T01:42:01.695 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:01.695 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":169,"btime":"2024-09-16T01:42:01:375610+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:01.375559+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:01.695 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 169 2024-09-16T01:42:01.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.695+0000 7fc8743c4640 1 -- 172.21.15.88:0/3582624021 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc85806c2b0 msgr2=0x7fc85806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:01.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.695+0000 7fc8743c4640 1 --2- 172.21.15.88:0/3582624021 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc85806c2b0 0x7fc85806e770 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fc86c0d2470 tx=0x7fc860009190 comp rx=0 tx=0).stop 2024-09-16T01:42:01.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.695+0000 7fc8743c4640 1 -- 172.21.15.88:0/3582624021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc86c073080 msgr2=0x7fc86c15f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:01.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.695+0000 7fc8743c4640 1 --2- 172.21.15.88:0/3582624021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc86c073080 0x7fc86c15f430 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fc85c009eb0 tx=0x7fc85c004a40 comp rx=0 tx=0).stop 2024-09-16T01:42:01.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.695+0000 7fc8743c4640 1 -- 172.21.15.88:0/3582624021 shutdown_connections 2024-09-16T01:42:01.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.695+0000 7fc8743c4640 1 --2- 172.21.15.88:0/3582624021 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc85806c2b0 0x7fc85806e770 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:01.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.695+0000 7fc8743c4640 1 --2- 172.21.15.88:0/3582624021 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc86c15c670 0x7fc86c1f22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:01.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.695+0000 7fc8743c4640 1 --2- 172.21.15.88:0/3582624021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc86c0b1990 0x7fc86c147ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:01.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.695+0000 7fc8743c4640 1 --2- 172.21.15.88:0/3582624021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc86c073080 0x7fc86c15f430 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:01.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.695+0000 7fc8743c4640 1 -- 172.21.15.88:0/3582624021 >> 172.21.15.88:0/3582624021 conn(0x7fc86c0d24d0 msgr2=0x7fc86c15ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:01.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.695+0000 7fc8743c4640 1 -- 172.21.15.88:0/3582624021 shutdown_connections 2024-09-16T01:42:01.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.695+0000 7fc8743c4640 1 -- 172.21.15.88:0/3582624021 wait complete. 2024-09-16T01:42:01.707 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:42:01.708 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:01.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.804+0000 7f2367aaf640 1 Processor -- start 2024-09-16T01:42:01.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.804+0000 7f2367aaf640 1 -- start start 2024-09-16T01:42:01.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.805+0000 7f2367aaf640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2360155800 0x7f23600b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:01.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.805+0000 7f2367aaf640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f23600cfc80 0x7f23600b4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:01.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.805+0000 7f2367aaf640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f236015c770 con 0x7f236015c330 2024-09-16T01:42:01.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.805+0000 7f2367aaf640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f23600b4e60 con 0x7f23600cfc80 2024-09-16T01:42:01.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.805+0000 7f2367aaf640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f23600b50a0 con 0x7f2360155800 2024-09-16T01:42:01.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.805+0000 7f236604c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f23600cfc80 0x7f23600b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:01.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.805+0000 7f236604c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f23600cfc80 0x7f23600b4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44592/0 (socket says 172.21.15.88:44592) 2024-09-16T01:42:01.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.805+0000 7f236604c640 1 -- 172.21.15.88:0/2684397307 learned_addr learned my addr 172.21.15.88:0/2684397307 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:01.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.806+0000 7f236584b640 1 --2- 172.21.15.88:0/2684397307 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2360155800 0x7f23600b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:01.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.806+0000 7f236604c640 1 -- 172.21.15.88:0/2684397307 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2360155800 msgr2=0x7f23600b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:01.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.806+0000 7f236604c640 1 --2- 172.21.15.88:0/2684397307 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2360155800 0x7f23600b6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:01.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.806+0000 7f236604c640 1 -- 172.21.15.88:0/2684397307 >> v1:172.21.15.88:6789/0 conn(0x7f236015c330 legacy=0x7f23600cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:01.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.806+0000 7f236604c640 1 -- 172.21.15.88:0/2684397307 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23600b5380 con 0x7f23600cfc80 2024-09-16T01:42:01.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.806+0000 7f2364849640 1 -- 172.21.15.88:0/2684397307 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4065146093 0 0) 0x7f2350001110 con 0x7f236015c330 2024-09-16T01:42:01.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.806+0000 7f236604c640 1 --2- 172.21.15.88:0/2684397307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23600cfc80 0x7f23600b4920 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f2358009e50 tx=0x7f235802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ac9c4acfdf8c4bb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:01.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.806+0000 7f2364849640 1 -- 172.21.15.88:0/2684397307 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2358036070 con 0x7f23600cfc80 2024-09-16T01:42:01.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.806+0000 7f2364849640 1 -- 172.21.15.88:0/2684397307 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2358031600 con 0x7f23600cfc80 2024-09-16T01:42:01.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.806+0000 7f2364849640 1 -- 172.21.15.88:0/2684397307 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23580319d0 con 0x7f23600cfc80 2024-09-16T01:42:01.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.807+0000 7f2367aaf640 1 -- 172.21.15.88:0/2684397307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23600cfc80 msgr2=0x7f23600b4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:01.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.807+0000 7f2367aaf640 1 --2- 172.21.15.88:0/2684397307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23600cfc80 0x7f23600b4920 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f2358009e50 tx=0x7f235802a710 comp rx=0 tx=0).stop 2024-09-16T01:42:01.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.807+0000 7f2367aaf640 1 -- 172.21.15.88:0/2684397307 shutdown_connections 2024-09-16T01:42:01.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.807+0000 7f2367aaf640 1 --2- 172.21.15.88:0/2684397307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f23600cfc80 0x7f23600b4920 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:01.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.807+0000 7f2367aaf640 1 --2- 172.21.15.88:0/2684397307 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2360155800 0x7f23600b6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:01.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.807+0000 7f2367aaf640 1 -- 172.21.15.88:0/2684397307 >> 172.21.15.88:0/2684397307 conn(0x7f236006bc10 msgr2=0x7f236006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:01.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.807+0000 7f2367aaf640 1 -- 172.21.15.88:0/2684397307 shutdown_connections 2024-09-16T01:42:01.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.807+0000 7f2367aaf640 1 -- 172.21.15.88:0/2684397307 wait complete. 2024-09-16T01:42:01.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.808+0000 7f2367aaf640 1 Processor -- start 2024-09-16T01:42:01.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.808+0000 7f2367aaf640 1 -- start start 2024-09-16T01:42:01.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.809+0000 7f2367aaf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23600cfc80 0x7f2360145e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:01.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.809+0000 7f2367aaf640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2360155800 0x7f2360157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:01.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.809+0000 7f2367aaf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f236015c330 0x7f2360157e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:01.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.809+0000 7f2367aaf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f23600b5c20 con 0x7f236015c330 2024-09-16T01:42:01.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.809+0000 7f2367aaf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f23600b5940 con 0x7f23600cfc80 2024-09-16T01:42:01.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.809+0000 7f2367aaf640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2360155be0 con 0x7f2360155800 2024-09-16T01:42:01.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.809+0000 7f236604c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f236015c330 0x7f2360157e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:01.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.809+0000 7f236604c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f236015c330 0x7f2360157e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44602/0 (socket says 172.21.15.88:44602) 2024-09-16T01:42:01.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.809+0000 7f236604c640 1 -- 172.21.15.88:0/3219424382 learned_addr learned my addr 172.21.15.88:0/3219424382 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:01.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.809+0000 7f236584b640 1 --2- 172.21.15.88:0/3219424382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23600cfc80 0x7f2360145e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:01.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.809+0000 7f236504a640 1 --2- 172.21.15.88:0/3219424382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2360155800 0x7f2360157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:01.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.809+0000 7f236604c640 1 -- 172.21.15.88:0/3219424382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2360155800 msgr2=0x7f2360157900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:01.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.809+0000 7f236604c640 1 --2- 172.21.15.88:0/3219424382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2360155800 0x7f2360157900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:01.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.809+0000 7f236604c640 1 -- 172.21.15.88:0/3219424382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23600cfc80 msgr2=0x7f2360145e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:01.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.809+0000 7f236604c640 1 --2- 172.21.15.88:0/3219424382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23600cfc80 0x7f2360145e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:01.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.809+0000 7f236604c640 1 -- 172.21.15.88:0/3219424382 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2360153e90 con 0x7f236015c330 2024-09-16T01:42:01.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.810+0000 7f236604c640 1 --2- 172.21.15.88:0/3219424382 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f236015c330 0x7f2360157e40 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f2358009f70 tx=0x7f2358004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:01.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.810+0000 7f23567fc640 1 -- 172.21.15.88:0/3219424382 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2358036050 con 0x7f236015c330 2024-09-16T01:42:01.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.810+0000 7f23567fc640 1 -- 172.21.15.88:0/3219424382 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2358042070 con 0x7f236015c330 2024-09-16T01:42:01.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.810+0000 7f2367aaf640 1 -- 172.21.15.88:0/3219424382 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2360154170 con 0x7f236015c330 2024-09-16T01:42:01.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.810+0000 7f23567fc640 1 -- 172.21.15.88:0/3219424382 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2358009510 con 0x7f236015c330 2024-09-16T01:42:01.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.810+0000 7f2367aaf640 1 -- 172.21.15.88:0/3219424382 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2360154730 con 0x7f236015c330 2024-09-16T01:42:01.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.811+0000 7f2367aaf640 1 -- 172.21.15.88:0/3219424382 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23600b4920 con 0x7f236015c330 2024-09-16T01:42:01.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.811+0000 7f23567fc640 1 -- 172.21.15.88:0/3219424382 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2358007bf0 con 0x7f236015c330 2024-09-16T01:42:01.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.812+0000 7f23567fc640 1 --2- 172.21.15.88:0/3219424382 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f234406c2b0 0x7f234406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:01.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.812+0000 7f23567fc640 1 -- 172.21.15.88:0/3219424382 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 7200+0+0 (secure 0 0 0) 0x7f23580ac830 con 0x7f236015c330 2024-09-16T01:42:01.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.812+0000 7f236584b640 1 --2- 172.21.15.88:0/3219424382 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f234406c2b0 0x7f234406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:01.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.812+0000 7f236584b640 1 --2- 172.21.15.88:0/3219424382 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f234406c2b0 0x7f234406e770 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f235c006fd0 tx=0x7f235c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:01.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:01.816+0000 7f23567fc640 1 -- 172.21.15.88:0/3219424382 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f235807aa60 con 0x7f236015c330 2024-09-16T01:42:02.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.017+0000 7f2367aaf640 1 -- 172.21.15.88:0/3219424382 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f23600b60b0 con 0x7f236015c330 2024-09-16T01:42:02.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.018+0000 7f23567fc640 1 -- 172.21.15.88:0/3219424382 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) ==== 95+0+3566 (secure 0 0 0) 0x7f235807a360 con 0x7f236015c330 2024-09-16T01:42:02.339 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:02.340 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":169,"btime":"2024-09-16T01:42:01:375610+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:01.375559+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:02.344 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 169 2024-09-16T01:42:02.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.019+0000 7f2367aaf640 1 -- 172.21.15.88:0/3219424382 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f234406c2b0 msgr2=0x7f234406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:02.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.019+0000 7f2367aaf640 1 --2- 172.21.15.88:0/3219424382 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f234406c2b0 0x7f234406e770 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f235c006fd0 tx=0x7f235c008040 comp rx=0 tx=0).stop 2024-09-16T01:42:02.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.019+0000 7f2367aaf640 1 -- 172.21.15.88:0/3219424382 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f236015c330 msgr2=0x7f2360157e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:02.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.019+0000 7f2367aaf640 1 --2- 172.21.15.88:0/3219424382 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f236015c330 0x7f2360157e40 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f2358009f70 tx=0x7f2358004a40 comp rx=0 tx=0).stop 2024-09-16T01:42:02.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.020+0000 7f2367aaf640 1 -- 172.21.15.88:0/3219424382 shutdown_connections 2024-09-16T01:42:02.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.020+0000 7f2367aaf640 1 --2- 172.21.15.88:0/3219424382 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f234406c2b0 0x7f234406e770 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:02.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.020+0000 7f2367aaf640 1 --2- 172.21.15.88:0/3219424382 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f236015c330 0x7f2360157e40 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:02.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.020+0000 7f2367aaf640 1 --2- 172.21.15.88:0/3219424382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2360155800 0x7f2360157900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:02.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.020+0000 7f2367aaf640 1 --2- 172.21.15.88:0/3219424382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23600cfc80 0x7f2360145e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:02.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.020+0000 7f2367aaf640 1 -- 172.21.15.88:0/3219424382 >> 172.21.15.88:0/3219424382 conn(0x7f236006bc10 msgr2=0x7f236015ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:02.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.020+0000 7f2367aaf640 1 -- 172.21.15.88:0/3219424382 shutdown_connections 2024-09-16T01:42:02.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.020+0000 7f2367aaf640 1 -- 172.21.15.88:0/3219424382 wait complete. 2024-09-16T01:42:02.346 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:42:02.346 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:02.347 INFO:tasks.ceph.mds.a.smithi088.stderr: -35> 2024-09-16T01:41:57.357+0000 7fba44961640 -1 mds.0.cache.ino(0x1) Corrupt inode 0x1: End of buffer [buffer:2] 2024-09-16T01:42:02.348 INFO:tasks.ceph.mds.a.smithi088.stderr: -33> 2024-09-16T01:41:57.357+0000 7fba44961640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (22) Invalid argument 2024-09-16T01:42:02.348 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:42:02.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.448+0000 7f885dc8d640 1 Processor -- start 2024-09-16T01:42:02.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.448+0000 7f885dc8d640 1 -- start start 2024-09-16T01:42:02.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.448+0000 7f885dc8d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f885815e800 0x7f88580b5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:02.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.448+0000 7f885dc8d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f885815ebe0 0x7f885815f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:02.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.448+0000 7f885dc8d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f885806ea60 con 0x7f88580b6e50 2024-09-16T01:42:02.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.448+0000 7f885dc8d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f885806ed40 con 0x7f885815e800 2024-09-16T01:42:02.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.449+0000 7f885dc8d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f885806cde0 con 0x7f885815ebe0 2024-09-16T01:42:02.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.449+0000 7f88577fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f885815e800 0x7f88580b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:02.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.449+0000 7f88577fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f885815e800 0x7f88580b5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44636/0 (socket says 172.21.15.88:44636) 2024-09-16T01:42:02.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.449+0000 7f88577fe640 1 -- 172.21.15.88:0/2070815568 learned_addr learned my addr 172.21.15.88:0/2070815568 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:02.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.449+0000 7f8856ffd640 1 --2- 172.21.15.88:0/2070815568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f885815ebe0 0x7f885815f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:02.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.449+0000 7f88577fe640 1 -- 172.21.15.88:0/2070815568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f885815ebe0 msgr2=0x7f885815f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:02.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.449+0000 7f88577fe640 1 --2- 172.21.15.88:0/2070815568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f885815ebe0 0x7f885815f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:02.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.449+0000 7f88577fe640 1 -- 172.21.15.88:0/2070815568 >> v1:172.21.15.154:6789/0 conn(0x7f88580b6e50 legacy=0x7f88580b7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:02.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.449+0000 7f88577fe640 1 -- 172.21.15.88:0/2070815568 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f885806d0c0 con 0x7f885815e800 2024-09-16T01:42:02.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.450+0000 7f88577fe640 1 --2- 172.21.15.88:0/2070815568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f885815e800 0x7f88580b5460 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f8848009870 tx=0x7f884802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a07fc1f2c52fd96b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:02.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.450+0000 7f88567fc640 1 -- 172.21.15.88:0/2070815568 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8848037070 con 0x7f885815e800 2024-09-16T01:42:02.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.450+0000 7f88567fc640 1 -- 172.21.15.88:0/2070815568 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8848032440 con 0x7f885815e800 2024-09-16T01:42:02.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.450+0000 7f88567fc640 1 -- 172.21.15.88:0/2070815568 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8848032810 con 0x7f885815e800 2024-09-16T01:42:02.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.450+0000 7f885dc8d640 1 -- 172.21.15.88:0/2070815568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f885815e800 msgr2=0x7f88580b5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:02.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.450+0000 7f885dc8d640 1 --2- 172.21.15.88:0/2070815568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f885815e800 0x7f88580b5460 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f8848009870 tx=0x7f884802a270 comp rx=0 tx=0).stop 2024-09-16T01:42:02.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.450+0000 7f885dc8d640 1 -- 172.21.15.88:0/2070815568 shutdown_connections 2024-09-16T01:42:02.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.450+0000 7f885dc8d640 1 --2- 172.21.15.88:0/2070815568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f885815ebe0 0x7f885815f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:02.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.450+0000 7f885dc8d640 1 --2- 172.21.15.88:0/2070815568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f885815e800 0x7f88580b5460 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:02.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.450+0000 7f885dc8d640 1 -- 172.21.15.88:0/2070815568 >> 172.21.15.88:0/2070815568 conn(0x7f88580d24d0 msgr2=0x7f88580d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:02.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.450+0000 7f885dc8d640 1 -- 172.21.15.88:0/2070815568 shutdown_connections 2024-09-16T01:42:02.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.450+0000 7f885dc8d640 1 -- 172.21.15.88:0/2070815568 wait complete. 2024-09-16T01:42:02.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.452+0000 7f885dc8d640 1 Processor -- start 2024-09-16T01:42:02.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.452+0000 7f885dc8d640 1 -- start start 2024-09-16T01:42:02.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.452+0000 7f885dc8d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88580b6e50 0x7f8858146e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:02.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.452+0000 7f885dc8d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f885815e800 0x7f8858151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:02.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.452+0000 7f885dc8d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f885815ebe0 0x7f88581524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:02.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.452+0000 7f885dc8d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f885806e680 con 0x7f885815e800 2024-09-16T01:42:02.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.452+0000 7f885dc8d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f88580d31e0 con 0x7f88580b6e50 2024-09-16T01:42:02.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.452+0000 7f885dc8d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f885806e800 con 0x7f885815ebe0 2024-09-16T01:42:02.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.453+0000 7f8856ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f885815e800 0x7f8858151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:02.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.453+0000 7f8856ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f885815e800 0x7f8858151fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44640/0 (socket says 172.21.15.88:44640) 2024-09-16T01:42:02.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.453+0000 7f8856ffd640 1 -- 172.21.15.88:0/2000378467 learned_addr learned my addr 172.21.15.88:0/2000378467 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:02.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.453+0000 7f88577fe640 1 --2- 172.21.15.88:0/2000378467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88580b6e50 0x7f8858146e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:02.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.453+0000 7f8857fff640 1 --2- 172.21.15.88:0/2000378467 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f885815ebe0 0x7f88581524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:02.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.453+0000 7f8856ffd640 1 -- 172.21.15.88:0/2000378467 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f885815ebe0 msgr2=0x7f88581524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:02.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.453+0000 7f8856ffd640 1 --2- 172.21.15.88:0/2000378467 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f885815ebe0 0x7f88581524e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:02.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.453+0000 7f8856ffd640 1 -- 172.21.15.88:0/2000378467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88580b6e50 msgr2=0x7f8858146e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:02.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.453+0000 7f8856ffd640 1 --2- 172.21.15.88:0/2000378467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88580b6e50 0x7f8858146e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:02.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.453+0000 7f8856ffd640 1 -- 172.21.15.88:0/2000378467 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8858150bd0 con 0x7f885815e800 2024-09-16T01:42:02.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.453+0000 7f8857fff640 1 --2- 172.21.15.88:0/2000378467 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f885815ebe0 0x7f88581524e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:02.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.453+0000 7f8856ffd640 1 --2- 172.21.15.88:0/2000378467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f885815e800 0x7f8858151fa0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f884c007e70 tx=0x7f884c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:02.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.453+0000 7f883ffff640 1 -- 172.21.15.88:0/2000378467 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c014420 con 0x7f885815e800 2024-09-16T01:42:02.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.453+0000 7f883ffff640 1 -- 172.21.15.88:0/2000378467 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f884c019070 con 0x7f885815e800 2024-09-16T01:42:02.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.453+0000 7f885dc8d640 1 -- 172.21.15.88:0/2000378467 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8858152b10 con 0x7f885815e800 2024-09-16T01:42:02.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.453+0000 7f883ffff640 1 -- 172.21.15.88:0/2000378467 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c014800 con 0x7f885815e800 2024-09-16T01:42:02.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.453+0000 7f885dc8d640 1 -- 172.21.15.88:0/2000378467 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88581533b0 con 0x7f885815e800 2024-09-16T01:42:02.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.454+0000 7f885dc8d640 1 -- 172.21.15.88:0/2000378467 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f885815f070 con 0x7f885815e800 2024-09-16T01:42:02.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.455+0000 7f883ffff640 1 -- 172.21.15.88:0/2000378467 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f884c0149a0 con 0x7f885815e800 2024-09-16T01:42:02.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.455+0000 7f883ffff640 1 --2- 172.21.15.88:0/2000378467 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f884406c2b0 0x7f884406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:02.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.455+0000 7f883ffff640 1 -- 172.21.15.88:0/2000378467 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 7200+0+0 (secure 0 0 0) 0x7f884c08c150 con 0x7f885815e800 2024-09-16T01:42:02.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.455+0000 7f88577fe640 1 --2- 172.21.15.88:0/2000378467 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f884406c2b0 0x7f884406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:02.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.456+0000 7f88577fe640 1 --2- 172.21.15.88:0/2000378467 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f884406c2b0 0x7f884406e770 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f8848034000 tx=0x7f884802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:02.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.459+0000 7f883ffff640 1 -- 172.21.15.88:0/2000378467 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f884c05a2f0 con 0x7f885815e800 2024-09-16T01:42:02.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.662+0000 7f885dc8d640 1 -- 172.21.15.88:0/2000378467 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88580b1910 con 0x7f885815e800 2024-09-16T01:42:02.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.662+0000 7f883ffff640 1 -- 172.21.15.88:0/2000378467 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) ==== 95+0+4392 (secure 0 0 0) 0x7f884c059bf0 con 0x7f885815e800 2024-09-16T01:42:02.664 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:02.664 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":170,"btime":"2024-09-16T01:42:02:376886+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3728363359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3728363359},{"type":"v1","addr":"172.21.15.88:6836","nonce":3728363359}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":170}],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:01.375559+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":104,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:02.664 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 170 2024-09-16T01:42:02.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.664+0000 7f885dc8d640 1 -- 172.21.15.88:0/2000378467 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f884406c2b0 msgr2=0x7f884406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:02.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.664+0000 7f885dc8d640 1 --2- 172.21.15.88:0/2000378467 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f884406c2b0 0x7f884406e770 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f8848034000 tx=0x7f884802e000 comp rx=0 tx=0).stop 2024-09-16T01:42:02.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.664+0000 7f885dc8d640 1 -- 172.21.15.88:0/2000378467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f885815e800 msgr2=0x7f8858151fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:02.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.664+0000 7f885dc8d640 1 --2- 172.21.15.88:0/2000378467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f885815e800 0x7f8858151fa0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f884c007e70 tx=0x7f884c00e4b0 comp rx=0 tx=0).stop 2024-09-16T01:42:02.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.664+0000 7f885dc8d640 1 -- 172.21.15.88:0/2000378467 shutdown_connections 2024-09-16T01:42:02.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.664+0000 7f885dc8d640 1 --2- 172.21.15.88:0/2000378467 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f884406c2b0 0x7f884406e770 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:02.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.664+0000 7f885dc8d640 1 --2- 172.21.15.88:0/2000378467 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f885815ebe0 0x7f88581524e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:02.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.664+0000 7f885dc8d640 1 --2- 172.21.15.88:0/2000378467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f885815e800 0x7f8858151fa0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:02.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.664+0000 7f885dc8d640 1 --2- 172.21.15.88:0/2000378467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88580b6e50 0x7f8858146e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:02.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.664+0000 7f885dc8d640 1 -- 172.21.15.88:0/2000378467 >> 172.21.15.88:0/2000378467 conn(0x7f88580d24d0 msgr2=0x7f88580b32c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:02.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.665+0000 7f885dc8d640 1 -- 172.21.15.88:0/2000378467 shutdown_connections 2024-09-16T01:42:02.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.665+0000 7f885dc8d640 1 -- 172.21.15.88:0/2000378467 wait complete. 2024-09-16T01:42:02.677 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt 200.00000000' led to DAMAGED state 2024-09-16T01:42:02.677 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt 200.00000001' 2024-09-16T01:42:02.677 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:42:02.677 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:42:02.693 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:42:02.693 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:42:02.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:02.693 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:42:02.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:42:02.749 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:42:02.749 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:42:02.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.887+0000 7f7e5724c640 1 Processor -- start 2024-09-16T01:42:02.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.887+0000 7f7e5724c640 1 -- start start 2024-09-16T01:42:02.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.888+0000 7f7e5724c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7e500b5470 0x7f7e5015f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:02.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.888+0000 7f7e5724c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7e5015cc20 0x7f7e5015d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:02.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.888+0000 7f7e5724c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e5015bb60 con 0x7f7e50155880 2024-09-16T01:42:02.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.888+0000 7f7e5724c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7e50155c60 con 0x7f7e5015cc20 2024-09-16T01:42:02.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.888+0000 7f7e5724c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7e5015fcb0 con 0x7f7e500b5470 2024-09-16T01:42:02.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.888+0000 7f7e557e9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7e5015cc20 0x7f7e5015d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:02.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.888+0000 7f7e54fe8640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7e50155880 0x7f7e500b4d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:40014/0 (socket says 172.21.15.88:40014) 2024-09-16T01:42:02.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.888+0000 7f7e557e9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7e5015cc20 0x7f7e5015d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44652/0 (socket says 172.21.15.88:44652) 2024-09-16T01:42:02.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.888+0000 7f7e54fe8640 1 -- 172.21.15.88:0/2234101408 learned_addr learned my addr 172.21.15.88:0/2234101408 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:02.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.888+0000 7f7e4ffff640 1 --2- 172.21.15.88:0/2234101408 >> v2:172.21.15.154:3300/0 conn(0x7f7e500b5470 0x7f7e5015f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:02.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.889+0000 7f7e557e9640 1 -- 172.21.15.88:0/2234101408 >> v2:172.21.15.154:3300/0 conn(0x7f7e500b5470 msgr2=0x7f7e5015f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:02.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.889+0000 7f7e557e9640 1 --2- 172.21.15.88:0/2234101408 >> v2:172.21.15.154:3300/0 conn(0x7f7e500b5470 0x7f7e5015f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:02.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.889+0000 7f7e557e9640 1 -- 172.21.15.88:0/2234101408 >> v1:172.21.15.88:6789/0 conn(0x7f7e50155880 legacy=0x7f7e500b4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:02.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.889+0000 7f7e4f7fe640 1 -- 172.21.15.88:0/2234101408 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1016703421 0 0) 0x7f7e5015bb60 con 0x7f7e50155880 2024-09-16T01:42:02.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.889+0000 7f7e557e9640 1 -- 172.21.15.88:0/2234101408 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e5015be40 con 0x7f7e5015cc20 2024-09-16T01:42:02.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.889+0000 7f7e557e9640 1 --2- 172.21.15.88:0/2234101408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e5015cc20 0x7f7e5015d0d0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f7e501592f0 tx=0x7f7e3802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82c94692cd840e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:02.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.889+0000 7f7e4f7fe640 1 -- 172.21.15.88:0/2234101408 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e38037070 con 0x7f7e5015cc20 2024-09-16T01:42:02.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.889+0000 7f7e4f7fe640 1 -- 172.21.15.88:0/2234101408 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e38032440 con 0x7f7e5015cc20 2024-09-16T01:42:02.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.889+0000 7f7e4f7fe640 1 -- 172.21.15.88:0/2234101408 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e38032810 con 0x7f7e5015cc20 2024-09-16T01:42:02.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.890+0000 7f7e5724c640 1 -- 172.21.15.88:0/2234101408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e5015cc20 msgr2=0x7f7e5015d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:02.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.890+0000 7f7e5724c640 1 --2- 172.21.15.88:0/2234101408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e5015cc20 0x7f7e5015d0d0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f7e501592f0 tx=0x7f7e3802a1a0 comp rx=0 tx=0).stop 2024-09-16T01:42:02.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.890+0000 7f7e5724c640 1 -- 172.21.15.88:0/2234101408 shutdown_connections 2024-09-16T01:42:02.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.890+0000 7f7e5724c640 1 --2- 172.21.15.88:0/2234101408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e5015cc20 0x7f7e5015d0d0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:02.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.890+0000 7f7e5724c640 1 --2- 172.21.15.88:0/2234101408 >> v2:172.21.15.154:3300/0 conn(0x7f7e500b5470 0x7f7e5015f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:02.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.890+0000 7f7e5724c640 1 -- 172.21.15.88:0/2234101408 >> 172.21.15.88:0/2234101408 conn(0x7f7e500d1390 msgr2=0x7f7e500d17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:02.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.890+0000 7f7e5724c640 1 -- 172.21.15.88:0/2234101408 shutdown_connections 2024-09-16T01:42:02.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.890+0000 7f7e5724c640 1 -- 172.21.15.88:0/2234101408 wait complete. 2024-09-16T01:42:02.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.891+0000 7f7e5724c640 1 Processor -- start 2024-09-16T01:42:02.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.891+0000 7f7e5724c640 1 -- start start 2024-09-16T01:42:02.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.892+0000 7f7e5724c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e500b5470 0x7f7e500ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:02.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.892+0000 7f7e5724c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e50155880 0x7f7e50150810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:02.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.892+0000 7f7e5724c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e5015cc20 0x7f7e500d0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:02.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.892+0000 7f7e5724c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7e5015c6e0 con 0x7f7e5015cc20 2024-09-16T01:42:02.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.892+0000 7f7e5724c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7e5015c400 con 0x7f7e50155880 2024-09-16T01:42:02.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.892+0000 7f7e5724c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7e5015b770 con 0x7f7e500b5470 2024-09-16T01:42:02.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.892+0000 7f7e557e9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e5015cc20 0x7f7e500d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:02.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.892+0000 7f7e557e9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e5015cc20 0x7f7e500d0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44654/0 (socket says 172.21.15.88:44654) 2024-09-16T01:42:02.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.892+0000 7f7e557e9640 1 -- 172.21.15.88:0/4169127806 learned_addr learned my addr 172.21.15.88:0/4169127806 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:02.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.892+0000 7f7e557e9640 1 -- 172.21.15.88:0/4169127806 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e500b5470 msgr2=0x7f7e500ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:02.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.892+0000 7f7e557e9640 1 --2- 172.21.15.88:0/4169127806 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e500b5470 0x7f7e500ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:02.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.892+0000 7f7e557e9640 1 -- 172.21.15.88:0/4169127806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e50155880 msgr2=0x7f7e50150810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:02.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.892+0000 7f7e557e9640 1 --2- 172.21.15.88:0/4169127806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e50155880 0x7f7e50150810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:02.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.892+0000 7f7e557e9640 1 -- 172.21.15.88:0/4169127806 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e50150d50 con 0x7f7e5015cc20 2024-09-16T01:42:02.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.893+0000 7f7e557e9640 1 --2- 172.21.15.88:0/4169127806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e5015cc20 0x7f7e500d0070 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f7e38009890 tx=0x7f7e380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:02.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.893+0000 7f7e4d7fa640 1 -- 172.21.15.88:0/4169127806 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e38037050 con 0x7f7e5015cc20 2024-09-16T01:42:02.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.893+0000 7f7e4d7fa640 1 -- 172.21.15.88:0/4169127806 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e38038040 con 0x7f7e5015cc20 2024-09-16T01:42:02.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.893+0000 7f7e5724c640 1 -- 172.21.15.88:0/4169127806 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e500cf0e0 con 0x7f7e5015cc20 2024-09-16T01:42:02.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.893+0000 7f7e4d7fa640 1 -- 172.21.15.88:0/4169127806 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e38004450 con 0x7f7e5015cc20 2024-09-16T01:42:02.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.893+0000 7f7e5724c640 1 -- 172.21.15.88:0/4169127806 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e50156270 con 0x7f7e5015cc20 2024-09-16T01:42:02.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.894+0000 7f7e5724c640 1 -- 172.21.15.88:0/4169127806 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e500cd8a0 con 0x7f7e5015cc20 2024-09-16T01:42:02.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.894+0000 7f7e4d7fa640 1 -- 172.21.15.88:0/4169127806 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7e38030980 con 0x7f7e5015cc20 2024-09-16T01:42:02.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.895+0000 7f7e4d7fa640 1 --2- 172.21.15.88:0/4169127806 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7e3006c2b0 0x7f7e3006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:02.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.895+0000 7f7e54fe8640 1 --2- 172.21.15.88:0/4169127806 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7e3006c2b0 0x7f7e3006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:02.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.895+0000 7f7e4d7fa640 1 -- 172.21.15.88:0/4169127806 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 7200+0+0 (secure 0 0 0) 0x7f7e380ac500 con 0x7f7e5015cc20 2024-09-16T01:42:02.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.895+0000 7f7e54fe8640 1 --2- 172.21.15.88:0/4169127806 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7e3006c2b0 0x7f7e3006e770 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f7e48007fd0 tx=0x7f7e48007920 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:02.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:02.899+0000 7f7e4d7fa640 1 -- 172.21.15.88:0/4169127806 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7e38032e20 con 0x7f7e5015cc20 2024-09-16T01:42:03.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.100+0000 7f7e5724c640 1 -- 172.21.15.88:0/4169127806 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f7e500d1140 con 0x7f7e5015cc20 2024-09-16T01:42:03.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.383+0000 7f7e4d7fa640 1 -- 172.21.15.88:0/4169127806 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v171) ==== 161+0+0 (secure 0 0 0) 0x7f7e3807a850 con 0x7f7e5015cc20 2024-09-16T01:42:03.384 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:42:03.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.385+0000 7f7e5724c640 1 -- 172.21.15.88:0/4169127806 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7e3006c2b0 msgr2=0x7f7e3006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:03.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.385+0000 7f7e5724c640 1 --2- 172.21.15.88:0/4169127806 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7e3006c2b0 0x7f7e3006e770 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f7e48007fd0 tx=0x7f7e48007920 comp rx=0 tx=0).stop 2024-09-16T01:42:03.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.385+0000 7f7e5724c640 1 -- 172.21.15.88:0/4169127806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e5015cc20 msgr2=0x7f7e500d0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:03.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.385+0000 7f7e5724c640 1 --2- 172.21.15.88:0/4169127806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e5015cc20 0x7f7e500d0070 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f7e38009890 tx=0x7f7e380046c0 comp rx=0 tx=0).stop 2024-09-16T01:42:03.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.385+0000 7f7e5724c640 1 -- 172.21.15.88:0/4169127806 shutdown_connections 2024-09-16T01:42:03.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.385+0000 7f7e5724c640 1 --2- 172.21.15.88:0/4169127806 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7e3006c2b0 0x7f7e3006e770 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:03.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.385+0000 7f7e5724c640 1 --2- 172.21.15.88:0/4169127806 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e5015cc20 0x7f7e500d0070 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:03.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.385+0000 7f7e5724c640 1 --2- 172.21.15.88:0/4169127806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e50155880 0x7f7e50150810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:03.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.385+0000 7f7e5724c640 1 --2- 172.21.15.88:0/4169127806 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7e500b5470 0x7f7e500ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:03.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.385+0000 7f7e5724c640 1 -- 172.21.15.88:0/4169127806 >> 172.21.15.88:0/4169127806 conn(0x7f7e500d1390 msgr2=0x7f7e5015ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:03.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.385+0000 7f7e5724c640 1 -- 172.21.15.88:0/4169127806 shutdown_connections 2024-09-16T01:42:03.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.385+0000 7f7e5724c640 1 -- 172.21.15.88:0/4169127806 wait complete. 2024-09-16T01:42:03.397 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:42:03.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.495+0000 7f208fce4640 1 Processor -- start 2024-09-16T01:42:03.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.495+0000 7f208fce4640 1 -- start start 2024-09-16T01:42:03.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.496+0000 7f208fce4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20880731c0 0x7f2088073600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:03.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.496+0000 7f208fce4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f208815de80 0x7f208815e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:03.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.496+0000 7f208fce4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f208815a430 con 0x7f208815a050 2024-09-16T01:42:03.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.496+0000 7f208fce4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2088073b40 con 0x7f208815de80 2024-09-16T01:42:03.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.496+0000 7f208fce4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f20880cd0a0 con 0x7f20880731c0 2024-09-16T01:42:03.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.496+0000 7f208da80640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f208815a050 0x7f208815d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:40018/0 (socket says 172.21.15.88:40018) 2024-09-16T01:42:03.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.496+0000 7f208da80640 1 -- 172.21.15.88:0/2412993865 learned_addr learned my addr 172.21.15.88:0/2412993865 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:03.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.496+0000 7f208d27f640 1 --2- 172.21.15.88:0/2412993865 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20880731c0 0x7f2088073600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:03.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.496+0000 7f208e281640 1 --2- 172.21.15.88:0/2412993865 >> v2:172.21.15.154:3300/0 conn(0x7f208815de80 0x7f208815e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:03.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.496+0000 7f208ca7e640 1 -- 172.21.15.88:0/2412993865 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1932752035 0 0) 0x7f208815a430 con 0x7f208815a050 2024-09-16T01:42:03.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.496+0000 7f208ca7e640 1 -- 172.21.15.88:0/2412993865 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2074003520 con 0x7f208815a050 2024-09-16T01:42:03.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.496+0000 7f208ca7e640 1 -- 172.21.15.88:0/2412993865 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3745370442 0 0) 0x7f2074003520 con 0x7f208815a050 2024-09-16T01:42:03.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.496+0000 7f208ca7e640 1 -- 172.21.15.88:0/2412993865 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20880731c0 msgr2=0x7f2088073600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:03.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.496+0000 7f208ca7e640 1 --2- 172.21.15.88:0/2412993865 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20880731c0 0x7f2088073600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:03.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.496+0000 7f208ca7e640 1 -- 172.21.15.88:0/2412993865 >> v2:172.21.15.154:3300/0 conn(0x7f208815de80 msgr2=0x7f208815e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:03.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.496+0000 7f208ca7e640 1 --2- 172.21.15.88:0/2412993865 >> v2:172.21.15.154:3300/0 conn(0x7f208815de80 0x7f208815e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:03.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.496+0000 7f208ca7e640 1 -- 172.21.15.88:0/2412993865 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20880cd380 con 0x7f208815a050 2024-09-16T01:42:03.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.496+0000 7f208ca7e640 1 -- 172.21.15.88:0/2412993865 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f2084001670 con 0x7f208815a050 2024-09-16T01:42:03.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.496+0000 7f208ca7e640 1 -- 172.21.15.88:0/2412993865 >> v1:172.21.15.88:6789/0 conn(0x7f208815a050 legacy=0x7f208815d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:03.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.496+0000 7f208ca7e640 1 --2- 172.21.15.88:0/2412993865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20740039b0 0x7f2074003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:03.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.497+0000 7f208ca7e640 1 --2- 172.21.15.88:0/2412993865 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20880731c0 0x7f2074004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:03.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.497+0000 7f208ca7e640 1 --2- 172.21.15.88:0/2412993865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2074004d40 0x7f20740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:03.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.497+0000 7f208ca7e640 1 -- 172.21.15.88:0/2412993865 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2088073b40 con 0x7f2074004d40 2024-09-16T01:42:03.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.497+0000 7f208ca7e640 1 -- 172.21.15.88:0/2412993865 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f20880cd0a0 con 0x7f20740039b0 2024-09-16T01:42:03.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.497+0000 7f208ca7e640 1 -- 172.21.15.88:0/2412993865 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f207400b9c0 con 0x7f20880731c0 2024-09-16T01:42:03.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.497+0000 7f208e281640 1 --2- 172.21.15.88:0/2412993865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2074004d40 0x7f20740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:03.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.497+0000 7f208da80640 1 --2- 172.21.15.88:0/2412993865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20740039b0 0x7f2074003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:03.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.497+0000 7f208d27f640 1 --2- 172.21.15.88:0/2412993865 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20880731c0 0x7f2074004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:03.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.497+0000 7f208e281640 1 -- 172.21.15.88:0/2412993865 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20880731c0 msgr2=0x7f2074004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:03.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.497+0000 7f208e281640 1 --2- 172.21.15.88:0/2412993865 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20880731c0 0x7f2074004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:03.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.497+0000 7f208e281640 1 -- 172.21.15.88:0/2412993865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20740039b0 msgr2=0x7f2074003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:03.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.497+0000 7f208e281640 1 --2- 172.21.15.88:0/2412993865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20740039b0 0x7f2074003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:03.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.497+0000 7f208e281640 1 -- 172.21.15.88:0/2412993865 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2074003520 con 0x7f2074004d40 2024-09-16T01:42:03.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.498+0000 7f208e281640 1 --2- 172.21.15.88:0/2412993865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2074004d40 0x7f20740051c0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f2080009fc0 tx=0x7f2074005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9723a6d4eee5bb0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:03.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.498+0000 7f208ca7e640 1 -- 172.21.15.88:0/2412993865 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2080009c10 con 0x7f2074004d40 2024-09-16T01:42:03.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.498+0000 7f208ca7e640 1 -- 172.21.15.88:0/2412993865 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2080036070 con 0x7f2074004d40 2024-09-16T01:42:03.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.498+0000 7f208ca7e640 1 -- 172.21.15.88:0/2412993865 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f208002d070 con 0x7f2074004d40 2024-09-16T01:42:03.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.498+0000 7f208fce4640 1 -- 172.21.15.88:0/2412993865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2074004d40 msgr2=0x7f20740051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:03.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.498+0000 7f208fce4640 1 --2- 172.21.15.88:0/2412993865 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2074004d40 0x7f20740051c0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f2080009fc0 tx=0x7f2074005830 comp rx=0 tx=0).stop 2024-09-16T01:42:03.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.498+0000 7f208e281640 1 -- 172.21.15.88:0/2412993865 reap_dead start 2024-09-16T01:42:03.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.498+0000 7f208fce4640 1 -- 172.21.15.88:0/2412993865 shutdown_connections 2024-09-16T01:42:03.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.498+0000 7f208fce4640 1 -- 172.21.15.88:0/2412993865 >> 172.21.15.88:0/2412993865 conn(0x7f208806d0f0 msgr2=0x7f208806d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:03.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.498+0000 7f208fce4640 1 -- 172.21.15.88:0/2412993865 shutdown_connections 2024-09-16T01:42:03.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.498+0000 7f208fce4640 1 -- 172.21.15.88:0/2412993865 wait complete. 2024-09-16T01:42:03.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.500+0000 7f208fce4640 1 Processor -- start 2024-09-16T01:42:03.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.500+0000 7f208fce4640 1 -- start start 2024-09-16T01:42:03.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.500+0000 7f208fce4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f208814fc10 0x7f2088150e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:03.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.500+0000 7f208fce4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f208814fff0 0x7f2088157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:03.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.500+0000 7f208fce4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2088155a10 0x7f2088158510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:03.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.500+0000 7f208fce4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f20880cdc20 con 0x7f208814fc10 2024-09-16T01:42:03.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.500+0000 7f208fce4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f20880cd940 con 0x7f2088155a10 2024-09-16T01:42:03.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.500+0000 7f208fce4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f20880b36e0 con 0x7f208814fff0 2024-09-16T01:42:03.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.500+0000 7f208da80640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f208814fc10 0x7f2088150e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:03.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.500+0000 7f208da80640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f208814fc10 0x7f2088150e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44672/0 (socket says 172.21.15.88:44672) 2024-09-16T01:42:03.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.500+0000 7f208da80640 1 -- 172.21.15.88:0/1087281638 learned_addr learned my addr 172.21.15.88:0/1087281638 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:03.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.501+0000 7f208e281640 1 --2- 172.21.15.88:0/1087281638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2088155a10 0x7f2088158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:03.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.501+0000 7f208da80640 1 -- 172.21.15.88:0/1087281638 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f208814fff0 msgr2=0x7f2088157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:03.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.501+0000 7f208d27f640 1 --2- 172.21.15.88:0/1087281638 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f208814fff0 0x7f2088157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:03.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.501+0000 7f208da80640 1 --2- 172.21.15.88:0/1087281638 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f208814fff0 0x7f2088157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:03.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.501+0000 7f208da80640 1 -- 172.21.15.88:0/1087281638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2088155a10 msgr2=0x7f2088158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:03.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.501+0000 7f208da80640 1 --2- 172.21.15.88:0/1087281638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2088155a10 0x7f2088158510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:03.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.501+0000 7f208da80640 1 -- 172.21.15.88:0/1087281638 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2088156400 con 0x7f208814fc10 2024-09-16T01:42:03.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.501+0000 7f208da80640 1 --2- 172.21.15.88:0/1087281638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f208814fc10 0x7f2088150e90 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f2084007290 tx=0x7f2084007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:03.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.501+0000 7f207e7fc640 1 -- 172.21.15.88:0/1087281638 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f208400f9c0 con 0x7f208814fc10 2024-09-16T01:42:03.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.501+0000 7f207e7fc640 1 -- 172.21.15.88:0/1087281638 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2084018070 con 0x7f208814fc10 2024-09-16T01:42:03.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.501+0000 7f208fce4640 1 -- 172.21.15.88:0/1087281638 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20881514e0 con 0x7f208814fc10 2024-09-16T01:42:03.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.501+0000 7f207e7fc640 1 -- 172.21.15.88:0/1087281638 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f208400fde0 con 0x7f208814fc10 2024-09-16T01:42:03.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.501+0000 7f208fce4640 1 -- 172.21.15.88:0/1087281638 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2088151aa0 con 0x7f208814fc10 2024-09-16T01:42:03.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.502+0000 7f208fce4640 1 -- 172.21.15.88:0/1087281638 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20880ccd00 con 0x7f208814fc10 2024-09-16T01:42:03.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.503+0000 7f207e7fc640 1 -- 172.21.15.88:0/1087281638 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f208401d460 con 0x7f208814fc10 2024-09-16T01:42:03.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.503+0000 7f207e7fc640 1 --2- 172.21.15.88:0/1087281638 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f206c06c2b0 0x7f206c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:03.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.503+0000 7f207e7fc640 1 -- 172.21.15.88:0/1087281638 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 7200+0+0 (secure 0 0 0) 0x7f208408bbd0 con 0x7f208814fc10 2024-09-16T01:42:03.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.503+0000 7f208d27f640 1 --2- 172.21.15.88:0/1087281638 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f206c06c2b0 0x7f206c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:03.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.504+0000 7f208d27f640 1 --2- 172.21.15.88:0/1087281638 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f206c06c2b0 0x7f206c06e770 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f2088158b90 tx=0x7f2078008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:03.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.507+0000 7f207e7fc640 1 -- 172.21.15.88:0/1087281638 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2084059d70 con 0x7f208814fc10 2024-09-16T01:42:03.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:03.743+0000 7f208fce4640 1 -- 172.21.15.88:0/1087281638 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f20880ce900 con 0x7f208814fc10 2024-09-16T01:42:04.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.385+0000 7f207e7fc640 1 -- 172.21.15.88:0/1087281638 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v172) ==== 101+0+0 (secure 0 0 0) 0x7f2084059670 con 0x7f208814fc10 2024-09-16T01:42:04.386 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:42:04.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.388+0000 7f208fce4640 1 -- 172.21.15.88:0/1087281638 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f206c06c2b0 msgr2=0x7f206c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:04.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.388+0000 7f208fce4640 1 --2- 172.21.15.88:0/1087281638 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f206c06c2b0 0x7f206c06e770 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f2088158b90 tx=0x7f2078008040 comp rx=0 tx=0).stop 2024-09-16T01:42:04.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.389+0000 7f208fce4640 1 -- 172.21.15.88:0/1087281638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f208814fc10 msgr2=0x7f2088150e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:04.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.389+0000 7f208fce4640 1 --2- 172.21.15.88:0/1087281638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f208814fc10 0x7f2088150e90 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f2084007290 tx=0x7f2084007750 comp rx=0 tx=0).stop 2024-09-16T01:42:04.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.389+0000 7f208fce4640 1 -- 172.21.15.88:0/1087281638 shutdown_connections 2024-09-16T01:42:04.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.389+0000 7f208fce4640 1 --2- 172.21.15.88:0/1087281638 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f206c06c2b0 0x7f206c06e770 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.389+0000 7f208fce4640 1 --2- 172.21.15.88:0/1087281638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2088155a10 0x7f2088158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.389+0000 7f208fce4640 1 --2- 172.21.15.88:0/1087281638 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f208814fff0 0x7f2088157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.389+0000 7f208fce4640 1 --2- 172.21.15.88:0/1087281638 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f208814fc10 0x7f2088150e90 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.389+0000 7f208fce4640 1 -- 172.21.15.88:0/1087281638 >> 172.21.15.88:0/1087281638 conn(0x7f208806d0f0 msgr2=0x7f20880b5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:04.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.389+0000 7f208fce4640 1 -- 172.21.15.88:0/1087281638 shutdown_connections 2024-09-16T01:42:04.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.389+0000 7f208fce4640 1 -- 172.21.15.88:0/1087281638 wait complete. 2024-09-16T01:42:04.404 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:42:04.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.449+0000 7fa5d783cf40 1 Processor -- start 2024-09-16T01:42:04.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.449+0000 7fa5d783cf40 1 -- start start 2024-09-16T01:42:04.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.449+0000 7fa5d783cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5c40030f0 0x564d8373d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.449+0000 7fa5d783cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d83723ef0 0x564d83695030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.449+0000 7fa5d783cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564d837cd3e0 0x564d837cd860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.449+0000 7fa5d783cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564d835b7510 con 0x564d83723ef0 2024-09-16T01:42:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.449+0000 7fa5d783cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564d83610c50 con 0x7fa5c40030f0 2024-09-16T01:42:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.449+0000 7fa5d783cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x564d836b05b0 con 0x564d837cd3e0 2024-09-16T01:42:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.450+0000 7fa5cffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d83723ef0 0x564d83695030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.450+0000 7fa5cffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d83723ef0 0x564d83695030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44686/0 (socket says 172.21.15.88:44686) 2024-09-16T01:42:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.450+0000 7fa5cffff640 1 -- 172.21.15.88:0/860472435 learned_addr learned my addr 172.21.15.88:0/860472435 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.450+0000 7fa5d5334640 1 --2- 172.21.15.88:0/860472435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5c40030f0 0x564d8373d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.450+0000 7fa5d5b35640 1 --2- 172.21.15.88:0/860472435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564d837cd3e0 0x564d837cd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.450+0000 7fa5cffff640 1 -- 172.21.15.88:0/860472435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564d837cd3e0 msgr2=0x564d837cd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:04.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.450+0000 7fa5cffff640 1 --2- 172.21.15.88:0/860472435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564d837cd3e0 0x564d837cd860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.450+0000 7fa5cffff640 1 -- 172.21.15.88:0/860472435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5c40030f0 msgr2=0x564d8373d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:04.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.450+0000 7fa5cffff640 1 --2- 172.21.15.88:0/860472435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5c40030f0 0x564d8373d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.450+0000 7fa5cffff640 1 -- 172.21.15.88:0/860472435 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d8373cc30 con 0x564d83723ef0 2024-09-16T01:42:04.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.450+0000 7fa5d5b35640 1 --2- 172.21.15.88:0/860472435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564d837cd3e0 0x564d837cd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:04.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.450+0000 7fa5cffff640 1 --2- 172.21.15.88:0/860472435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d83723ef0 0x564d83695030 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fa5c0000c00 tx=0x7fa5c0051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9787d21b4dc9356d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:04.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.450+0000 7fa5cf7fe640 1 -- 172.21.15.88:0/860472435 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c004f920 con 0x564d83723ef0 2024-09-16T01:42:04.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.450+0000 7fa5cf7fe640 1 -- 172.21.15.88:0/860472435 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5c004fac0 con 0x564d83723ef0 2024-09-16T01:42:04.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.450+0000 7fa5cf7fe640 1 -- 172.21.15.88:0/860472435 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0064460 con 0x564d83723ef0 2024-09-16T01:42:04.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.451+0000 7fa5d783cf40 1 -- 172.21.15.88:0/860472435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d83723ef0 msgr2=0x564d83695030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:04.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.451+0000 7fa5d783cf40 1 --2- 172.21.15.88:0/860472435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d83723ef0 0x564d83695030 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fa5c0000c00 tx=0x7fa5c0051000 comp rx=0 tx=0).stop 2024-09-16T01:42:04.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.451+0000 7fa5d783cf40 1 -- 172.21.15.88:0/860472435 shutdown_connections 2024-09-16T01:42:04.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.451+0000 7fa5d783cf40 1 --2- 172.21.15.88:0/860472435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x564d837cd3e0 0x564d837cd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.451+0000 7fa5d783cf40 1 --2- 172.21.15.88:0/860472435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d83723ef0 0x564d83695030 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.451+0000 7fa5d783cf40 1 --2- 172.21.15.88:0/860472435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5c40030f0 0x564d8373d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.451+0000 7fa5d783cf40 1 -- 172.21.15.88:0/860472435 >> 172.21.15.88:0/860472435 conn(0x564d836b0750 msgr2=0x564d83666130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:04.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.451+0000 7fa5d783cf40 1 -- 172.21.15.88:0/860472435 shutdown_connections 2024-09-16T01:42:04.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.451+0000 7fa5d783cf40 1 -- 172.21.15.88:0/860472435 wait complete. 2024-09-16T01:42:04.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.451+0000 7fa5d783cf40 1 Processor -- start 2024-09-16T01:42:04.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.451+0000 7fa5d783cf40 1 -- start start 2024-09-16T01:42:04.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.452+0000 7fa5d783cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d83723ef0 0x564d8373d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.452+0000 7fa5d783cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d837cd3e0 0x564d8371f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.452+0000 7fa5d783cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5c40030f0 0x564d83719ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.452+0000 7fa5d783cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564d837245e0 con 0x564d83723ef0 2024-09-16T01:42:04.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.452+0000 7fa5d783cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564d8373a3f0 con 0x564d837cd3e0 2024-09-16T01:42:04.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.452+0000 7fa5d783cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x564d8373c950 con 0x7fa5c40030f0 2024-09-16T01:42:04.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.452+0000 7fa5d5334640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d83723ef0 0x564d8373d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.452+0000 7fa5d5334640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d83723ef0 0x564d8373d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44702/0 (socket says 172.21.15.88:44702) 2024-09-16T01:42:04.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.452+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 learned_addr learned my addr 172.21.15.88:0/2646765612 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:04.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.452+0000 7fa5cffff640 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d837cd3e0 0x564d8371f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.452+0000 7fa5d5b35640 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5c40030f0 0x564d83719ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.453+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5c40030f0 msgr2=0x564d83719ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:04.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.453+0000 7fa5d5334640 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5c40030f0 0x564d83719ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.453+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d837cd3e0 msgr2=0x564d8371f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:04.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.453+0000 7fa5d5334640 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d837cd3e0 0x564d8371f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.453+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d8371fa90 con 0x564d83723ef0 2024-09-16T01:42:04.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.453+0000 7fa5d5b35640 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5c40030f0 0x564d83719ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:04.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.453+0000 7fa5d5334640 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d83723ef0 0x564d8373d930 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8003eb0 tx=0x7fa5c800e760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:04.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.453+0000 7fa5ce7fc640 1 -- 172.21.15.88:0/2646765612 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8003320 con 0x564d83723ef0 2024-09-16T01:42:04.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.453+0000 7fa5ce7fc640 1 -- 172.21.15.88:0/2646765612 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5c8017020 con 0x564d83723ef0 2024-09-16T01:42:04.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.453+0000 7fa5ce7fc640 1 -- 172.21.15.88:0/2646765612 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c80034c0 con 0x564d83723ef0 2024-09-16T01:42:04.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.453+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564d8371fd30 con 0x564d83723ef0 2024-09-16T01:42:04.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.453+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d837c7ea0 con 0x564d83723ef0 2024-09-16T01:42:04.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.454+0000 7fa5ce7fc640 1 -- 172.21.15.88:0/2646765612 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa5c8001680 con 0x564d83723ef0 2024-09-16T01:42:04.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.455+0000 7fa5ce7fc640 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa5b806ac90 0x7fa5b806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.455+0000 7fa5ce7fc640 1 -- 172.21.15.88:0/2646765612 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 7200+0+0 (secure 0 0 0) 0x7fa5c808ad30 con 0x564d83723ef0 2024-09-16T01:42:04.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.455+0000 7fa5cffff640 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa5b806ac90 0x7fa5b806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.460 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:42:04.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.455+0000 7fa5d783cf40 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x564d837c86a0 0x564d837c8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.455+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837c8fc0 con 0x564d837c86a0 2024-09-16T01:42:04.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.455+0000 7fa5d5b35640 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x564d837c86a0 0x564d837c8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.455+0000 7fa5cffff640 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa5b806ac90 0x7fa5b806d150 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fa5c0045c70 tx=0x7fa5c0053000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:04.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.456+0000 7fa5d5b35640 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x564d837c86a0 0x564d837c8a80 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:04.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.457+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v104'505 uv479 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.457+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e104) -- 0x564d837c9390 con 0x564d837c86a0 2024-09-16T01:42:04.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.458+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v104'506 uv506 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.458+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e05d0 con 0x564d837c86a0 2024-09-16T01:42:04.463 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:42:04.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.460+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v104'507 uv507 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.460+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e1980 con 0x564d837c86a0 2024-09-16T01:42:04.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.461+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v104'508 uv508 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.461+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e1d20 con 0x564d837c86a0 2024-09-16T01:42:04.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.462+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v104'509 uv509 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.462+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e0970 con 0x564d837c86a0 2024-09-16T01:42:04.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.463+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v104'510 uv510 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.463+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e0d10 con 0x564d837c86a0 2024-09-16T01:42:04.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.465+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v104'511 uv485 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.465+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e104) -- 0x564d837e10b0 con 0x564d837c86a0 2024-09-16T01:42:04.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.466+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v104'512 uv512 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.466+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e10b0 con 0x564d837c86a0 2024-09-16T01:42:04.468 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:42:04.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.467+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v104'513 uv513 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.467+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e10b0 con 0x564d837c86a0 2024-09-16T01:42:04.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.468+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v104'514 uv514 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.468+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e10b0 con 0x564d837c86a0 2024-09-16T01:42:04.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.469+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v104'515 uv515 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.469+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e10b0 con 0x564d837c86a0 2024-09-16T01:42:04.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.470+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v104'516 uv516 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.470+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e10b0 con 0x564d837c86a0 2024-09-16T01:42:04.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.472+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v104'517 uv492 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.472+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e104) -- 0x564d837e10b0 con 0x564d837c86a0 2024-09-16T01:42:04.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.473+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v104'518 uv518 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.473+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e10b0 con 0x564d837c86a0 2024-09-16T01:42:04.475 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:42:04.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.474+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v104'519 uv519 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.474+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e10b0 con 0x564d837c86a0 2024-09-16T01:42:04.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.475+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v104'520 uv520 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.475+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e10b0 con 0x564d837c86a0 2024-09-16T01:42:04.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.476+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v104'521 uv521 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.476+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e10b0 con 0x564d837c86a0 2024-09-16T01:42:04.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.477+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v104'522 uv522 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.477+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e10b0 con 0x564d837c86a0 2024-09-16T01:42:04.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.478+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v104'523 uv523 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.478+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e10b0 con 0x564d837c86a0 2024-09-16T01:42:04.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.480+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v104'524 uv498 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.480+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e104) -- 0x564d837e10b0 con 0x564d837c86a0 2024-09-16T01:42:04.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.481+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v104'525 uv525 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.481+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e10b0 con 0x564d837c86a0 2024-09-16T01:42:04.483 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:42:04.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.482+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v104'526 uv526 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.482+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e10b0 con 0x564d837c86a0 2024-09-16T01:42:04.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.483+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v104'527 uv527 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.483+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e10b0 con 0x564d837c86a0 2024-09-16T01:42:04.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.484+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v104'528 uv528 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.485+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e10b0 con 0x564d837c86a0 2024-09-16T01:42:04.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.486+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v104'529 uv529 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.486+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e10b0 con 0x564d837c86a0 2024-09-16T01:42:04.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.487+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v104'530 uv504 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.487+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e104) -- 0x564d837e10b0 con 0x564d837c86a0 2024-09-16T01:42:04.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.488+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v104'531 uv531 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.488+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e10b0 con 0x564d837c86a0 2024-09-16T01:42:04.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.489+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v104'532 uv532 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.490 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:42:04.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.489+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e10b0 con 0x564d837c86a0 2024-09-16T01:42:04.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.490+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v104'533 uv533 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.490+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e10b0 con 0x564d837c86a0 2024-09-16T01:42:04.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.491+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v104'534 uv534 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.491+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e10b0 con 0x564d837c86a0 2024-09-16T01:42:04.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.492+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v104'535 uv535 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.492+0000 7fa5d783cf40 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x564d837e20c0 0x564d837e44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.492+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e10b0 con 0x564d837e20c0 2024-09-16T01:42:04.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.492+0000 7fa5d5334640 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x564d837e20c0 0x564d837e44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.493+0000 7fa5d5334640 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x564d837e20c0 0x564d837e44b0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:04.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.494+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v104'375 uv357 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837e20c0 2024-09-16T01:42:04.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.494+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e104) -- 0x564d837e49f0 con 0x564d837e20c0 2024-09-16T01:42:04.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.495+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v104'376 uv376 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837e20c0 2024-09-16T01:42:04.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.495+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e4e50 con 0x564d837e20c0 2024-09-16T01:42:04.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.496+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v104'377 uv377 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837e20c0 2024-09-16T01:42:04.498 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:42:04.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.496+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e5710 con 0x564d837e20c0 2024-09-16T01:42:04.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.498+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v104'378 uv378 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837e20c0 2024-09-16T01:42:04.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.498+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e5ab0 con 0x564d837e20c0 2024-09-16T01:42:04.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.500+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v104'379 uv379 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837e20c0 2024-09-16T01:42:04.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.500+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e5e50 con 0x564d837e20c0 2024-09-16T01:42:04.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.501+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v104'380 uv374 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837e20c0 2024-09-16T01:42:04.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.501+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e104) -- 0x564d837e6240 con 0x564d837e20c0 2024-09-16T01:42:04.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.502+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v104'381 uv381 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837e20c0 2024-09-16T01:42:04.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.502+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e65e0 con 0x564d837e20c0 2024-09-16T01:42:04.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.503+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v104'382 uv382 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837e20c0 2024-09-16T01:42:04.504 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:42:04.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.503+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e69d0 con 0x564d837e20c0 2024-09-16T01:42:04.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.504+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v104'383 uv383 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837e20c0 2024-09-16T01:42:04.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.504+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e69d0 con 0x564d837e20c0 2024-09-16T01:42:04.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.505+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v104'384 uv384 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837e20c0 2024-09-16T01:42:04.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.505+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e6670 con 0x564d837e20c0 2024-09-16T01:42:04.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.506+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v104'385 uv367 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837e20c0 2024-09-16T01:42:04.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.506+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e104) -- 0x564d837e6620 con 0x564d837e20c0 2024-09-16T01:42:04.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.507+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v104'386 uv386 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837e20c0 2024-09-16T01:42:04.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.507+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e65e0 con 0x564d837e20c0 2024-09-16T01:42:04.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.508+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v104'387 uv387 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837e20c0 2024-09-16T01:42:04.510 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:42:04.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.509+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e65e0 con 0x564d837e20c0 2024-09-16T01:42:04.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.510+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v104'388 uv388 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837e20c0 2024-09-16T01:42:04.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.510+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e65e0 con 0x564d837e20c0 2024-09-16T01:42:04.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.511+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v104'389 uv389 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837e20c0 2024-09-16T01:42:04.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.511+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e65e0 con 0x564d837e20c0 2024-09-16T01:42:04.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.512+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v104'390 uv373 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837e20c0 2024-09-16T01:42:04.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.512+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e104) -- 0x564d837e65e0 con 0x564d837e20c0 2024-09-16T01:42:04.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.513+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v104'391 uv391 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837e20c0 2024-09-16T01:42:04.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.513+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e65e0 con 0x564d837e20c0 2024-09-16T01:42:04.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.514+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v104'392 uv392 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837e20c0 2024-09-16T01:42:04.515 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:42:04.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.514+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e65e0 con 0x564d837e20c0 2024-09-16T01:42:04.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.515+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v104'393 uv393 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837e20c0 2024-09-16T01:42:04.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.515+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e65e0 con 0x564d837e20c0 2024-09-16T01:42:04.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.516+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v104'394 uv394 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837e20c0 2024-09-16T01:42:04.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.516+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e65e0 con 0x564d837e20c0 2024-09-16T01:42:04.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.517+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v104'395 uv395 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837e20c0 2024-09-16T01:42:04.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.517+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e65e0 con 0x564d837c86a0 2024-09-16T01:42:04.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.518+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v104'99 uv98 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.518+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e104) -- 0x564d837e65e0 con 0x564d837c86a0 2024-09-16T01:42:04.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.519+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v104'100 uv100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.519+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e65e0 con 0x564d837c86a0 2024-09-16T01:42:04.521 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:42:04.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.520+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v104'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.520+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e65e0 con 0x564d837c86a0 2024-09-16T01:42:04.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.521+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v104'102 uv102 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.521+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e65e0 con 0x564d837c86a0 2024-09-16T01:42:04.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.522+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v104'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.522+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e65e0 con 0x564d837c86a0 2024-09-16T01:42:04.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.522+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v104'104 uv104 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.522+0000 7fa5d783cf40 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x564d837e6f20 0x564d837e7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.522+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e9b10 con 0x564d837e6f20 2024-09-16T01:42:04.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.523+0000 7fa5cffff640 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x564d837e6f20 0x564d837e7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.523+0000 7fa5cffff640 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x564d837e6f20 0x564d837e7360 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:04.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.524+0000 7fa5cffff640 1 -- 172.21.15.88:0/2646765612 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v104'77 uv76 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fa5c001a320 con 0x564d837e6f20 2024-09-16T01:42:04.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.524+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e104) -- 0x564d837e9eb0 con 0x564d837e6f20 2024-09-16T01:42:04.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.525+0000 7fa5cffff640 1 -- 172.21.15.88:0/2646765612 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v104'78 uv78 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa5c001a320 con 0x564d837e6f20 2024-09-16T01:42:04.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.525+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837ea2d0 con 0x564d837e6f20 2024-09-16T01:42:04.527 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:42:04.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.526+0000 7fa5cffff640 1 -- 172.21.15.88:0/2646765612 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v104'79 uv79 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa5c001a320 con 0x564d837e6f20 2024-09-16T01:42:04.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.526+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837e5320 con 0x564d837e6f20 2024-09-16T01:42:04.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.527+0000 7fa5cffff640 1 -- 172.21.15.88:0/2646765612 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v104'80 uv80 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa5c001a320 con 0x564d837e6f20 2024-09-16T01:42:04.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.527+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837eabc0 con 0x564d837c86a0 2024-09-16T01:42:04.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.528+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v104'99 uv98 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.528+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e104) -- 0x564d837eabc0 con 0x564d837c86a0 2024-09-16T01:42:04.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.529+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v104'100 uv100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.529+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837eabc0 con 0x564d837c86a0 2024-09-16T01:42:04.533 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:42:04.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.532+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v104'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.532+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837ebbd0 con 0x564d837c86a0 2024-09-16T01:42:04.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.534+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v104'102 uv102 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.534+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837ebbd0 con 0x564d837c86a0 2024-09-16T01:42:04.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.537+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v104'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.537+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837eabc0 con 0x564d837c86a0 2024-09-16T01:42:04.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.541+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v104'104 uv104 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.541+0000 7fa5d783cf40 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x564d837eb240 0x564d837ed690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.541+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837edc80 con 0x564d837eb240 2024-09-16T01:42:04.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.542+0000 7fa5d5b35640 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x564d837eb240 0x564d837ed690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.542+0000 7fa5d5b35640 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x564d837eb240 0x564d837ed690 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:04.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.543+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v104'182 uv176 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.543+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e104) -- 0x564d837ee050 con 0x564d837eb240 2024-09-16T01:42:04.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.544+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v104'183 uv183 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.544+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837ee050 con 0x564d837eb240 2024-09-16T01:42:04.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.545+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v104'184 uv184 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.547 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:42:04.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.545+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837ef060 con 0x564d837eb240 2024-09-16T01:42:04.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.546+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v104'185 uv185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.546+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837ef060 con 0x564d837eb240 2024-09-16T01:42:04.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.547+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v104'186 uv186 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.547+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837ee050 con 0x564d837eb240 2024-09-16T01:42:04.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.548+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v104'187 uv187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.548+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837ee050 con 0x564d837eb240 2024-09-16T01:42:04.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.549+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v104'188 uv181 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.550+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e104) -- 0x564d837ee050 con 0x564d837eb240 2024-09-16T01:42:04.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.550+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v104'189 uv189 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.550+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837ee050 con 0x564d837eb240 2024-09-16T01:42:04.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.551+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v104'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.553 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:42:04.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.551+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837ee050 con 0x564d837eb240 2024-09-16T01:42:04.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.552+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v104'191 uv191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.552+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837ee050 con 0x564d837eb240 2024-09-16T01:42:04.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.553+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v104'192 uv192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.553+0000 7fa5d783cf40 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x564d837ee6d0 0x564d837f0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.553+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f11f0 con 0x564d837ee6d0 2024-09-16T01:42:04.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.554+0000 7fa5d5334640 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x564d837ee6d0 0x564d837f0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.554+0000 7fa5d5334640 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x564d837ee6d0 0x564d837f0ac0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:04.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.555+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v104'81 uv80 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837ee6d0 2024-09-16T01:42:04.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.555+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e104) -- 0x564d837f15c0 con 0x564d837ee6d0 2024-09-16T01:42:04.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.556+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v104'82 uv82 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837ee6d0 2024-09-16T01:42:04.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.556+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f15c0 con 0x564d837ee6d0 2024-09-16T01:42:04.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.557+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v104'83 uv83 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837ee6d0 2024-09-16T01:42:04.558 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:42:04.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.557+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f15c0 con 0x564d837ee6d0 2024-09-16T01:42:04.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.558+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v104'84 uv84 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837ee6d0 2024-09-16T01:42:04.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.558+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f15c0 con 0x564d837ee6d0 2024-09-16T01:42:04.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.559+0000 7fa5d5334640 1 -- 172.21.15.88:0/2646765612 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v104'85 uv85 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa5c8023070 con 0x564d837ee6d0 2024-09-16T01:42:04.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.559+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f15c0 con 0x564d837c86a0 2024-09-16T01:42:04.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.559+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v104'99 uv98 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.559+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e104) -- 0x564d837f15c0 con 0x564d837c86a0 2024-09-16T01:42:04.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.560+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v104'100 uv100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.560+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f15c0 con 0x564d837c86a0 2024-09-16T01:42:04.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.561+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v104'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.563 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:42:04.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.561+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f2610 con 0x564d837c86a0 2024-09-16T01:42:04.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.562+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v104'102 uv102 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.562+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f25d0 con 0x564d837c86a0 2024-09-16T01:42:04.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.563+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v104'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.563+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f15c0 con 0x564d837c86a0 2024-09-16T01:42:04.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.564+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v104'104 uv104 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837c86a0 2024-09-16T01:42:04.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.564+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f15c0 con 0x564d837e6f20 2024-09-16T01:42:04.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.565+0000 7fa5cffff640 1 -- 172.21.15.88:0/2646765612 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v104'162 uv156 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fa5c001a320 con 0x564d837e6f20 2024-09-16T01:42:04.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.565+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e104) -- 0x564d837f19b0 con 0x564d837e6f20 2024-09-16T01:42:04.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.566+0000 7fa5cffff640 1 -- 172.21.15.88:0/2646765612 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v104'163 uv163 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa5c001a320 con 0x564d837e6f20 2024-09-16T01:42:04.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.566+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f1da0 con 0x564d837e6f20 2024-09-16T01:42:04.568 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:42:04.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.567+0000 7fa5cffff640 1 -- 172.21.15.88:0/2646765612 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v104'164 uv164 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa5c001a320 con 0x564d837e6f20 2024-09-16T01:42:04.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.567+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837ea760 con 0x564d837e6f20 2024-09-16T01:42:04.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.568+0000 7fa5cffff640 1 -- 172.21.15.88:0/2646765612 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v104'165 uv165 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa5c001a320 con 0x564d837e6f20 2024-09-16T01:42:04.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.568+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837ea760 con 0x564d837e6f20 2024-09-16T01:42:04.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.569+0000 7fa5cffff640 1 -- 172.21.15.88:0/2646765612 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v104'166 uv166 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa5c001a320 con 0x564d837e6f20 2024-09-16T01:42:04.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.569+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837ea760 con 0x564d837e6f20 2024-09-16T01:42:04.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.570+0000 7fa5cffff640 1 -- 172.21.15.88:0/2646765612 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v104'167 uv161 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa5c001a320 con 0x564d837e6f20 2024-09-16T01:42:04.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.570+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e104) -- 0x564d837ea760 con 0x564d837e6f20 2024-09-16T01:42:04.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.571+0000 7fa5cffff640 1 -- 172.21.15.88:0/2646765612 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v104'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5c001a320 con 0x564d837e6f20 2024-09-16T01:42:04.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.571+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837ea760 con 0x564d837e6f20 2024-09-16T01:42:04.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.572+0000 7fa5cffff640 1 -- 172.21.15.88:0/2646765612 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v104'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5c001a320 con 0x564d837e6f20 2024-09-16T01:42:04.573 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:42:04.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.572+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837ea760 con 0x564d837e6f20 2024-09-16T01:42:04.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.573+0000 7fa5cffff640 1 -- 172.21.15.88:0/2646765612 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v104'170 uv170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5c001a320 con 0x564d837e6f20 2024-09-16T01:42:04.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.573+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f52f0 con 0x564d837e6f20 2024-09-16T01:42:04.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.574+0000 7fa5cffff640 1 -- 172.21.15.88:0/2646765612 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v104'171 uv171 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5c001a320 con 0x564d837e6f20 2024-09-16T01:42:04.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.574+0000 7fa5d783cf40 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x564d837f2ea0 0x564d837f5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.574+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837f2ea0 2024-09-16T01:42:04.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.574+0000 7fa5cffff640 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x564d837f2ea0 0x564d837f5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.574+0000 7fa5cffff640 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x564d837f2ea0 0x564d837f5810 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:04.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.575+0000 7fa5cffff640 1 -- 172.21.15.88:0/2646765612 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v104'74 uv73 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fa5c001a320 con 0x564d837f2ea0 2024-09-16T01:42:04.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.575+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837f2ea0 2024-09-16T01:42:04.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.576+0000 7fa5cffff640 1 -- 172.21.15.88:0/2646765612 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v104'75 uv75 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fa5c001a320 con 0x564d837f2ea0 2024-09-16T01:42:04.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.576+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837f2ea0 2024-09-16T01:42:04.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.577+0000 7fa5cffff640 1 -- 172.21.15.88:0/2646765612 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v104'76 uv76 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fa5c001a320 con 0x564d837f2ea0 2024-09-16T01:42:04.578 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:42:04.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.577+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837f2ea0 2024-09-16T01:42:04.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.578+0000 7fa5cffff640 1 -- 172.21.15.88:0/2646765612 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v104'77 uv77 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fa5c001a320 con 0x564d837f2ea0 2024-09-16T01:42:04.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.578+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837e6f20 2024-09-16T01:42:04.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.579+0000 7fa5cffff640 1 -- 172.21.15.88:0/2646765612 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v104'99 uv98 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa5c001a320 con 0x564d837e6f20 2024-09-16T01:42:04.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.579+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837e6f20 2024-09-16T01:42:04.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.580+0000 7fa5cffff640 1 -- 172.21.15.88:0/2646765612 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v104'100 uv100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5c001a320 con 0x564d837e6f20 2024-09-16T01:42:04.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.580+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837e6f20 2024-09-16T01:42:04.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.581+0000 7fa5cffff640 1 -- 172.21.15.88:0/2646765612 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v104'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5c001a320 con 0x564d837e6f20 2024-09-16T01:42:04.583 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:42:04.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.581+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837e6f20 2024-09-16T01:42:04.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.582+0000 7fa5cffff640 1 -- 172.21.15.88:0/2646765612 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v104'102 uv102 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5c001a320 con 0x564d837e6f20 2024-09-16T01:42:04.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.582+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837e6f20 2024-09-16T01:42:04.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.583+0000 7fa5cffff640 1 -- 172.21.15.88:0/2646765612 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v104'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5c001a320 con 0x564d837e6f20 2024-09-16T01:42:04.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.583+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837e6f20 2024-09-16T01:42:04.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.584+0000 7fa5cffff640 1 -- 172.21.15.88:0/2646765612 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v104'104 uv104 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5c001a320 con 0x564d837e6f20 2024-09-16T01:42:04.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.584+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837eb240 2024-09-16T01:42:04.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.585+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v104'84 uv83 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.585+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837eb240 2024-09-16T01:42:04.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.586+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v104'85 uv85 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.586+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837eb240 2024-09-16T01:42:04.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.587+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v104'86 uv86 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.590 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:42:04.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.588+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837eb240 2024-09-16T01:42:04.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.589+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v104'87 uv87 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.589+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837eb240 2024-09-16T01:42:04.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.590+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v104'88 uv88 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.590+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837eb240 2024-09-16T01:42:04.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.591+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v104'204 uv196 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.591+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837eb240 2024-09-16T01:42:04.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.592+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v104'205 uv205 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.592+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837eb240 2024-09-16T01:42:04.594 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:42:04.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.593+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v104'206 uv206 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.593+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837eb240 2024-09-16T01:42:04.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.594+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v104'207 uv207 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.594+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837eb240 2024-09-16T01:42:04.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.595+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v104'208 uv208 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.595+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837eb240 2024-09-16T01:42:04.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.596+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v104'209 uv203 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.596+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837eb240 2024-09-16T01:42:04.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.597+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v104'210 uv210 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.597+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837eb240 2024-09-16T01:42:04.599 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:42:04.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.598+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v104'211 uv211 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.598+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837eb240 2024-09-16T01:42:04.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.599+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v104'212 uv212 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.599+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837eb240 2024-09-16T01:42:04.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.600+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v104'213 uv213 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.600+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5290 con 0x564d837eb240 2024-09-16T01:42:04.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.601+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v104'214 uv214 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.601+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x564d837f5d50 con 0x564d837eb240 2024-09-16T01:42:04.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.602+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v104'215 uv215 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa5d0003530 con 0x564d837eb240 2024-09-16T01:42:04.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.602+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x564d837c86a0 msgr2=0x564d837c8a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:04.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.602+0000 7fa5d783cf40 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x564d837c86a0 0x564d837c8a80 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.602+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x564d837e6f20 msgr2=0x564d837e7360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:04.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.602+0000 7fa5d783cf40 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x564d837e6f20 0x564d837e7360 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.602+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x564d837f2ea0 msgr2=0x564d837f5810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:04.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.602+0000 7fa5d783cf40 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x564d837f2ea0 0x564d837f5810 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.602+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x564d837e20c0 msgr2=0x564d837e44b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:04.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.602+0000 7fa5d783cf40 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x564d837e20c0 0x564d837e44b0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.602+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x564d837eb240 msgr2=0x564d837ed690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:04.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.602+0000 7fa5d783cf40 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x564d837eb240 0x564d837ed690 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.602+0000 7fa5d5b35640 1 -- 172.21.15.88:0/2646765612 reap_dead start 2024-09-16T01:42:04.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.602+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x564d837ee6d0 msgr2=0x564d837f0ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:04.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.602+0000 7fa5d783cf40 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x564d837ee6d0 0x564d837f0ac0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.602+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa5b806ac90 msgr2=0x7fa5b806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:04.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.602+0000 7fa5d783cf40 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa5b806ac90 0x7fa5b806d150 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fa5c0045c70 tx=0x7fa5c0053000 comp rx=0 tx=0).stop 2024-09-16T01:42:04.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.602+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d83723ef0 msgr2=0x564d8373d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:04.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.602+0000 7fa5d783cf40 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d83723ef0 0x564d8373d930 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8003eb0 tx=0x7fa5c800e760 comp rx=0 tx=0).stop 2024-09-16T01:42:04.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.603+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 shutdown_connections 2024-09-16T01:42:04.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.603+0000 7fa5d783cf40 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa5b806ac90 0x7fa5b806d150 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.603+0000 7fa5d783cf40 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x564d837ee6d0 0x564d837f0ac0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.603+0000 7fa5d783cf40 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x564d837eb240 0x564d837ed690 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.603+0000 7fa5d783cf40 1 --2- 172.21.15.88:0/2646765612 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x564d83723ef0 0x564d8373d930 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.603+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 >> 172.21.15.88:0/2646765612 conn(0x564d836b0750 msgr2=0x564d83666130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:04.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.603+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 shutdown_connections 2024-09-16T01:42:04.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.603+0000 7fa5d783cf40 1 -- 172.21.15.88:0/2646765612 wait complete. 2024-09-16T01:42:04.608 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put 200.00000001 - 2024-09-16T01:42:04.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.649+0000 7ffbebb2af40 1 Processor -- start 2024-09-16T01:42:04.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.649+0000 7ffbebb2af40 1 -- start start 2024-09-16T01:42:04.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.650+0000 7ffbebb2af40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56019d772e10 0x56019d8127d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.650+0000 7ffbebb2af40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56019d773a60 0x56019d817360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.650+0000 7ffbebb2af40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56019d78d250 0x56019d817f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.650+0000 7ffbebb2af40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56019d773220 con 0x56019d772e10 2024-09-16T01:42:04.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.650+0000 7ffbebb2af40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56019d81c120 con 0x56019d773a60 2024-09-16T01:42:04.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.650+0000 7ffbebb2af40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56019d81c3c0 con 0x56019d78d250 2024-09-16T01:42:04.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.650+0000 7ffbe8bba640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56019d772e10 0x56019d8127d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.650+0000 7ffbe8bba640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56019d772e10 0x56019d8127d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44724/0 (socket says 172.21.15.88:44724) 2024-09-16T01:42:04.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.650+0000 7ffbe8bba640 1 -- 172.21.15.88:0/1135066683 learned_addr learned my addr 172.21.15.88:0/1135066683 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:04.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.650+0000 7ffbe3fff640 1 --2- 172.21.15.88:0/1135066683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56019d773a60 0x56019d817360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.650+0000 7ffbe93bb640 1 --2- 172.21.15.88:0/1135066683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56019d78d250 0x56019d817f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.650+0000 7ffbe8bba640 1 -- 172.21.15.88:0/1135066683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56019d78d250 msgr2=0x56019d817f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:04.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.650+0000 7ffbe8bba640 1 --2- 172.21.15.88:0/1135066683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56019d78d250 0x56019d817f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.650+0000 7ffbe8bba640 1 -- 172.21.15.88:0/1135066683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56019d773a60 msgr2=0x56019d817360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:04.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.650+0000 7ffbe8bba640 1 --2- 172.21.15.88:0/1135066683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56019d773a60 0x56019d817360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.650+0000 7ffbe8bba640 1 -- 172.21.15.88:0/1135066683 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56019d66ecb0 con 0x56019d772e10 2024-09-16T01:42:04.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.650+0000 7ffbe8bba640 1 --2- 172.21.15.88:0/1135066683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56019d772e10 0x56019d8127d0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc00a7a0 tx=0x7ffbdc00a890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cacb9ef0d146155 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:04.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.651+0000 7ffbe2ffd640 1 -- 172.21.15.88:0/1135066683 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc018070 con 0x56019d772e10 2024-09-16T01:42:04.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.651+0000 7ffbe2ffd640 1 -- 172.21.15.88:0/1135066683 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbdc007600 con 0x56019d772e10 2024-09-16T01:42:04.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.651+0000 7ffbe2ffd640 1 -- 172.21.15.88:0/1135066683 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc0079b0 con 0x56019d772e10 2024-09-16T01:42:04.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.651+0000 7ffbebb2af40 1 -- 172.21.15.88:0/1135066683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56019d772e10 msgr2=0x56019d8127d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:04.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.651+0000 7ffbebb2af40 1 --2- 172.21.15.88:0/1135066683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56019d772e10 0x56019d8127d0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc00a7a0 tx=0x7ffbdc00a890 comp rx=0 tx=0).stop 2024-09-16T01:42:04.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.651+0000 7ffbebb2af40 1 -- 172.21.15.88:0/1135066683 shutdown_connections 2024-09-16T01:42:04.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.651+0000 7ffbebb2af40 1 --2- 172.21.15.88:0/1135066683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56019d78d250 0x56019d817f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.651+0000 7ffbebb2af40 1 --2- 172.21.15.88:0/1135066683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56019d773a60 0x56019d817360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.651+0000 7ffbebb2af40 1 --2- 172.21.15.88:0/1135066683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56019d772e10 0x56019d8127d0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.651+0000 7ffbebb2af40 1 -- 172.21.15.88:0/1135066683 >> 172.21.15.88:0/1135066683 conn(0x56019d78e750 msgr2=0x56019d772650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:04.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.651+0000 7ffbebb2af40 1 -- 172.21.15.88:0/1135066683 shutdown_connections 2024-09-16T01:42:04.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.651+0000 7ffbebb2af40 1 -- 172.21.15.88:0/1135066683 wait complete. 2024-09-16T01:42:04.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.652+0000 7ffbebb2af40 1 Processor -- start 2024-09-16T01:42:04.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.652+0000 7ffbebb2af40 1 -- start start 2024-09-16T01:42:04.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.652+0000 7ffbebb2af40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56019d772e10 0x56019d814120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.652+0000 7ffbebb2af40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56019d773a60 0x56019d80e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.652+0000 7ffbebb2af40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56019d78d250 0x56019d80e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.652+0000 7ffbebb2af40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56019d78cca0 con 0x56019d78d250 2024-09-16T01:42:04.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.652+0000 7ffbebb2af40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56019d818c70 con 0x56019d773a60 2024-09-16T01:42:04.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.652+0000 7ffbebb2af40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56019d81b810 con 0x56019d772e10 2024-09-16T01:42:04.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.653+0000 7ffbe93bb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56019d78d250 0x56019d80e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.653+0000 7ffbe93bb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56019d78d250 0x56019d80e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44728/0 (socket says 172.21.15.88:44728) 2024-09-16T01:42:04.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.653+0000 7ffbe8bba640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56019d772e10 0x56019d814120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.653+0000 7ffbe3fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56019d773a60 0x56019d80e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.653+0000 7ffbe93bb640 1 -- 172.21.15.88:0/2598137401 learned_addr learned my addr 172.21.15.88:0/2598137401 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:04.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.653+0000 7ffbe93bb640 1 -- 172.21.15.88:0/2598137401 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56019d772e10 msgr2=0x56019d814120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:04.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.653+0000 7ffbe93bb640 1 --2- 172.21.15.88:0/2598137401 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56019d772e10 0x56019d814120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.653+0000 7ffbe93bb640 1 -- 172.21.15.88:0/2598137401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56019d773a60 msgr2=0x56019d80e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:04.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.653+0000 7ffbe93bb640 1 --2- 172.21.15.88:0/2598137401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56019d773a60 0x56019d80e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.653+0000 7ffbe93bb640 1 -- 172.21.15.88:0/2598137401 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56019d814d10 con 0x56019d78d250 2024-09-16T01:42:04.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.653+0000 7ffbe93bb640 1 --2- 172.21.15.88:0/2598137401 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56019d78d250 0x56019d80e920 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7ffbe4025a20 tx=0x7ffbe4003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:04.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.653+0000 7ffbe1ffb640 1 -- 172.21.15.88:0/2598137401 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbe4068070 con 0x56019d78d250 2024-09-16T01:42:04.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.653+0000 7ffbe1ffb640 1 -- 172.21.15.88:0/2598137401 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbe4059450 con 0x56019d78d250 2024-09-16T01:42:04.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.653+0000 7ffbe1ffb640 1 -- 172.21.15.88:0/2598137401 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbe4054040 con 0x56019d78d250 2024-09-16T01:42:04.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.653+0000 7ffbebb2af40 1 -- 172.21.15.88:0/2598137401 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56019d8186b0 con 0x56019d78d250 2024-09-16T01:42:04.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.653+0000 7ffbebb2af40 1 -- 172.21.15.88:0/2598137401 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56019d814660 con 0x56019d78d250 2024-09-16T01:42:04.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.655+0000 7ffbe1ffb640 1 -- 172.21.15.88:0/2598137401 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ffbe406b070 con 0x56019d78d250 2024-09-16T01:42:04.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.655+0000 7ffbe1ffb640 1 --2- 172.21.15.88:0/2598137401 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ffbcc06ac90 0x7ffbcc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.655+0000 7ffbe8bba640 1 --2- 172.21.15.88:0/2598137401 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ffbcc06ac90 0x7ffbcc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.655+0000 7ffbe1ffb640 1 -- 172.21.15.88:0/2598137401 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 7200+0+0 (secure 0 0 0) 0x7ffbe40ce1b0 con 0x56019d78d250 2024-09-16T01:42:04.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.655+0000 7ffbebb2af40 1 --2- 172.21.15.88:0/2598137401 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x56019d80f370 0x56019d80b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.655+0000 7ffbebb2af40 1 -- 172.21.15.88:0/2598137401 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:1 12.14 12:2e2fa760:::200.00000001:head [writefull 0~80 in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x56019d80b910 con 0x56019d80f370 2024-09-16T01:42:04.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.655+0000 7ffbe8bba640 1 --2- 172.21.15.88:0/2598137401 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ffbcc06ac90 0x7ffbcc06d150 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc016040 tx=0x7ffbdc00c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:04.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.656+0000 7ffbe3fff640 1 --2- 172.21.15.88:0/2598137401 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x56019d80f370 0x56019d80b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.656+0000 7ffbe3fff640 1 --2- 172.21.15.88:0/2598137401 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x56019d80f370 0x56019d80b3d0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:04.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.658+0000 7ffbe3fff640 1 -- 172.21.15.88:0/2598137401 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(1 200.00000001 [writefull 0~80] v104'396 uv396 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ffbd4009850 con 0x56019d80f370 2024-09-16T01:42:04.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.658+0000 7ffbebb2af40 1 -- 172.21.15.88:0/2598137401 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x56019d80f370 msgr2=0x56019d80b3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:04.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.658+0000 7ffbebb2af40 1 --2- 172.21.15.88:0/2598137401 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x56019d80f370 0x56019d80b3d0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.658+0000 7ffbebb2af40 1 -- 172.21.15.88:0/2598137401 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ffbcc06ac90 msgr2=0x7ffbcc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:04.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.658+0000 7ffbebb2af40 1 --2- 172.21.15.88:0/2598137401 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ffbcc06ac90 0x7ffbcc06d150 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc016040 tx=0x7ffbdc00c000 comp rx=0 tx=0).stop 2024-09-16T01:42:04.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.658+0000 7ffbebb2af40 1 -- 172.21.15.88:0/2598137401 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56019d78d250 msgr2=0x56019d80e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:04.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.658+0000 7ffbebb2af40 1 --2- 172.21.15.88:0/2598137401 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56019d78d250 0x56019d80e920 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7ffbe4025a20 tx=0x7ffbe4003330 comp rx=0 tx=0).stop 2024-09-16T01:42:04.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.658+0000 7ffbe93bb640 1 -- 172.21.15.88:0/2598137401 reap_dead start 2024-09-16T01:42:04.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.658+0000 7ffbebb2af40 1 -- 172.21.15.88:0/2598137401 shutdown_connections 2024-09-16T01:42:04.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.658+0000 7ffbebb2af40 1 -- 172.21.15.88:0/2598137401 >> 172.21.15.88:0/2598137401 conn(0x56019d78e750 msgr2=0x56019d772650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:04.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.658+0000 7ffbebb2af40 1 -- 172.21.15.88:0/2598137401 shutdown_connections 2024-09-16T01:42:04.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.658+0000 7ffbebb2af40 1 -- 172.21.15.88:0/2598137401 wait complete. 2024-09-16T01:42:04.665 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:42:04.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.804+0000 7f9270053640 1 Processor -- start 2024-09-16T01:42:04.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.805+0000 7f9270053640 1 -- start start 2024-09-16T01:42:04.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.806+0000 7f9270053640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92680d0760 0x7f92680b5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.806+0000 7f9270053640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92680cd9b0 con 0x7f92680b4e60 2024-09-16T01:42:04.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.806+0000 7f9270053640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92680cdc90 con 0x7f926815cb20 2024-09-16T01:42:04.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.806+0000 7f9270053640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f92680ce5c0 con 0x7f92680d0760 2024-09-16T01:42:04.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.806+0000 7f926ddef640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f92680b4e60 0x7f92680b5c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:40040/0 (socket says 172.21.15.88:40040) 2024-09-16T01:42:04.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.806+0000 7f926ddef640 1 -- 172.21.15.88:0/1078038005 learned_addr learned my addr 172.21.15.88:0/1078038005 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:04.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.806+0000 7f926d5ee640 1 --2- 172.21.15.88:0/1078038005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92680d0760 0x7f92680b5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.806+0000 7f926cded640 1 -- 172.21.15.88:0/1078038005 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1605145274 0 0) 0x7f92680cd9b0 con 0x7f92680b4e60 2024-09-16T01:42:04.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.806+0000 7f926cded640 1 -- 172.21.15.88:0/1078038005 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9250003520 con 0x7f92680b4e60 2024-09-16T01:42:04.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.806+0000 7f926cded640 1 -- 172.21.15.88:0/1078038005 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 235004308 0 0) 0x7f92680cdc90 con 0x7f926815cb20 2024-09-16T01:42:04.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.806+0000 7f926cded640 1 -- 172.21.15.88:0/1078038005 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92680cd9b0 con 0x7f926815cb20 2024-09-16T01:42:04.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.806+0000 7f926d5ee640 1 -- 172.21.15.88:0/1078038005 >> v1:172.21.15.154:6789/0 conn(0x7f926815cb20 legacy=0x7f926815cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:04.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.806+0000 7f926d5ee640 1 -- 172.21.15.88:0/1078038005 >> v1:172.21.15.88:6789/0 conn(0x7f92680b4e60 legacy=0x7f92680b5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:04.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.806+0000 7f926cded640 1 -- 172.21.15.88:0/1078038005 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1610663888 0 0) 0x7f9250003520 con 0x7f92680b4e60 2024-09-16T01:42:04.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.806+0000 7f926d5ee640 1 -- 172.21.15.88:0/1078038005 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92680cdf70 con 0x7f92680d0760 2024-09-16T01:42:04.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.807+0000 7f926d5ee640 1 --2- 172.21.15.88:0/1078038005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92680d0760 0x7f92680b5240 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f925c0097d0 tx=0x7f925c02a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70bcadb4a36f4f8f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:04.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.807+0000 7f926cded640 1 -- 172.21.15.88:0/1078038005 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925c037070 con 0x7f92680d0760 2024-09-16T01:42:04.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.807+0000 7f926cded640 1 -- 172.21.15.88:0/1078038005 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f925c02ac30 con 0x7f92680d0760 2024-09-16T01:42:04.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.807+0000 7f926cded640 1 -- 172.21.15.88:0/1078038005 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9260001670 con 0x7f92680d0760 2024-09-16T01:42:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.808+0000 7f9270053640 1 -- 172.21.15.88:0/1078038005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92680d0760 msgr2=0x7f92680b5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.808+0000 7f9270053640 1 --2- 172.21.15.88:0/1078038005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92680d0760 0x7f92680b5240 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f925c0097d0 tx=0x7f925c02a1a0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.808+0000 7f9270053640 1 -- 172.21.15.88:0/1078038005 shutdown_connections 2024-09-16T01:42:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.808+0000 7f9270053640 1 --2- 172.21.15.88:0/1078038005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92680d0760 0x7f92680b5240 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.808+0000 7f9270053640 1 -- 172.21.15.88:0/1078038005 >> 172.21.15.88:0/1078038005 conn(0x7f926806cb50 msgr2=0x7f926806cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.808+0000 7f9270053640 1 -- 172.21.15.88:0/1078038005 shutdown_connections 2024-09-16T01:42:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.808+0000 7f9270053640 1 -- 172.21.15.88:0/1078038005 wait complete. 2024-09-16T01:42:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.809+0000 7f9270053640 1 Processor -- start 2024-09-16T01:42:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.809+0000 7f9270053640 1 -- start start 2024-09-16T01:42:04.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.810+0000 7f9270053640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92680b4e60 0x7f926814a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.810+0000 7f9270053640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92680d0760 0x7f92681542c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.810+0000 7f9270053640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f926815cb20 0x7f9268154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.810+0000 7f9270053640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f926815fc00 con 0x7f92680b4e60 2024-09-16T01:42:04.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.810+0000 7f9270053640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f92680b5780 con 0x7f92680d0760 2024-09-16T01:42:04.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.810+0000 7f9270053640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f926815fd80 con 0x7f926815cb20 2024-09-16T01:42:04.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.810+0000 7f926ddef640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92680b4e60 0x7f926814a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.810+0000 7f926ddef640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92680b4e60 0x7f926814a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44742/0 (socket says 172.21.15.88:44742) 2024-09-16T01:42:04.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.810+0000 7f926ddef640 1 -- 172.21.15.88:0/1182109478 learned_addr learned my addr 172.21.15.88:0/1182109478 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:04.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.810+0000 7f926d5ee640 1 --2- 172.21.15.88:0/1182109478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92680d0760 0x7f92681542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.810+0000 7f926e5f0640 1 --2- 172.21.15.88:0/1182109478 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f926815cb20 0x7f9268154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.810+0000 7f926ddef640 1 -- 172.21.15.88:0/1182109478 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f926815cb20 msgr2=0x7f9268154800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:04.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.810+0000 7f926ddef640 1 --2- 172.21.15.88:0/1182109478 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f926815cb20 0x7f9268154800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.810+0000 7f926ddef640 1 -- 172.21.15.88:0/1182109478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92680d0760 msgr2=0x7f92681542c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:04.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.810+0000 7f926ddef640 1 --2- 172.21.15.88:0/1182109478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92680d0760 0x7f92681542c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:04.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.810+0000 7f926ddef640 1 -- 172.21.15.88:0/1182109478 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92681572c0 con 0x7f92680b4e60 2024-09-16T01:42:04.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.810+0000 7f926ddef640 1 --2- 172.21.15.88:0/1182109478 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92680b4e60 0x7f926814a720 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f926000c4f0 tx=0x7f926000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:04.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.810+0000 7f92667fc640 1 -- 172.21.15.88:0/1182109478 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9260017070 con 0x7f92680b4e60 2024-09-16T01:42:04.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.810+0000 7f92667fc640 1 -- 172.21.15.88:0/1182109478 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9260005ce0 con 0x7f92680b4e60 2024-09-16T01:42:04.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.811+0000 7f9270053640 1 -- 172.21.15.88:0/1182109478 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9268154ea0 con 0x7f92680b4e60 2024-09-16T01:42:04.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.811+0000 7f92667fc640 1 -- 172.21.15.88:0/1182109478 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9260007290 con 0x7f92680b4e60 2024-09-16T01:42:04.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.811+0000 7f9270053640 1 -- 172.21.15.88:0/1182109478 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9268155460 con 0x7f92680b4e60 2024-09-16T01:42:04.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.811+0000 7f9270053640 1 -- 172.21.15.88:0/1182109478 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92680b5240 con 0x7f92680b4e60 2024-09-16T01:42:04.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.812+0000 7f92667fc640 1 -- 172.21.15.88:0/1182109478 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f926001a070 con 0x7f92680b4e60 2024-09-16T01:42:04.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.812+0000 7f92667fc640 1 --2- 172.21.15.88:0/1182109478 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f925406c2b0 0x7f925406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:04.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.812+0000 7f92667fc640 1 -- 172.21.15.88:0/1182109478 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 7200+0+0 (secure 0 0 0) 0x7f926008acd0 con 0x7f92680b4e60 2024-09-16T01:42:04.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.812+0000 7f926d5ee640 1 --2- 172.21.15.88:0/1182109478 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f925406c2b0 0x7f925406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:04.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.813+0000 7f926d5ee640 1 --2- 172.21.15.88:0/1182109478 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f925406c2b0 0x7f925406e770 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f925c034000 tx=0x7f925c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:04.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:04.817+0000 7f92667fc640 1 -- 172.21.15.88:0/1182109478 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f92600588b0 con 0x7f92680b4e60 2024-09-16T01:42:05.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.015+0000 7f9270053640 1 -- 172.21.15.88:0/1182109478 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f92680701a0 con 0x7f92680b4e60 2024-09-16T01:42:05.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.396+0000 7f92667fc640 1 -- 172.21.15.88:0/1182109478 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v173) ==== 162+0+0 (secure 0 0 0) 0x7f926005f0d0 con 0x7f92680b4e60 2024-09-16T01:42:05.398 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:42:05.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.399+0000 7f9270053640 1 -- 172.21.15.88:0/1182109478 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f925406c2b0 msgr2=0x7f925406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:05.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.399+0000 7f9270053640 1 --2- 172.21.15.88:0/1182109478 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f925406c2b0 0x7f925406e770 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f925c034000 tx=0x7f925c02e000 comp rx=0 tx=0).stop 2024-09-16T01:42:05.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.399+0000 7f9270053640 1 -- 172.21.15.88:0/1182109478 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92680b4e60 msgr2=0x7f926814a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:05.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.399+0000 7f9270053640 1 --2- 172.21.15.88:0/1182109478 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92680b4e60 0x7f926814a720 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f926000c4f0 tx=0x7f926000c9b0 comp rx=0 tx=0).stop 2024-09-16T01:42:05.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.399+0000 7f9270053640 1 -- 172.21.15.88:0/1182109478 shutdown_connections 2024-09-16T01:42:05.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.399+0000 7f9270053640 1 --2- 172.21.15.88:0/1182109478 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f925406c2b0 0x7f925406e770 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:05.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.399+0000 7f9270053640 1 --2- 172.21.15.88:0/1182109478 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f926815cb20 0x7f9268154800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:05.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.399+0000 7f9270053640 1 --2- 172.21.15.88:0/1182109478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92680d0760 0x7f92681542c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:05.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.399+0000 7f9270053640 1 --2- 172.21.15.88:0/1182109478 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92680b4e60 0x7f926814a720 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:05.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.399+0000 7f9270053640 1 -- 172.21.15.88:0/1182109478 >> 172.21.15.88:0/1182109478 conn(0x7f926806cb50 msgr2=0x7f92680b1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:05.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.399+0000 7f9270053640 1 -- 172.21.15.88:0/1182109478 shutdown_connections 2024-09-16T01:42:05.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.399+0000 7f9270053640 1 -- 172.21.15.88:0/1182109478 wait complete. 2024-09-16T01:42:05.412 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:42:05.411+0000 7f7498546640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (22) Invalid argument 2024-09-16T01:42:05.413 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:42:05.411+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:42:05.413 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:42:05.411+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:42:05.414 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:05.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.519+0000 7f2633253640 1 Processor -- start 2024-09-16T01:42:05.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.519+0000 7f2633253640 1 -- start start 2024-09-16T01:42:05.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.520+0000 7f2633253640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f262c0715e0 0x7f262c0b6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:05.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.520+0000 7f2633253640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f262c073160 0x7f262c073590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:05.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.520+0000 7f2633253640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f262c0d0270 con 0x7f262c15c550 2024-09-16T01:42:05.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.520+0000 7f2633253640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f262c0b7350 con 0x7f262c0715e0 2024-09-16T01:42:05.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.520+0000 7f2633253640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f262c15c930 con 0x7f262c073160 2024-09-16T01:42:05.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.520+0000 7f2623fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f262c0715e0 0x7f262c0b6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:05.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.520+0000 7f2623fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f262c0715e0 0x7f262c0b6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44752/0 (socket says 172.21.15.88:44752) 2024-09-16T01:42:05.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.520+0000 7f2623fff640 1 -- 172.21.15.88:0/4100425525 learned_addr learned my addr 172.21.15.88:0/4100425525 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:05.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.521+0000 7f2623fff640 1 -- 172.21.15.88:0/4100425525 >> v2:172.21.15.154:3300/0 conn(0x7f262c073160 msgr2=0x7f262c073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:05.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.521+0000 7f26317f0640 1 --2- 172.21.15.88:0/4100425525 >> v2:172.21.15.154:3300/0 conn(0x7f262c073160 0x7f262c073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:05.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.521+0000 7f2623fff640 1 --2- 172.21.15.88:0/4100425525 >> v2:172.21.15.154:3300/0 conn(0x7f262c073160 0x7f262c073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:05.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.521+0000 7f2623fff640 1 -- 172.21.15.88:0/4100425525 >> v1:172.21.15.154:6789/0 conn(0x7f262c15c550 legacy=0x7f262c0b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:05.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.521+0000 7f2623fff640 1 -- 172.21.15.88:0/4100425525 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f262c071a50 con 0x7f262c0715e0 2024-09-16T01:42:05.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.521+0000 7f2623fff640 1 --2- 172.21.15.88:0/4100425525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f262c0715e0 0x7f262c0b6e10 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f262c072d90 tx=0x7f261c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=511ecdb1b62aa39d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:05.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.521+0000 7f26237fe640 1 -- 172.21.15.88:0/4100425525 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261c037070 con 0x7f262c0715e0 2024-09-16T01:42:05.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.522+0000 7f26237fe640 1 -- 172.21.15.88:0/4100425525 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f261c032440 con 0x7f262c0715e0 2024-09-16T01:42:05.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.522+0000 7f26237fe640 1 -- 172.21.15.88:0/4100425525 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261c032810 con 0x7f262c0715e0 2024-09-16T01:42:05.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.522+0000 7f2633253640 1 -- 172.21.15.88:0/4100425525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f262c0715e0 msgr2=0x7f262c0b6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:05.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.522+0000 7f2633253640 1 --2- 172.21.15.88:0/4100425525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f262c0715e0 0x7f262c0b6e10 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f262c072d90 tx=0x7f261c02a230 comp rx=0 tx=0).stop 2024-09-16T01:42:05.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.522+0000 7f2633253640 1 -- 172.21.15.88:0/4100425525 shutdown_connections 2024-09-16T01:42:05.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.522+0000 7f2633253640 1 --2- 172.21.15.88:0/4100425525 >> v2:172.21.15.154:3300/0 conn(0x7f262c073160 0x7f262c073590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:05.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.522+0000 7f2633253640 1 --2- 172.21.15.88:0/4100425525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f262c0715e0 0x7f262c0b6e10 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:05.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.522+0000 7f2633253640 1 -- 172.21.15.88:0/4100425525 >> 172.21.15.88:0/4100425525 conn(0x7f262c0b4240 msgr2=0x7f262c0b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:05.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.522+0000 7f2633253640 1 -- 172.21.15.88:0/4100425525 shutdown_connections 2024-09-16T01:42:05.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.522+0000 7f2633253640 1 -- 172.21.15.88:0/4100425525 wait complete. 2024-09-16T01:42:05.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.523+0000 7f2633253640 1 Processor -- start 2024-09-16T01:42:05.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.523+0000 7f2633253640 1 -- start start 2024-09-16T01:42:05.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.524+0000 7f2633253640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f262c0715e0 0x7f262c15db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:05.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.524+0000 7f2633253640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f262c073160 0x7f262c1566c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:05.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.524+0000 7f2633253640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f262c15c550 0x7f262c14f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:05.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.524+0000 7f2633253640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f262c0b7580 con 0x7f262c073160 2024-09-16T01:42:05.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.524+0000 7f2633253640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f262c073ad0 con 0x7f262c0715e0 2024-09-16T01:42:05.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.524+0000 7f2633253640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f262c0b7860 con 0x7f262c15c550 2024-09-16T01:42:05.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.524+0000 7f2623fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f262c073160 0x7f262c1566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:05.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.524+0000 7f2623fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f262c073160 0x7f262c1566c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44758/0 (socket says 172.21.15.88:44758) 2024-09-16T01:42:05.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.524+0000 7f2623fff640 1 -- 172.21.15.88:0/4052515132 learned_addr learned my addr 172.21.15.88:0/4052515132 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:05.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.524+0000 7f2623fff640 1 -- 172.21.15.88:0/4052515132 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f262c15c550 msgr2=0x7f262c14f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:05.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.524+0000 7f2623fff640 1 --2- 172.21.15.88:0/4052515132 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f262c15c550 0x7f262c14f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:05.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.524+0000 7f2623fff640 1 -- 172.21.15.88:0/4052515132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f262c0715e0 msgr2=0x7f262c15db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:05.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.524+0000 7f2623fff640 1 --2- 172.21.15.88:0/4052515132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f262c0715e0 0x7f262c15db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:05.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.525+0000 7f2623fff640 1 -- 172.21.15.88:0/4052515132 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f262c15e040 con 0x7f262c073160 2024-09-16T01:42:05.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.525+0000 7f2623fff640 1 --2- 172.21.15.88:0/4052515132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f262c073160 0x7f262c1566c0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f261c000c00 tx=0x7f261c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:05.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.525+0000 7f26217fa640 1 -- 172.21.15.88:0/4052515132 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261c041020 con 0x7f262c073160 2024-09-16T01:42:05.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.525+0000 7f26217fa640 1 -- 172.21.15.88:0/4052515132 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f261c030970 con 0x7f262c073160 2024-09-16T01:42:05.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.525+0000 7f2633253640 1 -- 172.21.15.88:0/4052515132 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f262c1e5050 con 0x7f262c073160 2024-09-16T01:42:05.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.525+0000 7f26217fa640 1 -- 172.21.15.88:0/4052515132 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261c037040 con 0x7f262c073160 2024-09-16T01:42:05.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.525+0000 7f2633253640 1 -- 172.21.15.88:0/4052515132 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f262c15e8a0 con 0x7f262c073160 2024-09-16T01:42:05.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.526+0000 7f2633253640 1 -- 172.21.15.88:0/4052515132 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f262c0d07b0 con 0x7f262c073160 2024-09-16T01:42:05.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.526+0000 7f26217fa640 1 -- 172.21.15.88:0/4052515132 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f261c044020 con 0x7f262c073160 2024-09-16T01:42:05.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.527+0000 7f26217fa640 1 --2- 172.21.15.88:0/4052515132 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f261806c2b0 0x7f261806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:05.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.527+0000 7f26217fa640 1 -- 172.21.15.88:0/4052515132 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7286+0+0 (secure 0 0 0) 0x7f261c0ac4a0 con 0x7f262c073160 2024-09-16T01:42:05.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.527+0000 7f2630fef640 1 --2- 172.21.15.88:0/4052515132 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f261806c2b0 0x7f261806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:05.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.527+0000 7f2630fef640 1 --2- 172.21.15.88:0/4052515132 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f261806c2b0 0x7f261806e770 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2628003cc0 tx=0x7f26280079b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:05.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.531+0000 7f26217fa640 1 -- 172.21.15.88:0/4052515132 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f261c04b030 con 0x7f262c073160 2024-09-16T01:42:05.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.730+0000 7f2633253640 1 -- 172.21.15.88:0/4052515132 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f262c0693c0 con 0x7f262c073160 2024-09-16T01:42:05.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.731+0000 7f26217fa640 1 -- 172.21.15.88:0/4052515132 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) ==== 95+0+3566 (secure 0 0 0) 0x7f262c0d0270 con 0x7f262c073160 2024-09-16T01:42:05.733 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:05.733 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":175,"btime":"2024-09-16T01:42:05:412377+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:05.412321+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:05.733 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 175 2024-09-16T01:42:05.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.733+0000 7f2633253640 1 -- 172.21.15.88:0/4052515132 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f261806c2b0 msgr2=0x7f261806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:05.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.733+0000 7f2633253640 1 --2- 172.21.15.88:0/4052515132 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f261806c2b0 0x7f261806e770 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2628003cc0 tx=0x7f26280079b0 comp rx=0 tx=0).stop 2024-09-16T01:42:05.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.733+0000 7f2633253640 1 -- 172.21.15.88:0/4052515132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f262c073160 msgr2=0x7f262c1566c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:05.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.733+0000 7f2633253640 1 --2- 172.21.15.88:0/4052515132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f262c073160 0x7f262c1566c0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f261c000c00 tx=0x7f261c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:42:05.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.733+0000 7f2633253640 1 -- 172.21.15.88:0/4052515132 shutdown_connections 2024-09-16T01:42:05.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.733+0000 7f2633253640 1 --2- 172.21.15.88:0/4052515132 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f261806c2b0 0x7f261806e770 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:05.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.733+0000 7f2633253640 1 --2- 172.21.15.88:0/4052515132 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f262c15c550 0x7f262c14f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:05.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.733+0000 7f2633253640 1 --2- 172.21.15.88:0/4052515132 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f262c073160 0x7f262c1566c0 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:05.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.733+0000 7f2633253640 1 --2- 172.21.15.88:0/4052515132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f262c0715e0 0x7f262c15db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:05.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.733+0000 7f2633253640 1 -- 172.21.15.88:0/4052515132 >> 172.21.15.88:0/4052515132 conn(0x7f262c0b4240 msgr2=0x7f262c0ce620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:05.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.733+0000 7f2633253640 1 -- 172.21.15.88:0/4052515132 shutdown_connections 2024-09-16T01:42:05.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.733+0000 7f2633253640 1 -- 172.21.15.88:0/4052515132 wait complete. 2024-09-16T01:42:05.745 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:42:05.745 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:05.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.842+0000 7f971f882640 1 Processor -- start 2024-09-16T01:42:05.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.843+0000 7f971f882640 1 -- start start 2024-09-16T01:42:05.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.843+0000 7f971f882640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f97180b1990 0x7f97180d10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:05.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.843+0000 7f971f882640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9718071750 con 0x7f9718073080 2024-09-16T01:42:05.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.843+0000 7f971f882640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97180d0370 con 0x7f971815c670 2024-09-16T01:42:05.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.843+0000 7f971f882640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f97180d0c60 con 0x7f97180b1990 2024-09-16T01:42:05.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f971d61e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9718073080 0x7f97180b5460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:40052/0 (socket says 172.21.15.88:40052) 2024-09-16T01:42:05.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f971d61e640 1 -- 172.21.15.88:0/2648377766 learned_addr learned my addr 172.21.15.88:0/2648377766 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:05.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f971de1f640 1 --2- 172.21.15.88:0/2648377766 >> v2:172.21.15.154:3300/0 conn(0x7f97180b1990 0x7f97180d10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:05.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f9717fff640 1 -- 172.21.15.88:0/2648377766 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 147143547 0 0) 0x7f9718071750 con 0x7f9718073080 2024-09-16T01:42:05.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f9717fff640 1 -- 172.21.15.88:0/2648377766 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9700003520 con 0x7f9718073080 2024-09-16T01:42:05.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f9717fff640 1 -- 172.21.15.88:0/2648377766 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4267275421 0 0) 0x7f9700003520 con 0x7f9718073080 2024-09-16T01:42:05.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f9717fff640 1 -- 172.21.15.88:0/2648377766 >> v2:172.21.15.154:3300/0 conn(0x7f97180b1990 msgr2=0x7f97180d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:05.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f9717fff640 1 --2- 172.21.15.88:0/2648377766 >> v2:172.21.15.154:3300/0 conn(0x7f97180b1990 0x7f97180d10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:05.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f9717fff640 1 -- 172.21.15.88:0/2648377766 >> v1:172.21.15.154:6789/0 conn(0x7f971815c670 legacy=0x7f9718073460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:05.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f9717fff640 1 -- 172.21.15.88:0/2648377766 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9718071190 con 0x7f9718073080 2024-09-16T01:42:05.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f9717fff640 1 -- 172.21.15.88:0/2648377766 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f97080019f0 con 0x7f9718073080 2024-09-16T01:42:05.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f9717fff640 1 -- 172.21.15.88:0/2648377766 >> v1:172.21.15.88:6789/0 conn(0x7f9718073080 legacy=0x7f97180b5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:05.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f9717fff640 1 --2- 172.21.15.88:0/2648377766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97000039b0 0x7f9700003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:05.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f9717fff640 1 --2- 172.21.15.88:0/2648377766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9700004800 0x7f9700004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:05.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f9717fff640 1 --2- 172.21.15.88:0/2648377766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97000051c0 0x7f970000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:05.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f9717fff640 1 -- 172.21.15.88:0/2648377766 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f97180d0c60 con 0x7f97000051c0 2024-09-16T01:42:05.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f9717fff640 1 -- 172.21.15.88:0/2648377766 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f970000c060 con 0x7f97000039b0 2024-09-16T01:42:05.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f9717fff640 1 -- 172.21.15.88:0/2648377766 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f970000c260 con 0x7f9700004800 2024-09-16T01:42:05.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f971de1f640 1 --2- 172.21.15.88:0/2648377766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97000051c0 0x7f970000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:05.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f971d61e640 1 --2- 172.21.15.88:0/2648377766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97000039b0 0x7f9700003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:05.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f971ce1d640 1 --2- 172.21.15.88:0/2648377766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9700004800 0x7f9700004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:05.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f971de1f640 1 -- 172.21.15.88:0/2648377766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9700004800 msgr2=0x7f9700004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:05.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f971de1f640 1 --2- 172.21.15.88:0/2648377766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9700004800 0x7f9700004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:05.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f971de1f640 1 -- 172.21.15.88:0/2648377766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97000039b0 msgr2=0x7f9700003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:05.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f971de1f640 1 --2- 172.21.15.88:0/2648377766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97000039b0 0x7f9700003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:05.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f971de1f640 1 -- 172.21.15.88:0/2648377766 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9700003520 con 0x7f97000051c0 2024-09-16T01:42:05.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.844+0000 7f971d61e640 1 --2- 172.21.15.88:0/2648377766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97000039b0 0x7f9700003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:05.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.845+0000 7f971de1f640 1 -- 172.21.15.88:0/2648377766 reap_dead start 2024-09-16T01:42:05.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.845+0000 7f971de1f640 1 --2- 172.21.15.88:0/2648377766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97000051c0 0x7f970000b9c0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f9710009ac0 tx=0x7f9700005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=935f2ae74c794021 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:05.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.845+0000 7f9717fff640 1 -- 172.21.15.88:0/2648377766 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9710009690 con 0x7f97000051c0 2024-09-16T01:42:05.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.845+0000 7f9717fff640 1 -- 172.21.15.88:0/2648377766 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9710004020 con 0x7f97000051c0 2024-09-16T01:42:05.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.845+0000 7f9717fff640 1 -- 172.21.15.88:0/2648377766 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9710004410 con 0x7f97000051c0 2024-09-16T01:42:05.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.846+0000 7f971f882640 1 -- 172.21.15.88:0/2648377766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97000051c0 msgr2=0x7f970000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:05.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.846+0000 7f971f882640 1 --2- 172.21.15.88:0/2648377766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97000051c0 0x7f970000b9c0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f9710009ac0 tx=0x7f9700005830 comp rx=0 tx=0).stop 2024-09-16T01:42:05.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.846+0000 7f971f882640 1 -- 172.21.15.88:0/2648377766 shutdown_connections 2024-09-16T01:42:05.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.846+0000 7f971f882640 1 --2- 172.21.15.88:0/2648377766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97000051c0 0x7f970000b9c0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:05.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.846+0000 7f971f882640 1 -- 172.21.15.88:0/2648377766 >> 172.21.15.88:0/2648377766 conn(0x7f97180d24d0 msgr2=0x7f97180d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:05.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.846+0000 7f971f882640 1 -- 172.21.15.88:0/2648377766 shutdown_connections 2024-09-16T01:42:05.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.846+0000 7f971f882640 1 -- 172.21.15.88:0/2648377766 wait complete. 2024-09-16T01:42:05.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.847+0000 7f971f882640 1 Processor -- start 2024-09-16T01:42:05.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.847+0000 7f971f882640 1 -- start start 2024-09-16T01:42:05.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.848+0000 7f971f882640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97000051c0 0x7f97181581e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:05.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.848+0000 7f971f882640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97180b6080 0x7f971815a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:05.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.848+0000 7f971f882640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9718158aa0 0x7f9718158f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:05.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.848+0000 7f971f882640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f97180d08f0 con 0x7f97000051c0 2024-09-16T01:42:05.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.848+0000 7f971f882640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f97180d0610 con 0x7f9718158aa0 2024-09-16T01:42:05.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.848+0000 7f971f882640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f97180d1f00 con 0x7f97180b6080 2024-09-16T01:42:05.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.848+0000 7f971d61e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97000051c0 0x7f97181581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:05.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.848+0000 7f971d61e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97000051c0 0x7f97181581e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44766/0 (socket says 172.21.15.88:44766) 2024-09-16T01:42:05.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.848+0000 7f971d61e640 1 -- 172.21.15.88:0/1324007621 learned_addr learned my addr 172.21.15.88:0/1324007621 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:05.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.848+0000 7f971d61e640 1 -- 172.21.15.88:0/1324007621 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97180b6080 msgr2=0x7f971815a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:05.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.848+0000 7f971d61e640 1 --2- 172.21.15.88:0/1324007621 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97180b6080 0x7f971815a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:05.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.848+0000 7f971d61e640 1 -- 172.21.15.88:0/1324007621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9718158aa0 msgr2=0x7f9718158f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:05.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.848+0000 7f971d61e640 1 --2- 172.21.15.88:0/1324007621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9718158aa0 0x7f9718158f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:05.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.848+0000 7f971d61e640 1 -- 172.21.15.88:0/1324007621 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97180b78e0 con 0x7f97000051c0 2024-09-16T01:42:05.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.848+0000 7f971d61e640 1 --2- 172.21.15.88:0/1324007621 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97000051c0 0x7f97181581e0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f9708007990 tx=0x7f970800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:05.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.849+0000 7f9715ffb640 1 -- 172.21.15.88:0/1324007621 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f970800edf0 con 0x7f97000051c0 2024-09-16T01:42:05.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.849+0000 7f9715ffb640 1 -- 172.21.15.88:0/1324007621 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9708018070 con 0x7f97000051c0 2024-09-16T01:42:05.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.849+0000 7f971f882640 1 -- 172.21.15.88:0/1324007621 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97180b6840 con 0x7f97000051c0 2024-09-16T01:42:05.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.849+0000 7f9715ffb640 1 -- 172.21.15.88:0/1324007621 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9708013640 con 0x7f97000051c0 2024-09-16T01:42:05.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.849+0000 7f971f882640 1 -- 172.21.15.88:0/1324007621 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97180b6e00 con 0x7f97000051c0 2024-09-16T01:42:05.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.850+0000 7f971f882640 1 -- 172.21.15.88:0/1324007621 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97180b5460 con 0x7f97000051c0 2024-09-16T01:42:05.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.850+0000 7f9715ffb640 1 -- 172.21.15.88:0/1324007621 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f97080137e0 con 0x7f97000051c0 2024-09-16T01:42:05.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.851+0000 7f9715ffb640 1 --2- 172.21.15.88:0/1324007621 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f970406c2e0 0x7f970406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:05.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.851+0000 7f9715ffb640 1 -- 172.21.15.88:0/1324007621 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7286+0+0 (secure 0 0 0) 0x7f970808c050 con 0x7f97000051c0 2024-09-16T01:42:05.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.851+0000 7f971ce1d640 1 --2- 172.21.15.88:0/1324007621 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f970406c2e0 0x7f970406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:05.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.851+0000 7f971ce1d640 1 --2- 172.21.15.88:0/1324007621 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f970406c2e0 0x7f970406e7a0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f9718158170 tx=0x7f970c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:05.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:05.855+0000 7f9715ffb640 1 -- 172.21.15.88:0/1324007621 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f970805a100 con 0x7f97000051c0 2024-09-16T01:42:06.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.051+0000 7f971f882640 1 -- 172.21.15.88:0/1324007621 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9718155a80 con 0x7f97000051c0 2024-09-16T01:42:06.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.052+0000 7f9715ffb640 1 -- 172.21.15.88:0/1324007621 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) ==== 95+0+3566 (secure 0 0 0) 0x7f9708059a00 con 0x7f97000051c0 2024-09-16T01:42:06.054 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:06.054 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":175,"btime":"2024-09-16T01:42:05:412377+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:05.412321+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:06.054 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 175 2024-09-16T01:42:06.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.054+0000 7f971f882640 1 -- 172.21.15.88:0/1324007621 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f970406c2e0 msgr2=0x7f970406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:06.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.054+0000 7f971f882640 1 --2- 172.21.15.88:0/1324007621 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f970406c2e0 0x7f970406e7a0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f9718158170 tx=0x7f970c009040 comp rx=0 tx=0).stop 2024-09-16T01:42:06.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.054+0000 7f971f882640 1 -- 172.21.15.88:0/1324007621 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97000051c0 msgr2=0x7f97181581e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:06.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.054+0000 7f971f882640 1 --2- 172.21.15.88:0/1324007621 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97000051c0 0x7f97181581e0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f9708007990 tx=0x7f970800e530 comp rx=0 tx=0).stop 2024-09-16T01:42:06.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.054+0000 7f971f882640 1 -- 172.21.15.88:0/1324007621 shutdown_connections 2024-09-16T01:42:06.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.054+0000 7f971f882640 1 --2- 172.21.15.88:0/1324007621 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f970406c2e0 0x7f970406e7a0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:06.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.054+0000 7f971f882640 1 --2- 172.21.15.88:0/1324007621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9718158aa0 0x7f9718158f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:06.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.054+0000 7f971f882640 1 --2- 172.21.15.88:0/1324007621 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97180b6080 0x7f971815a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:06.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.054+0000 7f971f882640 1 --2- 172.21.15.88:0/1324007621 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97000051c0 0x7f97181581e0 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:06.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.054+0000 7f971f882640 1 -- 172.21.15.88:0/1324007621 >> 172.21.15.88:0/1324007621 conn(0x7f97180d24d0 msgr2=0x7f971815ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:06.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.054+0000 7f971f882640 1 -- 172.21.15.88:0/1324007621 shutdown_connections 2024-09-16T01:42:06.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.054+0000 7f971f882640 1 -- 172.21.15.88:0/1324007621 wait complete. 2024-09-16T01:42:06.067 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:42:06.067 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:06.127 INFO:tasks.ceph.mds.c.smithi088.stderr: -35> 2024-09-16T01:42:01.374+0000 7f35a9dca640 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x200: (22) Invalid argument 2024-09-16T01:42:06.150 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:42:06.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.166+0000 7fa5fce53640 1 Processor -- start 2024-09-16T01:42:06.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.166+0000 7fa5fce53640 1 -- start start 2024-09-16T01:42:06.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.167+0000 7fa5fce53640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa5f80d0fe0 0x7fa5f806cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:06.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.167+0000 7fa5fce53640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5f806d260 0x7fa5f80b5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:06.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.167+0000 7fa5fce53640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5f815f130 con 0x7fa5f815b460 2024-09-16T01:42:06.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.167+0000 7fa5fce53640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa5f815eee0 con 0x7fa5f80d0fe0 2024-09-16T01:42:06.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.167+0000 7fa5fce53640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa5f815f410 con 0x7fa5f806d260 2024-09-16T01:42:06.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.167+0000 7fa5f5d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa5f80d0fe0 0x7fa5f806cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:06.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.167+0000 7fa5f5d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa5f80d0fe0 0x7fa5f806cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44784/0 (socket says 172.21.15.88:44784) 2024-09-16T01:42:06.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.167+0000 7fa5f5d9b640 1 -- 172.21.15.88:0/3347295355 learned_addr learned my addr 172.21.15.88:0/3347295355 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:06.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.167+0000 7fa5f6d9d640 1 --2- 172.21.15.88:0/3347295355 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5f806d260 0x7fa5f80b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:06.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.168+0000 7fa5f5d9b640 1 -- 172.21.15.88:0/3347295355 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5f806d260 msgr2=0x7fa5f80b5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:06.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.168+0000 7fa5f5d9b640 1 --2- 172.21.15.88:0/3347295355 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5f806d260 0x7fa5f80b5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:06.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.168+0000 7fa5f5d9b640 1 -- 172.21.15.88:0/3347295355 >> v1:172.21.15.154:6789/0 conn(0x7fa5f815b460 legacy=0x7fa5f80b44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:06.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.168+0000 7fa5f5d9b640 1 -- 172.21.15.88:0/3347295355 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5f815f6f0 con 0x7fa5f80d0fe0 2024-09-16T01:42:06.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.168+0000 7fa5f5d9b640 1 --2- 172.21.15.88:0/3347295355 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5f80d0fe0 0x7fa5f806cd20 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec009870 tx=0x7fa5ec02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adbde0026286c7a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:06.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.168+0000 7fa5f559a640 1 -- 172.21.15.88:0/3347295355 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec037070 con 0x7fa5f80d0fe0 2024-09-16T01:42:06.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.168+0000 7fa5f559a640 1 -- 172.21.15.88:0/3347295355 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5ec032440 con 0x7fa5f80d0fe0 2024-09-16T01:42:06.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.168+0000 7fa5f559a640 1 -- 172.21.15.88:0/3347295355 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec032810 con 0x7fa5f80d0fe0 2024-09-16T01:42:06.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.169+0000 7fa5fce53640 1 -- 172.21.15.88:0/3347295355 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5f80d0fe0 msgr2=0x7fa5f806cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:06.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.169+0000 7fa5fce53640 1 --2- 172.21.15.88:0/3347295355 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5f80d0fe0 0x7fa5f806cd20 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec009870 tx=0x7fa5ec02a270 comp rx=0 tx=0).stop 2024-09-16T01:42:06.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.169+0000 7fa5fce53640 1 -- 172.21.15.88:0/3347295355 shutdown_connections 2024-09-16T01:42:06.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.169+0000 7fa5fce53640 1 --2- 172.21.15.88:0/3347295355 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5f806d260 0x7fa5f80b5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:06.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.169+0000 7fa5fce53640 1 --2- 172.21.15.88:0/3347295355 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5f80d0fe0 0x7fa5f806cd20 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:06.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.169+0000 7fa5fce53640 1 -- 172.21.15.88:0/3347295355 >> 172.21.15.88:0/3347295355 conn(0x7fa5f80b6ec0 msgr2=0x7fa5f80b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:06.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.169+0000 7fa5fce53640 1 -- 172.21.15.88:0/3347295355 shutdown_connections 2024-09-16T01:42:06.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.169+0000 7fa5fce53640 1 -- 172.21.15.88:0/3347295355 wait complete. 2024-09-16T01:42:06.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.170+0000 7fa5fce53640 1 Processor -- start 2024-09-16T01:42:06.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.170+0000 7fa5fce53640 1 -- start start 2024-09-16T01:42:06.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.171+0000 7fa5fce53640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5f806d260 0x7fa5f814eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.171+0000 7fa5fce53640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5f80d0fe0 0x7fa5f80cfee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.171+0000 7fa5fce53640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5f815b460 0x7fa5f80afee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.171+0000 7fa5fce53640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f80b5740 con 0x7fa5f80d0fe0 2024-09-16T01:42:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.171+0000 7fa5fce53640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f80b55c0 con 0x7fa5f815b460 2024-09-16T01:42:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.171+0000 7fa5fce53640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa5f80b58c0 con 0x7fa5f806d260 2024-09-16T01:42:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.171+0000 7fa5f5d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5f80d0fe0 0x7fa5f80cfee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.171+0000 7fa5f5d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5f80d0fe0 0x7fa5f80cfee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44798/0 (socket says 172.21.15.88:44798) 2024-09-16T01:42:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.171+0000 7fa5f5d9b640 1 -- 172.21.15.88:0/1909827715 learned_addr learned my addr 172.21.15.88:0/1909827715 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.171+0000 7fa5f659c640 1 --2- 172.21.15.88:0/1909827715 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5f806d260 0x7fa5f814eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.171+0000 7fa5f6d9d640 1 --2- 172.21.15.88:0/1909827715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5f815b460 0x7fa5f80afee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.172+0000 7fa5f5d9b640 1 -- 172.21.15.88:0/1909827715 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5f806d260 msgr2=0x7fa5f814eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.172+0000 7fa5f5d9b640 1 --2- 172.21.15.88:0/1909827715 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5f806d260 0x7fa5f814eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.172+0000 7fa5f5d9b640 1 -- 172.21.15.88:0/1909827715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5f815b460 msgr2=0x7fa5f80afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:06.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.172+0000 7fa5f5d9b640 1 --2- 172.21.15.88:0/1909827715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5f815b460 0x7fa5f80afee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:06.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.172+0000 7fa5f5d9b640 1 -- 172.21.15.88:0/1909827715 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5f80b0b60 con 0x7fa5f80d0fe0 2024-09-16T01:42:06.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.172+0000 7fa5f5d9b640 1 --2- 172.21.15.88:0/1909827715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5f80d0fe0 0x7fa5f80cfee0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec000c00 tx=0x7fa5ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:06.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.172+0000 7fa5e6ffd640 1 -- 172.21.15.88:0/1909827715 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec041020 con 0x7fa5f80d0fe0 2024-09-16T01:42:06.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.172+0000 7fa5e6ffd640 1 -- 172.21.15.88:0/1909827715 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5ec0306c0 con 0x7fa5f80d0fe0 2024-09-16T01:42:06.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.172+0000 7fa5e6ffd640 1 -- 172.21.15.88:0/1909827715 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec037040 con 0x7fa5f80d0fe0 2024-09-16T01:42:06.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.172+0000 7fa5fce53640 1 -- 172.21.15.88:0/1909827715 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5f814f0d0 con 0x7fa5f80d0fe0 2024-09-16T01:42:06.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.172+0000 7fa5fce53640 1 -- 172.21.15.88:0/1909827715 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5f80cf400 con 0x7fa5f80d0fe0 2024-09-16T01:42:06.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.173+0000 7fa5fce53640 1 -- 172.21.15.88:0/1909827715 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5f80b5080 con 0x7fa5f80d0fe0 2024-09-16T01:42:06.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.173+0000 7fa5e6ffd640 1 -- 172.21.15.88:0/1909827715 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa5ec044020 con 0x7fa5f80d0fe0 2024-09-16T01:42:06.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.174+0000 7fa5e6ffd640 1 --2- 172.21.15.88:0/1909827715 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa5d806c2b0 0x7fa5d806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:06.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.174+0000 7fa5e6ffd640 1 -- 172.21.15.88:0/1909827715 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7286+0+0 (secure 0 0 0) 0x7fa5ec0ac0a0 con 0x7fa5f80d0fe0 2024-09-16T01:42:06.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.174+0000 7fa5f659c640 1 --2- 172.21.15.88:0/1909827715 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa5d806c2b0 0x7fa5d806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:06.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.174+0000 7fa5f659c640 1 --2- 172.21.15.88:0/1909827715 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa5d806c2b0 0x7fa5d806e770 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fa5f814f620 tx=0x7fa5e8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:06.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.178+0000 7fa5e6ffd640 1 -- 172.21.15.88:0/1909827715 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa5ec04b030 con 0x7fa5f80d0fe0 2024-09-16T01:42:06.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.376+0000 7fa5fce53640 1 -- 172.21.15.88:0/1909827715 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa5f8159b00 con 0x7fa5f80d0fe0 2024-09-16T01:42:06.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.377+0000 7fa5e6ffd640 1 -- 172.21.15.88:0/1909827715 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) ==== 95+0+3566 (secure 0 0 0) 0x7fa5ec079c20 con 0x7fa5f80d0fe0 2024-09-16T01:42:06.378 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:06.378 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":175,"btime":"2024-09-16T01:42:05:412377+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:05.412321+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:06.378 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 175 2024-09-16T01:42:06.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.379+0000 7fa5fce53640 1 -- 172.21.15.88:0/1909827715 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa5d806c2b0 msgr2=0x7fa5d806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:06.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.379+0000 7fa5fce53640 1 --2- 172.21.15.88:0/1909827715 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa5d806c2b0 0x7fa5d806e770 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fa5f814f620 tx=0x7fa5e8009040 comp rx=0 tx=0).stop 2024-09-16T01:42:06.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.379+0000 7fa5fce53640 1 -- 172.21.15.88:0/1909827715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5f80d0fe0 msgr2=0x7fa5f80cfee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:06.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.379+0000 7fa5fce53640 1 --2- 172.21.15.88:0/1909827715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5f80d0fe0 0x7fa5f80cfee0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec000c00 tx=0x7fa5ec0046c0 comp rx=0 tx=0).stop 2024-09-16T01:42:06.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.379+0000 7fa5fce53640 1 -- 172.21.15.88:0/1909827715 shutdown_connections 2024-09-16T01:42:06.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.379+0000 7fa5fce53640 1 --2- 172.21.15.88:0/1909827715 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa5d806c2b0 0x7fa5d806e770 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:06.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.379+0000 7fa5fce53640 1 --2- 172.21.15.88:0/1909827715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5f815b460 0x7fa5f80afee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:06.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.379+0000 7fa5fce53640 1 --2- 172.21.15.88:0/1909827715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa5f80d0fe0 0x7fa5f80cfee0 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:06.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.379+0000 7fa5fce53640 1 --2- 172.21.15.88:0/1909827715 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5f806d260 0x7fa5f814eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:06.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.379+0000 7fa5fce53640 1 -- 172.21.15.88:0/1909827715 >> 172.21.15.88:0/1909827715 conn(0x7fa5f80b6ec0 msgr2=0x7fa5f8072760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:06.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.379+0000 7fa5fce53640 1 -- 172.21.15.88:0/1909827715 shutdown_connections 2024-09-16T01:42:06.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.379+0000 7fa5fce53640 1 -- 172.21.15.88:0/1909827715 wait complete. 2024-09-16T01:42:06.391 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt 200.00000001' led to DAMAGED state 2024-09-16T01:42:06.391 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt 500.00000000' 2024-09-16T01:42:06.391 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:42:06.391 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:42:06.407 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:42:06.407 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:42:06.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:06.408 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:42:06.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:42:06.466 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:42:06.466 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:42:06.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.605+0000 7f0304931640 1 Processor -- start 2024-09-16T01:42:06.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.605+0000 7f0304931640 1 -- start start 2024-09-16T01:42:06.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.606+0000 7f0304931640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f030015c710 0x7f0300073970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:06.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.606+0000 7f0304931640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0300073eb0 0x7f03000d2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:06.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.606+0000 7f0304931640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03000cdf50 con 0x7f03000b53f0 2024-09-16T01:42:06.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.606+0000 7f0304931640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f03000ce230 con 0x7f030015c710 2024-09-16T01:42:06.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.606+0000 7f0304931640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f03000ce510 con 0x7f0300073eb0 2024-09-16T01:42:06.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.606+0000 7f02fe59c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f030015c710 0x7f0300073970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:06.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.606+0000 7f02fe59c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f030015c710 0x7f0300073970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44804/0 (socket says 172.21.15.88:44804) 2024-09-16T01:42:06.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.606+0000 7f02fe59c640 1 -- 172.21.15.88:0/699668616 learned_addr learned my addr 172.21.15.88:0/699668616 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:06.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.607+0000 7f02fdd9b640 1 --2- 172.21.15.88:0/699668616 >> v2:172.21.15.154:3300/0 conn(0x7f0300073eb0 0x7f03000d2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:06.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.607+0000 7f02fe59c640 1 -- 172.21.15.88:0/699668616 >> v2:172.21.15.154:3300/0 conn(0x7f0300073eb0 msgr2=0x7f03000d2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:06.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.607+0000 7f02fe59c640 1 --2- 172.21.15.88:0/699668616 >> v2:172.21.15.154:3300/0 conn(0x7f0300073eb0 0x7f03000d2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:06.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.607+0000 7f02fe59c640 1 -- 172.21.15.88:0/699668616 >> v1:172.21.15.154:6789/0 conn(0x7f03000b53f0 legacy=0x7f03000d2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:06.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.607+0000 7f02fe59c640 1 -- 172.21.15.88:0/699668616 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03000ce7f0 con 0x7f030015c710 2024-09-16T01:42:06.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.607+0000 7f02fdd9b640 1 --2- 172.21.15.88:0/699668616 >> v2:172.21.15.154:3300/0 conn(0x7f0300073eb0 0x7f03000d2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:06.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.607+0000 7f02fe59c640 1 --2- 172.21.15.88:0/699668616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f030015c710 0x7f0300073970 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f03000d0000 tx=0x7f02f002a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=105e3ac5f350a1ad server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:06.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.608+0000 7f02fd59a640 1 -- 172.21.15.88:0/699668616 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f0036070 con 0x7f030015c710 2024-09-16T01:42:06.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.608+0000 7f02fd59a640 1 -- 172.21.15.88:0/699668616 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02f0031600 con 0x7f030015c710 2024-09-16T01:42:06.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.608+0000 7f02fd59a640 1 -- 172.21.15.88:0/699668616 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f00319d0 con 0x7f030015c710 2024-09-16T01:42:06.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.608+0000 7f0304931640 1 -- 172.21.15.88:0/699668616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f030015c710 msgr2=0x7f0300073970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:06.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.608+0000 7f0304931640 1 --2- 172.21.15.88:0/699668616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f030015c710 0x7f0300073970 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f03000d0000 tx=0x7f02f002a5d0 comp rx=0 tx=0).stop 2024-09-16T01:42:06.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.608+0000 7f0304931640 1 -- 172.21.15.88:0/699668616 shutdown_connections 2024-09-16T01:42:06.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.608+0000 7f0304931640 1 --2- 172.21.15.88:0/699668616 >> v2:172.21.15.154:3300/0 conn(0x7f0300073eb0 0x7f03000d2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:06.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.608+0000 7f0304931640 1 --2- 172.21.15.88:0/699668616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f030015c710 0x7f0300073970 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:06.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.608+0000 7f0304931640 1 -- 172.21.15.88:0/699668616 >> 172.21.15.88:0/699668616 conn(0x7f0300062a20 msgr2=0x7f0300062e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:06.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.608+0000 7f0304931640 1 -- 172.21.15.88:0/699668616 shutdown_connections 2024-09-16T01:42:06.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.608+0000 7f0304931640 1 -- 172.21.15.88:0/699668616 wait complete. 2024-09-16T01:42:06.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.610+0000 7f0304931640 1 Processor -- start 2024-09-16T01:42:06.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.610+0000 7f0304931640 1 -- start start 2024-09-16T01:42:06.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.610+0000 7f0304931640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0300073eb0 0x7f03000c9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:06.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.610+0000 7f0304931640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03000b53f0 0x7f0300146b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:06.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.610+0000 7f0304931640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f030015c710 0x7f03000cbf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:06.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.610+0000 7f0304931640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f03000ceff0 con 0x7f0300073eb0 2024-09-16T01:42:06.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.610+0000 7f0304931640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f03000cdc00 con 0x7f030015c710 2024-09-16T01:42:06.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.610+0000 7f0304931640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f03000cf170 con 0x7f03000b53f0 2024-09-16T01:42:06.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.610+0000 7f02fe59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0300073eb0 0x7f03000c9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:06.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.610+0000 7f02fe59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0300073eb0 0x7f03000c9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44820/0 (socket says 172.21.15.88:44820) 2024-09-16T01:42:06.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.610+0000 7f02fe59c640 1 -- 172.21.15.88:0/953393568 learned_addr learned my addr 172.21.15.88:0/953393568 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:06.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.611+0000 7f02fed9d640 1 --2- 172.21.15.88:0/953393568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f030015c710 0x7f03000cbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:06.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.611+0000 7f02fdd9b640 1 --2- 172.21.15.88:0/953393568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03000b53f0 0x7f0300146b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:06.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.611+0000 7f02fe59c640 1 -- 172.21.15.88:0/953393568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03000b53f0 msgr2=0x7f0300146b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:06.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.611+0000 7f02fe59c640 1 --2- 172.21.15.88:0/953393568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03000b53f0 0x7f0300146b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:06.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.611+0000 7f02fe59c640 1 -- 172.21.15.88:0/953393568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f030015c710 msgr2=0x7f03000cbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:06.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.611+0000 7f02fe59c640 1 --2- 172.21.15.88:0/953393568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f030015c710 0x7f03000cbf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:06.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.611+0000 7f02fe59c640 1 -- 172.21.15.88:0/953393568 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f030006cd20 con 0x7f0300073eb0 2024-09-16T01:42:06.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.611+0000 7f02fe59c640 1 --2- 172.21.15.88:0/953393568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0300073eb0 0x7f03000c9860 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f02f0000f80 tx=0x7f02f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:06.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.611+0000 7f02f6ffd640 1 -- 172.21.15.88:0/953393568 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f0036050 con 0x7f0300073eb0 2024-09-16T01:42:06.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.611+0000 7f02f6ffd640 1 -- 172.21.15.88:0/953393568 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02f0042070 con 0x7f0300073eb0 2024-09-16T01:42:06.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.611+0000 7f0304931640 1 -- 172.21.15.88:0/953393568 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f030006cfe0 con 0x7f0300073eb0 2024-09-16T01:42:06.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.611+0000 7f02f6ffd640 1 -- 172.21.15.88:0/953393568 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f0009510 con 0x7f0300073eb0 2024-09-16T01:42:06.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.611+0000 7f0304931640 1 -- 172.21.15.88:0/953393568 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f030006d5a0 con 0x7f0300073eb0 2024-09-16T01:42:06.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.612+0000 7f0304931640 1 -- 172.21.15.88:0/953393568 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0300073970 con 0x7f0300073eb0 2024-09-16T01:42:06.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.613+0000 7f02f6ffd640 1 -- 172.21.15.88:0/953393568 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f02f0007c70 con 0x7f0300073eb0 2024-09-16T01:42:06.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.613+0000 7f02f6ffd640 1 --2- 172.21.15.88:0/953393568 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f02e006c2b0 0x7f02e006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:06.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.613+0000 7f02f6ffd640 1 -- 172.21.15.88:0/953393568 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7286+0+0 (secure 0 0 0) 0x7f02f00ac9d0 con 0x7f0300073eb0 2024-09-16T01:42:06.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.613+0000 7f02fdd9b640 1 --2- 172.21.15.88:0/953393568 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f02e006c2b0 0x7f02e006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:06.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.614+0000 7f02fdd9b640 1 --2- 172.21.15.88:0/953393568 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f02e006c2b0 0x7f02e006e770 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f03001f7ba0 tx=0x7f02e8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:06.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.617+0000 7f02f6ffd640 1 -- 172.21.15.88:0/953393568 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f02f007ab10 con 0x7f0300073eb0 2024-09-16T01:42:06.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:06.821+0000 7f0304931640 1 -- 172.21.15.88:0/953393568 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f0300073970 con 0x7f0300073eb0 2024-09-16T01:42:07.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.418+0000 7f02f6ffd640 1 -- 172.21.15.88:0/953393568 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v177) ==== 161+0+0 (secure 0 0 0) 0x7f02f007a410 con 0x7f0300073eb0 2024-09-16T01:42:07.420 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:42:07.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.420+0000 7f0304931640 1 -- 172.21.15.88:0/953393568 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f02e006c2b0 msgr2=0x7f02e006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:07.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.420+0000 7f0304931640 1 --2- 172.21.15.88:0/953393568 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f02e006c2b0 0x7f02e006e770 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f03001f7ba0 tx=0x7f02e8008040 comp rx=0 tx=0).stop 2024-09-16T01:42:07.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.420+0000 7f0304931640 1 -- 172.21.15.88:0/953393568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0300073eb0 msgr2=0x7f03000c9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:07.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.420+0000 7f0304931640 1 --2- 172.21.15.88:0/953393568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0300073eb0 0x7f03000c9860 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f02f0000f80 tx=0x7f02f0004a40 comp rx=0 tx=0).stop 2024-09-16T01:42:07.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.420+0000 7f0304931640 1 -- 172.21.15.88:0/953393568 shutdown_connections 2024-09-16T01:42:07.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.420+0000 7f0304931640 1 --2- 172.21.15.88:0/953393568 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f02e006c2b0 0x7f02e006e770 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:07.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.420+0000 7f0304931640 1 --2- 172.21.15.88:0/953393568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f030015c710 0x7f03000cbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:07.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.420+0000 7f0304931640 1 --2- 172.21.15.88:0/953393568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f03000b53f0 0x7f0300146b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:07.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.420+0000 7f0304931640 1 --2- 172.21.15.88:0/953393568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0300073eb0 0x7f03000c9860 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:07.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.420+0000 7f0304931640 1 -- 172.21.15.88:0/953393568 >> 172.21.15.88:0/953393568 conn(0x7f0300062a20 msgr2=0x7f03000d1140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:07.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.420+0000 7f0304931640 1 -- 172.21.15.88:0/953393568 shutdown_connections 2024-09-16T01:42:07.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.420+0000 7f0304931640 1 -- 172.21.15.88:0/953393568 wait complete. 2024-09-16T01:42:07.435 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:42:07.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.533+0000 7ff1c8ad2640 1 Processor -- start 2024-09-16T01:42:07.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.533+0000 7ff1c8ad2640 1 -- start start 2024-09-16T01:42:07.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.534+0000 7ff1c8ad2640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff1c415e7a0 0x7ff1c406caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:07.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.534+0000 7ff1c8ad2640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1c40cf1f0 0x7ff1c40b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:07.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.534+0000 7ff1c8ad2640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff1c40b6720 0x7ff1c41623c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:07.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.534+0000 7ff1c8ad2640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff1c40d2ca0 con 0x7ff1c415e7a0 2024-09-16T01:42:07.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.534+0000 7ff1c8ad2640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff1c40cf860 con 0x7ff1c40b6720 2024-09-16T01:42:07.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.534+0000 7ff1c8ad2640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff1c40cfb40 con 0x7ff1c40cf1f0 2024-09-16T01:42:07.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.534+0000 7ff1c259c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff1c415e7a0 0x7ff1c406caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:07.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.534+0000 7ff1c259c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff1c415e7a0 0x7ff1c406caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44822/0 (socket says 172.21.15.88:44822) 2024-09-16T01:42:07.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.534+0000 7ff1c259c640 1 -- 172.21.15.88:0/1212939183 learned_addr learned my addr 172.21.15.88:0/1212939183 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:07.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.534+0000 7ff1b9d9b640 1 --2- 172.21.15.88:0/1212939183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1c40cf1f0 0x7ff1c40b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:07.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.534+0000 7ff1c2d9d640 1 --2- 172.21.15.88:0/1212939183 >> v2:172.21.15.154:3300/0 conn(0x7ff1c40b6720 0x7ff1c41623c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:07.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.535+0000 7ff1c259c640 1 -- 172.21.15.88:0/1212939183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1c40cf1f0 msgr2=0x7ff1c40b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:07.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.535+0000 7ff1c259c640 1 --2- 172.21.15.88:0/1212939183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1c40cf1f0 0x7ff1c40b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:07.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.535+0000 7ff1c259c640 1 -- 172.21.15.88:0/1212939183 >> v2:172.21.15.154:3300/0 conn(0x7ff1c40b6720 msgr2=0x7ff1c41623c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:07.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.535+0000 7ff1c259c640 1 --2- 172.21.15.88:0/1212939183 >> v2:172.21.15.154:3300/0 conn(0x7ff1c40b6720 0x7ff1c41623c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:07.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.535+0000 7ff1c259c640 1 -- 172.21.15.88:0/1212939183 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1c40cfe20 con 0x7ff1c415e7a0 2024-09-16T01:42:07.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.535+0000 7ff1c259c640 1 --2- 172.21.15.88:0/1212939183 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c415e7a0 0x7ff1c406caa0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4163460 tx=0x7ff1b402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f117de90e7ce525a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:07.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.535+0000 7ff1c1d9b640 1 -- 172.21.15.88:0/1212939183 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4037070 con 0x7ff1c415e7a0 2024-09-16T01:42:07.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.535+0000 7ff1c1d9b640 1 -- 172.21.15.88:0/1212939183 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1b4032440 con 0x7ff1c415e7a0 2024-09-16T01:42:07.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.535+0000 7ff1c1d9b640 1 -- 172.21.15.88:0/1212939183 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4032810 con 0x7ff1c415e7a0 2024-09-16T01:42:07.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.536+0000 7ff1c8ad2640 1 -- 172.21.15.88:0/1212939183 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c415e7a0 msgr2=0x7ff1c406caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:07.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.536+0000 7ff1c8ad2640 1 --2- 172.21.15.88:0/1212939183 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c415e7a0 0x7ff1c406caa0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4163460 tx=0x7ff1b402a140 comp rx=0 tx=0).stop 2024-09-16T01:42:07.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.536+0000 7ff1c8ad2640 1 -- 172.21.15.88:0/1212939183 shutdown_connections 2024-09-16T01:42:07.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.536+0000 7ff1c8ad2640 1 --2- 172.21.15.88:0/1212939183 >> v2:172.21.15.154:3300/0 conn(0x7ff1c40b6720 0x7ff1c41623c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:07.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.536+0000 7ff1c8ad2640 1 --2- 172.21.15.88:0/1212939183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1c40cf1f0 0x7ff1c40b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:07.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.536+0000 7ff1c8ad2640 1 --2- 172.21.15.88:0/1212939183 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c415e7a0 0x7ff1c406caa0 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:07.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.536+0000 7ff1c8ad2640 1 -- 172.21.15.88:0/1212939183 >> 172.21.15.88:0/1212939183 conn(0x7ff1c40d11e0 msgr2=0x7ff1c40d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:07.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.536+0000 7ff1c8ad2640 1 -- 172.21.15.88:0/1212939183 shutdown_connections 2024-09-16T01:42:07.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.536+0000 7ff1c8ad2640 1 -- 172.21.15.88:0/1212939183 wait complete. 2024-09-16T01:42:07.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.538+0000 7ff1c8ad2640 1 Processor -- start 2024-09-16T01:42:07.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.538+0000 7ff1c8ad2640 1 -- start start 2024-09-16T01:42:07.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.538+0000 7ff1c8ad2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1c40b6720 0x7ff1c40c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:07.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.538+0000 7ff1c8ad2640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1c40cf1f0 0x7ff1c4161060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:07.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.538+0000 7ff1c8ad2640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c415e7a0 0x7ff1c415ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:07.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.538+0000 7ff1c8ad2640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c406cfe0 con 0x7ff1c415e7a0 2024-09-16T01:42:07.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.538+0000 7ff1c8ad2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c40b7a00 con 0x7ff1c40b6720 2024-09-16T01:42:07.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.538+0000 7ff1c8ad2640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff1c415eb80 con 0x7ff1c40cf1f0 2024-09-16T01:42:07.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.538+0000 7ff1c2d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c415e7a0 0x7ff1c415ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:07.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.538+0000 7ff1c2d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c415e7a0 0x7ff1c415ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44834/0 (socket says 172.21.15.88:44834) 2024-09-16T01:42:07.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.538+0000 7ff1c2d9d640 1 -- 172.21.15.88:0/3425710715 learned_addr learned my addr 172.21.15.88:0/3425710715 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:07.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.538+0000 7ff1b9d9b640 1 --2- 172.21.15.88:0/3425710715 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1c40cf1f0 0x7ff1c4161060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:07.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.539+0000 7ff1c259c640 1 --2- 172.21.15.88:0/3425710715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1c40b6720 0x7ff1c40c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:07.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.539+0000 7ff1c2d9d640 1 -- 172.21.15.88:0/3425710715 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1c40cf1f0 msgr2=0x7ff1c4161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:07.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.539+0000 7ff1c2d9d640 1 --2- 172.21.15.88:0/3425710715 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1c40cf1f0 0x7ff1c4161060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:07.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.539+0000 7ff1c2d9d640 1 -- 172.21.15.88:0/3425710715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1c40b6720 msgr2=0x7ff1c40c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:07.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.539+0000 7ff1c2d9d640 1 --2- 172.21.15.88:0/3425710715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1c40b6720 0x7ff1c40c7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:07.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.539+0000 7ff1c2d9d640 1 -- 172.21.15.88:0/3425710715 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1c40c9870 con 0x7ff1c415e7a0 2024-09-16T01:42:07.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.539+0000 7ff1c2d9d640 1 --2- 172.21.15.88:0/3425710715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c415e7a0 0x7ff1c415ee40 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc007d20 tx=0x7ff1bc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:07.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.539+0000 7ff1bb7fe640 1 -- 172.21.15.88:0/3425710715 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc014420 con 0x7ff1c415e7a0 2024-09-16T01:42:07.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.539+0000 7ff1bb7fe640 1 -- 172.21.15.88:0/3425710715 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1bc019070 con 0x7ff1c415e7a0 2024-09-16T01:42:07.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.539+0000 7ff1c8ad2640 1 -- 172.21.15.88:0/3425710715 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1c40c8ad0 con 0x7ff1c415e7a0 2024-09-16T01:42:07.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.539+0000 7ff1bb7fe640 1 -- 172.21.15.88:0/3425710715 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc014820 con 0x7ff1c415e7a0 2024-09-16T01:42:07.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.539+0000 7ff1c8ad2640 1 -- 172.21.15.88:0/3425710715 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1c40c9090 con 0x7ff1c415e7a0 2024-09-16T01:42:07.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.540+0000 7ff1c8ad2640 1 -- 172.21.15.88:0/3425710715 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1c40b61e0 con 0x7ff1c415e7a0 2024-09-16T01:42:07.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.541+0000 7ff1bb7fe640 1 -- 172.21.15.88:0/3425710715 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff1bc0149c0 con 0x7ff1c415e7a0 2024-09-16T01:42:07.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.541+0000 7ff1bb7fe640 1 --2- 172.21.15.88:0/3425710715 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff1a406c2b0 0x7ff1a406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:07.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.541+0000 7ff1c259c640 1 --2- 172.21.15.88:0/3425710715 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff1a406c2b0 0x7ff1a406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:07.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.541+0000 7ff1bb7fe640 1 -- 172.21.15.88:0/3425710715 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7286+0+0 (secure 0 0 0) 0x7ff1bc08c100 con 0x7ff1c415e7a0 2024-09-16T01:42:07.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.542+0000 7ff1c259c640 1 --2- 172.21.15.88:0/3425710715 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff1a406c2b0 0x7ff1a406e770 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4034000 tx=0x7ff1b402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:07.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.545+0000 7ff1bb7fe640 1 -- 172.21.15.88:0/3425710715 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff1bc059bf0 con 0x7ff1c415e7a0 2024-09-16T01:42:07.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:07.777+0000 7ff1c8ad2640 1 -- 172.21.15.88:0/3425710715 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7ff1c40d2a80 con 0x7ff1c415e7a0 2024-09-16T01:42:08.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.425+0000 7ff1bb7fe640 1 -- 172.21.15.88:0/3425710715 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v178) ==== 101+0+0 (secure 0 0 0) 0x7ff1bc05def0 con 0x7ff1c415e7a0 2024-09-16T01:42:08.428 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:42:08.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.428+0000 7ff1c8ad2640 1 -- 172.21.15.88:0/3425710715 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff1a406c2b0 msgr2=0x7ff1a406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:08.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.428+0000 7ff1c8ad2640 1 --2- 172.21.15.88:0/3425710715 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff1a406c2b0 0x7ff1a406e770 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4034000 tx=0x7ff1b402e000 comp rx=0 tx=0).stop 2024-09-16T01:42:08.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.428+0000 7ff1c8ad2640 1 -- 172.21.15.88:0/3425710715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c415e7a0 msgr2=0x7ff1c415ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:08.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.428+0000 7ff1c8ad2640 1 --2- 172.21.15.88:0/3425710715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c415e7a0 0x7ff1c415ee40 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc007d20 tx=0x7ff1bc00e4b0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.428+0000 7ff1c8ad2640 1 -- 172.21.15.88:0/3425710715 shutdown_connections 2024-09-16T01:42:08.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.428+0000 7ff1c8ad2640 1 --2- 172.21.15.88:0/3425710715 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff1a406c2b0 0x7ff1a406e770 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.428+0000 7ff1c8ad2640 1 --2- 172.21.15.88:0/3425710715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c415e7a0 0x7ff1c415ee40 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.428+0000 7ff1c8ad2640 1 --2- 172.21.15.88:0/3425710715 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1c40cf1f0 0x7ff1c4161060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.428+0000 7ff1c8ad2640 1 --2- 172.21.15.88:0/3425710715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1c40b6720 0x7ff1c40c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.428+0000 7ff1c8ad2640 1 -- 172.21.15.88:0/3425710715 >> 172.21.15.88:0/3425710715 conn(0x7ff1c40d11e0 msgr2=0x7ff1c406c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:08.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.428+0000 7ff1c8ad2640 1 -- 172.21.15.88:0/3425710715 shutdown_connections 2024-09-16T01:42:08.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.428+0000 7ff1c8ad2640 1 -- 172.21.15.88:0/3425710715 wait complete. 2024-09-16T01:42:08.443 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:42:08.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.484+0000 7f63b1720f40 1 Processor -- start 2024-09-16T01:42:08.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.484+0000 7f63b1720f40 1 -- start start 2024-09-16T01:42:08.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.485+0000 7f63b1720f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3e070f940 0x55b3e07e2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.485+0000 7f63b1720f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e070fd20 0x55b3e0752dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.485+0000 7f63b1720f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3e07e3f30 0x55b3e0753300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.485+0000 7f63b1720f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b3e07102c0 con 0x55b3e070f940 2024-09-16T01:42:08.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.485+0000 7f63b1720f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b3e0660510 con 0x55b3e070fd20 2024-09-16T01:42:08.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.485+0000 7f63b1720f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55b3e070dc80 con 0x55b3e07e3f30 2024-09-16T01:42:08.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.485+0000 7f63ae7af640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3e070f940 0x55b3e07e2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.485+0000 7f63ae7af640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3e070f940 0x55b3e07e2b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44850/0 (socket says 172.21.15.88:44850) 2024-09-16T01:42:08.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.485+0000 7f63ae7af640 1 -- 172.21.15.88:0/4236822500 learned_addr learned my addr 172.21.15.88:0/4236822500 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:08.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.485+0000 7f63a5fae640 1 --2- 172.21.15.88:0/4236822500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e070fd20 0x55b3e0752dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.485+0000 7f63aefb0640 1 --2- 172.21.15.88:0/4236822500 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3e07e3f30 0x55b3e0753300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.486+0000 7f63ae7af640 1 -- 172.21.15.88:0/4236822500 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3e07e3f30 msgr2=0x55b3e0753300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:08.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.486+0000 7f63ae7af640 1 --2- 172.21.15.88:0/4236822500 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3e07e3f30 0x55b3e0753300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.486+0000 7f63ae7af640 1 -- 172.21.15.88:0/4236822500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e070fd20 msgr2=0x55b3e0752dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:08.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.486+0000 7f63ae7af640 1 --2- 172.21.15.88:0/4236822500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e070fd20 0x55b3e0752dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.486+0000 7f63ae7af640 1 -- 172.21.15.88:0/4236822500 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3e07539c0 con 0x55b3e070f940 2024-09-16T01:42:08.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.486+0000 7f63ae7af640 1 --2- 172.21.15.88:0/4236822500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3e070f940 0x55b3e07e2b40 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f63a000b2c0 tx=0x7f63a000b780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ee4216e6b0df940 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:08.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.486+0000 7f63ad7ad640 1 -- 172.21.15.88:0/4236822500 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a0008070 con 0x55b3e070f940 2024-09-16T01:42:08.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.486+0000 7f63ad7ad640 1 -- 172.21.15.88:0/4236822500 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63a000d500 con 0x55b3e070f940 2024-09-16T01:42:08.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.486+0000 7f63ad7ad640 1 -- 172.21.15.88:0/4236822500 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a000d8d0 con 0x55b3e070f940 2024-09-16T01:42:08.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.486+0000 7f63b1720f40 1 -- 172.21.15.88:0/4236822500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3e070f940 msgr2=0x55b3e07e2b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:08.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.486+0000 7f63b1720f40 1 --2- 172.21.15.88:0/4236822500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3e070f940 0x55b3e07e2b40 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f63a000b2c0 tx=0x7f63a000b780 comp rx=0 tx=0).stop 2024-09-16T01:42:08.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.486+0000 7f63b1720f40 1 -- 172.21.15.88:0/4236822500 shutdown_connections 2024-09-16T01:42:08.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.486+0000 7f63b1720f40 1 --2- 172.21.15.88:0/4236822500 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3e07e3f30 0x55b3e0753300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.486+0000 7f63b1720f40 1 --2- 172.21.15.88:0/4236822500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e070fd20 0x55b3e0752dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.486+0000 7f63b1720f40 1 --2- 172.21.15.88:0/4236822500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3e070f940 0x55b3e07e2b40 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.486+0000 7f63b1720f40 1 -- 172.21.15.88:0/4236822500 >> 172.21.15.88:0/4236822500 conn(0x55b3e0759750 msgr2=0x55b3e06fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:08.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.486+0000 7f63b1720f40 1 -- 172.21.15.88:0/4236822500 shutdown_connections 2024-09-16T01:42:08.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.486+0000 7f63b1720f40 1 -- 172.21.15.88:0/4236822500 wait complete. 2024-09-16T01:42:08.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.487+0000 7f63b1720f40 1 Processor -- start 2024-09-16T01:42:08.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.487+0000 7f63b1720f40 1 -- start start 2024-09-16T01:42:08.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.488+0000 7f63b1720f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3e070f940 0x55b3e07dcaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.488+0000 7f63b1720f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e070fd20 0x55b3e07dfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.488+0000 7f63b1720f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3e07e3f30 0x55b3e07dd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.488+0000 7f63b1720f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b3e0754910 con 0x55b3e070f940 2024-09-16T01:42:08.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.488+0000 7f63b1720f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b3e07e26b0 con 0x55b3e070fd20 2024-09-16T01:42:08.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.488+0000 7f63b1720f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55b3e0754bf0 con 0x55b3e07e3f30 2024-09-16T01:42:08.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.488+0000 7f63ae7af640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3e070f940 0x55b3e07dcaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.488+0000 7f63ae7af640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3e070f940 0x55b3e07dcaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44860/0 (socket says 172.21.15.88:44860) 2024-09-16T01:42:08.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.488+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 learned_addr learned my addr 172.21.15.88:0/3997726222 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:08.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.488+0000 7f63a5fae640 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e070fd20 0x55b3e07dfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.488+0000 7f63aefb0640 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3e07e3f30 0x55b3e07dd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.488+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3e07e3f30 msgr2=0x55b3e07dd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:08.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.488+0000 7f63ae7af640 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3e07e3f30 0x55b3e07dd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.488+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e070fd20 msgr2=0x55b3e07dfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:08.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.488+0000 7f63ae7af640 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e070fd20 0x55b3e07dfdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.488+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3e07c6940 con 0x55b3e070f940 2024-09-16T01:42:08.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.488+0000 7f63ae7af640 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3e070f940 0x55b3e07dcaf0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f63a0001ed0 tx=0x7f63a0000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:08.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.488+0000 7f63a7fff640 1 -- 172.21.15.88:0/3997726222 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a0021030 con 0x55b3e070f940 2024-09-16T01:42:08.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.488+0000 7f63a7fff640 1 -- 172.21.15.88:0/3997726222 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63a001c440 con 0x55b3e070f940 2024-09-16T01:42:08.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.488+0000 7f63a7fff640 1 -- 172.21.15.88:0/3997726222 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a001c6b0 con 0x55b3e070f940 2024-09-16T01:42:08.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.488+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b3e07c6c20 con 0x55b3e070f940 2024-09-16T01:42:08.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.488+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3e07c71e0 con 0x55b3e070f940 2024-09-16T01:42:08.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.490+0000 7f63a7fff640 1 -- 172.21.15.88:0/3997726222 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f63a0024070 con 0x55b3e070f940 2024-09-16T01:42:08.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.490+0000 7f63a7fff640 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f639006acc0 0x7f639006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.490+0000 7f63a7fff640 1 -- 172.21.15.88:0/3997726222 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7286+0+0 (secure 0 0 0) 0x7f63a008b510 con 0x55b3e070f940 2024-09-16T01:42:08.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.490+0000 7f63a5fae640 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f639006acc0 0x7f639006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.498 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:42:08.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.491+0000 7f63b1720f40 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55b3e07c79e0 0x55b3e07c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.491+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e07c8300 con 0x55b3e07c79e0 2024-09-16T01:42:08.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.491+0000 7f63aefb0640 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55b3e07c79e0 0x55b3e07c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.491+0000 7f63a5fae640 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f639006acc0 0x7f639006d180 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f6398057000 tx=0x7f6398052000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:08.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.491+0000 7f63aefb0640 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55b3e07c79e0 0x55b3e07c7dc0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:08.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.493+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v105'536 uv510 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.493+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e105) -- 0x55b3e07c86a0 con 0x55b3e07c79e0 2024-09-16T01:42:08.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.494+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v105'537 uv537 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.494+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e07c8ad0 con 0x55b3e07c79e0 2024-09-16T01:42:08.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.495+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v105'538 uv538 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.502 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:42:08.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.496+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0889ba0 con 0x55b3e07c79e0 2024-09-16T01:42:08.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.497+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v105'539 uv539 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.497+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0889f40 con 0x55b3e07c79e0 2024-09-16T01:42:08.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.498+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v105'540 uv540 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.498+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0888b90 con 0x55b3e07c79e0 2024-09-16T01:42:08.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.499+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v105'541 uv541 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.499+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0888f30 con 0x55b3e07c79e0 2024-09-16T01:42:08.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.501+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v105'542 uv516 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.501+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e105) -- 0x55b3e08892d0 con 0x55b3e07c79e0 2024-09-16T01:42:08.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.502+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v105'543 uv543 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.502+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e08892d0 con 0x55b3e07c79e0 2024-09-16T01:42:08.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.503+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v105'544 uv544 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.506 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:42:08.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.503+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e08892d0 con 0x55b3e07c79e0 2024-09-16T01:42:08.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.504+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v105'545 uv545 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.504+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e08892d0 con 0x55b3e07c79e0 2024-09-16T01:42:08.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.505+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v105'546 uv546 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.505+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e08892d0 con 0x55b3e07c79e0 2024-09-16T01:42:08.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.506+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v105'547 uv547 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.506+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e08892d0 con 0x55b3e07c79e0 2024-09-16T01:42:08.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.507+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v105'548 uv523 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.507+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e105) -- 0x55b3e08892d0 con 0x55b3e07c79e0 2024-09-16T01:42:08.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.509+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v105'549 uv549 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.509+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e08892d0 con 0x55b3e07c79e0 2024-09-16T01:42:08.512 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:42:08.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.510+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v105'550 uv550 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.510+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e08892d0 con 0x55b3e07c79e0 2024-09-16T01:42:08.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.511+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v105'551 uv551 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.511+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e08892d0 con 0x55b3e07c79e0 2024-09-16T01:42:08.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.512+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v105'552 uv552 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.512+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e08892d0 con 0x55b3e07c79e0 2024-09-16T01:42:08.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.514+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v105'553 uv553 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.514+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088a2e0 con 0x55b3e07c79e0 2024-09-16T01:42:08.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.515+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v105'554 uv554 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.515+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088a2e0 con 0x55b3e07c79e0 2024-09-16T01:42:08.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.516+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v105'555 uv529 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.516+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088a2e0 con 0x55b3e07c79e0 2024-09-16T01:42:08.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.517+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v105'556 uv556 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.517+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088a2e0 con 0x55b3e07c79e0 2024-09-16T01:42:08.520 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:42:08.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.518+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v105'557 uv557 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.518+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088b2f0 con 0x55b3e07c79e0 2024-09-16T01:42:08.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.520+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v105'558 uv558 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.520+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088b2f0 con 0x55b3e07c79e0 2024-09-16T01:42:08.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.521+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v105'559 uv559 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.521+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088a2e0 con 0x55b3e07c79e0 2024-09-16T01:42:08.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.522+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v105'560 uv560 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.522+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088a2e0 con 0x55b3e07c79e0 2024-09-16T01:42:08.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.524+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v105'561 uv535 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.524+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088a2e0 con 0x55b3e07c79e0 2024-09-16T01:42:08.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.525+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v105'562 uv562 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.525+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088a2e0 con 0x55b3e07c79e0 2024-09-16T01:42:08.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.526+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v105'563 uv563 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.528 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:42:08.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.526+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088b2f0 con 0x55b3e07c79e0 2024-09-16T01:42:08.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.527+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v105'564 uv564 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.527+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088b2f0 con 0x55b3e07c79e0 2024-09-16T01:42:08.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.528+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v105'565 uv565 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.529+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088a2e0 con 0x55b3e07c79e0 2024-09-16T01:42:08.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.530+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v105'566 uv566 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.530+0000 7f63b1720f40 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55b3e088a7f0 0x55b3e088cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.530+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088d120 con 0x55b3e088a7f0 2024-09-16T01:42:08.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.530+0000 7f63ae7af640 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55b3e088a7f0 0x55b3e088cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.530+0000 7f63ae7af640 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55b3e088a7f0 0x55b3e088cbe0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:08.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.532+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v105'397 uv379 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e088a7f0 2024-09-16T01:42:08.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.532+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088d4c0 con 0x55b3e088a7f0 2024-09-16T01:42:08.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.533+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v105'398 uv398 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e088a7f0 2024-09-16T01:42:08.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.533+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088d860 con 0x55b3e088a7f0 2024-09-16T01:42:08.536 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:42:08.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.534+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v105'399 uv399 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e088a7f0 2024-09-16T01:42:08.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.534+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088e0b0 con 0x55b3e088a7f0 2024-09-16T01:42:08.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.536+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v105'400 uv400 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e088a7f0 2024-09-16T01:42:08.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.536+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088e450 con 0x55b3e088a7f0 2024-09-16T01:42:08.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.537+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v105'401 uv401 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e088a7f0 2024-09-16T01:42:08.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.537+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088e7f0 con 0x55b3e088a7f0 2024-09-16T01:42:08.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.538+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v105'402 uv384 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e088a7f0 2024-09-16T01:42:08.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.538+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088eb90 con 0x55b3e088a7f0 2024-09-16T01:42:08.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.539+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v105'403 uv403 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e088a7f0 2024-09-16T01:42:08.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.539+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088ef30 con 0x55b3e088a7f0 2024-09-16T01:42:08.542 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:42:08.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.540+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v105'404 uv404 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e088a7f0 2024-09-16T01:42:08.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.540+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088ef30 con 0x55b3e088a7f0 2024-09-16T01:42:08.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.541+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v105'405 uv405 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e088a7f0 2024-09-16T01:42:08.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.541+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088ef30 con 0x55b3e088a7f0 2024-09-16T01:42:08.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.542+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v105'406 uv406 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e088a7f0 2024-09-16T01:42:08.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.542+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088ef30 con 0x55b3e088a7f0 2024-09-16T01:42:08.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.543+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v105'407 uv396 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e088a7f0 2024-09-16T01:42:08.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.543+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088ef30 con 0x55b3e088a7f0 2024-09-16T01:42:08.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.544+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v105'408 uv408 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e088a7f0 2024-09-16T01:42:08.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.544+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088ef30 con 0x55b3e088a7f0 2024-09-16T01:42:08.547 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:42:08.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.545+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v105'409 uv409 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e088a7f0 2024-09-16T01:42:08.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.545+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e08933b0 con 0x55b3e088a7f0 2024-09-16T01:42:08.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.546+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v105'410 uv410 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e088a7f0 2024-09-16T01:42:08.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.546+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e08933b0 con 0x55b3e088a7f0 2024-09-16T01:42:08.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.547+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v105'411 uv411 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e088a7f0 2024-09-16T01:42:08.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.547+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088ef30 con 0x55b3e088a7f0 2024-09-16T01:42:08.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.549+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v105'412 uv395 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e088a7f0 2024-09-16T01:42:08.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.549+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088ef30 con 0x55b3e088a7f0 2024-09-16T01:42:08.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.549+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v105'413 uv413 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e088a7f0 2024-09-16T01:42:08.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.550+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088ef30 con 0x55b3e088a7f0 2024-09-16T01:42:08.552 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:42:08.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.550+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v105'414 uv414 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e088a7f0 2024-09-16T01:42:08.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.551+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088ff40 con 0x55b3e088a7f0 2024-09-16T01:42:08.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.552+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v105'415 uv415 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e088a7f0 2024-09-16T01:42:08.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.552+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088ff40 con 0x55b3e088a7f0 2024-09-16T01:42:08.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.552+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v105'416 uv416 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e088a7f0 2024-09-16T01:42:08.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.552+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088ef30 con 0x55b3e088a7f0 2024-09-16T01:42:08.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.553+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v105'417 uv417 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e088a7f0 2024-09-16T01:42:08.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.554+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088ef30 con 0x55b3e07c79e0 2024-09-16T01:42:08.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.554+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v105'105 uv104 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.554+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088ef30 con 0x55b3e07c79e0 2024-09-16T01:42:08.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.556+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v105'106 uv106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.556+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088ef30 con 0x55b3e07c79e0 2024-09-16T01:42:08.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.556+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v105'107 uv107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.559 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:42:08.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.556+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088ff40 con 0x55b3e07c79e0 2024-09-16T01:42:08.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.557+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v105'108 uv108 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.557+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088ff40 con 0x55b3e07c79e0 2024-09-16T01:42:08.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.558+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v105'109 uv109 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.558+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088ef30 con 0x55b3e07c79e0 2024-09-16T01:42:08.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.559+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v105'110 uv110 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.559+0000 7f63b1720f40 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55b3e088f5b0 0x55b3e08919a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.559+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0891fc0 con 0x55b3e088f5b0 2024-09-16T01:42:08.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.559+0000 7f63a5fae640 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55b3e088f5b0 0x55b3e08919a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.560+0000 7f63a5fae640 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55b3e088f5b0 0x55b3e08919a0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:08.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.561+0000 7f63a5fae640 1 -- 172.21.15.88:0/3997726222 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v105'81 uv80 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f6398054ca0 con 0x55b3e088f5b0 2024-09-16T01:42:08.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.561+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e105) -- 0x55b3e08924d0 con 0x55b3e088f5b0 2024-09-16T01:42:08.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.562+0000 7f63a5fae640 1 -- 172.21.15.88:0/3997726222 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v105'82 uv82 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6398054ca0 con 0x55b3e088f5b0 2024-09-16T01:42:08.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.562+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e08929d0 con 0x55b3e088f5b0 2024-09-16T01:42:08.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.562+0000 7f63a5fae640 1 -- 172.21.15.88:0/3997726222 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v105'83 uv83 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6398054ca0 con 0x55b3e088f5b0 2024-09-16T01:42:08.565 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:42:08.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.563+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e088dce0 con 0x55b3e088f5b0 2024-09-16T01:42:08.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.563+0000 7f63a5fae640 1 -- 172.21.15.88:0/3997726222 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v105'84 uv84 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6398054ca0 con 0x55b3e088f5b0 2024-09-16T01:42:08.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.563+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0893300 con 0x55b3e07c79e0 2024-09-16T01:42:08.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.565+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v105'105 uv104 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.565+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0893300 con 0x55b3e07c79e0 2024-09-16T01:42:08.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.566+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v105'106 uv106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.566+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0893300 con 0x55b3e07c79e0 2024-09-16T01:42:08.569 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:42:08.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.567+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v105'107 uv107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.567+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0894310 con 0x55b3e07c79e0 2024-09-16T01:42:08.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.569+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v105'108 uv108 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.569+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0894310 con 0x55b3e07c79e0 2024-09-16T01:42:08.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.570+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v105'109 uv109 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.570+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0893300 con 0x55b3e07c79e0 2024-09-16T01:42:08.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.572+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v105'110 uv110 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.572+0000 7f63b1720f40 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b3e0893980 0x55b3e0893d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.572+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0896780 con 0x55b3e0893980 2024-09-16T01:42:08.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.572+0000 7f63aefb0640 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b3e0893980 0x55b3e0893d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.572+0000 7f63aefb0640 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b3e0893980 0x55b3e0893d80 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:08.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.573+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v105'193 uv187 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.573+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0896780 con 0x55b3e0893980 2024-09-16T01:42:08.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.574+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v105'194 uv194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.574+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0896780 con 0x55b3e0893980 2024-09-16T01:42:08.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.575+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v105'195 uv195 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.578 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:42:08.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.575+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0897790 con 0x55b3e0893980 2024-09-16T01:42:08.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.576+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v105'196 uv196 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.576+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0897790 con 0x55b3e0893980 2024-09-16T01:42:08.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.577+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v105'197 uv197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.577+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0896780 con 0x55b3e0893980 2024-09-16T01:42:08.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.578+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v105'198 uv198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.578+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0892e60 con 0x55b3e0893980 2024-09-16T01:42:08.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.579+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v105'199 uv192 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.579+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0892e60 con 0x55b3e0893980 2024-09-16T01:42:08.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.580+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v105'200 uv200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.580+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0892de0 con 0x55b3e0893980 2024-09-16T01:42:08.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.581+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v105'201 uv201 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.583 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:42:08.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.581+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0896780 con 0x55b3e0893980 2024-09-16T01:42:08.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.583+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v105'202 uv202 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.583+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0896780 con 0x55b3e0893980 2024-09-16T01:42:08.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.584+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v105'203 uv203 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.584+0000 7f63b1720f40 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55b3e08970d0 0x55b3e08994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.584+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0899a00 con 0x55b3e08970d0 2024-09-16T01:42:08.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.584+0000 7f63ae7af640 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55b3e08970d0 0x55b3e08994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.585+0000 7f63ae7af640 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55b3e08970d0 0x55b3e08994c0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:08.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.586+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v105'86 uv85 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e08970d0 2024-09-16T01:42:08.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.586+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0899ac0 con 0x55b3e08970d0 2024-09-16T01:42:08.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.587+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v105'87 uv87 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e08970d0 2024-09-16T01:42:08.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.587+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0899ee0 con 0x55b3e08970d0 2024-09-16T01:42:08.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.588+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v105'88 uv88 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e08970d0 2024-09-16T01:42:08.590 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:42:08.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.588+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0899ee0 con 0x55b3e08970d0 2024-09-16T01:42:08.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.590+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v105'89 uv89 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e08970d0 2024-09-16T01:42:08.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.590+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0899ee0 con 0x55b3e08970d0 2024-09-16T01:42:08.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.591+0000 7f63ae7af640 1 -- 172.21.15.88:0/3997726222 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v105'90 uv90 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f63a0055820 con 0x55b3e08970d0 2024-09-16T01:42:08.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.591+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0899ee0 con 0x55b3e07c79e0 2024-09-16T01:42:08.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.592+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v105'105 uv104 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.592+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0899ee0 con 0x55b3e07c79e0 2024-09-16T01:42:08.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.592+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v105'106 uv106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.593+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0899ee0 con 0x55b3e07c79e0 2024-09-16T01:42:08.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.593+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v105'107 uv107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.596 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:42:08.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.593+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089aef0 con 0x55b3e07c79e0 2024-09-16T01:42:08.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.594+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v105'108 uv108 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.594+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089aef0 con 0x55b3e07c79e0 2024-09-16T01:42:08.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.595+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v105'109 uv109 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.595+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0899ee0 con 0x55b3e07c79e0 2024-09-16T01:42:08.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.596+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v105'110 uv110 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e07c79e0 2024-09-16T01:42:08.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.596+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0899ee0 con 0x55b3e088f5b0 2024-09-16T01:42:08.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.597+0000 7f63a5fae640 1 -- 172.21.15.88:0/3997726222 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v105'172 uv166 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f6398054ca0 con 0x55b3e088f5b0 2024-09-16T01:42:08.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.597+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089a280 con 0x55b3e088f5b0 2024-09-16T01:42:08.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.598+0000 7f63a5fae640 1 -- 172.21.15.88:0/3997726222 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v105'173 uv173 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6398054ca0 con 0x55b3e088f5b0 2024-09-16T01:42:08.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.598+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089a620 con 0x55b3e088f5b0 2024-09-16T01:42:08.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.599+0000 7f63a5fae640 1 -- 172.21.15.88:0/3997726222 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v105'174 uv174 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6398054ca0 con 0x55b3e088f5b0 2024-09-16T01:42:08.601 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:42:08.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.599+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089aa00 con 0x55b3e088f5b0 2024-09-16T01:42:08.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.600+0000 7f63a5fae640 1 -- 172.21.15.88:0/3997726222 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v105'175 uv175 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6398054ca0 con 0x55b3e088f5b0 2024-09-16T01:42:08.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.600+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089aa00 con 0x55b3e088f5b0 2024-09-16T01:42:08.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.601+0000 7f63a5fae640 1 -- 172.21.15.88:0/3997726222 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v105'176 uv176 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6398054ca0 con 0x55b3e088f5b0 2024-09-16T01:42:08.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.601+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089a9c0 con 0x55b3e088f5b0 2024-09-16T01:42:08.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.602+0000 7f63a5fae640 1 -- 172.21.15.88:0/3997726222 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v105'177 uv171 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6398054ca0 con 0x55b3e088f5b0 2024-09-16T01:42:08.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.602+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089a9c0 con 0x55b3e088f5b0 2024-09-16T01:42:08.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.603+0000 7f63a5fae640 1 -- 172.21.15.88:0/3997726222 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v105'178 uv178 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6398054ca0 con 0x55b3e088f5b0 2024-09-16T01:42:08.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.603+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089a9c0 con 0x55b3e088f5b0 2024-09-16T01:42:08.606 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:42:08.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.604+0000 7f63a5fae640 1 -- 172.21.15.88:0/3997726222 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v105'179 uv179 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6398054ca0 con 0x55b3e088f5b0 2024-09-16T01:42:08.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.604+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089a9c0 con 0x55b3e088f5b0 2024-09-16T01:42:08.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.605+0000 7f63a5fae640 1 -- 172.21.15.88:0/3997726222 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v105'180 uv180 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6398054ca0 con 0x55b3e088f5b0 2024-09-16T01:42:08.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.605+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e0892ef0 con 0x55b3e088f5b0 2024-09-16T01:42:08.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.607+0000 7f63a5fae640 1 -- 172.21.15.88:0/3997726222 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v105'181 uv181 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6398054ca0 con 0x55b3e088f5b0 2024-09-16T01:42:08.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.607+0000 7f63b1720f40 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55b3e089b810 0x55b3e089df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.607+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089e4c0 con 0x55b3e089b810 2024-09-16T01:42:08.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.608+0000 7f63a5fae640 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55b3e089b810 0x55b3e089df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.608+0000 7f63a5fae640 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55b3e089b810 0x55b3e089df80 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:08.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.609+0000 7f63a5fae640 1 -- 172.21.15.88:0/3997726222 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v105'78 uv77 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f6398054ca0 con 0x55b3e089b810 2024-09-16T01:42:08.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.609+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089e4c0 con 0x55b3e089b810 2024-09-16T01:42:08.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.610+0000 7f63a5fae640 1 -- 172.21.15.88:0/3997726222 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v105'79 uv79 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f6398054ca0 con 0x55b3e089b810 2024-09-16T01:42:08.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.610+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089e4c0 con 0x55b3e089b810 2024-09-16T01:42:08.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.611+0000 7f63a5fae640 1 -- 172.21.15.88:0/3997726222 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v105'80 uv80 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f6398054ca0 con 0x55b3e089b810 2024-09-16T01:42:08.613 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:42:08.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.611+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089e4c0 con 0x55b3e089b810 2024-09-16T01:42:08.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.611+0000 7f63a5fae640 1 -- 172.21.15.88:0/3997726222 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v105'81 uv81 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f6398054ca0 con 0x55b3e089b810 2024-09-16T01:42:08.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.611+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089e4c0 con 0x55b3e088f5b0 2024-09-16T01:42:08.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.613+0000 7f63a5fae640 1 -- 172.21.15.88:0/3997726222 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v105'105 uv104 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6398054ca0 con 0x55b3e088f5b0 2024-09-16T01:42:08.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.613+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089e4c0 con 0x55b3e088f5b0 2024-09-16T01:42:08.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.614+0000 7f63a5fae640 1 -- 172.21.15.88:0/3997726222 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v105'106 uv106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6398054ca0 con 0x55b3e088f5b0 2024-09-16T01:42:08.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.614+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089e4c0 con 0x55b3e088f5b0 2024-09-16T01:42:08.617 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:42:08.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.615+0000 7f63a5fae640 1 -- 172.21.15.88:0/3997726222 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v105'107 uv107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6398054ca0 con 0x55b3e088f5b0 2024-09-16T01:42:08.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.615+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089f4d0 con 0x55b3e088f5b0 2024-09-16T01:42:08.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.615+0000 7f63a5fae640 1 -- 172.21.15.88:0/3997726222 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v105'108 uv108 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6398054ca0 con 0x55b3e088f5b0 2024-09-16T01:42:08.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.615+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089f4d0 con 0x55b3e088f5b0 2024-09-16T01:42:08.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.617+0000 7f63a5fae640 1 -- 172.21.15.88:0/3997726222 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v105'109 uv109 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6398054ca0 con 0x55b3e088f5b0 2024-09-16T01:42:08.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.617+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089e4c0 con 0x55b3e088f5b0 2024-09-16T01:42:08.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.617+0000 7f63a5fae640 1 -- 172.21.15.88:0/3997726222 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v105'110 uv110 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6398054ca0 con 0x55b3e088f5b0 2024-09-16T01:42:08.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.618+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089e4c0 con 0x55b3e0893980 2024-09-16T01:42:08.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.619+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v105'89 uv88 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.619+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089e4c0 con 0x55b3e0893980 2024-09-16T01:42:08.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.620+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v105'90 uv90 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.620+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089e4c0 con 0x55b3e0893980 2024-09-16T01:42:08.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.621+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v105'91 uv91 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.623 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:42:08.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.621+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089e4c0 con 0x55b3e0893980 2024-09-16T01:42:08.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.622+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v105'92 uv92 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.622+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089e4c0 con 0x55b3e0893980 2024-09-16T01:42:08.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.623+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v105'93 uv93 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.623+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089e4c0 con 0x55b3e0893980 2024-09-16T01:42:08.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.624+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v105'216 uv208 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.624+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089e4c0 con 0x55b3e0893980 2024-09-16T01:42:08.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.625+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v105'217 uv217 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.625+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089e4c0 con 0x55b3e0893980 2024-09-16T01:42:08.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.626+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v105'218 uv218 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.628 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:42:08.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.626+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089e4c0 con 0x55b3e0893980 2024-09-16T01:42:08.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.627+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v105'219 uv219 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.627+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089e4c0 con 0x55b3e0893980 2024-09-16T01:42:08.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.628+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v105'220 uv220 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.628+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089e4c0 con 0x55b3e0893980 2024-09-16T01:42:08.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.629+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v105'221 uv215 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.629+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089e4c0 con 0x55b3e0893980 2024-09-16T01:42:08.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.630+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v105'222 uv222 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.630+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089e4c0 con 0x55b3e0893980 2024-09-16T01:42:08.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.631+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v105'223 uv223 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.633 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:42:08.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.631+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089f4d0 con 0x55b3e0893980 2024-09-16T01:42:08.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.632+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v105'224 uv224 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.632+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089f4d0 con 0x55b3e0893980 2024-09-16T01:42:08.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.633+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v105'225 uv225 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.633+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089e4c0 con 0x55b3e0893980 2024-09-16T01:42:08.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.634+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v105'226 uv226 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.634+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x55b3e089e4c0 con 0x55b3e0893980 2024-09-16T01:42:08.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v105'227 uv227 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f63a800ac10 con 0x55b3e0893980 2024-09-16T01:42:08.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55b3e07c79e0 msgr2=0x55b3e07c7dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:08.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63b1720f40 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55b3e07c79e0 0x55b3e07c7dc0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55b3e088f5b0 msgr2=0x55b3e08919a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:08.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63b1720f40 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55b3e088f5b0 0x55b3e08919a0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55b3e089b810 msgr2=0x55b3e089df80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:08.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63b1720f40 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55b3e089b810 0x55b3e089df80 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55b3e088a7f0 msgr2=0x55b3e088cbe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:08.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63b1720f40 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55b3e088a7f0 0x55b3e088cbe0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 reap_dead start 2024-09-16T01:42:08.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b3e0893980 msgr2=0x55b3e0893d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:08.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63b1720f40 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b3e0893980 0x55b3e0893d80 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63aefb0640 1 -- 172.21.15.88:0/3997726222 reap_dead start 2024-09-16T01:42:08.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55b3e08970d0 msgr2=0x55b3e08994c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:08.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63b1720f40 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55b3e08970d0 0x55b3e08994c0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f639006acc0 msgr2=0x7f639006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:08.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63b1720f40 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f639006acc0 0x7f639006d180 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f6398057000 tx=0x7f6398052000 comp rx=0 tx=0).stop 2024-09-16T01:42:08.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3e070f940 msgr2=0x55b3e07dcaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:08.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63b1720f40 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3e070f940 0x55b3e07dcaf0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f63a0001ed0 tx=0x7f63a0000ed0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 shutdown_connections 2024-09-16T01:42:08.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63b1720f40 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f639006acc0 0x7f639006d180 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63b1720f40 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55b3e08970d0 0x55b3e08994c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63b1720f40 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b3e0893980 0x55b3e0893d80 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63b1720f40 1 --2- 172.21.15.88:0/3997726222 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3e070f940 0x55b3e07dcaf0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 >> 172.21.15.88:0/3997726222 conn(0x55b3e0759750 msgr2=0x55b3e07e0630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:08.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 shutdown_connections 2024-09-16T01:42:08.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.635+0000 7f63b1720f40 1 -- 172.21.15.88:0/3997726222 wait complete. 2024-09-16T01:42:08.641 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put 500.00000000 - 2024-09-16T01:42:08.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.725+0000 7ff4dd52ef40 1 Processor -- start 2024-09-16T01:42:08.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.725+0000 7ff4dd52ef40 1 -- start start 2024-09-16T01:42:08.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.726+0000 7ff4dd52ef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56467954d390 0x56467954c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.726+0000 7ff4dd52ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56467954f7c0 0x5646794a72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.726+0000 7ff4dd52ef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646795500e0 0x5646794bf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.726+0000 7ff4dd52ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5646793c9510 con 0x56467954f7c0 2024-09-16T01:42:08.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.726+0000 7ff4dd52ef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x564679422c50 con 0x56467954d390 2024-09-16T01:42:08.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.726+0000 7ff4dd52ef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x564679551230 con 0x5646795500e0 2024-09-16T01:42:08.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.726+0000 7ff4d9dbc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56467954f7c0 0x5646794a72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.726+0000 7ff4d9dbc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56467954f7c0 0x5646794a72d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44868/0 (socket says 172.21.15.88:44868) 2024-09-16T01:42:08.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.726+0000 7ff4d9dbc640 1 -- 172.21.15.88:0/684870891 learned_addr learned my addr 172.21.15.88:0/684870891 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:08.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.726+0000 7ff4da5bd640 1 --2- 172.21.15.88:0/684870891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56467954d390 0x56467954c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.726+0000 7ff4dadbe640 1 --2- 172.21.15.88:0/684870891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646795500e0 0x5646794bf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.726+0000 7ff4d9dbc640 1 -- 172.21.15.88:0/684870891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646795500e0 msgr2=0x5646794bf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:08.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.726+0000 7ff4d9dbc640 1 --2- 172.21.15.88:0/684870891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646795500e0 0x5646794bf3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.726+0000 7ff4d9dbc640 1 -- 172.21.15.88:0/684870891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56467954d390 msgr2=0x56467954c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:08.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.726+0000 7ff4d9dbc640 1 --2- 172.21.15.88:0/684870891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56467954d390 0x56467954c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.726+0000 7ff4d9dbc640 1 -- 172.21.15.88:0/684870891 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56467954cbd0 con 0x56467954f7c0 2024-09-16T01:42:08.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.726+0000 7ff4dadbe640 1 --2- 172.21.15.88:0/684870891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646795500e0 0x5646794bf3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:08.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.726+0000 7ff4d9dbc640 1 --2- 172.21.15.88:0/684870891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56467954f7c0 0x5646794a72d0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0046170 tx=0x7ff4d0051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=affd2bc36c9672b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:08.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.727+0000 7ff4d8dba640 1 -- 172.21.15.88:0/684870891 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0058540 con 0x56467954f7c0 2024-09-16T01:42:08.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.727+0000 7ff4d8dba640 1 -- 172.21.15.88:0/684870891 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4d0058950 con 0x56467954f7c0 2024-09-16T01:42:08.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.727+0000 7ff4d8dba640 1 -- 172.21.15.88:0/684870891 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0058d40 con 0x56467954f7c0 2024-09-16T01:42:08.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.727+0000 7ff4dd52ef40 1 -- 172.21.15.88:0/684870891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56467954f7c0 msgr2=0x5646794a72d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:08.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.727+0000 7ff4dd52ef40 1 --2- 172.21.15.88:0/684870891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56467954f7c0 0x5646794a72d0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0046170 tx=0x7ff4d0051000 comp rx=0 tx=0).stop 2024-09-16T01:42:08.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.727+0000 7ff4dd52ef40 1 -- 172.21.15.88:0/684870891 shutdown_connections 2024-09-16T01:42:08.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.727+0000 7ff4dd52ef40 1 --2- 172.21.15.88:0/684870891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646795500e0 0x5646794bf3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.727+0000 7ff4dd52ef40 1 --2- 172.21.15.88:0/684870891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56467954f7c0 0x5646794a72d0 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.727+0000 7ff4dd52ef40 1 --2- 172.21.15.88:0/684870891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56467954d390 0x56467954c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.727+0000 7ff4dd52ef40 1 -- 172.21.15.88:0/684870891 >> 172.21.15.88:0/684870891 conn(0x5646794a2680 msgr2=0x5646794c1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:08.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.727+0000 7ff4dd52ef40 1 -- 172.21.15.88:0/684870891 shutdown_connections 2024-09-16T01:42:08.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.727+0000 7ff4dd52ef40 1 -- 172.21.15.88:0/684870891 wait complete. 2024-09-16T01:42:08.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.728+0000 7ff4dd52ef40 1 Processor -- start 2024-09-16T01:42:08.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.728+0000 7ff4dd52ef40 1 -- start start 2024-09-16T01:42:08.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.728+0000 7ff4dd52ef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56467954d390 0x5646794c03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.728+0000 7ff4dd52ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56467954f7c0 0x5646795453a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.728+0000 7ff4dd52ef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646795500e0 0x564679542f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.728+0000 7ff4dd52ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x564679542d20 con 0x56467954f7c0 2024-09-16T01:42:08.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.728+0000 7ff4dd52ef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56467954bf40 con 0x56467954d390 2024-09-16T01:42:08.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.728+0000 7ff4dd52ef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5646794be4e0 con 0x5646795500e0 2024-09-16T01:42:08.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.728+0000 7ff4d9dbc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56467954f7c0 0x5646795453a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.729+0000 7ff4d9dbc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56467954f7c0 0x5646795453a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44884/0 (socket says 172.21.15.88:44884) 2024-09-16T01:42:08.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.729+0000 7ff4d9dbc640 1 -- 172.21.15.88:0/1074706389 learned_addr learned my addr 172.21.15.88:0/1074706389 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:08.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.729+0000 7ff4da5bd640 1 --2- 172.21.15.88:0/1074706389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56467954d390 0x5646794c03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.729+0000 7ff4dadbe640 1 --2- 172.21.15.88:0/1074706389 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646795500e0 0x564679542f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.729+0000 7ff4d9dbc640 1 -- 172.21.15.88:0/1074706389 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646795500e0 msgr2=0x564679542f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:08.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.729+0000 7ff4d9dbc640 1 --2- 172.21.15.88:0/1074706389 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5646795500e0 0x564679542f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.729+0000 7ff4d9dbc640 1 -- 172.21.15.88:0/1074706389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56467954d390 msgr2=0x5646794c03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:08.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.729+0000 7ff4d9dbc640 1 --2- 172.21.15.88:0/1074706389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56467954d390 0x5646794c03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.729+0000 7ff4d9dbc640 1 -- 172.21.15.88:0/1074706389 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646794a8520 con 0x56467954f7c0 2024-09-16T01:42:08.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.729+0000 7ff4d9dbc640 1 --2- 172.21.15.88:0/1074706389 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56467954f7c0 0x5646795453a0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0046170 tx=0x7ff4d0002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:08.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.729+0000 7ff4cb7fe640 1 -- 172.21.15.88:0/1074706389 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0070070 con 0x56467954f7c0 2024-09-16T01:42:08.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.729+0000 7ff4cb7fe640 1 -- 172.21.15.88:0/1074706389 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4d0058c80 con 0x56467954f7c0 2024-09-16T01:42:08.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.729+0000 7ff4dd52ef40 1 -- 172.21.15.88:0/1074706389 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5646794c0a60 con 0x56467954f7c0 2024-09-16T01:42:08.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.729+0000 7ff4cb7fe640 1 -- 172.21.15.88:0/1074706389 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d004f620 con 0x56467954f7c0 2024-09-16T01:42:08.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.729+0000 7ff4dd52ef40 1 -- 172.21.15.88:0/1074706389 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646794c1020 con 0x56467954f7c0 2024-09-16T01:42:08.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.731+0000 7ff4cb7fe640 1 -- 172.21.15.88:0/1074706389 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff4d0073070 con 0x56467954f7c0 2024-09-16T01:42:08.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.731+0000 7ff4cb7fe640 1 --2- 172.21.15.88:0/1074706389 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff4bc06ac90 0x7ff4bc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.731+0000 7ff4da5bd640 1 --2- 172.21.15.88:0/1074706389 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff4bc06ac90 0x7ff4bc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.731+0000 7ff4cb7fe640 1 -- 172.21.15.88:0/1074706389 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7286+0+0 (secure 0 0 0) 0x7ff4d00ceb10 con 0x56467954f7c0 2024-09-16T01:42:08.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.731+0000 7ff4dd52ef40 1 --2- 172.21.15.88:0/1074706389 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5646795458e0 0x564679540a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.731+0000 7ff4dd52ef40 1 -- 172.21.15.88:0/1074706389 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:1 12.15 12:aa448500:::500.00000000:head [writefull 0~80 in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) -- 0x5646794c1b00 con 0x5646795458e0 2024-09-16T01:42:08.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.731+0000 7ff4da5bd640 1 --2- 172.21.15.88:0/1074706389 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff4bc06ac90 0x7ff4bc06d150 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc003f10 tx=0x7ff4cc003890 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:08.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.731+0000 7ff4dadbe640 1 --2- 172.21.15.88:0/1074706389 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5646795458e0 0x564679540a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.732+0000 7ff4dadbe640 1 --2- 172.21.15.88:0/1074706389 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5646795458e0 0x564679540a80 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:08.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.733+0000 7ff4dadbe640 1 -- 172.21.15.88:0/1074706389 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(1 500.00000000 [writefull 0~80] v105'204 uv204 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff4d40034b0 con 0x5646795458e0 2024-09-16T01:42:08.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.733+0000 7ff4dd52ef40 1 -- 172.21.15.88:0/1074706389 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5646795458e0 msgr2=0x564679540a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:08.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.733+0000 7ff4dd52ef40 1 --2- 172.21.15.88:0/1074706389 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5646795458e0 0x564679540a80 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.733+0000 7ff4dd52ef40 1 -- 172.21.15.88:0/1074706389 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff4bc06ac90 msgr2=0x7ff4bc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:08.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.733+0000 7ff4dd52ef40 1 --2- 172.21.15.88:0/1074706389 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff4bc06ac90 0x7ff4bc06d150 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc003f10 tx=0x7ff4cc003890 comp rx=0 tx=0).stop 2024-09-16T01:42:08.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.733+0000 7ff4dd52ef40 1 -- 172.21.15.88:0/1074706389 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56467954f7c0 msgr2=0x5646795453a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:08.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.733+0000 7ff4dd52ef40 1 --2- 172.21.15.88:0/1074706389 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56467954f7c0 0x5646795453a0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0046170 tx=0x7ff4d0002fb0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.733+0000 7ff4dadbe640 1 -- 172.21.15.88:0/1074706389 reap_dead start 2024-09-16T01:42:08.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.733+0000 7ff4dd52ef40 1 -- 172.21.15.88:0/1074706389 shutdown_connections 2024-09-16T01:42:08.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.733+0000 7ff4dd52ef40 1 -- 172.21.15.88:0/1074706389 >> 172.21.15.88:0/1074706389 conn(0x5646794a2680 msgr2=0x564679549130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:08.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.733+0000 7ff4dd52ef40 1 -- 172.21.15.88:0/1074706389 shutdown_connections 2024-09-16T01:42:08.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.733+0000 7ff4dd52ef40 1 -- 172.21.15.88:0/1074706389 wait complete. 2024-09-16T01:42:08.743 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:42:08.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.880+0000 7fb9e5c4c640 1 Processor -- start 2024-09-16T01:42:08.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.880+0000 7fb9e5c4c640 1 -- start start 2024-09-16T01:42:08.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.881+0000 7fb9e5c4c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9e00731c0 0x7fb9e0073600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.881+0000 7fb9e5c4c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb9e015de80 0x7fb9e015e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.881+0000 7fb9e5c4c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9e015a430 con 0x7fb9e015a050 2024-09-16T01:42:08.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.881+0000 7fb9e5c4c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb9e0073b40 con 0x7fb9e015de80 2024-09-16T01:42:08.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.881+0000 7fb9e5c4c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb9e00cd0a0 con 0x7fb9e00731c0 2024-09-16T01:42:08.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.881+0000 7fb9df7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb9e015a050 0x7fb9e015d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:40076/0 (socket says 172.21.15.88:40076) 2024-09-16T01:42:08.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.881+0000 7fb9df7fe640 1 -- 172.21.15.88:0/3798280089 learned_addr learned my addr 172.21.15.88:0/3798280089 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:08.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.881+0000 7fb9dffff640 1 --2- 172.21.15.88:0/3798280089 >> v2:172.21.15.154:3300/0 conn(0x7fb9e015de80 0x7fb9e015e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.881+0000 7fb9deffd640 1 --2- 172.21.15.88:0/3798280089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9e00731c0 0x7fb9e0073600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.881+0000 7fb9de7fc640 1 -- 172.21.15.88:0/3798280089 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2309620129 0 0) 0x7fb9e015a430 con 0x7fb9e015a050 2024-09-16T01:42:08.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.881+0000 7fb9de7fc640 1 -- 172.21.15.88:0/3798280089 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9c8003520 con 0x7fb9e015a050 2024-09-16T01:42:08.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.882+0000 7fb9dffff640 1 -- 172.21.15.88:0/3798280089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9e00731c0 msgr2=0x7fb9e0073600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:08.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.882+0000 7fb9dffff640 1 --2- 172.21.15.88:0/3798280089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9e00731c0 0x7fb9e0073600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.882+0000 7fb9dffff640 1 -- 172.21.15.88:0/3798280089 >> v1:172.21.15.88:6789/0 conn(0x7fb9e015a050 legacy=0x7fb9e015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:08.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.882+0000 7fb9de7fc640 1 -- 172.21.15.88:0/3798280089 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2116899662 0 0) 0x7fb9c8003520 con 0x7fb9e015a050 2024-09-16T01:42:08.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.882+0000 7fb9dffff640 1 -- 172.21.15.88:0/3798280089 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9e00cd380 con 0x7fb9e015de80 2024-09-16T01:42:08.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.882+0000 7fb9dffff640 1 --2- 172.21.15.88:0/3798280089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9e015de80 0x7fb9e015e310 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8009d90 tx=0x7fb9d802a560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7ef03a959fa49e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:08.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.883+0000 7fb9de7fc640 1 -- 172.21.15.88:0/3798280089 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0001670 con 0x7fb9e015de80 2024-09-16T01:42:08.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.883+0000 7fb9de7fc640 1 -- 172.21.15.88:0/3798280089 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9d8007540 con 0x7fb9e015de80 2024-09-16T01:42:08.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.883+0000 7fb9de7fc640 1 -- 172.21.15.88:0/3798280089 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d80078f0 con 0x7fb9e015de80 2024-09-16T01:42:08.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.883+0000 7fb9e5c4c640 1 -- 172.21.15.88:0/3798280089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9e015de80 msgr2=0x7fb9e015e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:08.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.883+0000 7fb9e5c4c640 1 --2- 172.21.15.88:0/3798280089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9e015de80 0x7fb9e015e310 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8009d90 tx=0x7fb9d802a560 comp rx=0 tx=0).stop 2024-09-16T01:42:08.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.883+0000 7fb9e5c4c640 1 -- 172.21.15.88:0/3798280089 shutdown_connections 2024-09-16T01:42:08.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.883+0000 7fb9e5c4c640 1 --2- 172.21.15.88:0/3798280089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9e015de80 0x7fb9e015e310 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.883+0000 7fb9e5c4c640 1 --2- 172.21.15.88:0/3798280089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9e00731c0 0x7fb9e0073600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.883+0000 7fb9e5c4c640 1 -- 172.21.15.88:0/3798280089 >> 172.21.15.88:0/3798280089 conn(0x7fb9e006d0f0 msgr2=0x7fb9e006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:08.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.883+0000 7fb9e5c4c640 1 -- 172.21.15.88:0/3798280089 shutdown_connections 2024-09-16T01:42:08.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.883+0000 7fb9e5c4c640 1 -- 172.21.15.88:0/3798280089 wait complete. 2024-09-16T01:42:08.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.885+0000 7fb9e5c4c640 1 Processor -- start 2024-09-16T01:42:08.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.885+0000 7fb9e5c4c640 1 -- start start 2024-09-16T01:42:08.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.885+0000 7fb9e5c4c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9e00731c0 0x7fb9e01529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.885+0000 7fb9e5c4c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9e015a050 0x7fb9e0145070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.886+0000 7fb9e5c4c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9e015de80 0x7fb9e0151040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.886+0000 7fb9e5c4c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e00cdc20 con 0x7fb9e015a050 2024-09-16T01:42:08.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.886+0000 7fb9e5c4c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e00cd940 con 0x7fb9e00731c0 2024-09-16T01:42:08.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.886+0000 7fb9e5c4c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb9e00b36e0 con 0x7fb9e015de80 2024-09-16T01:42:08.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.886+0000 7fb9deffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9e015a050 0x7fb9e0145070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.886+0000 7fb9deffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9e015a050 0x7fb9e0145070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44890/0 (socket says 172.21.15.88:44890) 2024-09-16T01:42:08.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.886+0000 7fb9deffd640 1 -- 172.21.15.88:0/158352493 learned_addr learned my addr 172.21.15.88:0/158352493 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:08.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.886+0000 7fb9df7fe640 1 --2- 172.21.15.88:0/158352493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9e00731c0 0x7fb9e01529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.886+0000 7fb9dffff640 1 --2- 172.21.15.88:0/158352493 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9e015de80 0x7fb9e0151040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.886+0000 7fb9deffd640 1 -- 172.21.15.88:0/158352493 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9e015de80 msgr2=0x7fb9e0151040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:08.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.886+0000 7fb9deffd640 1 --2- 172.21.15.88:0/158352493 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9e015de80 0x7fb9e0151040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.886+0000 7fb9deffd640 1 -- 172.21.15.88:0/158352493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9e00731c0 msgr2=0x7fb9e01529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:08.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.886+0000 7fb9deffd640 1 --2- 172.21.15.88:0/158352493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9e00731c0 0x7fb9e01529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:08.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.886+0000 7fb9deffd640 1 -- 172.21.15.88:0/158352493 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9e0151940 con 0x7fb9e015a050 2024-09-16T01:42:08.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.886+0000 7fb9deffd640 1 --2- 172.21.15.88:0/158352493 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9e015a050 0x7fb9e0145070 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fb9d400e9e0 tx=0x7fb9d400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:08.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.886+0000 7fb9c7fff640 1 -- 172.21.15.88:0/158352493 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d4017070 con 0x7fb9e015a050 2024-09-16T01:42:08.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.886+0000 7fb9c7fff640 1 -- 172.21.15.88:0/158352493 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9d400c9d0 con 0x7fb9e015a050 2024-09-16T01:42:08.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.886+0000 7fb9e5c4c640 1 -- 172.21.15.88:0/158352493 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9e0151c20 con 0x7fb9e015a050 2024-09-16T01:42:08.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.886+0000 7fb9c7fff640 1 -- 172.21.15.88:0/158352493 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d400cdc0 con 0x7fb9e015a050 2024-09-16T01:42:08.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.886+0000 7fb9e5c4c640 1 -- 172.21.15.88:0/158352493 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9e01521e0 con 0x7fb9e015a050 2024-09-16T01:42:08.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.887+0000 7fb9e5c4c640 1 -- 172.21.15.88:0/158352493 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9e00ccd00 con 0x7fb9e015a050 2024-09-16T01:42:08.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.888+0000 7fb9c7fff640 1 -- 172.21.15.88:0/158352493 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb9d401a070 con 0x7fb9e015a050 2024-09-16T01:42:08.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.888+0000 7fb9c7fff640 1 --2- 172.21.15.88:0/158352493 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb9cc06c2b0 0x7fb9cc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:08.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.888+0000 7fb9c7fff640 1 -- 172.21.15.88:0/158352493 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7286+0+0 (secure 0 0 0) 0x7fb9d408bd60 con 0x7fb9e015a050 2024-09-16T01:42:08.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.888+0000 7fb9df7fe640 1 --2- 172.21.15.88:0/158352493 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb9cc06c2b0 0x7fb9cc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:08.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.889+0000 7fb9df7fe640 1 --2- 172.21.15.88:0/158352493 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb9cc06c2b0 0x7fb9cc06e770 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0005880 tx=0x7fb9d00057f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:08.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:08.893+0000 7fb9c7fff640 1 -- 172.21.15.88:0/158352493 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb9d4059850 con 0x7fb9e015a050 2024-09-16T01:42:09.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.094+0000 7fb9e5c4c640 1 -- 172.21.15.88:0/158352493 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fb9e015bf40 con 0x7fb9e015a050 2024-09-16T01:42:09.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.434+0000 7fb9c7fff640 1 -- 172.21.15.88:0/158352493 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v179) ==== 162+0+0 (secure 0 0 0) 0x7fb9d405db50 con 0x7fb9e015a050 2024-09-16T01:42:09.436 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:42:09.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.436+0000 7fb9e5c4c640 1 -- 172.21.15.88:0/158352493 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb9cc06c2b0 msgr2=0x7fb9cc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:09.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.436+0000 7fb9e5c4c640 1 --2- 172.21.15.88:0/158352493 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb9cc06c2b0 0x7fb9cc06e770 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0005880 tx=0x7fb9d00057f0 comp rx=0 tx=0).stop 2024-09-16T01:42:09.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.436+0000 7fb9e5c4c640 1 -- 172.21.15.88:0/158352493 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9e015a050 msgr2=0x7fb9e0145070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:09.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.436+0000 7fb9e5c4c640 1 --2- 172.21.15.88:0/158352493 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9e015a050 0x7fb9e0145070 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fb9d400e9e0 tx=0x7fb9d400eea0 comp rx=0 tx=0).stop 2024-09-16T01:42:09.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.436+0000 7fb9e5c4c640 1 -- 172.21.15.88:0/158352493 shutdown_connections 2024-09-16T01:42:09.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.436+0000 7fb9e5c4c640 1 --2- 172.21.15.88:0/158352493 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb9cc06c2b0 0x7fb9cc06e770 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:09.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.436+0000 7fb9e5c4c640 1 --2- 172.21.15.88:0/158352493 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9e015de80 0x7fb9e0151040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:09.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.436+0000 7fb9e5c4c640 1 --2- 172.21.15.88:0/158352493 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9e015a050 0x7fb9e0145070 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:09.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.436+0000 7fb9e5c4c640 1 --2- 172.21.15.88:0/158352493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9e00731c0 0x7fb9e01529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:09.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.436+0000 7fb9e5c4c640 1 -- 172.21.15.88:0/158352493 >> 172.21.15.88:0/158352493 conn(0x7fb9e006d0f0 msgr2=0x7fb9e015b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:09.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.436+0000 7fb9e5c4c640 1 -- 172.21.15.88:0/158352493 shutdown_connections 2024-09-16T01:42:09.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.436+0000 7fb9e5c4c640 1 -- 172.21.15.88:0/158352493 wait complete. 2024-09-16T01:42:09.443 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:42:09.441+0000 7fd3f783d640 -1 mds.0.purge_queue operator(): Error -22 loading Journaler 2024-09-16T01:42:09.443 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:42:09.441+0000 7fd3f783d640 -1 mds.0.180 unhandled write error (22) Invalid argument, force readonly... 2024-09-16T01:42:09.452 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:09.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.548+0000 7f83c08fa640 1 Processor -- start 2024-09-16T01:42:09.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.548+0000 7f83c08fa640 1 -- start start 2024-09-16T01:42:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.549+0000 7f83c08fa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f83bc145bb0 0x7f83bc06e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.549+0000 7f83c08fa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83bc06e5f0 0x7f83bc1439e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.549+0000 7f83c08fa640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83bc06ea10 con 0x7f83bc13f400 2024-09-16T01:42:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.549+0000 7f83c08fa640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f83bc0d03c0 con 0x7f83bc145bb0 2024-09-16T01:42:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.549+0000 7f83c08fa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f83bc0d06a0 con 0x7f83bc06e5f0 2024-09-16T01:42:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.549+0000 7f83b3fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f83bc145bb0 0x7f83bc06e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.549+0000 7f83b3fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f83bc145bb0 0x7f83bc06e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44906/0 (socket says 172.21.15.88:44906) 2024-09-16T01:42:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.549+0000 7f83b3fff640 1 -- 172.21.15.88:0/1839828160 learned_addr learned my addr 172.21.15.88:0/1839828160 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.549+0000 7f83bad9d640 1 --2- 172.21.15.88:0/1839828160 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83bc06e5f0 0x7f83bc1439e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.549+0000 7f83b3fff640 1 -- 172.21.15.88:0/1839828160 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83bc06e5f0 msgr2=0x7f83bc1439e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.549+0000 7f83b3fff640 1 --2- 172.21.15.88:0/1839828160 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83bc06e5f0 0x7f83bc1439e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:09.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.549+0000 7f83b3fff640 1 -- 172.21.15.88:0/1839828160 >> v1:172.21.15.154:6789/0 conn(0x7f83bc13f400 legacy=0x7f83bc0cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:09.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.549+0000 7f83b3fff640 1 -- 172.21.15.88:0/1839828160 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83bc1431e0 con 0x7f83bc145bb0 2024-09-16T01:42:09.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.550+0000 7f83b3fff640 1 --2- 172.21.15.88:0/1839828160 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83bc145bb0 0x7f83bc06e0b0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f83a4009870 tx=0x7f83a402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=669c1d978c49d9a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:09.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.550+0000 7f83b9d9b640 1 -- 172.21.15.88:0/1839828160 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a4037070 con 0x7f83bc145bb0 2024-09-16T01:42:09.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.550+0000 7f83b9d9b640 1 -- 172.21.15.88:0/1839828160 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83a402ae10 con 0x7f83bc145bb0 2024-09-16T01:42:09.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.550+0000 7f83b9d9b640 1 -- 172.21.15.88:0/1839828160 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a4032660 con 0x7f83bc145bb0 2024-09-16T01:42:09.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.551+0000 7f83c08fa640 1 -- 172.21.15.88:0/1839828160 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83bc145bb0 msgr2=0x7f83bc06e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:09.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.551+0000 7f83c08fa640 1 --2- 172.21.15.88:0/1839828160 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83bc145bb0 0x7f83bc06e0b0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f83a4009870 tx=0x7f83a402a100 comp rx=0 tx=0).stop 2024-09-16T01:42:09.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.551+0000 7f83c08fa640 1 -- 172.21.15.88:0/1839828160 shutdown_connections 2024-09-16T01:42:09.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.551+0000 7f83c08fa640 1 --2- 172.21.15.88:0/1839828160 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83bc06e5f0 0x7f83bc1439e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:09.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.551+0000 7f83c08fa640 1 --2- 172.21.15.88:0/1839828160 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83bc145bb0 0x7f83bc06e0b0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:09.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.551+0000 7f83c08fa640 1 -- 172.21.15.88:0/1839828160 >> 172.21.15.88:0/1839828160 conn(0x7f83bc06bc10 msgr2=0x7f83bc06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:09.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.551+0000 7f83c08fa640 1 -- 172.21.15.88:0/1839828160 shutdown_connections 2024-09-16T01:42:09.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.551+0000 7f83c08fa640 1 -- 172.21.15.88:0/1839828160 wait complete. 2024-09-16T01:42:09.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.552+0000 7f83c08fa640 1 Processor -- start 2024-09-16T01:42:09.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.552+0000 7f83c08fa640 1 -- start start 2024-09-16T01:42:09.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.553+0000 7f83c08fa640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83bc06e5f0 0x7f83bc0ad6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:09.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.553+0000 7f83c08fa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83bc13f400 0x7f83bc0adc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:09.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.553+0000 7f83c08fa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83bc145bb0 0x7f83bc0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:09.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.553+0000 7f83c08fa640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc13f960 con 0x7f83bc145bb0 2024-09-16T01:42:09.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.553+0000 7f83c08fa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc13f7e0 con 0x7f83bc13f400 2024-09-16T01:42:09.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.553+0000 7f83c08fa640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f83bc143f20 con 0x7f83bc06e5f0 2024-09-16T01:42:09.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.553+0000 7f83bad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83bc145bb0 0x7f83bc0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:09.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.553+0000 7f83bad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83bc145bb0 0x7f83bc0af8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44908/0 (socket says 172.21.15.88:44908) 2024-09-16T01:42:09.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.553+0000 7f83bad9d640 1 -- 172.21.15.88:0/2285442133 learned_addr learned my addr 172.21.15.88:0/2285442133 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:09.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.553+0000 7f83b3fff640 1 --2- 172.21.15.88:0/2285442133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83bc13f400 0x7f83bc0adc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:09.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.553+0000 7f83ba59c640 1 --2- 172.21.15.88:0/2285442133 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83bc06e5f0 0x7f83bc0ad6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:09.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.553+0000 7f83bad9d640 1 -- 172.21.15.88:0/2285442133 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83bc06e5f0 msgr2=0x7f83bc0ad6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:09.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.553+0000 7f83bad9d640 1 --2- 172.21.15.88:0/2285442133 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83bc06e5f0 0x7f83bc0ad6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:09.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.553+0000 7f83bad9d640 1 -- 172.21.15.88:0/2285442133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83bc13f400 msgr2=0x7f83bc0adc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:09.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.553+0000 7f83bad9d640 1 --2- 172.21.15.88:0/2285442133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83bc13f400 0x7f83bc0adc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:09.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.553+0000 7f83bad9d640 1 -- 172.21.15.88:0/2285442133 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83bc0ae2a0 con 0x7f83bc145bb0 2024-09-16T01:42:09.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.553+0000 7f83bad9d640 1 --2- 172.21.15.88:0/2285442133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83bc145bb0 0x7f83bc0af8d0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f83b4007e40 tx=0x7f83b400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:09.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.553+0000 7f83b2ffd640 1 -- 172.21.15.88:0/2285442133 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b4014620 con 0x7f83bc145bb0 2024-09-16T01:42:09.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.553+0000 7f83b2ffd640 1 -- 172.21.15.88:0/2285442133 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83b4019070 con 0x7f83bc145bb0 2024-09-16T01:42:09.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.553+0000 7f83b2ffd640 1 -- 172.21.15.88:0/2285442133 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b4014a20 con 0x7f83bc145bb0 2024-09-16T01:42:09.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.554+0000 7f83c08fa640 1 -- 172.21.15.88:0/2285442133 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83bc0aaf30 con 0x7f83bc145bb0 2024-09-16T01:42:09.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.554+0000 7f83c08fa640 1 -- 172.21.15.88:0/2285442133 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83bc0ab4f0 con 0x7f83bc145bb0 2024-09-16T01:42:09.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.554+0000 7f83c08fa640 1 -- 172.21.15.88:0/2285442133 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83bc1439e0 con 0x7f83bc145bb0 2024-09-16T01:42:09.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.555+0000 7f83b2ffd640 1 -- 172.21.15.88:0/2285442133 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f83b4014bc0 con 0x7f83bc145bb0 2024-09-16T01:42:09.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.555+0000 7f83b2ffd640 1 --2- 172.21.15.88:0/2285442133 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f839c06c2b0 0x7f839c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:09.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.555+0000 7f83b2ffd640 1 -- 172.21.15.88:0/2285442133 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7286+0+0 (secure 0 0 0) 0x7f83b408c220 con 0x7f83bc145bb0 2024-09-16T01:42:09.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.555+0000 7f83ba59c640 1 --2- 172.21.15.88:0/2285442133 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f839c06c2b0 0x7f839c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:09.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.556+0000 7f83ba59c640 1 --2- 172.21.15.88:0/2285442133 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f839c06c2b0 0x7f839c06e770 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f83ac004f60 tx=0x7f83ac007e30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:09.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.560+0000 7f83b2ffd640 1 -- 172.21.15.88:0/2285442133 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f83b4059d10 con 0x7f83bc145bb0 2024-09-16T01:42:09.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.759+0000 7f83c08fa640 1 -- 172.21.15.88:0/2285442133 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83bc06d530 con 0x7f83bc145bb0 2024-09-16T01:42:09.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.760+0000 7f83b2ffd640 1 -- 172.21.15.88:0/2285442133 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) ==== 95+0+4401 (secure 0 0 0) 0x7f83b405e010 con 0x7f83bc145bb0 2024-09-16T01:42:09.762 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:09.762 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":180,"btime":"2024-09-16T01:42:09:437467+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:09.437457+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8679},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8679":{"gid":8679,"name":"c","rank":0,"incarnation":180,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6837/1936493048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1936493048},{"type":"v1","addr":"172.21.15.88:6837","nonce":1936493048}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:09.763 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 180 2024-09-16T01:42:09.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.762+0000 7f83c08fa640 1 -- 172.21.15.88:0/2285442133 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f839c06c2b0 msgr2=0x7f839c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:09.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.762+0000 7f83c08fa640 1 --2- 172.21.15.88:0/2285442133 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f839c06c2b0 0x7f839c06e770 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f83ac004f60 tx=0x7f83ac007e30 comp rx=0 tx=0).stop 2024-09-16T01:42:09.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.762+0000 7f83c08fa640 1 -- 172.21.15.88:0/2285442133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83bc145bb0 msgr2=0x7f83bc0af8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:09.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.762+0000 7f83c08fa640 1 --2- 172.21.15.88:0/2285442133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83bc145bb0 0x7f83bc0af8d0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f83b4007e40 tx=0x7f83b400e4b0 comp rx=0 tx=0).stop 2024-09-16T01:42:09.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.762+0000 7f83c08fa640 1 -- 172.21.15.88:0/2285442133 shutdown_connections 2024-09-16T01:42:09.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.762+0000 7f83c08fa640 1 --2- 172.21.15.88:0/2285442133 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f839c06c2b0 0x7f839c06e770 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:09.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.762+0000 7f83c08fa640 1 --2- 172.21.15.88:0/2285442133 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83bc145bb0 0x7f83bc0af8d0 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:09.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.762+0000 7f83c08fa640 1 --2- 172.21.15.88:0/2285442133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83bc13f400 0x7f83bc0adc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:09.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.762+0000 7f83c08fa640 1 --2- 172.21.15.88:0/2285442133 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83bc06e5f0 0x7f83bc0ad6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:09.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.762+0000 7f83c08fa640 1 -- 172.21.15.88:0/2285442133 >> 172.21.15.88:0/2285442133 conn(0x7f83bc06bc10 msgr2=0x7f83bc15d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:09.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.762+0000 7f83c08fa640 1 -- 172.21.15.88:0/2285442133 shutdown_connections 2024-09-16T01:42:09.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:09.762+0000 7f83c08fa640 1 -- 172.21.15.88:0/2285442133 wait complete. 2024-09-16T01:42:10.163 INFO:tasks.ceph.mds.a.smithi088.stderr: -36> 2024-09-16T01:42:05.411+0000 7f7498546640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (22) Invalid argument 2024-09-16T01:42:10.177 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:10.187 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:42:10.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.275+0000 7fadd9e8b640 1 Processor -- start 2024-09-16T01:42:10.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.275+0000 7fadd9e8b640 1 -- start start 2024-09-16T01:42:10.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.276+0000 7fadd9e8b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fadd40cfc80 0x7fadd40b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:10.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.276+0000 7fadd9e8b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fadd415c330 0x7fadd40b4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:10.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.276+0000 7fadd9e8b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadd415c740 con 0x7fadd4155800 2024-09-16T01:42:10.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.276+0000 7fadd9e8b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fadd40b4e60 con 0x7fadd415c330 2024-09-16T01:42:10.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.276+0000 7fadd9e8b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fadd40b50a0 con 0x7fadd40cfc80 2024-09-16T01:42:10.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.276+0000 7fadd37fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fadd4155800 0x7fadd40cf570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:40096/0 (socket says 172.21.15.88:40096) 2024-09-16T01:42:10.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.276+0000 7fadd37fe640 1 -- 172.21.15.88:0/1917676499 learned_addr learned my addr 172.21.15.88:0/1917676499 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:10.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.276+0000 7fadd3fff640 1 --2- 172.21.15.88:0/1917676499 >> v2:172.21.15.88:3300/0 conn(0x7fadd415c330 0x7fadd40b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:10.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.276+0000 7fadd2ffd640 1 --2- 172.21.15.88:0/1917676499 >> v2:172.21.15.154:3300/0 conn(0x7fadd40cfc80 0x7fadd40b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:10.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.276+0000 7fadd3fff640 1 -- 172.21.15.88:0/1917676499 >> v2:172.21.15.154:3300/0 conn(0x7fadd40cfc80 msgr2=0x7fadd40b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:10.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.276+0000 7fadd3fff640 1 --2- 172.21.15.88:0/1917676499 >> v2:172.21.15.154:3300/0 conn(0x7fadd40cfc80 0x7fadd40b6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:10.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.276+0000 7fadd3fff640 1 -- 172.21.15.88:0/1917676499 >> v1:172.21.15.88:6789/0 conn(0x7fadd4155800 legacy=0x7fadd40cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:10.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.276+0000 7fadd27fc640 1 -- 172.21.15.88:0/1917676499 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 332087568 0 0) 0x7fadd415c740 con 0x7fadd4155800 2024-09-16T01:42:10.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.276+0000 7fadd3fff640 1 -- 172.21.15.88:0/1917676499 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd40b5380 con 0x7fadd415c330 2024-09-16T01:42:10.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.277+0000 7fadd3fff640 1 --2- 172.21.15.88:0/1917676499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd415c330 0x7fadd40b4920 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fadcc009d90 tx=0x7fadcc02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67d0d65de042bed server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:10.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.277+0000 7fadd27fc640 1 -- 172.21.15.88:0/1917676499 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc007740 con 0x7fadd415c330 2024-09-16T01:42:10.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.277+0000 7fadd27fc640 1 -- 172.21.15.88:0/1917676499 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadcc0078e0 con 0x7fadd415c330 2024-09-16T01:42:10.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.277+0000 7fadd27fc640 1 -- 172.21.15.88:0/1917676499 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc007cb0 con 0x7fadd415c330 2024-09-16T01:42:10.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.278+0000 7fadd9e8b640 1 -- 172.21.15.88:0/1917676499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd415c330 msgr2=0x7fadd40b4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:10.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.278+0000 7fadd9e8b640 1 --2- 172.21.15.88:0/1917676499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd415c330 0x7fadd40b4920 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fadcc009d90 tx=0x7fadcc02a6a0 comp rx=0 tx=0).stop 2024-09-16T01:42:10.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.278+0000 7fadd9e8b640 1 -- 172.21.15.88:0/1917676499 shutdown_connections 2024-09-16T01:42:10.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.278+0000 7fadd9e8b640 1 --2- 172.21.15.88:0/1917676499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd415c330 0x7fadd40b4920 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:10.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.278+0000 7fadd9e8b640 1 --2- 172.21.15.88:0/1917676499 >> v2:172.21.15.154:3300/0 conn(0x7fadd40cfc80 0x7fadd40b6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:10.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.278+0000 7fadd9e8b640 1 -- 172.21.15.88:0/1917676499 >> 172.21.15.88:0/1917676499 conn(0x7fadd406bc10 msgr2=0x7fadd406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:10.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.278+0000 7fadd9e8b640 1 -- 172.21.15.88:0/1917676499 shutdown_connections 2024-09-16T01:42:10.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.278+0000 7fadd9e8b640 1 -- 172.21.15.88:0/1917676499 wait complete. 2024-09-16T01:42:10.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.279+0000 7fadd9e8b640 1 Processor -- start 2024-09-16T01:42:10.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.279+0000 7fadd9e8b640 1 -- start start 2024-09-16T01:42:10.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.280+0000 7fadd9e8b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadd40cfc80 0x7fadd414ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:10.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.280+0000 7fadd9e8b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd4155800 0x7fadd4155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:10.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.280+0000 7fadd9e8b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd415c330 0x7fadd41598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:10.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.280+0000 7fadd9e8b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fadd40b5c20 con 0x7fadd4155800 2024-09-16T01:42:10.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.280+0000 7fadd9e8b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fadd40b5940 con 0x7fadd415c330 2024-09-16T01:42:10.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.280+0000 7fadd9e8b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fadd4155be0 con 0x7fadd40cfc80 2024-09-16T01:42:10.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.280+0000 7fadd2ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd4155800 0x7fadd4155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:10.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.280+0000 7fadd2ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd4155800 0x7fadd4155e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:44938/0 (socket says 172.21.15.88:44938) 2024-09-16T01:42:10.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.280+0000 7fadd2ffd640 1 -- 172.21.15.88:0/1196011002 learned_addr learned my addr 172.21.15.88:0/1196011002 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:10.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.280+0000 7fadd2ffd640 1 -- 172.21.15.88:0/1196011002 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadd40cfc80 msgr2=0x7fadd414ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.280+0000 7fadd2ffd640 1 --2- 172.21.15.88:0/1196011002 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadd40cfc80 0x7fadd414ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.280+0000 7fadd2ffd640 1 -- 172.21.15.88:0/1196011002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd415c330 msgr2=0x7fadd41598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.280+0000 7fadd2ffd640 1 --2- 172.21.15.88:0/1196011002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd415c330 0x7fadd41598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.280+0000 7fadd2ffd640 1 -- 172.21.15.88:0/1196011002 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd41565a0 con 0x7fadd4155800 2024-09-16T01:42:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.281+0000 7fadd2ffd640 1 --2- 172.21.15.88:0/1196011002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd4155800 0x7fadd4155e50 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fadbc00e8d0 tx=0x7fadbc00ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.281+0000 7fadcbfff640 1 -- 172.21.15.88:0/1196011002 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadbc017020 con 0x7fadd4155800 2024-09-16T01:42:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.281+0000 7fadcbfff640 1 -- 172.21.15.88:0/1196011002 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadbc004240 con 0x7fadd4155800 2024-09-16T01:42:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.281+0000 7fadd9e8b640 1 -- 172.21.15.88:0/1196011002 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadd4156880 con 0x7fadd4155800 2024-09-16T01:42:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.281+0000 7fadcbfff640 1 -- 172.21.15.88:0/1196011002 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadbc00c6d0 con 0x7fadd4155800 2024-09-16T01:42:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.281+0000 7fadd9e8b640 1 -- 172.21.15.88:0/1196011002 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadd4156e40 con 0x7fadd4155800 2024-09-16T01:42:10.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.282+0000 7fadd9e8b640 1 -- 172.21.15.88:0/1196011002 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadd40b6140 con 0x7fadd4155800 2024-09-16T01:42:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.282+0000 7fadcbfff640 1 -- 172.21.15.88:0/1196011002 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fadbc01a020 con 0x7fadd4155800 2024-09-16T01:42:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.283+0000 7fadcbfff640 1 --2- 172.21.15.88:0/1196011002 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fadb406c2e0 0x7fadb406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.283+0000 7fadcbfff640 1 -- 172.21.15.88:0/1196011002 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7286+0+0 (secure 0 0 0) 0x7fadbc08bbd0 con 0x7fadd4155800 2024-09-16T01:42:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.283+0000 7fadd37fe640 1 --2- 172.21.15.88:0/1196011002 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fadb406c2e0 0x7fadb406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:10.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.283+0000 7fadd37fe640 1 --2- 172.21.15.88:0/1196011002 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fadb406c2e0 0x7fadb406e7a0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fadc4007fd0 tx=0x7fadc4007920 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:10.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.287+0000 7fadcbfff640 1 -- 172.21.15.88:0/1196011002 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fadbc021030 con 0x7fadd4155800 2024-09-16T01:42:10.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.489+0000 7fadd9e8b640 1 -- 172.21.15.88:0/1196011002 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fadd415fac0 con 0x7fadd4155800 2024-09-16T01:42:10.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.489+0000 7fadcbfff640 1 -- 172.21.15.88:0/1196011002 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) ==== 95+0+5230 (secure 0 0 0) 0x7fadbc059da0 con 0x7fadd4155800 2024-09-16T01:42:10.492 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:10.492 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":181,"btime":"2024-09-16T01:42:10:439324+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8736,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3374372758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3374372758},{"type":"v1","addr":"172.21.15.88:6836","nonce":3374372758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":181}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:09.447668+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8679},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8679":{"gid":8679,"name":"c","rank":0,"incarnation":180,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.88:6837/1936493048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1936493048},{"type":"v1","addr":"172.21.15.88:6837","nonce":1936493048}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:10.492 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 181 2024-09-16T01:42:10.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.491+0000 7fadd9e8b640 1 -- 172.21.15.88:0/1196011002 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fadb406c2e0 msgr2=0x7fadb406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:10.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.491+0000 7fadd9e8b640 1 --2- 172.21.15.88:0/1196011002 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fadb406c2e0 0x7fadb406e7a0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fadc4007fd0 tx=0x7fadc4007920 comp rx=0 tx=0).stop 2024-09-16T01:42:10.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.491+0000 7fadd9e8b640 1 -- 172.21.15.88:0/1196011002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd4155800 msgr2=0x7fadd4155e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:10.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.491+0000 7fadd9e8b640 1 --2- 172.21.15.88:0/1196011002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd4155800 0x7fadd4155e50 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fadbc00e8d0 tx=0x7fadbc00ed90 comp rx=0 tx=0).stop 2024-09-16T01:42:10.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.491+0000 7fadd9e8b640 1 -- 172.21.15.88:0/1196011002 shutdown_connections 2024-09-16T01:42:10.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.491+0000 7fadd9e8b640 1 --2- 172.21.15.88:0/1196011002 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fadb406c2e0 0x7fadb406e7a0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:10.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.491+0000 7fadd9e8b640 1 --2- 172.21.15.88:0/1196011002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd415c330 0x7fadd41598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:10.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.491+0000 7fadd9e8b640 1 --2- 172.21.15.88:0/1196011002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadd4155800 0x7fadd4155e50 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:10.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.491+0000 7fadd9e8b640 1 --2- 172.21.15.88:0/1196011002 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadd40cfc80 0x7fadd414ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:10.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.491+0000 7fadd9e8b640 1 -- 172.21.15.88:0/1196011002 >> 172.21.15.88:0/1196011002 conn(0x7fadd406bc10 msgr2=0x7fadd40cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:10.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.491+0000 7fadd9e8b640 1 -- 172.21.15.88:0/1196011002 shutdown_connections 2024-09-16T01:42:10.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:10.492+0000 7fadd9e8b640 1 -- 172.21.15.88:0/1196011002 wait complete. 2024-09-16T01:42:10.504 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 181, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:40:03.597796+0000', 'modified': '2024-09-16T01:42:09.447668+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 105, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8679}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8679': {'gid': 8679, 'name': 'c', 'rank': 0, 'incarnation': 180, 'state': 'up:reconnect', 'state_seq': 2, 'addr': '172.21.15.88:6837/1936493048', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6835', 'nonce': 1936493048}, {'type': 'v1', 'addr': '172.21.15.88:6837', 'nonce': 1936493048}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-16T01:42:10.505 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8679:up:reconnect 2024-09-16T01:42:10.505 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-09-16T01:42:15.506 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:15.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.615+0000 7f326e6d9640 1 Processor -- start 2024-09-16T01:42:15.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.615+0000 7f326e6d9640 1 -- start start 2024-09-16T01:42:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.616+0000 7f326e6d9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3268062a70 0x7f32680b52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.616+0000 7f326e6d9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f326815e9b0 con 0x7f32680b5800 2024-09-16T01:42:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.616+0000 7f326e6d9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3268162a50 con 0x7f326815e5d0 2024-09-16T01:42:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.616+0000 7f326e6d9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3268162d30 con 0x7f3268062a70 2024-09-16T01:42:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.616+0000 7f326cc76640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f32680b5800 0x7f3268072e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44932/0 (socket says 172.21.15.88:44932) 2024-09-16T01:42:15.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.616+0000 7f326cc76640 1 -- 172.21.15.88:0/1061775856 learned_addr learned my addr 172.21.15.88:0/1061775856 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:15.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.616+0000 7f3266ffd640 1 -- 172.21.15.88:0/1061775856 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1054348612 0 0) 0x7f326815e9b0 con 0x7f32680b5800 2024-09-16T01:42:15.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.616+0000 7f3266ffd640 1 -- 172.21.15.88:0/1061775856 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3250003520 con 0x7f32680b5800 2024-09-16T01:42:15.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.616+0000 7f32677fe640 1 --2- 172.21.15.88:0/1061775856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3268062a70 0x7f32680b52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:15.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.617+0000 7f3266ffd640 1 -- 172.21.15.88:0/1061775856 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2674608242 0 0) 0x7f3250003520 con 0x7f32680b5800 2024-09-16T01:42:15.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.617+0000 7f3266ffd640 1 -- 172.21.15.88:0/1061775856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3268062a70 msgr2=0x7f32680b52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:15.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.617+0000 7f3266ffd640 1 --2- 172.21.15.88:0/1061775856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3268062a70 0x7f32680b52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:15.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.617+0000 7f3266ffd640 1 -- 172.21.15.88:0/1061775856 >> v1:172.21.15.154:6789/0 conn(0x7f326815e5d0 legacy=0x7f3268162170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:15.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.617+0000 7f3266ffd640 1 -- 172.21.15.88:0/1061775856 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3268163010 con 0x7f32680b5800 2024-09-16T01:42:15.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.617+0000 7f3266ffd640 1 -- 172.21.15.88:0/1061775856 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f32600019f0 con 0x7f32680b5800 2024-09-16T01:42:15.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.617+0000 7f3266ffd640 1 -- 172.21.15.88:0/1061775856 >> v1:172.21.15.88:6789/0 conn(0x7f32680b5800 legacy=0x7f3268072e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:15.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.617+0000 7f3266ffd640 1 --2- 172.21.15.88:0/1061775856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32500039e0 0x7f3250003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:15.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.617+0000 7f3266ffd640 1 --2- 172.21.15.88:0/1061775856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3250004390 0x7f3250004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:15.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.617+0000 7f3266ffd640 1 --2- 172.21.15.88:0/1061775856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3268062a70 0x7f3250005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:15.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.617+0000 7f3266ffd640 1 -- 172.21.15.88:0/1061775856 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3268162d30 con 0x7f3250004390 2024-09-16T01:42:15.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.617+0000 7f3266ffd640 1 -- 172.21.15.88:0/1061775856 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f325000baa0 con 0x7f32500039e0 2024-09-16T01:42:15.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.617+0000 7f3266ffd640 1 -- 172.21.15.88:0/1061775856 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f325000bca0 con 0x7f3268062a70 2024-09-16T01:42:15.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.617+0000 7f32677fe640 1 --2- 172.21.15.88:0/1061775856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3250004390 0x7f3250004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:15.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.617+0000 7f3267fff640 1 --2- 172.21.15.88:0/1061775856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32500039e0 0x7f3250003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:15.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.617+0000 7f326cc76640 1 --2- 172.21.15.88:0/1061775856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3268062a70 0x7f3250005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:15.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.617+0000 7f32677fe640 1 -- 172.21.15.88:0/1061775856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3268062a70 msgr2=0x7f3250005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:15.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.617+0000 7f32677fe640 1 --2- 172.21.15.88:0/1061775856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3268062a70 0x7f3250005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:15.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.617+0000 7f32677fe640 1 -- 172.21.15.88:0/1061775856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32500039e0 msgr2=0x7f3250003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:15.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.617+0000 7f32677fe640 1 --2- 172.21.15.88:0/1061775856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32500039e0 0x7f3250003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:15.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.617+0000 7f32677fe640 1 -- 172.21.15.88:0/1061775856 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3250003520 con 0x7f3250004390 2024-09-16T01:42:15.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.617+0000 7f326cc76640 1 --2- 172.21.15.88:0/1061775856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3268062a70 0x7f3250005160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:15.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.618+0000 7f32677fe640 1 --2- 172.21.15.88:0/1061775856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3250004390 0x7f3250004860 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f325c009a90 tx=0x7f3250005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a71c5964cff2fd27 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:15.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.618+0000 7f3266ffd640 1 -- 172.21.15.88:0/1061775856 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325c036070 con 0x7f3250004390 2024-09-16T01:42:15.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.618+0000 7f3266ffd640 1 -- 172.21.15.88:0/1061775856 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f325c031400 con 0x7f3250004390 2024-09-16T01:42:15.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.618+0000 7f3266ffd640 1 -- 172.21.15.88:0/1061775856 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325c009690 con 0x7f3250004390 2024-09-16T01:42:15.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.618+0000 7f326e6d9640 1 -- 172.21.15.88:0/1061775856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3250004390 msgr2=0x7f3250004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:15.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.618+0000 7f326e6d9640 1 --2- 172.21.15.88:0/1061775856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3250004390 0x7f3250004860 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f325c009a90 tx=0x7f3250005830 comp rx=0 tx=0).stop 2024-09-16T01:42:15.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.618+0000 7f326cc76640 1 -- 172.21.15.88:0/1061775856 reap_dead start 2024-09-16T01:42:15.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.619+0000 7f326e6d9640 1 -- 172.21.15.88:0/1061775856 shutdown_connections 2024-09-16T01:42:15.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.619+0000 7f326e6d9640 1 -- 172.21.15.88:0/1061775856 >> 172.21.15.88:0/1061775856 conn(0x7f326806bc10 msgr2=0x7f326806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:15.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.619+0000 7f326e6d9640 1 -- 172.21.15.88:0/1061775856 shutdown_connections 2024-09-16T01:42:15.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.619+0000 7f326e6d9640 1 -- 172.21.15.88:0/1061775856 wait complete. 2024-09-16T01:42:15.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.620+0000 7f326e6d9640 1 Processor -- start 2024-09-16T01:42:15.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.620+0000 7f326e6d9640 1 -- start start 2024-09-16T01:42:15.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.621+0000 7f326e6d9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3250004390 0x7f326814b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:15.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.621+0000 7f326e6d9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32681e7fb0 0x7f326814a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:15.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.621+0000 7f326e6d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32681e95c0 0x7f32681e9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:15.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.621+0000 7f326e6d9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3268073590 con 0x7f32681e95c0 2024-09-16T01:42:15.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.621+0000 7f326e6d9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3268071b90 con 0x7f32681e7fb0 2024-09-16T01:42:15.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.621+0000 7f326e6d9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3268073710 con 0x7f3250004390 2024-09-16T01:42:15.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.621+0000 7f326cc76640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32681e95c0 0x7f32681e9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:15.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.621+0000 7f326cc76640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32681e95c0 0x7f32681e9a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43718/0 (socket says 172.21.15.88:43718) 2024-09-16T01:42:15.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.621+0000 7f326cc76640 1 -- 172.21.15.88:0/3135602426 learned_addr learned my addr 172.21.15.88:0/3135602426 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:15.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.621+0000 7f32677fe640 1 --2- 172.21.15.88:0/3135602426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32681e7fb0 0x7f326814a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:15.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.621+0000 7f3267fff640 1 --2- 172.21.15.88:0/3135602426 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3250004390 0x7f326814b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:15.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.621+0000 7f326cc76640 1 -- 172.21.15.88:0/3135602426 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3250004390 msgr2=0x7f326814b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:15.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.621+0000 7f326cc76640 1 --2- 172.21.15.88:0/3135602426 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3250004390 0x7f326814b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:15.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.621+0000 7f326cc76640 1 -- 172.21.15.88:0/3135602426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32681e7fb0 msgr2=0x7f326814a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:15.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.621+0000 7f326cc76640 1 --2- 172.21.15.88:0/3135602426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32681e7fb0 0x7f326814a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:15.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.621+0000 7f326cc76640 1 -- 172.21.15.88:0/3135602426 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32681e8520 con 0x7f32681e95c0 2024-09-16T01:42:15.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.621+0000 7f32677fe640 1 --2- 172.21.15.88:0/3135602426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32681e7fb0 0x7f326814a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:15.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.621+0000 7f3267fff640 1 --2- 172.21.15.88:0/3135602426 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3250004390 0x7f326814b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:15.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.621+0000 7f326cc76640 1 --2- 172.21.15.88:0/3135602426 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32681e95c0 0x7f32681e9a40 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f3260002230 tx=0x7f326000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:15.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.621+0000 7f3264ff9640 1 -- 172.21.15.88:0/3135602426 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326000ec10 con 0x7f32681e95c0 2024-09-16T01:42:15.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.621+0000 7f3264ff9640 1 -- 172.21.15.88:0/3135602426 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3260018070 con 0x7f32681e95c0 2024-09-16T01:42:15.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.621+0000 7f326e6d9640 1 -- 172.21.15.88:0/3135602426 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32681e8800 con 0x7f32681e95c0 2024-09-16T01:42:15.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.621+0000 7f3264ff9640 1 -- 172.21.15.88:0/3135602426 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3260013420 con 0x7f32681e95c0 2024-09-16T01:42:15.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.622+0000 7f326e6d9640 1 -- 172.21.15.88:0/3135602426 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32681e8dc0 con 0x7f32681e95c0 2024-09-16T01:42:15.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.623+0000 7f326e6d9640 1 -- 172.21.15.88:0/3135602426 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32680b52c0 con 0x7f32681e95c0 2024-09-16T01:42:15.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.623+0000 7f3264ff9640 1 -- 172.21.15.88:0/3135602426 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f32600135c0 con 0x7f32681e95c0 2024-09-16T01:42:15.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.624+0000 7f3264ff9640 1 --2- 172.21.15.88:0/3135602426 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f325406c2e0 0x7f325406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:15.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.624+0000 7f3267fff640 1 --2- 172.21.15.88:0/3135602426 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f325406c2e0 0x7f325406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:15.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.624+0000 7f3264ff9640 1 -- 172.21.15.88:0/3135602426 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7286+0+0 (secure 0 0 0) 0x7f326008bd70 con 0x7f32681e95c0 2024-09-16T01:42:15.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.624+0000 7f3267fff640 1 --2- 172.21.15.88:0/3135602426 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f325406c2e0 0x7f325406e7a0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f3258000c00 tx=0x7f3258009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:15.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.629+0000 7f3264ff9640 1 -- 172.21.15.88:0/3135602426 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3260059eb0 con 0x7f32681e95c0 2024-09-16T01:42:15.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.828+0000 7f326e6d9640 1 -- 172.21.15.88:0/3135602426 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f326806f2f0 con 0x7f32681e95c0 2024-09-16T01:42:15.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.828+0000 7f3264ff9640 1 -- 172.21.15.88:0/3135602426 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) ==== 95+0+5234 (secure 0 0 0) 0x7f3260060080 con 0x7f32681e95c0 2024-09-16T01:42:15.831 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:15.831 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":183,"btime":"2024-09-16T01:42:12:447758+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8736,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3374372758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3374372758},{"type":"v1","addr":"172.21.15.88:6836","nonce":3374372758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":181}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:12.447749+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8679},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8679":{"gid":8679,"name":"c","rank":0,"incarnation":180,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6837/1936493048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1936493048},{"type":"v1","addr":"172.21.15.88:6837","nonce":1936493048}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":6}]} 2024-09-16T01:42:15.831 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 183 2024-09-16T01:42:15.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.831+0000 7f326e6d9640 1 -- 172.21.15.88:0/3135602426 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f325406c2e0 msgr2=0x7f325406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:15.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.831+0000 7f326e6d9640 1 --2- 172.21.15.88:0/3135602426 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f325406c2e0 0x7f325406e7a0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f3258000c00 tx=0x7f3258009040 comp rx=0 tx=0).stop 2024-09-16T01:42:15.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.831+0000 7f326e6d9640 1 -- 172.21.15.88:0/3135602426 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32681e95c0 msgr2=0x7f32681e9a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:15.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.831+0000 7f326e6d9640 1 --2- 172.21.15.88:0/3135602426 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32681e95c0 0x7f32681e9a40 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f3260002230 tx=0x7f326000e530 comp rx=0 tx=0).stop 2024-09-16T01:42:15.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.831+0000 7f326e6d9640 1 -- 172.21.15.88:0/3135602426 shutdown_connections 2024-09-16T01:42:15.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.831+0000 7f326e6d9640 1 --2- 172.21.15.88:0/3135602426 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f325406c2e0 0x7f325406e7a0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:15.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.831+0000 7f326e6d9640 1 --2- 172.21.15.88:0/3135602426 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32681e95c0 0x7f32681e9a40 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:15.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.831+0000 7f326e6d9640 1 --2- 172.21.15.88:0/3135602426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32681e7fb0 0x7f326814a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:15.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.831+0000 7f326e6d9640 1 --2- 172.21.15.88:0/3135602426 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3250004390 0x7f326814b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:15.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.831+0000 7f326e6d9640 1 -- 172.21.15.88:0/3135602426 >> 172.21.15.88:0/3135602426 conn(0x7f326806bc10 msgr2=0x7f326815bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:15.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.831+0000 7f326e6d9640 1 -- 172.21.15.88:0/3135602426 shutdown_connections 2024-09-16T01:42:15.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:15.831+0000 7f326e6d9640 1 -- 172.21.15.88:0/3135602426 wait complete. 2024-09-16T01:42:16.246 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:16.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.348+0000 7fae30333640 1 Processor -- start 2024-09-16T01:42:16.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.348+0000 7fae30333640 1 -- start start 2024-09-16T01:42:16.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.349+0000 7fae30333640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae2815a9c0 0x7fae2815e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.349+0000 7fae30333640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fae280b29a0 0x7fae280cde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.349+0000 7fae30333640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae280b6830 con 0x7fae280b5be0 2024-09-16T01:42:16.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.349+0000 7fae30333640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fae2815f410 con 0x7fae280b29a0 2024-09-16T01:42:16.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.349+0000 7fae30333640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fae280cd0b0 con 0x7fae2815a9c0 2024-09-16T01:42:16.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.349+0000 7fae2d8ce640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fae280b5be0 0x7fae2815eb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44936/0 (socket says 172.21.15.88:44936) 2024-09-16T01:42:16.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.349+0000 7fae2d8ce640 1 -- 172.21.15.88:0/2013369592 learned_addr learned my addr 172.21.15.88:0/2013369592 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:16.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.349+0000 7fae2e8d0640 1 --2- 172.21.15.88:0/2013369592 >> v2:172.21.15.88:3300/0 conn(0x7fae280b29a0 0x7fae280cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.349+0000 7fae2e0cf640 1 --2- 172.21.15.88:0/2013369592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae2815a9c0 0x7fae2815e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.349+0000 7fae2e8d0640 1 -- 172.21.15.88:0/2013369592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae2815a9c0 msgr2=0x7fae2815e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:16.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.349+0000 7fae2e8d0640 1 --2- 172.21.15.88:0/2013369592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae2815a9c0 0x7fae2815e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.349+0000 7fae2e8d0640 1 -- 172.21.15.88:0/2013369592 >> v1:172.21.15.88:6789/0 conn(0x7fae280b5be0 legacy=0x7fae2815eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:16.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.349+0000 7fae2e8d0640 1 -- 172.21.15.88:0/2013369592 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae280cd390 con 0x7fae280b29a0 2024-09-16T01:42:16.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.349+0000 7fae2d0cd640 1 -- 172.21.15.88:0/2013369592 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1953200710 0 0) 0x7fae280b6830 con 0x7fae280b5be0 2024-09-16T01:42:16.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.350+0000 7fae2e8d0640 1 --2- 172.21.15.88:0/2013369592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae280b29a0 0x7fae280cde70 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fae2000aea0 tx=0x7fae2002b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a48d58ce654fcb86 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:16.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.350+0000 7fae2d0cd640 1 -- 172.21.15.88:0/2013369592 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae20037070 con 0x7fae280b29a0 2024-09-16T01:42:16.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.350+0000 7fae2d0cd640 1 -- 172.21.15.88:0/2013369592 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae20008640 con 0x7fae280b29a0 2024-09-16T01:42:16.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.350+0000 7fae2d0cd640 1 -- 172.21.15.88:0/2013369592 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae20008a10 con 0x7fae280b29a0 2024-09-16T01:42:16.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.351+0000 7fae30333640 1 -- 172.21.15.88:0/2013369592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae280b29a0 msgr2=0x7fae280cde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:16.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.351+0000 7fae30333640 1 --2- 172.21.15.88:0/2013369592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae280b29a0 0x7fae280cde70 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fae2000aea0 tx=0x7fae2002b7b0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.351+0000 7fae30333640 1 -- 172.21.15.88:0/2013369592 shutdown_connections 2024-09-16T01:42:16.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.351+0000 7fae30333640 1 --2- 172.21.15.88:0/2013369592 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae280b29a0 0x7fae280cde70 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.351+0000 7fae30333640 1 --2- 172.21.15.88:0/2013369592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae2815a9c0 0x7fae2815e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.351+0000 7fae30333640 1 -- 172.21.15.88:0/2013369592 >> 172.21.15.88:0/2013369592 conn(0x7fae2806bc10 msgr2=0x7fae2806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:16.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.351+0000 7fae30333640 1 -- 172.21.15.88:0/2013369592 shutdown_connections 2024-09-16T01:42:16.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.351+0000 7fae30333640 1 -- 172.21.15.88:0/2013369592 wait complete. 2024-09-16T01:42:16.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.352+0000 7fae30333640 1 Processor -- start 2024-09-16T01:42:16.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.352+0000 7fae30333640 1 -- start start 2024-09-16T01:42:16.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.353+0000 7fae30333640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae280b29a0 0x7fae28143090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.353+0000 7fae30333640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae280b5be0 0x7fae28150020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.353+0000 7fae30333640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae2815a9c0 0x7fae2814c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.353+0000 7fae30333640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fae2815ada0 con 0x7fae2815a9c0 2024-09-16T01:42:16.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.353+0000 7fae30333640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fae280cd950 con 0x7fae280b5be0 2024-09-16T01:42:16.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.353+0000 7fae30333640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fae2806de40 con 0x7fae280b29a0 2024-09-16T01:42:16.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.353+0000 7fae2e8d0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae2815a9c0 0x7fae2814c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.353+0000 7fae2e8d0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae2815a9c0 0x7fae2814c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43746/0 (socket says 172.21.15.88:43746) 2024-09-16T01:42:16.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.353+0000 7fae2e8d0640 1 -- 172.21.15.88:0/3709156659 learned_addr learned my addr 172.21.15.88:0/3709156659 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:16.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.353+0000 7fae2d8ce640 1 --2- 172.21.15.88:0/3709156659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae280b5be0 0x7fae28150020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.353+0000 7fae2e0cf640 1 --2- 172.21.15.88:0/3709156659 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae280b29a0 0x7fae28143090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.353+0000 7fae2e8d0640 1 -- 172.21.15.88:0/3709156659 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae280b29a0 msgr2=0x7fae28143090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:16.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.353+0000 7fae2e8d0640 1 --2- 172.21.15.88:0/3709156659 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae280b29a0 0x7fae28143090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.353+0000 7fae2e8d0640 1 -- 172.21.15.88:0/3709156659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae280b5be0 msgr2=0x7fae28150020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:16.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.353+0000 7fae2e8d0640 1 --2- 172.21.15.88:0/3709156659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae280b5be0 0x7fae28150020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.353+0000 7fae2e8d0640 1 -- 172.21.15.88:0/3709156659 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae2814d0f0 con 0x7fae2815a9c0 2024-09-16T01:42:16.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.353+0000 7fae2e0cf640 1 --2- 172.21.15.88:0/3709156659 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae280b29a0 0x7fae28143090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:16.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.353+0000 7fae2d8ce640 1 --2- 172.21.15.88:0/3709156659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae280b5be0 0x7fae28150020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:16.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.353+0000 7fae2e8d0640 1 --2- 172.21.15.88:0/3709156659 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae2815a9c0 0x7fae2814c650 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fae2000aea0 tx=0x7fae20004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:16.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.354+0000 7fae1effd640 1 -- 172.21.15.88:0/3709156659 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae20041070 con 0x7fae2815a9c0 2024-09-16T01:42:16.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.354+0000 7fae1effd640 1 -- 172.21.15.88:0/3709156659 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae20003990 con 0x7fae2815a9c0 2024-09-16T01:42:16.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.354+0000 7fae1effd640 1 -- 172.21.15.88:0/3709156659 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae20037040 con 0x7fae2815a9c0 2024-09-16T01:42:16.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.354+0000 7fae30333640 1 -- 172.21.15.88:0/3709156659 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae2815afc0 con 0x7fae2815a9c0 2024-09-16T01:42:16.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.354+0000 7fae30333640 1 -- 172.21.15.88:0/3709156659 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae2814d3d0 con 0x7fae2815a9c0 2024-09-16T01:42:16.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.354+0000 7fae30333640 1 -- 172.21.15.88:0/3709156659 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae2806e220 con 0x7fae2815a9c0 2024-09-16T01:42:16.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.355+0000 7fae1effd640 1 -- 172.21.15.88:0/3709156659 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fae20033840 con 0x7fae2815a9c0 2024-09-16T01:42:16.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.355+0000 7fae1effd640 1 --2- 172.21.15.88:0/3709156659 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fae0c06c2b0 0x7fae0c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.356+0000 7fae1effd640 1 -- 172.21.15.88:0/3709156659 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7286+0+0 (secure 0 0 0) 0x7fae200ac2d0 con 0x7fae2815a9c0 2024-09-16T01:42:16.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.356+0000 7fae2e0cf640 1 --2- 172.21.15.88:0/3709156659 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fae0c06c2b0 0x7fae0c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.356+0000 7fae2e0cf640 1 --2- 172.21.15.88:0/3709156659 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fae0c06c2b0 0x7fae0c06e770 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fae2400b9f0 tx=0x7fae24008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:16.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.360+0000 7fae1effd640 1 -- 172.21.15.88:0/3709156659 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fae2004b020 con 0x7fae2815a9c0 2024-09-16T01:42:16.420 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:42:16.419+0000 7fd3fa843640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:42:16.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.561+0000 7fae30333640 1 -- 172.21.15.88:0/3709156659 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae280cf9e0 con 0x7fae2815a9c0 2024-09-16T01:42:16.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.561+0000 7fae1effd640 1 -- 172.21.15.88:0/3709156659 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) ==== 95+0+5234 (secure 0 0 0) 0x7fae2007a4d0 con 0x7fae2815a9c0 2024-09-16T01:42:16.563 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:16.563 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":183,"btime":"2024-09-16T01:42:12:447758+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8736,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3374372758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3374372758},{"type":"v1","addr":"172.21.15.88:6836","nonce":3374372758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":181}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:12.447749+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8679},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8679":{"gid":8679,"name":"c","rank":0,"incarnation":180,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6837/1936493048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1936493048},{"type":"v1","addr":"172.21.15.88:6837","nonce":1936493048}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":6}]} 2024-09-16T01:42:16.563 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 183 2024-09-16T01:42:16.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.563+0000 7fae30333640 1 -- 172.21.15.88:0/3709156659 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fae0c06c2b0 msgr2=0x7fae0c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:16.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.563+0000 7fae30333640 1 --2- 172.21.15.88:0/3709156659 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fae0c06c2b0 0x7fae0c06e770 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fae2400b9f0 tx=0x7fae24008040 comp rx=0 tx=0).stop 2024-09-16T01:42:16.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.563+0000 7fae30333640 1 -- 172.21.15.88:0/3709156659 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae2815a9c0 msgr2=0x7fae2814c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:16.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.563+0000 7fae30333640 1 --2- 172.21.15.88:0/3709156659 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae2815a9c0 0x7fae2814c650 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fae2000aea0 tx=0x7fae20004a40 comp rx=0 tx=0).stop 2024-09-16T01:42:16.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.563+0000 7fae30333640 1 -- 172.21.15.88:0/3709156659 shutdown_connections 2024-09-16T01:42:16.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.563+0000 7fae30333640 1 --2- 172.21.15.88:0/3709156659 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fae0c06c2b0 0x7fae0c06e770 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.563+0000 7fae30333640 1 --2- 172.21.15.88:0/3709156659 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae2815a9c0 0x7fae2814c650 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.563+0000 7fae30333640 1 --2- 172.21.15.88:0/3709156659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae280b5be0 0x7fae28150020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.563+0000 7fae30333640 1 --2- 172.21.15.88:0/3709156659 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fae280b29a0 0x7fae28143090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.564+0000 7fae30333640 1 -- 172.21.15.88:0/3709156659 >> 172.21.15.88:0/3709156659 conn(0x7fae2806bc10 msgr2=0x7fae280cc350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:16.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.564+0000 7fae30333640 1 -- 172.21.15.88:0/3709156659 shutdown_connections 2024-09-16T01:42:16.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.564+0000 7fae30333640 1 -- 172.21.15.88:0/3709156659 wait complete. 2024-09-16T01:42:16.576 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 183, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:40:03.597796+0000', 'modified': '2024-09-16T01:42:12.447749+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 105, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8679}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8679': {'gid': 8679, 'name': 'c', 'rank': 0, 'incarnation': 180, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.88:6837/1936493048', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6835', 'nonce': 1936493048}, {'type': 'v1', 'addr': '172.21.15.88:6837', 'nonce': 1936493048}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8679, 'qdb_cluster': [8679]} 2024-09-16T01:42:16.576 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T01:42:16.577 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-16T01:42:16.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.675+0000 7f993363c640 1 Processor -- start 2024-09-16T01:42:16.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.675+0000 7f993363c640 1 -- start start 2024-09-16T01:42:16.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.676+0000 7f993363c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f992c15c2a0 0x7f992c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.676+0000 7f993363c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f992c073b10 0x7f992c06f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.676+0000 7f993363c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f992c06fc50 0x7f992c06cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.676+0000 7f993363c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f992c15c680 con 0x7f992c15c2a0 2024-09-16T01:42:16.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.676+0000 7f993363c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f992c074120 con 0x7f992c06fc50 2024-09-16T01:42:16.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.676+0000 7f993363c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f992c15ef00 con 0x7f992c073b10 2024-09-16T01:42:16.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.676+0000 7f99313d8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f992c15c2a0 0x7f992c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.676+0000 7f99313d8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f992c15c2a0 0x7f992c0735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43760/0 (socket says 172.21.15.88:43760) 2024-09-16T01:42:16.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.676+0000 7f99313d8640 1 -- 172.21.15.88:0/433106297 learned_addr learned my addr 172.21.15.88:0/433106297 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:16.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.676+0000 7f9930bd7640 1 --2- 172.21.15.88:0/433106297 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f992c073b10 0x7f992c06f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.676+0000 7f9931bd9640 1 --2- 172.21.15.88:0/433106297 >> v2:172.21.15.154:3300/0 conn(0x7f992c06fc50 0x7f992c06cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.676+0000 7f99313d8640 1 -- 172.21.15.88:0/433106297 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f992c073b10 msgr2=0x7f992c06f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:16.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.676+0000 7f99313d8640 1 --2- 172.21.15.88:0/433106297 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f992c073b10 0x7f992c06f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.676+0000 7f99313d8640 1 -- 172.21.15.88:0/433106297 >> v2:172.21.15.154:3300/0 conn(0x7f992c06fc50 msgr2=0x7f992c06cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:16.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.676+0000 7f99313d8640 1 --2- 172.21.15.88:0/433106297 >> v2:172.21.15.154:3300/0 conn(0x7f992c06fc50 0x7f992c06cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.676+0000 7f99313d8640 1 -- 172.21.15.88:0/433106297 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f992c15f1e0 con 0x7f992c15c2a0 2024-09-16T01:42:16.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.676+0000 7f9930bd7640 1 --2- 172.21.15.88:0/433106297 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f992c073b10 0x7f992c06f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:16.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.677+0000 7f99313d8640 1 --2- 172.21.15.88:0/433106297 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f992c15c2a0 0x7f992c0735d0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f992c0cdd00 tx=0x7f992802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb00f437eb430361 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:16.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.677+0000 7f9923fff640 1 -- 172.21.15.88:0/433106297 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9928037070 con 0x7f992c15c2a0 2024-09-16T01:42:16.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.677+0000 7f9923fff640 1 -- 172.21.15.88:0/433106297 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9928032440 con 0x7f992c15c2a0 2024-09-16T01:42:16.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.677+0000 7f9923fff640 1 -- 172.21.15.88:0/433106297 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9928032810 con 0x7f992c15c2a0 2024-09-16T01:42:16.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.677+0000 7f993363c640 1 -- 172.21.15.88:0/433106297 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f992c15c2a0 msgr2=0x7f992c0735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:16.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.677+0000 7f993363c640 1 --2- 172.21.15.88:0/433106297 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f992c15c2a0 0x7f992c0735d0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f992c0cdd00 tx=0x7f992802a1e0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.678+0000 7f993363c640 1 -- 172.21.15.88:0/433106297 shutdown_connections 2024-09-16T01:42:16.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.678+0000 7f993363c640 1 --2- 172.21.15.88:0/433106297 >> v2:172.21.15.154:3300/0 conn(0x7f992c06fc50 0x7f992c06cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.678+0000 7f993363c640 1 --2- 172.21.15.88:0/433106297 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f992c073b10 0x7f992c06f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.678+0000 7f993363c640 1 --2- 172.21.15.88:0/433106297 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f992c15c2a0 0x7f992c0735d0 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.678+0000 7f993363c640 1 -- 172.21.15.88:0/433106297 >> 172.21.15.88:0/433106297 conn(0x7f992c0b3070 msgr2=0x7f992c0b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:16.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.678+0000 7f993363c640 1 -- 172.21.15.88:0/433106297 shutdown_connections 2024-09-16T01:42:16.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.678+0000 7f993363c640 1 -- 172.21.15.88:0/433106297 wait complete. 2024-09-16T01:42:16.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.679+0000 7f993363c640 1 Processor -- start 2024-09-16T01:42:16.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.679+0000 7f993363c640 1 -- start start 2024-09-16T01:42:16.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.679+0000 7f993363c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f992c06fc50 0x7f992c149ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.679+0000 7f993363c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f992c073b10 0x7f992c0cdd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.679+0000 7f993363c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f992c15c2a0 0x7f992c157d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.679+0000 7f993363c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f992c0b5f20 con 0x7f992c06fc50 2024-09-16T01:42:16.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.679+0000 7f993363c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f992c15ebf0 con 0x7f992c15c2a0 2024-09-16T01:42:16.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.680+0000 7f993363c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f992c0b60a0 con 0x7f992c073b10 2024-09-16T01:42:16.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.680+0000 7f99313d8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f992c06fc50 0x7f992c149ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.680+0000 7f99313d8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f992c06fc50 0x7f992c149ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43776/0 (socket says 172.21.15.88:43776) 2024-09-16T01:42:16.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.680+0000 7f99313d8640 1 -- 172.21.15.88:0/3357804019 learned_addr learned my addr 172.21.15.88:0/3357804019 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:16.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.680+0000 7f9931bd9640 1 --2- 172.21.15.88:0/3357804019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f992c15c2a0 0x7f992c157d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.680+0000 7f9930bd7640 1 --2- 172.21.15.88:0/3357804019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f992c073b10 0x7f992c0cdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.680+0000 7f99313d8640 1 -- 172.21.15.88:0/3357804019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f992c073b10 msgr2=0x7f992c0cdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:16.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.680+0000 7f99313d8640 1 --2- 172.21.15.88:0/3357804019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f992c073b10 0x7f992c0cdd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.680+0000 7f99313d8640 1 -- 172.21.15.88:0/3357804019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f992c15c2a0 msgr2=0x7f992c157d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:16.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.680+0000 7f99313d8640 1 --2- 172.21.15.88:0/3357804019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f992c15c2a0 0x7f992c157d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.680+0000 7f99313d8640 1 -- 172.21.15.88:0/3357804019 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f992c15b1a0 con 0x7f992c06fc50 2024-09-16T01:42:16.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.680+0000 7f99313d8640 1 --2- 172.21.15.88:0/3357804019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f992c06fc50 0x7f992c149ef0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f9928000c00 tx=0x7f99280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:16.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.680+0000 7f9921ffb640 1 -- 172.21.15.88:0/3357804019 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9928041070 con 0x7f992c06fc50 2024-09-16T01:42:16.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.680+0000 7f9921ffb640 1 -- 172.21.15.88:0/3357804019 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f992803c870 con 0x7f992c06fc50 2024-09-16T01:42:16.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.680+0000 7f9921ffb640 1 -- 172.21.15.88:0/3357804019 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9928037040 con 0x7f992c06fc50 2024-09-16T01:42:16.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.680+0000 7f993363c640 1 -- 172.21.15.88:0/3357804019 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f992c15b460 con 0x7f992c06fc50 2024-09-16T01:42:16.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.681+0000 7f993363c640 1 -- 172.21.15.88:0/3357804019 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f992c0d05a0 con 0x7f992c06fc50 2024-09-16T01:42:16.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.682+0000 7f9921ffb640 1 -- 172.21.15.88:0/3357804019 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9928044070 con 0x7f992c06fc50 2024-09-16T01:42:16.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.683+0000 7f9921ffb640 1 --2- 172.21.15.88:0/3357804019 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f991006c2b0 0x7f991006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.683+0000 7f9921ffb640 1 -- 172.21.15.88:0/3357804019 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7286+0+0 (secure 0 0 0) 0x7f99280ac370 con 0x7f992c06fc50 2024-09-16T01:42:16.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.683+0000 7f9930bd7640 1 --2- 172.21.15.88:0/3357804019 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f991006c2b0 0x7f991006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.683+0000 7f9930bd7640 1 --2- 172.21.15.88:0/3357804019 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f991006c2b0 0x7f991006e770 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f992c159060 tx=0x7f991c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:16.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.684+0000 7f99363d6740 1 Processor -- start 2024-09-16T01:42:16.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.684+0000 7f99363d6740 1 -- start start 2024-09-16T01:42:16.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.684+0000 7f99363d6740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b275f2c0 0x55b3b27a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.684+0000 7f99363d6740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3b275ca80 0x55b3b275f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.684+0000 7f99363d6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3b2760a50 0x55b3b27598b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.684+0000 7f99363d6740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b3b26c1520 con 0x55b3b275f2c0 2024-09-16T01:42:16.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.684+0000 7f99363d6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b3b26693e0 con 0x55b3b2760a50 2024-09-16T01:42:16.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.684+0000 7f99363d6740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55b3b26bd4e0 con 0x55b3b275ca80 2024-09-16T01:42:16.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.684+0000 7f99313d8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b275f2c0 0x55b3b27a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.684+0000 7f99313d8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b275f2c0 0x55b3b27a4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43792/0 (socket says 172.21.15.88:43792) 2024-09-16T01:42:16.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.684+0000 7f99313d8640 1 -- 172.21.15.88:0/3696934420 learned_addr learned my addr 172.21.15.88:0/3696934420 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:16.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.685+0000 7f9931bd9640 1 --2- 172.21.15.88:0/3696934420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3b2760a50 0x55b3b27598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.685+0000 7f9930bd7640 1 --2- 172.21.15.88:0/3696934420 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3b275ca80 0x55b3b275f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.685+0000 7f99313d8640 1 -- 172.21.15.88:0/3696934420 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3b275ca80 msgr2=0x55b3b275f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:16.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.685+0000 7f99313d8640 1 --2- 172.21.15.88:0/3696934420 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3b275ca80 0x55b3b275f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.685+0000 7f99313d8640 1 -- 172.21.15.88:0/3696934420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3b2760a50 msgr2=0x55b3b27598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:16.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.685+0000 7f99313d8640 1 --2- 172.21.15.88:0/3696934420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3b2760a50 0x55b3b27598b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.685+0000 7f99313d8640 1 -- 172.21.15.88:0/3696934420 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3b26842f0 con 0x55b3b275f2c0 2024-09-16T01:42:16.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.685+0000 7f99313d8640 1 --2- 172.21.15.88:0/3696934420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b275f2c0 0x55b3b27a4a60 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f99280aa240 tx=0x7f9928079850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b67e6250ecdaa6cc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:16.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.685+0000 7f990effd640 1 -- 172.21.15.88:0/3696934420 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f992807c030 con 0x55b3b275f2c0 2024-09-16T01:42:16.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.685+0000 7f990effd640 1 -- 172.21.15.88:0/3696934420 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9928082a20 con 0x55b3b275f2c0 2024-09-16T01:42:16.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.685+0000 7f990effd640 1 -- 172.21.15.88:0/3696934420 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f992804b050 con 0x55b3b275f2c0 2024-09-16T01:42:16.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.686+0000 7f99363d6740 1 -- 172.21.15.88:0/3696934420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b275f2c0 msgr2=0x55b3b27a4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:16.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.686+0000 7f99363d6740 1 --2- 172.21.15.88:0/3696934420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b275f2c0 0x55b3b27a4a60 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f99280aa240 tx=0x7f9928079850 comp rx=0 tx=0).stop 2024-09-16T01:42:16.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.686+0000 7f99363d6740 1 -- 172.21.15.88:0/3696934420 shutdown_connections 2024-09-16T01:42:16.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.686+0000 7f99363d6740 1 --2- 172.21.15.88:0/3696934420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3b2760a50 0x55b3b27598b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.686+0000 7f99363d6740 1 --2- 172.21.15.88:0/3696934420 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3b275ca80 0x55b3b275f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.686+0000 7f99363d6740 1 --2- 172.21.15.88:0/3696934420 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b275f2c0 0x55b3b27a4a60 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.686+0000 7f99363d6740 1 -- 172.21.15.88:0/3696934420 >> 172.21.15.88:0/3696934420 conn(0x55b3b275ba80 msgr2=0x55b3b27a3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:16.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.686+0000 7f99363d6740 1 -- 172.21.15.88:0/3696934420 shutdown_connections 2024-09-16T01:42:16.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.686+0000 7f99363d6740 1 -- 172.21.15.88:0/3696934420 wait complete. 2024-09-16T01:42:16.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.686+0000 7f99363d6740 1 -- start start 2024-09-16T01:42:16.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.686+0000 7f99363d6740 1 Processor -- start 2024-09-16T01:42:16.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.687+0000 7f99363d6740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3b275f2c0 0x55b3b277df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.687+0000 7f99363d6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3b2760a50 0x55b3b275da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.687+0000 7f99363d6740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b277faa0 0x55b3b275df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.687+0000 7f99363d6740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b3b26c1520 con 0x55b3b277faa0 2024-09-16T01:42:16.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.687+0000 7f99363d6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b3b264a450 con 0x55b3b2760a50 2024-09-16T01:42:16.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.687+0000 7f99363d6740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55b3b275af20 con 0x55b3b275f2c0 2024-09-16T01:42:16.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.687+0000 7f99363d6740 10 client.0 _ll_register_callbacks cb 0x55b3b23d3ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:42:16.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.687+0000 7f9931bd9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b277faa0 0x55b3b275df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.687+0000 7f9931bd9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b277faa0 0x55b3b275df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43804/0 (socket says 172.21.15.88:43804) 2024-09-16T01:42:16.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.687+0000 7f9931bd9640 1 -- 172.21.15.88:0/1053865637 learned_addr learned my addr 172.21.15.88:0/1053865637 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:16.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.687+0000 7f9930bd7640 1 --2- 172.21.15.88:0/1053865637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3b2760a50 0x55b3b275da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.687+0000 7f99313d8640 1 --2- 172.21.15.88:0/1053865637 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3b275f2c0 0x55b3b277df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.688+0000 7f9931bd9640 1 -- 172.21.15.88:0/1053865637 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3b275f2c0 msgr2=0x55b3b277df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:16.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.688+0000 7f9931bd9640 1 --2- 172.21.15.88:0/1053865637 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3b275f2c0 0x55b3b277df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.688+0000 7f9931bd9640 1 -- 172.21.15.88:0/1053865637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3b2760a50 msgr2=0x55b3b275da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:16.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.688+0000 7f9931bd9640 1 --2- 172.21.15.88:0/1053865637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3b2760a50 0x55b3b275da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.688+0000 7f9931bd9640 1 -- 172.21.15.88:0/1053865637 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3b23d42d0 con 0x55b3b277faa0 2024-09-16T01:42:16.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.688+0000 7f9931bd9640 1 --2- 172.21.15.88:0/1053865637 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b277faa0 0x55b3b275df70 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f9924003fb0 tx=0x7f99240044d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b28cfc83c2006f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:16.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.688+0000 7f9907fff640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:42:16.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.688+0000 7f9907fff640 1 -- 172.21.15.88:0/1053865637 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f992401a020 con 0x55b3b277faa0 2024-09-16T01:42:16.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.688+0000 7f9907fff640 1 -- 172.21.15.88:0/1053865637 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 7286+0+0 (secure 0 0 0) 0x7f99240077c0 con 0x55b3b277faa0 2024-09-16T01:42:16.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.688+0000 7f99363d6740 1 -- 172.21.15.88:0/1053865637 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9928082a20 con 0x55b3b277faa0 2024-09-16T01:42:16.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.688+0000 7f9907fff640 1 -- 172.21.15.88:0/1053865637 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=183) ==== 24+0+0 (secure 0 0 0) 0x7f99240079c0 con 0x55b3b277faa0 2024-09-16T01:42:16.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.688+0000 7f99363d6740 10 client.8772 fetch_fsmap learned FSMap version 183 2024-09-16T01:42:16.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.688+0000 7f99363d6740 1 -- 172.21.15.88:0/1053865637 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=183,monmap=2+}) -- 0x55b3b249e6b0 con 0x55b3b277faa0 2024-09-16T01:42:16.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.688+0000 7f9907fff640 1 -- 172.21.15.88:0/1053865637 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9928082a20 con 0x55b3b277faa0 2024-09-16T01:42:16.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.688+0000 7f9907fff640 1 -- 172.21.15.88:0/1053865637 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 183) ==== 3502+0+0 (secure 0 0 0) 0x7f9924007b60 con 0x55b3b277faa0 2024-09-16T01:42:16.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.688+0000 7f99363d6740 10 client.8772 fetch_fsmap finished waiting for FSMap version 183 2024-09-16T01:42:16.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.688+0000 7f99363d6740 10 client.8772 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T01:42:16.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.688+0000 7f99363d6740 20 client.8772 populate_metadata read hostname 'smithi088' 2024-09-16T01:42:16.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.688+0000 7f99363d6740 1 --2- 172.21.15.88:0/1053865637 >> [v2:172.21.15.88:6835/1936493048,v1:172.21.15.88:6837/1936493048] conn(0x55b3b2788a30 0x55b3b2788e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.688+0000 7f99363d6740 4 client.8772 mds_command: new command op to 8679 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:42:16.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.688+0000 7f99363d6740 1 -- 172.21.15.88:0/1053865637 --> [v2:172.21.15.88:6835/1936493048,v1:172.21.15.88:6837/1936493048] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3b26cec50 con 0x55b3b2788a30 2024-09-16T01:42:16.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.689+0000 7f99313d8640 1 --2- 172.21.15.88:0/1053865637 >> [v2:172.21.15.88:6835/1936493048,v1:172.21.15.88:6837/1936493048] conn(0x55b3b2788a30 0x55b3b2788e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.689+0000 7f9907fff640 10 client.8772 ms_handle_connect on v2:172.21.15.88:6835/1936493048 2024-09-16T01:42:16.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.689+0000 7f99313d8640 1 --2- 172.21.15.88:0/1053865637 >> [v2:172.21.15.88:6835/1936493048,v1:172.21.15.88:6837/1936493048] conn(0x55b3b2788a30 0x55b3b2788e10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=346c9fc23d276ecc server_cookie=675ee05f6e3f6c62 in_seq=0 out_seq=0 2024-09-16T01:42:16.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.689+0000 7f9907fff640 1 -- 172.21.15.88:0/1053865637 <== mds.0 v2:172.21.15.88:6835/1936493048 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b3b26cec50 con 0x55b3b2788a30 2024-09-16T01:42:16.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.689+0000 7f9907fff640 10 client.8772 handle_command_reply: tid=0 2024-09-16T01:42:16.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.689+0000 7f99363d6740 1 client.8772 shutdown 2024-09-16T01:42:16.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.690+0000 7f99363d6740 1 -- 172.21.15.88:0/1053865637 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b277faa0 msgr2=0x55b3b275df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:16.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.690+0000 7f99363d6740 1 --2- 172.21.15.88:0/1053865637 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b277faa0 0x55b3b275df70 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f9924003fb0 tx=0x7f99240044d0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.690+0000 7f99363d6740 1 -- 172.21.15.88:0/1053865637 shutdown_connections 2024-09-16T01:42:16.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.690+0000 7f99363d6740 1 --2- 172.21.15.88:0/1053865637 >> [v2:172.21.15.88:6835/1936493048,v1:172.21.15.88:6837/1936493048] conn(0x55b3b2788a30 0x55b3b2788e10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.690+0000 7f99363d6740 1 --2- 172.21.15.88:0/1053865637 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b277faa0 0x55b3b275df70 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.690+0000 7f99363d6740 1 --2- 172.21.15.88:0/1053865637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3b2760a50 0x55b3b275da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.690+0000 7f9907fff640 0 client.8772 ms_handle_reset on v2:172.21.15.88:6835/1936493048 2024-09-16T01:42:16.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.690+0000 7f99363d6740 1 --2- 172.21.15.88:0/1053865637 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3b275f2c0 0x55b3b277df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.690+0000 7f99363d6740 1 -- 172.21.15.88:0/1053865637 >> 172.21.15.88:0/1053865637 conn(0x55b3b275ca80 msgr2=0x55b3b27a3490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:16.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.690+0000 7f99363d6740 1 -- 172.21.15.88:0/1053865637 shutdown_connections 2024-09-16T01:42:16.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.690+0000 7f99363d6740 1 -- 172.21.15.88:0/1053865637 wait complete. 2024-09-16T01:42:16.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.690+0000 7f99363d6740 20 client.8772 trim_cache size 0 max 16384 2024-09-16T01:42:16.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.708+0000 7f99363d6740 1 Processor -- start 2024-09-16T01:42:16.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.708+0000 7f99363d6740 1 -- start start 2024-09-16T01:42:16.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.708+0000 7f99363d6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3b275f2c0 0x55b3b27a13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.708+0000 7f99363d6740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3b2760a50 0x55b3b27a4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.708+0000 7f99363d6740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b277faa0 0x55b3b275eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.708+0000 7f99363d6740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f992c200960 con 0x55b3b277faa0 2024-09-16T01:42:16.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.708+0000 7f99363d6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f992c1ffcf0 con 0x55b3b275f2c0 2024-09-16T01:42:16.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.708+0000 7f99363d6740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f992c200310 con 0x55b3b2760a50 2024-09-16T01:42:16.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.709+0000 7f9931bd9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b277faa0 0x55b3b275eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.709+0000 7f9931bd9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b277faa0 0x55b3b275eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43814/0 (socket says 172.21.15.88:43814) 2024-09-16T01:42:16.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.709+0000 7f9931bd9640 1 -- 172.21.15.88:0/4045860023 learned_addr learned my addr 172.21.15.88:0/4045860023 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:16.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.709+0000 7f99313d8640 1 --2- 172.21.15.88:0/4045860023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3b275f2c0 0x55b3b27a13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.709+0000 7f9930bd7640 1 --2- 172.21.15.88:0/4045860023 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3b2760a50 0x55b3b27a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.709+0000 7f9931bd9640 1 -- 172.21.15.88:0/4045860023 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3b2760a50 msgr2=0x55b3b27a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:16.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.709+0000 7f9931bd9640 1 --2- 172.21.15.88:0/4045860023 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3b2760a50 0x55b3b27a4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.709+0000 7f9931bd9640 1 -- 172.21.15.88:0/4045860023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3b275f2c0 msgr2=0x55b3b27a13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:16.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.709+0000 7f9931bd9640 1 --2- 172.21.15.88:0/4045860023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3b275f2c0 0x55b3b27a13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.709+0000 7f9931bd9640 1 -- 172.21.15.88:0/4045860023 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3b2786f60 con 0x55b3b277faa0 2024-09-16T01:42:16.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.709+0000 7f99313d8640 1 --2- 172.21.15.88:0/4045860023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3b275f2c0 0x55b3b27a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:16.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.709+0000 7f9931bd9640 1 --2- 172.21.15.88:0/4045860023 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b277faa0 0x55b3b275eb20 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f9924009880 tx=0x7f9924010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d97be0cac1da4b41 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:16.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.709+0000 7f990e7fc640 1 -- 172.21.15.88:0/4045860023 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f992401a030 con 0x55b3b277faa0 2024-09-16T01:42:16.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.709+0000 7f990e7fc640 1 -- 172.21.15.88:0/4045860023 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99240070a0 con 0x55b3b277faa0 2024-09-16T01:42:16.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.709+0000 7f990e7fc640 1 -- 172.21.15.88:0/4045860023 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9924007b60 con 0x55b3b277faa0 2024-09-16T01:42:16.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.710+0000 7f99363d6740 1 -- 172.21.15.88:0/4045860023 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b277faa0 msgr2=0x55b3b275eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:16.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.710+0000 7f99363d6740 1 --2- 172.21.15.88:0/4045860023 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b277faa0 0x55b3b275eb20 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f9924009880 tx=0x7f9924010000 comp rx=0 tx=0).stop 2024-09-16T01:42:16.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.710+0000 7f99363d6740 1 -- 172.21.15.88:0/4045860023 shutdown_connections 2024-09-16T01:42:16.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.710+0000 7f99363d6740 1 --2- 172.21.15.88:0/4045860023 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b277faa0 0x55b3b275eb20 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.710+0000 7f99363d6740 1 --2- 172.21.15.88:0/4045860023 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3b2760a50 0x55b3b27a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.710+0000 7f99363d6740 1 --2- 172.21.15.88:0/4045860023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3b275f2c0 0x55b3b27a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.710+0000 7f99363d6740 1 -- 172.21.15.88:0/4045860023 >> 172.21.15.88:0/4045860023 conn(0x55b3b275ca80 msgr2=0x55b3b277dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:16.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.710+0000 7f99363d6740 1 -- 172.21.15.88:0/4045860023 shutdown_connections 2024-09-16T01:42:16.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.710+0000 7f99363d6740 1 -- 172.21.15.88:0/4045860023 wait complete. 2024-09-16T01:42:16.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.710+0000 7f99363d6740 1 -- start start 2024-09-16T01:42:16.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.710+0000 7f99363d6740 1 Processor -- start 2024-09-16T01:42:16.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.711+0000 7f99363d6740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3b275f2c0 0x55b3b2785010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.711+0000 7f99363d6740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b2760a50 0x55b3b2821e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.711+0000 7f99363d6740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3b277faa0 0x55b3b27c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.711+0000 7f99363d6740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b3b275a850 con 0x55b3b2760a50 2024-09-16T01:42:16.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.711+0000 7f99363d6740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b3b2789990 con 0x55b3b275f2c0 2024-09-16T01:42:16.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.711+0000 7f99363d6740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55b3b27594e0 con 0x55b3b277faa0 2024-09-16T01:42:16.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.711+0000 7f99363d6740 10 client.0 _ll_register_callbacks cb 0x7f9924009b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:42:16.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.711+0000 7f9930bd7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b2760a50 0x55b3b2821e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.711+0000 7f9930bd7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b2760a50 0x55b3b2821e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43826/0 (socket says 172.21.15.88:43826) 2024-09-16T01:42:16.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.711+0000 7f9930bd7640 1 -- 172.21.15.88:0/3890689994 learned_addr learned my addr 172.21.15.88:0/3890689994 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:16.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.712+0000 7f99313d8640 1 --2- 172.21.15.88:0/3890689994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3b275f2c0 0x55b3b2785010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.712+0000 7f9931bd9640 1 --2- 172.21.15.88:0/3890689994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3b277faa0 0x55b3b27c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.712+0000 7f9930bd7640 1 -- 172.21.15.88:0/3890689994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3b277faa0 msgr2=0x55b3b27c73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:16.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.712+0000 7f9930bd7640 1 --2- 172.21.15.88:0/3890689994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3b277faa0 0x55b3b27c73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.712+0000 7f9930bd7640 1 -- 172.21.15.88:0/3890689994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3b275f2c0 msgr2=0x55b3b2785010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:16.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.712+0000 7f9930bd7640 1 --2- 172.21.15.88:0/3890689994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3b275f2c0 0x55b3b2785010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.712+0000 7f9930bd7640 1 -- 172.21.15.88:0/3890689994 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3b27c8200 con 0x55b3b2760a50 2024-09-16T01:42:16.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.712+0000 7f9931bd9640 1 --2- 172.21.15.88:0/3890689994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3b277faa0 0x55b3b27c73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:16.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.712+0000 7f9930bd7640 1 --2- 172.21.15.88:0/3890689994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b2760a50 0x55b3b2821e70 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f991c00ff30 tx=0x7f991c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4316ff0a2929224e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:16.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.712+0000 7f9907fff640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:42:16.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.712+0000 7f9907fff640 1 -- 172.21.15.88:0/3890689994 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991c01b040 con 0x55b3b2760a50 2024-09-16T01:42:16.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.712+0000 7f9907fff640 1 -- 172.21.15.88:0/3890689994 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 7286+0+0 (secure 0 0 0) 0x7f991c013930 con 0x55b3b2760a50 2024-09-16T01:42:16.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.712+0000 7f99363d6740 1 -- 172.21.15.88:0/3890689994 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99240070a0 con 0x55b3b2760a50 2024-09-16T01:42:16.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.712+0000 7f9907fff640 1 -- 172.21.15.88:0/3890689994 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=183) ==== 24+0+0 (secure 0 0 0) 0x7f991c013b30 con 0x55b3b2760a50 2024-09-16T01:42:16.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.712+0000 7f99363d6740 10 client.8778 fetch_fsmap learned FSMap version 183 2024-09-16T01:42:16.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.712+0000 7f99363d6740 1 -- 172.21.15.88:0/3890689994 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=183,monmap=2+}) -- 0x55b3b277fee0 con 0x55b3b2760a50 2024-09-16T01:42:16.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.712+0000 7f9907fff640 1 -- 172.21.15.88:0/3890689994 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99240070a0 con 0x55b3b2760a50 2024-09-16T01:42:16.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.712+0000 7f9907fff640 1 -- 172.21.15.88:0/3890689994 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 183) ==== 3502+0+0 (secure 0 0 0) 0x7f991c021040 con 0x55b3b2760a50 2024-09-16T01:42:16.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.712+0000 7f99363d6740 10 client.8778 fetch_fsmap finished waiting for FSMap version 183 2024-09-16T01:42:16.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.712+0000 7f99363d6740 10 client.8778 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-16T01:42:16.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.712+0000 7f99363d6740 20 client.8778 populate_metadata read hostname 'smithi088' 2024-09-16T01:42:16.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.712+0000 7f99363d6740 1 --2- 172.21.15.88:0/3890689994 >> [v2:172.21.15.88:6835/1936493048,v1:172.21.15.88:6837/1936493048] conn(0x55b3b275e140 0x55b3b2759730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.712+0000 7f99363d6740 4 client.8778 mds_command: new command op to 8679 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:42:16.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.712+0000 7f99363d6740 1 -- 172.21.15.88:0/3890689994 --> [v2:172.21.15.88:6835/1936493048,v1:172.21.15.88:6837/1936493048] -- command(tid 0: {"prefix": "status"}) -- 0x55b3b26c8fa0 con 0x55b3b275e140 2024-09-16T01:42:16.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.713+0000 7f99313d8640 1 --2- 172.21.15.88:0/3890689994 >> [v2:172.21.15.88:6835/1936493048,v1:172.21.15.88:6837/1936493048] conn(0x55b3b275e140 0x55b3b2759730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.713+0000 7f99313d8640 1 --2- 172.21.15.88:0/3890689994 >> [v2:172.21.15.88:6835/1936493048,v1:172.21.15.88:6837/1936493048] conn(0x55b3b275e140 0x55b3b2759730 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17c7f8a0b4089b96 server_cookie=5257889518331521 in_seq=0 out_seq=0 2024-09-16T01:42:16.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.713+0000 7f9907fff640 10 client.8778 ms_handle_connect on v2:172.21.15.88:6835/1936493048 2024-09-16T01:42:16.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.713+0000 7f9907fff640 1 -- 172.21.15.88:0/3890689994 <== mds.0 v2:172.21.15.88:6835/1936493048 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x55b3b26c8fa0 con 0x55b3b275e140 2024-09-16T01:42:16.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.713+0000 7f9907fff640 10 client.8778 handle_command_reply: tid=0 2024-09-16T01:42:16.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.713+0000 7f99363d6740 1 client.8778 shutdown 2024-09-16T01:42:16.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.713+0000 7f99363d6740 1 -- 172.21.15.88:0/3890689994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b2760a50 msgr2=0x55b3b2821e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:16.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.713+0000 7f99363d6740 1 --2- 172.21.15.88:0/3890689994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b2760a50 0x55b3b2821e70 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f991c00ff30 tx=0x7f991c004730 comp rx=0 tx=0).stop 2024-09-16T01:42:16.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.713+0000 7f99363d6740 1 -- 172.21.15.88:0/3890689994 shutdown_connections 2024-09-16T01:42:16.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.713+0000 7f99363d6740 1 --2- 172.21.15.88:0/3890689994 >> [v2:172.21.15.88:6835/1936493048,v1:172.21.15.88:6837/1936493048] conn(0x55b3b275e140 0x55b3b2759730 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.713+0000 7f99363d6740 1 --2- 172.21.15.88:0/3890689994 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b3b277faa0 0x55b3b27c73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.713+0000 7f99363d6740 1 --2- 172.21.15.88:0/3890689994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b3b2760a50 0x55b3b2821e70 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.713+0000 7f9907fff640 0 client.8778 ms_handle_reset on v2:172.21.15.88:6835/1936493048 2024-09-16T01:42:16.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.713+0000 7f99363d6740 1 --2- 172.21.15.88:0/3890689994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3b275f2c0 0x55b3b2785010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.713+0000 7f99363d6740 1 -- 172.21.15.88:0/3890689994 >> 172.21.15.88:0/3890689994 conn(0x55b3b275ca80 msgr2=0x55b3b277dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:16.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.713+0000 7f99363d6740 1 -- 172.21.15.88:0/3890689994 shutdown_connections 2024-09-16T01:42:16.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.713+0000 7f99363d6740 1 -- 172.21.15.88:0/3890689994 wait complete. 2024-09-16T01:42:16.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.713+0000 7f99363d6740 20 client.8778 trim_cache size 0 max 16384 2024-09-16T01:42:16.723 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:42:16.723 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "5ace2c91-46f9-482b-8a89-731fbdc171bf", 2024-09-16T01:42:16.724 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2024-09-16T01:42:16.724 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 8679, 2024-09-16T01:42:16.724 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2024-09-16T01:42:16.724 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2024-09-16T01:42:16.724 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2024-09-16T01:42:16.724 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 7.2744864079999996, 2024-09-16T01:42:16.724 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 183, 2024-09-16T01:42:16.724 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 105, 2024-09-16T01:42:16.724 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 105, 2024-09-16T01:42:16.724 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 10.563380618 2024-09-16T01:42:16.724 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:42:16.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.714+0000 7f993363c640 1 -- 172.21.15.88:0/3357804019 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f991006c2b0 msgr2=0x7f991006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:16.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.714+0000 7f993363c640 1 --2- 172.21.15.88:0/3357804019 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f991006c2b0 0x7f991006e770 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f992c159060 tx=0x7f991c008040 comp rx=0 tx=0).stop 2024-09-16T01:42:16.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.714+0000 7f993363c640 1 -- 172.21.15.88:0/3357804019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f992c06fc50 msgr2=0x7f992c149ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:16.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.714+0000 7f993363c640 1 --2- 172.21.15.88:0/3357804019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f992c06fc50 0x7f992c149ef0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f9928000c00 tx=0x7f99280046c0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.714+0000 7f993363c640 1 -- 172.21.15.88:0/3357804019 shutdown_connections 2024-09-16T01:42:16.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.714+0000 7f993363c640 1 --2- 172.21.15.88:0/3357804019 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f991006c2b0 0x7f991006e770 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.714+0000 7f993363c640 1 --2- 172.21.15.88:0/3357804019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f992c15c2a0 0x7f992c157d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.714+0000 7f993363c640 1 --2- 172.21.15.88:0/3357804019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f992c073b10 0x7f992c0cdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.714+0000 7f993363c640 1 --2- 172.21.15.88:0/3357804019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f992c06fc50 0x7f992c149ef0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.714+0000 7f993363c640 1 -- 172.21.15.88:0/3357804019 >> 172.21.15.88:0/3357804019 conn(0x7f992c0b3070 msgr2=0x7f992c0b4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:16.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.714+0000 7f993363c640 1 -- 172.21.15.88:0/3357804019 shutdown_connections 2024-09-16T01:42:16.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.714+0000 7f993363c640 1 -- 172.21.15.88:0/3357804019 wait complete. 2024-09-16T01:42:16.727 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-09-16T01:42:16.727 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:16.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.824+0000 7f108816d640 1 Processor -- start 2024-09-16T01:42:16.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.824+0000 7f108816d640 1 -- start start 2024-09-16T01:42:16.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.825+0000 7f108816d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f108006bc10 0x7f108006bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.825+0000 7f108816d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f108006c530 con 0x7f10800713c0 2024-09-16T01:42:16.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.825+0000 7f108816d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f108006e7b0 con 0x7f1080071830 2024-09-16T01:42:16.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.825+0000 7f108816d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f108006ea90 con 0x7f108006bc10 2024-09-16T01:42:16.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.825+0000 7f1085f09640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f108006bc10 0x7f108006bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.825+0000 7f1085f09640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f108006bc10 0x7f108006bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43840/0 (socket says 172.21.15.88:43840) 2024-09-16T01:42:16.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.825+0000 7f1085708640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f10800713c0 0x7f10800af510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44942/0 (socket says 172.21.15.88:44942) 2024-09-16T01:42:16.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.825+0000 7f1085f09640 1 -- 172.21.15.88:0/1696619098 learned_addr learned my addr 172.21.15.88:0/1696619098 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:16.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.825+0000 7f1085f09640 1 -- 172.21.15.88:0/1696619098 >> v1:172.21.15.154:6789/0 conn(0x7f1080071830 legacy=0x7f108006da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:16.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.826+0000 7f1085f09640 1 -- 172.21.15.88:0/1696619098 >> v1:172.21.15.88:6789/0 conn(0x7f10800713c0 legacy=0x7f10800af510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:16.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.826+0000 7f1085f09640 1 -- 172.21.15.88:0/1696619098 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10800aea30 con 0x7f108006bc10 2024-09-16T01:42:16.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.826+0000 7f1085f09640 1 --2- 172.21.15.88:0/1696619098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f108006bc10 0x7f108006bff0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f107c009870 tx=0x7f107c02a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85bdc27d436283dc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:16.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.826+0000 7f1084f07640 1 -- 172.21.15.88:0/1696619098 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107c037070 con 0x7f108006bc10 2024-09-16T01:42:16.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.826+0000 7f1084f07640 1 -- 172.21.15.88:0/1696619098 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f107c032440 con 0x7f108006bc10 2024-09-16T01:42:16.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.826+0000 7f1084f07640 1 -- 172.21.15.88:0/1696619098 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107c032810 con 0x7f108006bc10 2024-09-16T01:42:16.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.827+0000 7f108816d640 1 -- 172.21.15.88:0/1696619098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f108006bc10 msgr2=0x7f108006bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:16.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.827+0000 7f108816d640 1 --2- 172.21.15.88:0/1696619098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f108006bc10 0x7f108006bff0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f107c009870 tx=0x7f107c02a2a0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.827+0000 7f108816d640 1 -- 172.21.15.88:0/1696619098 shutdown_connections 2024-09-16T01:42:16.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.827+0000 7f108816d640 1 --2- 172.21.15.88:0/1696619098 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f108006bc10 0x7f108006bff0 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.827+0000 7f108816d640 1 -- 172.21.15.88:0/1696619098 >> 172.21.15.88:0/1696619098 conn(0x7f10800b2430 msgr2=0x7f10800b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:16.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.827+0000 7f108816d640 1 -- 172.21.15.88:0/1696619098 shutdown_connections 2024-09-16T01:42:16.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.827+0000 7f108816d640 1 -- 172.21.15.88:0/1696619098 wait complete. 2024-09-16T01:42:16.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.828+0000 7f108816d640 1 Processor -- start 2024-09-16T01:42:16.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.828+0000 7f108816d640 1 -- start start 2024-09-16T01:42:16.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.829+0000 7f108816d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f108006bc10 0x7f10801559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.829+0000 7f108816d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10800713c0 0x7f1080153180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.829+0000 7f108816d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1080071830 0x7f1080152020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.829+0000 7f108816d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f108015ea10 con 0x7f108006bc10 2024-09-16T01:42:16.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.829+0000 7f108816d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f10800aeff0 con 0x7f1080071830 2024-09-16T01:42:16.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.829+0000 7f108816d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f10801536c0 con 0x7f10800713c0 2024-09-16T01:42:16.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.829+0000 7f1085f09640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f108006bc10 0x7f10801559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.829+0000 7f1085f09640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f108006bc10 0x7f10801559e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43842/0 (socket says 172.21.15.88:43842) 2024-09-16T01:42:16.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.829+0000 7f1085f09640 1 -- 172.21.15.88:0/113420941 learned_addr learned my addr 172.21.15.88:0/113420941 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:16.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.829+0000 7f1085f09640 1 -- 172.21.15.88:0/113420941 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10800713c0 msgr2=0x7f1080153180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:16.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.829+0000 7f1085f09640 1 --2- 172.21.15.88:0/113420941 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10800713c0 0x7f1080153180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.829+0000 7f1085f09640 1 -- 172.21.15.88:0/113420941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1080071830 msgr2=0x7f1080152020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:16.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.829+0000 7f1085f09640 1 --2- 172.21.15.88:0/113420941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1080071830 0x7f1080152020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:16.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.829+0000 7f1085f09640 1 -- 172.21.15.88:0/113420941 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1080153920 con 0x7f108006bc10 2024-09-16T01:42:16.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.829+0000 7f1085f09640 1 --2- 172.21.15.88:0/113420941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f108006bc10 0x7f10801559e0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f107c009870 tx=0x7f107c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:16.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.830+0000 7f10767fc640 1 -- 172.21.15.88:0/113420941 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107c041070 con 0x7f108006bc10 2024-09-16T01:42:16.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.830+0000 7f10767fc640 1 -- 172.21.15.88:0/113420941 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f107c03c870 con 0x7f108006bc10 2024-09-16T01:42:16.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.830+0000 7f10767fc640 1 -- 172.21.15.88:0/113420941 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107c037040 con 0x7f108006bc10 2024-09-16T01:42:16.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.830+0000 7f108816d640 1 -- 172.21.15.88:0/113420941 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1080153c00 con 0x7f108006bc10 2024-09-16T01:42:16.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.830+0000 7f108816d640 1 -- 172.21.15.88:0/113420941 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10801578b0 con 0x7f108006bc10 2024-09-16T01:42:16.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.831+0000 7f108816d640 1 -- 172.21.15.88:0/113420941 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10800b5580 con 0x7f108006bc10 2024-09-16T01:42:16.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.831+0000 7f10767fc640 1 -- 172.21.15.88:0/113420941 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f107c044070 con 0x7f108006bc10 2024-09-16T01:42:16.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.831+0000 7f10767fc640 1 --2- 172.21.15.88:0/113420941 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f106406c2e0 0x7f106406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:16.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.832+0000 7f10767fc640 1 -- 172.21.15.88:0/113420941 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7286+0+0 (secure 0 0 0) 0x7f107c0ac7f0 con 0x7f108006bc10 2024-09-16T01:42:16.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.832+0000 7f1085708640 1 --2- 172.21.15.88:0/113420941 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f106406c2e0 0x7f106406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:16.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.832+0000 7f1085708640 1 --2- 172.21.15.88:0/113420941 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f106406c2e0 0x7f106406e7a0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f1070000f60 tx=0x7f1070007a00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:16.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:16.836+0000 7f10767fc640 1 -- 172.21.15.88:0/113420941 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f10800b5580 con 0x7f108006bc10 2024-09-16T01:42:17.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.035+0000 7f108816d640 1 -- 172.21.15.88:0/113420941 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1080153fa0 con 0x7f108006bc10 2024-09-16T01:42:17.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.036+0000 7f10767fc640 1 -- 172.21.15.88:0/113420941 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) ==== 95+0+5234 (secure 0 0 0) 0x7f107c07aae0 con 0x7f108006bc10 2024-09-16T01:42:17.037 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:17.038 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":183,"btime":"2024-09-16T01:42:12:447758+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8736,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3374372758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3374372758},{"type":"v1","addr":"172.21.15.88:6836","nonce":3374372758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":181}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:12.447749+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":105,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8679},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8679":{"gid":8679,"name":"c","rank":0,"incarnation":180,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6837/1936493048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":1936493048},{"type":"v1","addr":"172.21.15.88:6837","nonce":1936493048}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8679,"qdb_cluster":[8679]},"id":6}]} 2024-09-16T01:42:17.038 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 183 2024-09-16T01:42:17.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.038+0000 7f108816d640 1 -- 172.21.15.88:0/113420941 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f106406c2e0 msgr2=0x7f106406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:17.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.038+0000 7f108816d640 1 --2- 172.21.15.88:0/113420941 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f106406c2e0 0x7f106406e7a0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f1070000f60 tx=0x7f1070007a00 comp rx=0 tx=0).stop 2024-09-16T01:42:17.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.038+0000 7f108816d640 1 -- 172.21.15.88:0/113420941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f108006bc10 msgr2=0x7f10801559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:17.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.038+0000 7f108816d640 1 --2- 172.21.15.88:0/113420941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f108006bc10 0x7f10801559e0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f107c009870 tx=0x7f107c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:42:17.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.038+0000 7f108816d640 1 -- 172.21.15.88:0/113420941 shutdown_connections 2024-09-16T01:42:17.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.038+0000 7f108816d640 1 --2- 172.21.15.88:0/113420941 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f106406c2e0 0x7f106406e7a0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:17.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.038+0000 7f108816d640 1 --2- 172.21.15.88:0/113420941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1080071830 0x7f1080152020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:17.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.038+0000 7f108816d640 1 --2- 172.21.15.88:0/113420941 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10800713c0 0x7f1080153180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:17.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.038+0000 7f108816d640 1 --2- 172.21.15.88:0/113420941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f108006bc10 0x7f10801559e0 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:17.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.038+0000 7f108816d640 1 -- 172.21.15.88:0/113420941 >> 172.21.15.88:0/113420941 conn(0x7f10800b2430 msgr2=0x7f10800b0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:17.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.038+0000 7f108816d640 1 -- 172.21.15.88:0/113420941 shutdown_connections 2024-09-16T01:42:17.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.038+0000 7f108816d640 1 -- 172.21.15.88:0/113420941 wait complete. 2024-09-16T01:42:17.051 INFO:tasks.cephfs.test_damage:Mutation 'Corrupt 500.00000000' did not prevent MDS startup, attempting ls... 2024-09-16T01:42:17.051 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:42:17.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.148+0000 7f2b6c713640 1 Processor -- start 2024-09-16T01:42:17.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.148+0000 7f2b6c713640 1 -- start start 2024-09-16T01:42:17.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.149+0000 7f2b6c713640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2b6415e950 0x7f2b6406ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:17.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.149+0000 7f2b6c713640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b64072050 0x7f2b6406d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:17.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.149+0000 7f2b6c713640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b6406faf0 con 0x7f2b640735b0 2024-09-16T01:42:17.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.149+0000 7f2b6c713640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2b6406fdd0 con 0x7f2b6415e950 2024-09-16T01:42:17.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.149+0000 7f2b6c713640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2b640700b0 con 0x7f2b64072050 2024-09-16T01:42:17.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.149+0000 7f2b69cae640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f2b640735b0 0x7f2b6406bc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44956/0 (socket says 172.21.15.88:44956) 2024-09-16T01:42:17.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.149+0000 7f2b69cae640 1 -- 172.21.15.88:0/2003617106 learned_addr learned my addr 172.21.15.88:0/2003617106 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:17.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.149+0000 7f2b694ad640 1 -- 172.21.15.88:0/2003617106 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2884999927 0 0) 0x7f2b6406faf0 con 0x7f2b640735b0 2024-09-16T01:42:17.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.149+0000 7f2b6a4af640 1 --2- 172.21.15.88:0/2003617106 >> v2:172.21.15.154:3300/0 conn(0x7f2b6415e950 0x7f2b6406ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:17.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.149+0000 7f2b694ad640 1 -- 172.21.15.88:0/2003617106 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b4c003520 con 0x7f2b640735b0 2024-09-16T01:42:17.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.149+0000 7f2b6acb0640 1 --2- 172.21.15.88:0/2003617106 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b64072050 0x7f2b6406d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:17.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.149+0000 7f2b694ad640 1 -- 172.21.15.88:0/2003617106 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1270709807 0 0) 0x7f2b4c003520 con 0x7f2b640735b0 2024-09-16T01:42:17.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.149+0000 7f2b694ad640 1 -- 172.21.15.88:0/2003617106 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b64072050 msgr2=0x7f2b6406d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:17.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.149+0000 7f2b694ad640 1 --2- 172.21.15.88:0/2003617106 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b64072050 0x7f2b6406d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:17.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.149+0000 7f2b694ad640 1 -- 172.21.15.88:0/2003617106 >> v2:172.21.15.154:3300/0 conn(0x7f2b6415e950 msgr2=0x7f2b6406ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:17.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.149+0000 7f2b694ad640 1 --2- 172.21.15.88:0/2003617106 >> v2:172.21.15.154:3300/0 conn(0x7f2b6415e950 0x7f2b6406ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:17.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.149+0000 7f2b694ad640 1 -- 172.21.15.88:0/2003617106 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b64070390 con 0x7f2b640735b0 2024-09-16T01:42:17.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.149+0000 7f2b694ad640 1 -- 172.21.15.88:0/2003617106 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f2b58001670 con 0x7f2b640735b0 2024-09-16T01:42:17.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.149+0000 7f2b694ad640 1 -- 172.21.15.88:0/2003617106 >> v1:172.21.15.88:6789/0 conn(0x7f2b640735b0 legacy=0x7f2b6406bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:17.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.149+0000 7f2b694ad640 1 --2- 172.21.15.88:0/2003617106 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b4c0039b0 0x7f2b4c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:17.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.149+0000 7f2b694ad640 1 --2- 172.21.15.88:0/2003617106 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b64072050 0x7f2b4c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:17.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.149+0000 7f2b694ad640 1 --2- 172.21.15.88:0/2003617106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b4c004d40 0x7f2b4c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:17.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.149+0000 7f2b694ad640 1 -- 172.21.15.88:0/2003617106 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6406fdd0 con 0x7f2b4c004d40 2024-09-16T01:42:17.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.149+0000 7f2b694ad640 1 -- 172.21.15.88:0/2003617106 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2b640700b0 con 0x7f2b4c0039b0 2024-09-16T01:42:17.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.149+0000 7f2b694ad640 1 -- 172.21.15.88:0/2003617106 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2b4c00b9c0 con 0x7f2b64072050 2024-09-16T01:42:17.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.150+0000 7f2b6acb0640 1 --2- 172.21.15.88:0/2003617106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b4c004d40 0x7f2b4c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:17.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.150+0000 7f2b69cae640 1 --2- 172.21.15.88:0/2003617106 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b64072050 0x7f2b4c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:17.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.150+0000 7f2b6a4af640 1 --2- 172.21.15.88:0/2003617106 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b4c0039b0 0x7f2b4c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:17.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.150+0000 7f2b6acb0640 1 -- 172.21.15.88:0/2003617106 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b64072050 msgr2=0x7f2b4c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:17.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.150+0000 7f2b6acb0640 1 --2- 172.21.15.88:0/2003617106 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b64072050 0x7f2b4c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:17.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.150+0000 7f2b6acb0640 1 -- 172.21.15.88:0/2003617106 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b4c0039b0 msgr2=0x7f2b4c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:17.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.150+0000 7f2b6acb0640 1 --2- 172.21.15.88:0/2003617106 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b4c0039b0 0x7f2b4c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:17.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.150+0000 7f2b6acb0640 1 -- 172.21.15.88:0/2003617106 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b4c003520 con 0x7f2b4c004d40 2024-09-16T01:42:17.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.150+0000 7f2b6acb0640 1 --2- 172.21.15.88:0/2003617106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b4c004d40 0x7f2b4c0051c0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f2b5400e170 tx=0x7f2b4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ec55f21c4e6450 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:17.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.151+0000 7f2b694ad640 1 -- 172.21.15.88:0/2003617106 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b54034020 con 0x7f2b4c004d40 2024-09-16T01:42:17.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.151+0000 7f2b694ad640 1 -- 172.21.15.88:0/2003617106 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b540305f0 con 0x7f2b4c004d40 2024-09-16T01:42:17.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.151+0000 7f2b694ad640 1 -- 172.21.15.88:0/2003617106 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b54009c10 con 0x7f2b4c004d40 2024-09-16T01:42:17.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.151+0000 7f2b6c713640 1 -- 172.21.15.88:0/2003617106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b4c004d40 msgr2=0x7f2b4c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:17.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.151+0000 7f2b6c713640 1 --2- 172.21.15.88:0/2003617106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b4c004d40 0x7f2b4c0051c0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f2b5400e170 tx=0x7f2b4c005830 comp rx=0 tx=0).stop 2024-09-16T01:42:17.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.151+0000 7f2b6acb0640 1 -- 172.21.15.88:0/2003617106 reap_dead start 2024-09-16T01:42:17.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.151+0000 7f2b6c713640 1 -- 172.21.15.88:0/2003617106 shutdown_connections 2024-09-16T01:42:17.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.151+0000 7f2b6c713640 1 -- 172.21.15.88:0/2003617106 >> 172.21.15.88:0/2003617106 conn(0x7f2b640d14d0 msgr2=0x7f2b640d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:17.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.151+0000 7f2b6c713640 1 -- 172.21.15.88:0/2003617106 shutdown_connections 2024-09-16T01:42:17.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.151+0000 7f2b6c713640 1 -- 172.21.15.88:0/2003617106 wait complete. 2024-09-16T01:42:17.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.152+0000 7f2b6c713640 1 Processor -- start 2024-09-16T01:42:17.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.152+0000 7f2b6c713640 1 -- start start 2024-09-16T01:42:17.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.153+0000 7f2b6c713640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4c004d40 0x7f2b64157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:17.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.153+0000 7f2b6c713640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b640cb0a0 0x7f2b6414e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:17.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.153+0000 7f2b6c713640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b640cb520 0x7f2b6414e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:17.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.153+0000 7f2b6c713640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2b640d0230 con 0x7f2b640cb0a0 2024-09-16T01:42:17.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.153+0000 7f2b6c713640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2b640d00b0 con 0x7f2b640cb520 2024-09-16T01:42:17.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.153+0000 7f2b6c713640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2b640d03b0 con 0x7f2b4c004d40 2024-09-16T01:42:17.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.153+0000 7f2b69cae640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b640cb0a0 0x7f2b6414e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:17.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.153+0000 7f2b69cae640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b640cb0a0 0x7f2b6414e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43862/0 (socket says 172.21.15.88:43862) 2024-09-16T01:42:17.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.153+0000 7f2b69cae640 1 -- 172.21.15.88:0/3485261817 learned_addr learned my addr 172.21.15.88:0/3485261817 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:17.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.153+0000 7f2b6acb0640 1 --2- 172.21.15.88:0/3485261817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b640cb520 0x7f2b6414e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:17.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.153+0000 7f2b6a4af640 1 --2- 172.21.15.88:0/3485261817 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4c004d40 0x7f2b64157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:17.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.153+0000 7f2b69cae640 1 -- 172.21.15.88:0/3485261817 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4c004d40 msgr2=0x7f2b64157580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:17.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.153+0000 7f2b69cae640 1 --2- 172.21.15.88:0/3485261817 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4c004d40 0x7f2b64157580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:17.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.153+0000 7f2b69cae640 1 -- 172.21.15.88:0/3485261817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b640cb520 msgr2=0x7f2b6414e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:17.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.153+0000 7f2b69cae640 1 --2- 172.21.15.88:0/3485261817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b640cb520 0x7f2b6414e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:17.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.153+0000 7f2b69cae640 1 -- 172.21.15.88:0/3485261817 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b64154bb0 con 0x7f2b640cb0a0 2024-09-16T01:42:17.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.153+0000 7f2b6acb0640 1 --2- 172.21.15.88:0/3485261817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b640cb520 0x7f2b6414e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:17.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.153+0000 7f2b6a4af640 1 --2- 172.21.15.88:0/3485261817 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4c004d40 0x7f2b64157580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:17.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.154+0000 7f2b69cae640 1 --2- 172.21.15.88:0/3485261817 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b640cb0a0 0x7f2b6414e1e0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f2b58000fb0 tx=0x7f2b580077a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:17.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.154+0000 7f2b62ffd640 1 -- 172.21.15.88:0/3485261817 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5800f990 con 0x7f2b640cb0a0 2024-09-16T01:42:17.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.154+0000 7f2b62ffd640 1 -- 172.21.15.88:0/3485261817 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b58018070 con 0x7f2b640cb0a0 2024-09-16T01:42:17.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.154+0000 7f2b62ffd640 1 -- 172.21.15.88:0/3485261817 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5800fdb0 con 0x7f2b640cb0a0 2024-09-16T01:42:17.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.154+0000 7f2b6c713640 1 -- 172.21.15.88:0/3485261817 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b64154e90 con 0x7f2b640cb0a0 2024-09-16T01:42:17.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.154+0000 7f2b6c713640 1 -- 172.21.15.88:0/3485261817 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b64155450 con 0x7f2b640cb0a0 2024-09-16T01:42:17.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.155+0000 7f2b6c713640 1 -- 172.21.15.88:0/3485261817 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b6406ce30 con 0x7f2b640cb0a0 2024-09-16T01:42:17.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.155+0000 7f2b62ffd640 1 -- 172.21.15.88:0/3485261817 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2b5801d460 con 0x7f2b640cb0a0 2024-09-16T01:42:17.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.156+0000 7f2b62ffd640 1 --2- 172.21.15.88:0/3485261817 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b5006c2e0 0x7f2b5006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:17.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.156+0000 7f2b62ffd640 1 -- 172.21.15.88:0/3485261817 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7286+0+0 (secure 0 0 0) 0x7f2b5808bb90 con 0x7f2b640cb0a0 2024-09-16T01:42:17.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.156+0000 7f2b6a4af640 1 --2- 172.21.15.88:0/3485261817 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b5006c2e0 0x7f2b5006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:17.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.156+0000 7f2b6a4af640 1 --2- 172.21.15.88:0/3485261817 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b5006c2e0 0x7f2b5006e7a0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c009bc0 tx=0x7f2b5c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:17.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.160+0000 7f2b62ffd640 1 -- 172.21.15.88:0/3485261817 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2b58059d60 con 0x7f2b640cb0a0 2024-09-16T01:42:17.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.360+0000 7f2b6c713640 1 -- 172.21.15.88:0/3485261817 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2b64155620 con 0x7f2b640cb0a0 2024-09-16T01:42:17.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.360+0000 7f2b62ffd640 1 -- 172.21.15.88:0/3485261817 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v183) ==== 53+0+73 (secure 0 0 0) 0x7f2b58059660 con 0x7f2b640cb0a0 2024-09-16T01:42:17.362 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:42:17.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.362+0000 7f2b6c713640 1 -- 172.21.15.88:0/3485261817 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b5006c2e0 msgr2=0x7f2b5006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:17.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.362+0000 7f2b6c713640 1 --2- 172.21.15.88:0/3485261817 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b5006c2e0 0x7f2b5006e7a0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c009bc0 tx=0x7f2b5c008040 comp rx=0 tx=0).stop 2024-09-16T01:42:17.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.362+0000 7f2b6c713640 1 -- 172.21.15.88:0/3485261817 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b640cb0a0 msgr2=0x7f2b6414e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:17.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.362+0000 7f2b6c713640 1 --2- 172.21.15.88:0/3485261817 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b640cb0a0 0x7f2b6414e1e0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f2b58000fb0 tx=0x7f2b580077a0 comp rx=0 tx=0).stop 2024-09-16T01:42:17.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.362+0000 7f2b6c713640 1 -- 172.21.15.88:0/3485261817 shutdown_connections 2024-09-16T01:42:17.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.362+0000 7f2b6c713640 1 --2- 172.21.15.88:0/3485261817 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b5006c2e0 0x7f2b5006e7a0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:17.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.362+0000 7f2b6c713640 1 --2- 172.21.15.88:0/3485261817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b640cb520 0x7f2b6414e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:17.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.362+0000 7f2b6c713640 1 --2- 172.21.15.88:0/3485261817 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b640cb0a0 0x7f2b6414e1e0 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:17.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.362+0000 7f2b6c713640 1 --2- 172.21.15.88:0/3485261817 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4c004d40 0x7f2b64157580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:17.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.362+0000 7f2b6c713640 1 -- 172.21.15.88:0/3485261817 >> 172.21.15.88:0/3485261817 conn(0x7f2b640d14d0 msgr2=0x7f2b640b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:17.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.362+0000 7f2b6c713640 1 -- 172.21.15.88:0/3485261817 shutdown_connections 2024-09-16T01:42:17.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17.362+0000 7f2b6c713640 1 -- 172.21.15.88:0/3485261817 wait complete. 2024-09-16T01:42:17.375 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-16T01:42:17.375 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:42:17.375 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2024-09-16T01:42:17.375 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T01:42:17.375 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T01:42:17.375 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-16T01:42:17.375 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-16T01:42:17.375 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:42:17.375 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:42:17.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:17.375 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2024-09-16T01:42:17.391 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:42:17.391 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:42:17.391 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:42:17.391 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:42:17.391 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-09-16T01:42:17.392 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:42:17.392 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:42:17.392 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:42:17.392 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2024-09-16T01:42:17.392 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:42:17.392 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:42:17.392 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2024-09-16T01:42:17.392 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:42:17.392 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2024-09-16T01:42:17.392 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:42:17.392 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2024-09-16T01:42:17.393 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:42:17.393 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42287sec preferred_lft 42287sec 2024-09-16T01:42:17.393 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link noprefixroute 2024-09-16T01:42:17.393 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:42:17.393 INFO:teuthology.orchestra.run.smithi088.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:42:17.393 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff 2024-09-16T01:42:17.393 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T01:42:17.393 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:42:17.393 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::e486:6eff:fe81:a1c/64 scope link 2024-09-16T01:42:17.393 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:42:17.393 INFO:teuthology.orchestra.run.smithi088.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:42:17.393 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 12:6f:29:be:95:48 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:42:17.393 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::106f:29ff:febe:9548/64 scope link 2024-09-16T01:42:17.394 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:42:17.394 INFO:teuthology.orchestra.run.smithi088.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:42:17.394 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T01:42:17.394 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::c63:b4ff:feb3:ad9e/64 scope link 2024-09-16T01:42:17.394 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:42:17.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:17.395 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:42:17.454 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-09-16T01:42:17.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:17.456 DEBUG:teuthology.orchestra.run.smithi088:> route 2024-09-16T01:42:17.512 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2024-09-16T01:42:17.512 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:42:17.512 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:42:17.513 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:42:17.513 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:42:17.513 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:42:17.513 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:42:17.513 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:42:17.513 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:42:17.513 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:42:17.513 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:42:17.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:42:17.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:42:17.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:17.638 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:42:17.654 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T01:42:17.654 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T01:42:17.654 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:42:17.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:17.654 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2024-09-16T01:42:17.719 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:42:17.719 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T01:42:17.775 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:42:17.775 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:42:17.775 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:42:17.830 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-16T01:42:17.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:17.830 DEBUG:teuthology.orchestra.run.smithi088:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:42:17.913 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:42:17.913 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:42:17.913 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:42:17.922 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:42:17.922 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:42:17.922 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:42:17.922 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:42:17.922 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:42:17.932 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:42:17.932 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:42:17.932 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:42:17.932 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:42:17.932 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:42:17.932 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790" 2024-09-16T01:42:17.932 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2024-09-16T01:42:17.932 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5ace2c91-46f9-482b-8a89-731fbdc171bf" will pass to kernel 2024-09-16T01:42:17.936 INFO:teuthology.orchestra.run.smithi088.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-16T01:42:17.936 INFO:teuthology.orchestra.run.smithi088.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:42:17.937 INFO:teuthology.orchestra.run.smithi088.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:42:17.937 INFO:teuthology.orchestra.run.smithi088.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:42:17.937 INFO:teuthology.orchestra.run.smithi088.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:42:17.937 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:42:17.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:17.937 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:42:17.999 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:42:18.003 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:42:18.045 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:42:18.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:42:18.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:42:18.101 INFO:teuthology.orchestra.run.smithi088.stdout:client8796 2024-09-16T01:42:18.102 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:42:18.102 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:42:18.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:18.158 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R .) 2024-09-16T01:42:18.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:18.201 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir foo) 2024-09-16T01:42:18.215 INFO:teuthology.orchestra.run.smithi088.stdout:.: 2024-09-16T01:42:18.215 INFO:teuthology.orchestra.run.smithi088.stdout:subdir 2024-09-16T01:42:18.215 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:18.215 INFO:teuthology.orchestra.run.smithi088.stdout:./subdir: 2024-09-16T01:42:18.215 INFO:teuthology.orchestra.run.smithi088.stdout:sixmegs 2024-09-16T01:42:18.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:42:18.229 INFO:teuthology.orchestra.run.smithi088.stderr:mkdir: cannot create directory ‘foo’: Read-only file system 2024-09-16T01:42:18.229 INFO:tasks.cephfs.test_damage:mkdir: cannot create directory ‘foo’: Read-only file system 2024-09-16T01:42:18.229 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt mds_snaptable' 2024-09-16T01:42:18.230 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:42:18.230 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:42:18.287 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:42:18.287 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:42:18.343 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:42:18.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.481+0000 7f07b26d0640 1 Processor -- start 2024-09-16T01:42:18.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.481+0000 7f07b26d0640 1 -- start start 2024-09-16T01:42:18.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.482+0000 7f07b26d0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f07ac0ba440 0x7f07ac0b82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:18.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.482+0000 7f07b26d0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07ac15ed20 con 0x7f07ac0b6850 2024-09-16T01:42:18.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.482+0000 7f07b26d0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07ac0d32e0 con 0x7f07ac15e940 2024-09-16T01:42:18.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.482+0000 7f07b26d0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f07ac0d3ef0 con 0x7f07ac0ba440 2024-09-16T01:42:18.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.482+0000 7f07ab7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f07ac0b6850 0x7f07ac0b7ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44966/0 (socket says 172.21.15.88:44966) 2024-09-16T01:42:18.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.482+0000 7f07ab7fe640 1 -- 172.21.15.88:0/1947992965 learned_addr learned my addr 172.21.15.88:0/1947992965 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:18.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.482+0000 7f07b0c6d640 1 --2- 172.21.15.88:0/1947992965 >> v2:172.21.15.154:3300/0 conn(0x7f07ac0ba440 0x7f07ac0b82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:18.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.482+0000 7f07aaffd640 1 -- 172.21.15.88:0/1947992965 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 155644920 0 0) 0x7f07ac15ed20 con 0x7f07ac0b6850 2024-09-16T01:42:18.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.482+0000 7f07aaffd640 1 -- 172.21.15.88:0/1947992965 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0794003520 con 0x7f07ac0b6850 2024-09-16T01:42:18.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.482+0000 7f07aaffd640 1 -- 172.21.15.88:0/1947992965 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3990734556 0 0) 0x7f0794003520 con 0x7f07ac0b6850 2024-09-16T01:42:18.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.482+0000 7f07aaffd640 1 -- 172.21.15.88:0/1947992965 >> v2:172.21.15.154:3300/0 conn(0x7f07ac0ba440 msgr2=0x7f07ac0b82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:18.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.482+0000 7f07aaffd640 1 --2- 172.21.15.88:0/1947992965 >> v2:172.21.15.154:3300/0 conn(0x7f07ac0ba440 0x7f07ac0b82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:18.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.482+0000 7f07aaffd640 1 -- 172.21.15.88:0/1947992965 >> v1:172.21.15.154:6789/0 conn(0x7f07ac15e940 legacy=0x7f07ac0b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:18.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.482+0000 7f07aaffd640 1 -- 172.21.15.88:0/1947992965 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07ac0d35c0 con 0x7f07ac0b6850 2024-09-16T01:42:18.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.482+0000 7f07aaffd640 1 -- 172.21.15.88:0/1947992965 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f07a0001670 con 0x7f07ac0b6850 2024-09-16T01:42:18.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.482+0000 7f07aaffd640 1 -- 172.21.15.88:0/1947992965 >> v1:172.21.15.88:6789/0 conn(0x7f07ac0b6850 legacy=0x7f07ac0b7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:18.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.482+0000 7f07aaffd640 1 --2- 172.21.15.88:0/1947992965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07940039b0 0x7f0794003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:18.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.482+0000 7f07aaffd640 1 --2- 172.21.15.88:0/1947992965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0794004800 0x7f0794004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:18.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.482+0000 7f07aaffd640 1 --2- 172.21.15.88:0/1947992965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07940052e0 0x7f079400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:18.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.482+0000 7f07aaffd640 1 -- 172.21.15.88:0/1947992965 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac0d3ef0 con 0x7f0794004800 2024-09-16T01:42:18.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.482+0000 7f07aaffd640 1 -- 172.21.15.88:0/1947992965 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f079400c1b0 con 0x7f07940039b0 2024-09-16T01:42:18.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.482+0000 7f07aaffd640 1 -- 172.21.15.88:0/1947992965 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f079400c3b0 con 0x7f07940052e0 2024-09-16T01:42:18.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.483+0000 7f07ab7fe640 1 --2- 172.21.15.88:0/1947992965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0794004800 0x7f0794004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:18.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.483+0000 7f07abfff640 1 --2- 172.21.15.88:0/1947992965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07940039b0 0x7f0794003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:18.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.483+0000 7f07b0c6d640 1 --2- 172.21.15.88:0/1947992965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07940052e0 0x7f079400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:18.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.483+0000 7f07ab7fe640 1 -- 172.21.15.88:0/1947992965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07940052e0 msgr2=0x7f079400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:18.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.483+0000 7f07ab7fe640 1 --2- 172.21.15.88:0/1947992965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07940052e0 0x7f079400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:18.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.483+0000 7f07ab7fe640 1 -- 172.21.15.88:0/1947992965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07940039b0 msgr2=0x7f0794003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:18.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.483+0000 7f07ab7fe640 1 --2- 172.21.15.88:0/1947992965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07940039b0 0x7f0794003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:18.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.483+0000 7f07ab7fe640 1 -- 172.21.15.88:0/1947992965 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0794003520 con 0x7f0794004800 2024-09-16T01:42:18.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.483+0000 7f07b0c6d640 1 --2- 172.21.15.88:0/1947992965 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07940052e0 0x7f079400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:18.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.483+0000 7f07b0c6d640 1 -- 172.21.15.88:0/1947992965 reap_dead start 2024-09-16T01:42:18.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.484+0000 7f07ab7fe640 1 --2- 172.21.15.88:0/1947992965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0794004800 0x7f0794004c80 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f07a000c210 tx=0x7f0794005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=178698362ccc3d0a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:18.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.484+0000 7f07aaffd640 1 -- 172.21.15.88:0/1947992965 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a0007cf0 con 0x7f0794004800 2024-09-16T01:42:18.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.484+0000 7f07aaffd640 1 -- 172.21.15.88:0/1947992965 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07a0005af0 con 0x7f0794004800 2024-09-16T01:42:18.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.484+0000 7f07aaffd640 1 -- 172.21.15.88:0/1947992965 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a0007070 con 0x7f0794004800 2024-09-16T01:42:18.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.484+0000 7f07b26d0640 1 -- 172.21.15.88:0/1947992965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0794004800 msgr2=0x7f0794004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:18.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.484+0000 7f07b26d0640 1 --2- 172.21.15.88:0/1947992965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0794004800 0x7f0794004c80 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f07a000c210 tx=0x7f0794005830 comp rx=0 tx=0).stop 2024-09-16T01:42:18.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.484+0000 7f07b26d0640 1 -- 172.21.15.88:0/1947992965 shutdown_connections 2024-09-16T01:42:18.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.484+0000 7f07b26d0640 1 --2- 172.21.15.88:0/1947992965 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0794004800 0x7f0794004c80 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:18.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.484+0000 7f07b26d0640 1 -- 172.21.15.88:0/1947992965 >> 172.21.15.88:0/1947992965 conn(0x7f07ac0d1140 msgr2=0x7f07ac0d1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:18.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.484+0000 7f07b26d0640 1 -- 172.21.15.88:0/1947992965 shutdown_connections 2024-09-16T01:42:18.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.484+0000 7f07b26d0640 1 -- 172.21.15.88:0/1947992965 wait complete. 2024-09-16T01:42:18.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.485+0000 7f07b26d0640 1 Processor -- start 2024-09-16T01:42:18.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.486+0000 7f07b26d0640 1 -- start start 2024-09-16T01:42:18.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.486+0000 7f07b26d0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0794004800 0x7f07ac1f7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:18.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.486+0000 7f07b26d0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07ac160e40 0x7f07ac15f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:18.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.486+0000 7f07b26d0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07ac14a1d0 0x7f07ac14a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:18.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.486+0000 7f07b26d0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac0b7510 con 0x7f07ac14a1d0 2024-09-16T01:42:18.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.486+0000 7f07b26d0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac0d3b80 con 0x7f0794004800 2024-09-16T01:42:18.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.486+0000 7f07b26d0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f07ac0d2290 con 0x7f07ac160e40 2024-09-16T01:42:18.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.486+0000 7f07b0c6d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07ac14a1d0 0x7f07ac14a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:18.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.486+0000 7f07b0c6d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07ac14a1d0 0x7f07ac14a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43890/0 (socket says 172.21.15.88:43890) 2024-09-16T01:42:18.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.486+0000 7f07b0c6d640 1 -- 172.21.15.88:0/947625354 learned_addr learned my addr 172.21.15.88:0/947625354 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:18.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.487+0000 7f07abfff640 1 --2- 172.21.15.88:0/947625354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0794004800 0x7f07ac1f7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:18.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.487+0000 7f07ab7fe640 1 --2- 172.21.15.88:0/947625354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07ac160e40 0x7f07ac15f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:18.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.487+0000 7f07b0c6d640 1 -- 172.21.15.88:0/947625354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07ac160e40 msgr2=0x7f07ac15f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:18.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.487+0000 7f07b0c6d640 1 --2- 172.21.15.88:0/947625354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07ac160e40 0x7f07ac15f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:18.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.487+0000 7f07b0c6d640 1 -- 172.21.15.88:0/947625354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0794004800 msgr2=0x7f07ac1f7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:18.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.487+0000 7f07b0c6d640 1 --2- 172.21.15.88:0/947625354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0794004800 0x7f07ac1f7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:18.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.487+0000 7f07b0c6d640 1 -- 172.21.15.88:0/947625354 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07ac1513d0 con 0x7f07ac14a1d0 2024-09-16T01:42:18.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.487+0000 7f07ab7fe640 1 --2- 172.21.15.88:0/947625354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07ac160e40 0x7f07ac15f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:18.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.487+0000 7f07abfff640 1 --2- 172.21.15.88:0/947625354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0794004800 0x7f07ac1f7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:18.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.487+0000 7f07b0c6d640 1 --2- 172.21.15.88:0/947625354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07ac14a1d0 0x7f07ac14a660 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f07ac0ba0a0 tx=0x7f07a4010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:18.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.487+0000 7f07a8ff9640 1 -- 172.21.15.88:0/947625354 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a4003990 con 0x7f07ac14a1d0 2024-09-16T01:42:18.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.487+0000 7f07a8ff9640 1 -- 172.21.15.88:0/947625354 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07a4003b50 con 0x7f07ac14a1d0 2024-09-16T01:42:18.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.487+0000 7f07a8ff9640 1 -- 172.21.15.88:0/947625354 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a4004020 con 0x7f07ac14a1d0 2024-09-16T01:42:18.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.487+0000 7f07b26d0640 1 -- 172.21.15.88:0/947625354 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07ac1f84e0 con 0x7f07ac14a1d0 2024-09-16T01:42:18.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.487+0000 7f07b26d0640 1 -- 172.21.15.88:0/947625354 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07ac1619a0 con 0x7f07ac14a1d0 2024-09-16T01:42:18.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.488+0000 7f07b26d0640 1 -- 172.21.15.88:0/947625354 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07ac15ed20 con 0x7f07ac14a1d0 2024-09-16T01:42:18.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.489+0000 7f07a8ff9640 1 -- 172.21.15.88:0/947625354 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f07a401e030 con 0x7f07ac14a1d0 2024-09-16T01:42:18.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.489+0000 7f07a8ff9640 1 --2- 172.21.15.88:0/947625354 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f079806c2b0 0x7f079806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:18.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.489+0000 7f07a8ff9640 1 -- 172.21.15.88:0/947625354 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7286+0+0 (secure 0 0 0) 0x7f07a408c2f0 con 0x7f07ac14a1d0 2024-09-16T01:42:18.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.489+0000 7f07abfff640 1 --2- 172.21.15.88:0/947625354 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f079806c2b0 0x7f079806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:18.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.490+0000 7f07abfff640 1 --2- 172.21.15.88:0/947625354 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f079806c2b0 0x7f079806e770 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f079c005580 tx=0x7f079c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:18.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.494+0000 7f07a8ff9640 1 -- 172.21.15.88:0/947625354 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f07a4025020 con 0x7f07ac14a1d0 2024-09-16T01:42:18.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.674+0000 7f07b26d0640 1 -- 172.21.15.88:0/947625354 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f07ac1613e0 con 0x7f07ac14a1d0 2024-09-16T01:42:18.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.674+0000 7f07a8ff9640 1 -- 172.21.15.88:0/947625354 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 36 entries v105) ==== 81+0+2190 (secure 0 0 0) 0x7f07ac1513d0 con 0x7f07ac14a1d0 2024-09-16T01:42:18.676 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/3728363359 2024-09-17T01:42:05.412133+0000 2024-09-16T01:42:18.676 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/3163343083 2024-09-17T01:42:01.375396+0000 2024-09-16T01:42:18.676 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2550498835 2024-09-17T01:41:57.358287+0000 2024-09-16T01:42:18.676 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2550498835 2024-09-17T01:41:57.358287+0000 2024-09-16T01:42:18.676 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2408579939 2024-09-17T01:40:46.434492+0000 2024-09-16T01:42:18.676 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2733458327 2024-09-17T01:40:27.314712+0000 2024-09-16T01:42:18.676 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/1289932173 2024-09-17T01:40:42.399399+0000 2024-09-16T01:42:18.676 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/4162274767 2024-09-17T01:40:11.522091+0000 2024-09-16T01:42:18.676 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/3163343083 2024-09-17T01:42:01.375396+0000 2024-09-16T01:42:18.676 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2408579939 2024-09-17T01:40:46.434492+0000 2024-09-16T01:42:18.676 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/3356432718 2024-09-17T01:40:32.344252+0000 2024-09-16T01:42:18.677 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/1042715824 2024-09-17T01:40:59.080515+0000 2024-09-16T01:42:18.677 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2970185561 2024-09-17T01:40:38.255774+0000 2024-09-16T01:42:18.677 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/3356432718 2024-09-17T01:40:32.344252+0000 2024-09-16T01:42:18.677 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/1289932173 2024-09-17T01:40:42.399399+0000 2024-09-16T01:42:18.677 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2970185561 2024-09-17T01:40:38.255774+0000 2024-09-16T01:42:18.677 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/4162274767 2024-09-17T01:40:11.522091+0000 2024-09-16T01:42:18.677 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2957965650 2024-09-17T01:41:12.181485+0000 2024-09-16T01:42:18.677 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2733458327 2024-09-17T01:40:27.314712+0000 2024-09-16T01:42:18.677 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/1042715824 2024-09-17T01:40:59.080515+0000 2024-09-16T01:42:18.677 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/128058689 2024-09-17T01:41:10.163743+0000 2024-09-16T01:42:18.677 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2918194823 2024-09-17T01:41:29.275696+0000 2024-09-16T01:42:18.677 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/981209627 2024-09-17T01:41:51.268495+0000 2024-09-16T01:42:18.677 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/128058689 2024-09-17T01:41:10.163743+0000 2024-09-16T01:42:18.678 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/1162318664 2024-09-17T01:41:23.257870+0000 2024-09-16T01:42:18.678 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/3336935653 2024-09-17T01:41:44.295023+0000 2024-09-16T01:42:18.678 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2957965650 2024-09-17T01:41:12.181485+0000 2024-09-16T01:42:18.678 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/293764774 2024-09-17T01:41:16.203758+0000 2024-09-16T01:42:18.678 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/981209627 2024-09-17T01:41:51.268495+0000 2024-09-16T01:42:18.678 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/3336935653 2024-09-17T01:41:44.295023+0000 2024-09-16T01:42:18.678 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2918194823 2024-09-17T01:41:29.275696+0000 2024-09-16T01:42:18.678 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/1255135754 2024-09-17T01:41:41.905258+0000 2024-09-16T01:42:18.678 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/293764774 2024-09-17T01:41:16.203758+0000 2024-09-16T01:42:18.678 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/1162318664 2024-09-17T01:41:23.257870+0000 2024-09-16T01:42:18.678 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/3728363359 2024-09-17T01:42:05.412133+0000 2024-09-16T01:42:18.678 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/1255135754 2024-09-17T01:41:41.905258+0000 2024-09-16T01:42:18.679 INFO:teuthology.orchestra.run.smithi088.stderr:listed 36 entries 2024-09-16T01:42:18.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.676+0000 7f07b26d0640 1 -- 172.21.15.88:0/947625354 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f079806c2b0 msgr2=0x7f079806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:18.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.676+0000 7f07b26d0640 1 --2- 172.21.15.88:0/947625354 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f079806c2b0 0x7f079806e770 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f079c005580 tx=0x7f079c009040 comp rx=0 tx=0).stop 2024-09-16T01:42:18.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.676+0000 7f07b26d0640 1 -- 172.21.15.88:0/947625354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07ac14a1d0 msgr2=0x7f07ac14a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:18.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.676+0000 7f07b26d0640 1 --2- 172.21.15.88:0/947625354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07ac14a1d0 0x7f07ac14a660 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f07ac0ba0a0 tx=0x7f07a4010590 comp rx=0 tx=0).stop 2024-09-16T01:42:18.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.676+0000 7f07b26d0640 1 -- 172.21.15.88:0/947625354 shutdown_connections 2024-09-16T01:42:18.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.676+0000 7f07b26d0640 1 --2- 172.21.15.88:0/947625354 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f079806c2b0 0x7f079806e770 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:18.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.676+0000 7f07b26d0640 1 --2- 172.21.15.88:0/947625354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07ac14a1d0 0x7f07ac14a660 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:18.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.676+0000 7f07b26d0640 1 --2- 172.21.15.88:0/947625354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07ac160e40 0x7f07ac15f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:18.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.676+0000 7f07b26d0640 1 --2- 172.21.15.88:0/947625354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0794004800 0x7f07ac1f7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:18.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.676+0000 7f07b26d0640 1 -- 172.21.15.88:0/947625354 >> 172.21.15.88:0/947625354 conn(0x7f07ac0d1140 msgr2=0x7f07ac0b39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:18.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.677+0000 7f07b26d0640 1 -- 172.21.15.88:0/947625354 shutdown_connections 2024-09-16T01:42:18.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.677+0000 7f07b26d0640 1 -- 172.21.15.88:0/947625354 wait complete. 2024-09-16T01:42:18.688 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-16T01:42:18.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:18.689 DEBUG:teuthology.orchestra.run.smithi088:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-09-16T01:42:18.730 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:42:18.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:18.731 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:42:18.786 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:42:18.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.922+0000 7fb0f0ed7640 1 Processor -- start 2024-09-16T01:42:18.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.923+0000 7fb0f0ed7640 1 -- start start 2024-09-16T01:42:18.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.923+0000 7fb0f0ed7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0ec0b6290 0x7fb0ec071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:18.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.923+0000 7fb0f0ed7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb0ec072530 0x7fb0ec15bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:18.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.924+0000 7fb0f0ed7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0ec0ccff0 con 0x7fb0ec15b480 2024-09-16T01:42:18.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.924+0000 7fb0f0ed7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb0ec0cd2d0 con 0x7fb0ec072530 2024-09-16T01:42:18.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.924+0000 7fb0f0ed7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb0ec0cd5b0 con 0x7fb0ec0b6290 2024-09-16T01:42:18.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.924+0000 7fb0ead9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb0ec072530 0x7fb0ec15bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:18.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.924+0000 7fb0ead9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb0ec072530 0x7fb0ec15bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43902/0 (socket says 172.21.15.88:43902) 2024-09-16T01:42:18.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.924+0000 7fb0ead9d640 1 -- 172.21.15.88:0/1261100654 learned_addr learned my addr 172.21.15.88:0/1261100654 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:18.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.924+0000 7fb0e9d9b640 1 --2- 172.21.15.88:0/1261100654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0ec0b6290 0x7fb0ec071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:18.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.924+0000 7fb0ead9d640 1 -- 172.21.15.88:0/1261100654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0ec0b6290 msgr2=0x7fb0ec071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:18.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.924+0000 7fb0ead9d640 1 --2- 172.21.15.88:0/1261100654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0ec0b6290 0x7fb0ec071ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:18.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.924+0000 7fb0ead9d640 1 -- 172.21.15.88:0/1261100654 >> v1:172.21.15.154:6789/0 conn(0x7fb0ec15b480 legacy=0x7fb0ec0b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:18.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.924+0000 7fb0ead9d640 1 -- 172.21.15.88:0/1261100654 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0ec0cd890 con 0x7fb0ec072530 2024-09-16T01:42:18.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.924+0000 7fb0e9d9b640 1 --2- 172.21.15.88:0/1261100654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0ec0b6290 0x7fb0ec071ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:18.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.925+0000 7fb0ead9d640 1 --2- 172.21.15.88:0/1261100654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0ec072530 0x7fb0ec15bc60 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4009e50 tx=0x7fb0e402a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb519a8067a937fa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:18.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.925+0000 7fb0e959a640 1 -- 172.21.15.88:0/1261100654 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4036070 con 0x7fb0ec072530 2024-09-16T01:42:18.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.925+0000 7fb0e959a640 1 -- 172.21.15.88:0/1261100654 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0e4031600 con 0x7fb0ec072530 2024-09-16T01:42:18.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.925+0000 7fb0e959a640 1 -- 172.21.15.88:0/1261100654 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e40319d0 con 0x7fb0ec072530 2024-09-16T01:42:18.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.925+0000 7fb0f0ed7640 1 -- 172.21.15.88:0/1261100654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0ec072530 msgr2=0x7fb0ec15bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:18.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.925+0000 7fb0f0ed7640 1 --2- 172.21.15.88:0/1261100654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0ec072530 0x7fb0ec15bc60 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4009e50 tx=0x7fb0e402a630 comp rx=0 tx=0).stop 2024-09-16T01:42:18.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.925+0000 7fb0f0ed7640 1 -- 172.21.15.88:0/1261100654 shutdown_connections 2024-09-16T01:42:18.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.925+0000 7fb0f0ed7640 1 --2- 172.21.15.88:0/1261100654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0ec072530 0x7fb0ec15bc60 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:18.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.925+0000 7fb0f0ed7640 1 --2- 172.21.15.88:0/1261100654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0ec0b6290 0x7fb0ec071ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:18.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.925+0000 7fb0f0ed7640 1 -- 172.21.15.88:0/1261100654 >> 172.21.15.88:0/1261100654 conn(0x7fb0ec06d0f0 msgr2=0x7fb0ec06d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:18.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.926+0000 7fb0f0ed7640 1 -- 172.21.15.88:0/1261100654 shutdown_connections 2024-09-16T01:42:18.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.926+0000 7fb0f0ed7640 1 -- 172.21.15.88:0/1261100654 wait complete. 2024-09-16T01:42:18.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.927+0000 7fb0f0ed7640 1 Processor -- start 2024-09-16T01:42:18.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.927+0000 7fb0f0ed7640 1 -- start start 2024-09-16T01:42:18.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.928+0000 7fb0f0ed7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0ec072530 0x7fb0ec157700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:18.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.928+0000 7fb0f0ed7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0ec15b480 0x7fb0ec0c5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:18.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.928+0000 7fb0f0ed7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0ec157e50 0x7fb0ec1e8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:18.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.928+0000 7fb0f0ed7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb0ec15b9e0 con 0x7fb0ec157e50 2024-09-16T01:42:18.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.928+0000 7fb0f0ed7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb0ec15b860 con 0x7fb0ec072530 2024-09-16T01:42:18.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.928+0000 7fb0f0ed7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb0ec15c1a0 con 0x7fb0ec15b480 2024-09-16T01:42:18.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.928+0000 7fb0ead9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0ec157e50 0x7fb0ec1e8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:18.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.928+0000 7fb0ead9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0ec157e50 0x7fb0ec1e8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43910/0 (socket says 172.21.15.88:43910) 2024-09-16T01:42:18.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.928+0000 7fb0ead9d640 1 -- 172.21.15.88:0/971532750 learned_addr learned my addr 172.21.15.88:0/971532750 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:18.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.928+0000 7fb0ea59c640 1 --2- 172.21.15.88:0/971532750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0ec072530 0x7fb0ec157700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:18.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.928+0000 7fb0e9d9b640 1 --2- 172.21.15.88:0/971532750 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0ec15b480 0x7fb0ec0c5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:18.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.928+0000 7fb0ead9d640 1 -- 172.21.15.88:0/971532750 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0ec15b480 msgr2=0x7fb0ec0c5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:18.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.928+0000 7fb0ead9d640 1 --2- 172.21.15.88:0/971532750 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0ec15b480 0x7fb0ec0c5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:18.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.928+0000 7fb0ead9d640 1 -- 172.21.15.88:0/971532750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0ec072530 msgr2=0x7fb0ec157700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:18.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.928+0000 7fb0ead9d640 1 --2- 172.21.15.88:0/971532750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0ec072530 0x7fb0ec157700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:18.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.928+0000 7fb0ead9d640 1 -- 172.21.15.88:0/971532750 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0ec15d120 con 0x7fb0ec157e50 2024-09-16T01:42:18.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.928+0000 7fb0e9d9b640 1 --2- 172.21.15.88:0/971532750 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0ec15b480 0x7fb0ec0c5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:18.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.928+0000 7fb0ead9d640 1 --2- 172.21.15.88:0/971532750 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0ec157e50 0x7fb0ec1e8190 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4009e50 tx=0x7fb0e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:18.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.928+0000 7fb0e2ffd640 1 -- 172.21.15.88:0/971532750 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4036050 con 0x7fb0ec157e50 2024-09-16T01:42:18.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.928+0000 7fb0e2ffd640 1 -- 172.21.15.88:0/971532750 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0e4042070 con 0x7fb0ec157e50 2024-09-16T01:42:18.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.928+0000 7fb0e2ffd640 1 -- 172.21.15.88:0/971532750 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4009510 con 0x7fb0ec157e50 2024-09-16T01:42:18.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.928+0000 7fb0f0ed7640 1 -- 172.21.15.88:0/971532750 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0ec15d400 con 0x7fb0ec157e50 2024-09-16T01:42:18.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.929+0000 7fb0f0ed7640 1 -- 172.21.15.88:0/971532750 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0ec0c99a0 con 0x7fb0ec157e50 2024-09-16T01:42:18.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.929+0000 7fb0f0ed7640 1 -- 172.21.15.88:0/971532750 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0ec15bc60 con 0x7fb0ec157e50 2024-09-16T01:42:18.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.930+0000 7fb0e2ffd640 1 -- 172.21.15.88:0/971532750 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb0e4007bd0 con 0x7fb0ec157e50 2024-09-16T01:42:18.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.930+0000 7fb0e2ffd640 1 --2- 172.21.15.88:0/971532750 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb0cc06c2b0 0x7fb0cc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:18.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.930+0000 7fb0e2ffd640 1 -- 172.21.15.88:0/971532750 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 7286+0+0 (secure 0 0 0) 0x7fb0e40ac890 con 0x7fb0ec157e50 2024-09-16T01:42:18.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.930+0000 7fb0ea59c640 1 --2- 172.21.15.88:0/971532750 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb0cc06c2b0 0x7fb0cc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:18.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.931+0000 7fb0ea59c640 1 --2- 172.21.15.88:0/971532750 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb0cc06c2b0 0x7fb0cc06e770 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4006fd0 tx=0x7fb0d4009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:18.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:18.934+0000 7fb0e2ffd640 1 -- 172.21.15.88:0/971532750 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb0e407aa60 con 0x7fb0ec157e50 2024-09-16T01:42:19.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.132+0000 7fb0f0ed7640 1 -- 172.21.15.88:0/971532750 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fb0ec0cf760 con 0x7fb0ec157e50 2024-09-16T01:42:19.133 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:42:19.132+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:42:19.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.133+0000 7fb0e2ffd640 1 -- 172.21.15.88:0/971532750 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v184) ==== 161+0+0 (secure 0 0 0) 0x7fb0e407a360 con 0x7fb0ec157e50 2024-09-16T01:42:19.135 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:42:19.137 INFO:tasks.ceph.mds.c.smithi088.stderr: -1763> 2024-09-16T01:42:09.441+0000 7fd3f783d640 -1 mds.0.purge_queue operator(): Error -22 loading Journaler 2024-09-16T01:42:19.137 INFO:tasks.ceph.mds.c.smithi088.stderr: -1760> 2024-09-16T01:42:09.441+0000 7fd3f783d640 -1 mds.0.180 unhandled write error (22) Invalid argument, force readonly... 2024-09-16T01:42:19.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.136+0000 7fb0f0ed7640 1 -- 172.21.15.88:0/971532750 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb0cc06c2b0 msgr2=0x7fb0cc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:19.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.136+0000 7fb0f0ed7640 1 --2- 172.21.15.88:0/971532750 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb0cc06c2b0 0x7fb0cc06e770 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4006fd0 tx=0x7fb0d4009060 comp rx=0 tx=0).stop 2024-09-16T01:42:19.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.136+0000 7fb0f0ed7640 1 -- 172.21.15.88:0/971532750 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0ec157e50 msgr2=0x7fb0ec1e8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:19.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.136+0000 7fb0f0ed7640 1 --2- 172.21.15.88:0/971532750 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0ec157e50 0x7fb0ec1e8190 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4009e50 tx=0x7fb0e4004a40 comp rx=0 tx=0).stop 2024-09-16T01:42:19.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.136+0000 7fb0f0ed7640 1 -- 172.21.15.88:0/971532750 shutdown_connections 2024-09-16T01:42:19.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.136+0000 7fb0f0ed7640 1 --2- 172.21.15.88:0/971532750 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb0cc06c2b0 0x7fb0cc06e770 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:19.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.136+0000 7fb0f0ed7640 1 --2- 172.21.15.88:0/971532750 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb0ec157e50 0x7fb0ec1e8190 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:19.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.136+0000 7fb0f0ed7640 1 --2- 172.21.15.88:0/971532750 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0ec15b480 0x7fb0ec0c5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:19.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.136+0000 7fb0f0ed7640 1 --2- 172.21.15.88:0/971532750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0ec072530 0x7fb0ec157700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:19.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.136+0000 7fb0f0ed7640 1 -- 172.21.15.88:0/971532750 >> 172.21.15.88:0/971532750 conn(0x7fb0ec06d0f0 msgr2=0x7fb0ec0b2730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:19.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.137+0000 7fb0f0ed7640 1 -- 172.21.15.88:0/971532750 shutdown_connections 2024-09-16T01:42:19.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.137+0000 7fb0f0ed7640 1 -- 172.21.15.88:0/971532750 wait complete. 2024-09-16T01:42:19.139 INFO:tasks.ceph.mds.c.smithi088.stderr: -587> 2024-09-16T01:42:16.419+0000 7fd3fa843640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:42:19.149 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:42:19.163 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:42:19.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.248+0000 7f9ec58e4640 1 Processor -- start 2024-09-16T01:42:19.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.248+0000 7f9ec58e4640 1 -- start start 2024-09-16T01:42:19.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.249+0000 7f9ec58e4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9ec00cf1f0 0x7f9ec00b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:19.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.249+0000 7f9ec58e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ec00b6720 0x7f9ec00d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:19.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.249+0000 7f9ec58e4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ec00cf860 con 0x7f9ec015e7a0 2024-09-16T01:42:19.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.249+0000 7f9ec58e4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f9ec00d2ca0 con 0x7f9ec00cf1f0 2024-09-16T01:42:19.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.249+0000 7f9ec58e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9ec00cfb40 con 0x7f9ec00b6720 2024-09-16T01:42:19.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.249+0000 7f9ebe7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9ec00cf1f0 0x7f9ec00b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:19.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.249+0000 7f9ebe7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9ec00cf1f0 0x7f9ec00b61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43944/0 (socket says 172.21.15.88:43944) 2024-09-16T01:42:19.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.249+0000 7f9ebe7fc640 1 -- 172.21.15.88:0/2236619075 learned_addr learned my addr 172.21.15.88:0/2236619075 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:19.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.249+0000 7f9ebe7fc640 1 -- 172.21.15.88:0/2236619075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ec00b6720 msgr2=0x7f9ec00d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:19.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.249+0000 7f9ebe7fc640 1 --2- 172.21.15.88:0/2236619075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ec00b6720 0x7f9ec00d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:19.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.249+0000 7f9ebe7fc640 1 -- 172.21.15.88:0/2236619075 >> v1:172.21.15.154:6789/0 conn(0x7f9ec015e7a0 legacy=0x7f9ec006caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:19.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.249+0000 7f9ebe7fc640 1 -- 172.21.15.88:0/2236619075 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ec00cfe20 con 0x7f9ec00cf1f0 2024-09-16T01:42:19.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.250+0000 7f9ebe7fc640 1 --2- 172.21.15.88:0/2236619075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ec00cf1f0 0x7f9ec00b61e0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4009870 tx=0x7f9eb402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4002d3bdf32c7588 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:19.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.250+0000 7f9ebdffb640 1 -- 172.21.15.88:0/2236619075 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb4037070 con 0x7f9ec00cf1f0 2024-09-16T01:42:19.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.250+0000 7f9ebdffb640 1 -- 172.21.15.88:0/2236619075 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9eb4032440 con 0x7f9ec00cf1f0 2024-09-16T01:42:19.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.250+0000 7f9ebdffb640 1 -- 172.21.15.88:0/2236619075 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb4032810 con 0x7f9ec00cf1f0 2024-09-16T01:42:19.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.251+0000 7f9ec58e4640 1 -- 172.21.15.88:0/2236619075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ec00cf1f0 msgr2=0x7f9ec00b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:19.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.251+0000 7f9ec58e4640 1 --2- 172.21.15.88:0/2236619075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ec00cf1f0 0x7f9ec00b61e0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4009870 tx=0x7f9eb402a180 comp rx=0 tx=0).stop 2024-09-16T01:42:19.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.251+0000 7f9ec58e4640 1 -- 172.21.15.88:0/2236619075 shutdown_connections 2024-09-16T01:42:19.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.251+0000 7f9ec58e4640 1 --2- 172.21.15.88:0/2236619075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ec00b6720 0x7f9ec00d3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:19.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.251+0000 7f9ec58e4640 1 --2- 172.21.15.88:0/2236619075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ec00cf1f0 0x7f9ec00b61e0 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:19.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.251+0000 7f9ec58e4640 1 -- 172.21.15.88:0/2236619075 >> 172.21.15.88:0/2236619075 conn(0x7f9ec00d11e0 msgr2=0x7f9ec00d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:19.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.251+0000 7f9ec58e4640 1 -- 172.21.15.88:0/2236619075 shutdown_connections 2024-09-16T01:42:19.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.251+0000 7f9ec58e4640 1 -- 172.21.15.88:0/2236619075 wait complete. 2024-09-16T01:42:19.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.252+0000 7f9ec58e4640 1 Processor -- start 2024-09-16T01:42:19.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.252+0000 7f9ec58e4640 1 -- start start 2024-09-16T01:42:19.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.253+0000 7f9ec58e4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ec00b6720 0x7f9ec00ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:19.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.253+0000 7f9ec58e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ec00cf1f0 0x7f9ec00b8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:19.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.253+0000 7f9ec58e4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ec015e7a0 0x7f9ec00b92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:19.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.253+0000 7f9ec58e4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec015eb80 con 0x7f9ec00b6720 2024-09-16T01:42:19.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.253+0000 7f9ec58e4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec00b7a00 con 0x7f9ec015e7a0 2024-09-16T01:42:19.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.253+0000 7f9ec58e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9ec015ed00 con 0x7f9ec00cf1f0 2024-09-16T01:42:19.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.253+0000 7f9ebeffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ec00b6720 0x7f9ec00ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:19.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.253+0000 7f9ebeffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ec00b6720 0x7f9ec00ce5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43950/0 (socket says 172.21.15.88:43950) 2024-09-16T01:42:19.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.253+0000 7f9ebeffd640 1 -- 172.21.15.88:0/460544365 learned_addr learned my addr 172.21.15.88:0/460544365 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:19.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.253+0000 7f9ebf7fe640 1 --2- 172.21.15.88:0/460544365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ec015e7a0 0x7f9ec00b92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:19.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.253+0000 7f9ebe7fc640 1 --2- 172.21.15.88:0/460544365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ec00cf1f0 0x7f9ec00b8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:19.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.253+0000 7f9ebeffd640 1 -- 172.21.15.88:0/460544365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ec00cf1f0 msgr2=0x7f9ec00b8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:19.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.253+0000 7f9ebeffd640 1 --2- 172.21.15.88:0/460544365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ec00cf1f0 0x7f9ec00b8da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:19.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.253+0000 7f9ebeffd640 1 -- 172.21.15.88:0/460544365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ec015e7a0 msgr2=0x7f9ec00b92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:19.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.253+0000 7f9ebeffd640 1 --2- 172.21.15.88:0/460544365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ec015e7a0 0x7f9ec00b92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:19.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.253+0000 7f9ebeffd640 1 -- 172.21.15.88:0/460544365 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ec00cc3b0 con 0x7f9ec00b6720 2024-09-16T01:42:19.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.253+0000 7f9ebf7fe640 1 --2- 172.21.15.88:0/460544365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ec015e7a0 0x7f9ec00b92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:19.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.253+0000 7f9ebe7fc640 1 --2- 172.21.15.88:0/460544365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ec00cf1f0 0x7f9ec00b8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:19.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.253+0000 7f9ebeffd640 1 --2- 172.21.15.88:0/460544365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ec00b6720 0x7f9ec00ce5c0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f9eb000ba50 tx=0x7f9eb000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:19.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.253+0000 7f9ea7fff640 1 -- 172.21.15.88:0/460544365 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb0019070 con 0x7f9ec00b6720 2024-09-16T01:42:19.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.253+0000 7f9ea7fff640 1 -- 172.21.15.88:0/460544365 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9eb000cc20 con 0x7f9ec00b6720 2024-09-16T01:42:19.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.253+0000 7f9ec58e4640 1 -- 172.21.15.88:0/460544365 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ec00d4ab0 con 0x7f9ec00b6720 2024-09-16T01:42:19.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.253+0000 7f9ea7fff640 1 -- 172.21.15.88:0/460544365 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb0014680 con 0x7f9ec00b6720 2024-09-16T01:42:19.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.253+0000 7f9ec58e4640 1 -- 172.21.15.88:0/460544365 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ec00d5070 con 0x7f9ec00b6720 2024-09-16T01:42:19.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.254+0000 7f9ec58e4640 1 -- 172.21.15.88:0/460544365 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ec00b61e0 con 0x7f9ec00b6720 2024-09-16T01:42:19.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.255+0000 7f9ea7fff640 1 -- 172.21.15.88:0/460544365 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9eb001c070 con 0x7f9ec00b6720 2024-09-16T01:42:19.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.255+0000 7f9ea7fff640 1 --2- 172.21.15.88:0/460544365 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9eac06c2b0 0x7f9eac06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:19.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.255+0000 7f9ebe7fc640 1 --2- 172.21.15.88:0/460544365 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9eac06c2b0 0x7f9eac06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:19.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.255+0000 7f9ea7fff640 1 -- 172.21.15.88:0/460544365 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 7372+0+0 (secure 0 0 0) 0x7f9eb008b6d0 con 0x7f9ec00b6720 2024-09-16T01:42:19.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.256+0000 7f9ebe7fc640 1 --2- 172.21.15.88:0/460544365 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9eac06c2b0 0x7f9eac06e770 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4034000 tx=0x7f9eb402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:19.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.260+0000 7f9ea7fff640 1 -- 172.21.15.88:0/460544365 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9eb0059200 con 0x7f9ec00b6720 2024-09-16T01:42:19.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:19.496+0000 7f9ec58e4640 1 -- 172.21.15.88:0/460544365 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f9ec00d2a80 con 0x7f9ec00b6720 2024-09-16T01:42:20.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.135+0000 7f9ea7fff640 1 -- 172.21.15.88:0/460544365 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v185) ==== 107+0+0 (secure 0 0 0) 0x7f9eb005d500 con 0x7f9ec00b6720 2024-09-16T01:42:20.137 INFO:teuthology.orchestra.run.smithi088.stderr:nothing to do: rank is not damaged 2024-09-16T01:42:20.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.138+0000 7f9ec58e4640 1 -- 172.21.15.88:0/460544365 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9eac06c2b0 msgr2=0x7f9eac06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:20.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.138+0000 7f9ec58e4640 1 --2- 172.21.15.88:0/460544365 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9eac06c2b0 0x7f9eac06e770 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4034000 tx=0x7f9eb402e000 comp rx=0 tx=0).stop 2024-09-16T01:42:20.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.138+0000 7f9ec58e4640 1 -- 172.21.15.88:0/460544365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ec00b6720 msgr2=0x7f9ec00ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:20.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.138+0000 7f9ec58e4640 1 --2- 172.21.15.88:0/460544365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ec00b6720 0x7f9ec00ce5c0 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f9eb000ba50 tx=0x7f9eb000bf10 comp rx=0 tx=0).stop 2024-09-16T01:42:20.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.138+0000 7f9ec58e4640 1 -- 172.21.15.88:0/460544365 shutdown_connections 2024-09-16T01:42:20.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.138+0000 7f9ec58e4640 1 --2- 172.21.15.88:0/460544365 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9eac06c2b0 0x7f9eac06e770 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.138+0000 7f9ec58e4640 1 --2- 172.21.15.88:0/460544365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ec015e7a0 0x7f9ec00b92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.138+0000 7f9ec58e4640 1 --2- 172.21.15.88:0/460544365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ec00cf1f0 0x7f9ec00b8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.138+0000 7f9ec58e4640 1 --2- 172.21.15.88:0/460544365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ec00b6720 0x7f9ec00ce5c0 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.138+0000 7f9ec58e4640 1 -- 172.21.15.88:0/460544365 >> 172.21.15.88:0/460544365 conn(0x7f9ec00d11e0 msgr2=0x7f9ec01573b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:20.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.138+0000 7f9ec58e4640 1 -- 172.21.15.88:0/460544365 shutdown_connections 2024-09-16T01:42:20.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.138+0000 7f9ec58e4640 1 -- 172.21.15.88:0/460544365 wait complete. 2024-09-16T01:42:20.158 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:42:20.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.243+0000 7f02b201cf40 1 Processor -- start 2024-09-16T01:42:20.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.243+0000 7f02b201cf40 1 -- start start 2024-09-16T01:42:20.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.244+0000 7f02b201cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c1b91affe0 0x55c1b9257d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.244+0000 7f02b201cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c1b91b0a60 0x55c1b9254f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.244+0000 7f02b201cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1b91ca250 0x55c1b924f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.244+0000 7f02b201cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55c1b90d2510 con 0x55c1b91affe0 2024-09-16T01:42:20.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.244+0000 7f02b201cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c1b9259120 con 0x55c1b91ca250 2024-09-16T01:42:20.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.244+0000 7f02b201cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55c1b92578b0 con 0x55c1b91b0a60 2024-09-16T01:42:20.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.244+0000 7f02afb14640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c1b91affe0 0x55c1b9257d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.244+0000 7f02afb14640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c1b91affe0 0x55c1b9257d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43956/0 (socket says 172.21.15.88:43956) 2024-09-16T01:42:20.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.244+0000 7f02afb14640 1 -- 172.21.15.88:0/1024094355 learned_addr learned my addr 172.21.15.88:0/1024094355 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:20.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.244+0000 7f02b0315640 1 --2- 172.21.15.88:0/1024094355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1b91ca250 0x55c1b924f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.244+0000 7f02a7313640 1 --2- 172.21.15.88:0/1024094355 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c1b91b0a60 0x55c1b9254f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.244+0000 7f02afb14640 1 -- 172.21.15.88:0/1024094355 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c1b91b0a60 msgr2=0x55c1b9254f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:20.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.244+0000 7f02afb14640 1 --2- 172.21.15.88:0/1024094355 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c1b91b0a60 0x55c1b9254f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.244+0000 7f02afb14640 1 -- 172.21.15.88:0/1024094355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1b91ca250 msgr2=0x55c1b924f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:20.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.244+0000 7f02afb14640 1 --2- 172.21.15.88:0/1024094355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1b91ca250 0x55c1b924f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.244+0000 7f02afb14640 1 -- 172.21.15.88:0/1024094355 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1b91cbeb0 con 0x55c1b91affe0 2024-09-16T01:42:20.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.244+0000 7f02afb14640 1 --2- 172.21.15.88:0/1024094355 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c1b91affe0 0x55c1b9257d40 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f02a000a770 tx=0x7f02a000a860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20c217f4ca5ef76 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:20.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.244+0000 7f02aeb12640 1 -- 172.21.15.88:0/1024094355 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a0019070 con 0x55c1b91affe0 2024-09-16T01:42:20.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.244+0000 7f02aeb12640 1 -- 172.21.15.88:0/1024094355 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02a0007600 con 0x55c1b91affe0 2024-09-16T01:42:20.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.244+0000 7f02aeb12640 1 -- 172.21.15.88:0/1024094355 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a00079b0 con 0x55c1b91affe0 2024-09-16T01:42:20.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.245+0000 7f02b201cf40 1 -- 172.21.15.88:0/1024094355 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c1b91affe0 msgr2=0x55c1b9257d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:20.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.245+0000 7f02b201cf40 1 --2- 172.21.15.88:0/1024094355 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c1b91affe0 0x55c1b9257d40 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f02a000a770 tx=0x7f02a000a860 comp rx=0 tx=0).stop 2024-09-16T01:42:20.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.245+0000 7f02b201cf40 1 -- 172.21.15.88:0/1024094355 shutdown_connections 2024-09-16T01:42:20.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.245+0000 7f02b201cf40 1 --2- 172.21.15.88:0/1024094355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1b91ca250 0x55c1b924f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.245+0000 7f02b201cf40 1 --2- 172.21.15.88:0/1024094355 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c1b91b0a60 0x55c1b9254f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.245+0000 7f02b201cf40 1 --2- 172.21.15.88:0/1024094355 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c1b91affe0 0x55c1b9257d40 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.245+0000 7f02b201cf40 1 -- 172.21.15.88:0/1024094355 >> 172.21.15.88:0/1024094355 conn(0x55c1b91cb750 msgr2=0x55c1b925ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:20.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.245+0000 7f02b201cf40 1 -- 172.21.15.88:0/1024094355 shutdown_connections 2024-09-16T01:42:20.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.245+0000 7f02b201cf40 1 -- 172.21.15.88:0/1024094355 wait complete. 2024-09-16T01:42:20.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.245+0000 7f02b201cf40 1 Processor -- start 2024-09-16T01:42:20.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.246+0000 7f02b201cf40 1 -- start start 2024-09-16T01:42:20.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.246+0000 7f02b201cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1b91affe0 0x55c1b9248ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.246+0000 7f02b201cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c1b91b0a60 0x55c1b91c9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.246+0000 7f02b201cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c1b91ca250 0x55c1b91c9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.246+0000 7f02b201cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55c1b924ad40 con 0x55c1b91b0a60 2024-09-16T01:42:20.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.246+0000 7f02b201cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55c1b9258280 con 0x55c1b91affe0 2024-09-16T01:42:20.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.246+0000 7f02b201cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55c1b91cbbd0 con 0x55c1b91ca250 2024-09-16T01:42:20.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.246+0000 7f02a7313640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c1b91b0a60 0x55c1b91c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.246+0000 7f02a7313640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c1b91b0a60 0x55c1b91c9140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43972/0 (socket says 172.21.15.88:43972) 2024-09-16T01:42:20.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.246+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 learned_addr learned my addr 172.21.15.88:0/1051011209 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:20.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.246+0000 7f02afb14640 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1b91affe0 0x55c1b9248ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.247+0000 7f02b0315640 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c1b91ca250 0x55c1b91c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.247+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c1b91ca250 msgr2=0x55c1b91c9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:20.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.247+0000 7f02a7313640 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55c1b91ca250 0x55c1b91c9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.247+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1b91affe0 msgr2=0x55c1b9248ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:20.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.247+0000 7f02a7313640 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c1b91affe0 0x55c1b9248ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.247+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1b9255000 con 0x55c1b91b0a60 2024-09-16T01:42:20.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.247+0000 7f02a7313640 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c1b91b0a60 0x55c1b91c9140 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f029800ded0 tx=0x7f029800b5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:20.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.247+0000 7f02adb10640 1 -- 172.21.15.88:0/1051011209 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0298014020 con 0x55c1b91b0a60 2024-09-16T01:42:20.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.247+0000 7f02adb10640 1 -- 172.21.15.88:0/1051011209 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0298010410 con 0x55c1b91b0a60 2024-09-16T01:42:20.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.247+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c1b92498d0 con 0x55c1b91b0a60 2024-09-16T01:42:20.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.247+0000 7f02adb10640 1 -- 172.21.15.88:0/1051011209 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0298008030 con 0x55c1b91b0a60 2024-09-16T01:42:20.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.247+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1b9249530 con 0x55c1b91b0a60 2024-09-16T01:42:20.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.248+0000 7f02adb10640 1 -- 172.21.15.88:0/1051011209 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0298017020 con 0x55c1b91b0a60 2024-09-16T01:42:20.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.249+0000 7f02adb10640 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f029006ac90 0x7f029006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.249+0000 7f02adb10640 1 -- 172.21.15.88:0/1051011209 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 7372+0+0 (secure 0 0 0) 0x7f029808a610 con 0x55c1b91b0a60 2024-09-16T01:42:20.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.249+0000 7f02afb14640 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f029006ac90 0x7f029006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.256 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:42:20.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.249+0000 7f02b201cf40 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55c1b924c130 0x55c1b924caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.249+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b91c2300 con 0x55c1b924c130 2024-09-16T01:42:20.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.249+0000 7f02b0315640 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55c1b924c130 0x55c1b924caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.249+0000 7f02afb14640 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f029006ac90 0x7f029006d150 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f02a0017040 tx=0x7f02a000c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:20.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.250+0000 7f02b0315640 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55c1b924c130 0x55c1b924caf0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:20.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.251+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v106'567 uv541 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.251+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e106) -- 0x55c1b91c2710 con 0x55c1b924c130 2024-09-16T01:42:20.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.253+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v106'568 uv568 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.253+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fb250 con 0x55c1b924c130 2024-09-16T01:42:20.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.254+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v106'569 uv569 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.258 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:42:20.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.254+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fc600 con 0x55c1b924c130 2024-09-16T01:42:20.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.255+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v106'570 uv570 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.255+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fc9a0 con 0x55c1b924c130 2024-09-16T01:42:20.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.257+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v106'571 uv571 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.257+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fb5f0 con 0x55c1b924c130 2024-09-16T01:42:20.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.258+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v106'572 uv572 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.258+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fb990 con 0x55c1b924c130 2024-09-16T01:42:20.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.259+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v106'573 uv547 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.259+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fb990 con 0x55c1b924c130 2024-09-16T01:42:20.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.261+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v106'574 uv574 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.261+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fb990 con 0x55c1b924c130 2024-09-16T01:42:20.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.262+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v106'575 uv575 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.263 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:42:20.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.262+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fb990 con 0x55c1b924c130 2024-09-16T01:42:20.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.263+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v106'576 uv576 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.263+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fb990 con 0x55c1b924c130 2024-09-16T01:42:20.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.265+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v106'577 uv577 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.265+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fb990 con 0x55c1b924c130 2024-09-16T01:42:20.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.266+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v106'578 uv578 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.266+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fb990 con 0x55c1b924c130 2024-09-16T01:42:20.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.267+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v106'579 uv554 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.267+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fb990 con 0x55c1b924c130 2024-09-16T01:42:20.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.268+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v106'580 uv580 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.268+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fb990 con 0x55c1b924c130 2024-09-16T01:42:20.271 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:42:20.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.269+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v106'581 uv581 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.270+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fb990 con 0x55c1b924c130 2024-09-16T01:42:20.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.271+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v106'582 uv582 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.271+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fb990 con 0x55c1b924c130 2024-09-16T01:42:20.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.272+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v106'583 uv583 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.272+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fb990 con 0x55c1b924c130 2024-09-16T01:42:20.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.273+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v106'584 uv584 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.273+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fbd50 con 0x55c1b924c130 2024-09-16T01:42:20.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.274+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v106'585 uv585 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.274+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fbc40 con 0x55c1b924c130 2024-09-16T01:42:20.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.277+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v106'586 uv560 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.277+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fbc40 con 0x55c1b924c130 2024-09-16T01:42:20.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.278+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v106'587 uv587 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.278+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fbc40 con 0x55c1b924c130 2024-09-16T01:42:20.280 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:42:20.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.279+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v106'588 uv588 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.279+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fbc40 con 0x55c1b924c130 2024-09-16T01:42:20.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.281+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v106'589 uv589 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.281+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fbc40 con 0x55c1b924c130 2024-09-16T01:42:20.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.282+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v106'590 uv590 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.282+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fbc40 con 0x55c1b924c130 2024-09-16T01:42:20.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.283+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v106'591 uv591 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.283+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fbc40 con 0x55c1b924c130 2024-09-16T01:42:20.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.284+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v106'592 uv566 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.284+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fbc40 con 0x55c1b924c130 2024-09-16T01:42:20.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.286+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v106'593 uv593 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.286+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fbc40 con 0x55c1b924c130 2024-09-16T01:42:20.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.287+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v106'594 uv594 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.288 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:42:20.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.287+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fbc40 con 0x55c1b924c130 2024-09-16T01:42:20.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.288+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v106'595 uv595 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.288+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fbc40 con 0x55c1b924c130 2024-09-16T01:42:20.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.289+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v106'596 uv596 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.289+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92fbc40 con 0x55c1b924c130 2024-09-16T01:42:20.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.290+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v106'597 uv597 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.290+0000 7f02b201cf40 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55c1b92fbc40 0x55c1b92fed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.290+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92ff290 con 0x55c1b92fbc40 2024-09-16T01:42:20.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.290+0000 7f02afb14640 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55c1b92fbc40 0x55c1b92fed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.291+0000 7f02afb14640 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55c1b92fbc40 0x55c1b92fed50 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:20.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.292+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v106'422 uv401 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b92fbc40 2024-09-16T01:42:20.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.292+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92ff630 con 0x55c1b92fbc40 2024-09-16T01:42:20.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.293+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v106'423 uv423 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b92fbc40 2024-09-16T01:42:20.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.293+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92ff9d0 con 0x55c1b92fbc40 2024-09-16T01:42:20.295 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:42:20.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.294+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v106'424 uv424 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b92fbc40 2024-09-16T01:42:20.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.294+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9300220 con 0x55c1b92fbc40 2024-09-16T01:42:20.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.296+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v106'425 uv425 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b92fbc40 2024-09-16T01:42:20.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.296+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b93005c0 con 0x55c1b92fbc40 2024-09-16T01:42:20.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.297+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v106'426 uv426 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b92fbc40 2024-09-16T01:42:20.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.297+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9300960 con 0x55c1b92fbc40 2024-09-16T01:42:20.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.299+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v106'427 uv420 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b92fbc40 2024-09-16T01:42:20.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.299+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9300d00 con 0x55c1b92fbc40 2024-09-16T01:42:20.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.300+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v106'428 uv428 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b92fbc40 2024-09-16T01:42:20.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.300+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9300d00 con 0x55c1b92fbc40 2024-09-16T01:42:20.302 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:42:20.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.301+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v106'429 uv429 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b92fbc40 2024-09-16T01:42:20.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.301+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9300d00 con 0x55c1b92fbc40 2024-09-16T01:42:20.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.302+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v106'430 uv430 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b92fbc40 2024-09-16T01:42:20.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.302+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9300d00 con 0x55c1b92fbc40 2024-09-16T01:42:20.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.303+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v106'431 uv431 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b92fbc40 2024-09-16T01:42:20.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.303+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9300d00 con 0x55c1b92fbc40 2024-09-16T01:42:20.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.304+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v106'432 uv421 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b92fbc40 2024-09-16T01:42:20.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.304+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9300d00 con 0x55c1b92fbc40 2024-09-16T01:42:20.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.305+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v106'433 uv433 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b92fbc40 2024-09-16T01:42:20.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.305+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9300d00 con 0x55c1b92fbc40 2024-09-16T01:42:20.307 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:42:20.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.306+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v106'434 uv434 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b92fbc40 2024-09-16T01:42:20.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.306+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9305180 con 0x55c1b92fbc40 2024-09-16T01:42:20.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.308+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v106'435 uv435 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b92fbc40 2024-09-16T01:42:20.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.308+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9305180 con 0x55c1b92fbc40 2024-09-16T01:42:20.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.309+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v106'436 uv436 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b92fbc40 2024-09-16T01:42:20.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.309+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9300d00 con 0x55c1b92fbc40 2024-09-16T01:42:20.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.310+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v106'437 uv417 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b92fbc40 2024-09-16T01:42:20.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.310+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9300d00 con 0x55c1b92fbc40 2024-09-16T01:42:20.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.311+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v106'438 uv438 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b92fbc40 2024-09-16T01:42:20.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.311+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9300d00 con 0x55c1b92fbc40 2024-09-16T01:42:20.313 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:42:20.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.312+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v106'439 uv439 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b92fbc40 2024-09-16T01:42:20.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.312+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9301d10 con 0x55c1b92fbc40 2024-09-16T01:42:20.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.313+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v106'440 uv440 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b92fbc40 2024-09-16T01:42:20.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.313+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9301d10 con 0x55c1b92fbc40 2024-09-16T01:42:20.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.314+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v106'441 uv441 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b92fbc40 2024-09-16T01:42:20.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.314+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9300d00 con 0x55c1b92fbc40 2024-09-16T01:42:20.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.315+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v106'442 uv442 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b92fbc40 2024-09-16T01:42:20.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.315+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9300d00 con 0x55c1b924c130 2024-09-16T01:42:20.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.316+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v106'111 uv110 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.316+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9300d00 con 0x55c1b924c130 2024-09-16T01:42:20.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.317+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v106'112 uv112 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.317+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9300d00 con 0x55c1b924c130 2024-09-16T01:42:20.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.318+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v106'113 uv113 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.319 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:42:20.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.318+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9301d10 con 0x55c1b924c130 2024-09-16T01:42:20.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.319+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v106'114 uv114 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.319+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9301d10 con 0x55c1b924c130 2024-09-16T01:42:20.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.320+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v106'115 uv115 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.320+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9300d00 con 0x55c1b924c130 2024-09-16T01:42:20.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.321+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v106'116 uv116 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.321+0000 7f02b201cf40 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55c1b9301380 0x55c1b9303770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.321+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9303d60 con 0x55c1b9301380 2024-09-16T01:42:20.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.321+0000 7f02a7313640 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55c1b9301380 0x55c1b9303770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.321+0000 7f02a7313640 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55c1b9301380 0x55c1b9303770 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:20.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.322+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v106'85 uv84 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f0298020070 con 0x55c1b9301380 2024-09-16T01:42:20.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.322+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9304270 con 0x55c1b9301380 2024-09-16T01:42:20.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.323+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v106'86 uv86 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0298020070 con 0x55c1b9301380 2024-09-16T01:42:20.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.323+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9304770 con 0x55c1b9301380 2024-09-16T01:42:20.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.324+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v106'87 uv87 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0298020070 con 0x55c1b9301380 2024-09-16T01:42:20.325 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:42:20.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.324+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b92ffe50 con 0x55c1b9301380 2024-09-16T01:42:20.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.325+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v106'88 uv88 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0298020070 con 0x55c1b9301380 2024-09-16T01:42:20.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.325+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b93050a0 con 0x55c1b924c130 2024-09-16T01:42:20.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.326+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v106'111 uv110 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.326+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e106) -- 0x55c1b93050a0 con 0x55c1b924c130 2024-09-16T01:42:20.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.327+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v106'112 uv112 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.328+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b93050a0 con 0x55c1b924c130 2024-09-16T01:42:20.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.330+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v106'113 uv113 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.332 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:42:20.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.330+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b93050a0 con 0x55c1b924c130 2024-09-16T01:42:20.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.333+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v106'114 uv114 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.333+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b93050a0 con 0x55c1b924c130 2024-09-16T01:42:20.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.335+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v106'115 uv115 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.335+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b93050a0 con 0x55c1b924c130 2024-09-16T01:42:20.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.338+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v106'116 uv116 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.338+0000 7f02b201cf40 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55c1b9305b30 0x55c1b9305f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.338+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9308510 con 0x55c1b9305b30 2024-09-16T01:42:20.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.338+0000 7f02b0315640 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55c1b9305b30 0x55c1b9305f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.339+0000 7f02b0315640 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55c1b9305b30 0x55c1b9305f10 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:20.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.340+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v106'205 uv198 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.340+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9308510 con 0x55c1b9305b30 2024-09-16T01:42:20.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.341+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v106'206 uv206 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.341+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9308510 con 0x55c1b9305b30 2024-09-16T01:42:20.343 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:42:20.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.342+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v106'207 uv207 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.342+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9309560 con 0x55c1b9305b30 2024-09-16T01:42:20.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.343+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v106'208 uv208 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.343+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9309520 con 0x55c1b9305b30 2024-09-16T01:42:20.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.344+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v106'209 uv209 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.344+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9308510 con 0x55c1b9305b30 2024-09-16T01:42:20.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.345+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v106'210 uv210 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.345+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9308510 con 0x55c1b9305b30 2024-09-16T01:42:20.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.346+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v106'211 uv204 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.346+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9308510 con 0x55c1b9305b30 2024-09-16T01:42:20.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.347+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v106'212 uv212 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.347+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9308510 con 0x55c1b9305b30 2024-09-16T01:42:20.348 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:42:20.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.348+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v106'213 uv213 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.348+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9308510 con 0x55c1b9305b30 2024-09-16T01:42:20.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.349+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v106'214 uv214 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.349+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b9308510 con 0x55c1b9305b30 2024-09-16T01:42:20.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.350+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v106'215 uv215 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.350+0000 7f02b201cf40 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55c1b9308de0 0x55c1b9309220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.350+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930b950 con 0x55c1b9308de0 2024-09-16T01:42:20.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.350+0000 7f02afb14640 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55c1b9308de0 0x55c1b9309220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.351 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.350+0000 7f02afb14640 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55c1b9308de0 0x55c1b9309220 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:20.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.351+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v106'91 uv90 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b9308de0 2024-09-16T01:42:20.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.351+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930b950 con 0x55c1b9308de0 2024-09-16T01:42:20.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.352+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v106'92 uv92 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b9308de0 2024-09-16T01:42:20.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.352+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930b950 con 0x55c1b9308de0 2024-09-16T01:42:20.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.353+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v106'93 uv93 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b9308de0 2024-09-16T01:42:20.355 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:42:20.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.353+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930bd70 con 0x55c1b9308de0 2024-09-16T01:42:20.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.354+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v106'94 uv94 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b9308de0 2024-09-16T01:42:20.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.354+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930bd70 con 0x55c1b9308de0 2024-09-16T01:42:20.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.355+0000 7f02afb14640 1 -- 172.21.15.88:0/1051011209 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v106'95 uv95 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f02a001c040 con 0x55c1b9308de0 2024-09-16T01:42:20.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.355+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930bd70 con 0x55c1b924c130 2024-09-16T01:42:20.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.356+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v106'111 uv110 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.356+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930bd70 con 0x55c1b924c130 2024-09-16T01:42:20.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.357+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v106'112 uv112 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.357+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930bd70 con 0x55c1b924c130 2024-09-16T01:42:20.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.358+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v106'113 uv113 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.359 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:42:20.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.358+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930cd80 con 0x55c1b924c130 2024-09-16T01:42:20.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.359+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v106'114 uv114 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.359+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930cd80 con 0x55c1b924c130 2024-09-16T01:42:20.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.360+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v106'115 uv115 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.360+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930bd70 con 0x55c1b924c130 2024-09-16T01:42:20.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.361+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v106'116 uv116 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b924c130 2024-09-16T01:42:20.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.361+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930bd70 con 0x55c1b9301380 2024-09-16T01:42:20.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.362+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v106'182 uv176 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f0298020070 con 0x55c1b9301380 2024-09-16T01:42:20.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.362+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930c110 con 0x55c1b9301380 2024-09-16T01:42:20.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.363+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v106'183 uv183 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0298020070 con 0x55c1b9301380 2024-09-16T01:42:20.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.363+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930c4b0 con 0x55c1b9301380 2024-09-16T01:42:20.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.364+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v106'184 uv184 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0298020070 con 0x55c1b9301380 2024-09-16T01:42:20.365 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:42:20.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.364+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930c4b0 con 0x55c1b9301380 2024-09-16T01:42:20.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.365+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v106'185 uv185 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0298020070 con 0x55c1b9301380 2024-09-16T01:42:20.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.365+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930c4b0 con 0x55c1b9301380 2024-09-16T01:42:20.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.366+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v106'186 uv186 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0298020070 con 0x55c1b9301380 2024-09-16T01:42:20.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.366+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930c4b0 con 0x55c1b9301380 2024-09-16T01:42:20.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.366+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v106'187 uv181 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0298020070 con 0x55c1b9301380 2024-09-16T01:42:20.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.366+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930c4b0 con 0x55c1b9301380 2024-09-16T01:42:20.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.367+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v106'188 uv188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0298020070 con 0x55c1b9301380 2024-09-16T01:42:20.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.368+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930c4b0 con 0x55c1b9301380 2024-09-16T01:42:20.369 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:42:20.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.368+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v106'189 uv189 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0298020070 con 0x55c1b9301380 2024-09-16T01:42:20.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.368+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930c4b0 con 0x55c1b9301380 2024-09-16T01:42:20.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.369+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v106'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0298020070 con 0x55c1b9301380 2024-09-16T01:42:20.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.369+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f2c0 con 0x55c1b9301380 2024-09-16T01:42:20.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.370+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v106'191 uv191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0298020070 con 0x55c1b9301380 2024-09-16T01:42:20.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.370+0000 7f02b201cf40 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55c1b930cfe0 0x55c1b930f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.371+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b930cfe0 2024-09-16T01:42:20.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.371+0000 7f02a7313640 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55c1b930cfe0 0x55c1b930f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.371+0000 7f02a7313640 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55c1b930cfe0 0x55c1b930f8e0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:20.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.372+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v106'83 uv81 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f0298020070 con 0x55c1b930cfe0 2024-09-16T01:42:20.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.372+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b930cfe0 2024-09-16T01:42:20.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.373+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v106'84 uv84 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f0298020070 con 0x55c1b930cfe0 2024-09-16T01:42:20.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.373+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b930cfe0 2024-09-16T01:42:20.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.374+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v106'85 uv85 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f0298020070 con 0x55c1b930cfe0 2024-09-16T01:42:20.375 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:42:20.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.374+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b930cfe0 2024-09-16T01:42:20.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.375+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v106'86 uv86 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f0298020070 con 0x55c1b930cfe0 2024-09-16T01:42:20.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.375+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b9301380 2024-09-16T01:42:20.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.376+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v106'111 uv110 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0298020070 con 0x55c1b9301380 2024-09-16T01:42:20.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.376+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b9301380 2024-09-16T01:42:20.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.377+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v106'112 uv112 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0298020070 con 0x55c1b9301380 2024-09-16T01:42:20.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.377+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b9301380 2024-09-16T01:42:20.379 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:42:20.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.378+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v106'113 uv113 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0298020070 con 0x55c1b9301380 2024-09-16T01:42:20.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.378+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b9301380 2024-09-16T01:42:20.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.379+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v106'114 uv114 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0298020070 con 0x55c1b9301380 2024-09-16T01:42:20.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.379+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b9301380 2024-09-16T01:42:20.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.380+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v106'115 uv115 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0298020070 con 0x55c1b9301380 2024-09-16T01:42:20.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.380+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b9301380 2024-09-16T01:42:20.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.381+0000 7f02a7313640 1 -- 172.21.15.88:0/1051011209 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v106'116 uv116 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0298020070 con 0x55c1b9301380 2024-09-16T01:42:20.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.381+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b9305b30 2024-09-16T01:42:20.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.382+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v106'94 uv93 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.382+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b9305b30 2024-09-16T01:42:20.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.383+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v106'95 uv95 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.383+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b9305b30 2024-09-16T01:42:20.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.384+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v106'96 uv96 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.386 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:42:20.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.384+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b9305b30 2024-09-16T01:42:20.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.386+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v106'97 uv97 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.386+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b9305b30 2024-09-16T01:42:20.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.387+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v106'98 uv98 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.387+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b9305b30 2024-09-16T01:42:20.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.388+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v106'229 uv220 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.388+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b9305b30 2024-09-16T01:42:20.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.389+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v106'230 uv230 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.389+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b9305b30 2024-09-16T01:42:20.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.390+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v106'231 uv231 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.391 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:42:20.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.390+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b9305b30 2024-09-16T01:42:20.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.391+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v106'232 uv232 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.391+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b9305b30 2024-09-16T01:42:20.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.392+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v106'233 uv233 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.392+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b9305b30 2024-09-16T01:42:20.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.394+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v106'234 uv227 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.394+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b9305b30 2024-09-16T01:42:20.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.395+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v106'235 uv235 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.395+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b9305b30 2024-09-16T01:42:20.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.396+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v106'236 uv236 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.397 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:42:20.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.396+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b9305b30 2024-09-16T01:42:20.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.397+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v106'237 uv237 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.397+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b9305b30 2024-09-16T01:42:20.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.398+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v106'238 uv238 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.398+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930f3d0 con 0x55c1b9305b30 2024-09-16T01:42:20.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.399+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v106'239 uv239 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.399+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x55c1b930fe20 con 0x55c1b9305b30 2024-09-16T01:42:20.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v106'240 uv240 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f02a804e040 con 0x55c1b9305b30 2024-09-16T01:42:20.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55c1b924c130 msgr2=0x55c1b924caf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:20.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b201cf40 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55c1b924c130 0x55c1b924caf0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55c1b9301380 msgr2=0x55c1b9303770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:20.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b201cf40 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55c1b9301380 0x55c1b9303770 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55c1b930cfe0 msgr2=0x55c1b930f8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:20.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b201cf40 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55c1b930cfe0 0x55c1b930f8e0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55c1b92fbc40 msgr2=0x55c1b92fed50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:20.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b201cf40 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55c1b92fbc40 0x55c1b92fed50 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55c1b9305b30 msgr2=0x55c1b9305f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:20.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b201cf40 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55c1b9305b30 0x55c1b9305f10 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55c1b9308de0 msgr2=0x55c1b9309220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:20.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b201cf40 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55c1b9308de0 0x55c1b9309220 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 reap_dead start 2024-09-16T01:42:20.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 reap_dead start 2024-09-16T01:42:20.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b0315640 1 -- 172.21.15.88:0/1051011209 reap_dead start 2024-09-16T01:42:20.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f029006ac90 msgr2=0x7f029006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:20.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b201cf40 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f029006ac90 0x7f029006d150 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f02a0017040 tx=0x7f02a000c000 comp rx=0 tx=0).stop 2024-09-16T01:42:20.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c1b91b0a60 msgr2=0x55c1b91c9140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:20.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b201cf40 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c1b91b0a60 0x55c1b91c9140 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f029800ded0 tx=0x7f029800b5e0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 shutdown_connections 2024-09-16T01:42:20.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b201cf40 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f029006ac90 0x7f029006d150 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b201cf40 1 --2- 172.21.15.88:0/1051011209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55c1b91b0a60 0x55c1b91c9140 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 >> 172.21.15.88:0/1051011209 conn(0x55c1b91cb750 msgr2=0x55c1b925ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:20.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 shutdown_connections 2024-09-16T01:42:20.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.400+0000 7f02b201cf40 1 -- 172.21.15.88:0/1051011209 wait complete. 2024-09-16T01:42:20.405 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put mds_snaptable - 2024-09-16T01:42:20.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.487+0000 7f11c77eaf40 1 Processor -- start 2024-09-16T01:42:20.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.487+0000 7f11c77eaf40 1 -- start start 2024-09-16T01:42:20.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.488+0000 7f11c77eaf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5563302aaf00 0x55633037dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.488+0000 7f11c77eaf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556330360ed0 0x55633035edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.488+0000 7f11c77eaf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556330363190 0x55633035f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.488+0000 7f11c77eaf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5563301fa510 con 0x556330363190 2024-09-16T01:42:20.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.488+0000 7f11c77eaf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556330362210 con 0x556330360ed0 2024-09-16T01:42:20.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.488+0000 7f11c77eaf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5563302f1df0 con 0x5563302aaf00 2024-09-16T01:42:20.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.488+0000 7f11c5ae1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556330363190 0x55633035f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.488+0000 7f11c5ae1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556330363190 0x55633035f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49048/0 (socket says 172.21.15.88:49048) 2024-09-16T01:42:20.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.488+0000 7f11c5ae1640 1 -- 172.21.15.88:0/2795993816 learned_addr learned my addr 172.21.15.88:0/2795993816 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:20.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.488+0000 7f11c52e0640 1 --2- 172.21.15.88:0/2795993816 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5563302aaf00 0x55633037dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.488+0000 7f11bcadf640 1 --2- 172.21.15.88:0/2795993816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556330360ed0 0x55633035edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.488+0000 7f11c5ae1640 1 -- 172.21.15.88:0/2795993816 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5563302aaf00 msgr2=0x55633037dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:20.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.488+0000 7f11c5ae1640 1 --2- 172.21.15.88:0/2795993816 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5563302aaf00 0x55633037dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.488+0000 7f11c5ae1640 1 -- 172.21.15.88:0/2795993816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556330360ed0 msgr2=0x55633035edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:20.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.488+0000 7f11c5ae1640 1 --2- 172.21.15.88:0/2795993816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556330360ed0 0x55633035edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.488+0000 7f11c5ae1640 1 -- 172.21.15.88:0/2795993816 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556330366cb0 con 0x556330363190 2024-09-16T01:42:20.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.488+0000 7f11c5ae1640 1 --2- 172.21.15.88:0/2795993816 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556330363190 0x55633035f300 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f11c0001630 tx=0x7f11c0001af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=534753b61275b0a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:20.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.489+0000 7f11bffff640 1 -- 172.21.15.88:0/2795993816 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c0017070 con 0x556330363190 2024-09-16T01:42:20.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.489+0000 7f11bffff640 1 -- 172.21.15.88:0/2795993816 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11c0002690 con 0x556330363190 2024-09-16T01:42:20.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.489+0000 7f11bffff640 1 -- 172.21.15.88:0/2795993816 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c0002a80 con 0x556330363190 2024-09-16T01:42:20.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.489+0000 7f11c77eaf40 1 -- 172.21.15.88:0/2795993816 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556330363190 msgr2=0x55633035f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:20.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.489+0000 7f11c77eaf40 1 --2- 172.21.15.88:0/2795993816 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556330363190 0x55633035f300 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f11c0001630 tx=0x7f11c0001af0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.489+0000 7f11c77eaf40 1 -- 172.21.15.88:0/2795993816 shutdown_connections 2024-09-16T01:42:20.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.489+0000 7f11c77eaf40 1 --2- 172.21.15.88:0/2795993816 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556330363190 0x55633035f300 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.489+0000 7f11c77eaf40 1 --2- 172.21.15.88:0/2795993816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556330360ed0 0x55633035edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.489+0000 7f11c77eaf40 1 --2- 172.21.15.88:0/2795993816 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5563302aaf00 0x55633037dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.489+0000 7f11c77eaf40 1 -- 172.21.15.88:0/2795993816 >> 172.21.15.88:0/2795993816 conn(0x5563302a7be0 msgr2=0x5563303673d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:20.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.489+0000 7f11c77eaf40 1 -- 172.21.15.88:0/2795993816 shutdown_connections 2024-09-16T01:42:20.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.489+0000 7f11c77eaf40 1 -- 172.21.15.88:0/2795993816 wait complete. 2024-09-16T01:42:20.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.490+0000 7f11c77eaf40 1 Processor -- start 2024-09-16T01:42:20.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.490+0000 7f11c77eaf40 1 -- start start 2024-09-16T01:42:20.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.490+0000 7f11c77eaf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563302aaf00 0x55633035f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.490+0000 7f11c77eaf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556330360ed0 0x556330419290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.490+0000 7f11c77eaf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556330363190 0x5563302f0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.490+0000 7f11c77eaf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5563303663d0 con 0x5563302aaf00 2024-09-16T01:42:20.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.490+0000 7f11c77eaf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556330366f90 con 0x556330360ed0 2024-09-16T01:42:20.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.490+0000 7f11c77eaf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5563303668f0 con 0x556330363190 2024-09-16T01:42:20.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.491+0000 7f11c52e0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563302aaf00 0x55633035f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.491+0000 7f11c52e0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563302aaf00 0x55633035f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49058/0 (socket says 172.21.15.88:49058) 2024-09-16T01:42:20.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.491+0000 7f11c52e0640 1 -- 172.21.15.88:0/3517464892 learned_addr learned my addr 172.21.15.88:0/3517464892 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:20.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.491+0000 7f11bcadf640 1 --2- 172.21.15.88:0/3517464892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556330360ed0 0x556330419290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.491+0000 7f11c5ae1640 1 --2- 172.21.15.88:0/3517464892 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556330363190 0x5563302f0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.491+0000 7f11c52e0640 1 -- 172.21.15.88:0/3517464892 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556330363190 msgr2=0x5563302f0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:20.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.491+0000 7f11c52e0640 1 --2- 172.21.15.88:0/3517464892 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556330363190 0x5563302f0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.491+0000 7f11c52e0640 1 -- 172.21.15.88:0/3517464892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556330360ed0 msgr2=0x556330419290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:20.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.491+0000 7f11c52e0640 1 --2- 172.21.15.88:0/3517464892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556330360ed0 0x556330419290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.491+0000 7f11c52e0640 1 -- 172.21.15.88:0/3517464892 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563302e9c30 con 0x5563302aaf00 2024-09-16T01:42:20.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.491+0000 7f11c52e0640 1 --2- 172.21.15.88:0/3517464892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563302aaf00 0x55633035f750 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f11b8008740 tx=0x7f11b8008c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:20.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.491+0000 7f11beffd640 1 -- 172.21.15.88:0/3517464892 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b8019070 con 0x5563302aaf00 2024-09-16T01:42:20.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.491+0000 7f11beffd640 1 -- 172.21.15.88:0/3517464892 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11b8006b70 con 0x5563302aaf00 2024-09-16T01:42:20.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.491+0000 7f11c77eaf40 1 -- 172.21.15.88:0/3517464892 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5563302eee90 con 0x5563302aaf00 2024-09-16T01:42:20.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.491+0000 7f11beffd640 1 -- 172.21.15.88:0/3517464892 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b8014480 con 0x5563302aaf00 2024-09-16T01:42:20.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.491+0000 7f11c77eaf40 1 -- 172.21.15.88:0/3517464892 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563302ef450 con 0x5563302aaf00 2024-09-16T01:42:20.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.493+0000 7f11beffd640 1 -- 172.21.15.88:0/3517464892 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f11b80133e0 con 0x5563302aaf00 2024-09-16T01:42:20.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.493+0000 7f11beffd640 1 --2- 172.21.15.88:0/3517464892 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f11a806acc0 0x7f11a806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.493+0000 7f11beffd640 1 -- 172.21.15.88:0/3517464892 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 7372+0+0 (secure 0 0 0) 0x7f11b808a960 con 0x5563302aaf00 2024-09-16T01:42:20.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.493+0000 7f11bcadf640 1 --2- 172.21.15.88:0/3517464892 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f11a806acc0 0x7f11a806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.493+0000 7f11c77eaf40 1 --2- 172.21.15.88:0/3517464892 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x556330419c10 0x556330419ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.493+0000 7f11c77eaf40 1 -- 172.21.15.88:0/3517464892 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:1 12.d 12:b50e409b:::mds_snaptable:head [writefull 0~80 in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106) -- 0x5563304197d0 con 0x556330419c10 2024-09-16T01:42:20.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.493+0000 7f11bcadf640 1 --2- 172.21.15.88:0/3517464892 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f11a806acc0 0x7f11a806d180 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f11b0057000 tx=0x7f11b0052000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:20.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.494+0000 7f11c5ae1640 1 --2- 172.21.15.88:0/3517464892 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x556330419c10 0x556330419ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.494+0000 7f11c5ae1640 1 --2- 172.21.15.88:0/3517464892 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x556330419c10 0x556330419ff0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:20.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.496+0000 7f11c5ae1640 1 -- 172.21.15.88:0/3517464892 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(1 mds_snaptable [writefull 0~80] v106'96 uv96 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f11c000f070 con 0x556330419c10 2024-09-16T01:42:20.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.496+0000 7f11c77eaf40 1 -- 172.21.15.88:0/3517464892 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x556330419c10 msgr2=0x556330419ff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:20.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.496+0000 7f11c77eaf40 1 --2- 172.21.15.88:0/3517464892 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x556330419c10 0x556330419ff0 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.496+0000 7f11c77eaf40 1 -- 172.21.15.88:0/3517464892 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f11a806acc0 msgr2=0x7f11a806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:20.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.496+0000 7f11c77eaf40 1 --2- 172.21.15.88:0/3517464892 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f11a806acc0 0x7f11a806d180 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f11b0057000 tx=0x7f11b0052000 comp rx=0 tx=0).stop 2024-09-16T01:42:20.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.496+0000 7f11c77eaf40 1 -- 172.21.15.88:0/3517464892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563302aaf00 msgr2=0x55633035f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:20.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.496+0000 7f11c77eaf40 1 --2- 172.21.15.88:0/3517464892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5563302aaf00 0x55633035f750 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f11b8008740 tx=0x7f11b8008c00 comp rx=0 tx=0).stop 2024-09-16T01:42:20.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.496+0000 7f11c5ae1640 1 -- 172.21.15.88:0/3517464892 reap_dead start 2024-09-16T01:42:20.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.496+0000 7f11c77eaf40 1 -- 172.21.15.88:0/3517464892 shutdown_connections 2024-09-16T01:42:20.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.496+0000 7f11c77eaf40 1 -- 172.21.15.88:0/3517464892 >> 172.21.15.88:0/3517464892 conn(0x5563302a7be0 msgr2=0x556330364c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:20.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.496+0000 7f11c77eaf40 1 -- 172.21.15.88:0/3517464892 shutdown_connections 2024-09-16T01:42:20.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.496+0000 7f11c77eaf40 1 -- 172.21.15.88:0/3517464892 wait complete. 2024-09-16T01:42:20.502 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:42:20.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.644+0000 7f2873f59640 1 Processor -- start 2024-09-16T01:42:20.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.645+0000 7f2873f59640 1 -- start start 2024-09-16T01:42:20.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.645+0000 7f2873f59640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f286c155880 0x7f286c15f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.645+0000 7f2873f59640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f286c0b4d60 0x7f286c0b51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.645+0000 7f2873f59640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f286c15bb60 con 0x7f286c06f630 2024-09-16T01:42:20.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.645+0000 7f2873f59640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f286c155c60 con 0x7f286c0b4d60 2024-09-16T01:42:20.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.645+0000 7f2873f59640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f286c0b5700 con 0x7f286c155880 2024-09-16T01:42:20.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.646+0000 7f28724f6640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f286c06f630 0x7f286c06fa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43324/0 (socket says 172.21.15.88:43324) 2024-09-16T01:42:20.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.646+0000 7f28724f6640 1 -- 172.21.15.88:0/585213644 learned_addr learned my addr 172.21.15.88:0/585213644 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:20.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.646+0000 7f28714f4640 1 --2- 172.21.15.88:0/585213644 >> v2:172.21.15.154:3300/0 conn(0x7f286c0b4d60 0x7f286c0b51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.646+0000 7f2871cf5640 1 --2- 172.21.15.88:0/585213644 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f286c155880 0x7f286c15f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.646+0000 7f2870cf3640 1 -- 172.21.15.88:0/585213644 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1946696700 0 0) 0x7f286c15bb60 con 0x7f286c06f630 2024-09-16T01:42:20.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.646+0000 7f2870cf3640 1 -- 172.21.15.88:0/585213644 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2858003520 con 0x7f286c06f630 2024-09-16T01:42:20.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.646+0000 7f28714f4640 1 -- 172.21.15.88:0/585213644 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f286c155880 msgr2=0x7f286c15f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:20.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.646+0000 7f28714f4640 1 --2- 172.21.15.88:0/585213644 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f286c155880 0x7f286c15f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.646+0000 7f28714f4640 1 -- 172.21.15.88:0/585213644 >> v1:172.21.15.88:6789/0 conn(0x7f286c06f630 legacy=0x7f286c06fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:20.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.646+0000 7f28714f4640 1 -- 172.21.15.88:0/585213644 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f286c15be40 con 0x7f286c0b4d60 2024-09-16T01:42:20.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.646+0000 7f2871cf5640 1 --2- 172.21.15.88:0/585213644 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f286c155880 0x7f286c15f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:42:20.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.647+0000 7f28714f4640 1 --2- 172.21.15.88:0/585213644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f286c0b4d60 0x7f286c0b51c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f28680097a0 tx=0x7f286802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81bebce21bcc8ff5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:20.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.647+0000 7f2870cf3640 1 -- 172.21.15.88:0/585213644 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2868037070 con 0x7f286c0b4d60 2024-09-16T01:42:20.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.647+0000 7f2870cf3640 1 -- 172.21.15.88:0/585213644 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2868032440 con 0x7f286c0b4d60 2024-09-16T01:42:20.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.647+0000 7f2870cf3640 1 -- 172.21.15.88:0/585213644 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2868032810 con 0x7f286c0b4d60 2024-09-16T01:42:20.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.647+0000 7f2873f59640 1 -- 172.21.15.88:0/585213644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f286c0b4d60 msgr2=0x7f286c0b51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:20.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.647+0000 7f2873f59640 1 --2- 172.21.15.88:0/585213644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f286c0b4d60 0x7f286c0b51c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f28680097a0 tx=0x7f286802a180 comp rx=0 tx=0).stop 2024-09-16T01:42:20.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.648+0000 7f2873f59640 1 -- 172.21.15.88:0/585213644 shutdown_connections 2024-09-16T01:42:20.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.648+0000 7f2873f59640 1 --2- 172.21.15.88:0/585213644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f286c0b4d60 0x7f286c0b51c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.648+0000 7f2873f59640 1 --2- 172.21.15.88:0/585213644 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f286c155880 0x7f286c15f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.648+0000 7f2873f59640 1 -- 172.21.15.88:0/585213644 >> 172.21.15.88:0/585213644 conn(0x7f286c0d1390 msgr2=0x7f286c0d17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:20.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.648+0000 7f2873f59640 1 -- 172.21.15.88:0/585213644 shutdown_connections 2024-09-16T01:42:20.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.648+0000 7f2873f59640 1 -- 172.21.15.88:0/585213644 wait complete. 2024-09-16T01:42:20.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.649+0000 7f2873f59640 1 Processor -- start 2024-09-16T01:42:20.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.649+0000 7f2873f59640 1 -- start start 2024-09-16T01:42:20.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.649+0000 7f2873f59640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f286c06f630 0x7f286c14a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.649+0000 7f2873f59640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f286c0b4d60 0x7f286c153ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.649+0000 7f2873f59640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f286c155880 0x7f286c154000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.649+0000 7f2873f59640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f286c15c400 con 0x7f286c0b4d60 2024-09-16T01:42:20.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.649+0000 7f2873f59640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f286c15fcb0 con 0x7f286c06f630 2024-09-16T01:42:20.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.649+0000 7f2873f59640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f286c15c6e0 con 0x7f286c155880 2024-09-16T01:42:20.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.650+0000 7f28714f4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f286c0b4d60 0x7f286c153ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.650+0000 7f28714f4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f286c0b4d60 0x7f286c153ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49068/0 (socket says 172.21.15.88:49068) 2024-09-16T01:42:20.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.650+0000 7f28714f4640 1 -- 172.21.15.88:0/3256312854 learned_addr learned my addr 172.21.15.88:0/3256312854 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:20.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.650+0000 7f2871cf5640 1 --2- 172.21.15.88:0/3256312854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f286c06f630 0x7f286c14a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.650+0000 7f28724f6640 1 --2- 172.21.15.88:0/3256312854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f286c155880 0x7f286c154000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.650+0000 7f28714f4640 1 -- 172.21.15.88:0/3256312854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f286c155880 msgr2=0x7f286c154000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:20.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.650+0000 7f28714f4640 1 --2- 172.21.15.88:0/3256312854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f286c155880 0x7f286c154000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.650+0000 7f28714f4640 1 -- 172.21.15.88:0/3256312854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f286c06f630 msgr2=0x7f286c14a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:20.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.650+0000 7f28714f4640 1 --2- 172.21.15.88:0/3256312854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f286c06f630 0x7f286c14a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:20.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.650+0000 7f28714f4640 1 -- 172.21.15.88:0/3256312854 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f286c0cf6a0 con 0x7f286c0b4d60 2024-09-16T01:42:20.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.650+0000 7f28714f4640 1 --2- 172.21.15.88:0/3256312854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f286c0b4d60 0x7f286c153ac0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f28680098c0 tx=0x7f28680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:20.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.650+0000 7f285e7fc640 1 -- 172.21.15.88:0/3256312854 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2868041030 con 0x7f286c0b4d60 2024-09-16T01:42:20.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.650+0000 7f285e7fc640 1 -- 172.21.15.88:0/3256312854 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f286803d4e0 con 0x7f286c0b4d60 2024-09-16T01:42:20.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.650+0000 7f2873f59640 1 -- 172.21.15.88:0/3256312854 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f286c0cf980 con 0x7f286c0b4d60 2024-09-16T01:42:20.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.650+0000 7f285e7fc640 1 -- 172.21.15.88:0/3256312854 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2868037040 con 0x7f286c0b4d60 2024-09-16T01:42:20.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.650+0000 7f2873f59640 1 -- 172.21.15.88:0/3256312854 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f286c151c70 con 0x7f286c0b4d60 2024-09-16T01:42:20.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.651+0000 7f2873f59640 1 -- 172.21.15.88:0/3256312854 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f286c15b770 con 0x7f286c0b4d60 2024-09-16T01:42:20.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.652+0000 7f285e7fc640 1 -- 172.21.15.88:0/3256312854 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2868044070 con 0x7f286c0b4d60 2024-09-16T01:42:20.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.652+0000 7f285e7fc640 1 --2- 172.21.15.88:0/3256312854 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f285006c2b0 0x7f285006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:20.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.652+0000 7f2871cf5640 1 --2- 172.21.15.88:0/3256312854 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f285006c2b0 0x7f285006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:20.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.652+0000 7f285e7fc640 1 -- 172.21.15.88:0/3256312854 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 7372+0+0 (secure 0 0 0) 0x7f28680abff0 con 0x7f286c0b4d60 2024-09-16T01:42:20.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.653+0000 7f2871cf5640 1 --2- 172.21.15.88:0/3256312854 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f285006c2b0 0x7f285006e770 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2860009560 tx=0x7f2860009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:20.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.657+0000 7f285e7fc640 1 -- 172.21.15.88:0/3256312854 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2868079aa0 con 0x7f286c0b4d60 2024-09-16T01:42:20.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:20.855+0000 7f2873f59640 1 -- 172.21.15.88:0/3256312854 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f286c06bcb0 con 0x7f286c0b4d60 2024-09-16T01:42:21.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.474+0000 7f285e7fc640 1 -- 172.21.15.88:0/3256312854 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v186) ==== 162+0+0 (secure 0 0 0) 0x7f286807dda0 con 0x7f286c0b4d60 2024-09-16T01:42:21.476 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:42:21.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.477+0000 7f2873f59640 1 -- 172.21.15.88:0/3256312854 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f285006c2b0 msgr2=0x7f285006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:21.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.477+0000 7f2873f59640 1 --2- 172.21.15.88:0/3256312854 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f285006c2b0 0x7f285006e770 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2860009560 tx=0x7f2860009190 comp rx=0 tx=0).stop 2024-09-16T01:42:21.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.477+0000 7f2873f59640 1 -- 172.21.15.88:0/3256312854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f286c0b4d60 msgr2=0x7f286c153ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:21.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.477+0000 7f2873f59640 1 --2- 172.21.15.88:0/3256312854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f286c0b4d60 0x7f286c153ac0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f28680098c0 tx=0x7f28680046c0 comp rx=0 tx=0).stop 2024-09-16T01:42:21.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.478+0000 7f2873f59640 1 -- 172.21.15.88:0/3256312854 shutdown_connections 2024-09-16T01:42:21.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.478+0000 7f2873f59640 1 --2- 172.21.15.88:0/3256312854 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f285006c2b0 0x7f285006e770 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:21.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.478+0000 7f2873f59640 1 --2- 172.21.15.88:0/3256312854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f286c155880 0x7f286c154000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:21.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.478+0000 7f2873f59640 1 --2- 172.21.15.88:0/3256312854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f286c0b4d60 0x7f286c153ac0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:21.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.478+0000 7f2873f59640 1 --2- 172.21.15.88:0/3256312854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f286c06f630 0x7f286c14a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:21.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.478+0000 7f2873f59640 1 -- 172.21.15.88:0/3256312854 >> 172.21.15.88:0/3256312854 conn(0x7f286c0d1390 msgr2=0x7f286c06c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:21.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.478+0000 7f2873f59640 1 -- 172.21.15.88:0/3256312854 shutdown_connections 2024-09-16T01:42:21.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.478+0000 7f2873f59640 1 -- 172.21.15.88:0/3256312854 wait complete. 2024-09-16T01:42:21.485 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:42:21.484+0000 7fc5fd4be640 -1 log_channel(cluster) log [ERR] : error decoding table object 'mds_snaptable': Decoder at 'virtual void SnapServer::decode_server_state(ceph::buffer::v15_2_0::list::const_iterator&)' v=5 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-09-16T01:42:21.486 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:42:21.485+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:42:21.486 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:42:21.485+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:42:21.495 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:21.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.597+0000 7f17e381e640 1 Processor -- start 2024-09-16T01:42:21.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.597+0000 7f17e381e640 1 -- start start 2024-09-16T01:42:21.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.598+0000 7f17e381e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f17dc0d3310 0x7f17dc0d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:21.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.598+0000 7f17e381e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17dc15ea40 con 0x7f17dc0d1650 2024-09-16T01:42:21.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.598+0000 7f17e381e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17dc0d3870 con 0x7f17dc15e660 2024-09-16T01:42:21.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.598+0000 7f17e381e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f17dc0d1ac0 con 0x7f17dc0d3310 2024-09-16T01:42:21.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.598+0000 7f17e0db9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f17dc0d3310 0x7f17dc0d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:21.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.598+0000 7f17e0db9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f17dc0d3310 0x7f17dc0d1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49072/0 (socket says 172.21.15.88:49072) 2024-09-16T01:42:21.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.598+0000 7f17e1dbb640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f17dc0d1650 0x7f17dc0b7b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43326/0 (socket says 172.21.15.88:43326) 2024-09-16T01:42:21.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.598+0000 7f17e0db9640 1 -- 172.21.15.88:0/1547106031 learned_addr learned my addr 172.21.15.88:0/1547106031 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:21.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.598+0000 7f17e0db9640 1 -- 172.21.15.88:0/1547106031 >> v1:172.21.15.154:6789/0 conn(0x7f17dc15e660 legacy=0x7f17dc0cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:21.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.598+0000 7f17e0db9640 1 -- 172.21.15.88:0/1547106031 >> v1:172.21.15.88:6789/0 conn(0x7f17dc0d1650 legacy=0x7f17dc0b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:21.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.598+0000 7f17e0db9640 1 -- 172.21.15.88:0/1547106031 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17dc0d3b50 con 0x7f17dc0d3310 2024-09-16T01:42:21.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.598+0000 7f17d3fff640 1 -- 172.21.15.88:0/1547106031 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 696734578 0 0) 0x7f17dc15ea40 con 0x7f17dc0d1650 2024-09-16T01:42:21.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.599+0000 7f17e0db9640 1 --2- 172.21.15.88:0/1547106031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17dc0d3310 0x7f17dc0d1110 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009870 tx=0x7f17cc02a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69b1375d3c904c8a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:21.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.599+0000 7f17d3fff640 1 -- 172.21.15.88:0/1547106031 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc037070 con 0x7f17dc0d3310 2024-09-16T01:42:21.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.599+0000 7f17d3fff640 1 -- 172.21.15.88:0/1547106031 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17cc032440 con 0x7f17dc0d3310 2024-09-16T01:42:21.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.599+0000 7f17d3fff640 1 -- 172.21.15.88:0/1547106031 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc032810 con 0x7f17dc0d3310 2024-09-16T01:42:21.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.599+0000 7f17e381e640 1 -- 172.21.15.88:0/1547106031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17dc0d3310 msgr2=0x7f17dc0d1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:21.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.599+0000 7f17e381e640 1 --2- 172.21.15.88:0/1547106031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17dc0d3310 0x7f17dc0d1110 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009870 tx=0x7f17cc02a2a0 comp rx=0 tx=0).stop 2024-09-16T01:42:21.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.600+0000 7f17e381e640 1 -- 172.21.15.88:0/1547106031 shutdown_connections 2024-09-16T01:42:21.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.600+0000 7f17e381e640 1 --2- 172.21.15.88:0/1547106031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17dc0d3310 0x7f17dc0d1110 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:21.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.600+0000 7f17e381e640 1 -- 172.21.15.88:0/1547106031 >> 172.21.15.88:0/1547106031 conn(0x7f17dc0b6410 msgr2=0x7f17dc0b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:21.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.600+0000 7f17e381e640 1 -- 172.21.15.88:0/1547106031 shutdown_connections 2024-09-16T01:42:21.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.600+0000 7f17e381e640 1 -- 172.21.15.88:0/1547106031 wait complete. 2024-09-16T01:42:21.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.601+0000 7f17e381e640 1 Processor -- start 2024-09-16T01:42:21.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.601+0000 7f17e381e640 1 -- start start 2024-09-16T01:42:21.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.601+0000 7f17e381e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17dc0d1650 0x7f17dc1f50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:21.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.602+0000 7f17e381e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17dc0d3310 0x7f17dc158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:21.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.602+0000 7f17e381e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17dc15e660 0x7f17dc0cf520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:21.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.602+0000 7f17e381e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f17dc0d43f0 con 0x7f17dc0d3310 2024-09-16T01:42:21.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.602+0000 7f17e381e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f17dc0d4110 con 0x7f17dc15e660 2024-09-16T01:42:21.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.602+0000 7f17e381e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f17dc0b8280 con 0x7f17dc0d1650 2024-09-16T01:42:21.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.602+0000 7f17e0db9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17dc0d3310 0x7f17dc158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:21.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.602+0000 7f17e0db9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17dc0d3310 0x7f17dc158c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49086/0 (socket says 172.21.15.88:49086) 2024-09-16T01:42:21.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.602+0000 7f17e0db9640 1 -- 172.21.15.88:0/1100420916 learned_addr learned my addr 172.21.15.88:0/1100420916 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:21.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.602+0000 7f17e15ba640 1 --2- 172.21.15.88:0/1100420916 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17dc0d1650 0x7f17dc1f50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:21.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.602+0000 7f17e1dbb640 1 --2- 172.21.15.88:0/1100420916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17dc15e660 0x7f17dc0cf520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:21.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.602+0000 7f17e0db9640 1 -- 172.21.15.88:0/1100420916 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17dc0d1650 msgr2=0x7f17dc1f50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:21.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.602+0000 7f17e0db9640 1 --2- 172.21.15.88:0/1100420916 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17dc0d1650 0x7f17dc1f50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:21.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.602+0000 7f17e0db9640 1 -- 172.21.15.88:0/1100420916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17dc15e660 msgr2=0x7f17dc0cf520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:21.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.602+0000 7f17e0db9640 1 --2- 172.21.15.88:0/1100420916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17dc15e660 0x7f17dc0cf520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:21.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.602+0000 7f17e0db9640 1 -- 172.21.15.88:0/1100420916 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17dc0cfd50 con 0x7f17dc0d3310 2024-09-16T01:42:21.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.602+0000 7f17e0db9640 1 --2- 172.21.15.88:0/1100420916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17dc0d3310 0x7f17dc158c50 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009870 tx=0x7f17cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:21.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.602+0000 7f17d1ffb640 1 -- 172.21.15.88:0/1100420916 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc037040 con 0x7f17dc0d3310 2024-09-16T01:42:21.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.602+0000 7f17d1ffb640 1 -- 172.21.15.88:0/1100420916 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17cc004250 con 0x7f17dc0d3310 2024-09-16T01:42:21.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.602+0000 7f17d1ffb640 1 -- 172.21.15.88:0/1100420916 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc03c670 con 0x7f17dc0d3310 2024-09-16T01:42:21.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.602+0000 7f17e381e640 1 -- 172.21.15.88:0/1100420916 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17dc0ccf10 con 0x7f17dc0d3310 2024-09-16T01:42:21.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.602+0000 7f17e381e640 1 -- 172.21.15.88:0/1100420916 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17dc0cd4d0 con 0x7f17dc0d3310 2024-09-16T01:42:21.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.604+0000 7f17e381e640 1 -- 172.21.15.88:0/1100420916 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17dc0d1110 con 0x7f17dc0d3310 2024-09-16T01:42:21.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.604+0000 7f17d1ffb640 1 -- 172.21.15.88:0/1100420916 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f17cc03a070 con 0x7f17dc0d3310 2024-09-16T01:42:21.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.605+0000 7f17d1ffb640 1 --2- 172.21.15.88:0/1100420916 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f17c006c2b0 0x7f17c006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:21.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.605+0000 7f17e15ba640 1 --2- 172.21.15.88:0/1100420916 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f17c006c2b0 0x7f17c006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:21.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.605+0000 7f17d1ffb640 1 -- 172.21.15.88:0/1100420916 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 7458+0+0 (secure 0 0 0) 0x7f17cc0ac4c0 con 0x7f17dc0d3310 2024-09-16T01:42:21.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.605+0000 7f17e15ba640 1 --2- 172.21.15.88:0/1100420916 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f17c006c2b0 0x7f17c006e770 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f17d8006fd0 tx=0x7f17d8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:21.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.609+0000 7f17d1ffb640 1 -- 172.21.15.88:0/1100420916 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f17cc041020 con 0x7f17dc0d3310 2024-09-16T01:42:21.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.808+0000 7f17e381e640 1 -- 172.21.15.88:0/1100420916 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17dc0b4360 con 0x7f17dc0d3310 2024-09-16T01:42:21.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.808+0000 7f17d1ffb640 1 -- 172.21.15.88:0/1100420916 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) ==== 95+0+4392 (secure 0 0 0) 0x7f17cc07a550 con 0x7f17dc0d3310 2024-09-16T01:42:21.810 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:21.810 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":188,"btime":"2024-09-16T01:42:21:486016+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8736,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3374372758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3374372758},{"type":"v1","addr":"172.21.15.88:6836","nonce":3374372758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":181}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:21.485952+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:21.810 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 188 2024-09-16T01:42:21.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.810+0000 7f17e381e640 1 -- 172.21.15.88:0/1100420916 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f17c006c2b0 msgr2=0x7f17c006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:21.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.810+0000 7f17e381e640 1 --2- 172.21.15.88:0/1100420916 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f17c006c2b0 0x7f17c006e770 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f17d8006fd0 tx=0x7f17d8008040 comp rx=0 tx=0).stop 2024-09-16T01:42:21.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.810+0000 7f17e381e640 1 -- 172.21.15.88:0/1100420916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17dc0d3310 msgr2=0x7f17dc158c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:21.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.810+0000 7f17e381e640 1 --2- 172.21.15.88:0/1100420916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17dc0d3310 0x7f17dc158c50 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009870 tx=0x7f17cc0046c0 comp rx=0 tx=0).stop 2024-09-16T01:42:21.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.810+0000 7f17e381e640 1 -- 172.21.15.88:0/1100420916 shutdown_connections 2024-09-16T01:42:21.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.810+0000 7f17e381e640 1 --2- 172.21.15.88:0/1100420916 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f17c006c2b0 0x7f17c006e770 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:21.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.810+0000 7f17e381e640 1 --2- 172.21.15.88:0/1100420916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17dc15e660 0x7f17dc0cf520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:21.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.810+0000 7f17e381e640 1 --2- 172.21.15.88:0/1100420916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17dc0d3310 0x7f17dc158c50 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:21.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.810+0000 7f17e381e640 1 --2- 172.21.15.88:0/1100420916 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f17dc0d1650 0x7f17dc1f50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:21.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.810+0000 7f17e381e640 1 -- 172.21.15.88:0/1100420916 >> 172.21.15.88:0/1100420916 conn(0x7f17dc0b6410 msgr2=0x7f17dc15bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:21.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.810+0000 7f17e381e640 1 -- 172.21.15.88:0/1100420916 shutdown_connections 2024-09-16T01:42:21.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.810+0000 7f17e381e640 1 -- 172.21.15.88:0/1100420916 wait complete. 2024-09-16T01:42:21.823 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:42:21.824 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:21.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.921+0000 7fe50fb86640 1 Processor -- start 2024-09-16T01:42:21.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.922+0000 7fe50fb86640 1 -- start start 2024-09-16T01:42:21.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.922+0000 7fe50fb86640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe50815e660 0x7fe5080cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:21.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.922+0000 7fe50fb86640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe5080d1820 0x7fe5080b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:21.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.922+0000 7fe50fb86640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe50815ea40 con 0x7fe5080d3310 2024-09-16T01:42:21.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.922+0000 7fe50fb86640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe5080d3870 con 0x7fe5080d1820 2024-09-16T01:42:21.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.922+0000 7fe50fb86640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe5080d3b50 con 0x7fe50815e660 2024-09-16T01:42:21.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.923+0000 7fe50e123640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe5080d1820 0x7fe5080b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:21.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.923+0000 7fe50e123640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe5080d1820 0x7fe5080b7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49092/0 (socket says 172.21.15.88:49092) 2024-09-16T01:42:21.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.923+0000 7fe50e123640 1 -- 172.21.15.88:0/330574333 learned_addr learned my addr 172.21.15.88:0/330574333 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:21.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.923+0000 7fe50e123640 1 -- 172.21.15.88:0/330574333 >> v2:172.21.15.154:3300/0 conn(0x7fe50815e660 msgr2=0x7fe5080cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:21.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.923+0000 7fe50d922640 1 --2- 172.21.15.88:0/330574333 >> v2:172.21.15.154:3300/0 conn(0x7fe50815e660 0x7fe5080cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:21.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.923+0000 7fe50e123640 1 --2- 172.21.15.88:0/330574333 >> v2:172.21.15.154:3300/0 conn(0x7fe50815e660 0x7fe5080cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:21.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.923+0000 7fe50e123640 1 -- 172.21.15.88:0/330574333 >> v1:172.21.15.154:6789/0 conn(0x7fe5080d3310 legacy=0x7fe5080d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:21.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.923+0000 7fe50e123640 1 -- 172.21.15.88:0/330574333 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5080d3e30 con 0x7fe5080d1820 2024-09-16T01:42:21.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.923+0000 7fe50d922640 1 --2- 172.21.15.88:0/330574333 >> v2:172.21.15.154:3300/0 conn(0x7fe50815e660 0x7fe5080cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:21.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.924+0000 7fe50e123640 1 --2- 172.21.15.88:0/330574333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5080d1820 0x7fe5080b7b70 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fe5080d10e0 tx=0x7fe50002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8deeb079ab4f8cc0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:21.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.924+0000 7fe50c920640 1 -- 172.21.15.88:0/330574333 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe500036070 con 0x7fe5080d1820 2024-09-16T01:42:21.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.924+0000 7fe50c920640 1 -- 172.21.15.88:0/330574333 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe500031600 con 0x7fe5080d1820 2024-09-16T01:42:21.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.924+0000 7fe50c920640 1 -- 172.21.15.88:0/330574333 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5000319d0 con 0x7fe5080d1820 2024-09-16T01:42:21.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.924+0000 7fe50fb86640 1 -- 172.21.15.88:0/330574333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5080d1820 msgr2=0x7fe5080b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:21.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.924+0000 7fe50fb86640 1 --2- 172.21.15.88:0/330574333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5080d1820 0x7fe5080b7b70 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fe5080d10e0 tx=0x7fe50002a6f0 comp rx=0 tx=0).stop 2024-09-16T01:42:21.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.924+0000 7fe50fb86640 1 -- 172.21.15.88:0/330574333 shutdown_connections 2024-09-16T01:42:21.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.924+0000 7fe50fb86640 1 --2- 172.21.15.88:0/330574333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5080d1820 0x7fe5080b7b70 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:21.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.924+0000 7fe50fb86640 1 --2- 172.21.15.88:0/330574333 >> v2:172.21.15.154:3300/0 conn(0x7fe50815e660 0x7fe5080cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:21.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.924+0000 7fe50fb86640 1 -- 172.21.15.88:0/330574333 >> 172.21.15.88:0/330574333 conn(0x7fe5080b6410 msgr2=0x7fe5080b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:21.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.924+0000 7fe50fb86640 1 -- 172.21.15.88:0/330574333 shutdown_connections 2024-09-16T01:42:21.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.924+0000 7fe50fb86640 1 -- 172.21.15.88:0/330574333 wait complete. 2024-09-16T01:42:21.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.926+0000 7fe50fb86640 1 Processor -- start 2024-09-16T01:42:21.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.926+0000 7fe50fb86640 1 -- start start 2024-09-16T01:42:21.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.926+0000 7fe50fb86640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5080d1820 0x7fe5081f3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:21.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.926+0000 7fe50fb86640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5080d3310 0x7fe50815a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:21.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.926+0000 7fe50fb86640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe50815e660 0x7fe50815ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:21.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.926+0000 7fe50fb86640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe5080cfc70 con 0x7fe5080d1820 2024-09-16T01:42:21.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.926+0000 7fe50fb86640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe5080d43f0 con 0x7fe50815e660 2024-09-16T01:42:21.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.926+0000 7fe50fb86640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe5080b80b0 con 0x7fe5080d3310 2024-09-16T01:42:21.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.926+0000 7fe50d922640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5080d1820 0x7fe5081f3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:21.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.926+0000 7fe50d922640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5080d1820 0x7fe5081f3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49108/0 (socket says 172.21.15.88:49108) 2024-09-16T01:42:21.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.926+0000 7fe50d922640 1 -- 172.21.15.88:0/3522448463 learned_addr learned my addr 172.21.15.88:0/3522448463 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:21.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.927+0000 7fe50e123640 1 --2- 172.21.15.88:0/3522448463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe50815e660 0x7fe50815ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:21.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.927+0000 7fe50d121640 1 --2- 172.21.15.88:0/3522448463 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5080d3310 0x7fe50815a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:21.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.927+0000 7fe50d922640 1 -- 172.21.15.88:0/3522448463 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5080d3310 msgr2=0x7fe50815a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:21.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.927+0000 7fe50d922640 1 --2- 172.21.15.88:0/3522448463 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5080d3310 0x7fe50815a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:21.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.927+0000 7fe50d922640 1 -- 172.21.15.88:0/3522448463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe50815e660 msgr2=0x7fe50815ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:21.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.927+0000 7fe50d922640 1 --2- 172.21.15.88:0/3522448463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe50815e660 0x7fe50815ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:21.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.927+0000 7fe50d922640 1 -- 172.21.15.88:0/3522448463 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe50815b4c0 con 0x7fe5080d1820 2024-09-16T01:42:21.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.927+0000 7fe50e123640 1 --2- 172.21.15.88:0/3522448463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe50815e660 0x7fe50815ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:21.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.927+0000 7fe50d922640 1 --2- 172.21.15.88:0/3522448463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5080d1820 0x7fe5081f3dc0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc00c7f0 tx=0x7fe4fc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:21.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.927+0000 7fe4fa7fc640 1 -- 172.21.15.88:0/3522448463 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc015070 con 0x7fe5080d1820 2024-09-16T01:42:21.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.927+0000 7fe4fa7fc640 1 -- 172.21.15.88:0/3522448463 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4fc007820 con 0x7fe5080d1820 2024-09-16T01:42:21.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.927+0000 7fe50fb86640 1 -- 172.21.15.88:0/3522448463 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe50815ba80 con 0x7fe5080d1820 2024-09-16T01:42:21.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.927+0000 7fe4fa7fc640 1 -- 172.21.15.88:0/3522448463 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc007bd0 con 0x7fe5080d1820 2024-09-16T01:42:21.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.927+0000 7fe50fb86640 1 -- 172.21.15.88:0/3522448463 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe50815c040 con 0x7fe5080d1820 2024-09-16T01:42:21.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.928+0000 7fe50fb86640 1 -- 172.21.15.88:0/3522448463 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5080b7b70 con 0x7fe5080d1820 2024-09-16T01:42:21.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.929+0000 7fe4fa7fc640 1 -- 172.21.15.88:0/3522448463 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe4fc018070 con 0x7fe5080d1820 2024-09-16T01:42:21.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.929+0000 7fe4fa7fc640 1 --2- 172.21.15.88:0/3522448463 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe4ec06c2b0 0x7fe4ec06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:21.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.929+0000 7fe4fa7fc640 1 -- 172.21.15.88:0/3522448463 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 7458+0+0 (secure 0 0 0) 0x7fe4fc08bc80 con 0x7fe5080d1820 2024-09-16T01:42:21.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.929+0000 7fe50d121640 1 --2- 172.21.15.88:0/3522448463 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe4ec06c2b0 0x7fe4ec06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:21.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.929+0000 7fe50d121640 1 --2- 172.21.15.88:0/3522448463 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe4ec06c2b0 0x7fe4ec06e770 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fe508156d90 tx=0x7fe504009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:21.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:21.933+0000 7fe4fa7fc640 1 -- 172.21.15.88:0/3522448463 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe4fc059d10 con 0x7fe5080d1820 2024-09-16T01:42:22.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.134+0000 7fe50fb86640 1 -- 172.21.15.88:0/3522448463 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5081f4360 con 0x7fe5080d1820 2024-09-16T01:42:22.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.135+0000 7fe4fa7fc640 1 -- 172.21.15.88:0/3522448463 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) ==== 95+0+4392 (secure 0 0 0) 0x7fe4fc059610 con 0x7fe5080d1820 2024-09-16T01:42:22.136 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:22.137 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":188,"btime":"2024-09-16T01:42:21:486016+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8736,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3374372758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3374372758},{"type":"v1","addr":"172.21.15.88:6836","nonce":3374372758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":181}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:21.485952+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:22.137 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 188 2024-09-16T01:42:22.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.137+0000 7fe50fb86640 1 -- 172.21.15.88:0/3522448463 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe4ec06c2b0 msgr2=0x7fe4ec06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:22.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.137+0000 7fe50fb86640 1 --2- 172.21.15.88:0/3522448463 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe4ec06c2b0 0x7fe4ec06e770 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fe508156d90 tx=0x7fe504009040 comp rx=0 tx=0).stop 2024-09-16T01:42:22.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.137+0000 7fe50fb86640 1 -- 172.21.15.88:0/3522448463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5080d1820 msgr2=0x7fe5081f3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:22.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.137+0000 7fe50fb86640 1 --2- 172.21.15.88:0/3522448463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5080d1820 0x7fe5081f3dc0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc00c7f0 tx=0x7fe4fc00ccb0 comp rx=0 tx=0).stop 2024-09-16T01:42:22.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.137+0000 7fe50fb86640 1 -- 172.21.15.88:0/3522448463 shutdown_connections 2024-09-16T01:42:22.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.137+0000 7fe50fb86640 1 --2- 172.21.15.88:0/3522448463 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe4ec06c2b0 0x7fe4ec06e770 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:22.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.137+0000 7fe50fb86640 1 --2- 172.21.15.88:0/3522448463 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe50815e660 0x7fe50815ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:22.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.137+0000 7fe50fb86640 1 --2- 172.21.15.88:0/3522448463 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5080d3310 0x7fe50815a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:22.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.137+0000 7fe50fb86640 1 --2- 172.21.15.88:0/3522448463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5080d1820 0x7fe5081f3dc0 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:22.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.137+0000 7fe50fb86640 1 -- 172.21.15.88:0/3522448463 >> 172.21.15.88:0/3522448463 conn(0x7fe5080b6410 msgr2=0x7fe508158650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:22.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.137+0000 7fe50fb86640 1 -- 172.21.15.88:0/3522448463 shutdown_connections 2024-09-16T01:42:22.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.137+0000 7fe50fb86640 1 -- 172.21.15.88:0/3522448463 wait complete. 2024-09-16T01:42:22.149 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:42:22.150 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:22.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.247+0000 7f607c8a5640 1 Processor -- start 2024-09-16T01:42:22.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.247+0000 7f607c8a5640 1 -- start start 2024-09-16T01:42:22.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.248+0000 7f607c8a5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f607806e780 0x7f60780ae890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:22.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.248+0000 7f607c8a5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6078073130 0x7f6078073580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:22.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.248+0000 7f607c8a5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60780b5220 con 0x7f607815e560 2024-09-16T01:42:22.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.248+0000 7f607c8a5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f607806ebc0 con 0x7f607806e780 2024-09-16T01:42:22.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.248+0000 7f607c8a5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6078073ac0 con 0x7f6078073130 2024-09-16T01:42:22.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.248+0000 7f607659c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f607815e560 0x7f6078072220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43338/0 (socket says 172.21.15.88:43338) 2024-09-16T01:42:22.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.248+0000 7f607659c640 1 -- 172.21.15.88:0/95871415 learned_addr learned my addr 172.21.15.88:0/95871415 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:22.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.248+0000 7f6076d9d640 1 --2- 172.21.15.88:0/95871415 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6078073130 0x7f6078073580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:22.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.248+0000 7f6075d9b640 1 --2- 172.21.15.88:0/95871415 >> v2:172.21.15.154:3300/0 conn(0x7f607806e780 0x7f60780ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:22.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.248+0000 7f607559a640 1 -- 172.21.15.88:0/95871415 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 982066714 0 0) 0x7f60780b5220 con 0x7f607815e560 2024-09-16T01:42:22.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.248+0000 7f607559a640 1 -- 172.21.15.88:0/95871415 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6060003520 con 0x7f607815e560 2024-09-16T01:42:22.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.249+0000 7f607559a640 1 -- 172.21.15.88:0/95871415 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 5304157 0 0) 0x7f6060003520 con 0x7f607815e560 2024-09-16T01:42:22.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.249+0000 7f607559a640 1 -- 172.21.15.88:0/95871415 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6078073130 msgr2=0x7f6078073580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:22.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.249+0000 7f607559a640 1 --2- 172.21.15.88:0/95871415 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6078073130 0x7f6078073580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:22.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.249+0000 7f607559a640 1 -- 172.21.15.88:0/95871415 >> v2:172.21.15.154:3300/0 conn(0x7f607806e780 msgr2=0x7f60780ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:22.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.249+0000 7f607559a640 1 --2- 172.21.15.88:0/95871415 >> v2:172.21.15.154:3300/0 conn(0x7f607806e780 0x7f60780ae890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:22.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.249+0000 7f607559a640 1 -- 172.21.15.88:0/95871415 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60780b54c0 con 0x7f607815e560 2024-09-16T01:42:22.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.249+0000 7f607559a640 1 -- 172.21.15.88:0/95871415 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f6068001670 con 0x7f607815e560 2024-09-16T01:42:22.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.249+0000 7f607559a640 1 -- 172.21.15.88:0/95871415 >> v1:172.21.15.88:6789/0 conn(0x7f607815e560 legacy=0x7f6078072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:22.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.249+0000 7f607559a640 1 --2- 172.21.15.88:0/95871415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60600039b0 0x7f6060003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:22.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.249+0000 7f607559a640 1 --2- 172.21.15.88:0/95871415 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6078073130 0x7f6060004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:22.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.249+0000 7f607559a640 1 --2- 172.21.15.88:0/95871415 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6060004d40 0x7f60600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:22.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.249+0000 7f607559a640 1 -- 172.21.15.88:0/95871415 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f607806ebc0 con 0x7f60600039b0 2024-09-16T01:42:22.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.249+0000 7f607559a640 1 -- 172.21.15.88:0/95871415 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6078073ac0 con 0x7f6060004d40 2024-09-16T01:42:22.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.249+0000 7f607559a640 1 -- 172.21.15.88:0/95871415 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f606000b9c0 con 0x7f6078073130 2024-09-16T01:42:22.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.249+0000 7f607659c640 1 --2- 172.21.15.88:0/95871415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60600039b0 0x7f6060003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:22.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.249+0000 7f6075d9b640 1 --2- 172.21.15.88:0/95871415 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6078073130 0x7f6060004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:22.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.249+0000 7f6076d9d640 1 --2- 172.21.15.88:0/95871415 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6060004d40 0x7f60600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:22.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.249+0000 7f607659c640 1 -- 172.21.15.88:0/95871415 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6078073130 msgr2=0x7f6060004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:22.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.249+0000 7f607659c640 1 --2- 172.21.15.88:0/95871415 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6078073130 0x7f6060004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:22.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.249+0000 7f607659c640 1 -- 172.21.15.88:0/95871415 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6060004d40 msgr2=0x7f60600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:22.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.249+0000 7f607659c640 1 --2- 172.21.15.88:0/95871415 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6060004d40 0x7f60600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:22.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.249+0000 7f607659c640 1 -- 172.21.15.88:0/95871415 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6060003520 con 0x7f60600039b0 2024-09-16T01:42:22.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.250+0000 7f607659c640 1 --2- 172.21.15.88:0/95871415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60600039b0 0x7f6060003dd0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f606800c290 tx=0x7f6060005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=365e77040e362e39 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:22.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.250+0000 7f607559a640 1 -- 172.21.15.88:0/95871415 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6068007d50 con 0x7f60600039b0 2024-09-16T01:42:22.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.250+0000 7f607559a640 1 -- 172.21.15.88:0/95871415 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6068005af0 con 0x7f60600039b0 2024-09-16T01:42:22.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.250+0000 7f607559a640 1 -- 172.21.15.88:0/95871415 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6068007070 con 0x7f60600039b0 2024-09-16T01:42:22.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.250+0000 7f607c8a5640 1 -- 172.21.15.88:0/95871415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60600039b0 msgr2=0x7f6060003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:22.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.250+0000 7f607c8a5640 1 --2- 172.21.15.88:0/95871415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60600039b0 0x7f6060003dd0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f606800c290 tx=0x7f6060005830 comp rx=0 tx=0).stop 2024-09-16T01:42:22.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.250+0000 7f6076d9d640 1 -- 172.21.15.88:0/95871415 reap_dead start 2024-09-16T01:42:22.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.250+0000 7f607c8a5640 1 -- 172.21.15.88:0/95871415 shutdown_connections 2024-09-16T01:42:22.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.250+0000 7f607c8a5640 1 -- 172.21.15.88:0/95871415 >> 172.21.15.88:0/95871415 conn(0x7f607806bc10 msgr2=0x7f607806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:22.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.250+0000 7f607c8a5640 1 -- 172.21.15.88:0/95871415 shutdown_connections 2024-09-16T01:42:22.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.251+0000 7f607c8a5640 1 -- 172.21.15.88:0/95871415 wait complete. 2024-09-16T01:42:22.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.252+0000 7f607c8a5640 1 Processor -- start 2024-09-16T01:42:22.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.252+0000 7f607c8a5640 1 -- start start 2024-09-16T01:42:22.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.252+0000 7f607c8a5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60600039b0 0x7f607814edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:22.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.253+0000 7f607c8a5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f607815f5a0 0x7f60780cf790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:22.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.253+0000 7f607c8a5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f607814f520 0x7f60780cfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:22.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.253+0000 7f607c8a5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f60780b5d60 con 0x7f60600039b0 2024-09-16T01:42:22.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.253+0000 7f607c8a5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f60780b5a80 con 0x7f607815f5a0 2024-09-16T01:42:22.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.253+0000 7f607c8a5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f607815e940 con 0x7f607814f520 2024-09-16T01:42:22.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.253+0000 7f607659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60600039b0 0x7f607814edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:22.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.253+0000 7f607659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60600039b0 0x7f607814edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49134/0 (socket says 172.21.15.88:49134) 2024-09-16T01:42:22.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.253+0000 7f607659c640 1 -- 172.21.15.88:0/4051736450 learned_addr learned my addr 172.21.15.88:0/4051736450 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:22.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.253+0000 7f6075d9b640 1 --2- 172.21.15.88:0/4051736450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f607815f5a0 0x7f60780cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:22.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.253+0000 7f6076d9d640 1 --2- 172.21.15.88:0/4051736450 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f607814f520 0x7f60780cfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:22.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.253+0000 7f607659c640 1 -- 172.21.15.88:0/4051736450 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f607814f520 msgr2=0x7f60780cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:22.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.253+0000 7f607659c640 1 --2- 172.21.15.88:0/4051736450 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f607814f520 0x7f60780cfcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:22.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.253+0000 7f607659c640 1 -- 172.21.15.88:0/4051736450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f607815f5a0 msgr2=0x7f60780cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:22.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.253+0000 7f607659c640 1 --2- 172.21.15.88:0/4051736450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f607815f5a0 0x7f60780cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:22.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.253+0000 7f607659c640 1 -- 172.21.15.88:0/4051736450 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f607814f310 con 0x7f60600039b0 2024-09-16T01:42:22.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.253+0000 7f6076d9d640 1 --2- 172.21.15.88:0/4051736450 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f607814f520 0x7f60780cfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:22.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.253+0000 7f607659c640 1 --2- 172.21.15.88:0/4051736450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60600039b0 0x7f607814edd0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f606802ae50 tx=0x7f60780b5690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:22.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.253+0000 7f6066ffd640 1 -- 172.21.15.88:0/4051736450 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6068042070 con 0x7f60600039b0 2024-09-16T01:42:22.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.253+0000 7f6066ffd640 1 -- 172.21.15.88:0/4051736450 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6068007480 con 0x7f60600039b0 2024-09-16T01:42:22.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.253+0000 7f607c8a5640 1 -- 172.21.15.88:0/4051736450 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6078154950 con 0x7f60600039b0 2024-09-16T01:42:22.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.253+0000 7f6066ffd640 1 -- 172.21.15.88:0/4051736450 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6068001c80 con 0x7f60600039b0 2024-09-16T01:42:22.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.253+0000 7f607c8a5640 1 -- 172.21.15.88:0/4051736450 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6078154ed0 con 0x7f60600039b0 2024-09-16T01:42:22.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.254+0000 7f607c8a5640 1 -- 172.21.15.88:0/4051736450 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60780aec80 con 0x7f60600039b0 2024-09-16T01:42:22.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.255+0000 7f6066ffd640 1 -- 172.21.15.88:0/4051736450 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6068045070 con 0x7f60600039b0 2024-09-16T01:42:22.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.255+0000 7f6066ffd640 1 --2- 172.21.15.88:0/4051736450 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f605806c2e0 0x7f605806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:22.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.255+0000 7f6066ffd640 1 -- 172.21.15.88:0/4051736450 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 7458+0+0 (secure 0 0 0) 0x7f60680ac6b0 con 0x7f60600039b0 2024-09-16T01:42:22.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.255+0000 7f6075d9b640 1 --2- 172.21.15.88:0/4051736450 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f605806c2e0 0x7f605806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:22.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.256+0000 7f6075d9b640 1 --2- 172.21.15.88:0/4051736450 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f605806c2e0 0x7f605806e7a0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f60780b0af0 tx=0x7f606c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:22.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.259+0000 7f6066ffd640 1 -- 172.21.15.88:0/4051736450 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6068007df0 con 0x7f60600039b0 2024-09-16T01:42:22.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.459+0000 7f607c8a5640 1 -- 172.21.15.88:0/4051736450 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60780ae9a0 con 0x7f60600039b0 2024-09-16T01:42:22.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.460+0000 7f6066ffd640 1 -- 172.21.15.88:0/4051736450 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) ==== 95+0+4392 (secure 0 0 0) 0x7f6068005ac0 con 0x7f60600039b0 2024-09-16T01:42:22.461 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:22.462 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":188,"btime":"2024-09-16T01:42:21:486016+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":8736,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/3374372758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3374372758},{"type":"v1","addr":"172.21.15.88:6836","nonce":3374372758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":181}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:21.485952+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":107,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:22.462 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 188 2024-09-16T01:42:22.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.462+0000 7f607c8a5640 1 -- 172.21.15.88:0/4051736450 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f605806c2e0 msgr2=0x7f605806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:22.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.462+0000 7f607c8a5640 1 --2- 172.21.15.88:0/4051736450 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f605806c2e0 0x7f605806e7a0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f60780b0af0 tx=0x7f606c008040 comp rx=0 tx=0).stop 2024-09-16T01:42:22.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.462+0000 7f607c8a5640 1 -- 172.21.15.88:0/4051736450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60600039b0 msgr2=0x7f607814edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:22.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.462+0000 7f607c8a5640 1 --2- 172.21.15.88:0/4051736450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60600039b0 0x7f607814edd0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f606802ae50 tx=0x7f60780b5690 comp rx=0 tx=0).stop 2024-09-16T01:42:22.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.462+0000 7f607c8a5640 1 -- 172.21.15.88:0/4051736450 shutdown_connections 2024-09-16T01:42:22.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.462+0000 7f607c8a5640 1 --2- 172.21.15.88:0/4051736450 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f605806c2e0 0x7f605806e7a0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:22.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.462+0000 7f607c8a5640 1 --2- 172.21.15.88:0/4051736450 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f607814f520 0x7f60780cfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:22.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.462+0000 7f607c8a5640 1 --2- 172.21.15.88:0/4051736450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f607815f5a0 0x7f60780cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:22.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.462+0000 7f607c8a5640 1 --2- 172.21.15.88:0/4051736450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60600039b0 0x7f607814edd0 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:22.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.462+0000 7f607c8a5640 1 -- 172.21.15.88:0/4051736450 >> 172.21.15.88:0/4051736450 conn(0x7f607806bc10 msgr2=0x7f60780b1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:22.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.462+0000 7f607c8a5640 1 -- 172.21.15.88:0/4051736450 shutdown_connections 2024-09-16T01:42:22.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.462+0000 7f607c8a5640 1 -- 172.21.15.88:0/4051736450 wait complete. 2024-09-16T01:42:22.474 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt mds_snaptable' led to DAMAGED state 2024-09-16T01:42:22.474 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt mds0_inotable' 2024-09-16T01:42:22.474 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:42:22.474 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:42:22.490 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:42:22.490 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:42:22.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:22.490 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:42:22.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:42:22.546 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:42:22.546 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:42:22.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.684+0000 7fc6ad08d640 1 Processor -- start 2024-09-16T01:42:22.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.684+0000 7fc6ad08d640 1 -- start start 2024-09-16T01:42:22.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.685+0000 7fc6ad08d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc6a815e6f0 0x7fc6a806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:22.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.685+0000 7fc6ad08d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6a80b7450 0x7fc6a815f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:22.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.685+0000 7fc6ad08d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6a806e990 con 0x7fc6a80b54d0 2024-09-16T01:42:22.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.685+0000 7fc6ad08d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc6a806ec70 con 0x7fc6a815e6f0 2024-09-16T01:42:22.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.685+0000 7fc6ad08d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc6a815ec70 con 0x7fc6a80b7450 2024-09-16T01:42:22.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.685+0000 7fc6a6d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc6a815e6f0 0x7fc6a806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:22.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.685+0000 7fc69ffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc6a80b54d0 0x7fc6a80b6d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43342/0 (socket says 172.21.15.88:43342) 2024-09-16T01:42:22.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.685+0000 7fc6a6d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc6a815e6f0 0x7fc6a806fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49148/0 (socket says 172.21.15.88:49148) 2024-09-16T01:42:22.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.685+0000 7fc69ffff640 1 -- 172.21.15.88:0/2848309024 learned_addr learned my addr 172.21.15.88:0/2848309024 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:22.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.685+0000 7fc6a759e640 1 --2- 172.21.15.88:0/2848309024 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6a80b7450 0x7fc6a815f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:22.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.685+0000 7fc6a6d9d640 1 -- 172.21.15.88:0/2848309024 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6a80b7450 msgr2=0x7fc6a815f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:22.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.685+0000 7fc6a6d9d640 1 --2- 172.21.15.88:0/2848309024 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6a80b7450 0x7fc6a815f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:22.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.685+0000 7fc6a6d9d640 1 -- 172.21.15.88:0/2848309024 >> v1:172.21.15.88:6789/0 conn(0x7fc6a80b54d0 legacy=0x7fc6a80b6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:22.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.685+0000 7fc6a659c640 1 -- 172.21.15.88:0/2848309024 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3294150697 0 0) 0x7fc6a806e990 con 0x7fc6a80b54d0 2024-09-16T01:42:22.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.685+0000 7fc6a6d9d640 1 -- 172.21.15.88:0/2848309024 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6a815ef50 con 0x7fc6a815e6f0 2024-09-16T01:42:22.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.686+0000 7fc6a6d9d640 1 --2- 172.21.15.88:0/2848309024 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6a815e6f0 0x7fc6a806fd00 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fc6980097a0 tx=0x7fc69802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6f96855e47fe9eb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:22.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.686+0000 7fc6a659c640 1 -- 172.21.15.88:0/2848309024 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc698037070 con 0x7fc6a815e6f0 2024-09-16T01:42:22.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.686+0000 7fc6a659c640 1 -- 172.21.15.88:0/2848309024 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc698032440 con 0x7fc6a815e6f0 2024-09-16T01:42:22.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.686+0000 7fc6a659c640 1 -- 172.21.15.88:0/2848309024 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc698032810 con 0x7fc6a815e6f0 2024-09-16T01:42:22.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.687+0000 7fc6ad08d640 1 -- 172.21.15.88:0/2848309024 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6a815e6f0 msgr2=0x7fc6a806fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:22.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.687+0000 7fc6ad08d640 1 --2- 172.21.15.88:0/2848309024 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6a815e6f0 0x7fc6a806fd00 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fc6980097a0 tx=0x7fc69802a180 comp rx=0 tx=0).stop 2024-09-16T01:42:22.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.687+0000 7fc6ad08d640 1 -- 172.21.15.88:0/2848309024 shutdown_connections 2024-09-16T01:42:22.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.687+0000 7fc6ad08d640 1 --2- 172.21.15.88:0/2848309024 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6a80b7450 0x7fc6a815f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:22.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.687+0000 7fc6ad08d640 1 --2- 172.21.15.88:0/2848309024 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6a815e6f0 0x7fc6a806fd00 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:22.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.687+0000 7fc6ad08d640 1 -- 172.21.15.88:0/2848309024 >> 172.21.15.88:0/2848309024 conn(0x7fc6a80d2430 msgr2=0x7fc6a80d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:22.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.687+0000 7fc6ad08d640 1 -- 172.21.15.88:0/2848309024 shutdown_connections 2024-09-16T01:42:22.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.687+0000 7fc6ad08d640 1 -- 172.21.15.88:0/2848309024 wait complete. 2024-09-16T01:42:22.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.688+0000 7fc6ad08d640 1 Processor -- start 2024-09-16T01:42:22.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.688+0000 7fc6ad08d640 1 -- start start 2024-09-16T01:42:22.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.689+0000 7fc6ad08d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6a80b54d0 0x7fc6a8158520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:22.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.689+0000 7fc6ad08d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6a80b7450 0x7fc6a81619b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:22.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.689+0000 7fc6ad08d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6a815e6f0 0x7fc6a8161ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:22.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.689+0000 7fc6ad08d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a806e730 con 0x7fc6a80b7450 2024-09-16T01:42:22.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.689+0000 7fc6ad08d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc6a806e5b0 con 0x7fc6a815e6f0 2024-09-16T01:42:22.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.689+0000 7fc6ad08d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc6a8070830 con 0x7fc6a80b54d0 2024-09-16T01:42:22.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.689+0000 7fc69ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6a80b7450 0x7fc6a81619b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:22.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.689+0000 7fc69ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6a80b7450 0x7fc6a81619b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49162/0 (socket says 172.21.15.88:49162) 2024-09-16T01:42:22.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.689+0000 7fc69ffff640 1 -- 172.21.15.88:0/1505075680 learned_addr learned my addr 172.21.15.88:0/1505075680 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.689+0000 7fc69ffff640 1 -- 172.21.15.88:0/1505075680 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6a80b54d0 msgr2=0x7fc6a8158520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.689+0000 7fc69ffff640 1 --2- 172.21.15.88:0/1505075680 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6a80b54d0 0x7fc6a8158520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.689+0000 7fc6a759e640 1 --2- 172.21.15.88:0/1505075680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6a815e6f0 0x7fc6a8161ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.689+0000 7fc69ffff640 1 -- 172.21.15.88:0/1505075680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6a815e6f0 msgr2=0x7fc6a8161ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.689+0000 7fc69ffff640 1 --2- 172.21.15.88:0/1505075680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6a815e6f0 0x7fc6a8161ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.689+0000 7fc69ffff640 1 -- 172.21.15.88:0/1505075680 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6a81625c0 con 0x7fc6a80b7450 2024-09-16T01:42:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.690+0000 7fc69ffff640 1 --2- 172.21.15.88:0/1505075680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6a80b7450 0x7fc6a81619b0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fc69000c4b0 tx=0x7fc69000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.690+0000 7fc69f7fe640 1 -- 172.21.15.88:0/1505075680 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc690018070 con 0x7fc6a80b7450 2024-09-16T01:42:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.690+0000 7fc69f7fe640 1 -- 172.21.15.88:0/1505075680 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc690007500 con 0x7fc6a80b7450 2024-09-16T01:42:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.690+0000 7fc6ad08d640 1 -- 172.21.15.88:0/1505075680 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6a81f4380 con 0x7fc6a80b7450 2024-09-16T01:42:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.690+0000 7fc69f7fe640 1 -- 172.21.15.88:0/1505075680 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc690004a20 con 0x7fc6a80b7450 2024-09-16T01:42:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.690+0000 7fc6ad08d640 1 -- 172.21.15.88:0/1505075680 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6a81f4940 con 0x7fc6a80b7450 2024-09-16T01:42:22.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.691+0000 7fc6ad08d640 1 -- 172.21.15.88:0/1505075680 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6a806c6c0 con 0x7fc6a80b7450 2024-09-16T01:42:22.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.691+0000 7fc69f7fe640 1 -- 172.21.15.88:0/1505075680 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc69001b070 con 0x7fc6a80b7450 2024-09-16T01:42:22.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.692+0000 7fc69f7fe640 1 --2- 172.21.15.88:0/1505075680 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc68806c2b0 0x7fc68806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:22.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.692+0000 7fc69f7fe640 1 -- 172.21.15.88:0/1505075680 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 7458+0+0 (secure 0 0 0) 0x7fc69008ae20 con 0x7fc6a80b7450 2024-09-16T01:42:22.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.692+0000 7fc6a6d9d640 1 --2- 172.21.15.88:0/1505075680 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc68806c2b0 0x7fc68806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:22.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.692+0000 7fc6a6d9d640 1 --2- 172.21.15.88:0/1505075680 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc68806c2b0 0x7fc68806e770 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fc698034000 tx=0x7fc69802c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:22.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.696+0000 7fc69f7fe640 1 -- 172.21.15.88:0/1505075680 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc690059000 con 0x7fc6a80b7450 2024-09-16T01:42:22.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:22.893+0000 7fc6ad08d640 1 -- 172.21.15.88:0/1505075680 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc6a8159f30 con 0x7fc6a80b7450 2024-09-16T01:42:23.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.483+0000 7fc69f7fe640 1 -- 172.21.15.88:0/1505075680 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v189) ==== 161+0+0 (secure 0 0 0) 0x7fc690058900 con 0x7fc6a80b7450 2024-09-16T01:42:23.497 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:42:23.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.485+0000 7fc6ad08d640 1 -- 172.21.15.88:0/1505075680 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc68806c2b0 msgr2=0x7fc68806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:23.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.485+0000 7fc6ad08d640 1 --2- 172.21.15.88:0/1505075680 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc68806c2b0 0x7fc68806e770 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fc698034000 tx=0x7fc69802c000 comp rx=0 tx=0).stop 2024-09-16T01:42:23.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.485+0000 7fc6ad08d640 1 -- 172.21.15.88:0/1505075680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6a80b7450 msgr2=0x7fc6a81619b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:23.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.485+0000 7fc6ad08d640 1 --2- 172.21.15.88:0/1505075680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6a80b7450 0x7fc6a81619b0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fc69000c4b0 tx=0x7fc69000c970 comp rx=0 tx=0).stop 2024-09-16T01:42:23.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.485+0000 7fc6ad08d640 1 -- 172.21.15.88:0/1505075680 shutdown_connections 2024-09-16T01:42:23.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.485+0000 7fc6ad08d640 1 --2- 172.21.15.88:0/1505075680 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc68806c2b0 0x7fc68806e770 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:23.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.485+0000 7fc6ad08d640 1 --2- 172.21.15.88:0/1505075680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6a815e6f0 0x7fc6a8161ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:23.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.485+0000 7fc6ad08d640 1 --2- 172.21.15.88:0/1505075680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6a80b7450 0x7fc6a81619b0 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:23.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.486+0000 7fc6ad08d640 1 --2- 172.21.15.88:0/1505075680 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6a80b54d0 0x7fc6a8158520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:23.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.486+0000 7fc6ad08d640 1 -- 172.21.15.88:0/1505075680 >> 172.21.15.88:0/1505075680 conn(0x7fc6a80d2430 msgr2=0x7fc6a80b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:23.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.486+0000 7fc6ad08d640 1 -- 172.21.15.88:0/1505075680 shutdown_connections 2024-09-16T01:42:23.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.486+0000 7fc6ad08d640 1 -- 172.21.15.88:0/1505075680 wait complete. 2024-09-16T01:42:23.500 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:42:23.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.597+0000 7fe2ddbfd640 1 Processor -- start 2024-09-16T01:42:23.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.597+0000 7fe2ddbfd640 1 -- start start 2024-09-16T01:42:23.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.598+0000 7fe2ddbfd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2d815c2a0 0x7fe2d80735d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:23.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.598+0000 7fe2ddbfd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe2d806fe20 0x7fe2d806bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:23.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.598+0000 7fe2ddbfd640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2d8074120 con 0x7fe2d8073b10 2024-09-16T01:42:23.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.598+0000 7fe2ddbfd640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe2d815c680 con 0x7fe2d806fe20 2024-09-16T01:42:23.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.598+0000 7fe2ddbfd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe2d815ef00 con 0x7fe2d815c2a0 2024-09-16T01:42:23.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.598+0000 7fe2d7fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe2d806fe20 0x7fe2d806bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:23.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.598+0000 7fe2d7fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe2d806fe20 0x7fe2d806bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49176/0 (socket says 172.21.15.88:49176) 2024-09-16T01:42:23.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.598+0000 7fe2d7fff640 1 -- 172.21.15.88:0/3033624954 learned_addr learned my addr 172.21.15.88:0/3033624954 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:23.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.599+0000 7fe2d77fe640 1 --2- 172.21.15.88:0/3033624954 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2d815c2a0 0x7fe2d80735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:23.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.599+0000 7fe2d7fff640 1 -- 172.21.15.88:0/3033624954 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2d815c2a0 msgr2=0x7fe2d80735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:23.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.599+0000 7fe2d7fff640 1 --2- 172.21.15.88:0/3033624954 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2d815c2a0 0x7fe2d80735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:23.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.599+0000 7fe2d7fff640 1 -- 172.21.15.88:0/3033624954 >> v1:172.21.15.154:6789/0 conn(0x7fe2d8073b10 legacy=0x7fe2d806f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:23.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.599+0000 7fe2d7fff640 1 -- 172.21.15.88:0/3033624954 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d815f1e0 con 0x7fe2d806fe20 2024-09-16T01:42:23.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.599+0000 7fe2d77fe640 1 --2- 172.21.15.88:0/3033624954 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2d815c2a0 0x7fe2d80735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:23.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.599+0000 7fe2d7fff640 1 --2- 172.21.15.88:0/3033624954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2d806fe20 0x7fe2d806bc10 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0009e50 tx=0x7fe2d002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b3946e60dd7843e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:23.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.600+0000 7fe2d67fc640 1 -- 172.21.15.88:0/3033624954 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d0036070 con 0x7fe2d806fe20 2024-09-16T01:42:23.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.600+0000 7fe2d67fc640 1 -- 172.21.15.88:0/3033624954 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2d0031600 con 0x7fe2d806fe20 2024-09-16T01:42:23.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.600+0000 7fe2d67fc640 1 -- 172.21.15.88:0/3033624954 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d00319d0 con 0x7fe2d806fe20 2024-09-16T01:42:23.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.600+0000 7fe2ddbfd640 1 -- 172.21.15.88:0/3033624954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2d806fe20 msgr2=0x7fe2d806bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:23.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.600+0000 7fe2ddbfd640 1 --2- 172.21.15.88:0/3033624954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2d806fe20 0x7fe2d806bc10 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0009e50 tx=0x7fe2d002a710 comp rx=0 tx=0).stop 2024-09-16T01:42:23.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.600+0000 7fe2ddbfd640 1 -- 172.21.15.88:0/3033624954 shutdown_connections 2024-09-16T01:42:23.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.600+0000 7fe2ddbfd640 1 --2- 172.21.15.88:0/3033624954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2d806fe20 0x7fe2d806bc10 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:23.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.600+0000 7fe2ddbfd640 1 --2- 172.21.15.88:0/3033624954 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2d815c2a0 0x7fe2d80735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:23.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.600+0000 7fe2ddbfd640 1 -- 172.21.15.88:0/3033624954 >> 172.21.15.88:0/3033624954 conn(0x7fe2d80b3070 msgr2=0x7fe2d80b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:23.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.600+0000 7fe2ddbfd640 1 -- 172.21.15.88:0/3033624954 shutdown_connections 2024-09-16T01:42:23.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.600+0000 7fe2ddbfd640 1 -- 172.21.15.88:0/3033624954 wait complete. 2024-09-16T01:42:23.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.601+0000 7fe2ddbfd640 1 Processor -- start 2024-09-16T01:42:23.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.602+0000 7fe2ddbfd640 1 -- start start 2024-09-16T01:42:23.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.602+0000 7fe2ddbfd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2d806fe20 0x7fe2d80cdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:23.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.602+0000 7fe2ddbfd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2d8073b10 0x7fe2d81469b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:23.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.602+0000 7fe2ddbfd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2d815c2a0 0x7fe2d8146ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:23.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.602+0000 7fe2ddbfd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d80b5f20 con 0x7fe2d8073b10 2024-09-16T01:42:23.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.602+0000 7fe2ddbfd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d815ebf0 con 0x7fe2d806fe20 2024-09-16T01:42:23.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.602+0000 7fe2ddbfd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe2d80b60a0 con 0x7fe2d815c2a0 2024-09-16T01:42:23.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.602+0000 7fe2d6ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2d8073b10 0x7fe2d81469b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:23.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.603+0000 7fe2d6ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2d8073b10 0x7fe2d81469b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49192/0 (socket says 172.21.15.88:49192) 2024-09-16T01:42:23.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.603+0000 7fe2d6ffd640 1 -- 172.21.15.88:0/3787391430 learned_addr learned my addr 172.21.15.88:0/3787391430 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:23.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.603+0000 7fe2d77fe640 1 --2- 172.21.15.88:0/3787391430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2d806fe20 0x7fe2d80cdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:23.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.603+0000 7fe2d7fff640 1 --2- 172.21.15.88:0/3787391430 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2d815c2a0 0x7fe2d8146ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:23.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.603+0000 7fe2d6ffd640 1 -- 172.21.15.88:0/3787391430 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2d815c2a0 msgr2=0x7fe2d8146ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:23.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.603+0000 7fe2d6ffd640 1 --2- 172.21.15.88:0/3787391430 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2d815c2a0 0x7fe2d8146ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:23.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.603+0000 7fe2d6ffd640 1 -- 172.21.15.88:0/3787391430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2d806fe20 msgr2=0x7fe2d80cdfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:23.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.603+0000 7fe2d6ffd640 1 --2- 172.21.15.88:0/3787391430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2d806fe20 0x7fe2d80cdfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:23.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.603+0000 7fe2d6ffd640 1 -- 172.21.15.88:0/3787391430 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d80d0970 con 0x7fe2d8073b10 2024-09-16T01:42:23.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.603+0000 7fe2d6ffd640 1 --2- 172.21.15.88:0/3787391430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2d8073b10 0x7fe2d81469b0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc00c440 tx=0x7fe2cc001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:23.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.603+0000 7fe2bffff640 1 -- 172.21.15.88:0/3787391430 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc0052d0 con 0x7fe2d8073b10 2024-09-16T01:42:23.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.603+0000 7fe2bffff640 1 -- 172.21.15.88:0/3787391430 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2cc005470 con 0x7fe2d8073b10 2024-09-16T01:42:23.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.603+0000 7fe2bffff640 1 -- 172.21.15.88:0/3787391430 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc007260 con 0x7fe2d8073b10 2024-09-16T01:42:23.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.603+0000 7fe2ddbfd640 1 -- 172.21.15.88:0/3787391430 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2d80d0c50 con 0x7fe2d8073b10 2024-09-16T01:42:23.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.603+0000 7fe2ddbfd640 1 -- 172.21.15.88:0/3787391430 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2d81600e0 con 0x7fe2d8073b10 2024-09-16T01:42:23.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.604+0000 7fe2ddbfd640 1 -- 172.21.15.88:0/3787391430 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2d80b6220 con 0x7fe2d8073b10 2024-09-16T01:42:23.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.605+0000 7fe2bffff640 1 -- 172.21.15.88:0/3787391430 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe2cc007400 con 0x7fe2d8073b10 2024-09-16T01:42:23.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.605+0000 7fe2bffff640 1 --2- 172.21.15.88:0/3787391430 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe2c406c2b0 0x7fe2c406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:23.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.605+0000 7fe2bffff640 1 -- 172.21.15.88:0/3787391430 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 7458+0+0 (secure 0 0 0) 0x7fe2cc08af80 con 0x7fe2d8073b10 2024-09-16T01:42:23.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.605+0000 7fe2d77fe640 1 --2- 172.21.15.88:0/3787391430 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe2c406c2b0 0x7fe2c406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:23.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.605+0000 7fe2d77fe640 1 --2- 172.21.15.88:0/3787391430 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe2c406c2b0 0x7fe2c406e770 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8006fd0 tx=0x7fe2c8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:23.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.609+0000 7fe2bffff640 1 -- 172.21.15.88:0/3787391430 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe2d80b6220 con 0x7fe2d8073b10 2024-09-16T01:42:23.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:23.841+0000 7fe2ddbfd640 1 -- 172.21.15.88:0/3787391430 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fe2d81602b0 con 0x7fe2d8073b10 2024-09-16T01:42:24.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.493+0000 7fe2bffff640 1 -- 172.21.15.88:0/3787391430 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v190) ==== 101+0+0 (secure 0 0 0) 0x7fe2cc0589c0 con 0x7fe2d8073b10 2024-09-16T01:42:24.494 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:42:24.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.495+0000 7fe2ddbfd640 1 -- 172.21.15.88:0/3787391430 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe2c406c2b0 msgr2=0x7fe2c406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:24.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.495+0000 7fe2ddbfd640 1 --2- 172.21.15.88:0/3787391430 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe2c406c2b0 0x7fe2c406e770 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8006fd0 tx=0x7fe2c8008040 comp rx=0 tx=0).stop 2024-09-16T01:42:24.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.495+0000 7fe2ddbfd640 1 -- 172.21.15.88:0/3787391430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2d8073b10 msgr2=0x7fe2d81469b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:24.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.495+0000 7fe2ddbfd640 1 --2- 172.21.15.88:0/3787391430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2d8073b10 0x7fe2d81469b0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc00c440 tx=0x7fe2cc001e70 comp rx=0 tx=0).stop 2024-09-16T01:42:24.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.495+0000 7fe2ddbfd640 1 -- 172.21.15.88:0/3787391430 shutdown_connections 2024-09-16T01:42:24.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.495+0000 7fe2ddbfd640 1 --2- 172.21.15.88:0/3787391430 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe2c406c2b0 0x7fe2c406e770 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.495+0000 7fe2ddbfd640 1 --2- 172.21.15.88:0/3787391430 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2d815c2a0 0x7fe2d8146ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.495+0000 7fe2ddbfd640 1 --2- 172.21.15.88:0/3787391430 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2d8073b10 0x7fe2d81469b0 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.495+0000 7fe2ddbfd640 1 --2- 172.21.15.88:0/3787391430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2d806fe20 0x7fe2d80cdfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.495+0000 7fe2ddbfd640 1 -- 172.21.15.88:0/3787391430 >> 172.21.15.88:0/3787391430 conn(0x7fe2d80b3070 msgr2=0x7fe2d80b0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:24.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.496+0000 7fe2ddbfd640 1 -- 172.21.15.88:0/3787391430 shutdown_connections 2024-09-16T01:42:24.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.496+0000 7fe2ddbfd640 1 -- 172.21.15.88:0/3787391430 wait complete. 2024-09-16T01:42:24.509 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:42:24.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.552+0000 7f827c0f8f40 1 Processor -- start 2024-09-16T01:42:24.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.552+0000 7f827c0f8f40 1 -- start start 2024-09-16T01:42:24.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.553+0000 7f827c0f8f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558236b75f0 0x5558236e4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.553+0000 7f827c0f8f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5558236e7590 0x5558236e6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.553+0000 7f827c0f8f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55582378ac90 0x555823788270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.553+0000 7f827c0f8f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x555823608510 con 0x55582378ac90 2024-09-16T01:42:24.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.553+0000 7f827c0f8f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x555823661c50 con 0x5558236b75f0 2024-09-16T01:42:24.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.553+0000 7f827c0f8f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55582378bc30 con 0x5558236e7590 2024-09-16T01:42:24.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.553+0000 7f827a3ef640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55582378ac90 0x555823788270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.553+0000 7f827a3ef640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55582378ac90 0x555823788270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49218/0 (socket says 172.21.15.88:49218) 2024-09-16T01:42:24.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.553+0000 7f827a3ef640 1 -- 172.21.15.88:0/2078046834 learned_addr learned my addr 172.21.15.88:0/2078046834 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:24.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.553+0000 7f8279bee640 1 --2- 172.21.15.88:0/2078046834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558236b75f0 0x5558236e4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.553+0000 7f82713ed640 1 --2- 172.21.15.88:0/2078046834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5558236e7590 0x5558236e6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.553+0000 7f827a3ef640 1 -- 172.21.15.88:0/2078046834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5558236e7590 msgr2=0x5558236e6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:24.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.553+0000 7f827a3ef640 1 --2- 172.21.15.88:0/2078046834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5558236e7590 0x5558236e6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.553+0000 7f827a3ef640 1 -- 172.21.15.88:0/2078046834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558236b75f0 msgr2=0x5558236e4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:24.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.553+0000 7f827a3ef640 1 --2- 172.21.15.88:0/2078046834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558236b75f0 0x5558236e4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.553+0000 7f827a3ef640 1 -- 172.21.15.88:0/2078046834 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558236b9810 con 0x55582378ac90 2024-09-16T01:42:24.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.553+0000 7f8279bee640 1 --2- 172.21.15.88:0/2078046834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558236b75f0 0x5558236e4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:24.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.553+0000 7f827a3ef640 1 --2- 172.21.15.88:0/2078046834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55582378ac90 0x555823788270 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f8274025b90 tx=0x7f827404f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c83901b90ce4e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:24.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.553+0000 7f8278bec640 1 -- 172.21.15.88:0/2078046834 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f827405a050 con 0x55582378ac90 2024-09-16T01:42:24.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.553+0000 7f8278bec640 1 -- 172.21.15.88:0/2078046834 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8274066070 con 0x55582378ac90 2024-09-16T01:42:24.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.553+0000 7f8278bec640 1 -- 172.21.15.88:0/2078046834 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8274055c50 con 0x55582378ac90 2024-09-16T01:42:24.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.554+0000 7f827c0f8f40 1 -- 172.21.15.88:0/2078046834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55582378ac90 msgr2=0x555823788270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:24.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.554+0000 7f827c0f8f40 1 --2- 172.21.15.88:0/2078046834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55582378ac90 0x555823788270 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f8274025b90 tx=0x7f827404f000 comp rx=0 tx=0).stop 2024-09-16T01:42:24.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.554+0000 7f827c0f8f40 1 -- 172.21.15.88:0/2078046834 shutdown_connections 2024-09-16T01:42:24.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.554+0000 7f827c0f8f40 1 --2- 172.21.15.88:0/2078046834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55582378ac90 0x555823788270 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.554+0000 7f827c0f8f40 1 --2- 172.21.15.88:0/2078046834 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5558236e7590 0x5558236e6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.554+0000 7f827c0f8f40 1 --2- 172.21.15.88:0/2078046834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5558236b75f0 0x5558236e4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.554+0000 7f827c0f8f40 1 -- 172.21.15.88:0/2078046834 >> 172.21.15.88:0/2078046834 conn(0x5558236e1680 msgr2=0x555823786580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:24.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.554+0000 7f827c0f8f40 1 -- 172.21.15.88:0/2078046834 shutdown_connections 2024-09-16T01:42:24.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.554+0000 7f827c0f8f40 1 -- 172.21.15.88:0/2078046834 wait complete. 2024-09-16T01:42:24.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.555+0000 7f827c0f8f40 1 Processor -- start 2024-09-16T01:42:24.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.555+0000 7f827c0f8f40 1 -- start start 2024-09-16T01:42:24.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.555+0000 7f827c0f8f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5558236b75f0 0x5558236e6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.555+0000 7f827c0f8f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558236e7590 0x555823778fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.555+0000 7f827c0f8f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55582378ac90 0x55582377b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.555+0000 7f827c0f8f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5558236e5460 con 0x5558236e7590 2024-09-16T01:42:24.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.555+0000 7f827c0f8f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55582377efa0 con 0x55582378ac90 2024-09-16T01:42:24.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.555+0000 7f827c0f8f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5558236e5810 con 0x5558236b75f0 2024-09-16T01:42:24.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.555+0000 7f82713ed640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558236e7590 0x555823778fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.555+0000 7f82713ed640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558236e7590 0x555823778fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49232/0 (socket says 172.21.15.88:49232) 2024-09-16T01:42:24.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.555+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 learned_addr learned my addr 172.21.15.88:0/3793918950 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:24.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.556+0000 7f8279bee640 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5558236b75f0 0x5558236e6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.556+0000 7f827a3ef640 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55582378ac90 0x55582377b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.556+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5558236b75f0 msgr2=0x5558236e6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:24.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.556+0000 7f82713ed640 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5558236b75f0 0x5558236e6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.556+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55582378ac90 msgr2=0x55582377b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:24.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.556+0000 7f82713ed640 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55582378ac90 0x55582377b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.556+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558236fe290 con 0x5558236e7590 2024-09-16T01:42:24.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.556+0000 7f8279bee640 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5558236b75f0 0x5558236e6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:24.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.556+0000 7f827a3ef640 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55582378ac90 0x55582377b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:24.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.556+0000 7f82713ed640 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558236e7590 0x555823778fa0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f826400df40 tx=0x7f826400b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:24.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.556+0000 7f82737fe640 1 -- 172.21.15.88:0/3793918950 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82640032d0 con 0x5558236e7590 2024-09-16T01:42:24.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.556+0000 7f82737fe640 1 -- 172.21.15.88:0/3793918950 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8264016070 con 0x5558236e7590 2024-09-16T01:42:24.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.556+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55582377dcb0 con 0x5558236e7590 2024-09-16T01:42:24.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.556+0000 7f82737fe640 1 -- 172.21.15.88:0/3793918950 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82640034b0 con 0x5558236e7590 2024-09-16T01:42:24.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.556+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558236fe530 con 0x5558236e7590 2024-09-16T01:42:24.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.558+0000 7f82737fe640 1 -- 172.21.15.88:0/3793918950 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8264001870 con 0x5558236e7590 2024-09-16T01:42:24.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.558+0000 7f82737fe640 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f825c06ac90 0x7f825c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.558+0000 7f82737fe640 1 -- 172.21.15.88:0/3793918950 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 7458+0+0 (secure 0 0 0) 0x7f826408a910 con 0x5558236e7590 2024-09-16T01:42:24.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.558+0000 7f8279bee640 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f825c06ac90 0x7f825c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.564 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:42:24.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.558+0000 7f827c0f8f40 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55582377bf90 0x5558236ff240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.558+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x5558236ff780 con 0x55582377bf90 2024-09-16T01:42:24.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.558+0000 7f827a3ef640 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55582377bf90 0x5558236ff240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.559+0000 7f8279bee640 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f825c06ac90 0x7f825c06d150 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x55582377d8c0 tx=0x7f826c001680 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:24.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.559+0000 7f827a3ef640 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55582377bf90 0x5558236ff240 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:24.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.561+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v107'598 uv572 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.561+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e107) -- 0x55582382e490 con 0x55582377bf90 2024-09-16T01:42:24.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.562+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v107'599 uv599 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.562+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582382e830 con 0x55582377bf90 2024-09-16T01:42:24.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.563+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v107'600 uv600 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.566 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:42:24.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.563+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582382fbe0 con 0x55582377bf90 2024-09-16T01:42:24.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.564+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v107'601 uv601 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.565+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582382ff80 con 0x55582377bf90 2024-09-16T01:42:24.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.566+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v107'602 uv602 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.566+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582382ebd0 con 0x55582377bf90 2024-09-16T01:42:24.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.567+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v107'603 uv603 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.567+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582382efc0 con 0x55582377bf90 2024-09-16T01:42:24.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.568+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v107'604 uv578 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.568+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e107) -- 0x55582382f3b0 con 0x55582377bf90 2024-09-16T01:42:24.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.570+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v107'605 uv605 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.570+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582382f3b0 con 0x55582377bf90 2024-09-16T01:42:24.572 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:42:24.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.571+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v107'606 uv606 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.571+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582382f830 con 0x55582377bf90 2024-09-16T01:42:24.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.573+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v107'607 uv607 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.573+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582382f360 con 0x55582377bf90 2024-09-16T01:42:24.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.574+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v107'608 uv608 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.574+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582382f360 con 0x55582377bf90 2024-09-16T01:42:24.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.576+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v107'609 uv609 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.576+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582382f360 con 0x55582377bf90 2024-09-16T01:42:24.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.577+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v107'610 uv585 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.577+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e107) -- 0x55582382f360 con 0x55582377bf90 2024-09-16T01:42:24.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.579+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v107'611 uv611 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.579+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582382f360 con 0x55582377bf90 2024-09-16T01:42:24.582 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:42:24.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.580+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v107'612 uv612 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.580+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582382f360 con 0x55582377bf90 2024-09-16T01:42:24.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.582+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v107'613 uv613 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.582+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582382f360 con 0x55582377bf90 2024-09-16T01:42:24.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.583+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v107'614 uv614 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.583+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582382f360 con 0x55582377bf90 2024-09-16T01:42:24.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.584+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v107'615 uv615 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.584+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823830320 con 0x55582377bf90 2024-09-16T01:42:24.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.586+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v107'616 uv616 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.586+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823830320 con 0x55582377bf90 2024-09-16T01:42:24.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.587+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v107'617 uv591 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.587+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e107) -- 0x555823830320 con 0x55582377bf90 2024-09-16T01:42:24.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.589+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v107'618 uv618 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.589+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823830320 con 0x55582377bf90 2024-09-16T01:42:24.591 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:42:24.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.590+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v107'619 uv619 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.590+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823831330 con 0x55582377bf90 2024-09-16T01:42:24.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.591+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v107'620 uv620 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.591+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823831330 con 0x55582377bf90 2024-09-16T01:42:24.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.593+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v107'621 uv621 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.593+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823830320 con 0x55582377bf90 2024-09-16T01:42:24.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.594+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v107'622 uv622 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.594+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823830320 con 0x55582377bf90 2024-09-16T01:42:24.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.595+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v107'623 uv597 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.595+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e107) -- 0x555823830320 con 0x55582377bf90 2024-09-16T01:42:24.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.597+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v107'624 uv624 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.597+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823830320 con 0x55582377bf90 2024-09-16T01:42:24.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.598+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v107'625 uv625 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.599 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:42:24.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.598+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823831330 con 0x55582377bf90 2024-09-16T01:42:24.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.599+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v107'626 uv626 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.599+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823831330 con 0x55582377bf90 2024-09-16T01:42:24.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.600+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v107'627 uv627 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.600+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823830320 con 0x55582377bf90 2024-09-16T01:42:24.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.602+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v107'628 uv628 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.602+0000 7f827c0f8f40 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x555823830830 0x555823832c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.602+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823833160 con 0x555823830830 2024-09-16T01:42:24.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.602+0000 7f8279bee640 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x555823830830 0x555823832c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.602+0000 7f8279bee640 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x555823830830 0x555823832c20 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:24.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.604+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v107'443 uv426 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x555823830830 2024-09-16T01:42:24.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.604+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e107) -- 0x555823833500 con 0x555823830830 2024-09-16T01:42:24.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.605+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v107'444 uv444 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x555823830830 2024-09-16T01:42:24.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.605+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x5558238338a0 con 0x555823830830 2024-09-16T01:42:24.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.606+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v107'445 uv445 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x555823830830 2024-09-16T01:42:24.607 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:42:24.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.606+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x5558238340f0 con 0x555823830830 2024-09-16T01:42:24.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.607+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v107'446 uv446 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x555823830830 2024-09-16T01:42:24.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.607+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823834490 con 0x555823830830 2024-09-16T01:42:24.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.608+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v107'447 uv447 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x555823830830 2024-09-16T01:42:24.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.608+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823834830 con 0x555823830830 2024-09-16T01:42:24.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.609+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v107'448 uv431 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x555823830830 2024-09-16T01:42:24.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.609+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e107) -- 0x555823834bd0 con 0x555823830830 2024-09-16T01:42:24.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.611+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v107'449 uv449 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x555823830830 2024-09-16T01:42:24.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.611+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823834f70 con 0x555823830830 2024-09-16T01:42:24.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.612+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v107'450 uv450 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x555823830830 2024-09-16T01:42:24.613 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:42:24.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.612+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823834f70 con 0x555823830830 2024-09-16T01:42:24.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.613+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v107'451 uv451 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x555823830830 2024-09-16T01:42:24.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.613+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823834f70 con 0x555823830830 2024-09-16T01:42:24.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.614+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v107'452 uv452 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x555823830830 2024-09-16T01:42:24.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.614+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823834f70 con 0x555823830830 2024-09-16T01:42:24.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.616+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v107'453 uv436 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x555823830830 2024-09-16T01:42:24.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.616+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e107) -- 0x555823834f70 con 0x555823830830 2024-09-16T01:42:24.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.617+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v107'454 uv454 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x555823830830 2024-09-16T01:42:24.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.617+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823834f70 con 0x555823830830 2024-09-16T01:42:24.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.618+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v107'455 uv455 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x555823830830 2024-09-16T01:42:24.620 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:42:24.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.618+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823834f70 con 0x555823830830 2024-09-16T01:42:24.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.619+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v107'456 uv456 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x555823830830 2024-09-16T01:42:24.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.619+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823834f70 con 0x555823830830 2024-09-16T01:42:24.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.620+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v107'457 uv457 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x555823830830 2024-09-16T01:42:24.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.620+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823834f70 con 0x555823830830 2024-09-16T01:42:24.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.621+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v107'458 uv442 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x555823830830 2024-09-16T01:42:24.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.621+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e107) -- 0x555823834f70 con 0x555823830830 2024-09-16T01:42:24.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.622+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v107'459 uv459 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x555823830830 2024-09-16T01:42:24.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.622+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823834f70 con 0x555823830830 2024-09-16T01:42:24.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.624+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v107'460 uv460 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x555823830830 2024-09-16T01:42:24.626 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:42:24.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.624+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823834f70 con 0x555823830830 2024-09-16T01:42:24.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.625+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v107'461 uv461 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x555823830830 2024-09-16T01:42:24.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.625+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823834f70 con 0x555823830830 2024-09-16T01:42:24.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.626+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v107'462 uv462 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x555823830830 2024-09-16T01:42:24.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.626+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823834f70 con 0x555823830830 2024-09-16T01:42:24.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.627+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v107'463 uv463 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x555823830830 2024-09-16T01:42:24.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.627+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823834f70 con 0x55582377bf90 2024-09-16T01:42:24.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.628+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v107'117 uv116 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.628+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e107) -- 0x555823834f70 con 0x55582377bf90 2024-09-16T01:42:24.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.629+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v107'118 uv118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.629+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823834f70 con 0x55582377bf90 2024-09-16T01:42:24.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.630+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v107'119 uv119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.631 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:42:24.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.630+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823834f70 con 0x55582377bf90 2024-09-16T01:42:24.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.631+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v107'120 uv120 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.631+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823834f70 con 0x55582377bf90 2024-09-16T01:42:24.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.632+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v107'121 uv121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.632+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823834f70 con 0x55582377bf90 2024-09-16T01:42:24.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.633+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v107'122 uv122 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.633+0000 7f827c0f8f40 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x555823835870 0x555823835cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.633+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x5558238383d0 con 0x555823835870 2024-09-16T01:42:24.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.633+0000 7f82713ed640 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x555823835870 0x555823835cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.633+0000 7f82713ed640 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x555823835870 0x555823835cb0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:24.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.634+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v107'89 uv88 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f826400e040 con 0x555823835870 2024-09-16T01:42:24.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.634+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e107) -- 0x555823838770 con 0x555823835870 2024-09-16T01:42:24.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.635+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v107'90 uv90 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f826400e040 con 0x555823835870 2024-09-16T01:42:24.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.635+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823838c20 con 0x555823835870 2024-09-16T01:42:24.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.636+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v107'91 uv91 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f826400e040 con 0x555823835870 2024-09-16T01:42:24.638 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:42:24.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.636+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823833d20 con 0x555823835870 2024-09-16T01:42:24.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.637+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v107'92 uv92 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f826400e040 con 0x555823835870 2024-09-16T01:42:24.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.637+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x5558238390b0 con 0x55582377bf90 2024-09-16T01:42:24.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.639+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v107'117 uv116 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.639+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e107) -- 0x5558238390b0 con 0x55582377bf90 2024-09-16T01:42:24.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.642+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v107'118 uv118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.642+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x5558238390b0 con 0x55582377bf90 2024-09-16T01:42:24.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.644+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v107'119 uv119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.645 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:42:24.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.644+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x5558238390b0 con 0x55582377bf90 2024-09-16T01:42:24.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.646+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v107'120 uv120 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.646+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x5558238390b0 con 0x55582377bf90 2024-09-16T01:42:24.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.649+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v107'121 uv121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.649+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x5558238390b0 con 0x55582377bf90 2024-09-16T01:42:24.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.652+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v107'122 uv122 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.652+0000 7f827c0f8f40 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x555823839f10 0x55582383a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.652+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582383ca70 con 0x555823839f10 2024-09-16T01:42:24.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.653+0000 7f827a3ef640 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x555823839f10 0x55582383a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.653+0000 7f827a3ef640 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x555823839f10 0x55582383a310 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:24.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.654+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v107'216 uv210 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.655+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e107) -- 0x55582383ca70 con 0x555823839f10 2024-09-16T01:42:24.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.656+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v107'217 uv217 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.656+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582383ca70 con 0x555823839f10 2024-09-16T01:42:24.658 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:42:24.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.657+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v107'218 uv218 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.657+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582383dac0 con 0x555823839f10 2024-09-16T01:42:24.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.658+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v107'219 uv219 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.658+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582383da80 con 0x555823839f10 2024-09-16T01:42:24.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.659+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v107'220 uv220 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.659+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582383ca70 con 0x555823839f10 2024-09-16T01:42:24.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.660+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v107'221 uv221 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.660+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582383ca70 con 0x555823839f10 2024-09-16T01:42:24.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.661+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v107'222 uv215 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.661+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e107) -- 0x55582383ca70 con 0x555823839f10 2024-09-16T01:42:24.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.662+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v107'223 uv223 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.662+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582383ca70 con 0x555823839f10 2024-09-16T01:42:24.664 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:42:24.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.663+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v107'224 uv224 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.663+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582383ca70 con 0x555823839f10 2024-09-16T01:42:24.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.664+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v107'225 uv225 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.664+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582383ca70 con 0x555823839f10 2024-09-16T01:42:24.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.665+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v107'226 uv226 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.665+0000 7f827c0f8f40 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55582383d370 0x55582383d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.665+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582383fed0 con 0x55582383d370 2024-09-16T01:42:24.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.665+0000 7f8279bee640 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55582383d370 0x55582383d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.666+0000 7f8279bee640 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55582383d370 0x55582383d7d0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:24.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.667+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v107'97 uv96 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x55582383d370 2024-09-16T01:42:24.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.667+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e107) -- 0x55582383fed0 con 0x55582383d370 2024-09-16T01:42:24.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.668+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v107'98 uv98 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x55582383d370 2024-09-16T01:42:24.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.668+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582383fed0 con 0x55582383d370 2024-09-16T01:42:24.670 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:42:24.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.669+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v107'99 uv99 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x55582383d370 2024-09-16T01:42:24.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.669+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x5558238395b0 con 0x55582383d370 2024-09-16T01:42:24.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.670+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v107'100 uv100 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x55582383d370 2024-09-16T01:42:24.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.670+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823839570 con 0x55582383d370 2024-09-16T01:42:24.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.671+0000 7f8279bee640 1 -- 172.21.15.88:0/3793918950 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v107'101 uv101 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f826c01a040 con 0x55582383d370 2024-09-16T01:42:24.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.671+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823839570 con 0x55582377bf90 2024-09-16T01:42:24.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.672+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v107'117 uv116 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.672+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e107) -- 0x555823839570 con 0x55582377bf90 2024-09-16T01:42:24.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.673+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v107'118 uv118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.673+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823839570 con 0x55582377bf90 2024-09-16T01:42:24.675 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:42:24.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.674+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v107'119 uv119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.674+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823839570 con 0x55582377bf90 2024-09-16T01:42:24.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.674+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v107'120 uv120 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.674+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823839570 con 0x55582377bf90 2024-09-16T01:42:24.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.675+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v107'121 uv121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.675+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823839570 con 0x55582377bf90 2024-09-16T01:42:24.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.676+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v107'122 uv122 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x55582377bf90 2024-09-16T01:42:24.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.676+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823839570 con 0x555823835870 2024-09-16T01:42:24.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.677+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v107'192 uv186 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f826400e040 con 0x555823835870 2024-09-16T01:42:24.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.677+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e107) -- 0x555823840420 con 0x555823835870 2024-09-16T01:42:24.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.678+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v107'193 uv193 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f826400e040 con 0x555823835870 2024-09-16T01:42:24.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.678+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x5558238407c0 con 0x555823835870 2024-09-16T01:42:24.680 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:42:24.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.679+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v107'194 uv194 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f826400e040 con 0x555823835870 2024-09-16T01:42:24.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.679+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582383ff70 con 0x555823835870 2024-09-16T01:42:24.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.680+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v107'195 uv195 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f826400e040 con 0x555823835870 2024-09-16T01:42:24.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.680+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582383ff70 con 0x555823835870 2024-09-16T01:42:24.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.681+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v107'196 uv196 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f826400e040 con 0x555823835870 2024-09-16T01:42:24.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.681+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582383ff70 con 0x555823835870 2024-09-16T01:42:24.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.682+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v107'197 uv191 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f826400e040 con 0x555823835870 2024-09-16T01:42:24.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.682+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e107) -- 0x55582383ff70 con 0x555823835870 2024-09-16T01:42:24.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.683+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v107'198 uv198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f826400e040 con 0x555823835870 2024-09-16T01:42:24.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.683+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582383ff70 con 0x555823835870 2024-09-16T01:42:24.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.684+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v107'199 uv199 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f826400e040 con 0x555823835870 2024-09-16T01:42:24.685 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:42:24.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.684+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x55582383ff70 con 0x555823835870 2024-09-16T01:42:24.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.684+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v107'200 uv200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f826400e040 con 0x555823835870 2024-09-16T01:42:24.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.685+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823843a10 con 0x555823835870 2024-09-16T01:42:24.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.686+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v107'201 uv201 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f826400e040 con 0x555823835870 2024-09-16T01:42:24.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.686+0000 7f827c0f8f40 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x555823841070 0x5558238440d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.686+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823841070 2024-09-16T01:42:24.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.686+0000 7f82713ed640 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x555823841070 0x5558238440d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.687+0000 7f82713ed640 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x555823841070 0x5558238440d0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:24.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.688+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v107'87 uv86 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f826400e040 con 0x555823841070 2024-09-16T01:42:24.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.688+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823841070 2024-09-16T01:42:24.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.689+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v107'88 uv88 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f826400e040 con 0x555823841070 2024-09-16T01:42:24.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.689+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823841070 2024-09-16T01:42:24.691 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:42:24.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.690+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v107'89 uv89 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f826400e040 con 0x555823841070 2024-09-16T01:42:24.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.690+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823841070 2024-09-16T01:42:24.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.691+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v107'90 uv90 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f826400e040 con 0x555823841070 2024-09-16T01:42:24.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.691+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823835870 2024-09-16T01:42:24.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.692+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v107'117 uv116 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f826400e040 con 0x555823835870 2024-09-16T01:42:24.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.692+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823835870 2024-09-16T01:42:24.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.693+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v107'118 uv118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f826400e040 con 0x555823835870 2024-09-16T01:42:24.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.693+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823835870 2024-09-16T01:42:24.695 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:42:24.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.694+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v107'119 uv119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f826400e040 con 0x555823835870 2024-09-16T01:42:24.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.694+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823835870 2024-09-16T01:42:24.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.695+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v107'120 uv120 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f826400e040 con 0x555823835870 2024-09-16T01:42:24.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.695+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823835870 2024-09-16T01:42:24.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.696+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v107'121 uv121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f826400e040 con 0x555823835870 2024-09-16T01:42:24.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.696+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823835870 2024-09-16T01:42:24.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.697+0000 7f82713ed640 1 -- 172.21.15.88:0/3793918950 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v107'122 uv122 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f826400e040 con 0x555823835870 2024-09-16T01:42:24.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.697+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823839f10 2024-09-16T01:42:24.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.698+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v107'99 uv98 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.698+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823839f10 2024-09-16T01:42:24.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.699+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v107'100 uv100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.699+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823839f10 2024-09-16T01:42:24.702 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:42:24.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.701+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v107'101 uv101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.701+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823839f10 2024-09-16T01:42:24.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.702+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v107'102 uv102 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.702+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823839f10 2024-09-16T01:42:24.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.703+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v107'103 uv103 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.703+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823839f10 2024-09-16T01:42:24.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.704+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v107'241 uv233 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.704+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823839f10 2024-09-16T01:42:24.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.705+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v107'242 uv242 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.705+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823839f10 2024-09-16T01:42:24.707 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:42:24.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.706+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v107'243 uv243 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.706+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823839f10 2024-09-16T01:42:24.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.707+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v107'244 uv244 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.707+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823839f10 2024-09-16T01:42:24.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.708+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v107'245 uv245 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.708+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823839f10 2024-09-16T01:42:24.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.709+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v107'246 uv240 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.709+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823839f10 2024-09-16T01:42:24.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.710+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v107'247 uv247 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.710+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823839f10 2024-09-16T01:42:24.712 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:42:24.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.711+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v107'248 uv248 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.711+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823839f10 2024-09-16T01:42:24.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.712+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v107'249 uv249 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.712+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823839f10 2024-09-16T01:42:24.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.713+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v107'250 uv250 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.713+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823843460 con 0x555823839f10 2024-09-16T01:42:24.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.714+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v107'251 uv251 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.714+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x555823844610 con 0x555823839f10 2024-09-16T01:42:24.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.715+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v107'252 uv252 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f827405d070 con 0x555823839f10 2024-09-16T01:42:24.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.715+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55582377bf90 msgr2=0x5558236ff240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:24.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.715+0000 7f827c0f8f40 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55582377bf90 0x5558236ff240 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.715+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x555823835870 msgr2=0x555823835cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:24.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.715+0000 7f827c0f8f40 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x555823835870 0x555823835cb0 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.715+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x555823841070 msgr2=0x5558238440d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:24.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.715+0000 7f827c0f8f40 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x555823841070 0x5558238440d0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.715+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 reap_dead start 2024-09-16T01:42:24.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.715+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x555823830830 msgr2=0x555823832c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:24.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.715+0000 7f827c0f8f40 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x555823830830 0x555823832c20 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.715+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x555823839f10 msgr2=0x55582383a310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:24.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.715+0000 7f827c0f8f40 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x555823839f10 0x55582383a310 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.715+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55582383d370 msgr2=0x55582383d7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:24.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.715+0000 7f827c0f8f40 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55582383d370 0x55582383d7d0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.715+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f825c06ac90 msgr2=0x7f825c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:24.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.715+0000 7f827c0f8f40 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f825c06ac90 0x7f825c06d150 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x55582377d8c0 tx=0x7f826c001680 comp rx=0 tx=0).stop 2024-09-16T01:42:24.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.715+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558236e7590 msgr2=0x555823778fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:24.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.715+0000 7f827c0f8f40 1 --2- 172.21.15.88:0/3793918950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5558236e7590 0x555823778fa0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f826400df40 tx=0x7f826400b5b0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.715+0000 7f827a3ef640 1 -- 172.21.15.88:0/3793918950 reap_dead start 2024-09-16T01:42:24.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.715+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 shutdown_connections 2024-09-16T01:42:24.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.715+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 >> 172.21.15.88:0/3793918950 conn(0x5558236e1680 msgr2=0x555823786340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:24.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.715+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 shutdown_connections 2024-09-16T01:42:24.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.715+0000 7f827c0f8f40 1 -- 172.21.15.88:0/3793918950 wait complete. 2024-09-16T01:42:24.720 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put mds0_inotable - 2024-09-16T01:42:24.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.802+0000 7fc186b79f40 1 Processor -- start 2024-09-16T01:42:24.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.802+0000 7fc186b79f40 1 -- start start 2024-09-16T01:42:24.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.803+0000 7fc186b79f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600cd6bc400 0x5600cd677120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.803+0000 7fc186b79f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600cd6bcec0 0x5600cd743de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.803+0000 7fc186b79f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600cd750140 0x5600cd74a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.803+0000 7fc186b79f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5600cd6bf2c0 con 0x5600cd6bcec0 2024-09-16T01:42:24.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.803+0000 7fc186b79f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5600cd663fd0 con 0x5600cd6bc400 2024-09-16T01:42:24.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.803+0000 7fc186b79f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5600cd6bf040 con 0x5600cd750140 2024-09-16T01:42:24.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.803+0000 7fc17bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600cd6bcec0 0x5600cd743de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.803+0000 7fc17bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600cd6bcec0 0x5600cd743de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49254/0 (socket says 172.21.15.88:49254) 2024-09-16T01:42:24.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.803+0000 7fc17bfff640 1 -- 172.21.15.88:0/663978047 learned_addr learned my addr 172.21.15.88:0/663978047 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:24.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.803+0000 7fc18466f640 1 --2- 172.21.15.88:0/663978047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600cd6bc400 0x5600cd677120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.803+0000 7fc184e70640 1 --2- 172.21.15.88:0/663978047 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600cd750140 0x5600cd74a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.803+0000 7fc17bfff640 1 -- 172.21.15.88:0/663978047 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600cd750140 msgr2=0x5600cd74a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:24.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.803+0000 7fc17bfff640 1 --2- 172.21.15.88:0/663978047 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600cd750140 0x5600cd74a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.803+0000 7fc17bfff640 1 -- 172.21.15.88:0/663978047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600cd6bc400 msgr2=0x5600cd677120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:24.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.803+0000 7fc17bfff640 1 --2- 172.21.15.88:0/663978047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600cd6bc400 0x5600cd677120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.803+0000 7fc17bfff640 1 -- 172.21.15.88:0/663978047 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600cd5a3cb0 con 0x5600cd6bcec0 2024-09-16T01:42:24.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.803+0000 7fc18466f640 1 --2- 172.21.15.88:0/663978047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600cd6bc400 0x5600cd677120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:24.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.803+0000 7fc184e70640 1 --2- 172.21.15.88:0/663978047 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600cd750140 0x5600cd74a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:24.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.803+0000 7fc17bfff640 1 --2- 172.21.15.88:0/663978047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600cd6bcec0 0x5600cd743de0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fc16c009760 tx=0x7fc16c009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=506a5981aa8c7d04 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:24.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.804+0000 7fc18366d640 1 -- 172.21.15.88:0/663978047 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16c017070 con 0x5600cd6bcec0 2024-09-16T01:42:24.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.804+0000 7fc18366d640 1 -- 172.21.15.88:0/663978047 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc16c0090d0 con 0x5600cd6bcec0 2024-09-16T01:42:24.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.804+0000 7fc18366d640 1 -- 172.21.15.88:0/663978047 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16c006b60 con 0x5600cd6bcec0 2024-09-16T01:42:24.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.804+0000 7fc186b79f40 1 -- 172.21.15.88:0/663978047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600cd6bcec0 msgr2=0x5600cd743de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:24.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.804+0000 7fc186b79f40 1 --2- 172.21.15.88:0/663978047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600cd6bcec0 0x5600cd743de0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fc16c009760 tx=0x7fc16c009c20 comp rx=0 tx=0).stop 2024-09-16T01:42:24.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.804+0000 7fc186b79f40 1 -- 172.21.15.88:0/663978047 shutdown_connections 2024-09-16T01:42:24.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.804+0000 7fc186b79f40 1 --2- 172.21.15.88:0/663978047 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600cd750140 0x5600cd74a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.804+0000 7fc186b79f40 1 --2- 172.21.15.88:0/663978047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600cd6bcec0 0x5600cd743de0 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.804+0000 7fc186b79f40 1 --2- 172.21.15.88:0/663978047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600cd6bc400 0x5600cd677120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.804+0000 7fc186b79f40 1 -- 172.21.15.88:0/663978047 >> 172.21.15.88:0/663978047 conn(0x5600cd6a3960 msgr2=0x5600cd6c22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:24.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.804+0000 7fc186b79f40 1 -- 172.21.15.88:0/663978047 shutdown_connections 2024-09-16T01:42:24.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.804+0000 7fc186b79f40 1 -- 172.21.15.88:0/663978047 wait complete. 2024-09-16T01:42:24.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.805+0000 7fc186b79f40 1 Processor -- start 2024-09-16T01:42:24.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.805+0000 7fc186b79f40 1 -- start start 2024-09-16T01:42:24.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.805+0000 7fc186b79f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600cd6bc400 0x5600cd742b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.805+0000 7fc186b79f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600cd6bcec0 0x5600cd69a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.805+0000 7fc186b79f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600cd750140 0x5600cd7464a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.805+0000 7fc186b79f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5600cd74bcb0 con 0x5600cd6bcec0 2024-09-16T01:42:24.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.805+0000 7fc186b79f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5600cd74f8b0 con 0x5600cd750140 2024-09-16T01:42:24.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.805+0000 7fc186b79f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5600cd74e990 con 0x5600cd6bc400 2024-09-16T01:42:24.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.805+0000 7fc17bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600cd6bcec0 0x5600cd69a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.806+0000 7fc17bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600cd6bcec0 0x5600cd69a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49256/0 (socket says 172.21.15.88:49256) 2024-09-16T01:42:24.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.806+0000 7fc17bfff640 1 -- 172.21.15.88:0/3340173318 learned_addr learned my addr 172.21.15.88:0/3340173318 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:24.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.806+0000 7fc18466f640 1 --2- 172.21.15.88:0/3340173318 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600cd6bc400 0x5600cd742b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.806+0000 7fc184e70640 1 --2- 172.21.15.88:0/3340173318 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600cd750140 0x5600cd7464a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.806+0000 7fc17bfff640 1 -- 172.21.15.88:0/3340173318 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600cd6bc400 msgr2=0x5600cd742b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:24.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.806+0000 7fc17bfff640 1 --2- 172.21.15.88:0/3340173318 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600cd6bc400 0x5600cd742b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.806+0000 7fc17bfff640 1 -- 172.21.15.88:0/3340173318 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600cd750140 msgr2=0x5600cd7464a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:24.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.806+0000 7fc17bfff640 1 --2- 172.21.15.88:0/3340173318 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600cd750140 0x5600cd7464a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.806+0000 7fc17bfff640 1 -- 172.21.15.88:0/3340173318 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600cd67a940 con 0x5600cd6bcec0 2024-09-16T01:42:24.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.806+0000 7fc17bfff640 1 --2- 172.21.15.88:0/3340173318 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600cd6bcec0 0x5600cd69a460 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fc16c009730 tx=0x7fc16c003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:24.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.806+0000 7fc18266b640 1 -- 172.21.15.88:0/3340173318 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16c021020 con 0x5600cd6bcec0 2024-09-16T01:42:24.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.806+0000 7fc18266b640 1 -- 172.21.15.88:0/3340173318 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc16c01d670 con 0x5600cd6bcec0 2024-09-16T01:42:24.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.806+0000 7fc186b79f40 1 -- 172.21.15.88:0/3340173318 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5600cd67ac20 con 0x5600cd6bcec0 2024-09-16T01:42:24.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.806+0000 7fc18266b640 1 -- 172.21.15.88:0/3340173318 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16c017040 con 0x5600cd6bcec0 2024-09-16T01:42:24.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.806+0000 7fc186b79f40 1 -- 172.21.15.88:0/3340173318 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600cd744440 con 0x5600cd6bcec0 2024-09-16T01:42:24.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.808+0000 7fc18266b640 1 -- 172.21.15.88:0/3340173318 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc16c024020 con 0x5600cd6bcec0 2024-09-16T01:42:24.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.808+0000 7fc18266b640 1 --2- 172.21.15.88:0/3340173318 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc16406ac90 0x7fc16406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.808+0000 7fc18266b640 1 -- 172.21.15.88:0/3340173318 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 7458+0+0 (secure 0 0 0) 0x7fc16c08b7d0 con 0x5600cd6bcec0 2024-09-16T01:42:24.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.808+0000 7fc18466f640 1 --2- 172.21.15.88:0/3340173318 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc16406ac90 0x7fc16406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.808+0000 7fc186b79f40 1 --2- 172.21.15.88:0/3340173318 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5600cd7469e0 0x5600cd74e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.808+0000 7fc186b79f40 1 -- 172.21.15.88:0/3340173318 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:1 12.13 12:c91d4a1d:::mds0_inotable:head [writefull 0~80 in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107) -- 0x5600cd74b3d0 con 0x5600cd7469e0 2024-09-16T01:42:24.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.808+0000 7fc184e70640 1 --2- 172.21.15.88:0/3340173318 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5600cd7469e0 0x5600cd74e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.808+0000 7fc18466f640 1 --2- 172.21.15.88:0/3340173318 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc16406ac90 0x7fc16406d150 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fc174057000 tx=0x7fc17405e040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:24.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.809+0000 7fc184e70640 1 --2- 172.21.15.88:0/3340173318 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5600cd7469e0 0x5600cd74e230 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:24.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.810+0000 7fc184e70640 1 -- 172.21.15.88:0/3340173318 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(1 mds0_inotable [writefull 0~80] v107'202 uv202 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc17c00d860 con 0x5600cd7469e0 2024-09-16T01:42:24.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.810+0000 7fc186b79f40 1 -- 172.21.15.88:0/3340173318 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5600cd7469e0 msgr2=0x5600cd74e230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:24.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.810+0000 7fc186b79f40 1 --2- 172.21.15.88:0/3340173318 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5600cd7469e0 0x5600cd74e230 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.810+0000 7fc186b79f40 1 -- 172.21.15.88:0/3340173318 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc16406ac90 msgr2=0x7fc16406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:24.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.810+0000 7fc186b79f40 1 --2- 172.21.15.88:0/3340173318 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc16406ac90 0x7fc16406d150 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fc174057000 tx=0x7fc17405e040 comp rx=0 tx=0).stop 2024-09-16T01:42:24.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.810+0000 7fc186b79f40 1 -- 172.21.15.88:0/3340173318 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600cd6bcec0 msgr2=0x5600cd69a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:24.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.810+0000 7fc186b79f40 1 --2- 172.21.15.88:0/3340173318 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600cd6bcec0 0x5600cd69a460 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fc16c009730 tx=0x7fc16c003020 comp rx=0 tx=0).stop 2024-09-16T01:42:24.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.810+0000 7fc184e70640 1 -- 172.21.15.88:0/3340173318 reap_dead start 2024-09-16T01:42:24.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.810+0000 7fc186b79f40 1 -- 172.21.15.88:0/3340173318 shutdown_connections 2024-09-16T01:42:24.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.810+0000 7fc186b79f40 1 -- 172.21.15.88:0/3340173318 >> 172.21.15.88:0/3340173318 conn(0x5600cd6a3960 msgr2=0x5600cd6c22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:24.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.810+0000 7fc186b79f40 1 -- 172.21.15.88:0/3340173318 shutdown_connections 2024-09-16T01:42:24.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.810+0000 7fc186b79f40 1 -- 172.21.15.88:0/3340173318 wait complete. 2024-09-16T01:42:24.818 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:42:24.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.957+0000 7fa55021e640 1 Processor -- start 2024-09-16T01:42:24.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.957+0000 7fa55021e640 1 -- start start 2024-09-16T01:42:24.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.958+0000 7fa55021e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa54815b480 0x7fa5480b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.958+0000 7fa55021e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa548072700 0x7fa54815bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.958+0000 7fa55021e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5480ccff0 con 0x7fa5480b60c0 2024-09-16T01:42:24.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.958+0000 7fa55021e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa5480cd2d0 con 0x7fa54815b480 2024-09-16T01:42:24.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.958+0000 7fa55021e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa5480cd5b0 con 0x7fa548072700 2024-09-16T01:42:24.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.958+0000 7fa54dfba640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa54815b480 0x7fa5480b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.958+0000 7fa54dfba640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa54815b480 0x7fa5480b5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49262/0 (socket says 172.21.15.88:49262) 2024-09-16T01:42:24.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.958+0000 7fa54dfba640 1 -- 172.21.15.88:0/3035193384 learned_addr learned my addr 172.21.15.88:0/3035193384 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:24.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.958+0000 7fa54e7bb640 1 --2- 172.21.15.88:0/3035193384 >> v2:172.21.15.154:3300/0 conn(0x7fa548072700 0x7fa54815bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.958+0000 7fa54dfba640 1 -- 172.21.15.88:0/3035193384 >> v2:172.21.15.154:3300/0 conn(0x7fa548072700 msgr2=0x7fa54815bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:24.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.958+0000 7fa54dfba640 1 --2- 172.21.15.88:0/3035193384 >> v2:172.21.15.154:3300/0 conn(0x7fa548072700 0x7fa54815bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.958+0000 7fa54dfba640 1 -- 172.21.15.88:0/3035193384 >> v1:172.21.15.154:6789/0 conn(0x7fa5480b60c0 legacy=0x7fa548071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:24.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.958+0000 7fa54dfba640 1 -- 172.21.15.88:0/3035193384 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5480cd890 con 0x7fa54815b480 2024-09-16T01:42:24.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.959+0000 7fa54dfba640 1 --2- 172.21.15.88:0/3035193384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa54815b480 0x7fa5480b5b80 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fa5480cdf80 tx=0x7fa54402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bf18497aaac8771 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:24.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.959+0000 7fa54cfb8640 1 -- 172.21.15.88:0/3035193384 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa544037070 con 0x7fa54815b480 2024-09-16T01:42:24.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.959+0000 7fa54cfb8640 1 -- 172.21.15.88:0/3035193384 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa544032440 con 0x7fa54815b480 2024-09-16T01:42:24.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.959+0000 7fa54cfb8640 1 -- 172.21.15.88:0/3035193384 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa544032810 con 0x7fa54815b480 2024-09-16T01:42:24.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.959+0000 7fa55021e640 1 -- 172.21.15.88:0/3035193384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa54815b480 msgr2=0x7fa5480b5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:24.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.959+0000 7fa55021e640 1 --2- 172.21.15.88:0/3035193384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa54815b480 0x7fa5480b5b80 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fa5480cdf80 tx=0x7fa54402a1e0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.959+0000 7fa55021e640 1 -- 172.21.15.88:0/3035193384 shutdown_connections 2024-09-16T01:42:24.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.959+0000 7fa55021e640 1 --2- 172.21.15.88:0/3035193384 >> v2:172.21.15.154:3300/0 conn(0x7fa548072700 0x7fa54815bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.959+0000 7fa55021e640 1 --2- 172.21.15.88:0/3035193384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa54815b480 0x7fa5480b5b80 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.959+0000 7fa55021e640 1 -- 172.21.15.88:0/3035193384 >> 172.21.15.88:0/3035193384 conn(0x7fa54806d0f0 msgr2=0x7fa54806d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:24.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.960+0000 7fa55021e640 1 -- 172.21.15.88:0/3035193384 shutdown_connections 2024-09-16T01:42:24.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.960+0000 7fa55021e640 1 -- 172.21.15.88:0/3035193384 wait complete. 2024-09-16T01:42:24.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.961+0000 7fa55021e640 1 Processor -- start 2024-09-16T01:42:24.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.961+0000 7fa55021e640 1 -- start start 2024-09-16T01:42:24.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.961+0000 7fa55021e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa548072700 0x7fa548146b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.961+0000 7fa55021e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5480b60c0 0x7fa5480c76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.961+0000 7fa55021e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa54815b480 0x7fa5480c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.961+0000 7fa55021e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa54815b9e0 con 0x7fa548072700 2024-09-16T01:42:24.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.961+0000 7fa55021e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa54815b860 con 0x7fa54815b480 2024-09-16T01:42:24.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.961+0000 7fa55021e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa54815c1a0 con 0x7fa5480b60c0 2024-09-16T01:42:24.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.962+0000 7fa54dfba640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa548072700 0x7fa548146b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.962+0000 7fa54dfba640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa548072700 0x7fa548146b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49276/0 (socket says 172.21.15.88:49276) 2024-09-16T01:42:24.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.962+0000 7fa54dfba640 1 -- 172.21.15.88:0/1462645760 learned_addr learned my addr 172.21.15.88:0/1462645760 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:24.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.962+0000 7fa54d7b9640 1 --2- 172.21.15.88:0/1462645760 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5480b60c0 0x7fa5480c76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.962+0000 7fa54e7bb640 1 --2- 172.21.15.88:0/1462645760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa54815b480 0x7fa5480c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.962+0000 7fa54dfba640 1 -- 172.21.15.88:0/1462645760 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5480b60c0 msgr2=0x7fa5480c76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:24.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.962+0000 7fa54dfba640 1 --2- 172.21.15.88:0/1462645760 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5480b60c0 0x7fa5480c76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.962+0000 7fa54dfba640 1 -- 172.21.15.88:0/1462645760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa54815b480 msgr2=0x7fa5480c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:24.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.962+0000 7fa54dfba640 1 --2- 172.21.15.88:0/1462645760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa54815b480 0x7fa5480c65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:24.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.962+0000 7fa54dfba640 1 -- 172.21.15.88:0/1462645760 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5480c5580 con 0x7fa548072700 2024-09-16T01:42:24.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.962+0000 7fa54dfba640 1 --2- 172.21.15.88:0/1462645760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa548072700 0x7fa548146b80 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fa54402a220 tx=0x7fa5440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:24.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.962+0000 7fa53e7fc640 1 -- 172.21.15.88:0/1462645760 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa544041020 con 0x7fa548072700 2024-09-16T01:42:24.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.962+0000 7fa53e7fc640 1 -- 172.21.15.88:0/1462645760 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5440306c0 con 0x7fa548072700 2024-09-16T01:42:24.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.962+0000 7fa53e7fc640 1 -- 172.21.15.88:0/1462645760 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa544037040 con 0x7fa548072700 2024-09-16T01:42:24.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.962+0000 7fa55021e640 1 -- 172.21.15.88:0/1462645760 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5480c5840 con 0x7fa548072700 2024-09-16T01:42:24.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.962+0000 7fa55021e640 1 -- 172.21.15.88:0/1462645760 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5480c5e00 con 0x7fa548072700 2024-09-16T01:42:24.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.963+0000 7fa55021e640 1 -- 172.21.15.88:0/1462645760 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5480cdfd0 con 0x7fa548072700 2024-09-16T01:42:24.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.964+0000 7fa53e7fc640 1 -- 172.21.15.88:0/1462645760 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa544044020 con 0x7fa548072700 2024-09-16T01:42:24.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.964+0000 7fa53e7fc640 1 --2- 172.21.15.88:0/1462645760 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa52c06c2b0 0x7fa52c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:24.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.964+0000 7fa53e7fc640 1 -- 172.21.15.88:0/1462645760 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 7458+0+0 (secure 0 0 0) 0x7fa5440ac190 con 0x7fa548072700 2024-09-16T01:42:24.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.964+0000 7fa54d7b9640 1 --2- 172.21.15.88:0/1462645760 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa52c06c2b0 0x7fa52c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:24.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.964+0000 7fa54d7b9640 1 --2- 172.21.15.88:0/1462645760 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa52c06c2b0 0x7fa52c06e770 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fa5480d0e70 tx=0x7fa538009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:24.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:24.968+0000 7fa53e7fc640 1 -- 172.21.15.88:0/1462645760 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa5480cdfd0 con 0x7fa548072700 2024-09-16T01:42:25.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.169+0000 7fa55021e640 1 -- 172.21.15.88:0/1462645760 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fa54806dbc0 con 0x7fa548072700 2024-09-16T01:42:25.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.498+0000 7fa53e7fc640 1 -- 172.21.15.88:0/1462645760 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v191) ==== 162+0+0 (secure 0 0 0) 0x7fa54404b030 con 0x7fa548072700 2024-09-16T01:42:25.499 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:42:25.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.501+0000 7fa55021e640 1 -- 172.21.15.88:0/1462645760 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa52c06c2b0 msgr2=0x7fa52c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:25.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.501+0000 7fa55021e640 1 --2- 172.21.15.88:0/1462645760 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa52c06c2b0 0x7fa52c06e770 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fa5480d0e70 tx=0x7fa538009040 comp rx=0 tx=0).stop 2024-09-16T01:42:25.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.501+0000 7fa55021e640 1 -- 172.21.15.88:0/1462645760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa548072700 msgr2=0x7fa548146b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:25.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.501+0000 7fa55021e640 1 --2- 172.21.15.88:0/1462645760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa548072700 0x7fa548146b80 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fa54402a220 tx=0x7fa5440046c0 comp rx=0 tx=0).stop 2024-09-16T01:42:25.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.501+0000 7fa55021e640 1 -- 172.21.15.88:0/1462645760 shutdown_connections 2024-09-16T01:42:25.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.501+0000 7fa55021e640 1 --2- 172.21.15.88:0/1462645760 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa52c06c2b0 0x7fa52c06e770 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:25.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.501+0000 7fa55021e640 1 --2- 172.21.15.88:0/1462645760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa54815b480 0x7fa5480c65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:25.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.501+0000 7fa55021e640 1 --2- 172.21.15.88:0/1462645760 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5480b60c0 0x7fa5480c76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:25.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.501+0000 7fa55021e640 1 --2- 172.21.15.88:0/1462645760 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa548072700 0x7fa548146b80 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:25.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.501+0000 7fa55021e640 1 -- 172.21.15.88:0/1462645760 >> 172.21.15.88:0/1462645760 conn(0x7fa54806d0f0 msgr2=0x7fa5480ce4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:25.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.501+0000 7fa55021e640 1 -- 172.21.15.88:0/1462645760 shutdown_connections 2024-09-16T01:42:25.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.501+0000 7fa55021e640 1 -- 172.21.15.88:0/1462645760 wait complete. 2024-09-16T01:42:25.506 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:42:25.505+0000 7f6c35c28640 -1 log_channel(cluster) log [ERR] : error decoding table object 'mds0_inotable': Decoder at 'virtual void InoTable::decode_state(ceph::buffer::v15_2_0::list::const_iterator&)' v=2 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-09-16T01:42:25.506 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:42:25.505+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:42:25.507 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:42:25.505+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:42:25.514 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:25.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.611+0000 7f59abcf8640 1 Processor -- start 2024-09-16T01:42:25.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.611+0000 7f59abcf8640 1 -- start start 2024-09-16T01:42:25.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.612+0000 7f59abcf8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f59a415c2b0 0x7f59a40b6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:25.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.612+0000 7f59abcf8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59a40b72f0 0x7f59a415eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:25.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.612+0000 7f59abcf8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59a415fb10 con 0x7f59a415f030 2024-09-16T01:42:25.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.612+0000 7f59abcf8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f59a40b5900 con 0x7f59a415c2b0 2024-09-16T01:42:25.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.612+0000 7f59abcf8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f59a415f830 con 0x7f59a40b72f0 2024-09-16T01:42:25.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.612+0000 7f59a9a94640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f59a415c2b0 0x7f59a40b6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:25.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.612+0000 7f59a9a94640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f59a415c2b0 0x7f59a40b6db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49280/0 (socket says 172.21.15.88:49280) 2024-09-16T01:42:25.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.612+0000 7f59a9a94640 1 -- 172.21.15.88:0/265470319 learned_addr learned my addr 172.21.15.88:0/265470319 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:25.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.612+0000 7f59a9293640 1 --2- 172.21.15.88:0/265470319 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59a40b72f0 0x7f59a415eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:25.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.613+0000 7f59a9a94640 1 -- 172.21.15.88:0/265470319 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59a40b72f0 msgr2=0x7f59a415eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:25.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.613+0000 7f59a9a94640 1 --2- 172.21.15.88:0/265470319 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59a40b72f0 0x7f59a415eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:25.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.613+0000 7f59a9a94640 1 -- 172.21.15.88:0/265470319 >> v1:172.21.15.154:6789/0 conn(0x7f59a415f030 legacy=0x7f59a406f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:25.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.613+0000 7f59a9a94640 1 -- 172.21.15.88:0/265470319 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59a406ce30 con 0x7f59a415c2b0 2024-09-16T01:42:25.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.613+0000 7f59a9a94640 1 --2- 172.21.15.88:0/265470319 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59a415c2b0 0x7f59a40b6db0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f59a0009e50 tx=0x7f59a002a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ada098ca5e9e8953 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:25.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.613+0000 7f59a8a92640 1 -- 172.21.15.88:0/265470319 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a0036070 con 0x7f59a415c2b0 2024-09-16T01:42:25.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.613+0000 7f59a8a92640 1 -- 172.21.15.88:0/265470319 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59a0031600 con 0x7f59a415c2b0 2024-09-16T01:42:25.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.613+0000 7f59a8a92640 1 -- 172.21.15.88:0/265470319 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a00319d0 con 0x7f59a415c2b0 2024-09-16T01:42:25.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.614+0000 7f59abcf8640 1 -- 172.21.15.88:0/265470319 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59a415c2b0 msgr2=0x7f59a40b6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:25.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.614+0000 7f59abcf8640 1 --2- 172.21.15.88:0/265470319 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59a415c2b0 0x7f59a40b6db0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f59a0009e50 tx=0x7f59a002a6c0 comp rx=0 tx=0).stop 2024-09-16T01:42:25.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.614+0000 7f59abcf8640 1 -- 172.21.15.88:0/265470319 shutdown_connections 2024-09-16T01:42:25.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.614+0000 7f59abcf8640 1 --2- 172.21.15.88:0/265470319 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59a40b72f0 0x7f59a415eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:25.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.614+0000 7f59abcf8640 1 --2- 172.21.15.88:0/265470319 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59a415c2b0 0x7f59a40b6db0 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:25.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.614+0000 7f59abcf8640 1 -- 172.21.15.88:0/265470319 >> 172.21.15.88:0/265470319 conn(0x7f59a40b4240 msgr2=0x7f59a40b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:25.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.614+0000 7f59abcf8640 1 -- 172.21.15.88:0/265470319 shutdown_connections 2024-09-16T01:42:25.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.614+0000 7f59abcf8640 1 -- 172.21.15.88:0/265470319 wait complete. 2024-09-16T01:42:25.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.615+0000 7f59abcf8640 1 Processor -- start 2024-09-16T01:42:25.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.615+0000 7f59abcf8640 1 -- start start 2024-09-16T01:42:25.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.616+0000 7f59abcf8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59a40b72f0 0x7f59a40d0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:25.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.616+0000 7f59abcf8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59a415c2b0 0x7f59a415b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:25.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.616+0000 7f59abcf8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59a415f030 0x7f59a41e8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:25.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.616+0000 7f59abcf8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f59a40b7730 con 0x7f59a40b72f0 2024-09-16T01:42:25.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.616+0000 7f59abcf8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f59a406d3f0 con 0x7f59a415f030 2024-09-16T01:42:25.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.616+0000 7f59abcf8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f59a415f4c0 con 0x7f59a415c2b0 2024-09-16T01:42:25.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.616+0000 7f59a9a94640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59a40b72f0 0x7f59a40d0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:25.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.616+0000 7f59a9a94640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59a40b72f0 0x7f59a40d0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49292/0 (socket says 172.21.15.88:49292) 2024-09-16T01:42:25.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.616+0000 7f59a9a94640 1 -- 172.21.15.88:0/2503123977 learned_addr learned my addr 172.21.15.88:0/2503123977 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:25.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.616+0000 7f59aa295640 1 --2- 172.21.15.88:0/2503123977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59a415f030 0x7f59a41e8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:25.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.616+0000 7f59a9293640 1 --2- 172.21.15.88:0/2503123977 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59a415c2b0 0x7f59a415b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:25.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.616+0000 7f59a9a94640 1 -- 172.21.15.88:0/2503123977 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59a415c2b0 msgr2=0x7f59a415b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:25.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.616+0000 7f59a9a94640 1 --2- 172.21.15.88:0/2503123977 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59a415c2b0 0x7f59a415b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:25.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.616+0000 7f59a9a94640 1 -- 172.21.15.88:0/2503123977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59a415f030 msgr2=0x7f59a41e8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:25.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.616+0000 7f59a9a94640 1 --2- 172.21.15.88:0/2503123977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59a415f030 0x7f59a41e8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:25.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.616+0000 7f59a9a94640 1 -- 172.21.15.88:0/2503123977 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59a40d0e80 con 0x7f59a40b72f0 2024-09-16T01:42:25.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.616+0000 7f59aa295640 1 --2- 172.21.15.88:0/2503123977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59a415f030 0x7f59a41e8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:25.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.616+0000 7f59a9a94640 1 --2- 172.21.15.88:0/2503123977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59a40b72f0 0x7f59a40d0710 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f59a0009e50 tx=0x7f59a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:25.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.616+0000 7f599a7fc640 1 -- 172.21.15.88:0/2503123977 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a0036050 con 0x7f59a40b72f0 2024-09-16T01:42:25.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.617+0000 7f599a7fc640 1 -- 172.21.15.88:0/2503123977 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59a0042070 con 0x7f59a40b72f0 2024-09-16T01:42:25.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.617+0000 7f59abcf8640 1 -- 172.21.15.88:0/2503123977 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59a40d1160 con 0x7f59a40b72f0 2024-09-16T01:42:25.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.617+0000 7f599a7fc640 1 -- 172.21.15.88:0/2503123977 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a0009510 con 0x7f59a40b72f0 2024-09-16T01:42:25.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.617+0000 7f59abcf8640 1 -- 172.21.15.88:0/2503123977 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59a40aa7d0 con 0x7f59a40b72f0 2024-09-16T01:42:25.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.618+0000 7f59abcf8640 1 -- 172.21.15.88:0/2503123977 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59a40d1680 con 0x7f59a40b72f0 2024-09-16T01:42:25.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.618+0000 7f599a7fc640 1 -- 172.21.15.88:0/2503123977 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f59a0007c50 con 0x7f59a40b72f0 2024-09-16T01:42:25.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.618+0000 7f599a7fc640 1 --2- 172.21.15.88:0/2503123977 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f598806c2b0 0x7f598806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:25.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.618+0000 7f599a7fc640 1 -- 172.21.15.88:0/2503123977 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 7544+0+0 (secure 0 0 0) 0x7f59a00ac960 con 0x7f59a40b72f0 2024-09-16T01:42:25.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.618+0000 7f59a9293640 1 --2- 172.21.15.88:0/2503123977 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f598806c2b0 0x7f598806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:25.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.619+0000 7f59a9293640 1 --2- 172.21.15.88:0/2503123977 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f598806c2b0 0x7f598806e770 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f59a40b5d70 tx=0x7f5994008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:25.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.623+0000 7f599a7fc640 1 -- 172.21.15.88:0/2503123977 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f59a40d1680 con 0x7f59a40b72f0 2024-09-16T01:42:25.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.822+0000 7f59abcf8640 1 -- 172.21.15.88:0/2503123977 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59a415eaf0 con 0x7f59a40b72f0 2024-09-16T01:42:25.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.823+0000 7f599a7fc640 1 -- 172.21.15.88:0/2503123977 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) ==== 95+0+3566 (secure 0 0 0) 0x7f59a007aa30 con 0x7f59a40b72f0 2024-09-16T01:42:25.824 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:25.824 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":193,"btime":"2024-09-16T01:42:25:506576+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":193,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:25.506484+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:25.825 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 193 2024-09-16T01:42:25.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.824+0000 7f59abcf8640 1 -- 172.21.15.88:0/2503123977 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f598806c2b0 msgr2=0x7f598806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:25.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.824+0000 7f59abcf8640 1 --2- 172.21.15.88:0/2503123977 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f598806c2b0 0x7f598806e770 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f59a40b5d70 tx=0x7f5994008040 comp rx=0 tx=0).stop 2024-09-16T01:42:25.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.824+0000 7f59abcf8640 1 -- 172.21.15.88:0/2503123977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59a40b72f0 msgr2=0x7f59a40d0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:25.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.824+0000 7f59abcf8640 1 --2- 172.21.15.88:0/2503123977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59a40b72f0 0x7f59a40d0710 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f59a0009e50 tx=0x7f59a0004a40 comp rx=0 tx=0).stop 2024-09-16T01:42:25.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.824+0000 7f59abcf8640 1 -- 172.21.15.88:0/2503123977 shutdown_connections 2024-09-16T01:42:25.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.825+0000 7f59abcf8640 1 --2- 172.21.15.88:0/2503123977 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f598806c2b0 0x7f598806e770 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:25.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.825+0000 7f59abcf8640 1 --2- 172.21.15.88:0/2503123977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59a415f030 0x7f59a41e8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:25.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.825+0000 7f59abcf8640 1 --2- 172.21.15.88:0/2503123977 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59a415c2b0 0x7f59a415b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:25.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.825+0000 7f59abcf8640 1 --2- 172.21.15.88:0/2503123977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59a40b72f0 0x7f59a40d0710 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:25.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.825+0000 7f59abcf8640 1 -- 172.21.15.88:0/2503123977 >> 172.21.15.88:0/2503123977 conn(0x7f59a40b4240 msgr2=0x7f59a4160790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:25.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.825+0000 7f59abcf8640 1 -- 172.21.15.88:0/2503123977 shutdown_connections 2024-09-16T01:42:25.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.825+0000 7f59abcf8640 1 -- 172.21.15.88:0/2503123977 wait complete. 2024-09-16T01:42:25.836 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:42:25.837 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:25.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.933+0000 7ff7b2edd640 1 Processor -- start 2024-09-16T01:42:25.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.933+0000 7ff7b2edd640 1 -- start start 2024-09-16T01:42:25.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.934+0000 7ff7b2edd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff7ac0b4920 0x7ff7ac15b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:25.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.934+0000 7ff7b2edd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7ac0cd780 0x7ff7ac0cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:25.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.934+0000 7ff7b2edd640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7ac0ce4f0 con 0x7ff7ac0b4d00 2024-09-16T01:42:25.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.934+0000 7ff7b2edd640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff7ac0cf100 con 0x7ff7ac0b4920 2024-09-16T01:42:25.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.934+0000 7ff7b2edd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff7ac0ce7d0 con 0x7ff7ac0cd780 2024-09-16T01:42:25.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.934+0000 7ff7b0c79640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff7ac0b4920 0x7ff7ac15b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:25.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.934+0000 7ff7b147a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7ac0cd780 0x7ff7ac0cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:25.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.934+0000 7ff7b0c79640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff7ac0b4920 0x7ff7ac15b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:50372/0 (socket says 172.21.15.88:50372) 2024-09-16T01:42:25.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.934+0000 7ff7b0c79640 1 -- 172.21.15.88:0/3193865224 learned_addr learned my addr 172.21.15.88:0/3193865224 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:25.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.934+0000 7ff7abfff640 1 --1- 172.21.15.88:0/3193865224 >> v1:172.21.15.154:6789/0 conn(0x7ff7ac0b4d00 0x7ff7ac0b5170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:43996/0 (socket says 172.21.15.88:43996) 2024-09-16T01:42:25.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.935+0000 7ff7b147a640 1 -- 172.21.15.88:0/3193865224 >> v2:172.21.15.154:3300/0 conn(0x7ff7ac0b4920 msgr2=0x7ff7ac15b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:25.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.935+0000 7ff7b147a640 1 --2- 172.21.15.88:0/3193865224 >> v2:172.21.15.154:3300/0 conn(0x7ff7ac0b4920 0x7ff7ac15b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:25.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.935+0000 7ff7b147a640 1 -- 172.21.15.88:0/3193865224 >> v1:172.21.15.154:6789/0 conn(0x7ff7ac0b4d00 legacy=0x7ff7ac0b5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:25.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.935+0000 7ff7b147a640 1 -- 172.21.15.88:0/3193865224 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7ac0ceab0 con 0x7ff7ac0cd780 2024-09-16T01:42:25.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.935+0000 7ff7ab7fe640 1 -- 172.21.15.88:0/3193865224 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2360099201 0 0) 0x7ff7ac0ce4f0 con 0x7ff7ac0b4d00 2024-09-16T01:42:25.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.935+0000 7ff7b0c79640 1 --2- 172.21.15.88:0/3193865224 >> v2:172.21.15.154:3300/0 conn(0x7ff7ac0b4920 0x7ff7ac15b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:42:25.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.936+0000 7ff7b147a640 1 --2- 172.21.15.88:0/3193865224 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7ac0cd780 0x7ff7ac0cdbd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4009d90 tx=0x7ff7a402a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=977796107a5e392e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:25.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.936+0000 7ff7ab7fe640 1 -- 172.21.15.88:0/3193865224 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a4007740 con 0x7ff7ac0cd780 2024-09-16T01:42:25.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.936+0000 7ff7ab7fe640 1 -- 172.21.15.88:0/3193865224 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7a40078e0 con 0x7ff7ac0cd780 2024-09-16T01:42:25.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.936+0000 7ff7ab7fe640 1 -- 172.21.15.88:0/3193865224 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a4007cb0 con 0x7ff7ac0cd780 2024-09-16T01:42:25.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.936+0000 7ff7b2edd640 1 -- 172.21.15.88:0/3193865224 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7ac0cd780 msgr2=0x7ff7ac0cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:25.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.936+0000 7ff7b2edd640 1 --2- 172.21.15.88:0/3193865224 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7ac0cd780 0x7ff7ac0cdbd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4009d90 tx=0x7ff7a402a660 comp rx=0 tx=0).stop 2024-09-16T01:42:25.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.936+0000 7ff7b2edd640 1 -- 172.21.15.88:0/3193865224 shutdown_connections 2024-09-16T01:42:25.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.936+0000 7ff7b2edd640 1 --2- 172.21.15.88:0/3193865224 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7ac0cd780 0x7ff7ac0cdbd0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:25.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.936+0000 7ff7b2edd640 1 --2- 172.21.15.88:0/3193865224 >> v2:172.21.15.154:3300/0 conn(0x7ff7ac0b4920 0x7ff7ac15b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:25.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.936+0000 7ff7b2edd640 1 -- 172.21.15.88:0/3193865224 >> 172.21.15.88:0/3193865224 conn(0x7ff7ac06bc10 msgr2=0x7ff7ac06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:25.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.936+0000 7ff7b2edd640 1 -- 172.21.15.88:0/3193865224 shutdown_connections 2024-09-16T01:42:25.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.936+0000 7ff7b2edd640 1 -- 172.21.15.88:0/3193865224 wait complete. 2024-09-16T01:42:25.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.937+0000 7ff7b2edd640 1 Processor -- start 2024-09-16T01:42:25.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.938+0000 7ff7b2edd640 1 -- start start 2024-09-16T01:42:25.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.938+0000 7ff7b2edd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7ac0b4920 0x7ff7ac153a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:25.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.938+0000 7ff7b2edd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7ac0b4d00 0x7ff7ac150120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:25.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.938+0000 7ff7b2edd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7ac0cd780 0x7ff7ac151ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:25.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.938+0000 7ff7b2edd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ac0ce110 con 0x7ff7ac0b4d00 2024-09-16T01:42:25.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.938+0000 7ff7b2edd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ac15bd10 con 0x7ff7ac0b4920 2024-09-16T01:42:25.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.938+0000 7ff7b2edd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff7ac0ce290 con 0x7ff7ac0cd780 2024-09-16T01:42:25.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.939+0000 7ff7abfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7ac0b4d00 0x7ff7ac150120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:25.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.939+0000 7ff7abfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7ac0b4d00 0x7ff7ac150120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49308/0 (socket says 172.21.15.88:49308) 2024-09-16T01:42:25.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.939+0000 7ff7abfff640 1 -- 172.21.15.88:0/154573197 learned_addr learned my addr 172.21.15.88:0/154573197 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:25.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.939+0000 7ff7b0c79640 1 --2- 172.21.15.88:0/154573197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7ac0b4920 0x7ff7ac153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:25.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.939+0000 7ff7b147a640 1 --2- 172.21.15.88:0/154573197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7ac0cd780 0x7ff7ac151ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:25.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.939+0000 7ff7abfff640 1 -- 172.21.15.88:0/154573197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7ac0cd780 msgr2=0x7ff7ac151ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:25.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.939+0000 7ff7abfff640 1 --2- 172.21.15.88:0/154573197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7ac0cd780 0x7ff7ac151ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:25.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.939+0000 7ff7abfff640 1 -- 172.21.15.88:0/154573197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7ac0b4920 msgr2=0x7ff7ac153a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:25.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.939+0000 7ff7abfff640 1 --2- 172.21.15.88:0/154573197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7ac0b4920 0x7ff7ac153a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:25.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.939+0000 7ff7abfff640 1 -- 172.21.15.88:0/154573197 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7ac152240 con 0x7ff7ac0b4d00 2024-09-16T01:42:25.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.939+0000 7ff7abfff640 1 --2- 172.21.15.88:0/154573197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7ac0b4d00 0x7ff7ac150120 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7ff7a000c500 tx=0x7ff7a000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:25.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.939+0000 7ff7a97fa640 1 -- 172.21.15.88:0/154573197 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0018070 con 0x7ff7ac0b4d00 2024-09-16T01:42:25.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.939+0000 7ff7a97fa640 1 -- 172.21.15.88:0/154573197 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7a0007580 con 0x7ff7ac0b4d00 2024-09-16T01:42:25.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.939+0000 7ff7b2edd640 1 -- 172.21.15.88:0/154573197 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7ac152520 con 0x7ff7ac0b4d00 2024-09-16T01:42:25.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.939+0000 7ff7a97fa640 1 -- 172.21.15.88:0/154573197 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a0004a20 con 0x7ff7ac0b4d00 2024-09-16T01:42:25.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.939+0000 7ff7b2edd640 1 -- 172.21.15.88:0/154573197 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7ac152ae0 con 0x7ff7ac0b4d00 2024-09-16T01:42:25.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.940+0000 7ff7b2edd640 1 -- 172.21.15.88:0/154573197 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7ac15b7d0 con 0x7ff7ac0b4d00 2024-09-16T01:42:25.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.941+0000 7ff7a97fa640 1 -- 172.21.15.88:0/154573197 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff7a001b070 con 0x7ff7ac0b4d00 2024-09-16T01:42:25.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.941+0000 7ff7a97fa640 1 --2- 172.21.15.88:0/154573197 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff79806c2b0 0x7ff79806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:25.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.941+0000 7ff7a97fa640 1 -- 172.21.15.88:0/154573197 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 7544+0+0 (secure 0 0 0) 0x7ff7a008ad90 con 0x7ff7ac0b4d00 2024-09-16T01:42:25.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.941+0000 7ff7b0c79640 1 --2- 172.21.15.88:0/154573197 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff79806c2b0 0x7ff79806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:25.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.942+0000 7ff7b0c79640 1 --2- 172.21.15.88:0/154573197 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff79806c2b0 0x7ff79806e770 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff79c00b8d0 tx=0x7ff79c00b450 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:25.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:25.945+0000 7ff7a97fa640 1 -- 172.21.15.88:0/154573197 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff7a0058810 con 0x7ff7ac0b4d00 2024-09-16T01:42:26.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.143+0000 7ff7b2edd640 1 -- 172.21.15.88:0/154573197 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff7ac152ce0 con 0x7ff7ac0b4d00 2024-09-16T01:42:26.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.143+0000 7ff7a97fa640 1 -- 172.21.15.88:0/154573197 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) ==== 95+0+3566 (secure 0 0 0) 0x7ff7a005cb10 con 0x7ff7ac0b4d00 2024-09-16T01:42:26.145 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:26.145 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":193,"btime":"2024-09-16T01:42:25:506576+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":193,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:25.506484+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:26.145 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 193 2024-09-16T01:42:26.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.145+0000 7ff7b2edd640 1 -- 172.21.15.88:0/154573197 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff79806c2b0 msgr2=0x7ff79806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:26.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.145+0000 7ff7b2edd640 1 --2- 172.21.15.88:0/154573197 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff79806c2b0 0x7ff79806e770 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff79c00b8d0 tx=0x7ff79c00b450 comp rx=0 tx=0).stop 2024-09-16T01:42:26.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.145+0000 7ff7b2edd640 1 -- 172.21.15.88:0/154573197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7ac0b4d00 msgr2=0x7ff7ac150120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:26.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.145+0000 7ff7b2edd640 1 --2- 172.21.15.88:0/154573197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7ac0b4d00 0x7ff7ac150120 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7ff7a000c500 tx=0x7ff7a000c9c0 comp rx=0 tx=0).stop 2024-09-16T01:42:26.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.145+0000 7ff7b2edd640 1 -- 172.21.15.88:0/154573197 shutdown_connections 2024-09-16T01:42:26.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.145+0000 7ff7b2edd640 1 --2- 172.21.15.88:0/154573197 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff79806c2b0 0x7ff79806e770 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:26.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.145+0000 7ff7b2edd640 1 --2- 172.21.15.88:0/154573197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7ac0cd780 0x7ff7ac151ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:26.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.145+0000 7ff7b2edd640 1 --2- 172.21.15.88:0/154573197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7ac0b4d00 0x7ff7ac150120 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:26.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.145+0000 7ff7b2edd640 1 --2- 172.21.15.88:0/154573197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7ac0b4920 0x7ff7ac153a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:26.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.145+0000 7ff7b2edd640 1 -- 172.21.15.88:0/154573197 >> 172.21.15.88:0/154573197 conn(0x7ff7ac06bc10 msgr2=0x7ff7ac155ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:26.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.146+0000 7ff7b2edd640 1 -- 172.21.15.88:0/154573197 shutdown_connections 2024-09-16T01:42:26.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.146+0000 7ff7b2edd640 1 -- 172.21.15.88:0/154573197 wait complete. 2024-09-16T01:42:26.158 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:42:26.158 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:26.236 INFO:tasks.ceph.mds.c.smithi088.stderr: -45> 2024-09-16T01:42:21.484+0000 7fc5fd4be640 -1 log_channel(cluster) log [ERR] : error decoding table object 'mds_snaptable': Decoder at 'virtual void SnapServer::decode_server_state(ceph::buffer::v15_2_0::list::const_iterator&)' v=5 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-09-16T01:42:26.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.256+0000 7f540df2c640 1 Processor -- start 2024-09-16T01:42:26.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.256+0000 7f540df2c640 1 -- start start 2024-09-16T01:42:26.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.257+0000 7f540df2c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f540815e660 0x7f54080cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:26.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.257+0000 7f540df2c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f540815ea40 con 0x7f54080d1820 2024-09-16T01:42:26.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.257+0000 7f540df2c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54080d3870 con 0x7f54080d3310 2024-09-16T01:42:26.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.257+0000 7f540df2c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f54080d3b50 con 0x7f540815e660 2024-09-16T01:42:26.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.257+0000 7f54077fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f540815e660 0x7f54080cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:26.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.257+0000 7f54077fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f540815e660 0x7f54080cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49338/0 (socket says 172.21.15.88:49338) 2024-09-16T01:42:26.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.257+0000 7f5407fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f54080d1820 0x7f54080b7b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43374/0 (socket says 172.21.15.88:43374) 2024-09-16T01:42:26.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.257+0000 7f54077fe640 1 -- 172.21.15.88:0/1375895454 learned_addr learned my addr 172.21.15.88:0/1375895454 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:26.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.257+0000 7f54077fe640 1 -- 172.21.15.88:0/1375895454 >> v1:172.21.15.154:6789/0 conn(0x7f54080d3310 legacy=0x7f54080d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:26.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.257+0000 7f54077fe640 1 -- 172.21.15.88:0/1375895454 >> v1:172.21.15.88:6789/0 conn(0x7f54080d1820 legacy=0x7f54080b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:26.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.257+0000 7f54077fe640 1 -- 172.21.15.88:0/1375895454 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54080d3e30 con 0x7f540815e660 2024-09-16T01:42:26.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.257+0000 7f54067fc640 1 -- 172.21.15.88:0/1375895454 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 382687320 0 0) 0x7f540815ea40 con 0x7f54080d1820 2024-09-16T01:42:26.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.258+0000 7f54077fe640 1 --2- 172.21.15.88:0/1375895454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f540815e660 0x7f54080cf730 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f53f8009870 tx=0x7f53f802a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf3936978d859ca server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:26.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.258+0000 7f54067fc640 1 -- 172.21.15.88:0/1375895454 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f8037070 con 0x7f540815e660 2024-09-16T01:42:26.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.258+0000 7f54067fc640 1 -- 172.21.15.88:0/1375895454 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53f8032440 con 0x7f540815e660 2024-09-16T01:42:26.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.258+0000 7f54067fc640 1 -- 172.21.15.88:0/1375895454 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f8032810 con 0x7f540815e660 2024-09-16T01:42:26.261 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:42:26.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.259+0000 7f540df2c640 1 -- 172.21.15.88:0/1375895454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f540815e660 msgr2=0x7f54080cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:26.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.259+0000 7f540df2c640 1 --2- 172.21.15.88:0/1375895454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f540815e660 0x7f54080cf730 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f53f8009870 tx=0x7f53f802a2a0 comp rx=0 tx=0).stop 2024-09-16T01:42:26.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.259+0000 7f540df2c640 1 -- 172.21.15.88:0/1375895454 shutdown_connections 2024-09-16T01:42:26.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.259+0000 7f540df2c640 1 --2- 172.21.15.88:0/1375895454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f540815e660 0x7f54080cf730 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:26.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.259+0000 7f540df2c640 1 -- 172.21.15.88:0/1375895454 >> 172.21.15.88:0/1375895454 conn(0x7f54080b6410 msgr2=0x7f54080b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:26.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.259+0000 7f540df2c640 1 -- 172.21.15.88:0/1375895454 shutdown_connections 2024-09-16T01:42:26.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.259+0000 7f540df2c640 1 -- 172.21.15.88:0/1375895454 wait complete. 2024-09-16T01:42:26.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.260+0000 7f540df2c640 1 Processor -- start 2024-09-16T01:42:26.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.260+0000 7f540df2c640 1 -- start start 2024-09-16T01:42:26.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.261+0000 7f540df2c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54080d1820 0x7f5408151b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:26.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.261+0000 7f540df2c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54080d3310 0x7f5408160f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:26.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.261+0000 7f540df2c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f540815e660 0x7f5408161460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:26.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.261+0000 7f540df2c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f54080cfc70 con 0x7f54080d3310 2024-09-16T01:42:26.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.261+0000 7f540df2c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f54080d43f0 con 0x7f54080d1820 2024-09-16T01:42:26.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.261+0000 7f540df2c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f54080b8280 con 0x7f540815e660 2024-09-16T01:42:26.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.261+0000 7f5406ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54080d3310 0x7f5408160f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:26.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.261+0000 7f5406ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54080d3310 0x7f5408160f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49354/0 (socket says 172.21.15.88:49354) 2024-09-16T01:42:26.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.261+0000 7f5406ffd640 1 -- 172.21.15.88:0/2251260610 learned_addr learned my addr 172.21.15.88:0/2251260610 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:26.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.261+0000 7f54077fe640 1 --2- 172.21.15.88:0/2251260610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54080d1820 0x7f5408151b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:26.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.261+0000 7f5407fff640 1 --2- 172.21.15.88:0/2251260610 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f540815e660 0x7f5408161460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:26.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.262+0000 7f5406ffd640 1 -- 172.21.15.88:0/2251260610 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f540815e660 msgr2=0x7f5408161460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:26.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.262+0000 7f5406ffd640 1 --2- 172.21.15.88:0/2251260610 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f540815e660 0x7f5408161460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:26.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.262+0000 7f5406ffd640 1 -- 172.21.15.88:0/2251260610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54080d1820 msgr2=0x7f5408151b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:26.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.262+0000 7f5406ffd640 1 --2- 172.21.15.88:0/2251260610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54080d1820 0x7f5408151b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:26.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.262+0000 7f5406ffd640 1 -- 172.21.15.88:0/2251260610 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f540815f670 con 0x7f54080d3310 2024-09-16T01:42:26.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.262+0000 7f5406ffd640 1 --2- 172.21.15.88:0/2251260610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54080d3310 0x7f5408160f20 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f53fc00c430 tx=0x7f53fc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:26.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.262+0000 7f53effff640 1 -- 172.21.15.88:0/2251260610 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53fc011790 con 0x7f54080d3310 2024-09-16T01:42:26.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.262+0000 7f53effff640 1 -- 172.21.15.88:0/2251260610 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53fc017070 con 0x7f54080d3310 2024-09-16T01:42:26.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.262+0000 7f540df2c640 1 -- 172.21.15.88:0/2251260610 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f540815f910 con 0x7f54080d3310 2024-09-16T01:42:26.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.262+0000 7f53effff640 1 -- 172.21.15.88:0/2251260610 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53fc011b70 con 0x7f54080d3310 2024-09-16T01:42:26.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.262+0000 7f540df2c640 1 -- 172.21.15.88:0/2251260610 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f540815fed0 con 0x7f54080d3310 2024-09-16T01:42:26.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.263+0000 7f540df2c640 1 -- 172.21.15.88:0/2251260610 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54080cf730 con 0x7f54080d3310 2024-09-16T01:42:26.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.264+0000 7f53effff640 1 -- 172.21.15.88:0/2251260610 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f53fc011d10 con 0x7f54080d3310 2024-09-16T01:42:26.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.264+0000 7f53effff640 1 --2- 172.21.15.88:0/2251260610 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f53f406c2b0 0x7f53f406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:26.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.264+0000 7f53effff640 1 -- 172.21.15.88:0/2251260610 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 7544+0+0 (secure 0 0 0) 0x7f53fc08b9c0 con 0x7f54080d3310 2024-09-16T01:42:26.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.264+0000 7f54077fe640 1 --2- 172.21.15.88:0/2251260610 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f53f406c2b0 0x7f53f406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:26.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.264+0000 7f54077fe640 1 --2- 172.21.15.88:0/2251260610 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f53f406c2b0 0x7f53f406e770 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f53f8034000 tx=0x7f53f802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:26.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.268+0000 7f53effff640 1 -- 172.21.15.88:0/2251260610 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f53fc08de20 con 0x7f54080d3310 2024-09-16T01:42:26.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.466+0000 7f540df2c640 1 -- 172.21.15.88:0/2251260610 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5408153130 con 0x7f54080d3310 2024-09-16T01:42:26.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.467+0000 7f53effff640 1 -- 172.21.15.88:0/2251260610 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) ==== 95+0+3566 (secure 0 0 0) 0x7f53fc0592a0 con 0x7f54080d3310 2024-09-16T01:42:26.469 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:26.469 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":193,"btime":"2024-09-16T01:42:25:506576+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":193,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:25.506484+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":108,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:26.469 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 193 2024-09-16T01:42:26.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.469+0000 7f540df2c640 1 -- 172.21.15.88:0/2251260610 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f53f406c2b0 msgr2=0x7f53f406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:26.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.469+0000 7f540df2c640 1 --2- 172.21.15.88:0/2251260610 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f53f406c2b0 0x7f53f406e770 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f53f8034000 tx=0x7f53f802e000 comp rx=0 tx=0).stop 2024-09-16T01:42:26.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.469+0000 7f540df2c640 1 -- 172.21.15.88:0/2251260610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54080d3310 msgr2=0x7f5408160f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:26.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.469+0000 7f540df2c640 1 --2- 172.21.15.88:0/2251260610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54080d3310 0x7f5408160f20 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f53fc00c430 tx=0x7f53fc00c8f0 comp rx=0 tx=0).stop 2024-09-16T01:42:26.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.469+0000 7f540df2c640 1 -- 172.21.15.88:0/2251260610 shutdown_connections 2024-09-16T01:42:26.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.469+0000 7f540df2c640 1 --2- 172.21.15.88:0/2251260610 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f53f406c2b0 0x7f53f406e770 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:26.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.469+0000 7f540df2c640 1 --2- 172.21.15.88:0/2251260610 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f540815e660 0x7f5408161460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:26.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.469+0000 7f540df2c640 1 --2- 172.21.15.88:0/2251260610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54080d3310 0x7f5408160f20 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:26.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.469+0000 7f540df2c640 1 --2- 172.21.15.88:0/2251260610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54080d1820 0x7f5408151b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:26.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.469+0000 7f540df2c640 1 -- 172.21.15.88:0/2251260610 >> 172.21.15.88:0/2251260610 conn(0x7f54080b6410 msgr2=0x7f540815d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:26.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.469+0000 7f540df2c640 1 -- 172.21.15.88:0/2251260610 shutdown_connections 2024-09-16T01:42:26.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.469+0000 7f540df2c640 1 -- 172.21.15.88:0/2251260610 wait complete. 2024-09-16T01:42:26.482 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt mds0_inotable' led to DAMAGED state 2024-09-16T01:42:26.482 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt 400.00000000' 2024-09-16T01:42:26.482 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:42:26.482 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:42:26.507 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:42:26.508 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:42:26.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:26.508 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:42:26.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:42:26.581 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:42:26.581 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:42:26.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.729+0000 7f8790b98640 1 Processor -- start 2024-09-16T01:42:26.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.729+0000 7f8790b98640 1 -- start start 2024-09-16T01:42:26.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.730+0000 7f8790b98640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f878c15c2a0 0x7f878c0735d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:26.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.730+0000 7f8790b98640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f878c073b10 0x7f878c06f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:26.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.730+0000 7f8790b98640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f878c074120 con 0x7f878c06fc50 2024-09-16T01:42:26.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.730+0000 7f8790b98640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f878c15c680 con 0x7f878c073b10 2024-09-16T01:42:26.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.730+0000 7f8790b98640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f878c15ef00 con 0x7f878c15c2a0 2024-09-16T01:42:26.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.730+0000 7f8783fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f878c073b10 0x7f878c06f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:26.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.730+0000 7f8783fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f878c073b10 0x7f878c06f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49368/0 (socket says 172.21.15.88:49368) 2024-09-16T01:42:26.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.730+0000 7f8783fff640 1 -- 172.21.15.88:0/633352394 learned_addr learned my addr 172.21.15.88:0/633352394 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:26.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.730+0000 7f8783fff640 1 -- 172.21.15.88:0/633352394 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f878c15c2a0 msgr2=0x7f878c0735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:26.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.730+0000 7f8783fff640 1 --2- 172.21.15.88:0/633352394 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f878c15c2a0 0x7f878c0735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:26.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.730+0000 7f8783fff640 1 -- 172.21.15.88:0/633352394 >> v1:172.21.15.154:6789/0 conn(0x7f878c06fc50 legacy=0x7f878c06cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:26.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.730+0000 7f8783fff640 1 -- 172.21.15.88:0/633352394 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f878c15f1e0 con 0x7f878c073b10 2024-09-16T01:42:26.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.731+0000 7f8783fff640 1 --2- 172.21.15.88:0/633352394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f878c073b10 0x7f878c06f710 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f8774009870 tx=0x7f877402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b88f397ca3df24e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:26.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.731+0000 7f8789d9b640 1 -- 172.21.15.88:0/633352394 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8774037070 con 0x7f878c073b10 2024-09-16T01:42:26.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.731+0000 7f8789d9b640 1 -- 172.21.15.88:0/633352394 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8774032440 con 0x7f878c073b10 2024-09-16T01:42:26.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.731+0000 7f8789d9b640 1 -- 172.21.15.88:0/633352394 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8774032810 con 0x7f878c073b10 2024-09-16T01:42:26.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.732+0000 7f8790b98640 1 -- 172.21.15.88:0/633352394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f878c073b10 msgr2=0x7f878c06f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:26.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.732+0000 7f8790b98640 1 --2- 172.21.15.88:0/633352394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f878c073b10 0x7f878c06f710 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f8774009870 tx=0x7f877402a210 comp rx=0 tx=0).stop 2024-09-16T01:42:26.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.732+0000 7f8790b98640 1 -- 172.21.15.88:0/633352394 shutdown_connections 2024-09-16T01:42:26.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.732+0000 7f8790b98640 1 --2- 172.21.15.88:0/633352394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f878c073b10 0x7f878c06f710 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:26.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.732+0000 7f8790b98640 1 --2- 172.21.15.88:0/633352394 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f878c15c2a0 0x7f878c0735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:26.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.732+0000 7f8790b98640 1 -- 172.21.15.88:0/633352394 >> 172.21.15.88:0/633352394 conn(0x7f878c0b3070 msgr2=0x7f878c0b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:26.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.732+0000 7f8790b98640 1 -- 172.21.15.88:0/633352394 shutdown_connections 2024-09-16T01:42:26.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.732+0000 7f8790b98640 1 -- 172.21.15.88:0/633352394 wait complete. 2024-09-16T01:42:26.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.733+0000 7f8790b98640 1 Processor -- start 2024-09-16T01:42:26.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.733+0000 7f8790b98640 1 -- start start 2024-09-16T01:42:26.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.734+0000 7f8790b98640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f878c06fc50 0x7f878c144950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:26.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.734+0000 7f8790b98640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f878c073b10 0x7f878c0af370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:26.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.734+0000 7f8790b98640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f878c15c2a0 0x7f878c0af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:26.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.734+0000 7f8790b98640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f878c0b5f20 con 0x7f878c073b10 2024-09-16T01:42:26.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.734+0000 7f8790b98640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f878c15ebf0 con 0x7f878c15c2a0 2024-09-16T01:42:26.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.734+0000 7f8790b98640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f878c0b60a0 con 0x7f878c06fc50 2024-09-16T01:42:26.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.734+0000 7f8783fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f878c073b10 0x7f878c0af370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:26.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.734+0000 7f8783fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f878c073b10 0x7f878c0af370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49374/0 (socket says 172.21.15.88:49374) 2024-09-16T01:42:26.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.734+0000 7f8783fff640 1 -- 172.21.15.88:0/997612807 learned_addr learned my addr 172.21.15.88:0/997612807 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:26.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.734+0000 7f878a59c640 1 --2- 172.21.15.88:0/997612807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f878c06fc50 0x7f878c144950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:26.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.734+0000 7f878ad9d640 1 --2- 172.21.15.88:0/997612807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f878c15c2a0 0x7f878c0af8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:26.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.734+0000 7f8783fff640 1 -- 172.21.15.88:0/997612807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f878c06fc50 msgr2=0x7f878c144950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:26.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.734+0000 7f8783fff640 1 --2- 172.21.15.88:0/997612807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f878c06fc50 0x7f878c144950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:26.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.734+0000 7f8783fff640 1 -- 172.21.15.88:0/997612807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f878c15c2a0 msgr2=0x7f878c0af8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:26.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.734+0000 7f8783fff640 1 --2- 172.21.15.88:0/997612807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f878c15c2a0 0x7f878c0af8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:26.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.734+0000 7f8783fff640 1 -- 172.21.15.88:0/997612807 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f878c15cbf0 con 0x7f878c073b10 2024-09-16T01:42:26.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.734+0000 7f878ad9d640 1 --2- 172.21.15.88:0/997612807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f878c15c2a0 0x7f878c0af8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:26.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.734+0000 7f8783fff640 1 --2- 172.21.15.88:0/997612807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f878c073b10 0x7f878c0af370 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f8774000c00 tx=0x7f87740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:26.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.734+0000 7f8782ffd640 1 -- 172.21.15.88:0/997612807 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8774041070 con 0x7f878c073b10 2024-09-16T01:42:26.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.734+0000 7f8782ffd640 1 -- 172.21.15.88:0/997612807 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f877403c870 con 0x7f878c073b10 2024-09-16T01:42:26.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.734+0000 7f8790b98640 1 -- 172.21.15.88:0/997612807 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f878c15ced0 con 0x7f878c073b10 2024-09-16T01:42:26.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.735+0000 7f8782ffd640 1 -- 172.21.15.88:0/997612807 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8774037040 con 0x7f878c073b10 2024-09-16T01:42:26.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.735+0000 7f8790b98640 1 -- 172.21.15.88:0/997612807 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f878c15d490 con 0x7f878c073b10 2024-09-16T01:42:26.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.735+0000 7f8790b98640 1 -- 172.21.15.88:0/997612807 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f878c0d1dc0 con 0x7f878c073b10 2024-09-16T01:42:26.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.736+0000 7f8782ffd640 1 -- 172.21.15.88:0/997612807 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8774044070 con 0x7f878c073b10 2024-09-16T01:42:26.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.736+0000 7f8782ffd640 1 --2- 172.21.15.88:0/997612807 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f876c06c2b0 0x7f876c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:26.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.736+0000 7f8782ffd640 1 -- 172.21.15.88:0/997612807 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 7544+0+0 (secure 0 0 0) 0x7f87740ac660 con 0x7f878c073b10 2024-09-16T01:42:26.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.736+0000 7f878a59c640 1 --2- 172.21.15.88:0/997612807 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f876c06c2b0 0x7f876c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:26.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.737+0000 7f878a59c640 1 --2- 172.21.15.88:0/997612807 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f876c06c2b0 0x7f876c06e770 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f877c009990 tx=0x7f877c009110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:26.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.741+0000 7f8782ffd640 1 -- 172.21.15.88:0/997612807 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f877407a7c0 con 0x7f878c073b10 2024-09-16T01:42:26.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:26.939+0000 7f8790b98640 1 -- 172.21.15.88:0/997612807 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f878c0d1ae0 con 0x7f878c073b10 2024-09-16T01:42:27.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.514+0000 7f8782ffd640 1 -- 172.21.15.88:0/997612807 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v195) ==== 161+0+0 (secure 0 0 0) 0x7f877407a0c0 con 0x7f878c073b10 2024-09-16T01:42:27.518 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:42:27.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.517+0000 7f8790b98640 1 -- 172.21.15.88:0/997612807 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f876c06c2b0 msgr2=0x7f876c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:27.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.517+0000 7f8790b98640 1 --2- 172.21.15.88:0/997612807 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f876c06c2b0 0x7f876c06e770 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f877c009990 tx=0x7f877c009110 comp rx=0 tx=0).stop 2024-09-16T01:42:27.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.517+0000 7f8790b98640 1 -- 172.21.15.88:0/997612807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f878c073b10 msgr2=0x7f878c0af370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:27.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.517+0000 7f8790b98640 1 --2- 172.21.15.88:0/997612807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f878c073b10 0x7f878c0af370 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f8774000c00 tx=0x7f87740046c0 comp rx=0 tx=0).stop 2024-09-16T01:42:27.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.517+0000 7f8790b98640 1 -- 172.21.15.88:0/997612807 shutdown_connections 2024-09-16T01:42:27.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.517+0000 7f8790b98640 1 --2- 172.21.15.88:0/997612807 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f876c06c2b0 0x7f876c06e770 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:27.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.517+0000 7f8790b98640 1 --2- 172.21.15.88:0/997612807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f878c15c2a0 0x7f878c0af8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:27.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.517+0000 7f8790b98640 1 --2- 172.21.15.88:0/997612807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f878c073b10 0x7f878c0af370 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:27.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.517+0000 7f8790b98640 1 --2- 172.21.15.88:0/997612807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f878c06fc50 0x7f878c144950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:27.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.517+0000 7f8790b98640 1 -- 172.21.15.88:0/997612807 >> 172.21.15.88:0/997612807 conn(0x7f878c0b3070 msgr2=0x7f878c15b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:27.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.517+0000 7f8790b98640 1 -- 172.21.15.88:0/997612807 shutdown_connections 2024-09-16T01:42:27.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.517+0000 7f8790b98640 1 -- 172.21.15.88:0/997612807 wait complete. 2024-09-16T01:42:27.531 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:42:27.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.635+0000 7f18c502e640 1 Processor -- start 2024-09-16T01:42:27.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.635+0000 7f18c502e640 1 -- start start 2024-09-16T01:42:27.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.635+0000 7f18c502e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f18c015b480 0x7f18c00b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:27.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.636+0000 7f18c502e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f18c00b60c0 0x7f18c0071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:27.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.636+0000 7f18c502e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18c00ccff0 con 0x7f18c0072530 2024-09-16T01:42:27.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.636+0000 7f18c502e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f18c0072980 con 0x7f18c00b60c0 2024-09-16T01:42:27.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.636+0000 7f18c502e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f18c00cd2d0 con 0x7f18c015b480 2024-09-16T01:42:27.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.636+0000 7f18b659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f18c00b60c0 0x7f18c0071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:27.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.636+0000 7f18b659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f18c00b60c0 0x7f18c0071ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49384/0 (socket says 172.21.15.88:49384) 2024-09-16T01:42:27.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.636+0000 7f18b659c640 1 -- 172.21.15.88:0/4063481518 learned_addr learned my addr 172.21.15.88:0/4063481518 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:27.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.636+0000 7f18b659c640 1 -- 172.21.15.88:0/4063481518 >> v2:172.21.15.154:3300/0 conn(0x7f18c015b480 msgr2=0x7f18c00b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:27.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.636+0000 7f18b659c640 1 --2- 172.21.15.88:0/4063481518 >> v2:172.21.15.154:3300/0 conn(0x7f18c015b480 0x7f18c00b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:27.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.636+0000 7f18b659c640 1 -- 172.21.15.88:0/4063481518 >> v1:172.21.15.154:6789/0 conn(0x7f18c0072530 legacy=0x7f18c0070fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:27.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.636+0000 7f18b659c640 1 -- 172.21.15.88:0/4063481518 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c00cd5b0 con 0x7f18c00b60c0 2024-09-16T01:42:27.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.637+0000 7f18b659c640 1 --2- 172.21.15.88:0/4063481518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18c00b60c0 0x7f18c0071ff0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f18c00694d0 tx=0x7f18a802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a88ee6a5dcaab3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:27.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.637+0000 7f18be59c640 1 -- 172.21.15.88:0/4063481518 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a8037070 con 0x7f18c00b60c0 2024-09-16T01:42:27.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.637+0000 7f18be59c640 1 -- 172.21.15.88:0/4063481518 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18a8032440 con 0x7f18c00b60c0 2024-09-16T01:42:27.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.637+0000 7f18be59c640 1 -- 172.21.15.88:0/4063481518 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a8032810 con 0x7f18c00b60c0 2024-09-16T01:42:27.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.637+0000 7f18c502e640 1 -- 172.21.15.88:0/4063481518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18c00b60c0 msgr2=0x7f18c0071ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:27.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.637+0000 7f18c502e640 1 --2- 172.21.15.88:0/4063481518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18c00b60c0 0x7f18c0071ff0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f18c00694d0 tx=0x7f18a802a230 comp rx=0 tx=0).stop 2024-09-16T01:42:27.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.637+0000 7f18c502e640 1 -- 172.21.15.88:0/4063481518 shutdown_connections 2024-09-16T01:42:27.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.637+0000 7f18c502e640 1 --2- 172.21.15.88:0/4063481518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18c00b60c0 0x7f18c0071ff0 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:27.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.637+0000 7f18c502e640 1 --2- 172.21.15.88:0/4063481518 >> v2:172.21.15.154:3300/0 conn(0x7f18c015b480 0x7f18c00b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:27.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.637+0000 7f18c502e640 1 -- 172.21.15.88:0/4063481518 >> 172.21.15.88:0/4063481518 conn(0x7f18c006d0f0 msgr2=0x7f18c006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:27.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.637+0000 7f18c502e640 1 -- 172.21.15.88:0/4063481518 shutdown_connections 2024-09-16T01:42:27.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.637+0000 7f18c502e640 1 -- 172.21.15.88:0/4063481518 wait complete. 2024-09-16T01:42:27.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.639+0000 7f18c502e640 1 Processor -- start 2024-09-16T01:42:27.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.639+0000 7f18c502e640 1 -- start start 2024-09-16T01:42:27.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.640+0000 7f18c502e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18c0072530 0x7f18c015cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:27.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.640+0000 7f18c502e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18c00b60c0 0x7f18c01f5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:27.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.640+0000 7f18c502e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18c015b480 0x7f18c00b3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:27.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.640+0000 7f18c502e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f18c015b860 con 0x7f18c00b60c0 2024-09-16T01:42:27.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.640+0000 7f18c502e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f18c00cdb70 con 0x7f18c015b480 2024-09-16T01:42:27.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.640+0000 7f18c502e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f18c015b9e0 con 0x7f18c0072530 2024-09-16T01:42:27.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.640+0000 7f18b659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18c00b60c0 0x7f18c01f5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:27.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.640+0000 7f18b659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18c00b60c0 0x7f18c01f5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49392/0 (socket says 172.21.15.88:49392) 2024-09-16T01:42:27.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.640+0000 7f18b659c640 1 -- 172.21.15.88:0/2028152510 learned_addr learned my addr 172.21.15.88:0/2028152510 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:27.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.640+0000 7f18bf59e640 1 --2- 172.21.15.88:0/2028152510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18c015b480 0x7f18c00b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:27.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.640+0000 7f18bed9d640 1 --2- 172.21.15.88:0/2028152510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18c0072530 0x7f18c015cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:27.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.640+0000 7f18b659c640 1 -- 172.21.15.88:0/2028152510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18c0072530 msgr2=0x7f18c015cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:27.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.640+0000 7f18b659c640 1 --2- 172.21.15.88:0/2028152510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18c0072530 0x7f18c015cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:27.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.640+0000 7f18b659c640 1 -- 172.21.15.88:0/2028152510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18c015b480 msgr2=0x7f18c00b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:27.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.640+0000 7f18b659c640 1 --2- 172.21.15.88:0/2028152510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18c015b480 0x7f18c00b3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:27.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.640+0000 7f18b659c640 1 -- 172.21.15.88:0/2028152510 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c01e6530 con 0x7f18c00b60c0 2024-09-16T01:42:27.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.640+0000 7f18bf59e640 1 --2- 172.21.15.88:0/2028152510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18c015b480 0x7f18c00b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:27.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.640+0000 7f18b659c640 1 --2- 172.21.15.88:0/2028152510 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18c00b60c0 0x7f18c01f5390 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f18a8000c00 tx=0x7f18a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:27.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.640+0000 7f18b7fff640 1 -- 172.21.15.88:0/2028152510 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a8041070 con 0x7f18c00b60c0 2024-09-16T01:42:27.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.640+0000 7f18b7fff640 1 -- 172.21.15.88:0/2028152510 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18a803c870 con 0x7f18c00b60c0 2024-09-16T01:42:27.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.640+0000 7f18b7fff640 1 -- 172.21.15.88:0/2028152510 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a8037040 con 0x7f18c00b60c0 2024-09-16T01:42:27.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.640+0000 7f18c502e640 1 -- 172.21.15.88:0/2028152510 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18c01e6810 con 0x7f18c00b60c0 2024-09-16T01:42:27.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.641+0000 7f18c502e640 1 -- 172.21.15.88:0/2028152510 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18c01e6dd0 con 0x7f18c00b60c0 2024-09-16T01:42:27.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.642+0000 7f18c502e640 1 -- 172.21.15.88:0/2028152510 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18c0071ff0 con 0x7f18c00b60c0 2024-09-16T01:42:27.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.642+0000 7f18b7fff640 1 -- 172.21.15.88:0/2028152510 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f18a8044070 con 0x7f18c00b60c0 2024-09-16T01:42:27.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.642+0000 7f18b7fff640 1 --2- 172.21.15.88:0/2028152510 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f18a006c2e0 0x7f18a006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:27.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.643+0000 7f18b7fff640 1 -- 172.21.15.88:0/2028152510 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 7544+0+0 (secure 0 0 0) 0x7f18a80ac8c0 con 0x7f18c00b60c0 2024-09-16T01:42:27.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.643+0000 7f18bed9d640 1 --2- 172.21.15.88:0/2028152510 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f18a006c2e0 0x7f18a006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:27.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.643+0000 7f18bed9d640 1 --2- 172.21.15.88:0/2028152510 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f18a006c2e0 0x7f18a006e7a0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f18b000b810 tx=0x7f18b0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:27.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.647+0000 7f18b7fff640 1 -- 172.21.15.88:0/2028152510 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f18a807aab0 con 0x7f18c00b60c0 2024-09-16T01:42:27.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:27.880+0000 7f18c502e640 1 -- 172.21.15.88:0/2028152510 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f18c00d10c0 con 0x7f18c00b60c0 2024-09-16T01:42:28.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.519+0000 7f18b7fff640 1 -- 172.21.15.88:0/2028152510 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v196) ==== 101+0+0 (secure 0 0 0) 0x7f18a807a3b0 con 0x7f18c00b60c0 2024-09-16T01:42:28.521 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:42:28.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.521+0000 7f18c502e640 1 -- 172.21.15.88:0/2028152510 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f18a006c2e0 msgr2=0x7f18a006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:28.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.521+0000 7f18c502e640 1 --2- 172.21.15.88:0/2028152510 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f18a006c2e0 0x7f18a006e7a0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f18b000b810 tx=0x7f18b0008040 comp rx=0 tx=0).stop 2024-09-16T01:42:28.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.521+0000 7f18c502e640 1 -- 172.21.15.88:0/2028152510 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18c00b60c0 msgr2=0x7f18c01f5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:28.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.521+0000 7f18c502e640 1 --2- 172.21.15.88:0/2028152510 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18c00b60c0 0x7f18c01f5390 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f18a8000c00 tx=0x7f18a80046c0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.521+0000 7f18c502e640 1 -- 172.21.15.88:0/2028152510 shutdown_connections 2024-09-16T01:42:28.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.521+0000 7f18c502e640 1 --2- 172.21.15.88:0/2028152510 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f18a006c2e0 0x7f18a006e7a0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.521+0000 7f18c502e640 1 --2- 172.21.15.88:0/2028152510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18c015b480 0x7f18c00b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.521+0000 7f18c502e640 1 --2- 172.21.15.88:0/2028152510 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18c00b60c0 0x7f18c01f5390 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.521+0000 7f18c502e640 1 --2- 172.21.15.88:0/2028152510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18c0072530 0x7f18c015cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.521+0000 7f18c502e640 1 -- 172.21.15.88:0/2028152510 >> 172.21.15.88:0/2028152510 conn(0x7f18c006d0f0 msgr2=0x7f18c00d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:28.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.522+0000 7f18c502e640 1 -- 172.21.15.88:0/2028152510 shutdown_connections 2024-09-16T01:42:28.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.522+0000 7f18c502e640 1 -- 172.21.15.88:0/2028152510 wait complete. 2024-09-16T01:42:28.534 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:42:28.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.575+0000 7f353e11df40 1 Processor -- start 2024-09-16T01:42:28.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.575+0000 7f353e11df40 1 -- start start 2024-09-16T01:42:28.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.576+0000 7f353e11df40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d50624ddf0 0x55d5063b2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.576+0000 7f353e11df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d50624e710 0x55d506306f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.576+0000 7f353e11df40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d50630ae00 0x55d50624ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.576+0000 7f353e11df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d506323640 con 0x55d50624e710 2024-09-16T01:42:28.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.576+0000 7f353e11df40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d50619e510 con 0x55d50624ddf0 2024-09-16T01:42:28.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.576+0000 7f353e11df40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55d5063079e0 con 0x55d50630ae00 2024-09-16T01:42:28.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.576+0000 7f353a964640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d50624e710 0x55d506306f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.576+0000 7f353a964640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d50624e710 0x55d506306f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49410/0 (socket says 172.21.15.88:49410) 2024-09-16T01:42:28.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.576+0000 7f353a964640 1 -- 172.21.15.88:0/2558248249 learned_addr learned my addr 172.21.15.88:0/2558248249 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:28.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.576+0000 7f353b966640 1 --2- 172.21.15.88:0/2558248249 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d50630ae00 0x55d50624ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.576+0000 7f353b165640 1 --2- 172.21.15.88:0/2558248249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d50624ddf0 0x55d5063b2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.576+0000 7f353a964640 1 -- 172.21.15.88:0/2558248249 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d50630ae00 msgr2=0x55d50624ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:28.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.576+0000 7f353a964640 1 --2- 172.21.15.88:0/2558248249 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d50630ae00 0x55d50624ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.576+0000 7f353a964640 1 -- 172.21.15.88:0/2558248249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d50624ddf0 msgr2=0x55d5063b2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:28.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.576+0000 7f353a964640 1 --2- 172.21.15.88:0/2558248249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d50624ddf0 0x55d5063b2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.576+0000 7f353a964640 1 -- 172.21.15.88:0/2558248249 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d50627b810 con 0x55d50624e710 2024-09-16T01:42:28.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.576+0000 7f353b966640 1 --2- 172.21.15.88:0/2558248249 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d50630ae00 0x55d50624ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:28.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.577+0000 7f353a964640 1 --2- 172.21.15.88:0/2558248249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d50624e710 0x55d506306f70 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f3530045c00 tx=0x7f3530051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b5b043cfb7cdb88 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:28.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.577+0000 7f3539962640 1 -- 172.21.15.88:0/2558248249 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530068070 con 0x55d50624e710 2024-09-16T01:42:28.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.577+0000 7f3539962640 1 -- 172.21.15.88:0/2558248249 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3530058c10 con 0x55d50624e710 2024-09-16T01:42:28.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.577+0000 7f3539962640 1 -- 172.21.15.88:0/2558248249 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530063470 con 0x55d50624e710 2024-09-16T01:42:28.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.577+0000 7f353e11df40 1 -- 172.21.15.88:0/2558248249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d50624e710 msgr2=0x55d506306f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:28.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.577+0000 7f353e11df40 1 --2- 172.21.15.88:0/2558248249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d50624e710 0x55d506306f70 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f3530045c00 tx=0x7f3530051000 comp rx=0 tx=0).stop 2024-09-16T01:42:28.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.577+0000 7f353e11df40 1 -- 172.21.15.88:0/2558248249 shutdown_connections 2024-09-16T01:42:28.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.577+0000 7f353e11df40 1 --2- 172.21.15.88:0/2558248249 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d50630ae00 0x55d50624ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.577+0000 7f353e11df40 1 --2- 172.21.15.88:0/2558248249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d50624e710 0x55d506306f70 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.577+0000 7f353e11df40 1 --2- 172.21.15.88:0/2558248249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d50624ddf0 0x55d5063b2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.577+0000 7f353e11df40 1 -- 172.21.15.88:0/2558248249 >> 172.21.15.88:0/2558248249 conn(0x55d506277960 msgr2=0x55d506308cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:28.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.577+0000 7f353e11df40 1 -- 172.21.15.88:0/2558248249 shutdown_connections 2024-09-16T01:42:28.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.577+0000 7f353e11df40 1 -- 172.21.15.88:0/2558248249 wait complete. 2024-09-16T01:42:28.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.578+0000 7f353e11df40 1 Processor -- start 2024-09-16T01:42:28.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.578+0000 7f353e11df40 1 -- start start 2024-09-16T01:42:28.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.579+0000 7f353e11df40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d50624ddf0 0x55d50626e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.579+0000 7f353e11df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d50624e710 0x55d50624cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.579+0000 7f353e11df40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d50630ae00 0x55d50628dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.579+0000 7f353e11df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d50627b530 con 0x55d50624e710 2024-09-16T01:42:28.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.579+0000 7f353e11df40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d506276670 con 0x55d50624ddf0 2024-09-16T01:42:28.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.579+0000 7f353e11df40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55d50624bf90 con 0x55d50630ae00 2024-09-16T01:42:28.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.579+0000 7f353a964640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d50624e710 0x55d50624cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.579+0000 7f353a964640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d50624e710 0x55d50624cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49414/0 (socket says 172.21.15.88:49414) 2024-09-16T01:42:28.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.579+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 learned_addr learned my addr 172.21.15.88:0/4095020808 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:28.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.579+0000 7f353b165640 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d50624ddf0 0x55d50626e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.579+0000 7f353b966640 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d50630ae00 0x55d50628dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.579+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d50630ae00 msgr2=0x55d50628dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:28.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.579+0000 7f353a964640 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d50630ae00 0x55d50628dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.579+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d50624ddf0 msgr2=0x55d50626e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:28.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.579+0000 7f353a964640 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d50624ddf0 0x55d50626e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.579+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d506309510 con 0x55d50624e710 2024-09-16T01:42:28.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.579+0000 7f353b966640 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d50630ae00 0x55d50628dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:28.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.579+0000 7f353a964640 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d50624e710 0x55d50624cfd0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f3530025dc0 tx=0x7f3530002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:28.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.579+0000 7f3538960640 1 -- 172.21.15.88:0/4095020808 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530068070 con 0x55d50624e710 2024-09-16T01:42:28.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.579+0000 7f3538960640 1 -- 172.21.15.88:0/4095020808 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3530069040 con 0x55d50624e710 2024-09-16T01:42:28.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.579+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d506305990 con 0x55d50624e710 2024-09-16T01:42:28.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.579+0000 7f3538960640 1 -- 172.21.15.88:0/4095020808 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530063690 con 0x55d50624e710 2024-09-16T01:42:28.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.579+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d506305f50 con 0x55d50624e710 2024-09-16T01:42:28.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.581+0000 7f3538960640 1 -- 172.21.15.88:0/4095020808 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f353006d440 con 0x55d50624e710 2024-09-16T01:42:28.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.581+0000 7f3538960640 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f352806ac90 0x7f352806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.581+0000 7f3538960640 1 -- 172.21.15.88:0/4095020808 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 7544+0+0 (secure 0 0 0) 0x7f35300cfdb0 con 0x55d50624e710 2024-09-16T01:42:28.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.581+0000 7f353b165640 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f352806ac90 0x7f352806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.587 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:42:28.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.581+0000 7f353e11df40 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55d50624d510 0x55d5063be360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.581+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063229f0 con 0x55d50624d510 2024-09-16T01:42:28.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.582+0000 7f353b966640 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55d50624d510 0x55d5063be360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.582+0000 7f353b165640 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f352806ac90 0x7f352806d150 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f352c00d6e0 tx=0x7f352c00f040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:28.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.582+0000 7f353b966640 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55d50624d510 0x55d5063be360 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:28.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.583+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v108'629 uv603 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.584+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e108) -- 0x55d506322430 con 0x55d50624d510 2024-09-16T01:42:28.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.585+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v108'630 uv630 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.585+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063be8a0 con 0x55d50624d510 2024-09-16T01:42:28.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.586+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v108'631 uv631 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.589 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:42:28.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.586+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063bec40 con 0x55d50624d510 2024-09-16T01:42:28.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.587+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v108'632 uv632 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.587+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c8640 con 0x55d50624d510 2024-09-16T01:42:28.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.588+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v108'633 uv633 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.588+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7630 con 0x55d50624d510 2024-09-16T01:42:28.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.589+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v108'634 uv634 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.589+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c79d0 con 0x55d50624d510 2024-09-16T01:42:28.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.591+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v108'635 uv609 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.591+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7dc0 con 0x55d50624d510 2024-09-16T01:42:28.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.592+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v108'636 uv636 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.592+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7dc0 con 0x55d50624d510 2024-09-16T01:42:28.594 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:42:28.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.593+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v108'637 uv637 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.593+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7d70 con 0x55d50624d510 2024-09-16T01:42:28.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.594+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v108'638 uv638 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.594+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7d70 con 0x55d50624d510 2024-09-16T01:42:28.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.595+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v108'639 uv639 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.595+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7d70 con 0x55d50624d510 2024-09-16T01:42:28.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.597+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v108'640 uv640 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.597+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7d70 con 0x55d50624d510 2024-09-16T01:42:28.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.598+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v108'641 uv616 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.598+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7d70 con 0x55d50624d510 2024-09-16T01:42:28.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.599+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v108'642 uv642 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.599+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7d70 con 0x55d50624d510 2024-09-16T01:42:28.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.600+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v108'643 uv643 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.601 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:42:28.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.600+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7d70 con 0x55d50624d510 2024-09-16T01:42:28.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.601+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v108'644 uv644 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.601+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7d70 con 0x55d50624d510 2024-09-16T01:42:28.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.602+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v108'645 uv645 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.602+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7d70 con 0x55d50624d510 2024-09-16T01:42:28.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.603+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v108'646 uv646 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.604+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7d70 con 0x55d50624d510 2024-09-16T01:42:28.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.605+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v108'647 uv647 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.605+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7d70 con 0x55d50624d510 2024-09-16T01:42:28.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.606+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v108'648 uv622 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.606+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7d70 con 0x55d50624d510 2024-09-16T01:42:28.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.607+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v108'649 uv649 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.607+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7d70 con 0x55d50624d510 2024-09-16T01:42:28.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.608+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v108'650 uv650 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.610 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:42:28.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.608+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7d70 con 0x55d50624d510 2024-09-16T01:42:28.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.609+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v108'651 uv651 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.609+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7d70 con 0x55d50624d510 2024-09-16T01:42:28.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.610+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v108'652 uv652 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.610+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7d70 con 0x55d50624d510 2024-09-16T01:42:28.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.611+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v108'653 uv653 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.611+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7d70 con 0x55d50624d510 2024-09-16T01:42:28.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.613+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v108'654 uv628 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.613+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7d70 con 0x55d50624d510 2024-09-16T01:42:28.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.614+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v108'655 uv655 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.614+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7d70 con 0x55d50624d510 2024-09-16T01:42:28.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.615+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v108'656 uv656 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.616 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:42:28.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.615+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7d70 con 0x55d50624d510 2024-09-16T01:42:28.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.616+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v108'657 uv657 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.616+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7d70 con 0x55d50624d510 2024-09-16T01:42:28.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.617+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v108'658 uv658 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.617+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7d70 con 0x55d50624d510 2024-09-16T01:42:28.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.618+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v108'659 uv659 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.618+0000 7f353e11df40 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55d5063c81f0 0x55d5063caf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.618+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063c7d70 con 0x55d5063c81f0 2024-09-16T01:42:28.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.618+0000 7f353b165640 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55d5063c81f0 0x55d5063caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.618+0000 7f353b165640 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55d5063c81f0 0x55d5063caf00 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:28.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.620+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v108'464 uv447 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063c81f0 2024-09-16T01:42:28.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.620+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e108) -- 0x55d5063cb440 con 0x55d5063c81f0 2024-09-16T01:42:28.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.621+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v108'465 uv465 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063c81f0 2024-09-16T01:42:28.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.621+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063cb800 con 0x55d5063c81f0 2024-09-16T01:42:28.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.622+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v108'466 uv466 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063c81f0 2024-09-16T01:42:28.623 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:42:28.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.622+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063cc0f0 con 0x55d5063c81f0 2024-09-16T01:42:28.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.623+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v108'467 uv467 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063c81f0 2024-09-16T01:42:28.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.623+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063cc490 con 0x55d5063c81f0 2024-09-16T01:42:28.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.624+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v108'468 uv468 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063c81f0 2024-09-16T01:42:28.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.624+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063cc830 con 0x55d5063c81f0 2024-09-16T01:42:28.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.625+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v108'469 uv452 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063c81f0 2024-09-16T01:42:28.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.625+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e108) -- 0x55d5063ccbd0 con 0x55d5063c81f0 2024-09-16T01:42:28.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.626+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v108'470 uv470 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063c81f0 2024-09-16T01:42:28.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.626+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063ccf70 con 0x55d5063c81f0 2024-09-16T01:42:28.629 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:42:28.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.627+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v108'471 uv471 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063c81f0 2024-09-16T01:42:28.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.627+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063ccf70 con 0x55d5063c81f0 2024-09-16T01:42:28.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.629+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v108'472 uv472 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063c81f0 2024-09-16T01:42:28.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.629+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063ccf70 con 0x55d5063c81f0 2024-09-16T01:42:28.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.630+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v108'473 uv473 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063c81f0 2024-09-16T01:42:28.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.630+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063ccf70 con 0x55d5063c81f0 2024-09-16T01:42:28.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.631+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v108'474 uv457 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063c81f0 2024-09-16T01:42:28.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.631+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e108) -- 0x55d5063ccf70 con 0x55d5063c81f0 2024-09-16T01:42:28.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.632+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v108'475 uv475 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063c81f0 2024-09-16T01:42:28.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.632+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063ccf70 con 0x55d5063c81f0 2024-09-16T01:42:28.634 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:42:28.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.633+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v108'476 uv476 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063c81f0 2024-09-16T01:42:28.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.633+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d13f0 con 0x55d5063c81f0 2024-09-16T01:42:28.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.635+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v108'477 uv477 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063c81f0 2024-09-16T01:42:28.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.635+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d13f0 con 0x55d5063c81f0 2024-09-16T01:42:28.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.636+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v108'478 uv478 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063c81f0 2024-09-16T01:42:28.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.636+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063ccf70 con 0x55d5063c81f0 2024-09-16T01:42:28.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.637+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v108'479 uv463 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063c81f0 2024-09-16T01:42:28.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.637+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e108) -- 0x55d5063ccf70 con 0x55d5063c81f0 2024-09-16T01:42:28.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.638+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v108'480 uv480 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063c81f0 2024-09-16T01:42:28.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.638+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063ccf70 con 0x55d5063c81f0 2024-09-16T01:42:28.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.639+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v108'481 uv481 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063c81f0 2024-09-16T01:42:28.640 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:42:28.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.639+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063cdf80 con 0x55d5063c81f0 2024-09-16T01:42:28.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.640+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v108'482 uv482 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063c81f0 2024-09-16T01:42:28.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.640+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063cdf80 con 0x55d5063c81f0 2024-09-16T01:42:28.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.641+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v108'483 uv483 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063c81f0 2024-09-16T01:42:28.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.641+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063ccf70 con 0x55d5063c81f0 2024-09-16T01:42:28.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.642+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v108'484 uv484 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063c81f0 2024-09-16T01:42:28.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.642+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063ccf70 con 0x55d50624d510 2024-09-16T01:42:28.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.643+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v108'123 uv122 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.643+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e108) -- 0x55d5063ccf70 con 0x55d50624d510 2024-09-16T01:42:28.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.644+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v108'124 uv124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.644+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063ccf70 con 0x55d50624d510 2024-09-16T01:42:28.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.645+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v108'125 uv125 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.647 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:42:28.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.645+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063ce360 con 0x55d50624d510 2024-09-16T01:42:28.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.646+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v108'126 uv126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.646+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063ce360 con 0x55d50624d510 2024-09-16T01:42:28.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.647+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v108'127 uv127 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.647+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063ccf70 con 0x55d50624d510 2024-09-16T01:42:28.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.648+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v108'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.648+0000 7f353e11df40 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55d5063cd890 0x55d5063d0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.648+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063ccf70 con 0x55d5063cd890 2024-09-16T01:42:28.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.648+0000 7f353a964640 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55d5063cd890 0x55d5063d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.648+0000 7f353a964640 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55d5063cd890 0x55d5063d0370 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:28.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.649+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v108'93 uv92 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f353006b070 con 0x55d5063cd890 2024-09-16T01:42:28.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.649+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d08b0 con 0x55d5063cd890 2024-09-16T01:42:28.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.650+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v108'94 uv94 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f353006b070 con 0x55d5063cd890 2024-09-16T01:42:28.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.650+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d0c50 con 0x55d5063cd890 2024-09-16T01:42:28.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.651+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v108'95 uv95 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f353006b070 con 0x55d5063cd890 2024-09-16T01:42:28.652 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:42:28.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.651+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063cbd20 con 0x55d5063cd890 2024-09-16T01:42:28.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.652+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v108'96 uv96 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f353006b070 con 0x55d5063cd890 2024-09-16T01:42:28.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.652+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d1560 con 0x55d50624d510 2024-09-16T01:42:28.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.653+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v108'123 uv122 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.653+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d1560 con 0x55d50624d510 2024-09-16T01:42:28.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.657+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v108'124 uv124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.657+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d1560 con 0x55d50624d510 2024-09-16T01:42:28.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.658+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v108'125 uv125 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.660 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:42:28.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.659+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d29b0 con 0x55d50624d510 2024-09-16T01:42:28.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.661+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v108'126 uv126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.661+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d29b0 con 0x55d50624d510 2024-09-16T01:42:28.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.663+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v108'127 uv127 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.663+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d16c0 con 0x55d50624d510 2024-09-16T01:42:28.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.668+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v108'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.668+0000 7f353e11df40 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55d5063d1d40 0x55d5063d4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.668+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d4790 con 0x55d5063d1d40 2024-09-16T01:42:28.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.669+0000 7f353b966640 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55d5063d1d40 0x55d5063d4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.669+0000 7f353b966640 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55d5063d1d40 0x55d5063d4130 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:28.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.670+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v108'227 uv221 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.670+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d4850 con 0x55d5063d1d40 2024-09-16T01:42:28.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.671+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v108'228 uv228 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.671+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d4c40 con 0x55d5063d1d40 2024-09-16T01:42:28.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.672+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v108'229 uv229 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.674 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:42:28.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.672+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d5c50 con 0x55d5063d1d40 2024-09-16T01:42:28.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.673+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v108'230 uv230 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.673+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d5c50 con 0x55d5063d1d40 2024-09-16T01:42:28.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.675+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v108'231 uv231 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.675+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d4c40 con 0x55d5063d1d40 2024-09-16T01:42:28.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.676+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v108'232 uv232 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.676+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d4c40 con 0x55d5063d1d40 2024-09-16T01:42:28.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.677+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v108'233 uv226 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.677+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d4c40 con 0x55d5063d1d40 2024-09-16T01:42:28.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.678+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v108'234 uv234 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.678+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d4c40 con 0x55d5063d1d40 2024-09-16T01:42:28.680 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:42:28.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.679+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v108'235 uv235 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.679+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d4c40 con 0x55d5063d1d40 2024-09-16T01:42:28.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.680+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v108'236 uv236 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.680+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d4c40 con 0x55d5063d1d40 2024-09-16T01:42:28.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.681+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v108'237 uv237 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.681+0000 7f353e11df40 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55d5063d5150 0x55d5063d7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.681+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d7c00 con 0x55d5063d5150 2024-09-16T01:42:28.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.681+0000 7f353b165640 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55d5063d5150 0x55d5063d7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.682+0000 7f353b165640 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55d5063d5150 0x55d5063d7580 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:28.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.683+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v108'102 uv101 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063d5150 2024-09-16T01:42:28.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.683+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d7fd0 con 0x55d5063d5150 2024-09-16T01:42:28.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.684+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v108'103 uv103 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063d5150 2024-09-16T01:42:28.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.684+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d7fd0 con 0x55d5063d5150 2024-09-16T01:42:28.686 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:42:28.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.685+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v108'104 uv104 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063d5150 2024-09-16T01:42:28.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.685+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d7fd0 con 0x55d5063d5150 2024-09-16T01:42:28.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.686+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v108'105 uv105 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063d5150 2024-09-16T01:42:28.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.686+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d7fd0 con 0x55d5063d5150 2024-09-16T01:42:28.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.687+0000 7f353b165640 1 -- 172.21.15.88:0/4095020808 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v108'106 uv106 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f352c00b7f0 con 0x55d5063d5150 2024-09-16T01:42:28.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.687+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d7fd0 con 0x55d50624d510 2024-09-16T01:42:28.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.688+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v108'123 uv122 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.688+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d7fd0 con 0x55d50624d510 2024-09-16T01:42:28.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.689+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v108'124 uv124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.689+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d7fd0 con 0x55d50624d510 2024-09-16T01:42:28.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.690+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v108'125 uv125 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.691 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:42:28.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.690+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d8fe0 con 0x55d50624d510 2024-09-16T01:42:28.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.691+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v108'126 uv126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.691+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d8fe0 con 0x55d50624d510 2024-09-16T01:42:28.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.691+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v108'127 uv127 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.691+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d7fd0 con 0x55d50624d510 2024-09-16T01:42:28.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.692+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v108'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d50624d510 2024-09-16T01:42:28.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.692+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d7fd0 con 0x55d5063cd890 2024-09-16T01:42:28.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.693+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v108'203 uv202 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f353006b070 con 0x55d5063cd890 2024-09-16T01:42:28.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.693+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d8370 con 0x55d5063cd890 2024-09-16T01:42:28.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.694+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v108'204 uv204 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f353006b070 con 0x55d5063cd890 2024-09-16T01:42:28.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.694+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d8760 con 0x55d5063cd890 2024-09-16T01:42:28.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.695+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v108'205 uv205 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f353006b070 con 0x55d5063cd890 2024-09-16T01:42:28.697 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:42:28.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.695+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d10e0 con 0x55d5063cd890 2024-09-16T01:42:28.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.697+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v108'206 uv206 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f353006b070 con 0x55d5063cd890 2024-09-16T01:42:28.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.697+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d10e0 con 0x55d5063cd890 2024-09-16T01:42:28.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.698+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v108'207 uv207 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f353006b070 con 0x55d5063cd890 2024-09-16T01:42:28.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.698+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d10e0 con 0x55d5063cd890 2024-09-16T01:42:28.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.699+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v108'208 uv201 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f353006b070 con 0x55d5063cd890 2024-09-16T01:42:28.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.699+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d1120 con 0x55d5063cd890 2024-09-16T01:42:28.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.699+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v108'209 uv209 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f353006b070 con 0x55d5063cd890 2024-09-16T01:42:28.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.699+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d10e0 con 0x55d5063cd890 2024-09-16T01:42:28.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.700+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v108'210 uv210 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f353006b070 con 0x55d5063cd890 2024-09-16T01:42:28.702 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:42:28.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.700+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d10e0 con 0x55d5063cd890 2024-09-16T01:42:28.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.701+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v108'211 uv211 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f353006b070 con 0x55d5063cd890 2024-09-16T01:42:28.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.701+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063d8740 con 0x55d5063cd890 2024-09-16T01:42:28.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.702+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v108'212 uv212 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f353006b070 con 0x55d5063cd890 2024-09-16T01:42:28.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.702+0000 7f353e11df40 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55d5063d8bc0 0x55d5063db940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.702+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dbe80 con 0x55d5063d8bc0 2024-09-16T01:42:28.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.702+0000 7f353a964640 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55d5063d8bc0 0x55d5063db940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.703+0000 7f353a964640 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55d5063d8bc0 0x55d5063db940 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:28.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.704+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v108'91 uv90 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f353006b070 con 0x55d5063d8bc0 2024-09-16T01:42:28.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.704+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dbe80 con 0x55d5063d8bc0 2024-09-16T01:42:28.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.705+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v108'92 uv92 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f353006b070 con 0x55d5063d8bc0 2024-09-16T01:42:28.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.705+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dbe80 con 0x55d5063d8bc0 2024-09-16T01:42:28.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.705+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v108'93 uv93 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f353006b070 con 0x55d5063d8bc0 2024-09-16T01:42:28.707 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:42:28.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.705+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dbe80 con 0x55d5063d8bc0 2024-09-16T01:42:28.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.706+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v108'94 uv94 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f353006b070 con 0x55d5063d8bc0 2024-09-16T01:42:28.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.706+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dbe80 con 0x55d5063cd890 2024-09-16T01:42:28.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.708+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v108'123 uv122 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f353006b070 con 0x55d5063cd890 2024-09-16T01:42:28.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.708+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dbe80 con 0x55d5063cd890 2024-09-16T01:42:28.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.709+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v108'124 uv124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f353006b070 con 0x55d5063cd890 2024-09-16T01:42:28.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.709+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dbe80 con 0x55d5063cd890 2024-09-16T01:42:28.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.710+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v108'125 uv125 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f353006b070 con 0x55d5063cd890 2024-09-16T01:42:28.711 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:42:28.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.710+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dce90 con 0x55d5063cd890 2024-09-16T01:42:28.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.711+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v108'126 uv126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f353006b070 con 0x55d5063cd890 2024-09-16T01:42:28.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.711+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dce90 con 0x55d5063cd890 2024-09-16T01:42:28.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.711+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v108'127 uv127 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f353006b070 con 0x55d5063cd890 2024-09-16T01:42:28.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.711+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dbe80 con 0x55d5063cd890 2024-09-16T01:42:28.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.712+0000 7f353a964640 1 -- 172.21.15.88:0/4095020808 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v108'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f353006b070 con 0x55d5063cd890 2024-09-16T01:42:28.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.712+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dbe80 con 0x55d5063d1d40 2024-09-16T01:42:28.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.713+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v108'104 uv103 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.713+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dbe80 con 0x55d5063d1d40 2024-09-16T01:42:28.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.715+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v108'105 uv105 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.715+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dbe80 con 0x55d5063d1d40 2024-09-16T01:42:28.717 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:42:28.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.716+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v108'106 uv106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.716+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dbe80 con 0x55d5063d1d40 2024-09-16T01:42:28.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.717+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v108'107 uv107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.717+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dbe80 con 0x55d5063d1d40 2024-09-16T01:42:28.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.718+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v108'108 uv108 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.718+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dbe80 con 0x55d5063d1d40 2024-09-16T01:42:28.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.720+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v108'253 uv245 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.720+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dbe80 con 0x55d5063d1d40 2024-09-16T01:42:28.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.721+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v108'254 uv254 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.721+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dbe80 con 0x55d5063d1d40 2024-09-16T01:42:28.723 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:42:28.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.722+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v108'255 uv255 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.722+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dbe80 con 0x55d5063d1d40 2024-09-16T01:42:28.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.723+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v108'256 uv256 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.723+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dbe80 con 0x55d5063d1d40 2024-09-16T01:42:28.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.724+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v108'257 uv257 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.724+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dbe80 con 0x55d5063d1d40 2024-09-16T01:42:28.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.725+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v108'258 uv252 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.725+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dbe80 con 0x55d5063d1d40 2024-09-16T01:42:28.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.726+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v108'259 uv259 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.726+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dbe80 con 0x55d5063d1d40 2024-09-16T01:42:28.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.727+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v108'260 uv260 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.729 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:42:28.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.727+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dce90 con 0x55d5063d1d40 2024-09-16T01:42:28.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.728+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v108'261 uv261 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.728+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dce90 con 0x55d5063d1d40 2024-09-16T01:42:28.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.729+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v108'262 uv262 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.729+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dbe80 con 0x55d5063d1d40 2024-09-16T01:42:28.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.730+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v108'263 uv263 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.730+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x55d5063dc330 con 0x55d5063d1d40 2024-09-16T01:42:28.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.731+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v108'264 uv264 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f3534011040 con 0x55d5063d1d40 2024-09-16T01:42:28.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.731+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55d50624d510 msgr2=0x55d5063be360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:28.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.731+0000 7f353e11df40 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55d50624d510 0x55d5063be360 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.731+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55d5063cd890 msgr2=0x55d5063d0370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:28.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.731+0000 7f353e11df40 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55d5063cd890 0x55d5063d0370 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.731+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55d5063d8bc0 msgr2=0x55d5063db940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:28.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.731+0000 7f353e11df40 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55d5063d8bc0 0x55d5063db940 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.731+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55d5063c81f0 msgr2=0x55d5063caf00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:28.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.731+0000 7f353e11df40 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55d5063c81f0 0x55d5063caf00 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.731+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55d5063d1d40 msgr2=0x55d5063d4130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:28.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.731+0000 7f353b966640 1 -- 172.21.15.88:0/4095020808 reap_dead start 2024-09-16T01:42:28.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.731+0000 7f353e11df40 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55d5063d1d40 0x55d5063d4130 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.731+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55d5063d5150 msgr2=0x55d5063d7580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:28.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.731+0000 7f353e11df40 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55d5063d5150 0x55d5063d7580 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.732+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f352806ac90 msgr2=0x7f352806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.732+0000 7f353e11df40 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f352806ac90 0x7f352806d150 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f352c00d6e0 tx=0x7f352c00f040 comp rx=0 tx=0).stop 2024-09-16T01:42:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.732+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d50624e710 msgr2=0x55d50624cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.732+0000 7f353e11df40 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d50624e710 0x55d50624cfd0 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f3530025dc0 tx=0x7f3530002fb0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.732+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 shutdown_connections 2024-09-16T01:42:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.732+0000 7f353e11df40 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f352806ac90 0x7f352806d150 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.732+0000 7f353e11df40 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d50624e710 0x55d50624cfd0 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.732+0000 7f353e11df40 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55d5063d5150 0x55d5063d7580 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.732+0000 7f353e11df40 1 --2- 172.21.15.88:0/4095020808 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55d5063d1d40 0x55d5063d4130 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.732+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 >> 172.21.15.88:0/4095020808 conn(0x55d506277960 msgr2=0x55d50624c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:28.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.732+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 shutdown_connections 2024-09-16T01:42:28.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.732+0000 7f353e11df40 1 -- 172.21.15.88:0/4095020808 wait complete. 2024-09-16T01:42:28.738 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put 400.00000000 - 2024-09-16T01:42:28.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.821+0000 7feb1d886f40 1 Processor -- start 2024-09-16T01:42:28.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.821+0000 7feb1d886f40 1 -- start start 2024-09-16T01:42:28.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.821+0000 7feb1d886f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d0fcaba90 0x559d0fd90550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.821+0000 7feb1d886f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559d0fcc1fb0 0x559d0fd90a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.821+0000 7feb1d886f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d0fd94e40 0x559d0fd8b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.821+0000 7feb1d886f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x559d0fc12510 con 0x559d0fd94e40 2024-09-16T01:42:28.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.821+0000 7feb1d886f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559d0fd95220 con 0x559d0fcaba90 2024-09-16T01:42:28.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.821+0000 7feb1d886f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x559d0fd98690 con 0x559d0fcc1fb0 2024-09-16T01:42:28.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.822+0000 7feb1bb7d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d0fd94e40 0x559d0fd8b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.822+0000 7feb1bb7d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d0fd94e40 0x559d0fd8b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49432/0 (socket says 172.21.15.88:49432) 2024-09-16T01:42:28.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.822+0000 7feb1bb7d640 1 -- 172.21.15.88:0/1277897458 learned_addr learned my addr 172.21.15.88:0/1277897458 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:28.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.822+0000 7feb1b37c640 1 --2- 172.21.15.88:0/1277897458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d0fcaba90 0x559d0fd90550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.822+0000 7feb13fff640 1 --2- 172.21.15.88:0/1277897458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559d0fcc1fb0 0x559d0fd90a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.822+0000 7feb1bb7d640 1 -- 172.21.15.88:0/1277897458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559d0fcc1fb0 msgr2=0x559d0fd90a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:28.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.822+0000 7feb1bb7d640 1 --2- 172.21.15.88:0/1277897458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559d0fcc1fb0 0x559d0fd90a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.822+0000 7feb1bb7d640 1 -- 172.21.15.88:0/1277897458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d0fcaba90 msgr2=0x559d0fd90550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:28.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.822+0000 7feb1bb7d640 1 --2- 172.21.15.88:0/1277897458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d0fcaba90 0x559d0fd90550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.822+0000 7feb1bb7d640 1 -- 172.21.15.88:0/1277897458 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d0fd8e830 con 0x559d0fd94e40 2024-09-16T01:42:28.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.822+0000 7feb1b37c640 1 --2- 172.21.15.88:0/1277897458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d0fcaba90 0x559d0fd90550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:28.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.822+0000 7feb1bb7d640 1 --2- 172.21.15.88:0/1277897458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d0fd94e40 0x559d0fd8b9b0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7feb14025950 tx=0x7feb14051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=841c45c927e45a28 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:28.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.822+0000 7feb1a37a640 1 -- 172.21.15.88:0/1277897458 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb14069070 con 0x559d0fd94e40 2024-09-16T01:42:28.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.822+0000 7feb1a37a640 1 -- 172.21.15.88:0/1277897458 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb14051860 con 0x559d0fd94e40 2024-09-16T01:42:28.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.822+0000 7feb1a37a640 1 -- 172.21.15.88:0/1277897458 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb14055040 con 0x559d0fd94e40 2024-09-16T01:42:28.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.823+0000 7feb1d886f40 1 -- 172.21.15.88:0/1277897458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d0fd94e40 msgr2=0x559d0fd8b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:28.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.823+0000 7feb1d886f40 1 --2- 172.21.15.88:0/1277897458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d0fd94e40 0x559d0fd8b9b0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7feb14025950 tx=0x7feb14051000 comp rx=0 tx=0).stop 2024-09-16T01:42:28.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.823+0000 7feb1d886f40 1 -- 172.21.15.88:0/1277897458 shutdown_connections 2024-09-16T01:42:28.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.823+0000 7feb1d886f40 1 --2- 172.21.15.88:0/1277897458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d0fd94e40 0x559d0fd8b9b0 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.823+0000 7feb1d886f40 1 --2- 172.21.15.88:0/1277897458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559d0fcc1fb0 0x559d0fd90a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.823+0000 7feb1d886f40 1 --2- 172.21.15.88:0/1277897458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d0fcaba90 0x559d0fd90550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.823+0000 7feb1d886f40 1 -- 172.21.15.88:0/1277897458 >> 172.21.15.88:0/1277897458 conn(0x559d0fceb960 msgr2=0x559d0fcef760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:28.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.823+0000 7feb1d886f40 1 -- 172.21.15.88:0/1277897458 shutdown_connections 2024-09-16T01:42:28.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.823+0000 7feb1d886f40 1 -- 172.21.15.88:0/1277897458 wait complete. 2024-09-16T01:42:28.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.823+0000 7feb1d886f40 1 Processor -- start 2024-09-16T01:42:28.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.823+0000 7feb1d886f40 1 -- start start 2024-09-16T01:42:28.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.824+0000 7feb1d886f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d0fcaba90 0x559d0fd8db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.824+0000 7feb1d886f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d0fcc1fb0 0x559d0fd8a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.824+0000 7feb1d886f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559d0fd94e40 0x559d0fda0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.824+0000 7feb1d886f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x559d0fd96f50 con 0x559d0fcaba90 2024-09-16T01:42:28.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.824+0000 7feb1d886f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x559d0fd98c50 con 0x559d0fcc1fb0 2024-09-16T01:42:28.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.824+0000 7feb1d886f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x559d0fd97230 con 0x559d0fd94e40 2024-09-16T01:42:28.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.824+0000 7feb1b37c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d0fcaba90 0x559d0fd8db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.824+0000 7feb1b37c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d0fcaba90 0x559d0fd8db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49438/0 (socket says 172.21.15.88:49438) 2024-09-16T01:42:28.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.824+0000 7feb1b37c640 1 -- 172.21.15.88:0/1468569680 learned_addr learned my addr 172.21.15.88:0/1468569680 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:28.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.824+0000 7feb1bb7d640 1 --2- 172.21.15.88:0/1468569680 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559d0fd94e40 0x559d0fda0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.824+0000 7feb13fff640 1 --2- 172.21.15.88:0/1468569680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d0fcc1fb0 0x559d0fd8a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.824+0000 7feb1b37c640 1 -- 172.21.15.88:0/1468569680 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559d0fd94e40 msgr2=0x559d0fda0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:28.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.824+0000 7feb1b37c640 1 --2- 172.21.15.88:0/1468569680 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x559d0fd94e40 0x559d0fda0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.824+0000 7feb1b37c640 1 -- 172.21.15.88:0/1468569680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d0fcc1fb0 msgr2=0x559d0fd8a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:28.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.824+0000 7feb1b37c640 1 --2- 172.21.15.88:0/1468569680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d0fcc1fb0 0x559d0fd8a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.824+0000 7feb1b37c640 1 -- 172.21.15.88:0/1468569680 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d0fda1440 con 0x559d0fcaba90 2024-09-16T01:42:28.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.825+0000 7feb1b37c640 1 --2- 172.21.15.88:0/1468569680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d0fcaba90 0x559d0fd8db20 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7feb0c00be30 tx=0x7feb0c00bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:28.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.825+0000 7feb19378640 1 -- 172.21.15.88:0/1468569680 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c019070 con 0x559d0fcaba90 2024-09-16T01:42:28.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.825+0000 7feb19378640 1 -- 172.21.15.88:0/1468569680 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb0c007af0 con 0x559d0fcaba90 2024-09-16T01:42:28.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.825+0000 7feb1d886f40 1 -- 172.21.15.88:0/1468569680 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x559d0fda1720 con 0x559d0fcaba90 2024-09-16T01:42:28.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.825+0000 7feb19378640 1 -- 172.21.15.88:0/1468569680 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c014410 con 0x559d0fcaba90 2024-09-16T01:42:28.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.825+0000 7feb1d886f40 1 -- 172.21.15.88:0/1468569680 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d0fda1a00 con 0x559d0fcaba90 2024-09-16T01:42:28.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.826+0000 7feb19378640 1 -- 172.21.15.88:0/1468569680 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7feb0c01c070 con 0x559d0fcaba90 2024-09-16T01:42:28.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.826+0000 7feb19378640 1 --2- 172.21.15.88:0/1468569680 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7feafc06ac90 0x7feafc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.827+0000 7feb13fff640 1 --2- 172.21.15.88:0/1468569680 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7feafc06ac90 0x7feafc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.827+0000 7feb19378640 1 -- 172.21.15.88:0/1468569680 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 7544+0+0 (secure 0 0 0) 0x7feb0c089710 con 0x559d0fcaba90 2024-09-16T01:42:28.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.827+0000 7feb1d886f40 1 --2- 172.21.15.88:0/1468569680 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x559d0fd98170 0x559d0fd89910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.827+0000 7feb1d886f40 1 -- 172.21.15.88:0/1468569680 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:1 12.f 12:f1f69726:::400.00000000:head [writefull 0~80 in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108) -- 0x559d0fd078b0 con 0x559d0fd98170 2024-09-16T01:42:28.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.827+0000 7feb13fff640 1 --2- 172.21.15.88:0/1468569680 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7feafc06ac90 0x7feafc06d150 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7feb04000c00 tx=0x7feb040017f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:28.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.827+0000 7feb1bb7d640 1 --2- 172.21.15.88:0/1468569680 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x559d0fd98170 0x559d0fd89910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.827+0000 7feb1bb7d640 1 --2- 172.21.15.88:0/1468569680 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x559d0fd98170 0x559d0fd89910 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:28.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.829+0000 7feb1bb7d640 1 -- 172.21.15.88:0/1468569680 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(1 400.00000000 [writefull 0~80] v108'109 uv109 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7feb1406d040 con 0x559d0fd98170 2024-09-16T01:42:28.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.829+0000 7feb1d886f40 1 -- 172.21.15.88:0/1468569680 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x559d0fd98170 msgr2=0x559d0fd89910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:28.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.829+0000 7feb1d886f40 1 --2- 172.21.15.88:0/1468569680 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x559d0fd98170 0x559d0fd89910 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.829+0000 7feb1d886f40 1 -- 172.21.15.88:0/1468569680 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7feafc06ac90 msgr2=0x7feafc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:28.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.829+0000 7feb1d886f40 1 --2- 172.21.15.88:0/1468569680 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7feafc06ac90 0x7feafc06d150 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7feb04000c00 tx=0x7feb040017f0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.829+0000 7feb1d886f40 1 -- 172.21.15.88:0/1468569680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d0fcaba90 msgr2=0x559d0fd8db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:28.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.829+0000 7feb1d886f40 1 --2- 172.21.15.88:0/1468569680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d0fcaba90 0x559d0fd8db20 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7feb0c00be30 tx=0x7feb0c00bf20 comp rx=0 tx=0).stop 2024-09-16T01:42:28.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.829+0000 7feb1bb7d640 1 -- 172.21.15.88:0/1468569680 reap_dead start 2024-09-16T01:42:28.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.829+0000 7feb1d886f40 1 -- 172.21.15.88:0/1468569680 shutdown_connections 2024-09-16T01:42:28.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.829+0000 7feb1d886f40 1 -- 172.21.15.88:0/1468569680 >> 172.21.15.88:0/1468569680 conn(0x559d0fceb960 msgr2=0x559d0fd904a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:28.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.829+0000 7feb1d886f40 1 -- 172.21.15.88:0/1468569680 shutdown_connections 2024-09-16T01:42:28.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.829+0000 7feb1d886f40 1 -- 172.21.15.88:0/1468569680 wait complete. 2024-09-16T01:42:28.835 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:42:28.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.975+0000 7fbc88f2c640 1 Processor -- start 2024-09-16T01:42:28.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.976+0000 7fbc88f2c640 1 -- start start 2024-09-16T01:42:28.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.976+0000 7fbc88f2c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbc8415e7a0 0x7fbc8406caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.976+0000 7fbc88f2c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbc840cf1f0 0x7fbc840b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.976+0000 7fbc88f2c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc840b6720 0x7fbc841623c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.976+0000 7fbc88f2c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fbc840d2ca0 con 0x7fbc8415e7a0 2024-09-16T01:42:28.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.976+0000 7fbc88f2c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbc840cf860 con 0x7fbc840cf1f0 2024-09-16T01:42:28.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.976+0000 7fbc88f2c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbc840cfb40 con 0x7fbc840b6720 2024-09-16T01:42:28.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.977+0000 7fbc8259c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbc8415e7a0 0x7fbc8406caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.977+0000 7fbc8259c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbc8415e7a0 0x7fbc8406caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49442/0 (socket says 172.21.15.88:49442) 2024-09-16T01:42:28.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.977+0000 7fbc8259c640 1 -- 172.21.15.88:0/240041504 learned_addr learned my addr 172.21.15.88:0/240041504 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:28.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.977+0000 7fbc81d9b640 1 --2- 172.21.15.88:0/240041504 >> v2:172.21.15.154:3300/0 conn(0x7fbc840cf1f0 0x7fbc840b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.977+0000 7fbc82d9d640 1 --2- 172.21.15.88:0/240041504 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc840b6720 0x7fbc841623c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.977+0000 7fbc8259c640 1 -- 172.21.15.88:0/240041504 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc840b6720 msgr2=0x7fbc841623c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:28.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.977+0000 7fbc8259c640 1 --2- 172.21.15.88:0/240041504 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc840b6720 0x7fbc841623c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.977+0000 7fbc8259c640 1 -- 172.21.15.88:0/240041504 >> v2:172.21.15.154:3300/0 conn(0x7fbc840cf1f0 msgr2=0x7fbc840b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:28.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.977+0000 7fbc8259c640 1 --2- 172.21.15.88:0/240041504 >> v2:172.21.15.154:3300/0 conn(0x7fbc840cf1f0 0x7fbc840b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.977+0000 7fbc8259c640 1 -- 172.21.15.88:0/240041504 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc840cfe20 con 0x7fbc8415e7a0 2024-09-16T01:42:28.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.978+0000 7fbc8259c640 1 --2- 172.21.15.88:0/240041504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc8415e7a0 0x7fbc8406caa0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fbc840d2110 tx=0x7fbc7402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fca54b8d9d8e364 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:28.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.978+0000 7fbc8159a640 1 -- 172.21.15.88:0/240041504 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc74037070 con 0x7fbc8415e7a0 2024-09-16T01:42:28.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.978+0000 7fbc8159a640 1 -- 172.21.15.88:0/240041504 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc74032440 con 0x7fbc8415e7a0 2024-09-16T01:42:28.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.978+0000 7fbc8159a640 1 -- 172.21.15.88:0/240041504 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc74032810 con 0x7fbc8415e7a0 2024-09-16T01:42:28.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.978+0000 7fbc88f2c640 1 -- 172.21.15.88:0/240041504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc8415e7a0 msgr2=0x7fbc8406caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:28.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.978+0000 7fbc88f2c640 1 --2- 172.21.15.88:0/240041504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc8415e7a0 0x7fbc8406caa0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fbc840d2110 tx=0x7fbc7402a140 comp rx=0 tx=0).stop 2024-09-16T01:42:28.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.978+0000 7fbc88f2c640 1 -- 172.21.15.88:0/240041504 shutdown_connections 2024-09-16T01:42:28.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.978+0000 7fbc88f2c640 1 --2- 172.21.15.88:0/240041504 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc840b6720 0x7fbc841623c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.978+0000 7fbc88f2c640 1 --2- 172.21.15.88:0/240041504 >> v2:172.21.15.154:3300/0 conn(0x7fbc840cf1f0 0x7fbc840b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.978+0000 7fbc88f2c640 1 --2- 172.21.15.88:0/240041504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc8415e7a0 0x7fbc8406caa0 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.978+0000 7fbc88f2c640 1 -- 172.21.15.88:0/240041504 >> 172.21.15.88:0/240041504 conn(0x7fbc840d11e0 msgr2=0x7fbc840d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:28.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.978+0000 7fbc88f2c640 1 -- 172.21.15.88:0/240041504 shutdown_connections 2024-09-16T01:42:28.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.978+0000 7fbc88f2c640 1 -- 172.21.15.88:0/240041504 wait complete. 2024-09-16T01:42:28.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.980+0000 7fbc88f2c640 1 Processor -- start 2024-09-16T01:42:28.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.980+0000 7fbc88f2c640 1 -- start start 2024-09-16T01:42:28.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.980+0000 7fbc88f2c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc840b6720 0x7fbc8414e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.980+0000 7fbc88f2c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc840cf1f0 0x7fbc840cdda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.980+0000 7fbc88f2c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc8415e7a0 0x7fbc840ce2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.980+0000 7fbc88f2c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbc8406cfe0 con 0x7fbc8415e7a0 2024-09-16T01:42:28.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.980+0000 7fbc88f2c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbc840b7a00 con 0x7fbc840cf1f0 2024-09-16T01:42:28.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.980+0000 7fbc88f2c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbc8415eb80 con 0x7fbc840b6720 2024-09-16T01:42:28.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.981+0000 7fbc82d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc8415e7a0 0x7fbc840ce2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.981+0000 7fbc82d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc8415e7a0 0x7fbc840ce2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49456/0 (socket says 172.21.15.88:49456) 2024-09-16T01:42:28.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.981+0000 7fbc82d9d640 1 -- 172.21.15.88:0/3084792456 learned_addr learned my addr 172.21.15.88:0/3084792456 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:28.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.981+0000 7fbc81d9b640 1 --2- 172.21.15.88:0/3084792456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc840cf1f0 0x7fbc840cdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.981+0000 7fbc8259c640 1 --2- 172.21.15.88:0/3084792456 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc840b6720 0x7fbc8414e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.981+0000 7fbc82d9d640 1 -- 172.21.15.88:0/3084792456 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc840b6720 msgr2=0x7fbc8414e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:28.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.981+0000 7fbc82d9d640 1 --2- 172.21.15.88:0/3084792456 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc840b6720 0x7fbc8414e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.981+0000 7fbc82d9d640 1 -- 172.21.15.88:0/3084792456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc840cf1f0 msgr2=0x7fbc840cdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:28.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.981+0000 7fbc82d9d640 1 --2- 172.21.15.88:0/3084792456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc840cf1f0 0x7fbc840cdda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:28.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.981+0000 7fbc82d9d640 1 -- 172.21.15.88:0/3084792456 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc8414ed90 con 0x7fbc8415e7a0 2024-09-16T01:42:28.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.981+0000 7fbc82d9d640 1 --2- 172.21.15.88:0/3084792456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc8415e7a0 0x7fbc840ce2e0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c007d20 tx=0x7fbc7c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:28.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.981+0000 7fbc7affd640 1 -- 172.21.15.88:0/3084792456 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c014420 con 0x7fbc8415e7a0 2024-09-16T01:42:28.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.981+0000 7fbc7affd640 1 -- 172.21.15.88:0/3084792456 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc7c019070 con 0x7fbc8415e7a0 2024-09-16T01:42:28.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.981+0000 7fbc88f2c640 1 -- 172.21.15.88:0/3084792456 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc8414f000 con 0x7fbc8415e7a0 2024-09-16T01:42:28.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.981+0000 7fbc7affd640 1 -- 172.21.15.88:0/3084792456 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c014820 con 0x7fbc8415e7a0 2024-09-16T01:42:28.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.981+0000 7fbc88f2c640 1 -- 172.21.15.88:0/3084792456 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc8414f5c0 con 0x7fbc8415e7a0 2024-09-16T01:42:28.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.982+0000 7fbc88f2c640 1 -- 172.21.15.88:0/3084792456 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc840b61e0 con 0x7fbc8415e7a0 2024-09-16T01:42:28.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.983+0000 7fbc7affd640 1 -- 172.21.15.88:0/3084792456 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fbc7c0149c0 con 0x7fbc8415e7a0 2024-09-16T01:42:28.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.983+0000 7fbc7affd640 1 --2- 172.21.15.88:0/3084792456 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc6406c2e0 0x7fbc6406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:28.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.983+0000 7fbc7affd640 1 -- 172.21.15.88:0/3084792456 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 7544+0+0 (secure 0 0 0) 0x7fbc7c08c2f0 con 0x7fbc8415e7a0 2024-09-16T01:42:28.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.983+0000 7fbc8259c640 1 --2- 172.21.15.88:0/3084792456 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc6406c2e0 0x7fbc6406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:28.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.984+0000 7fbc8259c640 1 --2- 172.21.15.88:0/3084792456 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc6406c2e0 0x7fbc6406e7a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fbc74034000 tx=0x7fbc7402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:28.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:28.987+0000 7fbc7affd640 1 -- 172.21.15.88:0/3084792456 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fbc7c011260 con 0x7fbc8415e7a0 2024-09-16T01:42:29.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.189+0000 7fbc88f2c640 1 -- 172.21.15.88:0/3084792456 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fbc840d2a80 con 0x7fbc8415e7a0 2024-09-16T01:42:29.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.530+0000 7fbc7affd640 1 -- 172.21.15.88:0/3084792456 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v197) ==== 162+0+0 (secure 0 0 0) 0x7fbc7c05a330 con 0x7fbc8415e7a0 2024-09-16T01:42:29.531 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:42:29.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.532+0000 7fbc88f2c640 1 -- 172.21.15.88:0/3084792456 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc6406c2e0 msgr2=0x7fbc6406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:29.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.532+0000 7fbc88f2c640 1 --2- 172.21.15.88:0/3084792456 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc6406c2e0 0x7fbc6406e7a0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fbc74034000 tx=0x7fbc7402e000 comp rx=0 tx=0).stop 2024-09-16T01:42:29.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.532+0000 7fbc88f2c640 1 -- 172.21.15.88:0/3084792456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc8415e7a0 msgr2=0x7fbc840ce2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:29.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.532+0000 7fbc88f2c640 1 --2- 172.21.15.88:0/3084792456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc8415e7a0 0x7fbc840ce2e0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c007d20 tx=0x7fbc7c00e4b0 comp rx=0 tx=0).stop 2024-09-16T01:42:29.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.532+0000 7fbc88f2c640 1 -- 172.21.15.88:0/3084792456 shutdown_connections 2024-09-16T01:42:29.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.532+0000 7fbc88f2c640 1 --2- 172.21.15.88:0/3084792456 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbc6406c2e0 0x7fbc6406e7a0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:29.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.532+0000 7fbc88f2c640 1 --2- 172.21.15.88:0/3084792456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc8415e7a0 0x7fbc840ce2e0 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:29.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.532+0000 7fbc88f2c640 1 --2- 172.21.15.88:0/3084792456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc840cf1f0 0x7fbc840cdda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:29.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.532+0000 7fbc88f2c640 1 --2- 172.21.15.88:0/3084792456 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc840b6720 0x7fbc8414e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:29.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.532+0000 7fbc88f2c640 1 -- 172.21.15.88:0/3084792456 >> 172.21.15.88:0/3084792456 conn(0x7fbc840d11e0 msgr2=0x7fbc840cea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:29.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.532+0000 7fbc88f2c640 1 -- 172.21.15.88:0/3084792456 shutdown_connections 2024-09-16T01:42:29.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.532+0000 7fbc88f2c640 1 -- 172.21.15.88:0/3084792456 wait complete. 2024-09-16T01:42:29.541 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:42:29.540+0000 7f9385717640 -1 log_channel(cluster) log [ERR] : failed to read JournalPointer: -22 ((22) Invalid argument) 2024-09-16T01:42:29.541 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:42:29.540+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:42:29.541 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:42:29.540+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:42:29.546 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:29.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.643+0000 7fdb08cae640 1 Processor -- start 2024-09-16T01:42:29.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.643+0000 7fdb08cae640 1 -- start start 2024-09-16T01:42:29.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.644+0000 7fdb08cae640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdb040cfc80 0x7fdb040b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:29.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.644+0000 7fdb08cae640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb0415c740 con 0x7fdb0415c330 2024-09-16T01:42:29.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.644+0000 7fdb08cae640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb040b50a0 con 0x7fdb04155800 2024-09-16T01:42:29.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.644+0000 7fdb08cae640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fdb040b5380 con 0x7fdb040cfc80 2024-09-16T01:42:29.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.644+0000 7fdb02d9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fdb0415c330 0x7fdb0415c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43396/0 (socket says 172.21.15.88:43396) 2024-09-16T01:42:29.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.644+0000 7fdb02d9d640 1 -- 172.21.15.88:0/3480637431 learned_addr learned my addr 172.21.15.88:0/3480637431 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:29.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.644+0000 7fdb01d9b640 1 --2- 172.21.15.88:0/3480637431 >> v2:172.21.15.154:3300/0 conn(0x7fdb040cfc80 0x7fdb040b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:29.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.644+0000 7fdb0159a640 1 -- 172.21.15.88:0/3480637431 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1626732325 0 0) 0x7fdb0415c740 con 0x7fdb0415c330 2024-09-16T01:42:29.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.644+0000 7fdb0159a640 1 -- 172.21.15.88:0/3480637431 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdaf0003520 con 0x7fdb0415c330 2024-09-16T01:42:29.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.645+0000 7fdb0159a640 1 -- 172.21.15.88:0/3480637431 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3127293005 0 0) 0x7fdaf0003520 con 0x7fdb0415c330 2024-09-16T01:42:29.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.645+0000 7fdb0159a640 1 -- 172.21.15.88:0/3480637431 >> v2:172.21.15.154:3300/0 conn(0x7fdb040cfc80 msgr2=0x7fdb040b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:29.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.645+0000 7fdb0159a640 1 --2- 172.21.15.88:0/3480637431 >> v2:172.21.15.154:3300/0 conn(0x7fdb040cfc80 0x7fdb040b6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:29.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.645+0000 7fdb0159a640 1 -- 172.21.15.88:0/3480637431 >> v1:172.21.15.154:6789/0 conn(0x7fdb04155800 legacy=0x7fdb040cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:29.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.645+0000 7fdb0159a640 1 -- 172.21.15.88:0/3480637431 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb040b5660 con 0x7fdb0415c330 2024-09-16T01:42:29.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.645+0000 7fdb0159a640 1 -- 172.21.15.88:0/3480637431 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fdaf4001670 con 0x7fdb0415c330 2024-09-16T01:42:29.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.645+0000 7fdb0159a640 1 -- 172.21.15.88:0/3480637431 >> v1:172.21.15.88:6789/0 conn(0x7fdb0415c330 legacy=0x7fdb0415c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:29.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.645+0000 7fdb0159a640 1 --2- 172.21.15.88:0/3480637431 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdaf00039b0 0x7fdaf0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:29.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.645+0000 7fdb0159a640 1 --2- 172.21.15.88:0/3480637431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdaf0004800 0x7fdaf0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:29.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.645+0000 7fdb0159a640 1 --2- 172.21.15.88:0/3480637431 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdaf00051c0 0x7fdaf000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:29.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.645+0000 7fdb0159a640 1 -- 172.21.15.88:0/3480637431 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdb040b5380 con 0x7fdaf00051c0 2024-09-16T01:42:29.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.645+0000 7fdb0159a640 1 -- 172.21.15.88:0/3480637431 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf000c060 con 0x7fdaf0004800 2024-09-16T01:42:29.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.645+0000 7fdb0159a640 1 -- 172.21.15.88:0/3480637431 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdaf000c260 con 0x7fdaf00039b0 2024-09-16T01:42:29.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.645+0000 7fdb02d9d640 1 --2- 172.21.15.88:0/3480637431 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdaf00051c0 0x7fdaf000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:29.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.645+0000 7fdb01d9b640 1 --2- 172.21.15.88:0/3480637431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdaf0004800 0x7fdaf0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:29.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.645+0000 7fdb0259c640 1 --2- 172.21.15.88:0/3480637431 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdaf00039b0 0x7fdaf0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:29.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.645+0000 7fdb02d9d640 1 -- 172.21.15.88:0/3480637431 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdaf00039b0 msgr2=0x7fdaf0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:29.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.645+0000 7fdb02d9d640 1 --2- 172.21.15.88:0/3480637431 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdaf00039b0 0x7fdaf0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:29.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.645+0000 7fdb02d9d640 1 -- 172.21.15.88:0/3480637431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdaf0004800 msgr2=0x7fdaf0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:29.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.645+0000 7fdb02d9d640 1 --2- 172.21.15.88:0/3480637431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdaf0004800 0x7fdaf0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:29.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.645+0000 7fdb02d9d640 1 -- 172.21.15.88:0/3480637431 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaf0003520 con 0x7fdaf00051c0 2024-09-16T01:42:29.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.646+0000 7fdb02d9d640 1 -- 172.21.15.88:0/3480637431 reap_dead start 2024-09-16T01:42:29.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.646+0000 7fdb02d9d640 1 --2- 172.21.15.88:0/3480637431 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdaf00051c0 0x7fdaf000b9c0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fdaf400c240 tx=0x7fdaf0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a667b0d9297ae19b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.646+0000 7fdb0159a640 1 -- 172.21.15.88:0/3480637431 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf4007cf0 con 0x7fdaf00051c0 2024-09-16T01:42:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.646+0000 7fdb0159a640 1 -- 172.21.15.88:0/3480637431 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaf40058d0 con 0x7fdaf00051c0 2024-09-16T01:42:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.646+0000 7fdb0159a640 1 -- 172.21.15.88:0/3480637431 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf4005ce0 con 0x7fdaf00051c0 2024-09-16T01:42:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.646+0000 7fdb08cae640 1 -- 172.21.15.88:0/3480637431 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdaf00051c0 msgr2=0x7fdaf000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.646+0000 7fdb08cae640 1 --2- 172.21.15.88:0/3480637431 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdaf00051c0 0x7fdaf000b9c0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fdaf400c240 tx=0x7fdaf0005830 comp rx=0 tx=0).stop 2024-09-16T01:42:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.646+0000 7fdb08cae640 1 -- 172.21.15.88:0/3480637431 shutdown_connections 2024-09-16T01:42:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.646+0000 7fdb08cae640 1 --2- 172.21.15.88:0/3480637431 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdaf00051c0 0x7fdaf000b9c0 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.646+0000 7fdb08cae640 1 -- 172.21.15.88:0/3480637431 >> 172.21.15.88:0/3480637431 conn(0x7fdb0406bc10 msgr2=0x7fdb0406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.647+0000 7fdb08cae640 1 -- 172.21.15.88:0/3480637431 shutdown_connections 2024-09-16T01:42:29.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.647+0000 7fdb08cae640 1 -- 172.21.15.88:0/3480637431 wait complete. 2024-09-16T01:42:29.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.648+0000 7fdb08cae640 1 Processor -- start 2024-09-16T01:42:29.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.648+0000 7fdb08cae640 1 -- start start 2024-09-16T01:42:29.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.649+0000 7fdb08cae640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdaf00051c0 0x7fdb04156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:29.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.649+0000 7fdb08cae640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb0413fc30 0x7fdb04153d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:29.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.649+0000 7fdb08cae640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb041400b0 0x7fdb04154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:29.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.649+0000 7fdb08cae640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdb04155be0 con 0x7fdb0413fc30 2024-09-16T01:42:29.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.649+0000 7fdb08cae640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdb040b5c20 con 0x7fdb041400b0 2024-09-16T01:42:29.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.649+0000 7fdb08cae640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdb040b6140 con 0x7fdaf00051c0 2024-09-16T01:42:29.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.649+0000 7fdb01d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb0413fc30 0x7fdb04153d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:29.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.649+0000 7fdb01d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb0413fc30 0x7fdb04153d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49480/0 (socket says 172.21.15.88:49480) 2024-09-16T01:42:29.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.649+0000 7fdb01d9b640 1 -- 172.21.15.88:0/2540879437 learned_addr learned my addr 172.21.15.88:0/2540879437 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:29.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.649+0000 7fdb0259c640 1 --2- 172.21.15.88:0/2540879437 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdaf00051c0 0x7fdb04156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:29.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.649+0000 7fdb02d9d640 1 --2- 172.21.15.88:0/2540879437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb041400b0 0x7fdb04154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:29.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.649+0000 7fdb01d9b640 1 -- 172.21.15.88:0/2540879437 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdaf00051c0 msgr2=0x7fdb04156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:29.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.649+0000 7fdb01d9b640 1 --2- 172.21.15.88:0/2540879437 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdaf00051c0 0x7fdb04156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:29.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.649+0000 7fdb01d9b640 1 -- 172.21.15.88:0/2540879437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb041400b0 msgr2=0x7fdb04154290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:29.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.649+0000 7fdb01d9b640 1 --2- 172.21.15.88:0/2540879437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb041400b0 0x7fdb04154290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:29.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.649+0000 7fdb01d9b640 1 -- 172.21.15.88:0/2540879437 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb04154ac0 con 0x7fdb0413fc30 2024-09-16T01:42:29.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.649+0000 7fdb0259c640 1 --2- 172.21.15.88:0/2540879437 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdaf00051c0 0x7fdb04156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:29.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.649+0000 7fdb02d9d640 1 --2- 172.21.15.88:0/2540879437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb041400b0 0x7fdb04154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:29.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.649+0000 7fdb01d9b640 1 --2- 172.21.15.88:0/2540879437 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb0413fc30 0x7fdb04153d50 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fdaec011960 tx=0x7fdaec011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:29.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.649+0000 7fdafaffd640 1 -- 172.21.15.88:0/2540879437 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaec00faf0 con 0x7fdb0413fc30 2024-09-16T01:42:29.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.649+0000 7fdafaffd640 1 -- 172.21.15.88:0/2540879437 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaec00fc90 con 0x7fdb0413fc30 2024-09-16T01:42:29.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.649+0000 7fdafaffd640 1 -- 172.21.15.88:0/2540879437 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaec014440 con 0x7fdb0413fc30 2024-09-16T01:42:29.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.649+0000 7fdb08cae640 1 -- 172.21.15.88:0/2540879437 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb04140c20 con 0x7fdb0413fc30 2024-09-16T01:42:29.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.649+0000 7fdb08cae640 1 -- 172.21.15.88:0/2540879437 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb041547d0 con 0x7fdb0413fc30 2024-09-16T01:42:29.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.650+0000 7fdb08cae640 1 -- 172.21.15.88:0/2540879437 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb040b6470 con 0x7fdb0413fc30 2024-09-16T01:42:29.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.651+0000 7fdafaffd640 1 -- 172.21.15.88:0/2540879437 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fdaec012040 con 0x7fdb0413fc30 2024-09-16T01:42:29.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.651+0000 7fdafaffd640 1 --2- 172.21.15.88:0/2540879437 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdae406c2b0 0x7fdae406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:29.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.651+0000 7fdafaffd640 1 -- 172.21.15.88:0/2540879437 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 7630+0+0 (secure 0 0 0) 0x7fdaec08c250 con 0x7fdb0413fc30 2024-09-16T01:42:29.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.652+0000 7fdb0259c640 1 --2- 172.21.15.88:0/2540879437 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdae406c2b0 0x7fdae406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:29.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.652+0000 7fdb0259c640 1 --2- 172.21.15.88:0/2540879437 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdae406c2b0 0x7fdae406e770 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fdafc000f80 tx=0x7fdafc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:29.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.656+0000 7fdafaffd640 1 -- 172.21.15.88:0/2540879437 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fdaec019030 con 0x7fdb0413fc30 2024-09-16T01:42:29.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.853+0000 7fdb08cae640 1 -- 172.21.15.88:0/2540879437 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb040b4050 con 0x7fdb0413fc30 2024-09-16T01:42:29.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.853+0000 7fdafaffd640 1 -- 172.21.15.88:0/2540879437 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) ==== 95+0+3566 (secure 0 0 0) 0x7fdaec05a240 con 0x7fdb0413fc30 2024-09-16T01:42:29.855 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:29.855 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":199,"btime":"2024-09-16T01:42:29:541319+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:29.541276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:29.855 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 199 2024-09-16T01:42:29.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.855+0000 7fdb08cae640 1 -- 172.21.15.88:0/2540879437 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdae406c2b0 msgr2=0x7fdae406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:29.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.855+0000 7fdb08cae640 1 --2- 172.21.15.88:0/2540879437 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdae406c2b0 0x7fdae406e770 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fdafc000f80 tx=0x7fdafc009040 comp rx=0 tx=0).stop 2024-09-16T01:42:29.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.855+0000 7fdb08cae640 1 -- 172.21.15.88:0/2540879437 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb0413fc30 msgr2=0x7fdb04153d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:29.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.855+0000 7fdb08cae640 1 --2- 172.21.15.88:0/2540879437 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb0413fc30 0x7fdb04153d50 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fdaec011960 tx=0x7fdaec011e20 comp rx=0 tx=0).stop 2024-09-16T01:42:29.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.855+0000 7fdb08cae640 1 -- 172.21.15.88:0/2540879437 shutdown_connections 2024-09-16T01:42:29.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.855+0000 7fdb08cae640 1 --2- 172.21.15.88:0/2540879437 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdae406c2b0 0x7fdae406e770 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:29.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.855+0000 7fdb08cae640 1 --2- 172.21.15.88:0/2540879437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb041400b0 0x7fdb04154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:29.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.855+0000 7fdb08cae640 1 --2- 172.21.15.88:0/2540879437 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb0413fc30 0x7fdb04153d50 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:29.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.855+0000 7fdb08cae640 1 --2- 172.21.15.88:0/2540879437 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdaf00051c0 0x7fdb04156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:29.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.855+0000 7fdb08cae640 1 -- 172.21.15.88:0/2540879437 >> 172.21.15.88:0/2540879437 conn(0x7fdb0406bc10 msgr2=0x7fdb040cd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:29.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.856+0000 7fdb08cae640 1 -- 172.21.15.88:0/2540879437 shutdown_connections 2024-09-16T01:42:29.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.856+0000 7fdb08cae640 1 -- 172.21.15.88:0/2540879437 wait complete. 2024-09-16T01:42:29.868 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:42:29.868 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:29.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.965+0000 7fccb0ced640 1 Processor -- start 2024-09-16T01:42:29.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.965+0000 7fccb0ced640 1 -- start start 2024-09-16T01:42:29.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.966+0000 7fccb0ced640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fccac15e950 0x7fccac06ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:29.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.966+0000 7fccb0ced640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccac072050 0x7fccac06d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:29.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.966+0000 7fccb0ced640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccac06faf0 con 0x7fccac0735b0 2024-09-16T01:42:29.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.966+0000 7fccb0ced640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fccac06fdd0 con 0x7fccac15e950 2024-09-16T01:42:29.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.966+0000 7fccb0ced640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fccac0700b0 con 0x7fccac072050 2024-09-16T01:42:29.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.966+0000 7fccaa59c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fccac15e950 0x7fccac06ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:29.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.966+0000 7fccaa59c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fccac15e950 0x7fccac06ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49486/0 (socket says 172.21.15.88:49486) 2024-09-16T01:42:29.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.966+0000 7fccaa59c640 1 -- 172.21.15.88:0/3000458213 learned_addr learned my addr 172.21.15.88:0/3000458213 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:29.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.966+0000 7fccaad9d640 1 --2- 172.21.15.88:0/3000458213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccac072050 0x7fccac06d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:29.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.966+0000 7fccaa59c640 1 -- 172.21.15.88:0/3000458213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccac072050 msgr2=0x7fccac06d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:29.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.966+0000 7fccaa59c640 1 --2- 172.21.15.88:0/3000458213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccac072050 0x7fccac06d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:29.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.966+0000 7fccaa59c640 1 -- 172.21.15.88:0/3000458213 >> v1:172.21.15.154:6789/0 conn(0x7fccac0735b0 legacy=0x7fccac06bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:29.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.966+0000 7fccaa59c640 1 -- 172.21.15.88:0/3000458213 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccac070390 con 0x7fccac15e950 2024-09-16T01:42:29.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.967+0000 7fccaa59c640 1 --2- 172.21.15.88:0/3000458213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccac15e950 0x7fccac06ce30 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c009870 tx=0x7fcc9c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=347769da45b021c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:29.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.967+0000 7fcca9d9b640 1 -- 172.21.15.88:0/3000458213 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c037070 con 0x7fccac15e950 2024-09-16T01:42:29.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.967+0000 7fcca9d9b640 1 -- 172.21.15.88:0/3000458213 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc9c032440 con 0x7fccac15e950 2024-09-16T01:42:29.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.967+0000 7fcca9d9b640 1 -- 172.21.15.88:0/3000458213 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c032810 con 0x7fccac15e950 2024-09-16T01:42:29.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.967+0000 7fccb0ced640 1 -- 172.21.15.88:0/3000458213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccac15e950 msgr2=0x7fccac06ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:29.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.967+0000 7fccb0ced640 1 --2- 172.21.15.88:0/3000458213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccac15e950 0x7fccac06ce30 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c009870 tx=0x7fcc9c02a270 comp rx=0 tx=0).stop 2024-09-16T01:42:29.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.968+0000 7fccb0ced640 1 -- 172.21.15.88:0/3000458213 shutdown_connections 2024-09-16T01:42:29.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.968+0000 7fccb0ced640 1 --2- 172.21.15.88:0/3000458213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccac072050 0x7fccac06d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:29.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.968+0000 7fccb0ced640 1 --2- 172.21.15.88:0/3000458213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccac15e950 0x7fccac06ce30 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:29.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.968+0000 7fccb0ced640 1 -- 172.21.15.88:0/3000458213 >> 172.21.15.88:0/3000458213 conn(0x7fccac0d14d0 msgr2=0x7fccac0d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:29.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.968+0000 7fccb0ced640 1 -- 172.21.15.88:0/3000458213 shutdown_connections 2024-09-16T01:42:29.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.968+0000 7fccb0ced640 1 -- 172.21.15.88:0/3000458213 wait complete. 2024-09-16T01:42:29.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.969+0000 7fccb0ced640 1 Processor -- start 2024-09-16T01:42:29.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.969+0000 7fccb0ced640 1 -- start start 2024-09-16T01:42:29.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.970+0000 7fccb0ced640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccac072050 0x7fccac156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:29.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.970+0000 7fccb0ced640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccac0735b0 0x7fccac1541e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:29.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.970+0000 7fccb0ced640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccac15e950 0x7fccac14b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:29.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.970+0000 7fccb0ced640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fccac0d0230 con 0x7fccac072050 2024-09-16T01:42:29.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.970+0000 7fccb0ced640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fccac0d00b0 con 0x7fccac0735b0 2024-09-16T01:42:29.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.970+0000 7fccb0ced640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fccac0d03b0 con 0x7fccac15e950 2024-09-16T01:42:29.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.970+0000 7fccaa59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccac072050 0x7fccac156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:29.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.970+0000 7fccaa59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccac072050 0x7fccac156460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49500/0 (socket says 172.21.15.88:49500) 2024-09-16T01:42:29.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.970+0000 7fccaa59c640 1 -- 172.21.15.88:0/3220758764 learned_addr learned my addr 172.21.15.88:0/3220758764 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:29.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.970+0000 7fccaad9d640 1 --2- 172.21.15.88:0/3220758764 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccac15e950 0x7fccac14b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:29.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.970+0000 7fcca3fff640 1 --2- 172.21.15.88:0/3220758764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccac0735b0 0x7fccac1541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:29.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.970+0000 7fccaa59c640 1 -- 172.21.15.88:0/3220758764 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccac15e950 msgr2=0x7fccac14b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:29.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.970+0000 7fccaa59c640 1 --2- 172.21.15.88:0/3220758764 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccac15e950 0x7fccac14b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:29.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.970+0000 7fccaa59c640 1 -- 172.21.15.88:0/3220758764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccac0735b0 msgr2=0x7fccac1541e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:29.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.970+0000 7fccaa59c640 1 --2- 172.21.15.88:0/3220758764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccac0735b0 0x7fccac1541e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:29.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.970+0000 7fccaa59c640 1 -- 172.21.15.88:0/3220758764 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccac157680 con 0x7fccac072050 2024-09-16T01:42:29.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.970+0000 7fccaa59c640 1 --2- 172.21.15.88:0/3220758764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccac072050 0x7fccac156460 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c000c00 tx=0x7fcc9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:29.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.970+0000 7fcca2ffd640 1 -- 172.21.15.88:0/3220758764 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c041020 con 0x7fccac072050 2024-09-16T01:42:29.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.970+0000 7fcca2ffd640 1 -- 172.21.15.88:0/3220758764 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc9c0306c0 con 0x7fccac072050 2024-09-16T01:42:29.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.970+0000 7fccb0ced640 1 -- 172.21.15.88:0/3220758764 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccac1556a0 con 0x7fccac072050 2024-09-16T01:42:29.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.970+0000 7fcca2ffd640 1 -- 172.21.15.88:0/3220758764 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c037040 con 0x7fccac072050 2024-09-16T01:42:29.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.970+0000 7fccb0ced640 1 -- 172.21.15.88:0/3220758764 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccac155c60 con 0x7fccac072050 2024-09-16T01:42:29.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.971+0000 7fccb0ced640 1 -- 172.21.15.88:0/3220758764 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccac06ce30 con 0x7fccac072050 2024-09-16T01:42:29.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.972+0000 7fcca2ffd640 1 -- 172.21.15.88:0/3220758764 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fcc9c044020 con 0x7fccac072050 2024-09-16T01:42:29.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.972+0000 7fcca2ffd640 1 --2- 172.21.15.88:0/3220758764 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcc8c06c2b0 0x7fcc8c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:29.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.972+0000 7fcca2ffd640 1 -- 172.21.15.88:0/3220758764 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 7630+0+0 (secure 0 0 0) 0x7fcc9c0ac490 con 0x7fccac072050 2024-09-16T01:42:29.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.972+0000 7fcca3fff640 1 --2- 172.21.15.88:0/3220758764 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcc8c06c2b0 0x7fcc8c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:29.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.973+0000 7fcca3fff640 1 --2- 172.21.15.88:0/3220758764 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcc8c06c2b0 0x7fcc8c06e770 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fccac0739b0 tx=0x7fcc94009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:29.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:29.976+0000 7fcca2ffd640 1 -- 172.21.15.88:0/3220758764 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fcc9c04b030 con 0x7fccac072050 2024-09-16T01:42:30.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.175+0000 7fccb0ced640 1 -- 172.21.15.88:0/3220758764 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fccac0b3ee0 con 0x7fccac072050 2024-09-16T01:42:30.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.175+0000 7fcca2ffd640 1 -- 172.21.15.88:0/3220758764 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) ==== 95+0+3566 (secure 0 0 0) 0x7fcc9c07a3e0 con 0x7fccac072050 2024-09-16T01:42:30.177 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:30.177 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":199,"btime":"2024-09-16T01:42:29:541319+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:29.541276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:30.177 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 199 2024-09-16T01:42:30.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.177+0000 7fccb0ced640 1 -- 172.21.15.88:0/3220758764 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcc8c06c2b0 msgr2=0x7fcc8c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:30.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.177+0000 7fccb0ced640 1 --2- 172.21.15.88:0/3220758764 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcc8c06c2b0 0x7fcc8c06e770 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fccac0739b0 tx=0x7fcc94009040 comp rx=0 tx=0).stop 2024-09-16T01:42:30.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.177+0000 7fccb0ced640 1 -- 172.21.15.88:0/3220758764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccac072050 msgr2=0x7fccac156460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:30.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.177+0000 7fccb0ced640 1 --2- 172.21.15.88:0/3220758764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccac072050 0x7fccac156460 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c000c00 tx=0x7fcc9c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:42:30.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.177+0000 7fccb0ced640 1 -- 172.21.15.88:0/3220758764 shutdown_connections 2024-09-16T01:42:30.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.177+0000 7fccb0ced640 1 --2- 172.21.15.88:0/3220758764 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcc8c06c2b0 0x7fcc8c06e770 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:30.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.177+0000 7fccb0ced640 1 --2- 172.21.15.88:0/3220758764 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccac15e950 0x7fccac14b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:30.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.177+0000 7fccb0ced640 1 --2- 172.21.15.88:0/3220758764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccac0735b0 0x7fccac1541e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:30.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.177+0000 7fccb0ced640 1 --2- 172.21.15.88:0/3220758764 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccac072050 0x7fccac156460 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:30.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.177+0000 7fccb0ced640 1 -- 172.21.15.88:0/3220758764 >> 172.21.15.88:0/3220758764 conn(0x7fccac0d14d0 msgr2=0x7fccac0b33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:30.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.177+0000 7fccb0ced640 1 -- 172.21.15.88:0/3220758764 shutdown_connections 2024-09-16T01:42:30.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.177+0000 7fccb0ced640 1 -- 172.21.15.88:0/3220758764 wait complete. 2024-09-16T01:42:30.189 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:42:30.190 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:30.258 INFO:tasks.ceph.mds.a.smithi088.stderr: -42> 2024-09-16T01:42:25.505+0000 7f6c35c28640 -1 log_channel(cluster) log [ERR] : error decoding table object 'mds0_inotable': Decoder at 'virtual void InoTable::decode_state(ceph::buffer::v15_2_0::list::const_iterator&)' v=2 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-09-16T01:42:30.282 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:42:30.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.289+0000 7f822efbd640 1 Processor -- start 2024-09-16T01:42:30.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.289+0000 7f822efbd640 1 -- start start 2024-09-16T01:42:30.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.289+0000 7f822efbd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f822815e660 0x7f82280cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:30.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.289+0000 7f822efbd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f82280d3310 0x7f82280d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:30.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.289+0000 7f822efbd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82280d1650 0x7f822806bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:30.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.289+0000 7f822efbd640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f82280d1a90 con 0x7f822815e660 2024-09-16T01:42:30.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.289+0000 7f822efbd640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f822815ea40 con 0x7f82280d3310 2024-09-16T01:42:30.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.289+0000 7f822efbd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f82280d3870 con 0x7f82280d1650 2024-09-16T01:42:30.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.289+0000 7f822cd59640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f822815e660 0x7f82280cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:30.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.289+0000 7f822cd59640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f822815e660 0x7f82280cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:49516/0 (socket says 172.21.15.88:49516) 2024-09-16T01:42:30.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.289+0000 7f822cd59640 1 -- 172.21.15.88:0/3064881985 learned_addr learned my addr 172.21.15.88:0/3064881985 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:30.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.290+0000 7f822cd59640 1 -- 172.21.15.88:0/3064881985 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82280d1650 msgr2=0x7f822806bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:30.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.290+0000 7f822cd59640 1 --2- 172.21.15.88:0/3064881985 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82280d1650 0x7f822806bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:30.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.290+0000 7f822cd59640 1 -- 172.21.15.88:0/3064881985 >> v2:172.21.15.154:3300/0 conn(0x7f82280d3310 msgr2=0x7f82280d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:30.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.290+0000 7f822cd59640 1 --2- 172.21.15.88:0/3064881985 >> v2:172.21.15.154:3300/0 conn(0x7f82280d3310 0x7f82280d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:30.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.290+0000 7f822cd59640 1 -- 172.21.15.88:0/3064881985 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82280d3b50 con 0x7f822815e660 2024-09-16T01:42:30.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.290+0000 7f822cd59640 1 --2- 172.21.15.88:0/3064881985 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f822815e660 0x7f82280cf730 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f82280c8020 tx=0x7f821c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc02083462cd114f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:30.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.290+0000 7f82277fe640 1 -- 172.21.15.88:0/3064881985 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821c037070 con 0x7f822815e660 2024-09-16T01:42:30.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.290+0000 7f82277fe640 1 -- 172.21.15.88:0/3064881985 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f821c032440 con 0x7f822815e660 2024-09-16T01:42:30.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.291+0000 7f82277fe640 1 -- 172.21.15.88:0/3064881985 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821c032810 con 0x7f822815e660 2024-09-16T01:42:30.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.291+0000 7f822efbd640 1 -- 172.21.15.88:0/3064881985 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f822815e660 msgr2=0x7f82280cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:30.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.291+0000 7f822efbd640 1 --2- 172.21.15.88:0/3064881985 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f822815e660 0x7f82280cf730 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f82280c8020 tx=0x7f821c02a1e0 comp rx=0 tx=0).stop 2024-09-16T01:42:30.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.291+0000 7f822efbd640 1 -- 172.21.15.88:0/3064881985 shutdown_connections 2024-09-16T01:42:30.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.291+0000 7f822efbd640 1 --2- 172.21.15.88:0/3064881985 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82280d1650 0x7f822806bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:30.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.291+0000 7f822efbd640 1 --2- 172.21.15.88:0/3064881985 >> v2:172.21.15.154:3300/0 conn(0x7f82280d3310 0x7f82280d1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:30.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.291+0000 7f822efbd640 1 --2- 172.21.15.88:0/3064881985 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f822815e660 0x7f82280cf730 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:30.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.291+0000 7f822efbd640 1 -- 172.21.15.88:0/3064881985 >> 172.21.15.88:0/3064881985 conn(0x7f82280b6410 msgr2=0x7f82280b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:30.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.291+0000 7f822efbd640 1 -- 172.21.15.88:0/3064881985 shutdown_connections 2024-09-16T01:42:30.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.291+0000 7f822efbd640 1 -- 172.21.15.88:0/3064881985 wait complete. 2024-09-16T01:42:30.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.292+0000 7f822efbd640 1 Processor -- start 2024-09-16T01:42:30.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.292+0000 7f822efbd640 1 -- start start 2024-09-16T01:42:30.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.293+0000 7f822efbd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82280d1650 0x7f82280ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:30.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.293+0000 7f822efbd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82280d3310 0x7f822815fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:30.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.293+0000 7f822efbd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f822815e660 0x7f82281601e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:30.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.293+0000 7f822efbd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f82280d43f0 con 0x7f82280d3310 2024-09-16T01:42:30.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.293+0000 7f822efbd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f82280d4110 con 0x7f82280d1650 2024-09-16T01:42:30.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.293+0000 7f822efbd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f82280cfc70 con 0x7f822815e660 2024-09-16T01:42:30.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.293+0000 7f8227fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82280d3310 0x7f822815fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:30.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.293+0000 7f8227fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82280d3310 0x7f822815fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40008/0 (socket says 172.21.15.88:40008) 2024-09-16T01:42:30.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.293+0000 7f8227fff640 1 -- 172.21.15.88:0/324843206 learned_addr learned my addr 172.21.15.88:0/324843206 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:30.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.293+0000 7f8227fff640 1 -- 172.21.15.88:0/324843206 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f822815e660 msgr2=0x7f82281601e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:30.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.293+0000 7f8227fff640 1 --2- 172.21.15.88:0/324843206 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f822815e660 0x7f82281601e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:30.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.293+0000 7f8227fff640 1 -- 172.21.15.88:0/324843206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82280d1650 msgr2=0x7f82280ca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:30.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.293+0000 7f8227fff640 1 --2- 172.21.15.88:0/324843206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82280d1650 0x7f82280ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:30.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.293+0000 7f8227fff640 1 -- 172.21.15.88:0/324843206 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8228153370 con 0x7f82280d3310 2024-09-16T01:42:30.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.294+0000 7f8227fff640 1 --2- 172.21.15.88:0/324843206 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82280d3310 0x7f822815fca0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f821400b4a0 tx=0x7f821400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:30.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.294+0000 7f82257fa640 1 -- 172.21.15.88:0/324843206 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8214014070 con 0x7f82280d3310 2024-09-16T01:42:30.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.294+0000 7f82257fa640 1 -- 172.21.15.88:0/324843206 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8214010610 con 0x7f82280d3310 2024-09-16T01:42:30.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.294+0000 7f822efbd640 1 -- 172.21.15.88:0/324843206 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8228153650 con 0x7f82280d3310 2024-09-16T01:42:30.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.294+0000 7f82257fa640 1 -- 172.21.15.88:0/324843206 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8214010a00 con 0x7f82280d3310 2024-09-16T01:42:30.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.294+0000 7f822efbd640 1 -- 172.21.15.88:0/324843206 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8228153c10 con 0x7f82280d3310 2024-09-16T01:42:30.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.295+0000 7f822efbd640 1 -- 172.21.15.88:0/324843206 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8228153e10 con 0x7f82280d3310 2024-09-16T01:42:30.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.295+0000 7f82257fa640 1 -- 172.21.15.88:0/324843206 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8214017070 con 0x7f82280d3310 2024-09-16T01:42:30.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.296+0000 7f82257fa640 1 --2- 172.21.15.88:0/324843206 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f820c06c2e0 0x7f820c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:30.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.296+0000 7f82257fa640 1 -- 172.21.15.88:0/324843206 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 7630+0+0 (secure 0 0 0) 0x7f821408b620 con 0x7f82280d3310 2024-09-16T01:42:30.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.296+0000 7f822cd59640 1 --2- 172.21.15.88:0/324843206 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f820c06c2e0 0x7f820c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:30.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.296+0000 7f822cd59640 1 --2- 172.21.15.88:0/324843206 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f820c06c2e0 0x7f820c06e7a0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f821c034000 tx=0x7f821c02c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:30.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.300+0000 7f82257fa640 1 -- 172.21.15.88:0/324843206 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8228153e10 con 0x7f82280d3310 2024-09-16T01:42:30.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.505+0000 7f822efbd640 1 -- 172.21.15.88:0/324843206 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82280b88b0 con 0x7f82280d3310 2024-09-16T01:42:30.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.505+0000 7f82257fa640 1 -- 172.21.15.88:0/324843206 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) ==== 95+0+3566 (secure 0 0 0) 0x7f8214059610 con 0x7f82280d3310 2024-09-16T01:42:30.507 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:30.507 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":199,"btime":"2024-09-16T01:42:29:541319+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:29.541276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":109,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:30.507 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 199 2024-09-16T01:42:30.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.508+0000 7f822efbd640 1 -- 172.21.15.88:0/324843206 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f820c06c2e0 msgr2=0x7f820c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:30.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.508+0000 7f822efbd640 1 --2- 172.21.15.88:0/324843206 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f820c06c2e0 0x7f820c06e7a0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f821c034000 tx=0x7f821c02c000 comp rx=0 tx=0).stop 2024-09-16T01:42:30.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.508+0000 7f822efbd640 1 -- 172.21.15.88:0/324843206 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82280d3310 msgr2=0x7f822815fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:30.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.508+0000 7f822efbd640 1 --2- 172.21.15.88:0/324843206 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82280d3310 0x7f822815fca0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f821400b4a0 tx=0x7f821400b960 comp rx=0 tx=0).stop 2024-09-16T01:42:30.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.508+0000 7f822efbd640 1 -- 172.21.15.88:0/324843206 shutdown_connections 2024-09-16T01:42:30.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.508+0000 7f822efbd640 1 --2- 172.21.15.88:0/324843206 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f820c06c2e0 0x7f820c06e7a0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:30.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.508+0000 7f822efbd640 1 --2- 172.21.15.88:0/324843206 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f822815e660 0x7f82281601e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:30.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.508+0000 7f822efbd640 1 --2- 172.21.15.88:0/324843206 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82280d3310 0x7f822815fca0 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:30.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.508+0000 7f822efbd640 1 --2- 172.21.15.88:0/324843206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82280d1650 0x7f82280ca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:30.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.508+0000 7f822efbd640 1 -- 172.21.15.88:0/324843206 >> 172.21.15.88:0/324843206 conn(0x7f82280b6410 msgr2=0x7f822806c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:30.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.508+0000 7f822efbd640 1 -- 172.21.15.88:0/324843206 shutdown_connections 2024-09-16T01:42:30.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.508+0000 7f822efbd640 1 -- 172.21.15.88:0/324843206 wait complete. 2024-09-16T01:42:30.520 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt 400.00000000' led to DAMAGED state 2024-09-16T01:42:30.520 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt 100.00000000.inode' 2024-09-16T01:42:30.520 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:42:30.520 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:42:30.536 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:42:30.536 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:42:30.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:30.536 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:42:30.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:42:30.592 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:42:30.592 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:42:30.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.729+0000 7f612f0ae640 1 Processor -- start 2024-09-16T01:42:30.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.730+0000 7f612f0ae640 1 -- start start 2024-09-16T01:42:30.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.730+0000 7f612f0ae640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f61280b4e60 0x7f61280d0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:30.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.730+0000 7f612f0ae640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f61280b5c00 0x7f61280b5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:30.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.730+0000 7f612f0ae640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f612815ffc0 0x7f6128160430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:30.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.730+0000 7f612f0ae640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f61280ce5c0 con 0x7f61280b4e60 2024-09-16T01:42:30.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.730+0000 7f612f0ae640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f61280cd9b0 con 0x7f61280b5c00 2024-09-16T01:42:30.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.730+0000 7f612f0ae640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f61280cdc90 con 0x7f612815ffc0 2024-09-16T01:42:30.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.731+0000 7f612ce4a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f61280b4e60 0x7f61280d0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:30.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.731+0000 7f612ce4a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f61280b4e60 0x7f61280d0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40016/0 (socket says 172.21.15.88:40016) 2024-09-16T01:42:30.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.731+0000 7f612ce4a640 1 -- 172.21.15.88:0/1635561551 learned_addr learned my addr 172.21.15.88:0/1635561551 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:30.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.731+0000 7f612d64b640 1 --2- 172.21.15.88:0/1635561551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f612815ffc0 0x7f6128160430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:30.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.731+0000 7f6127fff640 1 --2- 172.21.15.88:0/1635561551 >> v2:172.21.15.154:3300/0 conn(0x7f61280b5c00 0x7f61280b5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:30.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.731+0000 7f612ce4a640 1 -- 172.21.15.88:0/1635561551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f612815ffc0 msgr2=0x7f6128160430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:30.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.731+0000 7f612ce4a640 1 --2- 172.21.15.88:0/1635561551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f612815ffc0 0x7f6128160430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:30.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.731+0000 7f612ce4a640 1 -- 172.21.15.88:0/1635561551 >> v2:172.21.15.154:3300/0 conn(0x7f61280b5c00 msgr2=0x7f61280b5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:30.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.731+0000 7f612ce4a640 1 --2- 172.21.15.88:0/1635561551 >> v2:172.21.15.154:3300/0 conn(0x7f61280b5c00 0x7f61280b5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:30.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.731+0000 7f612ce4a640 1 -- 172.21.15.88:0/1635561551 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61280cdf70 con 0x7f61280b4e60 2024-09-16T01:42:30.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.731+0000 7f612d64b640 1 --2- 172.21.15.88:0/1635561551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f612815ffc0 0x7f6128160430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:30.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.732+0000 7f612ce4a640 1 --2- 172.21.15.88:0/1635561551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61280b4e60 0x7f61280d0760 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f6128071380 tx=0x7f611c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b2b30fc748ec29a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:30.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.732+0000 7f61277fe640 1 -- 172.21.15.88:0/1635561551 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c037070 con 0x7f61280b4e60 2024-09-16T01:42:30.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.732+0000 7f61277fe640 1 -- 172.21.15.88:0/1635561551 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f611c032440 con 0x7f61280b4e60 2024-09-16T01:42:30.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.732+0000 7f61277fe640 1 -- 172.21.15.88:0/1635561551 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c032810 con 0x7f61280b4e60 2024-09-16T01:42:30.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.732+0000 7f612f0ae640 1 -- 172.21.15.88:0/1635561551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61280b4e60 msgr2=0x7f61280d0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:30.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.732+0000 7f612f0ae640 1 --2- 172.21.15.88:0/1635561551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61280b4e60 0x7f61280d0760 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f6128071380 tx=0x7f611c02a1e0 comp rx=0 tx=0).stop 2024-09-16T01:42:30.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.732+0000 7f612f0ae640 1 -- 172.21.15.88:0/1635561551 shutdown_connections 2024-09-16T01:42:30.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.732+0000 7f612f0ae640 1 --2- 172.21.15.88:0/1635561551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f612815ffc0 0x7f6128160430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:30.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.732+0000 7f612f0ae640 1 --2- 172.21.15.88:0/1635561551 >> v2:172.21.15.154:3300/0 conn(0x7f61280b5c00 0x7f61280b5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:30.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.732+0000 7f612f0ae640 1 --2- 172.21.15.88:0/1635561551 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61280b4e60 0x7f61280d0760 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:30.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.732+0000 7f612f0ae640 1 -- 172.21.15.88:0/1635561551 >> 172.21.15.88:0/1635561551 conn(0x7f612806cb50 msgr2=0x7f612806cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:30.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.732+0000 7f612f0ae640 1 -- 172.21.15.88:0/1635561551 shutdown_connections 2024-09-16T01:42:30.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.732+0000 7f612f0ae640 1 -- 172.21.15.88:0/1635561551 wait complete. 2024-09-16T01:42:30.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.734+0000 7f612f0ae640 1 Processor -- start 2024-09-16T01:42:30.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.734+0000 7f612f0ae640 1 -- start start 2024-09-16T01:42:30.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.734+0000 7f612f0ae640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61280b4e60 0x7f6128158a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:30.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.734+0000 7f612f0ae640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61280b5c00 0x7f6128153870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:30.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.734+0000 7f612f0ae640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f612815ffc0 0x7f612815a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:30.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.734+0000 7f612f0ae640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f61280b6090 con 0x7f612815ffc0 2024-09-16T01:42:30.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.734+0000 7f612f0ae640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f61280b5780 con 0x7f61280b5c00 2024-09-16T01:42:30.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.734+0000 7f612f0ae640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f61280b6210 con 0x7f61280b4e60 2024-09-16T01:42:30.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.735+0000 7f612d64b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f612815ffc0 0x7f612815a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:30.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.735+0000 7f612d64b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f612815ffc0 0x7f612815a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40030/0 (socket says 172.21.15.88:40030) 2024-09-16T01:42:30.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.735+0000 7f612d64b640 1 -- 172.21.15.88:0/2533024263 learned_addr learned my addr 172.21.15.88:0/2533024263 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:30.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.735+0000 7f6127fff640 1 --2- 172.21.15.88:0/2533024263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61280b5c00 0x7f6128153870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:30.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.735+0000 7f612ce4a640 1 --2- 172.21.15.88:0/2533024263 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61280b4e60 0x7f6128158a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:30.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.735+0000 7f612d64b640 1 -- 172.21.15.88:0/2533024263 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61280b4e60 msgr2=0x7f6128158a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:30.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.735+0000 7f612d64b640 1 --2- 172.21.15.88:0/2533024263 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61280b4e60 0x7f6128158a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:30.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.735+0000 7f612d64b640 1 -- 172.21.15.88:0/2533024263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61280b5c00 msgr2=0x7f6128153870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:30.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.735+0000 7f612d64b640 1 --2- 172.21.15.88:0/2533024263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61280b5c00 0x7f6128153870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:30.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.735+0000 7f612d64b640 1 -- 172.21.15.88:0/2533024263 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f612815b0f0 con 0x7f612815ffc0 2024-09-16T01:42:30.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.735+0000 7f612d64b640 1 --2- 172.21.15.88:0/2533024263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f612815ffc0 0x7f612815a9e0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f6120007f20 tx=0x7f612000d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:30.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.735+0000 7f61257fa640 1 -- 172.21.15.88:0/2533024263 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612001a070 con 0x7f612815ffc0 2024-09-16T01:42:30.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.735+0000 7f61257fa640 1 -- 172.21.15.88:0/2533024263 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6120003b90 con 0x7f612815ffc0 2024-09-16T01:42:30.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.735+0000 7f612f0ae640 1 -- 172.21.15.88:0/2533024263 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f612815b3d0 con 0x7f612815ffc0 2024-09-16T01:42:30.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.735+0000 7f61257fa640 1 -- 172.21.15.88:0/2533024263 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6120004030 con 0x7f612815ffc0 2024-09-16T01:42:30.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.735+0000 7f612f0ae640 1 -- 172.21.15.88:0/2533024263 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f612815b990 con 0x7f612815ffc0 2024-09-16T01:42:30.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.736+0000 7f612f0ae640 1 -- 172.21.15.88:0/2533024263 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f612815fc00 con 0x7f612815ffc0 2024-09-16T01:42:30.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.737+0000 7f61257fa640 1 -- 172.21.15.88:0/2533024263 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f612001d070 con 0x7f612815ffc0 2024-09-16T01:42:30.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.737+0000 7f61257fa640 1 --2- 172.21.15.88:0/2533024263 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f610c06c2e0 0x7f610c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:30.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.737+0000 7f612ce4a640 1 --2- 172.21.15.88:0/2533024263 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f610c06c2e0 0x7f610c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:30.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.737+0000 7f61257fa640 1 -- 172.21.15.88:0/2533024263 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 7630+0+0 (secure 0 0 0) 0x7f612008bd90 con 0x7f612815ffc0 2024-09-16T01:42:30.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.738+0000 7f612ce4a640 1 --2- 172.21.15.88:0/2533024263 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f610c06c2e0 0x7f610c06e7a0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f611c034000 tx=0x7f611c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:30.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.741+0000 7f61257fa640 1 -- 172.21.15.88:0/2533024263 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6120010070 con 0x7f612815ffc0 2024-09-16T01:42:30.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:30.944+0000 7f612f0ae640 1 -- 172.21.15.88:0/2533024263 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f61280701a0 con 0x7f612815ffc0 2024-09-16T01:42:31.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.553+0000 7f61257fa640 1 -- 172.21.15.88:0/2533024263 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v201) ==== 161+0+0 (secure 0 0 0) 0x7f6120059730 con 0x7f612815ffc0 2024-09-16T01:42:31.554 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:42:31.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.555+0000 7f612f0ae640 1 -- 172.21.15.88:0/2533024263 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f610c06c2e0 msgr2=0x7f610c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:31.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.555+0000 7f612f0ae640 1 --2- 172.21.15.88:0/2533024263 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f610c06c2e0 0x7f610c06e7a0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f611c034000 tx=0x7f611c02e000 comp rx=0 tx=0).stop 2024-09-16T01:42:31.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.555+0000 7f612f0ae640 1 -- 172.21.15.88:0/2533024263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f612815ffc0 msgr2=0x7f612815a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:31.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.555+0000 7f612f0ae640 1 --2- 172.21.15.88:0/2533024263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f612815ffc0 0x7f612815a9e0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f6120007f20 tx=0x7f612000d560 comp rx=0 tx=0).stop 2024-09-16T01:42:31.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.555+0000 7f612f0ae640 1 -- 172.21.15.88:0/2533024263 shutdown_connections 2024-09-16T01:42:31.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.555+0000 7f612f0ae640 1 --2- 172.21.15.88:0/2533024263 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f610c06c2e0 0x7f610c06e7a0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:31.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.555+0000 7f612f0ae640 1 --2- 172.21.15.88:0/2533024263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f612815ffc0 0x7f612815a9e0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:31.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.555+0000 7f612f0ae640 1 --2- 172.21.15.88:0/2533024263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61280b5c00 0x7f6128153870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:31.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.555+0000 7f612f0ae640 1 --2- 172.21.15.88:0/2533024263 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61280b4e60 0x7f6128158a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:31.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.555+0000 7f612f0ae640 1 -- 172.21.15.88:0/2533024263 >> 172.21.15.88:0/2533024263 conn(0x7f612806cb50 msgr2=0x7f6128154a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:31.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.556+0000 7f612f0ae640 1 -- 172.21.15.88:0/2533024263 shutdown_connections 2024-09-16T01:42:31.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.556+0000 7f612f0ae640 1 -- 172.21.15.88:0/2533024263 wait complete. 2024-09-16T01:42:31.569 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:42:31.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.668+0000 7f6384128640 1 Processor -- start 2024-09-16T01:42:31.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.669+0000 7f6384128640 1 -- start start 2024-09-16T01:42:31.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.669+0000 7f6384128640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f637c15f770 0x7f637c15d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:31.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.669+0000 7f6384128640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f637c15fc20 con 0x7f637c155880 2024-09-16T01:42:31.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.669+0000 7f6384128640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f637c15bb60 con 0x7f637c0b5470 2024-09-16T01:42:31.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.669+0000 7f6384128640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f637c155c60 con 0x7f637c15f770 2024-09-16T01:42:31.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.670+0000 7f63826c5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f637c15f770 0x7f637c15d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:31.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.670+0000 7f6381ec4640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f637c155880 0x7f637c0b4d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50808/0 (socket says 172.21.15.88:50808) 2024-09-16T01:42:31.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.670+0000 7f6381ec4640 1 -- 172.21.15.88:0/2120442125 learned_addr learned my addr 172.21.15.88:0/2120442125 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:31.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.670+0000 7f63826c5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f637c15f770 0x7f637c15d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40040/0 (socket says 172.21.15.88:40040) 2024-09-16T01:42:31.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.670+0000 7f63826c5640 1 -- 172.21.15.88:0/2120442125 >> v1:172.21.15.154:6789/0 conn(0x7f637c0b5470 legacy=0x7f637c15cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:31.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.670+0000 7f63826c5640 1 -- 172.21.15.88:0/2120442125 >> v1:172.21.15.88:6789/0 conn(0x7f637c155880 legacy=0x7f637c0b4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:31.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.670+0000 7f63826c5640 1 -- 172.21.15.88:0/2120442125 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f637c15be40 con 0x7f637c15f770 2024-09-16T01:42:31.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.670+0000 7f6380ec2640 1 -- 172.21.15.88:0/2120442125 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1738380837 0 0) 0x7f637c15fc20 con 0x7f637c155880 2024-09-16T01:42:31.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.671+0000 7f63826c5640 1 --2- 172.21.15.88:0/2120442125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c15f770 0x7f637c15d330 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f637400bfa0 tx=0x7f637402c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcab9f3c9c1bb17a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:31.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.671+0000 7f6380ec2640 1 -- 172.21.15.88:0/2120442125 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6374039070 con 0x7f637c15f770 2024-09-16T01:42:31.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.671+0000 7f6380ec2640 1 -- 172.21.15.88:0/2120442125 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6374009540 con 0x7f637c15f770 2024-09-16T01:42:31.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.671+0000 7f6380ec2640 1 -- 172.21.15.88:0/2120442125 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6374009910 con 0x7f637c15f770 2024-09-16T01:42:31.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.671+0000 7f6384128640 1 -- 172.21.15.88:0/2120442125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c15f770 msgr2=0x7f637c15d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:31.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.671+0000 7f6384128640 1 --2- 172.21.15.88:0/2120442125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c15f770 0x7f637c15d330 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f637400bfa0 tx=0x7f637402c8b0 comp rx=0 tx=0).stop 2024-09-16T01:42:31.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.671+0000 7f6384128640 1 -- 172.21.15.88:0/2120442125 shutdown_connections 2024-09-16T01:42:31.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.671+0000 7f6384128640 1 --2- 172.21.15.88:0/2120442125 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c15f770 0x7f637c15d330 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:31.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.671+0000 7f6384128640 1 -- 172.21.15.88:0/2120442125 >> 172.21.15.88:0/2120442125 conn(0x7f637c0d1390 msgr2=0x7f637c0d17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:31.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.671+0000 7f6384128640 1 -- 172.21.15.88:0/2120442125 shutdown_connections 2024-09-16T01:42:31.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.671+0000 7f6384128640 1 -- 172.21.15.88:0/2120442125 wait complete. 2024-09-16T01:42:31.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.672+0000 7f6384128640 1 Processor -- start 2024-09-16T01:42:31.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.672+0000 7f6384128640 1 -- start start 2024-09-16T01:42:31.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.673+0000 7f6384128640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f637c0b5470 0x7f637c148dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:31.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.673+0000 7f6384128640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c155880 0x7f637c155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:31.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.673+0000 7f6384128640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f637c15f770 0x7f637c14dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:31.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.673+0000 7f6384128640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f637c15c6e0 con 0x7f637c155880 2024-09-16T01:42:31.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.673+0000 7f6384128640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f637c15c400 con 0x7f637c0b5470 2024-09-16T01:42:31.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.673+0000 7f6384128640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f637c15b770 con 0x7f637c15f770 2024-09-16T01:42:31.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.673+0000 7f63816c3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c155880 0x7f637c155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:31.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.673+0000 7f63816c3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c155880 0x7f637c155df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40050/0 (socket says 172.21.15.88:40050) 2024-09-16T01:42:31.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.673+0000 7f63816c3640 1 -- 172.21.15.88:0/1122949251 learned_addr learned my addr 172.21.15.88:0/1122949251 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:31.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.673+0000 7f6381ec4640 1 --2- 172.21.15.88:0/1122949251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f637c0b5470 0x7f637c148dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:31.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.673+0000 7f63826c5640 1 --2- 172.21.15.88:0/1122949251 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f637c15f770 0x7f637c14dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:31.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.673+0000 7f63816c3640 1 -- 172.21.15.88:0/1122949251 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f637c15f770 msgr2=0x7f637c14dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:31.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.674+0000 7f63816c3640 1 --2- 172.21.15.88:0/1122949251 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f637c15f770 0x7f637c14dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:31.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.674+0000 7f63816c3640 1 -- 172.21.15.88:0/1122949251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f637c0b5470 msgr2=0x7f637c148dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:31.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.674+0000 7f63816c3640 1 --2- 172.21.15.88:0/1122949251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f637c0b5470 0x7f637c148dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:31.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.674+0000 7f63816c3640 1 -- 172.21.15.88:0/1122949251 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f637c150320 con 0x7f637c155880 2024-09-16T01:42:31.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.674+0000 7f6381ec4640 1 --2- 172.21.15.88:0/1122949251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f637c0b5470 0x7f637c148dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:31.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.674+0000 7f63826c5640 1 --2- 172.21.15.88:0/1122949251 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f637c15f770 0x7f637c14dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:31.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.674+0000 7f63816c3640 1 --2- 172.21.15.88:0/1122949251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c155880 0x7f637c155df0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f637000c400 tx=0x7f637000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:31.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.674+0000 7f636e7fc640 1 -- 172.21.15.88:0/1122949251 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6370011790 con 0x7f637c155880 2024-09-16T01:42:31.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.674+0000 7f636e7fc640 1 -- 172.21.15.88:0/1122949251 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6370017070 con 0x7f637c155880 2024-09-16T01:42:31.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.674+0000 7f636e7fc640 1 -- 172.21.15.88:0/1122949251 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6370011bb0 con 0x7f637c155880 2024-09-16T01:42:31.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.674+0000 7f6384128640 1 -- 172.21.15.88:0/1122949251 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f637c0cf360 con 0x7f637c155880 2024-09-16T01:42:31.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.674+0000 7f6384128640 1 -- 172.21.15.88:0/1122949251 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f637c0cf920 con 0x7f637c155880 2024-09-16T01:42:31.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.675+0000 7f6384128640 1 -- 172.21.15.88:0/1122949251 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f637c15eb20 con 0x7f637c155880 2024-09-16T01:42:31.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.675+0000 7f636e7fc640 1 -- 172.21.15.88:0/1122949251 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f637001c460 con 0x7f637c155880 2024-09-16T01:42:31.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.676+0000 7f636e7fc640 1 --2- 172.21.15.88:0/1122949251 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f636006c2b0 0x7f636006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:31.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.676+0000 7f636e7fc640 1 -- 172.21.15.88:0/1122949251 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 7630+0+0 (secure 0 0 0) 0x7f637008ba80 con 0x7f637c155880 2024-09-16T01:42:31.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.676+0000 7f6381ec4640 1 --2- 172.21.15.88:0/1122949251 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f636006c2b0 0x7f636006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:31.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.676+0000 7f6381ec4640 1 --2- 172.21.15.88:0/1122949251 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f636006c2b0 0x7f636006e770 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f6378000f60 tx=0x7f6378009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:31.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.680+0000 7f636e7fc640 1 -- 172.21.15.88:0/1122949251 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6370059a70 con 0x7f637c155880 2024-09-16T01:42:31.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:31.913+0000 7f6384128640 1 -- 172.21.15.88:0/1122949251 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f637c154a40 con 0x7f637c155880 2024-09-16T01:42:32.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.558+0000 7f636e7fc640 1 -- 172.21.15.88:0/1122949251 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v202) ==== 101+0+0 (secure 0 0 0) 0x7f6370059370 con 0x7f637c155880 2024-09-16T01:42:32.559 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:42:32.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.561+0000 7f6384128640 1 -- 172.21.15.88:0/1122949251 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f636006c2b0 msgr2=0x7f636006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:32.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.561+0000 7f6384128640 1 --2- 172.21.15.88:0/1122949251 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f636006c2b0 0x7f636006e770 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f6378000f60 tx=0x7f6378009040 comp rx=0 tx=0).stop 2024-09-16T01:42:32.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.561+0000 7f6384128640 1 -- 172.21.15.88:0/1122949251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c155880 msgr2=0x7f637c155df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:32.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.561+0000 7f6384128640 1 --2- 172.21.15.88:0/1122949251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c155880 0x7f637c155df0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f637000c400 tx=0x7f637000c8c0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.561+0000 7f6384128640 1 -- 172.21.15.88:0/1122949251 shutdown_connections 2024-09-16T01:42:32.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.561+0000 7f6384128640 1 --2- 172.21.15.88:0/1122949251 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f636006c2b0 0x7f636006e770 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.561+0000 7f6384128640 1 --2- 172.21.15.88:0/1122949251 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f637c15f770 0x7f637c14dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.561+0000 7f6384128640 1 --2- 172.21.15.88:0/1122949251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c155880 0x7f637c155df0 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.561+0000 7f6384128640 1 --2- 172.21.15.88:0/1122949251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f637c0b5470 0x7f637c148dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.561+0000 7f6384128640 1 -- 172.21.15.88:0/1122949251 >> 172.21.15.88:0/1122949251 conn(0x7f637c0d1390 msgr2=0x7f637c06bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:32.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.561+0000 7f6384128640 1 -- 172.21.15.88:0/1122949251 shutdown_connections 2024-09-16T01:42:32.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.561+0000 7f6384128640 1 -- 172.21.15.88:0/1122949251 wait complete. 2024-09-16T01:42:32.573 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:42:32.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.614+0000 7f1f3d11ef40 1 Processor -- start 2024-09-16T01:42:32.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.615+0000 7f1f3d11ef40 1 -- start start 2024-09-16T01:42:32.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.615+0000 7f1f3d11ef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5581b2f87fe0 0x5581b302ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:32.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.615+0000 7f1f3d11ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5581b2f88a60 0x5581b302f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:32.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.615+0000 7f1f3d11ef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5581b2fa2250 0x5581b302fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:32.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.615+0000 7f1f3d11ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5581b2eaa510 con 0x5581b2f88a60 2024-09-16T01:42:32.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.615+0000 7f1f3d11ef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5581b3031120 con 0x5581b2f87fe0 2024-09-16T01:42:32.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.615+0000 7f1f3d11ef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5581b3021480 con 0x5581b2fa2250 2024-09-16T01:42:32.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.615+0000 7f1f3a413640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5581b2f88a60 0x5581b302f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:32.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.615+0000 7f1f3a413640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5581b2f88a60 0x5581b302f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40064/0 (socket says 172.21.15.88:40064) 2024-09-16T01:42:32.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.615+0000 7f1f3a413640 1 -- 172.21.15.88:0/196258392 learned_addr learned my addr 172.21.15.88:0/196258392 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:32.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.615+0000 7f1f3ac14640 1 --2- 172.21.15.88:0/196258392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5581b2f87fe0 0x5581b302ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:32.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.615+0000 7f1f3b415640 1 --2- 172.21.15.88:0/196258392 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5581b2fa2250 0x5581b302fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:32.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.615+0000 7f1f3a413640 1 -- 172.21.15.88:0/196258392 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5581b2fa2250 msgr2=0x5581b302fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:32.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.616+0000 7f1f3a413640 1 --2- 172.21.15.88:0/196258392 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5581b2fa2250 0x5581b302fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.616+0000 7f1f3a413640 1 -- 172.21.15.88:0/196258392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5581b2f87fe0 msgr2=0x5581b302ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:32.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.616+0000 7f1f3a413640 1 --2- 172.21.15.88:0/196258392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5581b2f87fe0 0x5581b302ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.616+0000 7f1f3a413640 1 -- 172.21.15.88:0/196258392 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581b2f9f0f0 con 0x5581b2f88a60 2024-09-16T01:42:32.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.616+0000 7f1f3a413640 1 --2- 172.21.15.88:0/196258392 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5581b2f88a60 0x5581b302f8e0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f1f30002de0 tx=0x7f1f3000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34c8523843fb6746 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:32.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.616+0000 7f1f39411640 1 -- 172.21.15.88:0/196258392 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3000f400 con 0x5581b2f88a60 2024-09-16T01:42:32.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.616+0000 7f1f39411640 1 -- 172.21.15.88:0/196258392 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f30014070 con 0x5581b2f88a60 2024-09-16T01:42:32.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.616+0000 7f1f39411640 1 -- 172.21.15.88:0/196258392 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3000f7e0 con 0x5581b2f88a60 2024-09-16T01:42:32.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.616+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/196258392 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5581b2f88a60 msgr2=0x5581b302f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:32.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.616+0000 7f1f3d11ef40 1 --2- 172.21.15.88:0/196258392 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5581b2f88a60 0x5581b302f8e0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f1f30002de0 tx=0x7f1f3000b5b0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.616+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/196258392 shutdown_connections 2024-09-16T01:42:32.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.616+0000 7f1f3d11ef40 1 --2- 172.21.15.88:0/196258392 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5581b2fa2250 0x5581b302fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.616+0000 7f1f3d11ef40 1 --2- 172.21.15.88:0/196258392 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5581b2f88a60 0x5581b302f8e0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.616+0000 7f1f3d11ef40 1 --2- 172.21.15.88:0/196258392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5581b2f87fe0 0x5581b302ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.616+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/196258392 >> 172.21.15.88:0/196258392 conn(0x5581b2fa3750 msgr2=0x5581b2f84200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:32.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.616+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/196258392 shutdown_connections 2024-09-16T01:42:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.616+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/196258392 wait complete. 2024-09-16T01:42:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.617+0000 7f1f3d11ef40 1 Processor -- start 2024-09-16T01:42:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.617+0000 7f1f3d11ef40 1 -- start start 2024-09-16T01:42:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.618+0000 7f1f3d11ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5581b2f87fe0 0x5581b3020280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.618+0000 7f1f3d11ef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5581b2f88a60 0x5581b3023f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.618+0000 7f1f3d11ef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5581b2fa2250 0x5581b30244d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.618+0000 7f1f3d11ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5581b2fa33e0 con 0x5581b2f87fe0 2024-09-16T01:42:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.618+0000 7f1f3d11ef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5581b2fa30b0 con 0x5581b2fa2250 2024-09-16T01:42:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.618+0000 7f1f3d11ef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5581b2f9f3d0 con 0x5581b2f88a60 2024-09-16T01:42:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.618+0000 7f1f3ac14640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5581b2f87fe0 0x5581b3020280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.618+0000 7f1f3ac14640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5581b2f87fe0 0x5581b3020280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40066/0 (socket says 172.21.15.88:40066) 2024-09-16T01:42:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.618+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 learned_addr learned my addr 172.21.15.88:0/4170766770 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.618+0000 7f1f3b415640 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5581b2fa2250 0x5581b30244d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.618+0000 7f1f3a413640 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5581b2f88a60 0x5581b3023f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.618+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5581b2f88a60 msgr2=0x5581b3023f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.618+0000 7f1f3ac14640 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5581b2f88a60 0x5581b3023f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.618+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5581b2fa2250 msgr2=0x5581b30244d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.618+0000 7f1f3ac14640 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5581b2fa2250 0x5581b30244d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.618+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581b3022cb0 con 0x5581b2f87fe0 2024-09-16T01:42:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.618+0000 7f1f3a413640 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5581b2f88a60 0x5581b3023f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.618+0000 7f1f3ac14640 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5581b2f87fe0 0x5581b3020280 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c007e30 tx=0x7f1f2c006750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.618+0000 7f1f23fff640 1 -- 172.21.15.88:0/4170766770 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c018070 con 0x5581b2f87fe0 2024-09-16T01:42:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.618+0000 7f1f23fff640 1 -- 172.21.15.88:0/4170766770 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f2c011700 con 0x5581b2f87fe0 2024-09-16T01:42:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.618+0000 7f1f23fff640 1 -- 172.21.15.88:0/4170766770 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c011ad0 con 0x5581b2f87fe0 2024-09-16T01:42:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.618+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5581b3022f90 con 0x5581b2f87fe0 2024-09-16T01:42:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.618+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581b3023550 con 0x5581b2f87fe0 2024-09-16T01:42:32.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.620+0000 7f1f23fff640 1 -- 172.21.15.88:0/4170766770 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1f2c01b070 con 0x5581b2f87fe0 2024-09-16T01:42:32.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.620+0000 7f1f23fff640 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1f2806ac90 0x7f1f2806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:32.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.620+0000 7f1f23fff640 1 -- 172.21.15.88:0/4170766770 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 7630+0+0 (secure 0 0 0) 0x7f1f2c089b10 con 0x5581b2f87fe0 2024-09-16T01:42:32.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.620+0000 7f1f3a413640 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1f2806ac90 0x7f1f2806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:32.628 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:42:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.620+0000 7f1f3d11ef40 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5581b302fda0 0x5581b30c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.620+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b3023a70 con 0x5581b302fda0 2024-09-16T01:42:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.621+0000 7f1f3b415640 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5581b302fda0 0x5581b30c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.621+0000 7f1f3a413640 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1f2806ac90 0x7f1f2806d150 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x5581b2f833a0 tx=0x7f1f30008000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.621+0000 7f1f3b415640 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5581b302fda0 0x5581b30c73e0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.623+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v109'660 uv634 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.623+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e109) -- 0x5581b302f8e0 con 0x5581b302fda0 2024-09-16T01:42:32.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.624+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v109'661 uv661 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.624+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d3020 con 0x5581b302fda0 2024-09-16T01:42:32.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.625+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v109'662 uv662 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.630 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:42:32.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.625+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d43d0 con 0x5581b302fda0 2024-09-16T01:42:32.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.627+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v109'663 uv663 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.627+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d4770 con 0x5581b302fda0 2024-09-16T01:42:32.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.628+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v109'664 uv664 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.628+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d33c0 con 0x5581b302fda0 2024-09-16T01:42:32.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.629+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v109'665 uv665 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.629+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d37b0 con 0x5581b302fda0 2024-09-16T01:42:32.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.630+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v109'666 uv640 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.630+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d3ba0 con 0x5581b302fda0 2024-09-16T01:42:32.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.632+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v109'667 uv667 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.632+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d3f90 con 0x5581b302fda0 2024-09-16T01:42:32.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.633+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v109'668 uv668 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.634 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:42:32.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.633+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d3f90 con 0x5581b302fda0 2024-09-16T01:42:32.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.634+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v109'669 uv669 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.634+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d3f40 con 0x5581b302fda0 2024-09-16T01:42:32.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.636+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v109'670 uv670 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.636+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d3f40 con 0x5581b302fda0 2024-09-16T01:42:32.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.637+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v109'671 uv671 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.637+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d3f40 con 0x5581b302fda0 2024-09-16T01:42:32.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.638+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v109'672 uv647 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.638+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d3f40 con 0x5581b302fda0 2024-09-16T01:42:32.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.640+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v109'673 uv673 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.640+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d3f40 con 0x5581b302fda0 2024-09-16T01:42:32.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.641+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v109'674 uv674 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.642 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:42:32.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.641+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d3f40 con 0x5581b302fda0 2024-09-16T01:42:32.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.642+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v109'675 uv675 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.642+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d3f40 con 0x5581b302fda0 2024-09-16T01:42:32.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.644+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v109'676 uv676 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.644+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d3f40 con 0x5581b302fda0 2024-09-16T01:42:32.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.645+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v109'677 uv677 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.645+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d3f40 con 0x5581b302fda0 2024-09-16T01:42:32.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.646+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v109'678 uv678 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.646+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d3f40 con 0x5581b302fda0 2024-09-16T01:42:32.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.648+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v109'679 uv653 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.648+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d3f40 con 0x5581b302fda0 2024-09-16T01:42:32.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.649+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v109'680 uv680 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.649+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d3f40 con 0x5581b302fda0 2024-09-16T01:42:32.651 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:42:32.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.650+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v109'681 uv681 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.650+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d3f40 con 0x5581b302fda0 2024-09-16T01:42:32.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.651+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v109'682 uv682 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.651+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d3f40 con 0x5581b302fda0 2024-09-16T01:42:32.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.653+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v109'683 uv683 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.653+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d3f40 con 0x5581b302fda0 2024-09-16T01:42:32.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.654+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v109'684 uv684 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.654+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d3f40 con 0x5581b302fda0 2024-09-16T01:42:32.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.656+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v109'685 uv659 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.656+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d3f40 con 0x5581b302fda0 2024-09-16T01:42:32.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.657+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v109'686 uv686 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.657+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d3f40 con 0x5581b302fda0 2024-09-16T01:42:32.660 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:42:32.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.659+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v109'687 uv687 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.659+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d3f40 con 0x5581b302fda0 2024-09-16T01:42:32.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.660+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v109'688 uv688 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.660+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d3f40 con 0x5581b302fda0 2024-09-16T01:42:32.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.661+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v109'689 uv689 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.661+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d3f40 con 0x5581b302fda0 2024-09-16T01:42:32.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.662+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v109'690 uv690 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.662+0000 7f1f3d11ef40 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5581b30d5020 0x5581b30d7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:32.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.662+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d7950 con 0x5581b30d5020 2024-09-16T01:42:32.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.663+0000 7f1f3ac14640 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5581b30d5020 0x5581b30d7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:32.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.663+0000 7f1f3ac14640 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5581b30d5020 0x5581b30d7410 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:32.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.665+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v109'485 uv468 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30d5020 2024-09-16T01:42:32.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.665+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d7cf0 con 0x5581b30d5020 2024-09-16T01:42:32.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.666+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v109'486 uv486 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30d5020 2024-09-16T01:42:32.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.666+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d8090 con 0x5581b30d5020 2024-09-16T01:42:32.668 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:42:32.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.667+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v109'487 uv487 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30d5020 2024-09-16T01:42:32.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.667+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d88e0 con 0x5581b30d5020 2024-09-16T01:42:32.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.668+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v109'488 uv488 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30d5020 2024-09-16T01:42:32.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.668+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d8c80 con 0x5581b30d5020 2024-09-16T01:42:32.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.669+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v109'489 uv489 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30d5020 2024-09-16T01:42:32.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.669+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d9020 con 0x5581b30d5020 2024-09-16T01:42:32.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.670+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v109'490 uv473 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30d5020 2024-09-16T01:42:32.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.670+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d93c0 con 0x5581b30d5020 2024-09-16T01:42:32.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.671+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v109'491 uv491 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30d5020 2024-09-16T01:42:32.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.671+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d9760 con 0x5581b30d5020 2024-09-16T01:42:32.674 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:42:32.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.673+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v109'492 uv492 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30d5020 2024-09-16T01:42:32.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.673+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d9760 con 0x5581b30d5020 2024-09-16T01:42:32.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.674+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v109'493 uv493 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30d5020 2024-09-16T01:42:32.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.674+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d9760 con 0x5581b30d5020 2024-09-16T01:42:32.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.675+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v109'494 uv494 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30d5020 2024-09-16T01:42:32.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.675+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d9760 con 0x5581b30d5020 2024-09-16T01:42:32.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.676+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v109'495 uv478 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30d5020 2024-09-16T01:42:32.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.676+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d9760 con 0x5581b30d5020 2024-09-16T01:42:32.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.678+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v109'496 uv496 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30d5020 2024-09-16T01:42:32.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.678+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d9760 con 0x5581b30d5020 2024-09-16T01:42:32.680 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:42:32.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.679+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v109'497 uv497 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30d5020 2024-09-16T01:42:32.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.679+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30ddbe0 con 0x5581b30d5020 2024-09-16T01:42:32.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.680+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v109'498 uv498 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30d5020 2024-09-16T01:42:32.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.680+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30ddbe0 con 0x5581b30d5020 2024-09-16T01:42:32.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.682+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v109'499 uv499 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30d5020 2024-09-16T01:42:32.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.682+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d9760 con 0x5581b30d5020 2024-09-16T01:42:32.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.683+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v109'500 uv484 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30d5020 2024-09-16T01:42:32.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.683+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d9760 con 0x5581b30d5020 2024-09-16T01:42:32.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.684+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v109'501 uv501 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30d5020 2024-09-16T01:42:32.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.684+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d9760 con 0x5581b30d5020 2024-09-16T01:42:32.686 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:42:32.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.685+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v109'502 uv502 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30d5020 2024-09-16T01:42:32.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.685+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30da770 con 0x5581b30d5020 2024-09-16T01:42:32.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.686+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v109'503 uv503 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30d5020 2024-09-16T01:42:32.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.686+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30da770 con 0x5581b30d5020 2024-09-16T01:42:32.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.687+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v109'504 uv504 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30d5020 2024-09-16T01:42:32.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.687+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d9760 con 0x5581b30d5020 2024-09-16T01:42:32.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.689+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v109'505 uv505 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30d5020 2024-09-16T01:42:32.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.689+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d9760 con 0x5581b302fda0 2024-09-16T01:42:32.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.690+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v109'129 uv128 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.690+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d9760 con 0x5581b302fda0 2024-09-16T01:42:32.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.691+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v109'130 uv130 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.691+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d9760 con 0x5581b302fda0 2024-09-16T01:42:32.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.691+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v109'131 uv131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.693 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:42:32.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.692+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30da770 con 0x5581b302fda0 2024-09-16T01:42:32.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.692+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v109'132 uv132 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.692+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30da770 con 0x5581b302fda0 2024-09-16T01:42:32.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.693+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v109'133 uv133 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.693+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d9760 con 0x5581b302fda0 2024-09-16T01:42:32.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.694+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v109'134 uv134 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.694+0000 7f1f3d11ef40 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5581b30d9de0 0x5581b30dc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:32.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.694+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30dc710 con 0x5581b30d9de0 2024-09-16T01:42:32.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.694+0000 7f1f3a413640 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5581b30d9de0 0x5581b30dc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:32.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.694+0000 7f1f3a413640 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5581b30d9de0 0x5581b30dc1d0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:32.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.695+0000 7f1f3a413640 1 -- 172.21.15.88:0/4170766770 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v109'97 uv96 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f1f3001b7c0 con 0x5581b30d9de0 2024-09-16T01:42:32.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.695+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e109) -- 0x5581b30dcc10 con 0x5581b30d9de0 2024-09-16T01:42:32.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.696+0000 7f1f3a413640 1 -- 172.21.15.88:0/4170766770 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v109'98 uv98 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1f3001b7c0 con 0x5581b30d9de0 2024-09-16T01:42:32.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.696+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30dd110 con 0x5581b30d9de0 2024-09-16T01:42:32.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.697+0000 7f1f3a413640 1 -- 172.21.15.88:0/4170766770 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v109'99 uv99 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1f3001b7c0 con 0x5581b30d9de0 2024-09-16T01:42:32.699 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:42:32.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.697+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30d8510 con 0x5581b30d9de0 2024-09-16T01:42:32.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.698+0000 7f1f3a413640 1 -- 172.21.15.88:0/4170766770 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v109'100 uv100 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1f3001b7c0 con 0x5581b30d9de0 2024-09-16T01:42:32.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.698+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30dda40 con 0x5581b302fda0 2024-09-16T01:42:32.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.700+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v109'129 uv128 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.700+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e109) -- 0x5581b30dda40 con 0x5581b302fda0 2024-09-16T01:42:32.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.703+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v109'130 uv130 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.703+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30dda40 con 0x5581b302fda0 2024-09-16T01:42:32.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.705+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v109'131 uv131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.706 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:42:32.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.705+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30dea50 con 0x5581b302fda0 2024-09-16T01:42:32.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.706+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v109'132 uv132 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.706+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30dea50 con 0x5581b302fda0 2024-09-16T01:42:32.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.707+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v109'133 uv133 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.707+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30dda40 con 0x5581b302fda0 2024-09-16T01:42:32.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.708+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v109'134 uv134 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.708+0000 7f1f3d11ef40 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5581b30de110 0x5581b30de510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:32.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.708+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e0ea0 con 0x5581b30de110 2024-09-16T01:42:32.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.708+0000 7f1f3b415640 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5581b30de110 0x5581b30de510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:32.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.709+0000 7f1f3b415640 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5581b30de110 0x5581b30de510 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:32.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.710+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v109'238 uv232 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.710+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e0ea0 con 0x5581b30de110 2024-09-16T01:42:32.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.711+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v109'239 uv239 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.711+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e0ea0 con 0x5581b30de110 2024-09-16T01:42:32.713 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:42:32.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.712+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v109'240 uv240 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.712+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e1eb0 con 0x5581b30de110 2024-09-16T01:42:32.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.713+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v109'241 uv241 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.713+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e1eb0 con 0x5581b30de110 2024-09-16T01:42:32.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.714+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v109'242 uv242 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.714+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e0ea0 con 0x5581b30de110 2024-09-16T01:42:32.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.715+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v109'243 uv243 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.715+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e0ea0 con 0x5581b30de110 2024-09-16T01:42:32.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.716+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v109'244 uv237 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.716+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e0ea0 con 0x5581b30de110 2024-09-16T01:42:32.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.717+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v109'245 uv245 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.717+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e0ea0 con 0x5581b30de110 2024-09-16T01:42:32.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.718+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v109'246 uv246 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.719 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:42:32.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.718+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e0ea0 con 0x5581b30de110 2024-09-16T01:42:32.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.719+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v109'247 uv247 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.719+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e0ea0 con 0x5581b30de110 2024-09-16T01:42:32.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.720+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v109'248 uv248 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.720+0000 7f1f3d11ef40 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5581b30e17c0 0x5581b30e1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:32.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.720+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e42b0 con 0x5581b30e17c0 2024-09-16T01:42:32.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.720+0000 7f1f3ac14640 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5581b30e17c0 0x5581b30e1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:32.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.721+0000 7f1f3ac14640 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5581b30e17c0 0x5581b30e1bc0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:32.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.722+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v109'107 uv106 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30e17c0 2024-09-16T01:42:32.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.722+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e42b0 con 0x5581b30e17c0 2024-09-16T01:42:32.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.723+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v109'108 uv108 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30e17c0 2024-09-16T01:42:32.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.723+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e42b0 con 0x5581b30e17c0 2024-09-16T01:42:32.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.724+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v109'109 uv109 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30e17c0 2024-09-16T01:42:32.725 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:42:32.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.724+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e46d0 con 0x5581b30e17c0 2024-09-16T01:42:32.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.725+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v109'110 uv110 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30e17c0 2024-09-16T01:42:32.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.725+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e46d0 con 0x5581b30e17c0 2024-09-16T01:42:32.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.726+0000 7f1f3ac14640 1 -- 172.21.15.88:0/4170766770 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v109'111 uv111 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1f2c01a030 con 0x5581b30e17c0 2024-09-16T01:42:32.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.726+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e46d0 con 0x5581b302fda0 2024-09-16T01:42:32.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.727+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v109'129 uv128 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.727+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e46d0 con 0x5581b302fda0 2024-09-16T01:42:32.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.728+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v109'130 uv130 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.728+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e46d0 con 0x5581b302fda0 2024-09-16T01:42:32.729 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:42:32.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.728+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v109'131 uv131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.728+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e56e0 con 0x5581b302fda0 2024-09-16T01:42:32.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.729+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v109'132 uv132 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.729+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e56e0 con 0x5581b302fda0 2024-09-16T01:42:32.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.730+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v109'133 uv133 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.730+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e46d0 con 0x5581b302fda0 2024-09-16T01:42:32.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.731+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v109'134 uv134 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b302fda0 2024-09-16T01:42:32.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.731+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e46d0 con 0x5581b30d9de0 2024-09-16T01:42:32.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.732+0000 7f1f3a413640 1 -- 172.21.15.88:0/4170766770 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v109'213 uv207 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f1f3001b7c0 con 0x5581b30d9de0 2024-09-16T01:42:32.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.732+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e4a70 con 0x5581b30d9de0 2024-09-16T01:42:32.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.733+0000 7f1f3a413640 1 -- 172.21.15.88:0/4170766770 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v109'214 uv214 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1f3001b7c0 con 0x5581b30d9de0 2024-09-16T01:42:32.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.733+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e4e10 con 0x5581b30d9de0 2024-09-16T01:42:32.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.734+0000 7f1f3a413640 1 -- 172.21.15.88:0/4170766770 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v109'215 uv215 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1f3001b7c0 con 0x5581b30d9de0 2024-09-16T01:42:32.735 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:42:32.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.734+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e51b0 con 0x5581b30d9de0 2024-09-16T01:42:32.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.735+0000 7f1f3a413640 1 -- 172.21.15.88:0/4170766770 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v109'216 uv216 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1f3001b7c0 con 0x5581b30d9de0 2024-09-16T01:42:32.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.735+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e51b0 con 0x5581b30d9de0 2024-09-16T01:42:32.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.736+0000 7f1f3a413640 1 -- 172.21.15.88:0/4170766770 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v109'217 uv217 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1f3001b7c0 con 0x5581b30d9de0 2024-09-16T01:42:32.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.736+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e51b0 con 0x5581b30d9de0 2024-09-16T01:42:32.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.737+0000 7f1f3a413640 1 -- 172.21.15.88:0/4170766770 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v109'218 uv212 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1f3001b7c0 con 0x5581b30d9de0 2024-09-16T01:42:32.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.737+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e51b0 con 0x5581b30d9de0 2024-09-16T01:42:32.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.738+0000 7f1f3a413640 1 -- 172.21.15.88:0/4170766770 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v109'219 uv219 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3001b7c0 con 0x5581b30d9de0 2024-09-16T01:42:32.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.738+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e51b0 con 0x5581b30d9de0 2024-09-16T01:42:32.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.739+0000 7f1f3a413640 1 -- 172.21.15.88:0/4170766770 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v109'220 uv220 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3001b7c0 con 0x5581b30d9de0 2024-09-16T01:42:32.740 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:42:32.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.739+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e51b0 con 0x5581b30d9de0 2024-09-16T01:42:32.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.740+0000 7f1f3a413640 1 -- 172.21.15.88:0/4170766770 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v109'221 uv221 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3001b7c0 con 0x5581b30d9de0 2024-09-16T01:42:32.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.740+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e7fc0 con 0x5581b30d9de0 2024-09-16T01:42:32.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.741+0000 7f1f3a413640 1 -- 172.21.15.88:0/4170766770 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v109'222 uv222 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3001b7c0 con 0x5581b30d9de0 2024-09-16T01:42:32.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.741+0000 7f1f3d11ef40 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5581b30e5ce0 0x5581b30e8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:32.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.741+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30e5ce0 2024-09-16T01:42:32.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.741+0000 7f1f3a413640 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5581b30e5ce0 0x5581b30e8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:32.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.741+0000 7f1f3a413640 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5581b30e5ce0 0x5581b30e8600 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:32.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.742+0000 7f1f3a413640 1 -- 172.21.15.88:0/4170766770 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v109'95 uv94 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f1f3001b7c0 con 0x5581b30e5ce0 2024-09-16T01:42:32.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.742+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30e5ce0 2024-09-16T01:42:32.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.743+0000 7f1f3a413640 1 -- 172.21.15.88:0/4170766770 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v109'96 uv96 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f1f3001b7c0 con 0x5581b30e5ce0 2024-09-16T01:42:32.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.743+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30e5ce0 2024-09-16T01:42:32.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.744+0000 7f1f3a413640 1 -- 172.21.15.88:0/4170766770 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v109'97 uv97 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f1f3001b7c0 con 0x5581b30e5ce0 2024-09-16T01:42:32.745 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:42:32.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.744+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30e5ce0 2024-09-16T01:42:32.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.745+0000 7f1f3a413640 1 -- 172.21.15.88:0/4170766770 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v109'98 uv98 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f1f3001b7c0 con 0x5581b30e5ce0 2024-09-16T01:42:32.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.745+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30d9de0 2024-09-16T01:42:32.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.746+0000 7f1f3a413640 1 -- 172.21.15.88:0/4170766770 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v109'129 uv128 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1f3001b7c0 con 0x5581b30d9de0 2024-09-16T01:42:32.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.746+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30d9de0 2024-09-16T01:42:32.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.747+0000 7f1f3a413640 1 -- 172.21.15.88:0/4170766770 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v109'130 uv130 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3001b7c0 con 0x5581b30d9de0 2024-09-16T01:42:32.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.747+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30d9de0 2024-09-16T01:42:32.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.748+0000 7f1f3a413640 1 -- 172.21.15.88:0/4170766770 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v109'131 uv131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3001b7c0 con 0x5581b30d9de0 2024-09-16T01:42:32.749 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:42:32.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.748+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30d9de0 2024-09-16T01:42:32.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.749+0000 7f1f3a413640 1 -- 172.21.15.88:0/4170766770 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v109'132 uv132 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3001b7c0 con 0x5581b30d9de0 2024-09-16T01:42:32.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.749+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30d9de0 2024-09-16T01:42:32.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.750+0000 7f1f3a413640 1 -- 172.21.15.88:0/4170766770 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v109'133 uv133 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3001b7c0 con 0x5581b30d9de0 2024-09-16T01:42:32.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.750+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30d9de0 2024-09-16T01:42:32.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.751+0000 7f1f3a413640 1 -- 172.21.15.88:0/4170766770 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v109'134 uv134 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3001b7c0 con 0x5581b30d9de0 2024-09-16T01:42:32.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.751+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30de110 2024-09-16T01:42:32.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.752+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v109'110 uv109 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.752+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30de110 2024-09-16T01:42:32.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.754+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v109'111 uv111 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.754+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30de110 2024-09-16T01:42:32.756 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:42:32.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.755+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v109'112 uv112 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.755+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30de110 2024-09-16T01:42:32.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.756+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v109'113 uv113 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.756+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30de110 2024-09-16T01:42:32.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.757+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v109'114 uv114 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.757+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30de110 2024-09-16T01:42:32.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.759+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v109'265 uv257 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.759+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30de110 2024-09-16T01:42:32.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.760+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v109'266 uv266 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.760+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30de110 2024-09-16T01:42:32.763 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:42:32.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.761+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v109'267 uv267 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.761+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30de110 2024-09-16T01:42:32.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.762+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v109'268 uv268 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.762+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30de110 2024-09-16T01:42:32.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.763+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v109'269 uv269 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.763+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30de110 2024-09-16T01:42:32.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.764+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v109'270 uv264 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.764+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30de110 2024-09-16T01:42:32.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.765+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v109'271 uv271 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.765+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30de110 2024-09-16T01:42:32.767 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:42:32.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.766+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v109'272 uv272 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.766+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30de110 2024-09-16T01:42:32.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.767+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v109'273 uv273 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.767+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30de110 2024-09-16T01:42:32.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.768+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v109'274 uv274 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.768+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e80d0 con 0x5581b30de110 2024-09-16T01:42:32.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.769+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v109'275 uv275 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.770+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x5581b30e8b40 con 0x5581b30de110 2024-09-16T01:42:32.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v109'276 uv276 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1f3404e980 con 0x5581b30de110 2024-09-16T01:42:32.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5581b302fda0 msgr2=0x5581b30c73e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:32.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3d11ef40 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5581b302fda0 0x5581b30c73e0 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5581b30d9de0 msgr2=0x5581b30dc1d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:32.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3d11ef40 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5581b30d9de0 0x5581b30dc1d0 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5581b30e5ce0 msgr2=0x5581b30e8600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:32.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3d11ef40 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5581b30e5ce0 0x5581b30e8600 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5581b30d5020 msgr2=0x5581b30d7410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:32.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3d11ef40 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5581b30d5020 0x5581b30d7410 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5581b30de110 msgr2=0x5581b30de510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:32.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3b415640 1 -- 172.21.15.88:0/4170766770 reap_dead start 2024-09-16T01:42:32.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3d11ef40 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5581b30de110 0x5581b30de510 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5581b30e17c0 msgr2=0x5581b30e1bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:32.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3d11ef40 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5581b30e17c0 0x5581b30e1bc0 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1f2806ac90 msgr2=0x7f1f2806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:32.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3d11ef40 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1f2806ac90 0x7f1f2806d150 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x5581b2f833a0 tx=0x7f1f30008000 comp rx=0 tx=0).stop 2024-09-16T01:42:32.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5581b2f87fe0 msgr2=0x5581b3020280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:32.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3d11ef40 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5581b2f87fe0 0x5581b3020280 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c007e30 tx=0x7f1f2c006750 comp rx=0 tx=0).stop 2024-09-16T01:42:32.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 shutdown_connections 2024-09-16T01:42:32.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3d11ef40 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1f2806ac90 0x7f1f2806d150 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3d11ef40 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5581b30e17c0 0x5581b30e1bc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3d11ef40 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5581b30de110 0x5581b30de510 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3d11ef40 1 --2- 172.21.15.88:0/4170766770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5581b2f87fe0 0x5581b3020280 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 >> 172.21.15.88:0/4170766770 conn(0x5581b2fa3750 msgr2=0x5581b2f87650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:32.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 shutdown_connections 2024-09-16T01:42:32.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.771+0000 7f1f3d11ef40 1 -- 172.21.15.88:0/4170766770 wait complete. 2024-09-16T01:42:32.776 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put 100.00000000.inode - 2024-09-16T01:42:32.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.858+0000 7f38a7c73f40 1 Processor -- start 2024-09-16T01:42:32.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.858+0000 7f38a7c73f40 1 -- start start 2024-09-16T01:42:32.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.859+0000 7f38a7c73f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f2078ee0 0x55b2f2142bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:32.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.859+0000 7f38a7c73f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b2f214cc90 0x55b2f20c01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:32.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.859+0000 7f38a7c73f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b2f214d070 0x55b2f20c27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:32.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.859+0000 7f38a7c73f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b2f1fca510 con 0x55b2f214d070 2024-09-16T01:42:32.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.859+0000 7f38a7c73f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b2f2079840 con 0x55b2f2078ee0 2024-09-16T01:42:32.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.859+0000 7f38a7c73f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55b2f214c4d0 con 0x55b2f214cc90 2024-09-16T01:42:32.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.859+0000 7f38a5f6a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b2f214d070 0x55b2f20c27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:32.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.859+0000 7f38a5f6a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b2f214d070 0x55b2f20c27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40088/0 (socket says 172.21.15.88:40088) 2024-09-16T01:42:32.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.859+0000 7f38a5f6a640 1 -- 172.21.15.88:0/830834693 learned_addr learned my addr 172.21.15.88:0/830834693 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:32.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.859+0000 7f38a4f68640 1 --2- 172.21.15.88:0/830834693 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b2f214cc90 0x55b2f20c01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:32.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.859+0000 7f38a5f6a640 1 -- 172.21.15.88:0/830834693 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b2f214cc90 msgr2=0x55b2f20c01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:32.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.859+0000 7f38a5769640 1 --2- 172.21.15.88:0/830834693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f2078ee0 0x55b2f2142bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:32.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.859+0000 7f38a5f6a640 1 --2- 172.21.15.88:0/830834693 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b2f214cc90 0x55b2f20c01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.859+0000 7f38a5f6a640 1 -- 172.21.15.88:0/830834693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f2078ee0 msgr2=0x55b2f2142bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:32.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.859+0000 7f38a5f6a640 1 --2- 172.21.15.88:0/830834693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f2078ee0 0x55b2f2142bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.859+0000 7f38a5f6a640 1 -- 172.21.15.88:0/830834693 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2f214c260 con 0x55b2f214d070 2024-09-16T01:42:32.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.859+0000 7f38a4f68640 1 --2- 172.21.15.88:0/830834693 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b2f214cc90 0x55b2f20c01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:32.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.860+0000 7f38a5f6a640 1 --2- 172.21.15.88:0/830834693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b2f214d070 0x55b2f20c27b0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f38a0002880 tx=0x7f38a0002d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b972771b2e1cf71 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:32.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.860+0000 7f38977fe640 1 -- 172.21.15.88:0/830834693 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a0016020 con 0x55b2f214d070 2024-09-16T01:42:32.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.860+0000 7f38977fe640 1 -- 172.21.15.88:0/830834693 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38a000d9c0 con 0x55b2f214d070 2024-09-16T01:42:32.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.860+0000 7f38977fe640 1 -- 172.21.15.88:0/830834693 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a000ddb0 con 0x55b2f214d070 2024-09-16T01:42:32.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.860+0000 7f38a7c73f40 1 -- 172.21.15.88:0/830834693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b2f214d070 msgr2=0x55b2f20c27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:32.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.860+0000 7f38a7c73f40 1 --2- 172.21.15.88:0/830834693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b2f214d070 0x55b2f20c27b0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f38a0002880 tx=0x7f38a0002d40 comp rx=0 tx=0).stop 2024-09-16T01:42:32.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.860+0000 7f38a7c73f40 1 -- 172.21.15.88:0/830834693 shutdown_connections 2024-09-16T01:42:32.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.860+0000 7f38a7c73f40 1 --2- 172.21.15.88:0/830834693 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b2f214d070 0x55b2f20c27b0 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.860+0000 7f38a7c73f40 1 --2- 172.21.15.88:0/830834693 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b2f214cc90 0x55b2f20c01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.860+0000 7f38a7c73f40 1 --2- 172.21.15.88:0/830834693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f2078ee0 0x55b2f2142bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.860+0000 7f38a7c73f40 1 -- 172.21.15.88:0/830834693 >> 172.21.15.88:0/830834693 conn(0x55b2f20a3680 msgr2=0x55b2f214bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:32.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.860+0000 7f38a7c73f40 1 -- 172.21.15.88:0/830834693 shutdown_connections 2024-09-16T01:42:32.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.860+0000 7f38a7c73f40 1 -- 172.21.15.88:0/830834693 wait complete. 2024-09-16T01:42:32.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.861+0000 7f38a7c73f40 1 Processor -- start 2024-09-16T01:42:32.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.861+0000 7f38a7c73f40 1 -- start start 2024-09-16T01:42:32.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.861+0000 7f38a7c73f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b2f2078ee0 0x55b2f213afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:32.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.861+0000 7f38a7c73f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f214cc90 0x55b2f21e0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:32.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.861+0000 7f38a7c73f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b2f214d070 0x55b2f20a8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:32.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.861+0000 7f38a7c73f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b2f214b900 con 0x55b2f2078ee0 2024-09-16T01:42:32.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.861+0000 7f38a7c73f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b2f20c1170 con 0x55b2f214cc90 2024-09-16T01:42:32.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.861+0000 7f38a7c73f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55b2f2143c10 con 0x55b2f214d070 2024-09-16T01:42:32.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.862+0000 7f38a5769640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b2f2078ee0 0x55b2f213afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:32.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.862+0000 7f38a5769640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b2f2078ee0 0x55b2f213afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40094/0 (socket says 172.21.15.88:40094) 2024-09-16T01:42:32.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.862+0000 7f38a5769640 1 -- 172.21.15.88:0/1301029919 learned_addr learned my addr 172.21.15.88:0/1301029919 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:32.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.862+0000 7f38a4f68640 1 --2- 172.21.15.88:0/1301029919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f214cc90 0x55b2f21e0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:32.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.862+0000 7f38a5f6a640 1 --2- 172.21.15.88:0/1301029919 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b2f214d070 0x55b2f20a8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:32.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.862+0000 7f38a5769640 1 -- 172.21.15.88:0/1301029919 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b2f214d070 msgr2=0x55b2f20a8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:32.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.862+0000 7f38a5769640 1 --2- 172.21.15.88:0/1301029919 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b2f214d070 0x55b2f20a8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.862+0000 7f38a5769640 1 -- 172.21.15.88:0/1301029919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f214cc90 msgr2=0x55b2f21e0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:32.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.862+0000 7f38a5769640 1 --2- 172.21.15.88:0/1301029919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b2f214cc90 0x55b2f21e0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.862+0000 7f38a5769640 1 -- 172.21.15.88:0/1301029919 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2f2148470 con 0x55b2f2078ee0 2024-09-16T01:42:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.862+0000 7f38a5769640 1 --2- 172.21.15.88:0/1301029919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b2f2078ee0 0x55b2f213afc0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f3898046050 tx=0x7f3898002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.862+0000 7f38967fc640 1 -- 172.21.15.88:0/1301029919 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3898055070 con 0x55b2f2078ee0 2024-09-16T01:42:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.862+0000 7f38967fc640 1 -- 172.21.15.88:0/1301029919 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38980516b0 con 0x55b2f2078ee0 2024-09-16T01:42:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.862+0000 7f38a7c73f40 1 -- 172.21.15.88:0/1301029919 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b2f21de6c0 con 0x55b2f2078ee0 2024-09-16T01:42:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.862+0000 7f38967fc640 1 -- 172.21.15.88:0/1301029919 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3898051a60 con 0x55b2f2078ee0 2024-09-16T01:42:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.862+0000 7f38a7c73f40 1 -- 172.21.15.88:0/1301029919 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2f213ee50 con 0x55b2f2078ee0 2024-09-16T01:42:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.864+0000 7f38967fc640 1 -- 172.21.15.88:0/1301029919 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3898058070 con 0x55b2f2078ee0 2024-09-16T01:42:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.864+0000 7f38967fc640 1 --2- 172.21.15.88:0/1301029919 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f388806ac90 0x7f388806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.864+0000 7f38967fc640 1 -- 172.21.15.88:0/1301029919 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 7630+0+0 (secure 0 0 0) 0x7f38980ce220 con 0x55b2f2078ee0 2024-09-16T01:42:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.864+0000 7f38a4f68640 1 --2- 172.21.15.88:0/1301029919 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f388806ac90 0x7f388806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.864+0000 7f38a7c73f40 1 --2- 172.21.15.88:0/1301029919 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b2f20c0b20 0x55b2f20a88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.864+0000 7f38a7c73f40 1 -- 172.21.15.88:0/1301029919 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:1 12.1f 12:fe07bba1:::100.00000000.inode:head [writefull 0~80 in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) -- 0x55b2f213b500 con 0x55b2f20c0b20 2024-09-16T01:42:32.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.864+0000 7f38a5f6a640 1 --2- 172.21.15.88:0/1301029919 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b2f20c0b20 0x55b2f20a88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:32.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.864+0000 7f38a4f68640 1 --2- 172.21.15.88:0/1301029919 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f388806ac90 0x7f388806d150 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x55b2f21430c0 tx=0x7f389c0017d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:32.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.865+0000 7f38a5f6a640 1 --2- 172.21.15.88:0/1301029919 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b2f20c0b20 0x55b2f20a88b0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:32.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.866+0000 7f38a5f6a640 1 -- 172.21.15.88:0/1301029919 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(1 100.00000000.inode [writefull 0~80] v109'277 uv277 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f38a000dd50 con 0x55b2f20c0b20 2024-09-16T01:42:32.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.866+0000 7f38a7c73f40 1 -- 172.21.15.88:0/1301029919 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b2f20c0b20 msgr2=0x55b2f20a88b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:32.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.866+0000 7f38a7c73f40 1 --2- 172.21.15.88:0/1301029919 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b2f20c0b20 0x55b2f20a88b0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.866+0000 7f38a7c73f40 1 -- 172.21.15.88:0/1301029919 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f388806ac90 msgr2=0x7f388806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:32.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.866+0000 7f38a7c73f40 1 --2- 172.21.15.88:0/1301029919 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f388806ac90 0x7f388806d150 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x55b2f21430c0 tx=0x7f389c0017d0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.866+0000 7f38a7c73f40 1 -- 172.21.15.88:0/1301029919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b2f2078ee0 msgr2=0x55b2f213afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:32.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.866+0000 7f38a7c73f40 1 --2- 172.21.15.88:0/1301029919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b2f2078ee0 0x55b2f213afc0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f3898046050 tx=0x7f3898002fb0 comp rx=0 tx=0).stop 2024-09-16T01:42:32.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.866+0000 7f38a5f6a640 1 -- 172.21.15.88:0/1301029919 reap_dead start 2024-09-16T01:42:32.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.866+0000 7f38a7c73f40 1 -- 172.21.15.88:0/1301029919 shutdown_connections 2024-09-16T01:42:32.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.866+0000 7f38a7c73f40 1 -- 172.21.15.88:0/1301029919 >> 172.21.15.88:0/1301029919 conn(0x55b2f20a3680 msgr2=0x55b2f20a91b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:32.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.867+0000 7f38a7c73f40 1 -- 172.21.15.88:0/1301029919 shutdown_connections 2024-09-16T01:42:32.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:32.867+0000 7f38a7c73f40 1 -- 172.21.15.88:0/1301029919 wait complete. 2024-09-16T01:42:32.874 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:42:33.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.013+0000 7fd1b955f640 1 Processor -- start 2024-09-16T01:42:33.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.013+0000 7fd1b955f640 1 -- start start 2024-09-16T01:42:33.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.014+0000 7fd1b955f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1b415e7a0 0x7fd1b406caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:33.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.014+0000 7fd1b955f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd1b40b68f0 0x7fd1b40d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:33.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.014+0000 7fd1b955f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1b40cf860 con 0x7fd1b40cf1f0 2024-09-16T01:42:33.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.014+0000 7fd1b955f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd1b40d2ca0 con 0x7fd1b40b68f0 2024-09-16T01:42:33.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.014+0000 7fd1b955f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd1b40cfb40 con 0x7fd1b415e7a0 2024-09-16T01:42:33.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.014+0000 7fd1b37fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd1b40b68f0 0x7fd1b40d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:33.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.014+0000 7fd1b27fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd1b40cf1f0 0x7fd1b40b61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50824/0 (socket says 172.21.15.88:50824) 2024-09-16T01:42:33.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.014+0000 7fd1b27fc640 1 -- 172.21.15.88:0/1669251457 learned_addr learned my addr 172.21.15.88:0/1669251457 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:33.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.014+0000 7fd1b2ffd640 1 --2- 172.21.15.88:0/1669251457 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1b415e7a0 0x7fd1b406caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:33.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.015+0000 7fd1b37fe640 1 -- 172.21.15.88:0/1669251457 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1b415e7a0 msgr2=0x7fd1b406caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:33.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.015+0000 7fd1b37fe640 1 --2- 172.21.15.88:0/1669251457 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1b415e7a0 0x7fd1b406caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.015+0000 7fd1b37fe640 1 -- 172.21.15.88:0/1669251457 >> v1:172.21.15.88:6789/0 conn(0x7fd1b40cf1f0 legacy=0x7fd1b40b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:33.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.015+0000 7fd1b37fe640 1 -- 172.21.15.88:0/1669251457 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1b40cfe20 con 0x7fd1b40b68f0 2024-09-16T01:42:33.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.015+0000 7fd1b1ffb640 1 -- 172.21.15.88:0/1669251457 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 508245480 0 0) 0x7fd1b40cf860 con 0x7fd1b40cf1f0 2024-09-16T01:42:33.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.015+0000 7fd1b37fe640 1 --2- 172.21.15.88:0/1669251457 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1b40b68f0 0x7fd1b40d3410 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac0097a0 tx=0x7fd1ac02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78f6e051cc289207 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:33.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.016+0000 7fd1b1ffb640 1 -- 172.21.15.88:0/1669251457 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac037070 con 0x7fd1b40b68f0 2024-09-16T01:42:33.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.016+0000 7fd1b1ffb640 1 -- 172.21.15.88:0/1669251457 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1ac02ae50 con 0x7fd1b40b68f0 2024-09-16T01:42:33.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.016+0000 7fd1b1ffb640 1 -- 172.21.15.88:0/1669251457 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac032660 con 0x7fd1b40b68f0 2024-09-16T01:42:33.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.016+0000 7fd1b955f640 1 -- 172.21.15.88:0/1669251457 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1b40b68f0 msgr2=0x7fd1b40d3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:33.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.016+0000 7fd1b955f640 1 --2- 172.21.15.88:0/1669251457 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1b40b68f0 0x7fd1b40d3410 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac0097a0 tx=0x7fd1ac02a100 comp rx=0 tx=0).stop 2024-09-16T01:42:33.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.016+0000 7fd1b955f640 1 -- 172.21.15.88:0/1669251457 shutdown_connections 2024-09-16T01:42:33.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.016+0000 7fd1b955f640 1 --2- 172.21.15.88:0/1669251457 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1b40b68f0 0x7fd1b40d3410 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.016+0000 7fd1b955f640 1 --2- 172.21.15.88:0/1669251457 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1b415e7a0 0x7fd1b406caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.016+0000 7fd1b955f640 1 -- 172.21.15.88:0/1669251457 >> 172.21.15.88:0/1669251457 conn(0x7fd1b40d11e0 msgr2=0x7fd1b40d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:33.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.016+0000 7fd1b955f640 1 -- 172.21.15.88:0/1669251457 shutdown_connections 2024-09-16T01:42:33.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.016+0000 7fd1b955f640 1 -- 172.21.15.88:0/1669251457 wait complete. 2024-09-16T01:42:33.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.017+0000 7fd1b955f640 1 Processor -- start 2024-09-16T01:42:33.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.018+0000 7fd1b955f640 1 -- start start 2024-09-16T01:42:33.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.018+0000 7fd1b955f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1b40b68f0 0x7fd1b40c9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:33.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.018+0000 7fd1b955f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1b40cf1f0 0x7fd1b40ca900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:33.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.018+0000 7fd1b955f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1b415e7a0 0x7fd1b414e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:33.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.018+0000 7fd1b955f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b406cfe0 con 0x7fd1b415e7a0 2024-09-16T01:42:33.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.018+0000 7fd1b955f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b40b7a00 con 0x7fd1b40cf1f0 2024-09-16T01:42:33.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.018+0000 7fd1b955f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd1b415eb80 con 0x7fd1b40b68f0 2024-09-16T01:42:33.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.018+0000 7fd1b37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1b415e7a0 0x7fd1b414e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:33.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.018+0000 7fd1b37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1b415e7a0 0x7fd1b414e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40116/0 (socket says 172.21.15.88:40116) 2024-09-16T01:42:33.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.018+0000 7fd1b37fe640 1 -- 172.21.15.88:0/786310371 learned_addr learned my addr 172.21.15.88:0/786310371 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:33.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.019+0000 7fd1b2ffd640 1 --2- 172.21.15.88:0/786310371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1b40b68f0 0x7fd1b40c9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:33.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.019+0000 7fd1b27fc640 1 --2- 172.21.15.88:0/786310371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1b40cf1f0 0x7fd1b40ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:33.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.019+0000 7fd1b37fe640 1 -- 172.21.15.88:0/786310371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1b40b68f0 msgr2=0x7fd1b40c9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:33.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.019+0000 7fd1b37fe640 1 --2- 172.21.15.88:0/786310371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1b40b68f0 0x7fd1b40c9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.019+0000 7fd1b37fe640 1 -- 172.21.15.88:0/786310371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1b40cf1f0 msgr2=0x7fd1b40ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:33.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.019+0000 7fd1b37fe640 1 --2- 172.21.15.88:0/786310371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1b40cf1f0 0x7fd1b40ca900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.019+0000 7fd1b37fe640 1 -- 172.21.15.88:0/786310371 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1b40c83f0 con 0x7fd1b415e7a0 2024-09-16T01:42:33.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.019+0000 7fd1b2ffd640 1 --2- 172.21.15.88:0/786310371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1b40b68f0 0x7fd1b40c9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:33.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.019+0000 7fd1b27fc640 1 --2- 172.21.15.88:0/786310371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1b40cf1f0 0x7fd1b40ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:33.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.019+0000 7fd1b37fe640 1 --2- 172.21.15.88:0/786310371 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1b415e7a0 0x7fd1b414e5e0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac035040 tx=0x7fd1ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:33.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.019+0000 7fd19b7fe640 1 -- 172.21.15.88:0/786310371 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac041070 con 0x7fd1b415e7a0 2024-09-16T01:42:33.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.019+0000 7fd19b7fe640 1 -- 172.21.15.88:0/786310371 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1ac03c460 con 0x7fd1b415e7a0 2024-09-16T01:42:33.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.019+0000 7fd1b955f640 1 -- 172.21.15.88:0/786310371 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1b40c86d0 con 0x7fd1b415e7a0 2024-09-16T01:42:33.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.019+0000 7fd19b7fe640 1 -- 172.21.15.88:0/786310371 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac037040 con 0x7fd1b415e7a0 2024-09-16T01:42:33.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.019+0000 7fd1b955f640 1 -- 172.21.15.88:0/786310371 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1b40c8c90 con 0x7fd1b415e7a0 2024-09-16T01:42:33.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.020+0000 7fd1b955f640 1 -- 172.21.15.88:0/786310371 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1b406caa0 con 0x7fd1b415e7a0 2024-09-16T01:42:33.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.021+0000 7fd19b7fe640 1 -- 172.21.15.88:0/786310371 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd1ac044070 con 0x7fd1b415e7a0 2024-09-16T01:42:33.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.021+0000 7fd19b7fe640 1 --2- 172.21.15.88:0/786310371 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd1a006c2b0 0x7fd1a006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:33.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.021+0000 7fd19b7fe640 1 -- 172.21.15.88:0/786310371 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 7630+0+0 (secure 0 0 0) 0x7fd1ac0ac190 con 0x7fd1b415e7a0 2024-09-16T01:42:33.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.021+0000 7fd1b2ffd640 1 --2- 172.21.15.88:0/786310371 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd1a006c2b0 0x7fd1a006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:33.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.021+0000 7fd1b2ffd640 1 --2- 172.21.15.88:0/786310371 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd1a006c2b0 0x7fd1a006e770 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4007440 tx=0x7fd1a400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:33.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.025+0000 7fd19b7fe640 1 -- 172.21.15.88:0/786310371 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd1ac07a180 con 0x7fd1b415e7a0 2024-09-16T01:42:33.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.225+0000 7fd1b955f640 1 -- 172.21.15.88:0/786310371 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fd1b40b4640 con 0x7fd1b415e7a0 2024-09-16T01:42:33.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.267+0000 7fd19b7fe640 1 -- 172.21.15.88:0/786310371 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v203) ==== 162+0+0 (secure 0 0 0) 0x7fd1ac079a80 con 0x7fd1b415e7a0 2024-09-16T01:42:33.268 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:42:33.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.270+0000 7fd1b955f640 1 -- 172.21.15.88:0/786310371 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd1a006c2b0 msgr2=0x7fd1a006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:33.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.270+0000 7fd1b955f640 1 --2- 172.21.15.88:0/786310371 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd1a006c2b0 0x7fd1a006e770 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4007440 tx=0x7fd1a400b040 comp rx=0 tx=0).stop 2024-09-16T01:42:33.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.270+0000 7fd1b955f640 1 -- 172.21.15.88:0/786310371 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1b415e7a0 msgr2=0x7fd1b414e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:33.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.270+0000 7fd1b955f640 1 --2- 172.21.15.88:0/786310371 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1b415e7a0 0x7fd1b414e5e0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac035040 tx=0x7fd1ac0046c0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.270+0000 7fd1b955f640 1 -- 172.21.15.88:0/786310371 shutdown_connections 2024-09-16T01:42:33.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.270+0000 7fd1b955f640 1 --2- 172.21.15.88:0/786310371 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd1a006c2b0 0x7fd1a006e770 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.270+0000 7fd1b955f640 1 --2- 172.21.15.88:0/786310371 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1b415e7a0 0x7fd1b414e5e0 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.270+0000 7fd1b955f640 1 --2- 172.21.15.88:0/786310371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1b40cf1f0 0x7fd1b40ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.270+0000 7fd1b955f640 1 --2- 172.21.15.88:0/786310371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1b40b68f0 0x7fd1b40c9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.270+0000 7fd1b955f640 1 -- 172.21.15.88:0/786310371 >> 172.21.15.88:0/786310371 conn(0x7fd1b40d11e0 msgr2=0x7fd1b40d3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:33.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.270+0000 7fd1b955f640 1 -- 172.21.15.88:0/786310371 shutdown_connections 2024-09-16T01:42:33.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.270+0000 7fd1b955f640 1 -- 172.21.15.88:0/786310371 wait complete. 2024-09-16T01:42:33.276 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:42:33.274+0000 7f9d4d3c0640 -1 mds.0.cache.ino(0x100) Corrupt inode 0x100: End of buffer [buffer:2] 2024-09-16T01:42:33.276 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:42:33.275+0000 7f9d4d3c0640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (22) Invalid argument 2024-09-16T01:42:33.276 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:42:33.275+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:42:33.276 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:42:33.275+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:42:33.284 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:33.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.380+0000 7f6b2f8d7640 1 Processor -- start 2024-09-16T01:42:33.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.381+0000 7f6b2f8d7640 1 -- start start 2024-09-16T01:42:33.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.381+0000 7f6b2f8d7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b2815e660 0x7f6b280cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:33.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.381+0000 7f6b2f8d7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6b280d3310 0x7f6b280d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:33.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.381+0000 7f6b2f8d7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b2815ea40 con 0x7f6b280d1650 2024-09-16T01:42:33.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.382+0000 7f6b2f8d7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6b280d3870 con 0x7f6b280d3310 2024-09-16T01:42:33.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.382+0000 7f6b2f8d7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6b280d3b50 con 0x7f6b2815e660 2024-09-16T01:42:33.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.382+0000 7f6b2ce72640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6b280d3310 0x7f6b280d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:33.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.382+0000 7f6b2ce72640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6b280d3310 0x7f6b280d1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40130/0 (socket says 172.21.15.88:40130) 2024-09-16T01:42:33.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.382+0000 7f6b2ce72640 1 -- 172.21.15.88:0/2265772454 learned_addr learned my addr 172.21.15.88:0/2265772454 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:33.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.382+0000 7f6b2d673640 1 --2- 172.21.15.88:0/2265772454 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b2815e660 0x7f6b280cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:33.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.382+0000 7f6b2ce72640 1 -- 172.21.15.88:0/2265772454 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b2815e660 msgr2=0x7f6b280cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:33.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.382+0000 7f6b2ce72640 1 --2- 172.21.15.88:0/2265772454 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b2815e660 0x7f6b280cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.382+0000 7f6b2ce72640 1 -- 172.21.15.88:0/2265772454 >> v1:172.21.15.154:6789/0 conn(0x7f6b280d1650 legacy=0x7f6b2806bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:33.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.382+0000 7f6b2ce72640 1 -- 172.21.15.88:0/2265772454 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b280d3e30 con 0x7f6b280d3310 2024-09-16T01:42:33.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.383+0000 7f6b2ce72640 1 --2- 172.21.15.88:0/2265772454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b280d3310 0x7f6b280d1110 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f6b24009870 tx=0x7f6b2402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb2a51db62225cd6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:33.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.383+0000 7f6b1ffff640 1 -- 172.21.15.88:0/2265772454 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b24037070 con 0x7f6b280d3310 2024-09-16T01:42:33.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.383+0000 7f6b1ffff640 1 -- 172.21.15.88:0/2265772454 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b24032440 con 0x7f6b280d3310 2024-09-16T01:42:33.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.383+0000 7f6b1ffff640 1 -- 172.21.15.88:0/2265772454 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b24032810 con 0x7f6b280d3310 2024-09-16T01:42:33.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.383+0000 7f6b2f8d7640 1 -- 172.21.15.88:0/2265772454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b280d3310 msgr2=0x7f6b280d1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:33.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.383+0000 7f6b2f8d7640 1 --2- 172.21.15.88:0/2265772454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b280d3310 0x7f6b280d1110 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f6b24009870 tx=0x7f6b2402a270 comp rx=0 tx=0).stop 2024-09-16T01:42:33.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.383+0000 7f6b2f8d7640 1 -- 172.21.15.88:0/2265772454 shutdown_connections 2024-09-16T01:42:33.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.383+0000 7f6b2f8d7640 1 --2- 172.21.15.88:0/2265772454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b280d3310 0x7f6b280d1110 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.383+0000 7f6b2f8d7640 1 --2- 172.21.15.88:0/2265772454 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b2815e660 0x7f6b280cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.383+0000 7f6b2f8d7640 1 -- 172.21.15.88:0/2265772454 >> 172.21.15.88:0/2265772454 conn(0x7f6b280b6410 msgr2=0x7f6b280b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:33.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.384+0000 7f6b2f8d7640 1 -- 172.21.15.88:0/2265772454 shutdown_connections 2024-09-16T01:42:33.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.384+0000 7f6b2f8d7640 1 -- 172.21.15.88:0/2265772454 wait complete. 2024-09-16T01:42:33.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.385+0000 7f6b2f8d7640 1 Processor -- start 2024-09-16T01:42:33.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.385+0000 7f6b2f8d7640 1 -- start start 2024-09-16T01:42:33.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.385+0000 7f6b2f8d7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b280d1650 0x7f6b281f7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:33.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.385+0000 7f6b2f8d7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b280d3310 0x7f6b281f8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:33.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.385+0000 7f6b2f8d7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b2815e660 0x7f6b281f8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:33.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.385+0000 7f6b2f8d7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6b280cfc70 con 0x7f6b280d1650 2024-09-16T01:42:33.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.385+0000 7f6b2f8d7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6b280d43f0 con 0x7f6b280d3310 2024-09-16T01:42:33.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.385+0000 7f6b2f8d7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6b280b8b80 con 0x7f6b2815e660 2024-09-16T01:42:33.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.386+0000 7f6b2d673640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b280d1650 0x7f6b281f7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:33.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.386+0000 7f6b2d673640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b280d1650 0x7f6b281f7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40140/0 (socket says 172.21.15.88:40140) 2024-09-16T01:42:33.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.386+0000 7f6b2d673640 1 -- 172.21.15.88:0/3321784816 learned_addr learned my addr 172.21.15.88:0/3321784816 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:33.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.386+0000 7f6b2ce72640 1 --2- 172.21.15.88:0/3321784816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b280d3310 0x7f6b281f8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:33.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.386+0000 7f6b2de74640 1 --2- 172.21.15.88:0/3321784816 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b2815e660 0x7f6b281f8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:33.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.386+0000 7f6b2d673640 1 -- 172.21.15.88:0/3321784816 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b2815e660 msgr2=0x7f6b281f8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:33.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.386+0000 7f6b2d673640 1 --2- 172.21.15.88:0/3321784816 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b2815e660 0x7f6b281f8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.386+0000 7f6b2d673640 1 -- 172.21.15.88:0/3321784816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b280d3310 msgr2=0x7f6b281f8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:33.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.386+0000 7f6b2d673640 1 --2- 172.21.15.88:0/3321784816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b280d3310 0x7f6b281f8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.386+0000 7f6b2d673640 1 -- 172.21.15.88:0/3321784816 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b28152fe0 con 0x7f6b280d1650 2024-09-16T01:42:33.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.386+0000 7f6b2de74640 1 --2- 172.21.15.88:0/3321784816 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b2815e660 0x7f6b281f8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:33.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.386+0000 7f6b2d673640 1 --2- 172.21.15.88:0/3321784816 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b280d1650 0x7f6b281f7870 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f6b1800e870 tx=0x7f6b1800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:33.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.386+0000 7f6b1dffb640 1 -- 172.21.15.88:0/3321784816 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b18017070 con 0x7f6b280d1650 2024-09-16T01:42:33.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.386+0000 7f6b1dffb640 1 -- 172.21.15.88:0/3321784816 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b1800c8e0 con 0x7f6b280d1650 2024-09-16T01:42:33.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.386+0000 7f6b1dffb640 1 -- 172.21.15.88:0/3321784816 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1800ccd0 con 0x7f6b280d1650 2024-09-16T01:42:33.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.386+0000 7f6b2f8d7640 1 -- 172.21.15.88:0/3321784816 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b281f7db0 con 0x7f6b280d1650 2024-09-16T01:42:33.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.386+0000 7f6b2f8d7640 1 -- 172.21.15.88:0/3321784816 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b28160350 con 0x7f6b280d1650 2024-09-16T01:42:33.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.387+0000 7f6b2f8d7640 1 -- 172.21.15.88:0/3321784816 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b2815fe00 con 0x7f6b280d1650 2024-09-16T01:42:33.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.388+0000 7f6b1dffb640 1 -- 172.21.15.88:0/3321784816 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6b1801a070 con 0x7f6b280d1650 2024-09-16T01:42:33.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.388+0000 7f6b1dffb640 1 --2- 172.21.15.88:0/3321784816 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6b0c06c2b0 0x7f6b0c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:33.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.388+0000 7f6b1dffb640 1 -- 172.21.15.88:0/3321784816 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 7716+0+0 (secure 0 0 0) 0x7f6b1808bcc0 con 0x7f6b280d1650 2024-09-16T01:42:33.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.388+0000 7f6b2ce72640 1 --2- 172.21.15.88:0/3321784816 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6b0c06c2b0 0x7f6b0c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:33.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.389+0000 7f6b2ce72640 1 --2- 172.21.15.88:0/3321784816 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6b0c06c2b0 0x7f6b0c06e770 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f6b24034000 tx=0x7f6b2402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:33.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.393+0000 7f6b1dffb640 1 -- 172.21.15.88:0/3321784816 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6b18059690 con 0x7f6b280d1650 2024-09-16T01:42:33.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.593+0000 7f6b2f8d7640 1 -- 172.21.15.88:0/3321784816 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b280d1110 con 0x7f6b280d1650 2024-09-16T01:42:33.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.594+0000 7f6b1dffb640 1 -- 172.21.15.88:0/3321784816 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) ==== 95+0+3566 (secure 0 0 0) 0x7f6b1805d990 con 0x7f6b280d1650 2024-09-16T01:42:33.596 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:33.596 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":204,"btime":"2024-09-16T01:42:33:276330+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:33.276279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:33.596 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 204 2024-09-16T01:42:33.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.596+0000 7f6b2f8d7640 1 -- 172.21.15.88:0/3321784816 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6b0c06c2b0 msgr2=0x7f6b0c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:33.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.596+0000 7f6b2f8d7640 1 --2- 172.21.15.88:0/3321784816 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6b0c06c2b0 0x7f6b0c06e770 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f6b24034000 tx=0x7f6b2402e000 comp rx=0 tx=0).stop 2024-09-16T01:42:33.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.596+0000 7f6b2f8d7640 1 -- 172.21.15.88:0/3321784816 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b280d1650 msgr2=0x7f6b281f7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:33.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.596+0000 7f6b2f8d7640 1 --2- 172.21.15.88:0/3321784816 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b280d1650 0x7f6b281f7870 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f6b1800e870 tx=0x7f6b1800ed30 comp rx=0 tx=0).stop 2024-09-16T01:42:33.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.596+0000 7f6b2f8d7640 1 -- 172.21.15.88:0/3321784816 shutdown_connections 2024-09-16T01:42:33.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.596+0000 7f6b2f8d7640 1 --2- 172.21.15.88:0/3321784816 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6b0c06c2b0 0x7f6b0c06e770 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.596+0000 7f6b2f8d7640 1 --2- 172.21.15.88:0/3321784816 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b2815e660 0x7f6b281f8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.596+0000 7f6b2f8d7640 1 --2- 172.21.15.88:0/3321784816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b280d3310 0x7f6b281f8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.596+0000 7f6b2f8d7640 1 --2- 172.21.15.88:0/3321784816 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b280d1650 0x7f6b281f7870 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.596+0000 7f6b2f8d7640 1 -- 172.21.15.88:0/3321784816 >> 172.21.15.88:0/3321784816 conn(0x7f6b280b6410 msgr2=0x7f6b280cadd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:33.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.596+0000 7f6b2f8d7640 1 -- 172.21.15.88:0/3321784816 shutdown_connections 2024-09-16T01:42:33.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.596+0000 7f6b2f8d7640 1 -- 172.21.15.88:0/3321784816 wait complete. 2024-09-16T01:42:33.608 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:42:33.608 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:33.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.705+0000 7f2528d86640 1 Processor -- start 2024-09-16T01:42:33.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.705+0000 7f2528d86640 1 -- start start 2024-09-16T01:42:33.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.706+0000 7f2528d86640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f252415e560 0x7f252406e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:33.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.706+0000 7f2528d86640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2524073130 0x7f2524073580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:33.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.706+0000 7f2528d86640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25240b5220 con 0x7f2524072220 2024-09-16T01:42:33.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.706+0000 7f2528d86640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f25240aefa0 con 0x7f252415e560 2024-09-16T01:42:33.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.706+0000 7f2528d86640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2524073ac0 con 0x7f2524073130 2024-09-16T01:42:33.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.706+0000 7f252259c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f252415e560 0x7f252406e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:33.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.706+0000 7f252259c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f252415e560 0x7f252406e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40156/0 (socket says 172.21.15.88:40156) 2024-09-16T01:42:33.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.706+0000 7f252259c640 1 -- 172.21.15.88:0/187595532 learned_addr learned my addr 172.21.15.88:0/187595532 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:33.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.706+0000 7f2521d9b640 1 --1- 172.21.15.88:0/187595532 >> v1:172.21.15.88:6789/0 conn(0x7f2524072220 0x7f25240ae890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50826/0 (socket says 172.21.15.88:50826) 2024-09-16T01:42:33.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.706+0000 7f2522d9d640 1 --2- 172.21.15.88:0/187595532 >> v2:172.21.15.154:3300/0 conn(0x7f2524073130 0x7f2524073580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:33.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.706+0000 7f252259c640 1 -- 172.21.15.88:0/187595532 >> v2:172.21.15.154:3300/0 conn(0x7f2524073130 msgr2=0x7f2524073580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:33.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.706+0000 7f252259c640 1 --2- 172.21.15.88:0/187595532 >> v2:172.21.15.154:3300/0 conn(0x7f2524073130 0x7f2524073580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.706+0000 7f252159a640 1 -- 172.21.15.88:0/187595532 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4084162241 0 0) 0x7f25240b5220 con 0x7f2524072220 2024-09-16T01:42:33.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.706+0000 7f252259c640 1 -- 172.21.15.88:0/187595532 >> v1:172.21.15.88:6789/0 conn(0x7f2524072220 legacy=0x7f25240ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:33.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.706+0000 7f252259c640 1 -- 172.21.15.88:0/187595532 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25240b54c0 con 0x7f252415e560 2024-09-16T01:42:33.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.707+0000 7f252259c640 1 --2- 172.21.15.88:0/187595532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f252415e560 0x7f252406e780 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f25240b44f0 tx=0x7f251c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4825b97ea8d4aaab server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:33.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.707+0000 7f252159a640 1 -- 172.21.15.88:0/187595532 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251c037070 con 0x7f252415e560 2024-09-16T01:42:33.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.707+0000 7f252159a640 1 -- 172.21.15.88:0/187595532 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f251c032440 con 0x7f252415e560 2024-09-16T01:42:33.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.707+0000 7f252159a640 1 -- 172.21.15.88:0/187595532 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251c032810 con 0x7f252415e560 2024-09-16T01:42:33.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.708+0000 7f2528d86640 1 -- 172.21.15.88:0/187595532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f252415e560 msgr2=0x7f252406e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:33.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.708+0000 7f2528d86640 1 --2- 172.21.15.88:0/187595532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f252415e560 0x7f252406e780 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f25240b44f0 tx=0x7f251c02a1e0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.708+0000 7f2528d86640 1 -- 172.21.15.88:0/187595532 shutdown_connections 2024-09-16T01:42:33.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.708+0000 7f2528d86640 1 --2- 172.21.15.88:0/187595532 >> v2:172.21.15.154:3300/0 conn(0x7f2524073130 0x7f2524073580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.708+0000 7f2528d86640 1 --2- 172.21.15.88:0/187595532 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f252415e560 0x7f252406e780 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.708+0000 7f2528d86640 1 -- 172.21.15.88:0/187595532 >> 172.21.15.88:0/187595532 conn(0x7f252406bc10 msgr2=0x7f252406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:33.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.708+0000 7f2528d86640 1 -- 172.21.15.88:0/187595532 shutdown_connections 2024-09-16T01:42:33.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.708+0000 7f2528d86640 1 -- 172.21.15.88:0/187595532 wait complete. 2024-09-16T01:42:33.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.709+0000 7f2528d86640 1 Processor -- start 2024-09-16T01:42:33.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.709+0000 7f2528d86640 1 -- start start 2024-09-16T01:42:33.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.710+0000 7f2528d86640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2524072220 0x7f252415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:33.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.710+0000 7f2528d86640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2524073130 0x7f2524154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:33.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.710+0000 7f2528d86640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f252415e560 0x7f2524160d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:33.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.710+0000 7f2528d86640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f25240b5d60 con 0x7f2524072220 2024-09-16T01:42:33.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.710+0000 7f2528d86640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f25240b5a80 con 0x7f252415e560 2024-09-16T01:42:33.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.710+0000 7f2528d86640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2524072660 con 0x7f2524073130 2024-09-16T01:42:33.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.710+0000 7f252259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2524072220 0x7f252415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:33.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.710+0000 7f252259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2524072220 0x7f252415a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40168/0 (socket says 172.21.15.88:40168) 2024-09-16T01:42:33.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.710+0000 7f252259c640 1 -- 172.21.15.88:0/3529611311 learned_addr learned my addr 172.21.15.88:0/3529611311 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:33.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.710+0000 7f252259c640 1 -- 172.21.15.88:0/3529611311 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2524073130 msgr2=0x7f2524154100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:33.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.710+0000 7f252259c640 1 --2- 172.21.15.88:0/3529611311 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2524073130 0x7f2524154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.710+0000 7f252259c640 1 -- 172.21.15.88:0/3529611311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f252415e560 msgr2=0x7f2524160d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:33.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.710+0000 7f252259c640 1 --2- 172.21.15.88:0/3529611311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f252415e560 0x7f2524160d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.710+0000 7f252259c640 1 -- 172.21.15.88:0/3529611311 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f252415a730 con 0x7f2524072220 2024-09-16T01:42:33.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.710+0000 7f252259c640 1 --2- 172.21.15.88:0/3529611311 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2524072220 0x7f252415a9b0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f251c000c00 tx=0x7f251c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:33.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.711+0000 7f251affd640 1 -- 172.21.15.88:0/3529611311 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251c041020 con 0x7f2524072220 2024-09-16T01:42:33.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.711+0000 7f251affd640 1 -- 172.21.15.88:0/3529611311 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f251c0306c0 con 0x7f2524072220 2024-09-16T01:42:33.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.711+0000 7f251affd640 1 -- 172.21.15.88:0/3529611311 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251c037040 con 0x7f2524072220 2024-09-16T01:42:33.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.711+0000 7f2528d86640 1 -- 172.21.15.88:0/3529611311 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f252415b020 con 0x7f2524072220 2024-09-16T01:42:33.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.711+0000 7f2528d86640 1 -- 172.21.15.88:0/3529611311 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f252415b5e0 con 0x7f2524072220 2024-09-16T01:42:33.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.712+0000 7f2528d86640 1 -- 172.21.15.88:0/3529611311 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f252415b920 con 0x7f2524072220 2024-09-16T01:42:33.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.712+0000 7f251affd640 1 -- 172.21.15.88:0/3529611311 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f251c044020 con 0x7f2524072220 2024-09-16T01:42:33.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.713+0000 7f251affd640 1 --2- 172.21.15.88:0/3529611311 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f250406c2b0 0x7f250406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:33.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.713+0000 7f2521d9b640 1 --2- 172.21.15.88:0/3529611311 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f250406c2b0 0x7f250406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:33.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.713+0000 7f251affd640 1 -- 172.21.15.88:0/3529611311 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 7716+0+0 (secure 0 0 0) 0x7f251c0ac3c0 con 0x7f2524072220 2024-09-16T01:42:33.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.713+0000 7f2521d9b640 1 --2- 172.21.15.88:0/3529611311 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f250406c2b0 0x7f250406e770 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f250c007fd0 tx=0x7f250c007920 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:33.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.717+0000 7f251affd640 1 -- 172.21.15.88:0/3529611311 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f252415b920 con 0x7f2524072220 2024-09-16T01:42:33.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.915+0000 7f2528d86640 1 -- 172.21.15.88:0/3529611311 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25240b6590 con 0x7f2524072220 2024-09-16T01:42:33.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.915+0000 7f251affd640 1 -- 172.21.15.88:0/3529611311 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) ==== 95+0+3566 (secure 0 0 0) 0x7f251c04b030 con 0x7f2524072220 2024-09-16T01:42:33.917 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:33.917 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":204,"btime":"2024-09-16T01:42:33:276330+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:33.276279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:33.918 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 204 2024-09-16T01:42:33.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.917+0000 7f2528d86640 1 -- 172.21.15.88:0/3529611311 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f250406c2b0 msgr2=0x7f250406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:33.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.917+0000 7f2528d86640 1 --2- 172.21.15.88:0/3529611311 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f250406c2b0 0x7f250406e770 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f250c007fd0 tx=0x7f250c007920 comp rx=0 tx=0).stop 2024-09-16T01:42:33.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.917+0000 7f2528d86640 1 -- 172.21.15.88:0/3529611311 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2524072220 msgr2=0x7f252415a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:33.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.917+0000 7f2528d86640 1 --2- 172.21.15.88:0/3529611311 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2524072220 0x7f252415a9b0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f251c000c00 tx=0x7f251c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.917+0000 7f2528d86640 1 -- 172.21.15.88:0/3529611311 shutdown_connections 2024-09-16T01:42:33.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.917+0000 7f2528d86640 1 --2- 172.21.15.88:0/3529611311 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f250406c2b0 0x7f250406e770 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.917+0000 7f2528d86640 1 --2- 172.21.15.88:0/3529611311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f252415e560 0x7f2524160d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.917+0000 7f2528d86640 1 --2- 172.21.15.88:0/3529611311 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2524073130 0x7f2524154100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.917+0000 7f2528d86640 1 --2- 172.21.15.88:0/3529611311 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2524072220 0x7f252415a9b0 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:33.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.917+0000 7f2528d86640 1 -- 172.21.15.88:0/3529611311 >> 172.21.15.88:0/3529611311 conn(0x7f252406bc10 msgr2=0x7f25240b1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:33.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.917+0000 7f2528d86640 1 -- 172.21.15.88:0/3529611311 shutdown_connections 2024-09-16T01:42:33.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:33.917+0000 7f2528d86640 1 -- 172.21.15.88:0/3529611311 wait complete. 2024-09-16T01:42:33.929 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:42:33.930 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:34.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.026+0000 7f60eed2a640 1 Processor -- start 2024-09-16T01:42:34.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.026+0000 7f60eed2a640 1 -- start start 2024-09-16T01:42:34.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.027+0000 7f60eed2a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e80b5be0 0x7f60e815ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:34.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.027+0000 7f60eed2a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f60e80b29a0 0x7f60e80cde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:34.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.027+0000 7f60eed2a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60e80b6830 con 0x7f60e815a9c0 2024-09-16T01:42:34.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.027+0000 7f60eed2a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f60e815f410 con 0x7f60e80b29a0 2024-09-16T01:42:34.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.027+0000 7f60eed2a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f60e80cd0b0 con 0x7f60e80b5be0 2024-09-16T01:42:34.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.027+0000 7f60ed2c7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f60e80b29a0 0x7f60e80cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:34.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.027+0000 7f60ecac6640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f60e815a9c0 0x7f60e815e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50840/0 (socket says 172.21.15.88:50840) 2024-09-16T01:42:34.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.027+0000 7f60ed2c7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f60e80b29a0 0x7f60e80cde70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40172/0 (socket says 172.21.15.88:40172) 2024-09-16T01:42:34.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.027+0000 7f60ecac6640 1 -- 172.21.15.88:0/1607997037 learned_addr learned my addr 172.21.15.88:0/1607997037 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:34.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.027+0000 7f60e7fff640 1 --2- 172.21.15.88:0/1607997037 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e80b5be0 0x7f60e815ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:34.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.027+0000 7f60ed2c7640 1 -- 172.21.15.88:0/1607997037 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e80b5be0 msgr2=0x7f60e815ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:34.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.027+0000 7f60ed2c7640 1 --2- 172.21.15.88:0/1607997037 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e80b5be0 0x7f60e815ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:34.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.027+0000 7f60ed2c7640 1 -- 172.21.15.88:0/1607997037 >> v1:172.21.15.88:6789/0 conn(0x7f60e815a9c0 legacy=0x7f60e815e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:34.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.027+0000 7f60ed2c7640 1 -- 172.21.15.88:0/1607997037 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60e80cd390 con 0x7f60e80b29a0 2024-09-16T01:42:34.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.027+0000 7f60e77fe640 1 -- 172.21.15.88:0/1607997037 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3059904137 0 0) 0x7f60e80b6830 con 0x7f60e815a9c0 2024-09-16T01:42:34.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.028+0000 7f60ed2c7640 1 --2- 172.21.15.88:0/1607997037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60e80b29a0 0x7f60e80cde70 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f60e000aea0 tx=0x7f60e002b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58457d8a93558aad server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:34.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.028+0000 7f60e77fe640 1 -- 172.21.15.88:0/1607997037 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e0037070 con 0x7f60e80b29a0 2024-09-16T01:42:34.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.028+0000 7f60e77fe640 1 -- 172.21.15.88:0/1607997037 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60e0008640 con 0x7f60e80b29a0 2024-09-16T01:42:34.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.028+0000 7f60e77fe640 1 -- 172.21.15.88:0/1607997037 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e0008a10 con 0x7f60e80b29a0 2024-09-16T01:42:34.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.029+0000 7f60eed2a640 1 -- 172.21.15.88:0/1607997037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60e80b29a0 msgr2=0x7f60e80cde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:34.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.029+0000 7f60eed2a640 1 --2- 172.21.15.88:0/1607997037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60e80b29a0 0x7f60e80cde70 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f60e000aea0 tx=0x7f60e002b7b0 comp rx=0 tx=0).stop 2024-09-16T01:42:34.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.029+0000 7f60eed2a640 1 -- 172.21.15.88:0/1607997037 shutdown_connections 2024-09-16T01:42:34.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.029+0000 7f60eed2a640 1 --2- 172.21.15.88:0/1607997037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60e80b29a0 0x7f60e80cde70 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:34.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.029+0000 7f60eed2a640 1 --2- 172.21.15.88:0/1607997037 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e80b5be0 0x7f60e815ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:34.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.029+0000 7f60eed2a640 1 -- 172.21.15.88:0/1607997037 >> 172.21.15.88:0/1607997037 conn(0x7f60e806bc10 msgr2=0x7f60e806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:34.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.029+0000 7f60eed2a640 1 -- 172.21.15.88:0/1607997037 shutdown_connections 2024-09-16T01:42:34.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.029+0000 7f60eed2a640 1 -- 172.21.15.88:0/1607997037 wait complete. 2024-09-16T01:42:34.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.030+0000 7f60eed2a640 1 Processor -- start 2024-09-16T01:42:34.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.030+0000 7f60eed2a640 1 -- start start 2024-09-16T01:42:34.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.031+0000 7f60eed2a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60e80b29a0 0x7f60e814ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:34.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.031+0000 7f60eed2a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e80b5be0 0x7f60e814de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:34.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.031+0000 7f60eed2a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60e815a9c0 0x7f60e815d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:34.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.031+0000 7f60eed2a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f60e815ada0 con 0x7f60e80b29a0 2024-09-16T01:42:34.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.031+0000 7f60eed2a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f60e80cd950 con 0x7f60e815a9c0 2024-09-16T01:42:34.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.031+0000 7f60eed2a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f60e806de40 con 0x7f60e80b5be0 2024-09-16T01:42:34.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.031+0000 7f60ecac6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60e80b29a0 0x7f60e814ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:34.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.031+0000 7f60ecac6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60e80b29a0 0x7f60e814ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40184/0 (socket says 172.21.15.88:40184) 2024-09-16T01:42:34.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.031+0000 7f60ecac6640 1 -- 172.21.15.88:0/1449531168 learned_addr learned my addr 172.21.15.88:0/1449531168 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:34.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.031+0000 7f60ed2c7640 1 --2- 172.21.15.88:0/1449531168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60e815a9c0 0x7f60e815d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:34.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.031+0000 7f60e7fff640 1 --2- 172.21.15.88:0/1449531168 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e80b5be0 0x7f60e814de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:34.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.031+0000 7f60ecac6640 1 -- 172.21.15.88:0/1449531168 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e80b5be0 msgr2=0x7f60e814de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:34.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.031+0000 7f60ecac6640 1 --2- 172.21.15.88:0/1449531168 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e80b5be0 0x7f60e814de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.031+0000 7f60ecac6640 1 -- 172.21.15.88:0/1449531168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60e815a9c0 msgr2=0x7f60e815d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.031+0000 7f60ecac6640 1 --2- 172.21.15.88:0/1449531168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60e815a9c0 0x7f60e815d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.031+0000 7f60ecac6640 1 -- 172.21.15.88:0/1449531168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60e814b270 con 0x7f60e80b29a0 2024-09-16T01:42:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.031+0000 7f60ecac6640 1 --2- 172.21.15.88:0/1449531168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60e80b29a0 0x7f60e814ad30 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00c430 tx=0x7f60dc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.032+0000 7f60e57fa640 1 -- 172.21.15.88:0/1449531168 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60dc0074e0 con 0x7f60e80b29a0 2024-09-16T01:42:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.032+0000 7f60e57fa640 1 -- 172.21.15.88:0/1449531168 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60dc0047e0 con 0x7f60e80b29a0 2024-09-16T01:42:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.032+0000 7f60e57fa640 1 -- 172.21.15.88:0/1449531168 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60dc004bd0 con 0x7f60e80b29a0 2024-09-16T01:42:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.032+0000 7f60eed2a640 1 -- 172.21.15.88:0/1449531168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60e81f60d0 con 0x7f60e80b29a0 2024-09-16T01:42:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.032+0000 7f60eed2a640 1 -- 172.21.15.88:0/1449531168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60e814d640 con 0x7f60e80b29a0 2024-09-16T01:42:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.033+0000 7f60eed2a640 1 -- 172.21.15.88:0/1449531168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60e806e1f0 con 0x7f60e80b29a0 2024-09-16T01:42:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.033+0000 7f60e57fa640 1 -- 172.21.15.88:0/1449531168 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f60dc011070 con 0x7f60e80b29a0 2024-09-16T01:42:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.033+0000 7f60e57fa640 1 --2- 172.21.15.88:0/1449531168 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f60cc06c2b0 0x7f60cc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:34.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.034+0000 7f60e7fff640 1 --2- 172.21.15.88:0/1449531168 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f60cc06c2b0 0x7f60cc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:34.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.034+0000 7f60e57fa640 1 -- 172.21.15.88:0/1449531168 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 7716+0+0 (secure 0 0 0) 0x7f60dc08b7f0 con 0x7f60e80b29a0 2024-09-16T01:42:34.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.034+0000 7f60e7fff640 1 --2- 172.21.15.88:0/1449531168 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f60cc06c2b0 0x7f60cc06e770 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f60e8152310 tx=0x7f60d4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:34.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.038+0000 7f60e57fa640 1 -- 172.21.15.88:0/1449531168 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f60dc059810 con 0x7f60e80b29a0 2024-09-16T01:42:34.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.238+0000 7f60eed2a640 1 -- 172.21.15.88:0/1449531168 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60e8154460 con 0x7f60e80b29a0 2024-09-16T01:42:34.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.239+0000 7f60e57fa640 1 -- 172.21.15.88:0/1449531168 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) ==== 95+0+3566 (secure 0 0 0) 0x7f60dc059110 con 0x7f60e80b29a0 2024-09-16T01:42:34.240 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:34.240 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":204,"btime":"2024-09-16T01:42:33:276330+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:33.276279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:34.241 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 204 2024-09-16T01:42:34.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.241+0000 7f60eed2a640 1 -- 172.21.15.88:0/1449531168 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f60cc06c2b0 msgr2=0x7f60cc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:34.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.241+0000 7f60eed2a640 1 --2- 172.21.15.88:0/1449531168 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f60cc06c2b0 0x7f60cc06e770 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f60e8152310 tx=0x7f60d4008040 comp rx=0 tx=0).stop 2024-09-16T01:42:34.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.241+0000 7f60eed2a640 1 -- 172.21.15.88:0/1449531168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60e80b29a0 msgr2=0x7f60e814ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:34.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.241+0000 7f60eed2a640 1 --2- 172.21.15.88:0/1449531168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60e80b29a0 0x7f60e814ad30 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00c430 tx=0x7f60dc00c8f0 comp rx=0 tx=0).stop 2024-09-16T01:42:34.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.241+0000 7f60eed2a640 1 -- 172.21.15.88:0/1449531168 shutdown_connections 2024-09-16T01:42:34.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.241+0000 7f60eed2a640 1 --2- 172.21.15.88:0/1449531168 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f60cc06c2b0 0x7f60cc06e770 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:34.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.241+0000 7f60eed2a640 1 --2- 172.21.15.88:0/1449531168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60e815a9c0 0x7f60e815d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:34.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.241+0000 7f60eed2a640 1 --2- 172.21.15.88:0/1449531168 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f60e80b5be0 0x7f60e814de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:34.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.241+0000 7f60eed2a640 1 --2- 172.21.15.88:0/1449531168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f60e80b29a0 0x7f60e814ad30 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:34.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.241+0000 7f60eed2a640 1 -- 172.21.15.88:0/1449531168 >> 172.21.15.88:0/1449531168 conn(0x7f60e806bc10 msgr2=0x7f60e80ce4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:34.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.241+0000 7f60eed2a640 1 -- 172.21.15.88:0/1449531168 shutdown_connections 2024-09-16T01:42:34.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.241+0000 7f60eed2a640 1 -- 172.21.15.88:0/1449531168 wait complete. 2024-09-16T01:42:34.253 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt 100.00000000.inode' led to DAMAGED state 2024-09-16T01:42:34.253 INFO:tasks.cephfs.test_damage:Applying mutation 'Truncate 1.00000000.inode' 2024-09-16T01:42:34.254 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:42:34.254 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:42:34.269 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:42:34.269 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:42:34.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:34.270 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:42:34.293 INFO:tasks.ceph.mds.c.smithi088.stderr: -46> 2024-09-16T01:42:29.540+0000 7f9385717640 -1 log_channel(cluster) log [ERR] : failed to read JournalPointer: -22 ((22) Invalid argument) 2024-09-16T01:42:34.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:42:34.307 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:42:34.308 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:42:34.316 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:42:34.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.413+0000 7f5013dcf640 1 Processor -- start 2024-09-16T01:42:34.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.413+0000 7f5013dcf640 1 -- start start 2024-09-16T01:42:34.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.413+0000 7f5013dcf640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f500c0cf1f0 0x7f500c0b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:34.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.414+0000 7f5013dcf640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f500c0cf860 con 0x7f500c0b6720 2024-09-16T01:42:34.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.414+0000 7f5013dcf640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f500c0cfb40 con 0x7f500c15e7a0 2024-09-16T01:42:34.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.414+0000 7f5013dcf640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f500c0d2ca0 con 0x7f500c0cf1f0 2024-09-16T01:42:34.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.414+0000 7f501236c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f500c0b6720 0x7f500c0d3410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50846/0 (socket says 172.21.15.88:50846) 2024-09-16T01:42:34.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.414+0000 7f501236c640 1 -- 172.21.15.88:0/2064254116 learned_addr learned my addr 172.21.15.88:0/2064254116 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:34.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.414+0000 7f5010b69640 1 -- 172.21.15.88:0/2064254116 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3253898401 0 0) 0x7f500c0cf860 con 0x7f500c0b6720 2024-09-16T01:42:34.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.414+0000 7f5010b69640 1 -- 172.21.15.88:0/2064254116 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ff4003520 con 0x7f500c0b6720 2024-09-16T01:42:34.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.414+0000 7f501136a640 1 --2- 172.21.15.88:0/2064254116 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f500c0cf1f0 0x7f500c0b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:34.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.414+0000 7f5010b69640 1 -- 172.21.15.88:0/2064254116 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3928453949 0 0) 0x7f4ff4003520 con 0x7f500c0b6720 2024-09-16T01:42:34.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.414+0000 7f5010b69640 1 -- 172.21.15.88:0/2064254116 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f500c0cf1f0 msgr2=0x7f500c0b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:34.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.414+0000 7f5010b69640 1 --2- 172.21.15.88:0/2064254116 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f500c0cf1f0 0x7f500c0b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:34.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.414+0000 7f5010b69640 1 -- 172.21.15.88:0/2064254116 >> v1:172.21.15.154:6789/0 conn(0x7f500c15e7a0 legacy=0x7f500c06caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:34.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.414+0000 7f5010b69640 1 -- 172.21.15.88:0/2064254116 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f500c0cfe20 con 0x7f500c0b6720 2024-09-16T01:42:34.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.414+0000 7f5010b69640 1 -- 172.21.15.88:0/2064254116 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f4ffc001670 con 0x7f500c0b6720 2024-09-16T01:42:34.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.414+0000 7f501136a640 1 --2- 172.21.15.88:0/2064254116 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f500c0cf1f0 0x7f500c0b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:34.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.414+0000 7f5010b69640 1 -- 172.21.15.88:0/2064254116 >> v1:172.21.15.88:6789/0 conn(0x7f500c0b6720 legacy=0x7f500c0d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:34.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.414+0000 7f5010b69640 1 --2- 172.21.15.88:0/2064254116 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f500c0cf1f0 0x7f4ff40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:34.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.414+0000 7f5010b69640 1 --2- 172.21.15.88:0/2064254116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ff4003fd0 0x7f4ff4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:34.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.414+0000 7f5010b69640 1 --2- 172.21.15.88:0/2064254116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ff4004d40 0x7f4ff40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:34.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.414+0000 7f5010b69640 1 -- 172.21.15.88:0/2064254116 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f500c0d2ca0 con 0x7f4ff4004d40 2024-09-16T01:42:34.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.414+0000 7f5010b69640 1 -- 172.21.15.88:0/2064254116 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff400bbb0 con 0x7f4ff4003fd0 2024-09-16T01:42:34.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.414+0000 7f5010b69640 1 -- 172.21.15.88:0/2064254116 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4ff400bdb0 con 0x7f500c0cf1f0 2024-09-16T01:42:34.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.414+0000 7f501236c640 1 --2- 172.21.15.88:0/2064254116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ff4004d40 0x7f4ff40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:34.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.414+0000 7f5011b6b640 1 --2- 172.21.15.88:0/2064254116 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f500c0cf1f0 0x7f4ff40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:34.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.414+0000 7f501136a640 1 --2- 172.21.15.88:0/2064254116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ff4003fd0 0x7f4ff4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:34.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.415+0000 7f501236c640 1 -- 172.21.15.88:0/2064254116 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f500c0cf1f0 msgr2=0x7f4ff40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:34.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.415+0000 7f501236c640 1 --2- 172.21.15.88:0/2064254116 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f500c0cf1f0 0x7f4ff40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:34.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.415+0000 7f501236c640 1 -- 172.21.15.88:0/2064254116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ff4003fd0 msgr2=0x7f4ff4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:34.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.415+0000 7f501236c640 1 --2- 172.21.15.88:0/2064254116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ff4003fd0 0x7f4ff4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:34.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.415+0000 7f501236c640 1 -- 172.21.15.88:0/2064254116 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ff4003520 con 0x7f4ff4004d40 2024-09-16T01:42:34.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.415+0000 7f501236c640 1 --2- 172.21.15.88:0/2064254116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ff4004d40 0x7f4ff40051c0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc00c210 tx=0x7f4ff4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b383197179ed34a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:34.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.416+0000 7f5010b69640 1 -- 172.21.15.88:0/2064254116 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc007d30 con 0x7f4ff4004d40 2024-09-16T01:42:34.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.416+0000 7f5010b69640 1 -- 172.21.15.88:0/2064254116 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ffc005ad0 con 0x7f4ff4004d40 2024-09-16T01:42:34.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.416+0000 7f5010b69640 1 -- 172.21.15.88:0/2064254116 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc007070 con 0x7f4ff4004d40 2024-09-16T01:42:34.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.416+0000 7f5013dcf640 1 -- 172.21.15.88:0/2064254116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ff4004d40 msgr2=0x7f4ff40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:34.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.416+0000 7f5013dcf640 1 --2- 172.21.15.88:0/2064254116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ff4004d40 0x7f4ff40051c0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc00c210 tx=0x7f4ff4005830 comp rx=0 tx=0).stop 2024-09-16T01:42:34.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.416+0000 7f501236c640 1 -- 172.21.15.88:0/2064254116 reap_dead start 2024-09-16T01:42:34.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.416+0000 7f5013dcf640 1 -- 172.21.15.88:0/2064254116 shutdown_connections 2024-09-16T01:42:34.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.416+0000 7f5013dcf640 1 -- 172.21.15.88:0/2064254116 >> 172.21.15.88:0/2064254116 conn(0x7f500c0d11e0 msgr2=0x7f500c0d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:34.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.416+0000 7f5013dcf640 1 -- 172.21.15.88:0/2064254116 shutdown_connections 2024-09-16T01:42:34.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.416+0000 7f5013dcf640 1 -- 172.21.15.88:0/2064254116 wait complete. 2024-09-16T01:42:34.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.417+0000 7f5013dcf640 1 Processor -- start 2024-09-16T01:42:34.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.417+0000 7f5013dcf640 1 -- start start 2024-09-16T01:42:34.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.418+0000 7f5013dcf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f500c158af0 0x7f500c0c9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:34.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.418+0000 7f5013dcf640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f500c158ed0 0x7f500c0d4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:34.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.418+0000 7f5013dcf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f500c0c9330 0x7f500c0d4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:34.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.418+0000 7f5013dcf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f500c15eb80 con 0x7f500c158af0 2024-09-16T01:42:34.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.418+0000 7f5013dcf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f500c0b7a00 con 0x7f500c0c9330 2024-09-16T01:42:34.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.418+0000 7f5013dcf640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f500c15ed00 con 0x7f500c158ed0 2024-09-16T01:42:34.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.418+0000 7f5011b6b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f500c158af0 0x7f500c0c9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:34.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.418+0000 7f5011b6b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f500c158af0 0x7f500c0c9a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40210/0 (socket says 172.21.15.88:40210) 2024-09-16T01:42:34.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.418+0000 7f5011b6b640 1 -- 172.21.15.88:0/313076345 learned_addr learned my addr 172.21.15.88:0/313076345 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:34.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.418+0000 7f501236c640 1 --2- 172.21.15.88:0/313076345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f500c0c9330 0x7f500c0d4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:34.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.418+0000 7f501136a640 1 --2- 172.21.15.88:0/313076345 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f500c158ed0 0x7f500c0d4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:34.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.418+0000 7f5011b6b640 1 -- 172.21.15.88:0/313076345 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f500c158ed0 msgr2=0x7f500c0d4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:34.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.418+0000 7f5011b6b640 1 --2- 172.21.15.88:0/313076345 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f500c158ed0 0x7f500c0d4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:34.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.418+0000 7f5011b6b640 1 -- 172.21.15.88:0/313076345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f500c0c9330 msgr2=0x7f500c0d4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:34.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.418+0000 7f5011b6b640 1 --2- 172.21.15.88:0/313076345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f500c0c9330 0x7f500c0d4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:34.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.418+0000 7f5011b6b640 1 -- 172.21.15.88:0/313076345 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f500c0ca350 con 0x7f500c158af0 2024-09-16T01:42:34.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.418+0000 7f501236c640 1 --2- 172.21.15.88:0/313076345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f500c0c9330 0x7f500c0d4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:34.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.418+0000 7f501136a640 1 --2- 172.21.15.88:0/313076345 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f500c158ed0 0x7f500c0d4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:34.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.419+0000 7f5011b6b640 1 --2- 172.21.15.88:0/313076345 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f500c158af0 0x7f500c0c9a50 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f500400fb90 tx=0x7f500400d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:34.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.419+0000 7f500a7fc640 1 -- 172.21.15.88:0/313076345 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500400fe90 con 0x7f500c158af0 2024-09-16T01:42:34.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.419+0000 7f500a7fc640 1 -- 172.21.15.88:0/313076345 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5004018070 con 0x7f500c158af0 2024-09-16T01:42:34.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.419+0000 7f500a7fc640 1 -- 172.21.15.88:0/313076345 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5004003970 con 0x7f500c158af0 2024-09-16T01:42:34.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.419+0000 7f5013dcf640 1 -- 172.21.15.88:0/313076345 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f500c0ca630 con 0x7f500c158af0 2024-09-16T01:42:34.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.419+0000 7f5013dcf640 1 -- 172.21.15.88:0/313076345 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f500c0cabf0 con 0x7f500c158af0 2024-09-16T01:42:34.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.420+0000 7f5013dcf640 1 -- 172.21.15.88:0/313076345 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f500c0b61e0 con 0x7f500c158af0 2024-09-16T01:42:34.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.420+0000 7f500a7fc640 1 -- 172.21.15.88:0/313076345 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5004003b10 con 0x7f500c158af0 2024-09-16T01:42:34.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.421+0000 7f500a7fc640 1 --2- 172.21.15.88:0/313076345 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4ff806c450 0x7f4ff806e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:34.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.421+0000 7f500a7fc640 1 -- 172.21.15.88:0/313076345 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 7716+0+0 (secure 0 0 0) 0x7f500408bef0 con 0x7f500c158af0 2024-09-16T01:42:34.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.421+0000 7f501136a640 1 --2- 172.21.15.88:0/313076345 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4ff806c450 0x7f4ff806e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:34.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.421+0000 7f501136a640 1 --2- 172.21.15.88:0/313076345 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4ff806c450 0x7f4ff806e910 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f500c15fc50 tx=0x7f5000008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:34.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.425+0000 7f500a7fc640 1 -- 172.21.15.88:0/313076345 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5004059df0 con 0x7f500c158af0 2024-09-16T01:42:34.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:34.628+0000 7f5013dcf640 1 -- 172.21.15.88:0/313076345 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f500c0d2a80 con 0x7f500c158af0 2024-09-16T01:42:35.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.575+0000 7f500a7fc640 1 -- 172.21.15.88:0/313076345 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v206) ==== 161+0+0 (secure 0 0 0) 0x7f50040596f0 con 0x7f500c158af0 2024-09-16T01:42:35.576 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:42:35.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.578+0000 7f5013dcf640 1 -- 172.21.15.88:0/313076345 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4ff806c450 msgr2=0x7f4ff806e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:35.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.578+0000 7f5013dcf640 1 --2- 172.21.15.88:0/313076345 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4ff806c450 0x7f4ff806e910 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f500c15fc50 tx=0x7f5000008040 comp rx=0 tx=0).stop 2024-09-16T01:42:35.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.578+0000 7f5013dcf640 1 -- 172.21.15.88:0/313076345 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f500c158af0 msgr2=0x7f500c0c9a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:35.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.578+0000 7f5013dcf640 1 --2- 172.21.15.88:0/313076345 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f500c158af0 0x7f500c0c9a50 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f500400fb90 tx=0x7f500400d520 comp rx=0 tx=0).stop 2024-09-16T01:42:35.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.578+0000 7f5013dcf640 1 -- 172.21.15.88:0/313076345 shutdown_connections 2024-09-16T01:42:35.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.578+0000 7f5013dcf640 1 --2- 172.21.15.88:0/313076345 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4ff806c450 0x7f4ff806e910 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:35.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.578+0000 7f5013dcf640 1 --2- 172.21.15.88:0/313076345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f500c0c9330 0x7f500c0d4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:35.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.578+0000 7f5013dcf640 1 --2- 172.21.15.88:0/313076345 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f500c158ed0 0x7f500c0d4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:35.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.578+0000 7f5013dcf640 1 --2- 172.21.15.88:0/313076345 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f500c158af0 0x7f500c0c9a50 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:35.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.578+0000 7f5013dcf640 1 -- 172.21.15.88:0/313076345 >> 172.21.15.88:0/313076345 conn(0x7f500c0d11e0 msgr2=0x7f500c0b8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:35.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.578+0000 7f5013dcf640 1 -- 172.21.15.88:0/313076345 shutdown_connections 2024-09-16T01:42:35.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.578+0000 7f5013dcf640 1 -- 172.21.15.88:0/313076345 wait complete. 2024-09-16T01:42:35.594 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:42:35.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.697+0000 7f405c3e4640 1 Processor -- start 2024-09-16T01:42:35.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.697+0000 7f405c3e4640 1 -- start start 2024-09-16T01:42:35.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.698+0000 7f405c3e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f405415e7a0 0x7f405406caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:35.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.698+0000 7f405c3e4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f40540b68f0 0x7f40540d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:35.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.698+0000 7f405c3e4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40540cf860 con 0x7f40540cf1f0 2024-09-16T01:42:35.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.698+0000 7f405c3e4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f40540d2ca0 con 0x7f40540b68f0 2024-09-16T01:42:35.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.698+0000 7f405c3e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f40540cfb40 con 0x7f405415e7a0 2024-09-16T01:42:35.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.699+0000 7f405a981640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f40540b68f0 0x7f40540d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:35.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.699+0000 7f405a981640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f40540b68f0 0x7f40540d3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40216/0 (socket says 172.21.15.88:40216) 2024-09-16T01:42:35.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.699+0000 7f405a981640 1 -- 172.21.15.88:0/3238150087 learned_addr learned my addr 172.21.15.88:0/3238150087 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:35.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.699+0000 7f405a981640 1 -- 172.21.15.88:0/3238150087 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f405415e7a0 msgr2=0x7f405406caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:35.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.699+0000 7f405a180640 1 --2- 172.21.15.88:0/3238150087 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f405415e7a0 0x7f405406caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:35.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.699+0000 7f405a981640 1 --2- 172.21.15.88:0/3238150087 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f405415e7a0 0x7f405406caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:35.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.699+0000 7f405a981640 1 -- 172.21.15.88:0/3238150087 >> v1:172.21.15.154:6789/0 conn(0x7f40540cf1f0 legacy=0x7f40540b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:35.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.699+0000 7f405a981640 1 -- 172.21.15.88:0/3238150087 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40540cfe20 con 0x7f40540b68f0 2024-09-16T01:42:35.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.699+0000 7f405a180640 1 --2- 172.21.15.88:0/3238150087 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f405415e7a0 0x7f405406caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:35.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.700+0000 7f405a981640 1 --2- 172.21.15.88:0/3238150087 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40540b68f0 0x7f40540d3410 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f404c009e50 tx=0x7f404c02a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=199d50aae7c37d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:35.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.700+0000 7f405917e640 1 -- 172.21.15.88:0/3238150087 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f404c036070 con 0x7f40540b68f0 2024-09-16T01:42:35.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.700+0000 7f405917e640 1 -- 172.21.15.88:0/3238150087 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f404c031600 con 0x7f40540b68f0 2024-09-16T01:42:35.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.700+0000 7f405917e640 1 -- 172.21.15.88:0/3238150087 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f404c0319d0 con 0x7f40540b68f0 2024-09-16T01:42:35.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.700+0000 7f405c3e4640 1 -- 172.21.15.88:0/3238150087 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40540b68f0 msgr2=0x7f40540d3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:35.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.700+0000 7f405c3e4640 1 --2- 172.21.15.88:0/3238150087 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40540b68f0 0x7f40540d3410 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f404c009e50 tx=0x7f404c02a630 comp rx=0 tx=0).stop 2024-09-16T01:42:35.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.700+0000 7f405c3e4640 1 -- 172.21.15.88:0/3238150087 shutdown_connections 2024-09-16T01:42:35.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.700+0000 7f405c3e4640 1 --2- 172.21.15.88:0/3238150087 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f40540b68f0 0x7f40540d3410 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:35.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.700+0000 7f405c3e4640 1 --2- 172.21.15.88:0/3238150087 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f405415e7a0 0x7f405406caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:35.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.700+0000 7f405c3e4640 1 -- 172.21.15.88:0/3238150087 >> 172.21.15.88:0/3238150087 conn(0x7f40540d11e0 msgr2=0x7f40540d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:35.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.701+0000 7f405c3e4640 1 -- 172.21.15.88:0/3238150087 shutdown_connections 2024-09-16T01:42:35.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.701+0000 7f405c3e4640 1 -- 172.21.15.88:0/3238150087 wait complete. 2024-09-16T01:42:35.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.702+0000 7f405c3e4640 1 Processor -- start 2024-09-16T01:42:35.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.702+0000 7f405c3e4640 1 -- start start 2024-09-16T01:42:35.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.702+0000 7f405c3e4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40540b68f0 0x7f405414ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:35.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.702+0000 7f405c3e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40540cf1f0 0x7f40540c8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:35.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.702+0000 7f405c3e4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f405415e7a0 0x7f405415fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:35.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.702+0000 7f405c3e4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f405406cfe0 con 0x7f405415e7a0 2024-09-16T01:42:35.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.702+0000 7f405c3e4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f40540b7a00 con 0x7f40540b68f0 2024-09-16T01:42:35.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.702+0000 7f405c3e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f405415eb80 con 0x7f40540cf1f0 2024-09-16T01:42:35.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.703+0000 7f405a981640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f405415e7a0 0x7f405415fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:35.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.703+0000 7f405a981640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f405415e7a0 0x7f405415fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40220/0 (socket says 172.21.15.88:40220) 2024-09-16T01:42:35.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.703+0000 7f405a981640 1 -- 172.21.15.88:0/3728559918 learned_addr learned my addr 172.21.15.88:0/3728559918 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:35.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.703+0000 7f405a180640 1 --2- 172.21.15.88:0/3728559918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40540b68f0 0x7f405414ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:35.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.703+0000 7f405997f640 1 --2- 172.21.15.88:0/3728559918 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40540cf1f0 0x7f40540c8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:35.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.703+0000 7f405a981640 1 -- 172.21.15.88:0/3728559918 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40540cf1f0 msgr2=0x7f40540c8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:35.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.703+0000 7f405a981640 1 --2- 172.21.15.88:0/3728559918 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40540cf1f0 0x7f40540c8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:35.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.703+0000 7f405a981640 1 -- 172.21.15.88:0/3728559918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40540b68f0 msgr2=0x7f405414ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:35.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.703+0000 7f405a981640 1 --2- 172.21.15.88:0/3728559918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40540b68f0 0x7f405414ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:35.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.703+0000 7f405a981640 1 -- 172.21.15.88:0/3728559918 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40541608a0 con 0x7f405415e7a0 2024-09-16T01:42:35.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.703+0000 7f405a180640 1 --2- 172.21.15.88:0/3728559918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40540b68f0 0x7f405414ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:35.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.703+0000 7f405997f640 1 --2- 172.21.15.88:0/3728559918 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40540cf1f0 0x7f40540c8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:35.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.703+0000 7f405a981640 1 --2- 172.21.15.88:0/3728559918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f405415e7a0 0x7f405415fcc0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f404c034040 tx=0x7f404c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:35.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.703+0000 7f404affd640 1 -- 172.21.15.88:0/3728559918 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f404c036050 con 0x7f405415e7a0 2024-09-16T01:42:35.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.703+0000 7f404affd640 1 -- 172.21.15.88:0/3728559918 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f404c042070 con 0x7f405415e7a0 2024-09-16T01:42:35.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.703+0000 7f405c3e4640 1 -- 172.21.15.88:0/3728559918 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40541605c0 con 0x7f405415e7a0 2024-09-16T01:42:35.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.703+0000 7f404affd640 1 -- 172.21.15.88:0/3728559918 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f404c009510 con 0x7f405415e7a0 2024-09-16T01:42:35.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.703+0000 7f405c3e4640 1 -- 172.21.15.88:0/3728559918 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40540c86b0 con 0x7f405415e7a0 2024-09-16T01:42:35.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.704+0000 7f405c3e4640 1 -- 172.21.15.88:0/3728559918 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f405406caa0 con 0x7f405415e7a0 2024-09-16T01:42:35.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.705+0000 7f404affd640 1 -- 172.21.15.88:0/3728559918 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f404c007bd0 con 0x7f405415e7a0 2024-09-16T01:42:35.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.705+0000 7f404affd640 1 --2- 172.21.15.88:0/3728559918 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f403806c2b0 0x7f403806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:35.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.705+0000 7f404affd640 1 -- 172.21.15.88:0/3728559918 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 7716+0+0 (secure 0 0 0) 0x7f404c0acb70 con 0x7f405415e7a0 2024-09-16T01:42:35.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.705+0000 7f405a180640 1 --2- 172.21.15.88:0/3728559918 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f403806c2b0 0x7f403806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:35.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.705+0000 7f405a180640 1 --2- 172.21.15.88:0/3728559918 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f403806c2b0 0x7f403806e770 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f405000a990 tx=0x7f4050008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:35.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.709+0000 7f404affd640 1 -- 172.21.15.88:0/3728559918 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f404c045360 con 0x7f405415e7a0 2024-09-16T01:42:35.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:35.940+0000 7f405c3e4640 1 -- 172.21.15.88:0/3728559918 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f40540d1870 con 0x7f405415e7a0 2024-09-16T01:42:36.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.576+0000 7f404affd640 1 -- 172.21.15.88:0/3728559918 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v207) ==== 101+0+0 (secure 0 0 0) 0x7f404c07ab00 con 0x7f405415e7a0 2024-09-16T01:42:36.578 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:42:36.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.580+0000 7f405c3e4640 1 -- 172.21.15.88:0/3728559918 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f403806c2b0 msgr2=0x7f403806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:36.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.580+0000 7f405c3e4640 1 --2- 172.21.15.88:0/3728559918 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f403806c2b0 0x7f403806e770 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f405000a990 tx=0x7f4050008040 comp rx=0 tx=0).stop 2024-09-16T01:42:36.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.580+0000 7f405c3e4640 1 -- 172.21.15.88:0/3728559918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f405415e7a0 msgr2=0x7f405415fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:36.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.580+0000 7f405c3e4640 1 --2- 172.21.15.88:0/3728559918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f405415e7a0 0x7f405415fcc0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f404c034040 tx=0x7f404c004a40 comp rx=0 tx=0).stop 2024-09-16T01:42:36.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.580+0000 7f405c3e4640 1 -- 172.21.15.88:0/3728559918 shutdown_connections 2024-09-16T01:42:36.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.580+0000 7f405c3e4640 1 --2- 172.21.15.88:0/3728559918 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f403806c2b0 0x7f403806e770 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.580+0000 7f405c3e4640 1 --2- 172.21.15.88:0/3728559918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f405415e7a0 0x7f405415fcc0 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.580+0000 7f405c3e4640 1 --2- 172.21.15.88:0/3728559918 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40540cf1f0 0x7f40540c8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.580+0000 7f405c3e4640 1 --2- 172.21.15.88:0/3728559918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40540b68f0 0x7f405414ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.580+0000 7f405c3e4640 1 -- 172.21.15.88:0/3728559918 >> 172.21.15.88:0/3728559918 conn(0x7f40540d11e0 msgr2=0x7f405415d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:36.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.580+0000 7f405c3e4640 1 -- 172.21.15.88:0/3728559918 shutdown_connections 2024-09-16T01:42:36.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.580+0000 7f405c3e4640 1 -- 172.21.15.88:0/3728559918 wait complete. 2024-09-16T01:42:36.594 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:42:36.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.637+0000 7fd07cf1ff40 1 Processor -- start 2024-09-16T01:42:36.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.637+0000 7fd07cf1ff40 1 -- start start 2024-09-16T01:42:36.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.637+0000 7fd07cf1ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5614f0505ee0 0x5614f0535500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:36.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.637+0000 7fd07cf1ff40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5614f0536590 0x5614f0534460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:36.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.637+0000 7fd07cf1ff40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614f05d9c90 0x5614f05053f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:36.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.637+0000 7fd07cf1ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5614f0457510 con 0x5614f0505ee0 2024-09-16T01:42:36.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.637+0000 7fd07cf1ff40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5614f04b0c50 con 0x5614f05d9c90 2024-09-16T01:42:36.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.637+0000 7fd07cf1ff40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5614f05dac30 con 0x5614f0536590 2024-09-16T01:42:36.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.637+0000 7fd079f90640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5614f0505ee0 0x5614f0535500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:36.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.637+0000 7fd079f90640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5614f0505ee0 0x5614f0535500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40228/0 (socket says 172.21.15.88:40228) 2024-09-16T01:42:36.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.637+0000 7fd079f90640 1 -- 172.21.15.88:0/1149158397 learned_addr learned my addr 172.21.15.88:0/1149158397 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:36.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.638+0000 7fd07a791640 1 --2- 172.21.15.88:0/1149158397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614f05d9c90 0x5614f05053f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:36.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.638+0000 7fd079f90640 1 -- 172.21.15.88:0/1149158397 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5614f0536590 msgr2=0x5614f0534460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:36.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.638+0000 7fd079f90640 1 --2- 172.21.15.88:0/1149158397 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5614f0536590 0x5614f0534460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.638+0000 7fd079f90640 1 -- 172.21.15.88:0/1149158397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614f05d9c90 msgr2=0x5614f05053f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:36.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.638+0000 7fd079f90640 1 --2- 172.21.15.88:0/1149158397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614f05d9c90 0x5614f05053f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.638+0000 7fd079f90640 1 -- 172.21.15.88:0/1149158397 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614f0504960 con 0x5614f0505ee0 2024-09-16T01:42:36.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.638+0000 7fd07a791640 1 --2- 172.21.15.88:0/1149158397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614f05d9c90 0x5614f05053f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:36.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.638+0000 7fd079f90640 1 --2- 172.21.15.88:0/1149158397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5614f0505ee0 0x5614f0535500 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fd06c001dc0 tx=0x7fd06c00b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0d88b5ecded5d96 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:36.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.638+0000 7fd078f8e640 1 -- 172.21.15.88:0/1149158397 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06c012020 con 0x5614f0505ee0 2024-09-16T01:42:36.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.638+0000 7fd078f8e640 1 -- 172.21.15.88:0/1149158397 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd06c00e670 con 0x5614f0505ee0 2024-09-16T01:42:36.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.638+0000 7fd078f8e640 1 -- 172.21.15.88:0/1149158397 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06c00ea80 con 0x5614f0505ee0 2024-09-16T01:42:36.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.638+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/1149158397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5614f0505ee0 msgr2=0x5614f0535500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:36.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.638+0000 7fd07cf1ff40 1 --2- 172.21.15.88:0/1149158397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5614f0505ee0 0x5614f0535500 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fd06c001dc0 tx=0x7fd06c00b6c0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.638+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/1149158397 shutdown_connections 2024-09-16T01:42:36.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.638+0000 7fd07cf1ff40 1 --2- 172.21.15.88:0/1149158397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614f05d9c90 0x5614f05053f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.638+0000 7fd07cf1ff40 1 --2- 172.21.15.88:0/1149158397 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5614f0536590 0x5614f0534460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.638+0000 7fd07cf1ff40 1 --2- 172.21.15.88:0/1149158397 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5614f0505ee0 0x5614f0535500 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.638+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/1149158397 >> 172.21.15.88:0/1149158397 conn(0x5614f0530680 msgr2=0x5614f0504120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:36.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.639+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/1149158397 shutdown_connections 2024-09-16T01:42:36.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.639+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/1149158397 wait complete. 2024-09-16T01:42:36.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.639+0000 7fd07cf1ff40 1 Processor -- start 2024-09-16T01:42:36.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.639+0000 7fd07cf1ff40 1 -- start start 2024-09-16T01:42:36.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.640+0000 7fd07cf1ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5614f0505ee0 0x5614f0508220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:36.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.640+0000 7fd07cf1ff40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5614f0536590 0x5614f05ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:36.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.640+0000 7fd07cf1ff40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614f05d9c90 0x5614f05c5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:36.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.640+0000 7fd07cf1ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5614f05e8980 con 0x5614f0505ee0 2024-09-16T01:42:36.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.640+0000 7fd07cf1ff40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5614f05349a0 con 0x5614f05d9c90 2024-09-16T01:42:36.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.640+0000 7fd07cf1ff40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5614f0535bb0 con 0x5614f0536590 2024-09-16T01:42:36.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.640+0000 7fd079f90640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5614f0505ee0 0x5614f0508220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:36.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.640+0000 7fd079f90640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5614f0505ee0 0x5614f0508220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40238/0 (socket says 172.21.15.88:40238) 2024-09-16T01:42:36.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.640+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 learned_addr learned my addr 172.21.15.88:0/2282147158 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:36.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.640+0000 7fd07a791640 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614f05d9c90 0x5614f05c5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:36.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.640+0000 7fd07178f640 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5614f0536590 0x5614f05ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:36.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.640+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5614f0536590 msgr2=0x5614f05ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:36.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.640+0000 7fd079f90640 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5614f0536590 0x5614f05ca7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.640+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614f05d9c90 msgr2=0x5614f05c5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:36.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.640+0000 7fd079f90640 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614f05d9c90 0x5614f05c5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.640+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614f05c9740 con 0x5614f0505ee0 2024-09-16T01:42:36.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.640+0000 7fd07a791640 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614f05d9c90 0x5614f05c5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:36.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.640+0000 7fd079f90640 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5614f0505ee0 0x5614f0508220 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7fd06c00b700 tx=0x7fd06c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:36.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.641+0000 7fd0737fe640 1 -- 172.21.15.88:0/2282147158 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06c00e470 con 0x5614f0505ee0 2024-09-16T01:42:36.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.641+0000 7fd0737fe640 1 -- 172.21.15.88:0/2282147158 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd06c0197f0 con 0x5614f0505ee0 2024-09-16T01:42:36.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.641+0000 7fd0737fe640 1 -- 172.21.15.88:0/2282147158 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06c019bc0 con 0x5614f0505ee0 2024-09-16T01:42:36.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.641+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5614f05c9a20 con 0x5614f0505ee0 2024-09-16T01:42:36.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.641+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614f05c9fe0 con 0x5614f0505ee0 2024-09-16T01:42:36.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.642+0000 7fd0737fe640 1 -- 172.21.15.88:0/2282147158 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd06c015020 con 0x5614f0505ee0 2024-09-16T01:42:36.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.642+0000 7fd0737fe640 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd05c06acc0 0x7fd05c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:36.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.643+0000 7fd07178f640 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd05c06acc0 0x7fd05c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:36.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.643+0000 7fd0737fe640 1 -- 172.21.15.88:0/2282147158 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 7716+0+0 (secure 0 0 0) 0x7fd06c08c370 con 0x5614f0505ee0 2024-09-16T01:42:36.647 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:42:36.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.643+0000 7fd07cf1ff40 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5614f054c880 0x5614f054cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:36.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.643+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0680270 con 0x5614f054c880 2024-09-16T01:42:36.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.643+0000 7fd07a791640 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5614f054c880 0x5614f054cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:36.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.643+0000 7fd07178f640 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd05c06acc0 0x7fd05c06d180 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fd064057000 tx=0x7fd064051000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:36.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.643+0000 7fd07a791640 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5614f054c880 0x5614f054cc60 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:36.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.645+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v110'691 uv665 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.645+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e110) -- 0x5614f0680610 con 0x5614f054c880 2024-09-16T01:42:36.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.647+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v110'692 uv692 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.647+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06809b0 con 0x5614f054c880 2024-09-16T01:42:36.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.648+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v110'693 uv693 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.650 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:42:36.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.648+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0681d60 con 0x5614f054c880 2024-09-16T01:42:36.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.649+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v110'694 uv694 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.649+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0682100 con 0x5614f054c880 2024-09-16T01:42:36.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.651+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v110'695 uv695 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.651+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0680d50 con 0x5614f054c880 2024-09-16T01:42:36.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.652+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v110'696 uv696 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.652+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06810f0 con 0x5614f054c880 2024-09-16T01:42:36.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.653+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v110'697 uv671 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.653+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e110) -- 0x5614f06814e0 con 0x5614f054c880 2024-09-16T01:42:36.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.655+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v110'698 uv698 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.655+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06814e0 con 0x5614f054c880 2024-09-16T01:42:36.657 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:42:36.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.656+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v110'699 uv699 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.656+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0681490 con 0x5614f054c880 2024-09-16T01:42:36.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.657+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v110'700 uv700 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.657+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0681490 con 0x5614f054c880 2024-09-16T01:42:36.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.658+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v110'701 uv701 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.658+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0681490 con 0x5614f054c880 2024-09-16T01:42:36.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.660+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v110'702 uv702 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.660+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0681490 con 0x5614f054c880 2024-09-16T01:42:36.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.661+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v110'703 uv678 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.661+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e110) -- 0x5614f0681490 con 0x5614f054c880 2024-09-16T01:42:36.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.663+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v110'704 uv704 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.663+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0681490 con 0x5614f054c880 2024-09-16T01:42:36.665 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:42:36.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.664+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v110'705 uv705 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.664+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0681490 con 0x5614f054c880 2024-09-16T01:42:36.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.665+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v110'706 uv706 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.665+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0681490 con 0x5614f054c880 2024-09-16T01:42:36.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.667+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v110'707 uv707 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.667+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0681490 con 0x5614f054c880 2024-09-16T01:42:36.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.668+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v110'708 uv708 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.668+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0681490 con 0x5614f054c880 2024-09-16T01:42:36.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.669+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v110'709 uv709 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.669+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0681490 con 0x5614f054c880 2024-09-16T01:42:36.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.671+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v110'710 uv684 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.671+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e110) -- 0x5614f0681490 con 0x5614f054c880 2024-09-16T01:42:36.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.672+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v110'711 uv711 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.672+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0681490 con 0x5614f054c880 2024-09-16T01:42:36.674 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:42:36.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.673+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v110'712 uv712 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.673+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0681490 con 0x5614f054c880 2024-09-16T01:42:36.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.675+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v110'713 uv713 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.675+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0681490 con 0x5614f054c880 2024-09-16T01:42:36.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.676+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v110'714 uv714 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.676+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0681490 con 0x5614f054c880 2024-09-16T01:42:36.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.677+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v110'715 uv715 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.677+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0681490 con 0x5614f054c880 2024-09-16T01:42:36.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.679+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v110'716 uv690 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.679+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e110) -- 0x5614f0681490 con 0x5614f054c880 2024-09-16T01:42:36.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.680+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v110'717 uv717 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.680+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0681490 con 0x5614f054c880 2024-09-16T01:42:36.682 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:42:36.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.681+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v110'718 uv718 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.681+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0681490 con 0x5614f054c880 2024-09-16T01:42:36.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.682+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v110'719 uv719 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.682+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0681490 con 0x5614f054c880 2024-09-16T01:42:36.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.683+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v110'720 uv720 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.683+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0681490 con 0x5614f054c880 2024-09-16T01:42:36.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.684+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v110'721 uv721 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.684+0000 7fd07cf1ff40 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5614f0681880 0x5614f06849c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:36.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.684+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0681490 con 0x5614f0681880 2024-09-16T01:42:36.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.685+0000 7fd079f90640 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5614f0681880 0x5614f06849c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:36.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.685+0000 7fd079f90640 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5614f0681880 0x5614f06849c0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:36.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.687+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v110'506 uv489 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f0681880 2024-09-16T01:42:36.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.687+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e110) -- 0x5614f0684f00 con 0x5614f0681880 2024-09-16T01:42:36.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.688+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v110'507 uv507 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f0681880 2024-09-16T01:42:36.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.688+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06852a0 con 0x5614f0681880 2024-09-16T01:42:36.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.689+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v110'508 uv508 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f0681880 2024-09-16T01:42:36.690 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:42:36.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.689+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0685640 con 0x5614f0681880 2024-09-16T01:42:36.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.690+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v110'509 uv509 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f0681880 2024-09-16T01:42:36.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.690+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06859e0 con 0x5614f0681880 2024-09-16T01:42:36.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.691+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v110'510 uv510 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f0681880 2024-09-16T01:42:36.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.691+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0685d80 con 0x5614f0681880 2024-09-16T01:42:36.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.692+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v110'511 uv494 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f0681880 2024-09-16T01:42:36.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.692+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e110) -- 0x5614f0686120 con 0x5614f0681880 2024-09-16T01:42:36.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.693+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v110'512 uv512 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f0681880 2024-09-16T01:42:36.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.693+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0686120 con 0x5614f0681880 2024-09-16T01:42:36.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.694+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v110'513 uv513 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f0681880 2024-09-16T01:42:36.695 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:42:36.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.694+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0686120 con 0x5614f0681880 2024-09-16T01:42:36.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.695+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v110'514 uv514 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f0681880 2024-09-16T01:42:36.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.695+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0686120 con 0x5614f0681880 2024-09-16T01:42:36.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.697+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v110'515 uv515 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f0681880 2024-09-16T01:42:36.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.697+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0686120 con 0x5614f0681880 2024-09-16T01:42:36.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.698+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v110'516 uv499 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f0681880 2024-09-16T01:42:36.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.698+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e110) -- 0x5614f0686120 con 0x5614f0681880 2024-09-16T01:42:36.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.699+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v110'517 uv517 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f0681880 2024-09-16T01:42:36.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.699+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0686120 con 0x5614f0681880 2024-09-16T01:42:36.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.700+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v110'518 uv518 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f0681880 2024-09-16T01:42:36.701 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:42:36.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.700+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f068a5a0 con 0x5614f0681880 2024-09-16T01:42:36.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.701+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v110'519 uv519 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f0681880 2024-09-16T01:42:36.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.701+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f068a5a0 con 0x5614f0681880 2024-09-16T01:42:36.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.702+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v110'520 uv520 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f0681880 2024-09-16T01:42:36.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.702+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0686120 con 0x5614f0681880 2024-09-16T01:42:36.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.703+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v110'521 uv505 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f0681880 2024-09-16T01:42:36.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.703+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e110) -- 0x5614f0686120 con 0x5614f0681880 2024-09-16T01:42:36.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.704+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v110'522 uv522 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f0681880 2024-09-16T01:42:36.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.704+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0686120 con 0x5614f0681880 2024-09-16T01:42:36.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.705+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v110'523 uv523 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f0681880 2024-09-16T01:42:36.708 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:42:36.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.705+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0687130 con 0x5614f0681880 2024-09-16T01:42:36.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.706+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v110'524 uv524 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f0681880 2024-09-16T01:42:36.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.706+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0687130 con 0x5614f0681880 2024-09-16T01:42:36.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.707+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v110'525 uv525 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f0681880 2024-09-16T01:42:36.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.707+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0686120 con 0x5614f0681880 2024-09-16T01:42:36.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.709+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v110'526 uv526 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f0681880 2024-09-16T01:42:36.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.709+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0686120 con 0x5614f054c880 2024-09-16T01:42:36.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.710+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v110'135 uv134 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.710+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e110) -- 0x5614f0686120 con 0x5614f054c880 2024-09-16T01:42:36.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.711+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v110'136 uv136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.711+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0686120 con 0x5614f054c880 2024-09-16T01:42:36.713 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:42:36.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.712+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v110'137 uv137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.712+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0687130 con 0x5614f054c880 2024-09-16T01:42:36.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.713+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v110'138 uv138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.713+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0687130 con 0x5614f054c880 2024-09-16T01:42:36.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.714+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v110'139 uv139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.714+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0686120 con 0x5614f054c880 2024-09-16T01:42:36.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.714+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v110'140 uv140 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.715+0000 7fd07cf1ff40 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5614f0686630 0x5614f0688a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:36.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.715+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0689030 con 0x5614f0686630 2024-09-16T01:42:36.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.715+0000 7fd07178f640 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5614f0686630 0x5614f0688a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:36.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.715+0000 7fd07178f640 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5614f0686630 0x5614f0688a20 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:36.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.716+0000 7fd07178f640 1 -- 172.21.15.88:0/2282147158 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v110'101 uv100 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fd064054aa0 con 0x5614f0686630 2024-09-16T01:42:36.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.716+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e110) -- 0x5614f0689540 con 0x5614f0686630 2024-09-16T01:42:36.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.717+0000 7fd07178f640 1 -- 172.21.15.88:0/2282147158 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v110'102 uv102 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd064054aa0 con 0x5614f0686630 2024-09-16T01:42:36.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.717+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0689a10 con 0x5614f0686630 2024-09-16T01:42:36.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.718+0000 7fd07178f640 1 -- 172.21.15.88:0/2282147158 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v110'103 uv103 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd064054aa0 con 0x5614f0686630 2024-09-16T01:42:36.720 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:42:36.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.718+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f05d3dd0 con 0x5614f0686630 2024-09-16T01:42:36.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.719+0000 7fd07178f640 1 -- 172.21.15.88:0/2282147158 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v110'104 uv104 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fd064054aa0 con 0x5614f0686630 2024-09-16T01:42:36.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.719+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f068a340 con 0x5614f054c880 2024-09-16T01:42:36.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.720+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v110'135 uv134 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.720+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e110) -- 0x5614f068a340 con 0x5614f054c880 2024-09-16T01:42:36.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.722+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v110'136 uv136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.722+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f068a340 con 0x5614f054c880 2024-09-16T01:42:36.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.723+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v110'137 uv137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.724 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:42:36.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.723+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f068b350 con 0x5614f054c880 2024-09-16T01:42:36.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.724+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v110'138 uv138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.724+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f068b350 con 0x5614f054c880 2024-09-16T01:42:36.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.725+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v110'139 uv139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.725+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f068a340 con 0x5614f054c880 2024-09-16T01:42:36.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.726+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v110'140 uv140 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.726+0000 7fd07cf1ff40 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5614f068ac10 0x5614f068aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:36.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.726+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f068d740 con 0x5614f068ac10 2024-09-16T01:42:36.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.726+0000 7fd07a791640 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5614f068ac10 0x5614f068aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:36.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.727+0000 7fd07a791640 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5614f068ac10 0x5614f068aff0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:36.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.728+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v110'249 uv243 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.728+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e110) -- 0x5614f068d740 con 0x5614f068ac10 2024-09-16T01:42:36.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.729+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v110'250 uv250 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.729+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f068d740 con 0x5614f068ac10 2024-09-16T01:42:36.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.730+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v110'251 uv251 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.732 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:42:36.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.730+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f068e7f0 con 0x5614f068ac10 2024-09-16T01:42:36.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.731+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v110'252 uv252 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.731+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f068e7b0 con 0x5614f068ac10 2024-09-16T01:42:36.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.733+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v110'253 uv253 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.733+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f068e7b0 con 0x5614f068ac10 2024-09-16T01:42:36.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.734+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v110'254 uv254 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.734+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f068e7b0 con 0x5614f068ac10 2024-09-16T01:42:36.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.735+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v110'255 uv248 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.735+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e110) -- 0x5614f068e7b0 con 0x5614f068ac10 2024-09-16T01:42:36.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.736+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v110'256 uv256 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.736+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f068e7b0 con 0x5614f068ac10 2024-09-16T01:42:36.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.737+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v110'257 uv257 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.739 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:42:36.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.737+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f068e7b0 con 0x5614f068ac10 2024-09-16T01:42:36.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.739+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v110'258 uv258 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.739+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f068e7b0 con 0x5614f068ac10 2024-09-16T01:42:36.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.740+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v110'259 uv259 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.740+0000 7fd07cf1ff40 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5614f068dca0 0x5614f068e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:36.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.740+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0690c00 con 0x5614f068dca0 2024-09-16T01:42:36.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.740+0000 7fd079f90640 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5614f068dca0 0x5614f068e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:36.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.741+0000 7fd079f90640 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5614f068dca0 0x5614f068e100 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:36.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.742+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v110'112 uv111 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f068dca0 2024-09-16T01:42:36.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.742+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e110) -- 0x5614f0690c00 con 0x5614f068dca0 2024-09-16T01:42:36.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.743+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v110'113 uv113 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f068dca0 2024-09-16T01:42:36.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.743+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0690ff0 con 0x5614f068dca0 2024-09-16T01:42:36.745 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:42:36.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.744+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v110'114 uv114 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f068dca0 2024-09-16T01:42:36.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.744+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0690ff0 con 0x5614f068dca0 2024-09-16T01:42:36.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.745+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v110'115 uv115 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f068dca0 2024-09-16T01:42:36.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.745+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0690ff0 con 0x5614f068dca0 2024-09-16T01:42:36.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.746+0000 7fd079f90640 1 -- 172.21.15.88:0/2282147158 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v110'116 uv116 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd06c029020 con 0x5614f068dca0 2024-09-16T01:42:36.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.746+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0690ff0 con 0x5614f054c880 2024-09-16T01:42:36.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.747+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v110'135 uv134 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.747+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e110) -- 0x5614f0690ff0 con 0x5614f054c880 2024-09-16T01:42:36.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.748+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v110'136 uv136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.748+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0690ff0 con 0x5614f054c880 2024-09-16T01:42:36.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.749+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v110'137 uv137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.750 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:42:36.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.749+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0692000 con 0x5614f054c880 2024-09-16T01:42:36.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.750+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v110'138 uv138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.750+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0692000 con 0x5614f054c880 2024-09-16T01:42:36.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.751+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v110'139 uv139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.751+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0690ff0 con 0x5614f054c880 2024-09-16T01:42:36.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.751+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v110'140 uv140 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f054c880 2024-09-16T01:42:36.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.752+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0690ff0 con 0x5614f0686630 2024-09-16T01:42:36.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.753+0000 7fd07178f640 1 -- 172.21.15.88:0/2282147158 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v110'223 uv217 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fd064054aa0 con 0x5614f0686630 2024-09-16T01:42:36.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.753+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e110) -- 0x5614f0691390 con 0x5614f0686630 2024-09-16T01:42:36.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.754+0000 7fd07178f640 1 -- 172.21.15.88:0/2282147158 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v110'224 uv224 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd064054aa0 con 0x5614f0686630 2024-09-16T01:42:36.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.754+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0691730 con 0x5614f0686630 2024-09-16T01:42:36.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.754+0000 7fd07178f640 1 -- 172.21.15.88:0/2282147158 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v110'225 uv225 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd064054aa0 con 0x5614f0686630 2024-09-16T01:42:36.756 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:42:36.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.754+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0691ad0 con 0x5614f0686630 2024-09-16T01:42:36.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.756+0000 7fd07178f640 1 -- 172.21.15.88:0/2282147158 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v110'226 uv226 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd064054aa0 con 0x5614f0686630 2024-09-16T01:42:36.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.756+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0691ad0 con 0x5614f0686630 2024-09-16T01:42:36.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.756+0000 7fd07178f640 1 -- 172.21.15.88:0/2282147158 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v110'227 uv227 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fd064054aa0 con 0x5614f0686630 2024-09-16T01:42:36.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.756+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0691ad0 con 0x5614f0686630 2024-09-16T01:42:36.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.757+0000 7fd07178f640 1 -- 172.21.15.88:0/2282147158 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v110'228 uv222 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd064054aa0 con 0x5614f0686630 2024-09-16T01:42:36.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.757+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e110) -- 0x5614f0691ad0 con 0x5614f0686630 2024-09-16T01:42:36.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.758+0000 7fd07178f640 1 -- 172.21.15.88:0/2282147158 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v110'229 uv229 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd064054aa0 con 0x5614f0686630 2024-09-16T01:42:36.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.758+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0691ad0 con 0x5614f0686630 2024-09-16T01:42:36.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.759+0000 7fd07178f640 1 -- 172.21.15.88:0/2282147158 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v110'230 uv230 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd064054aa0 con 0x5614f0686630 2024-09-16T01:42:36.761 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:42:36.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.759+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0691ad0 con 0x5614f0686630 2024-09-16T01:42:36.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.760+0000 7fd07178f640 1 -- 172.21.15.88:0/2282147158 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v110'231 uv231 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd064054aa0 con 0x5614f0686630 2024-09-16T01:42:36.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.760+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06948e0 con 0x5614f0686630 2024-09-16T01:42:36.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.761+0000 7fd07178f640 1 -- 172.21.15.88:0/2282147158 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v110'232 uv232 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd064054aa0 con 0x5614f0686630 2024-09-16T01:42:36.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.761+0000 7fd07cf1ff40 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5614f0692600 0x5614f0694f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:36.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.761+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f0692600 2024-09-16T01:42:36.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.761+0000 7fd07178f640 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5614f0692600 0x5614f0694f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:36.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.762+0000 7fd07178f640 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5614f0692600 0x5614f0694f40 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:36.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.763+0000 7fd07178f640 1 -- 172.21.15.88:0/2282147158 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v110'99 uv98 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fd064054aa0 con 0x5614f0692600 2024-09-16T01:42:36.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.763+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f0692600 2024-09-16T01:42:36.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.764+0000 7fd07178f640 1 -- 172.21.15.88:0/2282147158 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v110'100 uv100 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fd064054aa0 con 0x5614f0692600 2024-09-16T01:42:36.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.764+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f0692600 2024-09-16T01:42:36.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.765+0000 7fd07178f640 1 -- 172.21.15.88:0/2282147158 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v110'101 uv101 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fd064054aa0 con 0x5614f0692600 2024-09-16T01:42:36.766 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:42:36.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.765+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f0692600 2024-09-16T01:42:36.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.766+0000 7fd07178f640 1 -- 172.21.15.88:0/2282147158 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v110'102 uv102 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fd064054aa0 con 0x5614f0692600 2024-09-16T01:42:36.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.766+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f0686630 2024-09-16T01:42:36.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.767+0000 7fd07178f640 1 -- 172.21.15.88:0/2282147158 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v110'135 uv134 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd064054aa0 con 0x5614f0686630 2024-09-16T01:42:36.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.767+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f0686630 2024-09-16T01:42:36.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.768+0000 7fd07178f640 1 -- 172.21.15.88:0/2282147158 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v110'136 uv136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd064054aa0 con 0x5614f0686630 2024-09-16T01:42:36.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.768+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f0686630 2024-09-16T01:42:36.776 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:42:36.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.775+0000 7fd07178f640 1 -- 172.21.15.88:0/2282147158 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v110'137 uv137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd064054aa0 con 0x5614f0686630 2024-09-16T01:42:36.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.775+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f0686630 2024-09-16T01:42:36.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.776+0000 7fd07178f640 1 -- 172.21.15.88:0/2282147158 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v110'138 uv138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd064054aa0 con 0x5614f0686630 2024-09-16T01:42:36.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.776+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f0686630 2024-09-16T01:42:36.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.777+0000 7fd07178f640 1 -- 172.21.15.88:0/2282147158 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v110'139 uv139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd064054aa0 con 0x5614f0686630 2024-09-16T01:42:36.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.777+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f0686630 2024-09-16T01:42:36.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.778+0000 7fd07178f640 1 -- 172.21.15.88:0/2282147158 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v110'140 uv140 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd064054aa0 con 0x5614f0686630 2024-09-16T01:42:36.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.778+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f068ac10 2024-09-16T01:42:36.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.779+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v110'115 uv114 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.779+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f068ac10 2024-09-16T01:42:36.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.780+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v110'116 uv116 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.780+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f068ac10 2024-09-16T01:42:36.783 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:42:36.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.781+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v110'117 uv117 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.782+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f068ac10 2024-09-16T01:42:36.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.783+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v110'118 uv118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.783+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f068ac10 2024-09-16T01:42:36.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.784+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v110'119 uv119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.784+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f068ac10 2024-09-16T01:42:36.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.785+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v110'278 uv277 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.785+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f068ac10 2024-09-16T01:42:36.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.786+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v110'279 uv279 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.786+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f068ac10 2024-09-16T01:42:36.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.787+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v110'280 uv280 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.788 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:42:36.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.787+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f068ac10 2024-09-16T01:42:36.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.789+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v110'281 uv281 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.789+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f068ac10 2024-09-16T01:42:36.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.790+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v110'282 uv282 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.790+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f068ac10 2024-09-16T01:42:36.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.791+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v110'283 uv276 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.791+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f068ac10 2024-09-16T01:42:36.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.792+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v110'284 uv284 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.792+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f068ac10 2024-09-16T01:42:36.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.793+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v110'285 uv285 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.794 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:42:36.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.793+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f068ac10 2024-09-16T01:42:36.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.794+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v110'286 uv286 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.794+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f068ac10 2024-09-16T01:42:36.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.795+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v110'287 uv287 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.795+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f06949f0 con 0x5614f068ac10 2024-09-16T01:42:36.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.797+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v110'288 uv288 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.797+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5614f0695480 con 0x5614f068ac10 2024-09-16T01:42:36.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.798+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v110'289 uv289 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd07400b770 con 0x5614f068ac10 2024-09-16T01:42:36.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.798+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5614f054c880 msgr2=0x5614f054cc60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:36.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.798+0000 7fd07cf1ff40 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5614f054c880 0x5614f054cc60 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.798+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5614f0686630 msgr2=0x5614f0688a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:36.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.798+0000 7fd07cf1ff40 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5614f0686630 0x5614f0688a20 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.798+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5614f0692600 msgr2=0x5614f0694f40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:36.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.798+0000 7fd07cf1ff40 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5614f0692600 0x5614f0694f40 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.798+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5614f0681880 msgr2=0x5614f06849c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:36.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.798+0000 7fd07cf1ff40 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5614f0681880 0x5614f06849c0 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.798+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5614f068ac10 msgr2=0x5614f068aff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:36.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.798+0000 7fd07cf1ff40 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5614f068ac10 0x5614f068aff0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.798+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 reap_dead start 2024-09-16T01:42:36.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.798+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5614f068dca0 msgr2=0x5614f068e100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:36.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.798+0000 7fd07cf1ff40 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5614f068dca0 0x5614f068e100 crc :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.798+0000 7fd07a791640 1 -- 172.21.15.88:0/2282147158 reap_dead start 2024-09-16T01:42:36.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.798+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd05c06acc0 msgr2=0x7fd05c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:36.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.798+0000 7fd07cf1ff40 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd05c06acc0 0x7fd05c06d180 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fd064057000 tx=0x7fd064051000 comp rx=0 tx=0).stop 2024-09-16T01:42:36.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.798+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5614f0505ee0 msgr2=0x5614f0508220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:36.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.798+0000 7fd07cf1ff40 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5614f0505ee0 0x5614f0508220 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7fd06c00b700 tx=0x7fd06c000bc0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.798+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 shutdown_connections 2024-09-16T01:42:36.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.798+0000 7fd07cf1ff40 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd05c06acc0 0x7fd05c06d180 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.798+0000 7fd07cf1ff40 1 --2- 172.21.15.88:0/2282147158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5614f0505ee0 0x5614f0508220 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.798+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 >> 172.21.15.88:0/2282147158 conn(0x5614f0530680 msgr2=0x5614f0504120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:36.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.798+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 shutdown_connections 2024-09-16T01:42:36.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.798+0000 7fd07cf1ff40 1 -- 172.21.15.88:0/2282147158 wait complete. 2024-09-16T01:42:36.803 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata truncate 1.00000000.inode 0 2024-09-16T01:42:36.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.885+0000 7ffab218df40 1 Processor -- start 2024-09-16T01:42:36.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.885+0000 7ffab218df40 1 -- start start 2024-09-16T01:42:36.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.886+0000 7ffab218df40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa980022a0 0x5615d93c8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:36.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.886+0000 7ffab218df40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615d9322f30 0x5615d93cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:36.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.886+0000 7ffab218df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5615d9341d60 0x5615d9458190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:36.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.886+0000 7ffab218df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5615d9248510 con 0x5615d9341d60 2024-09-16T01:42:36.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.886+0000 7ffab218df40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5615d93cf120 con 0x5615d9322f30 2024-09-16T01:42:36.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.886+0000 7ffab218df40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5615d9455fc0 con 0x7ffa980022a0 2024-09-16T01:42:36.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.886+0000 7ffab0484640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5615d9341d60 0x5615d9458190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:36.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.886+0000 7ffab0484640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5615d9341d60 0x5615d9458190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40260/0 (socket says 172.21.15.88:40260) 2024-09-16T01:42:36.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.886+0000 7ffab0484640 1 -- 172.21.15.88:0/2594282219 learned_addr learned my addr 172.21.15.88:0/2594282219 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:36.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.886+0000 7ffaafc83640 1 --2- 172.21.15.88:0/2594282219 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa980022a0 0x5615d93c8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:36.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.886+0000 7ffaaf482640 1 --2- 172.21.15.88:0/2594282219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615d9322f30 0x5615d93cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:36.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.886+0000 7ffab0484640 1 -- 172.21.15.88:0/2594282219 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa980022a0 msgr2=0x5615d93c8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:36.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.886+0000 7ffab0484640 1 --2- 172.21.15.88:0/2594282219 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa980022a0 0x5615d93c8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.886+0000 7ffab0484640 1 -- 172.21.15.88:0/2594282219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615d9322f30 msgr2=0x5615d93cdb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:36.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.886+0000 7ffab0484640 1 --2- 172.21.15.88:0/2594282219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615d9322f30 0x5615d93cdb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.886+0000 7ffab0484640 1 -- 172.21.15.88:0/2594282219 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615d93c87b0 con 0x5615d9341d60 2024-09-16T01:42:36.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.887+0000 7ffab0484640 1 --2- 172.21.15.88:0/2594282219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5615d9341d60 0x5615d9458190 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x5615d9341700 tx=0x7ffaa8005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e504df98b1d768a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:36.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.887+0000 7ffaae480640 1 -- 172.21.15.88:0/2594282219 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8017020 con 0x5615d9341d60 2024-09-16T01:42:36.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.887+0000 7ffaae480640 1 -- 172.21.15.88:0/2594282219 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaa8013740 con 0x5615d9341d60 2024-09-16T01:42:36.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.887+0000 7ffaae480640 1 -- 172.21.15.88:0/2594282219 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8013b10 con 0x5615d9341d60 2024-09-16T01:42:36.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.887+0000 7ffab218df40 1 -- 172.21.15.88:0/2594282219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5615d9341d60 msgr2=0x5615d9458190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:36.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.887+0000 7ffab218df40 1 --2- 172.21.15.88:0/2594282219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5615d9341d60 0x5615d9458190 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x5615d9341700 tx=0x7ffaa8005d80 comp rx=0 tx=0).stop 2024-09-16T01:42:36.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.887+0000 7ffab218df40 1 -- 172.21.15.88:0/2594282219 shutdown_connections 2024-09-16T01:42:36.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.887+0000 7ffab218df40 1 --2- 172.21.15.88:0/2594282219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5615d9341d60 0x5615d9458190 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.887+0000 7ffab218df40 1 --2- 172.21.15.88:0/2594282219 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615d9322f30 0x5615d93cdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.887+0000 7ffab218df40 1 --2- 172.21.15.88:0/2594282219 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa980022a0 0x5615d93c8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.887+0000 7ffab218df40 1 -- 172.21.15.88:0/2594282219 >> 172.21.15.88:0/2594282219 conn(0x5615d9341750 msgr2=0x5615d93cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:36.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.887+0000 7ffab218df40 1 -- 172.21.15.88:0/2594282219 shutdown_connections 2024-09-16T01:42:36.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.887+0000 7ffab218df40 1 -- 172.21.15.88:0/2594282219 wait complete. 2024-09-16T01:42:36.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.888+0000 7ffab218df40 1 Processor -- start 2024-09-16T01:42:36.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.888+0000 7ffab218df40 1 -- start start 2024-09-16T01:42:36.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.889+0000 7ffab218df40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5615d9322f30 0x5615d933c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:36.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.889+0000 7ffab218df40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5615d9341d60 0x5615d93b7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:36.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.889+0000 7ffab218df40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa980022a0 0x5615d93b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:36.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.889+0000 7ffab218df40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5615d93412f0 con 0x5615d9322f30 2024-09-16T01:42:36.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.889+0000 7ffab218df40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5615d9456c10 con 0x7ffa980022a0 2024-09-16T01:42:36.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.889+0000 7ffab218df40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5615d9456210 con 0x5615d9341d60 2024-09-16T01:42:36.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.889+0000 7ffaafc83640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5615d9322f30 0x5615d933c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:36.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.889+0000 7ffaafc83640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5615d9322f30 0x5615d933c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40264/0 (socket says 172.21.15.88:40264) 2024-09-16T01:42:36.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.889+0000 7ffaafc83640 1 -- 172.21.15.88:0/50900896 learned_addr learned my addr 172.21.15.88:0/50900896 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:36.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.889+0000 7ffaafc83640 1 -- 172.21.15.88:0/50900896 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5615d9341d60 msgr2=0x5615d93b7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:36.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.889+0000 7ffaaf482640 1 --2- 172.21.15.88:0/50900896 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5615d9341d60 0x5615d93b7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:36.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.889+0000 7ffaafc83640 1 --2- 172.21.15.88:0/50900896 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5615d9341d60 0x5615d93b7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.889+0000 7ffab0484640 1 --2- 172.21.15.88:0/50900896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa980022a0 0x5615d93b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:36.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.889+0000 7ffaafc83640 1 -- 172.21.15.88:0/50900896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa980022a0 msgr2=0x5615d93b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:36.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.889+0000 7ffaafc83640 1 --2- 172.21.15.88:0/50900896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa980022a0 0x5615d93b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.889+0000 7ffaafc83640 1 -- 172.21.15.88:0/50900896 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615d93c7130 con 0x5615d9322f30 2024-09-16T01:42:36.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.889+0000 7ffaafc83640 1 --2- 172.21.15.88:0/50900896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5615d9322f30 0x5615d933c450 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0025e70 tx=0x7ffaa0002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:36.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.889+0000 7ffaad47e640 1 -- 172.21.15.88:0/50900896 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa004f8a0 con 0x5615d9322f30 2024-09-16T01:42:36.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.889+0000 7ffaad47e640 1 -- 172.21.15.88:0/50900896 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaa0069070 con 0x5615d9322f30 2024-09-16T01:42:36.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.889+0000 7ffab218df40 1 -- 172.21.15.88:0/50900896 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5615d933ca30 con 0x5615d9322f30 2024-09-16T01:42:36.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.889+0000 7ffaad47e640 1 -- 172.21.15.88:0/50900896 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa004fc60 con 0x5615d9322f30 2024-09-16T01:42:36.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.889+0000 7ffab218df40 1 -- 172.21.15.88:0/50900896 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615d933cff0 con 0x5615d9322f30 2024-09-16T01:42:36.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.891+0000 7ffaad47e640 1 -- 172.21.15.88:0/50900896 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ffaa0063070 con 0x5615d9322f30 2024-09-16T01:42:36.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.891+0000 7ffaad47e640 1 --2- 172.21.15.88:0/50900896 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ffa9c06acc0 0x7ffa9c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:36.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.891+0000 7ffaad47e640 1 -- 172.21.15.88:0/50900896 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 7716+0+0 (secure 0 0 0) 0x7ffaa004fe00 con 0x5615d9322f30 2024-09-16T01:42:36.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.891+0000 7ffaaf482640 1 --2- 172.21.15.88:0/50900896 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ffa9c06acc0 0x7ffa9c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:36.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.891+0000 7ffab218df40 1 --2- 172.21.15.88:0/50900896 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5615d9459100 0x5615d9459570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:36.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.891+0000 7ffab218df40 1 -- 172.21.15.88:0/50900896 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:1 12.14 12:287034c4:::1.00000000.inode:head [truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x5615d9459ab0 con 0x5615d9459100 2024-09-16T01:42:36.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.891+0000 7ffaaf482640 1 --2- 172.21.15.88:0/50900896 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ffa9c06acc0 0x7ffa9c06d180 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x5615d93257a0 tx=0x7ffaa40014b0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:36.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.892+0000 7ffab0484640 1 --2- 172.21.15.88:0/50900896 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5615d9459100 0x5615d9459570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:36.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.892+0000 7ffab0484640 1 --2- 172.21.15.88:0/50900896 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5615d9459100 0x5615d9459570 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:36.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.893+0000 7ffab0484640 1 -- 172.21.15.88:0/50900896 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(1 1.00000000.inode [truncate 0] v110'527 uv527 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ffaa800e040 con 0x5615d9459100 2024-09-16T01:42:36.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.893+0000 7ffab218df40 1 -- 172.21.15.88:0/50900896 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5615d9459100 msgr2=0x5615d9459570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:36.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.893+0000 7ffab218df40 1 --2- 172.21.15.88:0/50900896 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5615d9459100 0x5615d9459570 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.893+0000 7ffab218df40 1 -- 172.21.15.88:0/50900896 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ffa9c06acc0 msgr2=0x7ffa9c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:36.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.893+0000 7ffab218df40 1 --2- 172.21.15.88:0/50900896 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ffa9c06acc0 0x7ffa9c06d180 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x5615d93257a0 tx=0x7ffaa40014b0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.893+0000 7ffab218df40 1 -- 172.21.15.88:0/50900896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5615d9322f30 msgr2=0x5615d933c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:36.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.893+0000 7ffab218df40 1 --2- 172.21.15.88:0/50900896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5615d9322f30 0x5615d933c450 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0025e70 tx=0x7ffaa0002fb0 comp rx=0 tx=0).stop 2024-09-16T01:42:36.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.893+0000 7ffab0484640 1 -- 172.21.15.88:0/50900896 reap_dead start 2024-09-16T01:42:36.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.893+0000 7ffab218df40 1 -- 172.21.15.88:0/50900896 shutdown_connections 2024-09-16T01:42:36.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.893+0000 7ffab218df40 1 -- 172.21.15.88:0/50900896 >> 172.21.15.88:0/50900896 conn(0x5615d9341750 msgr2=0x5615d93cc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:36.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.893+0000 7ffab218df40 1 -- 172.21.15.88:0/50900896 shutdown_connections 2024-09-16T01:42:36.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:36.893+0000 7ffab218df40 1 -- 172.21.15.88:0/50900896 wait complete. 2024-09-16T01:42:36.900 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:42:37.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.041+0000 7f4a8919f640 1 Processor -- start 2024-09-16T01:42:37.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.041+0000 7f4a8919f640 1 -- start start 2024-09-16T01:42:37.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.042+0000 7f4a8919f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a840b4a50 0x7f4a840b3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:37.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.042+0000 7f4a8919f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a840b0090 con 0x7f4a8415e460 2024-09-16T01:42:37.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.042+0000 7f4a8919f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a840af480 con 0x7f4a8415e840 2024-09-16T01:42:37.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.042+0000 7f4a8919f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4a840af760 con 0x7f4a840b4a50 2024-09-16T01:42:37.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.042+0000 7f4a82d9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4a8415e460 0x7f4a840b4340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50872/0 (socket says 172.21.15.88:50872) 2024-09-16T01:42:37.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.042+0000 7f4a82d9d640 1 -- 172.21.15.88:0/220001324 learned_addr learned my addr 172.21.15.88:0/220001324 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:37.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.042+0000 7f4a8259c640 1 -- 172.21.15.88:0/220001324 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3975854081 0 0) 0x7f4a840b0090 con 0x7f4a8415e460 2024-09-16T01:42:37.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.042+0000 7f4a8259c640 1 -- 172.21.15.88:0/220001324 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a70003520 con 0x7f4a8415e460 2024-09-16T01:42:37.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.042+0000 7f4a8259c640 1 -- 172.21.15.88:0/220001324 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1029388779 0 0) 0x7f4a70003520 con 0x7f4a8415e460 2024-09-16T01:42:37.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.042+0000 7f4a8259c640 1 -- 172.21.15.88:0/220001324 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a840b4a50 msgr2=0x7f4a840b3830 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:42:37.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.042+0000 7f4a8259c640 1 --2- 172.21.15.88:0/220001324 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a840b4a50 0x7f4a840b3830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:37.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.042+0000 7f4a8259c640 1 -- 172.21.15.88:0/220001324 >> v1:172.21.15.154:6789/0 conn(0x7f4a8415e840 legacy=0x7f4a84071d60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T01:42:37.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.042+0000 7f4a8259c640 1 -- 172.21.15.88:0/220001324 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a840717d0 con 0x7f4a8415e460 2024-09-16T01:42:37.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.043+0000 7f4a8259c640 1 -- 172.21.15.88:0/220001324 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f4a74001670 con 0x7f4a8415e460 2024-09-16T01:42:37.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.043+0000 7f4a8259c640 1 -- 172.21.15.88:0/220001324 >> v1:172.21.15.88:6789/0 conn(0x7f4a8415e460 legacy=0x7f4a840b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:37.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.043+0000 7f4a8259c640 1 --2- 172.21.15.88:0/220001324 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a700039e0 0x7f4a70003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:37.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.043+0000 7f4a8259c640 1 --2- 172.21.15.88:0/220001324 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a840b4a50 0x7f4a70004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:37.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.043+0000 7f4a8259c640 1 --2- 172.21.15.88:0/220001324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a70004e60 0x7f4a7000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:37.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.043+0000 7f4a8259c640 1 -- 172.21.15.88:0/220001324 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4a840af760 con 0x7f4a700039e0 2024-09-16T01:42:37.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.043+0000 7f4a8259c640 1 -- 172.21.15.88:0/220001324 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4a70005540 con 0x7f4a70004e60 2024-09-16T01:42:37.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.043+0000 7f4a8259c640 1 -- 172.21.15.88:0/220001324 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4a7000bf00 con 0x7f4a840b4a50 2024-09-16T01:42:37.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.043+0000 7f4a82d9d640 1 --2- 172.21.15.88:0/220001324 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a700039e0 0x7f4a70003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:37.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.043+0000 7f4a7a59c640 1 --2- 172.21.15.88:0/220001324 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a840b4a50 0x7f4a70004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:37.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.043+0000 7f4a8359e640 1 --2- 172.21.15.88:0/220001324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a70004e60 0x7f4a7000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:37.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.043+0000 7f4a82d9d640 1 -- 172.21.15.88:0/220001324 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a840b4a50 msgr2=0x7f4a70004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:37.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.043+0000 7f4a82d9d640 1 --2- 172.21.15.88:0/220001324 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a840b4a50 0x7f4a70004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:37.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.043+0000 7f4a82d9d640 1 -- 172.21.15.88:0/220001324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a70004e60 msgr2=0x7f4a7000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:37.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.043+0000 7f4a82d9d640 1 --2- 172.21.15.88:0/220001324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a70004e60 0x7f4a7000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:37.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.043+0000 7f4a82d9d640 1 -- 172.21.15.88:0/220001324 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a70003520 con 0x7f4a700039e0 2024-09-16T01:42:37.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.043+0000 7f4a7a59c640 1 --2- 172.21.15.88:0/220001324 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a840b4a50 0x7f4a70004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:37.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.043+0000 7f4a8359e640 1 --2- 172.21.15.88:0/220001324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a70004e60 0x7f4a7000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:37.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.044+0000 7f4a82d9d640 1 --2- 172.21.15.88:0/220001324 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a700039e0 0x7f4a70003de0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f4a7400c210 tx=0x7f4a70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfda90040ce364e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:37.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.044+0000 7f4a8259c640 1 -- 172.21.15.88:0/220001324 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7402e070 con 0x7f4a700039e0 2024-09-16T01:42:37.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.044+0000 7f4a8259c640 1 -- 172.21.15.88:0/220001324 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a74005ad0 con 0x7f4a700039e0 2024-09-16T01:42:37.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.044+0000 7f4a8259c640 1 -- 172.21.15.88:0/220001324 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a74007d50 con 0x7f4a700039e0 2024-09-16T01:42:37.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.044+0000 7f4a8919f640 1 -- 172.21.15.88:0/220001324 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a700039e0 msgr2=0x7f4a70003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:37.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.044+0000 7f4a8919f640 1 --2- 172.21.15.88:0/220001324 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a700039e0 0x7f4a70003de0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f4a7400c210 tx=0x7f4a70005830 comp rx=0 tx=0).stop 2024-09-16T01:42:37.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.044+0000 7f4a8359e640 1 -- 172.21.15.88:0/220001324 reap_dead start 2024-09-16T01:42:37.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.044+0000 7f4a8919f640 1 -- 172.21.15.88:0/220001324 shutdown_connections 2024-09-16T01:42:37.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.044+0000 7f4a8919f640 1 -- 172.21.15.88:0/220001324 >> 172.21.15.88:0/220001324 conn(0x7f4a8406bc10 msgr2=0x7f4a8406bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:37.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.044+0000 7f4a8919f640 1 -- 172.21.15.88:0/220001324 shutdown_connections 2024-09-16T01:42:37.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.044+0000 7f4a8919f640 1 -- 172.21.15.88:0/220001324 wait complete. 2024-09-16T01:42:37.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.046+0000 7f4a8919f640 1 Processor -- start 2024-09-16T01:42:37.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.046+0000 7f4a8919f640 1 -- start start 2024-09-16T01:42:37.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.047+0000 7f4a8919f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a700039e0 0x7f4a841f5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:37.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.047+0000 7f4a8919f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a841f7880 0x7f4a8406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:37.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.047+0000 7f4a8919f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a84158ea0 0x7f4a84159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:37.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.047+0000 7f4a8919f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4a840afe50 con 0x7f4a700039e0 2024-09-16T01:42:37.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.047+0000 7f4a8919f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4a840afcd0 con 0x7f4a84158ea0 2024-09-16T01:42:37.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.047+0000 7f4a8919f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4a840ce680 con 0x7f4a841f7880 2024-09-16T01:42:37.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.047+0000 7f4a82d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a700039e0 0x7f4a841f5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:37.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.047+0000 7f4a82d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a700039e0 0x7f4a841f5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40280/0 (socket says 172.21.15.88:40280) 2024-09-16T01:42:37.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.047+0000 7f4a82d9d640 1 -- 172.21.15.88:0/1558814120 learned_addr learned my addr 172.21.15.88:0/1558814120 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:37.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.047+0000 7f4a8359e640 1 --2- 172.21.15.88:0/1558814120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a84158ea0 0x7f4a84159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:37.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.047+0000 7f4a7a59c640 1 --2- 172.21.15.88:0/1558814120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a841f7880 0x7f4a8406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:37.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.047+0000 7f4a82d9d640 1 -- 172.21.15.88:0/1558814120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a841f7880 msgr2=0x7f4a8406ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:37.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.047+0000 7f4a82d9d640 1 --2- 172.21.15.88:0/1558814120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a841f7880 0x7f4a8406ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:37.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.047+0000 7f4a82d9d640 1 -- 172.21.15.88:0/1558814120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a84158ea0 msgr2=0x7f4a84159320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:37.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.047+0000 7f4a82d9d640 1 --2- 172.21.15.88:0/1558814120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a84158ea0 0x7f4a84159320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:37.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.047+0000 7f4a82d9d640 1 -- 172.21.15.88:0/1558814120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a841f5750 con 0x7f4a700039e0 2024-09-16T01:42:37.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.047+0000 7f4a7a59c640 1 --2- 172.21.15.88:0/1558814120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a841f7880 0x7f4a8406ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:37.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.047+0000 7f4a82d9d640 1 --2- 172.21.15.88:0/1558814120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a700039e0 0x7f4a841f5ef0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f4a7402ae30 tx=0x7f4a840af650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:37.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.047+0000 7f4a7bfff640 1 -- 172.21.15.88:0/1558814120 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a74042070 con 0x7f4a700039e0 2024-09-16T01:42:37.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.047+0000 7f4a7bfff640 1 -- 172.21.15.88:0/1558814120 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a74007480 con 0x7f4a700039e0 2024-09-16T01:42:37.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.047+0000 7f4a7bfff640 1 -- 172.21.15.88:0/1558814120 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7403e450 con 0x7f4a700039e0 2024-09-16T01:42:37.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.047+0000 7f4a8919f640 1 -- 172.21.15.88:0/1558814120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a841f5a30 con 0x7f4a700039e0 2024-09-16T01:42:37.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.047+0000 7f4a8919f640 1 -- 172.21.15.88:0/1558814120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a8406d8e0 con 0x7f4a700039e0 2024-09-16T01:42:37.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.048+0000 7f4a8919f640 1 -- 172.21.15.88:0/1558814120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a841f5c00 con 0x7f4a700039e0 2024-09-16T01:42:37.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.049+0000 7f4a7bfff640 1 -- 172.21.15.88:0/1558814120 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4a74045070 con 0x7f4a700039e0 2024-09-16T01:42:37.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.049+0000 7f4a7bfff640 1 --2- 172.21.15.88:0/1558814120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4a6406c2e0 0x7f4a6406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:37.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.050+0000 7f4a7bfff640 1 -- 172.21.15.88:0/1558814120 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 7716+0+0 (secure 0 0 0) 0x7f4a740ac930 con 0x7f4a700039e0 2024-09-16T01:42:37.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.050+0000 7f4a7a59c640 1 --2- 172.21.15.88:0/1558814120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4a6406c2e0 0x7f4a6406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:37.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.050+0000 7f4a7a59c640 1 --2- 172.21.15.88:0/1558814120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4a6406c2e0 0x7f4a6406e7a0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f4a840b4970 tx=0x7f4a6c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:37.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.054+0000 7f4a7bfff640 1 -- 172.21.15.88:0/1558814120 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4a7404c020 con 0x7f4a700039e0 2024-09-16T01:42:37.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.253+0000 7f4a8919f640 1 -- 172.21.15.88:0/1558814120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f4a840b3830 con 0x7f4a700039e0 2024-09-16T01:42:37.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.587+0000 7f4a7bfff640 1 -- 172.21.15.88:0/1558814120 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v208) ==== 162+0+0 (secure 0 0 0) 0x7f4a7407a8d0 con 0x7f4a700039e0 2024-09-16T01:42:37.589 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:42:37.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.589+0000 7f4a8919f640 1 -- 172.21.15.88:0/1558814120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4a6406c2e0 msgr2=0x7f4a6406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:37.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.589+0000 7f4a8919f640 1 --2- 172.21.15.88:0/1558814120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4a6406c2e0 0x7f4a6406e7a0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f4a840b4970 tx=0x7f4a6c008040 comp rx=0 tx=0).stop 2024-09-16T01:42:37.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.589+0000 7f4a8919f640 1 -- 172.21.15.88:0/1558814120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a700039e0 msgr2=0x7f4a841f5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:37.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.589+0000 7f4a8919f640 1 --2- 172.21.15.88:0/1558814120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a700039e0 0x7f4a841f5ef0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f4a7402ae30 tx=0x7f4a840af650 comp rx=0 tx=0).stop 2024-09-16T01:42:37.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.589+0000 7f4a8919f640 1 -- 172.21.15.88:0/1558814120 shutdown_connections 2024-09-16T01:42:37.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.589+0000 7f4a8919f640 1 --2- 172.21.15.88:0/1558814120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4a6406c2e0 0x7f4a6406e7a0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:37.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.589+0000 7f4a8919f640 1 --2- 172.21.15.88:0/1558814120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a84158ea0 0x7f4a84159320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:37.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.589+0000 7f4a8919f640 1 --2- 172.21.15.88:0/1558814120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a841f7880 0x7f4a8406ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:37.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.589+0000 7f4a8919f640 1 --2- 172.21.15.88:0/1558814120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a700039e0 0x7f4a841f5ef0 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:37.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.589+0000 7f4a8919f640 1 -- 172.21.15.88:0/1558814120 >> 172.21.15.88:0/1558814120 conn(0x7f4a8406bc10 msgr2=0x7f4a8415d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:37.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.590+0000 7f4a8919f640 1 -- 172.21.15.88:0/1558814120 shutdown_connections 2024-09-16T01:42:37.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.590+0000 7f4a8919f640 1 -- 172.21.15.88:0/1558814120 wait complete. 2024-09-16T01:42:37.600 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:42:37.598+0000 7fa987834640 -1 mds.0.cache.ino(0x1) No data while reading inode 0x1 2024-09-16T01:42:37.600 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:42:37.598+0000 7fa987834640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-09-16T01:42:37.601 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:42:37.599+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:42:37.601 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:42:37.599+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:42:37.604 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:37.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.701+0000 7f1b5dfb9640 1 Processor -- start 2024-09-16T01:42:37.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.701+0000 7f1b5dfb9640 1 -- start start 2024-09-16T01:42:37.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.701+0000 7f1b5dfb9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1b580d3310 0x7f1b580d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:37.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.701+0000 7f1b5dfb9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b5815ea40 con 0x7f1b580d1650 2024-09-16T01:42:37.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.701+0000 7f1b5dfb9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b580d3870 con 0x7f1b5815e660 2024-09-16T01:42:37.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.701+0000 7f1b5dfb9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f1b580d1ac0 con 0x7f1b580d3310 2024-09-16T01:42:37.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.702+0000 7f1b57fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1b580d1650 0x7f1b580b7b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50888/0 (socket says 172.21.15.88:50888) 2024-09-16T01:42:37.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.702+0000 7f1b57fff640 1 -- 172.21.15.88:0/1711742518 learned_addr learned my addr 172.21.15.88:0/1711742518 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:37.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.702+0000 7f1b56ffd640 1 -- 172.21.15.88:0/1711742518 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1835031291 0 0) 0x7f1b5815ea40 con 0x7f1b580d1650 2024-09-16T01:42:37.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.702+0000 7f1b56ffd640 1 -- 172.21.15.88:0/1711742518 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b44003520 con 0x7f1b580d1650 2024-09-16T01:42:37.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.702+0000 7f1b4ffff640 1 --2- 172.21.15.88:0/1711742518 >> v2:172.21.15.154:3300/0 conn(0x7f1b580d3310 0x7f1b580d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:37.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.702+0000 7f1b56ffd640 1 -- 172.21.15.88:0/1711742518 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1976074937 0 0) 0x7f1b44003520 con 0x7f1b580d1650 2024-09-16T01:42:37.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.702+0000 7f1b56ffd640 1 -- 172.21.15.88:0/1711742518 >> v2:172.21.15.154:3300/0 conn(0x7f1b580d3310 msgr2=0x7f1b580d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:37.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.702+0000 7f1b56ffd640 1 --2- 172.21.15.88:0/1711742518 >> v2:172.21.15.154:3300/0 conn(0x7f1b580d3310 0x7f1b580d1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:37.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.702+0000 7f1b56ffd640 1 -- 172.21.15.88:0/1711742518 >> v1:172.21.15.154:6789/0 conn(0x7f1b5815e660 legacy=0x7f1b580cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:37.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.702+0000 7f1b56ffd640 1 -- 172.21.15.88:0/1711742518 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b580d3b50 con 0x7f1b580d1650 2024-09-16T01:42:37.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.702+0000 7f1b56ffd640 1 -- 172.21.15.88:0/1711742518 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f1b500019f0 con 0x7f1b580d1650 2024-09-16T01:42:37.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.702+0000 7f1b56ffd640 1 -- 172.21.15.88:0/1711742518 >> v1:172.21.15.88:6789/0 conn(0x7f1b580d1650 legacy=0x7f1b580b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:37.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.702+0000 7f1b56ffd640 1 --2- 172.21.15.88:0/1711742518 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1b440039b0 0x7f1b44003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:37.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.702+0000 7f1b56ffd640 1 --2- 172.21.15.88:0/1711742518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b44004800 0x7f1b44004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:37.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.702+0000 7f1b56ffd640 1 --2- 172.21.15.88:0/1711742518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b440051c0 0x7f1b4400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:37.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.702+0000 7f1b56ffd640 1 -- 172.21.15.88:0/1711742518 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1b580d1ac0 con 0x7f1b440051c0 2024-09-16T01:42:37.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.702+0000 7f1b56ffd640 1 -- 172.21.15.88:0/1711742518 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1b4400c110 con 0x7f1b44004800 2024-09-16T01:42:37.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.702+0000 7f1b56ffd640 1 -- 172.21.15.88:0/1711742518 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1b4400c310 con 0x7f1b440039b0 2024-09-16T01:42:37.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.702+0000 7f1b57fff640 1 --2- 172.21.15.88:0/1711742518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b440051c0 0x7f1b4400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:37.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.702+0000 7f1b4ffff640 1 --2- 172.21.15.88:0/1711742518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b44004800 0x7f1b44004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:37.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.703+0000 7f1b577fe640 1 --2- 172.21.15.88:0/1711742518 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1b440039b0 0x7f1b44003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:37.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.703+0000 7f1b57fff640 1 -- 172.21.15.88:0/1711742518 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1b440039b0 msgr2=0x7f1b44003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:37.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.703+0000 7f1b57fff640 1 --2- 172.21.15.88:0/1711742518 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1b440039b0 0x7f1b44003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:37.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.703+0000 7f1b57fff640 1 -- 172.21.15.88:0/1711742518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b44004800 msgr2=0x7f1b44004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:37.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.703+0000 7f1b57fff640 1 --2- 172.21.15.88:0/1711742518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b44004800 0x7f1b44004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:37.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.703+0000 7f1b57fff640 1 -- 172.21.15.88:0/1711742518 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b44003520 con 0x7f1b440051c0 2024-09-16T01:42:37.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.703+0000 7f1b577fe640 1 --2- 172.21.15.88:0/1711742518 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1b440039b0 0x7f1b44003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:37.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.703+0000 7f1b57fff640 1 -- 172.21.15.88:0/1711742518 reap_dead start 2024-09-16T01:42:37.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.703+0000 7f1b57fff640 1 --2- 172.21.15.88:0/1711742518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b440051c0 0x7f1b4400b9c0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f1b5000a840 tx=0x7f1b44005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91a70757930c46bd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:37.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.704+0000 7f1b56ffd640 1 -- 172.21.15.88:0/1711742518 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b5000a460 con 0x7f1b440051c0 2024-09-16T01:42:37.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.704+0000 7f1b56ffd640 1 -- 172.21.15.88:0/1711742518 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b5002f070 con 0x7f1b440051c0 2024-09-16T01:42:37.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.704+0000 7f1b56ffd640 1 -- 172.21.15.88:0/1711742518 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b5002bcb0 con 0x7f1b440051c0 2024-09-16T01:42:37.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.704+0000 7f1b5dfb9640 1 -- 172.21.15.88:0/1711742518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b440051c0 msgr2=0x7f1b4400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:37.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.704+0000 7f1b5dfb9640 1 --2- 172.21.15.88:0/1711742518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b440051c0 0x7f1b4400b9c0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f1b5000a840 tx=0x7f1b44005830 comp rx=0 tx=0).stop 2024-09-16T01:42:37.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.704+0000 7f1b5dfb9640 1 -- 172.21.15.88:0/1711742518 shutdown_connections 2024-09-16T01:42:37.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.704+0000 7f1b5dfb9640 1 --2- 172.21.15.88:0/1711742518 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b440051c0 0x7f1b4400b9c0 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:37.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.704+0000 7f1b5dfb9640 1 -- 172.21.15.88:0/1711742518 >> 172.21.15.88:0/1711742518 conn(0x7f1b580b6410 msgr2=0x7f1b580b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:37.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.704+0000 7f1b5dfb9640 1 -- 172.21.15.88:0/1711742518 shutdown_connections 2024-09-16T01:42:37.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.704+0000 7f1b5dfb9640 1 -- 172.21.15.88:0/1711742518 wait complete. 2024-09-16T01:42:37.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.705+0000 7f1b5dfb9640 1 Processor -- start 2024-09-16T01:42:37.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.705+0000 7f1b5dfb9640 1 -- start start 2024-09-16T01:42:37.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.706+0000 7f1b5dfb9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b440051c0 0x7f1b58154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:37.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.706+0000 7f1b5dfb9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1b58161380 0x7f1b58151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:37.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.706+0000 7f1b5dfb9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b58153dc0 0x7f1b58151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:37.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.706+0000 7f1b5dfb9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1b580d43f0 con 0x7f1b440051c0 2024-09-16T01:42:37.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.706+0000 7f1b5dfb9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1b580d4110 con 0x7f1b58153dc0 2024-09-16T01:42:37.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.706+0000 7f1b5dfb9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1b580d1110 con 0x7f1b58161380 2024-09-16T01:42:37.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.706+0000 7f1b577fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b440051c0 0x7f1b58154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:37.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.706+0000 7f1b577fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b440051c0 0x7f1b58154580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40298/0 (socket says 172.21.15.88:40298) 2024-09-16T01:42:37.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.706+0000 7f1b577fe640 1 -- 172.21.15.88:0/643895770 learned_addr learned my addr 172.21.15.88:0/643895770 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:37.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.706+0000 7f1b57fff640 1 --2- 172.21.15.88:0/643895770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b58153dc0 0x7f1b58151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:37.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.706+0000 7f1b4ffff640 1 --2- 172.21.15.88:0/643895770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1b58161380 0x7f1b58151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:37.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.707+0000 7f1b577fe640 1 -- 172.21.15.88:0/643895770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1b58161380 msgr2=0x7f1b58151a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:37.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.707+0000 7f1b577fe640 1 --2- 172.21.15.88:0/643895770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1b58161380 0x7f1b58151a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:37.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.707+0000 7f1b577fe640 1 -- 172.21.15.88:0/643895770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b58153dc0 msgr2=0x7f1b58151fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:37.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.707+0000 7f1b577fe640 1 --2- 172.21.15.88:0/643895770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b58153dc0 0x7f1b58151fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:37.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.707+0000 7f1b577fe640 1 -- 172.21.15.88:0/643895770 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b581618d0 con 0x7f1b440051c0 2024-09-16T01:42:37.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.707+0000 7f1b57fff640 1 --2- 172.21.15.88:0/643895770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b58153dc0 0x7f1b58151fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:37.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.707+0000 7f1b4ffff640 1 --2- 172.21.15.88:0/643895770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1b58161380 0x7f1b58151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:37.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.707+0000 7f1b577fe640 1 --2- 172.21.15.88:0/643895770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b440051c0 0x7f1b58154580 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f1b4800d770 tx=0x7f1b4800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:37.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.707+0000 7f1b54ff9640 1 -- 172.21.15.88:0/643895770 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b48005270 con 0x7f1b440051c0 2024-09-16T01:42:37.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.707+0000 7f1b54ff9640 1 -- 172.21.15.88:0/643895770 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b48017070 con 0x7f1b440051c0 2024-09-16T01:42:37.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.707+0000 7f1b5dfb9640 1 -- 172.21.15.88:0/643895770 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b580cd390 con 0x7f1b440051c0 2024-09-16T01:42:37.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.707+0000 7f1b54ff9640 1 -- 172.21.15.88:0/643895770 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4800b6b0 con 0x7f1b440051c0 2024-09-16T01:42:37.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.707+0000 7f1b5dfb9640 1 -- 172.21.15.88:0/643895770 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b58153a90 con 0x7f1b440051c0 2024-09-16T01:42:37.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.708+0000 7f1b5dfb9640 1 -- 172.21.15.88:0/643895770 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b580d0f30 con 0x7f1b440051c0 2024-09-16T01:42:37.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.708+0000 7f1b54ff9640 1 -- 172.21.15.88:0/643895770 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1b4800b850 con 0x7f1b440051c0 2024-09-16T01:42:37.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.709+0000 7f1b54ff9640 1 --2- 172.21.15.88:0/643895770 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1b3806c2e0 0x7f1b3806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:37.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.709+0000 7f1b4ffff640 1 --2- 172.21.15.88:0/643895770 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1b3806c2e0 0x7f1b3806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:37.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.709+0000 7f1b54ff9640 1 -- 172.21.15.88:0/643895770 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 7802+0+0 (secure 0 0 0) 0x7f1b4808bb80 con 0x7f1b440051c0 2024-09-16T01:42:37.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.709+0000 7f1b4ffff640 1 --2- 172.21.15.88:0/643895770 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1b3806c2e0 0x7f1b3806e7a0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f1b580d2280 tx=0x7f1b40008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:37.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.713+0000 7f1b54ff9640 1 -- 172.21.15.88:0/643895770 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1b580d0f30 con 0x7f1b440051c0 2024-09-16T01:42:37.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.916+0000 7f1b5dfb9640 1 -- 172.21.15.88:0/643895770 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b580cf730 con 0x7f1b440051c0 2024-09-16T01:42:37.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.917+0000 7f1b54ff9640 1 -- 172.21.15.88:0/643895770 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) ==== 95+0+3566 (secure 0 0 0) 0x7f1b48019310 con 0x7f1b440051c0 2024-09-16T01:42:37.919 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:37.919 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":210,"btime":"2024-09-16T01:42:37:600124+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":210,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:37.600069+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:37.919 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 210 2024-09-16T01:42:37.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.919+0000 7f1b5dfb9640 1 -- 172.21.15.88:0/643895770 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1b3806c2e0 msgr2=0x7f1b3806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:37.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.919+0000 7f1b5dfb9640 1 --2- 172.21.15.88:0/643895770 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1b3806c2e0 0x7f1b3806e7a0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f1b580d2280 tx=0x7f1b40008040 comp rx=0 tx=0).stop 2024-09-16T01:42:37.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.919+0000 7f1b5dfb9640 1 -- 172.21.15.88:0/643895770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b440051c0 msgr2=0x7f1b58154580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:37.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.919+0000 7f1b5dfb9640 1 --2- 172.21.15.88:0/643895770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b440051c0 0x7f1b58154580 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f1b4800d770 tx=0x7f1b4800dc30 comp rx=0 tx=0).stop 2024-09-16T01:42:37.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.919+0000 7f1b5dfb9640 1 -- 172.21.15.88:0/643895770 shutdown_connections 2024-09-16T01:42:37.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.919+0000 7f1b5dfb9640 1 --2- 172.21.15.88:0/643895770 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1b3806c2e0 0x7f1b3806e7a0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:37.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.919+0000 7f1b5dfb9640 1 --2- 172.21.15.88:0/643895770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b58153dc0 0x7f1b58151fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:37.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.919+0000 7f1b5dfb9640 1 --2- 172.21.15.88:0/643895770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1b58161380 0x7f1b58151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:37.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.919+0000 7f1b5dfb9640 1 --2- 172.21.15.88:0/643895770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1b440051c0 0x7f1b58154580 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:37.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.919+0000 7f1b5dfb9640 1 -- 172.21.15.88:0/643895770 >> 172.21.15.88:0/643895770 conn(0x7f1b580b6410 msgr2=0x7f1b580ba140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:37.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.919+0000 7f1b5dfb9640 1 -- 172.21.15.88:0/643895770 shutdown_connections 2024-09-16T01:42:37.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:37.919+0000 7f1b5dfb9640 1 -- 172.21.15.88:0/643895770 wait complete. 2024-09-16T01:42:37.933 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:42:37.933 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:38.028 INFO:tasks.ceph.mds.a.smithi088.stderr: -46> 2024-09-16T01:42:33.274+0000 7f9d4d3c0640 -1 mds.0.cache.ino(0x100) Corrupt inode 0x100: End of buffer [buffer:2] 2024-09-16T01:42:38.028 INFO:tasks.ceph.mds.a.smithi088.stderr: -36> 2024-09-16T01:42:33.275+0000 7f9d4d3c0640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (22) Invalid argument 2024-09-16T01:42:38.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.030+0000 7fe9a4d4b640 1 Processor -- start 2024-09-16T01:42:38.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.030+0000 7fe9a4d4b640 1 -- start start 2024-09-16T01:42:38.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.031+0000 7fe9a4d4b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe9a006e5c0 0x7fe9a006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:38.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.031+0000 7fe9a4d4b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe9a00728e0 0x7fe9a015ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:38.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.031+0000 7fe9a4d4b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9a015fcc0 con 0x7fe9a015f260 2024-09-16T01:42:38.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.031+0000 7fe9a4d4b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe9a0160380 con 0x7fe9a006e5c0 2024-09-16T01:42:38.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.031+0000 7fe9a4d4b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe9a006eee0 con 0x7fe9a00728e0 2024-09-16T01:42:38.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.031+0000 7fe99e59c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe9a006e5c0 0x7fe9a006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:38.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.031+0000 7fe99ed9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe9a015f260 0x7fe9a006bc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50900/0 (socket says 172.21.15.88:50900) 2024-09-16T01:42:38.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.031+0000 7fe99ed9d640 1 -- 172.21.15.88:0/910782366 learned_addr learned my addr 172.21.15.88:0/910782366 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:38.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.032+0000 7fe99e59c640 1 -- 172.21.15.88:0/910782366 >> v2:172.21.15.154:3300/0 conn(0x7fe9a00728e0 msgr2=0x7fe9a015ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:38.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.032+0000 7fe99e59c640 1 --2- 172.21.15.88:0/910782366 >> v2:172.21.15.154:3300/0 conn(0x7fe9a00728e0 0x7fe9a015ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:38.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.032+0000 7fe99e59c640 1 -- 172.21.15.88:0/910782366 >> v1:172.21.15.88:6789/0 conn(0x7fe9a015f260 legacy=0x7fe9a006bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:38.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.032+0000 7fe99e59c640 1 -- 172.21.15.88:0/910782366 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9a015f700 con 0x7fe9a006e5c0 2024-09-16T01:42:38.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.032+0000 7fe99d59a640 1 -- 172.21.15.88:0/910782366 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 464402995 0 0) 0x7fe9a015fcc0 con 0x7fe9a015f260 2024-09-16T01:42:38.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.032+0000 7fe99e59c640 1 --2- 172.21.15.88:0/910782366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9a006e5c0 0x7fe9a006e9a0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fe9a0160500 tx=0x7fe99002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f4a448c8c8d601a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:38.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.033+0000 7fe99d59a640 1 -- 172.21.15.88:0/910782366 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe990037070 con 0x7fe9a006e5c0 2024-09-16T01:42:38.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.033+0000 7fe99d59a640 1 -- 172.21.15.88:0/910782366 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe990032440 con 0x7fe9a006e5c0 2024-09-16T01:42:38.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.033+0000 7fe99d59a640 1 -- 172.21.15.88:0/910782366 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99002ae60 con 0x7fe9a006e5c0 2024-09-16T01:42:38.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.033+0000 7fe9a4d4b640 1 -- 172.21.15.88:0/910782366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9a006e5c0 msgr2=0x7fe9a006e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:38.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.033+0000 7fe9a4d4b640 1 --2- 172.21.15.88:0/910782366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9a006e5c0 0x7fe9a006e9a0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fe9a0160500 tx=0x7fe99002a110 comp rx=0 tx=0).stop 2024-09-16T01:42:38.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.033+0000 7fe9a4d4b640 1 -- 172.21.15.88:0/910782366 shutdown_connections 2024-09-16T01:42:38.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.033+0000 7fe9a4d4b640 1 --2- 172.21.15.88:0/910782366 >> v2:172.21.15.154:3300/0 conn(0x7fe9a00728e0 0x7fe9a015ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:38.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.033+0000 7fe9a4d4b640 1 --2- 172.21.15.88:0/910782366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9a006e5c0 0x7fe9a006e9a0 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:38.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.033+0000 7fe9a4d4b640 1 -- 172.21.15.88:0/910782366 >> 172.21.15.88:0/910782366 conn(0x7fe9a006e1e0 msgr2=0x7fe9a00b67c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:38.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.033+0000 7fe9a4d4b640 1 -- 172.21.15.88:0/910782366 shutdown_connections 2024-09-16T01:42:38.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.033+0000 7fe9a4d4b640 1 -- 172.21.15.88:0/910782366 wait complete. 2024-09-16T01:42:38.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.035+0000 7fe9a4d4b640 1 Processor -- start 2024-09-16T01:42:38.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.035+0000 7fe9a4d4b640 1 -- start start 2024-09-16T01:42:38.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.035+0000 7fe9a4d4b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe9a006e5c0 0x7fe9a00cb140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:38.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.035+0000 7fe9a4d4b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9a00728e0 0x7fe9a01f50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:38.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.035+0000 7fe9a4d4b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9a015f260 0x7fe9a0154e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:38.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.035+0000 7fe9a4d4b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a015c670 con 0x7fe9a00728e0 2024-09-16T01:42:38.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.035+0000 7fe9a4d4b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a0160010 con 0x7fe9a015f260 2024-09-16T01:42:38.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.035+0000 7fe9a4d4b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe9a0161820 con 0x7fe9a006e5c0 2024-09-16T01:42:38.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.035+0000 7fe99dd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9a00728e0 0x7fe9a01f50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:38.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.036+0000 7fe99dd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9a00728e0 0x7fe9a01f50d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40314/0 (socket says 172.21.15.88:40314) 2024-09-16T01:42:38.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.036+0000 7fe99dd9b640 1 -- 172.21.15.88:0/1481477730 learned_addr learned my addr 172.21.15.88:0/1481477730 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:38.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.036+0000 7fe99ed9d640 1 --2- 172.21.15.88:0/1481477730 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9a015f260 0x7fe9a0154e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:38.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.036+0000 7fe99e59c640 1 --2- 172.21.15.88:0/1481477730 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe9a006e5c0 0x7fe9a00cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:38.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.036+0000 7fe99dd9b640 1 -- 172.21.15.88:0/1481477730 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe9a006e5c0 msgr2=0x7fe9a00cb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:38.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.036+0000 7fe99dd9b640 1 --2- 172.21.15.88:0/1481477730 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe9a006e5c0 0x7fe9a00cb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:38.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.036+0000 7fe99dd9b640 1 -- 172.21.15.88:0/1481477730 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9a015f260 msgr2=0x7fe9a0154e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:38.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.036+0000 7fe99dd9b640 1 --2- 172.21.15.88:0/1481477730 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9a015f260 0x7fe9a0154e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:38.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.036+0000 7fe99dd9b640 1 -- 172.21.15.88:0/1481477730 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9a00cb850 con 0x7fe9a00728e0 2024-09-16T01:42:38.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.036+0000 7fe99ed9d640 1 --2- 172.21.15.88:0/1481477730 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9a015f260 0x7fe9a0154e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:38.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.036+0000 7fe99dd9b640 1 --2- 172.21.15.88:0/1481477730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9a00728e0 0x7fe9a01f50d0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fe99400b480 tx=0x7fe99400b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:38.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.036+0000 7fe98effd640 1 -- 172.21.15.88:0/1481477730 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe994014070 con 0x7fe9a00728e0 2024-09-16T01:42:38.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.036+0000 7fe98effd640 1 -- 172.21.15.88:0/1481477730 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe994010870 con 0x7fe9a00728e0 2024-09-16T01:42:38.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.036+0000 7fe9a4d4b640 1 -- 172.21.15.88:0/1481477730 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9a00cbb30 con 0x7fe9a00728e0 2024-09-16T01:42:38.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.036+0000 7fe98effd640 1 -- 172.21.15.88:0/1481477730 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe994010c60 con 0x7fe9a00728e0 2024-09-16T01:42:38.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.036+0000 7fe9a4d4b640 1 -- 172.21.15.88:0/1481477730 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9a00cc0f0 con 0x7fe9a00728e0 2024-09-16T01:42:38.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.038+0000 7fe9a4d4b640 1 -- 172.21.15.88:0/1481477730 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9a006e9a0 con 0x7fe9a00728e0 2024-09-16T01:42:38.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.038+0000 7fe98effd640 1 -- 172.21.15.88:0/1481477730 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe994017070 con 0x7fe9a00728e0 2024-09-16T01:42:38.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.038+0000 7fe98effd640 1 --2- 172.21.15.88:0/1481477730 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe98006c2b0 0x7fe98006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:38.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.038+0000 7fe98effd640 1 -- 172.21.15.88:0/1481477730 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 7802+0+0 (secure 0 0 0) 0x7fe99408ba60 con 0x7fe9a00728e0 2024-09-16T01:42:38.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.038+0000 7fe99e59c640 1 --2- 172.21.15.88:0/1481477730 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe98006c2b0 0x7fe98006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:38.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.039+0000 7fe99e59c640 1 --2- 172.21.15.88:0/1481477730 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe98006c2b0 0x7fe98006e770 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fe990034000 tx=0x7fe99002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:38.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.043+0000 7fe98effd640 1 -- 172.21.15.88:0/1481477730 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe994059a30 con 0x7fe9a00728e0 2024-09-16T01:42:38.057 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:42:38.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.250+0000 7fe9a4d4b640 1 -- 172.21.15.88:0/1481477730 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe9a00b39d0 con 0x7fe9a00728e0 2024-09-16T01:42:38.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.250+0000 7fe98effd640 1 -- 172.21.15.88:0/1481477730 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) ==== 95+0+3566 (secure 0 0 0) 0x7fe994059330 con 0x7fe9a00728e0 2024-09-16T01:42:38.252 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:38.252 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":210,"btime":"2024-09-16T01:42:37:600124+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":210,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:37.600069+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:38.252 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 210 2024-09-16T01:42:38.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.252+0000 7fe9a4d4b640 1 -- 172.21.15.88:0/1481477730 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe98006c2b0 msgr2=0x7fe98006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:38.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.252+0000 7fe9a4d4b640 1 --2- 172.21.15.88:0/1481477730 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe98006c2b0 0x7fe98006e770 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fe990034000 tx=0x7fe99002e000 comp rx=0 tx=0).stop 2024-09-16T01:42:38.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.252+0000 7fe9a4d4b640 1 -- 172.21.15.88:0/1481477730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9a00728e0 msgr2=0x7fe9a01f50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:38.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.252+0000 7fe9a4d4b640 1 --2- 172.21.15.88:0/1481477730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9a00728e0 0x7fe9a01f50d0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fe99400b480 tx=0x7fe99400b940 comp rx=0 tx=0).stop 2024-09-16T01:42:38.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.252+0000 7fe9a4d4b640 1 -- 172.21.15.88:0/1481477730 shutdown_connections 2024-09-16T01:42:38.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.252+0000 7fe9a4d4b640 1 --2- 172.21.15.88:0/1481477730 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe98006c2b0 0x7fe98006e770 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:38.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.252+0000 7fe9a4d4b640 1 --2- 172.21.15.88:0/1481477730 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9a015f260 0x7fe9a0154e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:38.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.252+0000 7fe9a4d4b640 1 --2- 172.21.15.88:0/1481477730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9a00728e0 0x7fe9a01f50d0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:38.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.252+0000 7fe9a4d4b640 1 --2- 172.21.15.88:0/1481477730 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe9a006e5c0 0x7fe9a00cb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:38.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.252+0000 7fe9a4d4b640 1 -- 172.21.15.88:0/1481477730 >> 172.21.15.88:0/1481477730 conn(0x7fe9a006e1e0 msgr2=0x7fe9a006bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:38.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.253+0000 7fe9a4d4b640 1 -- 172.21.15.88:0/1481477730 shutdown_connections 2024-09-16T01:42:38.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.253+0000 7fe9a4d4b640 1 -- 172.21.15.88:0/1481477730 wait complete. 2024-09-16T01:42:38.265 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:42:38.265 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:38.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.361+0000 7fde92378640 1 Processor -- start 2024-09-16T01:42:38.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.362+0000 7fde92378640 1 -- start start 2024-09-16T01:42:38.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.362+0000 7fde92378640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fde8c0cf370 0x7fde8c0cf800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:38.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.362+0000 7fde92378640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde8c15c850 con 0x7fde8c15c470 2024-09-16T01:42:38.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.362+0000 7fde92378640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde8c0b4160 con 0x7fde8c0b52b0 2024-09-16T01:42:38.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.362+0000 7fde92378640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fde8c0b4440 con 0x7fde8c0cf370 2024-09-16T01:42:38.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.363+0000 7fde8b7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fde8c0cf370 0x7fde8c0cf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:38.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.363+0000 7fde8b7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fde8c0cf370 0x7fde8c0cf800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40338/0 (socket says 172.21.15.88:40338) 2024-09-16T01:42:38.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.363+0000 7fde8b7fe640 1 -- 172.21.15.88:0/116501969 learned_addr learned my addr 172.21.15.88:0/116501969 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:38.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.363+0000 7fde8bfff640 1 --1- 172.21.15.88:0/116501969 >> v1:172.21.15.88:6789/0 conn(0x7fde8c15c470 0x7fde8c0b3710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50918/0 (socket says 172.21.15.88:50918) 2024-09-16T01:42:38.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.363+0000 7fde8b7fe640 1 -- 172.21.15.88:0/116501969 >> v1:172.21.15.154:6789/0 conn(0x7fde8c0b52b0 legacy=0x7fde8c0b5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:38.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.363+0000 7fde8b7fe640 1 -- 172.21.15.88:0/116501969 >> v1:172.21.15.88:6789/0 conn(0x7fde8c15c470 legacy=0x7fde8c0b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:38.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.363+0000 7fde8b7fe640 1 -- 172.21.15.88:0/116501969 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde8c0b4720 con 0x7fde8c0cf370 2024-09-16T01:42:38.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.364+0000 7fde8b7fe640 1 --2- 172.21.15.88:0/116501969 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde8c0cf370 0x7fde8c0cf800 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fde740098a0 tx=0x7fde7402a320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfd448c0d49b40df server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:38.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.364+0000 7fde8affd640 1 -- 172.21.15.88:0/116501969 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde74037070 con 0x7fde8c0cf370 2024-09-16T01:42:38.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.364+0000 7fde8affd640 1 -- 172.21.15.88:0/116501969 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde74032640 con 0x7fde8c0cf370 2024-09-16T01:42:38.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.364+0000 7fde8affd640 1 -- 172.21.15.88:0/116501969 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde74032a10 con 0x7fde8c0cf370 2024-09-16T01:42:38.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.364+0000 7fde92378640 1 -- 172.21.15.88:0/116501969 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde8c0cf370 msgr2=0x7fde8c0cf800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:38.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.364+0000 7fde92378640 1 --2- 172.21.15.88:0/116501969 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde8c0cf370 0x7fde8c0cf800 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fde740098a0 tx=0x7fde7402a320 comp rx=0 tx=0).stop 2024-09-16T01:42:38.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.364+0000 7fde92378640 1 -- 172.21.15.88:0/116501969 shutdown_connections 2024-09-16T01:42:38.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.364+0000 7fde92378640 1 --2- 172.21.15.88:0/116501969 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde8c0cf370 0x7fde8c0cf800 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:38.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.364+0000 7fde92378640 1 -- 172.21.15.88:0/116501969 >> 172.21.15.88:0/116501969 conn(0x7fde8c06bc10 msgr2=0x7fde8c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:38.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.364+0000 7fde92378640 1 -- 172.21.15.88:0/116501969 shutdown_connections 2024-09-16T01:42:38.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.364+0000 7fde92378640 1 -- 172.21.15.88:0/116501969 wait complete. 2024-09-16T01:42:38.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.366+0000 7fde92378640 1 Processor -- start 2024-09-16T01:42:38.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.366+0000 7fde92378640 1 -- start start 2024-09-16T01:42:38.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.366+0000 7fde92378640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8c0b52b0 0x7fde8c14d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:38.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.366+0000 7fde92378640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde8c0cf370 0x7fde8c14ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:38.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.366+0000 7fde92378640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde8c15c470 0x7fde8c13ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:38.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.366+0000 7fde92378640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fde8c0b3e20 con 0x7fde8c15c470 2024-09-16T01:42:38.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.366+0000 7fde92378640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fde8c0b4d90 con 0x7fde8c0cf370 2024-09-16T01:42:38.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.366+0000 7fde92378640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fde8c06e1b0 con 0x7fde8c0b52b0 2024-09-16T01:42:38.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.367+0000 7fde90915640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde8c15c470 0x7fde8c13ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:38.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.367+0000 7fde90915640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde8c15c470 0x7fde8c13ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40342/0 (socket says 172.21.15.88:40342) 2024-09-16T01:42:38.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.367+0000 7fde90915640 1 -- 172.21.15.88:0/524257158 learned_addr learned my addr 172.21.15.88:0/524257158 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:38.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.367+0000 7fde8bfff640 1 --2- 172.21.15.88:0/524257158 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8c0b52b0 0x7fde8c14d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:38.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.367+0000 7fde8b7fe640 1 --2- 172.21.15.88:0/524257158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde8c0cf370 0x7fde8c14ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:38.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.367+0000 7fde90915640 1 -- 172.21.15.88:0/524257158 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8c0b52b0 msgr2=0x7fde8c14d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:38.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.367+0000 7fde90915640 1 --2- 172.21.15.88:0/524257158 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8c0b52b0 0x7fde8c14d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:38.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.367+0000 7fde90915640 1 -- 172.21.15.88:0/524257158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde8c0cf370 msgr2=0x7fde8c14ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.367+0000 7fde90915640 1 --2- 172.21.15.88:0/524257158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde8c0cf370 0x7fde8c14ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.367+0000 7fde90915640 1 -- 172.21.15.88:0/524257158 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde8c14dd80 con 0x7fde8c15c470 2024-09-16T01:42:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.367+0000 7fde90915640 1 --2- 172.21.15.88:0/524257158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde8c15c470 0x7fde8c13ae80 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fde8400c760 tx=0x7fde8400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.367+0000 7fde88ff9640 1 -- 172.21.15.88:0/524257158 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84003b70 con 0x7fde8c15c470 2024-09-16T01:42:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.367+0000 7fde88ff9640 1 -- 172.21.15.88:0/524257158 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde84003d30 con 0x7fde8c15c470 2024-09-16T01:42:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.367+0000 7fde92378640 1 -- 172.21.15.88:0/524257158 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde8c14e060 con 0x7fde8c15c470 2024-09-16T01:42:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.367+0000 7fde88ff9640 1 -- 172.21.15.88:0/524257158 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84005310 con 0x7fde8c15c470 2024-09-16T01:42:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.367+0000 7fde92378640 1 -- 172.21.15.88:0/524257158 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde8c14f9f0 con 0x7fde8c15c470 2024-09-16T01:42:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.368+0000 7fde92378640 1 -- 172.21.15.88:0/524257158 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde8c14e580 con 0x7fde8c15c470 2024-09-16T01:42:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.369+0000 7fde88ff9640 1 -- 172.21.15.88:0/524257158 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fde840054b0 con 0x7fde8c15c470 2024-09-16T01:42:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.369+0000 7fde88ff9640 1 --2- 172.21.15.88:0/524257158 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fde6c06c2b0 0x7fde6c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.369+0000 7fde88ff9640 1 -- 172.21.15.88:0/524257158 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 7802+0+0 (secure 0 0 0) 0x7fde8408ba20 con 0x7fde8c15c470 2024-09-16T01:42:38.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.369+0000 7fde8bfff640 1 --2- 172.21.15.88:0/524257158 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fde6c06c2b0 0x7fde6c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:38.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.370+0000 7fde8bfff640 1 --2- 172.21.15.88:0/524257158 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fde6c06c2b0 0x7fde6c06e770 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fde7c000f60 tx=0x7fde7c000ed0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:38.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.373+0000 7fde88ff9640 1 -- 172.21.15.88:0/524257158 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fde84059290 con 0x7fde8c15c470 2024-09-16T01:42:38.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.575+0000 7fde92378640 1 -- 172.21.15.88:0/524257158 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde8c0cf800 con 0x7fde8c15c470 2024-09-16T01:42:38.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.575+0000 7fde88ff9640 1 -- 172.21.15.88:0/524257158 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) ==== 95+0+3566 (secure 0 0 0) 0x7fde8405d590 con 0x7fde8c15c470 2024-09-16T01:42:38.577 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:38.577 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":210,"btime":"2024-09-16T01:42:37:600124+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":210,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:37.600069+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:38.577 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 210 2024-09-16T01:42:38.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.577+0000 7fde92378640 1 -- 172.21.15.88:0/524257158 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fde6c06c2b0 msgr2=0x7fde6c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:38.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.577+0000 7fde92378640 1 --2- 172.21.15.88:0/524257158 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fde6c06c2b0 0x7fde6c06e770 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fde7c000f60 tx=0x7fde7c000ed0 comp rx=0 tx=0).stop 2024-09-16T01:42:38.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.578+0000 7fde92378640 1 -- 172.21.15.88:0/524257158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde8c15c470 msgr2=0x7fde8c13ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:38.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.578+0000 7fde92378640 1 --2- 172.21.15.88:0/524257158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde8c15c470 0x7fde8c13ae80 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fde8400c760 tx=0x7fde8400cc20 comp rx=0 tx=0).stop 2024-09-16T01:42:38.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.578+0000 7fde92378640 1 -- 172.21.15.88:0/524257158 shutdown_connections 2024-09-16T01:42:38.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.578+0000 7fde92378640 1 --2- 172.21.15.88:0/524257158 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fde6c06c2b0 0x7fde6c06e770 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:38.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.578+0000 7fde92378640 1 --2- 172.21.15.88:0/524257158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde8c15c470 0x7fde8c13ae80 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:38.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.578+0000 7fde92378640 1 --2- 172.21.15.88:0/524257158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde8c0cf370 0x7fde8c14ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:38.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.578+0000 7fde92378640 1 --2- 172.21.15.88:0/524257158 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8c0b52b0 0x7fde8c14d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:38.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.578+0000 7fde92378640 1 -- 172.21.15.88:0/524257158 >> 172.21.15.88:0/524257158 conn(0x7fde8c06bc10 msgr2=0x7fde8c0b5720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:38.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.578+0000 7fde92378640 1 -- 172.21.15.88:0/524257158 shutdown_connections 2024-09-16T01:42:38.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.578+0000 7fde92378640 1 -- 172.21.15.88:0/524257158 wait complete. 2024-09-16T01:42:38.590 INFO:tasks.cephfs.test_damage:Result: Mutation 'Truncate 1.00000000.inode' led to DAMAGED state 2024-09-16T01:42:38.590 INFO:tasks.cephfs.test_damage:Applying mutation 'Truncate 200.00000000' 2024-09-16T01:42:38.590 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:42:38.590 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:42:38.607 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:42:38.607 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:42:38.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:38.608 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:42:38.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:42:38.663 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:42:38.663 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:42:38.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.801+0000 7fafc670a640 1 Processor -- start 2024-09-16T01:42:38.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.801+0000 7fafc670a640 1 -- start start 2024-09-16T01:42:38.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.802+0000 7fafc670a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fafc015c520 0x7fafc00b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:38.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.802+0000 7fafc670a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fafc006c320 0x7fafc00704a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:38.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.802+0000 7fafc670a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafc00cf3d0 con 0x7fafc0070040 2024-09-16T01:42:38.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.802+0000 7fafc670a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fafc00cffe0 con 0x7fafc006c320 2024-09-16T01:42:38.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.802+0000 7fafc670a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fafc00cf6b0 con 0x7fafc015c520 2024-09-16T01:42:38.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.802+0000 7fafbf7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fafc0070040 0x7fafc006bc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:50932/0 (socket says 172.21.15.88:50932) 2024-09-16T01:42:38.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.802+0000 7fafbf7fe640 1 -- 172.21.15.88:0/3247745365 learned_addr learned my addr 172.21.15.88:0/3247745365 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:38.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafbffff640 1 --2- 172.21.15.88:0/3247745365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fafc015c520 0x7fafc00b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:38.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafbeffd640 1 -- 172.21.15.88:0/3247745365 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1734371515 0 0) 0x7fafc00cf3d0 con 0x7fafc0070040 2024-09-16T01:42:38.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafc4ca7640 1 --2- 172.21.15.88:0/3247745365 >> v2:172.21.15.154:3300/0 conn(0x7fafc006c320 0x7fafc00704a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:38.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafbeffd640 1 -- 172.21.15.88:0/3247745365 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafa8003520 con 0x7fafc0070040 2024-09-16T01:42:38.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafbeffd640 1 -- 172.21.15.88:0/3247745365 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2408168842 0 0) 0x7fafa8003520 con 0x7fafc0070040 2024-09-16T01:42:38.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafbeffd640 1 -- 172.21.15.88:0/3247745365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fafc015c520 msgr2=0x7fafc00b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:38.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafbeffd640 1 --2- 172.21.15.88:0/3247745365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fafc015c520 0x7fafc00b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:38.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafbeffd640 1 -- 172.21.15.88:0/3247745365 >> v2:172.21.15.154:3300/0 conn(0x7fafc006c320 msgr2=0x7fafc00704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:38.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafbeffd640 1 --2- 172.21.15.88:0/3247745365 >> v2:172.21.15.154:3300/0 conn(0x7fafc006c320 0x7fafc00704a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:38.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafbeffd640 1 -- 172.21.15.88:0/3247745365 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc00cf990 con 0x7fafc0070040 2024-09-16T01:42:38.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafbeffd640 1 -- 172.21.15.88:0/3247745365 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fafb4001670 con 0x7fafc0070040 2024-09-16T01:42:38.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafbeffd640 1 -- 172.21.15.88:0/3247745365 >> v1:172.21.15.88:6789/0 conn(0x7fafc0070040 legacy=0x7fafc006bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:38.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafbeffd640 1 --2- 172.21.15.88:0/3247745365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fafc015c520 0x7fafa80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:38.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafbeffd640 1 --2- 172.21.15.88:0/3247745365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafa8003f30 0x7fafa8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:38.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafbeffd640 1 --2- 172.21.15.88:0/3247745365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa8004d40 0x7fafa80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:38.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafbeffd640 1 -- 172.21.15.88:0/3247745365 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fafc00cffe0 con 0x7fafa8004d40 2024-09-16T01:42:38.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafbeffd640 1 -- 172.21.15.88:0/3247745365 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fafc00cf6b0 con 0x7fafa8003f30 2024-09-16T01:42:38.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafbeffd640 1 -- 172.21.15.88:0/3247745365 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fafa800ba10 con 0x7fafc015c520 2024-09-16T01:42:38.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafc4ca7640 1 --2- 172.21.15.88:0/3247745365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa8004d40 0x7fafa80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:38.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafbffff640 1 --2- 172.21.15.88:0/3247745365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fafc015c520 0x7fafa80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:38.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafbf7fe640 1 --2- 172.21.15.88:0/3247745365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafa8003f30 0x7fafa8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:38.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafc4ca7640 1 -- 172.21.15.88:0/3247745365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fafc015c520 msgr2=0x7fafa80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:38.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafc4ca7640 1 --2- 172.21.15.88:0/3247745365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fafc015c520 0x7fafa80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:38.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafc4ca7640 1 -- 172.21.15.88:0/3247745365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafa8003f30 msgr2=0x7fafa8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:38.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafc4ca7640 1 --2- 172.21.15.88:0/3247745365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafa8003f30 0x7fafa8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafc4ca7640 1 -- 172.21.15.88:0/3247745365 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafa8003520 con 0x7fafa8004d40 2024-09-16T01:42:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.803+0000 7fafbffff640 1 --2- 172.21.15.88:0/3247745365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fafc015c520 0x7fafa80039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.804+0000 7fafc4ca7640 1 --2- 172.21.15.88:0/3247745365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa8004d40 0x7fafa80051c0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7fafb8009f90 tx=0x7fafa8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=217c1690ad6b1cea server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.804+0000 7fafbeffd640 1 -- 172.21.15.88:0/3247745365 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb8009c10 con 0x7fafa8004d40 2024-09-16T01:42:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.804+0000 7fafbeffd640 1 -- 172.21.15.88:0/3247745365 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafb8036070 con 0x7fafa8004d40 2024-09-16T01:42:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.804+0000 7fafbeffd640 1 -- 172.21.15.88:0/3247745365 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb8031880 con 0x7fafa8004d40 2024-09-16T01:42:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.805+0000 7fafc670a640 1 -- 172.21.15.88:0/3247745365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa8004d40 msgr2=0x7fafa80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.805+0000 7fafc670a640 1 --2- 172.21.15.88:0/3247745365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa8004d40 0x7fafa80051c0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7fafb8009f90 tx=0x7fafa8005830 comp rx=0 tx=0).stop 2024-09-16T01:42:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.805+0000 7fafc4ca7640 1 -- 172.21.15.88:0/3247745365 reap_dead start 2024-09-16T01:42:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.805+0000 7fafc670a640 1 -- 172.21.15.88:0/3247745365 shutdown_connections 2024-09-16T01:42:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.805+0000 7fafc670a640 1 -- 172.21.15.88:0/3247745365 >> 172.21.15.88:0/3247745365 conn(0x7fafc00d1390 msgr2=0x7fafc00d17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.805+0000 7fafc670a640 1 -- 172.21.15.88:0/3247745365 shutdown_connections 2024-09-16T01:42:38.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.805+0000 7fafc670a640 1 -- 172.21.15.88:0/3247745365 wait complete. 2024-09-16T01:42:38.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.806+0000 7fafc670a640 1 Processor -- start 2024-09-16T01:42:38.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.806+0000 7fafc670a640 1 -- start start 2024-09-16T01:42:38.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.807+0000 7fafc670a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa8004d40 0x7fafc01594c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:38.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.807+0000 7fafc670a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafc0151fc0 0x7fafc01532d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:38.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.807+0000 7fafc670a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fafc0143c70 0x7fafc0154660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:38.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.807+0000 7fafc670a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fafc00b4300 con 0x7fafa8004d40 2024-09-16T01:42:38.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.807+0000 7fafc670a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fafc00cdb00 con 0x7fafc0151fc0 2024-09-16T01:42:38.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.807+0000 7fafc670a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fafc00b4480 con 0x7fafc0143c70 2024-09-16T01:42:38.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.807+0000 7fafbffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa8004d40 0x7fafc01594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:38.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.807+0000 7fafbffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa8004d40 0x7fafc01594c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40362/0 (socket says 172.21.15.88:40362) 2024-09-16T01:42:38.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.807+0000 7fafbffff640 1 -- 172.21.15.88:0/1420420480 learned_addr learned my addr 172.21.15.88:0/1420420480 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:38.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.807+0000 7fafbf7fe640 1 --2- 172.21.15.88:0/1420420480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafc0151fc0 0x7fafc01532d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:38.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.807+0000 7fafc4ca7640 1 --2- 172.21.15.88:0/1420420480 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fafc0143c70 0x7fafc0154660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:38.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.807+0000 7fafbffff640 1 -- 172.21.15.88:0/1420420480 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fafc0143c70 msgr2=0x7fafc0154660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:38.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.807+0000 7fafbffff640 1 --2- 172.21.15.88:0/1420420480 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fafc0143c70 0x7fafc0154660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:38.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.807+0000 7fafbffff640 1 -- 172.21.15.88:0/1420420480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafc0151fc0 msgr2=0x7fafc01532d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:38.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.807+0000 7fafbffff640 1 --2- 172.21.15.88:0/1420420480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafc0151fc0 0x7fafc01532d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:38.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.807+0000 7fafbffff640 1 -- 172.21.15.88:0/1420420480 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc0153b20 con 0x7fafa8004d40 2024-09-16T01:42:38.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.807+0000 7fafbffff640 1 --2- 172.21.15.88:0/1420420480 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa8004d40 0x7fafc01594c0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fafb0011700 tx=0x7fafb0011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:38.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.808+0000 7fafbcff9640 1 -- 172.21.15.88:0/1420420480 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb000f9c0 con 0x7fafa8004d40 2024-09-16T01:42:38.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.808+0000 7fafbcff9640 1 -- 172.21.15.88:0/1420420480 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafb000fb60 con 0x7fafa8004d40 2024-09-16T01:42:38.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.808+0000 7fafc670a640 1 -- 172.21.15.88:0/1420420480 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafc0153e00 con 0x7fafa8004d40 2024-09-16T01:42:38.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.808+0000 7fafbcff9640 1 -- 172.21.15.88:0/1420420480 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb0015460 con 0x7fafa8004d40 2024-09-16T01:42:38.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.808+0000 7fafc670a640 1 -- 172.21.15.88:0/1420420480 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafc014f6c0 con 0x7fafa8004d40 2024-09-16T01:42:38.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.809+0000 7fafc670a640 1 -- 172.21.15.88:0/1420420480 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafc00704a0 con 0x7fafa8004d40 2024-09-16T01:42:38.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.809+0000 7fafbcff9640 1 -- 172.21.15.88:0/1420420480 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fafb0015600 con 0x7fafa8004d40 2024-09-16T01:42:38.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.810+0000 7fafbcff9640 1 --2- 172.21.15.88:0/1420420480 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fafac06c2b0 0x7fafac06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:38.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.810+0000 7fafbcff9640 1 -- 172.21.15.88:0/1420420480 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 7802+0+0 (secure 0 0 0) 0x7fafb008c5c0 con 0x7fafa8004d40 2024-09-16T01:42:38.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.810+0000 7fafbf7fe640 1 --2- 172.21.15.88:0/1420420480 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fafac06c2b0 0x7fafac06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:38.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.810+0000 7fafbf7fe640 1 --2- 172.21.15.88:0/1420420480 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fafac06c2b0 0x7fafac06e770 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fafb4000c00 tx=0x7fafb4007040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:38.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:38.814+0000 7fafbcff9640 1 -- 172.21.15.88:0/1420420480 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fafb0019030 con 0x7fafa8004d40 2024-09-16T01:42:39.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.013+0000 7fafc670a640 1 -- 172.21.15.88:0/1420420480 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fafc00b4d90 con 0x7fafa8004d40 2024-09-16T01:42:39.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.606+0000 7fafbcff9640 1 -- 172.21.15.88:0/1420420480 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v212) ==== 161+0+0 (secure 0 0 0) 0x7fafb005a500 con 0x7fafa8004d40 2024-09-16T01:42:39.607 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:42:39.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.608+0000 7fafc670a640 1 -- 172.21.15.88:0/1420420480 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fafac06c2b0 msgr2=0x7fafac06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:39.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.608+0000 7fafc670a640 1 --2- 172.21.15.88:0/1420420480 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fafac06c2b0 0x7fafac06e770 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fafb4000c00 tx=0x7fafb4007040 comp rx=0 tx=0).stop 2024-09-16T01:42:39.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.608+0000 7fafc670a640 1 -- 172.21.15.88:0/1420420480 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa8004d40 msgr2=0x7fafc01594c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:39.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.608+0000 7fafc670a640 1 --2- 172.21.15.88:0/1420420480 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa8004d40 0x7fafc01594c0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fafb0011700 tx=0x7fafb0011bc0 comp rx=0 tx=0).stop 2024-09-16T01:42:39.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.608+0000 7fafc670a640 1 -- 172.21.15.88:0/1420420480 shutdown_connections 2024-09-16T01:42:39.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.608+0000 7fafc670a640 1 --2- 172.21.15.88:0/1420420480 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fafac06c2b0 0x7fafac06e770 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:39.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.608+0000 7fafc670a640 1 --2- 172.21.15.88:0/1420420480 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fafc0143c70 0x7fafc0154660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:39.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.608+0000 7fafc670a640 1 --2- 172.21.15.88:0/1420420480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafc0151fc0 0x7fafc01532d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:39.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.608+0000 7fafc670a640 1 --2- 172.21.15.88:0/1420420480 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafa8004d40 0x7fafc01594c0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:39.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.608+0000 7fafc670a640 1 -- 172.21.15.88:0/1420420480 >> 172.21.15.88:0/1420420480 conn(0x7fafc00d1390 msgr2=0x7fafc015f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:39.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.608+0000 7fafc670a640 1 -- 172.21.15.88:0/1420420480 shutdown_connections 2024-09-16T01:42:39.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.608+0000 7fafc670a640 1 -- 172.21.15.88:0/1420420480 wait complete. 2024-09-16T01:42:39.622 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:42:39.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.721+0000 7f63e599f640 1 Processor -- start 2024-09-16T01:42:39.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.721+0000 7f63e599f640 1 -- start start 2024-09-16T01:42:39.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.722+0000 7f63e599f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63e015b480 0x7f63e00b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:39.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.722+0000 7f63e599f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f63e00b60c0 0x7f63e0071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:39.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.722+0000 7f63e599f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63e00ccff0 con 0x7f63e0072530 2024-09-16T01:42:39.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.722+0000 7f63e599f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f63e00b6500 con 0x7f63e00b60c0 2024-09-16T01:42:39.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.722+0000 7f63e599f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f63e0072990 con 0x7f63e015b480 2024-09-16T01:42:39.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.722+0000 7f63de7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f63e00b60c0 0x7f63e0071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:39.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.722+0000 7f63de7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f63e00b60c0 0x7f63e0071ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40378/0 (socket says 172.21.15.88:40378) 2024-09-16T01:42:39.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.722+0000 7f63de7fc640 1 -- 172.21.15.88:0/1787034686 learned_addr learned my addr 172.21.15.88:0/1787034686 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:39.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.723+0000 7f63deffd640 1 --2- 172.21.15.88:0/1787034686 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63e015b480 0x7f63e00b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:39.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.723+0000 7f63de7fc640 1 -- 172.21.15.88:0/1787034686 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63e015b480 msgr2=0x7f63e00b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:39.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.723+0000 7f63de7fc640 1 --2- 172.21.15.88:0/1787034686 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63e015b480 0x7f63e00b5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:39.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.723+0000 7f63de7fc640 1 -- 172.21.15.88:0/1787034686 >> v1:172.21.15.88:6789/0 conn(0x7f63e0072530 legacy=0x7f63e0070fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:39.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.723+0000 7f63de7fc640 1 -- 172.21.15.88:0/1787034686 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63e00cd2d0 con 0x7f63e00b60c0 2024-09-16T01:42:39.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.723+0000 7f63deffd640 1 --2- 172.21.15.88:0/1787034686 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63e015b480 0x7f63e00b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:39.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.723+0000 7f63de7fc640 1 --2- 172.21.15.88:0/1787034686 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63e00b60c0 0x7f63e0071ff0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f63d4009870 tx=0x7f63d402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f99f8b40cc6455fa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:39.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.724+0000 7f63ddffb640 1 -- 172.21.15.88:0/1787034686 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d4037070 con 0x7f63e00b60c0 2024-09-16T01:42:39.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.724+0000 7f63ddffb640 1 -- 172.21.15.88:0/1787034686 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63d4032440 con 0x7f63e00b60c0 2024-09-16T01:42:39.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.724+0000 7f63ddffb640 1 -- 172.21.15.88:0/1787034686 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d4032810 con 0x7f63e00b60c0 2024-09-16T01:42:39.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.724+0000 7f63e599f640 1 -- 172.21.15.88:0/1787034686 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63e00b60c0 msgr2=0x7f63e0071ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:39.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.724+0000 7f63e599f640 1 --2- 172.21.15.88:0/1787034686 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63e00b60c0 0x7f63e0071ff0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f63d4009870 tx=0x7f63d402a1c0 comp rx=0 tx=0).stop 2024-09-16T01:42:39.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.724+0000 7f63e599f640 1 -- 172.21.15.88:0/1787034686 shutdown_connections 2024-09-16T01:42:39.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.724+0000 7f63e599f640 1 --2- 172.21.15.88:0/1787034686 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63e00b60c0 0x7f63e0071ff0 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:39.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.724+0000 7f63e599f640 1 --2- 172.21.15.88:0/1787034686 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63e015b480 0x7f63e00b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:39.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.724+0000 7f63e599f640 1 -- 172.21.15.88:0/1787034686 >> 172.21.15.88:0/1787034686 conn(0x7f63e006d0f0 msgr2=0x7f63e006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:39.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.724+0000 7f63e599f640 1 -- 172.21.15.88:0/1787034686 shutdown_connections 2024-09-16T01:42:39.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.724+0000 7f63e599f640 1 -- 172.21.15.88:0/1787034686 wait complete. 2024-09-16T01:42:39.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.725+0000 7f63e599f640 1 Processor -- start 2024-09-16T01:42:39.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.726+0000 7f63e599f640 1 -- start start 2024-09-16T01:42:39.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.726+0000 7f63e599f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63e0072530 0x7f63e01e6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:39.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.726+0000 7f63e599f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63e00b60c0 0x7f63e01e6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:39.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.726+0000 7f63e599f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63e015b480 0x7f63e01e7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:39.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.726+0000 7f63e599f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f63e00cdb70 con 0x7f63e00b60c0 2024-09-16T01:42:39.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.726+0000 7f63e599f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f63e00cd890 con 0x7f63e0072530 2024-09-16T01:42:39.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.726+0000 7f63e599f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f63e015b860 con 0x7f63e015b480 2024-09-16T01:42:39.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.726+0000 7f63de7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63e00b60c0 0x7f63e01e6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:39.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.726+0000 7f63de7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63e00b60c0 0x7f63e01e6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40382/0 (socket says 172.21.15.88:40382) 2024-09-16T01:42:39.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.726+0000 7f63de7fc640 1 -- 172.21.15.88:0/2528703636 learned_addr learned my addr 172.21.15.88:0/2528703636 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:39.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.727+0000 7f63df7fe640 1 --2- 172.21.15.88:0/2528703636 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63e015b480 0x7f63e01e7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:39.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.727+0000 7f63deffd640 1 --2- 172.21.15.88:0/2528703636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63e0072530 0x7f63e01e6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:39.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.727+0000 7f63de7fc640 1 -- 172.21.15.88:0/2528703636 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63e015b480 msgr2=0x7f63e01e7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:39.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.727+0000 7f63de7fc640 1 --2- 172.21.15.88:0/2528703636 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63e015b480 0x7f63e01e7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:39.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.727+0000 7f63de7fc640 1 -- 172.21.15.88:0/2528703636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63e0072530 msgr2=0x7f63e01e6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:39.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.727+0000 7f63de7fc640 1 --2- 172.21.15.88:0/2528703636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63e0072530 0x7f63e01e6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:39.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.727+0000 7f63de7fc640 1 -- 172.21.15.88:0/2528703636 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63e0154790 con 0x7f63e00b60c0 2024-09-16T01:42:39.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.727+0000 7f63de7fc640 1 --2- 172.21.15.88:0/2528703636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63e00b60c0 0x7f63e01e6df0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f63d4000c00 tx=0x7f63d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:39.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.727+0000 7f63c7fff640 1 -- 172.21.15.88:0/2528703636 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d4037040 con 0x7f63e00b60c0 2024-09-16T01:42:39.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.727+0000 7f63c7fff640 1 -- 172.21.15.88:0/2528703636 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63d4042070 con 0x7f63e00b60c0 2024-09-16T01:42:39.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.727+0000 7f63e599f640 1 -- 172.21.15.88:0/2528703636 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63e0154a70 con 0x7f63e00b60c0 2024-09-16T01:42:39.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.727+0000 7f63c7fff640 1 -- 172.21.15.88:0/2528703636 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d4030bd0 con 0x7f63e00b60c0 2024-09-16T01:42:39.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.727+0000 7f63e599f640 1 -- 172.21.15.88:0/2528703636 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63e0155030 con 0x7f63e00b60c0 2024-09-16T01:42:39.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.728+0000 7f63e599f640 1 -- 172.21.15.88:0/2528703636 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63e0155410 con 0x7f63e00b60c0 2024-09-16T01:42:39.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.728+0000 7f63c7fff640 1 -- 172.21.15.88:0/2528703636 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f63d402ad10 con 0x7f63e00b60c0 2024-09-16T01:42:39.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.729+0000 7f63c7fff640 1 --2- 172.21.15.88:0/2528703636 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f63cc06c2b0 0x7f63cc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:39.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.729+0000 7f63c7fff640 1 -- 172.21.15.88:0/2528703636 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 7802+0+0 (secure 0 0 0) 0x7f63d40ac600 con 0x7f63e00b60c0 2024-09-16T01:42:39.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.729+0000 7f63deffd640 1 --2- 172.21.15.88:0/2528703636 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f63cc06c2b0 0x7f63cc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:39.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.729+0000 7f63deffd640 1 --2- 172.21.15.88:0/2528703636 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f63cc06c2b0 0x7f63cc06e770 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f63d000a950 tx=0x7f63d0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:39.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.733+0000 7f63c7fff640 1 -- 172.21.15.88:0/2528703636 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f63e0155410 con 0x7f63e00b60c0 2024-09-16T01:42:39.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:39.963+0000 7f63e599f640 1 -- 172.21.15.88:0/2528703636 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f63e0071ff0 con 0x7f63e00b60c0 2024-09-16T01:42:40.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.608+0000 7f63c7fff640 1 -- 172.21.15.88:0/2528703636 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v213) ==== 101+0+0 (secure 0 0 0) 0x7f63d407a540 con 0x7f63e00b60c0 2024-09-16T01:42:40.609 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:42:40.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.610+0000 7f63e599f640 1 -- 172.21.15.88:0/2528703636 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f63cc06c2b0 msgr2=0x7f63cc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:40.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.610+0000 7f63e599f640 1 --2- 172.21.15.88:0/2528703636 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f63cc06c2b0 0x7f63cc06e770 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f63d000a950 tx=0x7f63d0008040 comp rx=0 tx=0).stop 2024-09-16T01:42:40.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.610+0000 7f63e599f640 1 -- 172.21.15.88:0/2528703636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63e00b60c0 msgr2=0x7f63e01e6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:40.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.610+0000 7f63e599f640 1 --2- 172.21.15.88:0/2528703636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63e00b60c0 0x7f63e01e6df0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f63d4000c00 tx=0x7f63d40046c0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.610+0000 7f63e599f640 1 -- 172.21.15.88:0/2528703636 shutdown_connections 2024-09-16T01:42:40.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.610+0000 7f63e599f640 1 --2- 172.21.15.88:0/2528703636 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f63cc06c2b0 0x7f63cc06e770 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.610+0000 7f63e599f640 1 --2- 172.21.15.88:0/2528703636 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63e015b480 0x7f63e01e7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.610+0000 7f63e599f640 1 --2- 172.21.15.88:0/2528703636 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63e00b60c0 0x7f63e01e6df0 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.610+0000 7f63e599f640 1 --2- 172.21.15.88:0/2528703636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63e0072530 0x7f63e01e6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.610+0000 7f63e599f640 1 -- 172.21.15.88:0/2528703636 >> 172.21.15.88:0/2528703636 conn(0x7f63e006d0f0 msgr2=0x7f63e00cf5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:40.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.611+0000 7f63e599f640 1 -- 172.21.15.88:0/2528703636 shutdown_connections 2024-09-16T01:42:40.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.611+0000 7f63e599f640 1 -- 172.21.15.88:0/2528703636 wait complete. 2024-09-16T01:42:40.626 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:42:40.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.670+0000 7f63058e3f40 1 Processor -- start 2024-09-16T01:42:40.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.670+0000 7f63058e3f40 1 -- start start 2024-09-16T01:42:40.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.671+0000 7f63058e3f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62f0003630 0x558d5fac2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:40.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.671+0000 7f63058e3f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d5fac5c60 0x558d5fabb820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:40.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.671+0000 7f63058e3f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d5fac63d0 0x558d5fabeb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:40.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.671+0000 7f63058e3f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558d5fac5730 con 0x558d5fac63d0 2024-09-16T01:42:40.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.671+0000 7f63058e3f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558d5f93f510 con 0x558d5fac5c60 2024-09-16T01:42:40.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.671+0000 7f63058e3f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x558d5fac58b0 con 0x7f62f0003630 2024-09-16T01:42:40.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.671+0000 7f6303bda640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d5fac63d0 0x558d5fabeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:40.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.671+0000 7f6303bda640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d5fac63d0 0x558d5fabeb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36794/0 (socket says 172.21.15.88:36794) 2024-09-16T01:42:40.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.671+0000 7f6303bda640 1 -- 172.21.15.88:0/1499770227 learned_addr learned my addr 172.21.15.88:0/1499770227 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:40.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.671+0000 7f63033d9640 1 --2- 172.21.15.88:0/1499770227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62f0003630 0x558d5fac2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:40.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.671+0000 7f62fabd8640 1 --2- 172.21.15.88:0/1499770227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d5fac5c60 0x558d5fabb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:40.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.671+0000 7f6303bda640 1 -- 172.21.15.88:0/1499770227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62f0003630 msgr2=0x558d5fac2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:40.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.671+0000 7f6303bda640 1 --2- 172.21.15.88:0/1499770227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62f0003630 0x558d5fac2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.671+0000 7f6303bda640 1 -- 172.21.15.88:0/1499770227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d5fac5c60 msgr2=0x558d5fabb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:40.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.671+0000 7f6303bda640 1 --2- 172.21.15.88:0/1499770227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d5fac5c60 0x558d5fabb820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.671+0000 7f6303bda640 1 -- 172.21.15.88:0/1499770227 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d5fac4440 con 0x558d5fac63d0 2024-09-16T01:42:40.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.671+0000 7f62fabd8640 1 --2- 172.21.15.88:0/1499770227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d5fac5c60 0x558d5fabb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:40.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.671+0000 7f6303bda640 1 --2- 172.21.15.88:0/1499770227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d5fac63d0 0x558d5fabeb60 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f62fc059040 tx=0x7f62fc04f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33e341cc4a5eab7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:40.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.672+0000 7f63023d7640 1 -- 172.21.15.88:0/1499770227 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc00a100 con 0x558d5fac63d0 2024-09-16T01:42:40.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.672+0000 7f63023d7640 1 -- 172.21.15.88:0/1499770227 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62fc00a2a0 con 0x558d5fac63d0 2024-09-16T01:42:40.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.672+0000 7f63023d7640 1 -- 172.21.15.88:0/1499770227 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc04db10 con 0x558d5fac63d0 2024-09-16T01:42:40.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.672+0000 7f63058e3f40 1 -- 172.21.15.88:0/1499770227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d5fac63d0 msgr2=0x558d5fabeb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:40.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.672+0000 7f63058e3f40 1 --2- 172.21.15.88:0/1499770227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d5fac63d0 0x558d5fabeb60 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f62fc059040 tx=0x7f62fc04f000 comp rx=0 tx=0).stop 2024-09-16T01:42:40.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.672+0000 7f63058e3f40 1 -- 172.21.15.88:0/1499770227 shutdown_connections 2024-09-16T01:42:40.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.672+0000 7f63058e3f40 1 --2- 172.21.15.88:0/1499770227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d5fac63d0 0x558d5fabeb60 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.672+0000 7f63058e3f40 1 --2- 172.21.15.88:0/1499770227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d5fac5c60 0x558d5fabb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.672+0000 7f63058e3f40 1 --2- 172.21.15.88:0/1499770227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62f0003630 0x558d5fac2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.672+0000 7f63058e3f40 1 -- 172.21.15.88:0/1499770227 >> 172.21.15.88:0/1499770227 conn(0x558d5fa18960 msgr2=0x558d5fac2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:40.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.672+0000 7f63058e3f40 1 -- 172.21.15.88:0/1499770227 shutdown_connections 2024-09-16T01:42:40.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.672+0000 7f63058e3f40 1 -- 172.21.15.88:0/1499770227 wait complete. 2024-09-16T01:42:40.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.673+0000 7f63058e3f40 1 Processor -- start 2024-09-16T01:42:40.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.673+0000 7f63058e3f40 1 -- start start 2024-09-16T01:42:40.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.673+0000 7f63058e3f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d5fac5c60 0x558d5fab7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:40.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.673+0000 7f63058e3f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d5fac63d0 0x558d5fac74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:40.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.673+0000 7f63058e3f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62f0003630 0x558d5fac7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:40.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.673+0000 7f63058e3f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558d5fac3780 con 0x558d5fac63d0 2024-09-16T01:42:40.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.673+0000 7f63058e3f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558d5fac4720 con 0x558d5fac5c60 2024-09-16T01:42:40.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.673+0000 7f63058e3f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x558d5fac31c0 con 0x7f62f0003630 2024-09-16T01:42:40.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.674+0000 7f62fabd8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d5fac63d0 0x558d5fac74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:40.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.674+0000 7f62fabd8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d5fac63d0 0x558d5fac74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36798/0 (socket says 172.21.15.88:36798) 2024-09-16T01:42:40.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.674+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 learned_addr learned my addr 172.21.15.88:0/1570610088 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:40.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.674+0000 7f63033d9640 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d5fac5c60 0x558d5fab7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:40.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.674+0000 7f6303bda640 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62f0003630 0x558d5fac7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:40.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.674+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62f0003630 msgr2=0x558d5fac7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:40.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.674+0000 7f62fabd8640 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62f0003630 0x558d5fac7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.674+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d5fac5c60 msgr2=0x558d5fab7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:40.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.674+0000 7f62fabd8640 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558d5fac5c60 0x558d5fab7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.674+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d5fabcc40 con 0x558d5fac63d0 2024-09-16T01:42:40.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.674+0000 7f62fabd8640 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d5fac63d0 0x558d5fac74f0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f62ec009df0 tx=0x7f62ec009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:40.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.674+0000 7f63013d5640 1 -- 172.21.15.88:0/1570610088 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ec01a020 con 0x558d5fac63d0 2024-09-16T01:42:40.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.674+0000 7f63013d5640 1 -- 172.21.15.88:0/1570610088 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62ec016450 con 0x558d5fac63d0 2024-09-16T01:42:40.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.674+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x558d5fabcf20 con 0x558d5fac63d0 2024-09-16T01:42:40.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.674+0000 7f63013d5640 1 -- 172.21.15.88:0/1570610088 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ec016800 con 0x558d5fac63d0 2024-09-16T01:42:40.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.674+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d5fabd4e0 con 0x558d5fac63d0 2024-09-16T01:42:40.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.676+0000 7f63013d5640 1 -- 172.21.15.88:0/1570610088 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f62ec01d020 con 0x558d5fac63d0 2024-09-16T01:42:40.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.676+0000 7f63013d5640 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f62e406ac90 0x7f62e406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:40.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.676+0000 7f63013d5640 1 -- 172.21.15.88:0/1570610088 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 7802+0+0 (secure 0 0 0) 0x7f62ec08afc0 con 0x558d5fac63d0 2024-09-16T01:42:40.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.676+0000 7f63033d9640 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f62e406ac90 0x7f62e406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:40.680 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:42:40.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.676+0000 7f63058e3f40 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x558d5fabc710 0x558d5fb5c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:40.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.676+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb5ca00 con 0x558d5fabc710 2024-09-16T01:42:40.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.676+0000 7f6303bda640 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x558d5fabc710 0x558d5fb5c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:40.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.677+0000 7f63033d9640 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f62e406ac90 0x7f62e406d150 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f62f4001be0 tx=0x7f62f4005040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:40.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.677+0000 7f6303bda640 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x558d5fabc710 0x558d5fb5c4c0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:40.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.679+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v111'722 uv696 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.679+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb5ce10 con 0x558d5fabc710 2024-09-16T01:42:40.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.680+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v111'723 uv723 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.680+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68490 con 0x558d5fabc710 2024-09-16T01:42:40.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.681+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v111'724 uv724 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.683 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:42:40.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.681+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb69840 con 0x558d5fabc710 2024-09-16T01:42:40.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.682+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v111'725 uv725 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.682+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb69be0 con 0x558d5fabc710 2024-09-16T01:42:40.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.683+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v111'726 uv726 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.683+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68830 con 0x558d5fabc710 2024-09-16T01:42:40.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.684+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v111'727 uv727 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.684+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68bd0 con 0x558d5fabc710 2024-09-16T01:42:40.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.685+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v111'728 uv702 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.686+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68f70 con 0x558d5fabc710 2024-09-16T01:42:40.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.687+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v111'729 uv729 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.687+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68f70 con 0x558d5fabc710 2024-09-16T01:42:40.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.688+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v111'730 uv730 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.689 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:42:40.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.688+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68f70 con 0x558d5fabc710 2024-09-16T01:42:40.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.689+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v111'731 uv731 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.689+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68f70 con 0x558d5fabc710 2024-09-16T01:42:40.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.690+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v111'732 uv732 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.690+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68f70 con 0x558d5fabc710 2024-09-16T01:42:40.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.692+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v111'733 uv733 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.692+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68f70 con 0x558d5fabc710 2024-09-16T01:42:40.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.693+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v111'734 uv709 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.693+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68f70 con 0x558d5fabc710 2024-09-16T01:42:40.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.695+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v111'735 uv735 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.695+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68f70 con 0x558d5fabc710 2024-09-16T01:42:40.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.696+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v111'736 uv736 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.697 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:42:40.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.696+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68f70 con 0x558d5fabc710 2024-09-16T01:42:40.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.697+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v111'737 uv737 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.697+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68f70 con 0x558d5fabc710 2024-09-16T01:42:40.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.698+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v111'738 uv738 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.698+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68f70 con 0x558d5fabc710 2024-09-16T01:42:40.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.700+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v111'739 uv739 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.700+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68f70 con 0x558d5fabc710 2024-09-16T01:42:40.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.701+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v111'740 uv740 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.701+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68f70 con 0x558d5fabc710 2024-09-16T01:42:40.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.702+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v111'741 uv715 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.702+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68f70 con 0x558d5fabc710 2024-09-16T01:42:40.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.703+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v111'742 uv742 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.703+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68f70 con 0x558d5fabc710 2024-09-16T01:42:40.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.704+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v111'743 uv743 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.706 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:42:40.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.704+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68f70 con 0x558d5fabc710 2024-09-16T01:42:40.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.706+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v111'744 uv744 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.706+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68f70 con 0x558d5fabc710 2024-09-16T01:42:40.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.707+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v111'745 uv745 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.707+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68f70 con 0x558d5fabc710 2024-09-16T01:42:40.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.708+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v111'746 uv746 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.708+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68f70 con 0x558d5fabc710 2024-09-16T01:42:40.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.709+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v111'747 uv721 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.709+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68f70 con 0x558d5fabc710 2024-09-16T01:42:40.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.710+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v111'748 uv748 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.711+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68f70 con 0x558d5fabc710 2024-09-16T01:42:40.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.712+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v111'749 uv749 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.713 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:42:40.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.712+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68f70 con 0x558d5fabc710 2024-09-16T01:42:40.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.713+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v111'750 uv750 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.713+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68f70 con 0x558d5fabc710 2024-09-16T01:42:40.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.714+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v111'751 uv751 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.714+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb68f70 con 0x558d5fabc710 2024-09-16T01:42:40.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.715+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v111'752 uv752 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.715+0000 7f63058e3f40 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x558d5fb69f80 0x558d5fb6c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:40.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.715+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6c8b0 con 0x558d5fb69f80 2024-09-16T01:42:40.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.715+0000 7f63033d9640 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x558d5fb69f80 0x558d5fb6c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:40.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.716+0000 7f63033d9640 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x558d5fb69f80 0x558d5fb6c370 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:40.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.717+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v111'528 uv527 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb69f80 2024-09-16T01:42:40.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.717+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6cc50 con 0x558d5fb69f80 2024-09-16T01:42:40.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.719+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v111'529 uv529 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb69f80 2024-09-16T01:42:40.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.719+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6cff0 con 0x558d5fb69f80 2024-09-16T01:42:40.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.720+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v111'530 uv530 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb69f80 2024-09-16T01:42:40.721 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:42:40.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.720+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6d840 con 0x558d5fb69f80 2024-09-16T01:42:40.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.721+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v111'531 uv531 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb69f80 2024-09-16T01:42:40.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.721+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6dbe0 con 0x558d5fb69f80 2024-09-16T01:42:40.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.722+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v111'532 uv532 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb69f80 2024-09-16T01:42:40.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.722+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6df80 con 0x558d5fb69f80 2024-09-16T01:42:40.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.723+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v111'533 uv515 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb69f80 2024-09-16T01:42:40.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.723+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6e320 con 0x558d5fb69f80 2024-09-16T01:42:40.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.724+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v111'534 uv534 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb69f80 2024-09-16T01:42:40.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.724+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6e6c0 con 0x558d5fb69f80 2024-09-16T01:42:40.726 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:42:40.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.725+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v111'535 uv535 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb69f80 2024-09-16T01:42:40.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.725+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6e6c0 con 0x558d5fb69f80 2024-09-16T01:42:40.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.727+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v111'536 uv536 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb69f80 2024-09-16T01:42:40.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.727+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6e6c0 con 0x558d5fb69f80 2024-09-16T01:42:40.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.728+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v111'537 uv537 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb69f80 2024-09-16T01:42:40.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.728+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6e6c0 con 0x558d5fb69f80 2024-09-16T01:42:40.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.729+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v111'538 uv520 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb69f80 2024-09-16T01:42:40.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.729+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6e6c0 con 0x558d5fb69f80 2024-09-16T01:42:40.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.731+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v111'539 uv539 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb69f80 2024-09-16T01:42:40.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.731+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6e6c0 con 0x558d5fb69f80 2024-09-16T01:42:40.733 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:42:40.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.732+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v111'540 uv540 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb69f80 2024-09-16T01:42:40.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.732+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb72b40 con 0x558d5fb69f80 2024-09-16T01:42:40.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.733+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v111'541 uv541 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb69f80 2024-09-16T01:42:40.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.733+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb72b40 con 0x558d5fb69f80 2024-09-16T01:42:40.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.734+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v111'542 uv542 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb69f80 2024-09-16T01:42:40.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.734+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6e6c0 con 0x558d5fb69f80 2024-09-16T01:42:40.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.736+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v111'543 uv526 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb69f80 2024-09-16T01:42:40.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.736+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6e6c0 con 0x558d5fb69f80 2024-09-16T01:42:40.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.737+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v111'544 uv544 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb69f80 2024-09-16T01:42:40.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.737+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6e6c0 con 0x558d5fb69f80 2024-09-16T01:42:40.740 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:42:40.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.739+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v111'545 uv545 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb69f80 2024-09-16T01:42:40.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.739+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6f6d0 con 0x558d5fb69f80 2024-09-16T01:42:40.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.740+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v111'546 uv546 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb69f80 2024-09-16T01:42:40.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.740+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6f6d0 con 0x558d5fb69f80 2024-09-16T01:42:40.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.741+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v111'547 uv547 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb69f80 2024-09-16T01:42:40.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.741+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6e6c0 con 0x558d5fb69f80 2024-09-16T01:42:40.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.742+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v111'548 uv548 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb69f80 2024-09-16T01:42:40.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.742+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6e6c0 con 0x558d5fabc710 2024-09-16T01:42:40.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.743+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v111'141 uv140 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.743+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6e6c0 con 0x558d5fabc710 2024-09-16T01:42:40.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.744+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v111'142 uv142 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.744+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6e6c0 con 0x558d5fabc710 2024-09-16T01:42:40.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.745+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v111'143 uv143 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.746 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:42:40.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.745+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6f6d0 con 0x558d5fabc710 2024-09-16T01:42:40.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.746+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v111'144 uv144 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.746+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6f6d0 con 0x558d5fabc710 2024-09-16T01:42:40.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.747+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v111'145 uv145 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.747+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6e6c0 con 0x558d5fabc710 2024-09-16T01:42:40.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.748+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v111'146 uv146 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.748+0000 7f63058e3f40 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x558d5fb6ef90 0x558d5fb71380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:40.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.748+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb718c0 con 0x558d5fb6ef90 2024-09-16T01:42:40.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.748+0000 7f62fabd8640 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x558d5fb6ef90 0x558d5fb71380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:40.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.748+0000 7f62fabd8640 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x558d5fb6ef90 0x558d5fb71380 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:40.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.749+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v111'105 uv104 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f62ec026070 con 0x558d5fb6ef90 2024-09-16T01:42:40.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.749+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb71d40 con 0x558d5fb6ef90 2024-09-16T01:42:40.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.750+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v111'106 uv106 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f62ec026070 con 0x558d5fb6ef90 2024-09-16T01:42:40.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.750+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb72240 con 0x558d5fb6ef90 2024-09-16T01:42:40.752 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:42:40.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.751+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v111'107 uv107 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f62ec026070 con 0x558d5fb6ef90 2024-09-16T01:42:40.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.751+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb6d470 con 0x558d5fb6ef90 2024-09-16T01:42:40.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.752+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v111'108 uv108 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f62ec026070 con 0x558d5fb6ef90 2024-09-16T01:42:40.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.752+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb72b70 con 0x558d5fabc710 2024-09-16T01:42:40.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.753+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v111'141 uv140 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.753+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb72b70 con 0x558d5fabc710 2024-09-16T01:42:40.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.754+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v111'142 uv142 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.754+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb72b70 con 0x558d5fabc710 2024-09-16T01:42:40.756 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:42:40.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.755+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v111'143 uv143 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.755+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb73bc0 con 0x558d5fabc710 2024-09-16T01:42:40.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.756+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v111'144 uv144 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.756+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb73b80 con 0x558d5fabc710 2024-09-16T01:42:40.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.758+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v111'145 uv145 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.758+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb72b70 con 0x558d5fabc710 2024-09-16T01:42:40.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.760+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v111'146 uv146 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.760+0000 7f63058e3f40 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x558d5fb735b0 0x558d5fb739d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:40.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.760+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb75fb0 con 0x558d5fb735b0 2024-09-16T01:42:40.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.761+0000 7f6303bda640 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x558d5fb735b0 0x558d5fb739d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:40.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.762+0000 7f6303bda640 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x558d5fb735b0 0x558d5fb739d0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:40.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.763+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v111'260 uv254 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.763+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb75fb0 con 0x558d5fb735b0 2024-09-16T01:42:40.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.765+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v111'261 uv261 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.765+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb75fb0 con 0x558d5fb735b0 2024-09-16T01:42:40.767 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:42:40.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.766+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v111'262 uv262 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.766+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb77000 con 0x558d5fb735b0 2024-09-16T01:42:40.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.767+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v111'263 uv263 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.767+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb76fc0 con 0x558d5fb735b0 2024-09-16T01:42:40.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.768+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v111'264 uv264 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.768+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb75fb0 con 0x558d5fb735b0 2024-09-16T01:42:40.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.769+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v111'265 uv265 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.769+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb75fb0 con 0x558d5fb735b0 2024-09-16T01:42:40.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.770+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v111'266 uv259 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.770+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb75fb0 con 0x558d5fb735b0 2024-09-16T01:42:40.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.771+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v111'267 uv267 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.771+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb75fb0 con 0x558d5fb735b0 2024-09-16T01:42:40.773 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:42:40.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.772+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v111'268 uv268 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.772+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb75fb0 con 0x558d5fb735b0 2024-09-16T01:42:40.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.773+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v111'269 uv269 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.773+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb75fb0 con 0x558d5fb735b0 2024-09-16T01:42:40.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.774+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v111'270 uv270 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.774+0000 7f63058e3f40 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x558d5fb76900 0x558d5fb76d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:40.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.774+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb793f0 con 0x558d5fb76900 2024-09-16T01:42:40.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.774+0000 7f63033d9640 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x558d5fb76900 0x558d5fb76d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:40.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.775+0000 7f63033d9640 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x558d5fb76900 0x558d5fb76d00 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:40.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.776+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v111'117 uv116 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb76900 2024-09-16T01:42:40.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.776+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb793f0 con 0x558d5fb76900 2024-09-16T01:42:40.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.777+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v111'118 uv118 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb76900 2024-09-16T01:42:40.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.777+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb79430 con 0x558d5fb76900 2024-09-16T01:42:40.779 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:42:40.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.778+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v111'119 uv119 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb76900 2024-09-16T01:42:40.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.778+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb79820 con 0x558d5fb76900 2024-09-16T01:42:40.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.779+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v111'120 uv120 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb76900 2024-09-16T01:42:40.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.779+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb79820 con 0x558d5fb76900 2024-09-16T01:42:40.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.780+0000 7f63033d9640 1 -- 172.21.15.88:0/1570610088 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v111'121 uv121 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f62f4003b70 con 0x558d5fb76900 2024-09-16T01:42:40.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.780+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb79820 con 0x558d5fabc710 2024-09-16T01:42:40.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.781+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v111'141 uv140 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.781+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb79820 con 0x558d5fabc710 2024-09-16T01:42:40.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.782+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v111'142 uv142 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.782+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb79820 con 0x558d5fabc710 2024-09-16T01:42:40.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.783+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v111'143 uv143 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.784 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:42:40.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.783+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7a830 con 0x558d5fabc710 2024-09-16T01:42:40.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.784+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v111'144 uv144 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.784+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7a830 con 0x558d5fabc710 2024-09-16T01:42:40.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.785+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v111'145 uv145 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.785+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb79820 con 0x558d5fabc710 2024-09-16T01:42:40.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.786+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v111'146 uv146 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fabc710 2024-09-16T01:42:40.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.786+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb79820 con 0x558d5fb6ef90 2024-09-16T01:42:40.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.787+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v111'233 uv227 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f62ec026070 con 0x558d5fb6ef90 2024-09-16T01:42:40.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.787+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb79bc0 con 0x558d5fb6ef90 2024-09-16T01:42:40.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.788+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v111'234 uv234 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f62ec026070 con 0x558d5fb6ef90 2024-09-16T01:42:40.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.788+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb79f60 con 0x558d5fb6ef90 2024-09-16T01:42:40.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.789+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v111'235 uv235 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f62ec026070 con 0x558d5fb6ef90 2024-09-16T01:42:40.790 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:42:40.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.789+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7a300 con 0x558d5fb6ef90 2024-09-16T01:42:40.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.790+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v111'236 uv236 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f62ec026070 con 0x558d5fb6ef90 2024-09-16T01:42:40.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.790+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7a300 con 0x558d5fb6ef90 2024-09-16T01:42:40.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.790+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v111'237 uv237 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f62ec026070 con 0x558d5fb6ef90 2024-09-16T01:42:40.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.791+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7a300 con 0x558d5fb6ef90 2024-09-16T01:42:40.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.791+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v111'238 uv232 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f62ec026070 con 0x558d5fb6ef90 2024-09-16T01:42:40.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.791+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7a300 con 0x558d5fb6ef90 2024-09-16T01:42:40.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.792+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v111'239 uv239 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62ec026070 con 0x558d5fb6ef90 2024-09-16T01:42:40.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.792+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7a300 con 0x558d5fb6ef90 2024-09-16T01:42:40.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.793+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v111'240 uv240 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62ec026070 con 0x558d5fb6ef90 2024-09-16T01:42:40.794 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:42:40.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.793+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7a300 con 0x558d5fb6ef90 2024-09-16T01:42:40.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.794+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v111'241 uv241 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62ec026070 con 0x558d5fb6ef90 2024-09-16T01:42:40.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.794+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7d0f0 con 0x558d5fb6ef90 2024-09-16T01:42:40.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.795+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v111'242 uv242 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62ec026070 con 0x558d5fb6ef90 2024-09-16T01:42:40.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.795+0000 7f63058e3f40 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x558d5fb7ae10 0x558d5fb7d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:40.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.795+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7d7b0 con 0x558d5fb7ae10 2024-09-16T01:42:40.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.795+0000 7f62fabd8640 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x558d5fb7ae10 0x558d5fb7d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:40.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.796+0000 7f62fabd8640 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x558d5fb7ae10 0x558d5fb7d200 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:40.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.797+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v111'103 uv102 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f62ec026070 con 0x558d5fb7ae10 2024-09-16T01:42:40.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.797+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7d7b0 con 0x558d5fb7ae10 2024-09-16T01:42:40.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.798+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v111'104 uv104 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f62ec026070 con 0x558d5fb7ae10 2024-09-16T01:42:40.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.798+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7d7b0 con 0x558d5fb7ae10 2024-09-16T01:42:40.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.799+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v111'105 uv105 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f62ec026070 con 0x558d5fb7ae10 2024-09-16T01:42:40.800 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:42:40.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.799+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7dba0 con 0x558d5fb7ae10 2024-09-16T01:42:40.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.800+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v111'106 uv106 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f62ec026070 con 0x558d5fb7ae10 2024-09-16T01:42:40.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.800+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7dba0 con 0x558d5fb6ef90 2024-09-16T01:42:40.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.801+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v111'141 uv140 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f62ec026070 con 0x558d5fb6ef90 2024-09-16T01:42:40.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.801+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7dba0 con 0x558d5fb6ef90 2024-09-16T01:42:40.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.802+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v111'142 uv142 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62ec026070 con 0x558d5fb6ef90 2024-09-16T01:42:40.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.802+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7dba0 con 0x558d5fb6ef90 2024-09-16T01:42:40.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.804+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v111'143 uv143 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62ec026070 con 0x558d5fb6ef90 2024-09-16T01:42:40.805 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:42:40.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.804+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7ebb0 con 0x558d5fb6ef90 2024-09-16T01:42:40.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.805+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v111'144 uv144 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62ec026070 con 0x558d5fb6ef90 2024-09-16T01:42:40.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.805+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7ebb0 con 0x558d5fb6ef90 2024-09-16T01:42:40.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.806+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v111'145 uv145 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62ec026070 con 0x558d5fb6ef90 2024-09-16T01:42:40.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.806+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7dba0 con 0x558d5fb6ef90 2024-09-16T01:42:40.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.807+0000 7f62fabd8640 1 -- 172.21.15.88:0/1570610088 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v111'146 uv146 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62ec026070 con 0x558d5fb6ef90 2024-09-16T01:42:40.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.807+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7dba0 con 0x558d5fb735b0 2024-09-16T01:42:40.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.808+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v111'120 uv119 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.808+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7dba0 con 0x558d5fb735b0 2024-09-16T01:42:40.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.809+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v111'121 uv121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.810+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7dba0 con 0x558d5fb735b0 2024-09-16T01:42:40.811 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:42:40.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.810+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v111'122 uv122 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.810+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7dba0 con 0x558d5fb735b0 2024-09-16T01:42:40.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.811+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v111'123 uv123 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.811+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7dba0 con 0x558d5fb735b0 2024-09-16T01:42:40.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.813+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v111'124 uv124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.813+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7dba0 con 0x558d5fb735b0 2024-09-16T01:42:40.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.814+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v111'290 uv282 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.814+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7dba0 con 0x558d5fb735b0 2024-09-16T01:42:40.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.815+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v111'291 uv291 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.815+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7dba0 con 0x558d5fb735b0 2024-09-16T01:42:40.817 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:42:40.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.816+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v111'292 uv292 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.816+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7dba0 con 0x558d5fb735b0 2024-09-16T01:42:40.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.817+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v111'293 uv293 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.817+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7dba0 con 0x558d5fb735b0 2024-09-16T01:42:40.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.818+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v111'294 uv294 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.818+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7dba0 con 0x558d5fb735b0 2024-09-16T01:42:40.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.819+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v111'295 uv289 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.819+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7dba0 con 0x558d5fb735b0 2024-09-16T01:42:40.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.820+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v111'296 uv296 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.820+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7dba0 con 0x558d5fb735b0 2024-09-16T01:42:40.822 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:42:40.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.821+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v111'297 uv297 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.821+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7ebb0 con 0x558d5fb735b0 2024-09-16T01:42:40.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.822+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v111'298 uv298 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.822+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7ebb0 con 0x558d5fb735b0 2024-09-16T01:42:40.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.823+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v111'299 uv299 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.823+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7dba0 con 0x558d5fb735b0 2024-09-16T01:42:40.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.824+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v111'300 uv300 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.824+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x558d5fb7e050 con 0x558d5fb735b0 2024-09-16T01:42:40.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.825+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v111'301 uv301 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f62fc04d8d0 con 0x558d5fb735b0 2024-09-16T01:42:40.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.825+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x558d5fabc710 msgr2=0x558d5fb5c4c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:40.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.825+0000 7f63058e3f40 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x558d5fabc710 0x558d5fb5c4c0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.825+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x558d5fb6ef90 msgr2=0x558d5fb71380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:40.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.825+0000 7f63058e3f40 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x558d5fb6ef90 0x558d5fb71380 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.825+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x558d5fb7ae10 msgr2=0x558d5fb7d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:40.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.825+0000 7f63058e3f40 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x558d5fb7ae10 0x558d5fb7d200 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.825+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x558d5fb69f80 msgr2=0x558d5fb6c370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:40.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.825+0000 7f63058e3f40 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x558d5fb69f80 0x558d5fb6c370 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.825+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 reap_dead start 2024-09-16T01:42:40.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.825+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x558d5fb735b0 msgr2=0x558d5fb739d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:40.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.825+0000 7f63058e3f40 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x558d5fb735b0 0x558d5fb739d0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.825+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x558d5fb76900 msgr2=0x558d5fb76d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:40.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.825+0000 7f63058e3f40 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x558d5fb76900 0x558d5fb76d00 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.825+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f62e406ac90 msgr2=0x7f62e406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:40.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.825+0000 7f63058e3f40 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f62e406ac90 0x7f62e406d150 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f62f4001be0 tx=0x7f62f4005040 comp rx=0 tx=0).stop 2024-09-16T01:42:40.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.825+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d5fac63d0 msgr2=0x558d5fac74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:40.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.825+0000 7f63058e3f40 1 --2- 172.21.15.88:0/1570610088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558d5fac63d0 0x558d5fac74f0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f62ec009df0 tx=0x7f62ec009ee0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.825+0000 7f6303bda640 1 -- 172.21.15.88:0/1570610088 reap_dead start 2024-09-16T01:42:40.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.825+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 shutdown_connections 2024-09-16T01:42:40.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.825+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 >> 172.21.15.88:0/1570610088 conn(0x558d5fa18960 msgr2=0x558d5fa35750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:40.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.825+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 shutdown_connections 2024-09-16T01:42:40.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.825+0000 7f63058e3f40 1 -- 172.21.15.88:0/1570610088 wait complete. 2024-09-16T01:42:40.831 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata truncate 200.00000000 0 2024-09-16T01:42:40.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.873+0000 7fce3d363f40 1 Processor -- start 2024-09-16T01:42:40.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.874+0000 7fce3d363f40 1 -- start start 2024-09-16T01:42:40.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.874+0000 7fce3d363f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce28003bd0 0x5638c280ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:40.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.874+0000 7fce3d363f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638c288ec30 0x5638c28052b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:40.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.874+0000 7fce3d363f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5638c288f070 0x5638c28057f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:40.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.874+0000 7fce3d363f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5638c2713510 con 0x5638c288ec30 2024-09-16T01:42:40.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.874+0000 7fce3d363f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5638c276cc50 con 0x5638c288f070 2024-09-16T01:42:40.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.874+0000 7fce3d363f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5638c280baa0 con 0x7fce28003bd0 2024-09-16T01:42:40.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.874+0000 7fce32658640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638c288ec30 0x5638c28052b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:40.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.874+0000 7fce32658640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638c288ec30 0x5638c28052b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36814/0 (socket says 172.21.15.88:36814) 2024-09-16T01:42:40.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.874+0000 7fce32658640 1 -- 172.21.15.88:0/2128328329 learned_addr learned my addr 172.21.15.88:0/2128328329 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:40.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.874+0000 7fce3ae59640 1 --2- 172.21.15.88:0/2128328329 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce28003bd0 0x5638c280ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:40.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.874+0000 7fce3b65a640 1 --2- 172.21.15.88:0/2128328329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5638c288f070 0x5638c28057f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:40.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.875+0000 7fce32658640 1 -- 172.21.15.88:0/2128328329 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce28003bd0 msgr2=0x5638c280ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:40.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.875+0000 7fce32658640 1 --2- 172.21.15.88:0/2128328329 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce28003bd0 0x5638c280ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.875+0000 7fce32658640 1 -- 172.21.15.88:0/2128328329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5638c288f070 msgr2=0x5638c28057f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:40.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.875+0000 7fce32658640 1 --2- 172.21.15.88:0/2128328329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5638c288f070 0x5638c28057f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.875+0000 7fce32658640 1 -- 172.21.15.88:0/2128328329 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638c2898b10 con 0x5638c288ec30 2024-09-16T01:42:40.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.875+0000 7fce32658640 1 --2- 172.21.15.88:0/2128328329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638c288ec30 0x5638c28052b0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fce24003e90 tx=0x7fce2400e760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a7002d2b5d3893c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:40.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.875+0000 7fce39e57640 1 -- 172.21.15.88:0/2128328329 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce24015020 con 0x5638c288ec30 2024-09-16T01:42:40.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.875+0000 7fce39e57640 1 -- 172.21.15.88:0/2128328329 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce240029c0 con 0x5638c288ec30 2024-09-16T01:42:40.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.875+0000 7fce39e57640 1 -- 172.21.15.88:0/2128328329 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce24002db0 con 0x5638c288ec30 2024-09-16T01:42:40.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.875+0000 7fce3d363f40 1 -- 172.21.15.88:0/2128328329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638c288ec30 msgr2=0x5638c28052b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:40.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.875+0000 7fce3d363f40 1 --2- 172.21.15.88:0/2128328329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638c288ec30 0x5638c28052b0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fce24003e90 tx=0x7fce2400e760 comp rx=0 tx=0).stop 2024-09-16T01:42:40.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.875+0000 7fce3d363f40 1 -- 172.21.15.88:0/2128328329 shutdown_connections 2024-09-16T01:42:40.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.875+0000 7fce3d363f40 1 --2- 172.21.15.88:0/2128328329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5638c288f070 0x5638c28057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.875+0000 7fce3d363f40 1 --2- 172.21.15.88:0/2128328329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5638c288ec30 0x5638c28052b0 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.875+0000 7fce3d363f40 1 --2- 172.21.15.88:0/2128328329 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce28003bd0 0x5638c280ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.875+0000 7fce3d363f40 1 -- 172.21.15.88:0/2128328329 >> 172.21.15.88:0/2128328329 conn(0x5638c27ec960 msgr2=0x5638c27c0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:40.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.875+0000 7fce3d363f40 1 -- 172.21.15.88:0/2128328329 shutdown_connections 2024-09-16T01:42:40.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.875+0000 7fce3d363f40 1 -- 172.21.15.88:0/2128328329 wait complete. 2024-09-16T01:42:40.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.876+0000 7fce3d363f40 1 Processor -- start 2024-09-16T01:42:40.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.876+0000 7fce3d363f40 1 -- start start 2024-09-16T01:42:40.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.877+0000 7fce3d363f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5638c288ec30 0x5638c288b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:40.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.877+0000 7fce3d363f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5638c288f070 0x5638c288f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:40.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.877+0000 7fce3d363f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce28003bd0 0x5638c288d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:40.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.877+0000 7fce3d363f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5638c28963c0 con 0x7fce28003bd0 2024-09-16T01:42:40.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.877+0000 7fce3d363f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5638c280b0c0 con 0x5638c288ec30 2024-09-16T01:42:40.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.877+0000 7fce3d363f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5638c2898db0 con 0x5638c288f070 2024-09-16T01:42:40.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.877+0000 7fce3b65a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce28003bd0 0x5638c288d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:40.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.877+0000 7fce32658640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5638c288f070 0x5638c288f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:40.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.877+0000 7fce3ae59640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5638c288ec30 0x5638c288b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:40.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.877+0000 7fce3b65a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce28003bd0 0x5638c288d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36828/0 (socket says 172.21.15.88:36828) 2024-09-16T01:42:40.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.877+0000 7fce3b65a640 1 -- 172.21.15.88:0/413148310 learned_addr learned my addr 172.21.15.88:0/413148310 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:40.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.877+0000 7fce3b65a640 1 -- 172.21.15.88:0/413148310 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5638c288f070 msgr2=0x5638c288f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:40.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.877+0000 7fce3b65a640 1 --2- 172.21.15.88:0/413148310 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5638c288f070 0x5638c288f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.877+0000 7fce3b65a640 1 -- 172.21.15.88:0/413148310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5638c288ec30 msgr2=0x5638c288b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:40.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.877+0000 7fce3b65a640 1 --2- 172.21.15.88:0/413148310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5638c288ec30 0x5638c288b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.877+0000 7fce3b65a640 1 -- 172.21.15.88:0/413148310 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638c288e230 con 0x7fce28003bd0 2024-09-16T01:42:40.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.877+0000 7fce3ae59640 1 --2- 172.21.15.88:0/413148310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5638c288ec30 0x5638c288b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:42:40.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.877+0000 7fce3b65a640 1 --2- 172.21.15.88:0/413148310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce28003bd0 0x5638c288d140 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7fce34006d50 tx=0x5638c27c51f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:40.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.877+0000 7fce38e55640 1 -- 172.21.15.88:0/413148310 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce340069a0 con 0x7fce28003bd0 2024-09-16T01:42:40.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.877+0000 7fce38e55640 1 -- 172.21.15.88:0/413148310 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce340019b0 con 0x7fce28003bd0 2024-09-16T01:42:40.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.877+0000 7fce3d363f40 1 -- 172.21.15.88:0/413148310 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5638c288df50 con 0x7fce28003bd0 2024-09-16T01:42:40.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.878+0000 7fce38e55640 1 -- 172.21.15.88:0/413148310 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3404db70 con 0x7fce28003bd0 2024-09-16T01:42:40.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.878+0000 7fce3d363f40 1 -- 172.21.15.88:0/413148310 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638c28921d0 con 0x7fce28003bd0 2024-09-16T01:42:40.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.879+0000 7fce38e55640 1 -- 172.21.15.88:0/413148310 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fce34061020 con 0x7fce28003bd0 2024-09-16T01:42:40.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.879+0000 7fce38e55640 1 --2- 172.21.15.88:0/413148310 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fce1c06ac90 0x7fce1c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:40.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.879+0000 7fce38e55640 1 -- 172.21.15.88:0/413148310 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 7802+0+0 (secure 0 0 0) 0x7fce340cd4a0 con 0x7fce28003bd0 2024-09-16T01:42:40.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.879+0000 7fce3ae59640 1 --2- 172.21.15.88:0/413148310 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fce1c06ac90 0x7fce1c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:40.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.880+0000 7fce3d363f40 1 --2- 172.21.15.88:0/413148310 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5638c2890d40 0x5638c2891120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:40.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.880+0000 7fce3d363f40 1 -- 172.21.15.88:0/413148310 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:1 12.14 12:292cf221:::200.00000000:head [truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) -- 0x5638c2891660 con 0x5638c2890d40 2024-09-16T01:42:40.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.880+0000 7fce3ae59640 1 --2- 172.21.15.88:0/413148310 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fce1c06ac90 0x7fce1c06d150 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x5638c29336d0 tx=0x7fce2c00c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:40.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.880+0000 7fce32658640 1 --2- 172.21.15.88:0/413148310 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5638c2890d40 0x5638c2891120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:40.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.880+0000 7fce32658640 1 --2- 172.21.15.88:0/413148310 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5638c2890d40 0x5638c2891120 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:40.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.882+0000 7fce32658640 1 -- 172.21.15.88:0/413148310 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(1 200.00000000 [truncate 0] v111'549 uv549 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fce24002c70 con 0x5638c2890d40 2024-09-16T01:42:40.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.882+0000 7fce3d363f40 1 -- 172.21.15.88:0/413148310 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5638c2890d40 msgr2=0x5638c2891120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:40.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.882+0000 7fce3d363f40 1 --2- 172.21.15.88:0/413148310 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5638c2890d40 0x5638c2891120 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.882+0000 7fce3d363f40 1 -- 172.21.15.88:0/413148310 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fce1c06ac90 msgr2=0x7fce1c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:40.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.882+0000 7fce3d363f40 1 --2- 172.21.15.88:0/413148310 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fce1c06ac90 0x7fce1c06d150 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x5638c29336d0 tx=0x7fce2c00c040 comp rx=0 tx=0).stop 2024-09-16T01:42:40.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.882+0000 7fce3d363f40 1 -- 172.21.15.88:0/413148310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce28003bd0 msgr2=0x5638c288d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:40.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.882+0000 7fce3d363f40 1 --2- 172.21.15.88:0/413148310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce28003bd0 0x5638c288d140 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7fce34006d50 tx=0x5638c27c51f0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.882+0000 7fce3b65a640 1 -- 172.21.15.88:0/413148310 reap_dead start 2024-09-16T01:42:40.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.882+0000 7fce3d363f40 1 -- 172.21.15.88:0/413148310 shutdown_connections 2024-09-16T01:42:40.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.882+0000 7fce3d363f40 1 -- 172.21.15.88:0/413148310 >> 172.21.15.88:0/413148310 conn(0x5638c27ec960 msgr2=0x5638c27c0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:40.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.882+0000 7fce3d363f40 1 -- 172.21.15.88:0/413148310 shutdown_connections 2024-09-16T01:42:40.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.882+0000 7fce3d363f40 1 -- 172.21.15.88:0/413148310 wait complete. 2024-09-16T01:42:40.891 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:42:40.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.990+0000 7f0727bbd640 1 Processor -- start 2024-09-16T01:42:40.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.991+0000 7f0727bbd640 1 -- start start 2024-09-16T01:42:40.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.991+0000 7f0727bbd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f07200cf1f0 0x7f07200b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:40.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.991+0000 7f0727bbd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07200b6720 0x7f07200d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:40.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.991+0000 7f0727bbd640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07200cf860 con 0x7f072015e7a0 2024-09-16T01:42:40.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.991+0000 7f0727bbd640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f07200d2ca0 con 0x7f07200cf1f0 2024-09-16T01:42:40.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.991+0000 7f0727bbd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f07200cfb40 con 0x7f07200b6720 2024-09-16T01:42:40.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.992+0000 7f0725158640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f07200cf1f0 0x7f07200b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:40.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.992+0000 7f0725158640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f07200cf1f0 0x7f07200b61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36832/0 (socket says 172.21.15.88:36832) 2024-09-16T01:42:40.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.992+0000 7f0725158640 1 -- 172.21.15.88:0/2866747429 learned_addr learned my addr 172.21.15.88:0/2866747429 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:40.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.992+0000 7f072615a640 1 --2- 172.21.15.88:0/2866747429 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07200b6720 0x7f07200d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:40.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.992+0000 7f0725158640 1 -- 172.21.15.88:0/2866747429 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07200b6720 msgr2=0x7f07200d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:40.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.992+0000 7f0725158640 1 --2- 172.21.15.88:0/2866747429 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07200b6720 0x7f07200d3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.992+0000 7f0725158640 1 -- 172.21.15.88:0/2866747429 >> v1:172.21.15.154:6789/0 conn(0x7f072015e7a0 legacy=0x7f072006caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:40.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.992+0000 7f0725158640 1 -- 172.21.15.88:0/2866747429 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07200cfe20 con 0x7f07200cf1f0 2024-09-16T01:42:40.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.993+0000 7f0725158640 1 --2- 172.21.15.88:0/2866747429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07200cf1f0 0x7f07200b61e0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f0710009870 tx=0x7f071002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d02dcd679d75191 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:40.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.993+0000 7f0724957640 1 -- 172.21.15.88:0/2866747429 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0710037070 con 0x7f07200cf1f0 2024-09-16T01:42:40.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.993+0000 7f0724957640 1 -- 172.21.15.88:0/2866747429 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0710032440 con 0x7f07200cf1f0 2024-09-16T01:42:40.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.993+0000 7f0724957640 1 -- 172.21.15.88:0/2866747429 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0710032810 con 0x7f07200cf1f0 2024-09-16T01:42:40.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.993+0000 7f0727bbd640 1 -- 172.21.15.88:0/2866747429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07200cf1f0 msgr2=0x7f07200b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:40.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.993+0000 7f0727bbd640 1 --2- 172.21.15.88:0/2866747429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07200cf1f0 0x7f07200b61e0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f0710009870 tx=0x7f071002a1c0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.993+0000 7f0727bbd640 1 -- 172.21.15.88:0/2866747429 shutdown_connections 2024-09-16T01:42:40.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.993+0000 7f0727bbd640 1 --2- 172.21.15.88:0/2866747429 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07200b6720 0x7f07200d3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.993+0000 7f0727bbd640 1 --2- 172.21.15.88:0/2866747429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07200cf1f0 0x7f07200b61e0 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:40.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.993+0000 7f0727bbd640 1 -- 172.21.15.88:0/2866747429 >> 172.21.15.88:0/2866747429 conn(0x7f07200d11e0 msgr2=0x7f07200d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:40.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.993+0000 7f0727bbd640 1 -- 172.21.15.88:0/2866747429 shutdown_connections 2024-09-16T01:42:40.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.993+0000 7f0727bbd640 1 -- 172.21.15.88:0/2866747429 wait complete. 2024-09-16T01:42:41.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.995+0000 7f0727bbd640 1 Processor -- start 2024-09-16T01:42:41.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.995+0000 7f0727bbd640 1 -- start start 2024-09-16T01:42:41.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.995+0000 7f0727bbd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07200b6720 0x7f07200d5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:41.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.995+0000 7f0727bbd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07200cf1f0 0x7f072014d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:41.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.995+0000 7f0727bbd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f072015e7a0 0x7f072015f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:41.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.995+0000 7f0727bbd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f072015eb80 con 0x7f07200cf1f0 2024-09-16T01:42:41.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.995+0000 7f0727bbd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f07200b7a00 con 0x7f07200b6720 2024-09-16T01:42:41.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.995+0000 7f0727bbd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f072015ed00 con 0x7f072015e7a0 2024-09-16T01:42:41.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.995+0000 7f0725158640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07200cf1f0 0x7f072014d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:41.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.995+0000 7f0725158640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07200cf1f0 0x7f072014d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36836/0 (socket says 172.21.15.88:36836) 2024-09-16T01:42:41.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.995+0000 7f0725158640 1 -- 172.21.15.88:0/3193481246 learned_addr learned my addr 172.21.15.88:0/3193481246 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:41.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.995+0000 7f0725959640 1 --2- 172.21.15.88:0/3193481246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07200b6720 0x7f07200d5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:41.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.995+0000 7f072615a640 1 --2- 172.21.15.88:0/3193481246 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f072015e7a0 0x7f072015f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:41.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.996+0000 7f0725158640 1 -- 172.21.15.88:0/3193481246 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f072015e7a0 msgr2=0x7f072015f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:41.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.996+0000 7f0725158640 1 --2- 172.21.15.88:0/3193481246 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f072015e7a0 0x7f072015f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:41.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.996+0000 7f0725158640 1 -- 172.21.15.88:0/3193481246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07200b6720 msgr2=0x7f07200d5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:41.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.996+0000 7f0725158640 1 --2- 172.21.15.88:0/3193481246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07200b6720 0x7f07200d5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:41.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.996+0000 7f0725158640 1 -- 172.21.15.88:0/3193481246 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0720160640 con 0x7f07200cf1f0 2024-09-16T01:42:41.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.996+0000 7f0725158640 1 --2- 172.21.15.88:0/3193481246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07200cf1f0 0x7f072014d8d0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f0710000c00 tx=0x7f07100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:41.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.996+0000 7f07167fc640 1 -- 172.21.15.88:0/3193481246 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0710037040 con 0x7f07200cf1f0 2024-09-16T01:42:41.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.996+0000 7f07167fc640 1 -- 172.21.15.88:0/3193481246 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0710042070 con 0x7f07200cf1f0 2024-09-16T01:42:41.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.996+0000 7f0727bbd640 1 -- 172.21.15.88:0/3193481246 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0720160920 con 0x7f07200cf1f0 2024-09-16T01:42:41.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.996+0000 7f07167fc640 1 -- 172.21.15.88:0/3193481246 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0710030880 con 0x7f07200cf1f0 2024-09-16T01:42:41.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.996+0000 7f0727bbd640 1 -- 172.21.15.88:0/3193481246 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f072015f290 con 0x7f07200cf1f0 2024-09-16T01:42:41.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.997+0000 7f0727bbd640 1 -- 172.21.15.88:0/3193481246 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07200b61e0 con 0x7f07200cf1f0 2024-09-16T01:42:41.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.998+0000 7f07167fc640 1 -- 172.21.15.88:0/3193481246 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f071002ad10 con 0x7f07200cf1f0 2024-09-16T01:42:41.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.998+0000 7f07167fc640 1 --2- 172.21.15.88:0/3193481246 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f070406c2e0 0x7f070406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:41.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.998+0000 7f0725959640 1 --2- 172.21.15.88:0/3193481246 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f070406c2e0 0x7f070406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:41.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.998+0000 7f07167fc640 1 -- 172.21.15.88:0/3193481246 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 7802+0+0 (secure 0 0 0) 0x7f07100ac0c0 con 0x7f07200cf1f0 2024-09-16T01:42:41.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:40.999+0000 7f0725959640 1 --2- 172.21.15.88:0/3193481246 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f070406c2e0 0x7f070406e7a0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f071c0016e0 tx=0x7f071c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:41.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.003+0000 7f07167fc640 1 -- 172.21.15.88:0/3193481246 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0710079910 con 0x7f07200cf1f0 2024-09-16T01:42:41.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.201+0000 7f0727bbd640 1 -- 172.21.15.88:0/3193481246 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f07200d2a60 con 0x7f07200cf1f0 2024-09-16T01:42:41.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.618+0000 7f07167fc640 1 -- 172.21.15.88:0/3193481246 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v214) ==== 162+0+0 (secure 0 0 0) 0x7f071007dc10 con 0x7f07200cf1f0 2024-09-16T01:42:41.619 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:42:41.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.620+0000 7f0727bbd640 1 -- 172.21.15.88:0/3193481246 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f070406c2e0 msgr2=0x7f070406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:41.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.620+0000 7f0727bbd640 1 --2- 172.21.15.88:0/3193481246 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f070406c2e0 0x7f070406e7a0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f071c0016e0 tx=0x7f071c009040 comp rx=0 tx=0).stop 2024-09-16T01:42:41.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.620+0000 7f0727bbd640 1 -- 172.21.15.88:0/3193481246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07200cf1f0 msgr2=0x7f072014d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:41.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.620+0000 7f0727bbd640 1 --2- 172.21.15.88:0/3193481246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07200cf1f0 0x7f072014d8d0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f0710000c00 tx=0x7f07100046c0 comp rx=0 tx=0).stop 2024-09-16T01:42:41.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.620+0000 7f0727bbd640 1 -- 172.21.15.88:0/3193481246 shutdown_connections 2024-09-16T01:42:41.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.620+0000 7f0727bbd640 1 --2- 172.21.15.88:0/3193481246 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f070406c2e0 0x7f070406e7a0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:41.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.620+0000 7f0727bbd640 1 --2- 172.21.15.88:0/3193481246 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f072015e7a0 0x7f072015f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:41.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.620+0000 7f0727bbd640 1 --2- 172.21.15.88:0/3193481246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f07200cf1f0 0x7f072014d8d0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:41.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.620+0000 7f0727bbd640 1 --2- 172.21.15.88:0/3193481246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07200b6720 0x7f07200d5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:41.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.620+0000 7f0727bbd640 1 -- 172.21.15.88:0/3193481246 >> 172.21.15.88:0/3193481246 conn(0x7f07200d11e0 msgr2=0x7f072015d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:41.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.620+0000 7f0727bbd640 1 -- 172.21.15.88:0/3193481246 shutdown_connections 2024-09-16T01:42:41.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.620+0000 7f0727bbd640 1 -- 172.21.15.88:0/3193481246 wait complete. 2024-09-16T01:42:41.629 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:42:41.628+0000 7f7118bc5640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (22) Invalid argument 2024-09-16T01:42:41.630 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:42:41.629+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:42:41.630 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:42:41.629+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:42:41.634 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:41.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.731+0000 7fed284d9640 1 Processor -- start 2024-09-16T01:42:41.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.731+0000 7fed284d9640 1 -- start start 2024-09-16T01:42:41.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.732+0000 7fed284d9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fed2015c550 0x7fed200b5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:41.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.732+0000 7fed284d9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fed200b6310 0x7fed200b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:41.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.732+0000 7fed284d9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed200b4cf0 con 0x7fed2015c170 2024-09-16T01:42:41.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.732+0000 7fed284d9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fed200b4fd0 con 0x7fed200b6310 2024-09-16T01:42:41.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.732+0000 7fed284d9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fed200b52b0 con 0x7fed2015c550 2024-09-16T01:42:41.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.733+0000 7fed26a76640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fed200b6310 0x7fed200b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:41.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.733+0000 7fed26a76640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fed200b6310 0x7fed200b39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36840/0 (socket says 172.21.15.88:36840) 2024-09-16T01:42:41.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.733+0000 7fed26a76640 1 -- 172.21.15.88:0/2894041237 learned_addr learned my addr 172.21.15.88:0/2894041237 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:41.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.733+0000 7fed25a74640 1 --2- 172.21.15.88:0/2894041237 >> v2:172.21.15.154:3300/0 conn(0x7fed2015c550 0x7fed200b5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:41.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.733+0000 7fed26a76640 1 -- 172.21.15.88:0/2894041237 >> v2:172.21.15.154:3300/0 conn(0x7fed2015c550 msgr2=0x7fed200b5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:41.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.733+0000 7fed26a76640 1 --2- 172.21.15.88:0/2894041237 >> v2:172.21.15.154:3300/0 conn(0x7fed2015c550 0x7fed200b5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:41.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.733+0000 7fed26a76640 1 -- 172.21.15.88:0/2894041237 >> v1:172.21.15.88:6789/0 conn(0x7fed2015c170 legacy=0x7fed200cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:41.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.733+0000 7fed26a76640 1 -- 172.21.15.88:0/2894041237 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed200b5590 con 0x7fed200b6310 2024-09-16T01:42:41.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.733+0000 7fed25a74640 1 --2- 172.21.15.88:0/2894041237 >> v2:172.21.15.154:3300/0 conn(0x7fed2015c550 0x7fed200b5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:41.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.734+0000 7fed26a76640 1 --2- 172.21.15.88:0/2894041237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed200b6310 0x7fed200b39b0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fed1c009e50 tx=0x7fed1c02a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e9ccb4415593d8b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:41.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.734+0000 7fed25273640 1 -- 172.21.15.88:0/2894041237 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1c036070 con 0x7fed200b6310 2024-09-16T01:42:41.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.734+0000 7fed25273640 1 -- 172.21.15.88:0/2894041237 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed1c031600 con 0x7fed200b6310 2024-09-16T01:42:41.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.734+0000 7fed25273640 1 -- 172.21.15.88:0/2894041237 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1c0319d0 con 0x7fed200b6310 2024-09-16T01:42:41.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.734+0000 7fed284d9640 1 -- 172.21.15.88:0/2894041237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed200b6310 msgr2=0x7fed200b39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:41.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.734+0000 7fed284d9640 1 --2- 172.21.15.88:0/2894041237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed200b6310 0x7fed200b39b0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fed1c009e50 tx=0x7fed1c02a760 comp rx=0 tx=0).stop 2024-09-16T01:42:41.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.734+0000 7fed284d9640 1 -- 172.21.15.88:0/2894041237 shutdown_connections 2024-09-16T01:42:41.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.734+0000 7fed284d9640 1 --2- 172.21.15.88:0/2894041237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed200b6310 0x7fed200b39b0 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:41.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.734+0000 7fed284d9640 1 --2- 172.21.15.88:0/2894041237 >> v2:172.21.15.154:3300/0 conn(0x7fed2015c550 0x7fed200b5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:41.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.734+0000 7fed284d9640 1 -- 172.21.15.88:0/2894041237 >> 172.21.15.88:0/2894041237 conn(0x7fed2006bc10 msgr2=0x7fed2006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:41.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.734+0000 7fed284d9640 1 -- 172.21.15.88:0/2894041237 shutdown_connections 2024-09-16T01:42:41.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.734+0000 7fed284d9640 1 -- 172.21.15.88:0/2894041237 wait complete. 2024-09-16T01:42:41.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.735+0000 7fed284d9640 1 Processor -- start 2024-09-16T01:42:41.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.736+0000 7fed284d9640 1 -- start start 2024-09-16T01:42:41.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.736+0000 7fed284d9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed200b6310 0x7fed20148e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.736+0000 7fed284d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2015c170 0x7fed20155770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.736+0000 7fed284d9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed2015c550 0x7fed200cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.736+0000 7fed284d9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fed2015c9c0 con 0x7fed2015c170 2024-09-16T01:42:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.736+0000 7fed284d9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fed200b5b50 con 0x7fed2015c550 2024-09-16T01:42:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.736+0000 7fed284d9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fed200cfdd0 con 0x7fed200b6310 2024-09-16T01:42:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.736+0000 7fed25a74640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2015c170 0x7fed20155770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.736+0000 7fed25a74640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2015c170 0x7fed20155770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36848/0 (socket says 172.21.15.88:36848) 2024-09-16T01:42:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.736+0000 7fed25a74640 1 -- 172.21.15.88:0/2566425878 learned_addr learned my addr 172.21.15.88:0/2566425878 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.737+0000 7fed25a74640 1 -- 172.21.15.88:0/2566425878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed200b6310 msgr2=0x7fed20148e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.737+0000 7fed25a74640 1 --2- 172.21.15.88:0/2566425878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed200b6310 0x7fed20148e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:41.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.737+0000 7fed25a74640 1 -- 172.21.15.88:0/2566425878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed2015c550 msgr2=0x7fed200cdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.737+0000 7fed25a74640 1 --2- 172.21.15.88:0/2566425878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed2015c550 0x7fed200cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.737+0000 7fed25a74640 1 -- 172.21.15.88:0/2566425878 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed201545d0 con 0x7fed2015c170 2024-09-16T01:42:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.737+0000 7fed25a74640 1 --2- 172.21.15.88:0/2566425878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2015c170 0x7fed20155770 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fed1000d7d0 tx=0x7fed1000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.737+0000 7fed16ffd640 1 -- 172.21.15.88:0/2566425878 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed10004030 con 0x7fed2015c170 2024-09-16T01:42:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.737+0000 7fed16ffd640 1 -- 172.21.15.88:0/2566425878 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed100041d0 con 0x7fed2015c170 2024-09-16T01:42:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.737+0000 7fed284d9640 1 -- 172.21.15.88:0/2566425878 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed20153a50 con 0x7fed2015c170 2024-09-16T01:42:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.737+0000 7fed16ffd640 1 -- 172.21.15.88:0/2566425878 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed100096a0 con 0x7fed2015c170 2024-09-16T01:42:41.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.737+0000 7fed284d9640 1 -- 172.21.15.88:0/2566425878 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed201542f0 con 0x7fed2015c170 2024-09-16T01:42:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.738+0000 7fed284d9640 1 -- 172.21.15.88:0/2566425878 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed200b5dd0 con 0x7fed2015c170 2024-09-16T01:42:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.739+0000 7fed16ffd640 1 -- 172.21.15.88:0/2566425878 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fed10009840 con 0x7fed2015c170 2024-09-16T01:42:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.739+0000 7fed16ffd640 1 --2- 172.21.15.88:0/2566425878 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fed0406c2e0 0x7fed0406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:41.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.739+0000 7fed16ffd640 1 -- 172.21.15.88:0/2566425878 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 7888+0+0 (secure 0 0 0) 0x7fed1008bfa0 con 0x7fed2015c170 2024-09-16T01:42:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.739+0000 7fed26275640 1 --2- 172.21.15.88:0/2566425878 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fed0406c2e0 0x7fed0406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:41.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.740+0000 7fed26275640 1 --2- 172.21.15.88:0/2566425878 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fed0406c2e0 0x7fed0406e7a0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fed18000f60 tx=0x7fed180079d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:41.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.743+0000 7fed16ffd640 1 -- 172.21.15.88:0/2566425878 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fed10059e90 con 0x7fed2015c170 2024-09-16T01:42:41.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.943+0000 7fed284d9640 1 -- 172.21.15.88:0/2566425878 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed200b3e10 con 0x7fed2015c170 2024-09-16T01:42:41.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.944+0000 7fed16ffd640 1 -- 172.21.15.88:0/2566425878 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 216 v216) ==== 95+0+3566 (secure 0 0 0) 0x7fed10059790 con 0x7fed2015c170 2024-09-16T01:42:41.945 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:41.945 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":216,"btime":"2024-09-16T01:42:41:630129+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":216,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:41.630075+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:41.946 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 216 2024-09-16T01:42:41.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.946+0000 7fed284d9640 1 -- 172.21.15.88:0/2566425878 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fed0406c2e0 msgr2=0x7fed0406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:41.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.946+0000 7fed284d9640 1 --2- 172.21.15.88:0/2566425878 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fed0406c2e0 0x7fed0406e7a0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fed18000f60 tx=0x7fed180079d0 comp rx=0 tx=0).stop 2024-09-16T01:42:41.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.946+0000 7fed284d9640 1 -- 172.21.15.88:0/2566425878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2015c170 msgr2=0x7fed20155770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:41.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.946+0000 7fed284d9640 1 --2- 172.21.15.88:0/2566425878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2015c170 0x7fed20155770 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fed1000d7d0 tx=0x7fed1000dc90 comp rx=0 tx=0).stop 2024-09-16T01:42:41.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.946+0000 7fed284d9640 1 -- 172.21.15.88:0/2566425878 shutdown_connections 2024-09-16T01:42:41.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.946+0000 7fed284d9640 1 --2- 172.21.15.88:0/2566425878 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fed0406c2e0 0x7fed0406e7a0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:41.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.946+0000 7fed284d9640 1 --2- 172.21.15.88:0/2566425878 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed2015c550 0x7fed200cdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:41.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.946+0000 7fed284d9640 1 --2- 172.21.15.88:0/2566425878 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2015c170 0x7fed20155770 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:41.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.946+0000 7fed284d9640 1 --2- 172.21.15.88:0/2566425878 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed200b6310 0x7fed20148e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:41.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.946+0000 7fed284d9640 1 -- 172.21.15.88:0/2566425878 >> 172.21.15.88:0/2566425878 conn(0x7fed2006bc10 msgr2=0x7fed200b4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:41.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.946+0000 7fed284d9640 1 -- 172.21.15.88:0/2566425878 shutdown_connections 2024-09-16T01:42:41.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:41.946+0000 7fed284d9640 1 -- 172.21.15.88:0/2566425878 wait complete. 2024-09-16T01:42:41.958 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:42:41.958 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:42.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.055+0000 7ff350a0e640 1 Processor -- start 2024-09-16T01:42:42.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.055+0000 7ff350a0e640 1 -- start start 2024-09-16T01:42:42.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.056+0000 7ff350a0e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff34c06ef30 0x7ff34c06bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:42.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.056+0000 7ff350a0e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff34c0ceef0 con 0x7ff34c06c150 2024-09-16T01:42:42.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.056+0000 7ff350a0e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff34c0cf1d0 con 0x7ff34c15c610 2024-09-16T01:42:42.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.056+0000 7ff350a0e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff34c06e7b0 con 0x7ff34c06ef30 2024-09-16T01:42:42.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.056+0000 7ff341d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff34c06ef30 0x7ff34c06bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:42.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.056+0000 7ff341d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff34c06ef30 0x7ff34c06bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36850/0 (socket says 172.21.15.88:36850) 2024-09-16T01:42:42.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.056+0000 7ff341d9b640 1 -- 172.21.15.88:0/4093933822 learned_addr learned my addr 172.21.15.88:0/4093933822 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:42.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.056+0000 7ff341d9b640 1 -- 172.21.15.88:0/4093933822 >> v1:172.21.15.154:6789/0 conn(0x7ff34c15c610 legacy=0x7ff34c0ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:42.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.056+0000 7ff341d9b640 1 -- 172.21.15.88:0/4093933822 >> v1:172.21.15.88:6789/0 conn(0x7ff34c06c150 legacy=0x7ff34c1660c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:42.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.056+0000 7ff341d9b640 1 -- 172.21.15.88:0/4093933822 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff34c0cf4b0 con 0x7ff34c06ef30 2024-09-16T01:42:42.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.057+0000 7ff341d9b640 1 --2- 172.21.15.88:0/4093933822 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff34c06ef30 0x7ff34c06bc10 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7ff334009870 tx=0x7ff33402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3399a8db0bd39888 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:42.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.057+0000 7ff349d9b640 1 -- 172.21.15.88:0/4093933822 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff334037070 con 0x7ff34c06ef30 2024-09-16T01:42:42.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.057+0000 7ff349d9b640 1 -- 172.21.15.88:0/4093933822 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff33402ae50 con 0x7ff34c06ef30 2024-09-16T01:42:42.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.057+0000 7ff349d9b640 1 -- 172.21.15.88:0/4093933822 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff334032860 con 0x7ff34c06ef30 2024-09-16T01:42:42.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.057+0000 7ff350a0e640 1 -- 172.21.15.88:0/4093933822 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff34c06ef30 msgr2=0x7ff34c06bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:42.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.057+0000 7ff350a0e640 1 --2- 172.21.15.88:0/4093933822 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff34c06ef30 0x7ff34c06bc10 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7ff334009870 tx=0x7ff33402a300 comp rx=0 tx=0).stop 2024-09-16T01:42:42.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.057+0000 7ff350a0e640 1 -- 172.21.15.88:0/4093933822 shutdown_connections 2024-09-16T01:42:42.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.058+0000 7ff350a0e640 1 --2- 172.21.15.88:0/4093933822 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff34c06ef30 0x7ff34c06bc10 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:42.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.058+0000 7ff350a0e640 1 -- 172.21.15.88:0/4093933822 >> 172.21.15.88:0/4093933822 conn(0x7ff34c0b2430 msgr2=0x7ff34c0b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:42.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.058+0000 7ff350a0e640 1 -- 172.21.15.88:0/4093933822 shutdown_connections 2024-09-16T01:42:42.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.058+0000 7ff350a0e640 1 -- 172.21.15.88:0/4093933822 wait complete. 2024-09-16T01:42:42.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.059+0000 7ff350a0e640 1 Processor -- start 2024-09-16T01:42:42.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.059+0000 7ff350a0e640 1 -- start start 2024-09-16T01:42:42.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.060+0000 7ff350a0e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff34c06c150 0x7ff34c14f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:42.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.060+0000 7ff350a0e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff34c06ef30 0x7ff34c154150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:42.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.060+0000 7ff350a0e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff34c15c610 0x7ff34c154690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:42.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.060+0000 7ff350a0e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff34c06c580 con 0x7ff34c15c610 2024-09-16T01:42:42.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.060+0000 7ff350a0e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff34c0cecb0 con 0x7ff34c06c150 2024-09-16T01:42:42.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.060+0000 7ff350a0e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff34c0713c0 con 0x7ff34c06ef30 2024-09-16T01:42:42.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.060+0000 7ff34ad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff34c15c610 0x7ff34c154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:42.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.060+0000 7ff34ad9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff34c15c610 0x7ff34c154690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36854/0 (socket says 172.21.15.88:36854) 2024-09-16T01:42:42.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.060+0000 7ff34ad9d640 1 -- 172.21.15.88:0/2804173984 learned_addr learned my addr 172.21.15.88:0/2804173984 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:42.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.060+0000 7ff34a59c640 1 --2- 172.21.15.88:0/2804173984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff34c06c150 0x7ff34c14f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:42.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.060+0000 7ff341d9b640 1 --2- 172.21.15.88:0/2804173984 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff34c06ef30 0x7ff34c154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:42.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.060+0000 7ff34ad9d640 1 -- 172.21.15.88:0/2804173984 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff34c06ef30 msgr2=0x7ff34c154150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:42.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.060+0000 7ff34ad9d640 1 --2- 172.21.15.88:0/2804173984 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff34c06ef30 0x7ff34c154150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:42.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.060+0000 7ff34ad9d640 1 -- 172.21.15.88:0/2804173984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff34c06c150 msgr2=0x7ff34c14f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:42.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.060+0000 7ff34ad9d640 1 --2- 172.21.15.88:0/2804173984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff34c06c150 0x7ff34c14f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:42.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.060+0000 7ff34ad9d640 1 -- 172.21.15.88:0/2804173984 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff34c1599f0 con 0x7ff34c15c610 2024-09-16T01:42:42.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.060+0000 7ff34ad9d640 1 --2- 172.21.15.88:0/2804173984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff34c15c610 0x7ff34c154690 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7ff34400c760 tx=0x7ff34400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:42.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.061+0000 7ff3437fe640 1 -- 172.21.15.88:0/2804173984 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff344003b70 con 0x7ff34c15c610 2024-09-16T01:42:42.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.061+0000 7ff3437fe640 1 -- 172.21.15.88:0/2804173984 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff344003d30 con 0x7ff34c15c610 2024-09-16T01:42:42.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.061+0000 7ff350a0e640 1 -- 172.21.15.88:0/2804173984 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff34c1e4fa0 con 0x7ff34c15c610 2024-09-16T01:42:42.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.061+0000 7ff3437fe640 1 -- 172.21.15.88:0/2804173984 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff34400aa90 con 0x7ff34c15c610 2024-09-16T01:42:42.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.061+0000 7ff350a0e640 1 -- 172.21.15.88:0/2804173984 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff34c1e5520 con 0x7ff34c15c610 2024-09-16T01:42:42.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.062+0000 7ff350a0e640 1 -- 172.21.15.88:0/2804173984 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff34c06bc10 con 0x7ff34c15c610 2024-09-16T01:42:42.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.062+0000 7ff3437fe640 1 -- 172.21.15.88:0/2804173984 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff34400ac30 con 0x7ff34c15c610 2024-09-16T01:42:42.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.063+0000 7ff3437fe640 1 --2- 172.21.15.88:0/2804173984 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff32c06c2e0 0x7ff32c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:42.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.063+0000 7ff34a59c640 1 --2- 172.21.15.88:0/2804173984 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff32c06c2e0 0x7ff32c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:42.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.063+0000 7ff3437fe640 1 -- 172.21.15.88:0/2804173984 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 7888+0+0 (secure 0 0 0) 0x7ff34408b940 con 0x7ff34c15c610 2024-09-16T01:42:42.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.063+0000 7ff34a59c640 1 --2- 172.21.15.88:0/2804173984 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff32c06c2e0 0x7ff32c06e7a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0b0e00 tx=0x7ff33c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:42.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.067+0000 7ff3437fe640 1 -- 172.21.15.88:0/2804173984 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff344059270 con 0x7ff34c15c610 2024-09-16T01:42:42.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.270+0000 7ff350a0e640 1 -- 172.21.15.88:0/2804173984 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff34c0b1230 con 0x7ff34c15c610 2024-09-16T01:42:42.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.270+0000 7ff3437fe640 1 -- 172.21.15.88:0/2804173984 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 216 v216) ==== 95+0+3566 (secure 0 0 0) 0x7ff34405d570 con 0x7ff34c15c610 2024-09-16T01:42:42.272 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:42.272 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":216,"btime":"2024-09-16T01:42:41:630129+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":216,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:41.630075+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:42.272 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 216 2024-09-16T01:42:42.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.272+0000 7ff350a0e640 1 -- 172.21.15.88:0/2804173984 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff32c06c2e0 msgr2=0x7ff32c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:42.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.272+0000 7ff350a0e640 1 --2- 172.21.15.88:0/2804173984 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff32c06c2e0 0x7ff32c06e7a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0b0e00 tx=0x7ff33c009040 comp rx=0 tx=0).stop 2024-09-16T01:42:42.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.272+0000 7ff350a0e640 1 -- 172.21.15.88:0/2804173984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff34c15c610 msgr2=0x7ff34c154690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:42.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.272+0000 7ff350a0e640 1 --2- 172.21.15.88:0/2804173984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff34c15c610 0x7ff34c154690 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7ff34400c760 tx=0x7ff34400cc20 comp rx=0 tx=0).stop 2024-09-16T01:42:42.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.272+0000 7ff350a0e640 1 -- 172.21.15.88:0/2804173984 shutdown_connections 2024-09-16T01:42:42.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.272+0000 7ff350a0e640 1 --2- 172.21.15.88:0/2804173984 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff32c06c2e0 0x7ff32c06e7a0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:42.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.272+0000 7ff350a0e640 1 --2- 172.21.15.88:0/2804173984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff34c15c610 0x7ff34c154690 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:42.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.272+0000 7ff350a0e640 1 --2- 172.21.15.88:0/2804173984 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff34c06ef30 0x7ff34c154150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:42.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.272+0000 7ff350a0e640 1 --2- 172.21.15.88:0/2804173984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff34c06c150 0x7ff34c14f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:42.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.272+0000 7ff350a0e640 1 -- 172.21.15.88:0/2804173984 >> 172.21.15.88:0/2804173984 conn(0x7ff34c0b2430 msgr2=0x7ff34c071ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:42.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.273+0000 7ff350a0e640 1 -- 172.21.15.88:0/2804173984 shutdown_connections 2024-09-16T01:42:42.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.273+0000 7ff350a0e640 1 -- 172.21.15.88:0/2804173984 wait complete. 2024-09-16T01:42:42.285 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:42:42.285 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:42.351 INFO:tasks.ceph.mds.c.smithi088.stderr: -38> 2024-09-16T01:42:37.598+0000 7fa987834640 -1 mds.0.cache.ino(0x1) No data while reading inode 0x1 2024-09-16T01:42:42.351 INFO:tasks.ceph.mds.c.smithi088.stderr: -36> 2024-09-16T01:42:37.598+0000 7fa987834640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-09-16T01:42:42.374 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:42:42.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.384+0000 7f1dd4f59640 1 Processor -- start 2024-09-16T01:42:42.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.384+0000 7f1dd4f59640 1 -- start start 2024-09-16T01:42:42.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.385+0000 7f1dd4f59640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1dd01450d0 0x7f1dd00cd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:42.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.385+0000 7f1dd4f59640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1dd006ecd0 0x7f1dd00ce970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:42.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.385+0000 7f1dd4f59640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1dd015f1d0 con 0x7f1dd00cdcb0 2024-09-16T01:42:42.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.385+0000 7f1dd4f59640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f1dd015f4b0 con 0x7f1dd006ecd0 2024-09-16T01:42:42.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.385+0000 7f1dd4f59640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f1dd015f790 con 0x7f1dd01450d0 2024-09-16T01:42:42.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.385+0000 7f1dcdd9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1dd00cdcb0 0x7f1dd006e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49244/0 (socket says 172.21.15.88:49244) 2024-09-16T01:42:42.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.385+0000 7f1dcdd9b640 1 -- 172.21.15.88:0/2852723436 learned_addr learned my addr 172.21.15.88:0/2852723436 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:42.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.385+0000 7f1dced9d640 1 --2- 172.21.15.88:0/2852723436 >> v2:172.21.15.88:3300/0 conn(0x7f1dd006ecd0 0x7f1dd00ce970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:42.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.385+0000 7f1dcd59a640 1 -- 172.21.15.88:0/2852723436 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1481521452 0 0) 0x7f1dd015f1d0 con 0x7f1dd00cdcb0 2024-09-16T01:42:42.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.385+0000 7f1dcd59a640 1 -- 172.21.15.88:0/2852723436 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1db8003520 con 0x7f1dd00cdcb0 2024-09-16T01:42:42.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.385+0000 7f1dced9d640 1 -- 172.21.15.88:0/2852723436 >> v2:172.21.15.154:3300/0 conn(0x7f1dd01450d0 msgr2=0x7f1dd00cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:42.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.385+0000 7f1dced9d640 1 --2- 172.21.15.88:0/2852723436 >> v2:172.21.15.154:3300/0 conn(0x7f1dd01450d0 0x7f1dd00cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:42.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.385+0000 7f1dced9d640 1 -- 172.21.15.88:0/2852723436 >> v1:172.21.15.88:6789/0 conn(0x7f1dd00cdcb0 legacy=0x7f1dd006e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:42.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.385+0000 7f1dced9d640 1 -- 172.21.15.88:0/2852723436 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dd015fa70 con 0x7f1dd006ecd0 2024-09-16T01:42:42.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.386+0000 7f1dced9d640 1 --2- 172.21.15.88:0/2852723436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dd006ecd0 0x7f1dd00ce970 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8009d90 tx=0x7f1dc802a580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc6bdff64af064d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:42.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.386+0000 7f1dcd59a640 1 -- 172.21.15.88:0/2852723436 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc8007740 con 0x7f1dd006ecd0 2024-09-16T01:42:42.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.386+0000 7f1dcd59a640 1 -- 172.21.15.88:0/2852723436 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dc80078e0 con 0x7f1dd006ecd0 2024-09-16T01:42:42.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.386+0000 7f1dcd59a640 1 -- 172.21.15.88:0/2852723436 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc8007cb0 con 0x7f1dd006ecd0 2024-09-16T01:42:42.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.387+0000 7f1dd4f59640 1 -- 172.21.15.88:0/2852723436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dd006ecd0 msgr2=0x7f1dd00ce970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:42.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.387+0000 7f1dd4f59640 1 --2- 172.21.15.88:0/2852723436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dd006ecd0 0x7f1dd00ce970 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8009d90 tx=0x7f1dc802a580 comp rx=0 tx=0).stop 2024-09-16T01:42:42.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.387+0000 7f1dd4f59640 1 -- 172.21.15.88:0/2852723436 shutdown_connections 2024-09-16T01:42:42.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.387+0000 7f1dd4f59640 1 --2- 172.21.15.88:0/2852723436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dd006ecd0 0x7f1dd00ce970 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:42.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.387+0000 7f1dd4f59640 1 --2- 172.21.15.88:0/2852723436 >> v2:172.21.15.154:3300/0 conn(0x7f1dd01450d0 0x7f1dd00cd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:42.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.387+0000 7f1dd4f59640 1 -- 172.21.15.88:0/2852723436 >> 172.21.15.88:0/2852723436 conn(0x7f1dd006e1e0 msgr2=0x7f1dd00b67c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:42.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.387+0000 7f1dd4f59640 1 -- 172.21.15.88:0/2852723436 shutdown_connections 2024-09-16T01:42:42.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.387+0000 7f1dd4f59640 1 -- 172.21.15.88:0/2852723436 wait complete. 2024-09-16T01:42:42.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.388+0000 7f1dd4f59640 1 Processor -- start 2024-09-16T01:42:42.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.388+0000 7f1dd4f59640 1 -- start start 2024-09-16T01:42:42.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.389+0000 7f1dd4f59640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dd006ecd0 0x7f1dd01f2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:42.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.389+0000 7f1dd4f59640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dd00cdcb0 0x7f1dd01f3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:42.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.389+0000 7f1dd4f59640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1dd01450d0 0x7f1dd01f39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:42.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.389+0000 7f1dd4f59640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd00cd560 con 0x7f1dd00cdcb0 2024-09-16T01:42:42.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.389+0000 7f1dd4f59640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd00cd3e0 con 0x7f1dd006ecd0 2024-09-16T01:42:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.389+0000 7f1dd4f59640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1dd00ceeb0 con 0x7f1dd01450d0 2024-09-16T01:42:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.389+0000 7f1dcdd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dd00cdcb0 0x7f1dd01f3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.389+0000 7f1dcdd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dd00cdcb0 0x7f1dd01f3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36906/0 (socket says 172.21.15.88:36906) 2024-09-16T01:42:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.389+0000 7f1dcdd9b640 1 -- 172.21.15.88:0/3035742460 learned_addr learned my addr 172.21.15.88:0/3035742460 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.389+0000 7f1dced9d640 1 --2- 172.21.15.88:0/3035742460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1dd01450d0 0x7f1dd01f39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.389+0000 7f1dce59c640 1 --2- 172.21.15.88:0/3035742460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dd006ecd0 0x7f1dd01f2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.389+0000 7f1dcdd9b640 1 -- 172.21.15.88:0/3035742460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1dd01450d0 msgr2=0x7f1dd01f39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.389+0000 7f1dcdd9b640 1 --2- 172.21.15.88:0/3035742460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1dd01450d0 0x7f1dd01f39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.389+0000 7f1dcdd9b640 1 -- 172.21.15.88:0/3035742460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dd006ecd0 msgr2=0x7f1dd01f2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.389+0000 7f1dcdd9b640 1 --2- 172.21.15.88:0/3035742460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dd006ecd0 0x7f1dd01f2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.389+0000 7f1dcdd9b640 1 -- 172.21.15.88:0/3035742460 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dd015b400 con 0x7f1dd00cdcb0 2024-09-16T01:42:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.389+0000 7f1dced9d640 1 --2- 172.21.15.88:0/3035742460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1dd01450d0 0x7f1dd01f39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.389+0000 7f1dcdd9b640 1 --2- 172.21.15.88:0/3035742460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dd00cdcb0 0x7f1dd01f3490 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f1dc400c430 tx=0x7f1dc400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:42.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.390+0000 7f1dbeffd640 1 -- 172.21.15.88:0/3035742460 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc4017070 con 0x7f1dd00cdcb0 2024-09-16T01:42:42.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.390+0000 7f1dbeffd640 1 -- 172.21.15.88:0/3035742460 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dc4005ce0 con 0x7f1dd00cdcb0 2024-09-16T01:42:42.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.390+0000 7f1dbeffd640 1 -- 172.21.15.88:0/3035742460 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc4007310 con 0x7f1dd00cdcb0 2024-09-16T01:42:42.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.390+0000 7f1dd4f59640 1 -- 172.21.15.88:0/3035742460 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dd013cde0 con 0x7f1dd00cdcb0 2024-09-16T01:42:42.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.390+0000 7f1dd4f59640 1 -- 172.21.15.88:0/3035742460 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dd013d3a0 con 0x7f1dd00cdcb0 2024-09-16T01:42:42.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.391+0000 7f1dd4f59640 1 -- 172.21.15.88:0/3035742460 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dd00ce970 con 0x7f1dd00cdcb0 2024-09-16T01:42:42.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.391+0000 7f1dbeffd640 1 -- 172.21.15.88:0/3035742460 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1dc401a070 con 0x7f1dd00cdcb0 2024-09-16T01:42:42.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.392+0000 7f1dbeffd640 1 --2- 172.21.15.88:0/3035742460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1db006c2b0 0x7f1db006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:42.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.392+0000 7f1dbeffd640 1 -- 172.21.15.88:0/3035742460 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 7888+0+0 (secure 0 0 0) 0x7f1dc408af80 con 0x7f1dd00cdcb0 2024-09-16T01:42:42.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.392+0000 7f1dce59c640 1 --2- 172.21.15.88:0/3035742460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1db006c2b0 0x7f1db006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:42.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.392+0000 7f1dce59c640 1 --2- 172.21.15.88:0/3035742460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1db006c2b0 0x7f1db006e770 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f1dd00732e0 tx=0x7f1dc0009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:42.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.396+0000 7f1dbeffd640 1 -- 172.21.15.88:0/3035742460 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1dc40588b0 con 0x7f1dd00cdcb0 2024-09-16T01:42:42.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.594+0000 7f1dd4f59640 1 -- 172.21.15.88:0/3035742460 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1dd01423f0 con 0x7f1dd00cdcb0 2024-09-16T01:42:42.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.595+0000 7f1dbeffd640 1 -- 172.21.15.88:0/3035742460 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 216 v216) ==== 95+0+3566 (secure 0 0 0) 0x7f1dc405cbb0 con 0x7f1dd00cdcb0 2024-09-16T01:42:42.596 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:42.597 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":216,"btime":"2024-09-16T01:42:41:630129+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":216,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:41.630075+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":112,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:42.597 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 216 2024-09-16T01:42:42.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.597+0000 7f1dd4f59640 1 -- 172.21.15.88:0/3035742460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1db006c2b0 msgr2=0x7f1db006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:42.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.597+0000 7f1dd4f59640 1 --2- 172.21.15.88:0/3035742460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1db006c2b0 0x7f1db006e770 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f1dd00732e0 tx=0x7f1dc0009060 comp rx=0 tx=0).stop 2024-09-16T01:42:42.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.597+0000 7f1dd4f59640 1 -- 172.21.15.88:0/3035742460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dd00cdcb0 msgr2=0x7f1dd01f3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:42.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.597+0000 7f1dd4f59640 1 --2- 172.21.15.88:0/3035742460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dd00cdcb0 0x7f1dd01f3490 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f1dc400c430 tx=0x7f1dc400c8f0 comp rx=0 tx=0).stop 2024-09-16T01:42:42.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.597+0000 7f1dd4f59640 1 -- 172.21.15.88:0/3035742460 shutdown_connections 2024-09-16T01:42:42.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.597+0000 7f1dd4f59640 1 --2- 172.21.15.88:0/3035742460 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1db006c2b0 0x7f1db006e770 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:42.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.597+0000 7f1dd4f59640 1 --2- 172.21.15.88:0/3035742460 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1dd01450d0 0x7f1dd01f39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:42.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.597+0000 7f1dd4f59640 1 --2- 172.21.15.88:0/3035742460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1dd00cdcb0 0x7f1dd01f3490 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:42.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.597+0000 7f1dd4f59640 1 --2- 172.21.15.88:0/3035742460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dd006ecd0 0x7f1dd01f2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:42.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.597+0000 7f1dd4f59640 1 -- 172.21.15.88:0/3035742460 >> 172.21.15.88:0/3035742460 conn(0x7f1dd006e1e0 msgr2=0x7f1dd0144030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:42.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.597+0000 7f1dd4f59640 1 -- 172.21.15.88:0/3035742460 shutdown_connections 2024-09-16T01:42:42.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.597+0000 7f1dd4f59640 1 -- 172.21.15.88:0/3035742460 wait complete. 2024-09-16T01:42:42.609 INFO:tasks.cephfs.test_damage:Result: Mutation 'Truncate 200.00000000' led to DAMAGED state 2024-09-16T01:42:42.609 INFO:tasks.cephfs.test_damage:Applying mutation 'Truncate 200.00000001' 2024-09-16T01:42:42.610 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:42:42.610 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:42:42.625 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:42:42.626 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:42:42.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:42.626 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:42:42.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:42:42.682 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:42:42.682 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:42:42.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.820+0000 7f24c29fd640 1 Processor -- start 2024-09-16T01:42:42.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.821+0000 7f24c29fd640 1 -- start start 2024-09-16T01:42:42.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.821+0000 7f24c29fd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f24bc15b480 0x7f24bc0b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:42.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.821+0000 7f24c29fd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24bc0b60c0 0x7f24bc071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:42.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.821+0000 7f24c29fd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f24bc072530 0x7f24bc070fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:42.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.821+0000 7f24c29fd640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f24bc0b6500 con 0x7f24bc072530 2024-09-16T01:42:42.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.821+0000 7f24c29fd640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f24bc072970 con 0x7f24bc15b480 2024-09-16T01:42:42.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.821+0000 7f24c29fd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f24bc0ccff0 con 0x7f24bc0b60c0 2024-09-16T01:42:42.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.822+0000 7f24c0f9a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f24bc072530 0x7f24bc070fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:42.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.822+0000 7f24c0f9a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f24bc072530 0x7f24bc070fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36914/0 (socket says 172.21.15.88:36914) 2024-09-16T01:42:42.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.822+0000 7f24c0f9a640 1 -- 172.21.15.88:0/1588137973 learned_addr learned my addr 172.21.15.88:0/1588137973 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:42.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.822+0000 7f24bbfff640 1 --2- 172.21.15.88:0/1588137973 >> v2:172.21.15.154:3300/0 conn(0x7f24bc15b480 0x7f24bc0b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:42.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.822+0000 7f24bb7fe640 1 --2- 172.21.15.88:0/1588137973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24bc0b60c0 0x7f24bc071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:42.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.822+0000 7f24c0f9a640 1 -- 172.21.15.88:0/1588137973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24bc0b60c0 msgr2=0x7f24bc071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:42.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.822+0000 7f24c0f9a640 1 --2- 172.21.15.88:0/1588137973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24bc0b60c0 0x7f24bc071ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:42.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.822+0000 7f24c0f9a640 1 -- 172.21.15.88:0/1588137973 >> v2:172.21.15.154:3300/0 conn(0x7f24bc15b480 msgr2=0x7f24bc0b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:42.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.822+0000 7f24c0f9a640 1 --2- 172.21.15.88:0/1588137973 >> v2:172.21.15.154:3300/0 conn(0x7f24bc15b480 0x7f24bc0b5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:42.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.822+0000 7f24c0f9a640 1 -- 172.21.15.88:0/1588137973 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24bc0cd2d0 con 0x7f24bc072530 2024-09-16T01:42:42.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.823+0000 7f24c0f9a640 1 --2- 172.21.15.88:0/1588137973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24bc072530 0x7f24bc070fb0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f24bc0694d0 tx=0x7f24b402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0ddd192a25f4e75 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:42.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.823+0000 7f24baffd640 1 -- 172.21.15.88:0/1588137973 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b4036070 con 0x7f24bc072530 2024-09-16T01:42:42.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.823+0000 7f24baffd640 1 -- 172.21.15.88:0/1588137973 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24b4031600 con 0x7f24bc072530 2024-09-16T01:42:42.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.823+0000 7f24baffd640 1 -- 172.21.15.88:0/1588137973 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b40319d0 con 0x7f24bc072530 2024-09-16T01:42:42.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.823+0000 7f24c29fd640 1 -- 172.21.15.88:0/1588137973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24bc072530 msgr2=0x7f24bc070fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:42.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.823+0000 7f24c29fd640 1 --2- 172.21.15.88:0/1588137973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24bc072530 0x7f24bc070fb0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f24bc0694d0 tx=0x7f24b402a610 comp rx=0 tx=0).stop 2024-09-16T01:42:42.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.823+0000 7f24c29fd640 1 -- 172.21.15.88:0/1588137973 shutdown_connections 2024-09-16T01:42:42.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.823+0000 7f24c29fd640 1 --2- 172.21.15.88:0/1588137973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24bc072530 0x7f24bc070fb0 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:42.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.823+0000 7f24c29fd640 1 --2- 172.21.15.88:0/1588137973 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24bc0b60c0 0x7f24bc071ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:42.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.823+0000 7f24c29fd640 1 --2- 172.21.15.88:0/1588137973 >> v2:172.21.15.154:3300/0 conn(0x7f24bc15b480 0x7f24bc0b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:42.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.823+0000 7f24c29fd640 1 -- 172.21.15.88:0/1588137973 >> 172.21.15.88:0/1588137973 conn(0x7f24bc06d0f0 msgr2=0x7f24bc06d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:42.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.823+0000 7f24c29fd640 1 -- 172.21.15.88:0/1588137973 shutdown_connections 2024-09-16T01:42:42.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.823+0000 7f24c29fd640 1 -- 172.21.15.88:0/1588137973 wait complete. 2024-09-16T01:42:42.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.825+0000 7f24c29fd640 1 Processor -- start 2024-09-16T01:42:42.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.825+0000 7f24c29fd640 1 -- start start 2024-09-16T01:42:42.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.825+0000 7f24c29fd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24bc072530 0x7f24bc142b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:42.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.825+0000 7f24c29fd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24bc0b60c0 0x7f24bc15da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:42.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.825+0000 7f24c29fd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24bc15b480 0x7f24bc14c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:42.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.825+0000 7f24c29fd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f24bc0cdb70 con 0x7f24bc15b480 2024-09-16T01:42:42.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.825+0000 7f24c29fd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f24bc0cd890 con 0x7f24bc072530 2024-09-16T01:42:42.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.825+0000 7f24c29fd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f24bc15b860 con 0x7f24bc0b60c0 2024-09-16T01:42:42.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.826+0000 7f24c0f9a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24bc15b480 0x7f24bc14c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:42.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.826+0000 7f24c0f9a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24bc15b480 0x7f24bc14c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36930/0 (socket says 172.21.15.88:36930) 2024-09-16T01:42:42.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.826+0000 7f24c0f9a640 1 -- 172.21.15.88:0/478413746 learned_addr learned my addr 172.21.15.88:0/478413746 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:42.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.826+0000 7f24bbfff640 1 --2- 172.21.15.88:0/478413746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24bc072530 0x7f24bc142b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:42.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.826+0000 7f24bb7fe640 1 --2- 172.21.15.88:0/478413746 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24bc0b60c0 0x7f24bc15da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:42.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.826+0000 7f24c0f9a640 1 -- 172.21.15.88:0/478413746 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24bc0b60c0 msgr2=0x7f24bc15da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:42.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.826+0000 7f24c0f9a640 1 --2- 172.21.15.88:0/478413746 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24bc0b60c0 0x7f24bc15da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:42.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.826+0000 7f24c0f9a640 1 -- 172.21.15.88:0/478413746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24bc072530 msgr2=0x7f24bc142b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:42.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.826+0000 7f24c0f9a640 1 --2- 172.21.15.88:0/478413746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24bc072530 0x7f24bc142b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:42.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.826+0000 7f24c0f9a640 1 -- 172.21.15.88:0/478413746 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24bc15ca10 con 0x7f24bc15b480 2024-09-16T01:42:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.826+0000 7f24c0f9a640 1 --2- 172.21.15.88:0/478413746 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24bc15b480 0x7f24bc14c7a0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f24b4009f40 tx=0x7f24b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.826+0000 7f24b8ff9640 1 -- 172.21.15.88:0/478413746 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b4036050 con 0x7f24bc15b480 2024-09-16T01:42:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.826+0000 7f24b8ff9640 1 -- 172.21.15.88:0/478413746 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24b4042070 con 0x7f24bc15b480 2024-09-16T01:42:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.826+0000 7f24c29fd640 1 -- 172.21.15.88:0/478413746 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24bc15ccb0 con 0x7f24bc15b480 2024-09-16T01:42:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.826+0000 7f24b8ff9640 1 -- 172.21.15.88:0/478413746 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b4009510 con 0x7f24bc15b480 2024-09-16T01:42:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.826+0000 7f24c29fd640 1 -- 172.21.15.88:0/478413746 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24bc15d270 con 0x7f24bc15b480 2024-09-16T01:42:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.827+0000 7f24c29fd640 1 -- 172.21.15.88:0/478413746 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24bc069180 con 0x7f24bc15b480 2024-09-16T01:42:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.828+0000 7f24b8ff9640 1 -- 172.21.15.88:0/478413746 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f24b4007bd0 con 0x7f24bc15b480 2024-09-16T01:42:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.828+0000 7f24b8ff9640 1 --2- 172.21.15.88:0/478413746 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f249c06c2b0 0x7f249c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.828+0000 7f24b8ff9640 1 -- 172.21.15.88:0/478413746 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 7888+0+0 (secure 0 0 0) 0x7f24b40ac9e0 con 0x7f24bc15b480 2024-09-16T01:42:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.828+0000 7f24bbfff640 1 --2- 172.21.15.88:0/478413746 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f249c06c2b0 0x7f249c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:42.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.829+0000 7f24bbfff640 1 --2- 172.21.15.88:0/478413746 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f249c06c2b0 0x7f249c06e770 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0079a0 tx=0x7f24ac008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:42.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:42.833+0000 7f24b8ff9640 1 -- 172.21.15.88:0/478413746 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f24bc069180 con 0x7f24bc15b480 2024-09-16T01:42:43.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.030+0000 7f24c29fd640 1 -- 172.21.15.88:0/478413746 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f24bc0cff70 con 0x7f24bc15b480 2024-09-16T01:42:43.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.642+0000 7f24b8ff9640 1 -- 172.21.15.88:0/478413746 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v218) ==== 161+0+0 (secure 0 0 0) 0x7f24b407a280 con 0x7f24bc15b480 2024-09-16T01:42:43.644 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:42:43.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.645+0000 7f24c29fd640 1 -- 172.21.15.88:0/478413746 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f249c06c2b0 msgr2=0x7f249c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:43.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.645+0000 7f24c29fd640 1 --2- 172.21.15.88:0/478413746 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f249c06c2b0 0x7f249c06e770 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0079a0 tx=0x7f24ac008040 comp rx=0 tx=0).stop 2024-09-16T01:42:43.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.645+0000 7f24c29fd640 1 -- 172.21.15.88:0/478413746 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24bc15b480 msgr2=0x7f24bc14c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:43.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.645+0000 7f24c29fd640 1 --2- 172.21.15.88:0/478413746 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24bc15b480 0x7f24bc14c7a0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f24b4009f40 tx=0x7f24b4004a40 comp rx=0 tx=0).stop 2024-09-16T01:42:43.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.645+0000 7f24c29fd640 1 -- 172.21.15.88:0/478413746 shutdown_connections 2024-09-16T01:42:43.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.645+0000 7f24c29fd640 1 --2- 172.21.15.88:0/478413746 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f249c06c2b0 0x7f249c06e770 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:43.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.645+0000 7f24c29fd640 1 --2- 172.21.15.88:0/478413746 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24bc15b480 0x7f24bc14c7a0 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:43.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.645+0000 7f24c29fd640 1 --2- 172.21.15.88:0/478413746 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24bc0b60c0 0x7f24bc15da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:43.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.645+0000 7f24c29fd640 1 --2- 172.21.15.88:0/478413746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24bc072530 0x7f24bc142b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:43.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.645+0000 7f24c29fd640 1 -- 172.21.15.88:0/478413746 >> 172.21.15.88:0/478413746 conn(0x7f24bc06d0f0 msgr2=0x7f24bc0b3490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:43.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.645+0000 7f24c29fd640 1 -- 172.21.15.88:0/478413746 shutdown_connections 2024-09-16T01:42:43.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.645+0000 7f24c29fd640 1 -- 172.21.15.88:0/478413746 wait complete. 2024-09-16T01:42:43.659 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:42:43.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.758+0000 7f323880e640 1 Processor -- start 2024-09-16T01:42:43.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.759+0000 7f323880e640 1 -- start start 2024-09-16T01:42:43.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.759+0000 7f323880e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f323415e7a0 0x7f323406caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:43.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.759+0000 7f323880e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32340b68f0 0x7f32340d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:43.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.759+0000 7f323880e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32340cf860 con 0x7f32340cf1f0 2024-09-16T01:42:43.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.759+0000 7f323880e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f32340d2ca0 con 0x7f323415e7a0 2024-09-16T01:42:43.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.759+0000 7f323880e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f32340cfb40 con 0x7f32340b68f0 2024-09-16T01:42:43.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f3231d9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f32340cf1f0 0x7f32340b61e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49248/0 (socket says 172.21.15.88:49248) 2024-09-16T01:42:43.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f3231d9b640 1 -- 172.21.15.88:0/3562145770 learned_addr learned my addr 172.21.15.88:0/3562145770 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:43.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f323259c640 1 --2- 172.21.15.88:0/3562145770 >> v2:172.21.15.154:3300/0 conn(0x7f323415e7a0 0x7f323406caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:43.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f3232d9d640 1 --2- 172.21.15.88:0/3562145770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32340b68f0 0x7f32340d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:43.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f323159a640 1 -- 172.21.15.88:0/3562145770 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 641333704 0 0) 0x7f32340cf860 con 0x7f32340cf1f0 2024-09-16T01:42:43.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f323159a640 1 -- 172.21.15.88:0/3562145770 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f321c003520 con 0x7f32340cf1f0 2024-09-16T01:42:43.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f323159a640 1 -- 172.21.15.88:0/3562145770 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1683346695 0 0) 0x7f321c003520 con 0x7f32340cf1f0 2024-09-16T01:42:43.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f323159a640 1 -- 172.21.15.88:0/3562145770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32340b68f0 msgr2=0x7f32340d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:43.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f323159a640 1 --2- 172.21.15.88:0/3562145770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32340b68f0 0x7f32340d3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:43.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f323159a640 1 -- 172.21.15.88:0/3562145770 >> v2:172.21.15.154:3300/0 conn(0x7f323415e7a0 msgr2=0x7f323406caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:43.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f323159a640 1 --2- 172.21.15.88:0/3562145770 >> v2:172.21.15.154:3300/0 conn(0x7f323415e7a0 0x7f323406caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:43.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f323159a640 1 -- 172.21.15.88:0/3562145770 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32340cfe20 con 0x7f32340cf1f0 2024-09-16T01:42:43.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f323159a640 1 -- 172.21.15.88:0/3562145770 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f32280019f0 con 0x7f32340cf1f0 2024-09-16T01:42:43.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f323159a640 1 -- 172.21.15.88:0/3562145770 >> v1:172.21.15.88:6789/0 conn(0x7f32340cf1f0 legacy=0x7f32340b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:43.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f323159a640 1 --2- 172.21.15.88:0/3562145770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f321c0039b0 0x7f321c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:43.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f323159a640 1 --2- 172.21.15.88:0/3562145770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f321c004800 0x7f321c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:43.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f323159a640 1 --2- 172.21.15.88:0/3562145770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32340b68f0 0x7f321c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:43.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f323159a640 1 -- 172.21.15.88:0/3562145770 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f32340d2ca0 con 0x7f321c0039b0 2024-09-16T01:42:43.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f323159a640 1 -- 172.21.15.88:0/3562145770 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f32340cfb40 con 0x7f321c004800 2024-09-16T01:42:43.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f323159a640 1 -- 172.21.15.88:0/3562145770 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f321c00b9f0 con 0x7f32340b68f0 2024-09-16T01:42:43.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f323259c640 1 --2- 172.21.15.88:0/3562145770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f321c0039b0 0x7f321c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:43.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f3231d9b640 1 --2- 172.21.15.88:0/3562145770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f321c004800 0x7f321c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:43.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f3232d9d640 1 --2- 172.21.15.88:0/3562145770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32340b68f0 0x7f321c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:43.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f323259c640 1 -- 172.21.15.88:0/3562145770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32340b68f0 msgr2=0x7f321c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:43.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f323259c640 1 --2- 172.21.15.88:0/3562145770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32340b68f0 0x7f321c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:43.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f323259c640 1 -- 172.21.15.88:0/3562145770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f321c004800 msgr2=0x7f321c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:43.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f323259c640 1 --2- 172.21.15.88:0/3562145770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f321c004800 0x7f321c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:43.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.760+0000 7f323259c640 1 -- 172.21.15.88:0/3562145770 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f321c003520 con 0x7f321c0039b0 2024-09-16T01:42:43.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.761+0000 7f323259c640 1 --2- 172.21.15.88:0/3562145770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f321c0039b0 0x7f321c003dd0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f322c009ac0 tx=0x7f321c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f8f5de09c7cbf12 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:43.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.761+0000 7f323159a640 1 -- 172.21.15.88:0/3562145770 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322c036070 con 0x7f321c0039b0 2024-09-16T01:42:43.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.761+0000 7f323159a640 1 -- 172.21.15.88:0/3562145770 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f322c031400 con 0x7f321c0039b0 2024-09-16T01:42:43.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.761+0000 7f323159a640 1 -- 172.21.15.88:0/3562145770 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322c009690 con 0x7f321c0039b0 2024-09-16T01:42:43.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.762+0000 7f323880e640 1 -- 172.21.15.88:0/3562145770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f321c0039b0 msgr2=0x7f321c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:43.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.762+0000 7f323880e640 1 --2- 172.21.15.88:0/3562145770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f321c0039b0 0x7f321c003dd0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f322c009ac0 tx=0x7f321c005830 comp rx=0 tx=0).stop 2024-09-16T01:42:43.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.762+0000 7f3232d9d640 1 -- 172.21.15.88:0/3562145770 reap_dead start 2024-09-16T01:42:43.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.762+0000 7f323880e640 1 -- 172.21.15.88:0/3562145770 shutdown_connections 2024-09-16T01:42:43.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.762+0000 7f323880e640 1 -- 172.21.15.88:0/3562145770 >> 172.21.15.88:0/3562145770 conn(0x7f32340d11e0 msgr2=0x7f32340d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:43.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.762+0000 7f323880e640 1 -- 172.21.15.88:0/3562145770 shutdown_connections 2024-09-16T01:42:43.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.762+0000 7f323880e640 1 -- 172.21.15.88:0/3562145770 wait complete. 2024-09-16T01:42:43.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.763+0000 7f323880e640 1 Processor -- start 2024-09-16T01:42:43.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.763+0000 7f323880e640 1 -- start start 2024-09-16T01:42:43.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.764+0000 7f323880e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f321c0039b0 0x7f3234153eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:43.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.764+0000 7f323880e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32340b6480 0x7f3234149a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:43.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.764+0000 7f323880e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32340b5210 0x7f32340b5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:43.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.764+0000 7f323880e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f323406cfe0 con 0x7f32340b6480 2024-09-16T01:42:43.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.764+0000 7f323880e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f32340b7a00 con 0x7f321c0039b0 2024-09-16T01:42:43.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.764+0000 7f323880e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f323415eb80 con 0x7f32340b5210 2024-09-16T01:42:43.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.764+0000 7f3231d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32340b6480 0x7f3234149a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:43.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.764+0000 7f3231d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32340b6480 0x7f3234149a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36950/0 (socket says 172.21.15.88:36950) 2024-09-16T01:42:43.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.764+0000 7f3231d9b640 1 -- 172.21.15.88:0/2007609843 learned_addr learned my addr 172.21.15.88:0/2007609843 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:43.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.764+0000 7f323259c640 1 --2- 172.21.15.88:0/2007609843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f321c0039b0 0x7f3234153eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:43.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.764+0000 7f3232d9d640 1 --2- 172.21.15.88:0/2007609843 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32340b5210 0x7f32340b5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:43.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.764+0000 7f3231d9b640 1 -- 172.21.15.88:0/2007609843 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32340b5210 msgr2=0x7f32340b5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:43.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.764+0000 7f3231d9b640 1 --2- 172.21.15.88:0/2007609843 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32340b5210 0x7f32340b5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:43.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.764+0000 7f3231d9b640 1 -- 172.21.15.88:0/2007609843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f321c0039b0 msgr2=0x7f3234153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:43.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.764+0000 7f3231d9b640 1 --2- 172.21.15.88:0/2007609843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f321c0039b0 0x7f3234153eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:43.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.764+0000 7f3231d9b640 1 -- 172.21.15.88:0/2007609843 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f323415f060 con 0x7f32340b6480 2024-09-16T01:42:43.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.764+0000 7f3231d9b640 1 --2- 172.21.15.88:0/2007609843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32340b6480 0x7f3234149a30 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f3228007b00 tx=0x7f322800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:43.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.765+0000 7f3222ffd640 1 -- 172.21.15.88:0/2007609843 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322800edf0 con 0x7f32340b6480 2024-09-16T01:42:43.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.765+0000 7f3222ffd640 1 -- 172.21.15.88:0/2007609843 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3228018070 con 0x7f32340b6480 2024-09-16T01:42:43.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.765+0000 7f323880e640 1 -- 172.21.15.88:0/2007609843 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f323415f340 con 0x7f32340b6480 2024-09-16T01:42:43.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.765+0000 7f3222ffd640 1 -- 172.21.15.88:0/2007609843 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3228013640 con 0x7f32340b6480 2024-09-16T01:42:43.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.765+0000 7f323880e640 1 -- 172.21.15.88:0/2007609843 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f323415f900 con 0x7f32340b6480 2024-09-16T01:42:43.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.766+0000 7f323880e640 1 -- 172.21.15.88:0/2007609843 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32340cb200 con 0x7f32340b6480 2024-09-16T01:42:43.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.766+0000 7f3222ffd640 1 -- 172.21.15.88:0/2007609843 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f32280137e0 con 0x7f32340b6480 2024-09-16T01:42:43.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.766+0000 7f3222ffd640 1 --2- 172.21.15.88:0/2007609843 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f321406c2b0 0x7f321406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:43.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.767+0000 7f3222ffd640 1 -- 172.21.15.88:0/2007609843 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 7888+0+0 (secure 0 0 0) 0x7f322808c1a0 con 0x7f32340b6480 2024-09-16T01:42:43.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.767+0000 7f323259c640 1 --2- 172.21.15.88:0/2007609843 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f321406c2b0 0x7f321406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:43.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.767+0000 7f323259c640 1 --2- 172.21.15.88:0/2007609843 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f321406c2b0 0x7f321406e770 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f322c033000 tx=0x7f322c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:43.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:43.771+0000 7f3222ffd640 1 -- 172.21.15.88:0/2007609843 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f32340cb200 con 0x7f32340b6480 2024-09-16T01:42:44.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.004+0000 7f323880e640 1 -- 172.21.15.88:0/2007609843 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f323406caa0 con 0x7f32340b6480 2024-09-16T01:42:44.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.649+0000 7f3222ffd640 1 -- 172.21.15.88:0/2007609843 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v219) ==== 101+0+0 (secure 0 0 0) 0x7f3228059a40 con 0x7f32340b6480 2024-09-16T01:42:44.650 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:42:44.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.651+0000 7f323880e640 1 -- 172.21.15.88:0/2007609843 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f321406c2b0 msgr2=0x7f321406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:44.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.651+0000 7f323880e640 1 --2- 172.21.15.88:0/2007609843 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f321406c2b0 0x7f321406e770 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f322c033000 tx=0x7f322c008030 comp rx=0 tx=0).stop 2024-09-16T01:42:44.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.651+0000 7f323880e640 1 -- 172.21.15.88:0/2007609843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32340b6480 msgr2=0x7f3234149a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:44.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.651+0000 7f323880e640 1 --2- 172.21.15.88:0/2007609843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32340b6480 0x7f3234149a30 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f3228007b00 tx=0x7f322800e530 comp rx=0 tx=0).stop 2024-09-16T01:42:44.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.651+0000 7f323880e640 1 -- 172.21.15.88:0/2007609843 shutdown_connections 2024-09-16T01:42:44.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.651+0000 7f323880e640 1 --2- 172.21.15.88:0/2007609843 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f321406c2b0 0x7f321406e770 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:44.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.651+0000 7f323880e640 1 --2- 172.21.15.88:0/2007609843 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32340b5210 0x7f32340b5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:44.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.651+0000 7f323880e640 1 --2- 172.21.15.88:0/2007609843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32340b6480 0x7f3234149a30 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:44.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.651+0000 7f323880e640 1 --2- 172.21.15.88:0/2007609843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f321c0039b0 0x7f3234153eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:44.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.651+0000 7f323880e640 1 -- 172.21.15.88:0/2007609843 >> 172.21.15.88:0/2007609843 conn(0x7f32340d11e0 msgr2=0x7f323406c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:44.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.651+0000 7f323880e640 1 -- 172.21.15.88:0/2007609843 shutdown_connections 2024-09-16T01:42:44.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.651+0000 7f323880e640 1 -- 172.21.15.88:0/2007609843 wait complete. 2024-09-16T01:42:44.668 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:42:44.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.712+0000 7fc1fbd89f40 1 Processor -- start 2024-09-16T01:42:44.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.712+0000 7fc1fbd89f40 1 -- start start 2024-09-16T01:42:44.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.713+0000 7fc1fbd89f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1e80030f0 0x5566e22c0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:44.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.713+0000 7fc1fbd89f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5566e229e890 0x5566e229ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:44.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.713+0000 7fc1fbd89f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5566e2233840 0x5566e2233cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:44.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.713+0000 7fc1fbd89f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5566e213b510 con 0x5566e2233840 2024-09-16T01:42:44.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.713+0000 7fc1fbd89f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5566e2194c50 con 0x5566e229e890 2024-09-16T01:42:44.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.713+0000 7fc1fbd89f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5566e222f550 con 0x7fc1e80030f0 2024-09-16T01:42:44.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.713+0000 7fc1fa080640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5566e2233840 0x5566e2233cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:44.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.713+0000 7fc1fa080640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5566e2233840 0x5566e2233cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36964/0 (socket says 172.21.15.88:36964) 2024-09-16T01:42:44.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.713+0000 7fc1fa080640 1 -- 172.21.15.88:0/104714966 learned_addr learned my addr 172.21.15.88:0/104714966 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:44.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.713+0000 7fc1f907e640 1 --2- 172.21.15.88:0/104714966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5566e229e890 0x5566e229ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:44.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.713+0000 7fc1f987f640 1 --2- 172.21.15.88:0/104714966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1e80030f0 0x5566e22c0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:44.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.713+0000 7fc1fa080640 1 -- 172.21.15.88:0/104714966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1e80030f0 msgr2=0x5566e22c0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:44.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.713+0000 7fc1fa080640 1 --2- 172.21.15.88:0/104714966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1e80030f0 0x5566e22c0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:44.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.713+0000 7fc1fa080640 1 -- 172.21.15.88:0/104714966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5566e229e890 msgr2=0x5566e229ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:44.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.713+0000 7fc1fa080640 1 --2- 172.21.15.88:0/104714966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5566e229e890 0x5566e229ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:44.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.713+0000 7fc1fa080640 1 -- 172.21.15.88:0/104714966 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566e22c2430 con 0x5566e2233840 2024-09-16T01:42:44.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.713+0000 7fc1fa080640 1 --2- 172.21.15.88:0/104714966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5566e2233840 0x5566e2233cc0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x5566e22c4d70 tx=0x7fc1ec0017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9326c4d35fef821f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:44.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.713+0000 7fc1f3fff640 1 -- 172.21.15.88:0/104714966 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ec00f450 con 0x5566e2233840 2024-09-16T01:42:44.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.713+0000 7fc1f3fff640 1 -- 172.21.15.88:0/104714966 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1ec014020 con 0x5566e2233840 2024-09-16T01:42:44.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.713+0000 7fc1f3fff640 1 -- 172.21.15.88:0/104714966 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ec00f850 con 0x5566e2233840 2024-09-16T01:42:44.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.714+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/104714966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5566e2233840 msgr2=0x5566e2233cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:44.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.714+0000 7fc1fbd89f40 1 --2- 172.21.15.88:0/104714966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5566e2233840 0x5566e2233cc0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x5566e22c4d70 tx=0x7fc1ec0017a0 comp rx=0 tx=0).stop 2024-09-16T01:42:44.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.714+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/104714966 shutdown_connections 2024-09-16T01:42:44.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.714+0000 7fc1fbd89f40 1 --2- 172.21.15.88:0/104714966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5566e2233840 0x5566e2233cc0 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:44.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.714+0000 7fc1fbd89f40 1 --2- 172.21.15.88:0/104714966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5566e229e890 0x5566e229ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:44.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.714+0000 7fc1fbd89f40 1 --2- 172.21.15.88:0/104714966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1e80030f0 0x5566e22c0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:44.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.714+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/104714966 >> 172.21.15.88:0/104714966 conn(0x5566e21e8be0 msgr2=0x5566e21ea1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:44.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.714+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/104714966 shutdown_connections 2024-09-16T01:42:44.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.714+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/104714966 wait complete. 2024-09-16T01:42:44.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.714+0000 7fc1fbd89f40 1 Processor -- start 2024-09-16T01:42:44.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.714+0000 7fc1fbd89f40 1 -- start start 2024-09-16T01:42:44.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.715+0000 7fc1fbd89f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5566e2233840 0x5566e22bb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:44.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.715+0000 7fc1fbd89f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5566e229e890 0x5566e22a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:44.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.715+0000 7fc1fbd89f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1e80030f0 0x5566e22a5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:44.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.715+0000 7fc1fbd89f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5566e22a6c00 con 0x7fc1e80030f0 2024-09-16T01:42:44.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.715+0000 7fc1fbd89f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5566e22a6920 con 0x5566e229e890 2024-09-16T01:42:44.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.715+0000 7fc1fbd89f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5566e22c26d0 con 0x5566e2233840 2024-09-16T01:42:44.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.715+0000 7fc1fa080640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1e80030f0 0x5566e22a5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:44.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.715+0000 7fc1fa080640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1e80030f0 0x5566e22a5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36972/0 (socket says 172.21.15.88:36972) 2024-09-16T01:42:44.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.715+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 learned_addr learned my addr 172.21.15.88:0/2387703978 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:44.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.715+0000 7fc1f907e640 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5566e229e890 0x5566e22a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:44.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.715+0000 7fc1f987f640 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5566e2233840 0x5566e22bb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:44.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.715+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5566e2233840 msgr2=0x5566e22bb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:44.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.715+0000 7fc1fa080640 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5566e2233840 0x5566e22bb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:44.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.715+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5566e229e890 msgr2=0x5566e22a57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:44.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.715+0000 7fc1fa080640 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5566e229e890 0x5566e22a57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:44.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.715+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566e22a3580 con 0x7fc1e80030f0 2024-09-16T01:42:44.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.715+0000 7fc1f907e640 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5566e229e890 0x5566e22a57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:44.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.715+0000 7fc1f987f640 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5566e2233840 0x5566e22bb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:44.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.716+0000 7fc1fa080640 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1e80030f0 0x5566e22a5d00 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec012040 tx=0x7fc1ec001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:44.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.716+0000 7fc1f2ffd640 1 -- 172.21.15.88:0/2387703978 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ec029070 con 0x7fc1e80030f0 2024-09-16T01:42:44.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.716+0000 7fc1f2ffd640 1 -- 172.21.15.88:0/2387703978 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1ec00f660 con 0x7fc1e80030f0 2024-09-16T01:42:44.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.716+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5566e22bda30 con 0x7fc1e80030f0 2024-09-16T01:42:44.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.716+0000 7fc1f2ffd640 1 -- 172.21.15.88:0/2387703978 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ec013040 con 0x7fc1e80030f0 2024-09-16T01:42:44.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.716+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566e22bdff0 con 0x7fc1e80030f0 2024-09-16T01:42:44.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.717+0000 7fc1f2ffd640 1 -- 172.21.15.88:0/2387703978 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc1ec02c070 con 0x7fc1e80030f0 2024-09-16T01:42:44.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.718+0000 7fc1f2ffd640 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc1dc06ac90 0x7fc1dc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:44.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.718+0000 7fc1f2ffd640 1 -- 172.21.15.88:0/2387703978 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 7888+0+0 (secure 0 0 0) 0x7fc1ec08c560 con 0x7fc1e80030f0 2024-09-16T01:42:44.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.718+0000 7fc1f987f640 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc1dc06ac90 0x7fc1dc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:44.724 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:42:44.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.718+0000 7fc1fbd89f40 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5566e22c1fa0 0x5566e2358730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:44.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.718+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e22bb8d0 con 0x5566e22c1fa0 2024-09-16T01:42:44.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.718+0000 7fc1f907e640 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5566e22c1fa0 0x5566e2358730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:44.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.718+0000 7fc1f987f640 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc1dc06ac90 0x7fc1dc06d150 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x5566e21d6320 tx=0x7fc1f4007040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:44.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.718+0000 7fc1f907e640 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5566e22c1fa0 0x5566e2358730 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:44.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.720+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v112'753 uv727 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.720+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e112) -- 0x5566e2358c70 con 0x5566e22c1fa0 2024-09-16T01:42:44.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.722+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v112'754 uv754 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.722+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e23647e0 con 0x5566e22c1fa0 2024-09-16T01:42:44.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.723+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v112'755 uv755 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.726 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:42:44.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.723+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2365b90 con 0x5566e22c1fa0 2024-09-16T01:42:44.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.725+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v112'756 uv756 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.725+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2365f30 con 0x5566e22c1fa0 2024-09-16T01:42:44.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.726+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v112'757 uv757 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.726+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364b80 con 0x5566e22c1fa0 2024-09-16T01:42:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.728+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v112'758 uv758 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.728+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e22c1fa0 2024-09-16T01:42:44.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.729+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v112'759 uv733 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.729+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e22c1fa0 2024-09-16T01:42:44.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.730+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v112'760 uv760 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.730+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e22c1fa0 2024-09-16T01:42:44.733 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:42:44.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.732+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v112'761 uv761 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.732+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e22c1fa0 2024-09-16T01:42:44.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.733+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v112'762 uv762 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.733+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e22c1fa0 2024-09-16T01:42:44.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.735+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v112'763 uv763 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.735+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e22c1fa0 2024-09-16T01:42:44.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.736+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v112'764 uv764 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.736+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e22c1fa0 2024-09-16T01:42:44.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.737+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v112'765 uv740 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.737+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e22c1fa0 2024-09-16T01:42:44.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.739+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v112'766 uv766 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.739+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e22c1fa0 2024-09-16T01:42:44.741 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:42:44.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.740+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v112'767 uv767 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.740+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e22c1fa0 2024-09-16T01:42:44.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.741+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v112'768 uv768 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.741+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e22c1fa0 2024-09-16T01:42:44.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.742+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v112'769 uv769 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.742+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e22c1fa0 2024-09-16T01:42:44.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.744+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v112'770 uv770 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.744+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e22c1fa0 2024-09-16T01:42:44.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.745+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v112'771 uv771 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.745+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e22c1fa0 2024-09-16T01:42:44.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.746+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v112'772 uv746 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.746+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e22c1fa0 2024-09-16T01:42:44.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.748+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v112'773 uv773 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.748+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e22c1fa0 2024-09-16T01:42:44.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.749+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v112'774 uv774 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.750 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:42:44.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.749+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e22c1fa0 2024-09-16T01:42:44.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.750+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v112'775 uv775 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.750+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e22c1fa0 2024-09-16T01:42:44.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.751+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v112'776 uv776 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.751+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e22c1fa0 2024-09-16T01:42:44.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.752+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v112'777 uv777 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.752+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e22c1fa0 2024-09-16T01:42:44.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.754+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v112'778 uv752 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.754+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e22c1fa0 2024-09-16T01:42:44.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.755+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v112'779 uv779 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.755+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e22c1fa0 2024-09-16T01:42:44.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.756+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v112'780 uv780 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.757 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:42:44.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.756+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e22c1fa0 2024-09-16T01:42:44.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.757+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v112'781 uv781 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.757+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e22c1fa0 2024-09-16T01:42:44.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.758+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v112'782 uv782 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.758+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e22c1fa0 2024-09-16T01:42:44.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.760+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v112'783 uv783 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.760+0000 7fc1fbd89f40 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5566e23662d0 0x5566e23686c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:44.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.760+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2364f70 con 0x5566e23662d0 2024-09-16T01:42:44.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.760+0000 7fc1f987f640 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5566e23662d0 0x5566e23686c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:44.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.760+0000 7fc1f987f640 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5566e23662d0 0x5566e23686c0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:44.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.762+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v112'550 uv532 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e23662d0 2024-09-16T01:42:44.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.762+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e112) -- 0x5566e2368c00 con 0x5566e23662d0 2024-09-16T01:42:44.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.763+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v112'551 uv551 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e23662d0 2024-09-16T01:42:44.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.763+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2368fa0 con 0x5566e23662d0 2024-09-16T01:42:44.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.764+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v112'552 uv552 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e23662d0 2024-09-16T01:42:44.766 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:42:44.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.764+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2369340 con 0x5566e23662d0 2024-09-16T01:42:44.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.765+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v112'553 uv553 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e23662d0 2024-09-16T01:42:44.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.765+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e23696e0 con 0x5566e23662d0 2024-09-16T01:42:44.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.766+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v112'554 uv554 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e23662d0 2024-09-16T01:42:44.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.766+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2369a80 con 0x5566e23662d0 2024-09-16T01:42:44.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.768+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v112'555 uv549 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e23662d0 2024-09-16T01:42:44.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.768+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e112) -- 0x5566e2369e70 con 0x5566e23662d0 2024-09-16T01:42:44.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.769+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v112'556 uv556 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e23662d0 2024-09-16T01:42:44.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.769+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e236a290 con 0x5566e23662d0 2024-09-16T01:42:44.770 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:42:44.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.770+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v112'557 uv557 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e23662d0 2024-09-16T01:42:44.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.770+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e236a290 con 0x5566e23662d0 2024-09-16T01:42:44.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.771+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v112'558 uv558 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e23662d0 2024-09-16T01:42:44.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.771+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e236a290 con 0x5566e23662d0 2024-09-16T01:42:44.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.772+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v112'559 uv559 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e23662d0 2024-09-16T01:42:44.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.772+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e236a290 con 0x5566e23662d0 2024-09-16T01:42:44.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.773+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v112'560 uv542 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e23662d0 2024-09-16T01:42:44.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.773+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e112) -- 0x5566e236a290 con 0x5566e23662d0 2024-09-16T01:42:44.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.774+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v112'561 uv561 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e23662d0 2024-09-16T01:42:44.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.774+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e236a290 con 0x5566e23662d0 2024-09-16T01:42:44.776 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:42:44.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.775+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v112'562 uv562 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e23662d0 2024-09-16T01:42:44.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.775+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e236e710 con 0x5566e23662d0 2024-09-16T01:42:44.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.776+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v112'563 uv563 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e23662d0 2024-09-16T01:42:44.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.777+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e236e710 con 0x5566e23662d0 2024-09-16T01:42:44.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.777+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v112'564 uv564 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e23662d0 2024-09-16T01:42:44.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.778+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e236a290 con 0x5566e23662d0 2024-09-16T01:42:44.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.779+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v112'565 uv548 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e23662d0 2024-09-16T01:42:44.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.779+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e112) -- 0x5566e236a290 con 0x5566e23662d0 2024-09-16T01:42:44.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.780+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v112'566 uv566 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e23662d0 2024-09-16T01:42:44.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.780+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e236a290 con 0x5566e23662d0 2024-09-16T01:42:44.781 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:42:44.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.781+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v112'567 uv567 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e23662d0 2024-09-16T01:42:44.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.781+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e236b2a0 con 0x5566e23662d0 2024-09-16T01:42:44.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.781+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v112'568 uv568 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e23662d0 2024-09-16T01:42:44.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.782+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e236b2a0 con 0x5566e23662d0 2024-09-16T01:42:44.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.783+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v112'569 uv569 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e23662d0 2024-09-16T01:42:44.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.783+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e236b2a0 con 0x5566e23662d0 2024-09-16T01:42:44.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.784+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v112'570 uv570 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e23662d0 2024-09-16T01:42:44.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.784+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e236a210 con 0x5566e22c1fa0 2024-09-16T01:42:44.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.785+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v112'147 uv146 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.785+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e112) -- 0x5566e236a210 con 0x5566e22c1fa0 2024-09-16T01:42:44.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.786+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v112'148 uv148 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.786+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e236a210 con 0x5566e22c1fa0 2024-09-16T01:42:44.787 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:42:44.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.787+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v112'149 uv149 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.787+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e236b2a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.787+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v112'150 uv150 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.787+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e236b2a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.788+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v112'151 uv151 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.788+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e236b2a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.789+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v112'152 uv152 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.789+0000 7fc1fbd89f40 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5566e236a890 0x5566e236d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:44.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.789+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e236ae10 con 0x5566e236a890 2024-09-16T01:42:44.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.790+0000 7fc1f907e640 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5566e236a890 0x5566e236d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:44.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.790+0000 7fc1f907e640 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5566e236a890 0x5566e236d2b0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:44.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.791+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v112'109 uv108 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e236a890 2024-09-16T01:42:44.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.791+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e112) -- 0x5566e236d7f0 con 0x5566e236a890 2024-09-16T01:42:44.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.792+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v112'110 uv110 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e236a890 2024-09-16T01:42:44.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.792+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e236d7f0 con 0x5566e236a890 2024-09-16T01:42:44.794 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:42:44.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.793+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v112'111 uv111 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e236a890 2024-09-16T01:42:44.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.793+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e236d7f0 con 0x5566e236a890 2024-09-16T01:42:44.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.794+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v112'112 uv112 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e236a890 2024-09-16T01:42:44.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.794+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e236d7f0 con 0x5566e22c1fa0 2024-09-16T01:42:44.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.795+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v112'147 uv146 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.795+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e112) -- 0x5566e236d7f0 con 0x5566e22c1fa0 2024-09-16T01:42:44.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.796+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v112'148 uv148 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.796+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e236d7f0 con 0x5566e22c1fa0 2024-09-16T01:42:44.800 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:42:44.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.799+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v112'149 uv149 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.799+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e236e800 con 0x5566e22c1fa0 2024-09-16T01:42:44.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.800+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v112'150 uv150 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.800+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e236e800 con 0x5566e22c1fa0 2024-09-16T01:42:44.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.801+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v112'151 uv151 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.801+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e236d7f0 con 0x5566e22c1fa0 2024-09-16T01:42:44.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.802+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v112'152 uv152 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.802+0000 7fc1fbd89f40 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5566e236de70 0x5566e2370260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:44.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.802+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e23707e0 con 0x5566e236de70 2024-09-16T01:42:44.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.803+0000 7fc1fa080640 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5566e236de70 0x5566e2370260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:44.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.803+0000 7fc1fa080640 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5566e236de70 0x5566e2370260 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:44.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.804+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v112'271 uv265 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.804+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e112) -- 0x5566e2370c80 con 0x5566e236de70 2024-09-16T01:42:44.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.805+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v112'272 uv272 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.805+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2371160 con 0x5566e236de70 2024-09-16T01:42:44.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.806+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v112'273 uv273 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.807 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:42:44.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.806+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2372a10 con 0x5566e236de70 2024-09-16T01:42:44.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.807+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v112'274 uv274 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.807+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2372db0 con 0x5566e236de70 2024-09-16T01:42:44.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.809+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v112'275 uv275 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.809+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2371a00 con 0x5566e236de70 2024-09-16T01:42:44.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.810+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v112'276 uv276 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.810+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2371a00 con 0x5566e236de70 2024-09-16T01:42:44.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.811+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v112'277 uv270 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.811+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e112) -- 0x5566e2371a00 con 0x5566e236de70 2024-09-16T01:42:44.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.812+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v112'278 uv278 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.812+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2371a00 con 0x5566e236de70 2024-09-16T01:42:44.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.813+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v112'279 uv279 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.814 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:42:44.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.813+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2371a00 con 0x5566e236de70 2024-09-16T01:42:44.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.814+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v112'280 uv280 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.814+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2371a00 con 0x5566e236de70 2024-09-16T01:42:44.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.815+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v112'281 uv281 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.815+0000 7fc1fbd89f40 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5566e2372320 0x5566e2375160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:44.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.815+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2371a00 con 0x5566e2372320 2024-09-16T01:42:44.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.815+0000 7fc1f987f640 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5566e2372320 0x5566e2375160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:44.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.815+0000 7fc1f987f640 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5566e2372320 0x5566e2375160 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:44.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.817+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v112'122 uv121 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e2372320 2024-09-16T01:42:44.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.817+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e112) -- 0x5566e2371a00 con 0x5566e2372320 2024-09-16T01:42:44.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.818+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v112'123 uv123 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e2372320 2024-09-16T01:42:44.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.818+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e23756f0 con 0x5566e2372320 2024-09-16T01:42:44.820 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:42:44.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.819+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v112'124 uv124 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e2372320 2024-09-16T01:42:44.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.819+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e23756f0 con 0x5566e2372320 2024-09-16T01:42:44.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.820+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v112'125 uv125 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e2372320 2024-09-16T01:42:44.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.820+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e23756f0 con 0x5566e2372320 2024-09-16T01:42:44.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.821+0000 7fc1f987f640 1 -- 172.21.15.88:0/2387703978 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v112'126 uv126 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc1f4013ba0 con 0x5566e2372320 2024-09-16T01:42:44.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.821+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e23756f0 con 0x5566e22c1fa0 2024-09-16T01:42:44.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.823+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v112'147 uv146 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.823+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e112) -- 0x5566e23756f0 con 0x5566e22c1fa0 2024-09-16T01:42:44.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.823+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v112'148 uv148 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.823+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e23756f0 con 0x5566e22c1fa0 2024-09-16T01:42:44.826 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:42:44.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.824+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v112'149 uv149 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.824+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2376700 con 0x5566e22c1fa0 2024-09-16T01:42:44.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.825+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v112'150 uv150 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.825+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2376700 con 0x5566e22c1fa0 2024-09-16T01:42:44.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.826+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v112'151 uv151 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.826+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e23756f0 con 0x5566e22c1fa0 2024-09-16T01:42:44.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.827+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v112'152 uv152 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e22c1fa0 2024-09-16T01:42:44.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.827+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e23756f0 con 0x5566e236a890 2024-09-16T01:42:44.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.828+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v112'243 uv237 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e236a890 2024-09-16T01:42:44.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.828+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e112) -- 0x5566e23756f0 con 0x5566e236a890 2024-09-16T01:42:44.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.829+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v112'244 uv244 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e236a890 2024-09-16T01:42:44.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.829+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e23756f0 con 0x5566e236a890 2024-09-16T01:42:44.831 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:42:44.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.830+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v112'245 uv245 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e236a890 2024-09-16T01:42:44.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.830+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2371590 con 0x5566e236a890 2024-09-16T01:42:44.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.831+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v112'246 uv246 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e236a890 2024-09-16T01:42:44.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.831+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2371590 con 0x5566e236a890 2024-09-16T01:42:44.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.832+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v112'247 uv247 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e236a890 2024-09-16T01:42:44.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.832+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2371590 con 0x5566e236a890 2024-09-16T01:42:44.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.833+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v112'248 uv242 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e236a890 2024-09-16T01:42:44.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.833+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e112) -- 0x5566e2371590 con 0x5566e236a890 2024-09-16T01:42:44.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.834+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v112'249 uv249 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e236a890 2024-09-16T01:42:44.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.834+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2371570 con 0x5566e236a890 2024-09-16T01:42:44.836 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:42:44.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.835+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v112'250 uv250 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e236a890 2024-09-16T01:42:44.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.835+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2371570 con 0x5566e236a890 2024-09-16T01:42:44.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.836+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v112'251 uv251 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e236a890 2024-09-16T01:42:44.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.836+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378500 con 0x5566e236a890 2024-09-16T01:42:44.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.837+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v112'252 uv252 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e236a890 2024-09-16T01:42:44.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.837+0000 7fc1fbd89f40 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5566e2376220 0x5566e2378b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:44.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.837+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e2376220 2024-09-16T01:42:44.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.837+0000 7fc1f907e640 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5566e2376220 0x5566e2378b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:44.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.837+0000 7fc1f907e640 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5566e2376220 0x5566e2378b20 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:44.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.838+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v112'107 uv106 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e2376220 2024-09-16T01:42:44.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.838+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e2376220 2024-09-16T01:42:44.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.839+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v112'108 uv108 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e2376220 2024-09-16T01:42:44.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.839+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e2376220 2024-09-16T01:42:44.841 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:42:44.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.840+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v112'109 uv109 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e2376220 2024-09-16T01:42:44.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.840+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e2376220 2024-09-16T01:42:44.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.841+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v112'110 uv110 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e2376220 2024-09-16T01:42:44.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.841+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e236a890 2024-09-16T01:42:44.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.842+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v112'147 uv146 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e236a890 2024-09-16T01:42:44.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.842+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e236a890 2024-09-16T01:42:44.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.844+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v112'148 uv148 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e236a890 2024-09-16T01:42:44.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.844+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e236a890 2024-09-16T01:42:44.845 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:42:44.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.844+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v112'149 uv149 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e236a890 2024-09-16T01:42:44.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.845+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e236a890 2024-09-16T01:42:44.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.846+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v112'150 uv150 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e236a890 2024-09-16T01:42:44.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.846+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e236a890 2024-09-16T01:42:44.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.847+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v112'151 uv151 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e236a890 2024-09-16T01:42:44.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.847+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e236a890 2024-09-16T01:42:44.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.848+0000 7fc1f907e640 1 -- 172.21.15.88:0/2387703978 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v112'152 uv152 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1e40547a0 con 0x5566e236a890 2024-09-16T01:42:44.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.848+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e236de70 2024-09-16T01:42:44.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.849+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v112'125 uv124 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.849+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e236de70 2024-09-16T01:42:44.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.850+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v112'126 uv126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.850+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e236de70 2024-09-16T01:42:44.852 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:42:44.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.851+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v112'127 uv127 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.851+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e236de70 2024-09-16T01:42:44.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.852+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v112'128 uv128 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.852+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e236de70 2024-09-16T01:42:44.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.853+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v112'129 uv129 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.853+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e236de70 2024-09-16T01:42:44.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.855+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v112'302 uv294 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.855+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e236de70 2024-09-16T01:42:44.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.856+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v112'303 uv303 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.856+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e236de70 2024-09-16T01:42:44.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.857+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v112'304 uv304 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.858 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:42:44.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.857+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e236de70 2024-09-16T01:42:44.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.859+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v112'305 uv305 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.859+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e236de70 2024-09-16T01:42:44.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.860+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v112'306 uv306 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.860+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e236de70 2024-09-16T01:42:44.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.861+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v112'307 uv301 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.861+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e236de70 2024-09-16T01:42:44.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.862+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v112'308 uv308 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.862+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e236de70 2024-09-16T01:42:44.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.863+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v112'309 uv309 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.865 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:42:44.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.863+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e236de70 2024-09-16T01:42:44.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.865+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v112'310 uv310 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.865+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e236de70 2024-09-16T01:42:44.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.866+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v112'311 uv311 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.866+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2378610 con 0x5566e236de70 2024-09-16T01:42:44.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.867+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v112'312 uv312 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.867+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5566e2379060 con 0x5566e236de70 2024-09-16T01:42:44.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.868+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v112'313 uv313 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fc1ec02b030 con 0x5566e236de70 2024-09-16T01:42:44.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.868+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5566e22c1fa0 msgr2=0x5566e2358730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:44.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.868+0000 7fc1fbd89f40 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5566e22c1fa0 0x5566e2358730 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:44.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.868+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5566e236a890 msgr2=0x5566e236d2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:44.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.868+0000 7fc1fbd89f40 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5566e236a890 0x5566e236d2b0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:44.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.868+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5566e2376220 msgr2=0x5566e2378b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:44.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.868+0000 7fc1fbd89f40 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5566e2376220 0x5566e2378b20 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:44.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.868+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5566e23662d0 msgr2=0x5566e23686c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:44.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.868+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 reap_dead start 2024-09-16T01:42:44.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.868+0000 7fc1fbd89f40 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5566e23662d0 0x5566e23686c0 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:44.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.868+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5566e236de70 msgr2=0x5566e2370260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:44.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.868+0000 7fc1fbd89f40 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5566e236de70 0x5566e2370260 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:44.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.868+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5566e2372320 msgr2=0x5566e2375160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:44.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.868+0000 7fc1fbd89f40 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5566e2372320 0x5566e2375160 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:44.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.868+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc1dc06ac90 msgr2=0x7fc1dc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:44.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.868+0000 7fc1fbd89f40 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc1dc06ac90 0x7fc1dc06d150 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x5566e21d6320 tx=0x7fc1f4007040 comp rx=0 tx=0).stop 2024-09-16T01:42:44.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.868+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1e80030f0 msgr2=0x5566e22a5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:44.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.868+0000 7fc1fbd89f40 1 --2- 172.21.15.88:0/2387703978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1e80030f0 0x5566e22a5d00 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec012040 tx=0x7fc1ec001a40 comp rx=0 tx=0).stop 2024-09-16T01:42:44.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.868+0000 7fc1fa080640 1 -- 172.21.15.88:0/2387703978 reap_dead start 2024-09-16T01:42:44.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.868+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 shutdown_connections 2024-09-16T01:42:44.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.868+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 >> 172.21.15.88:0/2387703978 conn(0x5566e21e8be0 msgr2=0x5566e22c1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:44.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.868+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 shutdown_connections 2024-09-16T01:42:44.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.868+0000 7fc1fbd89f40 1 -- 172.21.15.88:0/2387703978 wait complete. 2024-09-16T01:42:44.876 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata truncate 200.00000001 0 2024-09-16T01:42:44.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.960+0000 7fcb0d721f40 1 Processor -- start 2024-09-16T01:42:45.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.960+0000 7fcb0d721f40 1 -- start start 2024-09-16T01:42:45.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.960+0000 7fcb0d721f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcaf80030f0 0x5619d9064410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:45.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.960+0000 7fcb0d721f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619d90df520 0x5619d9062c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:45.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.960+0000 7fcb0d721f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619d90fae80 0x5619d90631d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:45.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.960+0000 7fcb0d721f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5619d8f74510 con 0x5619d90fae80 2024-09-16T01:42:45.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.960+0000 7fcb0d721f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5619d8fcdc50 con 0x5619d90df520 2024-09-16T01:42:45.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.960+0000 7fcb0d721f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5619d90f76d0 con 0x7fcaf80030f0 2024-09-16T01:42:45.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.961+0000 7fcb0af9f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619d90fae80 0x5619d90631d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:45.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.961+0000 7fcb0af9f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619d90fae80 0x5619d90631d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36982/0 (socket says 172.21.15.88:36982) 2024-09-16T01:42:45.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.961+0000 7fcb0af9f640 1 -- 172.21.15.88:0/3501420836 learned_addr learned my addr 172.21.15.88:0/3501420836 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:45.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.961+0000 7fcb03fff640 1 --2- 172.21.15.88:0/3501420836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619d90df520 0x5619d9062c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:45.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.961+0000 7fcb0a79e640 1 --2- 172.21.15.88:0/3501420836 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcaf80030f0 0x5619d9064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:45.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.961+0000 7fcb0af9f640 1 -- 172.21.15.88:0/3501420836 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcaf80030f0 msgr2=0x5619d9064410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:45.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.961+0000 7fcb0af9f640 1 --2- 172.21.15.88:0/3501420836 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcaf80030f0 0x5619d9064410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.961+0000 7fcb0af9f640 1 -- 172.21.15.88:0/3501420836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619d90df520 msgr2=0x5619d9062c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:45.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.961+0000 7fcb0af9f640 1 --2- 172.21.15.88:0/3501420836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619d90df520 0x5619d9062c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.961+0000 7fcb0af9f640 1 -- 172.21.15.88:0/3501420836 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619d90f3aa0 con 0x5619d90fae80 2024-09-16T01:42:45.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.961+0000 7fcb03fff640 1 --2- 172.21.15.88:0/3501420836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619d90df520 0x5619d9062c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:45.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.961+0000 7fcb0af9f640 1 --2- 172.21.15.88:0/3501420836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619d90fae80 0x5619d90631d0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x5619d906ca10 tx=0x7fcb04004250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=148448991f639dec server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:45.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.961+0000 7fcb0979c640 1 -- 172.21.15.88:0/3501420836 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb04017070 con 0x5619d90fae80 2024-09-16T01:42:45.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.961+0000 7fcb0979c640 1 -- 172.21.15.88:0/3501420836 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb04006030 con 0x5619d90fae80 2024-09-16T01:42:45.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.961+0000 7fcb0979c640 1 -- 172.21.15.88:0/3501420836 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb04005440 con 0x5619d90fae80 2024-09-16T01:42:45.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.962+0000 7fcb0d721f40 1 -- 172.21.15.88:0/3501420836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619d90fae80 msgr2=0x5619d90631d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:45.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.962+0000 7fcb0d721f40 1 --2- 172.21.15.88:0/3501420836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619d90fae80 0x5619d90631d0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x5619d906ca10 tx=0x7fcb04004250 comp rx=0 tx=0).stop 2024-09-16T01:42:45.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.962+0000 7fcb0d721f40 1 -- 172.21.15.88:0/3501420836 shutdown_connections 2024-09-16T01:42:45.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.962+0000 7fcb0d721f40 1 --2- 172.21.15.88:0/3501420836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5619d90fae80 0x5619d90631d0 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.962+0000 7fcb0d721f40 1 --2- 172.21.15.88:0/3501420836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619d90df520 0x5619d9062c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.962+0000 7fcb0d721f40 1 --2- 172.21.15.88:0/3501420836 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcaf80030f0 0x5619d9064410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.962+0000 7fcb0d721f40 1 -- 172.21.15.88:0/3501420836 >> 172.21.15.88:0/3501420836 conn(0x5619d904d960 msgr2=0x5619d90df970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:45.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.962+0000 7fcb0d721f40 1 -- 172.21.15.88:0/3501420836 shutdown_connections 2024-09-16T01:42:45.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.962+0000 7fcb0d721f40 1 -- 172.21.15.88:0/3501420836 wait complete. 2024-09-16T01:42:45.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.962+0000 7fcb0d721f40 1 Processor -- start 2024-09-16T01:42:45.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.962+0000 7fcb0d721f40 1 -- start start 2024-09-16T01:42:45.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.963+0000 7fcb0d721f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5619d90df520 0x5619d90fa160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:45.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.963+0000 7fcb0d721f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619d90fae80 0x5619d90d9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:45.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.963+0000 7fcb0d721f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcaf80030f0 0x5619d90f5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:45.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.963+0000 7fcb0d721f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5619d90f3f70 con 0x7fcaf80030f0 2024-09-16T01:42:45.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.963+0000 7fcb0d721f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5619d90fa940 con 0x5619d90fae80 2024-09-16T01:42:45.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.963+0000 7fcb0d721f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5619d90e0460 con 0x5619d90df520 2024-09-16T01:42:45.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.963+0000 7fcb0af9f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcaf80030f0 0x5619d90f5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:45.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.963+0000 7fcb0af9f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcaf80030f0 0x5619d90f5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36990/0 (socket says 172.21.15.88:36990) 2024-09-16T01:42:45.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.963+0000 7fcb0af9f640 1 -- 172.21.15.88:0/3943848237 learned_addr learned my addr 172.21.15.88:0/3943848237 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:45.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.963+0000 7fcb03fff640 1 --2- 172.21.15.88:0/3943848237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619d90fae80 0x5619d90d9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:45.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.963+0000 7fcb0a79e640 1 --2- 172.21.15.88:0/3943848237 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5619d90df520 0x5619d90fa160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:45.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.963+0000 7fcb0af9f640 1 -- 172.21.15.88:0/3943848237 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5619d90df520 msgr2=0x5619d90fa160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:45.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.963+0000 7fcb0af9f640 1 --2- 172.21.15.88:0/3943848237 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5619d90df520 0x5619d90fa160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.963+0000 7fcb0af9f640 1 -- 172.21.15.88:0/3943848237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619d90fae80 msgr2=0x5619d90d9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:45.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.963+0000 7fcb0af9f640 1 --2- 172.21.15.88:0/3943848237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5619d90fae80 0x5619d90d9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.963+0000 7fcb0af9f640 1 -- 172.21.15.88:0/3943848237 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619d90f6ce0 con 0x7fcaf80030f0 2024-09-16T01:42:45.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.963+0000 7fcb0a79e640 1 --2- 172.21.15.88:0/3943848237 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5619d90df520 0x5619d90fa160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:45.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.963+0000 7fcb0af9f640 1 --2- 172.21.15.88:0/3943848237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcaf80030f0 0x5619d90f5f60 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x5619d9062040 tx=0x7fcb04001510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:45.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.964+0000 7fcb037fe640 1 -- 172.21.15.88:0/3943848237 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb04021020 con 0x7fcaf80030f0 2024-09-16T01:42:45.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.964+0000 7fcb037fe640 1 -- 172.21.15.88:0/3943848237 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb040058f0 con 0x7fcaf80030f0 2024-09-16T01:42:45.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.964+0000 7fcb0d721f40 1 -- 172.21.15.88:0/3943848237 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5619d90f9960 con 0x7fcaf80030f0 2024-09-16T01:42:45.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.964+0000 7fcb037fe640 1 -- 172.21.15.88:0/3943848237 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb04017040 con 0x7fcaf80030f0 2024-09-16T01:42:45.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.964+0000 7fcb0d721f40 1 -- 172.21.15.88:0/3943848237 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619d90da5c0 con 0x7fcaf80030f0 2024-09-16T01:42:45.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.965+0000 7fcb037fe640 1 -- 172.21.15.88:0/3943848237 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fcb04024020 con 0x7fcaf80030f0 2024-09-16T01:42:45.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.965+0000 7fcb037fe640 1 --2- 172.21.15.88:0/3943848237 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcaec06ac90 0x7fcaec06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:45.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.966+0000 7fcb037fe640 1 -- 172.21.15.88:0/3943848237 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 7888+0+0 (secure 0 0 0) 0x7fcb0408b2c0 con 0x7fcaf80030f0 2024-09-16T01:42:45.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.966+0000 7fcb0a79e640 1 --2- 172.21.15.88:0/3943848237 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcaec06ac90 0x7fcaec06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:45.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.966+0000 7fcb0d721f40 1 --2- 172.21.15.88:0/3943848237 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5619d9051640 0x5619d9066320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:45.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.966+0000 7fcb0d721f40 1 -- 172.21.15.88:0/3943848237 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:1 12.14 12:2e2fa760:::200.00000001:head [truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112) -- 0x5619d9066860 con 0x5619d9051640 2024-09-16T01:42:45.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.966+0000 7fcb0a79e640 1 --2- 172.21.15.88:0/3943848237 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcaec06ac90 0x7fcaec06d150 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fcafc001050 tx=0x7fcafc000fe0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:45.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.966+0000 7fcb03fff640 1 --2- 172.21.15.88:0/3943848237 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5619d9051640 0x5619d9066320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:45.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.966+0000 7fcb03fff640 1 --2- 172.21.15.88:0/3943848237 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5619d9051640 0x5619d9066320 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:45.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.968+0000 7fcb03fff640 1 -- 172.21.15.88:0/3943848237 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(1 200.00000001 [truncate 0] v112'571 uv571 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fcaf401a320 con 0x5619d9051640 2024-09-16T01:42:45.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.968+0000 7fcb0d721f40 1 -- 172.21.15.88:0/3943848237 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5619d9051640 msgr2=0x5619d9066320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:45.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.968+0000 7fcb0d721f40 1 --2- 172.21.15.88:0/3943848237 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5619d9051640 0x5619d9066320 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.968+0000 7fcb0d721f40 1 -- 172.21.15.88:0/3943848237 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcaec06ac90 msgr2=0x7fcaec06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:45.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.968+0000 7fcb0d721f40 1 --2- 172.21.15.88:0/3943848237 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcaec06ac90 0x7fcaec06d150 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fcafc001050 tx=0x7fcafc000fe0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.968+0000 7fcb0d721f40 1 -- 172.21.15.88:0/3943848237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcaf80030f0 msgr2=0x5619d90f5f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:45.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.968+0000 7fcb0d721f40 1 --2- 172.21.15.88:0/3943848237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcaf80030f0 0x5619d90f5f60 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x5619d9062040 tx=0x7fcb04001510 comp rx=0 tx=0).stop 2024-09-16T01:42:45.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.968+0000 7fcb0af9f640 1 -- 172.21.15.88:0/3943848237 reap_dead start 2024-09-16T01:42:45.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.968+0000 7fcb0d721f40 1 -- 172.21.15.88:0/3943848237 shutdown_connections 2024-09-16T01:42:45.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.968+0000 7fcb0d721f40 1 -- 172.21.15.88:0/3943848237 >> 172.21.15.88:0/3943848237 conn(0x5619d904d960 msgr2=0x5619d9064410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:45.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.968+0000 7fcb0d721f40 1 -- 172.21.15.88:0/3943848237 shutdown_connections 2024-09-16T01:42:45.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:44.968+0000 7fcb0d721f40 1 -- 172.21.15.88:0/3943848237 wait complete. 2024-09-16T01:42:45.055 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:42:45.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.156+0000 7f7d0b61e640 1 Processor -- start 2024-09-16T01:42:45.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.156+0000 7f7d0b61e640 1 -- start start 2024-09-16T01:42:45.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.157+0000 7f7d0b61e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7d040b6290 0x7f7d04071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:45.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.157+0000 7f7d0b61e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7d04072530 0x7f7d0415bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:45.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.157+0000 7f7d0b61e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d040ccff0 con 0x7f7d0415b480 2024-09-16T01:42:45.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.157+0000 7f7d0b61e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7d040cd2d0 con 0x7f7d040b6290 2024-09-16T01:42:45.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.157+0000 7f7d0b61e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7d040cd5b0 con 0x7f7d04072530 2024-09-16T01:42:45.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.157+0000 7f7d08bb9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7d040b6290 0x7f7d04071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:45.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.157+0000 7f7d093ba640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7d0415b480 0x7f7d040b5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49284/0 (socket says 172.21.15.88:49284) 2024-09-16T01:42:45.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.157+0000 7f7d093ba640 1 -- 172.21.15.88:0/512621650 learned_addr learned my addr 172.21.15.88:0/512621650 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:45.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.157+0000 7f7d09bbb640 1 --2- 172.21.15.88:0/512621650 >> v2:172.21.15.154:3300/0 conn(0x7f7d04072530 0x7f7d0415bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:45.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.158+0000 7f7d08bb9640 1 -- 172.21.15.88:0/512621650 >> v2:172.21.15.154:3300/0 conn(0x7f7d04072530 msgr2=0x7f7d0415bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:45.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.158+0000 7f7d08bb9640 1 --2- 172.21.15.88:0/512621650 >> v2:172.21.15.154:3300/0 conn(0x7f7d04072530 0x7f7d0415bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.158+0000 7f7d08bb9640 1 -- 172.21.15.88:0/512621650 >> v1:172.21.15.88:6789/0 conn(0x7f7d0415b480 legacy=0x7f7d040b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:45.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.158+0000 7f7d03fff640 1 -- 172.21.15.88:0/512621650 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4090230933 0 0) 0x7f7d040ccff0 con 0x7f7d0415b480 2024-09-16T01:42:45.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.158+0000 7f7d08bb9640 1 -- 172.21.15.88:0/512621650 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d040cd890 con 0x7f7d040b6290 2024-09-16T01:42:45.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.158+0000 7f7d09bbb640 1 --2- 172.21.15.88:0/512621650 >> v2:172.21.15.154:3300/0 conn(0x7f7d04072530 0x7f7d0415bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:45.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.158+0000 7f7d08bb9640 1 --2- 172.21.15.88:0/512621650 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d040b6290 0x7f7d04071ff0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f7d040733f0 tx=0x7f7cf802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6bd2995fef9862d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:45.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.158+0000 7f7d03fff640 1 -- 172.21.15.88:0/512621650 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf8037070 con 0x7f7d040b6290 2024-09-16T01:42:45.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.158+0000 7f7d03fff640 1 -- 172.21.15.88:0/512621650 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cf8032440 con 0x7f7d040b6290 2024-09-16T01:42:45.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.158+0000 7f7d03fff640 1 -- 172.21.15.88:0/512621650 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf802ae60 con 0x7f7d040b6290 2024-09-16T01:42:45.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.159+0000 7f7d0b61e640 1 -- 172.21.15.88:0/512621650 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d040b6290 msgr2=0x7f7d04071ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:45.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.159+0000 7f7d0b61e640 1 --2- 172.21.15.88:0/512621650 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d040b6290 0x7f7d04071ff0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f7d040733f0 tx=0x7f7cf802a110 comp rx=0 tx=0).stop 2024-09-16T01:42:45.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.159+0000 7f7d0b61e640 1 -- 172.21.15.88:0/512621650 shutdown_connections 2024-09-16T01:42:45.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.159+0000 7f7d0b61e640 1 --2- 172.21.15.88:0/512621650 >> v2:172.21.15.154:3300/0 conn(0x7f7d04072530 0x7f7d0415bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.159+0000 7f7d0b61e640 1 --2- 172.21.15.88:0/512621650 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d040b6290 0x7f7d04071ff0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.159+0000 7f7d0b61e640 1 -- 172.21.15.88:0/512621650 >> 172.21.15.88:0/512621650 conn(0x7f7d0406d0f0 msgr2=0x7f7d0406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:45.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.159+0000 7f7d0b61e640 1 -- 172.21.15.88:0/512621650 shutdown_connections 2024-09-16T01:42:45.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.159+0000 7f7d0b61e640 1 -- 172.21.15.88:0/512621650 wait complete. 2024-09-16T01:42:45.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.160+0000 7f7d0b61e640 1 Processor -- start 2024-09-16T01:42:45.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.160+0000 7f7d0b61e640 1 -- start start 2024-09-16T01:42:45.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.161+0000 7f7d0b61e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d04072530 0x7f7d04150890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:45.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.161+0000 7f7d0b61e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d0415b480 0x7f7d04147240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:45.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.161+0000 7f7d0b61e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d04151480 0x7f7d040d04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:45.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.161+0000 7f7d0b61e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0415b9e0 con 0x7f7d0415b480 2024-09-16T01:42:45.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.161+0000 7f7d0b61e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0415b860 con 0x7f7d04151480 2024-09-16T01:42:45.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.161+0000 7f7d0b61e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7d0415c1a0 con 0x7f7d04072530 2024-09-16T01:42:45.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.161+0000 7f7d08bb9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d0415b480 0x7f7d04147240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:45.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.161+0000 7f7d08bb9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d0415b480 0x7f7d04147240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37006/0 (socket says 172.21.15.88:37006) 2024-09-16T01:42:45.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.161+0000 7f7d08bb9640 1 -- 172.21.15.88:0/2968238656 learned_addr learned my addr 172.21.15.88:0/2968238656 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:45.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.161+0000 7f7d08bb9640 1 -- 172.21.15.88:0/2968238656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d04072530 msgr2=0x7f7d04150890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:45.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.161+0000 7f7d08bb9640 1 --2- 172.21.15.88:0/2968238656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d04072530 0x7f7d04150890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.161+0000 7f7d08bb9640 1 -- 172.21.15.88:0/2968238656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d04151480 msgr2=0x7f7d040d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:45.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.161+0000 7f7d08bb9640 1 --2- 172.21.15.88:0/2968238656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d04151480 0x7f7d040d04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.161+0000 7f7d08bb9640 1 -- 172.21.15.88:0/2968238656 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d04147780 con 0x7f7d0415b480 2024-09-16T01:42:45.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.161+0000 7f7d08bb9640 1 --2- 172.21.15.88:0/2968238656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d0415b480 0x7f7d04147240 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8009890 tx=0x7f7cf80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:45.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.162+0000 7f7d01ffb640 1 -- 172.21.15.88:0/2968238656 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf8037050 con 0x7f7d0415b480 2024-09-16T01:42:45.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.162+0000 7f7d01ffb640 1 -- 172.21.15.88:0/2968238656 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cf8038040 con 0x7f7d0415b480 2024-09-16T01:42:45.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.162+0000 7f7d0b61e640 1 -- 172.21.15.88:0/2968238656 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d04151f40 con 0x7f7d0415b480 2024-09-16T01:42:45.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.162+0000 7f7d01ffb640 1 -- 172.21.15.88:0/2968238656 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf8004450 con 0x7f7d0415b480 2024-09-16T01:42:45.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.162+0000 7f7d0b61e640 1 -- 172.21.15.88:0/2968238656 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d04152500 con 0x7f7d0415b480 2024-09-16T01:42:45.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.163+0000 7f7d0b61e640 1 -- 172.21.15.88:0/2968238656 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d0415bc60 con 0x7f7d0415b480 2024-09-16T01:42:45.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.163+0000 7f7d01ffb640 1 -- 172.21.15.88:0/2968238656 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7cf8030980 con 0x7f7d0415b480 2024-09-16T01:42:45.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.164+0000 7f7d01ffb640 1 --2- 172.21.15.88:0/2968238656 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7cf006c2e0 0x7f7cf006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:45.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.164+0000 7f7d01ffb640 1 -- 172.21.15.88:0/2968238656 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 7888+0+0 (secure 0 0 0) 0x7f7cf80ac7b0 con 0x7f7d0415b480 2024-09-16T01:42:45.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.164+0000 7f7d093ba640 1 --2- 172.21.15.88:0/2968238656 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7cf006c2e0 0x7f7cf006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:45.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.164+0000 7f7d093ba640 1 --2- 172.21.15.88:0/2968238656 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7cf006c2e0 0x7f7cf006e7a0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4007fd0 tx=0x7f7cf4007ce0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:45.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.168+0000 7f7d01ffb640 1 -- 172.21.15.88:0/2968238656 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7cf807a850 con 0x7f7d0415b480 2024-09-16T01:42:45.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.370+0000 7f7d0b61e640 1 -- 172.21.15.88:0/2968238656 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f7d040b4eb0 con 0x7f7d0415b480 2024-09-16T01:42:45.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.655+0000 7f7d01ffb640 1 -- 172.21.15.88:0/2968238656 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v220) ==== 162+0+0 (secure 0 0 0) 0x7f7cf807a150 con 0x7f7d0415b480 2024-09-16T01:42:45.657 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:42:45.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.657+0000 7f7d0b61e640 1 -- 172.21.15.88:0/2968238656 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7cf006c2e0 msgr2=0x7f7cf006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:45.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.657+0000 7f7d0b61e640 1 --2- 172.21.15.88:0/2968238656 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7cf006c2e0 0x7f7cf006e7a0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4007fd0 tx=0x7f7cf4007ce0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.657+0000 7f7d0b61e640 1 -- 172.21.15.88:0/2968238656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d0415b480 msgr2=0x7f7d04147240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:45.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.657+0000 7f7d0b61e640 1 --2- 172.21.15.88:0/2968238656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d0415b480 0x7f7d04147240 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8009890 tx=0x7f7cf80046c0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.657+0000 7f7d0b61e640 1 -- 172.21.15.88:0/2968238656 shutdown_connections 2024-09-16T01:42:45.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.657+0000 7f7d0b61e640 1 --2- 172.21.15.88:0/2968238656 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7cf006c2e0 0x7f7cf006e7a0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.657+0000 7f7d0b61e640 1 --2- 172.21.15.88:0/2968238656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d04151480 0x7f7d040d04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.657+0000 7f7d0b61e640 1 --2- 172.21.15.88:0/2968238656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7d0415b480 0x7f7d04147240 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.657+0000 7f7d0b61e640 1 --2- 172.21.15.88:0/2968238656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7d04072530 0x7f7d04150890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.657+0000 7f7d0b61e640 1 -- 172.21.15.88:0/2968238656 >> 172.21.15.88:0/2968238656 conn(0x7f7d0406d0f0 msgr2=0x7f7d0415c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:45.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.658+0000 7f7d0b61e640 1 -- 172.21.15.88:0/2968238656 shutdown_connections 2024-09-16T01:42:45.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.658+0000 7f7d0b61e640 1 -- 172.21.15.88:0/2968238656 wait complete. 2024-09-16T01:42:45.672 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:45.673 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:42:45.672+0000 7f24e0bcc640 -1 log_channel(cluster) log [ERR] : missing journal object 2024-09-16T01:42:45.674 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:42:45.673+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:42:45.674 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:42:45.673+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:42:45.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.772+0000 7f573abc4640 1 Processor -- start 2024-09-16T01:42:45.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.772+0000 7f573abc4640 1 -- start start 2024-09-16T01:42:45.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.773+0000 7f573abc4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f57340b5be0 0x7f573415ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:45.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.773+0000 7f573abc4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f57340b29a0 0x7f57340cde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:45.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.773+0000 7f573abc4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57340b6830 con 0x7f573415a9c0 2024-09-16T01:42:45.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.773+0000 7f573abc4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f573415f410 con 0x7f57340b29a0 2024-09-16T01:42:45.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.773+0000 7f573abc4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f57340cd0b0 con 0x7f57340b5be0 2024-09-16T01:42:45.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.773+0000 7f5738960640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f573415a9c0 0x7f573415e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49290/0 (socket says 172.21.15.88:49290) 2024-09-16T01:42:45.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.773+0000 7f5738960640 1 -- 172.21.15.88:0/3667845412 learned_addr learned my addr 172.21.15.88:0/3667845412 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:45.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.773+0000 7f5739161640 1 --2- 172.21.15.88:0/3667845412 >> v2:172.21.15.88:3300/0 conn(0x7f57340b29a0 0x7f57340cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:45.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.773+0000 7f57337fe640 1 -- 172.21.15.88:0/3667845412 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4225197274 0 0) 0x7f57340b6830 con 0x7f573415a9c0 2024-09-16T01:42:45.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.774+0000 7f57337fe640 1 -- 172.21.15.88:0/3667845412 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f571c003440 con 0x7f573415a9c0 2024-09-16T01:42:45.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.774+0000 7f5733fff640 1 --2- 172.21.15.88:0/3667845412 >> v2:172.21.15.154:3300/0 conn(0x7f57340b5be0 0x7f573415ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:45.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.774+0000 7f5739161640 1 -- 172.21.15.88:0/3667845412 >> v2:172.21.15.154:3300/0 conn(0x7f57340b5be0 msgr2=0x7f573415ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:45.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.774+0000 7f5739161640 1 --2- 172.21.15.88:0/3667845412 >> v2:172.21.15.154:3300/0 conn(0x7f57340b5be0 0x7f573415ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.774+0000 7f5739161640 1 -- 172.21.15.88:0/3667845412 >> v1:172.21.15.88:6789/0 conn(0x7f573415a9c0 legacy=0x7f573415e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:45.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.774+0000 7f5739161640 1 -- 172.21.15.88:0/3667845412 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57340cd390 con 0x7f57340b29a0 2024-09-16T01:42:45.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.774+0000 7f5739161640 1 --2- 172.21.15.88:0/3667845412 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57340b29a0 0x7f57340cde70 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f57340ccd90 tx=0x7f572c02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf31a6050c882e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:45.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.774+0000 7f57337fe640 1 -- 172.21.15.88:0/3667845412 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572c007740 con 0x7f57340b29a0 2024-09-16T01:42:45.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.774+0000 7f57337fe640 1 -- 172.21.15.88:0/3667845412 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f572c0078e0 con 0x7f57340b29a0 2024-09-16T01:42:45.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.774+0000 7f57337fe640 1 -- 172.21.15.88:0/3667845412 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572c007cb0 con 0x7f57340b29a0 2024-09-16T01:42:45.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.775+0000 7f573abc4640 1 -- 172.21.15.88:0/3667845412 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57340b29a0 msgr2=0x7f57340cde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:45.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.775+0000 7f573abc4640 1 --2- 172.21.15.88:0/3667845412 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57340b29a0 0x7f57340cde70 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f57340ccd90 tx=0x7f572c02a680 comp rx=0 tx=0).stop 2024-09-16T01:42:45.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.775+0000 7f573abc4640 1 -- 172.21.15.88:0/3667845412 shutdown_connections 2024-09-16T01:42:45.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.775+0000 7f573abc4640 1 --2- 172.21.15.88:0/3667845412 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57340b29a0 0x7f57340cde70 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.775+0000 7f573abc4640 1 --2- 172.21.15.88:0/3667845412 >> v2:172.21.15.154:3300/0 conn(0x7f57340b5be0 0x7f573415ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.775+0000 7f573abc4640 1 -- 172.21.15.88:0/3667845412 >> 172.21.15.88:0/3667845412 conn(0x7f573406bc10 msgr2=0x7f573406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:45.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.775+0000 7f573abc4640 1 -- 172.21.15.88:0/3667845412 shutdown_connections 2024-09-16T01:42:45.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.775+0000 7f573abc4640 1 -- 172.21.15.88:0/3667845412 wait complete. 2024-09-16T01:42:45.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.776+0000 7f573abc4640 1 Processor -- start 2024-09-16T01:42:45.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.776+0000 7f573abc4640 1 -- start start 2024-09-16T01:42:45.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.777+0000 7f573abc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57340b29a0 0x7f57341491f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:45.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.777+0000 7f573abc4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57340b5be0 0x7f5734155440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:45.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.777+0000 7f573abc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f573415a9c0 0x7f5734151460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:45.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.777+0000 7f573abc4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f573415ada0 con 0x7f57340b5be0 2024-09-16T01:42:45.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.777+0000 7f573abc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f57340cd950 con 0x7f573415a9c0 2024-09-16T01:42:45.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.777+0000 7f573abc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f573406de40 con 0x7f57340b29a0 2024-09-16T01:42:45.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.777+0000 7f5733fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57340b5be0 0x7f5734155440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:45.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.777+0000 7f5733fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57340b5be0 0x7f5734155440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37026/0 (socket says 172.21.15.88:37026) 2024-09-16T01:42:45.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.777+0000 7f5733fff640 1 -- 172.21.15.88:0/3319365520 learned_addr learned my addr 172.21.15.88:0/3319365520 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:45.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.777+0000 7f5733fff640 1 -- 172.21.15.88:0/3319365520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57340b29a0 msgr2=0x7f57341491f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:45.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.777+0000 7f5733fff640 1 --2- 172.21.15.88:0/3319365520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57340b29a0 0x7f57341491f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.777+0000 7f5733fff640 1 -- 172.21.15.88:0/3319365520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f573415a9c0 msgr2=0x7f5734151460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:45.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.777+0000 7f5733fff640 1 --2- 172.21.15.88:0/3319365520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f573415a9c0 0x7f5734151460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.777+0000 7f5733fff640 1 -- 172.21.15.88:0/3319365520 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5734155c90 con 0x7f57340b5be0 2024-09-16T01:42:45.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.778+0000 7f5733fff640 1 --2- 172.21.15.88:0/3319365520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57340b5be0 0x7f5734155440 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f572800c820 tx=0x7f572800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:45.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.778+0000 7f57317fa640 1 -- 172.21.15.88:0/3319365520 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5728015070 con 0x7f57340b5be0 2024-09-16T01:42:45.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.778+0000 7f57317fa640 1 -- 172.21.15.88:0/3319365520 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5728004220 con 0x7f57340b5be0 2024-09-16T01:42:45.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.778+0000 7f573abc4640 1 -- 172.21.15.88:0/3319365520 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5734155f70 con 0x7f57340b5be0 2024-09-16T01:42:45.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.778+0000 7f57317fa640 1 -- 172.21.15.88:0/3319365520 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572800ce10 con 0x7f57340b5be0 2024-09-16T01:42:45.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.778+0000 7f573abc4640 1 -- 172.21.15.88:0/3319365520 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5734156530 con 0x7f57340b5be0 2024-09-16T01:42:45.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.779+0000 7f573abc4640 1 -- 172.21.15.88:0/3319365520 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57340c5a90 con 0x7f57340b5be0 2024-09-16T01:42:45.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.779+0000 7f57317fa640 1 -- 172.21.15.88:0/3319365520 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5728007830 con 0x7f57340b5be0 2024-09-16T01:42:45.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.780+0000 7f57317fa640 1 --2- 172.21.15.88:0/3319365520 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f572006c2b0 0x7f572006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:45.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.780+0000 7f5738960640 1 --2- 172.21.15.88:0/3319365520 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f572006c2b0 0x7f572006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:45.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.780+0000 7f57317fa640 1 -- 172.21.15.88:0/3319365520 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 7974+0+0 (secure 0 0 0) 0x7f572808bb30 con 0x7f57340b5be0 2024-09-16T01:42:45.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.780+0000 7f5738960640 1 --2- 172.21.15.88:0/3319365520 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f572006c2b0 0x7f572006e770 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f5724000f60 tx=0x7f57240078f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:45.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.784+0000 7f57317fa640 1 -- 172.21.15.88:0/3319365520 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f57340c5a90 con 0x7f57340b5be0 2024-09-16T01:42:45.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.984+0000 7f573abc4640 1 -- 172.21.15.88:0/3319365520 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f573406e1d0 con 0x7f57340b5be0 2024-09-16T01:42:45.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.984+0000 7f57317fa640 1 -- 172.21.15.88:0/3319365520 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) ==== 95+0+3566 (secure 0 0 0) 0x7f5728059a50 con 0x7f57340b5be0 2024-09-16T01:42:45.986 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:45.986 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":222,"btime":"2024-09-16T01:42:45:674170+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:45.674119+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:45.986 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 222 2024-09-16T01:42:45.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.986+0000 7f573abc4640 1 -- 172.21.15.88:0/3319365520 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f572006c2b0 msgr2=0x7f572006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:45.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.986+0000 7f573abc4640 1 --2- 172.21.15.88:0/3319365520 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f572006c2b0 0x7f572006e770 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f5724000f60 tx=0x7f57240078f0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.986+0000 7f573abc4640 1 -- 172.21.15.88:0/3319365520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57340b5be0 msgr2=0x7f5734155440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:45.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.986+0000 7f573abc4640 1 --2- 172.21.15.88:0/3319365520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57340b5be0 0x7f5734155440 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f572800c820 tx=0x7f572800cce0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.986+0000 7f573abc4640 1 -- 172.21.15.88:0/3319365520 shutdown_connections 2024-09-16T01:42:45.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.986+0000 7f573abc4640 1 --2- 172.21.15.88:0/3319365520 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f572006c2b0 0x7f572006e770 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.986+0000 7f573abc4640 1 --2- 172.21.15.88:0/3319365520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f573415a9c0 0x7f5734151460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.986+0000 7f573abc4640 1 --2- 172.21.15.88:0/3319365520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57340b5be0 0x7f5734155440 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.986+0000 7f573abc4640 1 --2- 172.21.15.88:0/3319365520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f57340b29a0 0x7f57341491f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:45.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.986+0000 7f573abc4640 1 -- 172.21.15.88:0/3319365520 >> 172.21.15.88:0/3319365520 conn(0x7f573406bc10 msgr2=0x7f573406bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:45.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.987+0000 7f573abc4640 1 -- 172.21.15.88:0/3319365520 shutdown_connections 2024-09-16T01:42:45.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:45.987+0000 7f573abc4640 1 -- 172.21.15.88:0/3319365520 wait complete. 2024-09-16T01:42:45.999 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:42:45.999 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:46.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.095+0000 7fb52e2dd640 1 Processor -- start 2024-09-16T01:42:46.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.096+0000 7fb52e2dd640 1 -- start start 2024-09-16T01:42:46.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.096+0000 7fb52e2dd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb52815c170 0x7fb52815c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:46.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.096+0000 7fb52e2dd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5280b64e0 0x7fb5280cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:46.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.096+0000 7fb52e2dd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5280b4cf0 con 0x7fb5280cf6c0 2024-09-16T01:42:46.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.096+0000 7fb52e2dd640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb52815ca90 con 0x7fb52815c170 2024-09-16T01:42:46.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.096+0000 7fb52e2dd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb5280b49c0 con 0x7fb5280b64e0 2024-09-16T01:42:46.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.097+0000 7fb5277fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb5280cf6c0 0x7fb5280b5dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49304/0 (socket says 172.21.15.88:49304) 2024-09-16T01:42:46.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.097+0000 7fb5277fe640 1 -- 172.21.15.88:0/1362775005 learned_addr learned my addr 172.21.15.88:0/1362775005 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:46.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.097+0000 7fb52c87a640 1 --2- 172.21.15.88:0/1362775005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5280b64e0 0x7fb5280cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:46.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.097+0000 7fb527fff640 1 --2- 172.21.15.88:0/1362775005 >> v2:172.21.15.154:3300/0 conn(0x7fb52815c170 0x7fb52815c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:46.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.097+0000 7fb526ffd640 1 -- 172.21.15.88:0/1362775005 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1427216720 0 0) 0x7fb5280b4cf0 con 0x7fb5280cf6c0 2024-09-16T01:42:46.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.097+0000 7fb526ffd640 1 -- 172.21.15.88:0/1362775005 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb510003520 con 0x7fb5280cf6c0 2024-09-16T01:42:46.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.097+0000 7fb526ffd640 1 -- 172.21.15.88:0/1362775005 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2874762892 0 0) 0x7fb510003520 con 0x7fb5280cf6c0 2024-09-16T01:42:46.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.097+0000 7fb526ffd640 1 -- 172.21.15.88:0/1362775005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5280b64e0 msgr2=0x7fb5280cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:46.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.097+0000 7fb526ffd640 1 --2- 172.21.15.88:0/1362775005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5280b64e0 0x7fb5280cfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.097+0000 7fb526ffd640 1 -- 172.21.15.88:0/1362775005 >> v2:172.21.15.154:3300/0 conn(0x7fb52815c170 msgr2=0x7fb52815c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:46.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.097+0000 7fb526ffd640 1 --2- 172.21.15.88:0/1362775005 >> v2:172.21.15.154:3300/0 conn(0x7fb52815c170 0x7fb52815c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.097+0000 7fb526ffd640 1 -- 172.21.15.88:0/1362775005 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5280b4fd0 con 0x7fb5280cf6c0 2024-09-16T01:42:46.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.097+0000 7fb526ffd640 1 -- 172.21.15.88:0/1362775005 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fb51c001670 con 0x7fb5280cf6c0 2024-09-16T01:42:46.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.097+0000 7fb526ffd640 1 -- 172.21.15.88:0/1362775005 >> v1:172.21.15.88:6789/0 conn(0x7fb5280cf6c0 legacy=0x7fb5280b5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:46.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.097+0000 7fb526ffd640 1 --2- 172.21.15.88:0/1362775005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5280b64e0 0x7fb5100039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:46.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.097+0000 7fb526ffd640 1 --2- 172.21.15.88:0/1362775005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb510003f30 0x7fb510004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:46.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.097+0000 7fb526ffd640 1 --2- 172.21.15.88:0/1362775005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb510004d40 0x7fb5100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:46.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.097+0000 7fb526ffd640 1 -- 172.21.15.88:0/1362775005 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb52815ca90 con 0x7fb510004d40 2024-09-16T01:42:46.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.097+0000 7fb526ffd640 1 -- 172.21.15.88:0/1362775005 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb5280b49c0 con 0x7fb510003f30 2024-09-16T01:42:46.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.097+0000 7fb526ffd640 1 -- 172.21.15.88:0/1362775005 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb51000b9c0 con 0x7fb5280b64e0 2024-09-16T01:42:46.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.097+0000 7fb52c87a640 1 --2- 172.21.15.88:0/1362775005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb510004d40 0x7fb5100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:46.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.097+0000 7fb527fff640 1 --2- 172.21.15.88:0/1362775005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5280b64e0 0x7fb5100039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:46.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.097+0000 7fb5277fe640 1 --2- 172.21.15.88:0/1362775005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb510003f30 0x7fb510004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:46.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.098+0000 7fb52c87a640 1 -- 172.21.15.88:0/1362775005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5280b64e0 msgr2=0x7fb5100039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:46.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.098+0000 7fb52c87a640 1 --2- 172.21.15.88:0/1362775005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5280b64e0 0x7fb5100039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.098+0000 7fb52c87a640 1 -- 172.21.15.88:0/1362775005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb510003f30 msgr2=0x7fb510004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:46.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.098+0000 7fb52c87a640 1 --2- 172.21.15.88:0/1362775005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb510003f30 0x7fb510004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.098+0000 7fb52c87a640 1 -- 172.21.15.88:0/1362775005 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb510003520 con 0x7fb510004d40 2024-09-16T01:42:46.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.098+0000 7fb52c87a640 1 --2- 172.21.15.88:0/1362775005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb510004d40 0x7fb5100051c0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7fb520009fc0 tx=0x7fb510005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3b3787c288afcf3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:46.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.098+0000 7fb526ffd640 1 -- 172.21.15.88:0/1362775005 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb520009c10 con 0x7fb510004d40 2024-09-16T01:42:46.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.098+0000 7fb526ffd640 1 -- 172.21.15.88:0/1362775005 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb520036070 con 0x7fb510004d40 2024-09-16T01:42:46.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.098+0000 7fb526ffd640 1 -- 172.21.15.88:0/1362775005 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb520031670 con 0x7fb510004d40 2024-09-16T01:42:46.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.099+0000 7fb52e2dd640 1 -- 172.21.15.88:0/1362775005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb510004d40 msgr2=0x7fb5100051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:46.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.099+0000 7fb52e2dd640 1 --2- 172.21.15.88:0/1362775005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb510004d40 0x7fb5100051c0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7fb520009fc0 tx=0x7fb510005830 comp rx=0 tx=0).stop 2024-09-16T01:42:46.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.099+0000 7fb52c87a640 1 -- 172.21.15.88:0/1362775005 reap_dead start 2024-09-16T01:42:46.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.099+0000 7fb52e2dd640 1 -- 172.21.15.88:0/1362775005 shutdown_connections 2024-09-16T01:42:46.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.099+0000 7fb52e2dd640 1 -- 172.21.15.88:0/1362775005 >> 172.21.15.88:0/1362775005 conn(0x7fb52806bc10 msgr2=0x7fb52806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:46.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.099+0000 7fb52e2dd640 1 -- 172.21.15.88:0/1362775005 shutdown_connections 2024-09-16T01:42:46.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.099+0000 7fb52e2dd640 1 -- 172.21.15.88:0/1362775005 wait complete. 2024-09-16T01:42:46.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.100+0000 7fb52e2dd640 1 Processor -- start 2024-09-16T01:42:46.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.100+0000 7fb52e2dd640 1 -- start start 2024-09-16T01:42:46.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.101+0000 7fb52e2dd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb510004d40 0x7fb528147040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:46.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.101+0000 7fb52e2dd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb52815cff0 0x7fb52815d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:46.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.101+0000 7fb52e2dd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb52815a3e0 0x7fb52815a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:46.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.101+0000 7fb52e2dd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb5280b5870 con 0x7fb52815a3e0 2024-09-16T01:42:46.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.101+0000 7fb52e2dd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb5280b5590 con 0x7fb52815cff0 2024-09-16T01:42:46.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.101+0000 7fb52e2dd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb5280b5b50 con 0x7fb510004d40 2024-09-16T01:42:46.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.101+0000 7fb52c87a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb52815a3e0 0x7fb52815a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:46.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.101+0000 7fb52c87a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb52815a3e0 0x7fb52815a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37048/0 (socket says 172.21.15.88:37048) 2024-09-16T01:42:46.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.101+0000 7fb52c87a640 1 -- 172.21.15.88:0/378622412 learned_addr learned my addr 172.21.15.88:0/378622412 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:46.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.101+0000 7fb52c87a640 1 -- 172.21.15.88:0/378622412 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb510004d40 msgr2=0x7fb528147040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:46.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.101+0000 7fb52c87a640 1 --2- 172.21.15.88:0/378622412 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb510004d40 0x7fb528147040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.101+0000 7fb52c87a640 1 -- 172.21.15.88:0/378622412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb52815cff0 msgr2=0x7fb52815d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:46.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.101+0000 7fb52c87a640 1 --2- 172.21.15.88:0/378622412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb52815cff0 0x7fb52815d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.101+0000 7fb52c87a640 1 -- 172.21.15.88:0/378622412 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb52815da70 con 0x7fb52815a3e0 2024-09-16T01:42:46.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.101+0000 7fb52c87a640 1 --2- 172.21.15.88:0/378622412 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb52815a3e0 0x7fb52815a860 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fb52002aed0 tx=0x7fb520009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:46.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.102+0000 7fb524ff9640 1 -- 172.21.15.88:0/378622412 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb520009c10 con 0x7fb52815a3e0 2024-09-16T01:42:46.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.102+0000 7fb524ff9640 1 -- 172.21.15.88:0/378622412 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb52003f030 con 0x7fb52815a3e0 2024-09-16T01:42:46.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.102+0000 7fb524ff9640 1 -- 172.21.15.88:0/378622412 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb520003bb0 con 0x7fb52815a3e0 2024-09-16T01:42:46.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.102+0000 7fb52e2dd640 1 -- 172.21.15.88:0/378622412 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb52815dd50 con 0x7fb52815a3e0 2024-09-16T01:42:46.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.102+0000 7fb52e2dd640 1 -- 172.21.15.88:0/378622412 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb52815e310 con 0x7fb52815a3e0 2024-09-16T01:42:46.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.103+0000 7fb52e2dd640 1 -- 172.21.15.88:0/378622412 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb52815c550 con 0x7fb52815a3e0 2024-09-16T01:42:46.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.103+0000 7fb524ff9640 1 -- 172.21.15.88:0/378622412 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb520005d10 con 0x7fb52815a3e0 2024-09-16T01:42:46.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.103+0000 7fb524ff9640 1 --2- 172.21.15.88:0/378622412 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb51406c2e0 0x7fb51406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:46.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.104+0000 7fb524ff9640 1 -- 172.21.15.88:0/378622412 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 7974+0+0 (secure 0 0 0) 0x7fb5200acbb0 con 0x7fb52815a3e0 2024-09-16T01:42:46.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.104+0000 7fb527fff640 1 --2- 172.21.15.88:0/378622412 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb51406c2e0 0x7fb51406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:46.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.104+0000 7fb527fff640 1 --2- 172.21.15.88:0/378622412 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb51406c2e0 0x7fb51406e7a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fb518006e80 tx=0x7fb518006df0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:46.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.108+0000 7fb524ff9640 1 -- 172.21.15.88:0/378622412 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb520049020 con 0x7fb52815a3e0 2024-09-16T01:42:46.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.308+0000 7fb52e2dd640 1 -- 172.21.15.88:0/378622412 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb5280ccf40 con 0x7fb52815a3e0 2024-09-16T01:42:46.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.309+0000 7fb524ff9640 1 -- 172.21.15.88:0/378622412 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) ==== 95+0+3566 (secure 0 0 0) 0x7fb5200b0030 con 0x7fb52815a3e0 2024-09-16T01:42:46.310 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:46.311 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":222,"btime":"2024-09-16T01:42:45:674170+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:45.674119+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:46.311 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 222 2024-09-16T01:42:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.311+0000 7fb52e2dd640 1 -- 172.21.15.88:0/378622412 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb51406c2e0 msgr2=0x7fb51406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.311+0000 7fb52e2dd640 1 --2- 172.21.15.88:0/378622412 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb51406c2e0 0x7fb51406e7a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fb518006e80 tx=0x7fb518006df0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.311+0000 7fb52e2dd640 1 -- 172.21.15.88:0/378622412 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb52815a3e0 msgr2=0x7fb52815a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.311+0000 7fb52e2dd640 1 --2- 172.21.15.88:0/378622412 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb52815a3e0 0x7fb52815a860 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fb52002aed0 tx=0x7fb520009750 comp rx=0 tx=0).stop 2024-09-16T01:42:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.311+0000 7fb52e2dd640 1 -- 172.21.15.88:0/378622412 shutdown_connections 2024-09-16T01:42:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.311+0000 7fb52e2dd640 1 --2- 172.21.15.88:0/378622412 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb51406c2e0 0x7fb51406e7a0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.311+0000 7fb52e2dd640 1 --2- 172.21.15.88:0/378622412 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb52815a3e0 0x7fb52815a860 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.311+0000 7fb52e2dd640 1 --2- 172.21.15.88:0/378622412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb52815cff0 0x7fb52815d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.311+0000 7fb52e2dd640 1 --2- 172.21.15.88:0/378622412 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb510004d40 0x7fb528147040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.311+0000 7fb52e2dd640 1 -- 172.21.15.88:0/378622412 >> 172.21.15.88:0/378622412 conn(0x7fb52806bc10 msgr2=0x7fb52806ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:46.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.311+0000 7fb52e2dd640 1 -- 172.21.15.88:0/378622412 shutdown_connections 2024-09-16T01:42:46.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.311+0000 7fb52e2dd640 1 -- 172.21.15.88:0/378622412 wait complete. 2024-09-16T01:42:46.323 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:42:46.323 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:46.381 INFO:tasks.ceph.mds.a.smithi088.stderr: -36> 2024-09-16T01:42:41.628+0000 7f7118bc5640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (22) Invalid argument 2024-09-16T01:42:46.405 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:42:46.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.422+0000 7facadae6640 1 Processor -- start 2024-09-16T01:42:46.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.422+0000 7facadae6640 1 -- start start 2024-09-16T01:42:46.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.422+0000 7facadae6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faca815e660 0x7faca80cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:46.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.422+0000 7facadae6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faca815ea40 con 0x7faca80d1820 2024-09-16T01:42:46.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.422+0000 7facadae6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faca80d3870 con 0x7faca80d3310 2024-09-16T01:42:46.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.422+0000 7facadae6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7faca80d3b50 con 0x7faca815e660 2024-09-16T01:42:46.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.423+0000 7faca7fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7faca80d1820 0x7faca80b7b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49326/0 (socket says 172.21.15.88:49326) 2024-09-16T01:42:46.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.423+0000 7faca7fff640 1 -- 172.21.15.88:0/1490566959 learned_addr learned my addr 172.21.15.88:0/1490566959 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:46.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.423+0000 7faca77fe640 1 --2- 172.21.15.88:0/1490566959 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faca815e660 0x7faca80cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:46.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.423+0000 7faca6ffd640 1 -- 172.21.15.88:0/1490566959 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3789978322 0 0) 0x7faca815ea40 con 0x7faca80d1820 2024-09-16T01:42:46.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.423+0000 7faca6ffd640 1 -- 172.21.15.88:0/1490566959 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac94003520 con 0x7faca80d1820 2024-09-16T01:42:46.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.423+0000 7faca6ffd640 1 -- 172.21.15.88:0/1490566959 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2771790853 0 0) 0x7faca80d3870 con 0x7faca80d3310 2024-09-16T01:42:46.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.423+0000 7faca6ffd640 1 -- 172.21.15.88:0/1490566959 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faca815ea40 con 0x7faca80d3310 2024-09-16T01:42:46.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.423+0000 7faca6ffd640 1 -- 172.21.15.88:0/1490566959 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4069227776 0 0) 0x7fac94003520 con 0x7faca80d1820 2024-09-16T01:42:46.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.423+0000 7faca6ffd640 1 -- 172.21.15.88:0/1490566959 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faca815e660 msgr2=0x7faca80cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:46.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.423+0000 7faca6ffd640 1 --2- 172.21.15.88:0/1490566959 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faca815e660 0x7faca80cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.423+0000 7faca6ffd640 1 -- 172.21.15.88:0/1490566959 >> v1:172.21.15.154:6789/0 conn(0x7faca80d3310 legacy=0x7faca80d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:46.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.423+0000 7faca6ffd640 1 -- 172.21.15.88:0/1490566959 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faca80d3e30 con 0x7faca80d1820 2024-09-16T01:42:46.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.423+0000 7faca6ffd640 1 -- 172.21.15.88:0/1490566959 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7faca00019f0 con 0x7faca80d1820 2024-09-16T01:42:46.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.423+0000 7faca77fe640 1 --2- 172.21.15.88:0/1490566959 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faca815e660 0x7faca80cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:42:46.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.423+0000 7faca6ffd640 1 -- 172.21.15.88:0/1490566959 >> v1:172.21.15.88:6789/0 conn(0x7faca80d1820 legacy=0x7faca80b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:46.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.423+0000 7faca6ffd640 1 --2- 172.21.15.88:0/1490566959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac94003b70 0x7fac94003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:46.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.423+0000 7faca6ffd640 1 --2- 172.21.15.88:0/1490566959 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faca815e660 0x7fac94004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:46.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.423+0000 7faca6ffd640 1 --2- 172.21.15.88:0/1490566959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac94004d40 0x7fac940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:46.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.423+0000 7faca6ffd640 1 -- 172.21.15.88:0/1490566959 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7faca80d3b50 con 0x7fac94004d40 2024-09-16T01:42:46.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.423+0000 7faca6ffd640 1 -- 172.21.15.88:0/1490566959 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fac9400ba70 con 0x7fac94003b70 2024-09-16T01:42:46.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.423+0000 7faca6ffd640 1 -- 172.21.15.88:0/1490566959 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fac9400bbf0 con 0x7faca815e660 2024-09-16T01:42:46.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.423+0000 7faca7fff640 1 --2- 172.21.15.88:0/1490566959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac94004d40 0x7fac940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:46.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.423+0000 7faca77fe640 1 --2- 172.21.15.88:0/1490566959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac94003b70 0x7fac94003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:46.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.423+0000 7fac9effd640 1 --2- 172.21.15.88:0/1490566959 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faca815e660 0x7fac94004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:46.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.424+0000 7faca7fff640 1 -- 172.21.15.88:0/1490566959 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faca815e660 msgr2=0x7fac94004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:46.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.424+0000 7faca7fff640 1 --2- 172.21.15.88:0/1490566959 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faca815e660 0x7fac94004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.424+0000 7faca7fff640 1 -- 172.21.15.88:0/1490566959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac94003b70 msgr2=0x7fac94003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:46.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.424+0000 7faca7fff640 1 --2- 172.21.15.88:0/1490566959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac94003b70 0x7fac94003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.424+0000 7faca7fff640 1 -- 172.21.15.88:0/1490566959 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac94003520 con 0x7fac94004d40 2024-09-16T01:42:46.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.424+0000 7faca7fff640 1 --2- 172.21.15.88:0/1490566959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac94004d40 0x7fac940051c0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7faca000a810 tx=0x7fac94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e17d8feeeef65fdf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:46.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.424+0000 7faca6ffd640 1 -- 172.21.15.88:0/1490566959 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca000a430 con 0x7fac94004d40 2024-09-16T01:42:46.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.424+0000 7faca6ffd640 1 -- 172.21.15.88:0/1490566959 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faca002f070 con 0x7fac94004d40 2024-09-16T01:42:46.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.424+0000 7faca6ffd640 1 -- 172.21.15.88:0/1490566959 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca002bc30 con 0x7fac94004d40 2024-09-16T01:42:46.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.425+0000 7facadae6640 1 -- 172.21.15.88:0/1490566959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac94004d40 msgr2=0x7fac940051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:46.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.425+0000 7facadae6640 1 --2- 172.21.15.88:0/1490566959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac94004d40 0x7fac940051c0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7faca000a810 tx=0x7fac94005830 comp rx=0 tx=0).stop 2024-09-16T01:42:46.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.425+0000 7faca7fff640 1 -- 172.21.15.88:0/1490566959 reap_dead start 2024-09-16T01:42:46.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.425+0000 7facadae6640 1 -- 172.21.15.88:0/1490566959 shutdown_connections 2024-09-16T01:42:46.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.425+0000 7facadae6640 1 -- 172.21.15.88:0/1490566959 >> 172.21.15.88:0/1490566959 conn(0x7faca80b6410 msgr2=0x7faca80b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:46.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.425+0000 7facadae6640 1 -- 172.21.15.88:0/1490566959 shutdown_connections 2024-09-16T01:42:46.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.425+0000 7facadae6640 1 -- 172.21.15.88:0/1490566959 wait complete. 2024-09-16T01:42:46.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.426+0000 7facadae6640 1 Processor -- start 2024-09-16T01:42:46.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.426+0000 7facadae6640 1 -- start start 2024-09-16T01:42:46.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.427+0000 7facadae6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac94004d40 0x7faca80cee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:46.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.427+0000 7facadae6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faca81517b0 0x7faca8151c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:46.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.427+0000 7facadae6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faca815fd10 0x7faca8160170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:46.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.427+0000 7facadae6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7faca80cf730 con 0x7fac94004d40 2024-09-16T01:42:46.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.427+0000 7facadae6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faca80d43f0 con 0x7faca815fd10 2024-09-16T01:42:46.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.427+0000 7facadae6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7faca80cf8b0 con 0x7faca81517b0 2024-09-16T01:42:46.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.427+0000 7faca77fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac94004d40 0x7faca80cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:46.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.427+0000 7faca77fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac94004d40 0x7faca80cee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37086/0 (socket says 172.21.15.88:37086) 2024-09-16T01:42:46.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.427+0000 7faca77fe640 1 -- 172.21.15.88:0/1139109663 learned_addr learned my addr 172.21.15.88:0/1139109663 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:46.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.427+0000 7faca7fff640 1 --2- 172.21.15.88:0/1139109663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faca815fd10 0x7faca8160170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:46.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.427+0000 7fac9effd640 1 --2- 172.21.15.88:0/1139109663 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faca81517b0 0x7faca8151c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:46.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.427+0000 7faca77fe640 1 -- 172.21.15.88:0/1139109663 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faca81517b0 msgr2=0x7faca8151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:46.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.427+0000 7faca77fe640 1 --2- 172.21.15.88:0/1139109663 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faca81517b0 0x7faca8151c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.427+0000 7faca77fe640 1 -- 172.21.15.88:0/1139109663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faca815fd10 msgr2=0x7faca8160170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:46.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.427+0000 7faca77fe640 1 --2- 172.21.15.88:0/1139109663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faca815fd10 0x7faca8160170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.427+0000 7faca77fe640 1 -- 172.21.15.88:0/1139109663 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faca8152170 con 0x7fac94004d40 2024-09-16T01:42:46.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.428+0000 7faca77fe640 1 --2- 172.21.15.88:0/1139109663 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac94004d40 0x7faca80cee30 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fac980119d0 tx=0x7fac98011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:46.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.428+0000 7faca4ff9640 1 -- 172.21.15.88:0/1139109663 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac9800faf0 con 0x7fac94004d40 2024-09-16T01:42:46.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.428+0000 7faca4ff9640 1 -- 172.21.15.88:0/1139109663 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac9800fc90 con 0x7fac94004d40 2024-09-16T01:42:46.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.428+0000 7faca4ff9640 1 -- 172.21.15.88:0/1139109663 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac98004030 con 0x7fac94004d40 2024-09-16T01:42:46.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.428+0000 7facadae6640 1 -- 172.21.15.88:0/1139109663 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faca8160a80 con 0x7fac94004d40 2024-09-16T01:42:46.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.428+0000 7facadae6640 1 -- 172.21.15.88:0/1139109663 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faca8161040 con 0x7fac94004d40 2024-09-16T01:42:46.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.429+0000 7facadae6640 1 -- 172.21.15.88:0/1139109663 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faca80cfd10 con 0x7fac94004d40 2024-09-16T01:42:46.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.429+0000 7faca4ff9640 1 -- 172.21.15.88:0/1139109663 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fac98012040 con 0x7fac94004d40 2024-09-16T01:42:46.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.429+0000 7faca4ff9640 1 --2- 172.21.15.88:0/1139109663 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fac8806c2b0 0x7fac8806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:46.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.430+0000 7faca4ff9640 1 -- 172.21.15.88:0/1139109663 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 7974+0+0 (secure 0 0 0) 0x7fac9808c0c0 con 0x7fac94004d40 2024-09-16T01:42:46.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.430+0000 7fac9effd640 1 --2- 172.21.15.88:0/1139109663 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fac8806c2b0 0x7fac8806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:46.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.430+0000 7fac9effd640 1 --2- 172.21.15.88:0/1139109663 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fac8806c2b0 0x7fac8806e770 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7faca8159190 tx=0x7fac90009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:46.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.434+0000 7faca4ff9640 1 -- 172.21.15.88:0/1139109663 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fac98019070 con 0x7fac94004d40 2024-09-16T01:42:46.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.635+0000 7facadae6640 1 -- 172.21.15.88:0/1139109663 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faca81594c0 con 0x7fac94004d40 2024-09-16T01:42:46.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.635+0000 7faca4ff9640 1 -- 172.21.15.88:0/1139109663 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) ==== 95+0+3566 (secure 0 0 0) 0x7fac9805ee20 con 0x7fac94004d40 2024-09-16T01:42:46.637 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:46.637 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":222,"btime":"2024-09-16T01:42:45:674170+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:45.674119+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:46.637 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 222 2024-09-16T01:42:46.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.637+0000 7facadae6640 1 -- 172.21.15.88:0/1139109663 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fac8806c2b0 msgr2=0x7fac8806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:46.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.637+0000 7facadae6640 1 --2- 172.21.15.88:0/1139109663 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fac8806c2b0 0x7fac8806e770 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7faca8159190 tx=0x7fac90009040 comp rx=0 tx=0).stop 2024-09-16T01:42:46.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.637+0000 7facadae6640 1 -- 172.21.15.88:0/1139109663 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac94004d40 msgr2=0x7faca80cee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:46.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.637+0000 7facadae6640 1 --2- 172.21.15.88:0/1139109663 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac94004d40 0x7faca80cee30 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fac980119d0 tx=0x7fac98011e90 comp rx=0 tx=0).stop 2024-09-16T01:42:46.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.637+0000 7facadae6640 1 -- 172.21.15.88:0/1139109663 shutdown_connections 2024-09-16T01:42:46.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.637+0000 7facadae6640 1 --2- 172.21.15.88:0/1139109663 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fac8806c2b0 0x7fac8806e770 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.637+0000 7facadae6640 1 --2- 172.21.15.88:0/1139109663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faca815fd10 0x7faca8160170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.637+0000 7facadae6640 1 --2- 172.21.15.88:0/1139109663 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faca81517b0 0x7faca8151c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.637+0000 7facadae6640 1 --2- 172.21.15.88:0/1139109663 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fac94004d40 0x7faca80cee30 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.637+0000 7facadae6640 1 -- 172.21.15.88:0/1139109663 >> 172.21.15.88:0/1139109663 conn(0x7faca80b6410 msgr2=0x7faca815cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:46.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.638+0000 7facadae6640 1 -- 172.21.15.88:0/1139109663 shutdown_connections 2024-09-16T01:42:46.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.638+0000 7facadae6640 1 -- 172.21.15.88:0/1139109663 wait complete. 2024-09-16T01:42:46.650 INFO:tasks.cephfs.test_damage:Result: Mutation 'Truncate 200.00000001' led to DAMAGED state 2024-09-16T01:42:46.650 INFO:tasks.cephfs.test_damage:Applying mutation 'Truncate 500.00000000' 2024-09-16T01:42:46.650 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:42:46.650 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:42:46.666 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:42:46.666 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:42:46.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:46.667 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:42:46.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:42:46.722 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:42:46.722 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:42:46.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.859+0000 7f6801731640 1 Processor -- start 2024-09-16T01:42:46.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.859+0000 7f6801731640 1 -- start start 2024-09-16T01:42:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.860+0000 7f6801731640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f67fc15d3d0 0x7f67fc0d09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.860+0000 7f6801731640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67fc071770 con 0x7f67fc06fa00 2024-09-16T01:42:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.860+0000 7f6801731640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67fc0709b0 con 0x7f67fc15a060 2024-09-16T01:42:46.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.860+0000 7f6801731640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f67fc0712e0 con 0x7f67fc15d3d0 2024-09-16T01:42:46.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.860+0000 7f67f27fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f67fc06fa00 0x7f67fc0d02d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49328/0 (socket says 172.21.15.88:49328) 2024-09-16T01:42:46.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.860+0000 7f67f27fc640 1 -- 172.21.15.88:0/2124781562 learned_addr learned my addr 172.21.15.88:0/2124781562 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:46.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.861+0000 7f67fa7fc640 1 -- 172.21.15.88:0/2124781562 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1537655169 0 0) 0x7f67fc071770 con 0x7f67fc06fa00 2024-09-16T01:42:46.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.861+0000 7f67fa7fc640 1 -- 172.21.15.88:0/2124781562 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67e8003520 con 0x7f67fc06fa00 2024-09-16T01:42:46.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.861+0000 7f67fb7fe640 1 --2- 172.21.15.88:0/2124781562 >> v2:172.21.15.154:3300/0 conn(0x7f67fc15d3d0 0x7f67fc0d09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:46.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.861+0000 7f67fa7fc640 1 -- 172.21.15.88:0/2124781562 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 819691013 0 0) 0x7f67e8003520 con 0x7f67fc06fa00 2024-09-16T01:42:46.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.861+0000 7f67fa7fc640 1 -- 172.21.15.88:0/2124781562 >> v2:172.21.15.154:3300/0 conn(0x7f67fc15d3d0 msgr2=0x7f67fc0d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:46.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.861+0000 7f67fa7fc640 1 --2- 172.21.15.88:0/2124781562 >> v2:172.21.15.154:3300/0 conn(0x7f67fc15d3d0 0x7f67fc0d09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.861+0000 7f67fa7fc640 1 -- 172.21.15.88:0/2124781562 >> v1:172.21.15.154:6789/0 conn(0x7f67fc15a060 legacy=0x7f67fc06f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:46.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.861+0000 7f67fa7fc640 1 -- 172.21.15.88:0/2124781562 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67fc070c90 con 0x7f67fc06fa00 2024-09-16T01:42:46.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.861+0000 7f67fa7fc640 1 -- 172.21.15.88:0/2124781562 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f67e4001670 con 0x7f67fc06fa00 2024-09-16T01:42:46.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.861+0000 7f67fa7fc640 1 -- 172.21.15.88:0/2124781562 >> v1:172.21.15.88:6789/0 conn(0x7f67fc06fa00 legacy=0x7f67fc0d02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:46.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.861+0000 7f67fa7fc640 1 --2- 172.21.15.88:0/2124781562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67e80039b0 0x7f67e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:46.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.861+0000 7f67fa7fc640 1 --2- 172.21.15.88:0/2124781562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67e8004800 0x7f67e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:46.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.861+0000 7f67fa7fc640 1 --2- 172.21.15.88:0/2124781562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67e80051c0 0x7f67e800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:46.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.861+0000 7f67fa7fc640 1 -- 172.21.15.88:0/2124781562 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f67fc0712e0 con 0x7f67e8004800 2024-09-16T01:42:46.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.861+0000 7f67fa7fc640 1 -- 172.21.15.88:0/2124781562 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f67e800c060 con 0x7f67e80039b0 2024-09-16T01:42:46.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.861+0000 7f67fa7fc640 1 -- 172.21.15.88:0/2124781562 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f67e800c260 con 0x7f67e80051c0 2024-09-16T01:42:46.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.861+0000 7f67f27fc640 1 --2- 172.21.15.88:0/2124781562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67e8004800 0x7f67e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:46.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.861+0000 7f67faffd640 1 --2- 172.21.15.88:0/2124781562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67e80039b0 0x7f67e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:46.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.861+0000 7f67fb7fe640 1 --2- 172.21.15.88:0/2124781562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67e80051c0 0x7f67e800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:46.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.861+0000 7f67f27fc640 1 -- 172.21.15.88:0/2124781562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67e80051c0 msgr2=0x7f67e800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:46.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.861+0000 7f67f27fc640 1 --2- 172.21.15.88:0/2124781562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67e80051c0 0x7f67e800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.861+0000 7f67f27fc640 1 -- 172.21.15.88:0/2124781562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67e80039b0 msgr2=0x7f67e8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:46.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.861+0000 7f67f27fc640 1 --2- 172.21.15.88:0/2124781562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67e80039b0 0x7f67e8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.861+0000 7f67f27fc640 1 -- 172.21.15.88:0/2124781562 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67e8003520 con 0x7f67e8004800 2024-09-16T01:42:46.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.861+0000 7f67fb7fe640 1 -- 172.21.15.88:0/2124781562 reap_dead start 2024-09-16T01:42:46.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.862+0000 7f67f27fc640 1 --2- 172.21.15.88:0/2124781562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67e8004800 0x7f67e8004c80 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f67e400c240 tx=0x7f67e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1133b8b5f078d9e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:46.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.862+0000 7f67fa7fc640 1 -- 172.21.15.88:0/2124781562 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e4007cf0 con 0x7f67e8004800 2024-09-16T01:42:46.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.862+0000 7f67fa7fc640 1 -- 172.21.15.88:0/2124781562 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67e4005af0 con 0x7f67e8004800 2024-09-16T01:42:46.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.862+0000 7f67fa7fc640 1 -- 172.21.15.88:0/2124781562 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e4007070 con 0x7f67e8004800 2024-09-16T01:42:46.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.863+0000 7f6801731640 1 -- 172.21.15.88:0/2124781562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67e8004800 msgr2=0x7f67e8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:46.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.863+0000 7f6801731640 1 --2- 172.21.15.88:0/2124781562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67e8004800 0x7f67e8004c80 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f67e400c240 tx=0x7f67e8005830 comp rx=0 tx=0).stop 2024-09-16T01:42:46.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.863+0000 7f6801731640 1 -- 172.21.15.88:0/2124781562 shutdown_connections 2024-09-16T01:42:46.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.863+0000 7f6801731640 1 --2- 172.21.15.88:0/2124781562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67e8004800 0x7f67e8004c80 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.863+0000 7f6801731640 1 -- 172.21.15.88:0/2124781562 >> 172.21.15.88:0/2124781562 conn(0x7f67fc0d2430 msgr2=0x7f67fc0d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:46.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.863+0000 7f6801731640 1 -- 172.21.15.88:0/2124781562 shutdown_connections 2024-09-16T01:42:46.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.863+0000 7f6801731640 1 -- 172.21.15.88:0/2124781562 wait complete. 2024-09-16T01:42:46.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.864+0000 7f6801731640 1 Processor -- start 2024-09-16T01:42:46.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.864+0000 7f6801731640 1 -- start start 2024-09-16T01:42:46.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.865+0000 7f6801731640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67e8004800 0x7f67fc1457b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:46.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.865+0000 7f6801731640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67fc1542d0 0x7f67fc152ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:46.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.865+0000 7f6801731640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67fc153000 0x7f67fc0b6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:46.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.865+0000 7f6801731640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f67fc070030 con 0x7f67e8004800 2024-09-16T01:42:46.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.865+0000 7f6801731640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f67fc06feb0 con 0x7f67fc1542d0 2024-09-16T01:42:46.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.865+0000 7f6801731640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f67fc0b1990 con 0x7f67fc153000 2024-09-16T01:42:46.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.865+0000 7f67faffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67e8004800 0x7f67fc1457b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:46.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.865+0000 7f67faffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67e8004800 0x7f67fc1457b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37106/0 (socket says 172.21.15.88:37106) 2024-09-16T01:42:46.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.865+0000 7f67faffd640 1 -- 172.21.15.88:0/510811900 learned_addr learned my addr 172.21.15.88:0/510811900 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:46.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.865+0000 7f67f27fc640 1 --2- 172.21.15.88:0/510811900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67fc1542d0 0x7f67fc152ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:46.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.865+0000 7f67fb7fe640 1 --2- 172.21.15.88:0/510811900 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67fc153000 0x7f67fc0b6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:46.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.865+0000 7f67faffd640 1 -- 172.21.15.88:0/510811900 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67fc153000 msgr2=0x7f67fc0b6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:46.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.865+0000 7f67faffd640 1 --2- 172.21.15.88:0/510811900 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67fc153000 0x7f67fc0b6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.865+0000 7f67faffd640 1 -- 172.21.15.88:0/510811900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67fc1542d0 msgr2=0x7f67fc152ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:46.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.865+0000 7f67faffd640 1 --2- 172.21.15.88:0/510811900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67fc1542d0 0x7f67fc152ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:46.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.865+0000 7f67faffd640 1 -- 172.21.15.88:0/510811900 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67fc153510 con 0x7f67e8004800 2024-09-16T01:42:46.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.866+0000 7f67faffd640 1 --2- 172.21.15.88:0/510811900 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67e8004800 0x7f67fc1457b0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f67ec00e480 tx=0x7f67ec00e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:46.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.866+0000 7f67f3fff640 1 -- 172.21.15.88:0/510811900 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec0052e0 con 0x7f67e8004800 2024-09-16T01:42:46.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.866+0000 7f67f3fff640 1 -- 172.21.15.88:0/510811900 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67ec018070 con 0x7f67e8004800 2024-09-16T01:42:46.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.866+0000 7f6801731640 1 -- 172.21.15.88:0/510811900 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67fc1537f0 con 0x7f67e8004800 2024-09-16T01:42:46.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.866+0000 7f67f3fff640 1 -- 172.21.15.88:0/510811900 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec003830 con 0x7f67e8004800 2024-09-16T01:42:46.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.866+0000 7f6801731640 1 -- 172.21.15.88:0/510811900 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67fc15f6e0 con 0x7f67e8004800 2024-09-16T01:42:46.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.867+0000 7f6801731640 1 -- 172.21.15.88:0/510811900 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67fc06c6c0 con 0x7f67e8004800 2024-09-16T01:42:46.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.867+0000 7f67f3fff640 1 -- 172.21.15.88:0/510811900 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f67ec0039d0 con 0x7f67e8004800 2024-09-16T01:42:46.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.867+0000 7f67f3fff640 1 --2- 172.21.15.88:0/510811900 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f67dc06c2e0 0x7f67dc06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:46.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.868+0000 7f67f3fff640 1 -- 172.21.15.88:0/510811900 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 7974+0+0 (secure 0 0 0) 0x7f67ec08b820 con 0x7f67e8004800 2024-09-16T01:42:46.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.868+0000 7f67f27fc640 1 --2- 172.21.15.88:0/510811900 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f67dc06c2e0 0x7f67dc06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:46.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.868+0000 7f67f27fc640 1 --2- 172.21.15.88:0/510811900 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f67dc06c2e0 0x7f67dc06e7a0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f67e4032000 tx=0x7f67e4034040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:46.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:46.872+0000 7f67f3fff640 1 -- 172.21.15.88:0/510811900 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f67ec0590f0 con 0x7f67e8004800 2024-09-16T01:42:47.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.070+0000 7f6801731640 1 -- 172.21.15.88:0/510811900 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f67fc0d19d0 con 0x7f67e8004800 2024-09-16T01:42:47.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.688+0000 7f67f3fff640 1 -- 172.21.15.88:0/510811900 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v224) ==== 161+0+0 (secure 0 0 0) 0x7f67ec05d3f0 con 0x7f67e8004800 2024-09-16T01:42:47.690 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:42:47.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.690+0000 7f6801731640 1 -- 172.21.15.88:0/510811900 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f67dc06c2e0 msgr2=0x7f67dc06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:47.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.690+0000 7f6801731640 1 --2- 172.21.15.88:0/510811900 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f67dc06c2e0 0x7f67dc06e7a0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f67e4032000 tx=0x7f67e4034040 comp rx=0 tx=0).stop 2024-09-16T01:42:47.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.690+0000 7f6801731640 1 -- 172.21.15.88:0/510811900 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67e8004800 msgr2=0x7f67fc1457b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:47.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.690+0000 7f6801731640 1 --2- 172.21.15.88:0/510811900 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67e8004800 0x7f67fc1457b0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f67ec00e480 tx=0x7f67ec00e940 comp rx=0 tx=0).stop 2024-09-16T01:42:47.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.690+0000 7f6801731640 1 -- 172.21.15.88:0/510811900 shutdown_connections 2024-09-16T01:42:47.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.690+0000 7f6801731640 1 --2- 172.21.15.88:0/510811900 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f67dc06c2e0 0x7f67dc06e7a0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:47.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.690+0000 7f6801731640 1 --2- 172.21.15.88:0/510811900 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67fc153000 0x7f67fc0b6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:47.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.690+0000 7f6801731640 1 --2- 172.21.15.88:0/510811900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67fc1542d0 0x7f67fc152ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:47.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.690+0000 7f6801731640 1 --2- 172.21.15.88:0/510811900 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67e8004800 0x7f67fc1457b0 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:47.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.690+0000 7f6801731640 1 -- 172.21.15.88:0/510811900 >> 172.21.15.88:0/510811900 conn(0x7f67fc0d2430 msgr2=0x7f67fc0b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:47.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.690+0000 7f6801731640 1 -- 172.21.15.88:0/510811900 shutdown_connections 2024-09-16T01:42:47.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.691+0000 7f6801731640 1 -- 172.21.15.88:0/510811900 wait complete. 2024-09-16T01:42:47.716 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:42:47.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.815+0000 7f8c88a5e640 1 Processor -- start 2024-09-16T01:42:47.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.816+0000 7f8c88a5e640 1 -- start start 2024-09-16T01:42:47.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.816+0000 7f8c88a5e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c8415b480 0x7f8c840b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:47.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.816+0000 7f8c88a5e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c840ccff0 con 0x7f8c84072700 2024-09-16T01:42:47.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.816+0000 7f8c88a5e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c840cd2d0 con 0x7f8c840b60c0 2024-09-16T01:42:47.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.816+0000 7f8c88a5e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8c840b6500 con 0x7f8c8415b480 2024-09-16T01:42:47.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.817+0000 7f8c82d9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8c84072700 0x7f8c8415bc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49340/0 (socket says 172.21.15.88:49340) 2024-09-16T01:42:47.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.817+0000 7f8c82d9d640 1 -- 172.21.15.88:0/2898150225 learned_addr learned my addr 172.21.15.88:0/2898150225 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:47.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.817+0000 7f8c8159a640 1 -- 172.21.15.88:0/2898150225 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3834360054 0 0) 0x7f8c840ccff0 con 0x7f8c84072700 2024-09-16T01:42:47.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.817+0000 7f8c8159a640 1 -- 172.21.15.88:0/2898150225 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c70003520 con 0x7f8c84072700 2024-09-16T01:42:47.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.817+0000 7f8c8259c640 1 --2- 172.21.15.88:0/2898150225 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c8415b480 0x7f8c840b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:47.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.817+0000 7f8c8159a640 1 -- 172.21.15.88:0/2898150225 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1641002185 0 0) 0x7f8c70003520 con 0x7f8c84072700 2024-09-16T01:42:47.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.817+0000 7f8c8159a640 1 -- 172.21.15.88:0/2898150225 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c8415b480 msgr2=0x7f8c840b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:47.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.817+0000 7f8c8159a640 1 --2- 172.21.15.88:0/2898150225 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c8415b480 0x7f8c840b5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:47.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.817+0000 7f8c8159a640 1 -- 172.21.15.88:0/2898150225 >> v1:172.21.15.154:6789/0 conn(0x7f8c840b60c0 legacy=0x7f8c84071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:47.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.817+0000 7f8c8159a640 1 -- 172.21.15.88:0/2898150225 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c840cd5b0 con 0x7f8c84072700 2024-09-16T01:42:47.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.817+0000 7f8c8159a640 1 -- 172.21.15.88:0/2898150225 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f8c7c0019f0 con 0x7f8c84072700 2024-09-16T01:42:47.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.817+0000 7f8c8159a640 1 -- 172.21.15.88:0/2898150225 >> v1:172.21.15.88:6789/0 conn(0x7f8c84072700 legacy=0x7f8c8415bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:47.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.817+0000 7f8c8159a640 1 --2- 172.21.15.88:0/2898150225 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c8415b480 0x7f8c700039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:47.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.817+0000 7f8c8159a640 1 --2- 172.21.15.88:0/2898150225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c70003fb0 0x7f8c70004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:47.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.817+0000 7f8c8159a640 1 --2- 172.21.15.88:0/2898150225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c70004d40 0x7f8c700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:47.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.817+0000 7f8c8159a640 1 -- 172.21.15.88:0/2898150225 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8c840b6500 con 0x7f8c70004d40 2024-09-16T01:42:47.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.817+0000 7f8c8159a640 1 -- 172.21.15.88:0/2898150225 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7000bbb0 con 0x7f8c70003fb0 2024-09-16T01:42:47.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.817+0000 7f8c8159a640 1 -- 172.21.15.88:0/2898150225 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8c7000bdb0 con 0x7f8c8415b480 2024-09-16T01:42:47.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.817+0000 7f8c82d9d640 1 --2- 172.21.15.88:0/2898150225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c70004d40 0x7f8c700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:47.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.817+0000 7f8c8259c640 1 --2- 172.21.15.88:0/2898150225 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c8415b480 0x7f8c700039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:47.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.817+0000 7f8c81d9b640 1 --2- 172.21.15.88:0/2898150225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c70003fb0 0x7f8c70004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:47.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.818+0000 7f8c82d9d640 1 -- 172.21.15.88:0/2898150225 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c8415b480 msgr2=0x7f8c700039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:47.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.818+0000 7f8c82d9d640 1 --2- 172.21.15.88:0/2898150225 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c8415b480 0x7f8c700039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:47.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.818+0000 7f8c82d9d640 1 -- 172.21.15.88:0/2898150225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c70003fb0 msgr2=0x7f8c70004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:47.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.818+0000 7f8c82d9d640 1 --2- 172.21.15.88:0/2898150225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c70003fb0 0x7f8c70004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:47.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.818+0000 7f8c82d9d640 1 -- 172.21.15.88:0/2898150225 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c70003520 con 0x7f8c70004d40 2024-09-16T01:42:47.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.818+0000 7f8c82d9d640 1 --2- 172.21.15.88:0/2898150225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c70004d40 0x7f8c700051c0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c00a790 tx=0x7f8c70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12a75daa894fa792 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:47.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.818+0000 7f8c8159a640 1 -- 172.21.15.88:0/2898150225 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c00a3b0 con 0x7f8c70004d40 2024-09-16T01:42:47.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.818+0000 7f8c8159a640 1 -- 172.21.15.88:0/2898150225 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c7c02f070 con 0x7f8c70004d40 2024-09-16T01:42:47.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.818+0000 7f8c8159a640 1 -- 172.21.15.88:0/2898150225 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c02bc40 con 0x7f8c70004d40 2024-09-16T01:42:47.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.819+0000 7f8c88a5e640 1 -- 172.21.15.88:0/2898150225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c70004d40 msgr2=0x7f8c700051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:47.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.819+0000 7f8c88a5e640 1 --2- 172.21.15.88:0/2898150225 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c70004d40 0x7f8c700051c0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c00a790 tx=0x7f8c70005830 comp rx=0 tx=0).stop 2024-09-16T01:42:47.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.819+0000 7f8c82d9d640 1 -- 172.21.15.88:0/2898150225 reap_dead start 2024-09-16T01:42:47.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.819+0000 7f8c88a5e640 1 -- 172.21.15.88:0/2898150225 shutdown_connections 2024-09-16T01:42:47.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.819+0000 7f8c88a5e640 1 -- 172.21.15.88:0/2898150225 >> 172.21.15.88:0/2898150225 conn(0x7f8c8406d0f0 msgr2=0x7f8c8406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:47.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.819+0000 7f8c88a5e640 1 -- 172.21.15.88:0/2898150225 shutdown_connections 2024-09-16T01:42:47.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.819+0000 7f8c88a5e640 1 -- 172.21.15.88:0/2898150225 wait complete. 2024-09-16T01:42:47.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.820+0000 7f8c88a5e640 1 Processor -- start 2024-09-16T01:42:47.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.820+0000 7f8c88a5e640 1 -- start start 2024-09-16T01:42:47.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.821+0000 7f8c88a5e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c70004d40 0x7f8c84146220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:47.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.821+0000 7f8c88a5e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c841546c0 0x7f8c840cf590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:47.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.821+0000 7f8c88a5e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c840cfad0 0x7f8c84152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:47.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.821+0000 7f8c88a5e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8c8415b860 con 0x7f8c841546c0 2024-09-16T01:42:47.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.821+0000 7f8c88a5e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8c840cdb70 con 0x7f8c70004d40 2024-09-16T01:42:47.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.821+0000 7f8c88a5e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8c8415b9e0 con 0x7f8c840cfad0 2024-09-16T01:42:47.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.821+0000 7f8c81d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c841546c0 0x7f8c840cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:47.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.821+0000 7f8c81d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c841546c0 0x7f8c840cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37132/0 (socket says 172.21.15.88:37132) 2024-09-16T01:42:47.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.821+0000 7f8c81d9b640 1 -- 172.21.15.88:0/1043441831 learned_addr learned my addr 172.21.15.88:0/1043441831 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:47.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.821+0000 7f8c8259c640 1 --2- 172.21.15.88:0/1043441831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c70004d40 0x7f8c84146220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:47.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.821+0000 7f8c82d9d640 1 --2- 172.21.15.88:0/1043441831 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c840cfad0 0x7f8c84152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:47.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.822+0000 7f8c81d9b640 1 -- 172.21.15.88:0/1043441831 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c840cfad0 msgr2=0x7f8c84152960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:47.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.822+0000 7f8c81d9b640 1 --2- 172.21.15.88:0/1043441831 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c840cfad0 0x7f8c84152960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:47.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.822+0000 7f8c81d9b640 1 -- 172.21.15.88:0/1043441831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c70004d40 msgr2=0x7f8c84146220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:47.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.822+0000 7f8c81d9b640 1 --2- 172.21.15.88:0/1043441831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c70004d40 0x7f8c84146220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:47.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.822+0000 7f8c81d9b640 1 -- 172.21.15.88:0/1043441831 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c84153620 con 0x7f8c841546c0 2024-09-16T01:42:47.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.822+0000 7f8c8259c640 1 --2- 172.21.15.88:0/1043441831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c70004d40 0x7f8c84146220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:47.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.822+0000 7f8c81d9b640 1 --2- 172.21.15.88:0/1043441831 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c841546c0 0x7f8c840cf590 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c00f820 tx=0x7f8c6c00fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:47.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.822+0000 7f8c7affd640 1 -- 172.21.15.88:0/1043441831 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c0050d0 con 0x7f8c841546c0 2024-09-16T01:42:47.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.822+0000 7f8c7affd640 1 -- 172.21.15.88:0/1043441831 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c6c018070 con 0x7f8c841546c0 2024-09-16T01:42:47.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.822+0000 7f8c88a5e640 1 -- 172.21.15.88:0/1043441831 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c84153900 con 0x7f8c841546c0 2024-09-16T01:42:47.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.822+0000 7f8c7affd640 1 -- 172.21.15.88:0/1043441831 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c005490 con 0x7f8c841546c0 2024-09-16T01:42:47.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.822+0000 7f8c88a5e640 1 -- 172.21.15.88:0/1043441831 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c84153ec0 con 0x7f8c841546c0 2024-09-16T01:42:47.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.823+0000 7f8c88a5e640 1 -- 172.21.15.88:0/1043441831 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c840b5b80 con 0x7f8c841546c0 2024-09-16T01:42:47.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.823+0000 7f8c7affd640 1 -- 172.21.15.88:0/1043441831 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8c6c00d730 con 0x7f8c841546c0 2024-09-16T01:42:47.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.824+0000 7f8c7affd640 1 --2- 172.21.15.88:0/1043441831 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8c6406c2e0 0x7f8c6406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:47.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.824+0000 7f8c8259c640 1 --2- 172.21.15.88:0/1043441831 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8c6406c2e0 0x7f8c6406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:47.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.824+0000 7f8c7affd640 1 -- 172.21.15.88:0/1043441831 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 7974+0+0 (secure 0 0 0) 0x7f8c6c08bbc0 con 0x7f8c841546c0 2024-09-16T01:42:47.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.824+0000 7f8c8259c640 1 --2- 172.21.15.88:0/1043441831 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8c6406c2e0 0x7f8c6406e7a0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f8c74009ab0 tx=0x7f8c74008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:47.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:47.828+0000 7f8c7affd640 1 -- 172.21.15.88:0/1043441831 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8c6c0599c0 con 0x7f8c841546c0 2024-09-16T01:42:48.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.069+0000 7f8c88a5e640 1 -- 172.21.15.88:0/1043441831 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f8c840cee60 con 0x7f8c841546c0 2024-09-16T01:42:48.153 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:42:48.157 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:42:48.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.693+0000 7f8c7affd640 1 -- 172.21.15.88:0/1043441831 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v225) ==== 101+0+0 (secure 0 0 0) 0x7f8c6c0592c0 con 0x7f8c841546c0 2024-09-16T01:42:48.694 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:42:48.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.696+0000 7f8c88a5e640 1 -- 172.21.15.88:0/1043441831 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8c6406c2e0 msgr2=0x7f8c6406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:48.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.696+0000 7f8c88a5e640 1 --2- 172.21.15.88:0/1043441831 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8c6406c2e0 0x7f8c6406e7a0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f8c74009ab0 tx=0x7f8c74008040 comp rx=0 tx=0).stop 2024-09-16T01:42:48.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.696+0000 7f8c88a5e640 1 -- 172.21.15.88:0/1043441831 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c841546c0 msgr2=0x7f8c840cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:48.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.696+0000 7f8c88a5e640 1 --2- 172.21.15.88:0/1043441831 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c841546c0 0x7f8c840cf590 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c00f820 tx=0x7f8c6c00fce0 comp rx=0 tx=0).stop 2024-09-16T01:42:48.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.696+0000 7f8c88a5e640 1 -- 172.21.15.88:0/1043441831 shutdown_connections 2024-09-16T01:42:48.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.696+0000 7f8c88a5e640 1 --2- 172.21.15.88:0/1043441831 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8c6406c2e0 0x7f8c6406e7a0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:48.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.696+0000 7f8c88a5e640 1 --2- 172.21.15.88:0/1043441831 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c840cfad0 0x7f8c84152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:48.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.696+0000 7f8c88a5e640 1 --2- 172.21.15.88:0/1043441831 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8c841546c0 0x7f8c840cf590 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:48.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.696+0000 7f8c88a5e640 1 --2- 172.21.15.88:0/1043441831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c70004d40 0x7f8c84146220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:48.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.696+0000 7f8c88a5e640 1 -- 172.21.15.88:0/1043441831 >> 172.21.15.88:0/1043441831 conn(0x7f8c8406d0f0 msgr2=0x7f8c8406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:48.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.696+0000 7f8c88a5e640 1 -- 172.21.15.88:0/1043441831 shutdown_connections 2024-09-16T01:42:48.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.696+0000 7f8c88a5e640 1 -- 172.21.15.88:0/1043441831 wait complete. 2024-09-16T01:42:48.709 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:42:48.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.751+0000 7f6cc6d1cf40 1 Processor -- start 2024-09-16T01:42:48.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.751+0000 7f6cc6d1cf40 1 -- start start 2024-09-16T01:42:48.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.752+0000 7f6cc6d1cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b85414f530 0x55b8540b5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:48.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.752+0000 7f6cc6d1cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b85408ae60 0x55b8540b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:48.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.752+0000 7f6cc6d1cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b8541565d0 0x55b854156a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:48.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.752+0000 7f6cc6d1cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b853fd9510 con 0x55b85408ae60 2024-09-16T01:42:48.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.752+0000 7f6cc6d1cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b8540cfbd0 con 0x55b85414f530 2024-09-16T01:42:48.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.752+0000 7f6cc6d1cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55b8540d2f60 con 0x55b8541565d0 2024-09-16T01:42:48.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.752+0000 7f6cbffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b85408ae60 0x55b8540b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:48.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.752+0000 7f6cbffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b85408ae60 0x55b8540b5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37156/0 (socket says 172.21.15.88:37156) 2024-09-16T01:42:48.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.752+0000 7f6cbffff640 1 -- 172.21.15.88:0/374144851 learned_addr learned my addr 172.21.15.88:0/374144851 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:48.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.752+0000 7f6cc4812640 1 --2- 172.21.15.88:0/374144851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b85414f530 0x55b8540b5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:48.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.752+0000 7f6cc5013640 1 --2- 172.21.15.88:0/374144851 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b8541565d0 0x55b854156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:48.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.752+0000 7f6cbffff640 1 -- 172.21.15.88:0/374144851 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b8541565d0 msgr2=0x55b854156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:48.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.752+0000 7f6cbffff640 1 --2- 172.21.15.88:0/374144851 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b8541565d0 0x55b854156a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:48.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.752+0000 7f6cbffff640 1 -- 172.21.15.88:0/374144851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b85414f530 msgr2=0x55b8540b5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:48.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.752+0000 7f6cbffff640 1 --2- 172.21.15.88:0/374144851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b85414f530 0x55b8540b5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:48.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.752+0000 7f6cbffff640 1 -- 172.21.15.88:0/374144851 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8540b6750 con 0x55b85408ae60 2024-09-16T01:42:48.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.752+0000 7f6cbffff640 1 --2- 172.21.15.88:0/374144851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b85408ae60 0x55b8540b5db0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8009790 tx=0x7f6cb8005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46f538f4ff092534 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:48.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.752+0000 7f6cbeffd640 1 -- 172.21.15.88:0/374144851 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb80078f0 con 0x55b85408ae60 2024-09-16T01:42:48.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.752+0000 7f6cbeffd640 1 -- 172.21.15.88:0/374144851 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cb8007a90 con 0x55b85408ae60 2024-09-16T01:42:48.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.752+0000 7f6cbeffd640 1 -- 172.21.15.88:0/374144851 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb8007e60 con 0x55b85408ae60 2024-09-16T01:42:48.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.753+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/374144851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b85408ae60 msgr2=0x55b8540b5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:48.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.753+0000 7f6cc6d1cf40 1 --2- 172.21.15.88:0/374144851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b85408ae60 0x55b8540b5db0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f6cb8009790 tx=0x7f6cb8005a00 comp rx=0 tx=0).stop 2024-09-16T01:42:48.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.753+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/374144851 shutdown_connections 2024-09-16T01:42:48.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.753+0000 7f6cc6d1cf40 1 --2- 172.21.15.88:0/374144851 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b8541565d0 0x55b854156a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:48.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.753+0000 7f6cc6d1cf40 1 --2- 172.21.15.88:0/374144851 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b85408ae60 0x55b8540b5db0 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:48.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.753+0000 7f6cc6d1cf40 1 --2- 172.21.15.88:0/374144851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b85414f530 0x55b8540b5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:48.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.753+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/374144851 >> 172.21.15.88:0/374144851 conn(0x55b8540d2750 msgr2=0x55b85415ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:48.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.753+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/374144851 shutdown_connections 2024-09-16T01:42:48.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.753+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/374144851 wait complete. 2024-09-16T01:42:48.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.753+0000 7f6cc6d1cf40 1 Processor -- start 2024-09-16T01:42:48.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.754+0000 7f6cc6d1cf40 1 -- start start 2024-09-16T01:42:48.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.754+0000 7f6cc6d1cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b85408ae60 0x55b854087690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:48.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.754+0000 7f6cc6d1cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b85414f530 0x55b85415f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:48.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.754+0000 7f6cc6d1cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b8541565d0 0x55b854086b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:48.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.754+0000 7f6cc6d1cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b85415c620 con 0x55b8541565d0 2024-09-16T01:42:48.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.754+0000 7f6cc6d1cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b8540b3c20 con 0x55b85408ae60 2024-09-16T01:42:48.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.754+0000 7f6cc6d1cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55b85415c900 con 0x55b85414f530 2024-09-16T01:42:48.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.754+0000 7f6cc5013640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b8541565d0 0x55b854086b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:48.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.754+0000 7f6cc5013640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b8541565d0 0x55b854086b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37162/0 (socket says 172.21.15.88:37162) 2024-09-16T01:42:48.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.754+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 learned_addr learned my addr 172.21.15.88:0/378509145 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:48.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.754+0000 7f6cc4812640 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b85408ae60 0x55b854087690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:48.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.754+0000 7f6cbffff640 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b85414f530 0x55b85415f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:48.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.755+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b85414f530 msgr2=0x55b85415f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:48.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.755+0000 7f6cc5013640 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b85414f530 0x55b85415f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:48.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.755+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b85408ae60 msgr2=0x55b854087690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:48.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.755+0000 7f6cc5013640 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b85408ae60 0x55b854087690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:48.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.755+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b85414eb30 con 0x55b8541565d0 2024-09-16T01:42:48.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.755+0000 7f6cc5013640 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b8541565d0 0x55b854086b40 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0002230 tx=0x7f6cc00033a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:48.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.755+0000 7f6cbdffb640 1 -- 172.21.15.88:0/378509145 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0068070 con 0x55b8541565d0 2024-09-16T01:42:48.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.755+0000 7f6cbdffb640 1 -- 172.21.15.88:0/378509145 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cc004ede0 con 0x55b8541565d0 2024-09-16T01:42:48.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.755+0000 7f6cbdffb640 1 -- 172.21.15.88:0/378509145 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc000c280 con 0x55b8541565d0 2024-09-16T01:42:48.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.755+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b85414dd70 con 0x55b8541565d0 2024-09-16T01:42:48.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.755+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b85414e330 con 0x55b8541565d0 2024-09-16T01:42:48.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.756+0000 7f6cbdffb640 1 -- 172.21.15.88:0/378509145 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6cc006b070 con 0x55b8541565d0 2024-09-16T01:42:48.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.757+0000 7f6cbdffb640 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ca806cea0 0x7f6ca806f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:48.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.757+0000 7f6cbdffb640 1 -- 172.21.15.88:0/378509145 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 7974+0+0 (secure 0 0 0) 0x7f6cc00cdcf0 con 0x55b8541565d0 2024-09-16T01:42:48.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.757+0000 7f6cc4812640 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ca806cea0 0x7f6ca806f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:48.763 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:42:48.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.757+0000 7f6cc6d1cf40 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55b854087080 0x55b8541f6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:48.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.757+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8541512a0 con 0x55b854087080 2024-09-16T01:42:48.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.757+0000 7f6cbffff640 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55b854087080 0x55b8541f6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:48.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.757+0000 7f6cc4812640 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ca806cea0 0x7f6ca806f360 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0001920 tx=0x7f6cb0011040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:48.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.758+0000 7f6cbffff640 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55b854087080 0x55b8541f6670 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:48.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.759+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v113'784 uv758 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.759+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e113) -- 0x55b854151670 con 0x55b854087080 2024-09-16T01:42:48.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.761+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v113'785 uv785 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.761+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854151ad0 con 0x55b854087080 2024-09-16T01:42:48.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.762+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v113'786 uv786 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.766 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:42:48.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.762+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854151e70 con 0x55b854087080 2024-09-16T01:42:48.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.764+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v113'787 uv787 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.764+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854152270 con 0x55b854087080 2024-09-16T01:42:48.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.765+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v113'788 uv788 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.765+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8541fef80 con 0x55b854087080 2024-09-16T01:42:48.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.766+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v113'789 uv789 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.766+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8541ff320 con 0x55b854087080 2024-09-16T01:42:48.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.768+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v113'790 uv764 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.768+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e113) -- 0x55b8541ff6c0 con 0x55b854087080 2024-09-16T01:42:48.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.769+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v113'791 uv791 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.769+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8541ff6c0 con 0x55b854087080 2024-09-16T01:42:48.772 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:42:48.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.770+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v113'792 uv792 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.770+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542006d0 con 0x55b854087080 2024-09-16T01:42:48.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.772+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v113'793 uv793 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.772+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542006d0 con 0x55b854087080 2024-09-16T01:42:48.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.773+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v113'794 uv794 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.773+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8541ff6c0 con 0x55b854087080 2024-09-16T01:42:48.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.774+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v113'795 uv795 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.774+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8541ff6c0 con 0x55b854087080 2024-09-16T01:42:48.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.776+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v113'796 uv771 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.776+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e113) -- 0x55b8541ff6c0 con 0x55b854087080 2024-09-16T01:42:48.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.777+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v113'797 uv797 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.777+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8541ff6c0 con 0x55b854087080 2024-09-16T01:42:48.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.778+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v113'798 uv798 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.780 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:42:48.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.778+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542006d0 con 0x55b854087080 2024-09-16T01:42:48.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.780+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v113'799 uv799 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.780+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542006d0 con 0x55b854087080 2024-09-16T01:42:48.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.781+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v113'800 uv800 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.781+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8541ff6c0 con 0x55b854087080 2024-09-16T01:42:48.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.782+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v113'801 uv801 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.782+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8541ffe20 con 0x55b854087080 2024-09-16T01:42:48.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.783+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v113'802 uv802 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.783+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8541ffe20 con 0x55b854087080 2024-09-16T01:42:48.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.784+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v113'803 uv777 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.784+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e113) -- 0x55b8541ffe20 con 0x55b854087080 2024-09-16T01:42:48.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.786+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v113'804 uv804 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.786+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8541ffe20 con 0x55b854087080 2024-09-16T01:42:48.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.787+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v113'805 uv805 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.789 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:42:48.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.787+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854200e30 con 0x55b854087080 2024-09-16T01:42:48.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.788+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v113'806 uv806 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.788+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854200e30 con 0x55b854087080 2024-09-16T01:42:48.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.790+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v113'807 uv807 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.790+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8541ffe20 con 0x55b854087080 2024-09-16T01:42:48.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.791+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v113'808 uv808 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.791+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8541ffe20 con 0x55b854087080 2024-09-16T01:42:48.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.793+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v113'809 uv783 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.793+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e113) -- 0x55b8541ffe20 con 0x55b854087080 2024-09-16T01:42:48.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.794+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v113'810 uv810 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.794+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8541ffe20 con 0x55b854087080 2024-09-16T01:42:48.796 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:42:48.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.795+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v113'811 uv811 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.795+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854200e30 con 0x55b854087080 2024-09-16T01:42:48.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.796+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v113'812 uv812 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.796+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854200e30 con 0x55b854087080 2024-09-16T01:42:48.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.797+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v113'813 uv813 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.797+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8541ffe20 con 0x55b854087080 2024-09-16T01:42:48.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.798+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v113'814 uv814 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.798+0000 7f6cc6d1cf40 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55b854200330 0x55b854202720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:48.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.798+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854202c60 con 0x55b854200330 2024-09-16T01:42:48.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.799+0000 7f6cc4812640 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55b854200330 0x55b854202720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:48.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.799+0000 7f6cc4812640 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55b854200330 0x55b854202720 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:48.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.801+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v113'572 uv554 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b854200330 2024-09-16T01:42:48.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.801+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e113) -- 0x55b854203000 con 0x55b854200330 2024-09-16T01:42:48.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.802+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v113'573 uv573 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b854200330 2024-09-16T01:42:48.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.802+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542033c0 con 0x55b854200330 2024-09-16T01:42:48.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.803+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v113'574 uv574 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b854200330 2024-09-16T01:42:48.805 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:42:48.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.803+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854203cb0 con 0x55b854200330 2024-09-16T01:42:48.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.804+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v113'575 uv575 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b854200330 2024-09-16T01:42:48.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.804+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854204050 con 0x55b854200330 2024-09-16T01:42:48.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.805+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v113'576 uv576 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b854200330 2024-09-16T01:42:48.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.805+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854204440 con 0x55b854200330 2024-09-16T01:42:48.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.806+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v113'577 uv559 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b854200330 2024-09-16T01:42:48.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.806+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e113) -- 0x55b8542047e0 con 0x55b854200330 2024-09-16T01:42:48.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.807+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v113'578 uv578 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b854200330 2024-09-16T01:42:48.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.807+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854204bd0 con 0x55b854200330 2024-09-16T01:42:48.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.808+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v113'579 uv579 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b854200330 2024-09-16T01:42:48.810 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:42:48.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.808+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854204bd0 con 0x55b854200330 2024-09-16T01:42:48.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.809+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v113'580 uv580 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b854200330 2024-09-16T01:42:48.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.809+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854204b80 con 0x55b854200330 2024-09-16T01:42:48.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.810+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v113'581 uv581 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b854200330 2024-09-16T01:42:48.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.810+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854204b80 con 0x55b854200330 2024-09-16T01:42:48.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.812+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v113'582 uv571 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b854200330 2024-09-16T01:42:48.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.812+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e113) -- 0x55b854204b80 con 0x55b854200330 2024-09-16T01:42:48.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.813+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v113'583 uv583 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b854200330 2024-09-16T01:42:48.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.813+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854204b80 con 0x55b854200330 2024-09-16T01:42:48.815 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:42:48.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.814+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v113'584 uv584 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b854200330 2024-09-16T01:42:48.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.814+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854204b80 con 0x55b854200330 2024-09-16T01:42:48.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.815+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v113'585 uv585 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b854200330 2024-09-16T01:42:48.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.815+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854204b80 con 0x55b854200330 2024-09-16T01:42:48.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.816+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v113'586 uv586 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b854200330 2024-09-16T01:42:48.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.816+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854204b80 con 0x55b854200330 2024-09-16T01:42:48.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.818+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v113'587 uv570 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b854200330 2024-09-16T01:42:48.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.818+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e113) -- 0x55b854204b80 con 0x55b854200330 2024-09-16T01:42:48.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.819+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v113'588 uv588 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b854200330 2024-09-16T01:42:48.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.819+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854204b80 con 0x55b854200330 2024-09-16T01:42:48.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.820+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v113'589 uv589 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b854200330 2024-09-16T01:42:48.821 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:42:48.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.820+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854204bc0 con 0x55b854200330 2024-09-16T01:42:48.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.821+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v113'590 uv590 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b854200330 2024-09-16T01:42:48.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.821+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854206080 con 0x55b854200330 2024-09-16T01:42:48.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.822+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v113'591 uv591 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b854200330 2024-09-16T01:42:48.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.822+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854204f70 con 0x55b854200330 2024-09-16T01:42:48.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.823+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v113'592 uv592 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b854200330 2024-09-16T01:42:48.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.823+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854204f70 con 0x55b854087080 2024-09-16T01:42:48.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.824+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v113'153 uv152 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.824+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e113) -- 0x55b854204f70 con 0x55b854087080 2024-09-16T01:42:48.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.825+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v113'154 uv154 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.825+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854204f70 con 0x55b854087080 2024-09-16T01:42:48.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.826+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v113'155 uv155 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.828 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:42:48.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.826+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854206010 con 0x55b854087080 2024-09-16T01:42:48.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.827+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v113'156 uv156 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.827+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854205fd0 con 0x55b854087080 2024-09-16T01:42:48.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.828+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v113'157 uv157 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.828+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854205fd0 con 0x55b854087080 2024-09-16T01:42:48.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.829+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v113'158 uv158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.829+0000 7f6cc6d1cf40 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55b854205480 0x55b8542058b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:48.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.829+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854208400 con 0x55b854205480 2024-09-16T01:42:48.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.829+0000 7f6cbffff640 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55b854205480 0x55b8542058b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:48.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.829+0000 7f6cbffff640 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55b854205480 0x55b8542058b0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:48.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.830+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v113'113 uv112 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854205480 2024-09-16T01:42:48.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.830+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e113) -- 0x55b854208400 con 0x55b854205480 2024-09-16T01:42:48.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.831+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v113'114 uv114 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854205480 2024-09-16T01:42:48.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.831+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854208480 con 0x55b854205480 2024-09-16T01:42:48.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.832+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v113'115 uv115 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854205480 2024-09-16T01:42:48.835 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:42:48.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.832+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854208870 con 0x55b854205480 2024-09-16T01:42:48.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.833+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v113'116 uv116 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854205480 2024-09-16T01:42:48.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.833+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854208870 con 0x55b854087080 2024-09-16T01:42:48.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.835+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v113'153 uv152 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.835+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e113) -- 0x55b854208870 con 0x55b854087080 2024-09-16T01:42:48.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.836+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v113'154 uv154 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.836+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854208870 con 0x55b854087080 2024-09-16T01:42:48.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.837+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v113'155 uv155 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.839 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:42:48.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.837+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854208480 con 0x55b854087080 2024-09-16T01:42:48.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.838+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v113'156 uv156 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.838+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854208480 con 0x55b854087080 2024-09-16T01:42:48.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.839+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v113'157 uv157 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.839+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854208480 con 0x55b854087080 2024-09-16T01:42:48.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.841+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v113'158 uv158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.841+0000 7f6cc6d1cf40 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b854208d80 0x55b854209180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:48.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.841+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b85420b9a0 con 0x55b854208d80 2024-09-16T01:42:48.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.841+0000 7f6cc5013640 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b854208d80 0x55b854209180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:48.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.841+0000 7f6cc5013640 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b854208d80 0x55b854209180 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:48.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.842+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v113'282 uv276 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.842+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e113) -- 0x55b85420bd40 con 0x55b854208d80 2024-09-16T01:42:48.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.843+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v113'283 uv283 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.843+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b85420c0e0 con 0x55b854208d80 2024-09-16T01:42:48.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.844+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v113'284 uv284 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.846 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:42:48.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.844+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854203880 con 0x55b854208d80 2024-09-16T01:42:48.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.845+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v113'285 uv285 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.845+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b85420d990 con 0x55b854208d80 2024-09-16T01:42:48.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.846+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v113'286 uv286 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.846+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b85420c930 con 0x55b854208d80 2024-09-16T01:42:48.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.847+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v113'287 uv287 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.847+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b85420cd40 con 0x55b854208d80 2024-09-16T01:42:48.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.848+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v113'288 uv281 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.848+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e113) -- 0x55b85420cd40 con 0x55b854208d80 2024-09-16T01:42:48.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.849+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v113'289 uv289 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.849+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b85420cd40 con 0x55b854208d80 2024-09-16T01:42:48.851 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:42:48.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.850+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v113'290 uv290 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.850+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b85420cd40 con 0x55b854208d80 2024-09-16T01:42:48.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.851+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v113'291 uv291 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.851+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b85420cd40 con 0x55b854208d80 2024-09-16T01:42:48.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.852+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v113'292 uv292 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.852+0000 7f6cc6d1cf40 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55b85420dde0 0x55b8542101d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:48.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.852+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854210710 con 0x55b85420dde0 2024-09-16T01:42:48.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.852+0000 7f6cc4812640 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55b85420dde0 0x55b8542101d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:48.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.853+0000 7f6cc4812640 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55b85420dde0 0x55b8542101d0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:48.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.854+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v113'127 uv126 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b85420dde0 2024-09-16T01:42:48.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.854+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e113) -- 0x55b854210710 con 0x55b85420dde0 2024-09-16T01:42:48.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.855+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v113'128 uv128 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b85420dde0 2024-09-16T01:42:48.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.855+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854210710 con 0x55b85420dde0 2024-09-16T01:42:48.857 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:42:48.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.856+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v113'129 uv129 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b85420dde0 2024-09-16T01:42:48.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.856+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854210710 con 0x55b85420dde0 2024-09-16T01:42:48.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.857+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v113'130 uv130 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b85420dde0 2024-09-16T01:42:48.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.857+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854210710 con 0x55b85420dde0 2024-09-16T01:42:48.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.857+0000 7f6cc4812640 1 -- 172.21.15.88:0/378509145 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v113'131 uv131 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6cb000fc40 con 0x55b85420dde0 2024-09-16T01:42:48.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.858+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854210710 con 0x55b854087080 2024-09-16T01:42:48.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.859+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v113'153 uv152 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.859+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e113) -- 0x55b854210710 con 0x55b854087080 2024-09-16T01:42:48.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.860+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v113'154 uv154 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.860+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854210710 con 0x55b854087080 2024-09-16T01:42:48.866 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:42:48.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.861+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v113'155 uv155 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.861+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854210710 con 0x55b854087080 2024-09-16T01:42:48.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.862+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v113'156 uv156 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.862+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854210710 con 0x55b854087080 2024-09-16T01:42:48.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.863+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v113'157 uv157 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.863+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854210710 con 0x55b854087080 2024-09-16T01:42:48.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.864+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v113'158 uv158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854087080 2024-09-16T01:42:48.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.864+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854210710 con 0x55b854205480 2024-09-16T01:42:48.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.865+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v113'253 uv247 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854205480 2024-09-16T01:42:48.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.865+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e113) -- 0x55b854210710 con 0x55b854205480 2024-09-16T01:42:48.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.866+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v113'254 uv254 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854205480 2024-09-16T01:42:48.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.866+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854210710 con 0x55b854205480 2024-09-16T01:42:48.869 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:42:48.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.866+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v113'255 uv255 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854205480 2024-09-16T01:42:48.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.867+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854210b00 con 0x55b854205480 2024-09-16T01:42:48.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.868+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v113'256 uv256 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854205480 2024-09-16T01:42:48.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.868+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854210b00 con 0x55b854205480 2024-09-16T01:42:48.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.868+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v113'257 uv257 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854205480 2024-09-16T01:42:48.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.869+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854210b00 con 0x55b854205480 2024-09-16T01:42:48.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.869+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v113'258 uv252 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854205480 2024-09-16T01:42:48.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.869+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e113) -- 0x55b854210b00 con 0x55b854205480 2024-09-16T01:42:48.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.870+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v113'259 uv259 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854205480 2024-09-16T01:42:48.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.871+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854210b00 con 0x55b854205480 2024-09-16T01:42:48.872 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:42:48.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.871+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v113'260 uv260 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854205480 2024-09-16T01:42:48.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.871+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854210b00 con 0x55b854205480 2024-09-16T01:42:48.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.872+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v113'261 uv261 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854205480 2024-09-16T01:42:48.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.872+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854213910 con 0x55b854205480 2024-09-16T01:42:48.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.874+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v113'262 uv262 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854205480 2024-09-16T01:42:48.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.874+0000 7f6cc6d1cf40 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55b8542114c0 0x55b854213d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:48.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.874+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b8542114c0 2024-09-16T01:42:48.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.874+0000 7f6cbffff640 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55b8542114c0 0x55b854213d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:48.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.874+0000 7f6cbffff640 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55b8542114c0 0x55b854213d50 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:48.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.875+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v113'111 uv110 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b8542114c0 2024-09-16T01:42:48.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.875+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b8542114c0 2024-09-16T01:42:48.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.876+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v113'112 uv112 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b8542114c0 2024-09-16T01:42:48.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.876+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b8542114c0 2024-09-16T01:42:48.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.877+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v113'113 uv113 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b8542114c0 2024-09-16T01:42:48.879 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:42:48.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.877+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b8542114c0 2024-09-16T01:42:48.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.878+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v113'114 uv114 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b8542114c0 2024-09-16T01:42:48.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.878+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b854205480 2024-09-16T01:42:48.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.879+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v113'153 uv152 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854205480 2024-09-16T01:42:48.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.879+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b854205480 2024-09-16T01:42:48.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.881+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v113'154 uv154 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854205480 2024-09-16T01:42:48.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.881+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b854205480 2024-09-16T01:42:48.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.882+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v113'155 uv155 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854205480 2024-09-16T01:42:48.883 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:42:48.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.882+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b854205480 2024-09-16T01:42:48.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.882+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v113'156 uv156 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854205480 2024-09-16T01:42:48.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.882+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b854205480 2024-09-16T01:42:48.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.883+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v113'157 uv157 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854205480 2024-09-16T01:42:48.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.883+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b854205480 2024-09-16T01:42:48.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.885+0000 7f6cbffff640 1 -- 172.21.15.88:0/378509145 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v113'158 uv158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cb8007c40 con 0x55b854205480 2024-09-16T01:42:48.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.885+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b854208d80 2024-09-16T01:42:48.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.886+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v113'130 uv129 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.886+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b854208d80 2024-09-16T01:42:48.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.887+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v113'131 uv131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.887+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b854208d80 2024-09-16T01:42:48.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.888+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v113'132 uv132 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.889 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:42:48.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.888+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b854208d80 2024-09-16T01:42:48.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.889+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v113'133 uv133 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.889+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b854208d80 2024-09-16T01:42:48.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.890+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v113'134 uv134 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.890+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b854208d80 2024-09-16T01:42:48.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.891+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v113'314 uv306 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.891+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b854208d80 2024-09-16T01:42:48.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.892+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v113'315 uv315 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.892+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b854208d80 2024-09-16T01:42:48.895 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:42:48.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.893+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v113'316 uv316 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.893+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b854208d80 2024-09-16T01:42:48.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.895+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v113'317 uv317 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.895+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b854208d80 2024-09-16T01:42:48.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.896+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v113'318 uv318 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.896+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b854208d80 2024-09-16T01:42:48.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.897+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v113'319 uv313 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.897+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b854208d80 2024-09-16T01:42:48.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.898+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v113'320 uv320 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.898+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b854208d80 2024-09-16T01:42:48.900 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:42:48.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.898+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v113'321 uv321 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.898+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b854208d80 2024-09-16T01:42:48.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.899+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v113'322 uv322 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.899+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b854208d80 2024-09-16T01:42:48.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.900+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v113'323 uv323 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.900+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b8542138b0 con 0x55b854208d80 2024-09-16T01:42:48.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.901+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v113'324 uv324 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.901+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b854214290 con 0x55b854208d80 2024-09-16T01:42:48.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.902+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v113'325 uv325 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f6cc006a030 con 0x55b854208d80 2024-09-16T01:42:48.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.902+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55b854087080 msgr2=0x55b8541f6670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:48.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.902+0000 7f6cc6d1cf40 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55b854087080 0x55b8541f6670 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:48.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.902+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55b854205480 msgr2=0x55b8542058b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:48.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.902+0000 7f6cc6d1cf40 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55b854205480 0x55b8542058b0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:48.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.902+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55b8542114c0 msgr2=0x55b854213d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:48.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.902+0000 7f6cc6d1cf40 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55b8542114c0 0x55b854213d50 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:48.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.902+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55b854200330 msgr2=0x55b854202720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:48.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.902+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 reap_dead start 2024-09-16T01:42:48.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.902+0000 7f6cc6d1cf40 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55b854200330 0x55b854202720 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:48.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.902+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b854208d80 msgr2=0x55b854209180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:48.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.902+0000 7f6cc6d1cf40 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b854208d80 0x55b854209180 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:48.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.902+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55b85420dde0 msgr2=0x55b8542101d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:48.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.902+0000 7f6cc6d1cf40 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55b85420dde0 0x55b8542101d0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:48.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.902+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ca806cea0 msgr2=0x7f6ca806f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:48.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.902+0000 7f6cc6d1cf40 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ca806cea0 0x7f6ca806f360 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0001920 tx=0x7f6cb0011040 comp rx=0 tx=0).stop 2024-09-16T01:42:48.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.902+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b8541565d0 msgr2=0x55b854086b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:48.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.902+0000 7f6cc6d1cf40 1 --2- 172.21.15.88:0/378509145 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b8541565d0 0x55b854086b40 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0002230 tx=0x7f6cc00033a0 comp rx=0 tx=0).stop 2024-09-16T01:42:48.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.902+0000 7f6cc5013640 1 -- 172.21.15.88:0/378509145 reap_dead start 2024-09-16T01:42:48.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.903+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 shutdown_connections 2024-09-16T01:42:48.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.903+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 >> 172.21.15.88:0/378509145 conn(0x55b8540d2750 msgr2=0x55b8540b1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:48.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.903+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 shutdown_connections 2024-09-16T01:42:48.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.903+0000 7f6cc6d1cf40 1 -- 172.21.15.88:0/378509145 wait complete. 2024-09-16T01:42:48.909 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata truncate 500.00000000 0 2024-09-16T01:42:48.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.993+0000 7fda4c20cf40 1 Processor -- start 2024-09-16T01:42:48.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.993+0000 7fda4c20cf40 1 -- start start 2024-09-16T01:42:48.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.993+0000 7fda4c20cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b4e51e4dd0 0x55b4e51e3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:48.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.993+0000 7fda4c20cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b4e51e44f0 0x55b4e51c09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:48.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.993+0000 7fda4c20cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b4e513b0f0 0x55b4e51c0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:48.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.993+0000 7fda4c20cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b4e505e510 con 0x55b4e51e44f0 2024-09-16T01:42:48.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.993+0000 7fda4c20cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b4e51554a0 con 0x55b4e513b0f0 2024-09-16T01:42:48.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.993+0000 7fda4c20cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55b4e510f6a0 con 0x55b4e51e4dd0 2024-09-16T01:42:48.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.994+0000 7fda43fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b4e51e44f0 0x55b4e51c09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:48.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.994+0000 7fda43fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b4e51e44f0 0x55b4e51c09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37182/0 (socket says 172.21.15.88:37182) 2024-09-16T01:42:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.994+0000 7fda43fff640 1 -- 172.21.15.88:0/513654204 learned_addr learned my addr 172.21.15.88:0/513654204 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.994+0000 7fda48f97640 1 --2- 172.21.15.88:0/513654204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b4e51e4dd0 0x55b4e51e3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.994+0000 7fda49798640 1 --2- 172.21.15.88:0/513654204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b4e513b0f0 0x55b4e51c0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.994+0000 7fda43fff640 1 -- 172.21.15.88:0/513654204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b4e51e4dd0 msgr2=0x55b4e51e3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.994+0000 7fda43fff640 1 --2- 172.21.15.88:0/513654204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b4e51e4dd0 0x55b4e51e3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.994+0000 7fda43fff640 1 -- 172.21.15.88:0/513654204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b4e513b0f0 msgr2=0x55b4e51c0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.994+0000 7fda43fff640 1 --2- 172.21.15.88:0/513654204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b4e513b0f0 0x55b4e51c0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.994+0000 7fda43fff640 1 -- 172.21.15.88:0/513654204 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4e513b7e0 con 0x55b4e51e44f0 2024-09-16T01:42:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.994+0000 7fda49798640 1 --2- 172.21.15.88:0/513654204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b4e513b0f0 0x55b4e51c0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.994+0000 7fda43fff640 1 --2- 172.21.15.88:0/513654204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b4e51e44f0 0x55b4e51c09a0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7fda34025e70 tx=0x7fda34051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a43d4402d385a5e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.994+0000 7fda42ffd640 1 -- 172.21.15.88:0/513654204 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3404f8a0 con 0x55b4e51e44f0 2024-09-16T01:42:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.994+0000 7fda42ffd640 1 -- 172.21.15.88:0/513654204 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda3404fa40 con 0x55b4e51e44f0 2024-09-16T01:42:48.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.994+0000 7fda42ffd640 1 -- 172.21.15.88:0/513654204 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3404fe30 con 0x55b4e51e44f0 2024-09-16T01:42:48.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.995+0000 7fda4c20cf40 1 -- 172.21.15.88:0/513654204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b4e51e44f0 msgr2=0x55b4e51c09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:48.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.995+0000 7fda4c20cf40 1 --2- 172.21.15.88:0/513654204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b4e51e44f0 0x55b4e51c09a0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7fda34025e70 tx=0x7fda34051000 comp rx=0 tx=0).stop 2024-09-16T01:42:48.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.995+0000 7fda4c20cf40 1 -- 172.21.15.88:0/513654204 shutdown_connections 2024-09-16T01:42:48.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.995+0000 7fda4c20cf40 1 --2- 172.21.15.88:0/513654204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b4e513b0f0 0x55b4e51c0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:48.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.995+0000 7fda4c20cf40 1 --2- 172.21.15.88:0/513654204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b4e51e44f0 0x55b4e51c09a0 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:48.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.995+0000 7fda4c20cf40 1 --2- 172.21.15.88:0/513654204 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b4e51e4dd0 0x55b4e51e3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:48.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.995+0000 7fda4c20cf40 1 -- 172.21.15.88:0/513654204 >> 172.21.15.88:0/513654204 conn(0x55b4e5137960 msgr2=0x55b4e51e2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:48.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.995+0000 7fda4c20cf40 1 -- 172.21.15.88:0/513654204 shutdown_connections 2024-09-16T01:42:48.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.995+0000 7fda4c20cf40 1 -- 172.21.15.88:0/513654204 wait complete. 2024-09-16T01:42:48.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.995+0000 7fda4c20cf40 1 Processor -- start 2024-09-16T01:42:48.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.995+0000 7fda4c20cf40 1 -- start start 2024-09-16T01:42:49.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.996+0000 7fda4c20cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b4e513b0f0 0x55b4e527e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:49.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.996+0000 7fda4c20cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b4e51e44f0 0x55b4e5152d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:49.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.996+0000 7fda4c20cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b4e51e4dd0 0x55b4e51532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:49.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.996+0000 7fda4c20cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55b4e51e3ab0 con 0x55b4e51e44f0 2024-09-16T01:42:49.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.996+0000 7fda4c20cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55b4e513bac0 con 0x55b4e513b0f0 2024-09-16T01:42:49.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.996+0000 7fda4c20cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55b4e51afd10 con 0x55b4e51e4dd0 2024-09-16T01:42:49.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.996+0000 7fda43fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b4e51e44f0 0x55b4e5152d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:49.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.996+0000 7fda43fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b4e51e44f0 0x55b4e5152d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37194/0 (socket says 172.21.15.88:37194) 2024-09-16T01:42:49.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.996+0000 7fda43fff640 1 -- 172.21.15.88:0/1571483043 learned_addr learned my addr 172.21.15.88:0/1571483043 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:49.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.996+0000 7fda48f97640 1 --2- 172.21.15.88:0/1571483043 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b4e513b0f0 0x55b4e527e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:49.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.996+0000 7fda49798640 1 --2- 172.21.15.88:0/1571483043 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b4e51e4dd0 0x55b4e51532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:49.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.996+0000 7fda43fff640 1 -- 172.21.15.88:0/1571483043 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b4e51e4dd0 msgr2=0x55b4e51532c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:49.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.996+0000 7fda43fff640 1 --2- 172.21.15.88:0/1571483043 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55b4e51e4dd0 0x55b4e51532c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:49.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.996+0000 7fda43fff640 1 -- 172.21.15.88:0/1571483043 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b4e513b0f0 msgr2=0x55b4e527e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:49.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.996+0000 7fda43fff640 1 --2- 172.21.15.88:0/1571483043 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b4e513b0f0 0x55b4e527e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:49.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.996+0000 7fda43fff640 1 -- 172.21.15.88:0/1571483043 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4e51e2190 con 0x55b4e51e44f0 2024-09-16T01:42:49.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.997+0000 7fda43fff640 1 --2- 172.21.15.88:0/1571483043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b4e51e44f0 0x55b4e5152d80 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fda34025e70 tx=0x7fda34003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:49.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.997+0000 7fda41ffb640 1 -- 172.21.15.88:0/1571483043 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda34058480 con 0x55b4e51e44f0 2024-09-16T01:42:49.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.997+0000 7fda41ffb640 1 -- 172.21.15.88:0/1571483043 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda34058620 con 0x55b4e51e44f0 2024-09-16T01:42:49.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.997+0000 7fda4c20cf40 1 -- 172.21.15.88:0/1571483043 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b4e51e2470 con 0x55b4e51e44f0 2024-09-16T01:42:49.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.997+0000 7fda41ffb640 1 -- 172.21.15.88:0/1571483043 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda34058a10 con 0x55b4e51e44f0 2024-09-16T01:42:49.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.997+0000 7fda4c20cf40 1 -- 172.21.15.88:0/1571483043 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4e5151dc0 con 0x55b4e51e44f0 2024-09-16T01:42:49.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.998+0000 7fda41ffb640 1 -- 172.21.15.88:0/1571483043 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fda34058bb0 con 0x55b4e51e44f0 2024-09-16T01:42:49.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.998+0000 7fda41ffb640 1 --2- 172.21.15.88:0/1571483043 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fda2c06acc0 0x7fda2c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:49.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.999+0000 7fda41ffb640 1 -- 172.21.15.88:0/1571483043 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 7974+0+0 (secure 0 0 0) 0x7fda340cf380 con 0x55b4e51e44f0 2024-09-16T01:42:49.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.999+0000 7fda48f97640 1 --2- 172.21.15.88:0/1571483043 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fda2c06acc0 0x7fda2c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:49.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.999+0000 7fda4c20cf40 1 --2- 172.21.15.88:0/1571483043 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b4e510c340 0x55b4e510c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:49.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.999+0000 7fda4c20cf40 1 -- 172.21.15.88:0/1571483043 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:1 12.15 12:aa448500:::500.00000000:head [truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55b4e510cc60 con 0x55b4e510c340 2024-09-16T01:42:49.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.999+0000 7fda49798640 1 --2- 172.21.15.88:0/1571483043 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b4e510c340 0x55b4e510c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:49.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.999+0000 7fda48f97640 1 --2- 172.21.15.88:0/1571483043 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fda2c06acc0 0x7fda2c06d180 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fda3c0057f0 tx=0x7fda3c00c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:48.999+0000 7fda49798640 1 --2- 172.21.15.88:0/1571483043 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b4e510c340 0x55b4e510c720 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.000+0000 7fda49798640 1 -- 172.21.15.88:0/1571483043 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(1 500.00000000 [truncate 0] v113'293 uv293 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fda44012040 con 0x55b4e510c340 2024-09-16T01:42:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.001+0000 7fda4c20cf40 1 -- 172.21.15.88:0/1571483043 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b4e510c340 msgr2=0x55b4e510c720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.001+0000 7fda4c20cf40 1 --2- 172.21.15.88:0/1571483043 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55b4e510c340 0x55b4e510c720 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.001+0000 7fda4c20cf40 1 -- 172.21.15.88:0/1571483043 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fda2c06acc0 msgr2=0x7fda2c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.001+0000 7fda4c20cf40 1 --2- 172.21.15.88:0/1571483043 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fda2c06acc0 0x7fda2c06d180 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fda3c0057f0 tx=0x7fda3c00c040 comp rx=0 tx=0).stop 2024-09-16T01:42:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.001+0000 7fda4c20cf40 1 -- 172.21.15.88:0/1571483043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b4e51e44f0 msgr2=0x55b4e5152d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:49.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.001+0000 7fda4c20cf40 1 --2- 172.21.15.88:0/1571483043 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55b4e51e44f0 0x55b4e5152d80 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fda34025e70 tx=0x7fda34003020 comp rx=0 tx=0).stop 2024-09-16T01:42:49.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.001+0000 7fda49798640 1 -- 172.21.15.88:0/1571483043 reap_dead start 2024-09-16T01:42:49.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.001+0000 7fda4c20cf40 1 -- 172.21.15.88:0/1571483043 shutdown_connections 2024-09-16T01:42:49.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.001+0000 7fda4c20cf40 1 -- 172.21.15.88:0/1571483043 >> 172.21.15.88:0/1571483043 conn(0x55b4e5137960 msgr2=0x55b4e51e2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:49.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.001+0000 7fda4c20cf40 1 -- 172.21.15.88:0/1571483043 shutdown_connections 2024-09-16T01:42:49.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.001+0000 7fda4c20cf40 1 -- 172.21.15.88:0/1571483043 wait complete. 2024-09-16T01:42:49.009 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:42:49.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.112+0000 7fca3575f640 1 Processor -- start 2024-09-16T01:42:49.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.112+0000 7fca3575f640 1 -- start start 2024-09-16T01:42:49.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.113+0000 7fca3575f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fca3015c0f0 0x7fca300739d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:49.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.113+0000 7fca3575f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca30073f10 0x7fca3006bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:49.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.113+0000 7fca3575f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fca3006c150 0x7fca30162140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:49.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.113+0000 7fca3575f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fca3015c4d0 con 0x7fca3015c0f0 2024-09-16T01:42:49.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.113+0000 7fca3575f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fca3015ee60 con 0x7fca3006c150 2024-09-16T01:42:49.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.113+0000 7fca3575f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fca3015f140 con 0x7fca30073f10 2024-09-16T01:42:49.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.113+0000 7fca2effd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fca3015c0f0 0x7fca300739d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:49.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.113+0000 7fca2effd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fca3015c0f0 0x7fca300739d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37206/0 (socket says 172.21.15.88:37206) 2024-09-16T01:42:49.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.113+0000 7fca2effd640 1 -- 172.21.15.88:0/550312652 learned_addr learned my addr 172.21.15.88:0/550312652 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:49.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.114+0000 7fca2f7fe640 1 --2- 172.21.15.88:0/550312652 >> v2:172.21.15.154:3300/0 conn(0x7fca3006c150 0x7fca30162140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:49.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.114+0000 7fca2effd640 1 -- 172.21.15.88:0/550312652 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca30073f10 msgr2=0x7fca3006bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:49.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.114+0000 7fca2effd640 1 --2- 172.21.15.88:0/550312652 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca30073f10 0x7fca3006bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:49.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.114+0000 7fca2effd640 1 -- 172.21.15.88:0/550312652 >> v2:172.21.15.154:3300/0 conn(0x7fca3006c150 msgr2=0x7fca30162140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:49.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.114+0000 7fca2effd640 1 --2- 172.21.15.88:0/550312652 >> v2:172.21.15.154:3300/0 conn(0x7fca3006c150 0x7fca30162140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:49.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.114+0000 7fca2effd640 1 -- 172.21.15.88:0/550312652 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca3015f420 con 0x7fca3015c0f0 2024-09-16T01:42:49.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.114+0000 7fca2effd640 1 --2- 172.21.15.88:0/550312652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca3015c0f0 0x7fca300739d0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fca3015fc50 tx=0x7fca2002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a23de9be9484c02d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:49.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.115+0000 7fca2dffb640 1 -- 172.21.15.88:0/550312652 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca20037070 con 0x7fca3015c0f0 2024-09-16T01:42:49.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.115+0000 7fca2dffb640 1 -- 172.21.15.88:0/550312652 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca20032440 con 0x7fca3015c0f0 2024-09-16T01:42:49.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.115+0000 7fca2dffb640 1 -- 172.21.15.88:0/550312652 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca20032810 con 0x7fca3015c0f0 2024-09-16T01:42:49.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.115+0000 7fca3575f640 1 -- 172.21.15.88:0/550312652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca3015c0f0 msgr2=0x7fca300739d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:49.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.115+0000 7fca3575f640 1 --2- 172.21.15.88:0/550312652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca3015c0f0 0x7fca300739d0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fca3015fc50 tx=0x7fca2002a140 comp rx=0 tx=0).stop 2024-09-16T01:42:49.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.115+0000 7fca3575f640 1 -- 172.21.15.88:0/550312652 shutdown_connections 2024-09-16T01:42:49.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.115+0000 7fca3575f640 1 --2- 172.21.15.88:0/550312652 >> v2:172.21.15.154:3300/0 conn(0x7fca3006c150 0x7fca30162140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:49.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.115+0000 7fca3575f640 1 --2- 172.21.15.88:0/550312652 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca30073f10 0x7fca3006bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:49.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.115+0000 7fca3575f640 1 --2- 172.21.15.88:0/550312652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca3015c0f0 0x7fca300739d0 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:49.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.115+0000 7fca3575f640 1 -- 172.21.15.88:0/550312652 >> 172.21.15.88:0/550312652 conn(0x7fca300d1390 msgr2=0x7fca300d17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:49.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.115+0000 7fca3575f640 1 -- 172.21.15.88:0/550312652 shutdown_connections 2024-09-16T01:42:49.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.115+0000 7fca3575f640 1 -- 172.21.15.88:0/550312652 wait complete. 2024-09-16T01:42:49.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.117+0000 7fca3575f640 1 Processor -- start 2024-09-16T01:42:49.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.117+0000 7fca3575f640 1 -- start start 2024-09-16T01:42:49.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.117+0000 7fca3575f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca3006c150 0x7fca30150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:49.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.117+0000 7fca3575f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca30073f10 0x7fca3006f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:49.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.117+0000 7fca3575f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca3015c0f0 0x7fca3006fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:49.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.117+0000 7fca3575f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fca30072070 con 0x7fca3015c0f0 2024-09-16T01:42:49.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.117+0000 7fca3575f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fca30071ef0 con 0x7fca30073f10 2024-09-16T01:42:49.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.117+0000 7fca3575f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fca300b0ba0 con 0x7fca3006c150 2024-09-16T01:42:49.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.117+0000 7fca2f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca3015c0f0 0x7fca3006fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:49.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.117+0000 7fca2f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca3015c0f0 0x7fca3006fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37218/0 (socket says 172.21.15.88:37218) 2024-09-16T01:42:49.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.117+0000 7fca2f7fe640 1 -- 172.21.15.88:0/3077829278 learned_addr learned my addr 172.21.15.88:0/3077829278 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:49.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.117+0000 7fca2e7fc640 1 --2- 172.21.15.88:0/3077829278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca30073f10 0x7fca3006f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:49.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.117+0000 7fca2effd640 1 --2- 172.21.15.88:0/3077829278 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca3006c150 0x7fca30150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:49.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.118+0000 7fca2f7fe640 1 -- 172.21.15.88:0/3077829278 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca3006c150 msgr2=0x7fca30150a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:49.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.118+0000 7fca2f7fe640 1 --2- 172.21.15.88:0/3077829278 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca3006c150 0x7fca30150a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:49.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.118+0000 7fca2f7fe640 1 -- 172.21.15.88:0/3077829278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca30073f10 msgr2=0x7fca3006f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:49.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.118+0000 7fca2f7fe640 1 --2- 172.21.15.88:0/3077829278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca30073f10 0x7fca3006f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:49.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.118+0000 7fca2f7fe640 1 -- 172.21.15.88:0/3077829278 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca3014b750 con 0x7fca3015c0f0 2024-09-16T01:42:49.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.118+0000 7fca2f7fe640 1 --2- 172.21.15.88:0/3077829278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca3015c0f0 0x7fca3006fb90 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fca2800cd70 tx=0x7fca2800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:49.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.118+0000 7fca177fe640 1 -- 172.21.15.88:0/3077829278 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28003b90 con 0x7fca3015c0f0 2024-09-16T01:42:49.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.118+0000 7fca177fe640 1 -- 172.21.15.88:0/3077829278 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca28003d30 con 0x7fca3015c0f0 2024-09-16T01:42:49.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.118+0000 7fca177fe640 1 -- 172.21.15.88:0/3077829278 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28004240 con 0x7fca3015c0f0 2024-09-16T01:42:49.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.118+0000 7fca3575f640 1 -- 172.21.15.88:0/3077829278 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca3014ba30 con 0x7fca3015c0f0 2024-09-16T01:42:49.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.118+0000 7fca3575f640 1 -- 172.21.15.88:0/3077829278 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca3014bff0 con 0x7fca3015c0f0 2024-09-16T01:42:49.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.119+0000 7fca3575f640 1 -- 172.21.15.88:0/3077829278 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca30162ff0 con 0x7fca3015c0f0 2024-09-16T01:42:49.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.120+0000 7fca177fe640 1 -- 172.21.15.88:0/3077829278 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fca280043e0 con 0x7fca3015c0f0 2024-09-16T01:42:49.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.120+0000 7fca177fe640 1 --2- 172.21.15.88:0/3077829278 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fca1c06c2b0 0x7fca1c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:49.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.120+0000 7fca177fe640 1 -- 172.21.15.88:0/3077829278 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 7974+0+0 (secure 0 0 0) 0x7fca2808c610 con 0x7fca3015c0f0 2024-09-16T01:42:49.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.120+0000 7fca2effd640 1 --2- 172.21.15.88:0/3077829278 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fca1c06c2b0 0x7fca1c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:49.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.120+0000 7fca2effd640 1 --2- 172.21.15.88:0/3077829278 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fca1c06c2b0 0x7fca1c06e770 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fca20034000 tx=0x7fca2002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:49.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.124+0000 7fca177fe640 1 -- 172.21.15.88:0/3077829278 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fca30162ff0 con 0x7fca3015c0f0 2024-09-16T01:42:49.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.323+0000 7fca3575f640 1 -- 172.21.15.88:0/3077829278 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fca30068d10 con 0x7fca3015c0f0 2024-09-16T01:42:49.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.697+0000 7fca177fe640 1 -- 172.21.15.88:0/3077829278 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v226) ==== 162+0+0 (secure 0 0 0) 0x7fca30068d10 con 0x7fca3015c0f0 2024-09-16T01:42:49.699 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:42:49.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.700+0000 7fca3575f640 1 -- 172.21.15.88:0/3077829278 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fca1c06c2b0 msgr2=0x7fca1c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:49.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.700+0000 7fca3575f640 1 --2- 172.21.15.88:0/3077829278 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fca1c06c2b0 0x7fca1c06e770 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fca20034000 tx=0x7fca2002e000 comp rx=0 tx=0).stop 2024-09-16T01:42:49.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.700+0000 7fca3575f640 1 -- 172.21.15.88:0/3077829278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca3015c0f0 msgr2=0x7fca3006fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:49.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.700+0000 7fca3575f640 1 --2- 172.21.15.88:0/3077829278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca3015c0f0 0x7fca3006fb90 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fca2800cd70 tx=0x7fca2800a660 comp rx=0 tx=0).stop 2024-09-16T01:42:49.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.700+0000 7fca3575f640 1 -- 172.21.15.88:0/3077829278 shutdown_connections 2024-09-16T01:42:49.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.700+0000 7fca3575f640 1 --2- 172.21.15.88:0/3077829278 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fca1c06c2b0 0x7fca1c06e770 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:49.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.700+0000 7fca3575f640 1 --2- 172.21.15.88:0/3077829278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca3015c0f0 0x7fca3006fb90 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:49.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.700+0000 7fca3575f640 1 --2- 172.21.15.88:0/3077829278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca30073f10 0x7fca3006f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:49.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.700+0000 7fca3575f640 1 --2- 172.21.15.88:0/3077829278 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca3006c150 0x7fca30150a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:49.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.700+0000 7fca3575f640 1 -- 172.21.15.88:0/3077829278 >> 172.21.15.88:0/3077829278 conn(0x7fca300d1390 msgr2=0x7fca30072da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:49.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.700+0000 7fca3575f640 1 -- 172.21.15.88:0/3077829278 shutdown_connections 2024-09-16T01:42:49.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:49.700+0000 7fca3575f640 1 -- 172.21.15.88:0/3077829278 wait complete. 2024-09-16T01:42:50.117 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:50.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.219+0000 7f538f49b640 1 Processor -- start 2024-09-16T01:42:50.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.219+0000 7f538f49b640 1 -- start start 2024-09-16T01:42:50.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.220+0000 7f538f49b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f538806d540 0x7f53880cf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:50.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.220+0000 7f538f49b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f53880735b0 0x7f53880cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:50.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.220+0000 7f538f49b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f538806faf0 con 0x7f538815e950 2024-09-16T01:42:50.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.220+0000 7f538f49b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f538806fdd0 con 0x7f538806d540 2024-09-16T01:42:50.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.220+0000 7f538f49b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f53880700b0 con 0x7f53880735b0 2024-09-16T01:42:50.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.220+0000 7f538ca36640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f538806d540 0x7f53880cf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:50.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.220+0000 7f538ca36640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f538806d540 0x7f53880cf0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37220/0 (socket says 172.21.15.88:37220) 2024-09-16T01:42:50.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.220+0000 7f538ca36640 1 -- 172.21.15.88:0/735341075 learned_addr learned my addr 172.21.15.88:0/735341075 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:50.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.220+0000 7f538da38640 1 --2- 172.21.15.88:0/735341075 >> v2:172.21.15.154:3300/0 conn(0x7f53880735b0 0x7f53880cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:50.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.220+0000 7f538ca36640 1 -- 172.21.15.88:0/735341075 >> v2:172.21.15.154:3300/0 conn(0x7f53880735b0 msgr2=0x7f53880cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:50.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.220+0000 7f538ca36640 1 --2- 172.21.15.88:0/735341075 >> v2:172.21.15.154:3300/0 conn(0x7f53880735b0 0x7f53880cf5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:50.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.220+0000 7f538ca36640 1 -- 172.21.15.88:0/735341075 >> v1:172.21.15.154:6789/0 conn(0x7f538815e950 legacy=0x7f538806ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:50.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.220+0000 7f538ca36640 1 -- 172.21.15.88:0/735341075 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5388070390 con 0x7f538806d540 2024-09-16T01:42:50.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.221+0000 7f538ca36640 1 --2- 172.21.15.88:0/735341075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f538806d540 0x7f53880cf0a0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f5388071c80 tx=0x7f537802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3fbb63b35069a0b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:50.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.221+0000 7f537ffff640 1 -- 172.21.15.88:0/735341075 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5378037070 con 0x7f538806d540 2024-09-16T01:42:50.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.221+0000 7f537ffff640 1 -- 172.21.15.88:0/735341075 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5378032440 con 0x7f538806d540 2024-09-16T01:42:50.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.221+0000 7f537ffff640 1 -- 172.21.15.88:0/735341075 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5378032810 con 0x7f538806d540 2024-09-16T01:42:50.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.222+0000 7f538f49b640 1 -- 172.21.15.88:0/735341075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f538806d540 msgr2=0x7f53880cf0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:50.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.222+0000 7f538f49b640 1 --2- 172.21.15.88:0/735341075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f538806d540 0x7f53880cf0a0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f5388071c80 tx=0x7f537802a230 comp rx=0 tx=0).stop 2024-09-16T01:42:50.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.222+0000 7f538f49b640 1 -- 172.21.15.88:0/735341075 shutdown_connections 2024-09-16T01:42:50.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.222+0000 7f538f49b640 1 --2- 172.21.15.88:0/735341075 >> v2:172.21.15.154:3300/0 conn(0x7f53880735b0 0x7f53880cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:50.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.222+0000 7f538f49b640 1 --2- 172.21.15.88:0/735341075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f538806d540 0x7f53880cf0a0 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:50.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.222+0000 7f538f49b640 1 -- 172.21.15.88:0/735341075 >> 172.21.15.88:0/735341075 conn(0x7f53880d14d0 msgr2=0x7f53880d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:50.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.222+0000 7f538f49b640 1 -- 172.21.15.88:0/735341075 shutdown_connections 2024-09-16T01:42:50.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.222+0000 7f538f49b640 1 -- 172.21.15.88:0/735341075 wait complete. 2024-09-16T01:42:50.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.223+0000 7f538f49b640 1 Processor -- start 2024-09-16T01:42:50.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.223+0000 7f538f49b640 1 -- start start 2024-09-16T01:42:50.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.224+0000 7f538f49b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f538806d540 0x7f53881540d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:50.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.224+0000 7f538f49b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53880735b0 0x7f53881575c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:50.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.224+0000 7f538f49b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f538815e950 0x7f5388157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:50.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.224+0000 7f538f49b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f53880721d0 con 0x7f53880735b0 2024-09-16T01:42:50.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.224+0000 7f538f49b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5388072050 con 0x7f538806d540 2024-09-16T01:42:50.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.224+0000 7f538f49b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5388072350 con 0x7f538815e950 2024-09-16T01:42:50.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.224+0000 7f538ca36640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53880735b0 0x7f53881575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:50.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.224+0000 7f538ca36640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53880735b0 0x7f53881575c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:37234/0 (socket says 172.21.15.88:37234) 2024-09-16T01:42:50.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.224+0000 7f538ca36640 1 -- 172.21.15.88:0/2406449741 learned_addr learned my addr 172.21.15.88:0/2406449741 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:50.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.224+0000 7f538d237640 1 --2- 172.21.15.88:0/2406449741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f538806d540 0x7f53881540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:50.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.224+0000 7f538da38640 1 --2- 172.21.15.88:0/2406449741 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f538815e950 0x7f5388157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:50.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.224+0000 7f538ca36640 1 -- 172.21.15.88:0/2406449741 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f538815e950 msgr2=0x7f5388157b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:50.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.224+0000 7f538ca36640 1 --2- 172.21.15.88:0/2406449741 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f538815e950 0x7f5388157b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:50.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.224+0000 7f538ca36640 1 -- 172.21.15.88:0/2406449741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f538806d540 msgr2=0x7f53881540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:50.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.224+0000 7f538ca36640 1 --2- 172.21.15.88:0/2406449741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f538806d540 0x7f53881540d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:50.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.224+0000 7f538ca36640 1 -- 172.21.15.88:0/2406449741 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5388158360 con 0x7f53880735b0 2024-09-16T01:42:50.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.224+0000 7f538d237640 1 --2- 172.21.15.88:0/2406449741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f538806d540 0x7f53881540d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:50.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.224+0000 7f538ca36640 1 --2- 172.21.15.88:0/2406449741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53880735b0 0x7f53881575c0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f5378000c00 tx=0x7f53780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:50.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.224+0000 7f537dffb640 1 -- 172.21.15.88:0/2406449741 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5378041070 con 0x7f53880735b0 2024-09-16T01:42:50.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.224+0000 7f537dffb640 1 -- 172.21.15.88:0/2406449741 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f537803c870 con 0x7f53880735b0 2024-09-16T01:42:50.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.224+0000 7f538f49b640 1 -- 172.21.15.88:0/2406449741 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5388158620 con 0x7f53880735b0 2024-09-16T01:42:50.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.224+0000 7f537dffb640 1 -- 172.21.15.88:0/2406449741 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5378037040 con 0x7f53880735b0 2024-09-16T01:42:50.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.225+0000 7f538f49b640 1 -- 172.21.15.88:0/2406449741 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5388158be0 con 0x7f53880735b0 2024-09-16T01:42:50.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.225+0000 7f538f49b640 1 -- 172.21.15.88:0/2406449741 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f538806ce30 con 0x7f53880735b0 2024-09-16T01:42:50.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.226+0000 7f537dffb640 1 -- 172.21.15.88:0/2406449741 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5378044070 con 0x7f53880735b0 2024-09-16T01:42:50.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.226+0000 7f537dffb640 1 --2- 172.21.15.88:0/2406449741 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f536c06c2b0 0x7f536c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:50.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.226+0000 7f537dffb640 1 -- 172.21.15.88:0/2406449741 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 7974+0+0 (secure 0 0 0) 0x7f53780ac970 con 0x7f53880735b0 2024-09-16T01:42:50.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.226+0000 7f538d237640 1 --2- 172.21.15.88:0/2406449741 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f536c06c2b0 0x7f536c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:50.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.227+0000 7f538d237640 1 --2- 172.21.15.88:0/2406449741 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f536c06c2b0 0x7f536c06e770 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f5384003eb0 tx=0x7f5384009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:50.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.231+0000 7f537dffb640 1 -- 172.21.15.88:0/2406449741 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f537807a920 con 0x7f53880735b0 2024-09-16T01:42:50.425 INFO:tasks.ceph.mds.c.smithi088.stderr: -38> 2024-09-16T01:42:45.672+0000 7f24e0bcc640 -1 log_channel(cluster) log [ERR] : missing journal object 2024-09-16T01:42:50.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.431+0000 7f538f49b640 1 -- 172.21.15.88:0/2406449741 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53880d0560 con 0x7f53880735b0 2024-09-16T01:42:50.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.431+0000 7f537dffb640 1 -- 172.21.15.88:0/2406449741 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) ==== 95+0+4401 (secure 0 0 0) 0x7f537807a220 con 0x7f53880735b0 2024-09-16T01:42:50.433 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:50.433 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":227,"btime":"2024-09-16T01:42:49:700502+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:49.700493+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9276":{"gid":9276,"name":"a","rank":0,"incarnation":227,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6836/4065900771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4065900771},{"type":"v1","addr":"172.21.15.88:6836","nonce":4065900771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:42:50.433 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 227 2024-09-16T01:42:50.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.434+0000 7f538f49b640 1 -- 172.21.15.88:0/2406449741 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f536c06c2b0 msgr2=0x7f536c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:50.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.434+0000 7f538f49b640 1 --2- 172.21.15.88:0/2406449741 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f536c06c2b0 0x7f536c06e770 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f5384003eb0 tx=0x7f5384009040 comp rx=0 tx=0).stop 2024-09-16T01:42:50.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.434+0000 7f538f49b640 1 -- 172.21.15.88:0/2406449741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53880735b0 msgr2=0x7f53881575c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:50.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.434+0000 7f538f49b640 1 --2- 172.21.15.88:0/2406449741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53880735b0 0x7f53881575c0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f5378000c00 tx=0x7f53780046c0 comp rx=0 tx=0).stop 2024-09-16T01:42:50.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.434+0000 7f538f49b640 1 -- 172.21.15.88:0/2406449741 shutdown_connections 2024-09-16T01:42:50.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.434+0000 7f538f49b640 1 --2- 172.21.15.88:0/2406449741 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f536c06c2b0 0x7f536c06e770 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:50.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.434+0000 7f538f49b640 1 --2- 172.21.15.88:0/2406449741 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f538815e950 0x7f5388157b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:50.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.434+0000 7f538f49b640 1 --2- 172.21.15.88:0/2406449741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53880735b0 0x7f53881575c0 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:50.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.434+0000 7f538f49b640 1 --2- 172.21.15.88:0/2406449741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f538806d540 0x7f53881540d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:50.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.434+0000 7f538f49b640 1 -- 172.21.15.88:0/2406449741 >> 172.21.15.88:0/2406449741 conn(0x7f53880d14d0 msgr2=0x7f53880d0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:50.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.434+0000 7f538f49b640 1 -- 172.21.15.88:0/2406449741 shutdown_connections 2024-09-16T01:42:50.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:50.434+0000 7f538f49b640 1 -- 172.21.15.88:0/2406449741 wait complete. 2024-09-16T01:42:50.446 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 227, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:40:03.597796+0000', 'modified': '2024-09-16T01:42:49.700493+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 113, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9276}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9276': {'gid': 9276, 'name': 'a', 'rank': 0, 'incarnation': 227, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.88:6836/4065900771', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 4065900771}, {'type': 'v1', 'addr': '172.21.15.88:6836', 'nonce': 4065900771}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-16T01:42:50.446 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9276:up:replay 2024-09-16T01:42:50.447 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-09-16T01:42:50.448 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:42:55.851 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:42:55.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.962+0000 7fe57960e640 1 Processor -- start 2024-09-16T01:42:55.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.962+0000 7fe57960e640 1 -- start start 2024-09-16T01:42:55.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.963+0000 7fe57960e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe57415e660 0x7fe5740cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:55.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.963+0000 7fe57960e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe57415ea40 con 0x7fe5740d3310 2024-09-16T01:42:55.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.963+0000 7fe57960e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5740d3870 con 0x7fe5740d1820 2024-09-16T01:42:55.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.963+0000 7fe57960e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe5740d3b50 con 0x7fe57415e660 2024-09-16T01:42:55.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.963+0000 7fe5727fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe5740d3310 0x7fe5740d1110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42460/0 (socket says 172.21.15.88:42460) 2024-09-16T01:42:55.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.963+0000 7fe5727fc640 1 -- 172.21.15.88:0/1476633619 learned_addr learned my addr 172.21.15.88:0/1476633619 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:55.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.963+0000 7fe571ffb640 1 -- 172.21.15.88:0/1476633619 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1717540410 0 0) 0x7fe57415ea40 con 0x7fe5740d3310 2024-09-16T01:42:55.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.963+0000 7fe571ffb640 1 -- 172.21.15.88:0/1476633619 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe55c003520 con 0x7fe5740d3310 2024-09-16T01:42:55.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.963+0000 7fe572ffd640 1 --2- 172.21.15.88:0/1476633619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe57415e660 0x7fe5740cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:55.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.964+0000 7fe571ffb640 1 -- 172.21.15.88:0/1476633619 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2691978728 0 0) 0x7fe55c003520 con 0x7fe5740d3310 2024-09-16T01:42:55.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.964+0000 7fe571ffb640 1 -- 172.21.15.88:0/1476633619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe57415e660 msgr2=0x7fe5740cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:55.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.964+0000 7fe571ffb640 1 --2- 172.21.15.88:0/1476633619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe57415e660 0x7fe5740cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:55.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.964+0000 7fe571ffb640 1 -- 172.21.15.88:0/1476633619 >> v1:172.21.15.154:6789/0 conn(0x7fe5740d1820 legacy=0x7fe5740b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:55.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.964+0000 7fe571ffb640 1 -- 172.21.15.88:0/1476633619 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5740d3e30 con 0x7fe5740d3310 2024-09-16T01:42:55.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.964+0000 7fe571ffb640 1 -- 172.21.15.88:0/1476633619 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fe568001670 con 0x7fe5740d3310 2024-09-16T01:42:55.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.964+0000 7fe571ffb640 1 -- 172.21.15.88:0/1476633619 >> v1:172.21.15.88:6789/0 conn(0x7fe5740d3310 legacy=0x7fe5740d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:55.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.964+0000 7fe571ffb640 1 --2- 172.21.15.88:0/1476633619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe55c0039e0 0x7fe55c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:55.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.964+0000 7fe571ffb640 1 --2- 172.21.15.88:0/1476633619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe57415e660 0x7fe55c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:55.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.964+0000 7fe571ffb640 1 --2- 172.21.15.88:0/1476633619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe55c004d40 0x7fe55c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:55.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.964+0000 7fe571ffb640 1 -- 172.21.15.88:0/1476633619 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe5740d3b50 con 0x7fe55c0039e0 2024-09-16T01:42:55.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.964+0000 7fe571ffb640 1 -- 172.21.15.88:0/1476633619 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe55c00ba70 con 0x7fe55c004d40 2024-09-16T01:42:55.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.964+0000 7fe571ffb640 1 -- 172.21.15.88:0/1476633619 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe55c00bc70 con 0x7fe57415e660 2024-09-16T01:42:55.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.964+0000 7fe572ffd640 1 --2- 172.21.15.88:0/1476633619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe55c0039e0 0x7fe55c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:55.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.964+0000 7fe5727fc640 1 --2- 172.21.15.88:0/1476633619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe57415e660 0x7fe55c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:55.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.964+0000 7fe5737fe640 1 --2- 172.21.15.88:0/1476633619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe55c004d40 0x7fe55c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:55.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.964+0000 7fe572ffd640 1 -- 172.21.15.88:0/1476633619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe57415e660 msgr2=0x7fe55c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:55.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.964+0000 7fe572ffd640 1 --2- 172.21.15.88:0/1476633619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe57415e660 0x7fe55c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:55.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.964+0000 7fe572ffd640 1 -- 172.21.15.88:0/1476633619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe55c004d40 msgr2=0x7fe55c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:55.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.964+0000 7fe572ffd640 1 --2- 172.21.15.88:0/1476633619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe55c004d40 0x7fe55c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:55.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.964+0000 7fe572ffd640 1 -- 172.21.15.88:0/1476633619 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe55c003520 con 0x7fe55c0039e0 2024-09-16T01:42:55.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.964+0000 7fe5737fe640 1 --2- 172.21.15.88:0/1476633619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe55c004d40 0x7fe55c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:55.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.965+0000 7fe572ffd640 1 --2- 172.21.15.88:0/1476633619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe55c0039e0 0x7fe55c003e00 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7fe564009f90 tx=0x7fe55c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6549fe7c24d5d67 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:55.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.965+0000 7fe571ffb640 1 -- 172.21.15.88:0/1476633619 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe564009c10 con 0x7fe55c0039e0 2024-09-16T01:42:55.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.965+0000 7fe571ffb640 1 -- 172.21.15.88:0/1476633619 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe564036070 con 0x7fe55c0039e0 2024-09-16T01:42:55.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.965+0000 7fe571ffb640 1 -- 172.21.15.88:0/1476633619 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe564031850 con 0x7fe55c0039e0 2024-09-16T01:42:55.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.965+0000 7fe57960e640 1 -- 172.21.15.88:0/1476633619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe55c0039e0 msgr2=0x7fe55c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:55.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.965+0000 7fe57960e640 1 --2- 172.21.15.88:0/1476633619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe55c0039e0 0x7fe55c003e00 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7fe564009f90 tx=0x7fe55c005830 comp rx=0 tx=0).stop 2024-09-16T01:42:55.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.966+0000 7fe5737fe640 1 -- 172.21.15.88:0/1476633619 reap_dead start 2024-09-16T01:42:55.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.966+0000 7fe57960e640 1 -- 172.21.15.88:0/1476633619 shutdown_connections 2024-09-16T01:42:55.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.966+0000 7fe57960e640 1 -- 172.21.15.88:0/1476633619 >> 172.21.15.88:0/1476633619 conn(0x7fe5740b6410 msgr2=0x7fe5740b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:55.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.966+0000 7fe57960e640 1 -- 172.21.15.88:0/1476633619 shutdown_connections 2024-09-16T01:42:55.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.966+0000 7fe57960e640 1 -- 172.21.15.88:0/1476633619 wait complete. 2024-09-16T01:42:55.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.967+0000 7fe57960e640 1 Processor -- start 2024-09-16T01:42:55.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.967+0000 7fe57960e640 1 -- start start 2024-09-16T01:42:55.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.968+0000 7fe57960e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe55c0039e0 0x7fe574155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:55.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.968+0000 7fe57960e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe574160660 0x7fe574157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:55.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.968+0000 7fe57960e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe574157e80 0x7fe574158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:55.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.968+0000 7fe57960e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe5740cf730 con 0x7fe55c0039e0 2024-09-16T01:42:55.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.968+0000 7fe57960e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe5740d43f0 con 0x7fe574160660 2024-09-16T01:42:55.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.968+0000 7fe57960e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe5740cf8b0 con 0x7fe574157e80 2024-09-16T01:42:55.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.968+0000 7fe572ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe55c0039e0 0x7fe574155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:55.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.968+0000 7fe572ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe55c0039e0 0x7fe574155db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35022/0 (socket says 172.21.15.88:35022) 2024-09-16T01:42:55.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.968+0000 7fe572ffd640 1 -- 172.21.15.88:0/2489525574 learned_addr learned my addr 172.21.15.88:0/2489525574 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:55.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.968+0000 7fe5727fc640 1 --2- 172.21.15.88:0/2489525574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe574160660 0x7fe574157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:55.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.968+0000 7fe5737fe640 1 --2- 172.21.15.88:0/2489525574 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe574157e80 0x7fe574158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:55.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.968+0000 7fe572ffd640 1 -- 172.21.15.88:0/2489525574 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe574157e80 msgr2=0x7fe574158300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:55.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.968+0000 7fe572ffd640 1 --2- 172.21.15.88:0/2489525574 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe574157e80 0x7fe574158300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:55.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.968+0000 7fe572ffd640 1 -- 172.21.15.88:0/2489525574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe574160660 msgr2=0x7fe574157940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:55.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.968+0000 7fe572ffd640 1 --2- 172.21.15.88:0/2489525574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe574160660 0x7fe574157940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:55.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.968+0000 7fe572ffd640 1 -- 172.21.15.88:0/2489525574 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe574158840 con 0x7fe55c0039e0 2024-09-16T01:42:55.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.968+0000 7fe5737fe640 1 --2- 172.21.15.88:0/2489525574 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe574157e80 0x7fe574158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:55.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.968+0000 7fe572ffd640 1 --2- 172.21.15.88:0/2489525574 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe55c0039e0 0x7fe574155db0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fe56400e230 tx=0x7fe564009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:55.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.968+0000 7fe55b7fe640 1 -- 172.21.15.88:0/2489525574 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe564009c10 con 0x7fe55c0039e0 2024-09-16T01:42:55.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.968+0000 7fe55b7fe640 1 -- 172.21.15.88:0/2489525574 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe564045650 con 0x7fe55c0039e0 2024-09-16T01:42:55.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.968+0000 7fe57960e640 1 -- 172.21.15.88:0/2489525574 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe574160bb0 con 0x7fe55c0039e0 2024-09-16T01:42:55.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.969+0000 7fe55b7fe640 1 -- 172.21.15.88:0/2489525574 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe564045a40 con 0x7fe55c0039e0 2024-09-16T01:42:55.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.969+0000 7fe57960e640 1 -- 172.21.15.88:0/2489525574 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe574161170 con 0x7fe55c0039e0 2024-09-16T01:42:55.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.969+0000 7fe57960e640 1 -- 172.21.15.88:0/2489525574 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5740cfcc0 con 0x7fe55c0039e0 2024-09-16T01:42:55.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.970+0000 7fe55b7fe640 1 -- 172.21.15.88:0/2489525574 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe564005d10 con 0x7fe55c0039e0 2024-09-16T01:42:55.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.971+0000 7fe55b7fe640 1 --2- 172.21.15.88:0/2489525574 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe56006c2b0 0x7fe56006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:55.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.971+0000 7fe5727fc640 1 --2- 172.21.15.88:0/2489525574 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe56006c2b0 0x7fe56006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:55.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.971+0000 7fe55b7fe640 1 -- 172.21.15.88:0/2489525574 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 7974+0+0 (secure 0 0 0) 0x7fe5640ad830 con 0x7fe55c0039e0 2024-09-16T01:42:55.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.971+0000 7fe5727fc640 1 --2- 172.21.15.88:0/2489525574 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe56006c2b0 0x7fe56006e770 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fe5741619e0 tx=0x7fe568009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:55.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:55.975+0000 7fe55b7fe640 1 -- 172.21.15.88:0/2489525574 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe56402e050 con 0x7fe55c0039e0 2024-09-16T01:42:56.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.174+0000 7fe57960e640 1 -- 172.21.15.88:0/2489525574 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5741598b0 con 0x7fe55c0039e0 2024-09-16T01:42:56.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.174+0000 7fe55b7fe640 1 -- 172.21.15.88:0/2489525574 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) ==== 95+0+5234 (secure 0 0 0) 0x7fe56407b6c0 con 0x7fe55c0039e0 2024-09-16T01:42:56.176 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:56.176 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":230,"btime":"2024-09-16T01:42:52:722317+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9333,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2477051134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2477051134},{"type":"v1","addr":"172.21.15.88:6837","nonce":2477051134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":230,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:42:52.722311+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9276},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9276":{"gid":9276,"name":"a","rank":0,"incarnation":227,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6836/4065900771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4065900771},{"type":"v1","addr":"172.21.15.88:6836","nonce":4065900771}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9276,"qdb_cluster":[9276]},"id":6}]} 2024-09-16T01:42:56.176 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 230 2024-09-16T01:42:56.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.176+0000 7fe57960e640 1 -- 172.21.15.88:0/2489525574 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe56006c2b0 msgr2=0x7fe56006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:56.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.176+0000 7fe57960e640 1 --2- 172.21.15.88:0/2489525574 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe56006c2b0 0x7fe56006e770 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fe5741619e0 tx=0x7fe568009040 comp rx=0 tx=0).stop 2024-09-16T01:42:56.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.176+0000 7fe57960e640 1 -- 172.21.15.88:0/2489525574 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe55c0039e0 msgr2=0x7fe574155db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:56.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.176+0000 7fe57960e640 1 --2- 172.21.15.88:0/2489525574 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe55c0039e0 0x7fe574155db0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fe56400e230 tx=0x7fe564009750 comp rx=0 tx=0).stop 2024-09-16T01:42:56.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.176+0000 7fe57960e640 1 -- 172.21.15.88:0/2489525574 shutdown_connections 2024-09-16T01:42:56.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.176+0000 7fe57960e640 1 --2- 172.21.15.88:0/2489525574 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe56006c2b0 0x7fe56006e770 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.176+0000 7fe57960e640 1 --2- 172.21.15.88:0/2489525574 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe574157e80 0x7fe574158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.176+0000 7fe57960e640 1 --2- 172.21.15.88:0/2489525574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe574160660 0x7fe574157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.176+0000 7fe57960e640 1 --2- 172.21.15.88:0/2489525574 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe55c0039e0 0x7fe574155db0 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.176+0000 7fe57960e640 1 -- 172.21.15.88:0/2489525574 >> 172.21.15.88:0/2489525574 conn(0x7fe5740b6410 msgr2=0x7fe5740b5990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:56.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.176+0000 7fe57960e640 1 -- 172.21.15.88:0/2489525574 shutdown_connections 2024-09-16T01:42:56.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.176+0000 7fe57960e640 1 -- 172.21.15.88:0/2489525574 wait complete. 2024-09-16T01:42:56.189 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 230, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:40:03.597796+0000', 'modified': '2024-09-16T01:42:52.722311+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 113, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9276}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9276': {'gid': 9276, 'name': 'a', 'rank': 0, 'incarnation': 227, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.88:6836/4065900771', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 4065900771}, {'type': 'v1', 'addr': '172.21.15.88:6836', 'nonce': 4065900771}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9276, 'qdb_cluster': [9276]} 2024-09-16T01:42:56.189 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T01:42:56.189 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-16T01:42:56.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.287+0000 7f5ac743c640 1 Processor -- start 2024-09-16T01:42:56.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.287+0000 7f5ac743c640 1 -- start start 2024-09-16T01:42:56.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.288+0000 7f5ac743c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5ac015b480 0x7f5ac00b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:56.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.288+0000 7f5ac743c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5ac00b60c0 0x7f5ac0071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:56.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.288+0000 7f5ac743c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ac00ccff0 con 0x7f5ac0072530 2024-09-16T01:42:56.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.288+0000 7f5ac743c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f5ac0072980 con 0x7f5ac00b60c0 2024-09-16T01:42:56.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.288+0000 7f5ac743c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5ac00cd2d0 con 0x7f5ac015b480 2024-09-16T01:42:56.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.288+0000 7f5ac49d7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5ac00b60c0 0x7f5ac0071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:56.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.288+0000 7f5ac49d7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5ac00b60c0 0x7f5ac0071ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35034/0 (socket says 172.21.15.88:35034) 2024-09-16T01:42:56.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.288+0000 7f5ac49d7640 1 -- 172.21.15.88:0/3067302442 learned_addr learned my addr 172.21.15.88:0/3067302442 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:56.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.288+0000 7f5ac51d8640 1 --2- 172.21.15.88:0/3067302442 >> v2:172.21.15.154:3300/0 conn(0x7f5ac015b480 0x7f5ac00b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:56.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.288+0000 7f5ac49d7640 1 -- 172.21.15.88:0/3067302442 >> v2:172.21.15.154:3300/0 conn(0x7f5ac015b480 msgr2=0x7f5ac00b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:56.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.288+0000 7f5ac49d7640 1 --2- 172.21.15.88:0/3067302442 >> v2:172.21.15.154:3300/0 conn(0x7f5ac015b480 0x7f5ac00b5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.288+0000 7f5ac49d7640 1 -- 172.21.15.88:0/3067302442 >> v1:172.21.15.154:6789/0 conn(0x7f5ac0072530 legacy=0x7f5ac0070fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:56.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.288+0000 7f5ac49d7640 1 -- 172.21.15.88:0/3067302442 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac00cd5b0 con 0x7f5ac00b60c0 2024-09-16T01:42:56.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.288+0000 7f5ac51d8640 1 --2- 172.21.15.88:0/3067302442 >> v2:172.21.15.154:3300/0 conn(0x7f5ac015b480 0x7f5ac00b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:56.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.289+0000 7f5ac49d7640 1 --2- 172.21.15.88:0/3067302442 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ac00b60c0 0x7f5ac0071ff0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f5ac00694d0 tx=0x7f5ab002c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be189dc50bdc7bc6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:56.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.289+0000 7f5ab7fff640 1 -- 172.21.15.88:0/3067302442 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab00325a0 con 0x7f5ac00b60c0 2024-09-16T01:42:56.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.289+0000 7f5ab7fff640 1 -- 172.21.15.88:0/3067302442 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ab0038070 con 0x7f5ac00b60c0 2024-09-16T01:42:56.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.289+0000 7f5ab7fff640 1 -- 172.21.15.88:0/3067302442 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab0032960 con 0x7f5ac00b60c0 2024-09-16T01:42:56.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.290+0000 7f5ac743c640 1 -- 172.21.15.88:0/3067302442 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ac00b60c0 msgr2=0x7f5ac0071ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:56.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.290+0000 7f5ac743c640 1 --2- 172.21.15.88:0/3067302442 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ac00b60c0 0x7f5ac0071ff0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f5ac00694d0 tx=0x7f5ab002c8e0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.290+0000 7f5ac743c640 1 -- 172.21.15.88:0/3067302442 shutdown_connections 2024-09-16T01:42:56.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.290+0000 7f5ac743c640 1 --2- 172.21.15.88:0/3067302442 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ac00b60c0 0x7f5ac0071ff0 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.290+0000 7f5ac743c640 1 --2- 172.21.15.88:0/3067302442 >> v2:172.21.15.154:3300/0 conn(0x7f5ac015b480 0x7f5ac00b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.290+0000 7f5ac743c640 1 -- 172.21.15.88:0/3067302442 >> 172.21.15.88:0/3067302442 conn(0x7f5ac006d0f0 msgr2=0x7f5ac006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:56.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.290+0000 7f5ac743c640 1 -- 172.21.15.88:0/3067302442 shutdown_connections 2024-09-16T01:42:56.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.290+0000 7f5ac743c640 1 -- 172.21.15.88:0/3067302442 wait complete. 2024-09-16T01:42:56.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.291+0000 7f5ac743c640 1 Processor -- start 2024-09-16T01:42:56.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.291+0000 7f5ac743c640 1 -- start start 2024-09-16T01:42:56.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.292+0000 7f5ac743c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ac0072530 0x7f5ac00b3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:56.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.292+0000 7f5ac743c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ac00b60c0 0x7f5ac00c9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:56.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.292+0000 7f5ac743c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ac015b480 0x7f5ac00c7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:56.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.292+0000 7f5ac743c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac015b860 con 0x7f5ac00b60c0 2024-09-16T01:42:56.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.292+0000 7f5ac743c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac00cdb70 con 0x7f5ac0072530 2024-09-16T01:42:56.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.292+0000 7f5ac743c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5ac015b9e0 con 0x7f5ac015b480 2024-09-16T01:42:56.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.292+0000 7f5ac49d7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ac00b60c0 0x7f5ac00c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:56.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.292+0000 7f5ac49d7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ac00b60c0 0x7f5ac00c9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35042/0 (socket says 172.21.15.88:35042) 2024-09-16T01:42:56.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.292+0000 7f5ac49d7640 1 -- 172.21.15.88:0/1434057301 learned_addr learned my addr 172.21.15.88:0/1434057301 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:56.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.292+0000 7f5ac51d8640 1 --2- 172.21.15.88:0/1434057301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ac0072530 0x7f5ac00b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:56.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.292+0000 7f5ac59d9640 1 --2- 172.21.15.88:0/1434057301 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ac015b480 0x7f5ac00c7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:56.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.292+0000 7f5ac49d7640 1 -- 172.21.15.88:0/1434057301 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ac015b480 msgr2=0x7f5ac00c7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:56.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.292+0000 7f5ac49d7640 1 --2- 172.21.15.88:0/1434057301 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ac015b480 0x7f5ac00c7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.292+0000 7f5ac49d7640 1 -- 172.21.15.88:0/1434057301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ac0072530 msgr2=0x7f5ac00b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:56.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.292+0000 7f5ac49d7640 1 --2- 172.21.15.88:0/1434057301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ac0072530 0x7f5ac00b3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.292+0000 7f5ac49d7640 1 -- 172.21.15.88:0/1434057301 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac00b3b90 con 0x7f5ac00b60c0 2024-09-16T01:42:56.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.292+0000 7f5ac49d7640 1 --2- 172.21.15.88:0/1434057301 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ac00b60c0 0x7f5ac00c9950 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0000c00 tx=0x7f5ab00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:56.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.292+0000 7f5ab5ffb640 1 -- 172.21.15.88:0/1434057301 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab0033840 con 0x7f5ac00b60c0 2024-09-16T01:42:56.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.292+0000 7f5ab5ffb640 1 -- 172.21.15.88:0/1434057301 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ab0033a00 con 0x7f5ac00b60c0 2024-09-16T01:42:56.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.293+0000 7f5ac743c640 1 -- 172.21.15.88:0/1434057301 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ac00b3e70 con 0x7f5ac00b60c0 2024-09-16T01:42:56.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.293+0000 7f5ab5ffb640 1 -- 172.21.15.88:0/1434057301 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab0033df0 con 0x7f5ac00b60c0 2024-09-16T01:42:56.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.293+0000 7f5ac743c640 1 -- 172.21.15.88:0/1434057301 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ac00b4430 con 0x7f5ac00b60c0 2024-09-16T01:42:56.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.294+0000 7f5ab5ffb640 1 -- 172.21.15.88:0/1434057301 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5ab003c3f0 con 0x7f5ac00b60c0 2024-09-16T01:42:56.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.295+0000 7f5ab5ffb640 1 --2- 172.21.15.88:0/1434057301 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5aa406c2b0 0x7f5aa406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:56.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.295+0000 7f5ab5ffb640 1 -- 172.21.15.88:0/1434057301 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 7974+0+0 (secure 0 0 0) 0x7f5ab0030030 con 0x7f5ac00b60c0 2024-09-16T01:42:56.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.295+0000 7f5ac51d8640 1 --2- 172.21.15.88:0/1434057301 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5aa406c2b0 0x7f5aa406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:56.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.295+0000 7f5ac51d8640 1 --2- 172.21.15.88:0/1434057301 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5aa406c2b0 0x7f5aa406e770 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0156d80 tx=0x7f5abc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:56.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.295+0000 7f5aca1d5740 1 Processor -- start 2024-09-16T01:42:56.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.296+0000 7f5aca1d5740 1 -- start start 2024-09-16T01:42:56.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.296+0000 7f5aca1d5740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ebfb2c0 0x56051ec40a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:56.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.296+0000 7f5aca1d5740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56051ebf8a80 0x56051ebfb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:56.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.296+0000 7f5aca1d5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56051ebfca50 0x56051ebf58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:56.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.296+0000 7f5aca1d5740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56051eb5d520 con 0x56051ebfb2c0 2024-09-16T01:42:56.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.296+0000 7f5aca1d5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56051eb053e0 con 0x56051ebfca50 2024-09-16T01:42:56.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.296+0000 7f5aca1d5740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56051eb594e0 con 0x56051ebf8a80 2024-09-16T01:42:56.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.296+0000 7f5ac51d8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ebfb2c0 0x56051ec40a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:56.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.296+0000 7f5ac51d8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ebfb2c0 0x56051ec40a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35044/0 (socket says 172.21.15.88:35044) 2024-09-16T01:42:56.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.296+0000 7f5ac51d8640 1 -- 172.21.15.88:0/1800412733 learned_addr learned my addr 172.21.15.88:0/1800412733 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:56.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.296+0000 7f5ac49d7640 1 --2- 172.21.15.88:0/1800412733 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56051ebf8a80 0x56051ebfb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:56.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.296+0000 7f5ac59d9640 1 --2- 172.21.15.88:0/1800412733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56051ebfca50 0x56051ebf58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:56.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.297+0000 7f5ac51d8640 1 -- 172.21.15.88:0/1800412733 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56051ebf8a80 msgr2=0x56051ebfb6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:56.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.297+0000 7f5ac51d8640 1 --2- 172.21.15.88:0/1800412733 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56051ebf8a80 0x56051ebfb6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.297+0000 7f5ac51d8640 1 -- 172.21.15.88:0/1800412733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56051ebfca50 msgr2=0x56051ebf58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:56.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.297+0000 7f5ac51d8640 1 --2- 172.21.15.88:0/1800412733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56051ebfca50 0x56051ebf58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.297+0000 7f5ac51d8640 1 -- 172.21.15.88:0/1800412733 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56051eb202f0 con 0x56051ebfb2c0 2024-09-16T01:42:56.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.297+0000 7f5ac51d8640 1 --2- 172.21.15.88:0/1800412733 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ebfb2c0 0x56051ec40a60 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f5abc00fd20 tx=0x7f5abc012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d8e722eaf9bd107 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:56.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.297+0000 7f5aaaffd640 1 -- 172.21.15.88:0/1800412733 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5abc023070 con 0x56051ebfb2c0 2024-09-16T01:42:56.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.297+0000 7f5aaaffd640 1 -- 172.21.15.88:0/1800412733 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5abc019070 con 0x56051ebfb2c0 2024-09-16T01:42:56.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.297+0000 7f5aaaffd640 1 -- 172.21.15.88:0/1800412733 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5abc013820 con 0x56051ebfb2c0 2024-09-16T01:42:56.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.297+0000 7f5aca1d5740 1 -- 172.21.15.88:0/1800412733 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ebfb2c0 msgr2=0x56051ec40a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:56.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.297+0000 7f5aca1d5740 1 --2- 172.21.15.88:0/1800412733 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ebfb2c0 0x56051ec40a60 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f5abc00fd20 tx=0x7f5abc012000 comp rx=0 tx=0).stop 2024-09-16T01:42:56.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.298+0000 7f5aca1d5740 1 -- 172.21.15.88:0/1800412733 shutdown_connections 2024-09-16T01:42:56.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.298+0000 7f5aca1d5740 1 --2- 172.21.15.88:0/1800412733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56051ebfca50 0x56051ebf58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.298+0000 7f5aca1d5740 1 --2- 172.21.15.88:0/1800412733 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56051ebf8a80 0x56051ebfb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.298+0000 7f5aca1d5740 1 --2- 172.21.15.88:0/1800412733 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ebfb2c0 0x56051ec40a60 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.298+0000 7f5aca1d5740 1 -- 172.21.15.88:0/1800412733 >> 172.21.15.88:0/1800412733 conn(0x56051ebf7a80 msgr2=0x56051ec3ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:56.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.298+0000 7f5aca1d5740 1 -- 172.21.15.88:0/1800412733 shutdown_connections 2024-09-16T01:42:56.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.298+0000 7f5aca1d5740 1 -- 172.21.15.88:0/1800412733 wait complete. 2024-09-16T01:42:56.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.298+0000 7f5aca1d5740 1 -- start start 2024-09-16T01:42:56.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.298+0000 7f5aca1d5740 1 Processor -- start 2024-09-16T01:42:56.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.299+0000 7f5aca1d5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56051ebfb2c0 0x56051ec548d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:56.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.299+0000 7f5aca1d5740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56051ebfca50 0x56051ec3d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:56.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.299+0000 7f5aca1d5740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ec507a0 0x56051ec24c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:56.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.299+0000 7f5aca1d5740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56051ebf6f20 con 0x56051ec507a0 2024-09-16T01:42:56.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.299+0000 7f5aca1d5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56051eae6450 con 0x56051ebfb2c0 2024-09-16T01:42:56.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.299+0000 7f5aca1d5740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56051ec41290 con 0x56051ebfca50 2024-09-16T01:42:56.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.299+0000 7f5aca1d5740 10 client.0 _ll_register_callbacks cb 0x56051e86fac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:42:56.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.299+0000 7f5ac59d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ec507a0 0x56051ec24c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:56.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.299+0000 7f5ac59d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ec507a0 0x56051ec24c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35048/0 (socket says 172.21.15.88:35048) 2024-09-16T01:42:56.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.299+0000 7f5ac59d9640 1 -- 172.21.15.88:0/1864929568 learned_addr learned my addr 172.21.15.88:0/1864929568 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:56.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.299+0000 7f5ac49d7640 1 --2- 172.21.15.88:0/1864929568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56051ebfca50 0x56051ec3d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:56.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.299+0000 7f5ac59d9640 1 -- 172.21.15.88:0/1864929568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56051ebfca50 msgr2=0x56051ec3d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:56.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.299+0000 7f5ac59d9640 1 --2- 172.21.15.88:0/1864929568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56051ebfca50 0x56051ec3d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.299+0000 7f5ac59d9640 1 -- 172.21.15.88:0/1864929568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56051ebfb2c0 msgr2=0x56051ec548d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.299+0000 7f5ac59d9640 1 --2- 172.21.15.88:0/1864929568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56051ebfb2c0 0x56051ec548d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.299+0000 7f5ac59d9640 1 -- 172.21.15.88:0/1864929568 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56051e8702d0 con 0x56051ec507a0 2024-09-16T01:42:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.300+0000 7f5ac59d9640 1 --2- 172.21.15.88:0/1864929568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ec507a0 0x56051ec24c40 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80118d0 tx=0x7f5ab8011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab8e802202c52cfe server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.300+0000 7f5a9ffff640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:42:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.300+0000 7f5a9ffff640 1 -- 172.21.15.88:0/1864929568 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab801b020 con 0x56051ec507a0 2024-09-16T01:42:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.300+0000 7f5a9ffff640 1 -- 172.21.15.88:0/1864929568 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 7974+0+0 (secure 0 0 0) 0x7f5ab800f960 con 0x56051ec507a0 2024-09-16T01:42:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.300+0000 7f5aca1d5740 1 -- 172.21.15.88:0/1864929568 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5abc019070 con 0x56051ec507a0 2024-09-16T01:42:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.300+0000 7f5a9ffff640 1 -- 172.21.15.88:0/1864929568 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=230) ==== 24+0+0 (secure 0 0 0) 0x7f5ab8005810 con 0x56051ec507a0 2024-09-16T01:42:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.300+0000 7f5aca1d5740 10 client.9351 fetch_fsmap learned FSMap version 230 2024-09-16T01:42:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.300+0000 7f5aca1d5740 1 -- 172.21.15.88:0/1864929568 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=230,monmap=2+}) -- 0x56051e93a6b0 con 0x56051ec507a0 2024-09-16T01:42:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.300+0000 7f5a9ffff640 1 -- 172.21.15.88:0/1864929568 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5abc019070 con 0x56051ec507a0 2024-09-16T01:42:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.300+0000 7f5a9ffff640 1 -- 172.21.15.88:0/1864929568 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 230) ==== 3502+0+0 (secure 0 0 0) 0x7f5ab800fb60 con 0x56051ec507a0 2024-09-16T01:42:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.300+0000 7f5aca1d5740 10 client.9351 fetch_fsmap finished waiting for FSMap version 230 2024-09-16T01:42:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.300+0000 7f5aca1d5740 10 client.9351 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:42:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.300+0000 7f5aca1d5740 20 client.9351 populate_metadata read hostname 'smithi088' 2024-09-16T01:42:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.300+0000 7f5aca1d5740 1 --2- 172.21.15.88:0/1864929568 >> [v2:172.21.15.88:6834/4065900771,v1:172.21.15.88:6836/4065900771] conn(0x56051ec25180 0x56051ec1a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.300+0000 7f5aca1d5740 4 client.9351 mds_command: new command op to 9276 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:42:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.300+0000 7f5aca1d5740 1 -- 172.21.15.88:0/1864929568 --> [v2:172.21.15.88:6834/4065900771,v1:172.21.15.88:6836/4065900771] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56051eb6ac50 con 0x56051ec25180 2024-09-16T01:42:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.300+0000 7f5ac51d8640 1 --2- 172.21.15.88:0/1864929568 >> [v2:172.21.15.88:6834/4065900771,v1:172.21.15.88:6836/4065900771] conn(0x56051ec25180 0x56051ec1a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:56.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.301+0000 7f5ac51d8640 1 --2- 172.21.15.88:0/1864929568 >> [v2:172.21.15.88:6834/4065900771,v1:172.21.15.88:6836/4065900771] conn(0x56051ec25180 0x56051ec1a770 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=915c5217c2b46396 server_cookie=24a4acbfdc0096bd in_seq=0 out_seq=0 2024-09-16T01:42:56.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.301+0000 7f5a9ffff640 10 client.9351 ms_handle_connect on v2:172.21.15.88:6834/4065900771 2024-09-16T01:42:56.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.301+0000 7f5a9ffff640 1 -- 172.21.15.88:0/1864929568 <== mds.0 v2:172.21.15.88:6834/4065900771 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56051eb6ac50 con 0x56051ec25180 2024-09-16T01:42:56.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.301+0000 7f5a9ffff640 10 client.9351 handle_command_reply: tid=0 2024-09-16T01:42:56.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.301+0000 7f5aca1d5740 1 client.9351 shutdown 2024-09-16T01:42:56.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.301+0000 7f5aca1d5740 1 -- 172.21.15.88:0/1864929568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ec507a0 msgr2=0x56051ec24c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:56.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.301+0000 7f5aca1d5740 1 --2- 172.21.15.88:0/1864929568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ec507a0 0x56051ec24c40 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80118d0 tx=0x7f5ab8011d90 comp rx=0 tx=0).stop 2024-09-16T01:42:56.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.301+0000 7f5aca1d5740 1 -- 172.21.15.88:0/1864929568 shutdown_connections 2024-09-16T01:42:56.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.301+0000 7f5aca1d5740 1 --2- 172.21.15.88:0/1864929568 >> [v2:172.21.15.88:6834/4065900771,v1:172.21.15.88:6836/4065900771] conn(0x56051ec25180 0x56051ec1a770 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.301+0000 7f5aca1d5740 1 --2- 172.21.15.88:0/1864929568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ec507a0 0x56051ec24c40 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.301+0000 7f5a9ffff640 0 client.9351 ms_handle_reset on v2:172.21.15.88:6834/4065900771 2024-09-16T01:42:56.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.301+0000 7f5aca1d5740 1 --2- 172.21.15.88:0/1864929568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56051ebfca50 0x56051ec3d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.301+0000 7f5aca1d5740 1 --2- 172.21.15.88:0/1864929568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56051ebfb2c0 0x56051ec548d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.301+0000 7f5aca1d5740 1 -- 172.21.15.88:0/1864929568 >> 172.21.15.88:0/1864929568 conn(0x56051ebf8a80 msgr2=0x56051ec3dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:56.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.302+0000 7f5aca1d5740 1 -- 172.21.15.88:0/1864929568 shutdown_connections 2024-09-16T01:42:56.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.302+0000 7f5aca1d5740 1 -- 172.21.15.88:0/1864929568 wait complete. 2024-09-16T01:42:56.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.302+0000 7f5aca1d5740 20 client.9351 trim_cache size 0 max 16384 2024-09-16T01:42:56.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.319+0000 7f5aca1d5740 1 Processor -- start 2024-09-16T01:42:56.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.319+0000 7f5aca1d5740 1 -- start start 2024-09-16T01:42:56.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.320+0000 7f5aca1d5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56051ebfb2c0 0x56051ec644a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:56.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.320+0000 7f5aca1d5740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ebfca50 0x56051ec23ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:56.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.320+0000 7f5aca1d5740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56051ec507a0 0x56051ec43440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:56.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.320+0000 7f5aca1d5740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac01fcb90 con 0x56051ebfca50 2024-09-16T01:42:56.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.320+0000 7f5aca1d5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac01f42c0 con 0x56051ebfb2c0 2024-09-16T01:42:56.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.320+0000 7f5aca1d5740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5ac01fc540 con 0x56051ec507a0 2024-09-16T01:42:56.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.320+0000 7f5ac49d7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ebfca50 0x56051ec23ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:56.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.320+0000 7f5ac49d7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ebfca50 0x56051ec23ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35062/0 (socket says 172.21.15.88:35062) 2024-09-16T01:42:56.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.320+0000 7f5ac49d7640 1 -- 172.21.15.88:0/1692272064 learned_addr learned my addr 172.21.15.88:0/1692272064 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:56.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.320+0000 7f5ac51d8640 1 --2- 172.21.15.88:0/1692272064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56051ebfb2c0 0x56051ec644a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:56.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.320+0000 7f5ac49d7640 1 -- 172.21.15.88:0/1692272064 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56051ec507a0 msgr2=0x56051ec43440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:56.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.320+0000 7f5ac49d7640 1 --2- 172.21.15.88:0/1692272064 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56051ec507a0 0x56051ec43440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.320+0000 7f5ac49d7640 1 -- 172.21.15.88:0/1692272064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56051ebfb2c0 msgr2=0x56051ec644a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:56.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.320+0000 7f5ac49d7640 1 --2- 172.21.15.88:0/1692272064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56051ebfb2c0 0x56051ec644a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.320+0000 7f5ac49d7640 1 -- 172.21.15.88:0/1692272064 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56051ec540d0 con 0x56051ebfca50 2024-09-16T01:42:56.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.320+0000 7f5ac51d8640 1 --2- 172.21.15.88:0/1692272064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56051ebfb2c0 0x56051ec644a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:56.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.321+0000 7f5ac49d7640 1 --2- 172.21.15.88:0/1692272064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ebfca50 0x56051ec23ce0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f5ab007c630 tx=0x7f5ab000b3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dcf7d290b0833cb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:56.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.321+0000 7f5aaa7fc640 1 -- 172.21.15.88:0/1692272064 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab00809a0 con 0x56051ebfca50 2024-09-16T01:42:56.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.321+0000 7f5aaa7fc640 1 -- 172.21.15.88:0/1692272064 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ab0038040 con 0x56051ebfca50 2024-09-16T01:42:56.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.321+0000 7f5aaa7fc640 1 -- 172.21.15.88:0/1692272064 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab00851d0 con 0x56051ebfca50 2024-09-16T01:42:56.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.321+0000 7f5aca1d5740 1 -- 172.21.15.88:0/1692272064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ebfca50 msgr2=0x56051ec23ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:56.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.321+0000 7f5aca1d5740 1 --2- 172.21.15.88:0/1692272064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ebfca50 0x56051ec23ce0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f5ab007c630 tx=0x7f5ab000b3d0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.321+0000 7f5aca1d5740 1 -- 172.21.15.88:0/1692272064 shutdown_connections 2024-09-16T01:42:56.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.321+0000 7f5aca1d5740 1 --2- 172.21.15.88:0/1692272064 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56051ec507a0 0x56051ec43440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.321+0000 7f5aca1d5740 1 --2- 172.21.15.88:0/1692272064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ebfca50 0x56051ec23ce0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.321+0000 7f5aca1d5740 1 --2- 172.21.15.88:0/1692272064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56051ebfb2c0 0x56051ec644a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.321+0000 7f5aca1d5740 1 -- 172.21.15.88:0/1692272064 >> 172.21.15.88:0/1692272064 conn(0x56051ebf8a80 msgr2=0x56051ec1ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:56.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.321+0000 7f5aca1d5740 1 -- 172.21.15.88:0/1692272064 shutdown_connections 2024-09-16T01:42:56.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.321+0000 7f5aca1d5740 1 -- 172.21.15.88:0/1692272064 wait complete. 2024-09-16T01:42:56.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.322+0000 7f5aca1d5740 1 -- start start 2024-09-16T01:42:56.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.322+0000 7f5aca1d5740 1 Processor -- start 2024-09-16T01:42:56.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.323+0000 7f5aca1d5740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56051ebfb2c0 0x56051ecb36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:56.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.323+0000 7f5aca1d5740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56051ebfca50 0x56051ec20370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:56.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.323+0000 7f5aca1d5740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ec507a0 0x56051ec208b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:56.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.323+0000 7f5aca1d5740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56051ec3db30 con 0x56051ec507a0 2024-09-16T01:42:56.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.323+0000 7f5aca1d5740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56051ec51640 con 0x56051ebfca50 2024-09-16T01:42:56.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.323+0000 7f5aca1d5740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56051ec23350 con 0x56051ebfb2c0 2024-09-16T01:42:56.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.323+0000 7f5aca1d5740 10 client.0 _ll_register_callbacks cb 0x7f5ab80051b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:42:56.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.323+0000 7f5ac59d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ec507a0 0x56051ec208b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:56.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.323+0000 7f5ac59d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ec507a0 0x56051ec208b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35070/0 (socket says 172.21.15.88:35070) 2024-09-16T01:42:56.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.323+0000 7f5ac59d9640 1 -- 172.21.15.88:0/2920418827 learned_addr learned my addr 172.21.15.88:0/2920418827 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:56.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.323+0000 7f5ac49d7640 1 --2- 172.21.15.88:0/2920418827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56051ebfca50 0x56051ec20370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:56.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.323+0000 7f5ac51d8640 1 --2- 172.21.15.88:0/2920418827 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56051ebfb2c0 0x56051ecb36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:56.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.323+0000 7f5ac59d9640 1 -- 172.21.15.88:0/2920418827 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56051ebfb2c0 msgr2=0x56051ecb36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:56.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.323+0000 7f5ac59d9640 1 --2- 172.21.15.88:0/2920418827 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56051ebfb2c0 0x56051ecb36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.323+0000 7f5ac59d9640 1 -- 172.21.15.88:0/2920418827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56051ebfca50 msgr2=0x56051ec20370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:56.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.323+0000 7f5ac59d9640 1 --2- 172.21.15.88:0/2920418827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56051ebfca50 0x56051ec20370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.323+0000 7f5ac59d9640 1 -- 172.21.15.88:0/2920418827 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56051ec221f0 con 0x56051ec507a0 2024-09-16T01:42:56.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.323+0000 7f5ac51d8640 1 --2- 172.21.15.88:0/2920418827 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56051ebfb2c0 0x56051ecb36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:56.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.323+0000 7f5ac59d9640 1 --2- 172.21.15.88:0/2920418827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ec507a0 0x56051ec208b0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8011840 tx=0x7f5ab8003a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f68f72e2ad452daa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:56.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.323+0000 7f5a9ffff640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:42:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.324+0000 7f5a9ffff640 1 -- 172.21.15.88:0/2920418827 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab801b070 con 0x56051ec507a0 2024-09-16T01:42:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.324+0000 7f5a9ffff640 1 -- 172.21.15.88:0/2920418827 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 7974+0+0 (secure 0 0 0) 0x7f5ab8024760 con 0x56051ec507a0 2024-09-16T01:42:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.324+0000 7f5aca1d5740 1 -- 172.21.15.88:0/2920418827 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ab0038040 con 0x56051ec507a0 2024-09-16T01:42:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.324+0000 7f5a9ffff640 1 -- 172.21.15.88:0/2920418827 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=230) ==== 24+0+0 (secure 0 0 0) 0x7f5ab800f5a0 con 0x56051ec507a0 2024-09-16T01:42:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.324+0000 7f5aca1d5740 10 client.9357 fetch_fsmap learned FSMap version 230 2024-09-16T01:42:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.324+0000 7f5aca1d5740 1 -- 172.21.15.88:0/2920418827 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=230,monmap=2+}) -- 0x56051ec1ac90 con 0x56051ec507a0 2024-09-16T01:42:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.324+0000 7f5a9ffff640 1 -- 172.21.15.88:0/2920418827 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ab0038040 con 0x56051ec507a0 2024-09-16T01:42:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.324+0000 7f5a9ffff640 1 -- 172.21.15.88:0/2920418827 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 230) ==== 3502+0+0 (secure 0 0 0) 0x7f5ab802c020 con 0x56051ec507a0 2024-09-16T01:42:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.324+0000 7f5aca1d5740 10 client.9357 fetch_fsmap finished waiting for FSMap version 230 2024-09-16T01:42:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.324+0000 7f5aca1d5740 10 client.9357 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:42:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.324+0000 7f5aca1d5740 20 client.9357 populate_metadata read hostname 'smithi088' 2024-09-16T01:42:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.324+0000 7f5aca1d5740 1 --2- 172.21.15.88:0/2920418827 >> [v2:172.21.15.88:6834/4065900771,v1:172.21.15.88:6836/4065900771] conn(0x56051ec63ab0 0x56051ecb4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.324+0000 7f5aca1d5740 4 client.9357 mds_command: new command op to 9276 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:42:56.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.324+0000 7f5aca1d5740 1 -- 172.21.15.88:0/2920418827 --> [v2:172.21.15.88:6834/4065900771,v1:172.21.15.88:6836/4065900771] -- command(tid 0: {"prefix": "status"}) -- 0x56051eb64fa0 con 0x56051ec63ab0 2024-09-16T01:42:56.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.324+0000 7f5ac51d8640 1 --2- 172.21.15.88:0/2920418827 >> [v2:172.21.15.88:6834/4065900771,v1:172.21.15.88:6836/4065900771] conn(0x56051ec63ab0 0x56051ecb4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:56.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.324+0000 7f5a9ffff640 10 client.9357 ms_handle_connect on v2:172.21.15.88:6834/4065900771 2024-09-16T01:42:56.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.324+0000 7f5ac51d8640 1 --2- 172.21.15.88:0/2920418827 >> [v2:172.21.15.88:6834/4065900771,v1:172.21.15.88:6836/4065900771] conn(0x56051ec63ab0 0x56051ecb4380 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=edea6725b88dc784 server_cookie=3cf35ba5a63f31ce in_seq=0 out_seq=0 2024-09-16T01:42:56.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.325+0000 7f5a9ffff640 1 -- 172.21.15.88:0/2920418827 <== mds.0 v2:172.21.15.88:6834/4065900771 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x56051eb64fa0 con 0x56051ec63ab0 2024-09-16T01:42:56.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.325+0000 7f5a9ffff640 10 client.9357 handle_command_reply: tid=0 2024-09-16T01:42:56.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.325+0000 7f5aca1d5740 1 client.9357 shutdown 2024-09-16T01:42:56.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.325+0000 7f5aca1d5740 1 -- 172.21.15.88:0/2920418827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ec507a0 msgr2=0x56051ec208b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:56.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.325+0000 7f5aca1d5740 1 --2- 172.21.15.88:0/2920418827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ec507a0 0x56051ec208b0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8011840 tx=0x7f5ab8003a70 comp rx=0 tx=0).stop 2024-09-16T01:42:56.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.325+0000 7f5aca1d5740 1 -- 172.21.15.88:0/2920418827 shutdown_connections 2024-09-16T01:42:56.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.325+0000 7f5aca1d5740 1 --2- 172.21.15.88:0/2920418827 >> [v2:172.21.15.88:6834/4065900771,v1:172.21.15.88:6836/4065900771] conn(0x56051ec63ab0 0x56051ecb4380 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.325+0000 7f5aca1d5740 1 --2- 172.21.15.88:0/2920418827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56051ec507a0 0x56051ec208b0 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.325+0000 7f5a9ffff640 0 client.9357 ms_handle_reset on v2:172.21.15.88:6834/4065900771 2024-09-16T01:42:56.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.325+0000 7f5aca1d5740 1 --2- 172.21.15.88:0/2920418827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56051ebfca50 0x56051ec20370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.325+0000 7f5aca1d5740 1 --2- 172.21.15.88:0/2920418827 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56051ebfb2c0 0x56051ecb36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.325+0000 7f5aca1d5740 1 -- 172.21.15.88:0/2920418827 >> 172.21.15.88:0/2920418827 conn(0x56051ebf8a80 msgr2=0x56051ec43440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:56.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.325+0000 7f5aca1d5740 1 -- 172.21.15.88:0/2920418827 shutdown_connections 2024-09-16T01:42:56.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.325+0000 7f5aca1d5740 1 -- 172.21.15.88:0/2920418827 wait complete. 2024-09-16T01:42:56.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.325+0000 7f5aca1d5740 20 client.9357 trim_cache size 0 max 16384 2024-09-16T01:42:56.336 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:42:56.336 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "5ace2c91-46f9-482b-8a89-731fbdc171bf", 2024-09-16T01:42:56.336 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2024-09-16T01:42:56.336 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 9276, 2024-09-16T01:42:56.337 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2024-09-16T01:42:56.337 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2024-09-16T01:42:56.337 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2024-09-16T01:42:56.337 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 6.6217999460000003, 2024-09-16T01:42:56.337 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 230, 2024-09-16T01:42:56.337 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 113, 2024-09-16T01:42:56.337 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 113, 2024-09-16T01:42:56.337 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 9.9197830709999995 2024-09-16T01:42:56.337 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:42:56.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.325+0000 7f5ac743c640 1 -- 172.21.15.88:0/1434057301 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5aa406c2b0 msgr2=0x7f5aa406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:56.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.325+0000 7f5ac743c640 1 --2- 172.21.15.88:0/1434057301 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5aa406c2b0 0x7f5aa406e770 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0156d80 tx=0x7f5abc008040 comp rx=0 tx=0).stop 2024-09-16T01:42:56.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.325+0000 7f5ac743c640 1 -- 172.21.15.88:0/1434057301 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ac00b60c0 msgr2=0x7f5ac00c9950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:56.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.325+0000 7f5ac743c640 1 --2- 172.21.15.88:0/1434057301 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ac00b60c0 0x7f5ac00c9950 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0000c00 tx=0x7f5ab00046c0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.326+0000 7f5ac743c640 1 -- 172.21.15.88:0/1434057301 shutdown_connections 2024-09-16T01:42:56.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.326+0000 7f5ac743c640 1 --2- 172.21.15.88:0/1434057301 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5aa406c2b0 0x7f5aa406e770 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.326+0000 7f5ac743c640 1 --2- 172.21.15.88:0/1434057301 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ac015b480 0x7f5ac00c7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.326+0000 7f5ac743c640 1 --2- 172.21.15.88:0/1434057301 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ac00b60c0 0x7f5ac00c9950 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.326+0000 7f5ac743c640 1 --2- 172.21.15.88:0/1434057301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ac0072530 0x7f5ac00b3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.326+0000 7f5ac743c640 1 -- 172.21.15.88:0/1434057301 >> 172.21.15.88:0/1434057301 conn(0x7f5ac006d0f0 msgr2=0x7f5ac00d0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:56.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.326+0000 7f5ac743c640 1 -- 172.21.15.88:0/1434057301 shutdown_connections 2024-09-16T01:42:56.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.326+0000 7f5ac743c640 1 -- 172.21.15.88:0/1434057301 wait complete. 2024-09-16T01:42:56.340 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-09-16T01:42:56.340 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Truncate 500.00000000', proceeding to ls 2024-09-16T01:42:56.340 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:42:56.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.435+0000 7f6c19523640 1 Processor -- start 2024-09-16T01:42:56.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.436+0000 7f6c19523640 1 -- start start 2024-09-16T01:42:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.436+0000 7f6c19523640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6c140d4d00 0x7f6c140cf940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.436+0000 7f6c19523640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c140d3c60 con 0x7f6c140d18c0 2024-09-16T01:42:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.436+0000 7f6c19523640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c140d3f40 con 0x7f6c1415e640 2024-09-16T01:42:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.436+0000 7f6c19523640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6c140d5170 con 0x7f6c140d4d00 2024-09-16T01:42:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.436+0000 7f6c137fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6c140d4d00 0x7f6c140cf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.436+0000 7f6c127fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6c140d18c0 0x7f6c140b7850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42476/0 (socket says 172.21.15.88:42476) 2024-09-16T01:42:56.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.436+0000 7f6c137fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6c140d4d00 0x7f6c140cf940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35076/0 (socket says 172.21.15.88:35076) 2024-09-16T01:42:56.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.436+0000 7f6c127fc640 1 -- 172.21.15.88:0/3202192385 learned_addr learned my addr 172.21.15.88:0/3202192385 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:56.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.437+0000 7f6c137fe640 1 -- 172.21.15.88:0/3202192385 >> v1:172.21.15.154:6789/0 conn(0x7f6c1415e640 legacy=0x7f6c140d11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:56.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.437+0000 7f6c137fe640 1 -- 172.21.15.88:0/3202192385 >> v1:172.21.15.88:6789/0 conn(0x7f6c140d18c0 legacy=0x7f6c140b7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:56.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.437+0000 7f6c137fe640 1 -- 172.21.15.88:0/3202192385 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c140d4220 con 0x7f6c140d4d00 2024-09-16T01:42:56.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.437+0000 7f6c11ffb640 1 -- 172.21.15.88:0/3202192385 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3967760534 0 0) 0x7f6c140d3c60 con 0x7f6c140d18c0 2024-09-16T01:42:56.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.437+0000 7f6c137fe640 1 --2- 172.21.15.88:0/3202192385 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c140d4d00 0x7f6c140cf940 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c0097a0 tx=0x7f6c0c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff32168a663d98f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:56.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.438+0000 7f6c11ffb640 1 -- 172.21.15.88:0/3202192385 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c037070 con 0x7f6c140d4d00 2024-09-16T01:42:56.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.438+0000 7f6c11ffb640 1 -- 172.21.15.88:0/3202192385 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c0c032440 con 0x7f6c140d4d00 2024-09-16T01:42:56.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.438+0000 7f6c11ffb640 1 -- 172.21.15.88:0/3202192385 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c032810 con 0x7f6c140d4d00 2024-09-16T01:42:56.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.438+0000 7f6c19523640 1 -- 172.21.15.88:0/3202192385 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c140d4d00 msgr2=0x7f6c140cf940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:56.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.438+0000 7f6c19523640 1 --2- 172.21.15.88:0/3202192385 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c140d4d00 0x7f6c140cf940 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c0097a0 tx=0x7f6c0c02a230 comp rx=0 tx=0).stop 2024-09-16T01:42:56.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.438+0000 7f6c19523640 1 -- 172.21.15.88:0/3202192385 shutdown_connections 2024-09-16T01:42:56.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.438+0000 7f6c19523640 1 --2- 172.21.15.88:0/3202192385 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c140d4d00 0x7f6c140cf940 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.438+0000 7f6c19523640 1 -- 172.21.15.88:0/3202192385 >> 172.21.15.88:0/3202192385 conn(0x7f6c14062a50 msgr2=0x7f6c14062e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:56.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.438+0000 7f6c19523640 1 -- 172.21.15.88:0/3202192385 shutdown_connections 2024-09-16T01:42:56.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.438+0000 7f6c19523640 1 -- 172.21.15.88:0/3202192385 wait complete. 2024-09-16T01:42:56.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.439+0000 7f6c19523640 1 Processor -- start 2024-09-16T01:42:56.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.440+0000 7f6c19523640 1 -- start start 2024-09-16T01:42:56.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.440+0000 7f6c19523640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c140d18c0 0x7f6c140ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:56.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.440+0000 7f6c19523640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c140d4d00 0x7f6c14152d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:56.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.440+0000 7f6c19523640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c1415e640 0x7f6c140c8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:56.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.440+0000 7f6c19523640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1415eba0 con 0x7f6c140d18c0 2024-09-16T01:42:56.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.440+0000 7f6c19523640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1415ea20 con 0x7f6c140d4d00 2024-09-16T01:42:56.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.440+0000 7f6c19523640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6c140d34f0 con 0x7f6c1415e640 2024-09-16T01:42:56.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.440+0000 7f6c12ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c140d18c0 0x7f6c140ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:56.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.440+0000 7f6c12ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c140d18c0 0x7f6c140ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35092/0 (socket says 172.21.15.88:35092) 2024-09-16T01:42:56.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.440+0000 7f6c12ffd640 1 -- 172.21.15.88:0/2518556422 learned_addr learned my addr 172.21.15.88:0/2518556422 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:56.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.440+0000 7f6c127fc640 1 --2- 172.21.15.88:0/2518556422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c140d4d00 0x7f6c14152d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:56.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.441+0000 7f6c137fe640 1 --2- 172.21.15.88:0/2518556422 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c1415e640 0x7f6c140c8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:56.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.441+0000 7f6c12ffd640 1 -- 172.21.15.88:0/2518556422 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c1415e640 msgr2=0x7f6c140c8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:56.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.441+0000 7f6c12ffd640 1 --2- 172.21.15.88:0/2518556422 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c1415e640 0x7f6c140c8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.441+0000 7f6c12ffd640 1 -- 172.21.15.88:0/2518556422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c140d4d00 msgr2=0x7f6c14152d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:56.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.441+0000 7f6c12ffd640 1 --2- 172.21.15.88:0/2518556422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c140d4d00 0x7f6c14152d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.441+0000 7f6c12ffd640 1 -- 172.21.15.88:0/2518556422 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c140c93a0 con 0x7f6c140d18c0 2024-09-16T01:42:56.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.441+0000 7f6c12ffd640 1 --2- 172.21.15.88:0/2518556422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c140d18c0 0x7f6c140ce4f0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f6c0400c760 tx=0x7f6c0400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:56.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.441+0000 7f6bfb7fe640 1 -- 172.21.15.88:0/2518556422 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c04003b70 con 0x7f6c140d18c0 2024-09-16T01:42:56.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.441+0000 7f6bfb7fe640 1 -- 172.21.15.88:0/2518556422 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c04003d30 con 0x7f6c140d18c0 2024-09-16T01:42:56.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.441+0000 7f6c19523640 1 -- 172.21.15.88:0/2518556422 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c140cb9f0 con 0x7f6c140d18c0 2024-09-16T01:42:56.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.441+0000 7f6bfb7fe640 1 -- 172.21.15.88:0/2518556422 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c0400aa90 con 0x7f6c140d18c0 2024-09-16T01:42:56.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.441+0000 7f6c19523640 1 -- 172.21.15.88:0/2518556422 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c14154710 con 0x7f6c140d18c0 2024-09-16T01:42:56.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.442+0000 7f6c19523640 1 -- 172.21.15.88:0/2518556422 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c140d3890 con 0x7f6c140d18c0 2024-09-16T01:42:56.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.442+0000 7f6bfb7fe640 1 -- 172.21.15.88:0/2518556422 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6c0400ac30 con 0x7f6c140d18c0 2024-09-16T01:42:56.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.443+0000 7f6bfb7fe640 1 --2- 172.21.15.88:0/2518556422 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6c0006c2e0 0x7f6c0006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:56.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.443+0000 7f6bfb7fe640 1 -- 172.21.15.88:0/2518556422 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 7974+0+0 (secure 0 0 0) 0x7f6c0408ba90 con 0x7f6c140d18c0 2024-09-16T01:42:56.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.443+0000 7f6c127fc640 1 --2- 172.21.15.88:0/2518556422 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6c0006c2e0 0x7f6c0006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:56.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.443+0000 7f6c127fc640 1 --2- 172.21.15.88:0/2518556422 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6c0006c2e0 0x7f6c0006e7a0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f6c08000fb0 tx=0x7f6c08000f40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:56.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.447+0000 7f6bfb7fe640 1 -- 172.21.15.88:0/2518556422 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6c040592d0 con 0x7f6c140d18c0 2024-09-16T01:42:56.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.647+0000 7f6c19523640 1 -- 172.21.15.88:0/2518556422 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f6c140d2610 con 0x7f6c140d18c0 2024-09-16T01:42:56.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.647+0000 7f6bfb7fe640 1 -- 172.21.15.88:0/2518556422 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v230) ==== 53+0+73 (secure 0 0 0) 0x7f6c0405d5d0 con 0x7f6c140d18c0 2024-09-16T01:42:56.649 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:42:56.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.649+0000 7f6c19523640 1 -- 172.21.15.88:0/2518556422 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6c0006c2e0 msgr2=0x7f6c0006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:56.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.649+0000 7f6c19523640 1 --2- 172.21.15.88:0/2518556422 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6c0006c2e0 0x7f6c0006e7a0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f6c08000fb0 tx=0x7f6c08000f40 comp rx=0 tx=0).stop 2024-09-16T01:42:56.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.649+0000 7f6c19523640 1 -- 172.21.15.88:0/2518556422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c140d18c0 msgr2=0x7f6c140ce4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:56.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.649+0000 7f6c19523640 1 --2- 172.21.15.88:0/2518556422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c140d18c0 0x7f6c140ce4f0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f6c0400c760 tx=0x7f6c0400cc20 comp rx=0 tx=0).stop 2024-09-16T01:42:56.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.649+0000 7f6c19523640 1 -- 172.21.15.88:0/2518556422 shutdown_connections 2024-09-16T01:42:56.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.649+0000 7f6c19523640 1 --2- 172.21.15.88:0/2518556422 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6c0006c2e0 0x7f6c0006e7a0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.649+0000 7f6c19523640 1 --2- 172.21.15.88:0/2518556422 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c1415e640 0x7f6c140c8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.649+0000 7f6c19523640 1 --2- 172.21.15.88:0/2518556422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c140d4d00 0x7f6c14152d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.649+0000 7f6c19523640 1 --2- 172.21.15.88:0/2518556422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c140d18c0 0x7f6c140ce4f0 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:56.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.649+0000 7f6c19523640 1 -- 172.21.15.88:0/2518556422 >> 172.21.15.88:0/2518556422 conn(0x7f6c14062a50 msgr2=0x7f6c1415f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:56.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.649+0000 7f6c19523640 1 -- 172.21.15.88:0/2518556422 shutdown_connections 2024-09-16T01:42:56.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56.649+0000 7f6c19523640 1 -- 172.21.15.88:0/2518556422 wait complete. 2024-09-16T01:42:56.661 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-16T01:42:56.661 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:42:56.661 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2024-09-16T01:42:56.662 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T01:42:56.662 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T01:42:56.662 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-16T01:42:56.662 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-16T01:42:56.662 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:42:56.662 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:42:56.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:56.662 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2024-09-16T01:42:56.678 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:42:56.678 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:42:56.678 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:42:56.678 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:42:56.678 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-09-16T01:42:56.678 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:42:56.678 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:42:56.679 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:42:56.679 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2024-09-16T01:42:56.679 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:42:56.679 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:42:56.679 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2024-09-16T01:42:56.679 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:42:56.679 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2024-09-16T01:42:56.679 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:42:56.679 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2024-09-16T01:42:56.679 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:42:56.679 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42247sec preferred_lft 42247sec 2024-09-16T01:42:56.679 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link noprefixroute 2024-09-16T01:42:56.679 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:42:56.679 INFO:teuthology.orchestra.run.smithi088.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:42:56.680 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff 2024-09-16T01:42:56.680 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T01:42:56.680 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:42:56.680 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::e486:6eff:fe81:a1c/64 scope link 2024-09-16T01:42:56.680 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:42:56.680 INFO:teuthology.orchestra.run.smithi088.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:42:56.680 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 12:6f:29:be:95:48 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:42:56.680 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::106f:29ff:febe:9548/64 scope link 2024-09-16T01:42:56.680 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:42:56.680 INFO:teuthology.orchestra.run.smithi088.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:42:56.680 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T01:42:56.680 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::c63:b4ff:feb3:ad9e/64 scope link 2024-09-16T01:42:56.680 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:42:56.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:56.681 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:42:56.688 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:42:56.687+0000 7efe9b874640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:42:56.711 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-09-16T01:42:56.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:56.712 DEBUG:teuthology.orchestra.run.smithi088:> route 2024-09-16T01:42:56.769 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2024-09-16T01:42:56.769 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:42:56.769 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:42:56.769 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:42:56.769 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:42:56.770 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:42:56.770 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:42:56.770 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:42:56.770 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:42:56.770 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:42:56.770 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:42:56.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:42:56.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:42:56.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:56.894 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:42:56.909 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T01:42:56.910 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T01:42:56.910 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:42:56.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:56.910 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2024-09-16T01:42:56.975 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:42:56.975 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T01:42:57.030 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:42:57.031 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:42:57.031 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:42:57.086 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-16T01:42:57.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:57.086 DEBUG:teuthology.orchestra.run.smithi088:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:42:57.169 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:42:57.169 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:42:57.169 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:42:57.178 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:42:57.178 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:42:57.178 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:42:57.178 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:42:57.178 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:42:57.184 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:42:57.184 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:42:57.184 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:42:57.184 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:42:57.184 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:42:57.184 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790" 2024-09-16T01:42:57.184 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2024-09-16T01:42:57.185 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5ace2c91-46f9-482b-8a89-731fbdc171bf" will pass to kernel 2024-09-16T01:42:57.185 INFO:teuthology.orchestra.run.smithi088.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-16T01:42:57.185 INFO:teuthology.orchestra.run.smithi088.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:42:57.185 INFO:teuthology.orchestra.run.smithi088.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:42:57.185 INFO:teuthology.orchestra.run.smithi088.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:42:57.185 INFO:teuthology.orchestra.run.smithi088.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:42:57.187 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:42:57.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:57.187 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:42:57.252 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:42:57.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:42:57.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:42:57.349 INFO:teuthology.orchestra.run.smithi088.stdout:client7001 2024-09-16T01:42:57.349 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:42:57.350 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:42:57.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:57.366 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R .) 2024-09-16T01:42:57.408 INFO:teuthology.orchestra.run:waiting for 20 2024-09-16T01:42:57.423 INFO:teuthology.orchestra.run.smithi088.stdout:.: 2024-09-16T01:42:57.423 INFO:teuthology.orchestra.run.smithi088.stdout:subdir 2024-09-16T01:42:57.423 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:42:57.423 INFO:teuthology.orchestra.run.smithi088.stdout:./subdir: 2024-09-16T01:42:57.424 INFO:teuthology.orchestra.run.smithi088.stdout:sixmegs 2024-09-16T01:42:57.424 INFO:tasks.cephfs.test_damage:Result: Mutation 'Truncate 500.00000000' did not caused DAMAGED state 2024-09-16T01:42:57.424 INFO:tasks.cephfs.test_damage:Applying mutation 'Truncate mds_snaptable' 2024-09-16T01:42:57.424 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:42:57.424 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:42:57.479 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:42:57.480 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:42:57.538 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:42:57.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.675+0000 7f0d39a2e640 1 Processor -- start 2024-09-16T01:42:57.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.675+0000 7f0d39a2e640 1 -- start start 2024-09-16T01:42:57.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.675+0000 7f0d39a2e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d340d1130 0x7f0d340d1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:57.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.675+0000 7f0d39a2e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0d340d1a50 0x7f0d340b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:57.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.675+0000 7f0d39a2e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d340d4530 con 0x7f0d340b8160 2024-09-16T01:42:57.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.675+0000 7f0d39a2e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0d340d40a0 con 0x7f0d340d1a50 2024-09-16T01:42:57.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.675+0000 7f0d39a2e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0d340d3490 con 0x7f0d340d1130 2024-09-16T01:42:57.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.676+0000 7f0d337fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0d340b8160 0x7f0d340b8610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42484/0 (socket says 172.21.15.88:42484) 2024-09-16T01:42:57.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.676+0000 7f0d337fe640 1 -- 172.21.15.88:0/2153643668 learned_addr learned my addr 172.21.15.88:0/2153643668 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:57.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.676+0000 7f0d32ffd640 1 --2- 172.21.15.88:0/2153643668 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d340d1130 0x7f0d340d1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:57.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.676+0000 7f0d2a7fc640 1 --2- 172.21.15.88:0/2153643668 >> v2:172.21.15.154:3300/0 conn(0x7f0d340d1a50 0x7f0d340b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:57.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.676+0000 7f0d327fc640 1 -- 172.21.15.88:0/2153643668 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 126125788 0 0) 0x7f0d340d4530 con 0x7f0d340b8160 2024-09-16T01:42:57.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.676+0000 7f0d327fc640 1 -- 172.21.15.88:0/2153643668 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d20003520 con 0x7f0d340b8160 2024-09-16T01:42:57.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.676+0000 7f0d327fc640 1 -- 172.21.15.88:0/2153643668 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1809451545 0 0) 0x7f0d20003520 con 0x7f0d340b8160 2024-09-16T01:42:57.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.676+0000 7f0d327fc640 1 -- 172.21.15.88:0/2153643668 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d340d1130 msgr2=0x7f0d340d1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:57.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.676+0000 7f0d327fc640 1 --2- 172.21.15.88:0/2153643668 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d340d1130 0x7f0d340d1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:57.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.676+0000 7f0d327fc640 1 -- 172.21.15.88:0/2153643668 >> v2:172.21.15.154:3300/0 conn(0x7f0d340d1a50 msgr2=0x7f0d340b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:57.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.676+0000 7f0d327fc640 1 --2- 172.21.15.88:0/2153643668 >> v2:172.21.15.154:3300/0 conn(0x7f0d340d1a50 0x7f0d340b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:57.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.676+0000 7f0d327fc640 1 -- 172.21.15.88:0/2153643668 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d340d3770 con 0x7f0d340b8160 2024-09-16T01:42:57.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.676+0000 7f0d327fc640 1 -- 172.21.15.88:0/2153643668 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f0d2c0019f0 con 0x7f0d340b8160 2024-09-16T01:42:57.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.676+0000 7f0d327fc640 1 -- 172.21.15.88:0/2153643668 >> v1:172.21.15.88:6789/0 conn(0x7f0d340b8160 legacy=0x7f0d340b8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:57.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.676+0000 7f0d327fc640 1 --2- 172.21.15.88:0/2153643668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d200039b0 0x7f0d20003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:57.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.676+0000 7f0d327fc640 1 --2- 172.21.15.88:0/2153643668 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d340d1130 0x7f0d20004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:57.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.676+0000 7f0d327fc640 1 --2- 172.21.15.88:0/2153643668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d20004d40 0x7f0d200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:57.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.676+0000 7f0d327fc640 1 -- 172.21.15.88:0/2153643668 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0d340d40a0 con 0x7f0d200039b0 2024-09-16T01:42:57.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.676+0000 7f0d327fc640 1 -- 172.21.15.88:0/2153643668 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0d340d3490 con 0x7f0d20004d40 2024-09-16T01:42:57.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.676+0000 7f0d327fc640 1 -- 172.21.15.88:0/2153643668 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0d2000b9c0 con 0x7f0d340d1130 2024-09-16T01:42:57.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.676+0000 7f0d32ffd640 1 --2- 172.21.15.88:0/2153643668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d200039b0 0x7f0d20003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:57.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.676+0000 7f0d2a7fc640 1 --2- 172.21.15.88:0/2153643668 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d340d1130 0x7f0d20004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:57.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.676+0000 7f0d337fe640 1 --2- 172.21.15.88:0/2153643668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d20004d40 0x7f0d200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:57.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.677+0000 7f0d32ffd640 1 -- 172.21.15.88:0/2153643668 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d340d1130 msgr2=0x7f0d20004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:57.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.677+0000 7f0d32ffd640 1 --2- 172.21.15.88:0/2153643668 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d340d1130 0x7f0d20004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:57.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.677+0000 7f0d32ffd640 1 -- 172.21.15.88:0/2153643668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d20004d40 msgr2=0x7f0d200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:57.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.677+0000 7f0d32ffd640 1 --2- 172.21.15.88:0/2153643668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d20004d40 0x7f0d200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:57.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.677+0000 7f0d32ffd640 1 -- 172.21.15.88:0/2153643668 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d20003520 con 0x7f0d200039b0 2024-09-16T01:42:57.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.677+0000 7f0d2a7fc640 1 --2- 172.21.15.88:0/2153643668 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d340d1130 0x7f0d20004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:57.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.677+0000 7f0d32ffd640 1 --2- 172.21.15.88:0/2153643668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d200039b0 0x7f0d20003dd0 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f0d24009ac0 tx=0x7f0d20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fed63ebb4c0bec3c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:57.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.677+0000 7f0d327fc640 1 -- 172.21.15.88:0/2153643668 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d24036070 con 0x7f0d200039b0 2024-09-16T01:42:57.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.677+0000 7f0d327fc640 1 -- 172.21.15.88:0/2153643668 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d24031400 con 0x7f0d200039b0 2024-09-16T01:42:57.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.677+0000 7f0d327fc640 1 -- 172.21.15.88:0/2153643668 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d24009690 con 0x7f0d200039b0 2024-09-16T01:42:57.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.678+0000 7f0d39a2e640 1 -- 172.21.15.88:0/2153643668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d200039b0 msgr2=0x7f0d20003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:57.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.678+0000 7f0d39a2e640 1 --2- 172.21.15.88:0/2153643668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d200039b0 0x7f0d20003dd0 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f0d24009ac0 tx=0x7f0d20005830 comp rx=0 tx=0).stop 2024-09-16T01:42:57.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.678+0000 7f0d337fe640 1 -- 172.21.15.88:0/2153643668 reap_dead start 2024-09-16T01:42:57.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.678+0000 7f0d39a2e640 1 -- 172.21.15.88:0/2153643668 shutdown_connections 2024-09-16T01:42:57.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.678+0000 7f0d39a2e640 1 -- 172.21.15.88:0/2153643668 >> 172.21.15.88:0/2153643668 conn(0x7f0d3406c020 msgr2=0x7f0d3406c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:57.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.678+0000 7f0d39a2e640 1 -- 172.21.15.88:0/2153643668 shutdown_connections 2024-09-16T01:42:57.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.678+0000 7f0d39a2e640 1 -- 172.21.15.88:0/2153643668 wait complete. 2024-09-16T01:42:57.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.679+0000 7f0d39a2e640 1 Processor -- start 2024-09-16T01:42:57.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.679+0000 7f0d39a2e640 1 -- start start 2024-09-16T01:42:57.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.680+0000 7f0d39a2e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d200039b0 0x7f0d340b5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:57.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.680+0000 7f0d39a2e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d341531b0 0x7f0d340ad6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:57.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.680+0000 7f0d39a2e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d34150600 0x7f0d340cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:57.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.680+0000 7f0d39a2e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0d340b8d20 con 0x7f0d200039b0 2024-09-16T01:42:57.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.680+0000 7f0d39a2e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0d340d3d30 con 0x7f0d34150600 2024-09-16T01:42:57.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.680+0000 7f0d39a2e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0d340d1e90 con 0x7f0d341531b0 2024-09-16T01:42:57.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.680+0000 7f0d32ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d200039b0 0x7f0d340b5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:57.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.680+0000 7f0d32ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d200039b0 0x7f0d340b5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35108/0 (socket says 172.21.15.88:35108) 2024-09-16T01:42:57.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.680+0000 7f0d32ffd640 1 -- 172.21.15.88:0/2906344269 learned_addr learned my addr 172.21.15.88:0/2906344269 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:57.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.680+0000 7f0d337fe640 1 --2- 172.21.15.88:0/2906344269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d34150600 0x7f0d340cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:57.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.680+0000 7f0d2a7fc640 1 --2- 172.21.15.88:0/2906344269 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d341531b0 0x7f0d340ad6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:57.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.680+0000 7f0d32ffd640 1 -- 172.21.15.88:0/2906344269 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d341531b0 msgr2=0x7f0d340ad6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:57.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.680+0000 7f0d32ffd640 1 --2- 172.21.15.88:0/2906344269 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d341531b0 0x7f0d340ad6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:57.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.680+0000 7f0d32ffd640 1 -- 172.21.15.88:0/2906344269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d34150600 msgr2=0x7f0d340cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:57.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.680+0000 7f0d32ffd640 1 --2- 172.21.15.88:0/2906344269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d34150600 0x7f0d340cfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:57.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.680+0000 7f0d32ffd640 1 -- 172.21.15.88:0/2906344269 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d340adfa0 con 0x7f0d200039b0 2024-09-16T01:42:57.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.680+0000 7f0d2a7fc640 1 --2- 172.21.15.88:0/2906344269 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d341531b0 0x7f0d340ad6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:57.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.680+0000 7f0d337fe640 1 --2- 172.21.15.88:0/2906344269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d34150600 0x7f0d340cfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:57.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.681+0000 7f0d32ffd640 1 --2- 172.21.15.88:0/2906344269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d200039b0 0x7f0d340b5930 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f0d24009ac0 tx=0x7f0d24004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:57.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.681+0000 7f0d30ff9640 1 -- 172.21.15.88:0/2906344269 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d24040020 con 0x7f0d200039b0 2024-09-16T01:42:57.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.681+0000 7f0d30ff9640 1 -- 172.21.15.88:0/2906344269 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d24003bc0 con 0x7f0d200039b0 2024-09-16T01:42:57.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.681+0000 7f0d39a2e640 1 -- 172.21.15.88:0/2906344269 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d340ae280 con 0x7f0d200039b0 2024-09-16T01:42:57.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.681+0000 7f0d30ff9640 1 -- 172.21.15.88:0/2906344269 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d24036040 con 0x7f0d200039b0 2024-09-16T01:42:57.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.681+0000 7f0d39a2e640 1 -- 172.21.15.88:0/2906344269 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d340ae840 con 0x7f0d200039b0 2024-09-16T01:42:57.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.682+0000 7f0d39a2e640 1 -- 172.21.15.88:0/2906344269 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d3415e970 con 0x7f0d200039b0 2024-09-16T01:42:57.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.682+0000 7f0d30ff9640 1 -- 172.21.15.88:0/2906344269 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0d24043020 con 0x7f0d200039b0 2024-09-16T01:42:57.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.683+0000 7f0d30ff9640 1 --2- 172.21.15.88:0/2906344269 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0d1406c2e0 0x7f0d1406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:57.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.683+0000 7f0d30ff9640 1 -- 172.21.15.88:0/2906344269 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 7974+0+0 (secure 0 0 0) 0x7f0d240abf00 con 0x7f0d200039b0 2024-09-16T01:42:57.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.683+0000 7f0d2a7fc640 1 --2- 172.21.15.88:0/2906344269 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0d1406c2e0 0x7f0d1406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:57.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.683+0000 7f0d2a7fc640 1 --2- 172.21.15.88:0/2906344269 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0d1406c2e0 0x7f0d1406e7a0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f0d341f7a90 tx=0x7f0d1c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:57.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.687+0000 7f0d30ff9640 1 -- 172.21.15.88:0/2906344269 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0d3415e970 con 0x7f0d200039b0 2024-09-16T01:42:57.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.866+0000 7f0d39a2e640 1 -- 172.21.15.88:0/2906344269 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0d340d3070 con 0x7f0d200039b0 2024-09-16T01:42:57.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.867+0000 7f0d30ff9640 1 -- 172.21.15.88:0/2906344269 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 52 entries v113) ==== 81+0+3162 (secure 0 0 0) 0x7f0d2404a070 con 0x7f0d200039b0 2024-09-16T01:42:57.869 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/4222250252 2024-09-17T01:42:45.673956+0000 2024-09-16T01:42:57.870 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2680708900 2024-09-17T01:42:41.629904+0000 2024-09-16T01:42:57.870 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/1990650099 2024-09-17T01:42:37.599896+0000 2024-09-16T01:42:57.870 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/201188184 2024-09-17T01:42:33.276104+0000 2024-09-16T01:42:57.870 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/1990650099 2024-09-17T01:42:37.599896+0000 2024-09-16T01:42:57.870 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/3374372758 2024-09-17T01:42:25.506299+0000 2024-09-16T01:42:57.870 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/3374372758 2024-09-17T01:42:25.506299+0000 2024-09-16T01:42:57.870 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/880314974 2024-09-17T01:42:21.485748+0000 2024-09-16T01:42:57.870 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/1936493048 2024-09-17T01:42:19.133197+0000 2024-09-16T01:42:57.870 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/1936493048 2024-09-17T01:42:19.133197+0000 2024-09-16T01:42:57.871 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/3728363359 2024-09-17T01:42:05.412133+0000 2024-09-16T01:42:57.871 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/3163343083 2024-09-17T01:42:01.375396+0000 2024-09-16T01:42:57.871 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2550498835 2024-09-17T01:41:57.358287+0000 2024-09-16T01:42:57.871 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2285699282 2024-09-17T01:42:29.541113+0000 2024-09-16T01:42:57.871 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2550498835 2024-09-17T01:41:57.358287+0000 2024-09-16T01:42:57.871 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2408579939 2024-09-17T01:40:46.434492+0000 2024-09-16T01:42:57.871 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2733458327 2024-09-17T01:40:27.314712+0000 2024-09-16T01:42:57.871 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/1289932173 2024-09-17T01:40:42.399399+0000 2024-09-16T01:42:57.871 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/4162274767 2024-09-17T01:40:11.522091+0000 2024-09-16T01:42:57.871 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/4222250252 2024-09-17T01:42:45.673956+0000 2024-09-16T01:42:57.871 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/3163343083 2024-09-17T01:42:01.375396+0000 2024-09-16T01:42:57.871 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2408579939 2024-09-17T01:40:46.434492+0000 2024-09-16T01:42:57.871 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/3356432718 2024-09-17T01:40:32.344252+0000 2024-09-16T01:42:57.871 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/1042715824 2024-09-17T01:40:59.080515+0000 2024-09-16T01:42:57.872 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2970185561 2024-09-17T01:40:38.255774+0000 2024-09-16T01:42:57.872 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/3356432718 2024-09-17T01:40:32.344252+0000 2024-09-16T01:42:57.872 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/1289932173 2024-09-17T01:40:42.399399+0000 2024-09-16T01:42:57.872 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2970185561 2024-09-17T01:40:38.255774+0000 2024-09-16T01:42:57.872 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/4162274767 2024-09-17T01:40:11.522091+0000 2024-09-16T01:42:57.872 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/201188184 2024-09-17T01:42:33.276104+0000 2024-09-16T01:42:57.872 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2957965650 2024-09-17T01:41:12.181485+0000 2024-09-16T01:42:57.872 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2733458327 2024-09-17T01:40:27.314712+0000 2024-09-16T01:42:57.872 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/1042715824 2024-09-17T01:40:59.080515+0000 2024-09-16T01:42:57.872 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/128058689 2024-09-17T01:41:10.163743+0000 2024-09-16T01:42:57.872 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2918194823 2024-09-17T01:41:29.275696+0000 2024-09-16T01:42:57.872 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/880314974 2024-09-17T01:42:21.485748+0000 2024-09-16T01:42:57.872 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/981209627 2024-09-17T01:41:51.268495+0000 2024-09-16T01:42:57.873 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/128058689 2024-09-17T01:41:10.163743+0000 2024-09-16T01:42:57.873 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/1162318664 2024-09-17T01:41:23.257870+0000 2024-09-16T01:42:57.873 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2285699282 2024-09-17T01:42:29.541113+0000 2024-09-16T01:42:57.873 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/3336935653 2024-09-17T01:41:44.295023+0000 2024-09-16T01:42:57.873 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2957965650 2024-09-17T01:41:12.181485+0000 2024-09-16T01:42:57.873 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2680708900 2024-09-17T01:42:41.629904+0000 2024-09-16T01:42:57.873 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/293764774 2024-09-17T01:41:16.203758+0000 2024-09-16T01:42:57.873 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/981209627 2024-09-17T01:41:51.268495+0000 2024-09-16T01:42:57.873 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/3336935653 2024-09-17T01:41:44.295023+0000 2024-09-16T01:42:57.873 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2918194823 2024-09-17T01:41:29.275696+0000 2024-09-16T01:42:57.873 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/1255135754 2024-09-17T01:41:41.905258+0000 2024-09-16T01:42:57.873 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/293764774 2024-09-17T01:41:16.203758+0000 2024-09-16T01:42:57.873 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/1162318664 2024-09-17T01:41:23.257870+0000 2024-09-16T01:42:57.874 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/3728363359 2024-09-17T01:42:05.412133+0000 2024-09-16T01:42:57.874 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/1255135754 2024-09-17T01:41:41.905258+0000 2024-09-16T01:42:57.875 INFO:teuthology.orchestra.run.smithi088.stderr:listed 52 entries 2024-09-16T01:42:57.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.869+0000 7f0d39a2e640 1 -- 172.21.15.88:0/2906344269 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0d1406c2e0 msgr2=0x7f0d1406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:57.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.869+0000 7f0d39a2e640 1 --2- 172.21.15.88:0/2906344269 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0d1406c2e0 0x7f0d1406e7a0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f0d341f7a90 tx=0x7f0d1c008040 comp rx=0 tx=0).stop 2024-09-16T01:42:57.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.869+0000 7f0d39a2e640 1 -- 172.21.15.88:0/2906344269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d200039b0 msgr2=0x7f0d340b5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:57.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.869+0000 7f0d39a2e640 1 --2- 172.21.15.88:0/2906344269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d200039b0 0x7f0d340b5930 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f0d24009ac0 tx=0x7f0d24004730 comp rx=0 tx=0).stop 2024-09-16T01:42:57.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.869+0000 7f0d39a2e640 1 -- 172.21.15.88:0/2906344269 shutdown_connections 2024-09-16T01:42:57.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.869+0000 7f0d39a2e640 1 --2- 172.21.15.88:0/2906344269 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0d1406c2e0 0x7f0d1406e7a0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:57.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.869+0000 7f0d39a2e640 1 --2- 172.21.15.88:0/2906344269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d34150600 0x7f0d340cfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:57.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.869+0000 7f0d39a2e640 1 --2- 172.21.15.88:0/2906344269 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0d341531b0 0x7f0d340ad6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:57.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.869+0000 7f0d39a2e640 1 --2- 172.21.15.88:0/2906344269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0d200039b0 0x7f0d340b5930 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:57.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.869+0000 7f0d39a2e640 1 -- 172.21.15.88:0/2906344269 >> 172.21.15.88:0/2906344269 conn(0x7f0d3406c020 msgr2=0x7f0d3415abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:57.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.869+0000 7f0d39a2e640 1 -- 172.21.15.88:0/2906344269 shutdown_connections 2024-09-16T01:42:57.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:57.869+0000 7f0d39a2e640 1 -- 172.21.15.88:0/2906344269 wait complete. 2024-09-16T01:42:57.882 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-16T01:42:57.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:57.882 DEBUG:teuthology.orchestra.run.smithi088:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-09-16T01:42:57.923 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:42:57.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:42:57.924 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:42:57.979 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:42:58.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.116+0000 7fe689da1640 1 Processor -- start 2024-09-16T01:42:58.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.116+0000 7fe689da1640 1 -- start start 2024-09-16T01:42:58.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.117+0000 7fe689da1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe6840cf1f0 0x7fe6840b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:58.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.117+0000 7fe689da1640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe6840b6720 0x7fe6840d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:58.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.117+0000 7fe689da1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6840cf860 con 0x7fe68415e7a0 2024-09-16T01:42:58.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.117+0000 7fe689da1640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fe6840b6b90 con 0x7fe6840b6720 2024-09-16T01:42:58.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.117+0000 7fe689da1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe6840d2ca0 con 0x7fe6840cf1f0 2024-09-16T01:42:58.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.117+0000 7fe683fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe6840b6720 0x7fe6840d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:58.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.117+0000 7fe683fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fe6840b6720 0x7fe6840d3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35120/0 (socket says 172.21.15.88:35120) 2024-09-16T01:42:58.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.117+0000 7fe683fff640 1 -- 172.21.15.88:0/4179888054 learned_addr learned my addr 172.21.15.88:0/4179888054 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:58.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.117+0000 7fe67bfff640 1 --2- 172.21.15.88:0/4179888054 >> v2:172.21.15.154:3300/0 conn(0x7fe6840cf1f0 0x7fe6840b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:58.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.117+0000 7fe683fff640 1 -- 172.21.15.88:0/4179888054 >> v2:172.21.15.154:3300/0 conn(0x7fe6840cf1f0 msgr2=0x7fe6840b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:58.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.117+0000 7fe683fff640 1 --2- 172.21.15.88:0/4179888054 >> v2:172.21.15.154:3300/0 conn(0x7fe6840cf1f0 0x7fe6840b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:58.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.117+0000 7fe683fff640 1 -- 172.21.15.88:0/4179888054 >> v1:172.21.15.154:6789/0 conn(0x7fe68415e7a0 legacy=0x7fe68406caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:58.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.117+0000 7fe683fff640 1 -- 172.21.15.88:0/4179888054 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6840cfb40 con 0x7fe6840b6720 2024-09-16T01:42:58.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.118+0000 7fe683fff640 1 --2- 172.21.15.88:0/4179888054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6840b6720 0x7fe6840d3410 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7fe6840d2110 tx=0x7fe67c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0fa993e24de6cc5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:58.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.118+0000 7fe682ffd640 1 -- 172.21.15.88:0/4179888054 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe67c036070 con 0x7fe6840b6720 2024-09-16T01:42:58.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.118+0000 7fe682ffd640 1 -- 172.21.15.88:0/4179888054 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe67c031600 con 0x7fe6840b6720 2024-09-16T01:42:58.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.118+0000 7fe682ffd640 1 -- 172.21.15.88:0/4179888054 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe67c0319d0 con 0x7fe6840b6720 2024-09-16T01:42:58.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.118+0000 7fe689da1640 1 -- 172.21.15.88:0/4179888054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6840b6720 msgr2=0x7fe6840d3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:58.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.118+0000 7fe689da1640 1 --2- 172.21.15.88:0/4179888054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6840b6720 0x7fe6840d3410 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7fe6840d2110 tx=0x7fe67c02a6a0 comp rx=0 tx=0).stop 2024-09-16T01:42:58.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.118+0000 7fe689da1640 1 -- 172.21.15.88:0/4179888054 shutdown_connections 2024-09-16T01:42:58.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.118+0000 7fe689da1640 1 --2- 172.21.15.88:0/4179888054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6840b6720 0x7fe6840d3410 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:58.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.118+0000 7fe689da1640 1 --2- 172.21.15.88:0/4179888054 >> v2:172.21.15.154:3300/0 conn(0x7fe6840cf1f0 0x7fe6840b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:58.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.119+0000 7fe689da1640 1 -- 172.21.15.88:0/4179888054 >> 172.21.15.88:0/4179888054 conn(0x7fe6840d11e0 msgr2=0x7fe6840d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:58.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.119+0000 7fe689da1640 1 -- 172.21.15.88:0/4179888054 shutdown_connections 2024-09-16T01:42:58.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.119+0000 7fe689da1640 1 -- 172.21.15.88:0/4179888054 wait complete. 2024-09-16T01:42:58.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.120+0000 7fe689da1640 1 Processor -- start 2024-09-16T01:42:58.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.120+0000 7fe689da1640 1 -- start start 2024-09-16T01:42:58.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.120+0000 7fe689da1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6840b6720 0x7fe6840cb830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:58.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.120+0000 7fe689da1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6840cf1f0 0x7fe684147c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:58.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.120+0000 7fe689da1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe68415e7a0 0x7fe684148160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:58.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.120+0000 7fe689da1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe6840b7a00 con 0x7fe6840cf1f0 2024-09-16T01:42:58.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.120+0000 7fe689da1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe6840d2ef0 con 0x7fe68415e7a0 2024-09-16T01:42:58.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.120+0000 7fe689da1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe68415eb80 con 0x7fe6840b6720 2024-09-16T01:42:58.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.121+0000 7fe67bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6840cf1f0 0x7fe684147c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:58.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.121+0000 7fe67bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6840cf1f0 0x7fe684147c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35124/0 (socket says 172.21.15.88:35124) 2024-09-16T01:42:58.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.121+0000 7fe67bfff640 1 -- 172.21.15.88:0/775817957 learned_addr learned my addr 172.21.15.88:0/775817957 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:58.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.121+0000 7fe683fff640 1 --2- 172.21.15.88:0/775817957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe68415e7a0 0x7fe684148160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:58.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.121+0000 7fe6837fe640 1 --2- 172.21.15.88:0/775817957 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6840b6720 0x7fe6840cb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:58.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.121+0000 7fe67bfff640 1 -- 172.21.15.88:0/775817957 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6840b6720 msgr2=0x7fe6840cb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:58.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.121+0000 7fe67bfff640 1 --2- 172.21.15.88:0/775817957 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6840b6720 0x7fe6840cb830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:58.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.121+0000 7fe67bfff640 1 -- 172.21.15.88:0/775817957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe68415e7a0 msgr2=0x7fe684148160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:58.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.121+0000 7fe67bfff640 1 --2- 172.21.15.88:0/775817957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe68415e7a0 0x7fe684148160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:58.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.121+0000 7fe67bfff640 1 -- 172.21.15.88:0/775817957 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6840b7c20 con 0x7fe6840cf1f0 2024-09-16T01:42:58.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.121+0000 7fe6837fe640 1 --2- 172.21.15.88:0/775817957 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6840b6720 0x7fe6840cb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:58.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.121+0000 7fe67bfff640 1 --2- 172.21.15.88:0/775817957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6840cf1f0 0x7fe684147c20 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00d5f0 tx=0x7fe66c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:58.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.121+0000 7fe680ff9640 1 -- 172.21.15.88:0/775817957 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe66c0125f0 con 0x7fe6840cf1f0 2024-09-16T01:42:58.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.121+0000 7fe680ff9640 1 -- 172.21.15.88:0/775817957 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe66c019070 con 0x7fe6840cf1f0 2024-09-16T01:42:58.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.121+0000 7fe689da1640 1 -- 172.21.15.88:0/775817957 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6840cbec0 con 0x7fe6840cf1f0 2024-09-16T01:42:58.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.121+0000 7fe680ff9640 1 -- 172.21.15.88:0/775817957 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe66c0129d0 con 0x7fe6840cf1f0 2024-09-16T01:42:58.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.121+0000 7fe689da1640 1 -- 172.21.15.88:0/775817957 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6840b7ec0 con 0x7fe6840cf1f0 2024-09-16T01:42:58.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.122+0000 7fe689da1640 1 -- 172.21.15.88:0/775817957 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6840b61e0 con 0x7fe6840cf1f0 2024-09-16T01:42:58.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.123+0000 7fe680ff9640 1 -- 172.21.15.88:0/775817957 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe66c012b70 con 0x7fe6840cf1f0 2024-09-16T01:42:58.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.123+0000 7fe680ff9640 1 --2- 172.21.15.88:0/775817957 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe66406c2b0 0x7fe66406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:58.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.123+0000 7fe680ff9640 1 -- 172.21.15.88:0/775817957 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 7974+0+0 (secure 0 0 0) 0x7fe66c08c010 con 0x7fe6840cf1f0 2024-09-16T01:42:58.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.123+0000 7fe6837fe640 1 --2- 172.21.15.88:0/775817957 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe66406c2b0 0x7fe66406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:58.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.124+0000 7fe6837fe640 1 --2- 172.21.15.88:0/775817957 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe66406c2b0 0x7fe66406e770 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fe68415ba00 tx=0x7fe674009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:58.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.127+0000 7fe680ff9640 1 -- 172.21.15.88:0/775817957 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe66c059ea0 con 0x7fe6840cf1f0 2024-09-16T01:42:58.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.327+0000 7fe689da1640 1 -- 172.21.15.88:0/775817957 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fe6840d2a80 con 0x7fe6840cf1f0 2024-09-16T01:42:58.330 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:42:58.327+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:42:58.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.329+0000 7fe680ff9640 1 -- 172.21.15.88:0/775817957 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v231) ==== 161+0+0 (secure 0 0 0) 0x7fe66c0597a0 con 0x7fe6840cf1f0 2024-09-16T01:42:58.331 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:42:58.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.331+0000 7fe689da1640 1 -- 172.21.15.88:0/775817957 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe66406c2b0 msgr2=0x7fe66406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:58.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.331+0000 7fe689da1640 1 --2- 172.21.15.88:0/775817957 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe66406c2b0 0x7fe66406e770 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fe68415ba00 tx=0x7fe674009040 comp rx=0 tx=0).stop 2024-09-16T01:42:58.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.331+0000 7fe689da1640 1 -- 172.21.15.88:0/775817957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6840cf1f0 msgr2=0x7fe684147c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:58.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.331+0000 7fe689da1640 1 --2- 172.21.15.88:0/775817957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6840cf1f0 0x7fe684147c20 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fe66c00d5f0 tx=0x7fe66c00dab0 comp rx=0 tx=0).stop 2024-09-16T01:42:58.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.331+0000 7fe689da1640 1 -- 172.21.15.88:0/775817957 shutdown_connections 2024-09-16T01:42:58.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.331+0000 7fe689da1640 1 --2- 172.21.15.88:0/775817957 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe66406c2b0 0x7fe66406e770 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:58.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.331+0000 7fe689da1640 1 --2- 172.21.15.88:0/775817957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe68415e7a0 0x7fe684148160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:58.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.331+0000 7fe689da1640 1 --2- 172.21.15.88:0/775817957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6840cf1f0 0x7fe684147c20 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:58.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.331+0000 7fe689da1640 1 --2- 172.21.15.88:0/775817957 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6840b6720 0x7fe6840cb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:58.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.331+0000 7fe689da1640 1 -- 172.21.15.88:0/775817957 >> 172.21.15.88:0/775817957 conn(0x7fe6840d11e0 msgr2=0x7fe6840d15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:58.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.331+0000 7fe689da1640 1 -- 172.21.15.88:0/775817957 shutdown_connections 2024-09-16T01:42:58.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.331+0000 7fe689da1640 1 -- 172.21.15.88:0/775817957 wait complete. 2024-09-16T01:42:58.335 INFO:tasks.ceph.mds.a.smithi088.stderr: -525> 2024-09-16T01:42:56.687+0000 7efe9b874640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:42:58.345 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:42:58.359 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:42:58.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.442+0000 7f53f16f1640 1 Processor -- start 2024-09-16T01:42:58.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.442+0000 7f53f16f1640 1 -- start start 2024-09-16T01:42:58.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.443+0000 7f53f16f1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f53ec0b6290 0x7f53ec071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:58.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.443+0000 7f53f16f1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f53ec072530 0x7f53ec15bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:58.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.443+0000 7f53f16f1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53ec0ccff0 con 0x7f53ec15b480 2024-09-16T01:42:58.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.443+0000 7f53f16f1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f53ec0cd2d0 con 0x7f53ec072530 2024-09-16T01:42:58.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.443+0000 7f53f16f1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f53ec0cd5b0 con 0x7f53ec0b6290 2024-09-16T01:42:58.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.443+0000 7f53eaffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f53ec15b480 0x7f53ec0b5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42514/0 (socket says 172.21.15.88:42514) 2024-09-16T01:42:58.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.443+0000 7f53eaffd640 1 -- 172.21.15.88:0/3720098227 learned_addr learned my addr 172.21.15.88:0/3720098227 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:42:58.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.443+0000 7f53ea7fc640 1 --2- 172.21.15.88:0/3720098227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f53ec0b6290 0x7f53ec071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:58.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.443+0000 7f53eb7fe640 1 --2- 172.21.15.88:0/3720098227 >> v2:172.21.15.154:3300/0 conn(0x7f53ec072530 0x7f53ec15bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:58.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.443+0000 7f53e9ffb640 1 -- 172.21.15.88:0/3720098227 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4261066403 0 0) 0x7f53ec0ccff0 con 0x7f53ec15b480 2024-09-16T01:42:58.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.443+0000 7f53e9ffb640 1 -- 172.21.15.88:0/3720098227 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53d4003520 con 0x7f53ec15b480 2024-09-16T01:42:58.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.444+0000 7f53e9ffb640 1 -- 172.21.15.88:0/3720098227 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2248938277 0 0) 0x7f53d4003520 con 0x7f53ec15b480 2024-09-16T01:42:58.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.444+0000 7f53e9ffb640 1 -- 172.21.15.88:0/3720098227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f53ec0b6290 msgr2=0x7f53ec071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:58.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.444+0000 7f53e9ffb640 1 --2- 172.21.15.88:0/3720098227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f53ec0b6290 0x7f53ec071ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:58.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.444+0000 7f53e9ffb640 1 -- 172.21.15.88:0/3720098227 >> v2:172.21.15.154:3300/0 conn(0x7f53ec072530 msgr2=0x7f53ec15bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:58.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.444+0000 7f53e9ffb640 1 --2- 172.21.15.88:0/3720098227 >> v2:172.21.15.154:3300/0 conn(0x7f53ec072530 0x7f53ec15bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:58.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.444+0000 7f53e9ffb640 1 -- 172.21.15.88:0/3720098227 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53ec0cd890 con 0x7f53ec15b480 2024-09-16T01:42:58.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.444+0000 7f53e9ffb640 1 -- 172.21.15.88:0/3720098227 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f53dc001670 con 0x7f53ec15b480 2024-09-16T01:42:58.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.444+0000 7f53e9ffb640 1 -- 172.21.15.88:0/3720098227 >> v1:172.21.15.88:6789/0 conn(0x7f53ec15b480 legacy=0x7f53ec0b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:58.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.444+0000 7f53e9ffb640 1 --2- 172.21.15.88:0/3720098227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53d40039b0 0x7f53d4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:58.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.444+0000 7f53e9ffb640 1 --2- 172.21.15.88:0/3720098227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53d4004800 0x7f53d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:58.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.444+0000 7f53e9ffb640 1 --2- 172.21.15.88:0/3720098227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f53d40051c0 0x7f53d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:58.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.444+0000 7f53e9ffb640 1 -- 172.21.15.88:0/3720098227 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f53ec0cd2d0 con 0x7f53d40039b0 2024-09-16T01:42:58.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.444+0000 7f53e9ffb640 1 -- 172.21.15.88:0/3720098227 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f53ec0cd5b0 con 0x7f53d4004800 2024-09-16T01:42:58.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.444+0000 7f53e9ffb640 1 -- 172.21.15.88:0/3720098227 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f53d400bf00 con 0x7f53d40051c0 2024-09-16T01:42:58.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.444+0000 7f53eaffd640 1 --2- 172.21.15.88:0/3720098227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53d40039b0 0x7f53d4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:58.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.444+0000 7f53ea7fc640 1 --2- 172.21.15.88:0/3720098227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53d4004800 0x7f53d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:58.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.444+0000 7f53eb7fe640 1 --2- 172.21.15.88:0/3720098227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f53d40051c0 0x7f53d400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:58.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.444+0000 7f53eaffd640 1 -- 172.21.15.88:0/3720098227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f53d40051c0 msgr2=0x7f53d400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:58.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.444+0000 7f53eaffd640 1 --2- 172.21.15.88:0/3720098227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f53d40051c0 0x7f53d400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:58.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.444+0000 7f53eaffd640 1 -- 172.21.15.88:0/3720098227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53d4004800 msgr2=0x7f53d4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:58.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.444+0000 7f53eaffd640 1 --2- 172.21.15.88:0/3720098227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53d4004800 0x7f53d4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:58.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.444+0000 7f53eaffd640 1 -- 172.21.15.88:0/3720098227 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d4003520 con 0x7f53d40039b0 2024-09-16T01:42:58.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.445+0000 7f53eaffd640 1 --2- 172.21.15.88:0/3720098227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53d40039b0 0x7f53d4003dd0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f53dc00c290 tx=0x7f53d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=accfad27b9a6025e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:58.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.445+0000 7f53e9ffb640 1 -- 172.21.15.88:0/3720098227 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53dc007d50 con 0x7f53d40039b0 2024-09-16T01:42:58.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.445+0000 7f53e9ffb640 1 -- 172.21.15.88:0/3720098227 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53dc005ad0 con 0x7f53d40039b0 2024-09-16T01:42:58.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.445+0000 7f53e9ffb640 1 -- 172.21.15.88:0/3720098227 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53dc007070 con 0x7f53d40039b0 2024-09-16T01:42:58.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.445+0000 7f53f16f1640 1 -- 172.21.15.88:0/3720098227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53d40039b0 msgr2=0x7f53d4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:58.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.445+0000 7f53f16f1640 1 --2- 172.21.15.88:0/3720098227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53d40039b0 0x7f53d4003dd0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f53dc00c290 tx=0x7f53d4005830 comp rx=0 tx=0).stop 2024-09-16T01:42:58.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.445+0000 7f53eb7fe640 1 -- 172.21.15.88:0/3720098227 reap_dead start 2024-09-16T01:42:58.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.445+0000 7f53f16f1640 1 -- 172.21.15.88:0/3720098227 shutdown_connections 2024-09-16T01:42:58.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.445+0000 7f53f16f1640 1 -- 172.21.15.88:0/3720098227 >> 172.21.15.88:0/3720098227 conn(0x7f53ec06d0f0 msgr2=0x7f53ec06d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:58.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.445+0000 7f53f16f1640 1 -- 172.21.15.88:0/3720098227 shutdown_connections 2024-09-16T01:42:58.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.446+0000 7f53f16f1640 1 -- 172.21.15.88:0/3720098227 wait complete. 2024-09-16T01:42:58.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.447+0000 7f53f16f1640 1 Processor -- start 2024-09-16T01:42:58.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.447+0000 7f53f16f1640 1 -- start start 2024-09-16T01:42:58.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.447+0000 7f53f16f1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53d40039b0 0x7f53ec1493f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:58.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.448+0000 7f53f16f1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f53ec14d170 0x7f53ec0d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:58.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.448+0000 7f53f16f1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53ec14d5f0 0x7f53ec0d0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:58.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.448+0000 7f53f16f1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f53ec15b9e0 con 0x7f53d40039b0 2024-09-16T01:42:58.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.448+0000 7f53f16f1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f53ec15b860 con 0x7f53ec14d5f0 2024-09-16T01:42:58.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.448+0000 7f53f16f1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f53ec071ff0 con 0x7f53ec14d170 2024-09-16T01:42:58.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.448+0000 7f53eaffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53d40039b0 0x7f53ec1493f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:58.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.448+0000 7f53eaffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53d40039b0 0x7f53ec1493f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35152/0 (socket says 172.21.15.88:35152) 2024-09-16T01:42:58.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.448+0000 7f53eaffd640 1 -- 172.21.15.88:0/210127652 learned_addr learned my addr 172.21.15.88:0/210127652 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:58.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.448+0000 7f53ea7fc640 1 --2- 172.21.15.88:0/210127652 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f53ec14d170 0x7f53ec0d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:58.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.448+0000 7f53eb7fe640 1 --2- 172.21.15.88:0/210127652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53ec14d5f0 0x7f53ec0d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:58.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.448+0000 7f53eaffd640 1 -- 172.21.15.88:0/210127652 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f53ec14d170 msgr2=0x7f53ec0d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:58.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.448+0000 7f53eaffd640 1 --2- 172.21.15.88:0/210127652 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f53ec14d170 0x7f53ec0d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:58.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.448+0000 7f53eaffd640 1 -- 172.21.15.88:0/210127652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53ec14d5f0 msgr2=0x7f53ec0d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:58.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.448+0000 7f53eaffd640 1 --2- 172.21.15.88:0/210127652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53ec14d5f0 0x7f53ec0d0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:58.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.448+0000 7f53eaffd640 1 -- 172.21.15.88:0/210127652 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53ec14e460 con 0x7f53d40039b0 2024-09-16T01:42:58.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.448+0000 7f53eaffd640 1 --2- 172.21.15.88:0/210127652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53d40039b0 0x7f53ec1493f0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f53dc02ae50 tx=0x7f53ec0cd1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:58.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.448+0000 7f53d37fe640 1 -- 172.21.15.88:0/210127652 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53dc042070 con 0x7f53d40039b0 2024-09-16T01:42:58.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.448+0000 7f53d37fe640 1 -- 172.21.15.88:0/210127652 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53dc007480 con 0x7f53d40039b0 2024-09-16T01:42:58.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.448+0000 7f53f16f1640 1 -- 172.21.15.88:0/210127652 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53ec14dbc0 con 0x7f53d40039b0 2024-09-16T01:42:58.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.448+0000 7f53d37fe640 1 -- 172.21.15.88:0/210127652 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53dc001c80 con 0x7f53d40039b0 2024-09-16T01:42:58.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.448+0000 7f53f16f1640 1 -- 172.21.15.88:0/210127652 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53ec14e180 con 0x7f53d40039b0 2024-09-16T01:42:58.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.449+0000 7f53f16f1640 1 -- 172.21.15.88:0/210127652 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53ec0b5b80 con 0x7f53d40039b0 2024-09-16T01:42:58.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.450+0000 7f53d37fe640 1 -- 172.21.15.88:0/210127652 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f53dc045070 con 0x7f53d40039b0 2024-09-16T01:42:58.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.450+0000 7f53d37fe640 1 --2- 172.21.15.88:0/210127652 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f53d806c2b0 0x7f53d806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:58.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.450+0000 7f53d37fe640 1 -- 172.21.15.88:0/210127652 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8060+0+0 (secure 0 0 0) 0x7f53dc0acc50 con 0x7f53d40039b0 2024-09-16T01:42:58.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.450+0000 7f53ea7fc640 1 --2- 172.21.15.88:0/210127652 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f53d806c2b0 0x7f53d806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:58.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.451+0000 7f53ea7fc640 1 --2- 172.21.15.88:0/210127652 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f53d806c2b0 0x7f53d806e770 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f53ec155c60 tx=0x7f53e0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:58.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.455+0000 7f53d37fe640 1 -- 172.21.15.88:0/210127652 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f53dc0b2020 con 0x7f53d40039b0 2024-09-16T01:42:58.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:58.693+0000 7f53f16f1640 1 -- 172.21.15.88:0/210127652 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f53ec15d160 con 0x7f53d40039b0 2024-09-16T01:42:59.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.333+0000 7f53d37fe640 1 -- 172.21.15.88:0/210127652 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v232) ==== 107+0+0 (secure 0 0 0) 0x7f53dc04c020 con 0x7f53d40039b0 2024-09-16T01:42:59.336 INFO:teuthology.orchestra.run.smithi088.stderr:nothing to do: rank is not damaged 2024-09-16T01:42:59.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.336+0000 7f53f16f1640 1 -- 172.21.15.88:0/210127652 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f53d806c2b0 msgr2=0x7f53d806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:59.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.336+0000 7f53f16f1640 1 --2- 172.21.15.88:0/210127652 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f53d806c2b0 0x7f53d806e770 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f53ec155c60 tx=0x7f53e0008040 comp rx=0 tx=0).stop 2024-09-16T01:42:59.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.336+0000 7f53f16f1640 1 -- 172.21.15.88:0/210127652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53d40039b0 msgr2=0x7f53ec1493f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:59.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.336+0000 7f53f16f1640 1 --2- 172.21.15.88:0/210127652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53d40039b0 0x7f53ec1493f0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f53dc02ae50 tx=0x7f53ec0cd1c0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.336+0000 7f53f16f1640 1 -- 172.21.15.88:0/210127652 shutdown_connections 2024-09-16T01:42:59.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.336+0000 7f53f16f1640 1 --2- 172.21.15.88:0/210127652 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f53d806c2b0 0x7f53d806e770 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.336+0000 7f53f16f1640 1 --2- 172.21.15.88:0/210127652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53ec14d5f0 0x7f53ec0d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.336+0000 7f53f16f1640 1 --2- 172.21.15.88:0/210127652 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f53ec14d170 0x7f53ec0d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.336+0000 7f53f16f1640 1 --2- 172.21.15.88:0/210127652 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53d40039b0 0x7f53ec1493f0 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.336+0000 7f53f16f1640 1 -- 172.21.15.88:0/210127652 >> 172.21.15.88:0/210127652 conn(0x7f53ec06d0f0 msgr2=0x7f53ec15c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:59.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.336+0000 7f53f16f1640 1 -- 172.21.15.88:0/210127652 shutdown_connections 2024-09-16T01:42:59.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.336+0000 7f53f16f1640 1 -- 172.21.15.88:0/210127652 wait complete. 2024-09-16T01:42:59.350 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:42:59.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.432+0000 7fbd53905f40 1 Processor -- start 2024-09-16T01:42:59.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.432+0000 7fbd53905f40 1 -- start start 2024-09-16T01:42:59.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.433+0000 7fbd53905f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd400030f0 0x55f21bef5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.433+0000 7fbd53905f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21be50d00 0x55f21be4ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.433+0000 7fbd53905f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21be4b020 0x55f21beef2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.433+0000 7fbd53905f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f21bd72510 con 0x55f21be4b020 2024-09-16T01:42:59.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.433+0000 7fbd53905f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f21bdcbc50 con 0x55f21be50d00 2024-09-16T01:42:59.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.433+0000 7fbd53905f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55f21be22a70 con 0x7fbd400030f0 2024-09-16T01:42:59.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.433+0000 7fbd51bfc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21be4b020 0x55f21beef2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:59.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.433+0000 7fbd51bfc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21be4b020 0x55f21beef2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35176/0 (socket says 172.21.15.88:35176) 2024-09-16T01:42:59.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.433+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1058710165 learned_addr learned my addr 172.21.15.88:0/1058710165 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:59.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.433+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1058710165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd400030f0 msgr2=0x55f21bef5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:59.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.433+0000 7fbd51bfc640 1 --2- 172.21.15.88:0/1058710165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd400030f0 0x55f21bef5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.433+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1058710165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21be50d00 msgr2=0x55f21be4ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:59.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.433+0000 7fbd51bfc640 1 --2- 172.21.15.88:0/1058710165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21be50d00 0x55f21be4ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.433+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1058710165 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f21bef6270 con 0x55f21be4b020 2024-09-16T01:42:59.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.433+0000 7fbd51bfc640 1 --2- 172.21.15.88:0/1058710165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21be4b020 0x55f21beef2b0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x55f21bef6620 tx=0x7fbd4c005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f11bc02c65601bd1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:59.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.433+0000 7fbd4bfff640 1 -- 172.21.15.88:0/1058710165 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c013740 con 0x55f21be4b020 2024-09-16T01:42:59.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.433+0000 7fbd4bfff640 1 -- 172.21.15.88:0/1058710165 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd4c013900 con 0x55f21be4b020 2024-09-16T01:42:59.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.433+0000 7fbd4bfff640 1 -- 172.21.15.88:0/1058710165 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c013af0 con 0x55f21be4b020 2024-09-16T01:42:59.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.434+0000 7fbd53905f40 1 -- 172.21.15.88:0/1058710165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21be4b020 msgr2=0x55f21beef2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:59.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.434+0000 7fbd53905f40 1 --2- 172.21.15.88:0/1058710165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21be4b020 0x55f21beef2b0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x55f21bef6620 tx=0x7fbd4c005d80 comp rx=0 tx=0).stop 2024-09-16T01:42:59.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.434+0000 7fbd53905f40 1 -- 172.21.15.88:0/1058710165 shutdown_connections 2024-09-16T01:42:59.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.434+0000 7fbd53905f40 1 --2- 172.21.15.88:0/1058710165 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21be4b020 0x55f21beef2b0 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.434+0000 7fbd53905f40 1 --2- 172.21.15.88:0/1058710165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21be50d00 0x55f21be4ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.434+0000 7fbd53905f40 1 --2- 172.21.15.88:0/1058710165 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd400030f0 0x55f21bef5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.434+0000 7fbd53905f40 1 -- 172.21.15.88:0/1058710165 >> 172.21.15.88:0/1058710165 conn(0x55f21be4b680 msgr2=0x55f21be4ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:59.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.434+0000 7fbd53905f40 1 -- 172.21.15.88:0/1058710165 shutdown_connections 2024-09-16T01:42:59.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.434+0000 7fbd53905f40 1 -- 172.21.15.88:0/1058710165 wait complete. 2024-09-16T01:42:59.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.434+0000 7fbd53905f40 1 Processor -- start 2024-09-16T01:42:59.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.434+0000 7fbd53905f40 1 -- start start 2024-09-16T01:42:59.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.435+0000 7fbd53905f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21be4b020 0x55f21bee9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.435+0000 7fbd53905f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f21be50d00 0x55f21bedb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.435+0000 7fbd53905f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd400030f0 0x55f21beeda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.435+0000 7fbd53905f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f21be1f4a0 con 0x55f21be4b020 2024-09-16T01:42:59.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.435+0000 7fbd53905f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f21beefe00 con 0x7fbd400030f0 2024-09-16T01:42:59.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.435+0000 7fbd53905f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55f21bef4b20 con 0x55f21be50d00 2024-09-16T01:42:59.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.435+0000 7fbd513fb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21be4b020 0x55f21bee9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:59.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.435+0000 7fbd513fb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21be4b020 0x55f21bee9010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35184/0 (socket says 172.21.15.88:35184) 2024-09-16T01:42:59.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.435+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 learned_addr learned my addr 172.21.15.88:0/1723033203 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:59.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.435+0000 7fbd51bfc640 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd400030f0 0x55f21beeda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:59.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.435+0000 7fbd48bfa640 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f21be50d00 0x55f21bedb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:59.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.435+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f21be50d00 msgr2=0x55f21bedb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:59.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.435+0000 7fbd513fb640 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f21be50d00 0x55f21bedb160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.435+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd400030f0 msgr2=0x55f21beeda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:59.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.435+0000 7fbd513fb640 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd400030f0 0x55f21beeda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.436+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f21beedf80 con 0x55f21be4b020 2024-09-16T01:42:59.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.436+0000 7fbd51bfc640 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd400030f0 0x55f21beeda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:42:59.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.436+0000 7fbd513fb640 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21be4b020 0x55f21bee9010 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fbd44001ed0 tx=0x7fbd4400b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:59.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.436+0000 7fbd4affd640 1 -- 172.21.15.88:0/1723033203 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd44013070 con 0x55f21be4b020 2024-09-16T01:42:59.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.436+0000 7fbd4affd640 1 -- 172.21.15.88:0/1723033203 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd4400f880 con 0x55f21be4b020 2024-09-16T01:42:59.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.436+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f21beee220 con 0x55f21be4b020 2024-09-16T01:42:59.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.436+0000 7fbd4affd640 1 -- 172.21.15.88:0/1723033203 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4400fc70 con 0x55f21be4b020 2024-09-16T01:42:59.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.436+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f21beee7e0 con 0x55f21be4b020 2024-09-16T01:42:59.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.437+0000 7fbd4affd640 1 -- 172.21.15.88:0/1723033203 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fbd44016070 con 0x55f21be4b020 2024-09-16T01:42:59.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.437+0000 7fbd4affd640 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbd3406ac90 0x7fbd3406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.438+0000 7fbd4affd640 1 -- 172.21.15.88:0/1723033203 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8060+0+0 (secure 0 0 0) 0x7fbd4408ae50 con 0x55f21be4b020 2024-09-16T01:42:59.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.438+0000 7fbd48bfa640 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbd3406ac90 0x7fbd3406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:59.444 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:42:59.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.438+0000 7fbd53905f40 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55f21be4bf70 0x55f21bf8d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.438+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf8d5b0 con 0x55f21be4bf70 2024-09-16T01:42:59.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.438+0000 7fbd51bfc640 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55f21be4bf70 0x55f21bf8d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:59.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.438+0000 7fbd48bfa640 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbd3406ac90 0x7fbd3406d150 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c054000 tx=0x7fbd3c04e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:59.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.438+0000 7fbd51bfc640 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55f21be4bf70 0x55f21bf8d070 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:59.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.440+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v114'815 uv789 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.440+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf8d9b0 con 0x55f21be4bf70 2024-09-16T01:42:59.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.442+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v114'816 uv816 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.442+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf99630 con 0x55f21be4bf70 2024-09-16T01:42:59.446 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:42:59.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.443+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v114'817 uv817 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.443+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9a9e0 con 0x55f21be4bf70 2024-09-16T01:42:59.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.444+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v114'818 uv818 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.444+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9ad80 con 0x55f21be4bf70 2024-09-16T01:42:59.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.445+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v114'819 uv819 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.445+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf999d0 con 0x55f21be4bf70 2024-09-16T01:42:59.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.446+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v114'820 uv820 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.446+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf99d90 con 0x55f21be4bf70 2024-09-16T01:42:59.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.448+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v114'821 uv795 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.448+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9a130 con 0x55f21be4bf70 2024-09-16T01:42:59.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.449+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v114'822 uv822 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.449+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9a130 con 0x55f21be4bf70 2024-09-16T01:42:59.452 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:42:59.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.450+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v114'823 uv823 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.450+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9a130 con 0x55f21be4bf70 2024-09-16T01:42:59.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.451+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v114'824 uv824 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.451+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9a130 con 0x55f21be4bf70 2024-09-16T01:42:59.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.452+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v114'825 uv825 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.452+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9a130 con 0x55f21be4bf70 2024-09-16T01:42:59.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.454+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v114'826 uv826 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.454+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9a130 con 0x55f21be4bf70 2024-09-16T01:42:59.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.455+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v114'827 uv802 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.455+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9a130 con 0x55f21be4bf70 2024-09-16T01:42:59.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.456+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v114'828 uv828 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.456+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9a130 con 0x55f21be4bf70 2024-09-16T01:42:59.459 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:42:59.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.457+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v114'829 uv829 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.457+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9a130 con 0x55f21be4bf70 2024-09-16T01:42:59.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.459+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v114'830 uv830 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.459+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9a130 con 0x55f21be4bf70 2024-09-16T01:42:59.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.460+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v114'831 uv831 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.460+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9a130 con 0x55f21be4bf70 2024-09-16T01:42:59.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.461+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v114'832 uv832 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.461+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9b120 con 0x55f21be4bf70 2024-09-16T01:42:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.462+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v114'833 uv833 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.462+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9b120 con 0x55f21be4bf70 2024-09-16T01:42:59.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.464+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v114'834 uv808 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.464+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9b120 con 0x55f21be4bf70 2024-09-16T01:42:59.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.465+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v114'835 uv835 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.465+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9b120 con 0x55f21be4bf70 2024-09-16T01:42:59.469 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:42:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.467+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v114'836 uv836 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.467+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9c130 con 0x55f21be4bf70 2024-09-16T01:42:59.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.468+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v114'837 uv837 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.468+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9c130 con 0x55f21be4bf70 2024-09-16T01:42:59.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.469+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v114'838 uv838 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.469+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9b120 con 0x55f21be4bf70 2024-09-16T01:42:59.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.470+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v114'839 uv839 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.470+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9b120 con 0x55f21be4bf70 2024-09-16T01:42:59.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.472+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v114'840 uv814 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.472+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9b120 con 0x55f21be4bf70 2024-09-16T01:42:59.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.473+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v114'841 uv841 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.473+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9b120 con 0x55f21be4bf70 2024-09-16T01:42:59.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.474+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v114'842 uv842 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.477 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:42:59.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.474+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9c130 con 0x55f21be4bf70 2024-09-16T01:42:59.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.475+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v114'843 uv843 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.475+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9c130 con 0x55f21be4bf70 2024-09-16T01:42:59.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.477+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v114'844 uv844 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.477+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9b120 con 0x55f21be4bf70 2024-09-16T01:42:59.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.478+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v114'845 uv845 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.478+0000 7fbd53905f40 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55f21bf9b630 0x55f21bf9da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.478+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9df60 con 0x55f21bf9b630 2024-09-16T01:42:59.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.478+0000 7fbd513fb640 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55f21bf9b630 0x55f21bf9da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:59.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.479+0000 7fbd513fb640 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55f21bf9b630 0x55f21bf9da20 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:59.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.480+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v114'597 uv576 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bf9b630 2024-09-16T01:42:59.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.480+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9e300 con 0x55f21bf9b630 2024-09-16T01:42:59.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.482+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v114'598 uv598 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bf9b630 2024-09-16T01:42:59.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.482+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9e6a0 con 0x55f21bf9b630 2024-09-16T01:42:59.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.483+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v114'599 uv599 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bf9b630 2024-09-16T01:42:59.485 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:42:59.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.483+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9ea40 con 0x55f21bf9b630 2024-09-16T01:42:59.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.484+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v114'600 uv600 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bf9b630 2024-09-16T01:42:59.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.484+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9ede0 con 0x55f21bf9b630 2024-09-16T01:42:59.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.485+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v114'601 uv601 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bf9b630 2024-09-16T01:42:59.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.485+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9f180 con 0x55f21bf9b630 2024-09-16T01:42:59.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.487+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v114'602 uv595 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bf9b630 2024-09-16T01:42:59.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.487+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9f520 con 0x55f21bf9b630 2024-09-16T01:42:59.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.488+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v114'603 uv603 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bf9b630 2024-09-16T01:42:59.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.488+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9f520 con 0x55f21bf9b630 2024-09-16T01:42:59.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.489+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v114'604 uv604 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bf9b630 2024-09-16T01:42:59.492 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:42:59.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.489+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9f520 con 0x55f21bf9b630 2024-09-16T01:42:59.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.491+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v114'605 uv605 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bf9b630 2024-09-16T01:42:59.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.491+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9f520 con 0x55f21bf9b630 2024-09-16T01:42:59.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.492+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v114'606 uv606 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bf9b630 2024-09-16T01:42:59.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.492+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9f520 con 0x55f21bf9b630 2024-09-16T01:42:59.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.493+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v114'607 uv596 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bf9b630 2024-09-16T01:42:59.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.493+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9f520 con 0x55f21bf9b630 2024-09-16T01:42:59.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.494+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v114'608 uv608 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bf9b630 2024-09-16T01:42:59.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.494+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9f520 con 0x55f21bf9b630 2024-09-16T01:42:59.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.495+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v114'609 uv609 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bf9b630 2024-09-16T01:42:59.498 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:42:59.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.495+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa39a0 con 0x55f21bf9b630 2024-09-16T01:42:59.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.497+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v114'610 uv610 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bf9b630 2024-09-16T01:42:59.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.497+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa39a0 con 0x55f21bf9b630 2024-09-16T01:42:59.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.498+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v114'611 uv611 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bf9b630 2024-09-16T01:42:59.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.498+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9f520 con 0x55f21bf9b630 2024-09-16T01:42:59.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.499+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v114'612 uv592 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bf9b630 2024-09-16T01:42:59.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.499+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9f520 con 0x55f21bf9b630 2024-09-16T01:42:59.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.500+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v114'613 uv613 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bf9b630 2024-09-16T01:42:59.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.500+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9f520 con 0x55f21bf9b630 2024-09-16T01:42:59.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.501+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v114'614 uv614 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bf9b630 2024-09-16T01:42:59.503 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:42:59.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.501+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa0530 con 0x55f21bf9b630 2024-09-16T01:42:59.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.502+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v114'615 uv615 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bf9b630 2024-09-16T01:42:59.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.502+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa0530 con 0x55f21bf9b630 2024-09-16T01:42:59.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.503+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v114'616 uv616 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bf9b630 2024-09-16T01:42:59.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.503+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9f520 con 0x55f21bf9b630 2024-09-16T01:42:59.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.504+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v114'617 uv617 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bf9b630 2024-09-16T01:42:59.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.504+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9f520 con 0x55f21be4bf70 2024-09-16T01:42:59.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.505+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v114'159 uv158 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.505+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9f520 con 0x55f21be4bf70 2024-09-16T01:42:59.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.506+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v114'160 uv160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.506+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9f520 con 0x55f21be4bf70 2024-09-16T01:42:59.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.507+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v114'161 uv161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.509 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:42:59.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.507+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa0530 con 0x55f21be4bf70 2024-09-16T01:42:59.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.508+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v114'162 uv162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.508+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa0530 con 0x55f21be4bf70 2024-09-16T01:42:59.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.509+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v114'163 uv163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.509+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bf9f520 con 0x55f21be4bf70 2024-09-16T01:42:59.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.509+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v114'164 uv164 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.510+0000 7fbd53905f40 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55f21bf9fba0 0x55f21bfa1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.510+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa25c0 con 0x55f21bf9fba0 2024-09-16T01:42:59.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.510+0000 7fbd48bfa640 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55f21bf9fba0 0x55f21bfa1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:59.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.510+0000 7fbd48bfa640 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55f21bf9fba0 0x55f21bfa1f90 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:59.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.512+0000 7fbd48bfa640 1 -- 172.21.15.88:0/1723033203 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v114'119 uv118 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fbd3c0547a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.512+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa2ab0 con 0x55f21bf9fba0 2024-09-16T01:42:59.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.513+0000 7fbd48bfa640 1 -- 172.21.15.88:0/1723033203 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v114'120 uv120 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbd3c0547a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.513+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa2fb0 con 0x55f21bf9fba0 2024-09-16T01:42:59.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.514+0000 7fbd48bfa640 1 -- 172.21.15.88:0/1723033203 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v114'121 uv121 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbd3c0547a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.516 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:42:59.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.514+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bef2c40 con 0x55f21bf9fba0 2024-09-16T01:42:59.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.515+0000 7fbd48bfa640 1 -- 172.21.15.88:0/1723033203 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v114'122 uv122 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbd3c0547a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.515+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa38e0 con 0x55f21be4bf70 2024-09-16T01:42:59.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.516+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v114'159 uv158 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.516+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa38e0 con 0x55f21be4bf70 2024-09-16T01:42:59.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.518+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v114'160 uv160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.518+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa38e0 con 0x55f21be4bf70 2024-09-16T01:42:59.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.519+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v114'161 uv161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.521 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:42:59.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.519+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa38e0 con 0x55f21be4bf70 2024-09-16T01:42:59.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.520+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v114'162 uv162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.520+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa38e0 con 0x55f21be4bf70 2024-09-16T01:42:59.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.521+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v114'163 uv163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.521+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa3920 con 0x55f21be4bf70 2024-09-16T01:42:59.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.524+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v114'164 uv164 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.524+0000 7fbd53905f40 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f21bfa4240 0x55f21bfa4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.524+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa6d50 con 0x55f21bfa4240 2024-09-16T01:42:59.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.524+0000 7fbd51bfc640 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f21bfa4240 0x55f21bfa4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:59.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.524+0000 7fbd51bfc640 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f21bfa4240 0x55f21bfa4620 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:59.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.525+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v114'294 uv287 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.526+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa6d50 con 0x55f21bfa4240 2024-09-16T01:42:59.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.526+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v114'295 uv295 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.527+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa6d50 con 0x55f21bfa4240 2024-09-16T01:42:59.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.528+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v114'296 uv296 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.530 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:42:59.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.528+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa7da0 con 0x55f21bfa4240 2024-09-16T01:42:59.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.529+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v114'297 uv297 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.529+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa7d60 con 0x55f21bfa4240 2024-09-16T01:42:59.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.530+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v114'298 uv298 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.530+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa6d50 con 0x55f21bfa4240 2024-09-16T01:42:59.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.531+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v114'299 uv299 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.531+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa6d50 con 0x55f21bfa4240 2024-09-16T01:42:59.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.532+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v114'300 uv293 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.532+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa6d50 con 0x55f21bfa4240 2024-09-16T01:42:59.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.533+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v114'301 uv301 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.533+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa6d50 con 0x55f21bfa4240 2024-09-16T01:42:59.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.534+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v114'302 uv302 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.536 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:42:59.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.534+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa6d50 con 0x55f21bfa4240 2024-09-16T01:42:59.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.535+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v114'303 uv303 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.535+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfa6d50 con 0x55f21bfa4240 2024-09-16T01:42:59.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.536+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v114'304 uv304 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.536+0000 7fbd53905f40 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55f21bfa7650 0x55f21bfa7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.536+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfaa150 con 0x55f21bfa7650 2024-09-16T01:42:59.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.537+0000 7fbd513fb640 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55f21bfa7650 0x55f21bfa7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:59.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.537+0000 7fbd513fb640 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55f21bfa7650 0x55f21bfa7a90 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:59.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.538+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v114'132 uv131 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bfa7650 2024-09-16T01:42:59.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.538+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfaa150 con 0x55f21bfa7650 2024-09-16T01:42:59.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.539+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v114'133 uv133 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bfa7650 2024-09-16T01:42:59.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.539+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfaa1d0 con 0x55f21bfa7650 2024-09-16T01:42:59.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.540+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v114'134 uv134 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bfa7650 2024-09-16T01:42:59.542 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:42:59.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.540+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfaa5c0 con 0x55f21bfa7650 2024-09-16T01:42:59.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.541+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v114'135 uv135 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bfa7650 2024-09-16T01:42:59.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.541+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfaa5c0 con 0x55f21bfa7650 2024-09-16T01:42:59.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.542+0000 7fbd513fb640 1 -- 172.21.15.88:0/1723033203 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v114'136 uv136 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbd4401f070 con 0x55f21bfa7650 2024-09-16T01:42:59.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.542+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfaa5c0 con 0x55f21be4bf70 2024-09-16T01:42:59.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.543+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v114'159 uv158 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.543+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfaa5c0 con 0x55f21be4bf70 2024-09-16T01:42:59.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.544+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v114'160 uv160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.544+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfaa5c0 con 0x55f21be4bf70 2024-09-16T01:42:59.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.545+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v114'161 uv161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.547 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:42:59.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.545+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfab5d0 con 0x55f21be4bf70 2024-09-16T01:42:59.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.546+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v114'162 uv162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.546+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfab5d0 con 0x55f21be4bf70 2024-09-16T01:42:59.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.546+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v114'163 uv163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.546+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfaa5c0 con 0x55f21be4bf70 2024-09-16T01:42:59.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.547+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v114'164 uv164 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21be4bf70 2024-09-16T01:42:59.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.547+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfaa5c0 con 0x55f21bf9fba0 2024-09-16T01:42:59.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.548+0000 7fbd48bfa640 1 -- 172.21.15.88:0/1723033203 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v114'263 uv257 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fbd3c0547a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.548+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfaa960 con 0x55f21bf9fba0 2024-09-16T01:42:59.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.549+0000 7fbd48bfa640 1 -- 172.21.15.88:0/1723033203 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v114'264 uv264 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbd3c0547a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.549+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfaad00 con 0x55f21bf9fba0 2024-09-16T01:42:59.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.550+0000 7fbd48bfa640 1 -- 172.21.15.88:0/1723033203 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v114'265 uv265 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbd3c0547a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.554 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:42:59.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.550+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfab0a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.551+0000 7fbd48bfa640 1 -- 172.21.15.88:0/1723033203 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v114'266 uv266 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbd3c0547a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.551+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfab0a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.552+0000 7fbd48bfa640 1 -- 172.21.15.88:0/1723033203 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v114'267 uv267 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbd3c0547a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.552+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfab0a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.553+0000 7fbd48bfa640 1 -- 172.21.15.88:0/1723033203 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v114'268 uv262 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbd3c0547a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.553+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfab0a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.554+0000 7fbd48bfa640 1 -- 172.21.15.88:0/1723033203 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v114'269 uv269 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd3c0547a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.554+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfab0a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.555+0000 7fbd48bfa640 1 -- 172.21.15.88:0/1723033203 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v114'270 uv270 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd3c0547a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.557 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:42:59.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.555+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfab0a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.556+0000 7fbd48bfa640 1 -- 172.21.15.88:0/1723033203 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v114'271 uv271 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd3c0547a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.556+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadeb0 con 0x55f21bf9fba0 2024-09-16T01:42:59.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.557+0000 7fbd48bfa640 1 -- 172.21.15.88:0/1723033203 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v114'272 uv272 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd3c0547a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.557+0000 7fbd53905f40 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55f21bfabbd0 0x55f21bfae490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.557+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bfabbd0 2024-09-16T01:42:59.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.557+0000 7fbd48bfa640 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55f21bfabbd0 0x55f21bfae490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:59.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.557+0000 7fbd48bfa640 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55f21bfabbd0 0x55f21bfae490 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:59.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.558+0000 7fbd48bfa640 1 -- 172.21.15.88:0/1723033203 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v114'116 uv114 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fbd3c0547a0 con 0x55f21bfabbd0 2024-09-16T01:42:59.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.558+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bfabbd0 2024-09-16T01:42:59.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.559+0000 7fbd48bfa640 1 -- 172.21.15.88:0/1723033203 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v114'117 uv117 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fbd3c0547a0 con 0x55f21bfabbd0 2024-09-16T01:42:59.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.559+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bfabbd0 2024-09-16T01:42:59.562 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:42:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.560+0000 7fbd48bfa640 1 -- 172.21.15.88:0/1723033203 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v114'118 uv118 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fbd3c0547a0 con 0x55f21bfabbd0 2024-09-16T01:42:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.560+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bfabbd0 2024-09-16T01:42:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.561+0000 7fbd48bfa640 1 -- 172.21.15.88:0/1723033203 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v114'119 uv119 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fbd3c0547a0 con 0x55f21bfabbd0 2024-09-16T01:42:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.561+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bf9fba0 2024-09-16T01:42:59.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.562+0000 7fbd48bfa640 1 -- 172.21.15.88:0/1723033203 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v114'159 uv158 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbd3c0547a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.563+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bf9fba0 2024-09-16T01:42:59.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.564+0000 7fbd48bfa640 1 -- 172.21.15.88:0/1723033203 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v114'160 uv160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd3c0547a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.564+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bf9fba0 2024-09-16T01:42:59.567 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:42:59.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.564+0000 7fbd48bfa640 1 -- 172.21.15.88:0/1723033203 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v114'161 uv161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd3c0547a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.565+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bf9fba0 2024-09-16T01:42:59.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.565+0000 7fbd48bfa640 1 -- 172.21.15.88:0/1723033203 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v114'162 uv162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd3c0547a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.565+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bf9fba0 2024-09-16T01:42:59.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.566+0000 7fbd48bfa640 1 -- 172.21.15.88:0/1723033203 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v114'163 uv163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd3c0547a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.566+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bf9fba0 2024-09-16T01:42:59.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.567+0000 7fbd48bfa640 1 -- 172.21.15.88:0/1723033203 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v114'164 uv164 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd3c0547a0 con 0x55f21bf9fba0 2024-09-16T01:42:59.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.567+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bfa4240 2024-09-16T01:42:59.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.569+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v114'135 uv134 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.569+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bfa4240 2024-09-16T01:42:59.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.570+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v114'136 uv136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.570+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bfa4240 2024-09-16T01:42:59.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.571+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v114'137 uv137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.573 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:42:59.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.571+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bfa4240 2024-09-16T01:42:59.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.572+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v114'138 uv138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.572+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bfa4240 2024-09-16T01:42:59.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.573+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v114'139 uv139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.573+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bfa4240 2024-09-16T01:42:59.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.574+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v114'327 uv318 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.574+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bfa4240 2024-09-16T01:42:59.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.575+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v114'328 uv328 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.575+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bfa4240 2024-09-16T01:42:59.579 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:42:59.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.577+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v114'329 uv329 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.577+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bfa4240 2024-09-16T01:42:59.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.578+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v114'330 uv330 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.578+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bfa4240 2024-09-16T01:42:59.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.579+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v114'331 uv331 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.579+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bfa4240 2024-09-16T01:42:59.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.580+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v114'332 uv325 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.580+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bfa4240 2024-09-16T01:42:59.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.581+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v114'333 uv333 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.581+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bfa4240 2024-09-16T01:42:59.584 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:42:59.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.582+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v114'334 uv334 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.582+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bfa4240 2024-09-16T01:42:59.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.583+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v114'335 uv335 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.583+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bfa4240 2024-09-16T01:42:59.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.584+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v114'336 uv336 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.585+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfadfc0 con 0x55f21bfa4240 2024-09-16T01:42:59.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.586+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v114'337 uv337 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.586+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x55f21bfae9f0 con 0x55f21bfa4240 2024-09-16T01:42:59.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v114'338 uv338 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fbd4c013cb0 con 0x55f21bfa4240 2024-09-16T01:42:59.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55f21be4bf70 msgr2=0x55f21bf8d070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:59.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd53905f40 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55f21be4bf70 0x55f21bf8d070 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55f21bf9fba0 msgr2=0x55f21bfa1f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:59.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd53905f40 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55f21bf9fba0 0x55f21bfa1f90 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55f21bfabbd0 msgr2=0x55f21bfae490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:59.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd53905f40 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55f21bfabbd0 0x55f21bfae490 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55f21bf9b630 msgr2=0x55f21bf9da20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:59.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd53905f40 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55f21bf9b630 0x55f21bf9da20 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f21bfa4240 msgr2=0x55f21bfa4620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:59.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd51bfc640 1 -- 172.21.15.88:0/1723033203 reap_dead start 2024-09-16T01:42:59.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd53905f40 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f21bfa4240 0x55f21bfa4620 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55f21bfa7650 msgr2=0x55f21bfa7a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:59.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd53905f40 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55f21bfa7650 0x55f21bfa7a90 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbd3406ac90 msgr2=0x7fbd3406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:59.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd53905f40 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbd3406ac90 0x7fbd3406d150 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c054000 tx=0x7fbd3c04e000 comp rx=0 tx=0).stop 2024-09-16T01:42:59.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21be4b020 msgr2=0x55f21bee9010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:59.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd53905f40 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21be4b020 0x55f21bee9010 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fbd44001ed0 tx=0x7fbd4400b6c0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 shutdown_connections 2024-09-16T01:42:59.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd53905f40 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbd3406ac90 0x7fbd3406d150 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd53905f40 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55f21bfa7650 0x55f21bfa7a90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd53905f40 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f21bfa4240 0x55f21bfa4620 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd53905f40 1 --2- 172.21.15.88:0/1723033203 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21be4b020 0x55f21bee9010 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 >> 172.21.15.88:0/1723033203 conn(0x55f21be4b680 msgr2=0x55f21beebfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:59.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 shutdown_connections 2024-09-16T01:42:59.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.587+0000 7fbd53905f40 1 -- 172.21.15.88:0/1723033203 wait complete. 2024-09-16T01:42:59.593 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata truncate mds_snaptable 0 2024-09-16T01:42:59.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.675+0000 7fe64b411f40 1 Processor -- start 2024-09-16T01:42:59.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.675+0000 7fe64b411f40 1 -- start start 2024-09-16T01:42:59.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.675+0000 7fe64b411f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6380022a0 0x5655014a8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.675+0000 7fe64b411f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565501429270 0x5655014acb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.675+0000 7fe64b411f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5655014ad0a0 0x5655014b6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.675+0000 7fe64b411f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x565501331510 con 0x7fe6380022a0 2024-09-16T01:42:59.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.675+0000 7fe64b411f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56550138ac50 con 0x565501429270 2024-09-16T01:42:59.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.675+0000 7fe64b411f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5655014add80 con 0x5655014ad0a0 2024-09-16T01:42:59.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.675+0000 7fe648f07640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6380022a0 0x5655014a8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:59.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.676+0000 7fe648f07640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6380022a0 0x5655014a8560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35202/0 (socket says 172.21.15.88:35202) 2024-09-16T01:42:59.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.676+0000 7fe648f07640 1 -- 172.21.15.88:0/3556555539 learned_addr learned my addr 172.21.15.88:0/3556555539 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:59.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.676+0000 7fe643fff640 1 --2- 172.21.15.88:0/3556555539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565501429270 0x5655014acb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:59.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.676+0000 7fe649708640 1 --2- 172.21.15.88:0/3556555539 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5655014ad0a0 0x5655014b6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:59.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.676+0000 7fe648f07640 1 -- 172.21.15.88:0/3556555539 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5655014ad0a0 msgr2=0x5655014b6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:59.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.676+0000 7fe648f07640 1 --2- 172.21.15.88:0/3556555539 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5655014ad0a0 0x5655014b6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.676+0000 7fe648f07640 1 -- 172.21.15.88:0/3556555539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565501429270 msgr2=0x5655014acb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:59.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.676+0000 7fe648f07640 1 --2- 172.21.15.88:0/3556555539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565501429270 0x5655014acb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.676+0000 7fe648f07640 1 -- 172.21.15.88:0/3556555539 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5655013ce8a0 con 0x7fe6380022a0 2024-09-16T01:42:59.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.676+0000 7fe648f07640 1 --2- 172.21.15.88:0/3556555539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6380022a0 0x5655014a8560 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7fe63c045e80 tx=0x7fe63c04e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98f09f0756f13717 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:59.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.676+0000 7fe642ffd640 1 -- 172.21.15.88:0/3556555539 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63c0543f0 con 0x7fe6380022a0 2024-09-16T01:42:59.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.676+0000 7fe642ffd640 1 -- 172.21.15.88:0/3556555539 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe63c0547a0 con 0x7fe6380022a0 2024-09-16T01:42:59.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.676+0000 7fe642ffd640 1 -- 172.21.15.88:0/3556555539 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63c054b90 con 0x7fe6380022a0 2024-09-16T01:42:59.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.676+0000 7fe64b411f40 1 -- 172.21.15.88:0/3556555539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6380022a0 msgr2=0x5655014a8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:59.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.676+0000 7fe64b411f40 1 --2- 172.21.15.88:0/3556555539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6380022a0 0x5655014a8560 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7fe63c045e80 tx=0x7fe63c04e000 comp rx=0 tx=0).stop 2024-09-16T01:42:59.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.677+0000 7fe64b411f40 1 -- 172.21.15.88:0/3556555539 shutdown_connections 2024-09-16T01:42:59.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.677+0000 7fe64b411f40 1 --2- 172.21.15.88:0/3556555539 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5655014ad0a0 0x5655014b6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.677+0000 7fe64b411f40 1 --2- 172.21.15.88:0/3556555539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565501429270 0x5655014acb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.677+0000 7fe64b411f40 1 --2- 172.21.15.88:0/3556555539 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6380022a0 0x5655014a8560 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.677+0000 7fe64b411f40 1 -- 172.21.15.88:0/3556555539 >> 172.21.15.88:0/3556555539 conn(0x56550142a750 msgr2=0x56550142ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:59.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.677+0000 7fe64b411f40 1 -- 172.21.15.88:0/3556555539 shutdown_connections 2024-09-16T01:42:59.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.677+0000 7fe64b411f40 1 -- 172.21.15.88:0/3556555539 wait complete. 2024-09-16T01:42:59.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.677+0000 7fe64b411f40 1 Processor -- start 2024-09-16T01:42:59.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.677+0000 7fe64b411f40 1 -- start start 2024-09-16T01:42:59.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.678+0000 7fe64b411f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565501429270 0x5655014a8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.678+0000 7fe64b411f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5655014ad0a0 0x5655014aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.678+0000 7fe64b411f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6380022a0 0x5655014af1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.678+0000 7fe64b411f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56550140bee0 con 0x5655014ad0a0 2024-09-16T01:42:59.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.678+0000 7fe64b411f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56550140bc00 con 0x565501429270 2024-09-16T01:42:59.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.678+0000 7fe64b411f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5655014adfd0 con 0x7fe6380022a0 2024-09-16T01:42:59.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.678+0000 7fe643fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5655014ad0a0 0x5655014aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:59.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.678+0000 7fe643fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5655014ad0a0 0x5655014aecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35206/0 (socket says 172.21.15.88:35206) 2024-09-16T01:42:59.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.678+0000 7fe643fff640 1 -- 172.21.15.88:0/1420319502 learned_addr learned my addr 172.21.15.88:0/1420319502 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:59.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.678+0000 7fe648f07640 1 --2- 172.21.15.88:0/1420319502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565501429270 0x5655014a8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:59.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.678+0000 7fe649708640 1 --2- 172.21.15.88:0/1420319502 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6380022a0 0x5655014af1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:59.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.678+0000 7fe643fff640 1 -- 172.21.15.88:0/1420319502 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6380022a0 msgr2=0x5655014af1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:59.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.678+0000 7fe643fff640 1 --2- 172.21.15.88:0/1420319502 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6380022a0 0x5655014af1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.678+0000 7fe643fff640 1 -- 172.21.15.88:0/1420319502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565501429270 msgr2=0x5655014a8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:59.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.678+0000 7fe643fff640 1 --2- 172.21.15.88:0/1420319502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x565501429270 0x5655014a8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.678+0000 7fe643fff640 1 -- 172.21.15.88:0/1420319502 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5655014aad50 con 0x5655014ad0a0 2024-09-16T01:42:59.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.679+0000 7fe643fff640 1 --2- 172.21.15.88:0/1420319502 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5655014ad0a0 0x5655014aecb0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fe634008700 tx=0x7fe6340087f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:59.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.679+0000 7fe641ffb640 1 -- 172.21.15.88:0/1420319502 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe634016830 con 0x5655014ad0a0 2024-09-16T01:42:59.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.679+0000 7fe641ffb640 1 -- 172.21.15.88:0/1420319502 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6340169d0 con 0x5655014ad0a0 2024-09-16T01:42:59.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.679+0000 7fe641ffb640 1 -- 172.21.15.88:0/1420319502 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe634016da0 con 0x5655014ad0a0 2024-09-16T01:42:59.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.679+0000 7fe64b411f40 1 -- 172.21.15.88:0/1420319502 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5655014ab030 con 0x5655014ad0a0 2024-09-16T01:42:59.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.679+0000 7fe64b411f40 1 -- 172.21.15.88:0/1420319502 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5655014a8aa0 con 0x5655014ad0a0 2024-09-16T01:42:59.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.680+0000 7fe641ffb640 1 -- 172.21.15.88:0/1420319502 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe63401d3a0 con 0x5655014ad0a0 2024-09-16T01:42:59.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.680+0000 7fe641ffb640 1 --2- 172.21.15.88:0/1420319502 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe62c06ac90 0x7fe62c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.681+0000 7fe641ffb640 1 -- 172.21.15.88:0/1420319502 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8060+0+0 (secure 0 0 0) 0x7fe63408b6b0 con 0x5655014ad0a0 2024-09-16T01:42:59.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.681+0000 7fe648f07640 1 --2- 172.21.15.88:0/1420319502 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe62c06ac90 0x7fe62c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:59.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.681+0000 7fe64b411f40 1 --2- 172.21.15.88:0/1420319502 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5655014267e0 0x56550154e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.681+0000 7fe64b411f40 1 -- 172.21.15.88:0/1420319502 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:1 12.d 12:b50e409b:::mds_snaptable:head [truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) -- 0x56550154e950 con 0x5655014267e0 2024-09-16T01:42:59.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.681+0000 7fe649708640 1 --2- 172.21.15.88:0/1420319502 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5655014267e0 0x56550154e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:59.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.681+0000 7fe648f07640 1 --2- 172.21.15.88:0/1420319502 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe62c06ac90 0x7fe62c06d150 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fe63c045e80 tx=0x7fe63c065040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:59.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.681+0000 7fe649708640 1 --2- 172.21.15.88:0/1420319502 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5655014267e0 0x56550154e410 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:59.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.682+0000 7fe649708640 1 -- 172.21.15.88:0/1420319502 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(1 mds_snaptable [truncate 0] v114'137 uv137 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe6440060c0 con 0x5655014267e0 2024-09-16T01:42:59.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.682+0000 7fe64b411f40 1 -- 172.21.15.88:0/1420319502 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5655014267e0 msgr2=0x56550154e410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:59.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.682+0000 7fe64b411f40 1 --2- 172.21.15.88:0/1420319502 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5655014267e0 0x56550154e410 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.682+0000 7fe64b411f40 1 -- 172.21.15.88:0/1420319502 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe62c06ac90 msgr2=0x7fe62c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:59.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.682+0000 7fe64b411f40 1 --2- 172.21.15.88:0/1420319502 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe62c06ac90 0x7fe62c06d150 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fe63c045e80 tx=0x7fe63c065040 comp rx=0 tx=0).stop 2024-09-16T01:42:59.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.682+0000 7fe64b411f40 1 -- 172.21.15.88:0/1420319502 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5655014ad0a0 msgr2=0x5655014aecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:59.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.682+0000 7fe64b411f40 1 --2- 172.21.15.88:0/1420319502 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5655014ad0a0 0x5655014aecb0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fe634008700 tx=0x7fe6340087f0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.682+0000 7fe649708640 1 -- 172.21.15.88:0/1420319502 reap_dead start 2024-09-16T01:42:59.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.683+0000 7fe64b411f40 1 -- 172.21.15.88:0/1420319502 shutdown_connections 2024-09-16T01:42:59.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.683+0000 7fe64b411f40 1 -- 172.21.15.88:0/1420319502 >> 172.21.15.88:0/1420319502 conn(0x56550142a750 msgr2=0x56550142ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:59.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.683+0000 7fe64b411f40 1 -- 172.21.15.88:0/1420319502 shutdown_connections 2024-09-16T01:42:59.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.683+0000 7fe64b411f40 1 -- 172.21.15.88:0/1420319502 wait complete. 2024-09-16T01:42:59.689 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:42:59.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.788+0000 7f636cdb9640 1 Processor -- start 2024-09-16T01:42:59.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.788+0000 7f636cdb9640 1 -- start start 2024-09-16T01:42:59.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.789+0000 7f636cdb9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6368147fe0 0x7f63680d1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.789+0000 7f636cdb9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63680b66a0 con 0x7f63680b36a0 2024-09-16T01:42:59.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.789+0000 7f636cdb9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63681483c0 con 0x7f63680b3ad0 2024-09-16T01:42:59.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.789+0000 7f636cdb9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f63680b45f0 con 0x7f6368147fe0 2024-09-16T01:42:59.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.789+0000 7f636659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6368147fe0 0x7f63680d1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:59.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.789+0000 7f636659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6368147fe0 0x7f63680d1ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35212/0 (socket says 172.21.15.88:35212) 2024-09-16T01:42:59.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.789+0000 7f636659c640 1 -- 172.21.15.88:0/916192730 learned_addr learned my addr 172.21.15.88:0/916192730 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:59.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.789+0000 7f635dd9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f63680b36a0 0x7f63680732a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42550/0 (socket says 172.21.15.88:42550) 2024-09-16T01:42:59.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.790+0000 7f636659c640 1 -- 172.21.15.88:0/916192730 >> v1:172.21.15.154:6789/0 conn(0x7f63680b3ad0 legacy=0x7f63680739b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:42:59.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.790+0000 7f636659c640 1 -- 172.21.15.88:0/916192730 >> v1:172.21.15.88:6789/0 conn(0x7f63680b36a0 legacy=0x7f63680732a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:59.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.790+0000 7f6365d9b640 1 -- 172.21.15.88:0/916192730 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3485312223 0 0) 0x7f63680b66a0 con 0x7f63680b36a0 2024-09-16T01:42:59.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.790+0000 7f636659c640 1 -- 172.21.15.88:0/916192730 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63680b48d0 con 0x7f6368147fe0 2024-09-16T01:42:59.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.790+0000 7f636659c640 1 --2- 172.21.15.88:0/916192730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6368147fe0 0x7f63680d1ab0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f63680d2440 tx=0x7f635802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0c799f95d804335 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:59.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.790+0000 7f6365d9b640 1 -- 172.21.15.88:0/916192730 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6358037070 con 0x7f6368147fe0 2024-09-16T01:42:59.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.790+0000 7f6365d9b640 1 -- 172.21.15.88:0/916192730 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6358032440 con 0x7f6368147fe0 2024-09-16T01:42:59.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.790+0000 7f6365d9b640 1 -- 172.21.15.88:0/916192730 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6358032810 con 0x7f6368147fe0 2024-09-16T01:42:59.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.791+0000 7f636cdb9640 1 -- 172.21.15.88:0/916192730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6368147fe0 msgr2=0x7f63680d1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:59.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.791+0000 7f636cdb9640 1 --2- 172.21.15.88:0/916192730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6368147fe0 0x7f63680d1ab0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f63680d2440 tx=0x7f635802a1e0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.791+0000 7f636cdb9640 1 -- 172.21.15.88:0/916192730 shutdown_connections 2024-09-16T01:42:59.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.791+0000 7f636cdb9640 1 --2- 172.21.15.88:0/916192730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6368147fe0 0x7f63680d1ab0 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.791+0000 7f636cdb9640 1 -- 172.21.15.88:0/916192730 >> 172.21.15.88:0/916192730 conn(0x7f63680b6b50 msgr2=0x7f63680b6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:42:59.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.791+0000 7f636cdb9640 1 -- 172.21.15.88:0/916192730 shutdown_connections 2024-09-16T01:42:59.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.791+0000 7f636cdb9640 1 -- 172.21.15.88:0/916192730 wait complete. 2024-09-16T01:42:59.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.793+0000 7f636cdb9640 1 Processor -- start 2024-09-16T01:42:59.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.793+0000 7f636cdb9640 1 -- start start 2024-09-16T01:42:59.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.793+0000 7f636cdb9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63680b36a0 0x7f6368158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.793+0000 7f636cdb9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63680b3ad0 0x7f63680b0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.793+0000 7f636cdb9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6368147fe0 0x7f63680b0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.793+0000 7f636cdb9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f63680b5170 con 0x7f63680b36a0 2024-09-16T01:42:59.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.793+0000 7f636cdb9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f63680b4e90 con 0x7f6368147fe0 2024-09-16T01:42:59.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.793+0000 7f636cdb9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f63680b3f20 con 0x7f63680b3ad0 2024-09-16T01:42:59.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.793+0000 7f636659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63680b36a0 0x7f6368158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:59.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.793+0000 7f636659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63680b36a0 0x7f6368158140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35218/0 (socket says 172.21.15.88:35218) 2024-09-16T01:42:59.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.793+0000 7f636659c640 1 -- 172.21.15.88:0/3508298907 learned_addr learned my addr 172.21.15.88:0/3508298907 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:42:59.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.793+0000 7f6366d9d640 1 --2- 172.21.15.88:0/3508298907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6368147fe0 0x7f63680b0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:59.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.794+0000 7f635dd9b640 1 --2- 172.21.15.88:0/3508298907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63680b3ad0 0x7f63680b0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:59.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.794+0000 7f636659c640 1 -- 172.21.15.88:0/3508298907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63680b3ad0 msgr2=0x7f63680b0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:59.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.794+0000 7f636659c640 1 --2- 172.21.15.88:0/3508298907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63680b3ad0 0x7f63680b0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.794+0000 7f636659c640 1 -- 172.21.15.88:0/3508298907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6368147fe0 msgr2=0x7f63680b0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:42:59.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.794+0000 7f636659c640 1 --2- 172.21.15.88:0/3508298907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6368147fe0 0x7f63680b0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:42:59.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.794+0000 7f636659c640 1 -- 172.21.15.88:0/3508298907 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6368161010 con 0x7f63680b36a0 2024-09-16T01:42:59.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.794+0000 7f6366d9d640 1 --2- 172.21.15.88:0/3508298907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6368147fe0 0x7f63680b0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:42:59.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.794+0000 7f636659c640 1 --2- 172.21.15.88:0/3508298907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63680b36a0 0x7f6368158140 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f6358000c00 tx=0x7f63580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:59.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.794+0000 7f635f7fe640 1 -- 172.21.15.88:0/3508298907 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6358041070 con 0x7f63680b36a0 2024-09-16T01:42:59.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.794+0000 7f635f7fe640 1 -- 172.21.15.88:0/3508298907 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f635803c870 con 0x7f63680b36a0 2024-09-16T01:42:59.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.794+0000 7f636cdb9640 1 -- 172.21.15.88:0/3508298907 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63681612f0 con 0x7f63680b36a0 2024-09-16T01:42:59.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.794+0000 7f635f7fe640 1 -- 172.21.15.88:0/3508298907 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6358037040 con 0x7f63680b36a0 2024-09-16T01:42:59.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.794+0000 7f636cdb9640 1 -- 172.21.15.88:0/3508298907 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63681618b0 con 0x7f63680b36a0 2024-09-16T01:42:59.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.795+0000 7f636cdb9640 1 -- 172.21.15.88:0/3508298907 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63680ae890 con 0x7f63680b36a0 2024-09-16T01:42:59.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.795+0000 7f635f7fe640 1 -- 172.21.15.88:0/3508298907 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6358044070 con 0x7f63680b36a0 2024-09-16T01:42:59.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.796+0000 7f635f7fe640 1 --2- 172.21.15.88:0/3508298907 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f634806c2b0 0x7f634806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:42:59.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.796+0000 7f635f7fe640 1 -- 172.21.15.88:0/3508298907 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 8060+0+0 (secure 0 0 0) 0x7f63580ac990 con 0x7f63680b36a0 2024-09-16T01:42:59.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.796+0000 7f635dd9b640 1 --2- 172.21.15.88:0/3508298907 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f634806c2b0 0x7f634806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:42:59.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.796+0000 7f635dd9b640 1 --2- 172.21.15.88:0/3508298907 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f634806c2b0 0x7f634806e770 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f6368144d10 tx=0x7f6350007b70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:42:59.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.800+0000 7f635f7fe640 1 -- 172.21.15.88:0/3508298907 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f635807a8f0 con 0x7f63680b36a0 2024-09-16T01:43:00.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:42:59.999+0000 7f636cdb9640 1 -- 172.21.15.88:0/3508298907 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f636806dd20 con 0x7f63680b36a0 2024-09-16T01:43:00.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.338+0000 7f635f7fe640 1 -- 172.21.15.88:0/3508298907 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v233) ==== 162+0+0 (secure 0 0 0) 0x7f635807a1f0 con 0x7f63680b36a0 2024-09-16T01:43:00.341 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:43:00.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.341+0000 7f636cdb9640 1 -- 172.21.15.88:0/3508298907 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f634806c2b0 msgr2=0x7f634806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:00.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.341+0000 7f636cdb9640 1 --2- 172.21.15.88:0/3508298907 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f634806c2b0 0x7f634806e770 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f6368144d10 tx=0x7f6350007b70 comp rx=0 tx=0).stop 2024-09-16T01:43:00.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.341+0000 7f636cdb9640 1 -- 172.21.15.88:0/3508298907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63680b36a0 msgr2=0x7f6368158140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:00.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.341+0000 7f636cdb9640 1 --2- 172.21.15.88:0/3508298907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63680b36a0 0x7f6368158140 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f6358000c00 tx=0x7f63580046c0 comp rx=0 tx=0).stop 2024-09-16T01:43:00.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.341+0000 7f636cdb9640 1 -- 172.21.15.88:0/3508298907 shutdown_connections 2024-09-16T01:43:00.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.341+0000 7f636cdb9640 1 --2- 172.21.15.88:0/3508298907 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f634806c2b0 0x7f634806e770 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:00.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.341+0000 7f636cdb9640 1 --2- 172.21.15.88:0/3508298907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6368147fe0 0x7f63680b0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:00.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.341+0000 7f636cdb9640 1 --2- 172.21.15.88:0/3508298907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f63680b3ad0 0x7f63680b0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:00.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.341+0000 7f636cdb9640 1 --2- 172.21.15.88:0/3508298907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63680b36a0 0x7f6368158140 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:00.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.341+0000 7f636cdb9640 1 -- 172.21.15.88:0/3508298907 >> 172.21.15.88:0/3508298907 conn(0x7f63680b6b50 msgr2=0x7f63680b16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:00.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.341+0000 7f636cdb9640 1 -- 172.21.15.88:0/3508298907 shutdown_connections 2024-09-16T01:43:00.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.341+0000 7f636cdb9640 1 -- 172.21.15.88:0/3508298907 wait complete. 2024-09-16T01:43:00.350 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:43:00.348+0000 7f3832cb5640 -1 log_channel(cluster) log [ERR] : error decoding table object 'mds_snaptable': End of buffer [buffer:2] 2024-09-16T01:43:00.351 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:43:00.349+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:43:00.351 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:43:00.349+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:43:00.356 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:43:00.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.452+0000 7fbd2425e640 1 Processor -- start 2024-09-16T01:43:00.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.452+0000 7fbd2425e640 1 -- start start 2024-09-16T01:43:00.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.453+0000 7fbd2425e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbd1c06ef30 0x7fbd1c06bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:00.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.453+0000 7fbd2425e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd1c0ceef0 con 0x7fbd1c06c150 2024-09-16T01:43:00.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.453+0000 7fbd2425e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd1c0cf1d0 con 0x7fbd1c15c610 2024-09-16T01:43:00.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.453+0000 7fbd2425e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fbd1c06e7b0 con 0x7fbd1c06ef30 2024-09-16T01:43:00.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.453+0000 7fbd217f9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbd1c06ef30 0x7fbd1c06bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:00.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.453+0000 7fbd217f9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbd1c06ef30 0x7fbd1c06bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55550/0 (socket says 172.21.15.88:55550) 2024-09-16T01:43:00.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.453+0000 7fbd217f9640 1 -- 172.21.15.88:0/3444824269 learned_addr learned my addr 172.21.15.88:0/3444824269 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:00.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.453+0000 7fbd227fb640 1 --1- 172.21.15.88:0/3444824269 >> v1:172.21.15.88:6789/0 conn(0x7fbd1c06c150 0x7fbd1c1660c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57028/0 (socket says 172.21.15.88:57028) 2024-09-16T01:43:00.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.453+0000 7fbd217f9640 1 -- 172.21.15.88:0/3444824269 >> v1:172.21.15.154:6789/0 conn(0x7fbd1c15c610 legacy=0x7fbd1c0ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:00.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.453+0000 7fbd217f9640 1 -- 172.21.15.88:0/3444824269 >> v1:172.21.15.88:6789/0 conn(0x7fbd1c06c150 legacy=0x7fbd1c1660c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:00.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.453+0000 7fbd217f9640 1 -- 172.21.15.88:0/3444824269 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd1c0cf4b0 con 0x7fbd1c06ef30 2024-09-16T01:43:00.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.454+0000 7fbd217f9640 1 --2- 172.21.15.88:0/3444824269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd1c06ef30 0x7fbd1c06bc10 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c009870 tx=0x7fbd0c02a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8d42342fb614bbd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:00.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.454+0000 7fbd20ff8640 1 -- 172.21.15.88:0/3444824269 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c037070 con 0x7fbd1c06ef30 2024-09-16T01:43:00.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.454+0000 7fbd20ff8640 1 -- 172.21.15.88:0/3444824269 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd0c032440 con 0x7fbd1c06ef30 2024-09-16T01:43:00.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.454+0000 7fbd20ff8640 1 -- 172.21.15.88:0/3444824269 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c032810 con 0x7fbd1c06ef30 2024-09-16T01:43:00.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.455+0000 7fbd2425e640 1 -- 172.21.15.88:0/3444824269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd1c06ef30 msgr2=0x7fbd1c06bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:00.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.455+0000 7fbd2425e640 1 --2- 172.21.15.88:0/3444824269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd1c06ef30 0x7fbd1c06bc10 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c009870 tx=0x7fbd0c02a2a0 comp rx=0 tx=0).stop 2024-09-16T01:43:00.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.455+0000 7fbd2425e640 1 -- 172.21.15.88:0/3444824269 shutdown_connections 2024-09-16T01:43:00.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.455+0000 7fbd2425e640 1 --2- 172.21.15.88:0/3444824269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd1c06ef30 0x7fbd1c06bc10 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:00.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.455+0000 7fbd2425e640 1 -- 172.21.15.88:0/3444824269 >> 172.21.15.88:0/3444824269 conn(0x7fbd1c0b2430 msgr2=0x7fbd1c0b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:00.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.455+0000 7fbd2425e640 1 -- 172.21.15.88:0/3444824269 shutdown_connections 2024-09-16T01:43:00.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.455+0000 7fbd2425e640 1 -- 172.21.15.88:0/3444824269 wait complete. 2024-09-16T01:43:00.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.456+0000 7fbd2425e640 1 Processor -- start 2024-09-16T01:43:00.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.456+0000 7fbd2425e640 1 -- start start 2024-09-16T01:43:00.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.457+0000 7fbd2425e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd1c06c150 0x7fbd1c1e66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:00.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.457+0000 7fbd2425e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd1c06ef30 0x7fbd1c15e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:00.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.457+0000 7fbd2425e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd1c15c610 0x7fbd1c14e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:00.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.457+0000 7fbd2425e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1c06c580 con 0x7fbd1c06c150 2024-09-16T01:43:00.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.457+0000 7fbd2425e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1c0cecb0 con 0x7fbd1c06ef30 2024-09-16T01:43:00.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.457+0000 7fbd2425e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbd1c0713c0 con 0x7fbd1c15c610 2024-09-16T01:43:00.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.457+0000 7fbd21ffa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd1c06c150 0x7fbd1c1e66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:00.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.457+0000 7fbd21ffa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd1c06c150 0x7fbd1c1e66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55564/0 (socket says 172.21.15.88:55564) 2024-09-16T01:43:00.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.457+0000 7fbd21ffa640 1 -- 172.21.15.88:0/1418613796 learned_addr learned my addr 172.21.15.88:0/1418613796 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.457+0000 7fbd21ffa640 1 -- 172.21.15.88:0/1418613796 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd1c15c610 msgr2=0x7fbd1c14e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.457+0000 7fbd21ffa640 1 --2- 172.21.15.88:0/1418613796 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd1c15c610 0x7fbd1c14e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.457+0000 7fbd21ffa640 1 -- 172.21.15.88:0/1418613796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd1c06ef30 msgr2=0x7fbd1c15e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.457+0000 7fbd21ffa640 1 --2- 172.21.15.88:0/1418613796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd1c06ef30 0x7fbd1c15e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.457+0000 7fbd21ffa640 1 -- 172.21.15.88:0/1418613796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd1c1e4a80 con 0x7fbd1c06c150 2024-09-16T01:43:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.457+0000 7fbd21ffa640 1 --2- 172.21.15.88:0/1418613796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd1c06c150 0x7fbd1c1e66b0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7fbd1800b420 tx=0x7fbd1800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.457+0000 7fbd127fc640 1 -- 172.21.15.88:0/1418613796 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd18011830 con 0x7fbd1c06c150 2024-09-16T01:43:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.457+0000 7fbd127fc640 1 -- 172.21.15.88:0/1418613796 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd18016070 con 0x7fbd1c06c150 2024-09-16T01:43:00.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.457+0000 7fbd127fc640 1 -- 172.21.15.88:0/1418613796 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd18011c10 con 0x7fbd1c06c150 2024-09-16T01:43:00.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.457+0000 7fbd2425e640 1 -- 172.21.15.88:0/1418613796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd1c1e5b00 con 0x7fbd1c06c150 2024-09-16T01:43:00.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.457+0000 7fbd2425e640 1 -- 172.21.15.88:0/1418613796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd1c1e5020 con 0x7fbd1c06c150 2024-09-16T01:43:00.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.458+0000 7fbd2425e640 1 -- 172.21.15.88:0/1418613796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd1c06bc10 con 0x7fbd1c06c150 2024-09-16T01:43:00.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.459+0000 7fbd127fc640 1 -- 172.21.15.88:0/1418613796 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fbd1801b460 con 0x7fbd1c06c150 2024-09-16T01:43:00.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.459+0000 7fbd127fc640 1 --2- 172.21.15.88:0/1418613796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbd0006c2b0 0x7fbd0006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:00.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.459+0000 7fbd217f9640 1 --2- 172.21.15.88:0/1418613796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbd0006c2b0 0x7fbd0006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:00.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.459+0000 7fbd127fc640 1 -- 172.21.15.88:0/1418613796 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8146+0+0 (secure 0 0 0) 0x7fbd1808b8e0 con 0x7fbd1c06c150 2024-09-16T01:43:00.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.460+0000 7fbd217f9640 1 --2- 172.21.15.88:0/1418613796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbd0006c2b0 0x7fbd0006e770 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c034000 tx=0x7fbd0c02c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:00.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.464+0000 7fbd127fc640 1 -- 172.21.15.88:0/1418613796 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fbd18020020 con 0x7fbd1c06c150 2024-09-16T01:43:00.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.664+0000 7fbd2425e640 1 -- 172.21.15.88:0/1418613796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd1c0b14f0 con 0x7fbd1c06c150 2024-09-16T01:43:00.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.665+0000 7fbd127fc640 1 -- 172.21.15.88:0/1418613796 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) ==== 95+0+4392 (secure 0 0 0) 0x7fbd18059630 con 0x7fbd1c06c150 2024-09-16T01:43:00.667 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:00.668 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":235,"btime":"2024-09-16T01:43:00:351126+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9333,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2477051134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2477051134},{"type":"v1","addr":"172.21.15.88:6837","nonce":2477051134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":235,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:43:00.351059+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:43:00.668 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 235 2024-09-16T01:43:00.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.667+0000 7fbd2425e640 1 -- 172.21.15.88:0/1418613796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbd0006c2b0 msgr2=0x7fbd0006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:00.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.667+0000 7fbd2425e640 1 --2- 172.21.15.88:0/1418613796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbd0006c2b0 0x7fbd0006e770 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c034000 tx=0x7fbd0c02c000 comp rx=0 tx=0).stop 2024-09-16T01:43:00.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.667+0000 7fbd2425e640 1 -- 172.21.15.88:0/1418613796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd1c06c150 msgr2=0x7fbd1c1e66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:00.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.667+0000 7fbd2425e640 1 --2- 172.21.15.88:0/1418613796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd1c06c150 0x7fbd1c1e66b0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7fbd1800b420 tx=0x7fbd1800b8e0 comp rx=0 tx=0).stop 2024-09-16T01:43:00.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.667+0000 7fbd2425e640 1 -- 172.21.15.88:0/1418613796 shutdown_connections 2024-09-16T01:43:00.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.667+0000 7fbd2425e640 1 --2- 172.21.15.88:0/1418613796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbd0006c2b0 0x7fbd0006e770 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:00.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.667+0000 7fbd2425e640 1 --2- 172.21.15.88:0/1418613796 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd1c15c610 0x7fbd1c14e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:00.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.667+0000 7fbd2425e640 1 --2- 172.21.15.88:0/1418613796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd1c06ef30 0x7fbd1c15e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:00.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.667+0000 7fbd2425e640 1 --2- 172.21.15.88:0/1418613796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd1c06c150 0x7fbd1c1e66b0 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:00.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.667+0000 7fbd2425e640 1 -- 172.21.15.88:0/1418613796 >> 172.21.15.88:0/1418613796 conn(0x7fbd1c0b2430 msgr2=0x7fbd1c0724b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:00.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.667+0000 7fbd2425e640 1 -- 172.21.15.88:0/1418613796 shutdown_connections 2024-09-16T01:43:00.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.667+0000 7fbd2425e640 1 -- 172.21.15.88:0/1418613796 wait complete. 2024-09-16T01:43:00.681 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:43:00.681 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:43:00.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.777+0000 7ff792178640 1 Processor -- start 2024-09-16T01:43:00.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.777+0000 7ff792178640 1 -- start start 2024-09-16T01:43:00.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.778+0000 7ff792178640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff78c0cfc80 0x7ff78c0b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:00.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.778+0000 7ff792178640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff78c15c330 0x7ff78c0b4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:00.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.778+0000 7ff792178640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff78c15c780 con 0x7ff78c155800 2024-09-16T01:43:00.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.778+0000 7ff792178640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff78c0b4e60 con 0x7ff78c15c330 2024-09-16T01:43:00.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.778+0000 7ff792178640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff78c0b50a0 con 0x7ff78c0cfc80 2024-09-16T01:43:00.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.778+0000 7ff78bfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff78c15c330 0x7ff78c0b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:00.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.778+0000 7ff78bfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff78c15c330 0x7ff78c0b4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55574/0 (socket says 172.21.15.88:55574) 2024-09-16T01:43:00.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.778+0000 7ff78bfff640 1 -- 172.21.15.88:0/1041162989 learned_addr learned my addr 172.21.15.88:0/1041162989 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:00.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.778+0000 7ff78affd640 1 --2- 172.21.15.88:0/1041162989 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff78c0cfc80 0x7ff78c0b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:00.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.778+0000 7ff78bfff640 1 -- 172.21.15.88:0/1041162989 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff78c0cfc80 msgr2=0x7ff78c0b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:00.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.778+0000 7ff78bfff640 1 --2- 172.21.15.88:0/1041162989 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff78c0cfc80 0x7ff78c0b6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:00.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.778+0000 7ff78bfff640 1 -- 172.21.15.88:0/1041162989 >> v1:172.21.15.154:6789/0 conn(0x7ff78c155800 legacy=0x7ff78c0cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:00.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.778+0000 7ff78bfff640 1 -- 172.21.15.88:0/1041162989 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff78c0b5380 con 0x7ff78c15c330 2024-09-16T01:43:00.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.779+0000 7ff78bfff640 1 --2- 172.21.15.88:0/1041162989 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff78c15c330 0x7ff78c0b4920 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7ff784009870 tx=0x7ff78402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8164d99202bcb7c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:00.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.779+0000 7ff78a7fc640 1 -- 172.21.15.88:0/1041162989 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff784037070 con 0x7ff78c15c330 2024-09-16T01:43:00.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.779+0000 7ff78a7fc640 1 -- 172.21.15.88:0/1041162989 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff784032440 con 0x7ff78c15c330 2024-09-16T01:43:00.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.779+0000 7ff78a7fc640 1 -- 172.21.15.88:0/1041162989 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff784032810 con 0x7ff78c15c330 2024-09-16T01:43:00.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.780+0000 7ff792178640 1 -- 172.21.15.88:0/1041162989 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff78c15c330 msgr2=0x7ff78c0b4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:00.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.780+0000 7ff792178640 1 --2- 172.21.15.88:0/1041162989 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff78c15c330 0x7ff78c0b4920 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7ff784009870 tx=0x7ff78402a270 comp rx=0 tx=0).stop 2024-09-16T01:43:00.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.780+0000 7ff792178640 1 -- 172.21.15.88:0/1041162989 shutdown_connections 2024-09-16T01:43:00.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.780+0000 7ff792178640 1 --2- 172.21.15.88:0/1041162989 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff78c15c330 0x7ff78c0b4920 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:00.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.780+0000 7ff792178640 1 --2- 172.21.15.88:0/1041162989 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff78c0cfc80 0x7ff78c0b6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:00.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.780+0000 7ff792178640 1 -- 172.21.15.88:0/1041162989 >> 172.21.15.88:0/1041162989 conn(0x7ff78c06bc10 msgr2=0x7ff78c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:00.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.780+0000 7ff792178640 1 -- 172.21.15.88:0/1041162989 shutdown_connections 2024-09-16T01:43:00.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.780+0000 7ff792178640 1 -- 172.21.15.88:0/1041162989 wait complete. 2024-09-16T01:43:00.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.781+0000 7ff792178640 1 Processor -- start 2024-09-16T01:43:00.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.781+0000 7ff792178640 1 -- start start 2024-09-16T01:43:00.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.782+0000 7ff792178640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff78c0cfc80 0x7ff78c14f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:00.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.782+0000 7ff792178640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff78c155800 0x7ff78c155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:00.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.782+0000 7ff792178640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff78c15c330 0x7ff78c0720f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:00.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.782+0000 7ff792178640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c0b5c20 con 0x7ff78c15c330 2024-09-16T01:43:00.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.782+0000 7ff792178640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c0b5940 con 0x7ff78c0cfc80 2024-09-16T01:43:00.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.782+0000 7ff792178640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff78c155be0 con 0x7ff78c155800 2024-09-16T01:43:00.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.782+0000 7ff78bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff78c15c330 0x7ff78c0720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:00.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.782+0000 7ff78bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff78c15c330 0x7ff78c0720f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55590/0 (socket says 172.21.15.88:55590) 2024-09-16T01:43:00.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.782+0000 7ff78bfff640 1 -- 172.21.15.88:0/1012864069 learned_addr learned my addr 172.21.15.88:0/1012864069 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:00.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.782+0000 7ff78b7fe640 1 --2- 172.21.15.88:0/1012864069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff78c0cfc80 0x7ff78c14f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:00.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.782+0000 7ff78affd640 1 --2- 172.21.15.88:0/1012864069 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff78c155800 0x7ff78c155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:00.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.782+0000 7ff78bfff640 1 -- 172.21.15.88:0/1012864069 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff78c155800 msgr2=0x7ff78c155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:00.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.782+0000 7ff78bfff640 1 --2- 172.21.15.88:0/1012864069 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff78c155800 0x7ff78c155e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:00.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.782+0000 7ff78bfff640 1 -- 172.21.15.88:0/1012864069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff78c0cfc80 msgr2=0x7ff78c14f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:00.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.782+0000 7ff78bfff640 1 --2- 172.21.15.88:0/1012864069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff78c0cfc80 0x7ff78c14f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:00.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.782+0000 7ff78bfff640 1 -- 172.21.15.88:0/1012864069 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff78c156740 con 0x7ff78c15c330 2024-09-16T01:43:00.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.782+0000 7ff78b7fe640 1 --2- 172.21.15.88:0/1012864069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff78c0cfc80 0x7ff78c14f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:00.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.782+0000 7ff78affd640 1 --2- 172.21.15.88:0/1012864069 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff78c155800 0x7ff78c155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:00.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.782+0000 7ff78bfff640 1 --2- 172.21.15.88:0/1012864069 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff78c15c330 0x7ff78c0720f0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7ff78402a2b0 tx=0x7ff7840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:00.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.783+0000 7ff773fff640 1 -- 172.21.15.88:0/1012864069 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff784041020 con 0x7ff78c15c330 2024-09-16T01:43:00.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.783+0000 7ff773fff640 1 -- 172.21.15.88:0/1012864069 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff78403d640 con 0x7ff78c15c330 2024-09-16T01:43:00.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.783+0000 7ff773fff640 1 -- 172.21.15.88:0/1012864069 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff784037040 con 0x7ff78c15c330 2024-09-16T01:43:00.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.783+0000 7ff792178640 1 -- 172.21.15.88:0/1012864069 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff78c156a20 con 0x7ff78c15c330 2024-09-16T01:43:00.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.783+0000 7ff792178640 1 -- 172.21.15.88:0/1012864069 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff78c156fe0 con 0x7ff78c15c330 2024-09-16T01:43:00.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.784+0000 7ff792178640 1 -- 172.21.15.88:0/1012864069 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff78c06c5d0 con 0x7ff78c15c330 2024-09-16T01:43:00.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.784+0000 7ff773fff640 1 -- 172.21.15.88:0/1012864069 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff784044020 con 0x7ff78c15c330 2024-09-16T01:43:00.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.784+0000 7ff773fff640 1 --2- 172.21.15.88:0/1012864069 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff77806c2e0 0x7ff77806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:00.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.785+0000 7ff78b7fe640 1 --2- 172.21.15.88:0/1012864069 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff77806c2e0 0x7ff77806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:00.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.785+0000 7ff773fff640 1 -- 172.21.15.88:0/1012864069 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8146+0+0 (secure 0 0 0) 0x7ff7840ac460 con 0x7ff78c15c330 2024-09-16T01:43:00.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.785+0000 7ff78b7fe640 1 --2- 172.21.15.88:0/1012864069 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff77806c2e0 0x7ff77806e7a0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7ff78c1f5930 tx=0x7ff77c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:00.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.789+0000 7ff773fff640 1 -- 172.21.15.88:0/1012864069 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff78404b030 con 0x7ff78c15c330 2024-09-16T01:43:00.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.989+0000 7ff792178640 1 -- 172.21.15.88:0/1012864069 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff78c151a80 con 0x7ff78c15c330 2024-09-16T01:43:00.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.990+0000 7ff773fff640 1 -- 172.21.15.88:0/1012864069 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) ==== 95+0+4392 (secure 0 0 0) 0x7ff78407a250 con 0x7ff78c15c330 2024-09-16T01:43:00.992 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:00.992 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":235,"btime":"2024-09-16T01:43:00:351126+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9333,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2477051134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2477051134},{"type":"v1","addr":"172.21.15.88:6837","nonce":2477051134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":235,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:43:00.351059+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:43:00.993 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 235 2024-09-16T01:43:00.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.992+0000 7ff792178640 1 -- 172.21.15.88:0/1012864069 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff77806c2e0 msgr2=0x7ff77806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.992+0000 7ff792178640 1 --2- 172.21.15.88:0/1012864069 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff77806c2e0 0x7ff77806e7a0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7ff78c1f5930 tx=0x7ff77c009040 comp rx=0 tx=0).stop 2024-09-16T01:43:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.992+0000 7ff792178640 1 -- 172.21.15.88:0/1012864069 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff78c15c330 msgr2=0x7ff78c0720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.992+0000 7ff792178640 1 --2- 172.21.15.88:0/1012864069 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff78c15c330 0x7ff78c0720f0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7ff78402a2b0 tx=0x7ff7840046c0 comp rx=0 tx=0).stop 2024-09-16T01:43:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.992+0000 7ff792178640 1 -- 172.21.15.88:0/1012864069 shutdown_connections 2024-09-16T01:43:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.992+0000 7ff792178640 1 --2- 172.21.15.88:0/1012864069 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff77806c2e0 0x7ff77806e7a0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.992+0000 7ff792178640 1 --2- 172.21.15.88:0/1012864069 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff78c15c330 0x7ff78c0720f0 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.992+0000 7ff792178640 1 --2- 172.21.15.88:0/1012864069 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff78c155800 0x7ff78c155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.992+0000 7ff792178640 1 --2- 172.21.15.88:0/1012864069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff78c0cfc80 0x7ff78c14f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.992+0000 7ff792178640 1 -- 172.21.15.88:0/1012864069 >> 172.21.15.88:0/1012864069 conn(0x7ff78c06bc10 msgr2=0x7ff78c0b3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.992+0000 7ff792178640 1 -- 172.21.15.88:0/1012864069 shutdown_connections 2024-09-16T01:43:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:00.992+0000 7ff792178640 1 -- 172.21.15.88:0/1012864069 wait complete. 2024-09-16T01:43:01.005 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:43:01.005 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:43:01.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.101+0000 7f49f770a640 1 Processor -- start 2024-09-16T01:43:01.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.101+0000 7f49f770a640 1 -- start start 2024-09-16T01:43:01.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.102+0000 7f49f770a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f49f015e950 0x7f49f006ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:01.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.102+0000 7f49f770a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f49f0072050 0x7f49f006d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:01.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.102+0000 7f49f770a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49f006faf0 con 0x7f49f00735b0 2024-09-16T01:43:01.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.102+0000 7f49f770a640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f49f006fdd0 con 0x7f49f015e950 2024-09-16T01:43:01.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.102+0000 7f49f770a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f49f00700b0 con 0x7f49f0072050 2024-09-16T01:43:01.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.102+0000 7f49f54a6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f49f015e950 0x7f49f006ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:01.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.102+0000 7f49f54a6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f49f015e950 0x7f49f006ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55606/0 (socket says 172.21.15.88:55606) 2024-09-16T01:43:01.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.102+0000 7f49f4ca5640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f49f00735b0 0x7f49f006bc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57030/0 (socket says 172.21.15.88:57030) 2024-09-16T01:43:01.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.102+0000 7f49f54a6640 1 -- 172.21.15.88:0/400290826 learned_addr learned my addr 172.21.15.88:0/400290826 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:01.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.102+0000 7f49f5ca7640 1 --2- 172.21.15.88:0/400290826 >> v2:172.21.15.154:3300/0 conn(0x7f49f0072050 0x7f49f006d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:01.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.102+0000 7f49f54a6640 1 -- 172.21.15.88:0/400290826 >> v2:172.21.15.154:3300/0 conn(0x7f49f0072050 msgr2=0x7f49f006d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:01.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.102+0000 7f49f54a6640 1 --2- 172.21.15.88:0/400290826 >> v2:172.21.15.154:3300/0 conn(0x7f49f0072050 0x7f49f006d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:01.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.102+0000 7f49f54a6640 1 -- 172.21.15.88:0/400290826 >> v1:172.21.15.88:6789/0 conn(0x7f49f00735b0 legacy=0x7f49f006bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:01.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.102+0000 7f49f54a6640 1 -- 172.21.15.88:0/400290826 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49f0070390 con 0x7f49f015e950 2024-09-16T01:43:01.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.102+0000 7f49f5ca7640 1 --2- 172.21.15.88:0/400290826 >> v2:172.21.15.154:3300/0 conn(0x7f49f0072050 0x7f49f006d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:01.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.103+0000 7f49f54a6640 1 --2- 172.21.15.88:0/400290826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49f015e950 0x7f49f006ce30 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f49f00724e0 tx=0x7f49ec02a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dbb5d5fb8f2e6f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:01.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.103+0000 7f49e7fff640 1 -- 172.21.15.88:0/400290826 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ec036070 con 0x7f49f015e950 2024-09-16T01:43:01.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.103+0000 7f49e7fff640 1 -- 172.21.15.88:0/400290826 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49ec031600 con 0x7f49f015e950 2024-09-16T01:43:01.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.103+0000 7f49e7fff640 1 -- 172.21.15.88:0/400290826 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ec0319d0 con 0x7f49f015e950 2024-09-16T01:43:01.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.103+0000 7f49f770a640 1 -- 172.21.15.88:0/400290826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49f015e950 msgr2=0x7f49f006ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:01.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.103+0000 7f49f770a640 1 --2- 172.21.15.88:0/400290826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49f015e950 0x7f49f006ce30 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f49f00724e0 tx=0x7f49ec02a730 comp rx=0 tx=0).stop 2024-09-16T01:43:01.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.104+0000 7f49f770a640 1 -- 172.21.15.88:0/400290826 shutdown_connections 2024-09-16T01:43:01.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.104+0000 7f49f770a640 1 --2- 172.21.15.88:0/400290826 >> v2:172.21.15.154:3300/0 conn(0x7f49f0072050 0x7f49f006d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:01.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.104+0000 7f49f770a640 1 --2- 172.21.15.88:0/400290826 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49f015e950 0x7f49f006ce30 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:01.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.104+0000 7f49f770a640 1 -- 172.21.15.88:0/400290826 >> 172.21.15.88:0/400290826 conn(0x7f49f00d14d0 msgr2=0x7f49f00d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:01.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.104+0000 7f49f770a640 1 -- 172.21.15.88:0/400290826 shutdown_connections 2024-09-16T01:43:01.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.104+0000 7f49f770a640 1 -- 172.21.15.88:0/400290826 wait complete. 2024-09-16T01:43:01.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.105+0000 7f49f770a640 1 Processor -- start 2024-09-16T01:43:01.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.105+0000 7f49f770a640 1 -- start start 2024-09-16T01:43:01.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.106+0000 7f49f770a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49f0072050 0x7f49f00cd610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:01.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.106+0000 7f49f770a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49f00735b0 0x7f49f00cb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:01.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.106+0000 7f49f770a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49f015e950 0x7f49f01447f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:01.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.106+0000 7f49f770a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f49f00d0230 con 0x7f49f015e950 2024-09-16T01:43:01.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.106+0000 7f49f770a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f49f00d00b0 con 0x7f49f00735b0 2024-09-16T01:43:01.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.106+0000 7f49f770a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f49f00d03b0 con 0x7f49f0072050 2024-09-16T01:43:01.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.106+0000 7f49f5ca7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49f015e950 0x7f49f01447f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:01.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.106+0000 7f49f5ca7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49f015e950 0x7f49f01447f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55620/0 (socket says 172.21.15.88:55620) 2024-09-16T01:43:01.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.106+0000 7f49f5ca7640 1 -- 172.21.15.88:0/2137589689 learned_addr learned my addr 172.21.15.88:0/2137589689 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:01.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.106+0000 7f49f4ca5640 1 --2- 172.21.15.88:0/2137589689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49f00735b0 0x7f49f00cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:01.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.106+0000 7f49f54a6640 1 --2- 172.21.15.88:0/2137589689 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49f0072050 0x7f49f00cd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:01.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.106+0000 7f49f5ca7640 1 -- 172.21.15.88:0/2137589689 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49f0072050 msgr2=0x7f49f00cd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:01.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.106+0000 7f49f5ca7640 1 --2- 172.21.15.88:0/2137589689 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49f0072050 0x7f49f00cd610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:01.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.106+0000 7f49f5ca7640 1 -- 172.21.15.88:0/2137589689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49f00735b0 msgr2=0x7f49f00cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:01.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.106+0000 7f49f5ca7640 1 --2- 172.21.15.88:0/2137589689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49f00735b0 0x7f49f00cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:01.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.106+0000 7f49f5ca7640 1 -- 172.21.15.88:0/2137589689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49f00cbad0 con 0x7f49f015e950 2024-09-16T01:43:01.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.106+0000 7f49f4ca5640 1 --2- 172.21.15.88:0/2137589689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49f00735b0 0x7f49f00cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:01.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.106+0000 7f49f54a6640 1 --2- 172.21.15.88:0/2137589689 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49f0072050 0x7f49f00cd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:01.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.106+0000 7f49f5ca7640 1 --2- 172.21.15.88:0/2137589689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49f015e950 0x7f49f01447f0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f49e800c7c0 tx=0x7f49e800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:01.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.106+0000 7f49e5ffb640 1 -- 172.21.15.88:0/2137589689 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e8015020 con 0x7f49f015e950 2024-09-16T01:43:01.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.106+0000 7f49e5ffb640 1 -- 172.21.15.88:0/2137589689 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49e8004230 con 0x7f49f015e950 2024-09-16T01:43:01.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.106+0000 7f49f770a640 1 -- 172.21.15.88:0/2137589689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49f00cbdb0 con 0x7f49f015e950 2024-09-16T01:43:01.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.106+0000 7f49e5ffb640 1 -- 172.21.15.88:0/2137589689 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e8007620 con 0x7f49f015e950 2024-09-16T01:43:01.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.107+0000 7f49f770a640 1 -- 172.21.15.88:0/2137589689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49f00cb7a0 con 0x7f49f015e950 2024-09-16T01:43:01.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.108+0000 7f49f770a640 1 -- 172.21.15.88:0/2137589689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49f006ce30 con 0x7f49f015e950 2024-09-16T01:43:01.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.108+0000 7f49e5ffb640 1 -- 172.21.15.88:0/2137589689 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f49e8018020 con 0x7f49f015e950 2024-09-16T01:43:01.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.108+0000 7f49e5ffb640 1 --2- 172.21.15.88:0/2137589689 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f49d406c2b0 0x7f49d406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:01.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.109+0000 7f49e5ffb640 1 -- 172.21.15.88:0/2137589689 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8146+0+0 (secure 0 0 0) 0x7f49e808be10 con 0x7f49f015e950 2024-09-16T01:43:01.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.109+0000 7f49f54a6640 1 --2- 172.21.15.88:0/2137589689 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f49d406c2b0 0x7f49d406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:01.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.109+0000 7f49f54a6640 1 --2- 172.21.15.88:0/2137589689 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f49d406c2b0 0x7f49d406e770 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f49ec033000 tx=0x7f49ec009000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:01.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.113+0000 7f49e5ffb640 1 -- 172.21.15.88:0/2137589689 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f49e801f020 con 0x7f49f015e950 2024-09-16T01:43:01.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.312+0000 7f49f770a640 1 -- 172.21.15.88:0/2137589689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49f00b3900 con 0x7f49f015e950 2024-09-16T01:43:01.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.313+0000 7f49e5ffb640 1 -- 172.21.15.88:0/2137589689 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) ==== 95+0+4392 (secure 0 0 0) 0x7f49e8059d40 con 0x7f49f015e950 2024-09-16T01:43:01.315 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:01.316 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":235,"btime":"2024-09-16T01:43:00:351126+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9333,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/2477051134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":2477051134},{"type":"v1","addr":"172.21.15.88:6837","nonce":2477051134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":228}],"filesystems":[{"mdsmap":{"epoch":235,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:43:00.351059+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:43:01.316 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 235 2024-09-16T01:43:01.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.315+0000 7f49f770a640 1 -- 172.21.15.88:0/2137589689 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f49d406c2b0 msgr2=0x7f49d406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:01.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.315+0000 7f49f770a640 1 --2- 172.21.15.88:0/2137589689 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f49d406c2b0 0x7f49d406e770 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f49ec033000 tx=0x7f49ec009000 comp rx=0 tx=0).stop 2024-09-16T01:43:01.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.315+0000 7f49f770a640 1 -- 172.21.15.88:0/2137589689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49f015e950 msgr2=0x7f49f01447f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:01.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.315+0000 7f49f770a640 1 --2- 172.21.15.88:0/2137589689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49f015e950 0x7f49f01447f0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f49e800c7c0 tx=0x7f49e800cc80 comp rx=0 tx=0).stop 2024-09-16T01:43:01.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.315+0000 7f49f770a640 1 -- 172.21.15.88:0/2137589689 shutdown_connections 2024-09-16T01:43:01.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.315+0000 7f49f770a640 1 --2- 172.21.15.88:0/2137589689 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f49d406c2b0 0x7f49d406e770 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:01.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.315+0000 7f49f770a640 1 --2- 172.21.15.88:0/2137589689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49f015e950 0x7f49f01447f0 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:01.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.315+0000 7f49f770a640 1 --2- 172.21.15.88:0/2137589689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49f00735b0 0x7f49f00cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:01.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.315+0000 7f49f770a640 1 --2- 172.21.15.88:0/2137589689 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f49f0072050 0x7f49f00cd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:01.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.315+0000 7f49f770a640 1 -- 172.21.15.88:0/2137589689 >> 172.21.15.88:0/2137589689 conn(0x7f49f00d14d0 msgr2=0x7f49f006bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:01.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.315+0000 7f49f770a640 1 -- 172.21.15.88:0/2137589689 shutdown_connections 2024-09-16T01:43:01.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.315+0000 7f49f770a640 1 -- 172.21.15.88:0/2137589689 wait complete. 2024-09-16T01:43:01.328 INFO:tasks.cephfs.test_damage:Result: Mutation 'Truncate mds_snaptable' led to DAMAGED state 2024-09-16T01:43:01.328 INFO:tasks.cephfs.test_damage:Applying mutation 'Truncate mds0_inotable' 2024-09-16T01:43:01.328 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:43:01.329 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:43:01.344 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:43:01.345 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:43:01.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:01.345 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:43:01.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:43:01.401 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:43:01.401 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:43:01.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.538+0000 7f4a3629b640 1 Processor -- start 2024-09-16T01:43:01.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.538+0000 7f4a3629b640 1 -- start start 2024-09-16T01:43:01.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.539+0000 7f4a3629b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4a30073600 0x7f4a3015e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:01.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.539+0000 7f4a3629b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a3015a430 con 0x7f4a300731c0 2024-09-16T01:43:01.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.539+0000 7f4a3629b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a300cd0a0 con 0x7f4a3015a050 2024-09-16T01:43:01.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.539+0000 7f4a3629b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f4a300cd380 con 0x7f4a30073600 2024-09-16T01:43:01.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.539+0000 7f4a2f7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4a300731c0 0x7f4a3015de80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57044/0 (socket says 172.21.15.88:57044) 2024-09-16T01:43:01.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.539+0000 7f4a2f7fe640 1 -- 172.21.15.88:0/1208323873 learned_addr learned my addr 172.21.15.88:0/1208323873 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:01.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.539+0000 7f4a34838640 1 --2- 172.21.15.88:0/1208323873 >> v2:172.21.15.154:3300/0 conn(0x7f4a30073600 0x7f4a3015e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:01.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.539+0000 7f4a2effd640 1 -- 172.21.15.88:0/1208323873 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 292521330 0 0) 0x7f4a3015a430 con 0x7f4a300731c0 2024-09-16T01:43:01.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.539+0000 7f4a2effd640 1 -- 172.21.15.88:0/1208323873 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a18003520 con 0x7f4a300731c0 2024-09-16T01:43:01.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.540+0000 7f4a2effd640 1 -- 172.21.15.88:0/1208323873 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1097403423 0 0) 0x7f4a18003520 con 0x7f4a300731c0 2024-09-16T01:43:01.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.540+0000 7f4a2effd640 1 -- 172.21.15.88:0/1208323873 >> v2:172.21.15.154:3300/0 conn(0x7f4a30073600 msgr2=0x7f4a3015e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:01.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.540+0000 7f4a2effd640 1 --2- 172.21.15.88:0/1208323873 >> v2:172.21.15.154:3300/0 conn(0x7f4a30073600 0x7f4a3015e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:01.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.540+0000 7f4a2effd640 1 -- 172.21.15.88:0/1208323873 >> v1:172.21.15.154:6789/0 conn(0x7f4a3015a050 legacy=0x7f4a3015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:01.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.540+0000 7f4a2effd640 1 -- 172.21.15.88:0/1208323873 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a300cd660 con 0x7f4a300731c0 2024-09-16T01:43:01.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.540+0000 7f4a2effd640 1 -- 172.21.15.88:0/1208323873 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f4a24001670 con 0x7f4a300731c0 2024-09-16T01:43:01.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.540+0000 7f4a34838640 1 --2- 172.21.15.88:0/1208323873 >> v2:172.21.15.154:3300/0 conn(0x7f4a30073600 0x7f4a3015e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:43:01.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.540+0000 7f4a2effd640 1 -- 172.21.15.88:0/1208323873 >> v1:172.21.15.88:6789/0 conn(0x7f4a300731c0 legacy=0x7f4a3015de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:01.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.540+0000 7f4a2effd640 1 --2- 172.21.15.88:0/1208323873 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a180039b0 0x7f4a18003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:01.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.540+0000 7f4a2effd640 1 --2- 172.21.15.88:0/1208323873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a18004800 0x7f4a18004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:01.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.540+0000 7f4a2effd640 1 --2- 172.21.15.88:0/1208323873 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a180051c0 0x7f4a1800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:01.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.540+0000 7f4a2effd640 1 -- 172.21.15.88:0/1208323873 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4a300cd380 con 0x7f4a180039b0 2024-09-16T01:43:01.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.540+0000 7f4a2effd640 1 -- 172.21.15.88:0/1208323873 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1800c160 con 0x7f4a18004800 2024-09-16T01:43:01.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.540+0000 7f4a2effd640 1 -- 172.21.15.88:0/1208323873 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4a1800c360 con 0x7f4a180051c0 2024-09-16T01:43:01.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.540+0000 7f4a2ffff640 1 --2- 172.21.15.88:0/1208323873 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a180039b0 0x7f4a18003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:01.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.540+0000 7f4a2f7fe640 1 --2- 172.21.15.88:0/1208323873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a18004800 0x7f4a18004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:01.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.540+0000 7f4a34838640 1 --2- 172.21.15.88:0/1208323873 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a180051c0 0x7f4a1800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:01.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.540+0000 7f4a2ffff640 1 -- 172.21.15.88:0/1208323873 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a180051c0 msgr2=0x7f4a1800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:01.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.540+0000 7f4a2ffff640 1 --2- 172.21.15.88:0/1208323873 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a180051c0 0x7f4a1800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:01.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.540+0000 7f4a2ffff640 1 -- 172.21.15.88:0/1208323873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a18004800 msgr2=0x7f4a18004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:01.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.540+0000 7f4a2ffff640 1 --2- 172.21.15.88:0/1208323873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a18004800 0x7f4a18004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:01.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.540+0000 7f4a34838640 1 -- 172.21.15.88:0/1208323873 reap_dead start 2024-09-16T01:43:01.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.540+0000 7f4a2ffff640 1 -- 172.21.15.88:0/1208323873 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a18003520 con 0x7f4a180039b0 2024-09-16T01:43:01.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.540+0000 7f4a2f7fe640 1 --2- 172.21.15.88:0/1208323873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a18004800 0x7f4a18004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:01.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.541+0000 7f4a2ffff640 1 --2- 172.21.15.88:0/1208323873 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a180039b0 0x7f4a18003dd0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f4a20001750 tx=0x7f4a18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dd5366b1a7cd8f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:01.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.541+0000 7f4a2effd640 1 -- 172.21.15.88:0/1208323873 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a20001370 con 0x7f4a180039b0 2024-09-16T01:43:01.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.541+0000 7f4a2effd640 1 -- 172.21.15.88:0/1208323873 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a200052d0 con 0x7f4a180039b0 2024-09-16T01:43:01.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.541+0000 7f4a2effd640 1 -- 172.21.15.88:0/1208323873 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a2002abe0 con 0x7f4a180039b0 2024-09-16T01:43:01.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.541+0000 7f4a3629b640 1 -- 172.21.15.88:0/1208323873 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a180039b0 msgr2=0x7f4a18003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:01.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.541+0000 7f4a3629b640 1 --2- 172.21.15.88:0/1208323873 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a180039b0 0x7f4a18003dd0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f4a20001750 tx=0x7f4a18005830 comp rx=0 tx=0).stop 2024-09-16T01:43:01.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.541+0000 7f4a3629b640 1 -- 172.21.15.88:0/1208323873 shutdown_connections 2024-09-16T01:43:01.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.541+0000 7f4a3629b640 1 --2- 172.21.15.88:0/1208323873 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a180039b0 0x7f4a18003dd0 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:01.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.541+0000 7f4a3629b640 1 -- 172.21.15.88:0/1208323873 >> 172.21.15.88:0/1208323873 conn(0x7f4a3006d0f0 msgr2=0x7f4a3006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:01.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.541+0000 7f4a3629b640 1 -- 172.21.15.88:0/1208323873 shutdown_connections 2024-09-16T01:43:01.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.541+0000 7f4a3629b640 1 -- 172.21.15.88:0/1208323873 wait complete. 2024-09-16T01:43:01.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.543+0000 7f4a3629b640 1 Processor -- start 2024-09-16T01:43:01.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.543+0000 7f4a3629b640 1 -- start start 2024-09-16T01:43:01.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.543+0000 7f4a3629b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a180039b0 0x7f4a30156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:01.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.543+0000 7f4a3629b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a3014f450 0x7f4a301f1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:01.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.543+0000 7f4a3629b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a300cf370 0x7f4a300cf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:01.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.543+0000 7f4a3629b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4a30073a90 con 0x7f4a3014f450 2024-09-16T01:43:01.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.543+0000 7f4a3629b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4a300cdc20 con 0x7f4a300cf370 2024-09-16T01:43:01.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.543+0000 7f4a3629b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4a300b36e0 con 0x7f4a180039b0 2024-09-16T01:43:01.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.544+0000 7f4a2f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a3014f450 0x7f4a301f1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:01.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.544+0000 7f4a2f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a3014f450 0x7f4a301f1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55632/0 (socket says 172.21.15.88:55632) 2024-09-16T01:43:01.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.544+0000 7f4a2f7fe640 1 -- 172.21.15.88:0/1804924526 learned_addr learned my addr 172.21.15.88:0/1804924526 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:01.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.544+0000 7f4a34838640 1 --2- 172.21.15.88:0/1804924526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a300cf370 0x7f4a300cf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:01.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.544+0000 7f4a2ffff640 1 --2- 172.21.15.88:0/1804924526 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a180039b0 0x7f4a30156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:01.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.544+0000 7f4a2f7fe640 1 -- 172.21.15.88:0/1804924526 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a180039b0 msgr2=0x7f4a30156980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:01.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.544+0000 7f4a2f7fe640 1 --2- 172.21.15.88:0/1804924526 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a180039b0 0x7f4a30156980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:01.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.544+0000 7f4a2f7fe640 1 -- 172.21.15.88:0/1804924526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a300cf370 msgr2=0x7f4a300cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:01.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.544+0000 7f4a2f7fe640 1 --2- 172.21.15.88:0/1804924526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a300cf370 0x7f4a300cf7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:01.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.544+0000 7f4a2f7fe640 1 -- 172.21.15.88:0/1804924526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a30151f90 con 0x7f4a3014f450 2024-09-16T01:43:01.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.544+0000 7f4a2ffff640 1 --2- 172.21.15.88:0/1804924526 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a180039b0 0x7f4a30156980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:01.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.544+0000 7f4a2f7fe640 1 --2- 172.21.15.88:0/1804924526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a3014f450 0x7f4a301f1e10 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f4a2400e780 tx=0x7f4a2400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:01.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.544+0000 7f4a2cff9640 1 -- 172.21.15.88:0/1804924526 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a2400ee30 con 0x7f4a3014f450 2024-09-16T01:43:01.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.544+0000 7f4a2cff9640 1 -- 172.21.15.88:0/1804924526 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a24018070 con 0x7f4a3014f450 2024-09-16T01:43:01.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.544+0000 7f4a3629b640 1 -- 172.21.15.88:0/1804924526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a301f2720 con 0x7f4a3014f450 2024-09-16T01:43:01.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.544+0000 7f4a2cff9640 1 -- 172.21.15.88:0/1804924526 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a2400ca70 con 0x7f4a3014f450 2024-09-16T01:43:01.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.544+0000 7f4a3629b640 1 -- 172.21.15.88:0/1804924526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a301f2ce0 con 0x7f4a3014f450 2024-09-16T01:43:01.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.545+0000 7f4a3629b640 1 -- 172.21.15.88:0/1804924526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a300cfd30 con 0x7f4a3014f450 2024-09-16T01:43:01.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.546+0000 7f4a2cff9640 1 -- 172.21.15.88:0/1804924526 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4a2400cc10 con 0x7f4a3014f450 2024-09-16T01:43:01.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.546+0000 7f4a2cff9640 1 --2- 172.21.15.88:0/1804924526 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4a1c06c2e0 0x7f4a1c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:01.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.546+0000 7f4a2ffff640 1 --2- 172.21.15.88:0/1804924526 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4a1c06c2e0 0x7f4a1c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:01.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.546+0000 7f4a2cff9640 1 -- 172.21.15.88:0/1804924526 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8146+0+0 (secure 0 0 0) 0x7f4a2408b7b0 con 0x7f4a3014f450 2024-09-16T01:43:01.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.547+0000 7f4a2ffff640 1 --2- 172.21.15.88:0/1804924526 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4a1c06c2e0 0x7f4a1c06e7a0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f4a20032000 tx=0x7f4a2002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:01.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.550+0000 7f4a2cff9640 1 -- 172.21.15.88:0/1804924526 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4a24059590 con 0x7f4a3014f450 2024-09-16T01:43:01.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:01.748+0000 7f4a3629b640 1 -- 172.21.15.88:0/1804924526 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f4a300ccd00 con 0x7f4a3014f450 2024-09-16T01:43:02.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.355+0000 7f4a2cff9640 1 -- 172.21.15.88:0/1804924526 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v236) ==== 161+0+0 (secure 0 0 0) 0x7f4a24058e90 con 0x7f4a3014f450 2024-09-16T01:43:02.357 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:43:02.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.357+0000 7f4a3629b640 1 -- 172.21.15.88:0/1804924526 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4a1c06c2e0 msgr2=0x7f4a1c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:02.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.357+0000 7f4a3629b640 1 --2- 172.21.15.88:0/1804924526 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4a1c06c2e0 0x7f4a1c06e7a0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f4a20032000 tx=0x7f4a2002e000 comp rx=0 tx=0).stop 2024-09-16T01:43:02.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.357+0000 7f4a3629b640 1 -- 172.21.15.88:0/1804924526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a3014f450 msgr2=0x7f4a301f1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:02.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.357+0000 7f4a3629b640 1 --2- 172.21.15.88:0/1804924526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a3014f450 0x7f4a301f1e10 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f4a2400e780 tx=0x7f4a2400ec40 comp rx=0 tx=0).stop 2024-09-16T01:43:02.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.357+0000 7f4a3629b640 1 -- 172.21.15.88:0/1804924526 shutdown_connections 2024-09-16T01:43:02.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.357+0000 7f4a3629b640 1 --2- 172.21.15.88:0/1804924526 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4a1c06c2e0 0x7f4a1c06e7a0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:02.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.357+0000 7f4a3629b640 1 --2- 172.21.15.88:0/1804924526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a300cf370 0x7f4a300cf7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:02.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.357+0000 7f4a3629b640 1 --2- 172.21.15.88:0/1804924526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4a3014f450 0x7f4a301f1e10 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:02.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.357+0000 7f4a3629b640 1 --2- 172.21.15.88:0/1804924526 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4a180039b0 0x7f4a30156980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:02.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.357+0000 7f4a3629b640 1 -- 172.21.15.88:0/1804924526 >> 172.21.15.88:0/1804924526 conn(0x7f4a3006d0f0 msgr2=0x7f4a3015f310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:02.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.357+0000 7f4a3629b640 1 -- 172.21.15.88:0/1804924526 shutdown_connections 2024-09-16T01:43:02.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.357+0000 7f4a3629b640 1 -- 172.21.15.88:0/1804924526 wait complete. 2024-09-16T01:43:02.373 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:43:02.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.472+0000 7f10dbb66640 1 Processor -- start 2024-09-16T01:43:02.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.472+0000 7f10dbb66640 1 -- start start 2024-09-16T01:43:02.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.473+0000 7f10dbb66640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f10d415ad10 0x7f10d40719c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:02.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.473+0000 7f10dbb66640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f10d40cca40 0x7f10d4071f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:02.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.473+0000 7f10dbb66640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10d4071720 con 0x7f10d40cc360 2024-09-16T01:43:02.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.473+0000 7f10dbb66640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f10d4070fb0 con 0x7f10d415ad10 2024-09-16T01:43:02.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.473+0000 7f10dbb66640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f10d4073920 con 0x7f10d40cca40 2024-09-16T01:43:02.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.473+0000 7f10d9902640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f10d415ad10 0x7f10d40719c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:02.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.473+0000 7f10d9902640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f10d415ad10 0x7f10d40719c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55634/0 (socket says 172.21.15.88:55634) 2024-09-16T01:43:02.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.473+0000 7f10d9902640 1 -- 172.21.15.88:0/136508447 learned_addr learned my addr 172.21.15.88:0/136508447 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:02.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.473+0000 7f10d9902640 1 -- 172.21.15.88:0/136508447 >> v2:172.21.15.154:3300/0 conn(0x7f10d40cca40 msgr2=0x7f10d4071f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:02.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.473+0000 7f10d9902640 1 --2- 172.21.15.88:0/136508447 >> v2:172.21.15.154:3300/0 conn(0x7f10d40cca40 0x7f10d4071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:02.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.473+0000 7f10d9902640 1 -- 172.21.15.88:0/136508447 >> v1:172.21.15.154:6789/0 conn(0x7f10d40cc360 legacy=0x7f10d415e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:02.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.473+0000 7f10d9902640 1 -- 172.21.15.88:0/136508447 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10d4073c00 con 0x7f10d415ad10 2024-09-16T01:43:02.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.474+0000 7f10d9902640 1 --2- 172.21.15.88:0/136508447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10d415ad10 0x7f10d40719c0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f10d4070b50 tx=0x7f10d002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39137e37f939e208 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:02.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.474+0000 7f10d8900640 1 -- 172.21.15.88:0/136508447 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d0037070 con 0x7f10d415ad10 2024-09-16T01:43:02.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.474+0000 7f10d8900640 1 -- 172.21.15.88:0/136508447 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10d0032440 con 0x7f10d415ad10 2024-09-16T01:43:02.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.474+0000 7f10d8900640 1 -- 172.21.15.88:0/136508447 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d0032810 con 0x7f10d415ad10 2024-09-16T01:43:02.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.475+0000 7f10dbb66640 1 -- 172.21.15.88:0/136508447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10d415ad10 msgr2=0x7f10d40719c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:02.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.475+0000 7f10dbb66640 1 --2- 172.21.15.88:0/136508447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10d415ad10 0x7f10d40719c0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f10d4070b50 tx=0x7f10d002a1a0 comp rx=0 tx=0).stop 2024-09-16T01:43:02.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.475+0000 7f10dbb66640 1 -- 172.21.15.88:0/136508447 shutdown_connections 2024-09-16T01:43:02.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.475+0000 7f10dbb66640 1 --2- 172.21.15.88:0/136508447 >> v2:172.21.15.154:3300/0 conn(0x7f10d40cca40 0x7f10d4071f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:02.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.475+0000 7f10dbb66640 1 --2- 172.21.15.88:0/136508447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10d415ad10 0x7f10d40719c0 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:02.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.475+0000 7f10dbb66640 1 -- 172.21.15.88:0/136508447 >> 172.21.15.88:0/136508447 conn(0x7f10d406d0f0 msgr2=0x7f10d406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:02.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.475+0000 7f10dbb66640 1 -- 172.21.15.88:0/136508447 shutdown_connections 2024-09-16T01:43:02.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.475+0000 7f10dbb66640 1 -- 172.21.15.88:0/136508447 wait complete. 2024-09-16T01:43:02.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.476+0000 7f10dbb66640 1 Processor -- start 2024-09-16T01:43:02.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.476+0000 7f10dbb66640 1 -- start start 2024-09-16T01:43:02.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.477+0000 7f10dbb66640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10d40cc360 0x7f10d41f54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:02.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.477+0000 7f10dbb66640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10d40cca40 0x7f10d41f2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:02.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.477+0000 7f10dbb66640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10d415ad10 0x7f10d40cae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:02.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.477+0000 7f10dbb66640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f10d40b4eb0 con 0x7f10d415ad10 2024-09-16T01:43:02.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.477+0000 7f10dbb66640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f10d4071200 con 0x7f10d40cc360 2024-09-16T01:43:02.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.477+0000 7f10dbb66640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f10d40cd770 con 0x7f10d40cca40 2024-09-16T01:43:02.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.477+0000 7f10da103640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10d415ad10 0x7f10d40cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:02.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.477+0000 7f10da103640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10d415ad10 0x7f10d40cae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55646/0 (socket says 172.21.15.88:55646) 2024-09-16T01:43:02.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.477+0000 7f10da103640 1 -- 172.21.15.88:0/1168546112 learned_addr learned my addr 172.21.15.88:0/1168546112 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:02.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.477+0000 7f10d9902640 1 --2- 172.21.15.88:0/1168546112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10d40cc360 0x7f10d41f54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:02.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.477+0000 7f10d9101640 1 --2- 172.21.15.88:0/1168546112 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10d40cca40 0x7f10d41f2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:02.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.477+0000 7f10da103640 1 -- 172.21.15.88:0/1168546112 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10d40cca40 msgr2=0x7f10d41f2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:02.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.477+0000 7f10da103640 1 --2- 172.21.15.88:0/1168546112 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10d40cca40 0x7f10d41f2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:02.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.477+0000 7f10da103640 1 -- 172.21.15.88:0/1168546112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10d40cc360 msgr2=0x7f10d41f54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:02.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.477+0000 7f10da103640 1 --2- 172.21.15.88:0/1168546112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10d40cc360 0x7f10d41f54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:02.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.477+0000 7f10da103640 1 -- 172.21.15.88:0/1168546112 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10d41f3570 con 0x7f10d415ad10 2024-09-16T01:43:02.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.477+0000 7f10d9101640 1 --2- 172.21.15.88:0/1168546112 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10d40cca40 0x7f10d41f2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:02.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.477+0000 7f10da103640 1 --2- 172.21.15.88:0/1168546112 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10d415ad10 0x7f10d40cae30 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f10cc00b9e0 tx=0x7f10cc00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:02.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.477+0000 7f10ca7fc640 1 -- 172.21.15.88:0/1168546112 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10cc019070 con 0x7f10d415ad10 2024-09-16T01:43:02.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.477+0000 7f10ca7fc640 1 -- 172.21.15.88:0/1168546112 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10cc00cb20 con 0x7f10d415ad10 2024-09-16T01:43:02.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.477+0000 7f10dbb66640 1 -- 172.21.15.88:0/1168546112 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10d41f2cd0 con 0x7f10d415ad10 2024-09-16T01:43:02.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.477+0000 7f10ca7fc640 1 -- 172.21.15.88:0/1168546112 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10cc014440 con 0x7f10d415ad10 2024-09-16T01:43:02.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.477+0000 7f10dbb66640 1 -- 172.21.15.88:0/1168546112 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10d41f3290 con 0x7f10d415ad10 2024-09-16T01:43:02.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.478+0000 7f10dbb66640 1 -- 172.21.15.88:0/1168546112 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10d415dcc0 con 0x7f10d415ad10 2024-09-16T01:43:02.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.479+0000 7f10ca7fc640 1 -- 172.21.15.88:0/1168546112 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f10cc01c070 con 0x7f10d415ad10 2024-09-16T01:43:02.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.479+0000 7f10ca7fc640 1 --2- 172.21.15.88:0/1168546112 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f10b806c2b0 0x7f10b806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:02.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.479+0000 7f10d9902640 1 --2- 172.21.15.88:0/1168546112 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f10b806c2b0 0x7f10b806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:02.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.479+0000 7f10ca7fc640 1 -- 172.21.15.88:0/1168546112 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8146+0+0 (secure 0 0 0) 0x7f10cc08b990 con 0x7f10d415ad10 2024-09-16T01:43:02.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.480+0000 7f10d9902640 1 --2- 172.21.15.88:0/1168546112 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f10b806c2b0 0x7f10b806e770 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f10d0034000 tx=0x7f10d002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:02.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.484+0000 7f10ca7fc640 1 -- 172.21.15.88:0/1168546112 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f10cc0591b0 con 0x7f10d415ad10 2024-09-16T01:43:02.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:02.717+0000 7f10dbb66640 1 -- 172.21.15.88:0/1168546112 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f10d406d770 con 0x7f10d415ad10 2024-09-16T01:43:03.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.366+0000 7f10ca7fc640 1 -- 172.21.15.88:0/1168546112 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v237) ==== 101+0+0 (secure 0 0 0) 0x7f10cc05d4b0 con 0x7f10d415ad10 2024-09-16T01:43:03.368 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:43:03.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.369+0000 7f10dbb66640 1 -- 172.21.15.88:0/1168546112 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f10b806c2b0 msgr2=0x7f10b806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:03.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.369+0000 7f10dbb66640 1 --2- 172.21.15.88:0/1168546112 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f10b806c2b0 0x7f10b806e770 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f10d0034000 tx=0x7f10d002e000 comp rx=0 tx=0).stop 2024-09-16T01:43:03.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.369+0000 7f10dbb66640 1 -- 172.21.15.88:0/1168546112 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10d415ad10 msgr2=0x7f10d40cae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:03.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.369+0000 7f10dbb66640 1 --2- 172.21.15.88:0/1168546112 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10d415ad10 0x7f10d40cae30 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f10cc00b9e0 tx=0x7f10cc00bea0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.369+0000 7f10dbb66640 1 -- 172.21.15.88:0/1168546112 shutdown_connections 2024-09-16T01:43:03.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.369+0000 7f10dbb66640 1 --2- 172.21.15.88:0/1168546112 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f10b806c2b0 0x7f10b806e770 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.369+0000 7f10dbb66640 1 --2- 172.21.15.88:0/1168546112 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f10d415ad10 0x7f10d40cae30 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.369+0000 7f10dbb66640 1 --2- 172.21.15.88:0/1168546112 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10d40cca40 0x7f10d41f2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.369+0000 7f10dbb66640 1 --2- 172.21.15.88:0/1168546112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10d40cc360 0x7f10d41f54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.369+0000 7f10dbb66640 1 -- 172.21.15.88:0/1168546112 >> 172.21.15.88:0/1168546112 conn(0x7f10d406d0f0 msgr2=0x7f10d415bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:03.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.369+0000 7f10dbb66640 1 -- 172.21.15.88:0/1168546112 shutdown_connections 2024-09-16T01:43:03.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.369+0000 7f10dbb66640 1 -- 172.21.15.88:0/1168546112 wait complete. 2024-09-16T01:43:03.385 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:43:03.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.428+0000 7f9fbb078f40 1 Processor -- start 2024-09-16T01:43:03.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.428+0000 7f9fbb078f40 1 -- start start 2024-09-16T01:43:03.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.429+0000 7f9fbb078f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x562de77fffe0 0x562de789e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.429+0000 7f9fbb078f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562de78a4d20 0x562de77fbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.429+0000 7f9fbb078f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562de78a5100 0x562de77fc120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.429+0000 7f9fbb078f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x562de7722510 con 0x562de78a5100 2024-09-16T01:43:03.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.429+0000 7f9fbb078f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562de7818bd0 con 0x562de78a4d20 2024-09-16T01:43:03.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.429+0000 7f9fbb078f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x562de789e260 con 0x562de77fffe0 2024-09-16T01:43:03.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.429+0000 7f9fb936f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562de78a5100 0x562de77fc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:03.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.429+0000 7f9fb936f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562de78a5100 0x562de77fc120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55660/0 (socket says 172.21.15.88:55660) 2024-09-16T01:43:03.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.429+0000 7f9fb936f640 1 -- 172.21.15.88:0/1767475928 learned_addr learned my addr 172.21.15.88:0/1767475928 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:03.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.429+0000 7f9fb836d640 1 --2- 172.21.15.88:0/1767475928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562de78a4d20 0x562de77fbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:03.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.429+0000 7f9fb8b6e640 1 --2- 172.21.15.88:0/1767475928 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x562de77fffe0 0x562de789e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:03.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.429+0000 7f9fb936f640 1 -- 172.21.15.88:0/1767475928 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x562de77fffe0 msgr2=0x562de789e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:03.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.429+0000 7f9fb936f640 1 --2- 172.21.15.88:0/1767475928 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x562de77fffe0 0x562de789e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.429+0000 7f9fb936f640 1 -- 172.21.15.88:0/1767475928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562de78a4d20 msgr2=0x562de77fbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:03.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.429+0000 7f9fb936f640 1 --2- 172.21.15.88:0/1767475928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562de78a4d20 0x562de77fbbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.429+0000 7f9fb936f640 1 -- 172.21.15.88:0/1767475928 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562de789ec30 con 0x562de78a5100 2024-09-16T01:43:03.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.429+0000 7f9fb836d640 1 --2- 172.21.15.88:0/1767475928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562de78a4d20 0x562de77fbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:03.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.429+0000 7f9fb936f640 1 --2- 172.21.15.88:0/1767475928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562de78a5100 0x562de77fc120 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f9fb00095d0 tx=0x7f9fb0009a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bf74e1e4277a5a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:03.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.429+0000 7f9fb736b640 1 -- 172.21.15.88:0/1767475928 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fb0017070 con 0x562de78a5100 2024-09-16T01:43:03.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.429+0000 7f9fb736b640 1 -- 172.21.15.88:0/1767475928 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fb0006680 con 0x562de78a5100 2024-09-16T01:43:03.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.429+0000 7f9fb736b640 1 -- 172.21.15.88:0/1767475928 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fb0006a70 con 0x562de78a5100 2024-09-16T01:43:03.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.430+0000 7f9fbb078f40 1 -- 172.21.15.88:0/1767475928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562de78a5100 msgr2=0x562de77fc120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:03.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.430+0000 7f9fbb078f40 1 --2- 172.21.15.88:0/1767475928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562de78a5100 0x562de77fc120 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f9fb00095d0 tx=0x7f9fb0009a90 comp rx=0 tx=0).stop 2024-09-16T01:43:03.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.430+0000 7f9fbb078f40 1 -- 172.21.15.88:0/1767475928 shutdown_connections 2024-09-16T01:43:03.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.430+0000 7f9fbb078f40 1 --2- 172.21.15.88:0/1767475928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562de78a5100 0x562de77fc120 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.430+0000 7f9fbb078f40 1 --2- 172.21.15.88:0/1767475928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562de78a4d20 0x562de77fbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.430+0000 7f9fbb078f40 1 --2- 172.21.15.88:0/1767475928 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x562de77fffe0 0x562de789e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.430+0000 7f9fbb078f40 1 -- 172.21.15.88:0/1767475928 >> 172.21.15.88:0/1767475928 conn(0x562de781b750 msgr2=0x562de77fcc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:03.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.430+0000 7f9fbb078f40 1 -- 172.21.15.88:0/1767475928 shutdown_connections 2024-09-16T01:43:03.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.430+0000 7f9fbb078f40 1 -- 172.21.15.88:0/1767475928 wait complete. 2024-09-16T01:43:03.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.431+0000 7f9fbb078f40 1 Processor -- start 2024-09-16T01:43:03.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.431+0000 7f9fbb078f40 1 -- start start 2024-09-16T01:43:03.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.431+0000 7f9fbb078f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x562de77fffe0 0x562de77d3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.431+0000 7f9fbb078f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562de78a4d20 0x562de78a5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.431+0000 7f9fbb078f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562de78a5100 0x562de789a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.431+0000 7f9fbb078f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x562de77d0530 con 0x562de78a4d20 2024-09-16T01:43:03.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.431+0000 7f9fbb078f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562de789d2a0 con 0x562de78a5100 2024-09-16T01:43:03.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.431+0000 7f9fbb078f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x562de78af7c0 con 0x562de77fffe0 2024-09-16T01:43:03.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.431+0000 7f9fb836d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562de78a4d20 0x562de78a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:03.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.431+0000 7f9fb836d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562de78a4d20 0x562de78a5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55662/0 (socket says 172.21.15.88:55662) 2024-09-16T01:43:03.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.432+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 learned_addr learned my addr 172.21.15.88:0/3156652455 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:03.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.432+0000 7f9fb8b6e640 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x562de77fffe0 0x562de77d3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:03.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.432+0000 7f9fb936f640 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562de78a5100 0x562de789a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:03.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.432+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x562de77fffe0 msgr2=0x562de77d3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:03.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.432+0000 7f9fb836d640 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x562de77fffe0 0x562de77d3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.432+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562de78a5100 msgr2=0x562de789a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:03.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.432+0000 7f9fb836d640 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562de78a5100 0x562de789a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.432+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562de789bf70 con 0x562de78a4d20 2024-09-16T01:43:03.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.432+0000 7f9fb836d640 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562de78a4d20 0x562de78a5530 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x562de77fbb10 tx=0x7f9fa000b5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:03.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.432+0000 7f9fb6369640 1 -- 172.21.15.88:0/3156652455 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa0014020 con 0x562de78a4d20 2024-09-16T01:43:03.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.432+0000 7f9fb6369640 1 -- 172.21.15.88:0/3156652455 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fa0010410 con 0x562de78a4d20 2024-09-16T01:43:03.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.432+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x562de78af340 con 0x562de78a4d20 2024-09-16T01:43:03.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.432+0000 7f9fb6369640 1 -- 172.21.15.88:0/3156652455 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa0008030 con 0x562de78a4d20 2024-09-16T01:43:03.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.432+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562de789dbb0 con 0x562de78a4d20 2024-09-16T01:43:03.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.434+0000 7f9fb6369640 1 -- 172.21.15.88:0/3156652455 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9fa0017020 con 0x562de78a4d20 2024-09-16T01:43:03.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.434+0000 7f9fb6369640 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9f9806acc0 0x7f9f9806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.434+0000 7f9fb6369640 1 -- 172.21.15.88:0/3156652455 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8146+0+0 (secure 0 0 0) 0x7f9fa008aa80 con 0x562de78a4d20 2024-09-16T01:43:03.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.434+0000 7f9fb8b6e640 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9f9806acc0 0x7f9f9806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:03.440 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:43:03.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.434+0000 7f9fbb078f40 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x562de78a5a70 0x562de77d2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.434+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de78977c0 con 0x562de78a5a70 2024-09-16T01:43:03.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.434+0000 7f9fb936f640 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x562de78a5a70 0x562de77d2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:03.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.435+0000 7f9fb8b6e640 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9f9806acc0 0x7f9f9806d180 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8056000 tx=0x7f9fa8007030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:03.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.435+0000 7f9fb936f640 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x562de78a5a70 0x562de77d2f00 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:03.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.437+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v115'846 uv820 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.437+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e115) -- 0x562de7897b90 con 0x562de78a5a70 2024-09-16T01:43:03.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.438+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v115'847 uv847 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.438+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7897ff0 con 0x562de78a5a70 2024-09-16T01:43:03.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.440+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v115'848 uv848 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.442 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:43:03.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.440+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794c080 con 0x562de78a5a70 2024-09-16T01:43:03.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.441+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v115'849 uv849 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.441+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794c420 con 0x562de78a5a70 2024-09-16T01:43:03.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.442+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v115'850 uv850 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.442+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794b070 con 0x562de78a5a70 2024-09-16T01:43:03.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.444+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v115'851 uv851 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.444+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794b460 con 0x562de78a5a70 2024-09-16T01:43:03.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.445+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v115'852 uv826 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.445+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e115) -- 0x562de794b460 con 0x562de78a5a70 2024-09-16T01:43:03.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.447+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v115'853 uv853 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.447+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794b410 con 0x562de78a5a70 2024-09-16T01:43:03.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.448+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v115'854 uv854 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.451 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:43:03.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.448+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794b410 con 0x562de78a5a70 2024-09-16T01:43:03.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.449+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v115'855 uv855 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.449+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794b410 con 0x562de78a5a70 2024-09-16T01:43:03.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.451+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v115'856 uv856 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.451+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794b410 con 0x562de78a5a70 2024-09-16T01:43:03.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.453+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v115'857 uv857 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.453+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794b410 con 0x562de78a5a70 2024-09-16T01:43:03.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.454+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v115'858 uv833 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.454+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e115) -- 0x562de794b410 con 0x562de78a5a70 2024-09-16T01:43:03.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.456+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v115'859 uv859 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.456+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794b410 con 0x562de78a5a70 2024-09-16T01:43:03.459 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:43:03.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.457+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v115'860 uv860 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.457+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794b410 con 0x562de78a5a70 2024-09-16T01:43:03.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.458+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v115'861 uv861 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.458+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794b410 con 0x562de78a5a70 2024-09-16T01:43:03.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.459+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v115'862 uv862 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.459+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794b410 con 0x562de78a5a70 2024-09-16T01:43:03.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.461+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v115'863 uv863 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.461+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794bcc0 con 0x562de78a5a70 2024-09-16T01:43:03.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.462+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v115'864 uv864 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.462+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794bca0 con 0x562de78a5a70 2024-09-16T01:43:03.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.463+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v115'865 uv839 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.463+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e115) -- 0x562de794bca0 con 0x562de78a5a70 2024-09-16T01:43:03.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.465+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v115'866 uv866 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.465+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794bca0 con 0x562de78a5a70 2024-09-16T01:43:03.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.466+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v115'867 uv867 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.468 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:43:03.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.466+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794bb70 con 0x562de78a5a70 2024-09-16T01:43:03.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.467+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v115'868 uv868 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.467+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794bb70 con 0x562de78a5a70 2024-09-16T01:43:03.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.468+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v115'869 uv869 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.468+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794bb70 con 0x562de78a5a70 2024-09-16T01:43:03.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.470+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v115'870 uv870 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.470+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794bb70 con 0x562de78a5a70 2024-09-16T01:43:03.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.471+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v115'871 uv845 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.471+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e115) -- 0x562de794bb70 con 0x562de78a5a70 2024-09-16T01:43:03.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.472+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v115'872 uv872 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.472+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794bb70 con 0x562de78a5a70 2024-09-16T01:43:03.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.473+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v115'873 uv873 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.476 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:43:03.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.473+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794bb70 con 0x562de78a5a70 2024-09-16T01:43:03.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.474+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v115'874 uv874 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.474+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794bb70 con 0x562de78a5a70 2024-09-16T01:43:03.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.476+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v115'875 uv875 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.476+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794bb70 con 0x562de78a5a70 2024-09-16T01:43:03.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.477+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v115'876 uv876 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.477+0000 7f9fbb078f40 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x562de794ccd0 0x562de794f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.477+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794bb70 con 0x562de794ccd0 2024-09-16T01:43:03.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.477+0000 7f9fb8b6e640 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x562de794ccd0 0x562de794f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:03.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.477+0000 7f9fb8b6e640 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x562de794ccd0 0x562de794f0c0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:03.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.479+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v115'618 uv601 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de794ccd0 2024-09-16T01:43:03.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.479+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e115) -- 0x562de794f600 con 0x562de794ccd0 2024-09-16T01:43:03.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.480+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v115'619 uv619 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de794ccd0 2024-09-16T01:43:03.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.480+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794f9a0 con 0x562de794ccd0 2024-09-16T01:43:03.483 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:43:03.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.481+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v115'620 uv620 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de794ccd0 2024-09-16T01:43:03.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.481+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de79501f0 con 0x562de794ccd0 2024-09-16T01:43:03.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.482+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v115'621 uv621 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de794ccd0 2024-09-16T01:43:03.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.482+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7950590 con 0x562de794ccd0 2024-09-16T01:43:03.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.483+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v115'622 uv622 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de794ccd0 2024-09-16T01:43:03.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.483+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7950930 con 0x562de794ccd0 2024-09-16T01:43:03.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.485+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v115'623 uv606 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de794ccd0 2024-09-16T01:43:03.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.485+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e115) -- 0x562de7950cd0 con 0x562de794ccd0 2024-09-16T01:43:03.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.486+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v115'624 uv624 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de794ccd0 2024-09-16T01:43:03.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.486+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7950cd0 con 0x562de794ccd0 2024-09-16T01:43:03.490 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:43:03.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.488+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v115'625 uv625 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de794ccd0 2024-09-16T01:43:03.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.488+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7950cd0 con 0x562de794ccd0 2024-09-16T01:43:03.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.489+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v115'626 uv626 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de794ccd0 2024-09-16T01:43:03.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.489+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7950cd0 con 0x562de794ccd0 2024-09-16T01:43:03.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.490+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v115'627 uv627 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de794ccd0 2024-09-16T01:43:03.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.490+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7950cd0 con 0x562de794ccd0 2024-09-16T01:43:03.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.491+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v115'628 uv611 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de794ccd0 2024-09-16T01:43:03.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.491+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e115) -- 0x562de7950cd0 con 0x562de794ccd0 2024-09-16T01:43:03.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.492+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v115'629 uv629 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de794ccd0 2024-09-16T01:43:03.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.492+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7950cd0 con 0x562de794ccd0 2024-09-16T01:43:03.495 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:43:03.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.493+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v115'630 uv630 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de794ccd0 2024-09-16T01:43:03.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.493+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7955150 con 0x562de794ccd0 2024-09-16T01:43:03.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.494+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v115'631 uv631 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de794ccd0 2024-09-16T01:43:03.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.494+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7955150 con 0x562de794ccd0 2024-09-16T01:43:03.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.496+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v115'632 uv632 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de794ccd0 2024-09-16T01:43:03.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.496+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7950cd0 con 0x562de794ccd0 2024-09-16T01:43:03.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.497+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v115'633 uv617 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de794ccd0 2024-09-16T01:43:03.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.497+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e115) -- 0x562de7950cd0 con 0x562de794ccd0 2024-09-16T01:43:03.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.498+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v115'634 uv634 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de794ccd0 2024-09-16T01:43:03.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.498+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7950cd0 con 0x562de794ccd0 2024-09-16T01:43:03.501 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:43:03.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.499+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v115'635 uv635 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de794ccd0 2024-09-16T01:43:03.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.499+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7951ce0 con 0x562de794ccd0 2024-09-16T01:43:03.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.500+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v115'636 uv636 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de794ccd0 2024-09-16T01:43:03.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.500+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7951ce0 con 0x562de794ccd0 2024-09-16T01:43:03.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.501+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v115'637 uv637 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de794ccd0 2024-09-16T01:43:03.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.501+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7950cd0 con 0x562de794ccd0 2024-09-16T01:43:03.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.502+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v115'638 uv638 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de794ccd0 2024-09-16T01:43:03.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.502+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7950cd0 con 0x562de78a5a70 2024-09-16T01:43:03.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.503+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v115'165 uv164 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.503+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e115) -- 0x562de7950cd0 con 0x562de78a5a70 2024-09-16T01:43:03.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.504+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v115'166 uv166 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.504+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7950cd0 con 0x562de78a5a70 2024-09-16T01:43:03.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.505+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v115'167 uv167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.507 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:43:03.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.505+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7951ce0 con 0x562de78a5a70 2024-09-16T01:43:03.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.506+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v115'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.506+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7951ce0 con 0x562de78a5a70 2024-09-16T01:43:03.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.507+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v115'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.507+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7950cd0 con 0x562de78a5a70 2024-09-16T01:43:03.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.508+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v115'170 uv170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.508+0000 7f9fbb078f40 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x562de79511e0 0x562de7953610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.508+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7953ca0 con 0x562de79511e0 2024-09-16T01:43:03.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.508+0000 7f9fb836d640 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x562de79511e0 0x562de7953610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:03.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.508+0000 7f9fb836d640 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x562de79511e0 0x562de7953610 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:03.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.509+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v115'123 uv122 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f9fa0020070 con 0x562de79511e0 2024-09-16T01:43:03.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.509+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e115) -- 0x562de79541b0 con 0x562de79511e0 2024-09-16T01:43:03.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.510+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v115'124 uv124 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9fa0020070 con 0x562de79511e0 2024-09-16T01:43:03.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.510+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de79546b0 con 0x562de79511e0 2024-09-16T01:43:03.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.511+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v115'125 uv125 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9fa0020070 con 0x562de79511e0 2024-09-16T01:43:03.513 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:43:03.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.511+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de794fe20 con 0x562de79511e0 2024-09-16T01:43:03.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.512+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v115'126 uv126 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9fa0020070 con 0x562de79511e0 2024-09-16T01:43:03.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.512+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7954b40 con 0x562de78a5a70 2024-09-16T01:43:03.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.513+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v115'165 uv164 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.514+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e115) -- 0x562de7954b40 con 0x562de78a5a70 2024-09-16T01:43:03.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.515+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v115'166 uv166 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.515+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7954b40 con 0x562de78a5a70 2024-09-16T01:43:03.518 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:43:03.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.516+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v115'167 uv167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.516+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7954b40 con 0x562de78a5a70 2024-09-16T01:43:03.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.517+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v115'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.517+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7954b40 con 0x562de78a5a70 2024-09-16T01:43:03.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.518+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v115'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.518+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7954b40 con 0x562de78a5a70 2024-09-16T01:43:03.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.519+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v115'170 uv170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.519+0000 7f9fbb078f40 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x562de79559a0 0x562de7957d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.519+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de79582d0 con 0x562de79559a0 2024-09-16T01:43:03.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.520+0000 7f9fb936f640 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x562de79559a0 0x562de7957d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:03.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.520+0000 7f9fb936f640 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x562de79559a0 0x562de7957d90 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:03.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.521+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v115'305 uv299 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.521+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e115) -- 0x562de79582d0 con 0x562de79559a0 2024-09-16T01:43:03.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.523+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v115'306 uv306 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.523+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7958330 con 0x562de79559a0 2024-09-16T01:43:03.526 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:43:03.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.524+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v115'307 uv307 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.524+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7959760 con 0x562de79559a0 2024-09-16T01:43:03.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.525+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v115'308 uv308 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.525+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7959760 con 0x562de79559a0 2024-09-16T01:43:03.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.526+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v115'309 uv309 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.526+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7958750 con 0x562de79559a0 2024-09-16T01:43:03.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.527+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v115'310 uv310 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.527+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7958750 con 0x562de79559a0 2024-09-16T01:43:03.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.528+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v115'311 uv304 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.528+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e115) -- 0x562de7958750 con 0x562de79559a0 2024-09-16T01:43:03.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.529+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v115'312 uv312 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.529+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7958750 con 0x562de79559a0 2024-09-16T01:43:03.532 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:43:03.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.530+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v115'313 uv313 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.530+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7958750 con 0x562de79559a0 2024-09-16T01:43:03.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.531+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v115'314 uv314 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.531+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7958750 con 0x562de79559a0 2024-09-16T01:43:03.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.532+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v115'315 uv315 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.532+0000 7f9fbb078f40 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x562de7958c60 0x562de795b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.532+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795b6d0 con 0x562de7958c60 2024-09-16T01:43:03.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.532+0000 7f9fb8b6e640 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x562de7958c60 0x562de795b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:03.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.533+0000 7f9fb8b6e640 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x562de7958c60 0x562de795b050 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:03.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.534+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v115'138 uv137 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de7958c60 2024-09-16T01:43:03.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.534+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e115) -- 0x562de795baa0 con 0x562de7958c60 2024-09-16T01:43:03.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.535+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v115'139 uv139 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de7958c60 2024-09-16T01:43:03.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.535+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795baa0 con 0x562de7958c60 2024-09-16T01:43:03.538 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:43:03.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.536+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v115'140 uv140 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de7958c60 2024-09-16T01:43:03.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.536+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795baa0 con 0x562de7958c60 2024-09-16T01:43:03.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.536+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v115'141 uv141 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de7958c60 2024-09-16T01:43:03.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.537+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795baa0 con 0x562de7958c60 2024-09-16T01:43:03.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.537+0000 7f9fb8b6e640 1 -- 172.21.15.88:0/3156652455 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v115'142 uv142 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9fa806b040 con 0x562de7958c60 2024-09-16T01:43:03.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.537+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795baa0 con 0x562de78a5a70 2024-09-16T01:43:03.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.538+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v115'165 uv164 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.539+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e115) -- 0x562de795baa0 con 0x562de78a5a70 2024-09-16T01:43:03.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.539+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v115'166 uv166 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.539+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795baa0 con 0x562de78a5a70 2024-09-16T01:43:03.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.540+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v115'167 uv167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.543 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:43:03.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.540+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795caf0 con 0x562de78a5a70 2024-09-16T01:43:03.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.541+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v115'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.541+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795cab0 con 0x562de78a5a70 2024-09-16T01:43:03.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.542+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v115'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.542+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795baa0 con 0x562de78a5a70 2024-09-16T01:43:03.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.543+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v115'170 uv170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de78a5a70 2024-09-16T01:43:03.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.543+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795baa0 con 0x562de79511e0 2024-09-16T01:43:03.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.544+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v115'273 uv267 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f9fa0020070 con 0x562de79511e0 2024-09-16T01:43:03.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.544+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e115) -- 0x562de795be40 con 0x562de79511e0 2024-09-16T01:43:03.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.545+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v115'274 uv274 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9fa0020070 con 0x562de79511e0 2024-09-16T01:43:03.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.545+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795c230 con 0x562de79511e0 2024-09-16T01:43:03.548 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:43:03.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.546+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v115'275 uv275 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9fa0020070 con 0x562de79511e0 2024-09-16T01:43:03.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.546+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795c620 con 0x562de79511e0 2024-09-16T01:43:03.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.547+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v115'276 uv276 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9fa0020070 con 0x562de79511e0 2024-09-16T01:43:03.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.547+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795c620 con 0x562de79511e0 2024-09-16T01:43:03.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.548+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v115'277 uv277 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9fa0020070 con 0x562de79511e0 2024-09-16T01:43:03.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.548+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795c360 con 0x562de79511e0 2024-09-16T01:43:03.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.549+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v115'278 uv272 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9fa0020070 con 0x562de79511e0 2024-09-16T01:43:03.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.549+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e115) -- 0x562de795c320 con 0x562de79511e0 2024-09-16T01:43:03.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.550+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v115'279 uv279 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fa0020070 con 0x562de79511e0 2024-09-16T01:43:03.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.550+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795c2d0 con 0x562de79511e0 2024-09-16T01:43:03.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.551+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v115'280 uv280 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fa0020070 con 0x562de79511e0 2024-09-16T01:43:03.553 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:43:03.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.551+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795c310 con 0x562de79511e0 2024-09-16T01:43:03.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.552+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v115'281 uv281 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fa0020070 con 0x562de79511e0 2024-09-16T01:43:03.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.552+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de7955090 con 0x562de79511e0 2024-09-16T01:43:03.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.553+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v115'282 uv282 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fa0020070 con 0x562de79511e0 2024-09-16T01:43:03.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.553+0000 7f9fbb078f40 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x562de7955090 0x562de795f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.553+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de7955090 2024-09-16T01:43:03.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.553+0000 7f9fb836d640 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x562de7955090 0x562de795f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:03.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.554+0000 7f9fb836d640 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x562de7955090 0x562de795f8b0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:03.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.554+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v115'120 uv119 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f9fa0020070 con 0x562de7955090 2024-09-16T01:43:03.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.554+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de7955090 2024-09-16T01:43:03.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.555+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v115'121 uv121 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f9fa0020070 con 0x562de7955090 2024-09-16T01:43:03.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.555+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de7955090 2024-09-16T01:43:03.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.556+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v115'122 uv122 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f9fa0020070 con 0x562de7955090 2024-09-16T01:43:03.559 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:43:03.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.556+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de7955090 2024-09-16T01:43:03.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.557+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v115'123 uv123 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f9fa0020070 con 0x562de7955090 2024-09-16T01:43:03.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.557+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de79511e0 2024-09-16T01:43:03.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.559+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v115'165 uv164 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9fa0020070 con 0x562de79511e0 2024-09-16T01:43:03.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.559+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de79511e0 2024-09-16T01:43:03.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.560+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v115'166 uv166 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fa0020070 con 0x562de79511e0 2024-09-16T01:43:03.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.560+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de79511e0 2024-09-16T01:43:03.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.561+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v115'167 uv167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fa0020070 con 0x562de79511e0 2024-09-16T01:43:03.563 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:43:03.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.561+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de79511e0 2024-09-16T01:43:03.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.562+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v115'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fa0020070 con 0x562de79511e0 2024-09-16T01:43:03.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.562+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de79511e0 2024-09-16T01:43:03.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.563+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v115'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fa0020070 con 0x562de79511e0 2024-09-16T01:43:03.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.563+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de79511e0 2024-09-16T01:43:03.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.564+0000 7f9fb836d640 1 -- 172.21.15.88:0/3156652455 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v115'170 uv170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fa0020070 con 0x562de79511e0 2024-09-16T01:43:03.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.564+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de79559a0 2024-09-16T01:43:03.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.565+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v115'140 uv139 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.565+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de79559a0 2024-09-16T01:43:03.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.566+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v115'141 uv141 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.566+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de79559a0 2024-09-16T01:43:03.569 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:43:03.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.567+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v115'142 uv142 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.567+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de79559a0 2024-09-16T01:43:03.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.568+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v115'143 uv143 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.568+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de79559a0 2024-09-16T01:43:03.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.570+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v115'144 uv144 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.570+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de79559a0 2024-09-16T01:43:03.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.571+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v115'339 uv331 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.571+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de79559a0 2024-09-16T01:43:03.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.572+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v115'340 uv340 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.572+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de79559a0 2024-09-16T01:43:03.575 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:43:03.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.573+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v115'341 uv341 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.573+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de79559a0 2024-09-16T01:43:03.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.574+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v115'342 uv342 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.574+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de79559a0 2024-09-16T01:43:03.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.575+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v115'343 uv343 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.575+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de79559a0 2024-09-16T01:43:03.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.576+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v115'344 uv338 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.576+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de79559a0 2024-09-16T01:43:03.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.577+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v115'345 uv345 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.577+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de79559a0 2024-09-16T01:43:03.580 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:43:03.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.578+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v115'346 uv346 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.578+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de79559a0 2024-09-16T01:43:03.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.579+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v115'347 uv347 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.579+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de79559a0 2024-09-16T01:43:03.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.580+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v115'348 uv348 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.580+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795f3f0 con 0x562de79559a0 2024-09-16T01:43:03.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.581+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v115'349 uv349 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.581+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x562de795fdf0 con 0x562de79559a0 2024-09-16T01:43:03.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.582+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v115'350 uv350 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9fb00068b0 con 0x562de79559a0 2024-09-16T01:43:03.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.582+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x562de78a5a70 msgr2=0x562de77d2f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:03.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.582+0000 7f9fbb078f40 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x562de78a5a70 0x562de77d2f00 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.582+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x562de79511e0 msgr2=0x562de7953610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:03.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.582+0000 7f9fbb078f40 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x562de79511e0 0x562de7953610 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.582+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x562de7955090 msgr2=0x562de795f8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:03.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.582+0000 7f9fbb078f40 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x562de7955090 0x562de795f8b0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.582+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x562de794ccd0 msgr2=0x562de794f0c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:03.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.582+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 reap_dead start 2024-09-16T01:43:03.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.582+0000 7f9fbb078f40 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x562de794ccd0 0x562de794f0c0 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.582+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x562de79559a0 msgr2=0x562de7957d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:03.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.582+0000 7f9fbb078f40 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x562de79559a0 0x562de7957d90 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.582+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x562de7958c60 msgr2=0x562de795b050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:03.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.582+0000 7f9fbb078f40 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x562de7958c60 0x562de795b050 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.582+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9f9806acc0 msgr2=0x7f9f9806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:03.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.582+0000 7f9fbb078f40 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9f9806acc0 0x7f9f9806d180 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8056000 tx=0x7f9fa8007030 comp rx=0 tx=0).stop 2024-09-16T01:43:03.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.582+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562de78a4d20 msgr2=0x562de78a5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:03.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.582+0000 7f9fbb078f40 1 --2- 172.21.15.88:0/3156652455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562de78a4d20 0x562de78a5530 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x562de77fbb10 tx=0x7f9fa000b5e0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.582+0000 7f9fb936f640 1 -- 172.21.15.88:0/3156652455 reap_dead start 2024-09-16T01:43:03.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.582+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 shutdown_connections 2024-09-16T01:43:03.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.582+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 >> 172.21.15.88:0/3156652455 conn(0x562de781b750 msgr2=0x562de77fcad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:03.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.582+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 shutdown_connections 2024-09-16T01:43:03.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.582+0000 7f9fbb078f40 1 -- 172.21.15.88:0/3156652455 wait complete. 2024-09-16T01:43:03.591 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata truncate mds0_inotable 0 2024-09-16T01:43:03.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.633+0000 7f6cb2f63f40 1 Processor -- start 2024-09-16T01:43:03.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.633+0000 7f6cb2f63f40 1 -- start start 2024-09-16T01:43:03.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.633+0000 7f6cb2f63f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56155a3b8940 0x56155a488dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.633+0000 7f6cb2f63f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56155a3b8d50 0x56155a48ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.633+0000 7f6cb2f63f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56155a48cf30 0x56155a48bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.633+0000 7f6cb2f63f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56155a3b92c0 con 0x56155a3b8d50 2024-09-16T01:43:03.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.633+0000 7f6cb2f63f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56155a309510 con 0x56155a48cf30 2024-09-16T01:43:03.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.633+0000 7f6cb2f63f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56155a3b6c80 con 0x56155a3b8940 2024-09-16T01:43:03.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.633+0000 7f6ca8258640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56155a3b8d50 0x56155a48ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:03.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.633+0000 7f6ca8258640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56155a3b8d50 0x56155a48ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55682/0 (socket says 172.21.15.88:55682) 2024-09-16T01:43:03.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.633+0000 7f6ca8258640 1 -- 172.21.15.88:0/2251714909 learned_addr learned my addr 172.21.15.88:0/2251714909 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:03.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.634+0000 7f6ca8258640 1 -- 172.21.15.88:0/2251714909 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56155a3b8940 msgr2=0x56155a488dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:03.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.634+0000 7f6ca8258640 1 --2- 172.21.15.88:0/2251714909 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56155a3b8940 0x56155a488dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.634+0000 7f6cb125a640 1 --2- 172.21.15.88:0/2251714909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56155a48cf30 0x56155a48bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:03.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.634+0000 7f6ca8258640 1 -- 172.21.15.88:0/2251714909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56155a48cf30 msgr2=0x56155a48bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:03.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.634+0000 7f6ca8258640 1 --2- 172.21.15.88:0/2251714909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56155a48cf30 0x56155a48bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.634+0000 7f6ca8258640 1 -- 172.21.15.88:0/2251714909 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56155a3fbb40 con 0x56155a3b8d50 2024-09-16T01:43:03.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.634+0000 7f6ca8258640 1 --2- 172.21.15.88:0/2251714909 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56155a3b8d50 0x56155a48ba00 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800a5e0 tx=0x7f6c98005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b03d1fc238ee360b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:03.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.634+0000 7f6cab7fe640 1 -- 172.21.15.88:0/2251714909 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98008a40 con 0x56155a3b8d50 2024-09-16T01:43:03.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.634+0000 7f6cab7fe640 1 -- 172.21.15.88:0/2251714909 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c98008c00 con 0x56155a3b8d50 2024-09-16T01:43:03.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.634+0000 7f6cab7fe640 1 -- 172.21.15.88:0/2251714909 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98013440 con 0x56155a3b8d50 2024-09-16T01:43:03.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.634+0000 7f6cb2f63f40 1 -- 172.21.15.88:0/2251714909 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56155a3b8d50 msgr2=0x56155a48ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:03.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.634+0000 7f6cb2f63f40 1 --2- 172.21.15.88:0/2251714909 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56155a3b8d50 0x56155a48ba00 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800a5e0 tx=0x7f6c98005f90 comp rx=0 tx=0).stop 2024-09-16T01:43:03.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.634+0000 7f6cb2f63f40 1 -- 172.21.15.88:0/2251714909 shutdown_connections 2024-09-16T01:43:03.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.634+0000 7f6cb2f63f40 1 --2- 172.21.15.88:0/2251714909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56155a48cf30 0x56155a48bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.634+0000 7f6cb2f63f40 1 --2- 172.21.15.88:0/2251714909 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56155a3b8d50 0x56155a48ba00 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.634+0000 7f6cb2f63f40 1 --2- 172.21.15.88:0/2251714909 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56155a3b8940 0x56155a488dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.634+0000 7f6cb2f63f40 1 -- 172.21.15.88:0/2251714909 >> 172.21.15.88:0/2251714909 conn(0x56155a402750 msgr2=0x56155a402b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:03.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.635+0000 7f6cb2f63f40 1 -- 172.21.15.88:0/2251714909 shutdown_connections 2024-09-16T01:43:03.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.635+0000 7f6cb2f63f40 1 -- 172.21.15.88:0/2251714909 wait complete. 2024-09-16T01:43:03.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.635+0000 7f6cb2f63f40 1 Processor -- start 2024-09-16T01:43:03.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.635+0000 7f6cb2f63f40 1 -- start start 2024-09-16T01:43:03.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.636+0000 7f6cb2f63f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56155a3b8940 0x56155a525130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.636+0000 7f6cb2f63f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56155a3b8d50 0x56155a3ff0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.636+0000 7f6cb2f63f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56155a48cf30 0x56155a486290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.636+0000 7f6cb2f63f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56155a3fdc30 con 0x56155a3b8d50 2024-09-16T01:43:03.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.636+0000 7f6cb2f63f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56155a489710 con 0x56155a48cf30 2024-09-16T01:43:03.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.636+0000 7f6cb2f63f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56155a3fbde0 con 0x56155a3b8940 2024-09-16T01:43:03.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.636+0000 7f6ca8258640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56155a3b8d50 0x56155a3ff0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:03.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.636+0000 7f6ca8258640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56155a3b8d50 0x56155a3ff0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55684/0 (socket says 172.21.15.88:55684) 2024-09-16T01:43:03.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.636+0000 7f6ca8258640 1 -- 172.21.15.88:0/1659302622 learned_addr learned my addr 172.21.15.88:0/1659302622 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:03.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.636+0000 7f6cb125a640 1 --2- 172.21.15.88:0/1659302622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56155a48cf30 0x56155a486290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:03.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.636+0000 7f6cb0a59640 1 --2- 172.21.15.88:0/1659302622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56155a3b8940 0x56155a525130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:03.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.636+0000 7f6ca8258640 1 -- 172.21.15.88:0/1659302622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56155a3b8940 msgr2=0x56155a525130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:03.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.636+0000 7f6ca8258640 1 --2- 172.21.15.88:0/1659302622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56155a3b8940 0x56155a525130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.636+0000 7f6ca8258640 1 -- 172.21.15.88:0/1659302622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56155a48cf30 msgr2=0x56155a486290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:03.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.636+0000 7f6ca8258640 1 --2- 172.21.15.88:0/1659302622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56155a48cf30 0x56155a486290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.636+0000 7f6ca8258640 1 -- 172.21.15.88:0/1659302622 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56155a48e850 con 0x56155a3b8d50 2024-09-16T01:43:03.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.636+0000 7f6ca8258640 1 --2- 172.21.15.88:0/1659302622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56155a3b8d50 0x56155a3ff0a0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f6c98000c00 tx=0x7f6c98002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:03.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.636+0000 7f6caa7fc640 1 -- 172.21.15.88:0/1659302622 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98020070 con 0x56155a3b8d50 2024-09-16T01:43:03.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.636+0000 7f6caa7fc640 1 -- 172.21.15.88:0/1659302622 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c98013a80 con 0x56155a3b8d50 2024-09-16T01:43:03.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.637+0000 7f6caa7fc640 1 -- 172.21.15.88:0/1659302622 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98013e50 con 0x56155a3b8d50 2024-09-16T01:43:03.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.637+0000 7f6cb2f63f40 1 -- 172.21.15.88:0/1659302622 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56155a48eb30 con 0x56155a3b8d50 2024-09-16T01:43:03.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.637+0000 7f6cb2f63f40 1 -- 172.21.15.88:0/1659302622 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56155a48f0f0 con 0x56155a3b8d50 2024-09-16T01:43:03.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.638+0000 7f6caa7fc640 1 -- 172.21.15.88:0/1659302622 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6c98023070 con 0x56155a3b8d50 2024-09-16T01:43:03.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.638+0000 7f6caa7fc640 1 --2- 172.21.15.88:0/1659302622 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6c9006ae60 0x7f6c9006d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.638+0000 7f6caa7fc640 1 -- 172.21.15.88:0/1659302622 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8146+0+0 (secure 0 0 0) 0x7f6c9808c400 con 0x56155a3b8d50 2024-09-16T01:43:03.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.638+0000 7f6cb0a59640 1 --2- 172.21.15.88:0/1659302622 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6c9006ae60 0x7f6c9006d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:03.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.639+0000 7f6cb2f63f40 1 --2- 172.21.15.88:0/1659302622 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x56155a3ff5e0 0x56155a4878b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.639+0000 7f6cb2f63f40 1 -- 172.21.15.88:0/1659302622 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:1 12.13 12:c91d4a1d:::mds0_inotable:head [truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115) -- 0x56155a487df0 con 0x56155a3ff5e0 2024-09-16T01:43:03.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.639+0000 7f6cb125a640 1 --2- 172.21.15.88:0/1659302622 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x56155a3ff5e0 0x56155a4878b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:03.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.639+0000 7f6cb0a59640 1 --2- 172.21.15.88:0/1659302622 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6c9006ae60 0x7f6c9006d320 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x56155a528da0 tx=0x7f6ca0005820 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:03.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.639+0000 7f6cb125a640 1 --2- 172.21.15.88:0/1659302622 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x56155a3ff5e0 0x56155a4878b0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:03.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.640+0000 7f6cb125a640 1 -- 172.21.15.88:0/1659302622 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(1 mds0_inotable [truncate 0] v115'283 uv283 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6cac05e040 con 0x56155a3ff5e0 2024-09-16T01:43:03.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.640+0000 7f6cb2f63f40 1 -- 172.21.15.88:0/1659302622 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x56155a3ff5e0 msgr2=0x56155a4878b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:03.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.640+0000 7f6cb2f63f40 1 --2- 172.21.15.88:0/1659302622 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x56155a3ff5e0 0x56155a4878b0 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.640+0000 7f6cb2f63f40 1 -- 172.21.15.88:0/1659302622 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6c9006ae60 msgr2=0x7f6c9006d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:03.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.640+0000 7f6cb2f63f40 1 --2- 172.21.15.88:0/1659302622 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6c9006ae60 0x7f6c9006d320 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x56155a528da0 tx=0x7f6ca0005820 comp rx=0 tx=0).stop 2024-09-16T01:43:03.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.640+0000 7f6cb2f63f40 1 -- 172.21.15.88:0/1659302622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56155a3b8d50 msgr2=0x56155a3ff0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:03.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.640+0000 7f6cb2f63f40 1 --2- 172.21.15.88:0/1659302622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56155a3b8d50 0x56155a3ff0a0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f6c98000c00 tx=0x7f6c98002fb0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.640+0000 7f6cb125a640 1 -- 172.21.15.88:0/1659302622 reap_dead start 2024-09-16T01:43:03.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.640+0000 7f6cb2f63f40 1 -- 172.21.15.88:0/1659302622 shutdown_connections 2024-09-16T01:43:03.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.640+0000 7f6cb2f63f40 1 -- 172.21.15.88:0/1659302622 >> 172.21.15.88:0/1659302622 conn(0x56155a402750 msgr2=0x56155a402b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:03.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.640+0000 7f6cb2f63f40 1 -- 172.21.15.88:0/1659302622 shutdown_connections 2024-09-16T01:43:03.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.640+0000 7f6cb2f63f40 1 -- 172.21.15.88:0/1659302622 wait complete. 2024-09-16T01:43:03.649 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:43:03.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.748+0000 7fde15de8640 1 Processor -- start 2024-09-16T01:43:03.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.748+0000 7fde15de8640 1 -- start start 2024-09-16T01:43:03.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.749+0000 7fde15de8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde100d0760 0x7fde100b5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.749+0000 7fde15de8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde100cd9b0 con 0x7fde100b4e60 2024-09-16T01:43:03.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.749+0000 7fde15de8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde100cdc90 con 0x7fde1015cb20 2024-09-16T01:43:03.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.749+0000 7fde15de8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fde100ce5c0 con 0x7fde100d0760 2024-09-16T01:43:03.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.749+0000 7fde0f7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fde100b4e60 0x7fde100b5c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57066/0 (socket says 172.21.15.88:57066) 2024-09-16T01:43:03.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.749+0000 7fde0f7fe640 1 -- 172.21.15.88:0/1074175654 learned_addr learned my addr 172.21.15.88:0/1074175654 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:03.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.749+0000 7fde07fff640 1 --2- 172.21.15.88:0/1074175654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde100d0760 0x7fde100b5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:03.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.749+0000 7fde0effd640 1 -- 172.21.15.88:0/1074175654 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 522606613 0 0) 0x7fde100cd9b0 con 0x7fde100b4e60 2024-09-16T01:43:03.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.749+0000 7fde0effd640 1 -- 172.21.15.88:0/1074175654 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddfc003520 con 0x7fde100b4e60 2024-09-16T01:43:03.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.749+0000 7fde0effd640 1 -- 172.21.15.88:0/1074175654 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4207709583 0 0) 0x7fde100cdc90 con 0x7fde1015cb20 2024-09-16T01:43:03.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.749+0000 7fde0effd640 1 -- 172.21.15.88:0/1074175654 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde100cd9b0 con 0x7fde1015cb20 2024-09-16T01:43:03.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.749+0000 7fde07fff640 1 -- 172.21.15.88:0/1074175654 >> v1:172.21.15.154:6789/0 conn(0x7fde1015cb20 legacy=0x7fde1015cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:03.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.749+0000 7fde07fff640 1 -- 172.21.15.88:0/1074175654 >> v1:172.21.15.88:6789/0 conn(0x7fde100b4e60 legacy=0x7fde100b5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:03.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.749+0000 7fde07fff640 1 -- 172.21.15.88:0/1074175654 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde100cdf70 con 0x7fde100d0760 2024-09-16T01:43:03.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.750+0000 7fde07fff640 1 --2- 172.21.15.88:0/1074175654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde100d0760 0x7fde100b5240 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fddf80097d0 tx=0x7fddf802a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fc1339beb441e18 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:03.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.750+0000 7fde0effd640 1 -- 172.21.15.88:0/1074175654 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf8037070 con 0x7fde100d0760 2024-09-16T01:43:03.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.750+0000 7fde0effd640 1 -- 172.21.15.88:0/1074175654 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddf8032440 con 0x7fde100d0760 2024-09-16T01:43:03.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.750+0000 7fde0effd640 1 -- 172.21.15.88:0/1074175654 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf8032810 con 0x7fde100d0760 2024-09-16T01:43:03.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.751+0000 7fde15de8640 1 -- 172.21.15.88:0/1074175654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde100d0760 msgr2=0x7fde100b5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:03.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.751+0000 7fde15de8640 1 --2- 172.21.15.88:0/1074175654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde100d0760 0x7fde100b5240 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fddf80097d0 tx=0x7fddf802a210 comp rx=0 tx=0).stop 2024-09-16T01:43:03.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.751+0000 7fde15de8640 1 -- 172.21.15.88:0/1074175654 shutdown_connections 2024-09-16T01:43:03.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.751+0000 7fde15de8640 1 --2- 172.21.15.88:0/1074175654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde100d0760 0x7fde100b5240 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.751+0000 7fde15de8640 1 -- 172.21.15.88:0/1074175654 >> 172.21.15.88:0/1074175654 conn(0x7fde1006cb50 msgr2=0x7fde1006cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:03.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.751+0000 7fde15de8640 1 -- 172.21.15.88:0/1074175654 shutdown_connections 2024-09-16T01:43:03.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.751+0000 7fde15de8640 1 -- 172.21.15.88:0/1074175654 wait complete. 2024-09-16T01:43:03.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.752+0000 7fde15de8640 1 Processor -- start 2024-09-16T01:43:03.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.752+0000 7fde15de8640 1 -- start start 2024-09-16T01:43:03.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.753+0000 7fde15de8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde100b4e60 0x7fde1013ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.753+0000 7fde15de8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde100d0760 0x7fde1013d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.753+0000 7fde15de8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde1015cb20 0x7fde1013da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.753+0000 7fde15de8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fde1015fc00 con 0x7fde100b4e60 2024-09-16T01:43:03.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.753+0000 7fde15de8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fde100b5780 con 0x7fde100d0760 2024-09-16T01:43:03.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.753+0000 7fde15de8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fde1015fd80 con 0x7fde1015cb20 2024-09-16T01:43:03.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.753+0000 7fde0f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde100b4e60 0x7fde1013ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:03.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.753+0000 7fde0f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde100b4e60 0x7fde1013ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55694/0 (socket says 172.21.15.88:55694) 2024-09-16T01:43:03.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.753+0000 7fde0f7fe640 1 -- 172.21.15.88:0/1194550723 learned_addr learned my addr 172.21.15.88:0/1194550723 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:03.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.753+0000 7fde07fff640 1 --2- 172.21.15.88:0/1194550723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde100d0760 0x7fde1013d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:03.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.753+0000 7fde0ffff640 1 --2- 172.21.15.88:0/1194550723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde1015cb20 0x7fde1013da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:03.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.753+0000 7fde0f7fe640 1 -- 172.21.15.88:0/1194550723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde1015cb20 msgr2=0x7fde1013da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:03.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.753+0000 7fde0f7fe640 1 --2- 172.21.15.88:0/1194550723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde1015cb20 0x7fde1013da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.753+0000 7fde0f7fe640 1 -- 172.21.15.88:0/1194550723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde100d0760 msgr2=0x7fde1013d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:03.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.753+0000 7fde0f7fe640 1 --2- 172.21.15.88:0/1194550723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde100d0760 0x7fde1013d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:03.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.753+0000 7fde0f7fe640 1 -- 172.21.15.88:0/1194550723 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde1013b180 con 0x7fde100b4e60 2024-09-16T01:43:03.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.753+0000 7fde0ffff640 1 --2- 172.21.15.88:0/1194550723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde1015cb20 0x7fde1013da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:03.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.753+0000 7fde0f7fe640 1 --2- 172.21.15.88:0/1194550723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde100b4e60 0x7fde1013ac40 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7fde0000c430 tx=0x7fde0000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:03.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.753+0000 7fde0cff9640 1 -- 172.21.15.88:0/1194550723 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde00017070 con 0x7fde100b4e60 2024-09-16T01:43:03.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.753+0000 7fde0cff9640 1 -- 172.21.15.88:0/1194550723 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde00005ce0 con 0x7fde100b4e60 2024-09-16T01:43:03.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.753+0000 7fde15de8640 1 -- 172.21.15.88:0/1194550723 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde1013b5f0 con 0x7fde100b4e60 2024-09-16T01:43:03.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.753+0000 7fde0cff9640 1 -- 172.21.15.88:0/1194550723 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde000072f0 con 0x7fde100b4e60 2024-09-16T01:43:03.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.753+0000 7fde15de8640 1 -- 172.21.15.88:0/1194550723 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde1013bbb0 con 0x7fde100b4e60 2024-09-16T01:43:03.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.754+0000 7fde15de8640 1 -- 172.21.15.88:0/1194550723 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde100b5240 con 0x7fde100b4e60 2024-09-16T01:43:03.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.755+0000 7fde0cff9640 1 -- 172.21.15.88:0/1194550723 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fde0001a070 con 0x7fde100b4e60 2024-09-16T01:43:03.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.755+0000 7fde0cff9640 1 --2- 172.21.15.88:0/1194550723 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fddf006c2e0 0x7fddf006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:03.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.755+0000 7fde0cff9640 1 -- 172.21.15.88:0/1194550723 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8146+0+0 (secure 0 0 0) 0x7fde0008b200 con 0x7fde100b4e60 2024-09-16T01:43:03.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.755+0000 7fde07fff640 1 --2- 172.21.15.88:0/1194550723 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fddf006c2e0 0x7fddf006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:03.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.756+0000 7fde07fff640 1 --2- 172.21.15.88:0/1194550723 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fddf006c2e0 0x7fddf006e7a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fddf8034000 tx=0x7fddf802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:03.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.760+0000 7fde0cff9640 1 -- 172.21.15.88:0/1194550723 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fde00058fe0 con 0x7fde100b4e60 2024-09-16T01:43:03.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:03.959+0000 7fde15de8640 1 -- 172.21.15.88:0/1194550723 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fde1006d210 con 0x7fde100b4e60 2024-09-16T01:43:04.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.367+0000 7fde0cff9640 1 -- 172.21.15.88:0/1194550723 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v238) ==== 162+0+0 (secure 0 0 0) 0x7fde0005de20 con 0x7fde100b4e60 2024-09-16T01:43:04.369 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:43:04.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.369+0000 7fde15de8640 1 -- 172.21.15.88:0/1194550723 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fddf006c2e0 msgr2=0x7fddf006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:04.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.369+0000 7fde15de8640 1 --2- 172.21.15.88:0/1194550723 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fddf006c2e0 0x7fddf006e7a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fddf8034000 tx=0x7fddf802e000 comp rx=0 tx=0).stop 2024-09-16T01:43:04.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.369+0000 7fde15de8640 1 -- 172.21.15.88:0/1194550723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde100b4e60 msgr2=0x7fde1013ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:04.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.369+0000 7fde15de8640 1 --2- 172.21.15.88:0/1194550723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde100b4e60 0x7fde1013ac40 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7fde0000c430 tx=0x7fde0000c8f0 comp rx=0 tx=0).stop 2024-09-16T01:43:04.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.369+0000 7fde15de8640 1 -- 172.21.15.88:0/1194550723 shutdown_connections 2024-09-16T01:43:04.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.369+0000 7fde15de8640 1 --2- 172.21.15.88:0/1194550723 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fddf006c2e0 0x7fddf006e7a0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:04.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.369+0000 7fde15de8640 1 --2- 172.21.15.88:0/1194550723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde1015cb20 0x7fde1013da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:04.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.369+0000 7fde15de8640 1 --2- 172.21.15.88:0/1194550723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde100d0760 0x7fde1013d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:04.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.369+0000 7fde15de8640 1 --2- 172.21.15.88:0/1194550723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde100b4e60 0x7fde1013ac40 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:04.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.369+0000 7fde15de8640 1 -- 172.21.15.88:0/1194550723 >> 172.21.15.88:0/1194550723 conn(0x7fde1006cb50 msgr2=0x7fde1015dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:04.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.369+0000 7fde15de8640 1 -- 172.21.15.88:0/1194550723 shutdown_connections 2024-09-16T01:43:04.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.369+0000 7fde15de8640 1 -- 172.21.15.88:0/1194550723 wait complete. 2024-09-16T01:43:04.380 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:43:04.378+0000 7f268a7c6640 -1 log_channel(cluster) log [ERR] : error decoding table object 'mds0_inotable': End of buffer [buffer:2] 2024-09-16T01:43:04.381 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:43:04.379+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:43:04.381 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:43:04.379+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:43:04.385 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:43:04.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.481+0000 7f6d9f93b640 1 Processor -- start 2024-09-16T01:43:04.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.481+0000 7f6d9f93b640 1 -- start start 2024-09-16T01:43:04.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.482+0000 7f6d9f93b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6d9806e780 0x7f6d980ae890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:04.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.482+0000 7f6d9f93b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6d98073130 0x7f6d98073580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:04.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.482+0000 7f6d9f93b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d980b5220 con 0x7f6d9815e560 2024-09-16T01:43:04.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.482+0000 7f6d9f93b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6d9806ebc0 con 0x7f6d98073130 2024-09-16T01:43:04.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.482+0000 7f6d9f93b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6d98073ac0 con 0x7f6d9806e780 2024-09-16T01:43:04.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.482+0000 7f6d9ded8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6d98073130 0x7f6d98073580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:04.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.482+0000 7f6d9ded8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6d98073130 0x7f6d98073580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55706/0 (socket says 172.21.15.88:55706) 2024-09-16T01:43:04.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.482+0000 7f6d9ded8640 1 -- 172.21.15.88:0/1071326751 learned_addr learned my addr 172.21.15.88:0/1071326751 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:04.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.482+0000 7f6d9ded8640 1 -- 172.21.15.88:0/1071326751 >> v2:172.21.15.154:3300/0 conn(0x7f6d9806e780 msgr2=0x7f6d980ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:04.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.482+0000 7f6d9ded8640 1 --2- 172.21.15.88:0/1071326751 >> v2:172.21.15.154:3300/0 conn(0x7f6d9806e780 0x7f6d980ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:04.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.482+0000 7f6d9ded8640 1 -- 172.21.15.88:0/1071326751 >> v1:172.21.15.154:6789/0 conn(0x7f6d9815e560 legacy=0x7f6d98072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:04.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.482+0000 7f6d9ded8640 1 -- 172.21.15.88:0/1071326751 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d980b54c0 con 0x7f6d98073130 2024-09-16T01:43:04.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.483+0000 7f6d9ded8640 1 --2- 172.21.15.88:0/1071326751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d98073130 0x7f6d98073580 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f6d980b4cb0 tx=0x7f6d9002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a615d0e6ac370dae server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:04.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.483+0000 7f6d8ffff640 1 -- 172.21.15.88:0/1071326751 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d90036070 con 0x7f6d98073130 2024-09-16T01:43:04.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.483+0000 7f6d8ffff640 1 -- 172.21.15.88:0/1071326751 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d90031600 con 0x7f6d98073130 2024-09-16T01:43:04.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.483+0000 7f6d8ffff640 1 -- 172.21.15.88:0/1071326751 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d900319d0 con 0x7f6d98073130 2024-09-16T01:43:04.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.484+0000 7f6d9f93b640 1 -- 172.21.15.88:0/1071326751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d98073130 msgr2=0x7f6d98073580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:04.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.484+0000 7f6d9f93b640 1 --2- 172.21.15.88:0/1071326751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d98073130 0x7f6d98073580 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f6d980b4cb0 tx=0x7f6d9002a660 comp rx=0 tx=0).stop 2024-09-16T01:43:04.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.484+0000 7f6d9f93b640 1 -- 172.21.15.88:0/1071326751 shutdown_connections 2024-09-16T01:43:04.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.484+0000 7f6d9f93b640 1 --2- 172.21.15.88:0/1071326751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d98073130 0x7f6d98073580 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:04.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.484+0000 7f6d9f93b640 1 --2- 172.21.15.88:0/1071326751 >> v2:172.21.15.154:3300/0 conn(0x7f6d9806e780 0x7f6d980ae890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:04.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.484+0000 7f6d9f93b640 1 -- 172.21.15.88:0/1071326751 >> 172.21.15.88:0/1071326751 conn(0x7f6d9806bc10 msgr2=0x7f6d9806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:04.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.484+0000 7f6d9f93b640 1 -- 172.21.15.88:0/1071326751 shutdown_connections 2024-09-16T01:43:04.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.484+0000 7f6d9f93b640 1 -- 172.21.15.88:0/1071326751 wait complete. 2024-09-16T01:43:04.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.485+0000 7f6d9f93b640 1 Processor -- start 2024-09-16T01:43:04.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.485+0000 7f6d9f93b640 1 -- start start 2024-09-16T01:43:04.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.486+0000 7f6d9f93b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d9806e780 0x7f6d981f7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:04.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.486+0000 7f6d9f93b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d98073130 0x7f6d98150050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:04.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.486+0000 7f6d9f93b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d9815e560 0x7f6d981f5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:04.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.486+0000 7f6d9f93b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6d980b5d60 con 0x7f6d9806e780 2024-09-16T01:43:04.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.486+0000 7f6d9f93b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6d980b5a80 con 0x7f6d98073130 2024-09-16T01:43:04.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.486+0000 7f6d9f93b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6d9815e940 con 0x7f6d9815e560 2024-09-16T01:43:04.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.486+0000 7f6d9d6d7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d9806e780 0x7f6d981f7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:04.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.486+0000 7f6d9d6d7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d9806e780 0x7f6d981f7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55714/0 (socket says 172.21.15.88:55714) 2024-09-16T01:43:04.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.486+0000 7f6d9d6d7640 1 -- 172.21.15.88:0/2763894408 learned_addr learned my addr 172.21.15.88:0/2763894408 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:04.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.486+0000 7f6d9ced6640 1 --2- 172.21.15.88:0/2763894408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d98073130 0x7f6d98150050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:04.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.486+0000 7f6d9ded8640 1 --2- 172.21.15.88:0/2763894408 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d9815e560 0x7f6d981f5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:04.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.486+0000 7f6d9d6d7640 1 -- 172.21.15.88:0/2763894408 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d9815e560 msgr2=0x7f6d981f5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:04.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.486+0000 7f6d9d6d7640 1 --2- 172.21.15.88:0/2763894408 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d9815e560 0x7f6d981f5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:04.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.486+0000 7f6d9d6d7640 1 -- 172.21.15.88:0/2763894408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d98073130 msgr2=0x7f6d98150050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:04.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.486+0000 7f6d9d6d7640 1 --2- 172.21.15.88:0/2763894408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d98073130 0x7f6d98150050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:04.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.486+0000 7f6d9d6d7640 1 -- 172.21.15.88:0/2763894408 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d98155180 con 0x7f6d9806e780 2024-09-16T01:43:04.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.486+0000 7f6d9ced6640 1 --2- 172.21.15.88:0/2763894408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d98073130 0x7f6d98150050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:04.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.486+0000 7f6d9d6d7640 1 --2- 172.21.15.88:0/2763894408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d9806e780 0x7f6d981f7540 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f6d9400b3e0 tx=0x7f6d9400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:04.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.487+0000 7f6d8dffb640 1 -- 172.21.15.88:0/2763894408 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d94015020 con 0x7f6d9806e780 2024-09-16T01:43:04.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.487+0000 7f6d8dffb640 1 -- 172.21.15.88:0/2763894408 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d94011850 con 0x7f6d9806e780 2024-09-16T01:43:04.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.487+0000 7f6d9f93b640 1 -- 172.21.15.88:0/2763894408 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d98155460 con 0x7f6d9806e780 2024-09-16T01:43:04.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.487+0000 7f6d8dffb640 1 -- 172.21.15.88:0/2763894408 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d94011c40 con 0x7f6d9806e780 2024-09-16T01:43:04.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.487+0000 7f6d9f93b640 1 -- 172.21.15.88:0/2763894408 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d98150590 con 0x7f6d9806e780 2024-09-16T01:43:04.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.488+0000 7f6d9f93b640 1 -- 172.21.15.88:0/2763894408 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d98073580 con 0x7f6d9806e780 2024-09-16T01:43:04.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.488+0000 7f6d8dffb640 1 -- 172.21.15.88:0/2763894408 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6d94018020 con 0x7f6d9806e780 2024-09-16T01:43:04.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.489+0000 7f6d8dffb640 1 --2- 172.21.15.88:0/2763894408 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6d7c06c2b0 0x7f6d7c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:04.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.489+0000 7f6d8dffb640 1 -- 172.21.15.88:0/2763894408 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8232+0+0 (secure 0 0 0) 0x7f6d9408bde0 con 0x7f6d9806e780 2024-09-16T01:43:04.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.489+0000 7f6d9ced6640 1 --2- 172.21.15.88:0/2763894408 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6d7c06c2b0 0x7f6d7c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:04.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.489+0000 7f6d9ced6640 1 --2- 172.21.15.88:0/2763894408 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6d7c06c2b0 0x7f6d7c06e770 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f6d981579d0 tx=0x7f6d88008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:04.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.493+0000 7f6d8dffb640 1 -- 172.21.15.88:0/2763894408 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6d9401f020 con 0x7f6d9806e780 2024-09-16T01:43:04.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.694+0000 7f6d9f93b640 1 -- 172.21.15.88:0/2763894408 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d981e9570 con 0x7f6d9806e780 2024-09-16T01:43:04.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.694+0000 7f6d8dffb640 1 -- 172.21.15.88:0/2763894408 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) ==== 95+0+3566 (secure 0 0 0) 0x7f6d940172c0 con 0x7f6d9806e780 2024-09-16T01:43:04.697 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:04.697 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":240,"btime":"2024-09-16T01:43:04:381011+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:43:04.380964+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:43:04.698 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 240 2024-09-16T01:43:04.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.696+0000 7f6d9f93b640 1 -- 172.21.15.88:0/2763894408 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6d7c06c2b0 msgr2=0x7f6d7c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:04.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.696+0000 7f6d9f93b640 1 --2- 172.21.15.88:0/2763894408 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6d7c06c2b0 0x7f6d7c06e770 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f6d981579d0 tx=0x7f6d88008040 comp rx=0 tx=0).stop 2024-09-16T01:43:04.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.696+0000 7f6d9f93b640 1 -- 172.21.15.88:0/2763894408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d9806e780 msgr2=0x7f6d981f7540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:04.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.696+0000 7f6d9f93b640 1 --2- 172.21.15.88:0/2763894408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d9806e780 0x7f6d981f7540 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f6d9400b3e0 tx=0x7f6d9400b8a0 comp rx=0 tx=0).stop 2024-09-16T01:43:04.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.696+0000 7f6d9f93b640 1 -- 172.21.15.88:0/2763894408 shutdown_connections 2024-09-16T01:43:04.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.696+0000 7f6d9f93b640 1 --2- 172.21.15.88:0/2763894408 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6d7c06c2b0 0x7f6d7c06e770 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:04.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.696+0000 7f6d9f93b640 1 --2- 172.21.15.88:0/2763894408 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d9815e560 0x7f6d981f5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:04.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.696+0000 7f6d9f93b640 1 --2- 172.21.15.88:0/2763894408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d98073130 0x7f6d98150050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:04.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.696+0000 7f6d9f93b640 1 --2- 172.21.15.88:0/2763894408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d9806e780 0x7f6d981f7540 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:04.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.696+0000 7f6d9f93b640 1 -- 172.21.15.88:0/2763894408 >> 172.21.15.88:0/2763894408 conn(0x7f6d9806bc10 msgr2=0x7f6d981e8000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:04.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.697+0000 7f6d9f93b640 1 -- 172.21.15.88:0/2763894408 shutdown_connections 2024-09-16T01:43:04.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.697+0000 7f6d9f93b640 1 -- 172.21.15.88:0/2763894408 wait complete. 2024-09-16T01:43:04.710 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:43:04.710 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:43:04.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.805+0000 7fa4180af640 1 Processor -- start 2024-09-16T01:43:04.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.805+0000 7fa4180af640 1 -- start start 2024-09-16T01:43:04.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.806+0000 7fa4180af640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa41015e950 0x7fa41006ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.806+0000 7fa4180af640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa41006ca30 con 0x7fa4100735b0 2024-09-16T01:43:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.806+0000 7fa4180af640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa41006faf0 con 0x7fa410072050 2024-09-16T01:43:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.806+0000 7fa4180af640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa41006fdd0 con 0x7fa41015e950 2024-09-16T01:43:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.806+0000 7fa41564a640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa4100735b0 0x7fa41006bc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57072/0 (socket says 172.21.15.88:57072) 2024-09-16T01:43:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.806+0000 7fa41564a640 1 -- 172.21.15.88:0/764275770 learned_addr learned my addr 172.21.15.88:0/764275770 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.807+0000 7fa414e49640 1 -- 172.21.15.88:0/764275770 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2107218548 0 0) 0x7fa41006ca30 con 0x7fa4100735b0 2024-09-16T01:43:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.807+0000 7fa414e49640 1 -- 172.21.15.88:0/764275770 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3f8003520 con 0x7fa4100735b0 2024-09-16T01:43:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.807+0000 7fa415e4b640 1 --2- 172.21.15.88:0/764275770 >> v2:172.21.15.154:3300/0 conn(0x7fa41015e950 0x7fa41006ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.807+0000 7fa414e49640 1 -- 172.21.15.88:0/764275770 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2396219716 0 0) 0x7fa3f8003520 con 0x7fa4100735b0 2024-09-16T01:43:04.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.807+0000 7fa414e49640 1 -- 172.21.15.88:0/764275770 >> v2:172.21.15.154:3300/0 conn(0x7fa41015e950 msgr2=0x7fa41006ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:04.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.807+0000 7fa414e49640 1 --2- 172.21.15.88:0/764275770 >> v2:172.21.15.154:3300/0 conn(0x7fa41015e950 0x7fa41006ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:04.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.807+0000 7fa414e49640 1 -- 172.21.15.88:0/764275770 >> v1:172.21.15.154:6789/0 conn(0x7fa410072050 legacy=0x7fa41006c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:04.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.807+0000 7fa414e49640 1 -- 172.21.15.88:0/764275770 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4100700b0 con 0x7fa4100735b0 2024-09-16T01:43:04.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.807+0000 7fa414e49640 1 -- 172.21.15.88:0/764275770 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fa404001670 con 0x7fa4100735b0 2024-09-16T01:43:04.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.807+0000 7fa414e49640 1 -- 172.21.15.88:0/764275770 >> v1:172.21.15.88:6789/0 conn(0x7fa4100735b0 legacy=0x7fa41006bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:04.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.807+0000 7fa414e49640 1 --2- 172.21.15.88:0/764275770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3f80039b0 0x7fa3f8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:04.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.807+0000 7fa414e49640 1 --2- 172.21.15.88:0/764275770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3f8004800 0x7fa3f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:04.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.807+0000 7fa414e49640 1 --2- 172.21.15.88:0/764275770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3f80051c0 0x7fa3f800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:04.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.807+0000 7fa414e49640 1 -- 172.21.15.88:0/764275770 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa41006fdd0 con 0x7fa3f80051c0 2024-09-16T01:43:04.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.807+0000 7fa414e49640 1 -- 172.21.15.88:0/764275770 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa3f800c060 con 0x7fa3f8004800 2024-09-16T01:43:04.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.807+0000 7fa414e49640 1 -- 172.21.15.88:0/764275770 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa3f800c260 con 0x7fa3f80039b0 2024-09-16T01:43:04.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.807+0000 7fa41664c640 1 --2- 172.21.15.88:0/764275770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3f80051c0 0x7fa3f800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:04.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.807+0000 7fa41564a640 1 --2- 172.21.15.88:0/764275770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3f8004800 0x7fa3f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:04.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.807+0000 7fa415e4b640 1 --2- 172.21.15.88:0/764275770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3f80039b0 0x7fa3f8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:04.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.807+0000 7fa41664c640 1 -- 172.21.15.88:0/764275770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3f80039b0 msgr2=0x7fa3f8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:04.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.807+0000 7fa41664c640 1 --2- 172.21.15.88:0/764275770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa3f80039b0 0x7fa3f8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:04.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.807+0000 7fa41664c640 1 -- 172.21.15.88:0/764275770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3f8004800 msgr2=0x7fa3f8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:04.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.807+0000 7fa41664c640 1 --2- 172.21.15.88:0/764275770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3f8004800 0x7fa3f8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:04.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.807+0000 7fa41664c640 1 -- 172.21.15.88:0/764275770 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3f8003520 con 0x7fa3f80051c0 2024-09-16T01:43:04.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.808+0000 7fa41664c640 1 -- 172.21.15.88:0/764275770 reap_dead start 2024-09-16T01:43:04.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.808+0000 7fa41664c640 1 --2- 172.21.15.88:0/764275770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3f80051c0 0x7fa3f800b9c0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fa408001750 tx=0x7fa3f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a121772733a818be server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:04.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.808+0000 7fa414e49640 1 -- 172.21.15.88:0/764275770 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa408001370 con 0x7fa3f80051c0 2024-09-16T01:43:04.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.808+0000 7fa414e49640 1 -- 172.21.15.88:0/764275770 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa408005310 con 0x7fa3f80051c0 2024-09-16T01:43:04.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.808+0000 7fa414e49640 1 -- 172.21.15.88:0/764275770 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40802abe0 con 0x7fa3f80051c0 2024-09-16T01:43:04.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.809+0000 7fa4180af640 1 -- 172.21.15.88:0/764275770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3f80051c0 msgr2=0x7fa3f800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:04.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.809+0000 7fa4180af640 1 --2- 172.21.15.88:0/764275770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3f80051c0 0x7fa3f800b9c0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fa408001750 tx=0x7fa3f8005830 comp rx=0 tx=0).stop 2024-09-16T01:43:04.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.809+0000 7fa4180af640 1 -- 172.21.15.88:0/764275770 shutdown_connections 2024-09-16T01:43:04.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.809+0000 7fa4180af640 1 --2- 172.21.15.88:0/764275770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa3f80051c0 0x7fa3f800b9c0 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:04.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.809+0000 7fa4180af640 1 -- 172.21.15.88:0/764275770 >> 172.21.15.88:0/764275770 conn(0x7fa4100d14d0 msgr2=0x7fa4100d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:04.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.809+0000 7fa4180af640 1 -- 172.21.15.88:0/764275770 shutdown_connections 2024-09-16T01:43:04.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.809+0000 7fa4180af640 1 -- 172.21.15.88:0/764275770 wait complete. 2024-09-16T01:43:04.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.810+0000 7fa4180af640 1 Processor -- start 2024-09-16T01:43:04.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.810+0000 7fa4180af640 1 -- start start 2024-09-16T01:43:04.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.811+0000 7fa4180af640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3f80051c0 0x7fa4101542b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:04.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.811+0000 7fa4180af640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4101578b0 0x7fa4100cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:04.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.811+0000 7fa4180af640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa410157460 0x7fa410159c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:04.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.811+0000 7fa4180af640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa41006f8b0 con 0x7fa410157460 2024-09-16T01:43:04.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.811+0000 7fa4180af640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa41006f730 con 0x7fa3f80051c0 2024-09-16T01:43:04.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.811+0000 7fa4180af640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa41006ce30 con 0x7fa4101578b0 2024-09-16T01:43:04.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.811+0000 7fa41664c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa410157460 0x7fa410159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:04.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.811+0000 7fa41664c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa410157460 0x7fa410159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55726/0 (socket says 172.21.15.88:55726) 2024-09-16T01:43:04.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.811+0000 7fa41664c640 1 -- 172.21.15.88:0/1891189830 learned_addr learned my addr 172.21.15.88:0/1891189830 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:04.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.811+0000 7fa415e4b640 1 --2- 172.21.15.88:0/1891189830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3f80051c0 0x7fa4101542b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:04.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.811+0000 7fa41664c640 1 -- 172.21.15.88:0/1891189830 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4101578b0 msgr2=0x7fa4100cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:04.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.811+0000 7fa41564a640 1 --2- 172.21.15.88:0/1891189830 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4101578b0 0x7fa4100cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:04.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.811+0000 7fa41664c640 1 --2- 172.21.15.88:0/1891189830 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4101578b0 0x7fa4100cb0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:04.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.811+0000 7fa41664c640 1 -- 172.21.15.88:0/1891189830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3f80051c0 msgr2=0x7fa4101542b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:04.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.811+0000 7fa41664c640 1 --2- 172.21.15.88:0/1891189830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3f80051c0 0x7fa4101542b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:04.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.811+0000 7fa41664c640 1 -- 172.21.15.88:0/1891189830 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4100cb5e0 con 0x7fa410157460 2024-09-16T01:43:04.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.811+0000 7fa415e4b640 1 --2- 172.21.15.88:0/1891189830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3f80051c0 0x7fa4101542b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:04.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.811+0000 7fa41664c640 1 --2- 172.21.15.88:0/1891189830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa410157460 0x7fa410159c30 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fa40802a590 tx=0x7fa408001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:04.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.811+0000 7fa40e7fc640 1 -- 172.21.15.88:0/1891189830 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa408001370 con 0x7fa410157460 2024-09-16T01:43:04.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.811+0000 7fa40e7fc640 1 -- 172.21.15.88:0/1891189830 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa40802ae20 con 0x7fa410157460 2024-09-16T01:43:04.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.811+0000 7fa4180af640 1 -- 172.21.15.88:0/1891189830 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa410158280 con 0x7fa410157460 2024-09-16T01:43:04.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.812+0000 7fa40e7fc640 1 -- 172.21.15.88:0/1891189830 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa408007a30 con 0x7fa410157460 2024-09-16T01:43:04.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.812+0000 7fa4180af640 1 -- 172.21.15.88:0/1891189830 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa410155070 con 0x7fa410157460 2024-09-16T01:43:04.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.812+0000 7fa4180af640 1 -- 172.21.15.88:0/1891189830 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa410154b80 con 0x7fa410157460 2024-09-16T01:43:04.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.813+0000 7fa40e7fc640 1 -- 172.21.15.88:0/1891189830 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa40803c030 con 0x7fa410157460 2024-09-16T01:43:04.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.813+0000 7fa40e7fc640 1 --2- 172.21.15.88:0/1891189830 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa3fc06c2b0 0x7fa3fc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:04.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.813+0000 7fa40e7fc640 1 -- 172.21.15.88:0/1891189830 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8232+0+0 (secure 0 0 0) 0x7fa4080aaab0 con 0x7fa410157460 2024-09-16T01:43:04.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.813+0000 7fa415e4b640 1 --2- 172.21.15.88:0/1891189830 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa3fc06c2b0 0x7fa3fc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:04.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.814+0000 7fa415e4b640 1 --2- 172.21.15.88:0/1891189830 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa3fc06c2b0 0x7fa3fc06e770 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fa400007440 tx=0x7fa40000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:04.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:04.818+0000 7fa40e7fc640 1 -- 172.21.15.88:0/1891189830 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa4100cb5e0 con 0x7fa410157460 2024-09-16T01:43:05.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.019+0000 7fa4180af640 1 -- 172.21.15.88:0/1891189830 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa41006d1c0 con 0x7fa410157460 2024-09-16T01:43:05.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.019+0000 7fa40e7fc640 1 -- 172.21.15.88:0/1891189830 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) ==== 95+0+3566 (secure 0 0 0) 0x7fa4080781a0 con 0x7fa410157460 2024-09-16T01:43:05.022 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:05.022 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":240,"btime":"2024-09-16T01:43:04:381011+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:43:04.380964+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:43:05.022 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 240 2024-09-16T01:43:05.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.021+0000 7fa4180af640 1 -- 172.21.15.88:0/1891189830 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa3fc06c2b0 msgr2=0x7fa3fc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:05.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.021+0000 7fa4180af640 1 --2- 172.21.15.88:0/1891189830 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa3fc06c2b0 0x7fa3fc06e770 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fa400007440 tx=0x7fa40000b040 comp rx=0 tx=0).stop 2024-09-16T01:43:05.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.021+0000 7fa4180af640 1 -- 172.21.15.88:0/1891189830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa410157460 msgr2=0x7fa410159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:05.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.021+0000 7fa4180af640 1 --2- 172.21.15.88:0/1891189830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa410157460 0x7fa410159c30 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fa40802a590 tx=0x7fa408001120 comp rx=0 tx=0).stop 2024-09-16T01:43:05.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.022+0000 7fa4180af640 1 -- 172.21.15.88:0/1891189830 shutdown_connections 2024-09-16T01:43:05.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.022+0000 7fa4180af640 1 --2- 172.21.15.88:0/1891189830 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa3fc06c2b0 0x7fa3fc06e770 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:05.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.022+0000 7fa4180af640 1 --2- 172.21.15.88:0/1891189830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa410157460 0x7fa410159c30 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:05.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.022+0000 7fa4180af640 1 --2- 172.21.15.88:0/1891189830 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4101578b0 0x7fa4100cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:05.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.022+0000 7fa4180af640 1 --2- 172.21.15.88:0/1891189830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3f80051c0 0x7fa4101542b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:05.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.022+0000 7fa4180af640 1 -- 172.21.15.88:0/1891189830 >> 172.21.15.88:0/1891189830 conn(0x7fa4100d14d0 msgr2=0x7fa410072430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:05.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.022+0000 7fa4180af640 1 -- 172.21.15.88:0/1891189830 shutdown_connections 2024-09-16T01:43:05.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.022+0000 7fa4180af640 1 -- 172.21.15.88:0/1891189830 wait complete. 2024-09-16T01:43:05.035 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:43:05.035 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:43:05.104 INFO:tasks.ceph.mds.a.smithi088.stderr: -42> 2024-09-16T01:43:00.348+0000 7f3832cb5640 -1 log_channel(cluster) log [ERR] : error decoding table object 'mds_snaptable': End of buffer [buffer:2] 2024-09-16T01:43:05.125 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:43:05.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.134+0000 7fdab87fd640 1 Processor -- start 2024-09-16T01:43:05.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.134+0000 7fdab87fd640 1 -- start start 2024-09-16T01:43:05.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.135+0000 7fdab87fd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdab00d3310 0x7fdab00d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:05.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.135+0000 7fdab87fd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdab00d1650 0x7fdab00b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:05.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.135+0000 7fdab87fd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdab015ea40 con 0x7fdab015e660 2024-09-16T01:43:05.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.135+0000 7fdab87fd640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fdab00d1ac0 con 0x7fdab00d1650 2024-09-16T01:43:05.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.135+0000 7fdab87fd640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fdab00d3870 con 0x7fdab00d3310 2024-09-16T01:43:05.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.135+0000 7fdab6599640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fdab015e660 0x7fdab00cf730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57080/0 (socket says 172.21.15.88:57080) 2024-09-16T01:43:05.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.135+0000 7fdab6599640 1 -- 172.21.15.88:0/1292756957 learned_addr learned my addr 172.21.15.88:0/1292756957 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:05.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.135+0000 7fdab6d9a640 1 --2- 172.21.15.88:0/1292756957 >> v2:172.21.15.88:3300/0 conn(0x7fdab00d1650 0x7fdab00b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:05.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.135+0000 7fdab5d98640 1 --2- 172.21.15.88:0/1292756957 >> v2:172.21.15.154:3300/0 conn(0x7fdab00d3310 0x7fdab00d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:05.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.135+0000 7fdab6d9a640 1 -- 172.21.15.88:0/1292756957 >> v2:172.21.15.154:3300/0 conn(0x7fdab00d3310 msgr2=0x7fdab00d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:05.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.135+0000 7fdab6d9a640 1 --2- 172.21.15.88:0/1292756957 >> v2:172.21.15.154:3300/0 conn(0x7fdab00d3310 0x7fdab00d1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:05.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.135+0000 7fdab6d9a640 1 -- 172.21.15.88:0/1292756957 >> v1:172.21.15.88:6789/0 conn(0x7fdab015e660 legacy=0x7fdab00cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:05.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.135+0000 7fdab6d9a640 1 -- 172.21.15.88:0/1292756957 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab00d3b50 con 0x7fdab00d1650 2024-09-16T01:43:05.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.135+0000 7fdab5597640 1 -- 172.21.15.88:0/1292756957 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 829663214 0 0) 0x7fdab015ea40 con 0x7fdab015e660 2024-09-16T01:43:05.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.136+0000 7fdab5d98640 1 --2- 172.21.15.88:0/1292756957 >> v2:172.21.15.154:3300/0 conn(0x7fdab00d3310 0x7fdab00d1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:05.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.136+0000 7fdab6d9a640 1 --2- 172.21.15.88:0/1292756957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdab00d1650 0x7fdab00b7b70 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7fdab00c8020 tx=0x7fdaa802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e65b872f6cadd22 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:05.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.136+0000 7fdab5597640 1 -- 172.21.15.88:0/1292756957 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa8007740 con 0x7fdab00d1650 2024-09-16T01:43:05.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.136+0000 7fdab5597640 1 -- 172.21.15.88:0/1292756957 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaa80078e0 con 0x7fdab00d1650 2024-09-16T01:43:05.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.136+0000 7fdab5597640 1 -- 172.21.15.88:0/1292756957 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa8007cb0 con 0x7fdab00d1650 2024-09-16T01:43:05.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.137+0000 7fdab87fd640 1 -- 172.21.15.88:0/1292756957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdab00d1650 msgr2=0x7fdab00b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:05.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.137+0000 7fdab87fd640 1 --2- 172.21.15.88:0/1292756957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdab00d1650 0x7fdab00b7b70 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7fdab00c8020 tx=0x7fdaa802a680 comp rx=0 tx=0).stop 2024-09-16T01:43:05.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.137+0000 7fdab87fd640 1 -- 172.21.15.88:0/1292756957 shutdown_connections 2024-09-16T01:43:05.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.137+0000 7fdab87fd640 1 --2- 172.21.15.88:0/1292756957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdab00d1650 0x7fdab00b7b70 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:05.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.137+0000 7fdab87fd640 1 --2- 172.21.15.88:0/1292756957 >> v2:172.21.15.154:3300/0 conn(0x7fdab00d3310 0x7fdab00d1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:05.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.137+0000 7fdab87fd640 1 -- 172.21.15.88:0/1292756957 >> 172.21.15.88:0/1292756957 conn(0x7fdab00b6410 msgr2=0x7fdab00b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:05.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.137+0000 7fdab87fd640 1 -- 172.21.15.88:0/1292756957 shutdown_connections 2024-09-16T01:43:05.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.137+0000 7fdab87fd640 1 -- 172.21.15.88:0/1292756957 wait complete. 2024-09-16T01:43:05.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.138+0000 7fdab87fd640 1 Processor -- start 2024-09-16T01:43:05.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.138+0000 7fdab87fd640 1 -- start start 2024-09-16T01:43:05.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.139+0000 7fdab87fd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdab00d1650 0x7fdab01f4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:05.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.139+0000 7fdab87fd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdab00d3310 0x7fdab0155d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:05.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.139+0000 7fdab87fd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdab015e660 0x7fdab0156360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:05.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.139+0000 7fdab87fd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdab00d43f0 con 0x7fdab015e660 2024-09-16T01:43:05.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.139+0000 7fdab87fd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdab00d4110 con 0x7fdab00d1650 2024-09-16T01:43:05.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.139+0000 7fdab87fd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdab00b80b0 con 0x7fdab00d3310 2024-09-16T01:43:05.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.139+0000 7fdab6d9a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdab015e660 0x7fdab0156360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:05.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.139+0000 7fdab6d9a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdab015e660 0x7fdab0156360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55758/0 (socket says 172.21.15.88:55758) 2024-09-16T01:43:05.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.139+0000 7fdab6d9a640 1 -- 172.21.15.88:0/3932788382 learned_addr learned my addr 172.21.15.88:0/3932788382 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:05.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.139+0000 7fdab5d98640 1 --2- 172.21.15.88:0/3932788382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdab00d3310 0x7fdab0155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:05.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.139+0000 7fdab6599640 1 --2- 172.21.15.88:0/3932788382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdab00d1650 0x7fdab01f4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:05.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.139+0000 7fdab6d9a640 1 -- 172.21.15.88:0/3932788382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdab00d3310 msgr2=0x7fdab0155d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:05.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.139+0000 7fdab6d9a640 1 --2- 172.21.15.88:0/3932788382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdab00d3310 0x7fdab0155d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:05.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.139+0000 7fdab6d9a640 1 -- 172.21.15.88:0/3932788382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdab00d1650 msgr2=0x7fdab01f4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:05.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.139+0000 7fdab6d9a640 1 --2- 172.21.15.88:0/3932788382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdab00d1650 0x7fdab01f4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:05.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.139+0000 7fdab6d9a640 1 -- 172.21.15.88:0/3932788382 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab01592b0 con 0x7fdab015e660 2024-09-16T01:43:05.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.139+0000 7fdab5d98640 1 --2- 172.21.15.88:0/3932788382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdab00d3310 0x7fdab0155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:05.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.139+0000 7fdab6599640 1 --2- 172.21.15.88:0/3932788382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdab00d1650 0x7fdab01f4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:05.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.139+0000 7fdab6d9a640 1 --2- 172.21.15.88:0/3932788382 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdab015e660 0x7fdab0156360 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8009e80 tx=0x7fdaa8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:05.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.139+0000 7fdaa6ffd640 1 -- 172.21.15.88:0/3932788382 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa8040070 con 0x7fdab015e660 2024-09-16T01:43:05.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.140+0000 7fdaa6ffd640 1 -- 172.21.15.88:0/3932788382 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaa8031c40 con 0x7fdab015e660 2024-09-16T01:43:05.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.140+0000 7fdab87fd640 1 -- 172.21.15.88:0/3932788382 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdab0159590 con 0x7fdab015e660 2024-09-16T01:43:05.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.140+0000 7fdaa6ffd640 1 -- 172.21.15.88:0/3932788382 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa8003b90 con 0x7fdab015e660 2024-09-16T01:43:05.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.140+0000 7fdab87fd640 1 -- 172.21.15.88:0/3932788382 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdab01f4070 con 0x7fdab015e660 2024-09-16T01:43:05.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.140+0000 7fdab87fd640 1 -- 172.21.15.88:0/3932788382 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdab00b7b70 con 0x7fdab015e660 2024-09-16T01:43:05.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.141+0000 7fdaa6ffd640 1 -- 172.21.15.88:0/3932788382 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fdaa8039040 con 0x7fdab015e660 2024-09-16T01:43:05.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.141+0000 7fdaa6ffd640 1 --2- 172.21.15.88:0/3932788382 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fda9406c2b0 0x7fda9406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:05.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.142+0000 7fdaa6ffd640 1 -- 172.21.15.88:0/3932788382 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8232+0+0 (secure 0 0 0) 0x7fdaa80ad120 con 0x7fdab015e660 2024-09-16T01:43:05.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.142+0000 7fdab6599640 1 --2- 172.21.15.88:0/3932788382 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fda9406c2b0 0x7fda9406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:05.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.142+0000 7fdab6599640 1 --2- 172.21.15.88:0/3932788382 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fda9406c2b0 0x7fda9406e770 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fdaac000f60 tx=0x7fdaac009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:05.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.146+0000 7fdaa6ffd640 1 -- 172.21.15.88:0/3932788382 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fdaa807aef0 con 0x7fdab015e660 2024-09-16T01:43:05.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.344+0000 7fdab87fd640 1 -- 172.21.15.88:0/3932788382 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdab00c7e30 con 0x7fdab015e660 2024-09-16T01:43:05.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.345+0000 7fdaa6ffd640 1 -- 172.21.15.88:0/3932788382 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) ==== 95+0+3566 (secure 0 0 0) 0x7fdaa807a7f0 con 0x7fdab015e660 2024-09-16T01:43:05.348 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:05.348 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":240,"btime":"2024-09-16T01:43:04:381011+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:43:04.380964+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":116,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:43:05.348 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 240 2024-09-16T01:43:05.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.347+0000 7fdab87fd640 1 -- 172.21.15.88:0/3932788382 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fda9406c2b0 msgr2=0x7fda9406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:05.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.347+0000 7fdab87fd640 1 --2- 172.21.15.88:0/3932788382 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fda9406c2b0 0x7fda9406e770 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fdaac000f60 tx=0x7fdaac009040 comp rx=0 tx=0).stop 2024-09-16T01:43:05.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.347+0000 7fdab87fd640 1 -- 172.21.15.88:0/3932788382 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdab015e660 msgr2=0x7fdab0156360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:05.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.347+0000 7fdab87fd640 1 --2- 172.21.15.88:0/3932788382 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdab015e660 0x7fdab0156360 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8009e80 tx=0x7fdaa8004a40 comp rx=0 tx=0).stop 2024-09-16T01:43:05.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.347+0000 7fdab87fd640 1 -- 172.21.15.88:0/3932788382 shutdown_connections 2024-09-16T01:43:05.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.347+0000 7fdab87fd640 1 --2- 172.21.15.88:0/3932788382 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fda9406c2b0 0x7fda9406e770 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:05.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.347+0000 7fdab87fd640 1 --2- 172.21.15.88:0/3932788382 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdab015e660 0x7fdab0156360 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:05.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.347+0000 7fdab87fd640 1 --2- 172.21.15.88:0/3932788382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdab00d3310 0x7fdab0155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:05.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.347+0000 7fdab87fd640 1 --2- 172.21.15.88:0/3932788382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdab00d1650 0x7fdab01f4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:05.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.347+0000 7fdab87fd640 1 -- 172.21.15.88:0/3932788382 >> 172.21.15.88:0/3932788382 conn(0x7fdab00b6410 msgr2=0x7fdab015b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:05.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.347+0000 7fdab87fd640 1 -- 172.21.15.88:0/3932788382 shutdown_connections 2024-09-16T01:43:05.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.347+0000 7fdab87fd640 1 -- 172.21.15.88:0/3932788382 wait complete. 2024-09-16T01:43:05.360 INFO:tasks.cephfs.test_damage:Result: Mutation 'Truncate mds0_inotable' led to DAMAGED state 2024-09-16T01:43:05.360 INFO:tasks.cephfs.test_damage:Applying mutation 'Truncate 400.00000000' 2024-09-16T01:43:05.360 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:43:05.360 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:43:05.376 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:43:05.376 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:43:05.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:05.377 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:43:05.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:43:05.432 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:43:05.432 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:43:05.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.569+0000 7f6fca7f8640 1 Processor -- start 2024-09-16T01:43:05.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.569+0000 7f6fca7f8640 1 -- start start 2024-09-16T01:43:05.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.570+0000 7f6fca7f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fc40b4e60 0x7f6fc40d0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:05.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.570+0000 7f6fca7f8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6fc415cb20 0x7f6fc415cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:05.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.570+0000 7f6fca7f8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fc40cd9b0 con 0x7f6fc40b5c00 2024-09-16T01:43:05.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.570+0000 7f6fca7f8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6fc40ce5c0 con 0x7f6fc415cb20 2024-09-16T01:43:05.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.570+0000 7f6fca7f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6fc40cdc90 con 0x7f6fc40b4e60 2024-09-16T01:43:05.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.570+0000 7f6fc8d95640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6fc415cb20 0x7f6fc415cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:05.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.570+0000 7f6fc37fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6fc40b5c00 0x7f6fc40b5240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57092/0 (socket says 172.21.15.88:57092) 2024-09-16T01:43:05.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.570+0000 7f6fc8d95640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6fc415cb20 0x7f6fc415cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55762/0 (socket says 172.21.15.88:55762) 2024-09-16T01:43:05.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.570+0000 7f6fc37fe640 1 -- 172.21.15.88:0/744184478 learned_addr learned my addr 172.21.15.88:0/744184478 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:05.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.570+0000 7f6fc8d95640 1 -- 172.21.15.88:0/744184478 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fc40b4e60 msgr2=0x7f6fc40d0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:05.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.570+0000 7f6fc3fff640 1 --2- 172.21.15.88:0/744184478 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fc40b4e60 0x7f6fc40d0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:05.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.570+0000 7f6fc8d95640 1 --2- 172.21.15.88:0/744184478 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fc40b4e60 0x7f6fc40d0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:05.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.570+0000 7f6fc2ffd640 1 -- 172.21.15.88:0/744184478 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4156070972 0 0) 0x7f6fc40cd9b0 con 0x7f6fc40b5c00 2024-09-16T01:43:05.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.570+0000 7f6fc8d95640 1 -- 172.21.15.88:0/744184478 >> v1:172.21.15.88:6789/0 conn(0x7f6fc40b5c00 legacy=0x7f6fc40b5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:05.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.570+0000 7f6fc8d95640 1 -- 172.21.15.88:0/744184478 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fc40cdf70 con 0x7f6fc415cb20 2024-09-16T01:43:05.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.571+0000 7f6fc8d95640 1 --2- 172.21.15.88:0/744184478 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fc415cb20 0x7f6fc415cfb0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc009d90 tx=0x7f6fbc02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc286d01b37aa8d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:05.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.571+0000 7f6fc2ffd640 1 -- 172.21.15.88:0/744184478 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc007740 con 0x7f6fc415cb20 2024-09-16T01:43:05.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.571+0000 7f6fc2ffd640 1 -- 172.21.15.88:0/744184478 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fbc0078e0 con 0x7f6fc415cb20 2024-09-16T01:43:05.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.571+0000 7f6fc2ffd640 1 -- 172.21.15.88:0/744184478 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc007cb0 con 0x7f6fc415cb20 2024-09-16T01:43:05.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.571+0000 7f6fca7f8640 1 -- 172.21.15.88:0/744184478 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fc415cb20 msgr2=0x7f6fc415cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:05.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.571+0000 7f6fca7f8640 1 --2- 172.21.15.88:0/744184478 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fc415cb20 0x7f6fc415cfb0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc009d90 tx=0x7f6fbc02a660 comp rx=0 tx=0).stop 2024-09-16T01:43:05.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.572+0000 7f6fca7f8640 1 -- 172.21.15.88:0/744184478 shutdown_connections 2024-09-16T01:43:05.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.572+0000 7f6fca7f8640 1 --2- 172.21.15.88:0/744184478 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fc415cb20 0x7f6fc415cfb0 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:05.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.572+0000 7f6fca7f8640 1 --2- 172.21.15.88:0/744184478 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fc40b4e60 0x7f6fc40d0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:05.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.572+0000 7f6fca7f8640 1 -- 172.21.15.88:0/744184478 >> 172.21.15.88:0/744184478 conn(0x7f6fc406cb50 msgr2=0x7f6fc406cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:05.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.572+0000 7f6fca7f8640 1 -- 172.21.15.88:0/744184478 shutdown_connections 2024-09-16T01:43:05.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.572+0000 7f6fca7f8640 1 -- 172.21.15.88:0/744184478 wait complete. 2024-09-16T01:43:05.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.573+0000 7f6fca7f8640 1 Processor -- start 2024-09-16T01:43:05.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.573+0000 7f6fca7f8640 1 -- start start 2024-09-16T01:43:05.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.574+0000 7f6fca7f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fc40b4e60 0x7f6fc41f1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:05.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.574+0000 7f6fca7f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fc40b5c00 0x7f6fc41517f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:05.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.574+0000 7f6fca7f8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fc415cb20 0x7f6fc41435b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:05.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.574+0000 7f6fca7f8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc415fd80 con 0x7f6fc415cb20 2024-09-16T01:43:05.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.574+0000 7f6fca7f8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc415fc00 con 0x7f6fc40b5c00 2024-09-16T01:43:05.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.574+0000 7f6fca7f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6fc40b6060 con 0x7f6fc40b4e60 2024-09-16T01:43:05.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.574+0000 7f6fc8d95640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fc415cb20 0x7f6fc41435b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:05.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.574+0000 7f6fc8d95640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fc415cb20 0x7f6fc41435b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55772/0 (socket says 172.21.15.88:55772) 2024-09-16T01:43:05.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.574+0000 7f6fc8d95640 1 -- 172.21.15.88:0/1488760447 learned_addr learned my addr 172.21.15.88:0/1488760447 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:05.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.574+0000 7f6fc3fff640 1 --2- 172.21.15.88:0/1488760447 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fc40b4e60 0x7f6fc41f1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:05.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.574+0000 7f6fc37fe640 1 --2- 172.21.15.88:0/1488760447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fc40b5c00 0x7f6fc41517f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:05.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.574+0000 7f6fc8d95640 1 -- 172.21.15.88:0/1488760447 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fc40b4e60 msgr2=0x7f6fc41f1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:05.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.574+0000 7f6fc8d95640 1 --2- 172.21.15.88:0/1488760447 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fc40b4e60 0x7f6fc41f1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:05.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.574+0000 7f6fc8d95640 1 -- 172.21.15.88:0/1488760447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fc40b5c00 msgr2=0x7f6fc41517f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:05.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.574+0000 7f6fc8d95640 1 --2- 172.21.15.88:0/1488760447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fc40b5c00 0x7f6fc41517f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:05.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.574+0000 7f6fc8d95640 1 -- 172.21.15.88:0/1488760447 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fc41508f0 con 0x7f6fc415cb20 2024-09-16T01:43:05.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.574+0000 7f6fc37fe640 1 --2- 172.21.15.88:0/1488760447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fc40b5c00 0x7f6fc41517f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:05.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.574+0000 7f6fc8d95640 1 --2- 172.21.15.88:0/1488760447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fc415cb20 0x7f6fc41435b0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc009eb0 tx=0x7f6fbc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:05.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.574+0000 7f6fc0ff9640 1 -- 172.21.15.88:0/1488760447 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc040070 con 0x7f6fc415cb20 2024-09-16T01:43:05.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.574+0000 7f6fc0ff9640 1 -- 172.21.15.88:0/1488760447 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fbc031c90 con 0x7f6fc415cb20 2024-09-16T01:43:05.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.574+0000 7f6fca7f8640 1 -- 172.21.15.88:0/1488760447 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fc4143af0 con 0x7f6fc415cb20 2024-09-16T01:43:05.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.574+0000 7f6fc0ff9640 1 -- 172.21.15.88:0/1488760447 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc045680 con 0x7f6fc415cb20 2024-09-16T01:43:05.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.574+0000 7f6fca7f8640 1 -- 172.21.15.88:0/1488760447 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fc4151f20 con 0x7f6fc415cb20 2024-09-16T01:43:05.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.576+0000 7f6fca7f8640 1 -- 172.21.15.88:0/1488760447 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fc40d0760 con 0x7f6fc415cb20 2024-09-16T01:43:05.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.576+0000 7f6fc0ff9640 1 -- 172.21.15.88:0/1488760447 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6fbc039040 con 0x7f6fc415cb20 2024-09-16T01:43:05.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.576+0000 7f6fc0ff9640 1 --2- 172.21.15.88:0/1488760447 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6fb006c2b0 0x7f6fb006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:05.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.576+0000 7f6fc0ff9640 1 -- 172.21.15.88:0/1488760447 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8232+0+0 (secure 0 0 0) 0x7f6fbc0ad7e0 con 0x7f6fc415cb20 2024-09-16T01:43:05.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.576+0000 7f6fc3fff640 1 --2- 172.21.15.88:0/1488760447 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6fb006c2b0 0x7f6fb006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:05.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.577+0000 7f6fc3fff640 1 --2- 172.21.15.88:0/1488760447 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6fb006c2b0 0x7f6fb006e770 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4006fd0 tx=0x7f6fb4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:05.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.580+0000 7f6fc0ff9640 1 -- 172.21.15.88:0/1488760447 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6fc4143af0 con 0x7f6fc415cb20 2024-09-16T01:43:05.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:05.781+0000 7f6fca7f8640 1 -- 172.21.15.88:0/1488760447 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f6fc415f5c0 con 0x7f6fc415cb20 2024-09-16T01:43:06.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.394+0000 7f6fc0ff9640 1 -- 172.21.15.88:0/1488760447 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v242) ==== 161+0+0 (secure 0 0 0) 0x7f6fc41508f0 con 0x7f6fc415cb20 2024-09-16T01:43:06.396 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:43:06.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.397+0000 7f6fca7f8640 1 -- 172.21.15.88:0/1488760447 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6fb006c2b0 msgr2=0x7f6fb006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:06.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.397+0000 7f6fca7f8640 1 --2- 172.21.15.88:0/1488760447 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6fb006c2b0 0x7f6fb006e770 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4006fd0 tx=0x7f6fb4008040 comp rx=0 tx=0).stop 2024-09-16T01:43:06.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.397+0000 7f6fca7f8640 1 -- 172.21.15.88:0/1488760447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fc415cb20 msgr2=0x7f6fc41435b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:06.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.397+0000 7f6fca7f8640 1 --2- 172.21.15.88:0/1488760447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fc415cb20 0x7f6fc41435b0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc009eb0 tx=0x7f6fbc004a40 comp rx=0 tx=0).stop 2024-09-16T01:43:06.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.397+0000 7f6fca7f8640 1 -- 172.21.15.88:0/1488760447 shutdown_connections 2024-09-16T01:43:06.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.397+0000 7f6fca7f8640 1 --2- 172.21.15.88:0/1488760447 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6fb006c2b0 0x7f6fb006e770 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:06.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.397+0000 7f6fca7f8640 1 --2- 172.21.15.88:0/1488760447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fc415cb20 0x7f6fc41435b0 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:06.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.397+0000 7f6fca7f8640 1 --2- 172.21.15.88:0/1488760447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fc40b5c00 0x7f6fc41517f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:06.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.397+0000 7f6fca7f8640 1 --2- 172.21.15.88:0/1488760447 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fc40b4e60 0x7f6fc41f1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:06.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.397+0000 7f6fca7f8640 1 -- 172.21.15.88:0/1488760447 >> 172.21.15.88:0/1488760447 conn(0x7f6fc406cb50 msgr2=0x7f6fc406cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:06.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.397+0000 7f6fca7f8640 1 -- 172.21.15.88:0/1488760447 shutdown_connections 2024-09-16T01:43:06.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.397+0000 7f6fca7f8640 1 -- 172.21.15.88:0/1488760447 wait complete. 2024-09-16T01:43:06.411 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:43:06.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.513+0000 7fd9ff559640 1 Processor -- start 2024-09-16T01:43:06.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.513+0000 7fd9ff559640 1 -- start start 2024-09-16T01:43:06.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.514+0000 7fd9ff559640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd9f815c4c0 0x7fd9f80b55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:06.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.514+0000 7fd9ff559640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9f80b36a0 0x7fd9f80b3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:06.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.514+0000 7fd9ff559640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9f8072f40 con 0x7fd9f8159000 2024-09-16T01:43:06.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.514+0000 7fd9ff559640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd9f80b5130 con 0x7fd9f815c4c0 2024-09-16T01:43:06.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.514+0000 7fd9ff559640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd9f8072980 con 0x7fd9f80b36a0 2024-09-16T01:43:06.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.514+0000 7fd9fcaf4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd9f815c4c0 0x7fd9f80b55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:06.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.514+0000 7fd9fcaf4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd9f815c4c0 0x7fd9f80b55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55778/0 (socket says 172.21.15.88:55778) 2024-09-16T01:43:06.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.514+0000 7fd9fcaf4640 1 -- 172.21.15.88:0/2640787974 learned_addr learned my addr 172.21.15.88:0/2640787974 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:06.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.514+0000 7fd9fdaf6640 1 --2- 172.21.15.88:0/2640787974 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9f80b36a0 0x7fd9f80b3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:06.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.514+0000 7fd9fcaf4640 1 -- 172.21.15.88:0/2640787974 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9f80b36a0 msgr2=0x7fd9f80b3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:06.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.514+0000 7fd9fcaf4640 1 --2- 172.21.15.88:0/2640787974 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9f80b36a0 0x7fd9f80b3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:06.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.514+0000 7fd9fcaf4640 1 -- 172.21.15.88:0/2640787974 >> v1:172.21.15.154:6789/0 conn(0x7fd9f8159000 legacy=0x7fd9f8072190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:06.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.514+0000 7fd9fcaf4640 1 -- 172.21.15.88:0/2640787974 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f8072c60 con 0x7fd9f815c4c0 2024-09-16T01:43:06.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.515+0000 7fd9fcaf4640 1 --2- 172.21.15.88:0/2640787974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9f815c4c0 0x7fd9f80b55c0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8009870 tx=0x7fd9e802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=935067bcba55d4b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:06.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.515+0000 7fd9effff640 1 -- 172.21.15.88:0/2640787974 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e8037070 con 0x7fd9f815c4c0 2024-09-16T01:43:06.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.515+0000 7fd9effff640 1 -- 172.21.15.88:0/2640787974 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9e8032440 con 0x7fd9f815c4c0 2024-09-16T01:43:06.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.515+0000 7fd9effff640 1 -- 172.21.15.88:0/2640787974 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e8032810 con 0x7fd9f815c4c0 2024-09-16T01:43:06.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.516+0000 7fd9ff559640 1 -- 172.21.15.88:0/2640787974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9f815c4c0 msgr2=0x7fd9f80b55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:06.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.516+0000 7fd9ff559640 1 --2- 172.21.15.88:0/2640787974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9f815c4c0 0x7fd9f80b55c0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8009870 tx=0x7fd9e802a270 comp rx=0 tx=0).stop 2024-09-16T01:43:06.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.516+0000 7fd9ff559640 1 -- 172.21.15.88:0/2640787974 shutdown_connections 2024-09-16T01:43:06.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.516+0000 7fd9ff559640 1 --2- 172.21.15.88:0/2640787974 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9f80b36a0 0x7fd9f80b3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:06.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.516+0000 7fd9ff559640 1 --2- 172.21.15.88:0/2640787974 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9f815c4c0 0x7fd9f80b55c0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:06.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.516+0000 7fd9ff559640 1 -- 172.21.15.88:0/2640787974 >> 172.21.15.88:0/2640787974 conn(0x7fd9f80b6b50 msgr2=0x7fd9f80b6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:06.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.516+0000 7fd9ff559640 1 -- 172.21.15.88:0/2640787974 shutdown_connections 2024-09-16T01:43:06.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.516+0000 7fd9ff559640 1 -- 172.21.15.88:0/2640787974 wait complete. 2024-09-16T01:43:06.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.517+0000 7fd9ff559640 1 Processor -- start 2024-09-16T01:43:06.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.517+0000 7fd9ff559640 1 -- start start 2024-09-16T01:43:06.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.518+0000 7fd9ff559640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9f80b36a0 0x7fd9f81f1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:06.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.518+0000 7fd9ff559640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9f8159000 0x7fd9f814adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:06.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.518+0000 7fd9ff559640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9f815c4c0 0x7fd9f806bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:06.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.518+0000 7fd9ff559640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f815ca40 con 0x7fd9f80b36a0 2024-09-16T01:43:06.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.518+0000 7fd9ff559640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f80b5b00 con 0x7fd9f8159000 2024-09-16T01:43:06.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.518+0000 7fd9ff559640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd9f815cbc0 con 0x7fd9f815c4c0 2024-09-16T01:43:06.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.518+0000 7fd9fd2f5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9f80b36a0 0x7fd9f81f1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:06.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.518+0000 7fd9fd2f5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9f80b36a0 0x7fd9f81f1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55790/0 (socket says 172.21.15.88:55790) 2024-09-16T01:43:06.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.518+0000 7fd9fd2f5640 1 -- 172.21.15.88:0/3503362917 learned_addr learned my addr 172.21.15.88:0/3503362917 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:06.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.518+0000 7fd9fcaf4640 1 --2- 172.21.15.88:0/3503362917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9f8159000 0x7fd9f814adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:06.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.518+0000 7fd9fdaf6640 1 --2- 172.21.15.88:0/3503362917 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9f815c4c0 0x7fd9f806bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:06.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.518+0000 7fd9fd2f5640 1 -- 172.21.15.88:0/3503362917 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9f815c4c0 msgr2=0x7fd9f806bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:06.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.518+0000 7fd9fd2f5640 1 --2- 172.21.15.88:0/3503362917 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9f815c4c0 0x7fd9f806bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:06.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.518+0000 7fd9fd2f5640 1 -- 172.21.15.88:0/3503362917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9f8159000 msgr2=0x7fd9f814adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:06.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.518+0000 7fd9fd2f5640 1 --2- 172.21.15.88:0/3503362917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9f8159000 0x7fd9f814adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:06.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.518+0000 7fd9fd2f5640 1 -- 172.21.15.88:0/3503362917 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f80b22b0 con 0x7fd9f80b36a0 2024-09-16T01:43:06.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.518+0000 7fd9fcaf4640 1 --2- 172.21.15.88:0/3503362917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9f8159000 0x7fd9f814adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:06.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.518+0000 7fd9fd2f5640 1 --2- 172.21.15.88:0/3503362917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9f80b36a0 0x7fd9f81f1c60 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fd9f400c400 tx=0x7fd9f400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:06.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.518+0000 7fd9edffb640 1 -- 172.21.15.88:0/3503362917 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f40052d0 con 0x7fd9f80b36a0 2024-09-16T01:43:06.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.518+0000 7fd9edffb640 1 -- 172.21.15.88:0/3503362917 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9f4005490 con 0x7fd9f80b36a0 2024-09-16T01:43:06.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.518+0000 7fd9ff559640 1 -- 172.21.15.88:0/3503362917 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9f80b1fd0 con 0x7fd9f80b36a0 2024-09-16T01:43:06.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.518+0000 7fd9edffb640 1 -- 172.21.15.88:0/3503362917 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f4007260 con 0x7fd9f80b36a0 2024-09-16T01:43:06.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.519+0000 7fd9ff559640 1 -- 172.21.15.88:0/3503362917 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9f80acaa0 con 0x7fd9f80b36a0 2024-09-16T01:43:06.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.519+0000 7fd9ff559640 1 -- 172.21.15.88:0/3503362917 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9f80b55c0 con 0x7fd9f80b36a0 2024-09-16T01:43:06.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.520+0000 7fd9edffb640 1 -- 172.21.15.88:0/3503362917 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd9f4007400 con 0x7fd9f80b36a0 2024-09-16T01:43:06.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.520+0000 7fd9edffb640 1 --2- 172.21.15.88:0/3503362917 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd9dc06c2b0 0x7fd9dc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:06.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.521+0000 7fd9fcaf4640 1 --2- 172.21.15.88:0/3503362917 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd9dc06c2b0 0x7fd9dc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:06.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.521+0000 7fd9edffb640 1 -- 172.21.15.88:0/3503362917 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8232+0+0 (secure 0 0 0) 0x7fd9f408b310 con 0x7fd9f80b36a0 2024-09-16T01:43:06.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.521+0000 7fd9fcaf4640 1 --2- 172.21.15.88:0/3503362917 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd9dc06c2b0 0x7fd9dc06e770 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8034000 tx=0x7fd9e802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:06.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.525+0000 7fd9edffb640 1 -- 172.21.15.88:0/3503362917 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd9f80b22b0 con 0x7fd9f80b36a0 2024-09-16T01:43:06.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:06.760+0000 7fd9ff559640 1 -- 172.21.15.88:0/3503362917 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fd9f8063280 con 0x7fd9f80b36a0 2024-09-16T01:43:07.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.401+0000 7fd9edffb640 1 -- 172.21.15.88:0/3503362917 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v243) ==== 101+0+0 (secure 0 0 0) 0x7fd9f40590e0 con 0x7fd9f80b36a0 2024-09-16T01:43:07.402 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:43:07.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.403+0000 7fd9ff559640 1 -- 172.21.15.88:0/3503362917 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd9dc06c2b0 msgr2=0x7fd9dc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:07.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.403+0000 7fd9ff559640 1 --2- 172.21.15.88:0/3503362917 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd9dc06c2b0 0x7fd9dc06e770 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8034000 tx=0x7fd9e802e000 comp rx=0 tx=0).stop 2024-09-16T01:43:07.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.403+0000 7fd9ff559640 1 -- 172.21.15.88:0/3503362917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9f80b36a0 msgr2=0x7fd9f81f1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:07.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.403+0000 7fd9ff559640 1 --2- 172.21.15.88:0/3503362917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9f80b36a0 0x7fd9f81f1c60 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fd9f400c400 tx=0x7fd9f400c8c0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.403+0000 7fd9ff559640 1 -- 172.21.15.88:0/3503362917 shutdown_connections 2024-09-16T01:43:07.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.403+0000 7fd9ff559640 1 --2- 172.21.15.88:0/3503362917 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd9dc06c2b0 0x7fd9dc06e770 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.403+0000 7fd9ff559640 1 --2- 172.21.15.88:0/3503362917 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9f815c4c0 0x7fd9f806bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.403+0000 7fd9ff559640 1 --2- 172.21.15.88:0/3503362917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9f8159000 0x7fd9f814adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.403+0000 7fd9ff559640 1 --2- 172.21.15.88:0/3503362917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd9f80b36a0 0x7fd9f81f1c60 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.403+0000 7fd9ff559640 1 -- 172.21.15.88:0/3503362917 >> 172.21.15.88:0/3503362917 conn(0x7fd9f80b6b50 msgr2=0x7fd9f815aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:07.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.404+0000 7fd9ff559640 1 -- 172.21.15.88:0/3503362917 shutdown_connections 2024-09-16T01:43:07.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.404+0000 7fd9ff559640 1 -- 172.21.15.88:0/3503362917 wait complete. 2024-09-16T01:43:07.417 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:43:07.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.461+0000 7f1ee16fef40 1 Processor -- start 2024-09-16T01:43:07.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.461+0000 7f1ee16fef40 1 -- start start 2024-09-16T01:43:07.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.461+0000 7f1ee16fef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ecc0022d0 0x5653a2339f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.461+0000 7f1ee16fef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653a23e3cf0 0x5653a2310660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.461+0000 7f1ee16fef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5653a23d7ec0 0x5653a23d8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.461+0000 7f1ee16fef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5653a2261510 con 0x7f1ecc0022d0 2024-09-16T01:43:07.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.461+0000 7f1ee16fef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5653a22bac50 con 0x5653a23e3cf0 2024-09-16T01:43:07.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.461+0000 7f1ee16fef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5653a23e4150 con 0x5653a23d7ec0 2024-09-16T01:43:07.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.461+0000 7f1edf1f4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ecc0022d0 0x5653a2339f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.461+0000 7f1edf1f4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ecc0022d0 0x5653a2339f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55802/0 (socket says 172.21.15.88:55802) 2024-09-16T01:43:07.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.461+0000 7f1edf1f4640 1 -- 172.21.15.88:0/1881964797 learned_addr learned my addr 172.21.15.88:0/1881964797 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:07.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.462+0000 7f1ed7fff640 1 --2- 172.21.15.88:0/1881964797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653a23e3cf0 0x5653a2310660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.462+0000 7f1edf9f5640 1 --2- 172.21.15.88:0/1881964797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5653a23d7ec0 0x5653a23d8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.462+0000 7f1edf1f4640 1 -- 172.21.15.88:0/1881964797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5653a23d7ec0 msgr2=0x5653a23d8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:07.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.462+0000 7f1edf1f4640 1 --2- 172.21.15.88:0/1881964797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5653a23d7ec0 0x5653a23d8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.462+0000 7f1edf1f4640 1 -- 172.21.15.88:0/1881964797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653a23e3cf0 msgr2=0x5653a2310660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:07.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.462+0000 7f1edf1f4640 1 --2- 172.21.15.88:0/1881964797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653a23e3cf0 0x5653a2310660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.462+0000 7f1edf1f4640 1 -- 172.21.15.88:0/1881964797 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653a23e1840 con 0x7f1ecc0022d0 2024-09-16T01:43:07.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.462+0000 7f1edf1f4640 1 --2- 172.21.15.88:0/1881964797 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ecc0022d0 0x5653a2339f00 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009b00 tx=0x7f1ed0050000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60c713dcd8b73ec0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:07.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.462+0000 7f1ede1f2640 1 -- 172.21.15.88:0/1881964797 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0001630 con 0x7f1ecc0022d0 2024-09-16T01:43:07.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.462+0000 7f1ede1f2640 1 -- 172.21.15.88:0/1881964797 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ed0066070 con 0x7f1ecc0022d0 2024-09-16T01:43:07.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.462+0000 7f1ede1f2640 1 -- 172.21.15.88:0/1881964797 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0061650 con 0x7f1ecc0022d0 2024-09-16T01:43:07.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.462+0000 7f1ee16fef40 1 -- 172.21.15.88:0/1881964797 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ecc0022d0 msgr2=0x5653a2339f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:07.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.462+0000 7f1ee16fef40 1 --2- 172.21.15.88:0/1881964797 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ecc0022d0 0x5653a2339f00 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009b00 tx=0x7f1ed0050000 comp rx=0 tx=0).stop 2024-09-16T01:43:07.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.462+0000 7f1ee16fef40 1 -- 172.21.15.88:0/1881964797 shutdown_connections 2024-09-16T01:43:07.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.462+0000 7f1ee16fef40 1 --2- 172.21.15.88:0/1881964797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5653a23d7ec0 0x5653a23d8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.462+0000 7f1ee16fef40 1 --2- 172.21.15.88:0/1881964797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653a23e3cf0 0x5653a2310660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.462+0000 7f1ee16fef40 1 --2- 172.21.15.88:0/1881964797 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ecc0022d0 0x5653a2339f00 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.462+0000 7f1ee16fef40 1 -- 172.21.15.88:0/1881964797 >> 172.21.15.88:0/1881964797 conn(0x5653a233a680 msgr2=0x5653a233f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:07.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.462+0000 7f1ee16fef40 1 -- 172.21.15.88:0/1881964797 shutdown_connections 2024-09-16T01:43:07.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.462+0000 7f1ee16fef40 1 -- 172.21.15.88:0/1881964797 wait complete. 2024-09-16T01:43:07.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.463+0000 7f1ee16fef40 1 Processor -- start 2024-09-16T01:43:07.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.463+0000 7f1ee16fef40 1 -- start start 2024-09-16T01:43:07.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.464+0000 7f1ee16fef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5653a23d7ec0 0x5653a23d6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.464+0000 7f1ee16fef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653a23e3cf0 0x5653a23dc780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.464+0000 7f1ee16fef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ecc0022d0 0x5653a24730c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.464+0000 7f1ee16fef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5653a23e3780 con 0x7f1ecc0022d0 2024-09-16T01:43:07.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.464+0000 7f1ee16fef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5653a23e4380 con 0x5653a23e3cf0 2024-09-16T01:43:07.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.464+0000 7f1ee16fef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5653a23e1ae0 con 0x5653a23d7ec0 2024-09-16T01:43:07.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.464+0000 7f1edf9f5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ecc0022d0 0x5653a24730c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.464+0000 7f1edf9f5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ecc0022d0 0x5653a24730c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55806/0 (socket says 172.21.15.88:55806) 2024-09-16T01:43:07.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.464+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 learned_addr learned my addr 172.21.15.88:0/3141372312 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:07.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.464+0000 7f1ed7fff640 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653a23e3cf0 0x5653a23dc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.464+0000 7f1edf1f4640 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5653a23d7ec0 0x5653a23d6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.464+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5653a23d7ec0 msgr2=0x5653a23d6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:07.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.464+0000 7f1edf9f5640 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5653a23d7ec0 0x5653a23d6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.464+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653a23e3cf0 msgr2=0x5653a23dc780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:07.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.464+0000 7f1edf9f5640 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5653a23e3cf0 0x5653a23dc780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.464+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653a23e8010 con 0x7f1ecc0022d0 2024-09-16T01:43:07.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.464+0000 7f1edf9f5640 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ecc0022d0 0x5653a24730c0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x5653a233c4d0 tx=0x7f1ed800e5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:07.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.465+0000 7f1edd1f0640 1 -- 172.21.15.88:0/3141372312 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed800ee80 con 0x7f1ecc0022d0 2024-09-16T01:43:07.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.465+0000 7f1edd1f0640 1 -- 172.21.15.88:0/3141372312 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ed8017070 con 0x7f1ecc0022d0 2024-09-16T01:43:07.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.465+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5653a23db820 con 0x7f1ecc0022d0 2024-09-16T01:43:07.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.465+0000 7f1edd1f0640 1 -- 172.21.15.88:0/3141372312 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed8012630 con 0x7f1ecc0022d0 2024-09-16T01:43:07.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.465+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653a23dbde0 con 0x7f1ecc0022d0 2024-09-16T01:43:07.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.466+0000 7f1edd1f0640 1 -- 172.21.15.88:0/3141372312 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1ed80127d0 con 0x7f1ecc0022d0 2024-09-16T01:43:07.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.467+0000 7f1edd1f0640 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ec006ac90 0x7f1ec006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.467+0000 7f1edd1f0640 1 -- 172.21.15.88:0/3141372312 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8232+0+0 (secure 0 0 0) 0x7f1ed808a3a0 con 0x7f1ecc0022d0 2024-09-16T01:43:07.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.467+0000 7f1edf1f4640 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ec006ac90 0x7f1ec006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.472 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:43:07.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.467+0000 7f1ee16fef40 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5653a23d9810 0x5653a23e5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.467+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a23e6300 con 0x5653a23d9810 2024-09-16T01:43:07.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.467+0000 7f1ed7fff640 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5653a23d9810 0x5653a23e5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.467+0000 7f1edf1f4640 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ec006ac90 0x7f1ec006d150 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0046650 tx=0x7f1ed006f040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:07.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.467+0000 7f1ed7fff640 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5653a23d9810 0x5653a23e5dc0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:07.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.469+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v116'877 uv851 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.470+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e116) -- 0x5653a2488aa0 con 0x5653a23d9810 2024-09-16T01:43:07.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.471+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v116'878 uv878 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.471+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2488e40 con 0x5653a23d9810 2024-09-16T01:43:07.474 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:43:07.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.473+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v116'879 uv879 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.473+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a248a1f0 con 0x5653a23d9810 2024-09-16T01:43:07.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.474+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v116'880 uv880 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.474+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a248a590 con 0x5653a23d9810 2024-09-16T01:43:07.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.475+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v116'881 uv881 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.475+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a24891e0 con 0x5653a23d9810 2024-09-16T01:43:07.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.476+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v116'882 uv882 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.476+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a24895d0 con 0x5653a23d9810 2024-09-16T01:43:07.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.478+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v116'883 uv857 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.478+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e116) -- 0x5653a2489970 con 0x5653a23d9810 2024-09-16T01:43:07.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.480+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v116'884 uv884 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.480+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2489970 con 0x5653a23d9810 2024-09-16T01:43:07.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.481+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v116'885 uv885 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.483 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:43:07.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.481+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2489970 con 0x5653a23d9810 2024-09-16T01:43:07.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.482+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v116'886 uv886 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.482+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2489970 con 0x5653a23d9810 2024-09-16T01:43:07.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.483+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v116'887 uv887 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.483+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2489970 con 0x5653a23d9810 2024-09-16T01:43:07.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.485+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v116'888 uv888 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.486 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.485+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2489970 con 0x5653a23d9810 2024-09-16T01:43:07.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.486+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v116'889 uv864 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.486+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e116) -- 0x5653a2489970 con 0x5653a23d9810 2024-09-16T01:43:07.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.487+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v116'890 uv890 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.487+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2489970 con 0x5653a23d9810 2024-09-16T01:43:07.490 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:43:07.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.489+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v116'891 uv891 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.489+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2489970 con 0x5653a23d9810 2024-09-16T01:43:07.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.490+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v116'892 uv892 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.490+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2489970 con 0x5653a23d9810 2024-09-16T01:43:07.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.491+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v116'893 uv893 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.491+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2489970 con 0x5653a23d9810 2024-09-16T01:43:07.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.493+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v116'894 uv894 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.493+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2489970 con 0x5653a23d9810 2024-09-16T01:43:07.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.494+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v116'895 uv895 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.494+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2489970 con 0x5653a23d9810 2024-09-16T01:43:07.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.495+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v116'896 uv870 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.495+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e116) -- 0x5653a2489970 con 0x5653a23d9810 2024-09-16T01:43:07.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.497+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v116'897 uv897 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.497+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2489970 con 0x5653a23d9810 2024-09-16T01:43:07.502 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:43:07.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.498+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v116'898 uv898 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.498+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2489970 con 0x5653a23d9810 2024-09-16T01:43:07.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.499+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v116'899 uv899 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.499+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2489970 con 0x5653a23d9810 2024-09-16T01:43:07.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.500+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v116'900 uv900 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.500+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2489970 con 0x5653a23d9810 2024-09-16T01:43:07.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.502+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v116'901 uv901 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.502+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2489970 con 0x5653a23d9810 2024-09-16T01:43:07.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.503+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v116'902 uv876 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.503+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e116) -- 0x5653a2489970 con 0x5653a23d9810 2024-09-16T01:43:07.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.504+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v116'903 uv903 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.504+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2489970 con 0x5653a23d9810 2024-09-16T01:43:07.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.506+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v116'904 uv904 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.507 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:43:07.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.506+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2489970 con 0x5653a23d9810 2024-09-16T01:43:07.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.507+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v116'905 uv905 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.507+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2489970 con 0x5653a23d9810 2024-09-16T01:43:07.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.508+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v116'906 uv906 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.508+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2489970 con 0x5653a23d9810 2024-09-16T01:43:07.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.509+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v116'907 uv907 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.509+0000 7f1ee16fef40 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5653a2489d30 0x5653a248ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.509+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2489970 con 0x5653a2489d30 2024-09-16T01:43:07.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.510+0000 7f1edf1f4640 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5653a2489d30 0x5653a248ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.510+0000 7f1edf1f4640 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5653a2489d30 0x5653a248ce50 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:07.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.512+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v116'639 uv622 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2489d30 2024-09-16T01:43:07.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.512+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e116) -- 0x5653a248d390 con 0x5653a2489d30 2024-09-16T01:43:07.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.513+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v116'640 uv640 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2489d30 2024-09-16T01:43:07.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.513+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a248d730 con 0x5653a2489d30 2024-09-16T01:43:07.605 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:43:07.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.514+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v116'641 uv641 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2489d30 2024-09-16T01:43:07.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.514+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a248df80 con 0x5653a2489d30 2024-09-16T01:43:07.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.515+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v116'642 uv642 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2489d30 2024-09-16T01:43:07.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.515+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a248e320 con 0x5653a2489d30 2024-09-16T01:43:07.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.517+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v116'643 uv643 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2489d30 2024-09-16T01:43:07.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.517+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a248e6c0 con 0x5653a2489d30 2024-09-16T01:43:07.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.518+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v116'644 uv627 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2489d30 2024-09-16T01:43:07.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.518+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e116) -- 0x5653a248ea60 con 0x5653a2489d30 2024-09-16T01:43:07.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.520+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v116'645 uv645 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2489d30 2024-09-16T01:43:07.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.520+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a248ee00 con 0x5653a2489d30 2024-09-16T01:43:07.607 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:43:07.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.521+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v116'646 uv646 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2489d30 2024-09-16T01:43:07.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.521+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a248ee00 con 0x5653a2489d30 2024-09-16T01:43:07.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.522+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v116'647 uv647 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2489d30 2024-09-16T01:43:07.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.522+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a248ee00 con 0x5653a2489d30 2024-09-16T01:43:07.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.523+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v116'648 uv648 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2489d30 2024-09-16T01:43:07.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.523+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a248ee00 con 0x5653a2489d30 2024-09-16T01:43:07.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.524+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v116'649 uv632 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2489d30 2024-09-16T01:43:07.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.524+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e116) -- 0x5653a248ee00 con 0x5653a2489d30 2024-09-16T01:43:07.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.526+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v116'650 uv650 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2489d30 2024-09-16T01:43:07.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.526+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a248ee00 con 0x5653a2489d30 2024-09-16T01:43:07.609 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:43:07.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.527+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v116'651 uv651 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2489d30 2024-09-16T01:43:07.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.527+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2493280 con 0x5653a2489d30 2024-09-16T01:43:07.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.528+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v116'652 uv652 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2489d30 2024-09-16T01:43:07.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.528+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2493280 con 0x5653a2489d30 2024-09-16T01:43:07.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.530+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v116'653 uv653 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2489d30 2024-09-16T01:43:07.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.530+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a248ee00 con 0x5653a2489d30 2024-09-16T01:43:07.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.531+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v116'654 uv638 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2489d30 2024-09-16T01:43:07.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.531+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e116) -- 0x5653a248ee00 con 0x5653a2489d30 2024-09-16T01:43:07.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.532+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v116'655 uv655 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2489d30 2024-09-16T01:43:07.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.532+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a248ee00 con 0x5653a2489d30 2024-09-16T01:43:07.611 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:43:07.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.533+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v116'656 uv656 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2489d30 2024-09-16T01:43:07.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.533+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a248fe10 con 0x5653a2489d30 2024-09-16T01:43:07.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.534+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v116'657 uv657 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2489d30 2024-09-16T01:43:07.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.534+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a248fe10 con 0x5653a2489d30 2024-09-16T01:43:07.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.535+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v116'658 uv658 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2489d30 2024-09-16T01:43:07.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.535+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a248ee00 con 0x5653a2489d30 2024-09-16T01:43:07.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.537+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v116'659 uv659 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2489d30 2024-09-16T01:43:07.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.537+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a248ee00 con 0x5653a23d9810 2024-09-16T01:43:07.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.538+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v116'171 uv170 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.538+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e116) -- 0x5653a248ee00 con 0x5653a23d9810 2024-09-16T01:43:07.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.539+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v116'172 uv172 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.539+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a248ee00 con 0x5653a23d9810 2024-09-16T01:43:07.614 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:43:07.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.540+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v116'173 uv173 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.540+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a248fe10 con 0x5653a23d9810 2024-09-16T01:43:07.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.540+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v116'174 uv174 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.541+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a248fe10 con 0x5653a23d9810 2024-09-16T01:43:07.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.541+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v116'175 uv175 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.541+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a248ee00 con 0x5653a23d9810 2024-09-16T01:43:07.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.542+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v116'176 uv176 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.542+0000 7f1ee16fef40 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5653a248f480 0x5653a248f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.542+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2492210 con 0x5653a248f480 2024-09-16T01:43:07.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.542+0000 7f1ed7fff640 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5653a248f480 0x5653a248f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.543+0000 7f1ed7fff640 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5653a248f480 0x5653a248f8e0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:07.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.544+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v116'127 uv126 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a248f480 2024-09-16T01:43:07.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.544+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e116) -- 0x5653a2492210 con 0x5653a248f480 2024-09-16T01:43:07.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.545+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v116'128 uv128 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a248f480 2024-09-16T01:43:07.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.545+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2492210 con 0x5653a248f480 2024-09-16T01:43:07.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.546+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v116'129 uv129 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a248f480 2024-09-16T01:43:07.618 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:43:07.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.546+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2492210 con 0x5653a248f480 2024-09-16T01:43:07.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.547+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v116'130 uv130 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a248f480 2024-09-16T01:43:07.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.547+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2492210 con 0x5653a23d9810 2024-09-16T01:43:07.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.548+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v116'171 uv170 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.548+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e116) -- 0x5653a2492210 con 0x5653a23d9810 2024-09-16T01:43:07.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.549+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v116'172 uv172 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.549+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2492210 con 0x5653a23d9810 2024-09-16T01:43:07.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.550+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v116'173 uv173 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.620 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:43:07.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.551+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2493220 con 0x5653a23d9810 2024-09-16T01:43:07.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.552+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v116'174 uv174 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.552+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2493220 con 0x5653a23d9810 2024-09-16T01:43:07.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.552+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v116'175 uv175 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.553+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2492210 con 0x5653a23d9810 2024-09-16T01:43:07.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.554+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v116'176 uv176 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.554+0000 7f1ee16fef40 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5653a2492b10 0x5653a2492f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.554+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2495610 con 0x5653a2492b10 2024-09-16T01:43:07.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.554+0000 7f1edf9f5640 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5653a2492b10 0x5653a2492f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.554+0000 7f1edf9f5640 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5653a2492b10 0x5653a2492f10 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:07.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.556+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v116'316 uv310 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.556+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e116) -- 0x5653a24959b0 con 0x5653a2492b10 2024-09-16T01:43:07.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.557+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v116'317 uv317 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.557+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2495d50 con 0x5653a2492b10 2024-09-16T01:43:07.623 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:43:07.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.558+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v116'318 uv318 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.558+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a248dbd0 con 0x5653a2492b10 2024-09-16T01:43:07.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.559+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v116'319 uv319 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.559+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2497690 con 0x5653a2492b10 2024-09-16T01:43:07.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.560+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v116'320 uv320 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.560+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a24965a0 con 0x5653a2492b10 2024-09-16T01:43:07.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.561+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v116'321 uv321 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.561+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2496990 con 0x5653a2492b10 2024-09-16T01:43:07.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.562+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v116'322 uv315 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.562+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e116) -- 0x5653a2496d80 con 0x5653a2492b10 2024-09-16T01:43:07.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.563+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v116'323 uv323 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.563+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2497120 con 0x5653a2492b10 2024-09-16T01:43:07.626 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:43:07.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.564+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v116'324 uv324 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.564+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2497120 con 0x5653a2492b10 2024-09-16T01:43:07.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.567+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v116'325 uv325 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.567+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a2497120 con 0x5653a2492b10 2024-09-16T01:43:07.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.568+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v116'326 uv326 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.568+0000 7f1ee16fef40 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5653a2497fb0 0x5653a249a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.568+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249a8e0 con 0x5653a2497fb0 2024-09-16T01:43:07.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.568+0000 7f1edf1f4640 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5653a2497fb0 0x5653a249a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.569+0000 7f1edf1f4640 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5653a2497fb0 0x5653a249a3a0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:07.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.570+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v116'143 uv142 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2497fb0 2024-09-16T01:43:07.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.570+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e116) -- 0x5653a249a8e0 con 0x5653a2497fb0 2024-09-16T01:43:07.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.571+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v116'144 uv144 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2497fb0 2024-09-16T01:43:07.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.571+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249acd0 con 0x5653a2497fb0 2024-09-16T01:43:07.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.572+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v116'145 uv145 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2497fb0 2024-09-16T01:43:07.629 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:43:07.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.572+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249acd0 con 0x5653a2497fb0 2024-09-16T01:43:07.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.573+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v116'146 uv146 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2497fb0 2024-09-16T01:43:07.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.573+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249acd0 con 0x5653a2497fb0 2024-09-16T01:43:07.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.574+0000 7f1edf1f4640 1 -- 172.21.15.88:0/3141372312 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v116'147 uv147 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1ed0061450 con 0x5653a2497fb0 2024-09-16T01:43:07.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.575+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249acd0 con 0x5653a23d9810 2024-09-16T01:43:07.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.576+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v116'171 uv170 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.576+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e116) -- 0x5653a249acd0 con 0x5653a23d9810 2024-09-16T01:43:07.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.577+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v116'172 uv172 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.577+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249acd0 con 0x5653a23d9810 2024-09-16T01:43:07.630 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:43:07.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.578+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v116'173 uv173 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.578+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249bce0 con 0x5653a23d9810 2024-09-16T01:43:07.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.578+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v116'174 uv174 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.579+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249bce0 con 0x5653a23d9810 2024-09-16T01:43:07.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.579+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v116'175 uv175 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.579+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249acd0 con 0x5653a23d9810 2024-09-16T01:43:07.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.580+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v116'176 uv176 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a23d9810 2024-09-16T01:43:07.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.580+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249acd0 con 0x5653a248f480 2024-09-16T01:43:07.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.581+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v116'284 uv283 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a248f480 2024-09-16T01:43:07.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.581+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e116) -- 0x5653a249acd0 con 0x5653a248f480 2024-09-16T01:43:07.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.582+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v116'285 uv285 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a248f480 2024-09-16T01:43:07.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.582+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249acd0 con 0x5653a248f480 2024-09-16T01:43:07.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.583+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v116'286 uv286 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a248f480 2024-09-16T01:43:07.633 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:43:07.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.583+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249acd0 con 0x5653a248f480 2024-09-16T01:43:07.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.584+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v116'287 uv287 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a248f480 2024-09-16T01:43:07.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.584+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249acd0 con 0x5653a248f480 2024-09-16T01:43:07.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.585+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v116'288 uv288 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a248f480 2024-09-16T01:43:07.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.585+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249acd0 con 0x5653a248f480 2024-09-16T01:43:07.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.586+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v116'289 uv282 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a248f480 2024-09-16T01:43:07.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.586+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e116) -- 0x5653a249acd0 con 0x5653a248f480 2024-09-16T01:43:07.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.587+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v116'290 uv290 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a248f480 2024-09-16T01:43:07.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.587+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249acd0 con 0x5653a248f480 2024-09-16T01:43:07.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.588+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v116'291 uv291 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a248f480 2024-09-16T01:43:07.635 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:43:07.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.588+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249acd0 con 0x5653a248f480 2024-09-16T01:43:07.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.589+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v116'292 uv292 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a248f480 2024-09-16T01:43:07.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.589+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249da60 con 0x5653a248f480 2024-09-16T01:43:07.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.590+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v116'293 uv293 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a248f480 2024-09-16T01:43:07.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.590+0000 7f1ee16fef40 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5653a249b740 0x5653a249db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.590+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249e120 con 0x5653a249b740 2024-09-16T01:43:07.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.590+0000 7f1ed7fff640 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5653a249b740 0x5653a249db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.591+0000 7f1ed7fff640 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5653a249b740 0x5653a249db30 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:07.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.592+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v116'124 uv123 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a249b740 2024-09-16T01:43:07.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.592+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e116) -- 0x5653a249e120 con 0x5653a249b740 2024-09-16T01:43:07.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.593+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v116'125 uv125 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a249b740 2024-09-16T01:43:07.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.593+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249e120 con 0x5653a249b740 2024-09-16T01:43:07.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.593+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v116'126 uv126 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a249b740 2024-09-16T01:43:07.638 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:43:07.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.594+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249e1a0 con 0x5653a249b740 2024-09-16T01:43:07.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.594+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v116'127 uv127 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a249b740 2024-09-16T01:43:07.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.595+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249e1a0 con 0x5653a248f480 2024-09-16T01:43:07.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.596+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v116'171 uv170 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a248f480 2024-09-16T01:43:07.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.596+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e116) -- 0x5653a249e1a0 con 0x5653a248f480 2024-09-16T01:43:07.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.597+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v116'172 uv172 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a248f480 2024-09-16T01:43:07.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.597+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249e120 con 0x5653a248f480 2024-09-16T01:43:07.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.599+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v116'173 uv173 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a248f480 2024-09-16T01:43:07.640 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:43:07.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.599+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249f170 con 0x5653a248f480 2024-09-16T01:43:07.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.600+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v116'174 uv174 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a248f480 2024-09-16T01:43:07.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.600+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249f130 con 0x5653a248f480 2024-09-16T01:43:07.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.601+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v116'175 uv175 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a248f480 2024-09-16T01:43:07.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.601+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249e0d0 con 0x5653a248f480 2024-09-16T01:43:07.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.602+0000 7f1ed7fff640 1 -- 172.21.15.88:0/3141372312 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v116'176 uv176 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ec8009950 con 0x5653a248f480 2024-09-16T01:43:07.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.602+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249e0d0 con 0x5653a2492b10 2024-09-16T01:43:07.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.603+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v116'145 uv144 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.603+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e116) -- 0x5653a249e0d0 con 0x5653a2492b10 2024-09-16T01:43:07.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.605+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v116'146 uv146 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.605+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249e0d0 con 0x5653a2492b10 2024-09-16T01:43:07.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.606+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v116'147 uv147 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.642 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:43:07.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.606+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249e0d0 con 0x5653a2492b10 2024-09-16T01:43:07.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.607+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v116'148 uv148 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.607+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249e0d0 con 0x5653a2492b10 2024-09-16T01:43:07.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.608+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v116'149 uv149 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.608+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249e0d0 con 0x5653a2492b10 2024-09-16T01:43:07.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.610+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v116'351 uv343 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.610+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e116) -- 0x5653a249e0d0 con 0x5653a2492b10 2024-09-16T01:43:07.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.611+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v116'352 uv352 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.611+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249e0d0 con 0x5653a2492b10 2024-09-16T01:43:07.644 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:43:07.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.612+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v116'353 uv353 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.612+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249e0d0 con 0x5653a2492b10 2024-09-16T01:43:07.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.613+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v116'354 uv354 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.613+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249e0d0 con 0x5653a2492b10 2024-09-16T01:43:07.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.614+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v116'355 uv355 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.614+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249e0d0 con 0x5653a2492b10 2024-09-16T01:43:07.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.615+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v116'356 uv350 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.615+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e116) -- 0x5653a249e510 con 0x5653a2492b10 2024-09-16T01:43:07.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.616+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v116'357 uv357 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.616+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249e0d0 con 0x5653a2492b10 2024-09-16T01:43:07.646 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:43:07.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.617+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v116'358 uv358 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.617+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249f620 con 0x5653a2492b10 2024-09-16T01:43:07.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.618+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v116'359 uv359 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.618+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249f620 con 0x5653a2492b10 2024-09-16T01:43:07.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.619+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v116'360 uv360 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.619+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249e4c0 con 0x5653a2492b10 2024-09-16T01:43:07.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.620+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v116'361 uv361 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.620+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x5653a249e4c0 con 0x5653a2492b10 2024-09-16T01:43:07.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.621+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v116'362 uv362 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f1ed80540e0 con 0x5653a2492b10 2024-09-16T01:43:07.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.621+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5653a23d9810 msgr2=0x5653a23e5dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:07.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.621+0000 7f1ee16fef40 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5653a23d9810 0x5653a23e5dc0 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.621+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5653a248f480 msgr2=0x5653a248f8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:07.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.621+0000 7f1ee16fef40 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5653a248f480 0x5653a248f8e0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.621+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5653a249b740 msgr2=0x5653a249db30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:07.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.621+0000 7f1ee16fef40 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5653a249b740 0x5653a249db30 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.621+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5653a2489d30 msgr2=0x5653a248ce50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:07.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.621+0000 7f1ee16fef40 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5653a2489d30 0x5653a248ce50 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.621+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 reap_dead start 2024-09-16T01:43:07.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.621+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5653a2492b10 msgr2=0x5653a2492f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:07.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.621+0000 7f1ee16fef40 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5653a2492b10 0x5653a2492f10 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.621+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5653a2497fb0 msgr2=0x5653a249a3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:07.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.621+0000 7f1ee16fef40 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5653a2497fb0 0x5653a249a3a0 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.621+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ec006ac90 msgr2=0x7f1ec006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:07.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.621+0000 7f1ee16fef40 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ec006ac90 0x7f1ec006d150 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0046650 tx=0x7f1ed006f040 comp rx=0 tx=0).stop 2024-09-16T01:43:07.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.621+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ecc0022d0 msgr2=0x5653a24730c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:07.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.621+0000 7f1ee16fef40 1 --2- 172.21.15.88:0/3141372312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ecc0022d0 0x5653a24730c0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x5653a233c4d0 tx=0x7f1ed800e5a0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.621+0000 7f1edf9f5640 1 -- 172.21.15.88:0/3141372312 reap_dead start 2024-09-16T01:43:07.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.621+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 shutdown_connections 2024-09-16T01:43:07.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.621+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 >> 172.21.15.88:0/3141372312 conn(0x5653a233a680 msgr2=0x5653a2310660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:07.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.621+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 shutdown_connections 2024-09-16T01:43:07.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.621+0000 7f1ee16fef40 1 -- 172.21.15.88:0/3141372312 wait complete. 2024-09-16T01:43:07.651 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata truncate 400.00000000 0 2024-09-16T01:43:07.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.733+0000 7f2897e91f40 1 Processor -- start 2024-09-16T01:43:07.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.733+0000 7f2897e91f40 1 -- start start 2024-09-16T01:43:07.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.734+0000 7f2897e91f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1d0d3610 0x55cb1d1a0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.734+0000 7f2897e91f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cb1d188ed0 0x55cb1d1a7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.734+0000 7f2897e91f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55cb1d18b190 0x55cb1d1175a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.734+0000 7f2897e91f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55cb1d022510 con 0x55cb1d188ed0 2024-09-16T01:43:07.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.734+0000 7f2897e91f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cb1d18a210 con 0x55cb1d0d3610 2024-09-16T01:43:07.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.734+0000 7f2897e91f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55cb1d1a9520 con 0x55cb1d18b190 2024-09-16T01:43:07.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.734+0000 7f288ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cb1d188ed0 0x55cb1d1a7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.734+0000 7f288ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cb1d188ed0 0x55cb1d1a7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55830/0 (socket says 172.21.15.88:55830) 2024-09-16T01:43:07.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.734+0000 7f288ffff640 1 -- 172.21.15.88:0/553007859 learned_addr learned my addr 172.21.15.88:0/553007859 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:07.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.734+0000 7f2895987640 1 --2- 172.21.15.88:0/553007859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1d0d3610 0x55cb1d1a0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.734+0000 7f2896188640 1 --2- 172.21.15.88:0/553007859 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55cb1d18b190 0x55cb1d1175a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.734+0000 7f288ffff640 1 -- 172.21.15.88:0/553007859 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55cb1d18b190 msgr2=0x55cb1d1175a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:07.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.734+0000 7f288ffff640 1 --2- 172.21.15.88:0/553007859 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55cb1d18b190 0x55cb1d1175a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.734+0000 7f288ffff640 1 -- 172.21.15.88:0/553007859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1d0d3610 msgr2=0x55cb1d1a0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:07.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.734+0000 7f288ffff640 1 --2- 172.21.15.88:0/553007859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1d0d3610 0x55cb1d1a0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.734+0000 7f288ffff640 1 -- 172.21.15.88:0/553007859 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb1d111c30 con 0x55cb1d188ed0 2024-09-16T01:43:07.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.734+0000 7f288ffff640 1 --2- 172.21.15.88:0/553007859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cb1d188ed0 0x55cb1d1a7a20 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f2880001dd0 tx=0x7f288000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a51f5b6246d646e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:07.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.734+0000 7f2894985640 1 -- 172.21.15.88:0/553007859 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288000bde0 con 0x55cb1d188ed0 2024-09-16T01:43:07.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.734+0000 7f2894985640 1 -- 172.21.15.88:0/553007859 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f288000f400 con 0x55cb1d188ed0 2024-09-16T01:43:07.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.734+0000 7f2894985640 1 -- 172.21.15.88:0/553007859 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288000f830 con 0x55cb1d188ed0 2024-09-16T01:43:07.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.735+0000 7f2897e91f40 1 -- 172.21.15.88:0/553007859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cb1d188ed0 msgr2=0x55cb1d1a7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:07.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.735+0000 7f2897e91f40 1 --2- 172.21.15.88:0/553007859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cb1d188ed0 0x55cb1d1a7a20 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f2880001dd0 tx=0x7f288000b5b0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.735+0000 7f2897e91f40 1 -- 172.21.15.88:0/553007859 shutdown_connections 2024-09-16T01:43:07.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.735+0000 7f2897e91f40 1 --2- 172.21.15.88:0/553007859 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55cb1d18b190 0x55cb1d1175a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.735+0000 7f2897e91f40 1 --2- 172.21.15.88:0/553007859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cb1d188ed0 0x55cb1d1a7a20 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.735+0000 7f2897e91f40 1 --2- 172.21.15.88:0/553007859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1d0d3610 0x55cb1d1a0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.735+0000 7f2897e91f40 1 -- 172.21.15.88:0/553007859 >> 172.21.15.88:0/553007859 conn(0x55cb1d0cfbe0 msgr2=0x55cb1d116e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:07.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.735+0000 7f2897e91f40 1 -- 172.21.15.88:0/553007859 shutdown_connections 2024-09-16T01:43:07.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.735+0000 7f2897e91f40 1 -- 172.21.15.88:0/553007859 wait complete. 2024-09-16T01:43:07.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.735+0000 7f2897e91f40 1 Processor -- start 2024-09-16T01:43:07.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.735+0000 7f2897e91f40 1 -- start start 2024-09-16T01:43:07.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.736+0000 7f2897e91f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1d0d3610 0x55cb1d239630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.736+0000 7f2897e91f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cb1d188ed0 0x55cb1d19e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.736+0000 7f2897e91f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55cb1d18b190 0x55cb1d112af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.736+0000 7f2897e91f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55cb1d11a190 con 0x55cb1d188ed0 2024-09-16T01:43:07.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.736+0000 7f2897e91f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55cb1d119eb0 con 0x55cb1d0d3610 2024-09-16T01:43:07.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.736+0000 7f2897e91f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55cb1d1a91b0 con 0x55cb1d18b190 2024-09-16T01:43:07.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.736+0000 7f288ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cb1d188ed0 0x55cb1d19e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.736+0000 7f288ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cb1d188ed0 0x55cb1d19e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55840/0 (socket says 172.21.15.88:55840) 2024-09-16T01:43:07.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.736+0000 7f288ffff640 1 -- 172.21.15.88:0/236319793 learned_addr learned my addr 172.21.15.88:0/236319793 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:07.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.736+0000 7f2895987640 1 --2- 172.21.15.88:0/236319793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1d0d3610 0x55cb1d239630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.736+0000 7f2896188640 1 --2- 172.21.15.88:0/236319793 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55cb1d18b190 0x55cb1d112af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.736+0000 7f288ffff640 1 -- 172.21.15.88:0/236319793 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55cb1d18b190 msgr2=0x55cb1d112af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:07.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.737+0000 7f288ffff640 1 --2- 172.21.15.88:0/236319793 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55cb1d18b190 0x55cb1d112af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.737+0000 7f288ffff640 1 -- 172.21.15.88:0/236319793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1d0d3610 msgr2=0x55cb1d239630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:07.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.737+0000 7f288ffff640 1 --2- 172.21.15.88:0/236319793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb1d0d3610 0x55cb1d239630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.737+0000 7f288ffff640 1 -- 172.21.15.88:0/236319793 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb1d118750 con 0x55cb1d188ed0 2024-09-16T01:43:07.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.737+0000 7f2896188640 1 --2- 172.21.15.88:0/236319793 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55cb1d18b190 0x55cb1d112af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:07.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.737+0000 7f288ffff640 1 --2- 172.21.15.88:0/236319793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cb1d188ed0 0x55cb1d19e930 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x55cb1d0d35e0 tx=0x7f2880000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:07.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.737+0000 7f288effd640 1 -- 172.21.15.88:0/236319793 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288001c440 con 0x55cb1d188ed0 2024-09-16T01:43:07.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.737+0000 7f288effd640 1 -- 172.21.15.88:0/236319793 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2880021020 con 0x55cb1d188ed0 2024-09-16T01:43:07.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.737+0000 7f2897e91f40 1 -- 172.21.15.88:0/236319793 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55cb1d186e50 con 0x55cb1d188ed0 2024-09-16T01:43:07.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.737+0000 7f288effd640 1 -- 172.21.15.88:0/236319793 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288001c600 con 0x55cb1d188ed0 2024-09-16T01:43:07.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.737+0000 7f2897e91f40 1 -- 172.21.15.88:0/236319793 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb1d2391c0 con 0x55cb1d188ed0 2024-09-16T01:43:07.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.738+0000 7f288effd640 1 -- 172.21.15.88:0/236319793 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2880017070 con 0x55cb1d188ed0 2024-09-16T01:43:07.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.739+0000 7f288effd640 1 --2- 172.21.15.88:0/236319793 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f287806ac90 0x7f287806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.739+0000 7f288effd640 1 -- 172.21.15.88:0/236319793 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8232+0+0 (secure 0 0 0) 0x7f288008c140 con 0x55cb1d188ed0 2024-09-16T01:43:07.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.739+0000 7f2895987640 1 --2- 172.21.15.88:0/236319793 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f287806ac90 0x7f287806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.739+0000 7f2897e91f40 1 --2- 172.21.15.88:0/236319793 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55cb1d239b70 0x55cb1d2425a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.739+0000 7f2897e91f40 1 -- 172.21.15.88:0/236319793 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:1 12.f 12:f1f69726:::400.00000000:head [truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116) -- 0x55cb1d19ee70 con 0x55cb1d239b70 2024-09-16T01:43:07.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.739+0000 7f2895987640 1 --2- 172.21.15.88:0/236319793 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f287806ac90 0x7f287806d150 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f2888005fd0 tx=0x7f2888005af0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:07.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.739+0000 7f2896188640 1 --2- 172.21.15.88:0/236319793 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55cb1d239b70 0x55cb1d2425a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.740+0000 7f2896188640 1 --2- 172.21.15.88:0/236319793 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55cb1d239b70 0x55cb1d2425a0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:07.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.741+0000 7f2896188640 1 -- 172.21.15.88:0/236319793 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(1 400.00000000 [truncate 0] v116'150 uv150 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f289004e960 con 0x55cb1d239b70 2024-09-16T01:43:07.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.741+0000 7f2897e91f40 1 -- 172.21.15.88:0/236319793 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55cb1d239b70 msgr2=0x55cb1d2425a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:07.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.741+0000 7f2897e91f40 1 --2- 172.21.15.88:0/236319793 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55cb1d239b70 0x55cb1d2425a0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.741+0000 7f2897e91f40 1 -- 172.21.15.88:0/236319793 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f287806ac90 msgr2=0x7f287806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:07.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.741+0000 7f2897e91f40 1 --2- 172.21.15.88:0/236319793 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f287806ac90 0x7f287806d150 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f2888005fd0 tx=0x7f2888005af0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.741+0000 7f2897e91f40 1 -- 172.21.15.88:0/236319793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cb1d188ed0 msgr2=0x55cb1d19e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:07.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.741+0000 7f2897e91f40 1 --2- 172.21.15.88:0/236319793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55cb1d188ed0 0x55cb1d19e930 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x55cb1d0d35e0 tx=0x7f2880000ed0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.741+0000 7f2896188640 1 -- 172.21.15.88:0/236319793 reap_dead start 2024-09-16T01:43:07.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.741+0000 7f2897e91f40 1 -- 172.21.15.88:0/236319793 shutdown_connections 2024-09-16T01:43:07.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.741+0000 7f2897e91f40 1 -- 172.21.15.88:0/236319793 >> 172.21.15.88:0/236319793 conn(0x55cb1d0cfbe0 msgr2=0x55cb1d1898f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:07.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.741+0000 7f2897e91f40 1 -- 172.21.15.88:0/236319793 shutdown_connections 2024-09-16T01:43:07.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.741+0000 7f2897e91f40 1 -- 172.21.15.88:0/236319793 wait complete. 2024-09-16T01:43:07.747 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:43:07.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.889+0000 7f66ba73a640 1 Processor -- start 2024-09-16T01:43:07.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.889+0000 7f66ba73a640 1 -- start start 2024-09-16T01:43:07.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.890+0000 7f66ba73a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66b4072700 0x7f66b4166120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.890+0000 7f66ba73a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66b40ccff0 con 0x7f66b40b6290 2024-09-16T01:43:07.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.890+0000 7f66ba73a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66b40cd2d0 con 0x7f66b415b480 2024-09-16T01:43:07.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.890+0000 7f66ba73a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f66b40cd5b0 con 0x7f66b4072700 2024-09-16T01:43:07.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.890+0000 7f66b37fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f66b40b6290 0x7f66b4071ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57106/0 (socket says 172.21.15.88:57106) 2024-09-16T01:43:07.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.890+0000 7f66b37fe640 1 -- 172.21.15.88:0/1081572421 learned_addr learned my addr 172.21.15.88:0/1081572421 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:07.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.890+0000 7f66b8cd7640 1 --2- 172.21.15.88:0/1081572421 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66b4072700 0x7f66b4166120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.890+0000 7f66b2ffd640 1 -- 172.21.15.88:0/1081572421 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1485572303 0 0) 0x7f66b40ccff0 con 0x7f66b40b6290 2024-09-16T01:43:07.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.890+0000 7f66b2ffd640 1 -- 172.21.15.88:0/1081572421 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f669c003520 con 0x7f66b40b6290 2024-09-16T01:43:07.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.890+0000 7f66b2ffd640 1 -- 172.21.15.88:0/1081572421 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1878355898 0 0) 0x7f669c003520 con 0x7f66b40b6290 2024-09-16T01:43:07.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.890+0000 7f66b2ffd640 1 -- 172.21.15.88:0/1081572421 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66b4072700 msgr2=0x7f66b4166120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:07.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.890+0000 7f66b2ffd640 1 --2- 172.21.15.88:0/1081572421 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66b4072700 0x7f66b4166120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.890+0000 7f66b2ffd640 1 -- 172.21.15.88:0/1081572421 >> v1:172.21.15.154:6789/0 conn(0x7f66b415b480 legacy=0x7f66b40b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:07.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.890+0000 7f66b2ffd640 1 -- 172.21.15.88:0/1081572421 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b40cd890 con 0x7f66b40b6290 2024-09-16T01:43:07.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.890+0000 7f66b2ffd640 1 -- 172.21.15.88:0/1081572421 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f66a8001670 con 0x7f66b40b6290 2024-09-16T01:43:07.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.890+0000 7f66b2ffd640 1 -- 172.21.15.88:0/1081572421 >> v1:172.21.15.88:6789/0 conn(0x7f66b40b6290 legacy=0x7f66b4071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:07.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.890+0000 7f66b2ffd640 1 --2- 172.21.15.88:0/1081572421 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f669c0039e0 0x7f669c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.890+0000 7f66b2ffd640 1 --2- 172.21.15.88:0/1081572421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f669c004390 0x7f669c004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.890+0000 7f66b2ffd640 1 --2- 172.21.15.88:0/1081572421 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66b4072700 0x7f669c0050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.891+0000 7f66b2ffd640 1 -- 172.21.15.88:0/1081572421 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f66b40cd5b0 con 0x7f669c0039e0 2024-09-16T01:43:07.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.891+0000 7f66b2ffd640 1 -- 172.21.15.88:0/1081572421 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f669c00b9f0 con 0x7f669c004390 2024-09-16T01:43:07.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.891+0000 7f66b2ffd640 1 -- 172.21.15.88:0/1081572421 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f669c00bbf0 con 0x7f66b4072700 2024-09-16T01:43:07.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.891+0000 7f66b3fff640 1 --2- 172.21.15.88:0/1081572421 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f669c0039e0 0x7f669c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.891+0000 7f66b37fe640 1 --2- 172.21.15.88:0/1081572421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f669c004390 0x7f669c004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.891+0000 7f66b8cd7640 1 --2- 172.21.15.88:0/1081572421 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66b4072700 0x7f669c0050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.891+0000 7f66b3fff640 1 -- 172.21.15.88:0/1081572421 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66b4072700 msgr2=0x7f669c0050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:07.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.891+0000 7f66b3fff640 1 --2- 172.21.15.88:0/1081572421 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66b4072700 0x7f669c0050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.891+0000 7f66b3fff640 1 -- 172.21.15.88:0/1081572421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f669c004390 msgr2=0x7f669c004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:07.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.891+0000 7f66b3fff640 1 --2- 172.21.15.88:0/1081572421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f669c004390 0x7f669c004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.891+0000 7f66b3fff640 1 -- 172.21.15.88:0/1081572421 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f669c003520 con 0x7f669c0039e0 2024-09-16T01:43:07.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.891+0000 7f66b37fe640 1 --2- 172.21.15.88:0/1081572421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f669c004390 0x7f669c004860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:07.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.892+0000 7f66b3fff640 1 --2- 172.21.15.88:0/1081572421 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f669c0039e0 0x7f669c003dc0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f66a4001860 tx=0x7f669c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc12689668daa0d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:07.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.892+0000 7f66b2ffd640 1 -- 172.21.15.88:0/1081572421 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a4037070 con 0x7f669c0039e0 2024-09-16T01:43:07.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.892+0000 7f66b2ffd640 1 -- 172.21.15.88:0/1081572421 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66a402ac30 con 0x7f669c0039e0 2024-09-16T01:43:07.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.892+0000 7f66b2ffd640 1 -- 172.21.15.88:0/1081572421 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a4001430 con 0x7f669c0039e0 2024-09-16T01:43:07.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.892+0000 7f66ba73a640 1 -- 172.21.15.88:0/1081572421 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f669c0039e0 msgr2=0x7f669c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:07.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.892+0000 7f66ba73a640 1 --2- 172.21.15.88:0/1081572421 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f669c0039e0 0x7f669c003dc0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f66a4001860 tx=0x7f669c005830 comp rx=0 tx=0).stop 2024-09-16T01:43:07.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.892+0000 7f66b8cd7640 1 -- 172.21.15.88:0/1081572421 reap_dead start 2024-09-16T01:43:07.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.892+0000 7f66ba73a640 1 -- 172.21.15.88:0/1081572421 shutdown_connections 2024-09-16T01:43:07.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.892+0000 7f66ba73a640 1 -- 172.21.15.88:0/1081572421 >> 172.21.15.88:0/1081572421 conn(0x7f66b406d0f0 msgr2=0x7f66b406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:07.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.892+0000 7f66ba73a640 1 -- 172.21.15.88:0/1081572421 shutdown_connections 2024-09-16T01:43:07.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.892+0000 7f66ba73a640 1 -- 172.21.15.88:0/1081572421 wait complete. 2024-09-16T01:43:07.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.893+0000 7f66ba73a640 1 Processor -- start 2024-09-16T01:43:07.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.894+0000 7f66ba73a640 1 -- start start 2024-09-16T01:43:07.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.894+0000 7f66ba73a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f669c0039e0 0x7f66b414eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.894+0000 7f66ba73a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66b41f43b0 0x7f66b414e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.894+0000 7f66ba73a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66b4151810 0x7f66b414f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.894+0000 7f66ba73a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f66b415b9e0 con 0x7f669c0039e0 2024-09-16T01:43:07.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.894+0000 7f66ba73a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f66b415b860 con 0x7f66b41f43b0 2024-09-16T01:43:07.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.894+0000 7f66ba73a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f66b415cc70 con 0x7f66b4151810 2024-09-16T01:43:07.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.894+0000 7f66b3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f669c0039e0 0x7f66b414eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.894+0000 7f66b3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f669c0039e0 0x7f66b414eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55864/0 (socket says 172.21.15.88:55864) 2024-09-16T01:43:07.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.895+0000 7f66b3fff640 1 -- 172.21.15.88:0/1817886200 learned_addr learned my addr 172.21.15.88:0/1817886200 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:07.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.895+0000 7f66b37fe640 1 --2- 172.21.15.88:0/1817886200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66b41f43b0 0x7f66b414e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.895+0000 7f66b8cd7640 1 --2- 172.21.15.88:0/1817886200 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66b4151810 0x7f66b414f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.895+0000 7f66b3fff640 1 -- 172.21.15.88:0/1817886200 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66b4151810 msgr2=0x7f66b414f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:07.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.895+0000 7f66b3fff640 1 --2- 172.21.15.88:0/1817886200 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66b4151810 0x7f66b414f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.895+0000 7f66b3fff640 1 -- 172.21.15.88:0/1817886200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66b41f43b0 msgr2=0x7f66b414e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:07.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.895+0000 7f66b3fff640 1 --2- 172.21.15.88:0/1817886200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66b41f43b0 0x7f66b414e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:07.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.895+0000 7f66b3fff640 1 -- 172.21.15.88:0/1817886200 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b41f4830 con 0x7f669c0039e0 2024-09-16T01:43:07.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.895+0000 7f66b37fe640 1 --2- 172.21.15.88:0/1817886200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66b41f43b0 0x7f66b414e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:07.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.895+0000 7f66b3fff640 1 --2- 172.21.15.88:0/1817886200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f669c0039e0 0x7f66b414eb70 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f66a4001980 tx=0x7f66a4001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:07.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.895+0000 7f66b0ff9640 1 -- 172.21.15.88:0/1817886200 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a4040020 con 0x7f669c0039e0 2024-09-16T01:43:07.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.895+0000 7f66b0ff9640 1 -- 172.21.15.88:0/1817886200 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66a4005270 con 0x7f669c0039e0 2024-09-16T01:43:07.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.895+0000 7f66b0ff9640 1 -- 172.21.15.88:0/1817886200 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a4037040 con 0x7f669c0039e0 2024-09-16T01:43:07.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.895+0000 7f66ba73a640 1 -- 172.21.15.88:0/1817886200 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66b41520b0 con 0x7f669c0039e0 2024-09-16T01:43:07.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.895+0000 7f66ba73a640 1 -- 172.21.15.88:0/1817886200 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66b40cad30 con 0x7f669c0039e0 2024-09-16T01:43:07.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.896+0000 7f66ba73a640 1 -- 172.21.15.88:0/1817886200 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66b4071ff0 con 0x7f669c0039e0 2024-09-16T01:43:07.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.897+0000 7f66b0ff9640 1 -- 172.21.15.88:0/1817886200 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f66a4043020 con 0x7f669c0039e0 2024-09-16T01:43:07.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.897+0000 7f66b0ff9640 1 --2- 172.21.15.88:0/1817886200 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f66a006c2b0 0x7f66a006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:07.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.897+0000 7f66b37fe640 1 --2- 172.21.15.88:0/1817886200 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f66a006c2b0 0x7f66a006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:07.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.897+0000 7f66b0ff9640 1 -- 172.21.15.88:0/1817886200 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 8232+0+0 (secure 0 0 0) 0x7f66a40abe70 con 0x7f669c0039e0 2024-09-16T01:43:07.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.898+0000 7f66b37fe640 1 --2- 172.21.15.88:0/1817886200 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f66a006c2b0 0x7f66a006e770 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f66a8001eb0 tx=0x7f66a8001110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:07.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:07.902+0000 7f66b0ff9640 1 -- 172.21.15.88:0/1817886200 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f66a404a030 con 0x7f669c0039e0 2024-09-16T01:43:08.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.100+0000 7f66ba73a640 1 -- 172.21.15.88:0/1817886200 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f66b415a180 con 0x7f669c0039e0 2024-09-16T01:43:08.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.274+0000 7f66b0ff9640 1 -- 172.21.15.88:0/1817886200 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v244) ==== 162+0+0 (secure 0 0 0) 0x7f66a4079c40 con 0x7f669c0039e0 2024-09-16T01:43:08.276 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:43:08.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.277+0000 7f66ba73a640 1 -- 172.21.15.88:0/1817886200 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f66a006c2b0 msgr2=0x7f66a006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:08.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.277+0000 7f66ba73a640 1 --2- 172.21.15.88:0/1817886200 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f66a006c2b0 0x7f66a006e770 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f66a8001eb0 tx=0x7f66a8001110 comp rx=0 tx=0).stop 2024-09-16T01:43:08.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.277+0000 7f66ba73a640 1 -- 172.21.15.88:0/1817886200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f669c0039e0 msgr2=0x7f66b414eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:08.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.277+0000 7f66ba73a640 1 --2- 172.21.15.88:0/1817886200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f669c0039e0 0x7f66b414eb70 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f66a4001980 tx=0x7f66a4001230 comp rx=0 tx=0).stop 2024-09-16T01:43:08.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.277+0000 7f66ba73a640 1 -- 172.21.15.88:0/1817886200 shutdown_connections 2024-09-16T01:43:08.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.277+0000 7f66ba73a640 1 --2- 172.21.15.88:0/1817886200 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f66a006c2b0 0x7f66a006e770 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:08.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.277+0000 7f66ba73a640 1 --2- 172.21.15.88:0/1817886200 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66b4151810 0x7f66b414f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:08.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.277+0000 7f66ba73a640 1 --2- 172.21.15.88:0/1817886200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66b41f43b0 0x7f66b414e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:08.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.277+0000 7f66ba73a640 1 --2- 172.21.15.88:0/1817886200 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f669c0039e0 0x7f66b414eb70 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:08.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.277+0000 7f66ba73a640 1 -- 172.21.15.88:0/1817886200 >> 172.21.15.88:0/1817886200 conn(0x7f66b406d0f0 msgr2=0x7f66b4158f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:08.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.277+0000 7f66ba73a640 1 -- 172.21.15.88:0/1817886200 shutdown_connections 2024-09-16T01:43:08.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.277+0000 7f66ba73a640 1 -- 172.21.15.88:0/1817886200 wait complete. 2024-09-16T01:43:08.280 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:43:08.278+0000 7f598d015640 -1 log_channel(cluster) log [ERR] : failed to read JournalPointer: -22 ((22) Invalid argument) 2024-09-16T01:43:08.280 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:43:08.279+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:43:08.280 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:43:08.279+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:43:08.291 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:43:08.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.388+0000 7f6d09b59640 1 Processor -- start 2024-09-16T01:43:08.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.388+0000 7f6d09b59640 1 -- start start 2024-09-16T01:43:08.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.389+0000 7f6d09b59640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6d040b63d0 0x7f6d040d39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:08.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.389+0000 7f6d09b59640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d040cfc20 con 0x7f6d040cf6b0 2024-09-16T01:43:08.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.389+0000 7f6d09b59640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d040d2ed0 con 0x7f6d040d1c80 2024-09-16T01:43:08.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.389+0000 7f6d09b59640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6d040d31b0 con 0x7f6d040b63d0 2024-09-16T01:43:08.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.389+0000 7f6d037fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6d040b63d0 0x7f6d040d39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:08.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.389+0000 7f6d02ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6d040cf6b0 0x7f6d040b7880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57112/0 (socket says 172.21.15.88:57112) 2024-09-16T01:43:08.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.389+0000 7f6d037fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6d040b63d0 0x7f6d040d39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55876/0 (socket says 172.21.15.88:55876) 2024-09-16T01:43:08.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.389+0000 7f6d02ffd640 1 -- 172.21.15.88:0/3706170641 learned_addr learned my addr 172.21.15.88:0/3706170641 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:08.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.390+0000 7f6d037fe640 1 -- 172.21.15.88:0/3706170641 >> v1:172.21.15.154:6789/0 conn(0x7f6d040d1c80 legacy=0x7f6d041629f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:08.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.390+0000 7f6d037fe640 1 -- 172.21.15.88:0/3706170641 >> v1:172.21.15.88:6789/0 conn(0x7f6d040cf6b0 legacy=0x7f6d040b7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:08.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.390+0000 7f6d027fc640 1 -- 172.21.15.88:0/3706170641 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1546617082 0 0) 0x7f6d040cfc20 con 0x7f6d040cf6b0 2024-09-16T01:43:08.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.390+0000 7f6d037fe640 1 -- 172.21.15.88:0/3706170641 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d040d3490 con 0x7f6d040b63d0 2024-09-16T01:43:08.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.390+0000 7f6d037fe640 1 --2- 172.21.15.88:0/3706170641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d040b63d0 0x7f6d040d39b0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f6cf40097a0 tx=0x7f6cf402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=827c3f8fcc731f5e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:08.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.390+0000 7f6d027fc640 1 -- 172.21.15.88:0/3706170641 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4037070 con 0x7f6d040b63d0 2024-09-16T01:43:08.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.391+0000 7f6d027fc640 1 -- 172.21.15.88:0/3706170641 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cf402ae50 con 0x7f6d040b63d0 2024-09-16T01:43:08.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.391+0000 7f6d027fc640 1 -- 172.21.15.88:0/3706170641 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4032660 con 0x7f6d040b63d0 2024-09-16T01:43:08.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.391+0000 7f6d09b59640 1 -- 172.21.15.88:0/3706170641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d040b63d0 msgr2=0x7f6d040d39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:08.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.391+0000 7f6d09b59640 1 --2- 172.21.15.88:0/3706170641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d040b63d0 0x7f6d040d39b0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f6cf40097a0 tx=0x7f6cf402a140 comp rx=0 tx=0).stop 2024-09-16T01:43:08.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.391+0000 7f6d09b59640 1 -- 172.21.15.88:0/3706170641 shutdown_connections 2024-09-16T01:43:08.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.391+0000 7f6d09b59640 1 --2- 172.21.15.88:0/3706170641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d040b63d0 0x7f6d040d39b0 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:08.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.391+0000 7f6d09b59640 1 -- 172.21.15.88:0/3706170641 >> 172.21.15.88:0/3706170641 conn(0x7f6d040d1160 msgr2=0x7f6d040d1570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:08.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.391+0000 7f6d09b59640 1 -- 172.21.15.88:0/3706170641 shutdown_connections 2024-09-16T01:43:08.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.391+0000 7f6d09b59640 1 -- 172.21.15.88:0/3706170641 wait complete. 2024-09-16T01:43:08.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.392+0000 7f6d09b59640 1 Processor -- start 2024-09-16T01:43:08.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.392+0000 7f6d09b59640 1 -- start start 2024-09-16T01:43:08.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.393+0000 7f6d09b59640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d040b63d0 0x7f6d0415b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:08.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.393+0000 7f6d09b59640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d040cf6b0 0x7f6d04149780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:08.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.393+0000 7f6d09b59640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d040d1c80 0x7f6d0415d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:08.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.393+0000 7f6d09b59640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6d040cb5c0 con 0x7f6d040d1c80 2024-09-16T01:43:08.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.393+0000 7f6d09b59640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6d040cb440 con 0x7f6d040b63d0 2024-09-16T01:43:08.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.393+0000 7f6d09b59640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6d040d3ef0 con 0x7f6d040cf6b0 2024-09-16T01:43:08.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.393+0000 7f6d03fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d040d1c80 0x7f6d0415d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:08.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.393+0000 7f6d03fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d040d1c80 0x7f6d0415d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55878/0 (socket says 172.21.15.88:55878) 2024-09-16T01:43:08.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.393+0000 7f6d03fff640 1 -- 172.21.15.88:0/2224281663 learned_addr learned my addr 172.21.15.88:0/2224281663 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:08.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.393+0000 7f6d037fe640 1 --2- 172.21.15.88:0/2224281663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d040b63d0 0x7f6d0415b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:08.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.393+0000 7f6d03fff640 1 -- 172.21.15.88:0/2224281663 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d040cf6b0 msgr2=0x7f6d04149780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:08.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.393+0000 7f6d03fff640 1 --2- 172.21.15.88:0/2224281663 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d040cf6b0 0x7f6d04149780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:08.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.393+0000 7f6d03fff640 1 -- 172.21.15.88:0/2224281663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d040b63d0 msgr2=0x7f6d0415b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:08.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.394+0000 7f6d03fff640 1 --2- 172.21.15.88:0/2224281663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d040b63d0 0x7f6d0415b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:08.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.394+0000 7f6d03fff640 1 -- 172.21.15.88:0/2224281663 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0415c850 con 0x7f6d040d1c80 2024-09-16T01:43:08.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.394+0000 7f6d037fe640 1 --2- 172.21.15.88:0/2224281663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d040b63d0 0x7f6d0415b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:08.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.394+0000 7f6d03fff640 1 --2- 172.21.15.88:0/2224281663 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d040d1c80 0x7f6d0415d2f0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc00b970 tx=0x7f6cfc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:08.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.394+0000 7f6cebfff640 1 -- 172.21.15.88:0/2224281663 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc019070 con 0x7f6d040d1c80 2024-09-16T01:43:08.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.394+0000 7f6cebfff640 1 -- 172.21.15.88:0/2224281663 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cfc00cb40 con 0x7f6d040d1c80 2024-09-16T01:43:08.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.394+0000 7f6d09b59640 1 -- 172.21.15.88:0/2224281663 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d0414a1a0 con 0x7f6d040d1c80 2024-09-16T01:43:08.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.394+0000 7f6cebfff640 1 -- 172.21.15.88:0/2224281663 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc014440 con 0x7f6d040d1c80 2024-09-16T01:43:08.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.394+0000 7f6d09b59640 1 -- 172.21.15.88:0/2224281663 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d0415caf0 con 0x7f6d040d1c80 2024-09-16T01:43:08.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.395+0000 7f6d09b59640 1 -- 172.21.15.88:0/2224281663 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d0414a6c0 con 0x7f6d040d1c80 2024-09-16T01:43:08.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.395+0000 7f6cebfff640 1 -- 172.21.15.88:0/2224281663 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6cfc01c070 con 0x7f6d040d1c80 2024-09-16T01:43:08.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.396+0000 7f6cebfff640 1 --2- 172.21.15.88:0/2224281663 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6cf006c2b0 0x7f6cf006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:08.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.396+0000 7f6cebfff640 1 -- 172.21.15.88:0/2224281663 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8318+0+0 (secure 0 0 0) 0x7f6cfc08bbc0 con 0x7f6d040d1c80 2024-09-16T01:43:08.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.396+0000 7f6d037fe640 1 --2- 172.21.15.88:0/2224281663 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6cf006c2b0 0x7f6cf006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:08.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.396+0000 7f6d037fe640 1 --2- 172.21.15.88:0/2224281663 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6cf006c2b0 0x7f6cf006e770 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4034000 tx=0x7f6cf402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:08.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.400+0000 7f6cebfff640 1 -- 172.21.15.88:0/2224281663 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6d0414a6c0 con 0x7f6d040d1c80 2024-09-16T01:43:08.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.601+0000 7f6d09b59640 1 -- 172.21.15.88:0/2224281663 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d040cf280 con 0x7f6d040d1c80 2024-09-16T01:43:08.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.601+0000 7f6cebfff640 1 -- 172.21.15.88:0/2224281663 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 245 v245) ==== 95+0+3566 (secure 0 0 0) 0x7f6cfc0598b0 con 0x7f6d040d1c80 2024-09-16T01:43:08.603 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:08.603 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":245,"btime":"2024-09-16T01:43:08:280100+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:43:08.280054+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":117,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:43:08.603 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 245 2024-09-16T01:43:08.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.604+0000 7f6d09b59640 1 -- 172.21.15.88:0/2224281663 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6cf006c2b0 msgr2=0x7f6cf006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:08.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.604+0000 7f6d09b59640 1 --2- 172.21.15.88:0/2224281663 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6cf006c2b0 0x7f6cf006e770 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4034000 tx=0x7f6cf402e000 comp rx=0 tx=0).stop 2024-09-16T01:43:08.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.604+0000 7f6d09b59640 1 -- 172.21.15.88:0/2224281663 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d040d1c80 msgr2=0x7f6d0415d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:08.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.604+0000 7f6d09b59640 1 --2- 172.21.15.88:0/2224281663 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d040d1c80 0x7f6d0415d2f0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc00b970 tx=0x7f6cfc00be30 comp rx=0 tx=0).stop 2024-09-16T01:43:08.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.604+0000 7f6d09b59640 1 -- 172.21.15.88:0/2224281663 shutdown_connections 2024-09-16T01:43:08.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.604+0000 7f6d09b59640 1 --2- 172.21.15.88:0/2224281663 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6cf006c2b0 0x7f6cf006e770 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:08.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.604+0000 7f6d09b59640 1 --2- 172.21.15.88:0/2224281663 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d040d1c80 0x7f6d0415d2f0 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:08.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.604+0000 7f6d09b59640 1 --2- 172.21.15.88:0/2224281663 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d040cf6b0 0x7f6d04149780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:08.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.604+0000 7f6d09b59640 1 --2- 172.21.15.88:0/2224281663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d040b63d0 0x7f6d0415b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:08.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.604+0000 7f6d09b59640 1 -- 172.21.15.88:0/2224281663 >> 172.21.15.88:0/2224281663 conn(0x7f6d040d1160 msgr2=0x7f6d04160180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:08.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.604+0000 7f6d09b59640 1 -- 172.21.15.88:0/2224281663 shutdown_connections 2024-09-16T01:43:08.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.604+0000 7f6d09b59640 1 -- 172.21.15.88:0/2224281663 wait complete. 2024-09-16T01:43:08.616 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:43:08.616 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:43:08.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.713+0000 7f6e999f6640 1 Processor -- start 2024-09-16T01:43:08.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.713+0000 7f6e999f6640 1 -- start start 2024-09-16T01:43:08.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.714+0000 7f6e999f6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6e940d3310 0x7f6e940d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:08.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.714+0000 7f6e999f6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e9415ea40 con 0x7f6e9415e660 2024-09-16T01:43:08.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.714+0000 7f6e999f6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e940d3870 con 0x7f6e940d1650 2024-09-16T01:43:08.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.714+0000 7f6e999f6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6e940d1ac0 con 0x7f6e940d3310 2024-09-16T01:43:08.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.714+0000 7f6e92ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6e9415e660 0x7f6e940cf730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57124/0 (socket says 172.21.15.88:57124) 2024-09-16T01:43:08.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.714+0000 7f6e92ffd640 1 -- 172.21.15.88:0/1685850158 learned_addr learned my addr 172.21.15.88:0/1685850158 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:08.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.715+0000 7f6e91ffb640 1 -- 172.21.15.88:0/1685850158 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3033347080 0 0) 0x7f6e9415ea40 con 0x7f6e9415e660 2024-09-16T01:43:08.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.715+0000 7f6e91ffb640 1 -- 172.21.15.88:0/1685850158 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e7c003520 con 0x7f6e9415e660 2024-09-16T01:43:08.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.715+0000 7f6e927fc640 1 --2- 172.21.15.88:0/1685850158 >> v2:172.21.15.154:3300/0 conn(0x7f6e940d3310 0x7f6e940d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:08.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.715+0000 7f6e91ffb640 1 -- 172.21.15.88:0/1685850158 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 715779709 0 0) 0x7f6e7c003520 con 0x7f6e9415e660 2024-09-16T01:43:08.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.715+0000 7f6e91ffb640 1 -- 172.21.15.88:0/1685850158 >> v2:172.21.15.154:3300/0 conn(0x7f6e940d3310 msgr2=0x7f6e940d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:08.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.715+0000 7f6e91ffb640 1 --2- 172.21.15.88:0/1685850158 >> v2:172.21.15.154:3300/0 conn(0x7f6e940d3310 0x7f6e940d1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:08.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.715+0000 7f6e91ffb640 1 -- 172.21.15.88:0/1685850158 >> v1:172.21.15.154:6789/0 conn(0x7f6e940d1650 legacy=0x7f6e940b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:08.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.715+0000 7f6e91ffb640 1 -- 172.21.15.88:0/1685850158 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e940d3b50 con 0x7f6e9415e660 2024-09-16T01:43:08.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.715+0000 7f6e91ffb640 1 -- 172.21.15.88:0/1685850158 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f6e84001670 con 0x7f6e9415e660 2024-09-16T01:43:08.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.715+0000 7f6e91ffb640 1 -- 172.21.15.88:0/1685850158 >> v1:172.21.15.88:6789/0 conn(0x7f6e9415e660 legacy=0x7f6e940cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:08.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.715+0000 7f6e91ffb640 1 --2- 172.21.15.88:0/1685850158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e7c0039b0 0x7f6e7c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:08.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.715+0000 7f6e91ffb640 1 --2- 172.21.15.88:0/1685850158 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e7c004800 0x7f6e7c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:08.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.715+0000 7f6e91ffb640 1 --2- 172.21.15.88:0/1685850158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e7c0051c0 0x7f6e7c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:08.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.715+0000 7f6e91ffb640 1 -- 172.21.15.88:0/1685850158 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6e940d1ac0 con 0x7f6e7c0039b0 2024-09-16T01:43:08.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.715+0000 7f6e91ffb640 1 -- 172.21.15.88:0/1685850158 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7c00c060 con 0x7f6e7c0051c0 2024-09-16T01:43:08.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.715+0000 7f6e91ffb640 1 -- 172.21.15.88:0/1685850158 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6e7c00c260 con 0x7f6e7c004800 2024-09-16T01:43:08.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.715+0000 7f6e92ffd640 1 --2- 172.21.15.88:0/1685850158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e7c0039b0 0x7f6e7c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:08.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.715+0000 7f6e927fc640 1 --2- 172.21.15.88:0/1685850158 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e7c004800 0x7f6e7c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:08.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.715+0000 7f6e937fe640 1 --2- 172.21.15.88:0/1685850158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e7c0051c0 0x7f6e7c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:08.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.715+0000 7f6e92ffd640 1 -- 172.21.15.88:0/1685850158 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e7c004800 msgr2=0x7f6e7c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:08.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.715+0000 7f6e92ffd640 1 --2- 172.21.15.88:0/1685850158 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e7c004800 0x7f6e7c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:08.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.715+0000 7f6e92ffd640 1 -- 172.21.15.88:0/1685850158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e7c0051c0 msgr2=0x7f6e7c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:08.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.715+0000 7f6e92ffd640 1 --2- 172.21.15.88:0/1685850158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e7c0051c0 0x7f6e7c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:08.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.715+0000 7f6e937fe640 1 -- 172.21.15.88:0/1685850158 reap_dead start 2024-09-16T01:43:08.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.715+0000 7f6e92ffd640 1 -- 172.21.15.88:0/1685850158 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e7c003520 con 0x7f6e7c0039b0 2024-09-16T01:43:08.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.716+0000 7f6e92ffd640 1 --2- 172.21.15.88:0/1685850158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e7c0039b0 0x7f6e7c003dd0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f6e8400c240 tx=0x7f6e7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86ce3faf82378d0d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:08.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.716+0000 7f6e91ffb640 1 -- 172.21.15.88:0/1685850158 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e84007cf0 con 0x7f6e7c0039b0 2024-09-16T01:43:08.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.716+0000 7f6e91ffb640 1 -- 172.21.15.88:0/1685850158 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e840058d0 con 0x7f6e7c0039b0 2024-09-16T01:43:08.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.716+0000 7f6e91ffb640 1 -- 172.21.15.88:0/1685850158 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e84005ca0 con 0x7f6e7c0039b0 2024-09-16T01:43:08.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.717+0000 7f6e999f6640 1 -- 172.21.15.88:0/1685850158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e7c0039b0 msgr2=0x7f6e7c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:08.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.717+0000 7f6e999f6640 1 --2- 172.21.15.88:0/1685850158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e7c0039b0 0x7f6e7c003dd0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f6e8400c240 tx=0x7f6e7c005830 comp rx=0 tx=0).stop 2024-09-16T01:43:08.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.717+0000 7f6e999f6640 1 -- 172.21.15.88:0/1685850158 shutdown_connections 2024-09-16T01:43:08.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.717+0000 7f6e999f6640 1 --2- 172.21.15.88:0/1685850158 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e7c0039b0 0x7f6e7c003dd0 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:08.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.717+0000 7f6e999f6640 1 -- 172.21.15.88:0/1685850158 >> 172.21.15.88:0/1685850158 conn(0x7f6e940b6410 msgr2=0x7f6e940b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:08.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.717+0000 7f6e999f6640 1 -- 172.21.15.88:0/1685850158 shutdown_connections 2024-09-16T01:43:08.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.717+0000 7f6e999f6640 1 -- 172.21.15.88:0/1685850158 wait complete. 2024-09-16T01:43:08.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.718+0000 7f6e999f6640 1 Processor -- start 2024-09-16T01:43:08.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.718+0000 7f6e999f6640 1 -- start start 2024-09-16T01:43:08.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.719+0000 7f6e999f6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e7c0039b0 0x7f6e9415ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:08.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.719+0000 7f6e999f6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e9415f1c0 0x7f6e940c9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:08.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.719+0000 7f6e999f6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e940ca4e0 0x7f6e940cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:08.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.719+0000 7f6e999f6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6e940d43f0 con 0x7f6e7c0039b0 2024-09-16T01:43:08.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.719+0000 7f6e999f6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6e940d4110 con 0x7f6e9415f1c0 2024-09-16T01:43:08.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.719+0000 7f6e999f6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6e940d1110 con 0x7f6e940ca4e0 2024-09-16T01:43:08.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.719+0000 7f6e92ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e7c0039b0 0x7f6e9415ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:08.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.719+0000 7f6e92ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e7c0039b0 0x7f6e9415ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55896/0 (socket says 172.21.15.88:55896) 2024-09-16T01:43:08.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.719+0000 7f6e92ffd640 1 -- 172.21.15.88:0/3165515108 learned_addr learned my addr 172.21.15.88:0/3165515108 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:08.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.719+0000 7f6e927fc640 1 --2- 172.21.15.88:0/3165515108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e9415f1c0 0x7f6e940c9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:08.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.719+0000 7f6e92ffd640 1 -- 172.21.15.88:0/3165515108 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e940ca4e0 msgr2=0x7f6e940cdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:08.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.719+0000 7f6e92ffd640 1 --2- 172.21.15.88:0/3165515108 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e940ca4e0 0x7f6e940cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:08.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.719+0000 7f6e92ffd640 1 -- 172.21.15.88:0/3165515108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e9415f1c0 msgr2=0x7f6e940c9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:08.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.719+0000 7f6e92ffd640 1 --2- 172.21.15.88:0/3165515108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e9415f1c0 0x7f6e940c9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:08.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.719+0000 7f6e92ffd640 1 -- 172.21.15.88:0/3165515108 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e940ccfa0 con 0x7f6e7c0039b0 2024-09-16T01:43:08.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.719+0000 7f6e92ffd640 1 --2- 172.21.15.88:0/3165515108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e7c0039b0 0x7f6e9415ec80 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f6e84001970 tx=0x7f6e940d3a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:08.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.720+0000 7f6e7bfff640 1 -- 172.21.15.88:0/3165515108 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e84001c30 con 0x7f6e7c0039b0 2024-09-16T01:43:08.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.720+0000 7f6e7bfff640 1 -- 172.21.15.88:0/3165515108 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e84041070 con 0x7f6e7c0039b0 2024-09-16T01:43:08.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.720+0000 7f6e999f6640 1 -- 172.21.15.88:0/3165515108 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e940cd280 con 0x7f6e7c0039b0 2024-09-16T01:43:08.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.720+0000 7f6e7bfff640 1 -- 172.21.15.88:0/3165515108 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e840049f0 con 0x7f6e7c0039b0 2024-09-16T01:43:08.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.720+0000 7f6e999f6640 1 -- 172.21.15.88:0/3165515108 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e940cd840 con 0x7f6e7c0039b0 2024-09-16T01:43:08.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.721+0000 7f6e999f6640 1 -- 172.21.15.88:0/3165515108 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e940cf730 con 0x7f6e7c0039b0 2024-09-16T01:43:08.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.721+0000 7f6e7bfff640 1 -- 172.21.15.88:0/3165515108 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6e8403b070 con 0x7f6e7c0039b0 2024-09-16T01:43:08.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.722+0000 7f6e7bfff640 1 --2- 172.21.15.88:0/3165515108 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6e8006c2b0 0x7f6e8006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:08.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.722+0000 7f6e7bfff640 1 -- 172.21.15.88:0/3165515108 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8318+0+0 (secure 0 0 0) 0x7f6e840ab540 con 0x7f6e7c0039b0 2024-09-16T01:43:08.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.722+0000 7f6e927fc640 1 --2- 172.21.15.88:0/3165515108 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6e8006c2b0 0x7f6e8006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:08.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.722+0000 7f6e927fc640 1 --2- 172.21.15.88:0/3165515108 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6e8006c2b0 0x7f6e8006e770 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f6e9415d360 tx=0x7f6e88008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:08.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.726+0000 7f6e7bfff640 1 -- 172.21.15.88:0/3165515108 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6e840792c0 con 0x7f6e7c0039b0 2024-09-16T01:43:08.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.923+0000 7f6e999f6640 1 -- 172.21.15.88:0/3165515108 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e94159710 con 0x7f6e7c0039b0 2024-09-16T01:43:08.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.924+0000 7f6e7bfff640 1 -- 172.21.15.88:0/3165515108 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 245 v245) ==== 95+0+3566 (secure 0 0 0) 0x7f6e84078bc0 con 0x7f6e7c0039b0 2024-09-16T01:43:08.925 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:08.926 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":245,"btime":"2024-09-16T01:43:08:280100+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:43:08.280054+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":117,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:43:08.926 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 245 2024-09-16T01:43:08.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.926+0000 7f6e999f6640 1 -- 172.21.15.88:0/3165515108 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6e8006c2b0 msgr2=0x7f6e8006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:08.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.926+0000 7f6e999f6640 1 --2- 172.21.15.88:0/3165515108 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6e8006c2b0 0x7f6e8006e770 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f6e9415d360 tx=0x7f6e88008040 comp rx=0 tx=0).stop 2024-09-16T01:43:08.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.926+0000 7f6e999f6640 1 -- 172.21.15.88:0/3165515108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e7c0039b0 msgr2=0x7f6e9415ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:08.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.926+0000 7f6e999f6640 1 --2- 172.21.15.88:0/3165515108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e7c0039b0 0x7f6e9415ec80 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f6e84001970 tx=0x7f6e940d3a40 comp rx=0 tx=0).stop 2024-09-16T01:43:08.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.926+0000 7f6e999f6640 1 -- 172.21.15.88:0/3165515108 shutdown_connections 2024-09-16T01:43:08.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.926+0000 7f6e999f6640 1 --2- 172.21.15.88:0/3165515108 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6e8006c2b0 0x7f6e8006e770 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:08.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.926+0000 7f6e999f6640 1 --2- 172.21.15.88:0/3165515108 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e940ca4e0 0x7f6e940cdd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:08.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.926+0000 7f6e999f6640 1 --2- 172.21.15.88:0/3165515108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e9415f1c0 0x7f6e940c9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:08.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.926+0000 7f6e999f6640 1 --2- 172.21.15.88:0/3165515108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e7c0039b0 0x7f6e9415ec80 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:08.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.926+0000 7f6e999f6640 1 -- 172.21.15.88:0/3165515108 >> 172.21.15.88:0/3165515108 conn(0x7f6e940b6410 msgr2=0x7f6e9406c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:08.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.926+0000 7f6e999f6640 1 -- 172.21.15.88:0/3165515108 shutdown_connections 2024-09-16T01:43:08.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:08.926+0000 7f6e999f6640 1 -- 172.21.15.88:0/3165515108 wait complete. 2024-09-16T01:43:08.939 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:43:08.939 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:43:09.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.035+0000 7f8fdda5e640 1 Processor -- start 2024-09-16T01:43:09.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.035+0000 7f8fdda5e640 1 -- start start 2024-09-16T01:43:09.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.036+0000 7f8fdda5e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8fd815e660 0x7f8fd80cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:09.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.036+0000 7f8fdda5e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8fd80d1820 0x7f8fd80b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:09.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.036+0000 7f8fdda5e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fd815ea40 con 0x7f8fd80d3310 2024-09-16T01:43:09.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.036+0000 7f8fdda5e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f8fd80d3870 con 0x7f8fd815e660 2024-09-16T01:43:09.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.036+0000 7f8fdda5e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f8fd80d3b50 con 0x7f8fd80d1820 2024-09-16T01:43:09.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.036+0000 7f8fd6ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8fd815e660 0x7f8fd80cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:09.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.036+0000 7f8fd6ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8fd815e660 0x7f8fd80cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55898/0 (socket says 172.21.15.88:55898) 2024-09-16T01:43:09.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.036+0000 7f8fd6ffd640 1 -- 172.21.15.88:0/107258064 learned_addr learned my addr 172.21.15.88:0/107258064 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:09.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.037+0000 7f8fd77fe640 1 --2- 172.21.15.88:0/107258064 >> v2:172.21.15.154:3300/0 conn(0x7f8fd80d1820 0x7f8fd80b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:09.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.037+0000 7f8fd6ffd640 1 -- 172.21.15.88:0/107258064 >> v2:172.21.15.154:3300/0 conn(0x7f8fd80d1820 msgr2=0x7f8fd80b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:09.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.037+0000 7f8fd6ffd640 1 --2- 172.21.15.88:0/107258064 >> v2:172.21.15.154:3300/0 conn(0x7f8fd80d1820 0x7f8fd80b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:09.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.037+0000 7f8fd6ffd640 1 -- 172.21.15.88:0/107258064 >> v1:172.21.15.154:6789/0 conn(0x7f8fd80d3310 legacy=0x7f8fd80d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:09.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.037+0000 7f8fd6ffd640 1 -- 172.21.15.88:0/107258064 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd80d3e30 con 0x7f8fd815e660 2024-09-16T01:43:09.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.037+0000 7f8fd6ffd640 1 --2- 172.21.15.88:0/107258064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fd815e660 0x7f8fd80cf730 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f8fd80d26b0 tx=0x7f8fc802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aceb0b2542a7e641 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:09.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.038+0000 7f8fd67fc640 1 -- 172.21.15.88:0/107258064 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8037070 con 0x7f8fd815e660 2024-09-16T01:43:09.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.038+0000 7f8fd67fc640 1 -- 172.21.15.88:0/107258064 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fc8032440 con 0x7f8fd815e660 2024-09-16T01:43:09.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.038+0000 7f8fd67fc640 1 -- 172.21.15.88:0/107258064 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8032810 con 0x7f8fd815e660 2024-09-16T01:43:09.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.038+0000 7f8fdda5e640 1 -- 172.21.15.88:0/107258064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fd815e660 msgr2=0x7f8fd80cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:09.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.038+0000 7f8fdda5e640 1 --2- 172.21.15.88:0/107258064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fd815e660 0x7f8fd80cf730 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f8fd80d26b0 tx=0x7f8fc802a230 comp rx=0 tx=0).stop 2024-09-16T01:43:09.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.038+0000 7f8fdda5e640 1 -- 172.21.15.88:0/107258064 shutdown_connections 2024-09-16T01:43:09.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.038+0000 7f8fdda5e640 1 --2- 172.21.15.88:0/107258064 >> v2:172.21.15.154:3300/0 conn(0x7f8fd80d1820 0x7f8fd80b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:09.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.038+0000 7f8fdda5e640 1 --2- 172.21.15.88:0/107258064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fd815e660 0x7f8fd80cf730 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:09.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.038+0000 7f8fdda5e640 1 -- 172.21.15.88:0/107258064 >> 172.21.15.88:0/107258064 conn(0x7f8fd80b6410 msgr2=0x7f8fd80b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:09.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.038+0000 7f8fdda5e640 1 -- 172.21.15.88:0/107258064 shutdown_connections 2024-09-16T01:43:09.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.038+0000 7f8fdda5e640 1 -- 172.21.15.88:0/107258064 wait complete. 2024-09-16T01:43:09.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.039+0000 7f8fdda5e640 1 Processor -- start 2024-09-16T01:43:09.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.039+0000 7f8fdda5e640 1 -- start start 2024-09-16T01:43:09.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.040+0000 7f8fdda5e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8fd80d1820 0x7f8fd80cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:09.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.040+0000 7f8fdda5e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fd80d3310 0x7f8fd81e85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:09.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.040+0000 7f8fdda5e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fd815e660 0x7f8fd8150af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:09.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.040+0000 7f8fdda5e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd80cfc70 con 0x7f8fd815e660 2024-09-16T01:43:09.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.040+0000 7f8fdda5e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd80d43f0 con 0x7f8fd80d3310 2024-09-16T01:43:09.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.040+0000 7f8fdda5e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8fd80b80b0 con 0x7f8fd80d1820 2024-09-16T01:43:09.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.040+0000 7f8fd77fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fd815e660 0x7f8fd8150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:09.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.040+0000 7f8fd77fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fd815e660 0x7f8fd8150af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55902/0 (socket says 172.21.15.88:55902) 2024-09-16T01:43:09.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.040+0000 7f8fd77fe640 1 -- 172.21.15.88:0/394849891 learned_addr learned my addr 172.21.15.88:0/394849891 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:09.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.040+0000 7f8fd6ffd640 1 --2- 172.21.15.88:0/394849891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8fd80d1820 0x7f8fd80cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:09.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.040+0000 7f8fce7fc640 1 --2- 172.21.15.88:0/394849891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fd80d3310 0x7f8fd81e85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:09.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.041+0000 7f8fd77fe640 1 -- 172.21.15.88:0/394849891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8fd80d1820 msgr2=0x7f8fd80cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:09.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.041+0000 7f8fd77fe640 1 --2- 172.21.15.88:0/394849891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8fd80d1820 0x7f8fd80cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:09.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.041+0000 7f8fd77fe640 1 -- 172.21.15.88:0/394849891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fd80d3310 msgr2=0x7f8fd81e85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:09.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.041+0000 7f8fd77fe640 1 --2- 172.21.15.88:0/394849891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fd80d3310 0x7f8fd81e85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:09.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.041+0000 7f8fd77fe640 1 -- 172.21.15.88:0/394849891 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd81e8d60 con 0x7f8fd815e660 2024-09-16T01:43:09.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.041+0000 7f8fd6ffd640 1 --2- 172.21.15.88:0/394849891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8fd80d1820 0x7f8fd80cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:09.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.041+0000 7f8fd77fe640 1 --2- 172.21.15.88:0/394849891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fd815e660 0x7f8fd8150af0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0007e70 tx=0x7f8fd000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:09.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.041+0000 7f8fd4ff9640 1 -- 172.21.15.88:0/394849891 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0014620 con 0x7f8fd815e660 2024-09-16T01:43:09.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.041+0000 7f8fd4ff9640 1 -- 172.21.15.88:0/394849891 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fd0019070 con 0x7f8fd815e660 2024-09-16T01:43:09.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.041+0000 7f8fdda5e640 1 -- 172.21.15.88:0/394849891 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fd81e9040 con 0x7f8fd815e660 2024-09-16T01:43:09.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.041+0000 7f8fd4ff9640 1 -- 172.21.15.88:0/394849891 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0014a20 con 0x7f8fd815e660 2024-09-16T01:43:09.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.041+0000 7f8fdda5e640 1 -- 172.21.15.88:0/394849891 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fd8154c80 con 0x7f8fd815e660 2024-09-16T01:43:09.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.042+0000 7f8fdda5e640 1 -- 172.21.15.88:0/394849891 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fd8154e80 con 0x7f8fd815e660 2024-09-16T01:43:09.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.042+0000 7f8fd4ff9640 1 -- 172.21.15.88:0/394849891 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8fd0014bc0 con 0x7f8fd815e660 2024-09-16T01:43:09.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.043+0000 7f8fd4ff9640 1 --2- 172.21.15.88:0/394849891 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8fb806c2b0 0x7f8fb806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:09.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.043+0000 7f8fd4ff9640 1 -- 172.21.15.88:0/394849891 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8318+0+0 (secure 0 0 0) 0x7f8fd008c7b0 con 0x7f8fd815e660 2024-09-16T01:43:09.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.043+0000 7f8fd6ffd640 1 --2- 172.21.15.88:0/394849891 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8fb806c2b0 0x7f8fb806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:09.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.043+0000 7f8fd6ffd640 1 --2- 172.21.15.88:0/394849891 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8fb806c2b0 0x7f8fb806e770 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8034000 tx=0x7f8fc802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:09.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.047+0000 7f8fd4ff9640 1 -- 172.21.15.88:0/394849891 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8fd8154e80 con 0x7f8fd815e660 2024-09-16T01:43:09.132 INFO:tasks.ceph.mds.c.smithi088.stderr: -47> 2024-09-16T01:43:04.378+0000 7f268a7c6640 -1 log_channel(cluster) log [ERR] : error decoding table object 'mds0_inotable': End of buffer [buffer:2] 2024-09-16T01:43:09.155 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:43:09.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.251+0000 7f8fdda5e640 1 -- 172.21.15.88:0/394849891 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8fd80b7b70 con 0x7f8fd815e660 2024-09-16T01:43:09.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.251+0000 7f8fd4ff9640 1 -- 172.21.15.88:0/394849891 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 245 v245) ==== 95+0+3566 (secure 0 0 0) 0x7f8fd005a4a0 con 0x7f8fd815e660 2024-09-16T01:43:09.253 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:09.253 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":245,"btime":"2024-09-16T01:43:08:280100+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:43:08.280054+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":117,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:43:09.253 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 245 2024-09-16T01:43:09.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.253+0000 7f8fdda5e640 1 -- 172.21.15.88:0/394849891 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8fb806c2b0 msgr2=0x7f8fb806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:09.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.253+0000 7f8fdda5e640 1 --2- 172.21.15.88:0/394849891 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8fb806c2b0 0x7f8fb806e770 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8034000 tx=0x7f8fc802e000 comp rx=0 tx=0).stop 2024-09-16T01:43:09.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.253+0000 7f8fdda5e640 1 -- 172.21.15.88:0/394849891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fd815e660 msgr2=0x7f8fd8150af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:09.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.253+0000 7f8fdda5e640 1 --2- 172.21.15.88:0/394849891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fd815e660 0x7f8fd8150af0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0007e70 tx=0x7f8fd000e4b0 comp rx=0 tx=0).stop 2024-09-16T01:43:09.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.253+0000 7f8fdda5e640 1 -- 172.21.15.88:0/394849891 shutdown_connections 2024-09-16T01:43:09.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.253+0000 7f8fdda5e640 1 --2- 172.21.15.88:0/394849891 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8fb806c2b0 0x7f8fb806e770 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:09.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.253+0000 7f8fdda5e640 1 --2- 172.21.15.88:0/394849891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8fd815e660 0x7f8fd8150af0 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:09.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.254+0000 7f8fdda5e640 1 --2- 172.21.15.88:0/394849891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fd80d3310 0x7f8fd81e85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:09.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.254+0000 7f8fdda5e640 1 --2- 172.21.15.88:0/394849891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8fd80d1820 0x7f8fd80cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:09.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.254+0000 7f8fdda5e640 1 -- 172.21.15.88:0/394849891 >> 172.21.15.88:0/394849891 conn(0x7f8fd80b6410 msgr2=0x7f8fd80b9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:09.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.254+0000 7f8fdda5e640 1 -- 172.21.15.88:0/394849891 shutdown_connections 2024-09-16T01:43:09.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.254+0000 7f8fdda5e640 1 -- 172.21.15.88:0/394849891 wait complete. 2024-09-16T01:43:09.266 INFO:tasks.cephfs.test_damage:Result: Mutation 'Truncate 400.00000000' led to DAMAGED state 2024-09-16T01:43:09.266 INFO:tasks.cephfs.test_damage:Applying mutation 'Truncate 100.00000000.inode' 2024-09-16T01:43:09.266 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:43:09.266 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:43:09.282 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:43:09.282 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:43:09.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:09.282 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:43:09.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:43:09.337 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:43:09.338 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:43:09.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.477+0000 7f7432642640 1 Processor -- start 2024-09-16T01:43:09.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.477+0000 7f7432642640 1 -- start start 2024-09-16T01:43:09.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.478+0000 7f7432642640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f742c0b7450 0x7f742c06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:09.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.478+0000 7f7432642640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f742c06e990 con 0x7f742c0b54d0 2024-09-16T01:43:09.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.478+0000 7f7432642640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f742c06ec70 con 0x7f742c15e6f0 2024-09-16T01:43:09.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.478+0000 7f7432642640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f742c15ec70 con 0x7f742c0b7450 2024-09-16T01:43:09.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.478+0000 7f742b7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f742c0b54d0 0x7f742c070830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:57138/0 (socket says 172.21.15.88:57138) 2024-09-16T01:43:09.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.478+0000 7f742b7fe640 1 -- 172.21.15.88:0/1237409683 learned_addr learned my addr 172.21.15.88:0/1237409683 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:09.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f7430bdf640 1 --2- 172.21.15.88:0/1237409683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f742c0b7450 0x7f742c06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:09.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f742affd640 1 -- 172.21.15.88:0/1237409683 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 450698473 0 0) 0x7f742c06e990 con 0x7f742c0b54d0 2024-09-16T01:43:09.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f742affd640 1 -- 172.21.15.88:0/1237409683 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7414003520 con 0x7f742c0b54d0 2024-09-16T01:43:09.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f742affd640 1 -- 172.21.15.88:0/1237409683 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2099162492 0 0) 0x7f7414003520 con 0x7f742c0b54d0 2024-09-16T01:43:09.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f742affd640 1 -- 172.21.15.88:0/1237409683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f742c0b7450 msgr2=0x7f742c06fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:09.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f742affd640 1 --2- 172.21.15.88:0/1237409683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f742c0b7450 0x7f742c06fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:09.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f742affd640 1 -- 172.21.15.88:0/1237409683 >> v1:172.21.15.154:6789/0 conn(0x7f742c15e6f0 legacy=0x7f742c0b6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:09.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f742affd640 1 -- 172.21.15.88:0/1237409683 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f742c15ef50 con 0x7f742c0b54d0 2024-09-16T01:43:09.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f742affd640 1 -- 172.21.15.88:0/1237409683 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f7420001670 con 0x7f742c0b54d0 2024-09-16T01:43:09.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f742affd640 1 -- 172.21.15.88:0/1237409683 >> v1:172.21.15.88:6789/0 conn(0x7f742c0b54d0 legacy=0x7f742c070830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:09.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f742affd640 1 --2- 172.21.15.88:0/1237409683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74140039e0 0x7f7414003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:09.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f742affd640 1 --2- 172.21.15.88:0/1237409683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f742c0b7450 0x7f7414004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:09.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f742affd640 1 --2- 172.21.15.88:0/1237409683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7414004d40 0x7f74140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:09.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f742affd640 1 -- 172.21.15.88:0/1237409683 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f742c15ec70 con 0x7f7414004d40 2024-09-16T01:43:09.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f742affd640 1 -- 172.21.15.88:0/1237409683 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f741400b9f0 con 0x7f74140039e0 2024-09-16T01:43:09.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f742affd640 1 -- 172.21.15.88:0/1237409683 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f741400bbf0 con 0x7f742c0b7450 2024-09-16T01:43:09.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f7430bdf640 1 --2- 172.21.15.88:0/1237409683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7414004d40 0x7f74140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:09.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f742bfff640 1 --2- 172.21.15.88:0/1237409683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74140039e0 0x7f7414003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:09.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f742b7fe640 1 --2- 172.21.15.88:0/1237409683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f742c0b7450 0x7f7414004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:09.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f7430bdf640 1 -- 172.21.15.88:0/1237409683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f742c0b7450 msgr2=0x7f7414004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:09.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f7430bdf640 1 --2- 172.21.15.88:0/1237409683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f742c0b7450 0x7f7414004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:09.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f7430bdf640 1 -- 172.21.15.88:0/1237409683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74140039e0 msgr2=0x7f7414003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:09.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f7430bdf640 1 --2- 172.21.15.88:0/1237409683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74140039e0 0x7f7414003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:09.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f7430bdf640 1 -- 172.21.15.88:0/1237409683 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7414003520 con 0x7f7414004d40 2024-09-16T01:43:09.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f742bfff640 1 --2- 172.21.15.88:0/1237409683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74140039e0 0x7f7414003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:09.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.479+0000 7f742b7fe640 1 --2- 172.21.15.88:0/1237409683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f742c0b7450 0x7f7414004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:09.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.480+0000 7f7430bdf640 1 --2- 172.21.15.88:0/1237409683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7414004d40 0x7f74140051c0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f7424009f90 tx=0x7f7414005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e25b973ccce1674f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:09.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.480+0000 7f742affd640 1 -- 172.21.15.88:0/1237409683 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7424009c10 con 0x7f7414004d40 2024-09-16T01:43:09.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.480+0000 7f742affd640 1 -- 172.21.15.88:0/1237409683 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7424036070 con 0x7f7414004d40 2024-09-16T01:43:09.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.480+0000 7f742affd640 1 -- 172.21.15.88:0/1237409683 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7424031810 con 0x7f7414004d40 2024-09-16T01:43:09.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.481+0000 7f7432642640 1 -- 172.21.15.88:0/1237409683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7414004d40 msgr2=0x7f74140051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:09.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.481+0000 7f7432642640 1 --2- 172.21.15.88:0/1237409683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7414004d40 0x7f74140051c0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f7424009f90 tx=0x7f7414005830 comp rx=0 tx=0).stop 2024-09-16T01:43:09.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.481+0000 7f7430bdf640 1 -- 172.21.15.88:0/1237409683 reap_dead start 2024-09-16T01:43:09.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.481+0000 7f7432642640 1 -- 172.21.15.88:0/1237409683 shutdown_connections 2024-09-16T01:43:09.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.481+0000 7f7432642640 1 -- 172.21.15.88:0/1237409683 >> 172.21.15.88:0/1237409683 conn(0x7f742c0d2430 msgr2=0x7f742c0d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:09.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.481+0000 7f7432642640 1 -- 172.21.15.88:0/1237409683 shutdown_connections 2024-09-16T01:43:09.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.481+0000 7f7432642640 1 -- 172.21.15.88:0/1237409683 wait complete. 2024-09-16T01:43:09.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.482+0000 7f7432642640 1 Processor -- start 2024-09-16T01:43:09.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.482+0000 7f7432642640 1 -- start start 2024-09-16T01:43:09.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.483+0000 7f7432642640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7414004d40 0x7f742c152630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:09.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.483+0000 7f7432642640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f742c1533b0 0x7f742c1544c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:09.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.483+0000 7f7432642640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f742c154a00 0x7f742c15d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:09.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.483+0000 7f7432642640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f742c06e730 con 0x7f742c1533b0 2024-09-16T01:43:09.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.483+0000 7f7432642640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f742c06e5b0 con 0x7f7414004d40 2024-09-16T01:43:09.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.483+0000 7f7432642640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f742c06fd00 con 0x7f742c154a00 2024-09-16T01:43:09.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.483+0000 7f742b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f742c1533b0 0x7f742c1544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:09.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.483+0000 7f742b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f742c1533b0 0x7f742c1544c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:55924/0 (socket says 172.21.15.88:55924) 2024-09-16T01:43:09.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.483+0000 7f742b7fe640 1 -- 172.21.15.88:0/2496180515 learned_addr learned my addr 172.21.15.88:0/2496180515 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:09.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.483+0000 7f742bfff640 1 --2- 172.21.15.88:0/2496180515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7414004d40 0x7f742c152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:09.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.483+0000 7f7430bdf640 1 --2- 172.21.15.88:0/2496180515 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f742c154a00 0x7f742c15d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:09.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.483+0000 7f742b7fe640 1 -- 172.21.15.88:0/2496180515 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f742c154a00 msgr2=0x7f742c15d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:09.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.483+0000 7f742b7fe640 1 --2- 172.21.15.88:0/2496180515 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f742c154a00 0x7f742c15d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:09.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.483+0000 7f742b7fe640 1 -- 172.21.15.88:0/2496180515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7414004d40 msgr2=0x7f742c152630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:09.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.483+0000 7f742b7fe640 1 --2- 172.21.15.88:0/2496180515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7414004d40 0x7f742c152630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:09.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.483+0000 7f742b7fe640 1 -- 172.21.15.88:0/2496180515 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f742c1f7970 con 0x7f742c1533b0 2024-09-16T01:43:09.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.483+0000 7f742b7fe640 1 --2- 172.21.15.88:0/2496180515 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f742c1533b0 0x7f742c1544c0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f7420001b00 tx=0x7f742000ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:09.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.483+0000 7f7428ff9640 1 -- 172.21.15.88:0/2496180515 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f742000ee60 con 0x7f742c1533b0 2024-09-16T01:43:09.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.483+0000 7f7428ff9640 1 -- 172.21.15.88:0/2496180515 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7420018070 con 0x7f742c1533b0 2024-09-16T01:43:09.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.483+0000 7f7428ff9640 1 -- 172.21.15.88:0/2496180515 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f742000cac0 con 0x7f742c1533b0 2024-09-16T01:43:09.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.483+0000 7f7432642640 1 -- 172.21.15.88:0/2496180515 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f742c1f7c10 con 0x7f742c1533b0 2024-09-16T01:43:09.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.484+0000 7f7432642640 1 -- 172.21.15.88:0/2496180515 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f742c1f86f0 con 0x7f742c1533b0 2024-09-16T01:43:09.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.485+0000 7f7432642640 1 -- 172.21.15.88:0/2496180515 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f742c06c6c0 con 0x7f742c1533b0 2024-09-16T01:43:09.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.485+0000 7f7428ff9640 1 -- 172.21.15.88:0/2496180515 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f742000cc60 con 0x7f742c1533b0 2024-09-16T01:43:09.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.485+0000 7f7428ff9640 1 --2- 172.21.15.88:0/2496180515 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f741806c2b0 0x7f741806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:09.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.486+0000 7f742bfff640 1 --2- 172.21.15.88:0/2496180515 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f741806c2b0 0x7f741806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:09.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.486+0000 7f7428ff9640 1 -- 172.21.15.88:0/2496180515 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8318+0+0 (secure 0 0 0) 0x7f742008bd70 con 0x7f742c1533b0 2024-09-16T01:43:09.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.486+0000 7f742bfff640 1 --2- 172.21.15.88:0/2496180515 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f741806c2b0 0x7f741806e770 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f742c1f7830 tx=0x7f741c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:09.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.490+0000 7f7428ff9640 1 -- 172.21.15.88:0/2496180515 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7420059a60 con 0x7f742c1533b0 2024-09-16T01:43:09.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:09.694+0000 7f7432642640 1 -- 172.21.15.88:0/2496180515 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f742c159990 con 0x7f742c1533b0 2024-09-16T01:43:10.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.415+0000 7f7428ff9640 1 -- 172.21.15.88:0/2496180515 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v247) ==== 161+0+0 (secure 0 0 0) 0x7f7420059360 con 0x7f742c1533b0 2024-09-16T01:43:10.417 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:43:10.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.418+0000 7f7432642640 1 -- 172.21.15.88:0/2496180515 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f741806c2b0 msgr2=0x7f741806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:10.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.418+0000 7f7432642640 1 --2- 172.21.15.88:0/2496180515 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f741806c2b0 0x7f741806e770 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f742c1f7830 tx=0x7f741c009040 comp rx=0 tx=0).stop 2024-09-16T01:43:10.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.418+0000 7f7432642640 1 -- 172.21.15.88:0/2496180515 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f742c1533b0 msgr2=0x7f742c1544c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:10.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.418+0000 7f7432642640 1 --2- 172.21.15.88:0/2496180515 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f742c1533b0 0x7f742c1544c0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f7420001b00 tx=0x7f742000ec10 comp rx=0 tx=0).stop 2024-09-16T01:43:10.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.418+0000 7f7432642640 1 -- 172.21.15.88:0/2496180515 shutdown_connections 2024-09-16T01:43:10.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.418+0000 7f7432642640 1 --2- 172.21.15.88:0/2496180515 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f741806c2b0 0x7f741806e770 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:10.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.418+0000 7f7432642640 1 --2- 172.21.15.88:0/2496180515 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f742c154a00 0x7f742c15d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:10.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.418+0000 7f7432642640 1 --2- 172.21.15.88:0/2496180515 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f742c1533b0 0x7f742c1544c0 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:10.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.418+0000 7f7432642640 1 --2- 172.21.15.88:0/2496180515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7414004d40 0x7f742c152630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:10.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.418+0000 7f7432642640 1 -- 172.21.15.88:0/2496180515 >> 172.21.15.88:0/2496180515 conn(0x7f742c0d2430 msgr2=0x7f742c0b3990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:10.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.418+0000 7f7432642640 1 -- 172.21.15.88:0/2496180515 shutdown_connections 2024-09-16T01:43:10.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.418+0000 7f7432642640 1 -- 172.21.15.88:0/2496180515 wait complete. 2024-09-16T01:43:10.433 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:43:10.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.538+0000 7f2674f2c640 1 Processor -- start 2024-09-16T01:43:10.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.538+0000 7f2674f2c640 1 -- start start 2024-09-16T01:43:10.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.539+0000 7f2674f2c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26700b6290 0x7f2670071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:10.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.539+0000 7f2674f2c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2670072530 0x7f267015bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:10.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.539+0000 7f2674f2c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26700ccff0 con 0x7f267015b480 2024-09-16T01:43:10.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.539+0000 7f2674f2c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f26700cd2d0 con 0x7f2670072530 2024-09-16T01:43:10.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.539+0000 7f2674f2c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f26700cd5b0 con 0x7f26700b6290 2024-09-16T01:43:10.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.539+0000 7f266e59c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f267015b480 0x7f26700b5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59354/0 (socket says 172.21.15.88:59354) 2024-09-16T01:43:10.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.539+0000 7f266e59c640 1 -- 172.21.15.88:0/295594642 learned_addr learned my addr 172.21.15.88:0/295594642 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:10.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.539+0000 7f266ed9d640 1 --2- 172.21.15.88:0/295594642 >> v2:172.21.15.154:3300/0 conn(0x7f2670072530 0x7f267015bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:10.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.539+0000 7f266dd9b640 1 -- 172.21.15.88:0/295594642 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3121288672 0 0) 0x7f26700ccff0 con 0x7f267015b480 2024-09-16T01:43:10.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.539+0000 7f2665d9b640 1 --2- 172.21.15.88:0/295594642 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26700b6290 0x7f2670071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:10.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.539+0000 7f266dd9b640 1 -- 172.21.15.88:0/295594642 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f265c005730 con 0x7f267015b480 2024-09-16T01:43:10.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.540+0000 7f266dd9b640 1 -- 172.21.15.88:0/295594642 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2045473830 0 0) 0x7f265c005730 con 0x7f267015b480 2024-09-16T01:43:10.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.540+0000 7f266dd9b640 1 -- 172.21.15.88:0/295594642 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26700b6290 msgr2=0x7f2670071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:10.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.540+0000 7f266dd9b640 1 --2- 172.21.15.88:0/295594642 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f26700b6290 0x7f2670071ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:10.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.540+0000 7f266dd9b640 1 -- 172.21.15.88:0/295594642 >> v2:172.21.15.154:3300/0 conn(0x7f2670072530 msgr2=0x7f267015bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:10.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.540+0000 7f266dd9b640 1 --2- 172.21.15.88:0/295594642 >> v2:172.21.15.154:3300/0 conn(0x7f2670072530 0x7f267015bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:10.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.540+0000 7f266dd9b640 1 -- 172.21.15.88:0/295594642 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26700cd890 con 0x7f267015b480 2024-09-16T01:43:10.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.540+0000 7f266dd9b640 1 -- 172.21.15.88:0/295594642 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f2660001670 con 0x7f267015b480 2024-09-16T01:43:10.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.540+0000 7f266dd9b640 1 -- 172.21.15.88:0/295594642 >> v1:172.21.15.88:6789/0 conn(0x7f267015b480 legacy=0x7f26700b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:10.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.540+0000 7f266dd9b640 1 --2- 172.21.15.88:0/295594642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f265c005bc0 0x7f265c005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:10.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.540+0000 7f266dd9b640 1 --2- 172.21.15.88:0/295594642 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f265c006a10 0x7f265c006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:10.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.540+0000 7f266dd9b640 1 --2- 172.21.15.88:0/295594642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f265c007390 0x7f265c00dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:10.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.540+0000 7f266dd9b640 1 -- 172.21.15.88:0/295594642 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f26700cd2d0 con 0x7f265c005bc0 2024-09-16T01:43:10.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.540+0000 7f266dd9b640 1 -- 172.21.15.88:0/295594642 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f26700cd5b0 con 0x7f265c007390 2024-09-16T01:43:10.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.540+0000 7f266dd9b640 1 -- 172.21.15.88:0/295594642 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f265c00e110 con 0x7f265c006a10 2024-09-16T01:43:10.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.540+0000 7f266e59c640 1 --2- 172.21.15.88:0/295594642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f265c005bc0 0x7f265c005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:10.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.540+0000 7f2665d9b640 1 --2- 172.21.15.88:0/295594642 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f265c006a10 0x7f265c006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:10.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.540+0000 7f266ed9d640 1 --2- 172.21.15.88:0/295594642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f265c007390 0x7f265c00dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:10.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.540+0000 7f266e59c640 1 -- 172.21.15.88:0/295594642 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f265c006a10 msgr2=0x7f265c006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:10.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.540+0000 7f266e59c640 1 --2- 172.21.15.88:0/295594642 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f265c006a10 0x7f265c006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:10.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.540+0000 7f266e59c640 1 -- 172.21.15.88:0/295594642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f265c007390 msgr2=0x7f265c00dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:10.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.540+0000 7f266e59c640 1 --2- 172.21.15.88:0/295594642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f265c007390 0x7f265c00dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:10.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.540+0000 7f266e59c640 1 -- 172.21.15.88:0/295594642 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f265c005730 con 0x7f265c005bc0 2024-09-16T01:43:10.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.540+0000 7f266ed9d640 1 --2- 172.21.15.88:0/295594642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f265c007390 0x7f265c00dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:10.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.541+0000 7f266e59c640 1 --2- 172.21.15.88:0/295594642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f265c005bc0 0x7f265c005fe0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f2660007fd0 tx=0x7f265c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=502bb58d8a8b7df server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:10.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.541+0000 7f266dd9b640 1 -- 172.21.15.88:0/295594642 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2660007d50 con 0x7f265c005bc0 2024-09-16T01:43:10.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.541+0000 7f266dd9b640 1 -- 172.21.15.88:0/295594642 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2660005ad0 con 0x7f265c005bc0 2024-09-16T01:43:10.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.541+0000 7f266dd9b640 1 -- 172.21.15.88:0/295594642 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2660007070 con 0x7f265c005bc0 2024-09-16T01:43:10.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.541+0000 7f2674f2c640 1 -- 172.21.15.88:0/295594642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f265c005bc0 msgr2=0x7f265c005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:10.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.541+0000 7f2674f2c640 1 --2- 172.21.15.88:0/295594642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f265c005bc0 0x7f265c005fe0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f2660007fd0 tx=0x7f265c007a40 comp rx=0 tx=0).stop 2024-09-16T01:43:10.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.541+0000 7f266ed9d640 1 -- 172.21.15.88:0/295594642 reap_dead start 2024-09-16T01:43:10.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.541+0000 7f2674f2c640 1 -- 172.21.15.88:0/295594642 shutdown_connections 2024-09-16T01:43:10.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.541+0000 7f2674f2c640 1 -- 172.21.15.88:0/295594642 >> 172.21.15.88:0/295594642 conn(0x7f267006d0f0 msgr2=0x7f267006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:10.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.541+0000 7f2674f2c640 1 -- 172.21.15.88:0/295594642 shutdown_connections 2024-09-16T01:43:10.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.541+0000 7f2674f2c640 1 -- 172.21.15.88:0/295594642 wait complete. 2024-09-16T01:43:10.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.543+0000 7f2674f2c640 1 Processor -- start 2024-09-16T01:43:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.543+0000 7f2674f2c640 1 -- start start 2024-09-16T01:43:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.544+0000 7f2674f2c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f265c005bc0 0x7f267014cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.544+0000 7f2674f2c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2670158c20 0x7f26701590a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.544+0000 7f2674f2c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2670152ae0 0x7f2670152f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.544+0000 7f2674f2c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f267015b9e0 con 0x7f265c005bc0 2024-09-16T01:43:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.544+0000 7f2674f2c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f267015b860 con 0x7f2670152ae0 2024-09-16T01:43:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.544+0000 7f2674f2c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2670071ff0 con 0x7f2670158c20 2024-09-16T01:43:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.544+0000 7f266e59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f265c005bc0 0x7f267014cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.544+0000 7f266e59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f265c005bc0 0x7f267014cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58488/0 (socket says 172.21.15.88:58488) 2024-09-16T01:43:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.544+0000 7f266e59c640 1 -- 172.21.15.88:0/4047290090 learned_addr learned my addr 172.21.15.88:0/4047290090 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.544+0000 7f2665d9b640 1 --2- 172.21.15.88:0/4047290090 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2670158c20 0x7f26701590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.544+0000 7f266ed9d640 1 --2- 172.21.15.88:0/4047290090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2670152ae0 0x7f2670152f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:10.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.544+0000 7f266e59c640 1 -- 172.21.15.88:0/4047290090 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2670158c20 msgr2=0x7f26701590a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:10.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.544+0000 7f266e59c640 1 --2- 172.21.15.88:0/4047290090 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2670158c20 0x7f26701590a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:10.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.544+0000 7f266e59c640 1 -- 172.21.15.88:0/4047290090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2670152ae0 msgr2=0x7f2670152f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:10.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.544+0000 7f266e59c640 1 --2- 172.21.15.88:0/4047290090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2670152ae0 0x7f2670152f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:10.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.544+0000 7f266e59c640 1 -- 172.21.15.88:0/4047290090 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26701521b0 con 0x7f265c005bc0 2024-09-16T01:43:10.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.544+0000 7f266ed9d640 1 --2- 172.21.15.88:0/4047290090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2670152ae0 0x7f2670152f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:10.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.544+0000 7f266e59c640 1 --2- 172.21.15.88:0/4047290090 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f265c005bc0 0x7f267014cba0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f266000c300 tx=0x7f26700733f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:10.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.544+0000 7f26677fe640 1 -- 172.21.15.88:0/4047290090 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2660042020 con 0x7f265c005bc0 2024-09-16T01:43:10.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.544+0000 7f26677fe640 1 -- 172.21.15.88:0/4047290090 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2660004bf0 con 0x7f265c005bc0 2024-09-16T01:43:10.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.544+0000 7f2674f2c640 1 -- 172.21.15.88:0/4047290090 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26701537d0 con 0x7f265c005bc0 2024-09-16T01:43:10.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.544+0000 7f26677fe640 1 -- 172.21.15.88:0/4047290090 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2660007070 con 0x7f265c005bc0 2024-09-16T01:43:10.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.545+0000 7f2674f2c640 1 -- 172.21.15.88:0/4047290090 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26701516d0 con 0x7f265c005bc0 2024-09-16T01:43:10.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.545+0000 7f2674f2c640 1 -- 172.21.15.88:0/4047290090 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26700b5b80 con 0x7f265c005bc0 2024-09-16T01:43:10.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.546+0000 7f26677fe640 1 -- 172.21.15.88:0/4047290090 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2660045020 con 0x7f265c005bc0 2024-09-16T01:43:10.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.546+0000 7f26677fe640 1 --2- 172.21.15.88:0/4047290090 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f265006c2b0 0x7f265006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:10.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.547+0000 7f26677fe640 1 -- 172.21.15.88:0/4047290090 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8318+0+0 (secure 0 0 0) 0x7f26600ac710 con 0x7f265c005bc0 2024-09-16T01:43:10.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.547+0000 7f2665d9b640 1 --2- 172.21.15.88:0/4047290090 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f265006c2b0 0x7f265006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.547+0000 7f2665d9b640 1 --2- 172.21.15.88:0/4047290090 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f265006c2b0 0x7f265006e770 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f26700c5650 tx=0x7f2658008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:10.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.551+0000 7f26677fe640 1 -- 172.21.15.88:0/4047290090 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f266007a520 con 0x7f265c005bc0 2024-09-16T01:43:10.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:10.782+0000 7f2674f2c640 1 -- 172.21.15.88:0/4047290090 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f26700b43c0 con 0x7f265c005bc0 2024-09-16T01:43:11.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.423+0000 7f26677fe640 1 -- 172.21.15.88:0/4047290090 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v248) ==== 101+0+0 (secure 0 0 0) 0x7f2660079e20 con 0x7f265c005bc0 2024-09-16T01:43:11.425 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:43:11.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.427+0000 7f2674f2c640 1 -- 172.21.15.88:0/4047290090 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f265006c2b0 msgr2=0x7f265006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:11.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.427+0000 7f2674f2c640 1 --2- 172.21.15.88:0/4047290090 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f265006c2b0 0x7f265006e770 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f26700c5650 tx=0x7f2658008040 comp rx=0 tx=0).stop 2024-09-16T01:43:11.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.427+0000 7f2674f2c640 1 -- 172.21.15.88:0/4047290090 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f265c005bc0 msgr2=0x7f267014cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:11.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.427+0000 7f2674f2c640 1 --2- 172.21.15.88:0/4047290090 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f265c005bc0 0x7f267014cba0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f266000c300 tx=0x7f26700733f0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.427+0000 7f2674f2c640 1 -- 172.21.15.88:0/4047290090 shutdown_connections 2024-09-16T01:43:11.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.427+0000 7f2674f2c640 1 --2- 172.21.15.88:0/4047290090 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f265006c2b0 0x7f265006e770 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.427+0000 7f2674f2c640 1 --2- 172.21.15.88:0/4047290090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2670152ae0 0x7f2670152f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.427+0000 7f2674f2c640 1 --2- 172.21.15.88:0/4047290090 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2670158c20 0x7f26701590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.427+0000 7f2674f2c640 1 --2- 172.21.15.88:0/4047290090 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f265c005bc0 0x7f267014cba0 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.427+0000 7f2674f2c640 1 -- 172.21.15.88:0/4047290090 >> 172.21.15.88:0/4047290090 conn(0x7f267006d0f0 msgr2=0x7f26700b4bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:11.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.427+0000 7f2674f2c640 1 -- 172.21.15.88:0/4047290090 shutdown_connections 2024-09-16T01:43:11.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.427+0000 7f2674f2c640 1 -- 172.21.15.88:0/4047290090 wait complete. 2024-09-16T01:43:11.441 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:43:11.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.525+0000 7fbbac720f40 1 Processor -- start 2024-09-16T01:43:11.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.525+0000 7fbbac720f40 1 -- start start 2024-09-16T01:43:11.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.526+0000 7fbbac720f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe372858d0 0x55fe37256460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.526+0000 7fbbac720f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fe3732ad20 0x55fe37281f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.526+0000 7fbbac720f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe3732b100 0x55fe37282480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.526+0000 7fbbac720f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55fe37286220 con 0x55fe372858d0 2024-09-16T01:43:11.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.526+0000 7fbbac720f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fe3729ebd0 con 0x55fe3732b100 2024-09-16T01:43:11.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.526+0000 7fbbac720f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55fe3729ee70 con 0x55fe3732ad20 2024-09-16T01:43:11.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.526+0000 7fbba97af640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe372858d0 0x55fe37256460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.526+0000 7fbba97af640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe372858d0 0x55fe37256460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58502/0 (socket says 172.21.15.88:58502) 2024-09-16T01:43:11.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.526+0000 7fbba97af640 1 -- 172.21.15.88:0/3011158607 learned_addr learned my addr 172.21.15.88:0/3011158607 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:11.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.526+0000 7fbba0fae640 1 --2- 172.21.15.88:0/3011158607 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fe3732ad20 0x55fe37281f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.526+0000 7fbba9fb0640 1 --2- 172.21.15.88:0/3011158607 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe3732b100 0x55fe37282480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.526+0000 7fbba97af640 1 -- 172.21.15.88:0/3011158607 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fe3732ad20 msgr2=0x55fe37281f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:11.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.526+0000 7fbba97af640 1 --2- 172.21.15.88:0/3011158607 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fe3732ad20 0x55fe37281f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.526+0000 7fbba97af640 1 -- 172.21.15.88:0/3011158607 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe3732b100 msgr2=0x55fe37282480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:11.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.526+0000 7fbba97af640 1 --2- 172.21.15.88:0/3011158607 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe3732b100 0x55fe37282480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.526+0000 7fbba97af640 1 -- 172.21.15.88:0/3011158607 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe37181cb0 con 0x55fe372858d0 2024-09-16T01:43:11.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.526+0000 7fbba97af640 1 --2- 172.21.15.88:0/3011158607 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe372858d0 0x55fe37256460 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c009790 tx=0x7fbb9c009c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d2fa4a591074c0c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:11.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.526+0000 7fbba3fff640 1 -- 172.21.15.88:0/3011158607 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb9c017070 con 0x55fe372858d0 2024-09-16T01:43:11.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.526+0000 7fbba3fff640 1 -- 172.21.15.88:0/3011158607 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb9c0090d0 con 0x55fe372858d0 2024-09-16T01:43:11.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.526+0000 7fbba3fff640 1 -- 172.21.15.88:0/3011158607 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb9c006b60 con 0x55fe372858d0 2024-09-16T01:43:11.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.527+0000 7fbbac720f40 1 -- 172.21.15.88:0/3011158607 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe372858d0 msgr2=0x55fe37256460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:11.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.527+0000 7fbbac720f40 1 --2- 172.21.15.88:0/3011158607 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe372858d0 0x55fe37256460 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c009790 tx=0x7fbb9c009c50 comp rx=0 tx=0).stop 2024-09-16T01:43:11.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.527+0000 7fbbac720f40 1 -- 172.21.15.88:0/3011158607 shutdown_connections 2024-09-16T01:43:11.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.527+0000 7fbbac720f40 1 --2- 172.21.15.88:0/3011158607 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe3732b100 0x55fe37282480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.527+0000 7fbbac720f40 1 --2- 172.21.15.88:0/3011158607 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fe3732ad20 0x55fe37281f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.527+0000 7fbbac720f40 1 --2- 172.21.15.88:0/3011158607 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe372858d0 0x55fe37256460 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.527+0000 7fbbac720f40 1 -- 172.21.15.88:0/3011158607 >> 172.21.15.88:0/3011158607 conn(0x55fe372a1750 msgr2=0x55fe3732f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:11.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.527+0000 7fbbac720f40 1 -- 172.21.15.88:0/3011158607 shutdown_connections 2024-09-16T01:43:11.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.527+0000 7fbbac720f40 1 -- 172.21.15.88:0/3011158607 wait complete. 2024-09-16T01:43:11.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.527+0000 7fbbac720f40 1 Processor -- start 2024-09-16T01:43:11.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.528+0000 7fbbac720f40 1 -- start start 2024-09-16T01:43:11.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.528+0000 7fbbac720f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fe372858d0 0x55fe37319630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.528+0000 7fbbac720f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe3732ad20 0x55fe3729e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.528+0000 7fbbac720f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe3732b100 0x55fe3730e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.528+0000 7fbbac720f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55fe37281400 con 0x55fe3732ad20 2024-09-16T01:43:11.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.528+0000 7fbbac720f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55fe37281650 con 0x55fe3732b100 2024-09-16T01:43:11.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.528+0000 7fbbac720f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55fe3730e650 con 0x55fe372858d0 2024-09-16T01:43:11.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.528+0000 7fbba0fae640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe3732ad20 0x55fe3729e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.528+0000 7fbba0fae640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe3732ad20 0x55fe3729e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58516/0 (socket says 172.21.15.88:58516) 2024-09-16T01:43:11.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.528+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 learned_addr learned my addr 172.21.15.88:0/334505952 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:11.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.529+0000 7fbba9fb0640 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe3732b100 0x55fe3730e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.529+0000 7fbba97af640 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fe372858d0 0x55fe37319630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.529+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fe372858d0 msgr2=0x55fe37319630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:11.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.529+0000 7fbba0fae640 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55fe372858d0 0x55fe37319630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.529+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe3732b100 msgr2=0x55fe3730e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:11.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.529+0000 7fbba0fae640 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe3732b100 0x55fe3730e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.529+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe3731ac50 con 0x55fe3732ad20 2024-09-16T01:43:11.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.529+0000 7fbba0fae640 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe3732ad20 0x55fe3729e410 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7fbb9400a490 tx=0x7fbb9400a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:11.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.529+0000 7fbba2ffd640 1 -- 172.21.15.88:0/334505952 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb94019070 con 0x55fe3732ad20 2024-09-16T01:43:11.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.529+0000 7fbba2ffd640 1 -- 172.21.15.88:0/334505952 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb940086a0 con 0x55fe3732ad20 2024-09-16T01:43:11.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.529+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55fe3731aef0 con 0x55fe3732ad20 2024-09-16T01:43:11.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.529+0000 7fbba2ffd640 1 -- 172.21.15.88:0/334505952 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb94008a90 con 0x55fe3732ad20 2024-09-16T01:43:11.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.529+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe3731b4b0 con 0x55fe3732ad20 2024-09-16T01:43:11.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.530+0000 7fbba2ffd640 1 -- 172.21.15.88:0/334505952 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fbb9401c070 con 0x55fe3732ad20 2024-09-16T01:43:11.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.531+0000 7fbba2ffd640 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbb8c06ac90 0x7fbb8c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.531+0000 7fbba2ffd640 1 -- 172.21.15.88:0/334505952 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8318+0+0 (secure 0 0 0) 0x7fbb9408a5c0 con 0x55fe3732ad20 2024-09-16T01:43:11.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.531+0000 7fbba97af640 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbb8c06ac90 0x7fbb8c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.536 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:43:11.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.531+0000 7fbbac720f40 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55fe373c2e90 0x55fe373c3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.531+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373c37b0 con 0x55fe373c2e90 2024-09-16T01:43:11.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.531+0000 7fbba9fb0640 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55fe373c2e90 0x55fe373c3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.531+0000 7fbba97af640 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbb8c06ac90 0x7fbb8c06d150 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c009fd0 tx=0x7fbb9c00f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:11.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.532+0000 7fbba9fb0640 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55fe373c2e90 0x55fe373c3270 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:11.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.534+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v117'908 uv882 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.534+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e117) -- 0x55fe373c3b50 con 0x55fe373c2e90 2024-09-16T01:43:11.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.535+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v117'909 uv909 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.535+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373cf4d0 con 0x55fe373c2e90 2024-09-16T01:43:11.539 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:43:11.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.537+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v117'910 uv910 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.537+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d0880 con 0x55fe373c2e90 2024-09-16T01:43:11.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.538+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v117'911 uv911 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.538+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d0c20 con 0x55fe373c2e90 2024-09-16T01:43:11.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.539+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v117'912 uv912 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.539+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373cf870 con 0x55fe373c2e90 2024-09-16T01:43:11.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.540+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v117'913 uv913 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.541+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373cfc60 con 0x55fe373c2e90 2024-09-16T01:43:11.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.542+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v117'914 uv888 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.542+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e117) -- 0x55fe373cfc60 con 0x55fe373c2e90 2024-09-16T01:43:11.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.543+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v117'915 uv915 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.543+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373cfc60 con 0x55fe373c2e90 2024-09-16T01:43:11.546 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:43:11.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.545+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v117'916 uv916 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.545+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373cfc60 con 0x55fe373c2e90 2024-09-16T01:43:11.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.546+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v117'917 uv917 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.546+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373cfc60 con 0x55fe373c2e90 2024-09-16T01:43:11.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.547+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v117'918 uv918 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.547+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373cfc60 con 0x55fe373c2e90 2024-09-16T01:43:11.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.549+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v117'919 uv919 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.549+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373cfc60 con 0x55fe373c2e90 2024-09-16T01:43:11.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.550+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v117'920 uv895 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.550+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e117) -- 0x55fe373cfc60 con 0x55fe373c2e90 2024-09-16T01:43:11.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.552+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v117'921 uv921 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.552+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373cfc60 con 0x55fe373c2e90 2024-09-16T01:43:11.554 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:43:11.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.553+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v117'922 uv922 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.553+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373cfc60 con 0x55fe373c2e90 2024-09-16T01:43:11.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.554+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v117'923 uv923 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.554+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373cfc60 con 0x55fe373c2e90 2024-09-16T01:43:11.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.556+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v117'924 uv924 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.556+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373cfc60 con 0x55fe373c2e90 2024-09-16T01:43:11.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.557+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v117'925 uv925 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.557+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d0fc0 con 0x55fe373c2e90 2024-09-16T01:43:11.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.558+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v117'926 uv926 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.558+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d03c0 con 0x55fe373c2e90 2024-09-16T01:43:11.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.560+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v117'927 uv901 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.560+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d03c0 con 0x55fe373c2e90 2024-09-16T01:43:11.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.561+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v117'928 uv928 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.561+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d03c0 con 0x55fe373c2e90 2024-09-16T01:43:11.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.563+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v117'929 uv929 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.564 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:43:11.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.563+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d03c0 con 0x55fe373c2e90 2024-09-16T01:43:11.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.564+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v117'930 uv930 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.564+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d03c0 con 0x55fe373c2e90 2024-09-16T01:43:11.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.565+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v117'931 uv931 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.565+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d03c0 con 0x55fe373c2e90 2024-09-16T01:43:11.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.566+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v117'932 uv932 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.566+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d03c0 con 0x55fe373c2e90 2024-09-16T01:43:11.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.568+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v117'933 uv907 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.568+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d03c0 con 0x55fe373c2e90 2024-09-16T01:43:11.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.569+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v117'934 uv934 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.569+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d03c0 con 0x55fe373c2e90 2024-09-16T01:43:11.571 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:43:11.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.570+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v117'935 uv935 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.570+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d03c0 con 0x55fe373c2e90 2024-09-16T01:43:11.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.572+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v117'936 uv936 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.572+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d03c0 con 0x55fe373c2e90 2024-09-16T01:43:11.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.573+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v117'937 uv937 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.573+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d03c0 con 0x55fe373c2e90 2024-09-16T01:43:11.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.575+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v117'938 uv938 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.575+0000 7fbbac720f40 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55fe373d14d0 0x55fe373d38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.575+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d03c0 con 0x55fe373d14d0 2024-09-16T01:43:11.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.575+0000 7fbba97af640 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55fe373d14d0 0x55fe373d38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.575+0000 7fbba97af640 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55fe373d14d0 0x55fe373d38c0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:11.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.577+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v117'660 uv643 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373d14d0 2024-09-16T01:43:11.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.577+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d3e00 con 0x55fe373d14d0 2024-09-16T01:43:11.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.578+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v117'661 uv661 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373d14d0 2024-09-16T01:43:11.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.578+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d41a0 con 0x55fe373d14d0 2024-09-16T01:43:11.581 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:43:11.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.580+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v117'662 uv662 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373d14d0 2024-09-16T01:43:11.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.580+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d49f0 con 0x55fe373d14d0 2024-09-16T01:43:11.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.581+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v117'663 uv663 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373d14d0 2024-09-16T01:43:11.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.581+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d4d90 con 0x55fe373d14d0 2024-09-16T01:43:11.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.582+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v117'664 uv664 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373d14d0 2024-09-16T01:43:11.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.582+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d5130 con 0x55fe373d14d0 2024-09-16T01:43:11.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.583+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v117'665 uv648 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373d14d0 2024-09-16T01:43:11.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.583+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d54d0 con 0x55fe373d14d0 2024-09-16T01:43:11.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.585+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v117'666 uv666 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373d14d0 2024-09-16T01:43:11.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.585+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d54d0 con 0x55fe373d14d0 2024-09-16T01:43:11.587 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:43:11.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.586+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v117'667 uv667 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373d14d0 2024-09-16T01:43:11.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.586+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d54d0 con 0x55fe373d14d0 2024-09-16T01:43:11.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.587+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v117'668 uv668 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373d14d0 2024-09-16T01:43:11.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.587+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d54d0 con 0x55fe373d14d0 2024-09-16T01:43:11.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.588+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v117'669 uv669 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373d14d0 2024-09-16T01:43:11.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.588+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d54d0 con 0x55fe373d14d0 2024-09-16T01:43:11.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.589+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v117'670 uv653 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373d14d0 2024-09-16T01:43:11.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.590+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d54d0 con 0x55fe373d14d0 2024-09-16T01:43:11.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.591+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v117'671 uv671 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373d14d0 2024-09-16T01:43:11.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.591+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d54d0 con 0x55fe373d14d0 2024-09-16T01:43:11.593 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:43:11.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.592+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v117'672 uv672 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373d14d0 2024-09-16T01:43:11.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.592+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d9950 con 0x55fe373d14d0 2024-09-16T01:43:11.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.594+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v117'673 uv673 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373d14d0 2024-09-16T01:43:11.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.594+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d9950 con 0x55fe373d14d0 2024-09-16T01:43:11.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.595+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v117'674 uv674 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373d14d0 2024-09-16T01:43:11.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.595+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d54d0 con 0x55fe373d14d0 2024-09-16T01:43:11.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.596+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v117'675 uv659 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373d14d0 2024-09-16T01:43:11.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.596+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d54d0 con 0x55fe373d14d0 2024-09-16T01:43:11.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.597+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v117'676 uv676 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373d14d0 2024-09-16T01:43:11.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.597+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d54d0 con 0x55fe373d14d0 2024-09-16T01:43:11.599 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:43:11.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.598+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v117'677 uv677 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373d14d0 2024-09-16T01:43:11.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.598+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d64e0 con 0x55fe373d14d0 2024-09-16T01:43:11.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.599+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v117'678 uv678 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373d14d0 2024-09-16T01:43:11.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.599+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d64e0 con 0x55fe373d14d0 2024-09-16T01:43:11.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.601+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v117'679 uv679 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373d14d0 2024-09-16T01:43:11.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.601+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d54d0 con 0x55fe373d14d0 2024-09-16T01:43:11.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.602+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v117'680 uv680 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373d14d0 2024-09-16T01:43:11.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.602+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d54d0 con 0x55fe373c2e90 2024-09-16T01:43:11.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.603+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v117'177 uv176 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.603+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d54d0 con 0x55fe373c2e90 2024-09-16T01:43:11.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.604+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v117'178 uv178 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.604+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d54d0 con 0x55fe373c2e90 2024-09-16T01:43:11.606 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:43:11.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.605+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v117'179 uv179 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.605+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d64e0 con 0x55fe373c2e90 2024-09-16T01:43:11.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.606+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v117'180 uv180 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.606+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d64e0 con 0x55fe373c2e90 2024-09-16T01:43:11.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.607+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v117'181 uv181 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.607+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d54d0 con 0x55fe373c2e90 2024-09-16T01:43:11.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.608+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v117'182 uv182 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.608+0000 7fbbac720f40 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55fe373d59e0 0x55fe373d7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.608+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d8310 con 0x55fe373d59e0 2024-09-16T01:43:11.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.608+0000 7fbba0fae640 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55fe373d59e0 0x55fe373d7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.609+0000 7fbba0fae640 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55fe373d59e0 0x55fe373d7dd0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:11.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.610+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v117'131 uv130 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fbb9401b030 con 0x55fe373d59e0 2024-09-16T01:43:11.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.610+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d86b0 con 0x55fe373d59e0 2024-09-16T01:43:11.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.611+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v117'132 uv132 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbb9401b030 con 0x55fe373d59e0 2024-09-16T01:43:11.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.611+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d8b50 con 0x55fe373d59e0 2024-09-16T01:43:11.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.611+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v117'133 uv133 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbb9401b030 con 0x55fe373d59e0 2024-09-16T01:43:11.613 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:43:11.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.611+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d4620 con 0x55fe373d59e0 2024-09-16T01:43:11.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.612+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v117'134 uv134 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fbb9401b030 con 0x55fe373d59e0 2024-09-16T01:43:11.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.612+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d9480 con 0x55fe373c2e90 2024-09-16T01:43:11.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.614+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v117'177 uv176 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.614+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d9480 con 0x55fe373c2e90 2024-09-16T01:43:11.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.615+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v117'178 uv178 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.615+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d9480 con 0x55fe373c2e90 2024-09-16T01:43:11.617 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:43:11.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.616+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v117'179 uv179 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.616+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373da490 con 0x55fe373c2e90 2024-09-16T01:43:11.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.617+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v117'180 uv180 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.617+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373da490 con 0x55fe373c2e90 2024-09-16T01:43:11.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.618+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v117'181 uv181 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.618+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373d9480 con 0x55fe373c2e90 2024-09-16T01:43:11.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.619+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v117'182 uv182 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.619+0000 7fbbac720f40 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55fe373d9b00 0x55fe373d9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.619+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373dc880 con 0x55fe373d9b00 2024-09-16T01:43:11.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.619+0000 7fbba9fb0640 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55fe373d9b00 0x55fe373d9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.620+0000 7fbba9fb0640 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55fe373d9b00 0x55fe373d9ee0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:11.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.621+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v117'327 uv321 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.621+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e117) -- 0x55fe373dc880 con 0x55fe373d9b00 2024-09-16T01:43:11.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.622+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v117'328 uv328 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.622+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373dc880 con 0x55fe373d9b00 2024-09-16T01:43:11.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.623+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v117'329 uv329 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.625 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:43:11.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.623+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373dd890 con 0x55fe373d9b00 2024-09-16T01:43:11.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.624+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v117'330 uv330 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.624+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373dd890 con 0x55fe373d9b00 2024-09-16T01:43:11.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.625+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v117'331 uv331 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.625+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373dc880 con 0x55fe373d9b00 2024-09-16T01:43:11.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.626+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v117'332 uv332 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.626+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373dc880 con 0x55fe373d9b00 2024-09-16T01:43:11.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.627+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v117'333 uv326 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.627+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e117) -- 0x55fe373dc880 con 0x55fe373d9b00 2024-09-16T01:43:11.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.628+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v117'334 uv334 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.628+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373dc880 con 0x55fe373d9b00 2024-09-16T01:43:11.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.629+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v117'335 uv335 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.630 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:43:11.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.629+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373dc880 con 0x55fe373d9b00 2024-09-16T01:43:11.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.630+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v117'336 uv336 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.630+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373dc880 con 0x55fe373d9b00 2024-09-16T01:43:11.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.631+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v117'337 uv337 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.631+0000 7fbbac720f40 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55fe373dd1a0 0x55fe373dd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.631+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373dfcd0 con 0x55fe373dd1a0 2024-09-16T01:43:11.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.631+0000 7fbba97af640 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55fe373dd1a0 0x55fe373dd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.632+0000 7fbba97af640 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55fe373dd1a0 0x55fe373dd5c0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:11.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.633+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v117'148 uv147 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373dd1a0 2024-09-16T01:43:11.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.633+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e117) -- 0x55fe373dfcd0 con 0x55fe373dd1a0 2024-09-16T01:43:11.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.634+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v117'149 uv149 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373dd1a0 2024-09-16T01:43:11.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.634+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373dfd50 con 0x55fe373dd1a0 2024-09-16T01:43:11.637 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:43:11.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.636+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v117'150 uv150 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373dd1a0 2024-09-16T01:43:11.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.636+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e0140 con 0x55fe373dd1a0 2024-09-16T01:43:11.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.637+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v117'151 uv151 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373dd1a0 2024-09-16T01:43:11.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.637+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e0140 con 0x55fe373dd1a0 2024-09-16T01:43:11.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.639+0000 7fbba97af640 1 -- 172.21.15.88:0/334505952 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v117'152 uv152 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbb9c006bf0 con 0x55fe373dd1a0 2024-09-16T01:43:11.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.639+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e0140 con 0x55fe373c2e90 2024-09-16T01:43:11.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.640+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v117'177 uv176 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.640+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e0140 con 0x55fe373c2e90 2024-09-16T01:43:11.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.641+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v117'178 uv178 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.641+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e0140 con 0x55fe373c2e90 2024-09-16T01:43:11.643 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:43:11.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.642+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v117'179 uv179 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.642+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e1150 con 0x55fe373c2e90 2024-09-16T01:43:11.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.642+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v117'180 uv180 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.642+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e1150 con 0x55fe373c2e90 2024-09-16T01:43:11.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.643+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v117'181 uv181 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.643+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e0140 con 0x55fe373c2e90 2024-09-16T01:43:11.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.644+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v117'182 uv182 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373c2e90 2024-09-16T01:43:11.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.644+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e0140 con 0x55fe373d59e0 2024-09-16T01:43:11.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.645+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v117'294 uv288 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fbb9401b030 con 0x55fe373d59e0 2024-09-16T01:43:11.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.645+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e04e0 con 0x55fe373d59e0 2024-09-16T01:43:11.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.646+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v117'295 uv295 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbb9401b030 con 0x55fe373d59e0 2024-09-16T01:43:11.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.646+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e0880 con 0x55fe373d59e0 2024-09-16T01:43:11.651 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:43:11.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.647+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v117'296 uv296 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbb9401b030 con 0x55fe373d59e0 2024-09-16T01:43:11.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.647+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e0880 con 0x55fe373d59e0 2024-09-16T01:43:11.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.648+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v117'297 uv297 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbb9401b030 con 0x55fe373d59e0 2024-09-16T01:43:11.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.648+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e0880 con 0x55fe373d59e0 2024-09-16T01:43:11.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.649+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v117'298 uv298 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fbb9401b030 con 0x55fe373d59e0 2024-09-16T01:43:11.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.649+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e0880 con 0x55fe373d59e0 2024-09-16T01:43:11.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.650+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v117'299 uv293 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbb9401b030 con 0x55fe373d59e0 2024-09-16T01:43:11.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.650+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e0880 con 0x55fe373d59e0 2024-09-16T01:43:11.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.651+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v117'300 uv300 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbb9401b030 con 0x55fe373d59e0 2024-09-16T01:43:11.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.651+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e0880 con 0x55fe373d59e0 2024-09-16T01:43:11.654 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:43:11.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.652+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v117'301 uv301 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbb9401b030 con 0x55fe373d59e0 2024-09-16T01:43:11.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.652+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e0880 con 0x55fe373d59e0 2024-09-16T01:43:11.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.653+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v117'302 uv302 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbb9401b030 con 0x55fe373d59e0 2024-09-16T01:43:11.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.653+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e3690 con 0x55fe373d59e0 2024-09-16T01:43:11.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.654+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v117'303 uv303 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbb9401b030 con 0x55fe373d59e0 2024-09-16T01:43:11.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.654+0000 7fbbac720f40 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55fe373e13b0 0x55fe373e3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.654+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373e13b0 2024-09-16T01:43:11.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.655+0000 7fbba0fae640 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55fe373e13b0 0x55fe373e3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.655+0000 7fbba0fae640 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55fe373e13b0 0x55fe373e3d30 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:11.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.656+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v117'128 uv127 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fbb9401b030 con 0x55fe373e13b0 2024-09-16T01:43:11.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.656+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373e13b0 2024-09-16T01:43:11.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.657+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v117'129 uv129 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fbb9401b030 con 0x55fe373e13b0 2024-09-16T01:43:11.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.657+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373e13b0 2024-09-16T01:43:11.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.658+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v117'130 uv130 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fbb9401b030 con 0x55fe373e13b0 2024-09-16T01:43:11.660 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:43:11.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.658+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373e13b0 2024-09-16T01:43:11.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.659+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v117'131 uv131 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fbb9401b030 con 0x55fe373e13b0 2024-09-16T01:43:11.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.659+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373d59e0 2024-09-16T01:43:11.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.660+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v117'177 uv176 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbb9401b030 con 0x55fe373d59e0 2024-09-16T01:43:11.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.660+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373d59e0 2024-09-16T01:43:11.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.661+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v117'178 uv178 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbb9401b030 con 0x55fe373d59e0 2024-09-16T01:43:11.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.661+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373d59e0 2024-09-16T01:43:11.663 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:43:11.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.662+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v117'179 uv179 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbb9401b030 con 0x55fe373d59e0 2024-09-16T01:43:11.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.662+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373d59e0 2024-09-16T01:43:11.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.663+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v117'180 uv180 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbb9401b030 con 0x55fe373d59e0 2024-09-16T01:43:11.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.663+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373d59e0 2024-09-16T01:43:11.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.664+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v117'181 uv181 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbb9401b030 con 0x55fe373d59e0 2024-09-16T01:43:11.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.664+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373d59e0 2024-09-16T01:43:11.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.665+0000 7fbba0fae640 1 -- 172.21.15.88:0/334505952 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v117'182 uv182 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbb9401b030 con 0x55fe373d59e0 2024-09-16T01:43:11.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.665+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373d9b00 2024-09-16T01:43:11.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.666+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v117'151 uv150 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.666+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373d9b00 2024-09-16T01:43:11.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.667+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v117'152 uv152 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.667+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373d9b00 2024-09-16T01:43:11.669 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:43:11.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.668+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v117'153 uv153 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.668+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373d9b00 2024-09-16T01:43:11.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.670+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v117'154 uv154 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.670+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373d9b00 2024-09-16T01:43:11.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.671+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v117'155 uv155 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.671+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373d9b00 2024-09-16T01:43:11.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.672+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v117'363 uv355 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.672+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373d9b00 2024-09-16T01:43:11.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.673+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v117'364 uv364 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.673+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373d9b00 2024-09-16T01:43:11.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.674+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v117'365 uv365 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.676 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:43:11.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.674+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373d9b00 2024-09-16T01:43:11.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.675+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v117'366 uv366 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.675+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373d9b00 2024-09-16T01:43:11.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.676+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v117'367 uv367 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.676+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373d9b00 2024-09-16T01:43:11.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.677+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v117'368 uv362 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.677+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373d9b00 2024-09-16T01:43:11.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.680+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v117'369 uv369 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.680+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373d9b00 2024-09-16T01:43:11.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.682+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v117'370 uv370 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.683 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:43:11.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.682+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373d9b00 2024-09-16T01:43:11.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.683+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v117'371 uv371 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.683+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373d9b00 2024-09-16T01:43:11.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.684+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v117'372 uv372 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.684+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e37a0 con 0x55fe373d9b00 2024-09-16T01:43:11.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.685+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v117'373 uv373 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.685+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55fe373e4270 con 0x55fe373d9b00 2024-09-16T01:43:11.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.685+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v117'374 uv374 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fbba404f960 con 0x55fe373d9b00 2024-09-16T01:43:11.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55fe373c2e90 msgr2=0x55fe373c3270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:11.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbbac720f40 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55fe373c2e90 0x55fe373c3270 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55fe373d59e0 msgr2=0x55fe373d7dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:11.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbbac720f40 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55fe373d59e0 0x55fe373d7dd0 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55fe373e13b0 msgr2=0x55fe373e3d30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:11.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbbac720f40 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55fe373e13b0 0x55fe373e3d30 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55fe373d14d0 msgr2=0x55fe373d38c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:11.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbbac720f40 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55fe373d14d0 0x55fe373d38c0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 reap_dead start 2024-09-16T01:43:11.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55fe373d9b00 msgr2=0x55fe373d9ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:11.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbbac720f40 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55fe373d9b00 0x55fe373d9ee0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbba9fb0640 1 -- 172.21.15.88:0/334505952 reap_dead start 2024-09-16T01:43:11.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55fe373dd1a0 msgr2=0x55fe373dd5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:11.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbbac720f40 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55fe373dd1a0 0x55fe373dd5c0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbb8c06ac90 msgr2=0x7fbb8c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:11.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbbac720f40 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbb8c06ac90 0x7fbb8c06d150 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c009fd0 tx=0x7fbb9c00f000 comp rx=0 tx=0).stop 2024-09-16T01:43:11.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe3732ad20 msgr2=0x55fe3729e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:11.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbbac720f40 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe3732ad20 0x55fe3729e410 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7fbb9400a490 tx=0x7fbb9400a950 comp rx=0 tx=0).stop 2024-09-16T01:43:11.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 shutdown_connections 2024-09-16T01:43:11.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbbac720f40 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbb8c06ac90 0x7fbb8c06d150 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbbac720f40 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55fe373dd1a0 0x55fe373dd5c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbbac720f40 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55fe373d9b00 0x55fe373d9ee0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbbac720f40 1 --2- 172.21.15.88:0/334505952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe3732ad20 0x55fe3729e410 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 >> 172.21.15.88:0/334505952 conn(0x55fe372a1750 msgr2=0x55fe3732ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:11.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 shutdown_connections 2024-09-16T01:43:11.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.686+0000 7fbbac720f40 1 -- 172.21.15.88:0/334505952 wait complete. 2024-09-16T01:43:11.696 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata truncate 100.00000000.inode 0 2024-09-16T01:43:11.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.774+0000 7f1e6f00bf40 1 Processor -- start 2024-09-16T01:43:11.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.774+0000 7f1e6f00bf40 1 -- start start 2024-09-16T01:43:11.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.775+0000 7f1e6f00bf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21dc8f670 0x55f21dc98710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.775+0000 7f1e6f00bf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f21dd243c0 0x55f21dd07f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.775+0000 7f1e6f00bf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21dd26710 0x55f21dd21bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.775+0000 7f1e6f00bf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f21dd27c50 con 0x55f21dd26710 2024-09-16T01:43:11.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.775+0000 7f1e6f00bf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f21dc8eb30 con 0x55f21dc8f670 2024-09-16T01:43:11.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.775+0000 7f1e6f00bf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55f21dd27ef0 con 0x55f21dd243c0 2024-09-16T01:43:11.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.775+0000 7f1e6c61e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21dd26710 0x55f21dd21bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.775+0000 7f1e6c61e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21dd26710 0x55f21dd21bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58524/0 (socket says 172.21.15.88:58524) 2024-09-16T01:43:11.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.775+0000 7f1e6c61e640 1 -- 172.21.15.88:0/3268443807 learned_addr learned my addr 172.21.15.88:0/3268443807 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:11.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.775+0000 7f1e6be1d640 1 --2- 172.21.15.88:0/3268443807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21dc8f670 0x55f21dc98710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.775+0000 7f1e63fff640 1 --2- 172.21.15.88:0/3268443807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f21dd243c0 0x55f21dd07f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.775+0000 7f1e6c61e640 1 -- 172.21.15.88:0/3268443807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f21dd243c0 msgr2=0x55f21dd07f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:11.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.775+0000 7f1e6c61e640 1 --2- 172.21.15.88:0/3268443807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f21dd243c0 0x55f21dd07f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.775+0000 7f1e6c61e640 1 -- 172.21.15.88:0/3268443807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21dc8f670 msgr2=0x55f21dc98710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:11.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.775+0000 7f1e6c61e640 1 --2- 172.21.15.88:0/3268443807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21dc8f670 0x55f21dc98710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.775+0000 7f1e6c61e640 1 -- 172.21.15.88:0/3268443807 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f21db79cb0 con 0x55f21dd26710 2024-09-16T01:43:11.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.775+0000 7f1e6be1d640 1 --2- 172.21.15.88:0/3268443807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21dc8f670 0x55f21dc98710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:11.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.775+0000 7f1e6c61e640 1 --2- 172.21.15.88:0/3268443807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21dd26710 0x55f21dd21bf0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f1e64002800 tx=0x7f1e64002cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cabdf141cd5fa1e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:11.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.775+0000 7f1e6ae1b640 1 -- 172.21.15.88:0/3268443807 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e64016020 con 0x55f21dd26710 2024-09-16T01:43:11.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.776+0000 7f1e6ae1b640 1 -- 172.21.15.88:0/3268443807 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e6400d9c0 con 0x55f21dd26710 2024-09-16T01:43:11.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.776+0000 7f1e6ae1b640 1 -- 172.21.15.88:0/3268443807 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6400ddf0 con 0x55f21dd26710 2024-09-16T01:43:11.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.776+0000 7f1e6f00bf40 1 -- 172.21.15.88:0/3268443807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21dd26710 msgr2=0x55f21dd21bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:11.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.776+0000 7f1e6f00bf40 1 --2- 172.21.15.88:0/3268443807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21dd26710 0x55f21dd21bf0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f1e64002800 tx=0x7f1e64002cc0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.776+0000 7f1e6f00bf40 1 -- 172.21.15.88:0/3268443807 shutdown_connections 2024-09-16T01:43:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.776+0000 7f1e6f00bf40 1 --2- 172.21.15.88:0/3268443807 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21dd26710 0x55f21dd21bf0 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.776+0000 7f1e6f00bf40 1 --2- 172.21.15.88:0/3268443807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f21dd243c0 0x55f21dd07f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.776+0000 7f1e6f00bf40 1 --2- 172.21.15.88:0/3268443807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21dc8f670 0x55f21dc98710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.776+0000 7f1e6f00bf40 1 -- 172.21.15.88:0/3268443807 >> 172.21.15.88:0/3268443807 conn(0x55f21dc79680 msgr2=0x55f21dd20df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.776+0000 7f1e6f00bf40 1 -- 172.21.15.88:0/3268443807 shutdown_connections 2024-09-16T01:43:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.776+0000 7f1e6f00bf40 1 -- 172.21.15.88:0/3268443807 wait complete. 2024-09-16T01:43:11.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.777+0000 7f1e6f00bf40 1 Processor -- start 2024-09-16T01:43:11.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.777+0000 7f1e6f00bf40 1 -- start start 2024-09-16T01:43:11.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.777+0000 7f1e6f00bf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21dc8f670 0x55f21dc7dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.777+0000 7f1e6f00bf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f21dd243c0 0x55f21dc7e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.777+0000 7f1e6f00bf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21dd26710 0x55f21dc7e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.777+0000 7f1e6f00bf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f21dd23b90 con 0x55f21dc8f670 2024-09-16T01:43:11.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.777+0000 7f1e6f00bf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f21dd276c0 con 0x55f21dd26710 2024-09-16T01:43:11.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.777+0000 7f1e6f00bf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55f21dd1fe90 con 0x55f21dd243c0 2024-09-16T01:43:11.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.777+0000 7f1e6be1d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21dc8f670 0x55f21dc7dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.777+0000 7f1e6be1d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21dc8f670 0x55f21dc7dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58538/0 (socket says 172.21.15.88:58538) 2024-09-16T01:43:11.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.777+0000 7f1e6be1d640 1 -- 172.21.15.88:0/3652428668 learned_addr learned my addr 172.21.15.88:0/3652428668 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:11.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.778+0000 7f1e63fff640 1 --2- 172.21.15.88:0/3652428668 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f21dd243c0 0x55f21dc7e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.778+0000 7f1e6c61e640 1 --2- 172.21.15.88:0/3652428668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21dd26710 0x55f21dc7e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.778+0000 7f1e6be1d640 1 -- 172.21.15.88:0/3652428668 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f21dd243c0 msgr2=0x55f21dc7e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:11.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.778+0000 7f1e6be1d640 1 --2- 172.21.15.88:0/3652428668 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f21dd243c0 0x55f21dc7e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.778+0000 7f1e6be1d640 1 -- 172.21.15.88:0/3652428668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21dd26710 msgr2=0x55f21dc7e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:11.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.778+0000 7f1e6be1d640 1 --2- 172.21.15.88:0/3652428668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f21dd26710 0x55f21dc7e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.778+0000 7f1e6be1d640 1 -- 172.21.15.88:0/3652428668 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f21dd1cb90 con 0x55f21dc8f670 2024-09-16T01:43:11.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.778+0000 7f1e6be1d640 1 --2- 172.21.15.88:0/3652428668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21dc8f670 0x55f21dc7dc00 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c009f70 tx=0x7f1e5c0084c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:11.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.778+0000 7f1e69e19640 1 -- 172.21.15.88:0/3652428668 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c015460 con 0x55f21dc8f670 2024-09-16T01:43:11.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.778+0000 7f1e69e19640 1 -- 172.21.15.88:0/3652428668 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e5c015600 con 0x55f21dc8f670 2024-09-16T01:43:11.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.778+0000 7f1e6f00bf40 1 -- 172.21.15.88:0/3652428668 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f21dd07380 con 0x55f21dc8f670 2024-09-16T01:43:11.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.778+0000 7f1e69e19640 1 -- 172.21.15.88:0/3652428668 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c0159f0 con 0x55f21dc8f670 2024-09-16T01:43:11.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.778+0000 7f1e6f00bf40 1 -- 172.21.15.88:0/3652428668 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f21dd20530 con 0x55f21dc8f670 2024-09-16T01:43:11.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.779+0000 7f1e69e19640 1 -- 172.21.15.88:0/3652428668 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1e5c01c020 con 0x55f21dc8f670 2024-09-16T01:43:11.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.780+0000 7f1e69e19640 1 --2- 172.21.15.88:0/3652428668 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1e4c06ac90 0x7f1e4c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.780+0000 7f1e63fff640 1 --2- 172.21.15.88:0/3652428668 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1e4c06ac90 0x7f1e4c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.780+0000 7f1e69e19640 1 -- 172.21.15.88:0/3652428668 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8318+0+0 (secure 0 0 0) 0x7f1e5c08a880 con 0x55f21dc8f670 2024-09-16T01:43:11.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.780+0000 7f1e6f00bf40 1 --2- 172.21.15.88:0/3652428668 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f21dd06740 0x55f21dd06b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.780+0000 7f1e6f00bf40 1 -- 172.21.15.88:0/3652428668 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:1 12.1f 12:fe07bba1:::100.00000000.inode:head [truncate 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117) -- 0x55f21dc958f0 con 0x55f21dd06740 2024-09-16T01:43:11.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.780+0000 7f1e63fff640 1 --2- 172.21.15.88:0/3652428668 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1e4c06ac90 0x7f1e4c06d150 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f1e54057000 tx=0x7f1e54052000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:11.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.780+0000 7f1e6c61e640 1 --2- 172.21.15.88:0/3652428668 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f21dd06740 0x55f21dd06b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.781+0000 7f1e6c61e640 1 --2- 172.21.15.88:0/3652428668 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f21dd06740 0x55f21dd06b40 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.782+0000 7f1e6c61e640 1 -- 172.21.15.88:0/3652428668 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(1 100.00000000.inode [truncate 0] v117'375 uv375 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f1e6400dc70 con 0x55f21dd06740 2024-09-16T01:43:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.782+0000 7f1e6f00bf40 1 -- 172.21.15.88:0/3652428668 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f21dd06740 msgr2=0x55f21dd06b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:11.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.782+0000 7f1e6f00bf40 1 --2- 172.21.15.88:0/3652428668 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55f21dd06740 0x55f21dd06b40 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.782+0000 7f1e6f00bf40 1 -- 172.21.15.88:0/3652428668 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1e4c06ac90 msgr2=0x7f1e4c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:11.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.782+0000 7f1e6f00bf40 1 --2- 172.21.15.88:0/3652428668 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1e4c06ac90 0x7f1e4c06d150 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f1e54057000 tx=0x7f1e54052000 comp rx=0 tx=0).stop 2024-09-16T01:43:11.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.782+0000 7f1e6f00bf40 1 -- 172.21.15.88:0/3652428668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21dc8f670 msgr2=0x55f21dc7dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:11.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.782+0000 7f1e6f00bf40 1 --2- 172.21.15.88:0/3652428668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f21dc8f670 0x55f21dc7dc00 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c009f70 tx=0x7f1e5c0084c0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.782+0000 7f1e6c61e640 1 -- 172.21.15.88:0/3652428668 reap_dead start 2024-09-16T01:43:11.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.782+0000 7f1e6f00bf40 1 -- 172.21.15.88:0/3652428668 shutdown_connections 2024-09-16T01:43:11.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.782+0000 7f1e6f00bf40 1 -- 172.21.15.88:0/3652428668 >> 172.21.15.88:0/3652428668 conn(0x55f21dc79680 msgr2=0x55f21dd1f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:11.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.782+0000 7f1e6f00bf40 1 -- 172.21.15.88:0/3652428668 shutdown_connections 2024-09-16T01:43:11.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.782+0000 7f1e6f00bf40 1 -- 172.21.15.88:0/3652428668 wait complete. 2024-09-16T01:43:11.794 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:43:11.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.934+0000 7f1465d5b640 1 Processor -- start 2024-09-16T01:43:11.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.934+0000 7f1465d5b640 1 -- start start 2024-09-16T01:43:11.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.935+0000 7f1465d5b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f146015e6f0 0x7f146006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.935+0000 7f1465d5b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f14600b54d0 0x7f146015f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.935+0000 7f1465d5b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f146006e990 con 0x7f14600b6d40 2024-09-16T01:43:11.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.935+0000 7f1465d5b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f146006ec70 con 0x7f14600b54d0 2024-09-16T01:43:11.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.935+0000 7f1465d5b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f146015ec70 con 0x7f146015e6f0 2024-09-16T01:43:11.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.935+0000 7f145effd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f14600b54d0 0x7f146015f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.935+0000 7f145effd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f14600b54d0 0x7f146015f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58554/0 (socket says 172.21.15.88:58554) 2024-09-16T01:43:11.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.935+0000 7f145effd640 1 -- 172.21.15.88:0/93231488 learned_addr learned my addr 172.21.15.88:0/93231488 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:11.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.935+0000 7f145f7fe640 1 --2- 172.21.15.88:0/93231488 >> v2:172.21.15.154:3300/0 conn(0x7f146015e6f0 0x7f146006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.935+0000 7f145effd640 1 -- 172.21.15.88:0/93231488 >> v2:172.21.15.154:3300/0 conn(0x7f146015e6f0 msgr2=0x7f146006fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:11.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.935+0000 7f145effd640 1 --2- 172.21.15.88:0/93231488 >> v2:172.21.15.154:3300/0 conn(0x7f146015e6f0 0x7f146006fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.935+0000 7f145effd640 1 -- 172.21.15.88:0/93231488 >> v1:172.21.15.88:6789/0 conn(0x7f14600b6d40 legacy=0x7f14600b71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:11.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.935+0000 7f145effd640 1 -- 172.21.15.88:0/93231488 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f146015ef50 con 0x7f14600b54d0 2024-09-16T01:43:11.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.935+0000 7f145e7fc640 1 -- 172.21.15.88:0/93231488 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 602913981 0 0) 0x7f146006e990 con 0x7f14600b6d40 2024-09-16T01:43:11.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.936+0000 7f145effd640 1 --2- 172.21.15.88:0/93231488 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14600b54d0 0x7f146015f750 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f1460069920 tx=0x7f145402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efcb2b46c7e39fa1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:11.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.936+0000 7f145e7fc640 1 -- 172.21.15.88:0/93231488 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1454037070 con 0x7f14600b54d0 2024-09-16T01:43:11.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.936+0000 7f145e7fc640 1 -- 172.21.15.88:0/93231488 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1454032440 con 0x7f14600b54d0 2024-09-16T01:43:11.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.936+0000 7f145e7fc640 1 -- 172.21.15.88:0/93231488 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1454032810 con 0x7f14600b54d0 2024-09-16T01:43:11.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.936+0000 7f1465d5b640 1 -- 172.21.15.88:0/93231488 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14600b54d0 msgr2=0x7f146015f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:11.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.936+0000 7f1465d5b640 1 --2- 172.21.15.88:0/93231488 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14600b54d0 0x7f146015f750 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f1460069920 tx=0x7f145402a270 comp rx=0 tx=0).stop 2024-09-16T01:43:11.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.937+0000 7f1465d5b640 1 -- 172.21.15.88:0/93231488 shutdown_connections 2024-09-16T01:43:11.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.937+0000 7f1465d5b640 1 --2- 172.21.15.88:0/93231488 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14600b54d0 0x7f146015f750 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.937+0000 7f1465d5b640 1 --2- 172.21.15.88:0/93231488 >> v2:172.21.15.154:3300/0 conn(0x7f146015e6f0 0x7f146006fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.937+0000 7f1465d5b640 1 -- 172.21.15.88:0/93231488 >> 172.21.15.88:0/93231488 conn(0x7f14600d2430 msgr2=0x7f14600d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:11.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.937+0000 7f1465d5b640 1 -- 172.21.15.88:0/93231488 shutdown_connections 2024-09-16T01:43:11.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.937+0000 7f1465d5b640 1 -- 172.21.15.88:0/93231488 wait complete. 2024-09-16T01:43:11.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.938+0000 7f1465d5b640 1 Processor -- start 2024-09-16T01:43:11.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.938+0000 7f1465d5b640 1 -- start start 2024-09-16T01:43:11.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.939+0000 7f1465d5b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14600b54d0 0x7f1460160f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.939+0000 7f1465d5b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14600b6d40 0x7f1460150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.939+0000 7f1465d5b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f146015e6f0 0x7f1460159f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.939+0000 7f1465d5b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f146006e730 con 0x7f14600b6d40 2024-09-16T01:43:11.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.939+0000 7f1465d5b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f146006e5b0 con 0x7f14600b54d0 2024-09-16T01:43:11.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.939+0000 7f1465d5b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1460070830 con 0x7f146015e6f0 2024-09-16T01:43:11.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.939+0000 7f145effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14600b6d40 0x7f1460150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.939+0000 7f145effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14600b6d40 0x7f1460150ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58566/0 (socket says 172.21.15.88:58566) 2024-09-16T01:43:11.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.939+0000 7f145effd640 1 -- 172.21.15.88:0/2137455855 learned_addr learned my addr 172.21.15.88:0/2137455855 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:11.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.939+0000 7f145f7fe640 1 --2- 172.21.15.88:0/2137455855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14600b54d0 0x7f1460160f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.939+0000 7f145ffff640 1 --2- 172.21.15.88:0/2137455855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f146015e6f0 0x7f1460159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.939+0000 7f145effd640 1 -- 172.21.15.88:0/2137455855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f146015e6f0 msgr2=0x7f1460159f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:11.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.939+0000 7f145effd640 1 --2- 172.21.15.88:0/2137455855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f146015e6f0 0x7f1460159f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.939+0000 7f145effd640 1 -- 172.21.15.88:0/2137455855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14600b54d0 msgr2=0x7f1460160f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.939+0000 7f145effd640 1 --2- 172.21.15.88:0/2137455855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14600b54d0 0x7f1460160f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.939+0000 7f145effd640 1 -- 172.21.15.88:0/2137455855 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f146015fe60 con 0x7f14600b6d40 2024-09-16T01:43:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.939+0000 7f145f7fe640 1 --2- 172.21.15.88:0/2137455855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14600b54d0 0x7f1460160f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.939+0000 7f145ffff640 1 --2- 172.21.15.88:0/2137455855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f146015e6f0 0x7f1460159f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.939+0000 7f145effd640 1 --2- 172.21.15.88:0/2137455855 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14600b6d40 0x7f1460150ad0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f14540090b0 tx=0x7f14540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.939+0000 7f1447fff640 1 -- 172.21.15.88:0/2137455855 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1454037040 con 0x7f14600b6d40 2024-09-16T01:43:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.939+0000 7f1447fff640 1 -- 172.21.15.88:0/2137455855 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1454004030 con 0x7f14600b6d40 2024-09-16T01:43:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.939+0000 7f1465d5b640 1 -- 172.21.15.88:0/2137455855 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1460160140 con 0x7f14600b6d40 2024-09-16T01:43:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.939+0000 7f1447fff640 1 -- 172.21.15.88:0/2137455855 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1454004400 con 0x7f14600b6d40 2024-09-16T01:43:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.939+0000 7f1465d5b640 1 -- 172.21.15.88:0/2137455855 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1460160700 con 0x7f14600b6d40 2024-09-16T01:43:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.940+0000 7f1465d5b640 1 -- 172.21.15.88:0/2137455855 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f146006c6c0 con 0x7f14600b6d40 2024-09-16T01:43:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.941+0000 7f1447fff640 1 -- 172.21.15.88:0/2137455855 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f145403a070 con 0x7f14600b6d40 2024-09-16T01:43:11.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.941+0000 7f1447fff640 1 --2- 172.21.15.88:0/2137455855 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f144c06c2b0 0x7f144c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:11.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.941+0000 7f1447fff640 1 -- 172.21.15.88:0/2137455855 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(117..117 src has 1..117) ==== 8318+0+0 (secure 0 0 0) 0x7f14540ac500 con 0x7f14600b6d40 2024-09-16T01:43:11.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.941+0000 7f145f7fe640 1 --2- 172.21.15.88:0/2137455855 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f144c06c2b0 0x7f144c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:11.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.942+0000 7f145f7fe640 1 --2- 172.21.15.88:0/2137455855 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f144c06c2b0 0x7f144c06e770 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f1450006fd0 tx=0x7f1450008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:11.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:11.945+0000 7f1447fff640 1 -- 172.21.15.88:0/2137455855 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1454041020 con 0x7f14600b6d40 2024-09-16T01:43:12.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.144+0000 7f1465d5b640 1 -- 172.21.15.88:0/2137455855 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f14601570c0 con 0x7f14600b6d40 2024-09-16T01:43:12.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.428+0000 7f1447fff640 1 -- 172.21.15.88:0/2137455855 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v249) ==== 162+0+0 (secure 0 0 0) 0x7f145407a1f0 con 0x7f14600b6d40 2024-09-16T01:43:12.429 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:43:12.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.430+0000 7f1465d5b640 1 -- 172.21.15.88:0/2137455855 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f144c06c2b0 msgr2=0x7f144c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:12.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.430+0000 7f1465d5b640 1 --2- 172.21.15.88:0/2137455855 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f144c06c2b0 0x7f144c06e770 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f1450006fd0 tx=0x7f1450008040 comp rx=0 tx=0).stop 2024-09-16T01:43:12.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.430+0000 7f1465d5b640 1 -- 172.21.15.88:0/2137455855 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14600b6d40 msgr2=0x7f1460150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:12.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.430+0000 7f1465d5b640 1 --2- 172.21.15.88:0/2137455855 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14600b6d40 0x7f1460150ad0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f14540090b0 tx=0x7f14540046c0 comp rx=0 tx=0).stop 2024-09-16T01:43:12.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.430+0000 7f1465d5b640 1 -- 172.21.15.88:0/2137455855 shutdown_connections 2024-09-16T01:43:12.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.430+0000 7f1465d5b640 1 --2- 172.21.15.88:0/2137455855 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f144c06c2b0 0x7f144c06e770 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:12.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.430+0000 7f1465d5b640 1 --2- 172.21.15.88:0/2137455855 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f146015e6f0 0x7f1460159f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:12.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.430+0000 7f1465d5b640 1 --2- 172.21.15.88:0/2137455855 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14600b6d40 0x7f1460150ad0 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:12.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.430+0000 7f1465d5b640 1 --2- 172.21.15.88:0/2137455855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14600b54d0 0x7f1460160f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:12.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.430+0000 7f1465d5b640 1 -- 172.21.15.88:0/2137455855 >> 172.21.15.88:0/2137455855 conn(0x7f14600d2430 msgr2=0x7f14600d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:12.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.430+0000 7f1465d5b640 1 -- 172.21.15.88:0/2137455855 shutdown_connections 2024-09-16T01:43:12.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.430+0000 7f1465d5b640 1 -- 172.21.15.88:0/2137455855 wait complete. 2024-09-16T01:43:12.443 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:43:12.441+0000 7f355781f640 -1 mds.0.cache.ino(0x100) No data while reading inode 0x100 2024-09-16T01:43:12.443 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:43:12.442+0000 7f355781f640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-09-16T01:43:12.443 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:43:12.442+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:43:12.443 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:43:12.442+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:43:12.445 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:43:12.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.543+0000 7f82ea6d0640 1 Processor -- start 2024-09-16T01:43:12.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.543+0000 7f82ea6d0640 1 -- start start 2024-09-16T01:43:12.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.543+0000 7f82ea6d0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f82e4073130 0x7f82e4073580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:12.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.543+0000 7f82ea6d0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82e40b5220 con 0x7f82e406e780 2024-09-16T01:43:12.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.543+0000 7f82ea6d0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82e40b54c0 con 0x7f82e415e560 2024-09-16T01:43:12.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.543+0000 7f82ea6d0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f82e40aefa0 con 0x7f82e4073130 2024-09-16T01:43:12.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.544+0000 7f82e8c6d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f82e4073130 0x7f82e4073580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:12.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.544+0000 7f82e37fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f82e406e780 0x7f82e40ae890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59366/0 (socket says 172.21.15.88:59366) 2024-09-16T01:43:12.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.544+0000 7f82e37fe640 1 -- 172.21.15.88:0/752267798 learned_addr learned my addr 172.21.15.88:0/752267798 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:12.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.544+0000 7f82e8c6d640 1 -- 172.21.15.88:0/752267798 >> v1:172.21.15.154:6789/0 conn(0x7f82e415e560 legacy=0x7f82e4072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:12.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.544+0000 7f82e8c6d640 1 -- 172.21.15.88:0/752267798 >> v1:172.21.15.88:6789/0 conn(0x7f82e406e780 legacy=0x7f82e40ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:12.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.544+0000 7f82e8c6d640 1 -- 172.21.15.88:0/752267798 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e40b57a0 con 0x7f82e4073130 2024-09-16T01:43:12.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.544+0000 7f82e2ffd640 1 -- 172.21.15.88:0/752267798 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 62740389 0 0) 0x7f82e40b5220 con 0x7f82e406e780 2024-09-16T01:43:12.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.545+0000 7f82e8c6d640 1 --2- 172.21.15.88:0/752267798 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82e4073130 0x7f82e4073580 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009d90 tx=0x7f82dc02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=865f8ae1bbb3690 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:12.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.545+0000 7f82e2ffd640 1 -- 172.21.15.88:0/752267798 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82dc007740 con 0x7f82e4073130 2024-09-16T01:43:12.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.545+0000 7f82e2ffd640 1 -- 172.21.15.88:0/752267798 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82dc0078e0 con 0x7f82e4073130 2024-09-16T01:43:12.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.545+0000 7f82e2ffd640 1 -- 172.21.15.88:0/752267798 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82dc007cb0 con 0x7f82e4073130 2024-09-16T01:43:12.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.545+0000 7f82ea6d0640 1 -- 172.21.15.88:0/752267798 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82e4073130 msgr2=0x7f82e4073580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:12.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.545+0000 7f82ea6d0640 1 --2- 172.21.15.88:0/752267798 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82e4073130 0x7f82e4073580 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009d90 tx=0x7f82dc02a660 comp rx=0 tx=0).stop 2024-09-16T01:43:12.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.545+0000 7f82ea6d0640 1 -- 172.21.15.88:0/752267798 shutdown_connections 2024-09-16T01:43:12.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.545+0000 7f82ea6d0640 1 --2- 172.21.15.88:0/752267798 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82e4073130 0x7f82e4073580 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:12.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.545+0000 7f82ea6d0640 1 -- 172.21.15.88:0/752267798 >> 172.21.15.88:0/752267798 conn(0x7f82e406bc10 msgr2=0x7f82e406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:12.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.545+0000 7f82ea6d0640 1 -- 172.21.15.88:0/752267798 shutdown_connections 2024-09-16T01:43:12.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.545+0000 7f82ea6d0640 1 -- 172.21.15.88:0/752267798 wait complete. 2024-09-16T01:43:12.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.546+0000 7f82ea6d0640 1 Processor -- start 2024-09-16T01:43:12.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.547+0000 7f82ea6d0640 1 -- start start 2024-09-16T01:43:12.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.547+0000 7f82ea6d0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82e406e780 0x7f82e4154650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:12.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.547+0000 7f82ea6d0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82e4073130 0x7f82e40cf020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:12.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.547+0000 7f82ea6d0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82e415e560 0x7f82e4158b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:12.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.547+0000 7f82ea6d0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f82e4073ac0 con 0x7f82e406e780 2024-09-16T01:43:12.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.547+0000 7f82ea6d0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f82e406ebc0 con 0x7f82e4073130 2024-09-16T01:43:12.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.547+0000 7f82ea6d0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f82e40b5d60 con 0x7f82e415e560 2024-09-16T01:43:12.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.547+0000 7f82e3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82e406e780 0x7f82e4154650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:12.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.547+0000 7f82e3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82e406e780 0x7f82e4154650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58582/0 (socket says 172.21.15.88:58582) 2024-09-16T01:43:12.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.547+0000 7f82e3fff640 1 -- 172.21.15.88:0/3378162779 learned_addr learned my addr 172.21.15.88:0/3378162779 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:12.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.548+0000 7f82e37fe640 1 --2- 172.21.15.88:0/3378162779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82e4073130 0x7f82e40cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:12.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.548+0000 7f82e8c6d640 1 --2- 172.21.15.88:0/3378162779 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82e415e560 0x7f82e4158b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:12.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.548+0000 7f82e3fff640 1 -- 172.21.15.88:0/3378162779 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82e415e560 msgr2=0x7f82e4158b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:12.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.548+0000 7f82e3fff640 1 --2- 172.21.15.88:0/3378162779 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82e415e560 0x7f82e4158b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:12.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.548+0000 7f82e3fff640 1 -- 172.21.15.88:0/3378162779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82e4073130 msgr2=0x7f82e40cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:12.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.548+0000 7f82e3fff640 1 --2- 172.21.15.88:0/3378162779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82e4073130 0x7f82e40cf020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:12.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.548+0000 7f82e3fff640 1 -- 172.21.15.88:0/3378162779 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e41567d0 con 0x7f82e406e780 2024-09-16T01:43:12.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.548+0000 7f82e3fff640 1 --2- 172.21.15.88:0/3378162779 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82e406e780 0x7f82e4154650 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f82d400c400 tx=0x7f82d400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:12.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.548+0000 7f82e0ff9640 1 -- 172.21.15.88:0/3378162779 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d4017070 con 0x7f82e406e780 2024-09-16T01:43:12.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.548+0000 7f82e0ff9640 1 -- 172.21.15.88:0/3378162779 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82d40052d0 con 0x7f82e406e780 2024-09-16T01:43:12.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.548+0000 7f82ea6d0640 1 -- 172.21.15.88:0/3378162779 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82e4156ab0 con 0x7f82e406e780 2024-09-16T01:43:12.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.548+0000 7f82e0ff9640 1 -- 172.21.15.88:0/3378162779 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d4003830 con 0x7f82e406e780 2024-09-16T01:43:12.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.548+0000 7f82ea6d0640 1 -- 172.21.15.88:0/3378162779 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82e4157070 con 0x7f82e406e780 2024-09-16T01:43:12.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.549+0000 7f82ea6d0640 1 -- 172.21.15.88:0/3378162779 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82e415e9d0 con 0x7f82e406e780 2024-09-16T01:43:12.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.550+0000 7f82e0ff9640 1 -- 172.21.15.88:0/3378162779 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f82d401a070 con 0x7f82e406e780 2024-09-16T01:43:12.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.550+0000 7f82e0ff9640 1 --2- 172.21.15.88:0/3378162779 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f82c406c2e0 0x7f82c406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:12.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.550+0000 7f82e0ff9640 1 -- 172.21.15.88:0/3378162779 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8404+0+0 (secure 0 0 0) 0x7f82d408b210 con 0x7f82e406e780 2024-09-16T01:43:12.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.550+0000 7f82e37fe640 1 --2- 172.21.15.88:0/3378162779 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f82c406c2e0 0x7f82c406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:12.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.551+0000 7f82e37fe640 1 --2- 172.21.15.88:0/3378162779 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f82c406c2e0 0x7f82c406e7a0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f82cc007f70 tx=0x7f82cc007f00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:12.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.554+0000 7f82e0ff9640 1 -- 172.21.15.88:0/3378162779 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f82d4058850 con 0x7f82e406e780 2024-09-16T01:43:12.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.757+0000 7f82ea6d0640 1 -- 172.21.15.88:0/3378162779 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82e415a550 con 0x7f82e406e780 2024-09-16T01:43:12.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.757+0000 7f82e0ff9640 1 -- 172.21.15.88:0/3378162779 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 251 v251) ==== 95+0+3566 (secure 0 0 0) 0x7f82d405cb50 con 0x7f82e406e780 2024-09-16T01:43:12.759 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:12.759 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":251,"btime":"2024-09-16T01:43:12:443544+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":251,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:43:12.443489+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:43:12.759 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 251 2024-09-16T01:43:12.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.759+0000 7f82ea6d0640 1 -- 172.21.15.88:0/3378162779 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f82c406c2e0 msgr2=0x7f82c406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:12.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.759+0000 7f82ea6d0640 1 --2- 172.21.15.88:0/3378162779 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f82c406c2e0 0x7f82c406e7a0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f82cc007f70 tx=0x7f82cc007f00 comp rx=0 tx=0).stop 2024-09-16T01:43:12.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.759+0000 7f82ea6d0640 1 -- 172.21.15.88:0/3378162779 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82e406e780 msgr2=0x7f82e4154650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:12.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.759+0000 7f82ea6d0640 1 --2- 172.21.15.88:0/3378162779 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82e406e780 0x7f82e4154650 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f82d400c400 tx=0x7f82d400c8c0 comp rx=0 tx=0).stop 2024-09-16T01:43:12.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.759+0000 7f82ea6d0640 1 -- 172.21.15.88:0/3378162779 shutdown_connections 2024-09-16T01:43:12.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.759+0000 7f82ea6d0640 1 --2- 172.21.15.88:0/3378162779 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f82c406c2e0 0x7f82c406e7a0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:12.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.759+0000 7f82ea6d0640 1 --2- 172.21.15.88:0/3378162779 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82e415e560 0x7f82e4158b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:12.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.759+0000 7f82ea6d0640 1 --2- 172.21.15.88:0/3378162779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82e4073130 0x7f82e40cf020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:12.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.759+0000 7f82ea6d0640 1 --2- 172.21.15.88:0/3378162779 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82e406e780 0x7f82e4154650 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:12.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.759+0000 7f82ea6d0640 1 -- 172.21.15.88:0/3378162779 >> 172.21.15.88:0/3378162779 conn(0x7f82e406bc10 msgr2=0x7f82e40b6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:12.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.760+0000 7f82ea6d0640 1 -- 172.21.15.88:0/3378162779 shutdown_connections 2024-09-16T01:43:12.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.760+0000 7f82ea6d0640 1 -- 172.21.15.88:0/3378162779 wait complete. 2024-09-16T01:43:12.772 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:43:12.772 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:43:12.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.871+0000 7f7344f26640 1 Processor -- start 2024-09-16T01:43:12.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.871+0000 7f7344f26640 1 -- start start 2024-09-16T01:43:12.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.872+0000 7f7344f26640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f73400b4d00 0x7f73400b5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:12.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.872+0000 7f7344f26640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f73400cd780 0x7f73400cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:12.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.872+0000 7f7344f26640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73400ce4f0 con 0x7f73400b4920 2024-09-16T01:43:12.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.872+0000 7f7344f26640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f73400cf100 con 0x7f73400b4d00 2024-09-16T01:43:12.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.872+0000 7f7344f26640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f73400ce7d0 con 0x7f73400cd780 2024-09-16T01:43:12.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.872+0000 7f733dd9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f73400b4d00 0x7f73400b5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:12.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.872+0000 7f733dd9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f73400b4d00 0x7f73400b5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58596/0 (socket says 172.21.15.88:58596) 2024-09-16T01:43:12.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.872+0000 7f733dd9b640 1 -- 172.21.15.88:0/4035138066 learned_addr learned my addr 172.21.15.88:0/4035138066 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:12.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.872+0000 7f733dd9b640 1 -- 172.21.15.88:0/4035138066 >> v2:172.21.15.154:3300/0 conn(0x7f73400cd780 msgr2=0x7f73400cdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:12.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.872+0000 7f733dd9b640 1 --2- 172.21.15.88:0/4035138066 >> v2:172.21.15.154:3300/0 conn(0x7f73400cd780 0x7f73400cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:12.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.872+0000 7f733dd9b640 1 -- 172.21.15.88:0/4035138066 >> v1:172.21.15.154:6789/0 conn(0x7f73400b4920 legacy=0x7f734015b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:12.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.872+0000 7f733dd9b640 1 -- 172.21.15.88:0/4035138066 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73400ceab0 con 0x7f73400b4d00 2024-09-16T01:43:12.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.873+0000 7f733dd9b640 1 --2- 172.21.15.88:0/4035138066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73400b4d00 0x7f73400b5170 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f73400b34b0 tx=0x7f733402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c24a0a080ca4711 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:12.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.873+0000 7f733d59a640 1 -- 172.21.15.88:0/4035138066 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7334037070 con 0x7f73400b4d00 2024-09-16T01:43:12.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.873+0000 7f733d59a640 1 -- 172.21.15.88:0/4035138066 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7334032440 con 0x7f73400b4d00 2024-09-16T01:43:12.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.873+0000 7f733d59a640 1 -- 172.21.15.88:0/4035138066 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7334032810 con 0x7f73400b4d00 2024-09-16T01:43:12.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.874+0000 7f7344f26640 1 -- 172.21.15.88:0/4035138066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73400b4d00 msgr2=0x7f73400b5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:12.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.874+0000 7f7344f26640 1 --2- 172.21.15.88:0/4035138066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73400b4d00 0x7f73400b5170 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f73400b34b0 tx=0x7f733402a230 comp rx=0 tx=0).stop 2024-09-16T01:43:12.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.874+0000 7f7344f26640 1 -- 172.21.15.88:0/4035138066 shutdown_connections 2024-09-16T01:43:12.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.874+0000 7f7344f26640 1 --2- 172.21.15.88:0/4035138066 >> v2:172.21.15.154:3300/0 conn(0x7f73400cd780 0x7f73400cdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:12.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.874+0000 7f7344f26640 1 --2- 172.21.15.88:0/4035138066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73400b4d00 0x7f73400b5170 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:12.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.874+0000 7f7344f26640 1 -- 172.21.15.88:0/4035138066 >> 172.21.15.88:0/4035138066 conn(0x7f734006bc10 msgr2=0x7f734006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:12.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.874+0000 7f7344f26640 1 -- 172.21.15.88:0/4035138066 shutdown_connections 2024-09-16T01:43:12.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.874+0000 7f7344f26640 1 -- 172.21.15.88:0/4035138066 wait complete. 2024-09-16T01:43:12.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.875+0000 7f7344f26640 1 Processor -- start 2024-09-16T01:43:12.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.875+0000 7f7344f26640 1 -- start start 2024-09-16T01:43:12.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.876+0000 7f7344f26640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73400b4920 0x7f73400c9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:12.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.876+0000 7f7344f26640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73400b4d00 0x7f73400b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:12.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.876+0000 7f7344f26640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73400cd780 0x7f73400cc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:12.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.876+0000 7f7344f26640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f73400ce0d0 con 0x7f73400b4920 2024-09-16T01:43:12.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.876+0000 7f7344f26640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f73400b56b0 con 0x7f73400cd780 2024-09-16T01:43:12.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.876+0000 7f7344f26640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f73400ce250 con 0x7f73400b4d00 2024-09-16T01:43:12.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.876+0000 7f733e59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73400b4920 0x7f73400c9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:12.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.876+0000 7f733e59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73400b4920 0x7f73400c9fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58610/0 (socket says 172.21.15.88:58610) 2024-09-16T01:43:12.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.876+0000 7f733e59c640 1 -- 172.21.15.88:0/369659245 learned_addr learned my addr 172.21.15.88:0/369659245 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:12.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.876+0000 7f733ed9d640 1 --2- 172.21.15.88:0/369659245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73400cd780 0x7f73400cc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:12.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.876+0000 7f733dd9b640 1 --2- 172.21.15.88:0/369659245 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73400b4d00 0x7f73400b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:12.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.876+0000 7f733e59c640 1 -- 172.21.15.88:0/369659245 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73400b4d00 msgr2=0x7f73400b1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:12.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.876+0000 7f733e59c640 1 --2- 172.21.15.88:0/369659245 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73400b4d00 0x7f73400b1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:12.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.876+0000 7f733e59c640 1 -- 172.21.15.88:0/369659245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73400cd780 msgr2=0x7f73400cc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:12.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.876+0000 7f733e59c640 1 --2- 172.21.15.88:0/369659245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73400cd780 0x7f73400cc2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:12.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.876+0000 7f733e59c640 1 -- 172.21.15.88:0/369659245 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73400b2160 con 0x7f73400b4920 2024-09-16T01:43:12.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.876+0000 7f733dd9b640 1 --2- 172.21.15.88:0/369659245 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73400b4d00 0x7f73400b1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:12.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.876+0000 7f733e59c640 1 --2- 172.21.15.88:0/369659245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73400b4920 0x7f73400c9fd0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f733000c410 tx=0x7f733000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:12.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.876+0000 7f732effd640 1 -- 172.21.15.88:0/369659245 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7330017070 con 0x7f73400b4920 2024-09-16T01:43:12.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.876+0000 7f732effd640 1 -- 172.21.15.88:0/369659245 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73300035f0 con 0x7f73400b4920 2024-09-16T01:43:12.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.876+0000 7f7344f26640 1 -- 172.21.15.88:0/369659245 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73400c6f60 con 0x7f73400b4920 2024-09-16T01:43:12.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.876+0000 7f732effd640 1 -- 172.21.15.88:0/369659245 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7330003a20 con 0x7f73400b4920 2024-09-16T01:43:12.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.877+0000 7f7344f26640 1 -- 172.21.15.88:0/369659245 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73400c7520 con 0x7f73400b4920 2024-09-16T01:43:12.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.877+0000 7f7344f26640 1 -- 172.21.15.88:0/369659245 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73400b5170 con 0x7f73400b4920 2024-09-16T01:43:12.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.878+0000 7f732effd640 1 -- 172.21.15.88:0/369659245 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f733001a070 con 0x7f73400b4920 2024-09-16T01:43:12.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.878+0000 7f732effd640 1 --2- 172.21.15.88:0/369659245 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f732006c2b0 0x7f732006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:12.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.879+0000 7f732effd640 1 -- 172.21.15.88:0/369659245 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8404+0+0 (secure 0 0 0) 0x7f733008b470 con 0x7f73400b4920 2024-09-16T01:43:12.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.879+0000 7f733dd9b640 1 --2- 172.21.15.88:0/369659245 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f732006c2b0 0x7f732006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:12.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.879+0000 7f733dd9b640 1 --2- 172.21.15.88:0/369659245 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f732006c2b0 0x7f732006e770 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f7334034000 tx=0x7f733402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:12.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:12.883+0000 7f732effd640 1 -- 172.21.15.88:0/369659245 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7330059070 con 0x7f73400b4920 2024-09-16T01:43:13.031 INFO:tasks.ceph.mds.a.smithi088.stderr: -43> 2024-09-16T01:43:08.278+0000 7f598d015640 -1 log_channel(cluster) log [ERR] : failed to read JournalPointer: -22 ((22) Invalid argument) 2024-09-16T01:43:13.054 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:43:13.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.084+0000 7f7344f26640 1 -- 172.21.15.88:0/369659245 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f734015e310 con 0x7f73400b4920 2024-09-16T01:43:13.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.084+0000 7f732effd640 1 -- 172.21.15.88:0/369659245 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 251 v251) ==== 95+0+3566 (secure 0 0 0) 0x7f7330058970 con 0x7f73400b4920 2024-09-16T01:43:13.086 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:13.086 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":251,"btime":"2024-09-16T01:43:12:443544+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":251,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:43:12.443489+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:43:13.086 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 251 2024-09-16T01:43:13.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.086+0000 7f7344f26640 1 -- 172.21.15.88:0/369659245 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f732006c2b0 msgr2=0x7f732006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:13.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.086+0000 7f7344f26640 1 --2- 172.21.15.88:0/369659245 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f732006c2b0 0x7f732006e770 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f7334034000 tx=0x7f733402e000 comp rx=0 tx=0).stop 2024-09-16T01:43:13.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.086+0000 7f7344f26640 1 -- 172.21.15.88:0/369659245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73400b4920 msgr2=0x7f73400c9fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:13.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.086+0000 7f7344f26640 1 --2- 172.21.15.88:0/369659245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73400b4920 0x7f73400c9fd0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f733000c410 tx=0x7f733000c8d0 comp rx=0 tx=0).stop 2024-09-16T01:43:13.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.086+0000 7f7344f26640 1 -- 172.21.15.88:0/369659245 shutdown_connections 2024-09-16T01:43:13.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.086+0000 7f7344f26640 1 --2- 172.21.15.88:0/369659245 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f732006c2b0 0x7f732006e770 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:13.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.086+0000 7f7344f26640 1 --2- 172.21.15.88:0/369659245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73400cd780 0x7f73400cc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:13.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.086+0000 7f7344f26640 1 --2- 172.21.15.88:0/369659245 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73400b4d00 0x7f73400b1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:13.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.086+0000 7f7344f26640 1 --2- 172.21.15.88:0/369659245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f73400b4920 0x7f73400c9fd0 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:13.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.086+0000 7f7344f26640 1 -- 172.21.15.88:0/369659245 >> 172.21.15.88:0/369659245 conn(0x7f734006bc10 msgr2=0x7f7340072050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:13.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.086+0000 7f7344f26640 1 -- 172.21.15.88:0/369659245 shutdown_connections 2024-09-16T01:43:13.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.086+0000 7f7344f26640 1 -- 172.21.15.88:0/369659245 wait complete. 2024-09-16T01:43:13.098 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:43:13.099 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:43:13.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.196+0000 7fbd07f11640 1 Processor -- start 2024-09-16T01:43:13.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.196+0000 7fbd07f11640 1 -- start start 2024-09-16T01:43:13.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.197+0000 7fbd07f11640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbd00163a10 0x7fbd000d0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:13.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.197+0000 7fbd07f11640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd000ce030 con 0x7fbd00165740 2024-09-16T01:43:13.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.197+0000 7fbd07f11640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd000ce310 con 0x7fbd00162f20 2024-09-16T01:43:13.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.197+0000 7fbd07f11640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbd000ce5f0 con 0x7fbd00163a10 2024-09-16T01:43:13.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.197+0000 7fbd054ac640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fbd00165740 0x7fbd0016a4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59380/0 (socket says 172.21.15.88:59380) 2024-09-16T01:43:13.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.197+0000 7fbd054ac640 1 -- 172.21.15.88:0/3652658444 learned_addr learned my addr 172.21.15.88:0/3652658444 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:13.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.197+0000 7fbd064ae640 1 --2- 172.21.15.88:0/3652658444 >> v2:172.21.15.154:3300/0 conn(0x7fbd00163a10 0x7fbd000d0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:13.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.198+0000 7fbd04cab640 1 -- 172.21.15.88:0/3652658444 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3344954834 0 0) 0x7fbd000ce030 con 0x7fbd00165740 2024-09-16T01:43:13.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.198+0000 7fbd04cab640 1 -- 172.21.15.88:0/3652658444 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbce8003520 con 0x7fbd00165740 2024-09-16T01:43:13.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.198+0000 7fbd04cab640 1 -- 172.21.15.88:0/3652658444 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1497598421 0 0) 0x7fbce8003520 con 0x7fbd00165740 2024-09-16T01:43:13.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.198+0000 7fbd04cab640 1 -- 172.21.15.88:0/3652658444 >> v2:172.21.15.154:3300/0 conn(0x7fbd00163a10 msgr2=0x7fbd000d0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:13.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.198+0000 7fbd04cab640 1 --2- 172.21.15.88:0/3652658444 >> v2:172.21.15.154:3300/0 conn(0x7fbd00163a10 0x7fbd000d0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:13.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.198+0000 7fbd04cab640 1 -- 172.21.15.88:0/3652658444 >> v1:172.21.15.154:6789/0 conn(0x7fbd00162f20 legacy=0x7fbd00163300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:13.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.198+0000 7fbd04cab640 1 -- 172.21.15.88:0/3652658444 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd00164210 con 0x7fbd00165740 2024-09-16T01:43:13.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.198+0000 7fbd04cab640 1 -- 172.21.15.88:0/3652658444 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fbcf4001670 con 0x7fbd00165740 2024-09-16T01:43:13.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.198+0000 7fbd04cab640 1 -- 172.21.15.88:0/3652658444 >> v1:172.21.15.88:6789/0 conn(0x7fbd00165740 legacy=0x7fbd0016a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:13.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.198+0000 7fbd04cab640 1 --2- 172.21.15.88:0/3652658444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbce80039b0 0x7fbce8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:13.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.198+0000 7fbd04cab640 1 --2- 172.21.15.88:0/3652658444 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbce8004800 0x7fbce8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:13.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.198+0000 7fbd04cab640 1 --2- 172.21.15.88:0/3652658444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbce80051c0 0x7fbce800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:13.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.198+0000 7fbd04cab640 1 -- 172.21.15.88:0/3652658444 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbd000ce5f0 con 0x7fbce80051c0 2024-09-16T01:43:13.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.198+0000 7fbd04cab640 1 -- 172.21.15.88:0/3652658444 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbce800c060 con 0x7fbce80039b0 2024-09-16T01:43:13.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.198+0000 7fbd04cab640 1 -- 172.21.15.88:0/3652658444 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbce800c260 con 0x7fbce8004800 2024-09-16T01:43:13.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.198+0000 7fbd064ae640 1 --2- 172.21.15.88:0/3652658444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbce80051c0 0x7fbce800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:13.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.198+0000 7fbd05cad640 1 --2- 172.21.15.88:0/3652658444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbce80039b0 0x7fbce8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:13.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.198+0000 7fbd054ac640 1 --2- 172.21.15.88:0/3652658444 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbce8004800 0x7fbce8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:13.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.198+0000 7fbd064ae640 1 -- 172.21.15.88:0/3652658444 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbce8004800 msgr2=0x7fbce8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:13.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.198+0000 7fbd064ae640 1 --2- 172.21.15.88:0/3652658444 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbce8004800 0x7fbce8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:13.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.198+0000 7fbd064ae640 1 -- 172.21.15.88:0/3652658444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbce80039b0 msgr2=0x7fbce8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:13.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.198+0000 7fbd064ae640 1 --2- 172.21.15.88:0/3652658444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbce80039b0 0x7fbce8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:13.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.198+0000 7fbd064ae640 1 -- 172.21.15.88:0/3652658444 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce8003520 con 0x7fbce80051c0 2024-09-16T01:43:13.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.198+0000 7fbd05cad640 1 --2- 172.21.15.88:0/3652658444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbce80039b0 0x7fbce8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:13.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.199+0000 7fbd064ae640 1 -- 172.21.15.88:0/3652658444 reap_dead start 2024-09-16T01:43:13.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.199+0000 7fbd064ae640 1 --2- 172.21.15.88:0/3652658444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbce80051c0 0x7fbce800b9c0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8009ac0 tx=0x7fbce8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=291d5a86e7cfed7f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:13.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.199+0000 7fbd04cab640 1 -- 172.21.15.88:0/3652658444 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf8009690 con 0x7fbce80051c0 2024-09-16T01:43:13.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.199+0000 7fbd04cab640 1 -- 172.21.15.88:0/3652658444 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcf8004020 con 0x7fbce80051c0 2024-09-16T01:43:13.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.199+0000 7fbd04cab640 1 -- 172.21.15.88:0/3652658444 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf8004410 con 0x7fbce80051c0 2024-09-16T01:43:13.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.200+0000 7fbd07f11640 1 -- 172.21.15.88:0/3652658444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbce80051c0 msgr2=0x7fbce800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:13.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.200+0000 7fbd07f11640 1 --2- 172.21.15.88:0/3652658444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbce80051c0 0x7fbce800b9c0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8009ac0 tx=0x7fbce8005830 comp rx=0 tx=0).stop 2024-09-16T01:43:13.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.200+0000 7fbd07f11640 1 -- 172.21.15.88:0/3652658444 shutdown_connections 2024-09-16T01:43:13.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.200+0000 7fbd07f11640 1 --2- 172.21.15.88:0/3652658444 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbce80051c0 0x7fbce800b9c0 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:13.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.200+0000 7fbd07f11640 1 -- 172.21.15.88:0/3652658444 >> 172.21.15.88:0/3652658444 conn(0x7fbd0006bc10 msgr2=0x7fbd0006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:13.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.200+0000 7fbd07f11640 1 -- 172.21.15.88:0/3652658444 shutdown_connections 2024-09-16T01:43:13.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.200+0000 7fbd07f11640 1 -- 172.21.15.88:0/3652658444 wait complete. 2024-09-16T01:43:13.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.201+0000 7fbd07f11640 1 Processor -- start 2024-09-16T01:43:13.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.201+0000 7fbd07f11640 1 -- start start 2024-09-16T01:43:13.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.202+0000 7fbd07f11640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbce80051c0 0x7fbd0014af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:13.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.202+0000 7fbd07f11640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd00143ec0 0x7fbd00145600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:13.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.202+0000 7fbd07f11640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd00145b40 0x7fbd001f82b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:13.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.202+0000 7fbd07f11640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbd000d0190 con 0x7fbd00143ec0 2024-09-16T01:43:13.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.202+0000 7fbd07f11640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbd000d0010 con 0x7fbd00145b40 2024-09-16T01:43:13.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.202+0000 7fbd07f11640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbd000d0310 con 0x7fbce80051c0 2024-09-16T01:43:13.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.202+0000 7fbd054ac640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd00143ec0 0x7fbd00145600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:13.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.202+0000 7fbd054ac640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd00143ec0 0x7fbd00145600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58648/0 (socket says 172.21.15.88:58648) 2024-09-16T01:43:13.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.202+0000 7fbd054ac640 1 -- 172.21.15.88:0/911837554 learned_addr learned my addr 172.21.15.88:0/911837554 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:13.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.202+0000 7fbd064ae640 1 --2- 172.21.15.88:0/911837554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd00145b40 0x7fbd001f82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:13.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.202+0000 7fbd05cad640 1 --2- 172.21.15.88:0/911837554 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbce80051c0 0x7fbd0014af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:13.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.202+0000 7fbd054ac640 1 -- 172.21.15.88:0/911837554 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbce80051c0 msgr2=0x7fbd0014af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:13.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.202+0000 7fbd054ac640 1 --2- 172.21.15.88:0/911837554 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbce80051c0 0x7fbd0014af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:13.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.202+0000 7fbd054ac640 1 -- 172.21.15.88:0/911837554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd00145b40 msgr2=0x7fbd001f82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:13.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.202+0000 7fbd054ac640 1 --2- 172.21.15.88:0/911837554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd00145b40 0x7fbd001f82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:13.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.202+0000 7fbd054ac640 1 -- 172.21.15.88:0/911837554 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd00143100 con 0x7fbd00143ec0 2024-09-16T01:43:13.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.202+0000 7fbd054ac640 1 --2- 172.21.15.88:0/911837554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd00143ec0 0x7fbd00145600 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fbcf400e760 tx=0x7fbcf400ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:13.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.202+0000 7fbcfe7fc640 1 -- 172.21.15.88:0/911837554 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf400c6c0 con 0x7fbd00143ec0 2024-09-16T01:43:13.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.202+0000 7fbcfe7fc640 1 -- 172.21.15.88:0/911837554 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcf4018070 con 0x7fbd00143ec0 2024-09-16T01:43:13.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.202+0000 7fbd07f11640 1 -- 172.21.15.88:0/911837554 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd001433e0 con 0x7fbd00143ec0 2024-09-16T01:43:13.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.202+0000 7fbcfe7fc640 1 -- 172.21.15.88:0/911837554 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf400caa0 con 0x7fbd00143ec0 2024-09-16T01:43:13.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.202+0000 7fbd07f11640 1 -- 172.21.15.88:0/911837554 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd001f9140 con 0x7fbd00143ec0 2024-09-16T01:43:13.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.203+0000 7fbd07f11640 1 -- 172.21.15.88:0/911837554 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd000cd8c0 con 0x7fbd00143ec0 2024-09-16T01:43:13.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.204+0000 7fbcfe7fc640 1 -- 172.21.15.88:0/911837554 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fbcf400cc40 con 0x7fbd00143ec0 2024-09-16T01:43:13.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.204+0000 7fbcfe7fc640 1 --2- 172.21.15.88:0/911837554 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbcec06c2b0 0x7fbcec06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:13.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.204+0000 7fbcfe7fc640 1 -- 172.21.15.88:0/911837554 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8404+0+0 (secure 0 0 0) 0x7fbcf408bb00 con 0x7fbd00143ec0 2024-09-16T01:43:13.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.204+0000 7fbd05cad640 1 --2- 172.21.15.88:0/911837554 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbcec06c2b0 0x7fbcec06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:13.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.205+0000 7fbd05cad640 1 --2- 172.21.15.88:0/911837554 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbcec06c2b0 0x7fbcec06e770 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fbcf0000f80 tx=0x7fbcf0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:13.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.208+0000 7fbcfe7fc640 1 -- 172.21.15.88:0/911837554 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fbcf40591d0 con 0x7fbd00143ec0 2024-09-16T01:43:13.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.406+0000 7fbd07f11640 1 -- 172.21.15.88:0/911837554 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd00072bd0 con 0x7fbd00143ec0 2024-09-16T01:43:13.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.406+0000 7fbcfe7fc640 1 -- 172.21.15.88:0/911837554 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 251 v251) ==== 95+0+3566 (secure 0 0 0) 0x7fbcf405d4d0 con 0x7fbd00143ec0 2024-09-16T01:43:13.408 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:13.408 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":251,"btime":"2024-09-16T01:43:12:443544+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":251,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:43:12.443489+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:43:13.408 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 251 2024-09-16T01:43:13.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.408+0000 7fbd07f11640 1 -- 172.21.15.88:0/911837554 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbcec06c2b0 msgr2=0x7fbcec06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:13.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.408+0000 7fbd07f11640 1 --2- 172.21.15.88:0/911837554 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbcec06c2b0 0x7fbcec06e770 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fbcf0000f80 tx=0x7fbcf0009040 comp rx=0 tx=0).stop 2024-09-16T01:43:13.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.408+0000 7fbd07f11640 1 -- 172.21.15.88:0/911837554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd00143ec0 msgr2=0x7fbd00145600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:13.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.408+0000 7fbd07f11640 1 --2- 172.21.15.88:0/911837554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd00143ec0 0x7fbd00145600 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fbcf400e760 tx=0x7fbcf400ec20 comp rx=0 tx=0).stop 2024-09-16T01:43:13.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.408+0000 7fbd07f11640 1 -- 172.21.15.88:0/911837554 shutdown_connections 2024-09-16T01:43:13.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.408+0000 7fbd07f11640 1 --2- 172.21.15.88:0/911837554 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbcec06c2b0 0x7fbcec06e770 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:13.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.408+0000 7fbd07f11640 1 --2- 172.21.15.88:0/911837554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd00145b40 0x7fbd001f82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:13.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.408+0000 7fbd07f11640 1 --2- 172.21.15.88:0/911837554 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbd00143ec0 0x7fbd00145600 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:13.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.408+0000 7fbd07f11640 1 --2- 172.21.15.88:0/911837554 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbce80051c0 0x7fbd0014af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:13.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.408+0000 7fbd07f11640 1 -- 172.21.15.88:0/911837554 >> 172.21.15.88:0/911837554 conn(0x7fbd0006bc10 msgr2=0x7fbd00070f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:13.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.408+0000 7fbd07f11640 1 -- 172.21.15.88:0/911837554 shutdown_connections 2024-09-16T01:43:13.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.408+0000 7fbd07f11640 1 -- 172.21.15.88:0/911837554 wait complete. 2024-09-16T01:43:13.421 INFO:tasks.cephfs.test_damage:Result: Mutation 'Truncate 100.00000000.inode' led to DAMAGED state 2024-09-16T01:43:13.421 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap key 10000000000.00000000:sixmegs_head' 2024-09-16T01:43:13.422 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:43:13.422 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:43:13.437 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:43:13.438 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:43:13.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:13.438 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:43:13.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:43:13.495 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:43:13.495 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:43:13.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.635+0000 7f329b223640 1 Processor -- start 2024-09-16T01:43:13.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.635+0000 7f329b223640 1 -- start start 2024-09-16T01:43:13.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.636+0000 7f329b223640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f329415b480 0x7f32940b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:13.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.636+0000 7f329b223640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f32940b60c0 0x7f3294071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:13.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.636+0000 7f329b223640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3294072530 0x7f3294070fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:13.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.636+0000 7f329b223640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f32940b6500 con 0x7f32940b60c0 2024-09-16T01:43:13.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.636+0000 7f329b223640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3294072990 con 0x7f3294072530 2024-09-16T01:43:13.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.636+0000 7f329b223640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f32940ccff0 con 0x7f329415b480 2024-09-16T01:43:13.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.636+0000 7f328bfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f32940b60c0 0x7f3294071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:13.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.636+0000 7f328bfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f32940b60c0 0x7f3294071ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58658/0 (socket says 172.21.15.88:58658) 2024-09-16T01:43:13.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.636+0000 7f328bfff640 1 -- 172.21.15.88:0/941991031 learned_addr learned my addr 172.21.15.88:0/941991031 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:13.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.636+0000 7f32997c0640 1 --2- 172.21.15.88:0/941991031 >> v2:172.21.15.154:3300/0 conn(0x7f3294072530 0x7f3294070fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:13.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.636+0000 7f3298fbf640 1 --2- 172.21.15.88:0/941991031 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f329415b480 0x7f32940b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:13.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.637+0000 7f328bfff640 1 -- 172.21.15.88:0/941991031 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f329415b480 msgr2=0x7f32940b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:13.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.637+0000 7f328bfff640 1 --2- 172.21.15.88:0/941991031 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f329415b480 0x7f32940b5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:13.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.637+0000 7f328bfff640 1 -- 172.21.15.88:0/941991031 >> v2:172.21.15.154:3300/0 conn(0x7f3294072530 msgr2=0x7f3294070fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:13.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.637+0000 7f328bfff640 1 --2- 172.21.15.88:0/941991031 >> v2:172.21.15.154:3300/0 conn(0x7f3294072530 0x7f3294070fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:13.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.637+0000 7f328bfff640 1 -- 172.21.15.88:0/941991031 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32940cd2d0 con 0x7f32940b60c0 2024-09-16T01:43:13.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.637+0000 7f328bfff640 1 --2- 172.21.15.88:0/941991031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32940b60c0 0x7f3294071ff0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f32940b2eb0 tx=0x7f328402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f45311692206e62 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:13.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.637+0000 7f328b7fe640 1 -- 172.21.15.88:0/941991031 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3284037070 con 0x7f32940b60c0 2024-09-16T01:43:13.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.637+0000 7f328b7fe640 1 -- 172.21.15.88:0/941991031 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3284032440 con 0x7f32940b60c0 2024-09-16T01:43:13.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.637+0000 7f328b7fe640 1 -- 172.21.15.88:0/941991031 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3284032810 con 0x7f32940b60c0 2024-09-16T01:43:13.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.638+0000 7f329b223640 1 -- 172.21.15.88:0/941991031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32940b60c0 msgr2=0x7f3294071ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:13.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.638+0000 7f329b223640 1 --2- 172.21.15.88:0/941991031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32940b60c0 0x7f3294071ff0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f32940b2eb0 tx=0x7f328402a180 comp rx=0 tx=0).stop 2024-09-16T01:43:13.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.638+0000 7f329b223640 1 -- 172.21.15.88:0/941991031 shutdown_connections 2024-09-16T01:43:13.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.638+0000 7f329b223640 1 --2- 172.21.15.88:0/941991031 >> v2:172.21.15.154:3300/0 conn(0x7f3294072530 0x7f3294070fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:13.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.638+0000 7f329b223640 1 --2- 172.21.15.88:0/941991031 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f32940b60c0 0x7f3294071ff0 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:13.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.638+0000 7f329b223640 1 --2- 172.21.15.88:0/941991031 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f329415b480 0x7f32940b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:13.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.638+0000 7f329b223640 1 -- 172.21.15.88:0/941991031 >> 172.21.15.88:0/941991031 conn(0x7f329406d0f0 msgr2=0x7f329406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:13.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.638+0000 7f329b223640 1 -- 172.21.15.88:0/941991031 shutdown_connections 2024-09-16T01:43:13.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.638+0000 7f329b223640 1 -- 172.21.15.88:0/941991031 wait complete. 2024-09-16T01:43:13.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.639+0000 7f329b223640 1 Processor -- start 2024-09-16T01:43:13.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.639+0000 7f329b223640 1 -- start start 2024-09-16T01:43:13.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.640+0000 7f329b223640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3294072530 0x7f329415d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:13.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.640+0000 7f329b223640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32940b60c0 0x7f3294146590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:13.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.640+0000 7f329b223640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f329415b480 0x7f3294137b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:13.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.640+0000 7f329b223640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f32940cdb70 con 0x7f3294072530 2024-09-16T01:43:13.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.640+0000 7f329b223640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f32940cd890 con 0x7f32940b60c0 2024-09-16T01:43:13.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.640+0000 7f329b223640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f329415b860 con 0x7f329415b480 2024-09-16T01:43:13.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.640+0000 7f3298fbf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3294072530 0x7f329415d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:13.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.640+0000 7f3298fbf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3294072530 0x7f329415d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58662/0 (socket says 172.21.15.88:58662) 2024-09-16T01:43:13.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.640+0000 7f3298fbf640 1 -- 172.21.15.88:0/3710811399 learned_addr learned my addr 172.21.15.88:0/3710811399 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:13.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.640+0000 7f3298fbf640 1 -- 172.21.15.88:0/3710811399 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f329415b480 msgr2=0x7f3294137b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:13.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.640+0000 7f3298fbf640 1 --2- 172.21.15.88:0/3710811399 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f329415b480 0x7f3294137b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:13.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.640+0000 7f3298fbf640 1 -- 172.21.15.88:0/3710811399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32940b60c0 msgr2=0x7f3294146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:13.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.640+0000 7f3298fbf640 1 --2- 172.21.15.88:0/3710811399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32940b60c0 0x7f3294146590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:13.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.640+0000 7f3298fbf640 1 -- 172.21.15.88:0/3710811399 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3294149440 con 0x7f3294072530 2024-09-16T01:43:13.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.641+0000 7f3298fbf640 1 --2- 172.21.15.88:0/3710811399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3294072530 0x7f329415d030 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f329000d620 tx=0x7f329000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:13.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.641+0000 7f32897fa640 1 -- 172.21.15.88:0/3710811399 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f329000b620 con 0x7f3294072530 2024-09-16T01:43:13.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.641+0000 7f32897fa640 1 -- 172.21.15.88:0/3710811399 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3290019070 con 0x7f3294072530 2024-09-16T01:43:13.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.641+0000 7f329b223640 1 -- 172.21.15.88:0/3710811399 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32941496e0 con 0x7f3294072530 2024-09-16T01:43:13.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.641+0000 7f32897fa640 1 -- 172.21.15.88:0/3710811399 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f329000ba00 con 0x7f3294072530 2024-09-16T01:43:13.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.641+0000 7f329b223640 1 -- 172.21.15.88:0/3710811399 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3294149ca0 con 0x7f3294072530 2024-09-16T01:43:13.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.642+0000 7f329b223640 1 -- 172.21.15.88:0/3710811399 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3294071ff0 con 0x7f3294072530 2024-09-16T01:43:13.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.642+0000 7f32897fa640 1 -- 172.21.15.88:0/3710811399 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f329000bba0 con 0x7f3294072530 2024-09-16T01:43:13.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.643+0000 7f32897fa640 1 --2- 172.21.15.88:0/3710811399 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f328006c2e0 0x7f328006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:13.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.643+0000 7f32897fa640 1 -- 172.21.15.88:0/3710811399 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8404+0+0 (secure 0 0 0) 0x7f329008c2e0 con 0x7f3294072530 2024-09-16T01:43:13.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.643+0000 7f328bfff640 1 --2- 172.21.15.88:0/3710811399 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f328006c2e0 0x7f328006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:13.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.643+0000 7f328bfff640 1 --2- 172.21.15.88:0/3710811399 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f328006c2e0 0x7f328006e7a0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f3284034000 tx=0x7f328402c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:13.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.647+0000 7f32897fa640 1 -- 172.21.15.88:0/3710811399 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3290059f70 con 0x7f3294072530 2024-09-16T01:43:13.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:13.845+0000 7f329b223640 1 -- 172.21.15.88:0/3710811399 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f32940b4560 con 0x7f3294072530 2024-09-16T01:43:14.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.462+0000 7f32897fa640 1 -- 172.21.15.88:0/3710811399 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v253) ==== 161+0+0 (secure 0 0 0) 0x7f3290059870 con 0x7f3294072530 2024-09-16T01:43:14.464 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:43:14.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.465+0000 7f329b223640 1 -- 172.21.15.88:0/3710811399 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f328006c2e0 msgr2=0x7f328006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:14.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.465+0000 7f329b223640 1 --2- 172.21.15.88:0/3710811399 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f328006c2e0 0x7f328006e7a0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f3284034000 tx=0x7f328402c000 comp rx=0 tx=0).stop 2024-09-16T01:43:14.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.465+0000 7f329b223640 1 -- 172.21.15.88:0/3710811399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3294072530 msgr2=0x7f329415d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:14.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.465+0000 7f329b223640 1 --2- 172.21.15.88:0/3710811399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3294072530 0x7f329415d030 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f329000d620 tx=0x7f329000dae0 comp rx=0 tx=0).stop 2024-09-16T01:43:14.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.465+0000 7f329b223640 1 -- 172.21.15.88:0/3710811399 shutdown_connections 2024-09-16T01:43:14.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.465+0000 7f329b223640 1 --2- 172.21.15.88:0/3710811399 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f328006c2e0 0x7f328006e7a0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:14.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.465+0000 7f329b223640 1 --2- 172.21.15.88:0/3710811399 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f329415b480 0x7f3294137b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:14.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.465+0000 7f329b223640 1 --2- 172.21.15.88:0/3710811399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32940b60c0 0x7f3294146590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:14.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.465+0000 7f329b223640 1 --2- 172.21.15.88:0/3710811399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3294072530 0x7f329415d030 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:14.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.465+0000 7f329b223640 1 -- 172.21.15.88:0/3710811399 >> 172.21.15.88:0/3710811399 conn(0x7f329406d0f0 msgr2=0x7f329406d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:14.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.465+0000 7f329b223640 1 -- 172.21.15.88:0/3710811399 shutdown_connections 2024-09-16T01:43:14.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.465+0000 7f329b223640 1 -- 172.21.15.88:0/3710811399 wait complete. 2024-09-16T01:43:14.478 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:43:14.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.578+0000 7fc3b861e640 1 Processor -- start 2024-09-16T01:43:14.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.578+0000 7fc3b861e640 1 -- start start 2024-09-16T01:43:14.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.579+0000 7fc3b861e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc3b015e7a0 0x7fc3b006caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:14.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.579+0000 7fc3b861e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc3b00b68f0 0x7fc3b00d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:14.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.579+0000 7fc3b861e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3b00cf860 con 0x7fc3b00cf1f0 2024-09-16T01:43:14.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.579+0000 7fc3b861e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc3b00d2ca0 con 0x7fc3b00b68f0 2024-09-16T01:43:14.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.579+0000 7fc3b861e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc3b00cfb40 con 0x7fc3b015e7a0 2024-09-16T01:43:14.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.579+0000 7fc3b6bbb640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc3b00b68f0 0x7fc3b00d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:14.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.579+0000 7fc3b6bbb640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc3b00b68f0 0x7fc3b00d3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58676/0 (socket says 172.21.15.88:58676) 2024-09-16T01:43:14.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.579+0000 7fc3b6bbb640 1 -- 172.21.15.88:0/3704539340 learned_addr learned my addr 172.21.15.88:0/3704539340 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:14.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.580+0000 7fc3b6bbb640 1 -- 172.21.15.88:0/3704539340 >> v2:172.21.15.154:3300/0 conn(0x7fc3b015e7a0 msgr2=0x7fc3b006caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:14.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.580+0000 7fc3b6bbb640 1 --2- 172.21.15.88:0/3704539340 >> v2:172.21.15.154:3300/0 conn(0x7fc3b015e7a0 0x7fc3b006caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:14.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.580+0000 7fc3b6bbb640 1 -- 172.21.15.88:0/3704539340 >> v1:172.21.15.154:6789/0 conn(0x7fc3b00cf1f0 legacy=0x7fc3b00b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:14.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.580+0000 7fc3b6bbb640 1 -- 172.21.15.88:0/3704539340 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3b00cfe20 con 0x7fc3b00b68f0 2024-09-16T01:43:14.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.580+0000 7fc3b6bbb640 1 --2- 172.21.15.88:0/3704539340 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3b00b68f0 0x7fc3b00d3410 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7fc3b00d5820 tx=0x7fc3a802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cd0028a456f0156 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:14.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.581+0000 7fc3b53b8640 1 -- 172.21.15.88:0/3704539340 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a8037070 con 0x7fc3b00b68f0 2024-09-16T01:43:14.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.581+0000 7fc3b53b8640 1 -- 172.21.15.88:0/3704539340 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3a8032440 con 0x7fc3b00b68f0 2024-09-16T01:43:14.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.581+0000 7fc3b53b8640 1 -- 172.21.15.88:0/3704539340 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a8032810 con 0x7fc3b00b68f0 2024-09-16T01:43:14.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.581+0000 7fc3b861e640 1 -- 172.21.15.88:0/3704539340 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3b00b68f0 msgr2=0x7fc3b00d3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:14.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.581+0000 7fc3b861e640 1 --2- 172.21.15.88:0/3704539340 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3b00b68f0 0x7fc3b00d3410 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7fc3b00d5820 tx=0x7fc3a802a1a0 comp rx=0 tx=0).stop 2024-09-16T01:43:14.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.581+0000 7fc3b861e640 1 -- 172.21.15.88:0/3704539340 shutdown_connections 2024-09-16T01:43:14.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.581+0000 7fc3b861e640 1 --2- 172.21.15.88:0/3704539340 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3b00b68f0 0x7fc3b00d3410 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:14.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.581+0000 7fc3b861e640 1 --2- 172.21.15.88:0/3704539340 >> v2:172.21.15.154:3300/0 conn(0x7fc3b015e7a0 0x7fc3b006caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:14.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.581+0000 7fc3b861e640 1 -- 172.21.15.88:0/3704539340 >> 172.21.15.88:0/3704539340 conn(0x7fc3b00d11e0 msgr2=0x7fc3b00d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:14.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.581+0000 7fc3b861e640 1 -- 172.21.15.88:0/3704539340 shutdown_connections 2024-09-16T01:43:14.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.581+0000 7fc3b861e640 1 -- 172.21.15.88:0/3704539340 wait complete. 2024-09-16T01:43:14.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.583+0000 7fc3b861e640 1 Processor -- start 2024-09-16T01:43:14.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.583+0000 7fc3b861e640 1 -- start start 2024-09-16T01:43:14.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.583+0000 7fc3b861e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3b00b68f0 0x7fc3b00cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:14.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.583+0000 7fc3b861e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3b00cf1f0 0x7fc3b015f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:14.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.583+0000 7fc3b861e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3b015e7a0 0x7fc3b0156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:14.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.583+0000 7fc3b861e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b006cfe0 con 0x7fc3b015e7a0 2024-09-16T01:43:14.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.583+0000 7fc3b861e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b00b7a00 con 0x7fc3b00cf1f0 2024-09-16T01:43:14.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.583+0000 7fc3b861e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc3b015eb80 con 0x7fc3b00b68f0 2024-09-16T01:43:14.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.583+0000 7fc3b6bbb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3b015e7a0 0x7fc3b0156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:14.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.583+0000 7fc3b6bbb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3b015e7a0 0x7fc3b0156d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58680/0 (socket says 172.21.15.88:58680) 2024-09-16T01:43:14.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.583+0000 7fc3b6bbb640 1 -- 172.21.15.88:0/3210897343 learned_addr learned my addr 172.21.15.88:0/3210897343 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:14.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.584+0000 7fc3b5bb9640 1 --2- 172.21.15.88:0/3210897343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3b00cf1f0 0x7fc3b015f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:14.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.584+0000 7fc3b63ba640 1 --2- 172.21.15.88:0/3210897343 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3b00b68f0 0x7fc3b00cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:14.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.584+0000 7fc3b6bbb640 1 -- 172.21.15.88:0/3210897343 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3b00b68f0 msgr2=0x7fc3b00cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:14.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.584+0000 7fc3b6bbb640 1 --2- 172.21.15.88:0/3210897343 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3b00b68f0 0x7fc3b00cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:14.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.584+0000 7fc3b6bbb640 1 -- 172.21.15.88:0/3210897343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3b00cf1f0 msgr2=0x7fc3b015f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:14.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.584+0000 7fc3b6bbb640 1 --2- 172.21.15.88:0/3210897343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3b00cf1f0 0x7fc3b015f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:14.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.584+0000 7fc3b6bbb640 1 -- 172.21.15.88:0/3210897343 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3b00c9100 con 0x7fc3b015e7a0 2024-09-16T01:43:14.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.584+0000 7fc3b63ba640 1 --2- 172.21.15.88:0/3210897343 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3b00b68f0 0x7fc3b00cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:14.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.584+0000 7fc3b6bbb640 1 --2- 172.21.15.88:0/3210897343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3b015e7a0 0x7fc3b0156d10 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7fc3a80090b0 tx=0x7fc3a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:14.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.584+0000 7fc3aeffd640 1 -- 172.21.15.88:0/3210897343 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a8041070 con 0x7fc3b015e7a0 2024-09-16T01:43:14.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.584+0000 7fc3aeffd640 1 -- 172.21.15.88:0/3210897343 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3a803c950 con 0x7fc3b015e7a0 2024-09-16T01:43:14.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.584+0000 7fc3b861e640 1 -- 172.21.15.88:0/3210897343 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3b00c93e0 con 0x7fc3b015e7a0 2024-09-16T01:43:14.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.584+0000 7fc3aeffd640 1 -- 172.21.15.88:0/3210897343 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a8037040 con 0x7fc3b015e7a0 2024-09-16T01:43:14.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.584+0000 7fc3b861e640 1 -- 172.21.15.88:0/3210897343 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3b01567f0 con 0x7fc3b015e7a0 2024-09-16T01:43:14.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.585+0000 7fc3b861e640 1 -- 172.21.15.88:0/3210897343 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3b006caa0 con 0x7fc3b015e7a0 2024-09-16T01:43:14.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.586+0000 7fc3aeffd640 1 -- 172.21.15.88:0/3210897343 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc3a8044070 con 0x7fc3b015e7a0 2024-09-16T01:43:14.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.586+0000 7fc3aeffd640 1 --2- 172.21.15.88:0/3210897343 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc39c06c2b0 0x7fc39c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:14.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.586+0000 7fc3b63ba640 1 --2- 172.21.15.88:0/3210897343 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc39c06c2b0 0x7fc39c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:14.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.586+0000 7fc3aeffd640 1 -- 172.21.15.88:0/3210897343 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8404+0+0 (secure 0 0 0) 0x7fc3a80acc20 con 0x7fc3b015e7a0 2024-09-16T01:43:14.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.587+0000 7fc3b63ba640 1 --2- 172.21.15.88:0/3210897343 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc39c06c2b0 0x7fc39c06e770 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0007eb0 tx=0x7fc3a00073c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:14.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.590+0000 7fc3aeffd640 1 -- 172.21.15.88:0/3210897343 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc3a807a9d0 con 0x7fc3b015e7a0 2024-09-16T01:43:14.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:14.824+0000 7fc3b861e640 1 -- 172.21.15.88:0/3210897343 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fc3b00c95b0 con 0x7fc3b015e7a0 2024-09-16T01:43:15.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.468+0000 7fc3aeffd640 1 -- 172.21.15.88:0/3210897343 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v254) ==== 101+0+0 (secure 0 0 0) 0x7fc3a807a2d0 con 0x7fc3b015e7a0 2024-09-16T01:43:15.469 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:43:15.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.470+0000 7fc3b861e640 1 -- 172.21.15.88:0/3210897343 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc39c06c2b0 msgr2=0x7fc39c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:15.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.470+0000 7fc3b861e640 1 --2- 172.21.15.88:0/3210897343 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc39c06c2b0 0x7fc39c06e770 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0007eb0 tx=0x7fc3a00073c0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.470+0000 7fc3b861e640 1 -- 172.21.15.88:0/3210897343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3b015e7a0 msgr2=0x7fc3b0156d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:15.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.470+0000 7fc3b861e640 1 --2- 172.21.15.88:0/3210897343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3b015e7a0 0x7fc3b0156d10 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7fc3a80090b0 tx=0x7fc3a80046c0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.470+0000 7fc3b861e640 1 -- 172.21.15.88:0/3210897343 shutdown_connections 2024-09-16T01:43:15.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.470+0000 7fc3b861e640 1 --2- 172.21.15.88:0/3210897343 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc39c06c2b0 0x7fc39c06e770 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.470+0000 7fc3b861e640 1 --2- 172.21.15.88:0/3210897343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3b015e7a0 0x7fc3b0156d10 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.470+0000 7fc3b861e640 1 --2- 172.21.15.88:0/3210897343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3b00cf1f0 0x7fc3b015f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.470+0000 7fc3b861e640 1 --2- 172.21.15.88:0/3210897343 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3b00b68f0 0x7fc3b00cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.470+0000 7fc3b861e640 1 -- 172.21.15.88:0/3210897343 >> 172.21.15.88:0/3210897343 conn(0x7fc3b00d11e0 msgr2=0x7fc3b00d15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:15.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.470+0000 7fc3b861e640 1 -- 172.21.15.88:0/3210897343 shutdown_connections 2024-09-16T01:43:15.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.470+0000 7fc3b861e640 1 -- 172.21.15.88:0/3210897343 wait complete. 2024-09-16T01:43:15.485 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:43:15.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.529+0000 7f0753953f40 1 Processor -- start 2024-09-16T01:43:15.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.529+0000 7f0753953f40 1 -- start start 2024-09-16T01:43:15.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.530+0000 7f0753953f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07400022a0 0x5607d8f89440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.530+0000 7f0753953f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5607d8f87bb0 0x5607d8f84900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.530+0000 7f0753953f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5607d8f89f70 0x5607d8f8a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.530+0000 7f0753953f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5607d8ef6b30 con 0x5607d8f87bb0 2024-09-16T01:43:15.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.530+0000 7f0753953f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5607d8e08510 con 0x7f07400022a0 2024-09-16T01:43:15.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.530+0000 7f0753953f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5607d8f8fc50 con 0x5607d8f89f70 2024-09-16T01:43:15.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.530+0000 7f074bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5607d8f87bb0 0x5607d8f84900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:15.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.530+0000 7f074bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5607d8f87bb0 0x5607d8f84900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58708/0 (socket says 172.21.15.88:58708) 2024-09-16T01:43:15.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.530+0000 7f074bfff640 1 -- 172.21.15.88:0/2640834283 learned_addr learned my addr 172.21.15.88:0/2640834283 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:15.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.530+0000 7f074bfff640 1 -- 172.21.15.88:0/2640834283 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5607d8f89f70 msgr2=0x5607d8f8a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:15.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.530+0000 7f074bfff640 1 --2- 172.21.15.88:0/2640834283 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5607d8f89f70 0x5607d8f8a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.530+0000 7f074bfff640 1 -- 172.21.15.88:0/2640834283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07400022a0 msgr2=0x5607d8f89440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:15.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.530+0000 7f0751449640 1 --2- 172.21.15.88:0/2640834283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07400022a0 0x5607d8f89440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:15.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.530+0000 7f074bfff640 1 --2- 172.21.15.88:0/2640834283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07400022a0 0x5607d8f89440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.530+0000 7f074bfff640 1 -- 172.21.15.88:0/2640834283 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607d8f87790 con 0x5607d8f87bb0 2024-09-16T01:43:15.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.530+0000 7f074bfff640 1 --2- 172.21.15.88:0/2640834283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5607d8f87bb0 0x5607d8f84900 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f073c009730 tx=0x7f073c005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aa96c0b47442f5c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:15.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.531+0000 7f074b7fe640 1 -- 172.21.15.88:0/2640834283 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f073c0054c0 con 0x5607d8f87bb0 2024-09-16T01:43:15.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.531+0000 7f074b7fe640 1 -- 172.21.15.88:0/2640834283 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f073c01a070 con 0x5607d8f87bb0 2024-09-16T01:43:15.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.531+0000 7f074b7fe640 1 -- 172.21.15.88:0/2640834283 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f073c008a90 con 0x5607d8f87bb0 2024-09-16T01:43:15.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.531+0000 7f0753953f40 1 -- 172.21.15.88:0/2640834283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5607d8f87bb0 msgr2=0x5607d8f84900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:15.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.531+0000 7f0753953f40 1 --2- 172.21.15.88:0/2640834283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5607d8f87bb0 0x5607d8f84900 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f073c009730 tx=0x7f073c005a00 comp rx=0 tx=0).stop 2024-09-16T01:43:15.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.531+0000 7f0753953f40 1 -- 172.21.15.88:0/2640834283 shutdown_connections 2024-09-16T01:43:15.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.531+0000 7f0753953f40 1 --2- 172.21.15.88:0/2640834283 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5607d8f89f70 0x5607d8f8a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.531+0000 7f0753953f40 1 --2- 172.21.15.88:0/2640834283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5607d8f87bb0 0x5607d8f84900 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.531+0000 7f0753953f40 1 --2- 172.21.15.88:0/2640834283 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07400022a0 0x5607d8f89440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.531+0000 7f0753953f40 1 -- 172.21.15.88:0/2640834283 >> 172.21.15.88:0/2640834283 conn(0x5607d8ee1680 msgr2=0x5607d8ee6830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:15.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.531+0000 7f0753953f40 1 -- 172.21.15.88:0/2640834283 shutdown_connections 2024-09-16T01:43:15.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.531+0000 7f0753953f40 1 -- 172.21.15.88:0/2640834283 wait complete. 2024-09-16T01:43:15.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.532+0000 7f0753953f40 1 Processor -- start 2024-09-16T01:43:15.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.532+0000 7f0753953f40 1 -- start start 2024-09-16T01:43:15.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.532+0000 7f0753953f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5607d8f87bb0 0x5607d8ee5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.532+0000 7f0753953f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5607d8f89f70 0x5607d8f7f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.532+0000 7f0753953f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07400022a0 0x5607d8ee5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.532+0000 7f0753953f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5607d8f88310 con 0x5607d8f89f70 2024-09-16T01:43:15.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.532+0000 7f0753953f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5607d8f885f0 con 0x5607d8f87bb0 2024-09-16T01:43:15.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.532+0000 7f0753953f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5607d8f88f20 con 0x7f07400022a0 2024-09-16T01:43:15.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.533+0000 7f074bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5607d8f89f70 0x5607d8f7f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:15.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.533+0000 7f074bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5607d8f89f70 0x5607d8f7f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58716/0 (socket says 172.21.15.88:58716) 2024-09-16T01:43:15.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.533+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 learned_addr learned my addr 172.21.15.88:0/2057817653 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:15.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.533+0000 7f0751449640 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5607d8f87bb0 0x5607d8ee5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:15.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.533+0000 7f0751c4a640 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07400022a0 0x5607d8ee5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:15.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.533+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07400022a0 msgr2=0x5607d8ee5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:15.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.533+0000 7f074bfff640 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07400022a0 0x5607d8ee5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.533+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5607d8f87bb0 msgr2=0x5607d8ee5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:15.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.533+0000 7f074bfff640 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5607d8f87bb0 0x5607d8ee5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.533+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607d8efa530 con 0x5607d8f89f70 2024-09-16T01:43:15.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.533+0000 7f074bfff640 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5607d8f89f70 0x5607d8f7f190 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f073c000c00 tx=0x7f073c002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:15.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.533+0000 7f074a7fc640 1 -- 172.21.15.88:0/2057817653 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f073c015670 con 0x5607d8f89f70 2024-09-16T01:43:15.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.533+0000 7f074a7fc640 1 -- 172.21.15.88:0/2057817653 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f073c015830 con 0x5607d8f89f70 2024-09-16T01:43:15.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.533+0000 7f074a7fc640 1 -- 172.21.15.88:0/2057817653 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f073c015c20 con 0x5607d8f89f70 2024-09-16T01:43:15.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.533+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5607d8efae10 con 0x5607d8f89f70 2024-09-16T01:43:15.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.533+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607d8efb3d0 con 0x5607d8f89f70 2024-09-16T01:43:15.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.535+0000 7f074a7fc640 1 -- 172.21.15.88:0/2057817653 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f073c02d060 con 0x5607d8f89f70 2024-09-16T01:43:15.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.535+0000 7f074a7fc640 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f073406ac90 0x7f073406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.535+0000 7f074a7fc640 1 -- 172.21.15.88:0/2057817653 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8404+0+0 (secure 0 0 0) 0x7f073c08cad0 con 0x5607d8f89f70 2024-09-16T01:43:15.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.535+0000 7f0751449640 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f073406ac90 0x7f073406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:15.540 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:43:15.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.535+0000 7f0753953f40 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5607d8f86000 0x5607d8f863e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.535+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d8efa9a0 con 0x5607d8f86000 2024-09-16T01:43:15.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.536+0000 7f0751c4a640 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5607d8f86000 0x5607d8f863e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:15.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.536+0000 7f0751449640 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f073406ac90 0x7f073406d150 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f0744056000 tx=0x7f0744050000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:15.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.536+0000 7f0751c4a640 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5607d8f86000 0x5607d8f863e0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:15.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.538+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v118'939 uv913 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.538+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e118) -- 0x5607d8f86920 con 0x5607d8f86000 2024-09-16T01:43:15.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.540+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v118'940 uv940 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.540+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9031430 con 0x5607d8f86000 2024-09-16T01:43:15.542 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:43:15.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.541+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v118'941 uv941 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.541+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90327e0 con 0x5607d8f86000 2024-09-16T01:43:15.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.543+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v118'942 uv942 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.543+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9032b80 con 0x5607d8f86000 2024-09-16T01:43:15.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.544+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v118'943 uv943 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.544+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90317d0 con 0x5607d8f86000 2024-09-16T01:43:15.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.545+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v118'944 uv944 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.545+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9031b90 con 0x5607d8f86000 2024-09-16T01:43:15.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.547+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v118'945 uv919 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.547+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e118) -- 0x5607d9031f30 con 0x5607d8f86000 2024-09-16T01:43:15.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.549+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v118'946 uv946 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.549+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9031f30 con 0x5607d8f86000 2024-09-16T01:43:15.551 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:43:15.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.550+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v118'947 uv947 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.550+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9031f30 con 0x5607d8f86000 2024-09-16T01:43:15.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.551+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v118'948 uv948 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.551+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9031f30 con 0x5607d8f86000 2024-09-16T01:43:15.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.553+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v118'949 uv949 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.553+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9031f30 con 0x5607d8f86000 2024-09-16T01:43:15.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.555+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v118'950 uv950 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.555+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9031f30 con 0x5607d8f86000 2024-09-16T01:43:15.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.557+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v118'951 uv926 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.557+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e118) -- 0x5607d9031f30 con 0x5607d8f86000 2024-09-16T01:43:15.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.558+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v118'952 uv952 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.558+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9031f30 con 0x5607d8f86000 2024-09-16T01:43:15.561 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:43:15.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.560+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v118'953 uv953 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.560+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9031f30 con 0x5607d8f86000 2024-09-16T01:43:15.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.561+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v118'954 uv954 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.561+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9031f30 con 0x5607d8f86000 2024-09-16T01:43:15.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.562+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v118'955 uv955 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.562+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9031f30 con 0x5607d8f86000 2024-09-16T01:43:15.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.564+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v118'956 uv956 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.564+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9032f20 con 0x5607d8f86000 2024-09-16T01:43:15.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.565+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v118'957 uv957 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.565+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9032f20 con 0x5607d8f86000 2024-09-16T01:43:15.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.566+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v118'958 uv932 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.566+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e118) -- 0x5607d9032f20 con 0x5607d8f86000 2024-09-16T01:43:15.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.567+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v118'959 uv959 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.567+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9032f20 con 0x5607d8f86000 2024-09-16T01:43:15.570 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:43:15.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.569+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v118'960 uv960 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.569+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9033f30 con 0x5607d8f86000 2024-09-16T01:43:15.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.570+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v118'961 uv961 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.570+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9033f30 con 0x5607d8f86000 2024-09-16T01:43:15.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.571+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v118'962 uv962 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.571+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9032f20 con 0x5607d8f86000 2024-09-16T01:43:15.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.573+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v118'963 uv963 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.573+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9032f20 con 0x5607d8f86000 2024-09-16T01:43:15.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.574+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v118'964 uv938 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.574+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e118) -- 0x5607d9032f20 con 0x5607d8f86000 2024-09-16T01:43:15.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.576+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v118'965 uv965 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.576+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9032f20 con 0x5607d8f86000 2024-09-16T01:43:15.578 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:43:15.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.577+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v118'966 uv966 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.577+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9033f30 con 0x5607d8f86000 2024-09-16T01:43:15.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.578+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v118'967 uv967 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.578+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9033f30 con 0x5607d8f86000 2024-09-16T01:43:15.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.580+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v118'968 uv968 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.580+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9032f20 con 0x5607d8f86000 2024-09-16T01:43:15.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.581+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v118'969 uv969 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.581+0000 7f0753953f40 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5607d9033430 0x5607d9035820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.581+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9035d60 con 0x5607d9033430 2024-09-16T01:43:15.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.582+0000 7f0751449640 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5607d9033430 0x5607d9035820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:15.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.582+0000 7f0751449640 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5607d9033430 0x5607d9035820 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:15.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.584+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v118'681 uv664 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d9033430 2024-09-16T01:43:15.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.584+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e118) -- 0x5607d9036100 con 0x5607d9033430 2024-09-16T01:43:15.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.586+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v118'682 uv682 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d9033430 2024-09-16T01:43:15.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.586+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90364a0 con 0x5607d9033430 2024-09-16T01:43:15.588 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:43:15.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.587+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v118'683 uv683 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d9033430 2024-09-16T01:43:15.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.587+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9036cf0 con 0x5607d9033430 2024-09-16T01:43:15.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.588+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v118'684 uv684 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d9033430 2024-09-16T01:43:15.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.588+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9037090 con 0x5607d9033430 2024-09-16T01:43:15.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.589+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v118'685 uv685 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d9033430 2024-09-16T01:43:15.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.589+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9037430 con 0x5607d9033430 2024-09-16T01:43:15.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.590+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v118'686 uv669 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d9033430 2024-09-16T01:43:15.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.590+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e118) -- 0x5607d90377d0 con 0x5607d9033430 2024-09-16T01:43:15.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.591+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v118'687 uv687 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d9033430 2024-09-16T01:43:15.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.591+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9037b70 con 0x5607d9033430 2024-09-16T01:43:15.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.593+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v118'688 uv688 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d9033430 2024-09-16T01:43:15.594 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:43:15.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.593+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9037b70 con 0x5607d9033430 2024-09-16T01:43:15.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.594+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v118'689 uv689 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d9033430 2024-09-16T01:43:15.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.594+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9037b70 con 0x5607d9033430 2024-09-16T01:43:15.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.595+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v118'690 uv690 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d9033430 2024-09-16T01:43:15.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.595+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9037b70 con 0x5607d9033430 2024-09-16T01:43:15.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.596+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v118'691 uv674 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d9033430 2024-09-16T01:43:15.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.596+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e118) -- 0x5607d9037b70 con 0x5607d9033430 2024-09-16T01:43:15.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.598+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v118'692 uv692 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d9033430 2024-09-16T01:43:15.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.598+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9037b70 con 0x5607d9033430 2024-09-16T01:43:15.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.599+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v118'693 uv693 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d9033430 2024-09-16T01:43:15.600 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:43:15.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.599+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d903bff0 con 0x5607d9033430 2024-09-16T01:43:15.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.601+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v118'694 uv694 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d9033430 2024-09-16T01:43:15.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.601+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d903bff0 con 0x5607d9033430 2024-09-16T01:43:15.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.602+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v118'695 uv695 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d9033430 2024-09-16T01:43:15.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.602+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9037b70 con 0x5607d9033430 2024-09-16T01:43:15.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.603+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v118'696 uv680 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d9033430 2024-09-16T01:43:15.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.603+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e118) -- 0x5607d9037b70 con 0x5607d9033430 2024-09-16T01:43:15.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.605+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v118'697 uv697 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d9033430 2024-09-16T01:43:15.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.605+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9037b70 con 0x5607d9033430 2024-09-16T01:43:15.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.607+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v118'698 uv698 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d9033430 2024-09-16T01:43:15.609 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:43:15.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.607+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9038b80 con 0x5607d9033430 2024-09-16T01:43:15.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.608+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v118'699 uv699 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d9033430 2024-09-16T01:43:15.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.608+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9038b80 con 0x5607d9033430 2024-09-16T01:43:15.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.610+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v118'700 uv700 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d9033430 2024-09-16T01:43:15.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.610+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9037b70 con 0x5607d9033430 2024-09-16T01:43:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.611+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v118'701 uv701 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d9033430 2024-09-16T01:43:15.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.611+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9037b70 con 0x5607d8f86000 2024-09-16T01:43:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.612+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v118'183 uv182 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.612+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e118) -- 0x5607d9037b70 con 0x5607d8f86000 2024-09-16T01:43:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.613+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v118'184 uv184 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.613+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9037b70 con 0x5607d8f86000 2024-09-16T01:43:15.615 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:43:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.614+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v118'185 uv185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.614+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9038b80 con 0x5607d8f86000 2024-09-16T01:43:15.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.616+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v118'186 uv186 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.616+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9038b80 con 0x5607d8f86000 2024-09-16T01:43:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.616+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v118'187 uv187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.616+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9037b70 con 0x5607d8f86000 2024-09-16T01:43:15.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.617+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v118'188 uv188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.617+0000 7f0753953f40 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5607d9038080 0x5607d903a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.617+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d903a9b0 con 0x5607d9038080 2024-09-16T01:43:15.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.618+0000 7f074bfff640 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5607d9038080 0x5607d903a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:15.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.618+0000 7f074bfff640 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5607d9038080 0x5607d903a470 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:15.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.619+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v118'135 uv134 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f073c056810 con 0x5607d9038080 2024-09-16T01:43:15.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.619+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e118) -- 0x5607d903ad50 con 0x5607d9038080 2024-09-16T01:43:15.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.620+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v118'136 uv136 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f073c056810 con 0x5607d9038080 2024-09-16T01:43:15.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.620+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d903b0f0 con 0x5607d9038080 2024-09-16T01:43:15.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.621+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v118'137 uv137 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f073c056810 con 0x5607d9038080 2024-09-16T01:43:15.623 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:43:15.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.621+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d903b490 con 0x5607d9038080 2024-09-16T01:43:15.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.622+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v118'138 uv138 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f073c056810 con 0x5607d9038080 2024-09-16T01:43:15.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.622+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90368c0 con 0x5607d8f86000 2024-09-16T01:43:15.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.623+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v118'183 uv182 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.623+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e118) -- 0x5607d90368c0 con 0x5607d8f86000 2024-09-16T01:43:15.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.624+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v118'184 uv184 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.624+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90368c0 con 0x5607d8f86000 2024-09-16T01:43:15.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.625+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v118'185 uv185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.626 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:43:15.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.625+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90368c0 con 0x5607d8f86000 2024-09-16T01:43:15.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.626+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v118'186 uv186 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.626+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90368c0 con 0x5607d8f86000 2024-09-16T01:43:15.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.629+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v118'187 uv187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.629+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90368c0 con 0x5607d8f86000 2024-09-16T01:43:15.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.632+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v118'188 uv188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.632+0000 7f0753953f40 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5607d903c240 0x5607d903e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.632+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d903eb70 con 0x5607d903c240 2024-09-16T01:43:15.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.632+0000 7f0751c4a640 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5607d903c240 0x5607d903e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:15.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.632+0000 7f0751c4a640 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5607d903c240 0x5607d903e630 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:15.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.634+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v118'338 uv332 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.634+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e118) -- 0x5607d903eb70 con 0x5607d903c240 2024-09-16T01:43:15.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.635+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v118'339 uv339 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.635+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d903ec30 con 0x5607d903c240 2024-09-16T01:43:15.637 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:43:15.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.636+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v118'340 uv340 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.636+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9040060 con 0x5607d903c240 2024-09-16T01:43:15.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.637+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v118'341 uv341 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.637+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9040060 con 0x5607d903c240 2024-09-16T01:43:15.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.638+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v118'342 uv342 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.638+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d903f050 con 0x5607d903c240 2024-09-16T01:43:15.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.639+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v118'343 uv343 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.639+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d903f050 con 0x5607d903c240 2024-09-16T01:43:15.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.640+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v118'344 uv337 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.640+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e118) -- 0x5607d903f050 con 0x5607d903c240 2024-09-16T01:43:15.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.641+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v118'345 uv345 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.641+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d903f050 con 0x5607d903c240 2024-09-16T01:43:15.643 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:43:15.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.642+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v118'346 uv346 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.642+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d903f050 con 0x5607d903c240 2024-09-16T01:43:15.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.643+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v118'347 uv347 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.643+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d903f050 con 0x5607d903c240 2024-09-16T01:43:15.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.644+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v118'348 uv348 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.644+0000 7f0753953f40 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5607d903f560 0x5607d9041950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.644+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9041fb0 con 0x5607d903f560 2024-09-16T01:43:15.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.644+0000 7f0751449640 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5607d903f560 0x5607d9041950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:15.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.644+0000 7f0751449640 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5607d903f560 0x5607d9041950 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:15.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.645+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v118'153 uv152 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d903f560 2024-09-16T01:43:15.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.645+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e118) -- 0x5607d9042070 con 0x5607d903f560 2024-09-16T01:43:15.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.646+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v118'154 uv154 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d903f560 2024-09-16T01:43:15.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.646+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9042490 con 0x5607d903f560 2024-09-16T01:43:15.649 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:43:15.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.647+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v118'155 uv155 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d903f560 2024-09-16T01:43:15.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.647+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9042490 con 0x5607d903f560 2024-09-16T01:43:15.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.648+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v118'156 uv156 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d903f560 2024-09-16T01:43:15.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.648+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9042490 con 0x5607d903f560 2024-09-16T01:43:15.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.649+0000 7f0751449640 1 -- 172.21.15.88:0/2057817653 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v118'157 uv157 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f0744062430 con 0x5607d903f560 2024-09-16T01:43:15.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.649+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9042490 con 0x5607d8f86000 2024-09-16T01:43:15.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.650+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v118'183 uv182 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.650+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e118) -- 0x5607d9042490 con 0x5607d8f86000 2024-09-16T01:43:15.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.651+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v118'184 uv184 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.651+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9042490 con 0x5607d8f86000 2024-09-16T01:43:15.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.652+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v118'185 uv185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.653 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:43:15.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.652+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90434a0 con 0x5607d8f86000 2024-09-16T01:43:15.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.653+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v118'186 uv186 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.653+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90434a0 con 0x5607d8f86000 2024-09-16T01:43:15.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.654+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v118'187 uv187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.654+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9042490 con 0x5607d8f86000 2024-09-16T01:43:15.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.655+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v118'188 uv188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d8f86000 2024-09-16T01:43:15.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.655+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9042490 con 0x5607d9038080 2024-09-16T01:43:15.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.656+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v118'304 uv298 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f073c056810 con 0x5607d9038080 2024-09-16T01:43:15.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.656+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e118) -- 0x5607d9042830 con 0x5607d9038080 2024-09-16T01:43:15.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.657+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v118'305 uv305 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f073c056810 con 0x5607d9038080 2024-09-16T01:43:15.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.657+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9042c20 con 0x5607d9038080 2024-09-16T01:43:15.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.658+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v118'306 uv306 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f073c056810 con 0x5607d9038080 2024-09-16T01:43:15.659 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:43:15.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.658+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9043010 con 0x5607d9038080 2024-09-16T01:43:15.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.659+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v118'307 uv307 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f073c056810 con 0x5607d9038080 2024-09-16T01:43:15.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.659+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9043010 con 0x5607d9038080 2024-09-16T01:43:15.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.660+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v118'308 uv308 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f073c056810 con 0x5607d9038080 2024-09-16T01:43:15.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.660+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9042fc0 con 0x5607d9038080 2024-09-16T01:43:15.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.661+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v118'309 uv303 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f073c056810 con 0x5607d9038080 2024-09-16T01:43:15.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.661+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e118) -- 0x5607d9042fc0 con 0x5607d9038080 2024-09-16T01:43:15.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.662+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v118'310 uv310 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f073c056810 con 0x5607d9038080 2024-09-16T01:43:15.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.662+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9042fc0 con 0x5607d9038080 2024-09-16T01:43:15.664 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:43:15.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.663+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v118'311 uv311 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f073c056810 con 0x5607d9038080 2024-09-16T01:43:15.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.663+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9042fc0 con 0x5607d9038080 2024-09-16T01:43:15.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.663+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v118'312 uv312 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f073c056810 con 0x5607d9038080 2024-09-16T01:43:15.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.664+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d9043410 con 0x5607d9038080 2024-09-16T01:43:15.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.665+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v118'313 uv313 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f073c056810 con 0x5607d9038080 2024-09-16T01:43:15.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.665+0000 7f0753953f40 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5607d9043da0 0x5607d9046490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.665+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90469d0 con 0x5607d9043da0 2024-09-16T01:43:15.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.665+0000 7f074bfff640 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5607d9043da0 0x5607d9046490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:15.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.665+0000 7f074bfff640 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5607d9043da0 0x5607d9046490 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:15.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.666+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v118'132 uv131 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f073c056810 con 0x5607d9043da0 2024-09-16T01:43:15.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.666+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e118) -- 0x5607d90469d0 con 0x5607d9043da0 2024-09-16T01:43:15.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.667+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v118'133 uv133 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f073c056810 con 0x5607d9043da0 2024-09-16T01:43:15.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.667+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90469d0 con 0x5607d9043da0 2024-09-16T01:43:15.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.668+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v118'134 uv134 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f073c056810 con 0x5607d9043da0 2024-09-16T01:43:15.669 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:43:15.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.668+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90469d0 con 0x5607d9043da0 2024-09-16T01:43:15.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.669+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v118'135 uv135 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f073c056810 con 0x5607d9043da0 2024-09-16T01:43:15.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.669+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90469d0 con 0x5607d9038080 2024-09-16T01:43:15.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.670+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v118'183 uv182 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f073c056810 con 0x5607d9038080 2024-09-16T01:43:15.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.670+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e118) -- 0x5607d90469d0 con 0x5607d9038080 2024-09-16T01:43:15.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.671+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v118'184 uv184 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f073c056810 con 0x5607d9038080 2024-09-16T01:43:15.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.671+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90469d0 con 0x5607d9038080 2024-09-16T01:43:15.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.673+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v118'185 uv185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f073c056810 con 0x5607d9038080 2024-09-16T01:43:15.674 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:43:15.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.673+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90479e0 con 0x5607d9038080 2024-09-16T01:43:15.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.674+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v118'186 uv186 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f073c056810 con 0x5607d9038080 2024-09-16T01:43:15.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.674+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90479e0 con 0x5607d9038080 2024-09-16T01:43:15.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.674+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v118'187 uv187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f073c056810 con 0x5607d9038080 2024-09-16T01:43:15.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.675+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90469d0 con 0x5607d9038080 2024-09-16T01:43:15.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.676+0000 7f074bfff640 1 -- 172.21.15.88:0/2057817653 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v118'188 uv188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f073c056810 con 0x5607d9038080 2024-09-16T01:43:15.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.676+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90469d0 con 0x5607d903c240 2024-09-16T01:43:15.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.677+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v118'156 uv155 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.677+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e118) -- 0x5607d90469d0 con 0x5607d903c240 2024-09-16T01:43:15.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.678+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v118'157 uv157 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.678+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90469d0 con 0x5607d903c240 2024-09-16T01:43:15.682 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:43:15.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.679+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v118'158 uv158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.679+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90469d0 con 0x5607d903c240 2024-09-16T01:43:15.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.680+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v118'159 uv159 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.680+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90469d0 con 0x5607d903c240 2024-09-16T01:43:15.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.681+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v118'160 uv160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.681+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90469d0 con 0x5607d903c240 2024-09-16T01:43:15.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.683+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v118'376 uv375 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.683+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e118) -- 0x5607d90469d0 con 0x5607d903c240 2024-09-16T01:43:15.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.684+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v118'377 uv377 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.684+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90469d0 con 0x5607d903c240 2024-09-16T01:43:15.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.685+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v118'378 uv378 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.686 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:43:15.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.685+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90469d0 con 0x5607d903c240 2024-09-16T01:43:15.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.686+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v118'379 uv379 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.686+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90469d0 con 0x5607d903c240 2024-09-16T01:43:15.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.687+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v118'380 uv380 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.687+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90469d0 con 0x5607d903c240 2024-09-16T01:43:15.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.688+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v118'381 uv374 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.688+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e118) -- 0x5607d90469d0 con 0x5607d903c240 2024-09-16T01:43:15.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.689+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v118'382 uv382 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.689+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90469d0 con 0x5607d903c240 2024-09-16T01:43:15.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.690+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v118'383 uv383 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.691 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:43:15.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.690+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90479e0 con 0x5607d903c240 2024-09-16T01:43:15.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.691+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v118'384 uv384 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.691+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90479e0 con 0x5607d903c240 2024-09-16T01:43:15.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.692+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v118'385 uv385 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.692+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90469d0 con 0x5607d903c240 2024-09-16T01:43:15.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.693+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v118'386 uv386 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.693+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5607d90469d0 con 0x5607d903c240 2024-09-16T01:43:15.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v118'387 uv387 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f074c0027b0 con 0x5607d903c240 2024-09-16T01:43:15.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5607d8f86000 msgr2=0x5607d8f863e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:15.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0753953f40 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5607d8f86000 0x5607d8f863e0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5607d9038080 msgr2=0x5607d903a470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:15.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0753953f40 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5607d9038080 0x5607d903a470 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5607d9043da0 msgr2=0x5607d9046490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:15.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0753953f40 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5607d9043da0 0x5607d9046490 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5607d9033430 msgr2=0x5607d9035820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:15.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0753953f40 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5607d9033430 0x5607d9035820 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5607d903c240 msgr2=0x5607d903e630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:15.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0753953f40 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5607d903c240 0x5607d903e630 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5607d903f560 msgr2=0x5607d9041950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:15.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0753953f40 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5607d903f560 0x5607d9041950 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 reap_dead start 2024-09-16T01:43:15.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 reap_dead start 2024-09-16T01:43:15.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0751c4a640 1 -- 172.21.15.88:0/2057817653 reap_dead start 2024-09-16T01:43:15.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f073406ac90 msgr2=0x7f073406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:15.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0753953f40 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f073406ac90 0x7f073406d150 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f0744056000 tx=0x7f0744050000 comp rx=0 tx=0).stop 2024-09-16T01:43:15.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5607d8f89f70 msgr2=0x5607d8f7f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:15.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0753953f40 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5607d8f89f70 0x5607d8f7f190 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f073c000c00 tx=0x7f073c002fb0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 shutdown_connections 2024-09-16T01:43:15.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0753953f40 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f073406ac90 0x7f073406d150 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0753953f40 1 --2- 172.21.15.88:0/2057817653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5607d8f89f70 0x5607d8f7f190 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 >> 172.21.15.88:0/2057817653 conn(0x5607d8ee1680 msgr2=0x5607d8f8a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:15.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 shutdown_connections 2024-09-16T01:43:15.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.694+0000 7f0753953f40 1 -- 172.21.15.88:0/2057817653 wait complete. 2024-09-16T01:43:15.699 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 10000000000.00000000 sixmegs_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-09-16T01:43:15.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.783+0000 7f5e0c6eaf40 1 Processor -- start 2024-09-16T01:43:15.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.783+0000 7f5e0c6eaf40 1 -- start start 2024-09-16T01:43:15.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.783+0000 7f5e0c6eaf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640cf74f700 0x5640cf7d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.783+0000 7f5e0c6eaf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5640cf74fae0 0x5640cf704b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.783+0000 7f5e0c6eaf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5640cf7dd7b0 0x5640cf7050a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.783+0000 7f5e0c6eaf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5640cf657510 con 0x5640cf74fae0 2024-09-16T01:43:15.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.783+0000 7f5e0c6eaf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5640cf6b0c50 con 0x5640cf74f700 2024-09-16T01:43:15.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.783+0000 7f5e0c6eaf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5640cf707a70 con 0x5640cf7dd7b0 2024-09-16T01:43:15.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.783+0000 7f5e099df640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5640cf74fae0 0x5640cf704b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:15.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.783+0000 7f5e099df640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5640cf74fae0 0x5640cf704b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58736/0 (socket says 172.21.15.88:58736) 2024-09-16T01:43:15.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.783+0000 7f5e099df640 1 -- 172.21.15.88:0/2568198417 learned_addr learned my addr 172.21.15.88:0/2568198417 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:15.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.783+0000 7f5e0a9e1640 1 --2- 172.21.15.88:0/2568198417 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5640cf7dd7b0 0x5640cf7050a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:15.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.783+0000 7f5e0a1e0640 1 --2- 172.21.15.88:0/2568198417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640cf74f700 0x5640cf7d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:15.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.784+0000 7f5e099df640 1 -- 172.21.15.88:0/2568198417 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5640cf7dd7b0 msgr2=0x5640cf7050a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:15.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.784+0000 7f5e099df640 1 --2- 172.21.15.88:0/2568198417 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5640cf7dd7b0 0x5640cf7050a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.784+0000 7f5e099df640 1 -- 172.21.15.88:0/2568198417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640cf74f700 msgr2=0x5640cf7d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:15.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.784+0000 7f5e099df640 1 --2- 172.21.15.88:0/2568198417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640cf74f700 0x5640cf7d27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.784+0000 7f5e099df640 1 -- 172.21.15.88:0/2568198417 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640cf7d2cf0 con 0x5640cf74fae0 2024-09-16T01:43:15.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.784+0000 7f5e0a1e0640 1 --2- 172.21.15.88:0/2568198417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640cf74f700 0x5640cf7d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:15.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.784+0000 7f5e099df640 1 --2- 172.21.15.88:0/2568198417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5640cf74fae0 0x5640cf704b60 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f5df4055040 tx=0x7f5df4051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5b7c99c934e2453 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:15.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.784+0000 7f5e089dd640 1 -- 172.21.15.88:0/2568198417 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df4056040 con 0x5640cf74fae0 2024-09-16T01:43:15.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.784+0000 7f5e089dd640 1 -- 172.21.15.88:0/2568198417 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5df4052b30 con 0x5640cf74fae0 2024-09-16T01:43:15.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.784+0000 7f5e089dd640 1 -- 172.21.15.88:0/2568198417 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df4067480 con 0x5640cf74fae0 2024-09-16T01:43:15.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.784+0000 7f5e0c6eaf40 1 -- 172.21.15.88:0/2568198417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5640cf74fae0 msgr2=0x5640cf704b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:15.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.785+0000 7f5e0c6eaf40 1 --2- 172.21.15.88:0/2568198417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5640cf74fae0 0x5640cf704b60 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f5df4055040 tx=0x7f5df4051000 comp rx=0 tx=0).stop 2024-09-16T01:43:15.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.785+0000 7f5e0c6eaf40 1 -- 172.21.15.88:0/2568198417 shutdown_connections 2024-09-16T01:43:15.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.785+0000 7f5e0c6eaf40 1 --2- 172.21.15.88:0/2568198417 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5640cf7dd7b0 0x5640cf7050a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.785+0000 7f5e0c6eaf40 1 --2- 172.21.15.88:0/2568198417 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5640cf74fae0 0x5640cf704b60 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.785+0000 7f5e0c6eaf40 1 --2- 172.21.15.88:0/2568198417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640cf74f700 0x5640cf7d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.785+0000 7f5e0c6eaf40 1 -- 172.21.15.88:0/2568198417 >> 172.21.15.88:0/2568198417 conn(0x5640cf730680 msgr2=0x5640cf7d7a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:15.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.785+0000 7f5e0c6eaf40 1 -- 172.21.15.88:0/2568198417 shutdown_connections 2024-09-16T01:43:15.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.785+0000 7f5e0c6eaf40 1 -- 172.21.15.88:0/2568198417 wait complete. 2024-09-16T01:43:15.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.785+0000 7f5e0c6eaf40 1 Processor -- start 2024-09-16T01:43:15.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.785+0000 7f5e0c6eaf40 1 -- start start 2024-09-16T01:43:15.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.786+0000 7f5e0c6eaf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5640cf74f700 0x5640cf7ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.786+0000 7f5e0c6eaf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640cf74fae0 0x5640cf745e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.786+0000 7f5e0c6eaf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5640cf7dd7b0 0x5640cf7d3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.786+0000 7f5e0c6eaf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5640cf7d4f10 con 0x5640cf74f700 2024-09-16T01:43:15.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.786+0000 7f5e0c6eaf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5640cf705890 con 0x5640cf74fae0 2024-09-16T01:43:15.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.786+0000 7f5e0c6eaf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5640cf7da940 con 0x5640cf7dd7b0 2024-09-16T01:43:15.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.786+0000 7f5e0a1e0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5640cf74f700 0x5640cf7ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:15.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.786+0000 7f5e0a1e0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5640cf74f700 0x5640cf7ce0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58738/0 (socket says 172.21.15.88:58738) 2024-09-16T01:43:15.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.786+0000 7f5e0a1e0640 1 -- 172.21.15.88:0/56695196 learned_addr learned my addr 172.21.15.88:0/56695196 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:15.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.786+0000 7f5e0a1e0640 1 -- 172.21.15.88:0/56695196 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5640cf7dd7b0 msgr2=0x5640cf7d3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:15.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.786+0000 7f5e0a1e0640 1 --2- 172.21.15.88:0/56695196 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5640cf7dd7b0 0x5640cf7d3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.786+0000 7f5e0a1e0640 1 -- 172.21.15.88:0/56695196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640cf74fae0 msgr2=0x5640cf745e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:15.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.786+0000 7f5e0a1e0640 1 --2- 172.21.15.88:0/56695196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5640cf74fae0 0x5640cf745e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.786+0000 7f5e0a1e0640 1 -- 172.21.15.88:0/56695196 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640cf7ce5f0 con 0x5640cf74f700 2024-09-16T01:43:15.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.786+0000 7f5e0a1e0640 1 --2- 172.21.15.88:0/56695196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5640cf74f700 0x5640cf7ce0b0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f5e0400d440 tx=0x7f5e0400d900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:15.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.787+0000 7f5e037fe640 1 -- 172.21.15.88:0/56695196 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e04018070 con 0x5640cf74f700 2024-09-16T01:43:15.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.787+0000 7f5e037fe640 1 -- 172.21.15.88:0/56695196 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e040030d0 con 0x5640cf74f700 2024-09-16T01:43:15.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.787+0000 7f5e037fe640 1 -- 172.21.15.88:0/56695196 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e04001540 con 0x5640cf74f700 2024-09-16T01:43:15.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.787+0000 7f5e0c6eaf40 1 -- 172.21.15.88:0/56695196 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5640cf705fe0 con 0x5640cf74f700 2024-09-16T01:43:15.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.787+0000 7f5e0c6eaf40 1 -- 172.21.15.88:0/56695196 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640cf7065a0 con 0x5640cf74f700 2024-09-16T01:43:15.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.788+0000 7f5e037fe640 1 -- 172.21.15.88:0/56695196 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5e0401b070 con 0x5640cf74f700 2024-09-16T01:43:15.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.788+0000 7f5e037fe640 1 --2- 172.21.15.88:0/56695196 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5dec06acc0 0x7f5dec06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.788+0000 7f5e037fe640 1 -- 172.21.15.88:0/56695196 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8404+0+0 (secure 0 0 0) 0x7f5e0408a0a0 con 0x5640cf74f700 2024-09-16T01:43:15.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.788+0000 7f5e099df640 1 --2- 172.21.15.88:0/56695196 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5dec06acc0 0x7f5dec06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:15.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.789+0000 7f5e0c6eaf40 1 --2- 172.21.15.88:0/56695196 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5640cf746340 0x5640cf734040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.789+0000 7f5e0c6eaf40 1 -- 172.21.15.88:0/56695196 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=104b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118) -- 0x5640cf734580 con 0x5640cf746340 2024-09-16T01:43:15.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.789+0000 7f5e0a9e1640 1 --2- 172.21.15.88:0/56695196 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5640cf746340 0x5640cf734040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:15.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.789+0000 7f5e099df640 1 --2- 172.21.15.88:0/56695196 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5dec06acc0 0x7f5dec06d180 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f5df4063000 tx=0x7f5df4005a00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:15.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.789+0000 7f5e0a9e1640 1 --2- 172.21.15.88:0/56695196 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5640cf746340 0x5640cf734040 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:15.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.790+0000 7f5e0a9e1640 1 -- 172.21.15.88:0/56695196 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-vals] v118'970 uv970 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f5dfc00e580 con 0x5640cf746340 2024-09-16T01:43:15.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.790+0000 7f5e0c6eaf40 1 -- 172.21.15.88:0/56695196 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5640cf746340 msgr2=0x5640cf734040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:15.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.790+0000 7f5e0c6eaf40 1 --2- 172.21.15.88:0/56695196 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5640cf746340 0x5640cf734040 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.790+0000 7f5e0c6eaf40 1 -- 172.21.15.88:0/56695196 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5dec06acc0 msgr2=0x7f5dec06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:15.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.790+0000 7f5e0c6eaf40 1 --2- 172.21.15.88:0/56695196 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5dec06acc0 0x7f5dec06d180 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f5df4063000 tx=0x7f5df4005a00 comp rx=0 tx=0).stop 2024-09-16T01:43:15.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.790+0000 7f5e0c6eaf40 1 -- 172.21.15.88:0/56695196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5640cf74f700 msgr2=0x5640cf7ce0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:15.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.790+0000 7f5e0c6eaf40 1 --2- 172.21.15.88:0/56695196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5640cf74f700 0x5640cf7ce0b0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f5e0400d440 tx=0x7f5e0400d900 comp rx=0 tx=0).stop 2024-09-16T01:43:15.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.790+0000 7f5e0a9e1640 1 -- 172.21.15.88:0/56695196 reap_dead start 2024-09-16T01:43:15.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.790+0000 7f5e0c6eaf40 1 -- 172.21.15.88:0/56695196 shutdown_connections 2024-09-16T01:43:15.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.790+0000 7f5e0c6eaf40 1 -- 172.21.15.88:0/56695196 >> 172.21.15.88:0/56695196 conn(0x5640cf730680 msgr2=0x5640cf736510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:15.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.791+0000 7f5e0c6eaf40 1 -- 172.21.15.88:0/56695196 shutdown_connections 2024-09-16T01:43:15.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.791+0000 7f5e0c6eaf40 1 -- 172.21.15.88:0/56695196 wait complete. 2024-09-16T01:43:15.798 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:43:15.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.898+0000 7fc57172a640 1 Processor -- start 2024-09-16T01:43:15.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.898+0000 7fc57172a640 1 -- start start 2024-09-16T01:43:15.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.899+0000 7fc57172a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc56c0d1130 0x7fc56c0d1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.899+0000 7fc57172a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc56c0d4530 con 0x7fc56c0d1a50 2024-09-16T01:43:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.899+0000 7fc57172a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc56c0d3490 con 0x7fc56c0d0060 2024-09-16T01:43:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.899+0000 7fc57172a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc56c0d40a0 con 0x7fc56c0d1130 2024-09-16T01:43:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.899+0000 7fc5627fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc56c0d1a50 0x7fc56c0b7c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59394/0 (socket says 172.21.15.88:59394) 2024-09-16T01:43:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.899+0000 7fc5627fc640 1 -- 172.21.15.88:0/2507645162 learned_addr learned my addr 172.21.15.88:0/2507645162 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.899+0000 7fc56affd640 1 --2- 172.21.15.88:0/2507645162 >> v2:172.21.15.154:3300/0 conn(0x7fc56c0d1130 0x7fc56c0d1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.899+0000 7fc56a7fc640 1 -- 172.21.15.88:0/2507645162 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2390183511 0 0) 0x7fc56c0d4530 con 0x7fc56c0d1a50 2024-09-16T01:43:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.899+0000 7fc56a7fc640 1 -- 172.21.15.88:0/2507645162 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc558003520 con 0x7fc56c0d1a50 2024-09-16T01:43:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.899+0000 7fc56affd640 1 -- 172.21.15.88:0/2507645162 >> v1:172.21.15.154:6789/0 conn(0x7fc56c0d0060 legacy=0x7fc56c0b8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.899+0000 7fc56affd640 1 -- 172.21.15.88:0/2507645162 >> v1:172.21.15.88:6789/0 conn(0x7fc56c0d1a50 legacy=0x7fc56c0b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.899+0000 7fc56a7fc640 1 -- 172.21.15.88:0/2507645162 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1137971205 0 0) 0x7fc56c0d3490 con 0x7fc56c0d0060 2024-09-16T01:43:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.899+0000 7fc56affd640 1 -- 172.21.15.88:0/2507645162 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc56c0d3770 con 0x7fc56c0d1130 2024-09-16T01:43:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.900+0000 7fc56affd640 1 --2- 172.21.15.88:0/2507645162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc56c0d1130 0x7fc56c0d1510 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc55c0097a0 tx=0x7fc55c02a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc12fb4ed4198411 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.900+0000 7fc56a7fc640 1 -- 172.21.15.88:0/2507645162 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55c037070 con 0x7fc56c0d1130 2024-09-16T01:43:15.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.900+0000 7fc56a7fc640 1 -- 172.21.15.88:0/2507645162 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc55c032440 con 0x7fc56c0d1130 2024-09-16T01:43:15.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.900+0000 7fc56a7fc640 1 -- 172.21.15.88:0/2507645162 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55c032810 con 0x7fc56c0d1130 2024-09-16T01:43:15.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.901+0000 7fc57172a640 1 -- 172.21.15.88:0/2507645162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc56c0d1130 msgr2=0x7fc56c0d1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:15.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.901+0000 7fc57172a640 1 --2- 172.21.15.88:0/2507645162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc56c0d1130 0x7fc56c0d1510 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc55c0097a0 tx=0x7fc55c02a1a0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.901+0000 7fc57172a640 1 -- 172.21.15.88:0/2507645162 shutdown_connections 2024-09-16T01:43:15.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.901+0000 7fc57172a640 1 --2- 172.21.15.88:0/2507645162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc56c0d1130 0x7fc56c0d1510 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.901+0000 7fc57172a640 1 -- 172.21.15.88:0/2507645162 >> 172.21.15.88:0/2507645162 conn(0x7fc56c06c020 msgr2=0x7fc56c06c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:15.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.901+0000 7fc57172a640 1 -- 172.21.15.88:0/2507645162 shutdown_connections 2024-09-16T01:43:15.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.901+0000 7fc57172a640 1 -- 172.21.15.88:0/2507645162 wait complete. 2024-09-16T01:43:15.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.902+0000 7fc57172a640 1 Processor -- start 2024-09-16T01:43:15.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.902+0000 7fc57172a640 1 -- start start 2024-09-16T01:43:15.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.903+0000 7fc57172a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc56c0d1130 0x7fc56c1e4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.903+0000 7fc57172a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc56c0d1a50 0x7fc56c156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.903+0000 7fc57172a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc56c0cf930 0x7fc56c0cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.903+0000 7fc57172a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc56c0d1e90 con 0x7fc56c0cf930 2024-09-16T01:43:15.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.903+0000 7fc57172a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc56c0d3d30 con 0x7fc56c0d1130 2024-09-16T01:43:15.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.903+0000 7fc57172a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc56c0d3070 con 0x7fc56c0d1a50 2024-09-16T01:43:15.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.903+0000 7fc56b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc56c0cf930 0x7fc56c0cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:15.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.903+0000 7fc56b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc56c0cf930 0x7fc56c0cdb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58742/0 (socket says 172.21.15.88:58742) 2024-09-16T01:43:15.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.903+0000 7fc56b7fe640 1 -- 172.21.15.88:0/2255275220 learned_addr learned my addr 172.21.15.88:0/2255275220 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:15.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.903+0000 7fc56affd640 1 --2- 172.21.15.88:0/2255275220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc56c0d1130 0x7fc56c1e4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:15.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.903+0000 7fc5627fc640 1 --2- 172.21.15.88:0/2255275220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc56c0d1a50 0x7fc56c156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:15.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.903+0000 7fc56b7fe640 1 -- 172.21.15.88:0/2255275220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc56c0d1a50 msgr2=0x7fc56c156380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:15.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.903+0000 7fc56b7fe640 1 --2- 172.21.15.88:0/2255275220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc56c0d1a50 0x7fc56c156380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.903+0000 7fc56b7fe640 1 -- 172.21.15.88:0/2255275220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc56c0d1130 msgr2=0x7fc56c1e4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:15.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.903+0000 7fc56b7fe640 1 --2- 172.21.15.88:0/2255275220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc56c0d1130 0x7fc56c1e4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:15.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.903+0000 7fc56b7fe640 1 -- 172.21.15.88:0/2255275220 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc56c1e4bd0 con 0x7fc56c0cf930 2024-09-16T01:43:15.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.903+0000 7fc56b7fe640 1 --2- 172.21.15.88:0/2255275220 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc56c0cf930 0x7fc56c0cdb20 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fc56400c730 tx=0x7fc56400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:15.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.904+0000 7fc563fff640 1 -- 172.21.15.88:0/2255275220 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc564003da0 con 0x7fc56c0cf930 2024-09-16T01:43:15.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.904+0000 7fc563fff640 1 -- 172.21.15.88:0/2255275220 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc56400ce40 con 0x7fc56c0cf930 2024-09-16T01:43:15.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.904+0000 7fc57172a640 1 -- 172.21.15.88:0/2255275220 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc56c0cee50 con 0x7fc56c0cf930 2024-09-16T01:43:15.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.904+0000 7fc563fff640 1 -- 172.21.15.88:0/2255275220 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc564005270 con 0x7fc56c0cf930 2024-09-16T01:43:15.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.904+0000 7fc57172a640 1 -- 172.21.15.88:0/2255275220 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc56c0ceb60 con 0x7fc56c0cf930 2024-09-16T01:43:15.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.905+0000 7fc57172a640 1 -- 172.21.15.88:0/2255275220 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc56c0d1510 con 0x7fc56c0cf930 2024-09-16T01:43:15.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.905+0000 7fc563fff640 1 -- 172.21.15.88:0/2255275220 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc564005410 con 0x7fc56c0cf930 2024-09-16T01:43:15.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.906+0000 7fc563fff640 1 --2- 172.21.15.88:0/2255275220 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc54c06c2b0 0x7fc54c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:15.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.906+0000 7fc563fff640 1 -- 172.21.15.88:0/2255275220 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8404+0+0 (secure 0 0 0) 0x7fc56408bc80 con 0x7fc56c0cf930 2024-09-16T01:43:15.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.906+0000 7fc56affd640 1 --2- 172.21.15.88:0/2255275220 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc54c06c2b0 0x7fc54c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:15.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.906+0000 7fc56affd640 1 --2- 172.21.15.88:0/2255275220 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc54c06c2b0 0x7fc54c06e770 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fc55c034000 tx=0x7fc55c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:15.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:15.910+0000 7fc563fff640 1 -- 172.21.15.88:0/2255275220 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc5640592c0 con 0x7fc56c0cf930 2024-09-16T01:43:16.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.108+0000 7fc57172a640 1 -- 172.21.15.88:0/2255275220 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fc56c0b93b0 con 0x7fc56c0cf930 2024-09-16T01:43:16.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.474+0000 7fc563fff640 1 -- 172.21.15.88:0/2255275220 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v255) ==== 162+0+0 (secure 0 0 0) 0x7fc56405d5c0 con 0x7fc56c0cf930 2024-09-16T01:43:16.476 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:43:16.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.477+0000 7fc57172a640 1 -- 172.21.15.88:0/2255275220 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc54c06c2b0 msgr2=0x7fc54c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:16.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.477+0000 7fc57172a640 1 --2- 172.21.15.88:0/2255275220 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc54c06c2b0 0x7fc54c06e770 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fc55c034000 tx=0x7fc55c02e000 comp rx=0 tx=0).stop 2024-09-16T01:43:16.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.477+0000 7fc57172a640 1 -- 172.21.15.88:0/2255275220 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc56c0cf930 msgr2=0x7fc56c0cdb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:16.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.477+0000 7fc57172a640 1 --2- 172.21.15.88:0/2255275220 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc56c0cf930 0x7fc56c0cdb20 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fc56400c730 tx=0x7fc56400cbf0 comp rx=0 tx=0).stop 2024-09-16T01:43:16.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.477+0000 7fc57172a640 1 -- 172.21.15.88:0/2255275220 shutdown_connections 2024-09-16T01:43:16.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.477+0000 7fc57172a640 1 --2- 172.21.15.88:0/2255275220 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc54c06c2b0 0x7fc54c06e770 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:16.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.477+0000 7fc57172a640 1 --2- 172.21.15.88:0/2255275220 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc56c0cf930 0x7fc56c0cdb20 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:16.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.477+0000 7fc57172a640 1 --2- 172.21.15.88:0/2255275220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc56c0d1a50 0x7fc56c156380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:16.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.477+0000 7fc57172a640 1 --2- 172.21.15.88:0/2255275220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc56c0d1130 0x7fc56c1e4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:16.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.477+0000 7fc57172a640 1 -- 172.21.15.88:0/2255275220 >> 172.21.15.88:0/2255275220 conn(0x7fc56c06c020 msgr2=0x7fc56c0b5a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:16.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.477+0000 7fc57172a640 1 -- 172.21.15.88:0/2255275220 shutdown_connections 2024-09-16T01:43:16.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.477+0000 7fc57172a640 1 -- 172.21.15.88:0/2255275220 wait complete. 2024-09-16T01:43:16.893 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:43:16.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.991+0000 7ff93b65e640 1 Processor -- start 2024-09-16T01:43:16.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.991+0000 7ff93b65e640 1 -- start start 2024-09-16T01:43:16.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.992+0000 7ff93b65e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff9340b5be0 0x7ff93415ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:16.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.992+0000 7ff93b65e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff9340b29a0 0x7ff9340cde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:16.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.992+0000 7ff93b65e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9340b6830 con 0x7ff93415a9c0 2024-09-16T01:43:16.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.992+0000 7ff93b65e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff93415f410 con 0x7ff9340b5be0 2024-09-16T01:43:16.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.992+0000 7ff93b65e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff9340cd0b0 con 0x7ff9340b29a0 2024-09-16T01:43:16.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.992+0000 7ff938bf9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff9340b5be0 0x7ff93415ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:16.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.992+0000 7ff9393fa640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff93415a9c0 0x7ff93415e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59400/0 (socket says 172.21.15.88:59400) 2024-09-16T01:43:16.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.992+0000 7ff938bf9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff9340b5be0 0x7ff93415ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58756/0 (socket says 172.21.15.88:58756) 2024-09-16T01:43:16.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.992+0000 7ff9393fa640 1 -- 172.21.15.88:0/3447676196 learned_addr learned my addr 172.21.15.88:0/3447676196 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:16.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.992+0000 7ff939bfb640 1 --2- 172.21.15.88:0/3447676196 >> v2:172.21.15.154:3300/0 conn(0x7ff9340b29a0 0x7ff9340cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:16.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.992+0000 7ff938bf9640 1 -- 172.21.15.88:0/3447676196 >> v2:172.21.15.154:3300/0 conn(0x7ff9340b29a0 msgr2=0x7ff9340cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:16.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.992+0000 7ff938bf9640 1 --2- 172.21.15.88:0/3447676196 >> v2:172.21.15.154:3300/0 conn(0x7ff9340b29a0 0x7ff9340cde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:16.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.992+0000 7ff938bf9640 1 -- 172.21.15.88:0/3447676196 >> v1:172.21.15.88:6789/0 conn(0x7ff93415a9c0 legacy=0x7ff93415e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:16.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.992+0000 7ff938bf9640 1 -- 172.21.15.88:0/3447676196 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9340cd390 con 0x7ff9340b5be0 2024-09-16T01:43:16.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.992+0000 7ff92bfff640 1 -- 172.21.15.88:0/3447676196 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4178945284 0 0) 0x7ff9340b6830 con 0x7ff93415a9c0 2024-09-16T01:43:16.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.993+0000 7ff938bf9640 1 --2- 172.21.15.88:0/3447676196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9340b5be0 0x7ff93415ed10 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7ff93406f3e0 tx=0x7ff92402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fc63aaa5f540018 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:16.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.993+0000 7ff92bfff640 1 -- 172.21.15.88:0/3447676196 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff924037070 con 0x7ff9340b5be0 2024-09-16T01:43:16.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.993+0000 7ff92bfff640 1 -- 172.21.15.88:0/3447676196 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff924032440 con 0x7ff9340b5be0 2024-09-16T01:43:16.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.993+0000 7ff92bfff640 1 -- 172.21.15.88:0/3447676196 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff924032810 con 0x7ff9340b5be0 2024-09-16T01:43:16.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.994+0000 7ff93b65e640 1 -- 172.21.15.88:0/3447676196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9340b5be0 msgr2=0x7ff93415ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:16.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.994+0000 7ff93b65e640 1 --2- 172.21.15.88:0/3447676196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9340b5be0 0x7ff93415ed10 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7ff93406f3e0 tx=0x7ff92402a1a0 comp rx=0 tx=0).stop 2024-09-16T01:43:16.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.994+0000 7ff93b65e640 1 -- 172.21.15.88:0/3447676196 shutdown_connections 2024-09-16T01:43:16.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.994+0000 7ff93b65e640 1 --2- 172.21.15.88:0/3447676196 >> v2:172.21.15.154:3300/0 conn(0x7ff9340b29a0 0x7ff9340cde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:16.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.994+0000 7ff93b65e640 1 --2- 172.21.15.88:0/3447676196 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9340b5be0 0x7ff93415ed10 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:16.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.994+0000 7ff93b65e640 1 -- 172.21.15.88:0/3447676196 >> 172.21.15.88:0/3447676196 conn(0x7ff93406bc10 msgr2=0x7ff93406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:16.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.994+0000 7ff93b65e640 1 -- 172.21.15.88:0/3447676196 shutdown_connections 2024-09-16T01:43:16.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.994+0000 7ff93b65e640 1 -- 172.21.15.88:0/3447676196 wait complete. 2024-09-16T01:43:16.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.995+0000 7ff93b65e640 1 Processor -- start 2024-09-16T01:43:16.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.995+0000 7ff93b65e640 1 -- start start 2024-09-16T01:43:16.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.996+0000 7ff93b65e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9340b29a0 0x7ff9341542f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:16.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.996+0000 7ff93b65e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9340b5be0 0x7ff9341f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:16.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.996+0000 7ff93b65e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff93415a9c0 0x7ff93414fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:16.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.996+0000 7ff93b65e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff93415ada0 con 0x7ff9340b29a0 2024-09-16T01:43:17.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.996+0000 7ff93b65e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff9340cd950 con 0x7ff9340b5be0 2024-09-16T01:43:17.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.996+0000 7ff93b65e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff93406de40 con 0x7ff93415a9c0 2024-09-16T01:43:17.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.996+0000 7ff9393fa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9340b29a0 0x7ff9341542f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:17.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.996+0000 7ff9393fa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9340b29a0 0x7ff9341542f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:58770/0 (socket says 172.21.15.88:58770) 2024-09-16T01:43:17.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.996+0000 7ff9393fa640 1 -- 172.21.15.88:0/2940108262 learned_addr learned my addr 172.21.15.88:0/2940108262 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:17.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.996+0000 7ff9393fa640 1 -- 172.21.15.88:0/2940108262 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff93415a9c0 msgr2=0x7ff93414fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:17.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.996+0000 7ff9393fa640 1 --2- 172.21.15.88:0/2940108262 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff93415a9c0 0x7ff93414fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:17.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.996+0000 7ff9393fa640 1 -- 172.21.15.88:0/2940108262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9340b5be0 msgr2=0x7ff9341f3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:17.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.996+0000 7ff938bf9640 1 --2- 172.21.15.88:0/2940108262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9340b5be0 0x7ff9341f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:17.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.996+0000 7ff9393fa640 1 --2- 172.21.15.88:0/2940108262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9340b5be0 0x7ff9341f3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:17.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.996+0000 7ff9393fa640 1 -- 172.21.15.88:0/2940108262 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff934154830 con 0x7ff9340b29a0 2024-09-16T01:43:17.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.997+0000 7ff9393fa640 1 --2- 172.21.15.88:0/2940108262 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9340b29a0 0x7ff9341542f0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7ff93000c430 tx=0x7ff93000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:17.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.997+0000 7ff929ffb640 1 -- 172.21.15.88:0/2940108262 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9300074e0 con 0x7ff9340b29a0 2024-09-16T01:43:17.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.997+0000 7ff929ffb640 1 -- 172.21.15.88:0/2940108262 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9300047e0 con 0x7ff9340b29a0 2024-09-16T01:43:17.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.997+0000 7ff93b65e640 1 -- 172.21.15.88:0/2940108262 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9341f4290 con 0x7ff9340b29a0 2024-09-16T01:43:17.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.997+0000 7ff929ffb640 1 -- 172.21.15.88:0/2940108262 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff930004bd0 con 0x7ff9340b29a0 2024-09-16T01:43:17.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.997+0000 7ff93b65e640 1 -- 172.21.15.88:0/2940108262 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9341f4530 con 0x7ff9340b29a0 2024-09-16T01:43:17.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.998+0000 7ff93b65e640 1 -- 172.21.15.88:0/2940108262 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff93406e1d0 con 0x7ff9340b29a0 2024-09-16T01:43:17.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.998+0000 7ff929ffb640 1 -- 172.21.15.88:0/2940108262 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff930011070 con 0x7ff9340b29a0 2024-09-16T01:43:17.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.999+0000 7ff929ffb640 1 --2- 172.21.15.88:0/2940108262 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff91806c2b0 0x7ff91806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:17.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.999+0000 7ff938bf9640 1 --2- 172.21.15.88:0/2940108262 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff91806c2b0 0x7ff91806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:17.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.999+0000 7ff929ffb640 1 -- 172.21.15.88:0/2940108262 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8404+0+0 (secure 0 0 0) 0x7ff93008bb10 con 0x7ff9340b29a0 2024-09-16T01:43:17.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:16.999+0000 7ff938bf9640 1 --2- 172.21.15.88:0/2940108262 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff91806c2b0 0x7ff91806e770 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7ff924034000 tx=0x7ff92402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:17.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:17.003+0000 7ff929ffb640 1 -- 172.21.15.88:0/2940108262 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff934154830 con 0x7ff9340b29a0 2024-09-16T01:43:17.195 INFO:tasks.ceph.mds.c.smithi088.stderr: -46> 2024-09-16T01:43:12.441+0000 7f355781f640 -1 mds.0.cache.ino(0x100) No data while reading inode 0x100 2024-09-16T01:43:17.195 INFO:tasks.ceph.mds.c.smithi088.stderr: -36> 2024-09-16T01:43:12.442+0000 7f355781f640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-09-16T01:43:17.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:17.203+0000 7ff93b65e640 1 -- 172.21.15.88:0/2940108262 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff93414e2e0 con 0x7ff9340b29a0 2024-09-16T01:43:17.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:17.203+0000 7ff929ffb640 1 -- 172.21.15.88:0/2940108262 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 256 v256) ==== 95+0+4401 (secure 0 0 0) 0x7ff9300597a0 con 0x7ff9340b29a0 2024-09-16T01:43:17.205 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:17.205 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":256,"btime":"2024-09-16T01:43:16:477406+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":256,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:43:16.477396+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9636},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9636":{"gid":9636,"name":"a","rank":0,"incarnation":256,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6836/2555776364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2555776364},{"type":"v1","addr":"172.21.15.88:6836","nonce":2555776364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:43:17.205 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 256 2024-09-16T01:43:17.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:17.206+0000 7ff93b65e640 1 -- 172.21.15.88:0/2940108262 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff91806c2b0 msgr2=0x7ff91806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:17.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:17.206+0000 7ff93b65e640 1 --2- 172.21.15.88:0/2940108262 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff91806c2b0 0x7ff91806e770 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7ff924034000 tx=0x7ff92402e000 comp rx=0 tx=0).stop 2024-09-16T01:43:17.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:17.206+0000 7ff93b65e640 1 -- 172.21.15.88:0/2940108262 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9340b29a0 msgr2=0x7ff9341542f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:17.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:17.206+0000 7ff93b65e640 1 --2- 172.21.15.88:0/2940108262 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9340b29a0 0x7ff9341542f0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7ff93000c430 tx=0x7ff93000c8f0 comp rx=0 tx=0).stop 2024-09-16T01:43:17.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:17.206+0000 7ff93b65e640 1 -- 172.21.15.88:0/2940108262 shutdown_connections 2024-09-16T01:43:17.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:17.206+0000 7ff93b65e640 1 --2- 172.21.15.88:0/2940108262 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff91806c2b0 0x7ff91806e770 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:17.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:17.206+0000 7ff93b65e640 1 --2- 172.21.15.88:0/2940108262 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff93415a9c0 0x7ff93414fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:17.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:17.206+0000 7ff93b65e640 1 --2- 172.21.15.88:0/2940108262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9340b5be0 0x7ff9341f3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:17.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:17.206+0000 7ff93b65e640 1 --2- 172.21.15.88:0/2940108262 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9340b29a0 0x7ff9341542f0 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:17.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:17.206+0000 7ff93b65e640 1 -- 172.21.15.88:0/2940108262 >> 172.21.15.88:0/2940108262 conn(0x7ff93406bc10 msgr2=0x7ff934070f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:17.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:17.206+0000 7ff93b65e640 1 -- 172.21.15.88:0/2940108262 shutdown_connections 2024-09-16T01:43:17.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:17.206+0000 7ff93b65e640 1 -- 172.21.15.88:0/2940108262 wait complete. 2024-09-16T01:43:17.218 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 256, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:40:03.597796+0000', 'modified': '2024-09-16T01:43:16.477396+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 118, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9636}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9636': {'gid': 9636, 'name': 'a', 'rank': 0, 'incarnation': 256, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.88:6836/2555776364', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 2555776364}, {'type': 'v1', 'addr': '172.21.15.88:6836', 'nonce': 2555776364}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-16T01:43:17.219 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9636:up:replay 2024-09-16T01:43:17.219 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-09-16T01:43:17.219 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:43:18.258 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:43:18.261 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:43:22.622 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:43:22.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.731+0000 7fa7c8aa5640 1 Processor -- start 2024-09-16T01:43:22.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.732+0000 7fa7c8aa5640 1 -- start start 2024-09-16T01:43:22.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.732+0000 7fa7c8aa5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa7c406bc10 0x7fa7c406bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:22.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.732+0000 7fa7c8aa5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7c415cdb0 0x7fa7c415d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:22.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.732+0000 7fa7c8aa5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7c406c530 con 0x7fa7c415c710 2024-09-16T01:43:22.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.732+0000 7fa7c8aa5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa7c4072760 con 0x7fa7c406bc10 2024-09-16T01:43:22.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.732+0000 7fa7c8aa5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa7c4072a30 con 0x7fa7c415cdb0 2024-09-16T01:43:22.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.733+0000 7fa7c2d9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7c415cdb0 0x7fa7c415d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:22.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.733+0000 7fa7c259c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa7c406bc10 0x7fa7c406bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:22.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.733+0000 7fa7c2d9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7c415cdb0 0x7fa7c415d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:46920/0 (socket says 172.21.15.88:46920) 2024-09-16T01:43:22.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.733+0000 7fa7c2d9d640 1 -- 172.21.15.88:0/804310068 learned_addr learned my addr 172.21.15.88:0/804310068 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:22.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.733+0000 7fa7c1d9b640 1 -- 172.21.15.88:0/804310068 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3516614942 0 0) 0x7fa7c406c530 con 0x7fa7c415c710 2024-09-16T01:43:22.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.733+0000 7fa7c1d9b640 1 -- 172.21.15.88:0/804310068 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7b0003520 con 0x7fa7c415c710 2024-09-16T01:43:22.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.733+0000 7fa7c2d9d640 1 -- 172.21.15.88:0/804310068 >> v2:172.21.15.154:3300/0 conn(0x7fa7c406bc10 msgr2=0x7fa7c406bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:22.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.733+0000 7fa7c2d9d640 1 --2- 172.21.15.88:0/804310068 >> v2:172.21.15.154:3300/0 conn(0x7fa7c406bc10 0x7fa7c406bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:22.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.733+0000 7fa7c2d9d640 1 -- 172.21.15.88:0/804310068 >> v1:172.21.15.154:6789/0 conn(0x7fa7c415c710 legacy=0x7fa7c4072050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:22.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.733+0000 7fa7c2d9d640 1 -- 172.21.15.88:0/804310068 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7c4072d10 con 0x7fa7c415cdb0 2024-09-16T01:43:22.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.733+0000 7fa7c259c640 1 --2- 172.21.15.88:0/804310068 >> v2:172.21.15.154:3300/0 conn(0x7fa7c406bc10 0x7fa7c406bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:43:22.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.734+0000 7fa7c2d9d640 1 --2- 172.21.15.88:0/804310068 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7c415cdb0 0x7fa7c415d260 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc009e50 tx=0x7fa7bc02a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98eed1c5ea635da6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:22.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.734+0000 7fa7c1d9b640 1 -- 172.21.15.88:0/804310068 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc036070 con 0x7fa7c415cdb0 2024-09-16T01:43:22.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.734+0000 7fa7c1d9b640 1 -- 172.21.15.88:0/804310068 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7bc031600 con 0x7fa7c415cdb0 2024-09-16T01:43:22.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.734+0000 7fa7c1d9b640 1 -- 172.21.15.88:0/804310068 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc0319b0 con 0x7fa7c415cdb0 2024-09-16T01:43:22.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.735+0000 7fa7c8aa5640 1 -- 172.21.15.88:0/804310068 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7c415cdb0 msgr2=0x7fa7c415d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:22.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.735+0000 7fa7c8aa5640 1 --2- 172.21.15.88:0/804310068 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7c415cdb0 0x7fa7c415d260 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc009e50 tx=0x7fa7bc02a710 comp rx=0 tx=0).stop 2024-09-16T01:43:22.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.735+0000 7fa7c8aa5640 1 -- 172.21.15.88:0/804310068 shutdown_connections 2024-09-16T01:43:22.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.735+0000 7fa7c8aa5640 1 --2- 172.21.15.88:0/804310068 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7c415cdb0 0x7fa7c415d260 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:22.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.735+0000 7fa7c8aa5640 1 --2- 172.21.15.88:0/804310068 >> v2:172.21.15.154:3300/0 conn(0x7fa7c406bc10 0x7fa7c406bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:22.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.735+0000 7fa7c8aa5640 1 -- 172.21.15.88:0/804310068 >> 172.21.15.88:0/804310068 conn(0x7fa7c40d14d0 msgr2=0x7fa7c40d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:22.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.735+0000 7fa7c8aa5640 1 -- 172.21.15.88:0/804310068 shutdown_connections 2024-09-16T01:43:22.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.735+0000 7fa7c8aa5640 1 -- 172.21.15.88:0/804310068 wait complete. 2024-09-16T01:43:22.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.736+0000 7fa7c8aa5640 1 Processor -- start 2024-09-16T01:43:22.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.736+0000 7fa7c8aa5640 1 -- start start 2024-09-16T01:43:22.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.737+0000 7fa7c8aa5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7c406bc10 0x7fa7c4155230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:22.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.737+0000 7fa7c8aa5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7c415c710 0x7fa7c41568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:22.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.737+0000 7fa7c8aa5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7c415cdb0 0x7fa7c40cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:22.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.737+0000 7fa7c8aa5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c40735b0 con 0x7fa7c415c710 2024-09-16T01:43:22.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.737+0000 7fa7c8aa5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c40732d0 con 0x7fa7c406bc10 2024-09-16T01:43:22.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.737+0000 7fa7c8aa5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa7c406f850 con 0x7fa7c415cdb0 2024-09-16T01:43:22.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.737+0000 7fa7b9d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7c415c710 0x7fa7c41568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:22.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.737+0000 7fa7b9d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7c415c710 0x7fa7c41568d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43438/0 (socket says 172.21.15.88:43438) 2024-09-16T01:43:22.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.737+0000 7fa7b9d9b640 1 -- 172.21.15.88:0/1116253120 learned_addr learned my addr 172.21.15.88:0/1116253120 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:22.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.737+0000 7fa7c259c640 1 --2- 172.21.15.88:0/1116253120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7c406bc10 0x7fa7c4155230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:22.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.737+0000 7fa7c2d9d640 1 --2- 172.21.15.88:0/1116253120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7c415cdb0 0x7fa7c40cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:22.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.738+0000 7fa7b9d9b640 1 -- 172.21.15.88:0/1116253120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7c415cdb0 msgr2=0x7fa7c40cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:22.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.738+0000 7fa7b9d9b640 1 --2- 172.21.15.88:0/1116253120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7c415cdb0 0x7fa7c40cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:22.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.738+0000 7fa7b9d9b640 1 -- 172.21.15.88:0/1116253120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7c406bc10 msgr2=0x7fa7c4155230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:22.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.738+0000 7fa7b9d9b640 1 --2- 172.21.15.88:0/1116253120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7c406bc10 0x7fa7c4155230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:22.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.738+0000 7fa7b9d9b640 1 -- 172.21.15.88:0/1116253120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7c4155830 con 0x7fa7c415c710 2024-09-16T01:43:22.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.738+0000 7fa7b9d9b640 1 --2- 172.21.15.88:0/1116253120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7c415c710 0x7fa7c41568d0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac00c500 tx=0x7fa7ac00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:22.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.738+0000 7fa7bb7fe640 1 -- 172.21.15.88:0/1116253120 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac0052d0 con 0x7fa7c415c710 2024-09-16T01:43:22.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.738+0000 7fa7c8aa5640 1 -- 172.21.15.88:0/1116253120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7c4155b10 con 0x7fa7c415c710 2024-09-16T01:43:22.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.738+0000 7fa7bb7fe640 1 -- 172.21.15.88:0/1116253120 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7ac005490 con 0x7fa7c415c710 2024-09-16T01:43:22.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.738+0000 7fa7bb7fe640 1 -- 172.21.15.88:0/1116253120 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac012870 con 0x7fa7c415c710 2024-09-16T01:43:22.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.738+0000 7fa7c8aa5640 1 -- 172.21.15.88:0/1116253120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7c41560d0 con 0x7fa7c415c710 2024-09-16T01:43:22.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.739+0000 7fa7c8aa5640 1 -- 172.21.15.88:0/1116253120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7c406bff0 con 0x7fa7c415c710 2024-09-16T01:43:22.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.739+0000 7fa7bb7fe640 1 -- 172.21.15.88:0/1116253120 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa7ac012a10 con 0x7fa7c415c710 2024-09-16T01:43:22.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.740+0000 7fa7bb7fe640 1 --2- 172.21.15.88:0/1116253120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa7a406c2b0 0x7fa7a406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:22.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.740+0000 7fa7bb7fe640 1 -- 172.21.15.88:0/1116253120 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8404+0+0 (secure 0 0 0) 0x7fa7ac08baf0 con 0x7fa7c415c710 2024-09-16T01:43:22.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.740+0000 7fa7c259c640 1 --2- 172.21.15.88:0/1116253120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa7a406c2b0 0x7fa7a406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:22.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.740+0000 7fa7c259c640 1 --2- 172.21.15.88:0/1116253120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa7a406c2b0 0x7fa7a406e770 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4009560 tx=0x7fa7b4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:22.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.744+0000 7fa7bb7fe640 1 -- 172.21.15.88:0/1116253120 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa7ac059140 con 0x7fa7c415c710 2024-09-16T01:43:22.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.943+0000 7fa7c8aa5640 1 -- 172.21.15.88:0/1116253120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7c406c7c0 con 0x7fa7c415c710 2024-09-16T01:43:22.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.943+0000 7fa7bb7fe640 1 -- 172.21.15.88:0/1116253120 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 259 v259) ==== 95+0+5234 (secure 0 0 0) 0x7fa7ac05d440 con 0x7fa7c415c710 2024-09-16T01:43:22.945 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:22.946 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":259,"btime":"2024-09-16T01:43:19:502492+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9696,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4270197461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4270197461},{"type":"v1","addr":"172.21.15.88:6837","nonce":4270197461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":259,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:43:19.502487+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9636},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9636":{"gid":9636,"name":"a","rank":0,"incarnation":256,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6836/2555776364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2555776364},{"type":"v1","addr":"172.21.15.88:6836","nonce":2555776364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9636,"qdb_cluster":[9636]},"id":6}]} 2024-09-16T01:43:22.946 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 259 2024-09-16T01:43:22.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.945+0000 7fa7c8aa5640 1 -- 172.21.15.88:0/1116253120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa7a406c2b0 msgr2=0x7fa7a406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:22.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.945+0000 7fa7c8aa5640 1 --2- 172.21.15.88:0/1116253120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa7a406c2b0 0x7fa7a406e770 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4009560 tx=0x7fa7b4009190 comp rx=0 tx=0).stop 2024-09-16T01:43:22.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.945+0000 7fa7c8aa5640 1 -- 172.21.15.88:0/1116253120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7c415c710 msgr2=0x7fa7c41568d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:22.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.945+0000 7fa7c8aa5640 1 --2- 172.21.15.88:0/1116253120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7c415c710 0x7fa7c41568d0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac00c500 tx=0x7fa7ac00c9c0 comp rx=0 tx=0).stop 2024-09-16T01:43:22.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.945+0000 7fa7c8aa5640 1 -- 172.21.15.88:0/1116253120 shutdown_connections 2024-09-16T01:43:22.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.945+0000 7fa7c8aa5640 1 --2- 172.21.15.88:0/1116253120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa7a406c2b0 0x7fa7a406e770 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:22.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.945+0000 7fa7c8aa5640 1 --2- 172.21.15.88:0/1116253120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa7c415cdb0 0x7fa7c40cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:22.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.945+0000 7fa7c8aa5640 1 --2- 172.21.15.88:0/1116253120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7c415c710 0x7fa7c41568d0 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:22.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.945+0000 7fa7c8aa5640 1 --2- 172.21.15.88:0/1116253120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7c406bc10 0x7fa7c4155230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:22.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.945+0000 7fa7c8aa5640 1 -- 172.21.15.88:0/1116253120 >> 172.21.15.88:0/1116253120 conn(0x7fa7c40d14d0 msgr2=0x7fa7c415f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:22.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.946+0000 7fa7c8aa5640 1 -- 172.21.15.88:0/1116253120 shutdown_connections 2024-09-16T01:43:22.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:22.946+0000 7fa7c8aa5640 1 -- 172.21.15.88:0/1116253120 wait complete. 2024-09-16T01:43:22.958 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 259, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:40:03.597796+0000', 'modified': '2024-09-16T01:43:19.502487+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 118, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9636}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9636': {'gid': 9636, 'name': 'a', 'rank': 0, 'incarnation': 256, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.88:6836/2555776364', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 2555776364}, {'type': 'v1', 'addr': '172.21.15.88:6836', 'nonce': 2555776364}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9636, 'qdb_cluster': [9636]} 2024-09-16T01:43:22.959 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T01:43:22.959 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-16T01:43:23.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.057+0000 7f6f0b731640 1 Processor -- start 2024-09-16T01:43:23.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.058+0000 7f6f0b731640 1 -- start start 2024-09-16T01:43:23.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.059+0000 7f6f0b731640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6f0415e7a0 0x7f6f0406caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.059+0000 7f6f0b731640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6f040cf1f0 0x7f6f040b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.059+0000 7f6f0b731640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f040cf860 con 0x7f6f040b6720 2024-09-16T01:43:23.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.059+0000 7f6f0b731640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6f040b6b90 con 0x7f6f040cf1f0 2024-09-16T01:43:23.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.059+0000 7f6f0b731640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6f040d2ca0 con 0x7f6f0415e7a0 2024-09-16T01:43:23.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.059+0000 7f6f08ccc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6f040cf1f0 0x7f6f040b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:23.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.059+0000 7f6f08ccc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6f040cf1f0 0x7f6f040b61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43442/0 (socket says 172.21.15.88:43442) 2024-09-16T01:43:23.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.059+0000 7f6f08ccc640 1 -- 172.21.15.88:0/951236820 learned_addr learned my addr 172.21.15.88:0/951236820 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:23.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.059+0000 7f6f08ccc640 1 -- 172.21.15.88:0/951236820 >> v2:172.21.15.154:3300/0 conn(0x7f6f0415e7a0 msgr2=0x7f6f0406caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:23.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.059+0000 7f6f08ccc640 1 --2- 172.21.15.88:0/951236820 >> v2:172.21.15.154:3300/0 conn(0x7f6f0415e7a0 0x7f6f0406caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.059+0000 7f6f08ccc640 1 -- 172.21.15.88:0/951236820 >> v1:172.21.15.88:6789/0 conn(0x7f6f040b6720 legacy=0x7f6f041623c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:23.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.059+0000 7f6f08ccc640 1 -- 172.21.15.88:0/951236820 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f040cfb40 con 0x7f6f040cf1f0 2024-09-16T01:43:23.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.059+0000 7f6ef7fff640 1 -- 172.21.15.88:0/951236820 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1654130732 0 0) 0x7f6f040cf860 con 0x7f6f040b6720 2024-09-16T01:43:23.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.060+0000 7f6f08ccc640 1 --2- 172.21.15.88:0/951236820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f040cf1f0 0x7f6f040b61e0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f6f040d5820 tx=0x7f6f0002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e518bcb9e1d44ba server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:23.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.060+0000 7f6ef7fff640 1 -- 172.21.15.88:0/951236820 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f00037070 con 0x7f6f040cf1f0 2024-09-16T01:43:23.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.060+0000 7f6ef7fff640 1 -- 172.21.15.88:0/951236820 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f00032440 con 0x7f6f040cf1f0 2024-09-16T01:43:23.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.060+0000 7f6ef7fff640 1 -- 172.21.15.88:0/951236820 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f00032810 con 0x7f6f040cf1f0 2024-09-16T01:43:23.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.060+0000 7f6f0b731640 1 -- 172.21.15.88:0/951236820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f040cf1f0 msgr2=0x7f6f040b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:23.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.060+0000 7f6f0b731640 1 --2- 172.21.15.88:0/951236820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f040cf1f0 0x7f6f040b61e0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f6f040d5820 tx=0x7f6f0002a1e0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.060+0000 7f6f0b731640 1 -- 172.21.15.88:0/951236820 shutdown_connections 2024-09-16T01:43:23.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.060+0000 7f6f0b731640 1 --2- 172.21.15.88:0/951236820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f040cf1f0 0x7f6f040b61e0 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.060+0000 7f6f0b731640 1 --2- 172.21.15.88:0/951236820 >> v2:172.21.15.154:3300/0 conn(0x7f6f0415e7a0 0x7f6f0406caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.060+0000 7f6f0b731640 1 -- 172.21.15.88:0/951236820 >> 172.21.15.88:0/951236820 conn(0x7f6f040d11e0 msgr2=0x7f6f040d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:23.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.060+0000 7f6f0b731640 1 -- 172.21.15.88:0/951236820 shutdown_connections 2024-09-16T01:43:23.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.060+0000 7f6f0b731640 1 -- 172.21.15.88:0/951236820 wait complete. 2024-09-16T01:43:23.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.062+0000 7f6f0b731640 1 Processor -- start 2024-09-16T01:43:23.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.062+0000 7f6f0b731640 1 -- start start 2024-09-16T01:43:23.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.062+0000 7f6f0b731640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f040b6720 0x7f6f040cad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.062+0000 7f6f0b731640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f040cf1f0 0x7f6f0414b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.062+0000 7f6f0b731640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f0415e7a0 0x7f6f040cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.062+0000 7f6f0b731640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6f040b7a00 con 0x7f6f040cf1f0 2024-09-16T01:43:23.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.062+0000 7f6f0b731640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6f040d2ef0 con 0x7f6f0415e7a0 2024-09-16T01:43:23.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.062+0000 7f6f0b731640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6f0406cfe0 con 0x7f6f040b6720 2024-09-16T01:43:23.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.062+0000 7f6f08ccc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f040cf1f0 0x7f6f0414b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:23.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.062+0000 7f6f08ccc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f040cf1f0 0x7f6f0414b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43456/0 (socket says 172.21.15.88:43456) 2024-09-16T01:43:23.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.062+0000 7f6f08ccc640 1 -- 172.21.15.88:0/604311181 learned_addr learned my addr 172.21.15.88:0/604311181 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:23.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.063+0000 7f6f094cd640 1 --2- 172.21.15.88:0/604311181 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f040b6720 0x7f6f040cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:23.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.063+0000 7f6f09cce640 1 --2- 172.21.15.88:0/604311181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f0415e7a0 0x7f6f040cdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:23.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.063+0000 7f6f08ccc640 1 -- 172.21.15.88:0/604311181 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f040b6720 msgr2=0x7f6f040cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:23.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.063+0000 7f6f08ccc640 1 --2- 172.21.15.88:0/604311181 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f040b6720 0x7f6f040cad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.063+0000 7f6f08ccc640 1 -- 172.21.15.88:0/604311181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f0415e7a0 msgr2=0x7f6f040cdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:23.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.063+0000 7f6f08ccc640 1 --2- 172.21.15.88:0/604311181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f0415e7a0 0x7f6f040cdc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.063+0000 7f6f08ccc640 1 -- 172.21.15.88:0/604311181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f040cba80 con 0x7f6f040cf1f0 2024-09-16T01:43:23.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.063+0000 7f6f08ccc640 1 --2- 172.21.15.88:0/604311181 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f040cf1f0 0x7f6f0414b5b0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f6f00035040 tx=0x7f6f000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:23.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.063+0000 7f6ef5ffb640 1 -- 172.21.15.88:0/604311181 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f00037040 con 0x7f6f040cf1f0 2024-09-16T01:43:23.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.063+0000 7f6ef5ffb640 1 -- 172.21.15.88:0/604311181 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f00004230 con 0x7f6f040cf1f0 2024-09-16T01:43:23.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.063+0000 7f6f0b731640 1 -- 172.21.15.88:0/604311181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f040cbd60 con 0x7f6f040cf1f0 2024-09-16T01:43:23.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.063+0000 7f6ef5ffb640 1 -- 172.21.15.88:0/604311181 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0003c730 con 0x7f6f040cf1f0 2024-09-16T01:43:23.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.063+0000 7f6f0b731640 1 -- 172.21.15.88:0/604311181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f040cc320 con 0x7f6f040cf1f0 2024-09-16T01:43:23.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.065+0000 7f6ef5ffb640 1 -- 172.21.15.88:0/604311181 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6f0003a070 con 0x7f6f040cf1f0 2024-09-16T01:43:23.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.065+0000 7f6ef5ffb640 1 --2- 172.21.15.88:0/604311181 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ee806c2b0 0x7f6ee806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.065+0000 7f6ef5ffb640 1 -- 172.21.15.88:0/604311181 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8404+0+0 (secure 0 0 0) 0x7f6f000aca10 con 0x7f6f040cf1f0 2024-09-16T01:43:23.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.065+0000 7f6f094cd640 1 --2- 172.21.15.88:0/604311181 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ee806c2b0 0x7f6ee806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:23.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.066+0000 7f6f094cd640 1 --2- 172.21.15.88:0/604311181 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ee806c2b0 0x7f6ee806e770 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8009890 tx=0x7f6ef8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:23.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.066+0000 7f6f0e4d3740 1 Processor -- start 2024-09-16T01:43:23.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.066+0000 7f6f0e4d3740 1 -- start start 2024-09-16T01:43:23.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.067+0000 7f6f0e4d3740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600c68d82c0 0x5600c691da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.067+0000 7f6f0e4d3740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d5a80 0x5600c68d86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.067+0000 7f6f0e4d3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600c68d9a50 0x5600c68d28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.067+0000 7f6f0e4d3740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5600c683a520 con 0x5600c68d5a80 2024-09-16T01:43:23.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.067+0000 7f6f0e4d3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5600c67e23e0 con 0x5600c68d9a50 2024-09-16T01:43:23.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.067+0000 7f6f0e4d3740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5600c68364e0 con 0x5600c68d82c0 2024-09-16T01:43:23.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.067+0000 7f6f08ccc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d5a80 0x5600c68d86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:23.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.067+0000 7f6f08ccc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d5a80 0x5600c68d86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43468/0 (socket says 172.21.15.88:43468) 2024-09-16T01:43:23.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.067+0000 7f6f08ccc640 1 -- 172.21.15.88:0/4203249820 learned_addr learned my addr 172.21.15.88:0/4203249820 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:23.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.067+0000 7f6f094cd640 1 --2- 172.21.15.88:0/4203249820 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600c68d82c0 0x5600c691da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:23.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.067+0000 7f6f09cce640 1 --2- 172.21.15.88:0/4203249820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600c68d9a50 0x5600c68d28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:23.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.067+0000 7f6f08ccc640 1 -- 172.21.15.88:0/4203249820 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600c68d82c0 msgr2=0x5600c691da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:23.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.067+0000 7f6f08ccc640 1 --2- 172.21.15.88:0/4203249820 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600c68d82c0 0x5600c691da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.067+0000 7f6f08ccc640 1 -- 172.21.15.88:0/4203249820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600c68d9a50 msgr2=0x5600c68d28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:23.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.067+0000 7f6f08ccc640 1 --2- 172.21.15.88:0/4203249820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600c68d9a50 0x5600c68d28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.067+0000 7f6f08ccc640 1 -- 172.21.15.88:0/4203249820 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600c67fd2f0 con 0x5600c68d5a80 2024-09-16T01:43:23.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.067+0000 7f6f08ccc640 1 --2- 172.21.15.88:0/4203249820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d5a80 0x5600c68d86a0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f6f00076490 tx=0x7f6f00009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b7cca5fada3b8ba server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:23.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.068+0000 7f6eeeffd640 1 -- 172.21.15.88:0/4203249820 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0004a040 con 0x5600c68d5a80 2024-09-16T01:43:23.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.068+0000 7f6eeeffd640 1 -- 172.21.15.88:0/4203249820 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f00082ca0 con 0x5600c68d5a80 2024-09-16T01:43:23.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.068+0000 7f6eeeffd640 1 -- 172.21.15.88:0/4203249820 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f00092120 con 0x5600c68d5a80 2024-09-16T01:43:23.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.068+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/4203249820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d5a80 msgr2=0x5600c68d86a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:23.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.068+0000 7f6f0e4d3740 1 --2- 172.21.15.88:0/4203249820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d5a80 0x5600c68d86a0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f6f00076490 tx=0x7f6f00009240 comp rx=0 tx=0).stop 2024-09-16T01:43:23.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.068+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/4203249820 shutdown_connections 2024-09-16T01:43:23.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.068+0000 7f6f0e4d3740 1 --2- 172.21.15.88:0/4203249820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600c68d9a50 0x5600c68d28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.068+0000 7f6f0e4d3740 1 --2- 172.21.15.88:0/4203249820 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d5a80 0x5600c68d86a0 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.068+0000 7f6f0e4d3740 1 --2- 172.21.15.88:0/4203249820 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600c68d82c0 0x5600c691da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.068+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/4203249820 >> 172.21.15.88:0/4203249820 conn(0x5600c68d4a80 msgr2=0x5600c691cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:23.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.068+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/4203249820 shutdown_connections 2024-09-16T01:43:23.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.068+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/4203249820 wait complete. 2024-09-16T01:43:23.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.069+0000 7f6f0e4d3740 1 -- start start 2024-09-16T01:43:23.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.069+0000 7f6f0e4d3740 1 Processor -- start 2024-09-16T01:43:23.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.070+0000 7f6f0e4d3740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600c68d82c0 0x5600c6902970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.070+0000 7f6f0e4d3740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d9a50 0x5600c68f8b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.070+0000 7f6f0e4d3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600c691a7d0 0x5600c68fb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.070+0000 7f6f0e4d3740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5600c683a520 con 0x5600c68d9a50 2024-09-16T01:43:23.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.070+0000 7f6f0e4d3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5600c67c3450 con 0x5600c691a7d0 2024-09-16T01:43:23.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.070+0000 7f6f0e4d3740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5600c68d3f20 con 0x5600c68d82c0 2024-09-16T01:43:23.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.070+0000 7f6f0e4d3740 10 client.0 _ll_register_callbacks cb 0x5600c654cac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:43:23.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.070+0000 7f6f08ccc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d9a50 0x5600c68f8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:23.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.070+0000 7f6f08ccc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d9a50 0x5600c68f8b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43470/0 (socket says 172.21.15.88:43470) 2024-09-16T01:43:23.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.070+0000 7f6f08ccc640 1 -- 172.21.15.88:0/993152867 learned_addr learned my addr 172.21.15.88:0/993152867 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:23.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.070+0000 7f6f09cce640 1 --2- 172.21.15.88:0/993152867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600c691a7d0 0x5600c68fb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:23.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.070+0000 7f6f094cd640 1 --2- 172.21.15.88:0/993152867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600c68d82c0 0x5600c6902970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:23.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.070+0000 7f6f08ccc640 1 -- 172.21.15.88:0/993152867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600c68d82c0 msgr2=0x5600c6902970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:23.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.070+0000 7f6f08ccc640 1 --2- 172.21.15.88:0/993152867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600c68d82c0 0x5600c6902970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.070+0000 7f6f08ccc640 1 -- 172.21.15.88:0/993152867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600c691a7d0 msgr2=0x5600c68fb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:23.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.070+0000 7f6f08ccc640 1 --2- 172.21.15.88:0/993152867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600c691a7d0 0x5600c68fb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.070+0000 7f6f08ccc640 1 -- 172.21.15.88:0/993152867 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600c654d2d0 con 0x5600c68d9a50 2024-09-16T01:43:23.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.070+0000 7f6f08ccc640 1 --2- 172.21.15.88:0/993152867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d9a50 0x5600c68f8b70 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f6f00079d30 tx=0x7f6f0002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeb32fdcb14c6ba0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:23.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.070+0000 7f6ee3fff640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:43:23.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.070+0000 7f6ee3fff640 1 -- 172.21.15.88:0/993152867 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0004a040 con 0x5600c68d9a50 2024-09-16T01:43:23.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.070+0000 7f6ee3fff640 1 -- 172.21.15.88:0/993152867 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8404+0+0 (secure 0 0 0) 0x7f6f00082810 con 0x5600c68d9a50 2024-09-16T01:43:23.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.070+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/993152867 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f00082ca0 con 0x5600c68d9a50 2024-09-16T01:43:23.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.071+0000 7f6ee3fff640 1 -- 172.21.15.88:0/993152867 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=259) ==== 24+0+0 (secure 0 0 0) 0x7f6f00092120 con 0x5600c68d9a50 2024-09-16T01:43:23.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.071+0000 7f6f0e4d3740 10 client.9714 fetch_fsmap learned FSMap version 259 2024-09-16T01:43:23.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.071+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/993152867 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=259,monmap=2+}) -- 0x5600c66176b0 con 0x5600c68d9a50 2024-09-16T01:43:23.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.071+0000 7f6ee3fff640 1 -- 172.21.15.88:0/993152867 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f00082ca0 con 0x5600c68d9a50 2024-09-16T01:43:23.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.071+0000 7f6ee3fff640 1 -- 172.21.15.88:0/993152867 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 259) ==== 3502+0+0 (secure 0 0 0) 0x7f6f0008b020 con 0x5600c68d9a50 2024-09-16T01:43:23.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.071+0000 7f6f0e4d3740 10 client.9714 fetch_fsmap finished waiting for FSMap version 259 2024-09-16T01:43:23.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.071+0000 7f6f0e4d3740 10 client.9714 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:43:23.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.071+0000 7f6f0e4d3740 20 client.9714 populate_metadata read hostname 'smithi088' 2024-09-16T01:43:23.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.071+0000 7f6f0e4d3740 1 --2- 172.21.15.88:0/993152867 >> [v2:172.21.15.88:6834/2555776364,v1:172.21.15.88:6836/2555776364] conn(0x5600c6900320 0x5600c6930570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.071+0000 7f6f0e4d3740 4 client.9714 mds_command: new command op to 9636 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:43:23.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.071+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/993152867 --> [v2:172.21.15.88:6834/2555776364,v1:172.21.15.88:6836/2555776364] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600c6847c50 con 0x5600c6900320 2024-09-16T01:43:23.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.071+0000 7f6f094cd640 1 --2- 172.21.15.88:0/993152867 >> [v2:172.21.15.88:6834/2555776364,v1:172.21.15.88:6836/2555776364] conn(0x5600c6900320 0x5600c6930570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:23.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.071+0000 7f6f094cd640 1 --2- 172.21.15.88:0/993152867 >> [v2:172.21.15.88:6834/2555776364,v1:172.21.15.88:6836/2555776364] conn(0x5600c6900320 0x5600c6930570 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e7d3d38c68bc52 server_cookie=305156fe970815ed in_seq=0 out_seq=0 2024-09-16T01:43:23.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.071+0000 7f6ee3fff640 10 client.9714 ms_handle_connect on v2:172.21.15.88:6834/2555776364 2024-09-16T01:43:23.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.072+0000 7f6ee3fff640 1 -- 172.21.15.88:0/993152867 <== mds.0 v2:172.21.15.88:6834/2555776364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5600c6847c50 con 0x5600c6900320 2024-09-16T01:43:23.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.072+0000 7f6ee3fff640 10 client.9714 handle_command_reply: tid=0 2024-09-16T01:43:23.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.072+0000 7f6f0e4d3740 1 client.9714 shutdown 2024-09-16T01:43:23.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.072+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/993152867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d9a50 msgr2=0x5600c68f8b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:23.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.072+0000 7f6f0e4d3740 1 --2- 172.21.15.88:0/993152867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d9a50 0x5600c68f8b70 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f6f00079d30 tx=0x7f6f0002a1e0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.072+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/993152867 shutdown_connections 2024-09-16T01:43:23.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.072+0000 7f6f0e4d3740 1 --2- 172.21.15.88:0/993152867 >> [v2:172.21.15.88:6834/2555776364,v1:172.21.15.88:6836/2555776364] conn(0x5600c6900320 0x5600c6930570 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.072+0000 7f6f0e4d3740 1 --2- 172.21.15.88:0/993152867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600c691a7d0 0x5600c68fb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.072+0000 7f6f0e4d3740 1 --2- 172.21.15.88:0/993152867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d9a50 0x5600c68f8b70 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.072+0000 7f6ee3fff640 0 client.9714 ms_handle_reset on v2:172.21.15.88:6834/2555776364 2024-09-16T01:43:23.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.072+0000 7f6f0e4d3740 1 --2- 172.21.15.88:0/993152867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600c68d82c0 0x5600c6902970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.072+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/993152867 >> 172.21.15.88:0/993152867 conn(0x5600c68d5a80 msgr2=0x5600c691c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:23.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.072+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/993152867 shutdown_connections 2024-09-16T01:43:23.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.072+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/993152867 wait complete. 2024-09-16T01:43:23.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.072+0000 7f6f0e4d3740 20 client.9714 trim_cache size 0 max 16384 2024-09-16T01:43:23.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.090+0000 7f6f0e4d3740 1 Processor -- start 2024-09-16T01:43:23.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.090+0000 7f6f0e4d3740 1 -- start start 2024-09-16T01:43:23.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.091+0000 7f6f0e4d3740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d82c0 0x5600c691b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.091+0000 7f6f0e4d3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600c68d9a50 0x5600c691b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.091+0000 7f6f0e4d3740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600c68d9310 0x5600c691c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.091+0000 7f6f0e4d3740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6f04201660 con 0x5600c68d82c0 2024-09-16T01:43:23.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.091+0000 7f6f0e4d3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6f040c9060 con 0x5600c68d9a50 2024-09-16T01:43:23.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.091+0000 7f6f0e4d3740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6f04201040 con 0x5600c68d9310 2024-09-16T01:43:23.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.091+0000 7f6f094cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d82c0 0x5600c691b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:23.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.091+0000 7f6f094cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d82c0 0x5600c691b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43472/0 (socket says 172.21.15.88:43472) 2024-09-16T01:43:23.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.091+0000 7f6f094cd640 1 -- 172.21.15.88:0/2672225738 learned_addr learned my addr 172.21.15.88:0/2672225738 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:23.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.091+0000 7f6f094cd640 1 -- 172.21.15.88:0/2672225738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600c68d9310 msgr2=0x5600c691c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:23.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.091+0000 7f6f094cd640 1 --2- 172.21.15.88:0/2672225738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600c68d9310 0x5600c691c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.091+0000 7f6f094cd640 1 -- 172.21.15.88:0/2672225738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600c68d9a50 msgr2=0x5600c691b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:23.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.091+0000 7f6f094cd640 1 --2- 172.21.15.88:0/2672225738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600c68d9a50 0x5600c691b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.091+0000 7f6f094cd640 1 -- 172.21.15.88:0/2672225738 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600c66176b0 con 0x5600c68d82c0 2024-09-16T01:43:23.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.091+0000 7f6f094cd640 1 --2- 172.21.15.88:0/2672225738 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d82c0 0x5600c691b1a0 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8009750 tx=0x7f6ef8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d6fd6dfbd4b29c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:23.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.091+0000 7f6eee7fc640 1 -- 172.21.15.88:0/2672225738 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef802f070 con 0x5600c68d82c0 2024-09-16T01:43:23.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.091+0000 7f6eee7fc640 1 -- 172.21.15.88:0/2672225738 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ef800dd20 con 0x5600c68d82c0 2024-09-16T01:43:23.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.091+0000 7f6eee7fc640 1 -- 172.21.15.88:0/2672225738 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8014830 con 0x5600c68d82c0 2024-09-16T01:43:23.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.092+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/2672225738 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d82c0 msgr2=0x5600c691b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:23.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.092+0000 7f6f0e4d3740 1 --2- 172.21.15.88:0/2672225738 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d82c0 0x5600c691b1a0 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8009750 tx=0x7f6ef8013000 comp rx=0 tx=0).stop 2024-09-16T01:43:23.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.092+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/2672225738 shutdown_connections 2024-09-16T01:43:23.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.092+0000 7f6f0e4d3740 1 --2- 172.21.15.88:0/2672225738 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600c68d9310 0x5600c691c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.092+0000 7f6f0e4d3740 1 --2- 172.21.15.88:0/2672225738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600c68d9a50 0x5600c691b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.092+0000 7f6f0e4d3740 1 --2- 172.21.15.88:0/2672225738 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d82c0 0x5600c691b1a0 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.092+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/2672225738 >> 172.21.15.88:0/2672225738 conn(0x5600c68d5a80 msgr2=0x5600c6941130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:23.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.092+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/2672225738 shutdown_connections 2024-09-16T01:43:23.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.092+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/2672225738 wait complete. 2024-09-16T01:43:23.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.092+0000 7f6f0e4d3740 1 -- start start 2024-09-16T01:43:23.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.093+0000 7f6f0e4d3740 1 Processor -- start 2024-09-16T01:43:23.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.093+0000 7f6f0e4d3740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d82c0 0x5600c6981200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.093+0000 7f6f0e4d3740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600c68d9310 0x5600c691b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.093+0000 7f6f0e4d3740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600c68d9a50 0x5600c691b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.093+0000 7f6f0e4d3740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5600c683a520 con 0x5600c68d82c0 2024-09-16T01:43:23.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.093+0000 7f6f0e4d3740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5600c68d3f20 con 0x5600c68d9a50 2024-09-16T01:43:23.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.093+0000 7f6f0e4d3740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5600c68ffca0 con 0x5600c68d9310 2024-09-16T01:43:23.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.094+0000 7f6f0e4d3740 10 client.0 _ll_register_callbacks cb 0x7f6f00092070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:43:23.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.094+0000 7f6f094cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d82c0 0x5600c6981200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:23.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.094+0000 7f6f094cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d82c0 0x5600c6981200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43476/0 (socket says 172.21.15.88:43476) 2024-09-16T01:43:23.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.094+0000 7f6f094cd640 1 -- 172.21.15.88:0/966610726 learned_addr learned my addr 172.21.15.88:0/966610726 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:23.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.094+0000 7f6f08ccc640 1 --2- 172.21.15.88:0/966610726 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600c68d9310 0x5600c691b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:23.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.094+0000 7f6f09cce640 1 --2- 172.21.15.88:0/966610726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600c68d9a50 0x5600c691b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:23.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.094+0000 7f6f094cd640 1 -- 172.21.15.88:0/966610726 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600c68d9310 msgr2=0x5600c691b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:23.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.094+0000 7f6f094cd640 1 --2- 172.21.15.88:0/966610726 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600c68d9310 0x5600c691b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.094+0000 7f6f094cd640 1 -- 172.21.15.88:0/966610726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600c68d9a50 msgr2=0x5600c691b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:23.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.094+0000 7f6f094cd640 1 --2- 172.21.15.88:0/966610726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600c68d9a50 0x5600c691b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.094+0000 7f6f094cd640 1 -- 172.21.15.88:0/966610726 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600c68f88f0 con 0x5600c68d82c0 2024-09-16T01:43:23.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.094+0000 7f6f094cd640 1 --2- 172.21.15.88:0/966610726 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d82c0 0x5600c6981200 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8008010 tx=0x7f6ef80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4e78ff3c99d45e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:23.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.094+0000 7f6ee3fff640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:43:23.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.094+0000 7f6ee3fff640 1 -- 172.21.15.88:0/966610726 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef802f030 con 0x5600c68d82c0 2024-09-16T01:43:23.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.094+0000 7f6ee3fff640 1 -- 172.21.15.88:0/966610726 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8404+0+0 (secure 0 0 0) 0x7f6ef8029040 con 0x5600c68d82c0 2024-09-16T01:43:23.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.094+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/966610726 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ef800dd20 con 0x5600c68d82c0 2024-09-16T01:43:23.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.094+0000 7f6ee3fff640 1 -- 172.21.15.88:0/966610726 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=259) ==== 24+0+0 (secure 0 0 0) 0x7f6ef8014a40 con 0x5600c68d82c0 2024-09-16T01:43:23.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.094+0000 7f6f0e4d3740 10 client.9720 fetch_fsmap learned FSMap version 259 2024-09-16T01:43:23.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.094+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/966610726 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=259,monmap=2+}) -- 0x5600c6981740 con 0x5600c68d82c0 2024-09-16T01:43:23.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.094+0000 7f6ee3fff640 1 -- 172.21.15.88:0/966610726 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ef800dd20 con 0x5600c68d82c0 2024-09-16T01:43:23.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.095+0000 7f6ee3fff640 1 -- 172.21.15.88:0/966610726 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 259) ==== 3502+0+0 (secure 0 0 0) 0x7f6ef8013070 con 0x5600c68d82c0 2024-09-16T01:43:23.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.095+0000 7f6f0e4d3740 10 client.9720 fetch_fsmap finished waiting for FSMap version 259 2024-09-16T01:43:23.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.095+0000 7f6f0e4d3740 10 client.9720 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:43:23.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.095+0000 7f6f0e4d3740 20 client.9720 populate_metadata read hostname 'smithi088' 2024-09-16T01:43:23.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.095+0000 7f6f0e4d3740 1 --2- 172.21.15.88:0/966610726 >> [v2:172.21.15.88:6834/2555776364,v1:172.21.15.88:6836/2555776364] conn(0x5600c6982230 0x5600c68fce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.095+0000 7f6f0e4d3740 4 client.9720 mds_command: new command op to 9636 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:43:23.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.095+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/966610726 --> [v2:172.21.15.88:6834/2555776364,v1:172.21.15.88:6836/2555776364] -- command(tid 0: {"prefix": "status"}) -- 0x5600c6841fa0 con 0x5600c6982230 2024-09-16T01:43:23.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.095+0000 7f6f08ccc640 1 --2- 172.21.15.88:0/966610726 >> [v2:172.21.15.88:6834/2555776364,v1:172.21.15.88:6836/2555776364] conn(0x5600c6982230 0x5600c68fce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:23.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.095+0000 7f6f08ccc640 1 --2- 172.21.15.88:0/966610726 >> [v2:172.21.15.88:6834/2555776364,v1:172.21.15.88:6836/2555776364] conn(0x5600c6982230 0x5600c68fce00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17e3e3a1055b89b2 server_cookie=808d3f58f040e87c in_seq=0 out_seq=0 2024-09-16T01:43:23.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.095+0000 7f6ee3fff640 10 client.9720 ms_handle_connect on v2:172.21.15.88:6834/2555776364 2024-09-16T01:43:23.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.095+0000 7f6ee3fff640 1 -- 172.21.15.88:0/966610726 <== mds.0 v2:172.21.15.88:6834/2555776364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x5600c6841fa0 con 0x5600c6982230 2024-09-16T01:43:23.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.095+0000 7f6ee3fff640 10 client.9720 handle_command_reply: tid=0 2024-09-16T01:43:23.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.095+0000 7f6f0e4d3740 1 client.9720 shutdown 2024-09-16T01:43:23.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.095+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/966610726 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d82c0 msgr2=0x5600c6981200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:23.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.095+0000 7f6f0e4d3740 1 --2- 172.21.15.88:0/966610726 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d82c0 0x5600c6981200 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8008010 tx=0x7f6ef80046c0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.095+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/966610726 shutdown_connections 2024-09-16T01:43:23.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.095+0000 7f6f0e4d3740 1 --2- 172.21.15.88:0/966610726 >> [v2:172.21.15.88:6834/2555776364,v1:172.21.15.88:6836/2555776364] conn(0x5600c6982230 0x5600c68fce00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.095+0000 7f6f0e4d3740 1 --2- 172.21.15.88:0/966610726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600c68d9a50 0x5600c691b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.095+0000 7f6f0e4d3740 1 --2- 172.21.15.88:0/966610726 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5600c68d9310 0x5600c691b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.095+0000 7f6ee3fff640 0 client.9720 ms_handle_reset on v2:172.21.15.88:6834/2555776364 2024-09-16T01:43:23.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.095+0000 7f6f0e4d3740 1 --2- 172.21.15.88:0/966610726 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5600c68d82c0 0x5600c6981200 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.095+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/966610726 >> 172.21.15.88:0/966610726 conn(0x5600c68d5a80 msgr2=0x5600c691ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:23.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.095+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/966610726 shutdown_connections 2024-09-16T01:43:23.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.095+0000 7f6f0e4d3740 1 -- 172.21.15.88:0/966610726 wait complete. 2024-09-16T01:43:23.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.096+0000 7f6f0e4d3740 20 client.9720 trim_cache size 0 max 16384 2024-09-16T01:43:23.107 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:43:23.107 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "5ace2c91-46f9-482b-8a89-731fbdc171bf", 2024-09-16T01:43:23.107 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2024-09-16T01:43:23.107 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 9636, 2024-09-16T01:43:23.107 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2024-09-16T01:43:23.107 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2024-09-16T01:43:23.107 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2024-09-16T01:43:23.107 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 6.6153432219999999, 2024-09-16T01:43:23.107 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 259, 2024-09-16T01:43:23.107 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 118, 2024-09-16T01:43:23.107 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 118, 2024-09-16T01:43:23.108 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 10.041282225 2024-09-16T01:43:23.108 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:43:23.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.096+0000 7f6f0b731640 1 -- 172.21.15.88:0/604311181 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ee806c2b0 msgr2=0x7f6ee806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:23.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.096+0000 7f6f0b731640 1 --2- 172.21.15.88:0/604311181 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ee806c2b0 0x7f6ee806e770 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8009890 tx=0x7f6ef8008040 comp rx=0 tx=0).stop 2024-09-16T01:43:23.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.096+0000 7f6f0b731640 1 -- 172.21.15.88:0/604311181 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f040cf1f0 msgr2=0x7f6f0414b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:23.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.096+0000 7f6f0b731640 1 --2- 172.21.15.88:0/604311181 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f040cf1f0 0x7f6f0414b5b0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f6f00035040 tx=0x7f6f000046c0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.096+0000 7f6f0b731640 1 -- 172.21.15.88:0/604311181 shutdown_connections 2024-09-16T01:43:23.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.096+0000 7f6f0b731640 1 --2- 172.21.15.88:0/604311181 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6ee806c2b0 0x7f6ee806e770 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.096+0000 7f6f0b731640 1 --2- 172.21.15.88:0/604311181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f0415e7a0 0x7f6f040cdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.096+0000 7f6f0b731640 1 --2- 172.21.15.88:0/604311181 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f040cf1f0 0x7f6f0414b5b0 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.096+0000 7f6f0b731640 1 --2- 172.21.15.88:0/604311181 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f040b6720 0x7f6f040cad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.096+0000 7f6f0b731640 1 -- 172.21.15.88:0/604311181 >> 172.21.15.88:0/604311181 conn(0x7f6f040d11e0 msgr2=0x7f6f040b5b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:23.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.096+0000 7f6f0b731640 1 -- 172.21.15.88:0/604311181 shutdown_connections 2024-09-16T01:43:23.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.096+0000 7f6f0b731640 1 -- 172.21.15.88:0/604311181 wait complete. 2024-09-16T01:43:23.110 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-09-16T01:43:23.110 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Corrupt omap key 10000000000.00000000:sixmegs_head', proceeding to ls 2024-09-16T01:43:23.111 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:43:23.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.206+0000 7f6bb272c640 1 Processor -- start 2024-09-16T01:43:23.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.206+0000 7f6bb272c640 1 -- start start 2024-09-16T01:43:23.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.207+0000 7f6bb272c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6bac15c170 0x7f6bac15c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.207+0000 7f6bb272c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6bac0cf6c0 0x7f6bac0cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.207+0000 7f6bb272c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6bac0b5dd0 0x7f6bac0b6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.207+0000 7f6bb272c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6bac15ca90 con 0x7f6bac0b5dd0 2024-09-16T01:43:23.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.207+0000 7f6bb272c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6bac0b4cf0 con 0x7f6bac15c170 2024-09-16T01:43:23.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.207+0000 7f6bb272c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6bac0b4fd0 con 0x7f6bac0cf6c0 2024-09-16T01:43:23.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.207+0000 7f6bb0cc9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6bac0b5dd0 0x7f6bac0b6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:23.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.208+0000 7f6bb0cc9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6bac0b5dd0 0x7f6bac0b6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43482/0 (socket says 172.21.15.88:43482) 2024-09-16T01:43:23.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.208+0000 7f6bb0cc9640 1 -- 172.21.15.88:0/2734835909 learned_addr learned my addr 172.21.15.88:0/2734835909 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:23.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.208+0000 7f6bab7fe640 1 --2- 172.21.15.88:0/2734835909 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6bac0cf6c0 0x7f6bac0cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:23.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.208+0000 7f6babfff640 1 --2- 172.21.15.88:0/2734835909 >> v2:172.21.15.154:3300/0 conn(0x7f6bac15c170 0x7f6bac15c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:23.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.208+0000 7f6bb0cc9640 1 -- 172.21.15.88:0/2734835909 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6bac0cf6c0 msgr2=0x7f6bac0cfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:23.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.208+0000 7f6bb0cc9640 1 --2- 172.21.15.88:0/2734835909 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6bac0cf6c0 0x7f6bac0cfb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.208+0000 7f6bb0cc9640 1 -- 172.21.15.88:0/2734835909 >> v2:172.21.15.154:3300/0 conn(0x7f6bac15c170 msgr2=0x7f6bac15c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:23.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.208+0000 7f6bb0cc9640 1 --2- 172.21.15.88:0/2734835909 >> v2:172.21.15.154:3300/0 conn(0x7f6bac15c170 0x7f6bac15c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.208+0000 7f6bb0cc9640 1 -- 172.21.15.88:0/2734835909 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bac0b52b0 con 0x7f6bac0b5dd0 2024-09-16T01:43:23.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.208+0000 7f6babfff640 1 --2- 172.21.15.88:0/2734835909 >> v2:172.21.15.154:3300/0 conn(0x7f6bac15c170 0x7f6bac15c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:23.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.208+0000 7f6bb0cc9640 1 --2- 172.21.15.88:0/2734835909 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6bac0b5dd0 0x7f6bac0b6260 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f6bac06f340 tx=0x7f6ba402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73d91aee158b51df server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:23.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.209+0000 7f6baaffd640 1 -- 172.21.15.88:0/2734835909 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4036070 con 0x7f6bac0b5dd0 2024-09-16T01:43:23.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.209+0000 7f6baaffd640 1 -- 172.21.15.88:0/2734835909 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ba4031600 con 0x7f6bac0b5dd0 2024-09-16T01:43:23.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.209+0000 7f6baaffd640 1 -- 172.21.15.88:0/2734835909 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba40319d0 con 0x7f6bac0b5dd0 2024-09-16T01:43:23.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.209+0000 7f6bb272c640 1 -- 172.21.15.88:0/2734835909 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6bac0b5dd0 msgr2=0x7f6bac0b6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:23.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.209+0000 7f6bb272c640 1 --2- 172.21.15.88:0/2734835909 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6bac0b5dd0 0x7f6bac0b6260 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f6bac06f340 tx=0x7f6ba402a6a0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.209+0000 7f6bb272c640 1 -- 172.21.15.88:0/2734835909 shutdown_connections 2024-09-16T01:43:23.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.209+0000 7f6bb272c640 1 --2- 172.21.15.88:0/2734835909 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6bac0b5dd0 0x7f6bac0b6260 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.209+0000 7f6bb272c640 1 --2- 172.21.15.88:0/2734835909 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6bac0cf6c0 0x7f6bac0cfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.209+0000 7f6bb272c640 1 --2- 172.21.15.88:0/2734835909 >> v2:172.21.15.154:3300/0 conn(0x7f6bac15c170 0x7f6bac15c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.209+0000 7f6bb272c640 1 -- 172.21.15.88:0/2734835909 >> 172.21.15.88:0/2734835909 conn(0x7f6bac06bc10 msgr2=0x7f6bac06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:23.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.209+0000 7f6bb272c640 1 -- 172.21.15.88:0/2734835909 shutdown_connections 2024-09-16T01:43:23.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.209+0000 7f6bb272c640 1 -- 172.21.15.88:0/2734835909 wait complete. 2024-09-16T01:43:23.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.210+0000 7f6bb272c640 1 Processor -- start 2024-09-16T01:43:23.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.211+0000 7f6bb272c640 1 -- start start 2024-09-16T01:43:23.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.211+0000 7f6bb272c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6bac0b5dd0 0x7f6bac14a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.211+0000 7f6bb272c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bac0cf6c0 0x7f6bac0cbc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.211+0000 7f6bb272c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6bac15c170 0x7f6bac0cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.211+0000 7f6bb272c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6bac0b5b50 con 0x7f6bac15c170 2024-09-16T01:43:23.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.211+0000 7f6bb272c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6bac0b5870 con 0x7f6bac0cf6c0 2024-09-16T01:43:23.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.211+0000 7f6bb272c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6bac15ffb0 con 0x7f6bac0b5dd0 2024-09-16T01:43:23.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.211+0000 7f6bb0cc9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6bac15c170 0x7f6bac0cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:23.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.211+0000 7f6bb0cc9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6bac15c170 0x7f6bac0cc1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43484/0 (socket says 172.21.15.88:43484) 2024-09-16T01:43:23.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.211+0000 7f6bb0cc9640 1 -- 172.21.15.88:0/3991813531 learned_addr learned my addr 172.21.15.88:0/3991813531 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:23.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.212+0000 7f6babfff640 1 --2- 172.21.15.88:0/3991813531 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6bac0b5dd0 0x7f6bac14a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:23.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.212+0000 7f6bab7fe640 1 --2- 172.21.15.88:0/3991813531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bac0cf6c0 0x7f6bac0cbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:23.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.212+0000 7f6bb0cc9640 1 -- 172.21.15.88:0/3991813531 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6bac0b5dd0 msgr2=0x7f6bac14a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:23.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.212+0000 7f6bb0cc9640 1 --2- 172.21.15.88:0/3991813531 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6bac0b5dd0 0x7f6bac14a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.212+0000 7f6bb0cc9640 1 -- 172.21.15.88:0/3991813531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bac0cf6c0 msgr2=0x7f6bac0cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:23.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.212+0000 7f6bb0cc9640 1 --2- 172.21.15.88:0/3991813531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bac0cf6c0 0x7f6bac0cbc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.212+0000 7f6bb0cc9640 1 -- 172.21.15.88:0/3991813531 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bac1f5470 con 0x7f6bac15c170 2024-09-16T01:43:23.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.212+0000 7f6bb0cc9640 1 --2- 172.21.15.88:0/3991813531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6bac15c170 0x7f6bac0cc1a0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f6ba402ac90 tx=0x7f6ba4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:23.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.212+0000 7f6ba8ff9640 1 -- 172.21.15.88:0/3991813531 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba403f020 con 0x7f6bac15c170 2024-09-16T01:43:23.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.212+0000 7f6ba8ff9640 1 -- 172.21.15.88:0/3991813531 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ba4037040 con 0x7f6bac15c170 2024-09-16T01:43:23.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.212+0000 7f6bb272c640 1 -- 172.21.15.88:0/3991813531 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bac1f46b0 con 0x7f6bac15c170 2024-09-16T01:43:23.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.212+0000 7f6ba8ff9640 1 -- 172.21.15.88:0/3991813531 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4036070 con 0x7f6bac15c170 2024-09-16T01:43:23.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.212+0000 7f6bb272c640 1 -- 172.21.15.88:0/3991813531 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bac1f4c70 con 0x7f6bac15c170 2024-09-16T01:43:23.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.213+0000 7f6bb272c640 1 -- 172.21.15.88:0/3991813531 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bac15c550 con 0x7f6bac15c170 2024-09-16T01:43:23.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.214+0000 7f6ba8ff9640 1 -- 172.21.15.88:0/3991813531 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6ba4042020 con 0x7f6bac15c170 2024-09-16T01:43:23.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.214+0000 7f6ba8ff9640 1 --2- 172.21.15.88:0/3991813531 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6b9806c2e0 0x7f6b9806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:23.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.214+0000 7f6babfff640 1 --2- 172.21.15.88:0/3991813531 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6b9806c2e0 0x7f6b9806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:23.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.214+0000 7f6ba8ff9640 1 -- 172.21.15.88:0/3991813531 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8404+0+0 (secure 0 0 0) 0x7f6ba40ac030 con 0x7f6bac15c170 2024-09-16T01:43:23.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.215+0000 7f6babfff640 1 --2- 172.21.15.88:0/3991813531 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6b9806c2e0 0x7f6b9806e7a0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f6bac15f7a0 tx=0x7f6b9c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:23.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.219+0000 7f6ba8ff9640 1 -- 172.21.15.88:0/3991813531 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6ba4049030 con 0x7f6bac15c170 2024-09-16T01:43:23.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.419+0000 7f6bb272c640 1 -- 172.21.15.88:0/3991813531 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f6bac0b38e0 con 0x7f6bac15c170 2024-09-16T01:43:23.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.419+0000 7f6ba8ff9640 1 -- 172.21.15.88:0/3991813531 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v259) ==== 53+0+73 (secure 0 0 0) 0x7f6ba4079700 con 0x7f6bac15c170 2024-09-16T01:43:23.421 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:43:23.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.421+0000 7f6bb272c640 1 -- 172.21.15.88:0/3991813531 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6b9806c2e0 msgr2=0x7f6b9806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:23.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.421+0000 7f6bb272c640 1 --2- 172.21.15.88:0/3991813531 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6b9806c2e0 0x7f6b9806e7a0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f6bac15f7a0 tx=0x7f6b9c008040 comp rx=0 tx=0).stop 2024-09-16T01:43:23.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.421+0000 7f6bb272c640 1 -- 172.21.15.88:0/3991813531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6bac15c170 msgr2=0x7f6bac0cc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:23.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.421+0000 7f6bb272c640 1 --2- 172.21.15.88:0/3991813531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6bac15c170 0x7f6bac0cc1a0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f6ba402ac90 tx=0x7f6ba4004a40 comp rx=0 tx=0).stop 2024-09-16T01:43:23.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.421+0000 7f6bb272c640 1 -- 172.21.15.88:0/3991813531 shutdown_connections 2024-09-16T01:43:23.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.421+0000 7f6bb272c640 1 --2- 172.21.15.88:0/3991813531 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6b9806c2e0 0x7f6b9806e7a0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.421+0000 7f6bb272c640 1 --2- 172.21.15.88:0/3991813531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6bac15c170 0x7f6bac0cc1a0 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.421+0000 7f6bb272c640 1 --2- 172.21.15.88:0/3991813531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bac0cf6c0 0x7f6bac0cbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.421+0000 7f6bb272c640 1 --2- 172.21.15.88:0/3991813531 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6bac0b5dd0 0x7f6bac14a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:23.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.421+0000 7f6bb272c640 1 -- 172.21.15.88:0/3991813531 >> 172.21.15.88:0/3991813531 conn(0x7f6bac06bc10 msgr2=0x7f6bac0724f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:23.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.421+0000 7f6bb272c640 1 -- 172.21.15.88:0/3991813531 shutdown_connections 2024-09-16T01:43:23.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23.421+0000 7f6bb272c640 1 -- 172.21.15.88:0/3991813531 wait complete. 2024-09-16T01:43:23.433 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-16T01:43:23.433 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:43:23.433 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2024-09-16T01:43:23.433 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T01:43:23.434 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T01:43:23.434 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-16T01:43:23.434 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-16T01:43:23.434 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:43:23.434 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:43:23.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:23.434 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2024-09-16T01:43:23.450 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:43:23.450 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:43:23.450 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:43:23.450 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:43:23.450 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-09-16T01:43:23.450 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:43:23.450 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:43:23.450 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:43:23.451 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2024-09-16T01:43:23.451 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:43:23.451 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:43:23.451 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2024-09-16T01:43:23.451 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:43:23.451 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2024-09-16T01:43:23.451 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:43:23.451 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2024-09-16T01:43:23.451 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:43:23.451 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42221sec preferred_lft 42221sec 2024-09-16T01:43:23.451 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link noprefixroute 2024-09-16T01:43:23.451 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:43:23.451 INFO:teuthology.orchestra.run.smithi088.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:43:23.451 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff 2024-09-16T01:43:23.452 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T01:43:23.452 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:43:23.452 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::e486:6eff:fe81:a1c/64 scope link 2024-09-16T01:43:23.452 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:43:23.452 INFO:teuthology.orchestra.run.smithi088.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:43:23.452 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 12:6f:29:be:95:48 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:43:23.452 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::106f:29ff:febe:9548/64 scope link 2024-09-16T01:43:23.452 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:43:23.452 INFO:teuthology.orchestra.run.smithi088.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:43:23.452 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T01:43:23.452 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::c63:b4ff:feb3:ad9e/64 scope link 2024-09-16T01:43:23.452 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:43:23.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:23.453 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:43:23.459 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:43:23.458+0000 7fa851083640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:43:23.482 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-09-16T01:43:23.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:23.484 DEBUG:teuthology.orchestra.run.smithi088:> route 2024-09-16T01:43:23.540 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2024-09-16T01:43:23.540 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:43:23.540 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:43:23.540 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:43:23.540 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:43:23.541 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:43:23.541 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:43:23.541 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:43:23.541 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:43:23.541 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:43:23.541 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:43:23.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:43:23.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:43:23.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:23.666 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:43:23.681 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T01:43:23.681 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T01:43:23.682 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:43:23.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:23.682 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2024-09-16T01:43:23.747 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:43:23.747 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T01:43:23.802 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:43:23.803 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:43:23.803 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:43:23.858 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-16T01:43:23.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:23.859 DEBUG:teuthology.orchestra.run.smithi088:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:43:23.942 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:43:23.942 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:43:23.942 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:43:23.951 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:43:23.951 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:43:23.951 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:43:23.951 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:43:23.951 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:43:23.956 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:43:23.956 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:43:23.957 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:43:23.957 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:43:23.957 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:43:23.957 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790" 2024-09-16T01:43:23.957 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2024-09-16T01:43:23.957 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5ace2c91-46f9-482b-8a89-731fbdc171bf" will pass to kernel 2024-09-16T01:43:23.957 INFO:teuthology.orchestra.run.smithi088.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-16T01:43:23.957 INFO:teuthology.orchestra.run.smithi088.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:43:23.957 INFO:teuthology.orchestra.run.smithi088.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:43:23.958 INFO:teuthology.orchestra.run.smithi088.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:43:23.958 INFO:teuthology.orchestra.run.smithi088.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:43:23.959 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:43:23.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:23.959 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:43:24.025 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:43:24.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:43:24.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:43:24.120 INFO:teuthology.orchestra.run.smithi088.stdout:client7957 2024-09-16T01:43:24.121 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:43:24.121 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:43:24.137 DEBUG:teuthology.orchestra.run.smithi088:> stdin-killer -- python3 -c ' 2024-09-16T01:43:24.137 DEBUG:teuthology.orchestra.run.smithi088:> import os 2024-09-16T01:43:24.137 DEBUG:teuthology.orchestra.run.smithi088:> import stat 2024-09-16T01:43:24.137 DEBUG:teuthology.orchestra.run.smithi088:> import json 2024-09-16T01:43:24.138 DEBUG:teuthology.orchestra.run.smithi088:> import sys 2024-09-16T01:43:24.138 DEBUG:teuthology.orchestra.run.smithi088:> 2024-09-16T01:43:24.138 DEBUG:teuthology.orchestra.run.smithi088:> try: 2024-09-16T01:43:24.138 DEBUG:teuthology.orchestra.run.smithi088:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/./subdir/sixmegs'"'"') 2024-09-16T01:43:24.138 DEBUG:teuthology.orchestra.run.smithi088:> except OSError as e: 2024-09-16T01:43:24.138 DEBUG:teuthology.orchestra.run.smithi088:> sys.exit(e.errno) 2024-09-16T01:43:24.138 DEBUG:teuthology.orchestra.run.smithi088:> 2024-09-16T01:43:24.138 DEBUG:teuthology.orchestra.run.smithi088:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-09-16T01:43:24.138 DEBUG:teuthology.orchestra.run.smithi088:> print(json.dumps( 2024-09-16T01:43:24.138 DEBUG:teuthology.orchestra.run.smithi088:> dict([(a, getattr(s, a)) for a in attrs]), 2024-09-16T01:43:24.138 DEBUG:teuthology.orchestra.run.smithi088:> indent=2)) 2024-09-16T01:43:24.138 DEBUG:teuthology.orchestra.run.smithi088:> ' 2024-09-16T01:43:24.179 INFO:teuthology.orchestra.run:waiting for 20 2024-09-16T01:43:24.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24 stdin-killer INFO: command exited with status 5: exiting normally with same code! 2024-09-16T01:43:24.254 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-09-16T01:43:24.254 INFO:tasks.cephfs.test_damage:Result: EIO on client 2024-09-16T01:43:24.254 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:43:24.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.391+0000 7f099bfa8640 1 Processor -- start 2024-09-16T01:43:24.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.391+0000 7f099bfa8640 1 -- start start 2024-09-16T01:43:24.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.392+0000 7f099bfa8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f09940b6140 0x7f099415c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.392+0000 7f099bfa8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09940b50a0 con 0x7f0994155800 2024-09-16T01:43:24.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.392+0000 7f099bfa8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09940b5380 con 0x7f09940cfc80 2024-09-16T01:43:24.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.392+0000 7f099bfa8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f09940b6550 con 0x7f09940b6140 2024-09-16T01:43:24.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.392+0000 7f0999d44640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0994155800 0x7f09940cf570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36302/0 (socket says 172.21.15.88:36302) 2024-09-16T01:43:24.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.392+0000 7f0999d44640 1 -- 172.21.15.88:0/1502896228 learned_addr learned my addr 172.21.15.88:0/1502896228 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:24.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.393+0000 7f0998d42640 1 -- 172.21.15.88:0/1502896228 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3949515004 0 0) 0x7f09940b50a0 con 0x7f0994155800 2024-09-16T01:43:24.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.393+0000 7f099a545640 1 --2- 172.21.15.88:0/1502896228 >> v2:172.21.15.154:3300/0 conn(0x7f09940b6140 0x7f099415c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.393+0000 7f0998d42640 1 -- 172.21.15.88:0/1502896228 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0980003520 con 0x7f0994155800 2024-09-16T01:43:24.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.393+0000 7f0998d42640 1 -- 172.21.15.88:0/1502896228 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1662277068 0 0) 0x7f0980003520 con 0x7f0994155800 2024-09-16T01:43:24.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.393+0000 7f0998d42640 1 -- 172.21.15.88:0/1502896228 >> v2:172.21.15.154:3300/0 conn(0x7f09940b6140 msgr2=0x7f099415c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:24.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.393+0000 7f0998d42640 1 --2- 172.21.15.88:0/1502896228 >> v2:172.21.15.154:3300/0 conn(0x7f09940b6140 0x7f099415c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.393+0000 7f0998d42640 1 -- 172.21.15.88:0/1502896228 >> v1:172.21.15.154:6789/0 conn(0x7f09940cfc80 legacy=0x7f099415c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:24.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.393+0000 7f0998d42640 1 -- 172.21.15.88:0/1502896228 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09940b5660 con 0x7f0994155800 2024-09-16T01:43:24.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.393+0000 7f0998d42640 1 -- 172.21.15.88:0/1502896228 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f0990001670 con 0x7f0994155800 2024-09-16T01:43:24.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.393+0000 7f0998d42640 1 -- 172.21.15.88:0/1502896228 >> v1:172.21.15.88:6789/0 conn(0x7f0994155800 legacy=0x7f09940cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:24.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.393+0000 7f0998d42640 1 --2- 172.21.15.88:0/1502896228 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09800039b0 0x7f0980003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.393+0000 7f0998d42640 1 --2- 172.21.15.88:0/1502896228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0980004800 0x7f0980004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.393+0000 7f0998d42640 1 --2- 172.21.15.88:0/1502896228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09800051c0 0x7f098000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.393+0000 7f0998d42640 1 -- 172.21.15.88:0/1502896228 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f09940b6550 con 0x7f09800051c0 2024-09-16T01:43:24.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.393+0000 7f0998d42640 1 -- 172.21.15.88:0/1502896228 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f098000c060 con 0x7f0980004800 2024-09-16T01:43:24.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.393+0000 7f0998d42640 1 -- 172.21.15.88:0/1502896228 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f098000c260 con 0x7f09800039b0 2024-09-16T01:43:24.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.393+0000 7f099a545640 1 --2- 172.21.15.88:0/1502896228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09800051c0 0x7f098000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.393+0000 7f0999d44640 1 --2- 172.21.15.88:0/1502896228 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09800039b0 0x7f0980003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.393+0000 7f0999543640 1 --2- 172.21.15.88:0/1502896228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0980004800 0x7f0980004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.393+0000 7f099a545640 1 -- 172.21.15.88:0/1502896228 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09800039b0 msgr2=0x7f0980003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:24.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.393+0000 7f099a545640 1 --2- 172.21.15.88:0/1502896228 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09800039b0 0x7f0980003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.393+0000 7f099a545640 1 -- 172.21.15.88:0/1502896228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0980004800 msgr2=0x7f0980004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:24.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.393+0000 7f099a545640 1 --2- 172.21.15.88:0/1502896228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0980004800 0x7f0980004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.393+0000 7f099a545640 1 -- 172.21.15.88:0/1502896228 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0980003520 con 0x7f09800051c0 2024-09-16T01:43:24.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.394+0000 7f099a545640 1 -- 172.21.15.88:0/1502896228 reap_dead start 2024-09-16T01:43:24.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.394+0000 7f099a545640 1 --2- 172.21.15.88:0/1502896228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09800051c0 0x7f098000b9c0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f098c009fc0 tx=0x7f0980005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b19cc5299037fddf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:24.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.394+0000 7f0998d42640 1 -- 172.21.15.88:0/1502896228 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f098c035020 con 0x7f09800051c0 2024-09-16T01:43:24.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.394+0000 7f0998d42640 1 -- 172.21.15.88:0/1502896228 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f098c003bd0 con 0x7f09800051c0 2024-09-16T01:43:24.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.394+0000 7f0998d42640 1 -- 172.21.15.88:0/1502896228 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f098c009c10 con 0x7f09800051c0 2024-09-16T01:43:24.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.395+0000 7f099bfa8640 1 -- 172.21.15.88:0/1502896228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09800051c0 msgr2=0x7f098000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:24.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.395+0000 7f099bfa8640 1 --2- 172.21.15.88:0/1502896228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09800051c0 0x7f098000b9c0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f098c009fc0 tx=0x7f0980005830 comp rx=0 tx=0).stop 2024-09-16T01:43:24.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.395+0000 7f099bfa8640 1 -- 172.21.15.88:0/1502896228 shutdown_connections 2024-09-16T01:43:24.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.395+0000 7f099bfa8640 1 --2- 172.21.15.88:0/1502896228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09800051c0 0x7f098000b9c0 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.395+0000 7f099bfa8640 1 -- 172.21.15.88:0/1502896228 >> 172.21.15.88:0/1502896228 conn(0x7f099406bc10 msgr2=0x7f099406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:24.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.395+0000 7f099bfa8640 1 -- 172.21.15.88:0/1502896228 shutdown_connections 2024-09-16T01:43:24.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.395+0000 7f099bfa8640 1 -- 172.21.15.88:0/1502896228 wait complete. 2024-09-16T01:43:24.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.396+0000 7f099bfa8640 1 Processor -- start 2024-09-16T01:43:24.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.396+0000 7f099bfa8640 1 -- start start 2024-09-16T01:43:24.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.397+0000 7f099bfa8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09800051c0 0x7f09941f2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.397+0000 7f099bfa8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f099414bc40 0x7f099415ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.397+0000 7f099bfa8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f099414c0c0 0x7f0994072510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.397+0000 7f099bfa8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0994155be0 con 0x7f099414c0c0 2024-09-16T01:43:24.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.397+0000 7f099bfa8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f09940b5c20 con 0x7f099414bc40 2024-09-16T01:43:24.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.397+0000 7f099bfa8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f099415c330 con 0x7f09800051c0 2024-09-16T01:43:24.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.397+0000 7f099a545640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f099414c0c0 0x7f0994072510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.397+0000 7f099a545640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f099414c0c0 0x7f0994072510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43500/0 (socket says 172.21.15.88:43500) 2024-09-16T01:43:24.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.397+0000 7f099a545640 1 -- 172.21.15.88:0/1814394795 learned_addr learned my addr 172.21.15.88:0/1814394795 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:24.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.397+0000 7f0999d44640 1 --2- 172.21.15.88:0/1814394795 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09800051c0 0x7f09941f2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.397+0000 7f0999543640 1 --2- 172.21.15.88:0/1814394795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f099414bc40 0x7f099415ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.397+0000 7f099a545640 1 -- 172.21.15.88:0/1814394795 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09800051c0 msgr2=0x7f09941f2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:24.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.397+0000 7f099a545640 1 --2- 172.21.15.88:0/1814394795 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09800051c0 0x7f09941f2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.397+0000 7f099a545640 1 -- 172.21.15.88:0/1814394795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f099414bc40 msgr2=0x7f099415ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:24.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.397+0000 7f099a545640 1 --2- 172.21.15.88:0/1814394795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f099414bc40 0x7f099415ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.397+0000 7f099a545640 1 -- 172.21.15.88:0/1814394795 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f099414ee80 con 0x7f099414c0c0 2024-09-16T01:43:24.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.397+0000 7f099a545640 1 --2- 172.21.15.88:0/1814394795 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f099414c0c0 0x7f0994072510 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f098c00e230 tx=0x7f098c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:24.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.398+0000 7f098a7fc640 1 -- 172.21.15.88:0/1814394795 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f098c040070 con 0x7f099414c0c0 2024-09-16T01:43:24.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.398+0000 7f098a7fc640 1 -- 172.21.15.88:0/1814394795 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f098c032e80 con 0x7f099414c0c0 2024-09-16T01:43:24.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.398+0000 7f098a7fc640 1 -- 172.21.15.88:0/1814394795 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f098c035030 con 0x7f099414c0c0 2024-09-16T01:43:24.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.398+0000 7f099bfa8640 1 -- 172.21.15.88:0/1814394795 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f099414f160 con 0x7f099414c0c0 2024-09-16T01:43:24.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.398+0000 7f099bfa8640 1 -- 172.21.15.88:0/1814394795 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f099414f720 con 0x7f099414c0c0 2024-09-16T01:43:24.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.399+0000 7f099bfa8640 1 -- 172.21.15.88:0/1814394795 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f099415b7e0 con 0x7f099414c0c0 2024-09-16T01:43:24.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.399+0000 7f098a7fc640 1 -- 172.21.15.88:0/1814394795 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f098c043070 con 0x7f099414c0c0 2024-09-16T01:43:24.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.399+0000 7f098a7fc640 1 --2- 172.21.15.88:0/1814394795 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f097806c2b0 0x7f097806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.400+0000 7f098a7fc640 1 -- 172.21.15.88:0/1814394795 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8404+0+0 (secure 0 0 0) 0x7f098c0acf50 con 0x7f099414c0c0 2024-09-16T01:43:24.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.400+0000 7f0999d44640 1 --2- 172.21.15.88:0/1814394795 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f097806c2b0 0x7f097806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.400+0000 7f0999d44640 1 --2- 172.21.15.88:0/1814394795 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f097806c2b0 0x7f097806e770 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f0990005b10 tx=0x7f0990007040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:24.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.404+0000 7f098a7fc640 1 -- 172.21.15.88:0/1814394795 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f099415b7e0 con 0x7f099414c0c0 2024-09-16T01:43:24.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.604+0000 7f099bfa8640 1 -- 172.21.15.88:0/1814394795 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09941e5ea0 con 0x7f099414c0c0 2024-09-16T01:43:24.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.605+0000 7f098a7fc640 1 -- 172.21.15.88:0/1814394795 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 259 v259) ==== 95+0+5234 (secure 0 0 0) 0x7f098c07a590 con 0x7f099414c0c0 2024-09-16T01:43:24.606 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:24.606 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":259,"btime":"2024-09-16T01:43:19:502492+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9696,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4270197461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4270197461},{"type":"v1","addr":"172.21.15.88:6837","nonce":4270197461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":259,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:43:19.502487+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":118,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9636},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9636":{"gid":9636,"name":"a","rank":0,"incarnation":256,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6836/2555776364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2555776364},{"type":"v1","addr":"172.21.15.88:6836","nonce":2555776364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9636,"qdb_cluster":[9636]},"id":6}]} 2024-09-16T01:43:24.607 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 259 2024-09-16T01:43:24.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.606+0000 7f099bfa8640 1 -- 172.21.15.88:0/1814394795 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f097806c2b0 msgr2=0x7f097806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:24.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.606+0000 7f099bfa8640 1 --2- 172.21.15.88:0/1814394795 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f097806c2b0 0x7f097806e770 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f0990005b10 tx=0x7f0990007040 comp rx=0 tx=0).stop 2024-09-16T01:43:24.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.606+0000 7f099bfa8640 1 -- 172.21.15.88:0/1814394795 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f099414c0c0 msgr2=0x7f0994072510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:24.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.606+0000 7f099bfa8640 1 --2- 172.21.15.88:0/1814394795 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f099414c0c0 0x7f0994072510 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f098c00e230 tx=0x7f098c009750 comp rx=0 tx=0).stop 2024-09-16T01:43:24.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.607+0000 7f099bfa8640 1 -- 172.21.15.88:0/1814394795 shutdown_connections 2024-09-16T01:43:24.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.607+0000 7f099bfa8640 1 --2- 172.21.15.88:0/1814394795 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f097806c2b0 0x7f097806e770 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.607+0000 7f099bfa8640 1 --2- 172.21.15.88:0/1814394795 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f099414c0c0 0x7f0994072510 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.607+0000 7f099bfa8640 1 --2- 172.21.15.88:0/1814394795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f099414bc40 0x7f099415ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.607+0000 7f099bfa8640 1 --2- 172.21.15.88:0/1814394795 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09800051c0 0x7f09941f2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.607+0000 7f099bfa8640 1 -- 172.21.15.88:0/1814394795 >> 172.21.15.88:0/1814394795 conn(0x7f099406bc10 msgr2=0x7f09940b45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:24.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.607+0000 7f099bfa8640 1 -- 172.21.15.88:0/1814394795 shutdown_connections 2024-09-16T01:43:24.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.607+0000 7f099bfa8640 1 -- 172.21.15.88:0/1814394795 wait complete. 2024-09-16T01:43:24.619 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage ls --format=json-pretty 2024-09-16T01:43:24.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.718+0000 7fb1f545c640 1 Processor -- start 2024-09-16T01:43:24.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.718+0000 7fb1f545c640 1 -- start start 2024-09-16T01:43:24.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.719+0000 7fb1f545c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb1f00cf1f0 0x7fb1f00b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.719+0000 7fb1f545c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb1f00b6720 0x7fb1f00d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.719+0000 7fb1f545c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1f00cf860 con 0x7fb1f015e7a0 2024-09-16T01:43:24.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.719+0000 7fb1f545c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb1f00b6b90 con 0x7fb1f00b6720 2024-09-16T01:43:24.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.719+0000 7fb1f545c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb1f00d2ca0 con 0x7fb1f00cf1f0 2024-09-16T01:43:24.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.719+0000 7fb1ef7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb1f00b6720 0x7fb1f00d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.719+0000 7fb1ef7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb1f00b6720 0x7fb1f00d3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43502/0 (socket says 172.21.15.88:43502) 2024-09-16T01:43:24.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.719+0000 7fb1ef7fe640 1 -- 172.21.15.88:0/893390787 learned_addr learned my addr 172.21.15.88:0/893390787 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:24.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.719+0000 7fb1e7fff640 1 --2- 172.21.15.88:0/893390787 >> v2:172.21.15.154:3300/0 conn(0x7fb1f00cf1f0 0x7fb1f00b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.719+0000 7fb1ef7fe640 1 -- 172.21.15.88:0/893390787 >> v2:172.21.15.154:3300/0 conn(0x7fb1f00cf1f0 msgr2=0x7fb1f00b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:24.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.719+0000 7fb1ef7fe640 1 --2- 172.21.15.88:0/893390787 >> v2:172.21.15.154:3300/0 conn(0x7fb1f00cf1f0 0x7fb1f00b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.719+0000 7fb1ef7fe640 1 -- 172.21.15.88:0/893390787 >> v1:172.21.15.154:6789/0 conn(0x7fb1f015e7a0 legacy=0x7fb1f006caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:24.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.719+0000 7fb1ef7fe640 1 -- 172.21.15.88:0/893390787 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1f00cfb40 con 0x7fb1f00b6720 2024-09-16T01:43:24.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.719+0000 7fb1e7fff640 1 --2- 172.21.15.88:0/893390787 >> v2:172.21.15.154:3300/0 conn(0x7fb1f00cf1f0 0x7fb1f00b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:24.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.720+0000 7fb1ef7fe640 1 --2- 172.21.15.88:0/893390787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1f00b6720 0x7fb1f00d3410 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fb1f00d2110 tx=0x7fb1e802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f5119316d56f4c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:24.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.720+0000 7fb1ee7fc640 1 -- 172.21.15.88:0/893390787 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8036070 con 0x7fb1f00b6720 2024-09-16T01:43:24.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.720+0000 7fb1ee7fc640 1 -- 172.21.15.88:0/893390787 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1e8031600 con 0x7fb1f00b6720 2024-09-16T01:43:24.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.720+0000 7fb1ee7fc640 1 -- 172.21.15.88:0/893390787 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e80319d0 con 0x7fb1f00b6720 2024-09-16T01:43:24.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.721+0000 7fb1f545c640 1 -- 172.21.15.88:0/893390787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1f00b6720 msgr2=0x7fb1f00d3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:24.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.721+0000 7fb1f545c640 1 --2- 172.21.15.88:0/893390787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1f00b6720 0x7fb1f00d3410 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fb1f00d2110 tx=0x7fb1e802a660 comp rx=0 tx=0).stop 2024-09-16T01:43:24.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.721+0000 7fb1f545c640 1 -- 172.21.15.88:0/893390787 shutdown_connections 2024-09-16T01:43:24.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.721+0000 7fb1f545c640 1 --2- 172.21.15.88:0/893390787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1f00b6720 0x7fb1f00d3410 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.721+0000 7fb1f545c640 1 --2- 172.21.15.88:0/893390787 >> v2:172.21.15.154:3300/0 conn(0x7fb1f00cf1f0 0x7fb1f00b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.721+0000 7fb1f545c640 1 -- 172.21.15.88:0/893390787 >> 172.21.15.88:0/893390787 conn(0x7fb1f00d11e0 msgr2=0x7fb1f00d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:24.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.721+0000 7fb1f545c640 1 -- 172.21.15.88:0/893390787 shutdown_connections 2024-09-16T01:43:24.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.721+0000 7fb1f545c640 1 -- 172.21.15.88:0/893390787 wait complete. 2024-09-16T01:43:24.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.722+0000 7fb1f545c640 1 Processor -- start 2024-09-16T01:43:24.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.722+0000 7fb1f545c640 1 -- start start 2024-09-16T01:43:24.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.723+0000 7fb1f545c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1f00b6720 0x7fb1f00cae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.723+0000 7fb1f545c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1f00cf1f0 0x7fb1f01458e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.723+0000 7fb1f545c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1f015e7a0 0x7fb1f00ce250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.723+0000 7fb1f545c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f00b7a00 con 0x7fb1f00b6720 2024-09-16T01:43:24.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.723+0000 7fb1f545c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f00d2ef0 con 0x7fb1f00cf1f0 2024-09-16T01:43:24.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.723+0000 7fb1f545c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb1f015eb80 con 0x7fb1f015e7a0 2024-09-16T01:43:24.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.723+0000 7fb1eeffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1f00b6720 0x7fb1f00cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.723+0000 7fb1eeffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1f00b6720 0x7fb1f00cae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43514/0 (socket says 172.21.15.88:43514) 2024-09-16T01:43:24.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.723+0000 7fb1eeffd640 1 -- 172.21.15.88:0/3509718672 learned_addr learned my addr 172.21.15.88:0/3509718672 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:24.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.723+0000 7fb1e7fff640 1 --2- 172.21.15.88:0/3509718672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1f00cf1f0 0x7fb1f01458e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.723+0000 7fb1ef7fe640 1 --2- 172.21.15.88:0/3509718672 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1f015e7a0 0x7fb1f00ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.723+0000 7fb1eeffd640 1 -- 172.21.15.88:0/3509718672 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1f015e7a0 msgr2=0x7fb1f00ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:24.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.723+0000 7fb1eeffd640 1 --2- 172.21.15.88:0/3509718672 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1f015e7a0 0x7fb1f00ce250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.723+0000 7fb1eeffd640 1 -- 172.21.15.88:0/3509718672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1f00cf1f0 msgr2=0x7fb1f01458e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:24.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.723+0000 7fb1eeffd640 1 --2- 172.21.15.88:0/3509718672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1f00cf1f0 0x7fb1f01458e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.723+0000 7fb1eeffd640 1 -- 172.21.15.88:0/3509718672 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1f00d3310 con 0x7fb1f00b6720 2024-09-16T01:43:24.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.723+0000 7fb1eeffd640 1 --2- 172.21.15.88:0/3509718672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1f00b6720 0x7fb1f00cae20 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7fb1e000c460 tx=0x7fb1e000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:24.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.723+0000 7fb1e77fe640 1 -- 172.21.15.88:0/3509718672 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0016020 con 0x7fb1f00b6720 2024-09-16T01:43:24.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.723+0000 7fb1e77fe640 1 -- 172.21.15.88:0/3509718672 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1e0012870 con 0x7fb1f00b6720 2024-09-16T01:43:24.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.723+0000 7fb1f545c640 1 -- 172.21.15.88:0/3509718672 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1f00d35b0 con 0x7fb1f00b6720 2024-09-16T01:43:24.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.723+0000 7fb1e77fe640 1 -- 172.21.15.88:0/3509718672 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0012c60 con 0x7fb1f00b6720 2024-09-16T01:43:24.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.723+0000 7fb1f545c640 1 -- 172.21.15.88:0/3509718672 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1f00d3b70 con 0x7fb1f00b6720 2024-09-16T01:43:24.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.725+0000 7fb1e77fe640 1 -- 172.21.15.88:0/3509718672 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb1e0019020 con 0x7fb1f00b6720 2024-09-16T01:43:24.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.725+0000 7fb1e77fe640 1 --2- 172.21.15.88:0/3509718672 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb1d006c2b0 0x7fb1d006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.725+0000 7fb1e77fe640 1 -- 172.21.15.88:0/3509718672 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8404+0+0 (secure 0 0 0) 0x7fb1e008bca0 con 0x7fb1f00b6720 2024-09-16T01:43:24.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.725+0000 7fb1e7fff640 1 --2- 172.21.15.88:0/3509718672 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb1d006c2b0 0x7fb1d006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.726+0000 7fb1e7fff640 1 --2- 172.21.15.88:0/3509718672 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb1d006c2b0 0x7fb1d006e770 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fb1f01e9320 tx=0x7fb1d8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:24.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.726+0000 7fb1f81f4740 1 Processor -- start 2024-09-16T01:43:24.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.726+0000 7fb1f81f4740 1 -- start start 2024-09-16T01:43:24.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.727+0000 7fb1f81f4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601ef1aff00 0x5601ef1b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.727+0000 7fb1f81f4740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1ad6c0 0x5601ef1af250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.727+0000 7fb1f81f4740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601ef1adb20 0x5601ef18fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.727+0000 7fb1f81f4740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5601ef0cc930 con 0x5601ef1ad6c0 2024-09-16T01:43:24.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.727+0000 7fb1f81f4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5601ef074570 con 0x5601ef1aff00 2024-09-16T01:43:24.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.727+0000 7fb1f81f4740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5601ef0c88f0 con 0x5601ef1adb20 2024-09-16T01:43:24.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.727+0000 7fb1e7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1ad6c0 0x5601ef1af250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.727+0000 7fb1e7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1ad6c0 0x5601ef1af250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43526/0 (socket says 172.21.15.88:43526) 2024-09-16T01:43:24.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.727+0000 7fb1e7fff640 1 -- 172.21.15.88:0/2656539915 learned_addr learned my addr 172.21.15.88:0/2656539915 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:24.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.727+0000 7fb1eeffd640 1 --2- 172.21.15.88:0/2656539915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601ef1aff00 0x5601ef1b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.727+0000 7fb1ef7fe640 1 --2- 172.21.15.88:0/2656539915 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601ef1adb20 0x5601ef18fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.727+0000 7fb1e7fff640 1 -- 172.21.15.88:0/2656539915 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601ef1adb20 msgr2=0x5601ef18fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:24.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.727+0000 7fb1e7fff640 1 --2- 172.21.15.88:0/2656539915 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601ef1adb20 0x5601ef18fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.727+0000 7fb1e7fff640 1 -- 172.21.15.88:0/2656539915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601ef1aff00 msgr2=0x5601ef1b02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:24.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.727+0000 7fb1e7fff640 1 --2- 172.21.15.88:0/2656539915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601ef1aff00 0x5601ef1b02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.727+0000 7fb1e7fff640 1 -- 172.21.15.88:0/2656539915 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601ef089200 con 0x5601ef1ad6c0 2024-09-16T01:43:24.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.728+0000 7fb1e7fff640 1 --2- 172.21.15.88:0/2656539915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1ad6c0 0x5601ef1af250 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fb1d800eb70 tx=0x7fb1d8011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1257345d6d014ffe server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:24.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.728+0000 7fb1e4ff9640 1 -- 172.21.15.88:0/2656539915 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d801a050 con 0x5601ef1ad6c0 2024-09-16T01:43:24.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.728+0000 7fb1e4ff9640 1 -- 172.21.15.88:0/2656539915 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1d8024070 con 0x5601ef1ad6c0 2024-09-16T01:43:24.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.728+0000 7fb1e4ff9640 1 -- 172.21.15.88:0/2656539915 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d8013de0 con 0x5601ef1ad6c0 2024-09-16T01:43:24.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.728+0000 7fb1f81f4740 1 -- 172.21.15.88:0/2656539915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1ad6c0 msgr2=0x5601ef1af250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:24.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.728+0000 7fb1f81f4740 1 --2- 172.21.15.88:0/2656539915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1ad6c0 0x5601ef1af250 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fb1d800eb70 tx=0x7fb1d8011000 comp rx=0 tx=0).stop 2024-09-16T01:43:24.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.728+0000 7fb1f81f4740 1 -- 172.21.15.88:0/2656539915 shutdown_connections 2024-09-16T01:43:24.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.728+0000 7fb1f81f4740 1 --2- 172.21.15.88:0/2656539915 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601ef1adb20 0x5601ef18fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.728+0000 7fb1f81f4740 1 --2- 172.21.15.88:0/2656539915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1ad6c0 0x5601ef1af250 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.728+0000 7fb1f81f4740 1 --2- 172.21.15.88:0/2656539915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601ef1aff00 0x5601ef1b02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.728+0000 7fb1f81f4740 1 -- 172.21.15.88:0/2656539915 >> 172.21.15.88:0/2656539915 conn(0x5601ef165df0 msgr2=0x5601ef1adf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:24.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.729+0000 7fb1f81f4740 1 -- 172.21.15.88:0/2656539915 shutdown_connections 2024-09-16T01:43:24.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.729+0000 7fb1f81f4740 1 -- 172.21.15.88:0/2656539915 wait complete. 2024-09-16T01:43:24.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.729+0000 7fb1f81f4740 1 -- start start 2024-09-16T01:43:24.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.729+0000 7fb1f81f4740 1 Processor -- start 2024-09-16T01:43:24.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.730+0000 7fb1f81f4740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1adb20 0x5601ef18ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.730+0000 7fb1f81f4740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601ef1aff00 0x5601ef1a7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.730+0000 7fb1f81f4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601ef1a8230 0x5601ef192280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.730+0000 7fb1f81f4740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5601ef1ac620 con 0x5601ef1adb20 2024-09-16T01:43:24.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.730+0000 7fb1f81f4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5601ef0556a0 con 0x5601ef1a8230 2024-09-16T01:43:24.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.730+0000 7fb1f81f4740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5601ef1ac900 con 0x5601ef1aff00 2024-09-16T01:43:24.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.730+0000 7fb1f81f4740 10 client.0 _ll_register_callbacks cb 0x5601ef08d300 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:43:24.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.730+0000 7fb1eeffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1adb20 0x5601ef18ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.730+0000 7fb1eeffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1adb20 0x5601ef18ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43536/0 (socket says 172.21.15.88:43536) 2024-09-16T01:43:24.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.730+0000 7fb1eeffd640 1 -- 172.21.15.88:0/2856748795 learned_addr learned my addr 172.21.15.88:0/2856748795 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:24.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.730+0000 7fb1e7fff640 1 --2- 172.21.15.88:0/2856748795 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601ef1aff00 0x5601ef1a7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.730+0000 7fb1ef7fe640 1 --2- 172.21.15.88:0/2856748795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601ef1a8230 0x5601ef192280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.730+0000 7fb1eeffd640 1 -- 172.21.15.88:0/2856748795 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601ef1aff00 msgr2=0x5601ef1a7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:24.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.730+0000 7fb1eeffd640 1 --2- 172.21.15.88:0/2856748795 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601ef1aff00 0x5601ef1a7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.730+0000 7fb1eeffd640 1 -- 172.21.15.88:0/2856748795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601ef1a8230 msgr2=0x5601ef192280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:24.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.730+0000 7fb1eeffd640 1 --2- 172.21.15.88:0/2856748795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601ef1a8230 0x5601ef192280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.730+0000 7fb1eeffd640 1 -- 172.21.15.88:0/2856748795 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601eeeab7c0 con 0x5601ef1adb20 2024-09-16T01:43:24.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.730+0000 7fb1ef7fe640 1 --2- 172.21.15.88:0/2856748795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601ef1a8230 0x5601ef192280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:24.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.730+0000 7fb1eeffd640 1 --2- 172.21.15.88:0/2856748795 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1adb20 0x5601ef18ea60 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7fb1e00896f0 tx=0x7fb1e0061e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54b6e83995b5ea9d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:24.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.730+0000 7fb1cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:43:24.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.731+0000 7fb1cdffb640 1 -- 172.21.15.88:0/2856748795 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0071070 con 0x5601ef1adb20 2024-09-16T01:43:24.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.731+0000 7fb1cdffb640 1 -- 172.21.15.88:0/2856748795 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8404+0+0 (secure 0 0 0) 0x7fb1e00574a0 con 0x5601ef1adb20 2024-09-16T01:43:24.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.731+0000 7fb1f81f4740 1 -- 172.21.15.88:0/2856748795 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1d8024070 con 0x5601ef1adb20 2024-09-16T01:43:24.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.731+0000 7fb1cdffb640 1 -- 172.21.15.88:0/2856748795 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=259) ==== 24+0+0 (secure 0 0 0) 0x7fb1e0078050 con 0x5601ef1adb20 2024-09-16T01:43:24.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.731+0000 7fb1f81f4740 10 client.9744 fetch_fsmap learned FSMap version 259 2024-09-16T01:43:24.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.731+0000 7fb1f81f4740 1 -- 172.21.15.88:0/2856748795 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=259,monmap=2+}) -- 0x5601eef0bf50 con 0x5601ef1adb20 2024-09-16T01:43:24.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.731+0000 7fb1cdffb640 1 -- 172.21.15.88:0/2856748795 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1d8024070 con 0x5601ef1adb20 2024-09-16T01:43:24.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.731+0000 7fb1cdffb640 1 -- 172.21.15.88:0/2856748795 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 259) ==== 3502+0+0 (secure 0 0 0) 0x7fb1e0055810 con 0x5601ef1adb20 2024-09-16T01:43:24.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.731+0000 7fb1f81f4740 10 client.9744 fetch_fsmap finished waiting for FSMap version 259 2024-09-16T01:43:24.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.731+0000 7fb1f81f4740 10 client.9744 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:43:24.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.731+0000 7fb1f81f4740 20 client.9744 populate_metadata read hostname 'smithi088' 2024-09-16T01:43:24.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.731+0000 7fb1f81f4740 1 --2- 172.21.15.88:0/2856748795 >> [v2:172.21.15.88:6834/2555776364,v1:172.21.15.88:6836/2555776364] conn(0x5601ef1c9e10 0x5601ef1cc200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.731+0000 7fb1f81f4740 4 client.9744 mds_command: new command op to 9636 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:43:24.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.731+0000 7fb1f81f4740 1 -- 172.21.15.88:0/2856748795 --> [v2:172.21.15.88:6834/2555776364,v1:172.21.15.88:6836/2555776364] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5601ef0da060 con 0x5601ef1c9e10 2024-09-16T01:43:24.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.731+0000 7fb1e7fff640 1 --2- 172.21.15.88:0/2856748795 >> [v2:172.21.15.88:6834/2555776364,v1:172.21.15.88:6836/2555776364] conn(0x5601ef1c9e10 0x5601ef1cc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.731+0000 7fb1cdffb640 10 client.9744 ms_handle_connect on v2:172.21.15.88:6834/2555776364 2024-09-16T01:43:24.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.731+0000 7fb1e7fff640 1 --2- 172.21.15.88:0/2856748795 >> [v2:172.21.15.88:6834/2555776364,v1:172.21.15.88:6836/2555776364] conn(0x5601ef1c9e10 0x5601ef1cc200 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcf1044d32e25a06 server_cookie=6076bcc924ef184b in_seq=0 out_seq=0 2024-09-16T01:43:24.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.732+0000 7fb1cdffb640 1 -- 172.21.15.88:0/2856748795 <== mds.0 v2:172.21.15.88:6834/2555776364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5601ef0da060 con 0x5601ef1c9e10 2024-09-16T01:43:24.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.732+0000 7fb1cdffb640 10 client.9744 handle_command_reply: tid=0 2024-09-16T01:43:24.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.732+0000 7fb1f81f4740 1 client.9744 shutdown 2024-09-16T01:43:24.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.732+0000 7fb1f81f4740 1 -- 172.21.15.88:0/2856748795 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1adb20 msgr2=0x5601ef18ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:24.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.732+0000 7fb1f81f4740 1 --2- 172.21.15.88:0/2856748795 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1adb20 0x5601ef18ea60 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7fb1e00896f0 tx=0x7fb1e0061e10 comp rx=0 tx=0).stop 2024-09-16T01:43:24.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.732+0000 7fb1f81f4740 1 -- 172.21.15.88:0/2856748795 shutdown_connections 2024-09-16T01:43:24.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.732+0000 7fb1f81f4740 1 --2- 172.21.15.88:0/2856748795 >> [v2:172.21.15.88:6834/2555776364,v1:172.21.15.88:6836/2555776364] conn(0x5601ef1c9e10 0x5601ef1cc200 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.732+0000 7fb1f81f4740 1 --2- 172.21.15.88:0/2856748795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601ef1a8230 0x5601ef192280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.732+0000 7fb1f81f4740 1 --2- 172.21.15.88:0/2856748795 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601ef1aff00 0x5601ef1a7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.732+0000 7fb1f81f4740 1 --2- 172.21.15.88:0/2856748795 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1adb20 0x5601ef18ea60 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.732+0000 7fb1cdffb640 0 client.9744 ms_handle_reset on v2:172.21.15.88:6834/2555776364 2024-09-16T01:43:24.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.732+0000 7fb1f81f4740 1 -- 172.21.15.88:0/2856748795 >> 172.21.15.88:0/2856748795 conn(0x5601ef1ad6c0 msgr2=0x5601ef18fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:24.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.732+0000 7fb1f81f4740 1 -- 172.21.15.88:0/2856748795 shutdown_connections 2024-09-16T01:43:24.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.732+0000 7fb1f81f4740 1 -- 172.21.15.88:0/2856748795 wait complete. 2024-09-16T01:43:24.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.732+0000 7fb1f81f4740 20 client.9744 trim_cache size 0 max 16384 2024-09-16T01:43:24.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.745+0000 7fb1f81f4740 1 Processor -- start 2024-09-16T01:43:24.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.745+0000 7fb1f81f4740 1 -- start start 2024-09-16T01:43:24.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.746+0000 7fb1f81f4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601ef1a8230 0x5601ef1a5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.746+0000 7fb1f81f4740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601ef1adb20 0x5601ef190ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.746+0000 7fb1f81f4740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1aff00 0x5601ef191400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.746+0000 7fb1f81f4740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f02017b0 con 0x5601ef1aff00 2024-09-16T01:43:24.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.746+0000 7fb1f81f4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f0200b40 con 0x5601ef1a8230 2024-09-16T01:43:24.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.746+0000 7fb1f81f4740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb1f0201160 con 0x5601ef1adb20 2024-09-16T01:43:24.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.746+0000 7fb1ef7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1aff00 0x5601ef191400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.746+0000 7fb1ef7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1aff00 0x5601ef191400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43548/0 (socket says 172.21.15.88:43548) 2024-09-16T01:43:24.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.746+0000 7fb1ef7fe640 1 -- 172.21.15.88:0/1307783318 learned_addr learned my addr 172.21.15.88:0/1307783318 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:24.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.746+0000 7fb1ef7fe640 1 -- 172.21.15.88:0/1307783318 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601ef1adb20 msgr2=0x5601ef190ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:24.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.746+0000 7fb1ef7fe640 1 --2- 172.21.15.88:0/1307783318 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601ef1adb20 0x5601ef190ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.746+0000 7fb1ef7fe640 1 -- 172.21.15.88:0/1307783318 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601ef1a8230 msgr2=0x5601ef1a5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:24.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.746+0000 7fb1ef7fe640 1 --2- 172.21.15.88:0/1307783318 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601ef1a8230 0x5601ef1a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.746+0000 7fb1ef7fe640 1 -- 172.21.15.88:0/1307783318 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601eed9f0b0 con 0x5601ef1aff00 2024-09-16T01:43:24.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.746+0000 7fb1ef7fe640 1 --2- 172.21.15.88:0/1307783318 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1aff00 0x5601ef191400 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8034040 tx=0x7fb1e802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1648817f8b70636 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:24.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.747+0000 7fb1cffff640 1 -- 172.21.15.88:0/1307783318 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8036040 con 0x5601ef1aff00 2024-09-16T01:43:24.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.747+0000 7fb1cffff640 1 -- 172.21.15.88:0/1307783318 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1e8031740 con 0x5601ef1aff00 2024-09-16T01:43:24.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.747+0000 7fb1cffff640 1 -- 172.21.15.88:0/1307783318 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8031af0 con 0x5601ef1aff00 2024-09-16T01:43:24.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.747+0000 7fb1f81f4740 1 -- 172.21.15.88:0/1307783318 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1aff00 msgr2=0x5601ef191400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:24.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.747+0000 7fb1f81f4740 1 --2- 172.21.15.88:0/1307783318 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1aff00 0x5601ef191400 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8034040 tx=0x7fb1e802e000 comp rx=0 tx=0).stop 2024-09-16T01:43:24.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.747+0000 7fb1f81f4740 1 -- 172.21.15.88:0/1307783318 shutdown_connections 2024-09-16T01:43:24.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.747+0000 7fb1f81f4740 1 --2- 172.21.15.88:0/1307783318 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1aff00 0x5601ef191400 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.747+0000 7fb1f81f4740 1 --2- 172.21.15.88:0/1307783318 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601ef1adb20 0x5601ef190ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.747+0000 7fb1f81f4740 1 --2- 172.21.15.88:0/1307783318 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601ef1a8230 0x5601ef1a5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.747+0000 7fb1f81f4740 1 -- 172.21.15.88:0/1307783318 >> 172.21.15.88:0/1307783318 conn(0x5601ef1ad6c0 msgr2=0x5601ef1a9070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:24.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.747+0000 7fb1f81f4740 1 -- 172.21.15.88:0/1307783318 shutdown_connections 2024-09-16T01:43:24.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.747+0000 7fb1f81f4740 1 -- 172.21.15.88:0/1307783318 wait complete. 2024-09-16T01:43:24.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.748+0000 7fb1f81f4740 1 -- start start 2024-09-16T01:43:24.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.748+0000 7fb1f81f4740 1 Processor -- start 2024-09-16T01:43:24.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.748+0000 7fb1f81f4740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601ef1a8230 0x5601ef226960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.748+0000 7fb1f81f4740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601ef1adb20 0x5601ef1c0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.748+0000 7fb1f81f4740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1aff00 0x5601ef1c09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.748+0000 7fb1f81f4740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5601ef18eff0 con 0x5601ef1aff00 2024-09-16T01:43:24.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.748+0000 7fb1f81f4740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5601ef1a6d50 con 0x5601ef1a8230 2024-09-16T01:43:24.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.748+0000 7fb1f81f4740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5601ef1ab450 con 0x5601ef1adb20 2024-09-16T01:43:24.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.749+0000 7fb1f81f4740 10 client.0 _ll_register_callbacks cb 0x7fb1e000c3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:43:24.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.749+0000 7fb1ef7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1aff00 0x5601ef1c09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.749+0000 7fb1ef7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1aff00 0x5601ef1c09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43560/0 (socket says 172.21.15.88:43560) 2024-09-16T01:43:24.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.749+0000 7fb1ef7fe640 1 -- 172.21.15.88:0/2296609947 learned_addr learned my addr 172.21.15.88:0/2296609947 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:24.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.749+0000 7fb1e7fff640 1 --2- 172.21.15.88:0/2296609947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601ef1adb20 0x5601ef1c0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.749+0000 7fb1eeffd640 1 --2- 172.21.15.88:0/2296609947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601ef1a8230 0x5601ef226960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.749+0000 7fb1ef7fe640 1 -- 172.21.15.88:0/2296609947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601ef1adb20 msgr2=0x5601ef1c0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:24.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.749+0000 7fb1ef7fe640 1 --2- 172.21.15.88:0/2296609947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601ef1adb20 0x5601ef1c0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.749+0000 7fb1ef7fe640 1 -- 172.21.15.88:0/2296609947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601ef1a8230 msgr2=0x5601ef226960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:24.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.749+0000 7fb1ef7fe640 1 --2- 172.21.15.88:0/2296609947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601ef1a8230 0x5601ef226960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.749+0000 7fb1ef7fe640 1 -- 172.21.15.88:0/2296609947 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601ef1a5530 con 0x5601ef1aff00 2024-09-16T01:43:24.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.749+0000 7fb1ef7fe640 1 --2- 172.21.15.88:0/2296609947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1aff00 0x5601ef1c09d0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fb1e802ac50 tx=0x7fb1e8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcfa3d05ad9284c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:24.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.749+0000 7fb1cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:43:24.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.749+0000 7fb1cdffb640 1 -- 172.21.15.88:0/2296609947 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8036070 con 0x5601ef1aff00 2024-09-16T01:43:24.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.749+0000 7fb1f81f4740 1 -- 172.21.15.88:0/2296609947 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1e8031740 con 0x5601ef1aff00 2024-09-16T01:43:24.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.749+0000 7fb1cdffb640 1 -- 172.21.15.88:0/2296609947 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 8404+0+0 (secure 0 0 0) 0x7fb1e8003b90 con 0x5601ef1aff00 2024-09-16T01:43:24.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.749+0000 7fb1cdffb640 1 -- 172.21.15.88:0/2296609947 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=259) ==== 24+0+0 (secure 0 0 0) 0x7fb1e8003d90 con 0x5601ef1aff00 2024-09-16T01:43:24.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.749+0000 7fb1f81f4740 10 client.9750 fetch_fsmap learned FSMap version 259 2024-09-16T01:43:24.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.749+0000 7fb1f81f4740 1 -- 172.21.15.88:0/2296609947 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=259,monmap=2+}) -- 0x5601ef226ea0 con 0x5601ef1aff00 2024-09-16T01:43:24.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.749+0000 7fb1cdffb640 1 -- 172.21.15.88:0/2296609947 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1e8031740 con 0x5601ef1aff00 2024-09-16T01:43:24.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.750+0000 7fb1cdffb640 1 -- 172.21.15.88:0/2296609947 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 259) ==== 3502+0+0 (secure 0 0 0) 0x7fb1e8008040 con 0x5601ef1aff00 2024-09-16T01:43:24.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.750+0000 7fb1f81f4740 10 client.9750 fetch_fsmap finished waiting for FSMap version 259 2024-09-16T01:43:24.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.750+0000 7fb1f81f4740 10 client.9750 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:43:24.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.750+0000 7fb1f81f4740 20 client.9750 populate_metadata read hostname 'smithi088' 2024-09-16T01:43:24.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.750+0000 7fb1f81f4740 1 --2- 172.21.15.88:0/2296609947 >> [v2:172.21.15.88:6834/2555776364,v1:172.21.15.88:6836/2555776364] conn(0x5601ef1b2090 0x5601ef238190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:24.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.750+0000 7fb1f81f4740 4 client.9750 mds_command: new command op to 9636 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-09-16T01:43:24.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.750+0000 7fb1f81f4740 1 -- 172.21.15.88:0/2296609947 --> [v2:172.21.15.88:6834/2555776364,v1:172.21.15.88:6836/2555776364] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x5601ef0d43b0 con 0x5601ef1b2090 2024-09-16T01:43:24.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.750+0000 7fb1eeffd640 1 --2- 172.21.15.88:0/2296609947 >> [v2:172.21.15.88:6834/2555776364,v1:172.21.15.88:6836/2555776364] conn(0x5601ef1b2090 0x5601ef238190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:24.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.750+0000 7fb1cdffb640 10 client.9750 ms_handle_connect on v2:172.21.15.88:6834/2555776364 2024-09-16T01:43:24.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.750+0000 7fb1eeffd640 1 --2- 172.21.15.88:0/2296609947 >> [v2:172.21.15.88:6834/2555776364,v1:172.21.15.88:6836/2555776364] conn(0x5601ef1b2090 0x5601ef238190 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af607c64918e23a2 server_cookie=59dabc1ae77d9392 in_seq=0 out_seq=0 2024-09-16T01:43:24.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.750+0000 7fb1cdffb640 1 -- 172.21.15.88:0/2296609947 <== mds.0 v2:172.21.15.88:6834/2555776364 1 ==== command_reply(tid 0: 0 ) ==== 8+0+215 (crc 0 0 0) 0x5601ef0d43b0 con 0x5601ef1b2090 2024-09-16T01:43:24.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.750+0000 7fb1cdffb640 10 client.9750 handle_command_reply: tid=0 2024-09-16T01:43:24.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.750+0000 7fb1f81f4740 1 client.9750 shutdown 2024-09-16T01:43:24.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.750+0000 7fb1f81f4740 1 -- 172.21.15.88:0/2296609947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1aff00 msgr2=0x5601ef1c09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:24.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.750+0000 7fb1f81f4740 1 --2- 172.21.15.88:0/2296609947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1aff00 0x5601ef1c09d0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fb1e802ac50 tx=0x7fb1e8004ab0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.750+0000 7fb1f81f4740 1 -- 172.21.15.88:0/2296609947 shutdown_connections 2024-09-16T01:43:24.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.750+0000 7fb1f81f4740 1 --2- 172.21.15.88:0/2296609947 >> [v2:172.21.15.88:6834/2555776364,v1:172.21.15.88:6836/2555776364] conn(0x5601ef1b2090 0x5601ef238190 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.750+0000 7fb1f81f4740 1 --2- 172.21.15.88:0/2296609947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5601ef1aff00 0x5601ef1c09d0 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.750+0000 7fb1cdffb640 0 client.9750 ms_handle_reset on v2:172.21.15.88:6834/2555776364 2024-09-16T01:43:24.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.750+0000 7fb1f81f4740 1 --2- 172.21.15.88:0/2296609947 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5601ef1adb20 0x5601ef1c0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.750+0000 7fb1f81f4740 1 --2- 172.21.15.88:0/2296609947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5601ef1a8230 0x5601ef226960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.750+0000 7fb1f81f4740 1 -- 172.21.15.88:0/2296609947 >> 172.21.15.88:0/2296609947 conn(0x5601ef1ad6c0 msgr2=0x5601ef1a7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:24.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.750+0000 7fb1f81f4740 1 -- 172.21.15.88:0/2296609947 shutdown_connections 2024-09-16T01:43:24.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.751+0000 7fb1f81f4740 1 -- 172.21.15.88:0/2296609947 wait complete. 2024-09-16T01:43:24.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.751+0000 7fb1f81f4740 20 client.9750 trim_cache size 0 max 16384 2024-09-16T01:43:24.759 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:24.759 INFO:teuthology.orchestra.run.smithi088.stdout:[ 2024-09-16T01:43:24.759 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:43:24.759 INFO:teuthology.orchestra.run.smithi088.stdout: "damage_type": "dentry", 2024-09-16T01:43:24.759 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 2414809347, 2024-09-16T01:43:24.759 INFO:teuthology.orchestra.run.smithi088.stdout: "ino": 1099511627776, 2024-09-16T01:43:24.759 INFO:teuthology.orchestra.run.smithi088.stdout: "frag": "*", 2024-09-16T01:43:24.759 INFO:teuthology.orchestra.run.smithi088.stdout: "dname": "sixmegs", 2024-09-16T01:43:24.759 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_id": "head", 2024-09-16T01:43:24.759 INFO:teuthology.orchestra.run.smithi088.stdout: "path": "/subdir/sixmegs" 2024-09-16T01:43:24.760 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:43:24.760 INFO:teuthology.orchestra.run.smithi088.stdout:] 2024-09-16T01:43:24.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.751+0000 7fb1f545c640 1 -- 172.21.15.88:0/3509718672 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb1d006c2b0 msgr2=0x7fb1d006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:24.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.751+0000 7fb1f545c640 1 --2- 172.21.15.88:0/3509718672 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb1d006c2b0 0x7fb1d006e770 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fb1f01e9320 tx=0x7fb1d8008040 comp rx=0 tx=0).stop 2024-09-16T01:43:24.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.751+0000 7fb1f545c640 1 -- 172.21.15.88:0/3509718672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1f00b6720 msgr2=0x7fb1f00cae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:24.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.751+0000 7fb1f545c640 1 --2- 172.21.15.88:0/3509718672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1f00b6720 0x7fb1f00cae20 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7fb1e000c460 tx=0x7fb1e000c920 comp rx=0 tx=0).stop 2024-09-16T01:43:24.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.751+0000 7fb1f545c640 1 -- 172.21.15.88:0/3509718672 shutdown_connections 2024-09-16T01:43:24.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.751+0000 7fb1f545c640 1 --2- 172.21.15.88:0/3509718672 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb1d006c2b0 0x7fb1d006e770 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.751+0000 7fb1f545c640 1 --2- 172.21.15.88:0/3509718672 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1f015e7a0 0x7fb1f00ce250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.751+0000 7fb1f545c640 1 --2- 172.21.15.88:0/3509718672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1f00cf1f0 0x7fb1f01458e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.751+0000 7fb1f545c640 1 --2- 172.21.15.88:0/3509718672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1f00b6720 0x7fb1f00cae20 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:24.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.751+0000 7fb1f545c640 1 -- 172.21.15.88:0/3509718672 >> 172.21.15.88:0/3509718672 conn(0x7fb1f00d11e0 msgr2=0x7fb1f01e9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:24.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.751+0000 7fb1f545c640 1 -- 172.21.15.88:0/3509718672 shutdown_connections 2024-09-16T01:43:24.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.751+0000 7fb1f545c640 1 -- 172.21.15.88:0/3509718672 wait complete. 2024-09-16T01:43:24.763 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap key 100.00000000:stray0_head' 2024-09-16T01:43:24.763 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:43:24.763 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:43:24.779 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:43:24.780 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:43:24.836 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:43:24.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.974+0000 7fc09ae9b640 1 Processor -- start 2024-09-16T01:43:24.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.974+0000 7fc09ae9b640 1 -- start start 2024-09-16T01:43:24.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.975+0000 7fc09ae9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc09408c450 0x7fc09406e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:25.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.975+0000 7fc09ae9b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc09406ec20 0x7fc0940b4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:25.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.975+0000 7fc09ae9b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0940b3530 con 0x7fc09416b3e0 2024-09-16T01:43:25.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.975+0000 7fc09ae9b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc09406f090 con 0x7fc09408c450 2024-09-16T01:43:25.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.975+0000 7fc09ae9b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc09416b7c0 con 0x7fc09406ec20 2024-09-16T01:43:25.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.975+0000 7fc093fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc09408c450 0x7fc09406e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:25.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.975+0000 7fc093fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc09408c450 0x7fc09406e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43568/0 (socket says 172.21.15.88:43568) 2024-09-16T01:43:25.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.975+0000 7fc098c37640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc09416b3e0 0x7fc09406f330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36308/0 (socket says 172.21.15.88:36308) 2024-09-16T01:43:25.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.975+0000 7fc093fff640 1 -- 172.21.15.88:0/306504123 learned_addr learned my addr 172.21.15.88:0/306504123 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:25.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.975+0000 7fc0937fe640 1 -- 172.21.15.88:0/306504123 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4009672737 0 0) 0x7fc0940b3530 con 0x7fc09416b3e0 2024-09-16T01:43:25.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.975+0000 7fc099438640 1 --2- 172.21.15.88:0/306504123 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc09406ec20 0x7fc0940b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:25.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.998+0000 7fc0937fe640 1 -- 172.21.15.88:0/306504123 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc084003500 con 0x7fc09416b3e0 2024-09-16T01:43:25.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.998+0000 7fc093fff640 1 -- 172.21.15.88:0/306504123 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc09406ec20 msgr2=0x7fc0940b4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:25.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.998+0000 7fc093fff640 1 --2- 172.21.15.88:0/306504123 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc09406ec20 0x7fc0940b4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:25.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.998+0000 7fc093fff640 1 -- 172.21.15.88:0/306504123 >> v1:172.21.15.88:6789/0 conn(0x7fc09416b3e0 legacy=0x7fc09406f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:25.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.998+0000 7fc0937fe640 1 -- 172.21.15.88:0/306504123 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2766591228 0 0) 0x7fc084003500 con 0x7fc09416b3e0 2024-09-16T01:43:25.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.998+0000 7fc093fff640 1 -- 172.21.15.88:0/306504123 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0940b3810 con 0x7fc09408c450 2024-09-16T01:43:25.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.998+0000 7fc093fff640 1 --2- 172.21.15.88:0/306504123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc09408c450 0x7fc09406e6e0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7fc08000a910 tx=0x7fc08000a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b94730f6fca557a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:25.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.999+0000 7fc0937fe640 1 -- 172.21.15.88:0/306504123 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc080038070 con 0x7fc09408c450 2024-09-16T01:43:25.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.999+0000 7fc0937fe640 1 -- 172.21.15.88:0/306504123 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc080030480 con 0x7fc09408c450 2024-09-16T01:43:25.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.999+0000 7fc0937fe640 1 -- 172.21.15.88:0/306504123 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc088001670 con 0x7fc09408c450 2024-09-16T01:43:25.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.999+0000 7fc09ae9b640 1 -- 172.21.15.88:0/306504123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc09408c450 msgr2=0x7fc09406e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:25.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.999+0000 7fc09ae9b640 1 --2- 172.21.15.88:0/306504123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc09408c450 0x7fc09406e6e0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7fc08000a910 tx=0x7fc08000a250 comp rx=0 tx=0).stop 2024-09-16T01:43:25.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.999+0000 7fc09ae9b640 1 -- 172.21.15.88:0/306504123 shutdown_connections 2024-09-16T01:43:25.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.999+0000 7fc09ae9b640 1 --2- 172.21.15.88:0/306504123 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc09406ec20 0x7fc0940b4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:25.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.999+0000 7fc09ae9b640 1 --2- 172.21.15.88:0/306504123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc09408c450 0x7fc09406e6e0 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:25.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.999+0000 7fc09ae9b640 1 -- 172.21.15.88:0/306504123 >> 172.21.15.88:0/306504123 conn(0x7fc0940d03e0 msgr2=0x7fc0940d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:25.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.999+0000 7fc09ae9b640 1 -- 172.21.15.88:0/306504123 shutdown_connections 2024-09-16T01:43:25.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:24.999+0000 7fc09ae9b640 1 -- 172.21.15.88:0/306504123 wait complete. 2024-09-16T01:43:25.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.000+0000 7fc09ae9b640 1 Processor -- start 2024-09-16T01:43:25.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.000+0000 7fc09ae9b640 1 -- start start 2024-09-16T01:43:25.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.001+0000 7fc09ae9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc09406ec20 0x7fc0941f42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:25.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.001+0000 7fc09ae9b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc09408c450 0x7fc094167330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:25.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.001+0000 7fc09ae9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc09416b3e0 0x7fc0942046a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:25.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.001+0000 7fc09ae9b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc0940b40b0 con 0x7fc09416b3e0 2024-09-16T01:43:25.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.001+0000 7fc09ae9b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc0940b3dd0 con 0x7fc09406ec20 2024-09-16T01:43:25.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.001+0000 7fc09ae9b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc0940b4d80 con 0x7fc09408c450 2024-09-16T01:43:25.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.001+0000 7fc099438640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc09416b3e0 0x7fc0942046a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:25.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.001+0000 7fc099438640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc09416b3e0 0x7fc0942046a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43582/0 (socket says 172.21.15.88:43582) 2024-09-16T01:43:25.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.001+0000 7fc099438640 1 -- 172.21.15.88:0/123411329 learned_addr learned my addr 172.21.15.88:0/123411329 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:25.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.002+0000 7fc093fff640 1 --2- 172.21.15.88:0/123411329 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc09408c450 0x7fc094167330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:25.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.002+0000 7fc098c37640 1 --2- 172.21.15.88:0/123411329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc09406ec20 0x7fc0941f42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:25.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.002+0000 7fc099438640 1 -- 172.21.15.88:0/123411329 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc09408c450 msgr2=0x7fc094167330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:25.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.002+0000 7fc099438640 1 --2- 172.21.15.88:0/123411329 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc09408c450 0x7fc094167330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:25.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.002+0000 7fc099438640 1 -- 172.21.15.88:0/123411329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc09406ec20 msgr2=0x7fc0941f42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:25.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.002+0000 7fc099438640 1 --2- 172.21.15.88:0/123411329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc09406ec20 0x7fc0941f42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:25.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.002+0000 7fc099438640 1 -- 172.21.15.88:0/123411329 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0942011f0 con 0x7fc09416b3e0 2024-09-16T01:43:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.002+0000 7fc098c37640 1 --2- 172.21.15.88:0/123411329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc09406ec20 0x7fc0941f42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.002+0000 7fc099438640 1 --2- 172.21.15.88:0/123411329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc09416b3e0 0x7fc0942046a0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7fc08c00efc0 tx=0x7fc08c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.002+0000 7fc0917fa640 1 -- 172.21.15.88:0/123411329 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc08c018020 con 0x7fc09416b3e0 2024-09-16T01:43:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.002+0000 7fc0917fa640 1 -- 172.21.15.88:0/123411329 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc08c014480 con 0x7fc09416b3e0 2024-09-16T01:43:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.002+0000 7fc09ae9b640 1 -- 172.21.15.88:0/123411329 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0942014d0 con 0x7fc09416b3e0 2024-09-16T01:43:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.002+0000 7fc0917fa640 1 -- 172.21.15.88:0/123411329 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc08c014870 con 0x7fc09416b3e0 2024-09-16T01:43:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.002+0000 7fc09ae9b640 1 -- 172.21.15.88:0/123411329 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc094201a90 con 0x7fc09416b3e0 2024-09-16T01:43:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.003+0000 7fc09ae9b640 1 -- 172.21.15.88:0/123411329 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc094069180 con 0x7fc09416b3e0 2024-09-16T01:43:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.004+0000 7fc0917fa640 1 -- 172.21.15.88:0/123411329 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc08c005d10 con 0x7fc09416b3e0 2024-09-16T01:43:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.004+0000 7fc0917fa640 1 --2- 172.21.15.88:0/123411329 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc07806c2b0 0x7fc07806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.004+0000 7fc0917fa640 1 -- 172.21.15.88:0/123411329 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8404+0+0 (secure 0 0 0) 0x7fc08c08c9e0 con 0x7fc09416b3e0 2024-09-16T01:43:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.004+0000 7fc098c37640 1 --2- 172.21.15.88:0/123411329 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc07806c2b0 0x7fc07806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.004+0000 7fc098c37640 1 --2- 172.21.15.88:0/123411329 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc07806c2b0 0x7fc07806e770 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fc088000f60 tx=0x7fc088009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:25.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.008+0000 7fc0917fa640 1 -- 172.21.15.88:0/123411329 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc094069180 con 0x7fc09416b3e0 2024-09-16T01:43:25.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.184+0000 7fc09ae9b640 1 -- 172.21.15.88:0/123411329 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc0941633b0 con 0x7fc09416b3e0 2024-09-16T01:43:25.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.184+0000 7fc0917fa640 1 -- 172.21.15.88:0/123411329 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 62 entries v118) ==== 81+0+3770 (secure 0 0 0) 0x7fc08c022070 con 0x7fc09416b3e0 2024-09-16T01:43:25.186 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/1372908065 2024-09-17T01:43:12.443265+0000 2024-09-16T01:43:25.186 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/1372908065 2024-09-17T01:43:12.443265+0000 2024-09-16T01:43:25.186 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/650399848 2024-09-17T01:43:08.279879+0000 2024-09-16T01:43:25.186 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/1255135754 2024-09-17T01:41:41.905258+0000 2024-09-16T01:43:25.186 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/3728363359 2024-09-17T01:42:05.412133+0000 2024-09-16T01:43:25.186 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/293764774 2024-09-17T01:41:16.203758+0000 2024-09-16T01:43:25.186 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/1255135754 2024-09-17T01:41:41.905258+0000 2024-09-16T01:43:25.186 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2918194823 2024-09-17T01:41:29.275696+0000 2024-09-16T01:43:25.186 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/3336935653 2024-09-17T01:41:44.295023+0000 2024-09-16T01:43:25.186 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/981209627 2024-09-17T01:41:51.268495+0000 2024-09-16T01:43:25.187 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/293764774 2024-09-17T01:41:16.203758+0000 2024-09-16T01:43:25.187 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2680708900 2024-09-17T01:42:41.629904+0000 2024-09-16T01:43:25.187 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2957965650 2024-09-17T01:41:12.181485+0000 2024-09-16T01:43:25.187 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/3336935653 2024-09-17T01:41:44.295023+0000 2024-09-16T01:43:25.187 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2285699282 2024-09-17T01:42:29.541113+0000 2024-09-16T01:43:25.187 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/1162318664 2024-09-17T01:41:23.257870+0000 2024-09-16T01:43:25.187 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/128058689 2024-09-17T01:41:10.163743+0000 2024-09-16T01:43:25.187 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/981209627 2024-09-17T01:41:51.268495+0000 2024-09-16T01:43:25.187 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/128058689 2024-09-17T01:41:10.163743+0000 2024-09-16T01:43:25.187 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/1042715824 2024-09-17T01:40:59.080515+0000 2024-09-16T01:43:25.187 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2957965650 2024-09-17T01:41:12.181485+0000 2024-09-16T01:43:25.187 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2970185561 2024-09-17T01:40:38.255774+0000 2024-09-16T01:43:25.187 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/1162318664 2024-09-17T01:41:23.257870+0000 2024-09-16T01:43:25.188 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/1289932173 2024-09-17T01:40:42.399399+0000 2024-09-16T01:43:25.188 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/3356432718 2024-09-17T01:40:32.344252+0000 2024-09-16T01:43:25.188 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2970185561 2024-09-17T01:40:38.255774+0000 2024-09-16T01:43:25.188 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2733458327 2024-09-17T01:40:27.314712+0000 2024-09-16T01:43:25.188 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/3356432718 2024-09-17T01:40:32.344252+0000 2024-09-16T01:43:25.188 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2918194823 2024-09-17T01:41:29.275696+0000 2024-09-16T01:43:25.188 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/4144876777 2024-09-17T01:43:00.350808+0000 2024-09-16T01:43:25.188 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2408579939 2024-09-17T01:40:46.434492+0000 2024-09-16T01:43:25.188 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/3163343083 2024-09-17T01:42:01.375396+0000 2024-09-16T01:43:25.188 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/4222250252 2024-09-17T01:42:45.673956+0000 2024-09-16T01:43:25.188 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/4162274767 2024-09-17T01:40:11.522091+0000 2024-09-16T01:43:25.188 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2477051134 2024-09-17T01:43:04.380783+0000 2024-09-16T01:43:25.188 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/1289932173 2024-09-17T01:40:42.399399+0000 2024-09-16T01:43:25.189 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2733458327 2024-09-17T01:40:27.314712+0000 2024-09-16T01:43:25.189 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2550498835 2024-09-17T01:41:57.358287+0000 2024-09-16T01:43:25.189 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2550498835 2024-09-17T01:41:57.358287+0000 2024-09-16T01:43:25.189 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2285699282 2024-09-17T01:42:29.541113+0000 2024-09-16T01:43:25.189 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/3163343083 2024-09-17T01:42:01.375396+0000 2024-09-16T01:43:25.189 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/3728363359 2024-09-17T01:42:05.412133+0000 2024-09-16T01:43:25.189 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/880314974 2024-09-17T01:42:21.485748+0000 2024-09-16T01:43:25.189 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/1936493048 2024-09-17T01:42:19.133197+0000 2024-09-16T01:43:25.189 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/1936493048 2024-09-17T01:42:19.133197+0000 2024-09-16T01:43:25.189 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/880314974 2024-09-17T01:42:21.485748+0000 2024-09-16T01:43:25.189 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/3374372758 2024-09-17T01:42:25.506299+0000 2024-09-16T01:43:25.189 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/3374372758 2024-09-17T01:42:25.506299+0000 2024-09-16T01:43:25.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/201188184 2024-09-17T01:42:33.276104+0000 2024-09-16T01:43:25.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/1990650099 2024-09-17T01:42:37.599896+0000 2024-09-16T01:43:25.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/201188184 2024-09-17T01:42:33.276104+0000 2024-09-16T01:43:25.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/1990650099 2024-09-17T01:42:37.599896+0000 2024-09-16T01:43:25.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/4162274767 2024-09-17T01:40:11.522091+0000 2024-09-16T01:43:25.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2408579939 2024-09-17T01:40:46.434492+0000 2024-09-16T01:43:25.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2680708900 2024-09-17T01:42:41.629904+0000 2024-09-16T01:43:25.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/4222250252 2024-09-17T01:42:45.673956+0000 2024-09-16T01:43:25.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/1042715824 2024-09-17T01:40:59.080515+0000 2024-09-16T01:43:25.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/4065900771 2024-09-17T01:42:58.329389+0000 2024-09-16T01:43:25.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/4065900771 2024-09-17T01:42:58.329389+0000 2024-09-16T01:43:25.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/4144876777 2024-09-17T01:43:00.350808+0000 2024-09-16T01:43:25.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2477051134 2024-09-17T01:43:04.380783+0000 2024-09-16T01:43:25.191 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/650399848 2024-09-17T01:43:08.279879+0000 2024-09-16T01:43:25.191 INFO:teuthology.orchestra.run.smithi088.stderr:listed 62 entries 2024-09-16T01:43:25.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.186+0000 7fc09ae9b640 1 -- 172.21.15.88:0/123411329 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc07806c2b0 msgr2=0x7fc07806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:25.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.186+0000 7fc09ae9b640 1 --2- 172.21.15.88:0/123411329 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc07806c2b0 0x7fc07806e770 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fc088000f60 tx=0x7fc088009040 comp rx=0 tx=0).stop 2024-09-16T01:43:25.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.186+0000 7fc09ae9b640 1 -- 172.21.15.88:0/123411329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc09416b3e0 msgr2=0x7fc0942046a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:25.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.186+0000 7fc09ae9b640 1 --2- 172.21.15.88:0/123411329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc09416b3e0 0x7fc0942046a0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7fc08c00efc0 tx=0x7fc08c00c490 comp rx=0 tx=0).stop 2024-09-16T01:43:25.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.186+0000 7fc09ae9b640 1 -- 172.21.15.88:0/123411329 shutdown_connections 2024-09-16T01:43:25.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.186+0000 7fc09ae9b640 1 --2- 172.21.15.88:0/123411329 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc07806c2b0 0x7fc07806e770 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:25.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.186+0000 7fc09ae9b640 1 --2- 172.21.15.88:0/123411329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc09416b3e0 0x7fc0942046a0 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:25.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.186+0000 7fc09ae9b640 1 --2- 172.21.15.88:0/123411329 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc09408c450 0x7fc094167330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:25.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.186+0000 7fc09ae9b640 1 --2- 172.21.15.88:0/123411329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc09406ec20 0x7fc0941f42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:25.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.186+0000 7fc09ae9b640 1 -- 172.21.15.88:0/123411329 >> 172.21.15.88:0/123411329 conn(0x7fc0940d03e0 msgr2=0x7fc0940b5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:25.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.186+0000 7fc09ae9b640 1 -- 172.21.15.88:0/123411329 shutdown_connections 2024-09-16T01:43:25.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.186+0000 7fc09ae9b640 1 -- 172.21.15.88:0/123411329 wait complete. 2024-09-16T01:43:25.199 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-16T01:43:25.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:25.199 DEBUG:teuthology.orchestra.run.smithi088:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-09-16T01:43:25.225 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:43:25.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:25.225 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:43:25.281 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:43:25.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.419+0000 7f4b455af640 1 Processor -- start 2024-09-16T01:43:25.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.419+0000 7f4b455af640 1 -- start start 2024-09-16T01:43:25.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.420+0000 7f4b455af640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b4015e7a0 0x7f4b4006caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:25.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.420+0000 7f4b455af640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b400cf860 con 0x7f4b400b68f0 2024-09-16T01:43:25.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.420+0000 7f4b455af640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b400cfb40 con 0x7f4b400cf1f0 2024-09-16T01:43:25.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.420+0000 7f4b455af640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4b400d2ca0 con 0x7f4b4015e7a0 2024-09-16T01:43:25.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.420+0000 7f4b3f7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4b400b68f0 0x7f4b400d3410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36324/0 (socket says 172.21.15.88:36324) 2024-09-16T01:43:25.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.420+0000 7f4b3f7fe640 1 -- 172.21.15.88:0/1290574210 learned_addr learned my addr 172.21.15.88:0/1290574210 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:25.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.420+0000 7f4b3effd640 1 --2- 172.21.15.88:0/1290574210 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b4015e7a0 0x7f4b4006caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:25.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.420+0000 7f4b3e7fc640 1 -- 172.21.15.88:0/1290574210 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1631945484 0 0) 0x7f4b400cf860 con 0x7f4b400b68f0 2024-09-16T01:43:25.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.421+0000 7f4b3e7fc640 1 -- 172.21.15.88:0/1290574210 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b2c003520 con 0x7f4b400b68f0 2024-09-16T01:43:25.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.421+0000 7f4b3e7fc640 1 -- 172.21.15.88:0/1290574210 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2452428395 0 0) 0x7f4b2c003520 con 0x7f4b400b68f0 2024-09-16T01:43:25.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.421+0000 7f4b3e7fc640 1 -- 172.21.15.88:0/1290574210 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b4015e7a0 msgr2=0x7f4b4006caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:25.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.421+0000 7f4b3e7fc640 1 --2- 172.21.15.88:0/1290574210 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b4015e7a0 0x7f4b4006caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:25.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.421+0000 7f4b3e7fc640 1 -- 172.21.15.88:0/1290574210 >> v1:172.21.15.154:6789/0 conn(0x7f4b400cf1f0 legacy=0x7f4b400b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:25.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.421+0000 7f4b3e7fc640 1 -- 172.21.15.88:0/1290574210 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b400cfe20 con 0x7f4b400b68f0 2024-09-16T01:43:25.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.421+0000 7f4b3e7fc640 1 -- 172.21.15.88:0/1290574210 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f4b380019f0 con 0x7f4b400b68f0 2024-09-16T01:43:25.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.421+0000 7f4b3e7fc640 1 -- 172.21.15.88:0/1290574210 >> v1:172.21.15.88:6789/0 conn(0x7f4b400b68f0 legacy=0x7f4b400d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:25.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.421+0000 7f4b3e7fc640 1 --2- 172.21.15.88:0/1290574210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b2c0039e0 0x7f4b2c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:25.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.421+0000 7f4b3e7fc640 1 --2- 172.21.15.88:0/1290574210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b2c004390 0x7f4b2c004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:25.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.421+0000 7f4b3e7fc640 1 --2- 172.21.15.88:0/1290574210 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b4015e7a0 0x7f4b2c005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:25.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.421+0000 7f4b3e7fc640 1 -- 172.21.15.88:0/1290574210 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4b400d2ca0 con 0x7f4b2c004390 2024-09-16T01:43:25.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.421+0000 7f4b3e7fc640 1 -- 172.21.15.88:0/1290574210 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4b2c00ba70 con 0x7f4b2c0039e0 2024-09-16T01:43:25.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.421+0000 7f4b3e7fc640 1 -- 172.21.15.88:0/1290574210 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4b2c00bc70 con 0x7f4b4015e7a0 2024-09-16T01:43:25.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.421+0000 7f4b367fc640 1 --2- 172.21.15.88:0/1290574210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b2c004390 0x7f4b2c004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:25.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.421+0000 7f4b3effd640 1 --2- 172.21.15.88:0/1290574210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b2c0039e0 0x7f4b2c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:25.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.421+0000 7f4b3f7fe640 1 --2- 172.21.15.88:0/1290574210 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b4015e7a0 0x7f4b2c005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:25.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.421+0000 7f4b367fc640 1 -- 172.21.15.88:0/1290574210 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b4015e7a0 msgr2=0x7f4b2c005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:25.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.421+0000 7f4b367fc640 1 --2- 172.21.15.88:0/1290574210 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b4015e7a0 0x7f4b2c005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:25.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.421+0000 7f4b367fc640 1 -- 172.21.15.88:0/1290574210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b2c0039e0 msgr2=0x7f4b2c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:25.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.421+0000 7f4b367fc640 1 --2- 172.21.15.88:0/1290574210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b2c0039e0 0x7f4b2c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:25.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.421+0000 7f4b367fc640 1 -- 172.21.15.88:0/1290574210 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b2c003520 con 0x7f4b2c004390 2024-09-16T01:43:25.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.422+0000 7f4b367fc640 1 --2- 172.21.15.88:0/1290574210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b2c004390 0x7f4b2c004860 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f4b28001830 tx=0x7f4b2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83ea9543ac3302de server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:25.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.422+0000 7f4b3e7fc640 1 -- 172.21.15.88:0/1290574210 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b28037070 con 0x7f4b2c004390 2024-09-16T01:43:25.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.422+0000 7f4b3e7fc640 1 -- 172.21.15.88:0/1290574210 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b2802abe0 con 0x7f4b2c004390 2024-09-16T01:43:25.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.422+0000 7f4b3e7fc640 1 -- 172.21.15.88:0/1290574210 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b28001430 con 0x7f4b2c004390 2024-09-16T01:43:25.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.423+0000 7f4b455af640 1 -- 172.21.15.88:0/1290574210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b2c004390 msgr2=0x7f4b2c004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:25.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.423+0000 7f4b455af640 1 --2- 172.21.15.88:0/1290574210 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b2c004390 0x7f4b2c004860 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f4b28001830 tx=0x7f4b2c005830 comp rx=0 tx=0).stop 2024-09-16T01:43:25.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.423+0000 7f4b3f7fe640 1 -- 172.21.15.88:0/1290574210 reap_dead start 2024-09-16T01:43:25.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.423+0000 7f4b455af640 1 -- 172.21.15.88:0/1290574210 shutdown_connections 2024-09-16T01:43:25.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.423+0000 7f4b455af640 1 -- 172.21.15.88:0/1290574210 >> 172.21.15.88:0/1290574210 conn(0x7f4b400d11e0 msgr2=0x7f4b400d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:25.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.423+0000 7f4b455af640 1 -- 172.21.15.88:0/1290574210 shutdown_connections 2024-09-16T01:43:25.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.423+0000 7f4b455af640 1 -- 172.21.15.88:0/1290574210 wait complete. 2024-09-16T01:43:25.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.424+0000 7f4b455af640 1 Processor -- start 2024-09-16T01:43:25.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.424+0000 7f4b455af640 1 -- start start 2024-09-16T01:43:25.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.425+0000 7f4b455af640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b2c004390 0x7f4b40161bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:25.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.425+0000 7f4b455af640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b401f4c30 0x7f4b4015fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:25.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.425+0000 7f4b455af640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b401517c0 0x7f4b40151c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:25.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.425+0000 7f4b455af640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4015eb80 con 0x7f4b401f4c30 2024-09-16T01:43:25.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.425+0000 7f4b455af640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4b400b7a00 con 0x7f4b2c004390 2024-09-16T01:43:25.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.425+0000 7f4b455af640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4b4015ed00 con 0x7f4b401517c0 2024-09-16T01:43:25.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.425+0000 7f4b367fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b401f4c30 0x7f4b4015fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:25.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.425+0000 7f4b367fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b401f4c30 0x7f4b4015fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43592/0 (socket says 172.21.15.88:43592) 2024-09-16T01:43:25.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.425+0000 7f4b367fc640 1 -- 172.21.15.88:0/2689594065 learned_addr learned my addr 172.21.15.88:0/2689594065 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:25.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.425+0000 7f4b3effd640 1 --2- 172.21.15.88:0/2689594065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b2c004390 0x7f4b40161bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:25.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.425+0000 7f4b3f7fe640 1 --2- 172.21.15.88:0/2689594065 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b401517c0 0x7f4b40151c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:25.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.425+0000 7f4b367fc640 1 -- 172.21.15.88:0/2689594065 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b401517c0 msgr2=0x7f4b40151c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:25.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.425+0000 7f4b367fc640 1 --2- 172.21.15.88:0/2689594065 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b401517c0 0x7f4b40151c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:25.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.425+0000 7f4b367fc640 1 -- 172.21.15.88:0/2689594065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b2c004390 msgr2=0x7f4b40161bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:25.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.425+0000 7f4b367fc640 1 --2- 172.21.15.88:0/2689594065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b2c004390 0x7f4b40161bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:25.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.425+0000 7f4b367fc640 1 -- 172.21.15.88:0/2689594065 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b40162110 con 0x7f4b401f4c30 2024-09-16T01:43:25.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.425+0000 7f4b3f7fe640 1 --2- 172.21.15.88:0/2689594065 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b401517c0 0x7f4b40151c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:25.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.425+0000 7f4b367fc640 1 --2- 172.21.15.88:0/2689594065 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b401f4c30 0x7f4b4015fac0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f4b2802a670 tx=0x7f4b28001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:25.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.425+0000 7f4b37fff640 1 -- 172.21.15.88:0/2689594065 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2803f020 con 0x7f4b401f4c30 2024-09-16T01:43:25.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.425+0000 7f4b37fff640 1 -- 172.21.15.88:0/2689594065 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b28005270 con 0x7f4b401f4c30 2024-09-16T01:43:25.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.425+0000 7f4b37fff640 1 -- 172.21.15.88:0/2689594065 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b28037040 con 0x7f4b401f4c30 2024-09-16T01:43:25.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.425+0000 7f4b455af640 1 -- 172.21.15.88:0/2689594065 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b401f5530 con 0x7f4b401f4c30 2024-09-16T01:43:25.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.426+0000 7f4b455af640 1 -- 172.21.15.88:0/2689594065 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b401f5af0 con 0x7f4b401f4c30 2024-09-16T01:43:25.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.426+0000 7f4b455af640 1 -- 172.21.15.88:0/2689594065 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b400b61e0 con 0x7f4b401f4c30 2024-09-16T01:43:25.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.427+0000 7f4b37fff640 1 -- 172.21.15.88:0/2689594065 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4b28042020 con 0x7f4b401f4c30 2024-09-16T01:43:25.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.427+0000 7f4b37fff640 1 --2- 172.21.15.88:0/2689594065 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4b2006c2e0 0x7f4b2006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:25.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.427+0000 7f4b37fff640 1 -- 172.21.15.88:0/2689594065 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 8404+0+0 (secure 0 0 0) 0x7f4b280abd30 con 0x7f4b401f4c30 2024-09-16T01:43:25.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.427+0000 7f4b3effd640 1 --2- 172.21.15.88:0/2689594065 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4b2006c2e0 0x7f4b2006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:25.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.428+0000 7f4b3effd640 1 --2- 172.21.15.88:0/2689594065 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4b2006c2e0 0x7f4b2006e7a0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f4b4015d4f0 tx=0x7f4b30009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:25.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.432+0000 7f4b37fff640 1 -- 172.21.15.88:0/2689594065 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4b28049030 con 0x7f4b401f4c30 2024-09-16T01:43:25.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.634+0000 7f4b455af640 1 -- 172.21.15.88:0/2689594065 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f4b400c7be0 con 0x7f4b401f4c30 2024-09-16T01:43:25.636 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:43:25.635+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:43:25.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.637+0000 7f4b37fff640 1 -- 172.21.15.88:0/2689594065 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v260) ==== 161+0+0 (secure 0 0 0) 0x7f4b280799c0 con 0x7f4b401f4c30 2024-09-16T01:43:25.638 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:43:25.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.639+0000 7f4b455af640 1 -- 172.21.15.88:0/2689594065 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4b2006c2e0 msgr2=0x7f4b2006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:25.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.639+0000 7f4b455af640 1 --2- 172.21.15.88:0/2689594065 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4b2006c2e0 0x7f4b2006e7a0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f4b4015d4f0 tx=0x7f4b30009190 comp rx=0 tx=0).stop 2024-09-16T01:43:25.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.639+0000 7f4b455af640 1 -- 172.21.15.88:0/2689594065 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b401f4c30 msgr2=0x7f4b4015fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:25.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.639+0000 7f4b455af640 1 --2- 172.21.15.88:0/2689594065 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b401f4c30 0x7f4b4015fac0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f4b2802a670 tx=0x7f4b28001230 comp rx=0 tx=0).stop 2024-09-16T01:43:25.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.639+0000 7f4b455af640 1 -- 172.21.15.88:0/2689594065 shutdown_connections 2024-09-16T01:43:25.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.639+0000 7f4b455af640 1 --2- 172.21.15.88:0/2689594065 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4b2006c2e0 0x7f4b2006e7a0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:25.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.639+0000 7f4b455af640 1 --2- 172.21.15.88:0/2689594065 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b401517c0 0x7f4b40151c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:25.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.639+0000 7f4b455af640 1 --2- 172.21.15.88:0/2689594065 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b401f4c30 0x7f4b4015fac0 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:25.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.639+0000 7f4b455af640 1 --2- 172.21.15.88:0/2689594065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b2c004390 0x7f4b40161bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:25.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.639+0000 7f4b455af640 1 -- 172.21.15.88:0/2689594065 >> 172.21.15.88:0/2689594065 conn(0x7f4b400d11e0 msgr2=0x7f4b400b89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:25.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.639+0000 7f4b455af640 1 -- 172.21.15.88:0/2689594065 shutdown_connections 2024-09-16T01:43:25.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.639+0000 7f4b455af640 1 -- 172.21.15.88:0/2689594065 wait complete. 2024-09-16T01:43:25.642 INFO:tasks.ceph.mds.a.smithi088.stderr: -542> 2024-09-16T01:43:23.458+0000 7fa851083640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:43:25.652 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:43:25.666 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:43:25.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.751+0000 7f86126e4640 1 Processor -- start 2024-09-16T01:43:25.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.751+0000 7f86126e4640 1 -- start start 2024-09-16T01:43:25.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.752+0000 7f86126e4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f860c15b480 0x7f860c0b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:25.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.752+0000 7f86126e4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f860c072700 0x7f860c15bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:25.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.752+0000 7f86126e4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f860c0ccff0 con 0x7f860c0b60c0 2024-09-16T01:43:25.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.752+0000 7f86126e4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f860c0cd2d0 con 0x7f860c072700 2024-09-16T01:43:25.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.752+0000 7f86126e4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f860c0cd5b0 con 0x7f860c15b480 2024-09-16T01:43:25.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.752+0000 7f8610c81640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f860c072700 0x7f860c15bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:25.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.752+0000 7f8610c81640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f860c072700 0x7f860c15bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43608/0 (socket says 172.21.15.88:43608) 2024-09-16T01:43:25.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.752+0000 7f8610c81640 1 -- 172.21.15.88:0/4184944226 learned_addr learned my addr 172.21.15.88:0/4184944226 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:25.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.752+0000 7f8610c81640 1 -- 172.21.15.88:0/4184944226 >> v2:172.21.15.154:3300/0 conn(0x7f860c15b480 msgr2=0x7f860c0b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:25.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.752+0000 7f8610c81640 1 --2- 172.21.15.88:0/4184944226 >> v2:172.21.15.154:3300/0 conn(0x7f860c15b480 0x7f860c0b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:25.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.752+0000 7f8610c81640 1 -- 172.21.15.88:0/4184944226 >> v1:172.21.15.154:6789/0 conn(0x7f860c0b60c0 legacy=0x7f860c071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:25.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.752+0000 7f8610c81640 1 -- 172.21.15.88:0/4184944226 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860c0cd890 con 0x7f860c072700 2024-09-16T01:43:25.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.753+0000 7f8610c81640 1 --2- 172.21.15.88:0/4184944226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f860c072700 0x7f860c15bc60 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f860c0694d0 tx=0x7f860402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbd013d764506458 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:25.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.753+0000 7f860affd640 1 -- 172.21.15.88:0/4184944226 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8604036070 con 0x7f860c072700 2024-09-16T01:43:25.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.753+0000 7f860affd640 1 -- 172.21.15.88:0/4184944226 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8604031600 con 0x7f860c072700 2024-09-16T01:43:25.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.753+0000 7f860affd640 1 -- 172.21.15.88:0/4184944226 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86040319d0 con 0x7f860c072700 2024-09-16T01:43:25.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.753+0000 7f86126e4640 1 -- 172.21.15.88:0/4184944226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f860c072700 msgr2=0x7f860c15bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:25.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.753+0000 7f86126e4640 1 --2- 172.21.15.88:0/4184944226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f860c072700 0x7f860c15bc60 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f860c0694d0 tx=0x7f860402a660 comp rx=0 tx=0).stop 2024-09-16T01:43:25.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.753+0000 7f86126e4640 1 -- 172.21.15.88:0/4184944226 shutdown_connections 2024-09-16T01:43:25.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.753+0000 7f86126e4640 1 --2- 172.21.15.88:0/4184944226 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f860c072700 0x7f860c15bc60 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:25.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.753+0000 7f86126e4640 1 --2- 172.21.15.88:0/4184944226 >> v2:172.21.15.154:3300/0 conn(0x7f860c15b480 0x7f860c0b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:25.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.753+0000 7f86126e4640 1 -- 172.21.15.88:0/4184944226 >> 172.21.15.88:0/4184944226 conn(0x7f860c06d0f0 msgr2=0x7f860c06d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:25.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.753+0000 7f86126e4640 1 -- 172.21.15.88:0/4184944226 shutdown_connections 2024-09-16T01:43:25.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.754+0000 7f86126e4640 1 -- 172.21.15.88:0/4184944226 wait complete. 2024-09-16T01:43:25.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.755+0000 7f86126e4640 1 Processor -- start 2024-09-16T01:43:25.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.755+0000 7f86126e4640 1 -- start start 2024-09-16T01:43:25.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.755+0000 7f86126e4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f860c072700 0x7f860c143ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:25.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.756+0000 7f86126e4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f860c0b60c0 0x7f860c153b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:25.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.756+0000 7f86126e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f860c15b480 0x7f860c1e4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:25.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.756+0000 7f86126e4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f860c15b9e0 con 0x7f860c072700 2024-09-16T01:43:25.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.756+0000 7f86126e4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f860c15b860 con 0x7f860c0b60c0 2024-09-16T01:43:25.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.756+0000 7f86126e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f860c15c1a0 con 0x7f860c15b480 2024-09-16T01:43:25.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.756+0000 7f860bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f860c072700 0x7f860c143ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:25.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.756+0000 7f860bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f860c072700 0x7f860c143ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43612/0 (socket says 172.21.15.88:43612) 2024-09-16T01:43:25.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.756+0000 7f860bfff640 1 -- 172.21.15.88:0/4125088716 learned_addr learned my addr 172.21.15.88:0/4125088716 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:25.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.756+0000 7f860b7fe640 1 --2- 172.21.15.88:0/4125088716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f860c0b60c0 0x7f860c153b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:25.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.756+0000 7f8610c81640 1 --2- 172.21.15.88:0/4125088716 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f860c15b480 0x7f860c1e4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:25.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.756+0000 7f860bfff640 1 -- 172.21.15.88:0/4125088716 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f860c15b480 msgr2=0x7f860c1e4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:25.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.756+0000 7f860bfff640 1 --2- 172.21.15.88:0/4125088716 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f860c15b480 0x7f860c1e4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:25.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.756+0000 7f860bfff640 1 -- 172.21.15.88:0/4125088716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f860c0b60c0 msgr2=0x7f860c153b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:25.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.756+0000 7f860bfff640 1 --2- 172.21.15.88:0/4125088716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f860c0b60c0 0x7f860c153b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:25.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.756+0000 7f860bfff640 1 -- 172.21.15.88:0/4125088716 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860c154ba0 con 0x7f860c072700 2024-09-16T01:43:25.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.756+0000 7f860b7fe640 1 --2- 172.21.15.88:0/4125088716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f860c0b60c0 0x7f860c153b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:25.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.756+0000 7f860bfff640 1 --2- 172.21.15.88:0/4125088716 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f860c072700 0x7f860c143ea0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f85fc00b410 tx=0x7f85fc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:25.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.756+0000 7f8608ff9640 1 -- 172.21.15.88:0/4125088716 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85fc015020 con 0x7f860c072700 2024-09-16T01:43:25.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.756+0000 7f8608ff9640 1 -- 172.21.15.88:0/4125088716 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85fc011850 con 0x7f860c072700 2024-09-16T01:43:25.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.756+0000 7f86126e4640 1 -- 172.21.15.88:0/4125088716 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f860c1545e0 con 0x7f860c072700 2024-09-16T01:43:25.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.756+0000 7f8608ff9640 1 -- 172.21.15.88:0/4125088716 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85fc011c40 con 0x7f860c072700 2024-09-16T01:43:25.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.756+0000 7f86126e4640 1 -- 172.21.15.88:0/4125088716 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f860c1e3e20 con 0x7f860c072700 2024-09-16T01:43:25.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.757+0000 7f86126e4640 1 -- 172.21.15.88:0/4125088716 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f860c15bc60 con 0x7f860c072700 2024-09-16T01:43:25.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.758+0000 7f8608ff9640 1 -- 172.21.15.88:0/4125088716 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f85fc018020 con 0x7f860c072700 2024-09-16T01:43:25.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.758+0000 7f8608ff9640 1 --2- 172.21.15.88:0/4125088716 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f85f806c2e0 0x7f85f806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:25.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.758+0000 7f860b7fe640 1 --2- 172.21.15.88:0/4125088716 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f85f806c2e0 0x7f85f806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:25.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.758+0000 7f8608ff9640 1 -- 172.21.15.88:0/4125088716 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8490+0+0 (secure 0 0 0) 0x7f85fc08be50 con 0x7f860c072700 2024-09-16T01:43:25.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.759+0000 7f860b7fe640 1 --2- 172.21.15.88:0/4125088716 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f85f806c2e0 0x7f85f806e7a0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f860c1f3950 tx=0x7f8600009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:25.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.762+0000 7f8608ff9640 1 -- 172.21.15.88:0/4125088716 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f85fc01f020 con 0x7f860c072700 2024-09-16T01:43:25.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:25.995+0000 7f86126e4640 1 -- 172.21.15.88:0/4125088716 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f860c1572e0 con 0x7f860c072700 2024-09-16T01:43:26.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.638+0000 7f8608ff9640 1 -- 172.21.15.88:0/4125088716 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v261) ==== 107+0+0 (secure 0 0 0) 0x7f85fc0172c0 con 0x7f860c072700 2024-09-16T01:43:26.640 INFO:teuthology.orchestra.run.smithi088.stderr:nothing to do: rank is not damaged 2024-09-16T01:43:26.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.641+0000 7f86126e4640 1 -- 172.21.15.88:0/4125088716 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f85f806c2e0 msgr2=0x7f85f806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:26.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.641+0000 7f86126e4640 1 --2- 172.21.15.88:0/4125088716 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f85f806c2e0 0x7f85f806e7a0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f860c1f3950 tx=0x7f8600009040 comp rx=0 tx=0).stop 2024-09-16T01:43:26.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.641+0000 7f86126e4640 1 -- 172.21.15.88:0/4125088716 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f860c072700 msgr2=0x7f860c143ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:26.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.641+0000 7f86126e4640 1 --2- 172.21.15.88:0/4125088716 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f860c072700 0x7f860c143ea0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f85fc00b410 tx=0x7f85fc00b8d0 comp rx=0 tx=0).stop 2024-09-16T01:43:26.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.641+0000 7f86126e4640 1 -- 172.21.15.88:0/4125088716 shutdown_connections 2024-09-16T01:43:26.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.641+0000 7f86126e4640 1 --2- 172.21.15.88:0/4125088716 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f85f806c2e0 0x7f85f806e7a0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:26.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.641+0000 7f86126e4640 1 --2- 172.21.15.88:0/4125088716 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f860c15b480 0x7f860c1e4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:26.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.641+0000 7f86126e4640 1 --2- 172.21.15.88:0/4125088716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f860c0b60c0 0x7f860c153b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:26.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.641+0000 7f86126e4640 1 --2- 172.21.15.88:0/4125088716 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f860c072700 0x7f860c143ea0 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:26.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.641+0000 7f86126e4640 1 -- 172.21.15.88:0/4125088716 >> 172.21.15.88:0/4125088716 conn(0x7f860c06d0f0 msgr2=0x7f860c071c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:26.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.641+0000 7f86126e4640 1 -- 172.21.15.88:0/4125088716 shutdown_connections 2024-09-16T01:43:26.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.641+0000 7f86126e4640 1 -- 172.21.15.88:0/4125088716 wait complete. 2024-09-16T01:43:26.657 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:43:26.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.743+0000 7f6170fb6f40 1 Processor -- start 2024-09-16T01:43:26.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.743+0000 7f6170fb6f40 1 -- start start 2024-09-16T01:43:26.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.744+0000 7f6170fb6f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f615c0022a0 0x5617b64513e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:26.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.744+0000 7f6170fb6f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617b6462570 0x5617b6456100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:26.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.744+0000 7f6170fb6f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5617b645bf60 0x5617b6456640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:26.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.744+0000 7f6170fb6f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5617b62d9510 con 0x7f615c0022a0 2024-09-16T01:43:26.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.744+0000 7f6170fb6f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5617b63c7b80 con 0x5617b6462570 2024-09-16T01:43:26.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.744+0000 7f6170fb6f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5617b645c3d0 con 0x5617b645bf60 2024-09-16T01:43:26.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.744+0000 7f616eaac640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f615c0022a0 0x5617b64513e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:26.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.744+0000 7f616eaac640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f615c0022a0 0x5617b64513e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43630/0 (socket says 172.21.15.88:43630) 2024-09-16T01:43:26.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.744+0000 7f616eaac640 1 -- 172.21.15.88:0/999325367 learned_addr learned my addr 172.21.15.88:0/999325367 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:26.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.744+0000 7f61662ab640 1 --2- 172.21.15.88:0/999325367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617b6462570 0x5617b6456100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:26.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.744+0000 7f616eaac640 1 -- 172.21.15.88:0/999325367 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5617b645bf60 msgr2=0x5617b6456640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:26.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.744+0000 7f616eaac640 1 --2- 172.21.15.88:0/999325367 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5617b645bf60 0x5617b6456640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:26.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.744+0000 7f616eaac640 1 -- 172.21.15.88:0/999325367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617b6462570 msgr2=0x5617b6456100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:26.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.744+0000 7f616eaac640 1 --2- 172.21.15.88:0/999325367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617b6462570 0x5617b6456100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:26.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.744+0000 7f616eaac640 1 -- 172.21.15.88:0/999325367 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617b63c78a0 con 0x7f615c0022a0 2024-09-16T01:43:26.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.744+0000 7f61662ab640 1 --2- 172.21.15.88:0/999325367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617b6462570 0x5617b6456100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:26.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.744+0000 7f616eaac640 1 --2- 172.21.15.88:0/999325367 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f615c0022a0 0x5617b64513e0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f6160045c70 tx=0x7f6160051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5c06e11e1f6835c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:26.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.744+0000 7f616daaa640 1 -- 172.21.15.88:0/999325367 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f616004f860 con 0x7f615c0022a0 2024-09-16T01:43:26.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.744+0000 7f616daaa640 1 -- 172.21.15.88:0/999325367 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f616004fa00 con 0x7f615c0022a0 2024-09-16T01:43:26.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.744+0000 7f616daaa640 1 -- 172.21.15.88:0/999325367 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f616004fdf0 con 0x7f615c0022a0 2024-09-16T01:43:26.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.745+0000 7f6170fb6f40 1 -- 172.21.15.88:0/999325367 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f615c0022a0 msgr2=0x5617b64513e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:26.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.745+0000 7f6170fb6f40 1 --2- 172.21.15.88:0/999325367 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f615c0022a0 0x5617b64513e0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f6160045c70 tx=0x7f6160051000 comp rx=0 tx=0).stop 2024-09-16T01:43:26.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.745+0000 7f6170fb6f40 1 -- 172.21.15.88:0/999325367 shutdown_connections 2024-09-16T01:43:26.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.745+0000 7f6170fb6f40 1 --2- 172.21.15.88:0/999325367 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5617b645bf60 0x5617b6456640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:26.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.745+0000 7f6170fb6f40 1 --2- 172.21.15.88:0/999325367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5617b6462570 0x5617b6456100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:26.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.745+0000 7f6170fb6f40 1 --2- 172.21.15.88:0/999325367 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f615c0022a0 0x5617b64513e0 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:26.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.745+0000 7f6170fb6f40 1 -- 172.21.15.88:0/999325367 >> 172.21.15.88:0/999325367 conn(0x5617b63d2750 msgr2=0x5617b6461e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:26.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.745+0000 7f6170fb6f40 1 -- 172.21.15.88:0/999325367 shutdown_connections 2024-09-16T01:43:26.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.745+0000 7f6170fb6f40 1 -- 172.21.15.88:0/999325367 wait complete. 2024-09-16T01:43:26.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.745+0000 7f6170fb6f40 1 Processor -- start 2024-09-16T01:43:26.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.745+0000 7f6170fb6f40 1 -- start start 2024-09-16T01:43:26.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.746+0000 7f6170fb6f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b645bf60 0x5617b644c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:26.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.746+0000 7f6170fb6f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5617b6462570 0x5617b64502e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:26.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.746+0000 7f6170fb6f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f615c0022a0 0x5617b6450820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:26.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.746+0000 7f6170fb6f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5617b6451c90 con 0x5617b645bf60 2024-09-16T01:43:26.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.746+0000 7f6170fb6f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5617b645c900 con 0x7f615c0022a0 2024-09-16T01:43:26.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.746+0000 7f6170fb6f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5617b645c620 con 0x5617b6462570 2024-09-16T01:43:26.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.746+0000 7f616eaac640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b645bf60 0x5617b644c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:26.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.746+0000 7f616eaac640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b645bf60 0x5617b644c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43634/0 (socket says 172.21.15.88:43634) 2024-09-16T01:43:26.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.746+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 learned_addr learned my addr 172.21.15.88:0/2955094044 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:26.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.746+0000 7f616f2ad640 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f615c0022a0 0x5617b6450820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:26.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.746+0000 7f61662ab640 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5617b6462570 0x5617b64502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:26.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.746+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5617b6462570 msgr2=0x5617b64502e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:26.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.746+0000 7f616eaac640 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5617b6462570 0x5617b64502e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:26.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.746+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f615c0022a0 msgr2=0x5617b6450820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:26.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.746+0000 7f616eaac640 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f615c0022a0 0x5617b6450820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:26.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.746+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617b644c840 con 0x5617b645bf60 2024-09-16T01:43:26.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.747+0000 7f616f2ad640 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f615c0022a0 0x5617b6450820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:26.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.747+0000 7f61662ab640 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5617b6462570 0x5617b64502e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:26.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.747+0000 7f616eaac640 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b645bf60 0x5617b644c300 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7f6160046160 tx=0x7f6160003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:26.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.747+0000 7f616caa8640 1 -- 172.21.15.88:0/2955094044 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6160051c50 con 0x5617b645bf60 2024-09-16T01:43:26.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.747+0000 7f616caa8640 1 -- 172.21.15.88:0/2955094044 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f616006c070 con 0x5617b645bf60 2024-09-16T01:43:26.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.747+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5617b645dc30 con 0x5617b645bf60 2024-09-16T01:43:26.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.747+0000 7f616caa8640 1 -- 172.21.15.88:0/2955094044 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f616004fbb0 con 0x5617b645bf60 2024-09-16T01:43:26.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.747+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617b645e1f0 con 0x5617b645bf60 2024-09-16T01:43:26.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.748+0000 7f616caa8640 1 -- 172.21.15.88:0/2955094044 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6160051110 con 0x5617b645bf60 2024-09-16T01:43:26.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.749+0000 7f616caa8640 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f615006acc0 0x7f615006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:26.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.749+0000 7f616caa8640 1 -- 172.21.15.88:0/2955094044 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8490+0+0 (secure 0 0 0) 0x7f61600ceba0 con 0x5617b645bf60 2024-09-16T01:43:26.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.749+0000 7f61662ab640 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f615006acc0 0x7f615006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:26.756 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:43:26.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.749+0000 7f6170fb6f40 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5617b64ee240 0x5617b64ee690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:26.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.749+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b64eebd0 con 0x5617b64ee240 2024-09-16T01:43:26.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.749+0000 7f616f2ad640 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5617b64ee240 0x5617b64ee690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:26.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.749+0000 7f61662ab640 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f615006acc0 0x7f615006d180 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x5617b645f1e0 tx=0x7f615800c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:26.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.750+0000 7f616f2ad640 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5617b64ee240 0x5617b64ee690 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:26.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.752+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v119'971 uv944 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.752+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e119) -- 0x5617b64eefe0 con 0x5617b64ee240 2024-09-16T01:43:26.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.755+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v119'972 uv972 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.755+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6502d30 con 0x5617b64ee240 2024-09-16T01:43:26.759 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:43:26.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.758+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v119'973 uv973 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.758+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65040e0 con 0x5617b64ee240 2024-09-16T01:43:26.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.760+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v119'974 uv974 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.760+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6504480 con 0x5617b64ee240 2024-09-16T01:43:26.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.761+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v119'975 uv975 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.761+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65030d0 con 0x5617b64ee240 2024-09-16T01:43:26.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.762+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v119'976 uv976 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.763+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503470 con 0x5617b64ee240 2024-09-16T01:43:26.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.764+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v119'977 uv950 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.764+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503810 con 0x5617b64ee240 2024-09-16T01:43:26.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.766+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v119'978 uv978 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.766+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503810 con 0x5617b64ee240 2024-09-16T01:43:26.768 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:43:26.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.767+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v119'979 uv979 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.767+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503810 con 0x5617b64ee240 2024-09-16T01:43:26.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.768+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v119'980 uv980 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.768+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503810 con 0x5617b64ee240 2024-09-16T01:43:26.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.770+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v119'981 uv981 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.770+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503810 con 0x5617b64ee240 2024-09-16T01:43:26.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.771+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v119'982 uv982 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.771+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503810 con 0x5617b64ee240 2024-09-16T01:43:26.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.772+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v119'983 uv970 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.772+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503810 con 0x5617b64ee240 2024-09-16T01:43:26.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.774+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v119'984 uv984 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.774+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503810 con 0x5617b64ee240 2024-09-16T01:43:26.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.775+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v119'985 uv985 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.776 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:43:26.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.775+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503810 con 0x5617b64ee240 2024-09-16T01:43:26.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.777+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v119'986 uv986 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.777+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503810 con 0x5617b64ee240 2024-09-16T01:43:26.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.778+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v119'987 uv987 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.778+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503810 con 0x5617b64ee240 2024-09-16T01:43:26.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.779+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v119'988 uv988 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.779+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503bf0 con 0x5617b64ee240 2024-09-16T01:43:26.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.780+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v119'989 uv989 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.781+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503ac0 con 0x5617b64ee240 2024-09-16T01:43:26.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.783+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v119'990 uv963 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.783+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503ac0 con 0x5617b64ee240 2024-09-16T01:43:26.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.784+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v119'991 uv991 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.784+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503ac0 con 0x5617b64ee240 2024-09-16T01:43:26.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.785+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v119'992 uv992 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.787 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:43:26.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.785+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503ac0 con 0x5617b64ee240 2024-09-16T01:43:26.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.787+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v119'993 uv993 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.787+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503ac0 con 0x5617b64ee240 2024-09-16T01:43:26.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.788+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v119'994 uv994 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.788+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503ac0 con 0x5617b64ee240 2024-09-16T01:43:26.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.789+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v119'995 uv995 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.789+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503ac0 con 0x5617b64ee240 2024-09-16T01:43:26.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.790+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v119'996 uv969 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.790+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503ac0 con 0x5617b64ee240 2024-09-16T01:43:26.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.792+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v119'997 uv997 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.792+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503ac0 con 0x5617b64ee240 2024-09-16T01:43:26.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.793+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v119'998 uv998 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.794 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:43:26.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.793+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503ac0 con 0x5617b64ee240 2024-09-16T01:43:26.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.794+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v119'999 uv999 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.794+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503ac0 con 0x5617b64ee240 2024-09-16T01:43:26.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.795+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v119'1000 uv1000 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.795+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503ac0 con 0x5617b64ee240 2024-09-16T01:43:26.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.797+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v119'1001 uv1001 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.797+0000 7f6170fb6f40 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5617b6504d30 0x5617b6507120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:26.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.797+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6503ac0 con 0x5617b6504d30 2024-09-16T01:43:26.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.797+0000 7f616eaac640 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5617b6504d30 0x5617b6507120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:26.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.797+0000 7f616eaac640 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5617b6504d30 0x5617b6507120 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:26.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.799+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v119'706 uv685 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f6160050d60 con 0x5617b6504d30 2024-09-16T01:43:26.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.799+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e119) -- 0x5617b6507660 con 0x5617b6504d30 2024-09-16T01:43:26.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.800+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v119'707 uv707 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6160098890 con 0x5617b6504d30 2024-09-16T01:43:26.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.800+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6507a00 con 0x5617b6504d30 2024-09-16T01:43:26.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.802+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v119'708 uv708 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6160098890 con 0x5617b6504d30 2024-09-16T01:43:26.803 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:43:26.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.802+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6508250 con 0x5617b6504d30 2024-09-16T01:43:26.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.803+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v119'709 uv709 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6160098890 con 0x5617b6504d30 2024-09-16T01:43:26.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.803+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65085f0 con 0x5617b6504d30 2024-09-16T01:43:26.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.804+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v119'710 uv710 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6160098890 con 0x5617b6504d30 2024-09-16T01:43:26.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.804+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6508990 con 0x5617b6504d30 2024-09-16T01:43:26.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.805+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v119'711 uv704 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6160098890 con 0x5617b6504d30 2024-09-16T01:43:26.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.805+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e119) -- 0x5617b6508d30 con 0x5617b6504d30 2024-09-16T01:43:26.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.807+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v119'712 uv712 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6160098890 con 0x5617b6504d30 2024-09-16T01:43:26.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.807+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6508d30 con 0x5617b6504d30 2024-09-16T01:43:26.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.808+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v119'713 uv713 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6160098890 con 0x5617b6504d30 2024-09-16T01:43:26.809 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:43:26.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.808+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6508d30 con 0x5617b6504d30 2024-09-16T01:43:26.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.809+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v119'714 uv714 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6160098890 con 0x5617b6504d30 2024-09-16T01:43:26.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.809+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6508d30 con 0x5617b6504d30 2024-09-16T01:43:26.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.810+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v119'715 uv715 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6160098890 con 0x5617b6504d30 2024-09-16T01:43:26.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.810+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6508d30 con 0x5617b6504d30 2024-09-16T01:43:26.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.812+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v119'716 uv705 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6160098890 con 0x5617b6504d30 2024-09-16T01:43:26.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.812+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e119) -- 0x5617b6508d30 con 0x5617b6504d30 2024-09-16T01:43:26.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.813+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v119'717 uv717 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6160098890 con 0x5617b6504d30 2024-09-16T01:43:26.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.813+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6508d30 con 0x5617b6504d30 2024-09-16T01:43:26.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.815+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v119'718 uv718 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6160098890 con 0x5617b6504d30 2024-09-16T01:43:26.816 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:43:26.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.815+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b650d1b0 con 0x5617b6504d30 2024-09-16T01:43:26.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.816+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v119'719 uv719 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6160098890 con 0x5617b6504d30 2024-09-16T01:43:26.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.816+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b650d1b0 con 0x5617b6504d30 2024-09-16T01:43:26.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.817+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v119'720 uv720 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6160098890 con 0x5617b6504d30 2024-09-16T01:43:26.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.817+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6508d30 con 0x5617b6504d30 2024-09-16T01:43:26.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.819+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v119'721 uv701 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6160098890 con 0x5617b6504d30 2024-09-16T01:43:26.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.819+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e119) -- 0x5617b6508d30 con 0x5617b6504d30 2024-09-16T01:43:26.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.820+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v119'722 uv722 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6160098890 con 0x5617b6504d30 2024-09-16T01:43:26.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.820+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6508d30 con 0x5617b6504d30 2024-09-16T01:43:26.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.821+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v119'723 uv723 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6160098890 con 0x5617b6504d30 2024-09-16T01:43:26.823 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:43:26.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.821+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6509d40 con 0x5617b6504d30 2024-09-16T01:43:26.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.823+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v119'724 uv724 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6160098890 con 0x5617b6504d30 2024-09-16T01:43:26.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.823+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6509d40 con 0x5617b6504d30 2024-09-16T01:43:26.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.824+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v119'725 uv725 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6160098890 con 0x5617b6504d30 2024-09-16T01:43:26.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.824+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6508d30 con 0x5617b6504d30 2024-09-16T01:43:26.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.825+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v119'726 uv726 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6160098890 con 0x5617b6504d30 2024-09-16T01:43:26.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.825+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6508d30 con 0x5617b64ee240 2024-09-16T01:43:26.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.827+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v119'189 uv188 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.827+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e119) -- 0x5617b6508d30 con 0x5617b64ee240 2024-09-16T01:43:26.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.828+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v119'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.828+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6508d30 con 0x5617b64ee240 2024-09-16T01:43:26.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.829+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v119'191 uv191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.830 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:43:26.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.829+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6509d40 con 0x5617b64ee240 2024-09-16T01:43:26.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.830+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v119'192 uv192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.830+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6509d40 con 0x5617b64ee240 2024-09-16T01:43:26.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.831+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v119'193 uv193 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.831+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6508d30 con 0x5617b64ee240 2024-09-16T01:43:26.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.832+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v119'194 uv194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.832+0000 7f6170fb6f40 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5617b6509240 0x5617b650b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:26.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.832+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b650bb70 con 0x5617b6509240 2024-09-16T01:43:26.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.832+0000 7f61662ab640 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5617b6509240 0x5617b650b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:26.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.832+0000 7f61662ab640 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5617b6509240 0x5617b650b630 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:26.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.835+0000 7f61662ab640 1 -- 172.21.15.88:0/2955094044 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v119'142 uv141 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f6158016820 con 0x5617b6509240 2024-09-16T01:43:26.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.835+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e119) -- 0x5617b650bf10 con 0x5617b6509240 2024-09-16T01:43:26.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.836+0000 7f61662ab640 1 -- 172.21.15.88:0/2955094044 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v119'143 uv143 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6158020040 con 0x5617b6509240 2024-09-16T01:43:26.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.836+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b650c2b0 con 0x5617b6509240 2024-09-16T01:43:26.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.837+0000 7f61662ab640 1 -- 172.21.15.88:0/2955094044 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v119'144 uv144 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6158020040 con 0x5617b6509240 2024-09-16T01:43:26.839 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:43:26.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.837+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b650c650 con 0x5617b6509240 2024-09-16T01:43:26.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.839+0000 7f61662ab640 1 -- 172.21.15.88:0/2955094044 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v119'145 uv145 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f6158020040 con 0x5617b6509240 2024-09-16T01:43:26.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.839+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b650c9f0 con 0x5617b64ee240 2024-09-16T01:43:26.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.840+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v119'189 uv188 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.840+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e119) -- 0x5617b650c9f0 con 0x5617b64ee240 2024-09-16T01:43:26.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.841+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v119'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.841+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b650c9f0 con 0x5617b64ee240 2024-09-16T01:43:26.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.844+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v119'191 uv191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.845 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:43:26.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.844+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b650da00 con 0x5617b64ee240 2024-09-16T01:43:26.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.847+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v119'192 uv192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.847+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b650da00 con 0x5617b64ee240 2024-09-16T01:43:26.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.850+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v119'193 uv193 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.850+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b650c9f0 con 0x5617b64ee240 2024-09-16T01:43:26.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.851+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v119'194 uv194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.851+0000 7f6170fb6f40 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5617b650d070 0x5617b650f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:26.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.851+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b650fa80 con 0x5617b650d070 2024-09-16T01:43:26.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.851+0000 7f616f2ad640 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5617b650d070 0x5617b650f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:26.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.851+0000 7f616f2ad640 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5617b650d070 0x5617b650f460 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:26.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.853+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v119'349 uv343 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.853+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e119) -- 0x5617b650fe50 con 0x5617b650d070 2024-09-16T01:43:26.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.854+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v119'350 uv350 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.854+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b650fe50 con 0x5617b650d070 2024-09-16T01:43:26.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.855+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v119'351 uv351 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.856 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:43:26.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.855+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6510e60 con 0x5617b650d070 2024-09-16T01:43:26.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.856+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v119'352 uv352 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.856+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6510e60 con 0x5617b650d070 2024-09-16T01:43:26.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.857+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v119'353 uv353 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.857+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b650fe50 con 0x5617b650d070 2024-09-16T01:43:26.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.858+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v119'354 uv354 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.858+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b650fe50 con 0x5617b650d070 2024-09-16T01:43:26.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.860+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v119'355 uv348 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.860+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e119) -- 0x5617b650fe50 con 0x5617b650d070 2024-09-16T01:43:26.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.861+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v119'356 uv356 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.861+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b650fe50 con 0x5617b650d070 2024-09-16T01:43:26.863 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:43:26.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.862+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v119'357 uv357 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.862+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b650fe50 con 0x5617b650d070 2024-09-16T01:43:26.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.863+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v119'358 uv358 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.863+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b650fe50 con 0x5617b650d070 2024-09-16T01:43:26.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.864+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v119'359 uv359 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.864+0000 7f6170fb6f40 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5617b65104d0 0x5617b65128e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:26.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.864+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6512ff0 con 0x5617b65104d0 2024-09-16T01:43:26.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.864+0000 7f616eaac640 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5617b65104d0 0x5617b65128e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:26.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.864+0000 7f616eaac640 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5617b65104d0 0x5617b65128e0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:26.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.866+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v119'158 uv157 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f6160098890 con 0x5617b65104d0 2024-09-16T01:43:26.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.866+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e119) -- 0x5617b65133c0 con 0x5617b65104d0 2024-09-16T01:43:26.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.867+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v119'159 uv159 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6160098890 con 0x5617b65104d0 2024-09-16T01:43:26.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.867+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65133c0 con 0x5617b65104d0 2024-09-16T01:43:26.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.868+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v119'160 uv160 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6160098890 con 0x5617b65104d0 2024-09-16T01:43:26.869 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:43:26.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.868+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65133c0 con 0x5617b65104d0 2024-09-16T01:43:26.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.869+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v119'161 uv161 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6160098890 con 0x5617b65104d0 2024-09-16T01:43:26.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.869+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65133c0 con 0x5617b65104d0 2024-09-16T01:43:26.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.870+0000 7f616eaac640 1 -- 172.21.15.88:0/2955094044 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v119'162 uv162 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6160098890 con 0x5617b65104d0 2024-09-16T01:43:26.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.870+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65133c0 con 0x5617b64ee240 2024-09-16T01:43:26.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.871+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v119'189 uv188 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.871+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e119) -- 0x5617b65133c0 con 0x5617b64ee240 2024-09-16T01:43:26.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.872+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v119'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.872+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65133c0 con 0x5617b64ee240 2024-09-16T01:43:26.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.873+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v119'191 uv191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.874 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:43:26.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.873+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65143d0 con 0x5617b64ee240 2024-09-16T01:43:26.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.874+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v119'192 uv192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.874+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65143d0 con 0x5617b64ee240 2024-09-16T01:43:26.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.875+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v119'193 uv193 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.875+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65133c0 con 0x5617b64ee240 2024-09-16T01:43:26.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.876+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v119'194 uv194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b64ee240 2024-09-16T01:43:26.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.876+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65133c0 con 0x5617b6509240 2024-09-16T01:43:26.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.877+0000 7f61662ab640 1 -- 172.21.15.88:0/2955094044 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v119'314 uv308 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f6158020040 con 0x5617b6509240 2024-09-16T01:43:26.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.877+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e119) -- 0x5617b6513760 con 0x5617b6509240 2024-09-16T01:43:26.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.878+0000 7f61662ab640 1 -- 172.21.15.88:0/2955094044 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v119'315 uv315 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6158020040 con 0x5617b6509240 2024-09-16T01:43:26.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.878+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6513b50 con 0x5617b6509240 2024-09-16T01:43:26.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.879+0000 7f61662ab640 1 -- 172.21.15.88:0/2955094044 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v119'316 uv316 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6158020040 con 0x5617b6509240 2024-09-16T01:43:26.880 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:43:26.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.879+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65143a0 con 0x5617b6509240 2024-09-16T01:43:26.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.880+0000 7f61662ab640 1 -- 172.21.15.88:0/2955094044 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v119'317 uv317 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6158020040 con 0x5617b6509240 2024-09-16T01:43:26.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.880+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65143a0 con 0x5617b6509240 2024-09-16T01:43:26.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.881+0000 7f61662ab640 1 -- 172.21.15.88:0/2955094044 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v119'318 uv318 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f6158020040 con 0x5617b6509240 2024-09-16T01:43:26.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.881+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65143a0 con 0x5617b6509240 2024-09-16T01:43:26.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.882+0000 7f61662ab640 1 -- 172.21.15.88:0/2955094044 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v119'319 uv313 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6158020040 con 0x5617b6509240 2024-09-16T01:43:26.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.882+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e119) -- 0x5617b65143a0 con 0x5617b6509240 2024-09-16T01:43:26.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.883+0000 7f61662ab640 1 -- 172.21.15.88:0/2955094044 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v119'320 uv320 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6158020040 con 0x5617b6509240 2024-09-16T01:43:26.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.883+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6507e90 con 0x5617b6509240 2024-09-16T01:43:26.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.884+0000 7f61662ab640 1 -- 172.21.15.88:0/2955094044 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v119'321 uv321 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6158020040 con 0x5617b6509240 2024-09-16T01:43:26.885 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:43:26.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.884+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6507e90 con 0x5617b6509240 2024-09-16T01:43:26.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.885+0000 7f61662ab640 1 -- 172.21.15.88:0/2955094044 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v119'322 uv322 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6158020040 con 0x5617b6509240 2024-09-16T01:43:26.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.885+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6517110 con 0x5617b6509240 2024-09-16T01:43:26.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.886+0000 7f61662ab640 1 -- 172.21.15.88:0/2955094044 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v119'323 uv323 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6158020040 con 0x5617b6509240 2024-09-16T01:43:26.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.886+0000 7f6170fb6f40 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5617b6514cc0 0x5617b65170b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:26.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.886+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65175f0 con 0x5617b6514cc0 2024-09-16T01:43:26.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.886+0000 7f61662ab640 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5617b6514cc0 0x5617b65170b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:26.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.886+0000 7f61662ab640 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5617b6514cc0 0x5617b65170b0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:26.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.887+0000 7f61662ab640 1 -- 172.21.15.88:0/2955094044 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v119'137 uv135 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f6158020040 con 0x5617b6514cc0 2024-09-16T01:43:26.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.887+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e119) -- 0x5617b65175f0 con 0x5617b6514cc0 2024-09-16T01:43:26.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.888+0000 7f61662ab640 1 -- 172.21.15.88:0/2955094044 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v119'138 uv138 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f6158020040 con 0x5617b6514cc0 2024-09-16T01:43:26.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.888+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65175f0 con 0x5617b6514cc0 2024-09-16T01:43:26.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.889+0000 7f61662ab640 1 -- 172.21.15.88:0/2955094044 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v119'139 uv139 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f6158020040 con 0x5617b6514cc0 2024-09-16T01:43:26.890 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:43:26.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.889+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65175f0 con 0x5617b6514cc0 2024-09-16T01:43:26.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.890+0000 7f61662ab640 1 -- 172.21.15.88:0/2955094044 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v119'140 uv140 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f6158020040 con 0x5617b6514cc0 2024-09-16T01:43:26.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.890+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65175f0 con 0x5617b6509240 2024-09-16T01:43:26.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.891+0000 7f61662ab640 1 -- 172.21.15.88:0/2955094044 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v119'189 uv188 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f6158020040 con 0x5617b6509240 2024-09-16T01:43:26.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.891+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e119) -- 0x5617b65175f0 con 0x5617b6509240 2024-09-16T01:43:26.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.892+0000 7f61662ab640 1 -- 172.21.15.88:0/2955094044 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v119'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6158020040 con 0x5617b6509240 2024-09-16T01:43:26.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.892+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65175f0 con 0x5617b6509240 2024-09-16T01:43:26.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.893+0000 7f61662ab640 1 -- 172.21.15.88:0/2955094044 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v119'191 uv191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6158020040 con 0x5617b6509240 2024-09-16T01:43:26.895 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:43:26.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.893+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6518600 con 0x5617b6509240 2024-09-16T01:43:26.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.894+0000 7f61662ab640 1 -- 172.21.15.88:0/2955094044 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v119'192 uv192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6158020040 con 0x5617b6509240 2024-09-16T01:43:26.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.894+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6518600 con 0x5617b6509240 2024-09-16T01:43:26.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.895+0000 7f61662ab640 1 -- 172.21.15.88:0/2955094044 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v119'193 uv193 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6158020040 con 0x5617b6509240 2024-09-16T01:43:26.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.895+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65175f0 con 0x5617b6509240 2024-09-16T01:43:26.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.896+0000 7f61662ab640 1 -- 172.21.15.88:0/2955094044 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v119'194 uv194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f6158020040 con 0x5617b6509240 2024-09-16T01:43:26.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.896+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65175f0 con 0x5617b650d070 2024-09-16T01:43:26.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.898+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v119'161 uv160 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.898+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e119) -- 0x5617b65175f0 con 0x5617b650d070 2024-09-16T01:43:26.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.899+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v119'162 uv162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.899+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65175f0 con 0x5617b650d070 2024-09-16T01:43:26.901 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:43:26.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.900+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v119'163 uv163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.900+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65175f0 con 0x5617b650d070 2024-09-16T01:43:26.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.902+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v119'164 uv164 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.902+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65175f0 con 0x5617b650d070 2024-09-16T01:43:26.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.903+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v119'165 uv165 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.903+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65175f0 con 0x5617b650d070 2024-09-16T01:43:26.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.904+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v119'389 uv380 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.904+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e119) -- 0x5617b65175f0 con 0x5617b650d070 2024-09-16T01:43:26.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.905+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v119'390 uv390 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.905+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65175f0 con 0x5617b650d070 2024-09-16T01:43:26.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.906+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v119'391 uv391 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.907 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:43:26.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.906+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65175f0 con 0x5617b650d070 2024-09-16T01:43:26.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.907+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v119'392 uv392 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.907+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65175f0 con 0x5617b650d070 2024-09-16T01:43:26.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.908+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v119'393 uv393 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.908+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65175f0 con 0x5617b650d070 2024-09-16T01:43:26.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.909+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v119'394 uv387 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.909+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e119) -- 0x5617b65175f0 con 0x5617b650d070 2024-09-16T01:43:26.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.911+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v119'395 uv395 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.911+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65175f0 con 0x5617b650d070 2024-09-16T01:43:26.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.912+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v119'396 uv396 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.913 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:43:26.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.912+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6518600 con 0x5617b650d070 2024-09-16T01:43:26.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.913+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v119'397 uv397 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.913+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6518600 con 0x5617b650d070 2024-09-16T01:43:26.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.914+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v119'398 uv398 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.914+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b65175f0 con 0x5617b650d070 2024-09-16T01:43:26.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.915+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v119'399 uv399 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.915+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x5617b6517aa0 con 0x5617b650d070 2024-09-16T01:43:26.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.916+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v119'400 uv400 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f616800f1e0 con 0x5617b650d070 2024-09-16T01:43:26.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.916+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5617b64ee240 msgr2=0x5617b64ee690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:26.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.916+0000 7f6170fb6f40 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x5617b64ee240 0x5617b64ee690 crc :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:26.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.916+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5617b6509240 msgr2=0x5617b650b630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:26.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.916+0000 7f6170fb6f40 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x5617b6509240 0x5617b650b630 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:26.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.916+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5617b6514cc0 msgr2=0x5617b65170b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:26.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.916+0000 7f6170fb6f40 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x5617b6514cc0 0x5617b65170b0 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:26.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.916+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5617b6504d30 msgr2=0x5617b6507120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:26.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.916+0000 7f6170fb6f40 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x5617b6504d30 0x5617b6507120 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:26.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.916+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5617b650d070 msgr2=0x5617b650f460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:26.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.916+0000 7f6170fb6f40 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x5617b650d070 0x5617b650f460 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:26.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.916+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5617b65104d0 msgr2=0x5617b65128e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:26.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.916+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 reap_dead start 2024-09-16T01:43:26.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.916+0000 7f6170fb6f40 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x5617b65104d0 0x5617b65128e0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:26.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.916+0000 7f616f2ad640 1 -- 172.21.15.88:0/2955094044 reap_dead start 2024-09-16T01:43:26.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.916+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f615006acc0 msgr2=0x7f615006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:26.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.916+0000 7f6170fb6f40 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f615006acc0 0x7f615006d180 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x5617b645f1e0 tx=0x7f615800c040 comp rx=0 tx=0).stop 2024-09-16T01:43:26.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.916+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b645bf60 msgr2=0x5617b644c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:26.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.916+0000 7f6170fb6f40 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b645bf60 0x5617b644c300 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7f6160046160 tx=0x7f6160003020 comp rx=0 tx=0).stop 2024-09-16T01:43:26.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.916+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 shutdown_connections 2024-09-16T01:43:26.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.916+0000 7f6170fb6f40 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f615006acc0 0x7f615006d180 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:26.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.916+0000 7f6170fb6f40 1 --2- 172.21.15.88:0/2955094044 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5617b645bf60 0x5617b644c300 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:26.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.916+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 >> 172.21.15.88:0/2955094044 conn(0x5617b63d2750 msgr2=0x5617b644f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:26.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.916+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 shutdown_connections 2024-09-16T01:43:26.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:26.916+0000 7f6170fb6f40 1 -- 172.21.15.88:0/2955094044 wait complete. 2024-09-16T01:43:26.921 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 100.00000000 stray0_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-09-16T01:43:27.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.005+0000 7faf9d761f40 1 Processor -- start 2024-09-16T01:43:27.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.005+0000 7faf9d761f40 1 -- start start 2024-09-16T01:43:27.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.005+0000 7faf9d761f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ef7e8fbdd0 0x55ef7e9c7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:27.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.005+0000 7faf9d761f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef7e8fe230 0x55ef7e9c24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:27.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.005+0000 7faf9d761f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ef7e9ca540 0x55ef7e9cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:27.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.005+0000 7faf9d761f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55ef7e9455b0 con 0x55ef7e8fe230 2024-09-16T01:43:27.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.005+0000 7faf9d761f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ef7e84c510 con 0x55ef7e8fbdd0 2024-09-16T01:43:27.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.005+0000 7faf9d761f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55ef7e8fc1e0 con 0x55ef7e9ca540 2024-09-16T01:43:27.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.006+0000 7faf92a56640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef7e8fe230 0x55ef7e9c24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:27.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.006+0000 7faf9b257640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ef7e8fbdd0 0x55ef7e9c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:27.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.006+0000 7faf92a56640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef7e8fe230 0x55ef7e9c24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43646/0 (socket says 172.21.15.88:43646) 2024-09-16T01:43:27.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.006+0000 7faf92a56640 1 -- 172.21.15.88:0/2727996404 learned_addr learned my addr 172.21.15.88:0/2727996404 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:27.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.006+0000 7faf9b257640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ef7e8fbdd0 0x55ef7e9c7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:45752/0 (socket says 172.21.15.88:45752) 2024-09-16T01:43:27.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.006+0000 7faf9ba58640 1 --2- 172.21.15.88:0/2727996404 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ef7e9ca540 0x55ef7e9cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:27.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.006+0000 7faf92a56640 1 -- 172.21.15.88:0/2727996404 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ef7e9ca540 msgr2=0x55ef7e9cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:27.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.006+0000 7faf92a56640 1 --2- 172.21.15.88:0/2727996404 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ef7e9ca540 0x55ef7e9cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.006+0000 7faf92a56640 1 -- 172.21.15.88:0/2727996404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ef7e8fbdd0 msgr2=0x55ef7e9c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:27.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.006+0000 7faf92a56640 1 --2- 172.21.15.88:0/2727996404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ef7e8fbdd0 0x55ef7e9c7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.006+0000 7faf92a56640 1 -- 172.21.15.88:0/2727996404 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef7e945cb0 con 0x55ef7e8fe230 2024-09-16T01:43:27.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.006+0000 7faf9ba58640 1 --2- 172.21.15.88:0/2727996404 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ef7e9ca540 0x55ef7e9cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:27.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.006+0000 7faf92a56640 1 --2- 172.21.15.88:0/2727996404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef7e8fe230 0x55ef7e9c24e0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7faf84000c00 tx=0x7faf84051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db0aa94645b7730d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:27.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.006+0000 7faf9a255640 1 -- 172.21.15.88:0/2727996404 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf84063070 con 0x55ef7e8fe230 2024-09-16T01:43:27.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.006+0000 7faf9a255640 1 -- 172.21.15.88:0/2727996404 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf84055b20 con 0x55ef7e8fe230 2024-09-16T01:43:27.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.006+0000 7faf9a255640 1 -- 172.21.15.88:0/2727996404 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf84068460 con 0x55ef7e8fe230 2024-09-16T01:43:27.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.007+0000 7faf9d761f40 1 -- 172.21.15.88:0/2727996404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef7e8fe230 msgr2=0x55ef7e9c24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:27.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.007+0000 7faf9d761f40 1 --2- 172.21.15.88:0/2727996404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef7e8fe230 0x55ef7e9c24e0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7faf84000c00 tx=0x7faf84051000 comp rx=0 tx=0).stop 2024-09-16T01:43:27.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.007+0000 7faf9d761f40 1 -- 172.21.15.88:0/2727996404 shutdown_connections 2024-09-16T01:43:27.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.007+0000 7faf9d761f40 1 --2- 172.21.15.88:0/2727996404 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ef7e9ca540 0x55ef7e9cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.007+0000 7faf9d761f40 1 --2- 172.21.15.88:0/2727996404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef7e8fe230 0x55ef7e9c24e0 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.007+0000 7faf9d761f40 1 --2- 172.21.15.88:0/2727996404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ef7e8fbdd0 0x55ef7e9c7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.007+0000 7faf9d761f40 1 -- 172.21.15.88:0/2727996404 >> 172.21.15.88:0/2727996404 conn(0x55ef7e945750 msgr2=0x55ef7e929690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:27.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.007+0000 7faf9d761f40 1 -- 172.21.15.88:0/2727996404 shutdown_connections 2024-09-16T01:43:27.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.007+0000 7faf9d761f40 1 -- 172.21.15.88:0/2727996404 wait complete. 2024-09-16T01:43:27.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.007+0000 7faf9d761f40 1 Processor -- start 2024-09-16T01:43:27.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.007+0000 7faf9d761f40 1 -- start start 2024-09-16T01:43:27.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.008+0000 7faf9d761f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ef7e8fbdd0 0x55ef7e9c8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:27.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.008+0000 7faf9d761f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef7e8fe230 0x55ef7e9c82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:27.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.008+0000 7faf9d761f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ef7e9ca540 0x55ef7e9c8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:27.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.008+0000 7faf9d761f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55ef7e9d2050 con 0x55ef7e8fe230 2024-09-16T01:43:27.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.008+0000 7faf9d761f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ef7e9d22a0 con 0x55ef7e8fbdd0 2024-09-16T01:43:27.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.008+0000 7faf9d761f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55ef7e9d2580 con 0x55ef7e9ca540 2024-09-16T01:43:27.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.008+0000 7faf92a56640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef7e8fe230 0x55ef7e9c82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:27.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.008+0000 7faf92a56640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef7e8fe230 0x55ef7e9c82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43662/0 (socket says 172.21.15.88:43662) 2024-09-16T01:43:27.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.008+0000 7faf92a56640 1 -- 172.21.15.88:0/3225681023 learned_addr learned my addr 172.21.15.88:0/3225681023 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:27.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.008+0000 7faf9b257640 1 --2- 172.21.15.88:0/3225681023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ef7e8fbdd0 0x55ef7e9c8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:27.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.008+0000 7faf9ba58640 1 --2- 172.21.15.88:0/3225681023 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ef7e9ca540 0x55ef7e9c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:27.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.008+0000 7faf92a56640 1 -- 172.21.15.88:0/3225681023 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ef7e9ca540 msgr2=0x55ef7e9c8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:27.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.008+0000 7faf92a56640 1 --2- 172.21.15.88:0/3225681023 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ef7e9ca540 0x55ef7e9c8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.008+0000 7faf92a56640 1 -- 172.21.15.88:0/3225681023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ef7e8fbdd0 msgr2=0x55ef7e9c8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:27.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.008+0000 7faf92a56640 1 --2- 172.21.15.88:0/3225681023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ef7e8fbdd0 0x55ef7e9c8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.008+0000 7faf92a56640 1 -- 172.21.15.88:0/3225681023 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef7e9447b0 con 0x55ef7e8fe230 2024-09-16T01:43:27.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.008+0000 7faf9b257640 1 --2- 172.21.15.88:0/3225681023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ef7e8fbdd0 0x55ef7e9c8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:27.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.009+0000 7faf92a56640 1 --2- 172.21.15.88:0/3225681023 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef7e8fe230 0x55ef7e9c82d0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7faf84025bc0 tx=0x7faf84002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:27.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.009+0000 7faf99253640 1 -- 172.21.15.88:0/3225681023 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf84063070 con 0x55ef7e8fe230 2024-09-16T01:43:27.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.009+0000 7faf99253640 1 -- 172.21.15.88:0/3225681023 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf8406e070 con 0x55ef7e8fe230 2024-09-16T01:43:27.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.009+0000 7faf99253640 1 -- 172.21.15.88:0/3225681023 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf84055270 con 0x55ef7e8fe230 2024-09-16T01:43:27.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.009+0000 7faf9d761f40 1 -- 172.21.15.88:0/3225681023 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ef7e9c2730 con 0x55ef7e8fe230 2024-09-16T01:43:27.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.009+0000 7faf9d761f40 1 -- 172.21.15.88:0/3225681023 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef7e9c4980 con 0x55ef7e8fe230 2024-09-16T01:43:27.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.010+0000 7faf99253640 1 -- 172.21.15.88:0/3225681023 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7faf8405c040 con 0x55ef7e8fe230 2024-09-16T01:43:27.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.011+0000 7faf99253640 1 --2- 172.21.15.88:0/3225681023 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faf7c06ac90 0x7faf7c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:27.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.011+0000 7faf99253640 1 -- 172.21.15.88:0/3225681023 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8490+0+0 (secure 0 0 0) 0x7faf840cfea0 con 0x55ef7e8fe230 2024-09-16T01:43:27.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.011+0000 7faf9b257640 1 --2- 172.21.15.88:0/3225681023 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faf7c06ac90 0x7faf7c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:27.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.011+0000 7faf9d761f40 1 --2- 172.21.15.88:0/3225681023 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55ef7e9c2bb0 0x55ef7ea74d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:27.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.011+0000 7faf9d761f40 1 -- 172.21.15.88:0/3225681023 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:1 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=103b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119) -- 0x55ef7e9bf490 con 0x55ef7e9c2bb0 2024-09-16T01:43:27.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.011+0000 7faf9ba58640 1 --2- 172.21.15.88:0/3225681023 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55ef7e9c2bb0 0x55ef7ea74d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:27.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.011+0000 7faf9b257640 1 --2- 172.21.15.88:0/3225681023 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faf7c06ac90 0x7faf7c06d150 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7faf8c001ed0 tx=0x7faf8c0017f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:27.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.011+0000 7faf9ba58640 1 --2- 172.21.15.88:0/3225681023 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55ef7e9c2bb0 0x55ef7ea74d60 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:27.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.012+0000 7faf9ba58640 1 -- 172.21.15.88:0/3225681023 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(1 100.00000000 [omap-set-vals] v119'324 uv324 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7faf94011040 con 0x55ef7e9c2bb0 2024-09-16T01:43:27.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.013+0000 7faf9d761f40 1 -- 172.21.15.88:0/3225681023 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55ef7e9c2bb0 msgr2=0x55ef7ea74d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:27.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.013+0000 7faf9d761f40 1 --2- 172.21.15.88:0/3225681023 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55ef7e9c2bb0 0x55ef7ea74d60 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.013+0000 7faf9d761f40 1 -- 172.21.15.88:0/3225681023 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faf7c06ac90 msgr2=0x7faf7c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:27.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.013+0000 7faf9d761f40 1 --2- 172.21.15.88:0/3225681023 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faf7c06ac90 0x7faf7c06d150 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7faf8c001ed0 tx=0x7faf8c0017f0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.013+0000 7faf9d761f40 1 -- 172.21.15.88:0/3225681023 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef7e8fe230 msgr2=0x55ef7e9c82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:27.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.013+0000 7faf9d761f40 1 --2- 172.21.15.88:0/3225681023 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ef7e8fe230 0x55ef7e9c82d0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7faf84025bc0 tx=0x7faf84002fb0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.013+0000 7faf9ba58640 1 -- 172.21.15.88:0/3225681023 reap_dead start 2024-09-16T01:43:27.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.013+0000 7faf9d761f40 1 -- 172.21.15.88:0/3225681023 shutdown_connections 2024-09-16T01:43:27.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.013+0000 7faf9d761f40 1 -- 172.21.15.88:0/3225681023 >> 172.21.15.88:0/3225681023 conn(0x55ef7e945750 msgr2=0x55ef7e929690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:27.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.013+0000 7faf9d761f40 1 -- 172.21.15.88:0/3225681023 shutdown_connections 2024-09-16T01:43:27.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.013+0000 7faf9d761f40 1 -- 172.21.15.88:0/3225681023 wait complete. 2024-09-16T01:43:27.021 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:43:27.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.122+0000 7f8dd6323640 1 Processor -- start 2024-09-16T01:43:27.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.122+0000 7f8dd6323640 1 -- start start 2024-09-16T01:43:27.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.123+0000 7f8dd6323640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8dd015a050 0x7f8dd015d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:27.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.123+0000 7f8dd6323640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dd015a430 con 0x7f8dd00731c0 2024-09-16T01:43:27.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.123+0000 7f8dd6323640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dd0160df0 con 0x7f8dd015ee90 2024-09-16T01:43:27.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.123+0000 7f8dd6323640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f8dd015d9a0 con 0x7f8dd015a050 2024-09-16T01:43:27.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.123+0000 7f8dcffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8dd015a050 0x7f8dd015d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:27.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.123+0000 7f8dcffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8dd015a050 0x7f8dd015d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43668/0 (socket says 172.21.15.88:43668) 2024-09-16T01:43:27.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.123+0000 7f8dcf7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8dd00731c0 0x7f8dd015ffd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36344/0 (socket says 172.21.15.88:36344) 2024-09-16T01:43:27.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.123+0000 7f8dcffff640 1 -- 172.21.15.88:0/1184249770 learned_addr learned my addr 172.21.15.88:0/1184249770 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:27.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.123+0000 7f8dcffff640 1 -- 172.21.15.88:0/1184249770 >> v1:172.21.15.154:6789/0 conn(0x7f8dd015ee90 legacy=0x7f8dd01606e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:27.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.123+0000 7f8dcffff640 1 -- 172.21.15.88:0/1184249770 >> v1:172.21.15.88:6789/0 conn(0x7f8dd00731c0 legacy=0x7f8dd015ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:27.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.123+0000 7f8dcffff640 1 -- 172.21.15.88:0/1184249770 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dd00cd0a0 con 0x7f8dd015a050 2024-09-16T01:43:27.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.123+0000 7f8dceffd640 1 -- 172.21.15.88:0/1184249770 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 268958281 0 0) 0x7f8dd015a430 con 0x7f8dd00731c0 2024-09-16T01:43:27.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.124+0000 7f8dcffff640 1 --2- 172.21.15.88:0/1184249770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dd015a050 0x7f8dd015d460 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0009870 tx=0x7f8dc002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f635f5f148777aff server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:27.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.124+0000 7f8dceffd640 1 -- 172.21.15.88:0/1184249770 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc0037070 con 0x7f8dd015a050 2024-09-16T01:43:27.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.124+0000 7f8dceffd640 1 -- 172.21.15.88:0/1184249770 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dc0032440 con 0x7f8dd015a050 2024-09-16T01:43:27.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.124+0000 7f8dceffd640 1 -- 172.21.15.88:0/1184249770 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc0032810 con 0x7f8dd015a050 2024-09-16T01:43:27.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.124+0000 7f8dd6323640 1 -- 172.21.15.88:0/1184249770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dd015a050 msgr2=0x7f8dd015d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:27.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.124+0000 7f8dd6323640 1 --2- 172.21.15.88:0/1184249770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dd015a050 0x7f8dd015d460 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0009870 tx=0x7f8dc002a270 comp rx=0 tx=0).stop 2024-09-16T01:43:27.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.125+0000 7f8dd6323640 1 -- 172.21.15.88:0/1184249770 shutdown_connections 2024-09-16T01:43:27.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.125+0000 7f8dd6323640 1 --2- 172.21.15.88:0/1184249770 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dd015a050 0x7f8dd015d460 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.125+0000 7f8dd6323640 1 -- 172.21.15.88:0/1184249770 >> 172.21.15.88:0/1184249770 conn(0x7f8dd006d0f0 msgr2=0x7f8dd006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:27.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.125+0000 7f8dd6323640 1 -- 172.21.15.88:0/1184249770 shutdown_connections 2024-09-16T01:43:27.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.125+0000 7f8dd6323640 1 -- 172.21.15.88:0/1184249770 wait complete. 2024-09-16T01:43:27.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.126+0000 7f8dd6323640 1 Processor -- start 2024-09-16T01:43:27.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.126+0000 7f8dd6323640 1 -- start start 2024-09-16T01:43:27.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.127+0000 7f8dd6323640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dd00731c0 0x7f8dd0152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:27.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.127+0000 7f8dd6323640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dd015a050 0x7f8dd0151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:27.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.127+0000 7f8dd6323640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dd015ee90 0x7f8dd0151f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:27.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.127+0000 7f8dd6323640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd00cd940 con 0x7f8dd015ee90 2024-09-16T01:43:27.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.127+0000 7f8dd6323640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd00cd660 con 0x7f8dd00731c0 2024-09-16T01:43:27.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.127+0000 7f8dd6323640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8dd00cdc20 con 0x7f8dd015a050 2024-09-16T01:43:27.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.127+0000 7f8dd48c0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dd015ee90 0x7f8dd0151f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:27.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.127+0000 7f8dd48c0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dd015ee90 0x7f8dd0151f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43676/0 (socket says 172.21.15.88:43676) 2024-09-16T01:43:27.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.127+0000 7f8dd48c0640 1 -- 172.21.15.88:0/3096176395 learned_addr learned my addr 172.21.15.88:0/3096176395 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:27.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.127+0000 7f8dcffff640 1 --2- 172.21.15.88:0/3096176395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dd00731c0 0x7f8dd0152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:27.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.127+0000 7f8dcf7fe640 1 --2- 172.21.15.88:0/3096176395 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dd015a050 0x7f8dd0151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:27.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.127+0000 7f8dd48c0640 1 -- 172.21.15.88:0/3096176395 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dd015a050 msgr2=0x7f8dd0151580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:27.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.127+0000 7f8dd48c0640 1 --2- 172.21.15.88:0/3096176395 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dd015a050 0x7f8dd0151580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.127+0000 7f8dd48c0640 1 -- 172.21.15.88:0/3096176395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dd00731c0 msgr2=0x7f8dd0152a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:27.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.127+0000 7f8dd48c0640 1 --2- 172.21.15.88:0/3096176395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dd00731c0 0x7f8dd0152a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.127+0000 7f8dd48c0640 1 -- 172.21.15.88:0/3096176395 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dd01e8810 con 0x7f8dd015ee90 2024-09-16T01:43:27.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.127+0000 7f8dd48c0640 1 --2- 172.21.15.88:0/3096176395 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dd015ee90 0x7f8dd0151f50 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f8dc800c6b0 tx=0x7f8dc800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:27.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.127+0000 7f8dccff9640 1 -- 172.21.15.88:0/3096176395 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8003b70 con 0x7f8dd015ee90 2024-09-16T01:43:27.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.127+0000 7f8dccff9640 1 -- 172.21.15.88:0/3096176395 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dc8003d30 con 0x7f8dd015ee90 2024-09-16T01:43:27.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.127+0000 7f8dccff9640 1 -- 172.21.15.88:0/3096176395 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8005310 con 0x7f8dd015ee90 2024-09-16T01:43:27.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.127+0000 7f8dd6323640 1 -- 172.21.15.88:0/3096176395 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dd01e8af0 con 0x7f8dd015ee90 2024-09-16T01:43:27.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.127+0000 7f8dd6323640 1 -- 172.21.15.88:0/3096176395 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dd01e6c60 con 0x7f8dd015ee90 2024-09-16T01:43:27.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.129+0000 7f8dd6323640 1 -- 172.21.15.88:0/3096176395 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dd01e8cc0 con 0x7f8dd015ee90 2024-09-16T01:43:27.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.129+0000 7f8dccff9640 1 -- 172.21.15.88:0/3096176395 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8dc80054b0 con 0x7f8dd015ee90 2024-09-16T01:43:27.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.129+0000 7f8dccff9640 1 --2- 172.21.15.88:0/3096176395 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8dbc06c2b0 0x7f8dbc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:27.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.130+0000 7f8dcffff640 1 --2- 172.21.15.88:0/3096176395 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8dbc06c2b0 0x7f8dbc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:27.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.130+0000 7f8dccff9640 1 -- 172.21.15.88:0/3096176395 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8490+0+0 (secure 0 0 0) 0x7f8dc808bbe0 con 0x7f8dd015ee90 2024-09-16T01:43:27.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.130+0000 7f8dcffff640 1 --2- 172.21.15.88:0/3096176395 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8dbc06c2b0 0x7f8dbc06e770 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0034000 tx=0x7f8dc002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:27.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.134+0000 7f8dccff9640 1 -- 172.21.15.88:0/3096176395 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8dc8059260 con 0x7f8dd015ee90 2024-09-16T01:43:27.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.333+0000 7f8dd6323640 1 -- 172.21.15.88:0/3096176395 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f8dd00b36e0 con 0x7f8dd015ee90 2024-09-16T01:43:27.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.645+0000 7f8dccff9640 1 -- 172.21.15.88:0/3096176395 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v262) ==== 162+0+0 (secure 0 0 0) 0x7f8dc805d560 con 0x7f8dd015ee90 2024-09-16T01:43:27.647 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:43:27.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.648+0000 7f8dd6323640 1 -- 172.21.15.88:0/3096176395 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8dbc06c2b0 msgr2=0x7f8dbc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:27.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.648+0000 7f8dd6323640 1 --2- 172.21.15.88:0/3096176395 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8dbc06c2b0 0x7f8dbc06e770 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0034000 tx=0x7f8dc002e000 comp rx=0 tx=0).stop 2024-09-16T01:43:27.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.648+0000 7f8dd6323640 1 -- 172.21.15.88:0/3096176395 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dd015ee90 msgr2=0x7f8dd0151f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:27.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.648+0000 7f8dd6323640 1 --2- 172.21.15.88:0/3096176395 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dd015ee90 0x7f8dd0151f50 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f8dc800c6b0 tx=0x7f8dc800cb70 comp rx=0 tx=0).stop 2024-09-16T01:43:27.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.648+0000 7f8dd6323640 1 -- 172.21.15.88:0/3096176395 shutdown_connections 2024-09-16T01:43:27.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.648+0000 7f8dd6323640 1 --2- 172.21.15.88:0/3096176395 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8dbc06c2b0 0x7f8dbc06e770 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.648+0000 7f8dd6323640 1 --2- 172.21.15.88:0/3096176395 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8dd015ee90 0x7f8dd0151f50 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.648+0000 7f8dd6323640 1 --2- 172.21.15.88:0/3096176395 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dd015a050 0x7f8dd0151580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.648+0000 7f8dd6323640 1 --2- 172.21.15.88:0/3096176395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dd00731c0 0x7f8dd0152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.648+0000 7f8dd6323640 1 -- 172.21.15.88:0/3096176395 >> 172.21.15.88:0/3096176395 conn(0x7f8dd006d0f0 msgr2=0x7f8dd00cee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:27.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.648+0000 7f8dd6323640 1 -- 172.21.15.88:0/3096176395 shutdown_connections 2024-09-16T01:43:27.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.648+0000 7f8dd6323640 1 -- 172.21.15.88:0/3096176395 wait complete. 2024-09-16T01:43:27.663 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:43:27.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.759+0000 7fef5a5a8640 1 Processor -- start 2024-09-16T01:43:27.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.759+0000 7fef5a5a8640 1 -- start start 2024-09-16T01:43:27.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.760+0000 7fef5a5a8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fef540b3710 0x7fef540b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:27.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.760+0000 7fef5a5a8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fef540cf3d0 0x7fef540b4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:27.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.760+0000 7fef5a5a8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef540b4db0 con 0x7fef540b5be0 2024-09-16T01:43:27.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.760+0000 7fef5a5a8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fef540cdf30 con 0x7fef540cf3d0 2024-09-16T01:43:27.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.760+0000 7fef5a5a8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fef540b5090 con 0x7fef540b3710 2024-09-16T01:43:27.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.760+0000 7fef58b45640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fef540cf3d0 0x7fef540b4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:27.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.760+0000 7fef58b45640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fef540cf3d0 0x7fef540b4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43686/0 (socket says 172.21.15.88:43686) 2024-09-16T01:43:27.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.760+0000 7fef58b45640 1 -- 172.21.15.88:0/3434681094 learned_addr learned my addr 172.21.15.88:0/3434681094 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:27.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.760+0000 7fef53fff640 1 --2- 172.21.15.88:0/3434681094 >> v2:172.21.15.154:3300/0 conn(0x7fef540b3710 0x7fef540b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:27.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.760+0000 7fef58b45640 1 -- 172.21.15.88:0/3434681094 >> v2:172.21.15.154:3300/0 conn(0x7fef540b3710 msgr2=0x7fef540b3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:27.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.760+0000 7fef58b45640 1 --2- 172.21.15.88:0/3434681094 >> v2:172.21.15.154:3300/0 conn(0x7fef540b3710 0x7fef540b3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.760+0000 7fef58b45640 1 -- 172.21.15.88:0/3434681094 >> v1:172.21.15.154:6789/0 conn(0x7fef540b5be0 legacy=0x7fef540b4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:27.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.760+0000 7fef58b45640 1 -- 172.21.15.88:0/3434681094 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef540cdbc0 con 0x7fef540cf3d0 2024-09-16T01:43:27.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.761+0000 7fef58b45640 1 --2- 172.21.15.88:0/3434681094 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef540cf3d0 0x7fef540b4740 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7fef540ce0b0 tx=0x7fef4c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f5cfbd4a5bf2c65 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:27.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.761+0000 7fef52ffd640 1 -- 172.21.15.88:0/3434681094 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4c036070 con 0x7fef540cf3d0 2024-09-16T01:43:27.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.761+0000 7fef52ffd640 1 -- 172.21.15.88:0/3434681094 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef4c031600 con 0x7fef540cf3d0 2024-09-16T01:43:27.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.761+0000 7fef52ffd640 1 -- 172.21.15.88:0/3434681094 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4c0319d0 con 0x7fef540cf3d0 2024-09-16T01:43:27.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.762+0000 7fef5a5a8640 1 -- 172.21.15.88:0/3434681094 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef540cf3d0 msgr2=0x7fef540b4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:27.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.762+0000 7fef5a5a8640 1 --2- 172.21.15.88:0/3434681094 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef540cf3d0 0x7fef540b4740 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7fef540ce0b0 tx=0x7fef4c02a6a0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.762+0000 7fef5a5a8640 1 -- 172.21.15.88:0/3434681094 shutdown_connections 2024-09-16T01:43:27.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.762+0000 7fef5a5a8640 1 --2- 172.21.15.88:0/3434681094 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef540cf3d0 0x7fef540b4740 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.762+0000 7fef5a5a8640 1 --2- 172.21.15.88:0/3434681094 >> v2:172.21.15.154:3300/0 conn(0x7fef540b3710 0x7fef540b3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.762+0000 7fef5a5a8640 1 -- 172.21.15.88:0/3434681094 >> 172.21.15.88:0/3434681094 conn(0x7fef5406bc10 msgr2=0x7fef5406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:27.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.762+0000 7fef5a5a8640 1 -- 172.21.15.88:0/3434681094 shutdown_connections 2024-09-16T01:43:27.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.762+0000 7fef5a5a8640 1 -- 172.21.15.88:0/3434681094 wait complete. 2024-09-16T01:43:27.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.763+0000 7fef5a5a8640 1 Processor -- start 2024-09-16T01:43:27.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.763+0000 7fef5a5a8640 1 -- start start 2024-09-16T01:43:27.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.764+0000 7fef5a5a8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef540b3710 0x7fef5414f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:27.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.764+0000 7fef5a5a8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef540b5be0 0x7fef54138a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:27.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.764+0000 7fef5a5a8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef540cf3d0 0x7fef540721b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:27.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.764+0000 7fef5a5a8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fef541647b0 con 0x7fef540b5be0 2024-09-16T01:43:27.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.764+0000 7fef5a5a8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fef54164630 con 0x7fef540cf3d0 2024-09-16T01:43:27.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.764+0000 7fef5a5a8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fef54164930 con 0x7fef540b3710 2024-09-16T01:43:27.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.764+0000 7fef537fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef540b5be0 0x7fef54138a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:27.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.764+0000 7fef537fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef540b5be0 0x7fef54138a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43700/0 (socket says 172.21.15.88:43700) 2024-09-16T01:43:27.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.764+0000 7fef537fe640 1 -- 172.21.15.88:0/1527114616 learned_addr learned my addr 172.21.15.88:0/1527114616 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:27.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.764+0000 7fef58b45640 1 --2- 172.21.15.88:0/1527114616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef540cf3d0 0x7fef540721b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:27.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.764+0000 7fef53fff640 1 --2- 172.21.15.88:0/1527114616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef540b3710 0x7fef5414f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:27.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.764+0000 7fef537fe640 1 -- 172.21.15.88:0/1527114616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef540b3710 msgr2=0x7fef5414f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:27.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.764+0000 7fef537fe640 1 --2- 172.21.15.88:0/1527114616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef540b3710 0x7fef5414f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.764+0000 7fef537fe640 1 -- 172.21.15.88:0/1527114616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef540cf3d0 msgr2=0x7fef540721b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:27.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.764+0000 7fef537fe640 1 --2- 172.21.15.88:0/1527114616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef540cf3d0 0x7fef540721b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.764+0000 7fef537fe640 1 -- 172.21.15.88:0/1527114616 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef54138fa0 con 0x7fef540b5be0 2024-09-16T01:43:27.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.764+0000 7fef53fff640 1 --2- 172.21.15.88:0/1527114616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef540b3710 0x7fef5414f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:27.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.764+0000 7fef537fe640 1 --2- 172.21.15.88:0/1527114616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef540b5be0 0x7fef54138a60 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7fef4800c440 tx=0x7fef48001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:27.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.764+0000 7fef50ff9640 1 -- 172.21.15.88:0/1527114616 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef480052d0 con 0x7fef540b5be0 2024-09-16T01:43:27.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.764+0000 7fef50ff9640 1 -- 172.21.15.88:0/1527114616 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef48005470 con 0x7fef540b5be0 2024-09-16T01:43:27.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.764+0000 7fef5a5a8640 1 -- 172.21.15.88:0/1527114616 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef541392a0 con 0x7fef540b5be0 2024-09-16T01:43:27.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.764+0000 7fef50ff9640 1 -- 172.21.15.88:0/1527114616 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef48007260 con 0x7fef540b5be0 2024-09-16T01:43:27.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.765+0000 7fef5a5a8640 1 -- 172.21.15.88:0/1527114616 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef54139860 con 0x7fef540b5be0 2024-09-16T01:43:27.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.765+0000 7fef5a5a8640 1 -- 172.21.15.88:0/1527114616 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef54068fa0 con 0x7fef540b5be0 2024-09-16T01:43:27.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.766+0000 7fef50ff9640 1 -- 172.21.15.88:0/1527114616 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fef48007400 con 0x7fef540b5be0 2024-09-16T01:43:27.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.766+0000 7fef50ff9640 1 --2- 172.21.15.88:0/1527114616 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fef4006c2b0 0x7fef4006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:27.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.766+0000 7fef50ff9640 1 -- 172.21.15.88:0/1527114616 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8490+0+0 (secure 0 0 0) 0x7fef4808b460 con 0x7fef540b5be0 2024-09-16T01:43:27.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.766+0000 7fef53fff640 1 --2- 172.21.15.88:0/1527114616 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fef4006c2b0 0x7fef4006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:27.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.767+0000 7fef53fff640 1 --2- 172.21.15.88:0/1527114616 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fef4006c2b0 0x7fef4006e770 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fef44006fd0 tx=0x7fef44008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:27.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.771+0000 7fef50ff9640 1 -- 172.21.15.88:0/1527114616 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fef54068fa0 con 0x7fef540b5be0 2024-09-16T01:43:27.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.969+0000 7fef5a5a8640 1 -- 172.21.15.88:0/1527114616 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef540ceaa0 con 0x7fef540b5be0 2024-09-16T01:43:27.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.970+0000 7fef50ff9640 1 -- 172.21.15.88:0/1527114616 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 263 v263) ==== 95+0+5227 (secure 0 0 0) 0x7fef480589c0 con 0x7fef540b5be0 2024-09-16T01:43:27.971 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:27.971 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":263,"btime":"2024-09-16T01:43:27:648762+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9696,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4270197461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4270197461},{"type":"v1","addr":"172.21.15.88:6837","nonce":4270197461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":263,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:43:27.648747+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":119,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9771},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9771":{"gid":9771,"name":"a","rank":0,"incarnation":263,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6836/3121519215","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3121519215},{"type":"v1","addr":"172.21.15.88:6836","nonce":3121519215}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:43:27.972 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 263 2024-09-16T01:43:27.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.972+0000 7fef5a5a8640 1 -- 172.21.15.88:0/1527114616 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fef4006c2b0 msgr2=0x7fef4006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:27.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.972+0000 7fef5a5a8640 1 --2- 172.21.15.88:0/1527114616 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fef4006c2b0 0x7fef4006e770 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fef44006fd0 tx=0x7fef44008040 comp rx=0 tx=0).stop 2024-09-16T01:43:27.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.972+0000 7fef5a5a8640 1 -- 172.21.15.88:0/1527114616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef540b5be0 msgr2=0x7fef54138a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:27.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.972+0000 7fef5a5a8640 1 --2- 172.21.15.88:0/1527114616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef540b5be0 0x7fef54138a60 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7fef4800c440 tx=0x7fef48001e70 comp rx=0 tx=0).stop 2024-09-16T01:43:27.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.972+0000 7fef5a5a8640 1 -- 172.21.15.88:0/1527114616 shutdown_connections 2024-09-16T01:43:27.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.972+0000 7fef5a5a8640 1 --2- 172.21.15.88:0/1527114616 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fef4006c2b0 0x7fef4006e770 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.972+0000 7fef5a5a8640 1 --2- 172.21.15.88:0/1527114616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef540cf3d0 0x7fef540721b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.972+0000 7fef5a5a8640 1 --2- 172.21.15.88:0/1527114616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef540b5be0 0x7fef54138a60 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.972+0000 7fef5a5a8640 1 --2- 172.21.15.88:0/1527114616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fef540b3710 0x7fef5414f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:27.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.972+0000 7fef5a5a8640 1 -- 172.21.15.88:0/1527114616 >> 172.21.15.88:0/1527114616 conn(0x7fef5406bc10 msgr2=0x7fef54157540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:27.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.972+0000 7fef5a5a8640 1 -- 172.21.15.88:0/1527114616 shutdown_connections 2024-09-16T01:43:27.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:27.972+0000 7fef5a5a8640 1 -- 172.21.15.88:0/1527114616 wait complete. 2024-09-16T01:43:27.984 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-09-16T01:43:30.668 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:43:30.667+0000 7f46ecc06640 -1 mds.0.damage notify_dentry Damage to dentries in fragment * of ino 0x100is fatal because it is a system directory for this rank 2024-09-16T01:43:30.669 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:43:30.668+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:43:30.669 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:43:30.668+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:43:31.646 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:43:31.645+0000 7f46f0c0e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:43:32.985 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:43:33.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.095+0000 7f87087cd640 1 Processor -- start 2024-09-16T01:43:33.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.095+0000 7f87087cd640 1 -- start start 2024-09-16T01:43:33.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.096+0000 7f87087cd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f87000b4920 0x7f870015b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:33.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.096+0000 7f87087cd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87000ce4f0 con 0x7f87000cd780 2024-09-16T01:43:33.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.096+0000 7f87087cd640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87000ce7d0 con 0x7f87000b4d00 2024-09-16T01:43:33.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.096+0000 7f87087cd640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f87000cf100 con 0x7f87000b4920 2024-09-16T01:43:33.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.096+0000 7f8706d6a640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f87000cd780 0x7f87000cdb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54002/0 (socket says 172.21.15.88:54002) 2024-09-16T01:43:33.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.096+0000 7f8706d6a640 1 -- 172.21.15.88:0/1098778508 learned_addr learned my addr 172.21.15.88:0/1098778508 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:33.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.096+0000 7f8706569640 1 --2- 172.21.15.88:0/1098778508 >> v2:172.21.15.154:3300/0 conn(0x7f87000b4920 0x7f870015b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:33.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.096+0000 7f8705567640 1 -- 172.21.15.88:0/1098778508 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1755642448 0 0) 0x7f87000ce4f0 con 0x7f87000cd780 2024-09-16T01:43:33.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.096+0000 7f8705567640 1 -- 172.21.15.88:0/1098778508 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86ec003520 con 0x7f87000cd780 2024-09-16T01:43:33.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.096+0000 7f8705567640 1 -- 172.21.15.88:0/1098778508 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1197118250 0 0) 0x7f86ec003520 con 0x7f87000cd780 2024-09-16T01:43:33.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.096+0000 7f8705567640 1 -- 172.21.15.88:0/1098778508 >> v2:172.21.15.154:3300/0 conn(0x7f87000b4920 msgr2=0x7f870015b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:33.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.096+0000 7f8705567640 1 --2- 172.21.15.88:0/1098778508 >> v2:172.21.15.154:3300/0 conn(0x7f87000b4920 0x7f870015b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.096+0000 7f8705567640 1 -- 172.21.15.88:0/1098778508 >> v1:172.21.15.154:6789/0 conn(0x7f87000b4d00 legacy=0x7f87000b5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:33.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.096+0000 7f8705567640 1 -- 172.21.15.88:0/1098778508 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87000ceab0 con 0x7f87000cd780 2024-09-16T01:43:33.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.096+0000 7f8705567640 1 -- 172.21.15.88:0/1098778508 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f86f8001670 con 0x7f87000cd780 2024-09-16T01:43:33.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.097+0000 7f8705567640 1 -- 172.21.15.88:0/1098778508 >> v1:172.21.15.88:6789/0 conn(0x7f87000cd780 legacy=0x7f87000cdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:33.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.097+0000 7f8705567640 1 --2- 172.21.15.88:0/1098778508 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86ec0039b0 0x7f86ec003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:33.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.097+0000 7f8705567640 1 --2- 172.21.15.88:0/1098778508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86ec004800 0x7f86ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:33.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.097+0000 7f8705567640 1 --2- 172.21.15.88:0/1098778508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86ec0051c0 0x7f86ec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:33.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.097+0000 7f8705567640 1 -- 172.21.15.88:0/1098778508 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f87000cf100 con 0x7f86ec004800 2024-09-16T01:43:33.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.097+0000 7f8705567640 1 -- 172.21.15.88:0/1098778508 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f86ec00c090 con 0x7f86ec0051c0 2024-09-16T01:43:33.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.097+0000 7f8705567640 1 -- 172.21.15.88:0/1098778508 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f86ec00c290 con 0x7f86ec0039b0 2024-09-16T01:43:33.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.097+0000 7f8705d68640 1 --2- 172.21.15.88:0/1098778508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86ec004800 0x7f86ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:33.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.097+0000 7f8706d6a640 1 --2- 172.21.15.88:0/1098778508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86ec0051c0 0x7f86ec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:33.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.097+0000 7f8706569640 1 --2- 172.21.15.88:0/1098778508 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86ec0039b0 0x7f86ec003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:33.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.097+0000 7f8705d68640 1 -- 172.21.15.88:0/1098778508 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86ec0039b0 msgr2=0x7f86ec003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:33.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.097+0000 7f8705d68640 1 --2- 172.21.15.88:0/1098778508 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86ec0039b0 0x7f86ec003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.097+0000 7f8705d68640 1 -- 172.21.15.88:0/1098778508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86ec0051c0 msgr2=0x7f86ec00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:33.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.097+0000 7f8705d68640 1 --2- 172.21.15.88:0/1098778508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86ec0051c0 0x7f86ec00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.097+0000 7f8705d68640 1 -- 172.21.15.88:0/1098778508 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86ec003520 con 0x7f86ec004800 2024-09-16T01:43:33.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.097+0000 7f8706d6a640 1 -- 172.21.15.88:0/1098778508 reap_dead start 2024-09-16T01:43:33.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.097+0000 7f8706569640 1 --2- 172.21.15.88:0/1098778508 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86ec0039b0 0x7f86ec003df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:33.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.098+0000 7f8705d68640 1 --2- 172.21.15.88:0/1098778508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86ec004800 0x7f86ec004c80 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f86f0001750 tx=0x7f86ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7c37a3e8613116f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:33.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.098+0000 7f8705567640 1 -- 172.21.15.88:0/1098778508 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f0001370 con 0x7f86ec004800 2024-09-16T01:43:33.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.098+0000 7f8705567640 1 -- 172.21.15.88:0/1098778508 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86f00052d0 con 0x7f86ec004800 2024-09-16T01:43:33.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.098+0000 7f8705567640 1 -- 172.21.15.88:0/1098778508 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f002abc0 con 0x7f86ec004800 2024-09-16T01:43:33.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.098+0000 7f87087cd640 1 -- 172.21.15.88:0/1098778508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86ec004800 msgr2=0x7f86ec004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:33.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.098+0000 7f87087cd640 1 --2- 172.21.15.88:0/1098778508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86ec004800 0x7f86ec004c80 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f86f0001750 tx=0x7f86ec005830 comp rx=0 tx=0).stop 2024-09-16T01:43:33.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.098+0000 7f87087cd640 1 -- 172.21.15.88:0/1098778508 shutdown_connections 2024-09-16T01:43:33.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.098+0000 7f87087cd640 1 --2- 172.21.15.88:0/1098778508 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86ec004800 0x7f86ec004c80 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.098+0000 7f87087cd640 1 -- 172.21.15.88:0/1098778508 >> 172.21.15.88:0/1098778508 conn(0x7f870006bc10 msgr2=0x7f870006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:33.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.099+0000 7f87087cd640 1 -- 172.21.15.88:0/1098778508 shutdown_connections 2024-09-16T01:43:33.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.099+0000 7f87087cd640 1 -- 172.21.15.88:0/1098778508 wait complete. 2024-09-16T01:43:33.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.100+0000 7f87087cd640 1 Processor -- start 2024-09-16T01:43:33.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.100+0000 7f87087cd640 1 -- start start 2024-09-16T01:43:33.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.100+0000 7f87087cd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86ec004800 0x7f87001527f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:33.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.100+0000 7f87087cd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8700151760 0x7f870014d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:33.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.100+0000 7f87087cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87001544b0 0x7f8700154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:33.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.101+0000 7f87087cd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f87000b52f0 con 0x7f87001544b0 2024-09-16T01:43:33.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.101+0000 7f87087cd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f87000b5170 con 0x7f8700151760 2024-09-16T01:43:33.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.101+0000 7f87087cd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f87000b5470 con 0x7f86ec004800 2024-09-16T01:43:33.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.101+0000 7f8706d6a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87001544b0 0x7f8700154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:33.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.101+0000 7f8706d6a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87001544b0 0x7f8700154930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60142/0 (socket says 172.21.15.88:60142) 2024-09-16T01:43:33.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.101+0000 7f8706d6a640 1 -- 172.21.15.88:0/625995630 learned_addr learned my addr 172.21.15.88:0/625995630 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:33.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.101+0000 7f8705d68640 1 --2- 172.21.15.88:0/625995630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8700151760 0x7f870014d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:33.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.101+0000 7f8706569640 1 --2- 172.21.15.88:0/625995630 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86ec004800 0x7f87001527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:33.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.101+0000 7f8706d6a640 1 -- 172.21.15.88:0/625995630 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86ec004800 msgr2=0x7f87001527f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:33.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.101+0000 7f8706d6a640 1 --2- 172.21.15.88:0/625995630 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86ec004800 0x7f87001527f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.101+0000 7f8706d6a640 1 -- 172.21.15.88:0/625995630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8700151760 msgr2=0x7f870014d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:33.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.101+0000 7f8706d6a640 1 --2- 172.21.15.88:0/625995630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8700151760 0x7f870014d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.101+0000 7f8706d6a640 1 -- 172.21.15.88:0/625995630 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8700151bc0 con 0x7f87001544b0 2024-09-16T01:43:33.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.101+0000 7f8706d6a640 1 --2- 172.21.15.88:0/625995630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87001544b0 0x7f8700154930 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f87000cdb60 tx=0x7f86f8007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:33.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.101+0000 7f86f6ffd640 1 -- 172.21.15.88:0/625995630 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f801a070 con 0x7f87001544b0 2024-09-16T01:43:33.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.101+0000 7f86f6ffd640 1 -- 172.21.15.88:0/625995630 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86f800f620 con 0x7f87001544b0 2024-09-16T01:43:33.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.101+0000 7f87087cd640 1 -- 172.21.15.88:0/625995630 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8700152eb0 con 0x7f87001544b0 2024-09-16T01:43:33.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.101+0000 7f86f6ffd640 1 -- 172.21.15.88:0/625995630 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f800f9f0 con 0x7f87001544b0 2024-09-16T01:43:33.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.101+0000 7f87087cd640 1 -- 172.21.15.88:0/625995630 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8700153cb0 con 0x7f87001544b0 2024-09-16T01:43:33.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.102+0000 7f87087cd640 1 -- 172.21.15.88:0/625995630 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f870015b7d0 con 0x7f87001544b0 2024-09-16T01:43:33.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.103+0000 7f86f6ffd640 1 -- 172.21.15.88:0/625995630 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f86f801d070 con 0x7f87001544b0 2024-09-16T01:43:33.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.103+0000 7f86f6ffd640 1 --2- 172.21.15.88:0/625995630 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f86e406c2e0 0x7f86e406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:33.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.103+0000 7f86f6ffd640 1 -- 172.21.15.88:0/625995630 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8576+0+0 (secure 0 0 0) 0x7f86f808b360 con 0x7f87001544b0 2024-09-16T01:43:33.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.103+0000 7f8706569640 1 --2- 172.21.15.88:0/625995630 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f86e406c2e0 0x7f86e406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:33.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.104+0000 7f8706569640 1 --2- 172.21.15.88:0/625995630 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f86e406c2e0 0x7f86e406e7a0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f870006a350 tx=0x7f86fc00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:33.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.108+0000 7f86f6ffd640 1 -- 172.21.15.88:0/625995630 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f86f8058990 con 0x7f87001544b0 2024-09-16T01:43:33.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.307+0000 7f87087cd640 1 -- 172.21.15.88:0/625995630 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f870015e690 con 0x7f87001544b0 2024-09-16T01:43:33.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.307+0000 7f86f6ffd640 1 -- 172.21.15.88:0/625995630 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 267 v267) ==== 95+0+4392 (secure 0 0 0) 0x7f86f805de20 con 0x7f87001544b0 2024-09-16T01:43:33.309 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:33.309 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":267,"btime":"2024-09-16T01:43:30:669481+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9696,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4270197461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4270197461},{"type":"v1","addr":"172.21.15.88:6837","nonce":4270197461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":267,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:43:30.669477+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:43:33.309 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 267 2024-09-16T01:43:33.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.309+0000 7f87087cd640 1 -- 172.21.15.88:0/625995630 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f86e406c2e0 msgr2=0x7f86e406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:33.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.309+0000 7f87087cd640 1 --2- 172.21.15.88:0/625995630 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f86e406c2e0 0x7f86e406e7a0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f870006a350 tx=0x7f86fc00b040 comp rx=0 tx=0).stop 2024-09-16T01:43:33.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.309+0000 7f87087cd640 1 -- 172.21.15.88:0/625995630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87001544b0 msgr2=0x7f8700154930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:33.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.309+0000 7f87087cd640 1 --2- 172.21.15.88:0/625995630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87001544b0 0x7f8700154930 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f87000cdb60 tx=0x7f86f8007430 comp rx=0 tx=0).stop 2024-09-16T01:43:33.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.309+0000 7f87087cd640 1 -- 172.21.15.88:0/625995630 shutdown_connections 2024-09-16T01:43:33.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.309+0000 7f87087cd640 1 --2- 172.21.15.88:0/625995630 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f86e406c2e0 0x7f86e406e7a0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.309+0000 7f87087cd640 1 --2- 172.21.15.88:0/625995630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87001544b0 0x7f8700154930 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.309+0000 7f87087cd640 1 --2- 172.21.15.88:0/625995630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8700151760 0x7f870014d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.309+0000 7f87087cd640 1 --2- 172.21.15.88:0/625995630 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86ec004800 0x7f87001527f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.309+0000 7f87087cd640 1 -- 172.21.15.88:0/625995630 >> 172.21.15.88:0/625995630 conn(0x7f870006bc10 msgr2=0x7f870015d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:33.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.309+0000 7f87087cd640 1 -- 172.21.15.88:0/625995630 shutdown_connections 2024-09-16T01:43:33.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.309+0000 7f87087cd640 1 -- 172.21.15.88:0/625995630 wait complete. 2024-09-16T01:43:33.322 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-09-16T01:43:33.322 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:43:33.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.420+0000 7f4cf6015640 1 Processor -- start 2024-09-16T01:43:33.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.420+0000 7f4cf6015640 1 -- start start 2024-09-16T01:43:33.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.420+0000 7f4cf6015640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4cf015e660 0x7f4cf00cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:33.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.420+0000 7f4cf6015640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4cf00d3310 0x7f4cf00d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:33.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.420+0000 7f4cf6015640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cf015ea40 con 0x7f4cf00d1650 2024-09-16T01:43:33.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.420+0000 7f4cf6015640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f4cf00d3870 con 0x7f4cf00d3310 2024-09-16T01:43:33.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.420+0000 7f4cf6015640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4cf00d3b50 con 0x7f4cf015e660 2024-09-16T01:43:33.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.421+0000 7f4ceeffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4cf00d3310 0x7f4cf00d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:33.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.421+0000 7f4ceeffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4cf00d3310 0x7f4cf00d1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60148/0 (socket says 172.21.15.88:60148) 2024-09-16T01:43:33.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.421+0000 7f4ceeffd640 1 -- 172.21.15.88:0/352917218 learned_addr learned my addr 172.21.15.88:0/352917218 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:33.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.421+0000 7f4cef7fe640 1 --2- 172.21.15.88:0/352917218 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4cf015e660 0x7f4cf00cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:33.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.421+0000 7f4ceeffd640 1 -- 172.21.15.88:0/352917218 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4cf015e660 msgr2=0x7f4cf00cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:33.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.421+0000 7f4ceeffd640 1 --2- 172.21.15.88:0/352917218 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4cf015e660 0x7f4cf00cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.421+0000 7f4ceeffd640 1 -- 172.21.15.88:0/352917218 >> v1:172.21.15.154:6789/0 conn(0x7f4cf00d1650 legacy=0x7f4cf006bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:33.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.421+0000 7f4ceeffd640 1 -- 172.21.15.88:0/352917218 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cf00d3e30 con 0x7f4cf00d3310 2024-09-16T01:43:33.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.422+0000 7f4ceeffd640 1 --2- 172.21.15.88:0/352917218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cf00d3310 0x7f4cf00d1110 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4009870 tx=0x7f4ce402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab2fbb216890f8f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:33.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.422+0000 7f4cee7fc640 1 -- 172.21.15.88:0/352917218 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4037070 con 0x7f4cf00d3310 2024-09-16T01:43:33.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.422+0000 7f4cee7fc640 1 -- 172.21.15.88:0/352917218 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ce4032440 con 0x7f4cf00d3310 2024-09-16T01:43:33.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.422+0000 7f4cee7fc640 1 -- 172.21.15.88:0/352917218 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4032810 con 0x7f4cf00d3310 2024-09-16T01:43:33.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.422+0000 7f4cf6015640 1 -- 172.21.15.88:0/352917218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cf00d3310 msgr2=0x7f4cf00d1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:33.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.422+0000 7f4cf6015640 1 --2- 172.21.15.88:0/352917218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cf00d3310 0x7f4cf00d1110 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4009870 tx=0x7f4ce402a270 comp rx=0 tx=0).stop 2024-09-16T01:43:33.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.422+0000 7f4cf6015640 1 -- 172.21.15.88:0/352917218 shutdown_connections 2024-09-16T01:43:33.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.422+0000 7f4cf6015640 1 --2- 172.21.15.88:0/352917218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cf00d3310 0x7f4cf00d1110 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.422+0000 7f4cf6015640 1 --2- 172.21.15.88:0/352917218 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4cf015e660 0x7f4cf00cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.422+0000 7f4cf6015640 1 -- 172.21.15.88:0/352917218 >> 172.21.15.88:0/352917218 conn(0x7f4cf00b6410 msgr2=0x7f4cf00b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:33.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.422+0000 7f4cf6015640 1 -- 172.21.15.88:0/352917218 shutdown_connections 2024-09-16T01:43:33.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.422+0000 7f4cf6015640 1 -- 172.21.15.88:0/352917218 wait complete. 2024-09-16T01:43:33.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.423+0000 7f4cf6015640 1 Processor -- start 2024-09-16T01:43:33.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.424+0000 7f4cf6015640 1 -- start start 2024-09-16T01:43:33.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.424+0000 7f4cf6015640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cf00d1650 0x7f4cf00ca950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:33.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.424+0000 7f4cf6015640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cf00d3310 0x7f4cf01596e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:33.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.424+0000 7f4cf6015640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4cf015e660 0x7f4cf00cb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:33.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.424+0000 7f4cf6015640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf00cfc70 con 0x7f4cf00d3310 2024-09-16T01:43:33.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.424+0000 7f4cf6015640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf00d43f0 con 0x7f4cf00d1650 2024-09-16T01:43:33.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.424+0000 7f4cf6015640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4cf00b8b80 con 0x7f4cf015e660 2024-09-16T01:43:33.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.424+0000 7f4ceeffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cf00d3310 0x7f4cf01596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:33.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.425+0000 7f4ceeffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cf00d3310 0x7f4cf01596e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60156/0 (socket says 172.21.15.88:60156) 2024-09-16T01:43:33.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.425+0000 7f4ceeffd640 1 -- 172.21.15.88:0/4135683120 learned_addr learned my addr 172.21.15.88:0/4135683120 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:33.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.425+0000 7f4cef7fe640 1 --2- 172.21.15.88:0/4135683120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cf00d1650 0x7f4cf00ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:33.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.425+0000 7f4ceffff640 1 --2- 172.21.15.88:0/4135683120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4cf015e660 0x7f4cf00cb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:33.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.425+0000 7f4ceeffd640 1 -- 172.21.15.88:0/4135683120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4cf015e660 msgr2=0x7f4cf00cb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:33.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.425+0000 7f4ceeffd640 1 --2- 172.21.15.88:0/4135683120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4cf015e660 0x7f4cf00cb8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.425+0000 7f4ceeffd640 1 -- 172.21.15.88:0/4135683120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cf00d1650 msgr2=0x7f4cf00ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:33.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.425+0000 7f4ceeffd640 1 --2- 172.21.15.88:0/4135683120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cf00d1650 0x7f4cf00ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.425+0000 7f4ceeffd640 1 -- 172.21.15.88:0/4135683120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cf0159d70 con 0x7f4cf00d3310 2024-09-16T01:43:33.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.425+0000 7f4ceeffd640 1 --2- 172.21.15.88:0/4135683120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cf00d3310 0x7f4cf01596e0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4000c00 tx=0x7f4ce40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:33.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.425+0000 7f4cd7fff640 1 -- 172.21.15.88:0/4135683120 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4041020 con 0x7f4cf00d3310 2024-09-16T01:43:33.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.425+0000 7f4cd7fff640 1 -- 172.21.15.88:0/4135683120 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ce40306c0 con 0x7f4cf00d3310 2024-09-16T01:43:33.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.425+0000 7f4cf6015640 1 -- 172.21.15.88:0/4135683120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cf015a050 con 0x7f4cf00d3310 2024-09-16T01:43:33.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.425+0000 7f4cd7fff640 1 -- 172.21.15.88:0/4135683120 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4037040 con 0x7f4cf00d3310 2024-09-16T01:43:33.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.425+0000 7f4cf6015640 1 -- 172.21.15.88:0/4135683120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cf00ce8c0 con 0x7f4cf00d3310 2024-09-16T01:43:33.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.426+0000 7f4cf6015640 1 -- 172.21.15.88:0/4135683120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cf00ce6b0 con 0x7f4cf00d3310 2024-09-16T01:43:33.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.427+0000 7f4cd7fff640 1 -- 172.21.15.88:0/4135683120 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4ce4044020 con 0x7f4cf00d3310 2024-09-16T01:43:33.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.427+0000 7f4cd7fff640 1 --2- 172.21.15.88:0/4135683120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4cdc06c2b0 0x7f4cdc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:33.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.427+0000 7f4cd7fff640 1 -- 172.21.15.88:0/4135683120 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8576+0+0 (secure 0 0 0) 0x7f4ce40ac720 con 0x7f4cf00d3310 2024-09-16T01:43:33.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.427+0000 7f4cef7fe640 1 --2- 172.21.15.88:0/4135683120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4cdc06c2b0 0x7f4cdc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:33.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.428+0000 7f4cef7fe640 1 --2- 172.21.15.88:0/4135683120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4cdc06c2b0 0x7f4cdc06e770 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0006fd0 tx=0x7f4ce0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:33.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.431+0000 7f4cd7fff640 1 -- 172.21.15.88:0/4135683120 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4ce404b030 con 0x7f4cf00d3310 2024-09-16T01:43:33.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.632+0000 7f4cf6015640 1 -- 172.21.15.88:0/4135683120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4cf00d1110 con 0x7f4cf00d3310 2024-09-16T01:43:33.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.633+0000 7f4cd7fff640 1 -- 172.21.15.88:0/4135683120 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 267 v267) ==== 95+0+4392 (secure 0 0 0) 0x7f4ce4079cc0 con 0x7f4cf00d3310 2024-09-16T01:43:33.635 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:33.635 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":267,"btime":"2024-09-16T01:43:30:669481+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9696,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4270197461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4270197461},{"type":"v1","addr":"172.21.15.88:6837","nonce":4270197461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":267,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:43:30.669477+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:43:33.635 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 267 2024-09-16T01:43:33.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.635+0000 7f4cf6015640 1 -- 172.21.15.88:0/4135683120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4cdc06c2b0 msgr2=0x7f4cdc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:33.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.635+0000 7f4cf6015640 1 --2- 172.21.15.88:0/4135683120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4cdc06c2b0 0x7f4cdc06e770 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0006fd0 tx=0x7f4ce0008040 comp rx=0 tx=0).stop 2024-09-16T01:43:33.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.635+0000 7f4cf6015640 1 -- 172.21.15.88:0/4135683120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cf00d3310 msgr2=0x7f4cf01596e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:33.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.635+0000 7f4cf6015640 1 --2- 172.21.15.88:0/4135683120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cf00d3310 0x7f4cf01596e0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4000c00 tx=0x7f4ce40046c0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.635+0000 7f4cf6015640 1 -- 172.21.15.88:0/4135683120 shutdown_connections 2024-09-16T01:43:33.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.635+0000 7f4cf6015640 1 --2- 172.21.15.88:0/4135683120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4cdc06c2b0 0x7f4cdc06e770 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.635+0000 7f4cf6015640 1 --2- 172.21.15.88:0/4135683120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4cf015e660 0x7f4cf00cb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.635+0000 7f4cf6015640 1 --2- 172.21.15.88:0/4135683120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cf00d3310 0x7f4cf01596e0 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.635+0000 7f4cf6015640 1 --2- 172.21.15.88:0/4135683120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4cf00d1650 0x7f4cf00ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.635+0000 7f4cf6015640 1 -- 172.21.15.88:0/4135683120 >> 172.21.15.88:0/4135683120 conn(0x7f4cf00b6410 msgr2=0x7f4cf00b4d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:33.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.635+0000 7f4cf6015640 1 -- 172.21.15.88:0/4135683120 shutdown_connections 2024-09-16T01:43:33.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.635+0000 7f4cf6015640 1 -- 172.21.15.88:0/4135683120 wait complete. 2024-09-16T01:43:33.648 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:43:33.648 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:43:33.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.746+0000 7f25ed6d0640 1 Processor -- start 2024-09-16T01:43:33.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.746+0000 7f25ed6d0640 1 -- start start 2024-09-16T01:43:33.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.746+0000 7f25ed6d0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f25e806d540 0x7f25e80cf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:33.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.746+0000 7f25ed6d0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25e80735b0 0x7f25e80cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:33.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.746+0000 7f25ed6d0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25e806faf0 con 0x7f25e815e950 2024-09-16T01:43:33.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.746+0000 7f25ed6d0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f25e806fdd0 con 0x7f25e806d540 2024-09-16T01:43:33.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.746+0000 7f25ed6d0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f25e80700b0 con 0x7f25e80735b0 2024-09-16T01:43:33.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.747+0000 7f25e67fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f25e806d540 0x7f25e80cf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:33.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.747+0000 7f25e67fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f25e806d540 0x7f25e80cf0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60158/0 (socket says 172.21.15.88:60158) 2024-09-16T01:43:33.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.747+0000 7f25e67fc640 1 -- 172.21.15.88:0/1745326562 learned_addr learned my addr 172.21.15.88:0/1745326562 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:33.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.747+0000 7f25e77fe640 1 --2- 172.21.15.88:0/1745326562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25e80735b0 0x7f25e80cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:33.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.747+0000 7f25e67fc640 1 -- 172.21.15.88:0/1745326562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25e80735b0 msgr2=0x7f25e80cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:33.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.747+0000 7f25e67fc640 1 --2- 172.21.15.88:0/1745326562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25e80735b0 0x7f25e80cf5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.747+0000 7f25e67fc640 1 -- 172.21.15.88:0/1745326562 >> v1:172.21.15.154:6789/0 conn(0x7f25e815e950 legacy=0x7f25e806ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:33.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.747+0000 7f25e67fc640 1 -- 172.21.15.88:0/1745326562 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25e8070390 con 0x7f25e806d540 2024-09-16T01:43:33.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.747+0000 7f25e77fe640 1 --2- 172.21.15.88:0/1745326562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25e80735b0 0x7f25e80cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:33.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.748+0000 7f25e67fc640 1 --2- 172.21.15.88:0/1745326562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25e806d540 0x7f25e80cf0a0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f25d0009870 tx=0x7f25d002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cec433e72b499a5a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:33.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.748+0000 7f25e5ffb640 1 -- 172.21.15.88:0/1745326562 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d0037070 con 0x7f25e806d540 2024-09-16T01:43:33.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.748+0000 7f25e5ffb640 1 -- 172.21.15.88:0/1745326562 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25d0032440 con 0x7f25e806d540 2024-09-16T01:43:33.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.748+0000 7f25e5ffb640 1 -- 172.21.15.88:0/1745326562 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d0032810 con 0x7f25e806d540 2024-09-16T01:43:33.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.748+0000 7f25ed6d0640 1 -- 172.21.15.88:0/1745326562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25e806d540 msgr2=0x7f25e80cf0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:33.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.748+0000 7f25ed6d0640 1 --2- 172.21.15.88:0/1745326562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25e806d540 0x7f25e80cf0a0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f25d0009870 tx=0x7f25d002a210 comp rx=0 tx=0).stop 2024-09-16T01:43:33.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.748+0000 7f25ed6d0640 1 -- 172.21.15.88:0/1745326562 shutdown_connections 2024-09-16T01:43:33.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.748+0000 7f25ed6d0640 1 --2- 172.21.15.88:0/1745326562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25e80735b0 0x7f25e80cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.748+0000 7f25ed6d0640 1 --2- 172.21.15.88:0/1745326562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25e806d540 0x7f25e80cf0a0 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.748+0000 7f25ed6d0640 1 -- 172.21.15.88:0/1745326562 >> 172.21.15.88:0/1745326562 conn(0x7f25e80d14d0 msgr2=0x7f25e80d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:33.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.748+0000 7f25ed6d0640 1 -- 172.21.15.88:0/1745326562 shutdown_connections 2024-09-16T01:43:33.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.748+0000 7f25ed6d0640 1 -- 172.21.15.88:0/1745326562 wait complete. 2024-09-16T01:43:33.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.750+0000 7f25ed6d0640 1 Processor -- start 2024-09-16T01:43:33.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.750+0000 7f25ed6d0640 1 -- start start 2024-09-16T01:43:33.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.750+0000 7f25ed6d0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25e806d540 0x7f25e8159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:33.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.750+0000 7f25ed6d0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25e80735b0 0x7f25e8161680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:33.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.750+0000 7f25ed6d0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25e815e950 0x7f25e815a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:33.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.750+0000 7f25ed6d0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f25e80721d0 con 0x7f25e80735b0 2024-09-16T01:43:33.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.750+0000 7f25ed6d0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f25e8072050 con 0x7f25e815e950 2024-09-16T01:43:33.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.750+0000 7f25ed6d0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f25e8072350 con 0x7f25e806d540 2024-09-16T01:43:33.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.750+0000 7f25e67fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25e80735b0 0x7f25e8161680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:33.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.750+0000 7f25e67fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25e80735b0 0x7f25e8161680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60160/0 (socket says 172.21.15.88:60160) 2024-09-16T01:43:33.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.750+0000 7f25e67fc640 1 -- 172.21.15.88:0/4184895170 learned_addr learned my addr 172.21.15.88:0/4184895170 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:33.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.751+0000 7f25e77fe640 1 --2- 172.21.15.88:0/4184895170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25e815e950 0x7f25e815a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:33.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.751+0000 7f25e6ffd640 1 --2- 172.21.15.88:0/4184895170 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25e806d540 0x7f25e8159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:33.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.751+0000 7f25e67fc640 1 -- 172.21.15.88:0/4184895170 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25e806d540 msgr2=0x7f25e8159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:33.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.751+0000 7f25e67fc640 1 --2- 172.21.15.88:0/4184895170 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25e806d540 0x7f25e8159970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.751+0000 7f25e67fc640 1 -- 172.21.15.88:0/4184895170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25e815e950 msgr2=0x7f25e815a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:33.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.751+0000 7f25e67fc640 1 --2- 172.21.15.88:0/4184895170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25e815e950 0x7f25e815a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.751+0000 7f25e67fc640 1 -- 172.21.15.88:0/4184895170 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25e81588d0 con 0x7f25e80735b0 2024-09-16T01:43:33.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.751+0000 7f25e6ffd640 1 --2- 172.21.15.88:0/4184895170 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25e806d540 0x7f25e8159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:33.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.751+0000 7f25e67fc640 1 --2- 172.21.15.88:0/4184895170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25e80735b0 0x7f25e8161680 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f25d0009870 tx=0x7f25d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:33.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.751+0000 7f25df7fe640 1 -- 172.21.15.88:0/4184895170 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d0041020 con 0x7f25e80735b0 2024-09-16T01:43:33.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.751+0000 7f25df7fe640 1 -- 172.21.15.88:0/4184895170 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25d0030970 con 0x7f25e80735b0 2024-09-16T01:43:33.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.751+0000 7f25ed6d0640 1 -- 172.21.15.88:0/4184895170 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25e8158bb0 con 0x7f25e80735b0 2024-09-16T01:43:33.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.751+0000 7f25df7fe640 1 -- 172.21.15.88:0/4184895170 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d0037040 con 0x7f25e80735b0 2024-09-16T01:43:33.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.751+0000 7f25ed6d0640 1 -- 172.21.15.88:0/4184895170 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25e8159170 con 0x7f25e80735b0 2024-09-16T01:43:33.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.752+0000 7f25ed6d0640 1 -- 172.21.15.88:0/4184895170 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25e806ce30 con 0x7f25e80735b0 2024-09-16T01:43:33.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.752+0000 7f25df7fe640 1 -- 172.21.15.88:0/4184895170 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f25d0044020 con 0x7f25e80735b0 2024-09-16T01:43:33.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.753+0000 7f25df7fe640 1 --2- 172.21.15.88:0/4184895170 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f25c806c2b0 0x7f25c806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:33.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.753+0000 7f25df7fe640 1 -- 172.21.15.88:0/4184895170 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8576+0+0 (secure 0 0 0) 0x7f25d00ac8a0 con 0x7f25e80735b0 2024-09-16T01:43:33.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.753+0000 7f25e6ffd640 1 --2- 172.21.15.88:0/4184895170 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f25c806c2b0 0x7f25c806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:33.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.753+0000 7f25e6ffd640 1 --2- 172.21.15.88:0/4184895170 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f25c806c2b0 0x7f25c806e770 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f25d80045b0 tx=0x7f25d8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:33.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.757+0000 7f25df7fe640 1 -- 172.21.15.88:0/4184895170 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f25d004b030 con 0x7f25e80735b0 2024-09-16T01:43:33.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.957+0000 7f25ed6d0640 1 -- 172.21.15.88:0/4184895170 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25e80d0ec0 con 0x7f25e80735b0 2024-09-16T01:43:33.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.958+0000 7f25df7fe640 1 -- 172.21.15.88:0/4184895170 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 267 v267) ==== 95+0+4392 (secure 0 0 0) 0x7f25d007a520 con 0x7f25e80735b0 2024-09-16T01:43:33.959 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:33.959 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":267,"btime":"2024-09-16T01:43:30:669481+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9696,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4270197461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4270197461},{"type":"v1","addr":"172.21.15.88:6837","nonce":4270197461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":267,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:43:30.669477+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:43:33.960 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 267 2024-09-16T01:43:33.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.960+0000 7f25ed6d0640 1 -- 172.21.15.88:0/4184895170 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f25c806c2b0 msgr2=0x7f25c806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:33.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.960+0000 7f25ed6d0640 1 --2- 172.21.15.88:0/4184895170 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f25c806c2b0 0x7f25c806e770 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f25d80045b0 tx=0x7f25d8009040 comp rx=0 tx=0).stop 2024-09-16T01:43:33.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.960+0000 7f25ed6d0640 1 -- 172.21.15.88:0/4184895170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25e80735b0 msgr2=0x7f25e8161680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:33.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.960+0000 7f25ed6d0640 1 --2- 172.21.15.88:0/4184895170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25e80735b0 0x7f25e8161680 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f25d0009870 tx=0x7f25d00046c0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.960+0000 7f25ed6d0640 1 -- 172.21.15.88:0/4184895170 shutdown_connections 2024-09-16T01:43:33.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.960+0000 7f25ed6d0640 1 --2- 172.21.15.88:0/4184895170 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f25c806c2b0 0x7f25c806e770 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.960+0000 7f25ed6d0640 1 --2- 172.21.15.88:0/4184895170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25e815e950 0x7f25e815a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.960+0000 7f25ed6d0640 1 --2- 172.21.15.88:0/4184895170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25e80735b0 0x7f25e8161680 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.960+0000 7f25ed6d0640 1 --2- 172.21.15.88:0/4184895170 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25e806d540 0x7f25e8159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:33.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.960+0000 7f25ed6d0640 1 -- 172.21.15.88:0/4184895170 >> 172.21.15.88:0/4184895170 conn(0x7f25e80d14d0 msgr2=0x7f25e80b55f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:33.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.960+0000 7f25ed6d0640 1 -- 172.21.15.88:0/4184895170 shutdown_connections 2024-09-16T01:43:33.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:33.960+0000 7f25ed6d0640 1 -- 172.21.15.88:0/4184895170 wait complete. 2024-09-16T01:43:33.972 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt omap key 100.00000000:stray0_head' led to DAMAGED state 2024-09-16T01:43:33.972 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap key 1.00000000:subdir_head' 2024-09-16T01:43:33.973 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:43:33.973 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:43:33.988 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:43:33.988 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:43:33.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:33.989 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:43:34.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:43:34.044 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:43:34.044 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:43:34.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.182+0000 7fb1fc323640 1 Processor -- start 2024-09-16T01:43:34.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.182+0000 7fb1fc323640 1 -- start start 2024-09-16T01:43:34.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.183+0000 7fb1fc323640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1f406d6c0 0x7fb1f40cee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:34.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.183+0000 7fb1fc323640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1f406dba0 con 0x7fb1f415e6d0 2024-09-16T01:43:34.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.183+0000 7fb1fc323640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1f40d1070 con 0x7fb1f40cf350 2024-09-16T01:43:34.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.183+0000 7fb1fc323640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb1f40d1350 con 0x7fb1f406d6c0 2024-09-16T01:43:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.183+0000 7fb1fa0bf640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb1f415e6d0 0x7fb1f4073c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54014/0 (socket says 172.21.15.88:54014) 2024-09-16T01:43:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.183+0000 7fb1fa0bf640 1 -- 172.21.15.88:0/4263406172 learned_addr learned my addr 172.21.15.88:0/4263406172 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.183+0000 7fb1f90bd640 1 -- 172.21.15.88:0/4263406172 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2606916539 0 0) 0x7fb1f406dba0 con 0x7fb1f415e6d0 2024-09-16T01:43:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.183+0000 7fb1f98be640 1 --2- 172.21.15.88:0/4263406172 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1f406d6c0 0x7fb1f40cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.184+0000 7fb1f90bd640 1 -- 172.21.15.88:0/4263406172 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1dc003520 con 0x7fb1f415e6d0 2024-09-16T01:43:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.184+0000 7fb1f90bd640 1 -- 172.21.15.88:0/4263406172 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1641171154 0 0) 0x7fb1dc003520 con 0x7fb1f415e6d0 2024-09-16T01:43:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.184+0000 7fb1f90bd640 1 -- 172.21.15.88:0/4263406172 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1f406d6c0 msgr2=0x7fb1f40cee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.184+0000 7fb1f90bd640 1 --2- 172.21.15.88:0/4263406172 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1f406d6c0 0x7fb1f40cee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.184+0000 7fb1f90bd640 1 -- 172.21.15.88:0/4263406172 >> v1:172.21.15.154:6789/0 conn(0x7fb1f40cf350 legacy=0x7fb1f40d1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.184+0000 7fb1f90bd640 1 -- 172.21.15.88:0/4263406172 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1f415eab0 con 0x7fb1f415e6d0 2024-09-16T01:43:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.184+0000 7fb1f90bd640 1 -- 172.21.15.88:0/4263406172 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fb1e40019f0 con 0x7fb1f415e6d0 2024-09-16T01:43:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.184+0000 7fb1f90bd640 1 -- 172.21.15.88:0/4263406172 >> v1:172.21.15.88:6789/0 conn(0x7fb1f415e6d0 legacy=0x7fb1f4073c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:34.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.184+0000 7fb1f90bd640 1 --2- 172.21.15.88:0/4263406172 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1f406d6c0 0x7fb1dc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:34.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.184+0000 7fb1f90bd640 1 --2- 172.21.15.88:0/4263406172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1dc003f60 0x7fb1dc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:34.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.184+0000 7fb1f90bd640 1 --2- 172.21.15.88:0/4263406172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1dc004d40 0x7fb1dc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:34.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.184+0000 7fb1f90bd640 1 -- 172.21.15.88:0/4263406172 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f40d1350 con 0x7fb1dc004d40 2024-09-16T01:43:34.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.184+0000 7fb1f90bd640 1 -- 172.21.15.88:0/4263406172 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc00bb50 con 0x7fb1dc003f60 2024-09-16T01:43:34.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.184+0000 7fb1f90bd640 1 -- 172.21.15.88:0/4263406172 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb1dc00bd50 con 0x7fb1f406d6c0 2024-09-16T01:43:34.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.184+0000 7fb1fa8c0640 1 --2- 172.21.15.88:0/4263406172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1dc004d40 0x7fb1dc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:34.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.184+0000 7fb1fa0bf640 1 --2- 172.21.15.88:0/4263406172 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1f406d6c0 0x7fb1dc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:34.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.184+0000 7fb1f98be640 1 --2- 172.21.15.88:0/4263406172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1dc003f60 0x7fb1dc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:34.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.184+0000 7fb1fa8c0640 1 -- 172.21.15.88:0/4263406172 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1f406d6c0 msgr2=0x7fb1dc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:34.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.184+0000 7fb1fa8c0640 1 --2- 172.21.15.88:0/4263406172 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1f406d6c0 0x7fb1dc0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:34.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.184+0000 7fb1fa8c0640 1 -- 172.21.15.88:0/4263406172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1dc003f60 msgr2=0x7fb1dc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:34.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.184+0000 7fb1fa8c0640 1 --2- 172.21.15.88:0/4263406172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1dc003f60 0x7fb1dc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:34.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.184+0000 7fb1fa8c0640 1 -- 172.21.15.88:0/4263406172 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1dc003520 con 0x7fb1dc004d40 2024-09-16T01:43:34.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.185+0000 7fb1fa8c0640 1 --2- 172.21.15.88:0/4263406172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1dc004d40 0x7fb1dc0051c0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec001750 tx=0x7fb1dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=812496c18dbae872 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:34.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.185+0000 7fb1f90bd640 1 -- 172.21.15.88:0/4263406172 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec001370 con 0x7fb1dc004d40 2024-09-16T01:43:34.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.185+0000 7fb1f90bd640 1 -- 172.21.15.88:0/4263406172 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1ec0052d0 con 0x7fb1dc004d40 2024-09-16T01:43:34.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.185+0000 7fb1f90bd640 1 -- 172.21.15.88:0/4263406172 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec02abe0 con 0x7fb1dc004d40 2024-09-16T01:43:34.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.186+0000 7fb1fc323640 1 -- 172.21.15.88:0/4263406172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1dc004d40 msgr2=0x7fb1dc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:34.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.186+0000 7fb1fc323640 1 --2- 172.21.15.88:0/4263406172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1dc004d40 0x7fb1dc0051c0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec001750 tx=0x7fb1dc005830 comp rx=0 tx=0).stop 2024-09-16T01:43:34.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.186+0000 7fb1fa8c0640 1 -- 172.21.15.88:0/4263406172 reap_dead start 2024-09-16T01:43:34.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.186+0000 7fb1fc323640 1 -- 172.21.15.88:0/4263406172 shutdown_connections 2024-09-16T01:43:34.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.186+0000 7fb1fc323640 1 -- 172.21.15.88:0/4263406172 >> 172.21.15.88:0/4263406172 conn(0x7fb1f40b7d70 msgr2=0x7fb1f4062630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:34.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.186+0000 7fb1fc323640 1 -- 172.21.15.88:0/4263406172 shutdown_connections 2024-09-16T01:43:34.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.186+0000 7fb1fc323640 1 -- 172.21.15.88:0/4263406172 wait complete. 2024-09-16T01:43:34.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.187+0000 7fb1fc323640 1 Processor -- start 2024-09-16T01:43:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.187+0000 7fb1fc323640 1 -- start start 2024-09-16T01:43:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.188+0000 7fb1fc323640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1dc004d40 0x7fb1f41f3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.188+0000 7fb1fc323640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1f4157040 0x7fb1f4158f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.188+0000 7fb1fc323640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1f41574c0 0x7fb1f4159480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.188+0000 7fb1fc323640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f40cf1a0 con 0x7fb1f4157040 2024-09-16T01:43:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.188+0000 7fb1fc323640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f40cf020 con 0x7fb1dc004d40 2024-09-16T01:43:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.188+0000 7fb1fc323640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb1f40d2460 con 0x7fb1f41574c0 2024-09-16T01:43:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.188+0000 7fb1f98be640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1f4157040 0x7fb1f4158f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.188+0000 7fb1f98be640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1f4157040 0x7fb1f4158f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60176/0 (socket says 172.21.15.88:60176) 2024-09-16T01:43:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.188+0000 7fb1f98be640 1 -- 172.21.15.88:0/4242434018 learned_addr learned my addr 172.21.15.88:0/4242434018 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.188+0000 7fb1fa0bf640 1 --2- 172.21.15.88:0/4242434018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1dc004d40 0x7fb1f41f3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.188+0000 7fb1fa8c0640 1 --2- 172.21.15.88:0/4242434018 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1f41574c0 0x7fb1f4159480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:34.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.188+0000 7fb1f98be640 1 -- 172.21.15.88:0/4242434018 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1f41574c0 msgr2=0x7fb1f4159480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:34.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.188+0000 7fb1f98be640 1 --2- 172.21.15.88:0/4242434018 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1f41574c0 0x7fb1f4159480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:34.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.188+0000 7fb1f98be640 1 -- 172.21.15.88:0/4242434018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1dc004d40 msgr2=0x7fb1f41f3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:34.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.188+0000 7fb1f98be640 1 --2- 172.21.15.88:0/4242434018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1dc004d40 0x7fb1f41f3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:34.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.188+0000 7fb1f98be640 1 -- 172.21.15.88:0/4242434018 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1f4157a90 con 0x7fb1f4157040 2024-09-16T01:43:34.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.188+0000 7fb1fa8c0640 1 --2- 172.21.15.88:0/4242434018 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1f41574c0 0x7fb1f4159480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:34.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.188+0000 7fb1f98be640 1 --2- 172.21.15.88:0/4242434018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1f4157040 0x7fb1f4158f40 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8011910 tx=0x7fb1e8011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:34.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.188+0000 7fb1f2ffd640 1 -- 172.21.15.88:0/4242434018 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8019070 con 0x7fb1f4157040 2024-09-16T01:43:34.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.188+0000 7fb1f2ffd640 1 -- 172.21.15.88:0/4242434018 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1e8004240 con 0x7fb1f4157040 2024-09-16T01:43:34.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.188+0000 7fb1f2ffd640 1 -- 172.21.15.88:0/4242434018 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e800f8e0 con 0x7fb1f4157040 2024-09-16T01:43:34.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.188+0000 7fb1fc323640 1 -- 172.21.15.88:0/4242434018 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1f4157d70 con 0x7fb1f4157040 2024-09-16T01:43:34.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.188+0000 7fb1fc323640 1 -- 172.21.15.88:0/4242434018 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1f4158330 con 0x7fb1f4157040 2024-09-16T01:43:34.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.189+0000 7fb1fc323640 1 -- 172.21.15.88:0/4242434018 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1f40d1d50 con 0x7fb1f4157040 2024-09-16T01:43:34.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.190+0000 7fb1f2ffd640 1 -- 172.21.15.88:0/4242434018 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb1e801c070 con 0x7fb1f4157040 2024-09-16T01:43:34.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.190+0000 7fb1f2ffd640 1 --2- 172.21.15.88:0/4242434018 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb1e006c2e0 0x7fb1e006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:34.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.190+0000 7fb1f2ffd640 1 -- 172.21.15.88:0/4242434018 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8576+0+0 (secure 0 0 0) 0x7fb1e808c310 con 0x7fb1f4157040 2024-09-16T01:43:34.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.190+0000 7fb1fa0bf640 1 --2- 172.21.15.88:0/4242434018 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb1e006c2e0 0x7fb1e006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:34.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.191+0000 7fb1fa0bf640 1 --2- 172.21.15.88:0/4242434018 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb1e006c2e0 0x7fb1e006e7a0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fb1e40013d0 tx=0x7fb1e4006040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:34.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.195+0000 7fb1f2ffd640 1 -- 172.21.15.88:0/4242434018 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb1e8059940 con 0x7fb1f4157040 2024-09-16T01:43:34.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.395+0000 7fb1fc323640 1 -- 172.21.15.88:0/4242434018 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fb1f40b6510 con 0x7fb1f4157040 2024-09-16T01:43:34.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.672+0000 7fb1f2ffd640 1 -- 172.21.15.88:0/4242434018 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v268) ==== 161+0+0 (secure 0 0 0) 0x7fb1e805dc40 con 0x7fb1f4157040 2024-09-16T01:43:34.674 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:43:34.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.675+0000 7fb1fc323640 1 -- 172.21.15.88:0/4242434018 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb1e006c2e0 msgr2=0x7fb1e006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:34.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.675+0000 7fb1fc323640 1 --2- 172.21.15.88:0/4242434018 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb1e006c2e0 0x7fb1e006e7a0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fb1e40013d0 tx=0x7fb1e4006040 comp rx=0 tx=0).stop 2024-09-16T01:43:34.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.675+0000 7fb1fc323640 1 -- 172.21.15.88:0/4242434018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1f4157040 msgr2=0x7fb1f4158f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:34.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.675+0000 7fb1fc323640 1 --2- 172.21.15.88:0/4242434018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1f4157040 0x7fb1f4158f40 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8011910 tx=0x7fb1e8011dd0 comp rx=0 tx=0).stop 2024-09-16T01:43:34.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.675+0000 7fb1fc323640 1 -- 172.21.15.88:0/4242434018 shutdown_connections 2024-09-16T01:43:34.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.675+0000 7fb1fc323640 1 --2- 172.21.15.88:0/4242434018 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb1e006c2e0 0x7fb1e006e7a0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:34.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.675+0000 7fb1fc323640 1 --2- 172.21.15.88:0/4242434018 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1f41574c0 0x7fb1f4159480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:34.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.675+0000 7fb1fc323640 1 --2- 172.21.15.88:0/4242434018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb1f4157040 0x7fb1f4158f40 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:34.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.675+0000 7fb1fc323640 1 --2- 172.21.15.88:0/4242434018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1dc004d40 0x7fb1f41f3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:34.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.675+0000 7fb1fc323640 1 -- 172.21.15.88:0/4242434018 >> 172.21.15.88:0/4242434018 conn(0x7fb1f40b7d70 msgr2=0x7fb1f415d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:34.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.675+0000 7fb1fc323640 1 -- 172.21.15.88:0/4242434018 shutdown_connections 2024-09-16T01:43:34.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.675+0000 7fb1fc323640 1 -- 172.21.15.88:0/4242434018 wait complete. 2024-09-16T01:43:34.687 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:43:34.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.785+0000 7f482359e640 1 Processor -- start 2024-09-16T01:43:34.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.785+0000 7f482359e640 1 -- start start 2024-09-16T01:43:34.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.786+0000 7f482359e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f482415adf0 0x7f48240b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:34.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.786+0000 7f482359e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f482415b1d0 0x7f48240b1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:34.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.786+0000 7f482359e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48240b44f0 con 0x7f48240b22b0 2024-09-16T01:43:34.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.786+0000 7f482359e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f48240b47d0 con 0x7f482415adf0 2024-09-16T01:43:34.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.786+0000 7f482359e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f48240b4ab0 con 0x7f482415b1d0 2024-09-16T01:43:34.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f4822d9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f48240b22b0 0x7f482415ec80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54028/0 (socket says 172.21.15.88:54028) 2024-09-16T01:43:34.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f4822d9d640 1 -- 172.21.15.88:0/1145921409 learned_addr learned my addr 172.21.15.88:0/1145921409 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:34.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f482159a640 1 -- 172.21.15.88:0/1145921409 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3667411504 0 0) 0x7f48240b44f0 con 0x7f48240b22b0 2024-09-16T01:43:34.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f482259c640 1 --2- 172.21.15.88:0/1145921409 >> v2:172.21.15.154:3300/0 conn(0x7f482415adf0 0x7f48240b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:34.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f482159a640 1 -- 172.21.15.88:0/1145921409 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f480c005730 con 0x7f48240b22b0 2024-09-16T01:43:34.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f4821d9b640 1 --2- 172.21.15.88:0/1145921409 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f482415b1d0 0x7f48240b1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:34.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f482159a640 1 -- 172.21.15.88:0/1145921409 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3877580321 0 0) 0x7f480c005730 con 0x7f48240b22b0 2024-09-16T01:43:34.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f482159a640 1 -- 172.21.15.88:0/1145921409 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f482415b1d0 msgr2=0x7f48240b1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:34.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f482159a640 1 --2- 172.21.15.88:0/1145921409 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f482415b1d0 0x7f48240b1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:34.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f482159a640 1 -- 172.21.15.88:0/1145921409 >> v2:172.21.15.154:3300/0 conn(0x7f482415adf0 msgr2=0x7f48240b3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:34.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f482159a640 1 --2- 172.21.15.88:0/1145921409 >> v2:172.21.15.154:3300/0 conn(0x7f482415adf0 0x7f48240b3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:34.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f482159a640 1 -- 172.21.15.88:0/1145921409 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48240b4d90 con 0x7f48240b22b0 2024-09-16T01:43:34.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f482159a640 1 -- 172.21.15.88:0/1145921409 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f4814001670 con 0x7f48240b22b0 2024-09-16T01:43:34.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f482259c640 1 --2- 172.21.15.88:0/1145921409 >> v2:172.21.15.154:3300/0 conn(0x7f482415adf0 0x7f48240b3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:34.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f482159a640 1 -- 172.21.15.88:0/1145921409 >> v1:172.21.15.88:6789/0 conn(0x7f48240b22b0 legacy=0x7f482415ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:34.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f482159a640 1 --2- 172.21.15.88:0/1145921409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f480c005bc0 0x7f480c005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:34.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f482159a640 1 --2- 172.21.15.88:0/1145921409 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f482415b1d0 0x7f480c006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:34.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f482159a640 1 --2- 172.21.15.88:0/1145921409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f480c006f50 0x7f480c0073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:34.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f482159a640 1 -- 172.21.15.88:0/1145921409 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f48240b47d0 con 0x7f480c006f50 2024-09-16T01:43:34.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f482159a640 1 -- 172.21.15.88:0/1145921409 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f48240b4ab0 con 0x7f480c005bc0 2024-09-16T01:43:34.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f482159a640 1 -- 172.21.15.88:0/1145921409 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f480c00dd00 con 0x7f482415b1d0 2024-09-16T01:43:34.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f4822d9d640 1 --2- 172.21.15.88:0/1145921409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f480c006f50 0x7f480c0073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:34.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f482259c640 1 --2- 172.21.15.88:0/1145921409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f480c005bc0 0x7f480c005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:34.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f4821d9b640 1 --2- 172.21.15.88:0/1145921409 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f482415b1d0 0x7f480c006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:34.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f4822d9d640 1 -- 172.21.15.88:0/1145921409 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f482415b1d0 msgr2=0x7f480c006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:34.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f4822d9d640 1 --2- 172.21.15.88:0/1145921409 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f482415b1d0 0x7f480c006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:34.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f4822d9d640 1 -- 172.21.15.88:0/1145921409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f480c005bc0 msgr2=0x7f480c005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:34.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f4822d9d640 1 --2- 172.21.15.88:0/1145921409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f480c005bc0 0x7f480c005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:34.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f4822d9d640 1 -- 172.21.15.88:0/1145921409 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f480c005730 con 0x7f480c006f50 2024-09-16T01:43:34.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.787+0000 7f4821d9b640 1 --2- 172.21.15.88:0/1145921409 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f482415b1d0 0x7f480c006a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:34.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.788+0000 7f4822d9d640 1 --2- 172.21.15.88:0/1145921409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f480c006f50 0x7f480c0073d0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f481400c290 tx=0x7f480c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f19b9f587d09a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:34.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.788+0000 7f482159a640 1 -- 172.21.15.88:0/1145921409 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4814007d50 con 0x7f480c006f50 2024-09-16T01:43:34.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.788+0000 7f482159a640 1 -- 172.21.15.88:0/1145921409 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4814005af0 con 0x7f480c006f50 2024-09-16T01:43:34.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.788+0000 7f482159a640 1 -- 172.21.15.88:0/1145921409 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4814007070 con 0x7f480c006f50 2024-09-16T01:43:34.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.789+0000 7f482359e640 1 -- 172.21.15.88:0/1145921409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f480c006f50 msgr2=0x7f480c0073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:34.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.789+0000 7f482359e640 1 --2- 172.21.15.88:0/1145921409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f480c006f50 0x7f480c0073d0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f481400c290 tx=0x7f480c007a40 comp rx=0 tx=0).stop 2024-09-16T01:43:34.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.789+0000 7f4822d9d640 1 -- 172.21.15.88:0/1145921409 reap_dead start 2024-09-16T01:43:34.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.789+0000 7f482359e640 1 -- 172.21.15.88:0/1145921409 shutdown_connections 2024-09-16T01:43:34.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.789+0000 7f482359e640 1 -- 172.21.15.88:0/1145921409 >> 172.21.15.88:0/1145921409 conn(0x7f4824062a20 msgr2=0x7f482406bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:34.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.789+0000 7f482359e640 1 -- 172.21.15.88:0/1145921409 shutdown_connections 2024-09-16T01:43:34.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.789+0000 7f482359e640 1 -- 172.21.15.88:0/1145921409 wait complete. 2024-09-16T01:43:34.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.790+0000 7f482359e640 1 Processor -- start 2024-09-16T01:43:34.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.790+0000 7f482359e640 1 -- start start 2024-09-16T01:43:34.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.791+0000 7f482359e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f480c006f50 0x7f48240c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:34.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.791+0000 7f482359e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48241f4f70 0x7f482415ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:34.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.791+0000 7f482359e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f482415f0f0 0x7f48241f3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:34.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.791+0000 7f482359e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f48240b4030 con 0x7f480c006f50 2024-09-16T01:43:34.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.791+0000 7f482359e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f48240b5570 con 0x7f482415f0f0 2024-09-16T01:43:34.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.791+0000 7f482359e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4824071f10 con 0x7f48241f4f70 2024-09-16T01:43:34.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.791+0000 7f482259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f480c006f50 0x7f48240c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:34.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.791+0000 7f482259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f480c006f50 0x7f48240c65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60190/0 (socket says 172.21.15.88:60190) 2024-09-16T01:43:34.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.791+0000 7f482259c640 1 -- 172.21.15.88:0/4131757511 learned_addr learned my addr 172.21.15.88:0/4131757511 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:34.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.791+0000 7f4822d9d640 1 --2- 172.21.15.88:0/4131757511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f482415f0f0 0x7f48241f3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:34.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.791+0000 7f4821d9b640 1 --2- 172.21.15.88:0/4131757511 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48241f4f70 0x7f482415ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:34.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.791+0000 7f482259c640 1 -- 172.21.15.88:0/4131757511 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48241f4f70 msgr2=0x7f482415ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:34.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.791+0000 7f482259c640 1 --2- 172.21.15.88:0/4131757511 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48241f4f70 0x7f482415ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:34.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.791+0000 7f482259c640 1 -- 172.21.15.88:0/4131757511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f482415f0f0 msgr2=0x7f48241f3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:34.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.791+0000 7f482259c640 1 --2- 172.21.15.88:0/4131757511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f482415f0f0 0x7f48241f3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:34.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.791+0000 7f482259c640 1 -- 172.21.15.88:0/4131757511 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48240c5380 con 0x7f480c006f50 2024-09-16T01:43:34.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.791+0000 7f4821d9b640 1 --2- 172.21.15.88:0/4131757511 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48241f4f70 0x7f482415ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:34.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.792+0000 7f482259c640 1 --2- 172.21.15.88:0/4131757511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f480c006f50 0x7f48240c65f0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f481c0115d0 tx=0x7f481c011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:34.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.792+0000 7f4812ffd640 1 -- 172.21.15.88:0/4131757511 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c019070 con 0x7f480c006f50 2024-09-16T01:43:34.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.792+0000 7f4812ffd640 1 -- 172.21.15.88:0/4131757511 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f481c00f9b0 con 0x7f480c006f50 2024-09-16T01:43:34.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.792+0000 7f4812ffd640 1 -- 172.21.15.88:0/4131757511 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c00f040 con 0x7f480c006f50 2024-09-16T01:43:34.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.792+0000 7f482359e640 1 -- 172.21.15.88:0/4131757511 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48241f4a00 con 0x7f480c006f50 2024-09-16T01:43:34.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.792+0000 7f482359e640 1 -- 172.21.15.88:0/4131757511 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f482415cc10 con 0x7f480c006f50 2024-09-16T01:43:34.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.793+0000 7f482359e640 1 -- 172.21.15.88:0/4131757511 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48240ce050 con 0x7f480c006f50 2024-09-16T01:43:34.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.793+0000 7f4812ffd640 1 -- 172.21.15.88:0/4131757511 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f481c01c070 con 0x7f480c006f50 2024-09-16T01:43:34.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.793+0000 7f4812ffd640 1 --2- 172.21.15.88:0/4131757511 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f480406c2b0 0x7f480406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:34.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.794+0000 7f4812ffd640 1 -- 172.21.15.88:0/4131757511 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8576+0+0 (secure 0 0 0) 0x7f481c08c300 con 0x7f480c006f50 2024-09-16T01:43:34.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.794+0000 7f4821d9b640 1 --2- 172.21.15.88:0/4131757511 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f480406c2b0 0x7f480406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:34.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.794+0000 7f4821d9b640 1 --2- 172.21.15.88:0/4131757511 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f480406c2b0 0x7f480406e770 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f482415cac0 tx=0x7f481800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:34.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:34.798+0000 7f4812ffd640 1 -- 172.21.15.88:0/4131757511 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f48240c5380 con 0x7f480c006f50 2024-09-16T01:43:35.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.029+0000 7f482359e640 1 -- 172.21.15.88:0/4131757511 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f482406bfb0 con 0x7f480c006f50 2024-09-16T01:43:35.421 INFO:tasks.ceph.mds.a.smithi088.stderr: -45> 2024-09-16T01:43:30.667+0000 7f46ecc06640 -1 mds.0.damage notify_dentry Damage to dentries in fragment * of ino 0x100is fatal because it is a system directory for this rank 2024-09-16T01:43:35.712 INFO:tasks.ceph.mds.a.smithi088.stderr: -30> 2024-09-16T01:43:31.645+0000 7f46f0c0e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:43:35.714 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:43:35.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.679+0000 7f4812ffd640 1 -- 172.21.15.88:0/4131757511 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v269) ==== 101+0+0 (secure 0 0 0) 0x7f481c059ef0 con 0x7f480c006f50 2024-09-16T01:43:35.715 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:43:35.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.682+0000 7f482359e640 1 -- 172.21.15.88:0/4131757511 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f480406c2b0 msgr2=0x7f480406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:35.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.682+0000 7f482359e640 1 --2- 172.21.15.88:0/4131757511 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f480406c2b0 0x7f480406e770 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f482415cac0 tx=0x7f481800b040 comp rx=0 tx=0).stop 2024-09-16T01:43:35.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.682+0000 7f482359e640 1 -- 172.21.15.88:0/4131757511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f480c006f50 msgr2=0x7f48240c65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:35.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.682+0000 7f482359e640 1 --2- 172.21.15.88:0/4131757511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f480c006f50 0x7f48240c65f0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f481c0115d0 tx=0x7f481c011a90 comp rx=0 tx=0).stop 2024-09-16T01:43:35.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.682+0000 7f482359e640 1 -- 172.21.15.88:0/4131757511 shutdown_connections 2024-09-16T01:43:35.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.682+0000 7f482359e640 1 --2- 172.21.15.88:0/4131757511 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f480406c2b0 0x7f480406e770 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:35.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.682+0000 7f482359e640 1 --2- 172.21.15.88:0/4131757511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f482415f0f0 0x7f48241f3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:35.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.682+0000 7f482359e640 1 --2- 172.21.15.88:0/4131757511 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48241f4f70 0x7f482415ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:35.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.682+0000 7f482359e640 1 --2- 172.21.15.88:0/4131757511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f480c006f50 0x7f48240c65f0 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:35.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.682+0000 7f482359e640 1 -- 172.21.15.88:0/4131757511 >> 172.21.15.88:0/4131757511 conn(0x7f4824062a20 msgr2=0x7f48240b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:35.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.682+0000 7f482359e640 1 -- 172.21.15.88:0/4131757511 shutdown_connections 2024-09-16T01:43:35.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.682+0000 7f482359e640 1 -- 172.21.15.88:0/4131757511 wait complete. 2024-09-16T01:43:35.716 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:43:35.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.760+0000 7f5008481f40 1 Processor -- start 2024-09-16T01:43:35.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.760+0000 7f5008481f40 1 -- start start 2024-09-16T01:43:35.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.761+0000 7f5008481f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55898bf4b940 0x55898c0186b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:35.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.761+0000 7f5008481f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55898bf94b30 0x55898c01db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:35.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.761+0000 7f5008481f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55898c01ff30 0x55898c01e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:35.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.761+0000 7f5008481f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55898bf4c2c0 con 0x55898bf94b30 2024-09-16T01:43:35.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.761+0000 7f5008481f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55898be9c510 con 0x55898bf4b940 2024-09-16T01:43:35.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.761+0000 7f5008481f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55898bf49c80 con 0x55898c01ff30 2024-09-16T01:43:35.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.761+0000 7f4ffffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55898bf94b30 0x55898c01db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:35.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.761+0000 7f4ffffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55898bf94b30 0x55898c01db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60240/0 (socket says 172.21.15.88:60240) 2024-09-16T01:43:35.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.761+0000 7f4ffffff640 1 -- 172.21.15.88:0/1901093458 learned_addr learned my addr 172.21.15.88:0/1901093458 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:35.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.761+0000 7f5005f79640 1 --2- 172.21.15.88:0/1901093458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55898bf4b940 0x55898c0186b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:35.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.761+0000 7f500677a640 1 --2- 172.21.15.88:0/1901093458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55898c01ff30 0x55898c01e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:35.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.761+0000 7f4ffffff640 1 -- 172.21.15.88:0/1901093458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55898c01ff30 msgr2=0x55898c01e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:35.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.761+0000 7f4ffffff640 1 --2- 172.21.15.88:0/1901093458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55898c01ff30 0x55898c01e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:35.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.761+0000 7f4ffffff640 1 -- 172.21.15.88:0/1901093458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55898bf4b940 msgr2=0x55898c0186b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:35.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.761+0000 7f4ffffff640 1 --2- 172.21.15.88:0/1901093458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55898bf4b940 0x55898c0186b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:35.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.761+0000 7f4ffffff640 1 -- 172.21.15.88:0/1901093458 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55898bf91050 con 0x55898bf94b30 2024-09-16T01:43:35.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.761+0000 7f5005f79640 1 --2- 172.21.15.88:0/1901093458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55898bf4b940 0x55898c0186b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:35.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.761+0000 7f4ffffff640 1 --2- 172.21.15.88:0/1901093458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55898bf94b30 0x55898c01db20 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0002e10 tx=0x7f4ff000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8818233bb103f8d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:35.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.761+0000 7f5004f77640 1 -- 172.21.15.88:0/1901093458 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0013020 con 0x55898bf94b30 2024-09-16T01:43:35.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.761+0000 7f5004f77640 1 -- 172.21.15.88:0/1901093458 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ff000f400 con 0x55898bf94b30 2024-09-16T01:43:35.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.761+0000 7f5004f77640 1 -- 172.21.15.88:0/1901093458 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff000f7f0 con 0x55898bf94b30 2024-09-16T01:43:35.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.762+0000 7f5008481f40 1 -- 172.21.15.88:0/1901093458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55898bf94b30 msgr2=0x55898c01db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:35.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.762+0000 7f5008481f40 1 --2- 172.21.15.88:0/1901093458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55898bf94b30 0x55898c01db20 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0002e10 tx=0x7f4ff000b5b0 comp rx=0 tx=0).stop 2024-09-16T01:43:35.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.762+0000 7f5008481f40 1 -- 172.21.15.88:0/1901093458 shutdown_connections 2024-09-16T01:43:35.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.762+0000 7f5008481f40 1 --2- 172.21.15.88:0/1901093458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55898c01ff30 0x55898c01e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:35.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.762+0000 7f5008481f40 1 --2- 172.21.15.88:0/1901093458 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55898bf94b30 0x55898c01db20 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:35.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.762+0000 7f5008481f40 1 --2- 172.21.15.88:0/1901093458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55898bf4b940 0x55898c0186b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:35.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.762+0000 7f5008481f40 1 -- 172.21.15.88:0/1901093458 >> 172.21.15.88:0/1901093458 conn(0x55898bf95750 msgr2=0x55898bf76a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:35.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.762+0000 7f5008481f40 1 -- 172.21.15.88:0/1901093458 shutdown_connections 2024-09-16T01:43:35.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.762+0000 7f5008481f40 1 -- 172.21.15.88:0/1901093458 wait complete. 2024-09-16T01:43:35.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.762+0000 7f5008481f40 1 Processor -- start 2024-09-16T01:43:35.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.762+0000 7f5008481f40 1 -- start start 2024-09-16T01:43:35.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.763+0000 7f5008481f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55898bf4b940 0x55898c014c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:35.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.763+0000 7f5008481f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55898bf94b30 0x55898bf92400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:35.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.763+0000 7f5008481f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55898c01ff30 0x55898bf92940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:35.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.763+0000 7f5008481f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55898c01c480 con 0x55898c01ff30 2024-09-16T01:43:35.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.763+0000 7f5008481f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55898bf75050 con 0x55898bf94b30 2024-09-16T01:43:35.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.763+0000 7f5008481f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55898c01d600 con 0x55898bf4b940 2024-09-16T01:43:35.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.763+0000 7f500677a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55898c01ff30 0x55898bf92940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:35.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.763+0000 7f500677a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55898c01ff30 0x55898bf92940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60254/0 (socket says 172.21.15.88:60254) 2024-09-16T01:43:35.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.763+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 learned_addr learned my addr 172.21.15.88:0/3266065008 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:35.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.763+0000 7f5005f79640 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55898bf4b940 0x55898c014c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:35.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.763+0000 7f4ffffff640 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55898bf94b30 0x55898bf92400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:35.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.763+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55898bf4b940 msgr2=0x55898c014c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:35.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.764+0000 7f500677a640 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55898bf4b940 0x55898c014c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:35.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.764+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55898bf94b30 msgr2=0x55898bf92400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:35.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.764+0000 7f500677a640 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55898bf94b30 0x55898bf92400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:35.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.764+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55898c016960 con 0x55898c01ff30 2024-09-16T01:43:35.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.764+0000 7f500677a640 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55898c01ff30 0x55898bf92940 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f500000b4e0 tx=0x7f500000b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:35.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.764+0000 7f4ffeffd640 1 -- 172.21.15.88:0/3266065008 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5000016020 con 0x55898c01ff30 2024-09-16T01:43:35.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.764+0000 7f4ffeffd640 1 -- 172.21.15.88:0/3266065008 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5000012660 con 0x55898c01ff30 2024-09-16T01:43:35.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.764+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55898bf92f90 con 0x55898c01ff30 2024-09-16T01:43:35.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.764+0000 7f4ffeffd640 1 -- 172.21.15.88:0/3266065008 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5000012a30 con 0x55898c01ff30 2024-09-16T01:43:35.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.764+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55898bf93550 con 0x55898c01ff30 2024-09-16T01:43:35.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.765+0000 7f4ffeffd640 1 -- 172.21.15.88:0/3266065008 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5000019020 con 0x55898c01ff30 2024-09-16T01:43:35.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.766+0000 7f4ffeffd640 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4fe806acc0 0x7f4fe806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:35.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.766+0000 7f4ffeffd640 1 -- 172.21.15.88:0/3266065008 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8576+0+0 (secure 0 0 0) 0x7f500008ae70 con 0x55898c01ff30 2024-09-16T01:43:35.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.766+0000 7f5005f79640 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4fe806acc0 0x7f4fe806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:35.772 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:43:35.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.766+0000 7f5008481f40 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55898c0c1930 0x55898c0c3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:35.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.766+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c4260 con 0x55898c0c1930 2024-09-16T01:43:35.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.766+0000 7f4ffffff640 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55898c0c1930 0x55898c0c3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:35.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.766+0000 7f5005f79640 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4fe806acc0 0x7f4fe806d180 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8057000 tx=0x7f4ff805e040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:35.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.766+0000 7f4ffffff640 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55898c0c1930 0x55898c0c3d20 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:35.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.768+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v120'1002 uv976 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.769+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c4600 con 0x55898c0c1930 2024-09-16T01:43:35.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.770+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v120'1003 uv1003 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.770+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c49a0 con 0x55898c0c1930 2024-09-16T01:43:35.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.771+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v120'1004 uv1004 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.775 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:43:35.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.771+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c5d50 con 0x55898c0c1930 2024-09-16T01:43:35.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.773+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v120'1005 uv1005 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.773+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c60f0 con 0x55898c0c1930 2024-09-16T01:43:35.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.774+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v120'1006 uv1006 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.774+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c4d40 con 0x55898c0c1930 2024-09-16T01:43:35.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.775+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v120'1007 uv1007 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.775+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c5150 con 0x55898c0c1930 2024-09-16T01:43:35.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.777+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v120'1008 uv982 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.777+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c5150 con 0x55898c0c1930 2024-09-16T01:43:35.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.778+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v120'1009 uv1009 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.778+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c5100 con 0x55898c0c1930 2024-09-16T01:43:35.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.780+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v120'1010 uv1010 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.781 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:43:35.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.780+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c5100 con 0x55898c0c1930 2024-09-16T01:43:35.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.781+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v120'1011 uv1011 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.781+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c50e0 con 0x55898c0c1930 2024-09-16T01:43:35.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.782+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v120'1012 uv1012 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.782+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c50e0 con 0x55898c0c1930 2024-09-16T01:43:35.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.783+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v120'1013 uv1013 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.783+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c50e0 con 0x55898c0c1930 2024-09-16T01:43:35.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.785+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v120'1014 uv989 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.785+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c50e0 con 0x55898c0c1930 2024-09-16T01:43:35.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.786+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v120'1015 uv1015 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.786+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c50e0 con 0x55898c0c1930 2024-09-16T01:43:35.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.787+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v120'1016 uv1016 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.789 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:43:35.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.787+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c50e0 con 0x55898c0c1930 2024-09-16T01:43:35.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.788+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v120'1017 uv1017 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.788+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c50e0 con 0x55898c0c1930 2024-09-16T01:43:35.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.790+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v120'1018 uv1018 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.790+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c50e0 con 0x55898c0c1930 2024-09-16T01:43:35.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.791+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v120'1019 uv1019 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.791+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c55d0 con 0x55898c0c1930 2024-09-16T01:43:35.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.792+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v120'1020 uv1020 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.793 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.792+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c59b0 con 0x55898c0c1930 2024-09-16T01:43:35.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.794+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v120'1021 uv995 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.794+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c59b0 con 0x55898c0c1930 2024-09-16T01:43:35.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.795+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v120'1022 uv1022 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.795+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c59b0 con 0x55898c0c1930 2024-09-16T01:43:35.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.796+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v120'1023 uv1023 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.797 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:43:35.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.796+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c59b0 con 0x55898c0c1930 2024-09-16T01:43:35.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.797+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v120'1024 uv1024 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.797+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c55d0 con 0x55898c0c1930 2024-09-16T01:43:35.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.799+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v120'1025 uv1025 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.799+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c55d0 con 0x55898c0c1930 2024-09-16T01:43:35.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.800+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v120'1026 uv1026 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.800+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c55d0 con 0x55898c0c1930 2024-09-16T01:43:35.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.801+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v120'1027 uv1001 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.801+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c55d0 con 0x55898c0c1930 2024-09-16T01:43:35.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.802+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v120'1028 uv1028 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.802+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c55d0 con 0x55898c0c1930 2024-09-16T01:43:35.805 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:43:35.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.804+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v120'1029 uv1029 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.804+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c55d0 con 0x55898c0c1930 2024-09-16T01:43:35.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.805+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v120'1030 uv1030 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.805+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c55d0 con 0x55898c0c1930 2024-09-16T01:43:35.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.806+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v120'1031 uv1031 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.806+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c55d0 con 0x55898c0c1930 2024-09-16T01:43:35.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.807+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v120'1032 uv1032 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.807+0000 7f5008481f40 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55898c0c69a0 0x55898c0c8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:35.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.807+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c55d0 con 0x55898c0c69a0 2024-09-16T01:43:35.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.808+0000 7f5005f79640 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55898c0c69a0 0x55898c0c8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:35.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.808+0000 7f5005f79640 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55898c0c69a0 0x55898c0c8d90 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:35.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.810+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v120'728 uv710 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0c69a0 2024-09-16T01:43:35.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.810+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c92d0 con 0x55898c0c69a0 2024-09-16T01:43:35.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.811+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v120'729 uv729 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0c69a0 2024-09-16T01:43:35.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.811+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c9670 con 0x55898c0c69a0 2024-09-16T01:43:35.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.812+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v120'730 uv730 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0c69a0 2024-09-16T01:43:35.814 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:43:35.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.813+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0c9ec0 con 0x55898c0c69a0 2024-09-16T01:43:35.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.814+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v120'731 uv731 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0c69a0 2024-09-16T01:43:35.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.814+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0ca260 con 0x55898c0c69a0 2024-09-16T01:43:35.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.815+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v120'732 uv732 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0c69a0 2024-09-16T01:43:35.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.815+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0ca600 con 0x55898c0c69a0 2024-09-16T01:43:35.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.816+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v120'733 uv715 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0c69a0 2024-09-16T01:43:35.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.816+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e120) -- 0x55898c0ca9a0 con 0x55898c0c69a0 2024-09-16T01:43:35.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.817+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v120'734 uv734 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0c69a0 2024-09-16T01:43:35.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.817+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0ca9a0 con 0x55898c0c69a0 2024-09-16T01:43:35.819 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:43:35.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.818+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v120'735 uv735 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0c69a0 2024-09-16T01:43:35.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.818+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0ca9a0 con 0x55898c0c69a0 2024-09-16T01:43:35.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.819+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v120'736 uv736 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0c69a0 2024-09-16T01:43:35.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.819+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0ca9a0 con 0x55898c0c69a0 2024-09-16T01:43:35.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.820+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v120'737 uv737 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0c69a0 2024-09-16T01:43:35.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.820+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0ca9a0 con 0x55898c0c69a0 2024-09-16T01:43:35.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.822+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v120'738 uv720 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0c69a0 2024-09-16T01:43:35.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.822+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e120) -- 0x55898c0ca9a0 con 0x55898c0c69a0 2024-09-16T01:43:35.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.823+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v120'739 uv739 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0c69a0 2024-09-16T01:43:35.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.823+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0ca9a0 con 0x55898c0c69a0 2024-09-16T01:43:35.825 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:43:35.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.824+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v120'740 uv740 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0c69a0 2024-09-16T01:43:35.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.824+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0cee20 con 0x55898c0c69a0 2024-09-16T01:43:35.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.825+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v120'741 uv741 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0c69a0 2024-09-16T01:43:35.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.825+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0cee20 con 0x55898c0c69a0 2024-09-16T01:43:35.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.827+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v120'742 uv742 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0c69a0 2024-09-16T01:43:35.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.827+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0ca9a0 con 0x55898c0c69a0 2024-09-16T01:43:35.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.828+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v120'743 uv726 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0c69a0 2024-09-16T01:43:35.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.828+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e120) -- 0x55898c0ca9a0 con 0x55898c0c69a0 2024-09-16T01:43:35.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.829+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v120'744 uv744 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0c69a0 2024-09-16T01:43:35.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.829+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0ca9a0 con 0x55898c0c69a0 2024-09-16T01:43:35.831 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:43:35.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.830+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v120'745 uv745 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0c69a0 2024-09-16T01:43:35.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.830+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0cb9b0 con 0x55898c0c69a0 2024-09-16T01:43:35.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.831+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v120'746 uv746 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0c69a0 2024-09-16T01:43:35.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.831+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0cb9b0 con 0x55898c0c69a0 2024-09-16T01:43:35.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.832+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v120'747 uv747 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0c69a0 2024-09-16T01:43:35.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.832+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0ca9a0 con 0x55898c0c69a0 2024-09-16T01:43:35.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.833+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v120'748 uv748 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0c69a0 2024-09-16T01:43:35.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.833+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0ca9a0 con 0x55898c0c1930 2024-09-16T01:43:35.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.834+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v120'195 uv194 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.834+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e120) -- 0x55898c0ca9a0 con 0x55898c0c1930 2024-09-16T01:43:35.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.835+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v120'196 uv196 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.835+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0ca9a0 con 0x55898c0c1930 2024-09-16T01:43:35.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.836+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v120'197 uv197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.838 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:43:35.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.836+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0ca9a0 con 0x55898c0c1930 2024-09-16T01:43:35.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.837+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v120'198 uv198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.837+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0ca9a0 con 0x55898c0c1930 2024-09-16T01:43:35.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.838+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v120'199 uv199 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.838+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0ca9a0 con 0x55898c0c1930 2024-09-16T01:43:35.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.839+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v120'200 uv200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.839+0000 7f5008481f40 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55898c0cb2a0 0x55898c0cd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:35.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.839+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0cdbd0 con 0x55898c0cb2a0 2024-09-16T01:43:35.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.839+0000 7f4ffffff640 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55898c0cb2a0 0x55898c0cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:35.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.840+0000 7f4ffffff640 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55898c0cb2a0 0x55898c0cd690 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:35.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.841+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v120'146 uv145 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0cb2a0 2024-09-16T01:43:35.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.841+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e120) -- 0x55898c0cdfc0 con 0x55898c0cb2a0 2024-09-16T01:43:35.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.842+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v120'147 uv147 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0cb2a0 2024-09-16T01:43:35.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.842+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0cdfc0 con 0x55898c0cb2a0 2024-09-16T01:43:35.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.843+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v120'148 uv148 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0cb2a0 2024-09-16T01:43:35.844 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:43:35.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.843+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0cdfc0 con 0x55898c0cb2a0 2024-09-16T01:43:35.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.844+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v120'149 uv149 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0cb2a0 2024-09-16T01:43:35.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.844+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0cdfc0 con 0x55898c0c1930 2024-09-16T01:43:35.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.845+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v120'195 uv194 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.845+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e120) -- 0x55898c0cdfc0 con 0x55898c0c1930 2024-09-16T01:43:35.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.846+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v120'196 uv196 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.846+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0cdfc0 con 0x55898c0c1930 2024-09-16T01:43:35.850 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:43:35.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.849+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v120'197 uv197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.849+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0cefd0 con 0x55898c0c1930 2024-09-16T01:43:35.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.852+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v120'198 uv198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.852+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0cefd0 con 0x55898c0c1930 2024-09-16T01:43:35.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.853+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v120'199 uv199 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.853+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0cdfc0 con 0x55898c0c1930 2024-09-16T01:43:35.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.856+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v120'200 uv200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.856+0000 7f5008481f40 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55898c0ce980 0x55898c0d0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:35.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.856+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d12b0 con 0x55898c0ce980 2024-09-16T01:43:35.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.856+0000 7f500677a640 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55898c0ce980 0x55898c0d0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:35.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.856+0000 7f500677a640 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55898c0ce980 0x55898c0d0d70 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:35.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.858+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v120'360 uv354 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.858+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d1650 con 0x55898c0ce980 2024-09-16T01:43:35.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.859+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v120'361 uv361 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.859+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d1a70 con 0x55898c0ce980 2024-09-16T01:43:35.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.860+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v120'362 uv362 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.861 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:43:35.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.860+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d3320 con 0x55898c0ce980 2024-09-16T01:43:35.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.861+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v120'363 uv363 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.861+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d36c0 con 0x55898c0ce980 2024-09-16T01:43:35.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.862+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v120'364 uv364 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.862+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d2310 con 0x55898c0ce980 2024-09-16T01:43:35.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.863+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v120'365 uv365 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.863+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d2700 con 0x55898c0ce980 2024-09-16T01:43:35.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.864+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v120'366 uv359 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.864+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d2af0 con 0x55898c0ce980 2024-09-16T01:43:35.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.865+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v120'367 uv367 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.865+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d2af0 con 0x55898c0ce980 2024-09-16T01:43:35.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.866+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v120'368 uv368 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.867 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:43:35.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.866+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d2aa0 con 0x55898c0ce980 2024-09-16T01:43:35.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.867+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v120'369 uv369 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.867+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d2aa0 con 0x55898c0ce980 2024-09-16T01:43:35.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.868+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v120'370 uv370 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.868+0000 7f5008481f40 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55898c0d3a60 0x55898c0d5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:35.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.868+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d6390 con 0x55898c0d3a60 2024-09-16T01:43:35.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.868+0000 7f5005f79640 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55898c0d3a60 0x55898c0d5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:35.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.869+0000 7f5005f79640 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55898c0d3a60 0x55898c0d5e50 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:35.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.870+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v120'163 uv162 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0d3a60 2024-09-16T01:43:35.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.870+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d6390 con 0x55898c0d3a60 2024-09-16T01:43:35.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.871+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v120'164 uv164 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0d3a60 2024-09-16T01:43:35.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.871+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d6390 con 0x55898c0d3a60 2024-09-16T01:43:35.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.872+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v120'165 uv165 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0d3a60 2024-09-16T01:43:35.873 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:43:35.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.872+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d63f0 con 0x55898c0d3a60 2024-09-16T01:43:35.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.873+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v120'166 uv166 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0d3a60 2024-09-16T01:43:35.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.873+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d63b0 con 0x55898c0d3a60 2024-09-16T01:43:35.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.874+0000 7f5005f79640 1 -- 172.21.15.88:0/3266065008 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v120'167 uv167 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f4ff8057030 con 0x55898c0d3a60 2024-09-16T01:43:35.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.874+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d6390 con 0x55898c0c1930 2024-09-16T01:43:35.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.875+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v120'195 uv194 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.875+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d6390 con 0x55898c0c1930 2024-09-16T01:43:35.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.876+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v120'196 uv196 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.876+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d6390 con 0x55898c0c1930 2024-09-16T01:43:35.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.877+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v120'197 uv197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.878 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:43:35.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.877+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d63d0 con 0x55898c0c1930 2024-09-16T01:43:35.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.878+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v120'198 uv198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.878+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d6390 con 0x55898c0c1930 2024-09-16T01:43:35.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.879+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v120'199 uv199 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.879+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d6390 con 0x55898c0c1930 2024-09-16T01:43:35.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.880+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v120'200 uv200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0c1930 2024-09-16T01:43:35.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.880+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d6390 con 0x55898c0cb2a0 2024-09-16T01:43:35.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.881+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v120'325 uv318 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0cb2a0 2024-09-16T01:43:35.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.881+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d6390 con 0x55898c0cb2a0 2024-09-16T01:43:35.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.882+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v120'326 uv326 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0cb2a0 2024-09-16T01:43:35.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.882+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d6390 con 0x55898c0cb2a0 2024-09-16T01:43:35.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.883+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v120'327 uv327 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0cb2a0 2024-09-16T01:43:35.884 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:43:35.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.883+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d6880 con 0x55898c0cb2a0 2024-09-16T01:43:35.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.884+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v120'328 uv328 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0cb2a0 2024-09-16T01:43:35.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.884+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d6880 con 0x55898c0cb2a0 2024-09-16T01:43:35.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.885+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v120'329 uv329 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0cb2a0 2024-09-16T01:43:35.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.885+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d6880 con 0x55898c0cb2a0 2024-09-16T01:43:35.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.886+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v120'330 uv324 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0cb2a0 2024-09-16T01:43:35.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.886+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d6880 con 0x55898c0cb2a0 2024-09-16T01:43:35.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.887+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v120'331 uv331 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0cb2a0 2024-09-16T01:43:35.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.887+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d6880 con 0x55898c0cb2a0 2024-09-16T01:43:35.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.888+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v120'332 uv332 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0cb2a0 2024-09-16T01:43:35.889 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:43:35.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.888+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d6880 con 0x55898c0cb2a0 2024-09-16T01:43:35.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.889+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v120'333 uv333 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0cb2a0 2024-09-16T01:43:35.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.889+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d95a0 con 0x55898c0cb2a0 2024-09-16T01:43:35.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.890+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v120'334 uv334 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0cb2a0 2024-09-16T01:43:35.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.890+0000 7f5008481f40 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55898c0d7050 0x55898c0d9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:35.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.890+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d9a60 con 0x55898c0d7050 2024-09-16T01:43:35.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.890+0000 7f4ffffff640 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55898c0d7050 0x55898c0d9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:35.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.890+0000 7f4ffffff640 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55898c0d7050 0x55898c0d9440 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:35.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.891+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v120'142 uv140 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0d7050 2024-09-16T01:43:35.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.891+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d9a60 con 0x55898c0d7050 2024-09-16T01:43:35.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.892+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v120'143 uv143 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0d7050 2024-09-16T01:43:35.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.892+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d9a60 con 0x55898c0d7050 2024-09-16T01:43:35.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.893+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v120'144 uv144 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0d7050 2024-09-16T01:43:35.894 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:43:35.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.893+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d9a60 con 0x55898c0d7050 2024-09-16T01:43:35.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.894+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v120'145 uv145 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0d7050 2024-09-16T01:43:35.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.894+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d9e50 con 0x55898c0cb2a0 2024-09-16T01:43:35.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.895+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v120'195 uv194 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0cb2a0 2024-09-16T01:43:35.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.895+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d9e50 con 0x55898c0cb2a0 2024-09-16T01:43:35.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.896+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v120'196 uv196 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0cb2a0 2024-09-16T01:43:35.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.896+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d9ae0 con 0x55898c0cb2a0 2024-09-16T01:43:35.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.897+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v120'197 uv197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0cb2a0 2024-09-16T01:43:35.899 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:43:35.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.897+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0dab30 con 0x55898c0cb2a0 2024-09-16T01:43:35.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.898+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v120'198 uv198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0cb2a0 2024-09-16T01:43:35.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.898+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0daae0 con 0x55898c0cb2a0 2024-09-16T01:43:35.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.899+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v120'199 uv199 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0cb2a0 2024-09-16T01:43:35.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.899+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0daaa0 con 0x55898c0cb2a0 2024-09-16T01:43:35.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.900+0000 7f4ffffff640 1 -- 172.21.15.88:0/3266065008 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v120'200 uv200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4ff000f5b0 con 0x55898c0cb2a0 2024-09-16T01:43:35.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.900+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0daaa0 con 0x55898c0ce980 2024-09-16T01:43:35.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.901+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v120'166 uv165 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.902+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e120) -- 0x55898c0daaa0 con 0x55898c0ce980 2024-09-16T01:43:35.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.903+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v120'167 uv167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.903+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0daaa0 con 0x55898c0ce980 2024-09-16T01:43:35.905 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:43:35.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.904+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v120'168 uv168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.904+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0daaa0 con 0x55898c0ce980 2024-09-16T01:43:35.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.905+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v120'169 uv169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.905+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0daaa0 con 0x55898c0ce980 2024-09-16T01:43:35.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.906+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v120'170 uv170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.906+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0daaa0 con 0x55898c0ce980 2024-09-16T01:43:35.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.907+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v120'402 uv393 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.907+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e120) -- 0x55898c0dab00 con 0x55898c0ce980 2024-09-16T01:43:35.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.908+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v120'403 uv403 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.908+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0dab00 con 0x55898c0ce980 2024-09-16T01:43:35.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.910+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v120'404 uv404 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.911 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:43:35.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.910+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0dab00 con 0x55898c0ce980 2024-09-16T01:43:35.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.911+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v120'405 uv405 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.911+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0daaa0 con 0x55898c0ce980 2024-09-16T01:43:35.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.912+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v120'406 uv406 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.912+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0daaa0 con 0x55898c0ce980 2024-09-16T01:43:35.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.913+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v120'407 uv400 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.913+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e120) -- 0x55898c0daaa0 con 0x55898c0ce980 2024-09-16T01:43:35.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.914+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v120'408 uv408 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.914+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0daaa0 con 0x55898c0ce980 2024-09-16T01:43:35.916 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:43:35.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.915+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v120'409 uv409 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.915+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0daaa0 con 0x55898c0ce980 2024-09-16T01:43:35.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.916+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v120'410 uv410 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.916+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0daaa0 con 0x55898c0ce980 2024-09-16T01:43:35.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.917+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v120'411 uv411 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.917+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0daaa0 con 0x55898c0ce980 2024-09-16T01:43:35.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.918+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v120'412 uv412 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.918+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x55898c0d9b00 con 0x55898c0ce980 2024-09-16T01:43:35.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.919+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v120'413 uv413 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f5000054b10 con 0x55898c0ce980 2024-09-16T01:43:35.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.919+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55898c0c1930 msgr2=0x55898c0c3d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:35.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.919+0000 7f5008481f40 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55898c0c1930 0x55898c0c3d20 crc :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:35.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.919+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55898c0cb2a0 msgr2=0x55898c0cd690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:35.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.919+0000 7f5008481f40 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55898c0cb2a0 0x55898c0cd690 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:35.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.919+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55898c0d7050 msgr2=0x55898c0d9440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:35.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.919+0000 7f5008481f40 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55898c0d7050 0x55898c0d9440 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:35.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.919+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55898c0c69a0 msgr2=0x55898c0c8d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:35.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.919+0000 7f5008481f40 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55898c0c69a0 0x55898c0c8d90 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:35.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.919+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 reap_dead start 2024-09-16T01:43:35.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.919+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55898c0ce980 msgr2=0x55898c0d0d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:35.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.919+0000 7f5008481f40 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55898c0ce980 0x55898c0d0d70 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:35.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.919+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55898c0d3a60 msgr2=0x55898c0d5e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:35.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.919+0000 7f5008481f40 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55898c0d3a60 0x55898c0d5e50 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:35.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.919+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4fe806acc0 msgr2=0x7f4fe806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:35.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.919+0000 7f5008481f40 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4fe806acc0 0x7f4fe806d180 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8057000 tx=0x7f4ff805e040 comp rx=0 tx=0).stop 2024-09-16T01:43:35.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.919+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55898c01ff30 msgr2=0x55898bf92940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:35.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.919+0000 7f5008481f40 1 --2- 172.21.15.88:0/3266065008 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55898c01ff30 0x55898bf92940 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f500000b4e0 tx=0x7f500000b9a0 comp rx=0 tx=0).stop 2024-09-16T01:43:35.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.919+0000 7f500677a640 1 -- 172.21.15.88:0/3266065008 reap_dead start 2024-09-16T01:43:35.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.919+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 shutdown_connections 2024-09-16T01:43:35.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.919+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 >> 172.21.15.88:0/3266065008 conn(0x55898bf95750 msgr2=0x55898c01bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:35.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.919+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 shutdown_connections 2024-09-16T01:43:35.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:35.919+0000 7f5008481f40 1 -- 172.21.15.88:0/3266065008 wait complete. 2024-09-16T01:43:35.924 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 1.00000000 subdir_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-09-16T01:43:36.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.008+0000 7fd39b086f40 1 Processor -- start 2024-09-16T01:43:36.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.008+0000 7fd39b086f40 1 -- start start 2024-09-16T01:43:36.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.008+0000 7fd39b086f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558251ffa940 0x55825215d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:36.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.008+0000 7fd39b086f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558251ffad50 0x558252028810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:36.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.008+0000 7fd39b086f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5582520cef30 0x558252025320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:36.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.008+0000 7fd39b086f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558251ffb2c0 con 0x558251ffa940 2024-09-16T01:43:36.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.008+0000 7fd39b086f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558251f4b510 con 0x558251ffad50 2024-09-16T01:43:36.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.008+0000 7fd39b086f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x558251ff8c80 con 0x5582520cef30 2024-09-16T01:43:36.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.009+0000 7fd398b7c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558251ffa940 0x55825215d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:36.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.009+0000 7fd398b7c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558251ffa940 0x55825215d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60272/0 (socket says 172.21.15.88:60272) 2024-09-16T01:43:36.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.009+0000 7fd398b7c640 1 -- 172.21.15.88:0/3924403235 learned_addr learned my addr 172.21.15.88:0/3924403235 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:36.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.009+0000 7fd39037b640 1 --2- 172.21.15.88:0/3924403235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558251ffad50 0x558252028810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:36.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.009+0000 7fd39937d640 1 --2- 172.21.15.88:0/3924403235 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5582520cef30 0x558252025320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:36.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.009+0000 7fd398b7c640 1 -- 172.21.15.88:0/3924403235 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5582520cef30 msgr2=0x558252025320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:36.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.009+0000 7fd398b7c640 1 --2- 172.21.15.88:0/3924403235 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5582520cef30 0x558252025320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:36.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.009+0000 7fd398b7c640 1 -- 172.21.15.88:0/3924403235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558251ffad50 msgr2=0x558252028810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:36.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.009+0000 7fd398b7c640 1 --2- 172.21.15.88:0/3924403235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558251ffad50 0x558252028810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:36.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.009+0000 7fd398b7c640 1 -- 172.21.15.88:0/3924403235 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582520cb610 con 0x558251ffa940 2024-09-16T01:43:36.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.009+0000 7fd398b7c640 1 --2- 172.21.15.88:0/3924403235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558251ffa940 0x55825215d9d0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fd3880015f0 tx=0x7fd388001ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=600c9ce7b7f17c2f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:36.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.009+0000 7fd3937fe640 1 -- 172.21.15.88:0/3924403235 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd388016070 con 0x558251ffa940 2024-09-16T01:43:36.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.009+0000 7fd3937fe640 1 -- 172.21.15.88:0/3924403235 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd38800f730 con 0x558251ffa940 2024-09-16T01:43:36.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.009+0000 7fd3937fe640 1 -- 172.21.15.88:0/3924403235 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd38800fb20 con 0x558251ffa940 2024-09-16T01:43:36.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.010+0000 7fd39b086f40 1 -- 172.21.15.88:0/3924403235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558251ffa940 msgr2=0x55825215d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:36.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.010+0000 7fd39b086f40 1 --2- 172.21.15.88:0/3924403235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558251ffa940 0x55825215d9d0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fd3880015f0 tx=0x7fd388001ab0 comp rx=0 tx=0).stop 2024-09-16T01:43:36.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.010+0000 7fd39b086f40 1 -- 172.21.15.88:0/3924403235 shutdown_connections 2024-09-16T01:43:36.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.010+0000 7fd39b086f40 1 --2- 172.21.15.88:0/3924403235 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5582520cef30 0x558252025320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:36.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.010+0000 7fd39b086f40 1 --2- 172.21.15.88:0/3924403235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558251ffad50 0x558252028810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:36.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.010+0000 7fd39b086f40 1 --2- 172.21.15.88:0/3924403235 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558251ffa940 0x55825215d9d0 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:36.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.010+0000 7fd39b086f40 1 -- 172.21.15.88:0/3924403235 >> 172.21.15.88:0/3924403235 conn(0x558252044750 msgr2=0x558252024830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:36.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.010+0000 7fd39b086f40 1 -- 172.21.15.88:0/3924403235 shutdown_connections 2024-09-16T01:43:36.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.010+0000 7fd39b086f40 1 -- 172.21.15.88:0/3924403235 wait complete. 2024-09-16T01:43:36.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.010+0000 7fd39b086f40 1 Processor -- start 2024-09-16T01:43:36.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.010+0000 7fd39b086f40 1 -- start start 2024-09-16T01:43:36.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.011+0000 7fd39b086f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558251ffa940 0x55825203afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:36.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.011+0000 7fd39b086f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558251ffad50 0x5582520245b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:36.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.011+0000 7fd39b086f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5582520cef30 0x55825203ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:36.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.011+0000 7fd39b086f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5582520cbc10 con 0x558251ffad50 2024-09-16T01:43:36.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.011+0000 7fd39b086f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5582520cb9c0 con 0x558251ffa940 2024-09-16T01:43:36.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.011+0000 7fd39b086f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x558252023f30 con 0x5582520cef30 2024-09-16T01:43:36.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.011+0000 7fd39037b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558251ffad50 0x5582520245b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:36.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.011+0000 7fd39037b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558251ffad50 0x5582520245b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60288/0 (socket says 172.21.15.88:60288) 2024-09-16T01:43:36.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.011+0000 7fd39037b640 1 -- 172.21.15.88:0/2682964025 learned_addr learned my addr 172.21.15.88:0/2682964025 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:36.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.011+0000 7fd398b7c640 1 --2- 172.21.15.88:0/2682964025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558251ffa940 0x55825203afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:36.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.011+0000 7fd39937d640 1 --2- 172.21.15.88:0/2682964025 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5582520cef30 0x55825203ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:36.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.011+0000 7fd39037b640 1 -- 172.21.15.88:0/2682964025 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5582520cef30 msgr2=0x55825203ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:36.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.011+0000 7fd39037b640 1 --2- 172.21.15.88:0/2682964025 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5582520cef30 0x55825203ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:36.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.011+0000 7fd39037b640 1 -- 172.21.15.88:0/2682964025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558251ffa940 msgr2=0x55825203afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:36.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.011+0000 7fd39037b640 1 --2- 172.21.15.88:0/2682964025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558251ffa940 0x55825203afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:36.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.011+0000 7fd39037b640 1 -- 172.21.15.88:0/2682964025 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55825203b4e0 con 0x558251ffad50 2024-09-16T01:43:36.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.011+0000 7fd39037b640 1 --2- 172.21.15.88:0/2682964025 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558251ffad50 0x5582520245b0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fd380025e20 tx=0x7fd380002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:36.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.012+0000 7fd3927fc640 1 -- 172.21.15.88:0/2682964025 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd380068070 con 0x558251ffad50 2024-09-16T01:43:36.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.012+0000 7fd3927fc640 1 -- 172.21.15.88:0/2682964025 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd380050c70 con 0x558251ffad50 2024-09-16T01:43:36.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.012+0000 7fd39b086f40 1 -- 172.21.15.88:0/2682964025 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x558252026090 con 0x558251ffad50 2024-09-16T01:43:36.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.012+0000 7fd3927fc640 1 -- 172.21.15.88:0/2682964025 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd380063620 con 0x558251ffad50 2024-09-16T01:43:36.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.012+0000 7fd39b086f40 1 -- 172.21.15.88:0/2682964025 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582520c6ff0 con 0x558251ffad50 2024-09-16T01:43:36.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.013+0000 7fd3927fc640 1 -- 172.21.15.88:0/2682964025 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd38006b070 con 0x558251ffad50 2024-09-16T01:43:36.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.013+0000 7fd3927fc640 1 --2- 172.21.15.88:0/2682964025 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd37806acc0 0x7fd37806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:36.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.014+0000 7fd3927fc640 1 -- 172.21.15.88:0/2682964025 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8576+0+0 (secure 0 0 0) 0x7fd3800ce680 con 0x558251ffad50 2024-09-16T01:43:36.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.014+0000 7fd398b7c640 1 --2- 172.21.15.88:0/2682964025 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd37806acc0 0x7fd37806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:36.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.014+0000 7fd39b086f40 1 --2- 172.21.15.88:0/2682964025 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x558252024af0 0x5582520cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:36.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.014+0000 7fd39b086f40 1 -- 172.21.15.88:0/2682964025 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:1 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=103b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120) -- 0x558252025a30 con 0x558252024af0 2024-09-16T01:43:36.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.014+0000 7fd398b7c640 1 --2- 172.21.15.88:0/2682964025 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd37806acc0 0x7fd37806d180 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd388014040 tx=0x7fd388008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:36.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.014+0000 7fd39937d640 1 --2- 172.21.15.88:0/2682964025 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x558252024af0 0x5582520cd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:36.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.014+0000 7fd39937d640 1 --2- 172.21.15.88:0/2682964025 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x558252024af0 0x5582520cd8e0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:36.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.015+0000 7fd39937d640 1 -- 172.21.15.88:0/2682964025 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(1 1.00000000 [omap-set-vals] v120'414 uv414 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fd3940099b0 con 0x558252024af0 2024-09-16T01:43:36.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.015+0000 7fd39b086f40 1 -- 172.21.15.88:0/2682964025 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x558252024af0 msgr2=0x5582520cd8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:36.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.016+0000 7fd39b086f40 1 --2- 172.21.15.88:0/2682964025 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x558252024af0 0x5582520cd8e0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:36.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.016+0000 7fd39b086f40 1 -- 172.21.15.88:0/2682964025 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd37806acc0 msgr2=0x7fd37806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:36.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.016+0000 7fd39b086f40 1 --2- 172.21.15.88:0/2682964025 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd37806acc0 0x7fd37806d180 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd388014040 tx=0x7fd388008030 comp rx=0 tx=0).stop 2024-09-16T01:43:36.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.016+0000 7fd39b086f40 1 -- 172.21.15.88:0/2682964025 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558251ffad50 msgr2=0x5582520245b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:36.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.016+0000 7fd39b086f40 1 --2- 172.21.15.88:0/2682964025 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558251ffad50 0x5582520245b0 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fd380025e20 tx=0x7fd380002fb0 comp rx=0 tx=0).stop 2024-09-16T01:43:36.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.016+0000 7fd39937d640 1 -- 172.21.15.88:0/2682964025 reap_dead start 2024-09-16T01:43:36.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.016+0000 7fd39b086f40 1 -- 172.21.15.88:0/2682964025 shutdown_connections 2024-09-16T01:43:36.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.016+0000 7fd39b086f40 1 -- 172.21.15.88:0/2682964025 >> 172.21.15.88:0/2682964025 conn(0x558252044750 msgr2=0x558252025320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:36.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.016+0000 7fd39b086f40 1 -- 172.21.15.88:0/2682964025 shutdown_connections 2024-09-16T01:43:36.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.016+0000 7fd39b086f40 1 -- 172.21.15.88:0/2682964025 wait complete. 2024-09-16T01:43:36.021 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:43:36.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.120+0000 7fa0918ba640 1 Processor -- start 2024-09-16T01:43:36.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.120+0000 7fa0918ba640 1 -- start start 2024-09-16T01:43:36.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.121+0000 7fa0918ba640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa08c0b54d0 0x7fa08c06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:36.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.121+0000 7fa0918ba640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa08c0b7450 0x7fa08c15f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:36.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.121+0000 7fa0918ba640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa08c06e990 con 0x7fa08c15e6f0 2024-09-16T01:43:36.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.121+0000 7fa0918ba640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa08c06ec70 con 0x7fa08c0b54d0 2024-09-16T01:43:36.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.121+0000 7fa0918ba640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa08c15ec70 con 0x7fa08c0b7450 2024-09-16T01:43:36.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.121+0000 7fa08a7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa08c0b54d0 0x7fa08c06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:36.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.121+0000 7fa08a7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa08c0b54d0 0x7fa08c06fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60294/0 (socket says 172.21.15.88:60294) 2024-09-16T01:43:36.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.121+0000 7fa08a7fc640 1 -- 172.21.15.88:0/186694169 learned_addr learned my addr 172.21.15.88:0/186694169 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:36.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.121+0000 7fa08b7fe640 1 --2- 172.21.15.88:0/186694169 >> v2:172.21.15.154:3300/0 conn(0x7fa08c0b7450 0x7fa08c15f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:36.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.121+0000 7fa08a7fc640 1 -- 172.21.15.88:0/186694169 >> v2:172.21.15.154:3300/0 conn(0x7fa08c0b7450 msgr2=0x7fa08c15f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:36.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.121+0000 7fa08a7fc640 1 --2- 172.21.15.88:0/186694169 >> v2:172.21.15.154:3300/0 conn(0x7fa08c0b7450 0x7fa08c15f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:36.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.122+0000 7fa08a7fc640 1 -- 172.21.15.88:0/186694169 >> v1:172.21.15.154:6789/0 conn(0x7fa08c15e6f0 legacy=0x7fa08c0b6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:36.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.122+0000 7fa08a7fc640 1 -- 172.21.15.88:0/186694169 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa08c15ef50 con 0x7fa08c0b54d0 2024-09-16T01:43:36.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.122+0000 7fa08b7fe640 1 --2- 172.21.15.88:0/186694169 >> v2:172.21.15.154:3300/0 conn(0x7fa08c0b7450 0x7fa08c15f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:36.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.122+0000 7fa08a7fc640 1 --2- 172.21.15.88:0/186694169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa08c0b54d0 0x7fa08c06fd00 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7fa08c072df0 tx=0x7fa08002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c90db4bea1771949 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:36.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.122+0000 7fa089ffb640 1 -- 172.21.15.88:0/186694169 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa080037070 con 0x7fa08c0b54d0 2024-09-16T01:43:36.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.122+0000 7fa089ffb640 1 -- 172.21.15.88:0/186694169 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa080032440 con 0x7fa08c0b54d0 2024-09-16T01:43:36.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.122+0000 7fa089ffb640 1 -- 172.21.15.88:0/186694169 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa080032810 con 0x7fa08c0b54d0 2024-09-16T01:43:36.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.123+0000 7fa0918ba640 1 -- 172.21.15.88:0/186694169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa08c0b54d0 msgr2=0x7fa08c06fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:36.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.123+0000 7fa0918ba640 1 --2- 172.21.15.88:0/186694169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa08c0b54d0 0x7fa08c06fd00 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7fa08c072df0 tx=0x7fa08002a230 comp rx=0 tx=0).stop 2024-09-16T01:43:36.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.123+0000 7fa0918ba640 1 -- 172.21.15.88:0/186694169 shutdown_connections 2024-09-16T01:43:36.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.123+0000 7fa0918ba640 1 --2- 172.21.15.88:0/186694169 >> v2:172.21.15.154:3300/0 conn(0x7fa08c0b7450 0x7fa08c15f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:36.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.123+0000 7fa0918ba640 1 --2- 172.21.15.88:0/186694169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa08c0b54d0 0x7fa08c06fd00 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:36.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.123+0000 7fa0918ba640 1 -- 172.21.15.88:0/186694169 >> 172.21.15.88:0/186694169 conn(0x7fa08c0d2430 msgr2=0x7fa08c0d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:36.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.123+0000 7fa0918ba640 1 -- 172.21.15.88:0/186694169 shutdown_connections 2024-09-16T01:43:36.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.123+0000 7fa0918ba640 1 -- 172.21.15.88:0/186694169 wait complete. 2024-09-16T01:43:36.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.124+0000 7fa0918ba640 1 Processor -- start 2024-09-16T01:43:36.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.124+0000 7fa0918ba640 1 -- start start 2024-09-16T01:43:36.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.125+0000 7fa0918ba640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa08c0b54d0 0x7fa08c141560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:36.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.125+0000 7fa0918ba640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa08c0b7450 0x7fa08c154870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:36.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.125+0000 7fa0918ba640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa08c15e6f0 0x7fa08c154db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:36.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.125+0000 7fa0918ba640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa08c06e730 con 0x7fa08c15e6f0 2024-09-16T01:43:36.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.125+0000 7fa0918ba640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa08c06e5b0 con 0x7fa08c0b7450 2024-09-16T01:43:36.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.125+0000 7fa0918ba640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa08c070830 con 0x7fa08c0b54d0 2024-09-16T01:43:36.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.125+0000 7fa08b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa08c15e6f0 0x7fa08c154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:36.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.125+0000 7fa08b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa08c15e6f0 0x7fa08c154db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60296/0 (socket says 172.21.15.88:60296) 2024-09-16T01:43:36.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.125+0000 7fa08b7fe640 1 -- 172.21.15.88:0/4216742341 learned_addr learned my addr 172.21.15.88:0/4216742341 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:36.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.125+0000 7fa08a7fc640 1 --2- 172.21.15.88:0/4216742341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa08c0b7450 0x7fa08c154870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:36.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.125+0000 7fa08affd640 1 --2- 172.21.15.88:0/4216742341 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa08c0b54d0 0x7fa08c141560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:36.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.125+0000 7fa08b7fe640 1 -- 172.21.15.88:0/4216742341 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa08c0b54d0 msgr2=0x7fa08c141560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:36.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.125+0000 7fa08b7fe640 1 --2- 172.21.15.88:0/4216742341 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa08c0b54d0 0x7fa08c141560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:36.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.125+0000 7fa08b7fe640 1 -- 172.21.15.88:0/4216742341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa08c0b7450 msgr2=0x7fa08c154870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:36.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.125+0000 7fa08b7fe640 1 --2- 172.21.15.88:0/4216742341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa08c0b7450 0x7fa08c154870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:36.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.125+0000 7fa08b7fe640 1 -- 172.21.15.88:0/4216742341 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa08c1552f0 con 0x7fa08c15e6f0 2024-09-16T01:43:36.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.125+0000 7fa08a7fc640 1 --2- 172.21.15.88:0/4216742341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa08c0b7450 0x7fa08c154870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:36.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.126+0000 7fa08b7fe640 1 --2- 172.21.15.88:0/4216742341 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa08c15e6f0 0x7fa08c154db0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7fa08400c7f0 tx=0x7fa08400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:36.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.126+0000 7fa073fff640 1 -- 172.21.15.88:0/4216742341 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa084015070 con 0x7fa08c15e6f0 2024-09-16T01:43:36.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.126+0000 7fa073fff640 1 -- 172.21.15.88:0/4216742341 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa084007820 con 0x7fa08c15e6f0 2024-09-16T01:43:36.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.126+0000 7fa0918ba640 1 -- 172.21.15.88:0/4216742341 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa08c155b60 con 0x7fa08c15e6f0 2024-09-16T01:43:36.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.126+0000 7fa073fff640 1 -- 172.21.15.88:0/4216742341 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa084007c10 con 0x7fa08c15e6f0 2024-09-16T01:43:36.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.126+0000 7fa0918ba640 1 -- 172.21.15.88:0/4216742341 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa08c155880 con 0x7fa08c15e6f0 2024-09-16T01:43:36.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.127+0000 7fa0918ba640 1 -- 172.21.15.88:0/4216742341 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa08c06c6c0 con 0x7fa08c15e6f0 2024-09-16T01:43:36.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.127+0000 7fa073fff640 1 -- 172.21.15.88:0/4216742341 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa084018070 con 0x7fa08c15e6f0 2024-09-16T01:43:36.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.128+0000 7fa073fff640 1 --2- 172.21.15.88:0/4216742341 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa07806c2b0 0x7fa07806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:36.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.128+0000 7fa073fff640 1 -- 172.21.15.88:0/4216742341 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8576+0+0 (secure 0 0 0) 0x7fa08408c120 con 0x7fa08c15e6f0 2024-09-16T01:43:36.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.128+0000 7fa08affd640 1 --2- 172.21.15.88:0/4216742341 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa07806c2b0 0x7fa07806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:36.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.128+0000 7fa08affd640 1 --2- 172.21.15.88:0/4216742341 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa07806c2b0 0x7fa07806e770 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009f20 tx=0x7fa07c0094d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:36.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.132+0000 7fa073fff640 1 -- 172.21.15.88:0/4216742341 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa084059d10 con 0x7fa08c15e6f0 2024-09-16T01:43:36.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.333+0000 7fa0918ba640 1 -- 172.21.15.88:0/4216742341 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fa08c06cd20 con 0x7fa08c15e6f0 2024-09-16T01:43:36.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.682+0000 7fa073fff640 1 -- 172.21.15.88:0/4216742341 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v270) ==== 162+0+0 (secure 0 0 0) 0x7fa084059610 con 0x7fa08c15e6f0 2024-09-16T01:43:36.684 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:43:36.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.684+0000 7fa0918ba640 1 -- 172.21.15.88:0/4216742341 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa07806c2b0 msgr2=0x7fa07806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:36.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.684+0000 7fa0918ba640 1 --2- 172.21.15.88:0/4216742341 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa07806c2b0 0x7fa07806e770 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009f20 tx=0x7fa07c0094d0 comp rx=0 tx=0).stop 2024-09-16T01:43:36.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.684+0000 7fa0918ba640 1 -- 172.21.15.88:0/4216742341 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa08c15e6f0 msgr2=0x7fa08c154db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:36.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.684+0000 7fa0918ba640 1 --2- 172.21.15.88:0/4216742341 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa08c15e6f0 0x7fa08c154db0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7fa08400c7f0 tx=0x7fa08400ccb0 comp rx=0 tx=0).stop 2024-09-16T01:43:36.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.684+0000 7fa0918ba640 1 -- 172.21.15.88:0/4216742341 shutdown_connections 2024-09-16T01:43:36.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.684+0000 7fa0918ba640 1 --2- 172.21.15.88:0/4216742341 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa07806c2b0 0x7fa07806e770 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:36.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.684+0000 7fa0918ba640 1 --2- 172.21.15.88:0/4216742341 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa08c15e6f0 0x7fa08c154db0 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:36.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.684+0000 7fa0918ba640 1 --2- 172.21.15.88:0/4216742341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa08c0b7450 0x7fa08c154870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:36.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.684+0000 7fa0918ba640 1 --2- 172.21.15.88:0/4216742341 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa08c0b54d0 0x7fa08c141560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:36.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.684+0000 7fa0918ba640 1 -- 172.21.15.88:0/4216742341 >> 172.21.15.88:0/4216742341 conn(0x7fa08c0d2430 msgr2=0x7fa08c0b1850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:36.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.685+0000 7fa0918ba640 1 -- 172.21.15.88:0/4216742341 shutdown_connections 2024-09-16T01:43:36.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:36.685+0000 7fa0918ba640 1 -- 172.21.15.88:0/4216742341 wait complete. 2024-09-16T01:43:37.102 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:43:37.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.201+0000 7f1a602c9640 1 Processor -- start 2024-09-16T01:43:37.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.201+0000 7f1a602c9640 1 -- start start 2024-09-16T01:43:37.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.202+0000 7f1a602c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a5815e660 0x7f1a580cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:37.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.202+0000 7f1a602c9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1a580d3310 0x7f1a580d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:37.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.202+0000 7f1a602c9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1a580d1650 0x7f1a5806bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:37.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.202+0000 7f1a602c9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f1a5815ea40 con 0x7f1a580d1650 2024-09-16T01:43:37.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.202+0000 7f1a602c9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f1a580d3870 con 0x7f1a580d3310 2024-09-16T01:43:37.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.202+0000 7f1a602c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1a580d3b50 con 0x7f1a5815e660 2024-09-16T01:43:37.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.202+0000 7f1a5e866640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1a580d1650 0x7f1a5806bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:37.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.202+0000 7f1a5e866640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1a580d1650 0x7f1a5806bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60310/0 (socket says 172.21.15.88:60310) 2024-09-16T01:43:37.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.202+0000 7f1a5e866640 1 -- 172.21.15.88:0/282020084 learned_addr learned my addr 172.21.15.88:0/282020084 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:37.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.203+0000 7f1a5e866640 1 -- 172.21.15.88:0/282020084 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a5815e660 msgr2=0x7f1a580cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:37.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.203+0000 7f1a5e866640 1 --2- 172.21.15.88:0/282020084 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a5815e660 0x7f1a580cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:37.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.203+0000 7f1a5e866640 1 -- 172.21.15.88:0/282020084 >> v2:172.21.15.154:3300/0 conn(0x7f1a580d3310 msgr2=0x7f1a580d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:37.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.203+0000 7f1a5d864640 1 --2- 172.21.15.88:0/282020084 >> v2:172.21.15.154:3300/0 conn(0x7f1a580d3310 0x7f1a580d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:37.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.203+0000 7f1a5e866640 1 --2- 172.21.15.88:0/282020084 >> v2:172.21.15.154:3300/0 conn(0x7f1a580d3310 0x7f1a580d1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:37.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.203+0000 7f1a5e866640 1 -- 172.21.15.88:0/282020084 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a580d3e30 con 0x7f1a580d1650 2024-09-16T01:43:37.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.203+0000 7f1a5e866640 1 --2- 172.21.15.88:0/282020084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a580d1650 0x7f1a5806bfd0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f1a5808d140 tx=0x7f1a5002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=369c00d89d87c365 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:37.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.204+0000 7f1a5d063640 1 -- 172.21.15.88:0/282020084 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a50037070 con 0x7f1a580d1650 2024-09-16T01:43:37.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.204+0000 7f1a5d063640 1 -- 172.21.15.88:0/282020084 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a50032440 con 0x7f1a580d1650 2024-09-16T01:43:37.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.204+0000 7f1a5d063640 1 -- 172.21.15.88:0/282020084 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a50032810 con 0x7f1a580d1650 2024-09-16T01:43:37.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.204+0000 7f1a602c9640 1 -- 172.21.15.88:0/282020084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a580d1650 msgr2=0x7f1a5806bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:37.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.204+0000 7f1a602c9640 1 --2- 172.21.15.88:0/282020084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a580d1650 0x7f1a5806bfd0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f1a5808d140 tx=0x7f1a5002a1e0 comp rx=0 tx=0).stop 2024-09-16T01:43:37.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.204+0000 7f1a602c9640 1 -- 172.21.15.88:0/282020084 shutdown_connections 2024-09-16T01:43:37.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.204+0000 7f1a602c9640 1 --2- 172.21.15.88:0/282020084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a580d1650 0x7f1a5806bfd0 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:37.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.204+0000 7f1a602c9640 1 --2- 172.21.15.88:0/282020084 >> v2:172.21.15.154:3300/0 conn(0x7f1a580d3310 0x7f1a580d1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:37.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.204+0000 7f1a602c9640 1 --2- 172.21.15.88:0/282020084 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a5815e660 0x7f1a580cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:37.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.204+0000 7f1a602c9640 1 -- 172.21.15.88:0/282020084 >> 172.21.15.88:0/282020084 conn(0x7f1a580b6410 msgr2=0x7f1a580b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:37.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.204+0000 7f1a602c9640 1 -- 172.21.15.88:0/282020084 shutdown_connections 2024-09-16T01:43:37.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.204+0000 7f1a602c9640 1 -- 172.21.15.88:0/282020084 wait complete. 2024-09-16T01:43:37.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.205+0000 7f1a602c9640 1 Processor -- start 2024-09-16T01:43:37.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.205+0000 7f1a602c9640 1 -- start start 2024-09-16T01:43:37.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.206+0000 7f1a602c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a580d1650 0x7f1a581f14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:37.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.206+0000 7f1a602c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a580d3310 0x7f1a580b79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:37.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.206+0000 7f1a602c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a5815e660 0x7f1a581f5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:37.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.206+0000 7f1a602c9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1a580cfc70 con 0x7f1a5815e660 2024-09-16T01:43:37.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.206+0000 7f1a602c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1a580d43f0 con 0x7f1a580d3310 2024-09-16T01:43:37.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.206+0000 7f1a602c9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1a5806c510 con 0x7f1a580d1650 2024-09-16T01:43:37.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.206+0000 7f1a5e866640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a5815e660 0x7f1a581f5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:37.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.206+0000 7f1a5e866640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a5815e660 0x7f1a581f5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60320/0 (socket says 172.21.15.88:60320) 2024-09-16T01:43:37.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.206+0000 7f1a5e866640 1 -- 172.21.15.88:0/1938444164 learned_addr learned my addr 172.21.15.88:0/1938444164 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:37.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.206+0000 7f1a5e866640 1 -- 172.21.15.88:0/1938444164 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a580d1650 msgr2=0x7f1a581f14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:37.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.206+0000 7f1a5e866640 1 --2- 172.21.15.88:0/1938444164 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a580d1650 0x7f1a581f14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:37.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.206+0000 7f1a5e866640 1 -- 172.21.15.88:0/1938444164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a580d3310 msgr2=0x7f1a580b79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:37.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.206+0000 7f1a5e866640 1 --2- 172.21.15.88:0/1938444164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a580d3310 0x7f1a580b79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:37.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.206+0000 7f1a5e866640 1 -- 172.21.15.88:0/1938444164 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a580b8210 con 0x7f1a5815e660 2024-09-16T01:43:37.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.207+0000 7f1a5e866640 1 --2- 172.21.15.88:0/1938444164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a5815e660 0x7f1a581f5050 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f1a500090b0 tx=0x7f1a500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:37.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.207+0000 7f1a56ffd640 1 -- 172.21.15.88:0/1938444164 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a50041020 con 0x7f1a5815e660 2024-09-16T01:43:37.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.207+0000 7f1a56ffd640 1 -- 172.21.15.88:0/1938444164 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a50030990 con 0x7f1a5815e660 2024-09-16T01:43:37.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.207+0000 7f1a602c9640 1 -- 172.21.15.88:0/1938444164 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a580b84f0 con 0x7f1a5815e660 2024-09-16T01:43:37.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.207+0000 7f1a56ffd640 1 -- 172.21.15.88:0/1938444164 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a50037040 con 0x7f1a5815e660 2024-09-16T01:43:37.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.207+0000 7f1a602c9640 1 -- 172.21.15.88:0/1938444164 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a5815ebc0 con 0x7f1a5815e660 2024-09-16T01:43:37.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.208+0000 7f1a602c9640 1 -- 172.21.15.88:0/1938444164 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a580d1110 con 0x7f1a5815e660 2024-09-16T01:43:37.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.208+0000 7f1a56ffd640 1 -- 172.21.15.88:0/1938444164 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1a50044020 con 0x7f1a5815e660 2024-09-16T01:43:37.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.209+0000 7f1a56ffd640 1 --2- 172.21.15.88:0/1938444164 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1a4406c2e0 0x7f1a4406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:37.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.209+0000 7f1a56ffd640 1 -- 172.21.15.88:0/1938444164 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8576+0+0 (secure 0 0 0) 0x7f1a500ac930 con 0x7f1a5815e660 2024-09-16T01:43:37.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.209+0000 7f1a5e065640 1 --2- 172.21.15.88:0/1938444164 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1a4406c2e0 0x7f1a4406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:37.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.209+0000 7f1a5e065640 1 --2- 172.21.15.88:0/1938444164 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1a4406c2e0 0x7f1a4406e7a0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f1a580b9e70 tx=0x7f1a48009250 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:37.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.213+0000 7f1a56ffd640 1 -- 172.21.15.88:0/1938444164 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1a5004b030 con 0x7f1a5815e660 2024-09-16T01:43:37.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.412+0000 7f1a602c9640 1 -- 172.21.15.88:0/1938444164 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a580b6920 con 0x7f1a5815e660 2024-09-16T01:43:37.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.413+0000 7f1a56ffd640 1 -- 172.21.15.88:0/1938444164 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 271 v271) ==== 95+0+5227 (secure 0 0 0) 0x7f1a5007a520 con 0x7f1a5815e660 2024-09-16T01:43:37.415 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:37.415 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":271,"btime":"2024-09-16T01:43:36:685024+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9696,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4270197461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4270197461},{"type":"v1","addr":"172.21.15.88:6837","nonce":4270197461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":271,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:43:36.685009+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9852":{"gid":9852,"name":"a","rank":0,"incarnation":271,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6836/4135629195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4135629195},{"type":"v1","addr":"172.21.15.88:6836","nonce":4135629195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:43:37.415 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 271 2024-09-16T01:43:37.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.415+0000 7f1a602c9640 1 -- 172.21.15.88:0/1938444164 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1a4406c2e0 msgr2=0x7f1a4406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:37.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.415+0000 7f1a602c9640 1 --2- 172.21.15.88:0/1938444164 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1a4406c2e0 0x7f1a4406e7a0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f1a580b9e70 tx=0x7f1a48009250 comp rx=0 tx=0).stop 2024-09-16T01:43:37.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.415+0000 7f1a602c9640 1 -- 172.21.15.88:0/1938444164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a5815e660 msgr2=0x7f1a581f5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:37.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.415+0000 7f1a602c9640 1 --2- 172.21.15.88:0/1938444164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a5815e660 0x7f1a581f5050 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f1a500090b0 tx=0x7f1a500046c0 comp rx=0 tx=0).stop 2024-09-16T01:43:37.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.415+0000 7f1a602c9640 1 -- 172.21.15.88:0/1938444164 shutdown_connections 2024-09-16T01:43:37.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.415+0000 7f1a602c9640 1 --2- 172.21.15.88:0/1938444164 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1a4406c2e0 0x7f1a4406e7a0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:37.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.415+0000 7f1a602c9640 1 --2- 172.21.15.88:0/1938444164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1a5815e660 0x7f1a581f5050 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:37.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.415+0000 7f1a602c9640 1 --2- 172.21.15.88:0/1938444164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a580d3310 0x7f1a580b79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:37.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.415+0000 7f1a602c9640 1 --2- 172.21.15.88:0/1938444164 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a580d1650 0x7f1a581f14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:37.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.415+0000 7f1a602c9640 1 -- 172.21.15.88:0/1938444164 >> 172.21.15.88:0/1938444164 conn(0x7f1a580b6410 msgr2=0x7f1a58159790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:37.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.415+0000 7f1a602c9640 1 -- 172.21.15.88:0/1938444164 shutdown_connections 2024-09-16T01:43:37.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:37.415+0000 7f1a602c9640 1 -- 172.21.15.88:0/1938444164 wait complete. 2024-09-16T01:43:37.428 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 271, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:40:03.597796+0000', 'modified': '2024-09-16T01:43:36.685009+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 120, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9852}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9852': {'gid': 9852, 'name': 'a', 'rank': 0, 'incarnation': 271, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.88:6836/4135629195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 4135629195}, {'type': 'v1', 'addr': '172.21.15.88:6836', 'nonce': 4135629195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-16T01:43:37.428 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9852:up:replay 2024-09-16T01:43:37.428 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-09-16T01:43:40.687 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:43:40.686+0000 7fe826162640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:43:42.706 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:43:42.705+0000 7f213b330640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-09-16T01:43:42.832 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:43:42.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.942+0000 7f647eb3b640 1 Processor -- start 2024-09-16T01:43:42.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.942+0000 7f647eb3b640 1 -- start start 2024-09-16T01:43:42.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.942+0000 7f647eb3b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f647815e660 0x7f64780cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:42.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.942+0000 7f647eb3b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f64780d1820 0x7f64780b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:42.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.942+0000 7f647eb3b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f647815ea40 con 0x7f64780d3310 2024-09-16T01:43:42.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.942+0000 7f647eb3b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f64780d3870 con 0x7f64780d1820 2024-09-16T01:43:42.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.942+0000 7f647eb3b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f64780d3b50 con 0x7f647815e660 2024-09-16T01:43:42.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.943+0000 7f647d0d8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f64780d1820 0x7f64780b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:42.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.943+0000 7f647d0d8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f64780d1820 0x7f64780b7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35626/0 (socket says 172.21.15.88:35626) 2024-09-16T01:43:42.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.943+0000 7f647d0d8640 1 -- 172.21.15.88:0/2247861631 learned_addr learned my addr 172.21.15.88:0/2247861631 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:42.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.943+0000 7f647c8d7640 1 --2- 172.21.15.88:0/2247861631 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f647815e660 0x7f64780cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:42.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.943+0000 7f647d0d8640 1 -- 172.21.15.88:0/2247861631 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f647815e660 msgr2=0x7f64780cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:42.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.943+0000 7f647d0d8640 1 --2- 172.21.15.88:0/2247861631 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f647815e660 0x7f64780cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:42.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.943+0000 7f647d0d8640 1 -- 172.21.15.88:0/2247861631 >> v1:172.21.15.154:6789/0 conn(0x7f64780d3310 legacy=0x7f64780d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:42.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.943+0000 7f647d0d8640 1 -- 172.21.15.88:0/2247861631 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64780d3e30 con 0x7f64780d1820 2024-09-16T01:43:42.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.944+0000 7f647d0d8640 1 --2- 172.21.15.88:0/2247861631 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64780d1820 0x7f64780b7b70 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f6460009870 tx=0x7f646002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e434828955fd92d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:42.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.944+0000 7f64777fe640 1 -- 172.21.15.88:0/2247861631 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6460037070 con 0x7f64780d1820 2024-09-16T01:43:42.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.944+0000 7f64777fe640 1 -- 172.21.15.88:0/2247861631 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6460032440 con 0x7f64780d1820 2024-09-16T01:43:42.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.944+0000 7f64777fe640 1 -- 172.21.15.88:0/2247861631 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6460032810 con 0x7f64780d1820 2024-09-16T01:43:42.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.944+0000 7f647eb3b640 1 -- 172.21.15.88:0/2247861631 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64780d1820 msgr2=0x7f64780b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:42.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.944+0000 7f647eb3b640 1 --2- 172.21.15.88:0/2247861631 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64780d1820 0x7f64780b7b70 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f6460009870 tx=0x7f646002a250 comp rx=0 tx=0).stop 2024-09-16T01:43:42.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.944+0000 7f647eb3b640 1 -- 172.21.15.88:0/2247861631 shutdown_connections 2024-09-16T01:43:42.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.944+0000 7f647eb3b640 1 --2- 172.21.15.88:0/2247861631 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64780d1820 0x7f64780b7b70 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:42.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.944+0000 7f647eb3b640 1 --2- 172.21.15.88:0/2247861631 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f647815e660 0x7f64780cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:42.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.944+0000 7f647eb3b640 1 -- 172.21.15.88:0/2247861631 >> 172.21.15.88:0/2247861631 conn(0x7f64780b6410 msgr2=0x7f64780b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:42.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.945+0000 7f647eb3b640 1 -- 172.21.15.88:0/2247861631 shutdown_connections 2024-09-16T01:43:42.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.945+0000 7f647eb3b640 1 -- 172.21.15.88:0/2247861631 wait complete. 2024-09-16T01:43:42.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.946+0000 7f647eb3b640 1 Processor -- start 2024-09-16T01:43:42.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.946+0000 7f647eb3b640 1 -- start start 2024-09-16T01:43:42.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.947+0000 7f647eb3b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64780d1820 0x7f64781f4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:42.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.947+0000 7f647eb3b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64780d3310 0x7f64780ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:42.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.947+0000 7f647eb3b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f647815e660 0x7f64780ce690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:42.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.947+0000 7f647eb3b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f64780cfc70 con 0x7f64780d1820 2024-09-16T01:43:42.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.947+0000 7f647eb3b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f64780d43f0 con 0x7f64780d3310 2024-09-16T01:43:42.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.947+0000 7f647eb3b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f64780b80b0 con 0x7f647815e660 2024-09-16T01:43:42.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.947+0000 7f647c8d7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64780d1820 0x7f64781f4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:42.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.947+0000 7f647c8d7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64780d1820 0x7f64781f4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35628/0 (socket says 172.21.15.88:35628) 2024-09-16T01:43:42.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.947+0000 7f647c8d7640 1 -- 172.21.15.88:0/924271053 learned_addr learned my addr 172.21.15.88:0/924271053 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:42.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.947+0000 7f6477fff640 1 --2- 172.21.15.88:0/924271053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64780d3310 0x7f64780ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:42.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.947+0000 7f647d0d8640 1 --2- 172.21.15.88:0/924271053 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f647815e660 0x7f64780ce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:42.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.947+0000 7f647c8d7640 1 -- 172.21.15.88:0/924271053 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f647815e660 msgr2=0x7f64780ce690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:42.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.947+0000 7f647c8d7640 1 --2- 172.21.15.88:0/924271053 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f647815e660 0x7f64780ce690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:42.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.947+0000 7f647c8d7640 1 -- 172.21.15.88:0/924271053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64780d3310 msgr2=0x7f64780ccd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:42.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.947+0000 7f647c8d7640 1 --2- 172.21.15.88:0/924271053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64780d3310 0x7f64780ccd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:42.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.947+0000 7f647c8d7640 1 -- 172.21.15.88:0/924271053 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64780cbf90 con 0x7f64780d1820 2024-09-16T01:43:42.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.947+0000 7f6477fff640 1 --2- 172.21.15.88:0/924271053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64780d3310 0x7f64780ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:42.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.947+0000 7f647c8d7640 1 --2- 172.21.15.88:0/924271053 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64780d1820 0x7f64781f4840 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f647000d5f0 tx=0x7f647000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:42.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.947+0000 7f64757fa640 1 -- 172.21.15.88:0/924271053 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64700125f0 con 0x7f64780d1820 2024-09-16T01:43:42.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.947+0000 7f64757fa640 1 -- 172.21.15.88:0/924271053 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6470019070 con 0x7f64780d1820 2024-09-16T01:43:42.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.947+0000 7f64757fa640 1 -- 172.21.15.88:0/924271053 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64700129d0 con 0x7f64780d1820 2024-09-16T01:43:42.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.947+0000 7f647eb3b640 1 -- 172.21.15.88:0/924271053 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64780cc270 con 0x7f64780d1820 2024-09-16T01:43:42.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.947+0000 7f647eb3b640 1 -- 172.21.15.88:0/924271053 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64780cf130 con 0x7f64780d1820 2024-09-16T01:43:42.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.949+0000 7f647eb3b640 1 -- 172.21.15.88:0/924271053 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64780b7b70 con 0x7f64780d1820 2024-09-16T01:43:42.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.949+0000 7f64757fa640 1 -- 172.21.15.88:0/924271053 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6470012b70 con 0x7f64780d1820 2024-09-16T01:43:42.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.949+0000 7f64757fa640 1 --2- 172.21.15.88:0/924271053 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f645806c2b0 0x7f645806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:42.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.950+0000 7f64757fa640 1 -- 172.21.15.88:0/924271053 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8576+0+0 (secure 0 0 0) 0x7f647008c380 con 0x7f64780d1820 2024-09-16T01:43:42.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.950+0000 7f6477fff640 1 --2- 172.21.15.88:0/924271053 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f645806c2b0 0x7f645806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:42.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.950+0000 7f6477fff640 1 --2- 172.21.15.88:0/924271053 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f645806c2b0 0x7f645806e770 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f647808d600 tx=0x7f6468009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:42.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:42.954+0000 7f64757fa640 1 -- 172.21.15.88:0/924271053 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f647005a000 con 0x7f64780d1820 2024-09-16T01:43:43.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.153+0000 7f647eb3b640 1 -- 172.21.15.88:0/924271053 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f64780ce470 con 0x7f64780d1820 2024-09-16T01:43:43.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.154+0000 7f64757fa640 1 -- 172.21.15.88:0/924271053 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 275 v275) ==== 95+0+5234 (secure 0 0 0) 0x7f6470059900 con 0x7f64780d1820 2024-09-16T01:43:43.155 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:43.156 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":275,"btime":"2024-09-16T01:43:42:706187+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9696,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4270197461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4270197461},{"type":"v1","addr":"172.21.15.88:6837","nonce":4270197461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":275,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:43:42.694403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9852":{"gid":9852,"name":"a","rank":0,"incarnation":271,"state":"up:active","state_seq":5,"addr":"172.21.15.88:6836/4135629195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4135629195},{"type":"v1","addr":"172.21.15.88:6836","nonce":4135629195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9852,"qdb_cluster":[9852]},"id":6}]} 2024-09-16T01:43:43.156 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 275 2024-09-16T01:43:43.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.156+0000 7f647eb3b640 1 -- 172.21.15.88:0/924271053 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f645806c2b0 msgr2=0x7f645806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:43.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.156+0000 7f647eb3b640 1 --2- 172.21.15.88:0/924271053 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f645806c2b0 0x7f645806e770 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f647808d600 tx=0x7f6468009040 comp rx=0 tx=0).stop 2024-09-16T01:43:43.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.156+0000 7f647eb3b640 1 -- 172.21.15.88:0/924271053 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64780d1820 msgr2=0x7f64781f4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:43.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.156+0000 7f647eb3b640 1 --2- 172.21.15.88:0/924271053 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64780d1820 0x7f64781f4840 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f647000d5f0 tx=0x7f647000dab0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.156+0000 7f647eb3b640 1 -- 172.21.15.88:0/924271053 shutdown_connections 2024-09-16T01:43:43.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.156+0000 7f647eb3b640 1 --2- 172.21.15.88:0/924271053 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f645806c2b0 0x7f645806e770 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.156+0000 7f647eb3b640 1 --2- 172.21.15.88:0/924271053 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f647815e660 0x7f64780ce690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.156+0000 7f647eb3b640 1 --2- 172.21.15.88:0/924271053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64780d3310 0x7f64780ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.156+0000 7f647eb3b640 1 --2- 172.21.15.88:0/924271053 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64780d1820 0x7f64781f4840 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.156+0000 7f647eb3b640 1 -- 172.21.15.88:0/924271053 >> 172.21.15.88:0/924271053 conn(0x7f64780b6410 msgr2=0x7f647815bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:43.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.156+0000 7f647eb3b640 1 -- 172.21.15.88:0/924271053 shutdown_connections 2024-09-16T01:43:43.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.156+0000 7f647eb3b640 1 -- 172.21.15.88:0/924271053 wait complete. 2024-09-16T01:43:43.169 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 275, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:40:03.597796+0000', 'modified': '2024-09-16T01:43:42.694403+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 120, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9852}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9852': {'gid': 9852, 'name': 'a', 'rank': 0, 'incarnation': 271, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.88:6836/4135629195', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 4135629195}, {'type': 'v1', 'addr': '172.21.15.88:6836', 'nonce': 4135629195}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9852, 'qdb_cluster': [9852]} 2024-09-16T01:43:43.169 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T01:43:43.169 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-16T01:43:43.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.266+0000 7f06b4733640 1 Processor -- start 2024-09-16T01:43:43.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.266+0000 7f06b4733640 1 -- start start 2024-09-16T01:43:43.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.267+0000 7f06b4733640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f06ac0b6290 0x7f06ac071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.267+0000 7f06b4733640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06ac072530 0x7f06ac15bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.268+0000 7f06b4733640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06ac0ccff0 con 0x7f06ac15b480 2024-09-16T01:43:43.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.268+0000 7f06b4733640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f06ac0cd2d0 con 0x7f06ac0b6290 2024-09-16T01:43:43.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.268+0000 7f06b4733640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f06ac0cd5b0 con 0x7f06ac072530 2024-09-16T01:43:43.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.268+0000 7f06b24cf640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f06ac15b480 0x7f06ac0b5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42366/0 (socket says 172.21.15.88:42366) 2024-09-16T01:43:43.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.268+0000 7f06b24cf640 1 -- 172.21.15.88:0/2817162084 learned_addr learned my addr 172.21.15.88:0/2817162084 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:43.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.268+0000 7f06b14cd640 1 -- 172.21.15.88:0/2817162084 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3600904425 0 0) 0x7f06ac0ccff0 con 0x7f06ac15b480 2024-09-16T01:43:43.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.268+0000 7f06b14cd640 1 -- 172.21.15.88:0/2817162084 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0694003520 con 0x7f06ac15b480 2024-09-16T01:43:43.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.268+0000 7f06b2cd0640 1 --2- 172.21.15.88:0/2817162084 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06ac072530 0x7f06ac15bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.268+0000 7f06b1cce640 1 --2- 172.21.15.88:0/2817162084 >> v2:172.21.15.154:3300/0 conn(0x7f06ac0b6290 0x7f06ac071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.268+0000 7f06b14cd640 1 -- 172.21.15.88:0/2817162084 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 991114278 0 0) 0x7f0694003520 con 0x7f06ac15b480 2024-09-16T01:43:43.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.268+0000 7f06b14cd640 1 -- 172.21.15.88:0/2817162084 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06ac072530 msgr2=0x7f06ac15bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:43.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.268+0000 7f06b14cd640 1 --2- 172.21.15.88:0/2817162084 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06ac072530 0x7f06ac15bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.268+0000 7f06b14cd640 1 -- 172.21.15.88:0/2817162084 >> v2:172.21.15.154:3300/0 conn(0x7f06ac0b6290 msgr2=0x7f06ac071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:43.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.268+0000 7f06b14cd640 1 --2- 172.21.15.88:0/2817162084 >> v2:172.21.15.154:3300/0 conn(0x7f06ac0b6290 0x7f06ac071ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.268+0000 7f06b14cd640 1 -- 172.21.15.88:0/2817162084 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06ac0cd890 con 0x7f06ac15b480 2024-09-16T01:43:43.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.268+0000 7f06b14cd640 1 -- 172.21.15.88:0/2817162084 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f06a4001670 con 0x7f06ac15b480 2024-09-16T01:43:43.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.268+0000 7f06b14cd640 1 -- 172.21.15.88:0/2817162084 >> v1:172.21.15.88:6789/0 conn(0x7f06ac15b480 legacy=0x7f06ac0b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:43.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.268+0000 7f06b14cd640 1 --2- 172.21.15.88:0/2817162084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06940039b0 0x7f0694003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.268+0000 7f06b14cd640 1 --2- 172.21.15.88:0/2817162084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0694004800 0x7f0694004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.268+0000 7f06b14cd640 1 --2- 172.21.15.88:0/2817162084 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06ac072530 0x7f06940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.268+0000 7f06b14cd640 1 -- 172.21.15.88:0/2817162084 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f06ac0cd2d0 con 0x7f0694004800 2024-09-16T01:43:43.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.268+0000 7f06b14cd640 1 -- 172.21.15.88:0/2817162084 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f06ac0cd5b0 con 0x7f06940039b0 2024-09-16T01:43:43.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.268+0000 7f06b14cd640 1 -- 172.21.15.88:0/2817162084 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f069400dbd0 con 0x7f06ac072530 2024-09-16T01:43:43.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.268+0000 7f06b1cce640 1 --2- 172.21.15.88:0/2817162084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0694004800 0x7f0694004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.268+0000 7f06b24cf640 1 --2- 172.21.15.88:0/2817162084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06940039b0 0x7f0694003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.269+0000 7f06b2cd0640 1 --2- 172.21.15.88:0/2817162084 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06ac072530 0x7f06940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.269+0000 7f06b1cce640 1 -- 172.21.15.88:0/2817162084 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06ac072530 msgr2=0x7f06940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:43.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.269+0000 7f06b1cce640 1 --2- 172.21.15.88:0/2817162084 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06ac072530 0x7f06940051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.269+0000 7f06b1cce640 1 -- 172.21.15.88:0/2817162084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06940039b0 msgr2=0x7f0694003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:43.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.269+0000 7f06b1cce640 1 --2- 172.21.15.88:0/2817162084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06940039b0 0x7f0694003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.269+0000 7f06b1cce640 1 -- 172.21.15.88:0/2817162084 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0694003520 con 0x7f0694004800 2024-09-16T01:43:43.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.269+0000 7f06b24cf640 1 --2- 172.21.15.88:0/2817162084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06940039b0 0x7f0694003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:43.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.269+0000 7f06b1cce640 1 --2- 172.21.15.88:0/2817162084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0694004800 0x7f0694004c80 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009ac0 tx=0x7f0694005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17c2a094f66f0418 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:43.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.269+0000 7f06b14cd640 1 -- 172.21.15.88:0/2817162084 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a0036070 con 0x7f0694004800 2024-09-16T01:43:43.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.269+0000 7f06b14cd640 1 -- 172.21.15.88:0/2817162084 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06a0031400 con 0x7f0694004800 2024-09-16T01:43:43.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.269+0000 7f06b14cd640 1 -- 172.21.15.88:0/2817162084 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a0009690 con 0x7f0694004800 2024-09-16T01:43:43.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.270+0000 7f06b4733640 1 -- 172.21.15.88:0/2817162084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0694004800 msgr2=0x7f0694004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:43.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.270+0000 7f06b4733640 1 --2- 172.21.15.88:0/2817162084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0694004800 0x7f0694004c80 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009ac0 tx=0x7f0694005830 comp rx=0 tx=0).stop 2024-09-16T01:43:43.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.270+0000 7f06b2cd0640 1 -- 172.21.15.88:0/2817162084 reap_dead start 2024-09-16T01:43:43.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.270+0000 7f06b4733640 1 -- 172.21.15.88:0/2817162084 shutdown_connections 2024-09-16T01:43:43.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.270+0000 7f06b4733640 1 -- 172.21.15.88:0/2817162084 >> 172.21.15.88:0/2817162084 conn(0x7f06ac06d0f0 msgr2=0x7f06ac06d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:43.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.270+0000 7f06b4733640 1 -- 172.21.15.88:0/2817162084 shutdown_connections 2024-09-16T01:43:43.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.270+0000 7f06b4733640 1 -- 172.21.15.88:0/2817162084 wait complete. 2024-09-16T01:43:43.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.271+0000 7f06b4733640 1 Processor -- start 2024-09-16T01:43:43.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.271+0000 7f06b4733640 1 -- start start 2024-09-16T01:43:43.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.272+0000 7f06b4733640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0694004800 0x7f06ac1f1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.272+0000 7f06b4733640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06ac0cc4b0 0x7f06ac1f0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.272+0000 7f06b4733640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06ac153ca0 0x7f06ac0c7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.272+0000 7f06b4733640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f06ac15b9e0 con 0x7f06ac153ca0 2024-09-16T01:43:43.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.272+0000 7f06b4733640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f06ac15b860 con 0x7f0694004800 2024-09-16T01:43:43.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.272+0000 7f06b4733640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f06ac0b3590 con 0x7f06ac0cc4b0 2024-09-16T01:43:43.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.272+0000 7f06b2cd0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06ac153ca0 0x7f06ac0c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.272+0000 7f06b2cd0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06ac153ca0 0x7f06ac0c7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35644/0 (socket says 172.21.15.88:35644) 2024-09-16T01:43:43.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.272+0000 7f06b2cd0640 1 -- 172.21.15.88:0/1241195622 learned_addr learned my addr 172.21.15.88:0/1241195622 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:43.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.272+0000 7f06b24cf640 1 --2- 172.21.15.88:0/1241195622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0694004800 0x7f06ac1f1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.272+0000 7f06b1cce640 1 --2- 172.21.15.88:0/1241195622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06ac0cc4b0 0x7f06ac1f0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.272+0000 7f06b2cd0640 1 -- 172.21.15.88:0/1241195622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06ac0cc4b0 msgr2=0x7f06ac1f0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:43.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.272+0000 7f06b2cd0640 1 --2- 172.21.15.88:0/1241195622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06ac0cc4b0 0x7f06ac1f0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.272+0000 7f06b2cd0640 1 -- 172.21.15.88:0/1241195622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0694004800 msgr2=0x7f06ac1f1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:43.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.272+0000 7f06b2cd0640 1 --2- 172.21.15.88:0/1241195622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0694004800 0x7f06ac1f1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.272+0000 7f06b2cd0640 1 -- 172.21.15.88:0/1241195622 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06ac1f0ff0 con 0x7f06ac153ca0 2024-09-16T01:43:43.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.272+0000 7f06b1cce640 1 --2- 172.21.15.88:0/1241195622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06ac0cc4b0 0x7f06ac1f0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:43.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.273+0000 7f06b2cd0640 1 --2- 172.21.15.88:0/1241195622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06ac153ca0 0x7f06ac0c7880 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f06ac06e110 tx=0x7f069c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:43.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.273+0000 7f06aaffd640 1 -- 172.21.15.88:0/1241195622 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c011e10 con 0x7f06ac153ca0 2024-09-16T01:43:43.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.273+0000 7f06aaffd640 1 -- 172.21.15.88:0/1241195622 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f069c01b070 con 0x7f06ac153ca0 2024-09-16T01:43:43.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.273+0000 7f06b4733640 1 -- 172.21.15.88:0/1241195622 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06ac0cc9b0 con 0x7f06ac153ca0 2024-09-16T01:43:43.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.273+0000 7f06aaffd640 1 -- 172.21.15.88:0/1241195622 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c003990 con 0x7f06ac153ca0 2024-09-16T01:43:43.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.273+0000 7f06b4733640 1 -- 172.21.15.88:0/1241195622 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06ac0ce760 con 0x7f06ac153ca0 2024-09-16T01:43:43.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.274+0000 7f06aaffd640 1 -- 172.21.15.88:0/1241195622 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f069c005d10 con 0x7f06ac153ca0 2024-09-16T01:43:43.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.275+0000 7f06aaffd640 1 --2- 172.21.15.88:0/1241195622 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f069806c2e0 0x7f069806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.275+0000 7f06aaffd640 1 -- 172.21.15.88:0/1241195622 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8576+0+0 (secure 0 0 0) 0x7f069c08c1f0 con 0x7f06ac153ca0 2024-09-16T01:43:43.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.275+0000 7f06b24cf640 1 --2- 172.21.15.88:0/1241195622 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f069806c2e0 0x7f069806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.275+0000 7f06b24cf640 1 --2- 172.21.15.88:0/1241195622 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f069806c2e0 0x7f069806e7a0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f06a4001860 tx=0x7f06a4007040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:43.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.276+0000 7f06b74c8740 1 Processor -- start 2024-09-16T01:43:43.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.276+0000 7f06b74c8740 1 -- start start 2024-09-16T01:43:43.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.277+0000 7f06b74c8740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f9542c0 0x56473f999a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.277+0000 7f06b74c8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56473f951a80 0x56473f9546a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.277+0000 7f06b74c8740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56473f955a50 0x56473f94e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.277+0000 7f06b74c8740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56473f8b6520 con 0x56473f9542c0 2024-09-16T01:43:43.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.277+0000 7f06b74c8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56473f85e3e0 con 0x56473f951a80 2024-09-16T01:43:43.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.277+0000 7f06b74c8740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56473f8b24e0 con 0x56473f955a50 2024-09-16T01:43:43.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.277+0000 7f06b24cf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f9542c0 0x56473f999a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.277+0000 7f06b24cf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f9542c0 0x56473f999a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35650/0 (socket says 172.21.15.88:35650) 2024-09-16T01:43:43.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.277+0000 7f06b24cf640 1 -- 172.21.15.88:0/1361910242 learned_addr learned my addr 172.21.15.88:0/1361910242 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:43.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.277+0000 7f06b1cce640 1 --2- 172.21.15.88:0/1361910242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56473f951a80 0x56473f9546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.277+0000 7f06b2cd0640 1 --2- 172.21.15.88:0/1361910242 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56473f955a50 0x56473f94e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.277+0000 7f06b24cf640 1 -- 172.21.15.88:0/1361910242 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56473f955a50 msgr2=0x56473f94e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:43.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.277+0000 7f06b24cf640 1 --2- 172.21.15.88:0/1361910242 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56473f955a50 0x56473f94e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.277+0000 7f06b24cf640 1 -- 172.21.15.88:0/1361910242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56473f951a80 msgr2=0x56473f9546a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:43.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.277+0000 7f06b24cf640 1 --2- 172.21.15.88:0/1361910242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56473f951a80 0x56473f9546a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.277+0000 7f06b24cf640 1 -- 172.21.15.88:0/1361910242 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56473f8792f0 con 0x56473f9542c0 2024-09-16T01:43:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.277+0000 7f06b1cce640 1 --2- 172.21.15.88:0/1361910242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56473f951a80 0x56473f9546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.277+0000 7f06b24cf640 1 --2- 172.21.15.88:0/1361910242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f9542c0 0x56473f999a60 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f06a4011cb0 tx=0x7f06a4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18f2c690f1c9f428 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.277+0000 7f0693fff640 1 -- 172.21.15.88:0/1361910242 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a4027070 con 0x56473f9542c0 2024-09-16T01:43:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.277+0000 7f0693fff640 1 -- 172.21.15.88:0/1361910242 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06a4022420 con 0x56473f9542c0 2024-09-16T01:43:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.277+0000 7f0693fff640 1 -- 172.21.15.88:0/1361910242 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a40227d0 con 0x56473f9542c0 2024-09-16T01:43:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.278+0000 7f06b74c8740 1 -- 172.21.15.88:0/1361910242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f9542c0 msgr2=0x56473f999a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.278+0000 7f06b74c8740 1 --2- 172.21.15.88:0/1361910242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f9542c0 0x56473f999a60 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f06a4011cb0 tx=0x7f06a4014000 comp rx=0 tx=0).stop 2024-09-16T01:43:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.278+0000 7f06b74c8740 1 -- 172.21.15.88:0/1361910242 shutdown_connections 2024-09-16T01:43:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.278+0000 7f06b74c8740 1 --2- 172.21.15.88:0/1361910242 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56473f955a50 0x56473f94e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.278+0000 7f06b74c8740 1 --2- 172.21.15.88:0/1361910242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56473f951a80 0x56473f9546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.278+0000 7f06b74c8740 1 --2- 172.21.15.88:0/1361910242 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f9542c0 0x56473f999a60 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.278+0000 7f06b74c8740 1 -- 172.21.15.88:0/1361910242 >> 172.21.15.88:0/1361910242 conn(0x56473f950a80 msgr2=0x56473f998f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.278+0000 7f06b74c8740 1 -- 172.21.15.88:0/1361910242 shutdown_connections 2024-09-16T01:43:43.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.278+0000 7f06b74c8740 1 -- 172.21.15.88:0/1361910242 wait complete. 2024-09-16T01:43:43.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.279+0000 7f06b74c8740 1 -- start start 2024-09-16T01:43:43.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.279+0000 7f06b74c8740 1 Processor -- start 2024-09-16T01:43:43.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06b74c8740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56473f9542c0 0x56473f975540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06b74c8740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f955a50 0x56473f9724d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06b74c8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56473f94f860 0x56473f97d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06b74c8740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56473f94ff20 con 0x56473f955a50 2024-09-16T01:43:43.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06b74c8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56473f83f450 con 0x56473f94f860 2024-09-16T01:43:43.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06b74c8740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56473f99a290 con 0x56473f9542c0 2024-09-16T01:43:43.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06b74c8740 10 client.0 _ll_register_callbacks cb 0x56473f5c8ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:43:43.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06b1cce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f955a50 0x56473f9724d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06b1cce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f955a50 0x56473f9724d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35656/0 (socket says 172.21.15.88:35656) 2024-09-16T01:43:43.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06b1cce640 1 -- 172.21.15.88:0/3511945639 learned_addr learned my addr 172.21.15.88:0/3511945639 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:43.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06b2cd0640 1 --2- 172.21.15.88:0/3511945639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56473f94f860 0x56473f97d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06b24cf640 1 --2- 172.21.15.88:0/3511945639 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56473f9542c0 0x56473f975540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06b1cce640 1 -- 172.21.15.88:0/3511945639 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56473f9542c0 msgr2=0x56473f975540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:43.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06b1cce640 1 --2- 172.21.15.88:0/3511945639 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56473f9542c0 0x56473f975540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06b1cce640 1 -- 172.21.15.88:0/3511945639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56473f94f860 msgr2=0x56473f97d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:43.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06b1cce640 1 --2- 172.21.15.88:0/3511945639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56473f94f860 0x56473f97d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06b1cce640 1 -- 172.21.15.88:0/3511945639 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56473f5c92d0 con 0x56473f955a50 2024-09-16T01:43:43.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06b2cd0640 1 --2- 172.21.15.88:0/3511945639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56473f94f860 0x56473f97d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:43.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06b1cce640 1 --2- 172.21.15.88:0/3511945639 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f955a50 0x56473f9724d0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009be0 tx=0x7f06a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef819ca18ea1b2fe server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:43.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06917fa640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:43:43.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06917fa640 1 -- 172.21.15.88:0/3511945639 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a0036040 con 0x56473f955a50 2024-09-16T01:43:43.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06917fa640 1 -- 172.21.15.88:0/3511945639 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8576+0+0 (secure 0 0 0) 0x7f06a0031b20 con 0x56473f955a50 2024-09-16T01:43:43.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06b74c8740 1 -- 172.21.15.88:0/3511945639 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06a4022420 con 0x56473f955a50 2024-09-16T01:43:43.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06917fa640 1 -- 172.21.15.88:0/3511945639 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=275) ==== 24+0+0 (secure 0 0 0) 0x7f06a0031d20 con 0x56473f955a50 2024-09-16T01:43:43.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06b74c8740 10 client.9900 fetch_fsmap learned FSMap version 275 2024-09-16T01:43:43.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.280+0000 7f06b74c8740 1 -- 172.21.15.88:0/3511945639 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=275,monmap=2+}) -- 0x56473f6936b0 con 0x56473f955a50 2024-09-16T01:43:43.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.281+0000 7f06917fa640 1 -- 172.21.15.88:0/3511945639 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06a4022420 con 0x56473f955a50 2024-09-16T01:43:43.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.281+0000 7f06917fa640 1 -- 172.21.15.88:0/3511945639 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 275) ==== 3502+0+0 (secure 0 0 0) 0x7f06a004d430 con 0x56473f955a50 2024-09-16T01:43:43.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.281+0000 7f06b74c8740 10 client.9900 fetch_fsmap finished waiting for FSMap version 275 2024-09-16T01:43:43.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.281+0000 7f06b74c8740 10 client.9900 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:43:43.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.281+0000 7f06b74c8740 20 client.9900 populate_metadata read hostname 'smithi088' 2024-09-16T01:43:43.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.281+0000 7f06b74c8740 1 --2- 172.21.15.88:0/3511945639 >> [v2:172.21.15.88:6834/4135629195,v1:172.21.15.88:6836/4135629195] conn(0x56473f9737f0 0x56473f973bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.281+0000 7f06b74c8740 4 client.9900 mds_command: new command op to 9852 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:43:43.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.281+0000 7f06b74c8740 1 -- 172.21.15.88:0/3511945639 --> [v2:172.21.15.88:6834/4135629195,v1:172.21.15.88:6836/4135629195] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56473f8c3c50 con 0x56473f9737f0 2024-09-16T01:43:43.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.281+0000 7f06b24cf640 1 --2- 172.21.15.88:0/3511945639 >> [v2:172.21.15.88:6834/4135629195,v1:172.21.15.88:6836/4135629195] conn(0x56473f9737f0 0x56473f973bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.281+0000 7f06917fa640 10 client.9900 ms_handle_connect on v2:172.21.15.88:6834/4135629195 2024-09-16T01:43:43.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.281+0000 7f06b24cf640 1 --2- 172.21.15.88:0/3511945639 >> [v2:172.21.15.88:6834/4135629195,v1:172.21.15.88:6836/4135629195] conn(0x56473f9737f0 0x56473f973bd0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5810750860a68ee3 server_cookie=524e14383eec1cdb in_seq=0 out_seq=0 2024-09-16T01:43:43.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.282+0000 7f06917fa640 1 -- 172.21.15.88:0/3511945639 <== mds.0 v2:172.21.15.88:6834/4135629195 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56473f8c3c50 con 0x56473f9737f0 2024-09-16T01:43:43.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.282+0000 7f06917fa640 10 client.9900 handle_command_reply: tid=0 2024-09-16T01:43:43.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.282+0000 7f06b74c8740 1 client.9900 shutdown 2024-09-16T01:43:43.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.282+0000 7f06b74c8740 1 -- 172.21.15.88:0/3511945639 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f955a50 msgr2=0x56473f9724d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:43.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.282+0000 7f06b74c8740 1 --2- 172.21.15.88:0/3511945639 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f955a50 0x56473f9724d0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009be0 tx=0x7f06a00046c0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.282+0000 7f06b74c8740 1 -- 172.21.15.88:0/3511945639 shutdown_connections 2024-09-16T01:43:43.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.282+0000 7f06b74c8740 1 --2- 172.21.15.88:0/3511945639 >> [v2:172.21.15.88:6834/4135629195,v1:172.21.15.88:6836/4135629195] conn(0x56473f9737f0 0x56473f973bd0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.282+0000 7f06b74c8740 1 --2- 172.21.15.88:0/3511945639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56473f94f860 0x56473f97d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.282+0000 7f06b74c8740 1 --2- 172.21.15.88:0/3511945639 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f955a50 0x56473f9724d0 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.282+0000 7f06917fa640 0 client.9900 ms_handle_reset on v2:172.21.15.88:6834/4135629195 2024-09-16T01:43:43.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.282+0000 7f06b74c8740 1 --2- 172.21.15.88:0/3511945639 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56473f9542c0 0x56473f975540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.282+0000 7f06b74c8740 1 -- 172.21.15.88:0/3511945639 >> 172.21.15.88:0/3511945639 conn(0x56473f951a80 msgr2=0x56473f997fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:43.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.282+0000 7f06b74c8740 1 -- 172.21.15.88:0/3511945639 shutdown_connections 2024-09-16T01:43:43.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.282+0000 7f06b74c8740 1 -- 172.21.15.88:0/3511945639 wait complete. 2024-09-16T01:43:43.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.282+0000 7f06b74c8740 20 client.9900 trim_cache size 0 max 16384 2024-09-16T01:43:43.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.300+0000 7f06b74c8740 1 Processor -- start 2024-09-16T01:43:43.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.300+0000 7f06b74c8740 1 -- start start 2024-09-16T01:43:43.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.301+0000 7f06b74c8740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56473f94f860 0x56473f996ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.301+0000 7f06b74c8740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f9542c0 0x56473f997420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.301+0000 7f06b74c8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56473f955a50 0x56473f998e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.301+0000 7f06b74c8740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f06ac1fdaa0 con 0x56473f9542c0 2024-09-16T01:43:43.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.301+0000 7f06b74c8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f06ac1fce00 con 0x56473f955a50 2024-09-16T01:43:43.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.301+0000 7f06b74c8740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f06ac1fd450 con 0x56473f94f860 2024-09-16T01:43:43.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.301+0000 7f06b1cce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f9542c0 0x56473f997420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.301+0000 7f06b1cce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f9542c0 0x56473f997420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35658/0 (socket says 172.21.15.88:35658) 2024-09-16T01:43:43.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.301+0000 7f06b1cce640 1 -- 172.21.15.88:0/3862892720 learned_addr learned my addr 172.21.15.88:0/3862892720 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:43.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.301+0000 7f06b1cce640 1 -- 172.21.15.88:0/3862892720 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56473f94f860 msgr2=0x56473f996ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:43.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.301+0000 7f06b1cce640 1 --2- 172.21.15.88:0/3862892720 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56473f94f860 0x56473f996ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.301+0000 7f06b1cce640 1 -- 172.21.15.88:0/3862892720 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56473f955a50 msgr2=0x56473f998e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:43.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.301+0000 7f06b1cce640 1 --2- 172.21.15.88:0/3862892720 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56473f955a50 0x56473f998e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.301+0000 7f06b1cce640 1 -- 172.21.15.88:0/3862892720 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56473f94ef70 con 0x56473f9542c0 2024-09-16T01:43:43.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.301+0000 7f06b1cce640 1 --2- 172.21.15.88:0/3862892720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f9542c0 0x56473f997420 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f06a004d5e0 tx=0x7f06a003c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a103c62a7981787d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:43.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.301+0000 7f06937fe640 1 -- 172.21.15.88:0/3862892720 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a0036040 con 0x56473f9542c0 2024-09-16T01:43:43.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.302+0000 7f06937fe640 1 -- 172.21.15.88:0/3862892720 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06a004d930 con 0x56473f9542c0 2024-09-16T01:43:43.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.302+0000 7f06937fe640 1 -- 172.21.15.88:0/3862892720 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a00319a0 con 0x56473f9542c0 2024-09-16T01:43:43.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.302+0000 7f06b74c8740 1 -- 172.21.15.88:0/3862892720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f9542c0 msgr2=0x56473f997420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:43.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.302+0000 7f06b74c8740 1 --2- 172.21.15.88:0/3862892720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f9542c0 0x56473f997420 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f06a004d5e0 tx=0x7f06a003c000 comp rx=0 tx=0).stop 2024-09-16T01:43:43.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.302+0000 7f06b74c8740 1 -- 172.21.15.88:0/3862892720 shutdown_connections 2024-09-16T01:43:43.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.302+0000 7f06b74c8740 1 --2- 172.21.15.88:0/3862892720 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56473f955a50 0x56473f998e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.302+0000 7f06b74c8740 1 --2- 172.21.15.88:0/3862892720 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f9542c0 0x56473f997420 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.302+0000 7f06b74c8740 1 --2- 172.21.15.88:0/3862892720 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56473f94f860 0x56473f996ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.302+0000 7f06b74c8740 1 -- 172.21.15.88:0/3862892720 >> 172.21.15.88:0/3862892720 conn(0x56473f951a80 msgr2=0x56473f950290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:43.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.302+0000 7f06b74c8740 1 -- 172.21.15.88:0/3862892720 shutdown_connections 2024-09-16T01:43:43.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.302+0000 7f06b74c8740 1 -- 172.21.15.88:0/3862892720 wait complete. 2024-09-16T01:43:43.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.303+0000 7f06b74c8740 1 -- start start 2024-09-16T01:43:43.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.303+0000 7f06b74c8740 1 Processor -- start 2024-09-16T01:43:43.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06b74c8740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56473f94f860 0x56473f998970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06b74c8740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56473f9542c0 0x56473fa17550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06b74c8740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f955a50 0x56473f994970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06b74c8740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56473f97df10 con 0x56473f955a50 2024-09-16T01:43:43.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06b74c8740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56473f97c1d0 con 0x56473f94f860 2024-09-16T01:43:43.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06b74c8740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56473f952db0 con 0x56473f9542c0 2024-09-16T01:43:43.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06b74c8740 10 client.0 _ll_register_callbacks cb 0x7f06a0034070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:43:43.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06b2cd0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f955a50 0x56473f994970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06b2cd0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f955a50 0x56473f994970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35674/0 (socket says 172.21.15.88:35674) 2024-09-16T01:43:43.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06b2cd0640 1 -- 172.21.15.88:0/2171559127 learned_addr learned my addr 172.21.15.88:0/2171559127 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:43.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06b1cce640 1 --2- 172.21.15.88:0/2171559127 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56473f9542c0 0x56473fa17550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06b24cf640 1 --2- 172.21.15.88:0/2171559127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56473f94f860 0x56473f998970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06b2cd0640 1 -- 172.21.15.88:0/2171559127 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56473f9542c0 msgr2=0x56473fa17550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:43.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06b2cd0640 1 --2- 172.21.15.88:0/2171559127 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56473f9542c0 0x56473fa17550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06b2cd0640 1 -- 172.21.15.88:0/2171559127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56473f94f860 msgr2=0x56473f998970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:43.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06b2cd0640 1 --2- 172.21.15.88:0/2171559127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56473f94f860 0x56473f998970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06b2cd0640 1 -- 172.21.15.88:0/2171559127 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56473f978960 con 0x56473f955a50 2024-09-16T01:43:43.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06b24cf640 1 --2- 172.21.15.88:0/2171559127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56473f94f860 0x56473f998970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:43.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06b2cd0640 1 --2- 172.21.15.88:0/2171559127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f955a50 0x56473f994970 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f069c059470 tx=0x7f069c059190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de7c824186a8e1c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:43.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06917fa640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:43:43.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06917fa640 1 -- 172.21.15.88:0/2171559127 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c003d50 con 0x56473f955a50 2024-09-16T01:43:43.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06b74c8740 1 -- 172.21.15.88:0/2171559127 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06a004d930 con 0x56473f955a50 2024-09-16T01:43:43.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06917fa640 1 -- 172.21.15.88:0/2171559127 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8576+0+0 (secure 0 0 0) 0x7f069c058bc0 con 0x56473f955a50 2024-09-16T01:43:43.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06917fa640 1 -- 172.21.15.88:0/2171559127 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=275) ==== 24+0+0 (secure 0 0 0) 0x7f069c05db20 con 0x56473f955a50 2024-09-16T01:43:43.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06b74c8740 10 client.9906 fetch_fsmap learned FSMap version 275 2024-09-16T01:43:43.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.304+0000 7f06b74c8740 1 -- 172.21.15.88:0/2171559127 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=275,monmap=2+}) -- 0x56473f9bcc30 con 0x56473f955a50 2024-09-16T01:43:43.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.305+0000 7f06917fa640 1 -- 172.21.15.88:0/2171559127 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06a004d930 con 0x56473f955a50 2024-09-16T01:43:43.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.305+0000 7f06917fa640 1 -- 172.21.15.88:0/2171559127 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 275) ==== 3502+0+0 (secure 0 0 0) 0x7f069c0554c0 con 0x56473f955a50 2024-09-16T01:43:43.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.305+0000 7f06b74c8740 10 client.9906 fetch_fsmap finished waiting for FSMap version 275 2024-09-16T01:43:43.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.305+0000 7f06b74c8740 10 client.9906 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:43:43.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.305+0000 7f06b74c8740 20 client.9906 populate_metadata read hostname 'smithi088' 2024-09-16T01:43:43.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.305+0000 7f06b74c8740 1 --2- 172.21.15.88:0/2171559127 >> [v2:172.21.15.88:6834/4135629195,v1:172.21.15.88:6836/4135629195] conn(0x56473fa1b1b0 0x56473fa1b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.305+0000 7f06b74c8740 4 client.9906 mds_command: new command op to 9852 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:43:43.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.305+0000 7f06b74c8740 1 -- 172.21.15.88:0/2171559127 --> [v2:172.21.15.88:6834/4135629195,v1:172.21.15.88:6836/4135629195] -- command(tid 0: {"prefix": "status"}) -- 0x56473f8bdfa0 con 0x56473fa1b1b0 2024-09-16T01:43:43.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.305+0000 7f06b24cf640 1 --2- 172.21.15.88:0/2171559127 >> [v2:172.21.15.88:6834/4135629195,v1:172.21.15.88:6836/4135629195] conn(0x56473fa1b1b0 0x56473fa1b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.305+0000 7f06917fa640 10 client.9906 ms_handle_connect on v2:172.21.15.88:6834/4135629195 2024-09-16T01:43:43.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.305+0000 7f06b24cf640 1 --2- 172.21.15.88:0/2171559127 >> [v2:172.21.15.88:6834/4135629195,v1:172.21.15.88:6836/4135629195] conn(0x56473fa1b1b0 0x56473fa1b590 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1a27a1abd768454 server_cookie=1dc89d87449a3fdf in_seq=0 out_seq=0 2024-09-16T01:43:43.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.305+0000 7f06917fa640 1 -- 172.21.15.88:0/2171559127 <== mds.0 v2:172.21.15.88:6834/4135629195 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56473f8bdfa0 con 0x56473fa1b1b0 2024-09-16T01:43:43.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.305+0000 7f06917fa640 10 client.9906 handle_command_reply: tid=0 2024-09-16T01:43:43.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.305+0000 7f06b74c8740 1 client.9906 shutdown 2024-09-16T01:43:43.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.305+0000 7f06b74c8740 1 -- 172.21.15.88:0/2171559127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f955a50 msgr2=0x56473f994970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:43.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.305+0000 7f06b74c8740 1 --2- 172.21.15.88:0/2171559127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f955a50 0x56473f994970 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f069c059470 tx=0x7f069c059190 comp rx=0 tx=0).stop 2024-09-16T01:43:43.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.305+0000 7f06b74c8740 1 -- 172.21.15.88:0/2171559127 shutdown_connections 2024-09-16T01:43:43.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.305+0000 7f06b74c8740 1 --2- 172.21.15.88:0/2171559127 >> [v2:172.21.15.88:6834/4135629195,v1:172.21.15.88:6836/4135629195] conn(0x56473fa1b1b0 0x56473fa1b590 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.306+0000 7f06917fa640 0 client.9906 ms_handle_reset on v2:172.21.15.88:6834/4135629195 2024-09-16T01:43:43.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.305+0000 7f06b74c8740 1 --2- 172.21.15.88:0/2171559127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56473f955a50 0x56473f994970 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.306+0000 7f06b74c8740 1 --2- 172.21.15.88:0/2171559127 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56473f9542c0 0x56473fa17550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.306+0000 7f06b74c8740 1 --2- 172.21.15.88:0/2171559127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56473f94f860 0x56473f998970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.306+0000 7f06b74c8740 1 -- 172.21.15.88:0/2171559127 >> 172.21.15.88:0/2171559127 conn(0x56473f951a80 msgr2=0x56473f995e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:43.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.306+0000 7f06b74c8740 1 -- 172.21.15.88:0/2171559127 shutdown_connections 2024-09-16T01:43:43.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.306+0000 7f06b74c8740 1 -- 172.21.15.88:0/2171559127 wait complete. 2024-09-16T01:43:43.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.306+0000 7f06b74c8740 20 client.9906 trim_cache size 0 max 16384 2024-09-16T01:43:43.316 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:43:43.316 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "5ace2c91-46f9-482b-8a89-731fbdc171bf", 2024-09-16T01:43:43.316 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2024-09-16T01:43:43.316 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 9852, 2024-09-16T01:43:43.316 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2024-09-16T01:43:43.316 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2024-09-16T01:43:43.316 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2024-09-16T01:43:43.316 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 6.6199594370000003, 2024-09-16T01:43:43.316 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 275, 2024-09-16T01:43:43.316 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 120, 2024-09-16T01:43:43.317 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 120, 2024-09-16T01:43:43.317 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 7.860198317 2024-09-16T01:43:43.317 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:43:43.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.306+0000 7f06b4733640 1 -- 172.21.15.88:0/1241195622 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f069806c2e0 msgr2=0x7f069806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:43.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.306+0000 7f06b4733640 1 --2- 172.21.15.88:0/1241195622 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f069806c2e0 0x7f069806e7a0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f06a4001860 tx=0x7f06a4007040 comp rx=0 tx=0).stop 2024-09-16T01:43:43.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.306+0000 7f06b4733640 1 -- 172.21.15.88:0/1241195622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06ac153ca0 msgr2=0x7f06ac0c7880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:43.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.306+0000 7f06b4733640 1 --2- 172.21.15.88:0/1241195622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06ac153ca0 0x7f06ac0c7880 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f06ac06e110 tx=0x7f069c0115a0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.306+0000 7f06b4733640 1 -- 172.21.15.88:0/1241195622 shutdown_connections 2024-09-16T01:43:43.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.306+0000 7f06b4733640 1 --2- 172.21.15.88:0/1241195622 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f069806c2e0 0x7f069806e7a0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.306+0000 7f06b4733640 1 --2- 172.21.15.88:0/1241195622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06ac153ca0 0x7f06ac0c7880 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.306+0000 7f06b4733640 1 --2- 172.21.15.88:0/1241195622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06ac0cc4b0 0x7f06ac1f0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.306+0000 7f06b4733640 1 --2- 172.21.15.88:0/1241195622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0694004800 0x7f06ac1f1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.306+0000 7f06b4733640 1 -- 172.21.15.88:0/1241195622 >> 172.21.15.88:0/1241195622 conn(0x7f06ac06d0f0 msgr2=0x7f06ac159bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:43.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.306+0000 7f06b4733640 1 -- 172.21.15.88:0/1241195622 shutdown_connections 2024-09-16T01:43:43.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.306+0000 7f06b4733640 1 -- 172.21.15.88:0/1241195622 wait complete. 2024-09-16T01:43:43.319 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-09-16T01:43:43.319 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Corrupt omap key 1.00000000:subdir_head', proceeding to ls 2024-09-16T01:43:43.319 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:43:43.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.416+0000 7fbb7a42e640 1 Processor -- start 2024-09-16T01:43:43.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.416+0000 7fbb7a42e640 1 -- start start 2024-09-16T01:43:43.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.416+0000 7fbb7a42e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbb74073180 0x7fbb740735d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.416+0000 7fbb7a42e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb7415e920 con 0x7fbb7415e540 2024-09-16T01:43:43.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.417+0000 7fbb7a42e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb74162a50 con 0x7fbb740b01f0 2024-09-16T01:43:43.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.417+0000 7fbb7a42e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbb74073b10 con 0x7fbb74073180 2024-09-16T01:43:43.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.417+0000 7fbb73fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fbb7415e540 0x7fbb740b52c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42382/0 (socket says 172.21.15.88:42382) 2024-09-16T01:43:43.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.417+0000 7fbb73fff640 1 -- 172.21.15.88:0/3533204580 learned_addr learned my addr 172.21.15.88:0/3533204580 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:43.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.417+0000 7fbb72ffd640 1 -- 172.21.15.88:0/3533204580 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3970112199 0 0) 0x7fbb7415e920 con 0x7fbb7415e540 2024-09-16T01:43:43.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.417+0000 7fbb72ffd640 1 -- 172.21.15.88:0/3533204580 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb5c003520 con 0x7fbb7415e540 2024-09-16T01:43:43.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.417+0000 7fbb789cb640 1 --2- 172.21.15.88:0/3533204580 >> v2:172.21.15.154:3300/0 conn(0x7fbb74073180 0x7fbb740735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.417+0000 7fbb72ffd640 1 -- 172.21.15.88:0/3533204580 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 592015616 0 0) 0x7fbb5c003520 con 0x7fbb7415e540 2024-09-16T01:43:43.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.417+0000 7fbb72ffd640 1 -- 172.21.15.88:0/3533204580 >> v2:172.21.15.154:3300/0 conn(0x7fbb74073180 msgr2=0x7fbb740735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:43.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.417+0000 7fbb72ffd640 1 --2- 172.21.15.88:0/3533204580 >> v2:172.21.15.154:3300/0 conn(0x7fbb74073180 0x7fbb740735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.417+0000 7fbb72ffd640 1 -- 172.21.15.88:0/3533204580 >> v1:172.21.15.154:6789/0 conn(0x7fbb740b01f0 legacy=0x7fbb741620e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:43.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.417+0000 7fbb72ffd640 1 -- 172.21.15.88:0/3533204580 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb74162d30 con 0x7fbb7415e540 2024-09-16T01:43:43.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.417+0000 7fbb72ffd640 1 -- 172.21.15.88:0/3533204580 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fbb64001670 con 0x7fbb7415e540 2024-09-16T01:43:43.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.417+0000 7fbb789cb640 1 --2- 172.21.15.88:0/3533204580 >> v2:172.21.15.154:3300/0 conn(0x7fbb74073180 0x7fbb740735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:43.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.417+0000 7fbb72ffd640 1 -- 172.21.15.88:0/3533204580 >> v1:172.21.15.88:6789/0 conn(0x7fbb7415e540 legacy=0x7fbb740b52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:43.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.417+0000 7fbb72ffd640 1 --2- 172.21.15.88:0/3533204580 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb5c0039b0 0x7fbb5c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.417+0000 7fbb72ffd640 1 --2- 172.21.15.88:0/3533204580 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb5c004800 0x7fbb5c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.417+0000 7fbb72ffd640 1 --2- 172.21.15.88:0/3533204580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb5c0053c0 0x7fbb5c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.417+0000 7fbb72ffd640 1 -- 172.21.15.88:0/3533204580 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbb74073b10 con 0x7fbb5c0053c0 2024-09-16T01:43:43.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.417+0000 7fbb72ffd640 1 -- 172.21.15.88:0/3533204580 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5c00c250 con 0x7fbb5c004800 2024-09-16T01:43:43.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.417+0000 7fbb72ffd640 1 -- 172.21.15.88:0/3533204580 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbb5c00c450 con 0x7fbb5c0039b0 2024-09-16T01:43:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.417+0000 7fbb789cb640 1 --2- 172.21.15.88:0/3533204580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb5c0053c0 0x7fbb5c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.417+0000 7fbb737fe640 1 --2- 172.21.15.88:0/3533204580 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb5c004800 0x7fbb5c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.417+0000 7fbb73fff640 1 --2- 172.21.15.88:0/3533204580 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb5c0039b0 0x7fbb5c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.418+0000 7fbb789cb640 1 -- 172.21.15.88:0/3533204580 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb5c0039b0 msgr2=0x7fbb5c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.418+0000 7fbb789cb640 1 --2- 172.21.15.88:0/3533204580 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb5c0039b0 0x7fbb5c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.418+0000 7fbb789cb640 1 -- 172.21.15.88:0/3533204580 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb5c004800 msgr2=0x7fbb5c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.418+0000 7fbb789cb640 1 --2- 172.21.15.88:0/3533204580 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb5c004800 0x7fbb5c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.418+0000 7fbb789cb640 1 -- 172.21.15.88:0/3533204580 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb5c003520 con 0x7fbb5c0053c0 2024-09-16T01:43:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.418+0000 7fbb789cb640 1 -- 172.21.15.88:0/3533204580 reap_dead start 2024-09-16T01:43:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.418+0000 7fbb789cb640 1 --2- 172.21.15.88:0/3533204580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb5c0053c0 0x7fbb5c00b9c0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0073e0 tx=0x7fbb5c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c734e4fa26797b17 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.419+0000 7fbb72ffd640 1 -- 172.21.15.88:0/3533204580 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c009c80 con 0x7fbb5c0053c0 2024-09-16T01:43:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.419+0000 7fbb72ffd640 1 -- 172.21.15.88:0/3533204580 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb6c02c6c0 con 0x7fbb5c0053c0 2024-09-16T01:43:43.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.419+0000 7fbb72ffd640 1 -- 172.21.15.88:0/3533204580 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c02cab0 con 0x7fbb5c0053c0 2024-09-16T01:43:43.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.419+0000 7fbb7a42e640 1 -- 172.21.15.88:0/3533204580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb5c0053c0 msgr2=0x7fbb5c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:43.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.419+0000 7fbb7a42e640 1 --2- 172.21.15.88:0/3533204580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb5c0053c0 0x7fbb5c00b9c0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0073e0 tx=0x7fbb5c005830 comp rx=0 tx=0).stop 2024-09-16T01:43:43.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.419+0000 7fbb7a42e640 1 -- 172.21.15.88:0/3533204580 shutdown_connections 2024-09-16T01:43:43.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.419+0000 7fbb7a42e640 1 --2- 172.21.15.88:0/3533204580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb5c0053c0 0x7fbb5c00b9c0 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.419+0000 7fbb7a42e640 1 -- 172.21.15.88:0/3533204580 >> 172.21.15.88:0/3533204580 conn(0x7fbb7406bc10 msgr2=0x7fbb7406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:43.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.419+0000 7fbb7a42e640 1 -- 172.21.15.88:0/3533204580 shutdown_connections 2024-09-16T01:43:43.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.419+0000 7fbb7a42e640 1 -- 172.21.15.88:0/3533204580 wait complete. 2024-09-16T01:43:43.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.420+0000 7fbb7a42e640 1 Processor -- start 2024-09-16T01:43:43.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.421+0000 7fbb7a42e640 1 -- start start 2024-09-16T01:43:43.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.421+0000 7fbb7a42e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb5c0053c0 0x7fbb7414d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.421+0000 7fbb7a42e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb741604f0 0x7fbb741f7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.421+0000 7fbb7a42e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb741523f0 0x7fbb74152880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.421+0000 7fbb7a42e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbb740aeca0 con 0x7fbb5c0053c0 2024-09-16T01:43:43.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.421+0000 7fbb7a42e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbb74163010 con 0x7fbb741523f0 2024-09-16T01:43:43.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.421+0000 7fbb7a42e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbb740735d0 con 0x7fbb741604f0 2024-09-16T01:43:43.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.421+0000 7fbb73fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb5c0053c0 0x7fbb7414d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.421+0000 7fbb73fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb5c0053c0 0x7fbb7414d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35686/0 (socket says 172.21.15.88:35686) 2024-09-16T01:43:43.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.421+0000 7fbb73fff640 1 -- 172.21.15.88:0/1589585666 learned_addr learned my addr 172.21.15.88:0/1589585666 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:43.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.422+0000 7fbb73fff640 1 -- 172.21.15.88:0/1589585666 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb741604f0 msgr2=0x7fbb741f7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:43.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.422+0000 7fbb73fff640 1 --2- 172.21.15.88:0/1589585666 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb741604f0 0x7fbb741f7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.422+0000 7fbb73fff640 1 -- 172.21.15.88:0/1589585666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb741523f0 msgr2=0x7fbb74152880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:43.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.422+0000 7fbb73fff640 1 --2- 172.21.15.88:0/1589585666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb741523f0 0x7fbb74152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.422+0000 7fbb73fff640 1 -- 172.21.15.88:0/1589585666 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb74161a70 con 0x7fbb5c0053c0 2024-09-16T01:43:43.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.422+0000 7fbb73fff640 1 --2- 172.21.15.88:0/1589585666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb5c0053c0 0x7fbb7414d800 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7fbb64004f00 tx=0x7fbb64007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:43.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.422+0000 7fbb70ff9640 1 -- 172.21.15.88:0/1589585666 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6400f7b0 con 0x7fbb5c0053c0 2024-09-16T01:43:43.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.422+0000 7fbb70ff9640 1 -- 172.21.15.88:0/1589585666 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb64018070 con 0x7fbb5c0053c0 2024-09-16T01:43:43.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.422+0000 7fbb70ff9640 1 -- 172.21.15.88:0/1589585666 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6400fbd0 con 0x7fbb5c0053c0 2024-09-16T01:43:43.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.422+0000 7fbb7a42e640 1 -- 172.21.15.88:0/1589585666 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb740d0290 con 0x7fbb5c0053c0 2024-09-16T01:43:43.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.422+0000 7fbb7a42e640 1 -- 172.21.15.88:0/1589585666 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb74160cb0 con 0x7fbb5c0053c0 2024-09-16T01:43:43.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.423+0000 7fbb7a42e640 1 -- 172.21.15.88:0/1589585666 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb740b5a80 con 0x7fbb5c0053c0 2024-09-16T01:43:43.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.423+0000 7fbb70ff9640 1 -- 172.21.15.88:0/1589585666 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fbb6401d460 con 0x7fbb5c0053c0 2024-09-16T01:43:43.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.424+0000 7fbb70ff9640 1 --2- 172.21.15.88:0/1589585666 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbb6006c2b0 0x7fbb6006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:43.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.424+0000 7fbb70ff9640 1 -- 172.21.15.88:0/1589585666 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8576+0+0 (secure 0 0 0) 0x7fbb6408bed0 con 0x7fbb5c0053c0 2024-09-16T01:43:43.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.424+0000 7fbb737fe640 1 --2- 172.21.15.88:0/1589585666 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbb6006c2b0 0x7fbb6006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:43.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.424+0000 7fbb737fe640 1 --2- 172.21.15.88:0/1589585666 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbb6006c2b0 0x7fbb6006e770 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fbb68000c00 tx=0x7fbb68009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:43.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.428+0000 7fbb70ff9640 1 -- 172.21.15.88:0/1589585666 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fbb64059b50 con 0x7fbb5c0053c0 2024-09-16T01:43:43.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.628+0000 7fbb7a42e640 1 -- 172.21.15.88:0/1589585666 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fbb741543a0 con 0x7fbb5c0053c0 2024-09-16T01:43:43.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.629+0000 7fbb70ff9640 1 -- 172.21.15.88:0/1589585666 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v275) ==== 53+0+73 (secure 0 0 0) 0x7fbb64059450 con 0x7fbb5c0053c0 2024-09-16T01:43:43.630 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:43:43.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.631+0000 7fbb7a42e640 1 -- 172.21.15.88:0/1589585666 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbb6006c2b0 msgr2=0x7fbb6006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:43.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.631+0000 7fbb7a42e640 1 --2- 172.21.15.88:0/1589585666 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbb6006c2b0 0x7fbb6006e770 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fbb68000c00 tx=0x7fbb68009040 comp rx=0 tx=0).stop 2024-09-16T01:43:43.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.631+0000 7fbb7a42e640 1 -- 172.21.15.88:0/1589585666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb5c0053c0 msgr2=0x7fbb7414d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:43.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.631+0000 7fbb7a42e640 1 --2- 172.21.15.88:0/1589585666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb5c0053c0 0x7fbb7414d800 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7fbb64004f00 tx=0x7fbb64007430 comp rx=0 tx=0).stop 2024-09-16T01:43:43.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.631+0000 7fbb7a42e640 1 -- 172.21.15.88:0/1589585666 shutdown_connections 2024-09-16T01:43:43.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.631+0000 7fbb7a42e640 1 --2- 172.21.15.88:0/1589585666 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbb6006c2b0 0x7fbb6006e770 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.631+0000 7fbb7a42e640 1 --2- 172.21.15.88:0/1589585666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb741523f0 0x7fbb74152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.631+0000 7fbb7a42e640 1 --2- 172.21.15.88:0/1589585666 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbb741604f0 0x7fbb741f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.631+0000 7fbb7a42e640 1 --2- 172.21.15.88:0/1589585666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb5c0053c0 0x7fbb7414d800 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:43.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.631+0000 7fbb7a42e640 1 -- 172.21.15.88:0/1589585666 >> 172.21.15.88:0/1589585666 conn(0x7fbb7406bc10 msgr2=0x7fbb7406c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:43.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.631+0000 7fbb7a42e640 1 -- 172.21.15.88:0/1589585666 shutdown_connections 2024-09-16T01:43:43.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43.631+0000 7fbb7a42e640 1 -- 172.21.15.88:0/1589585666 wait complete. 2024-09-16T01:43:43.643 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-16T01:43:43.643 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:43:43.643 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2024-09-16T01:43:43.644 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T01:43:43.644 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T01:43:43.644 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-16T01:43:43.644 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-16T01:43:43.644 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:43:43.644 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:43:43.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:43.644 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2024-09-16T01:43:43.660 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:43:43.660 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:43:43.661 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:43:43.661 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:43:43.661 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-09-16T01:43:43.661 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:43:43.661 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:43:43.661 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:43:43.661 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2024-09-16T01:43:43.661 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:43:43.661 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:43:43.661 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2024-09-16T01:43:43.661 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:43:43.661 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2024-09-16T01:43:43.662 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:43:43.662 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2024-09-16T01:43:43.662 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:43:43.662 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42200sec preferred_lft 42200sec 2024-09-16T01:43:43.662 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link noprefixroute 2024-09-16T01:43:43.662 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:43:43.662 INFO:teuthology.orchestra.run.smithi088.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:43:43.662 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff 2024-09-16T01:43:43.662 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T01:43:43.662 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:43:43.663 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::e486:6eff:fe81:a1c/64 scope link 2024-09-16T01:43:43.663 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:43:43.663 INFO:teuthology.orchestra.run.smithi088.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:43:43.663 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 12:6f:29:be:95:48 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:43:43.663 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::106f:29ff:febe:9548/64 scope link 2024-09-16T01:43:43.663 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:43:43.663 INFO:teuthology.orchestra.run.smithi088.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:43:43.663 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T01:43:43.663 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::c63:b4ff:feb3:ad9e/64 scope link 2024-09-16T01:43:43.663 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:43:43.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:43.664 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:43:43.724 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-09-16T01:43:43.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:43.725 DEBUG:teuthology.orchestra.run.smithi088:> route 2024-09-16T01:43:43.782 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2024-09-16T01:43:43.782 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:43:43.782 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:43:43.782 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:43:43.783 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:43:43.783 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:43:43.783 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:43:43.783 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:43:43.783 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:43:43.783 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:43:43.783 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:43:43.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:43:43.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:43:43.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:43.909 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:43:43.926 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T01:43:43.926 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T01:43:43.927 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:43:43.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:43.928 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2024-09-16T01:43:43.992 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:43:43.992 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T01:43:44.049 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:43:44.049 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:43:44.050 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:43:44.104 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-16T01:43:44.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:44.105 DEBUG:teuthology.orchestra.run.smithi088:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:43:44.187 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:43:44.187 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:43:44.187 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:43:44.196 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:43:44.196 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:43:44.196 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:43:44.196 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:43:44.196 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:43:44.202 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:43:44.202 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:43:44.202 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:43:44.202 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:43:44.202 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:43:44.202 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790" 2024-09-16T01:43:44.202 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2024-09-16T01:43:44.203 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5ace2c91-46f9-482b-8a89-731fbdc171bf" will pass to kernel 2024-09-16T01:43:44.203 INFO:teuthology.orchestra.run.smithi088.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-16T01:43:44.203 INFO:teuthology.orchestra.run.smithi088.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:43:44.203 INFO:teuthology.orchestra.run.smithi088.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:43:44.203 INFO:teuthology.orchestra.run.smithi088.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:43:44.203 INFO:teuthology.orchestra.run.smithi088.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:43:44.204 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:43:44.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:44.205 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:43:44.270 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:43:44.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:43:44.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:43:44.366 INFO:teuthology.orchestra.run.smithi088.stdout:client7286 2024-09-16T01:43:44.367 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:43:44.367 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:43:44.385 DEBUG:teuthology.orchestra.run.smithi088:> stdin-killer -- python3 -c ' 2024-09-16T01:43:44.386 DEBUG:teuthology.orchestra.run.smithi088:> import os 2024-09-16T01:43:44.386 DEBUG:teuthology.orchestra.run.smithi088:> import stat 2024-09-16T01:43:44.386 DEBUG:teuthology.orchestra.run.smithi088:> import json 2024-09-16T01:43:44.386 DEBUG:teuthology.orchestra.run.smithi088:> import sys 2024-09-16T01:43:44.387 DEBUG:teuthology.orchestra.run.smithi088:> 2024-09-16T01:43:44.387 DEBUG:teuthology.orchestra.run.smithi088:> try: 2024-09-16T01:43:44.387 DEBUG:teuthology.orchestra.run.smithi088:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/./subdir'"'"') 2024-09-16T01:43:44.387 DEBUG:teuthology.orchestra.run.smithi088:> except OSError as e: 2024-09-16T01:43:44.387 DEBUG:teuthology.orchestra.run.smithi088:> sys.exit(e.errno) 2024-09-16T01:43:44.388 DEBUG:teuthology.orchestra.run.smithi088:> 2024-09-16T01:43:44.388 DEBUG:teuthology.orchestra.run.smithi088:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-09-16T01:43:44.388 DEBUG:teuthology.orchestra.run.smithi088:> print(json.dumps( 2024-09-16T01:43:44.388 DEBUG:teuthology.orchestra.run.smithi088:> dict([(a, getattr(s, a)) for a in attrs]), 2024-09-16T01:43:44.389 DEBUG:teuthology.orchestra.run.smithi088:> indent=2)) 2024-09-16T01:43:44.389 DEBUG:teuthology.orchestra.run.smithi088:> ' 2024-09-16T01:43:44.427 INFO:teuthology.orchestra.run:waiting for 20 2024-09-16T01:43:44.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44 stdin-killer INFO: command exited with status 5: exiting normally with same code! 2024-09-16T01:43:44.500 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-09-16T01:43:44.501 INFO:tasks.cephfs.test_damage:Result: EIO on client 2024-09-16T01:43:44.501 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:43:44.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.638+0000 7f7cd46f1640 1 Processor -- start 2024-09-16T01:43:44.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.638+0000 7f7cd46f1640 1 -- start start 2024-09-16T01:43:44.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.639+0000 7f7cd46f1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ccc073080 0x7f7ccc0b5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:44.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.639+0000 7f7cd46f1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ccc071750 con 0x7f7ccc15c670 2024-09-16T01:43:44.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.639+0000 7f7cd46f1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ccc0b59a0 con 0x7f7ccc0b1990 2024-09-16T01:43:44.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.639+0000 7f7cd46f1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7ccc0d0c60 con 0x7f7ccc073080 2024-09-16T01:43:44.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.639+0000 7f7cd1c8c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7ccc15c670 0x7f7ccc073460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42390/0 (socket says 172.21.15.88:42390) 2024-09-16T01:43:44.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.639+0000 7f7cd1c8c640 1 -- 172.21.15.88:0/3489058712 learned_addr learned my addr 172.21.15.88:0/3489058712 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:44.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.639+0000 7f7cd148b640 1 -- 172.21.15.88:0/3489058712 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1968762154 0 0) 0x7f7ccc071750 con 0x7f7ccc15c670 2024-09-16T01:43:44.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.639+0000 7f7cd248d640 1 --2- 172.21.15.88:0/3489058712 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ccc073080 0x7f7ccc0b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:44.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.639+0000 7f7cd148b640 1 -- 172.21.15.88:0/3489058712 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7cb8003520 con 0x7f7ccc15c670 2024-09-16T01:43:44.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.639+0000 7f7cd148b640 1 -- 172.21.15.88:0/3489058712 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1823050208 0 0) 0x7f7cb8003520 con 0x7f7ccc15c670 2024-09-16T01:43:44.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.639+0000 7f7cd148b640 1 -- 172.21.15.88:0/3489058712 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ccc073080 msgr2=0x7f7ccc0b5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:44.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.639+0000 7f7cd148b640 1 --2- 172.21.15.88:0/3489058712 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ccc073080 0x7f7ccc0b5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.639+0000 7f7cd148b640 1 -- 172.21.15.88:0/3489058712 >> v1:172.21.15.154:6789/0 conn(0x7f7ccc0b1990 legacy=0x7f7ccc0d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:44.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.639+0000 7f7cd148b640 1 -- 172.21.15.88:0/3489058712 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ccc0d0370 con 0x7f7ccc15c670 2024-09-16T01:43:44.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.639+0000 7f7cd148b640 1 -- 172.21.15.88:0/3489058712 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f7cbc001670 con 0x7f7ccc15c670 2024-09-16T01:43:44.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.639+0000 7f7cd148b640 1 -- 172.21.15.88:0/3489058712 >> v1:172.21.15.88:6789/0 conn(0x7f7ccc15c670 legacy=0x7f7ccc073460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:44.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.639+0000 7f7cd148b640 1 --2- 172.21.15.88:0/3489058712 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb80039e0 0x7f7cb8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:44.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.639+0000 7f7cd148b640 1 --2- 172.21.15.88:0/3489058712 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cb8004800 0x7f7cb8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:44.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.640+0000 7f7cd148b640 1 --2- 172.21.15.88:0/3489058712 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb80051c0 0x7f7cb800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:44.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.640+0000 7f7cd148b640 1 -- 172.21.15.88:0/3489058712 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc0d0c60 con 0x7f7cb80051c0 2024-09-16T01:43:44.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.640+0000 7f7cd148b640 1 -- 172.21.15.88:0/3489058712 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb800c090 con 0x7f7cb8004800 2024-09-16T01:43:44.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.640+0000 7f7cd148b640 1 -- 172.21.15.88:0/3489058712 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7cb800c290 con 0x7f7cb80039e0 2024-09-16T01:43:44.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.640+0000 7f7cd248d640 1 --2- 172.21.15.88:0/3489058712 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ccc073080 0x7f7ccc0b5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:44.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.640+0000 7f7cd2c8e640 1 --2- 172.21.15.88:0/3489058712 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb80051c0 0x7f7cb800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:44.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.640+0000 7f7cd1c8c640 1 --2- 172.21.15.88:0/3489058712 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cb8004800 0x7f7cb8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:44.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.640+0000 7f7cd248d640 1 --2- 172.21.15.88:0/3489058712 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb80039e0 0x7f7cb8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:44.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.640+0000 7f7cd2c8e640 1 -- 172.21.15.88:0/3489058712 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb80039e0 msgr2=0x7f7cb8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:44.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.640+0000 7f7cd2c8e640 1 --2- 172.21.15.88:0/3489058712 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb80039e0 0x7f7cb8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.640+0000 7f7cd2c8e640 1 -- 172.21.15.88:0/3489058712 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cb8004800 msgr2=0x7f7cb8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:44.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.640+0000 7f7cd2c8e640 1 --2- 172.21.15.88:0/3489058712 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cb8004800 0x7f7cb8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.640+0000 7f7cd2c8e640 1 -- 172.21.15.88:0/3489058712 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cb8003520 con 0x7f7cb80051c0 2024-09-16T01:43:44.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.641+0000 7f7cd2c8e640 1 --2- 172.21.15.88:0/3489058712 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb80051c0 0x7f7cb800b9c0 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4001e50 tx=0x7f7cb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c959cb8d2725773 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:44.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.641+0000 7f7cd148b640 1 -- 172.21.15.88:0/3489058712 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4001a70 con 0x7f7cb80051c0 2024-09-16T01:43:44.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.641+0000 7f7cd148b640 1 -- 172.21.15.88:0/3489058712 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cc4003970 con 0x7f7cb80051c0 2024-09-16T01:43:44.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.641+0000 7f7cd148b640 1 -- 172.21.15.88:0/3489058712 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4003d60 con 0x7f7cb80051c0 2024-09-16T01:43:44.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.641+0000 7f7cd46f1640 1 -- 172.21.15.88:0/3489058712 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb80051c0 msgr2=0x7f7cb800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:44.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.641+0000 7f7cd46f1640 1 --2- 172.21.15.88:0/3489058712 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb80051c0 0x7f7cb800b9c0 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4001e50 tx=0x7f7cb8005830 comp rx=0 tx=0).stop 2024-09-16T01:43:44.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.641+0000 7f7cd2c8e640 1 -- 172.21.15.88:0/3489058712 reap_dead start 2024-09-16T01:43:44.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.641+0000 7f7cd46f1640 1 -- 172.21.15.88:0/3489058712 shutdown_connections 2024-09-16T01:43:44.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.641+0000 7f7cd46f1640 1 -- 172.21.15.88:0/3489058712 >> 172.21.15.88:0/3489058712 conn(0x7f7ccc0d24d0 msgr2=0x7f7ccc0d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:44.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.641+0000 7f7cd46f1640 1 -- 172.21.15.88:0/3489058712 shutdown_connections 2024-09-16T01:43:44.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.641+0000 7f7cd46f1640 1 -- 172.21.15.88:0/3489058712 wait complete. 2024-09-16T01:43:44.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.643+0000 7f7cd46f1640 1 Processor -- start 2024-09-16T01:43:44.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.643+0000 7f7cd46f1640 1 -- start start 2024-09-16T01:43:44.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.643+0000 7f7cd46f1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb80051c0 0x7f7ccc14dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:44.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.643+0000 7f7cd46f1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ccc15ca70 0x7f7ccc15cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:44.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.643+0000 7f7cd46f1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ccc06e370 0x7f7ccc06e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:44.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.643+0000 7f7cd46f1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc0d0610 con 0x7f7ccc06e370 2024-09-16T01:43:44.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.643+0000 7f7cd46f1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc071470 con 0x7f7ccc15ca70 2024-09-16T01:43:44.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.643+0000 7f7cd46f1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7ccc0d08f0 con 0x7f7cb80051c0 2024-09-16T01:43:44.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.643+0000 7f7cd2c8e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ccc06e370 0x7f7ccc06e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:44.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.643+0000 7f7cd2c8e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ccc06e370 0x7f7ccc06e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35712/0 (socket says 172.21.15.88:35712) 2024-09-16T01:43:44.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.643+0000 7f7cd2c8e640 1 -- 172.21.15.88:0/4111820085 learned_addr learned my addr 172.21.15.88:0/4111820085 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:44.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.644+0000 7f7cd1c8c640 1 --2- 172.21.15.88:0/4111820085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ccc15ca70 0x7f7ccc15cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:44.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.644+0000 7f7cd248d640 1 --2- 172.21.15.88:0/4111820085 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb80051c0 0x7f7ccc14dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:44.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.644+0000 7f7cd2c8e640 1 -- 172.21.15.88:0/4111820085 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb80051c0 msgr2=0x7f7ccc14dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:44.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.644+0000 7f7cd2c8e640 1 --2- 172.21.15.88:0/4111820085 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb80051c0 0x7f7ccc14dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.644+0000 7f7cd2c8e640 1 -- 172.21.15.88:0/4111820085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ccc15ca70 msgr2=0x7f7ccc15cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:44.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.644+0000 7f7cd2c8e640 1 --2- 172.21.15.88:0/4111820085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ccc15ca70 0x7f7ccc15cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.644+0000 7f7cd2c8e640 1 -- 172.21.15.88:0/4111820085 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ccc06d2d0 con 0x7f7ccc06e370 2024-09-16T01:43:44.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.644+0000 7f7cd1c8c640 1 --2- 172.21.15.88:0/4111820085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ccc15ca70 0x7f7ccc15cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:44.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.644+0000 7f7cd248d640 1 --2- 172.21.15.88:0/4111820085 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb80051c0 0x7f7ccc14dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:44.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.644+0000 7f7cd2c8e640 1 --2- 172.21.15.88:0/4111820085 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ccc06e370 0x7f7ccc06e7f0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f7cc402be90 tx=0x7f7cc40014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:44.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.644+0000 7f7cc2ffd640 1 -- 172.21.15.88:0/4111820085 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4041070 con 0x7f7ccc06e370 2024-09-16T01:43:44.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.644+0000 7f7cc2ffd640 1 -- 172.21.15.88:0/4111820085 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cc4006d60 con 0x7f7ccc06e370 2024-09-16T01:43:44.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.644+0000 7f7cd46f1640 1 -- 172.21.15.88:0/4111820085 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ccc06d5b0 con 0x7f7ccc06e370 2024-09-16T01:43:44.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.644+0000 7f7cc2ffd640 1 -- 172.21.15.88:0/4111820085 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4007db0 con 0x7f7ccc06e370 2024-09-16T01:43:44.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.644+0000 7f7cd46f1640 1 -- 172.21.15.88:0/4111820085 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ccc06db70 con 0x7f7ccc06e370 2024-09-16T01:43:44.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.645+0000 7f7cd46f1640 1 -- 172.21.15.88:0/4111820085 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ccc0d1f00 con 0x7f7ccc06e370 2024-09-16T01:43:44.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.646+0000 7f7cc2ffd640 1 -- 172.21.15.88:0/4111820085 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7cc4044070 con 0x7f7ccc06e370 2024-09-16T01:43:44.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.646+0000 7f7cc2ffd640 1 --2- 172.21.15.88:0/4111820085 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7cb006c2b0 0x7f7cb006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:44.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.646+0000 7f7cc2ffd640 1 -- 172.21.15.88:0/4111820085 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8576+0+0 (secure 0 0 0) 0x7f7cc40ab0f0 con 0x7f7ccc06e370 2024-09-16T01:43:44.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.646+0000 7f7cd248d640 1 --2- 172.21.15.88:0/4111820085 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7cb006c2b0 0x7f7cb006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:44.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.646+0000 7f7cd248d640 1 --2- 172.21.15.88:0/4111820085 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7cb006c2b0 0x7f7cb006e770 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8006e80 tx=0x7f7cc8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:44.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.650+0000 7f7cc2ffd640 1 -- 172.21.15.88:0/4111820085 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7cc40787a0 con 0x7f7ccc06e370 2024-09-16T01:43:44.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.851+0000 7f7cd46f1640 1 -- 172.21.15.88:0/4111820085 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ccc0d3050 con 0x7f7ccc06e370 2024-09-16T01:43:44.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.851+0000 7f7cc2ffd640 1 -- 172.21.15.88:0/4111820085 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 275 v275) ==== 95+0+5234 (secure 0 0 0) 0x7f7cc407caa0 con 0x7f7ccc06e370 2024-09-16T01:43:44.853 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:44.853 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":275,"btime":"2024-09-16T01:43:42:706187+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9696,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4270197461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4270197461},{"type":"v1","addr":"172.21.15.88:6837","nonce":4270197461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":275,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:43:42.694403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":120,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9852},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9852":{"gid":9852,"name":"a","rank":0,"incarnation":271,"state":"up:active","state_seq":5,"addr":"172.21.15.88:6836/4135629195","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4135629195},{"type":"v1","addr":"172.21.15.88:6836","nonce":4135629195}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9852,"qdb_cluster":[9852]},"id":6}]} 2024-09-16T01:43:44.853 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 275 2024-09-16T01:43:44.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.853+0000 7f7cd46f1640 1 -- 172.21.15.88:0/4111820085 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7cb006c2b0 msgr2=0x7f7cb006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:44.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.853+0000 7f7cd46f1640 1 --2- 172.21.15.88:0/4111820085 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7cb006c2b0 0x7f7cb006e770 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8006e80 tx=0x7f7cc8008040 comp rx=0 tx=0).stop 2024-09-16T01:43:44.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.853+0000 7f7cd46f1640 1 -- 172.21.15.88:0/4111820085 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ccc06e370 msgr2=0x7f7ccc06e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:44.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.853+0000 7f7cd46f1640 1 --2- 172.21.15.88:0/4111820085 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ccc06e370 0x7f7ccc06e7f0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f7cc402be90 tx=0x7f7cc40014a0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.853+0000 7f7cd46f1640 1 -- 172.21.15.88:0/4111820085 shutdown_connections 2024-09-16T01:43:44.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.853+0000 7f7cd46f1640 1 --2- 172.21.15.88:0/4111820085 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7cb006c2b0 0x7f7cb006e770 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.853+0000 7f7cd46f1640 1 --2- 172.21.15.88:0/4111820085 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7ccc06e370 0x7f7ccc06e7f0 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.853+0000 7f7cd46f1640 1 --2- 172.21.15.88:0/4111820085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ccc15ca70 0x7f7ccc15cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.853+0000 7f7cd46f1640 1 --2- 172.21.15.88:0/4111820085 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cb80051c0 0x7f7ccc14dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.853+0000 7f7cd46f1640 1 -- 172.21.15.88:0/4111820085 >> 172.21.15.88:0/4111820085 conn(0x7f7ccc0d24d0 msgr2=0x7f7ccc0ce970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:44.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.853+0000 7f7cd46f1640 1 -- 172.21.15.88:0/4111820085 shutdown_connections 2024-09-16T01:43:44.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.853+0000 7f7cd46f1640 1 -- 172.21.15.88:0/4111820085 wait complete. 2024-09-16T01:43:44.866 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage ls --format=json-pretty 2024-09-16T01:43:44.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.965+0000 7f63757be640 1 Processor -- start 2024-09-16T01:43:44.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.965+0000 7f63757be640 1 -- start start 2024-09-16T01:43:44.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.966+0000 7f63757be640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f637015b480 0x7f63700b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:44.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.966+0000 7f63757be640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6370072700 0x7f637015bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:44.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.966+0000 7f63757be640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63700ccff0 con 0x7f63700b60c0 2024-09-16T01:43:44.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.966+0000 7f63757be640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f63700b6500 con 0x7f6370072700 2024-09-16T01:43:44.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.966+0000 7f63757be640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f63700cd2d0 con 0x7f637015b480 2024-09-16T01:43:44.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.966+0000 7f636effd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f637015b480 0x7f63700b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:44.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.966+0000 7f636f7fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6370072700 0x7f637015bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:44.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.966+0000 7f636e7fc640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f63700b60c0 0x7f6370071ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:50310/0 (socket says 172.21.15.88:50310) 2024-09-16T01:43:44.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.966+0000 7f636effd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f637015b480 0x7f63700b5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:34254/0 (socket says 172.21.15.88:34254) 2024-09-16T01:43:44.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.966+0000 7f636e7fc640 1 -- 172.21.15.88:0/2048116275 learned_addr learned my addr 172.21.15.88:0/2048116275 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:44.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.967+0000 7f636effd640 1 -- 172.21.15.88:0/2048116275 >> v2:172.21.15.154:3300/0 conn(0x7f6370072700 msgr2=0x7f637015bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:44.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.967+0000 7f636effd640 1 --2- 172.21.15.88:0/2048116275 >> v2:172.21.15.154:3300/0 conn(0x7f6370072700 0x7f637015bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.967+0000 7f636effd640 1 -- 172.21.15.88:0/2048116275 >> v1:172.21.15.154:6789/0 conn(0x7f63700b60c0 legacy=0x7f6370071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:44.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.967+0000 7f636effd640 1 -- 172.21.15.88:0/2048116275 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63700cd5b0 con 0x7f637015b480 2024-09-16T01:43:44.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.967+0000 7f636f7fe640 1 --2- 172.21.15.88:0/2048116275 >> v2:172.21.15.154:3300/0 conn(0x7f6370072700 0x7f637015bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:43:44.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.968+0000 7f636effd640 1 --2- 172.21.15.88:0/2048116275 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f637015b480 0x7f63700b5b80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f636000bfa0 tx=0x7f636002c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f0d0c95e7372955 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:44.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.968+0000 7f636dffb640 1 -- 172.21.15.88:0/2048116275 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6360038070 con 0x7f637015b480 2024-09-16T01:43:44.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.968+0000 7f636dffb640 1 -- 172.21.15.88:0/2048116275 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6360009540 con 0x7f637015b480 2024-09-16T01:43:44.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.968+0000 7f636dffb640 1 -- 172.21.15.88:0/2048116275 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6360009910 con 0x7f637015b480 2024-09-16T01:43:44.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.968+0000 7f63757be640 1 -- 172.21.15.88:0/2048116275 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f637015b480 msgr2=0x7f63700b5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:44.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.968+0000 7f63757be640 1 --2- 172.21.15.88:0/2048116275 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f637015b480 0x7f63700b5b80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f636000bfa0 tx=0x7f636002c7e0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.968+0000 7f63757be640 1 -- 172.21.15.88:0/2048116275 shutdown_connections 2024-09-16T01:43:44.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.968+0000 7f63757be640 1 --2- 172.21.15.88:0/2048116275 >> v2:172.21.15.154:3300/0 conn(0x7f6370072700 0x7f637015bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.968+0000 7f63757be640 1 --2- 172.21.15.88:0/2048116275 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f637015b480 0x7f63700b5b80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.968+0000 7f63757be640 1 -- 172.21.15.88:0/2048116275 >> 172.21.15.88:0/2048116275 conn(0x7f637006d0f0 msgr2=0x7f637006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:44.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.968+0000 7f63757be640 1 -- 172.21.15.88:0/2048116275 shutdown_connections 2024-09-16T01:43:44.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.968+0000 7f63757be640 1 -- 172.21.15.88:0/2048116275 wait complete. 2024-09-16T01:43:44.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.970+0000 7f63757be640 1 Processor -- start 2024-09-16T01:43:44.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.970+0000 7f63757be640 1 -- start start 2024-09-16T01:43:44.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.970+0000 7f63757be640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6370072700 0x7f63701561c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:44.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.970+0000 7f63757be640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63700b60c0 0x7f637015d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:44.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.970+0000 7f63757be640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f637015b480 0x7f6370146c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:44.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.970+0000 7f63757be640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f637015b860 con 0x7f63700b60c0 2024-09-16T01:43:44.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.970+0000 7f63757be640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f63700cdb70 con 0x7f637015b480 2024-09-16T01:43:44.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.970+0000 7f63757be640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f637015b9e0 con 0x7f6370072700 2024-09-16T01:43:44.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.971+0000 7f636e7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63700b60c0 0x7f637015d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:44.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.971+0000 7f636effd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6370072700 0x7f63701561c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:44.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.971+0000 7f636e7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63700b60c0 0x7f637015d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35724/0 (socket says 172.21.15.88:35724) 2024-09-16T01:43:44.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.971+0000 7f636e7fc640 1 -- 172.21.15.88:0/1234865423 learned_addr learned my addr 172.21.15.88:0/1234865423 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:44.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.971+0000 7f636f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f637015b480 0x7f6370146c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:44.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.971+0000 7f636e7fc640 1 -- 172.21.15.88:0/1234865423 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6370072700 msgr2=0x7f63701561c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:44.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.971+0000 7f636e7fc640 1 --2- 172.21.15.88:0/1234865423 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6370072700 0x7f63701561c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.971+0000 7f636e7fc640 1 -- 172.21.15.88:0/1234865423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f637015b480 msgr2=0x7f6370146c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:44.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.971+0000 7f636e7fc640 1 --2- 172.21.15.88:0/1234865423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f637015b480 0x7f6370146c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.971+0000 7f636e7fc640 1 -- 172.21.15.88:0/1234865423 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f637015bfd0 con 0x7f63700b60c0 2024-09-16T01:43:44.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.971+0000 7f636e7fc640 1 --2- 172.21.15.88:0/1234865423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63700b60c0 0x7f637015d2b0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f636400c500 tx=0x7f636400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:44.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.971+0000 7f63577fe640 1 -- 172.21.15.88:0/1234865423 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6364017070 con 0x7f63700b60c0 2024-09-16T01:43:44.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.971+0000 7f63577fe640 1 -- 172.21.15.88:0/1234865423 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6364005ce0 con 0x7f63700b60c0 2024-09-16T01:43:44.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.971+0000 7f63757be640 1 -- 172.21.15.88:0/1234865423 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f637015c2b0 con 0x7f63700b60c0 2024-09-16T01:43:44.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.971+0000 7f63577fe640 1 -- 172.21.15.88:0/1234865423 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63640072d0 con 0x7f63700b60c0 2024-09-16T01:43:44.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.971+0000 7f63757be640 1 -- 172.21.15.88:0/1234865423 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f637015c870 con 0x7f63700b60c0 2024-09-16T01:43:44.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.973+0000 7f63577fe640 1 -- 172.21.15.88:0/1234865423 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f636401a070 con 0x7f63700b60c0 2024-09-16T01:43:44.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.973+0000 7f63577fe640 1 --2- 172.21.15.88:0/1234865423 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f635c06c2b0 0x7f635c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:44.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.973+0000 7f63577fe640 1 -- 172.21.15.88:0/1234865423 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8576+0+0 (secure 0 0 0) 0x7f636408b100 con 0x7f63700b60c0 2024-09-16T01:43:44.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.973+0000 7f636effd640 1 --2- 172.21.15.88:0/1234865423 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f635c06c2b0 0x7f635c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:44.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.974+0000 7f636effd640 1 --2- 172.21.15.88:0/1234865423 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f635c06c2b0 0x7f635c06e770 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f6360035000 tx=0x7f636000b000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:44.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.974+0000 7f6378630740 1 Processor -- start 2024-09-16T01:43:44.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.974+0000 7f6378630740 1 -- start start 2024-09-16T01:43:44.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.975+0000 7f6378630740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dffb060f00 0x55dffb0612e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:44.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.975+0000 7f6378630740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb05e6c0 0x55dffb060250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:44.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.975+0000 7f6378630740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb05eb20 0x55dffb040e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:44.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.975+0000 7f6378630740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55dffaf7d930 con 0x55dffb05e6c0 2024-09-16T01:43:44.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.975+0000 7f6378630740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dffaf25570 con 0x55dffb05eb20 2024-09-16T01:43:44.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.975+0000 7f6378630740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55dffaf798f0 con 0x55dffb060f00 2024-09-16T01:43:44.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.975+0000 7f636e7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb05e6c0 0x55dffb060250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:44.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.975+0000 7f636e7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb05e6c0 0x55dffb060250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35732/0 (socket says 172.21.15.88:35732) 2024-09-16T01:43:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.975+0000 7f636e7fc640 1 -- 172.21.15.88:0/262323266 learned_addr learned my addr 172.21.15.88:0/262323266 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.975+0000 7f636effd640 1 --2- 172.21.15.88:0/262323266 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dffb060f00 0x55dffb0612e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.975+0000 7f636f7fe640 1 --2- 172.21.15.88:0/262323266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb05eb20 0x55dffb040e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.976+0000 7f636e7fc640 1 -- 172.21.15.88:0/262323266 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dffb060f00 msgr2=0x55dffb0612e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.976+0000 7f636e7fc640 1 --2- 172.21.15.88:0/262323266 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dffb060f00 0x55dffb0612e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.976+0000 7f636e7fc640 1 -- 172.21.15.88:0/262323266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb05eb20 msgr2=0x55dffb040e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.976+0000 7f636e7fc640 1 --2- 172.21.15.88:0/262323266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb05eb20 0x55dffb040e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.976+0000 7f636e7fc640 1 -- 172.21.15.88:0/262323266 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dffaf3a200 con 0x55dffb05e6c0 2024-09-16T01:43:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.976+0000 7f636f7fe640 1 --2- 172.21.15.88:0/262323266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb05eb20 0x55dffb040e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.976+0000 7f636e7fc640 1 --2- 172.21.15.88:0/262323266 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb05e6c0 0x55dffb060250 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f6364054c30 tx=0x7f63640587d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=412635dc629c6f04 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.976+0000 7f6354ff9640 1 -- 172.21.15.88:0/262323266 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6364061300 con 0x55dffb05e6c0 2024-09-16T01:43:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.976+0000 7f6354ff9640 1 -- 172.21.15.88:0/262323266 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6364056ad0 con 0x55dffb05e6c0 2024-09-16T01:43:44.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.976+0000 7f6354ff9640 1 -- 172.21.15.88:0/262323266 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636406cbb0 con 0x55dffb05e6c0 2024-09-16T01:43:44.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.976+0000 7f6378630740 1 -- 172.21.15.88:0/262323266 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb05e6c0 msgr2=0x55dffb060250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:44.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.976+0000 7f6378630740 1 --2- 172.21.15.88:0/262323266 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb05e6c0 0x55dffb060250 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f6364054c30 tx=0x7f63640587d0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.977+0000 7f6378630740 1 -- 172.21.15.88:0/262323266 shutdown_connections 2024-09-16T01:43:44.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.977+0000 7f6378630740 1 --2- 172.21.15.88:0/262323266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb05eb20 0x55dffb040e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.977+0000 7f6378630740 1 --2- 172.21.15.88:0/262323266 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb05e6c0 0x55dffb060250 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.977+0000 7f6378630740 1 --2- 172.21.15.88:0/262323266 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dffb060f00 0x55dffb0612e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.977+0000 7f6378630740 1 -- 172.21.15.88:0/262323266 >> 172.21.15.88:0/262323266 conn(0x55dffb016df0 msgr2=0x55dffb05ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:44.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.977+0000 7f6378630740 1 -- 172.21.15.88:0/262323266 shutdown_connections 2024-09-16T01:43:44.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.977+0000 7f6378630740 1 -- 172.21.15.88:0/262323266 wait complete. 2024-09-16T01:43:44.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.977+0000 7f6378630740 1 -- start start 2024-09-16T01:43:44.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.977+0000 7f6378630740 1 Processor -- start 2024-09-16T01:43:44.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.978+0000 7f6378630740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dffb05eb20 0x55dffb059560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:44.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.978+0000 7f6378630740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb060f00 0x55dffb072c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:44.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.978+0000 7f6378630740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb059aa0 0x55dffb0731b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:44.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.978+0000 7f6378630740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55dffb05d620 con 0x55dffb059aa0 2024-09-16T01:43:44.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.978+0000 7f6378630740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dffaf066a0 con 0x55dffb060f00 2024-09-16T01:43:44.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.978+0000 7f6378630740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55dffb05d900 con 0x55dffb05eb20 2024-09-16T01:43:44.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.978+0000 7f6378630740 10 client.0 _ll_register_callbacks cb 0x55dffaf3e300 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:43:44.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.978+0000 7f636f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb059aa0 0x55dffb0731b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:44.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.978+0000 7f636f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb059aa0 0x55dffb0731b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35748/0 (socket says 172.21.15.88:35748) 2024-09-16T01:43:44.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.978+0000 7f636f7fe640 1 -- 172.21.15.88:0/4226586074 learned_addr learned my addr 172.21.15.88:0/4226586074 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:44.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.978+0000 7f636f7fe640 1 -- 172.21.15.88:0/4226586074 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dffb05eb20 msgr2=0x55dffb059560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:44.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.978+0000 7f636f7fe640 1 --2- 172.21.15.88:0/4226586074 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dffb05eb20 0x55dffb059560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.978+0000 7f636e7fc640 1 --2- 172.21.15.88:0/4226586074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb060f00 0x55dffb072c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.978+0000 7f636f7fe640 1 -- 172.21.15.88:0/4226586074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb060f00 msgr2=0x55dffb072c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.978+0000 7f636f7fe640 1 --2- 172.21.15.88:0/4226586074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb060f00 0x55dffb072c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.978+0000 7f636f7fe640 1 -- 172.21.15.88:0/4226586074 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dffad5c7c0 con 0x55dffb059aa0 2024-09-16T01:43:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.979+0000 7f636f7fe640 1 --2- 172.21.15.88:0/4226586074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb059aa0 0x55dffb0731b0 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f6368014670 tx=0x7f6368014b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=842ad5191f6fb757 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.979+0000 7f634dffb640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:43:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.979+0000 7f634dffb640 1 -- 172.21.15.88:0/4226586074 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6368014e40 con 0x55dffb059aa0 2024-09-16T01:43:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.979+0000 7f634dffb640 1 -- 172.21.15.88:0/4226586074 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8576+0+0 (secure 0 0 0) 0x7f6368012ac0 con 0x55dffb059aa0 2024-09-16T01:43:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.979+0000 7f6378630740 1 -- 172.21.15.88:0/4226586074 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6364056ad0 con 0x55dffb059aa0 2024-09-16T01:43:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.979+0000 7f634dffb640 1 -- 172.21.15.88:0/4226586074 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=275) ==== 24+0+0 (secure 0 0 0) 0x7f6368012cc0 con 0x55dffb059aa0 2024-09-16T01:43:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.979+0000 7f6378630740 10 client.9927 fetch_fsmap learned FSMap version 275 2024-09-16T01:43:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.979+0000 7f6378630740 1 -- 172.21.15.88:0/4226586074 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=275,monmap=2+}) -- 0x55dffadbcf50 con 0x55dffb059aa0 2024-09-16T01:43:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.979+0000 7f634dffb640 1 -- 172.21.15.88:0/4226586074 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6364056ad0 con 0x55dffb059aa0 2024-09-16T01:43:44.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.979+0000 7f634dffb640 1 -- 172.21.15.88:0/4226586074 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 275) ==== 3502+0+0 (secure 0 0 0) 0x7f6368016430 con 0x55dffb059aa0 2024-09-16T01:43:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.979+0000 7f6378630740 10 client.9927 fetch_fsmap finished waiting for FSMap version 275 2024-09-16T01:43:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.979+0000 7f6378630740 10 client.9927 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:43:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.979+0000 7f6378630740 20 client.9927 populate_metadata read hostname 'smithi088' 2024-09-16T01:43:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.979+0000 7f6378630740 1 --2- 172.21.15.88:0/4226586074 >> [v2:172.21.15.88:6834/4135629195,v1:172.21.15.88:6836/4135629195] conn(0x55dffb06ed40 0x55dffb031da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.979+0000 7f6378630740 4 client.9927 mds_command: new command op to 9852 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:43:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.979+0000 7f6378630740 1 -- 172.21.15.88:0/4226586074 --> [v2:172.21.15.88:6834/4135629195,v1:172.21.15.88:6836/4135629195] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dffaf8b060 con 0x55dffb06ed40 2024-09-16T01:43:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.979+0000 7f636effd640 1 --2- 172.21.15.88:0/4226586074 >> [v2:172.21.15.88:6834/4135629195,v1:172.21.15.88:6836/4135629195] conn(0x55dffb06ed40 0x55dffb031da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.980+0000 7f636effd640 1 --2- 172.21.15.88:0/4226586074 >> [v2:172.21.15.88:6834/4135629195,v1:172.21.15.88:6836/4135629195] conn(0x55dffb06ed40 0x55dffb031da0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bcc6dc785497c0 server_cookie=c9df37da141e1293 in_seq=0 out_seq=0 2024-09-16T01:43:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.980+0000 7f634dffb640 10 client.9927 ms_handle_connect on v2:172.21.15.88:6834/4135629195 2024-09-16T01:43:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.980+0000 7f634dffb640 1 -- 172.21.15.88:0/4226586074 <== mds.0 v2:172.21.15.88:6834/4135629195 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dffaf8b060 con 0x55dffb06ed40 2024-09-16T01:43:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.980+0000 7f634dffb640 10 client.9927 handle_command_reply: tid=0 2024-09-16T01:43:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.980+0000 7f6378630740 1 client.9927 shutdown 2024-09-16T01:43:44.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.980+0000 7f6378630740 1 -- 172.21.15.88:0/4226586074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb059aa0 msgr2=0x55dffb0731b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:44.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.980+0000 7f6378630740 1 --2- 172.21.15.88:0/4226586074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb059aa0 0x55dffb0731b0 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f6368014670 tx=0x7f6368014b30 comp rx=0 tx=0).stop 2024-09-16T01:43:44.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.980+0000 7f6378630740 1 -- 172.21.15.88:0/4226586074 shutdown_connections 2024-09-16T01:43:44.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.980+0000 7f6378630740 1 --2- 172.21.15.88:0/4226586074 >> [v2:172.21.15.88:6834/4135629195,v1:172.21.15.88:6836/4135629195] conn(0x55dffb06ed40 0x55dffb031da0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.980+0000 7f6378630740 1 --2- 172.21.15.88:0/4226586074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb059aa0 0x55dffb0731b0 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.980+0000 7f6378630740 1 --2- 172.21.15.88:0/4226586074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb060f00 0x55dffb072c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.980+0000 7f6378630740 1 --2- 172.21.15.88:0/4226586074 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dffb05eb20 0x55dffb059560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.980+0000 7f634dffb640 0 client.9927 ms_handle_reset on v2:172.21.15.88:6834/4135629195 2024-09-16T01:43:44.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.980+0000 7f6378630740 1 -- 172.21.15.88:0/4226586074 >> 172.21.15.88:0/4226586074 conn(0x55dffb05e6c0 msgr2=0x55dffb040c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:44.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.980+0000 7f6378630740 1 -- 172.21.15.88:0/4226586074 shutdown_connections 2024-09-16T01:43:44.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.980+0000 7f6378630740 1 -- 172.21.15.88:0/4226586074 wait complete. 2024-09-16T01:43:44.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.981+0000 7f6378630740 20 client.9927 trim_cache size 0 max 16384 2024-09-16T01:43:44.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.994+0000 7f6378630740 1 Processor -- start 2024-09-16T01:43:44.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.994+0000 7f6378630740 1 -- start start 2024-09-16T01:43:44.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.994+0000 7f6378630740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb059aa0 0x55dffb040040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:44.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.994+0000 7f6378630740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb05eb20 0x55dffb083fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:44.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.994+0000 7f6378630740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dffb060f00 0x55dffb0d8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:44.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.994+0000 7f6378630740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6370201910 con 0x55dffb059aa0 2024-09-16T01:43:44.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.994+0000 7f6378630740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6370200c70 con 0x55dffb05eb20 2024-09-16T01:43:44.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.994+0000 7f6378630740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f63702012c0 con 0x55dffb060f00 2024-09-16T01:43:44.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.994+0000 7f636effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb059aa0 0x55dffb040040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:44.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.994+0000 7f636effd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb059aa0 0x55dffb040040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35762/0 (socket says 172.21.15.88:35762) 2024-09-16T01:43:44.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.994+0000 7f636effd640 1 -- 172.21.15.88:0/4167349428 learned_addr learned my addr 172.21.15.88:0/4167349428 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:44.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.995+0000 7f636effd640 1 -- 172.21.15.88:0/4167349428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dffb060f00 msgr2=0x55dffb0d8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:44.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.995+0000 7f636effd640 1 --2- 172.21.15.88:0/4167349428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dffb060f00 0x55dffb0d8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.995+0000 7f636effd640 1 -- 172.21.15.88:0/4167349428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb05eb20 msgr2=0x55dffb083fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:44.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.995+0000 7f636effd640 1 --2- 172.21.15.88:0/4167349428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb05eb20 0x55dffb083fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.995+0000 7f636effd640 1 -- 172.21.15.88:0/4167349428 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dffac500b0 con 0x55dffb059aa0 2024-09-16T01:43:44.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.995+0000 7f636effd640 1 --2- 172.21.15.88:0/4167349428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb059aa0 0x55dffb040040 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f6360033a20 tx=0x7f6360030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b0b1b15e0147087 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:44.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.995+0000 7f634ffff640 1 -- 172.21.15.88:0/4167349428 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6360038040 con 0x55dffb059aa0 2024-09-16T01:43:44.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.995+0000 7f634ffff640 1 -- 172.21.15.88:0/4167349428 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f636003e900 con 0x55dffb059aa0 2024-09-16T01:43:44.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.995+0000 7f634ffff640 1 -- 172.21.15.88:0/4167349428 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636003eab0 con 0x55dffb059aa0 2024-09-16T01:43:44.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.995+0000 7f6378630740 1 -- 172.21.15.88:0/4167349428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb059aa0 msgr2=0x55dffb040040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:44.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.995+0000 7f6378630740 1 --2- 172.21.15.88:0/4167349428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb059aa0 0x55dffb040040 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f6360033a20 tx=0x7f6360030000 comp rx=0 tx=0).stop 2024-09-16T01:43:44.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.996+0000 7f6378630740 1 -- 172.21.15.88:0/4167349428 shutdown_connections 2024-09-16T01:43:44.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.996+0000 7f6378630740 1 --2- 172.21.15.88:0/4167349428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dffb060f00 0x55dffb0d8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:44.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.996+0000 7f6378630740 1 --2- 172.21.15.88:0/4167349428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb05eb20 0x55dffb083fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.996+0000 7f6378630740 1 --2- 172.21.15.88:0/4167349428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb059aa0 0x55dffb040040 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.996+0000 7f6378630740 1 -- 172.21.15.88:0/4167349428 >> 172.21.15.88:0/4167349428 conn(0x55dffb05e6c0 msgr2=0x55dffb055140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:45.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.996+0000 7f6378630740 1 -- 172.21.15.88:0/4167349428 shutdown_connections 2024-09-16T01:43:45.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.996+0000 7f6378630740 1 -- 172.21.15.88:0/4167349428 wait complete. 2024-09-16T01:43:45.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.996+0000 7f6378630740 1 -- start start 2024-09-16T01:43:45.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.996+0000 7f6378630740 1 Processor -- start 2024-09-16T01:43:45.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.997+0000 7f6378630740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dffb059aa0 0x55dffb058480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:45.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.997+0000 7f6378630740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb05eb20 0x55dffb0db370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:45.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.997+0000 7f6378630740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb060f00 0x55dffb0e1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:45.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.997+0000 7f6378630740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55dffb06ebc0 con 0x55dffb05eb20 2024-09-16T01:43:45.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.997+0000 7f6378630740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55dffb043fe0 con 0x55dffb060f00 2024-09-16T01:43:45.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.997+0000 7f6378630740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55dffb06f210 con 0x55dffb059aa0 2024-09-16T01:43:45.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.997+0000 7f6378630740 10 client.0 _ll_register_callbacks cb 0x7f63680094e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:43:45.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.997+0000 7f636e7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb05eb20 0x55dffb0db370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:45.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.997+0000 7f636e7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb05eb20 0x55dffb0db370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35772/0 (socket says 172.21.15.88:35772) 2024-09-16T01:43:45.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.997+0000 7f636e7fc640 1 -- 172.21.15.88:0/2525362520 learned_addr learned my addr 172.21.15.88:0/2525362520 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:45.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.997+0000 7f636effd640 1 --2- 172.21.15.88:0/2525362520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dffb059aa0 0x55dffb058480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:45.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.997+0000 7f636f7fe640 1 --2- 172.21.15.88:0/2525362520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb060f00 0x55dffb0e1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:45.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.997+0000 7f636e7fc640 1 -- 172.21.15.88:0/2525362520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dffb059aa0 msgr2=0x55dffb058480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:45.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.997+0000 7f636e7fc640 1 --2- 172.21.15.88:0/2525362520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dffb059aa0 0x55dffb058480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.997+0000 7f636e7fc640 1 -- 172.21.15.88:0/2525362520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb060f00 msgr2=0x55dffb0e1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:45.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.997+0000 7f636e7fc640 1 --2- 172.21.15.88:0/2525362520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb060f00 0x55dffb0e1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.997+0000 7f636e7fc640 1 -- 172.21.15.88:0/2525362520 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dffb0db8b0 con 0x55dffb05eb20 2024-09-16T01:43:45.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.997+0000 7f636f7fe640 1 --2- 172.21.15.88:0/2525362520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb060f00 0x55dffb0e1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:45.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.997+0000 7f636effd640 1 --2- 172.21.15.88:0/2525362520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dffb059aa0 0x55dffb058480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:45.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.997+0000 7f636e7fc640 1 --2- 172.21.15.88:0/2525362520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb05eb20 0x55dffb0db370 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f6364000f60 tx=0x7f6364088ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4930fa5bb846829f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:45.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.997+0000 7f634dffb640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:43:45.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.998+0000 7f634dffb640 1 -- 172.21.15.88:0/2525362520 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636406cdb0 con 0x55dffb05eb20 2024-09-16T01:43:45.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.998+0000 7f634dffb640 1 -- 172.21.15.88:0/2525362520 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 8576+0+0 (secure 0 0 0) 0x7f6364007470 con 0x55dffb05eb20 2024-09-16T01:43:45.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.998+0000 7f6378630740 1 -- 172.21.15.88:0/2525362520 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f636003e900 con 0x55dffb05eb20 2024-09-16T01:43:45.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.998+0000 7f634dffb640 1 -- 172.21.15.88:0/2525362520 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=275) ==== 24+0+0 (secure 0 0 0) 0x7f6364061300 con 0x55dffb05eb20 2024-09-16T01:43:45.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.998+0000 7f6378630740 10 client.9933 fetch_fsmap learned FSMap version 275 2024-09-16T01:43:45.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.998+0000 7f6378630740 1 -- 172.21.15.88:0/2525362520 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=275,monmap=2+}) -- 0x55dffb0deb50 con 0x55dffb05eb20 2024-09-16T01:43:45.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.998+0000 7f634dffb640 1 -- 172.21.15.88:0/2525362520 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f636003e900 con 0x55dffb05eb20 2024-09-16T01:43:45.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.998+0000 7f634dffb640 1 -- 172.21.15.88:0/2525362520 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 275) ==== 3502+0+0 (secure 0 0 0) 0x7f6364057c20 con 0x55dffb05eb20 2024-09-16T01:43:45.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.998+0000 7f6378630740 10 client.9933 fetch_fsmap finished waiting for FSMap version 275 2024-09-16T01:43:45.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.998+0000 7f6378630740 10 client.9933 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:43:45.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.998+0000 7f6378630740 20 client.9933 populate_metadata read hostname 'smithi088' 2024-09-16T01:43:45.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.998+0000 7f6378630740 1 --2- 172.21.15.88:0/2525362520 >> [v2:172.21.15.88:6834/4135629195,v1:172.21.15.88:6836/4135629195] conn(0x55dffb06d390 0x55dffb06d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:45.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.998+0000 7f6378630740 4 client.9933 mds_command: new command op to 9852 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-09-16T01:43:45.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.998+0000 7f6378630740 1 -- 172.21.15.88:0/2525362520 --> [v2:172.21.15.88:6834/4135629195,v1:172.21.15.88:6836/4135629195] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x55dffaf853b0 con 0x55dffb06d390 2024-09-16T01:43:45.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.998+0000 7f636effd640 1 --2- 172.21.15.88:0/2525362520 >> [v2:172.21.15.88:6834/4135629195,v1:172.21.15.88:6836/4135629195] conn(0x55dffb06d390 0x55dffb06d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:45.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.998+0000 7f634dffb640 10 client.9933 ms_handle_connect on v2:172.21.15.88:6834/4135629195 2024-09-16T01:43:45.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.998+0000 7f636effd640 1 --2- 172.21.15.88:0/2525362520 >> [v2:172.21.15.88:6834/4135629195,v1:172.21.15.88:6836/4135629195] conn(0x55dffb06d390 0x55dffb06d770 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf8657011b5f04e server_cookie=80b247a76e357ad3 in_seq=0 out_seq=0 2024-09-16T01:43:45.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.999+0000 7f634dffb640 1 -- 172.21.15.88:0/2525362520 <== mds.0 v2:172.21.15.88:6834/4135629195 1 ==== command_reply(tid 0: 0 ) ==== 8+0+192 (crc 0 0 0) 0x55dffaf853b0 con 0x55dffb06d390 2024-09-16T01:43:45.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.999+0000 7f634dffb640 10 client.9933 handle_command_reply: tid=0 2024-09-16T01:43:45.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.999+0000 7f6378630740 1 client.9933 shutdown 2024-09-16T01:43:45.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.999+0000 7f6378630740 1 -- 172.21.15.88:0/2525362520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb05eb20 msgr2=0x55dffb0db370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:45.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.999+0000 7f6378630740 1 --2- 172.21.15.88:0/2525362520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb05eb20 0x55dffb0db370 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f6364000f60 tx=0x7f6364088ab0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.999+0000 7f6378630740 1 -- 172.21.15.88:0/2525362520 shutdown_connections 2024-09-16T01:43:45.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.999+0000 7f6378630740 1 --2- 172.21.15.88:0/2525362520 >> [v2:172.21.15.88:6834/4135629195,v1:172.21.15.88:6836/4135629195] conn(0x55dffb06d390 0x55dffb06d770 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.999+0000 7f6378630740 1 --2- 172.21.15.88:0/2525362520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dffb060f00 0x55dffb0e1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.999+0000 7f634dffb640 0 client.9933 ms_handle_reset on v2:172.21.15.88:6834/4135629195 2024-09-16T01:43:45.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.999+0000 7f6378630740 1 --2- 172.21.15.88:0/2525362520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55dffb05eb20 0x55dffb0db370 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.999+0000 7f6378630740 1 --2- 172.21.15.88:0/2525362520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55dffb059aa0 0x55dffb058480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.999+0000 7f6378630740 1 -- 172.21.15.88:0/2525362520 >> 172.21.15.88:0/2525362520 conn(0x55dffb05e6c0 msgr2=0x55dffb079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:45.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.999+0000 7f6378630740 1 -- 172.21.15.88:0/2525362520 shutdown_connections 2024-09-16T01:43:45.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.999+0000 7f6378630740 1 -- 172.21.15.88:0/2525362520 wait complete. 2024-09-16T01:43:45.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.999+0000 7f6378630740 20 client.9933 trim_cache size 0 max 16384 2024-09-16T01:43:45.011 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:45.011 INFO:teuthology.orchestra.run.smithi088.stdout:[ 2024-09-16T01:43:45.011 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:43:45.011 INFO:teuthology.orchestra.run.smithi088.stdout: "damage_type": "dentry", 2024-09-16T01:43:45.011 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 36845970, 2024-09-16T01:43:45.011 INFO:teuthology.orchestra.run.smithi088.stdout: "ino": 1, 2024-09-16T01:43:45.011 INFO:teuthology.orchestra.run.smithi088.stdout: "frag": "*", 2024-09-16T01:43:45.011 INFO:teuthology.orchestra.run.smithi088.stdout: "dname": "subdir", 2024-09-16T01:43:45.011 INFO:teuthology.orchestra.run.smithi088.stdout: "snap_id": "head", 2024-09-16T01:43:45.011 INFO:teuthology.orchestra.run.smithi088.stdout: "path": "/subdir" 2024-09-16T01:43:45.012 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:43:45.012 INFO:teuthology.orchestra.run.smithi088.stdout:] 2024-09-16T01:43:45.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.999+0000 7f63757be640 1 -- 172.21.15.88:0/1234865423 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f635c06c2b0 msgr2=0x7f635c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:45.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.999+0000 7f63757be640 1 --2- 172.21.15.88:0/1234865423 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f635c06c2b0 0x7f635c06e770 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f6360035000 tx=0x7f636000b000 comp rx=0 tx=0).stop 2024-09-16T01:43:45.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.999+0000 7f63757be640 1 -- 172.21.15.88:0/1234865423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63700b60c0 msgr2=0x7f637015d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:45.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.999+0000 7f63757be640 1 --2- 172.21.15.88:0/1234865423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63700b60c0 0x7f637015d2b0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f636400c500 tx=0x7f636400c900 comp rx=0 tx=0).stop 2024-09-16T01:43:45.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.999+0000 7f63757be640 1 -- 172.21.15.88:0/1234865423 shutdown_connections 2024-09-16T01:43:45.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.999+0000 7f63757be640 1 --2- 172.21.15.88:0/1234865423 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f635c06c2b0 0x7f635c06e770 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.999+0000 7f63757be640 1 --2- 172.21.15.88:0/1234865423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f637015b480 0x7f6370146c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.999+0000 7f63757be640 1 --2- 172.21.15.88:0/1234865423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f63700b60c0 0x7f637015d2b0 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.999+0000 7f63757be640 1 --2- 172.21.15.88:0/1234865423 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6370072700 0x7f63701561c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:44.999+0000 7f63757be640 1 -- 172.21.15.88:0/1234865423 >> 172.21.15.88:0/1234865423 conn(0x7f637006d0f0 msgr2=0x7f63700ce2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:45.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.000+0000 7f63757be640 1 -- 172.21.15.88:0/1234865423 shutdown_connections 2024-09-16T01:43:45.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.000+0000 7f63757be640 1 -- 172.21.15.88:0/1234865423 wait complete. 2024-09-16T01:43:45.014 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap header on 10000000000.00000000' 2024-09-16T01:43:45.014 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:43:45.014 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:43:45.030 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:43:45.031 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:43:45.087 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:43:45.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.225+0000 7ff996c53640 1 Processor -- start 2024-09-16T01:43:45.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.225+0000 7ff996c53640 1 -- start start 2024-09-16T01:43:45.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.226+0000 7ff996c53640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff99006c700 0x7ff99015d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:45.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.226+0000 7ff996c53640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff99006cb60 con 0x7ff990160720 2024-09-16T01:43:45.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.226+0000 7ff996c53640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff99006d600 con 0x7ff990156d80 2024-09-16T01:43:45.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.226+0000 7ff996c53640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff99006d3b0 con 0x7ff99006c700 2024-09-16T01:43:45.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.226+0000 7ff9951f0640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff990160720 0x7ff99006f090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42404/0 (socket says 172.21.15.88:42404) 2024-09-16T01:43:45.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.226+0000 7ff9951f0640 1 -- 172.21.15.88:0/4068942846 learned_addr learned my addr 172.21.15.88:0/4068942846 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:45.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.226+0000 7ff987fff640 1 --2- 172.21.15.88:0/4068942846 >> v2:172.21.15.154:3300/0 conn(0x7ff99006c700 0x7ff99015d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:45.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.226+0000 7ff9877fe640 1 -- 172.21.15.88:0/4068942846 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2070982009 0 0) 0x7ff99006cb60 con 0x7ff990160720 2024-09-16T01:43:45.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.226+0000 7ff9877fe640 1 -- 172.21.15.88:0/4068942846 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff978003520 con 0x7ff990160720 2024-09-16T01:43:45.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.226+0000 7ff9877fe640 1 -- 172.21.15.88:0/4068942846 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1885217988 0 0) 0x7ff978003520 con 0x7ff990160720 2024-09-16T01:43:45.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.226+0000 7ff9877fe640 1 -- 172.21.15.88:0/4068942846 >> v2:172.21.15.154:3300/0 conn(0x7ff99006c700 msgr2=0x7ff99015d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:45.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.226+0000 7ff9877fe640 1 --2- 172.21.15.88:0/4068942846 >> v2:172.21.15.154:3300/0 conn(0x7ff99006c700 0x7ff99015d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.226+0000 7ff9877fe640 1 -- 172.21.15.88:0/4068942846 >> v1:172.21.15.154:6789/0 conn(0x7ff990156d80 legacy=0x7ff9900ce680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:45.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.226+0000 7ff9877fe640 1 -- 172.21.15.88:0/4068942846 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff99006d8e0 con 0x7ff990160720 2024-09-16T01:43:45.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.226+0000 7ff9877fe640 1 -- 172.21.15.88:0/4068942846 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7ff9880019f0 con 0x7ff990160720 2024-09-16T01:43:45.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.226+0000 7ff9877fe640 1 -- 172.21.15.88:0/4068942846 >> v1:172.21.15.88:6789/0 conn(0x7ff990160720 legacy=0x7ff99006f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:45.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.227+0000 7ff9877fe640 1 --2- 172.21.15.88:0/4068942846 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff9780039b0 0x7ff978003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:45.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.227+0000 7ff9877fe640 1 --2- 172.21.15.88:0/4068942846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9780043b0 0x7ff978004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:45.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.227+0000 7ff9877fe640 1 --2- 172.21.15.88:0/4068942846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff978005120 0x7ff97800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:45.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.227+0000 7ff9877fe640 1 -- 172.21.15.88:0/4068942846 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff99006d3b0 con 0x7ff9780043b0 2024-09-16T01:43:45.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.227+0000 7ff9877fe640 1 -- 172.21.15.88:0/4068942846 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff97800c060 con 0x7ff978005120 2024-09-16T01:43:45.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.227+0000 7ff9877fe640 1 -- 172.21.15.88:0/4068942846 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff97800c260 con 0x7ff9780039b0 2024-09-16T01:43:45.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.227+0000 7ff987fff640 1 --2- 172.21.15.88:0/4068942846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9780043b0 0x7ff978004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:45.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.227+0000 7ff9949ef640 1 --2- 172.21.15.88:0/4068942846 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff9780039b0 0x7ff978003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:45.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.227+0000 7ff9951f0640 1 --2- 172.21.15.88:0/4068942846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff978005120 0x7ff97800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:45.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.227+0000 7ff987fff640 1 -- 172.21.15.88:0/4068942846 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff9780039b0 msgr2=0x7ff978003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:45.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.227+0000 7ff987fff640 1 --2- 172.21.15.88:0/4068942846 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff9780039b0 0x7ff978003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.227+0000 7ff987fff640 1 -- 172.21.15.88:0/4068942846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff978005120 msgr2=0x7ff97800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:45.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.227+0000 7ff987fff640 1 --2- 172.21.15.88:0/4068942846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff978005120 0x7ff97800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.227+0000 7ff987fff640 1 -- 172.21.15.88:0/4068942846 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff978003520 con 0x7ff9780043b0 2024-09-16T01:43:45.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.227+0000 7ff9951f0640 1 --2- 172.21.15.88:0/4068942846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff978005120 0x7ff97800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:45.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.227+0000 7ff9951f0640 1 -- 172.21.15.88:0/4068942846 reap_dead start 2024-09-16T01:43:45.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.228+0000 7ff987fff640 1 --2- 172.21.15.88:0/4068942846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9780043b0 0x7ff978004880 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7ff980009ac0 tx=0x7ff978005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dceb028a1e3f0ea2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:45.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.228+0000 7ff9877fe640 1 -- 172.21.15.88:0/4068942846 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff980036070 con 0x7ff9780043b0 2024-09-16T01:43:45.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.228+0000 7ff9877fe640 1 -- 172.21.15.88:0/4068942846 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff980031400 con 0x7ff9780043b0 2024-09-16T01:43:45.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.228+0000 7ff9877fe640 1 -- 172.21.15.88:0/4068942846 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff980009690 con 0x7ff9780043b0 2024-09-16T01:43:45.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.228+0000 7ff996c53640 1 -- 172.21.15.88:0/4068942846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9780043b0 msgr2=0x7ff978004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:45.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.228+0000 7ff996c53640 1 --2- 172.21.15.88:0/4068942846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9780043b0 0x7ff978004880 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7ff980009ac0 tx=0x7ff978005830 comp rx=0 tx=0).stop 2024-09-16T01:43:45.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.228+0000 7ff996c53640 1 -- 172.21.15.88:0/4068942846 shutdown_connections 2024-09-16T01:43:45.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.228+0000 7ff996c53640 1 --2- 172.21.15.88:0/4068942846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9780043b0 0x7ff978004880 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.228+0000 7ff996c53640 1 -- 172.21.15.88:0/4068942846 >> 172.21.15.88:0/4068942846 conn(0x7ff99006bc10 msgr2=0x7ff99006bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:45.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.228+0000 7ff996c53640 1 -- 172.21.15.88:0/4068942846 shutdown_connections 2024-09-16T01:43:45.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.228+0000 7ff996c53640 1 -- 172.21.15.88:0/4068942846 wait complete. 2024-09-16T01:43:45.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.230+0000 7ff996c53640 1 Processor -- start 2024-09-16T01:43:45.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.230+0000 7ff996c53640 1 -- start start 2024-09-16T01:43:45.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.230+0000 7ff996c53640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9780043b0 0x7ff990158670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:45.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.230+0000 7ff996c53640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff99014f390 0x7ff99014ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:45.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.230+0000 7ff996c53640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9900ae890 0x7ff9900aed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:45.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.230+0000 7ff996c53640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff990160b70 con 0x7ff9780043b0 2024-09-16T01:43:45.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.230+0000 7ff996c53640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff99006dbc0 con 0x7ff9900ae890 2024-09-16T01:43:45.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.230+0000 7ff996c53640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff990160cf0 con 0x7ff99014f390 2024-09-16T01:43:45.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.231+0000 7ff9949ef640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9780043b0 0x7ff990158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.231+0000 7ff9949ef640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9780043b0 0x7ff990158670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35792/0 (socket says 172.21.15.88:35792) 2024-09-16T01:43:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.231+0000 7ff9949ef640 1 -- 172.21.15.88:0/3888933194 learned_addr learned my addr 172.21.15.88:0/3888933194 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.231+0000 7ff9951f0640 1 --2- 172.21.15.88:0/3888933194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9900ae890 0x7ff9900aed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.231+0000 7ff987fff640 1 --2- 172.21.15.88:0/3888933194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff99014f390 0x7ff99014ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.231+0000 7ff9949ef640 1 -- 172.21.15.88:0/3888933194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff99014f390 msgr2=0x7ff99014ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.231+0000 7ff9949ef640 1 --2- 172.21.15.88:0/3888933194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff99014f390 0x7ff99014ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.231+0000 7ff9949ef640 1 -- 172.21.15.88:0/3888933194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9900ae890 msgr2=0x7ff9900aed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.231+0000 7ff9949ef640 1 --2- 172.21.15.88:0/3888933194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9900ae890 0x7ff9900aed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.231+0000 7ff9949ef640 1 -- 172.21.15.88:0/3888933194 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff990158ce0 con 0x7ff9780043b0 2024-09-16T01:43:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.231+0000 7ff9951f0640 1 --2- 172.21.15.88:0/3888933194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9900ae890 0x7ff9900aed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.231+0000 7ff987fff640 1 --2- 172.21.15.88:0/3888933194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff99014f390 0x7ff99014ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.231+0000 7ff9949ef640 1 --2- 172.21.15.88:0/3888933194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9780043b0 0x7ff990158670 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7ff98c00f820 tx=0x7ff98c00fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.231+0000 7ff9857fa640 1 -- 172.21.15.88:0/3888933194 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff98c01a070 con 0x7ff9780043b0 2024-09-16T01:43:45.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.231+0000 7ff9857fa640 1 -- 172.21.15.88:0/3888933194 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff98c005320 con 0x7ff9780043b0 2024-09-16T01:43:45.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.231+0000 7ff9857fa640 1 -- 172.21.15.88:0/3888933194 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff98c007270 con 0x7ff9780043b0 2024-09-16T01:43:45.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.231+0000 7ff996c53640 1 -- 172.21.15.88:0/3888933194 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff990158fc0 con 0x7ff9780043b0 2024-09-16T01:43:45.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.231+0000 7ff996c53640 1 -- 172.21.15.88:0/3888933194 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff990159580 con 0x7ff9780043b0 2024-09-16T01:43:45.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.232+0000 7ff996c53640 1 -- 172.21.15.88:0/3888933194 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff99015d5f0 con 0x7ff9780043b0 2024-09-16T01:43:45.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.233+0000 7ff9857fa640 1 -- 172.21.15.88:0/3888933194 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff98c01d070 con 0x7ff9780043b0 2024-09-16T01:43:45.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.233+0000 7ff9857fa640 1 --2- 172.21.15.88:0/3888933194 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff97c06c2b0 0x7ff97c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:45.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.233+0000 7ff9857fa640 1 -- 172.21.15.88:0/3888933194 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8576+0+0 (secure 0 0 0) 0x7ff98c08b800 con 0x7ff9780043b0 2024-09-16T01:43:45.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.233+0000 7ff987fff640 1 --2- 172.21.15.88:0/3888933194 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff97c06c2b0 0x7ff97c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:45.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.233+0000 7ff987fff640 1 --2- 172.21.15.88:0/3888933194 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff97c06c2b0 0x7ff97c06e770 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ff980033000 tx=0x7ff980008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:45.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.237+0000 7ff9857fa640 1 -- 172.21.15.88:0/3888933194 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff98c059480 con 0x7ff9780043b0 2024-09-16T01:43:45.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.414+0000 7ff996c53640 1 -- 172.21.15.88:0/3888933194 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff9900af250 con 0x7ff9780043b0 2024-09-16T01:43:45.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.414+0000 7ff9857fa640 1 -- 172.21.15.88:0/3888933194 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 66 entries v120) ==== 81+0+4014 (secure 0 0 0) 0x7ff98c058d80 con 0x7ff9780043b0 2024-09-16T01:43:45.416 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/3121519215 2024-09-17T01:43:30.669208+0000 2024-09-16T01:43:45.416 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/3121519215 2024-09-17T01:43:30.669208+0000 2024-09-16T01:43:45.416 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2555776364 2024-09-17T01:43:25.635976+0000 2024-09-16T01:43:45.416 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/1372908065 2024-09-17T01:43:12.443265+0000 2024-09-16T01:43:45.417 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/1372908065 2024-09-17T01:43:12.443265+0000 2024-09-16T01:43:45.417 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/650399848 2024-09-17T01:43:08.279879+0000 2024-09-16T01:43:45.417 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/1255135754 2024-09-17T01:41:41.905258+0000 2024-09-16T01:43:45.417 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/3728363359 2024-09-17T01:42:05.412133+0000 2024-09-16T01:43:45.417 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/293764774 2024-09-17T01:41:16.203758+0000 2024-09-16T01:43:45.417 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/1255135754 2024-09-17T01:41:41.905258+0000 2024-09-16T01:43:45.417 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2918194823 2024-09-17T01:41:29.275696+0000 2024-09-16T01:43:45.417 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/3336935653 2024-09-17T01:41:44.295023+0000 2024-09-16T01:43:45.417 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/981209627 2024-09-17T01:41:51.268495+0000 2024-09-16T01:43:45.417 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/293764774 2024-09-17T01:41:16.203758+0000 2024-09-16T01:43:45.418 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2680708900 2024-09-17T01:42:41.629904+0000 2024-09-16T01:43:45.418 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2957965650 2024-09-17T01:41:12.181485+0000 2024-09-16T01:43:45.418 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/3336935653 2024-09-17T01:41:44.295023+0000 2024-09-16T01:43:45.418 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2285699282 2024-09-17T01:42:29.541113+0000 2024-09-16T01:43:45.418 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/1162318664 2024-09-17T01:41:23.257870+0000 2024-09-16T01:43:45.418 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/128058689 2024-09-17T01:41:10.163743+0000 2024-09-16T01:43:45.418 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/981209627 2024-09-17T01:41:51.268495+0000 2024-09-16T01:43:45.418 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/128058689 2024-09-17T01:41:10.163743+0000 2024-09-16T01:43:45.418 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/1042715824 2024-09-17T01:40:59.080515+0000 2024-09-16T01:43:45.418 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2957965650 2024-09-17T01:41:12.181485+0000 2024-09-16T01:43:45.419 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2970185561 2024-09-17T01:40:38.255774+0000 2024-09-16T01:43:45.419 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/1162318664 2024-09-17T01:41:23.257870+0000 2024-09-16T01:43:45.419 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/1289932173 2024-09-17T01:40:42.399399+0000 2024-09-16T01:43:45.419 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/3356432718 2024-09-17T01:40:32.344252+0000 2024-09-16T01:43:45.419 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2555776364 2024-09-17T01:43:25.635976+0000 2024-09-16T01:43:45.419 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2970185561 2024-09-17T01:40:38.255774+0000 2024-09-16T01:43:45.419 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2733458327 2024-09-17T01:40:27.314712+0000 2024-09-16T01:43:45.419 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/3356432718 2024-09-17T01:40:32.344252+0000 2024-09-16T01:43:45.419 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2918194823 2024-09-17T01:41:29.275696+0000 2024-09-16T01:43:45.419 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/4144876777 2024-09-17T01:43:00.350808+0000 2024-09-16T01:43:45.419 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2408579939 2024-09-17T01:40:46.434492+0000 2024-09-16T01:43:45.419 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/3163343083 2024-09-17T01:42:01.375396+0000 2024-09-16T01:43:45.420 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/4222250252 2024-09-17T01:42:45.673956+0000 2024-09-16T01:43:45.420 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/4162274767 2024-09-17T01:40:11.522091+0000 2024-09-16T01:43:45.420 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2477051134 2024-09-17T01:43:04.380783+0000 2024-09-16T01:43:45.420 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/1289932173 2024-09-17T01:40:42.399399+0000 2024-09-16T01:43:45.420 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2733458327 2024-09-17T01:40:27.314712+0000 2024-09-16T01:43:45.420 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2550498835 2024-09-17T01:41:57.358287+0000 2024-09-16T01:43:45.420 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2550498835 2024-09-17T01:41:57.358287+0000 2024-09-16T01:43:45.420 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2285699282 2024-09-17T01:42:29.541113+0000 2024-09-16T01:43:45.420 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/3163343083 2024-09-17T01:42:01.375396+0000 2024-09-16T01:43:45.420 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/3728363359 2024-09-17T01:42:05.412133+0000 2024-09-16T01:43:45.421 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/880314974 2024-09-17T01:42:21.485748+0000 2024-09-16T01:43:45.421 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/1936493048 2024-09-17T01:42:19.133197+0000 2024-09-16T01:43:45.421 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/1936493048 2024-09-17T01:42:19.133197+0000 2024-09-16T01:43:45.421 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/880314974 2024-09-17T01:42:21.485748+0000 2024-09-16T01:43:45.421 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/3374372758 2024-09-17T01:42:25.506299+0000 2024-09-16T01:43:45.421 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/3374372758 2024-09-17T01:42:25.506299+0000 2024-09-16T01:43:45.421 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/201188184 2024-09-17T01:42:33.276104+0000 2024-09-16T01:43:45.421 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/1990650099 2024-09-17T01:42:37.599896+0000 2024-09-16T01:43:45.421 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/201188184 2024-09-17T01:42:33.276104+0000 2024-09-16T01:43:45.421 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/1990650099 2024-09-17T01:42:37.599896+0000 2024-09-16T01:43:45.421 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/4162274767 2024-09-17T01:40:11.522091+0000 2024-09-16T01:43:45.421 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2408579939 2024-09-17T01:40:46.434492+0000 2024-09-16T01:43:45.422 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2680708900 2024-09-17T01:42:41.629904+0000 2024-09-16T01:43:45.422 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/4222250252 2024-09-17T01:42:45.673956+0000 2024-09-16T01:43:45.422 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/1042715824 2024-09-17T01:40:59.080515+0000 2024-09-16T01:43:45.422 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/4065900771 2024-09-17T01:42:58.329389+0000 2024-09-16T01:43:45.422 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/4065900771 2024-09-17T01:42:58.329389+0000 2024-09-16T01:43:45.422 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/4144876777 2024-09-17T01:43:00.350808+0000 2024-09-16T01:43:45.422 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2477051134 2024-09-17T01:43:04.380783+0000 2024-09-16T01:43:45.422 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/650399848 2024-09-17T01:43:08.279879+0000 2024-09-16T01:43:45.423 INFO:teuthology.orchestra.run.smithi088.stderr:listed 66 entries 2024-09-16T01:43:45.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.417+0000 7ff996c53640 1 -- 172.21.15.88:0/3888933194 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff97c06c2b0 msgr2=0x7ff97c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:45.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.417+0000 7ff996c53640 1 --2- 172.21.15.88:0/3888933194 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff97c06c2b0 0x7ff97c06e770 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ff980033000 tx=0x7ff980008030 comp rx=0 tx=0).stop 2024-09-16T01:43:45.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.417+0000 7ff996c53640 1 -- 172.21.15.88:0/3888933194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9780043b0 msgr2=0x7ff990158670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:45.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.417+0000 7ff996c53640 1 --2- 172.21.15.88:0/3888933194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9780043b0 0x7ff990158670 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7ff98c00f820 tx=0x7ff98c00fce0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.417+0000 7ff996c53640 1 -- 172.21.15.88:0/3888933194 shutdown_connections 2024-09-16T01:43:45.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.417+0000 7ff996c53640 1 --2- 172.21.15.88:0/3888933194 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff97c06c2b0 0x7ff97c06e770 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.417+0000 7ff996c53640 1 --2- 172.21.15.88:0/3888933194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9900ae890 0x7ff9900aed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.417+0000 7ff996c53640 1 --2- 172.21.15.88:0/3888933194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff99014f390 0x7ff99014ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.417+0000 7ff996c53640 1 --2- 172.21.15.88:0/3888933194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9780043b0 0x7ff990158670 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.417+0000 7ff996c53640 1 -- 172.21.15.88:0/3888933194 >> 172.21.15.88:0/3888933194 conn(0x7ff99006bc10 msgr2=0x7ff990155f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:45.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.417+0000 7ff996c53640 1 -- 172.21.15.88:0/3888933194 shutdown_connections 2024-09-16T01:43:45.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.417+0000 7ff996c53640 1 -- 172.21.15.88:0/3888933194 wait complete. 2024-09-16T01:43:45.429 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-16T01:43:45.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:45.429 DEBUG:teuthology.orchestra.run.smithi088:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-09-16T01:43:45.455 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:43:45.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:45.455 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:43:45.511 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:43:45.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.650+0000 7f3bb2e47640 1 Processor -- start 2024-09-16T01:43:45.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.650+0000 7f3bb2e47640 1 -- start start 2024-09-16T01:43:45.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.651+0000 7f3bb2e47640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3bac072700 0x7f3bac166120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:45.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.651+0000 7f3bb2e47640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bac0ccff0 con 0x7f3bac15b480 2024-09-16T01:43:45.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.651+0000 7f3bb2e47640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bac0cd2d0 con 0x7f3bac0b6290 2024-09-16T01:43:45.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.651+0000 7f3bb2e47640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f3bac0cd5b0 con 0x7f3bac072700 2024-09-16T01:43:45.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.651+0000 7f3bb13e4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3bac072700 0x7f3bac166120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:45.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.651+0000 7f3bb13e4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3bac072700 0x7f3bac166120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35806/0 (socket says 172.21.15.88:35806) 2024-09-16T01:43:45.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.651+0000 7f3bb13e4640 1 -- 172.21.15.88:0/2980730227 learned_addr learned my addr 172.21.15.88:0/2980730227 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:45.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.651+0000 7f3bb0be3640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3bac15b480 0x7f3bac0b5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42410/0 (socket says 172.21.15.88:42410) 2024-09-16T01:43:45.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.651+0000 7f3bb13e4640 1 -- 172.21.15.88:0/2980730227 >> v1:172.21.15.154:6789/0 conn(0x7f3bac0b6290 legacy=0x7f3bac071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:45.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.651+0000 7f3bb13e4640 1 -- 172.21.15.88:0/2980730227 >> v1:172.21.15.88:6789/0 conn(0x7f3bac15b480 legacy=0x7f3bac0b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:45.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.651+0000 7f3bb13e4640 1 -- 172.21.15.88:0/2980730227 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bac0cd890 con 0x7f3bac072700 2024-09-16T01:43:45.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.651+0000 7f3bab7fe640 1 -- 172.21.15.88:0/2980730227 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1098740990 0 0) 0x7f3bac0ccff0 con 0x7f3bac15b480 2024-09-16T01:43:45.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.652+0000 7f3bb13e4640 1 --2- 172.21.15.88:0/2980730227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bac072700 0x7f3bac166120 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4009e50 tx=0x7f3ba402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbffcfdb5f52c9b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:45.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.652+0000 7f3bab7fe640 1 -- 172.21.15.88:0/2980730227 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4036070 con 0x7f3bac072700 2024-09-16T01:43:45.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.652+0000 7f3bab7fe640 1 -- 172.21.15.88:0/2980730227 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ba4031600 con 0x7f3bac072700 2024-09-16T01:43:45.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.652+0000 7f3bab7fe640 1 -- 172.21.15.88:0/2980730227 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba40319d0 con 0x7f3bac072700 2024-09-16T01:43:45.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.652+0000 7f3bb2e47640 1 -- 172.21.15.88:0/2980730227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bac072700 msgr2=0x7f3bac166120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:45.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.652+0000 7f3bb2e47640 1 --2- 172.21.15.88:0/2980730227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bac072700 0x7f3bac166120 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4009e50 tx=0x7f3ba402a730 comp rx=0 tx=0).stop 2024-09-16T01:43:45.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.653+0000 7f3bb2e47640 1 -- 172.21.15.88:0/2980730227 shutdown_connections 2024-09-16T01:43:45.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.653+0000 7f3bb2e47640 1 --2- 172.21.15.88:0/2980730227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bac072700 0x7f3bac166120 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.653+0000 7f3bb2e47640 1 -- 172.21.15.88:0/2980730227 >> 172.21.15.88:0/2980730227 conn(0x7f3bac06d0f0 msgr2=0x7f3bac06d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:45.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.653+0000 7f3bb2e47640 1 -- 172.21.15.88:0/2980730227 shutdown_connections 2024-09-16T01:43:45.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.653+0000 7f3bb2e47640 1 -- 172.21.15.88:0/2980730227 wait complete. 2024-09-16T01:43:45.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.654+0000 7f3bb2e47640 1 Processor -- start 2024-09-16T01:43:45.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.654+0000 7f3bb2e47640 1 -- start start 2024-09-16T01:43:45.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.654+0000 7f3bb2e47640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bac072700 0x7f3bac14afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:45.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.655+0000 7f3bb2e47640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bac15b480 0x7f3bac0c7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:45.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.655+0000 7f3bb2e47640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bac153410 0x7f3bac0b3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:45.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.655+0000 7f3bb2e47640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3bac15b9e0 con 0x7f3bac153410 2024-09-16T01:43:45.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.655+0000 7f3bb2e47640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3bac15b860 con 0x7f3bac15b480 2024-09-16T01:43:45.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.655+0000 7f3bb2e47640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3bac15cc70 con 0x7f3bac072700 2024-09-16T01:43:45.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.655+0000 7f3bb13e4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bac153410 0x7f3bac0b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:45.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.655+0000 7f3bb13e4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bac153410 0x7f3bac0b3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35822/0 (socket says 172.21.15.88:35822) 2024-09-16T01:43:45.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.655+0000 7f3bb13e4640 1 -- 172.21.15.88:0/32593654 learned_addr learned my addr 172.21.15.88:0/32593654 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:45.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.655+0000 7f3bb0be3640 1 --2- 172.21.15.88:0/32593654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bac072700 0x7f3bac14afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:45.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.655+0000 7f3babfff640 1 --2- 172.21.15.88:0/32593654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bac15b480 0x7f3bac0c7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:45.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.655+0000 7f3bb13e4640 1 -- 172.21.15.88:0/32593654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bac072700 msgr2=0x7f3bac14afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:45.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.655+0000 7f3bb13e4640 1 --2- 172.21.15.88:0/32593654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bac072700 0x7f3bac14afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.655+0000 7f3bb13e4640 1 -- 172.21.15.88:0/32593654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bac15b480 msgr2=0x7f3bac0c7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:45.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.655+0000 7f3bb13e4640 1 --2- 172.21.15.88:0/32593654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bac15b480 0x7f3bac0c7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.655+0000 7f3bb13e4640 1 -- 172.21.15.88:0/32593654 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bac0c8480 con 0x7f3bac153410 2024-09-16T01:43:45.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.655+0000 7f3bb0be3640 1 --2- 172.21.15.88:0/32593654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bac072700 0x7f3bac14afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:45.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.655+0000 7f3babfff640 1 --2- 172.21.15.88:0/32593654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bac15b480 0x7f3bac0c7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:45.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.655+0000 7f3bb13e4640 1 --2- 172.21.15.88:0/32593654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bac153410 0x7f3bac0b3700 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4009e50 tx=0x7f3ba4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:45.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.655+0000 7f3ba97fa640 1 -- 172.21.15.88:0/32593654 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4036050 con 0x7f3bac153410 2024-09-16T01:43:45.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.655+0000 7f3ba97fa640 1 -- 172.21.15.88:0/32593654 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ba4042070 con 0x7f3bac153410 2024-09-16T01:43:45.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.655+0000 7f3ba97fa640 1 -- 172.21.15.88:0/32593654 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4009510 con 0x7f3bac153410 2024-09-16T01:43:45.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.655+0000 7f3bb2e47640 1 -- 172.21.15.88:0/32593654 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bac0c6ea0 con 0x7f3bac153410 2024-09-16T01:43:45.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.655+0000 7f3bb2e47640 1 -- 172.21.15.88:0/32593654 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bac0c7460 con 0x7f3bac153410 2024-09-16T01:43:45.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.656+0000 7f3bb2e47640 1 -- 172.21.15.88:0/32593654 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bac071ff0 con 0x7f3bac153410 2024-09-16T01:43:45.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.657+0000 7f3ba97fa640 1 -- 172.21.15.88:0/32593654 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3ba4007bf0 con 0x7f3bac153410 2024-09-16T01:43:45.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.657+0000 7f3ba97fa640 1 --2- 172.21.15.88:0/32593654 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3b9006c2b0 0x7f3b9006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:45.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.657+0000 7f3ba97fa640 1 -- 172.21.15.88:0/32593654 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 8576+0+0 (secure 0 0 0) 0x7f3ba40acf20 con 0x7f3bac153410 2024-09-16T01:43:45.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.657+0000 7f3bb0be3640 1 --2- 172.21.15.88:0/32593654 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3b9006c2b0 0x7f3b9006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:45.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.658+0000 7f3bb0be3640 1 --2- 172.21.15.88:0/32593654 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3b9006c2b0 0x7f3b9006e770 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0007fd0 tx=0x7f3ba0007dc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:45.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.661+0000 7f3ba97fa640 1 -- 172.21.15.88:0/32593654 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3ba407ab10 con 0x7f3bac153410 2024-09-16T01:43:45.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.865+0000 7f3bb2e47640 1 -- 172.21.15.88:0/32593654 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f3bac0b28d0 con 0x7f3bac153410 2024-09-16T01:43:45.866 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:43:45.865+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:43:45.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.867+0000 7f3ba97fa640 1 -- 172.21.15.88:0/32593654 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v276) ==== 161+0+0 (secure 0 0 0) 0x7f3ba407a410 con 0x7f3bac153410 2024-09-16T01:43:45.868 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:43:45.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.869+0000 7f3bb2e47640 1 -- 172.21.15.88:0/32593654 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3b9006c2b0 msgr2=0x7f3b9006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:45.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.869+0000 7f3bb2e47640 1 --2- 172.21.15.88:0/32593654 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3b9006c2b0 0x7f3b9006e770 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0007fd0 tx=0x7f3ba0007dc0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.869+0000 7f3bb2e47640 1 -- 172.21.15.88:0/32593654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bac153410 msgr2=0x7f3bac0b3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:45.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.869+0000 7f3bb2e47640 1 --2- 172.21.15.88:0/32593654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bac153410 0x7f3bac0b3700 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4009e50 tx=0x7f3ba4004a40 comp rx=0 tx=0).stop 2024-09-16T01:43:45.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.870+0000 7f3bb2e47640 1 -- 172.21.15.88:0/32593654 shutdown_connections 2024-09-16T01:43:45.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.870+0000 7f3bb2e47640 1 --2- 172.21.15.88:0/32593654 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3b9006c2b0 0x7f3b9006e770 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.870+0000 7f3bb2e47640 1 --2- 172.21.15.88:0/32593654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bac153410 0x7f3bac0b3700 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.870+0000 7f3bb2e47640 1 --2- 172.21.15.88:0/32593654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bac15b480 0x7f3bac0c7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.870+0000 7f3bb2e47640 1 --2- 172.21.15.88:0/32593654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bac072700 0x7f3bac14afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.870+0000 7f3bb2e47640 1 -- 172.21.15.88:0/32593654 >> 172.21.15.88:0/32593654 conn(0x7f3bac06d0f0 msgr2=0x7f3bac159330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:45.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.870+0000 7f3bb2e47640 1 -- 172.21.15.88:0/32593654 shutdown_connections 2024-09-16T01:43:45.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.870+0000 7f3bb2e47640 1 -- 172.21.15.88:0/32593654 wait complete. 2024-09-16T01:43:45.872 INFO:tasks.ceph.mds.a.smithi088.stderr: -562> 2024-09-16T01:43:40.686+0000 7fe826162640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:43:45.883 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:43:45.898 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:43:45.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.983+0000 7fafb1465640 1 Processor -- start 2024-09-16T01:43:45.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.983+0000 7fafb1465640 1 -- start start 2024-09-16T01:43:45.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.984+0000 7fafb1465640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fafac0cf1f0 0x7fafac0b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:45.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.984+0000 7fafb1465640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fafac0b6720 0x7fafac0d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:45.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.984+0000 7fafb1465640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafac0cf860 con 0x7fafac15e7a0 2024-09-16T01:43:45.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.984+0000 7fafb1465640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fafac0b6b90 con 0x7fafac0b6720 2024-09-16T01:43:45.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.984+0000 7fafb1465640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fafac0d2ca0 con 0x7fafac0cf1f0 2024-09-16T01:43:45.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.984+0000 7fafab7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fafac0b6720 0x7fafac0d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:45.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.984+0000 7fafab7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fafac0b6720 0x7fafac0d3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35834/0 (socket says 172.21.15.88:35834) 2024-09-16T01:43:45.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.984+0000 7fafab7fe640 1 -- 172.21.15.88:0/734610247 learned_addr learned my addr 172.21.15.88:0/734610247 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:45.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.984+0000 7fafaa7fc640 1 --2- 172.21.15.88:0/734610247 >> v2:172.21.15.154:3300/0 conn(0x7fafac0cf1f0 0x7fafac0b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:45.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.984+0000 7fafab7fe640 1 -- 172.21.15.88:0/734610247 >> v2:172.21.15.154:3300/0 conn(0x7fafac0cf1f0 msgr2=0x7fafac0b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:45.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.984+0000 7fafab7fe640 1 --2- 172.21.15.88:0/734610247 >> v2:172.21.15.154:3300/0 conn(0x7fafac0cf1f0 0x7fafac0b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.984+0000 7fafab7fe640 1 -- 172.21.15.88:0/734610247 >> v1:172.21.15.154:6789/0 conn(0x7fafac15e7a0 legacy=0x7fafac06caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:45.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.984+0000 7fafab7fe640 1 -- 172.21.15.88:0/734610247 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafac0cfb40 con 0x7fafac0b6720 2024-09-16T01:43:45.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.985+0000 7fafab7fe640 1 --2- 172.21.15.88:0/734610247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafac0b6720 0x7fafac0d3410 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7fafac0d2110 tx=0x7fafa402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=865b117ff049e45c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:45.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.985+0000 7fafa9ffb640 1 -- 172.21.15.88:0/734610247 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa4036070 con 0x7fafac0b6720 2024-09-16T01:43:45.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.985+0000 7fafa9ffb640 1 -- 172.21.15.88:0/734610247 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafa4031600 con 0x7fafac0b6720 2024-09-16T01:43:45.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.985+0000 7fafa9ffb640 1 -- 172.21.15.88:0/734610247 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa40319d0 con 0x7fafac0b6720 2024-09-16T01:43:45.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.985+0000 7fafb1465640 1 -- 172.21.15.88:0/734610247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafac0b6720 msgr2=0x7fafac0d3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:45.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.985+0000 7fafb1465640 1 --2- 172.21.15.88:0/734610247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafac0b6720 0x7fafac0d3410 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7fafac0d2110 tx=0x7fafa402a660 comp rx=0 tx=0).stop 2024-09-16T01:43:45.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.985+0000 7fafb1465640 1 -- 172.21.15.88:0/734610247 shutdown_connections 2024-09-16T01:43:45.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.986+0000 7fafb1465640 1 --2- 172.21.15.88:0/734610247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafac0b6720 0x7fafac0d3410 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.986+0000 7fafb1465640 1 --2- 172.21.15.88:0/734610247 >> v2:172.21.15.154:3300/0 conn(0x7fafac0cf1f0 0x7fafac0b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.986+0000 7fafb1465640 1 -- 172.21.15.88:0/734610247 >> 172.21.15.88:0/734610247 conn(0x7fafac0d11e0 msgr2=0x7fafac0d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:45.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.986+0000 7fafb1465640 1 -- 172.21.15.88:0/734610247 shutdown_connections 2024-09-16T01:43:45.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.986+0000 7fafb1465640 1 -- 172.21.15.88:0/734610247 wait complete. 2024-09-16T01:43:45.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.987+0000 7fafb1465640 1 Processor -- start 2024-09-16T01:43:45.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.987+0000 7fafb1465640 1 -- start start 2024-09-16T01:43:45.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.988+0000 7fafb1465640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fafac0b6720 0x7fafac0cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:45.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.988+0000 7fafb1465640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafac0cf1f0 0x7fafac1f7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:45.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.988+0000 7fafb1465640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafac15e7a0 0x7fafac1f8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:45.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.988+0000 7fafb1465640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fafac0b7a00 con 0x7fafac0cf1f0 2024-09-16T01:43:45.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.988+0000 7fafb1465640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fafac0d2ef0 con 0x7fafac15e7a0 2024-09-16T01:43:45.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.988+0000 7fafb1465640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fafac15eb80 con 0x7fafac0b6720 2024-09-16T01:43:45.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.988+0000 7fafaa7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafac0cf1f0 0x7fafac1f7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:45.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.988+0000 7fafaa7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafac0cf1f0 0x7fafac1f7b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35846/0 (socket says 172.21.15.88:35846) 2024-09-16T01:43:45.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.988+0000 7fafaa7fc640 1 -- 172.21.15.88:0/1800828274 learned_addr learned my addr 172.21.15.88:0/1800828274 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:45.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.988+0000 7fafaaffd640 1 --2- 172.21.15.88:0/1800828274 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fafac0b6720 0x7fafac0cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:45.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.988+0000 7fafab7fe640 1 --2- 172.21.15.88:0/1800828274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafac15e7a0 0x7fafac1f8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:45.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.988+0000 7fafaa7fc640 1 -- 172.21.15.88:0/1800828274 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fafac0b6720 msgr2=0x7fafac0cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:45.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.988+0000 7fafaa7fc640 1 --2- 172.21.15.88:0/1800828274 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fafac0b6720 0x7fafac0cabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.988+0000 7fafaa7fc640 1 -- 172.21.15.88:0/1800828274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafac15e7a0 msgr2=0x7fafac1f8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:45.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.988+0000 7fafaa7fc640 1 --2- 172.21.15.88:0/1800828274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafac15e7a0 0x7fafac1f8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:45.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.988+0000 7fafaa7fc640 1 -- 172.21.15.88:0/1800828274 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafac158070 con 0x7fafac0cf1f0 2024-09-16T01:43:45.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.988+0000 7fafab7fe640 1 --2- 172.21.15.88:0/1800828274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafac15e7a0 0x7fafac1f8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:45.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.988+0000 7fafaa7fc640 1 --2- 172.21.15.88:0/1800828274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafac0cf1f0 0x7fafac1f7b00 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7fafa000d5f0 tx=0x7fafa000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:45.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.988+0000 7faf937fe640 1 -- 172.21.15.88:0/1800828274 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa00125f0 con 0x7fafac0cf1f0 2024-09-16T01:43:45.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.988+0000 7faf937fe640 1 -- 172.21.15.88:0/1800828274 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafa0019070 con 0x7fafac0cf1f0 2024-09-16T01:43:45.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.988+0000 7fafb1465640 1 -- 172.21.15.88:0/1800828274 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafac0cbdf0 con 0x7fafac0cf1f0 2024-09-16T01:43:45.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.988+0000 7faf937fe640 1 -- 172.21.15.88:0/1800828274 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa00129d0 con 0x7fafac0cf1f0 2024-09-16T01:43:45.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.989+0000 7fafb1465640 1 -- 172.21.15.88:0/1800828274 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafac0ce4c0 con 0x7fafac0cf1f0 2024-09-16T01:43:45.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.991+0000 7faf937fe640 1 -- 172.21.15.88:0/1800828274 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fafa0012b70 con 0x7fafac0cf1f0 2024-09-16T01:43:45.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.991+0000 7fafb1465640 1 -- 172.21.15.88:0/1800828274 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafac0b61e0 con 0x7fafac0cf1f0 2024-09-16T01:43:45.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.992+0000 7faf937fe640 1 --2- 172.21.15.88:0/1800828274 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faf9806c2b0 0x7faf9806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:45.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.992+0000 7faf937fe640 1 -- 172.21.15.88:0/1800828274 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8662+0+0 (secure 0 0 0) 0x7fafa008c380 con 0x7fafac0cf1f0 2024-09-16T01:43:45.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.992+0000 7fafaaffd640 1 --2- 172.21.15.88:0/1800828274 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faf9806c2b0 0x7faf9806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:45.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.992+0000 7fafaaffd640 1 --2- 172.21.15.88:0/1800828274 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faf9806c2b0 0x7faf9806e770 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fafac0b4b90 tx=0x7faf9c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:45.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:45.996+0000 7faf937fe640 1 -- 172.21.15.88:0/1800828274 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fafac158070 con 0x7fafac0cf1f0 2024-09-16T01:43:46.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.228+0000 7fafb1465640 1 -- 172.21.15.88:0/1800828274 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fafac0d2a80 con 0x7fafac0cf1f0 2024-09-16T01:43:46.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.875+0000 7faf937fe640 1 -- 172.21.15.88:0/1800828274 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v277) ==== 107+0+0 (secure 0 0 0) 0x7fafa0059fa0 con 0x7fafac0cf1f0 2024-09-16T01:43:46.877 INFO:teuthology.orchestra.run.smithi088.stderr:nothing to do: rank is not damaged 2024-09-16T01:43:46.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.879+0000 7fafb1465640 1 -- 172.21.15.88:0/1800828274 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faf9806c2b0 msgr2=0x7faf9806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:46.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.879+0000 7fafb1465640 1 --2- 172.21.15.88:0/1800828274 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faf9806c2b0 0x7faf9806e770 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fafac0b4b90 tx=0x7faf9c009040 comp rx=0 tx=0).stop 2024-09-16T01:43:46.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.879+0000 7fafb1465640 1 -- 172.21.15.88:0/1800828274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafac0cf1f0 msgr2=0x7fafac1f7b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:46.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.879+0000 7fafb1465640 1 --2- 172.21.15.88:0/1800828274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafac0cf1f0 0x7fafac1f7b00 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7fafa000d5f0 tx=0x7fafa000dab0 comp rx=0 tx=0).stop 2024-09-16T01:43:46.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.879+0000 7fafb1465640 1 -- 172.21.15.88:0/1800828274 shutdown_connections 2024-09-16T01:43:46.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.879+0000 7fafb1465640 1 --2- 172.21.15.88:0/1800828274 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7faf9806c2b0 0x7faf9806e770 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:46.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.879+0000 7fafb1465640 1 --2- 172.21.15.88:0/1800828274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fafac15e7a0 0x7fafac1f8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:46.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.879+0000 7fafb1465640 1 --2- 172.21.15.88:0/1800828274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fafac0cf1f0 0x7fafac1f7b00 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:46.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.879+0000 7fafb1465640 1 --2- 172.21.15.88:0/1800828274 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fafac0b6720 0x7fafac0cabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:46.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.879+0000 7fafb1465640 1 -- 172.21.15.88:0/1800828274 >> 172.21.15.88:0/1800828274 conn(0x7fafac0d11e0 msgr2=0x7fafac15c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:46.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.879+0000 7fafb1465640 1 -- 172.21.15.88:0/1800828274 shutdown_connections 2024-09-16T01:43:46.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.879+0000 7fafb1465640 1 -- 172.21.15.88:0/1800828274 wait complete. 2024-09-16T01:43:46.893 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:43:46.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.979+0000 7f7c35307f40 1 Processor -- start 2024-09-16T01:43:46.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.979+0000 7f7c35307f40 1 -- start start 2024-09-16T01:43:46.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.980+0000 7f7c35307f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561da3c17fe0 0x561da3c30960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:46.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.980+0000 7f7c35307f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561da3c18a60 0x561da3c14450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:46.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.980+0000 7f7c35307f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561da3c32250 0x561da3cbae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:46.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.980+0000 7f7c35307f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x561da3b3a510 con 0x561da3c17fe0 2024-09-16T01:43:46.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.980+0000 7f7c35307f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561da3cc1120 con 0x561da3c32250 2024-09-16T01:43:46.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.980+0000 7f7c35307f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x561da3c131e0 con 0x561da3c18a60 2024-09-16T01:43:46.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.980+0000 7f7c32dfd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561da3c17fe0 0x561da3c30960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:46.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.980+0000 7f7c32dfd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561da3c17fe0 0x561da3c30960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35874/0 (socket says 172.21.15.88:35874) 2024-09-16T01:43:46.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.980+0000 7f7c32dfd640 1 -- 172.21.15.88:0/11462237 learned_addr learned my addr 172.21.15.88:0/11462237 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:46.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.980+0000 7f7c335fe640 1 --2- 172.21.15.88:0/11462237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561da3c32250 0x561da3cbae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:46.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.980+0000 7f7c32dfd640 1 -- 172.21.15.88:0/11462237 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561da3c18a60 msgr2=0x561da3c14450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:46.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.980+0000 7f7c325fc640 1 --2- 172.21.15.88:0/11462237 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561da3c18a60 0x561da3c14450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:46.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.980+0000 7f7c32dfd640 1 --2- 172.21.15.88:0/11462237 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561da3c18a60 0x561da3c14450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:46.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.980+0000 7f7c32dfd640 1 -- 172.21.15.88:0/11462237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561da3c32250 msgr2=0x561da3cbae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:46.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.980+0000 7f7c32dfd640 1 --2- 172.21.15.88:0/11462237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561da3c32250 0x561da3cbae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:46.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.980+0000 7f7c32dfd640 1 -- 172.21.15.88:0/11462237 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561da3c12f00 con 0x561da3c17fe0 2024-09-16T01:43:46.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.980+0000 7f7c335fe640 1 --2- 172.21.15.88:0/11462237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561da3c32250 0x561da3cbae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:46.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.980+0000 7f7c325fc640 1 --2- 172.21.15.88:0/11462237 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561da3c18a60 0x561da3c14450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:46.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.980+0000 7f7c32dfd640 1 --2- 172.21.15.88:0/11462237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561da3c17fe0 0x561da3c30960 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f7c24025dc0 tx=0x7f7c24051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc215ddd94b0dc67 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:46.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.980+0000 7f7c315fa640 1 -- 172.21.15.88:0/11462237 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c24058950 con 0x561da3c17fe0 2024-09-16T01:43:46.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.980+0000 7f7c315fa640 1 -- 172.21.15.88:0/11462237 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c24058af0 con 0x561da3c17fe0 2024-09-16T01:43:46.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.980+0000 7f7c315fa640 1 -- 172.21.15.88:0/11462237 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c24064460 con 0x561da3c17fe0 2024-09-16T01:43:46.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.981+0000 7f7c35307f40 1 -- 172.21.15.88:0/11462237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561da3c17fe0 msgr2=0x561da3c30960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:46.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.981+0000 7f7c35307f40 1 --2- 172.21.15.88:0/11462237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561da3c17fe0 0x561da3c30960 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7f7c24025dc0 tx=0x7f7c24051000 comp rx=0 tx=0).stop 2024-09-16T01:43:46.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.981+0000 7f7c35307f40 1 -- 172.21.15.88:0/11462237 shutdown_connections 2024-09-16T01:43:46.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.981+0000 7f7c35307f40 1 --2- 172.21.15.88:0/11462237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561da3c32250 0x561da3cbae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:46.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.981+0000 7f7c35307f40 1 --2- 172.21.15.88:0/11462237 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561da3c18a60 0x561da3c14450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:46.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.981+0000 7f7c35307f40 1 --2- 172.21.15.88:0/11462237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561da3c17fe0 0x561da3c30960 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:46.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.981+0000 7f7c35307f40 1 -- 172.21.15.88:0/11462237 >> 172.21.15.88:0/11462237 conn(0x561da3c33750 msgr2=0x561da3cbbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:46.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.981+0000 7f7c35307f40 1 -- 172.21.15.88:0/11462237 shutdown_connections 2024-09-16T01:43:46.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.981+0000 7f7c35307f40 1 -- 172.21.15.88:0/11462237 wait complete. 2024-09-16T01:43:46.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.982+0000 7f7c35307f40 1 Processor -- start 2024-09-16T01:43:46.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.982+0000 7f7c35307f40 1 -- start start 2024-09-16T01:43:46.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.982+0000 7f7c35307f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561da3c17fe0 0x561da3cac510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:46.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.982+0000 7f7c35307f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561da3c18a60 0x561da3cb23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:46.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.982+0000 7f7c35307f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561da3c32250 0x561da3cb2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:46.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.982+0000 7f7c35307f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x561da3c14f50 con 0x561da3c18a60 2024-09-16T01:43:46.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.982+0000 7f7c35307f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561da3c2f1a0 con 0x561da3c32250 2024-09-16T01:43:46.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.982+0000 7f7c35307f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x561da3c15230 con 0x561da3c17fe0 2024-09-16T01:43:46.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.982+0000 7f7c325fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561da3c18a60 0x561da3cb23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:46.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.983+0000 7f7c325fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561da3c18a60 0x561da3cb23e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35880/0 (socket says 172.21.15.88:35880) 2024-09-16T01:43:46.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.983+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 learned_addr learned my addr 172.21.15.88:0/1557800989 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:46.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.983+0000 7f7c335fe640 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561da3c32250 0x561da3cb2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:46.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.983+0000 7f7c32dfd640 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561da3c17fe0 0x561da3cac510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:46.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.983+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561da3c17fe0 msgr2=0x561da3cac510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:46.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.983+0000 7f7c325fc640 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561da3c17fe0 0x561da3cac510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:46.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.983+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561da3c32250 msgr2=0x561da3cb2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:46.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.983+0000 7f7c325fc640 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561da3c32250 0x561da3cb2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:46.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.983+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561da3cb4e70 con 0x561da3c18a60 2024-09-16T01:43:46.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.983+0000 7f7c325fc640 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561da3c18a60 0x561da3cb23e0 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f7c28009700 tx=0x7f7c28009bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:46.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.983+0000 7f7c1bfff640 1 -- 172.21.15.88:0/1557800989 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c28006ac0 con 0x561da3c18a60 2024-09-16T01:43:46.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.983+0000 7f7c1bfff640 1 -- 172.21.15.88:0/1557800989 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c28006c60 con 0x561da3c18a60 2024-09-16T01:43:46.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.983+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x561da3cbfe00 con 0x561da3c18a60 2024-09-16T01:43:46.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.983+0000 7f7c1bfff640 1 -- 172.21.15.88:0/1557800989 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c28014680 con 0x561da3c18a60 2024-09-16T01:43:46.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.983+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561da3cb0700 con 0x561da3c18a60 2024-09-16T01:43:46.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.985+0000 7f7c1bfff640 1 -- 172.21.15.88:0/1557800989 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7c2801d440 con 0x561da3c18a60 2024-09-16T01:43:46.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.985+0000 7f7c1bfff640 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7c2006ac90 0x7f7c2006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:46.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.985+0000 7f7c32dfd640 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7c2006ac90 0x7f7c2006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:46.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.985+0000 7f7c1bfff640 1 -- 172.21.15.88:0/1557800989 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8662+0+0 (secure 0 0 0) 0x7f7c2808af70 con 0x561da3c18a60 2024-09-16T01:43:46.991 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:43:46.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.985+0000 7f7c35307f40 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x561da3cbc950 0x561da3cb6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:46.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.986+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3cb6e80 con 0x561da3cbc950 2024-09-16T01:43:46.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.986+0000 7f7c335fe640 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x561da3cbc950 0x561da3cb6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:46.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.986+0000 7f7c32dfd640 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7c2006ac90 0x7f7c2006d150 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f7c24025dc0 tx=0x7f7c24053000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:46.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.986+0000 7f7c335fe640 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x561da3cbc950 0x561da3cb6940 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:46.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.988+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v121'1033 uv1007 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:46.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.988+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e121) -- 0x561da3cb7250 con 0x561da3cbc950 2024-09-16T01:43:46.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.990+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v121'1034 uv1034 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:46.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.990+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63330 con 0x561da3cbc950 2024-09-16T01:43:46.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.991+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v121'1035 uv1035 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:46.993 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:43:46.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.991+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d646e0 con 0x561da3cbc950 2024-09-16T01:43:46.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.992+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v121'1036 uv1036 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:46.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.992+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d64a80 con 0x561da3cbc950 2024-09-16T01:43:46.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.993+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v121'1037 uv1037 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:46.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.993+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d636d0 con 0x561da3cbc950 2024-09-16T01:43:46.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.995+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v121'1038 uv1038 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:46.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.995+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63ac0 con 0x561da3cbc950 2024-09-16T01:43:46.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.996+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v121'1039 uv1013 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:46.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.996+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63eb0 con 0x561da3cbc950 2024-09-16T01:43:46.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.997+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v121'1040 uv1040 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:46.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.997+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63eb0 con 0x561da3cbc950 2024-09-16T01:43:47.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.999+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v121'1041 uv1041 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.000 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:43:47.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:46.999+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63e60 con 0x561da3cbc950 2024-09-16T01:43:47.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.000+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v121'1042 uv1042 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.000+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63e60 con 0x561da3cbc950 2024-09-16T01:43:47.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.001+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v121'1043 uv1043 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.001+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63e60 con 0x561da3cbc950 2024-09-16T01:43:47.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.003+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v121'1044 uv1044 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.003+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63e60 con 0x561da3cbc950 2024-09-16T01:43:47.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.004+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v121'1045 uv1020 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.004+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63e60 con 0x561da3cbc950 2024-09-16T01:43:47.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.006+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v121'1046 uv1046 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.006+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63e60 con 0x561da3cbc950 2024-09-16T01:43:47.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.007+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v121'1047 uv1047 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.010 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:43:47.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.007+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63e60 con 0x561da3cbc950 2024-09-16T01:43:47.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.008+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v121'1048 uv1048 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.008+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63e60 con 0x561da3cbc950 2024-09-16T01:43:47.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.009+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v121'1049 uv1049 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.009+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63e60 con 0x561da3cbc950 2024-09-16T01:43:47.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.011+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v121'1050 uv1050 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.011+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63e60 con 0x561da3cbc950 2024-09-16T01:43:47.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.013+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v121'1051 uv1051 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.013+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63e60 con 0x561da3cbc950 2024-09-16T01:43:47.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.015+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v121'1052 uv1026 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.015+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63e60 con 0x561da3cbc950 2024-09-16T01:43:47.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.016+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v121'1053 uv1053 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.016+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63e60 con 0x561da3cbc950 2024-09-16T01:43:47.018 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:43:47.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.017+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v121'1054 uv1054 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.017+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63e60 con 0x561da3cbc950 2024-09-16T01:43:47.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.019+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v121'1055 uv1055 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.019+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63e60 con 0x561da3cbc950 2024-09-16T01:43:47.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.020+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v121'1056 uv1056 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.020+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63e60 con 0x561da3cbc950 2024-09-16T01:43:47.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.021+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v121'1057 uv1057 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.021+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63e60 con 0x561da3cbc950 2024-09-16T01:43:47.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.023+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v121'1058 uv1032 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.023+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63e60 con 0x561da3cbc950 2024-09-16T01:43:47.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.024+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v121'1059 uv1059 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.024+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63e60 con 0x561da3cbc950 2024-09-16T01:43:47.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.025+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v121'1060 uv1060 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.027 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:43:47.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.025+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63e60 con 0x561da3cbc950 2024-09-16T01:43:47.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.027+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v121'1061 uv1061 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.027+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63e60 con 0x561da3cbc950 2024-09-16T01:43:47.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.028+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v121'1062 uv1062 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.028+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63e60 con 0x561da3cbc950 2024-09-16T01:43:47.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.029+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v121'1063 uv1063 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.029+0000 7f7c35307f40 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x561da3d64240 0x561da3d67340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:47.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.029+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d63e60 con 0x561da3d64240 2024-09-16T01:43:47.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.030+0000 7f7c32dfd640 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x561da3d64240 0x561da3d67340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:47.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.030+0000 7f7c32dfd640 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x561da3d64240 0x561da3d67340 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:47.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.032+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v121'753 uv732 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d64240 2024-09-16T01:43:47.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.032+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e121) -- 0x561da3d67880 con 0x561da3d64240 2024-09-16T01:43:47.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.033+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v121'754 uv754 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d64240 2024-09-16T01:43:47.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.033+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d67c20 con 0x561da3d64240 2024-09-16T01:43:47.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.034+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v121'755 uv755 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d64240 2024-09-16T01:43:47.036 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:43:47.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.034+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d68470 con 0x561da3d64240 2024-09-16T01:43:47.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.036+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v121'756 uv756 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d64240 2024-09-16T01:43:47.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.036+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d68810 con 0x561da3d64240 2024-09-16T01:43:47.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.037+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v121'757 uv757 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d64240 2024-09-16T01:43:47.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.037+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d68bb0 con 0x561da3d64240 2024-09-16T01:43:47.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.038+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v121'758 uv751 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d64240 2024-09-16T01:43:47.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.038+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e121) -- 0x561da3d68f50 con 0x561da3d64240 2024-09-16T01:43:47.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.039+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v121'759 uv759 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d64240 2024-09-16T01:43:47.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.039+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d692f0 con 0x561da3d64240 2024-09-16T01:43:47.042 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:43:47.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.041+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v121'760 uv760 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d64240 2024-09-16T01:43:47.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.041+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d692f0 con 0x561da3d64240 2024-09-16T01:43:47.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.042+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v121'761 uv761 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d64240 2024-09-16T01:43:47.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.042+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d692f0 con 0x561da3d64240 2024-09-16T01:43:47.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.043+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v121'762 uv762 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d64240 2024-09-16T01:43:47.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.043+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d692f0 con 0x561da3d64240 2024-09-16T01:43:47.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.045+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v121'763 uv752 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d64240 2024-09-16T01:43:47.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.045+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e121) -- 0x561da3d692f0 con 0x561da3d64240 2024-09-16T01:43:47.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.046+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v121'764 uv764 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d64240 2024-09-16T01:43:47.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.046+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d692f0 con 0x561da3d64240 2024-09-16T01:43:47.048 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:43:47.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.047+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v121'765 uv765 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d64240 2024-09-16T01:43:47.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.047+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d6d770 con 0x561da3d64240 2024-09-16T01:43:47.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.048+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v121'766 uv766 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d64240 2024-09-16T01:43:47.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.048+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d6d770 con 0x561da3d64240 2024-09-16T01:43:47.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.050+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v121'767 uv767 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d64240 2024-09-16T01:43:47.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.050+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d692f0 con 0x561da3d64240 2024-09-16T01:43:47.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.051+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v121'768 uv748 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d64240 2024-09-16T01:43:47.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.051+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e121) -- 0x561da3d692f0 con 0x561da3d64240 2024-09-16T01:43:47.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.052+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v121'769 uv769 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d64240 2024-09-16T01:43:47.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.052+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d692f0 con 0x561da3d64240 2024-09-16T01:43:47.054 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:43:47.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.053+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v121'770 uv770 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d64240 2024-09-16T01:43:47.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.053+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d692f0 con 0x561da3d64240 2024-09-16T01:43:47.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.055+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v121'771 uv771 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d64240 2024-09-16T01:43:47.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.055+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d6a300 con 0x561da3d64240 2024-09-16T01:43:47.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.056+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v121'772 uv772 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d64240 2024-09-16T01:43:47.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.056+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d692f0 con 0x561da3d64240 2024-09-16T01:43:47.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.057+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v121'773 uv773 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d64240 2024-09-16T01:43:47.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.057+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d692f0 con 0x561da3cbc950 2024-09-16T01:43:47.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.058+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v121'201 uv200 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.058+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e121) -- 0x561da3d692f0 con 0x561da3cbc950 2024-09-16T01:43:47.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.059+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v121'202 uv202 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.059+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d692f0 con 0x561da3cbc950 2024-09-16T01:43:47.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.060+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v121'203 uv203 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.061 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:43:47.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.060+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d692f0 con 0x561da3cbc950 2024-09-16T01:43:47.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.061+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v121'204 uv204 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.061+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d692f0 con 0x561da3cbc950 2024-09-16T01:43:47.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.062+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v121'205 uv205 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.062+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d692f0 con 0x561da3cbc950 2024-09-16T01:43:47.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.062+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v121'206 uv206 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.062+0000 7f7c35307f40 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x561da3d69bf0 0x561da3d6a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:47.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.062+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d6c740 con 0x561da3d69bf0 2024-09-16T01:43:47.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.063+0000 7f7c325fc640 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x561da3d69bf0 0x561da3d6a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:47.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.063+0000 7f7c325fc640 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x561da3d69bf0 0x561da3d6a060 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:47.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.064+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v121'151 uv150 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f7c28005040 con 0x561da3d69bf0 2024-09-16T01:43:47.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.064+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e121) -- 0x561da3d6cae0 con 0x561da3d69bf0 2024-09-16T01:43:47.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.065+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v121'152 uv152 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7c28005040 con 0x561da3d69bf0 2024-09-16T01:43:47.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.065+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d6cf90 con 0x561da3d69bf0 2024-09-16T01:43:47.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.066+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v121'153 uv153 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7c28005040 con 0x561da3d69bf0 2024-09-16T01:43:47.067 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:43:47.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.066+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d680a0 con 0x561da3d69bf0 2024-09-16T01:43:47.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.067+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v121'154 uv154 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f7c28005040 con 0x561da3d69bf0 2024-09-16T01:43:47.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.067+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d6d8c0 con 0x561da3cbc950 2024-09-16T01:43:47.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.068+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v121'201 uv200 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.068+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e121) -- 0x561da3d6d8c0 con 0x561da3cbc950 2024-09-16T01:43:47.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.070+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v121'202 uv202 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.070+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d6d8c0 con 0x561da3cbc950 2024-09-16T01:43:47.073 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:43:47.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.072+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v121'203 uv203 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.072+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d6e8d0 con 0x561da3cbc950 2024-09-16T01:43:47.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.077+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v121'204 uv204 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.077+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d6e8d0 con 0x561da3cbc950 2024-09-16T01:43:47.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.078+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v121'205 uv205 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.078+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d6d8c0 con 0x561da3cbc950 2024-09-16T01:43:47.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.081+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v121'206 uv206 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.081+0000 7f7c35307f40 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x561da3d6df40 0x561da3d6e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:47.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.081+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d70d20 con 0x561da3d6df40 2024-09-16T01:43:47.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.081+0000 7f7c335fe640 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x561da3d6df40 0x561da3d6e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:47.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.082+0000 7f7c335fe640 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x561da3d6df40 0x561da3d6e320 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:47.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.083+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v121'371 uv365 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.083+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e121) -- 0x561da3d70d20 con 0x561da3d6df40 2024-09-16T01:43:47.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.085+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v121'372 uv372 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.085+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d70d20 con 0x561da3d6df40 2024-09-16T01:43:47.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.086+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v121'373 uv373 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.088 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:43:47.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.086+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d71d30 con 0x561da3d6df40 2024-09-16T01:43:47.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.087+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v121'374 uv374 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.087+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d71d30 con 0x561da3d6df40 2024-09-16T01:43:47.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.088+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v121'375 uv375 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.088+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d70d20 con 0x561da3d6df40 2024-09-16T01:43:47.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.089+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v121'376 uv376 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.089+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d70d20 con 0x561da3d6df40 2024-09-16T01:43:47.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.090+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v121'377 uv370 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.090+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e121) -- 0x561da3d70d20 con 0x561da3d6df40 2024-09-16T01:43:47.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.090+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v121'378 uv378 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.090+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d70d20 con 0x561da3d6df40 2024-09-16T01:43:47.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.091+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v121'379 uv379 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.093 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:43:47.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.091+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d70d20 con 0x561da3d6df40 2024-09-16T01:43:47.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.093+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v121'380 uv380 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.093+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d70d20 con 0x561da3d6df40 2024-09-16T01:43:47.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.094+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v121'381 uv381 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.094+0000 7f7c35307f40 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x561da3d71620 0x561da3d73a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:47.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.094+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d73f50 con 0x561da3d71620 2024-09-16T01:43:47.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.094+0000 7f7c32dfd640 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x561da3d71620 0x561da3d73a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:47.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.095+0000 7f7c32dfd640 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x561da3d71620 0x561da3d73a10 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:47.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.096+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v121'168 uv167 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d71620 2024-09-16T01:43:47.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.096+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e121) -- 0x561da3d73fd0 con 0x561da3d71620 2024-09-16T01:43:47.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.097+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v121'169 uv169 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d71620 2024-09-16T01:43:47.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.097+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d743f0 con 0x561da3d71620 2024-09-16T01:43:47.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.098+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v121'170 uv170 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d71620 2024-09-16T01:43:47.100 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:43:47.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.098+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d743f0 con 0x561da3d71620 2024-09-16T01:43:47.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.099+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v121'171 uv171 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d71620 2024-09-16T01:43:47.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.099+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d743f0 con 0x561da3d71620 2024-09-16T01:43:47.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.100+0000 7f7c32dfd640 1 -- 172.21.15.88:0/1557800989 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v121'172 uv172 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7c240647c0 con 0x561da3d71620 2024-09-16T01:43:47.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.100+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d743f0 con 0x561da3cbc950 2024-09-16T01:43:47.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.101+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v121'201 uv200 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.101+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e121) -- 0x561da3d743f0 con 0x561da3cbc950 2024-09-16T01:43:47.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.102+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v121'202 uv202 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.102+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d743f0 con 0x561da3cbc950 2024-09-16T01:43:47.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.103+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v121'203 uv203 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.105 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:43:47.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.103+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d75400 con 0x561da3cbc950 2024-09-16T01:43:47.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.104+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v121'204 uv204 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.104+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d75400 con 0x561da3cbc950 2024-09-16T01:43:47.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.105+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v121'205 uv205 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.105+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d743f0 con 0x561da3cbc950 2024-09-16T01:43:47.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.106+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v121'206 uv206 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3cbc950 2024-09-16T01:43:47.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.106+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d743f0 con 0x561da3d69bf0 2024-09-16T01:43:47.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.107+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v121'335 uv329 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f7c28005040 con 0x561da3d69bf0 2024-09-16T01:43:47.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.107+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e121) -- 0x561da3d74790 con 0x561da3d69bf0 2024-09-16T01:43:47.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.108+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v121'336 uv336 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7c28005040 con 0x561da3d69bf0 2024-09-16T01:43:47.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.108+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d74b30 con 0x561da3d69bf0 2024-09-16T01:43:47.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.109+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v121'337 uv337 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7c28005040 con 0x561da3d69bf0 2024-09-16T01:43:47.111 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:43:47.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.109+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d74ed0 con 0x561da3d69bf0 2024-09-16T01:43:47.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.110+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v121'338 uv338 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7c28005040 con 0x561da3d69bf0 2024-09-16T01:43:47.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.110+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d74ed0 con 0x561da3d69bf0 2024-09-16T01:43:47.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.111+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v121'339 uv339 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f7c28005040 con 0x561da3d69bf0 2024-09-16T01:43:47.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.111+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d74ed0 con 0x561da3d69bf0 2024-09-16T01:43:47.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.112+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v121'340 uv334 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7c28005040 con 0x561da3d69bf0 2024-09-16T01:43:47.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.112+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e121) -- 0x561da3d74ed0 con 0x561da3d69bf0 2024-09-16T01:43:47.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.113+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v121'341 uv341 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c28005040 con 0x561da3d69bf0 2024-09-16T01:43:47.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.113+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d74ed0 con 0x561da3d69bf0 2024-09-16T01:43:47.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.113+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v121'342 uv342 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c28005040 con 0x561da3d69bf0 2024-09-16T01:43:47.116 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:43:47.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.113+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d74ed0 con 0x561da3d69bf0 2024-09-16T01:43:47.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.114+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v121'343 uv343 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c28005040 con 0x561da3d69bf0 2024-09-16T01:43:47.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.114+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d6d4b0 con 0x561da3d69bf0 2024-09-16T01:43:47.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.115+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v121'344 uv344 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c28005040 con 0x561da3d69bf0 2024-09-16T01:43:47.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.115+0000 7f7c35307f40 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x561da3d75c80 0x561da3d78440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:47.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.115+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d78980 con 0x561da3d75c80 2024-09-16T01:43:47.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.115+0000 7f7c325fc640 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x561da3d75c80 0x561da3d78440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:47.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.116+0000 7f7c325fc640 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x561da3d75c80 0x561da3d78440 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:47.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.117+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v121'147 uv145 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f7c28005040 con 0x561da3d75c80 2024-09-16T01:43:47.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.117+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e121) -- 0x561da3d78980 con 0x561da3d75c80 2024-09-16T01:43:47.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.118+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v121'148 uv148 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f7c28005040 con 0x561da3d75c80 2024-09-16T01:43:47.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.118+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d78980 con 0x561da3d75c80 2024-09-16T01:43:47.121 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:43:47.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.119+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v121'149 uv149 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f7c28005040 con 0x561da3d75c80 2024-09-16T01:43:47.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.119+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d78980 con 0x561da3d75c80 2024-09-16T01:43:47.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.120+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v121'150 uv150 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f7c28005040 con 0x561da3d75c80 2024-09-16T01:43:47.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.120+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d78980 con 0x561da3d69bf0 2024-09-16T01:43:47.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.121+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v121'201 uv200 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7c28005040 con 0x561da3d69bf0 2024-09-16T01:43:47.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.121+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e121) -- 0x561da3d78980 con 0x561da3d69bf0 2024-09-16T01:43:47.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.122+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v121'202 uv202 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c28005040 con 0x561da3d69bf0 2024-09-16T01:43:47.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.122+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d78980 con 0x561da3d69bf0 2024-09-16T01:43:47.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.123+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v121'203 uv203 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c28005040 con 0x561da3d69bf0 2024-09-16T01:43:47.126 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:43:47.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.123+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d79990 con 0x561da3d69bf0 2024-09-16T01:43:47.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.124+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v121'204 uv204 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c28005040 con 0x561da3d69bf0 2024-09-16T01:43:47.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.124+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d79990 con 0x561da3d69bf0 2024-09-16T01:43:47.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.125+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v121'205 uv205 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c28005040 con 0x561da3d69bf0 2024-09-16T01:43:47.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.126+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d78980 con 0x561da3d69bf0 2024-09-16T01:43:47.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.127+0000 7f7c325fc640 1 -- 172.21.15.88:0/1557800989 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v121'206 uv206 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c28005040 con 0x561da3d69bf0 2024-09-16T01:43:47.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.127+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d78980 con 0x561da3d6df40 2024-09-16T01:43:47.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.128+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v121'171 uv170 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.128+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e121) -- 0x561da3d78980 con 0x561da3d6df40 2024-09-16T01:43:47.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.129+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v121'172 uv172 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.129+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d78980 con 0x561da3d6df40 2024-09-16T01:43:47.132 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:43:47.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.130+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v121'173 uv173 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.130+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d78980 con 0x561da3d6df40 2024-09-16T01:43:47.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.132+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v121'174 uv174 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.132+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d78980 con 0x561da3d6df40 2024-09-16T01:43:47.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.133+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v121'175 uv175 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.133+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d78980 con 0x561da3d6df40 2024-09-16T01:43:47.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.134+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v121'416 uv406 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.134+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e121) -- 0x561da3d78980 con 0x561da3d6df40 2024-09-16T01:43:47.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.135+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v121'417 uv417 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.135+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d78980 con 0x561da3d6df40 2024-09-16T01:43:47.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.136+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v121'418 uv418 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.139 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:43:47.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.136+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d78980 con 0x561da3d6df40 2024-09-16T01:43:47.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.137+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v121'419 uv419 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.138+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d78980 con 0x561da3d6df40 2024-09-16T01:43:47.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.139+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v121'420 uv420 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.139+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d78980 con 0x561da3d6df40 2024-09-16T01:43:47.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.140+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v121'421 uv414 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.140+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e121) -- 0x561da3d78980 con 0x561da3d6df40 2024-09-16T01:43:47.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.141+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v121'422 uv422 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.141+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d78980 con 0x561da3d6df40 2024-09-16T01:43:47.144 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:43:47.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.142+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v121'423 uv423 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.142+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d79990 con 0x561da3d6df40 2024-09-16T01:43:47.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.143+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v121'424 uv424 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.143+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d79990 con 0x561da3d6df40 2024-09-16T01:43:47.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.144+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v121'425 uv425 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.144+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d78980 con 0x561da3d6df40 2024-09-16T01:43:47.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.145+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v121'426 uv426 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.146+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x561da3d78980 con 0x561da3d6df40 2024-09-16T01:43:47.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v121'427 uv427 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f7c2c00c880 con 0x561da3d6df40 2024-09-16T01:43:47.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x561da3cbc950 msgr2=0x561da3cb6940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:47.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c35307f40 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x561da3cbc950 0x561da3cb6940 crc :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x561da3d69bf0 msgr2=0x561da3d6a060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:47.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c35307f40 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x561da3d69bf0 0x561da3d6a060 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x561da3d75c80 msgr2=0x561da3d78440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:47.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c35307f40 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x561da3d75c80 0x561da3d78440 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x561da3d64240 msgr2=0x561da3d67340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:47.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c35307f40 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x561da3d64240 0x561da3d67340 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 reap_dead start 2024-09-16T01:43:47.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x561da3d6df40 msgr2=0x561da3d6e320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:47.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c35307f40 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x561da3d6df40 0x561da3d6e320 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c335fe640 1 -- 172.21.15.88:0/1557800989 reap_dead start 2024-09-16T01:43:47.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x561da3d71620 msgr2=0x561da3d73a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:47.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c35307f40 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x561da3d71620 0x561da3d73a10 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7c2006ac90 msgr2=0x7f7c2006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:47.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c35307f40 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7c2006ac90 0x7f7c2006d150 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f7c24025dc0 tx=0x7f7c24053000 comp rx=0 tx=0).stop 2024-09-16T01:43:47.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561da3c18a60 msgr2=0x561da3cb23e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:47.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c35307f40 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561da3c18a60 0x561da3cb23e0 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f7c28009700 tx=0x7f7c28009bc0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 shutdown_connections 2024-09-16T01:43:47.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c35307f40 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7c2006ac90 0x7f7c2006d150 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c35307f40 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x561da3d71620 0x561da3d73a10 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c35307f40 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x561da3d6df40 0x561da3d6e320 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c35307f40 1 --2- 172.21.15.88:0/1557800989 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x561da3c18a60 0x561da3cb23e0 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 >> 172.21.15.88:0/1557800989 conn(0x561da3c33750 msgr2=0x561da3c17750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:47.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 shutdown_connections 2024-09-16T01:43:47.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.147+0000 7f7c35307f40 1 -- 172.21.15.88:0/1557800989 wait complete. 2024-09-16T01:43:47.154 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapheader 10000000000.00000000 deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-09-16T01:43:47.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.236+0000 7fd781509f40 1 Processor -- start 2024-09-16T01:43:47.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.236+0000 7fd781509f40 1 -- start start 2024-09-16T01:43:47.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.236+0000 7fd781509f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f9cb65940 0x560f9cc35e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:47.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.236+0000 7fd781509f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f9cb65d50 0x560f9cc38840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:47.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.236+0000 7fd781509f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560f9cc39f30 0x560f9cc38d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:47.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.236+0000 7fd781509f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x560f9cab6510 con 0x560f9cb65d50 2024-09-16T01:43:47.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.236+0000 7fd781509f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560f9cb0fc50 con 0x560f9cb65940 2024-09-16T01:43:47.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.236+0000 7fd781509f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x560f9cb63c80 con 0x560f9cc39f30 2024-09-16T01:43:47.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.236+0000 7fd777fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f9cb65d50 0x560f9cc38840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:47.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.237+0000 7fd777fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f9cb65d50 0x560f9cc38840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35890/0 (socket says 172.21.15.88:35890) 2024-09-16T01:43:47.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.237+0000 7fd777fff640 1 -- 172.21.15.88:0/4284261104 learned_addr learned my addr 172.21.15.88:0/4284261104 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:47.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.237+0000 7fd77efff640 1 --2- 172.21.15.88:0/4284261104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f9cb65940 0x560f9cc35e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:47.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.237+0000 7fd77f800640 1 --2- 172.21.15.88:0/4284261104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560f9cc39f30 0x560f9cc38d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:47.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.237+0000 7fd777fff640 1 -- 172.21.15.88:0/4284261104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560f9cc39f30 msgr2=0x560f9cc38d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:47.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.237+0000 7fd777fff640 1 --2- 172.21.15.88:0/4284261104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560f9cc39f30 0x560f9cc38d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.237+0000 7fd777fff640 1 -- 172.21.15.88:0/4284261104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f9cb65940 msgr2=0x560f9cc35e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:47.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.237+0000 7fd777fff640 1 --2- 172.21.15.88:0/4284261104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f9cb65940 0x560f9cc35e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.237+0000 7fd777fff640 1 -- 172.21.15.88:0/4284261104 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f9cc382a0 con 0x560f9cb65d50 2024-09-16T01:43:47.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.237+0000 7fd777fff640 1 --2- 172.21.15.88:0/4284261104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f9cb65d50 0x560f9cc38840 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7fd768045d00 tx=0x7fd768051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c71f0b727554ee server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:47.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.237+0000 7fd77dffd640 1 -- 172.21.15.88:0/4284261104 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76804f9d0 con 0x560f9cb65d50 2024-09-16T01:43:47.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.237+0000 7fd77dffd640 1 -- 172.21.15.88:0/4284261104 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd76804fb70 con 0x560f9cb65d50 2024-09-16T01:43:47.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.237+0000 7fd77dffd640 1 -- 172.21.15.88:0/4284261104 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd768064420 con 0x560f9cb65d50 2024-09-16T01:43:47.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.238+0000 7fd781509f40 1 -- 172.21.15.88:0/4284261104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f9cb65d50 msgr2=0x560f9cc38840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:47.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.238+0000 7fd781509f40 1 --2- 172.21.15.88:0/4284261104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f9cb65d50 0x560f9cc38840 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7fd768045d00 tx=0x7fd768051000 comp rx=0 tx=0).stop 2024-09-16T01:43:47.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.238+0000 7fd781509f40 1 -- 172.21.15.88:0/4284261104 shutdown_connections 2024-09-16T01:43:47.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.238+0000 7fd781509f40 1 --2- 172.21.15.88:0/4284261104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560f9cc39f30 0x560f9cc38d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.238+0000 7fd781509f40 1 --2- 172.21.15.88:0/4284261104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f9cb65d50 0x560f9cc38840 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.238+0000 7fd781509f40 1 --2- 172.21.15.88:0/4284261104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f9cb65940 0x560f9cc35e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.238+0000 7fd781509f40 1 -- 172.21.15.88:0/4284261104 >> 172.21.15.88:0/4284261104 conn(0x560f9cbaf750 msgr2=0x560f9cc3d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:47.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.238+0000 7fd781509f40 1 -- 172.21.15.88:0/4284261104 shutdown_connections 2024-09-16T01:43:47.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.238+0000 7fd781509f40 1 -- 172.21.15.88:0/4284261104 wait complete. 2024-09-16T01:43:47.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.238+0000 7fd781509f40 1 Processor -- start 2024-09-16T01:43:47.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.238+0000 7fd781509f40 1 -- start start 2024-09-16T01:43:47.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.239+0000 7fd781509f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f9cb65940 0x560f9cc31b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:47.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.239+0000 7fd781509f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560f9cb65d50 0x560f9cc2ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:47.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.239+0000 7fd781509f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f9cc39f30 0x560f9cc28600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:47.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.239+0000 7fd781509f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x560f9cbaaae0 con 0x560f9cc39f30 2024-09-16T01:43:47.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.239+0000 7fd781509f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560f9cbab0a0 con 0x560f9cb65940 2024-09-16T01:43:47.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.239+0000 7fd781509f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x560f9cb539b0 con 0x560f9cb65d50 2024-09-16T01:43:47.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.239+0000 7fd77f800640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f9cc39f30 0x560f9cc28600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:47.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.239+0000 7fd77f800640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f9cc39f30 0x560f9cc28600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35896/0 (socket says 172.21.15.88:35896) 2024-09-16T01:43:47.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.239+0000 7fd77f800640 1 -- 172.21.15.88:0/1789170578 learned_addr learned my addr 172.21.15.88:0/1789170578 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:47.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.239+0000 7fd77efff640 1 --2- 172.21.15.88:0/1789170578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f9cb65940 0x560f9cc31b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:47.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.239+0000 7fd777fff640 1 --2- 172.21.15.88:0/1789170578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560f9cb65d50 0x560f9cc2ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:47.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.239+0000 7fd77f800640 1 -- 172.21.15.88:0/1789170578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560f9cb65d50 msgr2=0x560f9cc2ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:47.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.239+0000 7fd77f800640 1 --2- 172.21.15.88:0/1789170578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560f9cb65d50 0x560f9cc2ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.239+0000 7fd77f800640 1 -- 172.21.15.88:0/1789170578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f9cb65940 msgr2=0x560f9cc31b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:47.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.239+0000 7fd77f800640 1 --2- 172.21.15.88:0/1789170578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f9cb65940 0x560f9cc31b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.239+0000 7fd77f800640 1 -- 172.21.15.88:0/1789170578 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f9cb53610 con 0x560f9cc39f30 2024-09-16T01:43:47.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.239+0000 7fd777fff640 1 --2- 172.21.15.88:0/1789170578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560f9cb65d50 0x560f9cc2ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:47.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.239+0000 7fd77efff640 1 --2- 172.21.15.88:0/1789170578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560f9cb65940 0x560f9cc31b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:47.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.239+0000 7fd77f800640 1 --2- 172.21.15.88:0/1789170578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f9cc39f30 0x560f9cc28600 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7fd778004250 tx=0x7fd77800d5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:47.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.240+0000 7fd77cffb640 1 -- 172.21.15.88:0/1789170578 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd778018070 con 0x560f9cc39f30 2024-09-16T01:43:47.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.240+0000 7fd77cffb640 1 -- 172.21.15.88:0/1789170578 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd77800de20 con 0x560f9cc39f30 2024-09-16T01:43:47.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.240+0000 7fd781509f40 1 -- 172.21.15.88:0/1789170578 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x560f9cc320a0 con 0x560f9cc39f30 2024-09-16T01:43:47.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.240+0000 7fd77cffb640 1 -- 172.21.15.88:0/1789170578 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd778001540 con 0x560f9cc39f30 2024-09-16T01:43:47.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.240+0000 7fd781509f40 1 -- 172.21.15.88:0/1789170578 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f9cc36ea0 con 0x560f9cc39f30 2024-09-16T01:43:47.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.241+0000 7fd77cffb640 1 -- 172.21.15.88:0/1789170578 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd77801b070 con 0x560f9cc39f30 2024-09-16T01:43:47.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.241+0000 7fd77cffb640 1 --2- 172.21.15.88:0/1789170578 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd76006ac90 0x7fd76006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:47.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.242+0000 7fd77cffb640 1 -- 172.21.15.88:0/1789170578 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8662+0+0 (secure 0 0 0) 0x7fd77808a320 con 0x560f9cc39f30 2024-09-16T01:43:47.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.242+0000 7fd77efff640 1 --2- 172.21.15.88:0/1789170578 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd76006ac90 0x7fd76006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:47.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.242+0000 7fd781509f40 1 --2- 172.21.15.88:0/1789170578 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x560f9cc2f2b0 0x560f9cc2c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:47.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.242+0000 7fd781509f40 1 -- 172.21.15.88:0/1789170578 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121) -- 0x560f9cc28ff0 con 0x560f9cc2f2b0 2024-09-16T01:43:47.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.242+0000 7fd777fff640 1 --2- 172.21.15.88:0/1789170578 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x560f9cc2f2b0 0x560f9cc2c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:47.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.242+0000 7fd77efff640 1 --2- 172.21.15.88:0/1789170578 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd76006ac90 0x7fd76006d150 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x560f9cc39590 tx=0x7fd770001700 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:47.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.242+0000 7fd777fff640 1 --2- 172.21.15.88:0/1789170578 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x560f9cc2f2b0 0x560f9cc2c070 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:47.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.243+0000 7fd777fff640 1 -- 172.21.15.88:0/1789170578 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-header] v121'1064 uv1064 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd768058400 con 0x560f9cc2f2b0 2024-09-16T01:43:47.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.244+0000 7fd781509f40 1 -- 172.21.15.88:0/1789170578 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x560f9cc2f2b0 msgr2=0x560f9cc2c070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:47.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.244+0000 7fd781509f40 1 --2- 172.21.15.88:0/1789170578 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x560f9cc2f2b0 0x560f9cc2c070 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.244+0000 7fd781509f40 1 -- 172.21.15.88:0/1789170578 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd76006ac90 msgr2=0x7fd76006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:47.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.244+0000 7fd781509f40 1 --2- 172.21.15.88:0/1789170578 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd76006ac90 0x7fd76006d150 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x560f9cc39590 tx=0x7fd770001700 comp rx=0 tx=0).stop 2024-09-16T01:43:47.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.244+0000 7fd781509f40 1 -- 172.21.15.88:0/1789170578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f9cc39f30 msgr2=0x560f9cc28600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:47.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.244+0000 7fd781509f40 1 --2- 172.21.15.88:0/1789170578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560f9cc39f30 0x560f9cc28600 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7fd778004250 tx=0x7fd77800d5f0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.244+0000 7fd77f800640 1 -- 172.21.15.88:0/1789170578 reap_dead start 2024-09-16T01:43:47.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.244+0000 7fd781509f40 1 -- 172.21.15.88:0/1789170578 shutdown_connections 2024-09-16T01:43:47.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.244+0000 7fd781509f40 1 -- 172.21.15.88:0/1789170578 >> 172.21.15.88:0/1789170578 conn(0x560f9cbaf750 msgr2=0x560f9cba9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:47.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.244+0000 7fd781509f40 1 -- 172.21.15.88:0/1789170578 shutdown_connections 2024-09-16T01:43:47.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.244+0000 7fd781509f40 1 -- 172.21.15.88:0/1789170578 wait complete. 2024-09-16T01:43:47.255 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:43:47.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.387+0000 7ff151a77640 1 Processor -- start 2024-09-16T01:43:47.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.387+0000 7ff151a77640 1 -- start start 2024-09-16T01:43:47.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.388+0000 7ff151a77640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff14c158a00 0x7ff14c071f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:47.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.388+0000 7ff151a77640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff14c072470 0x7ff14c0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:47.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.388+0000 7ff151a77640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff14c0728f0 con 0x7ff14c158de0 2024-09-16T01:43:47.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.388+0000 7ff151a77640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff14c072bd0 con 0x7ff14c158a00 2024-09-16T01:43:47.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.388+0000 7ff151a77640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff14c072eb0 con 0x7ff14c072470 2024-09-16T01:43:47.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.388+0000 7ff14b7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff14c158a00 0x7ff14c071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:47.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.388+0000 7ff14bfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff14c158de0 0x7ff14c162110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42450/0 (socket says 172.21.15.88:42450) 2024-09-16T01:43:47.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.388+0000 7ff14bfff640 1 -- 172.21.15.88:0/2202514368 learned_addr learned my addr 172.21.15.88:0/2202514368 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:47.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.389+0000 7ff143fff640 1 --2- 172.21.15.88:0/2202514368 >> v2:172.21.15.154:3300/0 conn(0x7ff14c072470 0x7ff14c0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:47.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.389+0000 7ff14b7fe640 1 -- 172.21.15.88:0/2202514368 >> v2:172.21.15.154:3300/0 conn(0x7ff14c072470 msgr2=0x7ff14c0b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:47.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.389+0000 7ff14b7fe640 1 --2- 172.21.15.88:0/2202514368 >> v2:172.21.15.154:3300/0 conn(0x7ff14c072470 0x7ff14c0b4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.389+0000 7ff14b7fe640 1 -- 172.21.15.88:0/2202514368 >> v1:172.21.15.88:6789/0 conn(0x7ff14c158de0 legacy=0x7ff14c162110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:47.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.389+0000 7ff14affd640 1 -- 172.21.15.88:0/2202514368 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 179294093 0 0) 0x7ff14c0728f0 con 0x7ff14c158de0 2024-09-16T01:43:47.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.389+0000 7ff14b7fe640 1 -- 172.21.15.88:0/2202514368 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff14c073190 con 0x7ff14c158a00 2024-09-16T01:43:47.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.389+0000 7ff143fff640 1 --2- 172.21.15.88:0/2202514368 >> v2:172.21.15.154:3300/0 conn(0x7ff14c072470 0x7ff14c0b4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:47.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.389+0000 7ff14b7fe640 1 --2- 172.21.15.88:0/2202514368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff14c158a00 0x7ff14c071f30 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7ff14c15eb20 tx=0x7ff13c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc4e48f1fdf7ba4c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:47.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.389+0000 7ff14affd640 1 -- 172.21.15.88:0/2202514368 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13c037070 con 0x7ff14c158a00 2024-09-16T01:43:47.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.390+0000 7ff14affd640 1 -- 172.21.15.88:0/2202514368 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff13c032440 con 0x7ff14c158a00 2024-09-16T01:43:47.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.390+0000 7ff14affd640 1 -- 172.21.15.88:0/2202514368 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13c032810 con 0x7ff14c158a00 2024-09-16T01:43:47.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.390+0000 7ff151a77640 1 -- 172.21.15.88:0/2202514368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff14c158a00 msgr2=0x7ff14c071f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:47.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.390+0000 7ff151a77640 1 --2- 172.21.15.88:0/2202514368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff14c158a00 0x7ff14c071f30 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7ff14c15eb20 tx=0x7ff13c02a1a0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.390+0000 7ff151a77640 1 -- 172.21.15.88:0/2202514368 shutdown_connections 2024-09-16T01:43:47.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.390+0000 7ff151a77640 1 --2- 172.21.15.88:0/2202514368 >> v2:172.21.15.154:3300/0 conn(0x7ff14c072470 0x7ff14c0b4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.390+0000 7ff151a77640 1 --2- 172.21.15.88:0/2202514368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff14c158a00 0x7ff14c071f30 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.390+0000 7ff151a77640 1 -- 172.21.15.88:0/2202514368 >> 172.21.15.88:0/2202514368 conn(0x7ff14c06cb50 msgr2=0x7ff14c06cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:47.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.390+0000 7ff151a77640 1 -- 172.21.15.88:0/2202514368 shutdown_connections 2024-09-16T01:43:47.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.390+0000 7ff151a77640 1 -- 172.21.15.88:0/2202514368 wait complete. 2024-09-16T01:43:47.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.391+0000 7ff151a77640 1 Processor -- start 2024-09-16T01:43:47.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.391+0000 7ff151a77640 1 -- start start 2024-09-16T01:43:47.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.392+0000 7ff151a77640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff14c072470 0x7ff14c152380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:47.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.392+0000 7ff151a77640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff14c158a00 0x7ff14c0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:47.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.392+0000 7ff151a77640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff14c158de0 0x7ff14c153c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:47.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.392+0000 7ff151a77640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff14c0b5ca0 con 0x7ff14c158a00 2024-09-16T01:43:47.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.392+0000 7ff151a77640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff14c0b5b20 con 0x7ff14c072470 2024-09-16T01:43:47.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.392+0000 7ff151a77640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff14c06faf0 con 0x7ff14c158de0 2024-09-16T01:43:47.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.392+0000 7ff143fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff14c158a00 0x7ff14c0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:47.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.392+0000 7ff143fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff14c158a00 0x7ff14c0cc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35918/0 (socket says 172.21.15.88:35918) 2024-09-16T01:43:47.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.392+0000 7ff143fff640 1 -- 172.21.15.88:0/3017244627 learned_addr learned my addr 172.21.15.88:0/3017244627 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:47.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.392+0000 7ff14b7fe640 1 --2- 172.21.15.88:0/3017244627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff14c072470 0x7ff14c152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:47.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.392+0000 7ff14bfff640 1 --2- 172.21.15.88:0/3017244627 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff14c158de0 0x7ff14c153c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:47.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.392+0000 7ff143fff640 1 -- 172.21.15.88:0/3017244627 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff14c158de0 msgr2=0x7ff14c153c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:47.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.392+0000 7ff143fff640 1 --2- 172.21.15.88:0/3017244627 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff14c158de0 0x7ff14c153c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.392+0000 7ff143fff640 1 -- 172.21.15.88:0/3017244627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff14c072470 msgr2=0x7ff14c152380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:47.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.392+0000 7ff143fff640 1 --2- 172.21.15.88:0/3017244627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff14c072470 0x7ff14c152380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.392+0000 7ff143fff640 1 -- 172.21.15.88:0/3017244627 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff14c14f860 con 0x7ff14c158a00 2024-09-16T01:43:47.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.392+0000 7ff14b7fe640 1 --2- 172.21.15.88:0/3017244627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff14c072470 0x7ff14c152380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:47.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.393+0000 7ff143fff640 1 --2- 172.21.15.88:0/3017244627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff14c158a00 0x7ff14c0cc600 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7ff13400e840 tx=0x7ff13400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:47.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.393+0000 7ff148ff9640 1 -- 172.21.15.88:0/3017244627 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13400f040 con 0x7ff14c158a00 2024-09-16T01:43:47.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.393+0000 7ff148ff9640 1 -- 172.21.15.88:0/3017244627 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff134004230 con 0x7ff14c158a00 2024-09-16T01:43:47.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.393+0000 7ff151a77640 1 -- 172.21.15.88:0/3017244627 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff14c14fb40 con 0x7ff14c158a00 2024-09-16T01:43:47.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.393+0000 7ff148ff9640 1 -- 172.21.15.88:0/3017244627 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13400c670 con 0x7ff14c158a00 2024-09-16T01:43:47.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.393+0000 7ff151a77640 1 -- 172.21.15.88:0/3017244627 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff14c150100 con 0x7ff14c158a00 2024-09-16T01:43:47.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.394+0000 7ff151a77640 1 -- 172.21.15.88:0/3017244627 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff14c071f30 con 0x7ff14c158a00 2024-09-16T01:43:47.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.394+0000 7ff148ff9640 1 -- 172.21.15.88:0/3017244627 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff13400c810 con 0x7ff14c158a00 2024-09-16T01:43:47.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.395+0000 7ff148ff9640 1 --2- 172.21.15.88:0/3017244627 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff12c06c2b0 0x7ff12c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:47.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.395+0000 7ff148ff9640 1 -- 172.21.15.88:0/3017244627 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8662+0+0 (secure 0 0 0) 0x7ff13408bc50 con 0x7ff14c158a00 2024-09-16T01:43:47.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.395+0000 7ff14b7fe640 1 --2- 172.21.15.88:0/3017244627 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff12c06c2b0 0x7ff12c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:47.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.395+0000 7ff14b7fe640 1 --2- 172.21.15.88:0/3017244627 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff12c06c2b0 0x7ff12c06e770 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7ff13c034000 tx=0x7ff13c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:47.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.399+0000 7ff148ff9640 1 -- 172.21.15.88:0/3017244627 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff134091020 con 0x7ff14c158a00 2024-09-16T01:43:47.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.600+0000 7ff151a77640 1 -- 172.21.15.88:0/3017244627 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7ff14c0b6070 con 0x7ff14c158a00 2024-09-16T01:43:47.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.876+0000 7ff148ff9640 1 -- 172.21.15.88:0/3017244627 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v278) ==== 162+0+0 (secure 0 0 0) 0x7ff134017020 con 0x7ff14c158a00 2024-09-16T01:43:47.878 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:43:47.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.878+0000 7ff151a77640 1 -- 172.21.15.88:0/3017244627 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff12c06c2b0 msgr2=0x7ff12c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:47.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.878+0000 7ff151a77640 1 --2- 172.21.15.88:0/3017244627 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff12c06c2b0 0x7ff12c06e770 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7ff13c034000 tx=0x7ff13c02e000 comp rx=0 tx=0).stop 2024-09-16T01:43:47.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.878+0000 7ff151a77640 1 -- 172.21.15.88:0/3017244627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff14c158a00 msgr2=0x7ff14c0cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:47.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.878+0000 7ff151a77640 1 --2- 172.21.15.88:0/3017244627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff14c158a00 0x7ff14c0cc600 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7ff13400e840 tx=0x7ff13400ed00 comp rx=0 tx=0).stop 2024-09-16T01:43:47.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.878+0000 7ff151a77640 1 -- 172.21.15.88:0/3017244627 shutdown_connections 2024-09-16T01:43:47.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.878+0000 7ff151a77640 1 --2- 172.21.15.88:0/3017244627 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff12c06c2b0 0x7ff12c06e770 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.878+0000 7ff151a77640 1 --2- 172.21.15.88:0/3017244627 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff14c158de0 0x7ff14c153c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.878+0000 7ff151a77640 1 --2- 172.21.15.88:0/3017244627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff14c158a00 0x7ff14c0cc600 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.878+0000 7ff151a77640 1 --2- 172.21.15.88:0/3017244627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff14c072470 0x7ff14c152380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:47.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.878+0000 7ff151a77640 1 -- 172.21.15.88:0/3017244627 >> 172.21.15.88:0/3017244627 conn(0x7ff14c06cb50 msgr2=0x7ff14c0af9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:47.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.878+0000 7ff151a77640 1 -- 172.21.15.88:0/3017244627 shutdown_connections 2024-09-16T01:43:47.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:47.878+0000 7ff151a77640 1 -- 172.21.15.88:0/3017244627 wait complete. 2024-09-16T01:43:48.296 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:43:48.397 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:43:48.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.394+0000 7f6f473e4640 1 Processor -- start 2024-09-16T01:43:48.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.395+0000 7f6f473e4640 1 -- start start 2024-09-16T01:43:48.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.395+0000 7f6f473e4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6f400d3310 0x7f6f400d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:48.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.395+0000 7f6f473e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f400d1650 0x7f6f400b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:48.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.395+0000 7f6f473e4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f4015ea40 con 0x7f6f4015e660 2024-09-16T01:43:48.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.395+0000 7f6f473e4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6f400d1a90 con 0x7f6f400d3310 2024-09-16T01:43:48.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.395+0000 7f6f473e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6f400d3870 con 0x7f6f400d1650 2024-09-16T01:43:48.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f45180640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6f4015e660 0x7f6f400cf730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:42464/0 (socket says 172.21.15.88:42464) 2024-09-16T01:43:48.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f45180640 1 -- 172.21.15.88:0/3902074019 learned_addr learned my addr 172.21.15.88:0/3902074019 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:48.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f45981640 1 --2- 172.21.15.88:0/3902074019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f400d1650 0x7f6f400b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:48.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f4497f640 1 --2- 172.21.15.88:0/3902074019 >> v2:172.21.15.154:3300/0 conn(0x7f6f400d3310 0x7f6f400d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:48.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f37fff640 1 -- 172.21.15.88:0/3902074019 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1595385171 0 0) 0x7f6f4015ea40 con 0x7f6f4015e660 2024-09-16T01:43:48.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f37fff640 1 -- 172.21.15.88:0/3902074019 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f2c003520 con 0x7f6f4015e660 2024-09-16T01:43:48.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f37fff640 1 -- 172.21.15.88:0/3902074019 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 293220485 0 0) 0x7f6f2c003520 con 0x7f6f4015e660 2024-09-16T01:43:48.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f37fff640 1 -- 172.21.15.88:0/3902074019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f400d1650 msgr2=0x7f6f400b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:48.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f37fff640 1 --2- 172.21.15.88:0/3902074019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f400d1650 0x7f6f400b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:48.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f37fff640 1 -- 172.21.15.88:0/3902074019 >> v2:172.21.15.154:3300/0 conn(0x7f6f400d3310 msgr2=0x7f6f400d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:48.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f37fff640 1 --2- 172.21.15.88:0/3902074019 >> v2:172.21.15.154:3300/0 conn(0x7f6f400d3310 0x7f6f400d1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:48.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f37fff640 1 -- 172.21.15.88:0/3902074019 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f400d3b50 con 0x7f6f4015e660 2024-09-16T01:43:48.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f37fff640 1 -- 172.21.15.88:0/3902074019 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f6f3c001670 con 0x7f6f4015e660 2024-09-16T01:43:48.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f37fff640 1 -- 172.21.15.88:0/3902074019 >> v1:172.21.15.88:6789/0 conn(0x7f6f4015e660 legacy=0x7f6f400cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:48.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f37fff640 1 --2- 172.21.15.88:0/3902074019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f400d1650 0x7f6f2c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:48.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f37fff640 1 --2- 172.21.15.88:0/3902074019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f2c003f30 0x7f6f2c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:48.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f37fff640 1 --2- 172.21.15.88:0/3902074019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f2c004d40 0x7f6f2c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:48.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f37fff640 1 -- 172.21.15.88:0/3902074019 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6f400d1a90 con 0x7f6f2c003f30 2024-09-16T01:43:48.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f37fff640 1 -- 172.21.15.88:0/3902074019 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6f400d3870 con 0x7f6f2c004d40 2024-09-16T01:43:48.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f37fff640 1 -- 172.21.15.88:0/3902074019 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6f2c00b9c0 con 0x7f6f400d1650 2024-09-16T01:43:48.402 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:43:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f4497f640 1 --2- 172.21.15.88:0/3902074019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f2c003f30 0x7f6f2c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f45981640 1 --2- 172.21.15.88:0/3902074019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f2c004d40 0x7f6f2c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f45180640 1 --2- 172.21.15.88:0/3902074019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f400d1650 0x7f6f2c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f4497f640 1 -- 172.21.15.88:0/3902074019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f400d1650 msgr2=0x7f6f2c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f4497f640 1 --2- 172.21.15.88:0/3902074019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f400d1650 0x7f6f2c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f4497f640 1 -- 172.21.15.88:0/3902074019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f2c004d40 msgr2=0x7f6f2c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f4497f640 1 --2- 172.21.15.88:0/3902074019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f2c004d40 0x7f6f2c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f4497f640 1 -- 172.21.15.88:0/3902074019 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f2c003520 con 0x7f6f2c003f30 2024-09-16T01:43:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.396+0000 7f6f45981640 1 --2- 172.21.15.88:0/3902074019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f2c004d40 0x7f6f2c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.397+0000 7f6f4497f640 1 --2- 172.21.15.88:0/3902074019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f2c003f30 0x7f6f2c004800 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f6f30009ac0 tx=0x7f6f2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=497aaa270288763e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.397+0000 7f6f37fff640 1 -- 172.21.15.88:0/3902074019 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f30036070 con 0x7f6f2c003f30 2024-09-16T01:43:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.397+0000 7f6f37fff640 1 -- 172.21.15.88:0/3902074019 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f30031400 con 0x7f6f2c003f30 2024-09-16T01:43:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.397+0000 7f6f37fff640 1 -- 172.21.15.88:0/3902074019 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f30009690 con 0x7f6f2c003f30 2024-09-16T01:43:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.398+0000 7f6f473e4640 1 -- 172.21.15.88:0/3902074019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f2c003f30 msgr2=0x7f6f2c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.398+0000 7f6f473e4640 1 --2- 172.21.15.88:0/3902074019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f2c003f30 0x7f6f2c004800 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f6f30009ac0 tx=0x7f6f2c005830 comp rx=0 tx=0).stop 2024-09-16T01:43:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.398+0000 7f6f45981640 1 -- 172.21.15.88:0/3902074019 reap_dead start 2024-09-16T01:43:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.398+0000 7f6f473e4640 1 -- 172.21.15.88:0/3902074019 shutdown_connections 2024-09-16T01:43:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.398+0000 7f6f473e4640 1 -- 172.21.15.88:0/3902074019 >> 172.21.15.88:0/3902074019 conn(0x7f6f400b6410 msgr2=0x7f6f400b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.398+0000 7f6f473e4640 1 -- 172.21.15.88:0/3902074019 shutdown_connections 2024-09-16T01:43:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.398+0000 7f6f473e4640 1 -- 172.21.15.88:0/3902074019 wait complete. 2024-09-16T01:43:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.399+0000 7f6f473e4640 1 Processor -- start 2024-09-16T01:43:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.399+0000 7f6f473e4640 1 -- start start 2024-09-16T01:43:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.400+0000 7f6f473e4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f2c003f30 0x7f6f401f1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.400+0000 7f6f473e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f4015fcd0 0x7f6f40160150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.400+0000 7f6f473e4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f401518f0 0x7f6f40151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.400+0000 7f6f473e4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6f400d43f0 con 0x7f6f2c003f30 2024-09-16T01:43:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.400+0000 7f6f473e4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6f400d4110 con 0x7f6f401518f0 2024-09-16T01:43:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.400+0000 7f6f473e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6f400b7b70 con 0x7f6f4015fcd0 2024-09-16T01:43:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.400+0000 7f6f45180640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f2c003f30 0x7f6f401f1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.400+0000 7f6f45180640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f2c003f30 0x7f6f401f1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35942/0 (socket says 172.21.15.88:35942) 2024-09-16T01:43:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.400+0000 7f6f45180640 1 -- 172.21.15.88:0/2289488057 learned_addr learned my addr 172.21.15.88:0/2289488057 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.400+0000 7f6f45981640 1 --2- 172.21.15.88:0/2289488057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f401518f0 0x7f6f40151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.400+0000 7f6f4497f640 1 --2- 172.21.15.88:0/2289488057 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f4015fcd0 0x7f6f40160150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.401+0000 7f6f45180640 1 -- 172.21.15.88:0/2289488057 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f4015fcd0 msgr2=0x7f6f40160150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.401+0000 7f6f45180640 1 --2- 172.21.15.88:0/2289488057 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f4015fcd0 0x7f6f40160150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.401+0000 7f6f45180640 1 -- 172.21.15.88:0/2289488057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f401518f0 msgr2=0x7f6f40151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.401+0000 7f6f45180640 1 --2- 172.21.15.88:0/2289488057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f401518f0 0x7f6f40151d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.401+0000 7f6f45180640 1 -- 172.21.15.88:0/2289488057 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f40160790 con 0x7f6f2c003f30 2024-09-16T01:43:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.401+0000 7f6f45180640 1 --2- 172.21.15.88:0/2289488057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f2c003f30 0x7f6f401f1230 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c001860 tx=0x7f6f3c00ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.401+0000 7f6f35ffb640 1 -- 172.21.15.88:0/2289488057 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f3c00ee60 con 0x7f6f2c003f30 2024-09-16T01:43:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.401+0000 7f6f35ffb640 1 -- 172.21.15.88:0/2289488057 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f3c018070 con 0x7f6f2c003f30 2024-09-16T01:43:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.401+0000 7f6f473e4640 1 -- 172.21.15.88:0/2289488057 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f40160a70 con 0x7f6f2c003f30 2024-09-16T01:43:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.401+0000 7f6f35ffb640 1 -- 172.21.15.88:0/2289488057 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f3c00cb10 con 0x7f6f2c003f30 2024-09-16T01:43:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.401+0000 7f6f473e4640 1 -- 172.21.15.88:0/2289488057 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f400c8500 con 0x7f6f2c003f30 2024-09-16T01:43:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.402+0000 7f6f473e4640 1 -- 172.21.15.88:0/2289488057 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f400c8700 con 0x7f6f2c003f30 2024-09-16T01:43:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.403+0000 7f6f35ffb640 1 -- 172.21.15.88:0/2289488057 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6f3c00ccb0 con 0x7f6f2c003f30 2024-09-16T01:43:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.403+0000 7f6f35ffb640 1 --2- 172.21.15.88:0/2289488057 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6f2406c2b0 0x7f6f2406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.403+0000 7f6f35ffb640 1 -- 172.21.15.88:0/2289488057 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8662+0+0 (secure 0 0 0) 0x7f6f3c08bd80 con 0x7f6f2c003f30 2024-09-16T01:43:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.403+0000 7f6f4497f640 1 --2- 172.21.15.88:0/2289488057 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6f2406c2b0 0x7f6f2406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.404+0000 7f6f4497f640 1 --2- 172.21.15.88:0/2289488057 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6f2406c2b0 0x7f6f2406e770 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f6f30033000 tx=0x7f6f30008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:48.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.408+0000 7f6f35ffb640 1 -- 172.21.15.88:0/2289488057 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6f400c8700 con 0x7f6f2c003f30 2024-09-16T01:43:48.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.612+0000 7f6f473e4640 1 -- 172.21.15.88:0/2289488057 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f400b98a0 con 0x7f6f2c003f30 2024-09-16T01:43:48.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.612+0000 7f6f35ffb640 1 -- 172.21.15.88:0/2289488057 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 279 v279) ==== 95+0+5227 (secure 0 0 0) 0x7f6f3c0592c0 con 0x7f6f2c003f30 2024-09-16T01:43:48.615 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:48.615 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":279,"btime":"2024-09-16T01:43:47:880211+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9696,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4270197461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4270197461},{"type":"v1","addr":"172.21.15.88:6837","nonce":4270197461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":279,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:43:47.880197+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9954},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9954":{"gid":9954,"name":"a","rank":0,"incarnation":279,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6836/3892583353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3892583353},{"type":"v1","addr":"172.21.15.88:6836","nonce":3892583353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:43:48.615 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 279 2024-09-16T01:43:48.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.615+0000 7f6f473e4640 1 -- 172.21.15.88:0/2289488057 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6f2406c2b0 msgr2=0x7f6f2406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:48.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.615+0000 7f6f473e4640 1 --2- 172.21.15.88:0/2289488057 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6f2406c2b0 0x7f6f2406e770 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f6f30033000 tx=0x7f6f30008030 comp rx=0 tx=0).stop 2024-09-16T01:43:48.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.615+0000 7f6f473e4640 1 -- 172.21.15.88:0/2289488057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f2c003f30 msgr2=0x7f6f401f1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:48.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.615+0000 7f6f473e4640 1 --2- 172.21.15.88:0/2289488057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f2c003f30 0x7f6f401f1230 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c001860 tx=0x7f6f3c00ec10 comp rx=0 tx=0).stop 2024-09-16T01:43:48.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.615+0000 7f6f473e4640 1 -- 172.21.15.88:0/2289488057 shutdown_connections 2024-09-16T01:43:48.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.615+0000 7f6f473e4640 1 --2- 172.21.15.88:0/2289488057 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6f2406c2b0 0x7f6f2406e770 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:48.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.615+0000 7f6f473e4640 1 --2- 172.21.15.88:0/2289488057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f401518f0 0x7f6f40151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:48.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.615+0000 7f6f473e4640 1 --2- 172.21.15.88:0/2289488057 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f4015fcd0 0x7f6f40160150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:48.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.615+0000 7f6f473e4640 1 --2- 172.21.15.88:0/2289488057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f2c003f30 0x7f6f401f1230 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:48.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.615+0000 7f6f473e4640 1 -- 172.21.15.88:0/2289488057 >> 172.21.15.88:0/2289488057 conn(0x7f6f400b6410 msgr2=0x7f6f400b8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:48.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.615+0000 7f6f473e4640 1 -- 172.21.15.88:0/2289488057 shutdown_connections 2024-09-16T01:43:48.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:48.615+0000 7f6f473e4640 1 -- 172.21.15.88:0/2289488057 wait complete. 2024-09-16T01:43:48.628 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 279, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:40:03.597796+0000', 'modified': '2024-09-16T01:43:47.880197+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 121, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9954}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9954': {'gid': 9954, 'name': 'a', 'rank': 0, 'incarnation': 279, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.88:6836/3892583353', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 3892583353}, {'type': 'v1', 'addr': '172.21.15.88:6836', 'nonce': 3892583353}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-16T01:43:48.628 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9954:up:replay 2024-09-16T01:43:48.628 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-09-16T01:43:51.884 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:43:51.882+0000 7f5e6dfb8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:43:54.033 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:43:54.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.141+0000 7fca0d4d0640 1 Processor -- start 2024-09-16T01:43:54.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.141+0000 7fca0d4d0640 1 -- start start 2024-09-16T01:43:54.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.142+0000 7fca0d4d0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fca080b4d00 0x7fca080b5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.142+0000 7fca0d4d0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fca080cd780 0x7fca080cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.142+0000 7fca0d4d0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca080ce4f0 con 0x7fca080b4920 2024-09-16T01:43:54.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.142+0000 7fca0d4d0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fca080cf100 con 0x7fca080cd780 2024-09-16T01:43:54.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.142+0000 7fca0d4d0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fca080ce7d0 con 0x7fca080b4d00 2024-09-16T01:43:54.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.142+0000 7fca077fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fca080cd780 0x7fca080cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.142+0000 7fca077fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fca080cd780 0x7fca080cdb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38110/0 (socket says 172.21.15.88:38110) 2024-09-16T01:43:54.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.142+0000 7fca077fe640 1 -- 172.21.15.88:0/3432693084 learned_addr learned my addr 172.21.15.88:0/3432693084 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:54.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.142+0000 7fc9fe7fc640 1 --2- 172.21.15.88:0/3432693084 >> v2:172.21.15.154:3300/0 conn(0x7fca080b4d00 0x7fca080b5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.142+0000 7fca077fe640 1 -- 172.21.15.88:0/3432693084 >> v2:172.21.15.154:3300/0 conn(0x7fca080b4d00 msgr2=0x7fca080b5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:54.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.142+0000 7fca077fe640 1 --2- 172.21.15.88:0/3432693084 >> v2:172.21.15.154:3300/0 conn(0x7fca080b4d00 0x7fca080b5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.142+0000 7fca077fe640 1 -- 172.21.15.88:0/3432693084 >> v1:172.21.15.88:6789/0 conn(0x7fca080b4920 legacy=0x7fca0815b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:54.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.142+0000 7fca067fc640 1 -- 172.21.15.88:0/3432693084 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 848957912 0 0) 0x7fca080ce4f0 con 0x7fca080b4920 2024-09-16T01:43:54.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.142+0000 7fca077fe640 1 -- 172.21.15.88:0/3432693084 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca080ceab0 con 0x7fca080cd780 2024-09-16T01:43:54.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.143+0000 7fca077fe640 1 --2- 172.21.15.88:0/3432693084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca080cd780 0x7fca080cdb90 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7fca080b29e0 tx=0x7fca0002c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3056ed03746a0265 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:54.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.143+0000 7fca067fc640 1 -- 172.21.15.88:0/3432693084 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0000b2e0 con 0x7fca080cd780 2024-09-16T01:43:54.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.143+0000 7fca067fc640 1 -- 172.21.15.88:0/3432693084 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca00037070 con 0x7fca080cd780 2024-09-16T01:43:54.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.143+0000 7fca067fc640 1 -- 172.21.15.88:0/3432693084 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0000b480 con 0x7fca080cd780 2024-09-16T01:43:54.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.144+0000 7fca0d4d0640 1 -- 172.21.15.88:0/3432693084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca080cd780 msgr2=0x7fca080cdb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:54.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.144+0000 7fca0d4d0640 1 --2- 172.21.15.88:0/3432693084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca080cd780 0x7fca080cdb90 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7fca080b29e0 tx=0x7fca0002c940 comp rx=0 tx=0).stop 2024-09-16T01:43:54.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.144+0000 7fca0d4d0640 1 -- 172.21.15.88:0/3432693084 shutdown_connections 2024-09-16T01:43:54.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.144+0000 7fca0d4d0640 1 --2- 172.21.15.88:0/3432693084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca080cd780 0x7fca080cdb90 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.144+0000 7fca0d4d0640 1 --2- 172.21.15.88:0/3432693084 >> v2:172.21.15.154:3300/0 conn(0x7fca080b4d00 0x7fca080b5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.144+0000 7fca0d4d0640 1 -- 172.21.15.88:0/3432693084 >> 172.21.15.88:0/3432693084 conn(0x7fca0806bc10 msgr2=0x7fca0806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:54.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.144+0000 7fca0d4d0640 1 -- 172.21.15.88:0/3432693084 shutdown_connections 2024-09-16T01:43:54.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.144+0000 7fca0d4d0640 1 -- 172.21.15.88:0/3432693084 wait complete. 2024-09-16T01:43:54.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.145+0000 7fca0d4d0640 1 Processor -- start 2024-09-16T01:43:54.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.145+0000 7fca0d4d0640 1 -- start start 2024-09-16T01:43:54.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.146+0000 7fca0d4d0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca080b4920 0x7fca0815aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.146+0000 7fca0d4d0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca080b4d00 0x7fca08153080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.146+0000 7fca0d4d0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca080cd780 0x7fca08155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.146+0000 7fca0d4d0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fca080ce0d0 con 0x7fca080b4d00 2024-09-16T01:43:54.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.146+0000 7fca0d4d0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fca080b56b0 con 0x7fca080b4920 2024-09-16T01:43:54.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.146+0000 7fca0d4d0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fca080ce250 con 0x7fca080cd780 2024-09-16T01:43:54.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.146+0000 7fc9fe7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca080b4d00 0x7fca08153080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.146+0000 7fc9fe7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca080b4d00 0x7fca08153080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38118/0 (socket says 172.21.15.88:38118) 2024-09-16T01:43:54.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.146+0000 7fc9fe7fc640 1 -- 172.21.15.88:0/3672327970 learned_addr learned my addr 172.21.15.88:0/3672327970 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:54.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.146+0000 7fca06ffd640 1 --2- 172.21.15.88:0/3672327970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca080b4920 0x7fca0815aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.146+0000 7fc9fe7fc640 1 -- 172.21.15.88:0/3672327970 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca080cd780 msgr2=0x7fca08155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:54.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.146+0000 7fc9fe7fc640 1 --2- 172.21.15.88:0/3672327970 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca080cd780 0x7fca08155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.146+0000 7fc9fe7fc640 1 -- 172.21.15.88:0/3672327970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca080b4920 msgr2=0x7fca0815aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:54.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.146+0000 7fc9fe7fc640 1 --2- 172.21.15.88:0/3672327970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca080b4920 0x7fca0815aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.146+0000 7fc9fe7fc640 1 -- 172.21.15.88:0/3672327970 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca08151a30 con 0x7fca080b4d00 2024-09-16T01:43:54.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.146+0000 7fca06ffd640 1 --2- 172.21.15.88:0/3672327970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca080b4920 0x7fca0815aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:54.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.147+0000 7fc9fe7fc640 1 --2- 172.21.15.88:0/3672327970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca080b4d00 0x7fca08153080 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7fc9f000e9b0 tx=0x7fc9f000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:54.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.147+0000 7fc9fffff640 1 -- 172.21.15.88:0/3672327970 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f0017070 con 0x7fca080b4d00 2024-09-16T01:43:54.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.147+0000 7fc9fffff640 1 -- 172.21.15.88:0/3672327970 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9f000c9a0 con 0x7fca080b4d00 2024-09-16T01:43:54.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.147+0000 7fca0d4d0640 1 -- 172.21.15.88:0/3672327970 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca08151d10 con 0x7fca080b4d00 2024-09-16T01:43:54.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.147+0000 7fc9fffff640 1 -- 172.21.15.88:0/3672327970 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f000cd90 con 0x7fca080b4d00 2024-09-16T01:43:54.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.147+0000 7fca0d4d0640 1 -- 172.21.15.88:0/3672327970 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca081522d0 con 0x7fca080b4d00 2024-09-16T01:43:54.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.148+0000 7fca0d4d0640 1 -- 172.21.15.88:0/3672327970 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca080b5170 con 0x7fca080b4d00 2024-09-16T01:43:54.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.148+0000 7fc9fffff640 1 -- 172.21.15.88:0/3672327970 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc9f001a070 con 0x7fca080b4d00 2024-09-16T01:43:54.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.149+0000 7fc9fffff640 1 --2- 172.21.15.88:0/3672327970 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc9e806c2b0 0x7fc9e806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.149+0000 7fca06ffd640 1 --2- 172.21.15.88:0/3672327970 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc9e806c2b0 0x7fc9e806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.149+0000 7fc9fffff640 1 -- 172.21.15.88:0/3672327970 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8662+0+0 (secure 0 0 0) 0x7fc9f008c4a0 con 0x7fca080b4d00 2024-09-16T01:43:54.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.149+0000 7fca06ffd640 1 --2- 172.21.15.88:0/3672327970 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc9e806c2b0 0x7fc9e806e770 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fc9f8000f60 tx=0x7fc9f8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:54.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.153+0000 7fc9fffff640 1 -- 172.21.15.88:0/3672327970 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc9f0059fa0 con 0x7fca080b4d00 2024-09-16T01:43:54.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.352+0000 7fca0d4d0640 1 -- 172.21.15.88:0/3672327970 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca08072d60 con 0x7fca080b4d00 2024-09-16T01:43:54.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.353+0000 7fc9fffff640 1 -- 172.21.15.88:0/3672327970 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 282 v282) ==== 95+0+5234 (secure 0 0 0) 0x7fc9f00598a0 con 0x7fca080b4d00 2024-09-16T01:43:54.355 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:54.356 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":282,"btime":"2024-09-16T01:43:50:896967+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9696,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4270197461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4270197461},{"type":"v1","addr":"172.21.15.88:6837","nonce":4270197461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":282,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:43:50.896961+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":121,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9954},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9954":{"gid":9954,"name":"a","rank":0,"incarnation":279,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6836/3892583353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3892583353},{"type":"v1","addr":"172.21.15.88:6836","nonce":3892583353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9954,"qdb_cluster":[9954]},"id":6}]} 2024-09-16T01:43:54.356 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 282 2024-09-16T01:43:54.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.355+0000 7fca0d4d0640 1 -- 172.21.15.88:0/3672327970 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc9e806c2b0 msgr2=0x7fc9e806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:54.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.355+0000 7fca0d4d0640 1 --2- 172.21.15.88:0/3672327970 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc9e806c2b0 0x7fc9e806e770 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fc9f8000f60 tx=0x7fc9f8009040 comp rx=0 tx=0).stop 2024-09-16T01:43:54.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.355+0000 7fca0d4d0640 1 -- 172.21.15.88:0/3672327970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca080b4d00 msgr2=0x7fca08153080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:54.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.355+0000 7fca0d4d0640 1 --2- 172.21.15.88:0/3672327970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca080b4d00 0x7fca08153080 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7fc9f000e9b0 tx=0x7fc9f000ee70 comp rx=0 tx=0).stop 2024-09-16T01:43:54.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.355+0000 7fca0d4d0640 1 -- 172.21.15.88:0/3672327970 shutdown_connections 2024-09-16T01:43:54.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.355+0000 7fca0d4d0640 1 --2- 172.21.15.88:0/3672327970 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc9e806c2b0 0x7fc9e806e770 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.355+0000 7fca0d4d0640 1 --2- 172.21.15.88:0/3672327970 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca080cd780 0x7fca08155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.355+0000 7fca0d4d0640 1 --2- 172.21.15.88:0/3672327970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca080b4d00 0x7fca08153080 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.355+0000 7fca0d4d0640 1 --2- 172.21.15.88:0/3672327970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca080b4920 0x7fca0815aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.355+0000 7fca0d4d0640 1 -- 172.21.15.88:0/3672327970 >> 172.21.15.88:0/3672327970 conn(0x7fca0806bc10 msgr2=0x7fca08072330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:54.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.355+0000 7fca0d4d0640 1 -- 172.21.15.88:0/3672327970 shutdown_connections 2024-09-16T01:43:54.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.355+0000 7fca0d4d0640 1 -- 172.21.15.88:0/3672327970 wait complete. 2024-09-16T01:43:54.369 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 282, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:40:03.597796+0000', 'modified': '2024-09-16T01:43:50.896961+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 121, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9954}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9954': {'gid': 9954, 'name': 'a', 'rank': 0, 'incarnation': 279, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.88:6836/3892583353', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 3892583353}, {'type': 'v1', 'addr': '172.21.15.88:6836', 'nonce': 3892583353}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9954, 'qdb_cluster': [9954]} 2024-09-16T01:43:54.369 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T01:43:54.369 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-16T01:43:54.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.466+0000 7fc2fe59f640 1 Processor -- start 2024-09-16T01:43:54.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.466+0000 7fc2fe59f640 1 -- start start 2024-09-16T01:43:54.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.467+0000 7fc2fe59f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc2f80d2b20 0x7fc2f80d0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.467+0000 7fc2fe59f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f80d3060 con 0x7fc2f80d11b0 2024-09-16T01:43:54.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.467+0000 7fc2fe59f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f80d3340 con 0x7fc2f815e800 2024-09-16T01:43:54.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.467+0000 7fc2fe59f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc2f80d3620 con 0x7fc2f80d2b20 2024-09-16T01:43:54.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.467+0000 7fc2fdd9e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc2f80d11b0 0x7fc2f80b7cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49284/0 (socket says 172.21.15.88:49284) 2024-09-16T01:43:54.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.467+0000 7fc2fdd9e640 1 -- 172.21.15.88:0/1906102812 learned_addr learned my addr 172.21.15.88:0/1906102812 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:54.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.467+0000 7fc2fcd9c640 1 --2- 172.21.15.88:0/1906102812 >> v2:172.21.15.154:3300/0 conn(0x7fc2f80d2b20 0x7fc2f80d0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.467+0000 7fc2effff640 1 -- 172.21.15.88:0/1906102812 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3427404780 0 0) 0x7fc2f80d3060 con 0x7fc2f80d11b0 2024-09-16T01:43:54.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.467+0000 7fc2effff640 1 -- 172.21.15.88:0/1906102812 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2e4003520 con 0x7fc2f80d11b0 2024-09-16T01:43:54.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.468+0000 7fc2effff640 1 -- 172.21.15.88:0/1906102812 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1178701091 0 0) 0x7fc2e4003520 con 0x7fc2f80d11b0 2024-09-16T01:43:54.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.468+0000 7fc2effff640 1 -- 172.21.15.88:0/1906102812 >> v2:172.21.15.154:3300/0 conn(0x7fc2f80d2b20 msgr2=0x7fc2f80d0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:54.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.468+0000 7fc2effff640 1 --2- 172.21.15.88:0/1906102812 >> v2:172.21.15.154:3300/0 conn(0x7fc2f80d2b20 0x7fc2f80d0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.468+0000 7fc2effff640 1 -- 172.21.15.88:0/1906102812 >> v1:172.21.15.154:6789/0 conn(0x7fc2f815e800 legacy=0x7fc2f80d2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:54.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.468+0000 7fc2effff640 1 -- 172.21.15.88:0/1906102812 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f80d3900 con 0x7fc2f80d11b0 2024-09-16T01:43:54.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.468+0000 7fc2effff640 1 -- 172.21.15.88:0/1906102812 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fc2f00019f0 con 0x7fc2f80d11b0 2024-09-16T01:43:54.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.468+0000 7fc2effff640 1 -- 172.21.15.88:0/1906102812 >> v1:172.21.15.88:6789/0 conn(0x7fc2f80d11b0 legacy=0x7fc2f80b7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:54.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.468+0000 7fc2effff640 1 --2- 172.21.15.88:0/1906102812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2e40039b0 0x7fc2e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.468+0000 7fc2effff640 1 --2- 172.21.15.88:0/1906102812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2e4004800 0x7fc2e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.468+0000 7fc2effff640 1 --2- 172.21.15.88:0/1906102812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2e40051c0 0x7fc2e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.468+0000 7fc2effff640 1 -- 172.21.15.88:0/1906102812 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f80d3620 con 0x7fc2e4004800 2024-09-16T01:43:54.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.468+0000 7fc2effff640 1 -- 172.21.15.88:0/1906102812 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e400c060 con 0x7fc2e40039b0 2024-09-16T01:43:54.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.468+0000 7fc2effff640 1 -- 172.21.15.88:0/1906102812 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc2e400c260 con 0x7fc2e40051c0 2024-09-16T01:43:54.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.468+0000 7fc2fcd9c640 1 --2- 172.21.15.88:0/1906102812 >> v2:172.21.15.154:3300/0 conn(0x7fc2f80d2b20 0x7fc2f80d0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:43:54.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.468+0000 7fc2fcd9c640 1 --2- 172.21.15.88:0/1906102812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2e4004800 0x7fc2e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.468+0000 7fc2fd59d640 1 --2- 172.21.15.88:0/1906102812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2e40039b0 0x7fc2e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.468+0000 7fc2fdd9e640 1 --2- 172.21.15.88:0/1906102812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2e40051c0 0x7fc2e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.468+0000 7fc2fcd9c640 1 -- 172.21.15.88:0/1906102812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2e40051c0 msgr2=0x7fc2e400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:54.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.468+0000 7fc2fcd9c640 1 --2- 172.21.15.88:0/1906102812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2e40051c0 0x7fc2e400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.468+0000 7fc2fcd9c640 1 -- 172.21.15.88:0/1906102812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2e40039b0 msgr2=0x7fc2e4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:54.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.468+0000 7fc2fcd9c640 1 --2- 172.21.15.88:0/1906102812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2e40039b0 0x7fc2e4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.468+0000 7fc2fcd9c640 1 -- 172.21.15.88:0/1906102812 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2e4003520 con 0x7fc2e4004800 2024-09-16T01:43:54.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.468+0000 7fc2fdd9e640 1 -- 172.21.15.88:0/1906102812 reap_dead start 2024-09-16T01:43:54.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.469+0000 7fc2fcd9c640 1 --2- 172.21.15.88:0/1906102812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2e4004800 0x7fc2e4004c80 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8009c70 tx=0x7fc2e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2d0efd17b203b42 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:54.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.469+0000 7fc2effff640 1 -- 172.21.15.88:0/1906102812 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8009840 con 0x7fc2e4004800 2024-09-16T01:43:54.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.469+0000 7fc2effff640 1 -- 172.21.15.88:0/1906102812 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2e8004020 con 0x7fc2e4004800 2024-09-16T01:43:54.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.469+0000 7fc2effff640 1 -- 172.21.15.88:0/1906102812 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e80043d0 con 0x7fc2e4004800 2024-09-16T01:43:54.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.469+0000 7fc2fe59f640 1 -- 172.21.15.88:0/1906102812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2e4004800 msgr2=0x7fc2e4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:54.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.469+0000 7fc2fe59f640 1 --2- 172.21.15.88:0/1906102812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2e4004800 0x7fc2e4004c80 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8009c70 tx=0x7fc2e4005830 comp rx=0 tx=0).stop 2024-09-16T01:43:54.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.470+0000 7fc2fe59f640 1 -- 172.21.15.88:0/1906102812 shutdown_connections 2024-09-16T01:43:54.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.470+0000 7fc2fe59f640 1 --2- 172.21.15.88:0/1906102812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2e4004800 0x7fc2e4004c80 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.470+0000 7fc2fe59f640 1 -- 172.21.15.88:0/1906102812 >> 172.21.15.88:0/1906102812 conn(0x7fc2f8062630 msgr2=0x7fc2f8062a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:54.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.470+0000 7fc2fe59f640 1 -- 172.21.15.88:0/1906102812 shutdown_connections 2024-09-16T01:43:54.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.470+0000 7fc2fe59f640 1 -- 172.21.15.88:0/1906102812 wait complete. 2024-09-16T01:43:54.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.471+0000 7fc2fe59f640 1 Processor -- start 2024-09-16T01:43:54.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.471+0000 7fc2fe59f640 1 -- start start 2024-09-16T01:43:54.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.471+0000 7fc2fe59f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2e4004800 0x7fc2f8158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.472+0000 7fc2fe59f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2f81583f0 0x7fc2f81f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.472+0000 7fc2fe59f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2f815b230 0x7fc2f814d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.472+0000 7fc2fe59f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f815ebe0 con 0x7fc2f81583f0 2024-09-16T01:43:54.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.472+0000 7fc2fe59f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f80d1640 con 0x7fc2f815b230 2024-09-16T01:43:54.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.472+0000 7fc2fe59f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc2f80d0c70 con 0x7fc2e4004800 2024-09-16T01:43:54.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.472+0000 7fc2fcd9c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2f81583f0 0x7fc2f81f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.472+0000 7fc2fcd9c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2f81583f0 0x7fc2f81f43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38134/0 (socket says 172.21.15.88:38134) 2024-09-16T01:43:54.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.472+0000 7fc2fcd9c640 1 -- 172.21.15.88:0/3527270948 learned_addr learned my addr 172.21.15.88:0/3527270948 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:54.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.472+0000 7fc2fd59d640 1 --2- 172.21.15.88:0/3527270948 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2e4004800 0x7fc2f8158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.472+0000 7fc2fdd9e640 1 --2- 172.21.15.88:0/3527270948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2f815b230 0x7fc2f814d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.472+0000 7fc2fcd9c640 1 -- 172.21.15.88:0/3527270948 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2e4004800 msgr2=0x7fc2f8158e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:54.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.472+0000 7fc2fcd9c640 1 --2- 172.21.15.88:0/3527270948 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2e4004800 0x7fc2f8158e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.472+0000 7fc2fcd9c640 1 -- 172.21.15.88:0/3527270948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2f815b230 msgr2=0x7fc2f814d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:54.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.472+0000 7fc2fcd9c640 1 --2- 172.21.15.88:0/3527270948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2f815b230 0x7fc2f814d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.472+0000 7fc2fcd9c640 1 -- 172.21.15.88:0/3527270948 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f815a670 con 0x7fc2f81583f0 2024-09-16T01:43:54.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.472+0000 7fc2fcd9c640 1 --2- 172.21.15.88:0/3527270948 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2f81583f0 0x7fc2f81f43b0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8009c70 tx=0x7fc2e8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:54.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.472+0000 7fc2edffb640 1 -- 172.21.15.88:0/3527270948 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8040070 con 0x7fc2f81583f0 2024-09-16T01:43:54.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.472+0000 7fc2edffb640 1 -- 172.21.15.88:0/3527270948 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2e8003610 con 0x7fc2f81583f0 2024-09-16T01:43:54.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.472+0000 7fc2edffb640 1 -- 172.21.15.88:0/3527270948 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e80039c0 con 0x7fc2f81583f0 2024-09-16T01:43:54.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.472+0000 7fc2fe59f640 1 -- 172.21.15.88:0/3527270948 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f80cb860 con 0x7fc2f81583f0 2024-09-16T01:43:54.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.473+0000 7fc2fe59f640 1 -- 172.21.15.88:0/3527270948 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2f815b6b0 con 0x7fc2f81583f0 2024-09-16T01:43:54.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.474+0000 7fc2edffb640 1 -- 172.21.15.88:0/3527270948 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc2e8043070 con 0x7fc2f81583f0 2024-09-16T01:43:54.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.475+0000 7fc2edffb640 1 --2- 172.21.15.88:0/3527270948 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc2dc06c2b0 0x7fc2dc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.475+0000 7fc2edffb640 1 -- 172.21.15.88:0/3527270948 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8662+0+0 (secure 0 0 0) 0x7fc2e80ac950 con 0x7fc2f81583f0 2024-09-16T01:43:54.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.475+0000 7fc2fd59d640 1 --2- 172.21.15.88:0/3527270948 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc2dc06c2b0 0x7fc2dc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.475+0000 7fc2fd59d640 1 --2- 172.21.15.88:0/3527270948 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc2dc06c2b0 0x7fc2dc06e770 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4000c00 tx=0x7fc2f4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:54.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.476+0000 7fc302671740 1 Processor -- start 2024-09-16T01:43:54.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.476+0000 7fc302671740 1 -- start start 2024-09-16T01:43:54.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.476+0000 7fc302671740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56042606f4b0 0x5604260b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.476+0000 7fc302671740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042606cc70 0x56042606f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.476+0000 7fc302671740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560426070c40 0x560426069aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.476+0000 7fc302671740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x560425fd1520 con 0x560426070c40 2024-09-16T01:43:54.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.476+0000 7fc302671740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560425f793e0 con 0x56042606cc70 2024-09-16T01:43:54.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.476+0000 7fc302671740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x560425fcd4e0 con 0x56042606f4b0 2024-09-16T01:43:54.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.476+0000 7fc2fdd9e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560426070c40 0x560426069aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.476+0000 7fc2fdd9e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560426070c40 0x560426069aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38144/0 (socket says 172.21.15.88:38144) 2024-09-16T01:43:54.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.476+0000 7fc2fdd9e640 1 -- 172.21.15.88:0/2412030915 learned_addr learned my addr 172.21.15.88:0/2412030915 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:54.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.477+0000 7fc2fcd9c640 1 --2- 172.21.15.88:0/2412030915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042606cc70 0x56042606f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.477+0000 7fc2fd59d640 1 --2- 172.21.15.88:0/2412030915 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56042606f4b0 0x5604260b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.477+0000 7fc2fdd9e640 1 -- 172.21.15.88:0/2412030915 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56042606f4b0 msgr2=0x5604260b4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:54.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.477+0000 7fc2fdd9e640 1 --2- 172.21.15.88:0/2412030915 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56042606f4b0 0x5604260b4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.477+0000 7fc2fdd9e640 1 -- 172.21.15.88:0/2412030915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042606cc70 msgr2=0x56042606f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:54.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.477+0000 7fc2fdd9e640 1 --2- 172.21.15.88:0/2412030915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042606cc70 0x56042606f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.477+0000 7fc2fdd9e640 1 -- 172.21.15.88:0/2412030915 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560425f942f0 con 0x560426070c40 2024-09-16T01:43:54.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.477+0000 7fc2fdd9e640 1 --2- 172.21.15.88:0/2412030915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560426070c40 0x560426069aa0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0001990 tx=0x7fc2f0005f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b03f82e5725a08d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:54.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.477+0000 7fc2daffd640 1 -- 172.21.15.88:0/2412030915 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0007810 con 0x560426070c40 2024-09-16T01:43:54.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.477+0000 7fc2daffd640 1 -- 172.21.15.88:0/2412030915 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2f00079b0 con 0x560426070c40 2024-09-16T01:43:54.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.477+0000 7fc2daffd640 1 -- 172.21.15.88:0/2412030915 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0014640 con 0x560426070c40 2024-09-16T01:43:54.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.478+0000 7fc302671740 1 -- 172.21.15.88:0/2412030915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560426070c40 msgr2=0x560426069aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:54.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.478+0000 7fc302671740 1 --2- 172.21.15.88:0/2412030915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560426070c40 0x560426069aa0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0001990 tx=0x7fc2f0005f40 comp rx=0 tx=0).stop 2024-09-16T01:43:54.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.478+0000 7fc302671740 1 -- 172.21.15.88:0/2412030915 shutdown_connections 2024-09-16T01:43:54.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.478+0000 7fc302671740 1 --2- 172.21.15.88:0/2412030915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560426070c40 0x560426069aa0 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.491 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.478+0000 7fc302671740 1 --2- 172.21.15.88:0/2412030915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042606cc70 0x56042606f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.478+0000 7fc302671740 1 --2- 172.21.15.88:0/2412030915 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56042606f4b0 0x5604260b4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.478+0000 7fc302671740 1 -- 172.21.15.88:0/2412030915 >> 172.21.15.88:0/2412030915 conn(0x56042606bc70 msgr2=0x5604260b4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:54.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.478+0000 7fc302671740 1 -- 172.21.15.88:0/2412030915 shutdown_connections 2024-09-16T01:43:54.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.478+0000 7fc302671740 1 -- 172.21.15.88:0/2412030915 wait complete. 2024-09-16T01:43:54.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.478+0000 7fc302671740 1 -- start start 2024-09-16T01:43:54.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.478+0000 7fc302671740 1 Processor -- start 2024-09-16T01:43:54.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.479+0000 7fc302671740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042606f4b0 0x5604260979c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.479+0000 7fc302671740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560426070c40 0x560426099810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.479+0000 7fc302671740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56042608f7a0 0x5604260adf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.479+0000 7fc302671740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56042606b110 con 0x56042608f7a0 2024-09-16T01:43:54.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.479+0000 7fc302671740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560425f5a450 con 0x56042606f4b0 2024-09-16T01:43:54.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.479+0000 7fc302671740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5604260b5480 con 0x560426070c40 2024-09-16T01:43:54.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.479+0000 7fc302671740 10 client.0 _ll_register_callbacks cb 0x560425ce3ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:43:54.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.479+0000 7fc2fdd9e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56042608f7a0 0x5604260adf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.479+0000 7fc2fdd9e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56042608f7a0 0x5604260adf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38154/0 (socket says 172.21.15.88:38154) 2024-09-16T01:43:54.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.479+0000 7fc2fdd9e640 1 -- 172.21.15.88:0/347345093 learned_addr learned my addr 172.21.15.88:0/347345093 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:54.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.479+0000 7fc2fd59d640 1 --2- 172.21.15.88:0/347345093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042606f4b0 0x5604260979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.479+0000 7fc2fcd9c640 1 --2- 172.21.15.88:0/347345093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560426070c40 0x560426099810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.480+0000 7fc2fdd9e640 1 -- 172.21.15.88:0/347345093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560426070c40 msgr2=0x560426099810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:54.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.480+0000 7fc2fdd9e640 1 --2- 172.21.15.88:0/347345093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560426070c40 0x560426099810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.480+0000 7fc2fdd9e640 1 -- 172.21.15.88:0/347345093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042606f4b0 msgr2=0x5604260979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:54.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.480+0000 7fc2fdd9e640 1 --2- 172.21.15.88:0/347345093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042606f4b0 0x5604260979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.480+0000 7fc2fdd9e640 1 -- 172.21.15.88:0/347345093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560425ce42d0 con 0x56042608f7a0 2024-09-16T01:43:54.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.480+0000 7fc2fcd9c640 1 --2- 172.21.15.88:0/347345093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560426070c40 0x560426099810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:54.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.480+0000 7fc2fd59d640 1 --2- 172.21.15.88:0/347345093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042606f4b0 0x5604260979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:54.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.480+0000 7fc2fdd9e640 1 --2- 172.21.15.88:0/347345093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56042608f7a0 0x5604260adf00 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0014a30 tx=0x7fc2f0001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=492972465b766d86 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:54.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.480+0000 7fc2d3fff640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:43:54.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.480+0000 7fc2d3fff640 1 -- 172.21.15.88:0/347345093 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0007810 con 0x56042608f7a0 2024-09-16T01:43:54.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.480+0000 7fc2d3fff640 1 -- 172.21.15.88:0/347345093 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8662+0+0 (secure 0 0 0) 0x7fc2f001e640 con 0x56042608f7a0 2024-09-16T01:43:54.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.480+0000 7fc302671740 1 -- 172.21.15.88:0/347345093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2f00079b0 con 0x56042608f7a0 2024-09-16T01:43:54.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.480+0000 7fc2d3fff640 1 -- 172.21.15.88:0/347345093 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=282) ==== 24+0+0 (secure 0 0 0) 0x7fc2f002d050 con 0x56042608f7a0 2024-09-16T01:43:54.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.480+0000 7fc302671740 10 client.10017 fetch_fsmap learned FSMap version 282 2024-09-16T01:43:54.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.480+0000 7fc302671740 1 -- 172.21.15.88:0/347345093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=282,monmap=2+}) -- 0x560425dae6b0 con 0x56042608f7a0 2024-09-16T01:43:54.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.480+0000 7fc2d3fff640 1 -- 172.21.15.88:0/347345093 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2f00079b0 con 0x56042608f7a0 2024-09-16T01:43:54.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.480+0000 7fc2d3fff640 1 -- 172.21.15.88:0/347345093 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 282) ==== 3502+0+0 (secure 0 0 0) 0x7fc2f001e840 con 0x56042608f7a0 2024-09-16T01:43:54.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.480+0000 7fc302671740 10 client.10017 fetch_fsmap finished waiting for FSMap version 282 2024-09-16T01:43:54.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.480+0000 7fc302671740 10 client.10017 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:43:54.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.480+0000 7fc302671740 20 client.10017 populate_metadata read hostname 'smithi088' 2024-09-16T01:43:54.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.480+0000 7fc302671740 1 --2- 172.21.15.88:0/347345093 >> [v2:172.21.15.88:6834/3892583353,v1:172.21.15.88:6836/3892583353] conn(0x5604260c51e0 0x560426090370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.480+0000 7fc302671740 4 client.10017 mds_command: new command op to 9954 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:43:54.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.480+0000 7fc302671740 1 -- 172.21.15.88:0/347345093 --> [v2:172.21.15.88:6834/3892583353,v1:172.21.15.88:6836/3892583353] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560425fdec50 con 0x5604260c51e0 2024-09-16T01:43:54.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.480+0000 7fc2fd59d640 1 --2- 172.21.15.88:0/347345093 >> [v2:172.21.15.88:6834/3892583353,v1:172.21.15.88:6836/3892583353] conn(0x5604260c51e0 0x560426090370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.481+0000 7fc2d3fff640 10 client.10017 ms_handle_connect on v2:172.21.15.88:6834/3892583353 2024-09-16T01:43:54.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.481+0000 7fc2fd59d640 1 --2- 172.21.15.88:0/347345093 >> [v2:172.21.15.88:6834/3892583353,v1:172.21.15.88:6836/3892583353] conn(0x5604260c51e0 0x560426090370 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41d1c4e75e6669e2 server_cookie=dcb5ff116e252271 in_seq=0 out_seq=0 2024-09-16T01:43:54.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.481+0000 7fc2d3fff640 1 -- 172.21.15.88:0/347345093 <== mds.0 v2:172.21.15.88:6834/3892583353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560425fdec50 con 0x5604260c51e0 2024-09-16T01:43:54.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.481+0000 7fc2d3fff640 10 client.10017 handle_command_reply: tid=0 2024-09-16T01:43:54.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.481+0000 7fc302671740 1 client.10017 shutdown 2024-09-16T01:43:54.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.482+0000 7fc302671740 1 -- 172.21.15.88:0/347345093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56042608f7a0 msgr2=0x5604260adf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:54.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.482+0000 7fc302671740 1 --2- 172.21.15.88:0/347345093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56042608f7a0 0x5604260adf00 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0014a30 tx=0x7fc2f0001dc0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.482+0000 7fc302671740 1 -- 172.21.15.88:0/347345093 shutdown_connections 2024-09-16T01:43:54.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.482+0000 7fc302671740 1 --2- 172.21.15.88:0/347345093 >> [v2:172.21.15.88:6834/3892583353,v1:172.21.15.88:6836/3892583353] conn(0x5604260c51e0 0x560426090370 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.482+0000 7fc302671740 1 --2- 172.21.15.88:0/347345093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56042608f7a0 0x5604260adf00 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.482+0000 7fc302671740 1 --2- 172.21.15.88:0/347345093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x560426070c40 0x560426099810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.482+0000 7fc2d3fff640 0 client.10017 ms_handle_reset on v2:172.21.15.88:6834/3892583353 2024-09-16T01:43:54.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.482+0000 7fc302671740 1 --2- 172.21.15.88:0/347345093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042606f4b0 0x5604260979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.482+0000 7fc302671740 1 -- 172.21.15.88:0/347345093 >> 172.21.15.88:0/347345093 conn(0x56042606cc70 msgr2=0x5604260b5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:54.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.482+0000 7fc302671740 1 -- 172.21.15.88:0/347345093 shutdown_connections 2024-09-16T01:43:54.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.482+0000 7fc302671740 1 -- 172.21.15.88:0/347345093 wait complete. 2024-09-16T01:43:54.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.482+0000 7fc302671740 20 client.10017 trim_cache size 0 max 16384 2024-09-16T01:43:54.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.499+0000 7fc302671740 1 Processor -- start 2024-09-16T01:43:54.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.500+0000 7fc302671740 1 -- start start 2024-09-16T01:43:54.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.500+0000 7fc302671740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042606cc70 0x5604260b0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.500+0000 7fc302671740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56042606f4b0 0x5604260b20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.500+0000 7fc302671740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560426070c40 0x5604260b2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.500+0000 7fc302671740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f8201e70 con 0x560426070c40 2024-09-16T01:43:54.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.500+0000 7fc302671740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f82011d0 con 0x56042606cc70 2024-09-16T01:43:54.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.500+0000 7fc302671740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc2f8201820 con 0x56042606f4b0 2024-09-16T01:43:54.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.500+0000 7fc2fdd9e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560426070c40 0x5604260b2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.500+0000 7fc2fdd9e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560426070c40 0x5604260b2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38160/0 (socket says 172.21.15.88:38160) 2024-09-16T01:43:54.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.500+0000 7fc2fdd9e640 1 -- 172.21.15.88:0/313996792 learned_addr learned my addr 172.21.15.88:0/313996792 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:54.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.501+0000 7fc2fd59d640 1 --2- 172.21.15.88:0/313996792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042606cc70 0x5604260b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.501+0000 7fc2fcd9c640 1 --2- 172.21.15.88:0/313996792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56042606f4b0 0x5604260b20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.501+0000 7fc2fdd9e640 1 -- 172.21.15.88:0/313996792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56042606f4b0 msgr2=0x5604260b20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:54.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.501+0000 7fc2fdd9e640 1 --2- 172.21.15.88:0/313996792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56042606f4b0 0x5604260b20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.501+0000 7fc2fdd9e640 1 -- 172.21.15.88:0/313996792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042606cc70 msgr2=0x5604260b0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:54.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.501+0000 7fc2fdd9e640 1 --2- 172.21.15.88:0/313996792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042606cc70 0x5604260b0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.501+0000 7fc2fdd9e640 1 -- 172.21.15.88:0/313996792 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604260b5e20 con 0x560426070c40 2024-09-16T01:43:54.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.501+0000 7fc2fd59d640 1 --2- 172.21.15.88:0/313996792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042606cc70 0x5604260b0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:54.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.501+0000 7fc2fcd9c640 1 --2- 172.21.15.88:0/313996792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56042606f4b0 0x5604260b20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:54.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.501+0000 7fc2fdd9e640 1 --2- 172.21.15.88:0/313996792 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560426070c40 0x5604260b2610 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0009010 tx=0x7fc2f001e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f6b019f56c47f14 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:54.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.501+0000 7fc2da7fc640 1 -- 172.21.15.88:0/313996792 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0014a60 con 0x560426070c40 2024-09-16T01:43:54.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.501+0000 7fc2da7fc640 1 -- 172.21.15.88:0/313996792 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2f0024070 con 0x560426070c40 2024-09-16T01:43:54.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.501+0000 7fc2da7fc640 1 -- 172.21.15.88:0/313996792 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f001e680 con 0x560426070c40 2024-09-16T01:43:54.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.502+0000 7fc302671740 1 -- 172.21.15.88:0/313996792 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560426070c40 msgr2=0x5604260b2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:54.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.502+0000 7fc302671740 1 --2- 172.21.15.88:0/313996792 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560426070c40 0x5604260b2610 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0009010 tx=0x7fc2f001e000 comp rx=0 tx=0).stop 2024-09-16T01:43:54.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.502+0000 7fc302671740 1 -- 172.21.15.88:0/313996792 shutdown_connections 2024-09-16T01:43:54.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.502+0000 7fc302671740 1 --2- 172.21.15.88:0/313996792 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560426070c40 0x5604260b2610 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.502+0000 7fc302671740 1 --2- 172.21.15.88:0/313996792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56042606f4b0 0x5604260b20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.502+0000 7fc302671740 1 --2- 172.21.15.88:0/313996792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56042606cc70 0x5604260b0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.502+0000 7fc302671740 1 -- 172.21.15.88:0/313996792 >> 172.21.15.88:0/313996792 conn(0x5604260b5ff0 msgr2=0x560426091a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:54.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.502+0000 7fc302671740 1 -- 172.21.15.88:0/313996792 shutdown_connections 2024-09-16T01:43:54.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.502+0000 7fc302671740 1 -- 172.21.15.88:0/313996792 wait complete. 2024-09-16T01:43:54.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.502+0000 7fc302671740 1 -- start start 2024-09-16T01:43:54.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.502+0000 7fc302671740 1 Processor -- start 2024-09-16T01:43:54.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.503+0000 7fc302671740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56042606cc70 0x56042608d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.503+0000 7fc302671740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56042606f4b0 0x5604260b22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.503+0000 7fc302671740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560426070c40 0x560426136ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.503+0000 7fc302671740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5604260c56d0 con 0x56042606f4b0 2024-09-16T01:43:54.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.503+0000 7fc302671740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x560426099350 con 0x560426070c40 2024-09-16T01:43:54.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.503+0000 7fc302671740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x560426069610 con 0x56042606cc70 2024-09-16T01:43:54.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.503+0000 7fc302671740 10 client.0 _ll_register_callbacks cb 0x7fc2e8006eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:43:54.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.503+0000 7fc2fcd9c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56042606f4b0 0x5604260b22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.503+0000 7fc2fcd9c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56042606f4b0 0x5604260b22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38172/0 (socket says 172.21.15.88:38172) 2024-09-16T01:43:54.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.503+0000 7fc2fcd9c640 1 -- 172.21.15.88:0/2978385742 learned_addr learned my addr 172.21.15.88:0/2978385742 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:54.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.503+0000 7fc2fd59d640 1 --2- 172.21.15.88:0/2978385742 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56042606cc70 0x56042608d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.503+0000 7fc2fdd9e640 1 --2- 172.21.15.88:0/2978385742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560426070c40 0x560426136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.503+0000 7fc2fcd9c640 1 -- 172.21.15.88:0/2978385742 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56042606cc70 msgr2=0x56042608d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:54.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.503+0000 7fc2fcd9c640 1 --2- 172.21.15.88:0/2978385742 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56042606cc70 0x56042608d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.503+0000 7fc2fcd9c640 1 -- 172.21.15.88:0/2978385742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560426070c40 msgr2=0x560426136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:54.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.503+0000 7fc2fcd9c640 1 --2- 172.21.15.88:0/2978385742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560426070c40 0x560426136ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.503+0000 7fc2fcd9c640 1 -- 172.21.15.88:0/2978385742 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604260967e0 con 0x56042606f4b0 2024-09-16T01:43:54.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.504+0000 7fc2fd59d640 1 --2- 172.21.15.88:0/2978385742 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56042606cc70 0x56042608d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:54.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.504+0000 7fc2fcd9c640 1 --2- 172.21.15.88:0/2978385742 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56042606f4b0 0x5604260b22c0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8079d70 tx=0x7fc2e807a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=502f492e601f4bca server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:54.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.504+0000 7fc2d3fff640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:43:54.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.504+0000 7fc2d3fff640 1 -- 172.21.15.88:0/2978385742 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e80a8060 con 0x56042606f4b0 2024-09-16T01:43:54.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.504+0000 7fc2d3fff640 1 -- 172.21.15.88:0/2978385742 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 8662+0+0 (secure 0 0 0) 0x7fc2e8082730 con 0x56042606f4b0 2024-09-16T01:43:54.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.504+0000 7fc302671740 1 -- 172.21.15.88:0/2978385742 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2f0024070 con 0x56042606f4b0 2024-09-16T01:43:54.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.504+0000 7fc2d3fff640 1 -- 172.21.15.88:0/2978385742 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=282) ==== 24+0+0 (secure 0 0 0) 0x7fc2e804a040 con 0x56042606f4b0 2024-09-16T01:43:54.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.504+0000 7fc302671740 10 client.10023 fetch_fsmap learned FSMap version 282 2024-09-16T01:43:54.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.504+0000 7fc302671740 1 -- 172.21.15.88:0/2978385742 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=282,monmap=2+}) -- 0x56042608db20 con 0x56042606f4b0 2024-09-16T01:43:54.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.504+0000 7fc2d3fff640 1 -- 172.21.15.88:0/2978385742 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2e809a070 con 0x56042606f4b0 2024-09-16T01:43:54.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.504+0000 7fc2d3fff640 1 -- 172.21.15.88:0/2978385742 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 282) ==== 3502+0+0 (secure 0 0 0) 0x7fc2e8079230 con 0x56042606f4b0 2024-09-16T01:43:54.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.504+0000 7fc302671740 10 client.10023 fetch_fsmap finished waiting for FSMap version 282 2024-09-16T01:43:54.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.504+0000 7fc302671740 10 client.10023 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:43:54.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.504+0000 7fc302671740 20 client.10023 populate_metadata read hostname 'smithi088' 2024-09-16T01:43:54.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.504+0000 7fc302671740 1 --2- 172.21.15.88:0/2978385742 >> [v2:172.21.15.88:6834/3892583353,v1:172.21.15.88:6836/3892583353] conn(0x56042608f7a0 0x560426118910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.504+0000 7fc302671740 4 client.10023 mds_command: new command op to 9954 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:43:54.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.504+0000 7fc302671740 1 -- 172.21.15.88:0/2978385742 --> [v2:172.21.15.88:6834/3892583353,v1:172.21.15.88:6836/3892583353] -- command(tid 0: {"prefix": "status"}) -- 0x560425fd8fa0 con 0x56042608f7a0 2024-09-16T01:43:54.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.504+0000 7fc2fd59d640 1 --2- 172.21.15.88:0/2978385742 >> [v2:172.21.15.88:6834/3892583353,v1:172.21.15.88:6836/3892583353] conn(0x56042608f7a0 0x560426118910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.505+0000 7fc2d3fff640 10 client.10023 ms_handle_connect on v2:172.21.15.88:6834/3892583353 2024-09-16T01:43:54.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.505+0000 7fc2fd59d640 1 --2- 172.21.15.88:0/2978385742 >> [v2:172.21.15.88:6834/3892583353,v1:172.21.15.88:6836/3892583353] conn(0x56042608f7a0 0x560426118910 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf673bf24787d0d5 server_cookie=d830da177dbe4b2 in_seq=0 out_seq=0 2024-09-16T01:43:54.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.505+0000 7fc2d3fff640 1 -- 172.21.15.88:0/2978385742 <== mds.0 v2:172.21.15.88:6834/3892583353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x560425fd8fa0 con 0x56042608f7a0 2024-09-16T01:43:54.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.505+0000 7fc2d3fff640 10 client.10023 handle_command_reply: tid=0 2024-09-16T01:43:54.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.505+0000 7fc302671740 1 client.10023 shutdown 2024-09-16T01:43:54.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.505+0000 7fc302671740 1 -- 172.21.15.88:0/2978385742 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56042606f4b0 msgr2=0x5604260b22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:54.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.505+0000 7fc302671740 1 --2- 172.21.15.88:0/2978385742 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56042606f4b0 0x5604260b22c0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8079d70 tx=0x7fc2e807a610 comp rx=0 tx=0).stop 2024-09-16T01:43:54.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.505+0000 7fc302671740 1 -- 172.21.15.88:0/2978385742 shutdown_connections 2024-09-16T01:43:54.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.505+0000 7fc302671740 1 --2- 172.21.15.88:0/2978385742 >> [v2:172.21.15.88:6834/3892583353,v1:172.21.15.88:6836/3892583353] conn(0x56042608f7a0 0x560426118910 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.505+0000 7fc302671740 1 --2- 172.21.15.88:0/2978385742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560426070c40 0x560426136ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.505+0000 7fc302671740 1 --2- 172.21.15.88:0/2978385742 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56042606f4b0 0x5604260b22c0 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.505+0000 7fc2d3fff640 0 client.10023 ms_handle_reset on v2:172.21.15.88:6834/3892583353 2024-09-16T01:43:54.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.505+0000 7fc302671740 1 --2- 172.21.15.88:0/2978385742 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56042606cc70 0x56042608d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.505+0000 7fc302671740 1 -- 172.21.15.88:0/2978385742 >> 172.21.15.88:0/2978385742 conn(0x5604260b5ff0 msgr2=0x560426091a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:54.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.505+0000 7fc302671740 1 -- 172.21.15.88:0/2978385742 shutdown_connections 2024-09-16T01:43:54.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.505+0000 7fc302671740 1 -- 172.21.15.88:0/2978385742 wait complete. 2024-09-16T01:43:54.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.505+0000 7fc302671740 20 client.10023 trim_cache size 0 max 16384 2024-09-16T01:43:54.518 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:43:54.518 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "5ace2c91-46f9-482b-8a89-731fbdc171bf", 2024-09-16T01:43:54.518 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2024-09-16T01:43:54.518 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 9954, 2024-09-16T01:43:54.518 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2024-09-16T01:43:54.518 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2024-09-16T01:43:54.518 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2024-09-16T01:43:54.518 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 6.6251419660000002, 2024-09-16T01:43:54.518 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 282, 2024-09-16T01:43:54.518 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 121, 2024-09-16T01:43:54.518 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 121, 2024-09-16T01:43:54.519 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 8.6075405549999999 2024-09-16T01:43:54.519 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:43:54.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.506+0000 7fc2fe59f640 1 -- 172.21.15.88:0/3527270948 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc2dc06c2b0 msgr2=0x7fc2dc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:54.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.506+0000 7fc2fe59f640 1 --2- 172.21.15.88:0/3527270948 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc2dc06c2b0 0x7fc2dc06e770 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fc2f4000c00 tx=0x7fc2f4009040 comp rx=0 tx=0).stop 2024-09-16T01:43:54.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.506+0000 7fc2fe59f640 1 -- 172.21.15.88:0/3527270948 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2f81583f0 msgr2=0x7fc2f81f43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:54.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.506+0000 7fc2fe59f640 1 --2- 172.21.15.88:0/3527270948 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2f81583f0 0x7fc2f81f43b0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8009c70 tx=0x7fc2e8004730 comp rx=0 tx=0).stop 2024-09-16T01:43:54.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.506+0000 7fc2fe59f640 1 -- 172.21.15.88:0/3527270948 shutdown_connections 2024-09-16T01:43:54.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.506+0000 7fc2fe59f640 1 --2- 172.21.15.88:0/3527270948 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc2dc06c2b0 0x7fc2dc06e770 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.506+0000 7fc2fe59f640 1 --2- 172.21.15.88:0/3527270948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2f815b230 0x7fc2f814d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.506+0000 7fc2fe59f640 1 --2- 172.21.15.88:0/3527270948 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2f81583f0 0x7fc2f81f43b0 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.506+0000 7fc2fe59f640 1 --2- 172.21.15.88:0/3527270948 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2e4004800 0x7fc2f8158e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.506+0000 7fc2fe59f640 1 -- 172.21.15.88:0/3527270948 >> 172.21.15.88:0/3527270948 conn(0x7fc2f8062630 msgr2=0x7fc2f80cf810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:54.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.506+0000 7fc2fe59f640 1 -- 172.21.15.88:0/3527270948 shutdown_connections 2024-09-16T01:43:54.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.506+0000 7fc2fe59f640 1 -- 172.21.15.88:0/3527270948 wait complete. 2024-09-16T01:43:54.522 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-09-16T01:43:54.522 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Corrupt omap header on 10000000000.00000000', proceeding to ls 2024-09-16T01:43:54.522 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:43:54.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.617+0000 7fed48128640 1 Processor -- start 2024-09-16T01:43:54.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.618+0000 7fed48128640 1 -- start start 2024-09-16T01:43:54.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.618+0000 7fed48128640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fed400cfc80 0x7fed400b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.618+0000 7fed48128640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed4015c740 con 0x7fed40155800 2024-09-16T01:43:54.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.618+0000 7fed48128640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed400b50a0 con 0x7fed4015c330 2024-09-16T01:43:54.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.618+0000 7fed48128640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fed400b5380 con 0x7fed400cfc80 2024-09-16T01:43:54.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.619+0000 7fed45ec4640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fed40155800 0x7fed400cf570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49288/0 (socket says 172.21.15.88:49288) 2024-09-16T01:43:54.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.619+0000 7fed45ec4640 1 -- 172.21.15.88:0/2807547363 learned_addr learned my addr 172.21.15.88:0/2807547363 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:54.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.619+0000 7fed456c3640 1 --2- 172.21.15.88:0/2807547363 >> v2:172.21.15.154:3300/0 conn(0x7fed400cfc80 0x7fed400b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.619+0000 7fed44ec2640 1 -- 172.21.15.88:0/2807547363 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2034747422 0 0) 0x7fed4015c740 con 0x7fed40155800 2024-09-16T01:43:54.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.619+0000 7fed44ec2640 1 -- 172.21.15.88:0/2807547363 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed2c003520 con 0x7fed40155800 2024-09-16T01:43:54.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.619+0000 7fed44ec2640 1 -- 172.21.15.88:0/2807547363 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2454702578 0 0) 0x7fed2c003520 con 0x7fed40155800 2024-09-16T01:43:54.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.619+0000 7fed44ec2640 1 -- 172.21.15.88:0/2807547363 >> v2:172.21.15.154:3300/0 conn(0x7fed400cfc80 msgr2=0x7fed400b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:54.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.619+0000 7fed44ec2640 1 --2- 172.21.15.88:0/2807547363 >> v2:172.21.15.154:3300/0 conn(0x7fed400cfc80 0x7fed400b6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.619+0000 7fed44ec2640 1 -- 172.21.15.88:0/2807547363 >> v1:172.21.15.154:6789/0 conn(0x7fed4015c330 legacy=0x7fed4015c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:54.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.619+0000 7fed44ec2640 1 -- 172.21.15.88:0/2807547363 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed400b5660 con 0x7fed40155800 2024-09-16T01:43:54.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.619+0000 7fed44ec2640 1 -- 172.21.15.88:0/2807547363 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fed34001670 con 0x7fed40155800 2024-09-16T01:43:54.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.619+0000 7fed44ec2640 1 -- 172.21.15.88:0/2807547363 >> v1:172.21.15.88:6789/0 conn(0x7fed40155800 legacy=0x7fed400cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:54.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.619+0000 7fed44ec2640 1 --2- 172.21.15.88:0/2807547363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed2c0039b0 0x7fed2c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.619+0000 7fed44ec2640 1 --2- 172.21.15.88:0/2807547363 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2c004800 0x7fed2c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.619+0000 7fed44ec2640 1 --2- 172.21.15.88:0/2807547363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed2c0051c0 0x7fed2c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.619+0000 7fed44ec2640 1 -- 172.21.15.88:0/2807547363 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fed400b5380 con 0x7fed2c004800 2024-09-16T01:43:54.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.619+0000 7fed44ec2640 1 -- 172.21.15.88:0/2807547363 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fed2c00c060 con 0x7fed2c0039b0 2024-09-16T01:43:54.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.619+0000 7fed44ec2640 1 -- 172.21.15.88:0/2807547363 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fed2c00c260 con 0x7fed2c0051c0 2024-09-16T01:43:54.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.619+0000 7fed456c3640 1 --2- 172.21.15.88:0/2807547363 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2c004800 0x7fed2c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.619+0000 7fed45ec4640 1 --2- 172.21.15.88:0/2807547363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed2c0039b0 0x7fed2c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.619+0000 7fed466c5640 1 --2- 172.21.15.88:0/2807547363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed2c0051c0 0x7fed2c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.620+0000 7fed456c3640 1 -- 172.21.15.88:0/2807547363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed2c0051c0 msgr2=0x7fed2c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:54.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.620+0000 7fed456c3640 1 --2- 172.21.15.88:0/2807547363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed2c0051c0 0x7fed2c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.620+0000 7fed456c3640 1 -- 172.21.15.88:0/2807547363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed2c0039b0 msgr2=0x7fed2c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:54.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.620+0000 7fed456c3640 1 --2- 172.21.15.88:0/2807547363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed2c0039b0 0x7fed2c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.620+0000 7fed456c3640 1 -- 172.21.15.88:0/2807547363 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed2c003520 con 0x7fed2c004800 2024-09-16T01:43:54.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.620+0000 7fed466c5640 1 -- 172.21.15.88:0/2807547363 reap_dead start 2024-09-16T01:43:54.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.620+0000 7fed456c3640 1 --2- 172.21.15.88:0/2807547363 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2c004800 0x7fed2c004c80 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7fed3c009a90 tx=0x7fed2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dca756ade9bd8b1f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:54.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.620+0000 7fed44ec2640 1 -- 172.21.15.88:0/2807547363 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c036070 con 0x7fed2c004800 2024-09-16T01:43:54.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.620+0000 7fed44ec2640 1 -- 172.21.15.88:0/2807547363 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed3c031400 con 0x7fed2c004800 2024-09-16T01:43:54.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.620+0000 7fed44ec2640 1 -- 172.21.15.88:0/2807547363 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c009690 con 0x7fed2c004800 2024-09-16T01:43:54.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.621+0000 7fed48128640 1 -- 172.21.15.88:0/2807547363 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2c004800 msgr2=0x7fed2c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:54.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.621+0000 7fed48128640 1 --2- 172.21.15.88:0/2807547363 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2c004800 0x7fed2c004c80 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7fed3c009a90 tx=0x7fed2c005830 comp rx=0 tx=0).stop 2024-09-16T01:43:54.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.621+0000 7fed48128640 1 -- 172.21.15.88:0/2807547363 shutdown_connections 2024-09-16T01:43:54.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.621+0000 7fed48128640 1 --2- 172.21.15.88:0/2807547363 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2c004800 0x7fed2c004c80 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.621+0000 7fed48128640 1 -- 172.21.15.88:0/2807547363 >> 172.21.15.88:0/2807547363 conn(0x7fed4006bc10 msgr2=0x7fed4006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:54.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.621+0000 7fed48128640 1 -- 172.21.15.88:0/2807547363 shutdown_connections 2024-09-16T01:43:54.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.621+0000 7fed48128640 1 -- 172.21.15.88:0/2807547363 wait complete. 2024-09-16T01:43:54.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.622+0000 7fed48128640 1 Processor -- start 2024-09-16T01:43:54.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.622+0000 7fed48128640 1 -- start start 2024-09-16T01:43:54.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.623+0000 7fed48128640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2c004800 0x7fed40151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.623+0000 7fed48128640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed401f1590 0x7fed40158890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.623+0000 7fed48128640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed401f76c0 0x7fed401f7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.623+0000 7fed48128640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fed40155be0 con 0x7fed2c004800 2024-09-16T01:43:54.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.623+0000 7fed48128640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fed400b5c20 con 0x7fed401f76c0 2024-09-16T01:43:54.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.623+0000 7fed48128640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fed400b6140 con 0x7fed401f1590 2024-09-16T01:43:54.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.623+0000 7fed45ec4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2c004800 0x7fed40151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.623+0000 7fed45ec4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2c004800 0x7fed40151d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38198/0 (socket says 172.21.15.88:38198) 2024-09-16T01:43:54.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.623+0000 7fed45ec4640 1 -- 172.21.15.88:0/1043188214 learned_addr learned my addr 172.21.15.88:0/1043188214 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:54.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.623+0000 7fed466c5640 1 --2- 172.21.15.88:0/1043188214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed401f76c0 0x7fed401f7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.623+0000 7fed456c3640 1 --2- 172.21.15.88:0/1043188214 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed401f1590 0x7fed40158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.623+0000 7fed45ec4640 1 -- 172.21.15.88:0/1043188214 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed401f1590 msgr2=0x7fed40158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:54.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.623+0000 7fed45ec4640 1 --2- 172.21.15.88:0/1043188214 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed401f1590 0x7fed40158890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.623+0000 7fed45ec4640 1 -- 172.21.15.88:0/1043188214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed401f76c0 msgr2=0x7fed401f7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:54.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.623+0000 7fed45ec4640 1 --2- 172.21.15.88:0/1043188214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed401f76c0 0x7fed401f7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.623+0000 7fed45ec4640 1 -- 172.21.15.88:0/1043188214 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed401542d0 con 0x7fed2c004800 2024-09-16T01:43:54.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.623+0000 7fed466c5640 1 --2- 172.21.15.88:0/1043188214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed401f76c0 0x7fed401f7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:54.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.623+0000 7fed45ec4640 1 --2- 172.21.15.88:0/1043188214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2c004800 0x7fed40151d90 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7fed340072c0 tx=0x7fed34007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:54.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.624+0000 7fed327fc640 1 -- 172.21.15.88:0/1043188214 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3400f960 con 0x7fed2c004800 2024-09-16T01:43:54.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.624+0000 7fed327fc640 1 -- 172.21.15.88:0/1043188214 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed34018070 con 0x7fed2c004800 2024-09-16T01:43:54.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.624+0000 7fed327fc640 1 -- 172.21.15.88:0/1043188214 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3400fd60 con 0x7fed2c004800 2024-09-16T01:43:54.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.624+0000 7fed48128640 1 -- 172.21.15.88:0/1043188214 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed401545b0 con 0x7fed2c004800 2024-09-16T01:43:54.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.624+0000 7fed48128640 1 -- 172.21.15.88:0/1043188214 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed40154b70 con 0x7fed2c004800 2024-09-16T01:43:54.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.625+0000 7fed48128640 1 -- 172.21.15.88:0/1043188214 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed400b6470 con 0x7fed2c004800 2024-09-16T01:43:54.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.625+0000 7fed327fc640 1 -- 172.21.15.88:0/1043188214 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fed3401d460 con 0x7fed2c004800 2024-09-16T01:43:54.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.625+0000 7fed327fc640 1 --2- 172.21.15.88:0/1043188214 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fed2406c2e0 0x7fed2406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:54.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.626+0000 7fed456c3640 1 --2- 172.21.15.88:0/1043188214 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fed2406c2e0 0x7fed2406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:54.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.626+0000 7fed327fc640 1 -- 172.21.15.88:0/1043188214 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8662+0+0 (secure 0 0 0) 0x7fed3408c0a0 con 0x7fed2c004800 2024-09-16T01:43:54.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.626+0000 7fed456c3640 1 --2- 172.21.15.88:0/1043188214 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fed2406c2e0 0x7fed2406e7a0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fed3c033000 tx=0x7fed3c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:54.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.630+0000 7fed327fc640 1 -- 172.21.15.88:0/1043188214 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fed34059cc0 con 0x7fed2c004800 2024-09-16T01:43:54.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.829+0000 7fed48128640 1 -- 172.21.15.88:0/1043188214 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fed40154d40 con 0x7fed2c004800 2024-09-16T01:43:54.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.830+0000 7fed327fc640 1 -- 172.21.15.88:0/1043188214 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v282) ==== 53+0+73 (secure 0 0 0) 0x7fed340595c0 con 0x7fed2c004800 2024-09-16T01:43:54.937 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:43:54.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.832+0000 7fed48128640 1 -- 172.21.15.88:0/1043188214 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fed2406c2e0 msgr2=0x7fed2406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:54.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.832+0000 7fed48128640 1 --2- 172.21.15.88:0/1043188214 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fed2406c2e0 0x7fed2406e7a0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fed3c033000 tx=0x7fed3c008030 comp rx=0 tx=0).stop 2024-09-16T01:43:54.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.832+0000 7fed48128640 1 -- 172.21.15.88:0/1043188214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2c004800 msgr2=0x7fed40151d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:54.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.832+0000 7fed48128640 1 --2- 172.21.15.88:0/1043188214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2c004800 0x7fed40151d90 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7fed340072c0 tx=0x7fed34007780 comp rx=0 tx=0).stop 2024-09-16T01:43:54.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.832+0000 7fed48128640 1 -- 172.21.15.88:0/1043188214 shutdown_connections 2024-09-16T01:43:54.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.832+0000 7fed48128640 1 --2- 172.21.15.88:0/1043188214 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fed2406c2e0 0x7fed2406e7a0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.832+0000 7fed48128640 1 --2- 172.21.15.88:0/1043188214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed401f76c0 0x7fed401f7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.832+0000 7fed48128640 1 --2- 172.21.15.88:0/1043188214 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed401f1590 0x7fed40158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.832+0000 7fed48128640 1 --2- 172.21.15.88:0/1043188214 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed2c004800 0x7fed40151d90 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:54.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.832+0000 7fed48128640 1 -- 172.21.15.88:0/1043188214 >> 172.21.15.88:0/1043188214 conn(0x7fed4006bc10 msgr2=0x7fed400b37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:54.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.832+0000 7fed48128640 1 -- 172.21.15.88:0/1043188214 shutdown_connections 2024-09-16T01:43:54.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:54.832+0000 7fed48128640 1 -- 172.21.15.88:0/1043188214 wait complete. 2024-09-16T01:43:54.939 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-16T01:43:54.939 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:43:54.939 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2024-09-16T01:43:54.940 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T01:43:54.940 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T01:43:54.940 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-16T01:43:54.940 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-16T01:43:54.940 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:43:54.940 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:43:54.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:54.940 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2024-09-16T01:43:54.957 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:43:54.958 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:43:54.958 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:43:54.958 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:43:54.958 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-09-16T01:43:54.958 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:43:54.958 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:43:54.958 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:43:54.958 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2024-09-16T01:43:54.958 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:43:54.958 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:43:54.958 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2024-09-16T01:43:54.958 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:43:54.959 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2024-09-16T01:43:54.959 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:43:54.959 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2024-09-16T01:43:54.959 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:43:54.959 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42189sec preferred_lft 42189sec 2024-09-16T01:43:54.959 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link noprefixroute 2024-09-16T01:43:54.959 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:43:54.959 INFO:teuthology.orchestra.run.smithi088.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:43:54.959 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff 2024-09-16T01:43:54.959 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T01:43:54.959 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:43:54.959 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::e486:6eff:fe81:a1c/64 scope link 2024-09-16T01:43:54.959 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:43:54.960 INFO:teuthology.orchestra.run.smithi088.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:43:54.960 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 12:6f:29:be:95:48 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:43:54.960 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::106f:29ff:febe:9548/64 scope link 2024-09-16T01:43:54.960 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:43:54.960 INFO:teuthology.orchestra.run.smithi088.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:43:54.960 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T01:43:54.960 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::c63:b4ff:feb3:ad9e/64 scope link 2024-09-16T01:43:54.960 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:43:54.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:54.961 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:43:55.026 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-09-16T01:43:55.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:55.028 DEBUG:teuthology.orchestra.run.smithi088:> route 2024-09-16T01:43:55.085 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2024-09-16T01:43:55.085 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:43:55.086 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:43:55.086 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:43:55.086 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:43:55.086 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:43:55.087 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:43:55.087 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:43:55.087 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:43:55.087 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:43:55.087 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:43:55.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:43:55.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:43:55.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:55.215 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:43:55.232 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T01:43:55.232 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T01:43:55.232 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:43:55.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:55.232 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2024-09-16T01:43:55.298 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:43:55.298 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T01:43:55.353 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:43:55.354 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:43:55.354 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:43:55.409 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-16T01:43:55.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:55.409 DEBUG:teuthology.orchestra.run.smithi088:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:43:55.491 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:43:55.492 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:43:55.492 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:43:55.500 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:43:55.500 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:43:55.500 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:43:55.501 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:43:55.501 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:43:55.507 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:43:55.507 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:43:55.507 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:43:55.507 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:43:55.507 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:43:55.507 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790" 2024-09-16T01:43:55.507 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2024-09-16T01:43:55.507 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5ace2c91-46f9-482b-8a89-731fbdc171bf" will pass to kernel 2024-09-16T01:43:55.508 INFO:teuthology.orchestra.run.smithi088.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-16T01:43:55.508 INFO:teuthology.orchestra.run.smithi088.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:43:55.508 INFO:teuthology.orchestra.run.smithi088.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:43:55.508 INFO:teuthology.orchestra.run.smithi088.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:43:55.508 INFO:teuthology.orchestra.run.smithi088.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:43:55.509 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:43:55.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:55.510 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:43:55.575 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:43:55.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:43:55.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:43:55.672 INFO:teuthology.orchestra.run.smithi088.stdout:client7346 2024-09-16T01:43:55.673 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:43:55.673 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:43:55.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:55.690 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R .) 2024-09-16T01:43:55.733 INFO:teuthology.orchestra.run:waiting for 20 2024-09-16T01:43:55.747 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:43:55.746+0000 7f5e69fb0640 -1 mds.0.cache.dir(0x10000000000) Corrupt fnode in dirfrag 0x10000000000: Decoder at 'void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&)' v=3 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-09-16T01:43:55.748 INFO:teuthology.orchestra.run.smithi088.stdout:.: 2024-09-16T01:43:55.748 INFO:teuthology.orchestra.run.smithi088.stdout:subdir 2024-09-16T01:43:55.748 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:55.748 INFO:teuthology.orchestra.run.smithi088.stdout:./subdir: 2024-09-16T01:43:55.749 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt omap header on 10000000000.00000000' did not caused DAMAGED state 2024-09-16T01:43:55.749 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap header on 600.00000000' 2024-09-16T01:43:55.749 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:43:55.749 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:43:55.804 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:43:55.805 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:43:55.860 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:43:56.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.997+0000 7f6fc392a640 1 Processor -- start 2024-09-16T01:43:56.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.997+0000 7f6fc392a640 1 -- start start 2024-09-16T01:43:56.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.998+0000 7f6fc392a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6fbc0b6290 0x7f6fbc071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:56.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.998+0000 7f6fc392a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fbc072530 0x7f6fbc15bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:56.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.998+0000 7f6fc392a640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fbc0ccff0 con 0x7f6fbc15b480 2024-09-16T01:43:56.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.998+0000 7f6fc392a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6fbc0cd2d0 con 0x7f6fbc0b6290 2024-09-16T01:43:56.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.998+0000 7f6fc392a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6fbc0cd5b0 con 0x7f6fbc072530 2024-09-16T01:43:56.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.998+0000 7f6fc16c6640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6fbc15b480 0x7f6fbc0b5b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49302/0 (socket says 172.21.15.88:49302) 2024-09-16T01:43:56.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.998+0000 7f6fc16c6640 1 -- 172.21.15.88:0/2416058115 learned_addr learned my addr 172.21.15.88:0/2416058115 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:56.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.998+0000 7f6fb3fff640 1 -- 172.21.15.88:0/2416058115 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3969222372 0 0) 0x7f6fbc0ccff0 con 0x7f6fbc15b480 2024-09-16T01:43:56.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.998+0000 7f6fc1ec7640 1 --2- 172.21.15.88:0/2416058115 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fbc072530 0x7f6fbc15bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:56.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.998+0000 7f6fc0ec5640 1 --2- 172.21.15.88:0/2416058115 >> v2:172.21.15.154:3300/0 conn(0x7f6fbc0b6290 0x7f6fbc071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:56.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.998+0000 7f6fb3fff640 1 -- 172.21.15.88:0/2416058115 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fa8003520 con 0x7f6fbc15b480 2024-09-16T01:43:56.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.998+0000 7f6fb3fff640 1 -- 172.21.15.88:0/2416058115 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 516901562 0 0) 0x7f6fa8003520 con 0x7f6fbc15b480 2024-09-16T01:43:56.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.998+0000 7f6fb3fff640 1 -- 172.21.15.88:0/2416058115 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fbc072530 msgr2=0x7f6fbc15bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:56.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.998+0000 7f6fb3fff640 1 --2- 172.21.15.88:0/2416058115 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fbc072530 0x7f6fbc15bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:56.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.998+0000 7f6fb3fff640 1 -- 172.21.15.88:0/2416058115 >> v2:172.21.15.154:3300/0 conn(0x7f6fbc0b6290 msgr2=0x7f6fbc071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:56.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.998+0000 7f6fb3fff640 1 --2- 172.21.15.88:0/2416058115 >> v2:172.21.15.154:3300/0 conn(0x7f6fbc0b6290 0x7f6fbc071ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:56.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.998+0000 7f6fb3fff640 1 -- 172.21.15.88:0/2416058115 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fbc0cd890 con 0x7f6fbc15b480 2024-09-16T01:43:56.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.998+0000 7f6fb3fff640 1 -- 172.21.15.88:0/2416058115 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f6fb8001670 con 0x7f6fbc15b480 2024-09-16T01:43:56.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.999+0000 7f6fb3fff640 1 -- 172.21.15.88:0/2416058115 >> v1:172.21.15.88:6789/0 conn(0x7f6fbc15b480 legacy=0x7f6fbc0b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:56.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.999+0000 7f6fb3fff640 1 --2- 172.21.15.88:0/2416058115 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fa80039b0 0x7f6fa8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:56.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.999+0000 7f6fb3fff640 1 --2- 172.21.15.88:0/2416058115 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fbc072530 0x7f6fa8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:56.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.999+0000 7f6fb3fff640 1 --2- 172.21.15.88:0/2416058115 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fa8004d40 0x7f6fa80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:56.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.999+0000 7f6fb3fff640 1 -- 172.21.15.88:0/2416058115 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6fbc0cd2d0 con 0x7f6fa8004d40 2024-09-16T01:43:56.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.999+0000 7f6fb3fff640 1 -- 172.21.15.88:0/2416058115 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6fbc0cd5b0 con 0x7f6fa80039b0 2024-09-16T01:43:56.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.999+0000 7f6fb3fff640 1 -- 172.21.15.88:0/2416058115 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6fa800b9c0 con 0x7f6fbc072530 2024-09-16T01:43:56.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.999+0000 7f6fc1ec7640 1 --2- 172.21.15.88:0/2416058115 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fa8004d40 0x7f6fa80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:56.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.999+0000 7f6fc16c6640 1 --2- 172.21.15.88:0/2416058115 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fa80039b0 0x7f6fa8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:56.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.999+0000 7f6fc0ec5640 1 --2- 172.21.15.88:0/2416058115 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fbc072530 0x7f6fa8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:56.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.999+0000 7f6fc1ec7640 1 -- 172.21.15.88:0/2416058115 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fbc072530 msgr2=0x7f6fa8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:56.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.999+0000 7f6fc1ec7640 1 --2- 172.21.15.88:0/2416058115 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fbc072530 0x7f6fa8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:56.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.999+0000 7f6fc1ec7640 1 -- 172.21.15.88:0/2416058115 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fa80039b0 msgr2=0x7f6fa8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:56.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.999+0000 7f6fc1ec7640 1 --2- 172.21.15.88:0/2416058115 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fa80039b0 0x7f6fa8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:56.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.999+0000 7f6fc1ec7640 1 -- 172.21.15.88:0/2416058115 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fa8003520 con 0x7f6fa8004d40 2024-09-16T01:43:56.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:55.999+0000 7f6fc0ec5640 1 --2- 172.21.15.88:0/2416058115 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fbc072530 0x7f6fa8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:56.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.000+0000 7f6fc1ec7640 1 --2- 172.21.15.88:0/2416058115 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fa8004d40 0x7f6fa80051c0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4009fc0 tx=0x7f6fa8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f00547a815ca75 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:56.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.000+0000 7f6fb3fff640 1 -- 172.21.15.88:0/2416058115 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb4009c10 con 0x7f6fa8004d40 2024-09-16T01:43:56.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.000+0000 7f6fb3fff640 1 -- 172.21.15.88:0/2416058115 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fb4036070 con 0x7f6fa8004d40 2024-09-16T01:43:56.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.000+0000 7f6fb3fff640 1 -- 172.21.15.88:0/2416058115 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb4031880 con 0x7f6fa8004d40 2024-09-16T01:43:56.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.000+0000 7f6fc392a640 1 -- 172.21.15.88:0/2416058115 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fa8004d40 msgr2=0x7f6fa80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:56.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.000+0000 7f6fc392a640 1 --2- 172.21.15.88:0/2416058115 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fa8004d40 0x7f6fa80051c0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4009fc0 tx=0x7f6fa8005830 comp rx=0 tx=0).stop 2024-09-16T01:43:56.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.000+0000 7f6fc1ec7640 1 -- 172.21.15.88:0/2416058115 reap_dead start 2024-09-16T01:43:56.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.000+0000 7f6fc392a640 1 -- 172.21.15.88:0/2416058115 shutdown_connections 2024-09-16T01:43:56.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.000+0000 7f6fc392a640 1 -- 172.21.15.88:0/2416058115 >> 172.21.15.88:0/2416058115 conn(0x7f6fbc06d0f0 msgr2=0x7f6fbc06d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:56.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.000+0000 7f6fc392a640 1 -- 172.21.15.88:0/2416058115 shutdown_connections 2024-09-16T01:43:56.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.000+0000 7f6fc392a640 1 -- 172.21.15.88:0/2416058115 wait complete. 2024-09-16T01:43:56.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.002+0000 7f6fc392a640 1 Processor -- start 2024-09-16T01:43:56.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.002+0000 7f6fc392a640 1 -- start start 2024-09-16T01:43:56.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.002+0000 7f6fc392a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fa8004d40 0x7f6fbc0aae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:56.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.002+0000 7f6fc392a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fbc0c6fa0 0x7f6fbc14f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:56.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.002+0000 7f6fc392a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fbc0c6260 0x7f6fbc0b3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:56.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.002+0000 7f6fc392a640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6fbc15b9e0 con 0x7f6fbc0c6fa0 2024-09-16T01:43:56.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.002+0000 7f6fc392a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6fbc15b860 con 0x7f6fbc0c6260 2024-09-16T01:43:56.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.002+0000 7f6fc392a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6fbc15bc60 con 0x7f6fa8004d40 2024-09-16T01:43:56.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.002+0000 7f6fc0ec5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fbc0c6fa0 0x7f6fbc14f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:56.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.002+0000 7f6fc0ec5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fbc0c6fa0 0x7f6fbc14f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38230/0 (socket says 172.21.15.88:38230) 2024-09-16T01:43:56.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.002+0000 7f6fc0ec5640 1 -- 172.21.15.88:0/3603435811 learned_addr learned my addr 172.21.15.88:0/3603435811 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:56.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.002+0000 7f6fc1ec7640 1 --2- 172.21.15.88:0/3603435811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fbc0c6260 0x7f6fbc0b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:56.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.003+0000 7f6fc16c6640 1 --2- 172.21.15.88:0/3603435811 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fa8004d40 0x7f6fbc0aae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:56.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.003+0000 7f6fc0ec5640 1 -- 172.21.15.88:0/3603435811 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fa8004d40 msgr2=0x7f6fbc0aae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:56.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.003+0000 7f6fc0ec5640 1 --2- 172.21.15.88:0/3603435811 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fa8004d40 0x7f6fbc0aae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:56.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.003+0000 7f6fc0ec5640 1 -- 172.21.15.88:0/3603435811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fbc0c6260 msgr2=0x7f6fbc0b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:56.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.003+0000 7f6fc0ec5640 1 --2- 172.21.15.88:0/3603435811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fbc0c6260 0x7f6fbc0b3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:56.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.003+0000 7f6fc0ec5640 1 -- 172.21.15.88:0/3603435811 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fbc14f5e0 con 0x7f6fbc0c6fa0 2024-09-16T01:43:56.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.003+0000 7f6fc0ec5640 1 --2- 172.21.15.88:0/3603435811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fbc0c6fa0 0x7f6fbc14f0a0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f6fac011640 tx=0x7f6fac011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:56.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.003+0000 7f6fb1ffb640 1 -- 172.21.15.88:0/3603435811 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fac004220 con 0x7f6fbc0c6fa0 2024-09-16T01:43:56.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.003+0000 7f6fb1ffb640 1 -- 172.21.15.88:0/3603435811 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fac0043c0 con 0x7f6fbc0c6fa0 2024-09-16T01:43:56.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.003+0000 7f6fc392a640 1 -- 172.21.15.88:0/3603435811 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fbc0c7400 con 0x7f6fbc0c6fa0 2024-09-16T01:43:56.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.003+0000 7f6fb1ffb640 1 -- 172.21.15.88:0/3603435811 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fac00f940 con 0x7f6fbc0c6fa0 2024-09-16T01:43:56.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.003+0000 7f6fc392a640 1 -- 172.21.15.88:0/3603435811 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fbc0c5780 con 0x7f6fbc0c6fa0 2024-09-16T01:43:56.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.004+0000 7f6fc392a640 1 -- 172.21.15.88:0/3603435811 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fbc071ff0 con 0x7f6fbc0c6fa0 2024-09-16T01:43:56.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.005+0000 7f6fb1ffb640 1 -- 172.21.15.88:0/3603435811 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6fac00fae0 con 0x7f6fbc0c6fa0 2024-09-16T01:43:56.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.005+0000 7f6fb1ffb640 1 --2- 172.21.15.88:0/3603435811 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6fa006c2e0 0x7f6fa006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:56.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.005+0000 7f6fb1ffb640 1 -- 172.21.15.88:0/3603435811 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8662+0+0 (secure 0 0 0) 0x7f6fac08c1d0 con 0x7f6fbc0c6fa0 2024-09-16T01:43:56.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.005+0000 7f6fc16c6640 1 --2- 172.21.15.88:0/3603435811 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6fa006c2e0 0x7f6fa006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:56.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.006+0000 7f6fc16c6640 1 --2- 172.21.15.88:0/3603435811 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6fa006c2e0 0x7f6fa006e7a0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8001050 tx=0x7f6fb8007970 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:56.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.009+0000 7f6fb1ffb640 1 -- 172.21.15.88:0/3603435811 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6fbc14f5e0 con 0x7f6fbc0c6fa0 2024-09-16T01:43:56.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.186+0000 7f6fc392a640 1 -- 172.21.15.88:0/3603435811 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6fbc15d160 con 0x7f6fbc0c6fa0 2024-09-16T01:43:56.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.186+0000 7f6fb1ffb640 1 -- 172.21.15.88:0/3603435811 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 68 entries v121) ==== 81+0+4136 (secure 0 0 0) 0x7f6fac019030 con 0x7f6fbc0c6fa0 2024-09-16T01:43:56.189 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/4135629195 2024-09-17T01:43:45.866139+0000 2024-09-16T01:43:56.189 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/4135629195 2024-09-17T01:43:45.866139+0000 2024-09-16T01:43:56.189 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/3121519215 2024-09-17T01:43:30.669208+0000 2024-09-16T01:43:56.189 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/3121519215 2024-09-17T01:43:30.669208+0000 2024-09-16T01:43:56.189 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2555776364 2024-09-17T01:43:25.635976+0000 2024-09-16T01:43:56.189 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/1372908065 2024-09-17T01:43:12.443265+0000 2024-09-16T01:43:56.189 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/1372908065 2024-09-17T01:43:12.443265+0000 2024-09-16T01:43:56.189 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/650399848 2024-09-17T01:43:08.279879+0000 2024-09-16T01:43:56.189 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/1255135754 2024-09-17T01:41:41.905258+0000 2024-09-16T01:43:56.189 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/3728363359 2024-09-17T01:42:05.412133+0000 2024-09-16T01:43:56.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/293764774 2024-09-17T01:41:16.203758+0000 2024-09-16T01:43:56.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/1255135754 2024-09-17T01:41:41.905258+0000 2024-09-16T01:43:56.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2918194823 2024-09-17T01:41:29.275696+0000 2024-09-16T01:43:56.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/3336935653 2024-09-17T01:41:44.295023+0000 2024-09-16T01:43:56.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/981209627 2024-09-17T01:41:51.268495+0000 2024-09-16T01:43:56.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/293764774 2024-09-17T01:41:16.203758+0000 2024-09-16T01:43:56.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2680708900 2024-09-17T01:42:41.629904+0000 2024-09-16T01:43:56.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2957965650 2024-09-17T01:41:12.181485+0000 2024-09-16T01:43:56.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/3336935653 2024-09-17T01:41:44.295023+0000 2024-09-16T01:43:56.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2285699282 2024-09-17T01:42:29.541113+0000 2024-09-16T01:43:56.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/1162318664 2024-09-17T01:41:23.257870+0000 2024-09-16T01:43:56.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/128058689 2024-09-17T01:41:10.163743+0000 2024-09-16T01:43:56.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/981209627 2024-09-17T01:41:51.268495+0000 2024-09-16T01:43:56.190 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/128058689 2024-09-17T01:41:10.163743+0000 2024-09-16T01:43:56.191 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/1042715824 2024-09-17T01:40:59.080515+0000 2024-09-16T01:43:56.191 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2957965650 2024-09-17T01:41:12.181485+0000 2024-09-16T01:43:56.191 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2970185561 2024-09-17T01:40:38.255774+0000 2024-09-16T01:43:56.191 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/1162318664 2024-09-17T01:41:23.257870+0000 2024-09-16T01:43:56.191 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/1289932173 2024-09-17T01:40:42.399399+0000 2024-09-16T01:43:56.191 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/3356432718 2024-09-17T01:40:32.344252+0000 2024-09-16T01:43:56.191 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2555776364 2024-09-17T01:43:25.635976+0000 2024-09-16T01:43:56.191 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2970185561 2024-09-17T01:40:38.255774+0000 2024-09-16T01:43:56.191 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2733458327 2024-09-17T01:40:27.314712+0000 2024-09-16T01:43:56.191 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/3356432718 2024-09-17T01:40:32.344252+0000 2024-09-16T01:43:56.191 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2918194823 2024-09-17T01:41:29.275696+0000 2024-09-16T01:43:56.191 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/4144876777 2024-09-17T01:43:00.350808+0000 2024-09-16T01:43:56.191 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2408579939 2024-09-17T01:40:46.434492+0000 2024-09-16T01:43:56.191 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/3163343083 2024-09-17T01:42:01.375396+0000 2024-09-16T01:43:56.192 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/4222250252 2024-09-17T01:42:45.673956+0000 2024-09-16T01:43:56.192 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/4162274767 2024-09-17T01:40:11.522091+0000 2024-09-16T01:43:56.192 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2477051134 2024-09-17T01:43:04.380783+0000 2024-09-16T01:43:56.192 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/1289932173 2024-09-17T01:40:42.399399+0000 2024-09-16T01:43:56.192 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2733458327 2024-09-17T01:40:27.314712+0000 2024-09-16T01:43:56.192 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2550498835 2024-09-17T01:41:57.358287+0000 2024-09-16T01:43:56.192 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2550498835 2024-09-17T01:41:57.358287+0000 2024-09-16T01:43:56.192 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2285699282 2024-09-17T01:42:29.541113+0000 2024-09-16T01:43:56.192 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/3163343083 2024-09-17T01:42:01.375396+0000 2024-09-16T01:43:56.192 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/3728363359 2024-09-17T01:42:05.412133+0000 2024-09-16T01:43:56.192 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/880314974 2024-09-17T01:42:21.485748+0000 2024-09-16T01:43:56.192 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/1936493048 2024-09-17T01:42:19.133197+0000 2024-09-16T01:43:56.192 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/1936493048 2024-09-17T01:42:19.133197+0000 2024-09-16T01:43:56.193 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/880314974 2024-09-17T01:42:21.485748+0000 2024-09-16T01:43:56.193 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/3374372758 2024-09-17T01:42:25.506299+0000 2024-09-16T01:43:56.193 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/3374372758 2024-09-17T01:42:25.506299+0000 2024-09-16T01:43:56.193 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/201188184 2024-09-17T01:42:33.276104+0000 2024-09-16T01:43:56.193 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/1990650099 2024-09-17T01:42:37.599896+0000 2024-09-16T01:43:56.193 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/201188184 2024-09-17T01:42:33.276104+0000 2024-09-16T01:43:56.193 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/1990650099 2024-09-17T01:42:37.599896+0000 2024-09-16T01:43:56.193 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/4162274767 2024-09-17T01:40:11.522091+0000 2024-09-16T01:43:56.193 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2408579939 2024-09-17T01:40:46.434492+0000 2024-09-16T01:43:56.193 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2680708900 2024-09-17T01:42:41.629904+0000 2024-09-16T01:43:56.193 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/4222250252 2024-09-17T01:42:45.673956+0000 2024-09-16T01:43:56.193 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/1042715824 2024-09-17T01:40:59.080515+0000 2024-09-16T01:43:56.193 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/4065900771 2024-09-17T01:42:58.329389+0000 2024-09-16T01:43:56.193 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/4065900771 2024-09-17T01:42:58.329389+0000 2024-09-16T01:43:56.194 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/4144876777 2024-09-17T01:43:00.350808+0000 2024-09-16T01:43:56.194 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2477051134 2024-09-17T01:43:04.380783+0000 2024-09-16T01:43:56.194 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/650399848 2024-09-17T01:43:08.279879+0000 2024-09-16T01:43:56.195 INFO:teuthology.orchestra.run.smithi088.stderr:listed 68 entries 2024-09-16T01:43:56.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.188+0000 7f6fc392a640 1 -- 172.21.15.88:0/3603435811 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6fa006c2e0 msgr2=0x7f6fa006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:56.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.188+0000 7f6fc392a640 1 --2- 172.21.15.88:0/3603435811 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6fa006c2e0 0x7f6fa006e7a0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8001050 tx=0x7f6fb8007970 comp rx=0 tx=0).stop 2024-09-16T01:43:56.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.188+0000 7f6fc392a640 1 -- 172.21.15.88:0/3603435811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fbc0c6fa0 msgr2=0x7f6fbc14f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:56.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.188+0000 7f6fc392a640 1 --2- 172.21.15.88:0/3603435811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fbc0c6fa0 0x7f6fbc14f0a0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f6fac011640 tx=0x7f6fac011b00 comp rx=0 tx=0).stop 2024-09-16T01:43:56.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.188+0000 7f6fc392a640 1 -- 172.21.15.88:0/3603435811 shutdown_connections 2024-09-16T01:43:56.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.188+0000 7f6fc392a640 1 --2- 172.21.15.88:0/3603435811 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6fa006c2e0 0x7f6fa006e7a0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:56.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.188+0000 7f6fc392a640 1 --2- 172.21.15.88:0/3603435811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fbc0c6260 0x7f6fbc0b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:56.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.188+0000 7f6fc392a640 1 --2- 172.21.15.88:0/3603435811 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6fbc0c6fa0 0x7f6fbc14f0a0 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:56.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.188+0000 7f6fc392a640 1 --2- 172.21.15.88:0/3603435811 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fa8004d40 0x7f6fbc0aae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:56.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.188+0000 7f6fc392a640 1 -- 172.21.15.88:0/3603435811 >> 172.21.15.88:0/3603435811 conn(0x7f6fbc06d0f0 msgr2=0x7f6fbc15d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:56.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.188+0000 7f6fc392a640 1 -- 172.21.15.88:0/3603435811 shutdown_connections 2024-09-16T01:43:56.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.189+0000 7f6fc392a640 1 -- 172.21.15.88:0/3603435811 wait complete. 2024-09-16T01:43:56.202 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-16T01:43:56.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:56.202 DEBUG:teuthology.orchestra.run.smithi088:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-09-16T01:43:56.235 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:43:56.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:43:56.235 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:43:56.291 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:43:56.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.429+0000 7f5b8fdcf640 1 Processor -- start 2024-09-16T01:43:56.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.429+0000 7f5b8fdcf640 1 -- start start 2024-09-16T01:43:56.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.430+0000 7f5b8fdcf640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5b880720d0 0x7f5b880b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:56.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.430+0000 7f5b8fdcf640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b88071720 con 0x7f5b8815ad10 2024-09-16T01:43:56.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.430+0000 7f5b8fdcf640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b88073920 con 0x7f5b880cca40 2024-09-16T01:43:56.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.430+0000 7f5b8fdcf640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f5b88070fb0 con 0x7f5b880720d0 2024-09-16T01:43:56.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.430+0000 7f5b8d36a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5b880720d0 0x7f5b880b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:56.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.430+0000 7f5b8db6b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5b8815ad10 0x7f5b880719c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49304/0 (socket says 172.21.15.88:49304) 2024-09-16T01:43:56.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.430+0000 7f5b8d36a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5b880720d0 0x7f5b880b3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38232/0 (socket says 172.21.15.88:38232) 2024-09-16T01:43:56.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.430+0000 7f5b8db6b640 1 -- 172.21.15.88:0/215063282 learned_addr learned my addr 172.21.15.88:0/215063282 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:56.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.430+0000 7f5b8d36a640 1 -- 172.21.15.88:0/215063282 >> v1:172.21.15.154:6789/0 conn(0x7f5b880cca40 legacy=0x7f5b880ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:56.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.430+0000 7f5b8d36a640 1 -- 172.21.15.88:0/215063282 >> v1:172.21.15.88:6789/0 conn(0x7f5b8815ad10 legacy=0x7f5b880719c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:56.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.430+0000 7f5b8cb69640 1 -- 172.21.15.88:0/215063282 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3804107542 0 0) 0x7f5b88071720 con 0x7f5b8815ad10 2024-09-16T01:43:56.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.430+0000 7f5b8d36a640 1 -- 172.21.15.88:0/215063282 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b88073c00 con 0x7f5b880720d0 2024-09-16T01:43:56.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.430+0000 7f5b8d36a640 1 --2- 172.21.15.88:0/215063282 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b880720d0 0x7f5b880b3ea0 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f5b780097a0 tx=0x7f5b7802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=254a6c9fdd1d86f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:56.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.430+0000 7f5b8cb69640 1 -- 172.21.15.88:0/215063282 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b78037070 con 0x7f5b880720d0 2024-09-16T01:43:56.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.430+0000 7f5b8cb69640 1 -- 172.21.15.88:0/215063282 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b7802ae50 con 0x7f5b880720d0 2024-09-16T01:43:56.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.430+0000 7f5b8cb69640 1 -- 172.21.15.88:0/215063282 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b78032660 con 0x7f5b880720d0 2024-09-16T01:43:56.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.432+0000 7f5b8fdcf640 1 -- 172.21.15.88:0/215063282 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b880720d0 msgr2=0x7f5b880b3ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:56.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.432+0000 7f5b8fdcf640 1 --2- 172.21.15.88:0/215063282 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b880720d0 0x7f5b880b3ea0 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f5b780097a0 tx=0x7f5b7802a140 comp rx=0 tx=0).stop 2024-09-16T01:43:56.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.432+0000 7f5b8fdcf640 1 -- 172.21.15.88:0/215063282 shutdown_connections 2024-09-16T01:43:56.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.432+0000 7f5b8fdcf640 1 --2- 172.21.15.88:0/215063282 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b880720d0 0x7f5b880b3ea0 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:56.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.432+0000 7f5b8fdcf640 1 -- 172.21.15.88:0/215063282 >> 172.21.15.88:0/215063282 conn(0x7f5b8806d0f0 msgr2=0x7f5b8806d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:56.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.432+0000 7f5b8fdcf640 1 -- 172.21.15.88:0/215063282 shutdown_connections 2024-09-16T01:43:56.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.432+0000 7f5b8fdcf640 1 -- 172.21.15.88:0/215063282 wait complete. 2024-09-16T01:43:56.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.433+0000 7f5b8fdcf640 1 Processor -- start 2024-09-16T01:43:56.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.433+0000 7f5b8fdcf640 1 -- start start 2024-09-16T01:43:56.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.434+0000 7f5b8fdcf640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b880720d0 0x7f5b881f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:56.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.434+0000 7f5b8fdcf640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b880cca40 0x7f5b881543c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:56.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.434+0000 7f5b8fdcf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b8815ad10 0x7f5b88154900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:56.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.434+0000 7f5b8fdcf640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5b880cc360 con 0x7f5b880720d0 2024-09-16T01:43:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.434+0000 7f5b8fdcf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5b88071200 con 0x7f5b8815ad10 2024-09-16T01:43:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.434+0000 7f5b8fdcf640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5b880cc4e0 con 0x7f5b880cca40 2024-09-16T01:43:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.434+0000 7f5b8db6b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b880720d0 0x7f5b881f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.434+0000 7f5b8db6b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b880720d0 0x7f5b881f3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38244/0 (socket says 172.21.15.88:38244) 2024-09-16T01:43:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.434+0000 7f5b8db6b640 1 -- 172.21.15.88:0/1953043576 learned_addr learned my addr 172.21.15.88:0/1953043576 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.434+0000 7f5b8db6b640 1 -- 172.21.15.88:0/1953043576 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b880cca40 msgr2=0x7f5b881543c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.434+0000 7f5b8db6b640 1 --2- 172.21.15.88:0/1953043576 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b880cca40 0x7f5b881543c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.434+0000 7f5b8db6b640 1 -- 172.21.15.88:0/1953043576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b8815ad10 msgr2=0x7f5b88154900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.434+0000 7f5b8e36c640 1 --2- 172.21.15.88:0/1953043576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b8815ad10 0x7f5b88154900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.434+0000 7f5b8db6b640 1 --2- 172.21.15.88:0/1953043576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b8815ad10 0x7f5b88154900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.434+0000 7f5b8db6b640 1 -- 172.21.15.88:0/1953043576 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b8815bfc0 con 0x7f5b880720d0 2024-09-16T01:43:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.434+0000 7f5b8db6b640 1 --2- 172.21.15.88:0/1953043576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b880720d0 0x7f5b881f3d00 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f5b84005880 tx=0x7f5b8400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:56.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.435+0000 7f5b7e7fc640 1 -- 172.21.15.88:0/1953043576 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b84017070 con 0x7f5b880720d0 2024-09-16T01:43:56.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.435+0000 7f5b7e7fc640 1 -- 172.21.15.88:0/1953043576 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b84005ce0 con 0x7f5b880720d0 2024-09-16T01:43:56.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.435+0000 7f5b8fdcf640 1 -- 172.21.15.88:0/1953043576 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b880ca3c0 con 0x7f5b880720d0 2024-09-16T01:43:56.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.435+0000 7f5b7e7fc640 1 -- 172.21.15.88:0/1953043576 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b84007270 con 0x7f5b880720d0 2024-09-16T01:43:56.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.435+0000 7f5b8fdcf640 1 -- 172.21.15.88:0/1953043576 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b880ca980 con 0x7f5b880720d0 2024-09-16T01:43:56.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.435+0000 7f5b8fdcf640 1 -- 172.21.15.88:0/1953043576 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b880b4990 con 0x7f5b880720d0 2024-09-16T01:43:56.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.436+0000 7f5b7e7fc640 1 -- 172.21.15.88:0/1953043576 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5b8401a070 con 0x7f5b880720d0 2024-09-16T01:43:56.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.436+0000 7f5b7e7fc640 1 --2- 172.21.15.88:0/1953043576 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5b6c06c2e0 0x7f5b6c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:56.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.437+0000 7f5b7e7fc640 1 -- 172.21.15.88:0/1953043576 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 8662+0+0 (secure 0 0 0) 0x7f5b8408b270 con 0x7f5b880720d0 2024-09-16T01:43:56.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.437+0000 7f5b8d36a640 1 --2- 172.21.15.88:0/1953043576 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5b6c06c2e0 0x7f5b6c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:56.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.437+0000 7f5b8d36a640 1 --2- 172.21.15.88:0/1953043576 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5b6c06c2e0 0x7f5b6c06e7a0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f5b78034000 tx=0x7f5b7802c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:56.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.440+0000 7f5b7e7fc640 1 -- 172.21.15.88:0/1953043576 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5b840587b0 con 0x7f5b880720d0 2024-09-16T01:43:56.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.640+0000 7f5b8fdcf640 1 -- 172.21.15.88:0/1953043576 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f5b880cd6c0 con 0x7f5b880720d0 2024-09-16T01:43:56.642 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:43:56.641+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:43:56.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.642+0000 7f5b7e7fc640 1 -- 172.21.15.88:0/1953043576 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v283) ==== 161+0+0 (secure 0 0 0) 0x7f5b8405cab0 con 0x7f5b880720d0 2024-09-16T01:43:56.644 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:43:56.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.645+0000 7f5b8fdcf640 1 -- 172.21.15.88:0/1953043576 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5b6c06c2e0 msgr2=0x7f5b6c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:56.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.645+0000 7f5b8fdcf640 1 --2- 172.21.15.88:0/1953043576 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5b6c06c2e0 0x7f5b6c06e7a0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f5b78034000 tx=0x7f5b7802c000 comp rx=0 tx=0).stop 2024-09-16T01:43:56.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.645+0000 7f5b8fdcf640 1 -- 172.21.15.88:0/1953043576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b880720d0 msgr2=0x7f5b881f3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:56.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.645+0000 7f5b8fdcf640 1 --2- 172.21.15.88:0/1953043576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b880720d0 0x7f5b881f3d00 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f5b84005880 tx=0x7f5b8400c9b0 comp rx=0 tx=0).stop 2024-09-16T01:43:56.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.646+0000 7f5b8fdcf640 1 -- 172.21.15.88:0/1953043576 shutdown_connections 2024-09-16T01:43:56.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.646+0000 7f5b8fdcf640 1 --2- 172.21.15.88:0/1953043576 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5b6c06c2e0 0x7f5b6c06e7a0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:56.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.646+0000 7f5b8fdcf640 1 --2- 172.21.15.88:0/1953043576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b8815ad10 0x7f5b88154900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:56.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.646+0000 7f5b8fdcf640 1 --2- 172.21.15.88:0/1953043576 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b880cca40 0x7f5b881543c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:56.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.646+0000 7f5b8fdcf640 1 --2- 172.21.15.88:0/1953043576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b880720d0 0x7f5b881f3d00 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:56.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.646+0000 7f5b8fdcf640 1 -- 172.21.15.88:0/1953043576 >> 172.21.15.88:0/1953043576 conn(0x7f5b8806d0f0 msgr2=0x7f5b880cf7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:56.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.646+0000 7f5b8fdcf640 1 -- 172.21.15.88:0/1953043576 shutdown_connections 2024-09-16T01:43:56.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.646+0000 7f5b8fdcf640 1 -- 172.21.15.88:0/1953043576 wait complete. 2024-09-16T01:43:56.648 INFO:tasks.ceph.mds.a.smithi088.stderr: -631> 2024-09-16T01:43:51.882+0000 7f5e6dfb8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:43:56.649 INFO:tasks.ceph.mds.a.smithi088.stderr: -146> 2024-09-16T01:43:55.746+0000 7f5e69fb0640 -1 mds.0.cache.dir(0x10000000000) Corrupt fnode in dirfrag 0x10000000000: Decoder at 'void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&)' v=3 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-09-16T01:43:56.659 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:43:56.674 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:43:56.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.758+0000 7f3cbdf59640 1 Processor -- start 2024-09-16T01:43:56.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.758+0000 7f3cbdf59640 1 -- start start 2024-09-16T01:43:56.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.759+0000 7f3cbdf59640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3cb815d680 0x7f3cb806f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:56.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.759+0000 7f3cbdf59640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3cb806fc50 0x7f3cb80cf150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:56.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.759+0000 7f3cbdf59640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cb806cc20 con 0x7f3cb815c290 2024-09-16T01:43:56.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.759+0000 7f3cbdf59640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f3cb806bc10 con 0x7f3cb815d680 2024-09-16T01:43:56.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.759+0000 7f3cbdf59640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3cb8070080 con 0x7f3cb806fc50 2024-09-16T01:43:56.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.759+0000 7f3caffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3cb815d680 0x7f3cb806f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:56.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.759+0000 7f3caffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3cb815d680 0x7f3cb806f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38272/0 (socket says 172.21.15.88:38272) 2024-09-16T01:43:56.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.759+0000 7f3caffff640 1 -- 172.21.15.88:0/735109463 learned_addr learned my addr 172.21.15.88:0/735109463 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:56.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.759+0000 7f3cb77fe640 1 --1- 172.21.15.88:0/735109463 >> v1:172.21.15.88:6789/0 conn(0x7f3cb815c290 0x7f3cb806d500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49322/0 (socket says 172.21.15.88:49322) 2024-09-16T01:43:56.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.759+0000 7f3cb7fff640 1 --2- 172.21.15.88:0/735109463 >> v2:172.21.15.154:3300/0 conn(0x7f3cb806fc50 0x7f3cb80cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:56.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.759+0000 7f3caffff640 1 -- 172.21.15.88:0/735109463 >> v2:172.21.15.154:3300/0 conn(0x7f3cb806fc50 msgr2=0x7f3cb80cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:56.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.759+0000 7f3caffff640 1 --2- 172.21.15.88:0/735109463 >> v2:172.21.15.154:3300/0 conn(0x7f3cb806fc50 0x7f3cb80cf150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:56.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.759+0000 7f3caffff640 1 -- 172.21.15.88:0/735109463 >> v1:172.21.15.88:6789/0 conn(0x7f3cb815c290 legacy=0x7f3cb806d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:56.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.759+0000 7f3caffff640 1 -- 172.21.15.88:0/735109463 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cb806be60 con 0x7f3cb815d680 2024-09-16T01:43:56.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.759+0000 7f3cb6ffd640 1 -- 172.21.15.88:0/735109463 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3185932239 0 0) 0x7f3cb806cc20 con 0x7f3cb815c290 2024-09-16T01:43:56.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.760+0000 7f3caffff640 1 --2- 172.21.15.88:0/735109463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3cb815d680 0x7f3cb806f710 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f3cb80ce700 tx=0x7f3ca002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=697c2e2b89874f85 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:56.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.760+0000 7f3cb6ffd640 1 -- 172.21.15.88:0/735109463 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca0037070 con 0x7f3cb815d680 2024-09-16T01:43:56.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.760+0000 7f3cb6ffd640 1 -- 172.21.15.88:0/735109463 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ca0032440 con 0x7f3cb815d680 2024-09-16T01:43:56.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.760+0000 7f3cb6ffd640 1 -- 172.21.15.88:0/735109463 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca0032810 con 0x7f3cb815d680 2024-09-16T01:43:56.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.761+0000 7f3cbdf59640 1 -- 172.21.15.88:0/735109463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3cb815d680 msgr2=0x7f3cb806f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:56.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.761+0000 7f3cbdf59640 1 --2- 172.21.15.88:0/735109463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3cb815d680 0x7f3cb806f710 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f3cb80ce700 tx=0x7f3ca002a1e0 comp rx=0 tx=0).stop 2024-09-16T01:43:56.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.761+0000 7f3cbdf59640 1 -- 172.21.15.88:0/735109463 shutdown_connections 2024-09-16T01:43:56.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.761+0000 7f3cbdf59640 1 --2- 172.21.15.88:0/735109463 >> v2:172.21.15.154:3300/0 conn(0x7f3cb806fc50 0x7f3cb80cf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:56.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.761+0000 7f3cbdf59640 1 --2- 172.21.15.88:0/735109463 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3cb815d680 0x7f3cb806f710 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:56.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.761+0000 7f3cbdf59640 1 -- 172.21.15.88:0/735109463 >> 172.21.15.88:0/735109463 conn(0x7f3cb80b3070 msgr2=0x7f3cb80b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:56.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.761+0000 7f3cbdf59640 1 -- 172.21.15.88:0/735109463 shutdown_connections 2024-09-16T01:43:56.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.761+0000 7f3cbdf59640 1 -- 172.21.15.88:0/735109463 wait complete. 2024-09-16T01:43:56.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.762+0000 7f3cbdf59640 1 Processor -- start 2024-09-16T01:43:56.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.762+0000 7f3cbdf59640 1 -- start start 2024-09-16T01:43:56.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.763+0000 7f3cbdf59640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cb806fc50 0x7f3cb8155a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:56.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.763+0000 7f3cbdf59640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cb815c290 0x7f3cb814c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:56.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.763+0000 7f3cbdf59640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3cb815d680 0x7f3cb814f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:56.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.763+0000 7f3cbdf59640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb806c700 con 0x7f3cb815d680 2024-09-16T01:43:56.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.763+0000 7f3cbdf59640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb806c420 con 0x7f3cb806fc50 2024-09-16T01:43:56.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.763+0000 7f3cbdf59640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3cb815c670 con 0x7f3cb815c290 2024-09-16T01:43:56.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.763+0000 7f3cb7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3cb815d680 0x7f3cb814f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:56.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.763+0000 7f3cb7fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3cb815d680 0x7f3cb814f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38288/0 (socket says 172.21.15.88:38288) 2024-09-16T01:43:56.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.763+0000 7f3cb7fff640 1 -- 172.21.15.88:0/3995836247 learned_addr learned my addr 172.21.15.88:0/3995836247 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:56.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.763+0000 7f3cb77fe640 1 --2- 172.21.15.88:0/3995836247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cb806fc50 0x7f3cb8155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:56.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.763+0000 7f3caffff640 1 --2- 172.21.15.88:0/3995836247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cb815c290 0x7f3cb814c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:56.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.763+0000 7f3cb7fff640 1 -- 172.21.15.88:0/3995836247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cb815c290 msgr2=0x7f3cb814c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:56.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.763+0000 7f3cb7fff640 1 --2- 172.21.15.88:0/3995836247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cb815c290 0x7f3cb814c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:56.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.763+0000 7f3cb7fff640 1 -- 172.21.15.88:0/3995836247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cb806fc50 msgr2=0x7f3cb8155a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:56.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.763+0000 7f3cb7fff640 1 --2- 172.21.15.88:0/3995836247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cb806fc50 0x7f3cb8155a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:56.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.763+0000 7f3cb7fff640 1 -- 172.21.15.88:0/3995836247 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cb814f6e0 con 0x7f3cb815d680 2024-09-16T01:43:56.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.763+0000 7f3cb77fe640 1 --2- 172.21.15.88:0/3995836247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cb806fc50 0x7f3cb8155a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:43:56.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.763+0000 7f3cb7fff640 1 --2- 172.21.15.88:0/3995836247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3cb815d680 0x7f3cb814f1a0 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f3cb000efc0 tx=0x7f3cb000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:56.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.764+0000 7f3cb4ff9640 1 -- 172.21.15.88:0/3995836247 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0018020 con 0x7f3cb815d680 2024-09-16T01:43:56.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.764+0000 7f3cb4ff9640 1 -- 172.21.15.88:0/3995836247 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cb0014480 con 0x7f3cb815d680 2024-09-16T01:43:56.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.764+0000 7f3cbdf59640 1 -- 172.21.15.88:0/3995836247 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cb814f980 con 0x7f3cb815d680 2024-09-16T01:43:56.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.764+0000 7f3cb4ff9640 1 -- 172.21.15.88:0/3995836247 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0014870 con 0x7f3cb815d680 2024-09-16T01:43:56.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.764+0000 7f3cbdf59640 1 -- 172.21.15.88:0/3995836247 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cb814ff40 con 0x7f3cb815d680 2024-09-16T01:43:56.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.765+0000 7f3cbdf59640 1 -- 172.21.15.88:0/3995836247 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cb806f710 con 0x7f3cb815d680 2024-09-16T01:43:56.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.765+0000 7f3cb4ff9640 1 -- 172.21.15.88:0/3995836247 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3cb0005d10 con 0x7f3cb815d680 2024-09-16T01:43:56.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.766+0000 7f3cb4ff9640 1 --2- 172.21.15.88:0/3995836247 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3c9806c2b0 0x7f3c9806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:56.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.766+0000 7f3cb4ff9640 1 -- 172.21.15.88:0/3995836247 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 8748+0+0 (secure 0 0 0) 0x7f3cb008cc00 con 0x7f3cb815d680 2024-09-16T01:43:56.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.766+0000 7f3cb77fe640 1 --2- 172.21.15.88:0/3995836247 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3c9806c2b0 0x7f3c9806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:56.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.766+0000 7f3cb77fe640 1 --2- 172.21.15.88:0/3995836247 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3c9806c2b0 0x7f3c9806e770 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8000f60 tx=0x7f3ca8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:56.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:56.770+0000 7f3cb4ff9640 1 -- 172.21.15.88:0/3995836247 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3cb0093080 con 0x7f3cb815d680 2024-09-16T01:43:57.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.002+0000 7f3cbdf59640 1 -- 172.21.15.88:0/3995836247 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f3cb80d0420 con 0x7f3cb815d680 2024-09-16T01:43:57.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.647+0000 7f3cb4ff9640 1 -- 172.21.15.88:0/3995836247 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v284) ==== 107+0+0 (secure 0 0 0) 0x7f3cb0022070 con 0x7f3cb815d680 2024-09-16T01:43:57.648 INFO:teuthology.orchestra.run.smithi088.stderr:nothing to do: rank is not damaged 2024-09-16T01:43:57.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.649+0000 7f3cbdf59640 1 -- 172.21.15.88:0/3995836247 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3c9806c2b0 msgr2=0x7f3c9806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:57.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.649+0000 7f3cbdf59640 1 --2- 172.21.15.88:0/3995836247 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3c9806c2b0 0x7f3c9806e770 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8000f60 tx=0x7f3ca8009040 comp rx=0 tx=0).stop 2024-09-16T01:43:57.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.649+0000 7f3cbdf59640 1 -- 172.21.15.88:0/3995836247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3cb815d680 msgr2=0x7f3cb814f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:57.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.649+0000 7f3cbdf59640 1 --2- 172.21.15.88:0/3995836247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3cb815d680 0x7f3cb814f1a0 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f3cb000efc0 tx=0x7f3cb000c490 comp rx=0 tx=0).stop 2024-09-16T01:43:57.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.650+0000 7f3cbdf59640 1 -- 172.21.15.88:0/3995836247 shutdown_connections 2024-09-16T01:43:57.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.650+0000 7f3cbdf59640 1 --2- 172.21.15.88:0/3995836247 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3c9806c2b0 0x7f3c9806e770 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.650+0000 7f3cbdf59640 1 --2- 172.21.15.88:0/3995836247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3cb815d680 0x7f3cb814f1a0 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.650+0000 7f3cbdf59640 1 --2- 172.21.15.88:0/3995836247 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cb815c290 0x7f3cb814c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.650+0000 7f3cbdf59640 1 --2- 172.21.15.88:0/3995836247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cb806fc50 0x7f3cb8155a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.650+0000 7f3cbdf59640 1 -- 172.21.15.88:0/3995836247 >> 172.21.15.88:0/3995836247 conn(0x7f3cb80b3070 msgr2=0x7f3cb80d1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:57.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.650+0000 7f3cbdf59640 1 -- 172.21.15.88:0/3995836247 shutdown_connections 2024-09-16T01:43:57.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.650+0000 7f3cbdf59640 1 -- 172.21.15.88:0/3995836247 wait complete. 2024-09-16T01:43:57.665 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:43:57.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.752+0000 7fa430442f40 1 Processor -- start 2024-09-16T01:43:57.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.752+0000 7fa430442f40 1 -- start start 2024-09-16T01:43:57.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.753+0000 7fa430442f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562ed29bd4d0 0x562ed2918810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:57.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.753+0000 7fa430442f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x562ed2936120 0x562ed29c3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:57.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.753+0000 7fa430442f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ed29c3940 0x562ed2934c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:57.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.753+0000 7fa430442f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x562ed283f510 con 0x562ed29bd4d0 2024-09-16T01:43:57.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.753+0000 7fa430442f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562ed2919ec0 con 0x562ed29c3940 2024-09-16T01:43:57.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.753+0000 7fa430442f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x562ed29c8af0 con 0x562ed2936120 2024-09-16T01:43:57.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.753+0000 7fa42df38640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562ed29bd4d0 0x562ed2918810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:57.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.753+0000 7fa42df38640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562ed29bd4d0 0x562ed2918810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38304/0 (socket says 172.21.15.88:38304) 2024-09-16T01:43:57.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.753+0000 7fa42df38640 1 -- 172.21.15.88:0/418469629 learned_addr learned my addr 172.21.15.88:0/418469629 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:57.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.753+0000 7fa427fff640 1 --2- 172.21.15.88:0/418469629 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x562ed2936120 0x562ed29c3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:57.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.753+0000 7fa42e739640 1 --2- 172.21.15.88:0/418469629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ed29c3940 0x562ed2934c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:57.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.753+0000 7fa42df38640 1 -- 172.21.15.88:0/418469629 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x562ed2936120 msgr2=0x562ed29c3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:57.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.753+0000 7fa42df38640 1 --2- 172.21.15.88:0/418469629 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x562ed2936120 0x562ed29c3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.753+0000 7fa42df38640 1 -- 172.21.15.88:0/418469629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ed29c3940 msgr2=0x562ed2934c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:57.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.753+0000 7fa42df38640 1 --2- 172.21.15.88:0/418469629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ed29c3940 0x562ed2934c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.753+0000 7fa42df38640 1 -- 172.21.15.88:0/418469629 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ed291c940 con 0x562ed29bd4d0 2024-09-16T01:43:57.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.753+0000 7fa42e739640 1 --2- 172.21.15.88:0/418469629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ed29c3940 0x562ed2934c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:57.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.753+0000 7fa42df38640 1 --2- 172.21.15.88:0/418469629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562ed29bd4d0 0x562ed2918810 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fa4200096a0 tx=0x7fa420009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87a71945b83e3547 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:57.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.753+0000 7fa42cf36640 1 -- 172.21.15.88:0/418469629 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42000e9c0 con 0x562ed29bd4d0 2024-09-16T01:43:57.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.754+0000 7fa42cf36640 1 -- 172.21.15.88:0/418469629 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa420013030 con 0x562ed29bd4d0 2024-09-16T01:43:57.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.754+0000 7fa42cf36640 1 -- 172.21.15.88:0/418469629 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42000eda0 con 0x562ed29bd4d0 2024-09-16T01:43:57.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.754+0000 7fa430442f40 1 -- 172.21.15.88:0/418469629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562ed29bd4d0 msgr2=0x562ed2918810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:57.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.754+0000 7fa430442f40 1 --2- 172.21.15.88:0/418469629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562ed29bd4d0 0x562ed2918810 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fa4200096a0 tx=0x7fa420009b60 comp rx=0 tx=0).stop 2024-09-16T01:43:57.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.754+0000 7fa430442f40 1 -- 172.21.15.88:0/418469629 shutdown_connections 2024-09-16T01:43:57.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.754+0000 7fa430442f40 1 --2- 172.21.15.88:0/418469629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ed29c3940 0x562ed2934c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.754+0000 7fa430442f40 1 --2- 172.21.15.88:0/418469629 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x562ed2936120 0x562ed29c3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.754+0000 7fa430442f40 1 --2- 172.21.15.88:0/418469629 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562ed29bd4d0 0x562ed2918810 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.754+0000 7fa430442f40 1 -- 172.21.15.88:0/418469629 >> 172.21.15.88:0/418469629 conn(0x562ed2938750 msgr2=0x562ed291bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:57.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.754+0000 7fa430442f40 1 -- 172.21.15.88:0/418469629 shutdown_connections 2024-09-16T01:43:57.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.754+0000 7fa430442f40 1 -- 172.21.15.88:0/418469629 wait complete. 2024-09-16T01:43:57.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.755+0000 7fa430442f40 1 Processor -- start 2024-09-16T01:43:57.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.755+0000 7fa430442f40 1 -- start start 2024-09-16T01:43:57.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.755+0000 7fa430442f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ed2936120 0x562ed29a42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:57.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.755+0000 7fa430442f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x562ed29bd4d0 0x562ed29b6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:57.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.755+0000 7fa430442f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562ed29c3940 0x562ed29b66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:57.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.755+0000 7fa430442f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x562ed29bd8b0 con 0x562ed29c3940 2024-09-16T01:43:57.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.755+0000 7fa430442f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x562ed291c6f0 con 0x562ed2936120 2024-09-16T01:43:57.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.755+0000 7fa430442f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x562ed29bdbd0 con 0x562ed29bd4d0 2024-09-16T01:43:57.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.756+0000 7fa42e739640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562ed29c3940 0x562ed29b66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:57.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.756+0000 7fa42e739640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562ed29c3940 0x562ed29b66c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38318/0 (socket says 172.21.15.88:38318) 2024-09-16T01:43:57.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.756+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 learned_addr learned my addr 172.21.15.88:0/2204417374 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:57.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.756+0000 7fa427fff640 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x562ed29bd4d0 0x562ed29b6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:57.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.756+0000 7fa42df38640 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ed2936120 0x562ed29a42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:57.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.756+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x562ed29bd4d0 msgr2=0x562ed29b6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:57.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.756+0000 7fa42e739640 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x562ed29bd4d0 0x562ed29b6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.756+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ed2936120 msgr2=0x562ed29a42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:57.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.756+0000 7fa42e739640 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562ed2936120 0x562ed29a42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.756+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ed29bdeb0 con 0x562ed29c3940 2024-09-16T01:43:57.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.756+0000 7fa42e739640 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562ed29c3940 0x562ed29b66c0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7fa428025e40 tx=0x7fa4280033a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:57.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.756+0000 7fa426ffd640 1 -- 172.21.15.88:0/2204417374 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42801a420 con 0x562ed29c3940 2024-09-16T01:43:57.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.756+0000 7fa426ffd640 1 -- 172.21.15.88:0/2204417374 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa428067070 con 0x562ed29c3940 2024-09-16T01:43:57.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.756+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x562ed29be190 con 0x562ed29c3940 2024-09-16T01:43:57.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.756+0000 7fa426ffd640 1 -- 172.21.15.88:0/2204417374 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42800a3b0 con 0x562ed29c3940 2024-09-16T01:43:57.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.756+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ed29a47f0 con 0x562ed29c3940 2024-09-16T01:43:57.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.758+0000 7fa426ffd640 1 -- 172.21.15.88:0/2204417374 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa42804db30 con 0x562ed29c3940 2024-09-16T01:43:57.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.758+0000 7fa426ffd640 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa41006acc0 0x7fa41006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:57.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.758+0000 7fa426ffd640 1 -- 172.21.15.88:0/2204417374 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 8748+0+0 (secure 0 0 0) 0x7fa4280ce4e0 con 0x562ed29c3940 2024-09-16T01:43:57.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.758+0000 7fa42df38640 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa41006acc0 0x7fa41006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:57.765 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:43:57.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.759+0000 7fa430442f40 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x562ed29bec70 0x562ed2935120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:57.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.759+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2935660 con 0x562ed29bec70 2024-09-16T01:43:57.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.759+0000 7fa427fff640 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x562ed29bec70 0x562ed2935120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:57.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.759+0000 7fa42df38640 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa41006acc0 0x7fa41006d180 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fa420009ba0 tx=0x7fa420005000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:57.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.759+0000 7fa427fff640 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x562ed29bec70 0x562ed2935120 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:57.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.761+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v122'1065 uv1038 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.762+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e122) -- 0x562ed2935a20 con 0x562ed29bec70 2024-09-16T01:43:57.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.763+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v122'1066 uv1066 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.763+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a68bb0 con 0x562ed29bec70 2024-09-16T01:43:57.767 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:43:57.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.764+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v122'1067 uv1067 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.764+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a69f60 con 0x562ed29bec70 2024-09-16T01:43:57.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.766+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v122'1068 uv1068 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.766+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6a300 con 0x562ed29bec70 2024-09-16T01:43:57.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.767+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v122'1069 uv1069 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.767+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a68f50 con 0x562ed29bec70 2024-09-16T01:43:57.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.769+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v122'1070 uv1070 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.769+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a69340 con 0x562ed29bec70 2024-09-16T01:43:57.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.770+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v122'1071 uv1044 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.770+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a69730 con 0x562ed29bec70 2024-09-16T01:43:57.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.772+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v122'1072 uv1072 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.772+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a69730 con 0x562ed29bec70 2024-09-16T01:43:57.775 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:43:57.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.773+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v122'1073 uv1073 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.773+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a696e0 con 0x562ed29bec70 2024-09-16T01:43:57.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.775+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v122'1074 uv1074 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.775+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a696e0 con 0x562ed29bec70 2024-09-16T01:43:57.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.776+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v122'1075 uv1075 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.776+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a696e0 con 0x562ed29bec70 2024-09-16T01:43:57.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.778+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v122'1076 uv1076 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.778+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a696e0 con 0x562ed29bec70 2024-09-16T01:43:57.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.779+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v122'1077 uv1064 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.779+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a696e0 con 0x562ed29bec70 2024-09-16T01:43:57.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.781+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v122'1078 uv1078 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.781+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a696e0 con 0x562ed29bec70 2024-09-16T01:43:57.783 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:43:57.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.782+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v122'1079 uv1079 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.782+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a696e0 con 0x562ed29bec70 2024-09-16T01:43:57.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.784+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v122'1080 uv1080 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.784+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a696e0 con 0x562ed29bec70 2024-09-16T01:43:57.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.785+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v122'1081 uv1081 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.785+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a696e0 con 0x562ed29bec70 2024-09-16T01:43:57.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.787+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v122'1082 uv1082 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.787+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6a6a0 con 0x562ed29bec70 2024-09-16T01:43:57.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.788+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v122'1083 uv1083 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.788+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6a6a0 con 0x562ed29bec70 2024-09-16T01:43:57.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.790+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v122'1084 uv1057 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.790+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6a6a0 con 0x562ed29bec70 2024-09-16T01:43:57.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.791+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v122'1085 uv1085 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.791+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6a6a0 con 0x562ed29bec70 2024-09-16T01:43:57.794 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:43:57.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.792+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v122'1086 uv1086 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.792+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6b6b0 con 0x562ed29bec70 2024-09-16T01:43:57.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.793+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v122'1087 uv1087 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.793+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6b6b0 con 0x562ed29bec70 2024-09-16T01:43:57.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.794+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v122'1088 uv1088 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.795+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6a6a0 con 0x562ed29bec70 2024-09-16T01:43:57.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.796+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v122'1089 uv1089 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.796+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6a6a0 con 0x562ed29bec70 2024-09-16T01:43:57.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.797+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v122'1090 uv1063 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.798+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6a6a0 con 0x562ed29bec70 2024-09-16T01:43:57.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.799+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v122'1091 uv1091 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.799+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6a6a0 con 0x562ed29bec70 2024-09-16T01:43:57.802 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:43:57.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.800+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v122'1092 uv1092 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.802 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.800+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6b6b0 con 0x562ed29bec70 2024-09-16T01:43:57.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.802+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v122'1093 uv1093 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.802+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6b6b0 con 0x562ed29bec70 2024-09-16T01:43:57.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.803+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v122'1094 uv1094 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.803+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6a6a0 con 0x562ed29bec70 2024-09-16T01:43:57.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.805+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v122'1095 uv1095 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.805+0000 7fa430442f40 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x562ed2a6abb0 0x562ed2a6cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:57.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.805+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6d4e0 con 0x562ed2a6abb0 2024-09-16T01:43:57.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.805+0000 7fa42df38640 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x562ed2a6abb0 0x562ed2a6cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:57.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.805+0000 7fa42df38640 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x562ed2a6abb0 0x562ed2a6cfa0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:57.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.809+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v122'778 uv757 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fa42000ea40 con 0x562ed2a6abb0 2024-09-16T01:43:57.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.809+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6d880 con 0x562ed2a6abb0 2024-09-16T01:43:57.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.810+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v122'779 uv779 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa42000ea40 con 0x562ed2a6abb0 2024-09-16T01:43:57.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.810+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6dc20 con 0x562ed2a6abb0 2024-09-16T01:43:57.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.811+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v122'780 uv780 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa42000ea40 con 0x562ed2a6abb0 2024-09-16T01:43:57.813 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:43:57.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.811+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6e470 con 0x562ed2a6abb0 2024-09-16T01:43:57.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.813+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v122'781 uv781 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa42000ea40 con 0x562ed2a6abb0 2024-09-16T01:43:57.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.813+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6e810 con 0x562ed2a6abb0 2024-09-16T01:43:57.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.814+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v122'782 uv782 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa42000ea40 con 0x562ed2a6abb0 2024-09-16T01:43:57.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.814+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6ebb0 con 0x562ed2a6abb0 2024-09-16T01:43:57.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.816+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v122'783 uv776 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa42000ea40 con 0x562ed2a6abb0 2024-09-16T01:43:57.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.816+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6ef50 con 0x562ed2a6abb0 2024-09-16T01:43:57.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.817+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v122'784 uv784 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa42000ea40 con 0x562ed2a6abb0 2024-09-16T01:43:57.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.817+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6f2f0 con 0x562ed2a6abb0 2024-09-16T01:43:57.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.818+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v122'785 uv785 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa42000ea40 con 0x562ed2a6abb0 2024-09-16T01:43:57.820 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:43:57.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.818+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6f2f0 con 0x562ed2a6abb0 2024-09-16T01:43:57.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.819+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v122'786 uv786 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa42000ea40 con 0x562ed2a6abb0 2024-09-16T01:43:57.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.819+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6f2f0 con 0x562ed2a6abb0 2024-09-16T01:43:57.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.820+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v122'787 uv787 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa42000ea40 con 0x562ed2a6abb0 2024-09-16T01:43:57.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.820+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6f2f0 con 0x562ed2a6abb0 2024-09-16T01:43:57.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.822+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v122'788 uv777 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa42000ea40 con 0x562ed2a6abb0 2024-09-16T01:43:57.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.822+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6f2f0 con 0x562ed2a6abb0 2024-09-16T01:43:57.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.823+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v122'789 uv789 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa42000ea40 con 0x562ed2a6abb0 2024-09-16T01:43:57.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.823+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6f2f0 con 0x562ed2a6abb0 2024-09-16T01:43:57.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.824+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v122'790 uv790 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa42000ea40 con 0x562ed2a6abb0 2024-09-16T01:43:57.826 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:43:57.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.824+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a73770 con 0x562ed2a6abb0 2024-09-16T01:43:57.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.826+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v122'791 uv791 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa42000ea40 con 0x562ed2a6abb0 2024-09-16T01:43:57.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.826+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a73770 con 0x562ed2a6abb0 2024-09-16T01:43:57.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.827+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v122'792 uv792 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa42000ea40 con 0x562ed2a6abb0 2024-09-16T01:43:57.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.827+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6f2f0 con 0x562ed2a6abb0 2024-09-16T01:43:57.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.828+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v122'793 uv773 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa42000ea40 con 0x562ed2a6abb0 2024-09-16T01:43:57.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.828+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6f2f0 con 0x562ed2a6abb0 2024-09-16T01:43:57.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.829+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v122'794 uv794 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa42000ea40 con 0x562ed2a6abb0 2024-09-16T01:43:57.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.829+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6f2f0 con 0x562ed2a6abb0 2024-09-16T01:43:57.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.831+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v122'795 uv795 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa42000ea40 con 0x562ed2a6abb0 2024-09-16T01:43:57.833 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:43:57.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.831+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a70300 con 0x562ed2a6abb0 2024-09-16T01:43:57.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.832+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v122'796 uv796 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa42000ea40 con 0x562ed2a6abb0 2024-09-16T01:43:57.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.832+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a70300 con 0x562ed2a6abb0 2024-09-16T01:43:57.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.833+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v122'797 uv797 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa42000ea40 con 0x562ed2a6abb0 2024-09-16T01:43:57.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.833+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6f2f0 con 0x562ed2a6abb0 2024-09-16T01:43:57.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.834+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v122'798 uv798 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa42000ea40 con 0x562ed2a6abb0 2024-09-16T01:43:57.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.834+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6f2f0 con 0x562ed29bec70 2024-09-16T01:43:57.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.835+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v122'207 uv206 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.835+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6f2f0 con 0x562ed29bec70 2024-09-16T01:43:57.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.836+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v122'208 uv208 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.836+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6f2f0 con 0x562ed29bec70 2024-09-16T01:43:57.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.837+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v122'209 uv209 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.838 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:43:57.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.837+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a70300 con 0x562ed29bec70 2024-09-16T01:43:57.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.838+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v122'210 uv210 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.838+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a70300 con 0x562ed29bec70 2024-09-16T01:43:57.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.839+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v122'211 uv211 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.839+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a6f2f0 con 0x562ed29bec70 2024-09-16T01:43:57.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.840+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v122'212 uv212 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.840+0000 7fa430442f40 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x562ed2a6f800 0x562ed2a71bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:57.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.840+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a72130 con 0x562ed2a6f800 2024-09-16T01:43:57.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.840+0000 7fa427fff640 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x562ed2a6f800 0x562ed2a71bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:57.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.840+0000 7fa427fff640 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x562ed2a6f800 0x562ed2a71bf0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:57.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.841+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v122'157 uv156 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed2a6f800 2024-09-16T01:43:57.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.841+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a721b0 con 0x562ed2a6f800 2024-09-16T01:43:57.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.842+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v122'158 uv158 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed2a6f800 2024-09-16T01:43:57.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.842+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a725a0 con 0x562ed2a6f800 2024-09-16T01:43:57.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.843+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v122'159 uv159 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed2a6f800 2024-09-16T01:43:57.845 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:43:57.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.843+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a725a0 con 0x562ed2a6f800 2024-09-16T01:43:57.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.844+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v122'160 uv160 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed2a6f800 2024-09-16T01:43:57.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.844+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a725a0 con 0x562ed29bec70 2024-09-16T01:43:57.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.846+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v122'207 uv206 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.846+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a725a0 con 0x562ed29bec70 2024-09-16T01:43:57.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.847+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v122'208 uv208 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.847+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a725a0 con 0x562ed29bec70 2024-09-16T01:43:57.850 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:43:57.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.848+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v122'209 uv209 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.848+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a735b0 con 0x562ed29bec70 2024-09-16T01:43:57.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.851+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v122'210 uv210 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.851+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a735b0 con 0x562ed29bec70 2024-09-16T01:43:57.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.852+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v122'211 uv211 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.852+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a725a0 con 0x562ed29bec70 2024-09-16T01:43:57.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.853+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v122'212 uv212 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.853+0000 7fa430442f40 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x562ed2a72c20 0x562ed2a75010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:57.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.853+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a75550 con 0x562ed2a72c20 2024-09-16T01:43:57.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.853+0000 7fa42e739640 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x562ed2a72c20 0x562ed2a75010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:57.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.854+0000 7fa42e739640 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x562ed2a72c20 0x562ed2a75010 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:57.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.855+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v122'382 uv376 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.855+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a759b0 con 0x562ed2a72c20 2024-09-16T01:43:57.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.856+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v122'383 uv383 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.856+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a75e20 con 0x562ed2a72c20 2024-09-16T01:43:57.860 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:43:57.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.859+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v122'384 uv384 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.859+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a776d0 con 0x562ed2a72c20 2024-09-16T01:43:57.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.860+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v122'385 uv385 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.860+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a77a70 con 0x562ed2a72c20 2024-09-16T01:43:57.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.861+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v122'386 uv386 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.861+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a766c0 con 0x562ed2a72c20 2024-09-16T01:43:57.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.862+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v122'387 uv387 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.862+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a76ab0 con 0x562ed2a72c20 2024-09-16T01:43:57.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.863+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v122'388 uv381 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.863+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a76ab0 con 0x562ed2a72c20 2024-09-16T01:43:57.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.864+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v122'389 uv389 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.864+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a76ab0 con 0x562ed2a72c20 2024-09-16T01:43:57.867 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:43:57.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.865+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v122'390 uv390 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.865+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a76ab0 con 0x562ed2a72c20 2024-09-16T01:43:57.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.866+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v122'391 uv391 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.866+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a76ab0 con 0x562ed2a72c20 2024-09-16T01:43:57.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.867+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v122'392 uv392 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.867+0000 7fa430442f40 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x562ed2a77e10 0x562ed2a7a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:57.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.867+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a76ab0 con 0x562ed2a77e10 2024-09-16T01:43:57.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.868+0000 7fa42df38640 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x562ed2a77e10 0x562ed2a7a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:57.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.868+0000 7fa42df38640 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x562ed2a77e10 0x562ed2a7a200 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:57.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.869+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v122'173 uv172 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fa420032040 con 0x562ed2a77e10 2024-09-16T01:43:57.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.869+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a76ab0 con 0x562ed2a77e10 2024-09-16T01:43:57.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.870+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v122'174 uv174 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa420032040 con 0x562ed2a77e10 2024-09-16T01:43:57.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.870+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7a740 con 0x562ed2a77e10 2024-09-16T01:43:57.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.871+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v122'175 uv175 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa420032040 con 0x562ed2a77e10 2024-09-16T01:43:57.873 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:43:57.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.872+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7a740 con 0x562ed2a77e10 2024-09-16T01:43:57.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.873+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v122'176 uv176 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa420032040 con 0x562ed2a77e10 2024-09-16T01:43:57.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.873+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7a740 con 0x562ed2a77e10 2024-09-16T01:43:57.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.874+0000 7fa42df38640 1 -- 172.21.15.88:0/2204417374 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v122'177 uv177 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa420032040 con 0x562ed2a77e10 2024-09-16T01:43:57.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.874+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7a740 con 0x562ed29bec70 2024-09-16T01:43:57.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.875+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v122'207 uv206 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.875+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7a740 con 0x562ed29bec70 2024-09-16T01:43:57.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.876+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v122'208 uv208 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.876+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7a740 con 0x562ed29bec70 2024-09-16T01:43:57.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.876+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v122'209 uv209 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.878 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:43:57.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.877+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7b750 con 0x562ed29bec70 2024-09-16T01:43:57.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.878+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v122'210 uv210 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.878+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7b750 con 0x562ed29bec70 2024-09-16T01:43:57.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.878+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v122'211 uv211 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.878+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7a740 con 0x562ed29bec70 2024-09-16T01:43:57.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.879+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v122'212 uv212 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed29bec70 2024-09-16T01:43:57.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.879+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7a740 con 0x562ed2a6f800 2024-09-16T01:43:57.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.880+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v122'345 uv339 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed2a6f800 2024-09-16T01:43:57.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.880+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7a740 con 0x562ed2a6f800 2024-09-16T01:43:57.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.881+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v122'346 uv346 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed2a6f800 2024-09-16T01:43:57.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.881+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7a740 con 0x562ed2a6f800 2024-09-16T01:43:57.884 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:43:57.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.882+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v122'347 uv347 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed2a6f800 2024-09-16T01:43:57.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.882+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7a740 con 0x562ed2a6f800 2024-09-16T01:43:57.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.883+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v122'348 uv348 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed2a6f800 2024-09-16T01:43:57.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.883+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7a740 con 0x562ed2a6f800 2024-09-16T01:43:57.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.884+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v122'349 uv349 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed2a6f800 2024-09-16T01:43:57.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.884+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7a740 con 0x562ed2a6f800 2024-09-16T01:43:57.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.885+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v122'350 uv344 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed2a6f800 2024-09-16T01:43:57.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.885+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7a740 con 0x562ed2a6f800 2024-09-16T01:43:57.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.886+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v122'351 uv351 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed2a6f800 2024-09-16T01:43:57.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.886+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7a740 con 0x562ed2a6f800 2024-09-16T01:43:57.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.887+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v122'352 uv352 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed2a6f800 2024-09-16T01:43:57.889 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:43:57.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.887+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7a740 con 0x562ed2a6f800 2024-09-16T01:43:57.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.888+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v122'353 uv353 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed2a6f800 2024-09-16T01:43:57.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.888+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7d530 con 0x562ed2a6f800 2024-09-16T01:43:57.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.889+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v122'354 uv354 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed2a6f800 2024-09-16T01:43:57.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.889+0000 7fa430442f40 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x562ed2a7ab30 0x562ed2a7cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:57.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.889+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7dbf0 con 0x562ed2a7ab30 2024-09-16T01:43:57.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.889+0000 7fa427fff640 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x562ed2a7ab30 0x562ed2a7cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:57.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.890+0000 7fa427fff640 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x562ed2a7ab30 0x562ed2a7cf20 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:57.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.891+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v122'152 uv150 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed2a7ab30 2024-09-16T01:43:57.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.891+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7dbf0 con 0x562ed2a7ab30 2024-09-16T01:43:57.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.892+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v122'153 uv153 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed2a7ab30 2024-09-16T01:43:57.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.892+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7dbf0 con 0x562ed2a7ab30 2024-09-16T01:43:57.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.892+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v122'154 uv154 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed2a7ab30 2024-09-16T01:43:57.894 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:43:57.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.892+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7dbf0 con 0x562ed2a7ab30 2024-09-16T01:43:57.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.893+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v122'155 uv155 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed2a7ab30 2024-09-16T01:43:57.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.893+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7dbf0 con 0x562ed2a6f800 2024-09-16T01:43:57.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.895+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v122'207 uv206 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed2a6f800 2024-09-16T01:43:57.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.895+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7db70 con 0x562ed2a6f800 2024-09-16T01:43:57.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.896+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v122'208 uv208 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed2a6f800 2024-09-16T01:43:57.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.896+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7db70 con 0x562ed2a6f800 2024-09-16T01:43:57.898 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:43:57.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.897+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v122'209 uv209 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed2a6f800 2024-09-16T01:43:57.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.897+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7eb80 con 0x562ed2a6f800 2024-09-16T01:43:57.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.898+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v122'210 uv210 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed2a6f800 2024-09-16T01:43:57.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.898+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7eb80 con 0x562ed2a6f800 2024-09-16T01:43:57.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.898+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v122'211 uv211 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed2a6f800 2024-09-16T01:43:57.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.898+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7db70 con 0x562ed2a6f800 2024-09-16T01:43:57.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.900+0000 7fa427fff640 1 -- 172.21.15.88:0/2204417374 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v122'212 uv212 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa418008400 con 0x562ed2a6f800 2024-09-16T01:43:57.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.900+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7db70 con 0x562ed2a72c20 2024-09-16T01:43:57.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.901+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v122'176 uv175 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.901+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7db70 con 0x562ed2a72c20 2024-09-16T01:43:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.902+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v122'177 uv177 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.902+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7db70 con 0x562ed2a72c20 2024-09-16T01:43:57.904 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:43:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.903+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v122'178 uv178 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.903+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7db70 con 0x562ed2a72c20 2024-09-16T01:43:57.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.904+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v122'179 uv179 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.904+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7db70 con 0x562ed2a72c20 2024-09-16T01:43:57.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.905+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v122'180 uv180 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.905+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7db70 con 0x562ed2a72c20 2024-09-16T01:43:57.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.907+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v122'429 uv420 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.907+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7db70 con 0x562ed2a72c20 2024-09-16T01:43:57.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.908+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v122'430 uv430 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.908+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7db70 con 0x562ed2a72c20 2024-09-16T01:43:57.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.909+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v122'431 uv431 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.910 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:43:57.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.909+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7df80 con 0x562ed2a72c20 2024-09-16T01:43:57.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.910+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v122'432 uv432 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.910+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7df30 con 0x562ed2a72c20 2024-09-16T01:43:57.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.911+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v122'433 uv433 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.912+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7df30 con 0x562ed2a72c20 2024-09-16T01:43:57.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.914+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v122'434 uv427 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.914+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7db70 con 0x562ed2a72c20 2024-09-16T01:43:57.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.915+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v122'435 uv435 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.915+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7db70 con 0x562ed2a72c20 2024-09-16T01:43:57.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.916+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v122'436 uv436 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.917 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:43:57.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.916+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7db70 con 0x562ed2a72c20 2024-09-16T01:43:57.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.917+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v122'437 uv437 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.917+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7db70 con 0x562ed2a72c20 2024-09-16T01:43:57.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.918+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v122'438 uv438 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.918+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7db70 con 0x562ed2a72c20 2024-09-16T01:43:57.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.919+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v122'439 uv439 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.919+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x562ed2a7df30 con 0x562ed2a72c20 2024-09-16T01:43:57.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.920+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v122'440 uv440 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fa4280980d0 con 0x562ed2a72c20 2024-09-16T01:43:57.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.920+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x562ed29bec70 msgr2=0x562ed2935120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:57.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.920+0000 7fa430442f40 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x562ed29bec70 0x562ed2935120 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.920+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x562ed2a6f800 msgr2=0x562ed2a71bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:57.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.920+0000 7fa430442f40 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x562ed2a6f800 0x562ed2a71bf0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.920+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x562ed2a7ab30 msgr2=0x562ed2a7cf20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:57.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.920+0000 7fa430442f40 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x562ed2a7ab30 0x562ed2a7cf20 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.920+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x562ed2a6abb0 msgr2=0x562ed2a6cfa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:57.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.920+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 reap_dead start 2024-09-16T01:43:57.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.920+0000 7fa430442f40 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x562ed2a6abb0 0x562ed2a6cfa0 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.920+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x562ed2a72c20 msgr2=0x562ed2a75010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:57.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.920+0000 7fa430442f40 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x562ed2a72c20 0x562ed2a75010 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.920+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x562ed2a77e10 msgr2=0x562ed2a7a200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:57.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.920+0000 7fa430442f40 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x562ed2a77e10 0x562ed2a7a200 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.920+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa41006acc0 msgr2=0x7fa41006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:57.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.920+0000 7fa430442f40 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa41006acc0 0x7fa41006d180 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fa420009ba0 tx=0x7fa420005000 comp rx=0 tx=0).stop 2024-09-16T01:43:57.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.920+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562ed29c3940 msgr2=0x562ed29b66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:57.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.920+0000 7fa430442f40 1 --2- 172.21.15.88:0/2204417374 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562ed29c3940 0x562ed29b66c0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7fa428025e40 tx=0x7fa4280033a0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.920+0000 7fa42e739640 1 -- 172.21.15.88:0/2204417374 reap_dead start 2024-09-16T01:43:57.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.920+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 shutdown_connections 2024-09-16T01:43:57.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.920+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 >> 172.21.15.88:0/2204417374 conn(0x562ed2938750 msgr2=0x562ed29c4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:57.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.920+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 shutdown_connections 2024-09-16T01:43:57.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.920+0000 7fa430442f40 1 -- 172.21.15.88:0/2204417374 wait complete. 2024-09-16T01:43:57.924 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapheader 600.00000000 deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-09-16T01:43:57.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.966+0000 7fe5ab505f40 1 Processor -- start 2024-09-16T01:43:57.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.967+0000 7fe5ab505f40 1 -- start start 2024-09-16T01:43:57.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.967+0000 7fe5ab505f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ca9588fe10 0x55ca959350c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:57.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.967+0000 7fe5ab505f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca95890a60 0x55ca959ca760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:57.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.967+0000 7fe5ab505f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ca958aa250 0x55ca959caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:57.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.967+0000 7fe5ab505f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55ca95890220 con 0x55ca9588fe10 2024-09-16T01:43:57.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.967+0000 7fe5ab505f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ca95939120 con 0x55ca95890a60 2024-09-16T01:43:57.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.967+0000 7fe5ab505f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55ca959393c0 con 0x55ca958aa250 2024-09-16T01:43:57.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.967+0000 7fe5a8ffb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ca9588fe10 0x55ca959350c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:57.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.967+0000 7fe5a8ffb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ca9588fe10 0x55ca959350c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38326/0 (socket says 172.21.15.88:38326) 2024-09-16T01:43:57.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.967+0000 7fe5a8ffb640 1 -- 172.21.15.88:0/3837305110 learned_addr learned my addr 172.21.15.88:0/3837305110 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:57.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.967+0000 7fe5a07fa640 1 --2- 172.21.15.88:0/3837305110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca95890a60 0x55ca959ca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:57.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.967+0000 7fe5a97fc640 1 --2- 172.21.15.88:0/3837305110 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ca958aa250 0x55ca959caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:57.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.968+0000 7fe5a8ffb640 1 -- 172.21.15.88:0/3837305110 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ca958aa250 msgr2=0x55ca959caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:57.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.968+0000 7fe5a8ffb640 1 --2- 172.21.15.88:0/3837305110 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ca958aa250 0x55ca959caca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.968+0000 7fe5a8ffb640 1 -- 172.21.15.88:0/3837305110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca95890a60 msgr2=0x55ca959ca760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:57.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.968+0000 7fe5a8ffb640 1 --2- 172.21.15.88:0/3837305110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca95890a60 0x55ca959ca760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.968+0000 7fe5a8ffb640 1 -- 172.21.15.88:0/3837305110 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca9578bcb0 con 0x55ca9588fe10 2024-09-16T01:43:57.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.968+0000 7fe5a07fa640 1 --2- 172.21.15.88:0/3837305110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca95890a60 0x55ca959ca760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:57.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.968+0000 7fe5a8ffb640 1 --2- 172.21.15.88:0/3837305110 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ca9588fe10 0x55ca959350c0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7fe59800a770 tx=0x7fe59800a860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25f77c0c9e04eb5b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:57.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.968+0000 7fe5a37fe640 1 -- 172.21.15.88:0/3837305110 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe598019070 con 0x55ca9588fe10 2024-09-16T01:43:57.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.968+0000 7fe5a37fe640 1 -- 172.21.15.88:0/3837305110 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe598007600 con 0x55ca9588fe10 2024-09-16T01:43:57.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.968+0000 7fe5a37fe640 1 -- 172.21.15.88:0/3837305110 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5980079d0 con 0x55ca9588fe10 2024-09-16T01:43:57.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.968+0000 7fe5ab505f40 1 -- 172.21.15.88:0/3837305110 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ca9588fe10 msgr2=0x55ca959350c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:57.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.968+0000 7fe5ab505f40 1 --2- 172.21.15.88:0/3837305110 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ca9588fe10 0x55ca959350c0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7fe59800a770 tx=0x7fe59800a860 comp rx=0 tx=0).stop 2024-09-16T01:43:57.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.968+0000 7fe5ab505f40 1 -- 172.21.15.88:0/3837305110 shutdown_connections 2024-09-16T01:43:57.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.968+0000 7fe5ab505f40 1 --2- 172.21.15.88:0/3837305110 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ca958aa250 0x55ca959caca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.968+0000 7fe5ab505f40 1 --2- 172.21.15.88:0/3837305110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca95890a60 0x55ca959ca760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.968+0000 7fe5ab505f40 1 --2- 172.21.15.88:0/3837305110 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ca9588fe10 0x55ca959350c0 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.968+0000 7fe5ab505f40 1 -- 172.21.15.88:0/3837305110 >> 172.21.15.88:0/3837305110 conn(0x55ca958ab750 msgr2=0x55ca9588f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:57.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.968+0000 7fe5ab505f40 1 -- 172.21.15.88:0/3837305110 shutdown_connections 2024-09-16T01:43:57.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.968+0000 7fe5ab505f40 1 -- 172.21.15.88:0/3837305110 wait complete. 2024-09-16T01:43:57.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.969+0000 7fe5ab505f40 1 Processor -- start 2024-09-16T01:43:57.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.969+0000 7fe5ab505f40 1 -- start start 2024-09-16T01:43:57.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.970+0000 7fe5ab505f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ca9588fe10 0x55ca9592a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:57.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.970+0000 7fe5ab505f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ca95890a60 0x55ca9592cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:57.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.970+0000 7fe5ab505f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca958aa250 0x55ca959c3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:57.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.970+0000 7fe5ab505f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55ca9592b1b0 con 0x55ca9588fe10 2024-09-16T01:43:57.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.970+0000 7fe5ab505f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55ca9588d2b0 con 0x55ca958aa250 2024-09-16T01:43:57.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.970+0000 7fe5ab505f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55ca9592b490 con 0x55ca95890a60 2024-09-16T01:43:57.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.970+0000 7fe5a8ffb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ca9588fe10 0x55ca9592a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:57.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.970+0000 7fe5a8ffb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ca9588fe10 0x55ca9592a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38336/0 (socket says 172.21.15.88:38336) 2024-09-16T01:43:57.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.970+0000 7fe5a8ffb640 1 -- 172.21.15.88:0/3678553954 learned_addr learned my addr 172.21.15.88:0/3678553954 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:57.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.970+0000 7fe5a07fa640 1 --2- 172.21.15.88:0/3678553954 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ca95890a60 0x55ca9592cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:57.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.970+0000 7fe5a97fc640 1 --2- 172.21.15.88:0/3678553954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca958aa250 0x55ca959c3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:57.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.970+0000 7fe5a8ffb640 1 -- 172.21.15.88:0/3678553954 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ca95890a60 msgr2=0x55ca9592cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:57.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.970+0000 7fe5a8ffb640 1 --2- 172.21.15.88:0/3678553954 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55ca95890a60 0x55ca9592cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.970+0000 7fe5a8ffb640 1 -- 172.21.15.88:0/3678553954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca958aa250 msgr2=0x55ca959c3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:57.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.970+0000 7fe5a8ffb640 1 --2- 172.21.15.88:0/3678553954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ca958aa250 0x55ca959c3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.970+0000 7fe5a8ffb640 1 -- 172.21.15.88:0/3678553954 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca959c0a40 con 0x55ca9588fe10 2024-09-16T01:43:57.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.970+0000 7fe5a8ffb640 1 --2- 172.21.15.88:0/3678553954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ca9588fe10 0x55ca9592a350 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fe59800ad60 tx=0x7fe598003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:57.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.970+0000 7fe5a27fc640 1 -- 172.21.15.88:0/3678553954 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe598019040 con 0x55ca9588fe10 2024-09-16T01:43:57.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.970+0000 7fe5a27fc640 1 -- 172.21.15.88:0/3678553954 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe598022070 con 0x55ca9588fe10 2024-09-16T01:43:57.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.970+0000 7fe5ab505f40 1 -- 172.21.15.88:0/3678553954 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ca959c0d20 con 0x55ca9588fe10 2024-09-16T01:43:57.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.970+0000 7fe5a27fc640 1 -- 172.21.15.88:0/3678553954 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe598014470 con 0x55ca9588fe10 2024-09-16T01:43:57.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.970+0000 7fe5ab505f40 1 -- 172.21.15.88:0/3678553954 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca959c12e0 con 0x55ca9588fe10 2024-09-16T01:43:57.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.972+0000 7fe5a27fc640 1 -- 172.21.15.88:0/3678553954 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe598014610 con 0x55ca9588fe10 2024-09-16T01:43:57.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.972+0000 7fe5a27fc640 1 --2- 172.21.15.88:0/3678553954 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe58806ac90 0x7fe58806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:57.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.972+0000 7fe5a27fc640 1 -- 172.21.15.88:0/3678553954 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 8748+0+0 (secure 0 0 0) 0x7fe59808b6e0 con 0x55ca9588fe10 2024-09-16T01:43:57.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.972+0000 7fe5a07fa640 1 --2- 172.21.15.88:0/3678553954 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe58806ac90 0x7fe58806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:57.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.972+0000 7fe5ab505f40 1 --2- 172.21.15.88:0/3678553954 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55ca95937c40 0x55ca959c1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:57.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.972+0000 7fe5ab505f40 1 -- 172.21.15.88:0/3678553954 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122) -- 0x55ca959c2300 con 0x55ca95937c40 2024-09-16T01:43:57.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.972+0000 7fe5a97fc640 1 --2- 172.21.15.88:0/3678553954 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55ca95937c40 0x55ca959c1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:57.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.973+0000 7fe5a07fa640 1 --2- 172.21.15.88:0/3678553954 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe58806ac90 0x7fe58806d150 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fe590057000 tx=0x7fe590052000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:57.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.973+0000 7fe5a97fc640 1 --2- 172.21.15.88:0/3678553954 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55ca95937c40 0x55ca959c1dc0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:57.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.974+0000 7fe5a97fc640 1 -- 172.21.15.88:0/3678553954 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 600.00000000 [omap-set-header] v122'1096 uv1096 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe5a4002940 con 0x55ca95937c40 2024-09-16T01:43:57.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.974+0000 7fe5ab505f40 1 -- 172.21.15.88:0/3678553954 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55ca95937c40 msgr2=0x55ca959c1dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:57.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.974+0000 7fe5ab505f40 1 --2- 172.21.15.88:0/3678553954 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55ca95937c40 0x55ca959c1dc0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:57.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.974+0000 7fe5ab505f40 1 -- 172.21.15.88:0/3678553954 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe58806ac90 msgr2=0x7fe58806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:57.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.974+0000 7fe5ab505f40 1 --2- 172.21.15.88:0/3678553954 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe58806ac90 0x7fe58806d150 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fe590057000 tx=0x7fe590052000 comp rx=0 tx=0).stop 2024-09-16T01:43:57.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.974+0000 7fe5ab505f40 1 -- 172.21.15.88:0/3678553954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ca9588fe10 msgr2=0x55ca9592a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:57.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.974+0000 7fe5ab505f40 1 --2- 172.21.15.88:0/3678553954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55ca9588fe10 0x55ca9592a350 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fe59800ad60 tx=0x7fe598003020 comp rx=0 tx=0).stop 2024-09-16T01:43:57.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.974+0000 7fe5a97fc640 1 -- 172.21.15.88:0/3678553954 reap_dead start 2024-09-16T01:43:57.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.974+0000 7fe5ab505f40 1 -- 172.21.15.88:0/3678553954 shutdown_connections 2024-09-16T01:43:57.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.974+0000 7fe5ab505f40 1 -- 172.21.15.88:0/3678553954 >> 172.21.15.88:0/3678553954 conn(0x55ca958ab750 msgr2=0x55ca9592d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:57.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.975+0000 7fe5ab505f40 1 -- 172.21.15.88:0/3678553954 shutdown_connections 2024-09-16T01:43:57.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:57.975+0000 7fe5ab505f40 1 -- 172.21.15.88:0/3678553954 wait complete. 2024-09-16T01:43:57.980 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:43:58.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.081+0000 7f91129f1640 1 Processor -- start 2024-09-16T01:43:58.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.081+0000 7f91129f1640 1 -- start start 2024-09-16T01:43:58.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.081+0000 7f91129f1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f910c0ce9e0 0x7f910c0b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:58.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.081+0000 7f91129f1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f910c074090 con 0x7f910c0ce3b0 2024-09-16T01:43:58.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.081+0000 7f91129f1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f910c0ccd10 con 0x7f910c15e7c0 2024-09-16T01:43:58.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.081+0000 7f91129f1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f910c0ccff0 con 0x7f910c0ce9e0 2024-09-16T01:43:58.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.082+0000 7f9110f8e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f910c0ce3b0 0x7f910c0cd7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49344/0 (socket says 172.21.15.88:49344) 2024-09-16T01:43:58.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.082+0000 7f9110f8e640 1 -- 172.21.15.88:0/1124693566 learned_addr learned my addr 172.21.15.88:0/1124693566 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:58.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.082+0000 7f910b7fe640 1 --2- 172.21.15.88:0/1124693566 >> v2:172.21.15.154:3300/0 conn(0x7f910c0ce9e0 0x7f910c0b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:58.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.082+0000 7f910affd640 1 -- 172.21.15.88:0/1124693566 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2682866600 0 0) 0x7f910c074090 con 0x7f910c0ce3b0 2024-09-16T01:43:58.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.082+0000 7f910affd640 1 -- 172.21.15.88:0/1124693566 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90f4003520 con 0x7f910c0ce3b0 2024-09-16T01:43:58.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.082+0000 7f910affd640 1 -- 172.21.15.88:0/1124693566 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2598434537 0 0) 0x7f90f4003520 con 0x7f910c0ce3b0 2024-09-16T01:43:58.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.082+0000 7f910affd640 1 -- 172.21.15.88:0/1124693566 >> v2:172.21.15.154:3300/0 conn(0x7f910c0ce9e0 msgr2=0x7f910c0b44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:58.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.082+0000 7f910affd640 1 --2- 172.21.15.88:0/1124693566 >> v2:172.21.15.154:3300/0 conn(0x7f910c0ce9e0 0x7f910c0b44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:58.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.082+0000 7f910affd640 1 -- 172.21.15.88:0/1124693566 >> v1:172.21.15.154:6789/0 conn(0x7f910c15e7c0 legacy=0x7f910c0d1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:58.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.082+0000 7f910affd640 1 -- 172.21.15.88:0/1124693566 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f910c0cd2d0 con 0x7f910c0ce3b0 2024-09-16T01:43:58.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.082+0000 7f910affd640 1 -- 172.21.15.88:0/1124693566 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f91040019f0 con 0x7f910c0ce3b0 2024-09-16T01:43:58.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.082+0000 7f910affd640 1 -- 172.21.15.88:0/1124693566 >> v1:172.21.15.88:6789/0 conn(0x7f910c0ce3b0 legacy=0x7f910c0cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:58.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.082+0000 7f910affd640 1 --2- 172.21.15.88:0/1124693566 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90f40039b0 0x7f90f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:58.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.082+0000 7f910affd640 1 --2- 172.21.15.88:0/1124693566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90f4004800 0x7f90f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.082+0000 7f910affd640 1 --2- 172.21.15.88:0/1124693566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90f40051c0 0x7f90f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.082+0000 7f910affd640 1 -- 172.21.15.88:0/1124693566 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f910c0ccff0 con 0x7f90f40051c0 2024-09-16T01:43:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.082+0000 7f910affd640 1 -- 172.21.15.88:0/1124693566 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f90f400c060 con 0x7f90f4004800 2024-09-16T01:43:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.082+0000 7f910affd640 1 -- 172.21.15.88:0/1124693566 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f90f400c260 con 0x7f90f40039b0 2024-09-16T01:43:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.082+0000 7f9110f8e640 1 --2- 172.21.15.88:0/1124693566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90f40051c0 0x7f90f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.082+0000 7f910bfff640 1 --2- 172.21.15.88:0/1124693566 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90f40039b0 0x7f90f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.082+0000 7f910b7fe640 1 --2- 172.21.15.88:0/1124693566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90f4004800 0x7f90f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.083+0000 7f9110f8e640 1 -- 172.21.15.88:0/1124693566 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90f40039b0 msgr2=0x7f90f4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.083+0000 7f9110f8e640 1 --2- 172.21.15.88:0/1124693566 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90f40039b0 0x7f90f4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.083+0000 7f9110f8e640 1 -- 172.21.15.88:0/1124693566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90f4004800 msgr2=0x7f90f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.083+0000 7f9110f8e640 1 --2- 172.21.15.88:0/1124693566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90f4004800 0x7f90f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.083+0000 7f9110f8e640 1 -- 172.21.15.88:0/1124693566 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90f4003520 con 0x7f90f40051c0 2024-09-16T01:43:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.083+0000 7f9110f8e640 1 -- 172.21.15.88:0/1124693566 reap_dead start 2024-09-16T01:43:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.083+0000 7f9110f8e640 1 --2- 172.21.15.88:0/1124693566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90f40051c0 0x7f90f400b9c0 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f910400a840 tx=0x7f90f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b6cfab8f1ceeb53 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.083+0000 7f910affd640 1 -- 172.21.15.88:0/1124693566 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f910400a460 con 0x7f90f40051c0 2024-09-16T01:43:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.083+0000 7f910affd640 1 -- 172.21.15.88:0/1124693566 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f910402f070 con 0x7f90f40051c0 2024-09-16T01:43:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.083+0000 7f910affd640 1 -- 172.21.15.88:0/1124693566 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f910402bcb0 con 0x7f90f40051c0 2024-09-16T01:43:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.084+0000 7f91129f1640 1 -- 172.21.15.88:0/1124693566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90f40051c0 msgr2=0x7f90f400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.084+0000 7f91129f1640 1 --2- 172.21.15.88:0/1124693566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90f40051c0 0x7f90f400b9c0 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f910400a840 tx=0x7f90f4005830 comp rx=0 tx=0).stop 2024-09-16T01:43:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.084+0000 7f91129f1640 1 -- 172.21.15.88:0/1124693566 shutdown_connections 2024-09-16T01:43:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.084+0000 7f91129f1640 1 --2- 172.21.15.88:0/1124693566 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90f40051c0 0x7f90f400b9c0 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.084+0000 7f91129f1640 1 -- 172.21.15.88:0/1124693566 >> 172.21.15.88:0/1124693566 conn(0x7f910c062a20 msgr2=0x7f910c062e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.084+0000 7f91129f1640 1 -- 172.21.15.88:0/1124693566 shutdown_connections 2024-09-16T01:43:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.084+0000 7f91129f1640 1 -- 172.21.15.88:0/1124693566 wait complete. 2024-09-16T01:43:58.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.085+0000 7f91129f1640 1 Processor -- start 2024-09-16T01:43:58.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.085+0000 7f91129f1640 1 -- start start 2024-09-16T01:43:58.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.086+0000 7f91129f1640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90f40051c0 0x7f910c0c9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:58.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.086+0000 7f91129f1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f910c1f7e20 0x7f910c15ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:58.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.086+0000 7f91129f1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f910c0aff80 0x7f910c0b0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:58.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.086+0000 7f91129f1640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f910c073da0 con 0x7f910c1f7e20 2024-09-16T01:43:58.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.086+0000 7f91129f1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f910c073c20 con 0x7f910c0aff80 2024-09-16T01:43:58.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.086+0000 7f91129f1640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f910c0d1330 con 0x7f90f40051c0 2024-09-16T01:43:58.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.086+0000 7f910b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f910c1f7e20 0x7f910c15ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:58.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.086+0000 7f910b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f910c1f7e20 0x7f910c15ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38356/0 (socket says 172.21.15.88:38356) 2024-09-16T01:43:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.086+0000 7f910b7fe640 1 -- 172.21.15.88:0/1615031510 learned_addr learned my addr 172.21.15.88:0/1615031510 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.086+0000 7f9110f8e640 1 --2- 172.21.15.88:0/1615031510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f910c0aff80 0x7f910c0b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.086+0000 7f910bfff640 1 --2- 172.21.15.88:0/1615031510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90f40051c0 0x7f910c0c9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.086+0000 7f910b7fe640 1 -- 172.21.15.88:0/1615031510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90f40051c0 msgr2=0x7f910c0c9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.086+0000 7f910b7fe640 1 --2- 172.21.15.88:0/1615031510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90f40051c0 0x7f910c0c9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.086+0000 7f910b7fe640 1 -- 172.21.15.88:0/1615031510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f910c0aff80 msgr2=0x7f910c0b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.086+0000 7f910b7fe640 1 --2- 172.21.15.88:0/1615031510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f910c0aff80 0x7f910c0b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.086+0000 7f910b7fe640 1 -- 172.21.15.88:0/1615031510 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f910c0b8150 con 0x7f910c1f7e20 2024-09-16T01:43:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.086+0000 7f9110f8e640 1 --2- 172.21.15.88:0/1615031510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f910c0aff80 0x7f910c0b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.086+0000 7f910bfff640 1 --2- 172.21.15.88:0/1615031510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90f40051c0 0x7f910c0c9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.087+0000 7f910b7fe640 1 --2- 172.21.15.88:0/1615031510 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f910c1f7e20 0x7f910c15ba50 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f9100011930 tx=0x7f9100011df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.087+0000 7f9108ff9640 1 -- 172.21.15.88:0/1615031510 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f910000fa30 con 0x7f910c1f7e20 2024-09-16T01:43:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.087+0000 7f9108ff9640 1 -- 172.21.15.88:0/1615031510 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f910000fbd0 con 0x7f910c1f7e20 2024-09-16T01:43:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.087+0000 7f91129f1640 1 -- 172.21.15.88:0/1615031510 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f910c0b5520 con 0x7f910c1f7e20 2024-09-16T01:43:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.087+0000 7f9108ff9640 1 -- 172.21.15.88:0/1615031510 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9100014440 con 0x7f910c1f7e20 2024-09-16T01:43:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.087+0000 7f91129f1640 1 -- 172.21.15.88:0/1615031510 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f910c15bf90 con 0x7f910c1f7e20 2024-09-16T01:43:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.088+0000 7f91129f1640 1 -- 172.21.15.88:0/1615031510 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f910c0d16c0 con 0x7f910c1f7e20 2024-09-16T01:43:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.088+0000 7f9108ff9640 1 -- 172.21.15.88:0/1615031510 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9100012040 con 0x7f910c1f7e20 2024-09-16T01:43:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.089+0000 7f9108ff9640 1 --2- 172.21.15.88:0/1615031510 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f90f806c2e0 0x7f90f806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.089+0000 7f9108ff9640 1 -- 172.21.15.88:0/1615031510 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 8748+0+0 (secure 0 0 0) 0x7f910008c640 con 0x7f910c1f7e20 2024-09-16T01:43:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.089+0000 7f910bfff640 1 --2- 172.21.15.88:0/1615031510 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f90f806c2e0 0x7f90f806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.089+0000 7f910bfff640 1 --2- 172.21.15.88:0/1615031510 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f90f806c2e0 0x7f90f806e7a0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f90fc000c00 tx=0x7f90fc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:58.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.093+0000 7f9108ff9640 1 -- 172.21.15.88:0/1615031510 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9100019030 con 0x7f910c1f7e20 2024-09-16T01:43:58.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.292+0000 7f91129f1640 1 -- 172.21.15.88:0/1615031510 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f910c161ca0 con 0x7f910c1f7e20 2024-09-16T01:43:58.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.921+0000 7f9108ff9640 1 -- 172.21.15.88:0/1615031510 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v285) ==== 162+0+0 (secure 0 0 0) 0x7f9100059bc0 con 0x7f910c1f7e20 2024-09-16T01:43:58.928 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:43:58.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.923+0000 7f91129f1640 1 -- 172.21.15.88:0/1615031510 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f90f806c2e0 msgr2=0x7f90f806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:58.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.923+0000 7f91129f1640 1 --2- 172.21.15.88:0/1615031510 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f90f806c2e0 0x7f90f806e7a0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f90fc000c00 tx=0x7f90fc009040 comp rx=0 tx=0).stop 2024-09-16T01:43:58.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.923+0000 7f91129f1640 1 -- 172.21.15.88:0/1615031510 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f910c1f7e20 msgr2=0x7f910c15ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:58.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.923+0000 7f91129f1640 1 --2- 172.21.15.88:0/1615031510 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f910c1f7e20 0x7f910c15ba50 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f9100011930 tx=0x7f9100011df0 comp rx=0 tx=0).stop 2024-09-16T01:43:58.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.923+0000 7f91129f1640 1 -- 172.21.15.88:0/1615031510 shutdown_connections 2024-09-16T01:43:58.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.923+0000 7f91129f1640 1 --2- 172.21.15.88:0/1615031510 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f90f806c2e0 0x7f90f806e7a0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:58.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.923+0000 7f91129f1640 1 --2- 172.21.15.88:0/1615031510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f910c0aff80 0x7f910c0b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:58.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.923+0000 7f91129f1640 1 --2- 172.21.15.88:0/1615031510 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f910c1f7e20 0x7f910c15ba50 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:58.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.923+0000 7f91129f1640 1 --2- 172.21.15.88:0/1615031510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f90f40051c0 0x7f910c0c9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:58.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.923+0000 7f91129f1640 1 -- 172.21.15.88:0/1615031510 >> 172.21.15.88:0/1615031510 conn(0x7f910c062a20 msgr2=0x7f910c0b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:58.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.923+0000 7f91129f1640 1 -- 172.21.15.88:0/1615031510 shutdown_connections 2024-09-16T01:43:58.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:58.923+0000 7f91129f1640 1 -- 172.21.15.88:0/1615031510 wait complete. 2024-09-16T01:43:58.939 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:43:59.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.038+0000 7fae159f8640 1 Processor -- start 2024-09-16T01:43:59.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.038+0000 7fae159f8640 1 -- start start 2024-09-16T01:43:59.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.039+0000 7fae159f8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fae1015e660 0x7fae100cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:59.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.039+0000 7fae159f8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae1015ea40 con 0x7fae100d1820 2024-09-16T01:43:59.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.039+0000 7fae159f8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae100d3870 con 0x7fae100d3310 2024-09-16T01:43:59.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.039+0000 7fae159f8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fae100d3b50 con 0x7fae1015e660 2024-09-16T01:43:59.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.039+0000 7fae0f7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fae100d1820 0x7fae100b7b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:49358/0 (socket says 172.21.15.88:49358) 2024-09-16T01:43:59.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.039+0000 7fae0f7fe640 1 -- 172.21.15.88:0/2399984630 learned_addr learned my addr 172.21.15.88:0/2399984630 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:43:59.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.039+0000 7fae0effd640 1 --2- 172.21.15.88:0/2399984630 >> v2:172.21.15.154:3300/0 conn(0x7fae1015e660 0x7fae100cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:59.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.039+0000 7fae0dffb640 1 -- 172.21.15.88:0/2399984630 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2087041446 0 0) 0x7fae1015ea40 con 0x7fae100d1820 2024-09-16T01:43:59.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.040+0000 7fae0dffb640 1 -- 172.21.15.88:0/2399984630 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fadf8003520 con 0x7fae100d1820 2024-09-16T01:43:59.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.040+0000 7fae0dffb640 1 -- 172.21.15.88:0/2399984630 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1037148182 0 0) 0x7fadf8003520 con 0x7fae100d1820 2024-09-16T01:43:59.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.040+0000 7fae0dffb640 1 -- 172.21.15.88:0/2399984630 >> v2:172.21.15.154:3300/0 conn(0x7fae1015e660 msgr2=0x7fae100cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:59.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.040+0000 7fae0dffb640 1 --2- 172.21.15.88:0/2399984630 >> v2:172.21.15.154:3300/0 conn(0x7fae1015e660 0x7fae100cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:59.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.040+0000 7fae0dffb640 1 -- 172.21.15.88:0/2399984630 >> v1:172.21.15.154:6789/0 conn(0x7fae100d3310 legacy=0x7fae100d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:59.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.040+0000 7fae0dffb640 1 -- 172.21.15.88:0/2399984630 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae100d3e30 con 0x7fae100d1820 2024-09-16T01:43:59.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.040+0000 7fae0dffb640 1 -- 172.21.15.88:0/2399984630 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fae08001670 con 0x7fae100d1820 2024-09-16T01:43:59.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.040+0000 7fae0effd640 1 --2- 172.21.15.88:0/2399984630 >> v2:172.21.15.154:3300/0 conn(0x7fae1015e660 0x7fae100cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:43:59.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.040+0000 7fae0dffb640 1 -- 172.21.15.88:0/2399984630 >> v1:172.21.15.88:6789/0 conn(0x7fae100d1820 legacy=0x7fae100b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:59.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.040+0000 7fae0dffb640 1 --2- 172.21.15.88:0/2399984630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadf80039b0 0x7fadf8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:59.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.040+0000 7fae0dffb640 1 --2- 172.21.15.88:0/2399984630 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadf8004800 0x7fadf8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:59.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.040+0000 7fae0dffb640 1 --2- 172.21.15.88:0/2399984630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadf80051c0 0x7fadf800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:59.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.040+0000 7fae0dffb640 1 -- 172.21.15.88:0/2399984630 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fae100d3b50 con 0x7fadf80039b0 2024-09-16T01:43:59.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.040+0000 7fae0dffb640 1 -- 172.21.15.88:0/2399984630 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fadf800c060 con 0x7fadf80051c0 2024-09-16T01:43:59.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.040+0000 7fae0dffb640 1 -- 172.21.15.88:0/2399984630 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fadf800c260 con 0x7fadf8004800 2024-09-16T01:43:59.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.040+0000 7fae0effd640 1 --2- 172.21.15.88:0/2399984630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadf80039b0 0x7fadf8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:59.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.040+0000 7fae0f7fe640 1 --2- 172.21.15.88:0/2399984630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadf80051c0 0x7fadf800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:59.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.040+0000 7fae0e7fc640 1 --2- 172.21.15.88:0/2399984630 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadf8004800 0x7fadf8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:59.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.040+0000 7fae0effd640 1 -- 172.21.15.88:0/2399984630 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadf8004800 msgr2=0x7fadf8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:59.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.040+0000 7fae0effd640 1 --2- 172.21.15.88:0/2399984630 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadf8004800 0x7fadf8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:59.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.040+0000 7fae0effd640 1 -- 172.21.15.88:0/2399984630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadf80051c0 msgr2=0x7fadf800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:43:59.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.040+0000 7fae0effd640 1 --2- 172.21.15.88:0/2399984630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadf80051c0 0x7fadf800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:59.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.040+0000 7fae0effd640 1 -- 172.21.15.88:0/2399984630 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadf8003520 con 0x7fadf80039b0 2024-09-16T01:43:59.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.040+0000 7fae0f7fe640 1 --2- 172.21.15.88:0/2399984630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadf80051c0 0x7fadf800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:59.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.040+0000 7fae0f7fe640 1 -- 172.21.15.88:0/2399984630 reap_dead start 2024-09-16T01:43:59.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.041+0000 7fae0effd640 1 --2- 172.21.15.88:0/2399984630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadf80039b0 0x7fadf8003dd0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7fae0000e350 tx=0x7fadf8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=538eca64b8d98c81 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:59.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.041+0000 7fae0dffb640 1 -- 172.21.15.88:0/2399984630 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae00033070 con 0x7fadf80039b0 2024-09-16T01:43:59.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.041+0000 7fae0dffb640 1 -- 172.21.15.88:0/2399984630 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae0002f660 con 0x7fadf80039b0 2024-09-16T01:43:59.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.041+0000 7fae0dffb640 1 -- 172.21.15.88:0/2399984630 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae00009de0 con 0x7fadf80039b0 2024-09-16T01:43:59.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.042+0000 7fae159f8640 1 -- 172.21.15.88:0/2399984630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadf80039b0 msgr2=0x7fadf8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:59.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.042+0000 7fae159f8640 1 --2- 172.21.15.88:0/2399984630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadf80039b0 0x7fadf8003dd0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7fae0000e350 tx=0x7fadf8005830 comp rx=0 tx=0).stop 2024-09-16T01:43:59.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.042+0000 7fae159f8640 1 -- 172.21.15.88:0/2399984630 shutdown_connections 2024-09-16T01:43:59.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.042+0000 7fae159f8640 1 --2- 172.21.15.88:0/2399984630 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadf80039b0 0x7fadf8003dd0 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:59.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.042+0000 7fae159f8640 1 -- 172.21.15.88:0/2399984630 >> 172.21.15.88:0/2399984630 conn(0x7fae100b6410 msgr2=0x7fae100b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:59.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.042+0000 7fae159f8640 1 -- 172.21.15.88:0/2399984630 shutdown_connections 2024-09-16T01:43:59.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.042+0000 7fae159f8640 1 -- 172.21.15.88:0/2399984630 wait complete. 2024-09-16T01:43:59.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.043+0000 7fae159f8640 1 Processor -- start 2024-09-16T01:43:59.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.043+0000 7fae159f8640 1 -- start start 2024-09-16T01:43:59.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.044+0000 7fae159f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadf80039b0 0x7fae100c8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:59.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.044+0000 7fae159f8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae101618c0 0x7fae10160050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:59.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.044+0000 7fae159f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae101519f0 0x7fae100b5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:59.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.044+0000 7fae159f8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fae100d1110 con 0x7fae101618c0 2024-09-16T01:43:59.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.044+0000 7fae159f8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fae100d43f0 con 0x7fae101519f0 2024-09-16T01:43:59.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.044+0000 7fae159f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fae100d1290 con 0x7fadf80039b0 2024-09-16T01:43:59.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.044+0000 7fae0e7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae101618c0 0x7fae10160050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:59.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.044+0000 7fae0e7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae101618c0 0x7fae10160050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38378/0 (socket says 172.21.15.88:38378) 2024-09-16T01:43:59.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.044+0000 7fae0e7fc640 1 -- 172.21.15.88:0/746168194 learned_addr learned my addr 172.21.15.88:0/746168194 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:43:59.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.044+0000 7fae0effd640 1 --2- 172.21.15.88:0/746168194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadf80039b0 0x7fae100c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:59.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.044+0000 7fae0f7fe640 1 --2- 172.21.15.88:0/746168194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae101519f0 0x7fae100b5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:59.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.044+0000 7fae0e7fc640 1 -- 172.21.15.88:0/746168194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadf80039b0 msgr2=0x7fae100c8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:59.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.044+0000 7fae0e7fc640 1 --2- 172.21.15.88:0/746168194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadf80039b0 0x7fae100c8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:59.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.044+0000 7fae0e7fc640 1 -- 172.21.15.88:0/746168194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae101519f0 msgr2=0x7fae100b5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:59.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.044+0000 7fae0e7fc640 1 --2- 172.21.15.88:0/746168194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae101519f0 0x7fae100b5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:59.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.044+0000 7fae0e7fc640 1 -- 172.21.15.88:0/746168194 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae100c85c0 con 0x7fae101618c0 2024-09-16T01:43:59.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.044+0000 7fae0f7fe640 1 --2- 172.21.15.88:0/746168194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae101519f0 0x7fae100b5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:43:59.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.044+0000 7fae0e7fc640 1 --2- 172.21.15.88:0/746168194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae101618c0 0x7fae10160050 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7fae0400e4d0 tx=0x7fae0400e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:59.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.044+0000 7fadf7fff640 1 -- 172.21.15.88:0/746168194 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae040053a0 con 0x7fae101618c0 2024-09-16T01:43:59.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.044+0000 7fadf7fff640 1 -- 172.21.15.88:0/746168194 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae04018070 con 0x7fae101618c0 2024-09-16T01:43:59.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.044+0000 7fae159f8640 1 -- 172.21.15.88:0/746168194 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae1015ee20 con 0x7fae101618c0 2024-09-16T01:43:59.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.044+0000 7fadf7fff640 1 -- 172.21.15.88:0/746168194 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae04003810 con 0x7fae101618c0 2024-09-16T01:43:59.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.045+0000 7fae159f8640 1 -- 172.21.15.88:0/746168194 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae1015f850 con 0x7fae101618c0 2024-09-16T01:43:59.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.045+0000 7fae159f8640 1 -- 172.21.15.88:0/746168194 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae100cf730 con 0x7fae101618c0 2024-09-16T01:43:59.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.046+0000 7fadf7fff640 1 -- 172.21.15.88:0/746168194 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fae040039b0 con 0x7fae101618c0 2024-09-16T01:43:59.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.046+0000 7fadf7fff640 1 --2- 172.21.15.88:0/746168194 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fadfc06c2b0 0x7fadfc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:43:59.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.047+0000 7fadf7fff640 1 -- 172.21.15.88:0/746168194 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 8748+0+0 (secure 0 0 0) 0x7fae0408bc40 con 0x7fae101618c0 2024-09-16T01:43:59.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.047+0000 7fae0effd640 1 --2- 172.21.15.88:0/746168194 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fadfc06c2b0 0x7fadfc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:43:59.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.047+0000 7fae0effd640 1 --2- 172.21.15.88:0/746168194 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fadfc06c2b0 0x7fadfc06e770 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fae00035000 tx=0x7fae0002f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:43:59.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.051+0000 7fadf7fff640 1 -- 172.21.15.88:0/746168194 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fae04059810 con 0x7fae101618c0 2024-09-16T01:43:59.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.249+0000 7fae159f8640 1 -- 172.21.15.88:0/746168194 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae100c7e30 con 0x7fae101618c0 2024-09-16T01:43:59.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.249+0000 7fadf7fff640 1 -- 172.21.15.88:0/746168194 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 286 v286) ==== 95+0+5230 (secure 0 0 0) 0x7fae04059110 con 0x7fae101618c0 2024-09-16T01:43:59.252 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:43:59.252 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":286,"btime":"2024-09-16T01:43:58:923791+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9696,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4270197461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4270197461},{"type":"v1","addr":"172.21.15.88:6837","nonce":4270197461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":286,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:43:58.923779+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10053},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10053":{"gid":10053,"name":"a","rank":0,"incarnation":286,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6836/3078464816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3078464816},{"type":"v1","addr":"172.21.15.88:6836","nonce":3078464816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:43:59.252 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 286 2024-09-16T01:43:59.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.251+0000 7fae159f8640 1 -- 172.21.15.88:0/746168194 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fadfc06c2b0 msgr2=0x7fadfc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:59.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.251+0000 7fae159f8640 1 --2- 172.21.15.88:0/746168194 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fadfc06c2b0 0x7fadfc06e770 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fae00035000 tx=0x7fae0002f000 comp rx=0 tx=0).stop 2024-09-16T01:43:59.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.251+0000 7fae159f8640 1 -- 172.21.15.88:0/746168194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae101618c0 msgr2=0x7fae10160050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:43:59.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.251+0000 7fae159f8640 1 --2- 172.21.15.88:0/746168194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae101618c0 0x7fae10160050 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7fae0400e4d0 tx=0x7fae0400e990 comp rx=0 tx=0).stop 2024-09-16T01:43:59.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.251+0000 7fae159f8640 1 -- 172.21.15.88:0/746168194 shutdown_connections 2024-09-16T01:43:59.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.251+0000 7fae159f8640 1 --2- 172.21.15.88:0/746168194 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fadfc06c2b0 0x7fadfc06e770 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:59.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.251+0000 7fae159f8640 1 --2- 172.21.15.88:0/746168194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae101519f0 0x7fae100b5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:59.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.251+0000 7fae159f8640 1 --2- 172.21.15.88:0/746168194 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fae101618c0 0x7fae10160050 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:59.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.251+0000 7fae159f8640 1 --2- 172.21.15.88:0/746168194 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fadf80039b0 0x7fae100c8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:43:59.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.251+0000 7fae159f8640 1 -- 172.21.15.88:0/746168194 >> 172.21.15.88:0/746168194 conn(0x7fae100b6410 msgr2=0x7fae1006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:43:59.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.251+0000 7fae159f8640 1 -- 172.21.15.88:0/746168194 shutdown_connections 2024-09-16T01:43:59.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:43:59.252+0000 7fae159f8640 1 -- 172.21.15.88:0/746168194 wait complete. 2024-09-16T01:43:59.264 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-09-16T01:44:01.949 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:44:01.947+0000 7f48d62d0640 -1 mds.0.cache.dir(0x600) Corrupt fnode in dirfrag 0x600: Decoder at 'void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&)' v=3 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-09-16T01:44:02.057 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:44:01.947+0000 7f48d62d0640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x600 is fatal because it is a system directory for this rank 2024-09-16T01:44:02.058 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:44:01.948+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:44:02.059 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:44:01.948+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:44:02.655 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:44:02.654+0000 7f48da2d8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:44:04.265 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:44:04.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.375+0000 7fd1d54af640 1 Processor -- start 2024-09-16T01:44:04.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.375+0000 7fd1d54af640 1 -- start start 2024-09-16T01:44:04.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.376+0000 7fd1d54af640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd1d015e660 0x7fd1d00cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:04.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.376+0000 7fd1d54af640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd1d00d3310 0x7fd1d00d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:04.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.376+0000 7fd1d54af640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1d015ea40 con 0x7fd1d00d1650 2024-09-16T01:44:04.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.376+0000 7fd1d54af640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd1d00d1ac0 con 0x7fd1d015e660 2024-09-16T01:44:04.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.376+0000 7fd1d54af640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd1d00d3870 con 0x7fd1d00d3310 2024-09-16T01:44:04.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.376+0000 7fd1ceffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd1d015e660 0x7fd1d00cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:04.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.376+0000 7fd1cf7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd1d00d1650 0x7fd1d006bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44610/0 (socket says 172.21.15.88:44610) 2024-09-16T01:44:04.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.376+0000 7fd1cf7fe640 1 -- 172.21.15.88:0/473134844 learned_addr learned my addr 172.21.15.88:0/473134844 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:04.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.376+0000 7fd1c67fc640 1 --2- 172.21.15.88:0/473134844 >> v2:172.21.15.154:3300/0 conn(0x7fd1d00d3310 0x7fd1d00d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:04.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.376+0000 7fd1ceffd640 1 -- 172.21.15.88:0/473134844 >> v2:172.21.15.154:3300/0 conn(0x7fd1d00d3310 msgr2=0x7fd1d00d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:04.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.376+0000 7fd1ceffd640 1 --2- 172.21.15.88:0/473134844 >> v2:172.21.15.154:3300/0 conn(0x7fd1d00d3310 0x7fd1d00d1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:04.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.376+0000 7fd1ceffd640 1 -- 172.21.15.88:0/473134844 >> v1:172.21.15.88:6789/0 conn(0x7fd1d00d1650 legacy=0x7fd1d006bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:04.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.376+0000 7fd1ce7fc640 1 -- 172.21.15.88:0/473134844 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 736203466 0 0) 0x7fd1d015ea40 con 0x7fd1d00d1650 2024-09-16T01:44:04.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.376+0000 7fd1ceffd640 1 -- 172.21.15.88:0/473134844 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d00d3b50 con 0x7fd1d015e660 2024-09-16T01:44:04.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.377+0000 7fd1ceffd640 1 --2- 172.21.15.88:0/473134844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1d015e660 0x7fd1d00cf730 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7fd1d00c8020 tx=0x7fd1c002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=184071227c85b600 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:04.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.377+0000 7fd1ce7fc640 1 -- 172.21.15.88:0/473134844 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0037070 con 0x7fd1d015e660 2024-09-16T01:44:04.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.377+0000 7fd1ce7fc640 1 -- 172.21.15.88:0/473134844 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1c0032440 con 0x7fd1d015e660 2024-09-16T01:44:04.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.377+0000 7fd1ce7fc640 1 -- 172.21.15.88:0/473134844 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0032810 con 0x7fd1d015e660 2024-09-16T01:44:04.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.378+0000 7fd1d54af640 1 -- 172.21.15.88:0/473134844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1d015e660 msgr2=0x7fd1d00cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:04.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.378+0000 7fd1d54af640 1 --2- 172.21.15.88:0/473134844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1d015e660 0x7fd1d00cf730 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7fd1d00c8020 tx=0x7fd1c002a1a0 comp rx=0 tx=0).stop 2024-09-16T01:44:04.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.378+0000 7fd1d54af640 1 -- 172.21.15.88:0/473134844 shutdown_connections 2024-09-16T01:44:04.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.378+0000 7fd1d54af640 1 --2- 172.21.15.88:0/473134844 >> v2:172.21.15.154:3300/0 conn(0x7fd1d00d3310 0x7fd1d00d1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:04.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.378+0000 7fd1d54af640 1 --2- 172.21.15.88:0/473134844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1d015e660 0x7fd1d00cf730 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:04.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.378+0000 7fd1d54af640 1 -- 172.21.15.88:0/473134844 >> 172.21.15.88:0/473134844 conn(0x7fd1d00b6410 msgr2=0x7fd1d00b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:04.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.378+0000 7fd1d54af640 1 -- 172.21.15.88:0/473134844 shutdown_connections 2024-09-16T01:44:04.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.378+0000 7fd1d54af640 1 -- 172.21.15.88:0/473134844 wait complete. 2024-09-16T01:44:04.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.379+0000 7fd1d54af640 1 Processor -- start 2024-09-16T01:44:04.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.379+0000 7fd1d54af640 1 -- start start 2024-09-16T01:44:04.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.380+0000 7fd1d54af640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1d00d1650 0x7fd1d00ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:04.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.380+0000 7fd1d54af640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1d00d3310 0x7fd1d0152650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:04.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.380+0000 7fd1d54af640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1d015e660 0x7fd1d0152b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:04.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.380+0000 7fd1d54af640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d00d43f0 con 0x7fd1d015e660 2024-09-16T01:44:04.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.380+0000 7fd1d54af640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d00d4110 con 0x7fd1d00d3310 2024-09-16T01:44:04.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.380+0000 7fd1d54af640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd1d00cfc70 con 0x7fd1d00d1650 2024-09-16T01:44:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.380+0000 7fd1cf7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1d015e660 0x7fd1d0152b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.380+0000 7fd1cf7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1d015e660 0x7fd1d0152b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41216/0 (socket says 172.21.15.88:41216) 2024-09-16T01:44:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.380+0000 7fd1cf7fe640 1 -- 172.21.15.88:0/1521889129 learned_addr learned my addr 172.21.15.88:0/1521889129 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.380+0000 7fd1c67fc640 1 --2- 172.21.15.88:0/1521889129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1d00d3310 0x7fd1d0152650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.380+0000 7fd1ceffd640 1 --2- 172.21.15.88:0/1521889129 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1d00d1650 0x7fd1d00ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.380+0000 7fd1cf7fe640 1 -- 172.21.15.88:0/1521889129 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1d00d1650 msgr2=0x7fd1d00ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.380+0000 7fd1cf7fe640 1 --2- 172.21.15.88:0/1521889129 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1d00d1650 0x7fd1d00ca600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.380+0000 7fd1cf7fe640 1 -- 172.21.15.88:0/1521889129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1d00d3310 msgr2=0x7fd1d0152650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.380+0000 7fd1cf7fe640 1 --2- 172.21.15.88:0/1521889129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1d00d3310 0x7fd1d0152650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.380+0000 7fd1cf7fe640 1 -- 172.21.15.88:0/1521889129 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d0146620 con 0x7fd1d015e660 2024-09-16T01:44:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.381+0000 7fd1cf7fe640 1 --2- 172.21.15.88:0/1521889129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1d015e660 0x7fd1d0152b90 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fd1c800c730 tx=0x7fd1c800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.381+0000 7fd1c7fff640 1 -- 172.21.15.88:0/1521889129 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8018070 con 0x7fd1d015e660 2024-09-16T01:44:04.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.381+0000 7fd1c7fff640 1 -- 172.21.15.88:0/1521889129 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1c800ce40 con 0x7fd1d015e660 2024-09-16T01:44:04.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.381+0000 7fd1d54af640 1 -- 172.21.15.88:0/1521889129 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1d0156b70 con 0x7fd1d015e660 2024-09-16T01:44:04.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.381+0000 7fd1c7fff640 1 -- 172.21.15.88:0/1521889129 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c800a860 con 0x7fd1d015e660 2024-09-16T01:44:04.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.381+0000 7fd1d54af640 1 -- 172.21.15.88:0/1521889129 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1d0157130 con 0x7fd1d015e660 2024-09-16T01:44:04.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.382+0000 7fd1d54af640 1 -- 172.21.15.88:0/1521889129 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1d00b4120 con 0x7fd1d015e660 2024-09-16T01:44:04.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.382+0000 7fd1c7fff640 1 -- 172.21.15.88:0/1521889129 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd1c801b070 con 0x7fd1d015e660 2024-09-16T01:44:04.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.383+0000 7fd1c7fff640 1 --2- 172.21.15.88:0/1521889129 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd1b006c2b0 0x7fd1b006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:04.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.383+0000 7fd1c7fff640 1 -- 172.21.15.88:0/1521889129 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8834+0+0 (secure 0 0 0) 0x7fd1c808b590 con 0x7fd1d015e660 2024-09-16T01:44:04.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.383+0000 7fd1ceffd640 1 --2- 172.21.15.88:0/1521889129 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd1b006c2b0 0x7fd1b006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:04.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.383+0000 7fd1ceffd640 1 --2- 172.21.15.88:0/1521889129 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd1b006c2b0 0x7fd1b006e770 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0034000 tx=0x7fd1c002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:04.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.387+0000 7fd1c7fff640 1 -- 172.21.15.88:0/1521889129 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd1c8058a20 con 0x7fd1d015e660 2024-09-16T01:44:04.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.588+0000 7fd1d54af640 1 -- 172.21.15.88:0/1521889129 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1d00d1110 con 0x7fd1d015e660 2024-09-16T01:44:04.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.588+0000 7fd1c7fff640 1 -- 172.21.15.88:0/1521889129 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 290 v290) ==== 95+0+4392 (secure 0 0 0) 0x7fd1c805cd20 con 0x7fd1d015e660 2024-09-16T01:44:04.590 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:04.590 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":290,"btime":"2024-09-16T01:44:01:949065+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9696,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4270197461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4270197461},{"type":"v1","addr":"172.21.15.88:6837","nonce":4270197461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":290,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:44:01.949061+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:44:04.590 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 290 2024-09-16T01:44:04.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.590+0000 7fd1d54af640 1 -- 172.21.15.88:0/1521889129 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd1b006c2b0 msgr2=0x7fd1b006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:04.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.590+0000 7fd1d54af640 1 --2- 172.21.15.88:0/1521889129 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd1b006c2b0 0x7fd1b006e770 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0034000 tx=0x7fd1c002e000 comp rx=0 tx=0).stop 2024-09-16T01:44:04.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.590+0000 7fd1d54af640 1 -- 172.21.15.88:0/1521889129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1d015e660 msgr2=0x7fd1d0152b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:04.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.590+0000 7fd1d54af640 1 --2- 172.21.15.88:0/1521889129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1d015e660 0x7fd1d0152b90 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fd1c800c730 tx=0x7fd1c800cbf0 comp rx=0 tx=0).stop 2024-09-16T01:44:04.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.590+0000 7fd1d54af640 1 -- 172.21.15.88:0/1521889129 shutdown_connections 2024-09-16T01:44:04.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.590+0000 7fd1d54af640 1 --2- 172.21.15.88:0/1521889129 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd1b006c2b0 0x7fd1b006e770 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:04.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.590+0000 7fd1d54af640 1 --2- 172.21.15.88:0/1521889129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1d015e660 0x7fd1d0152b90 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:04.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.590+0000 7fd1d54af640 1 --2- 172.21.15.88:0/1521889129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1d00d3310 0x7fd1d0152650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:04.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.590+0000 7fd1d54af640 1 --2- 172.21.15.88:0/1521889129 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1d00d1650 0x7fd1d00ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:04.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.590+0000 7fd1d54af640 1 -- 172.21.15.88:0/1521889129 >> 172.21.15.88:0/1521889129 conn(0x7fd1d00b6410 msgr2=0x7fd1d00b7ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:04.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.590+0000 7fd1d54af640 1 -- 172.21.15.88:0/1521889129 shutdown_connections 2024-09-16T01:44:04.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.590+0000 7fd1d54af640 1 -- 172.21.15.88:0/1521889129 wait complete. 2024-09-16T01:44:04.606 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-09-16T01:44:04.606 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:44:04.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.704+0000 7f1615f26640 1 Processor -- start 2024-09-16T01:44:04.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.704+0000 7f1615f26640 1 -- start start 2024-09-16T01:44:04.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.705+0000 7f1615f26640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f161015e560 0x7f161006e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:04.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.705+0000 7f1615f26640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16100b5220 con 0x7f1610072220 2024-09-16T01:44:04.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.705+0000 7f1615f26640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16100b54c0 con 0x7f1610073130 2024-09-16T01:44:04.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.705+0000 7f1615f26640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f16100aefa0 con 0x7f161015e560 2024-09-16T01:44:04.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.705+0000 7f160f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f161015e560 0x7f161006e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:04.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.705+0000 7f160effd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1610072220 0x7f16100ae890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44612/0 (socket says 172.21.15.88:44612) 2024-09-16T01:44:04.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.705+0000 7f160effd640 1 -- 172.21.15.88:0/2959868593 learned_addr learned my addr 172.21.15.88:0/2959868593 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:04.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.705+0000 7f160f7fe640 1 -- 172.21.15.88:0/2959868593 >> v1:172.21.15.154:6789/0 conn(0x7f1610073130 legacy=0x7f16100b6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:04.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.705+0000 7f160f7fe640 1 -- 172.21.15.88:0/2959868593 >> v1:172.21.15.88:6789/0 conn(0x7f1610072220 legacy=0x7f16100ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:04.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.705+0000 7f160f7fe640 1 -- 172.21.15.88:0/2959868593 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16100b57a0 con 0x7f161015e560 2024-09-16T01:44:04.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.706+0000 7f160f7fe640 1 --2- 172.21.15.88:0/2959868593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f161015e560 0x7f161006e780 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f1600009d90 tx=0x7f160002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0003bd17eb14760 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:04.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.706+0000 7f160e7fc640 1 -- 172.21.15.88:0/2959868593 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1600031600 con 0x7f161015e560 2024-09-16T01:44:04.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.706+0000 7f160e7fc640 1 -- 172.21.15.88:0/2959868593 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16000317a0 con 0x7f161015e560 2024-09-16T01:44:04.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.706+0000 7f160e7fc640 1 -- 172.21.15.88:0/2959868593 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1600031b70 con 0x7f161015e560 2024-09-16T01:44:04.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.707+0000 7f1615f26640 1 -- 172.21.15.88:0/2959868593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f161015e560 msgr2=0x7f161006e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:04.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.707+0000 7f1615f26640 1 --2- 172.21.15.88:0/2959868593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f161015e560 0x7f161006e780 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f1600009d90 tx=0x7f160002a610 comp rx=0 tx=0).stop 2024-09-16T01:44:04.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.707+0000 7f1615f26640 1 -- 172.21.15.88:0/2959868593 shutdown_connections 2024-09-16T01:44:04.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.707+0000 7f1615f26640 1 --2- 172.21.15.88:0/2959868593 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f161015e560 0x7f161006e780 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:04.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.707+0000 7f1615f26640 1 -- 172.21.15.88:0/2959868593 >> 172.21.15.88:0/2959868593 conn(0x7f161006bc10 msgr2=0x7f161006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:04.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.707+0000 7f1615f26640 1 -- 172.21.15.88:0/2959868593 shutdown_connections 2024-09-16T01:44:04.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.707+0000 7f1615f26640 1 -- 172.21.15.88:0/2959868593 wait complete. 2024-09-16T01:44:04.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.708+0000 7f1615f26640 1 Processor -- start 2024-09-16T01:44:04.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.708+0000 7f1615f26640 1 -- start start 2024-09-16T01:44:04.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.709+0000 7f1615f26640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1610072220 0x7f16100cae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:04.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.709+0000 7f1615f26640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1610073130 0x7f16101545a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:04.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.709+0000 7f1615f26640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f161015e560 0x7f161015f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:04.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.709+0000 7f1615f26640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1610072660 con 0x7f1610072220 2024-09-16T01:44:04.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.709+0000 7f1615f26640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f16100b5d60 con 0x7f1610073130 2024-09-16T01:44:04.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.709+0000 7f1615f26640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f161015e940 con 0x7f161015e560 2024-09-16T01:44:04.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.709+0000 7f160f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1610072220 0x7f16100cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:04.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.709+0000 7f160f7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1610072220 0x7f16100cae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41238/0 (socket says 172.21.15.88:41238) 2024-09-16T01:44:04.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.709+0000 7f160f7fe640 1 -- 172.21.15.88:0/3291125796 learned_addr learned my addr 172.21.15.88:0/3291125796 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:04.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.709+0000 7f160f7fe640 1 -- 172.21.15.88:0/3291125796 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f161015e560 msgr2=0x7f161015f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:04.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.709+0000 7f160f7fe640 1 --2- 172.21.15.88:0/3291125796 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f161015e560 0x7f161015f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:04.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.709+0000 7f160f7fe640 1 -- 172.21.15.88:0/3291125796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1610073130 msgr2=0x7f16101545a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:04.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.709+0000 7f160f7fe640 1 --2- 172.21.15.88:0/3291125796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1610073130 0x7f16101545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:04.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.709+0000 7f160f7fe640 1 -- 172.21.15.88:0/3291125796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16100cb370 con 0x7f1610072220 2024-09-16T01:44:04.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.709+0000 7f160f7fe640 1 --2- 172.21.15.88:0/3291125796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1610072220 0x7f16100cae30 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f1600009d90 tx=0x7f1600004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:04.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.710+0000 7f15f7fff640 1 -- 172.21.15.88:0/3291125796 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f160003f070 con 0x7f1610072220 2024-09-16T01:44:04.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.710+0000 7f15f7fff640 1 -- 172.21.15.88:0/3291125796 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1600030be0 con 0x7f1610072220 2024-09-16T01:44:04.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.710+0000 7f1615f26640 1 -- 172.21.15.88:0/3291125796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16101f4db0 con 0x7f1610072220 2024-09-16T01:44:04.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.710+0000 7f15f7fff640 1 -- 172.21.15.88:0/3291125796 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1600044460 con 0x7f1610072220 2024-09-16T01:44:04.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.710+0000 7f1615f26640 1 -- 172.21.15.88:0/3291125796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16100c6a50 con 0x7f1610072220 2024-09-16T01:44:04.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.711+0000 7f1615f26640 1 -- 172.21.15.88:0/3291125796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f161006e780 con 0x7f1610072220 2024-09-16T01:44:04.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.711+0000 7f15f7fff640 1 -- 172.21.15.88:0/3291125796 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f160003b410 con 0x7f1610072220 2024-09-16T01:44:04.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.712+0000 7f15f7fff640 1 --2- 172.21.15.88:0/3291125796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f15fc06c2b0 0x7f15fc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:04.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.712+0000 7f160effd640 1 --2- 172.21.15.88:0/3291125796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f15fc06c2b0 0x7f15fc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:04.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.712+0000 7f15f7fff640 1 -- 172.21.15.88:0/3291125796 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8834+0+0 (secure 0 0 0) 0x7f16000ad820 con 0x7f1610072220 2024-09-16T01:44:04.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.712+0000 7f160effd640 1 --2- 172.21.15.88:0/3291125796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f15fc06c2b0 0x7f15fc06e770 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f16100c6200 tx=0x7f1604007a40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:04.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.716+0000 7f15f7fff640 1 -- 172.21.15.88:0/3291125796 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f16101f4db0 con 0x7f1610072220 2024-09-16T01:44:04.916 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.915+0000 7f1615f26640 1 -- 172.21.15.88:0/3291125796 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1610075690 con 0x7f1610072220 2024-09-16T01:44:04.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.916+0000 7f15f7fff640 1 -- 172.21.15.88:0/3291125796 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 290 v290) ==== 95+0+4392 (secure 0 0 0) 0x7f160007b300 con 0x7f1610072220 2024-09-16T01:44:04.917 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:04.917 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":290,"btime":"2024-09-16T01:44:01:949065+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9696,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4270197461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4270197461},{"type":"v1","addr":"172.21.15.88:6837","nonce":4270197461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":290,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:44:01.949061+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:44:04.918 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 290 2024-09-16T01:44:04.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.918+0000 7f1615f26640 1 -- 172.21.15.88:0/3291125796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f15fc06c2b0 msgr2=0x7f15fc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:04.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.918+0000 7f1615f26640 1 --2- 172.21.15.88:0/3291125796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f15fc06c2b0 0x7f15fc06e770 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f16100c6200 tx=0x7f1604007a40 comp rx=0 tx=0).stop 2024-09-16T01:44:04.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.918+0000 7f1615f26640 1 -- 172.21.15.88:0/3291125796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1610072220 msgr2=0x7f16100cae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:04.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.918+0000 7f1615f26640 1 --2- 172.21.15.88:0/3291125796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1610072220 0x7f16100cae30 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f1600009d90 tx=0x7f1600004a40 comp rx=0 tx=0).stop 2024-09-16T01:44:04.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.918+0000 7f1615f26640 1 -- 172.21.15.88:0/3291125796 shutdown_connections 2024-09-16T01:44:04.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.918+0000 7f1615f26640 1 --2- 172.21.15.88:0/3291125796 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f15fc06c2b0 0x7f15fc06e770 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:04.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.918+0000 7f1615f26640 1 --2- 172.21.15.88:0/3291125796 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f161015e560 0x7f161015f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:04.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.918+0000 7f1615f26640 1 --2- 172.21.15.88:0/3291125796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1610073130 0x7f16101545a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:04.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.918+0000 7f1615f26640 1 --2- 172.21.15.88:0/3291125796 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1610072220 0x7f16100cae30 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:04.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.918+0000 7f1615f26640 1 -- 172.21.15.88:0/3291125796 >> 172.21.15.88:0/3291125796 conn(0x7f161006bc10 msgr2=0x7f1610161680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:04.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.918+0000 7f1615f26640 1 -- 172.21.15.88:0/3291125796 shutdown_connections 2024-09-16T01:44:04.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:04.918+0000 7f1615f26640 1 -- 172.21.15.88:0/3291125796 wait complete. 2024-09-16T01:44:04.930 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:44:04.930 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:44:05.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.028+0000 7fcf622e4640 1 Processor -- start 2024-09-16T01:44:05.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.028+0000 7fcf622e4640 1 -- start start 2024-09-16T01:44:05.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.029+0000 7fcf622e4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf5c0b6140 0x7fcf5c15c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:05.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.029+0000 7fcf622e4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf5c0b50a0 con 0x7fcf5c0cfc80 2024-09-16T01:44:05.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.029+0000 7fcf622e4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf5c0b5380 con 0x7fcf5c155800 2024-09-16T01:44:05.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.029+0000 7fcf622e4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fcf5c0b6550 con 0x7fcf5c0b6140 2024-09-16T01:44:05.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.029+0000 7fcf60881640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf5c0b6140 0x7fcf5c15c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:05.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.029+0000 7fcf60881640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf5c0b6140 0x7fcf5c15c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41240/0 (socket says 172.21.15.88:41240) 2024-09-16T01:44:05.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.029+0000 7fcf60881640 1 -- 172.21.15.88:0/1238802834 learned_addr learned my addr 172.21.15.88:0/1238802834 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:05.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.029+0000 7fcf5b7fe640 1 --1- 172.21.15.88:0/1238802834 >> v1:172.21.15.88:6789/0 conn(0x7fcf5c0cfc80 0x7fcf5c15c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44628/0 (socket says 172.21.15.88:44628) 2024-09-16T01:44:05.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.029+0000 7fcf60881640 1 -- 172.21.15.88:0/1238802834 >> v1:172.21.15.154:6789/0 conn(0x7fcf5c155800 legacy=0x7fcf5c0cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:05.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.029+0000 7fcf60881640 1 -- 172.21.15.88:0/1238802834 >> v1:172.21.15.88:6789/0 conn(0x7fcf5c0cfc80 legacy=0x7fcf5c15c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:05.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.029+0000 7fcf60881640 1 -- 172.21.15.88:0/1238802834 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf5c0b5660 con 0x7fcf5c0b6140 2024-09-16T01:44:05.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.030+0000 7fcf60881640 1 --2- 172.21.15.88:0/1238802834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf5c0b6140 0x7fcf5c15c330 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7fcf54009e50 tx=0x7fcf5402a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30c73fee0f4ffa21 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:05.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.030+0000 7fcf5affd640 1 -- 172.21.15.88:0/1238802834 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54036070 con 0x7fcf5c0b6140 2024-09-16T01:44:05.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.030+0000 7fcf5affd640 1 -- 172.21.15.88:0/1238802834 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf54031600 con 0x7fcf5c0b6140 2024-09-16T01:44:05.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.030+0000 7fcf5affd640 1 -- 172.21.15.88:0/1238802834 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf540319d0 con 0x7fcf5c0b6140 2024-09-16T01:44:05.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.031+0000 7fcf622e4640 1 -- 172.21.15.88:0/1238802834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf5c0b6140 msgr2=0x7fcf5c15c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:05.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.031+0000 7fcf622e4640 1 --2- 172.21.15.88:0/1238802834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf5c0b6140 0x7fcf5c15c330 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7fcf54009e50 tx=0x7fcf5402a760 comp rx=0 tx=0).stop 2024-09-16T01:44:05.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.031+0000 7fcf622e4640 1 -- 172.21.15.88:0/1238802834 shutdown_connections 2024-09-16T01:44:05.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.031+0000 7fcf622e4640 1 --2- 172.21.15.88:0/1238802834 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf5c0b6140 0x7fcf5c15c330 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:05.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.031+0000 7fcf622e4640 1 -- 172.21.15.88:0/1238802834 >> 172.21.15.88:0/1238802834 conn(0x7fcf5c06bc10 msgr2=0x7fcf5c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:05.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.031+0000 7fcf622e4640 1 -- 172.21.15.88:0/1238802834 shutdown_connections 2024-09-16T01:44:05.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.031+0000 7fcf622e4640 1 -- 172.21.15.88:0/1238802834 wait complete. 2024-09-16T01:44:05.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.032+0000 7fcf622e4640 1 Processor -- start 2024-09-16T01:44:05.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.032+0000 7fcf622e4640 1 -- start start 2024-09-16T01:44:05.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.033+0000 7fcf622e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf5c0b6140 0x7fcf5c146130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:05.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.033+0000 7fcf622e4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf5c0cfc80 0x7fcf5c0b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:05.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.033+0000 7fcf622e4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf5c155800 0x7fcf5c157a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:05.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.033+0000 7fcf622e4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5c155be0 con 0x7fcf5c0cfc80 2024-09-16T01:44:05.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.033+0000 7fcf622e4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5c0b5c20 con 0x7fcf5c155800 2024-09-16T01:44:05.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.033+0000 7fcf622e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcf5c0b4920 con 0x7fcf5c0b6140 2024-09-16T01:44:05.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.033+0000 7fcf5b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf5c0cfc80 0x7fcf5c0b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:05.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.033+0000 7fcf5b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf5c0cfc80 0x7fcf5c0b3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41242/0 (socket says 172.21.15.88:41242) 2024-09-16T01:44:05.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.033+0000 7fcf5b7fe640 1 -- 172.21.15.88:0/3858724329 learned_addr learned my addr 172.21.15.88:0/3858724329 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:05.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.033+0000 7fcf5b7fe640 1 -- 172.21.15.88:0/3858724329 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf5c0b6140 msgr2=0x7fcf5c146130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:05.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.033+0000 7fcf60881640 1 --2- 172.21.15.88:0/3858724329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf5c155800 0x7fcf5c157a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:05.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.033+0000 7fcf5b7fe640 1 --2- 172.21.15.88:0/3858724329 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf5c0b6140 0x7fcf5c146130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:05.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.033+0000 7fcf5b7fe640 1 -- 172.21.15.88:0/3858724329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf5c155800 msgr2=0x7fcf5c157a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:05.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.033+0000 7fcf5b7fe640 1 --2- 172.21.15.88:0/3858724329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf5c155800 0x7fcf5c157a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:05.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.033+0000 7fcf5b7fe640 1 -- 172.21.15.88:0/3858724329 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf5c1560d0 con 0x7fcf5c0cfc80 2024-09-16T01:44:05.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.033+0000 7fcf5b7fe640 1 --2- 172.21.15.88:0/3858724329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf5c0cfc80 0x7fcf5c0b3ac0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7fcf5000c470 tx=0x7fcf50001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:05.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.033+0000 7fcf58ff9640 1 -- 172.21.15.88:0/3858724329 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf500074e0 con 0x7fcf5c0cfc80 2024-09-16T01:44:05.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.034+0000 7fcf58ff9640 1 -- 172.21.15.88:0/3858724329 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf50007680 con 0x7fcf5c0cfc80 2024-09-16T01:44:05.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.034+0000 7fcf622e4640 1 -- 172.21.15.88:0/3858724329 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf5c1563b0 con 0x7fcf5c0cfc80 2024-09-16T01:44:05.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.034+0000 7fcf58ff9640 1 -- 172.21.15.88:0/3858724329 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf50004a20 con 0x7fcf5c0cfc80 2024-09-16T01:44:05.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.034+0000 7fcf622e4640 1 -- 172.21.15.88:0/3858724329 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf5c156970 con 0x7fcf5c0cfc80 2024-09-16T01:44:05.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.035+0000 7fcf622e4640 1 -- 172.21.15.88:0/3858724329 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf5c0b4cb0 con 0x7fcf5c0cfc80 2024-09-16T01:44:05.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.035+0000 7fcf58ff9640 1 -- 172.21.15.88:0/3858724329 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fcf5001a020 con 0x7fcf5c0cfc80 2024-09-16T01:44:05.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.035+0000 7fcf58ff9640 1 --2- 172.21.15.88:0/3858724329 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcf4806c2b0 0x7fcf4806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:05.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.036+0000 7fcf58ff9640 1 -- 172.21.15.88:0/3858724329 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8834+0+0 (secure 0 0 0) 0x7fcf5008b950 con 0x7fcf5c0cfc80 2024-09-16T01:44:05.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.036+0000 7fcf5bfff640 1 --2- 172.21.15.88:0/3858724329 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcf4806c2b0 0x7fcf4806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:05.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.036+0000 7fcf5bfff640 1 --2- 172.21.15.88:0/3858724329 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcf4806c2b0 0x7fcf4806e770 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c006fd0 tx=0x7fcf4c006bc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:05.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.040+0000 7fcf58ff9640 1 -- 172.21.15.88:0/3858724329 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fcf500593a0 con 0x7fcf5c0cfc80 2024-09-16T01:44:05.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.240+0000 7fcf622e4640 1 -- 172.21.15.88:0/3858724329 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf5c071960 con 0x7fcf5c0cfc80 2024-09-16T01:44:05.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.240+0000 7fcf58ff9640 1 -- 172.21.15.88:0/3858724329 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 290 v290) ==== 95+0+4392 (secure 0 0 0) 0x7fcf50058ca0 con 0x7fcf5c0cfc80 2024-09-16T01:44:05.242 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:05.242 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":290,"btime":"2024-09-16T01:44:01:949065+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9696,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4270197461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4270197461},{"type":"v1","addr":"172.21.15.88:6837","nonce":4270197461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":290,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:44:01.949061+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:44:05.242 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 290 2024-09-16T01:44:05.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.242+0000 7fcf622e4640 1 -- 172.21.15.88:0/3858724329 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcf4806c2b0 msgr2=0x7fcf4806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:05.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.242+0000 7fcf622e4640 1 --2- 172.21.15.88:0/3858724329 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcf4806c2b0 0x7fcf4806e770 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c006fd0 tx=0x7fcf4c006bc0 comp rx=0 tx=0).stop 2024-09-16T01:44:05.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.242+0000 7fcf622e4640 1 -- 172.21.15.88:0/3858724329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf5c0cfc80 msgr2=0x7fcf5c0b3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:05.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.242+0000 7fcf622e4640 1 --2- 172.21.15.88:0/3858724329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf5c0cfc80 0x7fcf5c0b3ac0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7fcf5000c470 tx=0x7fcf50001e70 comp rx=0 tx=0).stop 2024-09-16T01:44:05.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.242+0000 7fcf622e4640 1 -- 172.21.15.88:0/3858724329 shutdown_connections 2024-09-16T01:44:05.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.242+0000 7fcf622e4640 1 --2- 172.21.15.88:0/3858724329 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcf4806c2b0 0x7fcf4806e770 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:05.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.242+0000 7fcf622e4640 1 --2- 172.21.15.88:0/3858724329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf5c155800 0x7fcf5c157a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:05.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.242+0000 7fcf622e4640 1 --2- 172.21.15.88:0/3858724329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf5c0cfc80 0x7fcf5c0b3ac0 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:05.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.242+0000 7fcf622e4640 1 --2- 172.21.15.88:0/3858724329 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf5c0b6140 0x7fcf5c146130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:05.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.242+0000 7fcf622e4640 1 -- 172.21.15.88:0/3858724329 >> 172.21.15.88:0/3858724329 conn(0x7fcf5c06bc10 msgr2=0x7fcf5c0ce8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:05.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.242+0000 7fcf622e4640 1 -- 172.21.15.88:0/3858724329 shutdown_connections 2024-09-16T01:44:05.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.242+0000 7fcf622e4640 1 -- 172.21.15.88:0/3858724329 wait complete. 2024-09-16T01:44:05.254 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt omap header on 600.00000000' led to DAMAGED state 2024-09-16T01:44:05.255 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap header on mds0_openfiles.0' 2024-09-16T01:44:05.255 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:44:05.255 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:44:05.270 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:44:05.270 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:44:05.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:05.271 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:44:05.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:44:05.327 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:44:05.327 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:44:05.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.466+0000 7ff80c566640 1 Processor -- start 2024-09-16T01:44:05.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.466+0000 7ff80c566640 1 -- start start 2024-09-16T01:44:05.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.467+0000 7ff80c566640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff80415b480 0x7ff8040b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:05.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.467+0000 7ff80c566640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff804072700 0x7ff80415bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:05.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.467+0000 7ff80c566640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8040ccff0 con 0x7ff8040b60c0 2024-09-16T01:44:05.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.467+0000 7ff80c566640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff8040cd2d0 con 0x7ff80415b480 2024-09-16T01:44:05.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.467+0000 7ff80c566640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff8040cd5b0 con 0x7ff804072700 2024-09-16T01:44:05.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.468+0000 7ff80ab03640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff804072700 0x7ff80415bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:05.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.468+0000 7ff80a302640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff80415b480 0x7ff8040b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:05.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.468+0000 7ff80ab03640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff804072700 0x7ff80415bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:39626/0 (socket says 172.21.15.88:39626) 2024-09-16T01:44:05.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.468+0000 7ff809b01640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7ff8040b60c0 0x7ff804071ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:51620/0 (socket says 172.21.15.88:51620) 2024-09-16T01:44:05.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.468+0000 7ff80ab03640 1 -- 172.21.15.88:0/1009910592 learned_addr learned my addr 172.21.15.88:0/1009910592 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:05.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.468+0000 7ff80a302640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff80415b480 0x7ff8040b5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:48900/0 (socket says 172.21.15.88:48900) 2024-09-16T01:44:05.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.468+0000 7ff80ab03640 1 -- 172.21.15.88:0/1009910592 >> v2:172.21.15.154:3300/0 conn(0x7ff80415b480 msgr2=0x7ff8040b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:05.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.468+0000 7ff80ab03640 1 --2- 172.21.15.88:0/1009910592 >> v2:172.21.15.154:3300/0 conn(0x7ff80415b480 0x7ff8040b5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:05.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.468+0000 7ff80ab03640 1 -- 172.21.15.88:0/1009910592 >> v1:172.21.15.154:6789/0 conn(0x7ff8040b60c0 legacy=0x7ff804071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:05.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.468+0000 7ff80ab03640 1 -- 172.21.15.88:0/1009910592 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8040cd890 con 0x7ff804072700 2024-09-16T01:44:05.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.468+0000 7ff809300640 1 -- 172.21.15.88:0/1009910592 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3869552395 0 0) 0x7ff8040ccff0 con 0x7ff8040b60c0 2024-09-16T01:44:05.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.468+0000 7ff80a302640 1 --2- 172.21.15.88:0/1009910592 >> v2:172.21.15.154:3300/0 conn(0x7ff80415b480 0x7ff8040b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:44:05.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.469+0000 7ff80ab03640 1 --2- 172.21.15.88:0/1009910592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff804072700 0x7ff80415bc60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc009e50 tx=0x7ff7fc02a6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a503d64aa9c6c77d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:05.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.469+0000 7ff809300640 1 -- 172.21.15.88:0/1009910592 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc036070 con 0x7ff804072700 2024-09-16T01:44:05.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.469+0000 7ff809300640 1 -- 172.21.15.88:0/1009910592 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7fc031600 con 0x7ff804072700 2024-09-16T01:44:05.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.469+0000 7ff809300640 1 -- 172.21.15.88:0/1009910592 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc0319d0 con 0x7ff804072700 2024-09-16T01:44:05.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.470+0000 7ff80c566640 1 -- 172.21.15.88:0/1009910592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff804072700 msgr2=0x7ff80415bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:05.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.470+0000 7ff80c566640 1 --2- 172.21.15.88:0/1009910592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff804072700 0x7ff80415bc60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc009e50 tx=0x7ff7fc02a6a0 comp rx=0 tx=0).stop 2024-09-16T01:44:05.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.470+0000 7ff80c566640 1 -- 172.21.15.88:0/1009910592 shutdown_connections 2024-09-16T01:44:05.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.470+0000 7ff80c566640 1 --2- 172.21.15.88:0/1009910592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff804072700 0x7ff80415bc60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:05.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.470+0000 7ff80c566640 1 --2- 172.21.15.88:0/1009910592 >> v2:172.21.15.154:3300/0 conn(0x7ff80415b480 0x7ff8040b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:05.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.470+0000 7ff80c566640 1 -- 172.21.15.88:0/1009910592 >> 172.21.15.88:0/1009910592 conn(0x7ff80406d0f0 msgr2=0x7ff80406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:05.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.470+0000 7ff80c566640 1 -- 172.21.15.88:0/1009910592 shutdown_connections 2024-09-16T01:44:05.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.470+0000 7ff80c566640 1 -- 172.21.15.88:0/1009910592 wait complete. 2024-09-16T01:44:05.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.471+0000 7ff80c566640 1 Processor -- start 2024-09-16T01:44:05.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.471+0000 7ff80c566640 1 -- start start 2024-09-16T01:44:05.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.472+0000 7ff80c566640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff804072700 0x7ff804149f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:05.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.472+0000 7ff80c566640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8040b60c0 0x7ff804159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:05.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.472+0000 7ff80c566640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff80415b480 0x7ff80415a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:05.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.472+0000 7ff80c566640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff80415b9e0 con 0x7ff8040b60c0 2024-09-16T01:44:05.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.472+0000 7ff80c566640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff80415b860 con 0x7ff804072700 2024-09-16T01:44:05.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.472+0000 7ff80c566640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff80415c1a0 con 0x7ff80415b480 2024-09-16T01:44:05.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.472+0000 7ff809b01640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8040b60c0 0x7ff804159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:05.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.472+0000 7ff809b01640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8040b60c0 0x7ff804159bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41246/0 (socket says 172.21.15.88:41246) 2024-09-16T01:44:05.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.472+0000 7ff809b01640 1 -- 172.21.15.88:0/1593400143 learned_addr learned my addr 172.21.15.88:0/1593400143 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:05.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.472+0000 7ff80a302640 1 --2- 172.21.15.88:0/1593400143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff804072700 0x7ff804149f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:05.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.472+0000 7ff80ab03640 1 --2- 172.21.15.88:0/1593400143 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff80415b480 0x7ff80415a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:05.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.472+0000 7ff809b01640 1 -- 172.21.15.88:0/1593400143 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff80415b480 msgr2=0x7ff80415a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:05.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.472+0000 7ff809b01640 1 --2- 172.21.15.88:0/1593400143 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff80415b480 0x7ff80415a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:05.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.472+0000 7ff809b01640 1 -- 172.21.15.88:0/1593400143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff804072700 msgr2=0x7ff804149f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:05.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.472+0000 7ff809b01640 1 --2- 172.21.15.88:0/1593400143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff804072700 0x7ff804149f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:05.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.472+0000 7ff809b01640 1 -- 172.21.15.88:0/1593400143 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8040c9670 con 0x7ff8040b60c0 2024-09-16T01:44:05.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.472+0000 7ff80a302640 1 --2- 172.21.15.88:0/1593400143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff804072700 0x7ff804149f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:05.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.473+0000 7ff809b01640 1 --2- 172.21.15.88:0/1593400143 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8040b60c0 0x7ff804159bf0 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7ff7f400c4d0 tx=0x7ff7f400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:05.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.473+0000 7ff7faffd640 1 -- 172.21.15.88:0/1593400143 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4018070 con 0x7ff8040b60c0 2024-09-16T01:44:05.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.473+0000 7ff7faffd640 1 -- 172.21.15.88:0/1593400143 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7f4007500 con 0x7ff8040b60c0 2024-09-16T01:44:05.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.473+0000 7ff80c566640 1 -- 172.21.15.88:0/1593400143 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8040c6170 con 0x7ff8040b60c0 2024-09-16T01:44:05.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.473+0000 7ff7faffd640 1 -- 172.21.15.88:0/1593400143 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4004a20 con 0x7ff8040b60c0 2024-09-16T01:44:05.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.473+0000 7ff80c566640 1 -- 172.21.15.88:0/1593400143 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8040c6730 con 0x7ff8040b60c0 2024-09-16T01:44:05.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.474+0000 7ff80c566640 1 -- 172.21.15.88:0/1593400143 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff80415bc60 con 0x7ff8040b60c0 2024-09-16T01:44:05.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.474+0000 7ff7faffd640 1 -- 172.21.15.88:0/1593400143 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff7f401b070 con 0x7ff8040b60c0 2024-09-16T01:44:05.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.475+0000 7ff7faffd640 1 --2- 172.21.15.88:0/1593400143 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff7e806c2b0 0x7ff7e806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:05.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.475+0000 7ff7faffd640 1 -- 172.21.15.88:0/1593400143 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8834+0+0 (secure 0 0 0) 0x7ff7f408b410 con 0x7ff8040b60c0 2024-09-16T01:44:05.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.475+0000 7ff80a302640 1 --2- 172.21.15.88:0/1593400143 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff7e806c2b0 0x7ff7e806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:05.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.475+0000 7ff80a302640 1 --2- 172.21.15.88:0/1593400143 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff7e806c2b0 0x7ff7e806e770 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7ff8040726d0 tx=0x7ff800008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:05.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.479+0000 7ff7faffd640 1 -- 172.21.15.88:0/1593400143 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff7f4058ef0 con 0x7ff8040b60c0 2024-09-16T01:44:05.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.682+0000 7ff80c566640 1 -- 172.21.15.88:0/1593400143 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ff80415eaf0 con 0x7ff8040b60c0 2024-09-16T01:44:05.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.966+0000 7ff7faffd640 1 -- 172.21.15.88:0/1593400143 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v291) ==== 161+0+0 (secure 0 0 0) 0x7ff7f40587f0 con 0x7ff8040b60c0 2024-09-16T01:44:05.968 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:44:05.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.969+0000 7ff80c566640 1 -- 172.21.15.88:0/1593400143 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff7e806c2b0 msgr2=0x7ff7e806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:05.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.969+0000 7ff80c566640 1 --2- 172.21.15.88:0/1593400143 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff7e806c2b0 0x7ff7e806e770 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7ff8040726d0 tx=0x7ff800008040 comp rx=0 tx=0).stop 2024-09-16T01:44:05.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.969+0000 7ff80c566640 1 -- 172.21.15.88:0/1593400143 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8040b60c0 msgr2=0x7ff804159bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:05.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.969+0000 7ff80c566640 1 --2- 172.21.15.88:0/1593400143 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8040b60c0 0x7ff804159bf0 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7ff7f400c4d0 tx=0x7ff7f400c990 comp rx=0 tx=0).stop 2024-09-16T01:44:05.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.969+0000 7ff80c566640 1 -- 172.21.15.88:0/1593400143 shutdown_connections 2024-09-16T01:44:05.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.969+0000 7ff80c566640 1 --2- 172.21.15.88:0/1593400143 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff7e806c2b0 0x7ff7e806e770 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:05.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.969+0000 7ff80c566640 1 --2- 172.21.15.88:0/1593400143 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff80415b480 0x7ff80415a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:05.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.969+0000 7ff80c566640 1 --2- 172.21.15.88:0/1593400143 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8040b60c0 0x7ff804159bf0 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:05.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.969+0000 7ff80c566640 1 --2- 172.21.15.88:0/1593400143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff804072700 0x7ff804149f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:05.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.969+0000 7ff80c566640 1 -- 172.21.15.88:0/1593400143 >> 172.21.15.88:0/1593400143 conn(0x7ff80406d0f0 msgr2=0x7ff8040710b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:05.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.969+0000 7ff80c566640 1 -- 172.21.15.88:0/1593400143 shutdown_connections 2024-09-16T01:44:05.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:05.969+0000 7ff80c566640 1 -- 172.21.15.88:0/1593400143 wait complete. 2024-09-16T01:44:05.981 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:44:06.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.079+0000 7f6229a0e640 1 Processor -- start 2024-09-16T01:44:06.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.080+0000 7f6229a0e640 1 -- start start 2024-09-16T01:44:06.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.080+0000 7f6229a0e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f622415e7c0 0x7f62240d1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:06.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.080+0000 7f6229a0e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62240ce9e0 0x7f62240b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:06.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.080+0000 7f6229a0e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6224074090 con 0x7f62240ce3b0 2024-09-16T01:44:06.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.080+0000 7f6229a0e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f62240ccd10 con 0x7f622415e7c0 2024-09-16T01:44:06.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.080+0000 7f6229a0e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f62240ccff0 con 0x7f62240ce9e0 2024-09-16T01:44:06.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.080+0000 7f6222ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f622415e7c0 0x7f62240d1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:06.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.081+0000 7f6222ffd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f622415e7c0 0x7f62240d1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41252/0 (socket says 172.21.15.88:41252) 2024-09-16T01:44:06.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.081+0000 7f6222ffd640 1 -- 172.21.15.88:0/3117366137 learned_addr learned my addr 172.21.15.88:0/3117366137 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:06.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.081+0000 7f6222ffd640 1 -- 172.21.15.88:0/3117366137 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62240ce9e0 msgr2=0x7f62240b44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:06.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.081+0000 7f6222ffd640 1 --2- 172.21.15.88:0/3117366137 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62240ce9e0 0x7f62240b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:06.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.081+0000 7f6222ffd640 1 -- 172.21.15.88:0/3117366137 >> v1:172.21.15.154:6789/0 conn(0x7f62240ce3b0 legacy=0x7f62240cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:06.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.081+0000 7f6222ffd640 1 -- 172.21.15.88:0/3117366137 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62240cd2d0 con 0x7f622415e7c0 2024-09-16T01:44:06.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.081+0000 7f6222ffd640 1 --2- 172.21.15.88:0/3117366137 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f622415e7c0 0x7f62240d1330 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f6214009870 tx=0x7f621402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=912372f198132eba server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:06.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.082+0000 7f6221ffb640 1 -- 172.21.15.88:0/3117366137 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6214037070 con 0x7f622415e7c0 2024-09-16T01:44:06.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.082+0000 7f6221ffb640 1 -- 172.21.15.88:0/3117366137 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6214032440 con 0x7f622415e7c0 2024-09-16T01:44:06.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.082+0000 7f6221ffb640 1 -- 172.21.15.88:0/3117366137 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6214032810 con 0x7f622415e7c0 2024-09-16T01:44:06.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.082+0000 7f6229a0e640 1 -- 172.21.15.88:0/3117366137 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f622415e7c0 msgr2=0x7f62240d1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:06.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.082+0000 7f6229a0e640 1 --2- 172.21.15.88:0/3117366137 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f622415e7c0 0x7f62240d1330 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f6214009870 tx=0x7f621402a210 comp rx=0 tx=0).stop 2024-09-16T01:44:06.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.082+0000 7f6229a0e640 1 -- 172.21.15.88:0/3117366137 shutdown_connections 2024-09-16T01:44:06.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.082+0000 7f6229a0e640 1 --2- 172.21.15.88:0/3117366137 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62240ce9e0 0x7f62240b44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:06.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.082+0000 7f6229a0e640 1 --2- 172.21.15.88:0/3117366137 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f622415e7c0 0x7f62240d1330 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:06.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.082+0000 7f6229a0e640 1 -- 172.21.15.88:0/3117366137 >> 172.21.15.88:0/3117366137 conn(0x7f6224062a20 msgr2=0x7f6224062e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:06.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.082+0000 7f6229a0e640 1 -- 172.21.15.88:0/3117366137 shutdown_connections 2024-09-16T01:44:06.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.082+0000 7f6229a0e640 1 -- 172.21.15.88:0/3117366137 wait complete. 2024-09-16T01:44:06.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.083+0000 7f6229a0e640 1 Processor -- start 2024-09-16T01:44:06.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.084+0000 7f6229a0e640 1 -- start start 2024-09-16T01:44:06.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.084+0000 7f6229a0e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62240ce3b0 0x7f62240cc240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:06.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.084+0000 7f6229a0e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62240ce9e0 0x7f6224156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:06.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.084+0000 7f6229a0e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f622415e7c0 0x7f62240c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:06.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.084+0000 7f6229a0e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6224073da0 con 0x7f622415e7c0 2024-09-16T01:44:06.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.084+0000 7f6229a0e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6224073c20 con 0x7f62240ce3b0 2024-09-16T01:44:06.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.084+0000 7f6229a0e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f62240cdf00 con 0x7f62240ce9e0 2024-09-16T01:44:06.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.084+0000 7f62237fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f622415e7c0 0x7f62240c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:06.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.084+0000 7f62237fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f622415e7c0 0x7f62240c9fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41260/0 (socket says 172.21.15.88:41260) 2024-09-16T01:44:06.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.084+0000 7f62237fe640 1 -- 172.21.15.88:0/1652051831 learned_addr learned my addr 172.21.15.88:0/1652051831 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.085+0000 7f62227fc640 1 --2- 172.21.15.88:0/1652051831 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62240ce9e0 0x7f6224156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.085+0000 7f6222ffd640 1 --2- 172.21.15.88:0/1652051831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62240ce3b0 0x7f62240cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.085+0000 7f62237fe640 1 -- 172.21.15.88:0/1652051831 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62240ce9e0 msgr2=0x7f6224156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.085+0000 7f62237fe640 1 --2- 172.21.15.88:0/1652051831 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62240ce9e0 0x7f6224156d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.085+0000 7f62237fe640 1 -- 172.21.15.88:0/1652051831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62240ce3b0 msgr2=0x7f62240cc240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.085+0000 7f62237fe640 1 --2- 172.21.15.88:0/1652051831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62240ce3b0 0x7f62240cc240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.085+0000 7f62237fe640 1 -- 172.21.15.88:0/1652051831 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62240c78b0 con 0x7f622415e7c0 2024-09-16T01:44:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.085+0000 7f62237fe640 1 --2- 172.21.15.88:0/1652051831 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f622415e7c0 0x7f62240c9fe0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f621c00c780 tx=0x7f621c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.085+0000 7f620bfff640 1 -- 172.21.15.88:0/1652051831 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c017070 con 0x7f622415e7c0 2024-09-16T01:44:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.085+0000 7f620bfff640 1 -- 172.21.15.88:0/1652051831 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f621c003da0 con 0x7f622415e7c0 2024-09-16T01:44:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.085+0000 7f6229a0e640 1 -- 172.21.15.88:0/1652051831 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62240c7b90 con 0x7f622415e7c0 2024-09-16T01:44:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.085+0000 7f620bfff640 1 -- 172.21.15.88:0/1652051831 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c0052b0 con 0x7f622415e7c0 2024-09-16T01:44:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.085+0000 7f6229a0e640 1 -- 172.21.15.88:0/1652051831 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62240c8150 con 0x7f622415e7c0 2024-09-16T01:44:06.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.086+0000 7f6229a0e640 1 -- 172.21.15.88:0/1652051831 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62240cd7f0 con 0x7f622415e7c0 2024-09-16T01:44:06.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.087+0000 7f620bfff640 1 -- 172.21.15.88:0/1652051831 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f621c01a070 con 0x7f622415e7c0 2024-09-16T01:44:06.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.087+0000 7f620bfff640 1 --2- 172.21.15.88:0/1652051831 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f621006c2b0 0x7f621006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:06.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.087+0000 7f620bfff640 1 -- 172.21.15.88:0/1652051831 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8834+0+0 (secure 0 0 0) 0x7f621c08b680 con 0x7f622415e7c0 2024-09-16T01:44:06.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.087+0000 7f6222ffd640 1 --2- 172.21.15.88:0/1652051831 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f621006c2b0 0x7f621006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:06.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.088+0000 7f6222ffd640 1 --2- 172.21.15.88:0/1652051831 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f621006c2b0 0x7f621006e770 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f6214034000 tx=0x7f621402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:06.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.091+0000 7f620bfff640 1 -- 172.21.15.88:0/1652051831 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f621c0591f0 con 0x7f622415e7c0 2024-09-16T01:44:06.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.321+0000 7f6229a0e640 1 -- 172.21.15.88:0/1652051831 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f62240b3cf0 con 0x7f622415e7c0 2024-09-16T01:44:06.700 INFO:tasks.ceph.mds.a.smithi088.stderr: -49> 2024-09-16T01:44:01.947+0000 7f48d62d0640 -1 mds.0.cache.dir(0x600) Corrupt fnode in dirfrag 0x600: Decoder at 'void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&)' v=3 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-09-16T01:44:06.700 INFO:tasks.ceph.mds.a.smithi088.stderr: -46> 2024-09-16T01:44:01.947+0000 7f48d62d0640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x600 is fatal because it is a system directory for this rank 2024-09-16T01:44:06.701 INFO:tasks.ceph.mds.a.smithi088.stderr: -34> 2024-09-16T01:44:02.654+0000 7f48da2d8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:44:06.725 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:44:06.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.968+0000 7f620bfff640 1 -- 172.21.15.88:0/1652051831 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v292) ==== 101+0+0 (secure 0 0 0) 0x7f621c058af0 con 0x7f622415e7c0 2024-09-16T01:44:06.969 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:44:06.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.970+0000 7f6229a0e640 1 -- 172.21.15.88:0/1652051831 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f621006c2b0 msgr2=0x7f621006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:06.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.970+0000 7f6229a0e640 1 --2- 172.21.15.88:0/1652051831 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f621006c2b0 0x7f621006e770 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f6214034000 tx=0x7f621402e000 comp rx=0 tx=0).stop 2024-09-16T01:44:06.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.970+0000 7f6229a0e640 1 -- 172.21.15.88:0/1652051831 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f622415e7c0 msgr2=0x7f62240c9fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:06.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.970+0000 7f6229a0e640 1 --2- 172.21.15.88:0/1652051831 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f622415e7c0 0x7f62240c9fe0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f621c00c780 tx=0x7f621c00cc40 comp rx=0 tx=0).stop 2024-09-16T01:44:06.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.970+0000 7f6229a0e640 1 -- 172.21.15.88:0/1652051831 shutdown_connections 2024-09-16T01:44:06.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.970+0000 7f6229a0e640 1 --2- 172.21.15.88:0/1652051831 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f621006c2b0 0x7f621006e770 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:06.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.971+0000 7f6229a0e640 1 --2- 172.21.15.88:0/1652051831 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f622415e7c0 0x7f62240c9fe0 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:06.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.971+0000 7f6229a0e640 1 --2- 172.21.15.88:0/1652051831 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62240ce9e0 0x7f6224156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:06.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.971+0000 7f6229a0e640 1 --2- 172.21.15.88:0/1652051831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62240ce3b0 0x7f62240cc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:06.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.971+0000 7f6229a0e640 1 -- 172.21.15.88:0/1652051831 >> 172.21.15.88:0/1652051831 conn(0x7f6224062a20 msgr2=0x7f62240b44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:06.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.971+0000 7f6229a0e640 1 -- 172.21.15.88:0/1652051831 shutdown_connections 2024-09-16T01:44:06.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:06.971+0000 7f6229a0e640 1 -- 172.21.15.88:0/1652051831 wait complete. 2024-09-16T01:44:06.985 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:44:07.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.028+0000 7fe9e8146f40 1 Processor -- start 2024-09-16T01:44:07.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.028+0000 7fe9e8146f40 1 -- start start 2024-09-16T01:44:07.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.028+0000 7fe9e8146f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9d4002820 0x558def554900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.029+0000 7fe9e8146f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558def555140 0x558def4c9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.029+0000 7fe9e8146f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558def576100 0x558def4c9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.029+0000 7fe9e8146f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558def4e85b0 con 0x7fe9d4002820 2024-09-16T01:44:07.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.029+0000 7fe9e8146f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558def3ef510 con 0x558def576100 2024-09-16T01:44:07.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.029+0000 7fe9e8146f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x558def49f1e0 con 0x558def555140 2024-09-16T01:44:07.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.029+0000 7fe9e5c3c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9d4002820 0x558def554900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.029+0000 7fe9e5c3c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9d4002820 0x558def554900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41308/0 (socket says 172.21.15.88:41308) 2024-09-16T01:44:07.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.029+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/2286040970 learned_addr learned my addr 172.21.15.88:0/2286040970 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:07.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.029+0000 7fe9dffff640 1 --2- 172.21.15.88:0/2286040970 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558def555140 0x558def4c9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.029+0000 7fe9e643d640 1 --2- 172.21.15.88:0/2286040970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558def576100 0x558def4c9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.029+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/2286040970 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558def555140 msgr2=0x558def4c9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:07.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.029+0000 7fe9e5c3c640 1 --2- 172.21.15.88:0/2286040970 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558def555140 0x558def4c9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.029+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/2286040970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558def576100 msgr2=0x558def4c9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:07.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.029+0000 7fe9e5c3c640 1 --2- 172.21.15.88:0/2286040970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558def576100 0x558def4c9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.029+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/2286040970 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558def573680 con 0x7fe9d4002820 2024-09-16T01:44:07.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.029+0000 7fe9dffff640 1 --2- 172.21.15.88:0/2286040970 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558def555140 0x558def4c9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:07.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.029+0000 7fe9e5c3c640 1 --2- 172.21.15.88:0/2286040970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9d4002820 0x558def554900 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8009810 tx=0x7fe9d8005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=153faaa038b7cb58 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:07.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.029+0000 7fe9e4c3a640 1 -- 172.21.15.88:0/2286040970 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8007800 con 0x7fe9d4002820 2024-09-16T01:44:07.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.029+0000 7fe9e4c3a640 1 -- 172.21.15.88:0/2286040970 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9d8019070 con 0x7fe9d4002820 2024-09-16T01:44:07.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.029+0000 7fe9e4c3a640 1 -- 172.21.15.88:0/2286040970 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8007ba0 con 0x7fe9d4002820 2024-09-16T01:44:07.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.030+0000 7fe9e8146f40 1 -- 172.21.15.88:0/2286040970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9d4002820 msgr2=0x558def554900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:07.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.030+0000 7fe9e8146f40 1 --2- 172.21.15.88:0/2286040970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9d4002820 0x558def554900 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8009810 tx=0x7fe9d8005a00 comp rx=0 tx=0).stop 2024-09-16T01:44:07.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.030+0000 7fe9e8146f40 1 -- 172.21.15.88:0/2286040970 shutdown_connections 2024-09-16T01:44:07.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.030+0000 7fe9e8146f40 1 --2- 172.21.15.88:0/2286040970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558def576100 0x558def4c9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.030+0000 7fe9e8146f40 1 --2- 172.21.15.88:0/2286040970 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558def555140 0x558def4c9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.030+0000 7fe9e8146f40 1 --2- 172.21.15.88:0/2286040970 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9d4002820 0x558def554900 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.030+0000 7fe9e8146f40 1 -- 172.21.15.88:0/2286040970 >> 172.21.15.88:0/2286040970 conn(0x558def4e8750 msgr2=0x558def49e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:07.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.030+0000 7fe9e8146f40 1 -- 172.21.15.88:0/2286040970 shutdown_connections 2024-09-16T01:44:07.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.030+0000 7fe9e8146f40 1 -- 172.21.15.88:0/2286040970 wait complete. 2024-09-16T01:44:07.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.030+0000 7fe9e8146f40 1 Processor -- start 2024-09-16T01:44:07.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.030+0000 7fe9e8146f40 1 -- start start 2024-09-16T01:44:07.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.031+0000 7fe9e8146f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558def555140 0x558def6055c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.031+0000 7fe9e8146f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558def576100 0x558def551060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.031+0000 7fe9e8146f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe9d4002820 0x558def573fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.031+0000 7fe9e8146f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x558def573960 con 0x558def555140 2024-09-16T01:44:07.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.031+0000 7fe9e8146f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558def575570 con 0x558def576100 2024-09-16T01:44:07.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.031+0000 7fe9e8146f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x558def575200 con 0x7fe9d4002820 2024-09-16T01:44:07.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.031+0000 7fe9e5c3c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558def555140 0x558def6055c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.031+0000 7fe9e5c3c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558def555140 0x558def6055c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41312/0 (socket says 172.21.15.88:41312) 2024-09-16T01:44:07.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.031+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 learned_addr learned my addr 172.21.15.88:0/1868876081 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:07.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.031+0000 7fe9dffff640 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558def576100 0x558def551060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.031+0000 7fe9e643d640 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe9d4002820 0x558def573fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.031+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe9d4002820 msgr2=0x558def573fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:07.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.031+0000 7fe9e5c3c640 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe9d4002820 0x558def573fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.031+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558def576100 msgr2=0x558def551060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:07.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.031+0000 7fe9e5c3c640 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558def576100 0x558def551060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.031+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558def554030 con 0x558def555140 2024-09-16T01:44:07.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.032+0000 7fe9e5c3c640 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558def555140 0x558def6055c0 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8000c00 tx=0x7fe9d8002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:07.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.032+0000 7fe9deffd640 1 -- 172.21.15.88:0/1868876081 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8027670 con 0x558def555140 2024-09-16T01:44:07.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.032+0000 7fe9deffd640 1 -- 172.21.15.88:0/1868876081 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9d8022070 con 0x558def555140 2024-09-16T01:44:07.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.032+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x558def554310 con 0x558def555140 2024-09-16T01:44:07.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.032+0000 7fe9deffd640 1 -- 172.21.15.88:0/1868876081 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8027a70 con 0x558def555140 2024-09-16T01:44:07.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.032+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558def6050a0 con 0x558def555140 2024-09-16T01:44:07.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.033+0000 7fe9deffd640 1 -- 172.21.15.88:0/1868876081 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe9d8013070 con 0x558def555140 2024-09-16T01:44:07.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.034+0000 7fe9deffd640 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe9c806acc0 0x7fe9c806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.034+0000 7fe9deffd640 1 -- 172.21.15.88:0/1868876081 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8834+0+0 (secure 0 0 0) 0x7fe9d808cc30 con 0x558def555140 2024-09-16T01:44:07.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.034+0000 7fe9dffff640 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe9c806acc0 0x7fe9c806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.041 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:44:07.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.034+0000 7fe9e8146f40 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x558def4c8ff0 0x558def569290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.034+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def5697d0 con 0x558def4c8ff0 2024-09-16T01:44:07.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.034+0000 7fe9e643d640 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x558def4c8ff0 0x558def569290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.034+0000 7fe9dffff640 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe9c806acc0 0x7fe9c806d180 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0009810 tx=0x7fe9d005a040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:07.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.034+0000 7fe9e643d640 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x558def4c8ff0 0x558def569290 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:07.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.037+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v123'1097 uv1070 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.037+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e123) -- 0x558def569c00 con 0x558def4c8ff0 2024-09-16T01:44:07.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.038+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v123'1098 uv1098 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.038+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6186e0 con 0x558def4c8ff0 2024-09-16T01:44:07.042 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:44:07.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.040+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v123'1099 uv1099 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.040+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def619a90 con 0x558def4c8ff0 2024-09-16T01:44:07.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.041+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v123'1100 uv1100 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.041+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def619e30 con 0x558def4c8ff0 2024-09-16T01:44:07.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.042+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v123'1101 uv1101 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.042+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def618a80 con 0x558def4c8ff0 2024-09-16T01:44:07.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.044+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v123'1102 uv1102 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.044+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def618e20 con 0x558def4c8ff0 2024-09-16T01:44:07.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.045+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v123'1103 uv1076 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.045+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e123) -- 0x558def619210 con 0x558def4c8ff0 2024-09-16T01:44:07.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.047+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v123'1104 uv1104 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.047+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def619210 con 0x558def4c8ff0 2024-09-16T01:44:07.049 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:44:07.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.048+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v123'1105 uv1105 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.048+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6191c0 con 0x558def4c8ff0 2024-09-16T01:44:07.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.050+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v123'1106 uv1106 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.050+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6191c0 con 0x558def4c8ff0 2024-09-16T01:44:07.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.051+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v123'1107 uv1107 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.051+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6191c0 con 0x558def4c8ff0 2024-09-16T01:44:07.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.053+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v123'1108 uv1108 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.053+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6191c0 con 0x558def4c8ff0 2024-09-16T01:44:07.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.054+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v123'1109 uv1083 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.054+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e123) -- 0x558def6191c0 con 0x558def4c8ff0 2024-09-16T01:44:07.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.056+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v123'1110 uv1110 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.056+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6191c0 con 0x558def4c8ff0 2024-09-16T01:44:07.058 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:44:07.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.057+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v123'1111 uv1111 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.057+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6191c0 con 0x558def4c8ff0 2024-09-16T01:44:07.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.059+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v123'1112 uv1112 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.059+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6191c0 con 0x558def4c8ff0 2024-09-16T01:44:07.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.060+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v123'1113 uv1113 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.060+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6191c0 con 0x558def4c8ff0 2024-09-16T01:44:07.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.062+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v123'1114 uv1114 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.062+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6191c0 con 0x558def4c8ff0 2024-09-16T01:44:07.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.063+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v123'1115 uv1115 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.063+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6191c0 con 0x558def4c8ff0 2024-09-16T01:44:07.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.064+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v123'1116 uv1096 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.064+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e123) -- 0x558def6191c0 con 0x558def4c8ff0 2024-09-16T01:44:07.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.066+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v123'1117 uv1117 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.066+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6191c0 con 0x558def4c8ff0 2024-09-16T01:44:07.068 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:44:07.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.067+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v123'1118 uv1118 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.067+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6191c0 con 0x558def4c8ff0 2024-09-16T01:44:07.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.068+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v123'1119 uv1119 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.068+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6191c0 con 0x558def4c8ff0 2024-09-16T01:44:07.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.070+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v123'1120 uv1120 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.070+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6191c0 con 0x558def4c8ff0 2024-09-16T01:44:07.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.071+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v123'1121 uv1121 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.071+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6191c0 con 0x558def4c8ff0 2024-09-16T01:44:07.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.072+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v123'1122 uv1095 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.072+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e123) -- 0x558def6191c0 con 0x558def4c8ff0 2024-09-16T01:44:07.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.074+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v123'1123 uv1123 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.074+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6191c0 con 0x558def4c8ff0 2024-09-16T01:44:07.076 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:44:07.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.075+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v123'1124 uv1124 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.075+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6191c0 con 0x558def4c8ff0 2024-09-16T01:44:07.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.077+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v123'1125 uv1125 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.077+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6191c0 con 0x558def4c8ff0 2024-09-16T01:44:07.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.078+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v123'1126 uv1126 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.078+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6191c0 con 0x558def4c8ff0 2024-09-16T01:44:07.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.080+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v123'1127 uv1127 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.080+0000 7fe9e8146f40 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x558def6195c0 0x558def61c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.080+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def61cc30 con 0x558def6195c0 2024-09-16T01:44:07.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.080+0000 7fe9e5c3c640 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x558def6195c0 0x558def61c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.080+0000 7fe9e5c3c640 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x558def6195c0 0x558def61c6f0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:07.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.082+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v123'800 uv782 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6195c0 2024-09-16T01:44:07.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.082+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e123) -- 0x558def61cfd0 con 0x558def6195c0 2024-09-16T01:44:07.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.083+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v123'801 uv801 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6195c0 2024-09-16T01:44:07.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.083+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def61d370 con 0x558def6195c0 2024-09-16T01:44:07.086 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:44:07.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.085+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v123'802 uv802 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6195c0 2024-09-16T01:44:07.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.085+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def61dbc0 con 0x558def6195c0 2024-09-16T01:44:07.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.086+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v123'803 uv803 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6195c0 2024-09-16T01:44:07.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.086+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def61df60 con 0x558def6195c0 2024-09-16T01:44:07.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.088+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v123'804 uv804 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6195c0 2024-09-16T01:44:07.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.088+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def61e300 con 0x558def6195c0 2024-09-16T01:44:07.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.089+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v123'805 uv787 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6195c0 2024-09-16T01:44:07.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.089+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e123) -- 0x558def61e6a0 con 0x558def6195c0 2024-09-16T01:44:07.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.091+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v123'806 uv806 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6195c0 2024-09-16T01:44:07.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.091+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def61e6a0 con 0x558def6195c0 2024-09-16T01:44:07.093 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:44:07.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.092+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v123'807 uv807 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6195c0 2024-09-16T01:44:07.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.092+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def61e6a0 con 0x558def6195c0 2024-09-16T01:44:07.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.093+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v123'808 uv808 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6195c0 2024-09-16T01:44:07.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.093+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def61e6a0 con 0x558def6195c0 2024-09-16T01:44:07.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.095+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v123'809 uv809 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6195c0 2024-09-16T01:44:07.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.095+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def61e6a0 con 0x558def6195c0 2024-09-16T01:44:07.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.096+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v123'810 uv792 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6195c0 2024-09-16T01:44:07.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.096+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e123) -- 0x558def61e6a0 con 0x558def6195c0 2024-09-16T01:44:07.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.097+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v123'811 uv811 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6195c0 2024-09-16T01:44:07.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.097+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def61e6a0 con 0x558def6195c0 2024-09-16T01:44:07.100 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:44:07.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.099+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v123'812 uv812 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6195c0 2024-09-16T01:44:07.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.099+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def622b20 con 0x558def6195c0 2024-09-16T01:44:07.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.100+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v123'813 uv813 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6195c0 2024-09-16T01:44:07.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.100+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def622b20 con 0x558def6195c0 2024-09-16T01:44:07.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.102+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v123'814 uv814 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6195c0 2024-09-16T01:44:07.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.102+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def61e6a0 con 0x558def6195c0 2024-09-16T01:44:07.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.103+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v123'815 uv798 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6195c0 2024-09-16T01:44:07.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.103+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e123) -- 0x558def61e6a0 con 0x558def6195c0 2024-09-16T01:44:07.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.104+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v123'816 uv816 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6195c0 2024-09-16T01:44:07.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.104+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def61e6a0 con 0x558def6195c0 2024-09-16T01:44:07.106 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:44:07.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.105+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v123'817 uv817 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6195c0 2024-09-16T01:44:07.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.105+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def61f6b0 con 0x558def6195c0 2024-09-16T01:44:07.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.106+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v123'818 uv818 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6195c0 2024-09-16T01:44:07.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.106+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def61f6b0 con 0x558def6195c0 2024-09-16T01:44:07.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.108+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v123'819 uv819 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6195c0 2024-09-16T01:44:07.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.108+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def61e6a0 con 0x558def6195c0 2024-09-16T01:44:07.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.109+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v123'820 uv820 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6195c0 2024-09-16T01:44:07.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.109+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def61e6a0 con 0x558def4c8ff0 2024-09-16T01:44:07.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.110+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v123'213 uv212 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.110+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e123) -- 0x558def61e6a0 con 0x558def4c8ff0 2024-09-16T01:44:07.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.111+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v123'214 uv214 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.111+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def61e6a0 con 0x558def4c8ff0 2024-09-16T01:44:07.113 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:44:07.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.112+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v123'215 uv215 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.112+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def61f6b0 con 0x558def4c8ff0 2024-09-16T01:44:07.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.113+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v123'216 uv216 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.113+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def61f6b0 con 0x558def4c8ff0 2024-09-16T01:44:07.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.114+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v123'217 uv217 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.114+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def61e6a0 con 0x558def4c8ff0 2024-09-16T01:44:07.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.115+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v123'218 uv218 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.115+0000 7fe9e8146f40 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x558def61ebb0 0x558def620fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.115+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6214e0 con 0x558def61ebb0 2024-09-16T01:44:07.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.115+0000 7fe9dffff640 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x558def61ebb0 0x558def620fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.115+0000 7fe9dffff640 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x558def61ebb0 0x558def620fa0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:07.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.116+0000 7fe9dffff640 1 -- 172.21.15.88:0/1868876081 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v123'161 uv160 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fe9d001a340 con 0x558def61ebb0 2024-09-16T01:44:07.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.116+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e123) -- 0x558def621940 con 0x558def61ebb0 2024-09-16T01:44:07.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.117+0000 7fe9dffff640 1 -- 172.21.15.88:0/1868876081 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v123'162 uv162 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe9d001a340 con 0x558def61ebb0 2024-09-16T01:44:07.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.117+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def621e40 con 0x558def61ebb0 2024-09-16T01:44:07.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.118+0000 7fe9dffff640 1 -- 172.21.15.88:0/1868876081 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v123'163 uv163 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe9d001a340 con 0x558def61ebb0 2024-09-16T01:44:07.120 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:44:07.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.118+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def61d7f0 con 0x558def61ebb0 2024-09-16T01:44:07.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.119+0000 7fe9dffff640 1 -- 172.21.15.88:0/1868876081 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v123'164 uv164 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe9d001a340 con 0x558def61ebb0 2024-09-16T01:44:07.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.119+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def622770 con 0x558def4c8ff0 2024-09-16T01:44:07.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.121+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v123'213 uv212 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.121+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e123) -- 0x558def622770 con 0x558def4c8ff0 2024-09-16T01:44:07.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.122+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v123'214 uv214 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.122+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def622770 con 0x558def4c8ff0 2024-09-16T01:44:07.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.123+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v123'215 uv215 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.124 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:44:07.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.123+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def622770 con 0x558def4c8ff0 2024-09-16T01:44:07.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.124+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v123'216 uv216 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.124+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def622770 con 0x558def4c8ff0 2024-09-16T01:44:07.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.125+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v123'217 uv217 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.125+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def622770 con 0x558def4c8ff0 2024-09-16T01:44:07.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.126+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v123'218 uv218 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.126+0000 7fe9e8146f40 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x558def623070 0x558def623470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.126+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def625ba0 con 0x558def623070 2024-09-16T01:44:07.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.126+0000 7fe9e643d640 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x558def623070 0x558def623470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.126+0000 7fe9e643d640 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x558def623070 0x558def623470 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:07.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.128+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v123'393 uv387 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.128+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e123) -- 0x558def625ba0 con 0x558def623070 2024-09-16T01:44:07.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.129+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v123'394 uv394 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.129+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def625ba0 con 0x558def623070 2024-09-16T01:44:07.131 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:44:07.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.130+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v123'395 uv395 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.130+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def626bf0 con 0x558def623070 2024-09-16T01:44:07.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.131+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v123'396 uv396 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.131+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def626bb0 con 0x558def623070 2024-09-16T01:44:07.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.132+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v123'397 uv397 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.132+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def625ba0 con 0x558def623070 2024-09-16T01:44:07.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.133+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v123'398 uv398 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.134+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def625ba0 con 0x558def623070 2024-09-16T01:44:07.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.135+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v123'399 uv392 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.135+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e123) -- 0x558def625ba0 con 0x558def623070 2024-09-16T01:44:07.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.136+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v123'400 uv400 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.136+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def625ba0 con 0x558def623070 2024-09-16T01:44:07.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.137+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v123'401 uv401 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.138 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:44:07.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.137+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def625ba0 con 0x558def623070 2024-09-16T01:44:07.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.138+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v123'402 uv402 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.138+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def625ba0 con 0x558def623070 2024-09-16T01:44:07.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.139+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v123'403 uv403 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.139+0000 7fe9e8146f40 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x558def6264f0 0x558def626930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.139+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def629000 con 0x558def6264f0 2024-09-16T01:44:07.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.140+0000 7fe9e5c3c640 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x558def6264f0 0x558def626930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.140+0000 7fe9e5c3c640 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x558def6264f0 0x558def626930 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:07.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.141+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v123'178 uv177 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6264f0 2024-09-16T01:44:07.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.141+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e123) -- 0x558def629000 con 0x558def6264f0 2024-09-16T01:44:07.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.143+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v123'179 uv179 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6264f0 2024-09-16T01:44:07.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.143+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6293f0 con 0x558def6264f0 2024-09-16T01:44:07.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.144+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v123'180 uv180 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6264f0 2024-09-16T01:44:07.145 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:44:07.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.144+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6293f0 con 0x558def6264f0 2024-09-16T01:44:07.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.145+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v123'181 uv181 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6264f0 2024-09-16T01:44:07.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.145+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6293f0 con 0x558def6264f0 2024-09-16T01:44:07.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.146+0000 7fe9e5c3c640 1 -- 172.21.15.88:0/1868876081 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v123'182 uv182 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe9d80567d0 con 0x558def6264f0 2024-09-16T01:44:07.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.146+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6293f0 con 0x558def4c8ff0 2024-09-16T01:44:07.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.147+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v123'213 uv212 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.147+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e123) -- 0x558def6293f0 con 0x558def4c8ff0 2024-09-16T01:44:07.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.148+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v123'214 uv214 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.148+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6293f0 con 0x558def4c8ff0 2024-09-16T01:44:07.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.149+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v123'215 uv215 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.151 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:44:07.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.149+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def62a400 con 0x558def4c8ff0 2024-09-16T01:44:07.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.150+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v123'216 uv216 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.150+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def62a400 con 0x558def4c8ff0 2024-09-16T01:44:07.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.151+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v123'217 uv217 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.151+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6293f0 con 0x558def4c8ff0 2024-09-16T01:44:07.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.151+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v123'218 uv218 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def4c8ff0 2024-09-16T01:44:07.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.152+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6293f0 con 0x558def61ebb0 2024-09-16T01:44:07.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.153+0000 7fe9dffff640 1 -- 172.21.15.88:0/1868876081 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v123'355 uv349 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fe9d001a340 con 0x558def61ebb0 2024-09-16T01:44:07.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.153+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e123) -- 0x558def629790 con 0x558def61ebb0 2024-09-16T01:44:07.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.154+0000 7fe9dffff640 1 -- 172.21.15.88:0/1868876081 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v123'356 uv356 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe9d001a340 con 0x558def61ebb0 2024-09-16T01:44:07.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.154+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def629b30 con 0x558def61ebb0 2024-09-16T01:44:07.156 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:44:07.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.155+0000 7fe9dffff640 1 -- 172.21.15.88:0/1868876081 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v123'357 uv357 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe9d001a340 con 0x558def61ebb0 2024-09-16T01:44:07.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.155+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6222f0 con 0x558def61ebb0 2024-09-16T01:44:07.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.156+0000 7fe9dffff640 1 -- 172.21.15.88:0/1868876081 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v123'358 uv358 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe9d001a340 con 0x558def61ebb0 2024-09-16T01:44:07.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.156+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6222f0 con 0x558def61ebb0 2024-09-16T01:44:07.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.157+0000 7fe9dffff640 1 -- 172.21.15.88:0/1868876081 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v123'359 uv359 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe9d001a340 con 0x558def61ebb0 2024-09-16T01:44:07.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.157+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6222f0 con 0x558def61ebb0 2024-09-16T01:44:07.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.158+0000 7fe9dffff640 1 -- 172.21.15.88:0/1868876081 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v123'360 uv354 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9d001a340 con 0x558def61ebb0 2024-09-16T01:44:07.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.158+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e123) -- 0x558def6222f0 con 0x558def61ebb0 2024-09-16T01:44:07.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.159+0000 7fe9dffff640 1 -- 172.21.15.88:0/1868876081 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v123'361 uv361 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d001a340 con 0x558def61ebb0 2024-09-16T01:44:07.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.159+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6222f0 con 0x558def61ebb0 2024-09-16T01:44:07.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.160+0000 7fe9dffff640 1 -- 172.21.15.88:0/1868876081 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v123'362 uv362 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d001a340 con 0x558def61ebb0 2024-09-16T01:44:07.161 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:44:07.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.160+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def6222f0 con 0x558def61ebb0 2024-09-16T01:44:07.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.162+0000 7fe9dffff640 1 -- 172.21.15.88:0/1868876081 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v123'363 uv363 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d001a340 con 0x558def61ebb0 2024-09-16T01:44:07.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.162+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def62c940 con 0x558def61ebb0 2024-09-16T01:44:07.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.163+0000 7fe9dffff640 1 -- 172.21.15.88:0/1868876081 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v123'364 uv364 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d001a340 con 0x558def61ebb0 2024-09-16T01:44:07.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.163+0000 7fe9e8146f40 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x558def62a660 0x558def62cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.163+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def62a660 2024-09-16T01:44:07.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.163+0000 7fe9dffff640 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x558def62a660 0x558def62cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.163+0000 7fe9dffff640 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x558def62a660 0x558def62cf80 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:07.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.164+0000 7fe9dffff640 1 -- 172.21.15.88:0/1868876081 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v123'157 uv155 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fe9d001a340 con 0x558def62a660 2024-09-16T01:44:07.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.164+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def62a660 2024-09-16T01:44:07.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.165+0000 7fe9dffff640 1 -- 172.21.15.88:0/1868876081 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v123'158 uv158 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fe9d001a340 con 0x558def62a660 2024-09-16T01:44:07.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.165+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def62a660 2024-09-16T01:44:07.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.166+0000 7fe9dffff640 1 -- 172.21.15.88:0/1868876081 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v123'159 uv159 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fe9d001a340 con 0x558def62a660 2024-09-16T01:44:07.167 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:44:07.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.166+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def62a660 2024-09-16T01:44:07.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.167+0000 7fe9dffff640 1 -- 172.21.15.88:0/1868876081 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v123'160 uv160 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fe9d001a340 con 0x558def62a660 2024-09-16T01:44:07.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.167+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def61ebb0 2024-09-16T01:44:07.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.168+0000 7fe9dffff640 1 -- 172.21.15.88:0/1868876081 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v123'213 uv212 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9d001a340 con 0x558def61ebb0 2024-09-16T01:44:07.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.168+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def61ebb0 2024-09-16T01:44:07.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.169+0000 7fe9dffff640 1 -- 172.21.15.88:0/1868876081 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v123'214 uv214 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d001a340 con 0x558def61ebb0 2024-09-16T01:44:07.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.169+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def61ebb0 2024-09-16T01:44:07.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.171+0000 7fe9dffff640 1 -- 172.21.15.88:0/1868876081 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v123'215 uv215 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d001a340 con 0x558def61ebb0 2024-09-16T01:44:07.172 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:44:07.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.171+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def61ebb0 2024-09-16T01:44:07.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.172+0000 7fe9dffff640 1 -- 172.21.15.88:0/1868876081 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v123'216 uv216 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d001a340 con 0x558def61ebb0 2024-09-16T01:44:07.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.172+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def61ebb0 2024-09-16T01:44:07.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.173+0000 7fe9dffff640 1 -- 172.21.15.88:0/1868876081 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v123'217 uv217 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d001a340 con 0x558def61ebb0 2024-09-16T01:44:07.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.173+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def61ebb0 2024-09-16T01:44:07.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.174+0000 7fe9dffff640 1 -- 172.21.15.88:0/1868876081 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v123'218 uv218 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d001a340 con 0x558def61ebb0 2024-09-16T01:44:07.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.174+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def623070 2024-09-16T01:44:07.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.175+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v123'181 uv180 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.175+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def623070 2024-09-16T01:44:07.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.176+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v123'182 uv182 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.176+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def623070 2024-09-16T01:44:07.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.177+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v123'183 uv183 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.179 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:44:07.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.177+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def623070 2024-09-16T01:44:07.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.179+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v123'184 uv184 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.179+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def623070 2024-09-16T01:44:07.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.180+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v123'185 uv185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.180+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def623070 2024-09-16T01:44:07.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.181+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v123'442 uv433 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.181+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def623070 2024-09-16T01:44:07.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.182+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v123'443 uv443 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.182+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def623070 2024-09-16T01:44:07.184 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:44:07.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.183+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v123'444 uv444 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.183+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def623070 2024-09-16T01:44:07.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.185+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v123'445 uv445 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.185+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def623070 2024-09-16T01:44:07.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.186+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v123'446 uv446 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.186+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def623070 2024-09-16T01:44:07.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.187+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v123'447 uv440 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.187+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def623070 2024-09-16T01:44:07.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.188+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v123'448 uv448 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.188+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def623070 2024-09-16T01:44:07.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.189+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v123'449 uv449 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.190 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:44:07.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.189+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def623070 2024-09-16T01:44:07.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.190+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v123'450 uv450 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.190+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def623070 2024-09-16T01:44:07.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.190+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v123'451 uv451 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.190+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def62ca50 con 0x558def623070 2024-09-16T01:44:07.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.191+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v123'452 uv452 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.191+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x558def62d4c0 con 0x558def623070 2024-09-16T01:44:07.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.192+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v123'453 uv453 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe9e0012330 con 0x558def623070 2024-09-16T01:44:07.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.192+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x558def4c8ff0 msgr2=0x558def569290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:07.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.192+0000 7fe9e8146f40 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x558def4c8ff0 0x558def569290 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.192+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x558def61ebb0 msgr2=0x558def620fa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:07.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.192+0000 7fe9e8146f40 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x558def61ebb0 0x558def620fa0 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.192+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x558def62a660 msgr2=0x558def62cf80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:07.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.192+0000 7fe9e8146f40 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x558def62a660 0x558def62cf80 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.192+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x558def6195c0 msgr2=0x558def61c6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:07.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.192+0000 7fe9e8146f40 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x558def6195c0 0x558def61c6f0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.192+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x558def623070 msgr2=0x558def623470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:07.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.192+0000 7fe9e643d640 1 -- 172.21.15.88:0/1868876081 reap_dead start 2024-09-16T01:44:07.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.192+0000 7fe9e8146f40 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x558def623070 0x558def623470 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.192+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x558def6264f0 msgr2=0x558def626930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:07.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.192+0000 7fe9e8146f40 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x558def6264f0 0x558def626930 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.193+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe9c806acc0 msgr2=0x7fe9c806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:07.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.193+0000 7fe9e8146f40 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe9c806acc0 0x7fe9c806d180 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0009810 tx=0x7fe9d005a040 comp rx=0 tx=0).stop 2024-09-16T01:44:07.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.193+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558def555140 msgr2=0x558def6055c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:07.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.193+0000 7fe9e8146f40 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558def555140 0x558def6055c0 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8000c00 tx=0x7fe9d8002fb0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.193+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 shutdown_connections 2024-09-16T01:44:07.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.193+0000 7fe9e8146f40 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe9c806acc0 0x7fe9c806d180 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.193+0000 7fe9e8146f40 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x558def6264f0 0x558def626930 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.193+0000 7fe9e8146f40 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x558def623070 0x558def623470 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.193+0000 7fe9e8146f40 1 --2- 172.21.15.88:0/1868876081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x558def555140 0x558def6055c0 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.193+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 >> 172.21.15.88:0/1868876081 conn(0x558def4e8750 msgr2=0x558def49e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:07.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.193+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 shutdown_connections 2024-09-16T01:44:07.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.193+0000 7fe9e8146f40 1 -- 172.21.15.88:0/1868876081 wait complete. 2024-09-16T01:44:07.197 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapheader mds0_openfiles.0 deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-09-16T01:44:07.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.279+0000 7f79fe4eaf40 1 Processor -- start 2024-09-16T01:44:07.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.280+0000 7f79fe4eaf40 1 -- start start 2024-09-16T01:44:07.283 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.280+0000 7f79fe4eaf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f025911df0 0x55f025a74980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.280+0000 7f79fe4eaf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f025912710 0x55f0259e6d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.280+0000 7f79fe4eaf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f0259cee00 0x55f0259e77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.280+0000 7f79fe4eaf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f0259e7640 con 0x55f0259cee00 2024-09-16T01:44:07.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.280+0000 7f79fe4eaf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f025862510 con 0x55f025912710 2024-09-16T01:44:07.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.280+0000 7f79fe4eaf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55f02593aa00 con 0x55f025911df0 2024-09-16T01:44:07.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.280+0000 7f79fc7e1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f0259cee00 0x55f0259e77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.280+0000 7f79fc7e1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f0259cee00 0x55f0259e77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41328/0 (socket says 172.21.15.88:41328) 2024-09-16T01:44:07.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.280+0000 7f79fc7e1640 1 -- 172.21.15.88:0/1987178154 learned_addr learned my addr 172.21.15.88:0/1987178154 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:07.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.280+0000 7f79f37df640 1 --2- 172.21.15.88:0/1987178154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f025912710 0x55f0259e6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.281+0000 7f79fbfe0640 1 --2- 172.21.15.88:0/1987178154 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f025911df0 0x55f025a74980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.281+0000 7f79fc7e1640 1 -- 172.21.15.88:0/1987178154 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f025911df0 msgr2=0x55f025a74980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:07.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.281+0000 7f79fc7e1640 1 --2- 172.21.15.88:0/1987178154 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f025911df0 0x55f025a74980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.281+0000 7f79fc7e1640 1 -- 172.21.15.88:0/1987178154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f025912710 msgr2=0x55f0259e6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:07.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.281+0000 7f79fc7e1640 1 --2- 172.21.15.88:0/1987178154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f025912710 0x55f0259e6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.281+0000 7f79fc7e1640 1 -- 172.21.15.88:0/1987178154 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f02593a790 con 0x55f0259cee00 2024-09-16T01:44:07.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.281+0000 7f79fc7e1640 1 --2- 172.21.15.88:0/1987178154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f0259cee00 0x55f0259e77c0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f79f40259d0 tx=0x7f79f4051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88778d489fdd47fb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:07.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.281+0000 7f79fafde640 1 -- 172.21.15.88:0/1987178154 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f4067070 con 0x55f0259cee00 2024-09-16T01:44:07.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.281+0000 7f79fafde640 1 -- 172.21.15.88:0/1987178154 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79f4050b60 con 0x55f0259cee00 2024-09-16T01:44:07.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.281+0000 7f79fafde640 1 -- 172.21.15.88:0/1987178154 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f4050610 con 0x55f0259cee00 2024-09-16T01:44:07.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.281+0000 7f79fe4eaf40 1 -- 172.21.15.88:0/1987178154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f0259cee00 msgr2=0x55f0259e77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:07.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.281+0000 7f79fe4eaf40 1 --2- 172.21.15.88:0/1987178154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f0259cee00 0x55f0259e77c0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f79f40259d0 tx=0x7f79f4051000 comp rx=0 tx=0).stop 2024-09-16T01:44:07.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.281+0000 7f79fe4eaf40 1 -- 172.21.15.88:0/1987178154 shutdown_connections 2024-09-16T01:44:07.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.281+0000 7f79fe4eaf40 1 --2- 172.21.15.88:0/1987178154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f0259cee00 0x55f0259e77c0 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.281+0000 7f79fe4eaf40 1 --2- 172.21.15.88:0/1987178154 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f025912710 0x55f0259e6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.281+0000 7f79fe4eaf40 1 --2- 172.21.15.88:0/1987178154 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f025911df0 0x55f025a74980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.281+0000 7f79fe4eaf40 1 -- 172.21.15.88:0/1987178154 >> 172.21.15.88:0/1987178154 conn(0x55f02593b960 msgr2=0x55f02593f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:07.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.282+0000 7f79fe4eaf40 1 -- 172.21.15.88:0/1987178154 shutdown_connections 2024-09-16T01:44:07.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.282+0000 7f79fe4eaf40 1 -- 172.21.15.88:0/1987178154 wait complete. 2024-09-16T01:44:07.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.282+0000 7f79fe4eaf40 1 Processor -- start 2024-09-16T01:44:07.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.282+0000 7f79fe4eaf40 1 -- start start 2024-09-16T01:44:07.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.283+0000 7f79fe4eaf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f025911df0 0x55f0259c4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.283+0000 7f79fe4eaf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f025912710 0x55f0259c3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.283+0000 7f79fe4eaf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f0259cee00 0x55f02593f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.283+0000 7f79fe4eaf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55f025956f00 con 0x55f025911df0 2024-09-16T01:44:07.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.283+0000 7f79fe4eaf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55f0259123b0 con 0x55f0259cee00 2024-09-16T01:44:07.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.283+0000 7f79fe4eaf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55f0259e8bb0 con 0x55f025912710 2024-09-16T01:44:07.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.283+0000 7f79fbfe0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f025911df0 0x55f0259c4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.283+0000 7f79fbfe0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f025911df0 0x55f0259c4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41336/0 (socket says 172.21.15.88:41336) 2024-09-16T01:44:07.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.283+0000 7f79fbfe0640 1 -- 172.21.15.88:0/3867383361 learned_addr learned my addr 172.21.15.88:0/3867383361 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:07.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.283+0000 7f79f37df640 1 --2- 172.21.15.88:0/3867383361 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f025912710 0x55f0259c3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.283+0000 7f79fc7e1640 1 --2- 172.21.15.88:0/3867383361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f0259cee00 0x55f02593f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.283+0000 7f79fbfe0640 1 -- 172.21.15.88:0/3867383361 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f025912710 msgr2=0x55f0259c3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:07.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.283+0000 7f79fbfe0640 1 --2- 172.21.15.88:0/3867383361 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55f025912710 0x55f0259c3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.283+0000 7f79fbfe0640 1 -- 172.21.15.88:0/3867383361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f0259cee00 msgr2=0x55f02593f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:07.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.283+0000 7f79fbfe0640 1 --2- 172.21.15.88:0/3867383361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f0259cee00 0x55f02593f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.283+0000 7f79fbfe0640 1 -- 172.21.15.88:0/3867383361 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0259c4840 con 0x55f025911df0 2024-09-16T01:44:07.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.283+0000 7f79fbfe0640 1 --2- 172.21.15.88:0/3867383361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f025911df0 0x55f0259c4260 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f79ec00dd00 tx=0x7f79ec00b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:07.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.283+0000 7f79f9fdc640 1 -- 172.21.15.88:0/3867383361 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ec00f650 con 0x55f025911df0 2024-09-16T01:44:07.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.283+0000 7f79f9fdc640 1 -- 172.21.15.88:0/3867383361 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79ec00f810 con 0x55f025911df0 2024-09-16T01:44:07.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.283+0000 7f79f9fdc640 1 -- 172.21.15.88:0/3867383361 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ec00fc00 con 0x55f025911df0 2024-09-16T01:44:07.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.283+0000 7f79fe4eaf40 1 -- 172.21.15.88:0/3867383361 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f0259c4b20 con 0x55f025911df0 2024-09-16T01:44:07.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.283+0000 7f79fe4eaf40 1 -- 172.21.15.88:0/3867383361 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0259c50e0 con 0x55f025911df0 2024-09-16T01:44:07.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.285+0000 7f79f9fdc640 1 -- 172.21.15.88:0/3867383361 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f79ec0203f0 con 0x55f025911df0 2024-09-16T01:44:07.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.285+0000 7f79f9fdc640 1 --2- 172.21.15.88:0/3867383361 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f79dc06ac90 0x7f79dc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.285+0000 7f79f9fdc640 1 -- 172.21.15.88:0/3867383361 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8834+0+0 (secure 0 0 0) 0x7f79ec08b520 con 0x55f025911df0 2024-09-16T01:44:07.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.285+0000 7f79f37df640 1 --2- 172.21.15.88:0/3867383361 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f79dc06ac90 0x7f79dc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.285+0000 7f79fe4eaf40 1 --2- 172.21.15.88:0/3867383361 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55f025a7c490 0x55f025a7c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.285+0000 7f79fe4eaf40 1 -- 172.21.15.88:0/3867383361 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:1 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123) -- 0x55f025a7cdf0 con 0x55f025a7c490 2024-09-16T01:44:07.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.286+0000 7f79fc7e1640 1 --2- 172.21.15.88:0/3867383361 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55f025a7c490 0x55f025a7c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.286+0000 7f79f37df640 1 --2- 172.21.15.88:0/3867383361 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f79dc06ac90 0x7f79dc06d150 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x55f025959090 tx=0x7f79e400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:07.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.286+0000 7f79fc7e1640 1 --2- 172.21.15.88:0/3867383361 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55f025a7c490 0x55f025a7c8b0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:07.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.287+0000 7f79fc7e1640 1 -- 172.21.15.88:0/3867383361 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(1 mds0_openfiles.0 [omap-set-header] v123'165 uv165 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f79f4062430 con 0x55f025a7c490 2024-09-16T01:44:07.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.287+0000 7f79fe4eaf40 1 -- 172.21.15.88:0/3867383361 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55f025a7c490 msgr2=0x55f025a7c8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:07.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.287+0000 7f79fe4eaf40 1 --2- 172.21.15.88:0/3867383361 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55f025a7c490 0x55f025a7c8b0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.287+0000 7f79fe4eaf40 1 -- 172.21.15.88:0/3867383361 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f79dc06ac90 msgr2=0x7f79dc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:07.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.287+0000 7f79fe4eaf40 1 --2- 172.21.15.88:0/3867383361 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f79dc06ac90 0x7f79dc06d150 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x55f025959090 tx=0x7f79e400b040 comp rx=0 tx=0).stop 2024-09-16T01:44:07.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.287+0000 7f79fe4eaf40 1 -- 172.21.15.88:0/3867383361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f025911df0 msgr2=0x55f0259c4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:07.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.287+0000 7f79fe4eaf40 1 --2- 172.21.15.88:0/3867383361 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f025911df0 0x55f0259c4260 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f79ec00dd00 tx=0x7f79ec00b6c0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.287+0000 7f79fc7e1640 1 -- 172.21.15.88:0/3867383361 reap_dead start 2024-09-16T01:44:07.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.287+0000 7f79fe4eaf40 1 -- 172.21.15.88:0/3867383361 shutdown_connections 2024-09-16T01:44:07.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.287+0000 7f79fe4eaf40 1 -- 172.21.15.88:0/3867383361 >> 172.21.15.88:0/3867383361 conn(0x55f02593b960 msgr2=0x55f0259cad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:07.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.287+0000 7f79fe4eaf40 1 -- 172.21.15.88:0/3867383361 shutdown_connections 2024-09-16T01:44:07.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.287+0000 7f79fe4eaf40 1 -- 172.21.15.88:0/3867383361 wait complete. 2024-09-16T01:44:07.293 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:44:07.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.434+0000 7ff82acd0640 1 Processor -- start 2024-09-16T01:44:07.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.434+0000 7ff82acd0640 1 -- start start 2024-09-16T01:44:07.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.435+0000 7ff82acd0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff82415e7a0 0x7ff82406caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.435+0000 7ff82acd0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff8240b68f0 0x7ff8240d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.435+0000 7ff82acd0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8240cf860 con 0x7ff8240cf1f0 2024-09-16T01:44:07.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.435+0000 7ff82acd0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff8240d2ca0 con 0x7ff82415e7a0 2024-09-16T01:44:07.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.435+0000 7ff82acd0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff8240cfb40 con 0x7ff8240b68f0 2024-09-16T01:44:07.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.435+0000 7ff828a6c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff82415e7a0 0x7ff82406caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.435+0000 7ff828a6c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff82415e7a0 0x7ff82406caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41340/0 (socket says 172.21.15.88:41340) 2024-09-16T01:44:07.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.435+0000 7ff828a6c640 1 -- 172.21.15.88:0/1984887191 learned_addr learned my addr 172.21.15.88:0/1984887191 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:07.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.436+0000 7ff82926d640 1 --2- 172.21.15.88:0/1984887191 >> v2:172.21.15.154:3300/0 conn(0x7ff8240b68f0 0x7ff8240d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.436+0000 7ff828a6c640 1 -- 172.21.15.88:0/1984887191 >> v2:172.21.15.154:3300/0 conn(0x7ff8240b68f0 msgr2=0x7ff8240d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:07.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.436+0000 7ff828a6c640 1 --2- 172.21.15.88:0/1984887191 >> v2:172.21.15.154:3300/0 conn(0x7ff8240b68f0 0x7ff8240d3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.436+0000 7ff828a6c640 1 -- 172.21.15.88:0/1984887191 >> v1:172.21.15.154:6789/0 conn(0x7ff8240cf1f0 legacy=0x7ff8240b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:07.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.436+0000 7ff828a6c640 1 -- 172.21.15.88:0/1984887191 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8240cfe20 con 0x7ff82415e7a0 2024-09-16T01:44:07.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.436+0000 7ff828a6c640 1 --2- 172.21.15.88:0/1984887191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff82415e7a0 0x7ff82406caa0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7ff82408d930 tx=0x7ff81802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab05093f0332a5d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:07.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.437+0000 7ff8237fe640 1 -- 172.21.15.88:0/1984887191 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff818037070 con 0x7ff82415e7a0 2024-09-16T01:44:07.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.437+0000 7ff8237fe640 1 -- 172.21.15.88:0/1984887191 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff818032440 con 0x7ff82415e7a0 2024-09-16T01:44:07.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.437+0000 7ff8237fe640 1 -- 172.21.15.88:0/1984887191 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff818032810 con 0x7ff82415e7a0 2024-09-16T01:44:07.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.437+0000 7ff82acd0640 1 -- 172.21.15.88:0/1984887191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff82415e7a0 msgr2=0x7ff82406caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:07.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.437+0000 7ff82acd0640 1 --2- 172.21.15.88:0/1984887191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff82415e7a0 0x7ff82406caa0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7ff82408d930 tx=0x7ff81802a1e0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.437+0000 7ff82acd0640 1 -- 172.21.15.88:0/1984887191 shutdown_connections 2024-09-16T01:44:07.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.437+0000 7ff82acd0640 1 --2- 172.21.15.88:0/1984887191 >> v2:172.21.15.154:3300/0 conn(0x7ff8240b68f0 0x7ff8240d3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.437+0000 7ff82acd0640 1 --2- 172.21.15.88:0/1984887191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff82415e7a0 0x7ff82406caa0 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.437+0000 7ff82acd0640 1 -- 172.21.15.88:0/1984887191 >> 172.21.15.88:0/1984887191 conn(0x7ff8240d11e0 msgr2=0x7ff8240d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:07.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.437+0000 7ff82acd0640 1 -- 172.21.15.88:0/1984887191 shutdown_connections 2024-09-16T01:44:07.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.437+0000 7ff82acd0640 1 -- 172.21.15.88:0/1984887191 wait complete. 2024-09-16T01:44:07.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.438+0000 7ff82acd0640 1 Processor -- start 2024-09-16T01:44:07.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.438+0000 7ff82acd0640 1 -- start start 2024-09-16T01:44:07.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.439+0000 7ff82acd0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8240b68f0 0x7ff8240cae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.439+0000 7ff82acd0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8240cf1f0 0x7ff8241f16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.439+0000 7ff82acd0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff82415e7a0 0x7ff824161850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.439+0000 7ff82acd0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff82406cfe0 con 0x7ff8240cf1f0 2024-09-16T01:44:07.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.439+0000 7ff82acd0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff8240b7a00 con 0x7ff82415e7a0 2024-09-16T01:44:07.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.439+0000 7ff82acd0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff82415eb80 con 0x7ff8240b68f0 2024-09-16T01:44:07.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.439+0000 7ff823fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8240cf1f0 0x7ff8241f16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.439+0000 7ff823fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8240cf1f0 0x7ff8241f16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41342/0 (socket says 172.21.15.88:41342) 2024-09-16T01:44:07.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.439+0000 7ff823fff640 1 -- 172.21.15.88:0/4171954119 learned_addr learned my addr 172.21.15.88:0/4171954119 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:07.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.439+0000 7ff82926d640 1 --2- 172.21.15.88:0/4171954119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff82415e7a0 0x7ff824161850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.439+0000 7ff828a6c640 1 --2- 172.21.15.88:0/4171954119 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8240b68f0 0x7ff8240cae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.440+0000 7ff823fff640 1 -- 172.21.15.88:0/4171954119 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8240b68f0 msgr2=0x7ff8240cae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:07.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.440+0000 7ff823fff640 1 --2- 172.21.15.88:0/4171954119 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8240b68f0 0x7ff8240cae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.440+0000 7ff823fff640 1 -- 172.21.15.88:0/4171954119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff82415e7a0 msgr2=0x7ff824161850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:07.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.440+0000 7ff823fff640 1 --2- 172.21.15.88:0/4171954119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff82415e7a0 0x7ff824161850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.440+0000 7ff823fff640 1 -- 172.21.15.88:0/4171954119 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8240c9dc0 con 0x7ff8240cf1f0 2024-09-16T01:44:07.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.440+0000 7ff823fff640 1 --2- 172.21.15.88:0/4171954119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8240cf1f0 0x7ff8241f16f0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7ff81000c430 tx=0x7ff81000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:07.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.440+0000 7ff8217fa640 1 -- 172.21.15.88:0/4171954119 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8100052d0 con 0x7ff8240cf1f0 2024-09-16T01:44:07.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.440+0000 7ff8217fa640 1 -- 172.21.15.88:0/4171954119 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff810005490 con 0x7ff8240cf1f0 2024-09-16T01:44:07.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.440+0000 7ff82acd0640 1 -- 172.21.15.88:0/4171954119 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8240ca0a0 con 0x7ff8240cf1f0 2024-09-16T01:44:07.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.440+0000 7ff8217fa640 1 -- 172.21.15.88:0/4171954119 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff810007260 con 0x7ff8240cf1f0 2024-09-16T01:44:07.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.440+0000 7ff82acd0640 1 -- 172.21.15.88:0/4171954119 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8240ca660 con 0x7ff8240cf1f0 2024-09-16T01:44:07.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.441+0000 7ff82acd0640 1 -- 172.21.15.88:0/4171954119 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff82406caa0 con 0x7ff8240cf1f0 2024-09-16T01:44:07.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.441+0000 7ff8217fa640 1 -- 172.21.15.88:0/4171954119 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff810007400 con 0x7ff8240cf1f0 2024-09-16T01:44:07.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.442+0000 7ff8217fa640 1 --2- 172.21.15.88:0/4171954119 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff80806c2b0 0x7ff80806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:07.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.442+0000 7ff8217fa640 1 -- 172.21.15.88:0/4171954119 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8834+0+0 (secure 0 0 0) 0x7ff81008b4c0 con 0x7ff8240cf1f0 2024-09-16T01:44:07.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.442+0000 7ff828a6c640 1 --2- 172.21.15.88:0/4171954119 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff80806c2b0 0x7ff80806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:07.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.442+0000 7ff828a6c640 1 --2- 172.21.15.88:0/4171954119 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff80806c2b0 0x7ff80806e770 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7ff818034000 tx=0x7ff81802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:07.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.446+0000 7ff8217fa640 1 -- 172.21.15.88:0/4171954119 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff8100589e0 con 0x7ff8240cf1f0 2024-09-16T01:44:07.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.647+0000 7ff82acd0640 1 -- 172.21.15.88:0/4171954119 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7ff8240d2a80 con 0x7ff8240cf1f0 2024-09-16T01:44:07.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.980+0000 7ff8217fa640 1 -- 172.21.15.88:0/4171954119 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v293) ==== 162+0+0 (secure 0 0 0) 0x7ff81005cce0 con 0x7ff8240cf1f0 2024-09-16T01:44:07.981 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:44:07.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.982+0000 7ff82acd0640 1 -- 172.21.15.88:0/4171954119 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff80806c2b0 msgr2=0x7ff80806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:07.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.982+0000 7ff82acd0640 1 --2- 172.21.15.88:0/4171954119 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff80806c2b0 0x7ff80806e770 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7ff818034000 tx=0x7ff81802e000 comp rx=0 tx=0).stop 2024-09-16T01:44:07.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.982+0000 7ff82acd0640 1 -- 172.21.15.88:0/4171954119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8240cf1f0 msgr2=0x7ff8241f16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:07.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.982+0000 7ff82acd0640 1 --2- 172.21.15.88:0/4171954119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8240cf1f0 0x7ff8241f16f0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7ff81000c430 tx=0x7ff81000c8f0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.982+0000 7ff82acd0640 1 -- 172.21.15.88:0/4171954119 shutdown_connections 2024-09-16T01:44:07.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.982+0000 7ff82acd0640 1 --2- 172.21.15.88:0/4171954119 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff80806c2b0 0x7ff80806e770 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.982+0000 7ff82acd0640 1 --2- 172.21.15.88:0/4171954119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff82415e7a0 0x7ff824161850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.982+0000 7ff82acd0640 1 --2- 172.21.15.88:0/4171954119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8240cf1f0 0x7ff8241f16f0 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.982+0000 7ff82acd0640 1 --2- 172.21.15.88:0/4171954119 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff8240b68f0 0x7ff8240cae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:07.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.982+0000 7ff82acd0640 1 -- 172.21.15.88:0/4171954119 >> 172.21.15.88:0/4171954119 conn(0x7ff8240d11e0 msgr2=0x7ff8240b90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:07.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.982+0000 7ff82acd0640 1 -- 172.21.15.88:0/4171954119 shutdown_connections 2024-09-16T01:44:07.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:07.982+0000 7ff82acd0640 1 -- 172.21.15.88:0/4171954119 wait complete. 2024-09-16T01:44:07.992 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:44:07.991+0000 7f4be32db640 -1 mds.0.openfiles _load_finish: corrupted header/values: End of buffer [buffer:2] 2024-09-16T01:44:08.399 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:44:08.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.496+0000 7fb9d1dc4640 1 Processor -- start 2024-09-16T01:44:08.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.496+0000 7fb9d1dc4640 1 -- start start 2024-09-16T01:44:08.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.497+0000 7fb9d1dc4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb9cc0d3310 0x7fb9cc0d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:08.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.497+0000 7fb9d1dc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9cc0d1650 0x7fb9cc0b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:08.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.497+0000 7fb9d1dc4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9cc15ea40 con 0x7fb9cc15e660 2024-09-16T01:44:08.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.497+0000 7fb9d1dc4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb9cc0d1a90 con 0x7fb9cc0d3310 2024-09-16T01:44:08.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.497+0000 7fb9d1dc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb9cc0d3870 con 0x7fb9cc0d1650 2024-09-16T01:44:08.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.497+0000 7fb9cb7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb9cc15e660 0x7fb9cc0cf730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44668/0 (socket says 172.21.15.88:44668) 2024-09-16T01:44:08.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.497+0000 7fb9cb7fe640 1 -- 172.21.15.88:0/2077557547 learned_addr learned my addr 172.21.15.88:0/2077557547 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:08.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.497+0000 7fb9ca7fc640 1 -- 172.21.15.88:0/2077557547 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1314403541 0 0) 0x7fb9cc15ea40 con 0x7fb9cc15e660 2024-09-16T01:44:08.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.497+0000 7fb9cbfff640 1 --2- 172.21.15.88:0/2077557547 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9cc0d1650 0x7fb9cc0b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:08.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.497+0000 7fb9ca7fc640 1 -- 172.21.15.88:0/2077557547 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9b4003520 con 0x7fb9cc15e660 2024-09-16T01:44:08.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.497+0000 7fb9caffd640 1 --2- 172.21.15.88:0/2077557547 >> v2:172.21.15.154:3300/0 conn(0x7fb9cc0d3310 0x7fb9cc0d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:08.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.497+0000 7fb9ca7fc640 1 -- 172.21.15.88:0/2077557547 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 334428730 0 0) 0x7fb9b4003520 con 0x7fb9cc15e660 2024-09-16T01:44:08.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.497+0000 7fb9ca7fc640 1 -- 172.21.15.88:0/2077557547 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9cc0d1650 msgr2=0x7fb9cc0b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:08.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.497+0000 7fb9ca7fc640 1 --2- 172.21.15.88:0/2077557547 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9cc0d1650 0x7fb9cc0b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:08.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.497+0000 7fb9ca7fc640 1 -- 172.21.15.88:0/2077557547 >> v2:172.21.15.154:3300/0 conn(0x7fb9cc0d3310 msgr2=0x7fb9cc0d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:08.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.497+0000 7fb9ca7fc640 1 --2- 172.21.15.88:0/2077557547 >> v2:172.21.15.154:3300/0 conn(0x7fb9cc0d3310 0x7fb9cc0d1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:08.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.497+0000 7fb9ca7fc640 1 -- 172.21.15.88:0/2077557547 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9cc0d3b50 con 0x7fb9cc15e660 2024-09-16T01:44:08.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.497+0000 7fb9ca7fc640 1 -- 172.21.15.88:0/2077557547 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fb9bc001670 con 0x7fb9cc15e660 2024-09-16T01:44:08.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.497+0000 7fb9ca7fc640 1 -- 172.21.15.88:0/2077557547 >> v1:172.21.15.88:6789/0 conn(0x7fb9cc15e660 legacy=0x7fb9cc0cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:08.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.497+0000 7fb9ca7fc640 1 --2- 172.21.15.88:0/2077557547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9b40039b0 0x7fb9b4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:08.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.498+0000 7fb9ca7fc640 1 --2- 172.21.15.88:0/2077557547 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9cc0d1650 0x7fb9b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:08.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.498+0000 7fb9ca7fc640 1 --2- 172.21.15.88:0/2077557547 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9b4004d40 0x7fb9b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:08.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.498+0000 7fb9ca7fc640 1 -- 172.21.15.88:0/2077557547 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc0d1a90 con 0x7fb9b40039b0 2024-09-16T01:44:08.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.498+0000 7fb9ca7fc640 1 -- 172.21.15.88:0/2077557547 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc0d3870 con 0x7fb9b4004d40 2024-09-16T01:44:08.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.498+0000 7fb9ca7fc640 1 -- 172.21.15.88:0/2077557547 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb9b400b9c0 con 0x7fb9cc0d1650 2024-09-16T01:44:08.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.498+0000 7fb9cb7fe640 1 --2- 172.21.15.88:0/2077557547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9b40039b0 0x7fb9b4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:08.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.498+0000 7fb9caffd640 1 --2- 172.21.15.88:0/2077557547 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9cc0d1650 0x7fb9b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:08.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.498+0000 7fb9cbfff640 1 --2- 172.21.15.88:0/2077557547 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9b4004d40 0x7fb9b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:08.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.498+0000 7fb9cb7fe640 1 -- 172.21.15.88:0/2077557547 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9cc0d1650 msgr2=0x7fb9b4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:08.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.498+0000 7fb9cb7fe640 1 --2- 172.21.15.88:0/2077557547 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9cc0d1650 0x7fb9b4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:08.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.498+0000 7fb9cb7fe640 1 -- 172.21.15.88:0/2077557547 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9b4004d40 msgr2=0x7fb9b40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:08.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.498+0000 7fb9cb7fe640 1 --2- 172.21.15.88:0/2077557547 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9b4004d40 0x7fb9b40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:08.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.498+0000 7fb9cb7fe640 1 -- 172.21.15.88:0/2077557547 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9b4003520 con 0x7fb9b40039b0 2024-09-16T01:44:08.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.498+0000 7fb9cbfff640 1 --2- 172.21.15.88:0/2077557547 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9b4004d40 0x7fb9b40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:08.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.499+0000 7fb9cb7fe640 1 --2- 172.21.15.88:0/2077557547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9b40039b0 0x7fb9b4003dd0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc00c240 tx=0x7fb9b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce426fce1e6e1f9b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:08.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.499+0000 7fb9ca7fc640 1 -- 172.21.15.88:0/2077557547 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc02e070 con 0x7fb9b40039b0 2024-09-16T01:44:08.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.499+0000 7fb9ca7fc640 1 -- 172.21.15.88:0/2077557547 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9bc005af0 con 0x7fb9b40039b0 2024-09-16T01:44:08.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.499+0000 7fb9ca7fc640 1 -- 172.21.15.88:0/2077557547 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc007d30 con 0x7fb9b40039b0 2024-09-16T01:44:08.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.499+0000 7fb9d1dc4640 1 -- 172.21.15.88:0/2077557547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9b40039b0 msgr2=0x7fb9b4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:08.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.499+0000 7fb9d1dc4640 1 --2- 172.21.15.88:0/2077557547 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9b40039b0 0x7fb9b4003dd0 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc00c240 tx=0x7fb9b4005830 comp rx=0 tx=0).stop 2024-09-16T01:44:08.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.499+0000 7fb9cbfff640 1 -- 172.21.15.88:0/2077557547 reap_dead start 2024-09-16T01:44:08.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.499+0000 7fb9d1dc4640 1 -- 172.21.15.88:0/2077557547 shutdown_connections 2024-09-16T01:44:08.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.499+0000 7fb9d1dc4640 1 -- 172.21.15.88:0/2077557547 >> 172.21.15.88:0/2077557547 conn(0x7fb9cc0b6410 msgr2=0x7fb9cc0b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:08.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.499+0000 7fb9d1dc4640 1 -- 172.21.15.88:0/2077557547 shutdown_connections 2024-09-16T01:44:08.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.499+0000 7fb9d1dc4640 1 -- 172.21.15.88:0/2077557547 wait complete. 2024-09-16T01:44:08.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.500+0000 7fb9d1dc4640 1 Processor -- start 2024-09-16T01:44:08.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.501+0000 7fb9d1dc4640 1 -- start start 2024-09-16T01:44:08.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.501+0000 7fb9d1dc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9b40039b0 0x7fb9cc0cd630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:08.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.501+0000 7fb9d1dc4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9cc0cf5f0 0x7fb9cc0b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:08.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.501+0000 7fb9d1dc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9cc14f790 0x7fb9cc14fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:08.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.501+0000 7fb9d1dc4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc0d43f0 con 0x7fb9cc0cf5f0 2024-09-16T01:44:08.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.501+0000 7fb9d1dc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc0d4110 con 0x7fb9b40039b0 2024-09-16T01:44:08.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.501+0000 7fb9d1dc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb9cc0b7b70 con 0x7fb9cc14f790 2024-09-16T01:44:08.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.501+0000 7fb9caffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9cc0cf5f0 0x7fb9cc0b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:08.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.501+0000 7fb9caffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9cc0cf5f0 0x7fb9cc0b8b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41368/0 (socket says 172.21.15.88:41368) 2024-09-16T01:44:08.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.501+0000 7fb9caffd640 1 -- 172.21.15.88:0/886571279 learned_addr learned my addr 172.21.15.88:0/886571279 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:08.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.502+0000 7fb9cb7fe640 1 --2- 172.21.15.88:0/886571279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9b40039b0 0x7fb9cc0cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:08.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.502+0000 7fb9cbfff640 1 --2- 172.21.15.88:0/886571279 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9cc14f790 0x7fb9cc14fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:08.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.502+0000 7fb9caffd640 1 -- 172.21.15.88:0/886571279 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9cc14f790 msgr2=0x7fb9cc14fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:08.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.502+0000 7fb9caffd640 1 --2- 172.21.15.88:0/886571279 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9cc14f790 0x7fb9cc14fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:08.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.502+0000 7fb9caffd640 1 -- 172.21.15.88:0/886571279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9b40039b0 msgr2=0x7fb9cc0cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:08.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.502+0000 7fb9caffd640 1 --2- 172.21.15.88:0/886571279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9b40039b0 0x7fb9cc0cd630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:08.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.502+0000 7fb9caffd640 1 -- 172.21.15.88:0/886571279 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9cc150130 con 0x7fb9cc0cf5f0 2024-09-16T01:44:08.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.502+0000 7fb9cbfff640 1 --2- 172.21.15.88:0/886571279 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9cc14f790 0x7fb9cc14fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:08.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.502+0000 7fb9caffd640 1 --2- 172.21.15.88:0/886571279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9cc0cf5f0 0x7fb9cc0b8b20 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0011a70 tx=0x7fb9c0011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:08.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.502+0000 7fb9b3fff640 1 -- 172.21.15.88:0/886571279 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c000fb50 con 0x7fb9cc0cf5f0 2024-09-16T01:44:08.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.502+0000 7fb9b3fff640 1 -- 172.21.15.88:0/886571279 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9c000fcf0 con 0x7fb9cc0cf5f0 2024-09-16T01:44:08.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.502+0000 7fb9d1dc4640 1 -- 172.21.15.88:0/886571279 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9cc0cde60 con 0x7fb9cc0cf5f0 2024-09-16T01:44:08.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.502+0000 7fb9b3fff640 1 -- 172.21.15.88:0/886571279 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c000fea0 con 0x7fb9cc0cf5f0 2024-09-16T01:44:08.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.502+0000 7fb9d1dc4640 1 -- 172.21.15.88:0/886571279 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9cc0ce420 con 0x7fb9cc0cf5f0 2024-09-16T01:44:08.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.503+0000 7fb9d1dc4640 1 -- 172.21.15.88:0/886571279 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9cc0d1110 con 0x7fb9cc0cf5f0 2024-09-16T01:44:08.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.504+0000 7fb9b3fff640 1 -- 172.21.15.88:0/886571279 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb9c0012040 con 0x7fb9cc0cf5f0 2024-09-16T01:44:08.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.504+0000 7fb9b3fff640 1 --2- 172.21.15.88:0/886571279 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb9b806c2e0 0x7fb9b806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:08.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.504+0000 7fb9b3fff640 1 -- 172.21.15.88:0/886571279 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8834+0+0 (secure 0 0 0) 0x7fb9c008c630 con 0x7fb9cc0cf5f0 2024-09-16T01:44:08.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.504+0000 7fb9cb7fe640 1 --2- 172.21.15.88:0/886571279 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb9b806c2e0 0x7fb9b806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:08.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.505+0000 7fb9cb7fe640 1 --2- 172.21.15.88:0/886571279 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb9b806c2e0 0x7fb9b806e7a0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc033000 tx=0x7fb9bc009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:08.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.508+0000 7fb9b3fff640 1 -- 172.21.15.88:0/886571279 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb9c0019030 con 0x7fb9cc0cf5f0 2024-09-16T01:44:08.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.711+0000 7fb9d1dc4640 1 -- 172.21.15.88:0/886571279 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9cc0b5400 con 0x7fb9cc0cf5f0 2024-09-16T01:44:08.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.711+0000 7fb9b3fff640 1 -- 172.21.15.88:0/886571279 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 294 v294) ==== 95+0+5230 (secure 0 0 0) 0x7fb9c005a110 con 0x7fb9cc0cf5f0 2024-09-16T01:44:08.713 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:08.713 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":294,"btime":"2024-09-16T01:44:07:982523+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9696,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4270197461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4270197461},{"type":"v1","addr":"172.21.15.88:6837","nonce":4270197461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":294,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:44:07.982512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10134},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10134":{"gid":10134,"name":"a","rank":0,"incarnation":294,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6836/1912564918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1912564918},{"type":"v1","addr":"172.21.15.88:6836","nonce":1912564918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:44:08.713 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 294 2024-09-16T01:44:08.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.713+0000 7fb9d1dc4640 1 -- 172.21.15.88:0/886571279 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb9b806c2e0 msgr2=0x7fb9b806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:08.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.713+0000 7fb9d1dc4640 1 --2- 172.21.15.88:0/886571279 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb9b806c2e0 0x7fb9b806e7a0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc033000 tx=0x7fb9bc009030 comp rx=0 tx=0).stop 2024-09-16T01:44:08.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.713+0000 7fb9d1dc4640 1 -- 172.21.15.88:0/886571279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9cc0cf5f0 msgr2=0x7fb9cc0b8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:08.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.713+0000 7fb9d1dc4640 1 --2- 172.21.15.88:0/886571279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9cc0cf5f0 0x7fb9cc0b8b20 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0011a70 tx=0x7fb9c0011f30 comp rx=0 tx=0).stop 2024-09-16T01:44:08.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.713+0000 7fb9d1dc4640 1 -- 172.21.15.88:0/886571279 shutdown_connections 2024-09-16T01:44:08.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.713+0000 7fb9d1dc4640 1 --2- 172.21.15.88:0/886571279 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb9b806c2e0 0x7fb9b806e7a0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:08.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.713+0000 7fb9d1dc4640 1 --2- 172.21.15.88:0/886571279 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb9cc14f790 0x7fb9cc14fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:08.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.713+0000 7fb9d1dc4640 1 --2- 172.21.15.88:0/886571279 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb9cc0cf5f0 0x7fb9cc0b8b20 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:08.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.713+0000 7fb9d1dc4640 1 --2- 172.21.15.88:0/886571279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9b40039b0 0x7fb9cc0cd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:08.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.713+0000 7fb9d1dc4640 1 -- 172.21.15.88:0/886571279 >> 172.21.15.88:0/886571279 conn(0x7fb9cc0b6410 msgr2=0x7fb9cc15be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:08.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.713+0000 7fb9d1dc4640 1 -- 172.21.15.88:0/886571279 shutdown_connections 2024-09-16T01:44:08.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:08.713+0000 7fb9d1dc4640 1 -- 172.21.15.88:0/886571279 wait complete. 2024-09-16T01:44:08.725 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 294, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:40:03.597796+0000', 'modified': '2024-09-16T01:44:07.982512+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 123, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10134}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10134': {'gid': 10134, 'name': 'a', 'rank': 0, 'incarnation': 294, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.88:6836/1912564918', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 1912564918}, {'type': 'v1', 'addr': '172.21.15.88:6836', 'nonce': 1912564918}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-16T01:44:08.725 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10134:up:replay 2024-09-16T01:44:08.726 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-09-16T01:44:11.976 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:44:11.975+0000 7f4be72e3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:44:14.131 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:44:14.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.240+0000 7fb6e156d640 1 Processor -- start 2024-09-16T01:44:14.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.240+0000 7fb6e156d640 1 -- start start 2024-09-16T01:44:14.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.241+0000 7fb6e156d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb6dc15e7a0 0x7fb6dc0ba240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.241+0000 7fb6e156d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb6dc0d3fe0 0x7fb6dc0d2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.241+0000 7fb6e156d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6dc0d4430 con 0x7fb6dc0cf2f0 2024-09-16T01:44:14.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.241+0000 7fb6e156d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb6dc0d2cc0 con 0x7fb6dc0d3fe0 2024-09-16T01:44:14.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.241+0000 7fb6e156d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb6dc0d2f40 con 0x7fb6dc15e7a0 2024-09-16T01:44:14.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.241+0000 7fb6d27fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb6dc0d3fe0 0x7fb6dc0d2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.241+0000 7fb6d27fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb6dc0d3fe0 0x7fb6dc0d2780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50404/0 (socket says 172.21.15.88:50404) 2024-09-16T01:44:14.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.241+0000 7fb6d27fc640 1 -- 172.21.15.88:0/2686654678 learned_addr learned my addr 172.21.15.88:0/2686654678 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:14.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.241+0000 7fb6db7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb6dc0cf2f0 0x7fb6dc0b7720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:37316/0 (socket says 172.21.15.88:37316) 2024-09-16T01:44:14.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.241+0000 7fb6daffd640 1 --2- 172.21.15.88:0/2686654678 >> v2:172.21.15.154:3300/0 conn(0x7fb6dc15e7a0 0x7fb6dc0ba240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.241+0000 7fb6d27fc640 1 -- 172.21.15.88:0/2686654678 >> v2:172.21.15.154:3300/0 conn(0x7fb6dc15e7a0 msgr2=0x7fb6dc0ba240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:14.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.241+0000 7fb6d27fc640 1 --2- 172.21.15.88:0/2686654678 >> v2:172.21.15.154:3300/0 conn(0x7fb6dc15e7a0 0x7fb6dc0ba240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.241+0000 7fb6d27fc640 1 -- 172.21.15.88:0/2686654678 >> v1:172.21.15.88:6789/0 conn(0x7fb6dc0cf2f0 legacy=0x7fb6dc0b7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:14.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.241+0000 7fb6d27fc640 1 -- 172.21.15.88:0/2686654678 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6dc0d3220 con 0x7fb6dc0d3fe0 2024-09-16T01:44:14.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.241+0000 7fb6da7fc640 1 -- 172.21.15.88:0/2686654678 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 936899793 0 0) 0x7fb6dc0d4430 con 0x7fb6dc0cf2f0 2024-09-16T01:44:14.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.242+0000 7fb6d27fc640 1 --2- 172.21.15.88:0/2686654678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6dc0d3fe0 0x7fb6dc0d2780 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc0b71e0 tx=0x7fb6c402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e23f06c326b17793 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:14.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.242+0000 7fb6da7fc640 1 -- 172.21.15.88:0/2686654678 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c4037070 con 0x7fb6dc0d3fe0 2024-09-16T01:44:14.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.242+0000 7fb6da7fc640 1 -- 172.21.15.88:0/2686654678 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6c402ae50 con 0x7fb6dc0d3fe0 2024-09-16T01:44:14.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.242+0000 7fb6da7fc640 1 -- 172.21.15.88:0/2686654678 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c4032860 con 0x7fb6dc0d3fe0 2024-09-16T01:44:14.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.243+0000 7fb6e156d640 1 -- 172.21.15.88:0/2686654678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6dc0d3fe0 msgr2=0x7fb6dc0d2780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:14.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.243+0000 7fb6e156d640 1 --2- 172.21.15.88:0/2686654678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6dc0d3fe0 0x7fb6dc0d2780 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc0b71e0 tx=0x7fb6c402a300 comp rx=0 tx=0).stop 2024-09-16T01:44:14.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.243+0000 7fb6e156d640 1 -- 172.21.15.88:0/2686654678 shutdown_connections 2024-09-16T01:44:14.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.243+0000 7fb6e156d640 1 --2- 172.21.15.88:0/2686654678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6dc0d3fe0 0x7fb6dc0d2780 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.243+0000 7fb6e156d640 1 --2- 172.21.15.88:0/2686654678 >> v2:172.21.15.154:3300/0 conn(0x7fb6dc15e7a0 0x7fb6dc0ba240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.243+0000 7fb6e156d640 1 -- 172.21.15.88:0/2686654678 >> 172.21.15.88:0/2686654678 conn(0x7fb6dc0d1550 msgr2=0x7fb6dc0d1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:14.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.243+0000 7fb6e156d640 1 -- 172.21.15.88:0/2686654678 shutdown_connections 2024-09-16T01:44:14.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.243+0000 7fb6e156d640 1 -- 172.21.15.88:0/2686654678 wait complete. 2024-09-16T01:44:14.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.244+0000 7fb6e156d640 1 Processor -- start 2024-09-16T01:44:14.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.244+0000 7fb6e156d640 1 -- start start 2024-09-16T01:44:14.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.245+0000 7fb6e156d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6dc0cf2f0 0x7fb6dc160660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.245+0000 7fb6e156d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6dc0d3fe0 0x7fb6dc14b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.245+0000 7fb6e156d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6dc15e7a0 0x7fb6dc0c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.245+0000 7fb6e156d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb6dc0d3ac0 con 0x7fb6dc15e7a0 2024-09-16T01:44:14.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.245+0000 7fb6e156d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb6dc0d37e0 con 0x7fb6dc0cf2f0 2024-09-16T01:44:14.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.245+0000 7fb6e156d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb6dc15eb80 con 0x7fb6dc0d3fe0 2024-09-16T01:44:14.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.245+0000 7fb6db7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6dc15e7a0 0x7fb6dc0c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.245+0000 7fb6db7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6dc15e7a0 0x7fb6dc0c9740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50408/0 (socket says 172.21.15.88:50408) 2024-09-16T01:44:14.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.245+0000 7fb6db7fe640 1 -- 172.21.15.88:0/1577210281 learned_addr learned my addr 172.21.15.88:0/1577210281 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:14.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.245+0000 7fb6daffd640 1 --2- 172.21.15.88:0/1577210281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6dc0cf2f0 0x7fb6dc160660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.245+0000 7fb6db7fe640 1 -- 172.21.15.88:0/1577210281 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6dc0d3fe0 msgr2=0x7fb6dc14b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:14.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.245+0000 7fb6d27fc640 1 --2- 172.21.15.88:0/1577210281 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6dc0d3fe0 0x7fb6dc14b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.245+0000 7fb6db7fe640 1 --2- 172.21.15.88:0/1577210281 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6dc0d3fe0 0x7fb6dc14b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.245+0000 7fb6db7fe640 1 -- 172.21.15.88:0/1577210281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6dc0cf2f0 msgr2=0x7fb6dc160660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:14.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.245+0000 7fb6db7fe640 1 --2- 172.21.15.88:0/1577210281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6dc0cf2f0 0x7fb6dc160660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.245+0000 7fb6db7fe640 1 -- 172.21.15.88:0/1577210281 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6dc157c70 con 0x7fb6dc15e7a0 2024-09-16T01:44:14.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.245+0000 7fb6d27fc640 1 --2- 172.21.15.88:0/1577210281 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6dc0d3fe0 0x7fb6dc14b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:14.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.245+0000 7fb6daffd640 1 --2- 172.21.15.88:0/1577210281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6dc0cf2f0 0x7fb6dc160660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:14.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.246+0000 7fb6db7fe640 1 --2- 172.21.15.88:0/1577210281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6dc15e7a0 0x7fb6dc0c9740 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7fb6d400c7a0 tx=0x7fb6d400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:14.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.246+0000 7fb6d3fff640 1 -- 172.21.15.88:0/1577210281 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d4018070 con 0x7fb6dc15e7a0 2024-09-16T01:44:14.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.246+0000 7fb6d3fff640 1 -- 172.21.15.88:0/1577210281 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6d400a5e0 con 0x7fb6dc15e7a0 2024-09-16T01:44:14.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.246+0000 7fb6e156d640 1 -- 172.21.15.88:0/1577210281 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6dc14b860 con 0x7fb6dc15e7a0 2024-09-16T01:44:14.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.246+0000 7fb6d3fff640 1 -- 172.21.15.88:0/1577210281 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d400aa50 con 0x7fb6dc15e7a0 2024-09-16T01:44:14.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.246+0000 7fb6e156d640 1 -- 172.21.15.88:0/1577210281 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6dc0cd890 con 0x7fb6dc15e7a0 2024-09-16T01:44:14.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.247+0000 7fb6e156d640 1 -- 172.21.15.88:0/1577210281 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6dc0d2780 con 0x7fb6dc15e7a0 2024-09-16T01:44:14.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.247+0000 7fb6d3fff640 1 -- 172.21.15.88:0/1577210281 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb6d401b070 con 0x7fb6dc15e7a0 2024-09-16T01:44:14.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.248+0000 7fb6d3fff640 1 --2- 172.21.15.88:0/1577210281 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb6bc06c2b0 0x7fb6bc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.248+0000 7fb6d3fff640 1 -- 172.21.15.88:0/1577210281 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8834+0+0 (secure 0 0 0) 0x7fb6d408b660 con 0x7fb6dc15e7a0 2024-09-16T01:44:14.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.248+0000 7fb6daffd640 1 --2- 172.21.15.88:0/1577210281 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb6bc06c2b0 0x7fb6bc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.248+0000 7fb6daffd640 1 --2- 172.21.15.88:0/1577210281 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb6bc06c2b0 0x7fb6bc06e770 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc006fd0 tx=0x7fb6cc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:14.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.252+0000 7fb6d3fff640 1 -- 172.21.15.88:0/1577210281 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb6dc14b860 con 0x7fb6dc15e7a0 2024-09-16T01:44:14.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.450+0000 7fb6e156d640 1 -- 172.21.15.88:0/1577210281 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6dc0ceaf0 con 0x7fb6dc15e7a0 2024-09-16T01:44:14.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.451+0000 7fb6d3fff640 1 -- 172.21.15.88:0/1577210281 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 297 v297) ==== 95+0+5239 (secure 0 0 0) 0x7fb6dc157c70 con 0x7fb6dc15e7a0 2024-09-16T01:44:14.453 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:14.453 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":297,"btime":"2024-09-16T01:44:11:000889+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9696,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4270197461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4270197461},{"type":"v1","addr":"172.21.15.88:6837","nonce":4270197461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":297,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:44:11.000884+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10134},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10134":{"gid":10134,"name":"a","rank":0,"incarnation":294,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6836/1912564918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1912564918},{"type":"v1","addr":"172.21.15.88:6836","nonce":1912564918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10134,"qdb_cluster":[10134]},"id":6}]} 2024-09-16T01:44:14.453 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 297 2024-09-16T01:44:14.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.453+0000 7fb6e156d640 1 -- 172.21.15.88:0/1577210281 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb6bc06c2b0 msgr2=0x7fb6bc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:14.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.453+0000 7fb6e156d640 1 --2- 172.21.15.88:0/1577210281 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb6bc06c2b0 0x7fb6bc06e770 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc006fd0 tx=0x7fb6cc008040 comp rx=0 tx=0).stop 2024-09-16T01:44:14.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.453+0000 7fb6e156d640 1 -- 172.21.15.88:0/1577210281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6dc15e7a0 msgr2=0x7fb6dc0c9740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:14.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.453+0000 7fb6e156d640 1 --2- 172.21.15.88:0/1577210281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6dc15e7a0 0x7fb6dc0c9740 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7fb6d400c7a0 tx=0x7fb6d400cc60 comp rx=0 tx=0).stop 2024-09-16T01:44:14.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.453+0000 7fb6e156d640 1 -- 172.21.15.88:0/1577210281 shutdown_connections 2024-09-16T01:44:14.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.453+0000 7fb6e156d640 1 --2- 172.21.15.88:0/1577210281 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb6bc06c2b0 0x7fb6bc06e770 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.453+0000 7fb6e156d640 1 --2- 172.21.15.88:0/1577210281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb6dc15e7a0 0x7fb6dc0c9740 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.453+0000 7fb6e156d640 1 --2- 172.21.15.88:0/1577210281 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb6dc0d3fe0 0x7fb6dc14b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.453+0000 7fb6e156d640 1 --2- 172.21.15.88:0/1577210281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6dc0cf2f0 0x7fb6dc160660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.453+0000 7fb6e156d640 1 -- 172.21.15.88:0/1577210281 >> 172.21.15.88:0/1577210281 conn(0x7fb6dc0d1550 msgr2=0x7fb6dc06d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:14.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.453+0000 7fb6e156d640 1 -- 172.21.15.88:0/1577210281 shutdown_connections 2024-09-16T01:44:14.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.453+0000 7fb6e156d640 1 -- 172.21.15.88:0/1577210281 wait complete. 2024-09-16T01:44:14.466 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 297, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:40:03.597796+0000', 'modified': '2024-09-16T01:44:11.000884+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 123, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10134}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10134': {'gid': 10134, 'name': 'a', 'rank': 0, 'incarnation': 294, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.88:6836/1912564918', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 1912564918}, {'type': 'v1', 'addr': '172.21.15.88:6836', 'nonce': 1912564918}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10134, 'qdb_cluster': [10134]} 2024-09-16T01:44:14.466 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T01:44:14.466 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-16T01:44:14.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.564+0000 7fddeece6640 1 Processor -- start 2024-09-16T01:44:14.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.564+0000 7fddeece6640 1 -- start start 2024-09-16T01:44:14.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.565+0000 7fddeece6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde80720d0 0x7fdde80b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.565+0000 7fddeece6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdde80cca40 0x7fdde80b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.565+0000 7fddeece6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdde8071720 con 0x7fdde815ad10 2024-09-16T01:44:14.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.565+0000 7fddeece6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fdde8070fb0 con 0x7fdde80cca40 2024-09-16T01:44:14.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.565+0000 7fddeece6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdde8073920 con 0x7fdde80720d0 2024-09-16T01:44:14.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.565+0000 7fdded283640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdde80cca40 0x7fdde80b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.565+0000 7fdded283640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdde80cca40 0x7fdde80b43e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50416/0 (socket says 172.21.15.88:50416) 2024-09-16T01:44:14.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.565+0000 7fdded283640 1 -- 172.21.15.88:0/1715568568 learned_addr learned my addr 172.21.15.88:0/1715568568 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:14.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.566+0000 7fdded283640 1 -- 172.21.15.88:0/1715568568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde80720d0 msgr2=0x7fdde80b3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:14.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.566+0000 7fdde7fff640 1 --2- 172.21.15.88:0/1715568568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde80720d0 0x7fdde80b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.566+0000 7fdded283640 1 --2- 172.21.15.88:0/1715568568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde80720d0 0x7fdde80b3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.566+0000 7fdded283640 1 -- 172.21.15.88:0/1715568568 >> v1:172.21.15.154:6789/0 conn(0x7fdde815ad10 legacy=0x7fdde80719c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:14.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.566+0000 7fdded283640 1 -- 172.21.15.88:0/1715568568 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdde8073c00 con 0x7fdde80cca40 2024-09-16T01:44:14.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.566+0000 7fdded283640 1 --2- 172.21.15.88:0/1715568568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdde80cca40 0x7fdde80b43e0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009870 tx=0x7fdde002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5233dc79ccd1842f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:14.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.566+0000 7fdde77fe640 1 -- 172.21.15.88:0/1715568568 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde0037070 con 0x7fdde80cca40 2024-09-16T01:44:14.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.566+0000 7fdde77fe640 1 -- 172.21.15.88:0/1715568568 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdde0032440 con 0x7fdde80cca40 2024-09-16T01:44:14.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.566+0000 7fdde77fe640 1 -- 172.21.15.88:0/1715568568 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde0032810 con 0x7fdde80cca40 2024-09-16T01:44:14.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.567+0000 7fddeece6640 1 -- 172.21.15.88:0/1715568568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdde80cca40 msgr2=0x7fdde80b43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:14.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.567+0000 7fddeece6640 1 --2- 172.21.15.88:0/1715568568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdde80cca40 0x7fdde80b43e0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009870 tx=0x7fdde002a180 comp rx=0 tx=0).stop 2024-09-16T01:44:14.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.567+0000 7fddeece6640 1 -- 172.21.15.88:0/1715568568 shutdown_connections 2024-09-16T01:44:14.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.567+0000 7fddeece6640 1 --2- 172.21.15.88:0/1715568568 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdde80cca40 0x7fdde80b43e0 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.567+0000 7fddeece6640 1 --2- 172.21.15.88:0/1715568568 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde80720d0 0x7fdde80b3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.567+0000 7fddeece6640 1 -- 172.21.15.88:0/1715568568 >> 172.21.15.88:0/1715568568 conn(0x7fdde806d0f0 msgr2=0x7fdde806d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:14.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.567+0000 7fddeece6640 1 -- 172.21.15.88:0/1715568568 shutdown_connections 2024-09-16T01:44:14.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.567+0000 7fddeece6640 1 -- 172.21.15.88:0/1715568568 wait complete. 2024-09-16T01:44:14.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.568+0000 7fddeece6640 1 Processor -- start 2024-09-16T01:44:14.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.568+0000 7fddeece6640 1 -- start start 2024-09-16T01:44:14.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.569+0000 7fddeece6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdde80720d0 0x7fdde8152f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.569+0000 7fddeece6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde80cca40 0x7fdde80ca330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.569+0000 7fddeece6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdde815ad10 0x7fdde813d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.569+0000 7fddeece6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdde80cc360 con 0x7fdde80720d0 2024-09-16T01:44:14.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.569+0000 7fddeece6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdde8071200 con 0x7fdde815ad10 2024-09-16T01:44:14.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.569+0000 7fddeece6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdde80cc4e0 con 0x7fdde80cca40 2024-09-16T01:44:14.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.569+0000 7fddeca82640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdde80720d0 0x7fdde8152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.569+0000 7fddeca82640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdde80720d0 0x7fdde8152f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50426/0 (socket says 172.21.15.88:50426) 2024-09-16T01:44:14.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.569+0000 7fddeca82640 1 -- 172.21.15.88:0/1655417571 learned_addr learned my addr 172.21.15.88:0/1655417571 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:14.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.569+0000 7fdde7fff640 1 --2- 172.21.15.88:0/1655417571 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde80cca40 0x7fdde80ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.569+0000 7fdded283640 1 --2- 172.21.15.88:0/1655417571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdde815ad10 0x7fdde813d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.570+0000 7fddeca82640 1 -- 172.21.15.88:0/1655417571 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde80cca40 msgr2=0x7fdde80ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:14.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.570+0000 7fddeca82640 1 --2- 172.21.15.88:0/1655417571 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde80cca40 0x7fdde80ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.570+0000 7fddeca82640 1 -- 172.21.15.88:0/1655417571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdde815ad10 msgr2=0x7fdde813d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:14.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.570+0000 7fddeca82640 1 --2- 172.21.15.88:0/1655417571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdde815ad10 0x7fdde813d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.570+0000 7fddeca82640 1 -- 172.21.15.88:0/1655417571 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdde8157060 con 0x7fdde80720d0 2024-09-16T01:44:14.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.570+0000 7fddeca82640 1 --2- 172.21.15.88:0/1655417571 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdde80720d0 0x7fdde8152f30 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7fddd800b3f0 tx=0x7fddd800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:14.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.570+0000 7fdde57fa640 1 -- 172.21.15.88:0/1655417571 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd8015020 con 0x7fdde80720d0 2024-09-16T01:44:14.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.570+0000 7fdde57fa640 1 -- 172.21.15.88:0/1655417571 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddd8011830 con 0x7fdde80720d0 2024-09-16T01:44:14.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.570+0000 7fdde57fa640 1 -- 172.21.15.88:0/1655417571 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd8011c20 con 0x7fdde80720d0 2024-09-16T01:44:14.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.570+0000 7fddeece6640 1 -- 172.21.15.88:0/1655417571 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdde813d6e0 con 0x7fdde80720d0 2024-09-16T01:44:14.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.570+0000 7fddeece6640 1 -- 172.21.15.88:0/1655417571 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdde80cad40 con 0x7fdde80720d0 2024-09-16T01:44:14.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.572+0000 7fdde57fa640 1 -- 172.21.15.88:0/1655417571 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fddd8018020 con 0x7fdde80720d0 2024-09-16T01:44:14.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.572+0000 7fdde57fa640 1 --2- 172.21.15.88:0/1655417571 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fddd406c2b0 0x7fddd406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.572+0000 7fdde57fa640 1 -- 172.21.15.88:0/1655417571 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8834+0+0 (secure 0 0 0) 0x7fddd808bbb0 con 0x7fdde80720d0 2024-09-16T01:44:14.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.572+0000 7fdde7fff640 1 --2- 172.21.15.88:0/1655417571 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fddd406c2b0 0x7fddd406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.572+0000 7fdde7fff640 1 --2- 172.21.15.88:0/1655417571 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fddd406c2b0 0x7fddd406e770 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fdde8153c90 tx=0x7fdddc007730 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:14.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.573+0000 7fddf1a7e740 1 Processor -- start 2024-09-16T01:44:14.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.573+0000 7fddf1a7e740 1 -- start start 2024-09-16T01:44:14.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.573+0000 7fddf1a7e740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5647fe8582c0 0x5647fe89da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.574+0000 7fddf1a7e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647fe855a80 0x5647fe8586a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.574+0000 7fddf1a7e740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe859a50 0x5647fe8528b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.574+0000 7fddf1a7e740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5647fe7ba520 con 0x5647fe859a50 2024-09-16T01:44:14.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.574+0000 7fddf1a7e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5647fe7623e0 con 0x5647fe855a80 2024-09-16T01:44:14.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.574+0000 7fddf1a7e740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5647fe7b64e0 con 0x5647fe8582c0 2024-09-16T01:44:14.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.574+0000 7fdded283640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe859a50 0x5647fe8528b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.574+0000 7fdded283640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe859a50 0x5647fe8528b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50432/0 (socket says 172.21.15.88:50432) 2024-09-16T01:44:14.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.574+0000 7fdded283640 1 -- 172.21.15.88:0/3811981 learned_addr learned my addr 172.21.15.88:0/3811981 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:14.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.574+0000 7fdde7fff640 1 --2- 172.21.15.88:0/3811981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647fe855a80 0x5647fe8586a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.574+0000 7fddeca82640 1 --2- 172.21.15.88:0/3811981 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5647fe8582c0 0x5647fe89da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.574+0000 7fdded283640 1 -- 172.21.15.88:0/3811981 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5647fe8582c0 msgr2=0x5647fe89da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:14.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.574+0000 7fdded283640 1 --2- 172.21.15.88:0/3811981 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5647fe8582c0 0x5647fe89da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.574+0000 7fdded283640 1 -- 172.21.15.88:0/3811981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647fe855a80 msgr2=0x5647fe8586a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:14.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.574+0000 7fdded283640 1 --2- 172.21.15.88:0/3811981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647fe855a80 0x5647fe8586a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.574+0000 7fdded283640 1 -- 172.21.15.88:0/3811981 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647fe77d2f0 con 0x5647fe859a50 2024-09-16T01:44:14.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.574+0000 7fdded283640 1 --2- 172.21.15.88:0/3811981 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe859a50 0x5647fe8528b0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7fdde002a1c0 tx=0x7fdde002c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcef0e597174c15c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:14.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.574+0000 7fddce7fc640 1 -- 172.21.15.88:0/3811981 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde0042070 con 0x5647fe859a50 2024-09-16T01:44:14.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.574+0000 7fddce7fc640 1 -- 172.21.15.88:0/3811981 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdde00326e0 con 0x5647fe859a50 2024-09-16T01:44:14.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.574+0000 7fddce7fc640 1 -- 172.21.15.88:0/3811981 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde0037040 con 0x5647fe859a50 2024-09-16T01:44:14.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.575+0000 7fddf1a7e740 1 -- 172.21.15.88:0/3811981 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe859a50 msgr2=0x5647fe8528b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:14.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.575+0000 7fddf1a7e740 1 --2- 172.21.15.88:0/3811981 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe859a50 0x5647fe8528b0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7fdde002a1c0 tx=0x7fdde002c000 comp rx=0 tx=0).stop 2024-09-16T01:44:14.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.575+0000 7fddf1a7e740 1 -- 172.21.15.88:0/3811981 shutdown_connections 2024-09-16T01:44:14.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.575+0000 7fddf1a7e740 1 --2- 172.21.15.88:0/3811981 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe859a50 0x5647fe8528b0 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.575+0000 7fddf1a7e740 1 --2- 172.21.15.88:0/3811981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647fe855a80 0x5647fe8586a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.575+0000 7fddf1a7e740 1 --2- 172.21.15.88:0/3811981 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5647fe8582c0 0x5647fe89da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.575+0000 7fddf1a7e740 1 -- 172.21.15.88:0/3811981 >> 172.21.15.88:0/3811981 conn(0x5647fe854a80 msgr2=0x5647fe89cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:14.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.575+0000 7fddf1a7e740 1 -- 172.21.15.88:0/3811981 shutdown_connections 2024-09-16T01:44:14.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.575+0000 7fddf1a7e740 1 -- 172.21.15.88:0/3811981 wait complete. 2024-09-16T01:44:14.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.575+0000 7fddf1a7e740 1 -- start start 2024-09-16T01:44:14.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.576+0000 7fddf1a7e740 1 Processor -- start 2024-09-16T01:44:14.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.576+0000 7fddf1a7e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647fe8582c0 0x5647fe8add40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.576+0000 7fddf1a7e740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5647fe859a50 0x5647fe878c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.576+0000 7fddf1a7e740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe8b0720 0x5647fe8b0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.576+0000 7fddf1a7e740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5647fe853f20 con 0x5647fe8b0720 2024-09-16T01:44:14.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.576+0000 7fddf1a7e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5647fe743450 con 0x5647fe8582c0 2024-09-16T01:44:14.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.576+0000 7fddf1a7e740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5647fe89e290 con 0x5647fe859a50 2024-09-16T01:44:14.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.577+0000 7fddf1a7e740 10 client.0 _ll_register_callbacks cb 0x5647fe4ccac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:44:14.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.577+0000 7fdded283640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe8b0720 0x5647fe8b0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.577+0000 7fdded283640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe8b0720 0x5647fe8b0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50448/0 (socket says 172.21.15.88:50448) 2024-09-16T01:44:14.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.577+0000 7fdded283640 1 -- 172.21.15.88:0/711372096 learned_addr learned my addr 172.21.15.88:0/711372096 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:14.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.577+0000 7fddeca82640 1 --2- 172.21.15.88:0/711372096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647fe8582c0 0x5647fe8add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.577+0000 7fdde7fff640 1 --2- 172.21.15.88:0/711372096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5647fe859a50 0x5647fe878c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.577+0000 7fdded283640 1 -- 172.21.15.88:0/711372096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5647fe859a50 msgr2=0x5647fe878c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:14.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.577+0000 7fdded283640 1 --2- 172.21.15.88:0/711372096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5647fe859a50 0x5647fe878c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.577+0000 7fdded283640 1 -- 172.21.15.88:0/711372096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647fe8582c0 msgr2=0x5647fe8add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:14.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.577+0000 7fdded283640 1 --2- 172.21.15.88:0/711372096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647fe8582c0 0x5647fe8add40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.577+0000 7fdded283640 1 -- 172.21.15.88:0/711372096 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647fe4cd2d0 con 0x5647fe8b0720 2024-09-16T01:44:14.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.577+0000 7fdded283640 1 --2- 172.21.15.88:0/711372096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe8b0720 0x5647fe8b0ba0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009990 tx=0x7fdde0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f97ba5b46391302 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:14.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.577+0000 7fddc77fe640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:44:14.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.577+0000 7fddc77fe640 1 -- 172.21.15.88:0/711372096 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde004b060 con 0x5647fe8b0720 2024-09-16T01:44:14.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.577+0000 7fddf1a7e740 1 -- 172.21.15.88:0/711372096 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdde00326e0 con 0x5647fe8b0720 2024-09-16T01:44:14.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.577+0000 7fddc77fe640 1 -- 172.21.15.88:0/711372096 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 8834+0+0 (secure 0 0 0) 0x7fdde0032a90 con 0x5647fe8b0720 2024-09-16T01:44:14.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.577+0000 7fddc77fe640 1 -- 172.21.15.88:0/711372096 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=297) ==== 24+0+0 (secure 0 0 0) 0x7fdde0052050 con 0x5647fe8b0720 2024-09-16T01:44:14.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.577+0000 7fddf1a7e740 10 client.10188 fetch_fsmap learned FSMap version 297 2024-09-16T01:44:14.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.577+0000 7fddf1a7e740 1 -- 172.21.15.88:0/711372096 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=297,monmap=2+}) -- 0x5647fe5976b0 con 0x5647fe8b0720 2024-09-16T01:44:14.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.577+0000 7fddc77fe640 1 -- 172.21.15.88:0/711372096 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdde00326e0 con 0x5647fe8b0720 2024-09-16T01:44:14.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.578+0000 7fddc77fe640 1 -- 172.21.15.88:0/711372096 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 297) ==== 3502+0+0 (secure 0 0 0) 0x7fdde0003610 con 0x5647fe8b0720 2024-09-16T01:44:14.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.578+0000 7fddf1a7e740 10 client.10188 fetch_fsmap finished waiting for FSMap version 297 2024-09-16T01:44:14.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.578+0000 7fddf1a7e740 10 client.10188 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:44:14.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.578+0000 7fddf1a7e740 20 client.10188 populate_metadata read hostname 'smithi088' 2024-09-16T01:44:14.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.578+0000 7fddf1a7e740 1 --2- 172.21.15.88:0/711372096 >> [v2:172.21.15.88:6834/1912564918,v1:172.21.15.88:6836/1912564918] conn(0x5647fe89b910 0x5647fe8517d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.578+0000 7fddf1a7e740 4 client.10188 mds_command: new command op to 10134 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:44:14.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.578+0000 7fddf1a7e740 1 -- 172.21.15.88:0/711372096 --> [v2:172.21.15.88:6834/1912564918,v1:172.21.15.88:6836/1912564918] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5647fe7c7c50 con 0x5647fe89b910 2024-09-16T01:44:14.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.578+0000 7fddeca82640 1 --2- 172.21.15.88:0/711372096 >> [v2:172.21.15.88:6834/1912564918,v1:172.21.15.88:6836/1912564918] conn(0x5647fe89b910 0x5647fe8517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.578+0000 7fddc77fe640 10 client.10188 ms_handle_connect on v2:172.21.15.88:6834/1912564918 2024-09-16T01:44:14.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.578+0000 7fddeca82640 1 --2- 172.21.15.88:0/711372096 >> [v2:172.21.15.88:6834/1912564918,v1:172.21.15.88:6836/1912564918] conn(0x5647fe89b910 0x5647fe8517d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d5f0e0081a6aa33 server_cookie=8a8b0a8d32998d69 in_seq=0 out_seq=0 2024-09-16T01:44:14.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.579+0000 7fddc77fe640 1 -- 172.21.15.88:0/711372096 <== mds.0 v2:172.21.15.88:6834/1912564918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5647fe7c7c50 con 0x5647fe89b910 2024-09-16T01:44:14.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.579+0000 7fddc77fe640 10 client.10188 handle_command_reply: tid=0 2024-09-16T01:44:14.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.579+0000 7fddf1a7e740 1 client.10188 shutdown 2024-09-16T01:44:14.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.579+0000 7fddf1a7e740 1 -- 172.21.15.88:0/711372096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe8b0720 msgr2=0x5647fe8b0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:14.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.579+0000 7fddf1a7e740 1 --2- 172.21.15.88:0/711372096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe8b0720 0x5647fe8b0ba0 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009990 tx=0x7fdde0004730 comp rx=0 tx=0).stop 2024-09-16T01:44:14.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.579+0000 7fddf1a7e740 1 -- 172.21.15.88:0/711372096 shutdown_connections 2024-09-16T01:44:14.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.579+0000 7fddf1a7e740 1 --2- 172.21.15.88:0/711372096 >> [v2:172.21.15.88:6834/1912564918,v1:172.21.15.88:6836/1912564918] conn(0x5647fe89b910 0x5647fe8517d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.579+0000 7fddf1a7e740 1 --2- 172.21.15.88:0/711372096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe8b0720 0x5647fe8b0ba0 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.579+0000 7fddf1a7e740 1 --2- 172.21.15.88:0/711372096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5647fe859a50 0x5647fe878c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.579+0000 7fddc77fe640 0 client.10188 ms_handle_reset on v2:172.21.15.88:6834/1912564918 2024-09-16T01:44:14.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.579+0000 7fddf1a7e740 1 --2- 172.21.15.88:0/711372096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647fe8582c0 0x5647fe8add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.579+0000 7fddf1a7e740 1 -- 172.21.15.88:0/711372096 >> 172.21.15.88:0/711372096 conn(0x5647fe855a80 msgr2=0x5647fe8a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:14.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.579+0000 7fddf1a7e740 1 -- 172.21.15.88:0/711372096 shutdown_connections 2024-09-16T01:44:14.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.579+0000 7fddf1a7e740 1 -- 172.21.15.88:0/711372096 wait complete. 2024-09-16T01:44:14.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.579+0000 7fddf1a7e740 20 client.10188 trim_cache size 0 max 16384 2024-09-16T01:44:14.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.597+0000 7fddf1a7e740 1 Processor -- start 2024-09-16T01:44:14.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.597+0000 7fddf1a7e740 1 -- start start 2024-09-16T01:44:14.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.597+0000 7fddf1a7e740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe8582c0 0x5647fe899970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.597+0000 7fddf1a7e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647fe859a50 0x5647fe876820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.597+0000 7fddf1a7e740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5647fe8b0720 0x5647fe876d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.597+0000 7fddf1a7e740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdde81ffa70 con 0x5647fe8582c0 2024-09-16T01:44:14.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.597+0000 7fddf1a7e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdde81fedd0 con 0x5647fe859a50 2024-09-16T01:44:14.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.597+0000 7fddf1a7e740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdde81ff420 con 0x5647fe8b0720 2024-09-16T01:44:14.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.597+0000 7fddeca82640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe8582c0 0x5647fe899970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.598+0000 7fddeca82640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe8582c0 0x5647fe899970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50450/0 (socket says 172.21.15.88:50450) 2024-09-16T01:44:14.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.598+0000 7fddeca82640 1 -- 172.21.15.88:0/617241485 learned_addr learned my addr 172.21.15.88:0/617241485 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:14.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.598+0000 7fdde7fff640 1 --2- 172.21.15.88:0/617241485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647fe859a50 0x5647fe876820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.598+0000 7fddeca82640 1 -- 172.21.15.88:0/617241485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5647fe8b0720 msgr2=0x5647fe876d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:14.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.598+0000 7fdded283640 1 --2- 172.21.15.88:0/617241485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5647fe8b0720 0x5647fe876d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.598+0000 7fddeca82640 1 --2- 172.21.15.88:0/617241485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5647fe8b0720 0x5647fe876d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.598+0000 7fddeca82640 1 -- 172.21.15.88:0/617241485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647fe859a50 msgr2=0x5647fe876820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:14.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.598+0000 7fddeca82640 1 --2- 172.21.15.88:0/617241485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647fe859a50 0x5647fe876820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.598+0000 7fddeca82640 1 -- 172.21.15.88:0/617241485 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647fe7f4800 con 0x5647fe8582c0 2024-09-16T01:44:14.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.598+0000 7fdded283640 1 --2- 172.21.15.88:0/617241485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5647fe8b0720 0x5647fe876d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:14.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.598+0000 7fddeca82640 1 --2- 172.21.15.88:0/617241485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe8582c0 0x5647fe899970 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fddd8058fb0 tx=0x7fddd808b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73eb848306e65e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:14.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.598+0000 7fddcdffb640 1 -- 172.21.15.88:0/617241485 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd8089530 con 0x5647fe8582c0 2024-09-16T01:44:14.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.598+0000 7fddcdffb640 1 -- 172.21.15.88:0/617241485 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddd8083e50 con 0x5647fe8582c0 2024-09-16T01:44:14.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.598+0000 7fddcdffb640 1 -- 172.21.15.88:0/617241485 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd8057320 con 0x5647fe8582c0 2024-09-16T01:44:14.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.599+0000 7fddf1a7e740 1 -- 172.21.15.88:0/617241485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe8582c0 msgr2=0x5647fe899970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:14.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.599+0000 7fddf1a7e740 1 --2- 172.21.15.88:0/617241485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe8582c0 0x5647fe899970 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fddd8058fb0 tx=0x7fddd808b040 comp rx=0 tx=0).stop 2024-09-16T01:44:14.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.599+0000 7fddf1a7e740 1 -- 172.21.15.88:0/617241485 shutdown_connections 2024-09-16T01:44:14.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.599+0000 7fddf1a7e740 1 --2- 172.21.15.88:0/617241485 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5647fe8b0720 0x5647fe876d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.599+0000 7fddf1a7e740 1 --2- 172.21.15.88:0/617241485 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647fe859a50 0x5647fe876820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.599+0000 7fddf1a7e740 1 --2- 172.21.15.88:0/617241485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe8582c0 0x5647fe899970 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.599+0000 7fddf1a7e740 1 -- 172.21.15.88:0/617241485 >> 172.21.15.88:0/617241485 conn(0x5647fe855a80 msgr2=0x5647fe8a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:14.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.599+0000 7fddf1a7e740 1 -- 172.21.15.88:0/617241485 shutdown_connections 2024-09-16T01:44:14.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.599+0000 7fddf1a7e740 1 -- 172.21.15.88:0/617241485 wait complete. 2024-09-16T01:44:14.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.599+0000 7fddf1a7e740 1 -- start start 2024-09-16T01:44:14.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.599+0000 7fddf1a7e740 1 Processor -- start 2024-09-16T01:44:14.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.600+0000 7fddf1a7e740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5647fe8582c0 0x5647fe877ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.600+0000 7fddf1a7e740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647fe859a50 0x5647fe91b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.600+0000 7fddf1a7e740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe8b0720 0x5647fe857360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.600+0000 7fddf1a7e740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5647fe8b12b0 con 0x5647fe8b0720 2024-09-16T01:44:14.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.600+0000 7fddf1a7e740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5647fe8ad5a0 con 0x5647fe859a50 2024-09-16T01:44:14.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.600+0000 7fddf1a7e740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5647fe89bd10 con 0x5647fe8582c0 2024-09-16T01:44:14.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.600+0000 7fddf1a7e740 10 client.0 _ll_register_callbacks cb 0x7fdddc015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:44:14.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.600+0000 7fdded283640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe8b0720 0x5647fe857360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.600+0000 7fdded283640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe8b0720 0x5647fe857360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50462/0 (socket says 172.21.15.88:50462) 2024-09-16T01:44:14.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.600+0000 7fdded283640 1 -- 172.21.15.88:0/1887100837 learned_addr learned my addr 172.21.15.88:0/1887100837 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:14.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.600+0000 7fddeca82640 1 --2- 172.21.15.88:0/1887100837 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5647fe8582c0 0x5647fe877ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.600+0000 7fdde7fff640 1 --2- 172.21.15.88:0/1887100837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647fe859a50 0x5647fe91b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.601+0000 7fdded283640 1 -- 172.21.15.88:0/1887100837 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5647fe8582c0 msgr2=0x5647fe877ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:14.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.601+0000 7fdded283640 1 --2- 172.21.15.88:0/1887100837 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5647fe8582c0 0x5647fe877ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.601+0000 7fdded283640 1 -- 172.21.15.88:0/1887100837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647fe859a50 msgr2=0x5647fe91b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:14.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.601+0000 7fdded283640 1 --2- 172.21.15.88:0/1887100837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647fe859a50 0x5647fe91b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.601+0000 7fdded283640 1 -- 172.21.15.88:0/1887100837 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647fe876e10 con 0x5647fe8b0720 2024-09-16T01:44:14.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.601+0000 7fdded283640 1 --2- 172.21.15.88:0/1887100837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe8b0720 0x5647fe857360 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fdde000b030 tx=0x7fdde002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d84ed1b9dc1e3c22 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:14.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.601+0000 7fddc77fe640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:44:14.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.601+0000 7fddc77fe640 1 -- 172.21.15.88:0/1887100837 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde004b040 con 0x5647fe8b0720 2024-09-16T01:44:14.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.601+0000 7fddc77fe640 1 -- 172.21.15.88:0/1887100837 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 8834+0+0 (secure 0 0 0) 0x7fdde00326e0 con 0x5647fe8b0720 2024-09-16T01:44:14.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.601+0000 7fddf1a7e740 1 -- 172.21.15.88:0/1887100837 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fddd8083e50 con 0x5647fe8b0720 2024-09-16T01:44:14.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.601+0000 7fddc77fe640 1 -- 172.21.15.88:0/1887100837 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=297) ==== 24+0+0 (secure 0 0 0) 0x7fdde0037040 con 0x5647fe8b0720 2024-09-16T01:44:14.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.601+0000 7fddf1a7e740 10 client.10194 fetch_fsmap learned FSMap version 297 2024-09-16T01:44:14.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.601+0000 7fddf1a7e740 1 -- 172.21.15.88:0/1887100837 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=297,monmap=2+}) -- 0x5647fe8724b0 con 0x5647fe8b0720 2024-09-16T01:44:14.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.601+0000 7fddc77fe640 1 -- 172.21.15.88:0/1887100837 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddd8083e50 con 0x5647fe8b0720 2024-09-16T01:44:14.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.601+0000 7fddc77fe640 1 -- 172.21.15.88:0/1887100837 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 297) ==== 3502+0+0 (secure 0 0 0) 0x7fdde00328e0 con 0x5647fe8b0720 2024-09-16T01:44:14.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.601+0000 7fddf1a7e740 10 client.10194 fetch_fsmap finished waiting for FSMap version 297 2024-09-16T01:44:14.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.601+0000 7fddf1a7e740 10 client.10194 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:44:14.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.601+0000 7fddf1a7e740 20 client.10194 populate_metadata read hostname 'smithi088' 2024-09-16T01:44:14.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.601+0000 7fddf1a7e740 1 --2- 172.21.15.88:0/1887100837 >> [v2:172.21.15.88:6834/1912564918,v1:172.21.15.88:6836/1912564918] conn(0x5647fe899950 0x5647fe8bfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.601+0000 7fddf1a7e740 4 client.10194 mds_command: new command op to 10134 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:44:14.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.601+0000 7fddf1a7e740 1 -- 172.21.15.88:0/1887100837 --> [v2:172.21.15.88:6834/1912564918,v1:172.21.15.88:6836/1912564918] -- command(tid 0: {"prefix": "status"}) -- 0x5647fe8810d0 con 0x5647fe899950 2024-09-16T01:44:14.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.601+0000 7fddeca82640 1 --2- 172.21.15.88:0/1887100837 >> [v2:172.21.15.88:6834/1912564918,v1:172.21.15.88:6836/1912564918] conn(0x5647fe899950 0x5647fe8bfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.602+0000 7fddc77fe640 10 client.10194 ms_handle_connect on v2:172.21.15.88:6834/1912564918 2024-09-16T01:44:14.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.602+0000 7fddeca82640 1 --2- 172.21.15.88:0/1887100837 >> [v2:172.21.15.88:6834/1912564918,v1:172.21.15.88:6836/1912564918] conn(0x5647fe899950 0x5647fe8bfbd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5336d1441aefa59c server_cookie=80efa04464496f74 in_seq=0 out_seq=0 2024-09-16T01:44:14.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.602+0000 7fddc77fe640 1 -- 172.21.15.88:0/1887100837 <== mds.0 v2:172.21.15.88:6834/1912564918 1 ==== command_reply(tid 0: 0 ) ==== 8+0+335 (crc 0 0 0) 0x5647fe8810d0 con 0x5647fe899950 2024-09-16T01:44:14.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.602+0000 7fddc77fe640 10 client.10194 handle_command_reply: tid=0 2024-09-16T01:44:14.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.602+0000 7fddf1a7e740 1 client.10194 shutdown 2024-09-16T01:44:14.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.602+0000 7fddf1a7e740 1 -- 172.21.15.88:0/1887100837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe8b0720 msgr2=0x5647fe857360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:14.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.602+0000 7fddf1a7e740 1 --2- 172.21.15.88:0/1887100837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe8b0720 0x5647fe857360 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fdde000b030 tx=0x7fdde002e000 comp rx=0 tx=0).stop 2024-09-16T01:44:14.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.602+0000 7fddf1a7e740 1 -- 172.21.15.88:0/1887100837 shutdown_connections 2024-09-16T01:44:14.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.602+0000 7fddf1a7e740 1 --2- 172.21.15.88:0/1887100837 >> [v2:172.21.15.88:6834/1912564918,v1:172.21.15.88:6836/1912564918] conn(0x5647fe899950 0x5647fe8bfbd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.602+0000 7fddf1a7e740 1 --2- 172.21.15.88:0/1887100837 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5647fe8b0720 0x5647fe857360 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.602+0000 7fddf1a7e740 1 --2- 172.21.15.88:0/1887100837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647fe859a50 0x5647fe91b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.602+0000 7fddc77fe640 0 client.10194 ms_handle_reset on v2:172.21.15.88:6834/1912564918 2024-09-16T01:44:14.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.602+0000 7fddf1a7e740 1 --2- 172.21.15.88:0/1887100837 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5647fe8582c0 0x5647fe877ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.602+0000 7fddf1a7e740 1 -- 172.21.15.88:0/1887100837 >> 172.21.15.88:0/1887100837 conn(0x5647fe855a80 msgr2=0x5647fe87ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:14.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.602+0000 7fddf1a7e740 1 -- 172.21.15.88:0/1887100837 shutdown_connections 2024-09-16T01:44:14.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.602+0000 7fddf1a7e740 1 -- 172.21.15.88:0/1887100837 wait complete. 2024-09-16T01:44:14.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.602+0000 7fddf1a7e740 20 client.10194 trim_cache size 0 max 16384 2024-09-16T01:44:14.616 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:44:14.616 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "5ace2c91-46f9-482b-8a89-731fbdc171bf", 2024-09-16T01:44:14.617 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2024-09-16T01:44:14.617 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 10134, 2024-09-16T01:44:14.617 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2024-09-16T01:44:14.617 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2024-09-16T01:44:14.617 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2024-09-16T01:44:14.617 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 6.6144309110000004, 2024-09-16T01:44:14.617 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 297, 2024-09-16T01:44:14.617 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 123, 2024-09-16T01:44:14.617 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 123, 2024-09-16T01:44:14.617 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 7.8771768890000002 2024-09-16T01:44:14.617 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:44:14.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.603+0000 7fddeece6640 1 -- 172.21.15.88:0/1655417571 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fddd406c2b0 msgr2=0x7fddd406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:14.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.603+0000 7fddeece6640 1 --2- 172.21.15.88:0/1655417571 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fddd406c2b0 0x7fddd406e770 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fdde8153c90 tx=0x7fdddc007730 comp rx=0 tx=0).stop 2024-09-16T01:44:14.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.603+0000 7fddeece6640 1 -- 172.21.15.88:0/1655417571 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdde80720d0 msgr2=0x7fdde8152f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:14.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.603+0000 7fddeece6640 1 --2- 172.21.15.88:0/1655417571 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdde80720d0 0x7fdde8152f30 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7fddd800b3f0 tx=0x7fddd800b8b0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.603+0000 7fddeece6640 1 -- 172.21.15.88:0/1655417571 shutdown_connections 2024-09-16T01:44:14.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.603+0000 7fddeece6640 1 --2- 172.21.15.88:0/1655417571 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fddd406c2b0 0x7fddd406e770 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.603+0000 7fddeece6640 1 --2- 172.21.15.88:0/1655417571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdde815ad10 0x7fdde813d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.603+0000 7fddeece6640 1 --2- 172.21.15.88:0/1655417571 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdde80cca40 0x7fdde80ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.603+0000 7fddeece6640 1 --2- 172.21.15.88:0/1655417571 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdde80720d0 0x7fdde8152f30 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.603+0000 7fddeece6640 1 -- 172.21.15.88:0/1655417571 >> 172.21.15.88:0/1655417571 conn(0x7fdde806d0f0 msgr2=0x7fdde815d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:14.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.603+0000 7fddeece6640 1 -- 172.21.15.88:0/1655417571 shutdown_connections 2024-09-16T01:44:14.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.603+0000 7fddeece6640 1 -- 172.21.15.88:0/1655417571 wait complete. 2024-09-16T01:44:14.620 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-09-16T01:44:14.620 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Corrupt omap header on mds0_openfiles.0', proceeding to ls 2024-09-16T01:44:14.620 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:44:14.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.717+0000 7f91aafb7640 1 Processor -- start 2024-09-16T01:44:14.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.718+0000 7f91aafb7640 1 -- start start 2024-09-16T01:44:14.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.718+0000 7f91aafb7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91a415e660 0x7f91a40cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.718+0000 7f91aafb7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f91a40d3310 0x7f91a40d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.718+0000 7f91aafb7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f91a40d1650 0x7f91a406bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.718+0000 7f91aafb7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f91a415ea40 con 0x7f91a40d3310 2024-09-16T01:44:14.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.718+0000 7f91aafb7640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f91a40d3870 con 0x7f91a40d1650 2024-09-16T01:44:14.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.718+0000 7f91aafb7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f91a40d3b50 con 0x7f91a415e660 2024-09-16T01:44:14.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.718+0000 7f91a3fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f91a40d3310 0x7f91a40d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.718+0000 7f91a3fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f91a40d3310 0x7f91a40d1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50472/0 (socket says 172.21.15.88:50472) 2024-09-16T01:44:14.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.718+0000 7f91a3fff640 1 -- 172.21.15.88:0/1271234456 learned_addr learned my addr 172.21.15.88:0/1271234456 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:14.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.719+0000 7f91a9554640 1 --2- 172.21.15.88:0/1271234456 >> v2:172.21.15.154:3300/0 conn(0x7f91a40d1650 0x7f91a406bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.719+0000 7f91a8d53640 1 --2- 172.21.15.88:0/1271234456 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91a415e660 0x7f91a40cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.719+0000 7f91a3fff640 1 -- 172.21.15.88:0/1271234456 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91a415e660 msgr2=0x7f91a40cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:14.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.719+0000 7f91a3fff640 1 --2- 172.21.15.88:0/1271234456 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91a415e660 0x7f91a40cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.719+0000 7f91a3fff640 1 -- 172.21.15.88:0/1271234456 >> v2:172.21.15.154:3300/0 conn(0x7f91a40d1650 msgr2=0x7f91a406bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:14.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.719+0000 7f91a3fff640 1 --2- 172.21.15.88:0/1271234456 >> v2:172.21.15.154:3300/0 conn(0x7f91a40d1650 0x7f91a406bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.719+0000 7f91a3fff640 1 -- 172.21.15.88:0/1271234456 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a40d3e30 con 0x7f91a40d3310 2024-09-16T01:44:14.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.719+0000 7f91a8d53640 1 --2- 172.21.15.88:0/1271234456 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91a415e660 0x7f91a40cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:14.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.719+0000 7f91a3fff640 1 --2- 172.21.15.88:0/1271234456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91a40d3310 0x7f91a40d1110 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f91a40cb760 tx=0x7f919802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=595ec42153c33874 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:14.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.720+0000 7f91a37fe640 1 -- 172.21.15.88:0/1271234456 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9198037070 con 0x7f91a40d3310 2024-09-16T01:44:14.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.720+0000 7f91a37fe640 1 -- 172.21.15.88:0/1271234456 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9198032440 con 0x7f91a40d3310 2024-09-16T01:44:14.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.720+0000 7f91a37fe640 1 -- 172.21.15.88:0/1271234456 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9198032810 con 0x7f91a40d3310 2024-09-16T01:44:14.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.720+0000 7f91aafb7640 1 -- 172.21.15.88:0/1271234456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91a40d3310 msgr2=0x7f91a40d1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:14.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.720+0000 7f91aafb7640 1 --2- 172.21.15.88:0/1271234456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91a40d3310 0x7f91a40d1110 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f91a40cb760 tx=0x7f919802a1e0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.720+0000 7f91aafb7640 1 -- 172.21.15.88:0/1271234456 shutdown_connections 2024-09-16T01:44:14.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.720+0000 7f91aafb7640 1 --2- 172.21.15.88:0/1271234456 >> v2:172.21.15.154:3300/0 conn(0x7f91a40d1650 0x7f91a406bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.720+0000 7f91aafb7640 1 --2- 172.21.15.88:0/1271234456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91a40d3310 0x7f91a40d1110 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.720+0000 7f91aafb7640 1 --2- 172.21.15.88:0/1271234456 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91a415e660 0x7f91a40cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.720+0000 7f91aafb7640 1 -- 172.21.15.88:0/1271234456 >> 172.21.15.88:0/1271234456 conn(0x7f91a40b6410 msgr2=0x7f91a40b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:14.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.720+0000 7f91aafb7640 1 -- 172.21.15.88:0/1271234456 shutdown_connections 2024-09-16T01:44:14.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.720+0000 7f91aafb7640 1 -- 172.21.15.88:0/1271234456 wait complete. 2024-09-16T01:44:14.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.721+0000 7f91aafb7640 1 Processor -- start 2024-09-16T01:44:14.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.721+0000 7f91aafb7640 1 -- start start 2024-09-16T01:44:14.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.722+0000 7f91aafb7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91a40d1650 0x7f91a4157810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.722+0000 7f91aafb7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91a40d3310 0x7f91a41f7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.722+0000 7f91aafb7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91a415e660 0x7f91a41f8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.722+0000 7f91aafb7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f91a40cfc70 con 0x7f91a415e660 2024-09-16T01:44:14.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.722+0000 7f91aafb7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f91a40d43f0 con 0x7f91a40d1650 2024-09-16T01:44:14.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.722+0000 7f91aafb7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f91a406c510 con 0x7f91a40d3310 2024-09-16T01:44:14.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.722+0000 7f91a9554640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91a415e660 0x7f91a41f8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.722+0000 7f91a9554640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91a415e660 0x7f91a41f8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50474/0 (socket says 172.21.15.88:50474) 2024-09-16T01:44:14.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.722+0000 7f91a9554640 1 -- 172.21.15.88:0/333406654 learned_addr learned my addr 172.21.15.88:0/333406654 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:14.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.723+0000 7f91a8d53640 1 --2- 172.21.15.88:0/333406654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91a40d1650 0x7f91a4157810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.723+0000 7f91a3fff640 1 --2- 172.21.15.88:0/333406654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91a40d3310 0x7f91a41f7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.723+0000 7f91a9554640 1 -- 172.21.15.88:0/333406654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91a40d3310 msgr2=0x7f91a41f7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:14.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.723+0000 7f91a9554640 1 --2- 172.21.15.88:0/333406654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91a40d3310 0x7f91a41f7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.723+0000 7f91a9554640 1 -- 172.21.15.88:0/333406654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91a40d1650 msgr2=0x7f91a4157810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:14.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.723+0000 7f91a9554640 1 --2- 172.21.15.88:0/333406654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91a40d1650 0x7f91a4157810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.723+0000 7f91a9554640 1 -- 172.21.15.88:0/333406654 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a41e9f70 con 0x7f91a415e660 2024-09-16T01:44:14.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.723+0000 7f91a9554640 1 --2- 172.21.15.88:0/333406654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91a415e660 0x7f91a41f8280 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f919c00e860 tx=0x7f919c00ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:14.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.723+0000 7f91a17fa640 1 -- 172.21.15.88:0/333406654 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919c017070 con 0x7f91a415e660 2024-09-16T01:44:14.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.723+0000 7f91a17fa640 1 -- 172.21.15.88:0/333406654 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f919c00c940 con 0x7f91a415e660 2024-09-16T01:44:14.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.723+0000 7f91a17fa640 1 -- 172.21.15.88:0/333406654 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919c00cd30 con 0x7f91a415e660 2024-09-16T01:44:14.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.723+0000 7f91aafb7640 1 -- 172.21.15.88:0/333406654 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91a41ea250 con 0x7f91a415e660 2024-09-16T01:44:14.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.723+0000 7f91aafb7640 1 -- 172.21.15.88:0/333406654 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91a41ea810 con 0x7f91a415e660 2024-09-16T01:44:14.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.724+0000 7f91aafb7640 1 -- 172.21.15.88:0/333406654 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91a40d1110 con 0x7f91a415e660 2024-09-16T01:44:14.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.725+0000 7f91a17fa640 1 -- 172.21.15.88:0/333406654 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f919c01a070 con 0x7f91a415e660 2024-09-16T01:44:14.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.725+0000 7f91a17fa640 1 --2- 172.21.15.88:0/333406654 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f919006c2b0 0x7f919006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:14.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.725+0000 7f91a8d53640 1 --2- 172.21.15.88:0/333406654 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f919006c2b0 0x7f919006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:14.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.725+0000 7f91a17fa640 1 -- 172.21.15.88:0/333406654 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8834+0+0 (secure 0 0 0) 0x7f919c08c350 con 0x7f91a415e660 2024-09-16T01:44:14.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.726+0000 7f91a8d53640 1 --2- 172.21.15.88:0/333406654 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f919006c2b0 0x7f919006e770 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f9194003fd0 tx=0x7f91940039f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:14.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.729+0000 7f91a17fa640 1 -- 172.21.15.88:0/333406654 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f919c059ec0 con 0x7f91a415e660 2024-09-16T01:44:14.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.928+0000 7f91aafb7640 1 -- 172.21.15.88:0/333406654 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f91a40d1e70 con 0x7f91a415e660 2024-09-16T01:44:14.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.928+0000 7f91a17fa640 1 -- 172.21.15.88:0/333406654 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v297) ==== 53+0+73 (secure 0 0 0) 0x7f919c0597c0 con 0x7f91a415e660 2024-09-16T01:44:14.931 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:44:14.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.930+0000 7f91aafb7640 1 -- 172.21.15.88:0/333406654 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f919006c2b0 msgr2=0x7f919006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:14.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.930+0000 7f91aafb7640 1 --2- 172.21.15.88:0/333406654 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f919006c2b0 0x7f919006e770 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f9194003fd0 tx=0x7f91940039f0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.930+0000 7f91aafb7640 1 -- 172.21.15.88:0/333406654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91a415e660 msgr2=0x7f91a41f8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:14.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.930+0000 7f91aafb7640 1 --2- 172.21.15.88:0/333406654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91a415e660 0x7f91a41f8280 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f919c00e860 tx=0x7f919c00ed20 comp rx=0 tx=0).stop 2024-09-16T01:44:14.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.930+0000 7f91aafb7640 1 -- 172.21.15.88:0/333406654 shutdown_connections 2024-09-16T01:44:14.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.930+0000 7f91aafb7640 1 --2- 172.21.15.88:0/333406654 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f919006c2b0 0x7f919006e770 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.930+0000 7f91aafb7640 1 --2- 172.21.15.88:0/333406654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f91a415e660 0x7f91a41f8280 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.930+0000 7f91aafb7640 1 --2- 172.21.15.88:0/333406654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f91a40d3310 0x7f91a41f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.930+0000 7f91aafb7640 1 --2- 172.21.15.88:0/333406654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91a40d1650 0x7f91a4157810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:14.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.930+0000 7f91aafb7640 1 -- 172.21.15.88:0/333406654 >> 172.21.15.88:0/333406654 conn(0x7f91a40b6410 msgr2=0x7f91a40b5550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:14.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.930+0000 7f91aafb7640 1 -- 172.21.15.88:0/333406654 shutdown_connections 2024-09-16T01:44:14.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:14.930+0000 7f91aafb7640 1 -- 172.21.15.88:0/333406654 wait complete. 2024-09-16T01:44:14.943 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-16T01:44:14.943 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:44:14.943 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2024-09-16T01:44:14.943 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T01:44:14.943 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T01:44:14.943 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-16T01:44:14.943 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-16T01:44:14.943 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:44:14.943 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:44:14.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:14.944 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2024-09-16T01:44:14.960 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:44:14.960 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:44:14.961 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:44:14.961 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:44:14.961 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-09-16T01:44:14.961 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:44:14.961 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:44:14.961 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:44:14.961 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2024-09-16T01:44:14.962 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:44:14.962 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:44:14.962 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2024-09-16T01:44:14.962 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:44:14.962 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2024-09-16T01:44:14.962 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:44:14.962 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2024-09-16T01:44:14.962 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:44:14.962 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42169sec preferred_lft 42169sec 2024-09-16T01:44:14.963 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link noprefixroute 2024-09-16T01:44:14.963 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:44:14.963 INFO:teuthology.orchestra.run.smithi088.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:44:14.963 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff 2024-09-16T01:44:14.963 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T01:44:14.963 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:44:14.963 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::e486:6eff:fe81:a1c/64 scope link 2024-09-16T01:44:14.963 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:44:14.963 INFO:teuthology.orchestra.run.smithi088.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:44:14.963 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 12:6f:29:be:95:48 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:44:14.964 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::106f:29ff:febe:9548/64 scope link 2024-09-16T01:44:14.964 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:44:14.964 INFO:teuthology.orchestra.run.smithi088.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:44:14.964 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T01:44:14.964 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::c63:b4ff:feb3:ad9e/64 scope link 2024-09-16T01:44:14.964 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:44:14.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:14.964 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:44:15.023 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-09-16T01:44:15.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:15.025 DEBUG:teuthology.orchestra.run.smithi088:> route 2024-09-16T01:44:15.081 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2024-09-16T01:44:15.081 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:44:15.081 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:44:15.081 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:44:15.081 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:44:15.082 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:44:15.082 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:44:15.082 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:44:15.082 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:44:15.082 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:44:15.082 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:44:15.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:44:15.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:44:15.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:15.207 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:44:15.222 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T01:44:15.223 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T01:44:15.223 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:44:15.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:15.223 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2024-09-16T01:44:15.288 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:44:15.288 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T01:44:15.343 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:44:15.343 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:44:15.343 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:44:15.398 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-16T01:44:15.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:15.398 DEBUG:teuthology.orchestra.run.smithi088:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:44:15.481 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:44:15.481 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:44:15.482 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:44:15.490 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:44:15.490 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:44:15.491 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:44:15.491 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:44:15.491 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:44:15.496 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:44:15.496 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:44:15.496 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:44:15.496 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:44:15.496 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:44:15.496 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790" 2024-09-16T01:44:15.496 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2024-09-16T01:44:15.496 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5ace2c91-46f9-482b-8a89-731fbdc171bf" will pass to kernel 2024-09-16T01:44:15.497 INFO:teuthology.orchestra.run.smithi088.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-16T01:44:15.497 INFO:teuthology.orchestra.run.smithi088.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:44:15.497 INFO:teuthology.orchestra.run.smithi088.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:44:15.497 INFO:teuthology.orchestra.run.smithi088.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:44:15.497 INFO:teuthology.orchestra.run.smithi088.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:44:15.498 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:44:15.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:15.499 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:44:15.563 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:44:15.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:44:15.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:44:15.657 INFO:teuthology.orchestra.run.smithi088.stdout:client10203 2024-09-16T01:44:15.658 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:44:15.658 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:44:15.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:15.673 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R .) 2024-09-16T01:44:15.716 INFO:teuthology.orchestra.run:waiting for 20 2024-09-16T01:44:15.731 INFO:teuthology.orchestra.run.smithi088.stdout:.: 2024-09-16T01:44:15.731 INFO:teuthology.orchestra.run.smithi088.stdout:subdir 2024-09-16T01:44:15.731 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:15.731 INFO:teuthology.orchestra.run.smithi088.stdout:./subdir: 2024-09-16T01:44:15.731 INFO:teuthology.orchestra.run.smithi088.stdout:sixmegs 2024-09-16T01:44:15.732 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt omap header on mds0_openfiles.0' did not caused DAMAGED state 2024-09-16T01:44:15.732 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap header on 100.00000000' 2024-09-16T01:44:15.732 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:44:15.732 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:44:15.787 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:44:15.787 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:44:15.842 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:44:15.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.981+0000 7ff733d2c640 1 Processor -- start 2024-09-16T01:44:15.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.981+0000 7ff733d2c640 1 -- start start 2024-09-16T01:44:15.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.982+0000 7ff733d2c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff72c15b480 0x7ff72c0b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:15.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.982+0000 7ff733d2c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff72c0b60c0 0x7ff72c071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:15.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.982+0000 7ff733d2c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff72c072530 0x7ff72c070fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:15.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.982+0000 7ff733d2c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff72c0729a0 con 0x7ff72c15b480 2024-09-16T01:44:15.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.982+0000 7ff733d2c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff72c0ccff0 con 0x7ff72c0b60c0 2024-09-16T01:44:15.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.982+0000 7ff733d2c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff72c0cd2d0 con 0x7ff72c072530 2024-09-16T01:44:15.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.982+0000 7ff731ac8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff72c15b480 0x7ff72c0b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:15.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.982+0000 7ff731ac8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff72c15b480 0x7ff72c0b5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50480/0 (socket says 172.21.15.88:50480) 2024-09-16T01:44:15.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.982+0000 7ff731ac8640 1 -- 172.21.15.88:0/1326772940 learned_addr learned my addr 172.21.15.88:0/1326772940 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:15.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.982+0000 7ff7312c7640 1 --2- 172.21.15.88:0/1326772940 >> v2:172.21.15.154:3300/0 conn(0x7ff72c0b60c0 0x7ff72c071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:15.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.982+0000 7ff7322c9640 1 --2- 172.21.15.88:0/1326772940 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff72c072530 0x7ff72c070fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:15.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.982+0000 7ff731ac8640 1 -- 172.21.15.88:0/1326772940 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff72c072530 msgr2=0x7ff72c070fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:15.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.982+0000 7ff731ac8640 1 --2- 172.21.15.88:0/1326772940 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff72c072530 0x7ff72c070fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:15.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.982+0000 7ff731ac8640 1 -- 172.21.15.88:0/1326772940 >> v2:172.21.15.154:3300/0 conn(0x7ff72c0b60c0 msgr2=0x7ff72c071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:15.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.982+0000 7ff731ac8640 1 --2- 172.21.15.88:0/1326772940 >> v2:172.21.15.154:3300/0 conn(0x7ff72c0b60c0 0x7ff72c071ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:15.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.982+0000 7ff731ac8640 1 -- 172.21.15.88:0/1326772940 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff72c0cd5b0 con 0x7ff72c15b480 2024-09-16T01:44:15.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.983+0000 7ff731ac8640 1 --2- 172.21.15.88:0/1326772940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff72c15b480 0x7ff72c0b5b80 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7ff72c0b2ee0 tx=0x7ff72802c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46b9a2016f90e814 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:15.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.983+0000 7ff730ac6640 1 -- 172.21.15.88:0/1326772940 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff728032790 con 0x7ff72c15b480 2024-09-16T01:44:15.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.983+0000 7ff730ac6640 1 -- 172.21.15.88:0/1326772940 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff728038070 con 0x7ff72c15b480 2024-09-16T01:44:15.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.983+0000 7ff730ac6640 1 -- 172.21.15.88:0/1326772940 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff728032b50 con 0x7ff72c15b480 2024-09-16T01:44:15.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.984+0000 7ff733d2c640 1 -- 172.21.15.88:0/1326772940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff72c15b480 msgr2=0x7ff72c0b5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:15.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.984+0000 7ff733d2c640 1 --2- 172.21.15.88:0/1326772940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff72c15b480 0x7ff72c0b5b80 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7ff72c0b2ee0 tx=0x7ff72802c890 comp rx=0 tx=0).stop 2024-09-16T01:44:15.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.984+0000 7ff733d2c640 1 -- 172.21.15.88:0/1326772940 shutdown_connections 2024-09-16T01:44:15.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.984+0000 7ff733d2c640 1 --2- 172.21.15.88:0/1326772940 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff72c072530 0x7ff72c070fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:15.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.984+0000 7ff733d2c640 1 --2- 172.21.15.88:0/1326772940 >> v2:172.21.15.154:3300/0 conn(0x7ff72c0b60c0 0x7ff72c071ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:15.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.984+0000 7ff733d2c640 1 --2- 172.21.15.88:0/1326772940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff72c15b480 0x7ff72c0b5b80 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:15.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.984+0000 7ff733d2c640 1 -- 172.21.15.88:0/1326772940 >> 172.21.15.88:0/1326772940 conn(0x7ff72c06d0f0 msgr2=0x7ff72c06d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:15.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.984+0000 7ff733d2c640 1 -- 172.21.15.88:0/1326772940 shutdown_connections 2024-09-16T01:44:15.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.984+0000 7ff733d2c640 1 -- 172.21.15.88:0/1326772940 wait complete. 2024-09-16T01:44:15.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.985+0000 7ff733d2c640 1 Processor -- start 2024-09-16T01:44:15.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.985+0000 7ff733d2c640 1 -- start start 2024-09-16T01:44:15.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.986+0000 7ff733d2c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff72c072530 0x7ff72c152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:15.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.986+0000 7ff733d2c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff72c0b60c0 0x7ff72c14cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:15.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.986+0000 7ff733d2c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff72c15b480 0x7ff72c1502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:15.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.986+0000 7ff733d2c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff72c15b860 con 0x7ff72c15b480 2024-09-16T01:44:15.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.986+0000 7ff733d2c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff72c0cdb70 con 0x7ff72c072530 2024-09-16T01:44:15.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.986+0000 7ff733d2c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff72c15b9e0 con 0x7ff72c0b60c0 2024-09-16T01:44:15.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.986+0000 7ff7322c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff72c15b480 0x7ff72c1502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:15.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.986+0000 7ff7322c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff72c15b480 0x7ff72c1502a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50486/0 (socket says 172.21.15.88:50486) 2024-09-16T01:44:15.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.986+0000 7ff7322c9640 1 -- 172.21.15.88:0/65122187 learned_addr learned my addr 172.21.15.88:0/65122187 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:15.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.986+0000 7ff7312c7640 1 --2- 172.21.15.88:0/65122187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff72c0b60c0 0x7ff72c14cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:15.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.986+0000 7ff731ac8640 1 --2- 172.21.15.88:0/65122187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff72c072530 0x7ff72c152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:15.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.986+0000 7ff7322c9640 1 -- 172.21.15.88:0/65122187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff72c0b60c0 msgr2=0x7ff72c14cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:15.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.986+0000 7ff7322c9640 1 --2- 172.21.15.88:0/65122187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff72c0b60c0 0x7ff72c14cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:15.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.986+0000 7ff7322c9640 1 -- 172.21.15.88:0/65122187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff72c072530 msgr2=0x7ff72c152440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:15.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.986+0000 7ff7322c9640 1 --2- 172.21.15.88:0/65122187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff72c072530 0x7ff72c152440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:15.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.986+0000 7ff7322c9640 1 -- 172.21.15.88:0/65122187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff72c0d0db0 con 0x7ff72c15b480 2024-09-16T01:44:15.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.986+0000 7ff7322c9640 1 --2- 172.21.15.88:0/65122187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff72c15b480 0x7ff72c1502a0 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7ff724007e60 tx=0x7ff72400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:15.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.986+0000 7ff7227fc640 1 -- 172.21.15.88:0/65122187 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff724014620 con 0x7ff72c15b480 2024-09-16T01:44:15.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.986+0000 7ff7227fc640 1 -- 172.21.15.88:0/65122187 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff724019070 con 0x7ff72c15b480 2024-09-16T01:44:15.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.986+0000 7ff7227fc640 1 -- 172.21.15.88:0/65122187 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff724014a20 con 0x7ff72c15b480 2024-09-16T01:44:15.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.986+0000 7ff733d2c640 1 -- 172.21.15.88:0/65122187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff72c0ce380 con 0x7ff72c15b480 2024-09-16T01:44:15.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.987+0000 7ff733d2c640 1 -- 172.21.15.88:0/65122187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff72c0cebe0 con 0x7ff72c15b480 2024-09-16T01:44:15.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.987+0000 7ff733d2c640 1 -- 172.21.15.88:0/65122187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff72c0cede0 con 0x7ff72c15b480 2024-09-16T01:44:15.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.988+0000 7ff7227fc640 1 -- 172.21.15.88:0/65122187 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff724014bc0 con 0x7ff72c15b480 2024-09-16T01:44:15.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.988+0000 7ff7227fc640 1 --2- 172.21.15.88:0/65122187 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff71006c2b0 0x7ff71006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:15.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.988+0000 7ff7227fc640 1 -- 172.21.15.88:0/65122187 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8834+0+0 (secure 0 0 0) 0x7ff72408c7f0 con 0x7ff72c15b480 2024-09-16T01:44:15.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.988+0000 7ff731ac8640 1 --2- 172.21.15.88:0/65122187 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff71006c2b0 0x7ff71006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:15.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.989+0000 7ff731ac8640 1 --2- 172.21.15.88:0/65122187 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff71006c2b0 0x7ff71006e770 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7ff728037000 tx=0x7ff728032000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:15.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:15.993+0000 7ff7227fc640 1 -- 172.21.15.88:0/65122187 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff72c0cede0 con 0x7ff72c15b480 2024-09-16T01:44:16.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.173+0000 7ff733d2c640 1 -- 172.21.15.88:0/65122187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff72c071ff0 con 0x7ff72c15b480 2024-09-16T01:44:16.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.173+0000 7ff7227fc640 1 -- 172.21.15.88:0/65122187 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 72 entries v123) ==== 81+0+4380 (secure 0 0 0) 0x7ff72c0d0db0 con 0x7ff72c15b480 2024-09-16T01:44:16.175 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/3078464816 2024-09-17T01:44:01.948759+0000 2024-09-16T01:44:16.175 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/3892583353 2024-09-17T01:43:56.641946+0000 2024-09-16T01:44:16.175 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/4135629195 2024-09-17T01:43:45.866139+0000 2024-09-16T01:44:16.175 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/3892583353 2024-09-17T01:43:56.641946+0000 2024-09-16T01:44:16.175 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/4135629195 2024-09-17T01:43:45.866139+0000 2024-09-16T01:44:16.175 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/3121519215 2024-09-17T01:43:30.669208+0000 2024-09-16T01:44:16.175 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/3121519215 2024-09-17T01:43:30.669208+0000 2024-09-16T01:44:16.175 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2555776364 2024-09-17T01:43:25.635976+0000 2024-09-16T01:44:16.175 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/1372908065 2024-09-17T01:43:12.443265+0000 2024-09-16T01:44:16.175 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/1372908065 2024-09-17T01:43:12.443265+0000 2024-09-16T01:44:16.175 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/650399848 2024-09-17T01:43:08.279879+0000 2024-09-16T01:44:16.175 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/1255135754 2024-09-17T01:41:41.905258+0000 2024-09-16T01:44:16.176 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/3728363359 2024-09-17T01:42:05.412133+0000 2024-09-16T01:44:16.176 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/293764774 2024-09-17T01:41:16.203758+0000 2024-09-16T01:44:16.176 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/1255135754 2024-09-17T01:41:41.905258+0000 2024-09-16T01:44:16.176 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2918194823 2024-09-17T01:41:29.275696+0000 2024-09-16T01:44:16.176 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/3336935653 2024-09-17T01:41:44.295023+0000 2024-09-16T01:44:16.176 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/981209627 2024-09-17T01:41:51.268495+0000 2024-09-16T01:44:16.176 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/293764774 2024-09-17T01:41:16.203758+0000 2024-09-16T01:44:16.176 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2680708900 2024-09-17T01:42:41.629904+0000 2024-09-16T01:44:16.176 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2957965650 2024-09-17T01:41:12.181485+0000 2024-09-16T01:44:16.176 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/3336935653 2024-09-17T01:41:44.295023+0000 2024-09-16T01:44:16.176 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2285699282 2024-09-17T01:42:29.541113+0000 2024-09-16T01:44:16.176 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/1162318664 2024-09-17T01:41:23.257870+0000 2024-09-16T01:44:16.176 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/128058689 2024-09-17T01:41:10.163743+0000 2024-09-16T01:44:16.176 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/981209627 2024-09-17T01:41:51.268495+0000 2024-09-16T01:44:16.177 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/128058689 2024-09-17T01:41:10.163743+0000 2024-09-16T01:44:16.177 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/1042715824 2024-09-17T01:40:59.080515+0000 2024-09-16T01:44:16.177 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2957965650 2024-09-17T01:41:12.181485+0000 2024-09-16T01:44:16.177 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2970185561 2024-09-17T01:40:38.255774+0000 2024-09-16T01:44:16.177 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/3078464816 2024-09-17T01:44:01.948759+0000 2024-09-16T01:44:16.177 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/1162318664 2024-09-17T01:41:23.257870+0000 2024-09-16T01:44:16.177 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/1289932173 2024-09-17T01:40:42.399399+0000 2024-09-16T01:44:16.177 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/3356432718 2024-09-17T01:40:32.344252+0000 2024-09-16T01:44:16.177 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2555776364 2024-09-17T01:43:25.635976+0000 2024-09-16T01:44:16.177 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2970185561 2024-09-17T01:40:38.255774+0000 2024-09-16T01:44:16.177 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2733458327 2024-09-17T01:40:27.314712+0000 2024-09-16T01:44:16.177 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/3356432718 2024-09-17T01:40:32.344252+0000 2024-09-16T01:44:16.177 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2918194823 2024-09-17T01:41:29.275696+0000 2024-09-16T01:44:16.178 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/4144876777 2024-09-17T01:43:00.350808+0000 2024-09-16T01:44:16.178 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2408579939 2024-09-17T01:40:46.434492+0000 2024-09-16T01:44:16.178 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/3163343083 2024-09-17T01:42:01.375396+0000 2024-09-16T01:44:16.178 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/4222250252 2024-09-17T01:42:45.673956+0000 2024-09-16T01:44:16.178 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/4162274767 2024-09-17T01:40:11.522091+0000 2024-09-16T01:44:16.178 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2477051134 2024-09-17T01:43:04.380783+0000 2024-09-16T01:44:16.178 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/1289932173 2024-09-17T01:40:42.399399+0000 2024-09-16T01:44:16.178 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2733458327 2024-09-17T01:40:27.314712+0000 2024-09-16T01:44:16.178 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2550498835 2024-09-17T01:41:57.358287+0000 2024-09-16T01:44:16.178 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/2550498835 2024-09-17T01:41:57.358287+0000 2024-09-16T01:44:16.178 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/2285699282 2024-09-17T01:42:29.541113+0000 2024-09-16T01:44:16.178 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/3163343083 2024-09-17T01:42:01.375396+0000 2024-09-16T01:44:16.178 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/3728363359 2024-09-17T01:42:05.412133+0000 2024-09-16T01:44:16.178 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/880314974 2024-09-17T01:42:21.485748+0000 2024-09-16T01:44:16.179 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/1936493048 2024-09-17T01:42:19.133197+0000 2024-09-16T01:44:16.179 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/1936493048 2024-09-17T01:42:19.133197+0000 2024-09-16T01:44:16.179 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/880314974 2024-09-17T01:42:21.485748+0000 2024-09-16T01:44:16.179 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/3374372758 2024-09-17T01:42:25.506299+0000 2024-09-16T01:44:16.179 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/3374372758 2024-09-17T01:42:25.506299+0000 2024-09-16T01:44:16.179 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/201188184 2024-09-17T01:42:33.276104+0000 2024-09-16T01:44:16.179 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/1990650099 2024-09-17T01:42:37.599896+0000 2024-09-16T01:44:16.179 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/201188184 2024-09-17T01:42:33.276104+0000 2024-09-16T01:44:16.179 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6835/1990650099 2024-09-17T01:42:37.599896+0000 2024-09-16T01:44:16.179 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/4162274767 2024-09-17T01:40:11.522091+0000 2024-09-16T01:44:16.179 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2408579939 2024-09-17T01:40:46.434492+0000 2024-09-16T01:44:16.180 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/2680708900 2024-09-17T01:42:41.629904+0000 2024-09-16T01:44:16.180 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/4222250252 2024-09-17T01:42:45.673956+0000 2024-09-16T01:44:16.180 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/1042715824 2024-09-17T01:40:59.080515+0000 2024-09-16T01:44:16.180 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/4065900771 2024-09-17T01:42:58.329389+0000 2024-09-16T01:44:16.180 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/4065900771 2024-09-17T01:42:58.329389+0000 2024-09-16T01:44:16.180 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/4144876777 2024-09-17T01:43:00.350808+0000 2024-09-16T01:44:16.180 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6837/2477051134 2024-09-17T01:43:04.380783+0000 2024-09-16T01:44:16.180 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/650399848 2024-09-17T01:43:08.279879+0000 2024-09-16T01:44:16.181 INFO:teuthology.orchestra.run.smithi088.stderr:listed 72 entries 2024-09-16T01:44:16.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.175+0000 7ff733d2c640 1 -- 172.21.15.88:0/65122187 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff71006c2b0 msgr2=0x7ff71006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:16.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.175+0000 7ff733d2c640 1 --2- 172.21.15.88:0/65122187 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff71006c2b0 0x7ff71006e770 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7ff728037000 tx=0x7ff728032000 comp rx=0 tx=0).stop 2024-09-16T01:44:16.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.175+0000 7ff733d2c640 1 -- 172.21.15.88:0/65122187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff72c15b480 msgr2=0x7ff72c1502a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:16.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.175+0000 7ff733d2c640 1 --2- 172.21.15.88:0/65122187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff72c15b480 0x7ff72c1502a0 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7ff724007e60 tx=0x7ff72400e4b0 comp rx=0 tx=0).stop 2024-09-16T01:44:16.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.175+0000 7ff733d2c640 1 -- 172.21.15.88:0/65122187 shutdown_connections 2024-09-16T01:44:16.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.175+0000 7ff733d2c640 1 --2- 172.21.15.88:0/65122187 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff71006c2b0 0x7ff71006e770 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:16.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.175+0000 7ff733d2c640 1 --2- 172.21.15.88:0/65122187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff72c15b480 0x7ff72c1502a0 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:16.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.175+0000 7ff733d2c640 1 --2- 172.21.15.88:0/65122187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff72c0b60c0 0x7ff72c14cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:16.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.175+0000 7ff733d2c640 1 --2- 172.21.15.88:0/65122187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff72c072530 0x7ff72c152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:16.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.175+0000 7ff733d2c640 1 -- 172.21.15.88:0/65122187 >> 172.21.15.88:0/65122187 conn(0x7ff72c06d0f0 msgr2=0x7ff72c0b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:16.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.175+0000 7ff733d2c640 1 -- 172.21.15.88:0/65122187 shutdown_connections 2024-09-16T01:44:16.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.175+0000 7ff733d2c640 1 -- 172.21.15.88:0/65122187 wait complete. 2024-09-16T01:44:16.187 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-16T01:44:16.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:16.188 DEBUG:teuthology.orchestra.run.smithi088:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-09-16T01:44:16.229 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:44:16.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:16.229 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:44:16.285 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:44:16.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.422+0000 7f704421e640 1 Processor -- start 2024-09-16T01:44:16.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.423+0000 7f704421e640 1 -- start start 2024-09-16T01:44:16.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.423+0000 7f704421e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f703c0b68f0 0x7f703c1696f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:16.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.423+0000 7f704421e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f703c0cf860 con 0x7f703c15e7a0 2024-09-16T01:44:16.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.423+0000 7f704421e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f703c0cfb40 con 0x7f703c0cf1f0 2024-09-16T01:44:16.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.423+0000 7f704421e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f703c0d2ca0 con 0x7f703c0b68f0 2024-09-16T01:44:16.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.423+0000 7f7041fba640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f703c15e7a0 0x7f703c06caa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:37332/0 (socket says 172.21.15.88:37332) 2024-09-16T01:44:16.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.423+0000 7f7041fba640 1 -- 172.21.15.88:0/2723539954 learned_addr learned my addr 172.21.15.88:0/2723539954 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:16.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.423+0000 7f70427bb640 1 --2- 172.21.15.88:0/2723539954 >> v2:172.21.15.88:3300/0 conn(0x7f703c0b68f0 0x7f703c1696f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:16.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.424+0000 7f7040fb8640 1 -- 172.21.15.88:0/2723539954 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2971676719 0 0) 0x7f703c0cf860 con 0x7f703c15e7a0 2024-09-16T01:44:16.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.424+0000 7f7040fb8640 1 -- 172.21.15.88:0/2723539954 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7028003520 con 0x7f703c15e7a0 2024-09-16T01:44:16.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.424+0000 7f70427bb640 1 -- 172.21.15.88:0/2723539954 >> v1:172.21.15.154:6789/0 conn(0x7f703c0cf1f0 legacy=0x7f703c0b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:16.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.424+0000 7f70427bb640 1 -- 172.21.15.88:0/2723539954 >> v1:172.21.15.88:6789/0 conn(0x7f703c15e7a0 legacy=0x7f703c06caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:16.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.424+0000 7f70427bb640 1 -- 172.21.15.88:0/2723539954 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f703c0cfe20 con 0x7f703c0b68f0 2024-09-16T01:44:16.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.424+0000 7f70427bb640 1 --2- 172.21.15.88:0/2723539954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f703c0b68f0 0x7f703c1696f0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7f70340097a0 tx=0x7f703402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a09ba28c1149047 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:16.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.425+0000 7f7040fb8640 1 -- 172.21.15.88:0/2723539954 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7034037070 con 0x7f703c0b68f0 2024-09-16T01:44:16.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.425+0000 7f7040fb8640 1 -- 172.21.15.88:0/2723539954 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7034032440 con 0x7f703c0b68f0 2024-09-16T01:44:16.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.425+0000 7f7040fb8640 1 -- 172.21.15.88:0/2723539954 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7034032810 con 0x7f703c0b68f0 2024-09-16T01:44:16.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.425+0000 7f704421e640 1 -- 172.21.15.88:0/2723539954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f703c0b68f0 msgr2=0x7f703c1696f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:16.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.425+0000 7f704421e640 1 --2- 172.21.15.88:0/2723539954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f703c0b68f0 0x7f703c1696f0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7f70340097a0 tx=0x7f703402a1a0 comp rx=0 tx=0).stop 2024-09-16T01:44:16.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.425+0000 7f704421e640 1 -- 172.21.15.88:0/2723539954 shutdown_connections 2024-09-16T01:44:16.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.425+0000 7f704421e640 1 --2- 172.21.15.88:0/2723539954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f703c0b68f0 0x7f703c1696f0 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:16.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.425+0000 7f704421e640 1 -- 172.21.15.88:0/2723539954 >> 172.21.15.88:0/2723539954 conn(0x7f703c0d11e0 msgr2=0x7f703c0d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:16.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.425+0000 7f704421e640 1 -- 172.21.15.88:0/2723539954 shutdown_connections 2024-09-16T01:44:16.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.425+0000 7f704421e640 1 -- 172.21.15.88:0/2723539954 wait complete. 2024-09-16T01:44:16.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.426+0000 7f704421e640 1 Processor -- start 2024-09-16T01:44:16.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.426+0000 7f704421e640 1 -- start start 2024-09-16T01:44:16.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.427+0000 7f704421e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f703c0b68f0 0x7f703c0b9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:16.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.427+0000 7f704421e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f703c0cf1f0 0x7f703c1512a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:16.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.427+0000 7f704421e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f703c15e7a0 0x7f703c157270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:16.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.427+0000 7f704421e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f703c0d4420 con 0x7f703c15e7a0 2024-09-16T01:44:16.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.427+0000 7f704421e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f703c0b7a00 con 0x7f703c0cf1f0 2024-09-16T01:44:16.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.427+0000 7f704421e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f703c0cd510 con 0x7f703c0b68f0 2024-09-16T01:44:16.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.427+0000 7f70427bb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f703c15e7a0 0x7f703c157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:16.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.427+0000 7f70427bb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f703c15e7a0 0x7f703c157270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50502/0 (socket says 172.21.15.88:50502) 2024-09-16T01:44:16.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.427+0000 7f70427bb640 1 -- 172.21.15.88:0/2499625271 learned_addr learned my addr 172.21.15.88:0/2499625271 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:16.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.427+0000 7f70417b9640 1 --2- 172.21.15.88:0/2499625271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f703c0cf1f0 0x7f703c1512a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:16.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.427+0000 7f7041fba640 1 --2- 172.21.15.88:0/2499625271 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f703c0b68f0 0x7f703c0b9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:16.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.427+0000 7f70427bb640 1 -- 172.21.15.88:0/2499625271 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f703c0b68f0 msgr2=0x7f703c0b9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:16.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.427+0000 7f70427bb640 1 --2- 172.21.15.88:0/2499625271 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f703c0b68f0 0x7f703c0b9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:16.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.428+0000 7f70427bb640 1 -- 172.21.15.88:0/2499625271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f703c0cf1f0 msgr2=0x7f703c1512a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:16.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.428+0000 7f70427bb640 1 --2- 172.21.15.88:0/2499625271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f703c0cf1f0 0x7f703c1512a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:16.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.428+0000 7f70427bb640 1 -- 172.21.15.88:0/2499625271 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f703c0cd7f0 con 0x7f703c15e7a0 2024-09-16T01:44:16.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.428+0000 7f7041fba640 1 --2- 172.21.15.88:0/2499625271 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f703c0b68f0 0x7f703c0b9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:16.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.428+0000 7f70427bb640 1 --2- 172.21.15.88:0/2499625271 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f703c15e7a0 0x7f703c157270 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f7034035040 tx=0x7f70340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:16.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.428+0000 7f70327fc640 1 -- 172.21.15.88:0/2499625271 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7034037040 con 0x7f703c15e7a0 2024-09-16T01:44:16.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.428+0000 7f70327fc640 1 -- 172.21.15.88:0/2499625271 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7034042070 con 0x7f703c15e7a0 2024-09-16T01:44:16.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.428+0000 7f704421e640 1 -- 172.21.15.88:0/2499625271 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f703c0cdad0 con 0x7f703c15e7a0 2024-09-16T01:44:16.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.428+0000 7f70327fc640 1 -- 172.21.15.88:0/2499625271 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70340308a0 con 0x7f703c15e7a0 2024-09-16T01:44:16.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.428+0000 7f704421e640 1 -- 172.21.15.88:0/2499625271 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f703c1517e0 con 0x7f703c15e7a0 2024-09-16T01:44:16.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.429+0000 7f704421e640 1 -- 172.21.15.88:0/2499625271 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f703c0cdff0 con 0x7f703c15e7a0 2024-09-16T01:44:16.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.429+0000 7f70327fc640 1 -- 172.21.15.88:0/2499625271 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7034030a40 con 0x7f703c15e7a0 2024-09-16T01:44:16.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.430+0000 7f70327fc640 1 --2- 172.21.15.88:0/2499625271 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f702006c2b0 0x7f702006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:16.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.430+0000 7f70327fc640 1 -- 172.21.15.88:0/2499625271 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 8834+0+0 (secure 0 0 0) 0x7f70340ac9a0 con 0x7f703c15e7a0 2024-09-16T01:44:16.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.430+0000 7f7041fba640 1 --2- 172.21.15.88:0/2499625271 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f702006c2b0 0x7f702006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:16.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.430+0000 7f7041fba640 1 --2- 172.21.15.88:0/2499625271 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f702006c2b0 0x7f702006e770 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f7038001560 tx=0x7f7038009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:16.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.434+0000 7f70327fc640 1 -- 172.21.15.88:0/2499625271 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f703c0cdff0 con 0x7f703c15e7a0 2024-09-16T01:44:16.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.633+0000 7f704421e640 1 -- 172.21.15.88:0/2499625271 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f703c15eb80 con 0x7f703c15e7a0 2024-09-16T01:44:16.635 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:44:16.634+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:44:16.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.635+0000 7f70327fc640 1 -- 172.21.15.88:0/2499625271 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v298) ==== 161+0+0 (secure 0 0 0) 0x7f703407a510 con 0x7f703c15e7a0 2024-09-16T01:44:16.637 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:44:16.639 INFO:tasks.ceph.mds.a.smithi088.stderr: -1775> 2024-09-16T01:44:07.991+0000 7f4be32db640 -1 mds.0.openfiles _load_finish: corrupted header/values: End of buffer [buffer:2] 2024-09-16T01:44:16.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.638+0000 7f704421e640 1 -- 172.21.15.88:0/2499625271 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f702006c2b0 msgr2=0x7f702006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:16.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.638+0000 7f704421e640 1 --2- 172.21.15.88:0/2499625271 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f702006c2b0 0x7f702006e770 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f7038001560 tx=0x7f7038009040 comp rx=0 tx=0).stop 2024-09-16T01:44:16.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.638+0000 7f704421e640 1 -- 172.21.15.88:0/2499625271 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f703c15e7a0 msgr2=0x7f703c157270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:16.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.638+0000 7f704421e640 1 --2- 172.21.15.88:0/2499625271 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f703c15e7a0 0x7f703c157270 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f7034035040 tx=0x7f70340046c0 comp rx=0 tx=0).stop 2024-09-16T01:44:16.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.638+0000 7f704421e640 1 -- 172.21.15.88:0/2499625271 shutdown_connections 2024-09-16T01:44:16.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.638+0000 7f704421e640 1 --2- 172.21.15.88:0/2499625271 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f702006c2b0 0x7f702006e770 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:16.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.638+0000 7f704421e640 1 --2- 172.21.15.88:0/2499625271 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f703c15e7a0 0x7f703c157270 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:16.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.638+0000 7f704421e640 1 --2- 172.21.15.88:0/2499625271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f703c0cf1f0 0x7f703c1512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:16.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.638+0000 7f704421e640 1 --2- 172.21.15.88:0/2499625271 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f703c0b68f0 0x7f703c0b9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:16.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.638+0000 7f704421e640 1 -- 172.21.15.88:0/2499625271 >> 172.21.15.88:0/2499625271 conn(0x7f703c0d11e0 msgr2=0x7f703c0b41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:16.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.638+0000 7f704421e640 1 -- 172.21.15.88:0/2499625271 shutdown_connections 2024-09-16T01:44:16.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.638+0000 7f704421e640 1 -- 172.21.15.88:0/2499625271 wait complete. 2024-09-16T01:44:16.641 INFO:tasks.ceph.mds.a.smithi088.stderr: -647> 2024-09-16T01:44:11.975+0000 7f4be72e3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:44:16.652 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:44:16.667 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:44:16.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.750+0000 7fc26a76f640 1 Processor -- start 2024-09-16T01:44:16.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.750+0000 7fc26a76f640 1 -- start start 2024-09-16T01:44:16.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.750+0000 7fc26a76f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc26415c710 0x7fc264073970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:16.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.750+0000 7fc26a76f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2640b53f0 0x7fc2640d2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:16.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.751+0000 7fc26a76f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2640cdf50 con 0x7fc264073eb0 2024-09-16T01:44:16.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.751+0000 7fc26a76f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc2640ce230 con 0x7fc26415c710 2024-09-16T01:44:16.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.751+0000 7fc26a76f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc2640ce510 con 0x7fc2640b53f0 2024-09-16T01:44:16.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.751+0000 7fc25bfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc264073eb0 0x7fc2640d2110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:37346/0 (socket says 172.21.15.88:37346) 2024-09-16T01:44:16.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.751+0000 7fc25bfff640 1 -- 172.21.15.88:0/2751579357 learned_addr learned my addr 172.21.15.88:0/2751579357 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:16.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.751+0000 7fc263fff640 1 --2- 172.21.15.88:0/2751579357 >> v2:172.21.15.154:3300/0 conn(0x7fc26415c710 0x7fc264073970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:16.755 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.751+0000 7fc268d0c640 1 --2- 172.21.15.88:0/2751579357 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2640b53f0 0x7fc2640d2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:16.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.751+0000 7fc2637fe640 1 -- 172.21.15.88:0/2751579357 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2985518980 0 0) 0x7fc2640cdf50 con 0x7fc264073eb0 2024-09-16T01:44:16.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.751+0000 7fc2637fe640 1 -- 172.21.15.88:0/2751579357 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc250003520 con 0x7fc264073eb0 2024-09-16T01:44:16.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.751+0000 7fc263fff640 1 -- 172.21.15.88:0/2751579357 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2640b53f0 msgr2=0x7fc2640d2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:16.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.751+0000 7fc263fff640 1 --2- 172.21.15.88:0/2751579357 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2640b53f0 0x7fc2640d2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:16.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.751+0000 7fc263fff640 1 -- 172.21.15.88:0/2751579357 >> v1:172.21.15.88:6789/0 conn(0x7fc264073eb0 legacy=0x7fc2640d2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:16.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.751+0000 7fc2637fe640 1 -- 172.21.15.88:0/2751579357 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 509725193 0 0) 0x7fc250003520 con 0x7fc264073eb0 2024-09-16T01:44:16.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.751+0000 7fc263fff640 1 -- 172.21.15.88:0/2751579357 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2640ce7f0 con 0x7fc26415c710 2024-09-16T01:44:16.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.752+0000 7fc263fff640 1 --2- 172.21.15.88:0/2751579357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc26415c710 0x7fc264073970 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fc2540097a0 tx=0x7fc25402a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81a73adc0d77e821 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:16.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.752+0000 7fc2637fe640 1 -- 172.21.15.88:0/2751579357 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc254037070 con 0x7fc26415c710 2024-09-16T01:44:16.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.752+0000 7fc2637fe640 1 -- 172.21.15.88:0/2751579357 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc25402aa90 con 0x7fc26415c710 2024-09-16T01:44:16.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.752+0000 7fc2637fe640 1 -- 172.21.15.88:0/2751579357 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24c001670 con 0x7fc26415c710 2024-09-16T01:44:16.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.753+0000 7fc26a76f640 1 -- 172.21.15.88:0/2751579357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc26415c710 msgr2=0x7fc264073970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:16.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.753+0000 7fc26a76f640 1 --2- 172.21.15.88:0/2751579357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc26415c710 0x7fc264073970 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fc2540097a0 tx=0x7fc25402a000 comp rx=0 tx=0).stop 2024-09-16T01:44:16.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.753+0000 7fc26a76f640 1 -- 172.21.15.88:0/2751579357 shutdown_connections 2024-09-16T01:44:16.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.753+0000 7fc26a76f640 1 --2- 172.21.15.88:0/2751579357 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2640b53f0 0x7fc2640d2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:16.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.753+0000 7fc26a76f640 1 --2- 172.21.15.88:0/2751579357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc26415c710 0x7fc264073970 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:16.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.753+0000 7fc26a76f640 1 -- 172.21.15.88:0/2751579357 >> 172.21.15.88:0/2751579357 conn(0x7fc264062a20 msgr2=0x7fc264062e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:16.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.753+0000 7fc26a76f640 1 -- 172.21.15.88:0/2751579357 shutdown_connections 2024-09-16T01:44:16.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.753+0000 7fc26a76f640 1 -- 172.21.15.88:0/2751579357 wait complete. 2024-09-16T01:44:16.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.754+0000 7fc26a76f640 1 Processor -- start 2024-09-16T01:44:16.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.754+0000 7fc26a76f640 1 -- start start 2024-09-16T01:44:16.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.755+0000 7fc26a76f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc264073eb0 0x7fc2641f3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:16.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.755+0000 7fc26a76f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2640b53f0 0x7fc26406e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:16.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.755+0000 7fc26a76f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc26415c710 0x7fc2640b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:16.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.755+0000 7fc26a76f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc2640ceff0 con 0x7fc2640b53f0 2024-09-16T01:44:16.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.755+0000 7fc26a76f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc2640cdc00 con 0x7fc26415c710 2024-09-16T01:44:16.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.755+0000 7fc26a76f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc2640cf170 con 0x7fc264073eb0 2024-09-16T01:44:16.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.755+0000 7fc25bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2640b53f0 0x7fc26406e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:16.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.755+0000 7fc25bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2640b53f0 0x7fc26406e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50524/0 (socket says 172.21.15.88:50524) 2024-09-16T01:44:16.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.755+0000 7fc25bfff640 1 -- 172.21.15.88:0/1221415829 learned_addr learned my addr 172.21.15.88:0/1221415829 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:16.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.755+0000 7fc268d0c640 1 --2- 172.21.15.88:0/1221415829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc26415c710 0x7fc2640b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:16.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.755+0000 7fc25bfff640 1 -- 172.21.15.88:0/1221415829 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc264073eb0 msgr2=0x7fc2641f3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:16.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.755+0000 7fc25bfff640 1 --2- 172.21.15.88:0/1221415829 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc264073eb0 0x7fc2641f3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:16.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.755+0000 7fc25bfff640 1 -- 172.21.15.88:0/1221415829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc26415c710 msgr2=0x7fc2640b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:16.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.755+0000 7fc25bfff640 1 --2- 172.21.15.88:0/1221415829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc26415c710 0x7fc2640b0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:16.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.755+0000 7fc25bfff640 1 -- 172.21.15.88:0/1221415829 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc26406ea20 con 0x7fc2640b53f0 2024-09-16T01:44:16.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.755+0000 7fc25bfff640 1 --2- 172.21.15.88:0/1221415829 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2640b53f0 0x7fc26406e4e0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7fc24c00c780 tx=0x7fc24c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:16.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.755+0000 7fc2617fa640 1 -- 172.21.15.88:0/1221415829 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24c017070 con 0x7fc2640b53f0 2024-09-16T01:44:16.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.755+0000 7fc2617fa640 1 -- 172.21.15.88:0/1221415829 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc24c005ad0 con 0x7fc2640b53f0 2024-09-16T01:44:16.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.755+0000 7fc26a76f640 1 -- 172.21.15.88:0/1221415829 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2641f46d0 con 0x7fc2640b53f0 2024-09-16T01:44:16.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.755+0000 7fc2617fa640 1 -- 172.21.15.88:0/1221415829 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24c007070 con 0x7fc2640b53f0 2024-09-16T01:44:16.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.755+0000 7fc26a76f640 1 -- 172.21.15.88:0/1221415829 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2641f4c90 con 0x7fc2640b53f0 2024-09-16T01:44:16.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.756+0000 7fc26a76f640 1 -- 172.21.15.88:0/1221415829 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2640cdf50 con 0x7fc2640b53f0 2024-09-16T01:44:16.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.757+0000 7fc2617fa640 1 -- 172.21.15.88:0/1221415829 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc24c01a070 con 0x7fc2640b53f0 2024-09-16T01:44:16.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.757+0000 7fc2617fa640 1 --2- 172.21.15.88:0/1221415829 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc24406c2b0 0x7fc24406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:16.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.758+0000 7fc263fff640 1 --2- 172.21.15.88:0/1221415829 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc24406c2b0 0x7fc24406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:16.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.758+0000 7fc2617fa640 1 -- 172.21.15.88:0/1221415829 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 8920+0+0 (secure 0 0 0) 0x7fc24c08b6a0 con 0x7fc2640b53f0 2024-09-16T01:44:16.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.758+0000 7fc263fff640 1 --2- 172.21.15.88:0/1221415829 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc24406c2b0 0x7fc24406e770 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fc254034000 tx=0x7fc25402c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:16.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.762+0000 7fc2617fa640 1 -- 172.21.15.88:0/1221415829 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc24c059130 con 0x7fc2640b53f0 2024-09-16T01:44:16.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:16.997+0000 7fc26a76f640 1 -- 172.21.15.88:0/1221415829 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fc2640b1170 con 0x7fc2640b53f0 2024-09-16T01:44:17.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.642+0000 7fc2617fa640 1 -- 172.21.15.88:0/1221415829 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v299) ==== 107+0+0 (secure 0 0 0) 0x7fc24c058a30 con 0x7fc2640b53f0 2024-09-16T01:44:17.644 INFO:teuthology.orchestra.run.smithi088.stderr:nothing to do: rank is not damaged 2024-09-16T01:44:17.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.645+0000 7fc26a76f640 1 -- 172.21.15.88:0/1221415829 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc24406c2b0 msgr2=0x7fc24406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:17.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.645+0000 7fc26a76f640 1 --2- 172.21.15.88:0/1221415829 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc24406c2b0 0x7fc24406e770 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fc254034000 tx=0x7fc25402c000 comp rx=0 tx=0).stop 2024-09-16T01:44:17.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.645+0000 7fc26a76f640 1 -- 172.21.15.88:0/1221415829 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2640b53f0 msgr2=0x7fc26406e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:17.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.645+0000 7fc26a76f640 1 --2- 172.21.15.88:0/1221415829 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2640b53f0 0x7fc26406e4e0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7fc24c00c780 tx=0x7fc24c00cc40 comp rx=0 tx=0).stop 2024-09-16T01:44:17.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.645+0000 7fc26a76f640 1 -- 172.21.15.88:0/1221415829 shutdown_connections 2024-09-16T01:44:17.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.645+0000 7fc26a76f640 1 --2- 172.21.15.88:0/1221415829 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc24406c2b0 0x7fc24406e770 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:17.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.645+0000 7fc26a76f640 1 --2- 172.21.15.88:0/1221415829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc26415c710 0x7fc2640b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:17.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.645+0000 7fc26a76f640 1 --2- 172.21.15.88:0/1221415829 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc2640b53f0 0x7fc26406e4e0 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:17.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.645+0000 7fc26a76f640 1 --2- 172.21.15.88:0/1221415829 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc264073eb0 0x7fc2641f3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:17.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.645+0000 7fc26a76f640 1 -- 172.21.15.88:0/1221415829 >> 172.21.15.88:0/1221415829 conn(0x7fc264062a20 msgr2=0x7fc2640b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:17.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.645+0000 7fc26a76f640 1 -- 172.21.15.88:0/1221415829 shutdown_connections 2024-09-16T01:44:17.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.645+0000 7fc26a76f640 1 -- 172.21.15.88:0/1221415829 wait complete. 2024-09-16T01:44:17.658 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:44:17.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.741+0000 7fe9da18ff40 1 Processor -- start 2024-09-16T01:44:17.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.741+0000 7fe9da18ff40 1 -- start start 2024-09-16T01:44:17.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.741+0000 7fe9da18ff40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcefa97e10 0x55bcefbd1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:17.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.741+0000 7fe9da18ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcefa98a60 0x55bcefb3cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:17.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.741+0000 7fe9da18ff40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bcefab2250 0x55bcefb3d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:17.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.741+0000 7fe9da18ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55bcef9ba510 con 0x55bcefa98a60 2024-09-16T01:44:17.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.741+0000 7fe9da18ff40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bcefb41120 con 0x55bcefa97e10 2024-09-16T01:44:17.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.741+0000 7fe9da18ff40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55bcefbd0010 con 0x55bcefab2250 2024-09-16T01:44:17.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.742+0000 7fe9d7484640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcefa98a60 0x55bcefb3cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:17.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.742+0000 7fe9d7484640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcefa98a60 0x55bcefb3cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50536/0 (socket says 172.21.15.88:50536) 2024-09-16T01:44:17.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.742+0000 7fe9d7484640 1 -- 172.21.15.88:0/311497319 learned_addr learned my addr 172.21.15.88:0/311497319 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:17.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.742+0000 7fe9d7c85640 1 --2- 172.21.15.88:0/311497319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcefa97e10 0x55bcefbd1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:17.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.742+0000 7fe9d8486640 1 --2- 172.21.15.88:0/311497319 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bcefab2250 0x55bcefb3d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:17.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.742+0000 7fe9d7484640 1 -- 172.21.15.88:0/311497319 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bcefab2250 msgr2=0x55bcefb3d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:17.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.742+0000 7fe9d7484640 1 --2- 172.21.15.88:0/311497319 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bcefab2250 0x55bcefb3d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:17.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.742+0000 7fe9d7484640 1 -- 172.21.15.88:0/311497319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcefa97e10 msgr2=0x55bcefbd1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:17.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.742+0000 7fe9d7484640 1 --2- 172.21.15.88:0/311497319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcefa97e10 0x55bcefbd1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:17.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.742+0000 7fe9d7484640 1 -- 172.21.15.88:0/311497319 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcefb401c0 con 0x55bcefa98a60 2024-09-16T01:44:17.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.742+0000 7fe9d7c85640 1 --2- 172.21.15.88:0/311497319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcefa97e10 0x55bcefbd1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:17.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.742+0000 7fe9d7484640 1 --2- 172.21.15.88:0/311497319 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcefa98a60 0x55bcefb3cd90 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc000c00 tx=0x7fe9cc051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1fed8e15ca8d974 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:17.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.742+0000 7fe9d6482640 1 -- 172.21.15.88:0/311497319 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc055070 con 0x55bcefa98a60 2024-09-16T01:44:17.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.742+0000 7fe9d6482640 1 -- 172.21.15.88:0/311497319 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9cc0518b0 con 0x55bcefa98a60 2024-09-16T01:44:17.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.742+0000 7fe9d6482640 1 -- 172.21.15.88:0/311497319 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc051c80 con 0x55bcefa98a60 2024-09-16T01:44:17.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.743+0000 7fe9da18ff40 1 -- 172.21.15.88:0/311497319 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcefa98a60 msgr2=0x55bcefb3cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:17.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.743+0000 7fe9da18ff40 1 --2- 172.21.15.88:0/311497319 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcefa98a60 0x55bcefb3cd90 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc000c00 tx=0x7fe9cc051000 comp rx=0 tx=0).stop 2024-09-16T01:44:17.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.743+0000 7fe9da18ff40 1 -- 172.21.15.88:0/311497319 shutdown_connections 2024-09-16T01:44:17.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.743+0000 7fe9da18ff40 1 --2- 172.21.15.88:0/311497319 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bcefab2250 0x55bcefb3d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:17.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.743+0000 7fe9da18ff40 1 --2- 172.21.15.88:0/311497319 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcefa98a60 0x55bcefb3cd90 unknown :-1 s=CLOSED pgs=2212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:17.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.743+0000 7fe9da18ff40 1 --2- 172.21.15.88:0/311497319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcefa97e10 0x55bcefbd1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:17.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.743+0000 7fe9da18ff40 1 -- 172.21.15.88:0/311497319 >> 172.21.15.88:0/311497319 conn(0x55bcefab3750 msgr2=0x55bcefb3bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:17.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.743+0000 7fe9da18ff40 1 -- 172.21.15.88:0/311497319 shutdown_connections 2024-09-16T01:44:17.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.743+0000 7fe9da18ff40 1 -- 172.21.15.88:0/311497319 wait complete. 2024-09-16T01:44:17.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.743+0000 7fe9da18ff40 1 Processor -- start 2024-09-16T01:44:17.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.743+0000 7fe9da18ff40 1 -- start start 2024-09-16T01:44:17.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.744+0000 7fe9da18ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcefa97e10 0x55bcefb3d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:17.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.744+0000 7fe9da18ff40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcefa98a60 0x55bcefb39870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:17.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.744+0000 7fe9da18ff40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bcefab2250 0x55bcefb2e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:17.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.744+0000 7fe9da18ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55bcefb3fee0 con 0x55bcefa97e10 2024-09-16T01:44:17.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.744+0000 7fe9da18ff40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bcefbcfaa0 con 0x55bcefa98a60 2024-09-16T01:44:17.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.744+0000 7fe9da18ff40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55bcefbcf730 con 0x55bcefab2250 2024-09-16T01:44:17.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.744+0000 7fe9d7c85640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcefa97e10 0x55bcefb3d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:17.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.744+0000 7fe9d7c85640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcefa97e10 0x55bcefb3d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50548/0 (socket says 172.21.15.88:50548) 2024-09-16T01:44:17.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.744+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 learned_addr learned my addr 172.21.15.88:0/1234101742 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:17.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.744+0000 7fe9d8486640 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bcefab2250 0x55bcefb2e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:17.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.744+0000 7fe9d7484640 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcefa98a60 0x55bcefb39870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:17.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.744+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bcefab2250 msgr2=0x55bcefb2e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:17.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.744+0000 7fe9d7c85640 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bcefab2250 0x55bcefb2e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:17.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.744+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcefa98a60 msgr2=0x55bcefb39870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:17.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.744+0000 7fe9d7c85640 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcefa98a60 0x55bcefb39870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:17.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.744+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcefb39db0 con 0x55bcefa97e10 2024-09-16T01:44:17.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.744+0000 7fe9d7484640 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bcefa98a60 0x55bcefb39870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:17.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.744+0000 7fe9d7c85640 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcefa97e10 0x55bcefb3d3c0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8007e10 tx=0x7fe9c8006750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:17.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.745+0000 7fe9d5480640 1 -- 172.21.15.88:0/1234101742 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c8018070 con 0x55bcefa97e10 2024-09-16T01:44:17.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.745+0000 7fe9d5480640 1 -- 172.21.15.88:0/1234101742 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9c8011700 con 0x55bcefa97e10 2024-09-16T01:44:17.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.745+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55bcefb42b60 con 0x55bcefa97e10 2024-09-16T01:44:17.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.745+0000 7fe9d5480640 1 -- 172.21.15.88:0/1234101742 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c8011ad0 con 0x55bcefa97e10 2024-09-16T01:44:17.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.745+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcefb43120 con 0x55bcefa97e10 2024-09-16T01:44:17.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.746+0000 7fe9d5480640 1 -- 172.21.15.88:0/1234101742 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe9c801b070 con 0x55bcefa97e10 2024-09-16T01:44:17.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.746+0000 7fe9d5480640 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe9c406acc0 0x7fe9c406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:17.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.747+0000 7fe9d5480640 1 -- 172.21.15.88:0/1234101742 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 8920+0+0 (secure 0 0 0) 0x7fe9c808a080 con 0x55bcefa97e10 2024-09-16T01:44:17.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.747+0000 7fe9d7484640 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe9c406acc0 0x7fe9c406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:17.754 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:44:17.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.747+0000 7fe9da18ff40 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55bcefb37820 0x55bcefb37c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:17.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.747+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefb38160 con 0x55bcefb37820 2024-09-16T01:44:17.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.747+0000 7fe9d8486640 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55bcefb37820 0x55bcefb37c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:17.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.747+0000 7fe9d7484640 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe9c406acc0 0x7fe9c406d180 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc064000 tx=0x7fe9cc053000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:17.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.747+0000 7fe9d8486640 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55bcefb37820 0x55bcefb37c20 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:17.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.749+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v124'1128 uv1102 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.749+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e124) -- 0x55bcefb38570 con 0x55bcefb37820 2024-09-16T01:44:17.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.751+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v124'1129 uv1129 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.751+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe2fc0 con 0x55bcefb37820 2024-09-16T01:44:17.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.752+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v124'1130 uv1130 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.757 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:44:17.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.752+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe4370 con 0x55bcefb37820 2024-09-16T01:44:17.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.753+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v124'1131 uv1131 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.753+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe4710 con 0x55bcefb37820 2024-09-16T01:44:17.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.754+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v124'1132 uv1132 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.755+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe3360 con 0x55bcefb37820 2024-09-16T01:44:17.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.757+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v124'1133 uv1133 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.757+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe3750 con 0x55bcefb37820 2024-09-16T01:44:17.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.758+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v124'1134 uv1108 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.758+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe3b40 con 0x55bcefb37820 2024-09-16T01:44:17.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.760+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v124'1135 uv1135 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.760+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe3b40 con 0x55bcefb37820 2024-09-16T01:44:17.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.761+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v124'1136 uv1136 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.762 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:44:17.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.761+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe3af0 con 0x55bcefb37820 2024-09-16T01:44:17.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.763+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v124'1137 uv1137 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.763+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe3af0 con 0x55bcefb37820 2024-09-16T01:44:17.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.764+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v124'1138 uv1138 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.764+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe3af0 con 0x55bcefb37820 2024-09-16T01:44:17.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.765+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v124'1139 uv1139 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.765+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe3af0 con 0x55bcefb37820 2024-09-16T01:44:17.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.767+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v124'1140 uv1115 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.767+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe3af0 con 0x55bcefb37820 2024-09-16T01:44:17.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.768+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v124'1141 uv1141 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.768+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe3af0 con 0x55bcefb37820 2024-09-16T01:44:17.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.769+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v124'1142 uv1142 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.771 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:44:17.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.769+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe3af0 con 0x55bcefb37820 2024-09-16T01:44:17.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.771+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v124'1143 uv1143 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.771+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe3af0 con 0x55bcefb37820 2024-09-16T01:44:17.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.772+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v124'1144 uv1144 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.772+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe3af0 con 0x55bcefb37820 2024-09-16T01:44:17.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.774+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v124'1145 uv1145 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.774+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe3eb0 con 0x55bcefb37820 2024-09-16T01:44:17.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.775+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v124'1146 uv1146 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.775+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe3eb0 con 0x55bcefb37820 2024-09-16T01:44:17.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.776+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v124'1147 uv1121 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.776+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe3da0 con 0x55bcefb37820 2024-09-16T01:44:17.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.778+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v124'1148 uv1148 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.778+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe3da0 con 0x55bcefb37820 2024-09-16T01:44:17.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.779+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v124'1149 uv1149 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.780 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:44:17.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.779+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe3da0 con 0x55bcefb37820 2024-09-16T01:44:17.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.780+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v124'1150 uv1150 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.780+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefb332b0 con 0x55bcefb37820 2024-09-16T01:44:17.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.781+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v124'1151 uv1151 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.781+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefb332b0 con 0x55bcefb37820 2024-09-16T01:44:17.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.783+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v124'1152 uv1152 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.783+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefb332b0 con 0x55bcefb37820 2024-09-16T01:44:17.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.784+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v124'1153 uv1127 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.784+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e124) -- 0x55bcefb332b0 con 0x55bcefb37820 2024-09-16T01:44:17.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.785+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v124'1154 uv1154 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.785+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefb332b0 con 0x55bcefb37820 2024-09-16T01:44:17.788 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:44:17.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.787+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v124'1155 uv1155 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.787+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefb332b0 con 0x55bcefb37820 2024-09-16T01:44:17.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.788+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v124'1156 uv1156 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.788+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe3da0 con 0x55bcefb37820 2024-09-16T01:44:17.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.789+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v124'1157 uv1157 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.789+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe3da0 con 0x55bcefb37820 2024-09-16T01:44:17.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.790+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v124'1158 uv1158 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.790+0000 7fe9da18ff40 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55bcefbe4fc0 0x55bcefbe73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:17.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.790+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe3da0 con 0x55bcefbe4fc0 2024-09-16T01:44:17.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.791+0000 7fe9d7c85640 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55bcefbe4fc0 0x55bcefbe73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:17.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.791+0000 7fe9d7c85640 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55bcefbe4fc0 0x55bcefbe73b0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:17.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.793+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v124'825 uv804 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbe4fc0 2024-09-16T01:44:17.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.793+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe78f0 con 0x55bcefbe4fc0 2024-09-16T01:44:17.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.794+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v124'826 uv826 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbe4fc0 2024-09-16T01:44:17.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.794+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe7c90 con 0x55bcefbe4fc0 2024-09-16T01:44:17.797 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:44:17.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.796+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v124'827 uv827 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbe4fc0 2024-09-16T01:44:17.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.796+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe84e0 con 0x55bcefbe4fc0 2024-09-16T01:44:17.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.797+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v124'828 uv828 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbe4fc0 2024-09-16T01:44:17.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.797+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe8880 con 0x55bcefbe4fc0 2024-09-16T01:44:17.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.798+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v124'829 uv829 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbe4fc0 2024-09-16T01:44:17.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.798+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe8c20 con 0x55bcefbe4fc0 2024-09-16T01:44:17.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.800+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v124'830 uv823 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbe4fc0 2024-09-16T01:44:17.801 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.800+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe8fc0 con 0x55bcefbe4fc0 2024-09-16T01:44:17.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.802+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v124'831 uv831 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbe4fc0 2024-09-16T01:44:17.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.802+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe9360 con 0x55bcefbe4fc0 2024-09-16T01:44:17.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.803+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v124'832 uv832 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbe4fc0 2024-09-16T01:44:17.804 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:44:17.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.803+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe9360 con 0x55bcefbe4fc0 2024-09-16T01:44:17.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.804+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v124'833 uv833 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbe4fc0 2024-09-16T01:44:17.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.804+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe9360 con 0x55bcefbe4fc0 2024-09-16T01:44:17.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.805+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v124'834 uv834 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbe4fc0 2024-09-16T01:44:17.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.805+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe9360 con 0x55bcefbe4fc0 2024-09-16T01:44:17.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.807+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v124'835 uv824 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbe4fc0 2024-09-16T01:44:17.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.807+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe9360 con 0x55bcefbe4fc0 2024-09-16T01:44:17.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.809+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v124'836 uv836 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbe4fc0 2024-09-16T01:44:17.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.809+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe9360 con 0x55bcefbe4fc0 2024-09-16T01:44:17.811 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:44:17.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.810+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v124'837 uv837 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbe4fc0 2024-09-16T01:44:17.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.810+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbed7e0 con 0x55bcefbe4fc0 2024-09-16T01:44:17.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.812+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v124'838 uv838 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbe4fc0 2024-09-16T01:44:17.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.812+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbed7e0 con 0x55bcefbe4fc0 2024-09-16T01:44:17.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.813+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v124'839 uv839 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbe4fc0 2024-09-16T01:44:17.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.813+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe9360 con 0x55bcefbe4fc0 2024-09-16T01:44:17.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.814+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v124'840 uv820 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbe4fc0 2024-09-16T01:44:17.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.814+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe9360 con 0x55bcefbe4fc0 2024-09-16T01:44:17.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.816+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v124'841 uv841 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbe4fc0 2024-09-16T01:44:17.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.816+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe9360 con 0x55bcefbe4fc0 2024-09-16T01:44:17.818 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:44:17.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.817+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v124'842 uv842 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbe4fc0 2024-09-16T01:44:17.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.817+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe9360 con 0x55bcefbe4fc0 2024-09-16T01:44:17.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.818+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v124'843 uv843 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbe4fc0 2024-09-16T01:44:17.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.818+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbea370 con 0x55bcefbe4fc0 2024-09-16T01:44:17.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.819+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v124'844 uv844 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbe4fc0 2024-09-16T01:44:17.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.819+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe9360 con 0x55bcefbe4fc0 2024-09-16T01:44:17.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.820+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v124'845 uv845 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbe4fc0 2024-09-16T01:44:17.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.820+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe9360 con 0x55bcefb37820 2024-09-16T01:44:17.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.821+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v124'219 uv218 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.821+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe9360 con 0x55bcefb37820 2024-09-16T01:44:17.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.822+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v124'220 uv220 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.822+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe9360 con 0x55bcefb37820 2024-09-16T01:44:17.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.823+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v124'221 uv221 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.825 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:44:17.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.823+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe9360 con 0x55bcefb37820 2024-09-16T01:44:17.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.824+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v124'222 uv222 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.824+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe9360 con 0x55bcefb37820 2024-09-16T01:44:17.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.825+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v124'223 uv223 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.825+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe9360 con 0x55bcefb37820 2024-09-16T01:44:17.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.826+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v124'224 uv224 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.826+0000 7fe9da18ff40 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55bcefbe9c60 0x55bcefbea060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:17.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.826+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe9360 con 0x55bcefbe9c60 2024-09-16T01:44:17.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.826+0000 7fe9d7484640 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55bcefbe9c60 0x55bcefbea060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:17.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.826+0000 7fe9d7484640 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55bcefbe9c60 0x55bcefbea060 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:17.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.827+0000 7fe9d7484640 1 -- 172.21.15.88:0/1234101742 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v124'168 uv167 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7fe9cc057030 con 0x55bcefbe9c60 2024-09-16T01:44:17.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.827+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbec960 con 0x55bcefbe9c60 2024-09-16T01:44:17.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.828+0000 7fe9d7484640 1 -- 172.21.15.88:0/1234101742 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v124'169 uv169 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe9cc057030 con 0x55bcefbe9c60 2024-09-16T01:44:17.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.828+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbece60 con 0x55bcefbe9c60 2024-09-16T01:44:17.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.829+0000 7fe9d7484640 1 -- 172.21.15.88:0/1234101742 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v124'170 uv170 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe9cc057030 con 0x55bcefbe9c60 2024-09-16T01:44:17.831 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:44:17.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.829+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbe8110 con 0x55bcefbe9c60 2024-09-16T01:44:17.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.830+0000 7fe9d7484640 1 -- 172.21.15.88:0/1234101742 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v124'171 uv171 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7fe9cc057030 con 0x55bcefbe9c60 2024-09-16T01:44:17.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.830+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbed790 con 0x55bcefb37820 2024-09-16T01:44:17.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.831+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v124'219 uv218 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.831+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbed790 con 0x55bcefb37820 2024-09-16T01:44:17.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.832+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v124'220 uv220 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.832+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbed790 con 0x55bcefb37820 2024-09-16T01:44:17.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.834+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v124'221 uv221 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.835 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:44:17.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.834+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbed790 con 0x55bcefb37820 2024-09-16T01:44:17.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.835+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v124'222 uv222 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.835+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbed790 con 0x55bcefb37820 2024-09-16T01:44:17.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.836+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v124'223 uv223 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.836+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbed790 con 0x55bcefb37820 2024-09-16T01:44:17.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.837+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v124'224 uv224 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.837+0000 7fe9da18ff40 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55bcefbee090 0x55bcefbf0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:17.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.837+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf09c0 con 0x55bcefbee090 2024-09-16T01:44:17.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.837+0000 7fe9d8486640 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55bcefbee090 0x55bcefbf0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:17.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.838+0000 7fe9d8486640 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55bcefbee090 0x55bcefbf0480 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:17.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.839+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v124'404 uv398 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.839+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf09c0 con 0x55bcefbee090 2024-09-16T01:44:17.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.840+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v124'405 uv405 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.840+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf0db0 con 0x55bcefbee090 2024-09-16T01:44:17.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.841+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v124'406 uv406 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.843 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:44:17.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.841+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf1dc0 con 0x55bcefbee090 2024-09-16T01:44:17.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.842+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v124'407 uv407 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.842+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf1dc0 con 0x55bcefbee090 2024-09-16T01:44:17.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.843+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v124'408 uv408 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.843+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf0db0 con 0x55bcefbee090 2024-09-16T01:44:17.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.845+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v124'409 uv409 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.845+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf0db0 con 0x55bcefbee090 2024-09-16T01:44:17.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.846+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v124'410 uv403 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.846+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf0db0 con 0x55bcefbee090 2024-09-16T01:44:17.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.847+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v124'411 uv411 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.847+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf0db0 con 0x55bcefbee090 2024-09-16T01:44:17.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.848+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v124'412 uv412 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.849 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:44:17.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.848+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf0db0 con 0x55bcefbee090 2024-09-16T01:44:17.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.849+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v124'413 uv413 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.849+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf0db0 con 0x55bcefbee090 2024-09-16T01:44:17.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.850+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v124'414 uv414 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.850+0000 7fe9da18ff40 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55bcefbf12c0 0x55bcefbf36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:17.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.850+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf3df0 con 0x55bcefbf12c0 2024-09-16T01:44:17.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.850+0000 7fe9d7c85640 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55bcefbf12c0 0x55bcefbf36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:17.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.850+0000 7fe9d7c85640 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55bcefbf12c0 0x55bcefbf36d0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:17.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.852+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v124'183 uv182 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbf12c0 2024-09-16T01:44:17.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.852+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf41c0 con 0x55bcefbf12c0 2024-09-16T01:44:17.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.853+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v124'184 uv184 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbf12c0 2024-09-16T01:44:17.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.853+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf41c0 con 0x55bcefbf12c0 2024-09-16T01:44:17.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.854+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v124'185 uv185 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbf12c0 2024-09-16T01:44:17.855 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:44:17.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.854+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf41c0 con 0x55bcefbf12c0 2024-09-16T01:44:17.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.855+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v124'186 uv186 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbf12c0 2024-09-16T01:44:17.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.855+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf41c0 con 0x55bcefbf12c0 2024-09-16T01:44:17.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.856+0000 7fe9d7c85640 1 -- 172.21.15.88:0/1234101742 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v124'187 uv187 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe9c801a030 con 0x55bcefbf12c0 2024-09-16T01:44:17.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.856+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf41c0 con 0x55bcefb37820 2024-09-16T01:44:17.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.857+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v124'219 uv218 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.857+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf41c0 con 0x55bcefb37820 2024-09-16T01:44:17.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.858+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v124'220 uv220 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.858+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf41c0 con 0x55bcefb37820 2024-09-16T01:44:17.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.859+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v124'221 uv221 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.860 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:44:17.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.859+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf5210 con 0x55bcefb37820 2024-09-16T01:44:17.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.860+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v124'222 uv222 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.860+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf51d0 con 0x55bcefb37820 2024-09-16T01:44:17.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.861+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v124'223 uv223 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.861+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf41c0 con 0x55bcefb37820 2024-09-16T01:44:17.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.861+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v124'224 uv224 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefb37820 2024-09-16T01:44:17.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.861+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf41c0 con 0x55bcefbe9c60 2024-09-16T01:44:17.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.863+0000 7fe9d7484640 1 -- 172.21.15.88:0/1234101742 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v124'365 uv359 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7fe9cc057030 con 0x55bcefbe9c60 2024-09-16T01:44:17.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.863+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf4560 con 0x55bcefbe9c60 2024-09-16T01:44:17.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.864+0000 7fe9d7484640 1 -- 172.21.15.88:0/1234101742 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v124'366 uv366 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe9cc057030 con 0x55bcefbe9c60 2024-09-16T01:44:17.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.864+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf4950 con 0x55bcefbe9c60 2024-09-16T01:44:17.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.865+0000 7fe9d7484640 1 -- 172.21.15.88:0/1234101742 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v124'367 uv367 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe9cc057030 con 0x55bcefbe9c60 2024-09-16T01:44:17.866 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:44:17.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.865+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf4d40 con 0x55bcefbe9c60 2024-09-16T01:44:17.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.866+0000 7fe9d7484640 1 -- 172.21.15.88:0/1234101742 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v124'368 uv368 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe9cc057030 con 0x55bcefbe9c60 2024-09-16T01:44:17.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.866+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf4d40 con 0x55bcefbe9c60 2024-09-16T01:44:17.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.867+0000 7fe9d7484640 1 -- 172.21.15.88:0/1234101742 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v124'369 uv369 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fe9cc057030 con 0x55bcefbe9c60 2024-09-16T01:44:17.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.867+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf4cf0 con 0x55bcefbe9c60 2024-09-16T01:44:17.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.868+0000 7fe9d7484640 1 -- 172.21.15.88:0/1234101742 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v124'370 uv364 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9cc057030 con 0x55bcefbe9c60 2024-09-16T01:44:17.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.868+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf4cf0 con 0x55bcefbe9c60 2024-09-16T01:44:17.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.869+0000 7fe9d7484640 1 -- 172.21.15.88:0/1234101742 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v124'371 uv371 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9cc057030 con 0x55bcefbe9c60 2024-09-16T01:44:17.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.869+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf4cf0 con 0x55bcefbe9c60 2024-09-16T01:44:17.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.869+0000 7fe9d7484640 1 -- 172.21.15.88:0/1234101742 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v124'372 uv372 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9cc057030 con 0x55bcefbe9c60 2024-09-16T01:44:17.871 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:44:17.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.869+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf4cf0 con 0x55bcefbe9c60 2024-09-16T01:44:17.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.870+0000 7fe9d7484640 1 -- 172.21.15.88:0/1234101742 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v124'373 uv373 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9cc057030 con 0x55bcefbe9c60 2024-09-16T01:44:17.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.870+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf8020 con 0x55bcefbe9c60 2024-09-16T01:44:17.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.871+0000 7fe9d7484640 1 -- 172.21.15.88:0/1234101742 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v124'374 uv374 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9cc057030 con 0x55bcefbe9c60 2024-09-16T01:44:17.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.871+0000 7fe9da18ff40 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55bcefbf5bd0 0x55bcefbf7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:17.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.871+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf8500 con 0x55bcefbf5bd0 2024-09-16T01:44:17.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.872+0000 7fe9d7484640 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55bcefbf5bd0 0x55bcefbf7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:17.873 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.872+0000 7fe9d7484640 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55bcefbf5bd0 0x55bcefbf7fc0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:17.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.873+0000 7fe9d7484640 1 -- 172.21.15.88:0/1234101742 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v124'162 uv160 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7fe9cc057030 con 0x55bcefbf5bd0 2024-09-16T01:44:17.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.873+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf8500 con 0x55bcefbf5bd0 2024-09-16T01:44:17.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.874+0000 7fe9d7484640 1 -- 172.21.15.88:0/1234101742 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v124'163 uv163 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fe9cc057030 con 0x55bcefbf5bd0 2024-09-16T01:44:17.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.874+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf8500 con 0x55bcefbf5bd0 2024-09-16T01:44:17.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.875+0000 7fe9d7484640 1 -- 172.21.15.88:0/1234101742 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v124'164 uv164 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fe9cc057030 con 0x55bcefbf5bd0 2024-09-16T01:44:17.876 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:44:17.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.875+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf8500 con 0x55bcefbf5bd0 2024-09-16T01:44:17.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.876+0000 7fe9d7484640 1 -- 172.21.15.88:0/1234101742 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v124'165 uv165 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7fe9cc057030 con 0x55bcefbf5bd0 2024-09-16T01:44:17.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.876+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf8500 con 0x55bcefbe9c60 2024-09-16T01:44:17.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.877+0000 7fe9d7484640 1 -- 172.21.15.88:0/1234101742 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v124'219 uv218 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9cc057030 con 0x55bcefbe9c60 2024-09-16T01:44:17.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.877+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf8500 con 0x55bcefbe9c60 2024-09-16T01:44:17.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.878+0000 7fe9d7484640 1 -- 172.21.15.88:0/1234101742 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v124'220 uv220 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9cc057030 con 0x55bcefbe9c60 2024-09-16T01:44:17.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.878+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf8500 con 0x55bcefbe9c60 2024-09-16T01:44:17.880 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:44:17.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.879+0000 7fe9d7484640 1 -- 172.21.15.88:0/1234101742 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v124'221 uv221 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9cc057030 con 0x55bcefbe9c60 2024-09-16T01:44:17.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.879+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf9510 con 0x55bcefbe9c60 2024-09-16T01:44:17.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.880+0000 7fe9d7484640 1 -- 172.21.15.88:0/1234101742 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v124'222 uv222 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9cc057030 con 0x55bcefbe9c60 2024-09-16T01:44:17.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.880+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf9510 con 0x55bcefbe9c60 2024-09-16T01:44:17.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.882+0000 7fe9d7484640 1 -- 172.21.15.88:0/1234101742 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v124'223 uv223 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9cc057030 con 0x55bcefbe9c60 2024-09-16T01:44:17.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.882+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf8500 con 0x55bcefbe9c60 2024-09-16T01:44:17.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.883+0000 7fe9d7484640 1 -- 172.21.15.88:0/1234101742 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v124'224 uv224 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9cc057030 con 0x55bcefbe9c60 2024-09-16T01:44:17.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.883+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf8500 con 0x55bcefbee090 2024-09-16T01:44:17.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.884+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v124'186 uv185 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.884+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf8500 con 0x55bcefbee090 2024-09-16T01:44:17.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.885+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v124'187 uv187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.885+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf8500 con 0x55bcefbee090 2024-09-16T01:44:17.887 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:44:17.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.886+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v124'188 uv188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.886+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf8500 con 0x55bcefbee090 2024-09-16T01:44:17.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.887+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v124'189 uv189 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.887+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf8500 con 0x55bcefbee090 2024-09-16T01:44:17.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.888+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v124'190 uv190 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.888+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf8500 con 0x55bcefbee090 2024-09-16T01:44:17.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.890+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v124'455 uv446 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.890+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf8500 con 0x55bcefbee090 2024-09-16T01:44:17.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.891+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v124'456 uv456 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.891+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf8500 con 0x55bcefbee090 2024-09-16T01:44:17.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.892+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v124'457 uv457 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.893 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:44:17.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.892+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf88f0 con 0x55bcefbee090 2024-09-16T01:44:17.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.893+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v124'458 uv458 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.893+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf85b0 con 0x55bcefbee090 2024-09-16T01:44:17.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.894+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v124'459 uv459 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.894+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf85b0 con 0x55bcefbee090 2024-09-16T01:44:17.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.895+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v124'460 uv453 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.895+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf8500 con 0x55bcefbee090 2024-09-16T01:44:17.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.896+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v124'461 uv461 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.896+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf8500 con 0x55bcefbee090 2024-09-16T01:44:17.898 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:44:17.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.897+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v124'462 uv462 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.897+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf9550 con 0x55bcefbee090 2024-09-16T01:44:17.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.898+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v124'463 uv463 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.898+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf9510 con 0x55bcefbee090 2024-09-16T01:44:17.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.899+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v124'464 uv464 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.899+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf8500 con 0x55bcefbee090 2024-09-16T01:44:17.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.901+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v124'465 uv465 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.901+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55bcefbf89b0 con 0x55bcefbee090 2024-09-16T01:44:17.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v124'466 uv466 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7fe9d0010660 con 0x55bcefbee090 2024-09-16T01:44:17.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55bcefb37820 msgr2=0x55bcefb37c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:17.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9da18ff40 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55bcefb37820 0x55bcefb37c20 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:17.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55bcefbe9c60 msgr2=0x55bcefbea060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:17.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9da18ff40 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55bcefbe9c60 0x55bcefbea060 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:17.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55bcefbf5bd0 msgr2=0x55bcefbf7fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:17.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9da18ff40 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55bcefbf5bd0 0x55bcefbf7fc0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:17.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55bcefbe4fc0 msgr2=0x55bcefbe73b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:17.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9da18ff40 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55bcefbe4fc0 0x55bcefbe73b0 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:17.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55bcefbee090 msgr2=0x55bcefbf0480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:17.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9da18ff40 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55bcefbee090 0x55bcefbf0480 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:17.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55bcefbf12c0 msgr2=0x55bcefbf36d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:17.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9da18ff40 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55bcefbf12c0 0x55bcefbf36d0 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:17.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 reap_dead start 2024-09-16T01:44:17.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 reap_dead start 2024-09-16T01:44:17.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9d8486640 1 -- 172.21.15.88:0/1234101742 reap_dead start 2024-09-16T01:44:17.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe9c406acc0 msgr2=0x7fe9c406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:17.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9da18ff40 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe9c406acc0 0x7fe9c406d180 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc064000 tx=0x7fe9cc053000 comp rx=0 tx=0).stop 2024-09-16T01:44:17.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcefa97e10 msgr2=0x55bcefb3d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:17.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9da18ff40 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcefa97e10 0x55bcefb3d3c0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8007e10 tx=0x7fe9c8006750 comp rx=0 tx=0).stop 2024-09-16T01:44:17.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 shutdown_connections 2024-09-16T01:44:17.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9da18ff40 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe9c406acc0 0x7fe9c406d180 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:17.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9da18ff40 1 --2- 172.21.15.88:0/1234101742 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bcefa97e10 0x55bcefb3d3c0 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:17.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 >> 172.21.15.88:0/1234101742 conn(0x55bcefab3750 msgr2=0x55bcefb365c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:17.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 shutdown_connections 2024-09-16T01:44:17.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.902+0000 7fe9da18ff40 1 -- 172.21.15.88:0/1234101742 wait complete. 2024-09-16T01:44:17.908 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapheader 100.00000000 deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-09-16T01:44:17.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.991+0000 7f4d283f8f40 1 Processor -- start 2024-09-16T01:44:17.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.991+0000 7f4d283f8f40 1 -- start start 2024-09-16T01:44:17.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.991+0000 7f4d283f8f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9aa453dd0 0x55d9aa51a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:17.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.991+0000 7f4d283f8f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9aa456230 0x55d9aa4529a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:17.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.991+0000 7f4d283f8f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9aa522540 0x55d9aa452ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:17.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.991+0000 7f4d283f8f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d9aa49d5b0 con 0x55d9aa456230 2024-09-16T01:44:17.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.991+0000 7f4d283f8f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d9aa3a4510 con 0x55d9aa522540 2024-09-16T01:44:17.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.991+0000 7f4d283f8f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55d9aa4541e0 con 0x55d9aa453dd0 2024-09-16T01:44:18.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.992+0000 7f4d1ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9aa456230 0x55d9aa4529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:18.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.992+0000 7f4d25ef0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9aa453dd0 0x55d9aa51a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:18.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.992+0000 7f4d1ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9aa456230 0x55d9aa4529a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50562/0 (socket says 172.21.15.88:50562) 2024-09-16T01:44:18.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.992+0000 7f4d1ffff640 1 -- 172.21.15.88:0/2648944002 learned_addr learned my addr 172.21.15.88:0/2648944002 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:18.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.992+0000 7f4d266f1640 1 --2- 172.21.15.88:0/2648944002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9aa522540 0x55d9aa452ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:18.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.992+0000 7f4d1ffff640 1 -- 172.21.15.88:0/2648944002 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9aa453dd0 msgr2=0x55d9aa51a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:18.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.992+0000 7f4d1ffff640 1 --2- 172.21.15.88:0/2648944002 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9aa453dd0 0x55d9aa51a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:18.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.992+0000 7f4d1ffff640 1 -- 172.21.15.88:0/2648944002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9aa522540 msgr2=0x55d9aa452ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:18.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.992+0000 7f4d1ffff640 1 --2- 172.21.15.88:0/2648944002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9aa522540 0x55d9aa452ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:18.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.992+0000 7f4d1ffff640 1 -- 172.21.15.88:0/2648944002 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9aa51f250 con 0x55d9aa456230 2024-09-16T01:44:18.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.992+0000 7f4d1ffff640 1 --2- 172.21.15.88:0/2648944002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9aa456230 0x55d9aa4529a0 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f4d180099d0 tx=0x7f4d18005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=131886fa92298926 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:18.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.992+0000 7f4d24eee640 1 -- 172.21.15.88:0/2648944002 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d18008a30 con 0x55d9aa456230 2024-09-16T01:44:18.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.992+0000 7f4d24eee640 1 -- 172.21.15.88:0/2648944002 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d18018070 con 0x55d9aa456230 2024-09-16T01:44:18.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.992+0000 7f4d24eee640 1 -- 172.21.15.88:0/2648944002 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d18008dd0 con 0x55d9aa456230 2024-09-16T01:44:18.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.993+0000 7f4d283f8f40 1 -- 172.21.15.88:0/2648944002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9aa456230 msgr2=0x55d9aa4529a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:18.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.993+0000 7f4d283f8f40 1 --2- 172.21.15.88:0/2648944002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9aa456230 0x55d9aa4529a0 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f4d180099d0 tx=0x7f4d18005a00 comp rx=0 tx=0).stop 2024-09-16T01:44:18.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.993+0000 7f4d283f8f40 1 -- 172.21.15.88:0/2648944002 shutdown_connections 2024-09-16T01:44:18.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.993+0000 7f4d283f8f40 1 --2- 172.21.15.88:0/2648944002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9aa522540 0x55d9aa452ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:18.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.993+0000 7f4d283f8f40 1 --2- 172.21.15.88:0/2648944002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9aa456230 0x55d9aa4529a0 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:18.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.993+0000 7f4d283f8f40 1 --2- 172.21.15.88:0/2648944002 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9aa453dd0 0x55d9aa51a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:18.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.993+0000 7f4d283f8f40 1 -- 172.21.15.88:0/2648944002 >> 172.21.15.88:0/2648944002 conn(0x55d9aa49d750 msgr2=0x55d9aa481690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:18.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.993+0000 7f4d283f8f40 1 -- 172.21.15.88:0/2648944002 shutdown_connections 2024-09-16T01:44:18.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.993+0000 7f4d283f8f40 1 -- 172.21.15.88:0/2648944002 wait complete. 2024-09-16T01:44:18.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.993+0000 7f4d283f8f40 1 Processor -- start 2024-09-16T01:44:18.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.993+0000 7f4d283f8f40 1 -- start start 2024-09-16T01:44:18.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.994+0000 7f4d283f8f40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9aa453dd0 0x55d9aa51e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:18.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.994+0000 7f4d283f8f40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9aa456230 0x55d9aa51d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:18.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.994+0000 7f4d283f8f40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9aa522540 0x55d9aa522920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:18.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.994+0000 7f4d283f8f40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d9aa520590 con 0x55d9aa522540 2024-09-16T01:44:18.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.994+0000 7f4d283f8f40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d9aa51ffd0 con 0x55d9aa453dd0 2024-09-16T01:44:18.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.994+0000 7f4d283f8f40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55d9aa520a20 con 0x55d9aa456230 2024-09-16T01:44:18.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.994+0000 7f4d266f1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9aa522540 0x55d9aa522920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:18.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.994+0000 7f4d266f1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9aa522540 0x55d9aa522920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50572/0 (socket says 172.21.15.88:50572) 2024-09-16T01:44:18.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.994+0000 7f4d1ffff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9aa456230 0x55d9aa51d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:18.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.994+0000 7f4d266f1640 1 -- 172.21.15.88:0/1136740913 learned_addr learned my addr 172.21.15.88:0/1136740913 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:18.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.994+0000 7f4d25ef0640 1 --2- 172.21.15.88:0/1136740913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9aa453dd0 0x55d9aa51e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:18.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.994+0000 7f4d266f1640 1 -- 172.21.15.88:0/1136740913 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9aa456230 msgr2=0x55d9aa51d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:18.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.994+0000 7f4d266f1640 1 --2- 172.21.15.88:0/1136740913 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d9aa456230 0x55d9aa51d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:18.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.994+0000 7f4d266f1640 1 -- 172.21.15.88:0/1136740913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9aa453dd0 msgr2=0x55d9aa51e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:18.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.994+0000 7f4d266f1640 1 --2- 172.21.15.88:0/1136740913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9aa453dd0 0x55d9aa51e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:18.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.995+0000 7f4d266f1640 1 -- 172.21.15.88:0/1136740913 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9aa518e10 con 0x55d9aa522540 2024-09-16T01:44:18.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.995+0000 7f4d266f1640 1 --2- 172.21.15.88:0/1136740913 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9aa522540 0x55d9aa522920 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f4d20003ea0 tx=0x7f4d20003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:18.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.995+0000 7f4d1effd640 1 -- 172.21.15.88:0/1136740913 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d200020a0 con 0x55d9aa522540 2024-09-16T01:44:18.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.995+0000 7f4d1effd640 1 -- 172.21.15.88:0/1136740913 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d20017070 con 0x55d9aa522540 2024-09-16T01:44:18.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.995+0000 7f4d1effd640 1 -- 172.21.15.88:0/1136740913 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d20011780 con 0x55d9aa522540 2024-09-16T01:44:18.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.995+0000 7f4d283f8f40 1 -- 172.21.15.88:0/1136740913 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d9aa5190b0 con 0x55d9aa522540 2024-09-16T01:44:18.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.995+0000 7f4d283f8f40 1 -- 172.21.15.88:0/1136740913 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9aa51a2a0 con 0x55d9aa522540 2024-09-16T01:44:18.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.996+0000 7f4d1effd640 1 -- 172.21.15.88:0/1136740913 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4d20011920 con 0x55d9aa522540 2024-09-16T01:44:18.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.997+0000 7f4d1effd640 1 --2- 172.21.15.88:0/1136740913 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d0c06ac90 0x7f4d0c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:18.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.997+0000 7f4d25ef0640 1 --2- 172.21.15.88:0/1136740913 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d0c06ac90 0x7f4d0c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:18.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.997+0000 7f4d1effd640 1 -- 172.21.15.88:0/1136740913 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 8920+0+0 (secure 0 0 0) 0x7f4d2008ab30 con 0x55d9aa522540 2024-09-16T01:44:18.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.997+0000 7f4d283f8f40 1 --2- 172.21.15.88:0/1136740913 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55d9aa5b8450 0x55d9aa5c2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:18.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.997+0000 7f4d283f8f40 1 -- 172.21.15.88:0/1136740913 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:1 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124) -- 0x55d9aa51d5c0 con 0x55d9aa5b8450 2024-09-16T01:44:18.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.997+0000 7f4d1ffff640 1 --2- 172.21.15.88:0/1136740913 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55d9aa5b8450 0x55d9aa5c2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:18.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.997+0000 7f4d25ef0640 1 --2- 172.21.15.88:0/1136740913 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d0c06ac90 0x7f4d0c06d150 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f4d10057000 tx=0x7f4d10052000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:18.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.997+0000 7f4d1ffff640 1 --2- 172.21.15.88:0/1136740913 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55d9aa5b8450 0x55d9aa5c2280 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:18.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.999+0000 7f4d1ffff640 1 -- 172.21.15.88:0/1136740913 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(1 100.00000000 [omap-set-header] v124'375 uv375 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4d180088b0 con 0x55d9aa5b8450 2024-09-16T01:44:18.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.999+0000 7f4d283f8f40 1 -- 172.21.15.88:0/1136740913 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55d9aa5b8450 msgr2=0x55d9aa5c2280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:18.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.999+0000 7f4d283f8f40 1 --2- 172.21.15.88:0/1136740913 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55d9aa5b8450 0x55d9aa5c2280 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:18.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.999+0000 7f4d283f8f40 1 -- 172.21.15.88:0/1136740913 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d0c06ac90 msgr2=0x7f4d0c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:18.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.999+0000 7f4d283f8f40 1 --2- 172.21.15.88:0/1136740913 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d0c06ac90 0x7f4d0c06d150 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f4d10057000 tx=0x7f4d10052000 comp rx=0 tx=0).stop 2024-09-16T01:44:18.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.999+0000 7f4d283f8f40 1 -- 172.21.15.88:0/1136740913 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9aa522540 msgr2=0x55d9aa522920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:18.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.999+0000 7f4d283f8f40 1 --2- 172.21.15.88:0/1136740913 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d9aa522540 0x55d9aa522920 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f4d20003ea0 tx=0x7f4d20003f90 comp rx=0 tx=0).stop 2024-09-16T01:44:18.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.999+0000 7f4d266f1640 1 -- 172.21.15.88:0/1136740913 reap_dead start 2024-09-16T01:44:18.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.999+0000 7f4d283f8f40 1 -- 172.21.15.88:0/1136740913 shutdown_connections 2024-09-16T01:44:18.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.999+0000 7f4d283f8f40 1 -- 172.21.15.88:0/1136740913 >> 172.21.15.88:0/1136740913 conn(0x55d9aa49d750 msgr2=0x55d9aa527c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:18.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.999+0000 7f4d283f8f40 1 -- 172.21.15.88:0/1136740913 shutdown_connections 2024-09-16T01:44:18.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:17.999+0000 7f4d283f8f40 1 -- 172.21.15.88:0/1136740913 wait complete. 2024-09-16T01:44:18.009 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:44:18.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.109+0000 7f0bc0398640 1 Processor -- start 2024-09-16T01:44:18.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.109+0000 7f0bc0398640 1 -- start start 2024-09-16T01:44:18.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.110+0000 7f0bc0398640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0bb815b480 0x7f0bb80b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:18.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.110+0000 7f0bc0398640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0bb8072700 0x7f0bb815bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:18.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.110+0000 7f0bc0398640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bb80ccff0 con 0x7f0bb80b60c0 2024-09-16T01:44:18.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.110+0000 7f0bc0398640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0bb80cd2d0 con 0x7f0bb815b480 2024-09-16T01:44:18.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.110+0000 7f0bc0398640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0bb80cd5b0 con 0x7f0bb8072700 2024-09-16T01:44:18.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.110+0000 7f0bbd933640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0bb80b60c0 0x7f0bb8071ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:37350/0 (socket says 172.21.15.88:37350) 2024-09-16T01:44:18.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.110+0000 7f0bbd933640 1 -- 172.21.15.88:0/492235621 learned_addr learned my addr 172.21.15.88:0/492235621 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:18.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.110+0000 7f0bbd132640 1 -- 172.21.15.88:0/492235621 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1836453131 0 0) 0x7f0bb80ccff0 con 0x7f0bb80b60c0 2024-09-16T01:44:18.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.110+0000 7f0bbe134640 1 --2- 172.21.15.88:0/492235621 >> v2:172.21.15.154:3300/0 conn(0x7f0bb815b480 0x7f0bb80b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:18.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.110+0000 7f0bbd132640 1 -- 172.21.15.88:0/492235621 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ba4003520 con 0x7f0bb80b60c0 2024-09-16T01:44:18.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.110+0000 7f0bbe935640 1 --2- 172.21.15.88:0/492235621 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0bb8072700 0x7f0bb815bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:18.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.111+0000 7f0bbd132640 1 -- 172.21.15.88:0/492235621 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2447692627 0 0) 0x7f0ba4003520 con 0x7f0bb80b60c0 2024-09-16T01:44:18.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.111+0000 7f0bbd132640 1 -- 172.21.15.88:0/492235621 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0bb8072700 msgr2=0x7f0bb815bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:18.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.111+0000 7f0bbd132640 1 --2- 172.21.15.88:0/492235621 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0bb8072700 0x7f0bb815bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:18.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.111+0000 7f0bbd132640 1 -- 172.21.15.88:0/492235621 >> v2:172.21.15.154:3300/0 conn(0x7f0bb815b480 msgr2=0x7f0bb80b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:18.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.111+0000 7f0bbd132640 1 --2- 172.21.15.88:0/492235621 >> v2:172.21.15.154:3300/0 conn(0x7f0bb815b480 0x7f0bb80b5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:18.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.111+0000 7f0bbd132640 1 -- 172.21.15.88:0/492235621 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bb80cd890 con 0x7f0bb80b60c0 2024-09-16T01:44:18.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.111+0000 7f0bbd132640 1 -- 172.21.15.88:0/492235621 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f0ba8001670 con 0x7f0bb80b60c0 2024-09-16T01:44:18.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.111+0000 7f0bbd132640 1 -- 172.21.15.88:0/492235621 >> v1:172.21.15.88:6789/0 conn(0x7f0bb80b60c0 legacy=0x7f0bb8071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:18.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.111+0000 7f0bbd132640 1 --2- 172.21.15.88:0/492235621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ba40039b0 0x7f0ba4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:18.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.111+0000 7f0bbd132640 1 --2- 172.21.15.88:0/492235621 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ba4004800 0x7f0ba4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:18.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.111+0000 7f0bbd132640 1 --2- 172.21.15.88:0/492235621 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0bb8072700 0x7f0ba40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:18.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.111+0000 7f0bbd132640 1 -- 172.21.15.88:0/492235621 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0bb80cd2d0 con 0x7f0ba4004800 2024-09-16T01:44:18.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.111+0000 7f0bbd132640 1 -- 172.21.15.88:0/492235621 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0bb80cd5b0 con 0x7f0ba40039b0 2024-09-16T01:44:18.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.111+0000 7f0bbd132640 1 -- 172.21.15.88:0/492235621 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0ba400dcb0 con 0x7f0bb8072700 2024-09-16T01:44:18.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.111+0000 7f0bbd933640 1 --2- 172.21.15.88:0/492235621 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ba4004800 0x7f0ba4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:18.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.111+0000 7f0bbe134640 1 --2- 172.21.15.88:0/492235621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ba40039b0 0x7f0ba4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:18.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.111+0000 7f0bbe935640 1 --2- 172.21.15.88:0/492235621 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0bb8072700 0x7f0ba40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:18.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.111+0000 7f0bbd933640 1 -- 172.21.15.88:0/492235621 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0bb8072700 msgr2=0x7f0ba40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:18.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.111+0000 7f0bbd933640 1 --2- 172.21.15.88:0/492235621 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0bb8072700 0x7f0ba40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:18.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.111+0000 7f0bbd933640 1 -- 172.21.15.88:0/492235621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ba40039b0 msgr2=0x7f0ba4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:18.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.111+0000 7f0bbd933640 1 --2- 172.21.15.88:0/492235621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ba40039b0 0x7f0ba4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:18.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.111+0000 7f0bbd933640 1 -- 172.21.15.88:0/492235621 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ba4003520 con 0x7f0ba4004800 2024-09-16T01:44:18.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.112+0000 7f0bbd933640 1 --2- 172.21.15.88:0/492235621 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ba4004800 0x7f0ba4004c80 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8007fd0 tx=0x7f0ba4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6573d4751934f921 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:18.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.112+0000 7f0bbd132640 1 -- 172.21.15.88:0/492235621 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba8007d50 con 0x7f0ba4004800 2024-09-16T01:44:18.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.112+0000 7f0bbd132640 1 -- 172.21.15.88:0/492235621 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ba8005ad0 con 0x7f0ba4004800 2024-09-16T01:44:18.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.112+0000 7f0bbd132640 1 -- 172.21.15.88:0/492235621 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba8007070 con 0x7f0ba4004800 2024-09-16T01:44:18.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.112+0000 7f0bc0398640 1 -- 172.21.15.88:0/492235621 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ba4004800 msgr2=0x7f0ba4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:18.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.112+0000 7f0bc0398640 1 --2- 172.21.15.88:0/492235621 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ba4004800 0x7f0ba4004c80 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8007fd0 tx=0x7f0ba4005830 comp rx=0 tx=0).stop 2024-09-16T01:44:18.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.112+0000 7f0bbe935640 1 -- 172.21.15.88:0/492235621 reap_dead start 2024-09-16T01:44:18.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.113+0000 7f0bc0398640 1 -- 172.21.15.88:0/492235621 shutdown_connections 2024-09-16T01:44:18.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.113+0000 7f0bc0398640 1 -- 172.21.15.88:0/492235621 >> 172.21.15.88:0/492235621 conn(0x7f0bb806d0f0 msgr2=0x7f0bb806d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:18.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.113+0000 7f0bc0398640 1 -- 172.21.15.88:0/492235621 shutdown_connections 2024-09-16T01:44:18.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.113+0000 7f0bc0398640 1 -- 172.21.15.88:0/492235621 wait complete. 2024-09-16T01:44:18.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.114+0000 7f0bc0398640 1 Processor -- start 2024-09-16T01:44:18.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.114+0000 7f0bc0398640 1 -- start start 2024-09-16T01:44:18.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.114+0000 7f0bc0398640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ba4004800 0x7f0bb8144180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:18.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.114+0000 7f0bc0398640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bb8152b90 0x7f0bb8153010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:18.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.114+0000 7f0bc0398640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0bb814e230 0x7f0bb814e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:18.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.114+0000 7f0bc0398640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0bb815b9e0 con 0x7f0bb8152b90 2024-09-16T01:44:18.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.114+0000 7f0bc0398640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0bb815b860 con 0x7f0ba4004800 2024-09-16T01:44:18.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.114+0000 7f0bc0398640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0bb80b3590 con 0x7f0bb814e230 2024-09-16T01:44:18.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.115+0000 7f0bbd933640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bb8152b90 0x7f0bb8153010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:18.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.115+0000 7f0bbd933640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bb8152b90 0x7f0bb8153010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50590/0 (socket says 172.21.15.88:50590) 2024-09-16T01:44:18.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.115+0000 7f0bbd933640 1 -- 172.21.15.88:0/956755609 learned_addr learned my addr 172.21.15.88:0/956755609 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:18.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.115+0000 7f0bbe935640 1 --2- 172.21.15.88:0/956755609 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0bb814e230 0x7f0bb814e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:18.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.115+0000 7f0bbe134640 1 --2- 172.21.15.88:0/956755609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ba4004800 0x7f0bb8144180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:18.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.115+0000 7f0bbd933640 1 -- 172.21.15.88:0/956755609 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0bb814e230 msgr2=0x7f0bb814e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:18.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.115+0000 7f0bbd933640 1 --2- 172.21.15.88:0/956755609 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0bb814e230 0x7f0bb814e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:18.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.115+0000 7f0bbd933640 1 -- 172.21.15.88:0/956755609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ba4004800 msgr2=0x7f0bb8144180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:18.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.115+0000 7f0bbd933640 1 --2- 172.21.15.88:0/956755609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ba4004800 0x7f0bb8144180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:18.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.115+0000 7f0bbd933640 1 -- 172.21.15.88:0/956755609 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bb81538c0 con 0x7f0bb8152b90 2024-09-16T01:44:18.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.115+0000 7f0bbd933640 1 --2- 172.21.15.88:0/956755609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bb8152b90 0x7f0bb8153010 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f0ba800c2c0 tx=0x7f0bb80cd1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:18.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.115+0000 7f0baeffd640 1 -- 172.21.15.88:0/956755609 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba8037040 con 0x7f0bb8152b90 2024-09-16T01:44:18.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.115+0000 7f0baeffd640 1 -- 172.21.15.88:0/956755609 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ba8041070 con 0x7f0bb8152b90 2024-09-16T01:44:18.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.115+0000 7f0bc0398640 1 -- 172.21.15.88:0/956755609 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bb8153ba0 con 0x7f0bb8152b90 2024-09-16T01:44:18.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.115+0000 7f0baeffd640 1 -- 172.21.15.88:0/956755609 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba80047e0 con 0x7f0bb8152b90 2024-09-16T01:44:18.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.115+0000 7f0bc0398640 1 -- 172.21.15.88:0/956755609 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bb8154160 con 0x7f0bb8152b90 2024-09-16T01:44:18.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.116+0000 7f0bc0398640 1 -- 172.21.15.88:0/956755609 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bb80b5b80 con 0x7f0bb8152b90 2024-09-16T01:44:18.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.117+0000 7f0baeffd640 1 -- 172.21.15.88:0/956755609 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0ba8004980 con 0x7f0bb8152b90 2024-09-16T01:44:18.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.117+0000 7f0baeffd640 1 --2- 172.21.15.88:0/956755609 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0b9c06c2e0 0x7f0b9c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:18.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.117+0000 7f0baeffd640 1 -- 172.21.15.88:0/956755609 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 8920+0+0 (secure 0 0 0) 0x7f0ba80ab9e0 con 0x7f0bb8152b90 2024-09-16T01:44:18.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.117+0000 7f0bbe134640 1 --2- 172.21.15.88:0/956755609 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0b9c06c2e0 0x7f0b9c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:18.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.118+0000 7f0bbe134640 1 --2- 172.21.15.88:0/956755609 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0b9c06c2e0 0x7f0b9c06e7a0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f0bb4009b00 tx=0x7f0bb4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:18.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.121+0000 7f0baeffd640 1 -- 172.21.15.88:0/956755609 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0ba8079500 con 0x7f0bb8152b90 2024-09-16T01:44:18.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:18.321+0000 7f0bc0398640 1 -- 172.21.15.88:0/956755609 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f0bb8159fc0 con 0x7f0bb8152b90 2024-09-16T01:44:18.551 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:44:18.553 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:44:19.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.024+0000 7f0baeffd640 1 -- 172.21.15.88:0/956755609 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v300) ==== 162+0+0 (secure 0 0 0) 0x7f0ba8078e00 con 0x7f0bb8152b90 2024-09-16T01:44:19.026 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:44:19.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.026+0000 7f0bc0398640 1 -- 172.21.15.88:0/956755609 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0b9c06c2e0 msgr2=0x7f0b9c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:19.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.026+0000 7f0bc0398640 1 --2- 172.21.15.88:0/956755609 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0b9c06c2e0 0x7f0b9c06e7a0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f0bb4009b00 tx=0x7f0bb4008040 comp rx=0 tx=0).stop 2024-09-16T01:44:19.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.026+0000 7f0bc0398640 1 -- 172.21.15.88:0/956755609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bb8152b90 msgr2=0x7f0bb8153010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:19.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.026+0000 7f0bc0398640 1 --2- 172.21.15.88:0/956755609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bb8152b90 0x7f0bb8153010 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f0ba800c2c0 tx=0x7f0bb80cd1c0 comp rx=0 tx=0).stop 2024-09-16T01:44:19.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.026+0000 7f0bc0398640 1 -- 172.21.15.88:0/956755609 shutdown_connections 2024-09-16T01:44:19.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.026+0000 7f0bc0398640 1 --2- 172.21.15.88:0/956755609 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0b9c06c2e0 0x7f0b9c06e7a0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:19.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.026+0000 7f0bc0398640 1 --2- 172.21.15.88:0/956755609 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0bb814e230 0x7f0bb814e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:19.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.026+0000 7f0bc0398640 1 --2- 172.21.15.88:0/956755609 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0bb8152b90 0x7f0bb8153010 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:19.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.026+0000 7f0bc0398640 1 --2- 172.21.15.88:0/956755609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ba4004800 0x7f0bb8144180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:19.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.026+0000 7f0bc0398640 1 -- 172.21.15.88:0/956755609 >> 172.21.15.88:0/956755609 conn(0x7f0bb806d0f0 msgr2=0x7f0bb8071ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:19.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.027+0000 7f0bc0398640 1 -- 172.21.15.88:0/956755609 shutdown_connections 2024-09-16T01:44:19.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.027+0000 7f0bc0398640 1 -- 172.21.15.88:0/956755609 wait complete. 2024-09-16T01:44:19.041 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:44:19.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.140+0000 7f2b4d053640 1 Processor -- start 2024-09-16T01:44:19.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.140+0000 7f2b4d053640 1 -- start start 2024-09-16T01:44:19.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.141+0000 7f2b4d053640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4815e540 0x7f2b480ae890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:19.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.141+0000 7f2b4d053640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2b48072d50 0x7f2b4806e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:19.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.141+0000 7f2b4d053640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2b4806e630 0x7f2b48071810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:19.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.141+0000 7f2b4d053640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2b480aedd0 con 0x7f2b4806e630 2024-09-16T01:44:19.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.141+0000 7f2b4d053640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2b480b52c0 con 0x7f2b48072d50 2024-09-16T01:44:19.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.141+0000 7f2b4d053640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2b480af000 con 0x7f2b4815e540 2024-09-16T01:44:19.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.141+0000 7f2b4759e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2b4806e630 0x7f2b48071810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:19.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.141+0000 7f2b4759e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2b4806e630 0x7f2b48071810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50606/0 (socket says 172.21.15.88:50606) 2024-09-16T01:44:19.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.141+0000 7f2b4759e640 1 -- 172.21.15.88:0/3233334474 learned_addr learned my addr 172.21.15.88:0/3233334474 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:19.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.141+0000 7f2b4759e640 1 -- 172.21.15.88:0/3233334474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4815e540 msgr2=0x7f2b480ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:19.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.141+0000 7f2b4659c640 1 --2- 172.21.15.88:0/3233334474 >> v2:172.21.15.154:3300/0 conn(0x7f2b48072d50 0x7f2b4806e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:19.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.141+0000 7f2b46d9d640 1 --2- 172.21.15.88:0/3233334474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4815e540 0x7f2b480ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:19.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.141+0000 7f2b4759e640 1 --2- 172.21.15.88:0/3233334474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4815e540 0x7f2b480ae890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:19.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.141+0000 7f2b4759e640 1 -- 172.21.15.88:0/3233334474 >> v2:172.21.15.154:3300/0 conn(0x7f2b48072d50 msgr2=0x7f2b4806e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:19.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.141+0000 7f2b4759e640 1 --2- 172.21.15.88:0/3233334474 >> v2:172.21.15.154:3300/0 conn(0x7f2b48072d50 0x7f2b4806e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:19.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.141+0000 7f2b4759e640 1 -- 172.21.15.88:0/3233334474 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b480af2e0 con 0x7f2b4806e630 2024-09-16T01:44:19.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.142+0000 7f2b4759e640 1 --2- 172.21.15.88:0/3233334474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b4806e630 0x7f2b48071810 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f2b4806ec80 tx=0x7f2b4002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cf8af39a18fc153 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:19.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.142+0000 7f2b45d9b640 1 -- 172.21.15.88:0/3233334474 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b40036070 con 0x7f2b4806e630 2024-09-16T01:44:19.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.142+0000 7f2b45d9b640 1 -- 172.21.15.88:0/3233334474 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b40031600 con 0x7f2b4806e630 2024-09-16T01:44:19.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.142+0000 7f2b45d9b640 1 -- 172.21.15.88:0/3233334474 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b400319d0 con 0x7f2b4806e630 2024-09-16T01:44:19.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.143+0000 7f2b4d053640 1 -- 172.21.15.88:0/3233334474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b4806e630 msgr2=0x7f2b48071810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:19.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.143+0000 7f2b4d053640 1 --2- 172.21.15.88:0/3233334474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b4806e630 0x7f2b48071810 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f2b4806ec80 tx=0x7f2b4002a610 comp rx=0 tx=0).stop 2024-09-16T01:44:19.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.143+0000 7f2b4d053640 1 -- 172.21.15.88:0/3233334474 shutdown_connections 2024-09-16T01:44:19.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.143+0000 7f2b4d053640 1 --2- 172.21.15.88:0/3233334474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b4806e630 0x7f2b48071810 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:19.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.143+0000 7f2b4d053640 1 --2- 172.21.15.88:0/3233334474 >> v2:172.21.15.154:3300/0 conn(0x7f2b48072d50 0x7f2b4806e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:19.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.143+0000 7f2b4d053640 1 --2- 172.21.15.88:0/3233334474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4815e540 0x7f2b480ae890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:19.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.143+0000 7f2b4d053640 1 -- 172.21.15.88:0/3233334474 >> 172.21.15.88:0/3233334474 conn(0x7f2b4806bc10 msgr2=0x7f2b4806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:19.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.143+0000 7f2b4d053640 1 -- 172.21.15.88:0/3233334474 shutdown_connections 2024-09-16T01:44:19.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.143+0000 7f2b4d053640 1 -- 172.21.15.88:0/3233334474 wait complete. 2024-09-16T01:44:19.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.144+0000 7f2b4d053640 1 Processor -- start 2024-09-16T01:44:19.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.144+0000 7f2b4d053640 1 -- start start 2024-09-16T01:44:19.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.145+0000 7f2b4d053640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b4806e630 0x7f2b481581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:19.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.145+0000 7f2b4d053640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b48072d50 0x7f2b480c67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:19.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.145+0000 7f2b4d053640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4815e540 0x7f2b480cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:19.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.145+0000 7f2b4d053640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4815e920 con 0x7f2b4806e630 2024-09-16T01:44:19.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.145+0000 7f2b4d053640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2b480af8a0 con 0x7f2b48072d50 2024-09-16T01:44:19.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.145+0000 7f2b4d053640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2b48071d50 con 0x7f2b4815e540 2024-09-16T01:44:19.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.145+0000 7f2b46d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b4806e630 0x7f2b481581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:19.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.145+0000 7f2b46d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b4806e630 0x7f2b481581b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:50610/0 (socket says 172.21.15.88:50610) 2024-09-16T01:44:19.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.145+0000 7f2b46d9d640 1 -- 172.21.15.88:0/3558377094 learned_addr learned my addr 172.21.15.88:0/3558377094 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:19.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.145+0000 7f2b4659c640 1 --2- 172.21.15.88:0/3558377094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b48072d50 0x7f2b480c67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:19.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.145+0000 7f2b4759e640 1 --2- 172.21.15.88:0/3558377094 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4815e540 0x7f2b480cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:19.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.146+0000 7f2b46d9d640 1 -- 172.21.15.88:0/3558377094 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4815e540 msgr2=0x7f2b480cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:19.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.146+0000 7f2b46d9d640 1 --2- 172.21.15.88:0/3558377094 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4815e540 0x7f2b480cfb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:19.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.146+0000 7f2b46d9d640 1 -- 172.21.15.88:0/3558377094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b48072d50 msgr2=0x7f2b480c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:19.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.146+0000 7f2b46d9d640 1 --2- 172.21.15.88:0/3558377094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b48072d50 0x7f2b480c67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:19.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.146+0000 7f2b46d9d640 1 -- 172.21.15.88:0/3558377094 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b481586f0 con 0x7f2b4806e630 2024-09-16T01:44:19.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.146+0000 7f2b4759e640 1 --2- 172.21.15.88:0/3558377094 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4815e540 0x7f2b480cfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:19.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.146+0000 7f2b46d9d640 1 --2- 172.21.15.88:0/3558377094 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b4806e630 0x7f2b481581b0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f2b3800c7f0 tx=0x7f2b3800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:19.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.146+0000 7f2b2f7fe640 1 -- 172.21.15.88:0/3558377094 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b38015070 con 0x7f2b4806e630 2024-09-16T01:44:19.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.146+0000 7f2b2f7fe640 1 -- 172.21.15.88:0/3558377094 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b38004220 con 0x7f2b4806e630 2024-09-16T01:44:19.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.146+0000 7f2b4d053640 1 -- 172.21.15.88:0/3558377094 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b4815c4d0 con 0x7f2b4806e630 2024-09-16T01:44:19.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.146+0000 7f2b2f7fe640 1 -- 172.21.15.88:0/3558377094 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b38007620 con 0x7f2b4806e630 2024-09-16T01:44:19.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.146+0000 7f2b4d053640 1 -- 172.21.15.88:0/3558377094 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b4815ca90 con 0x7f2b4806e630 2024-09-16T01:44:19.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.147+0000 7f2b4d053640 1 -- 172.21.15.88:0/3558377094 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b4806e0f0 con 0x7f2b4806e630 2024-09-16T01:44:19.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.147+0000 7f2b2f7fe640 1 -- 172.21.15.88:0/3558377094 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2b380079d0 con 0x7f2b4806e630 2024-09-16T01:44:19.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.148+0000 7f2b2f7fe640 1 --2- 172.21.15.88:0/3558377094 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b3406c2b0 0x7f2b3406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:19.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.148+0000 7f2b2f7fe640 1 -- 172.21.15.88:0/3558377094 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 8920+0+0 (secure 0 0 0) 0x7f2b3808c250 con 0x7f2b4806e630 2024-09-16T01:44:19.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.148+0000 7f2b4659c640 1 --2- 172.21.15.88:0/3558377094 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b3406c2b0 0x7f2b3406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:19.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.148+0000 7f2b4659c640 1 --2- 172.21.15.88:0/3558377094 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b3406c2b0 0x7f2b3406e770 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f2b48158cc0 tx=0x7f2b3c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:19.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.152+0000 7f2b2f7fe640 1 -- 172.21.15.88:0/3558377094 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2b380597c0 con 0x7f2b4806e630 2024-09-16T01:44:19.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.354+0000 7f2b4d053640 1 -- 172.21.15.88:0/3558377094 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b4806d1e0 con 0x7f2b4806e630 2024-09-16T01:44:19.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.355+0000 7f2b2f7fe640 1 -- 172.21.15.88:0/3558377094 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 301 v301) ==== 95+0+5230 (secure 0 0 0) 0x7f2b3805dac0 con 0x7f2b4806e630 2024-09-16T01:44:19.485 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:19.486 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":301,"btime":"2024-09-16T01:44:19:027436+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9696,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4270197461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4270197461},{"type":"v1","addr":"172.21.15.88:6837","nonce":4270197461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":301,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:44:19.027421+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10224},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10224":{"gid":10224,"name":"a","rank":0,"incarnation":301,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6836/1099469537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":1099469537},{"type":"v1","addr":"172.21.15.88:6836","nonce":1099469537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:44:19.487 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 301 2024-09-16T01:44:19.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.357+0000 7f2b4d053640 1 -- 172.21.15.88:0/3558377094 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b3406c2b0 msgr2=0x7f2b3406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:19.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.357+0000 7f2b4d053640 1 --2- 172.21.15.88:0/3558377094 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b3406c2b0 0x7f2b3406e770 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f2b48158cc0 tx=0x7f2b3c008040 comp rx=0 tx=0).stop 2024-09-16T01:44:19.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.357+0000 7f2b4d053640 1 -- 172.21.15.88:0/3558377094 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b4806e630 msgr2=0x7f2b481581b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:19.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.357+0000 7f2b4d053640 1 --2- 172.21.15.88:0/3558377094 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b4806e630 0x7f2b481581b0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f2b3800c7f0 tx=0x7f2b3800ccb0 comp rx=0 tx=0).stop 2024-09-16T01:44:19.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.357+0000 7f2b4d053640 1 -- 172.21.15.88:0/3558377094 shutdown_connections 2024-09-16T01:44:19.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.357+0000 7f2b4d053640 1 --2- 172.21.15.88:0/3558377094 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2b3406c2b0 0x7f2b3406e770 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:19.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.357+0000 7f2b4d053640 1 --2- 172.21.15.88:0/3558377094 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b4815e540 0x7f2b480cfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:19.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.357+0000 7f2b4d053640 1 --2- 172.21.15.88:0/3558377094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b48072d50 0x7f2b480c67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:19.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.357+0000 7f2b4d053640 1 --2- 172.21.15.88:0/3558377094 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2b4806e630 0x7f2b481581b0 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:19.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.357+0000 7f2b4d053640 1 -- 172.21.15.88:0/3558377094 >> 172.21.15.88:0/3558377094 conn(0x7f2b4806bc10 msgr2=0x7f2b48073250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:19.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.357+0000 7f2b4d053640 1 -- 172.21.15.88:0/3558377094 shutdown_connections 2024-09-16T01:44:19.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:19.357+0000 7f2b4d053640 1 -- 172.21.15.88:0/3558377094 wait complete. 2024-09-16T01:44:19.488 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-09-16T01:44:22.053 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:44:22.052+0000 7f634af03640 -1 mds.0.cache.dir(0x100) Corrupt fnode in dirfrag 0x100: Decoder at 'void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&)' v=3 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-09-16T01:44:22.055 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:44:22.054+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:44:22.055 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:44:22.054+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:44:22.649 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:44:22.648+0000 7f634ef0b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:44:24.490 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:44:24.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.599+0000 7f0aef2d0640 1 Processor -- start 2024-09-16T01:44:24.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.599+0000 7f0aef2d0640 1 -- start start 2024-09-16T01:44:24.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.600+0000 7f0aef2d0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0ae8155800 0x7f0ae80b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:24.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.600+0000 7f0aef2d0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ae815c330 0x7f0ae80b4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:24.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.600+0000 7f0aef2d0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ae80b50a0 con 0x7f0ae80cf570 2024-09-16T01:44:24.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.600+0000 7f0aef2d0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0ae80b4e60 con 0x7f0ae8155800 2024-09-16T01:44:24.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.600+0000 7f0aef2d0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0ae815c7a0 con 0x7f0ae815c330 2024-09-16T01:44:24.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.600+0000 7f0aed86d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0ae80cf570 0x7f0ae815c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44174/0 (socket says 172.21.15.88:44174) 2024-09-16T01:44:24.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.600+0000 7f0aed86d640 1 -- 172.21.15.88:0/3068341443 learned_addr learned my addr 172.21.15.88:0/3068341443 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:24.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.600+0000 7f0adffff640 1 -- 172.21.15.88:0/3068341443 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4098932393 0 0) 0x7f0ae80b50a0 con 0x7f0ae80cf570 2024-09-16T01:44:24.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.600+0000 7f0aec86b640 1 --2- 172.21.15.88:0/3068341443 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ae815c330 0x7f0ae80b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:24.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.601+0000 7f0adffff640 1 -- 172.21.15.88:0/3068341443 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ad4003520 con 0x7f0ae80cf570 2024-09-16T01:44:24.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.600+0000 7f0aed06c640 1 --2- 172.21.15.88:0/3068341443 >> v2:172.21.15.154:3300/0 conn(0x7f0ae8155800 0x7f0ae80b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:24.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.601+0000 7f0adffff640 1 -- 172.21.15.88:0/3068341443 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 496244942 0 0) 0x7f0ad4003520 con 0x7f0ae80cf570 2024-09-16T01:44:24.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.601+0000 7f0adffff640 1 -- 172.21.15.88:0/3068341443 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ae815c330 msgr2=0x7f0ae80b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:24.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.601+0000 7f0adffff640 1 --2- 172.21.15.88:0/3068341443 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ae815c330 0x7f0ae80b4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:24.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.601+0000 7f0adffff640 1 -- 172.21.15.88:0/3068341443 >> v2:172.21.15.154:3300/0 conn(0x7f0ae8155800 msgr2=0x7f0ae80b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:24.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.601+0000 7f0adffff640 1 --2- 172.21.15.88:0/3068341443 >> v2:172.21.15.154:3300/0 conn(0x7f0ae8155800 0x7f0ae80b6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:24.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.601+0000 7f0adffff640 1 -- 172.21.15.88:0/3068341443 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae80b5380 con 0x7f0ae80cf570 2024-09-16T01:44:24.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.601+0000 7f0adffff640 1 -- 172.21.15.88:0/3068341443 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f0ae00019f0 con 0x7f0ae80cf570 2024-09-16T01:44:24.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.601+0000 7f0adffff640 1 -- 172.21.15.88:0/3068341443 >> v1:172.21.15.88:6789/0 conn(0x7f0ae80cf570 legacy=0x7f0ae815c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:24.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.601+0000 7f0adffff640 1 --2- 172.21.15.88:0/3068341443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ad40039b0 0x7f0ad4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:24.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.601+0000 7f0adffff640 1 --2- 172.21.15.88:0/3068341443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ad4004800 0x7f0ad4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:24.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.601+0000 7f0adffff640 1 --2- 172.21.15.88:0/3068341443 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ae815c330 0x7f0ad40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:24.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.601+0000 7f0adffff640 1 -- 172.21.15.88:0/3068341443 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae80b4e60 con 0x7f0ad4004800 2024-09-16T01:44:24.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.601+0000 7f0adffff640 1 -- 172.21.15.88:0/3068341443 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae815c7a0 con 0x7f0ad40039b0 2024-09-16T01:44:24.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.601+0000 7f0adffff640 1 -- 172.21.15.88:0/3068341443 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0ad400b9c0 con 0x7f0ae815c330 2024-09-16T01:44:24.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.601+0000 7f0aec86b640 1 --2- 172.21.15.88:0/3068341443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ad4004800 0x7f0ad4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:24.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.601+0000 7f0aed06c640 1 --2- 172.21.15.88:0/3068341443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ad40039b0 0x7f0ad4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:24.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.601+0000 7f0aed86d640 1 --2- 172.21.15.88:0/3068341443 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ae815c330 0x7f0ad40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:24.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.601+0000 7f0aec86b640 1 -- 172.21.15.88:0/3068341443 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ae815c330 msgr2=0x7f0ad40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:24.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.601+0000 7f0aec86b640 1 --2- 172.21.15.88:0/3068341443 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ae815c330 0x7f0ad40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:24.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.601+0000 7f0aec86b640 1 -- 172.21.15.88:0/3068341443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ad40039b0 msgr2=0x7f0ad4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:24.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.601+0000 7f0aec86b640 1 --2- 172.21.15.88:0/3068341443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ad40039b0 0x7f0ad4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:24.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.601+0000 7f0aec86b640 1 -- 172.21.15.88:0/3068341443 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ad4003520 con 0x7f0ad4004800 2024-09-16T01:44:24.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.602+0000 7f0aec86b640 1 --2- 172.21.15.88:0/3068341443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ad4004800 0x7f0ad4004c80 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8009ac0 tx=0x7f0ad4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=254cb80d8a5cc5f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:24.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.602+0000 7f0adffff640 1 -- 172.21.15.88:0/3068341443 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad8036070 con 0x7f0ad4004800 2024-09-16T01:44:24.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.602+0000 7f0adffff640 1 -- 172.21.15.88:0/3068341443 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ad8031400 con 0x7f0ad4004800 2024-09-16T01:44:24.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.602+0000 7f0adffff640 1 -- 172.21.15.88:0/3068341443 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad8009690 con 0x7f0ad4004800 2024-09-16T01:44:24.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.603+0000 7f0aef2d0640 1 -- 172.21.15.88:0/3068341443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ad4004800 msgr2=0x7f0ad4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:24.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.603+0000 7f0aef2d0640 1 --2- 172.21.15.88:0/3068341443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ad4004800 0x7f0ad4004c80 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8009ac0 tx=0x7f0ad4005830 comp rx=0 tx=0).stop 2024-09-16T01:44:24.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.603+0000 7f0aed86d640 1 -- 172.21.15.88:0/3068341443 reap_dead start 2024-09-16T01:44:24.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.603+0000 7f0aef2d0640 1 -- 172.21.15.88:0/3068341443 shutdown_connections 2024-09-16T01:44:24.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.603+0000 7f0aef2d0640 1 -- 172.21.15.88:0/3068341443 >> 172.21.15.88:0/3068341443 conn(0x7f0ae806bc10 msgr2=0x7f0ae806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:24.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.603+0000 7f0aef2d0640 1 -- 172.21.15.88:0/3068341443 shutdown_connections 2024-09-16T01:44:24.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.603+0000 7f0aef2d0640 1 -- 172.21.15.88:0/3068341443 wait complete. 2024-09-16T01:44:24.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.604+0000 7f0aef2d0640 1 Processor -- start 2024-09-16T01:44:24.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.604+0000 7f0aef2d0640 1 -- start start 2024-09-16T01:44:24.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.605+0000 7f0aef2d0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ad4004800 0x7f0ae81511a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:24.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.605+0000 7f0aef2d0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ae8150790 0x7f0ae81571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:24.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.605+0000 7f0aef2d0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ae8150c70 0x7f0ae8153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:24.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.605+0000 7f0aef2d0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae80b5c20 con 0x7f0ae8150790 2024-09-16T01:44:24.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.605+0000 7f0aef2d0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae80b5940 con 0x7f0ae8150c70 2024-09-16T01:44:24.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.605+0000 7f0aef2d0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0ae8155be0 con 0x7f0ad4004800 2024-09-16T01:44:24.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.605+0000 7f0aec86b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ae8150790 0x7f0ae81571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:24.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.605+0000 7f0aec86b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ae8150790 0x7f0ae81571b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41534/0 (socket says 172.21.15.88:41534) 2024-09-16T01:44:24.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.605+0000 7f0aec86b640 1 -- 172.21.15.88:0/2381035922 learned_addr learned my addr 172.21.15.88:0/2381035922 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:24.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.605+0000 7f0aed86d640 1 --2- 172.21.15.88:0/2381035922 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ae8150c70 0x7f0ae8153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:24.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.605+0000 7f0aed06c640 1 --2- 172.21.15.88:0/2381035922 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ad4004800 0x7f0ae81511a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:24.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.605+0000 7f0aec86b640 1 -- 172.21.15.88:0/2381035922 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ad4004800 msgr2=0x7f0ae81511a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:24.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.605+0000 7f0aec86b640 1 --2- 172.21.15.88:0/2381035922 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ad4004800 0x7f0ae81511a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:24.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.605+0000 7f0aec86b640 1 -- 172.21.15.88:0/2381035922 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ae8150c70 msgr2=0x7f0ae8153ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:24.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.605+0000 7f0aec86b640 1 --2- 172.21.15.88:0/2381035922 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ae8150c70 0x7f0ae8153ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:24.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.605+0000 7f0aec86b640 1 -- 172.21.15.88:0/2381035922 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae8151cb0 con 0x7f0ae8150790 2024-09-16T01:44:24.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.605+0000 7f0aec86b640 1 --2- 172.21.15.88:0/2381035922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ae8150790 0x7f0ae81571b0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f0ad802abb0 tx=0x7f0ad8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:24.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.606+0000 7f0addffb640 1 -- 172.21.15.88:0/2381035922 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad8041020 con 0x7f0ae8150790 2024-09-16T01:44:24.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.606+0000 7f0addffb640 1 -- 172.21.15.88:0/2381035922 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ad803c470 con 0x7f0ae8150790 2024-09-16T01:44:24.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.606+0000 7f0aef2d0640 1 -- 172.21.15.88:0/2381035922 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ae81569f0 con 0x7f0ae8150790 2024-09-16T01:44:24.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.606+0000 7f0addffb640 1 -- 172.21.15.88:0/2381035922 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad8036040 con 0x7f0ae8150790 2024-09-16T01:44:24.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.606+0000 7f0aef2d0640 1 -- 172.21.15.88:0/2381035922 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ae8158f20 con 0x7f0ae8150790 2024-09-16T01:44:24.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.607+0000 7f0aef2d0640 1 -- 172.21.15.88:0/2381035922 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ae80b6140 con 0x7f0ae8150790 2024-09-16T01:44:24.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.607+0000 7f0addffb640 1 -- 172.21.15.88:0/2381035922 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0ad8044020 con 0x7f0ae8150790 2024-09-16T01:44:24.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.608+0000 7f0addffb640 1 --2- 172.21.15.88:0/2381035922 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0acc06c2e0 0x7f0acc06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:24.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.608+0000 7f0addffb640 1 -- 172.21.15.88:0/2381035922 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9006+0+0 (secure 0 0 0) 0x7f0ad80abeb0 con 0x7f0ae8150790 2024-09-16T01:44:24.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.608+0000 7f0aed06c640 1 --2- 172.21.15.88:0/2381035922 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0acc06c2e0 0x7f0acc06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:24.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.608+0000 7f0aed06c640 1 --2- 172.21.15.88:0/2381035922 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0acc06c2e0 0x7f0acc06e7a0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f0ae81f3370 tx=0x7f0ae4009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:24.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.612+0000 7f0addffb640 1 -- 172.21.15.88:0/2381035922 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0ae8151cb0 con 0x7f0ae8150790 2024-09-16T01:44:24.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.813+0000 7f0aef2d0640 1 -- 172.21.15.88:0/2381035922 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ae8159160 con 0x7f0ae8150790 2024-09-16T01:44:24.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.814+0000 7f0addffb640 1 -- 172.21.15.88:0/2381035922 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 305 v305) ==== 95+0+4392 (secure 0 0 0) 0x7f0ad80793c0 con 0x7f0ae8150790 2024-09-16T01:44:24.816 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:24.816 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":305,"btime":"2024-09-16T01:44:22:055031+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9696,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4270197461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4270197461},{"type":"v1","addr":"172.21.15.88:6837","nonce":4270197461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":305,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:44:22.055026+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:44:24.816 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 305 2024-09-16T01:44:24.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.816+0000 7f0aef2d0640 1 -- 172.21.15.88:0/2381035922 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0acc06c2e0 msgr2=0x7f0acc06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:24.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.816+0000 7f0aef2d0640 1 --2- 172.21.15.88:0/2381035922 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0acc06c2e0 0x7f0acc06e7a0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f0ae81f3370 tx=0x7f0ae4009190 comp rx=0 tx=0).stop 2024-09-16T01:44:24.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.816+0000 7f0aef2d0640 1 -- 172.21.15.88:0/2381035922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ae8150790 msgr2=0x7f0ae81571b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:24.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.816+0000 7f0aef2d0640 1 --2- 172.21.15.88:0/2381035922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ae8150790 0x7f0ae81571b0 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f0ad802abb0 tx=0x7f0ad8004730 comp rx=0 tx=0).stop 2024-09-16T01:44:24.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.816+0000 7f0aef2d0640 1 -- 172.21.15.88:0/2381035922 shutdown_connections 2024-09-16T01:44:24.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.816+0000 7f0aef2d0640 1 --2- 172.21.15.88:0/2381035922 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0acc06c2e0 0x7f0acc06e7a0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:24.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.816+0000 7f0aef2d0640 1 --2- 172.21.15.88:0/2381035922 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ae8150c70 0x7f0ae8153ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:24.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.816+0000 7f0aef2d0640 1 --2- 172.21.15.88:0/2381035922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0ae8150790 0x7f0ae81571b0 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:24.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.816+0000 7f0aef2d0640 1 --2- 172.21.15.88:0/2381035922 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ad4004800 0x7f0ae81511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:24.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.816+0000 7f0aef2d0640 1 -- 172.21.15.88:0/2381035922 >> 172.21.15.88:0/2381035922 conn(0x7f0ae806bc10 msgr2=0x7f0ae815b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:24.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.816+0000 7f0aef2d0640 1 -- 172.21.15.88:0/2381035922 shutdown_connections 2024-09-16T01:44:24.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.816+0000 7f0aef2d0640 1 -- 172.21.15.88:0/2381035922 wait complete. 2024-09-16T01:44:24.828 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-09-16T01:44:24.829 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:44:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.925+0000 7fc534b5f640 1 Processor -- start 2024-09-16T01:44:24.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.925+0000 7fc534b5f640 1 -- start start 2024-09-16T01:44:24.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.926+0000 7fc534b5f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc53015c170 0x7fc53015c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:24.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.926+0000 7fc534b5f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5300b64e0 0x7fc5300cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:24.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.926+0000 7fc534b5f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5300b4cf0 con 0x7fc5300cf6c0 2024-09-16T01:44:24.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.926+0000 7fc534b5f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc53015ca90 con 0x7fc53015c170 2024-09-16T01:44:24.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.926+0000 7fc534b5f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc5300b49c0 con 0x7fc5300b64e0 2024-09-16T01:44:24.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.926+0000 7fc52ed9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5300b64e0 0x7fc5300cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:24.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.926+0000 7fc52e59c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc53015c170 0x7fc53015c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:24.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.926+0000 7fc525d9b640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fc5300cf6c0 0x7fc5300b5dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:55822/0 (socket says 172.21.15.88:55822) 2024-09-16T01:44:24.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.926+0000 7fc525d9b640 1 -- 172.21.15.88:0/4033230554 learned_addr learned my addr 172.21.15.88:0/4033230554 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:24.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.926+0000 7fc52e59c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc53015c170 0x7fc53015c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:45944/0 (socket says 172.21.15.88:45944) 2024-09-16T01:44:24.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.926+0000 7fc52ed9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5300b64e0 0x7fc5300cfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:35234/0 (socket says 172.21.15.88:35234) 2024-09-16T01:44:24.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.926+0000 7fc52e59c640 1 -- 172.21.15.88:0/4033230554 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5300b64e0 msgr2=0x7fc5300cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:24.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.926+0000 7fc52e59c640 1 --2- 172.21.15.88:0/4033230554 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5300b64e0 0x7fc5300cfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:24.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.926+0000 7fc52e59c640 1 -- 172.21.15.88:0/4033230554 >> v1:172.21.15.154:6789/0 conn(0x7fc5300cf6c0 legacy=0x7fc5300b5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:24.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.926+0000 7fc52e59c640 1 -- 172.21.15.88:0/4033230554 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5300b4fd0 con 0x7fc53015c170 2024-09-16T01:44:24.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.926+0000 7fc52ed9d640 1 --2- 172.21.15.88:0/4033230554 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5300b64e0 0x7fc5300cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:44:24.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.927+0000 7fc52e59c640 1 --2- 172.21.15.88:0/4033230554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc53015c170 0x7fc53015c550 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fc5200097d0 tx=0x7fc52002a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50cde0093528b715 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:24.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.927+0000 7fc52dd9b640 1 -- 172.21.15.88:0/4033230554 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc520037070 con 0x7fc53015c170 2024-09-16T01:44:24.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.927+0000 7fc52dd9b640 1 -- 172.21.15.88:0/4033230554 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc520032440 con 0x7fc53015c170 2024-09-16T01:44:24.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.927+0000 7fc52dd9b640 1 -- 172.21.15.88:0/4033230554 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc520032810 con 0x7fc53015c170 2024-09-16T01:44:24.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.928+0000 7fc534b5f640 1 -- 172.21.15.88:0/4033230554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc53015c170 msgr2=0x7fc53015c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:24.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.928+0000 7fc534b5f640 1 --2- 172.21.15.88:0/4033230554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc53015c170 0x7fc53015c550 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fc5200097d0 tx=0x7fc52002a1c0 comp rx=0 tx=0).stop 2024-09-16T01:44:24.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.928+0000 7fc534b5f640 1 -- 172.21.15.88:0/4033230554 shutdown_connections 2024-09-16T01:44:24.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.928+0000 7fc534b5f640 1 --2- 172.21.15.88:0/4033230554 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5300b64e0 0x7fc5300cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:24.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.928+0000 7fc534b5f640 1 --2- 172.21.15.88:0/4033230554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc53015c170 0x7fc53015c550 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:24.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.928+0000 7fc534b5f640 1 -- 172.21.15.88:0/4033230554 >> 172.21.15.88:0/4033230554 conn(0x7fc53006bc10 msgr2=0x7fc53006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:24.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.928+0000 7fc534b5f640 1 -- 172.21.15.88:0/4033230554 shutdown_connections 2024-09-16T01:44:24.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.928+0000 7fc534b5f640 1 -- 172.21.15.88:0/4033230554 wait complete. 2024-09-16T01:44:24.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.929+0000 7fc534b5f640 1 Processor -- start 2024-09-16T01:44:24.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.930+0000 7fc534b5f640 1 -- start start 2024-09-16T01:44:24.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.930+0000 7fc534b5f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5300b64e0 0x7fc53015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:24.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.930+0000 7fc534b5f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5300cf6c0 0x7fc53015f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:24.935 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.930+0000 7fc534b5f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc53015c170 0x7fc5301e5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:24.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.930+0000 7fc534b5f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc5300b5870 con 0x7fc5300cf6c0 2024-09-16T01:44:24.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.930+0000 7fc534b5f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc5300b5590 con 0x7fc53015c170 2024-09-16T01:44:24.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.930+0000 7fc534b5f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc5300b5b50 con 0x7fc5300b64e0 2024-09-16T01:44:24.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.930+0000 7fc525d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5300cf6c0 0x7fc53015f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:24.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.930+0000 7fc525d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5300cf6c0 0x7fc53015f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41550/0 (socket says 172.21.15.88:41550) 2024-09-16T01:44:24.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.930+0000 7fc525d9b640 1 -- 172.21.15.88:0/4097275384 learned_addr learned my addr 172.21.15.88:0/4097275384 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:24.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.930+0000 7fc52e59c640 1 --2- 172.21.15.88:0/4097275384 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5300b64e0 0x7fc53015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:24.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.930+0000 7fc52ed9d640 1 --2- 172.21.15.88:0/4097275384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc53015c170 0x7fc5301e5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:24.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.931+0000 7fc525d9b640 1 -- 172.21.15.88:0/4097275384 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5300b64e0 msgr2=0x7fc53015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:24.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.931+0000 7fc525d9b640 1 --2- 172.21.15.88:0/4097275384 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5300b64e0 0x7fc53015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:24.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.931+0000 7fc525d9b640 1 -- 172.21.15.88:0/4097275384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc53015c170 msgr2=0x7fc5301e5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:24.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.931+0000 7fc525d9b640 1 --2- 172.21.15.88:0/4097275384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc53015c170 0x7fc5301e5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:24.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.931+0000 7fc525d9b640 1 -- 172.21.15.88:0/4097275384 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc53014d5e0 con 0x7fc5300cf6c0 2024-09-16T01:44:24.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.931+0000 7fc525d9b640 1 --2- 172.21.15.88:0/4097275384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5300cf6c0 0x7fc53015f2d0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7fc51800c400 tx=0x7fc51800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:24.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.931+0000 7fc5277fe640 1 -- 172.21.15.88:0/4097275384 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc518017070 con 0x7fc5300cf6c0 2024-09-16T01:44:24.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.931+0000 7fc5277fe640 1 -- 172.21.15.88:0/4097275384 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc518005ce0 con 0x7fc5300cf6c0 2024-09-16T01:44:24.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.931+0000 7fc534b5f640 1 -- 172.21.15.88:0/4097275384 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc53013be30 con 0x7fc5300cf6c0 2024-09-16T01:44:24.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.931+0000 7fc5277fe640 1 -- 172.21.15.88:0/4097275384 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5180072f0 con 0x7fc5300cf6c0 2024-09-16T01:44:24.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.931+0000 7fc534b5f640 1 -- 172.21.15.88:0/4097275384 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc53013c3f0 con 0x7fc5300cf6c0 2024-09-16T01:44:24.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.932+0000 7fc534b5f640 1 -- 172.21.15.88:0/4097275384 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5300cfb30 con 0x7fc5300cf6c0 2024-09-16T01:44:24.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.932+0000 7fc5277fe640 1 -- 172.21.15.88:0/4097275384 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc51801a070 con 0x7fc5300cf6c0 2024-09-16T01:44:24.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.933+0000 7fc5277fe640 1 --2- 172.21.15.88:0/4097275384 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc51006c2b0 0x7fc51006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:24.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.933+0000 7fc5277fe640 1 -- 172.21.15.88:0/4097275384 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9006+0+0 (secure 0 0 0) 0x7fc51808b540 con 0x7fc5300cf6c0 2024-09-16T01:44:24.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.933+0000 7fc52e59c640 1 --2- 172.21.15.88:0/4097275384 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc51006c2b0 0x7fc51006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:24.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.933+0000 7fc52e59c640 1 --2- 172.21.15.88:0/4097275384 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc51006c2b0 0x7fc51006e770 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fc520034000 tx=0x7fc52002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:24.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:24.937+0000 7fc5277fe640 1 -- 172.21.15.88:0/4097275384 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc5180588b0 con 0x7fc5300cf6c0 2024-09-16T01:44:25.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.137+0000 7fc534b5f640 1 -- 172.21.15.88:0/4097275384 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc53013c5c0 con 0x7fc5300cf6c0 2024-09-16T01:44:25.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.138+0000 7fc5277fe640 1 -- 172.21.15.88:0/4097275384 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 305 v305) ==== 95+0+4392 (secure 0 0 0) 0x7fc51805f0d0 con 0x7fc5300cf6c0 2024-09-16T01:44:25.139 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:25.139 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":305,"btime":"2024-09-16T01:44:22:055031+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9696,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4270197461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4270197461},{"type":"v1","addr":"172.21.15.88:6837","nonce":4270197461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":305,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:44:22.055026+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:44:25.140 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 305 2024-09-16T01:44:25.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.140+0000 7fc534b5f640 1 -- 172.21.15.88:0/4097275384 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc51006c2b0 msgr2=0x7fc51006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:25.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.140+0000 7fc534b5f640 1 --2- 172.21.15.88:0/4097275384 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc51006c2b0 0x7fc51006e770 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fc520034000 tx=0x7fc52002e000 comp rx=0 tx=0).stop 2024-09-16T01:44:25.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.140+0000 7fc534b5f640 1 -- 172.21.15.88:0/4097275384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5300cf6c0 msgr2=0x7fc53015f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:25.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.140+0000 7fc534b5f640 1 --2- 172.21.15.88:0/4097275384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5300cf6c0 0x7fc53015f2d0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7fc51800c400 tx=0x7fc51800c8c0 comp rx=0 tx=0).stop 2024-09-16T01:44:25.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.140+0000 7fc534b5f640 1 -- 172.21.15.88:0/4097275384 shutdown_connections 2024-09-16T01:44:25.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.140+0000 7fc534b5f640 1 --2- 172.21.15.88:0/4097275384 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc51006c2b0 0x7fc51006e770 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:25.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.140+0000 7fc534b5f640 1 --2- 172.21.15.88:0/4097275384 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc53015c170 0x7fc5301e5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:25.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.140+0000 7fc534b5f640 1 --2- 172.21.15.88:0/4097275384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5300cf6c0 0x7fc53015f2d0 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:25.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.140+0000 7fc534b5f640 1 --2- 172.21.15.88:0/4097275384 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc5300b64e0 0x7fc53015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:25.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.140+0000 7fc534b5f640 1 -- 172.21.15.88:0/4097275384 >> 172.21.15.88:0/4097275384 conn(0x7fc53006bc10 msgr2=0x7fc53006ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:25.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.140+0000 7fc534b5f640 1 -- 172.21.15.88:0/4097275384 shutdown_connections 2024-09-16T01:44:25.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.140+0000 7fc534b5f640 1 -- 172.21.15.88:0/4097275384 wait complete. 2024-09-16T01:44:25.152 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:44:25.153 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:44:25.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.250+0000 7f4b64877640 1 Processor -- start 2024-09-16T01:44:25.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.250+0000 7f4b64877640 1 -- start start 2024-09-16T01:44:25.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.251+0000 7f4b64877640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b6015c2b0 0x7f4b600b6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:25.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.251+0000 7f4b64877640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4b6015f200 0x7f4b6006d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:25.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.251+0000 7f4b64877640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b6015fb10 con 0x7f4b600b72f0 2024-09-16T01:44:25.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.251+0000 7f4b64877640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f4b600b5900 con 0x7f4b6015f200 2024-09-16T01:44:25.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.251+0000 7f4b64877640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4b6015f830 con 0x7f4b6015c2b0 2024-09-16T01:44:25.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.251+0000 7f4b5ed9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4b6015f200 0x7f4b6006d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:25.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.251+0000 7f4b5ed9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4b6015f200 0x7f4b6006d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41558/0 (socket says 172.21.15.88:41558) 2024-09-16T01:44:25.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.251+0000 7f4b5ed9d640 1 -- 172.21.15.88:0/3397555800 learned_addr learned my addr 172.21.15.88:0/3397555800 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:25.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.251+0000 7f4b5dd9b640 1 --1- 172.21.15.88:0/3397555800 >> v1:172.21.15.88:6789/0 conn(0x7f4b600b72f0 0x7f4b6015eaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44176/0 (socket says 172.21.15.88:44176) 2024-09-16T01:44:25.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.251+0000 7f4b5ed9d640 1 -- 172.21.15.88:0/3397555800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b6015c2b0 msgr2=0x7f4b600b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:25.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.251+0000 7f4b5ed9d640 1 --2- 172.21.15.88:0/3397555800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b6015c2b0 0x7f4b600b6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:25.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.251+0000 7f4b5ed9d640 1 -- 172.21.15.88:0/3397555800 >> v1:172.21.15.88:6789/0 conn(0x7f4b600b72f0 legacy=0x7f4b6015eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:25.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.251+0000 7f4b5ed9d640 1 -- 172.21.15.88:0/3397555800 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6006ce30 con 0x7f4b6015f200 2024-09-16T01:44:25.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.252+0000 7f4b5ed9d640 1 --2- 172.21.15.88:0/3397555800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b6015f200 0x7f4b6006d910 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f4b50009e50 tx=0x7f4b5002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aedbe4d691acaa0e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:25.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.252+0000 7f4b5d59a640 1 -- 172.21.15.88:0/3397555800 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b50036070 con 0x7f4b6015f200 2024-09-16T01:44:25.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.252+0000 7f4b5d59a640 1 -- 172.21.15.88:0/3397555800 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b50031600 con 0x7f4b6015f200 2024-09-16T01:44:25.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.252+0000 7f4b5d59a640 1 -- 172.21.15.88:0/3397555800 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b500319d0 con 0x7f4b6015f200 2024-09-16T01:44:25.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.253+0000 7f4b64877640 1 -- 172.21.15.88:0/3397555800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b6015f200 msgr2=0x7f4b6006d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:25.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.253+0000 7f4b64877640 1 --2- 172.21.15.88:0/3397555800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b6015f200 0x7f4b6006d910 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f4b50009e50 tx=0x7f4b5002a710 comp rx=0 tx=0).stop 2024-09-16T01:44:25.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.253+0000 7f4b64877640 1 -- 172.21.15.88:0/3397555800 shutdown_connections 2024-09-16T01:44:25.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.253+0000 7f4b64877640 1 --2- 172.21.15.88:0/3397555800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b6015f200 0x7f4b6006d910 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:25.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.253+0000 7f4b64877640 1 --2- 172.21.15.88:0/3397555800 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b6015c2b0 0x7f4b600b6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:25.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.253+0000 7f4b64877640 1 -- 172.21.15.88:0/3397555800 >> 172.21.15.88:0/3397555800 conn(0x7f4b600b4240 msgr2=0x7f4b600b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:25.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.253+0000 7f4b64877640 1 -- 172.21.15.88:0/3397555800 shutdown_connections 2024-09-16T01:44:25.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.253+0000 7f4b64877640 1 -- 172.21.15.88:0/3397555800 wait complete. 2024-09-16T01:44:25.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.254+0000 7f4b64877640 1 Processor -- start 2024-09-16T01:44:25.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.254+0000 7f4b64877640 1 -- start start 2024-09-16T01:44:25.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.255+0000 7f4b64877640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b600b72f0 0x7f4b60159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:25.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.255+0000 7f4b64877640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b6015c2b0 0x7f4b60157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:25.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.255+0000 7f4b64877640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b6015f200 0x7f4b60155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:25.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.255+0000 7f4b64877640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4b600b7730 con 0x7f4b600b72f0 2024-09-16T01:44:25.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.255+0000 7f4b64877640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6006d3f0 con 0x7f4b6015c2b0 2024-09-16T01:44:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.255+0000 7f4b64877640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4b6006de50 con 0x7f4b6015f200 2024-09-16T01:44:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.255+0000 7f4b5e59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b600b72f0 0x7f4b60159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.255+0000 7f4b5e59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b600b72f0 0x7f4b60159b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41568/0 (socket says 172.21.15.88:41568) 2024-09-16T01:44:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.255+0000 7f4b5e59c640 1 -- 172.21.15.88:0/1415734867 learned_addr learned my addr 172.21.15.88:0/1415734867 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.255+0000 7f4b5dd9b640 1 --2- 172.21.15.88:0/1415734867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b6015c2b0 0x7f4b60157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.255+0000 7f4b5ed9d640 1 --2- 172.21.15.88:0/1415734867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b6015f200 0x7f4b60155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.255+0000 7f4b5e59c640 1 -- 172.21.15.88:0/1415734867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b6015f200 msgr2=0x7f4b60155db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.255+0000 7f4b5e59c640 1 --2- 172.21.15.88:0/1415734867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b6015f200 0x7f4b60155db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.255+0000 7f4b5e59c640 1 -- 172.21.15.88:0/1415734867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b6015c2b0 msgr2=0x7f4b60157bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.255+0000 7f4b5e59c640 1 --2- 172.21.15.88:0/1415734867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b6015c2b0 0x7f4b60157bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.255+0000 7f4b5e59c640 1 -- 172.21.15.88:0/1415734867 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b60153620 con 0x7f4b600b72f0 2024-09-16T01:44:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.255+0000 7f4b5ed9d640 1 --2- 172.21.15.88:0/1415734867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b6015f200 0x7f4b60155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:25.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.255+0000 7f4b5dd9b640 1 --2- 172.21.15.88:0/1415734867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b6015c2b0 0x7f4b60157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.255+0000 7f4b5e59c640 1 --2- 172.21.15.88:0/1415734867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b600b72f0 0x7f4b60159b40 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f4b5800b4a0 tx=0x7f4b5800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.256+0000 7f4b56ffd640 1 -- 172.21.15.88:0/1415734867 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b58014070 con 0x7f4b600b72f0 2024-09-16T01:44:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.256+0000 7f4b56ffd640 1 -- 172.21.15.88:0/1415734867 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b58010610 con 0x7f4b600b72f0 2024-09-16T01:44:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.256+0000 7f4b64877640 1 -- 172.21.15.88:0/1415734867 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b60153900 con 0x7f4b600b72f0 2024-09-16T01:44:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.256+0000 7f4b56ffd640 1 -- 172.21.15.88:0/1415734867 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b58010a00 con 0x7f4b600b72f0 2024-09-16T01:44:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.256+0000 7f4b64877640 1 -- 172.21.15.88:0/1415734867 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b60153ec0 con 0x7f4b600b72f0 2024-09-16T01:44:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.257+0000 7f4b64877640 1 -- 172.21.15.88:0/1415734867 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b600b6db0 con 0x7f4b600b72f0 2024-09-16T01:44:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.257+0000 7f4b56ffd640 1 -- 172.21.15.88:0/1415734867 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4b58017070 con 0x7f4b600b72f0 2024-09-16T01:44:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.258+0000 7f4b56ffd640 1 --2- 172.21.15.88:0/1415734867 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4b4006c2b0 0x7f4b4006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.258+0000 7f4b56ffd640 1 -- 172.21.15.88:0/1415734867 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9006+0+0 (secure 0 0 0) 0x7f4b5808bb80 con 0x7f4b600b72f0 2024-09-16T01:44:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.258+0000 7f4b5dd9b640 1 --2- 172.21.15.88:0/1415734867 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4b4006c2b0 0x7f4b4006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:25.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.258+0000 7f4b5dd9b640 1 --2- 172.21.15.88:0/1415734867 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4b4006c2b0 0x7f4b4006e770 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f4b60157aa0 tx=0x7f4b48009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:25.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.262+0000 7f4b56ffd640 1 -- 172.21.15.88:0/1415734867 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4b58059650 con 0x7f4b600b72f0 2024-09-16T01:44:25.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.460+0000 7f4b64877640 1 -- 172.21.15.88:0/1415734867 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b60153300 con 0x7f4b600b72f0 2024-09-16T01:44:25.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.460+0000 7f4b56ffd640 1 -- 172.21.15.88:0/1415734867 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 305 v305) ==== 95+0+4392 (secure 0 0 0) 0x7f4b58058f50 con 0x7f4b600b72f0 2024-09-16T01:44:25.462 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:25.462 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":305,"btime":"2024-09-16T01:44:22:055031+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9696,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4270197461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4270197461},{"type":"v1","addr":"172.21.15.88:6837","nonce":4270197461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":305,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:44:22.055026+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":125,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:44:25.462 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 305 2024-09-16T01:44:25.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.462+0000 7f4b64877640 1 -- 172.21.15.88:0/1415734867 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4b4006c2b0 msgr2=0x7f4b4006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:25.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.462+0000 7f4b64877640 1 --2- 172.21.15.88:0/1415734867 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4b4006c2b0 0x7f4b4006e770 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f4b60157aa0 tx=0x7f4b48009040 comp rx=0 tx=0).stop 2024-09-16T01:44:25.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.462+0000 7f4b64877640 1 -- 172.21.15.88:0/1415734867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b600b72f0 msgr2=0x7f4b60159b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:25.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.462+0000 7f4b64877640 1 --2- 172.21.15.88:0/1415734867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b600b72f0 0x7f4b60159b40 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f4b5800b4a0 tx=0x7f4b5800b960 comp rx=0 tx=0).stop 2024-09-16T01:44:25.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.462+0000 7f4b64877640 1 -- 172.21.15.88:0/1415734867 shutdown_connections 2024-09-16T01:44:25.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.462+0000 7f4b64877640 1 --2- 172.21.15.88:0/1415734867 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4b4006c2b0 0x7f4b4006e770 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:25.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.462+0000 7f4b64877640 1 --2- 172.21.15.88:0/1415734867 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b6015f200 0x7f4b60155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:25.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.462+0000 7f4b64877640 1 --2- 172.21.15.88:0/1415734867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b6015c2b0 0x7f4b60157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:25.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.462+0000 7f4b64877640 1 --2- 172.21.15.88:0/1415734867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b600b72f0 0x7f4b60159b40 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:25.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.462+0000 7f4b64877640 1 -- 172.21.15.88:0/1415734867 >> 172.21.15.88:0/1415734867 conn(0x7f4b600b4240 msgr2=0x7f4b6006e280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:25.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.463+0000 7f4b64877640 1 -- 172.21.15.88:0/1415734867 shutdown_connections 2024-09-16T01:44:25.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.463+0000 7f4b64877640 1 -- 172.21.15.88:0/1415734867 wait complete. 2024-09-16T01:44:25.475 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt omap header on 100.00000000' led to DAMAGED state 2024-09-16T01:44:25.475 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap header on mds0_sessionmap' 2024-09-16T01:44:25.475 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:44:25.476 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:44:25.492 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:44:25.492 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:44:25.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:25.493 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:44:25.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:44:25.550 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:44:25.550 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:44:25.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.691+0000 7f859d87b640 1 Processor -- start 2024-09-16T01:44:25.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.691+0000 7f859d87b640 1 -- start start 2024-09-16T01:44:25.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.692+0000 7f859d87b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8598161d70 0x7f85980b3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:25.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.692+0000 7f859d87b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8598162150 0x7f85980b6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:25.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.692+0000 7f859d87b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85980cf2a0 con 0x7f85980d4260 2024-09-16T01:44:25.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.692+0000 7f859d87b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f85980cf760 con 0x7f8598162150 2024-09-16T01:44:25.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.692+0000 7f859d87b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f85980cf9b0 con 0x7f8598161d70 2024-09-16T01:44:25.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.692+0000 7f85967fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f8598162150 0x7f85980b6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:25.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.692+0000 7f85977fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f85980d4260 0x7f8598162610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44188/0 (socket says 172.21.15.88:44188) 2024-09-16T01:44:25.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.692+0000 7f85977fe640 1 -- 172.21.15.88:0/3784092095 learned_addr learned my addr 172.21.15.88:0/3784092095 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:25.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.693+0000 7f8596ffd640 1 --2- 172.21.15.88:0/3784092095 >> v2:172.21.15.154:3300/0 conn(0x7f8598161d70 0x7f85980b3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:25.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.693+0000 7f85967fc640 1 -- 172.21.15.88:0/3784092095 >> v2:172.21.15.154:3300/0 conn(0x7f8598161d70 msgr2=0x7f85980b3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:25.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.693+0000 7f85967fc640 1 --2- 172.21.15.88:0/3784092095 >> v2:172.21.15.154:3300/0 conn(0x7f8598161d70 0x7f85980b3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:25.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.693+0000 7f85967fc640 1 -- 172.21.15.88:0/3784092095 >> v1:172.21.15.88:6789/0 conn(0x7f85980d4260 legacy=0x7f8598162610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:25.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.693+0000 7f85967fc640 1 -- 172.21.15.88:0/3784092095 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85980cfc90 con 0x7f8598162150 2024-09-16T01:44:25.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.693+0000 7f8595ffb640 1 -- 172.21.15.88:0/3784092095 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 165735816 0 0) 0x7f85980cf2a0 con 0x7f85980d4260 2024-09-16T01:44:25.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.693+0000 7f85967fc640 1 --2- 172.21.15.88:0/3784092095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8598162150 0x7f85980b6180 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f859808dbd0 tx=0x7f858c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9774b72d817ba28d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:25.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.693+0000 7f8595ffb640 1 -- 172.21.15.88:0/3784092095 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f858c037070 con 0x7f8598162150 2024-09-16T01:44:25.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.694+0000 7f8595ffb640 1 -- 172.21.15.88:0/3784092095 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f858c032440 con 0x7f8598162150 2024-09-16T01:44:25.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.694+0000 7f8595ffb640 1 -- 172.21.15.88:0/3784092095 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f858c032810 con 0x7f8598162150 2024-09-16T01:44:25.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.694+0000 7f859d87b640 1 -- 172.21.15.88:0/3784092095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8598162150 msgr2=0x7f85980b6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:25.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.694+0000 7f859d87b640 1 --2- 172.21.15.88:0/3784092095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8598162150 0x7f85980b6180 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f859808dbd0 tx=0x7f858c02a1a0 comp rx=0 tx=0).stop 2024-09-16T01:44:25.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.694+0000 7f859d87b640 1 -- 172.21.15.88:0/3784092095 shutdown_connections 2024-09-16T01:44:25.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.694+0000 7f859d87b640 1 --2- 172.21.15.88:0/3784092095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8598162150 0x7f85980b6180 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:25.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.694+0000 7f859d87b640 1 --2- 172.21.15.88:0/3784092095 >> v2:172.21.15.154:3300/0 conn(0x7f8598161d70 0x7f85980b3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:25.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.694+0000 7f859d87b640 1 -- 172.21.15.88:0/3784092095 >> 172.21.15.88:0/3784092095 conn(0x7f859806c500 msgr2=0x7f859806c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:25.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.694+0000 7f859d87b640 1 -- 172.21.15.88:0/3784092095 shutdown_connections 2024-09-16T01:44:25.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.694+0000 7f859d87b640 1 -- 172.21.15.88:0/3784092095 wait complete. 2024-09-16T01:44:25.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.695+0000 7f859d87b640 1 Processor -- start 2024-09-16T01:44:25.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.695+0000 7f859d87b640 1 -- start start 2024-09-16T01:44:25.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.696+0000 7f859d87b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85980d4260 0x7f8598156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:25.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.696+0000 7f859d87b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8598161d70 0x7f85980b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:25.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.696+0000 7f859d87b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8598162150 0x7f85980b0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:25.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.696+0000 7f859d87b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f85980ceed0 con 0x7f8598162150 2024-09-16T01:44:25.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.696+0000 7f859d87b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f85980ced50 con 0x7f85980d4260 2024-09-16T01:44:25.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.696+0000 7f859d87b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f85980cf050 con 0x7f8598161d70 2024-09-16T01:44:25.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.696+0000 7f85977fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8598162150 0x7f85980b0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:25.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.696+0000 7f85977fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8598162150 0x7f85980b0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41580/0 (socket says 172.21.15.88:41580) 2024-09-16T01:44:25.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.696+0000 7f85977fe640 1 -- 172.21.15.88:0/1317074713 learned_addr learned my addr 172.21.15.88:0/1317074713 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:25.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.697+0000 7f85977fe640 1 -- 172.21.15.88:0/1317074713 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8598161d70 msgr2=0x7f85980b2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:25.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.697+0000 7f85977fe640 1 --2- 172.21.15.88:0/1317074713 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8598161d70 0x7f85980b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:25.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.697+0000 7f8596ffd640 1 --2- 172.21.15.88:0/1317074713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85980d4260 0x7f8598156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:25.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.697+0000 7f85977fe640 1 -- 172.21.15.88:0/1317074713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85980d4260 msgr2=0x7f8598156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:25.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.697+0000 7f85977fe640 1 --2- 172.21.15.88:0/1317074713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85980d4260 0x7f8598156a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:25.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.697+0000 7f85977fe640 1 -- 172.21.15.88:0/1317074713 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8598156fd0 con 0x7f8598162150 2024-09-16T01:44:25.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.697+0000 7f85977fe640 1 --2- 172.21.15.88:0/1317074713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8598162150 0x7f85980b0690 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f859000c480 tx=0x7f859000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:25.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.697+0000 7f857ffff640 1 -- 172.21.15.88:0/1317074713 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8590018070 con 0x7f8598162150 2024-09-16T01:44:25.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.697+0000 7f857ffff640 1 -- 172.21.15.88:0/1317074713 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85900074e0 con 0x7f8598162150 2024-09-16T01:44:25.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.697+0000 7f859d87b640 1 -- 172.21.15.88:0/1317074713 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85980b10c0 con 0x7f8598162150 2024-09-16T01:44:25.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.697+0000 7f857ffff640 1 -- 172.21.15.88:0/1317074713 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8590004a20 con 0x7f8598162150 2024-09-16T01:44:25.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.697+0000 7f859d87b640 1 -- 172.21.15.88:0/1317074713 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85980b1680 con 0x7f8598162150 2024-09-16T01:44:25.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.698+0000 7f859d87b640 1 -- 172.21.15.88:0/1317074713 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85980b3390 con 0x7f8598162150 2024-09-16T01:44:25.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.698+0000 7f857ffff640 1 -- 172.21.15.88:0/1317074713 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f859001b070 con 0x7f8598162150 2024-09-16T01:44:25.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.699+0000 7f857ffff640 1 --2- 172.21.15.88:0/1317074713 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f858406c2b0 0x7f858406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:25.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.699+0000 7f857ffff640 1 -- 172.21.15.88:0/1317074713 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9006+0+0 (secure 0 0 0) 0x7f859008b3d0 con 0x7f8598162150 2024-09-16T01:44:25.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.699+0000 7f8596ffd640 1 --2- 172.21.15.88:0/1317074713 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f858406c2b0 0x7f858406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:25.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.699+0000 7f8596ffd640 1 --2- 172.21.15.88:0/1317074713 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f858406c2b0 0x7f858406e770 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f8588009f30 tx=0x7f858800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:25.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.703+0000 7f857ffff640 1 -- 172.21.15.88:0/1317074713 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8590058ea0 con 0x7f8598162150 2024-09-16T01:44:25.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:25.901+0000 7f859d87b640 1 -- 172.21.15.88:0/1317074713 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f85980b6fa0 con 0x7f8598162150 2024-09-16T01:44:26.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.065+0000 7f857ffff640 1 -- 172.21.15.88:0/1317074713 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v306) ==== 161+0+0 (secure 0 0 0) 0x7f85900587a0 con 0x7f8598162150 2024-09-16T01:44:26.067 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:44:26.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.068+0000 7f859d87b640 1 -- 172.21.15.88:0/1317074713 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f858406c2b0 msgr2=0x7f858406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:26.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.068+0000 7f859d87b640 1 --2- 172.21.15.88:0/1317074713 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f858406c2b0 0x7f858406e770 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f8588009f30 tx=0x7f858800b040 comp rx=0 tx=0).stop 2024-09-16T01:44:26.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.068+0000 7f859d87b640 1 -- 172.21.15.88:0/1317074713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8598162150 msgr2=0x7f85980b0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:26.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.068+0000 7f859d87b640 1 --2- 172.21.15.88:0/1317074713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8598162150 0x7f85980b0690 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f859000c480 tx=0x7f859000c940 comp rx=0 tx=0).stop 2024-09-16T01:44:26.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.068+0000 7f859d87b640 1 -- 172.21.15.88:0/1317074713 shutdown_connections 2024-09-16T01:44:26.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.068+0000 7f859d87b640 1 --2- 172.21.15.88:0/1317074713 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f858406c2b0 0x7f858406e770 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:26.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.068+0000 7f859d87b640 1 --2- 172.21.15.88:0/1317074713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8598162150 0x7f85980b0690 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:26.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.068+0000 7f859d87b640 1 --2- 172.21.15.88:0/1317074713 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8598161d70 0x7f85980b2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:26.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.068+0000 7f859d87b640 1 --2- 172.21.15.88:0/1317074713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85980d4260 0x7f8598156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:26.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.068+0000 7f859d87b640 1 -- 172.21.15.88:0/1317074713 >> 172.21.15.88:0/1317074713 conn(0x7f859806c500 msgr2=0x7f85980b4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:26.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.068+0000 7f859d87b640 1 -- 172.21.15.88:0/1317074713 shutdown_connections 2024-09-16T01:44:26.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.068+0000 7f859d87b640 1 -- 172.21.15.88:0/1317074713 wait complete. 2024-09-16T01:44:26.081 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:44:26.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.187+0000 7f7f2eec9640 1 Processor -- start 2024-09-16T01:44:26.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.187+0000 7f7f2eec9640 1 -- start start 2024-09-16T01:44:26.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.188+0000 7f7f2eec9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7f28158a00 0x7f7f28071f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:26.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.188+0000 7f7f2eec9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f28072470 0x7f7f280b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:26.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.188+0000 7f7f2eec9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7f28158de0 0x7f7f280b53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:26.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.188+0000 7f7f2eec9640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7f280728f0 con 0x7f7f28158de0 2024-09-16T01:44:26.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.188+0000 7f7f2eec9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7f28072bd0 con 0x7f7f28158a00 2024-09-16T01:44:26.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.188+0000 7f7f2eec9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7f28072eb0 con 0x7f7f28072470 2024-09-16T01:44:26.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.188+0000 7f7f2d466640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7f28158de0 0x7f7f280b53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:26.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.188+0000 7f7f2d466640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7f28158de0 0x7f7f280b53a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41594/0 (socket says 172.21.15.88:41594) 2024-09-16T01:44:26.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.188+0000 7f7f2d466640 1 -- 172.21.15.88:0/3998934088 learned_addr learned my addr 172.21.15.88:0/3998934088 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:26.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.188+0000 7f7f2d466640 1 -- 172.21.15.88:0/3998934088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f28072470 msgr2=0x7f7f280b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:26.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.188+0000 7f7f2d466640 1 --2- 172.21.15.88:0/3998934088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f28072470 0x7f7f280b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:26.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.188+0000 7f7f2d466640 1 -- 172.21.15.88:0/3998934088 >> v2:172.21.15.154:3300/0 conn(0x7f7f28158a00 msgr2=0x7f7f28071f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:26.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.188+0000 7f7f2d466640 1 --2- 172.21.15.88:0/3998934088 >> v2:172.21.15.154:3300/0 conn(0x7f7f28158a00 0x7f7f28071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:26.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.188+0000 7f7f2d466640 1 -- 172.21.15.88:0/3998934088 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f28073190 con 0x7f7f28158de0 2024-09-16T01:44:26.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.189+0000 7f7f2d466640 1 --2- 172.21.15.88:0/3998934088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f28158de0 0x7f7f280b53a0 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f7f28069f90 tx=0x7f7f2002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e87d6a5161c8660 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:26.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.189+0000 7f7f1f7fe640 1 -- 172.21.15.88:0/3998934088 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f20037070 con 0x7f7f28158de0 2024-09-16T01:44:26.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.189+0000 7f7f1f7fe640 1 -- 172.21.15.88:0/3998934088 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f20032440 con 0x7f7f28158de0 2024-09-16T01:44:26.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.189+0000 7f7f1f7fe640 1 -- 172.21.15.88:0/3998934088 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f20032810 con 0x7f7f28158de0 2024-09-16T01:44:26.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.190+0000 7f7f2eec9640 1 -- 172.21.15.88:0/3998934088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f28158de0 msgr2=0x7f7f280b53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:26.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.190+0000 7f7f2eec9640 1 --2- 172.21.15.88:0/3998934088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f28158de0 0x7f7f280b53a0 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f7f28069f90 tx=0x7f7f2002a1e0 comp rx=0 tx=0).stop 2024-09-16T01:44:26.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.190+0000 7f7f2eec9640 1 -- 172.21.15.88:0/3998934088 shutdown_connections 2024-09-16T01:44:26.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.190+0000 7f7f2eec9640 1 --2- 172.21.15.88:0/3998934088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f28158de0 0x7f7f280b53a0 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:26.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.190+0000 7f7f2eec9640 1 --2- 172.21.15.88:0/3998934088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f28072470 0x7f7f280b4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:26.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.190+0000 7f7f2eec9640 1 --2- 172.21.15.88:0/3998934088 >> v2:172.21.15.154:3300/0 conn(0x7f7f28158a00 0x7f7f28071f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:26.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.190+0000 7f7f2eec9640 1 -- 172.21.15.88:0/3998934088 >> 172.21.15.88:0/3998934088 conn(0x7f7f2806cb50 msgr2=0x7f7f2806cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:26.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.190+0000 7f7f2eec9640 1 -- 172.21.15.88:0/3998934088 shutdown_connections 2024-09-16T01:44:26.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.190+0000 7f7f2eec9640 1 -- 172.21.15.88:0/3998934088 wait complete. 2024-09-16T01:44:26.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.191+0000 7f7f2eec9640 1 Processor -- start 2024-09-16T01:44:26.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.191+0000 7f7f2eec9640 1 -- start start 2024-09-16T01:44:26.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.192+0000 7f7f2eec9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f28072470 0x7f7f2814b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:26.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.192+0000 7f7f2eec9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f28158a00 0x7f7f281e9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:26.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.192+0000 7f7f2eec9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f28158de0 0x7f7f28153e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:26.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.192+0000 7f7f2eec9640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7f280b5ca0 con 0x7f7f28158a00 2024-09-16T01:44:26.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.192+0000 7f7f2eec9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7f280b5b20 con 0x7f7f28072470 2024-09-16T01:44:26.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.192+0000 7f7f2eec9640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7f28162110 con 0x7f7f28158de0 2024-09-16T01:44:26.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.192+0000 7f7f1ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f28158a00 0x7f7f281e9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:26.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.192+0000 7f7f1ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f28158a00 0x7f7f281e9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41602/0 (socket says 172.21.15.88:41602) 2024-09-16T01:44:26.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.192+0000 7f7f1ffff640 1 -- 172.21.15.88:0/3721858436 learned_addr learned my addr 172.21.15.88:0/3721858436 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:26.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.192+0000 7f7f2cc65640 1 --2- 172.21.15.88:0/3721858436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f28072470 0x7f7f2814b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:26.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.192+0000 7f7f2d466640 1 --2- 172.21.15.88:0/3721858436 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f28158de0 0x7f7f28153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:26.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.192+0000 7f7f1ffff640 1 -- 172.21.15.88:0/3721858436 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f28158de0 msgr2=0x7f7f28153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:26.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.192+0000 7f7f1ffff640 1 --2- 172.21.15.88:0/3721858436 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f28158de0 0x7f7f28153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:26.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.192+0000 7f7f1ffff640 1 -- 172.21.15.88:0/3721858436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f28072470 msgr2=0x7f7f2814b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:26.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.192+0000 7f7f1ffff640 1 --2- 172.21.15.88:0/3721858436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f28072470 0x7f7f2814b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:26.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.192+0000 7f7f1ffff640 1 -- 172.21.15.88:0/3721858436 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f281f82a0 con 0x7f7f28158a00 2024-09-16T01:44:26.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.192+0000 7f7f1ffff640 1 --2- 172.21.15.88:0/3721858436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f28158a00 0x7f7f281e9d10 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f7f1800b4a0 tx=0x7f7f1800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:26.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.192+0000 7f7f1d7fa640 1 -- 172.21.15.88:0/3721858436 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f18014070 con 0x7f7f28158a00 2024-09-16T01:44:26.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.192+0000 7f7f1d7fa640 1 -- 172.21.15.88:0/3721858436 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f18010610 con 0x7f7f28158a00 2024-09-16T01:44:26.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.192+0000 7f7f2eec9640 1 -- 172.21.15.88:0/3721858436 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f281f8540 con 0x7f7f28158a00 2024-09-16T01:44:26.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.192+0000 7f7f1d7fa640 1 -- 172.21.15.88:0/3721858436 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f18010a00 con 0x7f7f28158a00 2024-09-16T01:44:26.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.193+0000 7f7f2eec9640 1 -- 172.21.15.88:0/3721858436 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f281f8b00 con 0x7f7f28158a00 2024-09-16T01:44:26.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.193+0000 7f7f2eec9640 1 -- 172.21.15.88:0/3721858436 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f28071f30 con 0x7f7f28158a00 2024-09-16T01:44:26.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.194+0000 7f7f1d7fa640 1 -- 172.21.15.88:0/3721858436 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7f18017070 con 0x7f7f28158a00 2024-09-16T01:44:26.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.194+0000 7f7f1d7fa640 1 --2- 172.21.15.88:0/3721858436 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7f1406c2b0 0x7f7f1406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:26.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.194+0000 7f7f1d7fa640 1 -- 172.21.15.88:0/3721858436 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9006+0+0 (secure 0 0 0) 0x7f7f1808bb80 con 0x7f7f28158a00 2024-09-16T01:44:26.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.194+0000 7f7f2cc65640 1 --2- 172.21.15.88:0/3721858436 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7f1406c2b0 0x7f7f1406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:26.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.195+0000 7f7f2cc65640 1 --2- 172.21.15.88:0/3721858436 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7f1406c2b0 0x7f7f1406e770 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f7f24009f20 tx=0x7f7f240094f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:26.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.199+0000 7f7f1d7fa640 1 -- 172.21.15.88:0/3721858436 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7f180595c0 con 0x7f7f28158a00 2024-09-16T01:44:26.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:26.431+0000 7f7f2eec9640 1 -- 172.21.15.88:0/3721858436 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7f7f28073de0 con 0x7f7f28158a00 2024-09-16T01:44:26.805 INFO:tasks.ceph.mds.a.smithi088.stderr: -53> 2024-09-16T01:44:22.052+0000 7f634af03640 -1 mds.0.cache.dir(0x100) Corrupt fnode in dirfrag 0x100: Decoder at 'void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&)' v=3 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-09-16T01:44:26.806 INFO:tasks.ceph.mds.a.smithi088.stderr: -33> 2024-09-16T01:44:22.648+0000 7f634ef0b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:44:26.829 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:44:27.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.076+0000 7f7f1d7fa640 1 -- 172.21.15.88:0/3721858436 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v307) ==== 101+0+0 (secure 0 0 0) 0x7f7f18058ec0 con 0x7f7f28158a00 2024-09-16T01:44:27.078 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:44:27.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.080+0000 7f7f2eec9640 1 -- 172.21.15.88:0/3721858436 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7f1406c2b0 msgr2=0x7f7f1406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:27.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.080+0000 7f7f2eec9640 1 --2- 172.21.15.88:0/3721858436 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7f1406c2b0 0x7f7f1406e770 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f7f24009f20 tx=0x7f7f240094f0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.080+0000 7f7f2eec9640 1 -- 172.21.15.88:0/3721858436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f28158a00 msgr2=0x7f7f281e9d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:27.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.080+0000 7f7f2eec9640 1 --2- 172.21.15.88:0/3721858436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f28158a00 0x7f7f281e9d10 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f7f1800b4a0 tx=0x7f7f1800b960 comp rx=0 tx=0).stop 2024-09-16T01:44:27.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.080+0000 7f7f2eec9640 1 -- 172.21.15.88:0/3721858436 shutdown_connections 2024-09-16T01:44:27.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.080+0000 7f7f2eec9640 1 --2- 172.21.15.88:0/3721858436 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7f1406c2b0 0x7f7f1406e770 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.080+0000 7f7f2eec9640 1 --2- 172.21.15.88:0/3721858436 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7f28158de0 0x7f7f28153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.080+0000 7f7f2eec9640 1 --2- 172.21.15.88:0/3721858436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f28158a00 0x7f7f281e9d10 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.080+0000 7f7f2eec9640 1 --2- 172.21.15.88:0/3721858436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f28072470 0x7f7f2814b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.080+0000 7f7f2eec9640 1 -- 172.21.15.88:0/3721858436 >> 172.21.15.88:0/3721858436 conn(0x7f7f2806cb50 msgr2=0x7f7f280b11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:27.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.080+0000 7f7f2eec9640 1 -- 172.21.15.88:0/3721858436 shutdown_connections 2024-09-16T01:44:27.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.080+0000 7f7f2eec9640 1 -- 172.21.15.88:0/3721858436 wait complete. 2024-09-16T01:44:27.093 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:44:27.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.134+0000 7f497c8cef40 1 Processor -- start 2024-09-16T01:44:27.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.134+0000 7f497c8cef40 1 -- start start 2024-09-16T01:44:27.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.135+0000 7f497c8cef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a58e5f1fe0 0x55a58e5f19a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.135+0000 7f497c8cef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55a58e696d20 0x55a58e697fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.135+0000 7f497c8cef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a58e697100 0x55a58e5ee900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.135+0000 7f497c8cef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55a58e514510 con 0x55a58e5f1fe0 2024-09-16T01:44:27.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.135+0000 7f497c8cef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a58e60abd0 con 0x55a58e697100 2024-09-16T01:44:27.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.135+0000 7f497c8cef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55a58e5c57a0 con 0x55a58e696d20 2024-09-16T01:44:27.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.135+0000 7f497a3c4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a58e5f1fe0 0x55a58e5f19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:27.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.135+0000 7f497a3c4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a58e5f1fe0 0x55a58e5f19a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41622/0 (socket says 172.21.15.88:41622) 2024-09-16T01:44:27.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.135+0000 7f497a3c4640 1 -- 172.21.15.88:0/1688375104 learned_addr learned my addr 172.21.15.88:0/1688375104 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:27.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.135+0000 7f497a3c4640 1 -- 172.21.15.88:0/1688375104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55a58e696d20 msgr2=0x55a58e697fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:27.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.135+0000 7f497a3c4640 1 --2- 172.21.15.88:0/1688375104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55a58e696d20 0x55a58e697fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.135+0000 7f497a3c4640 1 -- 172.21.15.88:0/1688375104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a58e697100 msgr2=0x55a58e5ee900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:27.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.135+0000 7f497a3c4640 1 --2- 172.21.15.88:0/1688375104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a58e697100 0x55a58e5ee900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.135+0000 7f497a3c4640 1 -- 172.21.15.88:0/1688375104 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a58e60dbd0 con 0x55a58e5f1fe0 2024-09-16T01:44:27.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.136+0000 7f497a3c4640 1 --2- 172.21.15.88:0/1688375104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a58e5f1fe0 0x55a58e5f19a0 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7f496c00d4f0 tx=0x7f496c00d5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d07c644ceb4bcb9a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:27.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.136+0000 7f49793c2640 1 -- 172.21.15.88:0/1688375104 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496c00a620 con 0x55a58e5f1fe0 2024-09-16T01:44:27.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.136+0000 7f49793c2640 1 -- 172.21.15.88:0/1688375104 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f496c00a7c0 con 0x55a58e5f1fe0 2024-09-16T01:44:27.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.136+0000 7f49793c2640 1 -- 172.21.15.88:0/1688375104 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496c00ab70 con 0x55a58e5f1fe0 2024-09-16T01:44:27.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.136+0000 7f497c8cef40 1 -- 172.21.15.88:0/1688375104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a58e5f1fe0 msgr2=0x55a58e5f19a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:27.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.136+0000 7f497c8cef40 1 --2- 172.21.15.88:0/1688375104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a58e5f1fe0 0x55a58e5f19a0 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7f496c00d4f0 tx=0x7f496c00d5e0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.136+0000 7f497c8cef40 1 -- 172.21.15.88:0/1688375104 shutdown_connections 2024-09-16T01:44:27.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.136+0000 7f497c8cef40 1 --2- 172.21.15.88:0/1688375104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a58e697100 0x55a58e5ee900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.136+0000 7f497c8cef40 1 --2- 172.21.15.88:0/1688375104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55a58e696d20 0x55a58e697fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.136+0000 7f497c8cef40 1 --2- 172.21.15.88:0/1688375104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a58e5f1fe0 0x55a58e5f19a0 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.136+0000 7f497c8cef40 1 -- 172.21.15.88:0/1688375104 >> 172.21.15.88:0/1688375104 conn(0x55a58e60d750 msgr2=0x55a58e5eda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:27.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.136+0000 7f497c8cef40 1 -- 172.21.15.88:0/1688375104 shutdown_connections 2024-09-16T01:44:27.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.136+0000 7f497c8cef40 1 -- 172.21.15.88:0/1688375104 wait complete. 2024-09-16T01:44:27.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.137+0000 7f497c8cef40 1 Processor -- start 2024-09-16T01:44:27.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.137+0000 7f497c8cef40 1 -- start start 2024-09-16T01:44:27.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.138+0000 7f497c8cef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a58e5f1fe0 0x55a58e731510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.138+0000 7f497c8cef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a58e696d20 0x55a58e68bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.138+0000 7f497c8cef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55a58e697100 0x55a58e5c4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.138+0000 7f497c8cef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55a58e60deb0 con 0x55a58e5f1fe0 2024-09-16T01:44:27.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.138+0000 7f497c8cef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55a58e5c5a80 con 0x55a58e696d20 2024-09-16T01:44:27.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.138+0000 7f497c8cef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55a58e690080 con 0x55a58e697100 2024-09-16T01:44:27.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.138+0000 7f497a3c4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a58e5f1fe0 0x55a58e731510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:27.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.138+0000 7f497a3c4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a58e5f1fe0 0x55a58e731510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41636/0 (socket says 172.21.15.88:41636) 2024-09-16T01:44:27.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.138+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 learned_addr learned my addr 172.21.15.88:0/4054319233 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:27.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.138+0000 7f4973fff640 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a58e696d20 0x55a58e68bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:27.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.138+0000 7f497abc5640 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55a58e697100 0x55a58e5c4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:27.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.138+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55a58e697100 msgr2=0x55a58e5c4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:27.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.138+0000 7f497a3c4640 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55a58e697100 0x55a58e5c4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.138+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a58e696d20 msgr2=0x55a58e68bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:27.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.138+0000 7f497a3c4640 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a58e696d20 0x55a58e68bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.138+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a58e734200 con 0x55a58e5f1fe0 2024-09-16T01:44:27.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.138+0000 7f4973fff640 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a58e696d20 0x55a58e68bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:27.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.138+0000 7f497abc5640 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55a58e697100 0x55a58e5c4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:27.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.138+0000 7f497a3c4640 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a58e5f1fe0 0x55a58e731510 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f496c00d620 tx=0x7f496c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:27.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.138+0000 7f49737fe640 1 -- 172.21.15.88:0/4054319233 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496c025070 con 0x55a58e5f1fe0 2024-09-16T01:44:27.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.138+0000 7f49737fe640 1 -- 172.21.15.88:0/4054319233 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f496c00bc10 con 0x55a58e5f1fe0 2024-09-16T01:44:27.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.138+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a58e7344c0 con 0x55a58e5f1fe0 2024-09-16T01:44:27.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.138+0000 7f49737fe640 1 -- 172.21.15.88:0/4054319233 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496c008030 con 0x55a58e5f1fe0 2024-09-16T01:44:27.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.138+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a58e734a80 con 0x55a58e5f1fe0 2024-09-16T01:44:27.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.140+0000 7f49737fe640 1 -- 172.21.15.88:0/4054319233 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f496c028070 con 0x55a58e5f1fe0 2024-09-16T01:44:27.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.140+0000 7f49737fe640 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f495c06ac90 0x7f495c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.140+0000 7f49737fe640 1 -- 172.21.15.88:0/4054319233 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9006+0+0 (secure 0 0 0) 0x7f496c08b560 con 0x55a58e5f1fe0 2024-09-16T01:44:27.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.140+0000 7f4973fff640 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f495c06ac90 0x7f495c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:27.145 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:44:27.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.140+0000 7f497c8cef40 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55a58e697e20 0x55a58e698200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.140+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e698740 con 0x55a58e697e20 2024-09-16T01:44:27.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.140+0000 7f4973fff640 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f495c06ac90 0x7f495c06d150 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x55a58e69a820 tx=0x7f4964005620 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:27.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.141+0000 7f497abc5640 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55a58e697e20 0x55a58e698200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:27.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.141+0000 7f497abc5640 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55a58e697e20 0x55a58e698200 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:27.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.143+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v125'1159 uv1133 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.143+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e125) -- 0x55a58e698b00 con 0x55a58e697e20 2024-09-16T01:44:27.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.144+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v125'1160 uv1160 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.144+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73d1f0 con 0x55a58e697e20 2024-09-16T01:44:27.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.146+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v125'1161 uv1161 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.147 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:44:27.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.146+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73e5a0 con 0x55a58e697e20 2024-09-16T01:44:27.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.147+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v125'1162 uv1162 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.147+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73e940 con 0x55a58e697e20 2024-09-16T01:44:27.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.148+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v125'1163 uv1163 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.148+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73d590 con 0x55a58e697e20 2024-09-16T01:44:27.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.150+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v125'1164 uv1164 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.150+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73d980 con 0x55a58e697e20 2024-09-16T01:44:27.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.151+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v125'1165 uv1139 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.151+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73dd20 con 0x55a58e697e20 2024-09-16T01:44:27.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.152+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v125'1166 uv1166 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.152+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73dd20 con 0x55a58e697e20 2024-09-16T01:44:27.154 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:44:27.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.153+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v125'1167 uv1167 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.153+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73dd20 con 0x55a58e697e20 2024-09-16T01:44:27.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.155+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v125'1168 uv1168 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.155+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73dd20 con 0x55a58e697e20 2024-09-16T01:44:27.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.156+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v125'1169 uv1169 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.156+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73dd20 con 0x55a58e697e20 2024-09-16T01:44:27.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.157+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v125'1170 uv1170 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.157+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73dd20 con 0x55a58e697e20 2024-09-16T01:44:27.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.159+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v125'1171 uv1146 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.159+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73dd20 con 0x55a58e697e20 2024-09-16T01:44:27.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.160+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v125'1172 uv1172 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.160+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73dd20 con 0x55a58e697e20 2024-09-16T01:44:27.162 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:44:27.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.161+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v125'1173 uv1173 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.161+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73dd20 con 0x55a58e697e20 2024-09-16T01:44:27.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.163+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v125'1174 uv1174 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.163+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73dd20 con 0x55a58e697e20 2024-09-16T01:44:27.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.164+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v125'1175 uv1175 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.164+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73dd20 con 0x55a58e697e20 2024-09-16T01:44:27.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.165+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v125'1176 uv1176 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.165+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73dd20 con 0x55a58e697e20 2024-09-16T01:44:27.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.166+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v125'1177 uv1177 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.166+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73dd20 con 0x55a58e697e20 2024-09-16T01:44:27.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.168+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v125'1178 uv1152 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.168+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73dd20 con 0x55a58e697e20 2024-09-16T01:44:27.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.169+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v125'1179 uv1179 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.169+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73dd20 con 0x55a58e697e20 2024-09-16T01:44:27.171 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:44:27.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.170+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v125'1180 uv1180 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.170+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73dd20 con 0x55a58e697e20 2024-09-16T01:44:27.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.171+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v125'1181 uv1181 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.171+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73dd20 con 0x55a58e697e20 2024-09-16T01:44:27.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.172+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v125'1182 uv1182 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.173+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73dd20 con 0x55a58e697e20 2024-09-16T01:44:27.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.174+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v125'1183 uv1183 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.174+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73dd20 con 0x55a58e697e20 2024-09-16T01:44:27.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.175+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v125'1184 uv1158 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.175+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73dd20 con 0x55a58e697e20 2024-09-16T01:44:27.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.176+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v125'1185 uv1185 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.176+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73dd20 con 0x55a58e697e20 2024-09-16T01:44:27.179 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:44:27.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.178+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v125'1186 uv1186 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.178+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73dd20 con 0x55a58e697e20 2024-09-16T01:44:27.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.179+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v125'1187 uv1187 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.179+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73dd20 con 0x55a58e697e20 2024-09-16T01:44:27.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.180+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v125'1188 uv1188 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.180+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e73dd20 con 0x55a58e697e20 2024-09-16T01:44:27.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.181+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v125'1189 uv1189 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.181+0000 7f497c8cef40 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55a58e73ece0 0x55a58e7410d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.181+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e741610 con 0x55a58e73ece0 2024-09-16T01:44:27.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.181+0000 7f497a3c4640 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55a58e73ece0 0x55a58e7410d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:27.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.182+0000 7f497a3c4640 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55a58e73ece0 0x55a58e7410d0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:27.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.184+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v125'847 uv829 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e73ece0 2024-09-16T01:44:27.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.184+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e125) -- 0x55a58e7419b0 con 0x55a58e73ece0 2024-09-16T01:44:27.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.185+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v125'848 uv848 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e73ece0 2024-09-16T01:44:27.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.185+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e741d50 con 0x55a58e73ece0 2024-09-16T01:44:27.188 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:44:27.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.187+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v125'849 uv849 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e73ece0 2024-09-16T01:44:27.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.187+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e7420f0 con 0x55a58e73ece0 2024-09-16T01:44:27.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.188+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v125'850 uv850 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e73ece0 2024-09-16T01:44:27.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.188+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e742490 con 0x55a58e73ece0 2024-09-16T01:44:27.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.190+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v125'851 uv851 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e73ece0 2024-09-16T01:44:27.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.190+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e742880 con 0x55a58e73ece0 2024-09-16T01:44:27.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.191+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v125'852 uv834 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e73ece0 2024-09-16T01:44:27.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.192+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e125) -- 0x55a58e742c70 con 0x55a58e73ece0 2024-09-16T01:44:27.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.193+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v125'853 uv853 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e73ece0 2024-09-16T01:44:27.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.193+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e742c70 con 0x55a58e73ece0 2024-09-16T01:44:27.195 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:44:27.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.194+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v125'854 uv854 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e73ece0 2024-09-16T01:44:27.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.194+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e742c20 con 0x55a58e73ece0 2024-09-16T01:44:27.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.195+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v125'855 uv855 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e73ece0 2024-09-16T01:44:27.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.195+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e742c20 con 0x55a58e73ece0 2024-09-16T01:44:27.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.196+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v125'856 uv856 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e73ece0 2024-09-16T01:44:27.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.196+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e742c20 con 0x55a58e73ece0 2024-09-16T01:44:27.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.198+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v125'857 uv839 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e73ece0 2024-09-16T01:44:27.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.198+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e125) -- 0x55a58e742c20 con 0x55a58e73ece0 2024-09-16T01:44:27.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.199+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v125'858 uv858 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e73ece0 2024-09-16T01:44:27.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.199+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e742c20 con 0x55a58e73ece0 2024-09-16T01:44:27.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.200+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v125'859 uv859 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e73ece0 2024-09-16T01:44:27.202 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:44:27.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.200+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e742c20 con 0x55a58e73ece0 2024-09-16T01:44:27.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.202+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v125'860 uv860 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e73ece0 2024-09-16T01:44:27.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.202+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e742c20 con 0x55a58e73ece0 2024-09-16T01:44:27.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.203+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v125'861 uv861 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e73ece0 2024-09-16T01:44:27.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.203+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e742c20 con 0x55a58e73ece0 2024-09-16T01:44:27.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.204+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v125'862 uv845 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e73ece0 2024-09-16T01:44:27.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.204+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e125) -- 0x55a58e742c20 con 0x55a58e73ece0 2024-09-16T01:44:27.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.206+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v125'863 uv863 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e73ece0 2024-09-16T01:44:27.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.206+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e742c20 con 0x55a58e73ece0 2024-09-16T01:44:27.208 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:44:27.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.207+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v125'864 uv864 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e73ece0 2024-09-16T01:44:27.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.207+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e742c20 con 0x55a58e73ece0 2024-09-16T01:44:27.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.208+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v125'865 uv865 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e73ece0 2024-09-16T01:44:27.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.208+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e743c30 con 0x55a58e73ece0 2024-09-16T01:44:27.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.209+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v125'866 uv866 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e73ece0 2024-09-16T01:44:27.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.209+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e742c20 con 0x55a58e73ece0 2024-09-16T01:44:27.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.211+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v125'867 uv867 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e73ece0 2024-09-16T01:44:27.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.211+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e742c20 con 0x55a58e697e20 2024-09-16T01:44:27.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.212+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v125'225 uv224 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.212+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e125) -- 0x55a58e742c20 con 0x55a58e697e20 2024-09-16T01:44:27.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.213+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v125'226 uv226 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.213+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e742c20 con 0x55a58e697e20 2024-09-16T01:44:27.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.214+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v125'227 uv227 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.215 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:44:27.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.214+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e742c20 con 0x55a58e697e20 2024-09-16T01:44:27.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.214+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v125'228 uv228 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.214+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e742c20 con 0x55a58e697e20 2024-09-16T01:44:27.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.215+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v125'229 uv229 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.215+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e742c20 con 0x55a58e697e20 2024-09-16T01:44:27.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.216+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v125'230 uv230 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.216+0000 7f497c8cef40 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55a58e743520 0x55a58e743940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.216+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e746190 con 0x55a58e743520 2024-09-16T01:44:27.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.216+0000 7f4973fff640 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55a58e743520 0x55a58e743940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:27.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.217+0000 7f4973fff640 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55a58e743520 0x55a58e743940 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:27.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.218+0000 7f4973fff640 1 -- 172.21.15.88:0/4054319233 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v125'172 uv171 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f4964006870 con 0x55a58e743520 2024-09-16T01:44:27.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.218+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e125) -- 0x55a58e746530 con 0x55a58e743520 2024-09-16T01:44:27.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.219+0000 7f4973fff640 1 -- 172.21.15.88:0/4054319233 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v125'173 uv173 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f4964006870 con 0x55a58e743520 2024-09-16T01:44:27.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.219+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e746950 con 0x55a58e743520 2024-09-16T01:44:27.221 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:44:27.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.220+0000 7f4973fff640 1 -- 172.21.15.88:0/4054319233 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v125'174 uv174 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f4964006870 con 0x55a58e743520 2024-09-16T01:44:27.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.220+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e5edc90 con 0x55a58e743520 2024-09-16T01:44:27.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.221+0000 7f4973fff640 1 -- 172.21.15.88:0/4054319233 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v125'175 uv175 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f4964006870 con 0x55a58e743520 2024-09-16T01:44:27.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.221+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e747270 con 0x55a58e697e20 2024-09-16T01:44:27.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.222+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v125'225 uv224 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.222+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e125) -- 0x55a58e747270 con 0x55a58e697e20 2024-09-16T01:44:27.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.223+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v125'226 uv226 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.223+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e747270 con 0x55a58e697e20 2024-09-16T01:44:27.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.224+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v125'227 uv227 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.225 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:44:27.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.224+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e748280 con 0x55a58e697e20 2024-09-16T01:44:27.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.225+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v125'228 uv228 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.225+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e748280 con 0x55a58e697e20 2024-09-16T01:44:27.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.226+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v125'229 uv229 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.226+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e747270 con 0x55a58e697e20 2024-09-16T01:44:27.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.227+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v125'230 uv230 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.227+0000 7f497c8cef40 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55a58e747b40 0x55a58e747f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.227+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74a650 con 0x55a58e747b40 2024-09-16T01:44:27.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.227+0000 7f497abc5640 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55a58e747b40 0x55a58e747f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:27.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.228+0000 7f497abc5640 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55a58e747b40 0x55a58e747f60 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:27.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.229+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v125'415 uv409 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.229+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74a650 con 0x55a58e747b40 2024-09-16T01:44:27.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.230+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v125'416 uv416 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.230+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74a650 con 0x55a58e747b40 2024-09-16T01:44:27.232 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:44:27.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.231+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v125'417 uv417 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.231+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74b6c0 con 0x55a58e747b40 2024-09-16T01:44:27.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.232+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v125'418 uv418 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.233+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74b680 con 0x55a58e747b40 2024-09-16T01:44:27.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.234+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v125'419 uv419 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.234+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74b680 con 0x55a58e747b40 2024-09-16T01:44:27.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.235+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v125'420 uv420 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.235+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74b680 con 0x55a58e747b40 2024-09-16T01:44:27.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.236+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v125'421 uv414 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.236+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74a6a0 con 0x55a58e747b40 2024-09-16T01:44:27.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.237+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v125'422 uv422 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.237+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74a6a0 con 0x55a58e747b40 2024-09-16T01:44:27.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.238+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v125'423 uv423 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.240 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:44:27.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.238+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74a650 con 0x55a58e747b40 2024-09-16T01:44:27.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.239+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v125'424 uv424 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.239+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74a650 con 0x55a58e747b40 2024-09-16T01:44:27.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.240+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v125'425 uv425 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.240+0000 7f497c8cef40 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55a58e74af70 0x55a58e74b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.240+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74dad0 con 0x55a58e74af70 2024-09-16T01:44:27.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.240+0000 7f497a3c4640 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55a58e74af70 0x55a58e74b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:27.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.241+0000 7f497a3c4640 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55a58e74af70 0x55a58e74b3b0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:27.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.242+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v125'188 uv187 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e74af70 2024-09-16T01:44:27.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.242+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74dad0 con 0x55a58e74af70 2024-09-16T01:44:27.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.243+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v125'189 uv189 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e74af70 2024-09-16T01:44:27.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.243+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74db10 con 0x55a58e74af70 2024-09-16T01:44:27.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.244+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v125'190 uv190 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e74af70 2024-09-16T01:44:27.245 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:44:27.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.244+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74df00 con 0x55a58e74af70 2024-09-16T01:44:27.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.245+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v125'191 uv191 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e74af70 2024-09-16T01:44:27.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.245+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74df00 con 0x55a58e74af70 2024-09-16T01:44:27.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.246+0000 7f497a3c4640 1 -- 172.21.15.88:0/4054319233 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v125'192 uv192 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f496c027030 con 0x55a58e74af70 2024-09-16T01:44:27.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.246+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74df00 con 0x55a58e697e20 2024-09-16T01:44:27.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.247+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v125'225 uv224 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.247+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74df00 con 0x55a58e697e20 2024-09-16T01:44:27.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.248+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v125'226 uv226 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.248+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74df00 con 0x55a58e697e20 2024-09-16T01:44:27.250 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:44:27.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.249+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v125'227 uv227 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.249+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74ef10 con 0x55a58e697e20 2024-09-16T01:44:27.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.250+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v125'228 uv228 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.250+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74ef10 con 0x55a58e697e20 2024-09-16T01:44:27.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.251+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v125'229 uv229 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.251+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74df00 con 0x55a58e697e20 2024-09-16T01:44:27.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.252+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v125'230 uv230 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e697e20 2024-09-16T01:44:27.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.252+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74df00 con 0x55a58e743520 2024-09-16T01:44:27.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.253+0000 7f4973fff640 1 -- 172.21.15.88:0/4054319233 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v125'376 uv369 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f4964006870 con 0x55a58e743520 2024-09-16T01:44:27.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.253+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e2a0 con 0x55a58e743520 2024-09-16T01:44:27.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.254+0000 7f4973fff640 1 -- 172.21.15.88:0/4054319233 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v125'377 uv377 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f4964006870 con 0x55a58e743520 2024-09-16T01:44:27.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.254+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e640 con 0x55a58e743520 2024-09-16T01:44:27.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.255+0000 7f4973fff640 1 -- 172.21.15.88:0/4054319233 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v125'378 uv378 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f4964006870 con 0x55a58e743520 2024-09-16T01:44:27.257 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:44:27.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.255+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e743520 2024-09-16T01:44:27.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.256+0000 7f4973fff640 1 -- 172.21.15.88:0/4054319233 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v125'379 uv379 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f4964006870 con 0x55a58e743520 2024-09-16T01:44:27.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.256+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e743520 2024-09-16T01:44:27.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.257+0000 7f4973fff640 1 -- 172.21.15.88:0/4054319233 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v125'380 uv380 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f4964006870 con 0x55a58e743520 2024-09-16T01:44:27.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.257+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e743520 2024-09-16T01:44:27.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.258+0000 7f4973fff640 1 -- 172.21.15.88:0/4054319233 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v125'381 uv375 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f4964006870 con 0x55a58e743520 2024-09-16T01:44:27.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.258+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e743520 2024-09-16T01:44:27.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.259+0000 7f4973fff640 1 -- 172.21.15.88:0/4054319233 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v125'382 uv382 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4964006870 con 0x55a58e743520 2024-09-16T01:44:27.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.259+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e743520 2024-09-16T01:44:27.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.260+0000 7f4973fff640 1 -- 172.21.15.88:0/4054319233 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v125'383 uv383 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4964006870 con 0x55a58e743520 2024-09-16T01:44:27.261 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:44:27.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.260+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e743520 2024-09-16T01:44:27.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.261+0000 7f4973fff640 1 -- 172.21.15.88:0/4054319233 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v125'384 uv384 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4964006870 con 0x55a58e743520 2024-09-16T01:44:27.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.261+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e7517f0 con 0x55a58e743520 2024-09-16T01:44:27.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.262+0000 7f4973fff640 1 -- 172.21.15.88:0/4054319233 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v125'385 uv385 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4964006870 con 0x55a58e743520 2024-09-16T01:44:27.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.262+0000 7f497c8cef40 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55a58e74f510 0x55a58e751e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.262+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e751900 con 0x55a58e74f510 2024-09-16T01:44:27.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.262+0000 7f4973fff640 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55a58e74f510 0x55a58e751e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:27.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.262+0000 7f4973fff640 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55a58e74f510 0x55a58e751e30 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:27.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.263+0000 7f4973fff640 1 -- 172.21.15.88:0/4054319233 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v125'167 uv165 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f4964006870 con 0x55a58e74f510 2024-09-16T01:44:27.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.263+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e74f510 2024-09-16T01:44:27.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.264+0000 7f4973fff640 1 -- 172.21.15.88:0/4054319233 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v125'168 uv168 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f4964006870 con 0x55a58e74f510 2024-09-16T01:44:27.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.264+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e74f510 2024-09-16T01:44:27.266 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:44:27.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.265+0000 7f4973fff640 1 -- 172.21.15.88:0/4054319233 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v125'169 uv169 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f4964006870 con 0x55a58e74f510 2024-09-16T01:44:27.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.265+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e74f510 2024-09-16T01:44:27.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.266+0000 7f4973fff640 1 -- 172.21.15.88:0/4054319233 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v125'170 uv170 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f4964006870 con 0x55a58e74f510 2024-09-16T01:44:27.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.266+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e743520 2024-09-16T01:44:27.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.267+0000 7f4973fff640 1 -- 172.21.15.88:0/4054319233 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v125'225 uv224 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f4964006870 con 0x55a58e743520 2024-09-16T01:44:27.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.267+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e743520 2024-09-16T01:44:27.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.268+0000 7f4973fff640 1 -- 172.21.15.88:0/4054319233 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v125'226 uv226 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4964006870 con 0x55a58e743520 2024-09-16T01:44:27.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.268+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e743520 2024-09-16T01:44:27.270 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:44:27.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.269+0000 7f4973fff640 1 -- 172.21.15.88:0/4054319233 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v125'227 uv227 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4964006870 con 0x55a58e743520 2024-09-16T01:44:27.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.269+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e743520 2024-09-16T01:44:27.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.271+0000 7f4973fff640 1 -- 172.21.15.88:0/4054319233 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v125'228 uv228 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4964006870 con 0x55a58e743520 2024-09-16T01:44:27.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.271+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e743520 2024-09-16T01:44:27.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.272+0000 7f4973fff640 1 -- 172.21.15.88:0/4054319233 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v125'229 uv229 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4964006870 con 0x55a58e743520 2024-09-16T01:44:27.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.272+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e743520 2024-09-16T01:44:27.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.273+0000 7f4973fff640 1 -- 172.21.15.88:0/4054319233 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v125'230 uv230 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f4964006870 con 0x55a58e743520 2024-09-16T01:44:27.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.273+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e747b40 2024-09-16T01:44:27.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.274+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v125'191 uv190 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.274+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e747b40 2024-09-16T01:44:27.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.275+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v125'192 uv192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.275+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e747b40 2024-09-16T01:44:27.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.276+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v125'193 uv193 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.278 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:44:27.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.276+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e747b40 2024-09-16T01:44:27.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.277+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v125'194 uv194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.278+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e747b40 2024-09-16T01:44:27.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.279+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v125'195 uv195 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.279+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e747b40 2024-09-16T01:44:27.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.280+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v125'468 uv459 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.280+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e747b40 2024-09-16T01:44:27.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.281+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v125'469 uv469 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.281+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e747b40 2024-09-16T01:44:27.283 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:44:27.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.282+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v125'470 uv470 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.282+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e747b40 2024-09-16T01:44:27.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.283+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v125'471 uv471 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.283+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e747b40 2024-09-16T01:44:27.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.284+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v125'472 uv472 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.284+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e747b40 2024-09-16T01:44:27.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.285+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v125'473 uv466 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.285+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e747b40 2024-09-16T01:44:27.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.286+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v125'474 uv474 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.286+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e747b40 2024-09-16T01:44:27.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.287+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v125'475 uv475 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.288 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:44:27.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.287+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e747b40 2024-09-16T01:44:27.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.288+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v125'476 uv476 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.288+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e747b40 2024-09-16T01:44:27.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.289+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v125'477 uv477 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.289+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e74e9e0 con 0x55a58e747b40 2024-09-16T01:44:27.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.290+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v125'478 uv478 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.290+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x55a58e752370 con 0x55a58e747b40 2024-09-16T01:44:27.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v125'479 uv479 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f497405e040 con 0x55a58e747b40 2024-09-16T01:44:27.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55a58e697e20 msgr2=0x55a58e698200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:27.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497c8cef40 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55a58e697e20 0x55a58e698200 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55a58e743520 msgr2=0x55a58e743940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:27.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497c8cef40 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55a58e743520 0x55a58e743940 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55a58e74f510 msgr2=0x55a58e751e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:27.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497c8cef40 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x55a58e74f510 0x55a58e751e30 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55a58e73ece0 msgr2=0x55a58e7410d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:27.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497c8cef40 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x55a58e73ece0 0x55a58e7410d0 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55a58e747b40 msgr2=0x55a58e747f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:27.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497abc5640 1 -- 172.21.15.88:0/4054319233 reap_dead start 2024-09-16T01:44:27.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497c8cef40 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55a58e747b40 0x55a58e747f60 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55a58e74af70 msgr2=0x55a58e74b3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:27.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497c8cef40 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55a58e74af70 0x55a58e74b3b0 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f495c06ac90 msgr2=0x7f495c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:27.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497c8cef40 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f495c06ac90 0x7f495c06d150 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x55a58e69a820 tx=0x7f4964005620 comp rx=0 tx=0).stop 2024-09-16T01:44:27.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a58e5f1fe0 msgr2=0x55a58e731510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:27.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497c8cef40 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a58e5f1fe0 0x55a58e731510 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f496c00d620 tx=0x7f496c000ed0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 shutdown_connections 2024-09-16T01:44:27.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497c8cef40 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f495c06ac90 0x7f495c06d150 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497c8cef40 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55a58e74af70 0x55a58e74b3b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497c8cef40 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x55a58e747b40 0x55a58e747f60 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497c8cef40 1 --2- 172.21.15.88:0/4054319233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a58e5f1fe0 0x55a58e731510 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 >> 172.21.15.88:0/4054319233 conn(0x55a58e60d750 msgr2=0x55a58e693920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:27.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 shutdown_connections 2024-09-16T01:44:27.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.291+0000 7f497c8cef40 1 -- 172.21.15.88:0/4054319233 wait complete. 2024-09-16T01:44:27.297 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapheader mds0_sessionmap deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-09-16T01:44:27.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.379+0000 7f600435ff40 1 Processor -- start 2024-09-16T01:44:27.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.379+0000 7f600435ff40 1 -- start start 2024-09-16T01:44:27.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.380+0000 7f600435ff40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557e624b0fe0 0x557e62556120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.380+0000 7f600435ff40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e624b1a60 0x557e624ca150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.380+0000 7f600435ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e624cb250 0x557e6254f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.380+0000 7f600435ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x557e623d3510 con 0x557e624cb250 2024-09-16T01:44:27.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.380+0000 7f600435ff40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557e6255a120 con 0x557e624b1a60 2024-09-16T01:44:27.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.380+0000 7f600435ff40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x557e6254eb90 con 0x557e624b0fe0 2024-09-16T01:44:27.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.380+0000 7f6002656640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e624cb250 0x557e6254f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:27.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.380+0000 7f6002656640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e624cb250 0x557e6254f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41650/0 (socket says 172.21.15.88:41650) 2024-09-16T01:44:27.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.380+0000 7f6002656640 1 -- 172.21.15.88:0/1947034520 learned_addr learned my addr 172.21.15.88:0/1947034520 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:27.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.380+0000 7f6001e55640 1 --2- 172.21.15.88:0/1947034520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557e624b0fe0 0x557e62556120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:27.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.380+0000 7f5ff9654640 1 --2- 172.21.15.88:0/1947034520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e624b1a60 0x557e624ca150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:27.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.380+0000 7f6002656640 1 -- 172.21.15.88:0/1947034520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557e624b0fe0 msgr2=0x557e62556120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:27.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.380+0000 7f6002656640 1 --2- 172.21.15.88:0/1947034520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557e624b0fe0 0x557e62556120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.380+0000 7f6002656640 1 -- 172.21.15.88:0/1947034520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e624b1a60 msgr2=0x557e624ca150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:27.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.380+0000 7f6002656640 1 --2- 172.21.15.88:0/1947034520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e624b1a60 0x557e624ca150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.380+0000 7f6002656640 1 -- 172.21.15.88:0/1947034520 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e6254c250 con 0x557e624cb250 2024-09-16T01:44:27.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.380+0000 7f6002656640 1 --2- 172.21.15.88:0/1947034520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e624cb250 0x557e6254f0c0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc005fd0 tx=0x7f5ffc050000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc2f0a3517d57440 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:27.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.380+0000 7f6000e53640 1 -- 172.21.15.88:0/1947034520 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc04f990 con 0x557e624cb250 2024-09-16T01:44:27.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.380+0000 7f6000e53640 1 -- 172.21.15.88:0/1947034520 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ffc04fb30 con 0x557e624cb250 2024-09-16T01:44:27.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.380+0000 7f6000e53640 1 -- 172.21.15.88:0/1947034520 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc057400 con 0x557e624cb250 2024-09-16T01:44:27.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.381+0000 7f600435ff40 1 -- 172.21.15.88:0/1947034520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e624cb250 msgr2=0x557e6254f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:27.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.381+0000 7f600435ff40 1 --2- 172.21.15.88:0/1947034520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e624cb250 0x557e6254f0c0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc005fd0 tx=0x7f5ffc050000 comp rx=0 tx=0).stop 2024-09-16T01:44:27.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.381+0000 7f600435ff40 1 -- 172.21.15.88:0/1947034520 shutdown_connections 2024-09-16T01:44:27.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.381+0000 7f600435ff40 1 --2- 172.21.15.88:0/1947034520 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e624cb250 0x557e6254f0c0 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.381+0000 7f600435ff40 1 --2- 172.21.15.88:0/1947034520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e624b1a60 0x557e624ca150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.381+0000 7f600435ff40 1 --2- 172.21.15.88:0/1947034520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557e624b0fe0 0x557e62556120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.381+0000 7f600435ff40 1 -- 172.21.15.88:0/1947034520 >> 172.21.15.88:0/1947034520 conn(0x557e624cc750 msgr2=0x557e62555030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:27.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.381+0000 7f600435ff40 1 -- 172.21.15.88:0/1947034520 shutdown_connections 2024-09-16T01:44:27.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.381+0000 7f600435ff40 1 -- 172.21.15.88:0/1947034520 wait complete. 2024-09-16T01:44:27.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.382+0000 7f600435ff40 1 Processor -- start 2024-09-16T01:44:27.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.382+0000 7f600435ff40 1 -- start start 2024-09-16T01:44:27.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.382+0000 7f600435ff40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e624b0fe0 0x557e6254b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.382+0000 7f600435ff40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557e624b1a60 0x557e6254b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.382+0000 7f600435ff40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e624cb250 0x557e625551d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.382+0000 7f600435ff40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x557e6254e8c0 con 0x557e624b0fe0 2024-09-16T01:44:27.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.382+0000 7f600435ff40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x557e62550ca0 con 0x557e624cb250 2024-09-16T01:44:27.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.382+0000 7f600435ff40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x557e624c8bf0 con 0x557e624b1a60 2024-09-16T01:44:27.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.382+0000 7f6001e55640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e624b0fe0 0x557e6254b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:27.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.382+0000 7f6001e55640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e624b0fe0 0x557e6254b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41652/0 (socket says 172.21.15.88:41652) 2024-09-16T01:44:27.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.382+0000 7f6001e55640 1 -- 172.21.15.88:0/496020099 learned_addr learned my addr 172.21.15.88:0/496020099 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:27.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.382+0000 7f6002656640 1 --2- 172.21.15.88:0/496020099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e624cb250 0x557e625551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:27.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.382+0000 7f5ff9654640 1 --2- 172.21.15.88:0/496020099 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557e624b1a60 0x557e6254b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:27.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.383+0000 7f6001e55640 1 -- 172.21.15.88:0/496020099 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557e624b1a60 msgr2=0x557e6254b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:27.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.383+0000 7f6001e55640 1 --2- 172.21.15.88:0/496020099 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x557e624b1a60 0x557e6254b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.383+0000 7f6001e55640 1 -- 172.21.15.88:0/496020099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e624cb250 msgr2=0x557e625551d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:27.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.383+0000 7f6001e55640 1 --2- 172.21.15.88:0/496020099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e624cb250 0x557e625551d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.383+0000 7f6001e55640 1 -- 172.21.15.88:0/496020099 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e62550f40 con 0x557e624b0fe0 2024-09-16T01:44:27.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.383+0000 7f6001e55640 1 --2- 172.21.15.88:0/496020099 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e624b0fe0 0x557e6254b000 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f5ff40086a0 tx=0x7f5ff4008790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:27.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.383+0000 7f5ffb7fe640 1 -- 172.21.15.88:0/496020099 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff4019020 con 0x557e624b0fe0 2024-09-16T01:44:27.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.383+0000 7f5ffb7fe640 1 -- 172.21.15.88:0/496020099 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ff4015880 con 0x557e624b0fe0 2024-09-16T01:44:27.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.383+0000 7f5ffb7fe640 1 -- 172.21.15.88:0/496020099 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff4015c30 con 0x557e624b0fe0 2024-09-16T01:44:27.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.383+0000 7f600435ff40 1 -- 172.21.15.88:0/496020099 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x557e62551220 con 0x557e624b0fe0 2024-09-16T01:44:27.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.383+0000 7f600435ff40 1 -- 172.21.15.88:0/496020099 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e625517e0 con 0x557e624b0fe0 2024-09-16T01:44:27.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.384+0000 7f5ffb7fe640 1 -- 172.21.15.88:0/496020099 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5ff401c020 con 0x557e624b0fe0 2024-09-16T01:44:27.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.385+0000 7f5ffb7fe640 1 --2- 172.21.15.88:0/496020099 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5fe406acc0 0x7f5fe406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.385+0000 7f5ffb7fe640 1 -- 172.21.15.88:0/496020099 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9006+0+0 (secure 0 0 0) 0x7f5ff408b710 con 0x557e624b0fe0 2024-09-16T01:44:27.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.385+0000 7f5ff9654640 1 --2- 172.21.15.88:0/496020099 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5fe406acc0 0x7f5fe406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:27.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.385+0000 7f600435ff40 1 --2- 172.21.15.88:0/496020099 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x557e625ef220 0x557e625fa830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.385+0000 7f600435ff40 1 -- 172.21.15.88:0/496020099 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:1 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125) -- 0x557e625fad70 con 0x557e625ef220 2024-09-16T01:44:27.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.385+0000 7f6002656640 1 --2- 172.21.15.88:0/496020099 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x557e625ef220 0x557e625fa830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:27.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.385+0000 7f5ff9654640 1 --2- 172.21.15.88:0/496020099 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5fe406acc0 0x7f5fe406d180 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f5fec002fd0 tx=0x7f5fec001790 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:27.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.385+0000 7f6002656640 1 --2- 172.21.15.88:0/496020099 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x557e625ef220 0x557e625fa830 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:27.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.387+0000 7f6002656640 1 -- 172.21.15.88:0/496020099 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(1 mds0_sessionmap [omap-set-header] v125'171 uv171 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f5ffc060070 con 0x557e625ef220 2024-09-16T01:44:27.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.387+0000 7f600435ff40 1 -- 172.21.15.88:0/496020099 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x557e625ef220 msgr2=0x557e625fa830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:27.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.387+0000 7f600435ff40 1 --2- 172.21.15.88:0/496020099 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x557e625ef220 0x557e625fa830 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.387+0000 7f600435ff40 1 -- 172.21.15.88:0/496020099 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5fe406acc0 msgr2=0x7f5fe406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:27.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.387+0000 7f600435ff40 1 --2- 172.21.15.88:0/496020099 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5fe406acc0 0x7f5fe406d180 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f5fec002fd0 tx=0x7f5fec001790 comp rx=0 tx=0).stop 2024-09-16T01:44:27.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.387+0000 7f600435ff40 1 -- 172.21.15.88:0/496020099 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e624b0fe0 msgr2=0x557e6254b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:27.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.387+0000 7f600435ff40 1 --2- 172.21.15.88:0/496020099 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x557e624b0fe0 0x557e6254b000 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f5ff40086a0 tx=0x7f5ff4008790 comp rx=0 tx=0).stop 2024-09-16T01:44:27.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.387+0000 7f6002656640 1 -- 172.21.15.88:0/496020099 reap_dead start 2024-09-16T01:44:27.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.387+0000 7f600435ff40 1 -- 172.21.15.88:0/496020099 shutdown_connections 2024-09-16T01:44:27.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.387+0000 7f600435ff40 1 -- 172.21.15.88:0/496020099 >> 172.21.15.88:0/496020099 conn(0x557e624cc750 msgr2=0x557e624b0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:27.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.387+0000 7f600435ff40 1 -- 172.21.15.88:0/496020099 shutdown_connections 2024-09-16T01:44:27.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.387+0000 7f600435ff40 1 -- 172.21.15.88:0/496020099 wait complete. 2024-09-16T01:44:27.393 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:44:27.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.532+0000 7fb7c59a6640 1 Processor -- start 2024-09-16T01:44:27.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.532+0000 7fb7c59a6640 1 -- start start 2024-09-16T01:44:27.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.533+0000 7fb7c59a6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb7c006e5b0 0x7fb7c0073b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.533+0000 7fb7c59a6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7c015f7f0 0x7fb7c006cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.533+0000 7fb7c59a6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7c006eae0 con 0x7fb7c00b6d50 2024-09-16T01:44:27.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.533+0000 7fb7c59a6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fb7c015ed10 con 0x7fb7c006e5b0 2024-09-16T01:44:27.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.533+0000 7fb7c59a6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb7c015eff0 con 0x7fb7c015f7f0 2024-09-16T01:44:27.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.533+0000 7fb7beffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb7c00b6d50 0x7fb7c00b7130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44208/0 (socket says 172.21.15.88:44208) 2024-09-16T01:44:27.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.533+0000 7fb7beffd640 1 -- 172.21.15.88:0/3259489378 learned_addr learned my addr 172.21.15.88:0/3259489378 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:27.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.533+0000 7fb7b67fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fb7c006e5b0 0x7fb7c0073b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:27.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.534+0000 7fb7b67fc640 1 -- 172.21.15.88:0/3259489378 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7c015f7f0 msgr2=0x7fb7c006cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:27.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.534+0000 7fb7b67fc640 1 --2- 172.21.15.88:0/3259489378 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7c015f7f0 0x7fb7c006cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.534+0000 7fb7b67fc640 1 -- 172.21.15.88:0/3259489378 >> v1:172.21.15.88:6789/0 conn(0x7fb7c00b6d50 legacy=0x7fb7c00b7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:27.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.534+0000 7fb7b67fc640 1 -- 172.21.15.88:0/3259489378 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7c015f2d0 con 0x7fb7c006e5b0 2024-09-16T01:44:27.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.534+0000 7fb7be7fc640 1 -- 172.21.15.88:0/3259489378 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1504558894 0 0) 0x7fb7c006eae0 con 0x7fb7c00b6d50 2024-09-16T01:44:27.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.534+0000 7fb7b67fc640 1 --2- 172.21.15.88:0/3259489378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7c006e5b0 0x7fb7c0073b70 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7fb7a80097a0 tx=0x7fb7a802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=440e2864eac466f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:27.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.535+0000 7fb7be7fc640 1 -- 172.21.15.88:0/3259489378 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8037070 con 0x7fb7c006e5b0 2024-09-16T01:44:27.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.535+0000 7fb7be7fc640 1 -- 172.21.15.88:0/3259489378 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7a8032440 con 0x7fb7c006e5b0 2024-09-16T01:44:27.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.535+0000 7fb7be7fc640 1 -- 172.21.15.88:0/3259489378 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8032810 con 0x7fb7c006e5b0 2024-09-16T01:44:27.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.535+0000 7fb7c59a6640 1 -- 172.21.15.88:0/3259489378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7c006e5b0 msgr2=0x7fb7c0073b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:27.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.535+0000 7fb7c59a6640 1 --2- 172.21.15.88:0/3259489378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7c006e5b0 0x7fb7c0073b70 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7fb7a80097a0 tx=0x7fb7a802a180 comp rx=0 tx=0).stop 2024-09-16T01:44:27.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.535+0000 7fb7c59a6640 1 -- 172.21.15.88:0/3259489378 shutdown_connections 2024-09-16T01:44:27.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.535+0000 7fb7c59a6640 1 --2- 172.21.15.88:0/3259489378 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7c015f7f0 0x7fb7c006cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.535+0000 7fb7c59a6640 1 --2- 172.21.15.88:0/3259489378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7c006e5b0 0x7fb7c0073b70 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.535+0000 7fb7c59a6640 1 -- 172.21.15.88:0/3259489378 >> 172.21.15.88:0/3259489378 conn(0x7fb7c00d24d0 msgr2=0x7fb7c00d28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:27.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.535+0000 7fb7c59a6640 1 -- 172.21.15.88:0/3259489378 shutdown_connections 2024-09-16T01:44:27.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.535+0000 7fb7c59a6640 1 -- 172.21.15.88:0/3259489378 wait complete. 2024-09-16T01:44:27.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.536+0000 7fb7c59a6640 1 Processor -- start 2024-09-16T01:44:27.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.536+0000 7fb7c59a6640 1 -- start start 2024-09-16T01:44:27.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.537+0000 7fb7c59a6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7c006e5b0 0x7fb7c01e9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.537+0000 7fb7c59a6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7c00b6d50 0x7fb7c015bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.537+0000 7fb7c59a6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7c015f7f0 0x7fb7c0144a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.537+0000 7fb7c59a6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c015e8b0 con 0x7fb7c015f7f0 2024-09-16T01:44:27.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.537+0000 7fb7c59a6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c006edc0 con 0x7fb7c006e5b0 2024-09-16T01:44:27.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.537+0000 7fb7c59a6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb7c015ea30 con 0x7fb7c00b6d50 2024-09-16T01:44:27.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.537+0000 7fb7bf7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7c015f7f0 0x7fb7c0144a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:27.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.537+0000 7fb7bf7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7c015f7f0 0x7fb7c0144a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41662/0 (socket says 172.21.15.88:41662) 2024-09-16T01:44:27.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.537+0000 7fb7bf7fe640 1 -- 172.21.15.88:0/785423241 learned_addr learned my addr 172.21.15.88:0/785423241 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:27.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.537+0000 7fb7beffd640 1 --2- 172.21.15.88:0/785423241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7c006e5b0 0x7fb7c01e9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:27.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.537+0000 7fb7b67fc640 1 --2- 172.21.15.88:0/785423241 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7c00b6d50 0x7fb7c015bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:27.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.538+0000 7fb7bf7fe640 1 -- 172.21.15.88:0/785423241 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7c00b6d50 msgr2=0x7fb7c015bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:27.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.538+0000 7fb7bf7fe640 1 --2- 172.21.15.88:0/785423241 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7c00b6d50 0x7fb7c015bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.538+0000 7fb7bf7fe640 1 -- 172.21.15.88:0/785423241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7c006e5b0 msgr2=0x7fb7c01e9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:27.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.538+0000 7fb7bf7fe640 1 --2- 172.21.15.88:0/785423241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7c006e5b0 0x7fb7c01e9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:27.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.538+0000 7fb7bf7fe640 1 -- 172.21.15.88:0/785423241 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7c01ea300 con 0x7fb7c015f7f0 2024-09-16T01:44:27.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.538+0000 7fb7beffd640 1 --2- 172.21.15.88:0/785423241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7c006e5b0 0x7fb7c01e9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:27.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.538+0000 7fb7b67fc640 1 --2- 172.21.15.88:0/785423241 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7c00b6d50 0x7fb7c015bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:27.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.538+0000 7fb7bf7fe640 1 --2- 172.21.15.88:0/785423241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7c015f7f0 0x7fb7c0144a80 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7fb7b800b970 tx=0x7fb7b800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:27.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.538+0000 7fb7bcff9640 1 -- 172.21.15.88:0/785423241 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8019070 con 0x7fb7c015f7f0 2024-09-16T01:44:27.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.538+0000 7fb7bcff9640 1 -- 172.21.15.88:0/785423241 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7b800cb00 con 0x7fb7c015f7f0 2024-09-16T01:44:27.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.538+0000 7fb7c59a6640 1 -- 172.21.15.88:0/785423241 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7c01ea5e0 con 0x7fb7c015f7f0 2024-09-16T01:44:27.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.538+0000 7fb7bcff9640 1 -- 172.21.15.88:0/785423241 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8014440 con 0x7fb7c015f7f0 2024-09-16T01:44:27.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.538+0000 7fb7c59a6640 1 -- 172.21.15.88:0/785423241 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7c01eaba0 con 0x7fb7c015f7f0 2024-09-16T01:44:27.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.539+0000 7fb7c59a6640 1 -- 172.21.15.88:0/785423241 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7c006ae50 con 0x7fb7c015f7f0 2024-09-16T01:44:27.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.540+0000 7fb7bcff9640 1 -- 172.21.15.88:0/785423241 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb7b801c070 con 0x7fb7c015f7f0 2024-09-16T01:44:27.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.540+0000 7fb7bcff9640 1 --2- 172.21.15.88:0/785423241 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb7a006c2b0 0x7fb7a006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:27.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.540+0000 7fb7beffd640 1 --2- 172.21.15.88:0/785423241 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb7a006c2b0 0x7fb7a006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:27.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.540+0000 7fb7bcff9640 1 -- 172.21.15.88:0/785423241 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 9006+0+0 (secure 0 0 0) 0x7fb7b808be40 con 0x7fb7c015f7f0 2024-09-16T01:44:27.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.541+0000 7fb7beffd640 1 --2- 172.21.15.88:0/785423241 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb7a006c2b0 0x7fb7a006e770 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0000f60 tx=0x7fb7b0007ca0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:27.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.544+0000 7fb7bcff9640 1 -- 172.21.15.88:0/785423241 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb7c006ae50 con 0x7fb7c015f7f0 2024-09-16T01:44:27.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:27.747+0000 7fb7c59a6640 1 -- 172.21.15.88:0/785423241 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fb7c00d1e80 con 0x7fb7c015f7f0 2024-09-16T01:44:28.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.078+0000 7fb7bcff9640 1 -- 172.21.15.88:0/785423241 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v308) ==== 162+0+0 (secure 0 0 0) 0x7fb7b8059880 con 0x7fb7c015f7f0 2024-09-16T01:44:28.079 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:44:28.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.080+0000 7fb7c59a6640 1 -- 172.21.15.88:0/785423241 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb7a006c2b0 msgr2=0x7fb7a006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:28.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.080+0000 7fb7c59a6640 1 --2- 172.21.15.88:0/785423241 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb7a006c2b0 0x7fb7a006e770 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0000f60 tx=0x7fb7b0007ca0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.080+0000 7fb7c59a6640 1 -- 172.21.15.88:0/785423241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7c015f7f0 msgr2=0x7fb7c0144a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:28.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.080+0000 7fb7c59a6640 1 --2- 172.21.15.88:0/785423241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7c015f7f0 0x7fb7c0144a80 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7fb7b800b970 tx=0x7fb7b800be30 comp rx=0 tx=0).stop 2024-09-16T01:44:28.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.080+0000 7fb7c59a6640 1 -- 172.21.15.88:0/785423241 shutdown_connections 2024-09-16T01:44:28.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.080+0000 7fb7c59a6640 1 --2- 172.21.15.88:0/785423241 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb7a006c2b0 0x7fb7a006e770 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.080+0000 7fb7c59a6640 1 --2- 172.21.15.88:0/785423241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb7c015f7f0 0x7fb7c0144a80 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.080+0000 7fb7c59a6640 1 --2- 172.21.15.88:0/785423241 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb7c00b6d50 0x7fb7c015bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.080+0000 7fb7c59a6640 1 --2- 172.21.15.88:0/785423241 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7c006e5b0 0x7fb7c01e9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.080+0000 7fb7c59a6640 1 -- 172.21.15.88:0/785423241 >> 172.21.15.88:0/785423241 conn(0x7fb7c00d24d0 msgr2=0x7fb7c006dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:28.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.080+0000 7fb7c59a6640 1 -- 172.21.15.88:0/785423241 shutdown_connections 2024-09-16T01:44:28.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.080+0000 7fb7c59a6640 1 -- 172.21.15.88:0/785423241 wait complete. 2024-09-16T01:44:28.087 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:44:28.086+0000 7fbd679cb640 -1 log_channel(cluster) log [ERR] : corrupt sessionmap header: Decoder at 'virtual void SessionMapStore::decode_header(ceph::bufferlist&)' v=1 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-09-16T01:44:28.088 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:44:28.087+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:44:28.088 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:44:28.087+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:44:28.095 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:44:28.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.195+0000 7f67af549640 1 Processor -- start 2024-09-16T01:44:28.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.195+0000 7f67af549640 1 -- start start 2024-09-16T01:44:28.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.196+0000 7f67af549640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67a80d0fe0 0x7f67a806cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:28.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.196+0000 7f67af549640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f67a806d260 0x7f67a80b5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:28.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.196+0000 7f67af549640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67a815f130 con 0x7f67a815b460 2024-09-16T01:44:28.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.196+0000 7f67af549640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f67a806d6b0 con 0x7f67a806d260 2024-09-16T01:44:28.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.196+0000 7f67af549640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f67a815eee0 con 0x7f67a80d0fe0 2024-09-16T01:44:28.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.196+0000 7f67acae4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67a80d0fe0 0x7f67a806cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:28.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.196+0000 7f67adae6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f67a806d260 0x7f67a80b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:28.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.196+0000 7f67acae4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67a80d0fe0 0x7f67a806cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:35420/0 (socket says 172.21.15.88:35420) 2024-09-16T01:44:28.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.196+0000 7f67acae4640 1 -- 172.21.15.88:0/47217989 learned_addr learned my addr 172.21.15.88:0/47217989 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:28.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.196+0000 7f67acae4640 1 -- 172.21.15.88:0/47217989 >> v2:172.21.15.154:3300/0 conn(0x7f67a806d260 msgr2=0x7f67a80b5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:28.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.196+0000 7f67acae4640 1 --2- 172.21.15.88:0/47217989 >> v2:172.21.15.154:3300/0 conn(0x7f67a806d260 0x7f67a80b5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.196+0000 7f67acae4640 1 -- 172.21.15.88:0/47217989 >> v1:172.21.15.154:6789/0 conn(0x7f67a815b460 legacy=0x7f67a80b44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:28.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.196+0000 7f67acae4640 1 -- 172.21.15.88:0/47217989 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a815f410 con 0x7f67a80d0fe0 2024-09-16T01:44:28.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.196+0000 7f67adae6640 1 --2- 172.21.15.88:0/47217989 >> v2:172.21.15.154:3300/0 conn(0x7f67a806d260 0x7f67a80b5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:44:28.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.197+0000 7f67acae4640 1 --2- 172.21.15.88:0/47217989 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67a80d0fe0 0x7f67a806cd20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6798009870 tx=0x7f679802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8553432c25aae8c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:28.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.197+0000 7f679ffff640 1 -- 172.21.15.88:0/47217989 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6798037070 con 0x7f67a80d0fe0 2024-09-16T01:44:28.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.197+0000 7f679ffff640 1 -- 172.21.15.88:0/47217989 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6798032440 con 0x7f67a80d0fe0 2024-09-16T01:44:28.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.197+0000 7f679ffff640 1 -- 172.21.15.88:0/47217989 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6798032810 con 0x7f67a80d0fe0 2024-09-16T01:44:28.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.198+0000 7f67af549640 1 -- 172.21.15.88:0/47217989 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67a80d0fe0 msgr2=0x7f67a806cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:28.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.198+0000 7f67af549640 1 --2- 172.21.15.88:0/47217989 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67a80d0fe0 0x7f67a806cd20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6798009870 tx=0x7f679802a270 comp rx=0 tx=0).stop 2024-09-16T01:44:28.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.198+0000 7f67af549640 1 -- 172.21.15.88:0/47217989 shutdown_connections 2024-09-16T01:44:28.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.198+0000 7f67af549640 1 --2- 172.21.15.88:0/47217989 >> v2:172.21.15.154:3300/0 conn(0x7f67a806d260 0x7f67a80b5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.198+0000 7f67af549640 1 --2- 172.21.15.88:0/47217989 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67a80d0fe0 0x7f67a806cd20 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.198+0000 7f67af549640 1 -- 172.21.15.88:0/47217989 >> 172.21.15.88:0/47217989 conn(0x7f67a80b6ec0 msgr2=0x7f67a80b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:28.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.198+0000 7f67af549640 1 -- 172.21.15.88:0/47217989 shutdown_connections 2024-09-16T01:44:28.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.198+0000 7f67af549640 1 -- 172.21.15.88:0/47217989 wait complete. 2024-09-16T01:44:28.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.199+0000 7f67af549640 1 Processor -- start 2024-09-16T01:44:28.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.199+0000 7f67af549640 1 -- start start 2024-09-16T01:44:28.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.200+0000 7f67af549640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67a806d260 0x7f67a8147dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:28.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.200+0000 7f67af549640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67a80d0fe0 0x7f67a815bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:28.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.200+0000 7f67af549640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67a815b460 0x7f67a815c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:28.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.200+0000 7f67af549640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f67a80b55c0 con 0x7f67a806d260 2024-09-16T01:44:28.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.200+0000 7f67af549640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f67a815f9d0 con 0x7f67a815b460 2024-09-16T01:44:28.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.200+0000 7f67af549640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f67a80b5740 con 0x7f67a80d0fe0 2024-09-16T01:44:28.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.200+0000 7f67ad2e5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67a806d260 0x7f67a8147dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:28.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.200+0000 7f67ad2e5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67a806d260 0x7f67a8147dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41676/0 (socket says 172.21.15.88:41676) 2024-09-16T01:44:28.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.200+0000 7f67ad2e5640 1 -- 172.21.15.88:0/1052597111 learned_addr learned my addr 172.21.15.88:0/1052597111 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:28.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.200+0000 7f67acae4640 1 --2- 172.21.15.88:0/1052597111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67a80d0fe0 0x7f67a815bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:28.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.200+0000 7f67adae6640 1 --2- 172.21.15.88:0/1052597111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67a815b460 0x7f67a815c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:28.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.200+0000 7f67ad2e5640 1 -- 172.21.15.88:0/1052597111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67a80d0fe0 msgr2=0x7f67a815bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:28.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.200+0000 7f67ad2e5640 1 --2- 172.21.15.88:0/1052597111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67a80d0fe0 0x7f67a815bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.200+0000 7f67ad2e5640 1 -- 172.21.15.88:0/1052597111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67a815b460 msgr2=0x7f67a815c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:28.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.200+0000 7f67ad2e5640 1 --2- 172.21.15.88:0/1052597111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67a815b460 0x7f67a815c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.200+0000 7f67ad2e5640 1 -- 172.21.15.88:0/1052597111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a815c9e0 con 0x7f67a806d260 2024-09-16T01:44:28.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.201+0000 7f67ad2e5640 1 --2- 172.21.15.88:0/1052597111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67a806d260 0x7f67a8147dd0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f67a400c3c0 tx=0x7f67a400c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:28.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.201+0000 7f679dffb640 1 -- 172.21.15.88:0/1052597111 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a40052d0 con 0x7f67a806d260 2024-09-16T01:44:28.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.201+0000 7f679dffb640 1 -- 172.21.15.88:0/1052597111 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67a4005490 con 0x7f67a806d260 2024-09-16T01:44:28.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.201+0000 7f67af549640 1 -- 172.21.15.88:0/1052597111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67a815e160 con 0x7f67a806d260 2024-09-16T01:44:28.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.201+0000 7f679dffb640 1 -- 172.21.15.88:0/1052597111 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a4007260 con 0x7f67a806d260 2024-09-16T01:44:28.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.201+0000 7f67af549640 1 -- 172.21.15.88:0/1052597111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67a814f5e0 con 0x7f67a806d260 2024-09-16T01:44:28.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.202+0000 7f67af549640 1 -- 172.21.15.88:0/1052597111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67a80b5080 con 0x7f67a806d260 2024-09-16T01:44:28.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.202+0000 7f679dffb640 1 -- 172.21.15.88:0/1052597111 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f67a4007400 con 0x7f67a806d260 2024-09-16T01:44:28.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.203+0000 7f679dffb640 1 --2- 172.21.15.88:0/1052597111 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f678c06c2b0 0x7f678c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:28.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.203+0000 7f67acae4640 1 --2- 172.21.15.88:0/1052597111 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f678c06c2b0 0x7f678c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:28.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.203+0000 7f679dffb640 1 -- 172.21.15.88:0/1052597111 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 9092+0+0 (secure 0 0 0) 0x7f67a408b350 con 0x7f67a806d260 2024-09-16T01:44:28.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.203+0000 7f67acae4640 1 --2- 172.21.15.88:0/1052597111 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f678c06c2b0 0x7f678c06e770 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f6798034000 tx=0x7f679802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:28.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.207+0000 7f679dffb640 1 -- 172.21.15.88:0/1052597111 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f67a4058810 con 0x7f67a806d260 2024-09-16T01:44:28.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.413+0000 7f67af549640 1 -- 172.21.15.88:0/1052597111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67a80b4c20 con 0x7f67a806d260 2024-09-16T01:44:28.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.413+0000 7f679dffb640 1 -- 172.21.15.88:0/1052597111 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 310 v310) ==== 95+0+4392 (secure 0 0 0) 0x7f67a405cb10 con 0x7f67a806d260 2024-09-16T01:44:28.415 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:28.415 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":310,"btime":"2024-09-16T01:44:28:087878+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9696,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4270197461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4270197461},{"type":"v1","addr":"172.21.15.88:6837","nonce":4270197461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":310,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:44:28.087806+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:44:28.415 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 310 2024-09-16T01:44:28.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.415+0000 7f67af549640 1 -- 172.21.15.88:0/1052597111 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f678c06c2b0 msgr2=0x7f678c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:28.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.415+0000 7f67af549640 1 --2- 172.21.15.88:0/1052597111 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f678c06c2b0 0x7f678c06e770 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f6798034000 tx=0x7f679802e000 comp rx=0 tx=0).stop 2024-09-16T01:44:28.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.415+0000 7f67af549640 1 -- 172.21.15.88:0/1052597111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67a806d260 msgr2=0x7f67a8147dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:28.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.415+0000 7f67af549640 1 --2- 172.21.15.88:0/1052597111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67a806d260 0x7f67a8147dd0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f67a400c3c0 tx=0x7f67a400c7c0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.415+0000 7f67af549640 1 -- 172.21.15.88:0/1052597111 shutdown_connections 2024-09-16T01:44:28.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.415+0000 7f67af549640 1 --2- 172.21.15.88:0/1052597111 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f678c06c2b0 0x7f678c06e770 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.415+0000 7f67af549640 1 --2- 172.21.15.88:0/1052597111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f67a815b460 0x7f67a815c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.415+0000 7f67af549640 1 --2- 172.21.15.88:0/1052597111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f67a80d0fe0 0x7f67a815bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.415+0000 7f67af549640 1 --2- 172.21.15.88:0/1052597111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f67a806d260 0x7f67a8147dd0 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.415+0000 7f67af549640 1 -- 172.21.15.88:0/1052597111 >> 172.21.15.88:0/1052597111 conn(0x7f67a80b6ec0 msgr2=0x7f67a81553d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:28.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.415+0000 7f67af549640 1 -- 172.21.15.88:0/1052597111 shutdown_connections 2024-09-16T01:44:28.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.415+0000 7f67af549640 1 -- 172.21.15.88:0/1052597111 wait complete. 2024-09-16T01:44:28.427 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:44:28.428 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:44:28.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.525+0000 7f2e25bf8640 1 Processor -- start 2024-09-16T01:44:28.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.525+0000 7f2e25bf8640 1 -- start start 2024-09-16T01:44:28.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.526+0000 7f2e25bf8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2e20155800 0x7f2e200b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:28.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.526+0000 7f2e25bf8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e200b50a0 con 0x7f2e200cfc80 2024-09-16T01:44:28.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.526+0000 7f2e25bf8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e200b5380 con 0x7f2e2015c330 2024-09-16T01:44:28.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.526+0000 7f2e25bf8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2e2015c7a0 con 0x7f2e20155800 2024-09-16T01:44:28.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.526+0000 7f2e1f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2e20155800 0x7f2e200b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:28.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.526+0000 7f2e1ffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f2e200cfc80 0x7f2e2015c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44220/0 (socket says 172.21.15.88:44220) 2024-09-16T01:44:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.526+0000 7f2e1ffff640 1 -- 172.21.15.88:0/2671797653 learned_addr learned my addr 172.21.15.88:0/2671797653 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.526+0000 7f2e1f7fe640 1 --2- 172.21.15.88:0/2671797653 >> v2:172.21.15.88:3300/0 conn(0x7f2e20155800 0x7f2e200b6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41686/0 (socket says 172.21.15.88:41686) 2024-09-16T01:44:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.526+0000 7f2e1f7fe640 1 -- 172.21.15.88:0/2671797653 >> v1:172.21.15.154:6789/0 conn(0x7f2e2015c330 legacy=0x7f2e200cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.526+0000 7f2e1f7fe640 1 -- 172.21.15.88:0/2671797653 >> v1:172.21.15.88:6789/0 conn(0x7f2e200cfc80 legacy=0x7f2e2015c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.526+0000 7f2e1f7fe640 1 -- 172.21.15.88:0/2671797653 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e200b5660 con 0x7f2e20155800 2024-09-16T01:44:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.526+0000 7f2e1e7fc640 1 -- 172.21.15.88:0/2671797653 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3295890255 0 0) 0x7f2e200b50a0 con 0x7f2e200cfc80 2024-09-16T01:44:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.527+0000 7f2e1f7fe640 1 --2- 172.21.15.88:0/2671797653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e20155800 0x7f2e200b6140 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f2e100097a0 tx=0x7f2e1002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cc03da24f1db045 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.527+0000 7f2e1e7fc640 1 -- 172.21.15.88:0/2671797653 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e10037070 con 0x7f2e20155800 2024-09-16T01:44:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.527+0000 7f2e1e7fc640 1 -- 172.21.15.88:0/2671797653 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e10032440 con 0x7f2e20155800 2024-09-16T01:44:28.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.527+0000 7f2e1e7fc640 1 -- 172.21.15.88:0/2671797653 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e10032810 con 0x7f2e20155800 2024-09-16T01:44:28.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.527+0000 7f2e25bf8640 1 -- 172.21.15.88:0/2671797653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e20155800 msgr2=0x7f2e200b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:28.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.527+0000 7f2e25bf8640 1 --2- 172.21.15.88:0/2671797653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e20155800 0x7f2e200b6140 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f2e100097a0 tx=0x7f2e1002a1e0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.528+0000 7f2e25bf8640 1 -- 172.21.15.88:0/2671797653 shutdown_connections 2024-09-16T01:44:28.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.528+0000 7f2e25bf8640 1 --2- 172.21.15.88:0/2671797653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e20155800 0x7f2e200b6140 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.528+0000 7f2e25bf8640 1 -- 172.21.15.88:0/2671797653 >> 172.21.15.88:0/2671797653 conn(0x7f2e2006bc10 msgr2=0x7f2e2006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:28.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.528+0000 7f2e25bf8640 1 -- 172.21.15.88:0/2671797653 shutdown_connections 2024-09-16T01:44:28.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.528+0000 7f2e25bf8640 1 -- 172.21.15.88:0/2671797653 wait complete. 2024-09-16T01:44:28.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.529+0000 7f2e25bf8640 1 Processor -- start 2024-09-16T01:44:28.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.529+0000 7f2e25bf8640 1 -- start start 2024-09-16T01:44:28.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.530+0000 7f2e25bf8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e200cfc80 0x7f2e20149fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:28.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.530+0000 7f2e25bf8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e20155800 0x7f2e20158970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:28.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.530+0000 7f2e25bf8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e2015c330 0x7f2e200cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:28.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.530+0000 7f2e25bf8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2e20155be0 con 0x7f2e2015c330 2024-09-16T01:44:28.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.530+0000 7f2e25bf8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2e200b5c20 con 0x7f2e20155800 2024-09-16T01:44:28.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.530+0000 7f2e25bf8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2e200b4920 con 0x7f2e200cfc80 2024-09-16T01:44:28.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.530+0000 7f2e1ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e2015c330 0x7f2e200cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:28.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.530+0000 7f2e1ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e2015c330 0x7f2e200cc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41702/0 (socket says 172.21.15.88:41702) 2024-09-16T01:44:28.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.530+0000 7f2e1ffff640 1 -- 172.21.15.88:0/313020073 learned_addr learned my addr 172.21.15.88:0/313020073 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:28.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.530+0000 7f2e1effd640 1 --2- 172.21.15.88:0/313020073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e20155800 0x7f2e20158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:28.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.530+0000 7f2e1f7fe640 1 --2- 172.21.15.88:0/313020073 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e200cfc80 0x7f2e20149fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:28.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.530+0000 7f2e1ffff640 1 -- 172.21.15.88:0/313020073 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e200cfc80 msgr2=0x7f2e20149fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:28.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.530+0000 7f2e1ffff640 1 --2- 172.21.15.88:0/313020073 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e200cfc80 0x7f2e20149fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.530+0000 7f2e1ffff640 1 -- 172.21.15.88:0/313020073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e20155800 msgr2=0x7f2e20158970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:28.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.530+0000 7f2e1ffff640 1 --2- 172.21.15.88:0/313020073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e20155800 0x7f2e20158970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.530+0000 7f2e1ffff640 1 -- 172.21.15.88:0/313020073 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e2014a4e0 con 0x7f2e2015c330 2024-09-16T01:44:28.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.530+0000 7f2e1ffff640 1 --2- 172.21.15.88:0/313020073 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e2015c330 0x7f2e200cc0c0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f2e1800c800 tx=0x7f2e1800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:28.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.530+0000 7f2e07fff640 1 -- 172.21.15.88:0/313020073 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e18018070 con 0x7f2e2015c330 2024-09-16T01:44:28.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.530+0000 7f2e07fff640 1 -- 172.21.15.88:0/313020073 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e18004d80 con 0x7f2e2015c330 2024-09-16T01:44:28.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.530+0000 7f2e07fff640 1 -- 172.21.15.88:0/313020073 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1800a880 con 0x7f2e2015c330 2024-09-16T01:44:28.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.530+0000 7f2e25bf8640 1 -- 172.21.15.88:0/313020073 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e201572c0 con 0x7f2e2015c330 2024-09-16T01:44:28.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.530+0000 7f2e25bf8640 1 -- 172.21.15.88:0/313020073 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e20157880 con 0x7f2e2015c330 2024-09-16T01:44:28.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.531+0000 7f2e25bf8640 1 -- 172.21.15.88:0/313020073 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e200b4cb0 con 0x7f2e2015c330 2024-09-16T01:44:28.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.532+0000 7f2e07fff640 1 -- 172.21.15.88:0/313020073 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2e1801b070 con 0x7f2e2015c330 2024-09-16T01:44:28.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.532+0000 7f2e07fff640 1 --2- 172.21.15.88:0/313020073 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2e0c06c2b0 0x7f2e0c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:28.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.532+0000 7f2e1f7fe640 1 --2- 172.21.15.88:0/313020073 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2e0c06c2b0 0x7f2e0c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:28.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.533+0000 7f2e07fff640 1 -- 172.21.15.88:0/313020073 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 9092+0+0 (secure 0 0 0) 0x7f2e1808b770 con 0x7f2e2015c330 2024-09-16T01:44:28.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.533+0000 7f2e1f7fe640 1 --2- 172.21.15.88:0/313020073 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2e0c06c2b0 0x7f2e0c06e770 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f2e10034000 tx=0x7f2e1002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:28.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.537+0000 7f2e07fff640 1 -- 172.21.15.88:0/313020073 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2e180591b0 con 0x7f2e2015c330 2024-09-16T01:44:28.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.738+0000 7f2e25bf8640 1 -- 172.21.15.88:0/313020073 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e20158eb0 con 0x7f2e2015c330 2024-09-16T01:44:28.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.739+0000 7f2e07fff640 1 -- 172.21.15.88:0/313020073 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 310 v310) ==== 95+0+4392 (secure 0 0 0) 0x7f2e18058ab0 con 0x7f2e2015c330 2024-09-16T01:44:28.741 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:28.741 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":310,"btime":"2024-09-16T01:44:28:087878+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9696,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4270197461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4270197461},{"type":"v1","addr":"172.21.15.88:6837","nonce":4270197461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":310,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:44:28.087806+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:44:28.741 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 310 2024-09-16T01:44:28.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.741+0000 7f2e25bf8640 1 -- 172.21.15.88:0/313020073 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2e0c06c2b0 msgr2=0x7f2e0c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:28.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.741+0000 7f2e25bf8640 1 --2- 172.21.15.88:0/313020073 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2e0c06c2b0 0x7f2e0c06e770 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f2e10034000 tx=0x7f2e1002e000 comp rx=0 tx=0).stop 2024-09-16T01:44:28.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.741+0000 7f2e25bf8640 1 -- 172.21.15.88:0/313020073 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e2015c330 msgr2=0x7f2e200cc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:28.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.741+0000 7f2e25bf8640 1 --2- 172.21.15.88:0/313020073 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e2015c330 0x7f2e200cc0c0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f2e1800c800 tx=0x7f2e1800ccc0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.741+0000 7f2e25bf8640 1 -- 172.21.15.88:0/313020073 shutdown_connections 2024-09-16T01:44:28.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.741+0000 7f2e25bf8640 1 --2- 172.21.15.88:0/313020073 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2e0c06c2b0 0x7f2e0c06e770 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.741+0000 7f2e25bf8640 1 --2- 172.21.15.88:0/313020073 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e2015c330 0x7f2e200cc0c0 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.741+0000 7f2e25bf8640 1 --2- 172.21.15.88:0/313020073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e20155800 0x7f2e20158970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.741+0000 7f2e25bf8640 1 --2- 172.21.15.88:0/313020073 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2e200cfc80 0x7f2e20149fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.741+0000 7f2e25bf8640 1 -- 172.21.15.88:0/313020073 >> 172.21.15.88:0/313020073 conn(0x7f2e2006bc10 msgr2=0x7f2e200712e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:28.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.741+0000 7f2e25bf8640 1 -- 172.21.15.88:0/313020073 shutdown_connections 2024-09-16T01:44:28.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.741+0000 7f2e25bf8640 1 -- 172.21.15.88:0/313020073 wait complete. 2024-09-16T01:44:28.753 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:44:28.754 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:44:28.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.850+0000 7ff54fca5640 1 Processor -- start 2024-09-16T01:44:28.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.851+0000 7ff54fca5640 1 -- start start 2024-09-16T01:44:28.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.851+0000 7ff54fca5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff54815e950 0x7ff54806ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:28.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.851+0000 7ff54fca5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff5480735b0 0x7ff54806d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:28.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.851+0000 7ff54fca5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff54806faf0 con 0x7ff548072050 2024-09-16T01:44:28.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.852+0000 7ff54fca5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff54806fdd0 con 0x7ff5480735b0 2024-09-16T01:44:28.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.852+0000 7ff54fca5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff5480700b0 con 0x7ff54815e950 2024-09-16T01:44:28.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.852+0000 7ff54d240640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff5480735b0 0x7ff54806d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:28.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.852+0000 7ff54d240640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff5480735b0 0x7ff54806d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41714/0 (socket says 172.21.15.88:41714) 2024-09-16T01:44:28.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.852+0000 7ff54e242640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff548072050 0x7ff54806bc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44234/0 (socket says 172.21.15.88:44234) 2024-09-16T01:44:28.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.852+0000 7ff54d240640 1 -- 172.21.15.88:0/335016505 learned_addr learned my addr 172.21.15.88:0/335016505 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:28.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.852+0000 7ff54da41640 1 --2- 172.21.15.88:0/335016505 >> v2:172.21.15.154:3300/0 conn(0x7ff54815e950 0x7ff54806ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:28.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.852+0000 7ff54d240640 1 -- 172.21.15.88:0/335016505 >> v2:172.21.15.154:3300/0 conn(0x7ff54815e950 msgr2=0x7ff54806ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:28.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.852+0000 7ff54d240640 1 --2- 172.21.15.88:0/335016505 >> v2:172.21.15.154:3300/0 conn(0x7ff54815e950 0x7ff54806ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.852+0000 7ff54d240640 1 -- 172.21.15.88:0/335016505 >> v1:172.21.15.88:6789/0 conn(0x7ff548072050 legacy=0x7ff54806bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:28.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.852+0000 7ff54d240640 1 -- 172.21.15.88:0/335016505 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff548070390 con 0x7ff5480735b0 2024-09-16T01:44:28.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.852+0000 7ff54ca3f640 1 -- 172.21.15.88:0/335016505 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 149580545 0 0) 0x7ff54806faf0 con 0x7ff548072050 2024-09-16T01:44:28.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.853+0000 7ff54d240640 1 --2- 172.21.15.88:0/335016505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5480735b0 0x7ff54806d370 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7ff548069190 tx=0x7ff53c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3b75757a4b3baea server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:28.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.853+0000 7ff54ca3f640 1 -- 172.21.15.88:0/335016505 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff53c037070 con 0x7ff5480735b0 2024-09-16T01:44:28.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.853+0000 7ff54ca3f640 1 -- 172.21.15.88:0/335016505 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff53c032440 con 0x7ff5480735b0 2024-09-16T01:44:28.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.853+0000 7ff54ca3f640 1 -- 172.21.15.88:0/335016505 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff53c032810 con 0x7ff5480735b0 2024-09-16T01:44:28.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.853+0000 7ff54fca5640 1 -- 172.21.15.88:0/335016505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5480735b0 msgr2=0x7ff54806d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:28.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.853+0000 7ff54fca5640 1 --2- 172.21.15.88:0/335016505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5480735b0 0x7ff54806d370 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7ff548069190 tx=0x7ff53c02a270 comp rx=0 tx=0).stop 2024-09-16T01:44:28.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.853+0000 7ff54fca5640 1 -- 172.21.15.88:0/335016505 shutdown_connections 2024-09-16T01:44:28.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.853+0000 7ff54fca5640 1 --2- 172.21.15.88:0/335016505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5480735b0 0x7ff54806d370 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.853+0000 7ff54fca5640 1 --2- 172.21.15.88:0/335016505 >> v2:172.21.15.154:3300/0 conn(0x7ff54815e950 0x7ff54806ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.853+0000 7ff54fca5640 1 -- 172.21.15.88:0/335016505 >> 172.21.15.88:0/335016505 conn(0x7ff5480d14d0 msgr2=0x7ff5480d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:28.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.853+0000 7ff54fca5640 1 -- 172.21.15.88:0/335016505 shutdown_connections 2024-09-16T01:44:28.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.853+0000 7ff54fca5640 1 -- 172.21.15.88:0/335016505 wait complete. 2024-09-16T01:44:28.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.855+0000 7ff54fca5640 1 Processor -- start 2024-09-16T01:44:28.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.855+0000 7ff54fca5640 1 -- start start 2024-09-16T01:44:28.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.855+0000 7ff54fca5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff548072050 0x7ff54814b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:28.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.855+0000 7ff54fca5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5480735b0 0x7ff548156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:28.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.855+0000 7ff54fca5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff54815e950 0x7ff5481570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:28.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.855+0000 7ff54fca5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff5480d0230 con 0x7ff548072050 2024-09-16T01:44:28.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.855+0000 7ff54fca5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff5480d00b0 con 0x7ff54815e950 2024-09-16T01:44:28.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.855+0000 7ff54fca5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff5480d03b0 con 0x7ff5480735b0 2024-09-16T01:44:28.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.855+0000 7ff54da41640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff548072050 0x7ff54814b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:28.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.855+0000 7ff54da41640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff548072050 0x7ff54814b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41724/0 (socket says 172.21.15.88:41724) 2024-09-16T01:44:28.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.855+0000 7ff54da41640 1 -- 172.21.15.88:0/1385396976 learned_addr learned my addr 172.21.15.88:0/1385396976 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:28.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.856+0000 7ff54e242640 1 --2- 172.21.15.88:0/1385396976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff54815e950 0x7ff5481570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:28.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.856+0000 7ff54d240640 1 --2- 172.21.15.88:0/1385396976 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5480735b0 0x7ff548156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:28.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.856+0000 7ff54da41640 1 -- 172.21.15.88:0/1385396976 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5480735b0 msgr2=0x7ff548156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:28.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.856+0000 7ff54da41640 1 --2- 172.21.15.88:0/1385396976 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5480735b0 0x7ff548156b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.856+0000 7ff54da41640 1 -- 172.21.15.88:0/1385396976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff54815e950 msgr2=0x7ff5481570d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:28.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.856+0000 7ff54da41640 1 --2- 172.21.15.88:0/1385396976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff54815e950 0x7ff5481570d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:28.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.856+0000 7ff54da41640 1 -- 172.21.15.88:0/1385396976 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff54814b680 con 0x7ff548072050 2024-09-16T01:44:28.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.856+0000 7ff54e242640 1 --2- 172.21.15.88:0/1385396976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff54815e950 0x7ff5481570d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:28.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.856+0000 7ff54da41640 1 --2- 172.21.15.88:0/1385396976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff548072050 0x7ff54814b140 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7ff54400d830 tx=0x7ff54400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:28.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.856+0000 7ff53a7fc640 1 -- 172.21.15.88:0/1385396976 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54400b6c0 con 0x7ff548072050 2024-09-16T01:44:28.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.856+0000 7ff53a7fc640 1 -- 172.21.15.88:0/1385396976 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff544019070 con 0x7ff548072050 2024-09-16T01:44:28.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.856+0000 7ff53a7fc640 1 -- 172.21.15.88:0/1385396976 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54400ba80 con 0x7ff548072050 2024-09-16T01:44:28.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.856+0000 7ff54fca5640 1 -- 172.21.15.88:0/1385396976 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff548161ab0 con 0x7ff548072050 2024-09-16T01:44:28.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.856+0000 7ff54fca5640 1 -- 172.21.15.88:0/1385396976 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff54815fba0 con 0x7ff548072050 2024-09-16T01:44:28.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.857+0000 7ff54fca5640 1 -- 172.21.15.88:0/1385396976 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff54806ce30 con 0x7ff548072050 2024-09-16T01:44:28.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.857+0000 7ff53a7fc640 1 -- 172.21.15.88:0/1385396976 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff54401e460 con 0x7ff548072050 2024-09-16T01:44:28.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.858+0000 7ff53a7fc640 1 --2- 172.21.15.88:0/1385396976 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff52c06c2b0 0x7ff52c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:28.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.858+0000 7ff53a7fc640 1 -- 172.21.15.88:0/1385396976 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 9092+0+0 (secure 0 0 0) 0x7ff54408c800 con 0x7ff548072050 2024-09-16T01:44:28.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.858+0000 7ff54d240640 1 --2- 172.21.15.88:0/1385396976 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff52c06c2b0 0x7ff52c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:28.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.858+0000 7ff54d240640 1 --2- 172.21.15.88:0/1385396976 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff52c06c2b0 0x7ff52c06e770 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7ff53c034000 tx=0x7ff53c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:28.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:28.862+0000 7ff53a7fc640 1 -- 172.21.15.88:0/1385396976 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff54814b680 con 0x7ff548072050 2024-09-16T01:44:29.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.061+0000 7ff54fca5640 1 -- 172.21.15.88:0/1385396976 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5480b4d90 con 0x7ff548072050 2024-09-16T01:44:29.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.062+0000 7ff53a7fc640 1 -- 172.21.15.88:0/1385396976 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 310 v310) ==== 95+0+4392 (secure 0 0 0) 0x7ff54405a270 con 0x7ff548072050 2024-09-16T01:44:29.063 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:29.063 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":310,"btime":"2024-09-16T01:44:28:087878+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":9696,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/4270197461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4270197461},{"type":"v1","addr":"172.21.15.88:6837","nonce":4270197461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":257}],"filesystems":[{"mdsmap":{"epoch":310,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:44:28.087806+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:44:29.064 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 310 2024-09-16T01:44:29.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.064+0000 7ff54fca5640 1 -- 172.21.15.88:0/1385396976 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff52c06c2b0 msgr2=0x7ff52c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:29.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.064+0000 7ff54fca5640 1 --2- 172.21.15.88:0/1385396976 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff52c06c2b0 0x7ff52c06e770 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7ff53c034000 tx=0x7ff53c02e000 comp rx=0 tx=0).stop 2024-09-16T01:44:29.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.064+0000 7ff54fca5640 1 -- 172.21.15.88:0/1385396976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff548072050 msgr2=0x7ff54814b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:29.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.064+0000 7ff54fca5640 1 --2- 172.21.15.88:0/1385396976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff548072050 0x7ff54814b140 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7ff54400d830 tx=0x7ff54400dcf0 comp rx=0 tx=0).stop 2024-09-16T01:44:29.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.064+0000 7ff54fca5640 1 -- 172.21.15.88:0/1385396976 shutdown_connections 2024-09-16T01:44:29.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.064+0000 7ff54fca5640 1 --2- 172.21.15.88:0/1385396976 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff52c06c2b0 0x7ff52c06e770 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:29.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.064+0000 7ff54fca5640 1 --2- 172.21.15.88:0/1385396976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff54815e950 0x7ff5481570d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:29.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.064+0000 7ff54fca5640 1 --2- 172.21.15.88:0/1385396976 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff5480735b0 0x7ff548156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:29.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.064+0000 7ff54fca5640 1 --2- 172.21.15.88:0/1385396976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff548072050 0x7ff54814b140 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:29.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.064+0000 7ff54fca5640 1 -- 172.21.15.88:0/1385396976 >> 172.21.15.88:0/1385396976 conn(0x7ff5480d14d0 msgr2=0x7ff5480b5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:29.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.064+0000 7ff54fca5640 1 -- 172.21.15.88:0/1385396976 shutdown_connections 2024-09-16T01:44:29.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.064+0000 7ff54fca5640 1 -- 172.21.15.88:0/1385396976 wait complete. 2024-09-16T01:44:29.076 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt omap header on mds0_sessionmap' led to DAMAGED state 2024-09-16T01:44:29.076 INFO:tasks.cephfs.test_damage:Applying mutation 'Corrupt omap header on 1.00000000' 2024-09-16T01:44:29.076 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:44:29.076 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:44:29.092 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:44:29.092 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:44:29.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:29.092 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:44:29.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:44:29.148 INFO:teuthology.orchestra.run.smithi088.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T01:44:29.148 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:44:29.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.286+0000 7f6f7775b640 1 Processor -- start 2024-09-16T01:44:29.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.286+0000 7f6f7775b640 1 -- start start 2024-09-16T01:44:29.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.287+0000 7f6f7775b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6f7015b480 0x7f6f700b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:29.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.287+0000 7f6f7775b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6f700b60c0 0x7f6f70071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:29.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.287+0000 7f6f7775b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f700ccff0 con 0x7f6f70072530 2024-09-16T01:44:29.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.287+0000 7f6f7775b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6f70072980 con 0x7f6f700b60c0 2024-09-16T01:44:29.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.287+0000 7f6f7775b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6f700cd2d0 con 0x7f6f7015b480 2024-09-16T01:44:29.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.287+0000 7f6f74cf6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6f700b60c0 0x7f6f70071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:29.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.287+0000 7f6f75cf8640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6f70072530 0x7f6f70070fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:44246/0 (socket says 172.21.15.88:44246) 2024-09-16T01:44:29.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.287+0000 7f6f74cf6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6f700b60c0 0x7f6f70071ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41726/0 (socket says 172.21.15.88:41726) 2024-09-16T01:44:29.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.287+0000 7f6f75cf8640 1 -- 172.21.15.88:0/1495953740 learned_addr learned my addr 172.21.15.88:0/1495953740 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:29.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.287+0000 7f6f74cf6640 1 -- 172.21.15.88:0/1495953740 >> v2:172.21.15.154:3300/0 conn(0x7f6f7015b480 msgr2=0x7f6f700b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:29.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.287+0000 7f6f754f7640 1 --2- 172.21.15.88:0/1495953740 >> v2:172.21.15.154:3300/0 conn(0x7f6f7015b480 0x7f6f700b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:29.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.287+0000 7f6f74cf6640 1 --2- 172.21.15.88:0/1495953740 >> v2:172.21.15.154:3300/0 conn(0x7f6f7015b480 0x7f6f700b5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:29.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.288+0000 7f6f74cf6640 1 -- 172.21.15.88:0/1495953740 >> v1:172.21.15.88:6789/0 conn(0x7f6f70072530 legacy=0x7f6f70070fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:29.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.288+0000 7f6f74cf6640 1 -- 172.21.15.88:0/1495953740 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f700cd5b0 con 0x7f6f700b60c0 2024-09-16T01:44:29.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.288+0000 7f6f67fff640 1 -- 172.21.15.88:0/1495953740 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2967994051 0 0) 0x7f6f700ccff0 con 0x7f6f70072530 2024-09-16T01:44:29.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.288+0000 7f6f754f7640 1 --2- 172.21.15.88:0/1495953740 >> v2:172.21.15.154:3300/0 conn(0x7f6f7015b480 0x7f6f700b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:29.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.288+0000 7f6f74cf6640 1 --2- 172.21.15.88:0/1495953740 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f700b60c0 0x7f6f70071ff0 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f6f700694d0 tx=0x7f6f6002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c0ea5869afc59f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:29.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.288+0000 7f6f67fff640 1 -- 172.21.15.88:0/1495953740 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f60037070 con 0x7f6f700b60c0 2024-09-16T01:44:29.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.288+0000 7f6f67fff640 1 -- 172.21.15.88:0/1495953740 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f60032440 con 0x7f6f700b60c0 2024-09-16T01:44:29.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.288+0000 7f6f67fff640 1 -- 172.21.15.88:0/1495953740 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f60032810 con 0x7f6f700b60c0 2024-09-16T01:44:29.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.289+0000 7f6f7775b640 1 -- 172.21.15.88:0/1495953740 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f700b60c0 msgr2=0x7f6f70071ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:29.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.289+0000 7f6f7775b640 1 --2- 172.21.15.88:0/1495953740 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f700b60c0 0x7f6f70071ff0 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f6f700694d0 tx=0x7f6f6002a1a0 comp rx=0 tx=0).stop 2024-09-16T01:44:29.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.289+0000 7f6f7775b640 1 -- 172.21.15.88:0/1495953740 shutdown_connections 2024-09-16T01:44:29.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.289+0000 7f6f7775b640 1 --2- 172.21.15.88:0/1495953740 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f700b60c0 0x7f6f70071ff0 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:29.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.289+0000 7f6f7775b640 1 --2- 172.21.15.88:0/1495953740 >> v2:172.21.15.154:3300/0 conn(0x7f6f7015b480 0x7f6f700b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:29.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.289+0000 7f6f7775b640 1 -- 172.21.15.88:0/1495953740 >> 172.21.15.88:0/1495953740 conn(0x7f6f7006d0f0 msgr2=0x7f6f7006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:29.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.289+0000 7f6f7775b640 1 -- 172.21.15.88:0/1495953740 shutdown_connections 2024-09-16T01:44:29.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.289+0000 7f6f7775b640 1 -- 172.21.15.88:0/1495953740 wait complete. 2024-09-16T01:44:29.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.290+0000 7f6f7775b640 1 Processor -- start 2024-09-16T01:44:29.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.290+0000 7f6f7775b640 1 -- start start 2024-09-16T01:44:29.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.291+0000 7f6f7775b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f70072530 0x7f6f701442f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:29.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.291+0000 7f6f7775b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f700b60c0 0x7f6f700c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:29.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.291+0000 7f6f7775b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f7015b480 0x7f6f7014d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:29.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.291+0000 7f6f7775b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6f7015b860 con 0x7f6f70072530 2024-09-16T01:44:29.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.291+0000 7f6f7775b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6f700cdb70 con 0x7f6f700b60c0 2024-09-16T01:44:29.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.291+0000 7f6f7775b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6f7015b9e0 con 0x7f6f7015b480 2024-09-16T01:44:29.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.291+0000 7f6f754f7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f70072530 0x7f6f701442f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:29.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.291+0000 7f6f754f7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f70072530 0x7f6f701442f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41736/0 (socket says 172.21.15.88:41736) 2024-09-16T01:44:29.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.291+0000 7f6f754f7640 1 -- 172.21.15.88:0/837317677 learned_addr learned my addr 172.21.15.88:0/837317677 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:29.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.291+0000 7f6f74cf6640 1 --2- 172.21.15.88:0/837317677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f700b60c0 0x7f6f700c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:29.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.291+0000 7f6f75cf8640 1 --2- 172.21.15.88:0/837317677 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f7015b480 0x7f6f7014d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:29.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.291+0000 7f6f754f7640 1 -- 172.21.15.88:0/837317677 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f7015b480 msgr2=0x7f6f7014d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:29.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.291+0000 7f6f754f7640 1 --2- 172.21.15.88:0/837317677 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f7015b480 0x7f6f7014d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:29.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.291+0000 7f6f754f7640 1 -- 172.21.15.88:0/837317677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f700b60c0 msgr2=0x7f6f700c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:29.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.291+0000 7f6f754f7640 1 --2- 172.21.15.88:0/837317677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f700b60c0 0x7f6f700c9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:29.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.291+0000 7f6f754f7640 1 -- 172.21.15.88:0/837317677 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f70144ec0 con 0x7f6f70072530 2024-09-16T01:44:29.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.291+0000 7f6f75cf8640 1 --2- 172.21.15.88:0/837317677 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f7015b480 0x7f6f7014d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:29.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.291+0000 7f6f754f7640 1 --2- 172.21.15.88:0/837317677 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f70072530 0x7f6f701442f0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00c7f0 tx=0x7f6f6c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:29.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.292+0000 7f6f65ffb640 1 -- 172.21.15.88:0/837317677 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c015070 con 0x7f6f70072530 2024-09-16T01:44:29.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.292+0000 7f6f65ffb640 1 -- 172.21.15.88:0/837317677 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f6c007820 con 0x7f6f70072530 2024-09-16T01:44:29.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.292+0000 7f6f7775b640 1 -- 172.21.15.88:0/837317677 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f701529c0 con 0x7f6f70072530 2024-09-16T01:44:29.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.292+0000 7f6f65ffb640 1 -- 172.21.15.88:0/837317677 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c007bd0 con 0x7f6f70072530 2024-09-16T01:44:29.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.292+0000 7f6f7775b640 1 -- 172.21.15.88:0/837317677 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f70153200 con 0x7f6f70072530 2024-09-16T01:44:29.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.293+0000 7f6f7775b640 1 -- 172.21.15.88:0/837317677 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f70071ff0 con 0x7f6f70072530 2024-09-16T01:44:29.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.293+0000 7f6f65ffb640 1 -- 172.21.15.88:0/837317677 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6f6c018070 con 0x7f6f70072530 2024-09-16T01:44:29.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.294+0000 7f6f65ffb640 1 --2- 172.21.15.88:0/837317677 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6f5406c2b0 0x7f6f5406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:29.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.294+0000 7f6f65ffb640 1 -- 172.21.15.88:0/837317677 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 9092+0+0 (secure 0 0 0) 0x7f6f6c08c1f0 con 0x7f6f70072530 2024-09-16T01:44:29.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.294+0000 7f6f74cf6640 1 --2- 172.21.15.88:0/837317677 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6f5406c2b0 0x7f6f5406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:29.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.294+0000 7f6f74cf6640 1 --2- 172.21.15.88:0/837317677 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6f5406c2b0 0x7f6f5406e770 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f6f60034000 tx=0x7f6f6002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:29.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.298+0000 7f6f65ffb640 1 -- 172.21.15.88:0/837317677 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6f701529c0 con 0x7f6f70072530 2024-09-16T01:44:29.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:29.497+0000 7f6f7775b640 1 -- 172.21.15.88:0/837317677 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f6f7006ae50 con 0x7f6f70072530 2024-09-16T01:44:30.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.096+0000 7f6f65ffb640 1 -- 172.21.15.88:0/837317677 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v311) ==== 161+0+0 (secure 0 0 0) 0x7f6f6c059610 con 0x7f6f70072530 2024-09-16T01:44:30.098 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:44:30.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.099+0000 7f6f7775b640 1 -- 172.21.15.88:0/837317677 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6f5406c2b0 msgr2=0x7f6f5406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:30.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.099+0000 7f6f7775b640 1 --2- 172.21.15.88:0/837317677 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6f5406c2b0 0x7f6f5406e770 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f6f60034000 tx=0x7f6f6002e000 comp rx=0 tx=0).stop 2024-09-16T01:44:30.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.099+0000 7f6f7775b640 1 -- 172.21.15.88:0/837317677 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f70072530 msgr2=0x7f6f701442f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:30.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.099+0000 7f6f7775b640 1 --2- 172.21.15.88:0/837317677 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f70072530 0x7f6f701442f0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00c7f0 tx=0x7f6f6c00ccb0 comp rx=0 tx=0).stop 2024-09-16T01:44:30.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.099+0000 7f6f7775b640 1 -- 172.21.15.88:0/837317677 shutdown_connections 2024-09-16T01:44:30.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.099+0000 7f6f7775b640 1 --2- 172.21.15.88:0/837317677 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6f5406c2b0 0x7f6f5406e770 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:30.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.099+0000 7f6f7775b640 1 --2- 172.21.15.88:0/837317677 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f7015b480 0x7f6f7014d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:30.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.099+0000 7f6f7775b640 1 --2- 172.21.15.88:0/837317677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f700b60c0 0x7f6f700c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:30.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.099+0000 7f6f7775b640 1 --2- 172.21.15.88:0/837317677 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f70072530 0x7f6f701442f0 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:30.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.099+0000 7f6f7775b640 1 -- 172.21.15.88:0/837317677 >> 172.21.15.88:0/837317677 conn(0x7f6f7006d0f0 msgr2=0x7f6f7015a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:30.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.099+0000 7f6f7775b640 1 -- 172.21.15.88:0/837317677 shutdown_connections 2024-09-16T01:44:30.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.099+0000 7f6f7775b640 1 -- 172.21.15.88:0/837317677 wait complete. 2024-09-16T01:44:30.112 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-09-16T01:44:30.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.211+0000 7fa287b31640 1 Processor -- start 2024-09-16T01:44:30.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.211+0000 7fa287b31640 1 -- start start 2024-09-16T01:44:30.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.212+0000 7fa287b31640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa28015ad10 0x7fa2800719c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:30.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.212+0000 7fa287b31640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa2800cca40 0x7fa280071f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:30.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.212+0000 7fa287b31640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa2800cc360 0x7fa28015e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:30.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.212+0000 7fa287b31640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa280070fb0 con 0x7fa28015ad10 2024-09-16T01:44:30.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.212+0000 7fa287b31640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa280071720 con 0x7fa2800cca40 2024-09-16T01:44:30.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.212+0000 7fa287b31640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa2800b4fb0 con 0x7fa2800cc360 2024-09-16T01:44:30.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.212+0000 7fa2858cd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa28015ad10 0x7fa2800719c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:30.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.212+0000 7fa2858cd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa28015ad10 0x7fa2800719c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41744/0 (socket says 172.21.15.88:41744) 2024-09-16T01:44:30.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.212+0000 7fa2858cd640 1 -- 172.21.15.88:0/1949669818 learned_addr learned my addr 172.21.15.88:0/1949669818 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:30.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.212+0000 7fa2860ce640 1 --2- 172.21.15.88:0/1949669818 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa2800cc360 0x7fa28015e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:30.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.212+0000 7fa2850cc640 1 --2- 172.21.15.88:0/1949669818 >> v2:172.21.15.154:3300/0 conn(0x7fa2800cca40 0x7fa280071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:30.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.212+0000 7fa2858cd640 1 -- 172.21.15.88:0/1949669818 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa2800cc360 msgr2=0x7fa28015e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:30.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.212+0000 7fa2858cd640 1 --2- 172.21.15.88:0/1949669818 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa2800cc360 0x7fa28015e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:30.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.212+0000 7fa2858cd640 1 -- 172.21.15.88:0/1949669818 >> v2:172.21.15.154:3300/0 conn(0x7fa2800cca40 msgr2=0x7fa280071f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:30.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.212+0000 7fa2858cd640 1 --2- 172.21.15.88:0/1949669818 >> v2:172.21.15.154:3300/0 conn(0x7fa2800cca40 0x7fa280071f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:30.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.212+0000 7fa2858cd640 1 -- 172.21.15.88:0/1949669818 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa280073920 con 0x7fa28015ad10 2024-09-16T01:44:30.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.213+0000 7fa2858cd640 1 --2- 172.21.15.88:0/1949669818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa28015ad10 0x7fa2800719c0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7fa280070b50 tx=0x7fa27c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d135015d9908a49c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:30.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.213+0000 7fa2848cb640 1 -- 172.21.15.88:0/1949669818 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27c037070 con 0x7fa28015ad10 2024-09-16T01:44:30.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.213+0000 7fa2848cb640 1 -- 172.21.15.88:0/1949669818 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa27c032440 con 0x7fa28015ad10 2024-09-16T01:44:30.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.213+0000 7fa2848cb640 1 -- 172.21.15.88:0/1949669818 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27c032810 con 0x7fa28015ad10 2024-09-16T01:44:30.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.214+0000 7fa287b31640 1 -- 172.21.15.88:0/1949669818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa28015ad10 msgr2=0x7fa2800719c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:30.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.214+0000 7fa287b31640 1 --2- 172.21.15.88:0/1949669818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa28015ad10 0x7fa2800719c0 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7fa280070b50 tx=0x7fa27c02a180 comp rx=0 tx=0).stop 2024-09-16T01:44:30.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.214+0000 7fa287b31640 1 -- 172.21.15.88:0/1949669818 shutdown_connections 2024-09-16T01:44:30.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.214+0000 7fa287b31640 1 --2- 172.21.15.88:0/1949669818 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa2800cc360 0x7fa28015e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:30.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.214+0000 7fa287b31640 1 --2- 172.21.15.88:0/1949669818 >> v2:172.21.15.154:3300/0 conn(0x7fa2800cca40 0x7fa280071f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:30.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.214+0000 7fa287b31640 1 --2- 172.21.15.88:0/1949669818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa28015ad10 0x7fa2800719c0 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:30.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.214+0000 7fa287b31640 1 -- 172.21.15.88:0/1949669818 >> 172.21.15.88:0/1949669818 conn(0x7fa28006d0f0 msgr2=0x7fa28006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:30.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.214+0000 7fa287b31640 1 -- 172.21.15.88:0/1949669818 shutdown_connections 2024-09-16T01:44:30.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.214+0000 7fa287b31640 1 -- 172.21.15.88:0/1949669818 wait complete. 2024-09-16T01:44:30.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.215+0000 7fa287b31640 1 Processor -- start 2024-09-16T01:44:30.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.215+0000 7fa287b31640 1 -- start start 2024-09-16T01:44:30.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.216+0000 7fa287b31640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2800cc360 0x7fa280150090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:30.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.216+0000 7fa287b31640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2800cca40 0x7fa2801570b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:30.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.216+0000 7fa287b31640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa28015ad10 0x7fa280158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:30.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.216+0000 7fa287b31640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa280071200 con 0x7fa2800cc360 2024-09-16T01:44:30.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.216+0000 7fa287b31640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa280073ee0 con 0x7fa2800cca40 2024-09-16T01:44:30.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.216+0000 7fa287b31640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa28015ec50 con 0x7fa28015ad10 2024-09-16T01:44:30.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.216+0000 7fa2858cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2800cc360 0x7fa280150090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:30.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.216+0000 7fa2858cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2800cc360 0x7fa280150090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41758/0 (socket says 172.21.15.88:41758) 2024-09-16T01:44:30.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.216+0000 7fa2858cd640 1 -- 172.21.15.88:0/3730191621 learned_addr learned my addr 172.21.15.88:0/3730191621 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:30.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.216+0000 7fa2850cc640 1 --2- 172.21.15.88:0/3730191621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2800cca40 0x7fa2801570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:30.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.216+0000 7fa2860ce640 1 --2- 172.21.15.88:0/3730191621 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa28015ad10 0x7fa280158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:30.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.216+0000 7fa2858cd640 1 -- 172.21.15.88:0/3730191621 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa28015ad10 msgr2=0x7fa280158300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:30.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.216+0000 7fa2858cd640 1 --2- 172.21.15.88:0/3730191621 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa28015ad10 0x7fa280158300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:30.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.216+0000 7fa2858cd640 1 -- 172.21.15.88:0/3730191621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2800cca40 msgr2=0x7fa2801570b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:30.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.216+0000 7fa2858cd640 1 --2- 172.21.15.88:0/3730191621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2800cca40 0x7fa2801570b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:30.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.216+0000 7fa2858cd640 1 -- 172.21.15.88:0/3730191621 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2801505d0 con 0x7fa2800cc360 2024-09-16T01:44:30.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.216+0000 7fa2858cd640 1 --2- 172.21.15.88:0/3730191621 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2800cc360 0x7fa280150090 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7fa27c02a1c0 tx=0x7fa27c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:30.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.217+0000 7fa2767fc640 1 -- 172.21.15.88:0/3730191621 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27c041020 con 0x7fa2800cc360 2024-09-16T01:44:30.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.217+0000 7fa2767fc640 1 -- 172.21.15.88:0/3730191621 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa27c0306e0 con 0x7fa2800cc360 2024-09-16T01:44:30.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.217+0000 7fa287b31640 1 -- 172.21.15.88:0/3730191621 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa280157b50 con 0x7fa2800cc360 2024-09-16T01:44:30.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.217+0000 7fa2767fc640 1 -- 172.21.15.88:0/3730191621 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa27c037040 con 0x7fa2800cc360 2024-09-16T01:44:30.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.217+0000 7fa287b31640 1 -- 172.21.15.88:0/3730191621 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa28015c780 con 0x7fa2800cc360 2024-09-16T01:44:30.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.218+0000 7fa287b31640 1 -- 172.21.15.88:0/3730191621 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa28015e710 con 0x7fa2800cc360 2024-09-16T01:44:30.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.218+0000 7fa2767fc640 1 -- 172.21.15.88:0/3730191621 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa27c044020 con 0x7fa2800cc360 2024-09-16T01:44:30.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.218+0000 7fa2767fc640 1 --2- 172.21.15.88:0/3730191621 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa26406c2b0 0x7fa26406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:30.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.218+0000 7fa2767fc640 1 -- 172.21.15.88:0/3730191621 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 9092+0+0 (secure 0 0 0) 0x7fa27c0ac790 con 0x7fa2800cc360 2024-09-16T01:44:30.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.218+0000 7fa2850cc640 1 --2- 172.21.15.88:0/3730191621 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa26406c2b0 0x7fa26406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:30.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.219+0000 7fa2850cc640 1 --2- 172.21.15.88:0/3730191621 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa26406c2b0 0x7fa26406e770 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fa2801f7a30 tx=0x7fa270008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:30.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.223+0000 7fa2767fc640 1 -- 172.21.15.88:0/3730191621 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa27c04b030 con 0x7fa2800cc360 2024-09-16T01:44:30.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:30.453+0000 7fa287b31640 1 -- 172.21.15.88:0/3730191621 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) -- 0x7fa28006da00 con 0x7fa2800cc360 2024-09-16T01:44:31.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.104+0000 7fa2767fc640 1 -- 172.21.15.88:0/3730191621 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v312) ==== 101+0+0 (secure 0 0 0) 0x7fa27c079bb0 con 0x7fa2800cc360 2024-09-16T01:44:31.105 INFO:teuthology.orchestra.run.smithi088.stderr:repaired: restoring rank 6:0 2024-09-16T01:44:31.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.106+0000 7fa287b31640 1 -- 172.21.15.88:0/3730191621 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa26406c2b0 msgr2=0x7fa26406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:31.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.106+0000 7fa287b31640 1 --2- 172.21.15.88:0/3730191621 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa26406c2b0 0x7fa26406e770 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fa2801f7a30 tx=0x7fa270008040 comp rx=0 tx=0).stop 2024-09-16T01:44:31.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.106+0000 7fa287b31640 1 -- 172.21.15.88:0/3730191621 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2800cc360 msgr2=0x7fa280150090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:31.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.106+0000 7fa287b31640 1 --2- 172.21.15.88:0/3730191621 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2800cc360 0x7fa280150090 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7fa27c02a1c0 tx=0x7fa27c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.106+0000 7fa287b31640 1 -- 172.21.15.88:0/3730191621 shutdown_connections 2024-09-16T01:44:31.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.106+0000 7fa287b31640 1 --2- 172.21.15.88:0/3730191621 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa26406c2b0 0x7fa26406e770 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.106+0000 7fa287b31640 1 --2- 172.21.15.88:0/3730191621 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa28015ad10 0x7fa280158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.106+0000 7fa287b31640 1 --2- 172.21.15.88:0/3730191621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2800cca40 0x7fa2801570b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.106+0000 7fa287b31640 1 --2- 172.21.15.88:0/3730191621 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2800cc360 0x7fa280150090 unknown :-1 s=CLOSED pgs=2257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.106+0000 7fa287b31640 1 -- 172.21.15.88:0/3730191621 >> 172.21.15.88:0/3730191621 conn(0x7fa28006d0f0 msgr2=0x7fa2800cecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:31.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.106+0000 7fa287b31640 1 -- 172.21.15.88:0/3730191621 shutdown_connections 2024-09-16T01:44:31.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.106+0000 7fa287b31640 1 -- 172.21.15.88:0/3730191621 wait complete. 2024-09-16T01:44:31.119 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-09-16T01:44:31.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.160+0000 7f9d6ee0ef40 1 Processor -- start 2024-09-16T01:44:31.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.160+0000 7f9d6ee0ef40 1 -- start start 2024-09-16T01:44:31.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.161+0000 7f9d6ee0ef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a5433cee0 0x556a5436c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.161+0000 7f9d6ee0ef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556a54410c90 0x556a5436b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.161+0000 7f9d6ee0ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556a54411070 0x556a5440f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.161+0000 7f9d6ee0ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x556a5428e510 con 0x556a54411070 2024-09-16T01:44:31.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.161+0000 7f9d6ee0ef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556a544116b0 con 0x556a5433cee0 2024-09-16T01:44:31.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.161+0000 7f9d6ee0ef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x556a5433efd0 con 0x556a54410c90 2024-09-16T01:44:31.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.161+0000 7f9d6c4b6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556a54411070 0x556a5440f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:31.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.161+0000 7f9d6c4b6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556a54411070 0x556a5440f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46268/0 (socket says 172.21.15.88:46268) 2024-09-16T01:44:31.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.161+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/3692125745 learned_addr learned my addr 172.21.15.88:0/3692125745 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:31.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.161+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/3692125745 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556a54410c90 msgr2=0x556a5436b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:31.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.161+0000 7f9d6c4b6640 1 --2- 172.21.15.88:0/3692125745 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556a54410c90 0x556a5436b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.161+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/3692125745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a5433cee0 msgr2=0x556a5436c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:31.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.162+0000 7f9d6c4b6640 1 --2- 172.21.15.88:0/3692125745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a5433cee0 0x556a5436c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.162+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/3692125745 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a5440e380 con 0x556a54411070 2024-09-16T01:44:31.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.162+0000 7f9d6c4b6640 1 --2- 172.21.15.88:0/3692125745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556a54411070 0x556a5440f220 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f9d640015c0 tx=0x7f9d64001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5f559c505df7ff6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:31.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.162+0000 7f9d6acb3640 1 -- 172.21.15.88:0/3692125745 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d64003650 con 0x556a54411070 2024-09-16T01:44:31.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.162+0000 7f9d6acb3640 1 -- 172.21.15.88:0/3692125745 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d640037f0 con 0x556a54411070 2024-09-16T01:44:31.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.162+0000 7f9d6acb3640 1 -- 172.21.15.88:0/3692125745 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d64003c60 con 0x556a54411070 2024-09-16T01:44:31.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.162+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/3692125745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556a54411070 msgr2=0x556a5440f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:31.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.162+0000 7f9d6ee0ef40 1 --2- 172.21.15.88:0/3692125745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556a54411070 0x556a5440f220 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f9d640015c0 tx=0x7f9d64001a80 comp rx=0 tx=0).stop 2024-09-16T01:44:31.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.162+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/3692125745 shutdown_connections 2024-09-16T01:44:31.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.162+0000 7f9d6ee0ef40 1 --2- 172.21.15.88:0/3692125745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556a54411070 0x556a5440f220 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.162+0000 7f9d6ee0ef40 1 --2- 172.21.15.88:0/3692125745 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556a54410c90 0x556a5436b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.162+0000 7f9d6ee0ef40 1 --2- 172.21.15.88:0/3692125745 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a5433cee0 0x556a5436c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.162+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/3692125745 >> 172.21.15.88:0/3692125745 conn(0x556a54367680 msgr2=0x556a5440c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:31.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.162+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/3692125745 shutdown_connections 2024-09-16T01:44:31.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.162+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/3692125745 wait complete. 2024-09-16T01:44:31.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.163+0000 7f9d6ee0ef40 1 Processor -- start 2024-09-16T01:44:31.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.163+0000 7f9d6ee0ef40 1 -- start start 2024-09-16T01:44:31.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.164+0000 7f9d6ee0ef40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556a5433cee0 0x556a5440f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.164+0000 7f9d6ee0ef40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556a54410c90 0x556a544097b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.164+0000 7f9d6ee0ef40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a54411070 0x556a54406280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.164+0000 7f9d6ee0ef40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x556a5436c5b0 con 0x556a54410c90 2024-09-16T01:44:31.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.164+0000 7f9d6ee0ef40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x556a5440e660 con 0x556a54411070 2024-09-16T01:44:31.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.164+0000 7f9d6ee0ef40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x556a5440eb80 con 0x556a5433cee0 2024-09-16T01:44:31.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.164+0000 7f9d634b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556a54410c90 0x556a544097b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:31.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.164+0000 7f9d634b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556a54410c90 0x556a544097b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46282/0 (socket says 172.21.15.88:46282) 2024-09-16T01:44:31.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.164+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 learned_addr learned my addr 172.21.15.88:0/2900152179 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:31.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.164+0000 7f9d6bcb5640 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556a5433cee0 0x556a5440f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:31.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.164+0000 7f9d6c4b6640 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a54411070 0x556a54406280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:31.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.164+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556a5433cee0 msgr2=0x556a5440f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:31.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.164+0000 7f9d634b4640 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556a5433cee0 0x556a5440f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.164+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a54411070 msgr2=0x556a54406280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:31.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.164+0000 7f9d634b4640 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a54411070 0x556a54406280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.164+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a54407d40 con 0x556a54410c90 2024-09-16T01:44:31.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.164+0000 7f9d6bcb5640 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x556a5433cee0 0x556a5440f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:31.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.164+0000 7f9d6c4b6640 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556a54411070 0x556a54406280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:31.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.164+0000 7f9d634b4640 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556a54410c90 0x556a544097b0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f9d54000c00 tx=0x7f9d54052550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:31.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.164+0000 7f9d69cb1640 1 -- 172.21.15.88:0/2900152179 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d54056030 con 0x556a54410c90 2024-09-16T01:44:31.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.164+0000 7f9d69cb1640 1 -- 172.21.15.88:0/2900152179 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d54052b50 con 0x556a54410c90 2024-09-16T01:44:31.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.164+0000 7f9d69cb1640 1 -- 172.21.15.88:0/2900152179 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5405b450 con 0x556a54410c90 2024-09-16T01:44:31.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.164+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556a54408020 con 0x556a54410c90 2024-09-16T01:44:31.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.165+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a544085e0 con 0x556a54410c90 2024-09-16T01:44:31.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.166+0000 7f9d69cb1640 1 -- 172.21.15.88:0/2900152179 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9d54059020 con 0x556a54410c90 2024-09-16T01:44:31.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.166+0000 7f9d69cb1640 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9d4c06ac90 0x7f9d4c06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.166+0000 7f9d69cb1640 1 -- 172.21.15.88:0/2900152179 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 9092+0+0 (secure 0 0 0) 0x7f9d540cf270 con 0x556a54410c90 2024-09-16T01:44:31.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.166+0000 7f9d6bcb5640 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9d4c06ac90 0x7f9d4c06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:31.173 INFO:teuthology.orchestra.run.smithi088.stdout:Importing pool 2024-09-16T01:44:31.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.166+0000 7f9d6ee0ef40 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x556a54409cf0 0x556a54406cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.166+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:1 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a54402e10 con 0x556a54409cf0 2024-09-16T01:44:31.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.167+0000 7f9d6c4b6640 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x556a54409cf0 0x556a54406cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:31.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.167+0000 7f9d6bcb5640 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9d4c06ac90 0x7f9d4c06d150 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c005400 tx=0x7f9d5c005880 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:31.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.167+0000 7f9d6c4b6640 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x556a54409cf0 0x556a54406cd0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:31.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.170+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(1 601.00000000 [create] v126'1190 uv1164 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.170+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:2 12.10 12:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e126) -- 0x556a544031e0 con 0x556a54409cf0 2024-09-16T01:44:31.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.171+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(2 601.00000000 [delete] v126'1191 uv1191 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.171+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:3 12.10 12:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a54403640 con 0x556a54409cf0 2024-09-16T01:44:31.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.172+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(3 601.00000000 [create] v126'1192 uv1192 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.175 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-09-16T01:44:31.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.172+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:4 12.10 12:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b6320 con 0x556a54409cf0 2024-09-16T01:44:31.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.174+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v126'1193 uv1193 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.174+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:5 12.10 12:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b66c0 con 0x556a54409cf0 2024-09-16T01:44:31.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.175+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v126'1194 uv1194 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.175+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:6 12.10 12:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b5310 con 0x556a54409cf0 2024-09-16T01:44:31.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.176+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v126'1195 uv1195 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.176+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b56b0 con 0x556a54409cf0 2024-09-16T01:44:31.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.178+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 7 ==== osd_op_reply(7 602.00000000 [create] v126'1196 uv1170 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.178+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 12.10 12:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e126) -- 0x556a544b56b0 con 0x556a54409cf0 2024-09-16T01:44:31.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.179+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 8 ==== osd_op_reply(8 602.00000000 [delete] v126'1197 uv1197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.179+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 12.10 12:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b56b0 con 0x556a54409cf0 2024-09-16T01:44:31.182 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-09-16T01:44:31.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.181+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 9 ==== osd_op_reply(9 602.00000000 [create] v126'1198 uv1198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.181+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:10 12.10 12:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b56b0 con 0x556a54409cf0 2024-09-16T01:44:31.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.182+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v126'1199 uv1199 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.182+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 12.10 12:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b56b0 con 0x556a54409cf0 2024-09-16T01:44:31.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.183+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v126'1200 uv1200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.183+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:12 12.10 12:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b56b0 con 0x556a54409cf0 2024-09-16T01:44:31.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.185+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v126'1201 uv1201 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.185+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:13 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b56b0 con 0x556a54409cf0 2024-09-16T01:44:31.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.186+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 13 ==== osd_op_reply(13 10000000000.00000000 [create] v126'1202 uv1177 ondisk = -17 ((17) File exists)) ==== 164+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.186+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:14 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e126) -- 0x556a544b56b0 con 0x556a54409cf0 2024-09-16T01:44:31.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.188+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v126'1203 uv1203 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.188+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:15 12.10 12:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b56b0 con 0x556a54409cf0 2024-09-16T01:44:31.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.189+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 15 ==== osd_op_reply(15 10000000000.00000000 [create] v126'1204 uv1204 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.190 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-09-16T01:44:31.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.189+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:16 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b56b0 con 0x556a54409cf0 2024-09-16T01:44:31.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.190+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v126'1205 uv1205 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.190+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:17 12.10 12:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b56b0 con 0x556a54409cf0 2024-09-16T01:44:31.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.192+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v126'1206 uv1206 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.192+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:18 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b56b0 con 0x556a54409cf0 2024-09-16T01:44:31.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.193+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v126'1207 uv1207 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.193+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:19 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b56b0 con 0x556a54409cf0 2024-09-16T01:44:31.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.195+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v126'1208 uv1208 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.195+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:20 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b56b0 con 0x556a54409cf0 2024-09-16T01:44:31.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.196+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 20 ==== osd_op_reply(20 600.00000000 [create] v126'1209 uv1183 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.196+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:21 12.10 12:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e126) -- 0x556a544b56b0 con 0x556a54409cf0 2024-09-16T01:44:31.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.198+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 21 ==== osd_op_reply(21 600.00000000 [delete] v126'1210 uv1210 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.198+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:22 12.10 12:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b56b0 con 0x556a54409cf0 2024-09-16T01:44:31.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.199+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 22 ==== osd_op_reply(22 600.00000000 [create] v126'1211 uv1211 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.200 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-09-16T01:44:31.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.199+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:23 12.10 12:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b56b0 con 0x556a54409cf0 2024-09-16T01:44:31.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.200+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v126'1212 uv1212 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.200+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:24 12.10 12:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b56b0 con 0x556a54409cf0 2024-09-16T01:44:31.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.201+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v126'1213 uv1213 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.201+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:25 12.10 12:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b56b0 con 0x556a54409cf0 2024-09-16T01:44:31.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.203+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v126'1214 uv1214 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.203+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:26 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b56b0 con 0x556a54409cf0 2024-09-16T01:44:31.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.204+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 26 ==== osd_op_reply(26 603.00000000 [create] v126'1215 uv1189 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.204+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:27 12.10 12:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e126) -- 0x556a544b56b0 con 0x556a54409cf0 2024-09-16T01:44:31.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.205+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 27 ==== osd_op_reply(27 603.00000000 [delete] v126'1216 uv1216 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.206+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:28 12.10 12:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b56b0 con 0x556a54409cf0 2024-09-16T01:44:31.208 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-09-16T01:44:31.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.207+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 28 ==== osd_op_reply(28 603.00000000 [create] v126'1217 uv1217 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.207+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:29 12.10 12:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b56b0 con 0x556a54409cf0 2024-09-16T01:44:31.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.208+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v126'1218 uv1218 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.208+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:30 12.10 12:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b56b0 con 0x556a54409cf0 2024-09-16T01:44:31.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.209+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v126'1219 uv1219 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.209+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:31 12.10 12:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b56b0 con 0x556a54409cf0 2024-09-16T01:44:31.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.210+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v126'1220 uv1220 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.210+0000 7f9d6ee0ef40 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x556a544b6a60 0x556a544b8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.210+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:32 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b5f80 con 0x556a544b6a60 2024-09-16T01:44:31.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.211+0000 7f9d6bcb5640 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x556a544b6a60 0x556a544b8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:31.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.211+0000 7f9d6bcb5640 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x556a544b6a60 0x556a544b8e50 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:31.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.213+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== osd_op_reply(32 1.00000000.inode [create] v126'868 uv851 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544b6a60 2024-09-16T01:44:31.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.213+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:33 12.14 12:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e126) -- 0x556a544b9390 con 0x556a544b6a60 2024-09-16T01:44:31.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.215+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== osd_op_reply(33 1.00000000.inode [delete] v126'869 uv869 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544b6a60 2024-09-16T01:44:31.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.215+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:34 12.14 12:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b9730 con 0x556a544b6a60 2024-09-16T01:44:31.217 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-09-16T01:44:31.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.216+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.5 v2:172.21.15.154:6808/280070533 3 ==== osd_op_reply(34 1.00000000.inode [create] v126'870 uv870 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544b6a60 2024-09-16T01:44:31.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.216+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:35 12.14 12:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b9f80 con 0x556a544b6a60 2024-09-16T01:44:31.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.217+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.5 v2:172.21.15.154:6808/280070533 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v126'871 uv871 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544b6a60 2024-09-16T01:44:31.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.217+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:36 12.14 12:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544ba320 con 0x556a544b6a60 2024-09-16T01:44:31.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.218+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.5 v2:172.21.15.154:6808/280070533 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v126'872 uv872 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544b6a60 2024-09-16T01:44:31.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.218+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:37 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544ba6c0 con 0x556a544b6a60 2024-09-16T01:44:31.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.220+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.5 v2:172.21.15.154:6808/280070533 6 ==== osd_op_reply(37 200.00000000 [create] v126'873 uv856 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544b6a60 2024-09-16T01:44:31.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.220+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:38 12.14 12:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e126) -- 0x556a544baa60 con 0x556a544b6a60 2024-09-16T01:44:31.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.221+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.5 v2:172.21.15.154:6808/280070533 7 ==== osd_op_reply(38 200.00000000 [delete] v126'874 uv874 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544b6a60 2024-09-16T01:44:31.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.221+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:39 12.14 12:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bae00 con 0x556a544b6a60 2024-09-16T01:44:31.223 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.222+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.5 v2:172.21.15.154:6808/280070533 8 ==== osd_op_reply(39 200.00000000 [create] v126'875 uv875 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544b6a60 2024-09-16T01:44:31.223 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-09-16T01:44:31.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.222+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:40 12.14 12:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bae00 con 0x556a544b6a60 2024-09-16T01:44:31.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.224+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.5 v2:172.21.15.154:6808/280070533 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v126'876 uv876 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544b6a60 2024-09-16T01:44:31.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.224+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:41 12.14 12:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bae00 con 0x556a544b6a60 2024-09-16T01:44:31.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.225+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.5 v2:172.21.15.154:6808/280070533 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v126'877 uv877 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544b6a60 2024-09-16T01:44:31.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.225+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:42 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bae00 con 0x556a544b6a60 2024-09-16T01:44:31.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.226+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.5 v2:172.21.15.154:6808/280070533 11 ==== osd_op_reply(42 200.00000001 [create] v126'878 uv861 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544b6a60 2024-09-16T01:44:31.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.226+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:43 12.14 12:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e126) -- 0x556a544bae00 con 0x556a544b6a60 2024-09-16T01:44:31.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.228+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.5 v2:172.21.15.154:6808/280070533 12 ==== osd_op_reply(43 200.00000001 [delete] v126'879 uv879 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544b6a60 2024-09-16T01:44:31.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.228+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:44 12.14 12:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bae00 con 0x556a544b6a60 2024-09-16T01:44:31.230 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-09-16T01:44:31.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.229+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.5 v2:172.21.15.154:6808/280070533 13 ==== osd_op_reply(44 200.00000001 [create] v126'880 uv880 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544b6a60 2024-09-16T01:44:31.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.229+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:45 12.14 12:2e2fa760:::200.00000001:head [write 0~17490 in=17490b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bf280 con 0x556a544b6a60 2024-09-16T01:44:31.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.230+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.5 v2:172.21.15.154:6808/280070533 14 ==== osd_op_reply(45 200.00000001 [write 0~17490] v126'881 uv881 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544b6a60 2024-09-16T01:44:31.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.230+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:46 12.14 12:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bf280 con 0x556a544b6a60 2024-09-16T01:44:31.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.231+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.5 v2:172.21.15.154:6808/280070533 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v126'882 uv882 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544b6a60 2024-09-16T01:44:31.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.231+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:47 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bae00 con 0x556a544b6a60 2024-09-16T01:44:31.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.233+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.5 v2:172.21.15.154:6808/280070533 16 ==== osd_op_reply(47 606.00000000 [create] v126'883 uv867 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544b6a60 2024-09-16T01:44:31.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.233+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:48 12.14 12:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e126) -- 0x556a544bae00 con 0x556a544b6a60 2024-09-16T01:44:31.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.234+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.5 v2:172.21.15.154:6808/280070533 17 ==== osd_op_reply(48 606.00000000 [delete] v126'884 uv884 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544b6a60 2024-09-16T01:44:31.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.234+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:49 12.14 12:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bae00 con 0x556a544b6a60 2024-09-16T01:44:31.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.235+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.5 v2:172.21.15.154:6808/280070533 18 ==== osd_op_reply(49 606.00000000 [create] v126'885 uv885 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544b6a60 2024-09-16T01:44:31.236 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-09-16T01:44:31.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.235+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:50 12.14 12:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bbe10 con 0x556a544b6a60 2024-09-16T01:44:31.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.236+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.5 v2:172.21.15.154:6808/280070533 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v126'886 uv886 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544b6a60 2024-09-16T01:44:31.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.236+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:51 12.14 12:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bbe10 con 0x556a544b6a60 2024-09-16T01:44:31.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.237+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.5 v2:172.21.15.154:6808/280070533 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v126'887 uv887 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544b6a60 2024-09-16T01:44:31.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.237+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- osd_op(unknown.0.0:52 12.14 12:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bae00 con 0x556a544b6a60 2024-09-16T01:44:31.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.238+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.5 v2:172.21.15.154:6808/280070533 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v126'888 uv888 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544b6a60 2024-09-16T01:44:31.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.239+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:53 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bae00 con 0x556a54409cf0 2024-09-16T01:44:31.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.240+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 32 ==== osd_op_reply(53 607.00000000 [create] v126'231 uv230 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.240+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:54 12.1c 12:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e126) -- 0x556a544bae00 con 0x556a54409cf0 2024-09-16T01:44:31.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.241+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 33 ==== osd_op_reply(54 607.00000000 [delete] v126'232 uv232 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.241+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:55 12.1c 12:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bae00 con 0x556a54409cf0 2024-09-16T01:44:31.243 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-09-16T01:44:31.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.242+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 34 ==== osd_op_reply(55 607.00000000 [create] v126'233 uv233 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.242+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:56 12.1c 12:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bbe10 con 0x556a54409cf0 2024-09-16T01:44:31.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.243+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 35 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v126'234 uv234 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.243+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:57 12.1c 12:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bbe10 con 0x556a54409cf0 2024-09-16T01:44:31.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.244+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 36 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v126'235 uv235 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.244+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:58 12.1c 12:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bae00 con 0x556a54409cf0 2024-09-16T01:44:31.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.245+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 37 ==== osd_op_reply(58 607.00000000 [omap-set-header] v126'236 uv236 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.245+0000 7f9d6ee0ef40 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x556a544bb310 0x556a544bd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.245+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:59 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bdd10 con 0x556a544bb310 2024-09-16T01:44:31.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.245+0000 7f9d634b4640 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x556a544bb310 0x556a544bd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:31.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.245+0000 7f9d634b4640 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x556a544bb310 0x556a544bd700 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:31.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.246+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v126'176 uv175 ondisk = -17 ((17) File exists)) ==== 160+0+0 (crc 0 0 0) 0x7f9d5406c020 con 0x556a544bb310 2024-09-16T01:44:31.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.246+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:60 12.6 12:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e126) -- 0x556a544be220 con 0x556a544bb310 2024-09-16T01:44:31.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.247+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v126'177 uv177 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9d5406c020 con 0x556a544bb310 2024-09-16T01:44:31.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.247+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:61 12.6 12:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544be720 con 0x556a544bb310 2024-09-16T01:44:31.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.248+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v126'178 uv178 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9d5406c020 con 0x556a544bb310 2024-09-16T01:44:31.249 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-09-16T01:44:31.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.248+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:62 12.6 12:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544b9bb0 con 0x556a544bb310 2024-09-16T01:44:31.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.249+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 <== osd.1 v2:172.21.15.88:6824/3621315521 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v126'179 uv179 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7f9d5406c020 con 0x556a544bb310 2024-09-16T01:44:31.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.249+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:63 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bf050 con 0x556a54409cf0 2024-09-16T01:44:31.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.251+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 38 ==== osd_op_reply(63 608.00000000 [create] v126'231 uv230 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.251+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:64 12.e 12:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e126) -- 0x556a544bf050 con 0x556a54409cf0 2024-09-16T01:44:31.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.252+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 39 ==== osd_op_reply(64 608.00000000 [delete] v126'232 uv232 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.252+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:65 12.e 12:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bf050 con 0x556a54409cf0 2024-09-16T01:44:31.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.253+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 40 ==== osd_op_reply(65 608.00000000 [create] v126'233 uv233 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.255 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-09-16T01:44:31.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.253+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:66 12.e 12:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bf050 con 0x556a54409cf0 2024-09-16T01:44:31.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.254+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 41 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v126'234 uv234 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.254+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:67 12.e 12:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bf050 con 0x556a54409cf0 2024-09-16T01:44:31.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.255+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 42 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v126'235 uv235 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.255+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:68 12.e 12:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bf050 con 0x556a54409cf0 2024-09-16T01:44:31.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.257+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 43 ==== osd_op_reply(68 608.00000000 [omap-set-header] v126'236 uv236 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.257+0000 7f9d6ee0ef40 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x556a544bf970 0x556a544bfd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.257+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:69 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c24a0 con 0x556a544bf970 2024-09-16T01:44:31.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.257+0000 7f9d6c4b6640 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x556a544bf970 0x556a544bfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:31.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.257+0000 7f9d6c4b6640 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x556a544bf970 0x556a544bfd50 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:31.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.259+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(69 604.00000000 [create] v126'426 uv420 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.259+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:70 12.15 12:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e126) -- 0x556a544c24a0 con 0x556a544bf970 2024-09-16T01:44:31.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.260+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== osd_op_reply(70 604.00000000 [delete] v126'427 uv427 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.260+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:71 12.15 12:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c24a0 con 0x556a544bf970 2024-09-16T01:44:31.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.261+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 3 ==== osd_op_reply(71 604.00000000 [create] v126'428 uv428 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.262 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-09-16T01:44:31.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.261+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:72 12.15 12:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c34f0 con 0x556a544bf970 2024-09-16T01:44:31.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.262+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 4 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v126'429 uv429 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.262+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:73 12.15 12:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c34b0 con 0x556a544bf970 2024-09-16T01:44:31.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.263+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 5 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v126'430 uv430 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.263+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:74 12.15 12:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c24a0 con 0x556a544bf970 2024-09-16T01:44:31.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.265+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 6 ==== osd_op_reply(74 604.00000000 [omap-set-header] v126'431 uv431 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.265+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:75 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c24a0 con 0x556a544bf970 2024-09-16T01:44:31.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.266+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 7 ==== osd_op_reply(75 500.00000000 [create] v126'432 uv425 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.266+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:76 12.15 12:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e126) -- 0x556a544c24a0 con 0x556a544bf970 2024-09-16T01:44:31.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.267+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 8 ==== osd_op_reply(76 500.00000000 [delete] v126'433 uv433 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.267+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:77 12.15 12:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c24a0 con 0x556a544bf970 2024-09-16T01:44:31.269 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-09-16T01:44:31.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.268+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 9 ==== osd_op_reply(77 500.00000000 [create] v126'434 uv434 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.268+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:78 12.15 12:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c24a0 con 0x556a544bf970 2024-09-16T01:44:31.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.269+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 10 ==== osd_op_reply(78 500.00000000 [write 0~90] v126'435 uv435 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.269+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:79 12.15 12:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c24a0 con 0x556a544bf970 2024-09-16T01:44:31.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.270+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 11 ==== osd_op_reply(79 500.00000000 [omap-set-header] v126'436 uv436 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.270+0000 7f9d6ee0ef40 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x556a544c2e40 0x556a544c5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.270+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:80 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c57b0 con 0x556a544c2e40 2024-09-16T01:44:31.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.270+0000 7f9d6bcb5640 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x556a544c2e40 0x556a544c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:31.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.271+0000 7f9d6bcb5640 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x556a544c2e40 0x556a544c5230 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:31.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.272+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(80 mds_snaptable [create] v126'193 uv192 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544c2e40 2024-09-16T01:44:31.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.272+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:81 12.d 12:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e126) -- 0x556a544c5870 con 0x556a544c2e40 2024-09-16T01:44:31.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.273+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(81 mds_snaptable [delete] v126'194 uv194 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544c2e40 2024-09-16T01:44:31.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.273+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:82 12.d 12:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c5c90 con 0x556a544c2e40 2024-09-16T01:44:31.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.274+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.7 v2:172.21.15.154:6824/2200266987 3 ==== osd_op_reply(82 mds_snaptable [create] v126'195 uv195 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544c2e40 2024-09-16T01:44:31.275 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-09-16T01:44:31.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.274+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:83 12.d 12:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c5c90 con 0x556a544c2e40 2024-09-16T01:44:31.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.275+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.7 v2:172.21.15.154:6824/2200266987 4 ==== osd_op_reply(83 mds_snaptable [write 0~70] v126'196 uv196 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544c2e40 2024-09-16T01:44:31.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.275+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:84 12.d 12:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c5c90 con 0x556a544c2e40 2024-09-16T01:44:31.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.276+0000 7f9d6bcb5640 1 -- 172.21.15.88:0/2900152179 <== osd.7 v2:172.21.15.154:6824/2200266987 5 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v126'197 uv197 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9d5c013070 con 0x556a544c2e40 2024-09-16T01:44:31.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.276+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:85 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c5c90 con 0x556a54409cf0 2024-09-16T01:44:31.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.277+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 44 ==== osd_op_reply(85 605.00000000 [create] v126'231 uv230 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.277+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:86 12.1d 12:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e126) -- 0x556a544c5c90 con 0x556a54409cf0 2024-09-16T01:44:31.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.278+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 45 ==== osd_op_reply(86 605.00000000 [delete] v126'232 uv232 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.278+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:87 12.1d 12:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c5c90 con 0x556a54409cf0 2024-09-16T01:44:31.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.278+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 46 ==== osd_op_reply(87 605.00000000 [create] v126'233 uv233 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.280 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-09-16T01:44:31.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.279+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:88 12.1d 12:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c6ca0 con 0x556a54409cf0 2024-09-16T01:44:31.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.279+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 47 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v126'234 uv234 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.279+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c6ca0 con 0x556a54409cf0 2024-09-16T01:44:31.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.280+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 48 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v126'235 uv235 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.280+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:90 12.1d 12:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c5c90 con 0x556a54409cf0 2024-09-16T01:44:31.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.281+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.0 v2:172.21.15.88:6816/903680192 49 ==== osd_op_reply(90 605.00000000 [omap-set-header] v126'236 uv236 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a54409cf0 2024-09-16T01:44:31.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.281+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:91 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c5c90 con 0x556a544bb310 2024-09-16T01:44:31.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.283+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 <== osd.1 v2:172.21.15.88:6824/3621315521 5 ==== osd_op_reply(91 mds0_inotable [create] v126'386 uv380 ondisk = -17 ((17) File exists)) ==== 157+0+0 (crc 0 0 0) 0x7f9d5406c020 con 0x556a544bb310 2024-09-16T01:44:31.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.283+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:92 12.13 12:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e126) -- 0x556a544c6050 con 0x556a544bb310 2024-09-16T01:44:31.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.284+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 <== osd.1 v2:172.21.15.88:6824/3621315521 6 ==== osd_op_reply(92 mds0_inotable [delete] v126'387 uv387 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9d5406c020 con 0x556a544bb310 2024-09-16T01:44:31.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.284+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:93 12.13 12:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c63f0 con 0x556a544bb310 2024-09-16T01:44:31.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.284+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 <== osd.1 v2:172.21.15.88:6824/3621315521 7 ==== osd_op_reply(93 mds0_inotable [create] v126'388 uv388 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9d5406c020 con 0x556a544bb310 2024-09-16T01:44:31.286 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-09-16T01:44:31.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.285+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:94 12.13 12:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bebd0 con 0x556a544bb310 2024-09-16T01:44:31.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.286+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 <== osd.1 v2:172.21.15.88:6824/3621315521 8 ==== osd_op_reply(94 mds0_inotable [write 0~34] v126'389 uv389 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9d5406c020 con 0x556a544bb310 2024-09-16T01:44:31.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.286+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:95 12.13 12:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bebd0 con 0x556a544bb310 2024-09-16T01:44:31.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.287+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 <== osd.1 v2:172.21.15.88:6824/3621315521 9 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v126'390 uv390 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7f9d5406c020 con 0x556a544bb310 2024-09-16T01:44:31.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.287+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:96 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bebd0 con 0x556a544bb310 2024-09-16T01:44:31.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.287+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 <== osd.1 v2:172.21.15.88:6824/3621315521 10 ==== osd_op_reply(96 100.00000000 [create] v126'391 uv385 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d5406c020 con 0x556a544bb310 2024-09-16T01:44:31.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.287+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:97 12.13 12:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e126) -- 0x556a544bebd0 con 0x556a544bb310 2024-09-16T01:44:31.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.288+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 <== osd.1 v2:172.21.15.88:6824/3621315521 11 ==== osd_op_reply(97 100.00000000 [delete] v126'392 uv392 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d5406c020 con 0x556a544bb310 2024-09-16T01:44:31.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.288+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:98 12.13 12:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bebd0 con 0x556a544bb310 2024-09-16T01:44:31.290 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-09-16T01:44:31.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.289+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 <== osd.1 v2:172.21.15.88:6824/3621315521 12 ==== osd_op_reply(98 100.00000000 [create] v126'393 uv393 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d5406c020 con 0x556a544bb310 2024-09-16T01:44:31.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.289+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:99 12.13 12:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544bebd0 con 0x556a544bb310 2024-09-16T01:44:31.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.290+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 <== osd.1 v2:172.21.15.88:6824/3621315521 13 ==== osd_op_reply(99 100.00000000 [omap-set-header] v126'394 uv394 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d5406c020 con 0x556a544bb310 2024-09-16T01:44:31.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.290+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:100 12.13 12:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c9650 con 0x556a544bb310 2024-09-16T01:44:31.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.291+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 <== osd.1 v2:172.21.15.88:6824/3621315521 14 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v126'395 uv395 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d5406c020 con 0x556a544bb310 2024-09-16T01:44:31.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.291+0000 7f9d6ee0ef40 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x556a544c72c0 0x556a544c96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.291+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:101 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c9d10 con 0x556a544c72c0 2024-09-16T01:44:31.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.292+0000 7f9d634b4640 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x556a544c72c0 0x556a544c96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:31.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.292+0000 7f9d634b4640 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x556a544c72c0 0x556a544c96b0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:31.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.293+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== osd_op_reply(101 mds0_sessionmap [create] v126'172 uv171 ondisk = -17 ((17) File exists)) ==== 159+0+0 (crc 0 0 0) 0x7f9d5406c020 con 0x556a544c72c0 2024-09-16T01:44:31.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.293+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:102 12.b 12:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e126) -- 0x556a544c9d10 con 0x556a544c72c0 2024-09-16T01:44:31.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.294+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v126'173 uv173 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f9d5406c020 con 0x556a544c72c0 2024-09-16T01:44:31.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.294+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:103 12.b 12:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c9d10 con 0x556a544c72c0 2024-09-16T01:44:31.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.295+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 <== osd.3 v2:172.21.15.88:6800/697609987 3 ==== osd_op_reply(103 mds0_sessionmap [create] v126'174 uv174 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f9d5406c020 con 0x556a544c72c0 2024-09-16T01:44:31.296 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-09-16T01:44:31.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.295+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- osd_op(unknown.0.0:104 12.b 12:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c9d90 con 0x556a544c72c0 2024-09-16T01:44:31.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.295+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 <== osd.3 v2:172.21.15.88:6800/697609987 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v126'175 uv175 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7f9d5406c020 con 0x556a544c72c0 2024-09-16T01:44:31.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.296+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:105 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c9d90 con 0x556a544bb310 2024-09-16T01:44:31.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.297+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 <== osd.1 v2:172.21.15.88:6824/3621315521 15 ==== osd_op_reply(105 609.00000000 [create] v126'231 uv230 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d5406c020 con 0x556a544bb310 2024-09-16T01:44:31.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.297+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:106 12.7 12:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e126) -- 0x556a544c9d90 con 0x556a544bb310 2024-09-16T01:44:31.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.298+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 <== osd.1 v2:172.21.15.88:6824/3621315521 16 ==== osd_op_reply(106 609.00000000 [delete] v126'232 uv232 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d5406c020 con 0x556a544bb310 2024-09-16T01:44:31.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.298+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:107 12.7 12:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c9d10 con 0x556a544bb310 2024-09-16T01:44:31.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.299+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 <== osd.1 v2:172.21.15.88:6824/3621315521 17 ==== osd_op_reply(107 609.00000000 [create] v126'233 uv233 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d5406c020 con 0x556a544bb310 2024-09-16T01:44:31.300 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-09-16T01:44:31.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.299+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:108 12.7 12:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544cad20 con 0x556a544bb310 2024-09-16T01:44:31.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.300+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 <== osd.1 v2:172.21.15.88:6824/3621315521 18 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v126'234 uv234 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d5406c020 con 0x556a544bb310 2024-09-16T01:44:31.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.300+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:109 12.7 12:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544cad20 con 0x556a544bb310 2024-09-16T01:44:31.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.301+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 <== osd.1 v2:172.21.15.88:6824/3621315521 19 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v126'235 uv235 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d5406c020 con 0x556a544bb310 2024-09-16T01:44:31.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.301+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:110 12.7 12:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c9d10 con 0x556a544bb310 2024-09-16T01:44:31.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.302+0000 7f9d634b4640 1 -- 172.21.15.88:0/2900152179 <== osd.1 v2:172.21.15.88:6824/3621315521 20 ==== osd_op_reply(110 609.00000000 [omap-set-header] v126'236 uv236 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d5406c020 con 0x556a544bb310 2024-09-16T01:44:31.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.302+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:111 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c9d10 con 0x556a544bf970 2024-09-16T01:44:31.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.303+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 12 ==== osd_op_reply(111 400.00000000 [create] v126'196 uv195 ondisk = -17 ((17) File exists)) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.303+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:112 12.f 12:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e126) -- 0x556a544c9d10 con 0x556a544bf970 2024-09-16T01:44:31.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.305+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 13 ==== osd_op_reply(112 400.00000000 [delete] v126'197 uv197 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.305+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:113 12.f 12:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c9d10 con 0x556a544bf970 2024-09-16T01:44:31.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.306+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 14 ==== osd_op_reply(113 400.00000000 [create] v126'198 uv198 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.307 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-09-16T01:44:31.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.306+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:114 12.f 12:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c9d10 con 0x556a544bf970 2024-09-16T01:44:31.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.307+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 15 ==== osd_op_reply(114 400.00000000 [write 0~22] v126'199 uv199 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.307+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:115 12.f 12:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c9d10 con 0x556a544bf970 2024-09-16T01:44:31.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.309+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 16 ==== osd_op_reply(115 400.00000000 [omap-set-header] v126'200 uv200 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.309+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:116 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c9d10 con 0x556a544bf970 2024-09-16T01:44:31.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.310+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 17 ==== osd_op_reply(116 100.00000000.inode [create] v126'480 uv472 ondisk = -17 ((17) File exists)) ==== 162+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.310+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:117 12.1f 12:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e126) -- 0x556a544c9d70 con 0x556a544bf970 2024-09-16T01:44:31.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.311+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 18 ==== osd_op_reply(117 100.00000000.inode [delete] v126'481 uv481 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.311+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:118 12.1f 12:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c9d70 con 0x556a544bf970 2024-09-16T01:44:31.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.312+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 19 ==== osd_op_reply(118 100.00000000.inode [create] v126'482 uv482 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.313 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-09-16T01:44:31.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.312+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:119 12.1f 12:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c9d70 con 0x556a544bf970 2024-09-16T01:44:31.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.313+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 20 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v126'483 uv483 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.313+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:120 12.1f 12:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c9d50 con 0x556a544bf970 2024-09-16T01:44:31.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.314+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 21 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v126'484 uv484 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.314+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:121 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c9d50 con 0x556a544bf970 2024-09-16T01:44:31.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.315+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 22 ==== osd_op_reply(121 1.00000000 [create] v126'485 uv479 ondisk = -17 ((17) File exists)) ==== 154+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.315+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:122 12.1f 12:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e126) -- 0x556a544c9d30 con 0x556a544bf970 2024-09-16T01:44:31.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.316+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 23 ==== osd_op_reply(122 1.00000000 [delete] v126'486 uv486 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.316+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:123 12.1f 12:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c9d10 con 0x556a544bf970 2024-09-16T01:44:31.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.317+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 24 ==== osd_op_reply(123 1.00000000 [create] v126'487 uv487 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.319 INFO:teuthology.orchestra.run.smithi088.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-09-16T01:44:31.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.317+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:124 12.1f 12:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c9d50 con 0x556a544bf970 2024-09-16T01:44:31.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.318+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 25 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v126'488 uv488 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.318+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:125 12.1f 12:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c9d10 con 0x556a544bf970 2024-09-16T01:44:31.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.319+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 26 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v126'489 uv489 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.320+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:126 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544c9d10 con 0x556a544bf970 2024-09-16T01:44:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.320+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 27 ==== osd_op_reply(126 1.00000000 [omap-set-header] v126'490 uv490 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.321+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:127 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x556a544ca180 con 0x556a544bf970 2024-09-16T01:44:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 <== osd.6 v2:172.21.15.154:6800/189356541 28 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v126'491 uv491 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7f9d6400f030 con 0x556a544bf970 2024-09-16T01:44:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x556a54409cf0 msgr2=0x556a54406cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6ee0ef40 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x556a54409cf0 0x556a54406cd0 crc :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x556a544bb310 msgr2=0x556a544bd700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6ee0ef40 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x556a544bb310 0x556a544bd700 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x556a544c72c0 msgr2=0x556a544c96b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6ee0ef40 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x556a544c72c0 0x556a544c96b0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x556a544b6a60 msgr2=0x556a544b8e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6ee0ef40 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x556a544b6a60 0x556a544b8e50 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x556a544bf970 msgr2=0x556a544bfd50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6ee0ef40 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x556a544bf970 0x556a544bfd50 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x556a544c2e40 msgr2=0x556a544c5230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 reap_dead start 2024-09-16T01:44:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6ee0ef40 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x556a544c2e40 0x556a544c5230 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6c4b6640 1 -- 172.21.15.88:0/2900152179 reap_dead start 2024-09-16T01:44:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9d4c06ac90 msgr2=0x7f9d4c06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6ee0ef40 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9d4c06ac90 0x7f9d4c06d150 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c005400 tx=0x7f9d5c005880 comp rx=0 tx=0).stop 2024-09-16T01:44:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556a54410c90 msgr2=0x556a544097b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6ee0ef40 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556a54410c90 0x556a544097b0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f9d54000c00 tx=0x7f9d54052550 comp rx=0 tx=0).stop 2024-09-16T01:44:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 shutdown_connections 2024-09-16T01:44:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6ee0ef40 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9d4c06ac90 0x7f9d4c06d150 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6ee0ef40 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x556a544c2e40 0x556a544c5230 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6ee0ef40 1 --2- 172.21.15.88:0/2900152179 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556a54410c90 0x556a544097b0 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 >> 172.21.15.88:0/2900152179 conn(0x556a54367680 msgr2=0x556a5436d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 shutdown_connections 2024-09-16T01:44:31.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.322+0000 7f9d6ee0ef40 1 -- 172.21.15.88:0/2900152179 wait complete. 2024-09-16T01:44:31.327 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapheader 1.00000000 deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-09-16T01:44:31.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.409+0000 7efe8700af40 1 Processor -- start 2024-09-16T01:44:31.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.409+0000 7efe8700af40 1 -- start start 2024-09-16T01:44:31.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.410+0000 7efe8700af40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56245b10fdf0 0x56245b1c8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.410+0000 7efe8700af40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56245b110710 0x56245b1e6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.410+0000 7efe8700af40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56245b1cce00 0x56245b1cb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.410+0000 7efe8700af40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56245b1e5640 con 0x56245b10fdf0 2024-09-16T01:44:31.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.410+0000 7efe8700af40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56245b060510 con 0x56245b110710 2024-09-16T01:44:31.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.410+0000 7efe8700af40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56245b1c9540 con 0x56245b1cce00 2024-09-16T01:44:31.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.410+0000 7efe83eb3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56245b10fdf0 0x56245b1c8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:31.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.410+0000 7efe83eb3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56245b10fdf0 0x56245b1c8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46298/0 (socket says 172.21.15.88:46298) 2024-09-16T01:44:31.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.410+0000 7efe83eb3640 1 -- 172.21.15.88:0/2895847436 learned_addr learned my addr 172.21.15.88:0/2895847436 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:31.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.410+0000 7efe7b6b2640 1 --2- 172.21.15.88:0/2895847436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56245b110710 0x56245b1e6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:31.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.410+0000 7efe846b4640 1 --2- 172.21.15.88:0/2895847436 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56245b1cce00 0x56245b1cb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:31.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.410+0000 7efe83eb3640 1 -- 172.21.15.88:0/2895847436 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56245b1cce00 msgr2=0x56245b1cb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:31.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.410+0000 7efe83eb3640 1 --2- 172.21.15.88:0/2895847436 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56245b1cce00 0x56245b1cb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.410+0000 7efe83eb3640 1 -- 172.21.15.88:0/2895847436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56245b110710 msgr2=0x56245b1e6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:31.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.410+0000 7efe83eb3640 1 --2- 172.21.15.88:0/2895847436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56245b110710 0x56245b1e6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.410+0000 7efe83eb3640 1 -- 172.21.15.88:0/2895847436 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56245b1c20f0 con 0x56245b10fdf0 2024-09-16T01:44:31.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.410+0000 7efe83eb3640 1 --2- 172.21.15.88:0/2895847436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56245b10fdf0 0x56245b1c8d00 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7efe7400b240 tx=0x7efe7400b700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ec0e5b3594e040e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:31.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.411+0000 7efe82eb1640 1 -- 172.21.15.88:0/2895847436 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe74008070 con 0x56245b10fdf0 2024-09-16T01:44:31.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.411+0000 7efe82eb1640 1 -- 172.21.15.88:0/2895847436 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe7400d500 con 0x56245b10fdf0 2024-09-16T01:44:31.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.411+0000 7efe82eb1640 1 -- 172.21.15.88:0/2895847436 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe7400d8b0 con 0x56245b10fdf0 2024-09-16T01:44:31.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.411+0000 7efe8700af40 1 -- 172.21.15.88:0/2895847436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56245b10fdf0 msgr2=0x56245b1c8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:31.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.411+0000 7efe8700af40 1 --2- 172.21.15.88:0/2895847436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56245b10fdf0 0x56245b1c8d00 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7efe7400b240 tx=0x7efe7400b700 comp rx=0 tx=0).stop 2024-09-16T01:44:31.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.411+0000 7efe8700af40 1 -- 172.21.15.88:0/2895847436 shutdown_connections 2024-09-16T01:44:31.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.411+0000 7efe8700af40 1 --2- 172.21.15.88:0/2895847436 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56245b1cce00 0x56245b1cb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.411+0000 7efe8700af40 1 --2- 172.21.15.88:0/2895847436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56245b110710 0x56245b1e6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.411+0000 7efe8700af40 1 --2- 172.21.15.88:0/2895847436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56245b10fdf0 0x56245b1c8d00 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.411+0000 7efe8700af40 1 -- 172.21.15.88:0/2895847436 >> 172.21.15.88:0/2895847436 conn(0x56245b139960 msgr2=0x56245b1ca2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:31.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.411+0000 7efe8700af40 1 -- 172.21.15.88:0/2895847436 shutdown_connections 2024-09-16T01:44:31.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.411+0000 7efe8700af40 1 -- 172.21.15.88:0/2895847436 wait complete. 2024-09-16T01:44:31.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.412+0000 7efe8700af40 1 Processor -- start 2024-09-16T01:44:31.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.412+0000 7efe8700af40 1 -- start start 2024-09-16T01:44:31.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.412+0000 7efe8700af40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56245b10fdf0 0x56245b13d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.412+0000 7efe8700af40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56245b110710 0x56245b13d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.412+0000 7efe8700af40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56245b1cce00 0x56245b27e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.412+0000 7efe8700af40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56245b1c26b0 con 0x56245b1cce00 2024-09-16T01:44:31.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.412+0000 7efe8700af40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56245b1c23d0 con 0x56245b110710 2024-09-16T01:44:31.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.412+0000 7efe8700af40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56245b1c9800 con 0x56245b10fdf0 2024-09-16T01:44:31.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.413+0000 7efe846b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56245b1cce00 0x56245b27e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:31.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.413+0000 7efe83eb3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56245b10fdf0 0x56245b13d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:31.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.413+0000 7efe846b4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56245b1cce00 0x56245b27e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46310/0 (socket says 172.21.15.88:46310) 2024-09-16T01:44:31.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.413+0000 7efe846b4640 1 -- 172.21.15.88:0/1277285312 learned_addr learned my addr 172.21.15.88:0/1277285312 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:31.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.413+0000 7efe7b6b2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56245b110710 0x56245b13d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:31.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.413+0000 7efe846b4640 1 -- 172.21.15.88:0/1277285312 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56245b10fdf0 msgr2=0x56245b13d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:31.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.413+0000 7efe846b4640 1 --2- 172.21.15.88:0/1277285312 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56245b10fdf0 0x56245b13d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.413+0000 7efe846b4640 1 -- 172.21.15.88:0/1277285312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56245b110710 msgr2=0x56245b13d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:31.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.413+0000 7efe846b4640 1 --2- 172.21.15.88:0/1277285312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56245b110710 0x56245b13d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.413+0000 7efe846b4640 1 -- 172.21.15.88:0/1277285312 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56245b1c8e60 con 0x56245b1cce00 2024-09-16T01:44:31.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.413+0000 7efe846b4640 1 --2- 172.21.15.88:0/1277285312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56245b1cce00 0x56245b27e180 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7efe7c0259e0 tx=0x7efe7c003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:31.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.413+0000 7efe81eaf640 1 -- 172.21.15.88:0/1277285312 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe7c064480 con 0x56245b1cce00 2024-09-16T01:44:31.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.413+0000 7efe81eaf640 1 -- 172.21.15.88:0/1277285312 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe7c069070 con 0x56245b1cce00 2024-09-16T01:44:31.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.413+0000 7efe8700af40 1 -- 172.21.15.88:0/1277285312 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56245b13dbe0 con 0x56245b1cce00 2024-09-16T01:44:31.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.413+0000 7efe81eaf640 1 -- 172.21.15.88:0/1277285312 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe7c064820 con 0x56245b1cce00 2024-09-16T01:44:31.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.413+0000 7efe8700af40 1 -- 172.21.15.88:0/1277285312 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56245b1cbef0 con 0x56245b1cce00 2024-09-16T01:44:31.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.415+0000 7efe81eaf640 1 -- 172.21.15.88:0/1277285312 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7efe7c0649c0 con 0x56245b1cce00 2024-09-16T01:44:31.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.415+0000 7efe81eaf640 1 --2- 172.21.15.88:0/1277285312 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efe6406ac90 0x7efe6406d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.415+0000 7efe81eaf640 1 -- 172.21.15.88:0/1277285312 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 9092+0+0 (secure 0 0 0) 0x7efe7c0cf810 con 0x56245b1cce00 2024-09-16T01:44:31.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.415+0000 7efe83eb3640 1 --2- 172.21.15.88:0/1277285312 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efe6406ac90 0x7efe6406d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:31.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.415+0000 7efe8700af40 1 --2- 172.21.15.88:0/1277285312 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56245b1de420 0x56245b1de800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.415+0000 7efe8700af40 1 -- 172.21.15.88:0/1277285312 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- osd_op(unknown.0.0:1 12.1f 12:ff5b34d6:::1.00000000:head [omap-set-header in=80b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126) -- 0x56245b1e03c0 con 0x56245b1de420 2024-09-16T01:44:31.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.416+0000 7efe83eb3640 1 --2- 172.21.15.88:0/1277285312 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efe6406ac90 0x7efe6406d150 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7efe7400b740 tx=0x7efe74008000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:31.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.416+0000 7efe7b6b2640 1 --2- 172.21.15.88:0/1277285312 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56245b1de420 0x56245b1de800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:31.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.416+0000 7efe7b6b2640 1 --2- 172.21.15.88:0/1277285312 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56245b1de420 0x56245b1de800 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:31.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.417+0000 7efe7b6b2640 1 -- 172.21.15.88:0/1277285312 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== osd_op_reply(1 1.00000000 [omap-set-header] v126'492 uv492 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7efe6c009a30 con 0x56245b1de420 2024-09-16T01:44:31.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.417+0000 7efe8700af40 1 -- 172.21.15.88:0/1277285312 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56245b1de420 msgr2=0x56245b1de800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:31.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.417+0000 7efe8700af40 1 --2- 172.21.15.88:0/1277285312 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x56245b1de420 0x56245b1de800 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.417+0000 7efe8700af40 1 -- 172.21.15.88:0/1277285312 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efe6406ac90 msgr2=0x7efe6406d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:31.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.417+0000 7efe8700af40 1 --2- 172.21.15.88:0/1277285312 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efe6406ac90 0x7efe6406d150 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7efe7400b740 tx=0x7efe74008000 comp rx=0 tx=0).stop 2024-09-16T01:44:31.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.417+0000 7efe8700af40 1 -- 172.21.15.88:0/1277285312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56245b1cce00 msgr2=0x56245b27e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:31.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.417+0000 7efe8700af40 1 --2- 172.21.15.88:0/1277285312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56245b1cce00 0x56245b27e180 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7efe7c0259e0 tx=0x7efe7c003330 comp rx=0 tx=0).stop 2024-09-16T01:44:31.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.417+0000 7efe846b4640 1 -- 172.21.15.88:0/1277285312 reap_dead start 2024-09-16T01:44:31.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.417+0000 7efe8700af40 1 -- 172.21.15.88:0/1277285312 shutdown_connections 2024-09-16T01:44:31.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.417+0000 7efe8700af40 1 -- 172.21.15.88:0/1277285312 >> 172.21.15.88:0/1277285312 conn(0x56245b139960 msgr2=0x56245b1ca1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:31.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.418+0000 7efe8700af40 1 -- 172.21.15.88:0/1277285312 shutdown_connections 2024-09-16T01:44:31.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.418+0000 7efe8700af40 1 -- 172.21.15.88:0/1277285312 wait complete. 2024-09-16T01:44:31.424 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:44:31.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.563+0000 7efe28931640 1 Processor -- start 2024-09-16T01:44:31.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.563+0000 7efe28931640 1 -- start start 2024-09-16T01:44:31.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.564+0000 7efe28931640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efe240b54d0 0x7efe2406fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.564+0000 7efe28931640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe2406e990 con 0x7efe2415e6f0 2024-09-16T01:44:31.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.564+0000 7efe28931640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe2406ec70 con 0x7efe240b7450 2024-09-16T01:44:31.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.564+0000 7efe28931640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7efe2415ec70 con 0x7efe240b54d0 2024-09-16T01:44:31.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.564+0000 7efe2259c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7efe2415e6f0 0x7efe240b6d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:51998/0 (socket says 172.21.15.88:51998) 2024-09-16T01:44:31.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.564+0000 7efe2259c640 1 -- 172.21.15.88:0/2974053329 learned_addr learned my addr 172.21.15.88:0/2974053329 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:31.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.564+0000 7efe21d9b640 1 --2- 172.21.15.88:0/2974053329 >> v2:172.21.15.154:3300/0 conn(0x7efe240b54d0 0x7efe2406fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:31.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.564+0000 7efe2159a640 1 -- 172.21.15.88:0/2974053329 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3090870940 0 0) 0x7efe2406e990 con 0x7efe2415e6f0 2024-09-16T01:44:31.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.564+0000 7efe2159a640 1 -- 172.21.15.88:0/2974053329 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe0c003520 con 0x7efe2415e6f0 2024-09-16T01:44:31.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.564+0000 7efe21d9b640 1 -- 172.21.15.88:0/2974053329 >> v1:172.21.15.154:6789/0 conn(0x7efe240b7450 legacy=0x7efe24070830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:31.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.564+0000 7efe21d9b640 1 -- 172.21.15.88:0/2974053329 >> v1:172.21.15.88:6789/0 conn(0x7efe2415e6f0 legacy=0x7efe240b6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:31.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.564+0000 7efe21d9b640 1 -- 172.21.15.88:0/2974053329 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe2415ef50 con 0x7efe240b54d0 2024-09-16T01:44:31.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.565+0000 7efe21d9b640 1 --2- 172.21.15.88:0/2974053329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe240b54d0 0x7efe2406fd00 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7efe180097a0 tx=0x7efe1802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=802e8ae0de87556b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:31.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.565+0000 7efe2159a640 1 -- 172.21.15.88:0/2974053329 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe18037070 con 0x7efe240b54d0 2024-09-16T01:44:31.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.565+0000 7efe2159a640 1 -- 172.21.15.88:0/2974053329 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe18032440 con 0x7efe240b54d0 2024-09-16T01:44:31.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.565+0000 7efe2159a640 1 -- 172.21.15.88:0/2974053329 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe18032810 con 0x7efe240b54d0 2024-09-16T01:44:31.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.566+0000 7efe28931640 1 -- 172.21.15.88:0/2974053329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe240b54d0 msgr2=0x7efe2406fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:31.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.566+0000 7efe28931640 1 --2- 172.21.15.88:0/2974053329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe240b54d0 0x7efe2406fd00 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7efe180097a0 tx=0x7efe1802a230 comp rx=0 tx=0).stop 2024-09-16T01:44:31.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.566+0000 7efe28931640 1 -- 172.21.15.88:0/2974053329 shutdown_connections 2024-09-16T01:44:31.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.566+0000 7efe28931640 1 --2- 172.21.15.88:0/2974053329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe240b54d0 0x7efe2406fd00 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.566+0000 7efe28931640 1 -- 172.21.15.88:0/2974053329 >> 172.21.15.88:0/2974053329 conn(0x7efe240d2430 msgr2=0x7efe240d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:31.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.566+0000 7efe28931640 1 -- 172.21.15.88:0/2974053329 shutdown_connections 2024-09-16T01:44:31.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.566+0000 7efe28931640 1 -- 172.21.15.88:0/2974053329 wait complete. 2024-09-16T01:44:31.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.567+0000 7efe28931640 1 Processor -- start 2024-09-16T01:44:31.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.567+0000 7efe28931640 1 -- start start 2024-09-16T01:44:31.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.568+0000 7efe28931640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe240b54d0 0x7efe24151040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.568+0000 7efe28931640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe240b7450 0x7efe241567e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.568+0000 7efe28931640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe2415e6f0 0x7efe241f78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.568+0000 7efe28931640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7efe2406e730 con 0x7efe2415e6f0 2024-09-16T01:44:31.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.568+0000 7efe28931640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efe2406e5b0 con 0x7efe240b7450 2024-09-16T01:44:31.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.568+0000 7efe28931640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efe2415f750 con 0x7efe240b54d0 2024-09-16T01:44:31.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.568+0000 7efe22d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe2415e6f0 0x7efe241f78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:31.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.568+0000 7efe22d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe2415e6f0 0x7efe241f78b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46322/0 (socket says 172.21.15.88:46322) 2024-09-16T01:44:31.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.568+0000 7efe22d9d640 1 -- 172.21.15.88:0/1181494669 learned_addr learned my addr 172.21.15.88:0/1181494669 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:31.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.568+0000 7efe21d9b640 1 --2- 172.21.15.88:0/1181494669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe240b7450 0x7efe241567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:31.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.568+0000 7efe22d9d640 1 -- 172.21.15.88:0/1181494669 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe240b54d0 msgr2=0x7efe24151040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:31.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.568+0000 7efe2259c640 1 --2- 172.21.15.88:0/1181494669 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe240b54d0 0x7efe24151040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:31.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.568+0000 7efe22d9d640 1 --2- 172.21.15.88:0/1181494669 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe240b54d0 0x7efe24151040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.568+0000 7efe22d9d640 1 -- 172.21.15.88:0/1181494669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe240b7450 msgr2=0x7efe241567e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:31.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.568+0000 7efe22d9d640 1 --2- 172.21.15.88:0/1181494669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe240b7450 0x7efe241567e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:31.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.568+0000 7efe22d9d640 1 -- 172.21.15.88:0/1181494669 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe24157510 con 0x7efe2415e6f0 2024-09-16T01:44:31.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.568+0000 7efe21d9b640 1 --2- 172.21.15.88:0/1181494669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe240b7450 0x7efe241567e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:31.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.568+0000 7efe2259c640 1 --2- 172.21.15.88:0/1181494669 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe240b54d0 0x7efe24151040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:31.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.569+0000 7efe22d9d640 1 --2- 172.21.15.88:0/1181494669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe2415e6f0 0x7efe241f78b0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7efe1c00c7d0 tx=0x7efe1c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:31.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.569+0000 7efe12ffd640 1 -- 172.21.15.88:0/1181494669 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe1c003b70 con 0x7efe2415e6f0 2024-09-16T01:44:31.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.569+0000 7efe12ffd640 1 -- 172.21.15.88:0/1181494669 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe1c003d30 con 0x7efe2415e6f0 2024-09-16T01:44:31.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.569+0000 7efe12ffd640 1 -- 172.21.15.88:0/1181494669 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe1c00aa90 con 0x7efe2415e6f0 2024-09-16T01:44:31.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.569+0000 7efe28931640 1 -- 172.21.15.88:0/1181494669 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe24157230 con 0x7efe2415e6f0 2024-09-16T01:44:31.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.569+0000 7efe28931640 1 -- 172.21.15.88:0/1181494669 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe24160570 con 0x7efe2415e6f0 2024-09-16T01:44:31.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.570+0000 7efe28931640 1 -- 172.21.15.88:0/1181494669 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe2406c6c0 con 0x7efe2415e6f0 2024-09-16T01:44:31.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.570+0000 7efe12ffd640 1 -- 172.21.15.88:0/1181494669 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7efe1c00ac30 con 0x7efe2415e6f0 2024-09-16T01:44:31.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.571+0000 7efe12ffd640 1 --2- 172.21.15.88:0/1181494669 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efe0406c2b0 0x7efe0406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:31.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.571+0000 7efe12ffd640 1 -- 172.21.15.88:0/1181494669 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 9092+0+0 (secure 0 0 0) 0x7efe1c08c0e0 con 0x7efe2415e6f0 2024-09-16T01:44:31.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.571+0000 7efe2259c640 1 --2- 172.21.15.88:0/1181494669 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efe0406c2b0 0x7efe0406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:31.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.571+0000 7efe2259c640 1 --2- 172.21.15.88:0/1181494669 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efe0406c2b0 0x7efe0406e770 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7efe140011e0 tx=0x7efe14009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:31.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.575+0000 7efe12ffd640 1 -- 172.21.15.88:0/1181494669 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7efe1c059b50 con 0x7efe2415e6f0 2024-09-16T01:44:31.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:31.774+0000 7efe28931640 1 -- 172.21.15.88:0/1181494669 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7efe2415fae0 con 0x7efe2415e6f0 2024-09-16T01:44:32.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.105+0000 7efe12ffd640 1 -- 172.21.15.88:0/1181494669 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v313) ==== 162+0+0 (secure 0 0 0) 0x7efe1c059450 con 0x7efe2415e6f0 2024-09-16T01:44:32.107 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:44:32.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.107+0000 7efe28931640 1 -- 172.21.15.88:0/1181494669 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efe0406c2b0 msgr2=0x7efe0406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:32.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.107+0000 7efe28931640 1 --2- 172.21.15.88:0/1181494669 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efe0406c2b0 0x7efe0406e770 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7efe140011e0 tx=0x7efe14009040 comp rx=0 tx=0).stop 2024-09-16T01:44:32.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.107+0000 7efe28931640 1 -- 172.21.15.88:0/1181494669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe2415e6f0 msgr2=0x7efe241f78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:32.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.107+0000 7efe28931640 1 --2- 172.21.15.88:0/1181494669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe2415e6f0 0x7efe241f78b0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7efe1c00c7d0 tx=0x7efe1c00cc90 comp rx=0 tx=0).stop 2024-09-16T01:44:32.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.107+0000 7efe28931640 1 -- 172.21.15.88:0/1181494669 shutdown_connections 2024-09-16T01:44:32.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.107+0000 7efe28931640 1 --2- 172.21.15.88:0/1181494669 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efe0406c2b0 0x7efe0406e770 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:32.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.107+0000 7efe28931640 1 --2- 172.21.15.88:0/1181494669 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efe2415e6f0 0x7efe241f78b0 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:32.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.107+0000 7efe28931640 1 --2- 172.21.15.88:0/1181494669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe240b7450 0x7efe241567e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:32.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.107+0000 7efe28931640 1 --2- 172.21.15.88:0/1181494669 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe240b54d0 0x7efe24151040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:32.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.107+0000 7efe28931640 1 -- 172.21.15.88:0/1181494669 >> 172.21.15.88:0/1181494669 conn(0x7efe240d2430 msgr2=0x7efe2406cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:32.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.108+0000 7efe28931640 1 -- 172.21.15.88:0/1181494669 shutdown_connections 2024-09-16T01:44:32.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.108+0000 7efe28931640 1 -- 172.21.15.88:0/1181494669 wait complete. 2024-09-16T01:44:32.122 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:44:32.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.219+0000 7f2a60381640 1 Processor -- start 2024-09-16T01:44:32.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.219+0000 7f2a60381640 1 -- start start 2024-09-16T01:44:32.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.220+0000 7f2a60381640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2a580cfc80 0x7f2a580b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:32.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.220+0000 7f2a60381640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2a5815c330 0x7f2a580b4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:32.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.220+0000 7f2a60381640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a5815c740 con 0x7f2a58155800 2024-09-16T01:44:32.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.220+0000 7f2a60381640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f2a580b4e60 con 0x7f2a5815c330 2024-09-16T01:44:32.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.220+0000 7f2a60381640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2a580b50a0 con 0x7f2a580cfc80 2024-09-16T01:44:32.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.220+0000 7f2a5e91e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2a5815c330 0x7f2a580b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:32.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.220+0000 7f2a5e91e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f2a5815c330 0x7f2a580b4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46336/0 (socket says 172.21.15.88:46336) 2024-09-16T01:44:32.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.220+0000 7f2a5e91e640 1 -- 172.21.15.88:0/2357840799 learned_addr learned my addr 172.21.15.88:0/2357840799 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:32.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.220+0000 7f2a5d91c640 1 --2- 172.21.15.88:0/2357840799 >> v2:172.21.15.154:3300/0 conn(0x7f2a580cfc80 0x7f2a580b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:32.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.220+0000 7f2a5e91e640 1 -- 172.21.15.88:0/2357840799 >> v2:172.21.15.154:3300/0 conn(0x7f2a580cfc80 msgr2=0x7f2a580b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:32.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.220+0000 7f2a5e91e640 1 --2- 172.21.15.88:0/2357840799 >> v2:172.21.15.154:3300/0 conn(0x7f2a580cfc80 0x7f2a580b6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:32.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.220+0000 7f2a5e91e640 1 -- 172.21.15.88:0/2357840799 >> v1:172.21.15.154:6789/0 conn(0x7f2a58155800 legacy=0x7f2a580cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:32.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.220+0000 7f2a5e91e640 1 -- 172.21.15.88:0/2357840799 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a580b5380 con 0x7f2a5815c330 2024-09-16T01:44:32.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.221+0000 7f2a5e91e640 1 --2- 172.21.15.88:0/2357840799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a5815c330 0x7f2a580b4920 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f2a5000d150 tx=0x7f2a5002c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a03978b739653b70 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:32.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.221+0000 7f2a5d11b640 1 -- 172.21.15.88:0/2357840799 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a50009600 con 0x7f2a5815c330 2024-09-16T01:44:32.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.221+0000 7f2a5d11b640 1 -- 172.21.15.88:0/2357840799 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a5002f070 con 0x7f2a5815c330 2024-09-16T01:44:32.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.221+0000 7f2a5d11b640 1 -- 172.21.15.88:0/2357840799 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a500099c0 con 0x7f2a5815c330 2024-09-16T01:44:32.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.221+0000 7f2a60381640 1 -- 172.21.15.88:0/2357840799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a5815c330 msgr2=0x7f2a580b4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:32.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.221+0000 7f2a60381640 1 --2- 172.21.15.88:0/2357840799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a5815c330 0x7f2a580b4920 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f2a5000d150 tx=0x7f2a5002c930 comp rx=0 tx=0).stop 2024-09-16T01:44:32.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.222+0000 7f2a60381640 1 -- 172.21.15.88:0/2357840799 shutdown_connections 2024-09-16T01:44:32.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.222+0000 7f2a60381640 1 --2- 172.21.15.88:0/2357840799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a5815c330 0x7f2a580b4920 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:32.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.222+0000 7f2a60381640 1 --2- 172.21.15.88:0/2357840799 >> v2:172.21.15.154:3300/0 conn(0x7f2a580cfc80 0x7f2a580b6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:32.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.222+0000 7f2a60381640 1 -- 172.21.15.88:0/2357840799 >> 172.21.15.88:0/2357840799 conn(0x7f2a5806bc10 msgr2=0x7f2a5806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:32.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.222+0000 7f2a60381640 1 -- 172.21.15.88:0/2357840799 shutdown_connections 2024-09-16T01:44:32.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.222+0000 7f2a60381640 1 -- 172.21.15.88:0/2357840799 wait complete. 2024-09-16T01:44:32.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.223+0000 7f2a60381640 1 Processor -- start 2024-09-16T01:44:32.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.223+0000 7f2a60381640 1 -- start start 2024-09-16T01:44:32.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.223+0000 7f2a60381640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a580cfc80 0x7f2a581f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:32.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.224+0000 7f2a60381640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a58155800 0x7f2a5815a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:32.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.224+0000 7f2a60381640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a5815c330 0x7f2a5815a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:32.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.224+0000 7f2a60381640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2a580b5c20 con 0x7f2a580cfc80 2024-09-16T01:44:32.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.224+0000 7f2a60381640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2a580b5940 con 0x7f2a5815c330 2024-09-16T01:44:32.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.224+0000 7f2a60381640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2a58155be0 con 0x7f2a58155800 2024-09-16T01:44:32.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.224+0000 7f2a5e11d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a580cfc80 0x7f2a581f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:32.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.224+0000 7f2a5e11d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a580cfc80 0x7f2a581f6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46342/0 (socket says 172.21.15.88:46342) 2024-09-16T01:44:32.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.224+0000 7f2a5e11d640 1 -- 172.21.15.88:0/2366052258 learned_addr learned my addr 172.21.15.88:0/2366052258 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:32.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.224+0000 7f2a5d91c640 1 --2- 172.21.15.88:0/2366052258 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a58155800 0x7f2a5815a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:32.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.224+0000 7f2a5e91e640 1 --2- 172.21.15.88:0/2366052258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a5815c330 0x7f2a5815a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:32.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.224+0000 7f2a5e11d640 1 -- 172.21.15.88:0/2366052258 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a58155800 msgr2=0x7f2a5815a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:32.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.224+0000 7f2a5e11d640 1 --2- 172.21.15.88:0/2366052258 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a58155800 0x7f2a5815a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:32.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.224+0000 7f2a5e11d640 1 -- 172.21.15.88:0/2366052258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a5815c330 msgr2=0x7f2a5815a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:32.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.224+0000 7f2a5e11d640 1 --2- 172.21.15.88:0/2366052258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a5815c330 0x7f2a5815a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:32.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.224+0000 7f2a5e11d640 1 -- 172.21.15.88:0/2366052258 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a58153b40 con 0x7f2a580cfc80 2024-09-16T01:44:32.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.224+0000 7f2a5e91e640 1 --2- 172.21.15.88:0/2366052258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a5815c330 0x7f2a5815a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:32.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.224+0000 7f2a5d91c640 1 --2- 172.21.15.88:0/2366052258 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a58155800 0x7f2a5815a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:32.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.224+0000 7f2a5e11d640 1 --2- 172.21.15.88:0/2366052258 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a580cfc80 0x7f2a581f6e20 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7f2a5400c430 tx=0x7f2a5400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:32.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.224+0000 7f2a4effd640 1 -- 172.21.15.88:0/2366052258 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a54011790 con 0x7f2a580cfc80 2024-09-16T01:44:32.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.224+0000 7f2a4effd640 1 -- 172.21.15.88:0/2366052258 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a54017070 con 0x7f2a580cfc80 2024-09-16T01:44:32.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.224+0000 7f2a4effd640 1 -- 172.21.15.88:0/2366052258 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a54011bb0 con 0x7f2a580cfc80 2024-09-16T01:44:32.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.224+0000 7f2a60381640 1 -- 172.21.15.88:0/2366052258 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a581f4d00 con 0x7f2a580cfc80 2024-09-16T01:44:32.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.224+0000 7f2a60381640 1 -- 172.21.15.88:0/2366052258 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a581501a0 con 0x7f2a580cfc80 2024-09-16T01:44:32.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.225+0000 7f2a60381640 1 -- 172.21.15.88:0/2366052258 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a580b6140 con 0x7f2a580cfc80 2024-09-16T01:44:32.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.226+0000 7f2a4effd640 1 -- 172.21.15.88:0/2366052258 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2a5401c460 con 0x7f2a580cfc80 2024-09-16T01:44:32.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.226+0000 7f2a4effd640 1 --2- 172.21.15.88:0/2366052258 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2a3c06c2b0 0x7f2a3c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:32.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.226+0000 7f2a5d91c640 1 --2- 172.21.15.88:0/2366052258 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2a3c06c2b0 0x7f2a3c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:32.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.226+0000 7f2a4effd640 1 -- 172.21.15.88:0/2366052258 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 9092+0+0 (secure 0 0 0) 0x7f2a5408c010 con 0x7f2a580cfc80 2024-09-16T01:44:32.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.226+0000 7f2a5d91c640 1 --2- 172.21.15.88:0/2366052258 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2a3c06c2b0 0x7f2a3c06e770 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f2a5815e920 tx=0x7f2a48008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:32.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.230+0000 7f2a4effd640 1 -- 172.21.15.88:0/2366052258 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2a54059a80 con 0x7f2a580cfc80 2024-09-16T01:44:32.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.429+0000 7f2a60381640 1 -- 172.21.15.88:0/2366052258 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a580633d0 con 0x7f2a580cfc80 2024-09-16T01:44:32.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.430+0000 7f2a4effd640 1 -- 172.21.15.88:0/2366052258 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 314 v314) ==== 95+0+4401 (secure 0 0 0) 0x7f2a54059380 con 0x7f2a580cfc80 2024-09-16T01:44:32.432 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:32.432 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":314,"btime":"2024-09-16T01:44:32:108388+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":314,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:44:32.108378+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":126,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9696},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9696":{"gid":9696,"name":"c","rank":0,"incarnation":314,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6837/4270197461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":4270197461},{"type":"v1","addr":"172.21.15.88:6837","nonce":4270197461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:44:32.432 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 314 2024-09-16T01:44:32.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.432+0000 7f2a60381640 1 -- 172.21.15.88:0/2366052258 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2a3c06c2b0 msgr2=0x7f2a3c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:32.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.432+0000 7f2a60381640 1 --2- 172.21.15.88:0/2366052258 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2a3c06c2b0 0x7f2a3c06e770 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f2a5815e920 tx=0x7f2a48008040 comp rx=0 tx=0).stop 2024-09-16T01:44:32.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.432+0000 7f2a60381640 1 -- 172.21.15.88:0/2366052258 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a580cfc80 msgr2=0x7f2a581f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:32.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.432+0000 7f2a60381640 1 --2- 172.21.15.88:0/2366052258 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a580cfc80 0x7f2a581f6e20 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7f2a5400c430 tx=0x7f2a5400c8f0 comp rx=0 tx=0).stop 2024-09-16T01:44:32.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.432+0000 7f2a60381640 1 -- 172.21.15.88:0/2366052258 shutdown_connections 2024-09-16T01:44:32.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.432+0000 7f2a60381640 1 --2- 172.21.15.88:0/2366052258 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2a3c06c2b0 0x7f2a3c06e770 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:32.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.432+0000 7f2a60381640 1 --2- 172.21.15.88:0/2366052258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a5815c330 0x7f2a5815a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:32.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.432+0000 7f2a60381640 1 --2- 172.21.15.88:0/2366052258 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2a58155800 0x7f2a5815a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:32.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.432+0000 7f2a60381640 1 --2- 172.21.15.88:0/2366052258 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2a580cfc80 0x7f2a581f6e20 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:32.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.432+0000 7f2a60381640 1 -- 172.21.15.88:0/2366052258 >> 172.21.15.88:0/2366052258 conn(0x7f2a5806bc10 msgr2=0x7f2a580cdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:32.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.432+0000 7f2a60381640 1 -- 172.21.15.88:0/2366052258 shutdown_connections 2024-09-16T01:44:32.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:32.432+0000 7f2a60381640 1 -- 172.21.15.88:0/2366052258 wait complete. 2024-09-16T01:44:32.444 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-09-16T01:44:32.838 INFO:tasks.ceph.mds.a.smithi088.stderr: -45> 2024-09-16T01:44:28.086+0000 7fbd679cb640 -1 log_channel(cluster) log [ERR] : corrupt sessionmap header: Decoder at 'virtual void SessionMapStore::decode_header(ceph::bufferlist&)' v=1 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-09-16T01:44:32.862 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:44:35.136 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:44:35.135+0000 7fcfeed27640 -1 mds.0.cache.dir(0x1) Corrupt fnode in dirfrag 0x1: Decoder at 'void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&)' v=3 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-09-16T01:44:35.136 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:44:35.135+0000 7fcfeed27640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x1 is fatal because it is a system directory for this rank 2024-09-16T01:44:35.136 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:44:35.135+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:44:35.136 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:44:35.135+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-09-16T01:44:37.445 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:44:37.498 INFO:tasks.ceph.mds.c.smithi088.stderr:2024-09-16T01:44:37.497+0000 7fcff2d2f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:44:37.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.556+0000 7fe768fbd640 1 Processor -- start 2024-09-16T01:44:37.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.556+0000 7fe768fbd640 1 -- start start 2024-09-16T01:44:37.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.557+0000 7fe768fbd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe76415c550 0x7fe7640715e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:37.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.557+0000 7fe768fbd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7640d0270 con 0x7fe7640b6e10 2024-09-16T01:44:37.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.557+0000 7fe768fbd640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7640d04f0 con 0x7fe764073160 2024-09-16T01:44:37.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.557+0000 7fe768fbd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe76415c930 con 0x7fe76415c550 2024-09-16T01:44:37.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.557+0000 7fe759d9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fe7640b6e10 0x7fe7640b5540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52006/0 (socket says 172.21.15.88:52006) 2024-09-16T01:44:37.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.557+0000 7fe759d9b640 1 -- 172.21.15.88:0/2393747536 learned_addr learned my addr 172.21.15.88:0/2393747536 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:37.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.557+0000 7fe761d9b640 1 -- 172.21.15.88:0/2393747536 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 640928325 0 0) 0x7fe74c001110 con 0x7fe7640b6e10 2024-09-16T01:44:37.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.557+0000 7fe76259c640 1 --2- 172.21.15.88:0/2393747536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe76415c550 0x7fe7640715e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:37.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.557+0000 7fe761d9b640 1 -- 172.21.15.88:0/2393747536 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe750003520 con 0x7fe7640b6e10 2024-09-16T01:44:37.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.558+0000 7fe761d9b640 1 -- 172.21.15.88:0/2393747536 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2645700638 0 0) 0x7fe750003520 con 0x7fe7640b6e10 2024-09-16T01:44:37.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.558+0000 7fe761d9b640 1 -- 172.21.15.88:0/2393747536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe76415c550 msgr2=0x7fe7640715e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:37.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.558+0000 7fe761d9b640 1 --2- 172.21.15.88:0/2393747536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe76415c550 0x7fe7640715e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:37.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.558+0000 7fe761d9b640 1 -- 172.21.15.88:0/2393747536 >> v1:172.21.15.154:6789/0 conn(0x7fe764073160 legacy=0x7fe7640d0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:37.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.558+0000 7fe761d9b640 1 -- 172.21.15.88:0/2393747536 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7640b7580 con 0x7fe7640b6e10 2024-09-16T01:44:37.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.558+0000 7fe761d9b640 1 -- 172.21.15.88:0/2393747536 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fe74c001840 con 0x7fe7640b6e10 2024-09-16T01:44:37.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.558+0000 7fe761d9b640 1 -- 172.21.15.88:0/2393747536 >> v1:172.21.15.88:6789/0 conn(0x7fe7640b6e10 legacy=0x7fe7640b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:37.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.558+0000 7fe761d9b640 1 --2- 172.21.15.88:0/2393747536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7500039e0 0x7fe750003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:37.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.558+0000 7fe761d9b640 1 --2- 172.21.15.88:0/2393747536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe750004390 0x7fe750004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:37.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.558+0000 7fe761d9b640 1 --2- 172.21.15.88:0/2393747536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe76415c550 0x7fe7500050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:37.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.558+0000 7fe761d9b640 1 -- 172.21.15.88:0/2393747536 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe76415c930 con 0x7fe7500039e0 2024-09-16T01:44:37.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.558+0000 7fe761d9b640 1 -- 172.21.15.88:0/2393747536 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe75000b9c0 con 0x7fe750004390 2024-09-16T01:44:37.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.558+0000 7fe761d9b640 1 -- 172.21.15.88:0/2393747536 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe75000bbc0 con 0x7fe76415c550 2024-09-16T01:44:37.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.558+0000 7fe76259c640 1 --2- 172.21.15.88:0/2393747536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7500039e0 0x7fe750003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:37.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.558+0000 7fe759d9b640 1 --2- 172.21.15.88:0/2393747536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe750004390 0x7fe750004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:37.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.558+0000 7fe762d9d640 1 --2- 172.21.15.88:0/2393747536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe76415c550 0x7fe7500050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:37.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.558+0000 7fe76259c640 1 -- 172.21.15.88:0/2393747536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe76415c550 msgr2=0x7fe7500050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:37.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.558+0000 7fe76259c640 1 --2- 172.21.15.88:0/2393747536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe76415c550 0x7fe7500050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:37.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.558+0000 7fe76259c640 1 -- 172.21.15.88:0/2393747536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe750004390 msgr2=0x7fe750004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:37.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.558+0000 7fe76259c640 1 --2- 172.21.15.88:0/2393747536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe750004390 0x7fe750004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:37.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.558+0000 7fe76259c640 1 -- 172.21.15.88:0/2393747536 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe750003520 con 0x7fe7500039e0 2024-09-16T01:44:37.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.558+0000 7fe759d9b640 1 --2- 172.21.15.88:0/2393747536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe750004390 0x7fe750004860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:37.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.559+0000 7fe76259c640 1 --2- 172.21.15.88:0/2393747536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7500039e0 0x7fe750003dc0 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7fe754009ac0 tx=0x7fe750005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef9bdbb3847e244d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:37.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.559+0000 7fe761d9b640 1 -- 172.21.15.88:0/2393747536 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe754036070 con 0x7fe7500039e0 2024-09-16T01:44:37.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.559+0000 7fe761d9b640 1 -- 172.21.15.88:0/2393747536 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe754031400 con 0x7fe7500039e0 2024-09-16T01:44:37.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.559+0000 7fe761d9b640 1 -- 172.21.15.88:0/2393747536 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe754009690 con 0x7fe7500039e0 2024-09-16T01:44:37.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.559+0000 7fe768fbd640 1 -- 172.21.15.88:0/2393747536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7500039e0 msgr2=0x7fe750003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:37.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.559+0000 7fe768fbd640 1 --2- 172.21.15.88:0/2393747536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7500039e0 0x7fe750003dc0 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7fe754009ac0 tx=0x7fe750005830 comp rx=0 tx=0).stop 2024-09-16T01:44:37.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.559+0000 7fe762d9d640 1 -- 172.21.15.88:0/2393747536 reap_dead start 2024-09-16T01:44:37.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.560+0000 7fe768fbd640 1 -- 172.21.15.88:0/2393747536 shutdown_connections 2024-09-16T01:44:37.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.560+0000 7fe768fbd640 1 -- 172.21.15.88:0/2393747536 >> 172.21.15.88:0/2393747536 conn(0x7fe7640b4240 msgr2=0x7fe7640b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:37.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.560+0000 7fe768fbd640 1 -- 172.21.15.88:0/2393747536 shutdown_connections 2024-09-16T01:44:37.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.560+0000 7fe768fbd640 1 -- 172.21.15.88:0/2393747536 wait complete. 2024-09-16T01:44:37.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.561+0000 7fe768fbd640 1 Processor -- start 2024-09-16T01:44:37.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.561+0000 7fe768fbd640 1 -- start start 2024-09-16T01:44:37.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.562+0000 7fe768fbd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7500039e0 0x7fe7641564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:37.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.562+0000 7fe768fbd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe764152710 0x7fe7641f2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:37.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.562+0000 7fe768fbd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7641f28d0 0x7fe7641f2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:37.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.562+0000 7fe768fbd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe7640b7250 con 0x7fe7641f28d0 2024-09-16T01:44:37.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.562+0000 7fe768fbd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe7640d07b0 con 0x7fe764152710 2024-09-16T01:44:37.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.562+0000 7fe768fbd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe7640d13e0 con 0x7fe7500039e0 2024-09-16T01:44:37.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.562+0000 7fe762d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7641f28d0 0x7fe7641f2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:37.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.562+0000 7fe762d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7641f28d0 0x7fe7641f2d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46390/0 (socket says 172.21.15.88:46390) 2024-09-16T01:44:37.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.562+0000 7fe762d9d640 1 -- 172.21.15.88:0/2916638773 learned_addr learned my addr 172.21.15.88:0/2916638773 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:37.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.562+0000 7fe759d9b640 1 --2- 172.21.15.88:0/2916638773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe764152710 0x7fe7641f2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:37.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.562+0000 7fe76259c640 1 --2- 172.21.15.88:0/2916638773 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7500039e0 0x7fe7641564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:37.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.562+0000 7fe762d9d640 1 -- 172.21.15.88:0/2916638773 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7500039e0 msgr2=0x7fe7641564f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:37.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.562+0000 7fe762d9d640 1 --2- 172.21.15.88:0/2916638773 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7500039e0 0x7fe7641564f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:37.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.562+0000 7fe762d9d640 1 -- 172.21.15.88:0/2916638773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe764152710 msgr2=0x7fe7641f2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:37.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.562+0000 7fe762d9d640 1 --2- 172.21.15.88:0/2916638773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe764152710 0x7fe7641f2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:37.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.562+0000 7fe762d9d640 1 -- 172.21.15.88:0/2916638773 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe764153f90 con 0x7fe7641f28d0 2024-09-16T01:44:37.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.562+0000 7fe76259c640 1 --2- 172.21.15.88:0/2916638773 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7500039e0 0x7fe7641564f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:37.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.562+0000 7fe759d9b640 1 --2- 172.21.15.88:0/2916638773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe764152710 0x7fe7641f2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:37.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.563+0000 7fe762d9d640 1 --2- 172.21.15.88:0/2916638773 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7641f28d0 0x7fe7641f2d50 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7fe764074420 tx=0x7fe75c00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:37.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.563+0000 7fe75b7fe640 1 -- 172.21.15.88:0/2916638773 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe75c00ee50 con 0x7fe7641f28d0 2024-09-16T01:44:37.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.563+0000 7fe75b7fe640 1 -- 172.21.15.88:0/2916638773 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe75c018070 con 0x7fe7641f28d0 2024-09-16T01:44:37.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.563+0000 7fe768fbd640 1 -- 172.21.15.88:0/2916638773 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe764156ad0 con 0x7fe7641f28d0 2024-09-16T01:44:37.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.563+0000 7fe75b7fe640 1 -- 172.21.15.88:0/2916638773 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe75c003dc0 con 0x7fe7641f28d0 2024-09-16T01:44:37.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.563+0000 7fe768fbd640 1 -- 172.21.15.88:0/2916638773 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7640aa800 con 0x7fe7641f28d0 2024-09-16T01:44:37.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.564+0000 7fe768fbd640 1 -- 172.21.15.88:0/2916638773 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7640d1730 con 0x7fe7641f28d0 2024-09-16T01:44:37.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.564+0000 7fe75b7fe640 1 -- 172.21.15.88:0/2916638773 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe75c0050d0 con 0x7fe7641f28d0 2024-09-16T01:44:37.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.565+0000 7fe75b7fe640 1 --2- 172.21.15.88:0/2916638773 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe74406c2b0 0x7fe74406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:37.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.565+0000 7fe75b7fe640 1 -- 172.21.15.88:0/2916638773 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9178+0+0 (secure 0 0 0) 0x7fe75c08bf30 con 0x7fe7641f28d0 2024-09-16T01:44:37.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.565+0000 7fe76259c640 1 --2- 172.21.15.88:0/2916638773 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe74406c2b0 0x7fe74406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:37.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.565+0000 7fe76259c640 1 --2- 172.21.15.88:0/2916638773 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe74406c2b0 0x7fe74406e770 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fe754009ac0 tx=0x7fe754008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:37.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.569+0000 7fe75b7fe640 1 -- 172.21.15.88:0/2916638773 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe75c0598c0 con 0x7fe7641f28d0 2024-09-16T01:44:37.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.768+0000 7fe768fbd640 1 -- 172.21.15.88:0/2916638773 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7640d19e0 con 0x7fe7641f28d0 2024-09-16T01:44:37.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.769+0000 7fe75b7fe640 1 -- 172.21.15.88:0/2916638773 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 318 v318) ==== 95+0+4393 (secure 0 0 0) 0x7fe75c0591c0 con 0x7fe7641f28d0 2024-09-16T01:44:37.771 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:37.771 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":318,"btime":"2024-09-16T01:44:35:136561+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":10398,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2633617067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2633617067},{"type":"v1","addr":"172.21.15.88:6836","nonce":2633617067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":315}],"filesystems":[{"mdsmap":{"epoch":318,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:44:35.136557+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:44:37.771 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 318 2024-09-16T01:44:37.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.771+0000 7fe768fbd640 1 -- 172.21.15.88:0/2916638773 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe74406c2b0 msgr2=0x7fe74406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:37.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.771+0000 7fe768fbd640 1 --2- 172.21.15.88:0/2916638773 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe74406c2b0 0x7fe74406e770 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fe754009ac0 tx=0x7fe754008030 comp rx=0 tx=0).stop 2024-09-16T01:44:37.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.771+0000 7fe768fbd640 1 -- 172.21.15.88:0/2916638773 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7641f28d0 msgr2=0x7fe7641f2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:37.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.771+0000 7fe768fbd640 1 --2- 172.21.15.88:0/2916638773 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7641f28d0 0x7fe7641f2d50 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7fe764074420 tx=0x7fe75c00ec00 comp rx=0 tx=0).stop 2024-09-16T01:44:37.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.771+0000 7fe768fbd640 1 -- 172.21.15.88:0/2916638773 shutdown_connections 2024-09-16T01:44:37.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.771+0000 7fe768fbd640 1 --2- 172.21.15.88:0/2916638773 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe74406c2b0 0x7fe74406e770 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:37.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.771+0000 7fe768fbd640 1 --2- 172.21.15.88:0/2916638773 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7641f28d0 0x7fe7641f2d50 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:37.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.771+0000 7fe768fbd640 1 --2- 172.21.15.88:0/2916638773 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe764152710 0x7fe7641f2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:37.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.771+0000 7fe768fbd640 1 --2- 172.21.15.88:0/2916638773 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7500039e0 0x7fe7641564f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:37.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.771+0000 7fe768fbd640 1 -- 172.21.15.88:0/2916638773 >> 172.21.15.88:0/2916638773 conn(0x7fe7640b4240 msgr2=0x7fe764158320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:37.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.771+0000 7fe768fbd640 1 -- 172.21.15.88:0/2916638773 shutdown_connections 2024-09-16T01:44:37.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.771+0000 7fe768fbd640 1 -- 172.21.15.88:0/2916638773 wait complete. 2024-09-16T01:44:37.785 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-09-16T01:44:37.786 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:44:37.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.881+0000 7f6618da6640 1 Processor -- start 2024-09-16T01:44:37.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.881+0000 7f6618da6640 1 -- start start 2024-09-16T01:44:37.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.882+0000 7f6618da6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f661406e7c0 0x7f66141672a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:37.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.882+0000 7f6618da6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66140d03c0 con 0x7f6614145bb0 2024-09-16T01:44:37.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.882+0000 7f6618da6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66140d06a0 con 0x7f661413f400 2024-09-16T01:44:37.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.882+0000 7f6618da6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f66141431e0 con 0x7f661406e7c0 2024-09-16T01:44:37.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.882+0000 7f6611d9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6614145bb0 0x7f661406e0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52012/0 (socket says 172.21.15.88:52012) 2024-09-16T01:44:37.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.882+0000 7f6611d9b640 1 -- 172.21.15.88:0/391444589 learned_addr learned my addr 172.21.15.88:0/391444589 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:37.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.882+0000 7f6612d9d640 1 --2- 172.21.15.88:0/391444589 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f661406e7c0 0x7f66141672a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:37.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.883+0000 7f661159a640 1 -- 172.21.15.88:0/391444589 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 49034720 0 0) 0x7f66140d03c0 con 0x7f6614145bb0 2024-09-16T01:44:37.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.883+0000 7f661159a640 1 -- 172.21.15.88:0/391444589 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65fc003520 con 0x7f6614145bb0 2024-09-16T01:44:37.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.883+0000 7f661159a640 1 -- 172.21.15.88:0/391444589 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1050075139 0 0) 0x7f66140d06a0 con 0x7f661413f400 2024-09-16T01:44:37.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.883+0000 7f661159a640 1 -- 172.21.15.88:0/391444589 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66140d03c0 con 0x7f661413f400 2024-09-16T01:44:37.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.883+0000 7f6612d9d640 1 -- 172.21.15.88:0/391444589 >> v1:172.21.15.154:6789/0 conn(0x7f661413f400 legacy=0x7f66140cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:37.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.883+0000 7f6612d9d640 1 -- 172.21.15.88:0/391444589 >> v1:172.21.15.88:6789/0 conn(0x7f6614145bb0 legacy=0x7f661406e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:37.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.883+0000 7f6612d9d640 1 -- 172.21.15.88:0/391444589 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66141434c0 con 0x7f661406e7c0 2024-09-16T01:44:37.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.884+0000 7f6612d9d640 1 --2- 172.21.15.88:0/391444589 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f661406e7c0 0x7f66141672a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f660c0097d0 tx=0x7f660c02a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2e65eb61911474e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:37.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.884+0000 7f661159a640 1 -- 172.21.15.88:0/391444589 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660c037070 con 0x7f661406e7c0 2024-09-16T01:44:37.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.884+0000 7f661159a640 1 -- 172.21.15.88:0/391444589 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f660c02ae50 con 0x7f661406e7c0 2024-09-16T01:44:37.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.884+0000 7f661159a640 1 -- 172.21.15.88:0/391444589 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660c032640 con 0x7f661406e7c0 2024-09-16T01:44:37.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.884+0000 7f6618da6640 1 -- 172.21.15.88:0/391444589 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f661406e7c0 msgr2=0x7f66141672a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:37.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.884+0000 7f6618da6640 1 --2- 172.21.15.88:0/391444589 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f661406e7c0 0x7f66141672a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f660c0097d0 tx=0x7f660c02a100 comp rx=0 tx=0).stop 2024-09-16T01:44:37.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.884+0000 7f6618da6640 1 -- 172.21.15.88:0/391444589 shutdown_connections 2024-09-16T01:44:37.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.884+0000 7f6618da6640 1 --2- 172.21.15.88:0/391444589 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f661406e7c0 0x7f66141672a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:37.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.884+0000 7f6618da6640 1 -- 172.21.15.88:0/391444589 >> 172.21.15.88:0/391444589 conn(0x7f661406bc10 msgr2=0x7f661406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:37.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.884+0000 7f6618da6640 1 -- 172.21.15.88:0/391444589 shutdown_connections 2024-09-16T01:44:37.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.884+0000 7f6618da6640 1 -- 172.21.15.88:0/391444589 wait complete. 2024-09-16T01:44:37.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.886+0000 7f6618da6640 1 Processor -- start 2024-09-16T01:44:37.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.886+0000 7f6618da6640 1 -- start start 2024-09-16T01:44:37.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.886+0000 7f6618da6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f661406e7c0 0x7f661415ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:37.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.887+0000 7f6618da6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f661413f400 0x7f6614140fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:37.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.887+0000 7f6618da6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6614145bb0 0x7f66140b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:37.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.887+0000 7f6618da6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6614144b70 con 0x7f6614145bb0 2024-09-16T01:44:37.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.887+0000 7f6618da6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f66141449f0 con 0x7f661413f400 2024-09-16T01:44:37.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.887+0000 7f6618da6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6614144cf0 con 0x7f661406e7c0 2024-09-16T01:44:37.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.887+0000 7f6612d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6614145bb0 0x7f66140b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:37.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.887+0000 7f6612d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6614145bb0 0x7f66140b52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46400/0 (socket says 172.21.15.88:46400) 2024-09-16T01:44:37.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.887+0000 7f6612d9d640 1 -- 172.21.15.88:0/2704144542 learned_addr learned my addr 172.21.15.88:0/2704144542 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:37.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.887+0000 7f661259c640 1 --2- 172.21.15.88:0/2704144542 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f661406e7c0 0x7f661415ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:37.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.887+0000 7f6611d9b640 1 --2- 172.21.15.88:0/2704144542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f661413f400 0x7f6614140fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:37.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.887+0000 7f6612d9d640 1 -- 172.21.15.88:0/2704144542 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f661406e7c0 msgr2=0x7f661415ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:37.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.887+0000 7f6612d9d640 1 --2- 172.21.15.88:0/2704144542 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f661406e7c0 0x7f661415ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:37.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.887+0000 7f6612d9d640 1 -- 172.21.15.88:0/2704144542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f661413f400 msgr2=0x7f6614140fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:37.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.887+0000 7f6612d9d640 1 --2- 172.21.15.88:0/2704144542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f661413f400 0x7f6614140fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:37.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.887+0000 7f6612d9d640 1 -- 172.21.15.88:0/2704144542 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f661415b150 con 0x7f6614145bb0 2024-09-16T01:44:37.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.887+0000 7f6611d9b640 1 --2- 172.21.15.88:0/2704144542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f661413f400 0x7f6614140fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:37.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.887+0000 7f6612d9d640 1 --2- 172.21.15.88:0/2704144542 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6614145bb0 0x7f66140b52f0 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7f660c02a140 tx=0x7f660c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:37.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.887+0000 7f6602ffd640 1 -- 172.21.15.88:0/2704144542 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660c041070 con 0x7f6614145bb0 2024-09-16T01:44:37.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.887+0000 7f6602ffd640 1 -- 172.21.15.88:0/2704144542 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f660c03c460 con 0x7f6614145bb0 2024-09-16T01:44:37.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.887+0000 7f6602ffd640 1 -- 172.21.15.88:0/2704144542 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660c037040 con 0x7f6614145bb0 2024-09-16T01:44:37.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.887+0000 7f6618da6640 1 -- 172.21.15.88:0/2704144542 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6614141980 con 0x7f6614145bb0 2024-09-16T01:44:37.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.887+0000 7f6618da6640 1 -- 172.21.15.88:0/2704144542 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6614141f40 con 0x7f6614145bb0 2024-09-16T01:44:37.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.888+0000 7f6618da6640 1 -- 172.21.15.88:0/2704144542 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f661406e0b0 con 0x7f6614145bb0 2024-09-16T01:44:37.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.889+0000 7f6602ffd640 1 -- 172.21.15.88:0/2704144542 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f660c03a070 con 0x7f6614145bb0 2024-09-16T01:44:37.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.889+0000 7f6602ffd640 1 --2- 172.21.15.88:0/2704144542 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f65f406c2b0 0x7f65f406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:37.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.889+0000 7f6602ffd640 1 -- 172.21.15.88:0/2704144542 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9178+0+0 (secure 0 0 0) 0x7f660c0ac360 con 0x7f6614145bb0 2024-09-16T01:44:37.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.889+0000 7f661259c640 1 --2- 172.21.15.88:0/2704144542 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f65f406c2b0 0x7f65f406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:37.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.890+0000 7f661259c640 1 --2- 172.21.15.88:0/2704144542 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f65f406c2b0 0x7f65f406e770 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6604001ea0 tx=0x7f6604009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:37.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:37.894+0000 7f6602ffd640 1 -- 172.21.15.88:0/2704144542 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f660c0796d0 con 0x7f6614145bb0 2024-09-16T01:44:38.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.090+0000 7f6618da6640 1 -- 172.21.15.88:0/2704144542 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6614062c40 con 0x7f6614145bb0 2024-09-16T01:44:38.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.091+0000 7f6602ffd640 1 -- 172.21.15.88:0/2704144542 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 318 v318) ==== 95+0+4393 (secure 0 0 0) 0x7f660c07d9d0 con 0x7f6614145bb0 2024-09-16T01:44:38.093 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:38.093 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":318,"btime":"2024-09-16T01:44:35:136561+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":10398,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2633617067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2633617067},{"type":"v1","addr":"172.21.15.88:6836","nonce":2633617067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":315}],"filesystems":[{"mdsmap":{"epoch":318,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:44:35.136557+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:44:38.094 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 318 2024-09-16T01:44:38.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.093+0000 7f6618da6640 1 -- 172.21.15.88:0/2704144542 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f65f406c2b0 msgr2=0x7f65f406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:38.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.093+0000 7f6618da6640 1 --2- 172.21.15.88:0/2704144542 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f65f406c2b0 0x7f65f406e770 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6604001ea0 tx=0x7f6604009040 comp rx=0 tx=0).stop 2024-09-16T01:44:38.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.093+0000 7f6618da6640 1 -- 172.21.15.88:0/2704144542 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6614145bb0 msgr2=0x7f66140b52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:38.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.093+0000 7f6618da6640 1 --2- 172.21.15.88:0/2704144542 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6614145bb0 0x7f66140b52f0 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7f660c02a140 tx=0x7f660c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:44:38.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.093+0000 7f6618da6640 1 -- 172.21.15.88:0/2704144542 shutdown_connections 2024-09-16T01:44:38.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.093+0000 7f6618da6640 1 --2- 172.21.15.88:0/2704144542 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f65f406c2b0 0x7f65f406e770 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:38.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.093+0000 7f6618da6640 1 --2- 172.21.15.88:0/2704144542 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6614145bb0 0x7f66140b52f0 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:38.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.093+0000 7f6618da6640 1 --2- 172.21.15.88:0/2704144542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f661413f400 0x7f6614140fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:38.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.093+0000 7f6618da6640 1 --2- 172.21.15.88:0/2704144542 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f661406e7c0 0x7f661415ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:38.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.093+0000 7f6618da6640 1 -- 172.21.15.88:0/2704144542 >> 172.21.15.88:0/2704144542 conn(0x7f661406bc10 msgr2=0x7f661406bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:38.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.093+0000 7f6618da6640 1 -- 172.21.15.88:0/2704144542 shutdown_connections 2024-09-16T01:44:38.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.093+0000 7f6618da6640 1 -- 172.21.15.88:0/2704144542 wait complete. 2024-09-16T01:44:38.106 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-16T01:44:38.106 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:44:38.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.202+0000 7f7c0ce47640 1 Processor -- start 2024-09-16T01:44:38.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.202+0000 7f7c0ce47640 1 -- start start 2024-09-16T01:44:38.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.202+0000 7f7c0ce47640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c0815c430 0x7f7c081607f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:38.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.202+0000 7f7c0ce47640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7c0806e150 0x7f7c0806e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:38.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.202+0000 7f7c0ce47640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c080d0960 con 0x7f7c08062990 2024-09-16T01:44:38.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.202+0000 7f7c0ce47640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7c080d0180 con 0x7f7c0806e150 2024-09-16T01:44:38.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.202+0000 7f7c0ce47640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7c080d03a0 con 0x7f7c0815c430 2024-09-16T01:44:38.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.203+0000 7f7c06d9d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7c0806e150 0x7f7c0806e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:38.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.203+0000 7f7c0659c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c0815c430 0x7f7c081607f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:38.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.203+0000 7f7c06d9d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7c0806e150 0x7f7c0806e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:59854/0 (socket says 172.21.15.88:59854) 2024-09-16T01:44:38.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.203+0000 7f7bfffff640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f7c08062990 0x7f7c080cfa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:43028/0 (socket says 172.21.15.88:43028) 2024-09-16T01:44:38.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.203+0000 7f7c06d9d640 1 -- 172.21.15.88:0/434294073 learned_addr learned my addr 172.21.15.88:0/434294073 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:38.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.203+0000 7f7c0659c640 1 -- 172.21.15.88:0/434294073 >> v2:172.21.15.154:3300/0 conn(0x7f7c0806e150 msgr2=0x7f7c0806e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:38.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.203+0000 7f7c0659c640 1 --2- 172.21.15.88:0/434294073 >> v2:172.21.15.154:3300/0 conn(0x7f7c0806e150 0x7f7c0806e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:38.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.203+0000 7f7c0659c640 1 -- 172.21.15.88:0/434294073 >> v1:172.21.15.154:6789/0 conn(0x7f7c08062990 legacy=0x7f7c080cfa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:38.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.203+0000 7f7c05d9b640 1 -- 172.21.15.88:0/434294073 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1310879063 0 0) 0x7f7c080d0960 con 0x7f7c08062990 2024-09-16T01:44:38.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.203+0000 7f7c0659c640 1 -- 172.21.15.88:0/434294073 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c080d0680 con 0x7f7c0815c430 2024-09-16T01:44:38.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.203+0000 7f7c06d9d640 1 --2- 172.21.15.88:0/434294073 >> v2:172.21.15.154:3300/0 conn(0x7f7c0806e150 0x7f7c0806e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:44:38.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.204+0000 7f7c0659c640 1 --2- 172.21.15.88:0/434294073 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c0815c430 0x7f7c081607f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7bf80097a0 tx=0x7f7bf802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b7a1438e9a56e80 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:38.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.204+0000 7f7c05d9b640 1 -- 172.21.15.88:0/434294073 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf8037070 con 0x7f7c0815c430 2024-09-16T01:44:38.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.204+0000 7f7c05d9b640 1 -- 172.21.15.88:0/434294073 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bf8032440 con 0x7f7c0815c430 2024-09-16T01:44:38.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.204+0000 7f7c05d9b640 1 -- 172.21.15.88:0/434294073 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf8032810 con 0x7f7c0815c430 2024-09-16T01:44:38.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.205+0000 7f7c0ce47640 1 -- 172.21.15.88:0/434294073 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c0815c430 msgr2=0x7f7c081607f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:38.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.205+0000 7f7c0ce47640 1 --2- 172.21.15.88:0/434294073 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c0815c430 0x7f7c081607f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7bf80097a0 tx=0x7f7bf802a180 comp rx=0 tx=0).stop 2024-09-16T01:44:38.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.205+0000 7f7c0ce47640 1 -- 172.21.15.88:0/434294073 shutdown_connections 2024-09-16T01:44:38.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.205+0000 7f7c0ce47640 1 --2- 172.21.15.88:0/434294073 >> v2:172.21.15.154:3300/0 conn(0x7f7c0806e150 0x7f7c0806e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:38.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.205+0000 7f7c0ce47640 1 --2- 172.21.15.88:0/434294073 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c0815c430 0x7f7c081607f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:38.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.205+0000 7f7c0ce47640 1 -- 172.21.15.88:0/434294073 >> 172.21.15.88:0/434294073 conn(0x7f7c0806bc10 msgr2=0x7f7c0806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:38.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.205+0000 7f7c0ce47640 1 -- 172.21.15.88:0/434294073 shutdown_connections 2024-09-16T01:44:38.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.205+0000 7f7c0ce47640 1 -- 172.21.15.88:0/434294073 wait complete. 2024-09-16T01:44:38.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.206+0000 7f7c0ce47640 1 Processor -- start 2024-09-16T01:44:38.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.206+0000 7f7c0ce47640 1 -- start start 2024-09-16T01:44:38.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.207+0000 7f7c0ce47640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c08062990 0x7f7c081f5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:38.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.207+0000 7f7c0ce47640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c0806e150 0x7f7c08154990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:38.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.207+0000 7f7c0ce47640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c0815c430 0x7f7c081529f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:38.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.207+0000 7f7c0ce47640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7c08160d30 con 0x7f7c08062990 2024-09-16T01:44:38.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.207+0000 7f7c0ce47640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7c081602d0 con 0x7f7c0815c430 2024-09-16T01:44:38.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.207+0000 7f7c0ce47640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7c0815c810 con 0x7f7c0806e150 2024-09-16T01:44:38.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.207+0000 7f7c0659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c08062990 0x7f7c081f5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:38.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.207+0000 7f7c0659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c08062990 0x7f7c081f5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46404/0 (socket says 172.21.15.88:46404) 2024-09-16T01:44:38.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.207+0000 7f7c0659c640 1 -- 172.21.15.88:0/956376771 learned_addr learned my addr 172.21.15.88:0/956376771 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:38.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.207+0000 7f7c06d9d640 1 --2- 172.21.15.88:0/956376771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c0815c430 0x7f7c081529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:38.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.207+0000 7f7bfffff640 1 --2- 172.21.15.88:0/956376771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c0806e150 0x7f7c08154990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:38.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.208+0000 7f7c0659c640 1 -- 172.21.15.88:0/956376771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c0806e150 msgr2=0x7f7c08154990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:38.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.208+0000 7f7c0659c640 1 --2- 172.21.15.88:0/956376771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c0806e150 0x7f7c08154990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:38.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.208+0000 7f7c0659c640 1 -- 172.21.15.88:0/956376771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c0815c430 msgr2=0x7f7c081529f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:38.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.208+0000 7f7c0659c640 1 --2- 172.21.15.88:0/956376771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c0815c430 0x7f7c081529f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:38.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.208+0000 7f7c0659c640 1 -- 172.21.15.88:0/956376771 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c08158390 con 0x7f7c08062990 2024-09-16T01:44:38.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.208+0000 7f7c0659c640 1 --2- 172.21.15.88:0/956376771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c08062990 0x7f7c081f5ee0 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7f7bf80097a0 tx=0x7f7bf80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:38.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.208+0000 7f7bfeffd640 1 -- 172.21.15.88:0/956376771 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf8037040 con 0x7f7c08062990 2024-09-16T01:44:38.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.208+0000 7f7bfeffd640 1 -- 172.21.15.88:0/956376771 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bf8004230 con 0x7f7c08062990 2024-09-16T01:44:38.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.208+0000 7f7c0ce47640 1 -- 172.21.15.88:0/956376771 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c08158670 con 0x7f7c08062990 2024-09-16T01:44:38.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.208+0000 7f7bfeffd640 1 -- 172.21.15.88:0/956376771 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf8003610 con 0x7f7c08062990 2024-09-16T01:44:38.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.208+0000 7f7c0ce47640 1 -- 172.21.15.88:0/956376771 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c08158c30 con 0x7f7c08062990 2024-09-16T01:44:38.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.209+0000 7f7c0ce47640 1 -- 172.21.15.88:0/956376771 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c081607f0 con 0x7f7c08062990 2024-09-16T01:44:38.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.210+0000 7f7bfeffd640 1 -- 172.21.15.88:0/956376771 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7bf80037b0 con 0x7f7c08062990 2024-09-16T01:44:38.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.210+0000 7f7bfeffd640 1 --2- 172.21.15.88:0/956376771 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7be806c2b0 0x7f7be806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:38.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.210+0000 7f7bfeffd640 1 -- 172.21.15.88:0/956376771 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9178+0+0 (secure 0 0 0) 0x7f7bf80ad300 con 0x7f7c08062990 2024-09-16T01:44:38.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.210+0000 7f7bfffff640 1 --2- 172.21.15.88:0/956376771 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7be806c2b0 0x7f7be806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:38.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.210+0000 7f7bfffff640 1 --2- 172.21.15.88:0/956376771 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7be806c2b0 0x7f7be806e770 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f7c0815bb50 tx=0x7f7bf0007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:38.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.214+0000 7f7bfeffd640 1 -- 172.21.15.88:0/956376771 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7bf807a7f0 con 0x7f7c08062990 2024-09-16T01:44:38.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.413+0000 7f7c0ce47640 1 -- 172.21.15.88:0/956376771 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c080aff70 con 0x7f7c08062990 2024-09-16T01:44:38.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.413+0000 7f7bfeffd640 1 -- 172.21.15.88:0/956376771 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 318 v318) ==== 95+0+4393 (secure 0 0 0) 0x7f7bf807eaf0 con 0x7f7c08062990 2024-09-16T01:44:38.416 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:38.416 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":318,"btime":"2024-09-16T01:44:35:136561+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":10398,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2633617067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2633617067},{"type":"v1","addr":"172.21.15.88:6836","nonce":2633617067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":315}],"filesystems":[{"mdsmap":{"epoch":318,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:44:35.136557+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:44:38.416 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 318 2024-09-16T01:44:38.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.415+0000 7f7c0ce47640 1 -- 172.21.15.88:0/956376771 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7be806c2b0 msgr2=0x7f7be806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:38.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.415+0000 7f7c0ce47640 1 --2- 172.21.15.88:0/956376771 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7be806c2b0 0x7f7be806e770 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f7c0815bb50 tx=0x7f7bf0007d50 comp rx=0 tx=0).stop 2024-09-16T01:44:38.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.415+0000 7f7c0ce47640 1 -- 172.21.15.88:0/956376771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c08062990 msgr2=0x7f7c081f5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:38.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.415+0000 7f7c0ce47640 1 --2- 172.21.15.88:0/956376771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c08062990 0x7f7c081f5ee0 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7f7bf80097a0 tx=0x7f7bf80046c0 comp rx=0 tx=0).stop 2024-09-16T01:44:38.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.416+0000 7f7c0ce47640 1 -- 172.21.15.88:0/956376771 shutdown_connections 2024-09-16T01:44:38.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.416+0000 7f7c0ce47640 1 --2- 172.21.15.88:0/956376771 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7be806c2b0 0x7f7be806e770 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:38.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.416+0000 7f7c0ce47640 1 --2- 172.21.15.88:0/956376771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c0815c430 0x7f7c081529f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:38.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.416+0000 7f7c0ce47640 1 --2- 172.21.15.88:0/956376771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7c0806e150 0x7f7c08154990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:38.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.416+0000 7f7c0ce47640 1 --2- 172.21.15.88:0/956376771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c08062990 0x7f7c081f5ee0 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:38.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.416+0000 7f7c0ce47640 1 -- 172.21.15.88:0/956376771 >> 172.21.15.88:0/956376771 conn(0x7f7c0806bc10 msgr2=0x7f7c0806bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:38.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.416+0000 7f7c0ce47640 1 -- 172.21.15.88:0/956376771 shutdown_connections 2024-09-16T01:44:38.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.416+0000 7f7c0ce47640 1 -- 172.21.15.88:0/956376771 wait complete. 2024-09-16T01:44:38.429 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt omap header on 1.00000000' led to DAMAGED state 2024-09-16T01:44:38.429 INFO:tasks.cephfs.test_damage:All 39 mutations had expected outcomes 2024-09-16T01:44:38.429 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T01:44:38.430 DEBUG:teuthology.orchestra.run.smithi088:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T01:44:38.470 DEBUG:teuthology.orchestra.run.smithi154:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T01:44:38.513 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T01:44:38.513 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T01:44:38.513 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:44:38.513 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:44:38.528 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:44:38.528 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T01:44:38.529 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:44:38.529 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:44:38.571 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056026,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1347 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T01:44:38.571 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T01:44:38.571 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:44:38.571 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:44:38.587 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:44:38.587 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T01:44:38.587 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:44:38.587 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:44:38.630 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056026,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1347 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T01:44:38.631 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:44:38.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.727+0000 7fd539eba640 1 Processor -- start 2024-09-16T01:44:38.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.728+0000 7fd539eba640 1 -- start start 2024-09-16T01:44:38.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.728+0000 7fd539eba640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd53406e910 0x7fd5340735d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:38.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.728+0000 7fd539eba640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd534073b10 0x7fd5340b4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:38.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.728+0000 7fd539eba640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd53415eb10 con 0x7fd53415c100 2024-09-16T01:44:38.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.728+0000 7fd539eba640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd53415edd0 con 0x7fd53406e910 2024-09-16T01:44:38.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.728+0000 7fd539eba640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd53415f0b0 con 0x7fd534073b10 2024-09-16T01:44:38.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.728+0000 7fd52affd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd53406e910 0x7fd5340735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:38.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.729+0000 7fd52affd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd53406e910 0x7fd5340735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46418/0 (socket says 172.21.15.88:46418) 2024-09-16T01:44:38.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.729+0000 7fd52affd640 1 -- 172.21.15.88:0/2719303654 learned_addr learned my addr 172.21.15.88:0/2719303654 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:38.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.729+0000 7fd533fff640 1 --2- 172.21.15.88:0/2719303654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd534073b10 0x7fd5340b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:38.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.729+0000 7fd52affd640 1 -- 172.21.15.88:0/2719303654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd534073b10 msgr2=0x7fd5340b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:38.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.729+0000 7fd52affd640 1 --2- 172.21.15.88:0/2719303654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd534073b10 0x7fd5340b4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:38.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.729+0000 7fd52affd640 1 -- 172.21.15.88:0/2719303654 >> v1:172.21.15.154:6789/0 conn(0x7fd53415c100 legacy=0x7fd53406e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:38.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.729+0000 7fd52affd640 1 -- 172.21.15.88:0/2719303654 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd53415f390 con 0x7fd53406e910 2024-09-16T01:44:38.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.729+0000 7fd52affd640 1 --2- 172.21.15.88:0/2719303654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd53406e910 0x7fd5340735d0 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7fd51c009870 tx=0x7fd51c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46e87e1c77be5912 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:38.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.730+0000 7fd532ffd640 1 -- 172.21.15.88:0/2719303654 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51c037070 con 0x7fd53406e910 2024-09-16T01:44:38.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.730+0000 7fd532ffd640 1 -- 172.21.15.88:0/2719303654 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd51c032440 con 0x7fd53406e910 2024-09-16T01:44:38.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.730+0000 7fd532ffd640 1 -- 172.21.15.88:0/2719303654 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51c032810 con 0x7fd53406e910 2024-09-16T01:44:38.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.730+0000 7fd539eba640 1 -- 172.21.15.88:0/2719303654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd53406e910 msgr2=0x7fd5340735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:38.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.730+0000 7fd539eba640 1 --2- 172.21.15.88:0/2719303654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd53406e910 0x7fd5340735d0 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7fd51c009870 tx=0x7fd51c02a270 comp rx=0 tx=0).stop 2024-09-16T01:44:38.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.730+0000 7fd539eba640 1 -- 172.21.15.88:0/2719303654 shutdown_connections 2024-09-16T01:44:38.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.730+0000 7fd539eba640 1 --2- 172.21.15.88:0/2719303654 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd534073b10 0x7fd5340b4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:38.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.730+0000 7fd539eba640 1 --2- 172.21.15.88:0/2719303654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd53406e910 0x7fd5340735d0 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:38.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.730+0000 7fd539eba640 1 -- 172.21.15.88:0/2719303654 >> 172.21.15.88:0/2719303654 conn(0x7fd5340b6ec0 msgr2=0x7fd5340b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:38.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.730+0000 7fd539eba640 1 -- 172.21.15.88:0/2719303654 shutdown_connections 2024-09-16T01:44:38.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.730+0000 7fd539eba640 1 -- 172.21.15.88:0/2719303654 wait complete. 2024-09-16T01:44:38.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.731+0000 7fd539eba640 1 Processor -- start 2024-09-16T01:44:38.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.732+0000 7fd539eba640 1 -- start start 2024-09-16T01:44:38.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.732+0000 7fd539eba640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd53406e910 0x7fd5340cd570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:38.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.732+0000 7fd539eba640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd534073b10 0x7fd534146db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:38.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.732+0000 7fd539eba640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd53415c100 0x7fd5340b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:38.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.732+0000 7fd539eba640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd534073fc0 con 0x7fd53415c100 2024-09-16T01:44:38.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.732+0000 7fd539eba640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd53415c4e0 con 0x7fd534073b10 2024-09-16T01:44:38.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.732+0000 7fd539eba640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd534074140 con 0x7fd53406e910 2024-09-16T01:44:38.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.732+0000 7fd533fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd53415c100 0x7fd5340b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:38.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.732+0000 7fd533fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd53415c100 0x7fd5340b6130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46420/0 (socket says 172.21.15.88:46420) 2024-09-16T01:44:38.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.732+0000 7fd533fff640 1 -- 172.21.15.88:0/2872396604 learned_addr learned my addr 172.21.15.88:0/2872396604 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:38.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.732+0000 7fd52affd640 1 --2- 172.21.15.88:0/2872396604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd534073b10 0x7fd534146db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:38.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.732+0000 7fd5337fe640 1 --2- 172.21.15.88:0/2872396604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd53406e910 0x7fd5340cd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:38.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.733+0000 7fd533fff640 1 -- 172.21.15.88:0/2872396604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd53406e910 msgr2=0x7fd5340cd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:38.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.733+0000 7fd533fff640 1 --2- 172.21.15.88:0/2872396604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd53406e910 0x7fd5340cd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:38.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.733+0000 7fd533fff640 1 -- 172.21.15.88:0/2872396604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd534073b10 msgr2=0x7fd534146db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:38.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.733+0000 7fd533fff640 1 --2- 172.21.15.88:0/2872396604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd534073b10 0x7fd534146db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:38.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.733+0000 7fd533fff640 1 -- 172.21.15.88:0/2872396604 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5340b6670 con 0x7fd53415c100 2024-09-16T01:44:38.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.733+0000 7fd5337fe640 1 --2- 172.21.15.88:0/2872396604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd53406e910 0x7fd5340cd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:38.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.733+0000 7fd533fff640 1 --2- 172.21.15.88:0/2872396604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd53415c100 0x7fd5340b6130 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7fd52c007e40 tx=0x7fd52c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:38.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.733+0000 7fd530ff9640 1 -- 172.21.15.88:0/2872396604 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52c014620 con 0x7fd53415c100 2024-09-16T01:44:38.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.733+0000 7fd530ff9640 1 -- 172.21.15.88:0/2872396604 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd52c019070 con 0x7fd53415c100 2024-09-16T01:44:38.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.733+0000 7fd539eba640 1 -- 172.21.15.88:0/2872396604 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5340b4db0 con 0x7fd53415c100 2024-09-16T01:44:38.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.733+0000 7fd530ff9640 1 -- 172.21.15.88:0/2872396604 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52c014a20 con 0x7fd53415c100 2024-09-16T01:44:38.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.733+0000 7fd539eba640 1 -- 172.21.15.88:0/2872396604 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5340b5370 con 0x7fd53415c100 2024-09-16T01:44:38.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.734+0000 7fd539eba640 1 -- 172.21.15.88:0/2872396604 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5340b2350 con 0x7fd53415c100 2024-09-16T01:44:38.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.735+0000 7fd530ff9640 1 -- 172.21.15.88:0/2872396604 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd52c014bc0 con 0x7fd53415c100 2024-09-16T01:44:38.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.735+0000 7fd530ff9640 1 --2- 172.21.15.88:0/2872396604 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd51406c2b0 0x7fd51406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:38.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.735+0000 7fd5337fe640 1 --2- 172.21.15.88:0/2872396604 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd51406c2b0 0x7fd51406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:38.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.735+0000 7fd530ff9640 1 -- 172.21.15.88:0/2872396604 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9178+0+0 (secure 0 0 0) 0x7fd52c08cba0 con 0x7fd53415c100 2024-09-16T01:44:38.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.736+0000 7fd5337fe640 1 --2- 172.21.15.88:0/2872396604 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd51406c2b0 0x7fd51406e770 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fd524001660 tx=0x7fd524009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:38.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.740+0000 7fd530ff9640 1 -- 172.21.15.88:0/2872396604 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd52c05a420 con 0x7fd53415c100 2024-09-16T01:44:38.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.938+0000 7fd539eba640 1 -- 172.21.15.88:0/2872396604 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5340b2070 con 0x7fd53415c100 2024-09-16T01:44:38.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.939+0000 7fd530ff9640 1 -- 172.21.15.88:0/2872396604 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 318 v318) ==== 95+0+4393 (secure 0 0 0) 0x7fd52c059d20 con 0x7fd53415c100 2024-09-16T01:44:38.941 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:38.941 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":318,"btime":"2024-09-16T01:44:35:136561+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":10398,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2633617067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2633617067},{"type":"v1","addr":"172.21.15.88:6836","nonce":2633617067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":315}],"filesystems":[{"mdsmap":{"epoch":318,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:44:35.136557+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:44:38.942 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 318 2024-09-16T01:44:38.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.941+0000 7fd539eba640 1 -- 172.21.15.88:0/2872396604 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd51406c2b0 msgr2=0x7fd51406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:38.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.941+0000 7fd539eba640 1 --2- 172.21.15.88:0/2872396604 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd51406c2b0 0x7fd51406e770 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fd524001660 tx=0x7fd524009040 comp rx=0 tx=0).stop 2024-09-16T01:44:38.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.941+0000 7fd539eba640 1 -- 172.21.15.88:0/2872396604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd53415c100 msgr2=0x7fd5340b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:38.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.941+0000 7fd539eba640 1 --2- 172.21.15.88:0/2872396604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd53415c100 0x7fd5340b6130 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7fd52c007e40 tx=0x7fd52c00e4b0 comp rx=0 tx=0).stop 2024-09-16T01:44:38.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.941+0000 7fd539eba640 1 -- 172.21.15.88:0/2872396604 shutdown_connections 2024-09-16T01:44:38.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.941+0000 7fd539eba640 1 --2- 172.21.15.88:0/2872396604 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd51406c2b0 0x7fd51406e770 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:38.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.941+0000 7fd539eba640 1 --2- 172.21.15.88:0/2872396604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd53415c100 0x7fd5340b6130 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:38.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.941+0000 7fd539eba640 1 --2- 172.21.15.88:0/2872396604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd534073b10 0x7fd534146db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:38.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.941+0000 7fd539eba640 1 --2- 172.21.15.88:0/2872396604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd53406e910 0x7fd5340cd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:38.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.941+0000 7fd539eba640 1 -- 172.21.15.88:0/2872396604 >> 172.21.15.88:0/2872396604 conn(0x7fd5340b6ec0 msgr2=0x7fd53415d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:38.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.941+0000 7fd539eba640 1 -- 172.21.15.88:0/2872396604 shutdown_connections 2024-09-16T01:44:38.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:38.941+0000 7fd539eba640 1 -- 172.21.15.88:0/2872396604 wait complete. 2024-09-16T01:44:38.954 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:44:39.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.050+0000 7f3baaa27640 1 Processor -- start 2024-09-16T01:44:39.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.050+0000 7f3baaa27640 1 -- start start 2024-09-16T01:44:39.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.051+0000 7f3baaa27640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3ba40b4920 0x7f3ba415b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:39.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.051+0000 7f3baaa27640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ba40ce4f0 con 0x7f3ba40cd780 2024-09-16T01:44:39.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.051+0000 7f3baaa27640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ba40ce7d0 con 0x7f3ba40b4d00 2024-09-16T01:44:39.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.051+0000 7f3baaa27640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f3ba40cf100 con 0x7f3ba40b4920 2024-09-16T01:44:39.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.051+0000 7f3ba3fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3ba40b4920 0x7f3ba415b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:39.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.051+0000 7f3ba3fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3ba40b4920 0x7f3ba415b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46436/0 (socket says 172.21.15.88:46436) 2024-09-16T01:44:39.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.051+0000 7f3ba8fc4640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3ba40cd780 0x7f3ba40cdb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52022/0 (socket says 172.21.15.88:52022) 2024-09-16T01:44:39.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.051+0000 7f3ba3fff640 1 -- 172.21.15.88:0/4144922854 learned_addr learned my addr 172.21.15.88:0/4144922854 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:39.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.051+0000 7f3ba3fff640 1 -- 172.21.15.88:0/4144922854 >> v1:172.21.15.154:6789/0 conn(0x7f3ba40b4d00 legacy=0x7f3ba40b5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:39.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.051+0000 7f3ba3fff640 1 -- 172.21.15.88:0/4144922854 >> v1:172.21.15.88:6789/0 conn(0x7f3ba40cd780 legacy=0x7f3ba40cdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:39.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.051+0000 7f3ba3fff640 1 -- 172.21.15.88:0/4144922854 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ba40ceab0 con 0x7f3ba40b4920 2024-09-16T01:44:39.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.051+0000 7f3ba2ffd640 1 -- 172.21.15.88:0/4144922854 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 920877431 0 0) 0x7f3ba40ce4f0 con 0x7f3ba40cd780 2024-09-16T01:44:39.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.052+0000 7f3ba3fff640 1 --2- 172.21.15.88:0/4144922854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ba40b4920 0x7f3ba415b7d0 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7f3ba40cf280 tx=0x7f3b9402a2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abf08b2dcaeb85 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:39.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.052+0000 7f3ba2ffd640 1 -- 172.21.15.88:0/4144922854 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b94037070 con 0x7f3ba40b4920 2024-09-16T01:44:39.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.052+0000 7f3ba2ffd640 1 -- 172.21.15.88:0/4144922854 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b94032640 con 0x7f3ba40b4920 2024-09-16T01:44:39.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.052+0000 7f3ba2ffd640 1 -- 172.21.15.88:0/4144922854 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b94032a10 con 0x7f3ba40b4920 2024-09-16T01:44:39.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.053+0000 7f3baaa27640 1 -- 172.21.15.88:0/4144922854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ba40b4920 msgr2=0x7f3ba415b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:39.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.053+0000 7f3baaa27640 1 --2- 172.21.15.88:0/4144922854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ba40b4920 0x7f3ba415b7d0 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7f3ba40cf280 tx=0x7f3b9402a2e0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.053+0000 7f3baaa27640 1 -- 172.21.15.88:0/4144922854 shutdown_connections 2024-09-16T01:44:39.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.053+0000 7f3baaa27640 1 --2- 172.21.15.88:0/4144922854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ba40b4920 0x7f3ba415b7d0 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.053+0000 7f3baaa27640 1 -- 172.21.15.88:0/4144922854 >> 172.21.15.88:0/4144922854 conn(0x7f3ba406bc10 msgr2=0x7f3ba406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:39.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.053+0000 7f3baaa27640 1 -- 172.21.15.88:0/4144922854 shutdown_connections 2024-09-16T01:44:39.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.053+0000 7f3baaa27640 1 -- 172.21.15.88:0/4144922854 wait complete. 2024-09-16T01:44:39.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.054+0000 7f3baaa27640 1 Processor -- start 2024-09-16T01:44:39.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.054+0000 7f3baaa27640 1 -- start start 2024-09-16T01:44:39.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.055+0000 7f3baaa27640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ba40b4920 0x7f3ba415a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:39.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.055+0000 7f3baaa27640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ba40b4d00 0x7f3ba40c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:39.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.055+0000 7f3baaa27640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ba40cd780 0x7f3ba4144e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:39.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.055+0000 7f3baaa27640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba415bd10 con 0x7f3ba40b4920 2024-09-16T01:44:39.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.055+0000 7f3baaa27640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3ba40ce2a0 con 0x7f3ba40b4d00 2024-09-16T01:44:39.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.055+0000 7f3baaa27640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3ba40cf590 con 0x7f3ba40cd780 2024-09-16T01:44:39.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.055+0000 7f3ba3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ba40b4920 0x7f3ba415a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:39.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.055+0000 7f3ba3fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ba40b4920 0x7f3ba415a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46446/0 (socket says 172.21.15.88:46446) 2024-09-16T01:44:39.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.055+0000 7f3ba3fff640 1 -- 172.21.15.88:0/2799053178 learned_addr learned my addr 172.21.15.88:0/2799053178 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:39.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.055+0000 7f3ba8fc4640 1 --2- 172.21.15.88:0/2799053178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ba40cd780 0x7f3ba4144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:39.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.055+0000 7f3ba37fe640 1 --2- 172.21.15.88:0/2799053178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ba40b4d00 0x7f3ba40c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:39.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.055+0000 7f3ba3fff640 1 -- 172.21.15.88:0/2799053178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ba40cd780 msgr2=0x7f3ba4144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:39.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.055+0000 7f3ba3fff640 1 --2- 172.21.15.88:0/2799053178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ba40cd780 0x7f3ba4144e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.055+0000 7f3ba3fff640 1 -- 172.21.15.88:0/2799053178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ba40b4d00 msgr2=0x7f3ba40c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:39.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.055+0000 7f3ba3fff640 1 --2- 172.21.15.88:0/2799053178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ba40b4d00 0x7f3ba40c66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.055+0000 7f3ba3fff640 1 -- 172.21.15.88:0/2799053178 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ba40c6c30 con 0x7f3ba40b4920 2024-09-16T01:44:39.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.055+0000 7f3ba3fff640 1 --2- 172.21.15.88:0/2799053178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ba40b4920 0x7f3ba415a760 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f3b94000c00 tx=0x7f3b940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:39.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.055+0000 7f3ba0ff9640 1 -- 172.21.15.88:0/2799053178 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b94041020 con 0x7f3ba40b4920 2024-09-16T01:44:39.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.056+0000 7f3ba0ff9640 1 -- 172.21.15.88:0/2799053178 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b94030970 con 0x7f3ba40b4920 2024-09-16T01:44:39.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.056+0000 7f3baaa27640 1 -- 172.21.15.88:0/2799053178 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ba40c6ed0 con 0x7f3ba40b4920 2024-09-16T01:44:39.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.056+0000 7f3ba0ff9640 1 -- 172.21.15.88:0/2799053178 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b94037040 con 0x7f3ba40b4920 2024-09-16T01:44:39.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.056+0000 7f3baaa27640 1 -- 172.21.15.88:0/2799053178 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ba40c7490 con 0x7f3ba40b4920 2024-09-16T01:44:39.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.056+0000 7f3baaa27640 1 -- 172.21.15.88:0/2799053178 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ba415b7d0 con 0x7f3ba40b4920 2024-09-16T01:44:39.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.057+0000 7f3ba0ff9640 1 -- 172.21.15.88:0/2799053178 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3b94044020 con 0x7f3ba40b4920 2024-09-16T01:44:39.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.057+0000 7f3ba0ff9640 1 --2- 172.21.15.88:0/2799053178 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3b9006c2b0 0x7f3b9006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:39.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.057+0000 7f3ba0ff9640 1 -- 172.21.15.88:0/2799053178 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9178+0+0 (secure 0 0 0) 0x7f3b940aca80 con 0x7f3ba40b4920 2024-09-16T01:44:39.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.057+0000 7f3ba37fe640 1 --2- 172.21.15.88:0/2799053178 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3b9006c2b0 0x7f3b9006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:39.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.058+0000 7f3ba37fe640 1 --2- 172.21.15.88:0/2799053178 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3b9006c2b0 0x7f3b9006e770 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4155050 tx=0x7f3b98009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:39.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.062+0000 7f3ba0ff9640 1 -- 172.21.15.88:0/2799053178 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3b9404b030 con 0x7f3ba40b4920 2024-09-16T01:44:39.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.265+0000 7f3baaa27640 1 -- 172.21.15.88:0/2799053178 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ba4071530 con 0x7f3ba40b4920 2024-09-16T01:44:39.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.265+0000 7f3ba0ff9640 1 -- 172.21.15.88:0/2799053178 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 318 v318) ==== 95+0+4393 (secure 0 0 0) 0x7f3b94079e50 con 0x7f3ba40b4920 2024-09-16T01:44:39.268 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:39.268 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":318,"btime":"2024-09-16T01:44:35:136561+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":10398,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2633617067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2633617067},{"type":"v1","addr":"172.21.15.88:6836","nonce":2633617067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":315}],"filesystems":[{"mdsmap":{"epoch":318,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:44:35.136557+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:44:39.268 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 318 2024-09-16T01:44:39.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.267+0000 7f3baaa27640 1 -- 172.21.15.88:0/2799053178 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3b9006c2b0 msgr2=0x7f3b9006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:39.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.267+0000 7f3baaa27640 1 --2- 172.21.15.88:0/2799053178 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3b9006c2b0 0x7f3b9006e770 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4155050 tx=0x7f3b98009040 comp rx=0 tx=0).stop 2024-09-16T01:44:39.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.267+0000 7f3baaa27640 1 -- 172.21.15.88:0/2799053178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ba40b4920 msgr2=0x7f3ba415a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:39.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.267+0000 7f3baaa27640 1 --2- 172.21.15.88:0/2799053178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ba40b4920 0x7f3ba415a760 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7f3b94000c00 tx=0x7f3b940046c0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.267+0000 7f3baaa27640 1 -- 172.21.15.88:0/2799053178 shutdown_connections 2024-09-16T01:44:39.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.267+0000 7f3baaa27640 1 --2- 172.21.15.88:0/2799053178 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3b9006c2b0 0x7f3b9006e770 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.267+0000 7f3baaa27640 1 --2- 172.21.15.88:0/2799053178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ba40cd780 0x7f3ba4144e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.267+0000 7f3baaa27640 1 --2- 172.21.15.88:0/2799053178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ba40b4d00 0x7f3ba40c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.267+0000 7f3baaa27640 1 --2- 172.21.15.88:0/2799053178 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3ba40b4920 0x7f3ba415a760 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.267+0000 7f3baaa27640 1 -- 172.21.15.88:0/2799053178 >> 172.21.15.88:0/2799053178 conn(0x7f3ba406bc10 msgr2=0x7f3ba40cf920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:39.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.267+0000 7f3baaa27640 1 -- 172.21.15.88:0/2799053178 shutdown_connections 2024-09-16T01:44:39.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.267+0000 7f3baaa27640 1 -- 172.21.15.88:0/2799053178 wait complete. 2024-09-16T01:44:39.281 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:44:39.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.376+0000 7fad6e477640 1 Processor -- start 2024-09-16T01:44:39.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.376+0000 7fad6e477640 1 -- start start 2024-09-16T01:44:39.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.377+0000 7fad6e477640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fad6815e560 0x7fad6806e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:39.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.377+0000 7fad6e477640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad68072220 0x7fad680ae890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:39.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.377+0000 7fad6e477640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad680b5220 con 0x7fad68073130 2024-09-16T01:44:39.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.377+0000 7fad6e477640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fad680b54c0 con 0x7fad6815e560 2024-09-16T01:44:39.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.377+0000 7fad6e477640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fad680b57a0 con 0x7fad68072220 2024-09-16T01:44:39.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.377+0000 7fad677fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad68072220 0x7fad680ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:39.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.377+0000 7fad67fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fad6815e560 0x7fad6806e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:39.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.377+0000 7fad6ca14640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fad68073130 0x7fad680b6280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:43064/0 (socket says 172.21.15.88:43064) 2024-09-16T01:44:39.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.377+0000 7fad6ca14640 1 -- 172.21.15.88:0/2365448258 learned_addr learned my addr 172.21.15.88:0/2365448258 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:39.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.377+0000 7fad677fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad68072220 0x7fad680ae890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:47182/0 (socket says 172.21.15.88:47182) 2024-09-16T01:44:39.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.378+0000 7fad67fff640 1 -- 172.21.15.88:0/2365448258 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad68072220 msgr2=0x7fad680ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:39.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.378+0000 7fad67fff640 1 --2- 172.21.15.88:0/2365448258 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad68072220 0x7fad680ae890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.378+0000 7fad67fff640 1 -- 172.21.15.88:0/2365448258 >> v1:172.21.15.154:6789/0 conn(0x7fad68073130 legacy=0x7fad680b6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:39.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.378+0000 7fad67fff640 1 -- 172.21.15.88:0/2365448258 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad680b5a80 con 0x7fad6815e560 2024-09-16T01:44:39.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.378+0000 7fad677fe640 1 --2- 172.21.15.88:0/2365448258 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad68072220 0x7fad680ae890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:44:39.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.379+0000 7fad67fff640 1 --2- 172.21.15.88:0/2365448258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad6815e560 0x7fad6806e780 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fad580097a0 tx=0x7fad5802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a659149fab6304d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:39.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.379+0000 7fad66ffd640 1 -- 172.21.15.88:0/2365448258 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad58037070 con 0x7fad6815e560 2024-09-16T01:44:39.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.379+0000 7fad66ffd640 1 -- 172.21.15.88:0/2365448258 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad58032440 con 0x7fad6815e560 2024-09-16T01:44:39.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.379+0000 7fad66ffd640 1 -- 172.21.15.88:0/2365448258 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad58032810 con 0x7fad6815e560 2024-09-16T01:44:39.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.379+0000 7fad6e477640 1 -- 172.21.15.88:0/2365448258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad6815e560 msgr2=0x7fad6806e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:39.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.379+0000 7fad6e477640 1 --2- 172.21.15.88:0/2365448258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad6815e560 0x7fad6806e780 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fad580097a0 tx=0x7fad5802a180 comp rx=0 tx=0).stop 2024-09-16T01:44:39.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.379+0000 7fad6e477640 1 -- 172.21.15.88:0/2365448258 shutdown_connections 2024-09-16T01:44:39.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.379+0000 7fad6e477640 1 --2- 172.21.15.88:0/2365448258 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad68072220 0x7fad680ae890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.379+0000 7fad6e477640 1 --2- 172.21.15.88:0/2365448258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad6815e560 0x7fad6806e780 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.379+0000 7fad6e477640 1 -- 172.21.15.88:0/2365448258 >> 172.21.15.88:0/2365448258 conn(0x7fad6806bc10 msgr2=0x7fad6806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:39.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.379+0000 7fad6e477640 1 -- 172.21.15.88:0/2365448258 shutdown_connections 2024-09-16T01:44:39.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.379+0000 7fad6e477640 1 -- 172.21.15.88:0/2365448258 wait complete. 2024-09-16T01:44:39.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.380+0000 7fad6e477640 1 Processor -- start 2024-09-16T01:44:39.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.381+0000 7fad6e477640 1 -- start start 2024-09-16T01:44:39.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.381+0000 7fad6e477640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad68072220 0x7fad6815b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:39.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.381+0000 7fad6e477640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad68073130 0x7fad6814a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:39.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.381+0000 7fad6e477640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad6815e560 0x7fad6815d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:39.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.381+0000 7fad6e477640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fad6815e940 con 0x7fad6815e560 2024-09-16T01:44:39.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.381+0000 7fad6e477640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fad68072660 con 0x7fad68073130 2024-09-16T01:44:39.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.381+0000 7fad6e477640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fad6815eac0 con 0x7fad68072220 2024-09-16T01:44:39.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.381+0000 7fad6ca14640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad6815e560 0x7fad6815d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:39.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.382+0000 7fad6ca14640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad6815e560 0x7fad6815d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46448/0 (socket says 172.21.15.88:46448) 2024-09-16T01:44:39.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.382+0000 7fad6ca14640 1 -- 172.21.15.88:0/1263017074 learned_addr learned my addr 172.21.15.88:0/1263017074 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:39.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.382+0000 7fad67fff640 1 --2- 172.21.15.88:0/1263017074 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad68072220 0x7fad6815b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:39.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.382+0000 7fad677fe640 1 --2- 172.21.15.88:0/1263017074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad68073130 0x7fad6814a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:39.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.382+0000 7fad6ca14640 1 -- 172.21.15.88:0/1263017074 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad68072220 msgr2=0x7fad6815b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:39.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.382+0000 7fad6ca14640 1 --2- 172.21.15.88:0/1263017074 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad68072220 0x7fad6815b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.382+0000 7fad6ca14640 1 -- 172.21.15.88:0/1263017074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad68073130 msgr2=0x7fad6814a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:39.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.382+0000 7fad6ca14640 1 --2- 172.21.15.88:0/1263017074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad68073130 0x7fad6814a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.382+0000 7fad6ca14640 1 -- 172.21.15.88:0/1263017074 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad68156510 con 0x7fad6815e560 2024-09-16T01:44:39.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.382+0000 7fad6ca14640 1 --2- 172.21.15.88:0/1263017074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad6815e560 0x7fad6815d060 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7fad6000cae0 tx=0x7fad6000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:39.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.382+0000 7fad64ff9640 1 -- 172.21.15.88:0/1263017074 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad60017070 con 0x7fad6815e560 2024-09-16T01:44:39.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.382+0000 7fad64ff9640 1 -- 172.21.15.88:0/1263017074 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad6000ac40 con 0x7fad6815e560 2024-09-16T01:44:39.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.382+0000 7fad6e477640 1 -- 172.21.15.88:0/1263017074 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad6815bfc0 con 0x7fad6815e560 2024-09-16T01:44:39.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.382+0000 7fad64ff9640 1 -- 172.21.15.88:0/1263017074 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad60012400 con 0x7fad6815e560 2024-09-16T01:44:39.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.382+0000 7fad6e477640 1 -- 172.21.15.88:0/1263017074 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad6815c580 con 0x7fad6815e560 2024-09-16T01:44:39.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.383+0000 7fad6e477640 1 -- 172.21.15.88:0/1263017074 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad6806e780 con 0x7fad6815e560 2024-09-16T01:44:39.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.384+0000 7fad64ff9640 1 -- 172.21.15.88:0/1263017074 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fad6001a070 con 0x7fad6815e560 2024-09-16T01:44:39.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.384+0000 7fad64ff9640 1 --2- 172.21.15.88:0/1263017074 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fad5406c2e0 0x7fad5406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:39.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.384+0000 7fad64ff9640 1 -- 172.21.15.88:0/1263017074 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9178+0+0 (secure 0 0 0) 0x7fad6008beb0 con 0x7fad6815e560 2024-09-16T01:44:39.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.384+0000 7fad67fff640 1 --2- 172.21.15.88:0/1263017074 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fad5406c2e0 0x7fad5406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:39.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.384+0000 7fad67fff640 1 --2- 172.21.15.88:0/1263017074 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fad5406c2e0 0x7fad5406e7a0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fad58034000 tx=0x7fad5802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:39.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.388+0000 7fad64ff9640 1 -- 172.21.15.88:0/1263017074 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fad600591f0 con 0x7fad6815e560 2024-09-16T01:44:39.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.555+0000 7fad6e477640 1 -- 172.21.15.88:0/1263017074 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fad680cda40 con 0x7fad6815e560 2024-09-16T01:44:39.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.556+0000 7fad64ff9640 1 -- 172.21.15.88:0/1263017074 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) ==== 74+0+21934 (secure 0 0 0) 0x7fad6005d4f0 con 0x7fad6815e560 2024-09-16T01:44:39.559 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:39.559 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":127,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:44:35.136317+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:40:01.941882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-16T01:40:02.564641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.3b","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:6837/4270197461":"2024-09-17T01:44:35.136295+0000","172.21.15.88:6835/4270197461":"2024-09-17T01:44:35.136295+0000","172.21.15.88:6836/2111557063":"2024-09-17T01:44:28.087623+0000","172.21.15.88:6834/2111557063":"2024-09-17T01:44:28.087623+0000","172.21.15.88:6836/1912564918":"2024-09-17T01:44:16.634793+0000","172.21.15.88:6834/1912564918":"2024-09-17T01:44:16.634793+0000","172.21.15.88:6836/3078464816":"2024-09-17T01:44:01.948759+0000","172.21.15.88:6834/3892583353":"2024-09-17T01:43:56.641946+0000","172.21.15.88:6836/4135629195":"2024-09-17T01:43:45.866139+0000","172.21.15.88:6836/3892583353":"2024-09-17T01:43:56.641946+0000","172.21.15.88:6834/4135629195":"2024-09-17T01:43:45.866139+0000","172.21.15.88:6836/3121519215":"2024-09-17T01:43:30.669208+0000","172.21.15.88:6834/3121519215":"2024-09-17T01:43:30.669208+0000","172.21.15.88:6836/2555776364":"2024-09-17T01:43:25.635976+0000","172.21.15.88:6837/1372908065":"2024-09-17T01:43:12.443265+0000","172.21.15.88:6835/1372908065":"2024-09-17T01:43:12.443265+0000","172.21.15.88:6836/650399848":"2024-09-17T01:43:08.279879+0000","172.21.15.88:6835/1255135754":"2024-09-17T01:41:41.905258+0000","172.21.15.88:6836/3728363359":"2024-09-17T01:42:05.412133+0000","172.21.15.88:6837/293764774":"2024-09-17T01:41:16.203758+0000","172.21.15.88:6837/1255135754":"2024-09-17T01:41:41.905258+0000","172.21.15.88:6836/2918194823":"2024-09-17T01:41:29.275696+0000","172.21.15.88:6837/3336935653":"2024-09-17T01:41:44.295023+0000","172.21.15.88:6836/981209627":"2024-09-17T01:41:51.268495+0000","172.21.15.88:6835/293764774":"2024-09-17T01:41:16.203758+0000","172.21.15.88:6836/2680708900":"2024-09-17T01:42:41.629904+0000","172.21.15.88:6834/2957965650":"2024-09-17T01:41:12.181485+0000","172.21.15.88:6835/3336935653":"2024-09-17T01:41:44.295023+0000","172.21.15.88:6837/2285699282":"2024-09-17T01:42:29.541113+0000","172.21.15.88:6836/1162318664":"2024-09-17T01:41:23.257870+0000","172.21.15.88:6836/128058689":"2024-09-17T01:41:10.163743+0000","172.21.15.88:6834/981209627":"2024-09-17T01:41:51.268495+0000","172.21.15.88:6834/128058689":"2024-09-17T01:41:10.163743+0000","172.21.15.88:6836/1042715824":"2024-09-17T01:40:59.080515+0000","172.21.15.88:6836/2957965650":"2024-09-17T01:41:12.181485+0000","172.21.15.88:6835/2970185561":"2024-09-17T01:40:38.255774+0000","172.21.15.88:6834/3078464816":"2024-09-17T01:44:01.948759+0000","172.21.15.88:6834/1162318664":"2024-09-17T01:41:23.257870+0000","172.21.15.88:6836/1289932173":"2024-09-17T01:40:42.399399+0000","172.21.15.88:6835/3356432718":"2024-09-17T01:40:32.344252+0000","172.21.15.88:6836/1099469537":"2024-09-17T01:44:22.054752+0000","172.21.15.88:6834/2555776364":"2024-09-17T01:43:25.635976+0000","172.21.15.88:6837/2970185561":"2024-09-17T01:40:38.255774+0000","172.21.15.88:6835/2733458327":"2024-09-17T01:40:27.314712+0000","172.21.15.88:6837/3356432718":"2024-09-17T01:40:32.344252+0000","172.21.15.88:6834/2918194823":"2024-09-17T01:41:29.275696+0000","172.21.15.88:6834/4144876777":"2024-09-17T01:43:00.350808+0000","172.21.15.88:6835/2408579939":"2024-09-17T01:40:46.434492+0000","172.21.15.88:6835/3163343083":"2024-09-17T01:42:01.375396+0000","172.21.15.88:6835/4222250252":"2024-09-17T01:42:45.673956+0000","172.21.15.88:6835/4162274767":"2024-09-17T01:40:11.522091+0000","172.21.15.88:6835/2477051134":"2024-09-17T01:43:04.380783+0000","172.21.15.88:6834/1289932173":"2024-09-17T01:40:42.399399+0000","172.21.15.88:6837/2733458327":"2024-09-17T01:40:27.314712+0000","172.21.15.88:6834/2550498835":"2024-09-17T01:41:57.358287+0000","172.21.15.88:6836/2550498835":"2024-09-17T01:41:57.358287+0000","172.21.15.88:6835/2285699282":"2024-09-17T01:42:29.541113+0000","172.21.15.88:6837/3163343083":"2024-09-17T01:42:01.375396+0000","172.21.15.88:6834/3728363359":"2024-09-17T01:42:05.412133+0000","172.21.15.88:6834/1099469537":"2024-09-17T01:44:22.054752+0000","172.21.15.88:6835/880314974":"2024-09-17T01:42:21.485748+0000","172.21.15.88:6835/1936493048":"2024-09-17T01:42:19.133197+0000","172.21.15.88:6837/1936493048":"2024-09-17T01:42:19.133197+0000","172.21.15.88:6837/880314974":"2024-09-17T01:42:21.485748+0000","172.21.15.88:6834/3374372758":"2024-09-17T01:42:25.506299+0000","172.21.15.88:6836/3374372758":"2024-09-17T01:42:25.506299+0000","172.21.15.88:6836/201188184":"2024-09-17T01:42:33.276104+0000","172.21.15.88:6837/1990650099":"2024-09-17T01:42:37.599896+0000","172.21.15.88:6834/201188184":"2024-09-17T01:42:33.276104+0000","172.21.15.88:6835/1990650099":"2024-09-17T01:42:37.599896+0000","172.21.15.88:6837/4162274767":"2024-09-17T01:40:11.522091+0000","172.21.15.88:6837/2408579939":"2024-09-17T01:40:46.434492+0000","172.21.15.88:6834/2680708900":"2024-09-17T01:42:41.629904+0000","172.21.15.88:6837/4222250252":"2024-09-17T01:42:45.673956+0000","172.21.15.88:6834/1042715824":"2024-09-17T01:40:59.080515+0000","172.21.15.88:6834/4065900771":"2024-09-17T01:42:58.329389+0000","172.21.15.88:6836/4065900771":"2024-09-17T01:42:58.329389+0000","172.21.15.88:6836/4144876777":"2024-09-17T01:43:00.350808+0000","172.21.15.88:6837/2477051134":"2024-09-17T01:43:04.380783+0000","172.21.15.88:6834/650399848":"2024-09-17T01:43:08.279879+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:44:39.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.558+0000 7fad6e477640 1 -- 172.21.15.88:0/1263017074 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fad5406c2e0 msgr2=0x7fad5406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:39.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.558+0000 7fad6e477640 1 --2- 172.21.15.88:0/1263017074 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fad5406c2e0 0x7fad5406e7a0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fad58034000 tx=0x7fad5802e000 comp rx=0 tx=0).stop 2024-09-16T01:44:39.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.558+0000 7fad6e477640 1 -- 172.21.15.88:0/1263017074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad6815e560 msgr2=0x7fad6815d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:39.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.558+0000 7fad6e477640 1 --2- 172.21.15.88:0/1263017074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad6815e560 0x7fad6815d060 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7fad6000cae0 tx=0x7fad6000a060 comp rx=0 tx=0).stop 2024-09-16T01:44:39.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.559+0000 7fad6e477640 1 -- 172.21.15.88:0/1263017074 shutdown_connections 2024-09-16T01:44:39.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.559+0000 7fad6e477640 1 --2- 172.21.15.88:0/1263017074 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fad5406c2e0 0x7fad5406e7a0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.559+0000 7fad6e477640 1 --2- 172.21.15.88:0/1263017074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fad6815e560 0x7fad6815d060 unknown :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.559+0000 7fad6e477640 1 --2- 172.21.15.88:0/1263017074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad68073130 0x7fad6814a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.559+0000 7fad6e477640 1 --2- 172.21.15.88:0/1263017074 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fad68072220 0x7fad6815b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.559+0000 7fad6e477640 1 -- 172.21.15.88:0/1263017074 >> 172.21.15.88:0/1263017074 conn(0x7fad6806bc10 msgr2=0x7fad680b6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:39.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.559+0000 7fad6e477640 1 -- 172.21.15.88:0/1263017074 shutdown_connections 2024-09-16T01:44:39.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.559+0000 7fad6e477640 1 -- 172.21.15.88:0/1263017074 wait complete. 2024-09-16T01:44:39.572 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-16T01:44:39.573 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:44:39.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.669+0000 7fdb2d647640 1 Processor -- start 2024-09-16T01:44:39.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.669+0000 7fdb2d647640 1 -- start start 2024-09-16T01:44:39.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.670+0000 7fdb2d647640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdb280d3310 0x7fdb280d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:39.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.670+0000 7fdb2d647640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb2815ea40 con 0x7fdb280d1650 2024-09-16T01:44:39.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.670+0000 7fdb2d647640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb280d3870 con 0x7fdb2815e660 2024-09-16T01:44:39.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.670+0000 7fdb2d647640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fdb280d1ac0 con 0x7fdb280d3310 2024-09-16T01:44:39.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.670+0000 7fdb277fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fdb280d1650 0x7fdb280b7b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52028/0 (socket says 172.21.15.88:52028) 2024-09-16T01:44:39.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.670+0000 7fdb277fe640 1 -- 172.21.15.88:0/1472254771 learned_addr learned my addr 172.21.15.88:0/1472254771 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:39.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.671+0000 7fdb25ffb640 1 -- 172.21.15.88:0/1472254771 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 574400480 0 0) 0x7fdb2815ea40 con 0x7fdb280d1650 2024-09-16T01:44:39.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.671+0000 7fdb267fc640 1 --2- 172.21.15.88:0/1472254771 >> v2:172.21.15.154:3300/0 conn(0x7fdb280d3310 0x7fdb280d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:39.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.671+0000 7fdb25ffb640 1 -- 172.21.15.88:0/1472254771 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb14003520 con 0x7fdb280d1650 2024-09-16T01:44:39.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.671+0000 7fdb25ffb640 1 -- 172.21.15.88:0/1472254771 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2471626718 0 0) 0x7fdb14003520 con 0x7fdb280d1650 2024-09-16T01:44:39.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.671+0000 7fdb25ffb640 1 -- 172.21.15.88:0/1472254771 >> v2:172.21.15.154:3300/0 conn(0x7fdb280d3310 msgr2=0x7fdb280d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:39.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.671+0000 7fdb25ffb640 1 --2- 172.21.15.88:0/1472254771 >> v2:172.21.15.154:3300/0 conn(0x7fdb280d3310 0x7fdb280d1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.671+0000 7fdb25ffb640 1 -- 172.21.15.88:0/1472254771 >> v1:172.21.15.154:6789/0 conn(0x7fdb2815e660 legacy=0x7fdb280cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:39.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.671+0000 7fdb25ffb640 1 -- 172.21.15.88:0/1472254771 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb280d3b50 con 0x7fdb280d1650 2024-09-16T01:44:39.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.671+0000 7fdb25ffb640 1 -- 172.21.15.88:0/1472254771 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fdb20001670 con 0x7fdb280d1650 2024-09-16T01:44:39.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.671+0000 7fdb25ffb640 1 -- 172.21.15.88:0/1472254771 >> v1:172.21.15.88:6789/0 conn(0x7fdb280d1650 legacy=0x7fdb280b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:39.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.671+0000 7fdb25ffb640 1 --2- 172.21.15.88:0/1472254771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb140039b0 0x7fdb14003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:39.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.671+0000 7fdb25ffb640 1 --2- 172.21.15.88:0/1472254771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb14004800 0x7fdb14004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:39.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.671+0000 7fdb25ffb640 1 --2- 172.21.15.88:0/1472254771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb140051c0 0x7fdb1400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:39.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.671+0000 7fdb25ffb640 1 -- 172.21.15.88:0/1472254771 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdb280d1ac0 con 0x7fdb140051c0 2024-09-16T01:44:39.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.671+0000 7fdb25ffb640 1 -- 172.21.15.88:0/1472254771 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdb1400c060 con 0x7fdb140039b0 2024-09-16T01:44:39.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.671+0000 7fdb25ffb640 1 -- 172.21.15.88:0/1472254771 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdb1400c260 con 0x7fdb14004800 2024-09-16T01:44:39.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.671+0000 7fdb277fe640 1 --2- 172.21.15.88:0/1472254771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb140051c0 0x7fdb1400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:39.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.671+0000 7fdb26ffd640 1 --2- 172.21.15.88:0/1472254771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb140039b0 0x7fdb14003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:39.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.671+0000 7fdb267fc640 1 --2- 172.21.15.88:0/1472254771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb14004800 0x7fdb14004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:39.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.671+0000 7fdb277fe640 1 -- 172.21.15.88:0/1472254771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb14004800 msgr2=0x7fdb14004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:39.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.671+0000 7fdb277fe640 1 --2- 172.21.15.88:0/1472254771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb14004800 0x7fdb14004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.671+0000 7fdb277fe640 1 -- 172.21.15.88:0/1472254771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb140039b0 msgr2=0x7fdb14003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:39.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.671+0000 7fdb277fe640 1 --2- 172.21.15.88:0/1472254771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb140039b0 0x7fdb14003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.671+0000 7fdb277fe640 1 -- 172.21.15.88:0/1472254771 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb14003520 con 0x7fdb140051c0 2024-09-16T01:44:39.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.672+0000 7fdb277fe640 1 -- 172.21.15.88:0/1472254771 reap_dead start 2024-09-16T01:44:39.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.672+0000 7fdb277fe640 1 --2- 172.21.15.88:0/1472254771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb140051c0 0x7fdb1400b9c0 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7fdb2000c240 tx=0x7fdb14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9899ff486a284c88 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:39.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.672+0000 7fdb25ffb640 1 -- 172.21.15.88:0/1472254771 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb20007cf0 con 0x7fdb140051c0 2024-09-16T01:44:39.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.672+0000 7fdb25ffb640 1 -- 172.21.15.88:0/1472254771 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb200058d0 con 0x7fdb140051c0 2024-09-16T01:44:39.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.672+0000 7fdb25ffb640 1 -- 172.21.15.88:0/1472254771 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb20005ce0 con 0x7fdb140051c0 2024-09-16T01:44:39.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.673+0000 7fdb2d647640 1 -- 172.21.15.88:0/1472254771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb140051c0 msgr2=0x7fdb1400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:39.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.673+0000 7fdb2d647640 1 --2- 172.21.15.88:0/1472254771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb140051c0 0x7fdb1400b9c0 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7fdb2000c240 tx=0x7fdb14005830 comp rx=0 tx=0).stop 2024-09-16T01:44:39.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.673+0000 7fdb2d647640 1 -- 172.21.15.88:0/1472254771 shutdown_connections 2024-09-16T01:44:39.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.673+0000 7fdb2d647640 1 --2- 172.21.15.88:0/1472254771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb140051c0 0x7fdb1400b9c0 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.673+0000 7fdb2d647640 1 -- 172.21.15.88:0/1472254771 >> 172.21.15.88:0/1472254771 conn(0x7fdb280b6410 msgr2=0x7fdb280b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:39.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.673+0000 7fdb2d647640 1 -- 172.21.15.88:0/1472254771 shutdown_connections 2024-09-16T01:44:39.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.673+0000 7fdb2d647640 1 -- 172.21.15.88:0/1472254771 wait complete. 2024-09-16T01:44:39.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.674+0000 7fdb2d647640 1 Processor -- start 2024-09-16T01:44:39.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.674+0000 7fdb2d647640 1 -- start start 2024-09-16T01:44:39.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.675+0000 7fdb2d647640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb140051c0 0x7fdb280aba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:39.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.675+0000 7fdb2d647640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb280cd960 0x7fdb280b99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:39.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.675+0000 7fdb2d647640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb2815a870 0x7fdb2815acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:39.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.675+0000 7fdb2d647640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdb280d43f0 con 0x7fdb140051c0 2024-09-16T01:44:39.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.675+0000 7fdb2d647640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdb280d4110 con 0x7fdb280cd960 2024-09-16T01:44:39.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.675+0000 7fdb2d647640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdb280d1110 con 0x7fdb2815a870 2024-09-16T01:44:39.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.675+0000 7fdb26ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb140051c0 0x7fdb280aba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:39.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.675+0000 7fdb26ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb140051c0 0x7fdb280aba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46462/0 (socket says 172.21.15.88:46462) 2024-09-16T01:44:39.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.675+0000 7fdb26ffd640 1 -- 172.21.15.88:0/703974988 learned_addr learned my addr 172.21.15.88:0/703974988 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:39.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.675+0000 7fdb267fc640 1 --2- 172.21.15.88:0/703974988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb280cd960 0x7fdb280b99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:39.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.675+0000 7fdb277fe640 1 --2- 172.21.15.88:0/703974988 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb2815a870 0x7fdb2815acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:39.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.675+0000 7fdb26ffd640 1 -- 172.21.15.88:0/703974988 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb2815a870 msgr2=0x7fdb2815acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:39.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.675+0000 7fdb26ffd640 1 --2- 172.21.15.88:0/703974988 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb2815a870 0x7fdb2815acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.675+0000 7fdb26ffd640 1 -- 172.21.15.88:0/703974988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb280cd960 msgr2=0x7fdb280b99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:39.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.675+0000 7fdb26ffd640 1 --2- 172.21.15.88:0/703974988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb280cd960 0x7fdb280b99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.675+0000 7fdb26ffd640 1 -- 172.21.15.88:0/703974988 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb280ac160 con 0x7fdb140051c0 2024-09-16T01:44:39.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.675+0000 7fdb277fe640 1 --2- 172.21.15.88:0/703974988 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb2815a870 0x7fdb2815acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:39.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.675+0000 7fdb26ffd640 1 --2- 172.21.15.88:0/703974988 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb140051c0 0x7fdb280aba50 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7fdb1800f960 tx=0x7fdb1800fe20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:39.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.675+0000 7fdb1f7fe640 1 -- 172.21.15.88:0/703974988 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb18003b80 con 0x7fdb140051c0 2024-09-16T01:44:39.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.676+0000 7fdb1f7fe640 1 -- 172.21.15.88:0/703974988 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb18018070 con 0x7fdb140051c0 2024-09-16T01:44:39.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.676+0000 7fdb2d647640 1 -- 172.21.15.88:0/703974988 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb280ac440 con 0x7fdb140051c0 2024-09-16T01:44:39.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.676+0000 7fdb1f7fe640 1 -- 172.21.15.88:0/703974988 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb1800d7f0 con 0x7fdb140051c0 2024-09-16T01:44:39.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.676+0000 7fdb2d647640 1 -- 172.21.15.88:0/703974988 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb280aca00 con 0x7fdb140051c0 2024-09-16T01:44:39.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.677+0000 7fdb2d647640 1 -- 172.21.15.88:0/703974988 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb280cf730 con 0x7fdb140051c0 2024-09-16T01:44:39.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.677+0000 7fdb1f7fe640 1 -- 172.21.15.88:0/703974988 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fdb1800d990 con 0x7fdb140051c0 2024-09-16T01:44:39.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.678+0000 7fdb1f7fe640 1 --2- 172.21.15.88:0/703974988 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdb0806c2e0 0x7fdb0806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:39.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.678+0000 7fdb267fc640 1 --2- 172.21.15.88:0/703974988 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdb0806c2e0 0x7fdb0806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:39.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.678+0000 7fdb1f7fe640 1 -- 172.21.15.88:0/703974988 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9178+0+0 (secure 0 0 0) 0x7fdb1808c370 con 0x7fdb140051c0 2024-09-16T01:44:39.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.678+0000 7fdb267fc640 1 --2- 172.21.15.88:0/703974988 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdb0806c2e0 0x7fdb0806e7a0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fdb281f7650 tx=0x7fdb10008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:39.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.682+0000 7fdb1f7fe640 1 -- 172.21.15.88:0/703974988 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fdb1801a310 con 0x7fdb140051c0 2024-09-16T01:44:39.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.880+0000 7fdb2d647640 1 -- 172.21.15.88:0/703974988 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb280b6cc0 con 0x7fdb140051c0 2024-09-16T01:44:39.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.881+0000 7fdb1f7fe640 1 -- 172.21.15.88:0/703974988 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 318 v318) ==== 95+0+4393 (secure 0 0 0) 0x7fdb18059d00 con 0x7fdb140051c0 2024-09-16T01:44:39.883 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:39.883 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":318,"btime":"2024-09-16T01:44:35:136561+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":10398,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2633617067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2633617067},{"type":"v1","addr":"172.21.15.88:6836","nonce":2633617067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":315}],"filesystems":[{"mdsmap":{"epoch":318,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:44:35.136557+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:44:39.883 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 318 2024-09-16T01:44:39.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.883+0000 7fdb2d647640 1 -- 172.21.15.88:0/703974988 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdb0806c2e0 msgr2=0x7fdb0806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:39.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.883+0000 7fdb2d647640 1 --2- 172.21.15.88:0/703974988 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdb0806c2e0 0x7fdb0806e7a0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fdb281f7650 tx=0x7fdb10008040 comp rx=0 tx=0).stop 2024-09-16T01:44:39.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.883+0000 7fdb2d647640 1 -- 172.21.15.88:0/703974988 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb140051c0 msgr2=0x7fdb280aba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:39.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.883+0000 7fdb2d647640 1 --2- 172.21.15.88:0/703974988 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb140051c0 0x7fdb280aba50 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7fdb1800f960 tx=0x7fdb1800fe20 comp rx=0 tx=0).stop 2024-09-16T01:44:39.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.883+0000 7fdb2d647640 1 -- 172.21.15.88:0/703974988 shutdown_connections 2024-09-16T01:44:39.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.883+0000 7fdb2d647640 1 --2- 172.21.15.88:0/703974988 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdb0806c2e0 0x7fdb0806e7a0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.883+0000 7fdb2d647640 1 --2- 172.21.15.88:0/703974988 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb2815a870 0x7fdb2815acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.883+0000 7fdb2d647640 1 --2- 172.21.15.88:0/703974988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb280cd960 0x7fdb280b99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.883+0000 7fdb2d647640 1 --2- 172.21.15.88:0/703974988 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdb140051c0 0x7fdb280aba50 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:39.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.883+0000 7fdb2d647640 1 -- 172.21.15.88:0/703974988 >> 172.21.15.88:0/703974988 conn(0x7fdb280b6410 msgr2=0x7fdb2806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:39.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.883+0000 7fdb2d647640 1 -- 172.21.15.88:0/703974988 shutdown_connections 2024-09-16T01:44:39.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.883+0000 7fdb2d647640 1 -- 172.21.15.88:0/703974988 wait complete. 2024-09-16T01:44:39.890 INFO:tasks.ceph.mds.c.smithi088.stderr: -51> 2024-09-16T01:44:35.135+0000 7fcfeed27640 -1 mds.0.cache.dir(0x1) Corrupt fnode in dirfrag 0x1: Decoder at 'void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&)' v=3 cannot decode v=100 minimal_decoder=101: Malformed input [buffer:3] 2024-09-16T01:44:39.890 INFO:tasks.ceph.mds.c.smithi088.stderr: -48> 2024-09-16T01:44:35.135+0000 7fcfeed27640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x1 is fatal because it is a system directory for this rank 2024-09-16T01:44:39.891 INFO:tasks.ceph.mds.c.smithi088.stderr: -20> 2024-09-16T01:44:37.497+0000 7fcff2d2f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:44:39.898 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:44:39.916 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:44:39.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.995+0000 7f20f56d0640 1 Processor -- start 2024-09-16T01:44:39.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.996+0000 7f20f56d0640 1 -- start start 2024-09-16T01:44:39.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.996+0000 7f20f56d0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f20f0161820 0x7f20f0169350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:39.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.996+0000 7f20f56d0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20f015fcc0 con 0x7f20f00728e0 2024-09-16T01:44:39.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.996+0000 7f20f56d0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20f015f430 con 0x7f20f006e5c0 2024-09-16T01:44:39.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.996+0000 7f20f56d0640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f20f0160380 con 0x7f20f0161820 2024-09-16T01:44:39.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.996+0000 7f20ef7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f20f0161820 0x7f20f0169350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:40.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.997+0000 7f20ef7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f20f0161820 0x7f20f0169350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46494/0 (socket says 172.21.15.88:46494) 2024-09-16T01:44:40.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.997+0000 7f20ef7fe640 1 -- 172.21.15.88:0/3632716827 learned_addr learned my addr 172.21.15.88:0/3632716827 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:40.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.997+0000 7f20ee7fc640 1 --1- 172.21.15.88:0/3632716827 >> v1:172.21.15.88:6789/0 conn(0x7f20f00728e0 0x7f20f015ed20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:52052/0 (socket says 172.21.15.88:52052) 2024-09-16T01:44:40.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.997+0000 7f20ef7fe640 1 -- 172.21.15.88:0/3632716827 >> v1:172.21.15.154:6789/0 conn(0x7f20f006e5c0 legacy=0x7f20f006e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:40.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.997+0000 7f20ef7fe640 1 -- 172.21.15.88:0/3632716827 >> v1:172.21.15.88:6789/0 conn(0x7f20f00728e0 legacy=0x7f20f015ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:40.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.997+0000 7f20ef7fe640 1 -- 172.21.15.88:0/3632716827 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20f015f700 con 0x7f20f0161820 2024-09-16T01:44:40.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.997+0000 7f20edffb640 1 -- 172.21.15.88:0/3632716827 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3073430551 0 0) 0x7f20f015fcc0 con 0x7f20f00728e0 2024-09-16T01:44:40.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.997+0000 7f20ef7fe640 1 --2- 172.21.15.88:0/3632716827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f0161820 0x7f20f0169350 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f20e0009870 tx=0x7f20e002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee6b2bd1c26aa5c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:40.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.998+0000 7f20edffb640 1 -- 172.21.15.88:0/3632716827 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e0037070 con 0x7f20f0161820 2024-09-16T01:44:40.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.998+0000 7f20edffb640 1 -- 172.21.15.88:0/3632716827 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20e0032440 con 0x7f20f0161820 2024-09-16T01:44:40.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.998+0000 7f20edffb640 1 -- 172.21.15.88:0/3632716827 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e0032810 con 0x7f20f0161820 2024-09-16T01:44:40.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.998+0000 7f20f56d0640 1 -- 172.21.15.88:0/3632716827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f0161820 msgr2=0x7f20f0169350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:40.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.998+0000 7f20f56d0640 1 --2- 172.21.15.88:0/3632716827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f0161820 0x7f20f0169350 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f20e0009870 tx=0x7f20e002a270 comp rx=0 tx=0).stop 2024-09-16T01:44:40.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.998+0000 7f20f56d0640 1 -- 172.21.15.88:0/3632716827 shutdown_connections 2024-09-16T01:44:40.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.998+0000 7f20f56d0640 1 --2- 172.21.15.88:0/3632716827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f0161820 0x7f20f0169350 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:40.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.998+0000 7f20f56d0640 1 -- 172.21.15.88:0/3632716827 >> 172.21.15.88:0/3632716827 conn(0x7f20f006e1e0 msgr2=0x7f20f00b67c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:40.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.998+0000 7f20f56d0640 1 -- 172.21.15.88:0/3632716827 shutdown_connections 2024-09-16T01:44:40.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:39.998+0000 7f20f56d0640 1 -- 172.21.15.88:0/3632716827 wait complete. 2024-09-16T01:44:40.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.000+0000 7f20f56d0640 1 Processor -- start 2024-09-16T01:44:40.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.000+0000 7f20f56d0640 1 -- start start 2024-09-16T01:44:40.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.000+0000 7f20f56d0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20f006e5c0 0x7f20f00b6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:40.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.000+0000 7f20f56d0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f00728e0 0x7f20f01486c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:40.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.000+0000 7f20f56d0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20f0161820 0x7f20f0157440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:40.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.000+0000 7f20f56d0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f20f015c670 con 0x7f20f00728e0 2024-09-16T01:44:40.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.000+0000 7f20f56d0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f20f0160010 con 0x7f20f0161820 2024-09-16T01:44:40.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.000+0000 7f20f56d0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f20f0169890 con 0x7f20f006e5c0 2024-09-16T01:44:40.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.000+0000 7f20ee7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f00728e0 0x7f20f01486c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:40.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.000+0000 7f20ee7fc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f00728e0 0x7f20f01486c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:46502/0 (socket says 172.21.15.88:46502) 2024-09-16T01:44:40.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.000+0000 7f20ee7fc640 1 -- 172.21.15.88:0/3895183107 learned_addr learned my addr 172.21.15.88:0/3895183107 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:40.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.001+0000 7f20ef7fe640 1 --2- 172.21.15.88:0/3895183107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20f0161820 0x7f20f0157440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:40.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.001+0000 7f20eeffd640 1 --2- 172.21.15.88:0/3895183107 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20f006e5c0 0x7f20f00b6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:40.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.001+0000 7f20ee7fc640 1 -- 172.21.15.88:0/3895183107 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20f006e5c0 msgr2=0x7f20f00b6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:40.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.001+0000 7f20ee7fc640 1 --2- 172.21.15.88:0/3895183107 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20f006e5c0 0x7f20f00b6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:40.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.001+0000 7f20ee7fc640 1 -- 172.21.15.88:0/3895183107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20f0161820 msgr2=0x7f20f0157440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:40.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.001+0000 7f20ee7fc640 1 --2- 172.21.15.88:0/3895183107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20f0161820 0x7f20f0157440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:40.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.001+0000 7f20ee7fc640 1 -- 172.21.15.88:0/3895183107 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20f00b55d0 con 0x7f20f00728e0 2024-09-16T01:44:40.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.001+0000 7f20ef7fe640 1 --2- 172.21.15.88:0/3895183107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20f0161820 0x7f20f0157440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:40.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.001+0000 7f20ee7fc640 1 --2- 172.21.15.88:0/3895183107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f00728e0 0x7f20f01486c0 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f20e400c7e0 tx=0x7f20e400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:40.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.001+0000 7f20d77fe640 1 -- 172.21.15.88:0/3895183107 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e4018070 con 0x7f20f00728e0 2024-09-16T01:44:40.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.001+0000 7f20d77fe640 1 -- 172.21.15.88:0/3895183107 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20e400a640 con 0x7f20f00728e0 2024-09-16T01:44:40.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.001+0000 7f20f56d0640 1 -- 172.21.15.88:0/3895183107 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20f00b58b0 con 0x7f20f00728e0 2024-09-16T01:44:40.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.001+0000 7f20d77fe640 1 -- 172.21.15.88:0/3895183107 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e400aa10 con 0x7f20f00728e0 2024-09-16T01:44:40.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.001+0000 7f20f56d0640 1 -- 172.21.15.88:0/3895183107 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20f00b5e70 con 0x7f20f00728e0 2024-09-16T01:44:40.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.002+0000 7f20f56d0640 1 -- 172.21.15.88:0/3895183107 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20f00d0580 con 0x7f20f00728e0 2024-09-16T01:44:40.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.002+0000 7f20d77fe640 1 -- 172.21.15.88:0/3895183107 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f20e400abb0 con 0x7f20f00728e0 2024-09-16T01:44:40.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.003+0000 7f20d77fe640 1 --2- 172.21.15.88:0/3895183107 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f20dc06c2b0 0x7f20dc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:40.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.003+0000 7f20d77fe640 1 -- 172.21.15.88:0/3895183107 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9178+0+0 (secure 0 0 0) 0x7f20e408b9a0 con 0x7f20f00728e0 2024-09-16T01:44:40.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.003+0000 7f20eeffd640 1 --2- 172.21.15.88:0/3895183107 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f20dc06c2b0 0x7f20dc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:40.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.003+0000 7f20eeffd640 1 --2- 172.21.15.88:0/3895183107 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f20dc06c2b0 0x7f20dc06e770 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f20e8000f60 tx=0x7f20e8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:40.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.007+0000 7f20d77fe640 1 -- 172.21.15.88:0/3895183107 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f20f00d0580 con 0x7f20f00728e0 2024-09-16T01:44:40.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.209+0000 7f20f56d0640 1 -- 172.21.15.88:0/3895183107 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20f00b3c50 con 0x7f20f00728e0 2024-09-16T01:44:40.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.210+0000 7f20d77fe640 1 -- 172.21.15.88:0/3895183107 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 319 v319) ==== 95+0+5220 (secure 0 0 0) 0x7f20e40592b0 con 0x7f20f00728e0 2024-09-16T01:44:40.213 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:40.214 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":319,"btime":"2024-09-16T01:44:40:135399+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":10398,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2633617067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2633617067},{"type":"v1","addr":"172.21.15.88:6836","nonce":2633617067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":315},{"gid":10443,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/3726735780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":3726735780},{"type":"v1","addr":"172.21.15.88:6837","nonce":3726735780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":319}],"filesystems":[{"mdsmap":{"epoch":318,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:40:03.597796+0000","modified":"2024-09-16T01:44:35.136557+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-09-16T01:44:40.216 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 319 2024-09-16T01:44:40.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.212+0000 7f20f56d0640 1 -- 172.21.15.88:0/3895183107 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f20dc06c2b0 msgr2=0x7f20dc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:40.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.212+0000 7f20f56d0640 1 --2- 172.21.15.88:0/3895183107 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f20dc06c2b0 0x7f20dc06e770 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f20e8000f60 tx=0x7f20e8009040 comp rx=0 tx=0).stop 2024-09-16T01:44:40.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.212+0000 7f20f56d0640 1 -- 172.21.15.88:0/3895183107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f00728e0 msgr2=0x7f20f01486c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:40.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.212+0000 7f20f56d0640 1 --2- 172.21.15.88:0/3895183107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f00728e0 0x7f20f01486c0 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f20e400c7e0 tx=0x7f20e400cca0 comp rx=0 tx=0).stop 2024-09-16T01:44:40.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.212+0000 7f20f56d0640 1 -- 172.21.15.88:0/3895183107 shutdown_connections 2024-09-16T01:44:40.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.212+0000 7f20f56d0640 1 --2- 172.21.15.88:0/3895183107 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f20dc06c2b0 0x7f20dc06e770 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:40.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.212+0000 7f20f56d0640 1 --2- 172.21.15.88:0/3895183107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20f0161820 0x7f20f0157440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:40.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.212+0000 7f20f56d0640 1 --2- 172.21.15.88:0/3895183107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20f00728e0 0x7f20f01486c0 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:40.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.212+0000 7f20f56d0640 1 --2- 172.21.15.88:0/3895183107 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f20f006e5c0 0x7f20f00b6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:40.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.212+0000 7f20f56d0640 1 -- 172.21.15.88:0/3895183107 >> 172.21.15.88:0/3895183107 conn(0x7f20f006e1e0 msgr2=0x7f20f0159d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:40.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.212+0000 7f20f56d0640 1 -- 172.21.15.88:0/3895183107 shutdown_connections 2024-09-16T01:44:40.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.212+0000 7f20f56d0640 1 -- 172.21.15.88:0/3895183107 wait complete. 2024-09-16T01:44:40.225 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:44:40.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.361+0000 7fbd0154b640 1 Processor -- start 2024-09-16T01:44:40.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.361+0000 7fbd0154b640 1 -- start start 2024-09-16T01:44:40.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.362+0000 7fbd0154b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbcfc0b4d00 0x7fbcfc0b5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:40.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.362+0000 7fbd0154b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbcfc0cd780 0x7fbcfc0cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:40.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.362+0000 7fbd0154b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcfc0ce4f0 con 0x7fbcfc0b4920 2024-09-16T01:44:40.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.362+0000 7fbd0154b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbcfc0cf100 con 0x7fbcfc0b4d00 2024-09-16T01:44:40.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.362+0000 7fbd0154b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbcfc0ce7d0 con 0x7fbcfc0cd780 2024-09-16T01:44:40.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.362+0000 7fbcf27fc640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbcfc0b4d00 0x7fbcfc0b5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:40.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.362+0000 7fbcfb7fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbcfc0cd780 0x7fbcfc0cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:40.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.362+0000 7fbcfaffd640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fbcfc0b4920 0x7fbcfc15b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:44132/0 (socket says 172.21.15.88:44132) 2024-09-16T01:44:40.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.362+0000 7fbcf27fc640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbcfc0b4d00 0x7fbcfc0b5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:57242/0 (socket says 172.21.15.88:57242) 2024-09-16T01:44:40.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.362+0000 7fbcfaffd640 1 -- 172.21.15.88:0/658215310 learned_addr learned my addr 172.21.15.88:0/658215310 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:40.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.362+0000 7fbcfb7fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbcfc0cd780 0x7fbcfc0cdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:46468/0 (socket says 172.21.15.88:46468) 2024-09-16T01:44:40.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.362+0000 7fbcfb7fe640 1 -- 172.21.15.88:0/658215310 >> v2:172.21.15.154:3300/0 conn(0x7fbcfc0b4d00 msgr2=0x7fbcfc0b5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:40.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.362+0000 7fbcfb7fe640 1 --2- 172.21.15.88:0/658215310 >> v2:172.21.15.154:3300/0 conn(0x7fbcfc0b4d00 0x7fbcfc0b5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:40.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.362+0000 7fbcfb7fe640 1 -- 172.21.15.88:0/658215310 >> v1:172.21.15.154:6789/0 conn(0x7fbcfc0b4920 legacy=0x7fbcfc15b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:40.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.362+0000 7fbcfb7fe640 1 -- 172.21.15.88:0/658215310 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcfc0ceab0 con 0x7fbcfc0cd780 2024-09-16T01:44:40.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.363+0000 7fbcf27fc640 1 --2- 172.21.15.88:0/658215310 >> v2:172.21.15.154:3300/0 conn(0x7fbcfc0b4d00 0x7fbcfc0b5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:44:40.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.363+0000 7fbcfb7fe640 1 --2- 172.21.15.88:0/658215310 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbcfc0cd780 0x7fbcfc0cdbd0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4009d90 tx=0x7fbcf402a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7aa3365df8019736 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:40.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.363+0000 7fbcfa7fc640 1 -- 172.21.15.88:0/658215310 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf4007740 con 0x7fbcfc0cd780 2024-09-16T01:44:40.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.363+0000 7fbcfa7fc640 1 -- 172.21.15.88:0/658215310 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcf40078e0 con 0x7fbcfc0cd780 2024-09-16T01:44:40.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.364+0000 7fbcfa7fc640 1 -- 172.21.15.88:0/658215310 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf4007cb0 con 0x7fbcfc0cd780 2024-09-16T01:44:40.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.364+0000 7fbd0154b640 1 -- 172.21.15.88:0/658215310 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbcfc0cd780 msgr2=0x7fbcfc0cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:40.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.364+0000 7fbd0154b640 1 --2- 172.21.15.88:0/658215310 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbcfc0cd780 0x7fbcfc0cdbd0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4009d90 tx=0x7fbcf402a660 comp rx=0 tx=0).stop 2024-09-16T01:44:40.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.364+0000 7fbd0154b640 1 -- 172.21.15.88:0/658215310 shutdown_connections 2024-09-16T01:44:40.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.364+0000 7fbd0154b640 1 --2- 172.21.15.88:0/658215310 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbcfc0cd780 0x7fbcfc0cdbd0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:40.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.364+0000 7fbd0154b640 1 --2- 172.21.15.88:0/658215310 >> v2:172.21.15.154:3300/0 conn(0x7fbcfc0b4d00 0x7fbcfc0b5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:40.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.364+0000 7fbd0154b640 1 -- 172.21.15.88:0/658215310 >> 172.21.15.88:0/658215310 conn(0x7fbcfc06bc10 msgr2=0x7fbcfc06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:40.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.364+0000 7fbd0154b640 1 -- 172.21.15.88:0/658215310 shutdown_connections 2024-09-16T01:44:40.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.364+0000 7fbd0154b640 1 -- 172.21.15.88:0/658215310 wait complete. 2024-09-16T01:44:40.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.365+0000 7fbd0154b640 1 Processor -- start 2024-09-16T01:44:40.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.365+0000 7fbd0154b640 1 -- start start 2024-09-16T01:44:40.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.366+0000 7fbd0154b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbcfc0b4920 0x7fbcfc145fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:40.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.366+0000 7fbd0154b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbcfc0b4d00 0x7fbcfc1598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:40.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.366+0000 7fbd0154b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcfc0cd780 0x7fbcfc159e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:40.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.366+0000 7fbd0154b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbcfc0ce110 con 0x7fbcfc0cd780 2024-09-16T01:44:40.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.366+0000 7fbd0154b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbcfc0b56b0 con 0x7fbcfc0b4920 2024-09-16T01:44:40.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.366+0000 7fbd0154b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbcfc0ce290 con 0x7fbcfc0b4d00 2024-09-16T01:44:40.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.366+0000 7fbcfb7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcfc0cd780 0x7fbcfc159e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:40.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.366+0000 7fbcfb7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcfc0cd780 0x7fbcfc159e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41514/0 (socket says 172.21.15.88:41514) 2024-09-16T01:44:40.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.366+0000 7fbcfb7fe640 1 -- 172.21.15.88:0/3357342627 learned_addr learned my addr 172.21.15.88:0/3357342627 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:40.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.367+0000 7fbcfaffd640 1 --2- 172.21.15.88:0/3357342627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbcfc0b4920 0x7fbcfc145fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:40.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.367+0000 7fbcf27fc640 1 --2- 172.21.15.88:0/3357342627 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbcfc0b4d00 0x7fbcfc1598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:40.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.367+0000 7fbcfb7fe640 1 -- 172.21.15.88:0/3357342627 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbcfc0b4d00 msgr2=0x7fbcfc1598c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:40.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.367+0000 7fbcfb7fe640 1 --2- 172.21.15.88:0/3357342627 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbcfc0b4d00 0x7fbcfc1598c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:40.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.367+0000 7fbcfb7fe640 1 -- 172.21.15.88:0/3357342627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbcfc0b4920 msgr2=0x7fbcfc145fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:40.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.367+0000 7fbcfb7fe640 1 --2- 172.21.15.88:0/3357342627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbcfc0b4920 0x7fbcfc145fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:40.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.367+0000 7fbcfb7fe640 1 -- 172.21.15.88:0/3357342627 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcfc1590e0 con 0x7fbcfc0cd780 2024-09-16T01:44:40.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.367+0000 7fbcf27fc640 1 --2- 172.21.15.88:0/3357342627 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbcfc0b4d00 0x7fbcfc1598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:40.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.367+0000 7fbcfb7fe640 1 --2- 172.21.15.88:0/3357342627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcfc0cd780 0x7fbcfc159e00 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4009eb0 tx=0x7fbcf4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:40.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.367+0000 7fbcf3fff640 1 -- 172.21.15.88:0/3357342627 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf403f070 con 0x7fbcfc0cd780 2024-09-16T01:44:40.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.367+0000 7fbcf3fff640 1 -- 172.21.15.88:0/3357342627 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcf4031bc0 con 0x7fbcfc0cd780 2024-09-16T01:44:40.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.367+0000 7fbd0154b640 1 -- 172.21.15.88:0/3357342627 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcfc158040 con 0x7fbcfc0cd780 2024-09-16T01:44:40.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.367+0000 7fbcf3fff640 1 -- 172.21.15.88:0/3357342627 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf4009450 con 0x7fbcfc0cd780 2024-09-16T01:44:40.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.367+0000 7fbd0154b640 1 -- 172.21.15.88:0/3357342627 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcfc158600 con 0x7fbcfc0cd780 2024-09-16T01:44:40.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.368+0000 7fbd0154b640 1 -- 172.21.15.88:0/3357342627 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcfc0b5170 con 0x7fbcfc0cd780 2024-09-16T01:44:40.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.369+0000 7fbcf3fff640 1 -- 172.21.15.88:0/3357342627 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fbcf403b410 con 0x7fbcfc0cd780 2024-09-16T01:44:40.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.369+0000 7fbcf3fff640 1 --2- 172.21.15.88:0/3357342627 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbcdc06c2b0 0x7fbcdc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:40.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.369+0000 7fbcf3fff640 1 -- 172.21.15.88:0/3357342627 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9178+0+0 (secure 0 0 0) 0x7fbcf40ad770 con 0x7fbcfc0cd780 2024-09-16T01:44:40.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.369+0000 7fbcfaffd640 1 --2- 172.21.15.88:0/3357342627 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbcdc06c2b0 0x7fbcdc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:40.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.369+0000 7fbcfaffd640 1 --2- 172.21.15.88:0/3357342627 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbcdc06c2b0 0x7fbcdc06e770 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fbcec007fd0 tx=0x7fbcec007e80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:40.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.373+0000 7fbcf3fff640 1 -- 172.21.15.88:0/3357342627 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fbcf407b190 con 0x7fbcfc0cd780 2024-09-16T01:44:40.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.540+0000 7fbd0154b640 1 -- 172.21.15.88:0/3357342627 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbcfc15bbe0 con 0x7fbcfc0cd780 2024-09-16T01:44:40.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.541+0000 7fbcf3fff640 1 -- 172.21.15.88:0/3357342627 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) ==== 74+0+21934 (secure 0 0 0) 0x7fbcf407aa90 con 0x7fbcfc0cd780 2024-09-16T01:44:40.544 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:40.544 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":127,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:44:35.136317+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:40:01.941882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-16T01:40:02.564641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.3b","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:6837/4270197461":"2024-09-17T01:44:35.136295+0000","172.21.15.88:6835/4270197461":"2024-09-17T01:44:35.136295+0000","172.21.15.88:6836/2111557063":"2024-09-17T01:44:28.087623+0000","172.21.15.88:6834/2111557063":"2024-09-17T01:44:28.087623+0000","172.21.15.88:6836/1912564918":"2024-09-17T01:44:16.634793+0000","172.21.15.88:6834/1912564918":"2024-09-17T01:44:16.634793+0000","172.21.15.88:6836/3078464816":"2024-09-17T01:44:01.948759+0000","172.21.15.88:6834/3892583353":"2024-09-17T01:43:56.641946+0000","172.21.15.88:6836/4135629195":"2024-09-17T01:43:45.866139+0000","172.21.15.88:6836/3892583353":"2024-09-17T01:43:56.641946+0000","172.21.15.88:6834/4135629195":"2024-09-17T01:43:45.866139+0000","172.21.15.88:6836/3121519215":"2024-09-17T01:43:30.669208+0000","172.21.15.88:6834/3121519215":"2024-09-17T01:43:30.669208+0000","172.21.15.88:6836/2555776364":"2024-09-17T01:43:25.635976+0000","172.21.15.88:6837/1372908065":"2024-09-17T01:43:12.443265+0000","172.21.15.88:6835/1372908065":"2024-09-17T01:43:12.443265+0000","172.21.15.88:6836/650399848":"2024-09-17T01:43:08.279879+0000","172.21.15.88:6835/1255135754":"2024-09-17T01:41:41.905258+0000","172.21.15.88:6836/3728363359":"2024-09-17T01:42:05.412133+0000","172.21.15.88:6837/293764774":"2024-09-17T01:41:16.203758+0000","172.21.15.88:6837/1255135754":"2024-09-17T01:41:41.905258+0000","172.21.15.88:6836/2918194823":"2024-09-17T01:41:29.275696+0000","172.21.15.88:6837/3336935653":"2024-09-17T01:41:44.295023+0000","172.21.15.88:6836/981209627":"2024-09-17T01:41:51.268495+0000","172.21.15.88:6835/293764774":"2024-09-17T01:41:16.203758+0000","172.21.15.88:6836/2680708900":"2024-09-17T01:42:41.629904+0000","172.21.15.88:6834/2957965650":"2024-09-17T01:41:12.181485+0000","172.21.15.88:6835/3336935653":"2024-09-17T01:41:44.295023+0000","172.21.15.88:6837/2285699282":"2024-09-17T01:42:29.541113+0000","172.21.15.88:6836/1162318664":"2024-09-17T01:41:23.257870+0000","172.21.15.88:6836/128058689":"2024-09-17T01:41:10.163743+0000","172.21.15.88:6834/981209627":"2024-09-17T01:41:51.268495+0000","172.21.15.88:6834/128058689":"2024-09-17T01:41:10.163743+0000","172.21.15.88:6836/1042715824":"2024-09-17T01:40:59.080515+0000","172.21.15.88:6836/2957965650":"2024-09-17T01:41:12.181485+0000","172.21.15.88:6835/2970185561":"2024-09-17T01:40:38.255774+0000","172.21.15.88:6834/3078464816":"2024-09-17T01:44:01.948759+0000","172.21.15.88:6834/1162318664":"2024-09-17T01:41:23.257870+0000","172.21.15.88:6836/1289932173":"2024-09-17T01:40:42.399399+0000","172.21.15.88:6835/3356432718":"2024-09-17T01:40:32.344252+0000","172.21.15.88:6836/1099469537":"2024-09-17T01:44:22.054752+0000","172.21.15.88:6834/2555776364":"2024-09-17T01:43:25.635976+0000","172.21.15.88:6837/2970185561":"2024-09-17T01:40:38.255774+0000","172.21.15.88:6835/2733458327":"2024-09-17T01:40:27.314712+0000","172.21.15.88:6837/3356432718":"2024-09-17T01:40:32.344252+0000","172.21.15.88:6834/2918194823":"2024-09-17T01:41:29.275696+0000","172.21.15.88:6834/4144876777":"2024-09-17T01:43:00.350808+0000","172.21.15.88:6835/2408579939":"2024-09-17T01:40:46.434492+0000","172.21.15.88:6835/3163343083":"2024-09-17T01:42:01.375396+0000","172.21.15.88:6835/4222250252":"2024-09-17T01:42:45.673956+0000","172.21.15.88:6835/4162274767":"2024-09-17T01:40:11.522091+0000","172.21.15.88:6835/2477051134":"2024-09-17T01:43:04.380783+0000","172.21.15.88:6834/1289932173":"2024-09-17T01:40:42.399399+0000","172.21.15.88:6837/2733458327":"2024-09-17T01:40:27.314712+0000","172.21.15.88:6834/2550498835":"2024-09-17T01:41:57.358287+0000","172.21.15.88:6836/2550498835":"2024-09-17T01:41:57.358287+0000","172.21.15.88:6835/2285699282":"2024-09-17T01:42:29.541113+0000","172.21.15.88:6837/3163343083":"2024-09-17T01:42:01.375396+0000","172.21.15.88:6834/3728363359":"2024-09-17T01:42:05.412133+0000","172.21.15.88:6834/1099469537":"2024-09-17T01:44:22.054752+0000","172.21.15.88:6835/880314974":"2024-09-17T01:42:21.485748+0000","172.21.15.88:6835/1936493048":"2024-09-17T01:42:19.133197+0000","172.21.15.88:6837/1936493048":"2024-09-17T01:42:19.133197+0000","172.21.15.88:6837/880314974":"2024-09-17T01:42:21.485748+0000","172.21.15.88:6834/3374372758":"2024-09-17T01:42:25.506299+0000","172.21.15.88:6836/3374372758":"2024-09-17T01:42:25.506299+0000","172.21.15.88:6836/201188184":"2024-09-17T01:42:33.276104+0000","172.21.15.88:6837/1990650099":"2024-09-17T01:42:37.599896+0000","172.21.15.88:6834/201188184":"2024-09-17T01:42:33.276104+0000","172.21.15.88:6835/1990650099":"2024-09-17T01:42:37.599896+0000","172.21.15.88:6837/4162274767":"2024-09-17T01:40:11.522091+0000","172.21.15.88:6837/2408579939":"2024-09-17T01:40:46.434492+0000","172.21.15.88:6834/2680708900":"2024-09-17T01:42:41.629904+0000","172.21.15.88:6837/4222250252":"2024-09-17T01:42:45.673956+0000","172.21.15.88:6834/1042715824":"2024-09-17T01:40:59.080515+0000","172.21.15.88:6834/4065900771":"2024-09-17T01:42:58.329389+0000","172.21.15.88:6836/4065900771":"2024-09-17T01:42:58.329389+0000","172.21.15.88:6836/4144876777":"2024-09-17T01:43:00.350808+0000","172.21.15.88:6837/2477051134":"2024-09-17T01:43:04.380783+0000","172.21.15.88:6834/650399848":"2024-09-17T01:43:08.279879+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:44:40.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.543+0000 7fbd0154b640 1 -- 172.21.15.88:0/3357342627 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbcdc06c2b0 msgr2=0x7fbcdc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:40.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.543+0000 7fbd0154b640 1 --2- 172.21.15.88:0/3357342627 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbcdc06c2b0 0x7fbcdc06e770 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fbcec007fd0 tx=0x7fbcec007e80 comp rx=0 tx=0).stop 2024-09-16T01:44:40.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.543+0000 7fbd0154b640 1 -- 172.21.15.88:0/3357342627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcfc0cd780 msgr2=0x7fbcfc159e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:40.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.543+0000 7fbd0154b640 1 --2- 172.21.15.88:0/3357342627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcfc0cd780 0x7fbcfc159e00 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4009eb0 tx=0x7fbcf4004a40 comp rx=0 tx=0).stop 2024-09-16T01:44:40.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.543+0000 7fbd0154b640 1 -- 172.21.15.88:0/3357342627 shutdown_connections 2024-09-16T01:44:40.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.543+0000 7fbd0154b640 1 --2- 172.21.15.88:0/3357342627 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fbcdc06c2b0 0x7fbcdc06e770 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:40.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.543+0000 7fbd0154b640 1 --2- 172.21.15.88:0/3357342627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbcfc0cd780 0x7fbcfc159e00 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:40.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.543+0000 7fbd0154b640 1 --2- 172.21.15.88:0/3357342627 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbcfc0b4d00 0x7fbcfc1598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:40.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.543+0000 7fbd0154b640 1 --2- 172.21.15.88:0/3357342627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbcfc0b4920 0x7fbcfc145fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:40.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.543+0000 7fbd0154b640 1 -- 172.21.15.88:0/3357342627 >> 172.21.15.88:0/3357342627 conn(0x7fbcfc06bc10 msgr2=0x7fbcfc071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:40.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.543+0000 7fbd0154b640 1 -- 172.21.15.88:0/3357342627 shutdown_connections 2024-09-16T01:44:40.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.543+0000 7fbd0154b640 1 -- 172.21.15.88:0/3357342627 wait complete. 2024-09-16T01:44:40.557 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:44:40.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.655+0000 7f873b465640 1 Processor -- start 2024-09-16T01:44:40.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.656+0000 7f873b465640 1 -- start start 2024-09-16T01:44:40.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.657+0000 7f873b465640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f87340cf1f0 0x7f87340b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:40.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.657+0000 7f873b465640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f87340b6720 0x7f87340d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:40.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.657+0000 7f873b465640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87340cf860 con 0x7f873415e7a0 2024-09-16T01:44:40.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.657+0000 7f873b465640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f87340d2ca0 con 0x7f87340cf1f0 2024-09-16T01:44:40.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.657+0000 7f873b465640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f87340cfb40 con 0x7f87340b6720 2024-09-16T01:44:40.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.657+0000 7f8738a00640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f87340cf1f0 0x7f87340b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:40.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.657+0000 7f8738a00640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f87340cf1f0 0x7f87340b61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41516/0 (socket says 172.21.15.88:41516) 2024-09-16T01:44:40.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.657+0000 7f8738a00640 1 -- 172.21.15.88:0/1100452966 learned_addr learned my addr 172.21.15.88:0/1100452966 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:40.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.657+0000 7f8738a00640 1 -- 172.21.15.88:0/1100452966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f87340b6720 msgr2=0x7f87340d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:40.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.657+0000 7f8739a02640 1 --2- 172.21.15.88:0/1100452966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f87340b6720 0x7f87340d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:40.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.657+0000 7f8738a00640 1 --2- 172.21.15.88:0/1100452966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f87340b6720 0x7f87340d3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:40.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.657+0000 7f8738a00640 1 -- 172.21.15.88:0/1100452966 >> v1:172.21.15.154:6789/0 conn(0x7f873415e7a0 legacy=0x7f873406caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:40.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.657+0000 7f8738a00640 1 -- 172.21.15.88:0/1100452966 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87340cfe20 con 0x7f87340cf1f0 2024-09-16T01:44:40.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.658+0000 7f8738a00640 1 --2- 172.21.15.88:0/1100452966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87340cf1f0 0x7f87340b61e0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7f8724009870 tx=0x7f872402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16f1a584c98d40a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:40.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.658+0000 7f872bfff640 1 -- 172.21.15.88:0/1100452966 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8724037070 con 0x7f87340cf1f0 2024-09-16T01:44:40.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.658+0000 7f872bfff640 1 -- 172.21.15.88:0/1100452966 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8724032440 con 0x7f87340cf1f0 2024-09-16T01:44:40.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.658+0000 7f872bfff640 1 -- 172.21.15.88:0/1100452966 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8724032810 con 0x7f87340cf1f0 2024-09-16T01:44:40.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.659+0000 7f873b465640 1 -- 172.21.15.88:0/1100452966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87340cf1f0 msgr2=0x7f87340b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:40.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.659+0000 7f873b465640 1 --2- 172.21.15.88:0/1100452966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87340cf1f0 0x7f87340b61e0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7f8724009870 tx=0x7f872402a180 comp rx=0 tx=0).stop 2024-09-16T01:44:40.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.659+0000 7f873b465640 1 -- 172.21.15.88:0/1100452966 shutdown_connections 2024-09-16T01:44:40.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.659+0000 7f873b465640 1 --2- 172.21.15.88:0/1100452966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f87340b6720 0x7f87340d3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:40.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.659+0000 7f873b465640 1 --2- 172.21.15.88:0/1100452966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87340cf1f0 0x7f87340b61e0 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:40.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.659+0000 7f873b465640 1 -- 172.21.15.88:0/1100452966 >> 172.21.15.88:0/1100452966 conn(0x7f87340d11e0 msgr2=0x7f87340d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:40.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.659+0000 7f873b465640 1 -- 172.21.15.88:0/1100452966 shutdown_connections 2024-09-16T01:44:40.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.659+0000 7f873b465640 1 -- 172.21.15.88:0/1100452966 wait complete. 2024-09-16T01:44:40.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.660+0000 7f873b465640 1 Processor -- start 2024-09-16T01:44:40.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.660+0000 7f873b465640 1 -- start start 2024-09-16T01:44:40.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.661+0000 7f873b465640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87340b6720 0x7f8734156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:40.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.661+0000 7f873b465640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f87340cf1f0 0x7f8734161110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:40.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.661+0000 7f873b465640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f873415e7a0 0x7f8734156480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:40.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.661+0000 7f873b465640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f873415eb80 con 0x7f87340b6720 2024-09-16T01:44:40.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.661+0000 7f873b465640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f87340b7a00 con 0x7f873415e7a0 2024-09-16T01:44:40.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.661+0000 7f873b465640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f873415ed00 con 0x7f87340cf1f0 2024-09-16T01:44:40.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.661+0000 7f8739201640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87340b6720 0x7f8734156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:40.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.661+0000 7f8739201640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87340b6720 0x7f8734156ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41532/0 (socket says 172.21.15.88:41532) 2024-09-16T01:44:40.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.661+0000 7f8739201640 1 -- 172.21.15.88:0/1290272843 learned_addr learned my addr 172.21.15.88:0/1290272843 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:40.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.661+0000 7f8739a02640 1 --2- 172.21.15.88:0/1290272843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f873415e7a0 0x7f8734156480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:40.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.661+0000 7f8738a00640 1 --2- 172.21.15.88:0/1290272843 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f87340cf1f0 0x7f8734161110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:40.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.661+0000 7f8739201640 1 -- 172.21.15.88:0/1290272843 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f87340cf1f0 msgr2=0x7f8734161110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:40.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.661+0000 7f8739201640 1 --2- 172.21.15.88:0/1290272843 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f87340cf1f0 0x7f8734161110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:40.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.661+0000 7f8739201640 1 -- 172.21.15.88:0/1290272843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f873415e7a0 msgr2=0x7f8734156480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:40.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.661+0000 7f8739201640 1 --2- 172.21.15.88:0/1290272843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f873415e7a0 0x7f8734156480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:40.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.661+0000 7f8739201640 1 -- 172.21.15.88:0/1290272843 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87340d51d0 con 0x7f87340b6720 2024-09-16T01:44:40.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.661+0000 7f8739a02640 1 --2- 172.21.15.88:0/1290272843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f873415e7a0 0x7f8734156480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:40.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.661+0000 7f8739201640 1 --2- 172.21.15.88:0/1290272843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87340b6720 0x7f8734156ba0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7f873000ba50 tx=0x7f873000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:40.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.661+0000 7f8729ffb640 1 -- 172.21.15.88:0/1290272843 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8730019070 con 0x7f87340b6720 2024-09-16T01:44:40.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.661+0000 7f8729ffb640 1 -- 172.21.15.88:0/1290272843 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f873000cc20 con 0x7f87340b6720 2024-09-16T01:44:40.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.661+0000 7f873b465640 1 -- 172.21.15.88:0/1290272843 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87341618a0 con 0x7f87340b6720 2024-09-16T01:44:40.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.662+0000 7f8729ffb640 1 -- 172.21.15.88:0/1290272843 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8730014680 con 0x7f87340b6720 2024-09-16T01:44:40.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.662+0000 7f873b465640 1 -- 172.21.15.88:0/1290272843 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8734158940 con 0x7f87340b6720 2024-09-16T01:44:40.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.663+0000 7f873b465640 1 -- 172.21.15.88:0/1290272843 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87340b61e0 con 0x7f87340b6720 2024-09-16T01:44:40.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.663+0000 7f8729ffb640 1 -- 172.21.15.88:0/1290272843 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f873001c070 con 0x7f87340b6720 2024-09-16T01:44:40.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.663+0000 7f8729ffb640 1 --2- 172.21.15.88:0/1290272843 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f871806c2b0 0x7f871806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:40.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.664+0000 7f8729ffb640 1 -- 172.21.15.88:0/1290272843 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9178+0+0 (secure 0 0 0) 0x7f873008be30 con 0x7f87340b6720 2024-09-16T01:44:40.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.664+0000 7f8738a00640 1 --2- 172.21.15.88:0/1290272843 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f871806c2b0 0x7f871806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:40.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.664+0000 7f8738a00640 1 --2- 172.21.15.88:0/1290272843 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f871806c2b0 0x7f871806e770 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f8724034000 tx=0x7f872402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:40.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.668+0000 7f8729ffb640 1 -- 172.21.15.88:0/1290272843 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8730059200 con 0x7f87340b6720 2024-09-16T01:44:40.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:40.868+0000 7f873b465640 1 -- 172.21.15.88:0/1290272843 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f87340d2a80 con 0x7f87340b6720 2024-09-16T01:44:41.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.145+0000 7f8729ffb640 1 -- 172.21.15.88:0/1290272843 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v320) ==== 161+0+0 (secure 0 0 0) 0x7f873005d500 con 0x7f87340b6720 2024-09-16T01:44:41.210 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:44:41.212 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.148+0000 7f873b465640 1 -- 172.21.15.88:0/1290272843 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f871806c2b0 msgr2=0x7f871806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:41.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.148+0000 7f873b465640 1 --2- 172.21.15.88:0/1290272843 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f871806c2b0 0x7f871806e770 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f8724034000 tx=0x7f872402e000 comp rx=0 tx=0).stop 2024-09-16T01:44:41.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.148+0000 7f873b465640 1 -- 172.21.15.88:0/1290272843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87340b6720 msgr2=0x7f8734156ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:41.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.148+0000 7f873b465640 1 --2- 172.21.15.88:0/1290272843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87340b6720 0x7f8734156ba0 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7f873000ba50 tx=0x7f873000bf10 comp rx=0 tx=0).stop 2024-09-16T01:44:41.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.148+0000 7f873b465640 1 -- 172.21.15.88:0/1290272843 shutdown_connections 2024-09-16T01:44:41.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.148+0000 7f873b465640 1 --2- 172.21.15.88:0/1290272843 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f871806c2b0 0x7f871806e770 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:41.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.148+0000 7f873b465640 1 --2- 172.21.15.88:0/1290272843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f873415e7a0 0x7f8734156480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:41.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.148+0000 7f873b465640 1 --2- 172.21.15.88:0/1290272843 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f87340cf1f0 0x7f8734161110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:41.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.148+0000 7f873b465640 1 --2- 172.21.15.88:0/1290272843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f87340b6720 0x7f8734156ba0 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:41.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.148+0000 7f873b465640 1 -- 172.21.15.88:0/1290272843 >> 172.21.15.88:0/1290272843 conn(0x7f87340d11e0 msgr2=0x7f873415c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:41.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.148+0000 7f873b465640 1 -- 172.21.15.88:0/1290272843 shutdown_connections 2024-09-16T01:44:41.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.148+0000 7f873b465640 1 -- 172.21.15.88:0/1290272843 wait complete. 2024-09-16T01:44:41.214 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-16T01:44:41.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.324+0000 7f4db825e640 1 Processor -- start 2024-09-16T01:44:41.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.325+0000 7f4db825e640 1 -- start start 2024-09-16T01:44:41.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.325+0000 7f4db825e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4db015a050 0x7f4db015d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:41.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.326+0000 7f4db825e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4db015ee90 0x7f4db0073600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:41.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.326+0000 7f4db825e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4db015a430 con 0x7f4db00731c0 2024-09-16T01:44:41.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.326+0000 7f4db825e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f4db0073b40 con 0x7f4db015ee90 2024-09-16T01:44:41.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.326+0000 7f4db825e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4db015d9a0 con 0x7f4db015a050 2024-09-16T01:44:41.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.326+0000 7f4db67fb640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4db015ee90 0x7f4db0073600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:41.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.326+0000 7f4db67fb640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4db015ee90 0x7f4db0073600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41536/0 (socket says 172.21.15.88:41536) 2024-09-16T01:44:41.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.326+0000 7f4db57f9640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4db00731c0 0x7f4db015ffd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46222/0 (socket says 172.21.15.88:46222) 2024-09-16T01:44:41.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.326+0000 7f4db67fb640 1 -- 172.21.15.88:0/164115483 learned_addr learned my addr 172.21.15.88:0/164115483 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:41.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.326+0000 7f4db5ffa640 1 --2- 172.21.15.88:0/164115483 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4db015a050 0x7f4db015d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:41.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.326+0000 7f4db67fb640 1 -- 172.21.15.88:0/164115483 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4db015a050 msgr2=0x7f4db015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:41.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.326+0000 7f4db67fb640 1 --2- 172.21.15.88:0/164115483 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4db015a050 0x7f4db015d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:41.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.326+0000 7f4db67fb640 1 -- 172.21.15.88:0/164115483 >> v1:172.21.15.88:6789/0 conn(0x7f4db00731c0 legacy=0x7f4db015ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:41.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.326+0000 7f4db4ff8640 1 -- 172.21.15.88:0/164115483 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 723697510 0 0) 0x7f4db015a430 con 0x7f4db00731c0 2024-09-16T01:44:41.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.326+0000 7f4db67fb640 1 -- 172.21.15.88:0/164115483 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4db00cd0a0 con 0x7f4db015ee90 2024-09-16T01:44:41.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.327+0000 7f4db67fb640 1 --2- 172.21.15.88:0/164115483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4db015ee90 0x7f4db0073600 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f4dac009870 tx=0x7f4dac02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=294f2cdaf4d7b5fa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:41.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.327+0000 7f4db4ff8640 1 -- 172.21.15.88:0/164115483 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dac037070 con 0x7f4db015ee90 2024-09-16T01:44:41.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.327+0000 7f4db4ff8640 1 -- 172.21.15.88:0/164115483 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dac032440 con 0x7f4db015ee90 2024-09-16T01:44:41.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.327+0000 7f4db4ff8640 1 -- 172.21.15.88:0/164115483 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dac032810 con 0x7f4db015ee90 2024-09-16T01:44:41.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.327+0000 7f4db825e640 1 -- 172.21.15.88:0/164115483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4db015ee90 msgr2=0x7f4db0073600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:41.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.327+0000 7f4db825e640 1 --2- 172.21.15.88:0/164115483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4db015ee90 0x7f4db0073600 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f4dac009870 tx=0x7f4dac02a1c0 comp rx=0 tx=0).stop 2024-09-16T01:44:41.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.327+0000 7f4db825e640 1 -- 172.21.15.88:0/164115483 shutdown_connections 2024-09-16T01:44:41.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.327+0000 7f4db825e640 1 --2- 172.21.15.88:0/164115483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4db015ee90 0x7f4db0073600 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:41.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.327+0000 7f4db825e640 1 --2- 172.21.15.88:0/164115483 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4db015a050 0x7f4db015d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:41.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.327+0000 7f4db825e640 1 -- 172.21.15.88:0/164115483 >> 172.21.15.88:0/164115483 conn(0x7f4db006d0f0 msgr2=0x7f4db006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:41.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.328+0000 7f4db825e640 1 -- 172.21.15.88:0/164115483 shutdown_connections 2024-09-16T01:44:41.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.328+0000 7f4db825e640 1 -- 172.21.15.88:0/164115483 wait complete. 2024-09-16T01:44:41.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.329+0000 7f4db825e640 1 Processor -- start 2024-09-16T01:44:41.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.329+0000 7f4db825e640 1 -- start start 2024-09-16T01:44:41.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.329+0000 7f4db825e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4db00731c0 0x7f4db0158260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:41.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.329+0000 7f4db825e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4db015a050 0x7f4db01f44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:41.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.329+0000 7f4db825e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4db015ee90 0x7f4db0156550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:41.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.329+0000 7f4db825e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4db00cd940 con 0x7f4db015ee90 2024-09-16T01:44:41.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.329+0000 7f4db825e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4db00cd660 con 0x7f4db00731c0 2024-09-16T01:44:41.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.329+0000 7f4db825e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4db00cdc20 con 0x7f4db015a050 2024-09-16T01:44:41.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.330+0000 7f4db67fb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4db015ee90 0x7f4db0156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:41.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.330+0000 7f4db67fb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4db015ee90 0x7f4db0156550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41538/0 (socket says 172.21.15.88:41538) 2024-09-16T01:44:41.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.330+0000 7f4db67fb640 1 -- 172.21.15.88:0/1404031642 learned_addr learned my addr 172.21.15.88:0/1404031642 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:41.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.330+0000 7f4db67fb640 1 -- 172.21.15.88:0/1404031642 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4db015a050 msgr2=0x7f4db01f44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:41.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.330+0000 7f4db67fb640 1 --2- 172.21.15.88:0/1404031642 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4db015a050 0x7f4db01f44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:41.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.330+0000 7f4db5ffa640 1 --2- 172.21.15.88:0/1404031642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4db00731c0 0x7f4db0158260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:41.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.330+0000 7f4db67fb640 1 -- 172.21.15.88:0/1404031642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4db00731c0 msgr2=0x7f4db0158260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:41.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.330+0000 7f4db67fb640 1 --2- 172.21.15.88:0/1404031642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4db00731c0 0x7f4db0158260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:41.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.330+0000 7f4db67fb640 1 -- 172.21.15.88:0/1404031642 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4db00714b0 con 0x7f4db015ee90 2024-09-16T01:44:41.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.330+0000 7f4db67fb640 1 --2- 172.21.15.88:0/1404031642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4db015ee90 0x7f4db0156550 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7f4dac009870 tx=0x7f4dac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:41.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.330+0000 7f4da67fc640 1 -- 172.21.15.88:0/1404031642 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dac041020 con 0x7f4db015ee90 2024-09-16T01:44:41.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.330+0000 7f4da67fc640 1 -- 172.21.15.88:0/1404031642 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dac03d640 con 0x7f4db015ee90 2024-09-16T01:44:41.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.330+0000 7f4db825e640 1 -- 172.21.15.88:0/1404031642 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4db0071790 con 0x7f4db015ee90 2024-09-16T01:44:41.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.330+0000 7f4da67fc640 1 -- 172.21.15.88:0/1404031642 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dac037040 con 0x7f4db015ee90 2024-09-16T01:44:41.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.330+0000 7f4db825e640 1 -- 172.21.15.88:0/1404031642 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4db0071d50 con 0x7f4db015ee90 2024-09-16T01:44:41.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.331+0000 7f4db825e640 1 -- 172.21.15.88:0/1404031642 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4db00b36e0 con 0x7f4db015ee90 2024-09-16T01:44:41.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.332+0000 7f4da67fc640 1 -- 172.21.15.88:0/1404031642 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4dac044020 con 0x7f4db015ee90 2024-09-16T01:44:41.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.332+0000 7f4da67fc640 1 --2- 172.21.15.88:0/1404031642 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d9406c2b0 0x7f4d9406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:41.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.332+0000 7f4da67fc640 1 -- 172.21.15.88:0/1404031642 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9178+0+0 (secure 0 0 0) 0x7f4dac0ac910 con 0x7f4db015ee90 2024-09-16T01:44:41.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.332+0000 7f4db5ffa640 1 --2- 172.21.15.88:0/1404031642 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d9406c2b0 0x7f4d9406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:41.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.333+0000 7f4db5ffa640 1 --2- 172.21.15.88:0/1404031642 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d9406c2b0 0x7f4d9406e770 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f4da8009fd0 tx=0x7f4da800b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:41.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.336+0000 7f4da67fc640 1 -- 172.21.15.88:0/1404031642 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4dac04b030 con 0x7f4db015ee90 2024-09-16T01:44:41.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:41.537+0000 7f4db825e640 1 -- 172.21.15.88:0/1404031642 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f4db00b3d40 con 0x7f4db015ee90 2024-09-16T01:44:42.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.152+0000 7f4da67fc640 1 -- 172.21.15.88:0/1404031642 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v321) ==== 104+0+0 (secure 0 0 0) 0x7f4dac07a2b0 con 0x7f4db015ee90 2024-09-16T01:44:42.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.154+0000 7f4db825e640 1 -- 172.21.15.88:0/1404031642 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d9406c2b0 msgr2=0x7f4d9406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:42.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.155+0000 7f4db825e640 1 --2- 172.21.15.88:0/1404031642 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d9406c2b0 0x7f4d9406e770 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f4da8009fd0 tx=0x7f4da800b040 comp rx=0 tx=0).stop 2024-09-16T01:44:42.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.155+0000 7f4db825e640 1 -- 172.21.15.88:0/1404031642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4db015ee90 msgr2=0x7f4db0156550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:42.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.155+0000 7f4db825e640 1 --2- 172.21.15.88:0/1404031642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4db015ee90 0x7f4db0156550 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7f4dac009870 tx=0x7f4dac0046c0 comp rx=0 tx=0).stop 2024-09-16T01:44:42.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.155+0000 7f4db825e640 1 -- 172.21.15.88:0/1404031642 shutdown_connections 2024-09-16T01:44:42.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.155+0000 7f4db825e640 1 --2- 172.21.15.88:0/1404031642 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d9406c2b0 0x7f4d9406e770 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:42.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.155+0000 7f4db825e640 1 --2- 172.21.15.88:0/1404031642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4db015ee90 0x7f4db0156550 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:42.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.155+0000 7f4db825e640 1 --2- 172.21.15.88:0/1404031642 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4db015a050 0x7f4db01f44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:42.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.155+0000 7f4db825e640 1 --2- 172.21.15.88:0/1404031642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4db00731c0 0x7f4db0158260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:42.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.155+0000 7f4db825e640 1 -- 172.21.15.88:0/1404031642 >> 172.21.15.88:0/1404031642 conn(0x7f4db006d0f0 msgr2=0x7f4db00b45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:42.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.155+0000 7f4db825e640 1 -- 172.21.15.88:0/1404031642 shutdown_connections 2024-09-16T01:44:42.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.155+0000 7f4db825e640 1 -- 172.21.15.88:0/1404031642 wait complete. 2024-09-16T01:44:42.170 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-16T01:44:42.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.273+0000 7f5e427cd640 1 Processor -- start 2024-09-16T01:44:42.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.273+0000 7f5e427cd640 1 -- start start 2024-09-16T01:44:42.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.273+0000 7f5e427cd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5e3c15b480 0x7f5e3c0b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:42.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.273+0000 7f5e427cd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e3c072700 0x7f5e3c15bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:42.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.273+0000 7f5e427cd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e3c0ccff0 con 0x7f5e3c0b60c0 2024-09-16T01:44:42.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.273+0000 7f5e427cd640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5e3c0cd2d0 con 0x7f5e3c15b480 2024-09-16T01:44:42.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.274+0000 7f5e427cd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5e3c0cd5b0 con 0x7f5e3c072700 2024-09-16T01:44:42.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.274+0000 7f5e3b7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5e3c0b60c0 0x7f5e3c071ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46232/0 (socket says 172.21.15.88:46232) 2024-09-16T01:44:42.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.274+0000 7f5e3b7fe640 1 -- 172.21.15.88:0/3543198818 learned_addr learned my addr 172.21.15.88:0/3543198818 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:42.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.274+0000 7f5e3affd640 1 -- 172.21.15.88:0/3543198818 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3169715302 0 0) 0x7f5e3c0ccff0 con 0x7f5e3c0b60c0 2024-09-16T01:44:42.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.274+0000 7f5e3bfff640 1 --2- 172.21.15.88:0/3543198818 >> v2:172.21.15.154:3300/0 conn(0x7f5e3c15b480 0x7f5e3c0b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:42.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.274+0000 7f5e3affd640 1 -- 172.21.15.88:0/3543198818 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e28003520 con 0x7f5e3c0b60c0 2024-09-16T01:44:42.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.274+0000 7f5e40d6a640 1 --2- 172.21.15.88:0/3543198818 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e3c072700 0x7f5e3c15bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:42.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.274+0000 7f5e3affd640 1 -- 172.21.15.88:0/3543198818 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1135920858 0 0) 0x7f5e28003520 con 0x7f5e3c0b60c0 2024-09-16T01:44:42.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.274+0000 7f5e3affd640 1 -- 172.21.15.88:0/3543198818 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e3c072700 msgr2=0x7f5e3c15bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:42.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.274+0000 7f5e3affd640 1 --2- 172.21.15.88:0/3543198818 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e3c072700 0x7f5e3c15bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:42.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.274+0000 7f5e3affd640 1 -- 172.21.15.88:0/3543198818 >> v2:172.21.15.154:3300/0 conn(0x7f5e3c15b480 msgr2=0x7f5e3c0b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:42.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.274+0000 7f5e3affd640 1 --2- 172.21.15.88:0/3543198818 >> v2:172.21.15.154:3300/0 conn(0x7f5e3c15b480 0x7f5e3c0b5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:42.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.274+0000 7f5e3affd640 1 -- 172.21.15.88:0/3543198818 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e3c0cd890 con 0x7f5e3c0b60c0 2024-09-16T01:44:42.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.274+0000 7f5e3affd640 1 -- 172.21.15.88:0/3543198818 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f5e24001670 con 0x7f5e3c0b60c0 2024-09-16T01:44:42.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.274+0000 7f5e40d6a640 1 --2- 172.21.15.88:0/3543198818 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e3c072700 0x7f5e3c15bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:42.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.274+0000 7f5e3affd640 1 -- 172.21.15.88:0/3543198818 >> v1:172.21.15.88:6789/0 conn(0x7f5e3c0b60c0 legacy=0x7f5e3c071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:42.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.274+0000 7f5e3affd640 1 --2- 172.21.15.88:0/3543198818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e280039b0 0x7f5e28003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:42.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.274+0000 7f5e3affd640 1 --2- 172.21.15.88:0/3543198818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e28004800 0x7f5e28004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:42.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.274+0000 7f5e3affd640 1 --2- 172.21.15.88:0/3543198818 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e3c072700 0x7f5e280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:42.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.274+0000 7f5e3affd640 1 -- 172.21.15.88:0/3543198818 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3c0cd2d0 con 0x7f5e28004800 2024-09-16T01:44:42.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.274+0000 7f5e3affd640 1 -- 172.21.15.88:0/3543198818 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3c0cd5b0 con 0x7f5e280039b0 2024-09-16T01:44:42.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.274+0000 7f5e3affd640 1 -- 172.21.15.88:0/3543198818 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5e2800baa0 con 0x7f5e3c072700 2024-09-16T01:44:42.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.274+0000 7f5e3b7fe640 1 --2- 172.21.15.88:0/3543198818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e28004800 0x7f5e28004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:42.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.274+0000 7f5e3bfff640 1 --2- 172.21.15.88:0/3543198818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e280039b0 0x7f5e28003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:42.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.274+0000 7f5e40d6a640 1 --2- 172.21.15.88:0/3543198818 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e3c072700 0x7f5e280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:42.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.275+0000 7f5e3b7fe640 1 -- 172.21.15.88:0/3543198818 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e3c072700 msgr2=0x7f5e280051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:42.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.275+0000 7f5e3b7fe640 1 --2- 172.21.15.88:0/3543198818 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e3c072700 0x7f5e280051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:42.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.275+0000 7f5e3b7fe640 1 -- 172.21.15.88:0/3543198818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e280039b0 msgr2=0x7f5e28003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:42.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.275+0000 7f5e3b7fe640 1 --2- 172.21.15.88:0/3543198818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e280039b0 0x7f5e28003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:42.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.275+0000 7f5e3b7fe640 1 -- 172.21.15.88:0/3543198818 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e28003520 con 0x7f5e28004800 2024-09-16T01:44:42.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.275+0000 7f5e3bfff640 1 --2- 172.21.15.88:0/3543198818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e280039b0 0x7f5e28003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:42.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.275+0000 7f5e3b7fe640 1 --2- 172.21.15.88:0/3543198818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e28004800 0x7f5e28004c80 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f5e24001eb0 tx=0x7f5e28005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d0bdc497729b661 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:42.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.275+0000 7f5e3affd640 1 -- 172.21.15.88:0/3543198818 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e24007c80 con 0x7f5e28004800 2024-09-16T01:44:42.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.275+0000 7f5e3affd640 1 -- 172.21.15.88:0/3543198818 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e24005ad0 con 0x7f5e28004800 2024-09-16T01:44:42.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.275+0000 7f5e3affd640 1 -- 172.21.15.88:0/3543198818 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e24007070 con 0x7f5e28004800 2024-09-16T01:44:42.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.276+0000 7f5e427cd640 1 -- 172.21.15.88:0/3543198818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e28004800 msgr2=0x7f5e28004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:42.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.276+0000 7f5e427cd640 1 --2- 172.21.15.88:0/3543198818 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e28004800 0x7f5e28004c80 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f5e24001eb0 tx=0x7f5e28005830 comp rx=0 tx=0).stop 2024-09-16T01:44:42.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.276+0000 7f5e40d6a640 1 -- 172.21.15.88:0/3543198818 reap_dead start 2024-09-16T01:44:42.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.276+0000 7f5e427cd640 1 -- 172.21.15.88:0/3543198818 shutdown_connections 2024-09-16T01:44:42.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.276+0000 7f5e427cd640 1 -- 172.21.15.88:0/3543198818 >> 172.21.15.88:0/3543198818 conn(0x7f5e3c06d0f0 msgr2=0x7f5e3c06d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:42.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.276+0000 7f5e427cd640 1 -- 172.21.15.88:0/3543198818 shutdown_connections 2024-09-16T01:44:42.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.276+0000 7f5e427cd640 1 -- 172.21.15.88:0/3543198818 wait complete. 2024-09-16T01:44:42.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.277+0000 7f5e427cd640 1 Processor -- start 2024-09-16T01:44:42.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.277+0000 7f5e427cd640 1 -- start start 2024-09-16T01:44:42.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.278+0000 7f5e427cd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e28004800 0x7f5e3c14d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:42.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.278+0000 7f5e427cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e3c15bd70 0x7f5e3c0cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:42.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.278+0000 7f5e427cd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e3c0cc180 0x7f5e3c150570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:42.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.278+0000 7f5e427cd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3c15b9e0 con 0x7f5e3c15bd70 2024-09-16T01:44:42.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.278+0000 7f5e427cd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3c15b860 con 0x7f5e28004800 2024-09-16T01:44:42.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.278+0000 7f5e427cd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5e3c0b3590 con 0x7f5e3c0cc180 2024-09-16T01:44:42.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.278+0000 7f5e3b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e3c15bd70 0x7f5e3c0cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:42.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.278+0000 7f5e3b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e3c15bd70 0x7f5e3c0cbc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41560/0 (socket says 172.21.15.88:41560) 2024-09-16T01:44:42.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.278+0000 7f5e3b7fe640 1 -- 172.21.15.88:0/352815140 learned_addr learned my addr 172.21.15.88:0/352815140 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:42.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.278+0000 7f5e3bfff640 1 --2- 172.21.15.88:0/352815140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e28004800 0x7f5e3c14d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:42.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.278+0000 7f5e40d6a640 1 --2- 172.21.15.88:0/352815140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e3c0cc180 0x7f5e3c150570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:42.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.279+0000 7f5e3b7fe640 1 -- 172.21.15.88:0/352815140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e3c0cc180 msgr2=0x7f5e3c150570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:42.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.279+0000 7f5e3b7fe640 1 --2- 172.21.15.88:0/352815140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e3c0cc180 0x7f5e3c150570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:42.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.279+0000 7f5e3b7fe640 1 -- 172.21.15.88:0/352815140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e28004800 msgr2=0x7f5e3c14d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:42.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.279+0000 7f5e3b7fe640 1 --2- 172.21.15.88:0/352815140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e28004800 0x7f5e3c14d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:42.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.279+0000 7f5e3b7fe640 1 -- 172.21.15.88:0/352815140 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e3c151280 con 0x7f5e3c15bd70 2024-09-16T01:44:42.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.279+0000 7f5e3b7fe640 1 --2- 172.21.15.88:0/352815140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e3c15bd70 0x7f5e3c0cbc40 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f5e2402ad20 tx=0x7f5e3c0cd1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:42.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.279+0000 7f5e38ff9640 1 -- 172.21.15.88:0/352815140 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e24001630 con 0x7f5e3c15bd70 2024-09-16T01:44:42.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.279+0000 7f5e38ff9640 1 -- 172.21.15.88:0/352815140 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e24042070 con 0x7f5e3c15bd70 2024-09-16T01:44:42.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.279+0000 7f5e427cd640 1 -- 172.21.15.88:0/352815140 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e3c0c98b0 con 0x7f5e3c15bd70 2024-09-16T01:44:42.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.279+0000 7f5e38ff9640 1 -- 172.21.15.88:0/352815140 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e24007270 con 0x7f5e3c15bd70 2024-09-16T01:44:42.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.279+0000 7f5e427cd640 1 -- 172.21.15.88:0/352815140 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e3c0c9e70 con 0x7f5e3c15bd70 2024-09-16T01:44:42.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.280+0000 7f5e427cd640 1 -- 172.21.15.88:0/352815140 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e3c157f50 con 0x7f5e3c15bd70 2024-09-16T01:44:42.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.281+0000 7f5e38ff9640 1 -- 172.21.15.88:0/352815140 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5e24007410 con 0x7f5e3c15bd70 2024-09-16T01:44:42.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.281+0000 7f5e38ff9640 1 --2- 172.21.15.88:0/352815140 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e1c06c2b0 0x7f5e1c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:42.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.281+0000 7f5e38ff9640 1 -- 172.21.15.88:0/352815140 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9178+0+0 (secure 0 0 0) 0x7f5e240ac0f0 con 0x7f5e3c15bd70 2024-09-16T01:44:42.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.281+0000 7f5e3bfff640 1 --2- 172.21.15.88:0/352815140 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e1c06c2b0 0x7f5e1c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:42.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.281+0000 7f5e3bfff640 1 --2- 172.21.15.88:0/352815140 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e1c06c2b0 0x7f5e1c06e770 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009bc0 tx=0x7f5e2c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:42.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.285+0000 7f5e38ff9640 1 -- 172.21.15.88:0/352815140 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5e3c157f50 con 0x7f5e3c15bd70 2024-09-16T01:44:42.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:42.450+0000 7f5e427cd640 1 -- 172.21.15.88:0/352815140 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5e3c069180 con 0x7f5e3c15bd70 2024-09-16T01:44:43.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.161+0000 7f5e38ff9640 1 -- 172.21.15.88:0/352815140 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v128) ==== 181+0+0 (secure 0 0 0) 0x7f5e3c069180 con 0x7f5e3c15bd70 2024-09-16T01:44:43.163 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_metadata' removed 2024-09-16T01:44:43.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.163+0000 7f5e427cd640 1 -- 172.21.15.88:0/352815140 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e1c06c2b0 msgr2=0x7f5e1c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:43.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.163+0000 7f5e427cd640 1 --2- 172.21.15.88:0/352815140 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e1c06c2b0 0x7f5e1c06e770 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009bc0 tx=0x7f5e2c008040 comp rx=0 tx=0).stop 2024-09-16T01:44:43.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.163+0000 7f5e427cd640 1 -- 172.21.15.88:0/352815140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e3c15bd70 msgr2=0x7f5e3c0cbc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:43.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.163+0000 7f5e427cd640 1 --2- 172.21.15.88:0/352815140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e3c15bd70 0x7f5e3c0cbc40 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f5e2402ad20 tx=0x7f5e3c0cd1c0 comp rx=0 tx=0).stop 2024-09-16T01:44:43.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.163+0000 7f5e427cd640 1 -- 172.21.15.88:0/352815140 shutdown_connections 2024-09-16T01:44:43.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.163+0000 7f5e427cd640 1 --2- 172.21.15.88:0/352815140 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5e1c06c2b0 0x7f5e1c06e770 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:43.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.163+0000 7f5e427cd640 1 --2- 172.21.15.88:0/352815140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e3c0cc180 0x7f5e3c150570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:43.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.163+0000 7f5e427cd640 1 --2- 172.21.15.88:0/352815140 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5e3c15bd70 0x7f5e3c0cbc40 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:43.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.163+0000 7f5e427cd640 1 --2- 172.21.15.88:0/352815140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e28004800 0x7f5e3c14d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:43.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.163+0000 7f5e427cd640 1 -- 172.21.15.88:0/352815140 >> 172.21.15.88:0/352815140 conn(0x7f5e3c06d0f0 msgr2=0x7f5e3c157840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:43.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.164+0000 7f5e427cd640 1 -- 172.21.15.88:0/352815140 shutdown_connections 2024-09-16T01:44:43.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.164+0000 7f5e427cd640 1 -- 172.21.15.88:0/352815140 wait complete. 2024-09-16T01:44:43.180 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-16T01:44:43.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.283+0000 7f6e3d32c640 1 Processor -- start 2024-09-16T01:44:43.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.283+0000 7f6e3d32c640 1 -- start start 2024-09-16T01:44:43.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.284+0000 7f6e3d32c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e380b6290 0x7f6e38071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:43.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.284+0000 7f6e3d32c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e380ccff0 con 0x7f6e38072530 2024-09-16T01:44:43.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.284+0000 7f6e3d32c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e380cd2d0 con 0x7f6e3815b480 2024-09-16T01:44:43.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.284+0000 7f6e3d32c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6e380cd5b0 con 0x7f6e380b6290 2024-09-16T01:44:43.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.284+0000 7f6e377fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f6e38072530 0x7f6e3815bc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46240/0 (socket says 172.21.15.88:46240) 2024-09-16T01:44:43.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.284+0000 7f6e377fe640 1 -- 172.21.15.88:0/4274130876 learned_addr learned my addr 172.21.15.88:0/4274130876 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:43.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.284+0000 7f6e367fc640 1 --2- 172.21.15.88:0/4274130876 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e380b6290 0x7f6e38071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:43.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.284+0000 7f6e35ffb640 1 -- 172.21.15.88:0/4274130876 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3095478009 0 0) 0x7f6e380ccff0 con 0x7f6e38072530 2024-09-16T01:44:43.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.284+0000 7f6e35ffb640 1 -- 172.21.15.88:0/4274130876 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e24003520 con 0x7f6e38072530 2024-09-16T01:44:43.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.285+0000 7f6e35ffb640 1 -- 172.21.15.88:0/4274130876 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2163258971 0 0) 0x7f6e24003520 con 0x7f6e38072530 2024-09-16T01:44:43.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.285+0000 7f6e35ffb640 1 -- 172.21.15.88:0/4274130876 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e380b6290 msgr2=0x7f6e38071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:43.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.285+0000 7f6e35ffb640 1 --2- 172.21.15.88:0/4274130876 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e380b6290 0x7f6e38071ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:43.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.285+0000 7f6e35ffb640 1 -- 172.21.15.88:0/4274130876 >> v1:172.21.15.154:6789/0 conn(0x7f6e3815b480 legacy=0x7f6e380b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:43.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.285+0000 7f6e35ffb640 1 -- 172.21.15.88:0/4274130876 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e380cd890 con 0x7f6e38072530 2024-09-16T01:44:43.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.285+0000 7f6e35ffb640 1 -- 172.21.15.88:0/4274130876 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f6e30001670 con 0x7f6e38072530 2024-09-16T01:44:43.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.285+0000 7f6e35ffb640 1 -- 172.21.15.88:0/4274130876 >> v1:172.21.15.88:6789/0 conn(0x7f6e38072530 legacy=0x7f6e3815bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:43.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.285+0000 7f6e35ffb640 1 --2- 172.21.15.88:0/4274130876 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e240039e0 0x7f6e24003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:43.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.285+0000 7f6e35ffb640 1 --2- 172.21.15.88:0/4274130876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e24004300 0x7f6e24004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:43.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.285+0000 7f6e35ffb640 1 --2- 172.21.15.88:0/4274130876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e24004f90 0x7f6e2400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:43.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.285+0000 7f6e35ffb640 1 -- 172.21.15.88:0/4274130876 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6e380cd5b0 con 0x7f6e24004300 2024-09-16T01:44:43.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.285+0000 7f6e35ffb640 1 -- 172.21.15.88:0/4274130876 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2400bf00 con 0x7f6e24004f90 2024-09-16T01:44:43.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.285+0000 7f6e35ffb640 1 -- 172.21.15.88:0/4274130876 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6e2400c0b0 con 0x7f6e240039e0 2024-09-16T01:44:43.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.285+0000 7f6e367fc640 1 --2- 172.21.15.88:0/4274130876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e24004300 0x7f6e24004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:43.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.285+0000 7f6e377fe640 1 --2- 172.21.15.88:0/4274130876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e24004f90 0x7f6e2400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:43.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.285+0000 7f6e36ffd640 1 --2- 172.21.15.88:0/4274130876 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e240039e0 0x7f6e24003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:43.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.285+0000 7f6e367fc640 1 -- 172.21.15.88:0/4274130876 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e240039e0 msgr2=0x7f6e24003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:43.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.285+0000 7f6e367fc640 1 --2- 172.21.15.88:0/4274130876 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e240039e0 0x7f6e24003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:43.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.285+0000 7f6e367fc640 1 -- 172.21.15.88:0/4274130876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e24004f90 msgr2=0x7f6e2400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:43.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.285+0000 7f6e367fc640 1 --2- 172.21.15.88:0/4274130876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e24004f90 0x7f6e2400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:43.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.285+0000 7f6e367fc640 1 -- 172.21.15.88:0/4274130876 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e24003520 con 0x7f6e24004300 2024-09-16T01:44:43.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.285+0000 7f6e36ffd640 1 --2- 172.21.15.88:0/4274130876 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e240039e0 0x7f6e24003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:43.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.286+0000 7f6e367fc640 1 --2- 172.21.15.88:0/4274130876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e24004300 0x7f6e24004800 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f6e20009ac0 tx=0x7f6e24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9705247f4966266 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:43.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.286+0000 7f6e35ffb640 1 -- 172.21.15.88:0/4274130876 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e20036070 con 0x7f6e24004300 2024-09-16T01:44:43.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.286+0000 7f6e35ffb640 1 -- 172.21.15.88:0/4274130876 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e20031400 con 0x7f6e24004300 2024-09-16T01:44:43.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.286+0000 7f6e35ffb640 1 -- 172.21.15.88:0/4274130876 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e20009690 con 0x7f6e24004300 2024-09-16T01:44:43.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.286+0000 7f6e3d32c640 1 -- 172.21.15.88:0/4274130876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e24004300 msgr2=0x7f6e24004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:43.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.286+0000 7f6e3d32c640 1 --2- 172.21.15.88:0/4274130876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e24004300 0x7f6e24004800 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f6e20009ac0 tx=0x7f6e24005830 comp rx=0 tx=0).stop 2024-09-16T01:44:43.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.286+0000 7f6e377fe640 1 -- 172.21.15.88:0/4274130876 reap_dead start 2024-09-16T01:44:43.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.287+0000 7f6e3d32c640 1 -- 172.21.15.88:0/4274130876 shutdown_connections 2024-09-16T01:44:43.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.287+0000 7f6e3d32c640 1 -- 172.21.15.88:0/4274130876 >> 172.21.15.88:0/4274130876 conn(0x7f6e3806d0f0 msgr2=0x7f6e3806d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.287+0000 7f6e3d32c640 1 -- 172.21.15.88:0/4274130876 shutdown_connections 2024-09-16T01:44:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.287+0000 7f6e3d32c640 1 -- 172.21.15.88:0/4274130876 wait complete. 2024-09-16T01:44:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.288+0000 7f6e3d32c640 1 Processor -- start 2024-09-16T01:44:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.288+0000 7f6e3d32c640 1 -- start start 2024-09-16T01:44:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.289+0000 7f6e3d32c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e24004300 0x7f6e38146980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.289+0000 7f6e3d32c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e3815a1c0 0x7f6e3815a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.289+0000 7f6e3d32c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e381475a0 0x7f6e38147a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.289+0000 7f6e3d32c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3815b9e0 con 0x7f6e24004300 2024-09-16T01:44:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.289+0000 7f6e3d32c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3815b860 con 0x7f6e381475a0 2024-09-16T01:44:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.289+0000 7f6e3d32c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6e38071ff0 con 0x7f6e3815a1c0 2024-09-16T01:44:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.289+0000 7f6e36ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e24004300 0x7f6e38146980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.289+0000 7f6e36ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e24004300 0x7f6e38146980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41576/0 (socket says 172.21.15.88:41576) 2024-09-16T01:44:43.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.289+0000 7f6e36ffd640 1 -- 172.21.15.88:0/2160875675 learned_addr learned my addr 172.21.15.88:0/2160875675 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:43.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.289+0000 7f6e377fe640 1 --2- 172.21.15.88:0/2160875675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e381475a0 0x7f6e38147a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:43.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.289+0000 7f6e367fc640 1 --2- 172.21.15.88:0/2160875675 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e3815a1c0 0x7f6e3815a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:43.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.289+0000 7f6e36ffd640 1 -- 172.21.15.88:0/2160875675 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e3815a1c0 msgr2=0x7f6e3815a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:43.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.289+0000 7f6e36ffd640 1 --2- 172.21.15.88:0/2160875675 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e3815a1c0 0x7f6e3815a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:43.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.289+0000 7f6e36ffd640 1 -- 172.21.15.88:0/2160875675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e381475a0 msgr2=0x7f6e38147a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:43.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.289+0000 7f6e36ffd640 1 --2- 172.21.15.88:0/2160875675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e381475a0 0x7f6e38147a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:43.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.289+0000 7f6e36ffd640 1 -- 172.21.15.88:0/2160875675 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e38148370 con 0x7f6e24004300 2024-09-16T01:44:43.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.289+0000 7f6e377fe640 1 --2- 172.21.15.88:0/2160875675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e381475a0 0x7f6e38147a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:43.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.289+0000 7f6e36ffd640 1 --2- 172.21.15.88:0/2160875675 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e24004300 0x7f6e38146980 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f6e28010b50 tx=0x7f6e2800e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:43.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.289+0000 7f6e2f7fe640 1 -- 172.21.15.88:0/2160875675 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e28003b70 con 0x7f6e24004300 2024-09-16T01:44:43.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.289+0000 7f6e2f7fe640 1 -- 172.21.15.88:0/2160875675 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e28003d10 con 0x7f6e24004300 2024-09-16T01:44:43.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.289+0000 7f6e3d32c640 1 -- 172.21.15.88:0/2160875675 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e38148650 con 0x7f6e24004300 2024-09-16T01:44:43.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.289+0000 7f6e2f7fe640 1 -- 172.21.15.88:0/2160875675 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e280052f0 con 0x7f6e24004300 2024-09-16T01:44:43.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.289+0000 7f6e3d32c640 1 -- 172.21.15.88:0/2160875675 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e38148c10 con 0x7f6e24004300 2024-09-16T01:44:43.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.291+0000 7f6e3d32c640 1 -- 172.21.15.88:0/2160875675 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e380cdfd0 con 0x7f6e24004300 2024-09-16T01:44:43.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.291+0000 7f6e2f7fe640 1 -- 172.21.15.88:0/2160875675 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6e28005490 con 0x7f6e24004300 2024-09-16T01:44:43.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.291+0000 7f6e2f7fe640 1 --2- 172.21.15.88:0/2160875675 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6e1806c2e0 0x7f6e1806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:43.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.291+0000 7f6e2f7fe640 1 -- 172.21.15.88:0/2160875675 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 8740+0+0 (secure 0 0 0) 0x7f6e2808e750 con 0x7f6e24004300 2024-09-16T01:44:43.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.291+0000 7f6e367fc640 1 --2- 172.21.15.88:0/2160875675 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6e1806c2e0 0x7f6e1806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:43.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.292+0000 7f6e367fc640 1 --2- 172.21.15.88:0/2160875675 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6e1806c2e0 0x7f6e1806e7a0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f6e20033000 tx=0x7f6e20008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:43.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.295+0000 7f6e2f7fe640 1 -- 172.21.15.88:0/2160875675 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6e380cdfd0 con 0x7f6e24004300 2024-09-16T01:44:43.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:43.466+0000 7f6e3d32c640 1 -- 172.21.15.88:0/2160875675 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6e380b5b80 con 0x7f6e24004300 2024-09-16T01:44:44.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.162+0000 7f6e2f7fe640 1 -- 172.21.15.88:0/2160875675 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v129) ==== 169+0+0 (secure 0 0 0) 0x7f6e28016030 con 0x7f6e24004300 2024-09-16T01:44:44.165 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_data' removed 2024-09-16T01:44:44.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.166+0000 7f6e3d32c640 1 -- 172.21.15.88:0/2160875675 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6e1806c2e0 msgr2=0x7f6e1806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:44.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.166+0000 7f6e3d32c640 1 --2- 172.21.15.88:0/2160875675 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6e1806c2e0 0x7f6e1806e7a0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f6e20033000 tx=0x7f6e20008030 comp rx=0 tx=0).stop 2024-09-16T01:44:44.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.166+0000 7f6e3d32c640 1 -- 172.21.15.88:0/2160875675 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e24004300 msgr2=0x7f6e38146980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:44.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.166+0000 7f6e3d32c640 1 --2- 172.21.15.88:0/2160875675 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e24004300 0x7f6e38146980 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f6e28010b50 tx=0x7f6e2800e6c0 comp rx=0 tx=0).stop 2024-09-16T01:44:44.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.166+0000 7f6e3d32c640 1 -- 172.21.15.88:0/2160875675 shutdown_connections 2024-09-16T01:44:44.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.166+0000 7f6e3d32c640 1 --2- 172.21.15.88:0/2160875675 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6e1806c2e0 0x7f6e1806e7a0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:44.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.166+0000 7f6e3d32c640 1 --2- 172.21.15.88:0/2160875675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e381475a0 0x7f6e38147a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:44.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.166+0000 7f6e3d32c640 1 --2- 172.21.15.88:0/2160875675 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6e3815a1c0 0x7f6e3815a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:44.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.166+0000 7f6e3d32c640 1 --2- 172.21.15.88:0/2160875675 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e24004300 0x7f6e38146980 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:44.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.166+0000 7f6e3d32c640 1 -- 172.21.15.88:0/2160875675 >> 172.21.15.88:0/2160875675 conn(0x7f6e3806d0f0 msgr2=0x7f6e3815c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:44.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.166+0000 7f6e3d32c640 1 -- 172.21.15.88:0/2160875675 shutdown_connections 2024-09-16T01:44:44.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.166+0000 7f6e3d32c640 1 -- 172.21.15.88:0/2160875675 wait complete. 2024-09-16T01:44:44.183 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-16T01:44:44.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.290+0000 7fda0edc4640 1 Processor -- start 2024-09-16T01:44:44.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.290+0000 7fda0edc4640 1 -- start start 2024-09-16T01:44:44.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.291+0000 7fda0edc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda080b6890 0x7fda080b3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.291+0000 7fda0edc4640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda080cf2a0 con 0x7fda08162150 2024-09-16T01:44:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.291+0000 7fda0edc4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda080cf9b0 con 0x7fda08161d70 2024-09-16T01:44:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.291+0000 7fda0edc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fda080cf760 con 0x7fda080b6890 2024-09-16T01:44:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.291+0000 7fda0d361640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fda08162150 0x7fda080d22d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46246/0 (socket says 172.21.15.88:46246) 2024-09-16T01:44:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.291+0000 7fda0d361640 1 -- 172.21.15.88:0/1539059119 learned_addr learned my addr 172.21.15.88:0/1539059119 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.291+0000 7fda07fff640 1 --2- 172.21.15.88:0/1539059119 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda080b6890 0x7fda080b3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.291+0000 7fda077fe640 1 -- 172.21.15.88:0/1539059119 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 470882470 0 0) 0x7fda080cf2a0 con 0x7fda08162150 2024-09-16T01:44:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.291+0000 7fda077fe640 1 -- 172.21.15.88:0/1539059119 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9f8003520 con 0x7fda08162150 2024-09-16T01:44:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.291+0000 7fda077fe640 1 -- 172.21.15.88:0/1539059119 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1404843577 0 0) 0x7fda080cf9b0 con 0x7fda08161d70 2024-09-16T01:44:44.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.291+0000 7fda077fe640 1 -- 172.21.15.88:0/1539059119 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda080cf2a0 con 0x7fda08161d70 2024-09-16T01:44:44.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.291+0000 7fda07fff640 1 -- 172.21.15.88:0/1539059119 >> v1:172.21.15.154:6789/0 conn(0x7fda08161d70 legacy=0x7fda080b6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:44.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.291+0000 7fda07fff640 1 -- 172.21.15.88:0/1539059119 >> v1:172.21.15.88:6789/0 conn(0x7fda08162150 legacy=0x7fda080d22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:44.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.291+0000 7fda07fff640 1 -- 172.21.15.88:0/1539059119 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda080cfc90 con 0x7fda080b6890 2024-09-16T01:44:44.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.292+0000 7fda07fff640 1 --2- 172.21.15.88:0/1539059119 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda080b6890 0x7fda080b3390 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd9f40097d0 tx=0x7fd9f402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38cb6ff085172a93 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:44.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.292+0000 7fda077fe640 1 -- 172.21.15.88:0/1539059119 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f4037070 con 0x7fda080b6890 2024-09-16T01:44:44.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.292+0000 7fda077fe640 1 -- 172.21.15.88:0/1539059119 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9f4032440 con 0x7fda080b6890 2024-09-16T01:44:44.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.292+0000 7fda077fe640 1 -- 172.21.15.88:0/1539059119 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f40327f0 con 0x7fda080b6890 2024-09-16T01:44:44.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.293+0000 7fda0edc4640 1 -- 172.21.15.88:0/1539059119 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda080b6890 msgr2=0x7fda080b3390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:44.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.293+0000 7fda0edc4640 1 --2- 172.21.15.88:0/1539059119 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda080b6890 0x7fda080b3390 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd9f40097d0 tx=0x7fd9f402a210 comp rx=0 tx=0).stop 2024-09-16T01:44:44.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.293+0000 7fda0edc4640 1 -- 172.21.15.88:0/1539059119 shutdown_connections 2024-09-16T01:44:44.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.293+0000 7fda0edc4640 1 --2- 172.21.15.88:0/1539059119 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda080b6890 0x7fda080b3390 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:44.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.293+0000 7fda0edc4640 1 -- 172.21.15.88:0/1539059119 >> 172.21.15.88:0/1539059119 conn(0x7fda0806c500 msgr2=0x7fda0806c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:44.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.293+0000 7fda0edc4640 1 -- 172.21.15.88:0/1539059119 shutdown_connections 2024-09-16T01:44:44.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.293+0000 7fda0edc4640 1 -- 172.21.15.88:0/1539059119 wait complete. 2024-09-16T01:44:44.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.294+0000 7fda0edc4640 1 Processor -- start 2024-09-16T01:44:44.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.294+0000 7fda0edc4640 1 -- start start 2024-09-16T01:44:44.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.295+0000 7fda0edc4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda080b6890 0x7fda0814daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:44.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.295+0000 7fda0edc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda08161d70 0x7fda0815d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:44.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.295+0000 7fda0edc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda08162150 0x7fda0815dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:44.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.295+0000 7fda0edc4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fda080d43e0 con 0x7fda080b6890 2024-09-16T01:44:44.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.295+0000 7fda0edc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fda080d4260 con 0x7fda08162150 2024-09-16T01:44:44.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.295+0000 7fda0edc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fda080d4560 con 0x7fda08161d70 2024-09-16T01:44:44.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.295+0000 7fda0cb60640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda080b6890 0x7fda0814daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:44.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.295+0000 7fda0cb60640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda080b6890 0x7fda0814daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41592/0 (socket says 172.21.15.88:41592) 2024-09-16T01:44:44.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.295+0000 7fda0cb60640 1 -- 172.21.15.88:0/4199548275 learned_addr learned my addr 172.21.15.88:0/4199548275 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:44.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.295+0000 7fda07fff640 1 --2- 172.21.15.88:0/4199548275 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda08161d70 0x7fda0815d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:44.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.295+0000 7fda0d361640 1 --2- 172.21.15.88:0/4199548275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda08162150 0x7fda0815dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:44.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.295+0000 7fda0cb60640 1 -- 172.21.15.88:0/4199548275 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda08161d70 msgr2=0x7fda0815d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:44.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.295+0000 7fda0cb60640 1 --2- 172.21.15.88:0/4199548275 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda08161d70 0x7fda0815d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:44.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.295+0000 7fda0cb60640 1 -- 172.21.15.88:0/4199548275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda08162150 msgr2=0x7fda0815dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:44.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.295+0000 7fda0cb60640 1 --2- 172.21.15.88:0/4199548275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda08162150 0x7fda0815dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:44.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.295+0000 7fda0cb60640 1 -- 172.21.15.88:0/4199548275 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda0815c5f0 con 0x7fda080b6890 2024-09-16T01:44:44.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.295+0000 7fda07fff640 1 --2- 172.21.15.88:0/4199548275 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda08161d70 0x7fda0815d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:44.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.295+0000 7fda0d361640 1 --2- 172.21.15.88:0/4199548275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda08162150 0x7fda0815dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:44.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.295+0000 7fda0cb60640 1 --2- 172.21.15.88:0/4199548275 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda080b6890 0x7fda0814daa0 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc00c500 tx=0x7fd9fc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:44.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.296+0000 7fda057fa640 1 -- 172.21.15.88:0/4199548275 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc0052d0 con 0x7fda080b6890 2024-09-16T01:44:44.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.296+0000 7fda0edc4640 1 -- 172.21.15.88:0/4199548275 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda0815c860 con 0x7fda080b6890 2024-09-16T01:44:44.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.296+0000 7fda057fa640 1 -- 172.21.15.88:0/4199548275 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9fc005490 con 0x7fda080b6890 2024-09-16T01:44:44.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.296+0000 7fda057fa640 1 -- 172.21.15.88:0/4199548275 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9fc012870 con 0x7fda080b6890 2024-09-16T01:44:44.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.296+0000 7fda0edc4640 1 -- 172.21.15.88:0/4199548275 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda0815ce20 con 0x7fda080b6890 2024-09-16T01:44:44.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.297+0000 7fda0edc4640 1 -- 172.21.15.88:0/4199548275 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda080ced50 con 0x7fda080b6890 2024-09-16T01:44:44.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.297+0000 7fda057fa640 1 -- 172.21.15.88:0/4199548275 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd9fc012a10 con 0x7fda080b6890 2024-09-16T01:44:44.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.298+0000 7fda057fa640 1 --2- 172.21.15.88:0/4199548275 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd9ec06c2b0 0x7fd9ec06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:44.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.298+0000 7fda057fa640 1 -- 172.21.15.88:0/4199548275 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 8297+0+0 (secure 0 0 0) 0x7fd9fc08bc40 con 0x7fda080b6890 2024-09-16T01:44:44.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.298+0000 7fda07fff640 1 --2- 172.21.15.88:0/4199548275 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd9ec06c2b0 0x7fd9ec06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:44.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.298+0000 7fda07fff640 1 --2- 172.21.15.88:0/4199548275 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd9ec06c2b0 0x7fd9ec06e770 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4034000 tx=0x7fd9f402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:44.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.302+0000 7fda057fa640 1 -- 172.21.15.88:0/4199548275 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd9fc059950 con 0x7fda080b6890 2024-09-16T01:44:44.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.467+0000 7fda0edc4640 1 -- 172.21.15.88:0/4199548275 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fda080ba2d0 con 0x7fda080b6890 2024-09-16T01:44:44.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.471+0000 7fda057fa640 1 -- 172.21.15.88:0/4199548275 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) ==== 104+0+0 (secure 0 0 0) 0x7fd9fc059250 con 0x7fda080b6890 2024-09-16T01:44:44.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.473+0000 7fda0edc4640 1 -- 172.21.15.88:0/4199548275 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd9ec06c2b0 msgr2=0x7fd9ec06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:44.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.473+0000 7fda0edc4640 1 --2- 172.21.15.88:0/4199548275 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd9ec06c2b0 0x7fd9ec06e770 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4034000 tx=0x7fd9f402e000 comp rx=0 tx=0).stop 2024-09-16T01:44:44.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.473+0000 7fda0edc4640 1 -- 172.21.15.88:0/4199548275 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda080b6890 msgr2=0x7fda0814daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:44.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.473+0000 7fda0edc4640 1 --2- 172.21.15.88:0/4199548275 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda080b6890 0x7fda0814daa0 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc00c500 tx=0x7fd9fc00c9c0 comp rx=0 tx=0).stop 2024-09-16T01:44:44.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.473+0000 7fda0edc4640 1 -- 172.21.15.88:0/4199548275 shutdown_connections 2024-09-16T01:44:44.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.473+0000 7fda0edc4640 1 --2- 172.21.15.88:0/4199548275 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd9ec06c2b0 0x7fd9ec06e770 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:44.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.473+0000 7fda0edc4640 1 --2- 172.21.15.88:0/4199548275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda08162150 0x7fda0815dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:44.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.473+0000 7fda0edc4640 1 --2- 172.21.15.88:0/4199548275 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda08161d70 0x7fda0815d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:44.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.473+0000 7fda0edc4640 1 --2- 172.21.15.88:0/4199548275 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fda080b6890 0x7fda0814daa0 unknown :-1 s=CLOSED pgs=2304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:44.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.473+0000 7fda0edc4640 1 -- 172.21.15.88:0/4199548275 >> 172.21.15.88:0/4199548275 conn(0x7fda0806c500 msgr2=0x7fda080b4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:44.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.473+0000 7fda0edc4640 1 -- 172.21.15.88:0/4199548275 shutdown_connections 2024-09-16T01:44:44.477 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.473+0000 7fda0edc4640 1 -- 172.21.15.88:0/4199548275 wait complete. 2024-09-16T01:44:44.488 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_damage.TestDamage.test_object_deletion' 2024-09-16T01:44:44.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.584+0000 7f68349bd640 1 Processor -- start 2024-09-16T01:44:44.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.584+0000 7f68349bd640 1 -- start start 2024-09-16T01:44:44.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.585+0000 7f68349bd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6830155800 0x7f68300b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:44.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.585+0000 7f68349bd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f683015c330 0x7f68300b4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:44.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.585+0000 7f68349bd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68300b50a0 con 0x7f68300cf570 2024-09-16T01:44:44.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.585+0000 7f68349bd640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f68300b4e60 con 0x7f683015c330 2024-09-16T01:44:44.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.585+0000 7f68349bd640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f683015c7a0 con 0x7f6830155800 2024-09-16T01:44:44.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.585+0000 7f6825d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f683015c330 0x7f68300b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:44.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.585+0000 7f6825d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f683015c330 0x7f68300b4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41606/0 (socket says 172.21.15.88:41606) 2024-09-16T01:44:44.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.585+0000 7f6825d9b640 1 -- 172.21.15.88:0/3524471698 learned_addr learned my addr 172.21.15.88:0/3524471698 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:44.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.585+0000 7f682ed9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f68300cf570 0x7f68300cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46260/0 (socket says 172.21.15.88:46260) 2024-09-16T01:44:44.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.585+0000 7f682e59c640 1 --2- 172.21.15.88:0/3524471698 >> v2:172.21.15.154:3300/0 conn(0x7f6830155800 0x7f68300b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:44.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.585+0000 7f6825d9b640 1 -- 172.21.15.88:0/3524471698 >> v2:172.21.15.154:3300/0 conn(0x7f6830155800 msgr2=0x7f68300b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:44.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.585+0000 7f6825d9b640 1 --2- 172.21.15.88:0/3524471698 >> v2:172.21.15.154:3300/0 conn(0x7f6830155800 0x7f68300b6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:44.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.585+0000 7f682dd9b640 1 -- 172.21.15.88:0/3524471698 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1923180045 0 0) 0x7f68300b50a0 con 0x7f68300cf570 2024-09-16T01:44:44.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.585+0000 7f6825d9b640 1 -- 172.21.15.88:0/3524471698 >> v1:172.21.15.88:6789/0 conn(0x7f68300cf570 legacy=0x7f68300cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:44.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.585+0000 7f6825d9b640 1 -- 172.21.15.88:0/3524471698 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68300b5380 con 0x7f683015c330 2024-09-16T01:44:44.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.585+0000 7f682e59c640 1 --2- 172.21.15.88:0/3524471698 >> v2:172.21.15.154:3300/0 conn(0x7f6830155800 0x7f68300b6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:44.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.586+0000 7f6825d9b640 1 --2- 172.21.15.88:0/3524471698 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f683015c330 0x7f68300b4920 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f683006c7f0 tx=0x7f681802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acacd18f5933c04b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:44.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.586+0000 7f682dd9b640 1 -- 172.21.15.88:0/3524471698 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6818037070 con 0x7f683015c330 2024-09-16T01:44:44.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.586+0000 7f682dd9b640 1 -- 172.21.15.88:0/3524471698 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6818032440 con 0x7f683015c330 2024-09-16T01:44:44.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.586+0000 7f682dd9b640 1 -- 172.21.15.88:0/3524471698 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6818032810 con 0x7f683015c330 2024-09-16T01:44:44.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.586+0000 7f68349bd640 1 -- 172.21.15.88:0/3524471698 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f683015c330 msgr2=0x7f68300b4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:44.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.586+0000 7f68349bd640 1 --2- 172.21.15.88:0/3524471698 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f683015c330 0x7f68300b4920 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f683006c7f0 tx=0x7f681802a270 comp rx=0 tx=0).stop 2024-09-16T01:44:44.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.587+0000 7f68349bd640 1 -- 172.21.15.88:0/3524471698 shutdown_connections 2024-09-16T01:44:44.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.587+0000 7f68349bd640 1 --2- 172.21.15.88:0/3524471698 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f683015c330 0x7f68300b4920 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:44.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.587+0000 7f68349bd640 1 --2- 172.21.15.88:0/3524471698 >> v2:172.21.15.154:3300/0 conn(0x7f6830155800 0x7f68300b6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:44.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.587+0000 7f68349bd640 1 -- 172.21.15.88:0/3524471698 >> 172.21.15.88:0/3524471698 conn(0x7f683006bc10 msgr2=0x7f683006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:44.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.587+0000 7f68349bd640 1 -- 172.21.15.88:0/3524471698 shutdown_connections 2024-09-16T01:44:44.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.587+0000 7f68349bd640 1 -- 172.21.15.88:0/3524471698 wait complete. 2024-09-16T01:44:44.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.588+0000 7f68349bd640 1 Processor -- start 2024-09-16T01:44:44.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.588+0000 7f68349bd640 1 -- start start 2024-09-16T01:44:44.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.589+0000 7f68349bd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68300cf570 0x7f68301f3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:44.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.589+0000 7f68349bd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6830155800 0x7f6830151af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:44.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.589+0000 7f68349bd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f683015c330 0x7f68301548e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:44.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.589+0000 7f68349bd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f68300b5c20 con 0x7f6830155800 2024-09-16T01:44:44.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.589+0000 7f68349bd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f68300b5940 con 0x7f683015c330 2024-09-16T01:44:44.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.589+0000 7f68349bd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6830155be0 con 0x7f68300cf570 2024-09-16T01:44:44.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.589+0000 7f6825d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6830155800 0x7f6830151af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:44.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.589+0000 7f6825d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6830155800 0x7f6830151af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41608/0 (socket says 172.21.15.88:41608) 2024-09-16T01:44:44.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.589+0000 7f6825d9b640 1 -- 172.21.15.88:0/3533316244 learned_addr learned my addr 172.21.15.88:0/3533316244 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:44.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.589+0000 7f682e59c640 1 --2- 172.21.15.88:0/3533316244 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68300cf570 0x7f68301f3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:44.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.589+0000 7f682ed9d640 1 --2- 172.21.15.88:0/3533316244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f683015c330 0x7f68301548e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:44.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.589+0000 7f6825d9b640 1 -- 172.21.15.88:0/3533316244 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68300cf570 msgr2=0x7f68301f3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:44.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.589+0000 7f6825d9b640 1 --2- 172.21.15.88:0/3533316244 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68300cf570 0x7f68301f3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:44.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.589+0000 7f6825d9b640 1 -- 172.21.15.88:0/3533316244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f683015c330 msgr2=0x7f68301548e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:44.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.589+0000 7f6825d9b640 1 --2- 172.21.15.88:0/3533316244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f683015c330 0x7f68301548e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:44.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.589+0000 7f6825d9b640 1 -- 172.21.15.88:0/3533316244 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68301f23a0 con 0x7f6830155800 2024-09-16T01:44:44.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.589+0000 7f682ed9d640 1 --2- 172.21.15.88:0/3533316244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f683015c330 0x7f68301548e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:44.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.589+0000 7f6825d9b640 1 --2- 172.21.15.88:0/3533316244 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6830155800 0x7f6830151af0 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f6818000c00 tx=0x7f68180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:44.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.590+0000 7f68277fe640 1 -- 172.21.15.88:0/3533316244 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6818037040 con 0x7f6830155800 2024-09-16T01:44:44.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.590+0000 7f68277fe640 1 -- 172.21.15.88:0/3533316244 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6818004230 con 0x7f6830155800 2024-09-16T01:44:44.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.590+0000 7f68349bd640 1 -- 172.21.15.88:0/3533316244 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68301f2680 con 0x7f6830155800 2024-09-16T01:44:44.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.590+0000 7f68277fe640 1 -- 172.21.15.88:0/3533316244 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681803c670 con 0x7f6830155800 2024-09-16T01:44:44.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.590+0000 7f68349bd640 1 -- 172.21.15.88:0/3533316244 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68301f2c40 con 0x7f6830155800 2024-09-16T01:44:44.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.591+0000 7f68349bd640 1 -- 172.21.15.88:0/3533316244 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68300b6140 con 0x7f6830155800 2024-09-16T01:44:44.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.591+0000 7f68277fe640 1 -- 172.21.15.88:0/3533316244 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f681803a070 con 0x7f6830155800 2024-09-16T01:44:44.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.592+0000 7f68277fe640 1 --2- 172.21.15.88:0/3533316244 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f681006c2b0 0x7f681006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:44.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.592+0000 7f68277fe640 1 -- 172.21.15.88:0/3533316244 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 8297+0+0 (secure 0 0 0) 0x7f68180ac630 con 0x7f6830155800 2024-09-16T01:44:44.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.592+0000 7f682e59c640 1 --2- 172.21.15.88:0/3533316244 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f681006c2b0 0x7f681006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:44.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.592+0000 7f682e59c640 1 --2- 172.21.15.88:0/3533316244 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f681006c2b0 0x7f681006e770 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f682000a990 tx=0x7f6820008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:44.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.596+0000 7f68277fe640 1 -- 172.21.15.88:0/3533316244 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6818041020 con 0x7f6830155800 2024-09-16T01:44:44.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:44.832+0000 7f68349bd640 1 -- 172.21.15.88:0/3533316244 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_object_deletion"]} v 0) -- 0x7f683006e2a0 con 0x7f6830155800 2024-09-16T01:44:45.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.171+0000 7f68277fe640 1 -- 172.21.15.88:0/3533316244 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_object_deletion"]}]=0 v539) ==== 135+0+0 (secure 0 0 0) 0x7f681807a340 con 0x7f6830155800 2024-09-16T01:44:45.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.173+0000 7f68349bd640 1 -- 172.21.15.88:0/3533316244 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f681006c2b0 msgr2=0x7f681006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:45.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.173+0000 7f68349bd640 1 --2- 172.21.15.88:0/3533316244 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f681006c2b0 0x7f681006e770 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f682000a990 tx=0x7f6820008040 comp rx=0 tx=0).stop 2024-09-16T01:44:45.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.173+0000 7f68349bd640 1 -- 172.21.15.88:0/3533316244 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6830155800 msgr2=0x7f6830151af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:45.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.173+0000 7f68349bd640 1 --2- 172.21.15.88:0/3533316244 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6830155800 0x7f6830151af0 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f6818000c00 tx=0x7f68180046c0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.173+0000 7f68349bd640 1 -- 172.21.15.88:0/3533316244 shutdown_connections 2024-09-16T01:44:45.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.173+0000 7f68349bd640 1 --2- 172.21.15.88:0/3533316244 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f681006c2b0 0x7f681006e770 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.173+0000 7f68349bd640 1 --2- 172.21.15.88:0/3533316244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f683015c330 0x7f68301548e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.173+0000 7f68349bd640 1 --2- 172.21.15.88:0/3533316244 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6830155800 0x7f6830151af0 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.173+0000 7f68349bd640 1 --2- 172.21.15.88:0/3533316244 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68300cf570 0x7f68301f3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.173+0000 7f68349bd640 1 -- 172.21.15.88:0/3533316244 >> 172.21.15.88:0/3533316244 conn(0x7f683006bc10 msgr2=0x7f68300cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:45.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.173+0000 7f68349bd640 1 -- 172.21.15.88:0/3533316244 shutdown_connections 2024-09-16T01:44:45.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.173+0000 7f68349bd640 1 -- 172.21.15.88:0/3533316244 wait complete. 2024-09-16T01:44:45.196 INFO:tasks.cephfs_test_runner:That the MDS has a clean 'damaged' response to loss of any single metadata object ... ok 2024-09-16T01:44:45.196 INFO:tasks.cephfs_test_runner:Starting test: test_open_ino_errors (tasks.cephfs.test_damage.TestDamage) That errors encountered during opening inos are properly propagated 2024-09-16T01:44:45.197 INFO:tasks.cephfs_test_runner:test_open_ino_errors (tasks.cephfs.test_damage.TestDamage) 2024-09-16T01:44:45.197 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:44:45.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.294+0000 7f3e08378640 1 Processor -- start 2024-09-16T01:44:45.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.294+0000 7f3e08378640 1 -- start start 2024-09-16T01:44:45.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.295+0000 7f3e08378640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e0006bc10 0x7f3e0006bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:45.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.295+0000 7f3e08378640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e0006c530 con 0x7f3e000713c0 2024-09-16T01:44:45.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.295+0000 7f3e08378640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e0006e7b0 con 0x7f3e00071800 2024-09-16T01:44:45.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.295+0000 7f3e08378640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3e0006ea90 con 0x7f3e0006bc10 2024-09-16T01:44:45.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.295+0000 7f3e05913640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3e000713c0 0x7f3e000af510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46262/0 (socket says 172.21.15.88:46262) 2024-09-16T01:44:45.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e05913640 1 -- 172.21.15.88:0/2232724482 learned_addr learned my addr 172.21.15.88:0/2232724482 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:45.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e05112640 1 -- 172.21.15.88:0/2232724482 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2385284092 0 0) 0x7f3e0006c530 con 0x7f3e000713c0 2024-09-16T01:44:45.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e05112640 1 -- 172.21.15.88:0/2232724482 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dec003520 con 0x7f3e000713c0 2024-09-16T01:44:45.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e06114640 1 --2- 172.21.15.88:0/2232724482 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e0006bc10 0x7f3e0006bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:45.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e05112640 1 -- 172.21.15.88:0/2232724482 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2614555677 0 0) 0x7f3dec003520 con 0x7f3e000713c0 2024-09-16T01:44:45.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e05112640 1 -- 172.21.15.88:0/2232724482 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e0006bc10 msgr2=0x7f3e0006bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:45.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e05112640 1 --2- 172.21.15.88:0/2232724482 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e0006bc10 0x7f3e0006bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e05112640 1 -- 172.21.15.88:0/2232724482 >> v1:172.21.15.154:6789/0 conn(0x7f3e00071800 legacy=0x7f3e0006da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:45.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e05112640 1 -- 172.21.15.88:0/2232724482 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e000aea30 con 0x7f3e000713c0 2024-09-16T01:44:45.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e05112640 1 -- 172.21.15.88:0/2232724482 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f3dfc001670 con 0x7f3e000713c0 2024-09-16T01:44:45.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e05112640 1 -- 172.21.15.88:0/2232724482 >> v1:172.21.15.88:6789/0 conn(0x7f3e000713c0 legacy=0x7f3e000af510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:45.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e05112640 1 --2- 172.21.15.88:0/2232724482 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e0006bc10 0x7f3dec0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:45.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e05112640 1 --2- 172.21.15.88:0/2232724482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dec004010 0x7f3dec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:45.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e05112640 1 --2- 172.21.15.88:0/2232724482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3dec004d40 0x7f3dec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:45.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e05112640 1 -- 172.21.15.88:0/2232724482 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3e0006ea90 con 0x7f3dec004d40 2024-09-16T01:44:45.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e05112640 1 -- 172.21.15.88:0/2232724482 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3dec00bb80 con 0x7f3dec004010 2024-09-16T01:44:45.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e05112640 1 -- 172.21.15.88:0/2232724482 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3dec00bd80 con 0x7f3e0006bc10 2024-09-16T01:44:45.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e06915640 1 --2- 172.21.15.88:0/2232724482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3dec004d40 0x7f3dec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:45.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e06114640 1 --2- 172.21.15.88:0/2232724482 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e0006bc10 0x7f3dec0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:45.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e05913640 1 --2- 172.21.15.88:0/2232724482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dec004010 0x7f3dec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:45.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e06915640 1 -- 172.21.15.88:0/2232724482 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e0006bc10 msgr2=0x7f3dec0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:45.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e06915640 1 --2- 172.21.15.88:0/2232724482 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e0006bc10 0x7f3dec0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e06915640 1 -- 172.21.15.88:0/2232724482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dec004010 msgr2=0x7f3dec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:45.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e06915640 1 --2- 172.21.15.88:0/2232724482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dec004010 0x7f3dec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e06915640 1 -- 172.21.15.88:0/2232724482 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dec003520 con 0x7f3dec004d40 2024-09-16T01:44:45.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e06114640 1 --2- 172.21.15.88:0/2232724482 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e0006bc10 0x7f3dec0039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:45.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.296+0000 7f3e05913640 1 --2- 172.21.15.88:0/2232724482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dec004010 0x7f3dec004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:45.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.297+0000 7f3e06915640 1 --2- 172.21.15.88:0/2232724482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3dec004d40 0x7f3dec0051c0 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7f3e00071370 tx=0x7f3dec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4bf606c2dad320d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:45.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.297+0000 7f3e05112640 1 -- 172.21.15.88:0/2232724482 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df8001a70 con 0x7f3dec004d40 2024-09-16T01:44:45.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.297+0000 7f3e05112640 1 -- 172.21.15.88:0/2232724482 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3df802ae10 con 0x7f3dec004d40 2024-09-16T01:44:45.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.297+0000 7f3e05112640 1 -- 172.21.15.88:0/2232724482 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df8003bb0 con 0x7f3dec004d40 2024-09-16T01:44:45.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.298+0000 7f3e08378640 1 -- 172.21.15.88:0/2232724482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3dec004d40 msgr2=0x7f3dec0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:45.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.298+0000 7f3e08378640 1 --2- 172.21.15.88:0/2232724482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3dec004d40 0x7f3dec0051c0 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7f3e00071370 tx=0x7f3dec005830 comp rx=0 tx=0).stop 2024-09-16T01:44:45.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.298+0000 7f3e06915640 1 -- 172.21.15.88:0/2232724482 reap_dead start 2024-09-16T01:44:45.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.298+0000 7f3e08378640 1 -- 172.21.15.88:0/2232724482 shutdown_connections 2024-09-16T01:44:45.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.298+0000 7f3e08378640 1 -- 172.21.15.88:0/2232724482 >> 172.21.15.88:0/2232724482 conn(0x7f3e000b2430 msgr2=0x7f3e000b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:45.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.298+0000 7f3e08378640 1 -- 172.21.15.88:0/2232724482 shutdown_connections 2024-09-16T01:44:45.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.298+0000 7f3e08378640 1 -- 172.21.15.88:0/2232724482 wait complete. 2024-09-16T01:44:45.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.299+0000 7f3e08378640 1 Processor -- start 2024-09-16T01:44:45.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.299+0000 7f3e08378640 1 -- start start 2024-09-16T01:44:45.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.300+0000 7f3e08378640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3dec004d40 0x7f3e0013fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:45.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.300+0000 7f3e08378640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e001551b0 0x7f3e00155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:45.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.300+0000 7f3e08378640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e001423d0 0x7f3e00142880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:45.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.300+0000 7f3e08378640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3e0015ea10 con 0x7f3e001423d0 2024-09-16T01:44:45.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.300+0000 7f3e08378640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3e000aeff0 con 0x7f3e001551b0 2024-09-16T01:44:45.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.300+0000 7f3e08378640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3e00140b80 con 0x7f3dec004d40 2024-09-16T01:44:45.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.300+0000 7f3e06915640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e001423d0 0x7f3e00142880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:45.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.300+0000 7f3e06915640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e001423d0 0x7f3e00142880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41626/0 (socket says 172.21.15.88:41626) 2024-09-16T01:44:45.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.300+0000 7f3e06915640 1 -- 172.21.15.88:0/2754556062 learned_addr learned my addr 172.21.15.88:0/2754556062 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:45.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.300+0000 7f3e05913640 1 --2- 172.21.15.88:0/2754556062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e001551b0 0x7f3e00155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:45.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.300+0000 7f3e06114640 1 --2- 172.21.15.88:0/2754556062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3dec004d40 0x7f3e0013fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:45.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.300+0000 7f3e06915640 1 -- 172.21.15.88:0/2754556062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3dec004d40 msgr2=0x7f3e0013fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:45.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.300+0000 7f3e06915640 1 --2- 172.21.15.88:0/2754556062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3dec004d40 0x7f3e0013fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.300+0000 7f3e06915640 1 -- 172.21.15.88:0/2754556062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e001551b0 msgr2=0x7f3e00155630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:45.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.300+0000 7f3e06915640 1 --2- 172.21.15.88:0/2754556062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e001551b0 0x7f3e00155630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.300+0000 7f3e06915640 1 -- 172.21.15.88:0/2754556062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e00140290 con 0x7f3e001423d0 2024-09-16T01:44:45.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.300+0000 7f3e06915640 1 --2- 172.21.15.88:0/2754556062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e001423d0 0x7f3e00142880 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f3df8005040 tx=0x7f3df80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:45.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.301+0000 7f3df2ffd640 1 -- 172.21.15.88:0/2754556062 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df8037040 con 0x7f3e001423d0 2024-09-16T01:44:45.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.301+0000 7f3df2ffd640 1 -- 172.21.15.88:0/2754556062 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3df802ac10 con 0x7f3e001423d0 2024-09-16T01:44:45.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.301+0000 7f3e08378640 1 -- 172.21.15.88:0/2754556062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e0013ff80 con 0x7f3e001423d0 2024-09-16T01:44:45.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.301+0000 7f3df2ffd640 1 -- 172.21.15.88:0/2754556062 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df8001a70 con 0x7f3e001423d0 2024-09-16T01:44:45.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.301+0000 7f3e08378640 1 -- 172.21.15.88:0/2754556062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e00140530 con 0x7f3e001423d0 2024-09-16T01:44:45.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.302+0000 7f3e08378640 1 -- 172.21.15.88:0/2754556062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e00155b70 con 0x7f3e001423d0 2024-09-16T01:44:45.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.302+0000 7f3df2ffd640 1 -- 172.21.15.88:0/2754556062 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3df8042020 con 0x7f3e001423d0 2024-09-16T01:44:45.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.303+0000 7f3df2ffd640 1 --2- 172.21.15.88:0/2754556062 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3de406c2e0 0x7f3de406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:45.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.303+0000 7f3e06114640 1 --2- 172.21.15.88:0/2754556062 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3de406c2e0 0x7f3de406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:45.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.303+0000 7f3df2ffd640 1 -- 172.21.15.88:0/2754556062 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 8297+0+0 (secure 0 0 0) 0x7f3df80ab400 con 0x7f3e001423d0 2024-09-16T01:44:45.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.303+0000 7f3e06114640 1 --2- 172.21.15.88:0/2754556062 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3de406c2e0 0x7f3de406e7a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f3e001f7980 tx=0x7f3df4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:45.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.307+0000 7f3df2ffd640 1 -- 172.21.15.88:0/2754556062 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3df8079180 con 0x7f3e001423d0 2024-09-16T01:44:45.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.472+0000 7f3e08378640 1 -- 172.21.15.88:0/2754556062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3e0006bff0 con 0x7f3e001423d0 2024-09-16T01:44:45.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.473+0000 7f3df2ffd640 1 -- 172.21.15.88:0/2754556062 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) ==== 74+0+18068 (secure 0 0 0) 0x7f3df8078a80 con 0x7f3e001423d0 2024-09-16T01:44:45.476 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:45.477 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":129,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:44:44.162874+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:6837/4270197461":"2024-09-17T01:44:35.136295+0000","172.21.15.88:6835/4270197461":"2024-09-17T01:44:35.136295+0000","172.21.15.88:6836/2111557063":"2024-09-17T01:44:28.087623+0000","172.21.15.88:6834/2111557063":"2024-09-17T01:44:28.087623+0000","172.21.15.88:6836/1912564918":"2024-09-17T01:44:16.634793+0000","172.21.15.88:6834/1912564918":"2024-09-17T01:44:16.634793+0000","172.21.15.88:6836/3078464816":"2024-09-17T01:44:01.948759+0000","172.21.15.88:6834/3892583353":"2024-09-17T01:43:56.641946+0000","172.21.15.88:6836/4135629195":"2024-09-17T01:43:45.866139+0000","172.21.15.88:6836/3892583353":"2024-09-17T01:43:56.641946+0000","172.21.15.88:6834/4135629195":"2024-09-17T01:43:45.866139+0000","172.21.15.88:6836/3121519215":"2024-09-17T01:43:30.669208+0000","172.21.15.88:6834/3121519215":"2024-09-17T01:43:30.669208+0000","172.21.15.88:6836/2555776364":"2024-09-17T01:43:25.635976+0000","172.21.15.88:6837/1372908065":"2024-09-17T01:43:12.443265+0000","172.21.15.88:6835/1372908065":"2024-09-17T01:43:12.443265+0000","172.21.15.88:6836/650399848":"2024-09-17T01:43:08.279879+0000","172.21.15.88:6835/1255135754":"2024-09-17T01:41:41.905258+0000","172.21.15.88:6836/3728363359":"2024-09-17T01:42:05.412133+0000","172.21.15.88:6837/293764774":"2024-09-17T01:41:16.203758+0000","172.21.15.88:6837/1255135754":"2024-09-17T01:41:41.905258+0000","172.21.15.88:6836/2918194823":"2024-09-17T01:41:29.275696+0000","172.21.15.88:6837/3336935653":"2024-09-17T01:41:44.295023+0000","172.21.15.88:6836/981209627":"2024-09-17T01:41:51.268495+0000","172.21.15.88:6835/293764774":"2024-09-17T01:41:16.203758+0000","172.21.15.88:6836/2680708900":"2024-09-17T01:42:41.629904+0000","172.21.15.88:6834/2957965650":"2024-09-17T01:41:12.181485+0000","172.21.15.88:6835/3336935653":"2024-09-17T01:41:44.295023+0000","172.21.15.88:6837/2285699282":"2024-09-17T01:42:29.541113+0000","172.21.15.88:6836/1162318664":"2024-09-17T01:41:23.257870+0000","172.21.15.88:6836/128058689":"2024-09-17T01:41:10.163743+0000","172.21.15.88:6834/981209627":"2024-09-17T01:41:51.268495+0000","172.21.15.88:6834/128058689":"2024-09-17T01:41:10.163743+0000","172.21.15.88:6836/1042715824":"2024-09-17T01:40:59.080515+0000","172.21.15.88:6836/2957965650":"2024-09-17T01:41:12.181485+0000","172.21.15.88:6835/2970185561":"2024-09-17T01:40:38.255774+0000","172.21.15.88:6834/3078464816":"2024-09-17T01:44:01.948759+0000","172.21.15.88:6834/1162318664":"2024-09-17T01:41:23.257870+0000","172.21.15.88:6836/1289932173":"2024-09-17T01:40:42.399399+0000","172.21.15.88:6835/3356432718":"2024-09-17T01:40:32.344252+0000","172.21.15.88:6836/1099469537":"2024-09-17T01:44:22.054752+0000","172.21.15.88:6834/2555776364":"2024-09-17T01:43:25.635976+0000","172.21.15.88:6837/2970185561":"2024-09-17T01:40:38.255774+0000","172.21.15.88:6835/2733458327":"2024-09-17T01:40:27.314712+0000","172.21.15.88:6837/3356432718":"2024-09-17T01:40:32.344252+0000","172.21.15.88:6834/2918194823":"2024-09-17T01:41:29.275696+0000","172.21.15.88:6834/4144876777":"2024-09-17T01:43:00.350808+0000","172.21.15.88:6835/2408579939":"2024-09-17T01:40:46.434492+0000","172.21.15.88:6835/3163343083":"2024-09-17T01:42:01.375396+0000","172.21.15.88:6835/4222250252":"2024-09-17T01:42:45.673956+0000","172.21.15.88:6835/4162274767":"2024-09-17T01:40:11.522091+0000","172.21.15.88:6835/2477051134":"2024-09-17T01:43:04.380783+0000","172.21.15.88:6834/1289932173":"2024-09-17T01:40:42.399399+0000","172.21.15.88:6837/2733458327":"2024-09-17T01:40:27.314712+0000","172.21.15.88:6834/2550498835":"2024-09-17T01:41:57.358287+0000","172.21.15.88:6836/2550498835":"2024-09-17T01:41:57.358287+0000","172.21.15.88:6835/2285699282":"2024-09-17T01:42:29.541113+0000","172.21.15.88:6837/3163343083":"2024-09-17T01:42:01.375396+0000","172.21.15.88:6834/3728363359":"2024-09-17T01:42:05.412133+0000","172.21.15.88:6834/1099469537":"2024-09-17T01:44:22.054752+0000","172.21.15.88:6835/880314974":"2024-09-17T01:42:21.485748+0000","172.21.15.88:6835/1936493048":"2024-09-17T01:42:19.133197+0000","172.21.15.88:6837/1936493048":"2024-09-17T01:42:19.133197+0000","172.21.15.88:6837/880314974":"2024-09-17T01:42:21.485748+0000","172.21.15.88:6834/3374372758":"2024-09-17T01:42:25.506299+0000","172.21.15.88:6836/3374372758":"2024-09-17T01:42:25.506299+0000","172.21.15.88:6836/201188184":"2024-09-17T01:42:33.276104+0000","172.21.15.88:6837/1990650099":"2024-09-17T01:42:37.599896+0000","172.21.15.88:6834/201188184":"2024-09-17T01:42:33.276104+0000","172.21.15.88:6835/1990650099":"2024-09-17T01:42:37.599896+0000","172.21.15.88:6837/4162274767":"2024-09-17T01:40:11.522091+0000","172.21.15.88:6837/2408579939":"2024-09-17T01:40:46.434492+0000","172.21.15.88:6834/2680708900":"2024-09-17T01:42:41.629904+0000","172.21.15.88:6837/4222250252":"2024-09-17T01:42:45.673956+0000","172.21.15.88:6834/1042715824":"2024-09-17T01:40:59.080515+0000","172.21.15.88:6834/4065900771":"2024-09-17T01:42:58.329389+0000","172.21.15.88:6836/4065900771":"2024-09-17T01:42:58.329389+0000","172.21.15.88:6836/4144876777":"2024-09-17T01:43:00.350808+0000","172.21.15.88:6837/2477051134":"2024-09-17T01:43:04.380783+0000","172.21.15.88:6834/650399848":"2024-09-17T01:43:08.279879+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:44:45.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.475+0000 7f3e08378640 1 -- 172.21.15.88:0/2754556062 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3de406c2e0 msgr2=0x7f3de406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:45.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.475+0000 7f3e08378640 1 --2- 172.21.15.88:0/2754556062 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3de406c2e0 0x7f3de406e7a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f3e001f7980 tx=0x7f3df4008040 comp rx=0 tx=0).stop 2024-09-16T01:44:45.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.475+0000 7f3e08378640 1 -- 172.21.15.88:0/2754556062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e001423d0 msgr2=0x7f3e00142880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:45.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.475+0000 7f3e08378640 1 --2- 172.21.15.88:0/2754556062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e001423d0 0x7f3e00142880 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f3df8005040 tx=0x7f3df80014a0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.475+0000 7f3e08378640 1 -- 172.21.15.88:0/2754556062 shutdown_connections 2024-09-16T01:44:45.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.475+0000 7f3e08378640 1 --2- 172.21.15.88:0/2754556062 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3de406c2e0 0x7f3de406e7a0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.475+0000 7f3e08378640 1 --2- 172.21.15.88:0/2754556062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e001423d0 0x7f3e00142880 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.475+0000 7f3e08378640 1 --2- 172.21.15.88:0/2754556062 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e001551b0 0x7f3e00155630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.475+0000 7f3e08378640 1 --2- 172.21.15.88:0/2754556062 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3dec004d40 0x7f3e0013fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.475+0000 7f3e08378640 1 -- 172.21.15.88:0/2754556062 >> 172.21.15.88:0/2754556062 conn(0x7f3e000b2430 msgr2=0x7f3e000cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:45.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.476+0000 7f3e08378640 1 -- 172.21.15.88:0/2754556062 shutdown_connections 2024-09-16T01:44:45.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.476+0000 7f3e08378640 1 -- 172.21.15.88:0/2754556062 wait complete. 2024-09-16T01:44:45.490 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T01:34:10.547178+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T01:44:45.490 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T01:44:45.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.645+0000 7f7786d2c640 1 Processor -- start 2024-09-16T01:44:45.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.645+0000 7f7786d2c640 1 -- start start 2024-09-16T01:44:45.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.646+0000 7f7786d2c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f778015b480 0x7f77800b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:45.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.646+0000 7f7786d2c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7780072700 0x7f778015bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:45.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.646+0000 7f7786d2c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77800ccff0 con 0x7f77800b60c0 2024-09-16T01:44:45.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.646+0000 7f7786d2c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f77800cd2d0 con 0x7f778015b480 2024-09-16T01:44:45.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.646+0000 7f7786d2c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f77800cd5b0 con 0x7f7780072700 2024-09-16T01:44:45.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.646+0000 7f7784ac8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f778015b480 0x7f77800b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:45.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.646+0000 7f77852c9640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7780072700 0x7f778015bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:45.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.646+0000 7f777ffff640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f77800b60c0 0x7f7780071ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:44174/0 (socket says 172.21.15.88:44174) 2024-09-16T01:44:45.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.646+0000 7f7784ac8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f778015b480 0x7f77800b5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:57366/0 (socket says 172.21.15.88:57366) 2024-09-16T01:44:45.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.646+0000 7f777ffff640 1 -- 172.21.15.88:0/3413211693 learned_addr learned my addr 172.21.15.88:0/3413211693 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:45.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.647+0000 7f77852c9640 1 -- 172.21.15.88:0/3413211693 >> v2:172.21.15.154:3300/0 conn(0x7f778015b480 msgr2=0x7f77800b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:45.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.647+0000 7f77852c9640 1 --2- 172.21.15.88:0/3413211693 >> v2:172.21.15.154:3300/0 conn(0x7f778015b480 0x7f77800b5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.647+0000 7f77852c9640 1 -- 172.21.15.88:0/3413211693 >> v1:172.21.15.154:6789/0 conn(0x7f77800b60c0 legacy=0x7f7780071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:45.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.647+0000 7f77852c9640 1 -- 172.21.15.88:0/3413211693 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77800cd890 con 0x7f7780072700 2024-09-16T01:44:45.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.647+0000 7f7784ac8640 1 --2- 172.21.15.88:0/3413211693 >> v2:172.21.15.154:3300/0 conn(0x7f778015b480 0x7f77800b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:44:45.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.648+0000 7f77852c9640 1 --2- 172.21.15.88:0/3413211693 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7780072700 0x7f778015bc60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7778009d90 tx=0x7f777802a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c1a0af713254787 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:45.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.648+0000 7f777f7fe640 1 -- 172.21.15.88:0/3413211693 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7778031600 con 0x7f7780072700 2024-09-16T01:44:45.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.648+0000 7f777f7fe640 1 -- 172.21.15.88:0/3413211693 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77780317a0 con 0x7f7780072700 2024-09-16T01:44:45.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.648+0000 7f777f7fe640 1 -- 172.21.15.88:0/3413211693 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7778031b70 con 0x7f7780072700 2024-09-16T01:44:45.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.648+0000 7f7786d2c640 1 -- 172.21.15.88:0/3413211693 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7780072700 msgr2=0x7f778015bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:45.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.648+0000 7f7786d2c640 1 --2- 172.21.15.88:0/3413211693 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7780072700 0x7f778015bc60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7778009d90 tx=0x7f777802a5d0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.648+0000 7f7786d2c640 1 -- 172.21.15.88:0/3413211693 shutdown_connections 2024-09-16T01:44:45.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.648+0000 7f7786d2c640 1 --2- 172.21.15.88:0/3413211693 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7780072700 0x7f778015bc60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.648+0000 7f7786d2c640 1 --2- 172.21.15.88:0/3413211693 >> v2:172.21.15.154:3300/0 conn(0x7f778015b480 0x7f77800b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.648+0000 7f7786d2c640 1 -- 172.21.15.88:0/3413211693 >> 172.21.15.88:0/3413211693 conn(0x7f778006d0f0 msgr2=0x7f778006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:45.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.648+0000 7f7786d2c640 1 -- 172.21.15.88:0/3413211693 shutdown_connections 2024-09-16T01:44:45.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.648+0000 7f7786d2c640 1 -- 172.21.15.88:0/3413211693 wait complete. 2024-09-16T01:44:45.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.650+0000 7f7786d2c640 1 Processor -- start 2024-09-16T01:44:45.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.650+0000 7f7786d2c640 1 -- start start 2024-09-16T01:44:45.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.650+0000 7f7786d2c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7780072700 0x7f7780153c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:45.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.650+0000 7f7786d2c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77800b60c0 0x7f7780151990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:45.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.650+0000 7f7786d2c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f778015b480 0x7f77800c7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:45.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.650+0000 7f7786d2c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f778015b9e0 con 0x7f778015b480 2024-09-16T01:44:45.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.650+0000 7f7786d2c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f778015b860 con 0x7f77800b60c0 2024-09-16T01:44:45.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.650+0000 7f7786d2c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f778015c1a0 con 0x7f7780072700 2024-09-16T01:44:45.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.651+0000 7f77852c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f778015b480 0x7f77800c7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:45.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.651+0000 7f77852c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f778015b480 0x7f77800c7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41638/0 (socket says 172.21.15.88:41638) 2024-09-16T01:44:45.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.651+0000 7f77852c9640 1 -- 172.21.15.88:0/2987180866 learned_addr learned my addr 172.21.15.88:0/2987180866 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:45.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.651+0000 7f777ffff640 1 --2- 172.21.15.88:0/2987180866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77800b60c0 0x7f7780151990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:45.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.651+0000 7f7784ac8640 1 --2- 172.21.15.88:0/2987180866 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7780072700 0x7f7780153c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:45.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.651+0000 7f77852c9640 1 -- 172.21.15.88:0/2987180866 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7780072700 msgr2=0x7f7780153c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:45.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.651+0000 7f77852c9640 1 --2- 172.21.15.88:0/2987180866 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7780072700 0x7f7780153c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.651+0000 7f77852c9640 1 -- 172.21.15.88:0/2987180866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77800b60c0 msgr2=0x7f7780151990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:45.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.651+0000 7f77852c9640 1 --2- 172.21.15.88:0/2987180866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77800b60c0 0x7f7780151990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.651+0000 7f77852c9640 1 -- 172.21.15.88:0/2987180866 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7780152020 con 0x7f778015b480 2024-09-16T01:44:45.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.651+0000 7f77852c9640 1 --2- 172.21.15.88:0/2987180866 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f778015b480 0x7f77800c7d20 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f777802abc0 tx=0x7f7778004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:45.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.651+0000 7f777d7fa640 1 -- 172.21.15.88:0/2987180866 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7778040070 con 0x7f778015b480 2024-09-16T01:44:45.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.651+0000 7f777d7fa640 1 -- 172.21.15.88:0/2987180866 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7778030c00 con 0x7f778015b480 2024-09-16T01:44:45.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.651+0000 7f7786d2c640 1 -- 172.21.15.88:0/2987180866 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7780152300 con 0x7f778015b480 2024-09-16T01:44:45.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.651+0000 7f777d7fa640 1 -- 172.21.15.88:0/2987180866 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7778009450 con 0x7f778015b480 2024-09-16T01:44:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.651+0000 7f7786d2c640 1 -- 172.21.15.88:0/2987180866 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77801528c0 con 0x7f778015b480 2024-09-16T01:44:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.652+0000 7f7786d2c640 1 -- 172.21.15.88:0/2987180866 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f778015e5e0 con 0x7f778015b480 2024-09-16T01:44:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.653+0000 7f777d7fa640 1 -- 172.21.15.88:0/2987180866 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7778039040 con 0x7f778015b480 2024-09-16T01:44:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.653+0000 7f777d7fa640 1 --2- 172.21.15.88:0/2987180866 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f776406c2b0 0x7f776406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.653+0000 7f777d7fa640 1 -- 172.21.15.88:0/2987180866 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 8297+0+0 (secure 0 0 0) 0x7f77780acf50 con 0x7f778015b480 2024-09-16T01:44:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.653+0000 7f7784ac8640 1 --2- 172.21.15.88:0/2987180866 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f776406c2b0 0x7f776406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.654+0000 7f7784ac8640 1 --2- 172.21.15.88:0/2987180866 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f776406c2b0 0x7f776406e770 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f7780157330 tx=0x7f7774009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.657+0000 7f777d7fa640 1 -- 172.21.15.88:0/2987180866 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f778015e5e0 con 0x7f778015b480 2024-09-16T01:44:45.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.822+0000 7f7786d2c640 1 -- 172.21.15.88:0/2987180866 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f778015bc60 con 0x7f778015b480 2024-09-16T01:44:45.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.822+0000 7f777d7fa640 1 -- 172.21.15.88:0/2987180866 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v129) ==== 93+0+10 (secure 0 0 0) 0x7f777807a690 con 0x7f778015b480 2024-09-16T01:44:45.826 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2024-09-16T01:44:45.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.825+0000 7f7786d2c640 1 -- 172.21.15.88:0/2987180866 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f776406c2b0 msgr2=0x7f776406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:45.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.825+0000 7f7786d2c640 1 --2- 172.21.15.88:0/2987180866 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f776406c2b0 0x7f776406e770 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f7780157330 tx=0x7f7774009190 comp rx=0 tx=0).stop 2024-09-16T01:44:45.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.825+0000 7f7786d2c640 1 -- 172.21.15.88:0/2987180866 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f778015b480 msgr2=0x7f77800c7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:45.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.825+0000 7f7786d2c640 1 --2- 172.21.15.88:0/2987180866 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f778015b480 0x7f77800c7d20 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f777802abc0 tx=0x7f7778004a40 comp rx=0 tx=0).stop 2024-09-16T01:44:45.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.825+0000 7f7786d2c640 1 -- 172.21.15.88:0/2987180866 shutdown_connections 2024-09-16T01:44:45.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.825+0000 7f7786d2c640 1 --2- 172.21.15.88:0/2987180866 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f776406c2b0 0x7f776406e770 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.825+0000 7f7786d2c640 1 --2- 172.21.15.88:0/2987180866 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f778015b480 0x7f77800c7d20 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.825+0000 7f7786d2c640 1 --2- 172.21.15.88:0/2987180866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77800b60c0 0x7f7780151990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.825+0000 7f7786d2c640 1 --2- 172.21.15.88:0/2987180866 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7780072700 0x7f7780153c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.825+0000 7f7786d2c640 1 -- 172.21.15.88:0/2987180866 >> 172.21.15.88:0/2987180866 conn(0x7f778006d0f0 msgr2=0x7f778015a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:45.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.825+0000 7f7786d2c640 1 -- 172.21.15.88:0/2987180866 shutdown_connections 2024-09-16T01:44:45.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.825+0000 7f7786d2c640 1 -- 172.21.15.88:0/2987180866 wait complete. 2024-09-16T01:44:45.838 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_damage.TestDamage.test_open_ino_errors' 2024-09-16T01:44:45.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.934+0000 7f00d7c8b640 1 Processor -- start 2024-09-16T01:44:45.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.934+0000 7f00d7c8b640 1 -- start start 2024-09-16T01:44:45.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.935+0000 7f00d7c8b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f00d00b5be0 0x7f00d015ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:45.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.935+0000 7f00d7c8b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00d00b29a0 0x7f00d00cde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:45.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.935+0000 7f00d7c8b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00d00b6830 con 0x7f00d015a9c0 2024-09-16T01:44:45.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.935+0000 7f00d7c8b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f00d015f410 con 0x7f00d00b5be0 2024-09-16T01:44:45.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.935+0000 7f00d7c8b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f00d00cd0b0 con 0x7f00d00b29a0 2024-09-16T01:44:45.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.935+0000 7f00d5a27640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f00d015a9c0 0x7f00d015e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46272/0 (socket says 172.21.15.88:46272) 2024-09-16T01:44:45.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.935+0000 7f00d5a27640 1 -- 172.21.15.88:0/1617969250 learned_addr learned my addr 172.21.15.88:0/1617969250 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:45.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.935+0000 7f00d5226640 1 --2- 172.21.15.88:0/1617969250 >> v2:172.21.15.154:3300/0 conn(0x7f00d00b5be0 0x7f00d015ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:45.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.935+0000 7f00d6228640 1 --2- 172.21.15.88:0/1617969250 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00d00b29a0 0x7f00d00cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:45.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.936+0000 7f00d4a25640 1 -- 172.21.15.88:0/1617969250 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1937197316 0 0) 0x7f00d00b6830 con 0x7f00d015a9c0 2024-09-16T01:44:45.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.936+0000 7f00d4a25640 1 -- 172.21.15.88:0/1617969250 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f00b8003520 con 0x7f00d015a9c0 2024-09-16T01:44:45.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.936+0000 7f00d5226640 1 -- 172.21.15.88:0/1617969250 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00d00b29a0 msgr2=0x7f00d00cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:45.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.936+0000 7f00d5226640 1 --2- 172.21.15.88:0/1617969250 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00d00b29a0 0x7f00d00cde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.936+0000 7f00d5226640 1 -- 172.21.15.88:0/1617969250 >> v1:172.21.15.88:6789/0 conn(0x7f00d015a9c0 legacy=0x7f00d015e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:45.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.936+0000 7f00d5226640 1 -- 172.21.15.88:0/1617969250 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d00cd390 con 0x7f00d00b5be0 2024-09-16T01:44:45.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.936+0000 7f00d5226640 1 --2- 172.21.15.88:0/1617969250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00d00b5be0 0x7f00d015ed10 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f00c400a940 tx=0x7f00c402b550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f46d9566bb24b1a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:45.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.937+0000 7f00d4a25640 1 -- 172.21.15.88:0/1617969250 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c400b040 con 0x7f00d00b5be0 2024-09-16T01:44:45.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.937+0000 7f00d4a25640 1 -- 172.21.15.88:0/1617969250 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00c4030710 con 0x7f00d00b5be0 2024-09-16T01:44:45.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.937+0000 7f00d4a25640 1 -- 172.21.15.88:0/1617969250 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c4030ac0 con 0x7f00d00b5be0 2024-09-16T01:44:45.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.937+0000 7f00d7c8b640 1 -- 172.21.15.88:0/1617969250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00d00b5be0 msgr2=0x7f00d015ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:45.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.937+0000 7f00d7c8b640 1 --2- 172.21.15.88:0/1617969250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00d00b5be0 0x7f00d015ed10 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f00c400a940 tx=0x7f00c402b550 comp rx=0 tx=0).stop 2024-09-16T01:44:45.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.937+0000 7f00d7c8b640 1 -- 172.21.15.88:0/1617969250 shutdown_connections 2024-09-16T01:44:45.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.937+0000 7f00d7c8b640 1 --2- 172.21.15.88:0/1617969250 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00d00b29a0 0x7f00d00cde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.937+0000 7f00d7c8b640 1 --2- 172.21.15.88:0/1617969250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00d00b5be0 0x7f00d015ed10 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.937+0000 7f00d7c8b640 1 -- 172.21.15.88:0/1617969250 >> 172.21.15.88:0/1617969250 conn(0x7f00d006bc10 msgr2=0x7f00d006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:45.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.937+0000 7f00d7c8b640 1 -- 172.21.15.88:0/1617969250 shutdown_connections 2024-09-16T01:44:45.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.937+0000 7f00d7c8b640 1 -- 172.21.15.88:0/1617969250 wait complete. 2024-09-16T01:44:45.945 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.938+0000 7f00d7c8b640 1 Processor -- start 2024-09-16T01:44:45.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.939+0000 7f00d7c8b640 1 -- start start 2024-09-16T01:44:45.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.939+0000 7f00d7c8b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00d00b29a0 0x7f00d0159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:45.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.939+0000 7f00d7c8b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00d00b5be0 0x7f00d0149bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:45.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.939+0000 7f00d7c8b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00d015a9c0 0x7f00d01f18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:45.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.939+0000 7f00d7c8b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f00d015ada0 con 0x7f00d015a9c0 2024-09-16T01:44:45.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.939+0000 7f00d7c8b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f00d00cd950 con 0x7f00d00b29a0 2024-09-16T01:44:45.946 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.939+0000 7f00d7c8b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f00d006de40 con 0x7f00d00b5be0 2024-09-16T01:44:45.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.939+0000 7f00d6228640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00d015a9c0 0x7f00d01f18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:45.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.939+0000 7f00d6228640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00d015a9c0 0x7f00d01f18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41646/0 (socket says 172.21.15.88:41646) 2024-09-16T01:44:45.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.940+0000 7f00d6228640 1 -- 172.21.15.88:0/1867090488 learned_addr learned my addr 172.21.15.88:0/1867090488 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:45.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.940+0000 7f00d5226640 1 --2- 172.21.15.88:0/1867090488 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00d00b5be0 0x7f00d0149bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:45.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.940+0000 7f00d5a27640 1 --2- 172.21.15.88:0/1867090488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00d00b29a0 0x7f00d0159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:45.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.940+0000 7f00d6228640 1 -- 172.21.15.88:0/1867090488 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00d00b5be0 msgr2=0x7f00d0149bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:45.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.940+0000 7f00d6228640 1 --2- 172.21.15.88:0/1867090488 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00d00b5be0 0x7f00d0149bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.940+0000 7f00d6228640 1 -- 172.21.15.88:0/1867090488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00d00b29a0 msgr2=0x7f00d0159aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:45.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.940+0000 7f00d6228640 1 --2- 172.21.15.88:0/1867090488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00d00b29a0 0x7f00d0159aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:45.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.940+0000 7f00d6228640 1 -- 172.21.15.88:0/1867090488 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d014c0e0 con 0x7f00d015a9c0 2024-09-16T01:44:45.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.940+0000 7f00d6228640 1 --2- 172.21.15.88:0/1867090488 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00d015a9c0 0x7f00d01f18c0 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7f00c800e860 tx=0x7f00c800ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:45.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.940+0000 7f00ce7fc640 1 -- 172.21.15.88:0/1867090488 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c8017070 con 0x7f00d015a9c0 2024-09-16T01:44:45.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.940+0000 7f00ce7fc640 1 -- 172.21.15.88:0/1867090488 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00c800c6d0 con 0x7f00d015a9c0 2024-09-16T01:44:45.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.940+0000 7f00d7c8b640 1 -- 172.21.15.88:0/1867090488 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00d014c3c0 con 0x7f00d015a9c0 2024-09-16T01:44:45.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.940+0000 7f00ce7fc640 1 -- 172.21.15.88:0/1867090488 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c800cac0 con 0x7f00d015a9c0 2024-09-16T01:44:45.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.940+0000 7f00d7c8b640 1 -- 172.21.15.88:0/1867090488 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00d014c980 con 0x7f00d015a9c0 2024-09-16T01:44:45.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.941+0000 7f00d7c8b640 1 -- 172.21.15.88:0/1867090488 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00d006e1e0 con 0x7f00d015a9c0 2024-09-16T01:44:45.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.943+0000 7f00ce7fc640 1 -- 172.21.15.88:0/1867090488 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f00c801a070 con 0x7f00d015a9c0 2024-09-16T01:44:45.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.943+0000 7f00ce7fc640 1 --2- 172.21.15.88:0/1867090488 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f00bc06c2b0 0x7f00bc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:45.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.943+0000 7f00ce7fc640 1 -- 172.21.15.88:0/1867090488 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 8297+0+0 (secure 0 0 0) 0x7f00c808c3a0 con 0x7f00d015a9c0 2024-09-16T01:44:45.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.943+0000 7f00d5a27640 1 --2- 172.21.15.88:0/1867090488 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f00bc06c2b0 0x7f00bc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:45.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.943+0000 7f00d5a27640 1 --2- 172.21.15.88:0/1867090488 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f00bc06c2b0 0x7f00bc06e770 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f00c0005c00 tx=0x7f00c0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:45.950 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:45.946+0000 7f00ce7fc640 1 -- 172.21.15.88:0/1867090488 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f00c8059a50 con 0x7f00d015a9c0 2024-09-16T01:44:46.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:46.174+0000 7f00d7c8b640 1 -- 172.21.15.88:0/1867090488 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_open_ino_errors"]} v 0) -- 0x7f00d0153420 con 0x7f00d015a9c0 2024-09-16T01:44:47.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.180+0000 7f00ce7fc640 1 -- 172.21.15.88:0/1867090488 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_open_ino_errors"]}]=0 v541) ==== 138+0+0 (secure 0 0 0) 0x7f00c805dd50 con 0x7f00d015a9c0 2024-09-16T01:44:47.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.183+0000 7f00d7c8b640 1 -- 172.21.15.88:0/1867090488 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f00bc06c2b0 msgr2=0x7f00bc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:47.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.183+0000 7f00d7c8b640 1 --2- 172.21.15.88:0/1867090488 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f00bc06c2b0 0x7f00bc06e770 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f00c0005c00 tx=0x7f00c0009040 comp rx=0 tx=0).stop 2024-09-16T01:44:47.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.183+0000 7f00d7c8b640 1 -- 172.21.15.88:0/1867090488 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00d015a9c0 msgr2=0x7f00d01f18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:47.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.183+0000 7f00d7c8b640 1 --2- 172.21.15.88:0/1867090488 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00d015a9c0 0x7f00d01f18c0 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7f00c800e860 tx=0x7f00c800ed20 comp rx=0 tx=0).stop 2024-09-16T01:44:47.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.183+0000 7f00d7c8b640 1 -- 172.21.15.88:0/1867090488 shutdown_connections 2024-09-16T01:44:47.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.183+0000 7f00d7c8b640 1 --2- 172.21.15.88:0/1867090488 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f00bc06c2b0 0x7f00bc06e770 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:47.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.183+0000 7f00d7c8b640 1 --2- 172.21.15.88:0/1867090488 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00d015a9c0 0x7f00d01f18c0 unknown :-1 s=CLOSED pgs=2313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:47.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.183+0000 7f00d7c8b640 1 --2- 172.21.15.88:0/1867090488 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f00d00b5be0 0x7f00d0149bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:47.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.183+0000 7f00d7c8b640 1 --2- 172.21.15.88:0/1867090488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00d00b29a0 0x7f00d0159aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:47.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.183+0000 7f00d7c8b640 1 -- 172.21.15.88:0/1867090488 >> 172.21.15.88:0/1867090488 conn(0x7f00d006bc10 msgr2=0x7f00d006bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:47.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.183+0000 7f00d7c8b640 1 -- 172.21.15.88:0/1867090488 shutdown_connections 2024-09-16T01:44:47.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.183+0000 7f00d7c8b640 1 -- 172.21.15.88:0/1867090488 wait complete. 2024-09-16T01:44:47.198 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-16T01:44:47.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.304+0000 7f92a9c4c640 1 Processor -- start 2024-09-16T01:44:47.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.304+0000 7f92a9c4c640 1 -- start start 2024-09-16T01:44:47.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.305+0000 7f92a9c4c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f92a415a050 0x7f92a415d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:47.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.305+0000 7f92a9c4c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f92a415ee90 0x7f92a4073600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:47.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.305+0000 7f92a9c4c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92a415a430 con 0x7f92a40731c0 2024-09-16T01:44:47.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.305+0000 7f92a9c4c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f92a4073b40 con 0x7f92a415ee90 2024-09-16T01:44:47.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.305+0000 7f92a9c4c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f92a415d9a0 con 0x7f92a415a050 2024-09-16T01:44:47.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.305+0000 7f92a3fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f92a415ee90 0x7f92a4073600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:47.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.305+0000 7f92a3fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f92a415ee90 0x7f92a4073600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41664/0 (socket says 172.21.15.88:41664) 2024-09-16T01:44:47.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.305+0000 7f92a3fff640 1 -- 172.21.15.88:0/622996500 learned_addr learned my addr 172.21.15.88:0/622996500 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:47.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.305+0000 7f92a37fe640 1 --2- 172.21.15.88:0/622996500 >> v2:172.21.15.154:3300/0 conn(0x7f92a415a050 0x7f92a415d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:47.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.305+0000 7f92a3fff640 1 -- 172.21.15.88:0/622996500 >> v2:172.21.15.154:3300/0 conn(0x7f92a415a050 msgr2=0x7f92a415d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:47.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.305+0000 7f92a3fff640 1 --2- 172.21.15.88:0/622996500 >> v2:172.21.15.154:3300/0 conn(0x7f92a415a050 0x7f92a415d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:47.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.305+0000 7f92a3fff640 1 -- 172.21.15.88:0/622996500 >> v1:172.21.15.154:6789/0 conn(0x7f92a40731c0 legacy=0x7f92a415ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:47.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.305+0000 7f92a3fff640 1 -- 172.21.15.88:0/622996500 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a40cd0a0 con 0x7f92a415ee90 2024-09-16T01:44:47.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.306+0000 7f92a3fff640 1 --2- 172.21.15.88:0/622996500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92a415ee90 0x7f92a4073600 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7f92a4069470 tx=0x7f929c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102ec44b1bf770ee server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:47.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.306+0000 7f92a27fc640 1 -- 172.21.15.88:0/622996500 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f929c036070 con 0x7f92a415ee90 2024-09-16T01:44:47.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.306+0000 7f92a27fc640 1 -- 172.21.15.88:0/622996500 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f929c031600 con 0x7f92a415ee90 2024-09-16T01:44:47.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.306+0000 7f92a27fc640 1 -- 172.21.15.88:0/622996500 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f929c0319d0 con 0x7f92a415ee90 2024-09-16T01:44:47.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.307+0000 7f92a9c4c640 1 -- 172.21.15.88:0/622996500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92a415ee90 msgr2=0x7f92a4073600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:47.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.307+0000 7f92a9c4c640 1 --2- 172.21.15.88:0/622996500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92a415ee90 0x7f92a4073600 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7f92a4069470 tx=0x7f929c02a660 comp rx=0 tx=0).stop 2024-09-16T01:44:47.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.307+0000 7f92a9c4c640 1 -- 172.21.15.88:0/622996500 shutdown_connections 2024-09-16T01:44:47.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.307+0000 7f92a9c4c640 1 --2- 172.21.15.88:0/622996500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92a415ee90 0x7f92a4073600 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:47.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.307+0000 7f92a9c4c640 1 --2- 172.21.15.88:0/622996500 >> v2:172.21.15.154:3300/0 conn(0x7f92a415a050 0x7f92a415d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:47.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.307+0000 7f92a9c4c640 1 -- 172.21.15.88:0/622996500 >> 172.21.15.88:0/622996500 conn(0x7f92a406d0f0 msgr2=0x7f92a406d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:47.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.307+0000 7f92a9c4c640 1 -- 172.21.15.88:0/622996500 shutdown_connections 2024-09-16T01:44:47.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.307+0000 7f92a9c4c640 1 -- 172.21.15.88:0/622996500 wait complete. 2024-09-16T01:44:47.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.308+0000 7f92a9c4c640 1 Processor -- start 2024-09-16T01:44:47.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.308+0000 7f92a9c4c640 1 -- start start 2024-09-16T01:44:47.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.309+0000 7f92a9c4c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92a40731c0 0x7f92a4152300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:47.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.309+0000 7f92a9c4c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92a415a050 0x7f92a4148bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:47.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.309+0000 7f92a9c4c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92a415ee90 0x7f92a40c6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:47.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.309+0000 7f92a9c4c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f92a40cd940 con 0x7f92a415a050 2024-09-16T01:44:47.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.309+0000 7f92a9c4c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f92a40cd660 con 0x7f92a40731c0 2024-09-16T01:44:47.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.309+0000 7f92a9c4c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f92a40cdc20 con 0x7f92a415ee90 2024-09-16T01:44:47.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.309+0000 7f92a2ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92a415a050 0x7f92a4148bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:47.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.309+0000 7f92a2ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92a415a050 0x7f92a4148bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41678/0 (socket says 172.21.15.88:41678) 2024-09-16T01:44:47.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.309+0000 7f92a2ffd640 1 -- 172.21.15.88:0/1283480921 learned_addr learned my addr 172.21.15.88:0/1283480921 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:47.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.309+0000 7f92a37fe640 1 --2- 172.21.15.88:0/1283480921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92a40731c0 0x7f92a4152300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:47.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.309+0000 7f92a3fff640 1 --2- 172.21.15.88:0/1283480921 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92a415ee90 0x7f92a40c6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:47.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.309+0000 7f92a2ffd640 1 -- 172.21.15.88:0/1283480921 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92a415ee90 msgr2=0x7f92a40c6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:47.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.309+0000 7f92a2ffd640 1 --2- 172.21.15.88:0/1283480921 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92a415ee90 0x7f92a40c6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:47.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.309+0000 7f92a2ffd640 1 -- 172.21.15.88:0/1283480921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92a40731c0 msgr2=0x7f92a4152300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:47.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.309+0000 7f92a2ffd640 1 --2- 172.21.15.88:0/1283480921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92a40731c0 0x7f92a4152300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:47.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.309+0000 7f92a2ffd640 1 -- 172.21.15.88:0/1283480921 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a40c69d0 con 0x7f92a415a050 2024-09-16T01:44:47.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.309+0000 7f92a3fff640 1 --2- 172.21.15.88:0/1283480921 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92a415ee90 0x7f92a40c6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:47.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.309+0000 7f92a2ffd640 1 --2- 172.21.15.88:0/1283480921 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92a415a050 0x7f92a4148bf0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f929800c460 tx=0x7f929800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:47.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.310+0000 7f928bfff640 1 -- 172.21.15.88:0/1283480921 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9298016020 con 0x7f92a415a050 2024-09-16T01:44:47.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.310+0000 7f928bfff640 1 -- 172.21.15.88:0/1283480921 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9298012870 con 0x7f92a415a050 2024-09-16T01:44:47.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.310+0000 7f928bfff640 1 -- 172.21.15.88:0/1283480921 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9298012c60 con 0x7f92a415a050 2024-09-16T01:44:47.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.310+0000 7f92a9c4c640 1 -- 172.21.15.88:0/1283480921 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92a41556c0 con 0x7f92a415a050 2024-09-16T01:44:47.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.310+0000 7f92a9c4c640 1 -- 172.21.15.88:0/1283480921 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92a4155c80 con 0x7f92a415a050 2024-09-16T01:44:47.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.311+0000 7f92a9c4c640 1 -- 172.21.15.88:0/1283480921 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92a40b36e0 con 0x7f92a415a050 2024-09-16T01:44:47.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.311+0000 7f928bfff640 1 -- 172.21.15.88:0/1283480921 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9298019020 con 0x7f92a415a050 2024-09-16T01:44:47.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.311+0000 7f928bfff640 1 --2- 172.21.15.88:0/1283480921 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f929006c2b0 0x7f929006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:47.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.312+0000 7f928bfff640 1 -- 172.21.15.88:0/1283480921 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 8297+0+0 (secure 0 0 0) 0x7f929808b830 con 0x7f92a415a050 2024-09-16T01:44:47.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.312+0000 7f92a37fe640 1 --2- 172.21.15.88:0/1283480921 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f929006c2b0 0x7f929006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:47.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.312+0000 7f92a37fe640 1 --2- 172.21.15.88:0/1283480921 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f929006c2b0 0x7f929006e770 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f9294006fd0 tx=0x7f9294008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:47.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.316+0000 7f928bfff640 1 -- 172.21.15.88:0/1283480921 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9298020070 con 0x7f92a415a050 2024-09-16T01:44:47.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.485+0000 7f92a9c4c640 1 -- 172.21.15.88:0/1283480921 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f92a40b4780 con 0x7f92a415a050 2024-09-16T01:44:47.492 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.490+0000 7f928bfff640 1 -- 172.21.15.88:0/1283480921 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) ==== 122+0+0 (secure 0 0 0) 0x7f9298059530 con 0x7f92a415a050 2024-09-16T01:44:47.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.493+0000 7f92a9c4c640 1 -- 172.21.15.88:0/1283480921 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f929006c2b0 msgr2=0x7f929006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:47.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.493+0000 7f92a9c4c640 1 --2- 172.21.15.88:0/1283480921 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f929006c2b0 0x7f929006e770 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f9294006fd0 tx=0x7f9294008040 comp rx=0 tx=0).stop 2024-09-16T01:44:47.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.493+0000 7f92a9c4c640 1 -- 172.21.15.88:0/1283480921 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92a415a050 msgr2=0x7f92a4148bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:47.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.493+0000 7f92a9c4c640 1 --2- 172.21.15.88:0/1283480921 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92a415a050 0x7f92a4148bf0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f929800c460 tx=0x7f929800c920 comp rx=0 tx=0).stop 2024-09-16T01:44:47.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.493+0000 7f92a9c4c640 1 -- 172.21.15.88:0/1283480921 shutdown_connections 2024-09-16T01:44:47.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.493+0000 7f92a9c4c640 1 --2- 172.21.15.88:0/1283480921 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f929006c2b0 0x7f929006e770 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:47.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.493+0000 7f92a9c4c640 1 --2- 172.21.15.88:0/1283480921 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92a415ee90 0x7f92a40c6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:47.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.493+0000 7f92a9c4c640 1 --2- 172.21.15.88:0/1283480921 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f92a415a050 0x7f92a4148bf0 unknown :-1 s=CLOSED pgs=2315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:47.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.493+0000 7f92a9c4c640 1 --2- 172.21.15.88:0/1283480921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92a40731c0 0x7f92a4152300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:47.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.493+0000 7f92a9c4c640 1 -- 172.21.15.88:0/1283480921 >> 172.21.15.88:0/1283480921 conn(0x7f92a406d0f0 msgr2=0x7f92a415dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:47.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.493+0000 7f92a9c4c640 1 -- 172.21.15.88:0/1283480921 shutdown_connections 2024-09-16T01:44:47.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.493+0000 7f92a9c4c640 1 -- 172.21.15.88:0/1283480921 wait complete. 2024-09-16T01:44:47.505 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T01:44:47.506 DEBUG:teuthology.orchestra.run.smithi088:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T01:44:47.551 DEBUG:teuthology.orchestra.run.smithi154:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T01:44:47.594 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T01:44:47.594 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:44:47.594 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:44:47.610 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:44:47.611 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:44:47.611 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:44:47.653 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056026,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1347 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T01:44:47.654 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:44:47.654 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:44:47.673 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:44:47.673 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:44:47.674 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:44:47.712 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056026,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1347 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T01:44:47.712 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-16T01:44:47.712 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:44:47.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.810+0000 7f04429fd640 1 Processor -- start 2024-09-16T01:44:47.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.810+0000 7f04429fd640 1 -- start start 2024-09-16T01:44:47.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.811+0000 7f04429fd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f043c0b6140 0x7f043c15c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:47.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.811+0000 7f04429fd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f043c0b50a0 con 0x7f043c155800 2024-09-16T01:44:47.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.811+0000 7f04429fd640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f043c0b5380 con 0x7f043c0cfc80 2024-09-16T01:44:47.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.811+0000 7f04429fd640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f043c0b6550 con 0x7f043c0b6140 2024-09-16T01:44:47.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.811+0000 7f0440f9a640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f043c0b6140 0x7f043c15c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:47.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.811+0000 7f043bfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f043c155800 0x7f043c0cf570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46288/0 (socket says 172.21.15.88:46288) 2024-09-16T01:44:47.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.811+0000 7f043bfff640 1 -- 172.21.15.88:0/3184214366 learned_addr learned my addr 172.21.15.88:0/3184214366 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:47.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.811+0000 7f0440f9a640 1 -- 172.21.15.88:0/3184214366 >> v1:172.21.15.154:6789/0 conn(0x7f043c0cfc80 legacy=0x7f043c15c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:47.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.811+0000 7f0440f9a640 1 -- 172.21.15.88:0/3184214366 >> v1:172.21.15.88:6789/0 conn(0x7f043c155800 legacy=0x7f043c0cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:47.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.811+0000 7f0440f9a640 1 -- 172.21.15.88:0/3184214366 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f043c0b5660 con 0x7f043c0b6140 2024-09-16T01:44:47.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.812+0000 7f0440f9a640 1 --2- 172.21.15.88:0/3184214366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f043c0b6140 0x7f043c15c330 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f0434009d90 tx=0x7f043402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e2916bbde847f96 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:47.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.812+0000 7f043affd640 1 -- 172.21.15.88:0/3184214366 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0434007740 con 0x7f043c0b6140 2024-09-16T01:44:47.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.812+0000 7f043affd640 1 -- 172.21.15.88:0/3184214366 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04340078e0 con 0x7f043c0b6140 2024-09-16T01:44:47.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.812+0000 7f043affd640 1 -- 172.21.15.88:0/3184214366 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0434007cb0 con 0x7f043c0b6140 2024-09-16T01:44:47.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.812+0000 7f04429fd640 1 -- 172.21.15.88:0/3184214366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f043c0b6140 msgr2=0x7f043c15c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:47.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.812+0000 7f04429fd640 1 --2- 172.21.15.88:0/3184214366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f043c0b6140 0x7f043c15c330 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f0434009d90 tx=0x7f043402a6a0 comp rx=0 tx=0).stop 2024-09-16T01:44:47.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.812+0000 7f04429fd640 1 -- 172.21.15.88:0/3184214366 shutdown_connections 2024-09-16T01:44:47.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.812+0000 7f04429fd640 1 --2- 172.21.15.88:0/3184214366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f043c0b6140 0x7f043c15c330 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:47.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.812+0000 7f04429fd640 1 -- 172.21.15.88:0/3184214366 >> 172.21.15.88:0/3184214366 conn(0x7f043c06bc10 msgr2=0x7f043c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:47.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.812+0000 7f04429fd640 1 -- 172.21.15.88:0/3184214366 shutdown_connections 2024-09-16T01:44:47.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.812+0000 7f04429fd640 1 -- 172.21.15.88:0/3184214366 wait complete. 2024-09-16T01:44:47.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.814+0000 7f04429fd640 1 Processor -- start 2024-09-16T01:44:47.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.814+0000 7f04429fd640 1 -- start start 2024-09-16T01:44:47.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.814+0000 7f04429fd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f043c0b6140 0x7f043c1512f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:47.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.815+0000 7f04429fd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f043c0cfc80 0x7f043c15a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:47.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.815+0000 7f04429fd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f043c155800 0x7f043c15a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:47.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.815+0000 7f04429fd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f043c155be0 con 0x7f043c0b6140 2024-09-16T01:44:47.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.815+0000 7f04429fd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f043c0b5c20 con 0x7f043c0cfc80 2024-09-16T01:44:47.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.815+0000 7f04429fd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f043c0b4920 con 0x7f043c155800 2024-09-16T01:44:47.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.815+0000 7f043bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f043c0b6140 0x7f043c1512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:47.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.815+0000 7f043bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f043c0b6140 0x7f043c1512f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41704/0 (socket says 172.21.15.88:41704) 2024-09-16T01:44:47.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.815+0000 7f043bfff640 1 -- 172.21.15.88:0/866359844 learned_addr learned my addr 172.21.15.88:0/866359844 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:47.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.815+0000 7f0440f9a640 1 --2- 172.21.15.88:0/866359844 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f043c155800 0x7f043c15a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:47.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.815+0000 7f043b7fe640 1 --2- 172.21.15.88:0/866359844 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f043c0cfc80 0x7f043c15a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:47.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.815+0000 7f043bfff640 1 -- 172.21.15.88:0/866359844 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f043c155800 msgr2=0x7f043c15a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:47.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.815+0000 7f043bfff640 1 --2- 172.21.15.88:0/866359844 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f043c155800 0x7f043c15a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:47.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.815+0000 7f043bfff640 1 -- 172.21.15.88:0/866359844 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f043c0cfc80 msgr2=0x7f043c15a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:47.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.815+0000 7f043bfff640 1 --2- 172.21.15.88:0/866359844 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f043c0cfc80 0x7f043c15a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:47.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.815+0000 7f043bfff640 1 -- 172.21.15.88:0/866359844 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f043c15e8f0 con 0x7f043c0b6140 2024-09-16T01:44:47.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.815+0000 7f0440f9a640 1 --2- 172.21.15.88:0/866359844 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f043c155800 0x7f043c15a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:47.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.815+0000 7f043b7fe640 1 --2- 172.21.15.88:0/866359844 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f043c0cfc80 0x7f043c15a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:47.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.815+0000 7f043bfff640 1 --2- 172.21.15.88:0/866359844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f043c0b6140 0x7f043c1512f0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f042c00c3c0 tx=0x7f042c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:47.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.815+0000 7f0438ff9640 1 -- 172.21.15.88:0/866359844 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042c017070 con 0x7f043c0b6140 2024-09-16T01:44:47.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.815+0000 7f0438ff9640 1 -- 172.21.15.88:0/866359844 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f042c005ce0 con 0x7f043c0b6140 2024-09-16T01:44:47.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.815+0000 7f04429fd640 1 -- 172.21.15.88:0/866359844 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f043c1f4560 con 0x7f043c0b6140 2024-09-16T01:44:47.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.815+0000 7f0438ff9640 1 -- 172.21.15.88:0/866359844 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042c0072f0 con 0x7f043c0b6140 2024-09-16T01:44:47.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.815+0000 7f04429fd640 1 -- 172.21.15.88:0/866359844 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f043c1f4b20 con 0x7f043c0b6140 2024-09-16T01:44:47.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.816+0000 7f04429fd640 1 -- 172.21.15.88:0/866359844 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f043c0b4cb0 con 0x7f043c0b6140 2024-09-16T01:44:47.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.817+0000 7f0438ff9640 1 -- 172.21.15.88:0/866359844 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f042c01a070 con 0x7f043c0b6140 2024-09-16T01:44:47.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.817+0000 7f0438ff9640 1 --2- 172.21.15.88:0/866359844 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f042806c2b0 0x7f042806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:47.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.817+0000 7f0438ff9640 1 -- 172.21.15.88:0/866359844 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 8297+0+0 (secure 0 0 0) 0x7f042c08b1c0 con 0x7f043c0b6140 2024-09-16T01:44:47.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.817+0000 7f043b7fe640 1 --2- 172.21.15.88:0/866359844 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f042806c2b0 0x7f042806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:47.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.818+0000 7f043b7fe640 1 --2- 172.21.15.88:0/866359844 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f042806c2b0 0x7f042806e770 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f0430007fd0 tx=0x7f0430009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:47.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:47.821+0000 7f0438ff9640 1 -- 172.21.15.88:0/866359844 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f042c058f50 con 0x7f043c0b6140 2024-09-16T01:44:48.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.019+0000 7f04429fd640 1 -- 172.21.15.88:0/866359844 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f043c0ccc50 con 0x7f043c0b6140 2024-09-16T01:44:48.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.019+0000 7f0438ff9640 1 -- 172.21.15.88:0/866359844 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 321 v321) ==== 95+0+3919 (secure 0 0 0) 0x7f042c058850 con 0x7f043c0b6140 2024-09-16T01:44:48.021 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:48.021 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":321,"btime":"2024-09-16T01:44:42:148013+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/922420111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":922420111},{"type":"v1","addr":"172.21.15.154:6835","nonce":922420111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":6004,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/4070614136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":4070614136},{"type":"v1","addr":"172.21.15.154:6834","nonce":4070614136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67},{"gid":10398,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/2633617067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2633617067},{"type":"v1","addr":"172.21.15.88:6836","nonce":2633617067}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":315},{"gid":10443,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/3726735780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":3726735780},{"type":"v1","addr":"172.21.15.88:6837","nonce":3726735780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":319}],"filesystems":[]} 2024-09-16T01:44:48.021 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 321 2024-09-16T01:44:48.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.021+0000 7f04429fd640 1 -- 172.21.15.88:0/866359844 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f042806c2b0 msgr2=0x7f042806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:48.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.021+0000 7f04429fd640 1 --2- 172.21.15.88:0/866359844 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f042806c2b0 0x7f042806e770 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f0430007fd0 tx=0x7f0430009040 comp rx=0 tx=0).stop 2024-09-16T01:44:48.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.021+0000 7f04429fd640 1 -- 172.21.15.88:0/866359844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f043c0b6140 msgr2=0x7f043c1512f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:48.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.021+0000 7f04429fd640 1 --2- 172.21.15.88:0/866359844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f043c0b6140 0x7f043c1512f0 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f042c00c3c0 tx=0x7f042c00c880 comp rx=0 tx=0).stop 2024-09-16T01:44:48.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.021+0000 7f04429fd640 1 -- 172.21.15.88:0/866359844 shutdown_connections 2024-09-16T01:44:48.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.021+0000 7f04429fd640 1 --2- 172.21.15.88:0/866359844 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f042806c2b0 0x7f042806e770 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:48.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.021+0000 7f04429fd640 1 --2- 172.21.15.88:0/866359844 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f043c155800 0x7f043c15a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:48.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.021+0000 7f04429fd640 1 --2- 172.21.15.88:0/866359844 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f043c0cfc80 0x7f043c15a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:48.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.021+0000 7f04429fd640 1 --2- 172.21.15.88:0/866359844 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f043c0b6140 0x7f043c1512f0 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:48.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.021+0000 7f04429fd640 1 -- 172.21.15.88:0/866359844 >> 172.21.15.88:0/866359844 conn(0x7f043c06bc10 msgr2=0x7f043c0ce770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:48.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.021+0000 7f04429fd640 1 -- 172.21.15.88:0/866359844 shutdown_connections 2024-09-16T01:44:48.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.021+0000 7f04429fd640 1 -- 172.21.15.88:0/866359844 wait complete. 2024-09-16T01:44:48.034 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-16T01:44:48.034 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-16T01:44:48.034 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-16T01:44:48.034 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:44:48.035 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-16T01:44:48.035 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-16T01:44:48.035 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-16T01:44:48.035 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:44:48.036 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-16T01:44:48.036 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-16T01:44:48.036 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-16T01:44:48.036 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:44:48.036 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-16T01:44:48.036 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-16T01:44:48.036 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-16T01:44:48.036 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:44:48.044 INFO:tasks.ceph.mds.a:Stopped 2024-09-16T01:44:48.044 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-16T01:44:48.045 INFO:tasks.ceph.mds.c:Stopped 2024-09-16T01:44:48.045 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-16T01:44:48.048 INFO:tasks.ceph.mds.b:Stopped 2024-09-16T01:44:48.048 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-16T01:44:48.049 INFO:tasks.ceph.mds.d:Stopped 2024-09-16T01:44:48.049 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-16T01:44:48.088 INFO:tasks.ceph.mds.a:Started 2024-09-16T01:44:48.088 DEBUG:teuthology.parallel:result is None 2024-09-16T01:44:48.089 INFO:tasks.ceph.mds.c:Started 2024-09-16T01:44:48.089 DEBUG:teuthology.parallel:result is None 2024-09-16T01:44:48.092 INFO:tasks.ceph.mds.b:Started 2024-09-16T01:44:48.092 DEBUG:teuthology.parallel:result is None 2024-09-16T01:44:48.093 INFO:tasks.ceph.mds.d:Started 2024-09-16T01:44:48.093 DEBUG:teuthology.parallel:result is None 2024-09-16T01:44:48.093 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-16T01:44:48.162 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:44:48.162 INFO:tasks.ceph.mds.c.smithi088.stdout:starting mds.c at 2024-09-16T01:44:48.174 INFO:tasks.ceph.mds.d.smithi154.stdout:starting mds.d at 2024-09-16T01:44:48.174 INFO:tasks.ceph.mds.b.smithi154.stdout:starting mds.b at 2024-09-16T01:44:48.174 INFO:tasks.ceph.mds.b.smithi154.stderr:2024-09-16T01:44:48.172+0000 7fb850994640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-09-16T01:44:48.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.199+0000 7ff10cffd640 1 Processor -- start 2024-09-16T01:44:48.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.199+0000 7ff10cffd640 1 -- start start 2024-09-16T01:44:48.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.200+0000 7ff10cffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1080cf1f0 0x7ff1080b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:48.215 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.200+0000 7ff10cffd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1080cf860 con 0x7ff10815e7a0 2024-09-16T01:44:48.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.200+0000 7ff10cffd640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1080cfb40 con 0x7ff1080b6720 2024-09-16T01:44:48.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.200+0000 7ff10cffd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff1080d2ca0 con 0x7ff1080cf1f0 2024-09-16T01:44:48.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.200+0000 7ff10659c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff10815e7a0 0x7ff10806caa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46312/0 (socket says 172.21.15.88:46312) 2024-09-16T01:44:48.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.200+0000 7ff10659c640 1 -- 172.21.15.88:0/3092568167 learned_addr learned my addr 172.21.15.88:0/3092568167 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:48.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.200+0000 7ff105d9b640 1 --2- 172.21.15.88:0/3092568167 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1080cf1f0 0x7ff1080b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:48.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.200+0000 7ff10559a640 1 -- 172.21.15.88:0/3092568167 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2897022803 0 0) 0x7ff1080cf860 con 0x7ff10815e7a0 2024-09-16T01:44:48.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.200+0000 7ff10559a640 1 -- 172.21.15.88:0/3092568167 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0f0003520 con 0x7ff10815e7a0 2024-09-16T01:44:48.216 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.200+0000 7ff10559a640 1 -- 172.21.15.88:0/3092568167 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3093794688 0 0) 0x7ff1080cfb40 con 0x7ff1080b6720 2024-09-16T01:44:48.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.200+0000 7ff10559a640 1 -- 172.21.15.88:0/3092568167 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1080cf860 con 0x7ff1080b6720 2024-09-16T01:44:48.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.201+0000 7ff10559a640 1 -- 172.21.15.88:0/3092568167 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 870920419 0 0) 0x7ff0f0003520 con 0x7ff10815e7a0 2024-09-16T01:44:48.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.201+0000 7ff10559a640 1 -- 172.21.15.88:0/3092568167 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1080cf1f0 msgr2=0x7ff1080b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:48.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.201+0000 7ff10559a640 1 --2- 172.21.15.88:0/3092568167 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1080cf1f0 0x7ff1080b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:48.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.201+0000 7ff10559a640 1 -- 172.21.15.88:0/3092568167 >> v1:172.21.15.154:6789/0 conn(0x7ff1080b6720 legacy=0x7ff1080d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:48.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.201+0000 7ff10559a640 1 -- 172.21.15.88:0/3092568167 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1080cfe20 con 0x7ff10815e7a0 2024-09-16T01:44:48.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.201+0000 7ff10559a640 1 -- 172.21.15.88:0/3092568167 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7ff0f8001670 con 0x7ff10815e7a0 2024-09-16T01:44:48.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.201+0000 7ff10559a640 1 -- 172.21.15.88:0/3092568167 >> v1:172.21.15.88:6789/0 conn(0x7ff10815e7a0 legacy=0x7ff10806caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:48.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.201+0000 7ff10559a640 1 --2- 172.21.15.88:0/3092568167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0f0003b70 0x7ff0f0003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:48.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.201+0000 7ff10559a640 1 --2- 172.21.15.88:0/3092568167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0f0004800 0x7ff0f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:48.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.201+0000 7ff10559a640 1 --2- 172.21.15.88:0/3092568167 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1080cf1f0 0x7ff0f00051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:48.217 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.201+0000 7ff10559a640 1 -- 172.21.15.88:0/3092568167 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff1080d2ca0 con 0x7ff0f0003b70 2024-09-16T01:44:48.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.201+0000 7ff10559a640 1 -- 172.21.15.88:0/3092568167 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f000bad0 con 0x7ff0f0004800 2024-09-16T01:44:48.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.201+0000 7ff10559a640 1 -- 172.21.15.88:0/3092568167 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff0f000bcd0 con 0x7ff1080cf1f0 2024-09-16T01:44:48.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.201+0000 7ff10659c640 1 --2- 172.21.15.88:0/3092568167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0f0003b70 0x7ff0f0003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:48.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.201+0000 7ff105d9b640 1 --2- 172.21.15.88:0/3092568167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0f0004800 0x7ff0f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:48.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.201+0000 7ff106d9d640 1 --2- 172.21.15.88:0/3092568167 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1080cf1f0 0x7ff0f00051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:48.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.201+0000 7ff10659c640 1 -- 172.21.15.88:0/3092568167 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1080cf1f0 msgr2=0x7ff0f00051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:48.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.201+0000 7ff10659c640 1 --2- 172.21.15.88:0/3092568167 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1080cf1f0 0x7ff0f00051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:48.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.201+0000 7ff10659c640 1 -- 172.21.15.88:0/3092568167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0f0004800 msgr2=0x7ff0f0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:48.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.201+0000 7ff10659c640 1 --2- 172.21.15.88:0/3092568167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0f0004800 0x7ff0f0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:48.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.201+0000 7ff10659c640 1 -- 172.21.15.88:0/3092568167 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0f0003520 con 0x7ff0f0003b70 2024-09-16T01:44:48.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.202+0000 7ff10659c640 1 --2- 172.21.15.88:0/3092568167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0f0003b70 0x7ff0f0003f70 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800c1c0 tx=0x7ff0f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcb8daa1a553ee32 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:48.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.202+0000 7ff10559a640 1 -- 172.21.15.88:0/3092568167 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f8007cc0 con 0x7ff0f0003b70 2024-09-16T01:44:48.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.202+0000 7ff10559a640 1 -- 172.21.15.88:0/3092568167 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f8005af0 con 0x7ff0f0003b70 2024-09-16T01:44:48.218 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.202+0000 7ff10559a640 1 -- 172.21.15.88:0/3092568167 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f8007070 con 0x7ff0f0003b70 2024-09-16T01:44:48.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.202+0000 7ff10cffd640 1 -- 172.21.15.88:0/3092568167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0f0003b70 msgr2=0x7ff0f0003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:48.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.202+0000 7ff10cffd640 1 --2- 172.21.15.88:0/3092568167 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0f0003b70 0x7ff0f0003f70 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800c1c0 tx=0x7ff0f0005830 comp rx=0 tx=0).stop 2024-09-16T01:44:48.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.202+0000 7ff106d9d640 1 -- 172.21.15.88:0/3092568167 reap_dead start 2024-09-16T01:44:48.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.203+0000 7ff10cffd640 1 -- 172.21.15.88:0/3092568167 shutdown_connections 2024-09-16T01:44:48.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.203+0000 7ff10cffd640 1 -- 172.21.15.88:0/3092568167 >> 172.21.15.88:0/3092568167 conn(0x7ff1080d11e0 msgr2=0x7ff1080d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:48.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.203+0000 7ff10cffd640 1 -- 172.21.15.88:0/3092568167 shutdown_connections 2024-09-16T01:44:48.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.203+0000 7ff10cffd640 1 -- 172.21.15.88:0/3092568167 wait complete. 2024-09-16T01:44:48.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.204+0000 7ff10cffd640 1 Processor -- start 2024-09-16T01:44:48.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.204+0000 7ff10cffd640 1 -- start start 2024-09-16T01:44:48.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.205+0000 7ff10cffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0f0003b70 0x7ff1081f4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:48.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.205+0000 7ff10cffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1080cb830 0x7ff108152f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:48.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.205+0000 7ff10cffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1080cc1a0 0x7ff1080cc620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:48.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.205+0000 7ff10cffd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff10815eb80 con 0x7ff0f0003b70 2024-09-16T01:44:48.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.205+0000 7ff10cffd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff1080b7a00 con 0x7ff1080cc1a0 2024-09-16T01:44:48.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.205+0000 7ff10cffd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff10815ed00 con 0x7ff1080cb830 2024-09-16T01:44:48.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.205+0000 7ff10659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0f0003b70 0x7ff1081f4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:48.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.205+0000 7ff10659c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0f0003b70 0x7ff1081f4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41732/0 (socket says 172.21.15.88:41732) 2024-09-16T01:44:48.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.205+0000 7ff10659c640 1 -- 172.21.15.88:0/240594050 learned_addr learned my addr 172.21.15.88:0/240594050 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:48.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.205+0000 7ff106d9d640 1 --2- 172.21.15.88:0/240594050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1080cc1a0 0x7ff1080cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:48.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.205+0000 7ff105d9b640 1 --2- 172.21.15.88:0/240594050 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1080cb830 0x7ff108152f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:48.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.205+0000 7ff10659c640 1 -- 172.21.15.88:0/240594050 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1080cb830 msgr2=0x7ff108152f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:48.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.205+0000 7ff10659c640 1 --2- 172.21.15.88:0/240594050 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1080cb830 0x7ff108152f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:48.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.205+0000 7ff10659c640 1 -- 172.21.15.88:0/240594050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1080cc1a0 msgr2=0x7ff1080cc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:48.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.205+0000 7ff10659c640 1 --2- 172.21.15.88:0/240594050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1080cc1a0 0x7ff1080cc620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:48.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.205+0000 7ff10659c640 1 -- 172.21.15.88:0/240594050 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1080d39f0 con 0x7ff0f0003b70 2024-09-16T01:44:48.220 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.205+0000 7ff106d9d640 1 --2- 172.21.15.88:0/240594050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1080cc1a0 0x7ff1080cc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:48.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.205+0000 7ff105d9b640 1 --2- 172.21.15.88:0/240594050 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1080cb830 0x7ff108152f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:48.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.205+0000 7ff10659c640 1 --2- 172.21.15.88:0/240594050 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0f0003b70 0x7ff1081f4780 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8000f60 tx=0x7ff0f80019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:48.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.205+0000 7ff0f6ffd640 1 -- 172.21.15.88:0/240594050 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f8001670 con 0x7ff0f0003b70 2024-09-16T01:44:48.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.205+0000 7ff0f6ffd640 1 -- 172.21.15.88:0/240594050 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f8042070 con 0x7ff0f0003b70 2024-09-16T01:44:48.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.205+0000 7ff0f6ffd640 1 -- 172.21.15.88:0/240594050 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f8007270 con 0x7ff0f0003b70 2024-09-16T01:44:48.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.205+0000 7ff10cffd640 1 -- 172.21.15.88:0/240594050 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1081f5040 con 0x7ff0f0003b70 2024-09-16T01:44:48.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.205+0000 7ff10cffd640 1 -- 172.21.15.88:0/240594050 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1081f5600 con 0x7ff0f0003b70 2024-09-16T01:44:48.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.206+0000 7ff10cffd640 1 -- 172.21.15.88:0/240594050 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1080b61e0 con 0x7ff0f0003b70 2024-09-16T01:44:48.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.207+0000 7ff0f6ffd640 1 -- 172.21.15.88:0/240594050 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff0f8007410 con 0x7ff0f0003b70 2024-09-16T01:44:48.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.207+0000 7ff0f6ffd640 1 --2- 172.21.15.88:0/240594050 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff0e806c2e0 0x7ff0e806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:48.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.207+0000 7ff0f6ffd640 1 -- 172.21.15.88:0/240594050 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 8297+0+0 (secure 0 0 0) 0x7ff0f80abf30 con 0x7ff0f0003b70 2024-09-16T01:44:48.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.207+0000 7ff105d9b640 1 --2- 172.21.15.88:0/240594050 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff0e806c2e0 0x7ff0e806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:48.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.208+0000 7ff105d9b640 1 --2- 172.21.15.88:0/240594050 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff0e806c2e0 0x7ff0e806e7a0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7ff1080cec90 tx=0x7ff0fc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:48.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.212+0000 7ff0f6ffd640 1 -- 172.21.15.88:0/240594050 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff0f8079d50 con 0x7ff0f0003b70 2024-09-16T01:44:48.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:48.386+0000 7ff10cffd640 1 -- 172.21.15.88:0/240594050 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7ff1080d2a80 con 0x7ff0f0003b70 2024-09-16T01:44:48.610 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:44:48.613 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:44:49.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.195+0000 7ff0f6ffd640 1 -- 172.21.15.88:0/240594050 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v134) ==== 97+0+0 (secure 0 0 0) 0x7ff0f8079650 con 0x7ff0f0003b70 2024-09-16T01:44:49.197 INFO:teuthology.orchestra.run.smithi088.stderr: removed all blocklist entries 2024-09-16T01:44:49.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.198+0000 7ff10cffd640 1 -- 172.21.15.88:0/240594050 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff0e806c2e0 msgr2=0x7ff0e806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:49.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.198+0000 7ff10cffd640 1 --2- 172.21.15.88:0/240594050 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff0e806c2e0 0x7ff0e806e7a0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7ff1080cec90 tx=0x7ff0fc008040 comp rx=0 tx=0).stop 2024-09-16T01:44:49.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.198+0000 7ff10cffd640 1 -- 172.21.15.88:0/240594050 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0f0003b70 msgr2=0x7ff1081f4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:49.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.198+0000 7ff10cffd640 1 --2- 172.21.15.88:0/240594050 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0f0003b70 0x7ff1081f4780 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8000f60 tx=0x7ff0f80019c0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.198+0000 7ff10cffd640 1 -- 172.21.15.88:0/240594050 shutdown_connections 2024-09-16T01:44:49.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.198+0000 7ff10cffd640 1 --2- 172.21.15.88:0/240594050 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff0e806c2e0 0x7ff0e806e7a0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.198+0000 7ff10cffd640 1 --2- 172.21.15.88:0/240594050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1080cc1a0 0x7ff1080cc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.198+0000 7ff10cffd640 1 --2- 172.21.15.88:0/240594050 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1080cb830 0x7ff108152f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.198+0000 7ff10cffd640 1 --2- 172.21.15.88:0/240594050 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0f0003b70 0x7ff1081f4780 unknown :-1 s=CLOSED pgs=2326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.198+0000 7ff10cffd640 1 -- 172.21.15.88:0/240594050 >> 172.21.15.88:0/240594050 conn(0x7ff1080d11e0 msgr2=0x7ff10815d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:49.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.198+0000 7ff10cffd640 1 -- 172.21.15.88:0/240594050 shutdown_connections 2024-09-16T01:44:49.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.198+0000 7ff10cffd640 1 -- 172.21.15.88:0/240594050 wait complete. 2024-09-16T01:44:49.213 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-16T01:44:49.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.313+0000 7f6d562e4640 1 Processor -- start 2024-09-16T01:44:49.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.314+0000 7f6d562e4640 1 -- start start 2024-09-16T01:44:49.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.314+0000 7f6d562e4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6d5015e660 0x7f6d500cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:49.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.314+0000 7f6d562e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d500d3310 0x7f6d500d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:49.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.314+0000 7f6d562e4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6d500d1650 0x7f6d5006bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:49.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.314+0000 7f6d562e4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f6d500d1ac0 con 0x7f6d500d1650 2024-09-16T01:44:49.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.314+0000 7f6d562e4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6d5015ea40 con 0x7f6d5015e660 2024-09-16T01:44:49.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.314+0000 7f6d562e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6d500d3870 con 0x7f6d500d3310 2024-09-16T01:44:49.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.314+0000 7f6d54881640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6d500d1650 0x7f6d5006bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:49.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.314+0000 7f6d54881640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f6d500d1650 0x7f6d5006bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41746/0 (socket says 172.21.15.88:41746) 2024-09-16T01:44:49.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.314+0000 7f6d54881640 1 -- 172.21.15.88:0/941514880 learned_addr learned my addr 172.21.15.88:0/941514880 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:49.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.315+0000 7f6d54881640 1 -- 172.21.15.88:0/941514880 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d500d3310 msgr2=0x7f6d500d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:49.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.315+0000 7f6d54881640 1 --2- 172.21.15.88:0/941514880 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d500d3310 0x7f6d500d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.315+0000 7f6d54881640 1 -- 172.21.15.88:0/941514880 >> v2:172.21.15.154:3300/0 conn(0x7f6d5015e660 msgr2=0x7f6d500cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:49.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.315+0000 7f6d4ffff640 1 --2- 172.21.15.88:0/941514880 >> v2:172.21.15.154:3300/0 conn(0x7f6d5015e660 0x7f6d500cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:49.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.315+0000 7f6d54881640 1 --2- 172.21.15.88:0/941514880 >> v2:172.21.15.154:3300/0 conn(0x7f6d5015e660 0x7f6d500cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.315+0000 7f6d54881640 1 -- 172.21.15.88:0/941514880 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d500d3b50 con 0x7f6d500d1650 2024-09-16T01:44:49.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.315+0000 7f6d4ffff640 1 --2- 172.21.15.88:0/941514880 >> v2:172.21.15.154:3300/0 conn(0x7f6d5015e660 0x7f6d500cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:49.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.315+0000 7f6d54881640 1 --2- 172.21.15.88:0/941514880 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d500d1650 0x7f6d5006bfd0 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f6d500d10e0 tx=0x7f6d4802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec50a08f917b9c84 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:49.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.316+0000 7f6d4effd640 1 -- 172.21.15.88:0/941514880 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d48036070 con 0x7f6d500d1650 2024-09-16T01:44:49.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.316+0000 7f6d4effd640 1 -- 172.21.15.88:0/941514880 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d48031600 con 0x7f6d500d1650 2024-09-16T01:44:49.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.316+0000 7f6d4effd640 1 -- 172.21.15.88:0/941514880 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d480319d0 con 0x7f6d500d1650 2024-09-16T01:44:49.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.316+0000 7f6d562e4640 1 -- 172.21.15.88:0/941514880 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d500d1650 msgr2=0x7f6d5006bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:49.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.316+0000 7f6d562e4640 1 --2- 172.21.15.88:0/941514880 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d500d1650 0x7f6d5006bfd0 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f6d500d10e0 tx=0x7f6d4802a6a0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.316+0000 7f6d562e4640 1 -- 172.21.15.88:0/941514880 shutdown_connections 2024-09-16T01:44:49.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.316+0000 7f6d562e4640 1 --2- 172.21.15.88:0/941514880 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d500d1650 0x7f6d5006bfd0 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.316+0000 7f6d562e4640 1 --2- 172.21.15.88:0/941514880 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d500d3310 0x7f6d500d1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.316+0000 7f6d562e4640 1 --2- 172.21.15.88:0/941514880 >> v2:172.21.15.154:3300/0 conn(0x7f6d5015e660 0x7f6d500cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.316+0000 7f6d562e4640 1 -- 172.21.15.88:0/941514880 >> 172.21.15.88:0/941514880 conn(0x7f6d500b6410 msgr2=0x7f6d500b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:49.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.316+0000 7f6d562e4640 1 -- 172.21.15.88:0/941514880 shutdown_connections 2024-09-16T01:44:49.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.316+0000 7f6d562e4640 1 -- 172.21.15.88:0/941514880 wait complete. 2024-09-16T01:44:49.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.317+0000 7f6d562e4640 1 Processor -- start 2024-09-16T01:44:49.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.318+0000 7f6d562e4640 1 -- start start 2024-09-16T01:44:49.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.318+0000 7f6d562e4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d500d1650 0x7f6d501f4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:49.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.318+0000 7f6d562e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d500d3310 0x7f6d50160b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:49.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.318+0000 7f6d562e4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d5015e660 0x7f6d50161060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:49.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.318+0000 7f6d562e4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f6d500d43f0 con 0x7f6d5015e660 2024-09-16T01:44:49.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.318+0000 7f6d562e4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6d500d4110 con 0x7f6d500d1650 2024-09-16T01:44:49.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.318+0000 7f6d562e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6d500cfc70 con 0x7f6d500d3310 2024-09-16T01:44:49.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.318+0000 7f6d54881640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d5015e660 0x7f6d50161060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:49.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.318+0000 7f6d54881640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d5015e660 0x7f6d50161060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41754/0 (socket says 172.21.15.88:41754) 2024-09-16T01:44:49.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.318+0000 7f6d54881640 1 -- 172.21.15.88:0/1021754332 learned_addr learned my addr 172.21.15.88:0/1021754332 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:49.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.318+0000 7f6d4ffff640 1 --2- 172.21.15.88:0/1021754332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d500d1650 0x7f6d501f4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:49.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.318+0000 7f6d4f7fe640 1 --2- 172.21.15.88:0/1021754332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d500d3310 0x7f6d50160b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:49.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.319+0000 7f6d54881640 1 -- 172.21.15.88:0/1021754332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d500d3310 msgr2=0x7f6d50160b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:49.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.319+0000 7f6d54881640 1 --2- 172.21.15.88:0/1021754332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d500d3310 0x7f6d50160b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.319+0000 7f6d54881640 1 -- 172.21.15.88:0/1021754332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d500d1650 msgr2=0x7f6d501f4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:49.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.319+0000 7f6d54881640 1 --2- 172.21.15.88:0/1021754332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d500d1650 0x7f6d501f4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.319+0000 7f6d54881640 1 -- 172.21.15.88:0/1021754332 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d501f4f80 con 0x7f6d5015e660 2024-09-16T01:44:49.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.319+0000 7f6d54881640 1 --2- 172.21.15.88:0/1021754332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d5015e660 0x7f6d50161060 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f6d48009f40 tx=0x7f6d48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:49.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.319+0000 7f6d4cff9640 1 -- 172.21.15.88:0/1021754332 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d48036050 con 0x7f6d5015e660 2024-09-16T01:44:49.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.319+0000 7f6d4cff9640 1 -- 172.21.15.88:0/1021754332 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d48042070 con 0x7f6d5015e660 2024-09-16T01:44:49.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.319+0000 7f6d4cff9640 1 -- 172.21.15.88:0/1021754332 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d48009510 con 0x7f6d5015e660 2024-09-16T01:44:49.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.319+0000 7f6d562e4640 1 -- 172.21.15.88:0/1021754332 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d501f7740 con 0x7f6d5015e660 2024-09-16T01:44:49.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.319+0000 7f6d562e4640 1 -- 172.21.15.88:0/1021754332 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d501f7d00 con 0x7f6d5015e660 2024-09-16T01:44:49.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.320+0000 7f6d562e4640 1 -- 172.21.15.88:0/1021754332 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d500d1110 con 0x7f6d5015e660 2024-09-16T01:44:49.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.320+0000 7f6d4cff9640 1 -- 172.21.15.88:0/1021754332 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f6d48007bd0 con 0x7f6d5015e660 2024-09-16T01:44:49.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.321+0000 7f6d4cff9640 1 --2- 172.21.15.88:0/1021754332 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6d3c06c2b0 0x7f6d3c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:49.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.321+0000 7f6d4cff9640 1 -- 172.21.15.88:0/1021754332 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 4857+0+0 (secure 0 0 0) 0x7f6d480abef0 con 0x7f6d5015e660 2024-09-16T01:44:49.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.321+0000 7f6d4ffff640 1 --2- 172.21.15.88:0/1021754332 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6d3c06c2b0 0x7f6d3c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:49.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.321+0000 7f6d4ffff640 1 --2- 172.21.15.88:0/1021754332 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6d3c06c2b0 0x7f6d3c06e770 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f6d5015ae00 tx=0x7f6d40008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:49.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.325+0000 7f6d4cff9640 1 -- 172.21.15.88:0/1021754332 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6d4807a3f0 con 0x7f6d5015e660 2024-09-16T01:44:49.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.553+0000 7f6d562e4640 1 -- 172.21.15.88:0/1021754332 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f6d500cfa50 con 0x7f6d5015e660 2024-09-16T01:44:49.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.554+0000 7f6d4cff9640 1 -- 172.21.15.88:0/1021754332 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) ==== 80+0+5304 (secure 0 0 0) 0x7f6d4807e6f0 con 0x7f6d5015e660 2024-09-16T01:44:49.555 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:49.556 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:44:49.556 INFO:teuthology.orchestra.run.smithi088.stdout: "auth_dump": [ 2024-09-16T01:44:49.556 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:44:49.556 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mds.a", 2024-09-16T01:44:49.556 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmy+jJCRAAhnzX/li6D3D07NmThrjXkA==", 2024-09-16T01:44:49.556 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:44:49.556 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:44:49.556 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:44:49.556 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:44:49.556 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:44:49.556 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:44:49.556 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:44:49.557 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:44:49.557 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mds.b", 2024-09-16T01:44:49.557 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmx7p+FRAApxKUSZoR/nGUZsaYKlrYUg==", 2024-09-16T01:44:49.557 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:44:49.557 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:44:49.557 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:44:49.557 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:44:49.557 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:44:49.557 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:44:49.557 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:44:49.557 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:44:49.557 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mds.c", 2024-09-16T01:44:49.557 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmaYHUEBAAzM0qE53LEgHi+hThHJX3Tw==", 2024-09-16T01:44:49.558 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:44:49.558 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:44:49.558 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:44:49.558 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:44:49.558 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:44:49.558 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:44:49.558 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:44:49.558 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:44:49.558 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mds.d", 2024-09-16T01:44:49.558 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudm3TSLHBAAJzlR6H09cc1Pki2FjyoFiA==", 2024-09-16T01:44:49.558 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:44:49.558 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:44:49.558 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:44:49.558 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:44:49.559 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:44:49.559 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:44:49.559 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:44:49.559 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:44:49.559 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.0", 2024-09-16T01:44:49.559 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDhiudmZR7MORAAwDKvphubA+CbzBUWuaA80A==", 2024-09-16T01:44:49.559 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:44:49.559 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:44:49.559 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:44:49.559 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:44:49.559 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:44:49.559 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:44:49.559 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:44:49.559 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.1", 2024-09-16T01:44:49.560 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDliudmNd+SLBAArOUdoZ226c5jlAsKwqERmw==", 2024-09-16T01:44:49.560 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:44:49.560 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:44:49.560 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:44:49.560 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:44:49.560 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:44:49.560 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:44:49.560 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:44:49.560 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.2", 2024-09-16T01:44:49.560 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDpiudm75CUARAAWHtYOg9pE8g+M6IL+8XZ6g==", 2024-09-16T01:44:49.560 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:44:49.560 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:44:49.560 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:44:49.561 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:44:49.561 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:44:49.561 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:44:49.561 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:44:49.561 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.3", 2024-09-16T01:44:49.561 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDtiudm0fgWAxAAf2NOWrpXbOhb3dFXozJagQ==", 2024-09-16T01:44:49.561 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:44:49.561 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:44:49.561 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:44:49.561 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:44:49.562 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:44:49.562 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:44:49.562 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:44:49.562 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.4", 2024-09-16T01:44:49.562 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQD0iudmdddMCxAAtr2qfj4Nxj3/GHUpxRmtFg==", 2024-09-16T01:44:49.562 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:44:49.562 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:44:49.562 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:44:49.562 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:44:49.562 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:44:49.562 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:44:49.562 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:44:49.562 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.5", 2024-09-16T01:44:49.562 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQD3iudmT5RjOhAAcKwca6BG60gsxYnmy4twGg==", 2024-09-16T01:44:49.563 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:44:49.563 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:44:49.563 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:44:49.563 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:44:49.563 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:44:49.563 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:44:49.563 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:44:49.563 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.6", 2024-09-16T01:44:49.563 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQD7iudmT7daLRAAdfLFijKQku3rJcJ5xcfLsw==", 2024-09-16T01:44:49.563 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:44:49.563 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:44:49.563 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:44:49.563 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:44:49.563 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:44:49.564 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:44:49.564 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:44:49.564 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "osd.7", 2024-09-16T01:44:49.564 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQD/iudmmvtxLhAArYKhQ4zyFHtLgh/+bs5ZhA==", 2024-09-16T01:44:49.564 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:44:49.564 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow profile osd", 2024-09-16T01:44:49.564 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile osd", 2024-09-16T01:44:49.564 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:44:49.564 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:44:49.564 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:44:49.564 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:44:49.564 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.0", 2024-09-16T01:44:49.564 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmvPikIBAAyHxNNnwQKygxg09PNQLc/Q==", 2024-09-16T01:44:49.565 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:44:49.565 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:44:49.565 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow r", 2024-09-16T01:44:49.565 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-16T01:44:49.565 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:44:49.565 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:44:49.565 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:44:49.565 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.1", 2024-09-16T01:44:49.565 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmA69iKRAA8Abwe0Lg2PnZDErgJ1gEbQ==", 2024-09-16T01:44:49.565 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:44:49.565 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:44:49.566 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow r", 2024-09-16T01:44:49.566 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow rw", 2024-09-16T01:44:49.566 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow rwx" 2024-09-16T01:44:49.566 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:44:49.566 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:44:49.566 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:44:49.566 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.2", 2024-09-16T01:44:49.566 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmB+w9JhAAv63Is5T/cvDbbLOjIkI0wA==", 2024-09-16T01:44:49.566 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:44:49.567 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:44:49.567 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow r", 2024-09-16T01:44:49.567 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow rw", 2024-09-16T01:44:49.567 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow rwx" 2024-09-16T01:44:49.567 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:44:49.567 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:44:49.567 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:44:49.567 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.3", 2024-09-16T01:44:49.567 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudm4WoFLxAAnYGHGqHyBHHajQ8f3rCjEA==", 2024-09-16T01:44:49.567 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:44:49.567 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow", 2024-09-16T01:44:49.567 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow r", 2024-09-16T01:44:49.568 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow rw", 2024-09-16T01:44:49.568 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow rwx" 2024-09-16T01:44:49.568 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:44:49.568 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:44:49.568 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:44:49.569 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "client.admin", 2024-09-16T01:44:49.569 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDdiudmhcE0KBAANauQKyUY01/KlWMrN2x/5Q==", 2024-09-16T01:44:49.569 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:44:49.569 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow *", 2024-09-16T01:44:49.569 INFO:teuthology.orchestra.run.smithi088.stdout: "mgr": "allow *", 2024-09-16T01:44:49.569 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow *", 2024-09-16T01:44:49.569 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:44:49.569 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:44:49.569 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:44:49.569 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:44:49.569 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mgr.x", 2024-09-16T01:44:49.569 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudmToGYBhAAKwXS34PtfAVtEtdtHKHlQQ==", 2024-09-16T01:44:49.569 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:44:49.570 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow *", 2024-09-16T01:44:49.570 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile mgr", 2024-09-16T01:44:49.570 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:44:49.570 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:44:49.570 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2024-09-16T01:44:49.571 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:44:49.571 INFO:teuthology.orchestra.run.smithi088.stdout: "entity": "mgr.y", 2024-09-16T01:44:49.571 INFO:teuthology.orchestra.run.smithi088.stdout: "key": "AQDeiudm/BN5ARAAxbCopbsvxFUZP3drso32qw==", 2024-09-16T01:44:49.571 INFO:teuthology.orchestra.run.smithi088.stdout: "caps": { 2024-09-16T01:44:49.571 INFO:teuthology.orchestra.run.smithi088.stdout: "mds": "allow *", 2024-09-16T01:44:49.571 INFO:teuthology.orchestra.run.smithi088.stdout: "mon": "allow profile mgr", 2024-09-16T01:44:49.571 INFO:teuthology.orchestra.run.smithi088.stdout: "osd": "allow *" 2024-09-16T01:44:49.571 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:44:49.571 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:44:49.572 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2024-09-16T01:44:49.572 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:44:49.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.556+0000 7f6d562e4640 1 -- 172.21.15.88:0/1021754332 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6d3c06c2b0 msgr2=0x7f6d3c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:49.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.556+0000 7f6d562e4640 1 --2- 172.21.15.88:0/1021754332 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6d3c06c2b0 0x7f6d3c06e770 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f6d5015ae00 tx=0x7f6d40008040 comp rx=0 tx=0).stop 2024-09-16T01:44:49.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.556+0000 7f6d562e4640 1 -- 172.21.15.88:0/1021754332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d5015e660 msgr2=0x7f6d50161060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:49.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.556+0000 7f6d562e4640 1 --2- 172.21.15.88:0/1021754332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d5015e660 0x7f6d50161060 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f6d48009f40 tx=0x7f6d48004a40 comp rx=0 tx=0).stop 2024-09-16T01:44:49.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.556+0000 7f6d562e4640 1 -- 172.21.15.88:0/1021754332 shutdown_connections 2024-09-16T01:44:49.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.556+0000 7f6d562e4640 1 --2- 172.21.15.88:0/1021754332 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f6d3c06c2b0 0x7f6d3c06e770 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.556+0000 7f6d562e4640 1 --2- 172.21.15.88:0/1021754332 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d5015e660 0x7f6d50161060 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.556+0000 7f6d562e4640 1 --2- 172.21.15.88:0/1021754332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d500d3310 0x7f6d50160b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.556+0000 7f6d562e4640 1 --2- 172.21.15.88:0/1021754332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d500d1650 0x7f6d501f4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.556+0000 7f6d562e4640 1 -- 172.21.15.88:0/1021754332 >> 172.21.15.88:0/1021754332 conn(0x7f6d500b6410 msgr2=0x7f6d501596e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:49.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.556+0000 7f6d562e4640 1 -- 172.21.15.88:0/1021754332 shutdown_connections 2024-09-16T01:44:49.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.556+0000 7f6d562e4640 1 -- 172.21.15.88:0/1021754332 wait complete. 2024-09-16T01:44:49.577 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-16T01:44:49.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.674+0000 7f067198d640 1 Processor -- start 2024-09-16T01:44:49.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.674+0000 7f067198d640 1 -- start start 2024-09-16T01:44:49.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.674+0000 7f067198d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f066c15e660 0x7f066c0cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:49.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.674+0000 7f067198d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f066c15ea40 con 0x7f066c0d3310 2024-09-16T01:44:49.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.675+0000 7f067198d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f066c0d3870 con 0x7f066c0d1820 2024-09-16T01:44:49.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.675+0000 7f067198d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f066c0d3b50 con 0x7f066c15e660 2024-09-16T01:44:49.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.675+0000 7f066affd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f066c15e660 0x7f066c0cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:49.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.675+0000 7f06627fc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f066c0d3310 0x7f066c0d1110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46320/0 (socket says 172.21.15.88:46320) 2024-09-16T01:44:49.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.675+0000 7f06627fc640 1 -- 172.21.15.88:0/4122590192 learned_addr learned my addr 172.21.15.88:0/4122590192 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:49.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.675+0000 7f066affd640 1 -- 172.21.15.88:0/4122590192 >> v1:172.21.15.154:6789/0 conn(0x7f066c0d1820 legacy=0x7f066c0b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:49.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.675+0000 7f066affd640 1 -- 172.21.15.88:0/4122590192 >> v1:172.21.15.88:6789/0 conn(0x7f066c0d3310 legacy=0x7f066c0d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:49.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.675+0000 7f066a7fc640 1 -- 172.21.15.88:0/4122590192 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 556825428 0 0) 0x7f066c15ea40 con 0x7f066c0d3310 2024-09-16T01:44:49.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.675+0000 7f066affd640 1 -- 172.21.15.88:0/4122590192 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f066c0d3e30 con 0x7f066c15e660 2024-09-16T01:44:49.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.676+0000 7f066affd640 1 --2- 172.21.15.88:0/4122590192 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f066c15e660 0x7f066c0cf730 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7f065c0097a0 tx=0x7f065c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a0ad4adf1f7d6d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:49.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.676+0000 7f066a7fc640 1 -- 172.21.15.88:0/4122590192 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065c037070 con 0x7f066c15e660 2024-09-16T01:44:49.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.676+0000 7f066a7fc640 1 -- 172.21.15.88:0/4122590192 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f065c032440 con 0x7f066c15e660 2024-09-16T01:44:49.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.676+0000 7f066a7fc640 1 -- 172.21.15.88:0/4122590192 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065c032810 con 0x7f066c15e660 2024-09-16T01:44:49.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.676+0000 7f067198d640 1 -- 172.21.15.88:0/4122590192 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f066c15e660 msgr2=0x7f066c0cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:49.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.676+0000 7f067198d640 1 --2- 172.21.15.88:0/4122590192 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f066c15e660 0x7f066c0cf730 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7f065c0097a0 tx=0x7f065c02a1e0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.676+0000 7f067198d640 1 -- 172.21.15.88:0/4122590192 shutdown_connections 2024-09-16T01:44:49.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.676+0000 7f067198d640 1 --2- 172.21.15.88:0/4122590192 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f066c15e660 0x7f066c0cf730 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.676+0000 7f067198d640 1 -- 172.21.15.88:0/4122590192 >> 172.21.15.88:0/4122590192 conn(0x7f066c0b6410 msgr2=0x7f066c0b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:49.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.676+0000 7f067198d640 1 -- 172.21.15.88:0/4122590192 shutdown_connections 2024-09-16T01:44:49.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.676+0000 7f067198d640 1 -- 172.21.15.88:0/4122590192 wait complete. 2024-09-16T01:44:49.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.677+0000 7f067198d640 1 Processor -- start 2024-09-16T01:44:49.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.678+0000 7f067198d640 1 -- start start 2024-09-16T01:44:49.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.678+0000 7f067198d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f066c0d1820 0x7f066c1f37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:49.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.678+0000 7f067198d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f066c0d3310 0x7f066c161450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:49.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.678+0000 7f067198d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f066c15e660 0x7f066c0cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:49.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.678+0000 7f067198d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f066c0cfc70 con 0x7f066c15e660 2024-09-16T01:44:49.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.678+0000 7f067198d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f066c0d43f0 con 0x7f066c0d3310 2024-09-16T01:44:49.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.678+0000 7f067198d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f066c0b8280 con 0x7f066c0d1820 2024-09-16T01:44:49.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.679+0000 7f066b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f066c15e660 0x7f066c0cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:49.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.679+0000 7f066b7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f066c15e660 0x7f066c0cd6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41764/0 (socket says 172.21.15.88:41764) 2024-09-16T01:44:49.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.679+0000 7f066b7fe640 1 -- 172.21.15.88:0/460761383 learned_addr learned my addr 172.21.15.88:0/460761383 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:49.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.679+0000 7f06627fc640 1 --2- 172.21.15.88:0/460761383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f066c0d3310 0x7f066c161450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:49.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.679+0000 7f066affd640 1 --2- 172.21.15.88:0/460761383 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f066c0d1820 0x7f066c1f37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:49.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.679+0000 7f066b7fe640 1 -- 172.21.15.88:0/460761383 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f066c0d1820 msgr2=0x7f066c1f37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:49.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.679+0000 7f066b7fe640 1 --2- 172.21.15.88:0/460761383 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f066c0d1820 0x7f066c1f37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.679+0000 7f066b7fe640 1 -- 172.21.15.88:0/460761383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f066c0d3310 msgr2=0x7f066c161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:49.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.679+0000 7f066b7fe640 1 --2- 172.21.15.88:0/460761383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f066c0d3310 0x7f066c161450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.679+0000 7f066b7fe640 1 -- 172.21.15.88:0/460761383 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f066c0c9080 con 0x7f066c15e660 2024-09-16T01:44:49.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.679+0000 7f066b7fe640 1 --2- 172.21.15.88:0/460761383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f066c15e660 0x7f066c0cd6d0 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f066400c780 tx=0x7f066400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:49.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.679+0000 7f0668ff9640 1 -- 172.21.15.88:0/460761383 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0664004b60 con 0x7f066c15e660 2024-09-16T01:44:49.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.679+0000 7f0668ff9640 1 -- 172.21.15.88:0/460761383 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0664004d00 con 0x7f066c15e660 2024-09-16T01:44:49.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.679+0000 7f067198d640 1 -- 172.21.15.88:0/460761383 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f066c0cc670 con 0x7f066c15e660 2024-09-16T01:44:49.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.679+0000 7f0668ff9640 1 -- 172.21.15.88:0/460761383 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066400aa90 con 0x7f066c15e660 2024-09-16T01:44:49.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.679+0000 7f067198d640 1 -- 172.21.15.88:0/460761383 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f066c0ccc30 con 0x7f066c15e660 2024-09-16T01:44:49.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.680+0000 7f067198d640 1 -- 172.21.15.88:0/460761383 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f066c0cf730 con 0x7f066c15e660 2024-09-16T01:44:49.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.681+0000 7f0668ff9640 1 -- 172.21.15.88:0/460761383 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f066400ac30 con 0x7f066c15e660 2024-09-16T01:44:49.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.681+0000 7f0668ff9640 1 --2- 172.21.15.88:0/460761383 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f064c06c2b0 0x7f064c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:49.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.681+0000 7f066affd640 1 --2- 172.21.15.88:0/460761383 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f064c06c2b0 0x7f064c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:49.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.681+0000 7f0668ff9640 1 -- 172.21.15.88:0/460761383 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 4857+0+0 (secure 0 0 0) 0x7f066408b1f0 con 0x7f066c15e660 2024-09-16T01:44:49.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.682+0000 7f066affd640 1 --2- 172.21.15.88:0/460761383 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f064c06c2b0 0x7f064c06e770 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f065c034000 tx=0x7f065c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:49.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.685+0000 7f0668ff9640 1 -- 172.21.15.88:0/460761383 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0664020070 con 0x7f066c15e660 2024-09-16T01:44:49.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.852+0000 7f067198d640 1 -- 172.21.15.88:0/460761383 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f066c1591e0 con 0x7f066c15e660 2024-09-16T01:44:49.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.853+0000 7f0668ff9640 1 -- 172.21.15.88:0/460761383 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v134) ==== 84+0+65 (secure 0 0 0) 0x7f066c0c9080 con 0x7f066c15e660 2024-09-16T01:44:49.855 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:49.855 INFO:teuthology.orchestra.run.smithi088.stdout:[ 2024-09-16T01:44:49.855 INFO:teuthology.orchestra.run.smithi088.stdout: { 2024-09-16T01:44:49.855 INFO:teuthology.orchestra.run.smithi088.stdout: "poolnum": 1, 2024-09-16T01:44:49.855 INFO:teuthology.orchestra.run.smithi088.stdout: "poolname": ".mgr" 2024-09-16T01:44:49.855 INFO:teuthology.orchestra.run.smithi088.stdout: } 2024-09-16T01:44:49.855 INFO:teuthology.orchestra.run.smithi088.stdout:] 2024-09-16T01:44:49.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.854+0000 7f067198d640 1 -- 172.21.15.88:0/460761383 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f064c06c2b0 msgr2=0x7f064c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:49.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.854+0000 7f067198d640 1 --2- 172.21.15.88:0/460761383 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f064c06c2b0 0x7f064c06e770 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f065c034000 tx=0x7f065c02e000 comp rx=0 tx=0).stop 2024-09-16T01:44:49.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.854+0000 7f067198d640 1 -- 172.21.15.88:0/460761383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f066c15e660 msgr2=0x7f066c0cd6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:49.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.854+0000 7f067198d640 1 --2- 172.21.15.88:0/460761383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f066c15e660 0x7f066c0cd6d0 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f066400c780 tx=0x7f066400cc40 comp rx=0 tx=0).stop 2024-09-16T01:44:49.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.854+0000 7f067198d640 1 -- 172.21.15.88:0/460761383 shutdown_connections 2024-09-16T01:44:49.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.854+0000 7f067198d640 1 --2- 172.21.15.88:0/460761383 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f064c06c2b0 0x7f064c06e770 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.854+0000 7f067198d640 1 --2- 172.21.15.88:0/460761383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f066c15e660 0x7f066c0cd6d0 unknown :-1 s=CLOSED pgs=2331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.854+0000 7f067198d640 1 --2- 172.21.15.88:0/460761383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f066c0d3310 0x7f066c161450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.854+0000 7f067198d640 1 --2- 172.21.15.88:0/460761383 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f066c0d1820 0x7f066c1f37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.854+0000 7f067198d640 1 -- 172.21.15.88:0/460761383 >> 172.21.15.88:0/460761383 conn(0x7f066c0b6410 msgr2=0x7f066c0b8e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:49.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.855+0000 7f067198d640 1 -- 172.21.15.88:0/460761383 shutdown_connections 2024-09-16T01:44:49.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.855+0000 7f067198d640 1 -- 172.21.15.88:0/460761383 wait complete. 2024-09-16T01:44:49.867 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-16T01:44:49.867 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-16T01:44:49.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.969+0000 7f68a6f6d640 1 Processor -- start 2024-09-16T01:44:49.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.969+0000 7f68a6f6d640 1 -- start start 2024-09-16T01:44:49.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.970+0000 7f68a6f6d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f68a015e570 0x7f68a00b5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:49.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.970+0000 7f68a6f6d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68a006fb10 0x7f68a015e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:49.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.970+0000 7f68a6f6d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68a015f020 con 0x7f68a00714a0 2024-09-16T01:44:49.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.970+0000 7f68a6f6d640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f68a015fc30 con 0x7f68a015e570 2024-09-16T01:44:49.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.970+0000 7f68a6f6d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f68a015f300 con 0x7f68a006fb10 2024-09-16T01:44:49.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.970+0000 7f68a4d09640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f68a015e570 0x7f68a00b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:49.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.970+0000 7f68a4d09640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f68a015e570 0x7f68a00b5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41780/0 (socket says 172.21.15.88:41780) 2024-09-16T01:44:49.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.970+0000 7f68a4d09640 1 -- 172.21.15.88:0/530672868 learned_addr learned my addr 172.21.15.88:0/530672868 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:49.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.970+0000 7f689ffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f68a00714a0 0x7f68a006f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46326/0 (socket says 172.21.15.88:46326) 2024-09-16T01:44:49.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.970+0000 7f68a550a640 1 --2- 172.21.15.88:0/530672868 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68a006fb10 0x7f68a015e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:49.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.971+0000 7f68a4d09640 1 -- 172.21.15.88:0/530672868 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68a006fb10 msgr2=0x7f68a015e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:49.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.971+0000 7f68a4d09640 1 --2- 172.21.15.88:0/530672868 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68a006fb10 0x7f68a015e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.971+0000 7f68a4d09640 1 -- 172.21.15.88:0/530672868 >> v1:172.21.15.88:6789/0 conn(0x7f68a00714a0 legacy=0x7f68a006f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:49.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.971+0000 7f68a4d09640 1 -- 172.21.15.88:0/530672868 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68a015f5e0 con 0x7f68a015e570 2024-09-16T01:44:49.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.971+0000 7f689f7fe640 1 -- 172.21.15.88:0/530672868 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2472422309 0 0) 0x7f68a015f020 con 0x7f68a00714a0 2024-09-16T01:44:49.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.971+0000 7f68a4d09640 1 --2- 172.21.15.88:0/530672868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68a015e570 0x7f68a00b5540 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f6890009870 tx=0x7f689002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36ed31ef411eb95f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:49.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.971+0000 7f689f7fe640 1 -- 172.21.15.88:0/530672868 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6890037070 con 0x7f68a015e570 2024-09-16T01:44:49.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.971+0000 7f689f7fe640 1 -- 172.21.15.88:0/530672868 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6890032440 con 0x7f68a015e570 2024-09-16T01:44:49.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.971+0000 7f689f7fe640 1 -- 172.21.15.88:0/530672868 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6890032810 con 0x7f68a015e570 2024-09-16T01:44:49.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.972+0000 7f68a6f6d640 1 -- 172.21.15.88:0/530672868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68a015e570 msgr2=0x7f68a00b5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:49.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.972+0000 7f68a6f6d640 1 --2- 172.21.15.88:0/530672868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68a015e570 0x7f68a00b5540 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f6890009870 tx=0x7f689002a1c0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.972+0000 7f68a6f6d640 1 -- 172.21.15.88:0/530672868 shutdown_connections 2024-09-16T01:44:49.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.972+0000 7f68a6f6d640 1 --2- 172.21.15.88:0/530672868 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68a006fb10 0x7f68a015e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.972+0000 7f68a6f6d640 1 --2- 172.21.15.88:0/530672868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68a015e570 0x7f68a00b5540 unknown :-1 s=CLOSED pgs=2333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.972+0000 7f68a6f6d640 1 -- 172.21.15.88:0/530672868 >> 172.21.15.88:0/530672868 conn(0x7f68a00b4240 msgr2=0x7f68a00b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:49.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.972+0000 7f68a6f6d640 1 -- 172.21.15.88:0/530672868 shutdown_connections 2024-09-16T01:44:49.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.972+0000 7f68a6f6d640 1 -- 172.21.15.88:0/530672868 wait complete. 2024-09-16T01:44:49.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.973+0000 7f68a6f6d640 1 Processor -- start 2024-09-16T01:44:49.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.973+0000 7f68a6f6d640 1 -- start start 2024-09-16T01:44:49.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.974+0000 7f68a6f6d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68a006fb10 0x7f68a0147b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:49.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.974+0000 7f68a6f6d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68a00714a0 0x7f68a0141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:49.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.974+0000 7f68a6f6d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68a015e570 0x7f68a0144a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:49.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.974+0000 7f68a6f6d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f68a006d0e0 con 0x7f68a00714a0 2024-09-16T01:44:49.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.974+0000 7f68a6f6d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f68a006cf60 con 0x7f68a015e570 2024-09-16T01:44:49.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.974+0000 7f68a6f6d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f68a006d260 con 0x7f68a006fb10 2024-09-16T01:44:49.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.974+0000 7f689ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68a00714a0 0x7f68a0141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:49.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.974+0000 7f689ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68a00714a0 0x7f68a0141850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41786/0 (socket says 172.21.15.88:41786) 2024-09-16T01:44:49.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.974+0000 7f689ffff640 1 -- 172.21.15.88:0/2348582174 learned_addr learned my addr 172.21.15.88:0/2348582174 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:49.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.974+0000 7f68a550a640 1 --2- 172.21.15.88:0/2348582174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68a015e570 0x7f68a0144a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:49.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.974+0000 7f68a4d09640 1 --2- 172.21.15.88:0/2348582174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68a006fb10 0x7f68a0147b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:49.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.974+0000 7f689ffff640 1 -- 172.21.15.88:0/2348582174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68a006fb10 msgr2=0x7f68a0147b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:49.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.974+0000 7f689ffff640 1 --2- 172.21.15.88:0/2348582174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68a006fb10 0x7f68a0147b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.974+0000 7f689ffff640 1 -- 172.21.15.88:0/2348582174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68a015e570 msgr2=0x7f68a0144a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:49.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.974+0000 7f689ffff640 1 --2- 172.21.15.88:0/2348582174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68a015e570 0x7f68a0144a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:49.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.974+0000 7f689ffff640 1 -- 172.21.15.88:0/2348582174 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68a0144260 con 0x7f68a00714a0 2024-09-16T01:44:49.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.975+0000 7f68a4d09640 1 --2- 172.21.15.88:0/2348582174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68a006fb10 0x7f68a0147b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:49.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.975+0000 7f68a550a640 1 --2- 172.21.15.88:0/2348582174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68a015e570 0x7f68a0144a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:49.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.975+0000 7f689ffff640 1 --2- 172.21.15.88:0/2348582174 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68a00714a0 0x7f68a0141850 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f689400c890 tx=0x7f689400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:49.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.975+0000 7f689d7fa640 1 -- 172.21.15.88:0/2348582174 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6894018070 con 0x7f68a00714a0 2024-09-16T01:44:49.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.975+0000 7f689d7fa640 1 -- 172.21.15.88:0/2348582174 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f689400a640 con 0x7f68a00714a0 2024-09-16T01:44:49.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.975+0000 7f68a6f6d640 1 -- 172.21.15.88:0/2348582174 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68a0141d90 con 0x7f68a00714a0 2024-09-16T01:44:49.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.975+0000 7f689d7fa640 1 -- 172.21.15.88:0/2348582174 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689400aa30 con 0x7f68a00714a0 2024-09-16T01:44:49.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.975+0000 7f68a6f6d640 1 -- 172.21.15.88:0/2348582174 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68a0152570 con 0x7f68a00714a0 2024-09-16T01:44:49.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.976+0000 7f68a6f6d640 1 -- 172.21.15.88:0/2348582174 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68a00b5540 con 0x7f68a00714a0 2024-09-16T01:44:49.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.976+0000 7f689d7fa640 1 -- 172.21.15.88:0/2348582174 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f689401b070 con 0x7f68a00714a0 2024-09-16T01:44:49.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.977+0000 7f689d7fa640 1 --2- 172.21.15.88:0/2348582174 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f688c06c2b0 0x7f688c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:49.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.977+0000 7f689d7fa640 1 -- 172.21.15.88:0/2348582174 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 4857+0+0 (secure 0 0 0) 0x7f689408a700 con 0x7f68a00714a0 2024-09-16T01:44:49.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.977+0000 7f68a4d09640 1 --2- 172.21.15.88:0/2348582174 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f688c06c2b0 0x7f688c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:49.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.977+0000 7f68a4d09640 1 --2- 172.21.15.88:0/2348582174 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f688c06c2b0 0x7f688c06e770 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f6890034000 tx=0x7f689002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:49.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:49.981+0000 7f689d7fa640 1 -- 172.21.15.88:0/2348582174 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f6894059250 con 0x7f68a00714a0 2024-09-16T01:44:50.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.148+0000 7f68a6f6d640 1 -- 172.21.15.88:0/2348582174 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f68a00693c0 con 0x7f68a00714a0 2024-09-16T01:44:50.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.495+0000 7f689d7fa640 1 -- 172.21.15.88:0/2348582174 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v135) ==== 138+0+0 (secure 0 0 0) 0x7f6894058b50 con 0x7f68a00714a0 2024-09-16T01:44:50.496 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_metadata' created 2024-09-16T01:44:50.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.497+0000 7f68a6f6d640 1 -- 172.21.15.88:0/2348582174 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f688c06c2b0 msgr2=0x7f688c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:50.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.497+0000 7f68a6f6d640 1 --2- 172.21.15.88:0/2348582174 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f688c06c2b0 0x7f688c06e770 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f6890034000 tx=0x7f689002e000 comp rx=0 tx=0).stop 2024-09-16T01:44:50.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.497+0000 7f68a6f6d640 1 -- 172.21.15.88:0/2348582174 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68a00714a0 msgr2=0x7f68a0141850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:50.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.497+0000 7f68a6f6d640 1 --2- 172.21.15.88:0/2348582174 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68a00714a0 0x7f68a0141850 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f689400c890 tx=0x7f689400cd50 comp rx=0 tx=0).stop 2024-09-16T01:44:50.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.497+0000 7f68a6f6d640 1 -- 172.21.15.88:0/2348582174 shutdown_connections 2024-09-16T01:44:50.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.497+0000 7f68a6f6d640 1 --2- 172.21.15.88:0/2348582174 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f688c06c2b0 0x7f688c06e770 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:50.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.497+0000 7f68a6f6d640 1 --2- 172.21.15.88:0/2348582174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68a015e570 0x7f68a0144a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:50.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.497+0000 7f68a6f6d640 1 --2- 172.21.15.88:0/2348582174 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f68a00714a0 0x7f68a0141850 unknown :-1 s=CLOSED pgs=2334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:50.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.497+0000 7f68a6f6d640 1 --2- 172.21.15.88:0/2348582174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f68a006fb10 0x7f68a0147b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:50.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.497+0000 7f68a6f6d640 1 -- 172.21.15.88:0/2348582174 >> 172.21.15.88:0/2348582174 conn(0x7f68a00b4240 msgr2=0x7f68a006e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:50.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.497+0000 7f68a6f6d640 1 -- 172.21.15.88:0/2348582174 shutdown_connections 2024-09-16T01:44:50.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.497+0000 7f68a6f6d640 1 -- 172.21.15.88:0/2348582174 wait complete. 2024-09-16T01:44:50.513 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-16T01:44:50.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.618+0000 7f856e76d640 1 Processor -- start 2024-09-16T01:44:50.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.618+0000 7f856e76d640 1 -- start start 2024-09-16T01:44:50.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.619+0000 7f856e76d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f856815c2a0 0x7f85680735d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:50.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.619+0000 7f856e76d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8568073b10 0x7f856806f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:50.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.619+0000 7f856e76d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8568074120 con 0x7f856806fc50 2024-09-16T01:44:50.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.619+0000 7f856e76d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f856815c680 con 0x7f856815c2a0 2024-09-16T01:44:50.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.619+0000 7f856e76d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f856815ef00 con 0x7f8568073b10 2024-09-16T01:44:50.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.619+0000 7f85677fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8568073b10 0x7f856806f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:50.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.619+0000 7f8567fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f856815c2a0 0x7f85680735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:50.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.619+0000 7f8567fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f856815c2a0 0x7f85680735d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:50536/0 (socket says 172.21.15.88:50536) 2024-09-16T01:44:50.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.619+0000 7f85677fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8568073b10 0x7f856806f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:45312/0 (socket says 172.21.15.88:45312) 2024-09-16T01:44:50.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.619+0000 7f8567fff640 1 -- 172.21.15.88:0/423967667 learned_addr learned my addr 172.21.15.88:0/423967667 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:50.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.620+0000 7f8567fff640 1 -- 172.21.15.88:0/423967667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8568073b10 msgr2=0x7f856806f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:50.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.620+0000 7f8567fff640 1 --2- 172.21.15.88:0/423967667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8568073b10 0x7f856806f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:50.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.620+0000 7f8567fff640 1 -- 172.21.15.88:0/423967667 >> v1:172.21.15.154:6789/0 conn(0x7f856806fc50 legacy=0x7f856806cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:50.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.620+0000 7f8567fff640 1 -- 172.21.15.88:0/423967667 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f856815f1e0 con 0x7f856815c2a0 2024-09-16T01:44:50.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.620+0000 7f85677fe640 1 --2- 172.21.15.88:0/423967667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8568073b10 0x7f856806f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:44:50.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.620+0000 7f8567fff640 1 --2- 172.21.15.88:0/423967667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f856815c2a0 0x7f85680735d0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f8558009870 tx=0x7f855802a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b864f2a0f57bacd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:50.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.621+0000 7f8566ffd640 1 -- 172.21.15.88:0/423967667 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8558037070 con 0x7f856815c2a0 2024-09-16T01:44:50.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.621+0000 7f8566ffd640 1 -- 172.21.15.88:0/423967667 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8558032440 con 0x7f856815c2a0 2024-09-16T01:44:50.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.621+0000 7f8566ffd640 1 -- 172.21.15.88:0/423967667 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8558032810 con 0x7f856815c2a0 2024-09-16T01:44:50.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.621+0000 7f856e76d640 1 -- 172.21.15.88:0/423967667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f856815c2a0 msgr2=0x7f85680735d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:50.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.621+0000 7f856e76d640 1 --2- 172.21.15.88:0/423967667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f856815c2a0 0x7f85680735d0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f8558009870 tx=0x7f855802a270 comp rx=0 tx=0).stop 2024-09-16T01:44:50.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.621+0000 7f856e76d640 1 -- 172.21.15.88:0/423967667 shutdown_connections 2024-09-16T01:44:50.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.621+0000 7f856e76d640 1 --2- 172.21.15.88:0/423967667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8568073b10 0x7f856806f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:50.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.621+0000 7f856e76d640 1 --2- 172.21.15.88:0/423967667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f856815c2a0 0x7f85680735d0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:50.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.621+0000 7f856e76d640 1 -- 172.21.15.88:0/423967667 >> 172.21.15.88:0/423967667 conn(0x7f85680b3070 msgr2=0x7f85680b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:50.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.621+0000 7f856e76d640 1 -- 172.21.15.88:0/423967667 shutdown_connections 2024-09-16T01:44:50.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.621+0000 7f856e76d640 1 -- 172.21.15.88:0/423967667 wait complete. 2024-09-16T01:44:50.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.622+0000 7f856e76d640 1 Processor -- start 2024-09-16T01:44:50.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.623+0000 7f856e76d640 1 -- start start 2024-09-16T01:44:50.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.623+0000 7f856e76d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f856806fc50 0x7f8568149cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:50.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.623+0000 7f856e76d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8568073b10 0x7f856815b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:50.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.623+0000 7f856e76d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f856815c2a0 0x7f8568160e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:50.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.623+0000 7f856e76d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f85680b5f20 con 0x7f856815c2a0 2024-09-16T01:44:50.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.623+0000 7f856e76d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f856815ebf0 con 0x7f856806fc50 2024-09-16T01:44:50.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.623+0000 7f856e76d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f85680b60a0 con 0x7f8568073b10 2024-09-16T01:44:50.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.623+0000 7f856cd0a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f856815c2a0 0x7f8568160e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:50.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.623+0000 7f856cd0a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f856815c2a0 0x7f8568160e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35712/0 (socket says 172.21.15.88:35712) 2024-09-16T01:44:50.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.623+0000 7f856cd0a640 1 -- 172.21.15.88:0/718492846 learned_addr learned my addr 172.21.15.88:0/718492846 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:50.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.624+0000 7f8567fff640 1 --2- 172.21.15.88:0/718492846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f856806fc50 0x7f8568149cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:50.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.624+0000 7f85677fe640 1 --2- 172.21.15.88:0/718492846 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8568073b10 0x7f856815b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:50.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.624+0000 7f856cd0a640 1 -- 172.21.15.88:0/718492846 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8568073b10 msgr2=0x7f856815b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:50.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.624+0000 7f856cd0a640 1 --2- 172.21.15.88:0/718492846 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8568073b10 0x7f856815b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:50.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.624+0000 7f856cd0a640 1 -- 172.21.15.88:0/718492846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f856806fc50 msgr2=0x7f8568149cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:50.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.624+0000 7f856cd0a640 1 --2- 172.21.15.88:0/718492846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f856806fc50 0x7f8568149cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:50.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.624+0000 7f856cd0a640 1 -- 172.21.15.88:0/718492846 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85680d0480 con 0x7f856815c2a0 2024-09-16T01:44:50.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.624+0000 7f856cd0a640 1 --2- 172.21.15.88:0/718492846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f856815c2a0 0x7f8568160e80 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7f856000c6b0 tx=0x7f856000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:50.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.624+0000 7f8564ff9640 1 -- 172.21.15.88:0/718492846 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8560003b70 con 0x7f856815c2a0 2024-09-16T01:44:50.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.624+0000 7f8564ff9640 1 -- 172.21.15.88:0/718492846 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8560003d30 con 0x7f856815c2a0 2024-09-16T01:44:50.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.624+0000 7f856e76d640 1 -- 172.21.15.88:0/718492846 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85680d0760 con 0x7f856815c2a0 2024-09-16T01:44:50.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.624+0000 7f8564ff9640 1 -- 172.21.15.88:0/718492846 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8560005310 con 0x7f856815c2a0 2024-09-16T01:44:50.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.624+0000 7f856e76d640 1 -- 172.21.15.88:0/718492846 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f856814a200 con 0x7f856815c2a0 2024-09-16T01:44:50.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.625+0000 7f856e76d640 1 -- 172.21.15.88:0/718492846 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85680d0c80 con 0x7f856815c2a0 2024-09-16T01:44:50.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.626+0000 7f8564ff9640 1 -- 172.21.15.88:0/718492846 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f85600054b0 con 0x7f856815c2a0 2024-09-16T01:44:50.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.626+0000 7f8564ff9640 1 --2- 172.21.15.88:0/718492846 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f855406c2b0 0x7f855406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:50.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.626+0000 7f8564ff9640 1 -- 172.21.15.88:0/718492846 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 5227+0+0 (secure 0 0 0) 0x7f856008af20 con 0x7f856815c2a0 2024-09-16T01:44:50.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.626+0000 7f8567fff640 1 --2- 172.21.15.88:0/718492846 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f855406c2b0 0x7f855406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:50.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.627+0000 7f8567fff640 1 --2- 172.21.15.88:0/718492846 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f855406c2b0 0x7f855406e770 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f8558034000 tx=0x7f855802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:50.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.631+0000 7f8564ff9640 1 -- 172.21.15.88:0/718492846 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f85680d0c80 con 0x7f856815c2a0 2024-09-16T01:44:50.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:50.797+0000 7f856e76d640 1 -- 172.21.15.88:0/718492846 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f856815f9e0 con 0x7f856815c2a0 2024-09-16T01:44:51.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.501+0000 7f8564ff9640 1 -- 172.21.15.88:0/718492846 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v136) ==== 170+0+0 (secure 0 0 0) 0x7f8560059900 con 0x7f856815c2a0 2024-09-16T01:44:51.503 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_data' created 2024-09-16T01:44:51.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.504+0000 7f856e76d640 1 -- 172.21.15.88:0/718492846 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f855406c2b0 msgr2=0x7f855406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:51.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.504+0000 7f856e76d640 1 --2- 172.21.15.88:0/718492846 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f855406c2b0 0x7f855406e770 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f8558034000 tx=0x7f855802e000 comp rx=0 tx=0).stop 2024-09-16T01:44:51.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.504+0000 7f856e76d640 1 -- 172.21.15.88:0/718492846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f856815c2a0 msgr2=0x7f8568160e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:51.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.504+0000 7f856e76d640 1 --2- 172.21.15.88:0/718492846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f856815c2a0 0x7f8568160e80 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7f856000c6b0 tx=0x7f856000cb70 comp rx=0 tx=0).stop 2024-09-16T01:44:51.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.504+0000 7f856e76d640 1 -- 172.21.15.88:0/718492846 shutdown_connections 2024-09-16T01:44:51.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.504+0000 7f856e76d640 1 --2- 172.21.15.88:0/718492846 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f855406c2b0 0x7f855406e770 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:51.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.504+0000 7f856e76d640 1 --2- 172.21.15.88:0/718492846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f856815c2a0 0x7f8568160e80 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:51.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.504+0000 7f856e76d640 1 --2- 172.21.15.88:0/718492846 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8568073b10 0x7f856815b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:51.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.504+0000 7f856e76d640 1 --2- 172.21.15.88:0/718492846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f856806fc50 0x7f8568149cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:51.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.504+0000 7f856e76d640 1 -- 172.21.15.88:0/718492846 >> 172.21.15.88:0/718492846 conn(0x7f85680b3070 msgr2=0x7f85680af290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:51.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.504+0000 7f856e76d640 1 -- 172.21.15.88:0/718492846 shutdown_connections 2024-09-16T01:44:51.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.504+0000 7f856e76d640 1 -- 172.21.15.88:0/718492846 wait complete. 2024-09-16T01:44:51.530 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-16T01:44:51.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.632+0000 7f2931131640 1 Processor -- start 2024-09-16T01:44:51.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.632+0000 7f2931131640 1 -- start start 2024-09-16T01:44:51.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.633+0000 7f2931131640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f292c158a00 0x7f292c071f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:51.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.633+0000 7f2931131640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f292c072470 0x7f292c0b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:51.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.633+0000 7f2931131640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f292c0728f0 con 0x7f292c158de0 2024-09-16T01:44:51.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.633+0000 7f2931131640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f292c072bd0 con 0x7f292c072470 2024-09-16T01:44:51.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.633+0000 7f2931131640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f292c072eb0 con 0x7f292c158a00 2024-09-16T01:44:51.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.633+0000 7f292a59c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f292c072470 0x7f292c0b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:51.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.633+0000 7f292b59e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f292c158de0 0x7f292c162110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35018/0 (socket says 172.21.15.88:35018) 2024-09-16T01:44:51.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.633+0000 7f292b59e640 1 -- 172.21.15.88:0/3518568657 learned_addr learned my addr 172.21.15.88:0/3518568657 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:51.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.633+0000 7f292ad9d640 1 --2- 172.21.15.88:0/3518568657 >> v2:172.21.15.154:3300/0 conn(0x7f292c158a00 0x7f292c071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:51.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.633+0000 7f292a59c640 1 -- 172.21.15.88:0/3518568657 >> v2:172.21.15.154:3300/0 conn(0x7f292c158a00 msgr2=0x7f292c071f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:51.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.633+0000 7f292a59c640 1 --2- 172.21.15.88:0/3518568657 >> v2:172.21.15.154:3300/0 conn(0x7f292c158a00 0x7f292c071f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:51.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.633+0000 7f292a59c640 1 -- 172.21.15.88:0/3518568657 >> v1:172.21.15.88:6789/0 conn(0x7f292c158de0 legacy=0x7f292c162110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:51.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.633+0000 7f292a59c640 1 -- 172.21.15.88:0/3518568657 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f292c073190 con 0x7f292c072470 2024-09-16T01:44:51.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.633+0000 7f2929d9b640 1 -- 172.21.15.88:0/3518568657 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 343351425 0 0) 0x7f292c0728f0 con 0x7f292c158de0 2024-09-16T01:44:51.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.634+0000 7f292a59c640 1 --2- 172.21.15.88:0/3518568657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f292c072470 0x7f292c0b4e60 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7f292c069f90 tx=0x7f292002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=690add676c71aa69 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:51.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.634+0000 7f2929d9b640 1 -- 172.21.15.88:0/3518568657 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2920037070 con 0x7f292c072470 2024-09-16T01:44:51.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.634+0000 7f2929d9b640 1 -- 172.21.15.88:0/3518568657 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2920032440 con 0x7f292c072470 2024-09-16T01:44:51.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.634+0000 7f2929d9b640 1 -- 172.21.15.88:0/3518568657 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2920032810 con 0x7f292c072470 2024-09-16T01:44:51.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.634+0000 7f2931131640 1 -- 172.21.15.88:0/3518568657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f292c072470 msgr2=0x7f292c0b4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:51.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.634+0000 7f2931131640 1 --2- 172.21.15.88:0/3518568657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f292c072470 0x7f292c0b4e60 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7f292c069f90 tx=0x7f292002a1a0 comp rx=0 tx=0).stop 2024-09-16T01:44:51.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.634+0000 7f2931131640 1 -- 172.21.15.88:0/3518568657 shutdown_connections 2024-09-16T01:44:51.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.634+0000 7f2931131640 1 --2- 172.21.15.88:0/3518568657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f292c072470 0x7f292c0b4e60 unknown :-1 s=CLOSED pgs=2337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:51.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.634+0000 7f2931131640 1 --2- 172.21.15.88:0/3518568657 >> v2:172.21.15.154:3300/0 conn(0x7f292c158a00 0x7f292c071f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:51.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.634+0000 7f2931131640 1 -- 172.21.15.88:0/3518568657 >> 172.21.15.88:0/3518568657 conn(0x7f292c06cb50 msgr2=0x7f292c06cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:51.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.634+0000 7f2931131640 1 -- 172.21.15.88:0/3518568657 shutdown_connections 2024-09-16T01:44:51.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.634+0000 7f2931131640 1 -- 172.21.15.88:0/3518568657 wait complete. 2024-09-16T01:44:51.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.636+0000 7f2931131640 1 Processor -- start 2024-09-16T01:44:51.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.636+0000 7f2931131640 1 -- start start 2024-09-16T01:44:51.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.637+0000 7f2931131640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f292c072470 0x7f292c1f7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:51.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.637+0000 7f2931131640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f292c158a00 0x7f292c0cef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:51.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.637+0000 7f2931131640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f292c158de0 0x7f292c154780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:51.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.637+0000 7f2931131640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f292c0b5ca0 con 0x7f292c158de0 2024-09-16T01:44:51.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.637+0000 7f2931131640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f292c0b5b20 con 0x7f292c072470 2024-09-16T01:44:51.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.637+0000 7f2931131640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f292c06faf0 con 0x7f292c158a00 2024-09-16T01:44:51.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.637+0000 7f292b59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f292c158de0 0x7f292c154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:51.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.637+0000 7f292b59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f292c158de0 0x7f292c154780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35728/0 (socket says 172.21.15.88:35728) 2024-09-16T01:44:51.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.637+0000 7f292b59e640 1 -- 172.21.15.88:0/3539364879 learned_addr learned my addr 172.21.15.88:0/3539364879 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:51.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.637+0000 7f292ad9d640 1 --2- 172.21.15.88:0/3539364879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f292c072470 0x7f292c1f7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:51.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.637+0000 7f292a59c640 1 --2- 172.21.15.88:0/3539364879 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f292c158a00 0x7f292c0cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:51.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.637+0000 7f292b59e640 1 -- 172.21.15.88:0/3539364879 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f292c158a00 msgr2=0x7f292c0cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:51.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.637+0000 7f292b59e640 1 --2- 172.21.15.88:0/3539364879 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f292c158a00 0x7f292c0cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:51.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.637+0000 7f292b59e640 1 -- 172.21.15.88:0/3539364879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f292c072470 msgr2=0x7f292c1f7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:51.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.637+0000 7f292b59e640 1 --2- 172.21.15.88:0/3539364879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f292c072470 0x7f292c1f7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:51.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.637+0000 7f292b59e640 1 -- 172.21.15.88:0/3539364879 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f292c1f7ba0 con 0x7f292c158de0 2024-09-16T01:44:51.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.637+0000 7f292a59c640 1 --2- 172.21.15.88:0/3539364879 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f292c158a00 0x7f292c0cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:51.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.637+0000 7f292b59e640 1 --2- 172.21.15.88:0/3539364879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f292c158de0 0x7f292c154780 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f292400c700 tx=0x7f292400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:51.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.637+0000 7f29137fe640 1 -- 172.21.15.88:0/3539364879 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2924018070 con 0x7f292c158de0 2024-09-16T01:44:51.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.637+0000 7f29137fe640 1 -- 172.21.15.88:0/3539364879 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f292400ce10 con 0x7f292c158de0 2024-09-16T01:44:51.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.638+0000 7f2931131640 1 -- 172.21.15.88:0/3539364879 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f292c1f7e80 con 0x7f292c158de0 2024-09-16T01:44:51.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.638+0000 7f29137fe640 1 -- 172.21.15.88:0/3539364879 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f292400a860 con 0x7f292c158de0 2024-09-16T01:44:51.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.638+0000 7f2931131640 1 -- 172.21.15.88:0/3539364879 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f292c0cf4a0 con 0x7f292c158de0 2024-09-16T01:44:51.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.638+0000 7f2931131640 1 -- 172.21.15.88:0/3539364879 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f292c1f83a0 con 0x7f292c158de0 2024-09-16T01:44:51.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.639+0000 7f29137fe640 1 -- 172.21.15.88:0/3539364879 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f292401b070 con 0x7f292c158de0 2024-09-16T01:44:51.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.639+0000 7f29137fe640 1 --2- 172.21.15.88:0/3539364879 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f291806c2b0 0x7f291806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:51.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.639+0000 7f29137fe640 1 -- 172.21.15.88:0/3539364879 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5609+0+0 (secure 0 0 0) 0x7f292408a9e0 con 0x7f292c158de0 2024-09-16T01:44:51.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.640+0000 7f292ad9d640 1 --2- 172.21.15.88:0/3539364879 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f291806c2b0 0x7f291806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:51.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.640+0000 7f292ad9d640 1 --2- 172.21.15.88:0/3539364879 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f291806c2b0 0x7f291806e770 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f291c006fd0 tx=0x7f291c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:51.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.644+0000 7f29137fe640 1 -- 172.21.15.88:0/3539364879 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f29240591b0 con 0x7f292c158de0 2024-09-16T01:44:51.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:51.843+0000 7f2931131640 1 -- 172.21.15.88:0/3539364879 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f292c071f30 con 0x7f292c158de0 2024-09-16T01:44:52.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.503+0000 7f29137fe640 1 -- 172.21.15.88:0/3539364879 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v323) ==== 174+0+0 (secure 0 0 0) 0x7f2924058ab0 con 0x7f292c158de0 2024-09-16T01:44:52.505 INFO:teuthology.orchestra.run.smithi088.stderr:new fs with metadata pool 14 and data pool 15 2024-09-16T01:44:52.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.506+0000 7f2931131640 1 -- 172.21.15.88:0/3539364879 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f291806c2b0 msgr2=0x7f291806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:52.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.506+0000 7f2931131640 1 --2- 172.21.15.88:0/3539364879 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f291806c2b0 0x7f291806e770 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f291c006fd0 tx=0x7f291c008040 comp rx=0 tx=0).stop 2024-09-16T01:44:52.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.506+0000 7f2931131640 1 -- 172.21.15.88:0/3539364879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f292c158de0 msgr2=0x7f292c154780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:52.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.507+0000 7f2931131640 1 --2- 172.21.15.88:0/3539364879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f292c158de0 0x7f292c154780 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f292400c700 tx=0x7f292400cbc0 comp rx=0 tx=0).stop 2024-09-16T01:44:52.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.507+0000 7f2931131640 1 -- 172.21.15.88:0/3539364879 shutdown_connections 2024-09-16T01:44:52.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.507+0000 7f2931131640 1 --2- 172.21.15.88:0/3539364879 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f291806c2b0 0x7f291806e770 secure :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f291c006fd0 tx=0x7f291c008040 comp rx=0 tx=0).stop 2024-09-16T01:44:52.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.507+0000 7f2931131640 1 --2- 172.21.15.88:0/3539364879 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f292c158de0 0x7f292c154780 unknown :-1 s=CLOSED pgs=2338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:52.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.507+0000 7f2931131640 1 --2- 172.21.15.88:0/3539364879 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f292c158a00 0x7f292c0cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:52.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.507+0000 7f2931131640 1 --2- 172.21.15.88:0/3539364879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f292c072470 0x7f292c1f7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:52.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.507+0000 7f2931131640 1 -- 172.21.15.88:0/3539364879 >> 172.21.15.88:0/3539364879 conn(0x7f292c06cb50 msgr2=0x7f292c0b4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:52.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.507+0000 7f2931131640 1 -- 172.21.15.88:0/3539364879 shutdown_connections 2024-09-16T01:44:52.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.507+0000 7f2931131640 1 -- 172.21.15.88:0/3539364879 wait complete. 2024-09-16T01:44:52.538 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:44:52.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.634+0000 7f8ce5e77640 1 Processor -- start 2024-09-16T01:44:52.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.634+0000 7f8ce5e77640 1 -- start start 2024-09-16T01:44:52.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.635+0000 7f8ce5e77640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8ce015c170 0x7f8ce015c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:52.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.635+0000 7f8ce5e77640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce00b64e0 0x7f8ce00cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:52.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.635+0000 7f8ce5e77640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ce00b4cf0 con 0x7f8ce00cf6c0 2024-09-16T01:44:52.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.635+0000 7f8ce5e77640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f8ce015ca90 con 0x7f8ce015c170 2024-09-16T01:44:52.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.635+0000 7f8ce5e77640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8ce00b49c0 con 0x7f8ce00b64e0 2024-09-16T01:44:52.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.635+0000 7f8cdffff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce00b64e0 0x7f8ce00cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:52.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.635+0000 7f8cdf7fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8ce015c170 0x7f8ce015c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:52.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.635+0000 7f8cdffff640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce00b64e0 0x7f8ce00cfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:45354/0 (socket says 172.21.15.88:45354) 2024-09-16T01:44:52.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.635+0000 7f8cdf7fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8ce015c170 0x7f8ce015c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:50566/0 (socket says 172.21.15.88:50566) 2024-09-16T01:44:52.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.635+0000 7f8cdffff640 1 -- 172.21.15.88:0/322933769 learned_addr learned my addr 172.21.15.88:0/322933769 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:52.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.636+0000 7f8cdffff640 1 -- 172.21.15.88:0/322933769 >> v2:172.21.15.154:3300/0 conn(0x7f8ce015c170 msgr2=0x7f8ce015c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:52.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.636+0000 7f8cdffff640 1 --2- 172.21.15.88:0/322933769 >> v2:172.21.15.154:3300/0 conn(0x7f8ce015c170 0x7f8ce015c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:52.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.636+0000 7f8cdffff640 1 -- 172.21.15.88:0/322933769 >> v1:172.21.15.154:6789/0 conn(0x7f8ce00cf6c0 legacy=0x7f8ce00b5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:52.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.636+0000 7f8cdffff640 1 -- 172.21.15.88:0/322933769 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ce00b4fd0 con 0x7f8ce00b64e0 2024-09-16T01:44:52.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.636+0000 7f8cdffff640 1 --2- 172.21.15.88:0/322933769 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce00b64e0 0x7f8ce00cfb30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8cd80098a0 tx=0x7f8cd802a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1e68b3785e4fbed server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:52.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.637+0000 7f8cde7fc640 1 -- 172.21.15.88:0/322933769 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd8037070 con 0x7f8ce00b64e0 2024-09-16T01:44:52.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.637+0000 7f8cde7fc640 1 -- 172.21.15.88:0/322933769 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cd8032440 con 0x7f8ce00b64e0 2024-09-16T01:44:52.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.637+0000 7f8cde7fc640 1 -- 172.21.15.88:0/322933769 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd8032810 con 0x7f8ce00b64e0 2024-09-16T01:44:52.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.637+0000 7f8ce5e77640 1 -- 172.21.15.88:0/322933769 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce00b64e0 msgr2=0x7f8ce00cfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:52.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.637+0000 7f8ce5e77640 1 --2- 172.21.15.88:0/322933769 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce00b64e0 0x7f8ce00cfb30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8cd80098a0 tx=0x7f8cd802a290 comp rx=0 tx=0).stop 2024-09-16T01:44:52.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.637+0000 7f8ce5e77640 1 -- 172.21.15.88:0/322933769 shutdown_connections 2024-09-16T01:44:52.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.637+0000 7f8ce5e77640 1 --2- 172.21.15.88:0/322933769 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce00b64e0 0x7f8ce00cfb30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:52.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.637+0000 7f8ce5e77640 1 --2- 172.21.15.88:0/322933769 >> v2:172.21.15.154:3300/0 conn(0x7f8ce015c170 0x7f8ce015c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:52.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.637+0000 7f8ce5e77640 1 -- 172.21.15.88:0/322933769 >> 172.21.15.88:0/322933769 conn(0x7f8ce006bc10 msgr2=0x7f8ce006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:52.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.637+0000 7f8ce5e77640 1 -- 172.21.15.88:0/322933769 shutdown_connections 2024-09-16T01:44:52.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.637+0000 7f8ce5e77640 1 -- 172.21.15.88:0/322933769 wait complete. 2024-09-16T01:44:52.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.639+0000 7f8ce5e77640 1 Processor -- start 2024-09-16T01:44:52.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.639+0000 7f8ce5e77640 1 -- start start 2024-09-16T01:44:52.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.639+0000 7f8ce5e77640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ce00b64e0 0x7f8ce015a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:52.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.639+0000 7f8ce5e77640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ce00cf6c0 0x7f8ce01ee0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:52.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.639+0000 7f8ce5e77640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce015c170 0x7f8ce015ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:52.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.639+0000 7f8ce5e77640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f8ce00b5870 con 0x7f8ce00b64e0 2024-09-16T01:44:52.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.639+0000 7f8ce5e77640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8ce00b5590 con 0x7f8ce00cf6c0 2024-09-16T01:44:52.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.639+0000 7f8ce5e77640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8ce00b5b50 con 0x7f8ce015c170 2024-09-16T01:44:52.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.640+0000 7f8cdf7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ce00b64e0 0x7f8ce015a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:52.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.640+0000 7f8cdf7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ce00b64e0 0x7f8ce015a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35744/0 (socket says 172.21.15.88:35744) 2024-09-16T01:44:52.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.640+0000 7f8cdf7fe640 1 -- 172.21.15.88:0/512865494 learned_addr learned my addr 172.21.15.88:0/512865494 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:52.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.640+0000 7f8cdffff640 1 --2- 172.21.15.88:0/512865494 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce015c170 0x7f8ce015ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:52.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.640+0000 7f8cdeffd640 1 --2- 172.21.15.88:0/512865494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ce00cf6c0 0x7f8ce01ee0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:52.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.640+0000 7f8cdf7fe640 1 -- 172.21.15.88:0/512865494 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce015c170 msgr2=0x7f8ce015ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:52.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.640+0000 7f8cdf7fe640 1 --2- 172.21.15.88:0/512865494 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce015c170 0x7f8ce015ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:52.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.640+0000 7f8cdf7fe640 1 -- 172.21.15.88:0/512865494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ce00cf6c0 msgr2=0x7f8ce01ee0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:52.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.640+0000 7f8cdf7fe640 1 --2- 172.21.15.88:0/512865494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ce00cf6c0 0x7f8ce01ee0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:52.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.640+0000 7f8cdf7fe640 1 -- 172.21.15.88:0/512865494 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ce01f4860 con 0x7f8ce00b64e0 2024-09-16T01:44:52.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.640+0000 7f8cdf7fe640 1 --2- 172.21.15.88:0/512865494 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ce00b64e0 0x7f8ce015a170 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f8cd000efc0 tx=0x7f8cd000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:52.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.640+0000 7f8cc7fff640 1 -- 172.21.15.88:0/512865494 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd0018020 con 0x7f8ce00b64e0 2024-09-16T01:44:52.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.640+0000 7f8cc7fff640 1 -- 172.21.15.88:0/512865494 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cd0014480 con 0x7f8ce00b64e0 2024-09-16T01:44:52.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.640+0000 7f8ce5e77640 1 -- 172.21.15.88:0/512865494 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ce01f4b40 con 0x7f8ce00b64e0 2024-09-16T01:44:52.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.640+0000 7f8cc7fff640 1 -- 172.21.15.88:0/512865494 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd0014850 con 0x7f8ce00b64e0 2024-09-16T01:44:52.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.640+0000 7f8ce5e77640 1 -- 172.21.15.88:0/512865494 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ce0150470 con 0x7f8ce00b64e0 2024-09-16T01:44:52.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.641+0000 7f8ce5e77640 1 -- 172.21.15.88:0/512865494 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ce00cfb30 con 0x7f8ce00b64e0 2024-09-16T01:44:52.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.642+0000 7f8cc7fff640 1 -- 172.21.15.88:0/512865494 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f8cd0005d10 con 0x7f8ce00b64e0 2024-09-16T01:44:52.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.642+0000 7f8cc7fff640 1 --2- 172.21.15.88:0/512865494 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8ccc06c2b0 0x7f8ccc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:52.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.642+0000 7f8cc7fff640 1 -- 172.21.15.88:0/512865494 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 5709+0+0 (secure 0 0 0) 0x7f8cd008bd00 con 0x7f8ce00b64e0 2024-09-16T01:44:52.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.642+0000 7f8cdeffd640 1 --2- 172.21.15.88:0/512865494 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8ccc06c2b0 0x7f8ccc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:52.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.643+0000 7f8cdeffd640 1 --2- 172.21.15.88:0/512865494 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8ccc06c2b0 0x7f8ccc06e770 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f8cd4001780 tx=0x7f8cd4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:52.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.646+0000 7f8cc7fff640 1 -- 172.21.15.88:0/512865494 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f8cd0022070 con 0x7f8ce00b64e0 2024-09-16T01:44:52.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.816+0000 7f8ce5e77640 1 -- 172.21.15.88:0/512865494 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8ce00b2e20 con 0x7f8ce00b64e0 2024-09-16T01:44:52.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.817+0000 7f8cc7fff640 1 -- 172.21.15.88:0/512865494 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v137) ==== 74+0+16668 (secure 0 0 0) 0x7f8cd0010040 con 0x7f8ce00b64e0 2024-09-16T01:44:52.819 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:52.819 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":137,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:44:51.844752+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:44:50.167367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-09-16T01:44:50.816821+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:44:52.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.819+0000 7f8ce5e77640 1 -- 172.21.15.88:0/512865494 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8ccc06c2b0 msgr2=0x7f8ccc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:52.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.819+0000 7f8ce5e77640 1 --2- 172.21.15.88:0/512865494 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8ccc06c2b0 0x7f8ccc06e770 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f8cd4001780 tx=0x7f8cd4009040 comp rx=0 tx=0).stop 2024-09-16T01:44:52.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.819+0000 7f8ce5e77640 1 -- 172.21.15.88:0/512865494 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ce00b64e0 msgr2=0x7f8ce015a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:52.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.819+0000 7f8ce5e77640 1 --2- 172.21.15.88:0/512865494 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ce00b64e0 0x7f8ce015a170 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f8cd000efc0 tx=0x7f8cd000c490 comp rx=0 tx=0).stop 2024-09-16T01:44:52.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.819+0000 7f8ce5e77640 1 -- 172.21.15.88:0/512865494 shutdown_connections 2024-09-16T01:44:52.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.819+0000 7f8ce5e77640 1 --2- 172.21.15.88:0/512865494 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f8ccc06c2b0 0x7f8ccc06e770 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:52.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.819+0000 7f8ce5e77640 1 --2- 172.21.15.88:0/512865494 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce015c170 0x7f8ce015ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:52.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.819+0000 7f8ce5e77640 1 --2- 172.21.15.88:0/512865494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ce00cf6c0 0x7f8ce01ee0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:52.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.819+0000 7f8ce5e77640 1 --2- 172.21.15.88:0/512865494 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ce00b64e0 0x7f8ce015a170 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:52.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.819+0000 7f8ce5e77640 1 -- 172.21.15.88:0/512865494 >> 172.21.15.88:0/512865494 conn(0x7f8ce006bc10 msgr2=0x7f8ce00c5680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:52.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.819+0000 7f8ce5e77640 1 -- 172.21.15.88:0/512865494 shutdown_connections 2024-09-16T01:44:52.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.819+0000 7f8ce5e77640 1 -- 172.21.15.88:0/512865494 wait complete. 2024-09-16T01:44:52.832 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:44:52.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.928+0000 7f8043bc4640 1 Processor -- start 2024-09-16T01:44:52.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.928+0000 7f8043bc4640 1 -- start start 2024-09-16T01:44:52.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.929+0000 7f8043bc4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f803c0b4920 0x7f803c15b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:52.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.929+0000 7f8043bc4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f803c0b4d00 0x7f803c0b5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:52.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.929+0000 7f8043bc4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f803c0ce4f0 con 0x7f803c0cd780 2024-09-16T01:44:52.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.929+0000 7f8043bc4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f803c0cf100 con 0x7f803c0b4d00 2024-09-16T01:44:52.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.929+0000 7f8043bc4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f803c0ce2a0 con 0x7f803c0b4920 2024-09-16T01:44:52.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.929+0000 7f804115f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f803c0b4d00 0x7f803c0b5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:52.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.929+0000 7f804115f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f803c0b4d00 0x7f803c0b5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35750/0 (socket says 172.21.15.88:35750) 2024-09-16T01:44:52.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.929+0000 7f804115f640 1 -- 172.21.15.88:0/2638834301 learned_addr learned my addr 172.21.15.88:0/2638834301 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:52.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.929+0000 7f804115f640 1 -- 172.21.15.88:0/2638834301 >> v2:172.21.15.154:3300/0 conn(0x7f803c0b4920 msgr2=0x7f803c15b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:52.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.929+0000 7f804115f640 1 --2- 172.21.15.88:0/2638834301 >> v2:172.21.15.154:3300/0 conn(0x7f803c0b4920 0x7f803c15b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:52.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.929+0000 7f804115f640 1 -- 172.21.15.88:0/2638834301 >> v1:172.21.15.154:6789/0 conn(0x7f803c0cd780 legacy=0x7f803c0cdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:52.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.929+0000 7f804115f640 1 -- 172.21.15.88:0/2638834301 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f803c0ce7d0 con 0x7f803c0b4d00 2024-09-16T01:44:52.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.930+0000 7f804115f640 1 --2- 172.21.15.88:0/2638834301 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f803c0b4d00 0x7f803c0b5170 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f803c06cb00 tx=0x7f802c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=438d3612497e2317 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:52.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.930+0000 7f804095e640 1 -- 172.21.15.88:0/2638834301 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802c037070 con 0x7f803c0b4d00 2024-09-16T01:44:52.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.930+0000 7f804095e640 1 -- 172.21.15.88:0/2638834301 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f802c032440 con 0x7f803c0b4d00 2024-09-16T01:44:52.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.930+0000 7f804095e640 1 -- 172.21.15.88:0/2638834301 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802c032810 con 0x7f803c0b4d00 2024-09-16T01:44:52.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.930+0000 7f8043bc4640 1 -- 172.21.15.88:0/2638834301 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f803c0b4d00 msgr2=0x7f803c0b5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:52.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.930+0000 7f8043bc4640 1 --2- 172.21.15.88:0/2638834301 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f803c0b4d00 0x7f803c0b5170 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f803c06cb00 tx=0x7f802c02a230 comp rx=0 tx=0).stop 2024-09-16T01:44:52.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.931+0000 7f8043bc4640 1 -- 172.21.15.88:0/2638834301 shutdown_connections 2024-09-16T01:44:52.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.931+0000 7f8043bc4640 1 --2- 172.21.15.88:0/2638834301 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f803c0b4d00 0x7f803c0b5170 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:52.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.931+0000 7f8043bc4640 1 --2- 172.21.15.88:0/2638834301 >> v2:172.21.15.154:3300/0 conn(0x7f803c0b4920 0x7f803c15b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:52.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.931+0000 7f8043bc4640 1 -- 172.21.15.88:0/2638834301 >> 172.21.15.88:0/2638834301 conn(0x7f803c06bc10 msgr2=0x7f803c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:52.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.931+0000 7f8043bc4640 1 -- 172.21.15.88:0/2638834301 shutdown_connections 2024-09-16T01:44:52.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.931+0000 7f8043bc4640 1 -- 172.21.15.88:0/2638834301 wait complete. 2024-09-16T01:44:52.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.932+0000 7f8043bc4640 1 Processor -- start 2024-09-16T01:44:52.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.932+0000 7f8043bc4640 1 -- start start 2024-09-16T01:44:52.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.932+0000 7f8043bc4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f803c0b4920 0x7f803c144540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:52.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.932+0000 7f8043bc4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f803c0b4d00 0x7f803c14a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:52.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.932+0000 7f8043bc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f803c0cd780 0x7f803c14a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:52.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.932+0000 7f8043bc4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f803c0b56b0 con 0x7f803c0b4d00 2024-09-16T01:44:52.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.932+0000 7f8043bc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f803c0ced90 con 0x7f803c0cd780 2024-09-16T01:44:52.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.932+0000 7f8043bc4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f803c15bd10 con 0x7f803c0b4920 2024-09-16T01:44:52.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.933+0000 7f804115f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f803c0b4d00 0x7f803c14a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:52.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.933+0000 7f804115f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f803c0b4d00 0x7f803c14a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35764/0 (socket says 172.21.15.88:35764) 2024-09-16T01:44:52.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.933+0000 7f804115f640 1 -- 172.21.15.88:0/1964414422 learned_addr learned my addr 172.21.15.88:0/1964414422 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:52.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.933+0000 7f8041960640 1 --2- 172.21.15.88:0/1964414422 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f803c0b4920 0x7f803c144540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:52.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.933+0000 7f8042161640 1 --2- 172.21.15.88:0/1964414422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f803c0cd780 0x7f803c14a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:52.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.933+0000 7f804115f640 1 -- 172.21.15.88:0/1964414422 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f803c0b4920 msgr2=0x7f803c144540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:52.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.933+0000 7f804115f640 1 --2- 172.21.15.88:0/1964414422 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f803c0b4920 0x7f803c144540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:52.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.933+0000 7f804115f640 1 -- 172.21.15.88:0/1964414422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f803c0cd780 msgr2=0x7f803c14a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:52.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.933+0000 7f804115f640 1 --2- 172.21.15.88:0/1964414422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f803c0cd780 0x7f803c14a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:52.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.933+0000 7f804115f640 1 -- 172.21.15.88:0/1964414422 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f803c14f0b0 con 0x7f803c0b4d00 2024-09-16T01:44:52.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.933+0000 7f8042161640 1 --2- 172.21.15.88:0/1964414422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f803c0cd780 0x7f803c14a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:52.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.933+0000 7f804115f640 1 --2- 172.21.15.88:0/1964414422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f803c0b4d00 0x7f803c14a1b0 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7f802c000c00 tx=0x7f802c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:52.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.933+0000 7f80327fc640 1 -- 172.21.15.88:0/1964414422 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802c041070 con 0x7f803c0b4d00 2024-09-16T01:44:52.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.933+0000 7f80327fc640 1 -- 172.21.15.88:0/1964414422 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f802c03c870 con 0x7f803c0b4d00 2024-09-16T01:44:52.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.933+0000 7f8043bc4640 1 -- 172.21.15.88:0/1964414422 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f803c14f350 con 0x7f803c0b4d00 2024-09-16T01:44:52.938 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.933+0000 7f80327fc640 1 -- 172.21.15.88:0/1964414422 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802c037040 con 0x7f803c0b4d00 2024-09-16T01:44:52.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.933+0000 7f8043bc4640 1 -- 172.21.15.88:0/1964414422 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f803c14c720 con 0x7f803c0b4d00 2024-09-16T01:44:52.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.934+0000 7f8043bc4640 1 -- 172.21.15.88:0/1964414422 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f803c15b7d0 con 0x7f803c0b4d00 2024-09-16T01:44:52.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.935+0000 7f80327fc640 1 -- 172.21.15.88:0/1964414422 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f802c044070 con 0x7f803c0b4d00 2024-09-16T01:44:52.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.935+0000 7f80327fc640 1 --2- 172.21.15.88:0/1964414422 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f802006c2b0 0x7f802006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:52.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.935+0000 7f8041960640 1 --2- 172.21.15.88:0/1964414422 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f802006c2b0 0x7f802006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:52.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.935+0000 7f80327fc640 1 -- 172.21.15.88:0/1964414422 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 5709+0+0 (secure 0 0 0) 0x7f802c0abe70 con 0x7f803c0b4d00 2024-09-16T01:44:52.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.936+0000 7f8041960640 1 --2- 172.21.15.88:0/1964414422 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f802006c2b0 0x7f802006e770 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f8038009990 tx=0x7f8038009110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:52.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:52.940+0000 7f80327fc640 1 -- 172.21.15.88:0/1964414422 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f802c07a790 con 0x7f803c0b4d00 2024-09-16T01:44:53.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.107+0000 7f8043bc4640 1 -- 172.21.15.88:0/1964414422 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f803c14f520 con 0x7f803c0b4d00 2024-09-16T01:44:53.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.108+0000 7f80327fc640 1 -- 172.21.15.88:0/1964414422 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v137) ==== 74+0+16668 (secure 0 0 0) 0x7f802c07a090 con 0x7f803c0b4d00 2024-09-16T01:44:53.109 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:53.109 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":137,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:44:51.844752+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:44:50.167367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-09-16T01:44:50.816821+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:44:53.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.110+0000 7f8043bc4640 1 -- 172.21.15.88:0/1964414422 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f802006c2b0 msgr2=0x7f802006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:53.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.110+0000 7f8043bc4640 1 --2- 172.21.15.88:0/1964414422 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f802006c2b0 0x7f802006e770 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f8038009990 tx=0x7f8038009110 comp rx=0 tx=0).stop 2024-09-16T01:44:53.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.110+0000 7f8043bc4640 1 -- 172.21.15.88:0/1964414422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f803c0b4d00 msgr2=0x7f803c14a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:53.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.110+0000 7f8043bc4640 1 --2- 172.21.15.88:0/1964414422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f803c0b4d00 0x7f803c14a1b0 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7f802c000c00 tx=0x7f802c0046c0 comp rx=0 tx=0).stop 2024-09-16T01:44:53.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.110+0000 7f8043bc4640 1 -- 172.21.15.88:0/1964414422 shutdown_connections 2024-09-16T01:44:53.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.110+0000 7f8043bc4640 1 --2- 172.21.15.88:0/1964414422 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f802006c2b0 0x7f802006e770 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:53.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.110+0000 7f8043bc4640 1 --2- 172.21.15.88:0/1964414422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f803c0cd780 0x7f803c14a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:53.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.110+0000 7f8043bc4640 1 --2- 172.21.15.88:0/1964414422 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f803c0b4d00 0x7f803c14a1b0 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:53.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.110+0000 7f8043bc4640 1 --2- 172.21.15.88:0/1964414422 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f803c0b4920 0x7f803c144540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:53.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.110+0000 7f8043bc4640 1 -- 172.21.15.88:0/1964414422 >> 172.21.15.88:0/1964414422 conn(0x7f803c06bc10 msgr2=0x7f803c072920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:53.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.110+0000 7f8043bc4640 1 -- 172.21.15.88:0/1964414422 shutdown_connections 2024-09-16T01:44:53.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.110+0000 7f8043bc4640 1 -- 172.21.15.88:0/1964414422 wait complete. 2024-09-16T01:44:53.123 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-16T01:44:53.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.223+0000 7f7dce5f8640 1 Processor -- start 2024-09-16T01:44:53.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.223+0000 7f7dce5f8640 1 -- start start 2024-09-16T01:44:53.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.224+0000 7f7dce5f8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7dc815b480 0x7f7dc80b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:53.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.224+0000 7f7dce5f8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7dc80ccff0 con 0x7f7dc8072700 2024-09-16T01:44:53.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.224+0000 7f7dce5f8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7dc80cd2d0 con 0x7f7dc80b60c0 2024-09-16T01:44:53.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.224+0000 7f7dce5f8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7dc80cd5b0 con 0x7f7dc815b480 2024-09-16T01:44:53.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.224+0000 7f7dc7fff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f7dc815b480 0x7f7dc80b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:53.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.224+0000 7f7dccb95640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7dc8072700 0x7f7dc815bc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35030/0 (socket says 172.21.15.88:35030) 2024-09-16T01:44:53.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.224+0000 7f7dccb95640 1 -- 172.21.15.88:0/417109651 learned_addr learned my addr 172.21.15.88:0/417109651 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:53.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.224+0000 7f7dc7fff640 1 -- 172.21.15.88:0/417109651 >> v1:172.21.15.154:6789/0 conn(0x7f7dc80b60c0 legacy=0x7f7dc8071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:53.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.224+0000 7f7dc7fff640 1 -- 172.21.15.88:0/417109651 >> v1:172.21.15.88:6789/0 conn(0x7f7dc8072700 legacy=0x7f7dc815bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:53.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.224+0000 7f7dc7fff640 1 -- 172.21.15.88:0/417109651 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dc80cd890 con 0x7f7dc815b480 2024-09-16T01:44:53.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.225+0000 7f7dc7fff640 1 --2- 172.21.15.88:0/417109651 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7dc815b480 0x7f7dc80b5b80 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f7db80097a0 tx=0x7f7db802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=552c0ffc2aebf0ce server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:53.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.225+0000 7f7dc6ffd640 1 -- 172.21.15.88:0/417109651 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db8037070 con 0x7f7dc815b480 2024-09-16T01:44:53.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.225+0000 7f7dc6ffd640 1 -- 172.21.15.88:0/417109651 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7db802ae50 con 0x7f7dc815b480 2024-09-16T01:44:53.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.225+0000 7f7dc6ffd640 1 -- 172.21.15.88:0/417109651 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db8032660 con 0x7f7dc815b480 2024-09-16T01:44:53.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.226+0000 7f7dce5f8640 1 -- 172.21.15.88:0/417109651 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7dc815b480 msgr2=0x7f7dc80b5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:53.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.226+0000 7f7dce5f8640 1 --2- 172.21.15.88:0/417109651 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7dc815b480 0x7f7dc80b5b80 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f7db80097a0 tx=0x7f7db802a140 comp rx=0 tx=0).stop 2024-09-16T01:44:53.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.226+0000 7f7dce5f8640 1 -- 172.21.15.88:0/417109651 shutdown_connections 2024-09-16T01:44:53.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.226+0000 7f7dce5f8640 1 --2- 172.21.15.88:0/417109651 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7dc815b480 0x7f7dc80b5b80 unknown :-1 s=CLOSED pgs=2343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:53.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.226+0000 7f7dce5f8640 1 -- 172.21.15.88:0/417109651 >> 172.21.15.88:0/417109651 conn(0x7f7dc806d0f0 msgr2=0x7f7dc806d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:53.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.226+0000 7f7dce5f8640 1 -- 172.21.15.88:0/417109651 shutdown_connections 2024-09-16T01:44:53.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.226+0000 7f7dce5f8640 1 -- 172.21.15.88:0/417109651 wait complete. 2024-09-16T01:44:53.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.227+0000 7f7dce5f8640 1 Processor -- start 2024-09-16T01:44:53.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.227+0000 7f7dce5f8640 1 -- start start 2024-09-16T01:44:53.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.228+0000 7f7dce5f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7dc8072700 0x7f7dc8145f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:53.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.228+0000 7f7dce5f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7dc80b60c0 0x7f7dc8155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:53.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.228+0000 7f7dce5f8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7dc815b480 0x7f7dc80b3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:53.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.228+0000 7f7dce5f8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc815b9e0 con 0x7f7dc815b480 2024-09-16T01:44:53.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.228+0000 7f7dce5f8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc815b860 con 0x7f7dc8072700 2024-09-16T01:44:53.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.228+0000 7f7dce5f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7dc815c370 con 0x7f7dc80b60c0 2024-09-16T01:44:53.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.228+0000 7f7dccb95640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7dc815b480 0x7f7dc80b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:53.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.228+0000 7f7dccb95640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7dc815b480 0x7f7dc80b3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35778/0 (socket says 172.21.15.88:35778) 2024-09-16T01:44:53.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.228+0000 7f7dccb95640 1 -- 172.21.15.88:0/1553279697 learned_addr learned my addr 172.21.15.88:0/1553279697 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:53.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.228+0000 7f7dc7fff640 1 --2- 172.21.15.88:0/1553279697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7dc8072700 0x7f7dc8145f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:53.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.228+0000 7f7dc77fe640 1 --2- 172.21.15.88:0/1553279697 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7dc80b60c0 0x7f7dc8155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:53.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.228+0000 7f7dccb95640 1 -- 172.21.15.88:0/1553279697 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7dc80b60c0 msgr2=0x7f7dc8155f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:53.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.228+0000 7f7dccb95640 1 --2- 172.21.15.88:0/1553279697 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7dc80b60c0 0x7f7dc8155f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:53.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.228+0000 7f7dccb95640 1 -- 172.21.15.88:0/1553279697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7dc8072700 msgr2=0x7f7dc8145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:53.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.228+0000 7f7dccb95640 1 --2- 172.21.15.88:0/1553279697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7dc8072700 0x7f7dc8145f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:53.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.228+0000 7f7dccb95640 1 -- 172.21.15.88:0/1553279697 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dc8156f60 con 0x7f7dc815b480 2024-09-16T01:44:53.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.229+0000 7f7dccb95640 1 --2- 172.21.15.88:0/1553279697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7dc815b480 0x7f7dc80b3590 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f7dc000cbf0 tx=0x7f7dc000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:53.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.229+0000 7f7dc4ff9640 1 -- 172.21.15.88:0/1553279697 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc0017070 con 0x7f7dc815b480 2024-09-16T01:44:53.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.229+0000 7f7dc4ff9640 1 -- 172.21.15.88:0/1553279697 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dc000ad40 con 0x7f7dc815b480 2024-09-16T01:44:53.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.229+0000 7f7dce5f8640 1 -- 172.21.15.88:0/1553279697 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dc81569a0 con 0x7f7dc815b480 2024-09-16T01:44:53.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.229+0000 7f7dc4ff9640 1 -- 172.21.15.88:0/1553279697 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc0012600 con 0x7f7dc815b480 2024-09-16T01:44:53.232 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.229+0000 7f7dce5f8640 1 -- 172.21.15.88:0/1553279697 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dc8146b00 con 0x7f7dc815b480 2024-09-16T01:44:53.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.230+0000 7f7dce5f8640 1 -- 172.21.15.88:0/1553279697 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dc80b5b80 con 0x7f7dc815b480 2024-09-16T01:44:53.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.231+0000 7f7dc4ff9640 1 -- 172.21.15.88:0/1553279697 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7dc001a070 con 0x7f7dc815b480 2024-09-16T01:44:53.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.231+0000 7f7dc4ff9640 1 --2- 172.21.15.88:0/1553279697 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7db406c2e0 0x7f7db406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:53.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.231+0000 7f7dc4ff9640 1 -- 172.21.15.88:0/1553279697 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 5709+0+0 (secure 0 0 0) 0x7f7dc008b150 con 0x7f7dc815b480 2024-09-16T01:44:53.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.231+0000 7f7dc7fff640 1 --2- 172.21.15.88:0/1553279697 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7db406c2e0 0x7f7db406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:53.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.231+0000 7f7dc7fff640 1 --2- 172.21.15.88:0/1553279697 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7db406c2e0 0x7f7db406e7a0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f7db8034000 tx=0x7f7db802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:53.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.235+0000 7f7dc4ff9640 1 -- 172.21.15.88:0/1553279697 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f7dc00591d0 con 0x7f7dc815b480 2024-09-16T01:44:53.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:53.436+0000 7f7dce5f8640 1 -- 172.21.15.88:0/1553279697 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f7dc81464c0 con 0x7f7dc815b480 2024-09-16T01:44:53.497 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:44:53.496+0000 7f832c8b0640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:44:54.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.291+0000 7f7dc4ff9640 1 -- 172.21.15.88:0/1553279697 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v325) ==== 118+0+0 (secure 0 0 0) 0x7f7dc005d4d0 con 0x7f7dc815b480 2024-09-16T01:44:54.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.294+0000 7f7dce5f8640 1 -- 172.21.15.88:0/1553279697 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7db406c2e0 msgr2=0x7f7db406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:54.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.294+0000 7f7dce5f8640 1 --2- 172.21.15.88:0/1553279697 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7db406c2e0 0x7f7db406e7a0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f7db8034000 tx=0x7f7db802e000 comp rx=0 tx=0).stop 2024-09-16T01:44:54.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.294+0000 7f7dce5f8640 1 -- 172.21.15.88:0/1553279697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7dc815b480 msgr2=0x7f7dc80b3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:54.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.294+0000 7f7dce5f8640 1 --2- 172.21.15.88:0/1553279697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7dc815b480 0x7f7dc80b3590 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f7dc000cbf0 tx=0x7f7dc000a430 comp rx=0 tx=0).stop 2024-09-16T01:44:54.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.294+0000 7f7dce5f8640 1 -- 172.21.15.88:0/1553279697 shutdown_connections 2024-09-16T01:44:54.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.294+0000 7f7dce5f8640 1 --2- 172.21.15.88:0/1553279697 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f7db406c2e0 0x7f7db406e7a0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:54.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.294+0000 7f7dce5f8640 1 --2- 172.21.15.88:0/1553279697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7dc815b480 0x7f7dc80b3590 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:54.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.294+0000 7f7dce5f8640 1 --2- 172.21.15.88:0/1553279697 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7dc80b60c0 0x7f7dc8155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:54.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.294+0000 7f7dce5f8640 1 --2- 172.21.15.88:0/1553279697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7dc8072700 0x7f7dc8145f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:54.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.294+0000 7f7dce5f8640 1 -- 172.21.15.88:0/1553279697 >> 172.21.15.88:0/1553279697 conn(0x7f7dc806d0f0 msgr2=0x7f7dc8074c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:54.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.294+0000 7f7dce5f8640 1 -- 172.21.15.88:0/1553279697 shutdown_connections 2024-09-16T01:44:54.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.294+0000 7f7dce5f8640 1 -- 172.21.15.88:0/1553279697 wait complete. 2024-09-16T01:44:54.309 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-16T01:44:54.309 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-16T01:44:54.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.415+0000 7f1ec408c640 1 Processor -- start 2024-09-16T01:44:54.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.415+0000 7f1ec408c640 1 -- start start 2024-09-16T01:44:54.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.416+0000 7f1ec408c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1ebc0b4e60 0x7f1ebc0d0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:54.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.416+0000 7f1ec408c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1ebc15cb20 0x7f1ebc15cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:54.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.416+0000 7f1ec408c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ebc0cd9b0 con 0x7f1ebc0b5c00 2024-09-16T01:44:54.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.416+0000 7f1ec408c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f1ebc0ce5c0 con 0x7f1ebc15cb20 2024-09-16T01:44:54.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.416+0000 7f1ec408c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f1ebc0cdc90 con 0x7f1ebc0b4e60 2024-09-16T01:44:54.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.416+0000 7f1ec2629640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1ebc15cb20 0x7f1ebc15cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:54.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.416+0000 7f1ec2629640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f1ebc15cb20 0x7f1ebc15cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35784/0 (socket says 172.21.15.88:35784) 2024-09-16T01:44:54.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.416+0000 7f1ec1627640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1ebc0b5c00 0x7f1ebc0b5240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35038/0 (socket says 172.21.15.88:35038) 2024-09-16T01:44:54.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.416+0000 7f1ec2629640 1 -- 172.21.15.88:0/1868712545 learned_addr learned my addr 172.21.15.88:0/1868712545 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:54.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.416+0000 7f1ec1e28640 1 --2- 172.21.15.88:0/1868712545 >> v2:172.21.15.154:3300/0 conn(0x7f1ebc0b4e60 0x7f1ebc0d0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:54.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.416+0000 7f1ec2629640 1 -- 172.21.15.88:0/1868712545 >> v2:172.21.15.154:3300/0 conn(0x7f1ebc0b4e60 msgr2=0x7f1ebc0d0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:54.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.416+0000 7f1ec2629640 1 --2- 172.21.15.88:0/1868712545 >> v2:172.21.15.154:3300/0 conn(0x7f1ebc0b4e60 0x7f1ebc0d0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:54.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.416+0000 7f1ec2629640 1 -- 172.21.15.88:0/1868712545 >> v1:172.21.15.88:6789/0 conn(0x7f1ebc0b5c00 legacy=0x7f1ebc0b5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:54.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.416+0000 7f1ec0e26640 1 -- 172.21.15.88:0/1868712545 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4292700741 0 0) 0x7f1ebc0cd9b0 con 0x7f1ebc0b5c00 2024-09-16T01:44:54.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.416+0000 7f1ec2629640 1 -- 172.21.15.88:0/1868712545 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ebc0cdf70 con 0x7f1ebc15cb20 2024-09-16T01:44:54.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.417+0000 7f1ec2629640 1 --2- 172.21.15.88:0/1868712545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ebc15cb20 0x7f1ebc15cf50 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc063310 tx=0x7f1eb402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=820614cf326a41c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:54.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.417+0000 7f1ec0e26640 1 -- 172.21.15.88:0/1868712545 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb4036070 con 0x7f1ebc15cb20 2024-09-16T01:44:54.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.417+0000 7f1ec0e26640 1 -- 172.21.15.88:0/1868712545 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1eb4031600 con 0x7f1ebc15cb20 2024-09-16T01:44:54.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.417+0000 7f1ec0e26640 1 -- 172.21.15.88:0/1868712545 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb40319d0 con 0x7f1ebc15cb20 2024-09-16T01:44:54.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.418+0000 7f1ec408c640 1 -- 172.21.15.88:0/1868712545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ebc15cb20 msgr2=0x7f1ebc15cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:54.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.418+0000 7f1ec408c640 1 --2- 172.21.15.88:0/1868712545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ebc15cb20 0x7f1ebc15cf50 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc063310 tx=0x7f1eb402a730 comp rx=0 tx=0).stop 2024-09-16T01:44:54.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.418+0000 7f1ec408c640 1 -- 172.21.15.88:0/1868712545 shutdown_connections 2024-09-16T01:44:54.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.418+0000 7f1ec408c640 1 --2- 172.21.15.88:0/1868712545 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ebc15cb20 0x7f1ebc15cf50 unknown :-1 s=CLOSED pgs=2346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:54.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.418+0000 7f1ec408c640 1 --2- 172.21.15.88:0/1868712545 >> v2:172.21.15.154:3300/0 conn(0x7f1ebc0b4e60 0x7f1ebc0d0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:54.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.418+0000 7f1ec408c640 1 -- 172.21.15.88:0/1868712545 >> 172.21.15.88:0/1868712545 conn(0x7f1ebc06cb50 msgr2=0x7f1ebc06cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:54.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.418+0000 7f1ec408c640 1 -- 172.21.15.88:0/1868712545 shutdown_connections 2024-09-16T01:44:54.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.418+0000 7f1ec408c640 1 -- 172.21.15.88:0/1868712545 wait complete. 2024-09-16T01:44:54.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.419+0000 7f1ec408c640 1 Processor -- start 2024-09-16T01:44:54.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.419+0000 7f1ec408c640 1 -- start start 2024-09-16T01:44:54.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.420+0000 7f1ec408c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ebc0b4e60 0x7f1ebc071da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:54.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.420+0000 7f1ec408c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1ebc0b5c00 0x7f1ebc147660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:54.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.420+0000 7f1ec408c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ebc15cb20 0x7f1ebc155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:54.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.420+0000 7f1ec408c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1ebc0b6210 con 0x7f1ebc15cb20 2024-09-16T01:44:54.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.420+0000 7f1ec408c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1ebc0b6090 con 0x7f1ebc0b4e60 2024-09-16T01:44:54.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.420+0000 7f1ec408c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1ebc15fc00 con 0x7f1ebc0b5c00 2024-09-16T01:44:54.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.420+0000 7f1ec2629640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ebc15cb20 0x7f1ebc155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:54.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.420+0000 7f1ec2629640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ebc15cb20 0x7f1ebc155df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35790/0 (socket says 172.21.15.88:35790) 2024-09-16T01:44:54.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.420+0000 7f1ec2629640 1 -- 172.21.15.88:0/2883999653 learned_addr learned my addr 172.21.15.88:0/2883999653 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:54.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.420+0000 7f1ec1e28640 1 --2- 172.21.15.88:0/2883999653 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ebc0b4e60 0x7f1ebc071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:54.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.420+0000 7f1ec1627640 1 --2- 172.21.15.88:0/2883999653 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1ebc0b5c00 0x7f1ebc147660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:54.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.420+0000 7f1ec2629640 1 -- 172.21.15.88:0/2883999653 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1ebc0b5c00 msgr2=0x7f1ebc147660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:54.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.420+0000 7f1ec2629640 1 --2- 172.21.15.88:0/2883999653 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1ebc0b5c00 0x7f1ebc147660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:54.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.420+0000 7f1ec2629640 1 -- 172.21.15.88:0/2883999653 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ebc0b4e60 msgr2=0x7f1ebc071da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:54.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.420+0000 7f1ec2629640 1 --2- 172.21.15.88:0/2883999653 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ebc0b4e60 0x7f1ebc071da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:54.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.420+0000 7f1ec2629640 1 -- 172.21.15.88:0/2883999653 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ebc072380 con 0x7f1ebc15cb20 2024-09-16T01:44:54.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.420+0000 7f1ec1627640 1 --2- 172.21.15.88:0/2883999653 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1ebc0b5c00 0x7f1ebc147660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:54.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.420+0000 7f1ec2629640 1 --2- 172.21.15.88:0/2883999653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ebc15cb20 0x7f1ebc155df0 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4009f40 tx=0x7f1eb4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:54.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.421+0000 7f1eb27fc640 1 -- 172.21.15.88:0/2883999653 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb4036050 con 0x7f1ebc15cb20 2024-09-16T01:44:54.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.421+0000 7f1eb27fc640 1 -- 172.21.15.88:0/2883999653 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1eb4042070 con 0x7f1ebc15cb20 2024-09-16T01:44:54.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.421+0000 7f1ec408c640 1 -- 172.21.15.88:0/2883999653 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ebc072660 con 0x7f1ebc15cb20 2024-09-16T01:44:54.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.421+0000 7f1eb27fc640 1 -- 172.21.15.88:0/2883999653 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb4009510 con 0x7f1ebc15cb20 2024-09-16T01:44:54.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.421+0000 7f1ec408c640 1 -- 172.21.15.88:0/2883999653 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ebc072c20 con 0x7f1ebc15cb20 2024-09-16T01:44:54.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.422+0000 7f1ec408c640 1 -- 172.21.15.88:0/2883999653 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ebc0d0760 con 0x7f1ebc15cb20 2024-09-16T01:44:54.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.422+0000 7f1eb27fc640 1 -- 172.21.15.88:0/2883999653 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1eb4007bf0 con 0x7f1ebc15cb20 2024-09-16T01:44:54.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.423+0000 7f1eb27fc640 1 --2- 172.21.15.88:0/2883999653 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ea006c2b0 0x7f1ea006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:54.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.423+0000 7f1ec1e28640 1 --2- 172.21.15.88:0/2883999653 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ea006c2b0 0x7f1ea006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:54.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.423+0000 7f1eb27fc640 1 -- 172.21.15.88:0/2883999653 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5709+0+0 (secure 0 0 0) 0x7f1eb40ac280 con 0x7f1ebc15cb20 2024-09-16T01:44:54.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.423+0000 7f1ec1e28640 1 --2- 172.21.15.88:0/2883999653 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ea006c2b0 0x7f1ea006e770 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8006fd0 tx=0x7f1eb8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:54.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.427+0000 7f1eb27fc640 1 -- 172.21.15.88:0/2883999653 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1eb4045360 con 0x7f1ebc15cb20 2024-09-16T01:44:54.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:54.628+0000 7f1ec408c640 1 -- 172.21.15.88:0/2883999653 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f1ebc075620 con 0x7f1ebc15cb20 2024-09-16T01:44:55.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.538+0000 7f1eb27fc640 1 -- 172.21.15.88:0/2883999653 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v326) ==== 122+0+0 (secure 0 0 0) 0x7f1eb407a9f0 con 0x7f1ebc15cb20 2024-09-16T01:44:55.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.541+0000 7f1ec408c640 1 -- 172.21.15.88:0/2883999653 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ea006c2b0 msgr2=0x7f1ea006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:55.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.541+0000 7f1ec408c640 1 --2- 172.21.15.88:0/2883999653 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ea006c2b0 0x7f1ea006e770 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8006fd0 tx=0x7f1eb8008040 comp rx=0 tx=0).stop 2024-09-16T01:44:55.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.541+0000 7f1ec408c640 1 -- 172.21.15.88:0/2883999653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ebc15cb20 msgr2=0x7f1ebc155df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:55.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.541+0000 7f1ec408c640 1 --2- 172.21.15.88:0/2883999653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ebc15cb20 0x7f1ebc155df0 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4009f40 tx=0x7f1eb4004a40 comp rx=0 tx=0).stop 2024-09-16T01:44:55.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.541+0000 7f1ec408c640 1 -- 172.21.15.88:0/2883999653 shutdown_connections 2024-09-16T01:44:55.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.541+0000 7f1ec408c640 1 --2- 172.21.15.88:0/2883999653 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1ea006c2b0 0x7f1ea006e770 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:55.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.541+0000 7f1ec408c640 1 --2- 172.21.15.88:0/2883999653 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ebc15cb20 0x7f1ebc155df0 unknown :-1 s=CLOSED pgs=2347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:55.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.541+0000 7f1ec408c640 1 --2- 172.21.15.88:0/2883999653 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1ebc0b5c00 0x7f1ebc147660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:55.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.541+0000 7f1ec408c640 1 --2- 172.21.15.88:0/2883999653 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ebc0b4e60 0x7f1ebc071da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:55.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.541+0000 7f1ec408c640 1 -- 172.21.15.88:0/2883999653 >> 172.21.15.88:0/2883999653 conn(0x7f1ebc06cb50 msgr2=0x7f1ebc06cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:55.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.541+0000 7f1ec408c640 1 -- 172.21.15.88:0/2883999653 shutdown_connections 2024-09-16T01:44:55.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.541+0000 7f1ec408c640 1 -- 172.21.15.88:0/2883999653 wait complete. 2024-09-16T01:44:55.556 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-16T01:44:55.557 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:44:55.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.661+0000 7ffb14227640 1 Processor -- start 2024-09-16T01:44:55.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.661+0000 7ffb14227640 1 -- start start 2024-09-16T01:44:55.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.662+0000 7ffb14227640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ffb0c06e780 0x7ffb0c0ae890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:55.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.662+0000 7ffb14227640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ffb0c073130 0x7ffb0c073580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:55.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.662+0000 7ffb14227640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb0c0b5220 con 0x7ffb0c15e560 2024-09-16T01:44:55.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.662+0000 7ffb14227640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ffb0c06ebc0 con 0x7ffb0c073130 2024-09-16T01:44:55.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.662+0000 7ffb14227640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ffb0c073ac0 con 0x7ffb0c06e780 2024-09-16T01:44:55.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.662+0000 7ffb127c4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ffb0c073130 0x7ffb0c073580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:55.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.662+0000 7ffb11fc3640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ffb0c15e560 0x7ffb0c072220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35042/0 (socket says 172.21.15.88:35042) 2024-09-16T01:44:55.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.662+0000 7ffb11fc3640 1 -- 172.21.15.88:0/2265377247 learned_addr learned my addr 172.21.15.88:0/2265377247 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:44:55.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.663+0000 7ffb127c4640 1 -- 172.21.15.88:0/2265377247 >> v2:172.21.15.154:3300/0 conn(0x7ffb0c06e780 msgr2=0x7ffb0c0ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:55.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.663+0000 7ffb127c4640 1 --2- 172.21.15.88:0/2265377247 >> v2:172.21.15.154:3300/0 conn(0x7ffb0c06e780 0x7ffb0c0ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:55.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.663+0000 7ffb127c4640 1 -- 172.21.15.88:0/2265377247 >> v1:172.21.15.88:6789/0 conn(0x7ffb0c15e560 legacy=0x7ffb0c072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:55.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.663+0000 7ffb127c4640 1 -- 172.21.15.88:0/2265377247 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb0c0b54c0 con 0x7ffb0c073130 2024-09-16T01:44:55.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.663+0000 7ffb10fc1640 1 -- 172.21.15.88:0/2265377247 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1631053566 0 0) 0x7ffb0c0b5220 con 0x7ffb0c15e560 2024-09-16T01:44:55.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.663+0000 7ffb127c4640 1 --2- 172.21.15.88:0/2265377247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb0c073130 0x7ffb0c073580 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c0b4cb0 tx=0x7ffb0402a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85eaa397c78c2533 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:55.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.663+0000 7ffb10fc1640 1 -- 172.21.15.88:0/2265377247 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb04007740 con 0x7ffb0c073130 2024-09-16T01:44:55.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.663+0000 7ffb10fc1640 1 -- 172.21.15.88:0/2265377247 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb040078e0 con 0x7ffb0c073130 2024-09-16T01:44:55.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.663+0000 7ffb10fc1640 1 -- 172.21.15.88:0/2265377247 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb04007cb0 con 0x7ffb0c073130 2024-09-16T01:44:55.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.664+0000 7ffb14227640 1 -- 172.21.15.88:0/2265377247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb0c073130 msgr2=0x7ffb0c073580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:55.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.664+0000 7ffb14227640 1 --2- 172.21.15.88:0/2265377247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb0c073130 0x7ffb0c073580 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c0b4cb0 tx=0x7ffb0402a5f0 comp rx=0 tx=0).stop 2024-09-16T01:44:55.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.664+0000 7ffb14227640 1 -- 172.21.15.88:0/2265377247 shutdown_connections 2024-09-16T01:44:55.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.664+0000 7ffb14227640 1 --2- 172.21.15.88:0/2265377247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb0c073130 0x7ffb0c073580 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:55.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.664+0000 7ffb14227640 1 --2- 172.21.15.88:0/2265377247 >> v2:172.21.15.154:3300/0 conn(0x7ffb0c06e780 0x7ffb0c0ae890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:55.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.664+0000 7ffb14227640 1 -- 172.21.15.88:0/2265377247 >> 172.21.15.88:0/2265377247 conn(0x7ffb0c06bc10 msgr2=0x7ffb0c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:55.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.664+0000 7ffb14227640 1 -- 172.21.15.88:0/2265377247 shutdown_connections 2024-09-16T01:44:55.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.664+0000 7ffb14227640 1 -- 172.21.15.88:0/2265377247 wait complete. 2024-09-16T01:44:55.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.665+0000 7ffb14227640 1 Processor -- start 2024-09-16T01:44:55.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.665+0000 7ffb14227640 1 -- start start 2024-09-16T01:44:55.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.666+0000 7ffb14227640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffb0c06e780 0x7ffb0c157ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:55.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.666+0000 7ffb14227640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb0c073130 0x7ffb0c152610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:55.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.666+0000 7ffb14227640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb0c15e560 0x7ffb0c1560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:55.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.666+0000 7ffb14227640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0c0b5d60 con 0x7ffb0c15e560 2024-09-16T01:44:55.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.666+0000 7ffb14227640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0c0b5a80 con 0x7ffb0c073130 2024-09-16T01:44:55.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.666+0000 7ffb14227640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ffb0c15e940 con 0x7ffb0c06e780 2024-09-16T01:44:55.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.666+0000 7ffb127c4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb0c15e560 0x7ffb0c1560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:55.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.666+0000 7ffb127c4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb0c15e560 0x7ffb0c1560c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35808/0 (socket says 172.21.15.88:35808) 2024-09-16T01:44:55.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.666+0000 7ffb127c4640 1 -- 172.21.15.88:0/742849824 learned_addr learned my addr 172.21.15.88:0/742849824 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:55.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.666+0000 7ffb11fc3640 1 --2- 172.21.15.88:0/742849824 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffb0c06e780 0x7ffb0c157ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:55.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.666+0000 7ffb117c2640 1 --2- 172.21.15.88:0/742849824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb0c073130 0x7ffb0c152610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:55.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.666+0000 7ffb127c4640 1 -- 172.21.15.88:0/742849824 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffb0c06e780 msgr2=0x7ffb0c157ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:55.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.666+0000 7ffb127c4640 1 --2- 172.21.15.88:0/742849824 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffb0c06e780 0x7ffb0c157ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:55.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.667+0000 7ffb127c4640 1 -- 172.21.15.88:0/742849824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb0c073130 msgr2=0x7ffb0c152610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:55.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.667+0000 7ffb127c4640 1 --2- 172.21.15.88:0/742849824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb0c073130 0x7ffb0c152610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:55.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.667+0000 7ffb127c4640 1 -- 172.21.15.88:0/742849824 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb0c0cf1c0 con 0x7ffb0c15e560 2024-09-16T01:44:55.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.667+0000 7ffb117c2640 1 --2- 172.21.15.88:0/742849824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb0c073130 0x7ffb0c152610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:55.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.667+0000 7ffb127c4640 1 --2- 172.21.15.88:0/742849824 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb0c15e560 0x7ffb0c1560c0 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7ffb04009e80 tx=0x7ffb04004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:55.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.667+0000 7ffb027fc640 1 -- 172.21.15.88:0/742849824 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb04040070 con 0x7ffb0c15e560 2024-09-16T01:44:55.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.667+0000 7ffb027fc640 1 -- 172.21.15.88:0/742849824 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb04031c40 con 0x7ffb0c15e560 2024-09-16T01:44:55.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.667+0000 7ffb14227640 1 -- 172.21.15.88:0/742849824 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb0c1581d0 con 0x7ffb0c15e560 2024-09-16T01:44:55.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.667+0000 7ffb027fc640 1 -- 172.21.15.88:0/742849824 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb04003990 con 0x7ffb0c15e560 2024-09-16T01:44:55.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.667+0000 7ffb14227640 1 -- 172.21.15.88:0/742849824 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb0c0cf760 con 0x7ffb0c15e560 2024-09-16T01:44:55.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.668+0000 7ffb14227640 1 -- 172.21.15.88:0/742849824 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb0c073580 con 0x7ffb0c15e560 2024-09-16T01:44:55.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.668+0000 7ffb027fc640 1 -- 172.21.15.88:0/742849824 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ffb04039040 con 0x7ffb0c15e560 2024-09-16T01:44:55.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.669+0000 7ffb027fc640 1 --2- 172.21.15.88:0/742849824 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ffaf006c2b0 0x7ffaf006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:55.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.669+0000 7ffb027fc640 1 -- 172.21.15.88:0/742849824 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5709+0+0 (secure 0 0 0) 0x7ffb040ac410 con 0x7ffb0c15e560 2024-09-16T01:44:55.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.669+0000 7ffb11fc3640 1 --2- 172.21.15.88:0/742849824 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ffaf006c2b0 0x7ffaf006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:55.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.669+0000 7ffb11fc3640 1 --2- 172.21.15.88:0/742849824 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ffaf006c2b0 0x7ffaf006e770 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7ffb08000f60 tx=0x7ffb08007c00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:55.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.673+0000 7ffb027fc640 1 -- 172.21.15.88:0/742849824 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ffb0407ac10 con 0x7ffb0c15e560 2024-09-16T01:44:55.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.872+0000 7ffb14227640 1 -- 172.21.15.88:0/742849824 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb0c156600 con 0x7ffb0c15e560 2024-09-16T01:44:55.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.873+0000 7ffb027fc640 1 -- 172.21.15.88:0/742849824 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 326 v326) ==== 95+0+5239 (secure 0 0 0) 0x7ffb0407a510 con 0x7ffb0c15e560 2024-09-16T01:44:55.875 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:55.875 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":326,"btime":"2024-09-16T01:44:55:534079+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7649,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2706550508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2706550508},{"type":"v1","addr":"172.21.15.154:6834","nonce":2706550508}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322},{"gid":8569,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/3460839382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":3460839382},{"type":"v1","addr":"172.21.15.88:6837","nonce":3460839382}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322},{"gid":8581,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2822970755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2822970755},{"type":"v1","addr":"172.21.15.154:6835","nonce":2822970755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322}],"filesystems":[{"mdsmap":{"epoch":326,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:44:51.845091+0000","modified":"2024-09-16T01:44:54.630127+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10536},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10536":{"gid":10536,"name":"a","rank":0,"incarnation":323,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6836/656828696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":656828696},{"type":"v1","addr":"172.21.15.88:6836","nonce":656828696}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10536,"qdb_cluster":[10536]},"id":7}]} 2024-09-16T01:44:55.875 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 326 2024-09-16T01:44:55.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.875+0000 7ffb14227640 1 -- 172.21.15.88:0/742849824 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ffaf006c2b0 msgr2=0x7ffaf006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:55.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.875+0000 7ffb14227640 1 --2- 172.21.15.88:0/742849824 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ffaf006c2b0 0x7ffaf006e770 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7ffb08000f60 tx=0x7ffb08007c00 comp rx=0 tx=0).stop 2024-09-16T01:44:55.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.875+0000 7ffb14227640 1 -- 172.21.15.88:0/742849824 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb0c15e560 msgr2=0x7ffb0c1560c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:55.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.875+0000 7ffb14227640 1 --2- 172.21.15.88:0/742849824 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb0c15e560 0x7ffb0c1560c0 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7ffb04009e80 tx=0x7ffb04004a40 comp rx=0 tx=0).stop 2024-09-16T01:44:55.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.875+0000 7ffb14227640 1 -- 172.21.15.88:0/742849824 shutdown_connections 2024-09-16T01:44:55.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.875+0000 7ffb14227640 1 --2- 172.21.15.88:0/742849824 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ffaf006c2b0 0x7ffaf006e770 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:55.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.875+0000 7ffb14227640 1 --2- 172.21.15.88:0/742849824 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ffb0c15e560 0x7ffb0c1560c0 unknown :-1 s=CLOSED pgs=2350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:55.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.875+0000 7ffb14227640 1 --2- 172.21.15.88:0/742849824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb0c073130 0x7ffb0c152610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:55.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.875+0000 7ffb14227640 1 --2- 172.21.15.88:0/742849824 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffb0c06e780 0x7ffb0c157ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:55.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.875+0000 7ffb14227640 1 -- 172.21.15.88:0/742849824 >> 172.21.15.88:0/742849824 conn(0x7ffb0c06bc10 msgr2=0x7ffb0c0b6280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:55.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.875+0000 7ffb14227640 1 -- 172.21.15.88:0/742849824 shutdown_connections 2024-09-16T01:44:55.878 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.875+0000 7ffb14227640 1 -- 172.21.15.88:0/742849824 wait complete. 2024-09-16T01:44:55.887 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:44:55.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.984+0000 7fa04c25e640 1 Processor -- start 2024-09-16T01:44:55.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.984+0000 7fa04c25e640 1 -- start start 2024-09-16T01:44:55.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.985+0000 7fa04c25e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa04406bc10 0x7fa04406bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:55.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.985+0000 7fa04c25e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa04415c710 0x7fa044072050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:55.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.985+0000 7fa04c25e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa04406c530 con 0x7fa044072590 2024-09-16T01:44:55.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.985+0000 7fa04c25e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fa044072a30 con 0x7fa04406bc10 2024-09-16T01:44:55.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.985+0000 7fa04c25e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa044072d10 con 0x7fa04415c710 2024-09-16T01:44:55.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.985+0000 7fa049ffa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa04406bc10 0x7fa04406bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:55.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.985+0000 7fa049ffa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fa04406bc10 0x7fa04406bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35824/0 (socket says 172.21.15.88:35824) 2024-09-16T01:44:55.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.985+0000 7fa049ffa640 1 -- 172.21.15.88:0/2540275944 learned_addr learned my addr 172.21.15.88:0/2540275944 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:55.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.985+0000 7fa049ffa640 1 -- 172.21.15.88:0/2540275944 >> v2:172.21.15.154:3300/0 conn(0x7fa04415c710 msgr2=0x7fa044072050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:55.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.985+0000 7fa049ffa640 1 --2- 172.21.15.88:0/2540275944 >> v2:172.21.15.154:3300/0 conn(0x7fa04415c710 0x7fa044072050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:55.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.985+0000 7fa049ffa640 1 -- 172.21.15.88:0/2540275944 >> v1:172.21.15.154:6789/0 conn(0x7fa044072590 legacy=0x7fa0440d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:55.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.985+0000 7fa049ffa640 1 -- 172.21.15.88:0/2540275944 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa044072ff0 con 0x7fa04406bc10 2024-09-16T01:44:55.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.986+0000 7fa049ffa640 1 --2- 172.21.15.88:0/2540275944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa04406bc10 0x7fa04406bff0 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7fa0440b68d0 tx=0x7fa04002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622e8ab7b271ac7f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:55.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.986+0000 7fa048ff8640 1 -- 172.21.15.88:0/2540275944 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa040037070 con 0x7fa04406bc10 2024-09-16T01:44:55.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.986+0000 7fa048ff8640 1 -- 172.21.15.88:0/2540275944 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa040032440 con 0x7fa04406bc10 2024-09-16T01:44:55.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.986+0000 7fa048ff8640 1 -- 172.21.15.88:0/2540275944 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa040032810 con 0x7fa04406bc10 2024-09-16T01:44:55.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.986+0000 7fa04c25e640 1 -- 172.21.15.88:0/2540275944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa04406bc10 msgr2=0x7fa04406bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:55.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.986+0000 7fa04c25e640 1 --2- 172.21.15.88:0/2540275944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa04406bc10 0x7fa04406bff0 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7fa0440b68d0 tx=0x7fa04002a230 comp rx=0 tx=0).stop 2024-09-16T01:44:55.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.987+0000 7fa04c25e640 1 -- 172.21.15.88:0/2540275944 shutdown_connections 2024-09-16T01:44:55.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.987+0000 7fa04c25e640 1 --2- 172.21.15.88:0/2540275944 >> v2:172.21.15.154:3300/0 conn(0x7fa04415c710 0x7fa044072050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:55.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.987+0000 7fa04c25e640 1 --2- 172.21.15.88:0/2540275944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa04406bc10 0x7fa04406bff0 unknown :-1 s=CLOSED pgs=2351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:55.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.987+0000 7fa04c25e640 1 -- 172.21.15.88:0/2540275944 >> 172.21.15.88:0/2540275944 conn(0x7fa0440d14d0 msgr2=0x7fa0440d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:55.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.987+0000 7fa04c25e640 1 -- 172.21.15.88:0/2540275944 shutdown_connections 2024-09-16T01:44:55.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.987+0000 7fa04c25e640 1 -- 172.21.15.88:0/2540275944 wait complete. 2024-09-16T01:44:55.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.988+0000 7fa04c25e640 1 Processor -- start 2024-09-16T01:44:55.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.988+0000 7fa04c25e640 1 -- start start 2024-09-16T01:44:55.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.988+0000 7fa04c25e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa04406bc10 0x7fa0441f5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:55.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.988+0000 7fa04c25e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa044072590 0x7fa0440cd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:55.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.988+0000 7fa04c25e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa04415c710 0x7fa0441e6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:55.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.988+0000 7fa04c25e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa04406f850 con 0x7fa044072590 2024-09-16T01:44:55.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.988+0000 7fa04c25e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa0440735b0 con 0x7fa04406bc10 2024-09-16T01:44:55.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.988+0000 7fa04c25e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa04406f9d0 con 0x7fa04415c710 2024-09-16T01:44:55.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.989+0000 7fa0497f9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa044072590 0x7fa0440cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:55.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.989+0000 7fa0497f9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa044072590 0x7fa0440cd280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35838/0 (socket says 172.21.15.88:35838) 2024-09-16T01:44:55.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.989+0000 7fa0497f9640 1 -- 172.21.15.88:0/3380910136 learned_addr learned my addr 172.21.15.88:0/3380910136 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:55.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.989+0000 7fa049ffa640 1 --2- 172.21.15.88:0/3380910136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa04406bc10 0x7fa0441f5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:55.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.989+0000 7fa04a7fb640 1 --2- 172.21.15.88:0/3380910136 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa04415c710 0x7fa0441e6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:55.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.989+0000 7fa0497f9640 1 -- 172.21.15.88:0/3380910136 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa04415c710 msgr2=0x7fa0441e6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:55.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.989+0000 7fa0497f9640 1 --2- 172.21.15.88:0/3380910136 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa04415c710 0x7fa0441e6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:55.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.989+0000 7fa0497f9640 1 -- 172.21.15.88:0/3380910136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa04406bc10 msgr2=0x7fa0441f5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:55.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.989+0000 7fa0497f9640 1 --2- 172.21.15.88:0/3380910136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa04406bc10 0x7fa0441f5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:55.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.989+0000 7fa0497f9640 1 -- 172.21.15.88:0/3380910136 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0440cd7c0 con 0x7fa044072590 2024-09-16T01:44:55.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.989+0000 7fa049ffa640 1 --2- 172.21.15.88:0/3380910136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa04406bc10 0x7fa0441f5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:55.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.989+0000 7fa04a7fb640 1 --2- 172.21.15.88:0/3380910136 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa04415c710 0x7fa0441e6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:55.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.989+0000 7fa0497f9640 1 --2- 172.21.15.88:0/3380910136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa044072590 0x7fa0440cd280 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7fa03400b440 tx=0x7fa03400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:55.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.989+0000 7fa03a7fc640 1 -- 172.21.15.88:0/3380910136 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa034015020 con 0x7fa044072590 2024-09-16T01:44:55.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.989+0000 7fa03a7fc640 1 -- 172.21.15.88:0/3380910136 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa034011830 con 0x7fa044072590 2024-09-16T01:44:55.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.989+0000 7fa04c25e640 1 -- 172.21.15.88:0/3380910136 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0441e6990 con 0x7fa044072590 2024-09-16T01:44:55.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.989+0000 7fa03a7fc640 1 -- 172.21.15.88:0/3380910136 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa034011c20 con 0x7fa044072590 2024-09-16T01:44:55.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.989+0000 7fa04c25e640 1 -- 172.21.15.88:0/3380910136 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0441f6840 con 0x7fa044072590 2024-09-16T01:44:55.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.990+0000 7fa04c25e640 1 -- 172.21.15.88:0/3380910136 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa04406bff0 con 0x7fa044072590 2024-09-16T01:44:55.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.991+0000 7fa03a7fc640 1 -- 172.21.15.88:0/3380910136 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa034018020 con 0x7fa044072590 2024-09-16T01:44:55.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.991+0000 7fa03a7fc640 1 --2- 172.21.15.88:0/3380910136 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa02806c2b0 0x7fa02806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:55.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.992+0000 7fa049ffa640 1 --2- 172.21.15.88:0/3380910136 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa02806c2b0 0x7fa02806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:55.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.992+0000 7fa03a7fc640 1 -- 172.21.15.88:0/3380910136 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5709+0+0 (secure 0 0 0) 0x7fa03408b2f0 con 0x7fa044072590 2024-09-16T01:44:55.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.992+0000 7fa049ffa640 1 --2- 172.21.15.88:0/3380910136 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa02806c2b0 0x7fa02806e770 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fa040034000 tx=0x7fa04002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:55.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:55.996+0000 7fa03a7fc640 1 -- 172.21.15.88:0/3380910136 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fa03401f020 con 0x7fa044072590 2024-09-16T01:44:56.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.161+0000 7fa04c25e640 1 -- 172.21.15.88:0/3380910136 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa04415a5a0 con 0x7fa044072590 2024-09-16T01:44:56.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.162+0000 7fa03a7fc640 1 -- 172.21.15.88:0/3380910136 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) ==== 74+0+16655 (secure 0 0 0) 0x7fa0340172c0 con 0x7fa044072590 2024-09-16T01:44:56.164 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:56.164 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":138,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:44:53.504127+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:44:50.167367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-09-16T01:44:50.816821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:44:56.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.164+0000 7fa04c25e640 1 -- 172.21.15.88:0/3380910136 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa02806c2b0 msgr2=0x7fa02806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:56.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.164+0000 7fa04c25e640 1 --2- 172.21.15.88:0/3380910136 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa02806c2b0 0x7fa02806e770 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fa040034000 tx=0x7fa04002e000 comp rx=0 tx=0).stop 2024-09-16T01:44:56.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.164+0000 7fa04c25e640 1 -- 172.21.15.88:0/3380910136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa044072590 msgr2=0x7fa0440cd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:56.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.164+0000 7fa04c25e640 1 --2- 172.21.15.88:0/3380910136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa044072590 0x7fa0440cd280 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7fa03400b440 tx=0x7fa03400b900 comp rx=0 tx=0).stop 2024-09-16T01:44:56.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.164+0000 7fa04c25e640 1 -- 172.21.15.88:0/3380910136 shutdown_connections 2024-09-16T01:44:56.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.164+0000 7fa04c25e640 1 --2- 172.21.15.88:0/3380910136 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa02806c2b0 0x7fa02806e770 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.164+0000 7fa04c25e640 1 --2- 172.21.15.88:0/3380910136 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa04415c710 0x7fa0441e6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.164+0000 7fa04c25e640 1 --2- 172.21.15.88:0/3380910136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa044072590 0x7fa0440cd280 unknown :-1 s=CLOSED pgs=2352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.164+0000 7fa04c25e640 1 --2- 172.21.15.88:0/3380910136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa04406bc10 0x7fa0441f5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.164+0000 7fa04c25e640 1 -- 172.21.15.88:0/3380910136 >> 172.21.15.88:0/3380910136 conn(0x7fa0440d14d0 msgr2=0x7fa0440b5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:56.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.164+0000 7fa04c25e640 1 -- 172.21.15.88:0/3380910136 shutdown_connections 2024-09-16T01:44:56.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.164+0000 7fa04c25e640 1 -- 172.21.15.88:0/3380910136 wait complete. 2024-09-16T01:44:56.178 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T01:44:56.179 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:44:56.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.275+0000 7f66c325e640 1 Processor -- start 2024-09-16T01:44:56.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.275+0000 7f66c325e640 1 -- start start 2024-09-16T01:44:56.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.276+0000 7f66c325e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f66bc15e660 0x7f66bc0cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:56.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.276+0000 7f66c325e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66bc0d1820 0x7f66bc0b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:56.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.276+0000 7f66c325e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66bc15ea40 con 0x7f66bc0d3310 2024-09-16T01:44:56.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.276+0000 7f66c325e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f66bc0d3870 con 0x7f66bc15e660 2024-09-16T01:44:56.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.276+0000 7f66c325e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f66bc0d3b50 con 0x7f66bc0d1820 2024-09-16T01:44:56.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.276+0000 7f66c0ffa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f66bc15e660 0x7f66bc0cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:56.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.276+0000 7f66c0ffa640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f66bc15e660 0x7f66bc0cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35854/0 (socket says 172.21.15.88:35854) 2024-09-16T01:44:56.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.276+0000 7f66c0ffa640 1 -- 172.21.15.88:0/3866858273 learned_addr learned my addr 172.21.15.88:0/3866858273 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:56.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.277+0000 7f66c17fb640 1 --2- 172.21.15.88:0/3866858273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66bc0d1820 0x7f66bc0b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:56.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.277+0000 7f66c0ffa640 1 -- 172.21.15.88:0/3866858273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66bc0d1820 msgr2=0x7f66bc0b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:56.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.277+0000 7f66c0ffa640 1 --2- 172.21.15.88:0/3866858273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66bc0d1820 0x7f66bc0b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.277+0000 7f66c0ffa640 1 -- 172.21.15.88:0/3866858273 >> v1:172.21.15.154:6789/0 conn(0x7f66bc0d3310 legacy=0x7f66bc0d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:56.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.277+0000 7f66c0ffa640 1 -- 172.21.15.88:0/3866858273 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66bc0d3e30 con 0x7f66bc15e660 2024-09-16T01:44:56.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.278+0000 7f66c0ffa640 1 --2- 172.21.15.88:0/3866858273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66bc15e660 0x7f66bc0cf730 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7f66b0009870 tx=0x7f66b002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cfe4b0fc1eba47d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:56.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.278+0000 7f66bb7fe640 1 -- 172.21.15.88:0/3866858273 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b0037070 con 0x7f66bc15e660 2024-09-16T01:44:56.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.278+0000 7f66bb7fe640 1 -- 172.21.15.88:0/3866858273 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66b0032440 con 0x7f66bc15e660 2024-09-16T01:44:56.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.278+0000 7f66bb7fe640 1 -- 172.21.15.88:0/3866858273 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b0032810 con 0x7f66bc15e660 2024-09-16T01:44:56.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.278+0000 7f66c325e640 1 -- 172.21.15.88:0/3866858273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66bc15e660 msgr2=0x7f66bc0cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:56.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.278+0000 7f66c325e640 1 --2- 172.21.15.88:0/3866858273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66bc15e660 0x7f66bc0cf730 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7f66b0009870 tx=0x7f66b002a270 comp rx=0 tx=0).stop 2024-09-16T01:44:56.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.278+0000 7f66c325e640 1 -- 172.21.15.88:0/3866858273 shutdown_connections 2024-09-16T01:44:56.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.278+0000 7f66c325e640 1 --2- 172.21.15.88:0/3866858273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66bc0d1820 0x7f66bc0b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.278+0000 7f66c325e640 1 --2- 172.21.15.88:0/3866858273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66bc15e660 0x7f66bc0cf730 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.278+0000 7f66c325e640 1 -- 172.21.15.88:0/3866858273 >> 172.21.15.88:0/3866858273 conn(0x7f66bc0b6410 msgr2=0x7f66bc0b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:56.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.278+0000 7f66c325e640 1 -- 172.21.15.88:0/3866858273 shutdown_connections 2024-09-16T01:44:56.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.278+0000 7f66c325e640 1 -- 172.21.15.88:0/3866858273 wait complete. 2024-09-16T01:44:56.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.279+0000 7f66c325e640 1 Processor -- start 2024-09-16T01:44:56.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.280+0000 7f66c325e640 1 -- start start 2024-09-16T01:44:56.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.280+0000 7f66c325e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66bc0d1820 0x7f66bc0ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:56.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.280+0000 7f66c325e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66bc0d3310 0x7f66bc1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:56.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.280+0000 7f66c325e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66bc15e660 0x7f66bc151700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:56.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.280+0000 7f66c325e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f66bc0cfc70 con 0x7f66bc0d1820 2024-09-16T01:44:56.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.280+0000 7f66c325e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f66bc0d43f0 con 0x7f66bc0d3310 2024-09-16T01:44:56.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.280+0000 7f66c325e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f66bc0b80b0 con 0x7f66bc15e660 2024-09-16T01:44:56.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.280+0000 7f66c0ffa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66bc0d1820 0x7f66bc0ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:56.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.280+0000 7f66c0ffa640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66bc0d1820 0x7f66bc0ca7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35860/0 (socket says 172.21.15.88:35860) 2024-09-16T01:44:56.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.280+0000 7f66c0ffa640 1 -- 172.21.15.88:0/1972710523 learned_addr learned my addr 172.21.15.88:0/1972710523 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:56.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.281+0000 7f66bbfff640 1 --2- 172.21.15.88:0/1972710523 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66bc0d3310 0x7f66bc1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:56.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.281+0000 7f66c17fb640 1 --2- 172.21.15.88:0/1972710523 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66bc15e660 0x7f66bc151700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:56.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.281+0000 7f66c0ffa640 1 -- 172.21.15.88:0/1972710523 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66bc15e660 msgr2=0x7f66bc151700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:56.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.281+0000 7f66c0ffa640 1 --2- 172.21.15.88:0/1972710523 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66bc15e660 0x7f66bc151700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.281+0000 7f66c0ffa640 1 -- 172.21.15.88:0/1972710523 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66bc0d3310 msgr2=0x7f66bc1553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:56.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.281+0000 7f66c0ffa640 1 --2- 172.21.15.88:0/1972710523 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66bc0d3310 0x7f66bc1553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.281+0000 7f66c0ffa640 1 -- 172.21.15.88:0/1972710523 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66bc0cb0f0 con 0x7f66bc0d1820 2024-09-16T01:44:56.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.281+0000 7f66c0ffa640 1 --2- 172.21.15.88:0/1972710523 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66bc0d1820 0x7f66bc0ca7a0 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f66b0000c00 tx=0x7f66b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:56.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.281+0000 7f66b97fa640 1 -- 172.21.15.88:0/1972710523 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b0037040 con 0x7f66bc0d1820 2024-09-16T01:44:56.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.281+0000 7f66b97fa640 1 -- 172.21.15.88:0/1972710523 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66b0042070 con 0x7f66bc0d1820 2024-09-16T01:44:56.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.281+0000 7f66b97fa640 1 -- 172.21.15.88:0/1972710523 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b0030880 con 0x7f66bc0d1820 2024-09-16T01:44:56.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.281+0000 7f66c325e640 1 -- 172.21.15.88:0/1972710523 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66bc0cae10 con 0x7f66bc0d1820 2024-09-16T01:44:56.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.281+0000 7f66c325e640 1 -- 172.21.15.88:0/1972710523 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66bc150f00 con 0x7f66bc0d1820 2024-09-16T01:44:56.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.282+0000 7f66c325e640 1 -- 172.21.15.88:0/1972710523 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66bc0b7b70 con 0x7f66bc0d1820 2024-09-16T01:44:56.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.283+0000 7f66b97fa640 1 -- 172.21.15.88:0/1972710523 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f66b0030a20 con 0x7f66bc0d1820 2024-09-16T01:44:56.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.283+0000 7f66b97fa640 1 --2- 172.21.15.88:0/1972710523 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f66a006c2b0 0x7f66a006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:56.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.283+0000 7f66b97fa640 1 -- 172.21.15.88:0/1972710523 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5709+0+0 (secure 0 0 0) 0x7f66b00aba40 con 0x7f66bc0d1820 2024-09-16T01:44:56.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.283+0000 7f66bbfff640 1 --2- 172.21.15.88:0/1972710523 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f66a006c2b0 0x7f66a006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:56.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.284+0000 7f66bbfff640 1 --2- 172.21.15.88:0/1972710523 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f66a006c2b0 0x7f66a006e770 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f66bc06b010 tx=0x7f66a8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:56.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.288+0000 7f66b97fa640 1 -- 172.21.15.88:0/1972710523 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f66b0079bf0 con 0x7f66bc0d1820 2024-09-16T01:44:56.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.449+0000 7f66c325e640 1 -- 172.21.15.88:0/1972710523 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f66bc15c450 con 0x7f66a006c2b0 2024-09-16T01:44:56.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.454+0000 7f66b97fa640 1 -- 172.21.15.88:0/1972710523 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+240784 (secure 0 0 0) 0x7f66bc15c450 con 0x7f66a006c2b0 2024-09-16T01:44:56.458 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:56.459 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:44:56.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.457+0000 7f66c325e640 1 -- 172.21.15.88:0/1972710523 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f66a006c2b0 msgr2=0x7f66a006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:56.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.457+0000 7f66c325e640 1 --2- 172.21.15.88:0/1972710523 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f66a006c2b0 0x7f66a006e770 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f66bc06b010 tx=0x7f66a8009040 comp rx=0 tx=0).stop 2024-09-16T01:44:56.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.457+0000 7f66c325e640 1 -- 172.21.15.88:0/1972710523 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66bc0d1820 msgr2=0x7f66bc0ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:56.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.457+0000 7f66c325e640 1 --2- 172.21.15.88:0/1972710523 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66bc0d1820 0x7f66bc0ca7a0 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f66b0000c00 tx=0x7f66b00046c0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.457+0000 7f66c325e640 1 -- 172.21.15.88:0/1972710523 shutdown_connections 2024-09-16T01:44:56.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.457+0000 7f66c325e640 1 --2- 172.21.15.88:0/1972710523 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f66a006c2b0 0x7f66a006e770 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.457+0000 7f66c325e640 1 --2- 172.21.15.88:0/1972710523 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f66bc15e660 0x7f66bc151700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.457+0000 7f66c325e640 1 --2- 172.21.15.88:0/1972710523 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66bc0d3310 0x7f66bc1553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.457+0000 7f66c325e640 1 --2- 172.21.15.88:0/1972710523 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f66bc0d1820 0x7f66bc0ca7a0 unknown :-1 s=CLOSED pgs=2354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.457+0000 7f66c325e640 1 -- 172.21.15.88:0/1972710523 >> 172.21.15.88:0/1972710523 conn(0x7f66bc0b6410 msgr2=0x7f66bc0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:56.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.458+0000 7f66c325e640 1 -- 172.21.15.88:0/1972710523 shutdown_connections 2024-09-16T01:44:56.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.458+0000 7f66c325e640 1 -- 172.21.15.88:0/1972710523 wait complete. 2024-09-16T01:44:56.469 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":453,"stamp":"2024-09-16T01:44:54.655500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":70,"acting":70,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":297400,"kb_used_data":14576,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754677320,"statfs":{"total":773094113280,"available":772789575680,"internally_reserved":0,"allocated":14925824,"data_stored":7798046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":289394226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.159096"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.500840+0000","last_change":"2024-09-16T01:44:50.500840+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.500840+0000","last_undegraded":"2024-09-16T01:44:50.500840+0000","last_fullsized":"2024-09-16T01:44:50.500840+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.499735+0000","last_change":"2024-09-16T01:44:50.499735+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.499735+0000","last_undegraded":"2024-09-16T01:44:50.499735+0000","last_fullsized":"2024-09-16T01:44:50.499735+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.501653+0000","last_change":"2024-09-16T01:44:50.501653+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.501653+0000","last_undegraded":"2024-09-16T01:44:50.501653+0000","last_fullsized":"2024-09-16T01:44:50.501653+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.501562+0000","last_change":"2024-09-16T01:44:50.501562+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.501562+0000","last_undegraded":"2024-09-16T01:44:50.501562+0000","last_fullsized":"2024-09-16T01:44:50.501562+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-09-16T01:44:51.512056+0000","last_change":"2024-09-16T01:44:51.512056+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.512056+0000","last_undegraded":"2024-09-16T01:44:51.512056+0000","last_fullsized":"2024-09-16T01:44:51.512056+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.502267+0000","last_change":"2024-09-16T01:44:50.502267+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.502267+0000","last_undegraded":"2024-09-16T01:44:50.502267+0000","last_fullsized":"2024-09-16T01:44:50.502267+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.500927+0000","last_change":"2024-09-16T01:44:50.500927+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.500927+0000","last_undegraded":"2024-09-16T01:44:50.500927+0000","last_fullsized":"2024-09-16T01:44:50.500927+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.499217+0000","last_change":"2024-09-16T01:44:50.499217+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.499217+0000","last_undegraded":"2024-09-16T01:44:50.499217+0000","last_fullsized":"2024-09-16T01:44:50.499217+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.501534+0000","last_change":"2024-09-16T01:44:50.501534+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.501534+0000","last_undegraded":"2024-09-16T01:44:50.501534+0000","last_fullsized":"2024-09-16T01:44:50.501534+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.500923+0000","last_change":"2024-09-16T01:44:50.500923+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.500923+0000","last_undegraded":"2024-09-16T01:44:50.500923+0000","last_fullsized":"2024-09-16T01:44:50.500923+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.499189+0000","last_change":"2024-09-16T01:44:50.499189+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.499189+0000","last_undegraded":"2024-09-16T01:44:50.499189+0000","last_fullsized":"2024-09-16T01:44:50.499189+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.499683+0000","last_change":"2024-09-16T01:44:50.499683+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.499683+0000","last_undegraded":"2024-09-16T01:44:50.499683+0000","last_fullsized":"2024-09-16T01:44:50.499683+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.502516+0000","last_change":"2024-09-16T01:44:50.502516+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.502516+0000","last_undegraded":"2024-09-16T01:44:50.502516+0000","last_fullsized":"2024-09-16T01:44:50.502516+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.504649+0000","last_change":"2024-09-16T01:44:50.504649+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.504649+0000","last_undegraded":"2024-09-16T01:44:50.504649+0000","last_fullsized":"2024-09-16T01:44:50.504649+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.504649+0000","last_change":"2024-09-16T01:44:50.504649+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.504649+0000","last_undegraded":"2024-09-16T01:44:50.504649+0000","last_fullsized":"2024-09-16T01:44:50.504649+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.502510+0000","last_change":"2024-09-16T01:44:50.502510+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.502510+0000","last_undegraded":"2024-09-16T01:44:50.502510+0000","last_fullsized":"2024-09-16T01:44:50.502510+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.501039+0000","last_change":"2024-09-16T01:44:50.501039+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.501039+0000","last_undegraded":"2024-09-16T01:44:50.501039+0000","last_fullsized":"2024-09-16T01:44:50.501039+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.503546+0000","last_change":"2024-09-16T01:44:50.503546+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.503546+0000","last_undegraded":"2024-09-16T01:44:50.503546+0000","last_fullsized":"2024-09-16T01:44:50.503546+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":1,"reported_epoch":136,"state":"creating+peering","last_fresh":"2024-09-16T01:44:51.511053+0000","last_change":"2024-09-16T01:44:51.511053+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.511053+0000","last_undegraded":"2024-09-16T01:44:51.511053+0000","last_fullsized":"2024-09-16T01:44:51.511053+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.499285+0000","last_change":"2024-09-16T01:44:50.499285+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.499285+0000","last_undegraded":"2024-09-16T01:44:50.499285+0000","last_fullsized":"2024-09-16T01:44:50.499285+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.501072+0000","last_change":"2024-09-16T01:44:50.501072+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.501072+0000","last_undegraded":"2024-09-16T01:44:50.501072+0000","last_fullsized":"2024-09-16T01:44:50.501072+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.502152+0000","last_change":"2024-09-16T01:44:50.502152+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.502152+0000","last_undegraded":"2024-09-16T01:44:50.502152+0000","last_fullsized":"2024-09-16T01:44:50.502152+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":9,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T01:44:51.522793+0000","last_change":"2024-09-16T01:44:51.519279+0000","last_active":"2024-09-16T01:44:51.522793+0000","last_peered":"2024-09-16T01:44:51.522793+0000","last_clean":"2024-09-16T01:44:51.522793+0000","last_became_active":"2024-09-16T01:44:51.519002+0000","last_became_peered":"2024-09-16T01:44:51.519002+0000","last_unstale":"2024-09-16T01:44:51.522793+0000","last_undegraded":"2024-09-16T01:44:51.522793+0000","last_fullsized":"2024-09-16T01:44:51.522793+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:24:50.036463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":9,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T01:44:51.522703+0000","last_change":"2024-09-16T01:44:51.517843+0000","last_active":"2024-09-16T01:44:51.522703+0000","last_peered":"2024-09-16T01:44:51.522703+0000","last_clean":"2024-09-16T01:44:51.522703+0000","last_became_active":"2024-09-16T01:44:51.517548+0000","last_became_peered":"2024-09-16T01:44:51.517548+0000","last_unstale":"2024-09-16T01:44:51.522703+0000","last_undegraded":"2024-09-16T01:44:51.522703+0000","last_fullsized":"2024-09-16T01:44:51.522703+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:10:28.668777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.501688+0000","last_change":"2024-09-16T01:44:50.501688+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.501688+0000","last_undegraded":"2024-09-16T01:44:50.501688+0000","last_fullsized":"2024-09-16T01:44:50.501688+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":9,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T01:44:51.515647+0000","last_change":"2024-09-16T01:44:51.513903+0000","last_active":"2024-09-16T01:44:51.515647+0000","last_peered":"2024-09-16T01:44:51.515647+0000","last_clean":"2024-09-16T01:44:51.515647+0000","last_became_active":"2024-09-16T01:44:51.513583+0000","last_became_peered":"2024-09-16T01:44:51.513583+0000","last_unstale":"2024-09-16T01:44:51.515647+0000","last_undegraded":"2024-09-16T01:44:51.515647+0000","last_fullsized":"2024-09-16T01:44:51.515647+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:08.195158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.501031+0000","last_change":"2024-09-16T01:44:50.501031+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.501031+0000","last_undegraded":"2024-09-16T01:44:50.501031+0000","last_fullsized":"2024-09-16T01:44:50.501031+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.501121+0000","last_change":"2024-09-16T01:44:50.501121+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.501121+0000","last_undegraded":"2024-09-16T01:44:50.501121+0000","last_fullsized":"2024-09-16T01:44:50.501121+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.499212+0000","last_change":"2024-09-16T01:44:50.499212+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.499212+0000","last_undegraded":"2024-09-16T01:44:50.499212+0000","last_fullsized":"2024-09-16T01:44:50.499212+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.501208+0000","last_change":"2024-09-16T01:44:50.501208+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.501208+0000","last_undegraded":"2024-09-16T01:44:50.501208+0000","last_fullsized":"2024-09-16T01:44:50.501208+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.499938+0000","last_change":"2024-09-16T01:44:50.499938+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.499938+0000","last_undegraded":"2024-09-16T01:44:50.499938+0000","last_fullsized":"2024-09-16T01:44:50.499938+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.499207+0000","last_change":"2024-09-16T01:44:50.499207+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.499207+0000","last_undegraded":"2024-09-16T01:44:50.499207+0000","last_fullsized":"2024-09-16T01:44:50.499207+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.504658+0000","last_change":"2024-09-16T01:44:50.504658+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.504658+0000","last_undegraded":"2024-09-16T01:44:50.504658+0000","last_fullsized":"2024-09-16T01:44:50.504658+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T01:44:51.496468+0000","last_change":"2024-09-16T01:44:51.496468+0000","last_active":"2024-09-16T01:44:51.496468+0000","last_peered":"2024-09-16T01:44:51.496468+0000","last_clean":"2024-09-16T01:44:51.496468+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:51.496468+0000","last_undegraded":"2024-09-16T01:44:51.496468+0000","last_fullsized":"2024-09-16T01:44:51.496468+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":135,"state":"creating+peering","last_fresh":"2024-09-16T01:44:50.499468+0000","last_change":"2024-09-16T01:44:50.499468+0000","last_active":"2024-09-16T01:44:50.494064+0000","last_peered":"2024-09-16T01:44:50.494064+0000","last_clean":"2024-09-16T01:44:50.494064+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T01:44:50.499468+0000","last_undegraded":"2024-09-16T01:44:50.499468+0000","last_fullsized":"2024-09-16T01:44:50.499468+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":386,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T01:44:50.499251+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:44:50.499251+0000","last_peered":"2024-09-16T01:44:50.499251+0000","last_clean":"2024-09-16T01:44:50.499251+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:44:50.499251+0000","last_undegraded":"2024-09-16T01:44:50.499251+0000","last_fullsized":"2024-09-16T01:44:50.499251+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":4,"acting":4,"num_store_stats":1},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509573,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37060,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334780,"statfs":{"total":96636764160,"available":96598814720,"internally_reserved":0,"allocated":1732608,"data_stored":835200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542276,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37596,"kb_used_data":2248,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334244,"statfs":{"total":96636764160,"available":96598265856,"internally_reserved":0,"allocated":2301952,"data_stored":1411894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542276,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37028,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334812,"statfs":{"total":96636764160,"available":96598847488,"internally_reserved":0,"allocated":1720320,"data_stored":829383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542276,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32884,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338956,"statfs":{"total":96636764160,"available":96603090944,"internally_reserved":0,"allocated":1720320,"data_stored":829383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542275,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37028,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334812,"statfs":{"total":96636764160,"available":96598847488,"internally_reserved":0,"allocated":1720320,"data_stored":829383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542276,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37016,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334824,"statfs":{"total":96636764160,"available":96598859776,"internally_reserved":0,"allocated":1708032,"data_stored":821526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542275,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37032,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334808,"statfs":{"total":96636764160,"available":96598843392,"internally_reserved":0,"allocated":1720320,"data_stored":829383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542275,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41756,"kb_used_data":2248,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330084,"statfs":{"total":96636764160,"available":96594006016,"internally_reserved":0,"allocated":2301952,"data_stored":1411894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:44:56.522 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:44:56.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.620+0000 7ff6f3928640 1 Processor -- start 2024-09-16T01:44:56.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.620+0000 7ff6f3928640 1 -- start start 2024-09-16T01:44:56.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.621+0000 7ff6f3928640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff6ec0d1b40 0x7ff6ec1618a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:56.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.621+0000 7ff6f3928640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff6ec0b7290 0x7ff6ec161de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:56.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.621+0000 7ff6f3928640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6ec0708d0 con 0x7ff6ec0b6eb0 2024-09-16T01:44:56.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.621+0000 7ff6f3928640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff6ec070bb0 con 0x7ff6ec0d1b40 2024-09-16T01:44:56.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.621+0000 7ff6f3928640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff6ec070e90 con 0x7ff6ec0b7290 2024-09-16T01:44:56.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.621+0000 7ff6f0ec3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff6ec0d1b40 0x7ff6ec1618a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:56.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.621+0000 7ff6f0ec3640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff6ec0d1b40 0x7ff6ec1618a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35864/0 (socket says 172.21.15.88:35864) 2024-09-16T01:44:56.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.621+0000 7ff6f0ec3640 1 -- 172.21.15.88:0/3271437078 learned_addr learned my addr 172.21.15.88:0/3271437078 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:56.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.621+0000 7ff6f0ec3640 1 -- 172.21.15.88:0/3271437078 >> v2:172.21.15.154:3300/0 conn(0x7ff6ec0b7290 msgr2=0x7ff6ec161de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:56.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.621+0000 7ff6f1ec5640 1 --2- 172.21.15.88:0/3271437078 >> v2:172.21.15.154:3300/0 conn(0x7ff6ec0b7290 0x7ff6ec161de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:56.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.621+0000 7ff6f0ec3640 1 --2- 172.21.15.88:0/3271437078 >> v2:172.21.15.154:3300/0 conn(0x7ff6ec0b7290 0x7ff6ec161de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.621+0000 7ff6f0ec3640 1 -- 172.21.15.88:0/3271437078 >> v1:172.21.15.154:6789/0 conn(0x7ff6ec0b6eb0 legacy=0x7ff6ec06fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:56.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.621+0000 7ff6f0ec3640 1 -- 172.21.15.88:0/3271437078 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6ec071170 con 0x7ff6ec0d1b40 2024-09-16T01:44:56.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.621+0000 7ff6f1ec5640 1 --2- 172.21.15.88:0/3271437078 >> v2:172.21.15.154:3300/0 conn(0x7ff6ec0b7290 0x7ff6ec161de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:56.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.622+0000 7ff6f0ec3640 1 --2- 172.21.15.88:0/3271437078 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6ec0d1b40 0x7ff6ec1618a0 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec0759a0 tx=0x7ff6dc02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c3a28296b5d371a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:56.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.622+0000 7ff6e3fff640 1 -- 172.21.15.88:0/3271437078 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc037070 con 0x7ff6ec0d1b40 2024-09-16T01:44:56.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.622+0000 7ff6e3fff640 1 -- 172.21.15.88:0/3271437078 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6dc02ae20 con 0x7ff6ec0d1b40 2024-09-16T01:44:56.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.622+0000 7ff6e3fff640 1 -- 172.21.15.88:0/3271437078 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc032660 con 0x7ff6ec0d1b40 2024-09-16T01:44:56.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.623+0000 7ff6f3928640 1 -- 172.21.15.88:0/3271437078 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6ec0d1b40 msgr2=0x7ff6ec1618a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:56.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.623+0000 7ff6f3928640 1 --2- 172.21.15.88:0/3271437078 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6ec0d1b40 0x7ff6ec1618a0 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec0759a0 tx=0x7ff6dc02a110 comp rx=0 tx=0).stop 2024-09-16T01:44:56.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.623+0000 7ff6f3928640 1 -- 172.21.15.88:0/3271437078 shutdown_connections 2024-09-16T01:44:56.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.623+0000 7ff6f3928640 1 --2- 172.21.15.88:0/3271437078 >> v2:172.21.15.154:3300/0 conn(0x7ff6ec0b7290 0x7ff6ec161de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.623+0000 7ff6f3928640 1 --2- 172.21.15.88:0/3271437078 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6ec0d1b40 0x7ff6ec1618a0 unknown :-1 s=CLOSED pgs=2355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.623+0000 7ff6f3928640 1 -- 172.21.15.88:0/3271437078 >> 172.21.15.88:0/3271437078 conn(0x7ff6ec0d2450 msgr2=0x7ff6ec0d2830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:56.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.623+0000 7ff6f3928640 1 -- 172.21.15.88:0/3271437078 shutdown_connections 2024-09-16T01:44:56.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.623+0000 7ff6f3928640 1 -- 172.21.15.88:0/3271437078 wait complete. 2024-09-16T01:44:56.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.624+0000 7ff6f3928640 1 Processor -- start 2024-09-16T01:44:56.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.624+0000 7ff6f3928640 1 -- start start 2024-09-16T01:44:56.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.625+0000 7ff6f3928640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6ec0b6eb0 0x7ff6ec14e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:56.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.625+0000 7ff6f3928640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6ec0b7290 0x7ff6ec1f8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.625+0000 7ff6f3928640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6ec0d1b40 0x7ff6ec1f8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.625+0000 7ff6f3928640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ec071af0 con 0x7ff6ec0b7290 2024-09-16T01:44:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.625+0000 7ff6f3928640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ec071970 con 0x7ff6ec0d1b40 2024-09-16T01:44:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.625+0000 7ff6f3928640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff6ec162320 con 0x7ff6ec0b6eb0 2024-09-16T01:44:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.625+0000 7ff6f0ec3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6ec0b7290 0x7ff6ec1f8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.625+0000 7ff6f0ec3640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6ec0b7290 0x7ff6ec1f8350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35880/0 (socket says 172.21.15.88:35880) 2024-09-16T01:44:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.625+0000 7ff6f0ec3640 1 -- 172.21.15.88:0/1829135103 learned_addr learned my addr 172.21.15.88:0/1829135103 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.625+0000 7ff6f1ec5640 1 --2- 172.21.15.88:0/1829135103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6ec0d1b40 0x7ff6ec1f8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.625+0000 7ff6f16c4640 1 --2- 172.21.15.88:0/1829135103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6ec0b6eb0 0x7ff6ec14e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.625+0000 7ff6f0ec3640 1 -- 172.21.15.88:0/1829135103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6ec0b6eb0 msgr2=0x7ff6ec14e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.625+0000 7ff6f0ec3640 1 --2- 172.21.15.88:0/1829135103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6ec0b6eb0 0x7ff6ec14e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.625+0000 7ff6f0ec3640 1 -- 172.21.15.88:0/1829135103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6ec0d1b40 msgr2=0x7ff6ec1f8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:56.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.625+0000 7ff6f0ec3640 1 --2- 172.21.15.88:0/1829135103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6ec0d1b40 0x7ff6ec1f8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.625+0000 7ff6f0ec3640 1 -- 172.21.15.88:0/1829135103 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6ec1f4c40 con 0x7ff6ec0b7290 2024-09-16T01:44:56.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.625+0000 7ff6f1ec5640 1 --2- 172.21.15.88:0/1829135103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6ec0d1b40 0x7ff6ec1f8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:56.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.625+0000 7ff6f16c4640 1 --2- 172.21.15.88:0/1829135103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6ec0b6eb0 0x7ff6ec14e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:56.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.625+0000 7ff6f0ec3640 1 --2- 172.21.15.88:0/1829135103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6ec0b7290 0x7ff6ec1f8350 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc000c00 tx=0x7ff6dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:56.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.625+0000 7ff6e1ffb640 1 -- 172.21.15.88:0/1829135103 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc041070 con 0x7ff6ec0b7290 2024-09-16T01:44:56.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.625+0000 7ff6e1ffb640 1 -- 172.21.15.88:0/1829135103 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6dc03c670 con 0x7ff6ec0b7290 2024-09-16T01:44:56.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.625+0000 7ff6f3928640 1 -- 172.21.15.88:0/1829135103 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6ec160780 con 0x7ff6ec0b7290 2024-09-16T01:44:56.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.625+0000 7ff6e1ffb640 1 -- 172.21.15.88:0/1829135103 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc037040 con 0x7ff6ec0b7290 2024-09-16T01:44:56.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.626+0000 7ff6f3928640 1 -- 172.21.15.88:0/1829135103 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6ec154080 con 0x7ff6ec0b7290 2024-09-16T01:44:56.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.626+0000 7ff6f3928640 1 -- 172.21.15.88:0/1829135103 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6ec070130 con 0x7ff6ec0b7290 2024-09-16T01:44:56.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.627+0000 7ff6e1ffb640 1 -- 172.21.15.88:0/1829135103 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff6dc044070 con 0x7ff6ec0b7290 2024-09-16T01:44:56.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.627+0000 7ff6e1ffb640 1 --2- 172.21.15.88:0/1829135103 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff6d006c2b0 0x7ff6d006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:56.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.627+0000 7ff6e1ffb640 1 -- 172.21.15.88:0/1829135103 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5709+0+0 (secure 0 0 0) 0x7ff6dc0abbb0 con 0x7ff6ec0b7290 2024-09-16T01:44:56.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.627+0000 7ff6f16c4640 1 --2- 172.21.15.88:0/1829135103 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff6d006c2b0 0x7ff6d006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:56.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.628+0000 7ff6f16c4640 1 --2- 172.21.15.88:0/1829135103 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff6d006c2b0 0x7ff6d006e770 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec157280 tx=0x7ff6e8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:56.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.631+0000 7ff6e1ffb640 1 -- 172.21.15.88:0/1829135103 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff6dc07a320 con 0x7ff6ec0b7290 2024-09-16T01:44:56.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.793+0000 7ff6f3928640 1 -- 172.21.15.88:0/1829135103 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff6ec06fd00 con 0x7ff6d006c2b0 2024-09-16T01:44:56.800 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.799+0000 7ff6e1ffb640 1 -- 172.21.15.88:0/1829135103 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252484 (secure 0 0 0) 0x7ff6ec06fd00 con 0x7ff6d006c2b0 2024-09-16T01:44:56.803 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:56.805 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:44:56.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.802+0000 7ff6f3928640 1 -- 172.21.15.88:0/1829135103 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff6d006c2b0 msgr2=0x7ff6d006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:56.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.802+0000 7ff6f3928640 1 --2- 172.21.15.88:0/1829135103 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff6d006c2b0 0x7ff6d006e770 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec157280 tx=0x7ff6e8009040 comp rx=0 tx=0).stop 2024-09-16T01:44:56.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.802+0000 7ff6f3928640 1 -- 172.21.15.88:0/1829135103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6ec0b7290 msgr2=0x7ff6ec1f8350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:56.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.802+0000 7ff6f3928640 1 --2- 172.21.15.88:0/1829135103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6ec0b7290 0x7ff6ec1f8350 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc000c00 tx=0x7ff6dc0046c0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.802+0000 7ff6f3928640 1 -- 172.21.15.88:0/1829135103 shutdown_connections 2024-09-16T01:44:56.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.802+0000 7ff6f3928640 1 --2- 172.21.15.88:0/1829135103 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff6d006c2b0 0x7ff6d006e770 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.802+0000 7ff6f3928640 1 --2- 172.21.15.88:0/1829135103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6ec0d1b40 0x7ff6ec1f8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.802+0000 7ff6f3928640 1 --2- 172.21.15.88:0/1829135103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6ec0b7290 0x7ff6ec1f8350 unknown :-1 s=CLOSED pgs=2356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.802+0000 7ff6f3928640 1 --2- 172.21.15.88:0/1829135103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6ec0b6eb0 0x7ff6ec14e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.802+0000 7ff6f3928640 1 -- 172.21.15.88:0/1829135103 >> 172.21.15.88:0/1829135103 conn(0x7ff6ec0d2450 msgr2=0x7ff6ec15ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:56.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.802+0000 7ff6f3928640 1 -- 172.21.15.88:0/1829135103 shutdown_connections 2024-09-16T01:44:56.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.802+0000 7ff6f3928640 1 -- 172.21.15.88:0/1829135103 wait complete. 2024-09-16T01:44:56.814 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":454,"stamp":"2024-09-16T01:44:56.655991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":297816,"kb_used_data":15000,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754676904,"statfs":{"total":773094113280,"available":772789149696,"internally_reserved":0,"allocated":15360000,"data_stored":7996756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":289394226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001297"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.395254+0000","last_change":"2024-09-16T01:44:52.521713+0000","last_active":"2024-09-16T01:44:55.395254+0000","last_peered":"2024-09-16T01:44:55.395254+0000","last_clean":"2024-09-16T01:44:55.395254+0000","last_became_active":"2024-09-16T01:44:52.521266+0000","last_became_peered":"2024-09-16T01:44:52.521266+0000","last_unstale":"2024-09-16T01:44:55.395254+0000","last_undegraded":"2024-09-16T01:44:55.395254+0000","last_fullsized":"2024-09-16T01:44:55.395254+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:10:29.671181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:54.035167+0000","last_change":"2024-09-16T01:44:52.529566+0000","last_active":"2024-09-16T01:44:54.035167+0000","last_peered":"2024-09-16T01:44:54.035167+0000","last_clean":"2024-09-16T01:44:54.035167+0000","last_became_active":"2024-09-16T01:44:52.529299+0000","last_became_peered":"2024-09-16T01:44:52.529299+0000","last_unstale":"2024-09-16T01:44:54.035167+0000","last_undegraded":"2024-09-16T01:44:54.035167+0000","last_fullsized":"2024-09-16T01:44:54.035167+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:21:24.601465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.395204+0000","last_change":"2024-09-16T01:44:52.524484+0000","last_active":"2024-09-16T01:44:55.395204+0000","last_peered":"2024-09-16T01:44:55.395204+0000","last_clean":"2024-09-16T01:44:55.395204+0000","last_became_active":"2024-09-16T01:44:52.523981+0000","last_became_peered":"2024-09-16T01:44:52.523981+0000","last_unstale":"2024-09-16T01:44:55.395204+0000","last_undegraded":"2024-09-16T01:44:55.395204+0000","last_fullsized":"2024-09-16T01:44:55.395204+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:09.197562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.410257+0000","last_change":"2024-09-16T01:44:52.514263+0000","last_active":"2024-09-16T01:44:55.410257+0000","last_peered":"2024-09-16T01:44:55.410257+0000","last_clean":"2024-09-16T01:44:55.410257+0000","last_became_active":"2024-09-16T01:44:52.513998+0000","last_became_peered":"2024-09-16T01:44:52.513998+0000","last_unstale":"2024-09-16T01:44:55.410257+0000","last_undegraded":"2024-09-16T01:44:55.410257+0000","last_fullsized":"2024-09-16T01:44:55.410257+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:07:17.059934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.510771+0000","last_change":"2024-09-16T01:44:52.531523+0000","last_active":"2024-09-16T01:44:53.510771+0000","last_peered":"2024-09-16T01:44:53.510771+0000","last_clean":"2024-09-16T01:44:53.510771+0000","last_became_active":"2024-09-16T01:44:52.531249+0000","last_became_peered":"2024-09-16T01:44:52.531249+0000","last_unstale":"2024-09-16T01:44:53.510771+0000","last_undegraded":"2024-09-16T01:44:53.510771+0000","last_fullsized":"2024-09-16T01:44:53.510771+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:06:25.625371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409545+0000","last_change":"2024-09-16T01:44:52.529749+0000","last_active":"2024-09-16T01:44:55.409545+0000","last_peered":"2024-09-16T01:44:55.409545+0000","last_clean":"2024-09-16T01:44:55.409545+0000","last_became_active":"2024-09-16T01:44:52.529088+0000","last_became_peered":"2024-09-16T01:44:52.529088+0000","last_unstale":"2024-09-16T01:44:55.409545+0000","last_undegraded":"2024-09-16T01:44:55.409545+0000","last_fullsized":"2024-09-16T01:44:55.409545+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:44:58.041372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409546+0000","last_change":"2024-09-16T01:44:52.529364+0000","last_active":"2024-09-16T01:44:55.409546+0000","last_peered":"2024-09-16T01:44:55.409546+0000","last_clean":"2024-09-16T01:44:55.409546+0000","last_became_active":"2024-09-16T01:44:52.529066+0000","last_became_peered":"2024-09-16T01:44:52.529066+0000","last_unstale":"2024-09-16T01:44:55.409546+0000","last_undegraded":"2024-09-16T01:44:55.409546+0000","last_fullsized":"2024-09-16T01:44:55.409546+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:48:46.475463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:54.034846+0000","last_change":"2024-09-16T01:44:52.532066+0000","last_active":"2024-09-16T01:44:54.034846+0000","last_peered":"2024-09-16T01:44:54.034846+0000","last_clean":"2024-09-16T01:44:54.034846+0000","last_became_active":"2024-09-16T01:44:52.531776+0000","last_became_peered":"2024-09-16T01:44:52.531776+0000","last_unstale":"2024-09-16T01:44:54.034846+0000","last_undegraded":"2024-09-16T01:44:54.034846+0000","last_fullsized":"2024-09-16T01:44:54.034846+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:41:57.682050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.510453+0000","last_change":"2024-09-16T01:44:52.523930+0000","last_active":"2024-09-16T01:44:53.510453+0000","last_peered":"2024-09-16T01:44:53.510453+0000","last_clean":"2024-09-16T01:44:53.510453+0000","last_became_active":"2024-09-16T01:44:52.522040+0000","last_became_peered":"2024-09-16T01:44:52.522040+0000","last_unstale":"2024-09-16T01:44:53.510453+0000","last_undegraded":"2024-09-16T01:44:53.510453+0000","last_fullsized":"2024-09-16T01:44:53.510453+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:14:22.817576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.510475+0000","last_change":"2024-09-16T01:44:52.522963+0000","last_active":"2024-09-16T01:44:53.510475+0000","last_peered":"2024-09-16T01:44:53.510475+0000","last_clean":"2024-09-16T01:44:53.510475+0000","last_became_active":"2024-09-16T01:44:52.522235+0000","last_became_peered":"2024-09-16T01:44:52.522235+0000","last_unstale":"2024-09-16T01:44:53.510475+0000","last_undegraded":"2024-09-16T01:44:53.510475+0000","last_fullsized":"2024-09-16T01:44:53.510475+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:51:33.777392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409681+0000","last_change":"2024-09-16T01:44:52.528253+0000","last_active":"2024-09-16T01:44:55.409681+0000","last_peered":"2024-09-16T01:44:55.409681+0000","last_clean":"2024-09-16T01:44:55.409681+0000","last_became_active":"2024-09-16T01:44:52.527777+0000","last_became_peered":"2024-09-16T01:44:52.527777+0000","last_unstale":"2024-09-16T01:44:55.409681+0000","last_undegraded":"2024-09-16T01:44:55.409681+0000","last_fullsized":"2024-09-16T01:44:55.409681+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:10:36.216084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.511191+0000","last_change":"2024-09-16T01:44:52.527421+0000","last_active":"2024-09-16T01:44:53.511191+0000","last_peered":"2024-09-16T01:44:53.511191+0000","last_clean":"2024-09-16T01:44:53.511191+0000","last_became_active":"2024-09-16T01:44:52.526944+0000","last_became_peered":"2024-09-16T01:44:52.526944+0000","last_unstale":"2024-09-16T01:44:53.511191+0000","last_undegraded":"2024-09-16T01:44:53.511191+0000","last_fullsized":"2024-09-16T01:44:53.511191+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:06:25.625371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.410167+0000","last_change":"2024-09-16T01:44:52.526838+0000","last_active":"2024-09-16T01:44:55.410167+0000","last_peered":"2024-09-16T01:44:55.410167+0000","last_clean":"2024-09-16T01:44:55.410167+0000","last_became_active":"2024-09-16T01:44:52.526573+0000","last_became_peered":"2024-09-16T01:44:52.526573+0000","last_unstale":"2024-09-16T01:44:55.410167+0000","last_undegraded":"2024-09-16T01:44:55.410167+0000","last_fullsized":"2024-09-16T01:44:55.410167+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:06:25.625371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409556+0000","last_change":"2024-09-16T01:44:52.521091+0000","last_active":"2024-09-16T01:44:55.409556+0000","last_peered":"2024-09-16T01:44:55.409556+0000","last_clean":"2024-09-16T01:44:55.409556+0000","last_became_active":"2024-09-16T01:44:52.520834+0000","last_became_peered":"2024-09-16T01:44:52.520834+0000","last_unstale":"2024-09-16T01:44:55.409556+0000","last_undegraded":"2024-09-16T01:44:55.409556+0000","last_fullsized":"2024-09-16T01:44:55.409556+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:05:49.364680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.410084+0000","last_change":"2024-09-16T01:44:52.516228+0000","last_active":"2024-09-16T01:44:55.410084+0000","last_peered":"2024-09-16T01:44:55.410084+0000","last_clean":"2024-09-16T01:44:55.410084+0000","last_became_active":"2024-09-16T01:44:52.515977+0000","last_became_peered":"2024-09-16T01:44:52.515977+0000","last_unstale":"2024-09-16T01:44:55.410084+0000","last_undegraded":"2024-09-16T01:44:55.410084+0000","last_fullsized":"2024-09-16T01:44:55.410084+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:37:23.230828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409768+0000","last_change":"2024-09-16T01:44:52.534431+0000","last_active":"2024-09-16T01:44:55.409768+0000","last_peered":"2024-09-16T01:44:55.409768+0000","last_clean":"2024-09-16T01:44:55.409768+0000","last_became_active":"2024-09-16T01:44:52.534018+0000","last_became_peered":"2024-09-16T01:44:52.534018+0000","last_unstale":"2024-09-16T01:44:55.409768+0000","last_undegraded":"2024-09-16T01:44:55.409768+0000","last_fullsized":"2024-09-16T01:44:55.409768+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:26.024520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:54.035250+0000","last_change":"2024-09-16T01:44:52.518928+0000","last_active":"2024-09-16T01:44:54.035250+0000","last_peered":"2024-09-16T01:44:54.035250+0000","last_clean":"2024-09-16T01:44:54.035250+0000","last_became_active":"2024-09-16T01:44:52.518682+0000","last_became_peered":"2024-09-16T01:44:52.518682+0000","last_unstale":"2024-09-16T01:44:54.035250+0000","last_undegraded":"2024-09-16T01:44:54.035250+0000","last_fullsized":"2024-09-16T01:44:54.035250+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:54:54.638634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.410125+0000","last_change":"2024-09-16T01:44:52.517022+0000","last_active":"2024-09-16T01:44:55.410125+0000","last_peered":"2024-09-16T01:44:55.410125+0000","last_clean":"2024-09-16T01:44:55.410125+0000","last_became_active":"2024-09-16T01:44:52.516769+0000","last_became_peered":"2024-09-16T01:44:52.516769+0000","last_unstale":"2024-09-16T01:44:55.410125+0000","last_undegraded":"2024-09-16T01:44:55.410125+0000","last_fullsized":"2024-09-16T01:44:55.410125+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:45:57.188573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409836+0000","last_change":"2024-09-16T01:44:52.521772+0000","last_active":"2024-09-16T01:44:55.409836+0000","last_peered":"2024-09-16T01:44:55.409836+0000","last_clean":"2024-09-16T01:44:55.409836+0000","last_became_active":"2024-09-16T01:44:52.521514+0000","last_became_peered":"2024-09-16T01:44:52.521514+0000","last_unstale":"2024-09-16T01:44:55.409836+0000","last_undegraded":"2024-09-16T01:44:55.409836+0000","last_fullsized":"2024-09-16T01:44:55.409836+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:05.445566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.511209+0000","last_change":"2024-09-16T01:44:52.522256+0000","last_active":"2024-09-16T01:44:53.511209+0000","last_peered":"2024-09-16T01:44:53.511209+0000","last_clean":"2024-09-16T01:44:53.511209+0000","last_became_active":"2024-09-16T01:44:52.521982+0000","last_became_peered":"2024-09-16T01:44:52.521982+0000","last_unstale":"2024-09-16T01:44:53.511209+0000","last_undegraded":"2024-09-16T01:44:53.511209+0000","last_fullsized":"2024-09-16T01:44:53.511209+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:07:17.059934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409589+0000","last_change":"2024-09-16T01:44:52.522464+0000","last_active":"2024-09-16T01:44:55.409589+0000","last_peered":"2024-09-16T01:44:55.409589+0000","last_clean":"2024-09-16T01:44:55.409589+0000","last_became_active":"2024-09-16T01:44:52.522211+0000","last_became_peered":"2024-09-16T01:44:52.522211+0000","last_unstale":"2024-09-16T01:44:55.409589+0000","last_undegraded":"2024-09-16T01:44:55.409589+0000","last_fullsized":"2024-09-16T01:44:55.409589+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:05.727860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:54.035119+0000","last_change":"2024-09-16T01:44:52.517668+0000","last_active":"2024-09-16T01:44:54.035119+0000","last_peered":"2024-09-16T01:44:54.035119+0000","last_clean":"2024-09-16T01:44:54.035119+0000","last_became_active":"2024-09-16T01:44:52.517418+0000","last_became_peered":"2024-09-16T01:44:52.517418+0000","last_unstale":"2024-09-16T01:44:54.035119+0000","last_undegraded":"2024-09-16T01:44:54.035119+0000","last_fullsized":"2024-09-16T01:44:54.035119+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:13:59.043451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:54.034848+0000","last_change":"2024-09-16T01:44:52.517827+0000","last_active":"2024-09-16T01:44:54.034848+0000","last_peered":"2024-09-16T01:44:54.034848+0000","last_clean":"2024-09-16T01:44:54.034848+0000","last_became_active":"2024-09-16T01:44:52.517558+0000","last_became_peered":"2024-09-16T01:44:52.517558+0000","last_unstale":"2024-09-16T01:44:54.034848+0000","last_undegraded":"2024-09-16T01:44:54.034848+0000","last_fullsized":"2024-09-16T01:44:54.034848+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:54:00.714920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409647+0000","last_change":"2024-09-16T01:44:52.525939+0000","last_active":"2024-09-16T01:44:55.409647+0000","last_peered":"2024-09-16T01:44:55.409647+0000","last_clean":"2024-09-16T01:44:55.409647+0000","last_became_active":"2024-09-16T01:44:52.525599+0000","last_became_peered":"2024-09-16T01:44:52.525599+0000","last_unstale":"2024-09-16T01:44:55.409647+0000","last_undegraded":"2024-09-16T01:44:55.409647+0000","last_fullsized":"2024-09-16T01:44:55.409647+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:42:04.176536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.510800+0000","last_change":"2024-09-16T01:44:52.518677+0000","last_active":"2024-09-16T01:44:53.510800+0000","last_peered":"2024-09-16T01:44:53.510800+0000","last_clean":"2024-09-16T01:44:53.510800+0000","last_became_active":"2024-09-16T01:44:52.518389+0000","last_became_peered":"2024-09-16T01:44:52.518389+0000","last_unstale":"2024-09-16T01:44:53.510800+0000","last_undegraded":"2024-09-16T01:44:53.510800+0000","last_fullsized":"2024-09-16T01:44:53.510800+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:07:17.059934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:54.035218+0000","last_change":"2024-09-16T01:44:52.526290+0000","last_active":"2024-09-16T01:44:54.035218+0000","last_peered":"2024-09-16T01:44:54.035218+0000","last_clean":"2024-09-16T01:44:54.035218+0000","last_became_active":"2024-09-16T01:44:52.526049+0000","last_became_peered":"2024-09-16T01:44:52.526049+0000","last_unstale":"2024-09-16T01:44:54.035218+0000","last_undegraded":"2024-09-16T01:44:54.035218+0000","last_fullsized":"2024-09-16T01:44:54.035218+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:05.504546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409743+0000","last_change":"2024-09-16T01:44:52.528498+0000","last_active":"2024-09-16T01:44:55.409743+0000","last_peered":"2024-09-16T01:44:55.409743+0000","last_clean":"2024-09-16T01:44:55.409743+0000","last_became_active":"2024-09-16T01:44:52.527949+0000","last_became_peered":"2024-09-16T01:44:52.527949+0000","last_unstale":"2024-09-16T01:44:55.409743+0000","last_undegraded":"2024-09-16T01:44:55.409743+0000","last_fullsized":"2024-09-16T01:44:55.409743+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:47:56.247037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.511796+0000","last_change":"2024-09-16T01:44:52.531813+0000","last_active":"2024-09-16T01:44:53.511796+0000","last_peered":"2024-09-16T01:44:53.511796+0000","last_clean":"2024-09-16T01:44:53.511796+0000","last_became_active":"2024-09-16T01:44:52.531161+0000","last_became_peered":"2024-09-16T01:44:52.531161+0000","last_unstale":"2024-09-16T01:44:53.511796+0000","last_undegraded":"2024-09-16T01:44:53.511796+0000","last_fullsized":"2024-09-16T01:44:53.511796+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:54:00.714920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:54.034850+0000","last_change":"2024-09-16T01:44:52.520756+0000","last_active":"2024-09-16T01:44:54.034850+0000","last_peered":"2024-09-16T01:44:54.034850+0000","last_clean":"2024-09-16T01:44:54.034850+0000","last_became_active":"2024-09-16T01:44:52.520510+0000","last_became_peered":"2024-09-16T01:44:52.520510+0000","last_unstale":"2024-09-16T01:44:54.034850+0000","last_undegraded":"2024-09-16T01:44:54.034850+0000","last_fullsized":"2024-09-16T01:44:54.034850+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:54:28.443088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409407+0000","last_change":"2024-09-16T01:44:52.522380+0000","last_active":"2024-09-16T01:44:55.409407+0000","last_peered":"2024-09-16T01:44:55.409407+0000","last_clean":"2024-09-16T01:44:55.409407+0000","last_became_active":"2024-09-16T01:44:52.522112+0000","last_became_peered":"2024-09-16T01:44:52.522112+0000","last_unstale":"2024-09-16T01:44:55.409407+0000","last_undegraded":"2024-09-16T01:44:55.409407+0000","last_fullsized":"2024-09-16T01:44:55.409407+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:53:31.772509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.410168+0000","last_change":"2024-09-16T01:44:52.521499+0000","last_active":"2024-09-16T01:44:55.410168+0000","last_peered":"2024-09-16T01:44:55.410168+0000","last_clean":"2024-09-16T01:44:55.410168+0000","last_became_active":"2024-09-16T01:44:52.521253+0000","last_became_peered":"2024-09-16T01:44:52.521253+0000","last_unstale":"2024-09-16T01:44:55.410168+0000","last_undegraded":"2024-09-16T01:44:55.410168+0000","last_fullsized":"2024-09-16T01:44:55.410168+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:51:33.777392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.394866+0000","last_change":"2024-09-16T01:44:52.520919+0000","last_active":"2024-09-16T01:44:55.394866+0000","last_peered":"2024-09-16T01:44:55.394866+0000","last_clean":"2024-09-16T01:44:55.394866+0000","last_became_active":"2024-09-16T01:44:52.520207+0000","last_became_peered":"2024-09-16T01:44:52.520207+0000","last_unstale":"2024-09-16T01:44:55.394866+0000","last_undegraded":"2024-09-16T01:44:55.394866+0000","last_fullsized":"2024-09-16T01:44:55.394866+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:15:08.977917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.410188+0000","last_change":"2024-09-16T01:44:52.522831+0000","last_active":"2024-09-16T01:44:55.410188+0000","last_peered":"2024-09-16T01:44:55.410188+0000","last_clean":"2024-09-16T01:44:55.410188+0000","last_became_active":"2024-09-16T01:44:52.522522+0000","last_became_peered":"2024-09-16T01:44:52.522522+0000","last_unstale":"2024-09-16T01:44:55.410188+0000","last_undegraded":"2024-09-16T01:44:55.410188+0000","last_fullsized":"2024-09-16T01:44:55.410188+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:01:10.818442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"137'4","reported_seq":18,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:54.035190+0000","last_change":"2024-09-16T01:44:51.523626+0000","last_active":"2024-09-16T01:44:54.035190+0000","last_peered":"2024-09-16T01:44:54.035190+0000","last_clean":"2024-09-16T01:44:54.035190+0000","last_became_active":"2024-09-16T01:44:51.523382+0000","last_became_peered":"2024-09-16T01:44:51.523382+0000","last_unstale":"2024-09-16T01:44:54.035190+0000","last_undegraded":"2024-09-16T01:44:54.035190+0000","last_fullsized":"2024-09-16T01:44:54.035190+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:25:23.460308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.410127+0000","last_change":"2024-09-16T01:44:52.522303+0000","last_active":"2024-09-16T01:44:55.410127+0000","last_peered":"2024-09-16T01:44:55.410127+0000","last_clean":"2024-09-16T01:44:55.410127+0000","last_became_active":"2024-09-16T01:44:52.522057+0000","last_became_peered":"2024-09-16T01:44:52.522057+0000","last_unstale":"2024-09-16T01:44:55.410127+0000","last_undegraded":"2024-09-16T01:44:55.410127+0000","last_fullsized":"2024-09-16T01:44:55.410127+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:03.626997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"137'1","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409671+0000","last_change":"2024-09-16T01:44:51.515442+0000","last_active":"2024-09-16T01:44:55.409671+0000","last_peered":"2024-09-16T01:44:55.409671+0000","last_clean":"2024-09-16T01:44:55.409671+0000","last_became_active":"2024-09-16T01:44:51.515133+0000","last_became_peered":"2024-09-16T01:44:51.515133+0000","last_unstale":"2024-09-16T01:44:55.409671+0000","last_undegraded":"2024-09-16T01:44:55.409671+0000","last_fullsized":"2024-09-16T01:44:55.409671+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:58:13.470773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.395040+0000","last_change":"2024-09-16T01:44:52.523652+0000","last_active":"2024-09-16T01:44:55.395040+0000","last_peered":"2024-09-16T01:44:55.395040+0000","last_clean":"2024-09-16T01:44:55.395040+0000","last_became_active":"2024-09-16T01:44:52.523387+0000","last_became_peered":"2024-09-16T01:44:52.523387+0000","last_unstale":"2024-09-16T01:44:55.395040+0000","last_undegraded":"2024-09-16T01:44:55.395040+0000","last_fullsized":"2024-09-16T01:44:55.395040+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:18:03.085854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"137'2","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:54.034957+0000","last_change":"2024-09-16T01:44:51.526219+0000","last_active":"2024-09-16T01:44:54.034957+0000","last_peered":"2024-09-16T01:44:54.034957+0000","last_clean":"2024-09-16T01:44:54.034957+0000","last_became_active":"2024-09-16T01:44:51.525924+0000","last_became_peered":"2024-09-16T01:44:51.525924+0000","last_unstale":"2024-09-16T01:44:54.034957+0000","last_undegraded":"2024-09-16T01:44:54.034957+0000","last_fullsized":"2024-09-16T01:44:54.034957+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:22:56.611656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.511137+0000","last_change":"2024-09-16T01:44:52.525376+0000","last_active":"2024-09-16T01:44:53.511137+0000","last_peered":"2024-09-16T01:44:53.511137+0000","last_clean":"2024-09-16T01:44:53.511137+0000","last_became_active":"2024-09-16T01:44:52.525100+0000","last_became_peered":"2024-09-16T01:44:52.525100+0000","last_unstale":"2024-09-16T01:44:53.511137+0000","last_undegraded":"2024-09-16T01:44:53.511137+0000","last_fullsized":"2024-09-16T01:44:53.511137+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:51:33.777392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"137'2","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:54.034821+0000","last_change":"2024-09-16T01:44:51.521753+0000","last_active":"2024-09-16T01:44:54.034821+0000","last_peered":"2024-09-16T01:44:54.034821+0000","last_clean":"2024-09-16T01:44:54.034821+0000","last_became_active":"2024-09-16T01:44:51.521101+0000","last_became_peered":"2024-09-16T01:44:51.521101+0000","last_unstale":"2024-09-16T01:44:54.034821+0000","last_undegraded":"2024-09-16T01:44:54.034821+0000","last_fullsized":"2024-09-16T01:44:54.034821+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:10:11.718212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.512009+0000","last_change":"2024-09-16T01:44:52.524561+0000","last_active":"2024-09-16T01:44:53.512009+0000","last_peered":"2024-09-16T01:44:53.512009+0000","last_clean":"2024-09-16T01:44:53.512009+0000","last_became_active":"2024-09-16T01:44:52.524305+0000","last_became_peered":"2024-09-16T01:44:52.524305+0000","last_unstale":"2024-09-16T01:44:53.512009+0000","last_undegraded":"2024-09-16T01:44:53.512009+0000","last_fullsized":"2024-09-16T01:44:53.512009+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:28:23.484932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"137'1","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.394921+0000","last_change":"2024-09-16T01:44:51.513353+0000","last_active":"2024-09-16T01:44:55.394921+0000","last_peered":"2024-09-16T01:44:55.394921+0000","last_clean":"2024-09-16T01:44:55.394921+0000","last_became_active":"2024-09-16T01:44:51.513049+0000","last_became_peered":"2024-09-16T01:44:51.513049+0000","last_unstale":"2024-09-16T01:44:55.394921+0000","last_undegraded":"2024-09-16T01:44:55.394921+0000","last_fullsized":"2024-09-16T01:44:55.394921+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:35.808937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409663+0000","last_change":"2024-09-16T01:44:52.523810+0000","last_active":"2024-09-16T01:44:55.409663+0000","last_peered":"2024-09-16T01:44:55.409663+0000","last_clean":"2024-09-16T01:44:55.409663+0000","last_became_active":"2024-09-16T01:44:52.523487+0000","last_became_peered":"2024-09-16T01:44:52.523487+0000","last_unstale":"2024-09-16T01:44:55.409663+0000","last_undegraded":"2024-09-16T01:44:55.409663+0000","last_fullsized":"2024-09-16T01:44:55.409663+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:07:56.704658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"137'1","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:54.034896+0000","last_change":"2024-09-16T01:44:51.520021+0000","last_active":"2024-09-16T01:44:54.034896+0000","last_peered":"2024-09-16T01:44:54.034896+0000","last_clean":"2024-09-16T01:44:54.034896+0000","last_became_active":"2024-09-16T01:44:51.519734+0000","last_became_peered":"2024-09-16T01:44:51.519734+0000","last_unstale":"2024-09-16T01:44:54.034896+0000","last_undegraded":"2024-09-16T01:44:54.034896+0000","last_fullsized":"2024-09-16T01:44:54.034896+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:54:22.805176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409380+0000","last_change":"2024-09-16T01:44:52.523883+0000","last_active":"2024-09-16T01:44:55.409380+0000","last_peered":"2024-09-16T01:44:55.409380+0000","last_clean":"2024-09-16T01:44:55.409380+0000","last_became_active":"2024-09-16T01:44:52.523630+0000","last_became_peered":"2024-09-16T01:44:52.523630+0000","last_unstale":"2024-09-16T01:44:55.409380+0000","last_undegraded":"2024-09-16T01:44:55.409380+0000","last_fullsized":"2024-09-16T01:44:55.409380+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:19:18.531357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.511492+0000","last_change":"2024-09-16T01:44:51.516445+0000","last_active":"2024-09-16T01:44:53.511492+0000","last_peered":"2024-09-16T01:44:53.511492+0000","last_clean":"2024-09-16T01:44:53.511492+0000","last_became_active":"2024-09-16T01:44:51.516136+0000","last_became_peered":"2024-09-16T01:44:51.516136+0000","last_unstale":"2024-09-16T01:44:53.511492+0000","last_undegraded":"2024-09-16T01:44:53.511492+0000","last_fullsized":"2024-09-16T01:44:53.511492+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:17:04.206823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.510513+0000","last_change":"2024-09-16T01:44:52.525275+0000","last_active":"2024-09-16T01:44:53.510513+0000","last_peered":"2024-09-16T01:44:53.510513+0000","last_clean":"2024-09-16T01:44:53.510513+0000","last_became_active":"2024-09-16T01:44:52.525010+0000","last_became_peered":"2024-09-16T01:44:52.525010+0000","last_unstale":"2024-09-16T01:44:53.510513+0000","last_undegraded":"2024-09-16T01:44:53.510513+0000","last_fullsized":"2024-09-16T01:44:53.510513+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:37:23.230828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.394775+0000","last_change":"2024-09-16T01:44:51.518492+0000","last_active":"2024-09-16T01:44:55.394775+0000","last_peered":"2024-09-16T01:44:55.394775+0000","last_clean":"2024-09-16T01:44:55.394775+0000","last_became_active":"2024-09-16T01:44:51.518221+0000","last_became_peered":"2024-09-16T01:44:51.518221+0000","last_unstale":"2024-09-16T01:44:55.394775+0000","last_undegraded":"2024-09-16T01:44:55.394775+0000","last_fullsized":"2024-09-16T01:44:55.394775+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:53:30.770105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:54.035314+0000","last_change":"2024-09-16T01:44:52.525810+0000","last_active":"2024-09-16T01:44:54.035314+0000","last_peered":"2024-09-16T01:44:54.035314+0000","last_clean":"2024-09-16T01:44:54.035314+0000","last_became_active":"2024-09-16T01:44:52.525482+0000","last_became_peered":"2024-09-16T01:44:52.525482+0000","last_unstale":"2024-09-16T01:44:54.035314+0000","last_undegraded":"2024-09-16T01:44:54.035314+0000","last_fullsized":"2024-09-16T01:44:54.035314+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:36:36.587163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.395287+0000","last_change":"2024-09-16T01:44:51.513970+0000","last_active":"2024-09-16T01:44:55.395287+0000","last_peered":"2024-09-16T01:44:55.395287+0000","last_clean":"2024-09-16T01:44:55.395287+0000","last_became_active":"2024-09-16T01:44:51.513712+0000","last_became_peered":"2024-09-16T01:44:51.513712+0000","last_unstale":"2024-09-16T01:44:55.395287+0000","last_undegraded":"2024-09-16T01:44:55.395287+0000","last_fullsized":"2024-09-16T01:44:55.395287+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:32.101261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.510607+0000","last_change":"2024-09-16T01:44:52.519517+0000","last_active":"2024-09-16T01:44:53.510607+0000","last_peered":"2024-09-16T01:44:53.510607+0000","last_clean":"2024-09-16T01:44:53.510607+0000","last_became_active":"2024-09-16T01:44:52.519271+0000","last_became_peered":"2024-09-16T01:44:52.519271+0000","last_unstale":"2024-09-16T01:44:53.510607+0000","last_undegraded":"2024-09-16T01:44:53.510607+0000","last_fullsized":"2024-09-16T01:44:53.510607+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:45:57.188573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409951+0000","last_change":"2024-09-16T01:44:51.525079+0000","last_active":"2024-09-16T01:44:55.409951+0000","last_peered":"2024-09-16T01:44:55.409951+0000","last_clean":"2024-09-16T01:44:55.409951+0000","last_became_active":"2024-09-16T01:44:51.524798+0000","last_became_peered":"2024-09-16T01:44:51.524798+0000","last_unstale":"2024-09-16T01:44:55.409951+0000","last_undegraded":"2024-09-16T01:44:55.409951+0000","last_fullsized":"2024-09-16T01:44:55.409951+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:28.711338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.511898+0000","last_change":"2024-09-16T01:44:52.518198+0000","last_active":"2024-09-16T01:44:53.511898+0000","last_peered":"2024-09-16T01:44:53.511898+0000","last_clean":"2024-09-16T01:44:53.511898+0000","last_became_active":"2024-09-16T01:44:52.517865+0000","last_became_peered":"2024-09-16T01:44:52.517865+0000","last_unstale":"2024-09-16T01:44:53.511898+0000","last_undegraded":"2024-09-16T01:44:53.511898+0000","last_fullsized":"2024-09-16T01:44:53.511898+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:54:28.443088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"137'3","reported_seq":17,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409878+0000","last_change":"2024-09-16T01:44:51.514908+0000","last_active":"2024-09-16T01:44:55.409878+0000","last_peered":"2024-09-16T01:44:55.409878+0000","last_clean":"2024-09-16T01:44:55.409878+0000","last_became_active":"2024-09-16T01:44:51.514599+0000","last_became_peered":"2024-09-16T01:44:51.514599+0000","last_unstale":"2024-09-16T01:44:55.409878+0000","last_undegraded":"2024-09-16T01:44:55.409878+0000","last_fullsized":"2024-09-16T01:44:55.409878+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:45:01.912557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409836+0000","last_change":"2024-09-16T01:44:52.536971+0000","last_active":"2024-09-16T01:44:55.409836+0000","last_peered":"2024-09-16T01:44:55.409836+0000","last_clean":"2024-09-16T01:44:55.409836+0000","last_became_active":"2024-09-16T01:44:52.536661+0000","last_became_peered":"2024-09-16T01:44:52.536661+0000","last_unstale":"2024-09-16T01:44:55.409836+0000","last_undegraded":"2024-09-16T01:44:55.409836+0000","last_fullsized":"2024-09-16T01:44:55.409836+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:07:12.610889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"137'6","reported_seq":20,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.511226+0000","last_change":"2024-09-16T01:44:51.523419+0000","last_active":"2024-09-16T01:44:53.511226+0000","last_peered":"2024-09-16T01:44:53.511226+0000","last_clean":"2024-09-16T01:44:53.511226+0000","last_became_active":"2024-09-16T01:44:51.523189+0000","last_became_peered":"2024-09-16T01:44:51.523189+0000","last_unstale":"2024-09-16T01:44:53.511226+0000","last_undegraded":"2024-09-16T01:44:53.511226+0000","last_fullsized":"2024-09-16T01:44:53.511226+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:54:22.805176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.410082+0000","last_change":"2024-09-16T01:44:52.526130+0000","last_active":"2024-09-16T01:44:55.410082+0000","last_peered":"2024-09-16T01:44:55.410082+0000","last_clean":"2024-09-16T01:44:55.410082+0000","last_became_active":"2024-09-16T01:44:52.525683+0000","last_became_peered":"2024-09-16T01:44:52.525683+0000","last_unstale":"2024-09-16T01:44:55.410082+0000","last_undegraded":"2024-09-16T01:44:55.410082+0000","last_fullsized":"2024-09-16T01:44:55.410082+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:14:22.817576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"137'2","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409650+0000","last_change":"2024-09-16T01:44:51.514238+0000","last_active":"2024-09-16T01:44:55.409650+0000","last_peered":"2024-09-16T01:44:55.409650+0000","last_clean":"2024-09-16T01:44:55.409650+0000","last_became_active":"2024-09-16T01:44:51.513976+0000","last_became_peered":"2024-09-16T01:44:51.513976+0000","last_unstale":"2024-09-16T01:44:55.409650+0000","last_undegraded":"2024-09-16T01:44:55.409650+0000","last_fullsized":"2024-09-16T01:44:55.409650+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:06.828246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.394982+0000","last_change":"2024-09-16T01:44:52.520925+0000","last_active":"2024-09-16T01:44:55.394982+0000","last_peered":"2024-09-16T01:44:55.394982+0000","last_clean":"2024-09-16T01:44:55.394982+0000","last_became_active":"2024-09-16T01:44:52.520649+0000","last_became_peered":"2024-09-16T01:44:52.520649+0000","last_unstale":"2024-09-16T01:44:55.394982+0000","last_undegraded":"2024-09-16T01:44:55.394982+0000","last_fullsized":"2024-09-16T01:44:55.394982+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:11:00.141183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409830+0000","last_change":"2024-09-16T01:44:51.517978+0000","last_active":"2024-09-16T01:44:55.409830+0000","last_peered":"2024-09-16T01:44:55.409830+0000","last_clean":"2024-09-16T01:44:55.409830+0000","last_became_active":"2024-09-16T01:44:51.517696+0000","last_became_peered":"2024-09-16T01:44:51.517696+0000","last_unstale":"2024-09-16T01:44:55.409830+0000","last_undegraded":"2024-09-16T01:44:55.409830+0000","last_fullsized":"2024-09-16T01:44:55.409830+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:37:17.079391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.510750+0000","last_change":"2024-09-16T01:44:52.520251+0000","last_active":"2024-09-16T01:44:53.510750+0000","last_peered":"2024-09-16T01:44:53.510750+0000","last_clean":"2024-09-16T01:44:53.510750+0000","last_became_active":"2024-09-16T01:44:52.519948+0000","last_became_peered":"2024-09-16T01:44:52.519948+0000","last_unstale":"2024-09-16T01:44:53.510750+0000","last_undegraded":"2024-09-16T01:44:53.510750+0000","last_fullsized":"2024-09-16T01:44:53.510750+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:19:18.531357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.510570+0000","last_change":"2024-09-16T01:44:51.517473+0000","last_active":"2024-09-16T01:44:53.510570+0000","last_peered":"2024-09-16T01:44:53.510570+0000","last_clean":"2024-09-16T01:44:53.510570+0000","last_became_active":"2024-09-16T01:44:51.516338+0000","last_became_peered":"2024-09-16T01:44:51.516338+0000","last_unstale":"2024-09-16T01:44:53.510570+0000","last_undegraded":"2024-09-16T01:44:53.510570+0000","last_fullsized":"2024-09-16T01:44:53.510570+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:28.711338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409373+0000","last_change":"2024-09-16T01:44:52.529222+0000","last_active":"2024-09-16T01:44:55.409373+0000","last_peered":"2024-09-16T01:44:55.409373+0000","last_clean":"2024-09-16T01:44:55.409373+0000","last_became_active":"2024-09-16T01:44:52.528758+0000","last_became_peered":"2024-09-16T01:44:52.528758+0000","last_unstale":"2024-09-16T01:44:55.409373+0000","last_undegraded":"2024-09-16T01:44:55.409373+0000","last_fullsized":"2024-09-16T01:44:55.409373+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:05:36.811341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"137'8","reported_seq":22,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.394929+0000","last_change":"2024-09-16T01:44:51.513166+0000","last_active":"2024-09-16T01:44:55.394929+0000","last_peered":"2024-09-16T01:44:55.394929+0000","last_clean":"2024-09-16T01:44:55.394929+0000","last_became_active":"2024-09-16T01:44:51.512880+0000","last_became_peered":"2024-09-16T01:44:51.512880+0000","last_unstale":"2024-09-16T01:44:55.394929+0000","last_undegraded":"2024-09-16T01:44:55.394929+0000","last_fullsized":"2024-09-16T01:44:55.394929+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:05:48.362276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.510966+0000","last_change":"2024-09-16T01:44:52.524728+0000","last_active":"2024-09-16T01:44:53.510966+0000","last_peered":"2024-09-16T01:44:53.510966+0000","last_clean":"2024-09-16T01:44:53.510966+0000","last_became_active":"2024-09-16T01:44:52.524305+0000","last_became_peered":"2024-09-16T01:44:52.524305+0000","last_unstale":"2024-09-16T01:44:53.510966+0000","last_undegraded":"2024-09-16T01:44:53.510966+0000","last_fullsized":"2024-09-16T01:44:53.510966+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:58:59.379525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"137'2","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.510880+0000","last_change":"2024-09-16T01:44:51.517156+0000","last_active":"2024-09-16T01:44:53.510880+0000","last_peered":"2024-09-16T01:44:53.510880+0000","last_clean":"2024-09-16T01:44:53.510880+0000","last_became_active":"2024-09-16T01:44:51.516041+0000","last_became_peered":"2024-09-16T01:44:51.516041+0000","last_unstale":"2024-09-16T01:44:53.510880+0000","last_undegraded":"2024-09-16T01:44:53.510880+0000","last_fullsized":"2024-09-16T01:44:53.510880+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:23:27.981051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.512038+0000","last_change":"2024-09-16T01:44:52.518107+0000","last_active":"2024-09-16T01:44:53.512038+0000","last_peered":"2024-09-16T01:44:53.512038+0000","last_clean":"2024-09-16T01:44:53.512038+0000","last_became_active":"2024-09-16T01:44:52.517746+0000","last_became_peered":"2024-09-16T01:44:52.517746+0000","last_unstale":"2024-09-16T01:44:53.512038+0000","last_undegraded":"2024-09-16T01:44:53.512038+0000","last_fullsized":"2024-09-16T01:44:53.512038+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:57:53.722086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"137'1","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.511082+0000","last_change":"2024-09-16T01:44:51.514633+0000","last_active":"2024-09-16T01:44:53.511082+0000","last_peered":"2024-09-16T01:44:53.511082+0000","last_clean":"2024-09-16T01:44:53.511082+0000","last_became_active":"2024-09-16T01:44:51.514347+0000","last_became_peered":"2024-09-16T01:44:51.514347+0000","last_unstale":"2024-09-16T01:44:53.511082+0000","last_undegraded":"2024-09-16T01:44:53.511082+0000","last_fullsized":"2024-09-16T01:44:53.511082+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:25:23.460308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409884+0000","last_change":"2024-09-16T01:44:52.528221+0000","last_active":"2024-09-16T01:44:55.409884+0000","last_peered":"2024-09-16T01:44:55.409884+0000","last_clean":"2024-09-16T01:44:55.409884+0000","last_became_active":"2024-09-16T01:44:52.527434+0000","last_became_peered":"2024-09-16T01:44:52.527434+0000","last_unstale":"2024-09-16T01:44:55.409884+0000","last_undegraded":"2024-09-16T01:44:55.409884+0000","last_fullsized":"2024-09-16T01:44:55.409884+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:29:51.689510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:54.034928+0000","last_change":"2024-09-16T01:44:51.521681+0000","last_active":"2024-09-16T01:44:54.034928+0000","last_peered":"2024-09-16T01:44:54.034928+0000","last_clean":"2024-09-16T01:44:54.034928+0000","last_became_active":"2024-09-16T01:44:51.520744+0000","last_became_peered":"2024-09-16T01:44:51.520744+0000","last_unstale":"2024-09-16T01:44:54.034928+0000","last_undegraded":"2024-09-16T01:44:54.034928+0000","last_fullsized":"2024-09-16T01:44:54.034928+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:48:51.682039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409583+0000","last_change":"2024-09-16T01:44:52.524490+0000","last_active":"2024-09-16T01:44:55.409583+0000","last_peered":"2024-09-16T01:44:55.409583+0000","last_clean":"2024-09-16T01:44:55.409583+0000","last_became_active":"2024-09-16T01:44:52.524236+0000","last_became_peered":"2024-09-16T01:44:52.524236+0000","last_unstale":"2024-09-16T01:44:55.409583+0000","last_undegraded":"2024-09-16T01:44:55.409583+0000","last_fullsized":"2024-09-16T01:44:55.409583+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:29:30.647443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"137'2","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:54.034845+0000","last_change":"2024-09-16T01:44:51.524064+0000","last_active":"2024-09-16T01:44:54.034845+0000","last_peered":"2024-09-16T01:44:54.034845+0000","last_clean":"2024-09-16T01:44:54.034845+0000","last_became_active":"2024-09-16T01:44:51.523799+0000","last_became_peered":"2024-09-16T01:44:51.523799+0000","last_unstale":"2024-09-16T01:44:54.034845+0000","last_undegraded":"2024-09-16T01:44:54.034845+0000","last_fullsized":"2024-09-16T01:44:54.034845+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:24.176260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409675+0000","last_change":"2024-09-16T01:44:52.518664+0000","last_active":"2024-09-16T01:44:55.409675+0000","last_peered":"2024-09-16T01:44:55.409675+0000","last_clean":"2024-09-16T01:44:55.409675+0000","last_became_active":"2024-09-16T01:44:52.518409+0000","last_became_peered":"2024-09-16T01:44:52.518409+0000","last_unstale":"2024-09-16T01:44:55.409675+0000","last_undegraded":"2024-09-16T01:44:55.409675+0000","last_fullsized":"2024-09-16T01:44:55.409675+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:05:49.871850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.511951+0000","last_change":"2024-09-16T01:44:51.519279+0000","last_active":"2024-09-16T01:44:53.511951+0000","last_peered":"2024-09-16T01:44:53.511951+0000","last_clean":"2024-09-16T01:44:53.511951+0000","last_became_active":"2024-09-16T01:44:51.519002+0000","last_became_peered":"2024-09-16T01:44:51.519002+0000","last_unstale":"2024-09-16T01:44:53.511951+0000","last_undegraded":"2024-09-16T01:44:53.511951+0000","last_fullsized":"2024-09-16T01:44:53.511951+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:02:23.389323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409344+0000","last_change":"2024-09-16T01:44:52.524079+0000","last_active":"2024-09-16T01:44:55.409344+0000","last_peered":"2024-09-16T01:44:55.409344+0000","last_clean":"2024-09-16T01:44:55.409344+0000","last_became_active":"2024-09-16T01:44:52.523830+0000","last_became_peered":"2024-09-16T01:44:52.523830+0000","last_unstale":"2024-09-16T01:44:55.409344+0000","last_undegraded":"2024-09-16T01:44:55.409344+0000","last_fullsized":"2024-09-16T01:44:55.409344+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:33.103665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.511922+0000","last_change":"2024-09-16T01:44:51.517843+0000","last_active":"2024-09-16T01:44:53.511922+0000","last_peered":"2024-09-16T01:44:53.511922+0000","last_clean":"2024-09-16T01:44:53.511922+0000","last_became_active":"2024-09-16T01:44:51.517548+0000","last_became_peered":"2024-09-16T01:44:51.517548+0000","last_unstale":"2024-09-16T01:44:53.511922+0000","last_undegraded":"2024-09-16T01:44:53.511922+0000","last_fullsized":"2024-09-16T01:44:53.511922+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:41:26.888889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.511296+0000","last_change":"2024-09-16T01:44:52.519374+0000","last_active":"2024-09-16T01:44:53.511296+0000","last_peered":"2024-09-16T01:44:53.511296+0000","last_clean":"2024-09-16T01:44:53.511296+0000","last_became_active":"2024-09-16T01:44:52.519018+0000","last_became_peered":"2024-09-16T01:44:52.519018+0000","last_unstale":"2024-09-16T01:44:53.511296+0000","last_undegraded":"2024-09-16T01:44:53.511296+0000","last_fullsized":"2024-09-16T01:44:53.511296+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:58:34.846229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:54.034982+0000","last_change":"2024-09-16T01:44:51.523531+0000","last_active":"2024-09-16T01:44:54.034982+0000","last_peered":"2024-09-16T01:44:54.034982+0000","last_clean":"2024-09-16T01:44:54.034982+0000","last_became_active":"2024-09-16T01:44:51.520047+0000","last_became_peered":"2024-09-16T01:44:51.520047+0000","last_unstale":"2024-09-16T01:44:54.034982+0000","last_undegraded":"2024-09-16T01:44:54.034982+0000","last_fullsized":"2024-09-16T01:44:54.034982+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:19:17.528953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:54.035319+0000","last_change":"2024-09-16T01:44:52.528582+0000","last_active":"2024-09-16T01:44:54.035319+0000","last_peered":"2024-09-16T01:44:54.035319+0000","last_clean":"2024-09-16T01:44:54.035319+0000","last_became_active":"2024-09-16T01:44:52.528300+0000","last_became_peered":"2024-09-16T01:44:52.528300+0000","last_unstale":"2024-09-16T01:44:54.035319+0000","last_undegraded":"2024-09-16T01:44:54.035319+0000","last_fullsized":"2024-09-16T01:44:54.035319+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:23:29.680341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"137'2","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.511780+0000","last_change":"2024-09-16T01:44:51.513903+0000","last_active":"2024-09-16T01:44:53.511780+0000","last_peered":"2024-09-16T01:44:53.511780+0000","last_clean":"2024-09-16T01:44:53.511780+0000","last_became_active":"2024-09-16T01:44:51.513583+0000","last_became_peered":"2024-09-16T01:44:51.513583+0000","last_unstale":"2024-09-16T01:44:53.511780+0000","last_undegraded":"2024-09-16T01:44:53.511780+0000","last_fullsized":"2024-09-16T01:44:53.511780+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:26:45.936557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.395291+0000","last_change":"2024-09-16T01:44:52.520973+0000","last_active":"2024-09-16T01:44:55.395291+0000","last_peered":"2024-09-16T01:44:55.395291+0000","last_clean":"2024-09-16T01:44:55.395291+0000","last_became_active":"2024-09-16T01:44:52.520354+0000","last_became_peered":"2024-09-16T01:44:52.520354+0000","last_unstale":"2024-09-16T01:44:55.395291+0000","last_undegraded":"2024-09-16T01:44:55.395291+0000","last_fullsized":"2024-09-16T01:44:55.395291+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:24:51.038867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"137'1","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:54.035377+0000","last_change":"2024-09-16T01:44:51.525007+0000","last_active":"2024-09-16T01:44:54.035377+0000","last_peered":"2024-09-16T01:44:54.035377+0000","last_clean":"2024-09-16T01:44:54.035377+0000","last_became_active":"2024-09-16T01:44:51.524746+0000","last_became_peered":"2024-09-16T01:44:51.524746+0000","last_unstale":"2024-09-16T01:44:54.035377+0000","last_undegraded":"2024-09-16T01:44:54.035377+0000","last_fullsized":"2024-09-16T01:44:54.035377+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:24:41.046304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.395136+0000","last_change":"2024-09-16T01:44:52.518139+0000","last_active":"2024-09-16T01:44:55.395136+0000","last_peered":"2024-09-16T01:44:55.395136+0000","last_clean":"2024-09-16T01:44:55.395136+0000","last_became_active":"2024-09-16T01:44:52.517889+0000","last_became_peered":"2024-09-16T01:44:52.517889+0000","last_unstale":"2024-09-16T01:44:55.395136+0000","last_undegraded":"2024-09-16T01:44:55.395136+0000","last_fullsized":"2024-09-16T01:44:55.395136+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:59:34.704665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:54.035065+0000","last_change":"2024-09-16T01:44:51.521311+0000","last_active":"2024-09-16T01:44:54.035065+0000","last_peered":"2024-09-16T01:44:54.035065+0000","last_clean":"2024-09-16T01:44:54.035065+0000","last_became_active":"2024-09-16T01:44:51.520876+0000","last_became_peered":"2024-09-16T01:44:51.520876+0000","last_unstale":"2024-09-16T01:44:54.035065+0000","last_undegraded":"2024-09-16T01:44:54.035065+0000","last_fullsized":"2024-09-16T01:44:54.035065+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:19:49.416564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.511027+0000","last_change":"2024-09-16T01:44:52.531441+0000","last_active":"2024-09-16T01:44:53.511027+0000","last_peered":"2024-09-16T01:44:53.511027+0000","last_clean":"2024-09-16T01:44:53.511027+0000","last_became_active":"2024-09-16T01:44:52.531139+0000","last_became_peered":"2024-09-16T01:44:52.531139+0000","last_unstale":"2024-09-16T01:44:53.511027+0000","last_undegraded":"2024-09-16T01:44:53.511027+0000","last_fullsized":"2024-09-16T01:44:53.511027+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:38:51.745834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.511856+0000","last_change":"2024-09-16T01:44:51.515920+0000","last_active":"2024-09-16T01:44:53.511856+0000","last_peered":"2024-09-16T01:44:53.511856+0000","last_clean":"2024-09-16T01:44:53.511856+0000","last_became_active":"2024-09-16T01:44:51.515647+0000","last_became_peered":"2024-09-16T01:44:51.515647+0000","last_unstale":"2024-09-16T01:44:53.511856+0000","last_undegraded":"2024-09-16T01:44:53.511856+0000","last_fullsized":"2024-09-16T01:44:53.511856+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:22:56.611656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409766+0000","last_change":"2024-09-16T01:44:52.529616+0000","last_active":"2024-09-16T01:44:55.409766+0000","last_peered":"2024-09-16T01:44:55.409766+0000","last_clean":"2024-09-16T01:44:55.409766+0000","last_became_active":"2024-09-16T01:44:52.528923+0000","last_became_peered":"2024-09-16T01:44:52.528923+0000","last_unstale":"2024-09-16T01:44:55.409766+0000","last_undegraded":"2024-09-16T01:44:55.409766+0000","last_fullsized":"2024-09-16T01:44:55.409766+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:44:17.101959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:54.034909+0000","last_change":"2024-09-16T01:44:51.524660+0000","last_active":"2024-09-16T01:44:54.034909+0000","last_peered":"2024-09-16T01:44:54.034909+0000","last_clean":"2024-09-16T01:44:54.034909+0000","last_became_active":"2024-09-16T01:44:51.524396+0000","last_became_peered":"2024-09-16T01:44:51.524396+0000","last_unstale":"2024-09-16T01:44:54.034909+0000","last_undegraded":"2024-09-16T01:44:54.034909+0000","last_fullsized":"2024-09-16T01:44:54.034909+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:17:04.206823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.510674+0000","last_change":"2024-09-16T01:44:52.519507+0000","last_active":"2024-09-16T01:44:53.510674+0000","last_peered":"2024-09-16T01:44:53.510674+0000","last_clean":"2024-09-16T01:44:53.510674+0000","last_became_active":"2024-09-16T01:44:52.519140+0000","last_became_peered":"2024-09-16T01:44:52.519140+0000","last_unstale":"2024-09-16T01:44:53.510674+0000","last_undegraded":"2024-09-16T01:44:53.510674+0000","last_fullsized":"2024-09-16T01:44:53.510674+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:03.626997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.510390+0000","last_change":"2024-09-16T01:44:51.519274+0000","last_active":"2024-09-16T01:44:53.510390+0000","last_peered":"2024-09-16T01:44:53.510390+0000","last_clean":"2024-09-16T01:44:53.510390+0000","last_became_active":"2024-09-16T01:44:51.519022+0000","last_became_peered":"2024-09-16T01:44:51.519022+0000","last_unstale":"2024-09-16T01:44:53.510390+0000","last_undegraded":"2024-09-16T01:44:53.510390+0000","last_fullsized":"2024-09-16T01:44:53.510390+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:27:24.176260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409572+0000","last_change":"2024-09-16T01:44:52.529058+0000","last_active":"2024-09-16T01:44:55.409572+0000","last_peered":"2024-09-16T01:44:55.409572+0000","last_clean":"2024-09-16T01:44:55.409572+0000","last_became_active":"2024-09-16T01:44:52.528742+0000","last_became_peered":"2024-09-16T01:44:52.528742+0000","last_unstale":"2024-09-16T01:44:55.409572+0000","last_undegraded":"2024-09-16T01:44:55.409572+0000","last_fullsized":"2024-09-16T01:44:55.409572+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:58:05.828224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.410149+0000","last_change":"2024-09-16T01:44:51.516480+0000","last_active":"2024-09-16T01:44:55.410149+0000","last_peered":"2024-09-16T01:44:55.410149+0000","last_clean":"2024-09-16T01:44:55.410149+0000","last_became_active":"2024-09-16T01:44:51.516203+0000","last_became_peered":"2024-09-16T01:44:51.516203+0000","last_unstale":"2024-09-16T01:44:55.410149+0000","last_undegraded":"2024-09-16T01:44:55.410149+0000","last_fullsized":"2024-09-16T01:44:55.410149+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:54:22.805176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:53.511442+0000","last_change":"2024-09-16T01:44:52.531482+0000","last_active":"2024-09-16T01:44:53.511442+0000","last_peered":"2024-09-16T01:44:53.511442+0000","last_clean":"2024-09-16T01:44:53.511442+0000","last_became_active":"2024-09-16T01:44:52.531194+0000","last_became_peered":"2024-09-16T01:44:52.531194+0000","last_unstale":"2024-09-16T01:44:53.511442+0000","last_undegraded":"2024-09-16T01:44:53.511442+0000","last_fullsized":"2024-09-16T01:44:53.511442+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:41:57.682050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409405+0000","last_change":"2024-09-16T01:44:51.526216+0000","last_active":"2024-09-16T01:44:55.409405+0000","last_peered":"2024-09-16T01:44:55.409405+0000","last_clean":"2024-09-16T01:44:55.409405+0000","last_became_active":"2024-09-16T01:44:51.525902+0000","last_became_peered":"2024-09-16T01:44:51.525902+0000","last_unstale":"2024-09-16T01:44:55.409405+0000","last_undegraded":"2024-09-16T01:44:55.409405+0000","last_fullsized":"2024-09-16T01:44:55.409405+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:11:59.550424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.410271+0000","last_change":"2024-09-16T01:44:52.528788+0000","last_active":"2024-09-16T01:44:55.410271+0000","last_peered":"2024-09-16T01:44:55.410271+0000","last_clean":"2024-09-16T01:44:55.410271+0000","last_became_active":"2024-09-16T01:44:52.528506+0000","last_became_peered":"2024-09-16T01:44:52.528506+0000","last_unstale":"2024-09-16T01:44:55.410271+0000","last_undegraded":"2024-09-16T01:44:55.410271+0000","last_fullsized":"2024-09-16T01:44:55.410271+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:51.496468+0000","last_clean_scrub_stamp":"2024-09-16T01:44:51.496468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:49:43.638415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"137'1","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409641+0000","last_change":"2024-09-16T01:44:51.517160+0000","last_active":"2024-09-16T01:44:55.409641+0000","last_peered":"2024-09-16T01:44:55.409641+0000","last_clean":"2024-09-16T01:44:55.409641+0000","last_became_active":"2024-09-16T01:44:51.516881+0000","last_became_peered":"2024-09-16T01:44:51.516881+0000","last_unstale":"2024-09-16T01:44:55.409641+0000","last_undegraded":"2024-09-16T01:44:55.409641+0000","last_fullsized":"2024-09-16T01:44:55.409641+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":135,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:44:50.494064+0000","last_clean_scrub_stamp":"2024-09-16T01:44:50.494064+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:32:04.799253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":392,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-16T01:44:55.409728+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:44:55.409728+0000","last_peered":"2024-09-16T01:44:55.409728+0000","last_clean":"2024-09-16T01:44:55.409728+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:44:55.409728+0000","last_undegraded":"2024-09-16T01:44:55.409728+0000","last_fullsized":"2024-09-16T01:44:55.409728+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509574,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37088,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334752,"statfs":{"total":96636764160,"available":96598786048,"internally_reserved":0,"allocated":1773568,"data_stored":854620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542277,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37652,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334188,"statfs":{"total":96636764160,"available":96598208512,"internally_reserved":0,"allocated":2359296,"data_stored":1438730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542277,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37088,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334752,"statfs":{"total":96636764160,"available":96598786048,"internally_reserved":0,"allocated":1777664,"data_stored":855202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542277,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32940,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1789952,"data_stored":856286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542276,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37080,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334760,"statfs":{"total":96636764160,"available":96598794240,"internally_reserved":0,"allocated":1773568,"data_stored":849884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542277,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37068,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334772,"statfs":{"total":96636764160,"available":96598806528,"internally_reserved":0,"allocated":1761280,"data_stored":848284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542276,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37072,"kb_used_data":1716,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334768,"statfs":{"total":96636764160,"available":96598802432,"internally_reserved":0,"allocated":1757184,"data_stored":848250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542276,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41828,"kb_used_data":2312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330012,"statfs":{"total":96636764160,"available":96593932288,"internally_reserved":0,"allocated":2367488,"data_stored":1445500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:44:56.818 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T01:44:56.818 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-16T01:44:56.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.958+0000 7fc098480640 1 Processor -- start 2024-09-16T01:44:56.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.958+0000 7fc098480640 1 -- start start 2024-09-16T01:44:56.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.959+0000 7fc098480640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc09015a050 0x7fc09015d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:56.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.959+0000 7fc098480640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc0900731c0 0x7fc090073600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:56.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.959+0000 7fc098480640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc09015ffd0 0x7fc090160480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:56.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.959+0000 7fc098480640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc090073b40 con 0x7fc0900731c0 2024-09-16T01:44:56.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.959+0000 7fc098480640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc09015a430 con 0x7fc09015ffd0 2024-09-16T01:44:56.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.959+0000 7fc098480640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc09015d9a0 con 0x7fc09015a050 2024-09-16T01:44:56.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.959+0000 7fc095a1b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc0900731c0 0x7fc090073600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:56.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.959+0000 7fc095a1b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc0900731c0 0x7fc090073600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35890/0 (socket says 172.21.15.88:35890) 2024-09-16T01:44:56.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.959+0000 7fc095a1b640 1 -- 172.21.15.88:0/173583367 learned_addr learned my addr 172.21.15.88:0/173583367 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:56.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.960+0000 7fc095a1b640 1 -- 172.21.15.88:0/173583367 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc09015a050 msgr2=0x7fc09015d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:56.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.960+0000 7fc09621c640 1 --2- 172.21.15.88:0/173583367 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc09015a050 0x7fc09015d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:56.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.960+0000 7fc096a1d640 1 --2- 172.21.15.88:0/173583367 >> v2:172.21.15.154:3300/0 conn(0x7fc09015ffd0 0x7fc090160480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:56.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.960+0000 7fc095a1b640 1 --2- 172.21.15.88:0/173583367 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc09015a050 0x7fc09015d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.960+0000 7fc095a1b640 1 -- 172.21.15.88:0/173583367 >> v2:172.21.15.154:3300/0 conn(0x7fc09015ffd0 msgr2=0x7fc090160480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:56.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.960+0000 7fc095a1b640 1 --2- 172.21.15.88:0/173583367 >> v2:172.21.15.154:3300/0 conn(0x7fc09015ffd0 0x7fc090160480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.960+0000 7fc095a1b640 1 -- 172.21.15.88:0/173583367 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0900cd0a0 con 0x7fc0900731c0 2024-09-16T01:44:56.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.960+0000 7fc095a1b640 1 --2- 172.21.15.88:0/173583367 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0900731c0 0x7fc090073600 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7fc0900c6010 tx=0x7fc08002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f997d25a568aefd1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:56.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.960+0000 7fc09521a640 1 -- 172.21.15.88:0/173583367 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc080037070 con 0x7fc0900731c0 2024-09-16T01:44:56.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.961+0000 7fc09521a640 1 -- 172.21.15.88:0/173583367 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc080032440 con 0x7fc0900731c0 2024-09-16T01:44:56.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.961+0000 7fc09521a640 1 -- 172.21.15.88:0/173583367 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc080032810 con 0x7fc0900731c0 2024-09-16T01:44:56.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.961+0000 7fc098480640 1 -- 172.21.15.88:0/173583367 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0900731c0 msgr2=0x7fc090073600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:56.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.961+0000 7fc098480640 1 --2- 172.21.15.88:0/173583367 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0900731c0 0x7fc090073600 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7fc0900c6010 tx=0x7fc08002a140 comp rx=0 tx=0).stop 2024-09-16T01:44:56.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.961+0000 7fc098480640 1 -- 172.21.15.88:0/173583367 shutdown_connections 2024-09-16T01:44:56.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.961+0000 7fc098480640 1 --2- 172.21.15.88:0/173583367 >> v2:172.21.15.154:3300/0 conn(0x7fc09015ffd0 0x7fc090160480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.961+0000 7fc098480640 1 --2- 172.21.15.88:0/173583367 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0900731c0 0x7fc090073600 unknown :-1 s=CLOSED pgs=2357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.961+0000 7fc098480640 1 --2- 172.21.15.88:0/173583367 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc09015a050 0x7fc09015d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.961+0000 7fc098480640 1 -- 172.21.15.88:0/173583367 >> 172.21.15.88:0/173583367 conn(0x7fc09006d0f0 msgr2=0x7fc09006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:56.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.961+0000 7fc098480640 1 -- 172.21.15.88:0/173583367 shutdown_connections 2024-09-16T01:44:56.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.961+0000 7fc098480640 1 -- 172.21.15.88:0/173583367 wait complete. 2024-09-16T01:44:56.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.962+0000 7fc098480640 1 Processor -- start 2024-09-16T01:44:56.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.962+0000 7fc098480640 1 -- start start 2024-09-16T01:44:56.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.963+0000 7fc098480640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0900731c0 0x7fc090156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:56.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.963+0000 7fc098480640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc09015a050 0x7fc090152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:56.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.963+0000 7fc098480640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc09015ffd0 0x7fc090152760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:56.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.963+0000 7fc098480640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc0900cd940 con 0x7fc09015a050 2024-09-16T01:44:56.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.963+0000 7fc098480640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc0900cd660 con 0x7fc09015ffd0 2024-09-16T01:44:56.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.963+0000 7fc098480640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc0900cdc20 con 0x7fc0900731c0 2024-09-16T01:44:56.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.963+0000 7fc095a1b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc09015a050 0x7fc090152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:56.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.963+0000 7fc095a1b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc09015a050 0x7fc090152220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35904/0 (socket says 172.21.15.88:35904) 2024-09-16T01:44:56.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.963+0000 7fc095a1b640 1 -- 172.21.15.88:0/2056268812 learned_addr learned my addr 172.21.15.88:0/2056268812 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:56.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.963+0000 7fc096a1d640 1 --2- 172.21.15.88:0/2056268812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc09015ffd0 0x7fc090152760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:56.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.963+0000 7fc09621c640 1 --2- 172.21.15.88:0/2056268812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0900731c0 0x7fc090156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:56.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.963+0000 7fc095a1b640 1 -- 172.21.15.88:0/2056268812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0900731c0 msgr2=0x7fc090156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:56.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.963+0000 7fc095a1b640 1 --2- 172.21.15.88:0/2056268812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0900731c0 0x7fc090156d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.963+0000 7fc095a1b640 1 -- 172.21.15.88:0/2056268812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc09015ffd0 msgr2=0x7fc090152760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:56.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.963+0000 7fc095a1b640 1 --2- 172.21.15.88:0/2056268812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc09015ffd0 0x7fc090152760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:56.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.963+0000 7fc095a1b640 1 -- 172.21.15.88:0/2056268812 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc090151530 con 0x7fc09015a050 2024-09-16T01:44:56.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.963+0000 7fc09621c640 1 --2- 172.21.15.88:0/2056268812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0900731c0 0x7fc090156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:56.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.964+0000 7fc095a1b640 1 --2- 172.21.15.88:0/2056268812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc09015a050 0x7fc090152220 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7fc08002a180 tx=0x7fc0800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:56.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.964+0000 7fc086ffd640 1 -- 172.21.15.88:0/2056268812 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc080037040 con 0x7fc09015a050 2024-09-16T01:44:56.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.964+0000 7fc086ffd640 1 -- 172.21.15.88:0/2056268812 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc080042070 con 0x7fc09015a050 2024-09-16T01:44:56.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.964+0000 7fc098480640 1 -- 172.21.15.88:0/2056268812 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc090151810 con 0x7fc09015a050 2024-09-16T01:44:56.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.964+0000 7fc086ffd640 1 -- 172.21.15.88:0/2056268812 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc080030bb0 con 0x7fc09015a050 2024-09-16T01:44:56.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.964+0000 7fc098480640 1 -- 172.21.15.88:0/2056268812 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc090157270 con 0x7fc09015a050 2024-09-16T01:44:56.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.965+0000 7fc098480640 1 -- 172.21.15.88:0/2056268812 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0900b36e0 con 0x7fc09015a050 2024-09-16T01:44:56.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.965+0000 7fc086ffd640 1 -- 172.21.15.88:0/2056268812 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc080047460 con 0x7fc09015a050 2024-09-16T01:44:56.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.966+0000 7fc086ffd640 1 --2- 172.21.15.88:0/2056268812 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc07406c2b0 0x7fc07406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:56.969 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.966+0000 7fc086ffd640 1 -- 172.21.15.88:0/2056268812 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5709+0+0 (secure 0 0 0) 0x7fc0800ac030 con 0x7fc09015a050 2024-09-16T01:44:56.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.966+0000 7fc09621c640 1 --2- 172.21.15.88:0/2056268812 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc07406c2b0 0x7fc07406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:56.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.966+0000 7fc09621c640 1 --2- 172.21.15.88:0/2056268812 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc07406c2b0 0x7fc07406e770 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fc08c007f50 tx=0x7fc08c0074a0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:56.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:56.970+0000 7fc086ffd640 1 -- 172.21.15.88:0/2056268812 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc080045360 con 0x7fc09015a050 2024-09-16T01:44:57.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.199+0000 7fc098480640 1 -- 172.21.15.88:0/2056268812 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fc090156b10 con 0x7fc09015a050 2024-09-16T01:44:57.205 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.201+0000 7fc086ffd640 1 -- 172.21.15.88:0/2056268812 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) ==== 190+0+147 (secure 0 0 0) 0x7fc08007a830 con 0x7fc09015a050 2024-09-16T01:44:57.205 INFO:teuthology.orchestra.run.smithi088.stdout:[client.0] 2024-09-16T01:44:57.205 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQDeiudmvPikIBAAyHxNNnwQKygxg09PNQLc/Q== 2024-09-16T01:44:57.205 INFO:teuthology.orchestra.run.smithi088.stdout: caps mds = "allow" 2024-09-16T01:44:57.206 INFO:teuthology.orchestra.run.smithi088.stdout: caps mon = "allow r" 2024-09-16T01:44:57.206 INFO:teuthology.orchestra.run.smithi088.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-16T01:44:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:updated caps for client.0 2024-09-16T01:44:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.203+0000 7fc098480640 1 -- 172.21.15.88:0/2056268812 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc07406c2b0 msgr2=0x7fc07406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.203+0000 7fc098480640 1 --2- 172.21.15.88:0/2056268812 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc07406c2b0 0x7fc07406e770 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fc08c007f50 tx=0x7fc08c0074a0 comp rx=0 tx=0).stop 2024-09-16T01:44:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.203+0000 7fc098480640 1 -- 172.21.15.88:0/2056268812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc09015a050 msgr2=0x7fc090152220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.203+0000 7fc098480640 1 --2- 172.21.15.88:0/2056268812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc09015a050 0x7fc090152220 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7fc08002a180 tx=0x7fc0800046c0 comp rx=0 tx=0).stop 2024-09-16T01:44:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.203+0000 7fc098480640 1 -- 172.21.15.88:0/2056268812 shutdown_connections 2024-09-16T01:44:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.203+0000 7fc098480640 1 --2- 172.21.15.88:0/2056268812 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc07406c2b0 0x7fc07406e770 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.203+0000 7fc098480640 1 --2- 172.21.15.88:0/2056268812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc09015ffd0 0x7fc090152760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.203+0000 7fc098480640 1 --2- 172.21.15.88:0/2056268812 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc09015a050 0x7fc090152220 unknown :-1 s=CLOSED pgs=2358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.203+0000 7fc098480640 1 --2- 172.21.15.88:0/2056268812 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0900731c0 0x7fc090156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.203+0000 7fc098480640 1 -- 172.21.15.88:0/2056268812 >> 172.21.15.88:0/2056268812 conn(0x7fc09006d0f0 msgr2=0x7fc0900b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:57.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.203+0000 7fc098480640 1 -- 172.21.15.88:0/2056268812 shutdown_connections 2024-09-16T01:44:57.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.203+0000 7fc098480640 1 -- 172.21.15.88:0/2056268812 wait complete. 2024-09-16T01:44:57.216 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:44:57.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.312+0000 7f011c3e4640 1 Processor -- start 2024-09-16T01:44:57.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.312+0000 7f011c3e4640 1 -- start start 2024-09-16T01:44:57.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.313+0000 7f011c3e4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f011415c170 0x7f011415c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:57.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.313+0000 7f011c3e4640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f01140cf6c0 0x7f01140cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:57.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.313+0000 7f011c3e4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01140b49c0 con 0x7f01140b5dd0 2024-09-16T01:44:57.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.313+0000 7f011c3e4640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f011415ca90 con 0x7f01140cf6c0 2024-09-16T01:44:57.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.313+0000 7f011c3e4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f01140b4cf0 con 0x7f011415c170 2024-09-16T01:44:57.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.313+0000 7f011997f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f01140cf6c0 0x7f01140cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:57.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.313+0000 7f011997f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f01140cf6c0 0x7f01140cfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35910/0 (socket says 172.21.15.88:35910) 2024-09-16T01:44:57.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.313+0000 7f011997f640 1 -- 172.21.15.88:0/926091387 learned_addr learned my addr 172.21.15.88:0/926091387 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:57.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.314+0000 7f011a180640 1 --2- 172.21.15.88:0/926091387 >> v2:172.21.15.154:3300/0 conn(0x7f011415c170 0x7f011415c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:57.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.314+0000 7f011997f640 1 -- 172.21.15.88:0/926091387 >> v2:172.21.15.154:3300/0 conn(0x7f011415c170 msgr2=0x7f011415c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:57.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.314+0000 7f011997f640 1 --2- 172.21.15.88:0/926091387 >> v2:172.21.15.154:3300/0 conn(0x7f011415c170 0x7f011415c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:57.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.314+0000 7f011997f640 1 -- 172.21.15.88:0/926091387 >> v1:172.21.15.154:6789/0 conn(0x7f01140b5dd0 legacy=0x7f011415ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:57.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.314+0000 7f011997f640 1 -- 172.21.15.88:0/926091387 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01140b4fd0 con 0x7f01140cf6c0 2024-09-16T01:44:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.314+0000 7f011997f640 1 --2- 172.21.15.88:0/926091387 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01140cf6c0 0x7f01140cfb30 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f0108009870 tx=0x7f010802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=106b64e033aa1b3e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.314+0000 7f011917e640 1 -- 172.21.15.88:0/926091387 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0108037070 con 0x7f01140cf6c0 2024-09-16T01:44:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.314+0000 7f011917e640 1 -- 172.21.15.88:0/926091387 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0108032440 con 0x7f01140cf6c0 2024-09-16T01:44:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.314+0000 7f011917e640 1 -- 172.21.15.88:0/926091387 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0108032810 con 0x7f01140cf6c0 2024-09-16T01:44:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.315+0000 7f011c3e4640 1 -- 172.21.15.88:0/926091387 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01140cf6c0 msgr2=0x7f01140cfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.315+0000 7f011c3e4640 1 --2- 172.21.15.88:0/926091387 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01140cf6c0 0x7f01140cfb30 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f0108009870 tx=0x7f010802a270 comp rx=0 tx=0).stop 2024-09-16T01:44:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.315+0000 7f011c3e4640 1 -- 172.21.15.88:0/926091387 shutdown_connections 2024-09-16T01:44:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.315+0000 7f011c3e4640 1 --2- 172.21.15.88:0/926091387 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01140cf6c0 0x7f01140cfb30 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.315+0000 7f011c3e4640 1 --2- 172.21.15.88:0/926091387 >> v2:172.21.15.154:3300/0 conn(0x7f011415c170 0x7f011415c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.315+0000 7f011c3e4640 1 -- 172.21.15.88:0/926091387 >> 172.21.15.88:0/926091387 conn(0x7f011406bc10 msgr2=0x7f011406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.315+0000 7f011c3e4640 1 -- 172.21.15.88:0/926091387 shutdown_connections 2024-09-16T01:44:57.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.315+0000 7f011c3e4640 1 -- 172.21.15.88:0/926091387 wait complete. 2024-09-16T01:44:57.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.316+0000 7f011c3e4640 1 Processor -- start 2024-09-16T01:44:57.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.316+0000 7f011c3e4640 1 -- start start 2024-09-16T01:44:57.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.317+0000 7f011c3e4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01140b5dd0 0x7f011414ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:57.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.317+0000 7f011c3e4640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01140cf6c0 0x7f011415efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:57.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.317+0000 7f011c3e4640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f011415c170 0x7f011414e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:57.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.317+0000 7f011c3e4640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f01140b5870 con 0x7f01140cf6c0 2024-09-16T01:44:57.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.317+0000 7f011c3e4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f01140b5590 con 0x7f01140b5dd0 2024-09-16T01:44:57.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.317+0000 7f011c3e4640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f01140b5b50 con 0x7f011415c170 2024-09-16T01:44:57.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.317+0000 7f011997f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01140cf6c0 0x7f011415efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:57.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.317+0000 7f011997f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01140cf6c0 0x7f011415efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35924/0 (socket says 172.21.15.88:35924) 2024-09-16T01:44:57.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.317+0000 7f011997f640 1 -- 172.21.15.88:0/1184646123 learned_addr learned my addr 172.21.15.88:0/1184646123 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:57.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.317+0000 7f011997f640 1 -- 172.21.15.88:0/1184646123 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f011415c170 msgr2=0x7f011414e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:57.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.317+0000 7f011997f640 1 --2- 172.21.15.88:0/1184646123 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f011415c170 0x7f011414e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:57.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.317+0000 7f011997f640 1 -- 172.21.15.88:0/1184646123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01140b5dd0 msgr2=0x7f011414ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:57.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.317+0000 7f011a180640 1 --2- 172.21.15.88:0/1184646123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01140b5dd0 0x7f011414ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:57.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.317+0000 7f011997f640 1 --2- 172.21.15.88:0/1184646123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01140b5dd0 0x7f011414ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.318+0000 7f011997f640 1 -- 172.21.15.88:0/1184646123 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f011415df40 con 0x7f01140cf6c0 2024-09-16T01:44:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.318+0000 7f011a180640 1 --2- 172.21.15.88:0/1184646123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01140b5dd0 0x7f011414ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.318+0000 7f011997f640 1 --2- 172.21.15.88:0/1184646123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01140cf6c0 0x7f011415efb0 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7f0108009870 tx=0x7f01080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.318+0000 7f0112ffd640 1 -- 172.21.15.88:0/1184646123 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0108037040 con 0x7f01140cf6c0 2024-09-16T01:44:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.318+0000 7f0112ffd640 1 -- 172.21.15.88:0/1184646123 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0108004230 con 0x7f01140cf6c0 2024-09-16T01:44:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.318+0000 7f011c3e4640 1 -- 172.21.15.88:0/1184646123 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f011415e250 con 0x7f01140cf6c0 2024-09-16T01:44:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.318+0000 7f0112ffd640 1 -- 172.21.15.88:0/1184646123 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0108030620 con 0x7f01140cf6c0 2024-09-16T01:44:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.318+0000 7f011c3e4640 1 -- 172.21.15.88:0/1184646123 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f011415ea90 con 0x7f01140cf6c0 2024-09-16T01:44:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.319+0000 7f011c3e4640 1 -- 172.21.15.88:0/1184646123 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01140cfb30 con 0x7f01140cf6c0 2024-09-16T01:44:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.319+0000 7f0112ffd640 1 -- 172.21.15.88:0/1184646123 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f01080307c0 con 0x7f01140cf6c0 2024-09-16T01:44:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.320+0000 7f0112ffd640 1 --2- 172.21.15.88:0/1184646123 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f010006c2b0 0x7f010006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.320+0000 7f0112ffd640 1 -- 172.21.15.88:0/1184646123 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5709+0+0 (secure 0 0 0) 0x7f01080ac250 con 0x7f01140cf6c0 2024-09-16T01:44:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.320+0000 7f011a180640 1 --2- 172.21.15.88:0/1184646123 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f010006c2b0 0x7f010006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:57.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.320+0000 7f011a180640 1 --2- 172.21.15.88:0/1184646123 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f010006c2b0 0x7f010006e770 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f01141f5830 tx=0x7f0104007420 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:57.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.324+0000 7f0112ffd640 1 -- 172.21.15.88:0/1184646123 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f011415e250 con 0x7f01140cf6c0 2024-09-16T01:44:57.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.528+0000 7f011c3e4640 1 -- 172.21.15.88:0/1184646123 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f011415e430 con 0x7f01140cf6c0 2024-09-16T01:44:57.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.529+0000 7f0112ffd640 1 -- 172.21.15.88:0/1184646123 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 326 v326) ==== 95+0+5239 (secure 0 0 0) 0x7f010807aa50 con 0x7f01140cf6c0 2024-09-16T01:44:57.530 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:44:57.530 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":326,"btime":"2024-09-16T01:44:55:534079+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7649,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2706550508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2706550508},{"type":"v1","addr":"172.21.15.154:6834","nonce":2706550508}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322},{"gid":8569,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/3460839382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":3460839382},{"type":"v1","addr":"172.21.15.88:6837","nonce":3460839382}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322},{"gid":8581,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2822970755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2822970755},{"type":"v1","addr":"172.21.15.154:6835","nonce":2822970755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322}],"filesystems":[{"mdsmap":{"epoch":326,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:44:51.845091+0000","modified":"2024-09-16T01:44:54.630127+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10536},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10536":{"gid":10536,"name":"a","rank":0,"incarnation":323,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6836/656828696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":656828696},{"type":"v1","addr":"172.21.15.88:6836","nonce":656828696}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10536,"qdb_cluster":[10536]},"id":7}]} 2024-09-16T01:44:57.531 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 326 2024-09-16T01:44:57.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.531+0000 7f011c3e4640 1 -- 172.21.15.88:0/1184646123 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f010006c2b0 msgr2=0x7f010006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:57.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.531+0000 7f011c3e4640 1 --2- 172.21.15.88:0/1184646123 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f010006c2b0 0x7f010006e770 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f01141f5830 tx=0x7f0104007420 comp rx=0 tx=0).stop 2024-09-16T01:44:57.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.531+0000 7f011c3e4640 1 -- 172.21.15.88:0/1184646123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01140cf6c0 msgr2=0x7f011415efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:57.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.531+0000 7f011c3e4640 1 --2- 172.21.15.88:0/1184646123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01140cf6c0 0x7f011415efb0 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7f0108009870 tx=0x7f01080046c0 comp rx=0 tx=0).stop 2024-09-16T01:44:57.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.531+0000 7f011c3e4640 1 -- 172.21.15.88:0/1184646123 shutdown_connections 2024-09-16T01:44:57.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.531+0000 7f011c3e4640 1 --2- 172.21.15.88:0/1184646123 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f010006c2b0 0x7f010006e770 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:57.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.531+0000 7f011c3e4640 1 --2- 172.21.15.88:0/1184646123 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f011415c170 0x7f011414e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:57.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.531+0000 7f011c3e4640 1 --2- 172.21.15.88:0/1184646123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f01140cf6c0 0x7f011415efb0 unknown :-1 s=CLOSED pgs=2360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:57.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.531+0000 7f011c3e4640 1 --2- 172.21.15.88:0/1184646123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01140b5dd0 0x7f011414ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:57.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.531+0000 7f011c3e4640 1 -- 172.21.15.88:0/1184646123 >> 172.21.15.88:0/1184646123 conn(0x7f011406bc10 msgr2=0x7f0114071440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:57.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.531+0000 7f011c3e4640 1 -- 172.21.15.88:0/1184646123 shutdown_connections 2024-09-16T01:44:57.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:57.531+0000 7f011c3e4640 1 -- 172.21.15.88:0/1184646123 wait complete. 2024-09-16T01:44:57.945 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 326, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:44:51.845091+0000', 'modified': '2024-09-16T01:44:54.630127+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10536}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10536': {'gid': 10536, 'name': 'a', 'rank': 0, 'incarnation': 323, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.88:6836/656828696', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 656828696}, {'type': 'v1', 'addr': '172.21.15.88:6836', 'nonce': 656828696}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10536, 'qdb_cluster': [10536]} 2024-09-16T01:44:57.946 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T01:44:57.946 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-16T01:44:58.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.045+0000 7fccb9349640 1 Processor -- start 2024-09-16T01:44:58.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.045+0000 7fccb9349640 1 -- start start 2024-09-16T01:44:58.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.046+0000 7fccb9349640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccb40b54d0 0x7fccb406fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.046+0000 7fccb9349640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fccb40b7450 0x7fccb415f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.046+0000 7fccb9349640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccb406e990 con 0x7fccb415e6f0 2024-09-16T01:44:58.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.046+0000 7fccb9349640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fccb406ec70 con 0x7fccb40b7450 2024-09-16T01:44:58.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.046+0000 7fccb9349640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fccb415ec70 con 0x7fccb40b54d0 2024-09-16T01:44:58.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.046+0000 7fccb37fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fccb40b7450 0x7fccb415f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:58.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.046+0000 7fccb37fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fccb40b7450 0x7fccb415f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35938/0 (socket says 172.21.15.88:35938) 2024-09-16T01:44:58.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.046+0000 7fccb37fe640 1 -- 172.21.15.88:0/577890558 learned_addr learned my addr 172.21.15.88:0/577890558 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:58.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.046+0000 7fccaa7fc640 1 --2- 172.21.15.88:0/577890558 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccb40b54d0 0x7fccb406fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:58.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.046+0000 7fccb37fe640 1 -- 172.21.15.88:0/577890558 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccb40b54d0 msgr2=0x7fccb406fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:58.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.046+0000 7fccb37fe640 1 --2- 172.21.15.88:0/577890558 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccb40b54d0 0x7fccb406fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.046+0000 7fccb37fe640 1 -- 172.21.15.88:0/577890558 >> v1:172.21.15.154:6789/0 conn(0x7fccb415e6f0 legacy=0x7fccb40b6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:58.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.046+0000 7fccb37fe640 1 -- 172.21.15.88:0/577890558 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccb415ef50 con 0x7fccb40b7450 2024-09-16T01:44:58.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.047+0000 7fccb37fe640 1 --2- 172.21.15.88:0/577890558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccb40b7450 0x7fccb415f750 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7fccac009e50 tx=0x7fccac02a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91866e49db4f688 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:58.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.047+0000 7fccb27fc640 1 -- 172.21.15.88:0/577890558 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccac036070 con 0x7fccb40b7450 2024-09-16T01:44:58.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.047+0000 7fccb27fc640 1 -- 172.21.15.88:0/577890558 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccac031600 con 0x7fccb40b7450 2024-09-16T01:44:58.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.047+0000 7fccb27fc640 1 -- 172.21.15.88:0/577890558 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccac0319d0 con 0x7fccb40b7450 2024-09-16T01:44:58.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.047+0000 7fccb9349640 1 -- 172.21.15.88:0/577890558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccb40b7450 msgr2=0x7fccb415f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:58.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.047+0000 7fccb9349640 1 --2- 172.21.15.88:0/577890558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccb40b7450 0x7fccb415f750 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7fccac009e50 tx=0x7fccac02a6c0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.048+0000 7fccb9349640 1 -- 172.21.15.88:0/577890558 shutdown_connections 2024-09-16T01:44:58.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.048+0000 7fccb9349640 1 --2- 172.21.15.88:0/577890558 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccb40b7450 0x7fccb415f750 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.048+0000 7fccb9349640 1 --2- 172.21.15.88:0/577890558 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccb40b54d0 0x7fccb406fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.048+0000 7fccb9349640 1 -- 172.21.15.88:0/577890558 >> 172.21.15.88:0/577890558 conn(0x7fccb40d2430 msgr2=0x7fccb40d2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:58.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.048+0000 7fccb9349640 1 -- 172.21.15.88:0/577890558 shutdown_connections 2024-09-16T01:44:58.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.048+0000 7fccb9349640 1 -- 172.21.15.88:0/577890558 wait complete. 2024-09-16T01:44:58.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.049+0000 7fccb9349640 1 Processor -- start 2024-09-16T01:44:58.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.049+0000 7fccb9349640 1 -- start start 2024-09-16T01:44:58.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.050+0000 7fccb9349640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccb40b54d0 0x7fccb41607a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.050+0000 7fccb9349640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccb40b7450 0x7fccb40d0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.050+0000 7fccb9349640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccb415e6f0 0x7fccb4157bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.050+0000 7fccb9349640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fccb406e730 con 0x7fccb40b54d0 2024-09-16T01:44:58.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.050+0000 7fccb9349640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fccb406e5b0 con 0x7fccb415e6f0 2024-09-16T01:44:58.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.050+0000 7fccb9349640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fccb4070830 con 0x7fccb40b7450 2024-09-16T01:44:58.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.050+0000 7fccb2ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccb40b54d0 0x7fccb41607a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:58.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.050+0000 7fccb2ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccb40b54d0 0x7fccb41607a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35946/0 (socket says 172.21.15.88:35946) 2024-09-16T01:44:58.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.050+0000 7fccb2ffd640 1 -- 172.21.15.88:0/3634794371 learned_addr learned my addr 172.21.15.88:0/3634794371 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:58.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.050+0000 7fccaa7fc640 1 --2- 172.21.15.88:0/3634794371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccb40b7450 0x7fccb40d0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:58.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.050+0000 7fccb37fe640 1 --2- 172.21.15.88:0/3634794371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccb415e6f0 0x7fccb4157bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:58.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.050+0000 7fccb2ffd640 1 -- 172.21.15.88:0/3634794371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccb40b7450 msgr2=0x7fccb40d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:58.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.050+0000 7fccb2ffd640 1 --2- 172.21.15.88:0/3634794371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccb40b7450 0x7fccb40d0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.050+0000 7fccb2ffd640 1 -- 172.21.15.88:0/3634794371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccb415e6f0 msgr2=0x7fccb4157bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:58.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.050+0000 7fccb2ffd640 1 --2- 172.21.15.88:0/3634794371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccb415e6f0 0x7fccb4157bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.050+0000 7fccb2ffd640 1 -- 172.21.15.88:0/3634794371 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccb4161010 con 0x7fccb40b54d0 2024-09-16T01:44:58.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.050+0000 7fccaa7fc640 1 --2- 172.21.15.88:0/3634794371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccb40b7450 0x7fccb40d0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:58.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.050+0000 7fccb2ffd640 1 --2- 172.21.15.88:0/3634794371 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccb40b54d0 0x7fccb41607a0 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7fcca400b3f0 tx=0x7fcca400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:58.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.050+0000 7fccabfff640 1 -- 172.21.15.88:0/3634794371 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca4015020 con 0x7fccb40b54d0 2024-09-16T01:44:58.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.050+0000 7fccabfff640 1 -- 172.21.15.88:0/3634794371 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcca4011830 con 0x7fccb40b54d0 2024-09-16T01:44:58.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.050+0000 7fccabfff640 1 -- 172.21.15.88:0/3634794371 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca4011c20 con 0x7fccb40b54d0 2024-09-16T01:44:58.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.050+0000 7fccb9349640 1 -- 172.21.15.88:0/3634794371 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccb41612f0 con 0x7fccb40b54d0 2024-09-16T01:44:58.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.050+0000 7fccb9349640 1 -- 172.21.15.88:0/3634794371 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccb4161b90 con 0x7fccb40b54d0 2024-09-16T01:44:58.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.052+0000 7fccabfff640 1 -- 172.21.15.88:0/3634794371 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fcca4018020 con 0x7fccb40b54d0 2024-09-16T01:44:58.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.053+0000 7fccabfff640 1 --2- 172.21.15.88:0/3634794371 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcc9406c2b0 0x7fcc9406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.053+0000 7fccabfff640 1 -- 172.21.15.88:0/3634794371 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5709+0+0 (secure 0 0 0) 0x7fcca408b0d0 con 0x7fccb40b54d0 2024-09-16T01:44:58.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.053+0000 7fccaa7fc640 1 --2- 172.21.15.88:0/3634794371 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcc9406c2b0 0x7fcc9406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:58.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.053+0000 7fccaa7fc640 1 --2- 172.21.15.88:0/3634794371 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcc9406c2b0 0x7fcc9406e770 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fccb415d980 tx=0x7fcc9c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:58.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.053+0000 7fccbc1bd740 1 Processor -- start 2024-09-16T01:44:58.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.054+0000 7fccbc1bd740 1 -- start start 2024-09-16T01:44:58.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.054+0000 7fccbc1bd740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f4732c0 0x56117f4b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.054+0000 7fccbc1bd740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56117f470a80 0x56117f4736a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.054+0000 7fccbc1bd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56117f474a50 0x56117f46d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.054+0000 7fccbc1bd740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56117f3d5520 con 0x56117f4732c0 2024-09-16T01:44:58.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.054+0000 7fccbc1bd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56117f37d3e0 con 0x56117f474a50 2024-09-16T01:44:58.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.054+0000 7fccbc1bd740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56117f3d14e0 con 0x56117f470a80 2024-09-16T01:44:58.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.054+0000 7fccb2ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f4732c0 0x56117f4b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:58.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.055+0000 7fccb2ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f4732c0 0x56117f4b8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35952/0 (socket says 172.21.15.88:35952) 2024-09-16T01:44:58.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.055+0000 7fccb2ffd640 1 -- 172.21.15.88:0/220367932 learned_addr learned my addr 172.21.15.88:0/220367932 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:58.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.055+0000 7fccaa7fc640 1 --2- 172.21.15.88:0/220367932 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56117f470a80 0x56117f4736a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:58.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.055+0000 7fccb37fe640 1 --2- 172.21.15.88:0/220367932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56117f474a50 0x56117f46d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:58.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.055+0000 7fccb2ffd640 1 -- 172.21.15.88:0/220367932 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56117f470a80 msgr2=0x56117f4736a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:58.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.055+0000 7fccb2ffd640 1 --2- 172.21.15.88:0/220367932 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56117f470a80 0x56117f4736a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.055+0000 7fccb2ffd640 1 -- 172.21.15.88:0/220367932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56117f474a50 msgr2=0x56117f46d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:58.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.055+0000 7fccb2ffd640 1 --2- 172.21.15.88:0/220367932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56117f474a50 0x56117f46d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.055+0000 7fccb2ffd640 1 -- 172.21.15.88:0/220367932 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56117f3982f0 con 0x56117f4732c0 2024-09-16T01:44:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.055+0000 7fccb2ffd640 1 --2- 172.21.15.88:0/220367932 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f4732c0 0x56117f4b8a60 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7fcca4089540 tx=0x7fcca4059970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b074ae40b70bc78 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.055+0000 7fcca8ff9640 1 -- 172.21.15.88:0/220367932 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca40571e0 con 0x56117f4732c0 2024-09-16T01:44:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.055+0000 7fcca8ff9640 1 -- 172.21.15.88:0/220367932 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcca4011dc0 con 0x56117f4732c0 2024-09-16T01:44:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.055+0000 7fcca8ff9640 1 -- 172.21.15.88:0/220367932 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca405d1b0 con 0x56117f4732c0 2024-09-16T01:44:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.056+0000 7fccbc1bd740 1 -- 172.21.15.88:0/220367932 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f4732c0 msgr2=0x56117f4b8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.056+0000 7fccbc1bd740 1 --2- 172.21.15.88:0/220367932 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f4732c0 0x56117f4b8a60 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7fcca4089540 tx=0x7fcca4059970 comp rx=0 tx=0).stop 2024-09-16T01:44:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.056+0000 7fccbc1bd740 1 -- 172.21.15.88:0/220367932 shutdown_connections 2024-09-16T01:44:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.056+0000 7fccbc1bd740 1 --2- 172.21.15.88:0/220367932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56117f474a50 0x56117f46d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.056+0000 7fccbc1bd740 1 --2- 172.21.15.88:0/220367932 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56117f470a80 0x56117f4736a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.056+0000 7fccbc1bd740 1 --2- 172.21.15.88:0/220367932 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f4732c0 0x56117f4b8a60 unknown :-1 s=CLOSED pgs=2363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.056+0000 7fccbc1bd740 1 -- 172.21.15.88:0/220367932 >> 172.21.15.88:0/220367932 conn(0x56117f46fa80 msgr2=0x56117f4b7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:58.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.056+0000 7fccbc1bd740 1 -- 172.21.15.88:0/220367932 shutdown_connections 2024-09-16T01:44:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.056+0000 7fccbc1bd740 1 -- 172.21.15.88:0/220367932 wait complete. 2024-09-16T01:44:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.056+0000 7fccbc1bd740 1 -- start start 2024-09-16T01:44:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.056+0000 7fccbc1bd740 1 Processor -- start 2024-09-16T01:44:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.057+0000 7fccbc1bd740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f4732c0 0x56117f4c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.057+0000 7fccbc1bd740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56117f474a50 0x56117f4b54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.057+0000 7fccbc1bd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56117f495d40 0x56117f49d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.057+0000 7fccbc1bd740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56117f46ef20 con 0x56117f4732c0 2024-09-16T01:44:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.057+0000 7fccbc1bd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56117f35e450 con 0x56117f495d40 2024-09-16T01:44:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.057+0000 7fccbc1bd740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56117f4b9290 con 0x56117f474a50 2024-09-16T01:44:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.057+0000 7fccbc1bd740 10 client.0 _ll_register_callbacks cb 0x56117f0e7ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:44:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.057+0000 7fccb2ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f4732c0 0x56117f4c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.057+0000 7fccb2ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f4732c0 0x56117f4c8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35966/0 (socket says 172.21.15.88:35966) 2024-09-16T01:44:58.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.057+0000 7fccb2ffd640 1 -- 172.21.15.88:0/39085039 learned_addr learned my addr 172.21.15.88:0/39085039 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:58.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.057+0000 7fccaa7fc640 1 --2- 172.21.15.88:0/39085039 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56117f474a50 0x56117f4b54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:58.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.057+0000 7fccb37fe640 1 --2- 172.21.15.88:0/39085039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56117f495d40 0x56117f49d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:58.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.058+0000 7fccb2ffd640 1 -- 172.21.15.88:0/39085039 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56117f474a50 msgr2=0x56117f4b54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:58.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.058+0000 7fccb2ffd640 1 --2- 172.21.15.88:0/39085039 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56117f474a50 0x56117f4b54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.058+0000 7fccb2ffd640 1 -- 172.21.15.88:0/39085039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56117f495d40 msgr2=0x56117f49d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:58.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.058+0000 7fccb2ffd640 1 --2- 172.21.15.88:0/39085039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56117f495d40 0x56117f49d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.058+0000 7fccb2ffd640 1 -- 172.21.15.88:0/39085039 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56117f0e82d0 con 0x56117f4732c0 2024-09-16T01:44:58.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.058+0000 7fccb2ffd640 1 --2- 172.21.15.88:0/39085039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f4732c0 0x56117f4c8bd0 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7fcca4058e40 tx=0x7fcca401c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48da8d8e659e2805 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:58.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.058+0000 7fcc91ffb640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:44:58.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.058+0000 7fcc91ffb640 1 -- 172.21.15.88:0/39085039 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca4057440 con 0x56117f4732c0 2024-09-16T01:44:58.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.058+0000 7fcc91ffb640 1 -- 172.21.15.88:0/39085039 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(138..138 src has 1..138) ==== 5709+0+0 (secure 0 0 0) 0x7fcca405d1b0 con 0x56117f4732c0 2024-09-16T01:44:58.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.058+0000 7fccbc1bd740 1 -- 172.21.15.88:0/39085039 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcca4011dc0 con 0x56117f4732c0 2024-09-16T01:44:58.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.058+0000 7fcc91ffb640 1 -- 172.21.15.88:0/39085039 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=326) ==== 24+0+0 (secure 0 0 0) 0x7fcca40557b0 con 0x56117f4732c0 2024-09-16T01:44:58.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.058+0000 7fccbc1bd740 10 client.10671 fetch_fsmap learned FSMap version 326 2024-09-16T01:44:58.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.058+0000 7fccbc1bd740 1 -- 172.21.15.88:0/39085039 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=326,monmap=2+}) -- 0x56117f1b26b0 con 0x56117f4732c0 2024-09-16T01:44:58.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.058+0000 7fcc91ffb640 1 -- 172.21.15.88:0/39085039 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcca4011dc0 con 0x56117f4732c0 2024-09-16T01:44:58.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.058+0000 7fcc91ffb640 1 -- 172.21.15.88:0/39085039 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 326) ==== 3502+0+0 (secure 0 0 0) 0x7fcca4058590 con 0x56117f4732c0 2024-09-16T01:44:58.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.058+0000 7fccbc1bd740 10 client.10671 fetch_fsmap finished waiting for FSMap version 326 2024-09-16T01:44:58.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.058+0000 7fccbc1bd740 10 client.10671 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:44:58.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.058+0000 7fccbc1bd740 20 client.10671 populate_metadata read hostname 'smithi088' 2024-09-16T01:44:58.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.058+0000 7fccbc1bd740 1 --2- 172.21.15.88:0/39085039 >> [v2:172.21.15.88:6834/656828696,v1:172.21.15.88:6836/656828696] conn(0x56117f49b4b0 0x56117f4cb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.058+0000 7fccbc1bd740 4 client.10671 mds_command: new command op to 10536 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:44:58.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.058+0000 7fccbc1bd740 1 -- 172.21.15.88:0/39085039 --> [v2:172.21.15.88:6834/656828696,v1:172.21.15.88:6836/656828696] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56117f3e2c50 con 0x56117f49b4b0 2024-09-16T01:44:58.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.058+0000 7fccaa7fc640 1 --2- 172.21.15.88:0/39085039 >> [v2:172.21.15.88:6834/656828696,v1:172.21.15.88:6836/656828696] conn(0x56117f49b4b0 0x56117f4cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:58.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.059+0000 7fcc91ffb640 10 client.10671 ms_handle_connect on v2:172.21.15.88:6834/656828696 2024-09-16T01:44:58.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.059+0000 7fccaa7fc640 1 --2- 172.21.15.88:0/39085039 >> [v2:172.21.15.88:6834/656828696,v1:172.21.15.88:6836/656828696] conn(0x56117f49b4b0 0x56117f4cb740 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8723cd666276e7a5 server_cookie=636161a78f209f1a in_seq=0 out_seq=0 2024-09-16T01:44:58.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.059+0000 7fcc91ffb640 1 -- 172.21.15.88:0/39085039 <== mds.0 v2:172.21.15.88:6834/656828696 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56117f3e2c50 con 0x56117f49b4b0 2024-09-16T01:44:58.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.059+0000 7fcc91ffb640 10 client.10671 handle_command_reply: tid=0 2024-09-16T01:44:58.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.059+0000 7fccbc1bd740 1 client.10671 shutdown 2024-09-16T01:44:58.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.060+0000 7fccbc1bd740 1 -- 172.21.15.88:0/39085039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f4732c0 msgr2=0x56117f4c8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:58.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.060+0000 7fccbc1bd740 1 --2- 172.21.15.88:0/39085039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f4732c0 0x56117f4c8bd0 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7fcca4058e40 tx=0x7fcca401c040 comp rx=0 tx=0).stop 2024-09-16T01:44:58.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.060+0000 7fccbc1bd740 1 -- 172.21.15.88:0/39085039 shutdown_connections 2024-09-16T01:44:58.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.060+0000 7fccbc1bd740 1 --2- 172.21.15.88:0/39085039 >> [v2:172.21.15.88:6834/656828696,v1:172.21.15.88:6836/656828696] conn(0x56117f49b4b0 0x56117f4cb740 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.060+0000 7fccbc1bd740 1 --2- 172.21.15.88:0/39085039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56117f495d40 0x56117f49d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.060+0000 7fccbc1bd740 1 --2- 172.21.15.88:0/39085039 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56117f474a50 0x56117f4b54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.060+0000 7fcc91ffb640 0 client.10671 ms_handle_reset on v2:172.21.15.88:6834/656828696 2024-09-16T01:44:58.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.060+0000 7fccbc1bd740 1 --2- 172.21.15.88:0/39085039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f4732c0 0x56117f4c8bd0 unknown :-1 s=CLOSED pgs=2364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.060+0000 7fccbc1bd740 1 -- 172.21.15.88:0/39085039 >> 172.21.15.88:0/39085039 conn(0x56117f470a80 msgr2=0x56117f4b7510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:58.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.060+0000 7fccbc1bd740 1 -- 172.21.15.88:0/39085039 shutdown_connections 2024-09-16T01:44:58.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.060+0000 7fccbc1bd740 1 -- 172.21.15.88:0/39085039 wait complete. 2024-09-16T01:44:58.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.060+0000 7fccbc1bd740 20 client.10671 trim_cache size 0 max 16384 2024-09-16T01:44:58.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.077+0000 7fccbc1bd740 1 Processor -- start 2024-09-16T01:44:58.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.077+0000 7fccbc1bd740 1 -- start start 2024-09-16T01:44:58.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.078+0000 7fccbc1bd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56117f4732c0 0x56117f4dc370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.078+0000 7fccbc1bd740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56117f474a50 0x56117f46f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.078+0000 7fccbc1bd740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f495d40 0x56117f4968d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.078+0000 7fccbc1bd740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fccb41ffc40 con 0x56117f495d40 2024-09-16T01:44:58.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.078+0000 7fccbc1bd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fccb40cc8c0 con 0x56117f4732c0 2024-09-16T01:44:58.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.078+0000 7fccbc1bd740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fccb40ccf10 con 0x56117f474a50 2024-09-16T01:44:58.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.078+0000 7fccb37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f495d40 0x56117f4968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:58.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.078+0000 7fccb37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f495d40 0x56117f4968d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35982/0 (socket says 172.21.15.88:35982) 2024-09-16T01:44:58.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.078+0000 7fccb37fe640 1 -- 172.21.15.88:0/4242117292 learned_addr learned my addr 172.21.15.88:0/4242117292 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:58.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.078+0000 7fccb37fe640 1 -- 172.21.15.88:0/4242117292 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56117f474a50 msgr2=0x56117f46f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:58.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.078+0000 7fccb37fe640 1 --2- 172.21.15.88:0/4242117292 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56117f474a50 0x56117f46f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.078+0000 7fccb37fe640 1 -- 172.21.15.88:0/4242117292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56117f4732c0 msgr2=0x56117f4dc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:58.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.078+0000 7fccb37fe640 1 --2- 172.21.15.88:0/4242117292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56117f4732c0 0x56117f4dc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.078+0000 7fccb37fe640 1 -- 172.21.15.88:0/4242117292 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56117f4b76b0 con 0x56117f495d40 2024-09-16T01:44:58.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.078+0000 7fccb37fe640 1 --2- 172.21.15.88:0/4242117292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f495d40 0x56117f4968d0 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7fccac02a700 tx=0x7fccac02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62dd3f014b3366f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:58.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.079+0000 7fcc93fff640 1 -- 172.21.15.88:0/4242117292 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccac036040 con 0x56117f495d40 2024-09-16T01:44:58.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.079+0000 7fcc93fff640 1 -- 172.21.15.88:0/4242117292 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccac03e740 con 0x56117f495d40 2024-09-16T01:44:58.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.079+0000 7fcc93fff640 1 -- 172.21.15.88:0/4242117292 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccac03eb30 con 0x56117f495d40 2024-09-16T01:44:58.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.079+0000 7fccbc1bd740 1 -- 172.21.15.88:0/4242117292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f495d40 msgr2=0x56117f4968d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:58.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.079+0000 7fccbc1bd740 1 --2- 172.21.15.88:0/4242117292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f495d40 0x56117f4968d0 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7fccac02a700 tx=0x7fccac02e000 comp rx=0 tx=0).stop 2024-09-16T01:44:58.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.079+0000 7fccbc1bd740 1 -- 172.21.15.88:0/4242117292 shutdown_connections 2024-09-16T01:44:58.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.079+0000 7fccbc1bd740 1 --2- 172.21.15.88:0/4242117292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f495d40 0x56117f4968d0 unknown :-1 s=CLOSED pgs=2365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.079+0000 7fccbc1bd740 1 --2- 172.21.15.88:0/4242117292 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56117f474a50 0x56117f46f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.079+0000 7fccbc1bd740 1 --2- 172.21.15.88:0/4242117292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56117f4732c0 0x56117f4dc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.079+0000 7fccbc1bd740 1 -- 172.21.15.88:0/4242117292 >> 172.21.15.88:0/4242117292 conn(0x56117f470a80 msgr2=0x56117f4974d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:58.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.079+0000 7fccbc1bd740 1 -- 172.21.15.88:0/4242117292 shutdown_connections 2024-09-16T01:44:58.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.079+0000 7fccbc1bd740 1 -- 172.21.15.88:0/4242117292 wait complete. 2024-09-16T01:44:58.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.080+0000 7fccbc1bd740 1 -- start start 2024-09-16T01:44:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.080+0000 7fccbc1bd740 1 Processor -- start 2024-09-16T01:44:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fccbc1bd740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56117f4732c0 0x56117f53a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fccbc1bd740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56117f474a50 0x56117f49d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fccbc1bd740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f495d40 0x56117f535f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fccbc1bd740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x56117f46ef20 con 0x56117f495d40 2024-09-16T01:44:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fccbc1bd740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56117f4b9290 con 0x56117f474a50 2024-09-16T01:44:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fccbc1bd740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x56117f35e450 con 0x56117f4732c0 2024-09-16T01:44:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fccbc1bd740 10 client.0 _ll_register_callbacks cb 0x7fccac034070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:44:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fccb37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f495d40 0x56117f535f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fccb37fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f495d40 0x56117f535f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35988/0 (socket says 172.21.15.88:35988) 2024-09-16T01:44:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fccb37fe640 1 -- 172.21.15.88:0/3386136700 learned_addr learned my addr 172.21.15.88:0/3386136700 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:58.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fccb2ffd640 1 --2- 172.21.15.88:0/3386136700 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56117f4732c0 0x56117f53a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fccaa7fc640 1 --2- 172.21.15.88:0/3386136700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56117f474a50 0x56117f49d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fccb37fe640 1 -- 172.21.15.88:0/3386136700 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56117f4732c0 msgr2=0x56117f53a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fccb37fe640 1 --2- 172.21.15.88:0/3386136700 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56117f4732c0 0x56117f53a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fccb37fe640 1 -- 172.21.15.88:0/3386136700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56117f474a50 msgr2=0x56117f49d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fccb37fe640 1 --2- 172.21.15.88:0/3386136700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56117f474a50 0x56117f49d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fccb37fe640 1 -- 172.21.15.88:0/3386136700 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56117f4bbc80 con 0x56117f495d40 2024-09-16T01:44:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fccaa7fc640 1 --2- 172.21.15.88:0/3386136700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56117f474a50 0x56117f49d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:44:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fccb37fe640 1 --2- 172.21.15.88:0/3386136700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f495d40 0x56117f535f40 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7fccac02acb0 tx=0x7fccac004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=563ce6dc97bef04b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fcc91ffb640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:44:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fcc91ffb640 1 -- 172.21.15.88:0/3386136700 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccac036040 con 0x56117f495d40 2024-09-16T01:44:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fcc91ffb640 1 -- 172.21.15.88:0/3386136700 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(138..138 src has 1..138) ==== 5709+0+0 (secure 0 0 0) 0x7fccac003990 con 0x56117f495d40 2024-09-16T01:44:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fccbc1bd740 1 -- 172.21.15.88:0/3386136700 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fccac03e740 con 0x56117f495d40 2024-09-16T01:44:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fcc91ffb640 1 -- 172.21.15.88:0/3386136700 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=326) ==== 24+0+0 (secure 0 0 0) 0x7fccac003b90 con 0x56117f495d40 2024-09-16T01:44:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fccbc1bd740 10 client.10677 fetch_fsmap learned FSMap version 326 2024-09-16T01:44:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.081+0000 7fccbc1bd740 1 -- 172.21.15.88:0/3386136700 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=326,monmap=2+}) -- 0x56117f536900 con 0x56117f495d40 2024-09-16T01:44:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.082+0000 7fcc91ffb640 1 -- 172.21.15.88:0/3386136700 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccac03e740 con 0x56117f495d40 2024-09-16T01:44:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.082+0000 7fcc91ffb640 1 -- 172.21.15.88:0/3386136700 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 326) ==== 3502+0+0 (secure 0 0 0) 0x7fccac05f080 con 0x56117f495d40 2024-09-16T01:44:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.082+0000 7fccbc1bd740 10 client.10677 fetch_fsmap finished waiting for FSMap version 326 2024-09-16T01:44:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.082+0000 7fccbc1bd740 10 client.10677 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:44:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.082+0000 7fccbc1bd740 20 client.10677 populate_metadata read hostname 'smithi088' 2024-09-16T01:44:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.082+0000 7fccbc1bd740 1 --2- 172.21.15.88:0/3386136700 >> [v2:172.21.15.88:6834/656828696,v1:172.21.15.88:6836/656828696] conn(0x56117f4b5320 0x56117f46c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.082+0000 7fccbc1bd740 4 client.10677 mds_command: new command op to 10536 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:44:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.082+0000 7fccbc1bd740 1 -- 172.21.15.88:0/3386136700 --> [v2:172.21.15.88:6834/656828696,v1:172.21.15.88:6836/656828696] -- command(tid 0: {"prefix": "status"}) -- 0x56117f3dcfa0 con 0x56117f4b5320 2024-09-16T01:44:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.082+0000 7fccb2ffd640 1 --2- 172.21.15.88:0/3386136700 >> [v2:172.21.15.88:6834/656828696,v1:172.21.15.88:6836/656828696] conn(0x56117f4b5320 0x56117f46c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.082+0000 7fcc91ffb640 10 client.10677 ms_handle_connect on v2:172.21.15.88:6834/656828696 2024-09-16T01:44:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.082+0000 7fccb2ffd640 1 --2- 172.21.15.88:0/3386136700 >> [v2:172.21.15.88:6834/656828696,v1:172.21.15.88:6836/656828696] conn(0x56117f4b5320 0x56117f46c7d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6a25b9adfd19455 server_cookie=c6c7a5c09f8ac788 in_seq=0 out_seq=0 2024-09-16T01:44:58.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.082+0000 7fcc91ffb640 1 -- 172.21.15.88:0/3386136700 <== mds.0 v2:172.21.15.88:6834/656828696 1 ==== command_reply(tid 0: 0 ) ==== 8+0+335 (crc 0 0 0) 0x56117f3dcfa0 con 0x56117f4b5320 2024-09-16T01:44:58.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.082+0000 7fcc91ffb640 10 client.10677 handle_command_reply: tid=0 2024-09-16T01:44:58.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.082+0000 7fccbc1bd740 1 client.10677 shutdown 2024-09-16T01:44:58.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.082+0000 7fccbc1bd740 1 -- 172.21.15.88:0/3386136700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f495d40 msgr2=0x56117f535f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:58.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.082+0000 7fccbc1bd740 1 --2- 172.21.15.88:0/3386136700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f495d40 0x56117f535f40 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7fccac02acb0 tx=0x7fccac004ab0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.082+0000 7fccbc1bd740 1 -- 172.21.15.88:0/3386136700 shutdown_connections 2024-09-16T01:44:58.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.082+0000 7fccbc1bd740 1 --2- 172.21.15.88:0/3386136700 >> [v2:172.21.15.88:6834/656828696,v1:172.21.15.88:6836/656828696] conn(0x56117f4b5320 0x56117f46c7d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.082+0000 7fccbc1bd740 1 --2- 172.21.15.88:0/3386136700 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x56117f495d40 0x56117f535f40 unknown :-1 s=CLOSED pgs=2366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.082+0000 7fcc91ffb640 0 client.10677 ms_handle_reset on v2:172.21.15.88:6834/656828696 2024-09-16T01:44:58.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.082+0000 7fccbc1bd740 1 --2- 172.21.15.88:0/3386136700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56117f474a50 0x56117f49d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.082+0000 7fccbc1bd740 1 --2- 172.21.15.88:0/3386136700 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x56117f4732c0 0x56117f53a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.082+0000 7fccbc1bd740 1 -- 172.21.15.88:0/3386136700 >> 172.21.15.88:0/3386136700 conn(0x56117f470a80 msgr2=0x56117f46f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:58.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.083+0000 7fccbc1bd740 1 -- 172.21.15.88:0/3386136700 shutdown_connections 2024-09-16T01:44:58.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.083+0000 7fccbc1bd740 1 -- 172.21.15.88:0/3386136700 wait complete. 2024-09-16T01:44:58.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.083+0000 7fccbc1bd740 20 client.10677 trim_cache size 0 max 16384 2024-09-16T01:44:58.091 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:44:58.091 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "5ace2c91-46f9-482b-8a89-731fbdc171bf", 2024-09-16T01:44:58.091 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2024-09-16T01:44:58.091 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 10536, 2024-09-16T01:44:58.091 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2024-09-16T01:44:58.091 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2024-09-16T01:44:58.091 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2024-09-16T01:44:58.092 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 5.5788442150000002, 2024-09-16T01:44:58.092 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 326, 2024-09-16T01:44:58.092 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 138, 2024-09-16T01:44:58.092 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 138, 2024-09-16T01:44:58.092 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 9.9201847440000002 2024-09-16T01:44:58.092 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:44:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.083+0000 7fccb9349640 1 -- 172.21.15.88:0/3634794371 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcc9406c2b0 msgr2=0x7fcc9406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.083+0000 7fccb9349640 1 --2- 172.21.15.88:0/3634794371 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcc9406c2b0 0x7fcc9406e770 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fccb415d980 tx=0x7fcc9c008040 comp rx=0 tx=0).stop 2024-09-16T01:44:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.083+0000 7fccb9349640 1 -- 172.21.15.88:0/3634794371 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccb40b54d0 msgr2=0x7fccb41607a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.083+0000 7fccb9349640 1 --2- 172.21.15.88:0/3634794371 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccb40b54d0 0x7fccb41607a0 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7fcca400b3f0 tx=0x7fcca400b8b0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.083+0000 7fccb9349640 1 -- 172.21.15.88:0/3634794371 shutdown_connections 2024-09-16T01:44:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.083+0000 7fccb9349640 1 --2- 172.21.15.88:0/3634794371 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcc9406c2b0 0x7fcc9406e770 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.083+0000 7fccb9349640 1 --2- 172.21.15.88:0/3634794371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccb415e6f0 0x7fccb4157bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.083+0000 7fccb9349640 1 --2- 172.21.15.88:0/3634794371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fccb40b7450 0x7fccb40d0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.083+0000 7fccb9349640 1 --2- 172.21.15.88:0/3634794371 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fccb40b54d0 0x7fccb41607a0 unknown :-1 s=CLOSED pgs=2362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.083+0000 7fccb9349640 1 -- 172.21.15.88:0/3634794371 >> 172.21.15.88:0/3634794371 conn(0x7fccb40d2430 msgr2=0x7fccb40b2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:58.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.083+0000 7fccb9349640 1 -- 172.21.15.88:0/3634794371 shutdown_connections 2024-09-16T01:44:58.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.083+0000 7fccb9349640 1 -- 172.21.15.88:0/3634794371 wait complete. 2024-09-16T01:44:58.095 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:44:58.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.192+0000 7fe2e49d6640 1 Processor -- start 2024-09-16T01:44:58.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.192+0000 7fe2e49d6640 1 -- start start 2024-09-16T01:44:58.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.193+0000 7fe2e49d6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe2e0073080 0x7fe2e00b5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.193+0000 7fe2e49d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2e015c670 0x7fe2e0073460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.193+0000 7fe2e49d6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2e0071750 con 0x7fe2e00b1990 2024-09-16T01:44:58.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.193+0000 7fe2e49d6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe2e00d0c60 con 0x7fe2e0073080 2024-09-16T01:44:58.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.193+0000 7fe2e49d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe2e00d0370 con 0x7fe2e015c670 2024-09-16T01:44:58.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.194+0000 7fe2de59c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe2e0073080 0x7fe2e00b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:58.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.194+0000 7fe2ddd9b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2e015c670 0x7fe2e0073460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:58.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.194+0000 7fe2de59c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe2e0073080 0x7fe2e00b5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:50790/0 (socket says 172.21.15.88:50790) 2024-09-16T01:44:58.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.194+0000 7fe2ddd9b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2e015c670 0x7fe2e0073460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:45528/0 (socket says 172.21.15.88:45528) 2024-09-16T01:44:58.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.194+0000 7fe2de59c640 1 -- 172.21.15.88:0/2870888932 learned_addr learned my addr 172.21.15.88:0/2870888932 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:58.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.194+0000 7fe2de59c640 1 -- 172.21.15.88:0/2870888932 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2e015c670 msgr2=0x7fe2e0073460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:44:58.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.194+0000 7fe2de59c640 1 --2- 172.21.15.88:0/2870888932 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2e015c670 0x7fe2e0073460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.194+0000 7fe2de59c640 1 -- 172.21.15.88:0/2870888932 >> v1:172.21.15.154:6789/0 conn(0x7fe2e00b1990 legacy=0x7fe2e00d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:58.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.194+0000 7fe2de59c640 1 -- 172.21.15.88:0/2870888932 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2e00b59a0 con 0x7fe2e0073080 2024-09-16T01:44:58.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.194+0000 7fe2ddd9b640 1 --2- 172.21.15.88:0/2870888932 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2e015c670 0x7fe2e0073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:44:58.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.195+0000 7fe2de59c640 1 --2- 172.21.15.88:0/2870888932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2e0073080 0x7fe2e00b5460 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0009e50 tx=0x7fe2d002a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84c0853163d0e6e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:58.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.195+0000 7fe2dd59a640 1 -- 172.21.15.88:0/2870888932 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d0036070 con 0x7fe2e0073080 2024-09-16T01:44:58.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.195+0000 7fe2dd59a640 1 -- 172.21.15.88:0/2870888932 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2d0031600 con 0x7fe2e0073080 2024-09-16T01:44:58.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.195+0000 7fe2dd59a640 1 -- 172.21.15.88:0/2870888932 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d00319d0 con 0x7fe2e0073080 2024-09-16T01:44:58.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.195+0000 7fe2e49d6640 1 -- 172.21.15.88:0/2870888932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2e0073080 msgr2=0x7fe2e00b5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:58.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.195+0000 7fe2e49d6640 1 --2- 172.21.15.88:0/2870888932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2e0073080 0x7fe2e00b5460 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0009e50 tx=0x7fe2d002a710 comp rx=0 tx=0).stop 2024-09-16T01:44:58.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.196+0000 7fe2e49d6640 1 -- 172.21.15.88:0/2870888932 shutdown_connections 2024-09-16T01:44:58.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.196+0000 7fe2e49d6640 1 --2- 172.21.15.88:0/2870888932 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2e015c670 0x7fe2e0073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.196+0000 7fe2e49d6640 1 --2- 172.21.15.88:0/2870888932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2e0073080 0x7fe2e00b5460 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.196+0000 7fe2e49d6640 1 -- 172.21.15.88:0/2870888932 >> 172.21.15.88:0/2870888932 conn(0x7fe2e00d24d0 msgr2=0x7fe2e00d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:58.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.196+0000 7fe2e49d6640 1 -- 172.21.15.88:0/2870888932 shutdown_connections 2024-09-16T01:44:58.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.196+0000 7fe2e49d6640 1 -- 172.21.15.88:0/2870888932 wait complete. 2024-09-16T01:44:58.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.197+0000 7fe2e49d6640 1 Processor -- start 2024-09-16T01:44:58.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.197+0000 7fe2e49d6640 1 -- start start 2024-09-16T01:44:58.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.198+0000 7fe2e49d6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2e0073080 0x7fe2e0151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.198+0000 7fe2e49d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2e00b1990 0x7fe2e015caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.198+0000 7fe2e49d6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2e015c670 0x7fe2e0155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.198+0000 7fe2e49d6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e0071470 con 0x7fe2e0073080 2024-09-16T01:44:58.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.198+0000 7fe2e49d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e0071190 con 0x7fe2e00b1990 2024-09-16T01:44:58.201 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.198+0000 7fe2e49d6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe2e00d0610 con 0x7fe2e015c670 2024-09-16T01:44:58.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.198+0000 7fe2de59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2e0073080 0x7fe2e0151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:58.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.198+0000 7fe2de59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2e0073080 0x7fe2e0151b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:35996/0 (socket says 172.21.15.88:35996) 2024-09-16T01:44:58.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.198+0000 7fe2de59c640 1 -- 172.21.15.88:0/1967917311 learned_addr learned my addr 172.21.15.88:0/1967917311 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:44:58.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.198+0000 7fe2ddd9b640 1 --2- 172.21.15.88:0/1967917311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2e00b1990 0x7fe2e015caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:58.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.198+0000 7fe2ded9d640 1 --2- 172.21.15.88:0/1967917311 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2e015c670 0x7fe2e0155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:58.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.198+0000 7fe2de59c640 1 -- 172.21.15.88:0/1967917311 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2e015c670 msgr2=0x7fe2e0155fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:58.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.198+0000 7fe2de59c640 1 --2- 172.21.15.88:0/1967917311 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2e015c670 0x7fe2e0155fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.198+0000 7fe2de59c640 1 -- 172.21.15.88:0/1967917311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2e00b1990 msgr2=0x7fe2e015caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:58.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.198+0000 7fe2de59c640 1 --2- 172.21.15.88:0/1967917311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2e00b1990 0x7fe2e015caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.198+0000 7fe2de59c640 1 -- 172.21.15.88:0/1967917311 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2e015d380 con 0x7fe2e0073080 2024-09-16T01:44:58.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.198+0000 7fe2ddd9b640 1 --2- 172.21.15.88:0/1967917311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2e00b1990 0x7fe2e015caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:44:58.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.198+0000 7fe2de59c640 1 --2- 172.21.15.88:0/1967917311 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2e0073080 0x7fe2e0151b80 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0034040 tx=0x7fe2d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:58.202 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.198+0000 7fe2ceffd640 1 -- 172.21.15.88:0/1967917311 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d0041020 con 0x7fe2e0073080 2024-09-16T01:44:58.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.198+0000 7fe2ceffd640 1 -- 172.21.15.88:0/1967917311 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2d0007820 con 0x7fe2e0073080 2024-09-16T01:44:58.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.198+0000 7fe2e49d6640 1 -- 172.21.15.88:0/1967917311 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2e015d660 con 0x7fe2e0073080 2024-09-16T01:44:58.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.198+0000 7fe2ceffd640 1 -- 172.21.15.88:0/1967917311 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d0036040 con 0x7fe2e0073080 2024-09-16T01:44:58.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.199+0000 7fe2e49d6640 1 -- 172.21.15.88:0/1967917311 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2e015dc20 con 0x7fe2e0073080 2024-09-16T01:44:58.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.199+0000 7fe2e49d6640 1 -- 172.21.15.88:0/1967917311 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2e00d08f0 con 0x7fe2e0073080 2024-09-16T01:44:58.203 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.200+0000 7fe2ceffd640 1 -- 172.21.15.88:0/1967917311 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fe2d0044020 con 0x7fe2e0073080 2024-09-16T01:44:58.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.200+0000 7fe2ceffd640 1 --2- 172.21.15.88:0/1967917311 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe2c006c2b0 0x7fe2c006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:44:58.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.200+0000 7fe2ceffd640 1 -- 172.21.15.88:0/1967917311 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5709+0+0 (secure 0 0 0) 0x7fe2d00ac070 con 0x7fe2e0073080 2024-09-16T01:44:58.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.200+0000 7fe2ddd9b640 1 --2- 172.21.15.88:0/1967917311 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe2c006c2b0 0x7fe2c006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:44:58.204 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.201+0000 7fe2ddd9b640 1 --2- 172.21.15.88:0/1967917311 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe2c006c2b0 0x7fe2c006e770 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fe2e01f5950 tx=0x7fe2d4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:44:58.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.205+0000 7fe2ceffd640 1 -- 172.21.15.88:0/1967917311 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fe2d004b030 con 0x7fe2e0073080 2024-09-16T01:44:58.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.405+0000 7fe2e49d6640 1 -- 172.21.15.88:0/1967917311 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe2e00d3050 con 0x7fe2e0073080 2024-09-16T01:44:58.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.405+0000 7fe2ceffd640 1 -- 172.21.15.88:0/1967917311 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v326) ==== 53+0+73 (secure 0 0 0) 0x7fe2d007a870 con 0x7fe2e0073080 2024-09-16T01:44:58.407 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:44:58.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.407+0000 7fe2e49d6640 1 -- 172.21.15.88:0/1967917311 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe2c006c2b0 msgr2=0x7fe2c006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:58.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.407+0000 7fe2e49d6640 1 --2- 172.21.15.88:0/1967917311 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe2c006c2b0 0x7fe2c006e770 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fe2e01f5950 tx=0x7fe2d4008040 comp rx=0 tx=0).stop 2024-09-16T01:44:58.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.407+0000 7fe2e49d6640 1 -- 172.21.15.88:0/1967917311 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2e0073080 msgr2=0x7fe2e0151b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:44:58.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.407+0000 7fe2e49d6640 1 --2- 172.21.15.88:0/1967917311 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2e0073080 0x7fe2e0151b80 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0034040 tx=0x7fe2d0004a40 comp rx=0 tx=0).stop 2024-09-16T01:44:58.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.407+0000 7fe2e49d6640 1 -- 172.21.15.88:0/1967917311 shutdown_connections 2024-09-16T01:44:58.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.407+0000 7fe2e49d6640 1 --2- 172.21.15.88:0/1967917311 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fe2c006c2b0 0x7fe2c006e770 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.407+0000 7fe2e49d6640 1 --2- 172.21.15.88:0/1967917311 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2e015c670 0x7fe2e0155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.407+0000 7fe2e49d6640 1 --2- 172.21.15.88:0/1967917311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2e00b1990 0x7fe2e015caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.407+0000 7fe2e49d6640 1 --2- 172.21.15.88:0/1967917311 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe2e0073080 0x7fe2e0151b80 unknown :-1 s=CLOSED pgs=2367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:44:58.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.407+0000 7fe2e49d6640 1 -- 172.21.15.88:0/1967917311 >> 172.21.15.88:0/1967917311 conn(0x7fe2e00d24d0 msgr2=0x7fe2e00b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:44:58.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.407+0000 7fe2e49d6640 1 -- 172.21.15.88:0/1967917311 shutdown_connections 2024-09-16T01:44:58.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58.407+0000 7fe2e49d6640 1 -- 172.21.15.88:0/1967917311 wait complete. 2024-09-16T01:44:58.419 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-16T01:44:58.420 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:44:58.420 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2024-09-16T01:44:58.420 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T01:44:58.420 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T01:44:58.420 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-16T01:44:58.420 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-16T01:44:58.420 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:44:58.420 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:44:58.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:58.421 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2024-09-16T01:44:58.439 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:44:58.439 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:44:58.439 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:44:58.439 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:44:58.439 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-09-16T01:44:58.440 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:44:58.440 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:44:58.440 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:44:58.440 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2024-09-16T01:44:58.440 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:44:58.440 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:44:58.440 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2024-09-16T01:44:58.440 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:44:58.441 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2024-09-16T01:44:58.441 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:44:58.441 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2024-09-16T01:44:58.441 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:44:58.441 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42126sec preferred_lft 42126sec 2024-09-16T01:44:58.441 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link noprefixroute 2024-09-16T01:44:58.441 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:44:58.441 INFO:teuthology.orchestra.run.smithi088.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:44:58.441 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff 2024-09-16T01:44:58.441 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T01:44:58.441 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:44:58.441 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::e486:6eff:fe81:a1c/64 scope link 2024-09-16T01:44:58.442 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:44:58.442 INFO:teuthology.orchestra.run.smithi088.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:44:58.442 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 12:6f:29:be:95:48 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:44:58.442 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::106f:29ff:febe:9548/64 scope link 2024-09-16T01:44:58.442 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:44:58.442 INFO:teuthology.orchestra.run.smithi088.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:44:58.442 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T01:44:58.442 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::c63:b4ff:feb3:ad9e/64 scope link 2024-09-16T01:44:58.442 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:44:58.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:58.443 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:44:58.502 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-09-16T01:44:58.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:58.504 DEBUG:teuthology.orchestra.run.smithi088:> route 2024-09-16T01:44:58.560 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2024-09-16T01:44:58.560 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:44:58.560 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:44:58.560 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:44:58.560 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:44:58.561 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:44:58.561 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:44:58.561 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:44:58.561 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:44:58.561 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:44:58.561 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:44:58.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:44:58.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:44:58.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:58.687 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:44:58.702 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T01:44:58.702 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T01:44:58.702 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:44:58.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:58.702 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2024-09-16T01:44:58.767 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:44:58.767 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T01:44:58.822 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:44:58.822 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:44:58.822 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:44:58.877 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-16T01:44:58.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:58.877 DEBUG:teuthology.orchestra.run.smithi088:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:44:58.959 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:44:58.960 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:44:58.960 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:44:58.968 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:44:58.968 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:44:58.968 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:44:58.968 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:44:58.968 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:44:58.973 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:44:58.973 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:44:58.973 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:44:58.974 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:44:58.974 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:44:58.974 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790" 2024-09-16T01:44:58.974 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2024-09-16T01:44:58.974 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5ace2c91-46f9-482b-8a89-731fbdc171bf" will pass to kernel 2024-09-16T01:44:58.974 INFO:teuthology.orchestra.run.smithi088.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-16T01:44:58.974 INFO:teuthology.orchestra.run.smithi088.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:44:58.974 INFO:teuthology.orchestra.run.smithi088.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:44:58.974 INFO:teuthology.orchestra.run.smithi088.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:44:58.975 INFO:teuthology.orchestra.run.smithi088.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:44:58.976 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:44:58.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:58.976 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:44:59.041 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:44:59.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:59 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:44:59.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:44:59.140 INFO:teuthology.orchestra.run.smithi088.stdout:client10683 2024-09-16T01:44:59.141 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:44:59.141 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:44:59.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:59.156 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir1) 2024-09-16T01:44:59.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:59.217 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch dir1/file1) 2024-09-16T01:44:59.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:59.274 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir2) 2024-09-16T01:44:59.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:59.330 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch dir2/file2) 2024-09-16T01:44:59.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:59.390 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir testdir) 2024-09-16T01:44:59.446 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:59.447 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir1/file1 testdir/hardlink1) 2024-09-16T01:44:59.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:44:59.506 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir2/file2 testdir/hardlink2) 2024-09-16T01:44:59.562 DEBUG:teuthology.orchestra.run.smithi088:> stdin-killer -- python3 -c ' 2024-09-16T01:44:59.562 DEBUG:teuthology.orchestra.run.smithi088:> import os 2024-09-16T01:44:59.562 DEBUG:teuthology.orchestra.run.smithi088:> import stat 2024-09-16T01:44:59.562 DEBUG:teuthology.orchestra.run.smithi088:> 2024-09-16T01:44:59.563 DEBUG:teuthology.orchestra.run.smithi088:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir1/file1").st_ino) 2024-09-16T01:44:59.563 DEBUG:teuthology.orchestra.run.smithi088:> ' 2024-09-16T01:44:59.666 INFO:teuthology.orchestra.run.smithi088.stdout:1099511627777 2024-09-16T01:44:59.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:44:59.675 DEBUG:teuthology.orchestra.run.smithi088:> stdin-killer -- python3 -c ' 2024-09-16T01:44:59.675 DEBUG:teuthology.orchestra.run.smithi088:> import os 2024-09-16T01:44:59.675 DEBUG:teuthology.orchestra.run.smithi088:> import stat 2024-09-16T01:44:59.675 DEBUG:teuthology.orchestra.run.smithi088:> 2024-09-16T01:44:59.675 DEBUG:teuthology.orchestra.run.smithi088:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir2/file2").st_ino) 2024-09-16T01:44:59.675 DEBUG:teuthology.orchestra.run.smithi088:> ' 2024-09-16T01:44:59.780 INFO:teuthology.orchestra.run.smithi088.stdout:1099511628279 2024-09-16T01:44:59.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:44:59.788 DEBUG:teuthology.orchestra.run.smithi088:> stdin-killer -- python3 -c ' 2024-09-16T01:44:59.788 DEBUG:teuthology.orchestra.run.smithi088:> import os 2024-09-16T01:44:59.788 DEBUG:teuthology.orchestra.run.smithi088:> import stat 2024-09-16T01:44:59.788 DEBUG:teuthology.orchestra.run.smithi088:> 2024-09-16T01:44:59.789 DEBUG:teuthology.orchestra.run.smithi088:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir2").st_ino) 2024-09-16T01:44:59.789 DEBUG:teuthology.orchestra.run.smithi088:> ' 2024-09-16T01:44:59.893 INFO:teuthology.orchestra.run.smithi088.stdout:1099511628278 2024-09-16T01:44:59.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:44:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:44:59.902 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:44:59.902 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:44:59.958 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:44:59.958 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:45:00.014 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:45:00.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.151+0000 7fd04795b640 1 Processor -- start 2024-09-16T01:45:00.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.152+0000 7fd04795b640 1 -- start start 2024-09-16T01:45:00.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.152+0000 7fd04795b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd04015e7a0 0x7fd04006caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:00.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.152+0000 7fd04795b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd0400cf1f0 0x7fd0400b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:00.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.152+0000 7fd04795b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0400cf860 con 0x7fd0400b6720 2024-09-16T01:45:00.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.152+0000 7fd04795b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd0400b6b90 con 0x7fd04015e7a0 2024-09-16T01:45:00.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.152+0000 7fd04795b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd0400d2ca0 con 0x7fd0400cf1f0 2024-09-16T01:45:00.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.153+0000 7fd0456f7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd04015e7a0 0x7fd04006caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:00.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.153+0000 7fd0456f7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd04015e7a0 0x7fd04006caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36012/0 (socket says 172.21.15.88:36012) 2024-09-16T01:45:00.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.153+0000 7fd0456f7640 1 -- 172.21.15.88:0/3667321699 learned_addr learned my addr 172.21.15.88:0/3667321699 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:00.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.153+0000 7fd0456f7640 1 -- 172.21.15.88:0/3667321699 >> v2:172.21.15.154:3300/0 conn(0x7fd0400cf1f0 msgr2=0x7fd0400b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:00.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.153+0000 7fd0456f7640 1 --2- 172.21.15.88:0/3667321699 >> v2:172.21.15.154:3300/0 conn(0x7fd0400cf1f0 0x7fd0400b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:00.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.153+0000 7fd0456f7640 1 -- 172.21.15.88:0/3667321699 >> v1:172.21.15.154:6789/0 conn(0x7fd0400b6720 legacy=0x7fd0401623c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:00.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.153+0000 7fd0456f7640 1 -- 172.21.15.88:0/3667321699 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0400cfb40 con 0x7fd04015e7a0 2024-09-16T01:45:00.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.154+0000 7fd0456f7640 1 --2- 172.21.15.88:0/3667321699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd04015e7a0 0x7fd04006caa0 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7fd0400d2110 tx=0x7fd03802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb1f0404d60c4df server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:00.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.154+0000 7fd03ffff640 1 -- 172.21.15.88:0/3667321699 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd038037070 con 0x7fd04015e7a0 2024-09-16T01:45:00.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.154+0000 7fd03ffff640 1 -- 172.21.15.88:0/3667321699 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd038032440 con 0x7fd04015e7a0 2024-09-16T01:45:00.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.154+0000 7fd03ffff640 1 -- 172.21.15.88:0/3667321699 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd038032810 con 0x7fd04015e7a0 2024-09-16T01:45:00.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.154+0000 7fd04795b640 1 -- 172.21.15.88:0/3667321699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd04015e7a0 msgr2=0x7fd04006caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:00.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.154+0000 7fd04795b640 1 --2- 172.21.15.88:0/3667321699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd04015e7a0 0x7fd04006caa0 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7fd0400d2110 tx=0x7fd03802a1a0 comp rx=0 tx=0).stop 2024-09-16T01:45:00.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.154+0000 7fd04795b640 1 -- 172.21.15.88:0/3667321699 shutdown_connections 2024-09-16T01:45:00.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.154+0000 7fd04795b640 1 --2- 172.21.15.88:0/3667321699 >> v2:172.21.15.154:3300/0 conn(0x7fd0400cf1f0 0x7fd0400b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:00.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.154+0000 7fd04795b640 1 --2- 172.21.15.88:0/3667321699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd04015e7a0 0x7fd04006caa0 unknown :-1 s=CLOSED pgs=2369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:00.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.154+0000 7fd04795b640 1 -- 172.21.15.88:0/3667321699 >> 172.21.15.88:0/3667321699 conn(0x7fd0400d11e0 msgr2=0x7fd0400d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:00.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.154+0000 7fd04795b640 1 -- 172.21.15.88:0/3667321699 shutdown_connections 2024-09-16T01:45:00.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.154+0000 7fd04795b640 1 -- 172.21.15.88:0/3667321699 wait complete. 2024-09-16T01:45:00.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.156+0000 7fd04795b640 1 Processor -- start 2024-09-16T01:45:00.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.156+0000 7fd04795b640 1 -- start start 2024-09-16T01:45:00.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.156+0000 7fd04795b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0400b6720 0x7fd040150a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:00.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.156+0000 7fd04795b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0400cf1f0 0x7fd0401f7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:00.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.156+0000 7fd04795b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd04015e7a0 0x7fd0400ca190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:00.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.156+0000 7fd04795b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd0400b7a00 con 0x7fd0400b6720 2024-09-16T01:45:00.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.156+0000 7fd04795b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd0400d2ef0 con 0x7fd0400cf1f0 2024-09-16T01:45:00.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.156+0000 7fd04795b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd04006cfe0 con 0x7fd04015e7a0 2024-09-16T01:45:00.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.156+0000 7fd0456f7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0400b6720 0x7fd040150a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:00.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.156+0000 7fd0456f7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0400b6720 0x7fd040150a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:36022/0 (socket says 172.21.15.88:36022) 2024-09-16T01:45:00.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.156+0000 7fd0456f7640 1 -- 172.21.15.88:0/976076467 learned_addr learned my addr 172.21.15.88:0/976076467 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:00.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.157+0000 7fd044ef6640 1 --2- 172.21.15.88:0/976076467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0400cf1f0 0x7fd0401f7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:00.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.157+0000 7fd045ef8640 1 --2- 172.21.15.88:0/976076467 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd04015e7a0 0x7fd0400ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:00.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.157+0000 7fd0456f7640 1 -- 172.21.15.88:0/976076467 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd04015e7a0 msgr2=0x7fd0400ca190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:00.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.157+0000 7fd0456f7640 1 --2- 172.21.15.88:0/976076467 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd04015e7a0 0x7fd0400ca190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:00.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.157+0000 7fd0456f7640 1 -- 172.21.15.88:0/976076467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0400cf1f0 msgr2=0x7fd0401f7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:00.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.157+0000 7fd0456f7640 1 --2- 172.21.15.88:0/976076467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0400cf1f0 0x7fd0401f7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:00.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.157+0000 7fd0456f7640 1 -- 172.21.15.88:0/976076467 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0401f84e0 con 0x7fd0400b6720 2024-09-16T01:45:00.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.157+0000 7fd044ef6640 1 --2- 172.21.15.88:0/976076467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0400cf1f0 0x7fd0401f7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:45:00.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.157+0000 7fd0456f7640 1 --2- 172.21.15.88:0/976076467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0400b6720 0x7fd040150a20 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7fd038000c00 tx=0x7fd0380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:00.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.157+0000 7fd03dffb640 1 -- 172.21.15.88:0/976076467 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd038041070 con 0x7fd0400b6720 2024-09-16T01:45:00.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.157+0000 7fd03dffb640 1 -- 172.21.15.88:0/976076467 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd03803c870 con 0x7fd0400b6720 2024-09-16T01:45:00.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.157+0000 7fd04795b640 1 -- 172.21.15.88:0/976076467 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd04015ed90 con 0x7fd0400b6720 2024-09-16T01:45:00.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.157+0000 7fd03dffb640 1 -- 172.21.15.88:0/976076467 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd038037040 con 0x7fd0400b6720 2024-09-16T01:45:00.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.157+0000 7fd04795b640 1 -- 172.21.15.88:0/976076467 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0401f8a40 con 0x7fd0400b6720 2024-09-16T01:45:00.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.158+0000 7fd04795b640 1 -- 172.21.15.88:0/976076467 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd04015eb80 con 0x7fd0400b6720 2024-09-16T01:45:00.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.159+0000 7fd03dffb640 1 -- 172.21.15.88:0/976076467 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd038044070 con 0x7fd0400b6720 2024-09-16T01:45:00.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.159+0000 7fd03dffb640 1 --2- 172.21.15.88:0/976076467 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd02c06c2b0 0x7fd02c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:00.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.159+0000 7fd03dffb640 1 -- 172.21.15.88:0/976076467 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5709+0+0 (secure 0 0 0) 0x7fd0380ac1c0 con 0x7fd0400b6720 2024-09-16T01:45:00.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.159+0000 7fd044ef6640 1 --2- 172.21.15.88:0/976076467 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd02c06c2b0 0x7fd02c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:00.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.160+0000 7fd044ef6640 1 --2- 172.21.15.88:0/976076467 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd02c06c2b0 0x7fd02c06e770 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fd0401611e0 tx=0x7fd034009570 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:00.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.164+0000 7fd03dffb640 1 -- 172.21.15.88:0/976076467 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fd0401f84e0 con 0x7fd0400b6720 2024-09-16T01:45:00.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.341+0000 7fd04795b640 1 -- 172.21.15.88:0/976076467 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd0400c9f70 con 0x7fd0400b6720 2024-09-16T01:45:00.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.341+0000 7fd03dffb640 1 -- 172.21.15.88:0/976076467 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v138) ==== 80+0+0 (secure 0 0 0) 0x7fd03807aae0 con 0x7fd0400b6720 2024-09-16T01:45:00.343 INFO:teuthology.orchestra.run.smithi088.stderr:listed 0 entries 2024-09-16T01:45:00.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.343+0000 7fd04795b640 1 -- 172.21.15.88:0/976076467 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd02c06c2b0 msgr2=0x7fd02c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:00.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.343+0000 7fd04795b640 1 --2- 172.21.15.88:0/976076467 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd02c06c2b0 0x7fd02c06e770 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fd0401611e0 tx=0x7fd034009570 comp rx=0 tx=0).stop 2024-09-16T01:45:00.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.343+0000 7fd04795b640 1 -- 172.21.15.88:0/976076467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0400b6720 msgr2=0x7fd040150a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:00.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.343+0000 7fd04795b640 1 --2- 172.21.15.88:0/976076467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0400b6720 0x7fd040150a20 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7fd038000c00 tx=0x7fd0380046c0 comp rx=0 tx=0).stop 2024-09-16T01:45:00.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.343+0000 7fd04795b640 1 -- 172.21.15.88:0/976076467 shutdown_connections 2024-09-16T01:45:00.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.343+0000 7fd04795b640 1 --2- 172.21.15.88:0/976076467 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd02c06c2b0 0x7fd02c06e770 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:00.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.343+0000 7fd04795b640 1 --2- 172.21.15.88:0/976076467 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd04015e7a0 0x7fd0400ca190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:00.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.343+0000 7fd04795b640 1 --2- 172.21.15.88:0/976076467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0400cf1f0 0x7fd0401f7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:00.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.343+0000 7fd04795b640 1 --2- 172.21.15.88:0/976076467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0400b6720 0x7fd040150a20 unknown :-1 s=CLOSED pgs=2370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:00.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.343+0000 7fd04795b640 1 -- 172.21.15.88:0/976076467 >> 172.21.15.88:0/976076467 conn(0x7fd0400d11e0 msgr2=0x7fd04015d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:00.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.343+0000 7fd04795b640 1 -- 172.21.15.88:0/976076467 shutdown_connections 2024-09-16T01:45:00.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.343+0000 7fd04795b640 1 -- 172.21.15.88:0/976076467 wait complete. 2024-09-16T01:45:00.355 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-16T01:45:00.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:45:00.356 DEBUG:teuthology.orchestra.run.smithi088:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-16T01:45:00.399 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:45:00.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:45:00.400 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:45:00.456 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:45:00.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.596+0000 7f9514a7b640 1 Processor -- start 2024-09-16T01:45:00.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.596+0000 7f9514a7b640 1 -- start start 2024-09-16T01:45:00.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.597+0000 7f9514a7b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f951015c550 0x7f95100b5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:00.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.597+0000 7f9514a7b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95100b6310 0x7f95100b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:00.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.597+0000 7f9514a7b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95100b4cf0 con 0x7f951015c170 2024-09-16T01:45:00.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.597+0000 7f9514a7b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f95100b4fd0 con 0x7f951015c550 2024-09-16T01:45:00.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.597+0000 7f9514a7b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f95100b52b0 con 0x7f95100b6310 2024-09-16T01:45:00.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.597+0000 7f950e59c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f951015c170 0x7f95100cf6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35414/0 (socket says 172.21.15.88:35414) 2024-09-16T01:45:00.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.597+0000 7f950e59c640 1 -- 172.21.15.88:0/1400117077 learned_addr learned my addr 172.21.15.88:0/1400117077 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:00.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.597+0000 7f950d59a640 1 -- 172.21.15.88:0/1400117077 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1659934021 0 0) 0x7f95100b4cf0 con 0x7f951015c170 2024-09-16T01:45:00.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.597+0000 7f950dd9b640 1 --2- 172.21.15.88:0/1400117077 >> v2:172.21.15.154:3300/0 conn(0x7f951015c550 0x7f95100b5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:00.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.597+0000 7f950ed9d640 1 --2- 172.21.15.88:0/1400117077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95100b6310 0x7f95100b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:00.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.597+0000 7f950d59a640 1 -- 172.21.15.88:0/1400117077 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94f8003520 con 0x7f951015c170 2024-09-16T01:45:00.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.597+0000 7f950d59a640 1 -- 172.21.15.88:0/1400117077 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3426804047 0 0) 0x7f94f8003520 con 0x7f951015c170 2024-09-16T01:45:00.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.597+0000 7f950d59a640 1 -- 172.21.15.88:0/1400117077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95100b6310 msgr2=0x7f95100b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:00.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.597+0000 7f950d59a640 1 --2- 172.21.15.88:0/1400117077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95100b6310 0x7f95100b39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:00.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.597+0000 7f950d59a640 1 -- 172.21.15.88:0/1400117077 >> v2:172.21.15.154:3300/0 conn(0x7f951015c550 msgr2=0x7f95100b5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:00.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.597+0000 7f950d59a640 1 --2- 172.21.15.88:0/1400117077 >> v2:172.21.15.154:3300/0 conn(0x7f951015c550 0x7f95100b5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:00.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.597+0000 7f950d59a640 1 -- 172.21.15.88:0/1400117077 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95100b5590 con 0x7f951015c170 2024-09-16T01:45:00.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.597+0000 7f950d59a640 1 -- 172.21.15.88:0/1400117077 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f9500001670 con 0x7f951015c170 2024-09-16T01:45:00.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.598+0000 7f950d59a640 1 -- 172.21.15.88:0/1400117077 >> v1:172.21.15.88:6789/0 conn(0x7f951015c170 legacy=0x7f95100cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:00.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.598+0000 7f950d59a640 1 --2- 172.21.15.88:0/1400117077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94f80039b0 0x7f94f8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:00.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.598+0000 7f950d59a640 1 --2- 172.21.15.88:0/1400117077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95100b6310 0x7f94f8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:00.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.598+0000 7f950d59a640 1 --2- 172.21.15.88:0/1400117077 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94f8004d40 0x7f94f80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:00.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.598+0000 7f950d59a640 1 -- 172.21.15.88:0/1400117077 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f95100b4fd0 con 0x7f94f8004d40 2024-09-16T01:45:00.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.598+0000 7f950d59a640 1 -- 172.21.15.88:0/1400117077 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f95100b52b0 con 0x7f94f80039b0 2024-09-16T01:45:00.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.598+0000 7f950d59a640 1 -- 172.21.15.88:0/1400117077 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f94f800b9c0 con 0x7f95100b6310 2024-09-16T01:45:00.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.598+0000 7f950dd9b640 1 --2- 172.21.15.88:0/1400117077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95100b6310 0x7f94f8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:00.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.598+0000 7f950e59c640 1 --2- 172.21.15.88:0/1400117077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94f80039b0 0x7f94f8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:00.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.598+0000 7f950ed9d640 1 --2- 172.21.15.88:0/1400117077 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94f8004d40 0x7f94f80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:00.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.598+0000 7f950dd9b640 1 -- 172.21.15.88:0/1400117077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94f80039b0 msgr2=0x7f94f8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:00.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.598+0000 7f950dd9b640 1 --2- 172.21.15.88:0/1400117077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94f80039b0 0x7f94f8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:00.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.598+0000 7f950dd9b640 1 -- 172.21.15.88:0/1400117077 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94f8004d40 msgr2=0x7f94f80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:00.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.598+0000 7f950dd9b640 1 --2- 172.21.15.88:0/1400117077 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94f8004d40 0x7f94f80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:00.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.598+0000 7f950dd9b640 1 -- 172.21.15.88:0/1400117077 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94f8003520 con 0x7f95100b6310 2024-09-16T01:45:00.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.598+0000 7f950e59c640 1 --2- 172.21.15.88:0/1400117077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94f80039b0 0x7f94f8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:45:00.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.598+0000 7f950ed9d640 1 --2- 172.21.15.88:0/1400117077 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94f8004d40 0x7f94f80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:45:00.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.599+0000 7f950dd9b640 1 --2- 172.21.15.88:0/1400117077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95100b6310 0x7f94f8004800 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9504009ac0 tx=0x7f94f8005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=141e22e33c90d68d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:00.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.599+0000 7f950d59a640 1 -- 172.21.15.88:0/1400117077 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9504036070 con 0x7f95100b6310 2024-09-16T01:45:00.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.599+0000 7f950d59a640 1 -- 172.21.15.88:0/1400117077 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9504031400 con 0x7f95100b6310 2024-09-16T01:45:00.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.599+0000 7f950d59a640 1 -- 172.21.15.88:0/1400117077 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9504009690 con 0x7f95100b6310 2024-09-16T01:45:00.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.600+0000 7f9514a7b640 1 -- 172.21.15.88:0/1400117077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95100b6310 msgr2=0x7f94f8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:00.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.600+0000 7f9514a7b640 1 --2- 172.21.15.88:0/1400117077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95100b6310 0x7f94f8004800 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9504009ac0 tx=0x7f94f8005830 comp rx=0 tx=0).stop 2024-09-16T01:45:00.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.600+0000 7f950ed9d640 1 -- 172.21.15.88:0/1400117077 reap_dead start 2024-09-16T01:45:00.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.600+0000 7f9514a7b640 1 -- 172.21.15.88:0/1400117077 shutdown_connections 2024-09-16T01:45:00.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.600+0000 7f9514a7b640 1 -- 172.21.15.88:0/1400117077 >> 172.21.15.88:0/1400117077 conn(0x7f951006bc10 msgr2=0x7f951006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:00.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.600+0000 7f9514a7b640 1 -- 172.21.15.88:0/1400117077 shutdown_connections 2024-09-16T01:45:00.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.600+0000 7f9514a7b640 1 -- 172.21.15.88:0/1400117077 wait complete. 2024-09-16T01:45:00.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.601+0000 7f9514a7b640 1 Processor -- start 2024-09-16T01:45:00.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.601+0000 7f9514a7b640 1 -- start start 2024-09-16T01:45:00.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.602+0000 7f9514a7b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95100b6310 0x7f95100c7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:00.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.602+0000 7f9514a7b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95100c8cc0 0x7f951014def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:00.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.602+0000 7f9514a7b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95100b3840 0x7f95100b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:00.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.602+0000 7f9514a7b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f951015c9c0 con 0x7f95100c8cc0 2024-09-16T01:45:00.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.602+0000 7f9514a7b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f95100b5b50 con 0x7f95100b6310 2024-09-16T01:45:00.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.602+0000 7f9514a7b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f95100cfdd0 con 0x7f95100b3840 2024-09-16T01:45:00.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.602+0000 7f950dd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95100c8cc0 0x7f951014def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:00.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.602+0000 7f950dd9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95100c8cc0 0x7f951014def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54668/0 (socket says 172.21.15.88:54668) 2024-09-16T01:45:00.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.602+0000 7f950dd9b640 1 -- 172.21.15.88:0/2693983790 learned_addr learned my addr 172.21.15.88:0/2693983790 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:00.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.602+0000 7f950e59c640 1 --2- 172.21.15.88:0/2693983790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95100b6310 0x7f95100c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:00.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.602+0000 7f950ed9d640 1 --2- 172.21.15.88:0/2693983790 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95100b3840 0x7f95100b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:00.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.602+0000 7f950dd9b640 1 -- 172.21.15.88:0/2693983790 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95100b3840 msgr2=0x7f95100b3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:00.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.602+0000 7f950dd9b640 1 --2- 172.21.15.88:0/2693983790 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95100b3840 0x7f95100b3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:00.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.602+0000 7f950dd9b640 1 -- 172.21.15.88:0/2693983790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95100b6310 msgr2=0x7f95100c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:00.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.602+0000 7f950dd9b640 1 --2- 172.21.15.88:0/2693983790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95100b6310 0x7f95100c7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:00.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.602+0000 7f950dd9b640 1 -- 172.21.15.88:0/2693983790 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95100c7ec0 con 0x7f95100c8cc0 2024-09-16T01:45:00.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.602+0000 7f950ed9d640 1 --2- 172.21.15.88:0/2693983790 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95100b3840 0x7f95100b3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:45:00.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.603+0000 7f950dd9b640 1 --2- 172.21.15.88:0/2693983790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95100c8cc0 0x7f951014def0 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f9504009be0 tx=0x7f9504004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:00.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.603+0000 7f94feffd640 1 -- 172.21.15.88:0/2693983790 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9504040020 con 0x7f95100c8cc0 2024-09-16T01:45:00.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.603+0000 7f94feffd640 1 -- 172.21.15.88:0/2693983790 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9504003950 con 0x7f95100c8cc0 2024-09-16T01:45:00.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.603+0000 7f9514a7b640 1 -- 172.21.15.88:0/2693983790 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95100c81a0 con 0x7f95100c8cc0 2024-09-16T01:45:00.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.603+0000 7f94feffd640 1 -- 172.21.15.88:0/2693983790 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9504036040 con 0x7f95100c8cc0 2024-09-16T01:45:00.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.603+0000 7f9514a7b640 1 -- 172.21.15.88:0/2693983790 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95100c8760 con 0x7f95100c8cc0 2024-09-16T01:45:00.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.604+0000 7f9514a7b640 1 -- 172.21.15.88:0/2693983790 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95100b5dd0 con 0x7f95100c8cc0 2024-09-16T01:45:00.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.604+0000 7f94feffd640 1 -- 172.21.15.88:0/2693983790 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9504043020 con 0x7f95100c8cc0 2024-09-16T01:45:00.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.605+0000 7f94feffd640 1 --2- 172.21.15.88:0/2693983790 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f94f006c2e0 0x7f94f006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:00.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.605+0000 7f94feffd640 1 -- 172.21.15.88:0/2693983790 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5709+0+0 (secure 0 0 0) 0x7f95040ab550 con 0x7f95100c8cc0 2024-09-16T01:45:00.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.605+0000 7f950e59c640 1 --2- 172.21.15.88:0/2693983790 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f94f006c2e0 0x7f94f006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:00.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.605+0000 7f950e59c640 1 --2- 172.21.15.88:0/2693983790 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f94f006c2e0 0x7f94f006e7a0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f9500001610 tx=0x7f9500009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:00.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.609+0000 7f94feffd640 1 -- 172.21.15.88:0/2693983790 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f950404a070 con 0x7f95100c8cc0 2024-09-16T01:45:00.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.809+0000 7f9514a7b640 1 -- 172.21.15.88:0/2693983790 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f951015d560 con 0x7f95100c8cc0 2024-09-16T01:45:00.811 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.810+0000 7f94feffd640 1 -- 172.21.15.88:0/2693983790 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 326 v326) ==== 95+0+5239 (secure 0 0 0) 0x7f9504079d50 con 0x7f95100c8cc0 2024-09-16T01:45:00.811 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:45:00.811 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":326,"btime":"2024-09-16T01:44:55:534079+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7649,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2706550508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2706550508},{"type":"v1","addr":"172.21.15.154:6834","nonce":2706550508}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322},{"gid":8569,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/3460839382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":3460839382},{"type":"v1","addr":"172.21.15.88:6837","nonce":3460839382}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322},{"gid":8581,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2822970755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2822970755},{"type":"v1","addr":"172.21.15.154:6835","nonce":2822970755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322}],"filesystems":[{"mdsmap":{"epoch":326,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:44:51.845091+0000","modified":"2024-09-16T01:44:54.630127+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10536},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10536":{"gid":10536,"name":"a","rank":0,"incarnation":323,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6836/656828696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":656828696},{"type":"v1","addr":"172.21.15.88:6836","nonce":656828696}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10536,"qdb_cluster":[10536]},"id":7}]} 2024-09-16T01:45:00.812 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 326 2024-09-16T01:45:00.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.812+0000 7f9514a7b640 1 -- 172.21.15.88:0/2693983790 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f94f006c2e0 msgr2=0x7f94f006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:00.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.812+0000 7f9514a7b640 1 --2- 172.21.15.88:0/2693983790 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f94f006c2e0 0x7f94f006e7a0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f9500001610 tx=0x7f9500009040 comp rx=0 tx=0).stop 2024-09-16T01:45:00.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.812+0000 7f9514a7b640 1 -- 172.21.15.88:0/2693983790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95100c8cc0 msgr2=0x7f951014def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:00.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.812+0000 7f9514a7b640 1 --2- 172.21.15.88:0/2693983790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95100c8cc0 0x7f951014def0 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f9504009be0 tx=0x7f9504004730 comp rx=0 tx=0).stop 2024-09-16T01:45:00.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.812+0000 7f9514a7b640 1 -- 172.21.15.88:0/2693983790 shutdown_connections 2024-09-16T01:45:00.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.812+0000 7f9514a7b640 1 --2- 172.21.15.88:0/2693983790 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f94f006c2e0 0x7f94f006e7a0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:00.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.812+0000 7f9514a7b640 1 --2- 172.21.15.88:0/2693983790 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95100b3840 0x7f95100b3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:00.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.812+0000 7f9514a7b640 1 --2- 172.21.15.88:0/2693983790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95100c8cc0 0x7f951014def0 unknown :-1 s=CLOSED pgs=2372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:00.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.812+0000 7f9514a7b640 1 --2- 172.21.15.88:0/2693983790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95100b6310 0x7f95100c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:00.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.812+0000 7f9514a7b640 1 -- 172.21.15.88:0/2693983790 >> 172.21.15.88:0/2693983790 conn(0x7f951006bc10 msgr2=0x7f95100b2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:00.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.812+0000 7f9514a7b640 1 -- 172.21.15.88:0/2693983790 shutdown_connections 2024-09-16T01:45:00.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:00.812+0000 7f9514a7b640 1 -- 172.21.15.88:0/2693983790 wait complete. 2024-09-16T01:45:00.824 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-09-16T01:45:00.949 INFO:teuthology.orchestra.run.smithi088.stdout:{"message":"","return_code":0} 2024-09-16T01:45:00.950 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-16T01:45:00.950 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:45:01.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.048+0000 7fc23495f640 1 Processor -- start 2024-09-16T01:45:01.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.048+0000 7fc23495f640 1 -- start start 2024-09-16T01:45:01.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.049+0000 7fc23495f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc2300720d0 0x7fc2300b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.049+0000 7fc23495f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2300cca40 0x7fc2300b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.049+0000 7fc23495f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc230071720 con 0x7fc23015ad10 2024-09-16T01:45:01.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.049+0000 7fc23495f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc230070fb0 con 0x7fc2300720d0 2024-09-16T01:45:01.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.049+0000 7fc23495f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc230073920 con 0x7fc2300cca40 2024-09-16T01:45:01.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.049+0000 7fc22e59c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc23015ad10 0x7fc2300719c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35422/0 (socket says 172.21.15.88:35422) 2024-09-16T01:45:01.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.049+0000 7fc22e59c640 1 -- 172.21.15.88:0/3034855587 learned_addr learned my addr 172.21.15.88:0/3034855587 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:01.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.049+0000 7fc22dd9b640 1 --2- 172.21.15.88:0/3034855587 >> v2:172.21.15.154:3300/0 conn(0x7fc2300720d0 0x7fc2300b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.049+0000 7fc22ed9d640 1 --2- 172.21.15.88:0/3034855587 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2300cca40 0x7fc2300b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.049+0000 7fc22d59a640 1 -- 172.21.15.88:0/3034855587 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3486222110 0 0) 0x7fc230071720 con 0x7fc23015ad10 2024-09-16T01:45:01.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.049+0000 7fc22d59a640 1 -- 172.21.15.88:0/3034855587 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc218003520 con 0x7fc23015ad10 2024-09-16T01:45:01.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.050+0000 7fc22d59a640 1 -- 172.21.15.88:0/3034855587 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3061919130 0 0) 0x7fc218003520 con 0x7fc23015ad10 2024-09-16T01:45:01.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.050+0000 7fc22d59a640 1 -- 172.21.15.88:0/3034855587 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2300cca40 msgr2=0x7fc2300b43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:01.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.050+0000 7fc22d59a640 1 --2- 172.21.15.88:0/3034855587 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2300cca40 0x7fc2300b43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.050+0000 7fc22d59a640 1 -- 172.21.15.88:0/3034855587 >> v2:172.21.15.154:3300/0 conn(0x7fc2300720d0 msgr2=0x7fc2300b3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:01.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.050+0000 7fc22d59a640 1 --2- 172.21.15.88:0/3034855587 >> v2:172.21.15.154:3300/0 conn(0x7fc2300720d0 0x7fc2300b3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.050+0000 7fc22d59a640 1 -- 172.21.15.88:0/3034855587 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc230073c00 con 0x7fc23015ad10 2024-09-16T01:45:01.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.050+0000 7fc22d59a640 1 -- 172.21.15.88:0/3034855587 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fc220001670 con 0x7fc23015ad10 2024-09-16T01:45:01.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.050+0000 7fc22d59a640 1 -- 172.21.15.88:0/3034855587 >> v1:172.21.15.88:6789/0 conn(0x7fc23015ad10 legacy=0x7fc2300719c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:01.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.050+0000 7fc22d59a640 1 --2- 172.21.15.88:0/3034855587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2180039b0 0x7fc218003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.050+0000 7fc22d59a640 1 --2- 172.21.15.88:0/3034855587 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2300cca40 0x7fc218004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.050+0000 7fc22d59a640 1 --2- 172.21.15.88:0/3034855587 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc218004d40 0x7fc2180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.050+0000 7fc22d59a640 1 -- 172.21.15.88:0/3034855587 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc230070fb0 con 0x7fc218004d40 2024-09-16T01:45:01.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.050+0000 7fc22d59a640 1 -- 172.21.15.88:0/3034855587 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc230073920 con 0x7fc2180039b0 2024-09-16T01:45:01.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.050+0000 7fc22d59a640 1 -- 172.21.15.88:0/3034855587 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc21800b9f0 con 0x7fc2300cca40 2024-09-16T01:45:01.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.050+0000 7fc22ed9d640 1 --2- 172.21.15.88:0/3034855587 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc218004d40 0x7fc2180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.050+0000 7fc22e59c640 1 --2- 172.21.15.88:0/3034855587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2180039b0 0x7fc218003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.050+0000 7fc22dd9b640 1 --2- 172.21.15.88:0/3034855587 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2300cca40 0x7fc218004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.050+0000 7fc22ed9d640 1 -- 172.21.15.88:0/3034855587 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2300cca40 msgr2=0x7fc218004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:01.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.050+0000 7fc22ed9d640 1 --2- 172.21.15.88:0/3034855587 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2300cca40 0x7fc218004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.050+0000 7fc22ed9d640 1 -- 172.21.15.88:0/3034855587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2180039b0 msgr2=0x7fc218003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:01.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.050+0000 7fc22ed9d640 1 --2- 172.21.15.88:0/3034855587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc2180039b0 0x7fc218003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.050+0000 7fc22ed9d640 1 -- 172.21.15.88:0/3034855587 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc218003520 con 0x7fc218004d40 2024-09-16T01:45:01.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.051+0000 7fc22ed9d640 1 --2- 172.21.15.88:0/3034855587 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc218004d40 0x7fc2180051c0 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7fc228009fc0 tx=0x7fc218005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f9a57a51d786f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:01.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.051+0000 7fc22d59a640 1 -- 172.21.15.88:0/3034855587 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc228009c10 con 0x7fc218004d40 2024-09-16T01:45:01.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.051+0000 7fc22d59a640 1 -- 172.21.15.88:0/3034855587 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc22802e610 con 0x7fc218004d40 2024-09-16T01:45:01.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.051+0000 7fc22d59a640 1 -- 172.21.15.88:0/3034855587 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc22802ea00 con 0x7fc218004d40 2024-09-16T01:45:01.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.051+0000 7fc23495f640 1 -- 172.21.15.88:0/3034855587 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc218004d40 msgr2=0x7fc2180051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:01.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.051+0000 7fc23495f640 1 --2- 172.21.15.88:0/3034855587 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc218004d40 0x7fc2180051c0 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7fc228009fc0 tx=0x7fc218005830 comp rx=0 tx=0).stop 2024-09-16T01:45:01.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.051+0000 7fc22ed9d640 1 -- 172.21.15.88:0/3034855587 reap_dead start 2024-09-16T01:45:01.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.052+0000 7fc23495f640 1 -- 172.21.15.88:0/3034855587 shutdown_connections 2024-09-16T01:45:01.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.052+0000 7fc23495f640 1 -- 172.21.15.88:0/3034855587 >> 172.21.15.88:0/3034855587 conn(0x7fc23006d0f0 msgr2=0x7fc23006d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:01.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.052+0000 7fc23495f640 1 -- 172.21.15.88:0/3034855587 shutdown_connections 2024-09-16T01:45:01.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.052+0000 7fc23495f640 1 -- 172.21.15.88:0/3034855587 wait complete. 2024-09-16T01:45:01.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.053+0000 7fc23495f640 1 Processor -- start 2024-09-16T01:45:01.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.053+0000 7fc23495f640 1 -- start start 2024-09-16T01:45:01.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.054+0000 7fc23495f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc218004d40 0x7fc230156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.054+0000 7fc23495f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2301537c0 0x7fc2300c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.054+0000 7fc23495f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc230153c20 0x7fc2300c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.054+0000 7fc23495f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc2300cc360 con 0x7fc218004d40 2024-09-16T01:45:01.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.054+0000 7fc23495f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc230071200 con 0x7fc230153c20 2024-09-16T01:45:01.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.054+0000 7fc23495f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc2300cc4e0 con 0x7fc2301537c0 2024-09-16T01:45:01.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.054+0000 7fc22e59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc218004d40 0x7fc230156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.054+0000 7fc22e59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc218004d40 0x7fc230156f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54692/0 (socket says 172.21.15.88:54692) 2024-09-16T01:45:01.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.054+0000 7fc22e59c640 1 -- 172.21.15.88:0/1228652408 learned_addr learned my addr 172.21.15.88:0/1228652408 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:01.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.054+0000 7fc22e59c640 1 -- 172.21.15.88:0/1228652408 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2301537c0 msgr2=0x7fc2300c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:01.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.054+0000 7fc22e59c640 1 --2- 172.21.15.88:0/1228652408 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2301537c0 0x7fc2300c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.054+0000 7fc22e59c640 1 -- 172.21.15.88:0/1228652408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc230153c20 msgr2=0x7fc2300c9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:01.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.054+0000 7fc22e59c640 1 --2- 172.21.15.88:0/1228652408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc230153c20 0x7fc2300c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.054+0000 7fc22e59c640 1 -- 172.21.15.88:0/1228652408 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc230155ec0 con 0x7fc218004d40 2024-09-16T01:45:01.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.054+0000 7fc22e59c640 1 --2- 172.21.15.88:0/1228652408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc218004d40 0x7fc230156f60 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7fc220007310 tx=0x7fc2200077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:01.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.054+0000 7fc21effd640 1 -- 172.21.15.88:0/1228652408 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc22000f8e0 con 0x7fc218004d40 2024-09-16T01:45:01.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.054+0000 7fc21effd640 1 -- 172.21.15.88:0/1228652408 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc220018070 con 0x7fc218004d40 2024-09-16T01:45:01.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.054+0000 7fc21effd640 1 -- 172.21.15.88:0/1228652408 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc22000fca0 con 0x7fc218004d40 2024-09-16T01:45:01.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.054+0000 7fc23495f640 1 -- 172.21.15.88:0/1228652408 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2301561a0 con 0x7fc218004d40 2024-09-16T01:45:01.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.054+0000 7fc23495f640 1 -- 172.21.15.88:0/1228652408 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc230156760 con 0x7fc218004d40 2024-09-16T01:45:01.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.055+0000 7fc23495f640 1 -- 172.21.15.88:0/1228652408 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2300cf9a0 con 0x7fc218004d40 2024-09-16T01:45:01.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.056+0000 7fc21effd640 1 -- 172.21.15.88:0/1228652408 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc22001d460 con 0x7fc218004d40 2024-09-16T01:45:01.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.056+0000 7fc21effd640 1 --2- 172.21.15.88:0/1228652408 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc21006c2e0 0x7fc21006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.056+0000 7fc21effd640 1 -- 172.21.15.88:0/1228652408 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5709+0+0 (secure 0 0 0) 0x7fc22008b450 con 0x7fc218004d40 2024-09-16T01:45:01.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.056+0000 7fc22dd9b640 1 --2- 172.21.15.88:0/1228652408 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc21006c2e0 0x7fc21006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.057+0000 7fc22dd9b640 1 --2- 172.21.15.88:0/1228652408 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc21006c2e0 0x7fc21006e7a0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fc224006e80 tx=0x7fc224006df0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:01.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.060+0000 7fc21effd640 1 -- 172.21.15.88:0/1228652408 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc220059bc0 con 0x7fc218004d40 2024-09-16T01:45:01.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.260+0000 7fc23495f640 1 -- 172.21.15.88:0/1228652408 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc2300cf6c0 con 0x7fc218004d40 2024-09-16T01:45:01.262 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:45:01.261+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:45:01.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.262+0000 7fc21effd640 1 -- 172.21.15.88:0/1228652408 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v327) ==== 161+0+0 (secure 0 0 0) 0x7fc2200594c0 con 0x7fc218004d40 2024-09-16T01:45:01.264 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:45:01.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.265+0000 7fc23495f640 1 -- 172.21.15.88:0/1228652408 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc21006c2e0 msgr2=0x7fc21006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:01.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.265+0000 7fc23495f640 1 --2- 172.21.15.88:0/1228652408 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc21006c2e0 0x7fc21006e7a0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fc224006e80 tx=0x7fc224006df0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.265+0000 7fc23495f640 1 -- 172.21.15.88:0/1228652408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc218004d40 msgr2=0x7fc230156f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:01.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.265+0000 7fc23495f640 1 --2- 172.21.15.88:0/1228652408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc218004d40 0x7fc230156f60 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7fc220007310 tx=0x7fc2200077d0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.265+0000 7fc23495f640 1 -- 172.21.15.88:0/1228652408 shutdown_connections 2024-09-16T01:45:01.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.265+0000 7fc23495f640 1 --2- 172.21.15.88:0/1228652408 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc21006c2e0 0x7fc21006e7a0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.265+0000 7fc23495f640 1 --2- 172.21.15.88:0/1228652408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc230153c20 0x7fc2300c9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.265+0000 7fc23495f640 1 --2- 172.21.15.88:0/1228652408 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc2301537c0 0x7fc2300c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.265+0000 7fc23495f640 1 --2- 172.21.15.88:0/1228652408 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc218004d40 0x7fc230156f60 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.265+0000 7fc23495f640 1 -- 172.21.15.88:0/1228652408 >> 172.21.15.88:0/1228652408 conn(0x7fc23006d0f0 msgr2=0x7fc2300cb840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:01.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.265+0000 7fc23495f640 1 -- 172.21.15.88:0/1228652408 shutdown_connections 2024-09-16T01:45:01.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.266+0000 7fc23495f640 1 -- 172.21.15.88:0/1228652408 wait complete. 2024-09-16T01:45:01.270 INFO:tasks.ceph.mds.a.smithi088.stderr: -6390> 2024-09-16T01:44:53.496+0000 7f832c8b0640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:45:01.279 DEBUG:teuthology.orchestra.run.smithi088:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 journal reset --yes-i-really-really-mean-it 2024-09-16T01:45:01.302 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:45:01.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.320+0000 7fe286a94280 1 -- start start 2024-09-16T01:45:01.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.320+0000 7fe286a94280 1 Processor -- start 2024-09-16T01:45:01.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.321+0000 7fe286a94280 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55735f4c4c00 0x55735f540580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.321+0000 7fe286a94280 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55735f4c5800 0x55735f540b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.321+0000 7fe286a94280 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55735f4c5c00 0x55735f541080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.321+0000 7fe286a94280 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5573602b6180 con 0x55735f4c5c00 2024-09-16T01:45:01.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.321+0000 7fe286a94280 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5573602b6300 con 0x55735f4c4c00 2024-09-16T01:45:01.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.321+0000 7fe286a94280 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5573602b6480 con 0x55735f4c5800 2024-09-16T01:45:01.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.321+0000 7fe285a3c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55735f4c5c00 0x55735f541080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.321+0000 7fe285a3c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55735f4c5c00 0x55735f541080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54722/0 (socket says 172.21.15.88:54722) 2024-09-16T01:45:01.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.321+0000 7fe285a3c640 1 -- 172.21.15.88:0/4084761365 learned_addr learned my addr 172.21.15.88:0/4084761365 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:01.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.321+0000 7fe28523b640 1 --2- 172.21.15.88:0/4084761365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55735f4c4c00 0x55735f540580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.321+0000 7fe284a3a640 1 --2- 172.21.15.88:0/4084761365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55735f4c5800 0x55735f540b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.321+0000 7fe285a3c640 1 -- 172.21.15.88:0/4084761365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55735f4c5800 msgr2=0x55735f540b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:01.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.321+0000 7fe285a3c640 1 --2- 172.21.15.88:0/4084761365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55735f4c5800 0x55735f540b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.321+0000 7fe285a3c640 1 -- 172.21.15.88:0/4084761365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55735f4c4c00 msgr2=0x55735f540580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:01.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.321+0000 7fe285a3c640 1 --2- 172.21.15.88:0/4084761365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55735f4c4c00 0x55735f540580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.321+0000 7fe285a3c640 1 -- 172.21.15.88:0/4084761365 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55735f4fc780 con 0x55735f4c5c00 2024-09-16T01:45:01.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.321+0000 7fe28523b640 1 --2- 172.21.15.88:0/4084761365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55735f4c4c00 0x55735f540580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:45:01.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.321+0000 7fe284a3a640 1 --2- 172.21.15.88:0/4084761365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55735f4c5800 0x55735f540b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:45:01.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.321+0000 7fe285a3c640 1 --2- 172.21.15.88:0/4084761365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55735f4c5c00 0x55735f541080 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x5573602ed830 tx=0x557360269320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be0a715b47096713 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:01.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.321+0000 7fe282a36640 1 -- 172.21.15.88:0/4084761365 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55735f4ea680 con 0x55735f4c5c00 2024-09-16T01:45:01.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.322+0000 7fe282a36640 1 -- 172.21.15.88:0/4084761365 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55735f4f0700 con 0x55735f4c5c00 2024-09-16T01:45:01.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.322+0000 7fe282a36640 1 -- 172.21.15.88:0/4084761365 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55735f4ea9c0 con 0x55735f4c5c00 2024-09-16T01:45:01.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.322+0000 7fe286a94280 1 -- 172.21.15.88:0/4084761365 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55735f4fd680 con 0x55735f4c5c00 2024-09-16T01:45:01.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.322+0000 7fe286a94280 1 -- 172.21.15.88:0/4084761365 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55735f4fc000 con 0x55735f4c5c00 2024-09-16T01:45:01.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.322+0000 7fe286a94280 4 waiting for MDS map... 2024-09-16T01:45:01.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.322+0000 7fe282a36640 1 -- 172.21.15.88:0/4084761365 <== mon.0 v2:172.21.15.88:3300/0 4 ==== osd_map(139..139 src has 1..139) ==== 5795+0+0 (secure 0 0 0) 0x5573603aa400 con 0x55735f4c5c00 2024-09-16T01:45:01.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.322+0000 7fe282a36640 1 -- 172.21.15.88:0/4084761365 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 327) ==== 2893+0+0 (secure 0 0 0) 0x55735f51f180 con 0x55735f4c5c00 2024-09-16T01:45:01.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.322+0000 7fe286a94280 4 Got MDS map 327 2024-09-16T01:45:01.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.322+0000 7fe286a94280 10 main: JournalTool::main 2024-09-16T01:45:01.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.322+0000 7fe286a94280 4 main: JournalTool: connecting to RADOS... 2024-09-16T01:45:01.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.322+0000 7fe286a94280 1 Processor -- start 2024-09-16T01:45:01.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.323+0000 7fe286a94280 1 -- start start 2024-09-16T01:45:01.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.323+0000 7fe286a94280 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573603f2000 0x55735f541600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.323+0000 7fe286a94280 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5573603f2400 0x55735f541b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.323+0000 7fe286a94280 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573603f2800 0x55735f542100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.323+0000 7fe286a94280 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5573602b6600 con 0x5573603f2000 2024-09-16T01:45:01.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.323+0000 7fe286a94280 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5573602b6780 con 0x5573603f2800 2024-09-16T01:45:01.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.323+0000 7fe286a94280 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5573602b6900 con 0x5573603f2400 2024-09-16T01:45:01.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.323+0000 7fe284a3a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573603f2000 0x55735f541600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.323+0000 7fe284a3a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573603f2000 0x55735f541600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54736/0 (socket says 172.21.15.88:54736) 2024-09-16T01:45:01.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.323+0000 7fe284a3a640 1 -- 172.21.15.88:0/2627198985 learned_addr learned my addr 172.21.15.88:0/2627198985 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:01.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.323+0000 7fe284a3a640 1 -- 172.21.15.88:0/2627198985 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5573603f2400 msgr2=0x55735f541b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:01.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.323+0000 7fe284a3a640 1 --2- 172.21.15.88:0/2627198985 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5573603f2400 0x55735f541b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.323+0000 7fe284a3a640 1 -- 172.21.15.88:0/2627198985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573603f2800 msgr2=0x55735f542100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:01.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.323+0000 7fe284a3a640 1 --2- 172.21.15.88:0/2627198985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573603f2800 0x55735f542100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.323+0000 7fe284a3a640 1 -- 172.21.15.88:0/2627198985 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55735f4fc3c0 con 0x5573603f2000 2024-09-16T01:45:01.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.324+0000 7fe284a3a640 1 --2- 172.21.15.88:0/2627198985 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573603f2000 0x55735f541600 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x5573602d08a0 tx=0x5573603aec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d626ac49315ad5e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:01.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.324+0000 7fe280a32640 1 -- 172.21.15.88:0/2627198985 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55735f4ea680 con 0x5573603f2000 2024-09-16T01:45:01.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.324+0000 7fe280a32640 1 -- 172.21.15.88:0/2627198985 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55735f4f0e00 con 0x5573603f2000 2024-09-16T01:45:01.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.324+0000 7fe280a32640 1 -- 172.21.15.88:0/2627198985 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55735f4ea9c0 con 0x5573603f2000 2024-09-16T01:45:01.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.324+0000 7fe286a94280 1 -- 172.21.15.88:0/2627198985 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573603f2000 msgr2=0x55735f541600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:01.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.324+0000 7fe286a94280 1 --2- 172.21.15.88:0/2627198985 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573603f2000 0x55735f541600 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x5573602d08a0 tx=0x5573603aec00 comp rx=0 tx=0).stop 2024-09-16T01:45:01.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.324+0000 7fe286a94280 1 -- 172.21.15.88:0/2627198985 shutdown_connections 2024-09-16T01:45:01.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.324+0000 7fe286a94280 1 --2- 172.21.15.88:0/2627198985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573603f2800 0x55735f542100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.324+0000 7fe286a94280 1 --2- 172.21.15.88:0/2627198985 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5573603f2400 0x55735f541b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.324+0000 7fe286a94280 1 --2- 172.21.15.88:0/2627198985 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573603f2000 0x55735f541600 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.324+0000 7fe286a94280 1 -- 172.21.15.88:0/2627198985 >> 172.21.15.88:0/2627198985 conn(0x5573602b9c00 msgr2=0x557360253000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:01.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.324+0000 7fe286a94280 1 -- 172.21.15.88:0/2627198985 shutdown_connections 2024-09-16T01:45:01.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.324+0000 7fe286a94280 1 -- 172.21.15.88:0/2627198985 wait complete. 2024-09-16T01:45:01.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.325+0000 7fe286a94280 1 Processor -- start 2024-09-16T01:45:01.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.325+0000 7fe286a94280 1 -- start start 2024-09-16T01:45:01.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.325+0000 7fe286a94280 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573603f2000 0x55735f543180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.325+0000 7fe286a94280 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573603f2400 0x55735f542c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.325+0000 7fe286a94280 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5573603f2800 0x55735f542680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.325+0000 7fe286a94280 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5573602b7c80 con 0x5573603f2000 2024-09-16T01:45:01.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.325+0000 7fe286a94280 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5573602b7b00 con 0x5573603f2400 2024-09-16T01:45:01.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.325+0000 7fe286a94280 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5573602b7980 con 0x5573603f2800 2024-09-16T01:45:01.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.325+0000 7fe284a3a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573603f2000 0x55735f543180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.325+0000 7fe284a3a640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573603f2000 0x55735f543180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54742/0 (socket says 172.21.15.88:54742) 2024-09-16T01:45:01.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.325+0000 7fe284a3a640 1 -- 172.21.15.88:0/2571202435 learned_addr learned my addr 172.21.15.88:0/2571202435 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:01.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.326+0000 7fe28523b640 1 --2- 172.21.15.88:0/2571202435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573603f2400 0x55735f542c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.326+0000 7fe284a3a640 1 --2- 172.21.15.88:0/2571202435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5573603f2800 0x55735f542680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.326+0000 7fe284a3a640 1 -- 172.21.15.88:0/2571202435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5573603f2800 msgr2=0x55735f542680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:01.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.326+0000 7fe284a3a640 1 --2- 172.21.15.88:0/2571202435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5573603f2800 0x55735f542680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.326+0000 7fe284a3a640 1 -- 172.21.15.88:0/2571202435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573603f2400 msgr2=0x55735f542c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:01.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.326+0000 7fe284a3a640 1 --2- 172.21.15.88:0/2571202435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573603f2400 0x55735f542c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.326+0000 7fe284a3a640 1 -- 172.21.15.88:0/2571202435 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55735f4fc960 con 0x5573603f2000 2024-09-16T01:45:01.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.326+0000 7fe284a3a640 1 --2- 172.21.15.88:0/2571202435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573603f2000 0x55735f543180 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x5573602d1b60 tx=0x5573603aec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:01.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.326+0000 7fe27f22f640 1 -- 172.21.15.88:0/2571202435 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55735f4ea680 con 0x5573603f2000 2024-09-16T01:45:01.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.326+0000 7fe27f22f640 1 -- 172.21.15.88:0/2571202435 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55735f4f0c40 con 0x5573603f2000 2024-09-16T01:45:01.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.326+0000 7fe286a94280 1 -- 172.21.15.88:0/2571202435 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55735f4fd860 con 0x5573603f2000 2024-09-16T01:45:01.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.326+0000 7fe27f22f640 1 -- 172.21.15.88:0/2571202435 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55735f4ea9c0 con 0x5573603f2000 2024-09-16T01:45:01.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.326+0000 7fe286a94280 1 -- 172.21.15.88:0/2571202435 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55735f4fda40 con 0x5573603f2000 2024-09-16T01:45:01.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.326+0000 7fe286a94280 4 main: JournalTool: resolving pool 14 2024-09-16T01:45:01.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.327+0000 7fe27f22f640 1 -- 172.21.15.88:0/2571202435 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x55736040ac00 con 0x5573603f2000 2024-09-16T01:45:01.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.327+0000 7fe27f22f640 1 --2- 172.21.15.88:0/2571202435 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x55735f850000 0x55735f543700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.327+0000 7fe27f22f640 1 -- 172.21.15.88:0/2571202435 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5795+0+0 (secure 0 0 0) 0x55736041c600 con 0x5573603f2000 2024-09-16T01:45:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.328+0000 7fe286a94280 4 main: JournalTool: creating IoCtx.. 2024-09-16T01:45:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.328+0000 7fe286a94280 4 main: Executing for rank 0 2024-09-16T01:45:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.328+0000 7fe28523b640 1 --2- 172.21.15.88:0/2571202435 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x55735f850000 0x55735f543700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.328+0000 7fe28523b640 1 --2- 172.21.15.88:0/2571202435 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x55735f850000 0x55735f543700 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x55735f640de0 tx=0x5573603aeea0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.328+0000 7fe286a94280 1 -- start start 2024-09-16T01:45:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.328+0000 7fe286a94280 1 Processor -- start 2024-09-16T01:45:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.329+0000 7fe286a94280 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5573603f3000 0x55735f542100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.329+0000 7fe286a94280 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55735f851000 0x55735f541b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.329+0000 7fe286a94280 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55735f851400 0x55735f541600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.329+0000 7fe286a94280 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x5573602b7680 con 0x55735f851000 2024-09-16T01:45:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.329+0000 7fe286a94280 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5573602b6600 con 0x55735f851400 2024-09-16T01:45:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.329+0000 7fe286a94280 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5573602b6780 con 0x5573603f3000 2024-09-16T01:45:01.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.329+0000 7fe28523b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55735f851000 0x55735f541b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.329+0000 7fe28523b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55735f851000 0x55735f541b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54756/0 (socket says 172.21.15.88:54756) 2024-09-16T01:45:01.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.329+0000 7fe28523b640 1 -- 172.21.15.88:0/1035214929 learned_addr learned my addr 172.21.15.88:0/1035214929 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:01.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.329+0000 7fe284a3a640 1 --2- 172.21.15.88:0/1035214929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55735f851400 0x55735f541600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.329+0000 7fe285a3c640 1 --2- 172.21.15.88:0/1035214929 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5573603f3000 0x55735f542100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.329+0000 7fe28523b640 1 -- 172.21.15.88:0/1035214929 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5573603f3000 msgr2=0x55735f542100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:01.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.329+0000 7fe28523b640 1 --2- 172.21.15.88:0/1035214929 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5573603f3000 0x55735f542100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.329+0000 7fe28523b640 1 -- 172.21.15.88:0/1035214929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55735f851400 msgr2=0x55735f541600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:01.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.329+0000 7fe28523b640 1 --2- 172.21.15.88:0/1035214929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55735f851400 0x55735f541600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.329+0000 7fe28523b640 1 -- 172.21.15.88:0/1035214929 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55735f4fdc20 con 0x55735f851000 2024-09-16T01:45:01.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.329+0000 7fe28523b640 1 --2- 172.21.15.88:0/1035214929 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55735f851000 0x55735f541b80 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x55735f7f3b60 tx=0x5573602695c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca2c68bf2a75715 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:01.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.329+0000 7fe27c229640 1 -- 172.21.15.88:0/1035214929 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557360443d40 con 0x55735f851000 2024-09-16T01:45:01.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.329+0000 7fe27c229640 1 -- 172.21.15.88:0/1035214929 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55735f4f01c0 con 0x55735f851000 2024-09-16T01:45:01.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.329+0000 7fe27c229640 1 -- 172.21.15.88:0/1035214929 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557360451860 con 0x55735f851000 2024-09-16T01:45:01.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.330+0000 7fe286a94280 1 -- 172.21.15.88:0/1035214929 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55735f4fd4a0 con 0x55735f851000 2024-09-16T01:45:01.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.330+0000 7fe286a94280 1 -- 172.21.15.88:0/1035214929 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55735f4fc3c0 con 0x55735f851000 2024-09-16T01:45:01.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.330+0000 7fe286a94280 4 waiting for MDS map... 2024-09-16T01:45:01.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.330+0000 7fe27c229640 1 -- 172.21.15.88:0/1035214929 <== mon.0 v2:172.21.15.88:3300/0 4 ==== osd_map(139..139 src has 1..139) ==== 5795+0+0 (secure 0 0 0) 0x55735f8b3000 con 0x55735f851000 2024-09-16T01:45:01.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.330+0000 7fe27c229640 1 -- 172.21.15.88:0/1035214929 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 327) ==== 2893+0+0 (secure 0 0 0) 0x55735f51f500 con 0x55735f851000 2024-09-16T01:45:01.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.330+0000 7fe286a94280 4 Got MDS map 327 2024-09-16T01:45:01.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.330+0000 7fe286a94280 1 --2- 172.21.15.88:0/1035214929 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55735f851800 0x55735f8c8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.330+0000 7fe286a94280 1 -- 172.21.15.88:0/1035214929 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:1 14.f 14:f1f69726:::400.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139) -- 0x55735f851c00 con 0x55735f851800 2024-09-16T01:45:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.330+0000 7fe285a3c640 1 --2- 172.21.15.88:0/1035214929 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55735f851800 0x55735f8c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.330+0000 7fe285a3c640 1 --2- 172.21.15.88:0/1035214929 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55735f851800 0x55735f8c8000 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=e08d5bb23ecc4fec server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.333+0000 7fe285a3c640 1 -- 172.21.15.88:0/1035214929 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) ==== 156+0+22 (crc 0 0 0) 0x557360256000 con 0x55735f851800 2024-09-16T01:45:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.333+0000 7fe286a94280 1 --2- 172.21.15.88:0/1035214929 >> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] conn(0x55735f8b5400 0x55735f8c8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.333+0000 7fe286a94280 1 -- 172.21.15.88:0/1035214929 --> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] -- osd_op(unknown.0.0:2 14.14 14:292cf221:::200.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139) -- 0x55735f8b5800 con 0x55735f8b5400 2024-09-16T01:45:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.333+0000 7fe284a3a640 1 --2- 172.21.15.88:0/1035214929 >> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] conn(0x55735f8b5400 0x55735f8c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.333+0000 7fe284a3a640 1 --2- 172.21.15.88:0/1035214929 >> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] conn(0x55735f8b5400 0x55735f8c8580 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=e4bdda47c5976bef server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.334+0000 7fe284a3a640 1 -- 172.21.15.88:0/1035214929 <== osd.4 v2:172.21.15.154:6816/2520717231 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv22 ondisk = 0) ==== 156+0+90 (crc 0 0 0) 0x557360256500 con 0x55735f8b5400 2024-09-16T01:45:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.334+0000 7fe27aa26640 1 -- 172.21.15.88:0/1035214929 --> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] -- osd_op(unknown.0.0:3 14.14 14:2e2fa760:::200.00000001:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e139) -- 0x55735f8d4000 con 0x55735f8b5400 2024-09-16T01:45:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.334+0000 7fe27aa26640 1 --2- 172.21.15.88:0/1035214929 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55735f8d4400 0x55735f8ca680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.334+0000 7fe27aa26640 1 -- 172.21.15.88:0/1035214929 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:4 14.1b 14:ddb4e4d7:::200.00000002:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e139) -- 0x55735f8d4800 con 0x55735f8d4400 2024-09-16T01:45:01.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.334+0000 7fe285a3c640 1 --2- 172.21.15.88:0/1035214929 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55735f8d4400 0x55735f8ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.334+0000 7fe284a3a640 1 -- 172.21.15.88:0/1035214929 <== osd.4 v2:172.21.15.154:6816/2520717231 2 ==== osd_op_reply(3 200.00000001 [stat out=16b] v0'0 uv19 ondisk = 0) ==== 156+0+16 (crc 0 0 0) 0x557360256500 con 0x55735f8b5400 2024-09-16T01:45:01.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.334+0000 7fe285a3c640 1 --2- 172.21.15.88:0/1035214929 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55735f8d4400 0x55735f8ca680 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=db4ac6a327468731 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:01.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.334+0000 7fe285a3c640 1 -- 172.21.15.88:0/1035214929 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(4 200.00000002 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x557360256000 con 0x55735f8d4400 2024-09-16T01:45:01.345 INFO:teuthology.orchestra.run.smithi088.stdout:old journal was 4226044~2030 2024-09-16T01:45:01.345 INFO:teuthology.orchestra.run.smithi088.stdout:new journal start will be 8388608 (4160534 bytes past old end) 2024-09-16T01:45:01.345 INFO:teuthology.orchestra.run.smithi088.stdout:writing journal head 2024-09-16T01:45:01.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.334+0000 7fe286a94280 1 -- 172.21.15.88:0/1035214929 --> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] -- osd_op(unknown.0.0:5 14.14 14:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139) -- 0x55735f8b5c00 con 0x55735f8b5400 2024-09-16T01:45:01.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.336+0000 7fe284a3a640 1 -- 172.21.15.88:0/1035214929 <== osd.4 v2:172.21.15.154:6816/2520717231 3 ==== osd_op_reply(5 200.00000000 [writefull 0~90] v139'23 uv23 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x557360256500 con 0x55735f8b5400 2024-09-16T01:45:01.346 INFO:teuthology.orchestra.run.smithi088.stdout:writing EResetJournal entry 2024-09-16T01:45:01.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.336+0000 7fe286a94280 1 -- 172.21.15.88:0/1035214929 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:6 14.1b 14:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139) -- 0x55735f8d4c00 con 0x55735f8d4400 2024-09-16T01:45:01.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.336+0000 7fe286a94280 1 --2- 172.21.15.88:0/1035214929 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55735f8d5000 0x55735f8c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.336+0000 7fe286a94280 1 -- 172.21.15.88:0/1035214929 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:7 14.b 14:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139) -- 0x55735f8d5400 con 0x55735f8d5000 2024-09-16T01:45:01.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.336+0000 7fe286a94280 1 -- 172.21.15.88:0/1035214929 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:8 14.1f 14:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139) -- 0x55735f8d5800 con 0x55735f8d5000 2024-09-16T01:45:01.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.336+0000 7fe286a94280 1 -- 172.21.15.88:0/1035214929 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:9 14.1a 14:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139) -- 0x55735f8d5c00 con 0x55735f8d5000 2024-09-16T01:45:01.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.336+0000 7fe286a94280 1 -- 172.21.15.88:0/1035214929 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- osd_op(unknown.0.0:10 14.1e 14:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139) -- 0x55735f8de000 con 0x55735f851800 2024-09-16T01:45:01.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.336+0000 7fe286a94280 1 -- 172.21.15.88:0/1035214929 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- osd_op(unknown.0.0:11 14.1d 14:bec1251c:::200.00000007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139) -- 0x55735f8de400 con 0x55735f8d5000 2024-09-16T01:45:01.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.336+0000 7fe28523b640 1 --2- 172.21.15.88:0/1035214929 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55735f8d5000 0x55735f8c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.336+0000 7fe28523b640 1 --2- 172.21.15.88:0/1035214929 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55735f8d5000 0x55735f8c8b00 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=3b5c69c872d8eda5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:01.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.337+0000 7fe285a3c640 1 -- 172.21.15.88:0/1035214929 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== osd_op_reply(6 200.00000002 [delete] v139'2 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x557360256000 con 0x55735f8d4400 2024-09-16T01:45:01.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.337+0000 7fe285a3c640 1 -- 172.21.15.88:0/1035214929 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== osd_op_reply(10 200.00000006 [delete] v139'2 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x557360256000 con 0x55735f851800 2024-09-16T01:45:01.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.339+0000 7fe28523b640 1 -- 172.21.15.88:0/1035214929 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== osd_op_reply(9 200.00000005 [delete] v139'2 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x557360256780 con 0x55735f8d5000 2024-09-16T01:45:01.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.339+0000 7fe28523b640 1 -- 172.21.15.88:0/1035214929 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== osd_op_reply(7 200.00000003 [delete] v139'4 uv3 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x557360256780 con 0x55735f8d5000 2024-09-16T01:45:01.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.339+0000 7fe28523b640 1 -- 172.21.15.88:0/1035214929 <== osd.0 v2:172.21.15.88:6816/903680192 3 ==== osd_op_reply(11 200.00000007 [delete] v139'3 uv2 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x557360256780 con 0x55735f8d5000 2024-09-16T01:45:01.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.339+0000 7fe27aa26640 1 -- 172.21.15.88:0/1035214929 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:12 14.1b 14:ddb4e4d7:::200.00000002:head [write 0~48 in=48b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139) -- 0x55735f8e6000 con 0x55735f8d4400 2024-09-16T01:45:01.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.340+0000 7fe28523b640 1 -- 172.21.15.88:0/1035214929 <== osd.0 v2:172.21.15.88:6816/903680192 4 ==== osd_op_reply(8 200.00000004 [delete] v139'7 uv6 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x557360256780 con 0x55735f8d5000 2024-09-16T01:45:01.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.342+0000 7fe285a3c640 1 -- 172.21.15.88:0/1035214929 <== osd.1 v2:172.21.15.88:6824/3621315521 3 ==== osd_op_reply(12 200.00000002 [write 0~48] v139'3 uv3 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x557360256000 con 0x55735f8d4400 2024-09-16T01:45:01.349 INFO:teuthology.orchestra.run.smithi088.stdout:done 2024-09-16T01:45:01.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.342+0000 7fe286a94280 1 -- 172.21.15.88:0/1035214929 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55735f8d5000 msgr2=0x55735f8c8b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:01.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.342+0000 7fe286a94280 1 --2- 172.21.15.88:0/1035214929 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x55735f8d5000 0x55735f8c8b00 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.342+0000 7fe286a94280 1 -- 172.21.15.88:0/1035214929 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55735f8d4400 msgr2=0x55735f8ca680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:01.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.342+0000 7fe286a94280 1 --2- 172.21.15.88:0/1035214929 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55735f8d4400 0x55735f8ca680 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.342+0000 7fe286a94280 1 -- 172.21.15.88:0/1035214929 >> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] conn(0x55735f8b5400 msgr2=0x55735f8c8580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:01.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.342+0000 7fe286a94280 1 --2- 172.21.15.88:0/1035214929 >> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] conn(0x55735f8b5400 0x55735f8c8580 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.342+0000 7fe286a94280 1 -- 172.21.15.88:0/1035214929 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55735f851800 msgr2=0x55735f8c8000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:01.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.342+0000 7fe286a94280 1 --2- 172.21.15.88:0/1035214929 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x55735f851800 0x55735f8c8000 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.342+0000 7fe286a94280 1 -- 172.21.15.88:0/1035214929 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55735f851000 msgr2=0x55735f541b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:01.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.342+0000 7fe284a3a640 1 -- 172.21.15.88:0/1035214929 reap_dead start 2024-09-16T01:45:01.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.342+0000 7fe286a94280 1 --2- 172.21.15.88:0/1035214929 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55735f851000 0x55735f541b80 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x55735f7f3b60 tx=0x5573602695c0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.342+0000 7fe284a3a640 1 -- 172.21.15.88:0/1035214929 reap_dead start 2024-09-16T01:45:01.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.342+0000 7fe286a94280 1 -- 172.21.15.88:0/1035214929 shutdown_connections 2024-09-16T01:45:01.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.342+0000 7fe286a94280 1 --2- 172.21.15.88:0/1035214929 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55735f851000 0x55735f541b80 unknown :-1 s=CLOSED pgs=2384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.342+0000 7fe286a94280 1 -- 172.21.15.88:0/1035214929 >> 172.21.15.88:0/1035214929 conn(0x5573603f2c00 msgr2=0x557360253800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:01.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.343+0000 7fe286a94280 1 -- 172.21.15.88:0/1035214929 shutdown_connections 2024-09-16T01:45:01.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.343+0000 7fe286a94280 1 -- 172.21.15.88:0/1035214929 wait complete. 2024-09-16T01:45:01.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.343+0000 7fe286a94280 1 -- 172.21.15.88:0/2571202435 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x55735f850000 msgr2=0x55735f543700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:01.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.343+0000 7fe286a94280 1 --2- 172.21.15.88:0/2571202435 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x55735f850000 0x55735f543700 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x55735f640de0 tx=0x5573603aeea0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.343+0000 7fe286a94280 1 -- 172.21.15.88:0/2571202435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573603f2000 msgr2=0x55735f543180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:01.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.343+0000 7fe286a94280 1 --2- 172.21.15.88:0/2571202435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573603f2000 0x55735f543180 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x5573602d1b60 tx=0x5573603aec60 comp rx=0 tx=0).stop 2024-09-16T01:45:01.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.343+0000 7fe286a94280 1 -- 172.21.15.88:0/2571202435 shutdown_connections 2024-09-16T01:45:01.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.343+0000 7fe286a94280 1 --2- 172.21.15.88:0/2571202435 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x55735f850000 0x55735f543700 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.343+0000 7fe286a94280 1 --2- 172.21.15.88:0/2571202435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5573603f2800 0x55735f542680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.343+0000 7fe286a94280 1 --2- 172.21.15.88:0/2571202435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573603f2400 0x55735f542c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.343+0000 7fe286a94280 1 --2- 172.21.15.88:0/2571202435 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5573603f2000 0x55735f543180 unknown :-1 s=CLOSED pgs=2383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.343+0000 7fe286a94280 1 -- 172.21.15.88:0/2571202435 >> 172.21.15.88:0/2571202435 conn(0x5573602b9c00 msgr2=0x557360253000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:01.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.343+0000 7fe286a94280 1 -- 172.21.15.88:0/2571202435 shutdown_connections 2024-09-16T01:45:01.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.343+0000 7fe286a94280 1 -- 172.21.15.88:0/2571202435 wait complete. 2024-09-16T01:45:01.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.343+0000 7fe286a94280 1 -- 172.21.15.88:0/4084761365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55735f4c5c00 msgr2=0x55735f541080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:01.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.343+0000 7fe286a94280 1 --2- 172.21.15.88:0/4084761365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55735f4c5c00 0x55735f541080 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x5573602ed830 tx=0x557360269320 comp rx=0 tx=0).stop 2024-09-16T01:45:01.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.343+0000 7fe286a94280 1 -- 172.21.15.88:0/4084761365 shutdown_connections 2024-09-16T01:45:01.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.343+0000 7fe286a94280 1 --2- 172.21.15.88:0/4084761365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55735f4c5c00 0x55735f541080 unknown :-1 s=CLOSED pgs=2381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.343+0000 7fe286a94280 1 --2- 172.21.15.88:0/4084761365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55735f4c5800 0x55735f540b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.343+0000 7fe286a94280 1 --2- 172.21.15.88:0/4084761365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55735f4c4c00 0x55735f540580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.343+0000 7fe286a94280 1 -- 172.21.15.88:0/4084761365 >> 172.21.15.88:0/4084761365 conn(0x55735f4c5000 msgr2=0x557360252800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:01.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.343+0000 7fe286a94280 1 -- 172.21.15.88:0/4084761365 shutdown_connections 2024-09-16T01:45:01.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.343+0000 7fe286a94280 1 -- 172.21.15.88:0/4084761365 wait complete. 2024-09-16T01:45:01.356 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'journal', 'reset', '--yes-i-really-really-mean-it'] in time 0:00:00.077946, result: old journal was 4226044~2030 new journal start will be 8388608 (4160534 bytes past old end) writing journal head writing EResetJournal entry done 2024-09-16T01:45:01.357 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-16T01:45:01.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.492+0000 7ff108913640 1 Processor -- start 2024-09-16T01:45:01.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.492+0000 7ff108913640 1 -- start start 2024-09-16T01:45:01.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.493+0000 7ff108913640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff104155880 0x7ff10415f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.493+0000 7ff108913640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff1040b4d60 0x7ff1040b51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.493+0000 7ff108913640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff10415bb60 con 0x7ff10406f630 2024-09-16T01:45:01.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.493+0000 7ff108913640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7ff104155c60 con 0x7ff1040b4d60 2024-09-16T01:45:01.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.493+0000 7ff108913640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff1040b5700 con 0x7ff104155880 2024-09-16T01:45:01.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.493+0000 7ff101d9b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7ff1040b4d60 0x7ff1040b51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.493+0000 7ff102d9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff10406f630 0x7ff10406fa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35442/0 (socket says 172.21.15.88:35442) 2024-09-16T01:45:01.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.493+0000 7ff102d9d640 1 -- 172.21.15.88:0/3466361619 learned_addr learned my addr 172.21.15.88:0/3466361619 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:01.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.493+0000 7ff101d9b640 1 -- 172.21.15.88:0/3466361619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff104155880 msgr2=0x7ff10415f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:01.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.493+0000 7ff10259c640 1 --2- 172.21.15.88:0/3466361619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff104155880 0x7ff10415f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.493+0000 7ff10159a640 1 -- 172.21.15.88:0/3466361619 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2522082649 0 0) 0x7ff10415bb60 con 0x7ff10406f630 2024-09-16T01:45:01.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.493+0000 7ff101d9b640 1 --2- 172.21.15.88:0/3466361619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff104155880 0x7ff10415f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.493+0000 7ff101d9b640 1 -- 172.21.15.88:0/3466361619 >> v1:172.21.15.88:6789/0 conn(0x7ff10406f630 legacy=0x7ff10406fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:01.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.493+0000 7ff101d9b640 1 -- 172.21.15.88:0/3466361619 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff10415be40 con 0x7ff1040b4d60 2024-09-16T01:45:01.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.493+0000 7ff10259c640 1 --2- 172.21.15.88:0/3466361619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff104155880 0x7ff10415f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:45:01.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.494+0000 7ff101d9b640 1 --2- 172.21.15.88:0/3466361619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1040b4d60 0x7ff1040b51c0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7ff0f80097a0 tx=0x7ff0f802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b4bc3ac0920c0e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:01.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.494+0000 7ff10159a640 1 -- 172.21.15.88:0/3466361619 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f8037070 con 0x7ff1040b4d60 2024-09-16T01:45:01.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.494+0000 7ff10159a640 1 -- 172.21.15.88:0/3466361619 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f8032440 con 0x7ff1040b4d60 2024-09-16T01:45:01.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.494+0000 7ff10159a640 1 -- 172.21.15.88:0/3466361619 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f8032810 con 0x7ff1040b4d60 2024-09-16T01:45:01.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.495+0000 7ff108913640 1 -- 172.21.15.88:0/3466361619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1040b4d60 msgr2=0x7ff1040b51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:01.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.495+0000 7ff108913640 1 --2- 172.21.15.88:0/3466361619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1040b4d60 0x7ff1040b51c0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7ff0f80097a0 tx=0x7ff0f802a180 comp rx=0 tx=0).stop 2024-09-16T01:45:01.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.495+0000 7ff108913640 1 -- 172.21.15.88:0/3466361619 shutdown_connections 2024-09-16T01:45:01.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.495+0000 7ff108913640 1 --2- 172.21.15.88:0/3466361619 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1040b4d60 0x7ff1040b51c0 unknown :-1 s=CLOSED pgs=2386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.495+0000 7ff108913640 1 --2- 172.21.15.88:0/3466361619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff104155880 0x7ff10415f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.495+0000 7ff108913640 1 -- 172.21.15.88:0/3466361619 >> 172.21.15.88:0/3466361619 conn(0x7ff1040d1390 msgr2=0x7ff1040d17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:01.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.495+0000 7ff108913640 1 -- 172.21.15.88:0/3466361619 shutdown_connections 2024-09-16T01:45:01.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.495+0000 7ff108913640 1 -- 172.21.15.88:0/3466361619 wait complete. 2024-09-16T01:45:01.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.496+0000 7ff108913640 1 Processor -- start 2024-09-16T01:45:01.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.496+0000 7ff108913640 1 -- start start 2024-09-16T01:45:01.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.497+0000 7ff108913640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff10406f630 0x7ff1040d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.497+0000 7ff108913640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1040b4d60 0x7ff1041e5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.497+0000 7ff108913640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff104155880 0x7ff10406be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.497+0000 7ff108913640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7ff10415c400 con 0x7ff104155880 2024-09-16T01:45:01.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.497+0000 7ff108913640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff10415fcb0 con 0x7ff1040b4d60 2024-09-16T01:45:01.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.497+0000 7ff108913640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff10415c6e0 con 0x7ff10406f630 2024-09-16T01:45:01.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.497+0000 7ff102d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff104155880 0x7ff10406be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.497+0000 7ff102d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff104155880 0x7ff10406be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54778/0 (socket says 172.21.15.88:54778) 2024-09-16T01:45:01.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.497+0000 7ff102d9d640 1 -- 172.21.15.88:0/59920291 learned_addr learned my addr 172.21.15.88:0/59920291 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:01.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.497+0000 7ff101d9b640 1 --2- 172.21.15.88:0/59920291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1040b4d60 0x7ff1041e5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.497+0000 7ff10259c640 1 --2- 172.21.15.88:0/59920291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff10406f630 0x7ff1040d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.497+0000 7ff102d9d640 1 -- 172.21.15.88:0/59920291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff10406f630 msgr2=0x7ff1040d0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:01.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.497+0000 7ff102d9d640 1 --2- 172.21.15.88:0/59920291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff10406f630 0x7ff1040d0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.497+0000 7ff102d9d640 1 -- 172.21.15.88:0/59920291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1040b4d60 msgr2=0x7ff1041e5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:01.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.497+0000 7ff102d9d640 1 --2- 172.21.15.88:0/59920291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1040b4d60 0x7ff1041e5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:01.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.497+0000 7ff102d9d640 1 -- 172.21.15.88:0/59920291 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1040cfc80 con 0x7ff104155880 2024-09-16T01:45:01.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.497+0000 7ff101d9b640 1 --2- 172.21.15.88:0/59920291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1040b4d60 0x7ff1041e5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:45:01.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.498+0000 7ff102d9d640 1 --2- 172.21.15.88:0/59920291 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff104155880 0x7ff10406be80 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00c7a0 tx=0x7ff0fc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:01.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.498+0000 7ff0f2ffd640 1 -- 172.21.15.88:0/59920291 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc018070 con 0x7ff104155880 2024-09-16T01:45:01.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.498+0000 7ff0f2ffd640 1 -- 172.21.15.88:0/59920291 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0fc00a5e0 con 0x7ff104155880 2024-09-16T01:45:01.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.498+0000 7ff108913640 1 -- 172.21.15.88:0/59920291 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1040cff60 con 0x7ff104155880 2024-09-16T01:45:01.501 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.498+0000 7ff0f2ffd640 1 -- 172.21.15.88:0/59920291 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc00aa10 con 0x7ff104155880 2024-09-16T01:45:01.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.498+0000 7ff108913640 1 -- 172.21.15.88:0/59920291 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1040d0520 con 0x7ff104155880 2024-09-16T01:45:01.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.499+0000 7ff108913640 1 -- 172.21.15.88:0/59920291 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff10415b770 con 0x7ff104155880 2024-09-16T01:45:01.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.499+0000 7ff0f2ffd640 1 -- 172.21.15.88:0/59920291 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7ff0fc01b070 con 0x7ff104155880 2024-09-16T01:45:01.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.500+0000 7ff0f2ffd640 1 --2- 172.21.15.88:0/59920291 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff0e406c2b0 0x7ff0e406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:01.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.500+0000 7ff10259c640 1 --2- 172.21.15.88:0/59920291 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff0e406c2b0 0x7ff0e406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:01.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.500+0000 7ff0f2ffd640 1 -- 172.21.15.88:0/59920291 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5795+0+0 (secure 0 0 0) 0x7ff0fc08aa10 con 0x7ff104155880 2024-09-16T01:45:01.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.500+0000 7ff10259c640 1 --2- 172.21.15.88:0/59920291 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff0e406c2b0 0x7ff0e406e770 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400a950 tx=0x7ff0f4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:01.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.504+0000 7ff0f2ffd640 1 -- 172.21.15.88:0/59920291 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7ff0fc0591b0 con 0x7ff104155880 2024-09-16T01:45:01.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:01.707+0000 7ff108913640 1 -- 172.21.15.88:0/59920291 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7ff10415eb00 con 0x7ff104155880 2024-09-16T01:45:02.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.264+0000 7ff0f2ffd640 1 -- 172.21.15.88:0/59920291 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v328) ==== 162+0+0 (secure 0 0 0) 0x7ff0fc058ab0 con 0x7ff104155880 2024-09-16T01:45:02.266 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-16T01:45:02.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.267+0000 7ff108913640 1 -- 172.21.15.88:0/59920291 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff0e406c2b0 msgr2=0x7ff0e406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:02.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.267+0000 7ff108913640 1 --2- 172.21.15.88:0/59920291 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff0e406c2b0 0x7ff0e406e770 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400a950 tx=0x7ff0f4008040 comp rx=0 tx=0).stop 2024-09-16T01:45:02.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.267+0000 7ff108913640 1 -- 172.21.15.88:0/59920291 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff104155880 msgr2=0x7ff10406be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:02.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.267+0000 7ff108913640 1 --2- 172.21.15.88:0/59920291 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff104155880 0x7ff10406be80 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00c7a0 tx=0x7ff0fc00cc60 comp rx=0 tx=0).stop 2024-09-16T01:45:02.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.267+0000 7ff108913640 1 -- 172.21.15.88:0/59920291 shutdown_connections 2024-09-16T01:45:02.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.267+0000 7ff108913640 1 --2- 172.21.15.88:0/59920291 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7ff0e406c2b0 0x7ff0e406e770 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:02.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.267+0000 7ff108913640 1 --2- 172.21.15.88:0/59920291 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff104155880 0x7ff10406be80 unknown :-1 s=CLOSED pgs=2387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:02.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.267+0000 7ff108913640 1 --2- 172.21.15.88:0/59920291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1040b4d60 0x7ff1041e5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:02.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.267+0000 7ff108913640 1 --2- 172.21.15.88:0/59920291 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff10406f630 0x7ff1040d0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:02.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.267+0000 7ff108913640 1 -- 172.21.15.88:0/59920291 >> 172.21.15.88:0/59920291 conn(0x7ff1040d1390 msgr2=0x7ff10406fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:02.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.267+0000 7ff108913640 1 -- 172.21.15.88:0/59920291 shutdown_connections 2024-09-16T01:45:02.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.267+0000 7ff108913640 1 -- 172.21.15.88:0/59920291 wait complete. 2024-09-16T01:45:02.283 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:45:02.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.383+0000 7f70966dd640 1 Processor -- start 2024-09-16T01:45:02.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.383+0000 7f70966dd640 1 -- start start 2024-09-16T01:45:02.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.384+0000 7f70966dd640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f709015e560 0x7f709006e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:02.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.384+0000 7f70966dd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7090072220 0x7f70900ae890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:02.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.384+0000 7f70966dd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7090073130 0x7f7090073580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:02.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.384+0000 7f70966dd640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f7090073ac0 con 0x7f709015e560 2024-09-16T01:45:02.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.384+0000 7f70966dd640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f70900b5220 con 0x7f7090072220 2024-09-16T01:45:02.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.384+0000 7f70966dd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f70900b54c0 con 0x7f7090073130 2024-09-16T01:45:02.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.384+0000 7f708ffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f709015e560 0x7f709006e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:02.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.384+0000 7f708ffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f709015e560 0x7f709006e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54786/0 (socket says 172.21.15.88:54786) 2024-09-16T01:45:02.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.384+0000 7f708ffff640 1 -- 172.21.15.88:0/2532450540 learned_addr learned my addr 172.21.15.88:0/2532450540 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:02.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.384+0000 7f708ffff640 1 -- 172.21.15.88:0/2532450540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7090073130 msgr2=0x7f7090073580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:02.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.384+0000 7f708ffff640 1 --2- 172.21.15.88:0/2532450540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7090073130 0x7f7090073580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:02.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.384+0000 7f708ffff640 1 -- 172.21.15.88:0/2532450540 >> v2:172.21.15.154:3300/0 conn(0x7f7090072220 msgr2=0x7f70900ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:02.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.384+0000 7f708ffff640 1 --2- 172.21.15.88:0/2532450540 >> v2:172.21.15.154:3300/0 conn(0x7f7090072220 0x7f70900ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:02.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.384+0000 7f708ffff640 1 -- 172.21.15.88:0/2532450540 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70900b57a0 con 0x7f709015e560 2024-09-16T01:45:02.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.385+0000 7f708ffff640 1 --2- 172.21.15.88:0/2532450540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f709015e560 0x7f709006e780 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7f70900b4cb0 tx=0x7f708002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a48eb68698be8ea server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:02.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.385+0000 7f708effd640 1 -- 172.21.15.88:0/2532450540 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7080037070 con 0x7f709015e560 2024-09-16T01:45:02.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.385+0000 7f708effd640 1 -- 172.21.15.88:0/2532450540 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7080032440 con 0x7f709015e560 2024-09-16T01:45:02.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.385+0000 7f708effd640 1 -- 172.21.15.88:0/2532450540 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7080032810 con 0x7f709015e560 2024-09-16T01:45:02.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.386+0000 7f70966dd640 1 -- 172.21.15.88:0/2532450540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f709015e560 msgr2=0x7f709006e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:02.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.386+0000 7f70966dd640 1 --2- 172.21.15.88:0/2532450540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f709015e560 0x7f709006e780 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7f70900b4cb0 tx=0x7f708002a1e0 comp rx=0 tx=0).stop 2024-09-16T01:45:02.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.386+0000 7f70966dd640 1 -- 172.21.15.88:0/2532450540 shutdown_connections 2024-09-16T01:45:02.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.386+0000 7f70966dd640 1 --2- 172.21.15.88:0/2532450540 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7090073130 0x7f7090073580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.386+0000 7f70966dd640 1 --2- 172.21.15.88:0/2532450540 >> v2:172.21.15.154:3300/0 conn(0x7f7090072220 0x7f70900ae890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.386+0000 7f70966dd640 1 --2- 172.21.15.88:0/2532450540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f709015e560 0x7f709006e780 unknown :-1 s=CLOSED pgs=2388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.386+0000 7f70966dd640 1 -- 172.21.15.88:0/2532450540 >> 172.21.15.88:0/2532450540 conn(0x7f709006bc10 msgr2=0x7f709006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.386+0000 7f70966dd640 1 -- 172.21.15.88:0/2532450540 shutdown_connections 2024-09-16T01:45:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.386+0000 7f70966dd640 1 -- 172.21.15.88:0/2532450540 wait complete. 2024-09-16T01:45:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.387+0000 7f70966dd640 1 Processor -- start 2024-09-16T01:45:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.387+0000 7f70966dd640 1 -- start start 2024-09-16T01:45:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.388+0000 7f70966dd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7090072220 0x7f709015d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.388+0000 7f70966dd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7090073130 0x7f709015b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.388+0000 7f70966dd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f709015e560 0x7f70901534b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.388+0000 7f70966dd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f7090072660 con 0x7f7090072220 2024-09-16T01:45:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.388+0000 7f70966dd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f70900b5d60 con 0x7f709015e560 2024-09-16T01:45:02.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.388+0000 7f70966dd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f709015e940 con 0x7f7090073130 2024-09-16T01:45:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.388+0000 7f708ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7090072220 0x7f709015d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.388+0000 7f708ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7090072220 0x7f709015d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54788/0 (socket says 172.21.15.88:54788) 2024-09-16T01:45:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.388+0000 7f708ffff640 1 -- 172.21.15.88:0/4145606863 learned_addr learned my addr 172.21.15.88:0/4145606863 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.388+0000 7f708ffff640 1 -- 172.21.15.88:0/4145606863 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7090073130 msgr2=0x7f709015b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.388+0000 7f708ffff640 1 --2- 172.21.15.88:0/4145606863 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7090073130 0x7f709015b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.388+0000 7f708ffff640 1 -- 172.21.15.88:0/4145606863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f709015e560 msgr2=0x7f70901534b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.388+0000 7f708ffff640 1 --2- 172.21.15.88:0/4145606863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f709015e560 0x7f70901534b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.388+0000 7f708ffff640 1 -- 172.21.15.88:0/4145606863 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f709015a5e0 con 0x7f7090072220 2024-09-16T01:45:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.388+0000 7f708ffff640 1 --2- 172.21.15.88:0/4145606863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7090072220 0x7f709015d990 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f7080000c00 tx=0x7f70800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.389+0000 7f708cff9640 1 -- 172.21.15.88:0/4145606863 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7080041070 con 0x7f7090072220 2024-09-16T01:45:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.389+0000 7f708cff9640 1 -- 172.21.15.88:0/4145606863 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f708003c870 con 0x7f7090072220 2024-09-16T01:45:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.389+0000 7f70966dd640 1 -- 172.21.15.88:0/4145606863 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f709015a8c0 con 0x7f7090072220 2024-09-16T01:45:02.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.389+0000 7f708cff9640 1 -- 172.21.15.88:0/4145606863 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7080037040 con 0x7f7090072220 2024-09-16T01:45:02.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.389+0000 7f70966dd640 1 -- 172.21.15.88:0/4145606863 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f709015ae80 con 0x7f7090072220 2024-09-16T01:45:02.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.390+0000 7f70966dd640 1 -- 172.21.15.88:0/4145606863 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7090073580 con 0x7f7090072220 2024-09-16T01:45:02.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.390+0000 7f708cff9640 1 -- 172.21.15.88:0/4145606863 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f7080044070 con 0x7f7090072220 2024-09-16T01:45:02.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.390+0000 7f708cff9640 1 --2- 172.21.15.88:0/4145606863 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f707c06c2b0 0x7f707c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:02.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.391+0000 7f708cff9640 1 -- 172.21.15.88:0/4145606863 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5795+0+0 (secure 0 0 0) 0x7f70800ac0c0 con 0x7f7090072220 2024-09-16T01:45:02.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.391+0000 7f708f7fe640 1 --2- 172.21.15.88:0/4145606863 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f707c06c2b0 0x7f707c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:02.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.391+0000 7f708f7fe640 1 --2- 172.21.15.88:0/4145606863 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f707c06c2b0 0x7f707c06e770 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f7090159540 tx=0x7f7084006c90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:02.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.395+0000 7f708cff9640 1 -- 172.21.15.88:0/4145606863 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f708007a7e0 con 0x7f7090072220 2024-09-16T01:45:02.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.600+0000 7f70966dd640 1 -- 172.21.15.88:0/4145606863 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f709006c800 con 0x7f7090072220 2024-09-16T01:45:02.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.601+0000 7f708cff9640 1 -- 172.21.15.88:0/4145606863 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 329 v329) ==== 95+0+5235 (secure 0 0 0) 0x7f708007a0e0 con 0x7f7090072220 2024-09-16T01:45:02.602 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:45:02.602 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":329,"btime":"2024-09-16T01:45:02:267982+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7649,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2706550508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2706550508},{"type":"v1","addr":"172.21.15.154:6834","nonce":2706550508}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322},{"gid":8569,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/3460839382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":3460839382},{"type":"v1","addr":"172.21.15.88:6837","nonce":3460839382}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322},{"gid":8581,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2822970755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2822970755},{"type":"v1","addr":"172.21.15.154:6835","nonce":2822970755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322}],"filesystems":[{"mdsmap":{"epoch":329,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:44:51.845091+0000","modified":"2024-09-16T01:45:02.267968+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":139,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10707},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10707":{"gid":10707,"name":"a","rank":0,"incarnation":329,"state":"up:replay","state_seq":1,"addr":"172.21.15.88:6836/4245333273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4245333273},{"type":"v1","addr":"172.21.15.88:6836","nonce":4245333273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-09-16T01:45:02.603 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 329 2024-09-16T01:45:02.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.603+0000 7f70966dd640 1 -- 172.21.15.88:0/4145606863 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f707c06c2b0 msgr2=0x7f707c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:02.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.603+0000 7f70966dd640 1 --2- 172.21.15.88:0/4145606863 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f707c06c2b0 0x7f707c06e770 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f7090159540 tx=0x7f7084006c90 comp rx=0 tx=0).stop 2024-09-16T01:45:02.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.603+0000 7f70966dd640 1 -- 172.21.15.88:0/4145606863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7090072220 msgr2=0x7f709015d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:02.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.603+0000 7f70966dd640 1 --2- 172.21.15.88:0/4145606863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7090072220 0x7f709015d990 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f7080000c00 tx=0x7f70800046c0 comp rx=0 tx=0).stop 2024-09-16T01:45:02.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.603+0000 7f70966dd640 1 -- 172.21.15.88:0/4145606863 shutdown_connections 2024-09-16T01:45:02.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.603+0000 7f70966dd640 1 --2- 172.21.15.88:0/4145606863 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f707c06c2b0 0x7f707c06e770 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:02.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.603+0000 7f70966dd640 1 --2- 172.21.15.88:0/4145606863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f709015e560 0x7f70901534b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:02.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.603+0000 7f70966dd640 1 --2- 172.21.15.88:0/4145606863 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7090073130 0x7f709015b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:02.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.603+0000 7f70966dd640 1 --2- 172.21.15.88:0/4145606863 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7090072220 0x7f709015d990 unknown :-1 s=CLOSED pgs=2389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:02.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.603+0000 7f70966dd640 1 -- 172.21.15.88:0/4145606863 >> 172.21.15.88:0/4145606863 conn(0x7f709006bc10 msgr2=0x7f709006d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:02.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.603+0000 7f70966dd640 1 -- 172.21.15.88:0/4145606863 shutdown_connections 2024-09-16T01:45:02.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:02.603+0000 7f70966dd640 1 -- 172.21.15.88:0/4145606863 wait complete. 2024-09-16T01:45:03.017 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 329, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:44:51.845091+0000', 'modified': '2024-09-16T01:45:02.267968+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 139, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10707}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10707': {'gid': 10707, 'name': 'a', 'rank': 0, 'incarnation': 329, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.88:6836/4245333273', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 4245333273}, {'type': 'v1', 'addr': '172.21.15.88:6836', 'nonce': 4245333273}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-16T01:45:03.018 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10707:up:replay 2024-09-16T01:45:04.019 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:45:04.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.127+0000 7f3bcd39f640 1 Processor -- start 2024-09-16T01:45:04.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.127+0000 7f3bcd39f640 1 -- start start 2024-09-16T01:45:04.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.128+0000 7f3bcd39f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3bc80cf520 0x7f3bc806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:04.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.128+0000 7f3bcd39f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bc806e490 con 0x7f3bc80b0280 2024-09-16T01:45:04.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.128+0000 7f3bcd39f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bc80cfa20 con 0x7f3bc80af060 2024-09-16T01:45:04.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.128+0000 7f3bcd39f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f3bc806f490 con 0x7f3bc80cf520 2024-09-16T01:45:04.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.128+0000 7f3bbe7fc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3bc80cf520 0x7f3bc806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:04.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.128+0000 7f3bc77fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3bc80b0280 0x7f3bc8071ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35458/0 (socket says 172.21.15.88:35458) 2024-09-16T01:45:04.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.128+0000 7f3bc77fe640 1 -- 172.21.15.88:0/2278463464 learned_addr learned my addr 172.21.15.88:0/2278463464 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:04.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.129+0000 7f3bbe7fc640 1 -- 172.21.15.88:0/2278463464 >> v1:172.21.15.154:6789/0 conn(0x7f3bc80af060 legacy=0x7f3bc80af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:04.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.129+0000 7f3bbe7fc640 1 -- 172.21.15.88:0/2278463464 >> v1:172.21.15.88:6789/0 conn(0x7f3bc80b0280 legacy=0x7f3bc8071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:04.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.129+0000 7f3bc67fc640 1 -- 172.21.15.88:0/2278463464 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3514128703 0 0) 0x7f3bc806e490 con 0x7f3bc80b0280 2024-09-16T01:45:04.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.129+0000 7f3bbe7fc640 1 -- 172.21.15.88:0/2278463464 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc80cfd00 con 0x7f3bc80cf520 2024-09-16T01:45:04.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.130+0000 7f3bbe7fc640 1 --2- 172.21.15.88:0/2278463464 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc80cf520 0x7f3bc806ef50 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f3bb00097a0 tx=0x7f3bb002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca77351386a1b24e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:04.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.130+0000 7f3bc67fc640 1 -- 172.21.15.88:0/2278463464 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb0037070 con 0x7f3bc80cf520 2024-09-16T01:45:04.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.130+0000 7f3bc67fc640 1 -- 172.21.15.88:0/2278463464 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bb0032440 con 0x7f3bc80cf520 2024-09-16T01:45:04.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.130+0000 7f3bc67fc640 1 -- 172.21.15.88:0/2278463464 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb0032810 con 0x7f3bc80cf520 2024-09-16T01:45:04.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.130+0000 7f3bcd39f640 1 -- 172.21.15.88:0/2278463464 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc80cf520 msgr2=0x7f3bc806ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:04.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.130+0000 7f3bcd39f640 1 --2- 172.21.15.88:0/2278463464 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc80cf520 0x7f3bc806ef50 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f3bb00097a0 tx=0x7f3bb002a1e0 comp rx=0 tx=0).stop 2024-09-16T01:45:04.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.130+0000 7f3bcd39f640 1 -- 172.21.15.88:0/2278463464 shutdown_connections 2024-09-16T01:45:04.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.130+0000 7f3bcd39f640 1 --2- 172.21.15.88:0/2278463464 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc80cf520 0x7f3bc806ef50 unknown :-1 s=CLOSED pgs=2391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:04.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.130+0000 7f3bcd39f640 1 -- 172.21.15.88:0/2278463464 >> 172.21.15.88:0/2278463464 conn(0x7f3bc806bc10 msgr2=0x7f3bc806bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:04.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.130+0000 7f3bcd39f640 1 -- 172.21.15.88:0/2278463464 shutdown_connections 2024-09-16T01:45:04.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.130+0000 7f3bcd39f640 1 -- 172.21.15.88:0/2278463464 wait complete. 2024-09-16T01:45:04.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.131+0000 7f3bcd39f640 1 Processor -- start 2024-09-16T01:45:04.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.132+0000 7f3bcd39f640 1 -- start start 2024-09-16T01:45:04.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.132+0000 7f3bcd39f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc80af060 0x7f3bc8148790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:04.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.132+0000 7f3bcd39f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc80b0280 0x7f3bc81585a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:04.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.132+0000 7f3bcd39f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bc80cf520 0x7f3bc81f4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:04.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.132+0000 7f3bcd39f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc815e9f0 con 0x7f3bc80af060 2024-09-16T01:45:04.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.132+0000 7f3bcd39f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc806eac0 con 0x7f3bc80cf520 2024-09-16T01:45:04.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.132+0000 7f3bcd39f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3bc815eb70 con 0x7f3bc80b0280 2024-09-16T01:45:04.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.133+0000 7f3bc6ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc80af060 0x7f3bc8148790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:04.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.133+0000 7f3bc6ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc80af060 0x7f3bc8148790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54798/0 (socket says 172.21.15.88:54798) 2024-09-16T01:45:04.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.133+0000 7f3bc6ffd640 1 -- 172.21.15.88:0/578270689 learned_addr learned my addr 172.21.15.88:0/578270689 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:04.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.133+0000 7f3bbe7fc640 1 --2- 172.21.15.88:0/578270689 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc80b0280 0x7f3bc81585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:04.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.133+0000 7f3bc77fe640 1 --2- 172.21.15.88:0/578270689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bc80cf520 0x7f3bc81f4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:04.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.133+0000 7f3bc6ffd640 1 -- 172.21.15.88:0/578270689 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc80b0280 msgr2=0x7f3bc81585a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:04.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.133+0000 7f3bc6ffd640 1 --2- 172.21.15.88:0/578270689 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc80b0280 0x7f3bc81585a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:04.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.133+0000 7f3bc6ffd640 1 -- 172.21.15.88:0/578270689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bc80cf520 msgr2=0x7f3bc81f4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:04.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.133+0000 7f3bc6ffd640 1 --2- 172.21.15.88:0/578270689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bc80cf520 0x7f3bc81f4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:04.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.133+0000 7f3bc6ffd640 1 -- 172.21.15.88:0/578270689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc8153ad0 con 0x7f3bc80af060 2024-09-16T01:45:04.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.133+0000 7f3bc6ffd640 1 --2- 172.21.15.88:0/578270689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc80af060 0x7f3bc8148790 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f3bb800b3f0 tx=0x7f3bb800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:04.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.133+0000 7f3bbffff640 1 -- 172.21.15.88:0/578270689 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8015020 con 0x7f3bc80af060 2024-09-16T01:45:04.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.133+0000 7f3bbffff640 1 -- 172.21.15.88:0/578270689 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bb8011830 con 0x7f3bc80af060 2024-09-16T01:45:04.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.133+0000 7f3bcd39f640 1 -- 172.21.15.88:0/578270689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bc81591d0 con 0x7f3bc80af060 2024-09-16T01:45:04.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.133+0000 7f3bbffff640 1 -- 172.21.15.88:0/578270689 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8011c20 con 0x7f3bc80af060 2024-09-16T01:45:04.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.133+0000 7f3bcd39f640 1 -- 172.21.15.88:0/578270689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bc8154050 con 0x7f3bc80af060 2024-09-16T01:45:04.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.134+0000 7f3bcd39f640 1 -- 172.21.15.88:0/578270689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bc8068d70 con 0x7f3bc80af060 2024-09-16T01:45:04.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.135+0000 7f3bbffff640 1 -- 172.21.15.88:0/578270689 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3bb8018020 con 0x7f3bc80af060 2024-09-16T01:45:04.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.135+0000 7f3bbffff640 1 --2- 172.21.15.88:0/578270689 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3ba806c2b0 0x7f3ba806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:04.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.135+0000 7f3bbffff640 1 -- 172.21.15.88:0/578270689 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5795+0+0 (secure 0 0 0) 0x7f3bb808b040 con 0x7f3bc80af060 2024-09-16T01:45:04.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.135+0000 7f3bbe7fc640 1 --2- 172.21.15.88:0/578270689 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3ba806c2b0 0x7f3ba806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:04.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.136+0000 7f3bbe7fc640 1 --2- 172.21.15.88:0/578270689 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3ba806c2b0 0x7f3ba806e770 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0034000 tx=0x7f3bb002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:04.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.140+0000 7f3bbffff640 1 -- 172.21.15.88:0/578270689 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3bb801f030 con 0x7f3bc80af060 2024-09-16T01:45:04.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.342+0000 7f3bcd39f640 1 -- 172.21.15.88:0/578270689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3bc80cf300 con 0x7f3bc80af060 2024-09-16T01:45:04.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.343+0000 7f3bbffff640 1 -- 172.21.15.88:0/578270689 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 331 v331) ==== 95+0+5235 (secure 0 0 0) 0x7f3bb80597e0 con 0x7f3bc80af060 2024-09-16T01:45:04.422 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:45:04.422 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":331,"btime":"2024-09-16T01:45:04:271487+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7649,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2706550508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2706550508},{"type":"v1","addr":"172.21.15.154:6834","nonce":2706550508}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322},{"gid":8569,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/3460839382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":3460839382},{"type":"v1","addr":"172.21.15.88:6837","nonce":3460839382}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322},{"gid":8581,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2822970755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2822970755},{"type":"v1","addr":"172.21.15.154:6835","nonce":2822970755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322}],"filesystems":[{"mdsmap":{"epoch":331,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:44:51.845091+0000","modified":"2024-09-16T01:45:03.273730+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":139,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10707},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10707":{"gid":10707,"name":"a","rank":0,"incarnation":329,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.88:6836/4245333273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4245333273},{"type":"v1","addr":"172.21.15.88:6836","nonce":4245333273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-09-16T01:45:04.424 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 331 2024-09-16T01:45:04.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.345+0000 7f3bcd39f640 1 -- 172.21.15.88:0/578270689 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3ba806c2b0 msgr2=0x7f3ba806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:04.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.345+0000 7f3bcd39f640 1 --2- 172.21.15.88:0/578270689 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3ba806c2b0 0x7f3ba806e770 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0034000 tx=0x7f3bb002e000 comp rx=0 tx=0).stop 2024-09-16T01:45:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.345+0000 7f3bcd39f640 1 -- 172.21.15.88:0/578270689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc80af060 msgr2=0x7f3bc8148790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.345+0000 7f3bcd39f640 1 --2- 172.21.15.88:0/578270689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc80af060 0x7f3bc8148790 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f3bb800b3f0 tx=0x7f3bb800b8b0 comp rx=0 tx=0).stop 2024-09-16T01:45:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.345+0000 7f3bcd39f640 1 -- 172.21.15.88:0/578270689 shutdown_connections 2024-09-16T01:45:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.345+0000 7f3bcd39f640 1 --2- 172.21.15.88:0/578270689 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3ba806c2b0 0x7f3ba806e770 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.345+0000 7f3bcd39f640 1 --2- 172.21.15.88:0/578270689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bc80cf520 0x7f3bc81f4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.345+0000 7f3bcd39f640 1 --2- 172.21.15.88:0/578270689 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc80b0280 0x7f3bc81585a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.345+0000 7f3bcd39f640 1 --2- 172.21.15.88:0/578270689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc80af060 0x7f3bc8148790 unknown :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:04.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.345+0000 7f3bcd39f640 1 -- 172.21.15.88:0/578270689 >> 172.21.15.88:0/578270689 conn(0x7f3bc806bc10 msgr2=0x7f3bc815cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.345+0000 7f3bcd39f640 1 -- 172.21.15.88:0/578270689 shutdown_connections 2024-09-16T01:45:04.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:04.345+0000 7f3bcd39f640 1 -- 172.21.15.88:0/578270689 wait complete. 2024-09-16T01:45:04.829 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 331, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:44:51.845091+0000', 'modified': '2024-09-16T01:45:03.273730+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 139, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10707}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10707': {'gid': 10707, 'name': 'a', 'rank': 0, 'incarnation': 329, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.88:6836/4245333273', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 4245333273}, {'type': 'v1', 'addr': '172.21.15.88:6836', 'nonce': 4245333273}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-16T01:45:04.829 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10707:up:rejoin 2024-09-16T01:45:05.831 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:45:05.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.940+0000 7f185baa5640 1 Processor -- start 2024-09-16T01:45:05.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.940+0000 7f185baa5640 1 -- start start 2024-09-16T01:45:05.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.941+0000 7f185baa5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1854155800 0x7f18540b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:05.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.941+0000 7f185baa5640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f18540cfc80 0x7f18540b4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:05.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.941+0000 7f185baa5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18540b50a0 con 0x7f185415c330 2024-09-16T01:45:05.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.941+0000 7f185baa5640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f18540b4e60 con 0x7f18540cfc80 2024-09-16T01:45:05.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.941+0000 7f185baa5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f185415c7a0 con 0x7f1854155800 2024-09-16T01:45:05.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.941+0000 7f185a042640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f18540cfc80 0x7f18540b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:05.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.941+0000 7f1859040640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f185415c330 0x7f18540cf570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35470/0 (socket says 172.21.15.88:35470) 2024-09-16T01:45:05.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.941+0000 7f1859040640 1 -- 172.21.15.88:0/30366412 learned_addr learned my addr 172.21.15.88:0/30366412 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:05.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.941+0000 7f1859841640 1 --2- 172.21.15.88:0/30366412 >> v2:172.21.15.154:3300/0 conn(0x7f1854155800 0x7f18540b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:05.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.941+0000 7f185a042640 1 -- 172.21.15.88:0/30366412 >> v2:172.21.15.154:3300/0 conn(0x7f1854155800 msgr2=0x7f18540b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:05.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.941+0000 7f185a042640 1 --2- 172.21.15.88:0/30366412 >> v2:172.21.15.154:3300/0 conn(0x7f1854155800 0x7f18540b6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:05.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.941+0000 7f185a042640 1 -- 172.21.15.88:0/30366412 >> v1:172.21.15.88:6789/0 conn(0x7f185415c330 legacy=0x7f18540cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:05.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.941+0000 7f185a042640 1 -- 172.21.15.88:0/30366412 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18540b5380 con 0x7f18540cfc80 2024-09-16T01:45:05.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.941+0000 7f185883f640 1 -- 172.21.15.88:0/30366412 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2295662033 0 0) 0x7f18540b50a0 con 0x7f185415c330 2024-09-16T01:45:05.953 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.941+0000 7f1859841640 1 --2- 172.21.15.88:0/30366412 >> v2:172.21.15.154:3300/0 conn(0x7f1854155800 0x7f18540b6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:45:05.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.942+0000 7f185a042640 1 --2- 172.21.15.88:0/30366412 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18540cfc80 0x7f18540b4920 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7f185406c7f0 tx=0x7f184402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2b430f6dd28577d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:05.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.942+0000 7f185883f640 1 -- 172.21.15.88:0/30366412 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1844007740 con 0x7f18540cfc80 2024-09-16T01:45:05.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.942+0000 7f185883f640 1 -- 172.21.15.88:0/30366412 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18440078e0 con 0x7f18540cfc80 2024-09-16T01:45:05.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.942+0000 7f185883f640 1 -- 172.21.15.88:0/30366412 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1844007cb0 con 0x7f18540cfc80 2024-09-16T01:45:05.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.943+0000 7f185baa5640 1 -- 172.21.15.88:0/30366412 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18540cfc80 msgr2=0x7f18540b4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:05.954 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.943+0000 7f185baa5640 1 --2- 172.21.15.88:0/30366412 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18540cfc80 0x7f18540b4920 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7f185406c7f0 tx=0x7f184402a680 comp rx=0 tx=0).stop 2024-09-16T01:45:05.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.943+0000 7f185baa5640 1 -- 172.21.15.88:0/30366412 shutdown_connections 2024-09-16T01:45:05.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.943+0000 7f185baa5640 1 --2- 172.21.15.88:0/30366412 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18540cfc80 0x7f18540b4920 unknown :-1 s=CLOSED pgs=2394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:05.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.943+0000 7f185baa5640 1 --2- 172.21.15.88:0/30366412 >> v2:172.21.15.154:3300/0 conn(0x7f1854155800 0x7f18540b6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:05.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.943+0000 7f185baa5640 1 -- 172.21.15.88:0/30366412 >> 172.21.15.88:0/30366412 conn(0x7f185406bc10 msgr2=0x7f185406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:05.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.943+0000 7f185baa5640 1 -- 172.21.15.88:0/30366412 shutdown_connections 2024-09-16T01:45:05.955 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.943+0000 7f185baa5640 1 -- 172.21.15.88:0/30366412 wait complete. 2024-09-16T01:45:05.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.944+0000 7f185baa5640 1 Processor -- start 2024-09-16T01:45:05.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.944+0000 7f185baa5640 1 -- start start 2024-09-16T01:45:05.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.945+0000 7f185baa5640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18540cfc80 0x7f1854144f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:05.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.945+0000 7f185baa5640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1854155800 0x7f1854153800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:05.956 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.945+0000 7f185baa5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f185415c330 0x7f18541589a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:05.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.945+0000 7f185baa5640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f18540b5c20 con 0x7f18540cfc80 2024-09-16T01:45:05.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.945+0000 7f185baa5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f18540b5940 con 0x7f185415c330 2024-09-16T01:45:05.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.945+0000 7f185baa5640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1854155be0 con 0x7f1854155800 2024-09-16T01:45:05.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.945+0000 7f1859841640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18540cfc80 0x7f1854144f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:05.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.945+0000 7f1859841640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18540cfc80 0x7f1854144f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54818/0 (socket says 172.21.15.88:54818) 2024-09-16T01:45:05.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.945+0000 7f1859841640 1 -- 172.21.15.88:0/3747660715 learned_addr learned my addr 172.21.15.88:0/3747660715 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:05.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.945+0000 7f1859040640 1 --2- 172.21.15.88:0/3747660715 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1854155800 0x7f1854153800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:05.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.945+0000 7f185a042640 1 --2- 172.21.15.88:0/3747660715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f185415c330 0x7f18541589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:05.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.945+0000 7f1859841640 1 -- 172.21.15.88:0/3747660715 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1854155800 msgr2=0x7f1854153800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:05.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.945+0000 7f1859841640 1 --2- 172.21.15.88:0/3747660715 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1854155800 0x7f1854153800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:05.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.945+0000 7f1859841640 1 -- 172.21.15.88:0/3747660715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f185415c330 msgr2=0x7f18541589a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:05.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.945+0000 7f1859841640 1 --2- 172.21.15.88:0/3747660715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f185415c330 0x7f18541589a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:05.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.945+0000 7f1859841640 1 -- 172.21.15.88:0/3747660715 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18541560d0 con 0x7f18540cfc80 2024-09-16T01:45:05.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.945+0000 7f1859841640 1 --2- 172.21.15.88:0/3747660715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18540cfc80 0x7f1854144f20 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f184c00af70 tx=0x7f184c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:05.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.945+0000 7f18527fc640 1 -- 172.21.15.88:0/3747660715 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184c0079d0 con 0x7f18540cfc80 2024-09-16T01:45:05.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.945+0000 7f18527fc640 1 -- 172.21.15.88:0/3747660715 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f184c007b70 con 0x7f18540cfc80 2024-09-16T01:45:05.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.945+0000 7f185baa5640 1 -- 172.21.15.88:0/3747660715 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18541542a0 con 0x7f18540cfc80 2024-09-16T01:45:05.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.945+0000 7f18527fc640 1 -- 172.21.15.88:0/3747660715 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184c010440 con 0x7f18540cfc80 2024-09-16T01:45:05.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.945+0000 7f185baa5640 1 -- 172.21.15.88:0/3747660715 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1854154860 con 0x7f18540cfc80 2024-09-16T01:45:05.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.947+0000 7f185baa5640 1 -- 172.21.15.88:0/3747660715 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f185415bc60 con 0x7f18540cfc80 2024-09-16T01:45:05.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.947+0000 7f18527fc640 1 -- 172.21.15.88:0/3747660715 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f184c0105e0 con 0x7f18540cfc80 2024-09-16T01:45:05.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.948+0000 7f18527fc640 1 --2- 172.21.15.88:0/3747660715 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f184006c2e0 0x7f184006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:05.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.948+0000 7f18527fc640 1 -- 172.21.15.88:0/3747660715 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5795+0+0 (secure 0 0 0) 0x7f184c08d4c0 con 0x7f18540cfc80 2024-09-16T01:45:05.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.948+0000 7f1859040640 1 --2- 172.21.15.88:0/3747660715 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f184006c2e0 0x7f184006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:05.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.948+0000 7f1859040640 1 --2- 172.21.15.88:0/3747660715 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f184006c2e0 0x7f184006e7a0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f185415b070 tx=0x7f1848007c00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:05.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:05.952+0000 7f18527fc640 1 -- 172.21.15.88:0/3747660715 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f184c05bbd0 con 0x7f18540cfc80 2024-09-16T01:45:06.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.149+0000 7f185baa5640 1 -- 172.21.15.88:0/3747660715 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18540cf7c0 con 0x7f18540cfc80 2024-09-16T01:45:06.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.150+0000 7f18527fc640 1 -- 172.21.15.88:0/3747660715 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 332 v332) ==== 95+0+5244 (secure 0 0 0) 0x7f184c05b4d0 con 0x7f18540cfc80 2024-09-16T01:45:06.151 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:45:06.151 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":332,"btime":"2024-09-16T01:45:05:273239+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7649,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2706550508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2706550508},{"type":"v1","addr":"172.21.15.154:6834","nonce":2706550508}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322},{"gid":8569,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/3460839382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":3460839382},{"type":"v1","addr":"172.21.15.88:6837","nonce":3460839382}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322},{"gid":8581,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2822970755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2822970755},{"type":"v1","addr":"172.21.15.154:6835","nonce":2822970755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322}],"filesystems":[{"mdsmap":{"epoch":332,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:44:51.845091+0000","modified":"2024-09-16T01:45:05.273233+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":139,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10707},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10707":{"gid":10707,"name":"a","rank":0,"incarnation":329,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6836/4245333273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4245333273},{"type":"v1","addr":"172.21.15.88:6836","nonce":4245333273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10707,"qdb_cluster":[10707]},"id":7}]} 2024-09-16T01:45:06.152 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 332 2024-09-16T01:45:06.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.152+0000 7f185baa5640 1 -- 172.21.15.88:0/3747660715 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f184006c2e0 msgr2=0x7f184006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:06.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.152+0000 7f185baa5640 1 --2- 172.21.15.88:0/3747660715 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f184006c2e0 0x7f184006e7a0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f185415b070 tx=0x7f1848007c00 comp rx=0 tx=0).stop 2024-09-16T01:45:06.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.152+0000 7f185baa5640 1 -- 172.21.15.88:0/3747660715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18540cfc80 msgr2=0x7f1854144f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:06.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.152+0000 7f185baa5640 1 --2- 172.21.15.88:0/3747660715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18540cfc80 0x7f1854144f20 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f184c00af70 tx=0x7f184c00cc80 comp rx=0 tx=0).stop 2024-09-16T01:45:06.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.152+0000 7f185baa5640 1 -- 172.21.15.88:0/3747660715 shutdown_connections 2024-09-16T01:45:06.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.152+0000 7f185baa5640 1 --2- 172.21.15.88:0/3747660715 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f184006c2e0 0x7f184006e7a0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.152+0000 7f185baa5640 1 --2- 172.21.15.88:0/3747660715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f185415c330 0x7f18541589a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.152+0000 7f185baa5640 1 --2- 172.21.15.88:0/3747660715 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1854155800 0x7f1854153800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.152+0000 7f185baa5640 1 --2- 172.21.15.88:0/3747660715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f18540cfc80 0x7f1854144f20 unknown :-1 s=CLOSED pgs=2395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.152+0000 7f185baa5640 1 -- 172.21.15.88:0/3747660715 >> 172.21.15.88:0/3747660715 conn(0x7f185406bc10 msgr2=0x7f185406d650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:06.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.152+0000 7f185baa5640 1 -- 172.21.15.88:0/3747660715 shutdown_connections 2024-09-16T01:45:06.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.152+0000 7f185baa5640 1 -- 172.21.15.88:0/3747660715 wait complete. 2024-09-16T01:45:06.566 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 332, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-16T01:44:51.845091+0000', 'modified': '2024-09-16T01:45:05.273233+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 139, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10707}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10707': {'gid': 10707, 'name': 'a', 'rank': 0, 'incarnation': 329, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.88:6836/4245333273', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 4245333273}, {'type': 'v1', 'addr': '172.21.15.88:6836', 'nonce': 4245333273}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10707, 'qdb_cluster': [10707]} 2024-09-16T01:45:06.567 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-16T01:45:06.567 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-16T01:45:06.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.666+0000 7f4123331640 1 Processor -- start 2024-09-16T01:45:06.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.666+0000 7f4123331640 1 -- start start 2024-09-16T01:45:06.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.667+0000 7f4123331640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f411c0b6290 0x7f411c071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:06.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.667+0000 7f4123331640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f411c072530 0x7f411c15bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:06.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.667+0000 7f4123331640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f411c0ccff0 con 0x7f411c15b480 2024-09-16T01:45:06.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.667+0000 7f4123331640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f411c0cd2d0 con 0x7f411c0b6290 2024-09-16T01:45:06.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.667+0000 7f4123331640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f411c0cd5b0 con 0x7f411c072530 2024-09-16T01:45:06.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.667+0000 7f41208cc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f411c0b6290 0x7f411c071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:06.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.667+0000 7f41208cc640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f411c0b6290 0x7f411c071ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54822/0 (socket says 172.21.15.88:54822) 2024-09-16T01:45:06.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.667+0000 7f41208cc640 1 -- 172.21.15.88:0/431985062 learned_addr learned my addr 172.21.15.88:0/431985062 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:06.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.667+0000 7f41208cc640 1 -- 172.21.15.88:0/431985062 >> v2:172.21.15.154:3300/0 conn(0x7f411c072530 msgr2=0x7f411c15bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:06.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.667+0000 7f41218ce640 1 --2- 172.21.15.88:0/431985062 >> v2:172.21.15.154:3300/0 conn(0x7f411c072530 0x7f411c15bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:06.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.667+0000 7f41208cc640 1 --2- 172.21.15.88:0/431985062 >> v2:172.21.15.154:3300/0 conn(0x7f411c072530 0x7f411c15bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.667+0000 7f41208cc640 1 -- 172.21.15.88:0/431985062 >> v1:172.21.15.154:6789/0 conn(0x7f411c15b480 legacy=0x7f411c0b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:06.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.667+0000 7f41208cc640 1 -- 172.21.15.88:0/431985062 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f411c0cd890 con 0x7f411c0b6290 2024-09-16T01:45:06.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.667+0000 7f41218ce640 1 --2- 172.21.15.88:0/431985062 >> v2:172.21.15.154:3300/0 conn(0x7f411c072530 0x7f411c15bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:45:06.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.668+0000 7f41208cc640 1 --2- 172.21.15.88:0/431985062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f411c0b6290 0x7f411c071ff0 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7f411c0733f0 tx=0x7f411002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55adbd442c1421c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:06.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.668+0000 7f411bfff640 1 -- 172.21.15.88:0/431985062 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4110037070 con 0x7f411c0b6290 2024-09-16T01:45:06.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.668+0000 7f411bfff640 1 -- 172.21.15.88:0/431985062 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4110032440 con 0x7f411c0b6290 2024-09-16T01:45:06.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.668+0000 7f411bfff640 1 -- 172.21.15.88:0/431985062 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4110032810 con 0x7f411c0b6290 2024-09-16T01:45:06.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.669+0000 7f4123331640 1 -- 172.21.15.88:0/431985062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f411c0b6290 msgr2=0x7f411c071ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:06.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.669+0000 7f4123331640 1 --2- 172.21.15.88:0/431985062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f411c0b6290 0x7f411c071ff0 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7f411c0733f0 tx=0x7f411002a1a0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.669+0000 7f4123331640 1 -- 172.21.15.88:0/431985062 shutdown_connections 2024-09-16T01:45:06.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.669+0000 7f4123331640 1 --2- 172.21.15.88:0/431985062 >> v2:172.21.15.154:3300/0 conn(0x7f411c072530 0x7f411c15bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.669+0000 7f4123331640 1 --2- 172.21.15.88:0/431985062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f411c0b6290 0x7f411c071ff0 unknown :-1 s=CLOSED pgs=2396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.669+0000 7f4123331640 1 -- 172.21.15.88:0/431985062 >> 172.21.15.88:0/431985062 conn(0x7f411c06d0f0 msgr2=0x7f411c06d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:06.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.669+0000 7f4123331640 1 -- 172.21.15.88:0/431985062 shutdown_connections 2024-09-16T01:45:06.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.669+0000 7f4123331640 1 -- 172.21.15.88:0/431985062 wait complete. 2024-09-16T01:45:06.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.670+0000 7f4123331640 1 Processor -- start 2024-09-16T01:45:06.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.670+0000 7f4123331640 1 -- start start 2024-09-16T01:45:06.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.671+0000 7f4123331640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f411c072530 0x7f411c156760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:06.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.671+0000 7f4123331640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f411c15b480 0x7f411c0b3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:06.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.671+0000 7f4123331640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f411c0b3ad0 0x7f411c0c5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:06.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.671+0000 7f4123331640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f411c15b9e0 con 0x7f411c072530 2024-09-16T01:45:06.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.671+0000 7f4123331640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f411c15b860 con 0x7f411c15b480 2024-09-16T01:45:06.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.671+0000 7f4123331640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f411c15c1a0 con 0x7f411c0b3ad0 2024-09-16T01:45:06.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.671+0000 7f41210cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f411c072530 0x7f411c156760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:06.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.671+0000 7f41210cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f411c072530 0x7f411c156760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54836/0 (socket says 172.21.15.88:54836) 2024-09-16T01:45:06.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.671+0000 7f41210cd640 1 -- 172.21.15.88:0/382090591 learned_addr learned my addr 172.21.15.88:0/382090591 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:06.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.671+0000 7f41208cc640 1 --2- 172.21.15.88:0/382090591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f411c15b480 0x7f411c0b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:06.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.671+0000 7f41218ce640 1 --2- 172.21.15.88:0/382090591 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f411c0b3ad0 0x7f411c0c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:06.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.671+0000 7f41210cd640 1 -- 172.21.15.88:0/382090591 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f411c0b3ad0 msgr2=0x7f411c0c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:06.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.671+0000 7f41210cd640 1 --2- 172.21.15.88:0/382090591 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f411c0b3ad0 0x7f411c0c5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.671+0000 7f41210cd640 1 -- 172.21.15.88:0/382090591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f411c15b480 msgr2=0x7f411c0b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:06.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.671+0000 7f41210cd640 1 --2- 172.21.15.88:0/382090591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f411c15b480 0x7f411c0b3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.671+0000 7f41210cd640 1 -- 172.21.15.88:0/382090591 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f411c0c65e0 con 0x7f411c072530 2024-09-16T01:45:06.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.671+0000 7f41210cd640 1 --2- 172.21.15.88:0/382090591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f411c072530 0x7f411c156760 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f410c00ba80 tx=0x7f410c00bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:06.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.671+0000 7f4119ffb640 1 -- 172.21.15.88:0/382090591 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410c019070 con 0x7f411c072530 2024-09-16T01:45:06.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.672+0000 7f4119ffb640 1 -- 172.21.15.88:0/382090591 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f410c00cc90 con 0x7f411c072530 2024-09-16T01:45:06.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.672+0000 7f4123331640 1 -- 172.21.15.88:0/382090591 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f411c0c68c0 con 0x7f411c072530 2024-09-16T01:45:06.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.672+0000 7f4119ffb640 1 -- 172.21.15.88:0/382090591 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410c014680 con 0x7f411c072530 2024-09-16T01:45:06.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.672+0000 7f4123331640 1 -- 172.21.15.88:0/382090591 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f411c0c6e80 con 0x7f411c072530 2024-09-16T01:45:06.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.673+0000 7f4119ffb640 1 -- 172.21.15.88:0/382090591 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f410c007370 con 0x7f411c072530 2024-09-16T01:45:06.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.674+0000 7f4119ffb640 1 --2- 172.21.15.88:0/382090591 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f410806c2e0 0x7f410806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:06.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.674+0000 7f41208cc640 1 --2- 172.21.15.88:0/382090591 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f410806c2e0 0x7f410806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:06.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.674+0000 7f4119ffb640 1 -- 172.21.15.88:0/382090591 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5795+0+0 (secure 0 0 0) 0x7f410c08b100 con 0x7f411c072530 2024-09-16T01:45:06.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.674+0000 7f41208cc640 1 --2- 172.21.15.88:0/382090591 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f410806c2e0 0x7f410806e7a0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f4110034000 tx=0x7f411002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:06.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.675+0000 7f41260cb740 1 Processor -- start 2024-09-16T01:45:06.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.675+0000 7f41260cb740 1 -- start start 2024-09-16T01:45:06.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.676+0000 7f41260cb740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d720b202c0 0x55d720b65a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:06.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.676+0000 7f41260cb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d720b1da80 0x55d720b206a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:06.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.676+0000 7f41260cb740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b21a50 0x55d720b1a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:06.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.676+0000 7f41260cb740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d720a82520 con 0x55d720b21a50 2024-09-16T01:45:06.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.676+0000 7f41260cb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d720a2a3e0 con 0x55d720b1da80 2024-09-16T01:45:06.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.676+0000 7f41260cb740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55d720a7e4e0 con 0x55d720b202c0 2024-09-16T01:45:06.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.676+0000 7f41218ce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b21a50 0x55d720b1a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:06.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.676+0000 7f41218ce640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b21a50 0x55d720b1a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54852/0 (socket says 172.21.15.88:54852) 2024-09-16T01:45:06.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.676+0000 7f41218ce640 1 -- 172.21.15.88:0/1863844699 learned_addr learned my addr 172.21.15.88:0/1863844699 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:06.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.676+0000 7f41210cd640 1 --2- 172.21.15.88:0/1863844699 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d720b202c0 0x55d720b65a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:06.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.676+0000 7f41208cc640 1 --2- 172.21.15.88:0/1863844699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d720b1da80 0x55d720b206a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:06.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.676+0000 7f41218ce640 1 -- 172.21.15.88:0/1863844699 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d720b202c0 msgr2=0x55d720b65a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:06.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.676+0000 7f41218ce640 1 --2- 172.21.15.88:0/1863844699 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d720b202c0 0x55d720b65a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.676+0000 7f41218ce640 1 -- 172.21.15.88:0/1863844699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d720b1da80 msgr2=0x55d720b206a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:06.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.676+0000 7f41218ce640 1 --2- 172.21.15.88:0/1863844699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d720b1da80 0x55d720b206a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.676+0000 7f41218ce640 1 -- 172.21.15.88:0/1863844699 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d720a452f0 con 0x55d720b21a50 2024-09-16T01:45:06.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.676+0000 7f41208cc640 1 --2- 172.21.15.88:0/1863844699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d720b1da80 0x55d720b206a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:45:06.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.676+0000 7f41210cd640 1 --2- 172.21.15.88:0/1863844699 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d720b202c0 0x55d720b65a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:45:06.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.676+0000 7f41218ce640 1 --2- 172.21.15.88:0/1863844699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b21a50 0x55d720b1a8b0 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7f411400caf0 tx=0x7f411400e1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d34215e3e2516918 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:06.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.676+0000 7f4102ffd640 1 -- 172.21.15.88:0/1863844699 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4114017020 con 0x55d720b21a50 2024-09-16T01:45:06.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.676+0000 7f4102ffd640 1 -- 172.21.15.88:0/1863844699 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f411400ec10 con 0x55d720b21a50 2024-09-16T01:45:06.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.676+0000 7f4102ffd640 1 -- 172.21.15.88:0/1863844699 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4114013450 con 0x55d720b21a50 2024-09-16T01:45:06.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.677+0000 7f41260cb740 1 -- 172.21.15.88:0/1863844699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b21a50 msgr2=0x55d720b1a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:06.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.677+0000 7f41260cb740 1 --2- 172.21.15.88:0/1863844699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b21a50 0x55d720b1a8b0 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7f411400caf0 tx=0x7f411400e1d0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.677+0000 7f41260cb740 1 -- 172.21.15.88:0/1863844699 shutdown_connections 2024-09-16T01:45:06.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.677+0000 7f41260cb740 1 --2- 172.21.15.88:0/1863844699 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b21a50 0x55d720b1a8b0 unknown :-1 s=CLOSED pgs=2398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.677+0000 7f41260cb740 1 --2- 172.21.15.88:0/1863844699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d720b1da80 0x55d720b206a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.677+0000 7f41260cb740 1 --2- 172.21.15.88:0/1863844699 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d720b202c0 0x55d720b65a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.677+0000 7f41260cb740 1 -- 172.21.15.88:0/1863844699 >> 172.21.15.88:0/1863844699 conn(0x55d720b1ca80 msgr2=0x55d720b64f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:06.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.677+0000 7f41260cb740 1 -- 172.21.15.88:0/1863844699 shutdown_connections 2024-09-16T01:45:06.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.677+0000 7f41260cb740 1 -- 172.21.15.88:0/1863844699 wait complete. 2024-09-16T01:45:06.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.677+0000 7f41260cb740 1 -- start start 2024-09-16T01:45:06.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.678+0000 7f41260cb740 1 Processor -- start 2024-09-16T01:45:06.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.678+0000 7f41260cb740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b202c0 0x55d720b1ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:06.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.678+0000 7f41260cb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d720b21a50 0x55d720b3e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:06.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.678+0000 7f41260cb740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d720b3f110 0x55d720b40be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:06.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.679+0000 7f41260cb740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d720b1bf20 con 0x55d720b202c0 2024-09-16T01:45:06.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.679+0000 7f41260cb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55d720a0b450 con 0x55d720b21a50 2024-09-16T01:45:06.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.679+0000 7f41260cb740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55d720b66290 con 0x55d720b3f110 2024-09-16T01:45:06.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.679+0000 7f41210cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b202c0 0x55d720b1ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:06.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.679+0000 7f41260cb740 10 client.0 _ll_register_callbacks cb 0x55d720794ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:45:06.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.679+0000 7f41210cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b202c0 0x55d720b1ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54854/0 (socket says 172.21.15.88:54854) 2024-09-16T01:45:06.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.679+0000 7f41210cd640 1 -- 172.21.15.88:0/3861581656 learned_addr learned my addr 172.21.15.88:0/3861581656 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:06.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.679+0000 7f41208cc640 1 --2- 172.21.15.88:0/3861581656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d720b21a50 0x55d720b3e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:06.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.679+0000 7f41218ce640 1 --2- 172.21.15.88:0/3861581656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d720b3f110 0x55d720b40be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:06.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.679+0000 7f41210cd640 1 -- 172.21.15.88:0/3861581656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d720b3f110 msgr2=0x55d720b40be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:06.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.679+0000 7f41210cd640 1 --2- 172.21.15.88:0/3861581656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d720b3f110 0x55d720b40be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.679+0000 7f41210cd640 1 -- 172.21.15.88:0/3861581656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d720b21a50 msgr2=0x55d720b3e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:06.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.679+0000 7f41210cd640 1 --2- 172.21.15.88:0/3861581656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d720b21a50 0x55d720b3e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.679+0000 7f41210cd640 1 -- 172.21.15.88:0/3861581656 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7207952d0 con 0x55d720b202c0 2024-09-16T01:45:06.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.679+0000 7f41208cc640 1 --2- 172.21.15.88:0/3861581656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d720b21a50 0x55d720b3e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:45:06.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.679+0000 7f41210cd640 1 --2- 172.21.15.88:0/3861581656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b202c0 0x55d720b1ab80 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f410c00c760 tx=0x7f410c0598d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b62e87cc790f20 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:06.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.679+0000 7f40fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:45:06.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.679+0000 7f40fbfff640 1 -- 172.21.15.88:0/3861581656 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410c061cf0 con 0x55d720b202c0 2024-09-16T01:45:06.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.679+0000 7f40fbfff640 1 -- 172.21.15.88:0/3861581656 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 5795+0+0 (secure 0 0 0) 0x7f410c06ecb0 con 0x55d720b202c0 2024-09-16T01:45:06.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.679+0000 7f41260cb740 1 -- 172.21.15.88:0/3861581656 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f411400ec10 con 0x55d720b202c0 2024-09-16T01:45:06.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.679+0000 7f40fbfff640 1 -- 172.21.15.88:0/3861581656 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=332) ==== 24+0+0 (secure 0 0 0) 0x7f410c06eeb0 con 0x55d720b202c0 2024-09-16T01:45:06.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.679+0000 7f41260cb740 10 client.10770 fetch_fsmap learned FSMap version 332 2024-09-16T01:45:06.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.679+0000 7f41260cb740 1 -- 172.21.15.88:0/3861581656 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=332,monmap=2+}) -- 0x55d72085f6b0 con 0x55d720b202c0 2024-09-16T01:45:06.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.679+0000 7f40fbfff640 1 -- 172.21.15.88:0/3861581656 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f411400ec10 con 0x55d720b202c0 2024-09-16T01:45:06.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.680+0000 7f40fbfff640 1 -- 172.21.15.88:0/3861581656 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 332) ==== 3502+0+0 (secure 0 0 0) 0x7f410c061860 con 0x55d720b202c0 2024-09-16T01:45:06.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.680+0000 7f41260cb740 10 client.10770 fetch_fsmap finished waiting for FSMap version 332 2024-09-16T01:45:06.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.680+0000 7f41260cb740 10 client.10770 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:45:06.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.680+0000 7f41260cb740 20 client.10770 populate_metadata read hostname 'smithi088' 2024-09-16T01:45:06.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.680+0000 7f41260cb740 1 --2- 172.21.15.88:0/3861581656 >> [v2:172.21.15.88:6834/4245333273,v1:172.21.15.88:6836/4245333273] conn(0x55d720b80760 0x55d720b82b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:06.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.680+0000 7f41260cb740 4 client.10770 mds_command: new command op to 10707 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-16T01:45:06.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.680+0000 7f41260cb740 1 -- 172.21.15.88:0/3861581656 --> [v2:172.21.15.88:6834/4245333273,v1:172.21.15.88:6836/4245333273] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d720a8fc50 con 0x55d720b80760 2024-09-16T01:45:06.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.680+0000 7f41208cc640 1 --2- 172.21.15.88:0/3861581656 >> [v2:172.21.15.88:6834/4245333273,v1:172.21.15.88:6836/4245333273] conn(0x55d720b80760 0x55d720b82b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:06.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.680+0000 7f41208cc640 1 --2- 172.21.15.88:0/3861581656 >> [v2:172.21.15.88:6834/4245333273,v1:172.21.15.88:6836/4245333273] conn(0x55d720b80760 0x55d720b82b50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9938ec866bb4f20 server_cookie=f26ac96bb7f44804 in_seq=0 out_seq=0 2024-09-16T01:45:06.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.680+0000 7f40fbfff640 10 client.10770 ms_handle_connect on v2:172.21.15.88:6834/4245333273 2024-09-16T01:45:06.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.681+0000 7f40fbfff640 1 -- 172.21.15.88:0/3861581656 <== mds.0 v2:172.21.15.88:6834/4245333273 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d720a8fc50 con 0x55d720b80760 2024-09-16T01:45:06.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.681+0000 7f40fbfff640 10 client.10770 handle_command_reply: tid=0 2024-09-16T01:45:06.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.681+0000 7f41260cb740 1 client.10770 shutdown 2024-09-16T01:45:06.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.681+0000 7f41260cb740 1 -- 172.21.15.88:0/3861581656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b202c0 msgr2=0x55d720b1ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:06.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.681+0000 7f41260cb740 1 --2- 172.21.15.88:0/3861581656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b202c0 0x55d720b1ab80 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f410c00c760 tx=0x7f410c0598d0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.681+0000 7f41260cb740 1 -- 172.21.15.88:0/3861581656 shutdown_connections 2024-09-16T01:45:06.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.681+0000 7f41260cb740 1 --2- 172.21.15.88:0/3861581656 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d720b3f110 0x55d720b40be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.681+0000 7f41260cb740 1 --2- 172.21.15.88:0/3861581656 >> [v2:172.21.15.88:6834/4245333273,v1:172.21.15.88:6836/4245333273] conn(0x55d720b80760 0x55d720b82b50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.681+0000 7f41260cb740 1 --2- 172.21.15.88:0/3861581656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d720b21a50 0x55d720b3e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.681+0000 7f40fbfff640 0 client.10770 ms_handle_reset on v2:172.21.15.88:6834/4245333273 2024-09-16T01:45:06.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.681+0000 7f41260cb740 1 --2- 172.21.15.88:0/3861581656 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b202c0 0x55d720b1ab80 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.681+0000 7f41260cb740 1 -- 172.21.15.88:0/3861581656 >> 172.21.15.88:0/3861581656 conn(0x55d720b1da80 msgr2=0x55d720b68440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:06.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.681+0000 7f41260cb740 1 -- 172.21.15.88:0/3861581656 shutdown_connections 2024-09-16T01:45:06.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.681+0000 7f41260cb740 1 -- 172.21.15.88:0/3861581656 wait complete. 2024-09-16T01:45:06.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.681+0000 7f41260cb740 20 client.10770 trim_cache size 0 max 16384 2024-09-16T01:45:06.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.699+0000 7f41260cb740 1 Processor -- start 2024-09-16T01:45:06.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.699+0000 7f41260cb740 1 -- start start 2024-09-16T01:45:06.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.700+0000 7f41260cb740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b202c0 0x55d720b40bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:06.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.700+0000 7f41260cb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d720b21a50 0x55d720bd9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:06.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.700+0000 7f41260cb740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d720b3f110 0x55d720b687d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:06.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.700+0000 7f41260cb740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f411c1fea30 con 0x55d720b202c0 2024-09-16T01:45:06.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.700+0000 7f41260cb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f411c0cb430 con 0x55d720b21a50 2024-09-16T01:45:06.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.700+0000 7f41260cb740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f411c15d000 con 0x55d720b3f110 2024-09-16T01:45:06.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.700+0000 7f41210cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b202c0 0x55d720b40bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:06.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.700+0000 7f41210cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b202c0 0x55d720b40bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54868/0 (socket says 172.21.15.88:54868) 2024-09-16T01:45:06.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.700+0000 7f41210cd640 1 -- 172.21.15.88:0/2785385300 learned_addr learned my addr 172.21.15.88:0/2785385300 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:06.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.700+0000 7f41210cd640 1 -- 172.21.15.88:0/2785385300 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d720b3f110 msgr2=0x55d720b687d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:06.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.700+0000 7f41210cd640 1 --2- 172.21.15.88:0/2785385300 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d720b3f110 0x55d720b687d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.700+0000 7f41210cd640 1 -- 172.21.15.88:0/2785385300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d720b21a50 msgr2=0x55d720bd9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:06.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.700+0000 7f41210cd640 1 --2- 172.21.15.88:0/2785385300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d720b21a50 0x55d720bd9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.700+0000 7f41210cd640 1 -- 172.21.15.88:0/2785385300 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d720b168b0 con 0x55d720b202c0 2024-09-16T01:45:06.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.700+0000 7f41210cd640 1 --2- 172.21.15.88:0/2785385300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b202c0 0x55d720b40bd0 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f410c0590a0 tx=0x7f410c01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1f4374988e5ef2f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:06.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.700+0000 7f41027fc640 1 -- 172.21.15.88:0/2785385300 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410c089510 con 0x55d720b202c0 2024-09-16T01:45:06.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.700+0000 7f41027fc640 1 -- 172.21.15.88:0/2785385300 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f410c061cf0 con 0x55d720b202c0 2024-09-16T01:45:06.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.700+0000 7f41027fc640 1 -- 172.21.15.88:0/2785385300 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410c009090 con 0x55d720b202c0 2024-09-16T01:45:06.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.701+0000 7f41260cb740 1 -- 172.21.15.88:0/2785385300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b202c0 msgr2=0x55d720b40bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:06.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.701+0000 7f41260cb740 1 --2- 172.21.15.88:0/2785385300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b202c0 0x55d720b40bd0 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f410c0590a0 tx=0x7f410c01f000 comp rx=0 tx=0).stop 2024-09-16T01:45:06.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.701+0000 7f41260cb740 1 -- 172.21.15.88:0/2785385300 shutdown_connections 2024-09-16T01:45:06.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.701+0000 7f41260cb740 1 --2- 172.21.15.88:0/2785385300 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d720b3f110 0x55d720b687d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.701+0000 7f41260cb740 1 --2- 172.21.15.88:0/2785385300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d720b21a50 0x55d720bd9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.701+0000 7f41260cb740 1 --2- 172.21.15.88:0/2785385300 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b202c0 0x55d720b40bd0 unknown :-1 s=CLOSED pgs=2400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.701+0000 7f41260cb740 1 -- 172.21.15.88:0/2785385300 >> 172.21.15.88:0/2785385300 conn(0x55d720b1da80 msgr2=0x55d720b15f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:06.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.701+0000 7f41260cb740 1 -- 172.21.15.88:0/2785385300 shutdown_connections 2024-09-16T01:45:06.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.701+0000 7f41260cb740 1 -- 172.21.15.88:0/2785385300 wait complete. 2024-09-16T01:45:06.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.701+0000 7f41260cb740 1 -- start start 2024-09-16T01:45:06.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.701+0000 7f41260cb740 1 Processor -- start 2024-09-16T01:45:06.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.702+0000 7f41260cb740 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b202c0 0x55d720b689c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:06.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.702+0000 7f41260cb740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d720b21a50 0x55d720bc14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:06.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.702+0000 7f41260cb740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d720b3f110 0x55d720bc1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:06.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.702+0000 7f41260cb740 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55d720b41530 con 0x55d720b202c0 2024-09-16T01:45:06.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.702+0000 7f41260cb740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f411c0cade0 con 0x55d720b3f110 2024-09-16T01:45:06.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.702+0000 7f41260cb740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55d720b48f90 con 0x55d720b21a50 2024-09-16T01:45:06.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.702+0000 7f41260cb740 10 client.0 _ll_register_callbacks cb 0x7f410c009970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-16T01:45:06.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.702+0000 7f41210cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b202c0 0x55d720b689c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:06.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.702+0000 7f41210cd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b202c0 0x55d720b689c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54872/0 (socket says 172.21.15.88:54872) 2024-09-16T01:45:06.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.702+0000 7f41210cd640 1 -- 172.21.15.88:0/897557171 learned_addr learned my addr 172.21.15.88:0/897557171 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:06.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.703+0000 7f41208cc640 1 --2- 172.21.15.88:0/897557171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d720b21a50 0x55d720bc14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:06.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.703+0000 7f41218ce640 1 --2- 172.21.15.88:0/897557171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d720b3f110 0x55d720bc1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:06.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.703+0000 7f41210cd640 1 -- 172.21.15.88:0/897557171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d720b21a50 msgr2=0x55d720bc14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:06.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.703+0000 7f41210cd640 1 --2- 172.21.15.88:0/897557171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d720b21a50 0x55d720bc14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.703+0000 7f41210cd640 1 -- 172.21.15.88:0/897557171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d720b3f110 msgr2=0x55d720bc1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:06.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.703+0000 7f41210cd640 1 --2- 172.21.15.88:0/897557171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d720b3f110 0x55d720bc1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.703+0000 7f41210cd640 1 -- 172.21.15.88:0/897557171 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d720bc2140 con 0x55d720b202c0 2024-09-16T01:45:06.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.703+0000 7f41210cd640 1 --2- 172.21.15.88:0/897557171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b202c0 0x55d720b689c0 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7f410c058dc0 tx=0x7f410c0894a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca32dd5c79fdb222 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:06.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.703+0000 7f40fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2024-09-16T01:45:06.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.703+0000 7f40fbfff640 1 -- 172.21.15.88:0/897557171 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410c0587b0 con 0x55d720b202c0 2024-09-16T01:45:06.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.703+0000 7f40fbfff640 1 -- 172.21.15.88:0/897557171 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(139..139 src has 1..139) ==== 5795+0+0 (secure 0 0 0) 0x7f410c079bc0 con 0x55d720b202c0 2024-09-16T01:45:06.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.703+0000 7f41260cb740 1 -- 172.21.15.88:0/897557171 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f410c061cf0 con 0x55d720b202c0 2024-09-16T01:45:06.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.703+0000 7f40fbfff640 1 -- 172.21.15.88:0/897557171 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=332) ==== 24+0+0 (secure 0 0 0) 0x7f410c079dc0 con 0x55d720b202c0 2024-09-16T01:45:06.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.703+0000 7f41260cb740 10 client.10776 fetch_fsmap learned FSMap version 332 2024-09-16T01:45:06.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.703+0000 7f41260cb740 1 -- 172.21.15.88:0/897557171 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=332,monmap=2+}) -- 0x55d720bc2140 con 0x55d720b202c0 2024-09-16T01:45:06.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.703+0000 7f40fbfff640 1 -- 172.21.15.88:0/897557171 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f410c061cf0 con 0x55d720b202c0 2024-09-16T01:45:06.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.703+0000 7f40fbfff640 1 -- 172.21.15.88:0/897557171 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 332) ==== 3502+0+0 (secure 0 0 0) 0x7f410c089040 con 0x55d720b202c0 2024-09-16T01:45:06.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.703+0000 7f41260cb740 10 client.10776 fetch_fsmap finished waiting for FSMap version 332 2024-09-16T01:45:06.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.703+0000 7f41260cb740 10 client.10776 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-16T01:45:06.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.703+0000 7f41260cb740 20 client.10776 populate_metadata read hostname 'smithi088' 2024-09-16T01:45:06.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.703+0000 7f41260cb740 1 --2- 172.21.15.88:0/897557171 >> [v2:172.21.15.88:6834/4245333273,v1:172.21.15.88:6836/4245333273] conn(0x55d720b46c60 0x55d720b47040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:06.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.703+0000 7f41260cb740 4 client.10776 mds_command: new command op to 10707 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-16T01:45:06.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.703+0000 7f41260cb740 1 -- 172.21.15.88:0/897557171 --> [v2:172.21.15.88:6834/4245333273,v1:172.21.15.88:6836/4245333273] -- command(tid 0: {"prefix": "status"}) -- 0x55d720a89fa0 con 0x55d720b46c60 2024-09-16T01:45:06.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.703+0000 7f41208cc640 1 --2- 172.21.15.88:0/897557171 >> [v2:172.21.15.88:6834/4245333273,v1:172.21.15.88:6836/4245333273] conn(0x55d720b46c60 0x55d720b47040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:06.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.704+0000 7f40fbfff640 10 client.10776 ms_handle_connect on v2:172.21.15.88:6834/4245333273 2024-09-16T01:45:06.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.704+0000 7f41208cc640 1 --2- 172.21.15.88:0/897557171 >> [v2:172.21.15.88:6834/4245333273,v1:172.21.15.88:6836/4245333273] conn(0x55d720b46c60 0x55d720b47040 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b53800f20ac5742 server_cookie=32b7652281c6e9b in_seq=0 out_seq=0 2024-09-16T01:45:06.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.704+0000 7f40fbfff640 1 -- 172.21.15.88:0/897557171 <== mds.0 v2:172.21.15.88:6834/4245333273 1 ==== command_reply(tid 0: 0 ) ==== 8+0+335 (crc 0 0 0) 0x55d720a89fa0 con 0x55d720b46c60 2024-09-16T01:45:06.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.704+0000 7f40fbfff640 10 client.10776 handle_command_reply: tid=0 2024-09-16T01:45:06.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.704+0000 7f41260cb740 1 client.10776 shutdown 2024-09-16T01:45:06.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.704+0000 7f41260cb740 1 -- 172.21.15.88:0/897557171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b202c0 msgr2=0x55d720b689c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:06.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.704+0000 7f41260cb740 1 --2- 172.21.15.88:0/897557171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b202c0 0x55d720b689c0 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7f410c058dc0 tx=0x7f410c0894a0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.704+0000 7f41260cb740 1 -- 172.21.15.88:0/897557171 shutdown_connections 2024-09-16T01:45:06.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.704+0000 7f41260cb740 1 --2- 172.21.15.88:0/897557171 >> [v2:172.21.15.88:6834/4245333273,v1:172.21.15.88:6836/4245333273] conn(0x55d720b46c60 0x55d720b47040 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.704+0000 7f41260cb740 1 --2- 172.21.15.88:0/897557171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d720b3f110 0x55d720bc1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.704+0000 7f41260cb740 1 --2- 172.21.15.88:0/897557171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55d720b21a50 0x55d720bc14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.704+0000 7f40fbfff640 0 client.10776 ms_handle_reset on v2:172.21.15.88:6834/4245333273 2024-09-16T01:45:06.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.704+0000 7f41260cb740 1 --2- 172.21.15.88:0/897557171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d720b202c0 0x55d720b689c0 unknown :-1 s=CLOSED pgs=2401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.704+0000 7f41260cb740 1 -- 172.21.15.88:0/897557171 >> 172.21.15.88:0/897557171 conn(0x55d720b1da80 msgr2=0x55d720b409a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:06.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.704+0000 7f41260cb740 1 -- 172.21.15.88:0/897557171 shutdown_connections 2024-09-16T01:45:06.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.704+0000 7f41260cb740 1 -- 172.21.15.88:0/897557171 wait complete. 2024-09-16T01:45:06.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.704+0000 7f41260cb740 20 client.10776 trim_cache size 0 max 16384 2024-09-16T01:45:06.712 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:45:06.713 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "5ace2c91-46f9-482b-8a89-731fbdc171bf", 2024-09-16T01:45:06.713 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2024-09-16T01:45:06.713 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 10707, 2024-09-16T01:45:06.713 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2024-09-16T01:45:06.713 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2024-09-16T01:45:06.713 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2024-09-16T01:45:06.713 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 4.4321490890000002, 2024-09-16T01:45:06.713 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 332, 2024-09-16T01:45:06.713 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 139, 2024-09-16T01:45:06.713 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 139, 2024-09-16T01:45:06.713 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 5.4021838080000002 2024-09-16T01:45:06.713 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:45:06.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.705+0000 7f4123331640 1 -- 172.21.15.88:0/382090591 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f410806c2e0 msgr2=0x7f410806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:06.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.705+0000 7f4123331640 1 --2- 172.21.15.88:0/382090591 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f410806c2e0 0x7f410806e7a0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f4110034000 tx=0x7f411002e000 comp rx=0 tx=0).stop 2024-09-16T01:45:06.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.705+0000 7f4123331640 1 -- 172.21.15.88:0/382090591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f411c072530 msgr2=0x7f411c156760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:06.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.705+0000 7f4123331640 1 --2- 172.21.15.88:0/382090591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f411c072530 0x7f411c156760 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f410c00ba80 tx=0x7f410c00bf40 comp rx=0 tx=0).stop 2024-09-16T01:45:06.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.705+0000 7f4123331640 1 -- 172.21.15.88:0/382090591 shutdown_connections 2024-09-16T01:45:06.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.705+0000 7f4123331640 1 --2- 172.21.15.88:0/382090591 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f410806c2e0 0x7f410806e7a0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.705+0000 7f4123331640 1 --2- 172.21.15.88:0/382090591 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f411c0b3ad0 0x7f411c0c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.705+0000 7f4123331640 1 --2- 172.21.15.88:0/382090591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f411c15b480 0x7f411c0b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.705+0000 7f4123331640 1 --2- 172.21.15.88:0/382090591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f411c072530 0x7f411c156760 unknown :-1 s=CLOSED pgs=2397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.705+0000 7f4123331640 1 -- 172.21.15.88:0/382090591 >> 172.21.15.88:0/382090591 conn(0x7f411c06d0f0 msgr2=0x7f411c06d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:06.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.705+0000 7f4123331640 1 -- 172.21.15.88:0/382090591 shutdown_connections 2024-09-16T01:45:06.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.705+0000 7f4123331640 1 -- 172.21.15.88:0/382090591 wait complete. 2024-09-16T01:45:06.717 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-16T01:45:06.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.814+0000 7f4060423640 1 Processor -- start 2024-09-16T01:45:06.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.814+0000 7f4060423640 1 -- start start 2024-09-16T01:45:06.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.815+0000 7f4060423640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f405815c710 0x7f4058072050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:06.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.815+0000 7f4060423640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4058072a30 con 0x7f4058072590 2024-09-16T01:45:06.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.815+0000 7f4060423640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4058072d10 con 0x7f405806bc10 2024-09-16T01:45:06.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.815+0000 7f4060423640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f4058072ff0 con 0x7f405815c710 2024-09-16T01:45:06.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.815+0000 7f405d9be640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f405815c710 0x7f4058072050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:06.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.815+0000 7f405e9c0640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4058072590 0x7f405815cdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35476/0 (socket says 172.21.15.88:35476) 2024-09-16T01:45:06.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.815+0000 7f405e9c0640 1 -- 172.21.15.88:0/2403856227 learned_addr learned my addr 172.21.15.88:0/2403856227 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:06.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.815+0000 7f405d9be640 1 -- 172.21.15.88:0/2403856227 >> v1:172.21.15.154:6789/0 conn(0x7f405806bc10 legacy=0x7f405806bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:06.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.815+0000 7f405d9be640 1 -- 172.21.15.88:0/2403856227 >> v1:172.21.15.88:6789/0 conn(0x7f4058072590 legacy=0x7f405815cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:06.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.815+0000 7f405d9be640 1 -- 172.21.15.88:0/2403856227 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40580732d0 con 0x7f405815c710 2024-09-16T01:45:06.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.815+0000 7f405d1bd640 1 -- 172.21.15.88:0/2403856227 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1716162871 0 0) 0x7f4058072a30 con 0x7f4058072590 2024-09-16T01:45:06.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.816+0000 7f405d9be640 1 --2- 172.21.15.88:0/2403856227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f405815c710 0x7f4058072050 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f40480097a0 tx=0x7f404802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b95ff33bb00d31 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:06.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.816+0000 7f405d1bd640 1 -- 172.21.15.88:0/2403856227 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4048037070 con 0x7f405815c710 2024-09-16T01:45:06.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.816+0000 7f405d1bd640 1 -- 172.21.15.88:0/2403856227 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4048032440 con 0x7f405815c710 2024-09-16T01:45:06.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.816+0000 7f405d1bd640 1 -- 172.21.15.88:0/2403856227 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4048032810 con 0x7f405815c710 2024-09-16T01:45:06.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.817+0000 7f4060423640 1 -- 172.21.15.88:0/2403856227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f405815c710 msgr2=0x7f4058072050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:06.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.817+0000 7f4060423640 1 --2- 172.21.15.88:0/2403856227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f405815c710 0x7f4058072050 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f40480097a0 tx=0x7f404802a1e0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.817+0000 7f4060423640 1 -- 172.21.15.88:0/2403856227 shutdown_connections 2024-09-16T01:45:06.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.817+0000 7f4060423640 1 --2- 172.21.15.88:0/2403856227 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f405815c710 0x7f4058072050 unknown :-1 s=CLOSED pgs=2403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.817+0000 7f4060423640 1 -- 172.21.15.88:0/2403856227 >> 172.21.15.88:0/2403856227 conn(0x7f40580d14d0 msgr2=0x7f40580d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:06.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.817+0000 7f4060423640 1 -- 172.21.15.88:0/2403856227 shutdown_connections 2024-09-16T01:45:06.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.817+0000 7f4060423640 1 -- 172.21.15.88:0/2403856227 wait complete. 2024-09-16T01:45:06.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.818+0000 7f4060423640 1 Processor -- start 2024-09-16T01:45:06.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.818+0000 7f4060423640 1 -- start start 2024-09-16T01:45:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.819+0000 7f4060423640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f405806bc10 0x7f40580cb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.819+0000 7f4060423640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4058072590 0x7f4058152420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.819+0000 7f4060423640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f405815c710 0x7f4058152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.819+0000 7f4060423640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f405806f9d0 con 0x7f405815c710 2024-09-16T01:45:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.819+0000 7f4060423640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f405806f850 con 0x7f4058072590 2024-09-16T01:45:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.819+0000 7f4060423640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f405815d4c0 con 0x7f405806bc10 2024-09-16T01:45:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.819+0000 7f405e9c0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f405815c710 0x7f4058152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.819+0000 7f405e9c0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f405815c710 0x7f4058152960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54894/0 (socket says 172.21.15.88:54894) 2024-09-16T01:45:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.819+0000 7f405e9c0640 1 -- 172.21.15.88:0/2143676001 learned_addr learned my addr 172.21.15.88:0/2143676001 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.819+0000 7f405d9be640 1 --2- 172.21.15.88:0/2143676001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4058072590 0x7f4058152420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.819+0000 7f405e9c0640 1 -- 172.21.15.88:0/2143676001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f405806bc10 msgr2=0x7f40580cb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.819+0000 7f405e1bf640 1 --2- 172.21.15.88:0/2143676001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f405806bc10 0x7f40580cb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.819+0000 7f405e9c0640 1 --2- 172.21.15.88:0/2143676001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f405806bc10 0x7f40580cb2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.819+0000 7f405e9c0640 1 -- 172.21.15.88:0/2143676001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4058072590 msgr2=0x7f4058152420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:06.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.819+0000 7f405e9c0640 1 --2- 172.21.15.88:0/2143676001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4058072590 0x7f4058152420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:06.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.819+0000 7f405e9c0640 1 -- 172.21.15.88:0/2143676001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40581532f0 con 0x7f405815c710 2024-09-16T01:45:06.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.819+0000 7f405d9be640 1 --2- 172.21.15.88:0/2143676001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4058072590 0x7f4058152420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:45:06.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.819+0000 7f405e1bf640 1 --2- 172.21.15.88:0/2143676001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f405806bc10 0x7f40580cb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:45:06.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.819+0000 7f405e9c0640 1 --2- 172.21.15.88:0/2143676001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f405815c710 0x7f4058152960 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7f405000c780 tx=0x7f405000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:06.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.819+0000 7f404effd640 1 -- 172.21.15.88:0/2143676001 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4050018070 con 0x7f405815c710 2024-09-16T01:45:06.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.819+0000 7f404effd640 1 -- 172.21.15.88:0/2143676001 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f405000a640 con 0x7f405815c710 2024-09-16T01:45:06.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.819+0000 7f404effd640 1 -- 172.21.15.88:0/2143676001 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405000aab0 con 0x7f405815c710 2024-09-16T01:45:06.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.819+0000 7f4060423640 1 -- 172.21.15.88:0/2143676001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40580ce9b0 con 0x7f405815c710 2024-09-16T01:45:06.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.819+0000 7f4060423640 1 -- 172.21.15.88:0/2143676001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40580cef60 con 0x7f405815c710 2024-09-16T01:45:06.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.820+0000 7f4060423640 1 -- 172.21.15.88:0/2143676001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4058072050 con 0x7f405815c710 2024-09-16T01:45:06.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.821+0000 7f404effd640 1 -- 172.21.15.88:0/2143676001 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f405000ac50 con 0x7f405815c710 2024-09-16T01:45:06.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.821+0000 7f404effd640 1 --2- 172.21.15.88:0/2143676001 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f403c06c2b0 0x7f403c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:06.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.822+0000 7f405e1bf640 1 --2- 172.21.15.88:0/2143676001 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f403c06c2b0 0x7f403c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:06.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.822+0000 7f404effd640 1 -- 172.21.15.88:0/2143676001 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5795+0+0 (secure 0 0 0) 0x7f405008ab20 con 0x7f405815c710 2024-09-16T01:45:06.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.822+0000 7f405e1bf640 1 --2- 172.21.15.88:0/2143676001 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f403c06c2b0 0x7f403c06e770 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f4054003eb0 tx=0x7f4054009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:06.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:06.826+0000 7f404effd640 1 -- 172.21.15.88:0/2143676001 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4050059230 con 0x7f405815c710 2024-09-16T01:45:07.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.023+0000 7f4060423640 1 -- 172.21.15.88:0/2143676001 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f40580b3600 con 0x7f405815c710 2024-09-16T01:45:07.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.023+0000 7f404effd640 1 -- 172.21.15.88:0/2143676001 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v332) ==== 53+0+73 (secure 0 0 0) 0x7f4050058b30 con 0x7f405815c710 2024-09-16T01:45:07.025 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-16T01:45:07.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.025+0000 7f4060423640 1 -- 172.21.15.88:0/2143676001 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f403c06c2b0 msgr2=0x7f403c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:07.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.025+0000 7f4060423640 1 --2- 172.21.15.88:0/2143676001 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f403c06c2b0 0x7f403c06e770 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f4054003eb0 tx=0x7f4054009040 comp rx=0 tx=0).stop 2024-09-16T01:45:07.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.025+0000 7f4060423640 1 -- 172.21.15.88:0/2143676001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f405815c710 msgr2=0x7f4058152960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:07.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.025+0000 7f4060423640 1 --2- 172.21.15.88:0/2143676001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f405815c710 0x7f4058152960 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7f405000c780 tx=0x7f405000cc40 comp rx=0 tx=0).stop 2024-09-16T01:45:07.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.025+0000 7f4060423640 1 -- 172.21.15.88:0/2143676001 shutdown_connections 2024-09-16T01:45:07.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.025+0000 7f4060423640 1 --2- 172.21.15.88:0/2143676001 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f403c06c2b0 0x7f403c06e770 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:07.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.025+0000 7f4060423640 1 --2- 172.21.15.88:0/2143676001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f405815c710 0x7f4058152960 unknown :-1 s=CLOSED pgs=2404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:07.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.025+0000 7f4060423640 1 --2- 172.21.15.88:0/2143676001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4058072590 0x7f4058152420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:07.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.025+0000 7f4060423640 1 --2- 172.21.15.88:0/2143676001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f405806bc10 0x7f40580cb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:07.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.025+0000 7f4060423640 1 -- 172.21.15.88:0/2143676001 >> 172.21.15.88:0/2143676001 conn(0x7f40580d14d0 msgr2=0x7f40580b4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:07.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.025+0000 7f4060423640 1 -- 172.21.15.88:0/2143676001 shutdown_connections 2024-09-16T01:45:07.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.025+0000 7f4060423640 1 -- 172.21.15.88:0/2143676001 wait complete. 2024-09-16T01:45:07.037 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-16T01:45:07.038 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-16T01:45:07.038 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2024-09-16T01:45:07.038 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-16T01:45:07.038 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-16T01:45:07.038 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-16T01:45:07.038 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-16T01:45:07.038 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-16T01:45:07.038 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:45:07.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:45:07.038 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2024-09-16T01:45:07.054 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-16T01:45:07.054 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-16T01:45:07.054 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2024-09-16T01:45:07.054 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:45:07.054 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2024-09-16T01:45:07.054 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:45:07.054 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:45:07.054 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:45:07.055 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f0 2024-09-16T01:45:07.055 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:45:07.055 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2024-09-16T01:45:07.055 INFO:teuthology.orchestra.run.smithi088.stdout: altname enp7s0f1 2024-09-16T01:45:07.055 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-16T01:45:07.055 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2024-09-16T01:45:07.055 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-16T01:45:07.055 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2024-09-16T01:45:07.055 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-16T01:45:07.055 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 42117sec preferred_lft 42117sec 2024-09-16T01:45:07.055 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::ec4:7aff:fe8f:d42d/64 scope link noprefixroute 2024-09-16T01:45:07.055 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:45:07.055 INFO:teuthology.orchestra.run.smithi088.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-16T01:45:07.056 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff 2024-09-16T01:45:07.056 INFO:teuthology.orchestra.run.smithi088.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-16T01:45:07.056 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:45:07.056 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::e486:6eff:fe81:a1c/64 scope link 2024-09-16T01:45:07.056 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:45:07.056 INFO:teuthology.orchestra.run.smithi088.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:45:07.056 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 12:6f:29:be:95:48 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:45:07.056 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::106f:29ff:febe:9548/64 scope link 2024-09-16T01:45:07.056 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:45:07.056 INFO:teuthology.orchestra.run.smithi088.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-16T01:45:07.056 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0e:63:b4:b3:ad:9e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T01:45:07.056 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::c63:b4ff:feb3:ad9e/64 scope link 2024-09-16T01:45:07.057 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2024-09-16T01:45:07.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:45:07.057 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-16T01:45:07.117 INFO:teuthology.orchestra.run.smithi088.stdout:1 2024-09-16T01:45:07.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:45:07.118 DEBUG:teuthology.orchestra.run.smithi088:> route 2024-09-16T01:45:07.174 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2024-09-16T01:45:07.175 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-16T01:45:07.175 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-16T01:45:07.175 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-16T01:45:07.175 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-16T01:45:07.175 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-16T01:45:07.175 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2024-09-16T01:45:07.175 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-16T01:45:07.175 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-16T01:45:07.175 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-16T01:45:07.175 DEBUG:teuthology.orchestra.run.smithi088:> ') 2024-09-16T01:45:07.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-16T01:45:07.274 INFO:tasks.ceph.mds.a.smithi088.stderr:2024-09-16T01:45:07.273+0000 7f7534874640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:45:07.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:45:07.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:45:07.301 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:45:07.316 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T01:45:07.316 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T01:45:07.317 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:45:07.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:45:07.317 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2024-09-16T01:45:07.382 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:45:07.382 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-16T01:45:07.438 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-16T01:45:07.438 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-16T01:45:07.439 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:45:07.494 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-16T01:45:07.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:45:07.494 DEBUG:teuthology.orchestra.run.smithi088:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:45:07.578 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:45:07.578 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:45:07.578 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:45:07.586 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:45:07.586 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:45:07.587 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:45:07.587 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:45:07.587 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:45:07.591 INFO:teuthology.orchestra.run.smithi088.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-16T01:45:07.592 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-16T01:45:07.592 INFO:teuthology.orchestra.run.smithi088.stdout:invalid new device string format 2024-09-16T01:45:07.592 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:3300,172.21.15.154:3300,172.21.15.154:3301" 2024-09-16T01:45:07.592 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-16T01:45:07.592 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: resolved to: "172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790" 2024-09-16T01:45:07.593 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.88:6789,172.21.15.154:6789,172.21.15.154:6790:/ 2024-09-16T01:45:07.593 INFO:teuthology.orchestra.run.smithi088.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=5ace2c91-46f9-482b-8a89-731fbdc171bf" will pass to kernel 2024-09-16T01:45:07.596 INFO:teuthology.orchestra.run.smithi088.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-16T01:45:07.596 INFO:teuthology.orchestra.run.smithi088.stdout: You just mounted a file system that supports labels which does not 2024-09-16T01:45:07.596 INFO:teuthology.orchestra.run.smithi088.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-16T01:45:07.597 INFO:teuthology.orchestra.run.smithi088.stdout: applications will generate AVC messages and not be allowed access to 2024-09-16T01:45:07.597 INFO:teuthology.orchestra.run.smithi088.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-16T01:45:07.597 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-16T01:45:07.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:45:07.597 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-16T01:45:07.670 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-16T01:45:07.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-16T01:45:07.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-16T01:45:07.772 INFO:teuthology.orchestra.run.smithi088.stdout:client10788 2024-09-16T01:45:07.773 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:45:07.773 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:45:07.789 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-09-16T01:45:07.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.874+0000 7f29b711cf40 1 Processor -- start 2024-09-16T01:45:07.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.874+0000 7f29b711cf40 1 -- start start 2024-09-16T01:45:07.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.875+0000 7f29b711cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfc97a4670 0x55bfc982db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:07.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.875+0000 7f29b711cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfc98393c0 0x55bfc983a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:07.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.875+0000 7f29b711cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bfc983b710 0x55bfc983a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:07.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.875+0000 7f29b711cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55bfc983cc50 con 0x55bfc97a4670 2024-09-16T01:45:07.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.875+0000 7f29b711cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bfc97a3b30 con 0x55bfc98393c0 2024-09-16T01:45:07.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.875+0000 7f29b711cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55bfc983cef0 con 0x55bfc983b710 2024-09-16T01:45:07.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.875+0000 7f29b41ad640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfc97a4670 0x55bfc982db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:07.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.875+0000 7f29b41ad640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfc97a4670 0x55bfc982db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54898/0 (socket says 172.21.15.88:54898) 2024-09-16T01:45:07.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.875+0000 7f29b41ad640 1 -- 172.21.15.88:0/234690413 learned_addr learned my addr 172.21.15.88:0/234690413 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:07.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.875+0000 7f29b39ac640 1 --2- 172.21.15.88:0/234690413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfc98393c0 0x55bfc983a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:07.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.875+0000 7f29b49ae640 1 --2- 172.21.15.88:0/234690413 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bfc983b710 0x55bfc983a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:07.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.875+0000 7f29b41ad640 1 -- 172.21.15.88:0/234690413 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bfc983b710 msgr2=0x55bfc983a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:07.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.875+0000 7f29b41ad640 1 --2- 172.21.15.88:0/234690413 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bfc983b710 0x55bfc983a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:07.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.875+0000 7f29b41ad640 1 -- 172.21.15.88:0/234690413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfc98393c0 msgr2=0x55bfc983a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:07.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.875+0000 7f29b41ad640 1 --2- 172.21.15.88:0/234690413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfc98393c0 0x55bfc983a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:07.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.875+0000 7f29b41ad640 1 -- 172.21.15.88:0/234690413 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfc968ecb0 con 0x55bfc97a4670 2024-09-16T01:45:07.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.875+0000 7f29b39ac640 1 --2- 172.21.15.88:0/234690413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfc98393c0 0x55bfc983a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:45:07.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.876+0000 7f29b41ad640 1 --2- 172.21.15.88:0/234690413 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfc97a4670 0x55bfc982db80 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7f29a40095d0 tx=0x7f29a4009a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58b82167de56558 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:07.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.876+0000 7f29b29aa640 1 -- 172.21.15.88:0/234690413 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a4017070 con 0x55bfc97a4670 2024-09-16T01:45:07.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.876+0000 7f29b29aa640 1 -- 172.21.15.88:0/234690413 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29a4006680 con 0x55bfc97a4670 2024-09-16T01:45:07.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.876+0000 7f29b29aa640 1 -- 172.21.15.88:0/234690413 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a4006a30 con 0x55bfc97a4670 2024-09-16T01:45:07.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.876+0000 7f29b711cf40 1 -- 172.21.15.88:0/234690413 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfc97a4670 msgr2=0x55bfc982db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:07.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.876+0000 7f29b711cf40 1 --2- 172.21.15.88:0/234690413 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfc97a4670 0x55bfc982db80 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7f29a40095d0 tx=0x7f29a4009a90 comp rx=0 tx=0).stop 2024-09-16T01:45:07.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.876+0000 7f29b711cf40 1 -- 172.21.15.88:0/234690413 shutdown_connections 2024-09-16T01:45:07.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.876+0000 7f29b711cf40 1 --2- 172.21.15.88:0/234690413 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bfc983b710 0x55bfc983a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:07.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.876+0000 7f29b711cf40 1 --2- 172.21.15.88:0/234690413 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfc98393c0 0x55bfc983a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:07.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.876+0000 7f29b711cf40 1 --2- 172.21.15.88:0/234690413 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfc97a4670 0x55bfc982db80 unknown :-1 s=CLOSED pgs=2407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:07.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.876+0000 7f29b711cf40 1 -- 172.21.15.88:0/234690413 >> 172.21.15.88:0/234690413 conn(0x55bfc978e680 msgr2=0x55bfc98364c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:07.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.876+0000 7f29b711cf40 1 -- 172.21.15.88:0/234690413 shutdown_connections 2024-09-16T01:45:07.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.876+0000 7f29b711cf40 1 -- 172.21.15.88:0/234690413 wait complete. 2024-09-16T01:45:07.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.877+0000 7f29b711cf40 1 Processor -- start 2024-09-16T01:45:07.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.877+0000 7f29b711cf40 1 -- start start 2024-09-16T01:45:07.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.878+0000 7f29b711cf40 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bfc97a4670 0x55bfc98cbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:07.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.878+0000 7f29b711cf40 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfc98393c0 0x55bfc9839fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:07.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.878+0000 7f29b711cf40 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfc983b710 0x55bfc97aa690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:07.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.878+0000 7f29b711cf40 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x55bfc9837450 con 0x55bfc983b710 2024-09-16T01:45:07.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.878+0000 7f29b711cf40 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x55bfc9837200 con 0x55bfc98393c0 2024-09-16T01:45:07.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.878+0000 7f29b711cf40 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55bfc9837730 con 0x55bfc97a4670 2024-09-16T01:45:07.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.878+0000 7f29b49ae640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfc983b710 0x55bfc97aa690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:07.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.878+0000 7f29b49ae640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfc983b710 0x55bfc97aa690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54908/0 (socket says 172.21.15.88:54908) 2024-09-16T01:45:07.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.878+0000 7f29b49ae640 1 -- 172.21.15.88:0/2809083044 learned_addr learned my addr 172.21.15.88:0/2809083044 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:07.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.878+0000 7f29b39ac640 1 --2- 172.21.15.88:0/2809083044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfc98393c0 0x55bfc9839fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:07.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.878+0000 7f29b41ad640 1 --2- 172.21.15.88:0/2809083044 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bfc97a4670 0x55bfc98cbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:07.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.878+0000 7f29b49ae640 1 -- 172.21.15.88:0/2809083044 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bfc97a4670 msgr2=0x55bfc98cbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:07.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.878+0000 7f29b49ae640 1 --2- 172.21.15.88:0/2809083044 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x55bfc97a4670 0x55bfc98cbfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:07.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.878+0000 7f29b49ae640 1 -- 172.21.15.88:0/2809083044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfc98393c0 msgr2=0x55bfc9839fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:07.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.878+0000 7f29b49ae640 1 --2- 172.21.15.88:0/2809083044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bfc98393c0 0x55bfc9839fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:07.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.878+0000 7f29b49ae640 1 -- 172.21.15.88:0/2809083044 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfc9794400 con 0x55bfc983b710 2024-09-16T01:45:07.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.878+0000 7f29b49ae640 1 --2- 172.21.15.88:0/2809083044 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfc983b710 0x55bfc97aa690 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f29ac025b20 tx=0x7f29ac003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:07.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.878+0000 7f29b19a8640 1 -- 172.21.15.88:0/2809083044 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ac066020 con 0x55bfc983b710 2024-09-16T01:45:07.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.878+0000 7f29b19a8640 1 -- 172.21.15.88:0/2809083044 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29ac05c040 con 0x55bfc983b710 2024-09-16T01:45:07.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.878+0000 7f29b19a8640 1 -- 172.21.15.88:0/2809083044 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ac05b040 con 0x55bfc983b710 2024-09-16T01:45:07.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.878+0000 7f29b711cf40 1 -- 172.21.15.88:0/2809083044 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55bfc97a8330 con 0x55bfc983b710 2024-09-16T01:45:07.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.878+0000 7f29b711cf40 1 -- 172.21.15.88:0/2809083044 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfc97a88f0 con 0x55bfc983b710 2024-09-16T01:45:07.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.880+0000 7f29b19a8640 1 -- 172.21.15.88:0/2809083044 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f29ac069020 con 0x55bfc983b710 2024-09-16T01:45:07.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.880+0000 7f29b19a8640 1 --2- 172.21.15.88:0/2809083044 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f29a006ac90 0x7f29a006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:07.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.880+0000 7f29b19a8640 1 -- 172.21.15.88:0/2809083044 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5795+0+0 (secure 0 0 0) 0x7f29ac0ce630 con 0x55bfc983b710 2024-09-16T01:45:07.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.880+0000 7f29b41ad640 1 --2- 172.21.15.88:0/2809083044 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f29a006ac90 0x7f29a006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:07.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.880+0000 7f29b711cf40 1 --2- 172.21.15.88:0/2809083044 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55bfc983a4e0 0x55bfc97a9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:07.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.880+0000 7f29b711cf40 1 -- 172.21.15.88:0/2809083044 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- osd_op(unknown.0.0:1 15.7 15:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e139) -- 0x55bfc97aabd0 con 0x55bfc983a4e0 2024-09-16T01:45:07.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.880+0000 7f29b39ac640 1 --2- 172.21.15.88:0/2809083044 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55bfc983a4e0 0x55bfc97a9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:07.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.881+0000 7f29b41ad640 1 --2- 172.21.15.88:0/2809083044 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f29a006ac90 0x7f29a006d150 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f29a4000c00 tx=0x7f29a400f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:07.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.881+0000 7f29b39ac640 1 --2- 172.21.15.88:0/2809083044 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55bfc983a4e0 0x55bfc97a9320 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:07.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.881+0000 7f29b39ac640 1 -- 172.21.15.88:0/2809083044 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (91) out=91b] v0'0 uv6 ondisk = 0) ==== 164+0+91 (crc 0 0 0) 0x7f29a8007920 con 0x55bfc983a4e0 2024-09-16T01:45:07.886 INFO:teuthology.orchestra.run.smithi088.stdout:Ufile1dir1 2024-09-16T01:45:07.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.881+0000 7f29b711cf40 1 -- 172.21.15.88:0/2809083044 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55bfc983a4e0 msgr2=0x55bfc97a9320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:07.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.881+0000 7f29b711cf40 1 --2- 172.21.15.88:0/2809083044 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x55bfc983a4e0 0x55bfc97a9320 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:07.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.881+0000 7f29b711cf40 1 -- 172.21.15.88:0/2809083044 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f29a006ac90 msgr2=0x7f29a006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:07.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.881+0000 7f29b711cf40 1 --2- 172.21.15.88:0/2809083044 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f29a006ac90 0x7f29a006d150 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f29a4000c00 tx=0x7f29a400f000 comp rx=0 tx=0).stop 2024-09-16T01:45:07.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.881+0000 7f29b711cf40 1 -- 172.21.15.88:0/2809083044 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfc983b710 msgr2=0x55bfc97aa690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:07.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.881+0000 7f29b711cf40 1 --2- 172.21.15.88:0/2809083044 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55bfc983b710 0x55bfc97aa690 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f29ac025b20 tx=0x7f29ac003330 comp rx=0 tx=0).stop 2024-09-16T01:45:07.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.881+0000 7f29b49ae640 1 -- 172.21.15.88:0/2809083044 reap_dead start 2024-09-16T01:45:07.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.881+0000 7f29b711cf40 1 -- 172.21.15.88:0/2809083044 shutdown_connections 2024-09-16T01:45:07.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.881+0000 7f29b711cf40 1 -- 172.21.15.88:0/2809083044 >> 172.21.15.88:0/2809083044 conn(0x55bfc978e680 msgr2=0x55bfc98363a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:07.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.881+0000 7f29b711cf40 1 -- 172.21.15.88:0/2809083044 shutdown_connections 2024-09-16T01:45:07.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:07.881+0000 7f29b711cf40 1 -- 172.21.15.88:0/2809083044 wait complete. 2024-09-16T01:45:07.889 DEBUG:teuthology.orchestra.run.smithi088:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-09-16T01:45:08.023 INFO:teuthology.orchestra.run.smithi088.stderr:error: stray data at end of buffer, offset 91 2024-09-16T01:45:08.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:45:08.033 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-16T01:45:08.033 DEBUG:teuthology.orchestra.run.smithi088:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T01:45:08.075 DEBUG:teuthology.orchestra.run.smithi154:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-16T01:45:08.122 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-16T01:45:08.122 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T01:45:08.122 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:45:08.122 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:45:08.138 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:45:08.138 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:45:08.195 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-16T01:45:08.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.334+0000 7f3e20d9f640 1 Processor -- start 2024-09-16T01:45:08.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.334+0000 7f3e20d9f640 1 -- start start 2024-09-16T01:45:08.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.335+0000 7f3e20d9f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e1c15adf0 0x7f3e1c0b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:08.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.335+0000 7f3e20d9f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3e1c15b1d0 0x7f3e1c0b1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:08.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.335+0000 7f3e20d9f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e1c0b44f0 con 0x7f3e1c0b22b0 2024-09-16T01:45:08.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.335+0000 7f3e20d9f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3e1c0b47d0 con 0x7f3e1c15b1d0 2024-09-16T01:45:08.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.335+0000 7f3e20d9f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3e1c0b4ab0 con 0x7f3e1c15adf0 2024-09-16T01:45:08.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.335+0000 7f3e1b7fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e1c15adf0 0x7f3e1c0b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:08.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.335+0000 7f3e1affd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3e1c15b1d0 0x7f3e1c0b1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:08.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.335+0000 7f3e1b7fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e1c15adf0 0x7f3e1c0b3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:51302/0 (socket says 172.21.15.88:51302) 2024-09-16T01:45:08.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.335+0000 7f3e1b7fe640 1 -- 172.21.15.88:0/4253253861 learned_addr learned my addr 172.21.15.88:0/4253253861 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:08.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.336+0000 7f3e1affd640 1 -- 172.21.15.88:0/4253253861 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e1c15adf0 msgr2=0x7f3e1c0b3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:08.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.336+0000 7f3e1affd640 1 --2- 172.21.15.88:0/4253253861 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e1c15adf0 0x7f3e1c0b3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:08.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.336+0000 7f3e1affd640 1 -- 172.21.15.88:0/4253253861 >> v1:172.21.15.154:6789/0 conn(0x7f3e1c0b22b0 legacy=0x7f3e1c15ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:08.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.336+0000 7f3e1affd640 1 -- 172.21.15.88:0/4253253861 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1c0b4d90 con 0x7f3e1c15b1d0 2024-09-16T01:45:08.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.336+0000 7f3e1b7fe640 1 --2- 172.21.15.88:0/4253253861 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e1c15adf0 0x7f3e1c0b3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:45:08.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.337+0000 7f3e1affd640 1 --2- 172.21.15.88:0/4253253861 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e1c15b1d0 0x7f3e1c0b1d70 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f3e100097a0 tx=0x7f3e1002a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9354e09b05baff server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:08.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.337+0000 7f3e1a7fc640 1 -- 172.21.15.88:0/4253253861 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e10037070 con 0x7f3e1c15b1d0 2024-09-16T01:45:08.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.337+0000 7f3e1a7fc640 1 -- 172.21.15.88:0/4253253861 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e10032440 con 0x7f3e1c15b1d0 2024-09-16T01:45:08.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.337+0000 7f3e1a7fc640 1 -- 172.21.15.88:0/4253253861 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e10032810 con 0x7f3e1c15b1d0 2024-09-16T01:45:08.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.337+0000 7f3e20d9f640 1 -- 172.21.15.88:0/4253253861 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e1c15b1d0 msgr2=0x7f3e1c0b1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:08.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.337+0000 7f3e20d9f640 1 --2- 172.21.15.88:0/4253253861 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e1c15b1d0 0x7f3e1c0b1d70 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f3e100097a0 tx=0x7f3e1002a210 comp rx=0 tx=0).stop 2024-09-16T01:45:08.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.337+0000 7f3e20d9f640 1 -- 172.21.15.88:0/4253253861 shutdown_connections 2024-09-16T01:45:08.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.337+0000 7f3e20d9f640 1 --2- 172.21.15.88:0/4253253861 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e1c15b1d0 0x7f3e1c0b1d70 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:08.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.337+0000 7f3e20d9f640 1 --2- 172.21.15.88:0/4253253861 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e1c15adf0 0x7f3e1c0b3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:08.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.337+0000 7f3e20d9f640 1 -- 172.21.15.88:0/4253253861 >> 172.21.15.88:0/4253253861 conn(0x7f3e1c062a20 msgr2=0x7f3e1c06bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:08.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.337+0000 7f3e20d9f640 1 -- 172.21.15.88:0/4253253861 shutdown_connections 2024-09-16T01:45:08.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.337+0000 7f3e20d9f640 1 -- 172.21.15.88:0/4253253861 wait complete. 2024-09-16T01:45:08.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.339+0000 7f3e20d9f640 1 Processor -- start 2024-09-16T01:45:08.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.339+0000 7f3e20d9f640 1 -- start start 2024-09-16T01:45:08.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.339+0000 7f3e20d9f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e1c0b22b0 0x7f3e1c15b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:08.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.339+0000 7f3e20d9f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e1c15adf0 0x7f3e1c14c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:08.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.339+0000 7f3e20d9f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e1c15b1d0 0x7f3e1c06d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:08.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.339+0000 7f3e20d9f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c0b4030 con 0x7f3e1c15adf0 2024-09-16T01:45:08.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.339+0000 7f3e20d9f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c0b5570 con 0x7f3e1c15b1d0 2024-09-16T01:45:08.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.339+0000 7f3e20d9f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3e1c071f10 con 0x7f3e1c0b22b0 2024-09-16T01:45:08.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.340+0000 7f3e1affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e1c15adf0 0x7f3e1c14c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:08.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.340+0000 7f3e1affd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e1c15adf0 0x7f3e1c14c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54918/0 (socket says 172.21.15.88:54918) 2024-09-16T01:45:08.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.340+0000 7f3e1affd640 1 -- 172.21.15.88:0/28640117 learned_addr learned my addr 172.21.15.88:0/28640117 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:08.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.340+0000 7f3e1b7fe640 1 --2- 172.21.15.88:0/28640117 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e1c0b22b0 0x7f3e1c15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:08.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.340+0000 7f3e1bfff640 1 --2- 172.21.15.88:0/28640117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e1c15b1d0 0x7f3e1c06d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:08.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.340+0000 7f3e1affd640 1 -- 172.21.15.88:0/28640117 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e1c0b22b0 msgr2=0x7f3e1c15b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:08.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.340+0000 7f3e1affd640 1 --2- 172.21.15.88:0/28640117 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e1c0b22b0 0x7f3e1c15b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:08.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.340+0000 7f3e1affd640 1 -- 172.21.15.88:0/28640117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e1c15b1d0 msgr2=0x7f3e1c06d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:08.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.340+0000 7f3e1affd640 1 --2- 172.21.15.88:0/28640117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e1c15b1d0 0x7f3e1c06d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:08.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.340+0000 7f3e1affd640 1 -- 172.21.15.88:0/28640117 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1c15de90 con 0x7f3e1c15adf0 2024-09-16T01:45:08.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.340+0000 7f3e1b7fe640 1 --2- 172.21.15.88:0/28640117 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e1c0b22b0 0x7f3e1c15b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:45:08.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.340+0000 7f3e1bfff640 1 --2- 172.21.15.88:0/28640117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e1c15b1d0 0x7f3e1c06d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:45:08.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.340+0000 7f3e1affd640 1 --2- 172.21.15.88:0/28640117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e1c15adf0 0x7f3e1c14c9c0 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f3e10000c00 tx=0x7f3e100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:08.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.340+0000 7f3e03fff640 1 -- 172.21.15.88:0/28640117 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e10037040 con 0x7f3e1c15adf0 2024-09-16T01:45:08.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.340+0000 7f3e03fff640 1 -- 172.21.15.88:0/28640117 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e10042070 con 0x7f3e1c15adf0 2024-09-16T01:45:08.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.340+0000 7f3e20d9f640 1 -- 172.21.15.88:0/28640117 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e1c15bd20 con 0x7f3e1c15adf0 2024-09-16T01:45:08.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.340+0000 7f3e03fff640 1 -- 172.21.15.88:0/28640117 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e10030920 con 0x7f3e1c15adf0 2024-09-16T01:45:08.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.340+0000 7f3e20d9f640 1 -- 172.21.15.88:0/28640117 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e1c15c2e0 con 0x7f3e1c15adf0 2024-09-16T01:45:08.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.341+0000 7f3e20d9f640 1 -- 172.21.15.88:0/28640117 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e1c0b3830 con 0x7f3e1c15adf0 2024-09-16T01:45:08.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.342+0000 7f3e03fff640 1 -- 172.21.15.88:0/28640117 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3e10030ac0 con 0x7f3e1c15adf0 2024-09-16T01:45:08.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.342+0000 7f3e03fff640 1 --2- 172.21.15.88:0/28640117 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3e0806c2b0 0x7f3e0806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:08.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.342+0000 7f3e03fff640 1 -- 172.21.15.88:0/28640117 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5795+0+0 (secure 0 0 0) 0x7f3e100abda0 con 0x7f3e1c15adf0 2024-09-16T01:45:08.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.342+0000 7f3e1b7fe640 1 --2- 172.21.15.88:0/28640117 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3e0806c2b0 0x7f3e0806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:08.349 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.343+0000 7f3e1b7fe640 1 --2- 172.21.15.88:0/28640117 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3e0806c2b0 0x7f3e0806e770 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c009630 tx=0x7f3e0c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:08.350 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.347+0000 7f3e03fff640 1 -- 172.21.15.88:0/28640117 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3e1c0b3830 con 0x7f3e1c15adf0 2024-09-16T01:45:08.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.528+0000 7f3e20d9f640 1 -- 172.21.15.88:0/28640117 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3e1c0af5c0 con 0x7f3e1c15adf0 2024-09-16T01:45:08.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.528+0000 7f3e03fff640 1 -- 172.21.15.88:0/28640117 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v139) ==== 80+0+120 (secure 0 0 0) 0x7f3e10079e70 con 0x7f3e1c15adf0 2024-09-16T01:45:08.530 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6836/656828696 2024-09-17T01:45:01.261737+0000 2024-09-16T01:45:08.530 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6834/656828696 2024-09-17T01:45:01.261737+0000 2024-09-16T01:45:08.530 INFO:teuthology.orchestra.run.smithi088.stderr:listed 2 entries 2024-09-16T01:45:08.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.530+0000 7f3e20d9f640 1 -- 172.21.15.88:0/28640117 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3e0806c2b0 msgr2=0x7f3e0806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:08.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.530+0000 7f3e20d9f640 1 --2- 172.21.15.88:0/28640117 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3e0806c2b0 0x7f3e0806e770 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c009630 tx=0x7f3e0c008040 comp rx=0 tx=0).stop 2024-09-16T01:45:08.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.530+0000 7f3e20d9f640 1 -- 172.21.15.88:0/28640117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e1c15adf0 msgr2=0x7f3e1c14c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:08.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.530+0000 7f3e20d9f640 1 --2- 172.21.15.88:0/28640117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e1c15adf0 0x7f3e1c14c9c0 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f3e10000c00 tx=0x7f3e100046c0 comp rx=0 tx=0).stop 2024-09-16T01:45:08.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.530+0000 7f3e20d9f640 1 -- 172.21.15.88:0/28640117 shutdown_connections 2024-09-16T01:45:08.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.530+0000 7f3e20d9f640 1 --2- 172.21.15.88:0/28640117 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3e0806c2b0 0x7f3e0806e770 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:08.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.530+0000 7f3e20d9f640 1 --2- 172.21.15.88:0/28640117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e1c15b1d0 0x7f3e1c06d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:08.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.530+0000 7f3e20d9f640 1 --2- 172.21.15.88:0/28640117 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e1c15adf0 0x7f3e1c14c9c0 unknown :-1 s=CLOSED pgs=2409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:08.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.530+0000 7f3e20d9f640 1 --2- 172.21.15.88:0/28640117 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e1c0b22b0 0x7f3e1c15b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:08.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.530+0000 7f3e20d9f640 1 -- 172.21.15.88:0/28640117 >> 172.21.15.88:0/28640117 conn(0x7f3e1c062a20 msgr2=0x7f3e1c06bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:08.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.530+0000 7f3e20d9f640 1 -- 172.21.15.88:0/28640117 shutdown_connections 2024-09-16T01:45:08.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.530+0000 7f3e20d9f640 1 -- 172.21.15.88:0/28640117 wait complete. 2024-09-16T01:45:08.542 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-16T01:45:08.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:45:08.543 DEBUG:teuthology.orchestra.run.smithi088:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-16T01:45:08.570 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:45:08.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:45:08.571 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-16T01:45:08.627 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T01:45:08.627 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:45:08.627 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:45:08.647 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056026,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1347 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T01:45:08.647 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T01:45:08.648 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:45:08.648 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:45:08.683 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:45:08.684 INFO:tasks.cephfs.mount:Terminating background process 2024-09-16T01:45:08.684 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:45:08.684 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:45:08.707 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056026,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1347 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T01:45:08.707 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:45:08.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.822+0000 7f984508d640 1 Processor -- start 2024-09-16T01:45:08.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.822+0000 7f984508d640 1 -- start start 2024-09-16T01:45:08.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f984508d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f98400b4d00 0x7f98400b5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:08.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f984508d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98400cd780 0x7f98400cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:08.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f984508d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98400ce4f0 con 0x7f98400b4920 2024-09-16T01:45:08.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f984508d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f98400cf100 con 0x7f98400b4d00 2024-09-16T01:45:08.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f984508d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f98400ce7d0 con 0x7f98400cd780 2024-09-16T01:45:08.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f983ed9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f98400b4920 0x7f984015b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35482/0 (socket says 172.21.15.88:35482) 2024-09-16T01:45:08.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f983ed9d640 1 -- 172.21.15.88:0/3705657019 learned_addr learned my addr 172.21.15.88:0/3705657019 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:08.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f983dd9b640 1 -- 172.21.15.88:0/3705657019 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2162816203 0 0) 0x7f98400ce4f0 con 0x7f98400b4920 2024-09-16T01:45:08.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f983dd9b640 1 -- 172.21.15.88:0/3705657019 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9828003520 con 0x7f98400b4920 2024-09-16T01:45:08.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f983f59e640 1 --2- 172.21.15.88:0/3705657019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98400cd780 0x7f98400cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:08.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f983e59c640 1 --2- 172.21.15.88:0/3705657019 >> v2:172.21.15.154:3300/0 conn(0x7f98400b4d00 0x7f98400b5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:08.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f983dd9b640 1 -- 172.21.15.88:0/3705657019 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 921701342 0 0) 0x7f9828003520 con 0x7f98400b4920 2024-09-16T01:45:08.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f983dd9b640 1 -- 172.21.15.88:0/3705657019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98400cd780 msgr2=0x7f98400cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:08.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f983dd9b640 1 --2- 172.21.15.88:0/3705657019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98400cd780 0x7f98400cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:08.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f983dd9b640 1 -- 172.21.15.88:0/3705657019 >> v2:172.21.15.154:3300/0 conn(0x7f98400b4d00 msgr2=0x7f98400b5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:08.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f983dd9b640 1 --2- 172.21.15.88:0/3705657019 >> v2:172.21.15.154:3300/0 conn(0x7f98400b4d00 0x7f98400b5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:08.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f983dd9b640 1 -- 172.21.15.88:0/3705657019 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98400ceab0 con 0x7f98400b4920 2024-09-16T01:45:08.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f983dd9b640 1 -- 172.21.15.88:0/3705657019 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f9830001670 con 0x7f98400b4920 2024-09-16T01:45:08.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f983dd9b640 1 -- 172.21.15.88:0/3705657019 >> v1:172.21.15.88:6789/0 conn(0x7f98400b4920 legacy=0x7f984015b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:08.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f983dd9b640 1 --2- 172.21.15.88:0/3705657019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98280039b0 0x7f9828003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:08.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f983dd9b640 1 --2- 172.21.15.88:0/3705657019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9828004800 0x7f9828004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:08.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f983dd9b640 1 --2- 172.21.15.88:0/3705657019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98400cd780 0x7f98280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:08.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f983dd9b640 1 -- 172.21.15.88:0/3705657019 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f98400cf100 con 0x7f9828004800 2024-09-16T01:45:08.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f983dd9b640 1 -- 172.21.15.88:0/3705657019 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f98400ce7d0 con 0x7f98280039b0 2024-09-16T01:45:08.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f983dd9b640 1 -- 172.21.15.88:0/3705657019 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f982800b9c0 con 0x7f98400cd780 2024-09-16T01:45:08.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.823+0000 7f983e59c640 1 --2- 172.21.15.88:0/3705657019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9828004800 0x7f9828004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:08.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.824+0000 7f983f59e640 1 --2- 172.21.15.88:0/3705657019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98400cd780 0x7f98280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:08.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.824+0000 7f983ed9d640 1 --2- 172.21.15.88:0/3705657019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98280039b0 0x7f9828003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:08.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.824+0000 7f983e59c640 1 -- 172.21.15.88:0/3705657019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98400cd780 msgr2=0x7f98280051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:08.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.824+0000 7f983e59c640 1 --2- 172.21.15.88:0/3705657019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98400cd780 0x7f98280051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:08.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.824+0000 7f983e59c640 1 -- 172.21.15.88:0/3705657019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98280039b0 msgr2=0x7f9828003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:08.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.824+0000 7f983e59c640 1 --2- 172.21.15.88:0/3705657019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98280039b0 0x7f9828003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:08.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.824+0000 7f983e59c640 1 -- 172.21.15.88:0/3705657019 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9828003520 con 0x7f9828004800 2024-09-16T01:45:08.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.824+0000 7f983e59c640 1 --2- 172.21.15.88:0/3705657019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9828004800 0x7f9828004c80 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f9834009ac0 tx=0x7f9828005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6f6570cd1c44e28 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:08.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.825+0000 7f983dd9b640 1 -- 172.21.15.88:0/3705657019 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9834036070 con 0x7f9828004800 2024-09-16T01:45:08.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.825+0000 7f983dd9b640 1 -- 172.21.15.88:0/3705657019 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9834031400 con 0x7f9828004800 2024-09-16T01:45:08.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.825+0000 7f983dd9b640 1 -- 172.21.15.88:0/3705657019 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9834009690 con 0x7f9828004800 2024-09-16T01:45:08.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.825+0000 7f984508d640 1 -- 172.21.15.88:0/3705657019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9828004800 msgr2=0x7f9828004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:08.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.825+0000 7f984508d640 1 --2- 172.21.15.88:0/3705657019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9828004800 0x7f9828004c80 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f9834009ac0 tx=0x7f9828005830 comp rx=0 tx=0).stop 2024-09-16T01:45:08.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.825+0000 7f983f59e640 1 -- 172.21.15.88:0/3705657019 reap_dead start 2024-09-16T01:45:08.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.825+0000 7f984508d640 1 -- 172.21.15.88:0/3705657019 shutdown_connections 2024-09-16T01:45:08.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.825+0000 7f984508d640 1 -- 172.21.15.88:0/3705657019 >> 172.21.15.88:0/3705657019 conn(0x7f984006bc10 msgr2=0x7f984006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:08.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.825+0000 7f984508d640 1 -- 172.21.15.88:0/3705657019 shutdown_connections 2024-09-16T01:45:08.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.825+0000 7f984508d640 1 -- 172.21.15.88:0/3705657019 wait complete. 2024-09-16T01:45:08.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.827+0000 7f984508d640 1 Processor -- start 2024-09-16T01:45:08.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.827+0000 7f984508d640 1 -- start start 2024-09-16T01:45:08.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.827+0000 7f984508d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9828004800 0x7f98401531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:08.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.827+0000 7f984508d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f984014f730 0x7f984014fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:08.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.827+0000 7f984508d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f984015d720 0x7f984015dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:08.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.827+0000 7f984508d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f98400cdbd0 con 0x7f984015d720 2024-09-16T01:45:08.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.827+0000 7f984508d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f98400b56b0 con 0x7f9828004800 2024-09-16T01:45:08.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.827+0000 7f984508d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f98400cdd50 con 0x7f984014f730 2024-09-16T01:45:08.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.827+0000 7f983f59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f984015d720 0x7f984015dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:08.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.827+0000 7f983f59e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f984015d720 0x7f984015dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54940/0 (socket says 172.21.15.88:54940) 2024-09-16T01:45:08.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.828+0000 7f983f59e640 1 -- 172.21.15.88:0/970021657 learned_addr learned my addr 172.21.15.88:0/970021657 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:08.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.828+0000 7f983ed9d640 1 --2- 172.21.15.88:0/970021657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9828004800 0x7f98401531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:08.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.828+0000 7f983e59c640 1 --2- 172.21.15.88:0/970021657 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f984014f730 0x7f984014fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:08.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.828+0000 7f983f59e640 1 -- 172.21.15.88:0/970021657 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f984014f730 msgr2=0x7f984014fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:08.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.828+0000 7f983f59e640 1 --2- 172.21.15.88:0/970021657 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f984014f730 0x7f984014fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:08.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.828+0000 7f983f59e640 1 -- 172.21.15.88:0/970021657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9828004800 msgr2=0x7f98401531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:08.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.828+0000 7f983f59e640 1 --2- 172.21.15.88:0/970021657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9828004800 0x7f98401531c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:08.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.828+0000 7f983f59e640 1 -- 172.21.15.88:0/970021657 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98401500f0 con 0x7f984015d720 2024-09-16T01:45:08.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.828+0000 7f983f59e640 1 --2- 172.21.15.88:0/970021657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f984015d720 0x7f984015dbd0 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7f9840070e70 tx=0x7f9838011f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:08.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.828+0000 7f98277fe640 1 -- 172.21.15.88:0/970021657 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f983800fd90 con 0x7f984015d720 2024-09-16T01:45:08.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.828+0000 7f98277fe640 1 -- 172.21.15.88:0/970021657 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9838015430 con 0x7f984015d720 2024-09-16T01:45:08.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.828+0000 7f98277fe640 1 -- 172.21.15.88:0/970021657 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9838015820 con 0x7f984015d720 2024-09-16T01:45:08.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.828+0000 7f984508d640 1 -- 172.21.15.88:0/970021657 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9840151410 con 0x7f984015d720 2024-09-16T01:45:08.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.828+0000 7f984508d640 1 -- 172.21.15.88:0/970021657 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98401519d0 con 0x7f984015d720 2024-09-16T01:45:08.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.829+0000 7f984508d640 1 -- 172.21.15.88:0/970021657 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98400ce160 con 0x7f984015d720 2024-09-16T01:45:08.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.830+0000 7f98277fe640 1 -- 172.21.15.88:0/970021657 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f98380159c0 con 0x7f984015d720 2024-09-16T01:45:08.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.830+0000 7f98277fe640 1 --2- 172.21.15.88:0/970021657 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f982c06c2e0 0x7f982c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:08.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.830+0000 7f98277fe640 1 -- 172.21.15.88:0/970021657 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5795+0+0 (secure 0 0 0) 0x7f983808c020 con 0x7f984015d720 2024-09-16T01:45:08.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.830+0000 7f983ed9d640 1 --2- 172.21.15.88:0/970021657 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f982c06c2e0 0x7f982c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:08.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.831+0000 7f983ed9d640 1 --2- 172.21.15.88:0/970021657 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f982c06c2e0 0x7f982c06e7a0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f9830001050 tx=0x7f9830007970 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:08.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:08.834+0000 7f98277fe640 1 -- 172.21.15.88:0/970021657 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9838019030 con 0x7f984015d720 2024-09-16T01:45:09.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.034+0000 7f984508d640 1 -- 172.21.15.88:0/970021657 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98400718b0 con 0x7f984015d720 2024-09-16T01:45:09.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.034+0000 7f98277fe640 1 -- 172.21.15.88:0/970021657 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 332 v332) ==== 95+0+5244 (secure 0 0 0) 0x7f9838010040 con 0x7f984015d720 2024-09-16T01:45:09.036 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:45:09.036 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":332,"btime":"2024-09-16T01:45:05:273239+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7649,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2706550508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2706550508},{"type":"v1","addr":"172.21.15.154:6834","nonce":2706550508}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322},{"gid":8569,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/3460839382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":3460839382},{"type":"v1","addr":"172.21.15.88:6837","nonce":3460839382}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322},{"gid":8581,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2822970755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2822970755},{"type":"v1","addr":"172.21.15.154:6835","nonce":2822970755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322}],"filesystems":[{"mdsmap":{"epoch":332,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:44:51.845091+0000","modified":"2024-09-16T01:45:05.273233+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":139,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10707},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10707":{"gid":10707,"name":"a","rank":0,"incarnation":329,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6836/4245333273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4245333273},{"type":"v1","addr":"172.21.15.88:6836","nonce":4245333273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10707,"qdb_cluster":[10707]},"id":7}]} 2024-09-16T01:45:09.037 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 332 2024-09-16T01:45:09.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.036+0000 7f984508d640 1 -- 172.21.15.88:0/970021657 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f982c06c2e0 msgr2=0x7f982c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:09.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.036+0000 7f984508d640 1 --2- 172.21.15.88:0/970021657 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f982c06c2e0 0x7f982c06e7a0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f9830001050 tx=0x7f9830007970 comp rx=0 tx=0).stop 2024-09-16T01:45:09.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.036+0000 7f984508d640 1 -- 172.21.15.88:0/970021657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f984015d720 msgr2=0x7f984015dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:09.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.036+0000 7f984508d640 1 --2- 172.21.15.88:0/970021657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f984015d720 0x7f984015dbd0 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7f9840070e70 tx=0x7f9838011f50 comp rx=0 tx=0).stop 2024-09-16T01:45:09.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.036+0000 7f984508d640 1 -- 172.21.15.88:0/970021657 shutdown_connections 2024-09-16T01:45:09.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.036+0000 7f984508d640 1 --2- 172.21.15.88:0/970021657 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f982c06c2e0 0x7f982c06e7a0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.036+0000 7f984508d640 1 --2- 172.21.15.88:0/970021657 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f984015d720 0x7f984015dbd0 unknown :-1 s=CLOSED pgs=2412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.036+0000 7f984508d640 1 --2- 172.21.15.88:0/970021657 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f984014f730 0x7f984014fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.036+0000 7f984508d640 1 --2- 172.21.15.88:0/970021657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9828004800 0x7f98401531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.036+0000 7f984508d640 1 -- 172.21.15.88:0/970021657 >> 172.21.15.88:0/970021657 conn(0x7f984006bc10 msgr2=0x7f98400720b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:09.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.036+0000 7f984508d640 1 -- 172.21.15.88:0/970021657 shutdown_connections 2024-09-16T01:45:09.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.036+0000 7f984508d640 1 -- 172.21.15.88:0/970021657 wait complete. 2024-09-16T01:45:09.049 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:45:09.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.146+0000 7f161042e640 1 Processor -- start 2024-09-16T01:45:09.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.146+0000 7f161042e640 1 -- start start 2024-09-16T01:45:09.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.147+0000 7f161042e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f16080cfc80 0x7f16080b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:09.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.147+0000 7f161042e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f160815c330 0x7f16080b4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:09.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.147+0000 7f161042e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f160815c740 con 0x7f1608155800 2024-09-16T01:45:09.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.147+0000 7f161042e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f16080b4e60 con 0x7f160815c330 2024-09-16T01:45:09.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.147+0000 7f161042e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f16080b50a0 con 0x7f16080cfc80 2024-09-16T01:45:09.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.147+0000 7f160e9cb640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f160815c330 0x7f16080b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:09.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.147+0000 7f160e9cb640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f160815c330 0x7f16080b4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54948/0 (socket says 172.21.15.88:54948) 2024-09-16T01:45:09.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.147+0000 7f160e9cb640 1 -- 172.21.15.88:0/3265457769 learned_addr learned my addr 172.21.15.88:0/3265457769 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:09.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.147+0000 7f160d9c9640 1 --2- 172.21.15.88:0/3265457769 >> v2:172.21.15.154:3300/0 conn(0x7f16080cfc80 0x7f16080b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:09.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.147+0000 7f160e9cb640 1 -- 172.21.15.88:0/3265457769 >> v2:172.21.15.154:3300/0 conn(0x7f16080cfc80 msgr2=0x7f16080b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:09.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.147+0000 7f160e9cb640 1 --2- 172.21.15.88:0/3265457769 >> v2:172.21.15.154:3300/0 conn(0x7f16080cfc80 0x7f16080b6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.147+0000 7f160e9cb640 1 -- 172.21.15.88:0/3265457769 >> v1:172.21.15.154:6789/0 conn(0x7f1608155800 legacy=0x7f16080cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:09.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.147+0000 7f160e9cb640 1 -- 172.21.15.88:0/3265457769 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16080b5380 con 0x7f160815c330 2024-09-16T01:45:09.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.148+0000 7f160e9cb640 1 --2- 172.21.15.88:0/3265457769 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f160815c330 0x7f16080b4920 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7f1600009e50 tx=0x7f160002a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c57f580cc0e1342f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:09.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.148+0000 7f160d1c8640 1 -- 172.21.15.88:0/3265457769 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1600036070 con 0x7f160815c330 2024-09-16T01:45:09.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.148+0000 7f160d1c8640 1 -- 172.21.15.88:0/3265457769 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1600031600 con 0x7f160815c330 2024-09-16T01:45:09.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.148+0000 7f160d1c8640 1 -- 172.21.15.88:0/3265457769 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16000319d0 con 0x7f160815c330 2024-09-16T01:45:09.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.149+0000 7f161042e640 1 -- 172.21.15.88:0/3265457769 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f160815c330 msgr2=0x7f16080b4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:09.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.149+0000 7f161042e640 1 --2- 172.21.15.88:0/3265457769 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f160815c330 0x7f16080b4920 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7f1600009e50 tx=0x7f160002a760 comp rx=0 tx=0).stop 2024-09-16T01:45:09.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.149+0000 7f161042e640 1 -- 172.21.15.88:0/3265457769 shutdown_connections 2024-09-16T01:45:09.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.149+0000 7f161042e640 1 --2- 172.21.15.88:0/3265457769 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f160815c330 0x7f16080b4920 unknown :-1 s=CLOSED pgs=2413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.149+0000 7f161042e640 1 --2- 172.21.15.88:0/3265457769 >> v2:172.21.15.154:3300/0 conn(0x7f16080cfc80 0x7f16080b6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.149+0000 7f161042e640 1 -- 172.21.15.88:0/3265457769 >> 172.21.15.88:0/3265457769 conn(0x7f160806bc10 msgr2=0x7f160806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:09.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.149+0000 7f161042e640 1 -- 172.21.15.88:0/3265457769 shutdown_connections 2024-09-16T01:45:09.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.149+0000 7f161042e640 1 -- 172.21.15.88:0/3265457769 wait complete. 2024-09-16T01:45:09.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.150+0000 7f161042e640 1 Processor -- start 2024-09-16T01:45:09.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.150+0000 7f161042e640 1 -- start start 2024-09-16T01:45:09.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.151+0000 7f161042e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16080cfc80 0x7f16081f13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:09.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.151+0000 7f161042e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1608155800 0x7f16080b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:09.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.151+0000 7f161042e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f160815c330 0x7f16080b3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:09.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.151+0000 7f161042e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f16080b5c20 con 0x7f1608155800 2024-09-16T01:45:09.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.151+0000 7f161042e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f16080b5940 con 0x7f16080cfc80 2024-09-16T01:45:09.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.151+0000 7f161042e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1608155be0 con 0x7f160815c330 2024-09-16T01:45:09.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.151+0000 7f160d9c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1608155800 0x7f16080b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:09.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.151+0000 7f160d9c9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1608155800 0x7f16080b3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54956/0 (socket says 172.21.15.88:54956) 2024-09-16T01:45:09.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.151+0000 7f160d9c9640 1 -- 172.21.15.88:0/698876122 learned_addr learned my addr 172.21.15.88:0/698876122 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:09.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.151+0000 7f160e9cb640 1 --2- 172.21.15.88:0/698876122 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f160815c330 0x7f16080b3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:09.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.151+0000 7f160d9c9640 1 -- 172.21.15.88:0/698876122 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f160815c330 msgr2=0x7f16080b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:09.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.151+0000 7f160d9c9640 1 --2- 172.21.15.88:0/698876122 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f160815c330 0x7f16080b3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.151+0000 7f160d9c9640 1 -- 172.21.15.88:0/698876122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16080cfc80 msgr2=0x7f16081f13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:09.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.151+0000 7f160e1ca640 1 --2- 172.21.15.88:0/698876122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16080cfc80 0x7f16081f13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:09.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.151+0000 7f160d9c9640 1 --2- 172.21.15.88:0/698876122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16080cfc80 0x7f16081f13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.151+0000 7f160d9c9640 1 -- 172.21.15.88:0/698876122 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16081f1910 con 0x7f1608155800 2024-09-16T01:45:09.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.151+0000 7f160d9c9640 1 --2- 172.21.15.88:0/698876122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1608155800 0x7f16080b3810 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7f15f800e980 tx=0x7f15f800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:09.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.151+0000 7f15feffd640 1 -- 172.21.15.88:0/698876122 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f8017070 con 0x7f1608155800 2024-09-16T01:45:09.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.151+0000 7f15feffd640 1 -- 172.21.15.88:0/698876122 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15f800c9a0 con 0x7f1608155800 2024-09-16T01:45:09.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.151+0000 7f161042e640 1 -- 172.21.15.88:0/698876122 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1608140510 con 0x7f1608155800 2024-09-16T01:45:09.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.151+0000 7f15feffd640 1 -- 172.21.15.88:0/698876122 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f800cd70 con 0x7f1608155800 2024-09-16T01:45:09.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.151+0000 7f161042e640 1 -- 172.21.15.88:0/698876122 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1608140ad0 con 0x7f1608155800 2024-09-16T01:45:09.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.152+0000 7f161042e640 1 -- 172.21.15.88:0/698876122 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16080b6140 con 0x7f1608155800 2024-09-16T01:45:09.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.153+0000 7f15feffd640 1 -- 172.21.15.88:0/698876122 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f15f801a070 con 0x7f1608155800 2024-09-16T01:45:09.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.153+0000 7f15feffd640 1 --2- 172.21.15.88:0/698876122 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f15ec06c2b0 0x7f15ec06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:09.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.153+0000 7f15feffd640 1 -- 172.21.15.88:0/698876122 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5795+0+0 (secure 0 0 0) 0x7f15f808b380 con 0x7f1608155800 2024-09-16T01:45:09.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.153+0000 7f160e1ca640 1 --2- 172.21.15.88:0/698876122 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f15ec06c2b0 0x7f15ec06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:09.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.154+0000 7f160e1ca640 1 --2- 172.21.15.88:0/698876122 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f15ec06c2b0 0x7f15ec06e770 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f16080b46d0 tx=0x7f1604009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:09.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.158+0000 7f15feffd640 1 -- 172.21.15.88:0/698876122 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f15f8059b20 con 0x7f1608155800 2024-09-16T01:45:09.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.359+0000 7f161042e640 1 -- 172.21.15.88:0/698876122 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16080633d0 con 0x7f1608155800 2024-09-16T01:45:09.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.359+0000 7f15feffd640 1 -- 172.21.15.88:0/698876122 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 332 v332) ==== 95+0+5244 (secure 0 0 0) 0x7f15f8059420 con 0x7f1608155800 2024-09-16T01:45:09.362 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:45:09.362 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":332,"btime":"2024-09-16T01:45:05:273239+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7649,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2706550508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2706550508},{"type":"v1","addr":"172.21.15.154:6834","nonce":2706550508}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322},{"gid":8569,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/3460839382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":3460839382},{"type":"v1","addr":"172.21.15.88:6837","nonce":3460839382}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322},{"gid":8581,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2822970755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2822970755},{"type":"v1","addr":"172.21.15.154:6835","nonce":2822970755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322}],"filesystems":[{"mdsmap":{"epoch":332,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:44:51.845091+0000","modified":"2024-09-16T01:45:05.273233+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":139,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10707},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10707":{"gid":10707,"name":"a","rank":0,"incarnation":329,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6836/4245333273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4245333273},{"type":"v1","addr":"172.21.15.88:6836","nonce":4245333273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10707,"qdb_cluster":[10707]},"id":7}]} 2024-09-16T01:45:09.363 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 332 2024-09-16T01:45:09.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.361+0000 7f161042e640 1 -- 172.21.15.88:0/698876122 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f15ec06c2b0 msgr2=0x7f15ec06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:09.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.361+0000 7f161042e640 1 --2- 172.21.15.88:0/698876122 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f15ec06c2b0 0x7f15ec06e770 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f16080b46d0 tx=0x7f1604009040 comp rx=0 tx=0).stop 2024-09-16T01:45:09.363 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.361+0000 7f161042e640 1 -- 172.21.15.88:0/698876122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1608155800 msgr2=0x7f16080b3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:09.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.361+0000 7f161042e640 1 --2- 172.21.15.88:0/698876122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1608155800 0x7f16080b3810 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7f15f800e980 tx=0x7f15f800ee40 comp rx=0 tx=0).stop 2024-09-16T01:45:09.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.361+0000 7f161042e640 1 -- 172.21.15.88:0/698876122 shutdown_connections 2024-09-16T01:45:09.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.361+0000 7f161042e640 1 --2- 172.21.15.88:0/698876122 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f15ec06c2b0 0x7f15ec06e770 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.361+0000 7f161042e640 1 --2- 172.21.15.88:0/698876122 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f160815c330 0x7f16080b3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.361+0000 7f161042e640 1 --2- 172.21.15.88:0/698876122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1608155800 0x7f16080b3810 unknown :-1 s=CLOSED pgs=2414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.361+0000 7f161042e640 1 --2- 172.21.15.88:0/698876122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16080cfc80 0x7f16081f13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.361+0000 7f161042e640 1 -- 172.21.15.88:0/698876122 >> 172.21.15.88:0/698876122 conn(0x7f160806bc10 msgr2=0x7f160806ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:09.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.361+0000 7f161042e640 1 -- 172.21.15.88:0/698876122 shutdown_connections 2024-09-16T01:45:09.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.361+0000 7f161042e640 1 -- 172.21.15.88:0/698876122 wait complete. 2024-09-16T01:45:09.374 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:45:09.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.470+0000 7fc1f53b7640 1 Processor -- start 2024-09-16T01:45:09.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.470+0000 7fc1f53b7640 1 -- start start 2024-09-16T01:45:09.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.471+0000 7fc1f53b7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc1f00d0fe0 0x7fc1f006cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:09.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.471+0000 7fc1f53b7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1f015f130 con 0x7fc1f006d260 2024-09-16T01:45:09.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.471+0000 7fc1f53b7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1f015f410 con 0x7fc1f015b460 2024-09-16T01:45:09.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.471+0000 7fc1f53b7640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc1f006d6b0 con 0x7fc1f00d0fe0 2024-09-16T01:45:09.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.471+0000 7fc1ef7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc1f006d260 0x7fc1f00b5080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35496/0 (socket says 172.21.15.88:35496) 2024-09-16T01:45:09.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.471+0000 7fc1ef7fe640 1 -- 172.21.15.88:0/355059138 learned_addr learned my addr 172.21.15.88:0/355059138 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:09.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.472+0000 7fc1ee7fc640 1 --2- 172.21.15.88:0/355059138 >> v2:172.21.15.154:3300/0 conn(0x7fc1f00d0fe0 0x7fc1f006cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:09.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.472+0000 7fc1edffb640 1 -- 172.21.15.88:0/355059138 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1779223364 0 0) 0x7fc1f015f130 con 0x7fc1f006d260 2024-09-16T01:45:09.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.472+0000 7fc1edffb640 1 -- 172.21.15.88:0/355059138 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1dc003520 con 0x7fc1f006d260 2024-09-16T01:45:09.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.472+0000 7fc1ee7fc640 1 -- 172.21.15.88:0/355059138 >> v1:172.21.15.154:6789/0 conn(0x7fc1f015b460 legacy=0x7fc1f00b44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:09.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.472+0000 7fc1ee7fc640 1 -- 172.21.15.88:0/355059138 >> v1:172.21.15.88:6789/0 conn(0x7fc1f006d260 legacy=0x7fc1f00b5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:09.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.472+0000 7fc1ee7fc640 1 -- 172.21.15.88:0/355059138 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1f015f6f0 con 0x7fc1f00d0fe0 2024-09-16T01:45:09.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.473+0000 7fc1ee7fc640 1 --2- 172.21.15.88:0/355059138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1f00d0fe0 0x7fc1f006cd20 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fc1d80097a0 tx=0x7fc1d802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d7a2499660a76f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:09.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.473+0000 7fc1edffb640 1 -- 172.21.15.88:0/355059138 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d8037070 con 0x7fc1f00d0fe0 2024-09-16T01:45:09.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.473+0000 7fc1edffb640 1 -- 172.21.15.88:0/355059138 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1d8032440 con 0x7fc1f00d0fe0 2024-09-16T01:45:09.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.473+0000 7fc1edffb640 1 -- 172.21.15.88:0/355059138 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d8032810 con 0x7fc1f00d0fe0 2024-09-16T01:45:09.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.473+0000 7fc1f53b7640 1 -- 172.21.15.88:0/355059138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1f00d0fe0 msgr2=0x7fc1f006cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:09.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.473+0000 7fc1f53b7640 1 --2- 172.21.15.88:0/355059138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1f00d0fe0 0x7fc1f006cd20 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fc1d80097a0 tx=0x7fc1d802a230 comp rx=0 tx=0).stop 2024-09-16T01:45:09.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.473+0000 7fc1f53b7640 1 -- 172.21.15.88:0/355059138 shutdown_connections 2024-09-16T01:45:09.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.473+0000 7fc1f53b7640 1 --2- 172.21.15.88:0/355059138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1f00d0fe0 0x7fc1f006cd20 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.473+0000 7fc1f53b7640 1 -- 172.21.15.88:0/355059138 >> 172.21.15.88:0/355059138 conn(0x7fc1f00b6ec0 msgr2=0x7fc1f00b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:09.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.474+0000 7fc1f53b7640 1 -- 172.21.15.88:0/355059138 shutdown_connections 2024-09-16T01:45:09.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.474+0000 7fc1f53b7640 1 -- 172.21.15.88:0/355059138 wait complete. 2024-09-16T01:45:09.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.475+0000 7fc1f53b7640 1 Processor -- start 2024-09-16T01:45:09.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.475+0000 7fc1f53b7640 1 -- start start 2024-09-16T01:45:09.478 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.476+0000 7fc1f53b7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1f006d260 0x7fc1f0152af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:09.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.476+0000 7fc1f53b7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1f00d0fe0 0x7fc1f0153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:09.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.476+0000 7fc1f53b7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1f015b460 0x7fc1f01519e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:09.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.476+0000 7fc1f53b7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f00b5790 con 0x7fc1f015b460 2024-09-16T01:45:09.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.476+0000 7fc1f53b7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f015eee0 con 0x7fc1f00d0fe0 2024-09-16T01:45:09.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.476+0000 7fc1f53b7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc1f00b5910 con 0x7fc1f006d260 2024-09-16T01:45:09.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.476+0000 7fc1ef7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1f015b460 0x7fc1f01519e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:09.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.476+0000 7fc1ef7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1f015b460 0x7fc1f01519e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54960/0 (socket says 172.21.15.88:54960) 2024-09-16T01:45:09.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.476+0000 7fc1ef7fe640 1 -- 172.21.15.88:0/3411823578 learned_addr learned my addr 172.21.15.88:0/3411823578 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:09.479 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.476+0000 7fc1ee7fc640 1 --2- 172.21.15.88:0/3411823578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1f00d0fe0 0x7fc1f0153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:09.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.476+0000 7fc1ef7fe640 1 -- 172.21.15.88:0/3411823578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1f006d260 msgr2=0x7fc1f0152af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:09.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.476+0000 7fc1eeffd640 1 --2- 172.21.15.88:0/3411823578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1f006d260 0x7fc1f0152af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:09.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.476+0000 7fc1ef7fe640 1 --2- 172.21.15.88:0/3411823578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1f006d260 0x7fc1f0152af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.476+0000 7fc1ef7fe640 1 -- 172.21.15.88:0/3411823578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1f00d0fe0 msgr2=0x7fc1f0153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:09.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.476+0000 7fc1ef7fe640 1 --2- 172.21.15.88:0/3411823578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1f00d0fe0 0x7fc1f0153030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.476+0000 7fc1ef7fe640 1 -- 172.21.15.88:0/3411823578 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1f01517a0 con 0x7fc1f015b460 2024-09-16T01:45:09.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.476+0000 7fc1ef7fe640 1 --2- 172.21.15.88:0/3411823578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1f015b460 0x7fc1f01519e0 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7fc1e000cb40 tx=0x7fc1e000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:09.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.476+0000 7fc1e77fe640 1 -- 172.21.15.88:0/3411823578 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e0017070 con 0x7fc1f015b460 2024-09-16T01:45:09.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.476+0000 7fc1e77fe640 1 -- 172.21.15.88:0/3411823578 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1e000aca0 con 0x7fc1f015b460 2024-09-16T01:45:09.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.476+0000 7fc1f53b7640 1 -- 172.21.15.88:0/3411823578 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1f0154f60 con 0x7fc1f015b460 2024-09-16T01:45:09.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.476+0000 7fc1e77fe640 1 -- 172.21.15.88:0/3411823578 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e0012620 con 0x7fc1f015b460 2024-09-16T01:45:09.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.476+0000 7fc1f53b7640 1 -- 172.21.15.88:0/3411823578 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1f0155520 con 0x7fc1f015b460 2024-09-16T01:45:09.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.477+0000 7fc1f53b7640 1 -- 172.21.15.88:0/3411823578 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1f015e6c0 con 0x7fc1f015b460 2024-09-16T01:45:09.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.478+0000 7fc1e77fe640 1 -- 172.21.15.88:0/3411823578 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc1e001a070 con 0x7fc1f015b460 2024-09-16T01:45:09.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.478+0000 7fc1e77fe640 1 --2- 172.21.15.88:0/3411823578 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc1d006c2b0 0x7fc1d006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:09.481 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.478+0000 7fc1e77fe640 1 -- 172.21.15.88:0/3411823578 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5795+0+0 (secure 0 0 0) 0x7fc1e008b390 con 0x7fc1f015b460 2024-09-16T01:45:09.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.478+0000 7fc1eeffd640 1 --2- 172.21.15.88:0/3411823578 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc1d006c2b0 0x7fc1d006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:09.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.479+0000 7fc1eeffd640 1 --2- 172.21.15.88:0/3411823578 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc1d006c2b0 0x7fc1d006e770 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8001500 tx=0x7fc1e8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:09.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.482+0000 7fc1e77fe640 1 -- 172.21.15.88:0/3411823578 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fc1f015e6c0 con 0x7fc1f015b460 2024-09-16T01:45:09.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.649+0000 7fc1f53b7640 1 -- 172.21.15.88:0/3411823578 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc1f006e8f0 con 0x7fc1f015b460 2024-09-16T01:45:09.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.650+0000 7fc1e77fe640 1 -- 172.21.15.88:0/3411823578 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) ==== 74+0+16782 (secure 0 0 0) 0x7fc1e0059a20 con 0x7fc1f015b460 2024-09-16T01:45:09.651 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:45:09.652 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":139,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:45:01.261760+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:44:50.167367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-09-16T01:44:50.816821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:6836/656828696":"2024-09-17T01:45:01.261737+0000","172.21.15.88:6834/656828696":"2024-09-17T01:45:01.261737+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:45:09.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.652+0000 7fc1f53b7640 1 -- 172.21.15.88:0/3411823578 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc1d006c2b0 msgr2=0x7fc1d006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:09.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.652+0000 7fc1f53b7640 1 --2- 172.21.15.88:0/3411823578 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc1d006c2b0 0x7fc1d006e770 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8001500 tx=0x7fc1e8009040 comp rx=0 tx=0).stop 2024-09-16T01:45:09.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.652+0000 7fc1f53b7640 1 -- 172.21.15.88:0/3411823578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1f015b460 msgr2=0x7fc1f01519e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:09.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.652+0000 7fc1f53b7640 1 --2- 172.21.15.88:0/3411823578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1f015b460 0x7fc1f01519e0 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7fc1e000cb40 tx=0x7fc1e000a060 comp rx=0 tx=0).stop 2024-09-16T01:45:09.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.652+0000 7fc1f53b7640 1 -- 172.21.15.88:0/3411823578 shutdown_connections 2024-09-16T01:45:09.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.652+0000 7fc1f53b7640 1 --2- 172.21.15.88:0/3411823578 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc1d006c2b0 0x7fc1d006e770 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.652+0000 7fc1f53b7640 1 --2- 172.21.15.88:0/3411823578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1f015b460 0x7fc1f01519e0 unknown :-1 s=CLOSED pgs=2416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.652+0000 7fc1f53b7640 1 --2- 172.21.15.88:0/3411823578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1f00d0fe0 0x7fc1f0153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.652+0000 7fc1f53b7640 1 --2- 172.21.15.88:0/3411823578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc1f006d260 0x7fc1f0152af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.652+0000 7fc1f53b7640 1 -- 172.21.15.88:0/3411823578 >> 172.21.15.88:0/3411823578 conn(0x7fc1f00b6ec0 msgr2=0x7fc1f00b5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:09.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.652+0000 7fc1f53b7640 1 -- 172.21.15.88:0/3411823578 shutdown_connections 2024-09-16T01:45:09.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.652+0000 7fc1f53b7640 1 -- 172.21.15.88:0/3411823578 wait complete. 2024-09-16T01:45:09.665 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-16T01:45:09.665 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:45:09.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.762+0000 7efdc8080640 1 Processor -- start 2024-09-16T01:45:09.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.763+0000 7efdc8080640 1 -- start start 2024-09-16T01:45:09.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.763+0000 7efdc8080640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efdc015e660 0x7efdc00cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:09.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.763+0000 7efdc8080640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efdc00d1820 0x7efdc00b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:09.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.764+0000 7efdc8080640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdc015ea40 con 0x7efdc00d3310 2024-09-16T01:45:09.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.764+0000 7efdc8080640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7efdc00d3870 con 0x7efdc015e660 2024-09-16T01:45:09.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.764+0000 7efdc8080640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efdc00d3b50 con 0x7efdc00d1820 2024-09-16T01:45:09.766 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.764+0000 7efdc5e1c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efdc015e660 0x7efdc00cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:09.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.764+0000 7efdc5e1c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7efdc015e660 0x7efdc00cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54970/0 (socket says 172.21.15.88:54970) 2024-09-16T01:45:09.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.764+0000 7efdc5e1c640 1 -- 172.21.15.88:0/1626720246 learned_addr learned my addr 172.21.15.88:0/1626720246 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:09.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.764+0000 7efdc661d640 1 --2- 172.21.15.88:0/1626720246 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efdc00d1820 0x7efdc00b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:09.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.764+0000 7efdc5e1c640 1 -- 172.21.15.88:0/1626720246 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efdc00d1820 msgr2=0x7efdc00b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:09.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.764+0000 7efdc5e1c640 1 --2- 172.21.15.88:0/1626720246 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efdc00d1820 0x7efdc00b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.764+0000 7efdc5e1c640 1 -- 172.21.15.88:0/1626720246 >> v1:172.21.15.154:6789/0 conn(0x7efdc00d3310 legacy=0x7efdc00d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:09.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.764+0000 7efdc5e1c640 1 -- 172.21.15.88:0/1626720246 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdc00d3e30 con 0x7efdc015e660 2024-09-16T01:45:09.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.765+0000 7efdc5e1c640 1 --2- 172.21.15.88:0/1626720246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efdc015e660 0x7efdc00cf730 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7efdb4009870 tx=0x7efdb402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e6ef9d575eed5dc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:09.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.765+0000 7efdc4e1a640 1 -- 172.21.15.88:0/1626720246 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb4037070 con 0x7efdc015e660 2024-09-16T01:45:09.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.765+0000 7efdc4e1a640 1 -- 172.21.15.88:0/1626720246 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdb4032440 con 0x7efdc015e660 2024-09-16T01:45:09.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.765+0000 7efdc4e1a640 1 -- 172.21.15.88:0/1626720246 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb4032810 con 0x7efdc015e660 2024-09-16T01:45:09.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.765+0000 7efdc8080640 1 -- 172.21.15.88:0/1626720246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efdc015e660 msgr2=0x7efdc00cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:09.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.765+0000 7efdc8080640 1 --2- 172.21.15.88:0/1626720246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efdc015e660 0x7efdc00cf730 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7efdb4009870 tx=0x7efdb402a270 comp rx=0 tx=0).stop 2024-09-16T01:45:09.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.765+0000 7efdc8080640 1 -- 172.21.15.88:0/1626720246 shutdown_connections 2024-09-16T01:45:09.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.765+0000 7efdc8080640 1 --2- 172.21.15.88:0/1626720246 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efdc00d1820 0x7efdc00b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.765+0000 7efdc8080640 1 --2- 172.21.15.88:0/1626720246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efdc015e660 0x7efdc00cf730 unknown :-1 s=CLOSED pgs=2417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.765+0000 7efdc8080640 1 -- 172.21.15.88:0/1626720246 >> 172.21.15.88:0/1626720246 conn(0x7efdc00b6410 msgr2=0x7efdc00b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:09.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.765+0000 7efdc8080640 1 -- 172.21.15.88:0/1626720246 shutdown_connections 2024-09-16T01:45:09.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.765+0000 7efdc8080640 1 -- 172.21.15.88:0/1626720246 wait complete. 2024-09-16T01:45:09.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.767+0000 7efdc8080640 1 Processor -- start 2024-09-16T01:45:09.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.767+0000 7efdc8080640 1 -- start start 2024-09-16T01:45:09.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.767+0000 7efdc8080640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efdc00d1820 0x7efdc00ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:09.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.767+0000 7efdc8080640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efdc00d3310 0x7efdc01553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:09.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.767+0000 7efdc8080640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdc015e660 0x7efdc0151700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:09.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.767+0000 7efdc8080640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7efdc00cfc70 con 0x7efdc00d3310 2024-09-16T01:45:09.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.767+0000 7efdc8080640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efdc00d43f0 con 0x7efdc015e660 2024-09-16T01:45:09.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.767+0000 7efdc8080640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efdc00b80b0 con 0x7efdc00d1820 2024-09-16T01:45:09.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.767+0000 7efdc561b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efdc00d3310 0x7efdc01553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:09.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.767+0000 7efdc561b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efdc00d3310 0x7efdc01553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54978/0 (socket says 172.21.15.88:54978) 2024-09-16T01:45:09.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.767+0000 7efdc561b640 1 -- 172.21.15.88:0/2718572317 learned_addr learned my addr 172.21.15.88:0/2718572317 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:09.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.768+0000 7efdc5e1c640 1 --2- 172.21.15.88:0/2718572317 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efdc00d1820 0x7efdc00ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:09.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.768+0000 7efdc661d640 1 --2- 172.21.15.88:0/2718572317 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdc015e660 0x7efdc0151700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:09.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.768+0000 7efdc561b640 1 -- 172.21.15.88:0/2718572317 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efdc00d1820 msgr2=0x7efdc00ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:09.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.768+0000 7efdc561b640 1 --2- 172.21.15.88:0/2718572317 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efdc00d1820 0x7efdc00ca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.768+0000 7efdc561b640 1 -- 172.21.15.88:0/2718572317 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdc015e660 msgr2=0x7efdc0151700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:09.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.768+0000 7efdc561b640 1 --2- 172.21.15.88:0/2718572317 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdc015e660 0x7efdc0151700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.768+0000 7efdc561b640 1 -- 172.21.15.88:0/2718572317 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdc00cb0f0 con 0x7efdc00d3310 2024-09-16T01:45:09.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.768+0000 7efdc561b640 1 --2- 172.21.15.88:0/2718572317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efdc00d3310 0x7efdc01553a0 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7efdbc00c430 tx=0x7efdbc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:09.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.768+0000 7efdb27fc640 1 -- 172.21.15.88:0/2718572317 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdbc0052d0 con 0x7efdc00d3310 2024-09-16T01:45:09.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.768+0000 7efdb27fc640 1 -- 172.21.15.88:0/2718572317 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdbc005490 con 0x7efdc00d3310 2024-09-16T01:45:09.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.768+0000 7efdc8080640 1 -- 172.21.15.88:0/2718572317 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdc00cae10 con 0x7efdc00d3310 2024-09-16T01:45:09.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.768+0000 7efdb27fc640 1 -- 172.21.15.88:0/2718572317 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdbc007260 con 0x7efdc00d3310 2024-09-16T01:45:09.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.768+0000 7efdc8080640 1 -- 172.21.15.88:0/2718572317 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdc0150f00 con 0x7efdc00d3310 2024-09-16T01:45:09.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.769+0000 7efdc8080640 1 -- 172.21.15.88:0/2718572317 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdc00b7b70 con 0x7efdc00d3310 2024-09-16T01:45:09.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.769+0000 7efdb27fc640 1 -- 172.21.15.88:0/2718572317 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7efdbc007400 con 0x7efdc00d3310 2024-09-16T01:45:09.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.770+0000 7efdb27fc640 1 --2- 172.21.15.88:0/2718572317 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efda406c2b0 0x7efda406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:09.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.770+0000 7efdc5e1c640 1 --2- 172.21.15.88:0/2718572317 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efda406c2b0 0x7efda406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:09.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.770+0000 7efdb27fc640 1 -- 172.21.15.88:0/2718572317 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5795+0+0 (secure 0 0 0) 0x7efdbc08a860 con 0x7efdc00d3310 2024-09-16T01:45:09.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.770+0000 7efdc5e1c640 1 --2- 172.21.15.88:0/2718572317 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efda406c2b0 0x7efda406e770 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7efdb4034000 tx=0x7efdb402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:09.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.774+0000 7efdb27fc640 1 -- 172.21.15.88:0/2718572317 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7efdbc0589b0 con 0x7efdc00d3310 2024-09-16T01:45:09.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.972+0000 7efdc8080640 1 -- 172.21.15.88:0/2718572317 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efdc015c450 con 0x7efdc00d3310 2024-09-16T01:45:09.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.973+0000 7efdb27fc640 1 -- 172.21.15.88:0/2718572317 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 332 v332) ==== 95+0+5244 (secure 0 0 0) 0x7efdbc05ccb0 con 0x7efdc00d3310 2024-09-16T01:45:09.975 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:45:09.975 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":332,"btime":"2024-09-16T01:45:05:273239+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7649,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2706550508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2706550508},{"type":"v1","addr":"172.21.15.154:6834","nonce":2706550508}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322},{"gid":8569,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/3460839382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":3460839382},{"type":"v1","addr":"172.21.15.88:6837","nonce":3460839382}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322},{"gid":8581,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2822970755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2822970755},{"type":"v1","addr":"172.21.15.154:6835","nonce":2822970755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322}],"filesystems":[{"mdsmap":{"epoch":332,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:44:51.845091+0000","modified":"2024-09-16T01:45:05.273233+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":139,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10707},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10707":{"gid":10707,"name":"a","rank":0,"incarnation":329,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6836/4245333273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4245333273},{"type":"v1","addr":"172.21.15.88:6836","nonce":4245333273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10707,"qdb_cluster":[10707]},"id":7}]} 2024-09-16T01:45:09.975 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 332 2024-09-16T01:45:09.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.975+0000 7efdc8080640 1 -- 172.21.15.88:0/2718572317 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efda406c2b0 msgr2=0x7efda406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:09.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.975+0000 7efdc8080640 1 --2- 172.21.15.88:0/2718572317 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efda406c2b0 0x7efda406e770 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7efdb4034000 tx=0x7efdb402e000 comp rx=0 tx=0).stop 2024-09-16T01:45:09.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.975+0000 7efdc8080640 1 -- 172.21.15.88:0/2718572317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efdc00d3310 msgr2=0x7efdc01553a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:09.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.975+0000 7efdc8080640 1 --2- 172.21.15.88:0/2718572317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efdc00d3310 0x7efdc01553a0 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7efdbc00c430 tx=0x7efdbc00c8f0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.975+0000 7efdc8080640 1 -- 172.21.15.88:0/2718572317 shutdown_connections 2024-09-16T01:45:09.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.975+0000 7efdc8080640 1 --2- 172.21.15.88:0/2718572317 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7efda406c2b0 0x7efda406e770 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.975+0000 7efdc8080640 1 --2- 172.21.15.88:0/2718572317 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdc015e660 0x7efdc0151700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.975+0000 7efdc8080640 1 --2- 172.21.15.88:0/2718572317 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efdc00d3310 0x7efdc01553a0 unknown :-1 s=CLOSED pgs=2418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.975+0000 7efdc8080640 1 --2- 172.21.15.88:0/2718572317 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efdc00d1820 0x7efdc00ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:09.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.975+0000 7efdc8080640 1 -- 172.21.15.88:0/2718572317 >> 172.21.15.88:0/2718572317 conn(0x7efdc00b6410 msgr2=0x7efdc00b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:09.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.975+0000 7efdc8080640 1 -- 172.21.15.88:0/2718572317 shutdown_connections 2024-09-16T01:45:09.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:09.975+0000 7efdc8080640 1 -- 172.21.15.88:0/2718572317 wait complete. 2024-09-16T01:45:09.987 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:45:10.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.084+0000 7f24a753b640 1 Processor -- start 2024-09-16T01:45:10.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.085+0000 7f24a753b640 1 -- start start 2024-09-16T01:45:10.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.085+0000 7f24a753b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f24a006e780 0x7f24a00ae890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:10.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.085+0000 7f24a753b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24a0073130 0x7f24a0073580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:10.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.085+0000 7f24a753b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24a00b5220 con 0x7f24a015e560 2024-09-16T01:45:10.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.085+0000 7f24a753b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f24a006ebc0 con 0x7f24a006e780 2024-09-16T01:45:10.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.085+0000 7f24a753b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f24a0073ac0 con 0x7f24a0073130 2024-09-16T01:45:10.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.086+0000 7f24a52d7640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f24a015e560 0x7f24a0072220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35502/0 (socket says 172.21.15.88:35502) 2024-09-16T01:45:10.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.086+0000 7f24a52d7640 1 -- 172.21.15.88:0/1921556474 learned_addr learned my addr 172.21.15.88:0/1921556474 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:10.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.086+0000 7f24a4ad6640 1 --2- 172.21.15.88:0/1921556474 >> v2:172.21.15.154:3300/0 conn(0x7f24a006e780 0x7f24a00ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:10.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.086+0000 7f24a5ad8640 1 --2- 172.21.15.88:0/1921556474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24a0073130 0x7f24a0073580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:10.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.086+0000 7f2493fff640 1 -- 172.21.15.88:0/1921556474 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1196851119 0 0) 0x7f24a00b5220 con 0x7f24a015e560 2024-09-16T01:45:10.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.086+0000 7f2493fff640 1 -- 172.21.15.88:0/1921556474 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f248c003520 con 0x7f24a015e560 2024-09-16T01:45:10.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.086+0000 7f24a4ad6640 1 -- 172.21.15.88:0/1921556474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24a0073130 msgr2=0x7f24a0073580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:10.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.086+0000 7f24a4ad6640 1 --2- 172.21.15.88:0/1921556474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24a0073130 0x7f24a0073580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.086+0000 7f24a4ad6640 1 -- 172.21.15.88:0/1921556474 >> v1:172.21.15.88:6789/0 conn(0x7f24a015e560 legacy=0x7f24a0072220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:10.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.086+0000 7f24a4ad6640 1 -- 172.21.15.88:0/1921556474 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a00b54c0 con 0x7f24a006e780 2024-09-16T01:45:10.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.087+0000 7f24a4ad6640 1 --2- 172.21.15.88:0/1921556474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24a006e780 0x7f24a00ae890 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f24940097a0 tx=0x7f249402a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4ded4950f351ce9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:10.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.087+0000 7f2493fff640 1 -- 172.21.15.88:0/1921556474 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2494037070 con 0x7f24a006e780 2024-09-16T01:45:10.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.087+0000 7f2493fff640 1 -- 172.21.15.88:0/1921556474 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f249402ae50 con 0x7f24a006e780 2024-09-16T01:45:10.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.087+0000 7f2493fff640 1 -- 172.21.15.88:0/1921556474 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2494032660 con 0x7f24a006e780 2024-09-16T01:45:10.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.087+0000 7f24a753b640 1 -- 172.21.15.88:0/1921556474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24a006e780 msgr2=0x7f24a00ae890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:10.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.087+0000 7f24a753b640 1 --2- 172.21.15.88:0/1921556474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24a006e780 0x7f24a00ae890 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f24940097a0 tx=0x7f249402a100 comp rx=0 tx=0).stop 2024-09-16T01:45:10.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.087+0000 7f24a753b640 1 -- 172.21.15.88:0/1921556474 shutdown_connections 2024-09-16T01:45:10.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.087+0000 7f24a753b640 1 --2- 172.21.15.88:0/1921556474 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24a0073130 0x7f24a0073580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.087+0000 7f24a753b640 1 --2- 172.21.15.88:0/1921556474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24a006e780 0x7f24a00ae890 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.087+0000 7f24a753b640 1 -- 172.21.15.88:0/1921556474 >> 172.21.15.88:0/1921556474 conn(0x7f24a006bc10 msgr2=0x7f24a006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:10.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.088+0000 7f24a753b640 1 -- 172.21.15.88:0/1921556474 shutdown_connections 2024-09-16T01:45:10.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.088+0000 7f24a753b640 1 -- 172.21.15.88:0/1921556474 wait complete. 2024-09-16T01:45:10.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.089+0000 7f24a753b640 1 Processor -- start 2024-09-16T01:45:10.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.089+0000 7f24a753b640 1 -- start start 2024-09-16T01:45:10.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.089+0000 7f24a753b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24a006e780 0x7f24a00b1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:10.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.089+0000 7f24a753b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24a0073130 0x7f24a00ae890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:10.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.089+0000 7f24a753b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24a015e560 0x7f24a00aedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:10.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.089+0000 7f24a753b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f24a00b5d60 con 0x7f24a0073130 2024-09-16T01:45:10.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.089+0000 7f24a753b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f24a00b5a80 con 0x7f24a015e560 2024-09-16T01:45:10.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.089+0000 7f24a753b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f24a015e940 con 0x7f24a006e780 2024-09-16T01:45:10.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.090+0000 7f24a4ad6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24a0073130 0x7f24a00ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:10.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.090+0000 7f24a4ad6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24a0073130 0x7f24a00ae890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:54994/0 (socket says 172.21.15.88:54994) 2024-09-16T01:45:10.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.090+0000 7f24a4ad6640 1 -- 172.21.15.88:0/780577525 learned_addr learned my addr 172.21.15.88:0/780577525 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:10.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.090+0000 7f24a5ad8640 1 --2- 172.21.15.88:0/780577525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24a015e560 0x7f24a00aedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:10.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.090+0000 7f24a52d7640 1 --2- 172.21.15.88:0/780577525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24a006e780 0x7f24a00b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:10.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.090+0000 7f24a4ad6640 1 -- 172.21.15.88:0/780577525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24a006e780 msgr2=0x7f24a00b1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:10.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.090+0000 7f24a4ad6640 1 --2- 172.21.15.88:0/780577525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24a006e780 0x7f24a00b1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.090+0000 7f24a4ad6640 1 -- 172.21.15.88:0/780577525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24a015e560 msgr2=0x7f24a00aedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:10.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.090+0000 7f24a4ad6640 1 --2- 172.21.15.88:0/780577525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24a015e560 0x7f24a00aedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.090+0000 7f24a4ad6640 1 -- 172.21.15.88:0/780577525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a014d6d0 con 0x7f24a0073130 2024-09-16T01:45:10.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.090+0000 7f24a52d7640 1 --2- 172.21.15.88:0/780577525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24a006e780 0x7f24a00b1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:45:10.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.090+0000 7f24a4ad6640 1 --2- 172.21.15.88:0/780577525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24a0073130 0x7f24a00ae890 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f2494035040 tx=0x7f24940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:10.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.090+0000 7f2491ffb640 1 -- 172.21.15.88:0/780577525 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2494041070 con 0x7f24a0073130 2024-09-16T01:45:10.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.090+0000 7f2491ffb640 1 -- 172.21.15.88:0/780577525 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f249403c460 con 0x7f24a0073130 2024-09-16T01:45:10.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.090+0000 7f2491ffb640 1 -- 172.21.15.88:0/780577525 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2494037040 con 0x7f24a0073130 2024-09-16T01:45:10.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.090+0000 7f24a753b640 1 -- 172.21.15.88:0/780577525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24a0150020 con 0x7f24a0073130 2024-09-16T01:45:10.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.090+0000 7f24a753b640 1 -- 172.21.15.88:0/780577525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24a00cd8a0 con 0x7f24a0073130 2024-09-16T01:45:10.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.091+0000 7f24a753b640 1 -- 172.21.15.88:0/780577525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24a0073580 con 0x7f24a0073130 2024-09-16T01:45:10.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.092+0000 7f2491ffb640 1 -- 172.21.15.88:0/780577525 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f249403a070 con 0x7f24a0073130 2024-09-16T01:45:10.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.092+0000 7f2491ffb640 1 --2- 172.21.15.88:0/780577525 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f248406c2b0 0x7f248406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:10.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.092+0000 7f2491ffb640 1 -- 172.21.15.88:0/780577525 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5795+0+0 (secure 0 0 0) 0x7f24940ab510 con 0x7f24a0073130 2024-09-16T01:45:10.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.092+0000 7f24a52d7640 1 --2- 172.21.15.88:0/780577525 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f248406c2b0 0x7f248406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:10.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.093+0000 7f24a52d7640 1 --2- 172.21.15.88:0/780577525 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f248406c2b0 0x7f248406e770 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f249c007fd0 tx=0x7f249c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:10.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.096+0000 7f2491ffb640 1 -- 172.21.15.88:0/780577525 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2494079660 con 0x7f24a0073130 2024-09-16T01:45:10.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.295+0000 7f24a753b640 1 -- 172.21.15.88:0/780577525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24a006d0f0 con 0x7f24a0073130 2024-09-16T01:45:10.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.296+0000 7f2491ffb640 1 -- 172.21.15.88:0/780577525 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 332 v332) ==== 95+0+5244 (secure 0 0 0) 0x7f249407d960 con 0x7f24a0073130 2024-09-16T01:45:10.297 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:45:10.298 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":332,"btime":"2024-09-16T01:45:05:273239+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7649,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2706550508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2706550508},{"type":"v1","addr":"172.21.15.154:6834","nonce":2706550508}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322},{"gid":8569,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/3460839382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":3460839382},{"type":"v1","addr":"172.21.15.88:6837","nonce":3460839382}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322},{"gid":8581,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2822970755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2822970755},{"type":"v1","addr":"172.21.15.154:6835","nonce":2822970755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322}],"filesystems":[{"mdsmap":{"epoch":332,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-16T01:44:51.845091+0000","modified":"2024-09-16T01:45:05.273233+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":139,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10707},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10707":{"gid":10707,"name":"a","rank":0,"incarnation":329,"state":"up:active","state_seq":4,"addr":"172.21.15.88:6836/4245333273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":4245333273},{"type":"v1","addr":"172.21.15.88:6836","nonce":4245333273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10707,"qdb_cluster":[10707]},"id":7}]} 2024-09-16T01:45:10.298 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 332 2024-09-16T01:45:10.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.298+0000 7f24a753b640 1 -- 172.21.15.88:0/780577525 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f248406c2b0 msgr2=0x7f248406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:10.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.298+0000 7f24a753b640 1 --2- 172.21.15.88:0/780577525 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f248406c2b0 0x7f248406e770 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f249c007fd0 tx=0x7f249c009040 comp rx=0 tx=0).stop 2024-09-16T01:45:10.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.298+0000 7f24a753b640 1 -- 172.21.15.88:0/780577525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24a0073130 msgr2=0x7f24a00ae890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:10.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.298+0000 7f24a753b640 1 --2- 172.21.15.88:0/780577525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24a0073130 0x7f24a00ae890 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f2494035040 tx=0x7f24940046c0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.298+0000 7f24a753b640 1 -- 172.21.15.88:0/780577525 shutdown_connections 2024-09-16T01:45:10.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.298+0000 7f24a753b640 1 --2- 172.21.15.88:0/780577525 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f248406c2b0 0x7f248406e770 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.298+0000 7f24a753b640 1 --2- 172.21.15.88:0/780577525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24a015e560 0x7f24a00aedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.298+0000 7f24a753b640 1 --2- 172.21.15.88:0/780577525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24a0073130 0x7f24a00ae890 unknown :-1 s=CLOSED pgs=2420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.298+0000 7f24a753b640 1 --2- 172.21.15.88:0/780577525 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24a006e780 0x7f24a00b1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.298+0000 7f24a753b640 1 -- 172.21.15.88:0/780577525 >> 172.21.15.88:0/780577525 conn(0x7f24a006bc10 msgr2=0x7f24a00b6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:10.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.298+0000 7f24a753b640 1 -- 172.21.15.88:0/780577525 shutdown_connections 2024-09-16T01:45:10.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.298+0000 7f24a753b640 1 -- 172.21.15.88:0/780577525 wait complete. 2024-09-16T01:45:10.311 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:45:10.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.409+0000 7f013cd9f640 1 Processor -- start 2024-09-16T01:45:10.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.409+0000 7f013cd9f640 1 -- start start 2024-09-16T01:45:10.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.410+0000 7f013cd9f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0138154e70 0x7f01380b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:10.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.410+0000 7f013cd9f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f01380ccf60 0x7f01380d0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:10.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.410+0000 7f013cd9f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01380b47f0 con 0x7f013815bbf0 2024-09-16T01:45:10.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.410+0000 7f013cd9f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f01380b5400 con 0x7f0138154e70 2024-09-16T01:45:10.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.410+0000 7f013cd9f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f01380b4ad0 con 0x7f01380ccf60 2024-09-16T01:45:10.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.410+0000 7f0137fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f013815bbf0 0x7f013815e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59924/0 (socket says 172.21.15.88:59924) 2024-09-16T01:45:10.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.410+0000 7f01377fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0138154e70 0x7f01380b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:10.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.410+0000 7f0137fff640 1 -- 172.21.15.88:0/1745168271 learned_addr learned my addr 172.21.15.88:0/1745168271 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:10.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.410+0000 7f0136ffd640 1 --2- 172.21.15.88:0/1745168271 >> v2:172.21.15.154:3300/0 conn(0x7f01380ccf60 0x7f01380d0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:10.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.410+0000 7f01377fe640 1 -- 172.21.15.88:0/1745168271 >> v2:172.21.15.154:3300/0 conn(0x7f01380ccf60 msgr2=0x7f01380d0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:10.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.410+0000 7f01377fe640 1 --2- 172.21.15.88:0/1745168271 >> v2:172.21.15.154:3300/0 conn(0x7f01380ccf60 0x7f01380d0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.410+0000 7f01377fe640 1 -- 172.21.15.88:0/1745168271 >> v1:172.21.15.88:6789/0 conn(0x7f013815bbf0 legacy=0x7f013815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:10.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.410+0000 7f01367fc640 1 -- 172.21.15.88:0/1745168271 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 685553635 0 0) 0x7f01380b47f0 con 0x7f013815bbf0 2024-09-16T01:45:10.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.410+0000 7f01377fe640 1 -- 172.21.15.88:0/1745168271 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01380b4db0 con 0x7f0138154e70 2024-09-16T01:45:10.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.411+0000 7f01377fe640 1 --2- 172.21.15.88:0/1745168271 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0138154e70 0x7f01380b2670 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f01380cc6c0 tx=0x7f012802b220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddeda5e042974887 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:10.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.411+0000 7f01367fc640 1 -- 172.21.15.88:0/1745168271 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0128038070 con 0x7f0138154e70 2024-09-16T01:45:10.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.411+0000 7f01367fc640 1 -- 172.21.15.88:0/1745168271 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0128033440 con 0x7f0138154e70 2024-09-16T01:45:10.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.411+0000 7f01367fc640 1 -- 172.21.15.88:0/1745168271 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0128033810 con 0x7f0138154e70 2024-09-16T01:45:10.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.412+0000 7f013cd9f640 1 -- 172.21.15.88:0/1745168271 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0138154e70 msgr2=0x7f01380b2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:10.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.412+0000 7f013cd9f640 1 --2- 172.21.15.88:0/1745168271 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0138154e70 0x7f01380b2670 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f01380cc6c0 tx=0x7f012802b220 comp rx=0 tx=0).stop 2024-09-16T01:45:10.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.412+0000 7f013cd9f640 1 -- 172.21.15.88:0/1745168271 shutdown_connections 2024-09-16T01:45:10.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.412+0000 7f013cd9f640 1 --2- 172.21.15.88:0/1745168271 >> v2:172.21.15.154:3300/0 conn(0x7f01380ccf60 0x7f01380d0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.412+0000 7f013cd9f640 1 --2- 172.21.15.88:0/1745168271 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0138154e70 0x7f01380b2670 unknown :-1 s=CLOSED pgs=2422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.412+0000 7f013cd9f640 1 -- 172.21.15.88:0/1745168271 >> 172.21.15.88:0/1745168271 conn(0x7f0138062a20 msgr2=0x7f013806bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:10.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.412+0000 7f013cd9f640 1 -- 172.21.15.88:0/1745168271 shutdown_connections 2024-09-16T01:45:10.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.412+0000 7f013cd9f640 1 -- 172.21.15.88:0/1745168271 wait complete. 2024-09-16T01:45:10.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.413+0000 7f013cd9f640 1 Processor -- start 2024-09-16T01:45:10.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.413+0000 7f013cd9f640 1 -- start start 2024-09-16T01:45:10.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.414+0000 7f013cd9f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01380ccf60 0x7f0138148de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:10.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.414+0000 7f013cd9f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0138154e70 0x7f0138151920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:10.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.414+0000 7f013cd9f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f013815bbf0 0x7f0138151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:10.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.414+0000 7f013cd9f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0138155250 con 0x7f013815bbf0 2024-09-16T01:45:10.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.414+0000 7f013cd9f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f01380b2bb0 con 0x7f01380ccf60 2024-09-16T01:45:10.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.414+0000 7f013cd9f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f01381553d0 con 0x7f0138154e70 2024-09-16T01:45:10.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.414+0000 7f0137fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f013815bbf0 0x7f0138151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:10.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.414+0000 7f0137fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f013815bbf0 0x7f0138151e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59860/0 (socket says 172.21.15.88:59860) 2024-09-16T01:45:10.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.414+0000 7f0137fff640 1 -- 172.21.15.88:0/343322784 learned_addr learned my addr 172.21.15.88:0/343322784 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:10.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.414+0000 7f0137fff640 1 -- 172.21.15.88:0/343322784 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0138154e70 msgr2=0x7f0138151920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:10.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.414+0000 7f0137fff640 1 --2- 172.21.15.88:0/343322784 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0138154e70 0x7f0138151920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.414+0000 7f0137fff640 1 -- 172.21.15.88:0/343322784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01380ccf60 msgr2=0x7f0138148de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:10.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.414+0000 7f01377fe640 1 --2- 172.21.15.88:0/343322784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01380ccf60 0x7f0138148de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:10.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.414+0000 7f0137fff640 1 --2- 172.21.15.88:0/343322784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01380ccf60 0x7f0138148de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.414+0000 7f0137fff640 1 -- 172.21.15.88:0/343322784 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f013814a2c0 con 0x7f013815bbf0 2024-09-16T01:45:10.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.414+0000 7f01377fe640 1 --2- 172.21.15.88:0/343322784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01380ccf60 0x7f0138148de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:45:10.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.414+0000 7f0137fff640 1 --2- 172.21.15.88:0/343322784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f013815bbf0 0x7f0138151e60 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f013000c7a0 tx=0x7f013000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:10.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.414+0000 7f011ffff640 1 -- 172.21.15.88:0/343322784 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0130018070 con 0x7f013815bbf0 2024-09-16T01:45:10.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.415+0000 7f011ffff640 1 -- 172.21.15.88:0/343322784 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f013000a640 con 0x7f013815bbf0 2024-09-16T01:45:10.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.415+0000 7f011ffff640 1 -- 172.21.15.88:0/343322784 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013000aab0 con 0x7f013815bbf0 2024-09-16T01:45:10.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.415+0000 7f013cd9f640 1 -- 172.21.15.88:0/343322784 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f013814a5a0 con 0x7f013815bbf0 2024-09-16T01:45:10.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.415+0000 7f013cd9f640 1 -- 172.21.15.88:0/343322784 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f013814ab60 con 0x7f013815bbf0 2024-09-16T01:45:10.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.416+0000 7f013cd9f640 1 -- 172.21.15.88:0/343322784 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f013814f750 con 0x7f013815bbf0 2024-09-16T01:45:10.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.416+0000 7f011ffff640 1 -- 172.21.15.88:0/343322784 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f013001b070 con 0x7f013815bbf0 2024-09-16T01:45:10.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.417+0000 7f011ffff640 1 --2- 172.21.15.88:0/343322784 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f012406c2b0 0x7f012406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:10.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.417+0000 7f01377fe640 1 --2- 172.21.15.88:0/343322784 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f012406c2b0 0x7f012406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:10.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.417+0000 7f011ffff640 1 -- 172.21.15.88:0/343322784 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5795+0+0 (secure 0 0 0) 0x7f013008aad0 con 0x7f013815bbf0 2024-09-16T01:45:10.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.417+0000 7f01377fe640 1 --2- 172.21.15.88:0/343322784 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f012406c2b0 0x7f012406e770 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f0128035000 tx=0x7f012802f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:10.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.421+0000 7f011ffff640 1 -- 172.21.15.88:0/343322784 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f01300591e0 con 0x7f013815bbf0 2024-09-16T01:45:10.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.589+0000 7f013cd9f640 1 -- 172.21.15.88:0/343322784 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f01380cff80 con 0x7f013815bbf0 2024-09-16T01:45:10.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.590+0000 7f011ffff640 1 -- 172.21.15.88:0/343322784 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) ==== 74+0+16782 (secure 0 0 0) 0x7f0130058ae0 con 0x7f013815bbf0 2024-09-16T01:45:10.592 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:45:10.592 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":139,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:45:01.261760+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-09-16T01:44:50.167367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-09-16T01:44:50.816821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:6836/656828696":"2024-09-17T01:45:01.261737+0000","172.21.15.88:6834/656828696":"2024-09-17T01:45:01.261737+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:45:10.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.592+0000 7f013cd9f640 1 -- 172.21.15.88:0/343322784 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f012406c2b0 msgr2=0x7f012406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:10.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.592+0000 7f013cd9f640 1 --2- 172.21.15.88:0/343322784 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f012406c2b0 0x7f012406e770 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f0128035000 tx=0x7f012802f000 comp rx=0 tx=0).stop 2024-09-16T01:45:10.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.592+0000 7f013cd9f640 1 -- 172.21.15.88:0/343322784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f013815bbf0 msgr2=0x7f0138151e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:10.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.592+0000 7f013cd9f640 1 --2- 172.21.15.88:0/343322784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f013815bbf0 0x7f0138151e60 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f013000c7a0 tx=0x7f013000cc60 comp rx=0 tx=0).stop 2024-09-16T01:45:10.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.592+0000 7f013cd9f640 1 -- 172.21.15.88:0/343322784 shutdown_connections 2024-09-16T01:45:10.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.592+0000 7f013cd9f640 1 --2- 172.21.15.88:0/343322784 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f012406c2b0 0x7f012406e770 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.592+0000 7f013cd9f640 1 --2- 172.21.15.88:0/343322784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f013815bbf0 0x7f0138151e60 unknown :-1 s=CLOSED pgs=2423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.592+0000 7f013cd9f640 1 --2- 172.21.15.88:0/343322784 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0138154e70 0x7f0138151920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.592+0000 7f013cd9f640 1 --2- 172.21.15.88:0/343322784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01380ccf60 0x7f0138148de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.592+0000 7f013cd9f640 1 -- 172.21.15.88:0/343322784 >> 172.21.15.88:0/343322784 conn(0x7f0138062a20 msgr2=0x7f01380b12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:10.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.592+0000 7f013cd9f640 1 -- 172.21.15.88:0/343322784 shutdown_connections 2024-09-16T01:45:10.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.592+0000 7f013cd9f640 1 -- 172.21.15.88:0/343322784 wait complete. 2024-09-16T01:45:10.606 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-16T01:45:10.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.707+0000 7fbab7215640 1 Processor -- start 2024-09-16T01:45:10.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.707+0000 7fbab7215640 1 -- start start 2024-09-16T01:45:10.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.708+0000 7fbab7215640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbab00cf1f0 0x7fbab00b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:10.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.708+0000 7fbab7215640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbab00b6720 0x7fbab00d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:10.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.708+0000 7fbab7215640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbab00cf860 con 0x7fbab015e7a0 2024-09-16T01:45:10.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.708+0000 7fbab7215640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fbab00b6b90 con 0x7fbab00cf1f0 2024-09-16T01:45:10.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.708+0000 7fbab7215640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbab00d2ca0 con 0x7fbab00b6720 2024-09-16T01:45:10.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.708+0000 7fbaaffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbab00cf1f0 0x7fbab00b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:10.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.708+0000 7fbaaffff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fbab00cf1f0 0x7fbab00b61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59876/0 (socket says 172.21.15.88:59876) 2024-09-16T01:45:10.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.708+0000 7fbaaffff640 1 -- 172.21.15.88:0/2245174147 learned_addr learned my addr 172.21.15.88:0/2245174147 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:10.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.708+0000 7fbab57b2640 1 --2- 172.21.15.88:0/2245174147 >> v2:172.21.15.154:3300/0 conn(0x7fbab00b6720 0x7fbab00d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:10.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.708+0000 7fbaaffff640 1 -- 172.21.15.88:0/2245174147 >> v2:172.21.15.154:3300/0 conn(0x7fbab00b6720 msgr2=0x7fbab00d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:10.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.708+0000 7fbaaffff640 1 --2- 172.21.15.88:0/2245174147 >> v2:172.21.15.154:3300/0 conn(0x7fbab00b6720 0x7fbab00d3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.708+0000 7fbaaffff640 1 -- 172.21.15.88:0/2245174147 >> v1:172.21.15.154:6789/0 conn(0x7fbab015e7a0 legacy=0x7fbab006caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:10.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.708+0000 7fbaaffff640 1 -- 172.21.15.88:0/2245174147 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbab00cfb40 con 0x7fbab00cf1f0 2024-09-16T01:45:10.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.709+0000 7fbaaffff640 1 --2- 172.21.15.88:0/2245174147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbab00cf1f0 0x7fbab00b61e0 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7fbab008d930 tx=0x7fba9c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65b83752ab926fe server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:10.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.709+0000 7fbaaf7fe640 1 -- 172.21.15.88:0/2245174147 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c037070 con 0x7fbab00cf1f0 2024-09-16T01:45:10.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.709+0000 7fbaaf7fe640 1 -- 172.21.15.88:0/2245174147 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba9c032440 con 0x7fbab00cf1f0 2024-09-16T01:45:10.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.709+0000 7fbaaf7fe640 1 -- 172.21.15.88:0/2245174147 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c032810 con 0x7fbab00cf1f0 2024-09-16T01:45:10.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.709+0000 7fbab7215640 1 -- 172.21.15.88:0/2245174147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbab00cf1f0 msgr2=0x7fbab00b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:10.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.709+0000 7fbab7215640 1 --2- 172.21.15.88:0/2245174147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbab00cf1f0 0x7fbab00b61e0 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7fbab008d930 tx=0x7fba9c02a1e0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.709+0000 7fbab7215640 1 -- 172.21.15.88:0/2245174147 shutdown_connections 2024-09-16T01:45:10.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.709+0000 7fbab7215640 1 --2- 172.21.15.88:0/2245174147 >> v2:172.21.15.154:3300/0 conn(0x7fbab00b6720 0x7fbab00d3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.709+0000 7fbab7215640 1 --2- 172.21.15.88:0/2245174147 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbab00cf1f0 0x7fbab00b61e0 unknown :-1 s=CLOSED pgs=2424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.710+0000 7fbab7215640 1 -- 172.21.15.88:0/2245174147 >> 172.21.15.88:0/2245174147 conn(0x7fbab00d11e0 msgr2=0x7fbab00d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:10.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.710+0000 7fbab7215640 1 -- 172.21.15.88:0/2245174147 shutdown_connections 2024-09-16T01:45:10.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.710+0000 7fbab7215640 1 -- 172.21.15.88:0/2245174147 wait complete. 2024-09-16T01:45:10.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.711+0000 7fbab7215640 1 Processor -- start 2024-09-16T01:45:10.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.711+0000 7fbab7215640 1 -- start start 2024-09-16T01:45:10.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.712+0000 7fbab7215640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbab00b6720 0x7fbab0156280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:10.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.712+0000 7fbab7215640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbab00cf1f0 0x7fbab014d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:10.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.712+0000 7fbab7215640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbab015e7a0 0x7fbab00cab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:10.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.712+0000 7fbab7215640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fbab00b7a00 con 0x7fbab00b6720 2024-09-16T01:45:10.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.712+0000 7fbab7215640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbab00d2ef0 con 0x7fbab015e7a0 2024-09-16T01:45:10.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.712+0000 7fbab7215640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbab015eb80 con 0x7fbab00cf1f0 2024-09-16T01:45:10.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.712+0000 7fbab4fb1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbab00b6720 0x7fbab0156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:10.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.712+0000 7fbab4fb1640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbab00b6720 0x7fbab0156280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59890/0 (socket says 172.21.15.88:59890) 2024-09-16T01:45:10.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.712+0000 7fbab4fb1640 1 -- 172.21.15.88:0/465323740 learned_addr learned my addr 172.21.15.88:0/465323740 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:10.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.712+0000 7fbab57b2640 1 --2- 172.21.15.88:0/465323740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbab015e7a0 0x7fbab00cab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:10.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.712+0000 7fbaaffff640 1 --2- 172.21.15.88:0/465323740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbab00cf1f0 0x7fbab014d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:10.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.712+0000 7fbab4fb1640 1 -- 172.21.15.88:0/465323740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbab00cf1f0 msgr2=0x7fbab014d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:10.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.712+0000 7fbab4fb1640 1 --2- 172.21.15.88:0/465323740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbab00cf1f0 0x7fbab014d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.712+0000 7fbab4fb1640 1 -- 172.21.15.88:0/465323740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbab015e7a0 msgr2=0x7fbab00cab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:10.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.712+0000 7fbab4fb1640 1 --2- 172.21.15.88:0/465323740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbab015e7a0 0x7fbab00cab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.712+0000 7fbab4fb1640 1 -- 172.21.15.88:0/465323740 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbab014df30 con 0x7fbab00b6720 2024-09-16T01:45:10.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.712+0000 7fbab57b2640 1 --2- 172.21.15.88:0/465323740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbab015e7a0 0x7fbab00cab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:45:10.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.712+0000 7fbab4fb1640 1 --2- 172.21.15.88:0/465323740 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbab00b6720 0x7fbab0156280 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7fbaa400c400 tx=0x7fbaa400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:10.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.712+0000 7fbaad7fa640 1 -- 172.21.15.88:0/465323740 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa40052d0 con 0x7fbab00b6720 2024-09-16T01:45:10.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.712+0000 7fbaad7fa640 1 -- 172.21.15.88:0/465323740 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaa4005490 con 0x7fbab00b6720 2024-09-16T01:45:10.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.712+0000 7fbab7215640 1 -- 172.21.15.88:0/465323740 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbab00c9a90 con 0x7fbab00b6720 2024-09-16T01:45:10.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.712+0000 7fbaad7fa640 1 -- 172.21.15.88:0/465323740 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa4007260 con 0x7fbab00b6720 2024-09-16T01:45:10.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.712+0000 7fbab7215640 1 -- 172.21.15.88:0/465323740 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbab00ca050 con 0x7fbab00b6720 2024-09-16T01:45:10.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.713+0000 7fbab7215640 1 -- 172.21.15.88:0/465323740 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbab00b61e0 con 0x7fbab00b6720 2024-09-16T01:45:10.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.714+0000 7fbaad7fa640 1 -- 172.21.15.88:0/465323740 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fbaa4007400 con 0x7fbab00b6720 2024-09-16T01:45:10.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.714+0000 7fbaad7fa640 1 --2- 172.21.15.88:0/465323740 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba9406c2e0 0x7fba9406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:10.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.714+0000 7fbaad7fa640 1 -- 172.21.15.88:0/465323740 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5795+0+0 (secure 0 0 0) 0x7fbaa408a9d0 con 0x7fbab00b6720 2024-09-16T01:45:10.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.714+0000 7fbaaffff640 1 --2- 172.21.15.88:0/465323740 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba9406c2e0 0x7fba9406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:10.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.715+0000 7fbaaffff640 1 --2- 172.21.15.88:0/465323740 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba9406c2e0 0x7fba9406e7a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fba9c034000 tx=0x7fba9c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:10.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.719+0000 7fbaad7fa640 1 -- 172.21.15.88:0/465323740 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fbaa40590e0 con 0x7fbab00b6720 2024-09-16T01:45:10.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.918+0000 7fbab7215640 1 -- 172.21.15.88:0/465323740 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fbab00cefd0 con 0x7fbab00b6720 2024-09-16T01:45:10.920 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-09-16T01:45:10.919+0000 7f2138b2b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-16T01:45:10.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.921+0000 7fbaad7fa640 1 -- 172.21.15.88:0/465323740 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v333) ==== 161+0+0 (secure 0 0 0) 0x7fbaa40589e0 con 0x7fbab00b6720 2024-09-16T01:45:10.922 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-16T01:45:10.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.924+0000 7fbab7215640 1 -- 172.21.15.88:0/465323740 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba9406c2e0 msgr2=0x7fba9406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:10.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.924+0000 7fbab7215640 1 --2- 172.21.15.88:0/465323740 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba9406c2e0 0x7fba9406e7a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fba9c034000 tx=0x7fba9c02e000 comp rx=0 tx=0).stop 2024-09-16T01:45:10.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.924+0000 7fbab7215640 1 -- 172.21.15.88:0/465323740 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbab00b6720 msgr2=0x7fbab0156280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:10.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.924+0000 7fbab7215640 1 --2- 172.21.15.88:0/465323740 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbab00b6720 0x7fbab0156280 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7fbaa400c400 tx=0x7fbaa400c8c0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.924+0000 7fbab7215640 1 -- 172.21.15.88:0/465323740 shutdown_connections 2024-09-16T01:45:10.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.924+0000 7fbab7215640 1 --2- 172.21.15.88:0/465323740 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fba9406c2e0 0x7fba9406e7a0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.924+0000 7fbab7215640 1 --2- 172.21.15.88:0/465323740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbab015e7a0 0x7fbab00cab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.924+0000 7fbab7215640 1 --2- 172.21.15.88:0/465323740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbab00cf1f0 0x7fbab014d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.924+0000 7fbab7215640 1 --2- 172.21.15.88:0/465323740 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbab00b6720 0x7fbab0156280 unknown :-1 s=CLOSED pgs=2425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:10.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.924+0000 7fbab7215640 1 -- 172.21.15.88:0/465323740 >> 172.21.15.88:0/465323740 conn(0x7fbab00d11e0 msgr2=0x7fbab015d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:10.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.924+0000 7fbab7215640 1 -- 172.21.15.88:0/465323740 shutdown_connections 2024-09-16T01:45:10.926 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:10.924+0000 7fbab7215640 1 -- 172.21.15.88:0/465323740 wait complete. 2024-09-16T01:45:10.927 INFO:tasks.ceph.mds.a.smithi088.stderr: -357> 2024-09-16T01:45:07.273+0000 7f7534874640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-16T01:45:10.937 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-16T01:45:10.951 INFO:tasks.ceph.mds.a.smithi088.stdout:starting mds.a at 2024-09-16T01:45:11.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.036+0000 7fb960877640 1 Processor -- start 2024-09-16T01:45:11.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.036+0000 7fb960877640 1 -- start start 2024-09-16T01:45:11.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.036+0000 7fb960877640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb95c0cf1f0 0x7fb95c0b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:11.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.036+0000 7fb960877640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb95c0b6720 0x7fb95c0d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:11.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.037+0000 7fb960877640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb95c0cf860 con 0x7fb95c15e7a0 2024-09-16T01:45:11.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.037+0000 7fb960877640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb95c0d2ca0 con 0x7fb95c0cf1f0 2024-09-16T01:45:11.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.037+0000 7fb960877640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb95c0cfb40 con 0x7fb95c0b6720 2024-09-16T01:45:11.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.037+0000 7fb959d9b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb95c0cf1f0 0x7fb95c0b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:11.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.037+0000 7fb959d9b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb95c0cf1f0 0x7fb95c0b61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:53538/0 (socket says 172.21.15.88:53538) 2024-09-16T01:45:11.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.037+0000 7fb95ad9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb95c0b6720 0x7fb95c0d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:11.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.037+0000 7fb959d9b640 1 -- 172.21.15.88:0/3981918451 learned_addr learned my addr 172.21.15.88:0/3981918451 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:11.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.037+0000 7fb959d9b640 1 -- 172.21.15.88:0/3981918451 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb95c0b6720 msgr2=0x7fb95c0d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:11.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.037+0000 7fb959d9b640 1 --2- 172.21.15.88:0/3981918451 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb95c0b6720 0x7fb95c0d3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:11.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.037+0000 7fb959d9b640 1 -- 172.21.15.88:0/3981918451 >> v1:172.21.15.154:6789/0 conn(0x7fb95c15e7a0 legacy=0x7fb95c06caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:11.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.037+0000 7fb959d9b640 1 -- 172.21.15.88:0/3981918451 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb95c0cfe20 con 0x7fb95c0cf1f0 2024-09-16T01:45:11.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.038+0000 7fb959d9b640 1 --2- 172.21.15.88:0/3981918451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb95c0cf1f0 0x7fb95c0b61e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fb944009870 tx=0x7fb94402a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c2b03a8e7505176 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:11.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.038+0000 7fb95959a640 1 -- 172.21.15.88:0/3981918451 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb944037070 con 0x7fb95c0cf1f0 2024-09-16T01:45:11.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.038+0000 7fb95959a640 1 -- 172.21.15.88:0/3981918451 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb944032440 con 0x7fb95c0cf1f0 2024-09-16T01:45:11.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.038+0000 7fb95959a640 1 -- 172.21.15.88:0/3981918451 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb944032810 con 0x7fb95c0cf1f0 2024-09-16T01:45:11.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.039+0000 7fb960877640 1 -- 172.21.15.88:0/3981918451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb95c0cf1f0 msgr2=0x7fb95c0b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:11.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.039+0000 7fb960877640 1 --2- 172.21.15.88:0/3981918451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb95c0cf1f0 0x7fb95c0b61e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fb944009870 tx=0x7fb94402a1c0 comp rx=0 tx=0).stop 2024-09-16T01:45:11.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.039+0000 7fb960877640 1 -- 172.21.15.88:0/3981918451 shutdown_connections 2024-09-16T01:45:11.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.039+0000 7fb960877640 1 --2- 172.21.15.88:0/3981918451 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb95c0b6720 0x7fb95c0d3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:11.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.039+0000 7fb960877640 1 --2- 172.21.15.88:0/3981918451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb95c0cf1f0 0x7fb95c0b61e0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:11.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.039+0000 7fb960877640 1 -- 172.21.15.88:0/3981918451 >> 172.21.15.88:0/3981918451 conn(0x7fb95c0d11e0 msgr2=0x7fb95c0d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:11.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.039+0000 7fb960877640 1 -- 172.21.15.88:0/3981918451 shutdown_connections 2024-09-16T01:45:11.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.039+0000 7fb960877640 1 -- 172.21.15.88:0/3981918451 wait complete. 2024-09-16T01:45:11.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.040+0000 7fb960877640 1 Processor -- start 2024-09-16T01:45:11.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.040+0000 7fb960877640 1 -- start start 2024-09-16T01:45:11.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.041+0000 7fb960877640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb95c0b6720 0x7fb95c1f4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:11.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.041+0000 7fb960877640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb95c0cf1f0 0x7fb95c159c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:11.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.041+0000 7fb960877640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb95c15e7a0 0x7fb95c15a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:11.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.041+0000 7fb960877640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fb95c15eb80 con 0x7fb95c0cf1f0 2024-09-16T01:45:11.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.041+0000 7fb960877640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb95c0b7a00 con 0x7fb95c15e7a0 2024-09-16T01:45:11.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.041+0000 7fb960877640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb95c15ed00 con 0x7fb95c0b6720 2024-09-16T01:45:11.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.041+0000 7fb959d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb95c0cf1f0 0x7fb95c159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:11.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.041+0000 7fb959d9b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb95c0cf1f0 0x7fb95c159c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59918/0 (socket says 172.21.15.88:59918) 2024-09-16T01:45:11.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.041+0000 7fb959d9b640 1 -- 172.21.15.88:0/166129453 learned_addr learned my addr 172.21.15.88:0/166129453 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:11.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.041+0000 7fb95ad9d640 1 --2- 172.21.15.88:0/166129453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb95c15e7a0 0x7fb95c15a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:11.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.041+0000 7fb95a59c640 1 --2- 172.21.15.88:0/166129453 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb95c0b6720 0x7fb95c1f4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:11.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.041+0000 7fb959d9b640 1 -- 172.21.15.88:0/166129453 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb95c0b6720 msgr2=0x7fb95c1f4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:11.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.041+0000 7fb959d9b640 1 --2- 172.21.15.88:0/166129453 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb95c0b6720 0x7fb95c1f4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:11.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.041+0000 7fb959d9b640 1 -- 172.21.15.88:0/166129453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb95c15e7a0 msgr2=0x7fb95c15a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:11.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.041+0000 7fb959d9b640 1 --2- 172.21.15.88:0/166129453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb95c15e7a0 0x7fb95c15a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:11.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.041+0000 7fb959d9b640 1 -- 172.21.15.88:0/166129453 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb95c1520b0 con 0x7fb95c0cf1f0 2024-09-16T01:45:11.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.041+0000 7fb959d9b640 1 --2- 172.21.15.88:0/166129453 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb95c0cf1f0 0x7fb95c159c00 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7fb944000c00 tx=0x7fb9440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:11.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.041+0000 7fb952ffd640 1 -- 172.21.15.88:0/166129453 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb944037040 con 0x7fb95c0cf1f0 2024-09-16T01:45:11.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.041+0000 7fb952ffd640 1 -- 172.21.15.88:0/166129453 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb944042070 con 0x7fb95c0cf1f0 2024-09-16T01:45:11.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.041+0000 7fb960877640 1 -- 172.21.15.88:0/166129453 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb95c152390 con 0x7fb95c0cf1f0 2024-09-16T01:45:11.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.041+0000 7fb952ffd640 1 -- 172.21.15.88:0/166129453 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb944030860 con 0x7fb95c0cf1f0 2024-09-16T01:45:11.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.042+0000 7fb960877640 1 -- 172.21.15.88:0/166129453 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb95c0b7dd0 con 0x7fb95c0cf1f0 2024-09-16T01:45:11.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.042+0000 7fb960877640 1 -- 172.21.15.88:0/166129453 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb95c0b61e0 con 0x7fb95c0cf1f0 2024-09-16T01:45:11.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.043+0000 7fb952ffd640 1 -- 172.21.15.88:0/166129453 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fb94402ad10 con 0x7fb95c0cf1f0 2024-09-16T01:45:11.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.043+0000 7fb952ffd640 1 --2- 172.21.15.88:0/166129453 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb93c06c2b0 0x7fb93c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:11.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.043+0000 7fb952ffd640 1 -- 172.21.15.88:0/166129453 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 5881+0+0 (secure 0 0 0) 0x7fb9440abab0 con 0x7fb95c0cf1f0 2024-09-16T01:45:11.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.044+0000 7fb95a59c640 1 --2- 172.21.15.88:0/166129453 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb93c06c2b0 0x7fb93c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:11.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.044+0000 7fb95a59c640 1 --2- 172.21.15.88:0/166129453 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb93c06c2b0 0x7fb93c06e770 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fb954001ca0 tx=0x7fb954009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:11.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.048+0000 7fb952ffd640 1 -- 172.21.15.88:0/166129453 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fb94407a170 con 0x7fb95c0cf1f0 2024-09-16T01:45:11.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.247+0000 7fb960877640 1 -- 172.21.15.88:0/166129453 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fb95c0d2a80 con 0x7fb95c0cf1f0 2024-09-16T01:45:11.924 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.922+0000 7fb952ffd640 1 -- 172.21.15.88:0/166129453 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v334) ==== 104+0+0 (secure 0 0 0) 0x7fb944079a70 con 0x7fb95c0cf1f0 2024-09-16T01:45:11.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.925+0000 7fb960877640 1 -- 172.21.15.88:0/166129453 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb93c06c2b0 msgr2=0x7fb93c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:11.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.925+0000 7fb960877640 1 --2- 172.21.15.88:0/166129453 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb93c06c2b0 0x7fb93c06e770 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fb954001ca0 tx=0x7fb954009040 comp rx=0 tx=0).stop 2024-09-16T01:45:11.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.925+0000 7fb960877640 1 -- 172.21.15.88:0/166129453 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb95c0cf1f0 msgr2=0x7fb95c159c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:11.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.925+0000 7fb960877640 1 --2- 172.21.15.88:0/166129453 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb95c0cf1f0 0x7fb95c159c00 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7fb944000c00 tx=0x7fb9440046c0 comp rx=0 tx=0).stop 2024-09-16T01:45:11.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.926+0000 7fb960877640 1 -- 172.21.15.88:0/166129453 shutdown_connections 2024-09-16T01:45:11.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.926+0000 7fb960877640 1 --2- 172.21.15.88:0/166129453 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fb93c06c2b0 0x7fb93c06e770 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:11.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.926+0000 7fb960877640 1 --2- 172.21.15.88:0/166129453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb95c15e7a0 0x7fb95c15a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:11.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.926+0000 7fb960877640 1 --2- 172.21.15.88:0/166129453 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb95c0cf1f0 0x7fb95c159c00 unknown :-1 s=CLOSED pgs=2429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:11.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.926+0000 7fb960877640 1 --2- 172.21.15.88:0/166129453 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb95c0b6720 0x7fb95c1f4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:11.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.926+0000 7fb960877640 1 -- 172.21.15.88:0/166129453 >> 172.21.15.88:0/166129453 conn(0x7fb95c0d11e0 msgr2=0x7fb95c15d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:11.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.926+0000 7fb960877640 1 -- 172.21.15.88:0/166129453 shutdown_connections 2024-09-16T01:45:11.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:11.926+0000 7fb960877640 1 -- 172.21.15.88:0/166129453 wait complete. 2024-09-16T01:45:11.939 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-16T01:45:12.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.083+0000 7f9f8ba77640 1 Processor -- start 2024-09-16T01:45:12.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.083+0000 7f9f8ba77640 1 -- start start 2024-09-16T01:45:12.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.084+0000 7f9f8ba77640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9f8415db30 0x7f9f84167140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:12.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.084+0000 7f9f8ba77640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f840cd9b0 con 0x7f9f840d0760 2024-09-16T01:45:12.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.084+0000 7f9f8ba77640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f840cdc90 con 0x7f9f840b4e60 2024-09-16T01:45:12.085 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.084+0000 7f9f8ba77640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f9f840ce5c0 con 0x7f9f8415db30 2024-09-16T01:45:12.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.084+0000 7f9f8a014640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f9f8415db30 0x7f9f84167140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:12.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.084+0000 7f9f89012640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9f840d0760 0x7f9f840b5240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59942/0 (socket says 172.21.15.88:59942) 2024-09-16T01:45:12.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.084+0000 7f9f89012640 1 -- 172.21.15.88:0/3404632058 learned_addr learned my addr 172.21.15.88:0/3404632058 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:12.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.084+0000 7f9f8a014640 1 -- 172.21.15.88:0/3404632058 >> v1:172.21.15.154:6789/0 conn(0x7f9f840b4e60 legacy=0x7f9f840b5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:12.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.084+0000 7f9f8a014640 1 -- 172.21.15.88:0/3404632058 >> v1:172.21.15.88:6789/0 conn(0x7f9f840d0760 legacy=0x7f9f840b5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:12.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.084+0000 7f9f8a014640 1 -- 172.21.15.88:0/3404632058 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f840cdf70 con 0x7f9f8415db30 2024-09-16T01:45:12.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.085+0000 7f9f8a014640 1 --2- 172.21.15.88:0/3404632058 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f8415db30 0x7f9f84167140 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c009d90 tx=0x7f9f7c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1ebed02c87e21a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:12.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.085+0000 7f9f88811640 1 -- 172.21.15.88:0/3404632058 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c007740 con 0x7f9f8415db30 2024-09-16T01:45:12.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.085+0000 7f9f88811640 1 -- 172.21.15.88:0/3404632058 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f7c0078e0 con 0x7f9f8415db30 2024-09-16T01:45:12.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.085+0000 7f9f88811640 1 -- 172.21.15.88:0/3404632058 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c007cb0 con 0x7f9f8415db30 2024-09-16T01:45:12.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.085+0000 7f9f8ba77640 1 -- 172.21.15.88:0/3404632058 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f8415db30 msgr2=0x7f9f84167140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:12.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.085+0000 7f9f8ba77640 1 --2- 172.21.15.88:0/3404632058 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f8415db30 0x7f9f84167140 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c009d90 tx=0x7f9f7c02a6a0 comp rx=0 tx=0).stop 2024-09-16T01:45:12.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.086+0000 7f9f8ba77640 1 -- 172.21.15.88:0/3404632058 shutdown_connections 2024-09-16T01:45:12.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.086+0000 7f9f8ba77640 1 --2- 172.21.15.88:0/3404632058 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f8415db30 0x7f9f84167140 unknown :-1 s=CLOSED pgs=2431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:12.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.086+0000 7f9f8ba77640 1 -- 172.21.15.88:0/3404632058 >> 172.21.15.88:0/3404632058 conn(0x7f9f8406cb50 msgr2=0x7f9f8406cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:12.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.086+0000 7f9f8ba77640 1 -- 172.21.15.88:0/3404632058 shutdown_connections 2024-09-16T01:45:12.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.086+0000 7f9f8ba77640 1 -- 172.21.15.88:0/3404632058 wait complete. 2024-09-16T01:45:12.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.087+0000 7f9f8ba77640 1 Processor -- start 2024-09-16T01:45:12.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.087+0000 7f9f8ba77640 1 -- start start 2024-09-16T01:45:12.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.087+0000 7f9f8ba77640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f840b4e60 0x7f9f840aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:12.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.087+0000 7f9f8ba77640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f840d0760 0x7f9f8414b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:12.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.087+0000 7f9f8ba77640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f8415db30 0x7f9f84155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:12.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.087+0000 7f9f8ba77640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8415fd80 con 0x7f9f840b4e60 2024-09-16T01:45:12.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.087+0000 7f9f8ba77640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8415fc00 con 0x7f9f840d0760 2024-09-16T01:45:12.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.087+0000 7f9f8ba77640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9f84167680 con 0x7f9f8415db30 2024-09-16T01:45:12.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.088+0000 7f9f89813640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f840b4e60 0x7f9f840aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:12.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.088+0000 7f9f89813640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f840b4e60 0x7f9f840aecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59934/0 (socket says 172.21.15.88:59934) 2024-09-16T01:45:12.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.088+0000 7f9f89813640 1 -- 172.21.15.88:0/3879534915 learned_addr learned my addr 172.21.15.88:0/3879534915 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:12.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.088+0000 7f9f89012640 1 --2- 172.21.15.88:0/3879534915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f840d0760 0x7f9f8414b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:12.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.088+0000 7f9f8a014640 1 --2- 172.21.15.88:0/3879534915 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f8415db30 0x7f9f84155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:12.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.088+0000 7f9f89813640 1 -- 172.21.15.88:0/3879534915 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f8415db30 msgr2=0x7f9f84155920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:12.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.088+0000 7f9f89813640 1 --2- 172.21.15.88:0/3879534915 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f8415db30 0x7f9f84155920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:12.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.088+0000 7f9f89813640 1 -- 172.21.15.88:0/3879534915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f840d0760 msgr2=0x7f9f8414b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:12.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.088+0000 7f9f89813640 1 --2- 172.21.15.88:0/3879534915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f840d0760 0x7f9f8414b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:12.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.088+0000 7f9f89813640 1 -- 172.21.15.88:0/3879534915 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f8414bee0 con 0x7f9f840b4e60 2024-09-16T01:45:12.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.088+0000 7f9f89012640 1 --2- 172.21.15.88:0/3879534915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f840d0760 0x7f9f8414b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:45:12.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.088+0000 7f9f89813640 1 --2- 172.21.15.88:0/3879534915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f840b4e60 0x7f9f840aecb0 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f9f8000b3f0 tx=0x7f9f8000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:12.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.088+0000 7f9f7a7fc640 1 -- 172.21.15.88:0/3879534915 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f80015020 con 0x7f9f840b4e60 2024-09-16T01:45:12.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.088+0000 7f9f7a7fc640 1 -- 172.21.15.88:0/3879534915 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f80011830 con 0x7f9f840b4e60 2024-09-16T01:45:12.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.088+0000 7f9f8ba77640 1 -- 172.21.15.88:0/3879534915 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f84150610 con 0x7f9f840b4e60 2024-09-16T01:45:12.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.088+0000 7f9f7a7fc640 1 -- 172.21.15.88:0/3879534915 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f80011c20 con 0x7f9f840b4e60 2024-09-16T01:45:12.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.088+0000 7f9f8ba77640 1 -- 172.21.15.88:0/3879534915 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f84150bd0 con 0x7f9f840b4e60 2024-09-16T01:45:12.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.089+0000 7f9f8ba77640 1 -- 172.21.15.88:0/3879534915 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f840b5c00 con 0x7f9f840b4e60 2024-09-16T01:45:12.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.090+0000 7f9f7a7fc640 1 -- 172.21.15.88:0/3879534915 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9f80018020 con 0x7f9f840b4e60 2024-09-16T01:45:12.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.090+0000 7f9f7a7fc640 1 --2- 172.21.15.88:0/3879534915 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9f6806c2b0 0x7f9f6806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:12.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.090+0000 7f9f89012640 1 --2- 172.21.15.88:0/3879534915 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9f6806c2b0 0x7f9f6806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:12.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.090+0000 7f9f7a7fc640 1 -- 172.21.15.88:0/3879534915 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 5881+0+0 (secure 0 0 0) 0x7f9f8008b0f0 con 0x7f9f840b4e60 2024-09-16T01:45:12.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.091+0000 7f9f89012640 1 --2- 172.21.15.88:0/3879534915 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9f6806c2b0 0x7f9f6806e770 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f9f8414fc10 tx=0x7f9f74009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:12.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.094+0000 7f9f7a7fc640 1 -- 172.21.15.88:0/3879534915 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f9f8001f030 con 0x7f9f840b4e60 2024-09-16T01:45:12.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.261+0000 7f9f8ba77640 1 -- 172.21.15.88:0/3879534915 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9f8414c880 con 0x7f9f840b4e60 2024-09-16T01:45:12.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.924+0000 7f9f7a7fc640 1 -- 172.21.15.88:0/3879534915 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v141) ==== 181+0+0 (secure 0 0 0) 0x7f9f800172c0 con 0x7f9f840b4e60 2024-09-16T01:45:12.925 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_metadata' removed 2024-09-16T01:45:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.927+0000 7f9f8ba77640 1 -- 172.21.15.88:0/3879534915 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9f6806c2b0 msgr2=0x7f9f6806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.927+0000 7f9f8ba77640 1 --2- 172.21.15.88:0/3879534915 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9f6806c2b0 0x7f9f6806e770 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f9f8414fc10 tx=0x7f9f74009040 comp rx=0 tx=0).stop 2024-09-16T01:45:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.927+0000 7f9f8ba77640 1 -- 172.21.15.88:0/3879534915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f840b4e60 msgr2=0x7f9f840aecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.927+0000 7f9f8ba77640 1 --2- 172.21.15.88:0/3879534915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f840b4e60 0x7f9f840aecb0 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f9f8000b3f0 tx=0x7f9f8000b8b0 comp rx=0 tx=0).stop 2024-09-16T01:45:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.927+0000 7f9f8ba77640 1 -- 172.21.15.88:0/3879534915 shutdown_connections 2024-09-16T01:45:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.927+0000 7f9f8ba77640 1 --2- 172.21.15.88:0/3879534915 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f9f6806c2b0 0x7f9f6806e770 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.927+0000 7f9f8ba77640 1 --2- 172.21.15.88:0/3879534915 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f8415db30 0x7f9f84155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.927+0000 7f9f8ba77640 1 --2- 172.21.15.88:0/3879534915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f840d0760 0x7f9f8414b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.927+0000 7f9f8ba77640 1 --2- 172.21.15.88:0/3879534915 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f840b4e60 0x7f9f840aecb0 unknown :-1 s=CLOSED pgs=2432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:12.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.927+0000 7f9f8ba77640 1 -- 172.21.15.88:0/3879534915 >> 172.21.15.88:0/3879534915 conn(0x7f9f8406cb50 msgr2=0x7f9f8406cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:12.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.928+0000 7f9f8ba77640 1 -- 172.21.15.88:0/3879534915 shutdown_connections 2024-09-16T01:45:12.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:12.928+0000 7f9f8ba77640 1 -- 172.21.15.88:0/3879534915 wait complete. 2024-09-16T01:45:12.944 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-16T01:45:13.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.050+0000 7fca84050640 1 Processor -- start 2024-09-16T01:45:13.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.050+0000 7fca84050640 1 -- start start 2024-09-16T01:45:13.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.051+0000 7fca84050640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca7c15a050 0x7fca7c15d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:13.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.051+0000 7fca84050640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca7c15a430 con 0x7fca7c0731c0 2024-09-16T01:45:13.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.051+0000 7fca84050640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca7c160df0 con 0x7fca7c15ee90 2024-09-16T01:45:13.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.051+0000 7fca84050640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fca7c15d9a0 con 0x7fca7c15a050 2024-09-16T01:45:13.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.051+0000 7fca815eb640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fca7c0731c0 0x7fca7c15ffd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59952/0 (socket says 172.21.15.88:59952) 2024-09-16T01:45:13.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.051+0000 7fca815eb640 1 -- 172.21.15.88:0/1971381597 learned_addr learned my addr 172.21.15.88:0/1971381597 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:13.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.051+0000 7fca81dec640 1 --2- 172.21.15.88:0/1971381597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca7c15a050 0x7fca7c15d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:13.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.051+0000 7fca80dea640 1 -- 172.21.15.88:0/1971381597 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1641081812 0 0) 0x7fca7c15a430 con 0x7fca7c0731c0 2024-09-16T01:45:13.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.051+0000 7fca80dea640 1 -- 172.21.15.88:0/1971381597 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca68003520 con 0x7fca7c0731c0 2024-09-16T01:45:13.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.051+0000 7fca80dea640 1 -- 172.21.15.88:0/1971381597 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2160634067 0 0) 0x7fca68003520 con 0x7fca7c0731c0 2024-09-16T01:45:13.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.051+0000 7fca80dea640 1 -- 172.21.15.88:0/1971381597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca7c15a050 msgr2=0x7fca7c15d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:13.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.051+0000 7fca80dea640 1 --2- 172.21.15.88:0/1971381597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca7c15a050 0x7fca7c15d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:13.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.051+0000 7fca80dea640 1 -- 172.21.15.88:0/1971381597 >> v1:172.21.15.154:6789/0 conn(0x7fca7c15ee90 legacy=0x7fca7c1606e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:13.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.051+0000 7fca80dea640 1 -- 172.21.15.88:0/1971381597 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca7c0cd0a0 con 0x7fca7c0731c0 2024-09-16T01:45:13.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.051+0000 7fca80dea640 1 -- 172.21.15.88:0/1971381597 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fca6c001670 con 0x7fca7c0731c0 2024-09-16T01:45:13.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.051+0000 7fca80dea640 1 -- 172.21.15.88:0/1971381597 >> v1:172.21.15.88:6789/0 conn(0x7fca7c0731c0 legacy=0x7fca7c15ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:13.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.051+0000 7fca80dea640 1 --2- 172.21.15.88:0/1971381597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca680039e0 0x7fca68003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:13.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.052+0000 7fca80dea640 1 --2- 172.21.15.88:0/1971381597 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca68004800 0x7fca68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:13.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.052+0000 7fca80dea640 1 --2- 172.21.15.88:0/1971381597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca7c15a050 0x7fca68005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:13.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.052+0000 7fca80dea640 1 -- 172.21.15.88:0/1971381597 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fca7c15d9a0 con 0x7fca68004800 2024-09-16T01:45:13.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.052+0000 7fca80dea640 1 -- 172.21.15.88:0/1971381597 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fca6800baf0 con 0x7fca680039e0 2024-09-16T01:45:13.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.052+0000 7fca80dea640 1 -- 172.21.15.88:0/1971381597 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fca6800bcf0 con 0x7fca7c15a050 2024-09-16T01:45:13.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.052+0000 7fca815eb640 1 --2- 172.21.15.88:0/1971381597 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca68004800 0x7fca68004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:13.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.052+0000 7fca81dec640 1 --2- 172.21.15.88:0/1971381597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca680039e0 0x7fca68003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:13.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.052+0000 7fca825ed640 1 --2- 172.21.15.88:0/1971381597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca7c15a050 0x7fca68005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:13.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.052+0000 7fca815eb640 1 -- 172.21.15.88:0/1971381597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca7c15a050 msgr2=0x7fca68005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:13.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.052+0000 7fca815eb640 1 --2- 172.21.15.88:0/1971381597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca7c15a050 0x7fca68005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:13.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.052+0000 7fca815eb640 1 -- 172.21.15.88:0/1971381597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca680039e0 msgr2=0x7fca68003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:13.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.052+0000 7fca815eb640 1 --2- 172.21.15.88:0/1971381597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca680039e0 0x7fca68003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:13.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.052+0000 7fca815eb640 1 -- 172.21.15.88:0/1971381597 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca68003520 con 0x7fca68004800 2024-09-16T01:45:13.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.052+0000 7fca825ed640 1 --2- 172.21.15.88:0/1971381597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca7c15a050 0x7fca68005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:45:13.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.053+0000 7fca815eb640 1 --2- 172.21.15.88:0/1971381597 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca68004800 0x7fca68004c80 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7fca6c00c210 tx=0x7fca68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc0872276cdea81b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:13.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.053+0000 7fca80dea640 1 -- 172.21.15.88:0/1971381597 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6c007d10 con 0x7fca68004800 2024-09-16T01:45:13.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.053+0000 7fca80dea640 1 -- 172.21.15.88:0/1971381597 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca6c0058d0 con 0x7fca68004800 2024-09-16T01:45:13.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.053+0000 7fca80dea640 1 -- 172.21.15.88:0/1971381597 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6c005d00 con 0x7fca68004800 2024-09-16T01:45:13.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.053+0000 7fca84050640 1 -- 172.21.15.88:0/1971381597 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca68004800 msgr2=0x7fca68004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:13.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.053+0000 7fca84050640 1 --2- 172.21.15.88:0/1971381597 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca68004800 0x7fca68004c80 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7fca6c00c210 tx=0x7fca68005830 comp rx=0 tx=0).stop 2024-09-16T01:45:13.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.053+0000 7fca825ed640 1 -- 172.21.15.88:0/1971381597 reap_dead start 2024-09-16T01:45:13.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.053+0000 7fca84050640 1 -- 172.21.15.88:0/1971381597 shutdown_connections 2024-09-16T01:45:13.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.053+0000 7fca84050640 1 -- 172.21.15.88:0/1971381597 >> 172.21.15.88:0/1971381597 conn(0x7fca7c06d0f0 msgr2=0x7fca7c06d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:13.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.053+0000 7fca84050640 1 -- 172.21.15.88:0/1971381597 shutdown_connections 2024-09-16T01:45:13.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.053+0000 7fca84050640 1 -- 172.21.15.88:0/1971381597 wait complete. 2024-09-16T01:45:13.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.055+0000 7fca84050640 1 Processor -- start 2024-09-16T01:45:13.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.055+0000 7fca84050640 1 -- start start 2024-09-16T01:45:13.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.055+0000 7fca84050640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca68004800 0x7fca7c0d0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:13.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.055+0000 7fca84050640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca7c0ce910 0x7fca7c15ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:13.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.055+0000 7fca84050640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca7c15cf60 0x7fca7c13aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:13.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.055+0000 7fca84050640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fca7c0cd940 con 0x7fca7c0ce910 2024-09-16T01:45:13.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.055+0000 7fca84050640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fca7c0cd660 con 0x7fca68004800 2024-09-16T01:45:13.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.055+0000 7fca84050640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fca7c0cdc20 con 0x7fca7c15cf60 2024-09-16T01:45:13.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.055+0000 7fca815eb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca7c0ce910 0x7fca7c15ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:13.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.055+0000 7fca815eb640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca7c0ce910 0x7fca7c15ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59962/0 (socket says 172.21.15.88:59962) 2024-09-16T01:45:13.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.055+0000 7fca815eb640 1 -- 172.21.15.88:0/508765120 learned_addr learned my addr 172.21.15.88:0/508765120 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:13.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.056+0000 7fca81dec640 1 --2- 172.21.15.88:0/508765120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca68004800 0x7fca7c0d0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:13.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.056+0000 7fca825ed640 1 --2- 172.21.15.88:0/508765120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca7c15cf60 0x7fca7c13aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:13.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.056+0000 7fca815eb640 1 -- 172.21.15.88:0/508765120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca7c15cf60 msgr2=0x7fca7c13aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:13.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.056+0000 7fca815eb640 1 --2- 172.21.15.88:0/508765120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca7c15cf60 0x7fca7c13aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:13.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.056+0000 7fca815eb640 1 -- 172.21.15.88:0/508765120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca68004800 msgr2=0x7fca7c0d0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:13.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.056+0000 7fca815eb640 1 --2- 172.21.15.88:0/508765120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca68004800 0x7fca7c0d0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:13.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.056+0000 7fca815eb640 1 -- 172.21.15.88:0/508765120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca7c13ba00 con 0x7fca7c0ce910 2024-09-16T01:45:13.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.056+0000 7fca815eb640 1 --2- 172.21.15.88:0/508765120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca7c0ce910 0x7fca7c15ca20 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7fca6c00c210 tx=0x7fca7c0cd270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:13.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.056+0000 7fca727fc640 1 -- 172.21.15.88:0/508765120 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6c007d10 con 0x7fca7c0ce910 2024-09-16T01:45:13.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.056+0000 7fca727fc640 1 -- 172.21.15.88:0/508765120 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca6c005240 con 0x7fca7c0ce910 2024-09-16T01:45:13.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.056+0000 7fca727fc640 1 -- 172.21.15.88:0/508765120 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6c0055f0 con 0x7fca7c0ce910 2024-09-16T01:45:13.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.056+0000 7fca84050640 1 -- 172.21.15.88:0/508765120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca7c13b720 con 0x7fca7c0ce910 2024-09-16T01:45:13.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.056+0000 7fca84050640 1 -- 172.21.15.88:0/508765120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca7c0d0780 con 0x7fca7c0ce910 2024-09-16T01:45:13.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.057+0000 7fca84050640 1 -- 172.21.15.88:0/508765120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca7c0d0290 con 0x7fca7c0ce910 2024-09-16T01:45:13.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.057+0000 7fca727fc640 1 -- 172.21.15.88:0/508765120 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fca6c0047e0 con 0x7fca7c0ce910 2024-09-16T01:45:13.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.058+0000 7fca727fc640 1 --2- 172.21.15.88:0/508765120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fca6006c2e0 0x7fca6006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:13.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.058+0000 7fca727fc640 1 -- 172.21.15.88:0/508765120 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 5443+0+0 (secure 0 0 0) 0x7fca6c0aba30 con 0x7fca7c0ce910 2024-09-16T01:45:13.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.058+0000 7fca81dec640 1 --2- 172.21.15.88:0/508765120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fca6006c2e0 0x7fca6006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:13.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.058+0000 7fca81dec640 1 --2- 172.21.15.88:0/508765120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fca6006c2e0 0x7fca6006e7a0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fca78009bc0 tx=0x7fca78009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:13.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.062+0000 7fca727fc640 1 -- 172.21.15.88:0/508765120 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fca6c07a2a0 con 0x7fca7c0ce910 2024-09-16T01:45:13.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.227+0000 7fca84050640 1 -- 172.21.15.88:0/508765120 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fca7c0b36e0 con 0x7fca7c0ce910 2024-09-16T01:45:13.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.931+0000 7fca727fc640 1 -- 172.21.15.88:0/508765120 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v142) ==== 169+0+0 (secure 0 0 0) 0x7fca6c079ba0 con 0x7fca7c0ce910 2024-09-16T01:45:13.933 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_data' removed 2024-09-16T01:45:13.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.937+0000 7fca84050640 1 -- 172.21.15.88:0/508765120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fca6006c2e0 msgr2=0x7fca6006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:13.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.937+0000 7fca84050640 1 --2- 172.21.15.88:0/508765120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fca6006c2e0 0x7fca6006e7a0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fca78009bc0 tx=0x7fca78009190 comp rx=0 tx=0).stop 2024-09-16T01:45:13.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.937+0000 7fca84050640 1 -- 172.21.15.88:0/508765120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca7c0ce910 msgr2=0x7fca7c15ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:13.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.937+0000 7fca84050640 1 --2- 172.21.15.88:0/508765120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca7c0ce910 0x7fca7c15ca20 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7fca6c00c210 tx=0x7fca7c0cd270 comp rx=0 tx=0).stop 2024-09-16T01:45:13.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.937+0000 7fca84050640 1 -- 172.21.15.88:0/508765120 shutdown_connections 2024-09-16T01:45:13.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.937+0000 7fca84050640 1 --2- 172.21.15.88:0/508765120 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fca6006c2e0 0x7fca6006e7a0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:13.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.937+0000 7fca84050640 1 --2- 172.21.15.88:0/508765120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca7c15cf60 0x7fca7c13aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:13.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.937+0000 7fca84050640 1 --2- 172.21.15.88:0/508765120 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca7c0ce910 0x7fca7c15ca20 unknown :-1 s=CLOSED pgs=2435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:13.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.937+0000 7fca84050640 1 --2- 172.21.15.88:0/508765120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca68004800 0x7fca7c0d0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:13.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.937+0000 7fca84050640 1 -- 172.21.15.88:0/508765120 >> 172.21.15.88:0/508765120 conn(0x7fca7c06d0f0 msgr2=0x7fca7c15bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:13.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.937+0000 7fca84050640 1 -- 172.21.15.88:0/508765120 shutdown_connections 2024-09-16T01:45:13.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:13.938+0000 7fca84050640 1 -- 172.21.15.88:0/508765120 wait complete. 2024-09-16T01:45:13.954 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-16T01:45:14.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.058+0000 7f3335ad0640 1 Processor -- start 2024-09-16T01:45:14.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.058+0000 7f3335ad0640 1 -- start start 2024-09-16T01:45:14.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.059+0000 7f3335ad0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f333015e460 0x7f33300b3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:14.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.059+0000 7f3335ad0640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33300b0090 con 0x7f333015e840 2024-09-16T01:45:14.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.059+0000 7f3335ad0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33300af480 con 0x7f33300b4a50 2024-09-16T01:45:14.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.059+0000 7f3335ad0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f33300af760 con 0x7f333015e460 2024-09-16T01:45:14.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.059+0000 7f332effd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f333015e840 0x7f33300b4340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59968/0 (socket says 172.21.15.88:59968) 2024-09-16T01:45:14.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.059+0000 7f332effd640 1 -- 172.21.15.88:0/1594246668 learned_addr learned my addr 172.21.15.88:0/1594246668 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:14.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.059+0000 7f332e7fc640 1 -- 172.21.15.88:0/1594246668 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 178603536 0 0) 0x7f33300b0090 con 0x7f333015e840 2024-09-16T01:45:14.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.059+0000 7f332e7fc640 1 -- 172.21.15.88:0/1594246668 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3318003520 con 0x7f333015e840 2024-09-16T01:45:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.059+0000 7f332f7fe640 1 --2- 172.21.15.88:0/1594246668 >> v2:172.21.15.154:3300/0 conn(0x7f333015e460 0x7f33300b3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.059+0000 7f332e7fc640 1 -- 172.21.15.88:0/1594246668 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3840312146 0 0) 0x7f3318003520 con 0x7f333015e840 2024-09-16T01:45:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.059+0000 7f332e7fc640 1 -- 172.21.15.88:0/1594246668 >> v2:172.21.15.154:3300/0 conn(0x7f333015e460 msgr2=0x7f33300b3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.059+0000 7f332e7fc640 1 --2- 172.21.15.88:0/1594246668 >> v2:172.21.15.154:3300/0 conn(0x7f333015e460 0x7f33300b3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.059+0000 7f332e7fc640 1 -- 172.21.15.88:0/1594246668 >> v1:172.21.15.154:6789/0 conn(0x7f33300b4a50 legacy=0x7f3330071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.059+0000 7f332e7fc640 1 -- 172.21.15.88:0/1594246668 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33300717d0 con 0x7f333015e840 2024-09-16T01:45:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.059+0000 7f332e7fc640 1 -- 172.21.15.88:0/1594246668 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f3324001670 con 0x7f333015e840 2024-09-16T01:45:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.059+0000 7f332e7fc640 1 -- 172.21.15.88:0/1594246668 >> v1:172.21.15.88:6789/0 conn(0x7f333015e840 legacy=0x7f33300b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.060+0000 7f332e7fc640 1 --2- 172.21.15.88:0/1594246668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33180039b0 0x7f3318003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.060+0000 7f332e7fc640 1 --2- 172.21.15.88:0/1594246668 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3318004800 0x7f3318004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.060+0000 7f332e7fc640 1 --2- 172.21.15.88:0/1594246668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33180051c0 0x7f331800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.060+0000 7f332e7fc640 1 -- 172.21.15.88:0/1594246668 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f33300af760 con 0x7f33180051c0 2024-09-16T01:45:14.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.060+0000 7f332e7fc640 1 -- 172.21.15.88:0/1594246668 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f331800c0f0 con 0x7f33180039b0 2024-09-16T01:45:14.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.060+0000 7f332e7fc640 1 -- 172.21.15.88:0/1594246668 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f331800c2f0 con 0x7f3318004800 2024-09-16T01:45:14.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.060+0000 7f332ffff640 1 --2- 172.21.15.88:0/1594246668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33180051c0 0x7f331800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:14.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.060+0000 7f332f7fe640 1 --2- 172.21.15.88:0/1594246668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33180039b0 0x7f3318003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:14.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.060+0000 7f332effd640 1 --2- 172.21.15.88:0/1594246668 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3318004800 0x7f3318004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:14.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.060+0000 7f332ffff640 1 -- 172.21.15.88:0/1594246668 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3318004800 msgr2=0x7f3318004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:14.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.060+0000 7f332ffff640 1 --2- 172.21.15.88:0/1594246668 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3318004800 0x7f3318004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:14.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.060+0000 7f332ffff640 1 -- 172.21.15.88:0/1594246668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33180039b0 msgr2=0x7f3318003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:14.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.060+0000 7f332ffff640 1 --2- 172.21.15.88:0/1594246668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33180039b0 0x7f3318003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:14.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.060+0000 7f332ffff640 1 -- 172.21.15.88:0/1594246668 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3318003520 con 0x7f33180051c0 2024-09-16T01:45:14.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.061+0000 7f332ffff640 1 -- 172.21.15.88:0/1594246668 reap_dead start 2024-09-16T01:45:14.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.061+0000 7f332ffff640 1 --2- 172.21.15.88:0/1594246668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33180051c0 0x7f331800b9c0 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f3328004480 tx=0x7f3318005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89fa52667a889e34 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:14.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.061+0000 7f332e7fc640 1 -- 172.21.15.88:0/1594246668 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3328004080 con 0x7f33180051c0 2024-09-16T01:45:14.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.061+0000 7f332e7fc640 1 -- 172.21.15.88:0/1594246668 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f332802ad80 con 0x7f33180051c0 2024-09-16T01:45:14.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.061+0000 7f332e7fc640 1 -- 172.21.15.88:0/1594246668 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3328006d90 con 0x7f33180051c0 2024-09-16T01:45:14.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.061+0000 7f3335ad0640 1 -- 172.21.15.88:0/1594246668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33180051c0 msgr2=0x7f331800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:14.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.061+0000 7f3335ad0640 1 --2- 172.21.15.88:0/1594246668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33180051c0 0x7f331800b9c0 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f3328004480 tx=0x7f3318005830 comp rx=0 tx=0).stop 2024-09-16T01:45:14.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.061+0000 7f3335ad0640 1 -- 172.21.15.88:0/1594246668 shutdown_connections 2024-09-16T01:45:14.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.061+0000 7f3335ad0640 1 --2- 172.21.15.88:0/1594246668 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33180051c0 0x7f331800b9c0 unknown :-1 s=CLOSED pgs=2437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:14.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.061+0000 7f3335ad0640 1 -- 172.21.15.88:0/1594246668 >> 172.21.15.88:0/1594246668 conn(0x7f333006bc10 msgr2=0x7f333006bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:14.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.061+0000 7f3335ad0640 1 -- 172.21.15.88:0/1594246668 shutdown_connections 2024-09-16T01:45:14.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.061+0000 7f3335ad0640 1 -- 172.21.15.88:0/1594246668 wait complete. 2024-09-16T01:45:14.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.062+0000 7f3335ad0640 1 Processor -- start 2024-09-16T01:45:14.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.063+0000 7f3335ad0640 1 -- start start 2024-09-16T01:45:14.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.063+0000 7f3335ad0640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33180051c0 0x7f33301524c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:14.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.063+0000 7f3335ad0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33301542f0 0x7f3330152a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:14.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.063+0000 7f3335ad0640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33301f2fb0 0x7f33301f3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:14.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.063+0000 7f3335ad0640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f33300afe50 con 0x7f33301f2fb0 2024-09-16T01:45:14.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.063+0000 7f3335ad0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f33300afcd0 con 0x7f33301542f0 2024-09-16T01:45:14.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.063+0000 7f3335ad0640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f33300ce680 con 0x7f33180051c0 2024-09-16T01:45:14.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.064+0000 7f332ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33301f2fb0 0x7f33301f3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:14.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.064+0000 7f332ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33301f2fb0 0x7f33301f3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59984/0 (socket says 172.21.15.88:59984) 2024-09-16T01:45:14.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.064+0000 7f332ffff640 1 -- 172.21.15.88:0/3600138111 learned_addr learned my addr 172.21.15.88:0/3600138111 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:14.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.064+0000 7f332f7fe640 1 --2- 172.21.15.88:0/3600138111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33180051c0 0x7f33301524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:14.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.064+0000 7f332effd640 1 --2- 172.21.15.88:0/3600138111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33301542f0 0x7f3330152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:14.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.064+0000 7f332ffff640 1 -- 172.21.15.88:0/3600138111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33180051c0 msgr2=0x7f33301524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:14.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.064+0000 7f332ffff640 1 --2- 172.21.15.88:0/3600138111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33180051c0 0x7f33301524c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:14.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.064+0000 7f332ffff640 1 -- 172.21.15.88:0/3600138111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33301542f0 msgr2=0x7f3330152a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:14.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.064+0000 7f332ffff640 1 --2- 172.21.15.88:0/3600138111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33301542f0 0x7f3330152a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:14.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.064+0000 7f332ffff640 1 -- 172.21.15.88:0/3600138111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33301547d0 con 0x7f33301f2fb0 2024-09-16T01:45:14.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.064+0000 7f332ffff640 1 --2- 172.21.15.88:0/3600138111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33301f2fb0 0x7f33301f3440 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7f3328004480 tx=0x7f3328003a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:14.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.064+0000 7f3317fff640 1 -- 172.21.15.88:0/3600138111 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3328007860 con 0x7f33301f2fb0 2024-09-16T01:45:14.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.064+0000 7f3317fff640 1 -- 172.21.15.88:0/3600138111 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3328043070 con 0x7f33301f2fb0 2024-09-16T01:45:14.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.064+0000 7f3335ad0640 1 -- 172.21.15.88:0/3600138111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3330154ab0 con 0x7f33301f2fb0 2024-09-16T01:45:14.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.064+0000 7f3317fff640 1 -- 172.21.15.88:0/3600138111 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3328032440 con 0x7f33301f2fb0 2024-09-16T01:45:14.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.064+0000 7f3335ad0640 1 -- 172.21.15.88:0/3600138111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3330155070 con 0x7f33301f2fb0 2024-09-16T01:45:14.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.065+0000 7f3335ad0640 1 -- 172.21.15.88:0/3600138111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33300b3830 con 0x7f33301f2fb0 2024-09-16T01:45:14.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.066+0000 7f3317fff640 1 -- 172.21.15.88:0/3600138111 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f33280325e0 con 0x7f33301f2fb0 2024-09-16T01:45:14.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.066+0000 7f3317fff640 1 --2- 172.21.15.88:0/3600138111 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f331c06c2e0 0x7f331c06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:14.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.066+0000 7f332f7fe640 1 --2- 172.21.15.88:0/3600138111 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f331c06c2e0 0x7f331c06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:14.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.066+0000 7f3317fff640 1 -- 172.21.15.88:0/3600138111 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7f33280ab630 con 0x7f33301f2fb0 2024-09-16T01:45:14.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.067+0000 7f332f7fe640 1 --2- 172.21.15.88:0/3600138111 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f331c06c2e0 0x7f331c06e7a0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f3320006fd0 tx=0x7f3320008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:14.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.071+0000 7f3317fff640 1 -- 172.21.15.88:0/3600138111 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3328079b10 con 0x7f33301f2fb0 2024-09-16T01:45:14.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.236+0000 7f3335ad0640 1 -- 172.21.15.88:0/3600138111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f3330075490 con 0x7f33301f2fb0 2024-09-16T01:45:14.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.239+0000 7f3317fff640 1 -- 172.21.15.88:0/3600138111 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) ==== 104+0+0 (secure 0 0 0) 0x7f332807de10 con 0x7f33301f2fb0 2024-09-16T01:45:14.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.241+0000 7f3335ad0640 1 -- 172.21.15.88:0/3600138111 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f331c06c2e0 msgr2=0x7f331c06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:14.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.241+0000 7f3335ad0640 1 --2- 172.21.15.88:0/3600138111 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f331c06c2e0 0x7f331c06e7a0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f3320006fd0 tx=0x7f3320008040 comp rx=0 tx=0).stop 2024-09-16T01:45:14.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.241+0000 7f3335ad0640 1 -- 172.21.15.88:0/3600138111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33301f2fb0 msgr2=0x7f33301f3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:14.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.241+0000 7f3335ad0640 1 --2- 172.21.15.88:0/3600138111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33301f2fb0 0x7f33301f3440 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7f3328004480 tx=0x7f3328003a70 comp rx=0 tx=0).stop 2024-09-16T01:45:14.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.241+0000 7f3335ad0640 1 -- 172.21.15.88:0/3600138111 shutdown_connections 2024-09-16T01:45:14.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.241+0000 7f3335ad0640 1 --2- 172.21.15.88:0/3600138111 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f331c06c2e0 0x7f331c06e7a0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:14.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.241+0000 7f3335ad0640 1 --2- 172.21.15.88:0/3600138111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33301f2fb0 0x7f33301f3440 unknown :-1 s=CLOSED pgs=2438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:14.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.241+0000 7f3335ad0640 1 --2- 172.21.15.88:0/3600138111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33301542f0 0x7f3330152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:14.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.241+0000 7f3335ad0640 1 --2- 172.21.15.88:0/3600138111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33180051c0 0x7f33301524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:14.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.241+0000 7f3335ad0640 1 -- 172.21.15.88:0/3600138111 >> 172.21.15.88:0/3600138111 conn(0x7f333006bc10 msgr2=0x7f33300cf850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:14.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.241+0000 7f3335ad0640 1 -- 172.21.15.88:0/3600138111 shutdown_connections 2024-09-16T01:45:14.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.241+0000 7f3335ad0640 1 -- 172.21.15.88:0/3600138111 wait complete. 2024-09-16T01:45:14.254 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_damage.TestDamage.test_open_ino_errors' 2024-09-16T01:45:14.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.356+0000 7f4d841f8640 1 Processor -- start 2024-09-16T01:45:14.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.356+0000 7f4d841f8640 1 -- start start 2024-09-16T01:45:14.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.357+0000 7f4d841f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c073080 0x7f4d7c0b5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:14.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.357+0000 7f4d841f8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4d7c15c670 0x7f4d7c073460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:14.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.357+0000 7f4d841f8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4d7c0b1990 0x7f4d7c0d10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:14.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.357+0000 7f4d841f8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f4d7c0d0c60 con 0x7f4d7c15c670 2024-09-16T01:45:14.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.357+0000 7f4d841f8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f4d7c071750 con 0x7f4d7c0b1990 2024-09-16T01:45:14.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.357+0000 7f4d841f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4d7c0b59a0 con 0x7f4d7c073080 2024-09-16T01:45:14.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.357+0000 7f4d81793640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4d7c15c670 0x7f4d7c073460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:14.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.357+0000 7f4d81793640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f4d7c15c670 0x7f4d7c073460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59988/0 (socket says 172.21.15.88:59988) 2024-09-16T01:45:14.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.357+0000 7f4d81793640 1 -- 172.21.15.88:0/3337790221 learned_addr learned my addr 172.21.15.88:0/3337790221 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:14.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.358+0000 7f4d81f94640 1 --2- 172.21.15.88:0/3337790221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c073080 0x7f4d7c0b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:14.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.358+0000 7f4d81793640 1 -- 172.21.15.88:0/3337790221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c073080 msgr2=0x7f4d7c0b5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:14.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.358+0000 7f4d81793640 1 --2- 172.21.15.88:0/3337790221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c073080 0x7f4d7c0b5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:14.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.358+0000 7f4d81793640 1 -- 172.21.15.88:0/3337790221 >> v2:172.21.15.154:3300/0 conn(0x7f4d7c0b1990 msgr2=0x7f4d7c0d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:14.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.358+0000 7f4d82795640 1 --2- 172.21.15.88:0/3337790221 >> v2:172.21.15.154:3300/0 conn(0x7f4d7c0b1990 0x7f4d7c0d10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:14.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.358+0000 7f4d81793640 1 --2- 172.21.15.88:0/3337790221 >> v2:172.21.15.154:3300/0 conn(0x7f4d7c0b1990 0x7f4d7c0d10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:14.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.358+0000 7f4d81793640 1 -- 172.21.15.88:0/3337790221 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d7c0d0370 con 0x7f4d7c15c670 2024-09-16T01:45:14.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.358+0000 7f4d81f94640 1 --2- 172.21.15.88:0/3337790221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c073080 0x7f4d7c0b5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:45:14.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.358+0000 7f4d82795640 1 --2- 172.21.15.88:0/3337790221 >> v2:172.21.15.154:3300/0 conn(0x7f4d7c0b1990 0x7f4d7c0d10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:45:14.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.358+0000 7f4d81793640 1 --2- 172.21.15.88:0/3337790221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d7c15c670 0x7f4d7c073460 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c1610c0 tx=0x7f4d7002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7727f7a3e54843e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:14.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.359+0000 7f4d80f92640 1 -- 172.21.15.88:0/3337790221 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d70037070 con 0x7f4d7c15c670 2024-09-16T01:45:14.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.359+0000 7f4d80f92640 1 -- 172.21.15.88:0/3337790221 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d70032440 con 0x7f4d7c15c670 2024-09-16T01:45:14.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.359+0000 7f4d80f92640 1 -- 172.21.15.88:0/3337790221 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d70032810 con 0x7f4d7c15c670 2024-09-16T01:45:14.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.359+0000 7f4d841f8640 1 -- 172.21.15.88:0/3337790221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d7c15c670 msgr2=0x7f4d7c073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:14.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.359+0000 7f4d841f8640 1 --2- 172.21.15.88:0/3337790221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d7c15c670 0x7f4d7c073460 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c1610c0 tx=0x7f4d7002a1e0 comp rx=0 tx=0).stop 2024-09-16T01:45:14.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.359+0000 7f4d841f8640 1 -- 172.21.15.88:0/3337790221 shutdown_connections 2024-09-16T01:45:14.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.359+0000 7f4d841f8640 1 --2- 172.21.15.88:0/3337790221 >> v2:172.21.15.154:3300/0 conn(0x7f4d7c0b1990 0x7f4d7c0d10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:14.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.359+0000 7f4d841f8640 1 --2- 172.21.15.88:0/3337790221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d7c15c670 0x7f4d7c073460 unknown :-1 s=CLOSED pgs=2439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:14.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.359+0000 7f4d841f8640 1 --2- 172.21.15.88:0/3337790221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c073080 0x7f4d7c0b5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:14.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.359+0000 7f4d841f8640 1 -- 172.21.15.88:0/3337790221 >> 172.21.15.88:0/3337790221 conn(0x7f4d7c0d24d0 msgr2=0x7f4d7c0d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:14.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.359+0000 7f4d841f8640 1 -- 172.21.15.88:0/3337790221 shutdown_connections 2024-09-16T01:45:14.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.359+0000 7f4d841f8640 1 -- 172.21.15.88:0/3337790221 wait complete. 2024-09-16T01:45:14.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.360+0000 7f4d841f8640 1 Processor -- start 2024-09-16T01:45:14.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.360+0000 7f4d841f8640 1 -- start start 2024-09-16T01:45:14.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.361+0000 7f4d841f8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d7c073080 0x7f4d7c14dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:14.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.361+0000 7f4d841f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c0b1990 0x7f4d7c1e6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:14.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.361+0000 7f4d841f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d7c15c670 0x7f4d7c1e73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:14.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.361+0000 7f4d841f8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c071470 con 0x7f4d7c073080 2024-09-16T01:45:14.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.361+0000 7f4d841f8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4d7c071190 con 0x7f4d7c15c670 2024-09-16T01:45:14.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.361+0000 7f4d841f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4d7c0d0610 con 0x7f4d7c0b1990 2024-09-16T01:45:14.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.361+0000 7f4d81f94640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d7c073080 0x7f4d7c14dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:14.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.361+0000 7f4d81f94640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d7c073080 0x7f4d7c14dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59992/0 (socket says 172.21.15.88:59992) 2024-09-16T01:45:14.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.361+0000 7f4d81f94640 1 -- 172.21.15.88:0/1770609359 learned_addr learned my addr 172.21.15.88:0/1770609359 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:14.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.361+0000 7f4d81793640 1 --2- 172.21.15.88:0/1770609359 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c0b1990 0x7f4d7c1e6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:14.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.361+0000 7f4d82795640 1 --2- 172.21.15.88:0/1770609359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d7c15c670 0x7f4d7c1e73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:14.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.361+0000 7f4d81f94640 1 -- 172.21.15.88:0/1770609359 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c0b1990 msgr2=0x7f4d7c1e6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:14.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.361+0000 7f4d81f94640 1 --2- 172.21.15.88:0/1770609359 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c0b1990 0x7f4d7c1e6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:14.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.361+0000 7f4d81f94640 1 -- 172.21.15.88:0/1770609359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d7c15c670 msgr2=0x7f4d7c1e73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:14.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.361+0000 7f4d81f94640 1 --2- 172.21.15.88:0/1770609359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d7c15c670 0x7f4d7c1e73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:14.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.361+0000 7f4d81f94640 1 -- 172.21.15.88:0/1770609359 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d7c151ea0 con 0x7f4d7c073080 2024-09-16T01:45:14.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.362+0000 7f4d81f94640 1 --2- 172.21.15.88:0/1770609359 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d7c073080 0x7f4d7c14dcf0 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c00ce30 tx=0x7f4d6c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:14.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.362+0000 7f4d7a7fc640 1 -- 172.21.15.88:0/1770609359 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c00f680 con 0x7f4d7c073080 2024-09-16T01:45:14.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.362+0000 7f4d7a7fc640 1 -- 172.21.15.88:0/1770609359 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d6c00f820 con 0x7f4d7c073080 2024-09-16T01:45:14.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.362+0000 7f4d841f8640 1 -- 172.21.15.88:0/1770609359 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d7c152180 con 0x7f4d7c073080 2024-09-16T01:45:14.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.362+0000 7f4d7a7fc640 1 -- 172.21.15.88:0/1770609359 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c00fc10 con 0x7f4d7c073080 2024-09-16T01:45:14.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.362+0000 7f4d841f8640 1 -- 172.21.15.88:0/1770609359 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d7c152740 con 0x7f4d7c073080 2024-09-16T01:45:14.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.363+0000 7f4d841f8640 1 -- 172.21.15.88:0/1770609359 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d7c0d08f0 con 0x7f4d7c073080 2024-09-16T01:45:14.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.363+0000 7f4d7a7fc640 1 -- 172.21.15.88:0/1770609359 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4d6c018400 con 0x7f4d7c073080 2024-09-16T01:45:14.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.364+0000 7f4d7a7fc640 1 --2- 172.21.15.88:0/1770609359 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d6806c2b0 0x7f4d6806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:14.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.364+0000 7f4d7a7fc640 1 -- 172.21.15.88:0/1770609359 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7f4d6c08bed0 con 0x7f4d7c073080 2024-09-16T01:45:14.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.364+0000 7f4d81793640 1 --2- 172.21.15.88:0/1770609359 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d6806c2b0 0x7f4d6806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:14.368 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.364+0000 7f4d81793640 1 --2- 172.21.15.88:0/1770609359 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d6806c2b0 0x7f4d6806e770 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f4d70034000 tx=0x7f4d7002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:14.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.368+0000 7f4d7a7fc640 1 -- 172.21.15.88:0/1770609359 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4d6c014030 con 0x7f4d7c073080 2024-09-16T01:45:14.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.601+0000 7f4d841f8640 1 -- 172.21.15.88:0/1770609359 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_open_ino_errors"]} v 0) -- 0x7f4d7c0d3050 con 0x7f4d7c073080 2024-09-16T01:45:14.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.932+0000 7f4d7a7fc640 1 -- 172.21.15.88:0/1770609359 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_open_ino_errors"]}]=0 v567) ==== 135+0+0 (secure 0 0 0) 0x7f4d6c05a970 con 0x7f4d7c073080 2024-09-16T01:45:14.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.935+0000 7f4d841f8640 1 -- 172.21.15.88:0/1770609359 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d6806c2b0 msgr2=0x7f4d6806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:14.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.935+0000 7f4d841f8640 1 --2- 172.21.15.88:0/1770609359 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d6806c2b0 0x7f4d6806e770 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f4d70034000 tx=0x7f4d7002e000 comp rx=0 tx=0).stop 2024-09-16T01:45:14.936 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.935+0000 7f4d841f8640 1 -- 172.21.15.88:0/1770609359 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d7c073080 msgr2=0x7f4d7c14dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:14.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.935+0000 7f4d841f8640 1 --2- 172.21.15.88:0/1770609359 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d7c073080 0x7f4d7c14dcf0 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c00ce30 tx=0x7f4d6c00a650 comp rx=0 tx=0).stop 2024-09-16T01:45:14.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.935+0000 7f4d841f8640 1 -- 172.21.15.88:0/1770609359 shutdown_connections 2024-09-16T01:45:14.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.935+0000 7f4d841f8640 1 --2- 172.21.15.88:0/1770609359 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4d6806c2b0 0x7f4d6806e770 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:14.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.935+0000 7f4d841f8640 1 --2- 172.21.15.88:0/1770609359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d7c15c670 0x7f4d7c1e73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:14.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.935+0000 7f4d841f8640 1 --2- 172.21.15.88:0/1770609359 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d7c0b1990 0x7f4d7c1e6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:14.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.935+0000 7f4d841f8640 1 --2- 172.21.15.88:0/1770609359 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d7c073080 0x7f4d7c14dcf0 unknown :-1 s=CLOSED pgs=2440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:14.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.935+0000 7f4d841f8640 1 -- 172.21.15.88:0/1770609359 >> 172.21.15.88:0/1770609359 conn(0x7f4d7c0d24d0 msgr2=0x7f4d7c0b7040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:14.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.935+0000 7f4d841f8640 1 -- 172.21.15.88:0/1770609359 shutdown_connections 2024-09-16T01:45:14.937 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:14.935+0000 7f4d841f8640 1 -- 172.21.15.88:0/1770609359 wait complete. 2024-09-16T01:45:14.952 INFO:tasks.cephfs_test_runner:That errors encountered during opening inos are properly propagated ... ERROR 2024-09-16T01:45:14.953 INFO:tasks.cephfs_test_runner: 2024-09-16T01:45:14.953 INFO:tasks.cephfs_test_runner:====================================================================== 2024-09-16T01:45:14.953 INFO:tasks.cephfs_test_runner:ERROR: test_open_ino_errors (tasks.cephfs.test_damage.TestDamage) 2024-09-16T01:45:14.953 INFO:tasks.cephfs_test_runner:That errors encountered during opening inos are properly propagated 2024-09-16T01:45:14.953 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-09-16T01:45:14.953 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-09-16T01:45:14.953 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks/cephfs/test_damage.py", line 510, in test_open_ino_errors 2024-09-16T01:45:14.953 INFO:tasks.cephfs_test_runner: self.fs.read_backtrace(file1_ino) 2024-09-16T01:45:14.953 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks/cephfs/filesystem.py", line 1508, in read_backtrace 2024-09-16T01:45:14.953 INFO:tasks.cephfs_test_runner: return self._read_data_xattr(ino_no, "parent", "inode_backtrace_t", pool) 2024-09-16T01:45:14.953 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks/cephfs/filesystem.py", line 1464, in _read_data_xattr 2024-09-16T01:45:14.953 INFO:tasks.cephfs_test_runner: return json.loads(self.dencoder(obj_type, obj_blob).strip()) 2024-09-16T01:45:14.954 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks/cephfs/filesystem.py", line 1267, in dencoder 2024-09-16T01:45:14.954 INFO:tasks.cephfs_test_runner: p = self.mon_manager.controller.run(args=args, stdin=BytesIO(obj_blob), stdout=BytesIO()) 2024-09-16T01:45:14.954 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run 2024-09-16T01:45:14.954 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-09-16T01:45:14.954 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run 2024-09-16T01:45:14.954 INFO:tasks.cephfs_test_runner: r.wait() 2024-09-16T01:45:14.954 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait 2024-09-16T01:45:14.954 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-09-16T01:45:14.954 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-09-16T01:45:14.954 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-09-16T01:45:14.954 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 1: 'ceph-dencoder type inode_backtrace_t import - decode dump_json' 2024-09-16T01:45:14.954 INFO:tasks.cephfs_test_runner: 2024-09-16T01:45:14.954 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-09-16T01:45:14.954 INFO:tasks.cephfs_test_runner:Ran 6 tests in 630.857s 2024-09-16T01:45:14.954 INFO:tasks.cephfs_test_runner: 2024-09-16T01:45:14.955 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-09-16T01:45:14.955 INFO:tasks.cephfs_test_runner: 2024-09-16T01:45:14.955 INFO:tasks.cephfs_test_runner:====================================================================== 2024-09-16T01:45:14.955 INFO:tasks.cephfs_test_runner:ERROR: test_open_ino_errors (tasks.cephfs.test_damage.TestDamage) 2024-09-16T01:45:14.955 INFO:tasks.cephfs_test_runner:That errors encountered during opening inos are properly propagated 2024-09-16T01:45:14.955 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-09-16T01:45:14.955 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-09-16T01:45:14.955 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks/cephfs/test_damage.py", line 510, in test_open_ino_errors 2024-09-16T01:45:14.955 INFO:tasks.cephfs_test_runner: self.fs.read_backtrace(file1_ino) 2024-09-16T01:45:14.955 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks/cephfs/filesystem.py", line 1508, in read_backtrace 2024-09-16T01:45:14.955 INFO:tasks.cephfs_test_runner: return self._read_data_xattr(ino_no, "parent", "inode_backtrace_t", pool) 2024-09-16T01:45:14.955 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks/cephfs/filesystem.py", line 1464, in _read_data_xattr 2024-09-16T01:45:14.956 INFO:tasks.cephfs_test_runner: return json.loads(self.dencoder(obj_type, obj_blob).strip()) 2024-09-16T01:45:14.956 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks/cephfs/filesystem.py", line 1267, in dencoder 2024-09-16T01:45:14.956 INFO:tasks.cephfs_test_runner: p = self.mon_manager.controller.run(args=args, stdin=BytesIO(obj_blob), stdout=BytesIO()) 2024-09-16T01:45:14.956 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run 2024-09-16T01:45:14.956 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-09-16T01:45:14.956 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run 2024-09-16T01:45:14.956 INFO:tasks.cephfs_test_runner: r.wait() 2024-09-16T01:45:14.956 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait 2024-09-16T01:45:14.956 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-09-16T01:45:14.956 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-09-16T01:45:14.956 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-09-16T01:45:14.956 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 1: 'ceph-dencoder type inode_backtrace_t import - decode dump_json' 2024-09-16T01:45:14.956 INFO:tasks.cephfs_test_runner: 2024-09-16T01:45:14.957 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 112, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_open_ino_errors (tasks.cephfs.test_damage.TestDamage) 2024-09-16T01:45:15.130 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=8981d074651243e4aff2cda2d1ec9639 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 112, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_open_ino_errors (tasks.cephfs.test_damage.TestDamage) 2024-09-16T01:45:15.132 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-09-16T01:45:15.140 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-09-16T01:45:15.150 INFO:tasks.kclient:Unmounting kernel clients... 2024-09-16T01:45:15.151 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:45:15.151 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:45:15.171 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:45:15.171 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-09-16T01:45:15.171 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:45:15.228 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056025,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19827 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-16T01:45:15.228 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:45:15.228 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:45:15.248 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056026,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1347 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T01:45:15.248 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T01:45:15.248 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-16T01:45:15.307 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056026,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1347 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248728k,nr_inodes=812182,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497460k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-16T01:45:15.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:45:15.307 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:45:15.324 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-16T01:45:15.325 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-16T01:45:15.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:45:15.325 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-16T01:45:15.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:45:15.392 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-16T01:45:15.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:45:15.458 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link delete ceph-brx 2024-09-16T01:45:15.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:45:15.559 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2024-09-16T01:45:15.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:45:15.575 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link delete ceph-brx 2024-09-16T01:45:15.643 INFO:teuthology.orchestra.run.smithi088.stderr:Cannot find device "ceph-brx" 2024-09-16T01:45:15.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:45:15.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:45:15.645 DEBUG:teuthology.orchestra.run.smithi154:> ip netns list 2024-09-16T01:45:15.665 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-16T01:45:15.665 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-16T01:45:15.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:45:15.666 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-16T01:45:15.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:45:15.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-16T01:45:15.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:45:15.805 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip link delete ceph-brx 2024-09-16T01:45:15.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:45:15.918 DEBUG:teuthology.orchestra.run.smithi154:> ip netns list 2024-09-16T01:45:15.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-16T01:45:15.934 DEBUG:teuthology.orchestra.run.smithi154:> sudo ip link delete ceph-brx 2024-09-16T01:45:16.002 INFO:teuthology.orchestra.run.smithi154.stderr:Cannot find device "ceph-brx" 2024-09-16T01:45:16.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:45:16.004 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T01:45:16.013 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T01:45:16.014 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:45:16.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.115+0000 7f7907566640 1 Processor -- start 2024-09-16T01:45:16.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.115+0000 7f7907566640 1 -- start start 2024-09-16T01:45:16.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.116+0000 7f7907566640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7900155800 0x7f79000b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:16.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.116+0000 7f7907566640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f790015c330 0x7f79000b4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:16.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.116+0000 7f7907566640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f790015c770 con 0x7f79000cf570 2024-09-16T01:45:16.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.116+0000 7f7907566640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f79000b4e60 con 0x7f790015c330 2024-09-16T01:45:16.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.116+0000 7f7907566640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f79000b50a0 con 0x7f7900155800 2024-09-16T01:45:16.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.116+0000 7f7905302640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7900155800 0x7f79000b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:16.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.116+0000 7f7904b01640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f790015c330 0x7f79000b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:16.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.116+0000 7f7905302640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7900155800 0x7f79000b6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:48284/0 (socket says 172.21.15.88:48284) 2024-09-16T01:45:16.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.116+0000 7f7905302640 1 -- 172.21.15.88:0/4011626621 learned_addr learned my addr 172.21.15.88:0/4011626621 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:16.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.116+0000 7f7904b01640 1 -- 172.21.15.88:0/4011626621 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7900155800 msgr2=0x7f79000b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:16.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.116+0000 7f7904b01640 1 --2- 172.21.15.88:0/4011626621 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7900155800 0x7f79000b6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.116+0000 7f7904b01640 1 -- 172.21.15.88:0/4011626621 >> v1:172.21.15.154:6789/0 conn(0x7f79000cf570 legacy=0x7f790015c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:16.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.116+0000 7f7904b01640 1 -- 172.21.15.88:0/4011626621 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79000b5380 con 0x7f790015c330 2024-09-16T01:45:16.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.116+0000 7f7905302640 1 --2- 172.21.15.88:0/4011626621 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7900155800 0x7f79000b6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:45:16.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.117+0000 7f7904b01640 1 --2- 172.21.15.88:0/4011626621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f790015c330 0x7f79000b4920 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f78f00097a0 tx=0x7f78f002a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=386eebb712af8515 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:16.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.117+0000 7f78f7fff640 1 -- 172.21.15.88:0/4011626621 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f0037070 con 0x7f790015c330 2024-09-16T01:45:16.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.117+0000 7f78f7fff640 1 -- 172.21.15.88:0/4011626621 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78f0032440 con 0x7f790015c330 2024-09-16T01:45:16.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.117+0000 7f78f7fff640 1 -- 172.21.15.88:0/4011626621 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f0032810 con 0x7f790015c330 2024-09-16T01:45:16.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.118+0000 7f7907566640 1 -- 172.21.15.88:0/4011626621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f790015c330 msgr2=0x7f79000b4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:16.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.118+0000 7f7907566640 1 --2- 172.21.15.88:0/4011626621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f790015c330 0x7f79000b4920 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f78f00097a0 tx=0x7f78f002a180 comp rx=0 tx=0).stop 2024-09-16T01:45:16.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.118+0000 7f7907566640 1 -- 172.21.15.88:0/4011626621 shutdown_connections 2024-09-16T01:45:16.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.118+0000 7f7907566640 1 --2- 172.21.15.88:0/4011626621 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f790015c330 0x7f79000b4920 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.118+0000 7f7907566640 1 --2- 172.21.15.88:0/4011626621 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7900155800 0x7f79000b6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.118+0000 7f7907566640 1 -- 172.21.15.88:0/4011626621 >> 172.21.15.88:0/4011626621 conn(0x7f790006bc10 msgr2=0x7f790006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:16.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.118+0000 7f7907566640 1 -- 172.21.15.88:0/4011626621 shutdown_connections 2024-09-16T01:45:16.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.118+0000 7f7907566640 1 -- 172.21.15.88:0/4011626621 wait complete. 2024-09-16T01:45:16.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.119+0000 7f7907566640 1 Processor -- start 2024-09-16T01:45:16.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.119+0000 7f7907566640 1 -- start start 2024-09-16T01:45:16.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.120+0000 7f7907566640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79000cf570 0x7f7900147fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:16.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.120+0000 7f7907566640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7900155800 0x7f79000b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:16.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.120+0000 7f7907566640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f790015c330 0x7f79000b3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:16.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.120+0000 7f7907566640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f79000b5c20 con 0x7f790015c330 2024-09-16T01:45:16.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.120+0000 7f7907566640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f79000b5940 con 0x7f7900155800 2024-09-16T01:45:16.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.120+0000 7f7907566640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7900155be0 con 0x7f79000cf570 2024-09-16T01:45:16.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.120+0000 7f7905b03640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f790015c330 0x7f79000b3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:16.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.120+0000 7f7905302640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79000cf570 0x7f7900147fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:16.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.120+0000 7f7904b01640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7900155800 0x7f79000b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:16.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.120+0000 7f7905b03640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f790015c330 0x7f79000b3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60006/0 (socket says 172.21.15.88:60006) 2024-09-16T01:45:16.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.120+0000 7f7905b03640 1 -- 172.21.15.88:0/2620062095 learned_addr learned my addr 172.21.15.88:0/2620062095 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:16.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.120+0000 7f7905b03640 1 -- 172.21.15.88:0/2620062095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79000cf570 msgr2=0x7f7900147fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:16.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.120+0000 7f7905b03640 1 --2- 172.21.15.88:0/2620062095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79000cf570 0x7f7900147fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.120+0000 7f7905b03640 1 -- 172.21.15.88:0/2620062095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7900155800 msgr2=0x7f79000b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:16.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.120+0000 7f7905b03640 1 --2- 172.21.15.88:0/2620062095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7900155800 0x7f79000b3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.120+0000 7f7905b03640 1 -- 172.21.15.88:0/2620062095 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f790015be60 con 0x7f790015c330 2024-09-16T01:45:16.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.120+0000 7f7904b01640 1 --2- 172.21.15.88:0/2620062095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7900155800 0x7f79000b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:45:16.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.120+0000 7f7905b03640 1 --2- 172.21.15.88:0/2620062095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f790015c330 0x7f79000b3d50 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7f78f800c760 tx=0x7f78f800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:16.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.120+0000 7f78f5ffb640 1 -- 172.21.15.88:0/2620062095 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f8003b70 con 0x7f790015c330 2024-09-16T01:45:16.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.121+0000 7f78f5ffb640 1 -- 172.21.15.88:0/2620062095 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78f8003d30 con 0x7f790015c330 2024-09-16T01:45:16.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.121+0000 7f7907566640 1 -- 172.21.15.88:0/2620062095 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79000cac80 con 0x7f790015c330 2024-09-16T01:45:16.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.121+0000 7f78f5ffb640 1 -- 172.21.15.88:0/2620062095 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f800aa90 con 0x7f790015c330 2024-09-16T01:45:16.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.121+0000 7f7907566640 1 -- 172.21.15.88:0/2620062095 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79000cb240 con 0x7f790015c330 2024-09-16T01:45:16.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.122+0000 7f7907566640 1 -- 172.21.15.88:0/2620062095 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79000b6140 con 0x7f790015c330 2024-09-16T01:45:16.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.122+0000 7f78f5ffb640 1 -- 172.21.15.88:0/2620062095 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f78f800ac30 con 0x7f790015c330 2024-09-16T01:45:16.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.123+0000 7f78f5ffb640 1 --2- 172.21.15.88:0/2620062095 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f78e406c2b0 0x7f78e406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:16.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.123+0000 7f78f5ffb640 1 -- 172.21.15.88:0/2620062095 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7f78f808ae20 con 0x7f790015c330 2024-09-16T01:45:16.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.123+0000 7f7905302640 1 --2- 172.21.15.88:0/2620062095 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f78e406c2b0 0x7f78e406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:16.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.123+0000 7f7905302640 1 --2- 172.21.15.88:0/2620062095 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f78e406c2b0 0x7f78e406e770 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f78fc006ca0 tx=0x7f78fc009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:16.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.127+0000 7f78f5ffb640 1 -- 172.21.15.88:0/2620062095 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f790015be60 con 0x7f790015c330 2024-09-16T01:45:16.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.288+0000 7f7907566640 1 -- 172.21.15.88:0/2620062095 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f79000b2c10 con 0x7f78e406c2b0 2024-09-16T01:45:16.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.289+0000 7f78f5ffb640 1 -- 172.21.15.88:0/2620062095 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13424 (secure 0 0 0) 0x7f79000b2c10 con 0x7f78e406c2b0 2024-09-16T01:45:16.291 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:45:16.291 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:45:16.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.291+0000 7f7907566640 1 -- 172.21.15.88:0/2620062095 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f78e406c2b0 msgr2=0x7f78e406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:16.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.291+0000 7f7907566640 1 --2- 172.21.15.88:0/2620062095 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f78e406c2b0 0x7f78e406e770 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f78fc006ca0 tx=0x7f78fc009240 comp rx=0 tx=0).stop 2024-09-16T01:45:16.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.291+0000 7f7907566640 1 -- 172.21.15.88:0/2620062095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f790015c330 msgr2=0x7f79000b3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:16.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.291+0000 7f7907566640 1 --2- 172.21.15.88:0/2620062095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f790015c330 0x7f79000b3d50 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7f78f800c760 tx=0x7f78f800cc20 comp rx=0 tx=0).stop 2024-09-16T01:45:16.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.291+0000 7f7907566640 1 -- 172.21.15.88:0/2620062095 shutdown_connections 2024-09-16T01:45:16.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.291+0000 7f7907566640 1 --2- 172.21.15.88:0/2620062095 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f78e406c2b0 0x7f78e406e770 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.291+0000 7f7907566640 1 --2- 172.21.15.88:0/2620062095 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f790015c330 0x7f79000b3d50 unknown :-1 s=CLOSED pgs=2441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.291+0000 7f7907566640 1 --2- 172.21.15.88:0/2620062095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7900155800 0x7f79000b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.291+0000 7f7907566640 1 --2- 172.21.15.88:0/2620062095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f79000cf570 0x7f7900147fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.291+0000 7f7907566640 1 -- 172.21.15.88:0/2620062095 >> 172.21.15.88:0/2620062095 conn(0x7f790006bc10 msgr2=0x7f79000ce3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:16.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.291+0000 7f7907566640 1 -- 172.21.15.88:0/2620062095 shutdown_connections 2024-09-16T01:45:16.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.291+0000 7f7907566640 1 -- 172.21.15.88:0/2620062095 wait complete. 2024-09-16T01:45:16.303 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":467,"stamp":"2024-09-16T01:45:14.660068+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":298004,"kb_used_data":15228,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754676716,"statfs":{"total":773094113280,"available":772788957184,"internally_reserved":0,"allocated":15593472,"data_stored":8143353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":289394226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-35304,"num_objects":-28,"num_object_clones":0,"num_object_copies":-56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-28,"num_whiteouts":0,"num_read":-83,"num_read_kb":-85,"num_write":-91,"num_write_kb":-67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001383"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":394,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T01:45:01.333733+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:45:01.333733+0000","last_peered":"2024-09-16T01:45:01.333733+0000","last_clean":"2024-09-16T01:45:01.333733+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:45:01.333733+0000","last_undegraded":"2024-09-16T01:45:01.333733+0000","last_fullsized":"2024-09-16T01:45:01.333733+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509577,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37108,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334732,"statfs":{"total":96636764160,"available":96598765568,"internally_reserved":0,"allocated":1798144,"data_stored":866868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542280,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37676,"kb_used_data":2328,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334164,"statfs":{"total":96636764160,"available":96598183936,"internally_reserved":0,"allocated":2383872,"data_stored":1451111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542280,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37096,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334744,"statfs":{"total":96636764160,"available":96598777856,"internally_reserved":0,"allocated":1794048,"data_stored":861728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542280,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32980,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338860,"statfs":{"total":96636764160,"available":96602992640,"internally_reserved":0,"allocated":1835008,"data_stored":895213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542279,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37136,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334704,"statfs":{"total":96636764160,"available":96598736896,"internally_reserved":0,"allocated":1830912,"data_stored":895095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542280,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37088,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334752,"statfs":{"total":96636764160,"available":96598786048,"internally_reserved":0,"allocated":1785856,"data_stored":860681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542279,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37088,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334752,"statfs":{"total":96636764160,"available":96598786048,"internally_reserved":0,"allocated":1785856,"data_stored":861076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542279,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41832,"kb_used_data":2324,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330008,"statfs":{"total":96636764160,"available":96593928192,"internally_reserved":0,"allocated":2379776,"data_stored":1451581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:45:16.304 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:45:16.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.401+0000 7f5b47eb7640 1 Processor -- start 2024-09-16T01:45:16.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.401+0000 7f5b47eb7640 1 -- start start 2024-09-16T01:45:16.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.402+0000 7f5b47eb7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b4015c2b0 0x7f5b400b6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:16.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.402+0000 7f5b47eb7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5b400b72f0 0x7f5b4015eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:16.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.402+0000 7f5b47eb7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b4015fb10 con 0x7f5b4015f030 2024-09-16T01:45:16.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.402+0000 7f5b47eb7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f5b400b5900 con 0x7f5b400b72f0 2024-09-16T01:45:16.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.402+0000 7f5b47eb7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5b4015f830 con 0x7f5b4015c2b0 2024-09-16T01:45:16.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.402+0000 7f5b45452640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5b400b72f0 0x7f5b4015eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:16.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.402+0000 7f5b45452640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5b400b72f0 0x7f5b4015eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60020/0 (socket says 172.21.15.88:60020) 2024-09-16T01:45:16.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.402+0000 7f5b45452640 1 -- 172.21.15.88:0/3828261108 learned_addr learned my addr 172.21.15.88:0/3828261108 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:16.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.402+0000 7f5b45c53640 1 --2- 172.21.15.88:0/3828261108 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b4015c2b0 0x7f5b400b6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:16.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.402+0000 7f5b45452640 1 -- 172.21.15.88:0/3828261108 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b4015c2b0 msgr2=0x7f5b400b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:16.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.402+0000 7f5b45452640 1 --2- 172.21.15.88:0/3828261108 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b4015c2b0 0x7f5b400b6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.402+0000 7f5b45452640 1 -- 172.21.15.88:0/3828261108 >> v1:172.21.15.154:6789/0 conn(0x7f5b4015f030 legacy=0x7f5b4006f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:16.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.402+0000 7f5b45452640 1 -- 172.21.15.88:0/3828261108 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b4006ce30 con 0x7f5b400b72f0 2024-09-16T01:45:16.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.402+0000 7f5b45c53640 1 --2- 172.21.15.88:0/3828261108 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b4015c2b0 0x7f5b400b6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:45:16.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.403+0000 7f5b45452640 1 --2- 172.21.15.88:0/3828261108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b400b72f0 0x7f5b4015eaf0 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f5b30009870 tx=0x7f5b3002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0715b364276f67 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:16.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.403+0000 7f5b44c51640 1 -- 172.21.15.88:0/3828261108 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b30037070 con 0x7f5b400b72f0 2024-09-16T01:45:16.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.403+0000 7f5b44c51640 1 -- 172.21.15.88:0/3828261108 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b30032440 con 0x7f5b400b72f0 2024-09-16T01:45:16.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.403+0000 7f5b44c51640 1 -- 172.21.15.88:0/3828261108 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b30032810 con 0x7f5b400b72f0 2024-09-16T01:45:16.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.404+0000 7f5b47eb7640 1 -- 172.21.15.88:0/3828261108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b400b72f0 msgr2=0x7f5b4015eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:16.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.404+0000 7f5b47eb7640 1 --2- 172.21.15.88:0/3828261108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b400b72f0 0x7f5b4015eaf0 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f5b30009870 tx=0x7f5b3002a210 comp rx=0 tx=0).stop 2024-09-16T01:45:16.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.404+0000 7f5b47eb7640 1 -- 172.21.15.88:0/3828261108 shutdown_connections 2024-09-16T01:45:16.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.404+0000 7f5b47eb7640 1 --2- 172.21.15.88:0/3828261108 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b400b72f0 0x7f5b4015eaf0 unknown :-1 s=CLOSED pgs=2442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.404+0000 7f5b47eb7640 1 --2- 172.21.15.88:0/3828261108 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b4015c2b0 0x7f5b400b6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.404+0000 7f5b47eb7640 1 -- 172.21.15.88:0/3828261108 >> 172.21.15.88:0/3828261108 conn(0x7f5b400b4240 msgr2=0x7f5b400b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:16.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.404+0000 7f5b47eb7640 1 -- 172.21.15.88:0/3828261108 shutdown_connections 2024-09-16T01:45:16.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.404+0000 7f5b47eb7640 1 -- 172.21.15.88:0/3828261108 wait complete. 2024-09-16T01:45:16.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.405+0000 7f5b47eb7640 1 Processor -- start 2024-09-16T01:45:16.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.405+0000 7f5b47eb7640 1 -- start start 2024-09-16T01:45:16.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.406+0000 7f5b47eb7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b400b72f0 0x7f5b40151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:16.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.406+0000 7f5b47eb7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b4015c2b0 0x7f5b400cc820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:16.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.406+0000 7f5b47eb7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b4015f030 0x7f5b400cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:16.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.406+0000 7f5b47eb7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5b400b7730 con 0x7f5b400b72f0 2024-09-16T01:45:16.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.406+0000 7f5b47eb7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5b4006d3f0 con 0x7f5b4015f030 2024-09-16T01:45:16.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.406+0000 7f5b47eb7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5b4015f4e0 con 0x7f5b4015c2b0 2024-09-16T01:45:16.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.406+0000 7f5b45c53640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b400b72f0 0x7f5b40151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:16.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.406+0000 7f5b45c53640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b400b72f0 0x7f5b40151ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60034/0 (socket says 172.21.15.88:60034) 2024-09-16T01:45:16.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.406+0000 7f5b45c53640 1 -- 172.21.15.88:0/1385978123 learned_addr learned my addr 172.21.15.88:0/1385978123 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:16.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.406+0000 7f5b45452640 1 --2- 172.21.15.88:0/1385978123 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b4015c2b0 0x7f5b400cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:16.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.406+0000 7f5b46454640 1 --2- 172.21.15.88:0/1385978123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b4015f030 0x7f5b400cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:16.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.406+0000 7f5b45c53640 1 -- 172.21.15.88:0/1385978123 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b4015c2b0 msgr2=0x7f5b400cc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:16.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.406+0000 7f5b45c53640 1 --2- 172.21.15.88:0/1385978123 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b4015c2b0 0x7f5b400cc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.406+0000 7f5b45c53640 1 -- 172.21.15.88:0/1385978123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b4015f030 msgr2=0x7f5b400cdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:16.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.406+0000 7f5b45c53640 1 --2- 172.21.15.88:0/1385978123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b4015f030 0x7f5b400cdcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.406+0000 7f5b45c53640 1 -- 172.21.15.88:0/1385978123 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b40152610 con 0x7f5b400b72f0 2024-09-16T01:45:16.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.406+0000 7f5b45c53640 1 --2- 172.21.15.88:0/1385978123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b400b72f0 0x7f5b40151ba0 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c00c7f0 tx=0x7f5b3c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:16.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.407+0000 7f5b367fc640 1 -- 172.21.15.88:0/1385978123 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c015070 con 0x7f5b400b72f0 2024-09-16T01:45:16.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.407+0000 7f5b367fc640 1 -- 172.21.15.88:0/1385978123 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b3c007a30 con 0x7f5b400b72f0 2024-09-16T01:45:16.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.407+0000 7f5b47eb7640 1 -- 172.21.15.88:0/1385978123 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b400ce830 con 0x7f5b400b72f0 2024-09-16T01:45:16.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.407+0000 7f5b367fc640 1 -- 172.21.15.88:0/1385978123 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c007e00 con 0x7f5b400b72f0 2024-09-16T01:45:16.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.407+0000 7f5b47eb7640 1 -- 172.21.15.88:0/1385978123 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b400cba60 con 0x7f5b400b72f0 2024-09-16T01:45:16.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.408+0000 7f5b47eb7640 1 -- 172.21.15.88:0/1385978123 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b400cc450 con 0x7f5b400b72f0 2024-09-16T01:45:16.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.408+0000 7f5b367fc640 1 -- 172.21.15.88:0/1385978123 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5b3c018070 con 0x7f5b400b72f0 2024-09-16T01:45:16.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.409+0000 7f5b367fc640 1 --2- 172.21.15.88:0/1385978123 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5b2406c2b0 0x7f5b2406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:16.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.409+0000 7f5b45452640 1 --2- 172.21.15.88:0/1385978123 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5b2406c2b0 0x7f5b2406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:16.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.409+0000 7f5b367fc640 1 -- 172.21.15.88:0/1385978123 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7f5b3c08b0a0 con 0x7f5b400b72f0 2024-09-16T01:45:16.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.409+0000 7f5b45452640 1 --2- 172.21.15.88:0/1385978123 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5b2406c2b0 0x7f5b2406e770 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f5b30034000 tx=0x7f5b3002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:16.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.413+0000 7f5b367fc640 1 -- 172.21.15.88:0/1385978123 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5b3c059460 con 0x7f5b400b72f0 2024-09-16T01:45:16.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.577+0000 7f5b47eb7640 1 -- 172.21.15.88:0/1385978123 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5b401520e0 con 0x7f5b2406c2b0 2024-09-16T01:45:16.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.578+0000 7f5b367fc640 1 -- 172.21.15.88:0/1385978123 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13424 (secure 0 0 0) 0x7f5b30030340 con 0x7f5b2406c2b0 2024-09-16T01:45:16.580 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:45:16.580 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:45:16.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.580+0000 7f5b47eb7640 1 -- 172.21.15.88:0/1385978123 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5b2406c2b0 msgr2=0x7f5b2406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:16.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.580+0000 7f5b47eb7640 1 --2- 172.21.15.88:0/1385978123 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5b2406c2b0 0x7f5b2406e770 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f5b30034000 tx=0x7f5b3002e000 comp rx=0 tx=0).stop 2024-09-16T01:45:16.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.580+0000 7f5b47eb7640 1 -- 172.21.15.88:0/1385978123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b400b72f0 msgr2=0x7f5b40151ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:16.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.580+0000 7f5b47eb7640 1 --2- 172.21.15.88:0/1385978123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b400b72f0 0x7f5b40151ba0 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c00c7f0 tx=0x7f5b3c00cbf0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.580+0000 7f5b47eb7640 1 -- 172.21.15.88:0/1385978123 shutdown_connections 2024-09-16T01:45:16.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.580+0000 7f5b47eb7640 1 --2- 172.21.15.88:0/1385978123 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5b2406c2b0 0x7f5b2406e770 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.580+0000 7f5b47eb7640 1 --2- 172.21.15.88:0/1385978123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b4015f030 0x7f5b400cdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.580+0000 7f5b47eb7640 1 --2- 172.21.15.88:0/1385978123 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b4015c2b0 0x7f5b400cc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.580+0000 7f5b47eb7640 1 --2- 172.21.15.88:0/1385978123 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b400b72f0 0x7f5b40151ba0 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.580+0000 7f5b47eb7640 1 -- 172.21.15.88:0/1385978123 >> 172.21.15.88:0/1385978123 conn(0x7f5b400b4240 msgr2=0x7f5b400b3480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:16.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.581+0000 7f5b47eb7640 1 -- 172.21.15.88:0/1385978123 shutdown_connections 2024-09-16T01:45:16.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.581+0000 7f5b47eb7640 1 -- 172.21.15.88:0/1385978123 wait complete. 2024-09-16T01:45:16.592 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":467,"stamp":"2024-09-16T01:45:14.660068+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":298004,"kb_used_data":15228,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754676716,"statfs":{"total":773094113280,"available":772788957184,"internally_reserved":0,"allocated":15593472,"data_stored":8143353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":289394226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-35304,"num_objects":-28,"num_object_clones":0,"num_object_copies":-56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-28,"num_whiteouts":0,"num_read":-83,"num_read_kb":-85,"num_write":-91,"num_write_kb":-67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001383"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":394,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T01:45:01.333733+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:45:01.333733+0000","last_peered":"2024-09-16T01:45:01.333733+0000","last_clean":"2024-09-16T01:45:01.333733+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:45:01.333733+0000","last_undegraded":"2024-09-16T01:45:01.333733+0000","last_fullsized":"2024-09-16T01:45:01.333733+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509577,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37108,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334732,"statfs":{"total":96636764160,"available":96598765568,"internally_reserved":0,"allocated":1798144,"data_stored":866868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542280,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37676,"kb_used_data":2328,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334164,"statfs":{"total":96636764160,"available":96598183936,"internally_reserved":0,"allocated":2383872,"data_stored":1451111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542280,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37096,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334744,"statfs":{"total":96636764160,"available":96598777856,"internally_reserved":0,"allocated":1794048,"data_stored":861728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542280,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32980,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338860,"statfs":{"total":96636764160,"available":96602992640,"internally_reserved":0,"allocated":1835008,"data_stored":895213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542279,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37136,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334704,"statfs":{"total":96636764160,"available":96598736896,"internally_reserved":0,"allocated":1830912,"data_stored":895095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542280,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37088,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334752,"statfs":{"total":96636764160,"available":96598786048,"internally_reserved":0,"allocated":1785856,"data_stored":860681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542279,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37088,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334752,"statfs":{"total":96636764160,"available":96598786048,"internally_reserved":0,"allocated":1785856,"data_stored":861076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542279,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41832,"kb_used_data":2324,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330008,"statfs":{"total":96636764160,"available":96593928192,"internally_reserved":0,"allocated":2379776,"data_stored":1451581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:45:16.593 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T01:45:16.593 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:45:16.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.691+0000 7f0685c65640 1 Processor -- start 2024-09-16T01:45:16.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.691+0000 7f0685c65640 1 -- start start 2024-09-16T01:45:16.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.691+0000 7f0685c65640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0680155800 0x7f06800b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:16.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.691+0000 7f0685c65640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06800b50a0 con 0x7f06800cfc80 2024-09-16T01:45:16.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.691+0000 7f0685c65640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06800b5380 con 0x7f068015c330 2024-09-16T01:45:16.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.692+0000 7f0685c65640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f068015c7a0 con 0x7f0680155800 2024-09-16T01:45:16.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.692+0000 7f067f7fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f0680155800 0x7f06800b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:16.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.692+0000 7f067ffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f06800cfc80 0x7f068015c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59980/0 (socket says 172.21.15.88:59980) 2024-09-16T01:45:16.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.692+0000 7f067ffff640 1 -- 172.21.15.88:0/84337947 learned_addr learned my addr 172.21.15.88:0/84337947 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:16.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.692+0000 7f067f7fe640 1 -- 172.21.15.88:0/84337947 >> v1:172.21.15.154:6789/0 conn(0x7f068015c330 legacy=0x7f06800cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:16.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.692+0000 7f067f7fe640 1 -- 172.21.15.88:0/84337947 >> v1:172.21.15.88:6789/0 conn(0x7f06800cfc80 legacy=0x7f068015c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:16.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.692+0000 7f067e7fc640 1 -- 172.21.15.88:0/84337947 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3914989768 0 0) 0x7f06800b50a0 con 0x7f06800cfc80 2024-09-16T01:45:16.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.692+0000 7f067f7fe640 1 -- 172.21.15.88:0/84337947 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06800b5660 con 0x7f0680155800 2024-09-16T01:45:16.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.693+0000 7f067f7fe640 1 --2- 172.21.15.88:0/84337947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0680155800 0x7f06800b6140 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7f06700097a0 tx=0x7f067002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd04f4be6cafadc4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:16.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.693+0000 7f067e7fc640 1 -- 172.21.15.88:0/84337947 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0670037070 con 0x7f0680155800 2024-09-16T01:45:16.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.693+0000 7f067e7fc640 1 -- 172.21.15.88:0/84337947 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0670032440 con 0x7f0680155800 2024-09-16T01:45:16.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.693+0000 7f067e7fc640 1 -- 172.21.15.88:0/84337947 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0670032810 con 0x7f0680155800 2024-09-16T01:45:16.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.693+0000 7f0685c65640 1 -- 172.21.15.88:0/84337947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0680155800 msgr2=0x7f06800b6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:16.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.693+0000 7f0685c65640 1 --2- 172.21.15.88:0/84337947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0680155800 0x7f06800b6140 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7f06700097a0 tx=0x7f067002a1e0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.693+0000 7f0685c65640 1 -- 172.21.15.88:0/84337947 shutdown_connections 2024-09-16T01:45:16.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.693+0000 7f0685c65640 1 --2- 172.21.15.88:0/84337947 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0680155800 0x7f06800b6140 unknown :-1 s=CLOSED pgs=2445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.693+0000 7f0685c65640 1 -- 172.21.15.88:0/84337947 >> 172.21.15.88:0/84337947 conn(0x7f068006bc10 msgr2=0x7f068006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:16.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.693+0000 7f0685c65640 1 -- 172.21.15.88:0/84337947 shutdown_connections 2024-09-16T01:45:16.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.693+0000 7f0685c65640 1 -- 172.21.15.88:0/84337947 wait complete. 2024-09-16T01:45:16.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.694+0000 7f0685c65640 1 Processor -- start 2024-09-16T01:45:16.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.695+0000 7f0685c65640 1 -- start start 2024-09-16T01:45:16.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.695+0000 7f0685c65640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06800cfc80 0x7f06801385e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:16.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.695+0000 7f0685c65640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0680155800 0x7f06800cdab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:16.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.695+0000 7f0685c65640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f068015c330 0x7f06800cdff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:16.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.695+0000 7f0685c65640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0680155be0 con 0x7f068015c330 2024-09-16T01:45:16.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.695+0000 7f0685c65640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f06800b5c20 con 0x7f0680155800 2024-09-16T01:45:16.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.695+0000 7f0685c65640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f068015d080 con 0x7f06800cfc80 2024-09-16T01:45:16.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.695+0000 7f067ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f068015c330 0x7f06800cdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:16.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.696+0000 7f067ffff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f068015c330 0x7f06800cdff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60050/0 (socket says 172.21.15.88:60050) 2024-09-16T01:45:16.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.696+0000 7f067ffff640 1 -- 172.21.15.88:0/177319248 learned_addr learned my addr 172.21.15.88:0/177319248 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:16.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.696+0000 7f067f7fe640 1 --2- 172.21.15.88:0/177319248 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06800cfc80 0x7f06801385e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:16.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.696+0000 7f067effd640 1 --2- 172.21.15.88:0/177319248 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0680155800 0x7f06800cdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:16.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.696+0000 7f067ffff640 1 -- 172.21.15.88:0/177319248 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06800cfc80 msgr2=0x7f06801385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:16.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.696+0000 7f067ffff640 1 --2- 172.21.15.88:0/177319248 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06800cfc80 0x7f06801385e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.696+0000 7f067ffff640 1 -- 172.21.15.88:0/177319248 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0680155800 msgr2=0x7f06800cdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:16.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.696+0000 7f067ffff640 1 --2- 172.21.15.88:0/177319248 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0680155800 0x7f06800cdab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.696+0000 7f067ffff640 1 -- 172.21.15.88:0/177319248 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06801403d0 con 0x7f068015c330 2024-09-16T01:45:16.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.696+0000 7f067ffff640 1 --2- 172.21.15.88:0/177319248 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f068015c330 0x7f06800cdff0 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f067800c800 tx=0x7f067800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:16.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.696+0000 7f0667fff640 1 -- 172.21.15.88:0/177319248 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0678018070 con 0x7f068015c330 2024-09-16T01:45:16.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.696+0000 7f0667fff640 1 -- 172.21.15.88:0/177319248 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0678004d80 con 0x7f068015c330 2024-09-16T01:45:16.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.696+0000 7f0685c65640 1 -- 172.21.15.88:0/177319248 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06801406b0 con 0x7f068015c330 2024-09-16T01:45:16.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.696+0000 7f0667fff640 1 -- 172.21.15.88:0/177319248 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067800a880 con 0x7f068015c330 2024-09-16T01:45:16.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.696+0000 7f0685c65640 1 -- 172.21.15.88:0/177319248 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0680140c70 con 0x7f068015c330 2024-09-16T01:45:16.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.697+0000 7f0685c65640 1 -- 172.21.15.88:0/177319248 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f068015e590 con 0x7f068015c330 2024-09-16T01:45:16.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.698+0000 7f0667fff640 1 -- 172.21.15.88:0/177319248 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f067801b070 con 0x7f068015c330 2024-09-16T01:45:16.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.698+0000 7f0667fff640 1 --2- 172.21.15.88:0/177319248 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f066c06c2b0 0x7f066c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:16.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.698+0000 7f067f7fe640 1 --2- 172.21.15.88:0/177319248 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f066c06c2b0 0x7f066c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:16.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.698+0000 7f0667fff640 1 -- 172.21.15.88:0/177319248 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7f067808a5a0 con 0x7f068015c330 2024-09-16T01:45:16.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.699+0000 7f067f7fe640 1 --2- 172.21.15.88:0/177319248 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f066c06c2b0 0x7f066c06e770 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f0670034000 tx=0x7f067002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:16.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.702+0000 7f0667fff640 1 -- 172.21.15.88:0/177319248 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0678058ab0 con 0x7f068015c330 2024-09-16T01:45:16.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.866+0000 7f0685c65640 1 -- 172.21.15.88:0/177319248 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f06801497a0 con 0x7f066c06c2b0 2024-09-16T01:45:16.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.867+0000 7f0667fff640 1 -- 172.21.15.88:0/177319248 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13497 (secure 0 0 0) 0x7f0670032750 con 0x7f066c06c2b0 2024-09-16T01:45:16.868 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:45:16.869 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:45:16.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.869+0000 7f0685c65640 1 -- 172.21.15.88:0/177319248 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f066c06c2b0 msgr2=0x7f066c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:16.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.869+0000 7f0685c65640 1 --2- 172.21.15.88:0/177319248 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f066c06c2b0 0x7f066c06e770 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f0670034000 tx=0x7f067002e000 comp rx=0 tx=0).stop 2024-09-16T01:45:16.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.869+0000 7f0685c65640 1 -- 172.21.15.88:0/177319248 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f068015c330 msgr2=0x7f06800cdff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:16.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.869+0000 7f0685c65640 1 --2- 172.21.15.88:0/177319248 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f068015c330 0x7f06800cdff0 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f067800c800 tx=0x7f067800ccc0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.869+0000 7f0685c65640 1 -- 172.21.15.88:0/177319248 shutdown_connections 2024-09-16T01:45:16.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.869+0000 7f0685c65640 1 --2- 172.21.15.88:0/177319248 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f066c06c2b0 0x7f066c06e770 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.869+0000 7f0685c65640 1 --2- 172.21.15.88:0/177319248 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f068015c330 0x7f06800cdff0 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.869+0000 7f0685c65640 1 --2- 172.21.15.88:0/177319248 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0680155800 0x7f06800cdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.869+0000 7f0685c65640 1 --2- 172.21.15.88:0/177319248 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f06800cfc80 0x7f06801385e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.869+0000 7f0685c65640 1 -- 172.21.15.88:0/177319248 >> 172.21.15.88:0/177319248 conn(0x7f068006bc10 msgr2=0x7f06800cf570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:16.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.869+0000 7f0685c65640 1 -- 172.21.15.88:0/177319248 shutdown_connections 2024-09-16T01:45:16.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.869+0000 7f0685c65640 1 -- 172.21.15.88:0/177319248 wait complete. 2024-09-16T01:45:16.881 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-09-16T01:45:16.660474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":113,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":298232,"kb_used_data":15432,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754676488,"statfs":{"total":773094113280,"available":772788723712,"internally_reserved":0,"allocated":15802368,"data_stored":8209040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":289394226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-35701,"num_objects":-28,"num_object_clones":0,"num_object_copies":-56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-28,"num_whiteouts":0,"num_read":-84,"num_read_kb":-86,"num_write":-95,"num_write_kb":-70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001497"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":400,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T01:45:13.941508+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:45:13.941508+0000","last_peered":"2024-09-16T01:45:13.941508+0000","last_clean":"2024-09-16T01:45:13.941508+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:45:13.941508+0000","last_undegraded":"2024-09-16T01:45:13.941508+0000","last_fullsized":"2024-09-16T01:45:13.941508+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509578,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37132,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334708,"statfs":{"total":96636764160,"available":96598740992,"internally_reserved":0,"allocated":1826816,"data_stored":878538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542281,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37716,"kb_used_data":2364,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334124,"statfs":{"total":96636764160,"available":96598142976,"internally_reserved":0,"allocated":2420736,"data_stored":1468906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542281,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37132,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334708,"statfs":{"total":96636764160,"available":96598740992,"internally_reserved":0,"allocated":1826816,"data_stored":878538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542281,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32972,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1826816,"data_stored":878538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542280,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37136,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334704,"statfs":{"total":96636764160,"available":96598736896,"internally_reserved":0,"allocated":1826816,"data_stored":878538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542281,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37136,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334704,"statfs":{"total":96636764160,"available":96598736896,"internally_reserved":0,"allocated":1826816,"data_stored":878538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542280,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37132,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334708,"statfs":{"total":96636764160,"available":96598740992,"internally_reserved":0,"allocated":1826816,"data_stored":878538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542280,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41876,"kb_used_data":2364,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329964,"statfs":{"total":96636764160,"available":96593883136,"internally_reserved":0,"allocated":2420736,"data_stored":1468906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:45:16.882 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T01:45:16.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.980+0000 7f5de35b7640 1 Processor -- start 2024-09-16T01:45:16.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.980+0000 7f5de35b7640 1 -- start start 2024-09-16T01:45:16.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.981+0000 7f5de35b7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5ddc15c170 0x7f5ddc15c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:16.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.981+0000 7f5de35b7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5ddc0cf6c0 0x7f5ddc0cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:16.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.981+0000 7f5de35b7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ddc0b49c0 con 0x7f5ddc0b5dd0 2024-09-16T01:45:16.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.981+0000 7f5de35b7640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f5ddc15ca90 con 0x7f5ddc15c170 2024-09-16T01:45:16.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.981+0000 7f5de35b7640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5ddc0b4cf0 con 0x7f5ddc0cf6c0 2024-09-16T01:45:16.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.981+0000 7f5de1353640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f5ddc15c170 0x7f5ddc15c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:16.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.981+0000 7f5de1b54640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5ddc0b5dd0 0x7f5ddc15ffb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:59994/0 (socket says 172.21.15.88:59994) 2024-09-16T01:45:16.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.981+0000 7f5de1b54640 1 -- 172.21.15.88:0/2786774519 learned_addr learned my addr 172.21.15.88:0/2786774519 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:16.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.981+0000 7f5de0b52640 1 --2- 172.21.15.88:0/2786774519 >> v2:172.21.15.154:3300/0 conn(0x7f5ddc0cf6c0 0x7f5ddc0cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:16.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.981+0000 7f5de1353640 1 -- 172.21.15.88:0/2786774519 >> v2:172.21.15.154:3300/0 conn(0x7f5ddc0cf6c0 msgr2=0x7f5ddc0cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:16.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.981+0000 7f5de1353640 1 --2- 172.21.15.88:0/2786774519 >> v2:172.21.15.154:3300/0 conn(0x7f5ddc0cf6c0 0x7f5ddc0cfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.981+0000 7f5de1353640 1 -- 172.21.15.88:0/2786774519 >> v1:172.21.15.88:6789/0 conn(0x7f5ddc0b5dd0 legacy=0x7f5ddc15ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:16.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.981+0000 7f5dd3fff640 1 -- 172.21.15.88:0/2786774519 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3258962047 0 0) 0x7f5ddc0b49c0 con 0x7f5ddc0b5dd0 2024-09-16T01:45:16.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.981+0000 7f5de1353640 1 -- 172.21.15.88:0/2786774519 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ddc0b4fd0 con 0x7f5ddc15c170 2024-09-16T01:45:16.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.982+0000 7f5de1353640 1 --2- 172.21.15.88:0/2786774519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ddc15c170 0x7f5ddc15c550 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f5dd80097a0 tx=0x7f5dd802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=268f144a5cbe663d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:16.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.982+0000 7f5dd3fff640 1 -- 172.21.15.88:0/2786774519 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd8037070 con 0x7f5ddc15c170 2024-09-16T01:45:16.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.982+0000 7f5dd3fff640 1 -- 172.21.15.88:0/2786774519 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dd8032440 con 0x7f5ddc15c170 2024-09-16T01:45:16.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.982+0000 7f5dd3fff640 1 -- 172.21.15.88:0/2786774519 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd8032810 con 0x7f5ddc15c170 2024-09-16T01:45:16.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.982+0000 7f5de35b7640 1 -- 172.21.15.88:0/2786774519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ddc15c170 msgr2=0x7f5ddc15c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:16.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.982+0000 7f5de35b7640 1 --2- 172.21.15.88:0/2786774519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ddc15c170 0x7f5ddc15c550 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f5dd80097a0 tx=0x7f5dd802a1e0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.983+0000 7f5de35b7640 1 -- 172.21.15.88:0/2786774519 shutdown_connections 2024-09-16T01:45:16.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.983+0000 7f5de35b7640 1 --2- 172.21.15.88:0/2786774519 >> v2:172.21.15.154:3300/0 conn(0x7f5ddc0cf6c0 0x7f5ddc0cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.983+0000 7f5de35b7640 1 --2- 172.21.15.88:0/2786774519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ddc15c170 0x7f5ddc15c550 unknown :-1 s=CLOSED pgs=2448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.983+0000 7f5de35b7640 1 -- 172.21.15.88:0/2786774519 >> 172.21.15.88:0/2786774519 conn(0x7f5ddc06bc10 msgr2=0x7f5ddc06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:16.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.983+0000 7f5de35b7640 1 -- 172.21.15.88:0/2786774519 shutdown_connections 2024-09-16T01:45:16.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.983+0000 7f5de35b7640 1 -- 172.21.15.88:0/2786774519 wait complete. 2024-09-16T01:45:16.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.984+0000 7f5de35b7640 1 Processor -- start 2024-09-16T01:45:16.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.984+0000 7f5de35b7640 1 -- start start 2024-09-16T01:45:16.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.985+0000 7f5de35b7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ddc0b5dd0 0x7f5ddc0c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:16.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.985+0000 7f5de35b7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ddc0cf6c0 0x7f5ddc14ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:16.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.985+0000 7f5de35b7640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ddc15c170 0x7f5ddc0cc350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:16.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.985+0000 7f5de35b7640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f5ddc0b5870 con 0x7f5ddc0cf6c0 2024-09-16T01:45:16.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.985+0000 7f5de35b7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5ddc0b5590 con 0x7f5ddc0b5dd0 2024-09-16T01:45:16.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.985+0000 7f5de35b7640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5ddc0b5b50 con 0x7f5ddc15c170 2024-09-16T01:45:16.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.985+0000 7f5de0b52640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ddc0cf6c0 0x7f5ddc14ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:16.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.985+0000 7f5de0b52640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ddc0cf6c0 0x7f5ddc14ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60066/0 (socket says 172.21.15.88:60066) 2024-09-16T01:45:16.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.985+0000 7f5de0b52640 1 -- 172.21.15.88:0/4821536 learned_addr learned my addr 172.21.15.88:0/4821536 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:16.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.985+0000 7f5de1353640 1 --2- 172.21.15.88:0/4821536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ddc0b5dd0 0x7f5ddc0c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:16.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.985+0000 7f5de1b54640 1 --2- 172.21.15.88:0/4821536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ddc15c170 0x7f5ddc0cc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:16.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.985+0000 7f5de0b52640 1 -- 172.21.15.88:0/4821536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ddc15c170 msgr2=0x7f5ddc0cc350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:16.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.985+0000 7f5de0b52640 1 --2- 172.21.15.88:0/4821536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ddc15c170 0x7f5ddc0cc350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.985+0000 7f5de0b52640 1 -- 172.21.15.88:0/4821536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ddc0b5dd0 msgr2=0x7f5ddc0c8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:16.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.985+0000 7f5de0b52640 1 --2- 172.21.15.88:0/4821536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ddc0b5dd0 0x7f5ddc0c8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:16.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.985+0000 7f5de0b52640 1 -- 172.21.15.88:0/4821536 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ddc0cb2b0 con 0x7f5ddc0cf6c0 2024-09-16T01:45:16.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.985+0000 7f5de0b52640 1 --2- 172.21.15.88:0/4821536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ddc0cf6c0 0x7f5ddc14ef60 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc00d800 tx=0x7f5dcc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:16.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.985+0000 7f5dd1ffb640 1 -- 172.21.15.88:0/4821536 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc00b6c0 con 0x7f5ddc0cf6c0 2024-09-16T01:45:16.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.985+0000 7f5dd1ffb640 1 -- 172.21.15.88:0/4821536 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dcc019070 con 0x7f5ddc0cf6c0 2024-09-16T01:45:16.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.985+0000 7f5de35b7640 1 -- 172.21.15.88:0/4821536 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ddc0cb590 con 0x7f5ddc0cf6c0 2024-09-16T01:45:16.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.985+0000 7f5dd1ffb640 1 -- 172.21.15.88:0/4821536 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc00ba80 con 0x7f5ddc0cf6c0 2024-09-16T01:45:16.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.986+0000 7f5de35b7640 1 -- 172.21.15.88:0/4821536 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ddc0cbb50 con 0x7f5ddc0cf6c0 2024-09-16T01:45:16.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.986+0000 7f5de35b7640 1 -- 172.21.15.88:0/4821536 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ddc0cfb30 con 0x7f5ddc0cf6c0 2024-09-16T01:45:16.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.987+0000 7f5dd1ffb640 1 -- 172.21.15.88:0/4821536 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f5dcc00bc20 con 0x7f5ddc0cf6c0 2024-09-16T01:45:16.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.987+0000 7f5dd1ffb640 1 --2- 172.21.15.88:0/4821536 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5dc006c2b0 0x7f5dc006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:16.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.987+0000 7f5dd1ffb640 1 -- 172.21.15.88:0/4821536 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7f5dcc08b650 con 0x7f5ddc0cf6c0 2024-09-16T01:45:16.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.987+0000 7f5de1353640 1 --2- 172.21.15.88:0/4821536 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5dc006c2b0 0x7f5dc006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:16.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.988+0000 7f5de1353640 1 --2- 172.21.15.88:0/4821536 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5dc006c2b0 0x7f5dc006e770 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8034000 tx=0x7f5dd802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:16.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:16.992+0000 7f5dd1ffb640 1 -- 172.21.15.88:0/4821536 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f5dcc059fd0 con 0x7f5ddc0cf6c0 2024-09-16T01:45:17.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:17.158+0000 7f5de35b7640 1 -- 172.21.15.88:0/4821536 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5ddc1526f0 con 0x7f5ddc0cf6c0 2024-09-16T01:45:17.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:17.158+0000 7f5dd1ffb640 1 -- 172.21.15.88:0/4821536 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v142) ==== 74+0+13146 (secure 0 0 0) 0x7f5dcc0598d0 con 0x7f5ddc0cf6c0 2024-09-16T01:45:17.160 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:45:17.160 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":142,"fsid":"5ace2c91-46f9-482b-8a89-731fbdc171bf","created":"2024-09-16T01:33:58.412606+0000","modified":"2024-09-16T01:45:13.925175+0000","last_up_change":"2024-09-16T01:34:10.944097+0000","last_in_change":"2024-09-16T01:34:00.481168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T01:34:10.547178+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"29743152-8c11-4c1d-81bd-f6945c3d1401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6817","nonce":903680192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6819","nonce":903680192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6823","nonce":903680192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":903680192},{"type":"v1","addr":"172.21.15.88:6821","nonce":903680192}]},"public_addr":"172.21.15.88:6817/903680192","cluster_addr":"172.21.15.88:6819/903680192","heartbeat_back_addr":"172.21.15.88:6823/903680192","heartbeat_front_addr":"172.21.15.88:6821/903680192","state":["exists","up"]},{"osd":1,"uuid":"191015d7-d294-4241-81e8-509f25cc024d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6825","nonce":3621315521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6827","nonce":3621315521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6831","nonce":3621315521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3621315521},{"type":"v1","addr":"172.21.15.88:6829","nonce":3621315521}]},"public_addr":"172.21.15.88:6825/3621315521","cluster_addr":"172.21.15.88:6827/3621315521","heartbeat_back_addr":"172.21.15.88:6831/3621315521","heartbeat_front_addr":"172.21.15.88:6829/3621315521","state":["exists","up"]},{"osd":2,"uuid":"ba5bc56d-a42c-4ba7-a392-40afa4a9cf04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6809","nonce":2918260303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6811","nonce":2918260303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6815","nonce":2918260303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":2918260303},{"type":"v1","addr":"172.21.15.88:6813","nonce":2918260303}]},"public_addr":"172.21.15.88:6809/2918260303","cluster_addr":"172.21.15.88:6811/2918260303","heartbeat_back_addr":"172.21.15.88:6815/2918260303","heartbeat_front_addr":"172.21.15.88:6813/2918260303","state":["exists","up"]},{"osd":3,"uuid":"b6d9e2eb-da9c-4d8a-87df-b3a30a5c663a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6801","nonce":697609987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6803","nonce":697609987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6807","nonce":697609987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":697609987},{"type":"v1","addr":"172.21.15.88:6805","nonce":697609987}]},"public_addr":"172.21.15.88:6801/697609987","cluster_addr":"172.21.15.88:6803/697609987","heartbeat_back_addr":"172.21.15.88:6807/697609987","heartbeat_front_addr":"172.21.15.88:6805/697609987","state":["exists","up"]},{"osd":4,"uuid":"cb4dded8-72ed-4ac9-8761-39dd4b43692c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6817","nonce":2520717231}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6819","nonce":2520717231}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6823","nonce":2520717231}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2520717231},{"type":"v1","addr":"172.21.15.154:6821","nonce":2520717231}]},"public_addr":"172.21.15.154:6817/2520717231","cluster_addr":"172.21.15.154:6819/2520717231","heartbeat_back_addr":"172.21.15.154:6823/2520717231","heartbeat_front_addr":"172.21.15.154:6821/2520717231","state":["exists","up"]},{"osd":5,"uuid":"35e0ad41-838d-43c4-b049-c158fe2b5da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6809","nonce":280070533}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6811","nonce":280070533}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6815","nonce":280070533}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":280070533},{"type":"v1","addr":"172.21.15.154:6813","nonce":280070533}]},"public_addr":"172.21.15.154:6809/280070533","cluster_addr":"172.21.15.154:6811/280070533","heartbeat_back_addr":"172.21.15.154:6815/280070533","heartbeat_front_addr":"172.21.15.154:6813/280070533","state":["exists","up"]},{"osd":6,"uuid":"fdbaae79-3ed1-45bb-a292-2dd48d3aaffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6801","nonce":189356541}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6803","nonce":189356541}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6807","nonce":189356541}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":189356541},{"type":"v1","addr":"172.21.15.154:6805","nonce":189356541}]},"public_addr":"172.21.15.154:6801/189356541","cluster_addr":"172.21.15.154:6803/189356541","heartbeat_back_addr":"172.21.15.154:6807/189356541","heartbeat_front_addr":"172.21.15.154:6805/189356541","state":["exists","up"]},{"osd":7,"uuid":"f6f6b035-c28a-4a9a-bff5-13cf359c2eac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6825","nonce":2200266987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6827","nonce":2200266987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6831","nonce":2200266987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2200266987},{"type":"v1","addr":"172.21.15.154:6829","nonce":2200266987}]},"public_addr":"172.21.15.154:6825/2200266987","cluster_addr":"172.21.15.154:6827/2200266987","heartbeat_back_addr":"172.21.15.154:6831/2200266987","heartbeat_front_addr":"172.21.15.154:6829/2200266987","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.441593+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:08.012270+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.794380+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.026727+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:09.049034+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.805694+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.289315+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T01:34:07.880551+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:6836/4245333273":"2024-09-17T01:45:10.920064+0000","172.21.15.88:6834/4245333273":"2024-09-17T01:45:10.920064+0000","172.21.15.88:6836/656828696":"2024-09-17T01:45:01.261737+0000","172.21.15.88:6834/656828696":"2024-09-17T01:45:01.261737+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T01:45:17.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:17.160+0000 7f5de35b7640 1 -- 172.21.15.88:0/4821536 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5dc006c2b0 msgr2=0x7f5dc006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:17.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:17.160+0000 7f5de35b7640 1 --2- 172.21.15.88:0/4821536 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5dc006c2b0 0x7f5dc006e770 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8034000 tx=0x7f5dd802e000 comp rx=0 tx=0).stop 2024-09-16T01:45:17.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:17.160+0000 7f5de35b7640 1 -- 172.21.15.88:0/4821536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ddc0cf6c0 msgr2=0x7f5ddc14ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:17.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:17.160+0000 7f5de35b7640 1 --2- 172.21.15.88:0/4821536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ddc0cf6c0 0x7f5ddc14ef60 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc00d800 tx=0x7f5dcc00dcc0 comp rx=0 tx=0).stop 2024-09-16T01:45:17.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:17.161+0000 7f5de35b7640 1 -- 172.21.15.88:0/4821536 shutdown_connections 2024-09-16T01:45:17.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:17.161+0000 7f5de35b7640 1 --2- 172.21.15.88:0/4821536 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f5dc006c2b0 0x7f5dc006e770 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:17.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:17.161+0000 7f5de35b7640 1 --2- 172.21.15.88:0/4821536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5ddc15c170 0x7f5ddc0cc350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:17.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:17.161+0000 7f5de35b7640 1 --2- 172.21.15.88:0/4821536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ddc0cf6c0 0x7f5ddc14ef60 unknown :-1 s=CLOSED pgs=2449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:17.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:17.161+0000 7f5de35b7640 1 --2- 172.21.15.88:0/4821536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ddc0b5dd0 0x7f5ddc0c8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:17.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:17.161+0000 7f5de35b7640 1 -- 172.21.15.88:0/4821536 >> 172.21.15.88:0/4821536 conn(0x7f5ddc06bc10 msgr2=0x7f5ddc0b27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:17.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:17.161+0000 7f5de35b7640 1 -- 172.21.15.88:0/4821536 shutdown_connections 2024-09-16T01:45:17.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:17.161+0000 7f5de35b7640 1 -- 172.21.15.88:0/4821536 wait complete. 2024-09-16T01:45:18.173 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T01:45:18.174 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T01:45:18.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.286+0000 7fc0bba1e640 1 Processor -- start 2024-09-16T01:45:18.288 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.287+0000 7fc0bba1e640 1 -- start start 2024-09-16T01:45:18.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.287+0000 7fc0bba1e640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc0b40b3920 0x7fc0b4162b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.287+0000 7fc0bba1e640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0b415ed70 con 0x7fc0b40d29a0 2024-09-16T01:45:18.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.287+0000 7fc0bba1e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0b40d1fa0 con 0x7fc0b415e990 2024-09-16T01:45:18.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.287+0000 7fc0bba1e640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fc0b40cff20 con 0x7fc0b40b3920 2024-09-16T01:45:18.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.288+0000 7fc0b8fb9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fc0b40b3920 0x7fc0b4162b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.288+0000 7fc0b9fbb640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc0b40d29a0 0x7fc0b40d2e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60008/0 (socket says 172.21.15.88:60008) 2024-09-16T01:45:18.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.288+0000 7fc0b9fbb640 1 -- 172.21.15.88:0/1521825026 learned_addr learned my addr 172.21.15.88:0/1521825026 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:18.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.288+0000 7fc0b8fb9640 1 -- 172.21.15.88:0/1521825026 >> v1:172.21.15.154:6789/0 conn(0x7fc0b415e990 legacy=0x7fc0b4162450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:18.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.288+0000 7fc0b8fb9640 1 -- 172.21.15.88:0/1521825026 >> v1:172.21.15.88:6789/0 conn(0x7fc0b40d29a0 legacy=0x7fc0b40d2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:18.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.288+0000 7fc0b8fb9640 1 -- 172.21.15.88:0/1521825026 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0b40cf960 con 0x7fc0b40b3920 2024-09-16T01:45:18.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.288+0000 7fc0abfff640 1 -- 172.21.15.88:0/1521825026 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 459295446 0 0) 0x7fc0ac001490 con 0x7fc0b40d29a0 2024-09-16T01:45:18.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.289+0000 7fc0b8fb9640 1 --2- 172.21.15.88:0/1521825026 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0b40b3920 0x7fc0b4162b60 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7fc0a40097a0 tx=0x7fc0a402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76a8bcd4a5695688 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:18.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.289+0000 7fc0abfff640 1 -- 172.21.15.88:0/1521825026 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0a4037070 con 0x7fc0b40b3920 2024-09-16T01:45:18.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.289+0000 7fc0abfff640 1 -- 172.21.15.88:0/1521825026 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0a402ae50 con 0x7fc0b40b3920 2024-09-16T01:45:18.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.289+0000 7fc0abfff640 1 -- 172.21.15.88:0/1521825026 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0a4032660 con 0x7fc0b40b3920 2024-09-16T01:45:18.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.289+0000 7fc0bba1e640 1 -- 172.21.15.88:0/1521825026 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0b40b3920 msgr2=0x7fc0b4162b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:18.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.289+0000 7fc0bba1e640 1 --2- 172.21.15.88:0/1521825026 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0b40b3920 0x7fc0b4162b60 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7fc0a40097a0 tx=0x7fc0a402a140 comp rx=0 tx=0).stop 2024-09-16T01:45:18.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.289+0000 7fc0bba1e640 1 -- 172.21.15.88:0/1521825026 shutdown_connections 2024-09-16T01:45:18.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.289+0000 7fc0bba1e640 1 --2- 172.21.15.88:0/1521825026 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0b40b3920 0x7fc0b4162b60 unknown :-1 s=CLOSED pgs=2451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.289+0000 7fc0bba1e640 1 -- 172.21.15.88:0/1521825026 >> 172.21.15.88:0/1521825026 conn(0x7fc0b40d1200 msgr2=0x7fc0b40d1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:18.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.289+0000 7fc0bba1e640 1 -- 172.21.15.88:0/1521825026 shutdown_connections 2024-09-16T01:45:18.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.289+0000 7fc0bba1e640 1 -- 172.21.15.88:0/1521825026 wait complete. 2024-09-16T01:45:18.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.291+0000 7fc0bba1e640 1 Processor -- start 2024-09-16T01:45:18.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.291+0000 7fc0bba1e640 1 -- start start 2024-09-16T01:45:18.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.291+0000 7fc0bba1e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0b40b3920 0x7fc0b414c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.291+0000 7fc0bba1e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0b40d29a0 0x7fc0b414f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.291+0000 7fc0bba1e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b415e990 0x7fc0b40cb850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.291+0000 7fc0bba1e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fc0b41630a0 con 0x7fc0b40b3920 2024-09-16T01:45:18.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.291+0000 7fc0bba1e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc0b40b3d60 con 0x7fc0b415e990 2024-09-16T01:45:18.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.291+0000 7fc0bba1e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc0b4163220 con 0x7fc0b40d29a0 2024-09-16T01:45:18.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.291+0000 7fc0b97ba640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0b40b3920 0x7fc0b414c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.292+0000 7fc0b97ba640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0b40b3920 0x7fc0b414c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60080/0 (socket says 172.21.15.88:60080) 2024-09-16T01:45:18.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.292+0000 7fc0b97ba640 1 -- 172.21.15.88:0/3257132072 learned_addr learned my addr 172.21.15.88:0/3257132072 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:18.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.292+0000 7fc0b9fbb640 1 --2- 172.21.15.88:0/3257132072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b415e990 0x7fc0b40cb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.292+0000 7fc0b8fb9640 1 --2- 172.21.15.88:0/3257132072 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0b40d29a0 0x7fc0b414f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.292+0000 7fc0b97ba640 1 -- 172.21.15.88:0/3257132072 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0b40d29a0 msgr2=0x7fc0b414f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:18.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.292+0000 7fc0b97ba640 1 --2- 172.21.15.88:0/3257132072 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0b40d29a0 0x7fc0b414f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.292+0000 7fc0b97ba640 1 -- 172.21.15.88:0/3257132072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b415e990 msgr2=0x7fc0b40cb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:18.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.292+0000 7fc0b97ba640 1 --2- 172.21.15.88:0/3257132072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b415e990 0x7fc0b40cb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.292+0000 7fc0b97ba640 1 -- 172.21.15.88:0/3257132072 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0b4152d40 con 0x7fc0b40b3920 2024-09-16T01:45:18.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.292+0000 7fc0b9fbb640 1 --2- 172.21.15.88:0/3257132072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b415e990 0x7fc0b40cb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:45:18.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.292+0000 7fc0b97ba640 1 --2- 172.21.15.88:0/3257132072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0b40b3920 0x7fc0b414c750 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7fc0b000b3f0 tx=0x7fc0b000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:18.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.293+0000 7fc0a9ffb640 1 -- 172.21.15.88:0/3257132072 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0015020 con 0x7fc0b40b3920 2024-09-16T01:45:18.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.293+0000 7fc0a9ffb640 1 -- 172.21.15.88:0/3257132072 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0b0011830 con 0x7fc0b40b3920 2024-09-16T01:45:18.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.293+0000 7fc0a9ffb640 1 -- 172.21.15.88:0/3257132072 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0011c20 con 0x7fc0b40b3920 2024-09-16T01:45:18.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.293+0000 7fc0bba1e640 1 -- 172.21.15.88:0/3257132072 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0b414fe20 con 0x7fc0b40b3920 2024-09-16T01:45:18.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.293+0000 7fc0bba1e640 1 -- 172.21.15.88:0/3257132072 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0b40ce1c0 con 0x7fc0b40b3920 2024-09-16T01:45:18.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.294+0000 7fc0bba1e640 1 -- 172.21.15.88:0/3257132072 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc0b4153250 con 0x7fc0b40b3920 2024-09-16T01:45:18.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.294+0000 7fc0a9ffb640 1 -- 172.21.15.88:0/3257132072 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fc0b0018020 con 0x7fc0b40b3920 2024-09-16T01:45:18.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.295+0000 7fc0a9ffb640 1 --2- 172.21.15.88:0/3257132072 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc09806c2b0 0x7fc09806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.295+0000 7fc0a9ffb640 1 -- 172.21.15.88:0/3257132072 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7fc0b008ad40 con 0x7fc0b40b3920 2024-09-16T01:45:18.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.295+0000 7fc0b8fb9640 1 --2- 172.21.15.88:0/3257132072 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc09806c2b0 0x7fc09806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.295+0000 7fc0a9ffb640 1 --2- 172.21.15.88:0/3257132072 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x7fc098072210 0x7fc098074630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.295+0000 7fc0a9ffb640 1 -- 172.21.15.88:0/3257132072 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc098074ce0 con 0x7fc098072210 2024-09-16T01:45:18.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.295+0000 7fc0a9ffb640 1 -- 172.21.15.88:0/3257132072 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=142) ==== 24+0+0 (secure 0 0 0) 0x7fc0b008d440 con 0x7fc0b40b3920 2024-09-16T01:45:18.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.295+0000 7fc0b9fbb640 1 --2- 172.21.15.88:0/3257132072 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x7fc098072210 0x7fc098074630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.295+0000 7fc0b8fb9640 1 --2- 172.21.15.88:0/3257132072 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc09806c2b0 0x7fc09806e770 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4034000 tx=0x7fc0a402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:18.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.295+0000 7fc0b9fbb640 1 --2- 172.21.15.88:0/3257132072 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x7fc098072210 0x7fc098074630 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:18.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.296+0000 7fc0a9ffb640 1 -- 172.21.15.88:0/3257132072 <== osd.0 v2:172.21.15.88:6816/903680192 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc098074ce0 con 0x7fc098072210 2024-09-16T01:45:18.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.309+0000 7fc0bba1e640 1 -- 172.21.15.88:0/3257132072 --> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc0b4162450 con 0x7fc098072210 2024-09-16T01:45:18.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.309+0000 7fc0a9ffb640 1 -- 172.21.15.88:0/3257132072 <== osd.0 v2:172.21.15.88:6816/903680192 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc0b4162450 con 0x7fc098072210 2024-09-16T01:45:18.310 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:45:18.310 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T01:45:18.311 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:45:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.309+0000 7fc0bba1e640 1 -- 172.21.15.88:0/3257132072 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x7fc098072210 msgr2=0x7fc098074630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.309+0000 7fc0bba1e640 1 --2- 172.21.15.88:0/3257132072 >> [v2:172.21.15.88:6816/903680192,v1:172.21.15.88:6817/903680192] conn(0x7fc098072210 0x7fc098074630 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.309+0000 7fc0bba1e640 1 -- 172.21.15.88:0/3257132072 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc09806c2b0 msgr2=0x7fc09806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.309+0000 7fc0bba1e640 1 --2- 172.21.15.88:0/3257132072 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fc09806c2b0 0x7fc09806e770 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4034000 tx=0x7fc0a402e000 comp rx=0 tx=0).stop 2024-09-16T01:45:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.309+0000 7fc0bba1e640 1 -- 172.21.15.88:0/3257132072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0b40b3920 msgr2=0x7fc0b414c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.309+0000 7fc0bba1e640 1 --2- 172.21.15.88:0/3257132072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0b40b3920 0x7fc0b414c750 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7fc0b000b3f0 tx=0x7fc0b000b8b0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.309+0000 7fc0b9fbb640 1 -- 172.21.15.88:0/3257132072 reap_dead start 2024-09-16T01:45:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.309+0000 7fc0bba1e640 1 -- 172.21.15.88:0/3257132072 shutdown_connections 2024-09-16T01:45:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.309+0000 7fc0bba1e640 1 -- 172.21.15.88:0/3257132072 >> 172.21.15.88:0/3257132072 conn(0x7fc0b40d1200 msgr2=0x7fc0b415fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.309+0000 7fc0bba1e640 1 -- 172.21.15.88:0/3257132072 shutdown_connections 2024-09-16T01:45:18.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.309+0000 7fc0bba1e640 1 -- 172.21.15.88:0/3257132072 wait complete. 2024-09-16T01:45:18.321 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T01:45:18.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.419+0000 7fdd8dffd640 1 Processor -- start 2024-09-16T01:45:18.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.419+0000 7fdd8dffd640 1 -- start start 2024-09-16T01:45:18.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.420+0000 7fdd8dffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd8815e350 0x7fdd8806f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.420+0000 7fdd8dffd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd8806bd40 con 0x7fdd88073f30 2024-09-16T01:45:18.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.420+0000 7fdd8dffd640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd8806c020 con 0x7fdd8815e730 2024-09-16T01:45:18.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.420+0000 7fdd8dffd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdd8806c300 con 0x7fdd8815e350 2024-09-16T01:45:18.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.420+0000 7fdd87fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fdd88073f30 0x7fdd880ae990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60012/0 (socket says 172.21.15.88:60012) 2024-09-16T01:45:18.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.420+0000 7fdd87fff640 1 -- 172.21.15.88:0/1612465670 learned_addr learned my addr 172.21.15.88:0/1612465670 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:18.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.420+0000 7fdd877fe640 1 --2- 172.21.15.88:0/1612465670 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd8815e350 0x7fdd8806f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.420+0000 7fdd867fc640 1 -- 172.21.15.88:0/1612465670 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 57570709 0 0) 0x7fdd8806bd40 con 0x7fdd88073f30 2024-09-16T01:45:18.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.420+0000 7fdd867fc640 1 -- 172.21.15.88:0/1612465670 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd70003520 con 0x7fdd88073f30 2024-09-16T01:45:18.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.420+0000 7fdd867fc640 1 -- 172.21.15.88:0/1612465670 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2681306174 0 0) 0x7fdd70003520 con 0x7fdd88073f30 2024-09-16T01:45:18.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.420+0000 7fdd867fc640 1 -- 172.21.15.88:0/1612465670 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd8815e350 msgr2=0x7fdd8806f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:18.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.420+0000 7fdd867fc640 1 --2- 172.21.15.88:0/1612465670 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd8815e350 0x7fdd8806f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.420+0000 7fdd867fc640 1 -- 172.21.15.88:0/1612465670 >> v1:172.21.15.154:6789/0 conn(0x7fdd8815e730 legacy=0x7fdd88073820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:18.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.420+0000 7fdd867fc640 1 -- 172.21.15.88:0/1612465670 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd8806c5e0 con 0x7fdd88073f30 2024-09-16T01:45:18.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.420+0000 7fdd867fc640 1 -- 172.21.15.88:0/1612465670 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fdd800019f0 con 0x7fdd88073f30 2024-09-16T01:45:18.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.420+0000 7fdd867fc640 1 -- 172.21.15.88:0/1612465670 >> v1:172.21.15.88:6789/0 conn(0x7fdd88073f30 legacy=0x7fdd880ae990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:18.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.420+0000 7fdd867fc640 1 --2- 172.21.15.88:0/1612465670 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd8815e350 0x7fdd700039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.420+0000 7fdd867fc640 1 --2- 172.21.15.88:0/1612465670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd70003f60 0x7fdd70004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.420+0000 7fdd867fc640 1 --2- 172.21.15.88:0/1612465670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd70004d40 0x7fdd700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.420+0000 7fdd867fc640 1 -- 172.21.15.88:0/1612465670 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8806c300 con 0x7fdd70003f60 2024-09-16T01:45:18.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.420+0000 7fdd867fc640 1 -- 172.21.15.88:0/1612465670 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdd7000bb50 con 0x7fdd70004d40 2024-09-16T01:45:18.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.420+0000 7fdd867fc640 1 -- 172.21.15.88:0/1612465670 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdd7000bd50 con 0x7fdd8815e350 2024-09-16T01:45:18.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.421+0000 7fdd86ffd640 1 --2- 172.21.15.88:0/1612465670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd70003f60 0x7fdd70004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.421+0000 7fdd877fe640 1 --2- 172.21.15.88:0/1612465670 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd8815e350 0x7fdd700039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.421+0000 7fdd87fff640 1 --2- 172.21.15.88:0/1612465670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd70004d40 0x7fdd700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.421+0000 7fdd86ffd640 1 -- 172.21.15.88:0/1612465670 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd8815e350 msgr2=0x7fdd700039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:18.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.421+0000 7fdd86ffd640 1 --2- 172.21.15.88:0/1612465670 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd8815e350 0x7fdd700039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.421+0000 7fdd86ffd640 1 -- 172.21.15.88:0/1612465670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd70004d40 msgr2=0x7fdd700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:18.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.421+0000 7fdd86ffd640 1 --2- 172.21.15.88:0/1612465670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd70004d40 0x7fdd700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.421+0000 7fdd86ffd640 1 -- 172.21.15.88:0/1612465670 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd70003520 con 0x7fdd70003f60 2024-09-16T01:45:18.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.421+0000 7fdd877fe640 1 --2- 172.21.15.88:0/1612465670 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd8815e350 0x7fdd700039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:45:18.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.421+0000 7fdd87fff640 1 --2- 172.21.15.88:0/1612465670 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd70004d40 0x7fdd700051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:45:18.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.421+0000 7fdd86ffd640 1 --2- 172.21.15.88:0/1612465670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd70003f60 0x7fdd70004800 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c001720 tx=0x7fdd70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7f7e552c8053103 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:18.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.422+0000 7fdd867fc640 1 -- 172.21.15.88:0/1612465670 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c001370 con 0x7fdd70003f60 2024-09-16T01:45:18.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.422+0000 7fdd867fc640 1 -- 172.21.15.88:0/1612465670 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd7c0052d0 con 0x7fdd70003f60 2024-09-16T01:45:18.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.422+0000 7fdd867fc640 1 -- 172.21.15.88:0/1612465670 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c02abc0 con 0x7fdd70003f60 2024-09-16T01:45:18.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.422+0000 7fdd8dffd640 1 -- 172.21.15.88:0/1612465670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd70003f60 msgr2=0x7fdd70004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:18.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.422+0000 7fdd8dffd640 1 --2- 172.21.15.88:0/1612465670 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd70003f60 0x7fdd70004800 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c001720 tx=0x7fdd70005830 comp rx=0 tx=0).stop 2024-09-16T01:45:18.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.422+0000 7fdd87fff640 1 -- 172.21.15.88:0/1612465670 reap_dead start 2024-09-16T01:45:18.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.422+0000 7fdd8dffd640 1 -- 172.21.15.88:0/1612465670 shutdown_connections 2024-09-16T01:45:18.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.422+0000 7fdd8dffd640 1 -- 172.21.15.88:0/1612465670 >> 172.21.15.88:0/1612465670 conn(0x7fdd880b3070 msgr2=0x7fdd880b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:18.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.422+0000 7fdd8dffd640 1 -- 172.21.15.88:0/1612465670 shutdown_connections 2024-09-16T01:45:18.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.422+0000 7fdd8dffd640 1 -- 172.21.15.88:0/1612465670 wait complete. 2024-09-16T01:45:18.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.423+0000 7fdd8dffd640 1 Processor -- start 2024-09-16T01:45:18.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.424+0000 7fdd8dffd640 1 -- start start 2024-09-16T01:45:18.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.424+0000 7fdd8dffd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd70003f60 0x7fdd88155750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.424+0000 7fdd8dffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd88161540 0x7fdd881f7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.424+0000 7fdd8dffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd88159e50 0x7fdd8815a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.424+0000 7fdd8dffd640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8806f890 con 0x7fdd88159e50 2024-09-16T01:45:18.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.424+0000 7fdd8dffd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8806f710 con 0x7fdd88161540 2024-09-16T01:45:18.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.424+0000 7fdd8dffd640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdd8806fa10 con 0x7fdd70003f60 2024-09-16T01:45:18.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.424+0000 7fdd87fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd88159e50 0x7fdd8815a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.424+0000 7fdd87fff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd88159e50 0x7fdd8815a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60100/0 (socket says 172.21.15.88:60100) 2024-09-16T01:45:18.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.424+0000 7fdd87fff640 1 -- 172.21.15.88:0/1138822640 learned_addr learned my addr 172.21.15.88:0/1138822640 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:18.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.424+0000 7fdd877fe640 1 --2- 172.21.15.88:0/1138822640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd70003f60 0x7fdd88155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.424+0000 7fdd86ffd640 1 --2- 172.21.15.88:0/1138822640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd88161540 0x7fdd881f7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.425+0000 7fdd87fff640 1 -- 172.21.15.88:0/1138822640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd70003f60 msgr2=0x7fdd88155750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:18.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.425+0000 7fdd87fff640 1 --2- 172.21.15.88:0/1138822640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd70003f60 0x7fdd88155750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.425+0000 7fdd87fff640 1 -- 172.21.15.88:0/1138822640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd88161540 msgr2=0x7fdd881f7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:18.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.425+0000 7fdd87fff640 1 --2- 172.21.15.88:0/1138822640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd88161540 0x7fdd881f7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.425+0000 7fdd87fff640 1 -- 172.21.15.88:0/1138822640 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd88156010 con 0x7fdd88159e50 2024-09-16T01:45:18.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.425+0000 7fdd87fff640 1 --2- 172.21.15.88:0/1138822640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd88159e50 0x7fdd8815a2d0 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7fdd80000f80 tx=0x7fdd8000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:18.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.425+0000 7fdd6ffff640 1 -- 172.21.15.88:0/1138822640 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd8000eb30 con 0x7fdd88159e50 2024-09-16T01:45:18.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.425+0000 7fdd6ffff640 1 -- 172.21.15.88:0/1138822640 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd80018070 con 0x7fdd88159e50 2024-09-16T01:45:18.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.425+0000 7fdd8dffd640 1 -- 172.21.15.88:0/1138822640 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd881562f0 con 0x7fdd88159e50 2024-09-16T01:45:18.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.425+0000 7fdd6ffff640 1 -- 172.21.15.88:0/1138822640 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd80013420 con 0x7fdd88159e50 2024-09-16T01:45:18.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.425+0000 7fdd8dffd640 1 -- 172.21.15.88:0/1138822640 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd881568b0 con 0x7fdd88159e50 2024-09-16T01:45:18.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.426+0000 7fdd8dffd640 1 -- 172.21.15.88:0/1138822640 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd8806fe40 con 0x7fdd88159e50 2024-09-16T01:45:18.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.426+0000 7fdd6ffff640 1 -- 172.21.15.88:0/1138822640 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fdd800135c0 con 0x7fdd88159e50 2024-09-16T01:45:18.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.427+0000 7fdd6ffff640 1 --2- 172.21.15.88:0/1138822640 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdd7406c2b0 0x7fdd7406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.427+0000 7fdd877fe640 1 --2- 172.21.15.88:0/1138822640 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdd7406c2b0 0x7fdd7406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.427+0000 7fdd6ffff640 1 -- 172.21.15.88:0/1138822640 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7fdd8008b400 con 0x7fdd88159e50 2024-09-16T01:45:18.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.427+0000 7fdd877fe640 1 --2- 172.21.15.88:0/1138822640 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdd7406c2b0 0x7fdd7406e770 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fdd78009a90 tx=0x7fdd78008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:18.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.431+0000 7fdd6ffff640 1 -- 172.21.15.88:0/1138822640 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fdd80059ea0 con 0x7fdd88159e50 2024-09-16T01:45:18.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.599+0000 7fdd8dffd640 1 -- 172.21.15.88:0/1138822640 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fdd880b4e60 con 0x7fdd7406c2b0 2024-09-16T01:45:18.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.599+0000 7fdd6ffff640 1 -- 172.21.15.88:0/1138822640 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fdd880b4e60 con 0x7fdd7406c2b0 2024-09-16T01:45:18.601 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T01:45:18.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.601+0000 7fdd8dffd640 1 -- 172.21.15.88:0/1138822640 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdd7406c2b0 msgr2=0x7fdd7406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:18.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.601+0000 7fdd8dffd640 1 --2- 172.21.15.88:0/1138822640 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdd7406c2b0 0x7fdd7406e770 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fdd78009a90 tx=0x7fdd78008040 comp rx=0 tx=0).stop 2024-09-16T01:45:18.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.601+0000 7fdd8dffd640 1 -- 172.21.15.88:0/1138822640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd88159e50 msgr2=0x7fdd8815a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:18.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.601+0000 7fdd8dffd640 1 --2- 172.21.15.88:0/1138822640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd88159e50 0x7fdd8815a2d0 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7fdd80000f80 tx=0x7fdd8000e530 comp rx=0 tx=0).stop 2024-09-16T01:45:18.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.601+0000 7fdd8dffd640 1 -- 172.21.15.88:0/1138822640 shutdown_connections 2024-09-16T01:45:18.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.601+0000 7fdd8dffd640 1 --2- 172.21.15.88:0/1138822640 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdd7406c2b0 0x7fdd7406e770 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.601+0000 7fdd8dffd640 1 --2- 172.21.15.88:0/1138822640 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdd88159e50 0x7fdd8815a2d0 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.601+0000 7fdd8dffd640 1 --2- 172.21.15.88:0/1138822640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd88161540 0x7fdd881f7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.601+0000 7fdd8dffd640 1 --2- 172.21.15.88:0/1138822640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd70003f60 0x7fdd88155750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.601+0000 7fdd8dffd640 1 -- 172.21.15.88:0/1138822640 >> 172.21.15.88:0/1138822640 conn(0x7fdd880b3070 msgr2=0x7fdd880b5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:18.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.601+0000 7fdd8dffd640 1 -- 172.21.15.88:0/1138822640 shutdown_connections 2024-09-16T01:45:18.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.601+0000 7fdd8dffd640 1 -- 172.21.15.88:0/1138822640 wait complete. 2024-09-16T01:45:18.613 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T01:45:18.613 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T01:45:18.660 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:45:18.663 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:45:18.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.718+0000 7f5607265640 1 Processor -- start 2024-09-16T01:45:18.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.718+0000 7f5607265640 1 -- start start 2024-09-16T01:45:18.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.719+0000 7f5607265640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f56000b3920 0x7f5600162b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.719+0000 7f5607265640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56000d29a0 0x7f56000d2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.719+0000 7f5607265640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f560015ed70 con 0x7f560015e990 2024-09-16T01:45:18.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.719+0000 7f5607265640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f56000cff20 con 0x7f56000b3920 2024-09-16T01:45:18.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.719+0000 7f5607265640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f56000b3d60 con 0x7f56000d29a0 2024-09-16T01:45:18.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.719+0000 7f5604800640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f56000b3920 0x7f5600162b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.719+0000 7f5604800640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f56000b3920 0x7f5600162b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60102/0 (socket says 172.21.15.88:60102) 2024-09-16T01:45:18.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.719+0000 7f5604800640 1 -- 172.21.15.88:0/274858843 learned_addr learned my addr 172.21.15.88:0/274858843 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:18.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.720+0000 7f5605802640 1 --2- 172.21.15.88:0/274858843 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56000d29a0 0x7f56000d2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.720+0000 7f5604800640 1 -- 172.21.15.88:0/274858843 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56000d29a0 msgr2=0x7f56000d2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:18.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.720+0000 7f5604800640 1 --2- 172.21.15.88:0/274858843 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56000d29a0 0x7f56000d2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.720+0000 7f5604800640 1 -- 172.21.15.88:0/274858843 >> v1:172.21.15.154:6789/0 conn(0x7f560015e990 legacy=0x7f5600162450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:18.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.720+0000 7f5604800640 1 -- 172.21.15.88:0/274858843 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56000d1fa0 con 0x7f56000b3920 2024-09-16T01:45:18.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.720+0000 7f5604800640 1 --2- 172.21.15.88:0/274858843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56000b3920 0x7f5600162b60 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7f55f0009870 tx=0x7f55f002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44820d0b5a45b30c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:18.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.721+0000 7f55f7fff640 1 -- 172.21.15.88:0/274858843 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f0037070 con 0x7f56000b3920 2024-09-16T01:45:18.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.721+0000 7f55f7fff640 1 -- 172.21.15.88:0/274858843 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55f0032440 con 0x7f56000b3920 2024-09-16T01:45:18.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.721+0000 7f55f7fff640 1 -- 172.21.15.88:0/274858843 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f0032810 con 0x7f56000b3920 2024-09-16T01:45:18.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.721+0000 7f5607265640 1 -- 172.21.15.88:0/274858843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56000b3920 msgr2=0x7f5600162b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:18.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.721+0000 7f5607265640 1 --2- 172.21.15.88:0/274858843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56000b3920 0x7f5600162b60 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7f55f0009870 tx=0x7f55f002a180 comp rx=0 tx=0).stop 2024-09-16T01:45:18.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.721+0000 7f5607265640 1 -- 172.21.15.88:0/274858843 shutdown_connections 2024-09-16T01:45:18.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.721+0000 7f5607265640 1 --2- 172.21.15.88:0/274858843 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56000d29a0 0x7f56000d2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.721+0000 7f5607265640 1 --2- 172.21.15.88:0/274858843 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56000b3920 0x7f5600162b60 unknown :-1 s=CLOSED pgs=2456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.721+0000 7f5607265640 1 -- 172.21.15.88:0/274858843 >> 172.21.15.88:0/274858843 conn(0x7f56000d1200 msgr2=0x7f56000d1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:18.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.722+0000 7f5607265640 1 -- 172.21.15.88:0/274858843 shutdown_connections 2024-09-16T01:45:18.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.722+0000 7f5607265640 1 -- 172.21.15.88:0/274858843 wait complete. 2024-09-16T01:45:18.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.723+0000 7f5607265640 1 Processor -- start 2024-09-16T01:45:18.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.723+0000 7f5607265640 1 -- start start 2024-09-16T01:45:18.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.723+0000 7f5607265640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56000b3920 0x7f56000c9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.723+0000 7f5607265640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56000d29a0 0x7f56001f46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.723+0000 7f5607265640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f560015e990 0x7f560015cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.723+0000 7f5607265640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f56001630a0 con 0x7f56000b3920 2024-09-16T01:45:18.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.723+0000 7f5607265640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f56000cfc40 con 0x7f560015e990 2024-09-16T01:45:18.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.723+0000 7f5607265640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5600163220 con 0x7f56000d29a0 2024-09-16T01:45:18.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.724+0000 7f5605001640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56000b3920 0x7f56000c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.724+0000 7f5605001640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56000b3920 0x7f56000c9640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60112/0 (socket says 172.21.15.88:60112) 2024-09-16T01:45:18.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.724+0000 7f5605001640 1 -- 172.21.15.88:0/4146034827 learned_addr learned my addr 172.21.15.88:0/4146034827 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:18.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.724+0000 7f5604800640 1 --2- 172.21.15.88:0/4146034827 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56000d29a0 0x7f56001f46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.724+0000 7f5605802640 1 --2- 172.21.15.88:0/4146034827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f560015e990 0x7f560015cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.724+0000 7f5605001640 1 -- 172.21.15.88:0/4146034827 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56000d29a0 msgr2=0x7f56001f46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:18.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.724+0000 7f5605001640 1 --2- 172.21.15.88:0/4146034827 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f56000d29a0 0x7f56001f46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.724+0000 7f5605001640 1 -- 172.21.15.88:0/4146034827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f560015e990 msgr2=0x7f560015cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:18.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.724+0000 7f5605001640 1 --2- 172.21.15.88:0/4146034827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f560015e990 0x7f560015cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.724+0000 7f5605001640 1 -- 172.21.15.88:0/4146034827 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56000cb4e0 con 0x7f56000b3920 2024-09-16T01:45:18.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.724+0000 7f5605001640 1 --2- 172.21.15.88:0/4146034827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56000b3920 0x7f56000c9640 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7f55fc00b420 tx=0x7f55fc00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:18.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.724+0000 7f55f5ffb640 1 -- 172.21.15.88:0/4146034827 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55fc011830 con 0x7f56000b3920 2024-09-16T01:45:18.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.724+0000 7f55f5ffb640 1 -- 172.21.15.88:0/4146034827 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55fc016070 con 0x7f56000b3920 2024-09-16T01:45:18.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.724+0000 7f55f5ffb640 1 -- 172.21.15.88:0/4146034827 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55fc011c10 con 0x7f56000b3920 2024-09-16T01:45:18.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.724+0000 7f5607265640 1 -- 172.21.15.88:0/4146034827 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56000ccec0 con 0x7f56000b3920 2024-09-16T01:45:18.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.724+0000 7f5607265640 1 -- 172.21.15.88:0/4146034827 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56000cd480 con 0x7f56000b3920 2024-09-16T01:45:18.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.725+0000 7f5607265640 1 -- 172.21.15.88:0/4146034827 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5600162450 con 0x7f56000b3920 2024-09-16T01:45:18.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.726+0000 7f55f5ffb640 1 -- 172.21.15.88:0/4146034827 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f55fc01b460 con 0x7f56000b3920 2024-09-16T01:45:18.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.726+0000 7f55f5ffb640 1 --2- 172.21.15.88:0/4146034827 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f55e406c2e0 0x7f55e406e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.726+0000 7f55f5ffb640 1 -- 172.21.15.88:0/4146034827 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7f55fc08ab10 con 0x7f56000b3920 2024-09-16T01:45:18.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.726+0000 7f5604800640 1 --2- 172.21.15.88:0/4146034827 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f55e406c2e0 0x7f55e406e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.726+0000 7f55f5ffb640 1 --2- 172.21.15.88:0/4146034827 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x7f55e4072240 0x7f55e4074660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.726+0000 7f55f5ffb640 1 -- 172.21.15.88:0/4146034827 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f55e4074d10 con 0x7f55e4072240 2024-09-16T01:45:18.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.726+0000 7f55f5ffb640 1 -- 172.21.15.88:0/4146034827 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=142) ==== 24+0+0 (secure 0 0 0) 0x7f55fc08f2c0 con 0x7f56000b3920 2024-09-16T01:45:18.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.727+0000 7f5605802640 1 --2- 172.21.15.88:0/4146034827 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x7f55e4072240 0x7f55e4074660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.727+0000 7f5604800640 1 --2- 172.21.15.88:0/4146034827 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f55e406c2e0 0x7f55e406e7a0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f55f0034000 tx=0x7f55f002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:18.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.727+0000 7f5605802640 1 --2- 172.21.15.88:0/4146034827 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x7f55e4072240 0x7f55e4074660 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:18.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.727+0000 7f55f5ffb640 1 -- 172.21.15.88:0/4146034827 <== osd.1 v2:172.21.15.88:6824/3621315521 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f55e4074d10 con 0x7f55e4072240 2024-09-16T01:45:18.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.740+0000 7f5607265640 1 -- 172.21.15.88:0/4146034827 --> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f56000ba700 con 0x7f55e4072240 2024-09-16T01:45:18.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.740+0000 7f55f5ffb640 1 -- 172.21.15.88:0/4146034827 <== osd.1 v2:172.21.15.88:6824/3621315521 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f56000ba700 con 0x7f55e4072240 2024-09-16T01:45:18.742 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:45:18.742 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T01:45:18.742 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:45:18.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.740+0000 7f5607265640 1 -- 172.21.15.88:0/4146034827 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x7f55e4072240 msgr2=0x7f55e4074660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:18.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.740+0000 7f5607265640 1 --2- 172.21.15.88:0/4146034827 >> [v2:172.21.15.88:6824/3621315521,v1:172.21.15.88:6825/3621315521] conn(0x7f55e4072240 0x7f55e4074660 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.741+0000 7f5607265640 1 -- 172.21.15.88:0/4146034827 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f55e406c2e0 msgr2=0x7f55e406e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:18.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.741+0000 7f5607265640 1 --2- 172.21.15.88:0/4146034827 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f55e406c2e0 0x7f55e406e7a0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f55f0034000 tx=0x7f55f002e000 comp rx=0 tx=0).stop 2024-09-16T01:45:18.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.741+0000 7f5607265640 1 -- 172.21.15.88:0/4146034827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56000b3920 msgr2=0x7f56000c9640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:18.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.741+0000 7f5607265640 1 --2- 172.21.15.88:0/4146034827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56000b3920 0x7f56000c9640 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7f55fc00b420 tx=0x7f55fc00b8e0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.741+0000 7f5605802640 1 -- 172.21.15.88:0/4146034827 reap_dead start 2024-09-16T01:45:18.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.741+0000 7f5607265640 1 -- 172.21.15.88:0/4146034827 shutdown_connections 2024-09-16T01:45:18.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.741+0000 7f5607265640 1 -- 172.21.15.88:0/4146034827 >> 172.21.15.88:0/4146034827 conn(0x7f56000d1200 msgr2=0x7f5600160e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:18.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.741+0000 7f5607265640 1 -- 172.21.15.88:0/4146034827 shutdown_connections 2024-09-16T01:45:18.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.741+0000 7f5607265640 1 -- 172.21.15.88:0/4146034827 wait complete. 2024-09-16T01:45:18.752 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T01:45:18.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.889+0000 7fcd0c23c640 1 Processor -- start 2024-09-16T01:45:18.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.890+0000 7fcd0c23c640 1 -- start start 2024-09-16T01:45:18.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.890+0000 7fcd0c23c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcd04158a00 0x7fcd04071f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.890+0000 7fcd0c23c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd04072470 0x7fcd040b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.890+0000 7fcd0c23c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcd04158de0 0x7fcd040b53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.891+0000 7fcd0c23c640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fcd040728f0 con 0x7fcd04158de0 2024-09-16T01:45:18.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.891+0000 7fcd0c23c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fcd04072bd0 con 0x7fcd04158a00 2024-09-16T01:45:18.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.891+0000 7fcd0c23c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcd04072eb0 con 0x7fcd04072470 2024-09-16T01:45:18.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.891+0000 7fcd0a7d9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcd04158de0 0x7fcd040b53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.891+0000 7fcd0a7d9640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcd04158de0 0x7fcd040b53a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60124/0 (socket says 172.21.15.88:60124) 2024-09-16T01:45:18.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.891+0000 7fcd0a7d9640 1 -- 172.21.15.88:0/1940300183 learned_addr learned my addr 172.21.15.88:0/1940300183 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:18.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.891+0000 7fcd09fd8640 1 --2- 172.21.15.88:0/1940300183 >> v2:172.21.15.154:3300/0 conn(0x7fcd04158a00 0x7fcd04071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.891+0000 7fcd097d7640 1 --2- 172.21.15.88:0/1940300183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd04072470 0x7fcd040b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.891+0000 7fcd0a7d9640 1 -- 172.21.15.88:0/1940300183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd04072470 msgr2=0x7fcd040b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:18.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.891+0000 7fcd0a7d9640 1 --2- 172.21.15.88:0/1940300183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd04072470 0x7fcd040b4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.891+0000 7fcd0a7d9640 1 -- 172.21.15.88:0/1940300183 >> v2:172.21.15.154:3300/0 conn(0x7fcd04158a00 msgr2=0x7fcd04071f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:18.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.891+0000 7fcd0a7d9640 1 --2- 172.21.15.88:0/1940300183 >> v2:172.21.15.154:3300/0 conn(0x7fcd04158a00 0x7fcd04071f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.891+0000 7fcd0a7d9640 1 -- 172.21.15.88:0/1940300183 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd04073190 con 0x7fcd04158de0 2024-09-16T01:45:18.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.892+0000 7fcd0a7d9640 1 --2- 172.21.15.88:0/1940300183 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd04158de0 0x7fcd040b53a0 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7fcd04069f90 tx=0x7fccf402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d116b01b0cc0895 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:18.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.892+0000 7fcd08fd6640 1 -- 172.21.15.88:0/1940300183 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf4036070 con 0x7fcd04158de0 2024-09-16T01:45:18.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.892+0000 7fcd08fd6640 1 -- 172.21.15.88:0/1940300183 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccf4031600 con 0x7fcd04158de0 2024-09-16T01:45:18.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.892+0000 7fcd08fd6640 1 -- 172.21.15.88:0/1940300183 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf40319d0 con 0x7fcd04158de0 2024-09-16T01:45:18.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.892+0000 7fcd0c23c640 1 -- 172.21.15.88:0/1940300183 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd04158de0 msgr2=0x7fcd040b53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:18.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.892+0000 7fcd0c23c640 1 --2- 172.21.15.88:0/1940300183 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd04158de0 0x7fcd040b53a0 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7fcd04069f90 tx=0x7fccf402a6a0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.892+0000 7fcd0c23c640 1 -- 172.21.15.88:0/1940300183 shutdown_connections 2024-09-16T01:45:18.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.892+0000 7fcd0c23c640 1 --2- 172.21.15.88:0/1940300183 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd04158de0 0x7fcd040b53a0 unknown :-1 s=CLOSED pgs=2458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.892+0000 7fcd0c23c640 1 --2- 172.21.15.88:0/1940300183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd04072470 0x7fcd040b4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.892+0000 7fcd0c23c640 1 --2- 172.21.15.88:0/1940300183 >> v2:172.21.15.154:3300/0 conn(0x7fcd04158a00 0x7fcd04071f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.892+0000 7fcd0c23c640 1 -- 172.21.15.88:0/1940300183 >> 172.21.15.88:0/1940300183 conn(0x7fcd0406cb50 msgr2=0x7fcd0406cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:18.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.893+0000 7fcd0c23c640 1 -- 172.21.15.88:0/1940300183 shutdown_connections 2024-09-16T01:45:18.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.893+0000 7fcd0c23c640 1 -- 172.21.15.88:0/1940300183 wait complete. 2024-09-16T01:45:18.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.894+0000 7fcd0c23c640 1 Processor -- start 2024-09-16T01:45:18.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.894+0000 7fcd0c23c640 1 -- start start 2024-09-16T01:45:18.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.894+0000 7fcd0c23c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd04072470 0x7fcd04153d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.894+0000 7fcd0c23c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd04158a00 0x7fcd0413d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.894+0000 7fcd0c23c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd04158de0 0x7fcd0413dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.894+0000 7fcd0c23c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcd040b5ca0 con 0x7fcd04158de0 2024-09-16T01:45:18.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.894+0000 7fcd0c23c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcd040b5b20 con 0x7fcd04072470 2024-09-16T01:45:18.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.894+0000 7fcd0c23c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcd04162110 con 0x7fcd04158a00 2024-09-16T01:45:18.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.895+0000 7fcd0a7d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd04158de0 0x7fcd0413dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.895+0000 7fcd0a7d9640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd04158de0 0x7fcd0413dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60140/0 (socket says 172.21.15.88:60140) 2024-09-16T01:45:18.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.895+0000 7fcd0a7d9640 1 -- 172.21.15.88:0/852905343 learned_addr learned my addr 172.21.15.88:0/852905343 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:18.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.895+0000 7fcd097d7640 1 --2- 172.21.15.88:0/852905343 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd04158a00 0x7fcd0413d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.895+0000 7fcd09fd8640 1 --2- 172.21.15.88:0/852905343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd04072470 0x7fcd04153d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.895+0000 7fcd0a7d9640 1 -- 172.21.15.88:0/852905343 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd04158a00 msgr2=0x7fcd0413d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:18.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.895+0000 7fcd0a7d9640 1 --2- 172.21.15.88:0/852905343 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd04158a00 0x7fcd0413d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.895+0000 7fcd0a7d9640 1 -- 172.21.15.88:0/852905343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd04072470 msgr2=0x7fcd04153d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:18.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.895+0000 7fcd0a7d9640 1 --2- 172.21.15.88:0/852905343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd04072470 0x7fcd04153d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:18.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.895+0000 7fcd0a7d9640 1 -- 172.21.15.88:0/852905343 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd040ce510 con 0x7fcd04158de0 2024-09-16T01:45:18.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.895+0000 7fcd0a7d9640 1 --2- 172.21.15.88:0/852905343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd04158de0 0x7fcd0413dcd0 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7fccf4009f40 tx=0x7fccf4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:18.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.895+0000 7fcd027fc640 1 -- 172.21.15.88:0/852905343 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf4036050 con 0x7fcd04158de0 2024-09-16T01:45:18.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.895+0000 7fcd027fc640 1 -- 172.21.15.88:0/852905343 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccf4042070 con 0x7fcd04158de0 2024-09-16T01:45:18.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.895+0000 7fcd0c23c640 1 -- 172.21.15.88:0/852905343 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd0413e460 con 0x7fcd04158de0 2024-09-16T01:45:18.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.895+0000 7fcd027fc640 1 -- 172.21.15.88:0/852905343 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf4009510 con 0x7fcd04158de0 2024-09-16T01:45:18.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.895+0000 7fcd0c23c640 1 -- 172.21.15.88:0/852905343 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd0413ea20 con 0x7fcd04158de0 2024-09-16T01:45:18.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.896+0000 7fcd0c23c640 1 -- 172.21.15.88:0/852905343 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd04071f30 con 0x7fcd04158de0 2024-09-16T01:45:18.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.897+0000 7fcd027fc640 1 -- 172.21.15.88:0/852905343 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fccf4007c50 con 0x7fcd04158de0 2024-09-16T01:45:18.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.897+0000 7fcd027fc640 1 --2- 172.21.15.88:0/852905343 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fccf006c2b0 0x7fccf006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:18.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.897+0000 7fcd027fc640 1 -- 172.21.15.88:0/852905343 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7fccf40abf20 con 0x7fcd04158de0 2024-09-16T01:45:18.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.897+0000 7fcd09fd8640 1 --2- 172.21.15.88:0/852905343 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fccf006c2b0 0x7fccf006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:18.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.898+0000 7fcd09fd8640 1 --2- 172.21.15.88:0/852905343 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fccf006c2b0 0x7fccf006e770 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fccfc006fd0 tx=0x7fccfc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:18.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:18.901+0000 7fcd027fc640 1 -- 172.21.15.88:0/852905343 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fccf407a2e0 con 0x7fcd04158de0 2024-09-16T01:45:19.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.069+0000 7fcd0c23c640 1 -- 172.21.15.88:0/852905343 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fcd0406f490 con 0x7fccf006c2b0 2024-09-16T01:45:19.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.069+0000 7fcd027fc640 1 -- 172.21.15.88:0/852905343 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fcd0406f490 con 0x7fccf006c2b0 2024-09-16T01:45:19.071 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T01:45:19.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.071+0000 7fcd0c23c640 1 -- 172.21.15.88:0/852905343 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fccf006c2b0 msgr2=0x7fccf006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.071+0000 7fcd0c23c640 1 --2- 172.21.15.88:0/852905343 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fccf006c2b0 0x7fccf006e770 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fccfc006fd0 tx=0x7fccfc008040 comp rx=0 tx=0).stop 2024-09-16T01:45:19.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.071+0000 7fcd0c23c640 1 -- 172.21.15.88:0/852905343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd04158de0 msgr2=0x7fcd0413dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.071+0000 7fcd0c23c640 1 --2- 172.21.15.88:0/852905343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd04158de0 0x7fcd0413dcd0 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7fccf4009f40 tx=0x7fccf4004a40 comp rx=0 tx=0).stop 2024-09-16T01:45:19.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.071+0000 7fcd0c23c640 1 -- 172.21.15.88:0/852905343 shutdown_connections 2024-09-16T01:45:19.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.071+0000 7fcd0c23c640 1 --2- 172.21.15.88:0/852905343 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fccf006c2b0 0x7fccf006e770 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.071+0000 7fcd0c23c640 1 --2- 172.21.15.88:0/852905343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcd04158de0 0x7fcd0413dcd0 unknown :-1 s=CLOSED pgs=2459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.071+0000 7fcd0c23c640 1 --2- 172.21.15.88:0/852905343 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd04158a00 0x7fcd0413d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.071+0000 7fcd0c23c640 1 --2- 172.21.15.88:0/852905343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd04072470 0x7fcd04153d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.071+0000 7fcd0c23c640 1 -- 172.21.15.88:0/852905343 >> 172.21.15.88:0/852905343 conn(0x7fcd0406cb50 msgr2=0x7fcd040b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:19.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.071+0000 7fcd0c23c640 1 -- 172.21.15.88:0/852905343 shutdown_connections 2024-09-16T01:45:19.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.071+0000 7fcd0c23c640 1 -- 172.21.15.88:0/852905343 wait complete. 2024-09-16T01:45:19.083 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T01:45:19.083 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T01:45:19.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.183+0000 7f0a6498d640 1 Processor -- start 2024-09-16T01:45:19.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.184+0000 7f0a6498d640 1 -- start start 2024-09-16T01:45:19.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.184+0000 7f0a6498d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a6015e350 0x7f0a6006f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.184+0000 7f0a6498d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0a60073f30 0x7f0a6006fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.184+0000 7f0a6498d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a6006bd40 con 0x7f0a6015e730 2024-09-16T01:45:19.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.184+0000 7f0a6498d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0a6006c020 con 0x7f0a60073f30 2024-09-16T01:45:19.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.184+0000 7f0a6498d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0a6006c300 con 0x7f0a6015e350 2024-09-16T01:45:19.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.185+0000 7f0a5ed9d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0a60073f30 0x7f0a6006fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.185+0000 7f0a5ed9d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0a60073f30 0x7f0a6006fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:53782/0 (socket says 172.21.15.88:53782) 2024-09-16T01:45:19.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.185+0000 7f0a5e59c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a6015e350 0x7f0a6006f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.185+0000 7f0a5ed9d640 1 -- 172.21.15.88:0/4077475538 learned_addr learned my addr 172.21.15.88:0/4077475538 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:19.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.185+0000 7f0a5e59c640 1 -- 172.21.15.88:0/4077475538 >> v2:172.21.15.154:3300/0 conn(0x7f0a60073f30 msgr2=0x7f0a6006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:19.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.185+0000 7f0a5e59c640 1 --2- 172.21.15.88:0/4077475538 >> v2:172.21.15.154:3300/0 conn(0x7f0a60073f30 0x7f0a6006fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.185+0000 7f0a5e59c640 1 -- 172.21.15.88:0/4077475538 >> v1:172.21.15.154:6789/0 conn(0x7f0a6015e730 legacy=0x7f0a60073820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.185+0000 7f0a5e59c640 1 -- 172.21.15.88:0/4077475538 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a6006c5e0 con 0x7f0a6015e350 2024-09-16T01:45:19.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.185+0000 7f0a5ed9d640 1 --2- 172.21.15.88:0/4077475538 >> v2:172.21.15.154:3300/0 conn(0x7f0a60073f30 0x7f0a6006fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:45:19.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.186+0000 7f0a5e59c640 1 --2- 172.21.15.88:0/4077475538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a6015e350 0x7f0a6006f710 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0a500097a0 tx=0x7f0a5002a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a77721fa5ed5793b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:19.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.186+0000 7f0a5dd9b640 1 -- 172.21.15.88:0/4077475538 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a50037070 con 0x7f0a6015e350 2024-09-16T01:45:19.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.186+0000 7f0a5dd9b640 1 -- 172.21.15.88:0/4077475538 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a5002ae50 con 0x7f0a6015e350 2024-09-16T01:45:19.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.186+0000 7f0a5dd9b640 1 -- 172.21.15.88:0/4077475538 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a50032660 con 0x7f0a6015e350 2024-09-16T01:45:19.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.187+0000 7f0a6498d640 1 -- 172.21.15.88:0/4077475538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a6015e350 msgr2=0x7f0a6006f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.187+0000 7f0a6498d640 1 --2- 172.21.15.88:0/4077475538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a6015e350 0x7f0a6006f710 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0a500097a0 tx=0x7f0a5002a100 comp rx=0 tx=0).stop 2024-09-16T01:45:19.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.187+0000 7f0a6498d640 1 -- 172.21.15.88:0/4077475538 shutdown_connections 2024-09-16T01:45:19.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.187+0000 7f0a6498d640 1 --2- 172.21.15.88:0/4077475538 >> v2:172.21.15.154:3300/0 conn(0x7f0a60073f30 0x7f0a6006fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.187+0000 7f0a6498d640 1 --2- 172.21.15.88:0/4077475538 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a6015e350 0x7f0a6006f710 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.187+0000 7f0a6498d640 1 -- 172.21.15.88:0/4077475538 >> 172.21.15.88:0/4077475538 conn(0x7f0a600b3070 msgr2=0x7f0a600b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:19.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.187+0000 7f0a6498d640 1 -- 172.21.15.88:0/4077475538 shutdown_connections 2024-09-16T01:45:19.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.187+0000 7f0a6498d640 1 -- 172.21.15.88:0/4077475538 wait complete. 2024-09-16T01:45:19.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.188+0000 7f0a6498d640 1 Processor -- start 2024-09-16T01:45:19.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.188+0000 7f0a6498d640 1 -- start start 2024-09-16T01:45:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.189+0000 7f0a6498d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a60073f30 0x7f0a6015d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.189+0000 7f0a6498d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a6015e350 0x7f0a601496c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.189+0000 7f0a6498d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a6015e730 0x7f0a601f7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.189+0000 7f0a6498d640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0a600aeb10 con 0x7f0a60073f30 2024-09-16T01:45:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.189+0000 7f0a6498d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0a600ae990 con 0x7f0a6015e350 2024-09-16T01:45:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.189+0000 7f0a6498d640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0a600aec90 con 0x7f0a6015e730 2024-09-16T01:45:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.189+0000 7f0a5e59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a60073f30 0x7f0a6015d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.189+0000 7f0a5e59c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a60073f30 0x7f0a6015d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60146/0 (socket says 172.21.15.88:60146) 2024-09-16T01:45:19.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.189+0000 7f0a5e59c640 1 -- 172.21.15.88:0/786065956 learned_addr learned my addr 172.21.15.88:0/786065956 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:19.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.189+0000 7f0a5ed9d640 1 --2- 172.21.15.88:0/786065956 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a6015e730 0x7f0a601f7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.189+0000 7f0a57fff640 1 --2- 172.21.15.88:0/786065956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a6015e350 0x7f0a601496c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.189+0000 7f0a5e59c640 1 -- 172.21.15.88:0/786065956 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a6015e730 msgr2=0x7f0a601f7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.189+0000 7f0a5e59c640 1 --2- 172.21.15.88:0/786065956 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a6015e730 0x7f0a601f7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.189+0000 7f0a5e59c640 1 -- 172.21.15.88:0/786065956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a6015e350 msgr2=0x7f0a601496c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.189+0000 7f0a5e59c640 1 --2- 172.21.15.88:0/786065956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a6015e350 0x7f0a601496c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.189+0000 7f0a5e59c640 1 -- 172.21.15.88:0/786065956 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a6015f5d0 con 0x7f0a60073f30 2024-09-16T01:45:19.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.189+0000 7f0a57fff640 1 --2- 172.21.15.88:0/786065956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a6015e350 0x7f0a601496c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:45:19.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.189+0000 7f0a5e59c640 1 --2- 172.21.15.88:0/786065956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a60073f30 0x7f0a6015d6e0 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7f0a5002a140 tx=0x7f0a500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:19.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.190+0000 7f0a56ffd640 1 -- 172.21.15.88:0/786065956 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a50041070 con 0x7f0a60073f30 2024-09-16T01:45:19.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.190+0000 7f0a56ffd640 1 -- 172.21.15.88:0/786065956 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a5003c670 con 0x7f0a60073f30 2024-09-16T01:45:19.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.190+0000 7f0a6498d640 1 -- 172.21.15.88:0/786065956 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a6015f8b0 con 0x7f0a60073f30 2024-09-16T01:45:19.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.190+0000 7f0a56ffd640 1 -- 172.21.15.88:0/786065956 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a50037040 con 0x7f0a60073f30 2024-09-16T01:45:19.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.190+0000 7f0a6498d640 1 -- 172.21.15.88:0/786065956 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a6015cd60 con 0x7f0a60073f30 2024-09-16T01:45:19.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.190+0000 7f0a6498d640 1 -- 172.21.15.88:0/786065956 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0a60073820 con 0x7f0a60073f30 2024-09-16T01:45:19.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.191+0000 7f0a56ffd640 1 -- 172.21.15.88:0/786065956 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0a50044070 con 0x7f0a60073f30 2024-09-16T01:45:19.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.192+0000 7f0a56ffd640 1 --2- 172.21.15.88:0/786065956 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0a4006c2b0 0x7f0a4006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.192+0000 7f0a56ffd640 1 -- 172.21.15.88:0/786065956 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7f0a500ab740 con 0x7f0a60073f30 2024-09-16T01:45:19.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.192+0000 7f0a57fff640 1 --2- 172.21.15.88:0/786065956 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0a4006c2b0 0x7f0a4006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.192+0000 7f0a56ffd640 1 --2- 172.21.15.88:0/786065956 >> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] conn(0x7f0a40072210 0x7f0a40074630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.192+0000 7f0a56ffd640 1 -- 172.21.15.88:0/786065956 --> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0a50033e50 con 0x7f0a40072210 2024-09-16T01:45:19.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.192+0000 7f0a56ffd640 1 -- 172.21.15.88:0/786065956 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=142) ==== 24+0+0 (secure 0 0 0) 0x7f0a500b2050 con 0x7f0a60073f30 2024-09-16T01:45:19.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.192+0000 7f0a5ed9d640 1 --2- 172.21.15.88:0/786065956 >> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] conn(0x7f0a40072210 0x7f0a40074630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.192+0000 7f0a57fff640 1 --2- 172.21.15.88:0/786065956 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0a4006c2b0 0x7f0a4006e770 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f0a60158e90 tx=0x7f0a48009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:19.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.192+0000 7f0a5ed9d640 1 --2- 172.21.15.88:0/786065956 >> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] conn(0x7f0a40072210 0x7f0a40074630 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:19.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.193+0000 7f0a56ffd640 1 -- 172.21.15.88:0/786065956 <== osd.2 v2:172.21.15.88:6808/2918260303 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0a50033e50 con 0x7f0a40072210 2024-09-16T01:45:19.206 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.205+0000 7f0a6498d640 1 -- 172.21.15.88:0/786065956 --> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f0a600b2c10 con 0x7f0a40072210 2024-09-16T01:45:19.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.205+0000 7f0a56ffd640 1 -- 172.21.15.88:0/786065956 <== osd.2 v2:172.21.15.88:6808/2918260303 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f0a600b2c10 con 0x7f0a40072210 2024-09-16T01:45:19.207 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:45:19.207 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T01:45:19.207 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:45:19.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.206+0000 7f0a6498d640 1 -- 172.21.15.88:0/786065956 >> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] conn(0x7f0a40072210 msgr2=0x7f0a40074630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.208 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.206+0000 7f0a6498d640 1 --2- 172.21.15.88:0/786065956 >> [v2:172.21.15.88:6808/2918260303,v1:172.21.15.88:6809/2918260303] conn(0x7f0a40072210 0x7f0a40074630 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.206+0000 7f0a6498d640 1 -- 172.21.15.88:0/786065956 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0a4006c2b0 msgr2=0x7f0a4006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.206+0000 7f0a6498d640 1 --2- 172.21.15.88:0/786065956 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0a4006c2b0 0x7f0a4006e770 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f0a60158e90 tx=0x7f0a48009040 comp rx=0 tx=0).stop 2024-09-16T01:45:19.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.206+0000 7f0a6498d640 1 -- 172.21.15.88:0/786065956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a60073f30 msgr2=0x7f0a6015d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.206+0000 7f0a6498d640 1 --2- 172.21.15.88:0/786065956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0a60073f30 0x7f0a6015d6e0 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7f0a5002a140 tx=0x7f0a500046c0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.206+0000 7f0a5ed9d640 1 -- 172.21.15.88:0/786065956 reap_dead start 2024-09-16T01:45:19.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.206+0000 7f0a6498d640 1 -- 172.21.15.88:0/786065956 shutdown_connections 2024-09-16T01:45:19.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.206+0000 7f0a6498d640 1 -- 172.21.15.88:0/786065956 >> 172.21.15.88:0/786065956 conn(0x7f0a600b3070 msgr2=0x7f0a600b5390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:19.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.206+0000 7f0a6498d640 1 -- 172.21.15.88:0/786065956 shutdown_connections 2024-09-16T01:45:19.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.206+0000 7f0a6498d640 1 -- 172.21.15.88:0/786065956 wait complete. 2024-09-16T01:45:19.217 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T01:45:19.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.315+0000 7f0fc6d91640 1 Processor -- start 2024-09-16T01:45:19.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.315+0000 7f0fc6d91640 1 -- start start 2024-09-16T01:45:19.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.316+0000 7f0fc6d91640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0fc0158a00 0x7f0fc0071f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.316+0000 7f0fc6d91640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fc0072470 0x7f0fc00b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.316+0000 7f0fc6d91640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fc00728f0 con 0x7f0fc0158de0 2024-09-16T01:45:19.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.316+0000 7f0fc6d91640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0fc0072bd0 con 0x7f0fc0158a00 2024-09-16T01:45:19.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.316+0000 7f0fc6d91640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0fc0072eb0 con 0x7f0fc0072470 2024-09-16T01:45:19.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.316+0000 7f0fc532e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0fc0158de0 0x7f0fc0162110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60026/0 (socket says 172.21.15.88:60026) 2024-09-16T01:45:19.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.316+0000 7f0fc532e640 1 -- 172.21.15.88:0/1853295608 learned_addr learned my addr 172.21.15.88:0/1853295608 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:19.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.316+0000 7f0fc4b2d640 1 --2- 172.21.15.88:0/1853295608 >> v2:172.21.15.154:3300/0 conn(0x7f0fc0158a00 0x7f0fc0071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.316+0000 7f0fbffff640 1 --2- 172.21.15.88:0/1853295608 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fc0072470 0x7f0fc00b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.317+0000 7f0fbf7fe640 1 -- 172.21.15.88:0/1853295608 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4207412358 0 0) 0x7f0fc00728f0 con 0x7f0fc0158de0 2024-09-16T01:45:19.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.317+0000 7f0fbf7fe640 1 -- 172.21.15.88:0/1853295608 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fa8003520 con 0x7f0fc0158de0 2024-09-16T01:45:19.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.317+0000 7f0fc4b2d640 1 -- 172.21.15.88:0/1853295608 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fc0072470 msgr2=0x7f0fc00b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:19.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.317+0000 7f0fc4b2d640 1 --2- 172.21.15.88:0/1853295608 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fc0072470 0x7f0fc00b4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.317+0000 7f0fc4b2d640 1 -- 172.21.15.88:0/1853295608 >> v1:172.21.15.88:6789/0 conn(0x7f0fc0158de0 legacy=0x7f0fc0162110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.317+0000 7f0fbf7fe640 1 -- 172.21.15.88:0/1853295608 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2257710638 0 0) 0x7f0fa8003520 con 0x7f0fc0158de0 2024-09-16T01:45:19.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.317+0000 7f0fc4b2d640 1 -- 172.21.15.88:0/1853295608 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc0073190 con 0x7f0fc0158a00 2024-09-16T01:45:19.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.318+0000 7f0fc4b2d640 1 --2- 172.21.15.88:0/1853295608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fc0158a00 0x7f0fc0071f30 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f0fb00097a0 tx=0x7f0fb002a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=821a4e0c118d0477 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:19.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.318+0000 7f0fbf7fe640 1 -- 172.21.15.88:0/1853295608 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb0037070 con 0x7f0fc0158a00 2024-09-16T01:45:19.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.318+0000 7f0fbf7fe640 1 -- 172.21.15.88:0/1853295608 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fb002aba0 con 0x7f0fc0158a00 2024-09-16T01:45:19.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.318+0000 7f0fbf7fe640 1 -- 172.21.15.88:0/1853295608 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb80019f0 con 0x7f0fc0158a00 2024-09-16T01:45:19.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.318+0000 7f0fc6d91640 1 -- 172.21.15.88:0/1853295608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fc0158a00 msgr2=0x7f0fc0071f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.318+0000 7f0fc6d91640 1 --2- 172.21.15.88:0/1853295608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fc0158a00 0x7f0fc0071f30 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f0fb00097a0 tx=0x7f0fb002a110 comp rx=0 tx=0).stop 2024-09-16T01:45:19.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.318+0000 7f0fc6d91640 1 -- 172.21.15.88:0/1853295608 shutdown_connections 2024-09-16T01:45:19.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.318+0000 7f0fc6d91640 1 --2- 172.21.15.88:0/1853295608 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fc0072470 0x7f0fc00b4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.318+0000 7f0fc6d91640 1 --2- 172.21.15.88:0/1853295608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fc0158a00 0x7f0fc0071f30 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.318+0000 7f0fc6d91640 1 -- 172.21.15.88:0/1853295608 >> 172.21.15.88:0/1853295608 conn(0x7f0fc006cb50 msgr2=0x7f0fc006cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:19.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.318+0000 7f0fc6d91640 1 -- 172.21.15.88:0/1853295608 shutdown_connections 2024-09-16T01:45:19.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.318+0000 7f0fc6d91640 1 -- 172.21.15.88:0/1853295608 wait complete. 2024-09-16T01:45:19.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.320+0000 7f0fc6d91640 1 Processor -- start 2024-09-16T01:45:19.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.320+0000 7f0fc6d91640 1 -- start start 2024-09-16T01:45:19.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.320+0000 7f0fc6d91640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fc0072470 0x7f0fc01626f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.320+0000 7f0fc6d91640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fc0158a00 0x7f0fc01f39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.320+0000 7f0fc6d91640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fc0158de0 0x7f0fc01f73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.320+0000 7f0fc6d91640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc00b5ca0 con 0x7f0fc0158de0 2024-09-16T01:45:19.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.320+0000 7f0fc6d91640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc00b5b20 con 0x7f0fc0072470 2024-09-16T01:45:19.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.320+0000 7f0fc6d91640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0fc006faf0 con 0x7f0fc0158a00 2024-09-16T01:45:19.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.321+0000 7f0fc532e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fc0158de0 0x7f0fc01f73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.321+0000 7f0fc532e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fc0158de0 0x7f0fc01f73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60154/0 (socket says 172.21.15.88:60154) 2024-09-16T01:45:19.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.321+0000 7f0fc532e640 1 -- 172.21.15.88:0/1061435371 learned_addr learned my addr 172.21.15.88:0/1061435371 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:19.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.321+0000 7f0fc4b2d640 1 --2- 172.21.15.88:0/1061435371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fc0072470 0x7f0fc01626f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.321+0000 7f0fbffff640 1 --2- 172.21.15.88:0/1061435371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fc0158a00 0x7f0fc01f39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.321+0000 7f0fc532e640 1 -- 172.21.15.88:0/1061435371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fc0158a00 msgr2=0x7f0fc01f39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.321+0000 7f0fc532e640 1 --2- 172.21.15.88:0/1061435371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fc0158a00 0x7f0fc01f39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.321+0000 7f0fc532e640 1 -- 172.21.15.88:0/1061435371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fc0072470 msgr2=0x7f0fc01626f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.321+0000 7f0fc532e640 1 --2- 172.21.15.88:0/1061435371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fc0072470 0x7f0fc01626f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.321+0000 7f0fc532e640 1 -- 172.21.15.88:0/1061435371 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc0161b50 con 0x7f0fc0158de0 2024-09-16T01:45:19.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.321+0000 7f0fc532e640 1 --2- 172.21.15.88:0/1061435371 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fc0158de0 0x7f0fc01f73e0 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7f0fb800cc00 tx=0x7f0fb800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:19.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.321+0000 7f0fbd7fa640 1 -- 172.21.15.88:0/1061435371 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8017070 con 0x7f0fc0158de0 2024-09-16T01:45:19.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.321+0000 7f0fbd7fa640 1 -- 172.21.15.88:0/1061435371 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fb800ce10 con 0x7f0fc0158de0 2024-09-16T01:45:19.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.321+0000 7f0fc6d91640 1 -- 172.21.15.88:0/1061435371 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fc01f43e0 con 0x7f0fc0158de0 2024-09-16T01:45:19.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.321+0000 7f0fbd7fa640 1 -- 172.21.15.88:0/1061435371 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8012640 con 0x7f0fc0158de0 2024-09-16T01:45:19.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.321+0000 7f0fc6d91640 1 -- 172.21.15.88:0/1061435371 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fc01f49a0 con 0x7f0fc0158de0 2024-09-16T01:45:19.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.322+0000 7f0fc6d91640 1 -- 172.21.15.88:0/1061435371 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fc0071f30 con 0x7f0fc0158de0 2024-09-16T01:45:19.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.323+0000 7f0fbd7fa640 1 -- 172.21.15.88:0/1061435371 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f0fb801a070 con 0x7f0fc0158de0 2024-09-16T01:45:19.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.323+0000 7f0fbd7fa640 1 --2- 172.21.15.88:0/1061435371 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0fac06c2e0 0x7f0fac06e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.323+0000 7f0fbd7fa640 1 -- 172.21.15.88:0/1061435371 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7f0fb808aef0 con 0x7f0fc0158de0 2024-09-16T01:45:19.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.323+0000 7f0fc4b2d640 1 --2- 172.21.15.88:0/1061435371 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0fac06c2e0 0x7f0fac06e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.324+0000 7f0fc4b2d640 1 --2- 172.21.15.88:0/1061435371 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0fac06c2e0 0x7f0fac06e7a0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0034000 tx=0x7f0fb002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:19.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.327+0000 7f0fbd7fa640 1 -- 172.21.15.88:0/1061435371 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f0fc0161b50 con 0x7f0fc0158de0 2024-09-16T01:45:19.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.492+0000 7f0fc6d91640 1 -- 172.21.15.88:0/1061435371 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f0fc00b1770 con 0x7f0fac06c2e0 2024-09-16T01:45:19.493 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.492+0000 7f0fbd7fa640 1 -- 172.21.15.88:0/1061435371 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f0fc00b1770 con 0x7f0fac06c2e0 2024-09-16T01:45:19.494 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T01:45:19.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.494+0000 7f0fc6d91640 1 -- 172.21.15.88:0/1061435371 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0fac06c2e0 msgr2=0x7f0fac06e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.494+0000 7f0fc6d91640 1 --2- 172.21.15.88:0/1061435371 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0fac06c2e0 0x7f0fac06e7a0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0034000 tx=0x7f0fb002e000 comp rx=0 tx=0).stop 2024-09-16T01:45:19.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.494+0000 7f0fc6d91640 1 -- 172.21.15.88:0/1061435371 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fc0158de0 msgr2=0x7f0fc01f73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.494+0000 7f0fc6d91640 1 --2- 172.21.15.88:0/1061435371 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fc0158de0 0x7f0fc01f73e0 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7f0fb800cc00 tx=0x7f0fb800a430 comp rx=0 tx=0).stop 2024-09-16T01:45:19.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.494+0000 7f0fc6d91640 1 -- 172.21.15.88:0/1061435371 shutdown_connections 2024-09-16T01:45:19.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.494+0000 7f0fc6d91640 1 --2- 172.21.15.88:0/1061435371 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f0fac06c2e0 0x7f0fac06e7a0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.496 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.494+0000 7f0fc6d91640 1 --2- 172.21.15.88:0/1061435371 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0fc0158de0 0x7f0fc01f73e0 unknown :-1 s=CLOSED pgs=2462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.494+0000 7f0fc6d91640 1 --2- 172.21.15.88:0/1061435371 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0fc0158a00 0x7f0fc01f39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.494+0000 7f0fc6d91640 1 --2- 172.21.15.88:0/1061435371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fc0072470 0x7f0fc01626f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.494+0000 7f0fc6d91640 1 -- 172.21.15.88:0/1061435371 >> 172.21.15.88:0/1061435371 conn(0x7f0fc006cb50 msgr2=0x7f0fc00b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:19.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.494+0000 7f0fc6d91640 1 -- 172.21.15.88:0/1061435371 shutdown_connections 2024-09-16T01:45:19.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.494+0000 7f0fc6d91640 1 -- 172.21.15.88:0/1061435371 wait complete. 2024-09-16T01:45:19.507 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T01:45:19.507 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T01:45:19.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.608+0000 7fa1fefd6640 1 Processor -- start 2024-09-16T01:45:19.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.608+0000 7fa1fefd6640 1 -- start start 2024-09-16T01:45:19.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.609+0000 7fa1fefd6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa1f806fb10 0x7fa1f80b7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.609+0000 7fa1fefd6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1f80719c0 con 0x7fa1f8071450 2024-09-16T01:45:19.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.609+0000 7fa1fefd6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1f80b6ab0 con 0x7fa1f815c490 2024-09-16T01:45:19.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.609+0000 7fa1fefd6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa1f80b6860 con 0x7fa1f806fb10 2024-09-16T01:45:19.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.609+0000 7fa1fd573640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa1f8071450 0x7fa1f80d0270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60042/0 (socket says 172.21.15.88:60042) 2024-09-16T01:45:19.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.609+0000 7fa1fd573640 1 -- 172.21.15.88:0/3981296047 learned_addr learned my addr 172.21.15.88:0/3981296047 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:19.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.609+0000 7fa1f7fff640 1 --2- 172.21.15.88:0/3981296047 >> v2:172.21.15.154:3300/0 conn(0x7fa1f806fb10 0x7fa1f80b7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.609+0000 7fa1f77fe640 1 -- 172.21.15.88:0/3981296047 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4038787618 0 0) 0x7fa1f80719c0 con 0x7fa1f8071450 2024-09-16T01:45:19.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.609+0000 7fa1f77fe640 1 -- 172.21.15.88:0/3981296047 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1e0003520 con 0x7fa1f8071450 2024-09-16T01:45:19.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.609+0000 7fa1f77fe640 1 -- 172.21.15.88:0/3981296047 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3586484013 0 0) 0x7fa1e0003520 con 0x7fa1f8071450 2024-09-16T01:45:19.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.609+0000 7fa1f77fe640 1 -- 172.21.15.88:0/3981296047 >> v2:172.21.15.154:3300/0 conn(0x7fa1f806fb10 msgr2=0x7fa1f80b7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:19.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.609+0000 7fa1f77fe640 1 --2- 172.21.15.88:0/3981296047 >> v2:172.21.15.154:3300/0 conn(0x7fa1f806fb10 0x7fa1f80b7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.609+0000 7fa1f77fe640 1 -- 172.21.15.88:0/3981296047 >> v1:172.21.15.154:6789/0 conn(0x7fa1f815c490 legacy=0x7fa1f806f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.609+0000 7fa1f77fe640 1 -- 172.21.15.88:0/3981296047 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1f80b6d90 con 0x7fa1f8071450 2024-09-16T01:45:19.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.609+0000 7fa1f77fe640 1 -- 172.21.15.88:0/3981296047 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7fa1f0001670 con 0x7fa1f8071450 2024-09-16T01:45:19.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.609+0000 7fa1f77fe640 1 -- 172.21.15.88:0/3981296047 >> v1:172.21.15.88:6789/0 conn(0x7fa1f8071450 legacy=0x7fa1f80d0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.610+0000 7fa1f77fe640 1 --2- 172.21.15.88:0/3981296047 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1e00039b0 0x7fa1e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.610+0000 7fa1f77fe640 1 --2- 172.21.15.88:0/3981296047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1e0004800 0x7fa1e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.610+0000 7fa1f77fe640 1 --2- 172.21.15.88:0/3981296047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1e00051c0 0x7fa1e000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.610+0000 7fa1f77fe640 1 -- 172.21.15.88:0/3981296047 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f80b6860 con 0x7fa1e0004800 2024-09-16T01:45:19.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.610+0000 7fa1f77fe640 1 -- 172.21.15.88:0/3981296047 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa1e000c060 con 0x7fa1e00051c0 2024-09-16T01:45:19.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.610+0000 7fa1f77fe640 1 -- 172.21.15.88:0/3981296047 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa1e000c260 con 0x7fa1e00039b0 2024-09-16T01:45:19.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.610+0000 7fa1f7fff640 1 --2- 172.21.15.88:0/3981296047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1e0004800 0x7fa1e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.610+0000 7fa1fd573640 1 --2- 172.21.15.88:0/3981296047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1e00051c0 0x7fa1e000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.610+0000 7fa1fcd72640 1 --2- 172.21.15.88:0/3981296047 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1e00039b0 0x7fa1e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.610+0000 7fa1f7fff640 1 -- 172.21.15.88:0/3981296047 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1e00039b0 msgr2=0x7fa1e0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:19.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.610+0000 7fa1f7fff640 1 --2- 172.21.15.88:0/3981296047 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1e00039b0 0x7fa1e0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.610+0000 7fa1f7fff640 1 -- 172.21.15.88:0/3981296047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1e00051c0 msgr2=0x7fa1e000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:19.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.610+0000 7fa1f7fff640 1 --2- 172.21.15.88:0/3981296047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1e00051c0 0x7fa1e000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.610+0000 7fa1fd573640 1 -- 172.21.15.88:0/3981296047 reap_dead start 2024-09-16T01:45:19.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.610+0000 7fa1f7fff640 1 -- 172.21.15.88:0/3981296047 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1e0003520 con 0x7fa1e0004800 2024-09-16T01:45:19.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.611+0000 7fa1f7fff640 1 --2- 172.21.15.88:0/3981296047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1e0004800 0x7fa1e0004c80 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec009fc0 tx=0x7fa1e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f80290a780c75773 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:19.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.611+0000 7fa1f77fe640 1 -- 172.21.15.88:0/3981296047 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec009c10 con 0x7fa1e0004800 2024-09-16T01:45:19.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.611+0000 7fa1f77fe640 1 -- 172.21.15.88:0/3981296047 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1ec036070 con 0x7fa1e0004800 2024-09-16T01:45:19.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.611+0000 7fa1f77fe640 1 -- 172.21.15.88:0/3981296047 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec031880 con 0x7fa1e0004800 2024-09-16T01:45:19.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.611+0000 7fa1fefd6640 1 -- 172.21.15.88:0/3981296047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1e0004800 msgr2=0x7fa1e0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.611+0000 7fa1fefd6640 1 --2- 172.21.15.88:0/3981296047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1e0004800 0x7fa1e0004c80 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec009fc0 tx=0x7fa1e0005830 comp rx=0 tx=0).stop 2024-09-16T01:45:19.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.611+0000 7fa1fefd6640 1 -- 172.21.15.88:0/3981296047 shutdown_connections 2024-09-16T01:45:19.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.611+0000 7fa1fefd6640 1 --2- 172.21.15.88:0/3981296047 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1e0004800 0x7fa1e0004c80 unknown :-1 s=CLOSED pgs=2464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.611+0000 7fa1fefd6640 1 -- 172.21.15.88:0/3981296047 >> 172.21.15.88:0/3981296047 conn(0x7fa1f80b4240 msgr2=0x7fa1f80b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:19.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.611+0000 7fa1fefd6640 1 -- 172.21.15.88:0/3981296047 shutdown_connections 2024-09-16T01:45:19.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.611+0000 7fa1fefd6640 1 -- 172.21.15.88:0/3981296047 wait complete. 2024-09-16T01:45:19.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.612+0000 7fa1fefd6640 1 Processor -- start 2024-09-16T01:45:19.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.613+0000 7fa1fefd6640 1 -- start start 2024-09-16T01:45:19.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.613+0000 7fa1fefd6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1e0004800 0x7fa1f8151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.613+0000 7fa1fefd6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1f81510f0 0x7fa1f8146fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.613+0000 7fa1fefd6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1f815a680 0x7fa1f815ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.613+0000 7fa1fefd6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f80b7590 con 0x7fa1e0004800 2024-09-16T01:45:19.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.613+0000 7fa1fefd6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f815c870 con 0x7fa1f81510f0 2024-09-16T01:45:19.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.613+0000 7fa1fefd6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa1f80b7710 con 0x7fa1f815a680 2024-09-16T01:45:19.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.613+0000 7fa1fcd72640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1e0004800 0x7fa1f8151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.613+0000 7fa1fcd72640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1e0004800 0x7fa1f8151890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60164/0 (socket says 172.21.15.88:60164) 2024-09-16T01:45:19.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.613+0000 7fa1fcd72640 1 -- 172.21.15.88:0/1464322105 learned_addr learned my addr 172.21.15.88:0/1464322105 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:19.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.614+0000 7fa1fd573640 1 --2- 172.21.15.88:0/1464322105 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1f815a680 0x7fa1f815ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.614+0000 7fa1f7fff640 1 --2- 172.21.15.88:0/1464322105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1f81510f0 0x7fa1f8146fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.614+0000 7fa1fcd72640 1 -- 172.21.15.88:0/1464322105 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1f815a680 msgr2=0x7fa1f815ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.614+0000 7fa1fcd72640 1 --2- 172.21.15.88:0/1464322105 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1f815a680 0x7fa1f815ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.614+0000 7fa1fcd72640 1 -- 172.21.15.88:0/1464322105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1f81510f0 msgr2=0x7fa1f8146fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.614+0000 7fa1fcd72640 1 --2- 172.21.15.88:0/1464322105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1f81510f0 0x7fa1f8146fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.614+0000 7fa1fcd72640 1 -- 172.21.15.88:0/1464322105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1f81f5a80 con 0x7fa1e0004800 2024-09-16T01:45:19.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.614+0000 7fa1fd573640 1 --2- 172.21.15.88:0/1464322105 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1f815a680 0x7fa1f815ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:45:19.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.614+0000 7fa1f7fff640 1 --2- 172.21.15.88:0/1464322105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1f81510f0 0x7fa1f8146fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:45:19.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.614+0000 7fa1fcd72640 1 --2- 172.21.15.88:0/1464322105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1e0004800 0x7fa1f8151890 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7fa1e800e500 tx=0x7fa1e800e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:19.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.614+0000 7fa1f57fa640 1 -- 172.21.15.88:0/1464322105 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e80053a0 con 0x7fa1e0004800 2024-09-16T01:45:19.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.614+0000 7fa1f57fa640 1 -- 172.21.15.88:0/1464322105 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1e8018070 con 0x7fa1e0004800 2024-09-16T01:45:19.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.614+0000 7fa1fefd6640 1 -- 172.21.15.88:0/1464322105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1f81f5d60 con 0x7fa1e0004800 2024-09-16T01:45:19.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.614+0000 7fa1f57fa640 1 -- 172.21.15.88:0/1464322105 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e8003810 con 0x7fa1e0004800 2024-09-16T01:45:19.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.614+0000 7fa1fefd6640 1 -- 172.21.15.88:0/1464322105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1f81f6320 con 0x7fa1e0004800 2024-09-16T01:45:19.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.615+0000 7fa1fefd6640 1 -- 172.21.15.88:0/1464322105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa1f81f5830 con 0x7fa1e0004800 2024-09-16T01:45:19.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.616+0000 7fa1f57fa640 1 -- 172.21.15.88:0/1464322105 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fa1e80039b0 con 0x7fa1e0004800 2024-09-16T01:45:19.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.616+0000 7fa1f57fa640 1 --2- 172.21.15.88:0/1464322105 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa1e406c2b0 0x7fa1e406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.616+0000 7fa1f7fff640 1 --2- 172.21.15.88:0/1464322105 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa1e406c2b0 0x7fa1e406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.616+0000 7fa1f57fa640 1 -- 172.21.15.88:0/1464322105 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7fa1e808ae00 con 0x7fa1e0004800 2024-09-16T01:45:19.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.616+0000 7fa1f57fa640 1 --2- 172.21.15.88:0/1464322105 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x7fa1e4072210 0x7fa1e4074630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.616+0000 7fa1f57fa640 1 -- 172.21.15.88:0/1464322105 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa1e4074ce0 con 0x7fa1e4072210 2024-09-16T01:45:19.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.616+0000 7fa1f57fa640 1 -- 172.21.15.88:0/1464322105 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=142) ==== 24+0+0 (secure 0 0 0) 0x7fa1e808d500 con 0x7fa1e0004800 2024-09-16T01:45:19.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.616+0000 7fa1fd573640 1 --2- 172.21.15.88:0/1464322105 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x7fa1e4072210 0x7fa1e4074630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.616+0000 7fa1f7fff640 1 --2- 172.21.15.88:0/1464322105 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa1e406c2b0 0x7fa1e406e770 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec035000 tx=0x7fa1ec02f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:19.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.617+0000 7fa1fd573640 1 --2- 172.21.15.88:0/1464322105 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x7fa1e4072210 0x7fa1e4074630 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:19.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.617+0000 7fa1f57fa640 1 -- 172.21.15.88:0/1464322105 <== osd.3 v2:172.21.15.88:6800/697609987 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa1e4074ce0 con 0x7fa1e4072210 2024-09-16T01:45:19.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.629+0000 7fa1fefd6640 1 -- 172.21.15.88:0/1464322105 --> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa1f8076170 con 0x7fa1e4072210 2024-09-16T01:45:19.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.630+0000 7fa1f57fa640 1 -- 172.21.15.88:0/1464322105 <== osd.3 v2:172.21.15.88:6800/697609987 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa1f8076170 con 0x7fa1e4072210 2024-09-16T01:45:19.631 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:45:19.631 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T01:45:19.632 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:45:19.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.630+0000 7fa1fefd6640 1 -- 172.21.15.88:0/1464322105 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x7fa1e4072210 msgr2=0x7fa1e4074630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.630+0000 7fa1fefd6640 1 --2- 172.21.15.88:0/1464322105 >> [v2:172.21.15.88:6800/697609987,v1:172.21.15.88:6801/697609987] conn(0x7fa1e4072210 0x7fa1e4074630 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.630+0000 7fa1fefd6640 1 -- 172.21.15.88:0/1464322105 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa1e406c2b0 msgr2=0x7fa1e406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.630+0000 7fa1fefd6640 1 --2- 172.21.15.88:0/1464322105 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fa1e406c2b0 0x7fa1e406e770 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec035000 tx=0x7fa1ec02f000 comp rx=0 tx=0).stop 2024-09-16T01:45:19.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.630+0000 7fa1fefd6640 1 -- 172.21.15.88:0/1464322105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1e0004800 msgr2=0x7fa1f8151890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.630+0000 7fa1fefd6640 1 --2- 172.21.15.88:0/1464322105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1e0004800 0x7fa1f8151890 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7fa1e800e500 tx=0x7fa1e800e9c0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.630+0000 7fa1fd573640 1 -- 172.21.15.88:0/1464322105 reap_dead start 2024-09-16T01:45:19.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.630+0000 7fa1fefd6640 1 -- 172.21.15.88:0/1464322105 shutdown_connections 2024-09-16T01:45:19.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.630+0000 7fa1fefd6640 1 -- 172.21.15.88:0/1464322105 >> 172.21.15.88:0/1464322105 conn(0x7fa1f80b4240 msgr2=0x7fa1f80d1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:19.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.630+0000 7fa1fefd6640 1 -- 172.21.15.88:0/1464322105 shutdown_connections 2024-09-16T01:45:19.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.630+0000 7fa1fefd6640 1 -- 172.21.15.88:0/1464322105 wait complete. 2024-09-16T01:45:19.642 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T01:45:19.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.740+0000 7f1e11d86640 1 Processor -- start 2024-09-16T01:45:19.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.740+0000 7f1e11d86640 1 -- start start 2024-09-16T01:45:19.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.741+0000 7f1e11d86640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1e0c06fda0 0x7f1e0c0b5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.741+0000 7f1e11d86640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e0c0d1540 con 0x7f1e0c0b59a0 2024-09-16T01:45:19.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.741+0000 7f1e11d86640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e0c06d810 con 0x7f1e0c15e760 2024-09-16T01:45:19.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.741+0000 7f1e11d86640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f1e0c06daf0 con 0x7f1e0c06fda0 2024-09-16T01:45:19.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.741+0000 7f1e0bfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1e0c0b59a0 0x7f1e0c0d0350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60054/0 (socket says 172.21.15.88:60054) 2024-09-16T01:45:19.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.741+0000 7f1e0bfff640 1 -- 172.21.15.88:0/1580363149 learned_addr learned my addr 172.21.15.88:0/1580363149 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:19.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.741+0000 7f1e0affd640 1 --2- 172.21.15.88:0/1580363149 >> v2:172.21.15.154:3300/0 conn(0x7f1e0c06fda0 0x7f1e0c0b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.741+0000 7f1e0a7fc640 1 -- 172.21.15.88:0/1580363149 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3700527530 0 0) 0x7f1e0c0d1540 con 0x7f1e0c0b59a0 2024-09-16T01:45:19.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.741+0000 7f1e0a7fc640 1 -- 172.21.15.88:0/1580363149 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1df4003520 con 0x7f1e0c0b59a0 2024-09-16T01:45:19.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.741+0000 7f1e0affd640 1 -- 172.21.15.88:0/1580363149 >> v1:172.21.15.154:6789/0 conn(0x7f1e0c15e760 legacy=0x7f1e0c073d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.741+0000 7f1e0affd640 1 -- 172.21.15.88:0/1580363149 >> v1:172.21.15.88:6789/0 conn(0x7f1e0c0b59a0 legacy=0x7f1e0c0d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.741+0000 7f1e0a7fc640 1 -- 172.21.15.88:0/1580363149 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 405189950 0 0) 0x7f1df4003520 con 0x7f1e0c0b59a0 2024-09-16T01:45:19.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.742+0000 7f1e0affd640 1 -- 172.21.15.88:0/1580363149 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e0c15eb40 con 0x7f1e0c06fda0 2024-09-16T01:45:19.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.742+0000 7f1e0affd640 1 --2- 172.21.15.88:0/1580363149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e0c06fda0 0x7f1e0c0b5460 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f1e000097a0 tx=0x7f1e0002a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86ae6c99300004eb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:19.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.742+0000 7f1e0a7fc640 1 -- 172.21.15.88:0/1580363149 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e00037070 con 0x7f1e0c06fda0 2024-09-16T01:45:19.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.742+0000 7f1e0a7fc640 1 -- 172.21.15.88:0/1580363149 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e0002ac50 con 0x7f1e0c06fda0 2024-09-16T01:45:19.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.743+0000 7f1e0a7fc640 1 -- 172.21.15.88:0/1580363149 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e040019f0 con 0x7f1e0c06fda0 2024-09-16T01:45:19.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.743+0000 7f1e11d86640 1 -- 172.21.15.88:0/1580363149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e0c06fda0 msgr2=0x7f1e0c0b5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.743+0000 7f1e11d86640 1 --2- 172.21.15.88:0/1580363149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e0c06fda0 0x7f1e0c0b5460 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f1e000097a0 tx=0x7f1e0002a1c0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.743+0000 7f1e11d86640 1 -- 172.21.15.88:0/1580363149 shutdown_connections 2024-09-16T01:45:19.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.743+0000 7f1e11d86640 1 --2- 172.21.15.88:0/1580363149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e0c06fda0 0x7f1e0c0b5460 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.743+0000 7f1e11d86640 1 -- 172.21.15.88:0/1580363149 >> 172.21.15.88:0/1580363149 conn(0x7f1e0c0b7d70 msgr2=0x7f1e0c062630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:19.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.743+0000 7f1e11d86640 1 -- 172.21.15.88:0/1580363149 shutdown_connections 2024-09-16T01:45:19.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.743+0000 7f1e11d86640 1 -- 172.21.15.88:0/1580363149 wait complete. 2024-09-16T01:45:19.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.744+0000 7f1e11d86640 1 Processor -- start 2024-09-16T01:45:19.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.744+0000 7f1e11d86640 1 -- start start 2024-09-16T01:45:19.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.745+0000 7f1e11d86640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e0c06fda0 0x7f1e0c144330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.745+0000 7f1e11d86640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e0c0b59a0 0x7f1e0c15f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.745+0000 7f1e11d86640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e0c15e760 0x7f1e0c159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.745+0000 7f1e11d86640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f1e0c0b5fb0 con 0x7f1e0c15e760 2024-09-16T01:45:19.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.745+0000 7f1e11d86640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1e0c0b5e30 con 0x7f1e0c06fda0 2024-09-16T01:45:19.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.745+0000 7f1e11d86640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1e0c070230 con 0x7f1e0c0b59a0 2024-09-16T01:45:19.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.745+0000 7f1e0bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e0c15e760 0x7f1e0c159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.745+0000 7f1e0bfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e0c15e760 0x7f1e0c159fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60166/0 (socket says 172.21.15.88:60166) 2024-09-16T01:45:19.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.745+0000 7f1e0bfff640 1 -- 172.21.15.88:0/1877097546 learned_addr learned my addr 172.21.15.88:0/1877097546 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:19.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.745+0000 7f1e0b7fe640 1 --2- 172.21.15.88:0/1877097546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e0c06fda0 0x7f1e0c144330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.745+0000 7f1e0affd640 1 --2- 172.21.15.88:0/1877097546 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e0c0b59a0 0x7f1e0c15f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.746+0000 7f1e0bfff640 1 -- 172.21.15.88:0/1877097546 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e0c0b59a0 msgr2=0x7f1e0c15f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.746+0000 7f1e0bfff640 1 --2- 172.21.15.88:0/1877097546 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e0c0b59a0 0x7f1e0c15f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.746+0000 7f1e0bfff640 1 -- 172.21.15.88:0/1877097546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e0c06fda0 msgr2=0x7f1e0c144330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.746+0000 7f1e0bfff640 1 --2- 172.21.15.88:0/1877097546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e0c06fda0 0x7f1e0c144330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.746+0000 7f1e0bfff640 1 -- 172.21.15.88:0/1877097546 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e0c160180 con 0x7f1e0c15e760 2024-09-16T01:45:19.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.746+0000 7f1e0b7fe640 1 --2- 172.21.15.88:0/1877097546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e0c06fda0 0x7f1e0c144330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:45:19.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.746+0000 7f1e0bfff640 1 --2- 172.21.15.88:0/1877097546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e0c15e760 0x7f1e0c159fd0 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f1e0400cca0 tx=0x7f1e0400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:19.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.746+0000 7f1df3fff640 1 -- 172.21.15.88:0/1877097546 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e04017070 con 0x7f1e0c15e760 2024-09-16T01:45:19.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.746+0000 7f1df3fff640 1 -- 172.21.15.88:0/1877097546 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e04012400 con 0x7f1e0c15e760 2024-09-16T01:45:19.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.746+0000 7f1e11d86640 1 -- 172.21.15.88:0/1877097546 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e0c160460 con 0x7f1e0c15e760 2024-09-16T01:45:19.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.746+0000 7f1df3fff640 1 -- 172.21.15.88:0/1877097546 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e040127f0 con 0x7f1e0c15e760 2024-09-16T01:45:19.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.746+0000 7f1e11d86640 1 -- 172.21.15.88:0/1877097546 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e0c15fd20 con 0x7f1e0c15e760 2024-09-16T01:45:19.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.747+0000 7f1e11d86640 1 -- 172.21.15.88:0/1877097546 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e0c06c740 con 0x7f1e0c15e760 2024-09-16T01:45:19.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.747+0000 7f1df3fff640 1 -- 172.21.15.88:0/1877097546 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f1e0401a070 con 0x7f1e0c15e760 2024-09-16T01:45:19.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.748+0000 7f1df3fff640 1 --2- 172.21.15.88:0/1877097546 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1df806c2b0 0x7f1df806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:19.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.748+0000 7f1df3fff640 1 -- 172.21.15.88:0/1877097546 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7f1e0408b0d0 con 0x7f1e0c15e760 2024-09-16T01:45:19.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.748+0000 7f1e0b7fe640 1 --2- 172.21.15.88:0/1877097546 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1df806c2b0 0x7f1df806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:19.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.748+0000 7f1e0b7fe640 1 --2- 172.21.15.88:0/1877097546 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1df806c2b0 0x7f1df806e770 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc007fd0 tx=0x7f1dfc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:19.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.752+0000 7f1df3fff640 1 -- 172.21.15.88:0/1877097546 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f1e04059b70 con 0x7f1e0c15e760 2024-09-16T01:45:19.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.918+0000 7f1e11d86640 1 -- 172.21.15.88:0/1877097546 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f1e0c160980 con 0x7f1df806c2b0 2024-09-16T01:45:19.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.918+0000 7f1df3fff640 1 -- 172.21.15.88:0/1877097546 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f1e0c160980 con 0x7f1df806c2b0 2024-09-16T01:45:19.920 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T01:45:19.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.920+0000 7f1e11d86640 1 -- 172.21.15.88:0/1877097546 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1df806c2b0 msgr2=0x7f1df806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.920+0000 7f1e11d86640 1 --2- 172.21.15.88:0/1877097546 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1df806c2b0 0x7f1df806e770 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc007fd0 tx=0x7f1dfc009040 comp rx=0 tx=0).stop 2024-09-16T01:45:19.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.920+0000 7f1e11d86640 1 -- 172.21.15.88:0/1877097546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e0c15e760 msgr2=0x7f1e0c159fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:19.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.920+0000 7f1e11d86640 1 --2- 172.21.15.88:0/1877097546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e0c15e760 0x7f1e0c159fd0 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f1e0400cca0 tx=0x7f1e0400a430 comp rx=0 tx=0).stop 2024-09-16T01:45:19.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.920+0000 7f1e11d86640 1 -- 172.21.15.88:0/1877097546 shutdown_connections 2024-09-16T01:45:19.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.920+0000 7f1e11d86640 1 --2- 172.21.15.88:0/1877097546 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f1df806c2b0 0x7f1df806e770 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.920+0000 7f1e11d86640 1 --2- 172.21.15.88:0/1877097546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e0c15e760 0x7f1e0c159fd0 unknown :-1 s=CLOSED pgs=2467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.920+0000 7f1e11d86640 1 --2- 172.21.15.88:0/1877097546 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e0c0b59a0 0x7f1e0c15f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.920+0000 7f1e11d86640 1 --2- 172.21.15.88:0/1877097546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e0c06fda0 0x7f1e0c144330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:19.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.920+0000 7f1e11d86640 1 -- 172.21.15.88:0/1877097546 >> 172.21.15.88:0/1877097546 conn(0x7f1e0c0b7d70 msgr2=0x7f1e0c0b6130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:19.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.920+0000 7f1e11d86640 1 -- 172.21.15.88:0/1877097546 shutdown_connections 2024-09-16T01:45:19.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:19.920+0000 7f1e11d86640 1 -- 172.21.15.88:0/1877097546 wait complete. 2024-09-16T01:45:19.932 INFO:tasks.ceph:Scrubbing osd.4 2024-09-16T01:45:19.933 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-16T01:45:20.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.035+0000 7f9228952640 1 Processor -- start 2024-09-16T01:45:20.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.035+0000 7f9228952640 1 -- start start 2024-09-16T01:45:20.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.036+0000 7f9228952640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f922415e570 0x7f92240b5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.036+0000 7f9228952640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f922406fb10 0x7f922415e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.036+0000 7f9228952640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f922415f020 con 0x7f92240714a0 2024-09-16T01:45:20.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.036+0000 7f9228952640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f922415fc30 con 0x7f922406fb10 2024-09-16T01:45:20.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.036+0000 7f9228952640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f922415f300 con 0x7f922415e570 2024-09-16T01:45:20.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.036+0000 7f9222d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f922406fb10 0x7f922415e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.036+0000 7f9219d9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f92240714a0 0x7f922406f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60058/0 (socket says 172.21.15.88:60058) 2024-09-16T01:45:20.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.036+0000 7f9219d9b640 1 -- 172.21.15.88:0/908929096 learned_addr learned my addr 172.21.15.88:0/908929096 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:20.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.036+0000 7f922259c640 1 --2- 172.21.15.88:0/908929096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f922415e570 0x7f92240b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.036+0000 7f9222d9d640 1 -- 172.21.15.88:0/908929096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f922415e570 msgr2=0x7f92240b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:20.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.036+0000 7f9222d9d640 1 --2- 172.21.15.88:0/908929096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f922415e570 0x7f92240b5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.036+0000 7f9222d9d640 1 -- 172.21.15.88:0/908929096 >> v1:172.21.15.88:6789/0 conn(0x7f92240714a0 legacy=0x7f922406f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.036+0000 7f9222d9d640 1 -- 172.21.15.88:0/908929096 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f922415f5e0 con 0x7f922406fb10 2024-09-16T01:45:20.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.036+0000 7f9221d9b640 1 -- 172.21.15.88:0/908929096 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3451380048 0 0) 0x7f922415f020 con 0x7f92240714a0 2024-09-16T01:45:20.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.037+0000 7f9222d9d640 1 --2- 172.21.15.88:0/908929096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f922406fb10 0x7f922415e950 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f921c009d90 tx=0x7f921c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=249e9f6ec1704a4d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:20.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.037+0000 7f9221d9b640 1 -- 172.21.15.88:0/908929096 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f921c007740 con 0x7f922406fb10 2024-09-16T01:45:20.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.037+0000 7f9221d9b640 1 -- 172.21.15.88:0/908929096 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f921c0078e0 con 0x7f922406fb10 2024-09-16T01:45:20.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.037+0000 7f9221d9b640 1 -- 172.21.15.88:0/908929096 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f921c007cb0 con 0x7f922406fb10 2024-09-16T01:45:20.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.037+0000 7f9228952640 1 -- 172.21.15.88:0/908929096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f922406fb10 msgr2=0x7f922415e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.037+0000 7f9228952640 1 --2- 172.21.15.88:0/908929096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f922406fb10 0x7f922415e950 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f921c009d90 tx=0x7f921c02a660 comp rx=0 tx=0).stop 2024-09-16T01:45:20.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.037+0000 7f9228952640 1 -- 172.21.15.88:0/908929096 shutdown_connections 2024-09-16T01:45:20.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.037+0000 7f9228952640 1 --2- 172.21.15.88:0/908929096 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f922406fb10 0x7f922415e950 unknown :-1 s=CLOSED pgs=2469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.037+0000 7f9228952640 1 --2- 172.21.15.88:0/908929096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f922415e570 0x7f92240b5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.037+0000 7f9228952640 1 -- 172.21.15.88:0/908929096 >> 172.21.15.88:0/908929096 conn(0x7f92240b4240 msgr2=0x7f92240b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:20.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.037+0000 7f9228952640 1 -- 172.21.15.88:0/908929096 shutdown_connections 2024-09-16T01:45:20.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.037+0000 7f9228952640 1 -- 172.21.15.88:0/908929096 wait complete. 2024-09-16T01:45:20.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.039+0000 7f9228952640 1 Processor -- start 2024-09-16T01:45:20.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.039+0000 7f9228952640 1 -- start start 2024-09-16T01:45:20.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.040+0000 7f9228952640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f922406fb10 0x7f9224151ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.040+0000 7f9228952640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92240714a0 0x7f92241f4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.040+0000 7f9228952640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f922415e570 0x7f922415c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.040+0000 7f9228952640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f922406d0e0 con 0x7f922406fb10 2024-09-16T01:45:20.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.040+0000 7f9228952640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f922406cf60 con 0x7f92240714a0 2024-09-16T01:45:20.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.040+0000 7f9228952640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f922406d260 con 0x7f922415e570 2024-09-16T01:45:20.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.040+0000 7f922259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f922406fb10 0x7f9224151ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.040+0000 7f922259c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f922406fb10 0x7f9224151ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60192/0 (socket says 172.21.15.88:60192) 2024-09-16T01:45:20.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.040+0000 7f922259c640 1 -- 172.21.15.88:0/1888641146 learned_addr learned my addr 172.21.15.88:0/1888641146 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:20.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.040+0000 7f9222d9d640 1 --2- 172.21.15.88:0/1888641146 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f922415e570 0x7f922415c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.040+0000 7f9219d9b640 1 --2- 172.21.15.88:0/1888641146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92240714a0 0x7f92241f4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.040+0000 7f922259c640 1 -- 172.21.15.88:0/1888641146 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f922415e570 msgr2=0x7f922415c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.040+0000 7f922259c640 1 --2- 172.21.15.88:0/1888641146 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f922415e570 0x7f922415c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.040+0000 7f922259c640 1 -- 172.21.15.88:0/1888641146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92240714a0 msgr2=0x7f92241f4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.040+0000 7f922259c640 1 --2- 172.21.15.88:0/1888641146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92240714a0 0x7f92241f4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.040+0000 7f922259c640 1 -- 172.21.15.88:0/1888641146 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92241f5150 con 0x7f922406fb10 2024-09-16T01:45:20.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.040+0000 7f9222d9d640 1 --2- 172.21.15.88:0/1888641146 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f922415e570 0x7f922415c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:45:20.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.040+0000 7f922259c640 1 --2- 172.21.15.88:0/1888641146 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f922406fb10 0x7f9224151ca0 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f921400d5f0 tx=0x7f921400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:20.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.040+0000 7f921b7fe640 1 -- 172.21.15.88:0/1888641146 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92140125f0 con 0x7f922406fb10 2024-09-16T01:45:20.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.040+0000 7f921b7fe640 1 -- 172.21.15.88:0/1888641146 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9214019070 con 0x7f922406fb10 2024-09-16T01:45:20.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.040+0000 7f9228952640 1 -- 172.21.15.88:0/1888641146 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9224152c70 con 0x7f922406fb10 2024-09-16T01:45:20.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.040+0000 7f921b7fe640 1 -- 172.21.15.88:0/1888641146 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92140129d0 con 0x7f922406fb10 2024-09-16T01:45:20.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.040+0000 7f9228952640 1 -- 172.21.15.88:0/1888641146 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9224153230 con 0x7f922406fb10 2024-09-16T01:45:20.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.041+0000 7f9228952640 1 -- 172.21.15.88:0/1888641146 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f92240b6920 con 0x7f922406fb10 2024-09-16T01:45:20.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.042+0000 7f921b7fe640 1 -- 172.21.15.88:0/1888641146 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f9214012b70 con 0x7f922406fb10 2024-09-16T01:45:20.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.042+0000 7f921b7fe640 1 --2- 172.21.15.88:0/1888641146 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f920406c2b0 0x7f920406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.042+0000 7f921b7fe640 1 -- 172.21.15.88:0/1888641146 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7f921408b5a0 con 0x7f922406fb10 2024-09-16T01:45:20.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.042+0000 7f9219d9b640 1 --2- 172.21.15.88:0/1888641146 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f920406c2b0 0x7f920406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.042+0000 7f921b7fe640 1 --2- 172.21.15.88:0/1888641146 >> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] conn(0x7f9204072210 0x7f9204074630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.042+0000 7f921b7fe640 1 -- 172.21.15.88:0/1888641146 --> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9204074ce0 con 0x7f9204072210 2024-09-16T01:45:20.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.042+0000 7f921b7fe640 1 -- 172.21.15.88:0/1888641146 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=142) ==== 24+0+0 (secure 0 0 0) 0x7f9214092050 con 0x7f922406fb10 2024-09-16T01:45:20.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.043+0000 7f9222d9d640 1 --2- 172.21.15.88:0/1888641146 >> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] conn(0x7f9204072210 0x7f9204074630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.043+0000 7f9219d9b640 1 --2- 172.21.15.88:0/1888641146 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f920406c2b0 0x7f920406e770 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f92240b36d0 tx=0x7f920c007b50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:20.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.043+0000 7f9222d9d640 1 --2- 172.21.15.88:0/1888641146 >> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] conn(0x7f9204072210 0x7f9204074630 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:20.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.044+0000 7f921b7fe640 1 -- 172.21.15.88:0/1888641146 <== osd.4 v2:172.21.15.154:6816/2520717231 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9204074ce0 con 0x7f9204072210 2024-09-16T01:45:20.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.056+0000 7f9228952640 1 -- 172.21.15.88:0/1888641146 --> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f9224152920 con 0x7f9204072210 2024-09-16T01:45:20.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.057+0000 7f921b7fe640 1 -- 172.21.15.88:0/1888641146 <== osd.4 v2:172.21.15.154:6816/2520717231 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f9224152920 con 0x7f9204072210 2024-09-16T01:45:20.058 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:45:20.059 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T01:45:20.059 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:45:20.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.057+0000 7f9228952640 1 -- 172.21.15.88:0/1888641146 >> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] conn(0x7f9204072210 msgr2=0x7f9204074630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.057+0000 7f9228952640 1 --2- 172.21.15.88:0/1888641146 >> [v2:172.21.15.154:6816/2520717231,v1:172.21.15.154:6817/2520717231] conn(0x7f9204072210 0x7f9204074630 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.057+0000 7f9228952640 1 -- 172.21.15.88:0/1888641146 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f920406c2b0 msgr2=0x7f920406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.057+0000 7f9228952640 1 --2- 172.21.15.88:0/1888641146 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f920406c2b0 0x7f920406e770 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f92240b36d0 tx=0x7f920c007b50 comp rx=0 tx=0).stop 2024-09-16T01:45:20.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.057+0000 7f9228952640 1 -- 172.21.15.88:0/1888641146 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f922406fb10 msgr2=0x7f9224151ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.057+0000 7f9228952640 1 --2- 172.21.15.88:0/1888641146 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f922406fb10 0x7f9224151ca0 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f921400d5f0 tx=0x7f921400dab0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.057+0000 7f9222d9d640 1 -- 172.21.15.88:0/1888641146 reap_dead start 2024-09-16T01:45:20.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.057+0000 7f9228952640 1 -- 172.21.15.88:0/1888641146 shutdown_connections 2024-09-16T01:45:20.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.057+0000 7f9228952640 1 -- 172.21.15.88:0/1888641146 >> 172.21.15.88:0/1888641146 conn(0x7f92240b4240 msgr2=0x7f9224073980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:20.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.057+0000 7f9228952640 1 -- 172.21.15.88:0/1888641146 shutdown_connections 2024-09-16T01:45:20.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.057+0000 7f9228952640 1 -- 172.21.15.88:0/1888641146 wait complete. 2024-09-16T01:45:20.069 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-16T01:45:20.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.167+0000 7f70cc9f8640 1 Processor -- start 2024-09-16T01:45:20.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.167+0000 7f70cc9f8640 1 -- start start 2024-09-16T01:45:20.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.168+0000 7f70cc9f8640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f70c815c260 0x7f70c80d1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.168+0000 7f70cc9f8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70c8072c70 con 0x7f70c80b3570 2024-09-16T01:45:20.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.168+0000 7f70cc9f8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70c80d1190 con 0x7f70c8072690 2024-09-16T01:45:20.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.168+0000 7f70cc9f8640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f70c80d21b0 con 0x7f70c815c260 2024-09-16T01:45:20.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.168+0000 7f70c659c640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f70c815c260 0x7f70c80d1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.168+0000 7f70bffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f70c80b3570 0x7f70c8071f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60068/0 (socket says 172.21.15.88:60068) 2024-09-16T01:45:20.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.168+0000 7f70bffff640 1 -- 172.21.15.88:0/4186917578 learned_addr learned my addr 172.21.15.88:0/4186917578 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:20.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.168+0000 7f70c659c640 1 -- 172.21.15.88:0/4186917578 >> v1:172.21.15.154:6789/0 conn(0x7f70c8072690 legacy=0x7f70c806ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.168+0000 7f70c659c640 1 -- 172.21.15.88:0/4186917578 >> v1:172.21.15.88:6789/0 conn(0x7f70c80b3570 legacy=0x7f70c8071f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.168+0000 7f70c5d9b640 1 -- 172.21.15.88:0/4186917578 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3726969083 0 0) 0x7f70c8072c70 con 0x7f70c80b3570 2024-09-16T01:45:20.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.168+0000 7f70c659c640 1 -- 172.21.15.88:0/4186917578 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70c80d1470 con 0x7f70c815c260 2024-09-16T01:45:20.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.169+0000 7f70c659c640 1 --2- 172.21.15.88:0/4186917578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70c815c260 0x7f70c80d1c70 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f70b80097a0 tx=0x7f70b802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4a7afe4e77e823a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:20.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.169+0000 7f70c5d9b640 1 -- 172.21.15.88:0/4186917578 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b8037070 con 0x7f70c815c260 2024-09-16T01:45:20.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.169+0000 7f70c5d9b640 1 -- 172.21.15.88:0/4186917578 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70b8032440 con 0x7f70c815c260 2024-09-16T01:45:20.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.169+0000 7f70c5d9b640 1 -- 172.21.15.88:0/4186917578 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b8032810 con 0x7f70c815c260 2024-09-16T01:45:20.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.170+0000 7f70cc9f8640 1 -- 172.21.15.88:0/4186917578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70c815c260 msgr2=0x7f70c80d1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.170+0000 7f70cc9f8640 1 --2- 172.21.15.88:0/4186917578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70c815c260 0x7f70c80d1c70 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f70b80097a0 tx=0x7f70b802a1a0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.170+0000 7f70cc9f8640 1 -- 172.21.15.88:0/4186917578 shutdown_connections 2024-09-16T01:45:20.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.170+0000 7f70cc9f8640 1 --2- 172.21.15.88:0/4186917578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70c815c260 0x7f70c80d1c70 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.170+0000 7f70cc9f8640 1 -- 172.21.15.88:0/4186917578 >> 172.21.15.88:0/4186917578 conn(0x7f70c80b6b50 msgr2=0x7f70c80b6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:20.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.170+0000 7f70cc9f8640 1 -- 172.21.15.88:0/4186917578 shutdown_connections 2024-09-16T01:45:20.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.170+0000 7f70cc9f8640 1 -- 172.21.15.88:0/4186917578 wait complete. 2024-09-16T01:45:20.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.171+0000 7f70cc9f8640 1 Processor -- start 2024-09-16T01:45:20.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.171+0000 7f70cc9f8640 1 -- start start 2024-09-16T01:45:20.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.172+0000 7f70cc9f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f70c8072690 0x7f70c8142b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.172+0000 7f70cc9f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70c80b3570 0x7f70c81467b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.172+0000 7f70cc9f8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70c815c260 0x7f70c81541c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.172+0000 7f70cc9f8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f70c80d17e0 con 0x7f70c815c260 2024-09-16T01:45:20.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.172+0000 7f70cc9f8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f70c8072f50 con 0x7f70c80b3570 2024-09-16T01:45:20.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.172+0000 7f70cc9f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f70c806d150 con 0x7f70c8072690 2024-09-16T01:45:20.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.172+0000 7f70c6d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70c815c260 0x7f70c81541c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.172+0000 7f70c6d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70c815c260 0x7f70c81541c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60204/0 (socket says 172.21.15.88:60204) 2024-09-16T01:45:20.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.172+0000 7f70c6d9d640 1 -- 172.21.15.88:0/3234790862 learned_addr learned my addr 172.21.15.88:0/3234790862 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:20.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.172+0000 7f70c659c640 1 --2- 172.21.15.88:0/3234790862 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f70c8072690 0x7f70c8142b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.172+0000 7f70bffff640 1 --2- 172.21.15.88:0/3234790862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70c80b3570 0x7f70c81467b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.172+0000 7f70c6d9d640 1 -- 172.21.15.88:0/3234790862 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f70c8072690 msgr2=0x7f70c8142b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.172+0000 7f70c6d9d640 1 --2- 172.21.15.88:0/3234790862 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f70c8072690 0x7f70c8142b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.172+0000 7f70c6d9d640 1 -- 172.21.15.88:0/3234790862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70c80b3570 msgr2=0x7f70c81467b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.172+0000 7f70c6d9d640 1 --2- 172.21.15.88:0/3234790862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70c80b3570 0x7f70c81467b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.172+0000 7f70c6d9d640 1 -- 172.21.15.88:0/3234790862 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70c8143460 con 0x7f70c815c260 2024-09-16T01:45:20.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.172+0000 7f70bffff640 1 --2- 172.21.15.88:0/3234790862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70c80b3570 0x7f70c81467b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:45:20.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.172+0000 7f70c659c640 1 --2- 172.21.15.88:0/3234790862 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f70c8072690 0x7f70c8142b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:45:20.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.172+0000 7f70c6d9d640 1 --2- 172.21.15.88:0/3234790862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70c815c260 0x7f70c81541c0 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f70c000c7a0 tx=0x7f70c000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:20.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.172+0000 7f70beffd640 1 -- 172.21.15.88:0/3234790862 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c0003b70 con 0x7f70c815c260 2024-09-16T01:45:20.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.172+0000 7f70beffd640 1 -- 172.21.15.88:0/3234790862 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70c0003d30 con 0x7f70c815c260 2024-09-16T01:45:20.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.172+0000 7f70cc9f8640 1 -- 172.21.15.88:0/3234790862 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70c8143740 con 0x7f70c815c260 2024-09-16T01:45:20.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.172+0000 7f70beffd640 1 -- 172.21.15.88:0/3234790862 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c000aa90 con 0x7f70c815c260 2024-09-16T01:45:20.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.172+0000 7f70cc9f8640 1 -- 172.21.15.88:0/3234790862 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70c8143d00 con 0x7f70c815c260 2024-09-16T01:45:20.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.173+0000 7f70cc9f8640 1 -- 172.21.15.88:0/3234790862 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70c806d4e0 con 0x7f70c815c260 2024-09-16T01:45:20.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.174+0000 7f70beffd640 1 -- 172.21.15.88:0/3234790862 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f70c000ac30 con 0x7f70c815c260 2024-09-16T01:45:20.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.174+0000 7f70beffd640 1 --2- 172.21.15.88:0/3234790862 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f70a806c2b0 0x7f70a806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.174+0000 7f70c659c640 1 --2- 172.21.15.88:0/3234790862 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f70a806c2b0 0x7f70a806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.174+0000 7f70beffd640 1 -- 172.21.15.88:0/3234790862 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7f70c008b0d0 con 0x7f70c815c260 2024-09-16T01:45:20.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.175+0000 7f70c659c640 1 --2- 172.21.15.88:0/3234790862 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f70a806c2b0 0x7f70a806e770 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f70b8034000 tx=0x7f70b802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:20.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.179+0000 7f70beffd640 1 -- 172.21.15.88:0/3234790862 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f70c0059b70 con 0x7f70c815c260 2024-09-16T01:45:20.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.343+0000 7f70cc9f8640 1 -- 172.21.15.88:0/3234790862 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f70c8073160 con 0x7f70a806c2b0 2024-09-16T01:45:20.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.343+0000 7f70beffd640 1 -- 172.21.15.88:0/3234790862 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f70c8073160 con 0x7f70a806c2b0 2024-09-16T01:45:20.345 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 4 to deep-scrub 2024-09-16T01:45:20.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.345+0000 7f70cc9f8640 1 -- 172.21.15.88:0/3234790862 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f70a806c2b0 msgr2=0x7f70a806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.345+0000 7f70cc9f8640 1 --2- 172.21.15.88:0/3234790862 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f70a806c2b0 0x7f70a806e770 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f70b8034000 tx=0x7f70b802e000 comp rx=0 tx=0).stop 2024-09-16T01:45:20.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.345+0000 7f70cc9f8640 1 -- 172.21.15.88:0/3234790862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70c815c260 msgr2=0x7f70c81541c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.345+0000 7f70cc9f8640 1 --2- 172.21.15.88:0/3234790862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70c815c260 0x7f70c81541c0 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f70c000c7a0 tx=0x7f70c000cc60 comp rx=0 tx=0).stop 2024-09-16T01:45:20.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.345+0000 7f70cc9f8640 1 -- 172.21.15.88:0/3234790862 shutdown_connections 2024-09-16T01:45:20.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.345+0000 7f70cc9f8640 1 --2- 172.21.15.88:0/3234790862 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f70a806c2b0 0x7f70a806e770 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.345+0000 7f70cc9f8640 1 --2- 172.21.15.88:0/3234790862 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70c815c260 0x7f70c81541c0 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.345+0000 7f70cc9f8640 1 --2- 172.21.15.88:0/3234790862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70c80b3570 0x7f70c81467b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.345+0000 7f70cc9f8640 1 --2- 172.21.15.88:0/3234790862 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f70c8072690 0x7f70c8142b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.345+0000 7f70cc9f8640 1 -- 172.21.15.88:0/3234790862 >> 172.21.15.88:0/3234790862 conn(0x7f70c80b6b50 msgr2=0x7f70c8073380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:20.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.345+0000 7f70cc9f8640 1 -- 172.21.15.88:0/3234790862 shutdown_connections 2024-09-16T01:45:20.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.345+0000 7f70cc9f8640 1 -- 172.21.15.88:0/3234790862 wait complete. 2024-09-16T01:45:20.357 INFO:tasks.ceph:Scrubbing osd.5 2024-09-16T01:45:20.357 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-16T01:45:20.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.459+0000 7fd075d9f640 1 Processor -- start 2024-09-16T01:45:20.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.459+0000 7fd075d9f640 1 -- start start 2024-09-16T01:45:20.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.460+0000 7fd075d9f640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd070158d70 0x7fd07015ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.460+0000 7fd075d9f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0700b5620 0x7fd070072f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.460+0000 7fd075d9f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd0700734a0 0x7fd0701621c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.460+0000 7fd075d9f640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fd07006ce50 con 0x7fd070158d70 2024-09-16T01:45:20.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.460+0000 7fd075d9f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd07015f200 con 0x7fd0700734a0 2024-09-16T01:45:20.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.460+0000 7fd075d9f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd07015f4e0 con 0x7fd0700b5620 2024-09-16T01:45:20.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.460+0000 7fd074d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd070158d70 0x7fd07015ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.460+0000 7fd074d9d640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fd070158d70 0x7fd07015ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47612/0 (socket says 172.21.15.88:47612) 2024-09-16T01:45:20.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.460+0000 7fd074d9d640 1 -- 172.21.15.88:0/333433944 learned_addr learned my addr 172.21.15.88:0/333433944 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:20.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.460+0000 7fd07559e640 1 --2- 172.21.15.88:0/333433944 >> v2:172.21.15.154:3300/0 conn(0x7fd0700734a0 0x7fd0701621c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.460+0000 7fd06ffff640 1 --2- 172.21.15.88:0/333433944 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0700b5620 0x7fd070072f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.461+0000 7fd074d9d640 1 -- 172.21.15.88:0/333433944 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0700b5620 msgr2=0x7fd070072f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:20.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.461+0000 7fd074d9d640 1 --2- 172.21.15.88:0/333433944 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0700b5620 0x7fd070072f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.461+0000 7fd074d9d640 1 -- 172.21.15.88:0/333433944 >> v2:172.21.15.154:3300/0 conn(0x7fd0700734a0 msgr2=0x7fd0701621c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:20.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.461+0000 7fd074d9d640 1 --2- 172.21.15.88:0/333433944 >> v2:172.21.15.154:3300/0 conn(0x7fd0700734a0 0x7fd0701621c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.461+0000 7fd074d9d640 1 -- 172.21.15.88:0/333433944 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd07015f7c0 con 0x7fd070158d70 2024-09-16T01:45:20.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.461+0000 7fd074d9d640 1 --2- 172.21.15.88:0/333433944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd070158d70 0x7fd07015ebd0 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7fd070163260 tx=0x7fd06002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86dcbafd6f801ba8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:20.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.462+0000 7fd06f7fe640 1 -- 172.21.15.88:0/333433944 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd060037070 con 0x7fd070158d70 2024-09-16T01:45:20.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.462+0000 7fd06f7fe640 1 -- 172.21.15.88:0/333433944 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd060032440 con 0x7fd070158d70 2024-09-16T01:45:20.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.462+0000 7fd06f7fe640 1 -- 172.21.15.88:0/333433944 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd060032810 con 0x7fd070158d70 2024-09-16T01:45:20.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.462+0000 7fd075d9f640 1 -- 172.21.15.88:0/333433944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd070158d70 msgr2=0x7fd07015ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.462+0000 7fd075d9f640 1 --2- 172.21.15.88:0/333433944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd070158d70 0x7fd07015ebd0 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7fd070163260 tx=0x7fd06002a1e0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.462+0000 7fd075d9f640 1 -- 172.21.15.88:0/333433944 shutdown_connections 2024-09-16T01:45:20.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.462+0000 7fd075d9f640 1 --2- 172.21.15.88:0/333433944 >> v2:172.21.15.154:3300/0 conn(0x7fd0700734a0 0x7fd0701621c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.462+0000 7fd075d9f640 1 --2- 172.21.15.88:0/333433944 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0700b5620 0x7fd070072f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.462+0000 7fd075d9f640 1 --2- 172.21.15.88:0/333433944 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd070158d70 0x7fd07015ebd0 unknown :-1 s=CLOSED pgs=2474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.462+0000 7fd075d9f640 1 -- 172.21.15.88:0/333433944 >> 172.21.15.88:0/333433944 conn(0x7fd070062630 msgr2=0x7fd070062a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:20.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.462+0000 7fd075d9f640 1 -- 172.21.15.88:0/333433944 shutdown_connections 2024-09-16T01:45:20.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.462+0000 7fd075d9f640 1 -- 172.21.15.88:0/333433944 wait complete. 2024-09-16T01:45:20.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.463+0000 7fd075d9f640 1 Processor -- start 2024-09-16T01:45:20.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.463+0000 7fd075d9f640 1 -- start start 2024-09-16T01:45:20.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.464+0000 7fd075d9f640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0700734a0 0x7fd0701f4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.464+0000 7fd075d9f640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0700b5620 0x7fd0701f53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.464+0000 7fd075d9f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd070158d70 0x7fd0700cc780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.464+0000 7fd075d9f640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fd0700b5be0 con 0x7fd0700734a0 2024-09-16T01:45:20.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.464+0000 7fd075d9f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd0700b5a60 con 0x7fd070158d70 2024-09-16T01:45:20.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.464+0000 7fd075d9f640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd070162700 con 0x7fd0700b5620 2024-09-16T01:45:20.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.464+0000 7fd074d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0700734a0 0x7fd0701f4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.464+0000 7fd074d9d640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0700734a0 0x7fd0701f4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47624/0 (socket says 172.21.15.88:47624) 2024-09-16T01:45:20.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.464+0000 7fd074d9d640 1 -- 172.21.15.88:0/597751757 learned_addr learned my addr 172.21.15.88:0/597751757 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:20.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.464+0000 7fd06ffff640 1 --2- 172.21.15.88:0/597751757 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0700b5620 0x7fd0701f53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.464+0000 7fd07559e640 1 --2- 172.21.15.88:0/597751757 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd070158d70 0x7fd0700cc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.465+0000 7fd074d9d640 1 -- 172.21.15.88:0/597751757 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0700b5620 msgr2=0x7fd0701f53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.465+0000 7fd074d9d640 1 --2- 172.21.15.88:0/597751757 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd0700b5620 0x7fd0701f53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.465+0000 7fd074d9d640 1 -- 172.21.15.88:0/597751757 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd070158d70 msgr2=0x7fd0700cc780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.465+0000 7fd074d9d640 1 --2- 172.21.15.88:0/597751757 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd070158d70 0x7fd0700cc780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.465+0000 7fd074d9d640 1 -- 172.21.15.88:0/597751757 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0701f4840 con 0x7fd0700734a0 2024-09-16T01:45:20.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.465+0000 7fd074d9d640 1 --2- 172.21.15.88:0/597751757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0700734a0 0x7fd0701f4230 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7fd060000c00 tx=0x7fd0600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:20.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.465+0000 7fd06d7fa640 1 -- 172.21.15.88:0/597751757 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd060041070 con 0x7fd0700734a0 2024-09-16T01:45:20.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.465+0000 7fd06d7fa640 1 -- 172.21.15.88:0/597751757 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd060032e50 con 0x7fd0700734a0 2024-09-16T01:45:20.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.465+0000 7fd06d7fa640 1 -- 172.21.15.88:0/597751757 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd060037040 con 0x7fd0700734a0 2024-09-16T01:45:20.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.465+0000 7fd075d9f640 1 -- 172.21.15.88:0/597751757 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0701f4b20 con 0x7fd0700734a0 2024-09-16T01:45:20.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.465+0000 7fd075d9f640 1 -- 172.21.15.88:0/597751757 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0701f50e0 con 0x7fd0700734a0 2024-09-16T01:45:20.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.466+0000 7fd075d9f640 1 -- 172.21.15.88:0/597751757 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd070072f60 con 0x7fd0700734a0 2024-09-16T01:45:20.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.467+0000 7fd06d7fa640 1 -- 172.21.15.88:0/597751757 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fd060044070 con 0x7fd0700734a0 2024-09-16T01:45:20.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.467+0000 7fd06d7fa640 1 --2- 172.21.15.88:0/597751757 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd05c06c2b0 0x7fd05c06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.467+0000 7fd06d7fa640 1 -- 172.21.15.88:0/597751757 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7fd0600abf10 con 0x7fd0700734a0 2024-09-16T01:45:20.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.467+0000 7fd06ffff640 1 --2- 172.21.15.88:0/597751757 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd05c06c2b0 0x7fd05c06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.467+0000 7fd06d7fa640 1 --2- 172.21.15.88:0/597751757 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x7fd05c072210 0x7fd05c074630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.467+0000 7fd06d7fa640 1 -- 172.21.15.88:0/597751757 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd060032440 con 0x7fd05c072210 2024-09-16T01:45:20.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.467+0000 7fd06d7fa640 1 -- 172.21.15.88:0/597751757 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=142) ==== 24+0+0 (secure 0 0 0) 0x7fd0600ac3a0 con 0x7fd0700734a0 2024-09-16T01:45:20.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.467+0000 7fd07559e640 1 --2- 172.21.15.88:0/597751757 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x7fd05c072210 0x7fd05c074630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.467+0000 7fd06ffff640 1 --2- 172.21.15.88:0/597751757 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd05c06c2b0 0x7fd05c06e770 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fd0701f4770 tx=0x7fd064009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:20.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.468+0000 7fd07559e640 1 --2- 172.21.15.88:0/597751757 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x7fd05c072210 0x7fd05c074630 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:20.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.468+0000 7fd06d7fa640 1 -- 172.21.15.88:0/597751757 <== osd.5 v2:172.21.15.154:6808/280070533 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd060032440 con 0x7fd05c072210 2024-09-16T01:45:20.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.481+0000 7fd075d9f640 1 -- 172.21.15.88:0/597751757 --> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd070074250 con 0x7fd05c072210 2024-09-16T01:45:20.482 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.481+0000 7fd06d7fa640 1 -- 172.21.15.88:0/597751757 <== osd.5 v2:172.21.15.154:6808/280070533 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd070074250 con 0x7fd05c072210 2024-09-16T01:45:20.482 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:45:20.483 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T01:45:20.483 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:45:20.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.481+0000 7fd075d9f640 1 -- 172.21.15.88:0/597751757 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x7fd05c072210 msgr2=0x7fd05c074630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.481+0000 7fd075d9f640 1 --2- 172.21.15.88:0/597751757 >> [v2:172.21.15.154:6808/280070533,v1:172.21.15.154:6809/280070533] conn(0x7fd05c072210 0x7fd05c074630 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.481+0000 7fd075d9f640 1 -- 172.21.15.88:0/597751757 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd05c06c2b0 msgr2=0x7fd05c06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.481+0000 7fd075d9f640 1 --2- 172.21.15.88:0/597751757 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fd05c06c2b0 0x7fd05c06e770 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fd0701f4770 tx=0x7fd064009190 comp rx=0 tx=0).stop 2024-09-16T01:45:20.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.481+0000 7fd075d9f640 1 -- 172.21.15.88:0/597751757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0700734a0 msgr2=0x7fd0701f4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.484 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.481+0000 7fd075d9f640 1 --2- 172.21.15.88:0/597751757 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0700734a0 0x7fd0701f4230 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7fd060000c00 tx=0x7fd0600046c0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.481+0000 7fd07559e640 1 -- 172.21.15.88:0/597751757 reap_dead start 2024-09-16T01:45:20.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.481+0000 7fd075d9f640 1 -- 172.21.15.88:0/597751757 shutdown_connections 2024-09-16T01:45:20.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.482+0000 7fd075d9f640 1 -- 172.21.15.88:0/597751757 >> 172.21.15.88:0/597751757 conn(0x7fd070062630 msgr2=0x7fd070161140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:20.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.482+0000 7fd075d9f640 1 -- 172.21.15.88:0/597751757 shutdown_connections 2024-09-16T01:45:20.485 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.482+0000 7fd075d9f640 1 -- 172.21.15.88:0/597751757 wait complete. 2024-09-16T01:45:20.493 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-16T01:45:20.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.590+0000 7f74f464c640 1 Processor -- start 2024-09-16T01:45:20.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.590+0000 7f74f464c640 1 -- start start 2024-09-16T01:45:20.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.591+0000 7f74f464c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74ec15e460 0x7f74ec0b3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.591+0000 7f74f464c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f74ec15e840 0x7f74ec0b4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.591+0000 7f74f464c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74ec0b0090 con 0x7f74ec0ce680 2024-09-16T01:45:20.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.591+0000 7f74f464c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f74ec0af480 con 0x7f74ec15e840 2024-09-16T01:45:20.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.591+0000 7f74f464c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f74ec0af760 con 0x7f74ec15e460 2024-09-16T01:45:20.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.591+0000 7f74f23e8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74ec15e460 0x7f74ec0b3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.591+0000 7f74f1be7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f74ec15e840 0x7f74ec0b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.591+0000 7f74f1be7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f74ec15e840 0x7f74ec0b4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:43954/0 (socket says 172.21.15.88:43954) 2024-09-16T01:45:20.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.591+0000 7f74f23e8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74ec15e460 0x7f74ec0b3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.88:60780/0 (socket says 172.21.15.88:60780) 2024-09-16T01:45:20.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.592+0000 7f74f1be7640 1 -- 172.21.15.88:0/2229100831 learned_addr learned my addr 172.21.15.88:0/2229100831 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:20.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.592+0000 7f74f2be9640 1 --1- 172.21.15.88:0/2229100831 >> v1:172.21.15.154:6789/0 conn(0x7f74ec0ce680 0x7f74ec071d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.88:37780/0 (socket says 172.21.15.88:37780) 2024-09-16T01:45:20.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.592+0000 7f74f1be7640 1 -- 172.21.15.88:0/2229100831 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74ec15e460 msgr2=0x7f74ec0b3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:20.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.592+0000 7f74f1be7640 1 --2- 172.21.15.88:0/2229100831 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74ec15e460 0x7f74ec0b3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.592+0000 7f74f1be7640 1 -- 172.21.15.88:0/2229100831 >> v1:172.21.15.154:6789/0 conn(0x7f74ec0ce680 legacy=0x7f74ec071d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.592+0000 7f74f1be7640 1 -- 172.21.15.88:0/2229100831 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74ec0717d0 con 0x7f74ec15e840 2024-09-16T01:45:20.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.592+0000 7f74f23e8640 1 --2- 172.21.15.88:0/2229100831 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74ec15e460 0x7f74ec0b3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:45:20.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.593+0000 7f74f1be7640 1 --2- 172.21.15.88:0/2229100831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74ec15e840 0x7f74ec0b4340 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f74dc009870 tx=0x7f74dc02a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=181fdc6f47032121 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:20.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.593+0000 7f74f13e6640 1 -- 172.21.15.88:0/2229100831 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74dc037070 con 0x7f74ec15e840 2024-09-16T01:45:20.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.593+0000 7f74f13e6640 1 -- 172.21.15.88:0/2229100831 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74dc032440 con 0x7f74ec15e840 2024-09-16T01:45:20.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.593+0000 7f74f13e6640 1 -- 172.21.15.88:0/2229100831 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74dc032810 con 0x7f74ec15e840 2024-09-16T01:45:20.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.593+0000 7f74f464c640 1 -- 172.21.15.88:0/2229100831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74ec15e840 msgr2=0x7f74ec0b4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.593+0000 7f74f464c640 1 --2- 172.21.15.88:0/2229100831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74ec15e840 0x7f74ec0b4340 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f74dc009870 tx=0x7f74dc02a270 comp rx=0 tx=0).stop 2024-09-16T01:45:20.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.593+0000 7f74f464c640 1 -- 172.21.15.88:0/2229100831 shutdown_connections 2024-09-16T01:45:20.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.593+0000 7f74f464c640 1 --2- 172.21.15.88:0/2229100831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74ec15e840 0x7f74ec0b4340 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.593+0000 7f74f464c640 1 --2- 172.21.15.88:0/2229100831 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74ec15e460 0x7f74ec0b3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.593+0000 7f74f464c640 1 -- 172.21.15.88:0/2229100831 >> 172.21.15.88:0/2229100831 conn(0x7f74ec06bc10 msgr2=0x7f74ec06bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:20.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.593+0000 7f74f464c640 1 -- 172.21.15.88:0/2229100831 shutdown_connections 2024-09-16T01:45:20.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.593+0000 7f74f464c640 1 -- 172.21.15.88:0/2229100831 wait complete. 2024-09-16T01:45:20.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.595+0000 7f74f464c640 1 Processor -- start 2024-09-16T01:45:20.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.595+0000 7f74f464c640 1 -- start start 2024-09-16T01:45:20.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.595+0000 7f74f464c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74ec0ce680 0x7f74ec155e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.595+0000 7f74f464c640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74ec15e460 0x7f74ec150b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.595+0000 7f74f464c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74ec15e840 0x7f74ec1533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.595+0000 7f74f464c640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f74ec0afe50 con 0x7f74ec15e460 2024-09-16T01:45:20.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.595+0000 7f74f464c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f74ec0afcd0 con 0x7f74ec15e840 2024-09-16T01:45:20.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.595+0000 7f74f464c640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f74ec0b4880 con 0x7f74ec0ce680 2024-09-16T01:45:20.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.595+0000 7f74f1be7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74ec15e460 0x7f74ec150b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.595+0000 7f74f1be7640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74ec15e460 0x7f74ec150b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47626/0 (socket says 172.21.15.88:47626) 2024-09-16T01:45:20.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.595+0000 7f74f1be7640 1 -- 172.21.15.88:0/122230667 learned_addr learned my addr 172.21.15.88:0/122230667 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:20.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.595+0000 7f74f23e8640 1 --2- 172.21.15.88:0/122230667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74ec0ce680 0x7f74ec155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.595+0000 7f74f2be9640 1 --2- 172.21.15.88:0/122230667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74ec15e840 0x7f74ec1533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.596+0000 7f74f1be7640 1 -- 172.21.15.88:0/122230667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74ec0ce680 msgr2=0x7f74ec155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.596+0000 7f74f1be7640 1 --2- 172.21.15.88:0/122230667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74ec0ce680 0x7f74ec155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.596+0000 7f74f1be7640 1 -- 172.21.15.88:0/122230667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74ec15e840 msgr2=0x7f74ec1533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.596+0000 7f74f1be7640 1 --2- 172.21.15.88:0/122230667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74ec15e840 0x7f74ec1533d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.596+0000 7f74f1be7640 1 -- 172.21.15.88:0/122230667 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74ec1563d0 con 0x7f74ec15e460 2024-09-16T01:45:20.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.596+0000 7f74f23e8640 1 --2- 172.21.15.88:0/122230667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74ec0ce680 0x7f74ec155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:45:20.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.596+0000 7f74f1be7640 1 --2- 172.21.15.88:0/122230667 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74ec15e460 0x7f74ec150b90 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7f74dc0090b0 tx=0x7f74dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:20.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.596+0000 7f74e2ffd640 1 -- 172.21.15.88:0/122230667 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74dc041030 con 0x7f74ec15e460 2024-09-16T01:45:20.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.596+0000 7f74e2ffd640 1 -- 172.21.15.88:0/122230667 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74dc030440 con 0x7f74ec15e460 2024-09-16T01:45:20.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.596+0000 7f74e2ffd640 1 -- 172.21.15.88:0/122230667 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74dc037040 con 0x7f74ec15e460 2024-09-16T01:45:20.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.596+0000 7f74f464c640 1 -- 172.21.15.88:0/122230667 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74ec153e20 con 0x7f74ec15e460 2024-09-16T01:45:20.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.596+0000 7f74f464c640 1 -- 172.21.15.88:0/122230667 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74ec1543e0 con 0x7f74ec15e460 2024-09-16T01:45:20.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.597+0000 7f74f464c640 1 -- 172.21.15.88:0/122230667 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74ec0b4340 con 0x7f74ec15e460 2024-09-16T01:45:20.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.598+0000 7f74e2ffd640 1 -- 172.21.15.88:0/122230667 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f74dc044070 con 0x7f74ec15e460 2024-09-16T01:45:20.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.598+0000 7f74e2ffd640 1 --2- 172.21.15.88:0/122230667 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f74d006c2b0 0x7f74d006e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.598+0000 7f74e2ffd640 1 -- 172.21.15.88:0/122230667 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7f74dc0ab650 con 0x7f74ec15e460 2024-09-16T01:45:20.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.598+0000 7f74f23e8640 1 --2- 172.21.15.88:0/122230667 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f74d006c2b0 0x7f74d006e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.599+0000 7f74f23e8640 1 --2- 172.21.15.88:0/122230667 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f74d006c2b0 0x7f74d006e770 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f74e8009560 tx=0x7f74e8009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:20.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.602+0000 7f74e2ffd640 1 -- 172.21.15.88:0/122230667 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f74dc079aa0 con 0x7f74ec15e460 2024-09-16T01:45:20.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.769+0000 7f74f464c640 1 -- 172.21.15.88:0/122230667 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f74ec151ad0 con 0x7f74d006c2b0 2024-09-16T01:45:20.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.770+0000 7f74e2ffd640 1 -- 172.21.15.88:0/122230667 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f74ec151ad0 con 0x7f74d006c2b0 2024-09-16T01:45:20.771 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 5 to deep-scrub 2024-09-16T01:45:20.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.771+0000 7f74f464c640 1 -- 172.21.15.88:0/122230667 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f74d006c2b0 msgr2=0x7f74d006e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.771+0000 7f74f464c640 1 --2- 172.21.15.88:0/122230667 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f74d006c2b0 0x7f74d006e770 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f74e8009560 tx=0x7f74e8009190 comp rx=0 tx=0).stop 2024-09-16T01:45:20.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.771+0000 7f74f464c640 1 -- 172.21.15.88:0/122230667 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74ec15e460 msgr2=0x7f74ec150b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.771+0000 7f74f464c640 1 --2- 172.21.15.88:0/122230667 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74ec15e460 0x7f74ec150b90 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7f74dc0090b0 tx=0x7f74dc0046c0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.771+0000 7f74f464c640 1 -- 172.21.15.88:0/122230667 shutdown_connections 2024-09-16T01:45:20.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.771+0000 7f74f464c640 1 --2- 172.21.15.88:0/122230667 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f74d006c2b0 0x7f74d006e770 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.771+0000 7f74f464c640 1 --2- 172.21.15.88:0/122230667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74ec15e840 0x7f74ec1533d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.771+0000 7f74f464c640 1 --2- 172.21.15.88:0/122230667 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74ec15e460 0x7f74ec150b90 unknown :-1 s=CLOSED pgs=2476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.771+0000 7f74f464c640 1 --2- 172.21.15.88:0/122230667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f74ec0ce680 0x7f74ec155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.772+0000 7f74f464c640 1 -- 172.21.15.88:0/122230667 >> 172.21.15.88:0/122230667 conn(0x7f74ec06bc10 msgr2=0x7f74ec072c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:20.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.772+0000 7f74f464c640 1 -- 172.21.15.88:0/122230667 shutdown_connections 2024-09-16T01:45:20.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.772+0000 7f74f464c640 1 -- 172.21.15.88:0/122230667 wait complete. 2024-09-16T01:45:20.784 INFO:tasks.ceph:Scrubbing osd.6 2024-09-16T01:45:20.784 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-16T01:45:20.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.884+0000 7fdec2f31640 1 Processor -- start 2024-09-16T01:45:20.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.884+0000 7fdec2f31640 1 -- start start 2024-09-16T01:45:20.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.885+0000 7fdec2f31640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdebc06fe20 0x7fdebc073820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.885+0000 7fdec2f31640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdebc06bd40 con 0x7fdebc15e350 2024-09-16T01:45:20.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.885+0000 7fdec2f31640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdebc06c020 con 0x7fdebc15e730 2024-09-16T01:45:20.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.885+0000 7fdec2f31640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fdebc06c300 con 0x7fdebc06fe20 2024-09-16T01:45:20.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.885+0000 7fdebbfff640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fdebc06fe20 0x7fdebc073820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.885+0000 7fdec0ccd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fdebc15e350 0x7fdebc06f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:40732/0 (socket says 172.21.15.88:40732) 2024-09-16T01:45:20.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.885+0000 7fdec0ccd640 1 -- 172.21.15.88:0/4037917304 learned_addr learned my addr 172.21.15.88:0/4037917304 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:20.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.885+0000 7fdebbfff640 1 -- 172.21.15.88:0/4037917304 >> v1:172.21.15.154:6789/0 conn(0x7fdebc15e730 legacy=0x7fdebc0ae990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:20.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.885+0000 7fdebbfff640 1 -- 172.21.15.88:0/4037917304 >> v1:172.21.15.88:6789/0 conn(0x7fdebc15e350 legacy=0x7fdebc06f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.885+0000 7fdebb7fe640 1 -- 172.21.15.88:0/4037917304 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4143233176 0 0) 0x7fdebc06bd40 con 0x7fdebc15e350 2024-09-16T01:45:20.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.885+0000 7fdebbfff640 1 -- 172.21.15.88:0/4037917304 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdebc06c5e0 con 0x7fdebc06fe20 2024-09-16T01:45:20.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.886+0000 7fdebbfff640 1 --2- 172.21.15.88:0/4037917304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdebc06fe20 0x7fdebc073820 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7fdea80097a0 tx=0x7fdea802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d1d35381cae0dd2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:20.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.886+0000 7fdebb7fe640 1 -- 172.21.15.88:0/4037917304 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea8037070 con 0x7fdebc06fe20 2024-09-16T01:45:20.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.886+0000 7fdebb7fe640 1 -- 172.21.15.88:0/4037917304 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdea802ae50 con 0x7fdebc06fe20 2024-09-16T01:45:20.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.886+0000 7fdebb7fe640 1 -- 172.21.15.88:0/4037917304 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea8032660 con 0x7fdebc06fe20 2024-09-16T01:45:20.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.887+0000 7fdec2f31640 1 -- 172.21.15.88:0/4037917304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdebc06fe20 msgr2=0x7fdebc073820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.887+0000 7fdec2f31640 1 --2- 172.21.15.88:0/4037917304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdebc06fe20 0x7fdebc073820 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7fdea80097a0 tx=0x7fdea802a140 comp rx=0 tx=0).stop 2024-09-16T01:45:20.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.887+0000 7fdec2f31640 1 -- 172.21.15.88:0/4037917304 shutdown_connections 2024-09-16T01:45:20.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.887+0000 7fdec2f31640 1 --2- 172.21.15.88:0/4037917304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdebc06fe20 0x7fdebc073820 unknown :-1 s=CLOSED pgs=2478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.887+0000 7fdec2f31640 1 -- 172.21.15.88:0/4037917304 >> 172.21.15.88:0/4037917304 conn(0x7fdebc0b3070 msgr2=0x7fdebc0b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:20.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.887+0000 7fdec2f31640 1 -- 172.21.15.88:0/4037917304 shutdown_connections 2024-09-16T01:45:20.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.887+0000 7fdec2f31640 1 -- 172.21.15.88:0/4037917304 wait complete. 2024-09-16T01:45:20.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.888+0000 7fdec2f31640 1 Processor -- start 2024-09-16T01:45:20.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.888+0000 7fdec2f31640 1 -- start start 2024-09-16T01:45:20.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.889+0000 7fdec2f31640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdebc06fe20 0x7fdebc146e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.889+0000 7fdec2f31640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdebc15e350 0x7fdebc159ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.889+0000 7fdec2f31640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdebc15e730 0x7fdebc152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.889+0000 7fdec2f31640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc073ee0 con 0x7fdebc06fe20 2024-09-16T01:45:20.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.889+0000 7fdec2f31640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc073d60 con 0x7fdebc15e350 2024-09-16T01:45:20.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.889+0000 7fdec2f31640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdebc074060 con 0x7fdebc15e730 2024-09-16T01:45:20.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.889+0000 7fdec0ccd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdebc06fe20 0x7fdebc146e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.889+0000 7fdec0ccd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdebc06fe20 0x7fdebc146e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47644/0 (socket says 172.21.15.88:47644) 2024-09-16T01:45:20.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.889+0000 7fdec0ccd640 1 -- 172.21.15.88:0/374008732 learned_addr learned my addr 172.21.15.88:0/374008732 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:20.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.889+0000 7fdebbfff640 1 --2- 172.21.15.88:0/374008732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdebc15e350 0x7fdebc159ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.889+0000 7fdec14ce640 1 --2- 172.21.15.88:0/374008732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdebc15e730 0x7fdebc152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.889+0000 7fdec0ccd640 1 -- 172.21.15.88:0/374008732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdebc15e730 msgr2=0x7fdebc152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.889+0000 7fdec0ccd640 1 --2- 172.21.15.88:0/374008732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdebc15e730 0x7fdebc152470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.889+0000 7fdec0ccd640 1 -- 172.21.15.88:0/374008732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdebc15e350 msgr2=0x7fdebc159ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.889+0000 7fdec0ccd640 1 --2- 172.21.15.88:0/374008732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdebc15e350 0x7fdebc159ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.889+0000 7fdec0ccd640 1 -- 172.21.15.88:0/374008732 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdebc15a5d0 con 0x7fdebc06fe20 2024-09-16T01:45:20.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.889+0000 7fdebbfff640 1 --2- 172.21.15.88:0/374008732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdebc15e350 0x7fdebc159ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:45:20.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.889+0000 7fdec0ccd640 1 --2- 172.21.15.88:0/374008732 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdebc06fe20 0x7fdebc146e30 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7fdeb000c500 tx=0x7fdeb000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:20.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.889+0000 7fdeb97fa640 1 -- 172.21.15.88:0/374008732 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb0018070 con 0x7fdebc06fe20 2024-09-16T01:45:20.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.889+0000 7fdeb97fa640 1 -- 172.21.15.88:0/374008732 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeb0007580 con 0x7fdebc06fe20 2024-09-16T01:45:20.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.890+0000 7fdec2f31640 1 -- 172.21.15.88:0/374008732 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdebc15f580 con 0x7fdebc06fe20 2024-09-16T01:45:20.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.890+0000 7fdeb97fa640 1 -- 172.21.15.88:0/374008732 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb0004a20 con 0x7fdebc06fe20 2024-09-16T01:45:20.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.890+0000 7fdec2f31640 1 -- 172.21.15.88:0/374008732 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdebc147370 con 0x7fdebc06fe20 2024-09-16T01:45:20.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.890+0000 7fdec2f31640 1 -- 172.21.15.88:0/374008732 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdebc073820 con 0x7fdebc06fe20 2024-09-16T01:45:20.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.891+0000 7fdeb97fa640 1 -- 172.21.15.88:0/374008732 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fdeb001b070 con 0x7fdebc06fe20 2024-09-16T01:45:20.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.891+0000 7fdeb97fa640 1 --2- 172.21.15.88:0/374008732 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdea006c2e0 0x7fdea006e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.892+0000 7fdeb97fa640 1 -- 172.21.15.88:0/374008732 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7fdeb008aa70 con 0x7fdebc06fe20 2024-09-16T01:45:20.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.892+0000 7fdebbfff640 1 --2- 172.21.15.88:0/374008732 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdea006c2e0 0x7fdea006e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.892+0000 7fdeb97fa640 1 --2- 172.21.15.88:0/374008732 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x7fdea0072240 0x7fdea0074660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:20.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.892+0000 7fdeb97fa640 1 -- 172.21.15.88:0/374008732 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdeb0015ec0 con 0x7fdea0072240 2024-09-16T01:45:20.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.892+0000 7fdeb97fa640 1 -- 172.21.15.88:0/374008732 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=142) ==== 24+0+0 (secure 0 0 0) 0x7fdeb008af00 con 0x7fdebc06fe20 2024-09-16T01:45:20.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.892+0000 7fdec14ce640 1 --2- 172.21.15.88:0/374008732 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x7fdea0072240 0x7fdea0074660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:20.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.892+0000 7fdebbfff640 1 --2- 172.21.15.88:0/374008732 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdea006c2e0 0x7fdea006e7a0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fdea8034000 tx=0x7fdea802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:20.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.892+0000 7fdec14ce640 1 --2- 172.21.15.88:0/374008732 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x7fdea0072240 0x7fdea0074660 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:20.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.893+0000 7fdeb97fa640 1 -- 172.21.15.88:0/374008732 <== osd.6 v2:172.21.15.154:6800/189356541 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdeb0015ec0 con 0x7fdea0072240 2024-09-16T01:45:20.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.905+0000 7fdec2f31640 1 -- 172.21.15.88:0/374008732 --> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fdebc15d890 con 0x7fdea0072240 2024-09-16T01:45:20.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.906+0000 7fdeb97fa640 1 -- 172.21.15.88:0/374008732 <== osd.6 v2:172.21.15.154:6800/189356541 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fdebc15d890 con 0x7fdea0072240 2024-09-16T01:45:20.907 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:45:20.907 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T01:45:20.908 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:45:20.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.906+0000 7fdec2f31640 1 -- 172.21.15.88:0/374008732 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x7fdea0072240 msgr2=0x7fdea0074660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.906+0000 7fdec2f31640 1 --2- 172.21.15.88:0/374008732 >> [v2:172.21.15.154:6800/189356541,v1:172.21.15.154:6801/189356541] conn(0x7fdea0072240 0x7fdea0074660 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.906+0000 7fdec2f31640 1 -- 172.21.15.88:0/374008732 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdea006c2e0 msgr2=0x7fdea006e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.906+0000 7fdec2f31640 1 --2- 172.21.15.88:0/374008732 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fdea006c2e0 0x7fdea006e7a0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fdea8034000 tx=0x7fdea802e000 comp rx=0 tx=0).stop 2024-09-16T01:45:20.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.906+0000 7fdec2f31640 1 -- 172.21.15.88:0/374008732 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdebc06fe20 msgr2=0x7fdebc146e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:20.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.906+0000 7fdec2f31640 1 --2- 172.21.15.88:0/374008732 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdebc06fe20 0x7fdebc146e30 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7fdeb000c500 tx=0x7fdeb000c9c0 comp rx=0 tx=0).stop 2024-09-16T01:45:20.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.906+0000 7fdec14ce640 1 -- 172.21.15.88:0/374008732 reap_dead start 2024-09-16T01:45:20.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.906+0000 7fdec2f31640 1 -- 172.21.15.88:0/374008732 shutdown_connections 2024-09-16T01:45:20.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.906+0000 7fdec2f31640 1 -- 172.21.15.88:0/374008732 >> 172.21.15.88:0/374008732 conn(0x7fdebc0b3070 msgr2=0x7fdebc0b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:20.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.906+0000 7fdec2f31640 1 -- 172.21.15.88:0/374008732 shutdown_connections 2024-09-16T01:45:20.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:20.906+0000 7fdec2f31640 1 -- 172.21.15.88:0/374008732 wait complete. 2024-09-16T01:45:20.918 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-16T01:45:21.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.017+0000 7f2ac260e640 1 Processor -- start 2024-09-16T01:45:21.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.017+0000 7f2ac260e640 1 -- start start 2024-09-16T01:45:21.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.017+0000 7f2ac260e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2abc0cf1f0 0x7f2abc0b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.017+0000 7f2ac260e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2abc0b6720 0x7f2abc0d3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.017+0000 7f2ac260e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2abc0cf860 con 0x7f2abc15e7a0 2024-09-16T01:45:21.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.017+0000 7f2ac260e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2abc0d2ca0 con 0x7f2abc0b6720 2024-09-16T01:45:21.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.017+0000 7f2ac260e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2abc0cfb40 con 0x7f2abc0cf1f0 2024-09-16T01:45:21.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.018+0000 7f2ac0bab640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2abc0b6720 0x7f2abc0d3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.018+0000 7f2abb7fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2abc0cf1f0 0x7f2abc0b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.018+0000 7f2ac0bab640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2abc0b6720 0x7f2abc0d3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.88:43962/0 (socket says 172.21.15.88:43962) 2024-09-16T01:45:21.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.018+0000 7f2ac0bab640 1 -- 172.21.15.88:0/3532382213 learned_addr learned my addr 172.21.15.88:0/3532382213 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:21.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.018+0000 7f2ac0bab640 1 -- 172.21.15.88:0/3532382213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2abc0cf1f0 msgr2=0x7f2abc0b61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:21.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.018+0000 7f2ac0bab640 1 --2- 172.21.15.88:0/3532382213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2abc0cf1f0 0x7f2abc0b61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.018+0000 7f2ac0bab640 1 -- 172.21.15.88:0/3532382213 >> v1:172.21.15.154:6789/0 conn(0x7f2abc15e7a0 legacy=0x7f2abc06caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.018+0000 7f2ac0bab640 1 -- 172.21.15.88:0/3532382213 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2abc0cfe20 con 0x7f2abc0b6720 2024-09-16T01:45:21.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.018+0000 7f2abb7fe640 1 --2- 172.21.15.88:0/3532382213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2abc0cf1f0 0x7f2abc0b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:45:21.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.019+0000 7f2ac0bab640 1 --2- 172.21.15.88:0/3532382213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2abc0b6720 0x7f2abc0d3410 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4009e50 tx=0x7f2ab402a6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d210de415aab38e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:21.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.019+0000 7f2abaffd640 1 -- 172.21.15.88:0/3532382213 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab4036070 con 0x7f2abc0b6720 2024-09-16T01:45:21.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.019+0000 7f2abaffd640 1 -- 172.21.15.88:0/3532382213 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ab4031600 con 0x7f2abc0b6720 2024-09-16T01:45:21.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.019+0000 7f2abaffd640 1 -- 172.21.15.88:0/3532382213 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab40319d0 con 0x7f2abc0b6720 2024-09-16T01:45:21.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.019+0000 7f2ac260e640 1 -- 172.21.15.88:0/3532382213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2abc0b6720 msgr2=0x7f2abc0d3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.019+0000 7f2ac260e640 1 --2- 172.21.15.88:0/3532382213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2abc0b6720 0x7f2abc0d3410 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4009e50 tx=0x7f2ab402a6a0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.019+0000 7f2ac260e640 1 -- 172.21.15.88:0/3532382213 shutdown_connections 2024-09-16T01:45:21.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.019+0000 7f2ac260e640 1 --2- 172.21.15.88:0/3532382213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2abc0b6720 0x7f2abc0d3410 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.019+0000 7f2ac260e640 1 --2- 172.21.15.88:0/3532382213 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2abc0cf1f0 0x7f2abc0b61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.019+0000 7f2ac260e640 1 -- 172.21.15.88:0/3532382213 >> 172.21.15.88:0/3532382213 conn(0x7f2abc0d11e0 msgr2=0x7f2abc0d15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:21.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.019+0000 7f2ac260e640 1 -- 172.21.15.88:0/3532382213 shutdown_connections 2024-09-16T01:45:21.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.019+0000 7f2ac260e640 1 -- 172.21.15.88:0/3532382213 wait complete. 2024-09-16T01:45:21.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.021+0000 7f2ac260e640 1 Processor -- start 2024-09-16T01:45:21.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.021+0000 7f2ac260e640 1 -- start start 2024-09-16T01:45:21.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.021+0000 7f2ac260e640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2abc0b6720 0x7f2abc14fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.021+0000 7f2ac260e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2abc0cf1f0 0x7f2abc15ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.021+0000 7f2ac260e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2abc15e7a0 0x7f2abc0ce130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.021+0000 7f2ac260e640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc15eb80 con 0x7f2abc0b6720 2024-09-16T01:45:21.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.021+0000 7f2ac260e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc0b7a00 con 0x7f2abc0cf1f0 2024-09-16T01:45:21.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.021+0000 7f2ac260e640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2abc15ed00 con 0x7f2abc15e7a0 2024-09-16T01:45:21.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.022+0000 7f2abbfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2abc0b6720 0x7f2abc14fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.022+0000 7f2abbfff640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2abc0b6720 0x7f2abc14fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47648/0 (socket says 172.21.15.88:47648) 2024-09-16T01:45:21.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.022+0000 7f2abbfff640 1 -- 172.21.15.88:0/4038626978 learned_addr learned my addr 172.21.15.88:0/4038626978 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:21.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.022+0000 7f2abb7fe640 1 --2- 172.21.15.88:0/4038626978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2abc0cf1f0 0x7f2abc15ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.022+0000 7f2ac0bab640 1 --2- 172.21.15.88:0/4038626978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2abc15e7a0 0x7f2abc0ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.022+0000 7f2abbfff640 1 -- 172.21.15.88:0/4038626978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2abc15e7a0 msgr2=0x7f2abc0ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.022+0000 7f2abbfff640 1 --2- 172.21.15.88:0/4038626978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2abc15e7a0 0x7f2abc0ce130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.022+0000 7f2abbfff640 1 -- 172.21.15.88:0/4038626978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2abc0cf1f0 msgr2=0x7f2abc15ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.022+0000 7f2abbfff640 1 --2- 172.21.15.88:0/4038626978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2abc0cf1f0 0x7f2abc15ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.022+0000 7f2abbfff640 1 -- 172.21.15.88:0/4038626978 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2abc0cc6b0 con 0x7f2abc0b6720 2024-09-16T01:45:21.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.022+0000 7f2ac0bab640 1 --2- 172.21.15.88:0/4038626978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2abc15e7a0 0x7f2abc0ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T01:45:21.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.022+0000 7f2abbfff640 1 --2- 172.21.15.88:0/4038626978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2abc0b6720 0x7f2abc14fcb0 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7f2aac00c3c0 tx=0x7f2aac00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:21.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.022+0000 7f2ab8ff9640 1 -- 172.21.15.88:0/4038626978 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aac0052d0 con 0x7f2abc0b6720 2024-09-16T01:45:21.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.022+0000 7f2ab8ff9640 1 -- 172.21.15.88:0/4038626978 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aac005490 con 0x7f2abc0b6720 2024-09-16T01:45:21.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.022+0000 7f2ab8ff9640 1 -- 172.21.15.88:0/4038626978 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aac007260 con 0x7f2abc0b6720 2024-09-16T01:45:21.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.022+0000 7f2ac260e640 1 -- 172.21.15.88:0/4038626978 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2abc0cc990 con 0x7f2abc0b6720 2024-09-16T01:45:21.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.022+0000 7f2ac260e640 1 -- 172.21.15.88:0/4038626978 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2abc0d3dd0 con 0x7f2abc0b6720 2024-09-16T01:45:21.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.023+0000 7f2ac260e640 1 -- 172.21.15.88:0/4038626978 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2abc0d2a80 con 0x7f2abc0b6720 2024-09-16T01:45:21.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.024+0000 7f2ab8ff9640 1 -- 172.21.15.88:0/4038626978 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f2aac007400 con 0x7f2abc0b6720 2024-09-16T01:45:21.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.024+0000 7f2ab8ff9640 1 --2- 172.21.15.88:0/4038626978 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2aa806c2b0 0x7f2aa806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.024+0000 7f2abb7fe640 1 --2- 172.21.15.88:0/4038626978 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2aa806c2b0 0x7f2aa806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.025+0000 7f2ab8ff9640 1 -- 172.21.15.88:0/4038626978 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7f2aac08a5c0 con 0x7f2abc0b6720 2024-09-16T01:45:21.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.025+0000 7f2abb7fe640 1 --2- 172.21.15.88:0/4038626978 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2aa806c2b0 0x7f2aa806e770 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f2abc0d3be0 tx=0x7f2ab0009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:21.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.029+0000 7f2ab8ff9640 1 -- 172.21.15.88:0/4038626978 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f2aac058980 con 0x7f2abc0b6720 2024-09-16T01:45:21.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.194+0000 7f2ac260e640 1 -- 172.21.15.88:0/4038626978 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f2abc0b4f30 con 0x7f2aa806c2b0 2024-09-16T01:45:21.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.194+0000 7f2ab8ff9640 1 -- 172.21.15.88:0/4038626978 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f2abc0b4f30 con 0x7f2aa806c2b0 2024-09-16T01:45:21.196 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 6 to deep-scrub 2024-09-16T01:45:21.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.196+0000 7f2ac260e640 1 -- 172.21.15.88:0/4038626978 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2aa806c2b0 msgr2=0x7f2aa806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.196+0000 7f2ac260e640 1 --2- 172.21.15.88:0/4038626978 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2aa806c2b0 0x7f2aa806e770 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f2abc0d3be0 tx=0x7f2ab0009190 comp rx=0 tx=0).stop 2024-09-16T01:45:21.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.196+0000 7f2ac260e640 1 -- 172.21.15.88:0/4038626978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2abc0b6720 msgr2=0x7f2abc14fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.196+0000 7f2ac260e640 1 --2- 172.21.15.88:0/4038626978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2abc0b6720 0x7f2abc14fcb0 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7f2aac00c3c0 tx=0x7f2aac00c880 comp rx=0 tx=0).stop 2024-09-16T01:45:21.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.196+0000 7f2ac260e640 1 -- 172.21.15.88:0/4038626978 shutdown_connections 2024-09-16T01:45:21.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.196+0000 7f2ac260e640 1 --2- 172.21.15.88:0/4038626978 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f2aa806c2b0 0x7f2aa806e770 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.198 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.196+0000 7f2ac260e640 1 --2- 172.21.15.88:0/4038626978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2abc15e7a0 0x7f2abc0ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.196+0000 7f2ac260e640 1 --2- 172.21.15.88:0/4038626978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2abc0cf1f0 0x7f2abc15ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.196+0000 7f2ac260e640 1 --2- 172.21.15.88:0/4038626978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2abc0b6720 0x7f2abc14fcb0 unknown :-1 s=CLOSED pgs=2480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.196+0000 7f2ac260e640 1 -- 172.21.15.88:0/4038626978 >> 172.21.15.88:0/4038626978 conn(0x7f2abc0d11e0 msgr2=0x7f2abc06c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:21.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.196+0000 7f2ac260e640 1 -- 172.21.15.88:0/4038626978 shutdown_connections 2024-09-16T01:45:21.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.196+0000 7f2ac260e640 1 -- 172.21.15.88:0/4038626978 wait complete. 2024-09-16T01:45:21.209 INFO:tasks.ceph:Scrubbing osd.7 2024-09-16T01:45:21.209 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-16T01:45:21.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.309+0000 7f04f54e6640 1 Processor -- start 2024-09-16T01:45:21.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.309+0000 7f04f54e6640 1 -- start start 2024-09-16T01:45:21.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.310+0000 7f04f54e6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f04f00b3920 0x7f04f0162b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.310+0000 7f04f54e6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04f015ed70 con 0x7f04f015e990 2024-09-16T01:45:21.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.310+0000 7f04f54e6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04f00d1fa0 con 0x7f04f00d29a0 2024-09-16T01:45:21.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.310+0000 7f04f54e6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f04f00cff20 con 0x7f04f00b3920 2024-09-16T01:45:21.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.310+0000 7f04eeffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f04f015e990 0x7f04f0162450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:40740/0 (socket says 172.21.15.88:40740) 2024-09-16T01:45:21.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.310+0000 7f04eeffd640 1 -- 172.21.15.88:0/139737182 learned_addr learned my addr 172.21.15.88:0/139737182 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:21.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.310+0000 7f04ee7fc640 1 --2- 172.21.15.88:0/139737182 >> v2:172.21.15.154:3300/0 conn(0x7f04f00b3920 0x7f04f0162b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.311+0000 7f04edffb640 1 -- 172.21.15.88:0/139737182 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 505985627 0 0) 0x7f04e0001490 con 0x7f04f015e990 2024-09-16T01:45:21.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.311+0000 7f04edffb640 1 -- 172.21.15.88:0/139737182 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04d8003520 con 0x7f04f015e990 2024-09-16T01:45:21.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.311+0000 7f04ee7fc640 1 -- 172.21.15.88:0/139737182 >> v1:172.21.15.154:6789/0 conn(0x7f04f00d29a0 legacy=0x7f04f00d2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.311+0000 7f04ee7fc640 1 -- 172.21.15.88:0/139737182 >> v1:172.21.15.88:6789/0 conn(0x7f04f015e990 legacy=0x7f04f0162450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.311+0000 7f04ee7fc640 1 -- 172.21.15.88:0/139737182 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04f00cf960 con 0x7f04f00b3920 2024-09-16T01:45:21.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.311+0000 7f04ee7fc640 1 --2- 172.21.15.88:0/139737182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04f00b3920 0x7f04f0162b60 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f04e40097a0 tx=0x7f04e402a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13cb6097cb3809e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:21.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.312+0000 7f04edffb640 1 -- 172.21.15.88:0/139737182 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e4037070 con 0x7f04f00b3920 2024-09-16T01:45:21.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.312+0000 7f04edffb640 1 -- 172.21.15.88:0/139737182 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04e4032440 con 0x7f04f00b3920 2024-09-16T01:45:21.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.312+0000 7f04edffb640 1 -- 172.21.15.88:0/139737182 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e40327f0 con 0x7f04f00b3920 2024-09-16T01:45:21.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.312+0000 7f04f54e6640 1 -- 172.21.15.88:0/139737182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04f00b3920 msgr2=0x7f04f0162b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.312+0000 7f04f54e6640 1 --2- 172.21.15.88:0/139737182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04f00b3920 0x7f04f0162b60 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f04e40097a0 tx=0x7f04e402a1a0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.312+0000 7f04f54e6640 1 -- 172.21.15.88:0/139737182 shutdown_connections 2024-09-16T01:45:21.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.312+0000 7f04f54e6640 1 --2- 172.21.15.88:0/139737182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04f00b3920 0x7f04f0162b60 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.312+0000 7f04f54e6640 1 -- 172.21.15.88:0/139737182 >> 172.21.15.88:0/139737182 conn(0x7f04f00d1200 msgr2=0x7f04f00d1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:21.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.312+0000 7f04f54e6640 1 -- 172.21.15.88:0/139737182 shutdown_connections 2024-09-16T01:45:21.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.312+0000 7f04f54e6640 1 -- 172.21.15.88:0/139737182 wait complete. 2024-09-16T01:45:21.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.314+0000 7f04f54e6640 1 Processor -- start 2024-09-16T01:45:21.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.314+0000 7f04f54e6640 1 -- start start 2024-09-16T01:45:21.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.314+0000 7f04f54e6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04f00b3920 0x7f04f00b5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.314+0000 7f04f54e6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04f00d29a0 0x7f04f015dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.314+0000 7f04f54e6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04f015e990 0x7f04f00b6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.314+0000 7f04f54e6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f04f01630a0 con 0x7f04f00b3920 2024-09-16T01:45:21.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.314+0000 7f04f54e6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f04f00b3d60 con 0x7f04f00d29a0 2024-09-16T01:45:21.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.314+0000 7f04f54e6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f04f0163220 con 0x7f04f015e990 2024-09-16T01:45:21.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.314+0000 7f04eeffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04f00b3920 0x7f04f00b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.314+0000 7f04eeffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04f00b3920 0x7f04f00b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47654/0 (socket says 172.21.15.88:47654) 2024-09-16T01:45:21.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.314+0000 7f04eeffd640 1 -- 172.21.15.88:0/1347090935 learned_addr learned my addr 172.21.15.88:0/1347090935 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:21.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.314+0000 7f04ef7fe640 1 --2- 172.21.15.88:0/1347090935 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04f015e990 0x7f04f00b6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.315+0000 7f04ee7fc640 1 --2- 172.21.15.88:0/1347090935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04f00d29a0 0x7f04f015dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.315+0000 7f04eeffd640 1 -- 172.21.15.88:0/1347090935 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04f015e990 msgr2=0x7f04f00b6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.315+0000 7f04eeffd640 1 --2- 172.21.15.88:0/1347090935 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04f015e990 0x7f04f00b6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.315+0000 7f04eeffd640 1 -- 172.21.15.88:0/1347090935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04f00d29a0 msgr2=0x7f04f015dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.315+0000 7f04eeffd640 1 --2- 172.21.15.88:0/1347090935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04f00d29a0 0x7f04f015dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.315+0000 7f04eeffd640 1 -- 172.21.15.88:0/1347090935 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04f00ccf40 con 0x7f04f00b3920 2024-09-16T01:45:21.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.315+0000 7f04eeffd640 1 --2- 172.21.15.88:0/1347090935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04f00b3920 0x7f04f00b5640 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7f04e000cbc0 tx=0x7f04e000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:21.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.315+0000 7f04d77fe640 1 -- 172.21.15.88:0/1347090935 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e0017070 con 0x7f04f00b3920 2024-09-16T01:45:21.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.315+0000 7f04d77fe640 1 -- 172.21.15.88:0/1347090935 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04e000ad40 con 0x7f04f00b3920 2024-09-16T01:45:21.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.315+0000 7f04f54e6640 1 -- 172.21.15.88:0/1347090935 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04f00b4b60 con 0x7f04f00b3920 2024-09-16T01:45:21.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.315+0000 7f04d77fe640 1 -- 172.21.15.88:0/1347090935 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e0012600 con 0x7f04f00b3920 2024-09-16T01:45:21.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.315+0000 7f04f54e6640 1 -- 172.21.15.88:0/1347090935 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04f0152e20 con 0x7f04f00b3920 2024-09-16T01:45:21.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.316+0000 7f04f54e6640 1 -- 172.21.15.88:0/1347090935 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f04f00b5220 con 0x7f04f00b3920 2024-09-16T01:45:21.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.317+0000 7f04d77fe640 1 -- 172.21.15.88:0/1347090935 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f04e001a070 con 0x7f04f00b3920 2024-09-16T01:45:21.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.317+0000 7f04d77fe640 1 --2- 172.21.15.88:0/1347090935 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f04dc06c2b0 0x7f04dc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.317+0000 7f04d77fe640 1 -- 172.21.15.88:0/1347090935 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7f04e008ae80 con 0x7f04f00b3920 2024-09-16T01:45:21.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.317+0000 7f04ee7fc640 1 --2- 172.21.15.88:0/1347090935 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f04dc06c2b0 0x7f04dc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.317+0000 7f04d77fe640 1 --2- 172.21.15.88:0/1347090935 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x7f04dc072210 0x7f04dc074630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.317+0000 7f04d77fe640 1 -- 172.21.15.88:0/1347090935 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f04dc074ce0 con 0x7f04dc072210 2024-09-16T01:45:21.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.317+0000 7f04d77fe640 1 -- 172.21.15.88:0/1347090935 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=142) ==== 24+0+0 (secure 0 0 0) 0x7f04e008d580 con 0x7f04f00b3920 2024-09-16T01:45:21.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.317+0000 7f04ef7fe640 1 --2- 172.21.15.88:0/1347090935 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x7f04dc072210 0x7f04dc074630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.318+0000 7f04ee7fc640 1 --2- 172.21.15.88:0/1347090935 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f04dc06c2b0 0x7f04dc06e770 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f04e4034000 tx=0x7f04e402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:21.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.318+0000 7f04ef7fe640 1 --2- 172.21.15.88:0/1347090935 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x7f04dc072210 0x7f04dc074630 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:21.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.319+0000 7f04d77fe640 1 -- 172.21.15.88:0/1347090935 <== osd.7 v2:172.21.15.154:6824/2200266987 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f04dc074ce0 con 0x7f04dc072210 2024-09-16T01:45:21.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.331+0000 7f04f54e6640 1 -- 172.21.15.88:0/1347090935 --> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f04f00d0a40 con 0x7f04dc072210 2024-09-16T01:45:21.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.331+0000 7f04d77fe640 1 -- 172.21.15.88:0/1347090935 <== osd.7 v2:172.21.15.154:6824/2200266987 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f04f00d0a40 con 0x7f04dc072210 2024-09-16T01:45:21.333 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-09-16T01:45:21.333 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T01:45:21.333 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-09-16T01:45:21.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.332+0000 7f04f54e6640 1 -- 172.21.15.88:0/1347090935 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x7f04dc072210 msgr2=0x7f04dc074630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.332+0000 7f04f54e6640 1 --2- 172.21.15.88:0/1347090935 >> [v2:172.21.15.154:6824/2200266987,v1:172.21.15.154:6825/2200266987] conn(0x7f04dc072210 0x7f04dc074630 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.332+0000 7f04f54e6640 1 -- 172.21.15.88:0/1347090935 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f04dc06c2b0 msgr2=0x7f04dc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.332+0000 7f04f54e6640 1 --2- 172.21.15.88:0/1347090935 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f04dc06c2b0 0x7f04dc06e770 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f04e4034000 tx=0x7f04e402e000 comp rx=0 tx=0).stop 2024-09-16T01:45:21.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.332+0000 7f04f54e6640 1 -- 172.21.15.88:0/1347090935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04f00b3920 msgr2=0x7f04f00b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.332+0000 7f04f54e6640 1 --2- 172.21.15.88:0/1347090935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04f00b3920 0x7f04f00b5640 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7f04e000cbc0 tx=0x7f04e000a430 comp rx=0 tx=0).stop 2024-09-16T01:45:21.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.332+0000 7f04ef7fe640 1 -- 172.21.15.88:0/1347090935 reap_dead start 2024-09-16T01:45:21.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.332+0000 7f04f54e6640 1 -- 172.21.15.88:0/1347090935 shutdown_connections 2024-09-16T01:45:21.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.332+0000 7f04f54e6640 1 -- 172.21.15.88:0/1347090935 >> 172.21.15.88:0/1347090935 conn(0x7f04f00d1200 msgr2=0x7f04f01e8110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:21.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.332+0000 7f04f54e6640 1 -- 172.21.15.88:0/1347090935 shutdown_connections 2024-09-16T01:45:21.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.332+0000 7f04f54e6640 1 -- 172.21.15.88:0/1347090935 wait complete. 2024-09-16T01:45:21.344 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-16T01:45:21.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.441+0000 7f91647f8640 1 Processor -- start 2024-09-16T01:45:21.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.441+0000 7f91647f8640 1 -- start start 2024-09-16T01:45:21.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.442+0000 7f91647f8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f915c0d3060 0x7f915c0b5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.442+0000 7f91647f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f915c0b5a80 0x7f915c073160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.442+0000 7f91647f8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f915c0d0370 con 0x7f915c15c550 2024-09-16T01:45:21.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.442+0000 7f91647f8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f915c071530 con 0x7f915c0d3060 2024-09-16T01:45:21.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.442+0000 7f91647f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f915c15c930 con 0x7f915c0b5a80 2024-09-16T01:45:21.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.443+0000 7f9162594640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f915c15c550 0x7f915c0b7270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:40750/0 (socket says 172.21.15.88:40750) 2024-09-16T01:45:21.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.443+0000 7f9162594640 1 -- 172.21.15.88:0/3348637267 learned_addr learned my addr 172.21.15.88:0/3348637267 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:21.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.443+0000 7f9161d93640 1 --2- 172.21.15.88:0/3348637267 >> v2:172.21.15.154:3300/0 conn(0x7f915c0d3060 0x7f915c0b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.443+0000 7f9162d95640 1 --2- 172.21.15.88:0/3348637267 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f915c0b5a80 0x7f915c073160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.443+0000 7f9161592640 1 -- 172.21.15.88:0/3348637267 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2287374230 0 0) 0x7f914c001490 con 0x7f915c15c550 2024-09-16T01:45:21.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.443+0000 7f9161592640 1 -- 172.21.15.88:0/3348637267 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9144003520 con 0x7f915c15c550 2024-09-16T01:45:21.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.443+0000 7f9161592640 1 -- 172.21.15.88:0/3348637267 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4180819598 0 0) 0x7f9144003520 con 0x7f915c15c550 2024-09-16T01:45:21.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.443+0000 7f9161592640 1 -- 172.21.15.88:0/3348637267 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f915c0b5a80 msgr2=0x7f915c073160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:21.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.443+0000 7f9161592640 1 --2- 172.21.15.88:0/3348637267 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f915c0b5a80 0x7f915c073160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.443+0000 7f9161592640 1 -- 172.21.15.88:0/3348637267 >> v2:172.21.15.154:3300/0 conn(0x7f915c0d3060 msgr2=0x7f915c0b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:21.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.443+0000 7f9161592640 1 --2- 172.21.15.88:0/3348637267 >> v2:172.21.15.154:3300/0 conn(0x7f915c0d3060 0x7f915c0b5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.443+0000 7f9161592640 1 -- 172.21.15.88:0/3348637267 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f915c070920 con 0x7f915c15c550 2024-09-16T01:45:21.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.443+0000 7f9161592640 1 -- 172.21.15.88:0/3348637267 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f914c001bc0 con 0x7f915c15c550 2024-09-16T01:45:21.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.443+0000 7f9161592640 1 -- 172.21.15.88:0/3348637267 >> v1:172.21.15.88:6789/0 conn(0x7f915c15c550 legacy=0x7f915c0b7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.443+0000 7f9161592640 1 --2- 172.21.15.88:0/3348637267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91440039b0 0x7f9144003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.443+0000 7f9161592640 1 --2- 172.21.15.88:0/3348637267 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f915c0b5a80 0x7f9144004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.443+0000 7f9161592640 1 --2- 172.21.15.88:0/3348637267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9144004d40 0x7f91440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.443+0000 7f9161592640 1 -- 172.21.15.88:0/3348637267 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f915c071530 con 0x7f9144004d40 2024-09-16T01:45:21.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.443+0000 7f9161592640 1 -- 172.21.15.88:0/3348637267 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f915c15c930 con 0x7f91440039b0 2024-09-16T01:45:21.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.443+0000 7f9161592640 1 -- 172.21.15.88:0/3348637267 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f914400b9c0 con 0x7f915c0b5a80 2024-09-16T01:45:21.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.443+0000 7f9162d95640 1 --2- 172.21.15.88:0/3348637267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9144004d40 0x7f91440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.443+0000 7f9162594640 1 --2- 172.21.15.88:0/3348637267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91440039b0 0x7f9144003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.443+0000 7f9161d93640 1 --2- 172.21.15.88:0/3348637267 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f915c0b5a80 0x7f9144004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.444+0000 7f9162d95640 1 -- 172.21.15.88:0/3348637267 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f915c0b5a80 msgr2=0x7f9144004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:21.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.444+0000 7f9162d95640 1 --2- 172.21.15.88:0/3348637267 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f915c0b5a80 0x7f9144004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.444+0000 7f9162d95640 1 -- 172.21.15.88:0/3348637267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91440039b0 msgr2=0x7f9144003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:21.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.444+0000 7f9162d95640 1 --2- 172.21.15.88:0/3348637267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91440039b0 0x7f9144003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.444+0000 7f9162d95640 1 -- 172.21.15.88:0/3348637267 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9144003520 con 0x7f9144004d40 2024-09-16T01:45:21.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.444+0000 7f9162d95640 1 --2- 172.21.15.88:0/3348637267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9144004d40 0x7f91440051c0 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7f9154009ac0 tx=0x7f9144005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa9e0707af499ade server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:21.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.444+0000 7f9161592640 1 -- 172.21.15.88:0/3348637267 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9154036070 con 0x7f9144004d40 2024-09-16T01:45:21.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.444+0000 7f9161592640 1 -- 172.21.15.88:0/3348637267 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9154031400 con 0x7f9144004d40 2024-09-16T01:45:21.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.444+0000 7f9161592640 1 -- 172.21.15.88:0/3348637267 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9154009690 con 0x7f9144004d40 2024-09-16T01:45:21.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.445+0000 7f91647f8640 1 -- 172.21.15.88:0/3348637267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9144004d40 msgr2=0x7f91440051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.445+0000 7f91647f8640 1 --2- 172.21.15.88:0/3348637267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9144004d40 0x7f91440051c0 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7f9154009ac0 tx=0x7f9144005830 comp rx=0 tx=0).stop 2024-09-16T01:45:21.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.445+0000 7f9162d95640 1 -- 172.21.15.88:0/3348637267 reap_dead start 2024-09-16T01:45:21.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.445+0000 7f91647f8640 1 -- 172.21.15.88:0/3348637267 shutdown_connections 2024-09-16T01:45:21.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.445+0000 7f91647f8640 1 -- 172.21.15.88:0/3348637267 >> 172.21.15.88:0/3348637267 conn(0x7f915c0d24d0 msgr2=0x7f915c0d28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:21.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.445+0000 7f91647f8640 1 -- 172.21.15.88:0/3348637267 shutdown_connections 2024-09-16T01:45:21.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.445+0000 7f91647f8640 1 -- 172.21.15.88:0/3348637267 wait complete. 2024-09-16T01:45:21.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.446+0000 7f91647f8640 1 Processor -- start 2024-09-16T01:45:21.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.446+0000 7f91647f8640 1 -- start start 2024-09-16T01:45:21.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.447+0000 7f91647f8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9144004d40 0x7f915c0ce030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.447+0000 7f91647f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f915c14ac80 0x7f915c0cbe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.447+0000 7f91647f8640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f915c15fed0 0x7f915c160350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.447+0000 7f91647f8640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f915c0711c0 con 0x7f9144004d40 2024-09-16T01:45:21.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.447+0000 7f91647f8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f915c070ee0 con 0x7f915c14ac80 2024-09-16T01:45:21.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.447+0000 7f91647f8640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f915c073160 con 0x7f915c15fed0 2024-09-16T01:45:21.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.447+0000 7f9162594640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9144004d40 0x7f915c0ce030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.447+0000 7f9162594640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9144004d40 0x7f915c0ce030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47666/0 (socket says 172.21.15.88:47666) 2024-09-16T01:45:21.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.447+0000 7f9162594640 1 -- 172.21.15.88:0/1576622293 learned_addr learned my addr 172.21.15.88:0/1576622293 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:21.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.447+0000 7f9162d95640 1 --2- 172.21.15.88:0/1576622293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f915c15fed0 0x7f915c160350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.447+0000 7f9161d93640 1 --2- 172.21.15.88:0/1576622293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f915c14ac80 0x7f915c0cbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.447+0000 7f9162594640 1 -- 172.21.15.88:0/1576622293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f915c15fed0 msgr2=0x7f915c160350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.447+0000 7f9162594640 1 --2- 172.21.15.88:0/1576622293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f915c15fed0 0x7f915c160350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.447+0000 7f9162594640 1 -- 172.21.15.88:0/1576622293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f915c14ac80 msgr2=0x7f915c0cbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.447+0000 7f9162594640 1 --2- 172.21.15.88:0/1576622293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f915c14ac80 0x7f915c0cbe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.447+0000 7f9162594640 1 -- 172.21.15.88:0/1576622293 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f915c14b450 con 0x7f9144004d40 2024-09-16T01:45:21.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.448+0000 7f9162594640 1 --2- 172.21.15.88:0/1576622293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9144004d40 0x7f915c0ce030 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7f914c00ac50 tx=0x7f914c007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:21.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.448+0000 7f915affd640 1 -- 172.21.15.88:0/1576622293 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914c017020 con 0x7f9144004d40 2024-09-16T01:45:21.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.448+0000 7f915affd640 1 -- 172.21.15.88:0/1576622293 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f914c0133f0 con 0x7f9144004d40 2024-09-16T01:45:21.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.448+0000 7f91647f8640 1 -- 172.21.15.88:0/1576622293 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f915c14b730 con 0x7f9144004d40 2024-09-16T01:45:21.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.448+0000 7f915affd640 1 -- 172.21.15.88:0/1576622293 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914c0137e0 con 0x7f9144004d40 2024-09-16T01:45:21.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.448+0000 7f91647f8640 1 -- 172.21.15.88:0/1576622293 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f915c0cc3a0 con 0x7f9144004d40 2024-09-16T01:45:21.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.449+0000 7f91647f8640 1 -- 172.21.15.88:0/1576622293 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f915c148c50 con 0x7f9144004d40 2024-09-16T01:45:21.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.449+0000 7f915affd640 1 -- 172.21.15.88:0/1576622293 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f914c01a020 con 0x7f9144004d40 2024-09-16T01:45:21.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.450+0000 7f915affd640 1 --2- 172.21.15.88:0/1576622293 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f914806c2e0 0x7f914806e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.450+0000 7f915affd640 1 -- 172.21.15.88:0/1576622293 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7f914c08bda0 con 0x7f9144004d40 2024-09-16T01:45:21.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.450+0000 7f9161d93640 1 --2- 172.21.15.88:0/1576622293 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f914806c2e0 0x7f914806e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.450+0000 7f9161d93640 1 --2- 172.21.15.88:0/1576622293 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f914806c2e0 0x7f914806e7a0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f915c1f7b40 tx=0x7f9150008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:21.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.454+0000 7f915affd640 1 -- 172.21.15.88:0/1576622293 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f914c021020 con 0x7f9144004d40 2024-09-16T01:45:21.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.620+0000 7f91647f8640 1 -- 172.21.15.88:0/1576622293 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f915c14bc50 con 0x7f914806c2e0 2024-09-16T01:45:21.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.620+0000 7f915affd640 1 -- 172.21.15.88:0/1576622293 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f9150014b50 con 0x7f914806c2e0 2024-09-16T01:45:21.622 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 7 to deep-scrub 2024-09-16T01:45:21.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.622+0000 7f91647f8640 1 -- 172.21.15.88:0/1576622293 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f914806c2e0 msgr2=0x7f914806e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.622+0000 7f91647f8640 1 --2- 172.21.15.88:0/1576622293 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f914806c2e0 0x7f914806e7a0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f915c1f7b40 tx=0x7f9150008040 comp rx=0 tx=0).stop 2024-09-16T01:45:21.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.622+0000 7f91647f8640 1 -- 172.21.15.88:0/1576622293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9144004d40 msgr2=0x7f915c0ce030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.622+0000 7f91647f8640 1 --2- 172.21.15.88:0/1576622293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9144004d40 0x7f915c0ce030 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7f914c00ac50 tx=0x7f914c007430 comp rx=0 tx=0).stop 2024-09-16T01:45:21.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.622+0000 7f91647f8640 1 -- 172.21.15.88:0/1576622293 shutdown_connections 2024-09-16T01:45:21.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.622+0000 7f91647f8640 1 --2- 172.21.15.88:0/1576622293 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f914806c2e0 0x7f914806e7a0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.622+0000 7f91647f8640 1 --2- 172.21.15.88:0/1576622293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f915c15fed0 0x7f915c160350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.622+0000 7f91647f8640 1 --2- 172.21.15.88:0/1576622293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f915c14ac80 0x7f915c0cbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.622+0000 7f91647f8640 1 --2- 172.21.15.88:0/1576622293 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9144004d40 0x7f915c0ce030 unknown :-1 s=CLOSED pgs=2485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.622+0000 7f91647f8640 1 -- 172.21.15.88:0/1576622293 >> 172.21.15.88:0/1576622293 conn(0x7f915c0d24d0 msgr2=0x7f915c0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:21.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.622+0000 7f91647f8640 1 -- 172.21.15.88:0/1576622293 shutdown_connections 2024-09-16T01:45:21.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.622+0000 7f91647f8640 1 -- 172.21.15.88:0/1576622293 wait complete. 2024-09-16T01:45:21.634 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:45:21.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.731+0000 7f3bcdff6640 1 Processor -- start 2024-09-16T01:45:21.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.731+0000 7f3bcdff6640 1 -- start start 2024-09-16T01:45:21.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.731+0000 7f3bcdff6640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3bc815c170 0x7f3bc815c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.731+0000 7f3bcdff6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bc80b4cf0 con 0x7f3bc80cf6c0 2024-09-16T01:45:21.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.731+0000 7f3bcdff6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bc80b4fd0 con 0x7f3bc80b64e0 2024-09-16T01:45:21.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.731+0000 7f3bcdff6640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f3bc815ca90 con 0x7f3bc815c170 2024-09-16T01:45:21.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.732+0000 7f3bc77fe640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3bc815c170 0x7f3bc815c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.732+0000 7f3bc6ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3bc80cf6c0 0x7f3bc80b5dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:40766/0 (socket says 172.21.15.88:40766) 2024-09-16T01:45:21.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.732+0000 7f3bc6ffd640 1 -- 172.21.15.88:0/1345363310 learned_addr learned my addr 172.21.15.88:0/1345363310 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:21.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.732+0000 7f3bc77fe640 1 -- 172.21.15.88:0/1345363310 >> v1:172.21.15.154:6789/0 conn(0x7f3bc80b64e0 legacy=0x7f3bc80b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.732+0000 7f3bc77fe640 1 -- 172.21.15.88:0/1345363310 >> v1:172.21.15.88:6789/0 conn(0x7f3bc80cf6c0 legacy=0x7f3bc80b5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.732+0000 7f3bc77fe640 1 -- 172.21.15.88:0/1345363310 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc80b52b0 con 0x7f3bc815c170 2024-09-16T01:45:21.735 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.732+0000 7f3bc67fc640 1 -- 172.21.15.88:0/1345363310 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3037250562 0 0) 0x7f3bc80b4cf0 con 0x7f3bc80cf6c0 2024-09-16T01:45:21.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.733+0000 7f3bc77fe640 1 --2- 172.21.15.88:0/1345363310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc815c170 0x7f3bc815c550 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7f3bb80097d0 tx=0x7f3bb802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5104273f0b49eee9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:21.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.733+0000 7f3bc67fc640 1 -- 172.21.15.88:0/1345363310 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8037070 con 0x7f3bc815c170 2024-09-16T01:45:21.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.733+0000 7f3bc67fc640 1 -- 172.21.15.88:0/1345363310 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bb8032440 con 0x7f3bc815c170 2024-09-16T01:45:21.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.733+0000 7f3bc67fc640 1 -- 172.21.15.88:0/1345363310 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8032810 con 0x7f3bc815c170 2024-09-16T01:45:21.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.733+0000 7f3bcdff6640 1 -- 172.21.15.88:0/1345363310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc815c170 msgr2=0x7f3bc815c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.733+0000 7f3bcdff6640 1 --2- 172.21.15.88:0/1345363310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc815c170 0x7f3bc815c550 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7f3bb80097d0 tx=0x7f3bb802a1c0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.733+0000 7f3bcdff6640 1 -- 172.21.15.88:0/1345363310 shutdown_connections 2024-09-16T01:45:21.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.733+0000 7f3bcdff6640 1 --2- 172.21.15.88:0/1345363310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc815c170 0x7f3bc815c550 unknown :-1 s=CLOSED pgs=2487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.733+0000 7f3bcdff6640 1 -- 172.21.15.88:0/1345363310 >> 172.21.15.88:0/1345363310 conn(0x7f3bc806bc10 msgr2=0x7f3bc806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:21.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.733+0000 7f3bcdff6640 1 -- 172.21.15.88:0/1345363310 shutdown_connections 2024-09-16T01:45:21.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.733+0000 7f3bcdff6640 1 -- 172.21.15.88:0/1345363310 wait complete. 2024-09-16T01:45:21.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.734+0000 7f3bcdff6640 1 Processor -- start 2024-09-16T01:45:21.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.735+0000 7f3bcdff6640 1 -- start start 2024-09-16T01:45:21.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.735+0000 7f3bcdff6640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc80b64e0 0x7f3bc80cc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.735+0000 7f3bcdff6640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc80cf6c0 0x7f3bc80cae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.735+0000 7f3bcdff6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bc815c170 0x7f3bc80c8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.735+0000 7f3bcdff6640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc80b5b50 con 0x7f3bc80cf6c0 2024-09-16T01:45:21.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.735+0000 7f3bcdff6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc80b58b0 con 0x7f3bc815c170 2024-09-16T01:45:21.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.735+0000 7f3bcdff6640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3bc80cfb30 con 0x7f3bc80b64e0 2024-09-16T01:45:21.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.735+0000 7f3bc6ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc80cf6c0 0x7f3bc80cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.735+0000 7f3bc6ffd640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc80cf6c0 0x7f3bc80cae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:47690/0 (socket says 172.21.15.88:47690) 2024-09-16T01:45:21.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.735+0000 7f3bc6ffd640 1 -- 172.21.15.88:0/1596550708 learned_addr learned my addr 172.21.15.88:0/1596550708 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:21.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.736+0000 7f3bc77fe640 1 --2- 172.21.15.88:0/1596550708 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc80b64e0 0x7f3bc80cc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.736+0000 7f3bc7fff640 1 --2- 172.21.15.88:0/1596550708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bc815c170 0x7f3bc80c8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.736+0000 7f3bc6ffd640 1 -- 172.21.15.88:0/1596550708 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc80b64e0 msgr2=0x7f3bc80cc510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.736+0000 7f3bc6ffd640 1 --2- 172.21.15.88:0/1596550708 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc80b64e0 0x7f3bc80cc510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.736+0000 7f3bc6ffd640 1 -- 172.21.15.88:0/1596550708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bc815c170 msgr2=0x7f3bc80c8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.736+0000 7f3bc6ffd640 1 --2- 172.21.15.88:0/1596550708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bc815c170 0x7f3bc80c8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.736+0000 7f3bc6ffd640 1 -- 172.21.15.88:0/1596550708 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc80cb470 con 0x7f3bc80cf6c0 2024-09-16T01:45:21.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.736+0000 7f3bc6ffd640 1 --2- 172.21.15.88:0/1596550708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc80cf6c0 0x7f3bc80cae20 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc00c4d0 tx=0x7f3bbc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:21.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.736+0000 7f3baffff640 1 -- 172.21.15.88:0/1596550708 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc018070 con 0x7f3bc80cf6c0 2024-09-16T01:45:21.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.736+0000 7f3baffff640 1 -- 172.21.15.88:0/1596550708 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bbc0074e0 con 0x7f3bc80cf6c0 2024-09-16T01:45:21.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.736+0000 7f3bcdff6640 1 -- 172.21.15.88:0/1596550708 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bc80cb750 con 0x7f3bc80cf6c0 2024-09-16T01:45:21.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.736+0000 7f3baffff640 1 -- 172.21.15.88:0/1596550708 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc004a20 con 0x7f3bc80cf6c0 2024-09-16T01:45:21.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.736+0000 7f3bcdff6640 1 -- 172.21.15.88:0/1596550708 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bc80cbd10 con 0x7f3bc80cf6c0 2024-09-16T01:45:21.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.737+0000 7f3bcdff6640 1 -- 172.21.15.88:0/1596550708 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bc80cff20 con 0x7f3bc80cf6c0 2024-09-16T01:45:21.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.738+0000 7f3baffff640 1 -- 172.21.15.88:0/1596550708 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3bbc01b070 con 0x7f3bc80cf6c0 2024-09-16T01:45:21.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.738+0000 7f3baffff640 1 --2- 172.21.15.88:0/1596550708 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3bb406c2b0 0x7f3bb406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:21.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.738+0000 7f3baffff640 1 -- 172.21.15.88:0/1596550708 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7f3bbc08a3f0 con 0x7f3bc80cf6c0 2024-09-16T01:45:21.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.738+0000 7f3bc77fe640 1 --2- 172.21.15.88:0/1596550708 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3bb406c2b0 0x7f3bb406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:21.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.738+0000 7f3bc77fe640 1 --2- 172.21.15.88:0/1596550708 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3bb406c2b0 0x7f3bb406e770 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8034000 tx=0x7f3bb802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:21.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.742+0000 7f3baffff640 1 -- 172.21.15.88:0/1596550708 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3bbc058840 con 0x7f3bc80cf6c0 2024-09-16T01:45:21.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.905+0000 7f3bcdff6640 1 -- 172.21.15.88:0/1596550708 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3bc80c92f0 con 0x7f3bb406c2b0 2024-09-16T01:45:21.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.905+0000 7f3baffff640 1 -- 172.21.15.88:0/1596550708 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13414 (secure 0 0 0) 0x7f3bc80c92f0 con 0x7f3bb406c2b0 2024-09-16T01:45:21.907 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:45:21.907 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:45:21.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.907+0000 7f3bcdff6640 1 -- 172.21.15.88:0/1596550708 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3bb406c2b0 msgr2=0x7f3bb406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.907+0000 7f3bcdff6640 1 --2- 172.21.15.88:0/1596550708 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3bb406c2b0 0x7f3bb406e770 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8034000 tx=0x7f3bb802e000 comp rx=0 tx=0).stop 2024-09-16T01:45:21.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.907+0000 7f3bcdff6640 1 -- 172.21.15.88:0/1596550708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc80cf6c0 msgr2=0x7f3bc80cae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:21.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.907+0000 7f3bcdff6640 1 --2- 172.21.15.88:0/1596550708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc80cf6c0 0x7f3bc80cae20 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc00c4d0 tx=0x7f3bbc00c990 comp rx=0 tx=0).stop 2024-09-16T01:45:21.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.907+0000 7f3bcdff6640 1 -- 172.21.15.88:0/1596550708 shutdown_connections 2024-09-16T01:45:21.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.907+0000 7f3bcdff6640 1 --2- 172.21.15.88:0/1596550708 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3bb406c2b0 0x7f3bb406e770 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.907+0000 7f3bcdff6640 1 --2- 172.21.15.88:0/1596550708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bc815c170 0x7f3bc80c8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.907+0000 7f3bcdff6640 1 --2- 172.21.15.88:0/1596550708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bc80cf6c0 0x7f3bc80cae20 unknown :-1 s=CLOSED pgs=2488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.907+0000 7f3bcdff6640 1 --2- 172.21.15.88:0/1596550708 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc80b64e0 0x7f3bc80cc510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:21.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.907+0000 7f3bcdff6640 1 -- 172.21.15.88:0/1596550708 >> 172.21.15.88:0/1596550708 conn(0x7f3bc806bc10 msgr2=0x7f3bc80727a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:21.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.908+0000 7f3bcdff6640 1 -- 172.21.15.88:0/1596550708 shutdown_connections 2024-09-16T01:45:21.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:21.908+0000 7f3bcdff6640 1 -- 172.21.15.88:0/1596550708 wait complete. 2024-09-16T01:45:21.920 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-09-16T01:45:20.661080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":298232,"kb_used_data":15432,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754676488,"statfs":{"total":773094113280,"available":772788723712,"internally_reserved":0,"allocated":15802368,"data_stored":8209040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":289394226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-35701,"num_objects":-28,"num_object_clones":0,"num_object_copies":-56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-28,"num_whiteouts":0,"num_read":-84,"num_read_kb":-86,"num_write":-95,"num_write_kb":-70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001596"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":400,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T01:45:13.941508+0000","last_change":"2024-09-16T01:34:12.315118+0000","last_active":"2024-09-16T01:45:13.941508+0000","last_peered":"2024-09-16T01:45:13.941508+0000","last_clean":"2024-09-16T01:45:13.941508+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:45:13.941508+0000","last_undegraded":"2024-09-16T01:45:13.941508+0000","last_fullsized":"2024-09-16T01:45:13.941508+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T01:34:10.944097+0000","last_clean_scrub_stamp":"2024-09-16T01:34:10.944097+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:39.989971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509578,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37132,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334708,"statfs":{"total":96636764160,"available":96598740992,"internally_reserved":0,"allocated":1826816,"data_stored":878538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542282,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37716,"kb_used_data":2364,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334124,"statfs":{"total":96636764160,"available":96598142976,"internally_reserved":0,"allocated":2420736,"data_stored":1468906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542282,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37132,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334708,"statfs":{"total":96636764160,"available":96598740992,"internally_reserved":0,"allocated":1826816,"data_stored":878538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542282,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32972,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1826816,"data_stored":878538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542281,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37136,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334704,"statfs":{"total":96636764160,"available":96598736896,"internally_reserved":0,"allocated":1826816,"data_stored":878538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542282,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37136,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334704,"statfs":{"total":96636764160,"available":96598736896,"internally_reserved":0,"allocated":1826816,"data_stored":878538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542281,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37132,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334708,"statfs":{"total":96636764160,"available":96598740992,"internally_reserved":0,"allocated":1826816,"data_stored":878538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542281,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41876,"kb_used_data":2364,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329964,"statfs":{"total":96636764160,"available":96593883136,"internally_reserved":0,"allocated":2420736,"data_stored":1468906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:45:21.922 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T01:34:10.944097+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=34, tm_sec=10, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=1, tm_min=45, tm_sec=17, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T01:45:21.923 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T01:45:41.924 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T01:45:42.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.035+0000 7f3ac2823640 1 Processor -- start 2024-09-16T01:45:42.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.035+0000 7f3ac2823640 1 -- start start 2024-09-16T01:45:42.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.035+0000 7f3ac2823640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3abc15e660 0x7f3abc0cf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:42.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.036+0000 7f3ac2823640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3abc0d1820 0x7f3abc0b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:42.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.036+0000 7f3ac2823640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3abc15ea40 con 0x7f3abc0d3310 2024-09-16T01:45:42.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.036+0000 7f3ac2823640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7f3abc0d3870 con 0x7f3abc0d1820 2024-09-16T01:45:42.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.036+0000 7f3ac2823640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3abc0d3b50 con 0x7f3abc15e660 2024-09-16T01:45:42.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.036+0000 7f3ac0dc0640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7f3abc0d1820 0x7f3abc0b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:42.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.036+0000 7f3abb7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3abc0d3310 0x7f3abc0d1110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35396/0 (socket says 172.21.15.88:35396) 2024-09-16T01:45:42.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.036+0000 7f3abb7fe640 1 -- 172.21.15.88:0/2994945538 learned_addr learned my addr 172.21.15.88:0/2994945538 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:42.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.036+0000 7f3abbfff640 1 --2- 172.21.15.88:0/2994945538 >> v2:172.21.15.154:3300/0 conn(0x7f3abc15e660 0x7f3abc0cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:42.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.036+0000 7f3ac0dc0640 1 -- 172.21.15.88:0/2994945538 >> v2:172.21.15.154:3300/0 conn(0x7f3abc15e660 msgr2=0x7f3abc0cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:42.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.036+0000 7f3ac0dc0640 1 --2- 172.21.15.88:0/2994945538 >> v2:172.21.15.154:3300/0 conn(0x7f3abc15e660 0x7f3abc0cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.036+0000 7f3ac0dc0640 1 -- 172.21.15.88:0/2994945538 >> v1:172.21.15.88:6789/0 conn(0x7f3abc0d3310 legacy=0x7f3abc0d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:42.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.036+0000 7f3ac0dc0640 1 -- 172.21.15.88:0/2994945538 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3abc0d3e30 con 0x7f3abc0d1820 2024-09-16T01:45:42.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.036+0000 7f3abaffd640 1 -- 172.21.15.88:0/2994945538 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2223289911 0 0) 0x7f3abc15ea40 con 0x7f3abc0d3310 2024-09-16T01:45:42.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.037+0000 7f3ac0dc0640 1 --2- 172.21.15.88:0/2994945538 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3abc0d1820 0x7f3abc0b7b70 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7f3abc0d10e0 tx=0x7f3ab402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6396221e9d07b86 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:42.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.037+0000 7f3abaffd640 1 -- 172.21.15.88:0/2994945538 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab4007740 con 0x7f3abc0d1820 2024-09-16T01:45:42.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.037+0000 7f3abaffd640 1 -- 172.21.15.88:0/2994945538 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ab40078e0 con 0x7f3abc0d1820 2024-09-16T01:45:42.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.037+0000 7f3abaffd640 1 -- 172.21.15.88:0/2994945538 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab4007cb0 con 0x7f3abc0d1820 2024-09-16T01:45:42.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.037+0000 7f3ac2823640 1 -- 172.21.15.88:0/2994945538 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3abc0d1820 msgr2=0x7f3abc0b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.037+0000 7f3ac2823640 1 --2- 172.21.15.88:0/2994945538 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3abc0d1820 0x7f3abc0b7b70 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7f3abc0d10e0 tx=0x7f3ab402a680 comp rx=0 tx=0).stop 2024-09-16T01:45:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.038+0000 7f3ac2823640 1 -- 172.21.15.88:0/2994945538 shutdown_connections 2024-09-16T01:45:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.038+0000 7f3ac2823640 1 --2- 172.21.15.88:0/2994945538 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3abc0d1820 0x7f3abc0b7b70 unknown :-1 s=CLOSED pgs=2490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.038+0000 7f3ac2823640 1 --2- 172.21.15.88:0/2994945538 >> v2:172.21.15.154:3300/0 conn(0x7f3abc15e660 0x7f3abc0cf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.038+0000 7f3ac2823640 1 -- 172.21.15.88:0/2994945538 >> 172.21.15.88:0/2994945538 conn(0x7f3abc0b6410 msgr2=0x7f3abc0b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.038+0000 7f3ac2823640 1 -- 172.21.15.88:0/2994945538 shutdown_connections 2024-09-16T01:45:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.038+0000 7f3ac2823640 1 -- 172.21.15.88:0/2994945538 wait complete. 2024-09-16T01:45:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.039+0000 7f3ac2823640 1 Processor -- start 2024-09-16T01:45:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.039+0000 7f3ac2823640 1 -- start start 2024-09-16T01:45:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.040+0000 7f3ac2823640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3abc0d1820 0x7f3abc152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.040+0000 7f3ac2823640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3abc0d3310 0x7f3abc0cae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.040+0000 7f3ac2823640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3abc15e660 0x7f3abc1521c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.040+0000 7f3ac2823640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f3abc0cfc70 con 0x7f3abc0d3310 2024-09-16T01:45:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.040+0000 7f3ac2823640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3abc0d43f0 con 0x7f3abc0d1820 2024-09-16T01:45:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.040+0000 7f3ac2823640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3abc0b80b0 con 0x7f3abc15e660 2024-09-16T01:45:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.040+0000 7f3abb7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3abc0d3310 0x7f3abc0cae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.040+0000 7f3abb7fe640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3abc0d3310 0x7f3abc0cae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38782/0 (socket says 172.21.15.88:38782) 2024-09-16T01:45:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.040+0000 7f3abb7fe640 1 -- 172.21.15.88:0/2522532741 learned_addr learned my addr 172.21.15.88:0/2522532741 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.040+0000 7f3ac0dc0640 1 --2- 172.21.15.88:0/2522532741 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3abc15e660 0x7f3abc1521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.040+0000 7f3abbfff640 1 --2- 172.21.15.88:0/2522532741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3abc0d1820 0x7f3abc152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.040+0000 7f3abb7fe640 1 -- 172.21.15.88:0/2522532741 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3abc15e660 msgr2=0x7f3abc1521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.040+0000 7f3abb7fe640 1 --2- 172.21.15.88:0/2522532741 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3abc15e660 0x7f3abc1521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.040+0000 7f3abb7fe640 1 -- 172.21.15.88:0/2522532741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3abc0d1820 msgr2=0x7f3abc152960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.040+0000 7f3abb7fe640 1 --2- 172.21.15.88:0/2522532741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3abc0d1820 0x7f3abc152960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.040+0000 7f3abb7fe640 1 -- 172.21.15.88:0/2522532741 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3abc1600a0 con 0x7f3abc0d3310 2024-09-16T01:45:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.040+0000 7f3abb7fe640 1 --2- 172.21.15.88:0/2522532741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3abc0d3310 0x7f3abc0cae70 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7f3ab000c500 tx=0x7f3ab000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.040+0000 7f3ab8ff9640 1 -- 172.21.15.88:0/2522532741 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0018070 con 0x7f3abc0d3310 2024-09-16T01:45:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.040+0000 7f3ab8ff9640 1 -- 172.21.15.88:0/2522532741 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ab0007580 con 0x7f3abc0d3310 2024-09-16T01:45:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.040+0000 7f3ab8ff9640 1 -- 172.21.15.88:0/2522532741 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0004a20 con 0x7f3abc0d3310 2024-09-16T01:45:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.041+0000 7f3ac2823640 1 -- 172.21.15.88:0/2522532741 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3abc160380 con 0x7f3abc0d3310 2024-09-16T01:45:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.041+0000 7f3ac2823640 1 -- 172.21.15.88:0/2522532741 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3abc160940 con 0x7f3abc0d3310 2024-09-16T01:45:42.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.042+0000 7f3ac2823640 1 -- 172.21.15.88:0/2522532741 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3abc0b7b70 con 0x7f3abc0d3310 2024-09-16T01:45:42.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.042+0000 7f3ab8ff9640 1 -- 172.21.15.88:0/2522532741 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f3ab001b070 con 0x7f3abc0d3310 2024-09-16T01:45:42.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.042+0000 7f3ab8ff9640 1 --2- 172.21.15.88:0/2522532741 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3aa806c2b0 0x7f3aa806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:42.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.042+0000 7f3ab8ff9640 1 -- 172.21.15.88:0/2522532741 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7f3ab008a470 con 0x7f3abc0d3310 2024-09-16T01:45:42.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.043+0000 7f3abbfff640 1 --2- 172.21.15.88:0/2522532741 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3aa806c2b0 0x7f3aa806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:42.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.043+0000 7f3abbfff640 1 --2- 172.21.15.88:0/2522532741 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3aa806c2b0 0x7f3aa806e770 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f3aac006fd0 tx=0x7f3aac008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:42.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.047+0000 7f3ab8ff9640 1 -- 172.21.15.88:0/2522532741 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f3ab0058f10 con 0x7f3abc0d3310 2024-09-16T01:45:42.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.208+0000 7f3ac2823640 1 -- 172.21.15.88:0/2522532741 --> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3abc152700 con 0x7f3aa806c2b0 2024-09-16T01:45:42.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.209+0000 7f3ab8ff9640 1 -- 172.21.15.88:0/2522532741 <== mgr.4107 v2:172.21.15.88:6832/3938986063 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13379 (secure 0 0 0) 0x7f3abc152700 con 0x7f3aa806c2b0 2024-09-16T01:45:42.211 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:45:42.212 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-09-16T01:45:42.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.211+0000 7f3ac2823640 1 -- 172.21.15.88:0/2522532741 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3aa806c2b0 msgr2=0x7f3aa806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:42.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.211+0000 7f3ac2823640 1 --2- 172.21.15.88:0/2522532741 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3aa806c2b0 0x7f3aa806e770 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f3aac006fd0 tx=0x7f3aac008040 comp rx=0 tx=0).stop 2024-09-16T01:45:42.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.211+0000 7f3ac2823640 1 -- 172.21.15.88:0/2522532741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3abc0d3310 msgr2=0x7f3abc0cae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:42.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.211+0000 7f3ac2823640 1 --2- 172.21.15.88:0/2522532741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3abc0d3310 0x7f3abc0cae70 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7f3ab000c500 tx=0x7f3ab000c9c0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.211+0000 7f3ac2823640 1 -- 172.21.15.88:0/2522532741 shutdown_connections 2024-09-16T01:45:42.213 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.211+0000 7f3ac2823640 1 --2- 172.21.15.88:0/2522532741 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f3aa806c2b0 0x7f3aa806e770 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.211+0000 7f3ac2823640 1 --2- 172.21.15.88:0/2522532741 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3abc15e660 0x7f3abc1521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.211+0000 7f3ac2823640 1 --2- 172.21.15.88:0/2522532741 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3abc0d3310 0x7f3abc0cae70 unknown :-1 s=CLOSED pgs=2491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.211+0000 7f3ac2823640 1 --2- 172.21.15.88:0/2522532741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3abc0d1820 0x7f3abc152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.211+0000 7f3ac2823640 1 -- 172.21.15.88:0/2522532741 >> 172.21.15.88:0/2522532741 conn(0x7f3abc0b6410 msgr2=0x7f3abc15b990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:42.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.211+0000 7f3ac2823640 1 -- 172.21.15.88:0/2522532741 shutdown_connections 2024-09-16T01:45:42.214 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.211+0000 7f3ac2823640 1 -- 172.21.15.88:0/2522532741 wait complete. 2024-09-16T01:45:42.223 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":480,"stamp":"2024-09-16T01:45:40.664901+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":298216,"kb_used_data":15432,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754676504,"statfs":{"total":773094113280,"available":772788740096,"internally_reserved":0,"allocated":15802368,"data_stored":8209040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":289394226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002309"},"pg_stats":[{"pgid":"1.0","version":"142'119","reported_seq":411,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T01:45:22.131455+0000","last_change":"2024-09-16T01:45:22.131486+0000","last_active":"2024-09-16T01:45:22.131455+0000","last_peered":"2024-09-16T01:45:22.131455+0000","last_clean":"2024-09-16T01:45:22.131455+0000","last_became_active":"2024-09-16T01:34:12.314074+0000","last_became_peered":"2024-09-16T01:34:12.314074+0000","last_unstale":"2024-09-16T01:45:22.131455+0000","last_undegraded":"2024-09-16T01:45:22.131455+0000","last_fullsized":"2024-09-16T01:45:22.131455+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'119","last_scrub_stamp":"2024-09-16T01:45:22.131368+0000","last_deep_scrub":"142'119","last_deep_scrub_stamp":"2024-09-16T01:45:22.131368+0000","last_clean_scrub_stamp":"2024-09-16T01:45:22.131368+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:37:23.052133+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509582,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37132,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334708,"statfs":{"total":96636764160,"available":96598740992,"internally_reserved":0,"allocated":1826816,"data_stored":878538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542286,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37708,"kb_used_data":2364,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334132,"statfs":{"total":96636764160,"available":96598151168,"internally_reserved":0,"allocated":2420736,"data_stored":1468906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542286,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37132,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334708,"statfs":{"total":96636764160,"available":96598740992,"internally_reserved":0,"allocated":1826816,"data_stored":878538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542286,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32972,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1826816,"data_stored":878538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542285,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37136,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334704,"statfs":{"total":96636764160,"available":96598736896,"internally_reserved":0,"allocated":1826816,"data_stored":878538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542286,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37136,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334704,"statfs":{"total":96636764160,"available":96598736896,"internally_reserved":0,"allocated":1826816,"data_stored":878538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542285,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37132,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334708,"statfs":{"total":96636764160,"available":96598740992,"internally_reserved":0,"allocated":1826816,"data_stored":878538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542285,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41868,"kb_used_data":2364,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329972,"statfs":{"total":96636764160,"available":96593891328,"internally_reserved":0,"allocated":2420736,"data_stored":1468906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T01:45:42.224 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T01:45:42.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.321+0000 7f4ff3f31640 1 Processor -- start 2024-09-16T01:45:42.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.321+0000 7f4ff3f31640 1 -- start start 2024-09-16T01:45:42.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.322+0000 7f4ff3f31640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4fec158a00 0x7f4fec071f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:42.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.322+0000 7f4ff3f31640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fec158de0 0x7f4fec06eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:42.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.322+0000 7f4ff3f31640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fec0728f0 con 0x7f4fec072470 2024-09-16T01:45:42.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.322+0000 7f4ff3f31640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f4fec072bd0 con 0x7f4fec158a00 2024-09-16T01:45:42.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.322+0000 7f4ff3f31640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4fec072eb0 con 0x7f4fec158de0 2024-09-16T01:45:42.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.322+0000 7f4ff14cc640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4fec072470 0x7f4fec0b4e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:35408/0 (socket says 172.21.15.88:35408) 2024-09-16T01:45:42.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.322+0000 7f4ff14cc640 1 -- 172.21.15.88:0/956617503 learned_addr learned my addr 172.21.15.88:0/956617503 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-09-16T01:45:42.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.322+0000 7f4ff0ccb640 1 -- 172.21.15.88:0/956617503 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1922939563 0 0) 0x7f4fec0728f0 con 0x7f4fec072470 2024-09-16T01:45:42.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.322+0000 7f4ff0ccb640 1 -- 172.21.15.88:0/956617503 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4fd4003520 con 0x7f4fec072470 2024-09-16T01:45:42.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.322+0000 7f4ff1ccd640 1 --2- 172.21.15.88:0/956617503 >> v2:172.21.15.154:3300/0 conn(0x7f4fec158a00 0x7f4fec071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:42.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.322+0000 7f4ff24ce640 1 --2- 172.21.15.88:0/956617503 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fec158de0 0x7f4fec06eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:42.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.323+0000 7f4ff0ccb640 1 -- 172.21.15.88:0/956617503 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 782730024 0 0) 0x7f4fd4003520 con 0x7f4fec072470 2024-09-16T01:45:42.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.323+0000 7f4ff0ccb640 1 -- 172.21.15.88:0/956617503 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fec158de0 msgr2=0x7f4fec06eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:42.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.323+0000 7f4ff0ccb640 1 --2- 172.21.15.88:0/956617503 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fec158de0 0x7f4fec06eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.323+0000 7f4ff0ccb640 1 -- 172.21.15.88:0/956617503 >> v2:172.21.15.154:3300/0 conn(0x7f4fec158a00 msgr2=0x7f4fec071f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:42.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.323+0000 7f4ff0ccb640 1 --2- 172.21.15.88:0/956617503 >> v2:172.21.15.154:3300/0 conn(0x7f4fec158a00 0x7f4fec071f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.323+0000 7f4ff0ccb640 1 -- 172.21.15.88:0/956617503 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fec073190 con 0x7f4fec072470 2024-09-16T01:45:42.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.323+0000 7f4ff0ccb640 1 -- 172.21.15.88:0/956617503 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1780662380 0 0) 0x7f4fe0001670 con 0x7f4fec072470 2024-09-16T01:45:42.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.323+0000 7f4ff0ccb640 1 -- 172.21.15.88:0/956617503 >> v1:172.21.15.88:6789/0 conn(0x7f4fec072470 legacy=0x7f4fec0b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:42.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.323+0000 7f4ff0ccb640 1 --2- 172.21.15.88:0/956617503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fd40039b0 0x7f4fd4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:42.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.323+0000 7f4ff0ccb640 1 --2- 172.21.15.88:0/956617503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fd4004800 0x7f4fd4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:42.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.323+0000 7f4ff0ccb640 1 --2- 172.21.15.88:0/956617503 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fd40051c0 0x7f4fd400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:42.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.323+0000 7f4ff0ccb640 1 -- 172.21.15.88:0/956617503 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4fec072bd0 con 0x7f4fd40039b0 2024-09-16T01:45:42.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.323+0000 7f4ff0ccb640 1 -- 172.21.15.88:0/956617503 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4fec072eb0 con 0x7f4fd4004800 2024-09-16T01:45:42.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.323+0000 7f4ff0ccb640 1 -- 172.21.15.88:0/956617503 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4fd400bf30 con 0x7f4fd40051c0 2024-09-16T01:45:42.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.323+0000 7f4ff24ce640 1 --2- 172.21.15.88:0/956617503 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fec158de0 0x7f4fec06eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:45:42.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.323+0000 7f4ff1ccd640 1 --2- 172.21.15.88:0/956617503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fd40039b0 0x7f4fd4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:42.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.323+0000 7f4ff14cc640 1 --2- 172.21.15.88:0/956617503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fd4004800 0x7f4fd4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:42.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.323+0000 7f4ff24ce640 1 --2- 172.21.15.88:0/956617503 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fd40051c0 0x7f4fd400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:42.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.323+0000 7f4ff1ccd640 1 -- 172.21.15.88:0/956617503 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fd40051c0 msgr2=0x7f4fd400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:42.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.323+0000 7f4ff1ccd640 1 --2- 172.21.15.88:0/956617503 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fd40051c0 0x7f4fd400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.323+0000 7f4ff1ccd640 1 -- 172.21.15.88:0/956617503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fd4004800 msgr2=0x7f4fd4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:42.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.323+0000 7f4ff1ccd640 1 --2- 172.21.15.88:0/956617503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fd4004800 0x7f4fd4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.323+0000 7f4ff1ccd640 1 -- 172.21.15.88:0/956617503 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd4003520 con 0x7f4fd40039b0 2024-09-16T01:45:42.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.323+0000 7f4ff14cc640 1 --2- 172.21.15.88:0/956617503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fd4004800 0x7f4fd4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:45:42.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.324+0000 7f4ff1ccd640 1 --2- 172.21.15.88:0/956617503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fd40039b0 0x7f4fd4003dd0 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc009fc0 tx=0x7f4fd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b1d1a9f9b0f005 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:42.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.324+0000 7f4ff0ccb640 1 -- 172.21.15.88:0/956617503 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc009c10 con 0x7f4fd40039b0 2024-09-16T01:45:42.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.324+0000 7f4ff0ccb640 1 -- 172.21.15.88:0/956617503 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fdc036070 con 0x7f4fd40039b0 2024-09-16T01:45:42.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.324+0000 7f4ff0ccb640 1 -- 172.21.15.88:0/956617503 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc02d070 con 0x7f4fd40039b0 2024-09-16T01:45:42.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.324+0000 7f4ff3f31640 1 -- 172.21.15.88:0/956617503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fd40039b0 msgr2=0x7f4fd4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:42.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.324+0000 7f4ff3f31640 1 --2- 172.21.15.88:0/956617503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fd40039b0 0x7f4fd4003dd0 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc009fc0 tx=0x7f4fd4005830 comp rx=0 tx=0).stop 2024-09-16T01:45:42.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.324+0000 7f4ff24ce640 1 -- 172.21.15.88:0/956617503 reap_dead start 2024-09-16T01:45:42.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.325+0000 7f4ff3f31640 1 -- 172.21.15.88:0/956617503 shutdown_connections 2024-09-16T01:45:42.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.325+0000 7f4ff3f31640 1 -- 172.21.15.88:0/956617503 >> 172.21.15.88:0/956617503 conn(0x7f4fec06cb50 msgr2=0x7f4fec06cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:42.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.325+0000 7f4ff3f31640 1 -- 172.21.15.88:0/956617503 shutdown_connections 2024-09-16T01:45:42.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.325+0000 7f4ff3f31640 1 -- 172.21.15.88:0/956617503 wait complete. 2024-09-16T01:45:42.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.326+0000 7f4ff3f31640 1 Processor -- start 2024-09-16T01:45:42.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.326+0000 7f4ff3f31640 1 -- start start 2024-09-16T01:45:42.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.326+0000 7f4ff3f31640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fd40039b0 0x7f4fec150230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:42.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.326+0000 7f4ff3f31640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fec0ce780 0x7f4fec0cec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:42.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.326+0000 7f4ff3f31640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fec1574c0 0x7f4fec157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:42.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.326+0000 7f4ff3f31640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7f4fec0b5ca0 con 0x7f4fec0ce780 2024-09-16T01:45:42.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.326+0000 7f4ff3f31640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4fec0b5b20 con 0x7f4fd40039b0 2024-09-16T01:45:42.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.327+0000 7f4ff3f31640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4fec0b5570 con 0x7f4fec1574c0 2024-09-16T01:45:42.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.327+0000 7f4ff14cc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fec0ce780 0x7f4fec0cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:42.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.327+0000 7f4ff14cc640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fec0ce780 0x7f4fec0cec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38810/0 (socket says 172.21.15.88:38810) 2024-09-16T01:45:42.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.327+0000 7f4ff14cc640 1 -- 172.21.15.88:0/2277433305 learned_addr learned my addr 172.21.15.88:0/2277433305 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:42.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.327+0000 7f4ff1ccd640 1 --2- 172.21.15.88:0/2277433305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fd40039b0 0x7f4fec150230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:42.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.327+0000 7f4ff24ce640 1 --2- 172.21.15.88:0/2277433305 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fec1574c0 0x7f4fec157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:42.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.327+0000 7f4ff14cc640 1 -- 172.21.15.88:0/2277433305 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fec1574c0 msgr2=0x7f4fec157940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:42.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.327+0000 7f4ff14cc640 1 --2- 172.21.15.88:0/2277433305 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fec1574c0 0x7f4fec157940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.327+0000 7f4ff14cc640 1 -- 172.21.15.88:0/2277433305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fd40039b0 msgr2=0x7f4fec150230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:42.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.327+0000 7f4ff14cc640 1 --2- 172.21.15.88:0/2277433305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fd40039b0 0x7f4fec150230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.327+0000 7f4ff14cc640 1 -- 172.21.15.88:0/2277433305 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fec15fdd0 con 0x7f4fec0ce780 2024-09-16T01:45:42.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.327+0000 7f4ff1ccd640 1 --2- 172.21.15.88:0/2277433305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fd40039b0 0x7f4fec150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:45:42.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.327+0000 7f4ff24ce640 1 --2- 172.21.15.88:0/2277433305 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fec1574c0 0x7f4fec157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T01:45:42.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.327+0000 7f4ff14cc640 1 --2- 172.21.15.88:0/2277433305 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fec0ce780 0x7f4fec0cec00 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0001050 tx=0x7f4fe000ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:42.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.327+0000 7f4fea7fc640 1 -- 172.21.15.88:0/2277433305 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe000c8d0 con 0x7f4fec0ce780 2024-09-16T01:45:42.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.327+0000 7f4fea7fc640 1 -- 172.21.15.88:0/2277433305 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fe0018070 con 0x7f4fec0ce780 2024-09-16T01:45:42.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.327+0000 7f4ff3f31640 1 -- 172.21.15.88:0/2277433305 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fec1f5860 con 0x7f4fec0ce780 2024-09-16T01:45:42.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.327+0000 7f4fea7fc640 1 -- 172.21.15.88:0/2277433305 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe000cc70 con 0x7f4fec0ce780 2024-09-16T01:45:42.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.327+0000 7f4ff3f31640 1 -- 172.21.15.88:0/2277433305 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fec1f4d80 con 0x7f4fec0ce780 2024-09-16T01:45:42.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.328+0000 7f4ff3f31640 1 -- 172.21.15.88:0/2277433305 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fec150b90 con 0x7f4fec0ce780 2024-09-16T01:45:42.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.329+0000 7f4fea7fc640 1 -- 172.21.15.88:0/2277433305 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7f4fe001d460 con 0x7f4fec0ce780 2024-09-16T01:45:42.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.329+0000 7f4fea7fc640 1 --2- 172.21.15.88:0/2277433305 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4fd806c2b0 0x7f4fd806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:42.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.329+0000 7f4fea7fc640 1 -- 172.21.15.88:0/2277433305 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7f4fe008afd0 con 0x7f4fec0ce780 2024-09-16T01:45:42.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.329+0000 7f4ff1ccd640 1 --2- 172.21.15.88:0/2277433305 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4fd806c2b0 0x7f4fd806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:42.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.330+0000 7f4ff1ccd640 1 --2- 172.21.15.88:0/2277433305 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4fd806c2b0 0x7f4fd806e770 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc035000 tx=0x7f4fdc02f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:42.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.333+0000 7f4fea7fc640 1 -- 172.21.15.88:0/2277433305 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7f4fec15fdd0 con 0x7f4fec0ce780 2024-09-16T01:45:42.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.500+0000 7f4ff3f31640 1 -- 172.21.15.88:0/2277433305 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f4fec071f30 con 0x7f4fec0ce780 2024-09-16T01:45:42.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.504+0000 7f4fea7fc640 1 -- 172.21.15.88:0/2277433305 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v30)=0 v30) ==== 123+0+0 (secure 0 0 0) 0x7f4fe0059a70 con 0x7f4fec0ce780 2024-09-16T01:45:42.506 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.505+0000 7f4fea7fc640 1 -- 172.21.15.88:0/2277433305 <== mon.0 v2:172.21.15.88:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f4fe0059370 con 0x7f4fec0ce780 2024-09-16T01:45:42.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.506+0000 7f4ff3f31640 1 -- 172.21.15.88:0/2277433305 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4fd806c2b0 msgr2=0x7f4fd806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:42.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.506+0000 7f4ff3f31640 1 --2- 172.21.15.88:0/2277433305 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4fd806c2b0 0x7f4fd806e770 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc035000 tx=0x7f4fdc02f000 comp rx=0 tx=0).stop 2024-09-16T01:45:42.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.506+0000 7f4ff3f31640 1 -- 172.21.15.88:0/2277433305 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fec0ce780 msgr2=0x7f4fec0cec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:42.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.506+0000 7f4ff3f31640 1 --2- 172.21.15.88:0/2277433305 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fec0ce780 0x7f4fec0cec00 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0001050 tx=0x7f4fe000ec60 comp rx=0 tx=0).stop 2024-09-16T01:45:42.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.506+0000 7f4ff3f31640 1 -- 172.21.15.88:0/2277433305 shutdown_connections 2024-09-16T01:45:42.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.506+0000 7f4ff3f31640 1 --2- 172.21.15.88:0/2277433305 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7f4fd806c2b0 0x7f4fd806e770 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.506+0000 7f4ff3f31640 1 --2- 172.21.15.88:0/2277433305 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4fec1574c0 0x7f4fec157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.506+0000 7f4ff3f31640 1 --2- 172.21.15.88:0/2277433305 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4fec0ce780 0x7f4fec0cec00 unknown :-1 s=CLOSED pgs=2494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.506+0000 7f4ff3f31640 1 --2- 172.21.15.88:0/2277433305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4fd40039b0 0x7f4fec150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.506+0000 7f4ff3f31640 1 -- 172.21.15.88:0/2277433305 >> 172.21.15.88:0/2277433305 conn(0x7f4fec06cb50 msgr2=0x7f4fec1534f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:42.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.506+0000 7f4ff3f31640 1 -- 172.21.15.88:0/2277433305 shutdown_connections 2024-09-16T01:45:42.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.506+0000 7f4ff3f31640 1 -- 172.21.15.88:0/2277433305 wait complete. 2024-09-16T01:45:42.519 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-16T01:45:42.519 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-16T01:45:42.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.616+0000 7fcf6b34b640 1 Processor -- start 2024-09-16T01:45:42.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.616+0000 7fcf6b34b640 1 -- start start 2024-09-16T01:45:42.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.617+0000 7fcf6b34b640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf6415e950 0x7fcf6406ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:42.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.617+0000 7fcf6b34b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcf640735b0 0x7fcf6406d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:42.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.617+0000 7fcf6b34b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf6406faf0 con 0x7fcf64072050 2024-09-16T01:45:42.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.617+0000 7fcf6b34b640 1 -- --> v2:172.21.15.88:3300/0 -- mon_getmap magic: 0 -- 0x7fcf6406fdd0 con 0x7fcf6415e950 2024-09-16T01:45:42.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.617+0000 7fcf6b34b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fcf640700b0 con 0x7fcf640735b0 2024-09-16T01:45:42.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.617+0000 7fcf690e7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf6415e950 0x7fcf6406ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:42.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.617+0000 7fcf690e7640 1 --2- >> v2:172.21.15.88:3300/0 conn(0x7fcf6415e950 0x7fcf6406ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38824/0 (socket says 172.21.15.88:38824) 2024-09-16T01:45:42.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.617+0000 7fcf690e7640 1 -- 172.21.15.88:0/1454800296 learned_addr learned my addr 172.21.15.88:0/1454800296 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:42.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.617+0000 7fcf688e6640 1 --2- 172.21.15.88:0/1454800296 >> v2:172.21.15.154:3300/0 conn(0x7fcf640735b0 0x7fcf6406d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:42.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.617+0000 7fcf690e7640 1 -- 172.21.15.88:0/1454800296 >> v2:172.21.15.154:3300/0 conn(0x7fcf640735b0 msgr2=0x7fcf6406d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:42.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.617+0000 7fcf690e7640 1 --2- 172.21.15.88:0/1454800296 >> v2:172.21.15.154:3300/0 conn(0x7fcf640735b0 0x7fcf6406d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.617+0000 7fcf690e7640 1 -- 172.21.15.88:0/1454800296 >> v1:172.21.15.154:6789/0 conn(0x7fcf64072050 legacy=0x7fcf6406bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T01:45:42.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.617+0000 7fcf690e7640 1 -- 172.21.15.88:0/1454800296 --> v2:172.21.15.88:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf64070390 con 0x7fcf6415e950 2024-09-16T01:45:42.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.618+0000 7fcf690e7640 1 --2- 172.21.15.88:0/1454800296 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf6415e950 0x7fcf6406ce30 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7fcf640743c0 tx=0x7fcf6002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c4f78d919d20321 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:42.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.618+0000 7fcf5bfff640 1 -- 172.21.15.88:0/1454800296 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf60037070 con 0x7fcf6415e950 2024-09-16T01:45:42.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.618+0000 7fcf5bfff640 1 -- 172.21.15.88:0/1454800296 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fcf60032440 con 0x7fcf6415e950 2024-09-16T01:45:42.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.618+0000 7fcf5bfff640 1 -- 172.21.15.88:0/1454800296 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf60032830 con 0x7fcf6415e950 2024-09-16T01:45:42.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.618+0000 7fcf6b34b640 1 -- 172.21.15.88:0/1454800296 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf6415e950 msgr2=0x7fcf6406ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:42.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.618+0000 7fcf6b34b640 1 --2- 172.21.15.88:0/1454800296 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf6415e950 0x7fcf6406ce30 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7fcf640743c0 tx=0x7fcf6002a230 comp rx=0 tx=0).stop 2024-09-16T01:45:42.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.619+0000 7fcf6b34b640 1 -- 172.21.15.88:0/1454800296 shutdown_connections 2024-09-16T01:45:42.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.619+0000 7fcf6b34b640 1 --2- 172.21.15.88:0/1454800296 >> v2:172.21.15.154:3300/0 conn(0x7fcf640735b0 0x7fcf6406d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.619+0000 7fcf6b34b640 1 --2- 172.21.15.88:0/1454800296 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf6415e950 0x7fcf6406ce30 unknown :-1 s=CLOSED pgs=2495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.619+0000 7fcf6b34b640 1 -- 172.21.15.88:0/1454800296 >> 172.21.15.88:0/1454800296 conn(0x7fcf640d14d0 msgr2=0x7fcf640d18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:42.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.619+0000 7fcf6b34b640 1 -- 172.21.15.88:0/1454800296 shutdown_connections 2024-09-16T01:45:42.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.619+0000 7fcf6b34b640 1 -- 172.21.15.88:0/1454800296 wait complete. 2024-09-16T01:45:42.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.620+0000 7fcf6b34b640 1 Processor -- start 2024-09-16T01:45:42.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.620+0000 7fcf6b34b640 1 -- start start 2024-09-16T01:45:42.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.620+0000 7fcf6b34b640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf64072050 0x7fcf640ca900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:42.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.620+0000 7fcf6b34b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf640735b0 0x7fcf6414b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:42.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.621+0000 7fcf6b34b640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf6415e950 0x7fcf640cc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:42.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.621+0000 7fcf6b34b640 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 -- 0x7fcf640d0230 con 0x7fcf6415e950 2024-09-16T01:45:42.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.621+0000 7fcf6b34b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcf640d00b0 con 0x7fcf640735b0 2024-09-16T01:45:42.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.621+0000 7fcf6b34b640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcf640d03b0 con 0x7fcf64072050 2024-09-16T01:45:42.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.621+0000 7fcf698e8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf6415e950 0x7fcf640cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:42.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.621+0000 7fcf698e8640 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf6415e950 0x7fcf640cc390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:38838/0 (socket says 172.21.15.88:38838) 2024-09-16T01:45:42.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.621+0000 7fcf698e8640 1 -- 172.21.15.88:0/261335331 learned_addr learned my addr 172.21.15.88:0/261335331 (peer_addr_for_me v2:172.21.15.88:0/0) 2024-09-16T01:45:42.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.621+0000 7fcf688e6640 1 --2- 172.21.15.88:0/261335331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf640735b0 0x7fcf6414b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:42.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.621+0000 7fcf690e7640 1 --2- 172.21.15.88:0/261335331 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf64072050 0x7fcf640ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:42.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.621+0000 7fcf698e8640 1 -- 172.21.15.88:0/261335331 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf64072050 msgr2=0x7fcf640ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:42.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.621+0000 7fcf698e8640 1 --2- 172.21.15.88:0/261335331 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf64072050 0x7fcf640ca900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.621+0000 7fcf698e8640 1 -- 172.21.15.88:0/261335331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf640735b0 msgr2=0x7fcf6414b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:42.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.621+0000 7fcf698e8640 1 --2- 172.21.15.88:0/261335331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf640735b0 0x7fcf6414b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.621+0000 7fcf698e8640 1 -- 172.21.15.88:0/261335331 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf640caf00 con 0x7fcf6415e950 2024-09-16T01:45:42.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.621+0000 7fcf688e6640 1 --2- 172.21.15.88:0/261335331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf640735b0 0x7fcf6414b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T01:45:42.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.621+0000 7fcf698e8640 1 --2- 172.21.15.88:0/261335331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf6415e950 0x7fcf640cc390 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00c7b0 tx=0x7fcf5c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:42.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.621+0000 7fcf59ffb640 1 -- 172.21.15.88:0/261335331 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c004b60 con 0x7fcf6415e950 2024-09-16T01:45:42.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.621+0000 7fcf59ffb640 1 -- 172.21.15.88:0/261335331 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fcf5c004d00 con 0x7fcf6415e950 2024-09-16T01:45:42.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.621+0000 7fcf6b34b640 1 -- 172.21.15.88:0/261335331 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf640cb1e0 con 0x7fcf6415e950 2024-09-16T01:45:42.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.621+0000 7fcf59ffb640 1 -- 172.21.15.88:0/261335331 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c00aab0 con 0x7fcf6415e950 2024-09-16T01:45:42.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.621+0000 7fcf6b34b640 1 -- 172.21.15.88:0/261335331 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf640cb7a0 con 0x7fcf6415e950 2024-09-16T01:45:42.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.622+0000 7fcf6b34b640 1 -- 172.21.15.88:0/261335331 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf6406ce30 con 0x7fcf6415e950 2024-09-16T01:45:42.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.623+0000 7fcf59ffb640 1 -- 172.21.15.88:0/261335331 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) ==== 90846+0+0 (secure 0 0 0) 0x7fcf5c00ac50 con 0x7fcf6415e950 2024-09-16T01:45:42.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.623+0000 7fcf59ffb640 1 --2- 172.21.15.88:0/261335331 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcf4806c2b0 0x7fcf4806e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T01:45:42.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.623+0000 7fcf59ffb640 1 -- 172.21.15.88:0/261335331 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5029+0+0 (secure 0 0 0) 0x7fcf5c08b470 con 0x7fcf6415e950 2024-09-16T01:45:42.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.623+0000 7fcf690e7640 1 --2- 172.21.15.88:0/261335331 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcf4806c2b0 0x7fcf4806e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T01:45:42.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.624+0000 7fcf690e7640 1 --2- 172.21.15.88:0/261335331 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcf4806c2b0 0x7fcf4806e770 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fcf60034000 tx=0x7fcf6002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T01:45:42.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.628+0000 7fcf59ffb640 1 -- 172.21.15.88:0/261335331 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187697 (secure 0 0 0) 0x7fcf5c020070 con 0x7fcf6415e950 2024-09-16T01:45:42.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.829+0000 7fcf6b34b640 1 -- 172.21.15.88:0/261335331 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf640b3db0 con 0x7fcf6415e950 2024-09-16T01:45:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.829+0000 7fcf59ffb640 1 -- 172.21.15.88:0/261335331 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 334 v334) ==== 95+0+3917 (secure 0 0 0) 0x7fcf5c059e80 con 0x7fcf6415e950 2024-09-16T01:45:42.831 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-09-16T01:45:42.831 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":334,"btime":"2024-09-16T01:45:11:921883+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7649,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6834/2706550508","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2706550508},{"type":"v1","addr":"172.21.15.154:6834","nonce":2706550508}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322},{"gid":8569,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/3460839382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6835","nonce":3460839382},{"type":"v1","addr":"172.21.15.88:6837","nonce":3460839382}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322},{"gid":8581,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2822970755","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6833","nonce":2822970755},{"type":"v1","addr":"172.21.15.154:6835","nonce":2822970755}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":322},{"gid":10854,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6836/62928880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":62928880},{"type":"v1","addr":"172.21.15.88:6836","nonce":62928880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":334}],"filesystems":[]} 2024-09-16T01:45:42.831 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 334 2024-09-16T01:45:42.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.831+0000 7fcf6b34b640 1 -- 172.21.15.88:0/261335331 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcf4806c2b0 msgr2=0x7fcf4806e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:42.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.831+0000 7fcf6b34b640 1 --2- 172.21.15.88:0/261335331 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcf4806c2b0 0x7fcf4806e770 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fcf60034000 tx=0x7fcf6002e000 comp rx=0 tx=0).stop 2024-09-16T01:45:42.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.831+0000 7fcf6b34b640 1 -- 172.21.15.88:0/261335331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf6415e950 msgr2=0x7fcf640cc390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T01:45:42.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.831+0000 7fcf6b34b640 1 --2- 172.21.15.88:0/261335331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf6415e950 0x7fcf640cc390 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00c7b0 tx=0x7fcf5c00cc70 comp rx=0 tx=0).stop 2024-09-16T01:45:42.833 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.831+0000 7fcf6b34b640 1 -- 172.21.15.88:0/261335331 shutdown_connections 2024-09-16T01:45:42.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.831+0000 7fcf6b34b640 1 --2- 172.21.15.88:0/261335331 >> [v2:172.21.15.88:6832/3938986063,v1:172.21.15.88:6833/3938986063] conn(0x7fcf4806c2b0 0x7fcf4806e770 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.831+0000 7fcf6b34b640 1 --2- 172.21.15.88:0/261335331 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf6415e950 0x7fcf640cc390 unknown :-1 s=CLOSED pgs=2496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.831+0000 7fcf6b34b640 1 --2- 172.21.15.88:0/261335331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf640735b0 0x7fcf6414b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.831+0000 7fcf6b34b640 1 --2- 172.21.15.88:0/261335331 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcf64072050 0x7fcf640ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T01:45:42.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.831+0000 7fcf6b34b640 1 -- 172.21.15.88:0/261335331 >> 172.21.15.88:0/261335331 conn(0x7fcf640d14d0 msgr2=0x7fcf640b29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T01:45:42.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.831+0000 7fcf6b34b640 1 -- 172.21.15.88:0/261335331 shutdown_connections 2024-09-16T01:45:42.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-09-16T01:45:42.831+0000 7fcf6b34b640 1 -- 172.21.15.88:0/261335331 wait complete. 2024-09-16T01:45:42.844 DEBUG:tasks.cephfs.filesystem:already dead... 2024-09-16T01:45:42.844 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T01:45:42.845 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-16T01:45:42.845 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:45:42.853 INFO:tasks.ceph.mds.a:Stopped 2024-09-16T01:45:42.854 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-16T01:45:42.854 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:45:42.903 INFO:tasks.ceph.mds.c:Stopped 2024-09-16T01:45:42.904 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-16T01:45:42.904 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:45:42.916 INFO:tasks.ceph.mds.b:Stopped 2024-09-16T01:45:42.917 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-16T01:45:42.917 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:45:42.968 INFO:tasks.ceph.mds.d:Stopped 2024-09-16T01:45:42.969 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T01:45:42.969 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T01:45:42.969 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:45:43.000 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T01:45:43.001 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T01:45:43.001 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:45:43.065 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T01:45:43.065 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T01:45:43.065 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:45:43.133 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T01:45:43.133 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T01:45:43.133 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:45:43.205 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T01:45:43.206 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-16T01:45:43.206 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:45:43.241 INFO:tasks.ceph.osd.4:Stopped 2024-09-16T01:45:43.241 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-16T01:45:43.241 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:45:43.307 INFO:tasks.ceph.osd.5:Stopped 2024-09-16T01:45:43.308 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-16T01:45:43.308 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:45:43.375 INFO:tasks.ceph.osd.6:Stopped 2024-09-16T01:45:43.375 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-16T01:45:43.375 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:45:43.449 INFO:tasks.ceph.osd.7:Stopped 2024-09-16T01:45:43.450 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T01:45:43.450 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-16T01:45:43.450 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:45:43.477 INFO:tasks.ceph.mgr.y:Stopped 2024-09-16T01:45:43.477 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T01:45:43.477 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:45:43.518 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T01:45:43.518 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T01:45:43.518 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T01:45:43.518 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:45:43.530 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T01:45:43.530 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T01:45:43.530 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:45:43.571 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T01:45:43.571 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T01:45:43.571 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T01:45:43.623 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T01:45:43.624 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T01:45:43.624 DEBUG:teuthology.orchestra.run.smithi088:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'bad backtrace' | egrep -v 'object missing on disk' | egrep -v 'error reading table object' | egrep -v 'error reading sessionmap' | egrep -v 'Error loading MDS rank' | egrep -v 'missing journal object' | egrep -v 'Error recovering journal' | egrep -v 'error decoding table object' | egrep -v 'failed to read JournalPointer' | egrep -v 'Corrupt directory entry' | egrep -v 'Corrupt fnode header' | egrep -v 'corrupt sessionmap header' | egrep -v 'Corrupt dentry' | egrep -v 'Scrub error on inode' | egrep -v 'Metadata damage detected' | egrep -v MDS_READ_ONLY | egrep -v 'force file system read-only' | egrep -v 'with standby daemon mds' | egrep -v 'MDS abort because newly corrupt dentry' | head -n 1 2024-09-16T01:45:43.665 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:45:43.665 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T01:45:43.851 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:45:43.852 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T01:45:43.939 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:45:43.940 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T01:45:44.005 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:45:44.005 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T01:45:44.058 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-16T01:45:44.058 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-16T01:45:44.248 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-16T01:45:44.248 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-16T01:45:44.310 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-16T01:45:44.311 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-16T01:45:44.367 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-16T01:45:44.368 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-16T01:45:44.426 INFO:tasks.ceph:Archiving mon data... 2024-09-16T01:45:44.428 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906477/data/mon.a.tgz 2024-09-16T01:45:44.428 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2024-09-16T01:45:44.445 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.VwHuTIOTl0 2024-09-16T01:45:44.445 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.VwHuTIOTl0 2024-09-16T01:45:44.627 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.VwHuTIOTl0 2024-09-16T01:45:44.710 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.VwHuTIOTl0 is 553KB 2024-09-16T01:45:44.809 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.VwHuTIOTl0 2024-09-16T01:45:44.825 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906477/data/mon.b.tgz 2024-09-16T01:45:44.825 DEBUG:teuthology.orchestra.run.smithi154:> mktemp 2024-09-16T01:45:44.843 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.9Pk0To3PkC 2024-09-16T01:45:44.844 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.9Pk0To3PkC 2024-09-16T01:45:45.027 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.9Pk0To3PkC 2024-09-16T01:45:45.111 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.9Pk0To3PkC is 557KB 2024-09-16T01:45:45.211 DEBUG:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.9Pk0To3PkC 2024-09-16T01:45:45.226 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906477/data/mon.c.tgz 2024-09-16T01:45:45.226 DEBUG:teuthology.orchestra.run.smithi154:> mktemp 2024-09-16T01:45:45.291 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.29yj82VxLL 2024-09-16T01:45:45.292 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.29yj82VxLL 2024-09-16T01:45:45.465 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.29yj82VxLL 2024-09-16T01:45:45.556 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.29yj82VxLL is 557KB 2024-09-16T01:45:45.742 DEBUG:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.29yj82VxLL 2024-09-16T01:45:45.757 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T01:45:45.757 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T01:45:45.760 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T01:45:45.840 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T01:45:45.877 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T01:45:45.877 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T01:45:45.879 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906477/remote/smithi088/crash 2024-09-16T01:45:45.880 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T01:45:45.913 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906477/remote/smithi154/crash 2024-09-16T01:45:45.914 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T01:45:45.968 INFO:tasks.ceph:Shutting down logrotate 2024-09-16T01:45:45.969 DEBUG:teuthology.orchestra.run.smithi088:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:45:45.971 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-16T01:45:46.056 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-16T01:45:46.081 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-16T01:45:46.105 INFO:tasks.ceph:Compressing logs... 2024-09-16T01:45:46.105 DEBUG:teuthology.orchestra.run.smithi088:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T01:45:46.123 DEBUG:teuthology.orchestra.run.smithi154:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T01:45:46.145 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-16T01:45:46.145 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79515.log 2024-09-16T01:45:46.189 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36756.log 2024-09-16T01:45:46.189 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-16T01:45:46.189 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.42828.log 2024-09-16T01:45:46.190 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42419.log 2024-09-16T01:45:46.190 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-osd.7.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36756.log.gz 2024-09-16T01:45:46.190 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.3.42828.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.42828.log.gz 2024-09-16T01:45:46.190 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36632.log 2024-09-16T01:45:46.190 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.1.42419.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-09-16T01:45:46.190 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.42419.log.gz 2024-09-16T01:45:46.190 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36632.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-16T01:45:46.190 INFO:teuthology.orchestra.run.smithi154.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36632.log.gz 2024-09-16T01:45:46.191 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T01:45:46.191 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42424.log 2024-09-16T01:45:46.191 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36843.log 2024-09-16T01:45:46.191 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.1.42424.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-16T01:45:46.191 INFO:teuthology.orchestra.run.smithi154.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.42424.log.gz 2024-09-16T01:45:46.191 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36552.log 2024-09-16T01:45:46.191 INFO:teuthology.orchestra.run.smithi154.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36843.log.gz 2024-09-16T01:45:46.191 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36712.log 2024-09-16T01:45:46.192 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36672.log 2024-09-16T01:45:46.192 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36712.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36552.log.gz 2024-09-16T01:45:46.192 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-16T01:45:46.192 INFO:teuthology.orchestra.run.smithi154.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.36712.log.gz 2024-09-16T01:45:46.192 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36803.log 2024-09-16T01:45:46.192 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36672.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-16T01:45:46.192 INFO:teuthology.orchestra.run.smithi154.stderr: 92.7% 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36672.log.gz 2024-09-16T01:45:46.192 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-osd.6.log: -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-09-16T01:45:46.193 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36803.log: gzip -5 87.2% -- replaced with /var/log/ceph/ceph-client.admin.36803.log.gz 2024-09-16T01:45:46.193 INFO:teuthology.orchestra.run.smithi154.stderr: --verbose -- /var/log/ceph/ceph.log 2024-09-16T01:45:46.193 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-09-16T01:45:46.193 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-16T01:45:46.193 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-mds.d.log: 94.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T01:45:46.193 INFO:teuthology.orchestra.run.smithi154.stderr: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T01:45:46.205 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-16T01:45:46.205 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.42832.log 2024-09-16T01:45:46.211 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-mon.b.log: 94.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-16T01:45:46.211 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36592.log 2024-09-16T01:45:46.211 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.3.42832.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.42832.log.gz 2024-09-16T01:45:46.211 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.36592.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36592.log.gz 2024-09-16T01:45:46.214 INFO:teuthology.orchestra.run.smithi154.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-09-16T01:45:46.546 INFO:teuthology.orchestra.run.smithi154.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-16T01:45:46.577 INFO:teuthology.orchestra.run.smithi154.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-16T01:45:46.592 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67246.log 2024-09-16T01:45:46.592 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52249.log 2024-09-16T01:45:46.592 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.79515.log: /var/log/ceph/ceph-client.admin.67246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76841.log 2024-09-16T01:45:46.592 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.79515.log.gz 2024-09-16T01:45:46.592 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51572.log 2024-09-16T01:45:46.593 INFO:teuthology.orchestra.run.smithi088.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.67246.log.gz 2024-09-16T01:45:46.593 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51404.log 2024-09-16T01:45:46.593 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87976.log 2024-09-16T01:45:46.593 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.76841.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52249.log.gz 2024-09-16T01:45:46.593 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51572.log: /var/log/ceph/ceph-client.admin.51404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84269.log 2024-09-16T01:45:46.593 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.76841.log.gz 2024-09-16T01:45:46.593 INFO:teuthology.orchestra.run.smithi088.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51404.log.gz 2024-09-16T01:45:46.593 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.87976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88144.log 2024-09-16T01:45:46.594 INFO:teuthology.orchestra.run.smithi088.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.51572.log.gz 2024-09-16T01:45:46.594 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.84269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59415.log 2024-09-16T01:45:46.594 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.87976.log.gz 2024-09-16T01:45:46.594 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.84269.log.gz 2024-09-16T01:45:46.594 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.88144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54075.log 2024-09-16T01:45:46.594 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.59415.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88144.log.gz 2024-09-16T01:45:46.594 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58181.log 2024-09-16T01:45:46.595 INFO:teuthology.orchestra.run.smithi088.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59415.log.gz 2024-09-16T01:45:46.595 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42943.log 2024-09-16T01:45:46.595 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.58181.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54075.log.gz 2024-09-16T01:45:46.595 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81361.log 2024-09-16T01:45:46.595 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.42943.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58181.log.gz 2024-09-16T01:45:46.595 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77327.log 2024-09-16T01:45:46.595 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42943.log.gz 2024-09-16T01:45:46.595 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.81361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83206.log 2024-09-16T01:45:46.596 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.77327.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.81361.log.gz 2024-09-16T01:45:46.596 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81721.log 2024-09-16T01:45:46.596 INFO:teuthology.orchestra.run.smithi088.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.77327.log.gz 2024-09-16T01:45:46.596 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.83206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59084.log 2024-09-16T01:45:46.596 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.83206.log.gz 2024-09-16T01:45:46.596 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.81721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.85697.log 2024-09-16T01:45:46.596 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.59084.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81721.log.gz 2024-09-16T01:45:46.596 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45348.log 2024-09-16T01:45:46.597 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59084.log.gz 2024-09-16T01:45:46.597 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.85697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53810.log 2024-09-16T01:45:46.597 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.85697.log.gz 2024-09-16T01:45:46.597 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73483.log 2024-09-16T01:45:46.597 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53810.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45348.log.gz 2024-09-16T01:45:46.597 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77851.log 2024-09-16T01:45:46.597 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53810.log.gz 2024-09-16T01:45:46.597 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.73483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65245.log 2024-09-16T01:45:46.598 INFO:teuthology.orchestra.run.smithi088.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.73483.log.gz 2024-09-16T01:45:46.598 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.77851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71190.log 2024-09-16T01:45:46.598 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.65245.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77851.log.gz 2024-09-16T01:45:46.598 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43521.log 2024-09-16T01:45:46.598 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.71190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74240.log 2024-09-16T01:45:46.598 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71190.log.gz 2024-09-16T01:45:46.598 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.76577.log 2024-09-16T01:45:46.599 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43521.log.gz 2024-09-16T01:45:46.599 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55370.log 2024-09-16T01:45:46.599 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.74240.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.65245.log.gz 2024-09-16T01:45:46.599 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.76577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50980.log 2024-09-16T01:45:46.599 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.76577.log.gz 2024-09-16T01:45:46.599 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.74240.log.gz 2024-09-16T01:45:46.599 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.49448.log 2024-09-16T01:45:46.599 INFO:teuthology.orchestra.run.smithi088.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55370.log.gz 2024-09-16T01:45:46.599 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58267.log 2024-09-16T01:45:46.600 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.49448.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50980.log.gz 2024-09-16T01:45:46.600 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68765.log 2024-09-16T01:45:46.600 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.49448.log.gz 2024-09-16T01:45:46.600 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.58267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74030.log 2024-09-16T01:45:46.600 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58267.log.gz 2024-09-16T01:45:46.600 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.68765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47066.log 2024-09-16T01:45:46.600 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.74030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61344.log 2024-09-16T01:45:46.601 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74030.log.gz 2024-09-16T01:45:46.601 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.47066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42474.log 2024-09-16T01:45:46.601 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47066.log.gz 2024-09-16T01:45:46.601 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63709.log 2024-09-16T01:45:46.601 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.61344.log: /var/log/ceph/ceph-client.admin.42474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79795.log 2024-09-16T01:45:46.601 INFO:teuthology.orchestra.run.smithi088.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.68765.log.gz 2024-09-16T01:45:46.601 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61344.log.gz 2024-09-16T01:45:46.601 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42474.log.gz 2024-09-16T01:45:46.602 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.63709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81421.log 2024-09-16T01:45:46.602 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.79795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72219.log 2024-09-16T01:45:46.602 INFO:teuthology.orchestra.run.smithi088.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.63709.log.gz 2024-09-16T01:45:46.602 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.81421.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.79795.log.gz 2024-09-16T01:45:46.602 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84395.log 2024-09-16T01:45:46.602 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1%/var/log/ceph/ceph-client.admin.72219.log: -- replaced with /var/log/ceph/ceph-client.admin.81421.log.gz 2024-09-16T01:45:46.602 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72673.log 2024-09-16T01:45:46.602 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.84395.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72219.log.gz 2024-09-16T01:45:46.603 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86324.log 2024-09-16T01:45:46.603 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.84395.log.gz 2024-09-16T01:45:46.603 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.72673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61638.log 2024-09-16T01:45:46.603 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.86324.log: gzip -5 --verbose -- 88.3% /var/log/ceph/ceph-client.admin.66344.log 2024-09-16T01:45:46.603 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.72673.log.gz 2024-09-16T01:45:46.603 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.61638.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86324.log.gz 2024-09-16T01:45:46.603 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52207.log 2024-09-16T01:45:46.603 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.66344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45848.log 2024-09-16T01:45:46.604 INFO:teuthology.orchestra.run.smithi088.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.61638.log.gz 2024-09-16T01:45:46.604 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52207.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.66344.log.gz 2024-09-16T01:45:46.604 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52052.log 2024-09-16T01:45:46.604 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52207.log.gz 2024-09-16T01:45:46.604 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69565.log 2024-09-16T01:45:46.604 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45848.log: /var/log/ceph/ceph-client.admin.52052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59946.log 2024-09-16T01:45:46.604 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45848.log.gz 2024-09-16T01:45:46.605 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52052.log.gz 2024-09-16T01:45:46.605 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85019.log 2024-09-16T01:45:46.605 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.69565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74520.log 2024-09-16T01:45:46.605 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.59946.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.69565.log.gz 2024-09-16T01:45:46.605 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.85019.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59946.log.gz 2024-09-16T01:45:46.605 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85233.log 2024-09-16T01:45:46.605 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.85019.log.gz 2024-09-16T01:45:46.605 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88102.log 2024-09-16T01:45:46.606 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.74520.log: /var/log/ceph/ceph-client.admin.85233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54523.log 2024-09-16T01:45:46.606 INFO:teuthology.orchestra.run.smithi088.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.74520.log.gz 2024-09-16T01:45:46.606 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.85233.log.gz 2024-09-16T01:45:46.606 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.88102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83735.log 2024-09-16T01:45:46.606 INFO:teuthology.orchestra.run.smithi088.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.88102.log.gz 2024-09-16T01:45:46.606 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59499.log 2024-09-16T01:45:46.606 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.83735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50841.log 2024-09-16T01:45:46.607 INFO:teuthology.orchestra.run.smithi088.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.54523.log.gz 2024-09-16T01:45:46.607 INFO:teuthology.orchestra.run.smithi088.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.83735.log.gz 2024-09-16T01:45:46.607 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.59499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43274.log 2024-09-16T01:45:46.607 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59499.log.gz 2024-09-16T01:45:46.607 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66190.log 2024-09-16T01:45:46.607 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43274.log: gzip 88.4% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.50841.log.gz -- 2024-09-16T01:45:46.607 INFO:teuthology.orchestra.run.smithi088.stderr: /var/log/ceph/ceph-client.admin.53301.log 2024-09-16T01:45:46.607 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.66190.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43274.log.gz 2024-09-16T01:45:46.608 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72355.log 2024-09-16T01:45:46.608 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66190.log.gz 2024-09-16T01:45:46.608 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.53301.log: /var/log/ceph/ceph-client.admin.59594.log 2024-09-16T01:45:46.608 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.72355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68335.log 2024-09-16T01:45:46.608 INFO:teuthology.orchestra.run.smithi088.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53301.log.gz 2024-09-16T01:45:46.608 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.59594.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.72355.log.gz 2024-09-16T01:45:46.608 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68026.log 2024-09-16T01:45:46.609 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59594.log.gz 2024-09-16T01:45:46.609 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82697.log 2024-09-16T01:45:46.609 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.68335.log: /var/log/ceph/ceph-client.admin.68026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85401.log 2024-09-16T01:45:46.609 INFO:teuthology.orchestra.run.smithi088.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.68335.log.gz 2024-09-16T01:45:46.609 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.82697.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.68026.log.gz 2024-09-16T01:45:46.609 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69364.log 2024-09-16T01:45:46.609 INFO:teuthology.orchestra.run.smithi088.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.82697.log.gz 2024-09-16T01:45:46.609 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.85401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57627.log 2024-09-16T01:45:46.610 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.69364.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.85401.log.gz 2024-09-16T01:45:46.610 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75194.log 2024-09-16T01:45:46.610 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.57627.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.69364.log.gz 2024-09-16T01:45:46.610 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53553.log 2024-09-16T01:45:46.610 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57627.log.gz 2024-09-16T01:45:46.610 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.75194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45681.log 2024-09-16T01:45:46.610 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75446.log 2024-09-16T01:45:46.610 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.75194.log.gz 2024-09-16T01:45:46.611 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53553.log: /var/log/ceph/ceph-client.admin.45681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82499.log 2024-09-16T01:45:46.611 INFO:teuthology.orchestra.run.smithi088.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.53553.log.gz 2024-09-16T01:45:46.611 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45681.log.gz 2024-09-16T01:45:46.611 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.75446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61428.log 2024-09-16T01:45:46.611 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75446.log.gz 2024-09-16T01:45:46.611 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.82499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83300.log 2024-09-16T01:45:46.611 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.61428.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.82499.log.gz 2024-09-16T01:45:46.611 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64264.log 2024-09-16T01:45:46.612 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.61428.log.gz 2024-09-16T01:45:46.612 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.83300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64786.log 2024-09-16T01:45:46.612 INFO:teuthology.orchestra.run.smithi088.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.83300.log.gz 2024-09-16T01:45:46.612 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.64264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65390.log 2024-09-16T01:45:46.612 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.64786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72631.log 2024-09-16T01:45:46.612 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64786.log.gz 2024-09-16T01:45:46.612 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.65390.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.64264.log.gz 2024-09-16T01:45:46.612 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62747.log 2024-09-16T01:45:46.613 INFO:teuthology.orchestra.run.smithi088.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.65390.log.gz 2024-09-16T01:45:46.613 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87344.log 2024-09-16T01:45:46.613 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.72631.log: /var/log/ceph/ceph-client.admin.62747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65591.log 2024-09-16T01:45:46.613 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.87344.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.72631.log.gz 2024-09-16T01:45:46.613 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46629.log 2024-09-16T01:45:46.613 INFO:teuthology.orchestra.run.smithi088.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.62747.log.gz 2024-09-16T01:45:46.613 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.65591.log: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.87344.log.gz 2024-09-16T01:45:46.613 INFO:teuthology.orchestra.run.smithi088.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.65474.log 2024-09-16T01:45:46.614 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46629.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.65591.log.gz 2024-09-16T01:45:46.614 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88270.log 2024-09-16T01:45:46.614 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46629.log.gz 2024-09-16T01:45:46.614 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.65474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53511.log 2024-09-16T01:45:46.614 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.88270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62413.log 2024-09-16T01:45:46.614 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.65474.log.gz 2024-09-16T01:45:46.614 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53511.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.88270.log.gz 2024-09-16T01:45:46.614 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85103.log 2024-09-16T01:45:46.615 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53511.log.gz 2024-09-16T01:45:46.615 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.62413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67989.log 2024-09-16T01:45:46.615 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.85103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43986.log 2024-09-16T01:45:46.615 INFO:teuthology.orchestra.run.smithi088.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.62413.log.gz 2024-09-16T01:45:46.615 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.67989.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.85103.log.gz 2024-09-16T01:45:46.615 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84796.log 2024-09-16T01:45:46.615 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62302.log 2024-09-16T01:45:46.615 INFO:teuthology.orchestra.run.smithi088.stderr: 87.5%/var/log/ceph/ceph-client.admin.84796.log: -- replaced with /var/log/ceph/ceph-client.admin.43986.log.gz 2024-09-16T01:45:46.616 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36724.log 2024-09-16T01:45:46.616 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.84796.log.gz 2024-09-16T01:45:46.616 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-16T01:45:46.616 INFO:teuthology.orchestra.run.smithi088.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.67989.log.gz 2024-09-16T01:45:46.616 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.62302.log: /var/log/ceph/ceph-client.admin.36724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63598.log 2024-09-16T01:45:46.616 INFO:teuthology.orchestra.run.smithi088.stderr: 87.8%/var/log/ceph/ceph-osd.1.log: -- replaced with /var/log/ceph/ceph-client.admin.36724.log.gz 2024-09-16T01:45:46.616 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62224.log 2024-09-16T01:45:46.617 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.63598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80898.log 2024-09-16T01:45:46.617 INFO:teuthology.orchestra.run.smithi088.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.62302.log.gz 2024-09-16T01:45:46.617 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.62224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75352.log 2024-09-16T01:45:46.617 INFO:teuthology.orchestra.run.smithi088.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.63598.log.gz 2024-09-16T01:45:46.617 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62224.log.gz 2024-09-16T01:45:46.617 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.80898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67905.log 2024-09-16T01:45:46.617 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.75352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82180.log 2024-09-16T01:45:46.617 INFO:teuthology.orchestra.run.smithi088.stderr: 91.2% 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80898.log.gz -- replaced with /var/log/ceph/ceph-client.admin.75352.log.gz 2024-09-16T01:45:46.618 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.618 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.67905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61596.log 2024-09-16T01:45:46.618 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.82180.log: gzip 89.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.67905.log.gz -- 2024-09-16T01:45:46.618 INFO:teuthology.orchestra.run.smithi088.stderr: /var/log/ceph/ceph-client.admin.69135.log 2024-09-16T01:45:46.618 INFO:teuthology.orchestra.run.smithi088.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.82180.log.gz 2024-09-16T01:45:46.618 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.61596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66555.log 2024-09-16T01:45:46.618 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81591.log 2024-09-16T01:45:46.618 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.69135.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61596.log.gz 2024-09-16T01:45:46.619 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.66555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49580.log 2024-09-16T01:45:46.619 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.69135.log.gz 2024-09-16T01:45:46.619 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66555.log.gz 2024-09-16T01:45:46.619 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.81591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82636.log 2024-09-16T01:45:46.619 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69060.log 2024-09-16T01:45:46.619 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81591.log.gz 2024-09-16T01:45:46.619 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.82636.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49580.log.gz 2024-09-16T01:45:46.619 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84696.log 2024-09-16T01:45:46.620 INFO:teuthology.orchestra.run.smithi088.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.82636.log.gz 2024-09-16T01:45:46.620 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.69060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73431.log 2024-09-16T01:45:46.620 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.84696.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.69060.log.gz 2024-09-16T01:45:46.620 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68218.log 2024-09-16T01:45:46.620 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.84696.log.gz 2024-09-16T01:45:46.620 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68293.log 2024-09-16T01:45:46.620 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.73431.log: /var/log/ceph/ceph-client.admin.68218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84479.log 2024-09-16T01:45:46.620 INFO:teuthology.orchestra.run.smithi088.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.73431.log.gz 2024-09-16T01:45:46.621 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.68293.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.68218.log.gz 2024-09-16T01:45:46.621 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53469.log 2024-09-16T01:45:46.621 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68293.log.gz 2024-09-16T01:45:46.621 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83693.log 2024-09-16T01:45:46.621 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.84479.log: /var/log/ceph/ceph-client.admin.53469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80739.log 2024-09-16T01:45:46.621 INFO:teuthology.orchestra.run.smithi088.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.84479.log.gz 2024-09-16T01:45:46.621 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53469.log.gz 2024-09-16T01:45:46.621 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.83693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81538.log 2024-09-16T01:45:46.622 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.83693.log.gz 2024-09-16T01:45:46.622 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68545.log 2024-09-16T01:45:46.622 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.80739.log: /var/log/ceph/ceph-client.admin.81538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69625.log 2024-09-16T01:45:46.622 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.80739.log.gz 2024-09-16T01:45:46.622 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.68545.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81538.log.gz 2024-09-16T01:45:46.622 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46936.log 2024-09-16T01:45:46.622 INFO:teuthology.orchestra.run.smithi088.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68545.log.gz 2024-09-16T01:45:46.622 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68082.log 2024-09-16T01:45:46.623 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.69625.log: /var/log/ceph/ceph-client.admin.46936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.78927.log 2024-09-16T01:45:46.623 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46936.log.gz 2024-09-16T01:45:46.623 INFO:teuthology.orchestra.run.smithi088.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.69625.log.gz 2024-09-16T01:45:46.623 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.68082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55722.log 2024-09-16T01:45:46.623 INFO:teuthology.orchestra.run.smithi088.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.68082.log.gz 2024-09-16T01:45:46.623 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83459.log 2024-09-16T01:45:46.623 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.78927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.78927.log.gz 2024-09-16T01:45:46.623 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66232.log 2024-09-16T01:45:46.624 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55722.log.gz 2024-09-16T01:45:46.624 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.83459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70616.log 2024-09-16T01:45:46.624 INFO:teuthology.orchestra.run.smithi088.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.83459.log.gz 2024-09-16T01:45:46.624 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77715.log 2024-09-16T01:45:46.624 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.66232.log: /var/log/ceph/ceph-client.admin.70616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63376.log 2024-09-16T01:45:46.624 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66232.log.gz 2024-09-16T01:45:46.624 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.70616.log.gz 2024-09-16T01:45:46.624 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.77715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87115.log 2024-09-16T01:45:46.625 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77715.log.gz 2024-09-16T01:45:46.625 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49106.log 2024-09-16T01:45:46.625 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.63376.log: /var/log/ceph/ceph-client.admin.87115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78435.log 2024-09-16T01:45:46.625 INFO:teuthology.orchestra.run.smithi088.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63376.log.gz 2024-09-16T01:45:46.625 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.87115.log.gz 2024-09-16T01:45:46.625 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42941.log 2024-09-16T01:45:46.625 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.78435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42260.log 2024-09-16T01:45:46.625 INFO:teuthology.orchestra.run.smithi088.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.49106.log.gz 2024-09-16T01:45:46.626 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.78435.log.gz 2024-09-16T01:45:46.626 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.42941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71989.log 2024-09-16T01:45:46.626 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.42260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78398.log 2024-09-16T01:45:46.626 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42941.log.gz 2024-09-16T01:45:46.626 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42260.log.gz 2024-09-16T01:45:46.626 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.71989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65861.log 2024-09-16T01:45:46.626 INFO:teuthology.orchestra.run.smithi088.stderr:gzip/var/log/ceph/ceph-client.admin.78398.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.83914.log 2024-09-16T01:45:46.626 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.65861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77169.log 2024-09-16T01:45:46.627 INFO:teuthology.orchestra.run.smithi088.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.65861.log.gz 2024-09-16T01:45:46.627 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.83914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74132.log 2024-09-16T01:45:46.627 INFO:teuthology.orchestra.run.smithi088.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.83914.log.gz 2024-09-16T01:45:46.627 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.77169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57314.log 2024-09-16T01:45:46.627 INFO:teuthology.orchestra.run.smithi088.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.71989.log.gz 2024-09-16T01:45:46.627 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.74132.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.77169.log.gz 2024-09-16T01:45:46.627 INFO:teuthology.orchestra.run.smithi088.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.78398.log.gz 2024-09-16T01:45:46.627 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88354.log 2024-09-16T01:45:46.628 INFO:teuthology.orchestra.run.smithi088.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.74132.log.gz 2024-09-16T01:45:46.628 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.57314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54033.log 2024-09-16T01:45:46.628 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.88354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61033.log 2024-09-16T01:45:46.628 INFO:teuthology.orchestra.run.smithi088.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57314.log.gz 2024-09-16T01:45:46.628 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88354.log.gz 2024-09-16T01:45:46.628 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68976.log 2024-09-16T01:45:46.628 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.61033.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54033.log.gz 2024-09-16T01:45:46.628 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47528.log 2024-09-16T01:45:46.629 INFO:teuthology.orchestra.run.smithi088.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61033.log.gz 2024-09-16T01:45:46.629 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.68976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76865.log 2024-09-16T01:45:46.629 INFO:teuthology.orchestra.run.smithi088.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.68976.log.gz 2024-09-16T01:45:46.629 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.47528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46100.log 2024-09-16T01:45:46.629 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.76865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45771.log 2024-09-16T01:45:46.629 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46100.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.47528.log.gz 2024-09-16T01:45:46.629 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70363.log 2024-09-16T01:45:46.629 INFO:teuthology.orchestra.run.smithi088.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.46100.log.gz 2024-09-16T01:45:46.630 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66502.log 2024-09-16T01:45:46.630 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.70363.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45771.log.gz 2024-09-16T01:45:46.630 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47276.log 2024-09-16T01:45:46.630 INFO:teuthology.orchestra.run.smithi088.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.76865.log.gz 2024-09-16T01:45:46.630 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.66502.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70363.log.gz 2024-09-16T01:45:46.631 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65161.log 2024-09-16T01:45:46.631 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66502.log.gz 2024-09-16T01:45:46.631 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.47276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.54715.log 2024-09-16T01:45:46.631 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47276.log.gz 2024-09-16T01:45:46.631 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43522.log 2024-09-16T01:45:46.631 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.65161.log: /var/log/ceph/ceph-client.0.54715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80622.log 2024-09-16T01:45:46.631 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.54715.log.gz 2024-09-16T01:45:46.631 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.65161.log.gz 2024-09-16T01:45:46.631 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50675.log 2024-09-16T01:45:46.632 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.80622.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43522.log.gz 2024-09-16T01:45:46.632 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86058.log 2024-09-16T01:45:46.632 INFO:teuthology.orchestra.run.smithi088.stderr:gzip 88.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.80622.log.gz --verbose 2024-09-16T01:45:46.632 INFO:teuthology.orchestra.run.smithi088.stderr: -- /var/log/ceph/ceph-client.admin.42946.log 2024-09-16T01:45:46.632 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50675.log: /var/log/ceph/ceph-client.admin.86058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48296.log 2024-09-16T01:45:46.632 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50675.log.gz 2024-09-16T01:45:46.632 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.86058.log.gz 2024-09-16T01:45:46.633 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.42946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65801.log 2024-09-16T01:45:46.633 INFO:teuthology.orchestra.run.smithi088.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42946.log.gz 2024-09-16T01:45:46.633 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65759.log 2024-09-16T01:45:46.633 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48296.log: /var/log/ceph/ceph-client.admin.65801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48660.log 2024-09-16T01:45:46.633 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48296.log.gz 2024-09-16T01:45:46.633 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.65759.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.65801.log.gz 2024-09-16T01:45:46.633 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79416.log 2024-09-16T01:45:46.633 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.65759.log.gz 2024-09-16T01:45:46.634 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55568.log 2024-09-16T01:45:46.634 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.48660.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.48660.log.gz 2024-09-16T01:45:46.634 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.79416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43627.log 2024-09-16T01:45:46.634 INFO:teuthology.orchestra.run.smithi088.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.79416.log.gz 2024-09-16T01:45:46.634 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88480.log 2024-09-16T01:45:46.634 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86478.log 2024-09-16T01:45:46.634 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55568.log.gz 2024-09-16T01:45:46.635 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43627.log: /var/log/ceph/ceph-client.admin.88480.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43627.log.gz 2024-09-16T01:45:46.635 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79469.log 2024-09-16T01:45:46.635 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70226.log 2024-09-16T01:45:46.635 INFO:teuthology.orchestra.run.smithi088.stderr: 87.3%/var/log/ceph/ceph-client.admin.86478.log: -- replaced with /var/log/ceph/ceph-client.admin.88480.log.gz 2024-09-16T01:45:46.635 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.79469.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.86478.log.gz 2024-09-16T01:45:46.635 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84058.log 2024-09-16T01:45:46.635 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 88.7% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.79469.log.gz /var/log/ceph/ceph-client.admin.61386.log 2024-09-16T01:45:46.635 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.636 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.70226.log: /var/log/ceph/ceph-client.admin.84058.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70226.log.gz 2024-09-16T01:45:46.636 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52701.log 2024-09-16T01:45:46.636 INFO:teuthology.orchestra.run.smithi088.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.84058.log.gz 2024-09-16T01:45:46.636 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81637.log 2024-09-16T01:45:46.636 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.61386.log: /var/log/ceph/ceph-client.admin.52701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65651.log 2024-09-16T01:45:46.636 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61386.log.gz 2024-09-16T01:45:46.636 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52701.log.gz 2024-09-16T01:45:46.636 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.81637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87874.log 2024-09-16T01:45:46.637 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.65651.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81637.log.gz 2024-09-16T01:45:46.637 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51152.log 2024-09-16T01:45:46.637 INFO:teuthology.orchestra.run.smithi088.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.65651.log.gz 2024-09-16T01:45:46.637 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77463.log 2024-09-16T01:45:46.637 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.87874.log: /var/log/ceph/ceph-client.admin.51152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57543.log 2024-09-16T01:45:46.637 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.87874.log.gz 2024-09-16T01:45:46.637 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.77463.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51152.log.gz 2024-09-16T01:45:46.637 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61512.log 2024-09-16T01:45:46.638 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77463.log.gz 2024-09-16T01:45:46.638 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72313.log 2024-09-16T01:45:46.638 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.57543.log: /var/log/ceph/ceph-client.admin.61512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87832.log 2024-09-16T01:45:46.638 INFO:teuthology.orchestra.run.smithi088.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57543.log.gz 2024-09-16T01:45:46.638 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.72313.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61512.log.gz 2024-09-16T01:45:46.638 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65044.log 2024-09-16T01:45:46.638 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72313.log.gz 2024-09-16T01:45:46.638 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79599.log 2024-09-16T01:45:46.639 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.87832.log: /var/log/ceph/ceph-client.admin.65044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.85701.log 2024-09-16T01:45:46.639 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.87832.log.gz/var/log/ceph/ceph-client.admin.79599.log: 2024-09-16T01:45:46.639 INFO:teuthology.orchestra.run.smithi088.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65044.log.gz 2024-09-16T01:45:46.639 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55293.log 2024-09-16T01:45:46.639 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.79599.log.gz 2024-09-16T01:45:46.639 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46586.log 2024-09-16T01:45:46.639 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.85701.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.85701.log.gz 2024-09-16T01:45:46.639 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64175.log 2024-09-16T01:45:46.640 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46586.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.55293.log.gz 2024-09-16T01:45:46.640 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87748.log 2024-09-16T01:45:46.640 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50019.log 2024-09-16T01:45:46.640 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46586.log.gz 2024-09-16T01:45:46.640 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.64175.log: /var/log/ceph/ceph-client.admin.87748.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.64175.log.gz 2024-09-16T01:45:46.640 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68911.log 2024-09-16T01:45:46.640 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.87748.log.gz 2024-09-16T01:45:46.640 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48402.log 2024-09-16T01:45:46.641 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50019.log: /var/log/ceph/ceph-client.admin.68911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75488.log 2024-09-16T01:45:46.641 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50019.log.gz 2024-09-16T01:45:46.641 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.68911.log.gz 2024-09-16T01:45:46.641 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50206.log 2024-09-16T01:45:46.641 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60731.log 2024-09-16T01:45:46.641 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.75488.log: /var/log/ceph/ceph-client.admin.50206.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.48402.log.gz 2024-09-16T01:45:46.641 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41665.log 2024-09-16T01:45:46.641 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75488.log.gz 2024-09-16T01:45:46.642 INFO:teuthology.orchestra.run.smithi088.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.50206.log.gz 2024-09-16T01:45:46.642 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67185.log 2024-09-16T01:45:46.642 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.60731.log: /var/log/ceph/ceph-client.admin.41665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60605.log 2024-09-16T01:45:46.642 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60731.log.gz 2024-09-16T01:45:46.642 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41665.log.gz 2024-09-16T01:45:46.642 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.67185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71601.log 2024-09-16T01:45:46.642 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.67185.log.gz 2024-09-16T01:45:46.643 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85275.log 2024-09-16T01:45:46.643 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.60605.log: /var/log/ceph/ceph-client.admin.71601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76158.log 2024-09-16T01:45:46.643 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60605.log.gz 2024-09-16T01:45:46.643 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.85275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53385.log 2024-09-16T01:45:46.643 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.85275.log.gz 2024-09-16T01:45:46.643 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.76158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62247.log 2024-09-16T01:45:46.643 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.76158.log.gz 2024-09-16T01:45:46.643 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42306.log 2024-09-16T01:45:46.644 INFO:teuthology.orchestra.run.smithi088.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.71601.log.gz 2024-09-16T01:45:46.644 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77211.log 2024-09-16T01:45:46.644 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53385.log.gz 2024-09-16T01:45:46.644 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.62247.log: /var/log/ceph/ceph-client.admin.42306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43284.log 2024-09-16T01:45:46.644 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.77211.log: gzip 89.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.42306.log.gz -- 2024-09-16T01:45:46.644 INFO:teuthology.orchestra.run.smithi088.stderr: /var/log/ceph/ceph-client.admin.45432.log 2024-09-16T01:45:46.644 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77211.log.gz 2024-09-16T01:45:46.644 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73894.log 2024-09-16T01:45:46.645 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58453.log 2024-09-16T01:45:46.645 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45432.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43284.log.gz 2024-09-16T01:45:46.645 INFO:teuthology.orchestra.run.smithi088.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.62247.log.gz 2024-09-16T01:45:46.645 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.73894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79064.log 2024-09-16T01:45:46.645 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45432.log.gz 2024-09-16T01:45:46.645 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73894.log.gz 2024-09-16T01:45:46.645 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86282.log 2024-09-16T01:45:46.645 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.58453.log: /var/log/ceph/ceph-client.admin.79064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75404.log 2024-09-16T01:45:46.646 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58453.log.gz 2024-09-16T01:45:46.646 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.86282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55614.log 2024-09-16T01:45:46.646 INFO:teuthology.orchestra.run.smithi088.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.79064.log.gz 2024-09-16T01:45:46.646 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86282.log.gz 2024-09-16T01:45:46.646 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.75404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.55177.log 2024-09-16T01:45:46.646 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75404.log.gz 2024-09-16T01:45:46.646 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67048.log 2024-09-16T01:45:46.646 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55614.log: /var/log/ceph/ceph-client.0.55177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71924.log 2024-09-16T01:45:46.647 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.55177.log.gz 2024-09-16T01:45:46.647 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55614.log.gz 2024-09-16T01:45:46.647 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.67048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54373.log 2024-09-16T01:45:46.647 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.71924.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67048.log.gz 2024-09-16T01:45:46.647 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70424.log 2024-09-16T01:45:46.647 INFO:teuthology.orchestra.run.smithi088.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71924.log.gz 2024-09-16T01:45:46.647 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63117.log 2024-09-16T01:45:46.647 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54373.log: /var/log/ceph/ceph-client.admin.70424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46287.log 2024-09-16T01:45:46.648 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54373.log.gz 2024-09-16T01:45:46.648 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.63117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83482.log 2024-09-16T01:45:46.648 INFO:teuthology.orchestra.run.smithi088.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.70424.log.gz 2024-09-16T01:45:46.648 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46287.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.63117.log.gz 2024-09-16T01:45:46.648 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65432.log 2024-09-16T01:45:46.648 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46287.log.gz 2024-09-16T01:45:46.648 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57272.log 2024-09-16T01:45:46.649 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.83482.log: gzip/var/log/ceph/ceph-client.admin.65432.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.87706.log 2024-09-16T01:45:46.649 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.57272.log: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.65432.log.gz -5 2024-09-16T01:45:46.649 INFO:teuthology.orchestra.run.smithi088.stderr: --verbose -- /var/log/ceph/ceph-client.admin.88312.log 2024-09-16T01:45:46.649 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57272.log.gz 2024-09-16T01:45:46.649 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.87706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57669.log 2024-09-16T01:45:46.649 INFO:teuthology.orchestra.run.smithi088.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.87706.log.gz 2024-09-16T01:45:46.649 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.88312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43902.log 2024-09-16T01:45:46.649 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.57669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81880.log 2024-09-16T01:45:46.650 INFO:teuthology.orchestra.run.smithi088.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.83482.log.gz 2024-09-16T01:45:46.650 INFO:teuthology.orchestra.run.smithi088.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.88312.log.gz 2024-09-16T01:45:46.650 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43902.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57669.log.gz 2024-09-16T01:45:46.650 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50163.log 2024-09-16T01:45:46.650 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.81880.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43902.log.gz 2024-09-16T01:45:46.650 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49190.log 2024-09-16T01:45:46.650 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64649.log 2024-09-16T01:45:46.650 INFO:teuthology.orchestra.run.smithi088.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50163.log.gz 2024-09-16T01:45:46.651 INFO:teuthology.orchestra.run.smithi088.stderr:gzip/var/log/ceph/ceph-client.admin.49190.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.67828.log 2024-09-16T01:45:46.651 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.64649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85317.log 2024-09-16T01:45:46.651 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64649.log.gz 2024-09-16T01:45:46.651 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.67828.log: gzip -5 90.9% -- replaced with /var/log/ceph/ceph-client.admin.49190.log.gz 2024-09-16T01:45:46.651 INFO:teuthology.orchestra.run.smithi088.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.81880.log.gz 2024-09-16T01:45:46.651 INFO:teuthology.orchestra.run.smithi088.stderr: --verbose -- /var/log/ceph/ceph-client.admin.60292.log 2024-09-16T01:45:46.651 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.85317.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67828.log.gz 2024-09-16T01:45:46.651 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42390.log 2024-09-16T01:45:46.652 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.85317.log.gz 2024-09-16T01:45:46.652 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.60292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49758.log 2024-09-16T01:45:46.652 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.42390.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60292.log.gz 2024-09-16T01:45:46.652 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65707.log 2024-09-16T01:45:46.652 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47594.log 2024-09-16T01:45:46.652 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49758.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42390.log.gz 2024-09-16T01:45:46.652 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.65707.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49758.log.gz 2024-09-16T01:45:46.652 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52988.log 2024-09-16T01:45:46.653 INFO:teuthology.orchestra.run.smithi088.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.65707.log.gz 2024-09-16T01:45:46.653 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76902.log 2024-09-16T01:45:46.653 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.47594.log: /var/log/ceph/ceph-client.admin.52988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49524.log 2024-09-16T01:45:46.653 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47594.log.gz 2024-09-16T01:45:46.653 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.76902.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52988.log.gz 2024-09-16T01:45:46.653 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77421.log 2024-09-16T01:45:46.653 INFO:teuthology.orchestra.run.smithi088.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.76902.log.gz 2024-09-16T01:45:46.654 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46245.log 2024-09-16T01:45:46.654 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50799.log 2024-09-16T01:45:46.654 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.77421.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.49524.log.gz 2024-09-16T01:45:46.654 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78197.log 2024-09-16T01:45:46.654 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77421.log.gz 2024-09-16T01:45:46.654 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46245.log.gz 2024-09-16T01:45:46.654 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54923.log 2024-09-16T01:45:46.654 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50799.log: /var/log/ceph/ceph-client.admin.78197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77379.log 2024-09-16T01:45:46.655 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50799.log.gz 2024-09-16T01:45:46.655 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78197.log.gz 2024-09-16T01:45:46.655 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47964.log 2024-09-16T01:45:46.655 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.77379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52617.log 2024-09-16T01:45:46.655 INFO:teuthology.orchestra.run.smithi088.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.54923.log.gz 2024-09-16T01:45:46.655 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77379.log.gz 2024-09-16T01:45:46.655 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.47964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57891.log 2024-09-16T01:45:46.655 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47964.log.gz 2024-09-16T01:45:46.656 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68395.log 2024-09-16T01:45:46.656 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52617.log: /var/log/ceph/ceph-client.admin.57891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78375.log 2024-09-16T01:45:46.656 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52617.log.gz 2024-09-16T01:45:46.656 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.68395.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57891.log.gz 2024-09-16T01:45:46.656 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79950.log 2024-09-16T01:45:46.656 INFO:teuthology.orchestra.run.smithi088.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.68395.log.gz 2024-09-16T01:45:46.656 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.78375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.82368.log 2024-09-16T01:45:46.656 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.79950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60689.log 2024-09-16T01:45:46.657 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78375.log.gz 2024-09-16T01:45:46.657 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.79950.log.gz 2024-09-16T01:45:46.657 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.82368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83998.log 2024-09-16T01:45:46.657 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.82368.log.gz 2024-09-16T01:45:46.657 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46713.log 2024-09-16T01:45:46.657 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.60689.log: /var/log/ceph/ceph-client.admin.83998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74282.log 2024-09-16T01:45:46.657 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60689.log.gz 2024-09-16T01:45:46.657 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.83998.log.gz 2024-09-16T01:45:46.658 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55656.log 2024-09-16T01:45:46.658 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54247.log 2024-09-16T01:45:46.658 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.74282.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46713.log.gz 2024-09-16T01:45:46.658 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55656.log: gzip 88.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.74282.log.gz --verbose 2024-09-16T01:45:46.658 INFO:teuthology.orchestra.run.smithi088.stderr: -- /var/log/ceph/ceph.audit.log 2024-09-16T01:45:46.658 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42081.log 2024-09-16T01:45:46.658 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54247.log: 91.3%/var/log/ceph/ceph.audit.log: -- replaced with /var/log/ceph/ceph-client.admin.55656.log.gz 2024-09-16T01:45:46.658 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44028.log 2024-09-16T01:45:46.659 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54247.log.gz 2024-09-16T01:45:46.659 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51530.log 2024-09-16T01:45:46.659 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.42081.log: /var/log/ceph/ceph-client.admin.44028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79758.log 2024-09-16T01:45:46.659 INFO:teuthology.orchestra.run.smithi088.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42081.log.gz 2024-09-16T01:45:46.659 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44028.log.gz 2024-09-16T01:45:46.659 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46841.log 2024-09-16T01:45:46.659 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76958.log 2024-09-16T01:45:46.659 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.79758.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51530.log.gz 2024-09-16T01:45:46.660 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63820.log 2024-09-16T01:45:46.660 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46841.log.gz 2024-09-16T01:45:46.660 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.76958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60040.log 2024-09-16T01:45:46.660 INFO:teuthology.orchestra.run.smithi088.stderr:gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.76958.log.gz 2024-09-16T01:45:46.660 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.63820.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.86854.log 2024-09-16T01:45:46.660 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.60040.log: gzip 89.9% -5 --verbose -- /var/log/ceph/ceph-client.admin.48103.log 2024-09-16T01:45:46.660 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63820.log.gz 2024-09-16T01:45:46.660 INFO:teuthology.orchestra.run.smithi088.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.79758.log.gz 2024-09-16T01:45:46.661 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60040.log.gz 2024-09-16T01:45:46.661 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.86854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85981.log 2024-09-16T01:45:46.661 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48103.log: 89.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T01:45:46.661 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86854.log.gz 2024-09-16T01:45:46.661 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82954.log 2024-09-16T01:45:46.661 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48103.log.gz 2024-09-16T01:45:46.661 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.85981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75740.log 2024-09-16T01:45:46.662 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.85981.log.gz 2024-09-16T01:45:46.662 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48126.log 2024-09-16T01:45:46.662 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.82954.log: /var/log/ceph/ceph-client.admin.75740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71031.log 2024-09-16T01:45:46.662 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.82954.log.gz 2024-09-16T01:45:46.662 INFO:teuthology.orchestra.run.smithi088.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.75740.log.gz 2024-09-16T01:45:46.662 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44600.log 2024-09-16T01:45:46.662 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84227.log 2024-09-16T01:45:46.662 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.71031.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48126.log.gz 2024-09-16T01:45:46.663 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.44600.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.44600.log.gz 2024-09-16T01:45:46.663 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86228.log 2024-09-16T01:45:46.663 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71031.log.gz 2024-09-16T01:45:46.663 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77010.log 2024-09-16T01:45:46.663 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.84227.log: /var/log/ceph/ceph-client.admin.86228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70387.log 2024-09-16T01:45:46.663 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.84227.log.gz 2024-09-16T01:45:46.663 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.77010.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.86228.log.gz 2024-09-16T01:45:46.663 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75582.log 2024-09-16T01:45:46.664 INFO:teuthology.orchestra.run.smithi088.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.77010.log.gz 2024-09-16T01:45:46.664 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.70387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78491.log 2024-09-16T01:45:46.664 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.75582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45492.log 2024-09-16T01:45:46.664 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.75582.log.gz 2024-09-16T01:45:46.664 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.78491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56791.log 2024-09-16T01:45:46.664 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.82372.log 2024-09-16T01:45:46.664 INFO:teuthology.orchestra.run.smithi088.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78491.log.gz 2024-09-16T01:45:46.664 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45492.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.70387.log.gz 2024-09-16T01:45:46.665 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.56791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83029.log 2024-09-16T01:45:46.665 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.56791.log.gz 2024-09-16T01:45:46.665 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45492.log.gz 2024-09-16T01:45:46.665 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.82372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75016.log 2024-09-16T01:45:46.665 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.82372.log.gz 2024-09-16T01:45:46.665 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.83029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60250.log 2024-09-16T01:45:46.665 INFO:teuthology.orchestra.run.smithi088.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.83029.log.gz 2024-09-16T01:45:46.665 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.75016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64726.log 2024-09-16T01:45:46.666 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54828.log 2024-09-16T01:45:46.666 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.60250.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.75016.log.gz 2024-09-16T01:45:46.666 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.64726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.64518.log 2024-09-16T01:45:46.666 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60250.log.gz 2024-09-16T01:45:46.666 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64726.log.gz 2024-09-16T01:45:46.666 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85061.log 2024-09-16T01:45:46.666 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54828.log: /var/log/ceph/ceph-client.0.64518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64066.log 2024-09-16T01:45:46.667 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.64518.log.gz 2024-09-16T01:45:46.667 INFO:teuthology.orchestra.run.smithi088.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54828.log.gz 2024-09-16T01:45:46.667 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.85061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77622.log 2024-09-16T01:45:46.667 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.64066.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.85061.log.gz 2024-09-16T01:45:46.667 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88060.log 2024-09-16T01:45:46.667 INFO:teuthology.orchestra.run.smithi088.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.64066.log.gz 2024-09-16T01:45:46.667 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.77622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72177.log 2024-09-16T01:45:46.667 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51446.log 2024-09-16T01:45:46.668 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.88060.log: /var/log/ceph/ceph-client.admin.72177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87199.log 2024-09-16T01:45:46.668 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.88060.log.gz 2024-09-16T01:45:46.668 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.72177.log.gz 2024-09-16T01:45:46.668 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75236.log 2024-09-16T01:45:46.668 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.87199.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.77622.log.gz 88.3% 2024-09-16T01:45:46.668 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51446.log.gz 2024-09-16T01:45:46.668 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84185.log 2024-09-16T01:45:46.668 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77987.log 2024-09-16T01:45:46.669 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.87199.log.gz 2024-09-16T01:45:46.669 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.75236.log: /var/log/ceph/ceph-client.admin.84185.log: gzip 88.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.75236.log.gz --verbose 2024-09-16T01:45:46.669 INFO:teuthology.orchestra.run.smithi088.stderr: -- /var/log/ceph/ceph-client.admin.65979.log 2024-09-16T01:45:46.669 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.84185.log.gz 2024-09-16T01:45:46.669 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72781.log 2024-09-16T01:45:46.669 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.77987.log: /var/log/ceph/ceph-client.admin.65979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65282.log 2024-09-16T01:45:46.669 INFO:teuthology.orchestra.run.smithi088.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.77987.log.gz 2024-09-16T01:45:46.670 INFO:teuthology.orchestra.run.smithi088.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.65979.log.gz 2024-09-16T01:45:46.670 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.72781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61119.log 2024-09-16T01:45:46.670 INFO:teuthology.orchestra.run.smithi088.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.72781.log.gz 2024-09-16T01:45:46.670 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72378.log 2024-09-16T01:45:46.670 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.65282.log: /var/log/ceph/ceph-client.admin.61119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49256.log 2024-09-16T01:45:46.670 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65282.log.gz 2024-09-16T01:45:46.670 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.72378.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61119.log.gz 2024-09-16T01:45:46.670 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46491.log 2024-09-16T01:45:46.670 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78627.log 2024-09-16T01:45:46.671 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49256.log.gz 2024-09-16T01:45:46.671 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78669.log 2024-09-16T01:45:46.671 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.73015.log 2024-09-16T01:45:46.671 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.78627.log: 92.5% 89.3% -- replaced with /var/log/ceph/ceph-client.admin.72378.log.gz -- replaced with /var/log/ceph/ceph-client.admin.46491.log.gz 2024-09-16T01:45:46.671 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.671 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80058.log 2024-09-16T01:45:46.671 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.78669.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.78627.log.gz 2024-09-16T01:45:46.671 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77659.log 2024-09-16T01:45:46.672 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.73015.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.73015.log.gz 2024-09-16T01:45:46.672 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.80058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50500.log 2024-09-16T01:45:46.672 INFO:teuthology.orchestra.run.smithi088.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.78669.log.gz 2024-09-16T01:45:46.672 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.80058.log.gz 2024-09-16T01:45:46.672 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.77659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74576.log 2024-09-16T01:45:46.672 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53072.log 2024-09-16T01:45:46.672 INFO:teuthology.orchestra.run.smithi088.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.77659.log.gz 2024-09-16T01:45:46.672 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.74576.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50500.log.gzgzip 2024-09-16T01:45:46.673 INFO:teuthology.orchestra.run.smithi088.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.70574.log 2024-09-16T01:45:46.673 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74576.log.gz 2024-09-16T01:45:46.673 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48963.log 2024-09-16T01:45:46.673 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53072.log.gz 2024-09-16T01:45:46.673 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43860.log 2024-09-16T01:45:46.673 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.70574.log: /var/log/ceph/ceph-client.admin.48963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56795.log 2024-09-16T01:45:46.673 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.70574.log.gz 2024-09-16T01:45:46.673 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43860.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48963.log.gz 2024-09-16T01:45:46.674 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71073.log 2024-09-16T01:45:46.674 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49935.log 2024-09-16T01:45:46.674 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43860.log.gz 2024-09-16T01:45:46.674 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.56795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.56795.log.gz 2024-09-16T01:45:46.674 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.71073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70691.log 2024-09-16T01:45:46.674 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71073.log.gz 2024-09-16T01:45:46.674 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88396.log 2024-09-16T01:45:46.674 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49935.log.gz 2024-09-16T01:45:46.675 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74848.log 2024-09-16T01:45:46.675 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.70691.log: /var/log/ceph/ceph-client.admin.88396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74908.log 2024-09-16T01:45:46.675 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70691.log.gz 2024-09-16T01:45:46.675 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.74848.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.88396.log.gz 2024-09-16T01:45:46.675 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.80250.log 2024-09-16T01:45:46.675 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67302.log 2024-09-16T01:45:46.675 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.74848.log.gz 2024-09-16T01:45:46.675 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.74908.log: /var/log/ceph/ceph-client.0.80250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57585.log 2024-09-16T01:45:46.675 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.80250.log.gz 2024-09-16T01:45:46.676 INFO:teuthology.orchestra.run.smithi088.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.74908.log.gz 2024-09-16T01:45:46.676 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5/var/log/ceph/ceph-client.admin.67302.log: --verbose -- /var/log/ceph/ceph-client.admin.46978.log 2024-09-16T01:45:46.676 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.57585.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30355.log 2024-09-16T01:45:46.676 INFO:teuthology.orchestra.run.smithi088.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.67302.log.gz 2024-09-16T01:45:46.676 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57585.log.gz 2024-09-16T01:45:46.676 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59827.log 2024-09-16T01:45:46.676 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46978.log.gz 2024-09-16T01:45:46.676 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76065.log 2024-09-16T01:45:46.677 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph.tmp-client.admin.30355.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30355.log.gz 2024-09-16T01:45:46.677 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.59827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88228.log 2024-09-16T01:45:46.677 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.76065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73254.log 2024-09-16T01:45:46.677 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.88228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45390.log 2024-09-16T01:45:46.677 INFO:teuthology.orchestra.run.smithi088.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.59827.log.gz 2024-09-16T01:45:46.677 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.88228.log.gz 2024-09-16T01:45:46.677 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.73254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87622.log 2024-09-16T01:45:46.677 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57146.log 2024-09-16T01:45:46.678 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73254.log.gz 2024-09-16T01:45:46.678 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45390.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.76065.log.gz 2024-09-16T01:45:46.678 INFO:teuthology.orchestra.run.smithi088.stderr:gzip/var/log/ceph/ceph-client.admin.87622.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.79904.log 2024-09-16T01:45:46.678 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45390.log.gz 2024-09-16T01:45:46.678 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80345.log 2024-09-16T01:45:46.678 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.57146.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.87622.log.gz 2024-09-16T01:45:46.678 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.79904.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57146.log.gz 2024-09-16T01:45:46.678 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42950.log 2024-09-16T01:45:46.679 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.79904.log.gz 2024-09-16T01:45:46.679 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84880.log 2024-09-16T01:45:46.679 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.80345.log: /var/log/ceph/ceph-client.admin.42950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74805.log 2024-09-16T01:45:46.679 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.80345.log.gz 2024-09-16T01:45:46.679 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42950.log.gz 2024-09-16T01:45:46.679 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.84880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86989.log 2024-09-16T01:45:46.679 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.84880.log.gz 2024-09-16T01:45:46.680 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85509.log 2024-09-16T01:45:46.680 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.74805.log: /var/log/ceph/ceph-client.admin.86989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60437.log 2024-09-16T01:45:46.680 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.74805.log.gz 2024-09-16T01:45:46.680 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.85509.log: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82114.log 2024-09-16T01:45:46.680 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.86989.log.gz 2024-09-16T01:45:46.680 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.85509.log.gz 2024-09-16T01:45:46.680 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.60437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67509.log 2024-09-16T01:45:46.680 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60437.log.gz 2024-09-16T01:45:46.681 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50885.log 2024-09-16T01:45:46.681 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.82114.log: /var/log/ceph/ceph-client.admin.67509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77271.log 2024-09-16T01:45:46.681 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67509.log.gz 2024-09-16T01:45:46.681 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64122.log 2024-09-16T01:45:46.681 INFO:teuthology.orchestra.run.smithi088.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.82114.log.gz 2024-09-16T01:45:46.681 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51830.log 2024-09-16T01:45:46.681 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.77271.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50885.log.gz 2024-09-16T01:45:46.681 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.64122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53648.log 2024-09-16T01:45:46.682 INFO:teuthology.orchestra.run.smithi088.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.77271.log.gz 2024-09-16T01:45:46.682 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64122.log.gz 2024-09-16T01:45:46.682 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.51830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74483.log 2024-09-16T01:45:46.682 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.51830.log.gz 2024-09-16T01:45:46.682 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49851.log 2024-09-16T01:45:46.682 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58137.log 2024-09-16T01:45:46.682 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.74483.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53648.log.gz 2024-09-16T01:45:46.683 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48041.log 2024-09-16T01:45:46.683 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49851.log.gz 2024-09-16T01:45:46.683 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.58137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59700.log 2024-09-16T01:45:46.683 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58706.log 2024-09-16T01:45:46.683 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48041.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58137.log.gz 2024-09-16T01:45:46.683 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.59700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43285.log 2024-09-16T01:45:46.683 INFO:teuthology.orchestra.run.smithi088.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48041.log.gz 2024-09-16T01:45:46.683 INFO:teuthology.orchestra.run.smithi088.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.74483.log.gz 2024-09-16T01:45:46.684 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59700.log.gz 2024-09-16T01:45:46.684 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.58706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80683.log 2024-09-16T01:45:46.684 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81796.log 2024-09-16T01:45:46.684 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.80683.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43285.log.gz 2024-09-16T01:45:46.684 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51638.log 2024-09-16T01:45:46.684 INFO:teuthology.orchestra.run.smithi088.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.58706.log.gz 2024-09-16T01:45:46.684 INFO:teuthology.orchestra.run.smithi088.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.80683.log.gz 2024-09-16T01:45:46.684 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.81796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71377.log 2024-09-16T01:45:46.685 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70480.log 2024-09-16T01:45:46.685 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51638.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81796.log.gz 2024-09-16T01:45:46.685 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.71377.log: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.51638.log.gz 2024-09-16T01:45:46.685 INFO:teuthology.orchestra.run.smithi088.stderr: -5 --verbose -- /var/log/ceph/ceph-client.0.66921.log 2024-09-16T01:45:46.685 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71377.log.gz 2024-09-16T01:45:46.685 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81482.log 2024-09-16T01:45:46.685 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.70480.log: /var/log/ceph/ceph-client.0.66921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60479.log 2024-09-16T01:45:46.685 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.66921.log.gz 2024-09-16T01:45:46.686 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70480.log.gz 2024-09-16T01:45:46.686 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.81482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63857.log 2024-09-16T01:45:46.686 INFO:teuthology.orchestra.run.smithi088.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.81482.log.gz 2024-09-16T01:45:46.686 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43277.log 2024-09-16T01:45:46.686 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.60479.log: /var/log/ceph/ceph-client.admin.63857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.81152.log 2024-09-16T01:45:46.686 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60479.log.gz 2024-09-16T01:45:46.686 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43277.log: gzip -5 89.7% -- replaced with /var/log/ceph/ceph-client.admin.63857.log.gz 2024-09-16T01:45:46.686 INFO:teuthology.orchestra.run.smithi088.stderr: --verbose -- /var/log/ceph/ceph-client.admin.83872.log 2024-09-16T01:45:46.687 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.81152.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43277.log.gz 2024-09-16T01:45:46.687 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.81152.log.gz 2024-09-16T01:45:46.687 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63154.log 2024-09-16T01:45:46.687 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60907.log 2024-09-16T01:45:46.687 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.83872.log: gzip/var/log/ceph/ceph-client.admin.63154.log: -5 --verbose -- /var/log/ceph/ceph-client.0.49444.log 2024-09-16T01:45:46.687 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.83872.log.gz 2024-09-16T01:45:46.687 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.60907.log: gzip 89.5% -- replaced with /var/log/ceph/ceph-client.admin.63154.log.gz 2024-09-16T01:45:46.687 INFO:teuthology.orchestra.run.smithi088.stderr: -5 --verbose -- /var/log/ceph/ceph-client.0.58960.log 2024-09-16T01:45:46.688 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60907.log.gz 2024-09-16T01:45:46.688 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.49444.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.49444.log.gz 2024-09-16T01:45:46.688 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62784.log 2024-09-16T01:45:46.688 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.58960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74964.log 2024-09-16T01:45:46.688 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.58960.log.gz 2024-09-16T01:45:46.688 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.62784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50595.log 2024-09-16T01:45:46.688 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.74964.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.62784.log.gz 2024-09-16T01:45:46.688 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58309.log 2024-09-16T01:45:46.689 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74964.log.gz 2024-09-16T01:45:46.689 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44070.log 2024-09-16T01:45:46.689 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50595.log: /var/log/ceph/ceph-client.admin.58309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57971.log 2024-09-16T01:45:46.689 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50595.log.gz 2024-09-16T01:45:46.689 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44070.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58309.log.gz 2024-09-16T01:45:46.689 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83417.log 2024-09-16T01:45:46.689 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64222.log 2024-09-16T01:45:46.690 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44070.log.gz 2024-09-16T01:45:46.690 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.57971.log: /var/log/ceph/ceph-client.admin.83417.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57971.log.gz 2024-09-16T01:45:46.690 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51278.log 2024-09-16T01:45:46.690 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.83417.log.gz 2024-09-16T01:45:46.690 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68176.log 2024-09-16T01:45:46.690 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.64222.log: /var/log/ceph/ceph-client.admin.51278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88522.log 2024-09-16T01:45:46.690 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64222.log.gz 2024-09-16T01:45:46.690 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.68176.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51278.log.gz 2024-09-16T01:45:46.691 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56333.log 2024-09-16T01:45:46.691 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68176.log.gz 2024-09-16T01:45:46.691 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82576.log 2024-09-16T01:45:46.691 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.88522.log: /var/log/ceph/ceph-client.admin.56333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83777.log 2024-09-16T01:45:46.691 INFO:teuthology.orchestra.run.smithi088.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.88522.log.gz 2024-09-16T01:45:46.691 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.82576.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56333.log.gz 2024-09-16T01:45:46.691 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47234.log 2024-09-16T01:45:46.691 INFO:teuthology.orchestra.run.smithi088.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.82576.log.gz 2024-09-16T01:45:46.692 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75971.log 2024-09-16T01:45:46.692 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.83777.log: /var/log/ceph/ceph-client.admin.47234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68858.log 2024-09-16T01:45:46.692 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.83777.log.gz 2024-09-16T01:45:46.692 INFO:teuthology.orchestra.run.smithi088.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47234.log.gz 2024-09-16T01:45:46.692 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.75971.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-09-16T01:45:46.692 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75971.log.gz 2024-09-16T01:45:46.692 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69799.log 2024-09-16T01:45:46.692 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.68858.log: /var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51064.log 2024-09-16T01:45:46.693 INFO:teuthology.orchestra.run.smithi088.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.68858.log.gz 2024-09-16T01:45:46.693 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.69799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86143.log 2024-09-16T01:45:46.693 INFO:teuthology.orchestra.run.smithi088.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.69799.log.gz 2024-09-16T01:45:46.693 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85359.log 2024-09-16T01:45:46.693 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51236.log 2024-09-16T01:45:46.693 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.86143.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51064.log.gz 2024-09-16T01:45:46.693 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.85359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54163.log 2024-09-16T01:45:46.693 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51236.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.85359.log.gz 2024-09-16T01:45:46.694 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66621.log 2024-09-16T01:45:46.694 INFO:teuthology.orchestra.run.smithi088.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.86143.log.gz 2024-09-16T01:45:46.694 INFO:teuthology.orchestra.run.smithi088.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51236.log.gz 2024-09-16T01:45:46.694 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72026.log 2024-09-16T01:45:46.694 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54163.log.gz 2024-09-16T01:45:46.694 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46142.log 2024-09-16T01:45:46.694 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.66621.log: /var/log/ceph/ceph-client.admin.72026.log: gzip 89.2% -5 --verbose -- /var/log/ceph/ceph-client.admin.62932.log 2024-09-16T01:45:46.694 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.66621.log.gz 2024-09-16T01:45:46.695 INFO:teuthology.orchestra.run.smithi088.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.72026.log.gz 2024-09-16T01:45:46.695 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69681.log 2024-09-16T01:45:46.695 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.46142.log.gz 2024-09-16T01:45:46.695 INFO:teuthology.orchestra.run.smithi088.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.76385.log 2024-09-16T01:45:46.695 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.62932.log: /var/log/ceph/ceph-client.admin.69681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62074.log 2024-09-16T01:45:46.695 INFO:teuthology.orchestra.run.smithi088.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.62932.log.gz 2024-09-16T01:45:46.695 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69681.log.gz 2024-09-16T01:45:46.695 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.76385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71848.log 2024-09-16T01:45:46.696 INFO:teuthology.orchestra.run.smithi088.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.76385.log.gz 2024-09-16T01:45:46.696 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55430.log 2024-09-16T01:45:46.696 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.62074.log: /var/log/ceph/ceph-client.admin.71848.log: gzip -5 88.3% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.62074.log.gz /var/log/ceph/ceph-client.admin.74188.log 2024-09-16T01:45:46.696 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.696 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.71848.log.gz 2024-09-16T01:45:46.696 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43275.log 2024-09-16T01:45:46.696 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78047.log 2024-09-16T01:45:46.696 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55430.log.gz 2024-09-16T01:45:46.697 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.74188.log: /var/log/ceph/ceph-client.admin.43275.log: gzip -5 89.2% -- replaced with /var/log/ceph/ceph-client.admin.74188.log.gz 2024-09-16T01:45:46.697 INFO:teuthology.orchestra.run.smithi088.stderr: --verbose -- /var/log/ceph/ceph-client.admin.70923.log 2024-09-16T01:45:46.697 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43275.log.gz 2024-09-16T01:45:46.697 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.78047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71269.log 2024-09-16T01:45:46.697 INFO:teuthology.orchestra.run.smithi088.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.78047.log.gz 2024-09-16T01:45:46.697 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.70923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54331.log 2024-09-16T01:45:46.697 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.70923.log.gz 2024-09-16T01:45:46.697 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.71269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73800.log 2024-09-16T01:45:46.698 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78333.log 2024-09-16T01:45:46.698 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54331.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.71269.log.gz 2024-09-16T01:45:46.698 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.73800.log: gzip -5 --verbose 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54331.log.gz 2024-09-16T01:45:46.698 INFO:teuthology.orchestra.run.smithi088.stderr: -- /var/log/ceph/ceph-client.admin.74072.log 2024-09-16T01:45:46.698 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73800.log.gz 2024-09-16T01:45:46.698 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.78333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67401.log 2024-09-16T01:45:46.698 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.78333.log.gz 2024-09-16T01:45:46.698 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53894.log 2024-09-16T01:45:46.699 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.74072.log: /var/log/ceph/ceph-client.admin.67401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85149.log 2024-09-16T01:45:46.699 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74072.log.gz 2024-09-16T01:45:46.699 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67401.log.gz 2024-09-16T01:45:46.699 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.61892.log 2024-09-16T01:45:46.699 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58095.log 2024-09-16T01:45:46.699 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.85149.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53894.log.gz 2024-09-16T01:45:46.699 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.61892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57011.log 2024-09-16T01:45:46.699 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.61892.log.gz 2024-09-16T01:45:46.700 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.85149.log.gz 2024-09-16T01:45:46.700 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55932.log/var/log/ceph/ceph-client.admin.58095.log: 2024-09-16T01:45:46.700 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.57011.log: gzip 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58095.log.gz 2024-09-16T01:45:46.700 INFO:teuthology.orchestra.run.smithi088.stderr: -5 --verbose -- 88.7% /var/log/ceph/ceph-client.admin.67355.log 2024-09-16T01:45:46.700 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57011.log.gz 2024-09-16T01:45:46.700 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67965.log 2024-09-16T01:45:46.700 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.67355.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55932.log.gz 2024-09-16T01:45:46.701 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42218.log 2024-09-16T01:45:46.701 INFO:teuthology.orchestra.run.smithi088.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67355.log.gz 2024-09-16T01:45:46.701 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.67965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52858.log 2024-09-16T01:45:46.701 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67965.log.gz 2024-09-16T01:45:46.701 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.42218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82026.log 2024-09-16T01:45:46.701 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52858.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42218.log.gz 2024-09-16T01:45:46.701 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45932.log 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52858.log.gz 2024-09-16T01:45:46.701 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.702 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.82026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42684.log 2024-09-16T01:45:46.702 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45932.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.82026.log.gz 2024-09-16T01:45:46.702 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73642.log 2024-09-16T01:45:46.702 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.42684.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45932.log.gz 2024-09-16T01:45:46.702 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72584.log 2024-09-16T01:45:46.702 INFO:teuthology.orchestra.run.smithi088.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42684.log.gz 2024-09-16T01:45:46.702 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.73642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84521.log 2024-09-16T01:45:46.702 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.72584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53728.log 2024-09-16T01:45:46.703 INFO:teuthology.orchestra.run.smithi088.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.73642.log.gz 2024-09-16T01:45:46.703 INFO:teuthology.orchestra.run.smithi088.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72584.log.gz 2024-09-16T01:45:46.703 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.84521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43273.log 2024-09-16T01:45:46.703 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84521.log.gz 2024-09-16T01:45:46.703 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42726.log 2024-09-16T01:45:46.703 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53728.log.gz 2024-09-16T01:45:46.703 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60647.log 2024-09-16T01:45:46.703 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.42726.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43273.log.gz 2024-09-16T01:45:46.704 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61554.log 88.9% 2024-09-16T01:45:46.704 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42726.log.gz 2024-09-16T01:45:46.704 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.60647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49040.log 2024-09-16T01:45:46.704 INFO:teuthology.orchestra.run.smithi088.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.60647.log.gz 2024-09-16T01:45:46.704 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.61554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45222.log 2024-09-16T01:45:46.704 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44739.log 2024-09-16T01:45:46.704 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61554.log.gz 2024-09-16T01:45:46.704 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66663.log 2024-09-16T01:45:46.705 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44739.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45222.log.gz 2024-09-16T01:45:46.705 INFO:teuthology.orchestra.run.smithi088.stderr:gzip 91.2% -- replaced with /var/log/ceph/ceph-client.admin.49040.log.gz 2024-09-16T01:45:46.705 INFO:teuthology.orchestra.run.smithi088.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60563.log 2024-09-16T01:45:46.705 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.66663.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44739.log.gz 2024-09-16T01:45:46.705 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64810.log 2024-09-16T01:45:46.705 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.60563.log: gzip -5 91.1% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.66663.log.gz 2024-09-16T01:45:46.705 INFO:teuthology.orchestra.run.smithi088.stderr: /var/log/ceph/ceph-client.admin.62673.log 2024-09-16T01:45:46.705 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60563.log.gz 2024-09-16T01:45:46.706 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.64810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63783.log 2024-09-16T01:45:46.706 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.62673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63339.log 2024-09-16T01:45:46.706 INFO:teuthology.orchestra.run.smithi088.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.62673.log.gz 2024-09-16T01:45:46.706 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.63783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83150.log 2024-09-16T01:45:46.706 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.63339.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.63783.log.gz 2024-09-16T01:45:46.706 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44250.log 2024-09-16T01:45:46.706 INFO:teuthology.orchestra.run.smithi088.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.64810.log.gz 2024-09-16T01:45:46.706 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.83150.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.63339.log.gz 2024-09-16T01:45:46.706 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55825.log 2024-09-16T01:45:46.707 INFO:teuthology.orchestra.run.smithi088.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.83150.log.gz 2024-09-16T01:45:46.707 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44250.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-16T01:45:46.707 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55825.log: gzip 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44250.log.gz 2024-09-16T01:45:46.707 INFO:teuthology.orchestra.run.smithi088.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.69523.log 2024-09-16T01:45:46.707 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59042.log 92.3% -- replaced with /var/log/ceph/ceph-client.admin.55825.log.gz 2024-09-16T01:45:46.707 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.707 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.69523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57459.log 2024-09-16T01:45:46.707 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69523.log.gz 2024-09-16T01:45:46.708 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.59042.log: gzip -5 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59042.log.gz 2024-09-16T01:45:46.708 INFO:teuthology.orchestra.run.smithi088.stderr: --verbose -- /var/log/ceph/ceph-client.admin.66095.log 2024-09-16T01:45:46.708 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.57459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68681.log 2024-09-16T01:45:46.708 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.66095.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57459.log.gz 2024-09-16T01:45:46.708 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48254.log 2024-09-16T01:45:46.708 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.66095.log.gz 2024-09-16T01:45:46.708 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.68681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47108.log 2024-09-16T01:45:46.708 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68681.log.gz 2024-09-16T01:45:46.709 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57188.log 2024-09-16T01:45:46.709 INFO:teuthology.orchestra.run.smithi088.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48254.log.gz 2024-09-16T01:45:46.709 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46541.log 2024-09-16T01:45:46.709 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.47108.log: /var/log/ceph/ceph-client.admin.57188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65002.log 2024-09-16T01:45:46.709 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47108.log.gz 2024-09-16T01:45:46.709 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46541.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57188.log.gz 2024-09-16T01:45:46.709 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48814.log 2024-09-16T01:45:46.709 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46541.log.gz 2024-09-16T01:45:46.710 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.65002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82072.log 2024-09-16T01:45:46.710 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48814.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65002.log.gz 2024-09-16T01:45:46.710 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42176.log 2024-09-16T01:45:46.710 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.82072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69256.log 91.2% -- replaced with /var/log/ceph/ceph-client.admin.48814.log.gz 2024-09-16T01:45:46.710 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.710 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.82072.log.gz 2024-09-16T01:45:46.710 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.42176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69907.log 2024-09-16T01:45:46.710 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.69256.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42176.log.gz 2024-09-16T01:45:46.710 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.70099.log 2024-09-16T01:45:46.711 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.69907.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.69256.log.gz 2024-09-16T01:45:46.711 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43558.log 89.4% -- replaced with /var/log/ceph/ceph-client.admin.69907.log.gz 2024-09-16T01:45:46.711 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.711 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.70099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.78923.log 2024-09-16T01:45:46.711 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.70099.log.gz 2024-09-16T01:45:46.711 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71419.log 2024-09-16T01:45:46.711 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.78923.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43558.log.gz 2024-09-16T01:45:46.711 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.78923.log.gz 2024-09-16T01:45:46.712 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71536.log 2024-09-16T01:45:46.712 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58772.log 2024-09-16T01:45:46.712 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.71419.log: /var/log/ceph/ceph-client.admin.71536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63524.log 2024-09-16T01:45:46.712 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.71419.log.gz 2024-09-16T01:45:46.712 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.71536.log.gz 2024-09-16T01:45:46.712 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63228.log 2024-09-16T01:45:46.712 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.58772.log: /var/log/ceph/ceph-client.admin.63524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57796.log 2024-09-16T01:45:46.713 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58772.log.gz 2024-09-16T01:45:46.713 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.63228.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.63524.log.gz 2024-09-16T01:45:46.713 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48468.log 2024-09-16T01:45:46.713 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.57796.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.63228.log.gz 2024-09-16T01:45:46.713 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73684.log 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57796.log.gz 2024-09-16T01:45:46.713 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.713 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72135.log 2024-09-16T01:45:46.713 INFO:teuthology.orchestra.run.smithi088.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48468.log.gz 2024-09-16T01:45:46.714 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.73684.log: gzip -5 88.6% --verbose -- /var/log/ceph/ceph-client.admin.76781.log 2024-09-16T01:45:46.714 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.73684.log.gz 2024-09-16T01:45:46.714 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.72135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77599.log 2024-09-16T01:45:46.714 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.72135.log.gz 2024-09-16T01:45:46.714 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.76781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64903.log 2024-09-16T01:45:46.714 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.76781.log.gz 2024-09-16T01:45:46.714 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.77599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75058.log 88.7% 2024-09-16T01:45:46.714 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.77599.log.gz 2024-09-16T01:45:46.714 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.64903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58537.log 2024-09-16T01:45:46.715 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64903.log.gz/var/log/ceph/ceph-client.admin.75058.log: 2024-09-16T01:45:46.715 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50061.log 2024-09-16T01:45:46.715 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75058.log.gz 2024-09-16T01:45:46.715 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.58537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85191.log 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58537.log.gz 2024-09-16T01:45:46.715 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.715 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.64522.log 2024-09-16T01:45:46.715 INFO:teuthology.orchestra.run.smithi088.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50061.log.gz 2024-09-16T01:45:46.716 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.85191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62524.log 2024-09-16T01:45:46.716 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.64522.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.85191.log.gz 2024-09-16T01:45:46.716 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.64522.log.gz 2024-09-16T01:45:46.716 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77809.log 2024-09-16T01:45:46.716 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70756.log 2024-09-16T01:45:46.716 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.62524.log: /var/log/ceph/ceph-client.admin.77809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61256.log 2024-09-16T01:45:46.716 INFO:teuthology.orchestra.run.smithi088.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.62524.log.gz 2024-09-16T01:45:46.716 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.70756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80646.log 2024-09-16T01:45:46.717 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77809.log.gz 2024-09-16T01:45:46.717 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52129.log 2024-09-16T01:45:46.717 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.61256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61163.log 2024-09-16T01:45:46.717 INFO:teuthology.orchestra.run.smithi088.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61256.log.gz 2024-09-16T01:45:46.717 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.80646.log: gzip/var/log/ceph/ceph-client.admin.52129.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.49893.log 2024-09-16T01:45:46.717 INFO:teuthology.orchestra.run.smithi088.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.70756.log.gz 2024-09-16T01:45:46.717 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.61163.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52129.log.gz 2024-09-16T01:45:46.717 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79323.log 2024-09-16T01:45:46.718 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61163.log.gz 2024-09-16T01:45:46.718 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55837.log 2024-09-16T01:45:46.718 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.79323.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49893.log.gz 2024-09-16T01:45:46.718 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81973.log 2024-09-16T01:45:46.718 INFO:teuthology.orchestra.run.smithi088.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.80646.log.gz 2024-09-16T01:45:46.718 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73338.log 2024-09-16T01:45:46.718 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55837.log: /var/log/ceph/ceph-client.admin.81973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43520.log 2024-09-16T01:45:46.719 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.73338.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.55837.log.gz 2024-09-16T01:45:46.719 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81973.log.gz 2024-09-16T01:45:46.719 INFO:teuthology.orchestra.run.smithi088.stderr:gzip 92.4% -- replaced with /var/log/ceph/ceph-client.admin.79323.log.gz 2024-09-16T01:45:46.719 INFO:teuthology.orchestra.run.smithi088.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42600.log 2024-09-16T01:45:46.719 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57230.log 2024-09-16T01:45:46.719 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43520.log.gz/var/log/ceph/ceph-client.admin.42600.log: 2024-09-16T01:45:46.719 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55473.log 2024-09-16T01:45:46.719 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42600.log.gz 2024-09-16T01:45:46.720 INFO:teuthology.orchestra.run.smithi088.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.73338.log.gz 2024-09-16T01:45:46.720 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76212.log 2024-09-16T01:45:46.720 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.57230.log: /var/log/ceph/ceph-client.admin.55473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61704.log 2024-09-16T01:45:46.720 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57230.log.gz 2024-09-16T01:45:46.720 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.76212.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55473.log.gz 2024-09-16T01:45:46.720 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-16T01:45:46.720 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.76212.log.gz 2024-09-16T01:45:46.720 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.61704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60166.log 2024-09-16T01:45:46.721 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.2.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61704.log.gz 2024-09-16T01:45:46.721 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42558.log 2024-09-16T01:45:46.721 INFO:teuthology.orchestra.run.smithi088.stderr:gzip/var/log/ceph/ceph-client.admin.60166.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.61075.log 2024-09-16T01:45:46.721 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.42558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42945.log 2024-09-16T01:45:46.721 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60166.log.gz 2024-09-16T01:45:46.721 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42558.log.gz/var/log/ceph/ceph-client.admin.61075.log: 2024-09-16T01:45:46.721 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62561.log 2024-09-16T01:45:46.721 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2%/var/log/ceph/ceph-client.admin.42945.log: -- replaced with /var/log/ceph/ceph-client.admin.61075.log.gz 2024-09-16T01:45:46.722 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53343.log 2024-09-16T01:45:46.722 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42945.log.gz 2024-09-16T01:45:46.722 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.62561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57711.log 2024-09-16T01:45:46.722 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53343.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.62561.log.gz 2024-09-16T01:45:46.722 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.54711.log 2024-09-16T01:45:46.722 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1%/var/log/ceph/ceph-client.admin.57711.log: -- replaced with /var/log/ceph/ceph-client.admin.53343.log.gz 2024-09-16T01:45:46.722 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85443.log 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57711.log.gz 2024-09-16T01:45:46.722 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.723 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.54711.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.54711.log.gz 2024-09-16T01:45:46.723 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80856.log 2024-09-16T01:45:46.723 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5/var/log/ceph/ceph-client.admin.85443.log: --verbose -- /var/log/ceph/ceph-client.admin.88562.log 2024-09-16T01:45:46.723 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.80856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70733.log 2024-09-16T01:45:46.723 INFO:teuthology.orchestra.run.smithi088.stderr: 91.2% 88.3% -- replaced with /var/log/ceph/ceph-client.admin.85443.log.gz -- replaced with /var/log/ceph/ceph-client.admin.80856.log.gz 2024-09-16T01:45:46.723 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.723 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.88562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49622.log 2024-09-16T01:45:46.723 INFO:teuthology.orchestra.run.smithi088.stderr: 89.5%/var/log/ceph/ceph-client.admin.70733.log: -- replaced with /var/log/ceph/ceph-client.admin.88562.log.gz 2024-09-16T01:45:46.724 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84311.log 2024-09-16T01:45:46.724 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70733.log.gz/var/log/ceph/ceph-client.admin.49622.log: 2024-09-16T01:45:46.724 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76102.log 2024-09-16T01:45:46.724 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.84311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84101.log 2024-09-16T01:45:46.724 INFO:teuthology.orchestra.run.smithi088.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.49622.log.gz 2024-09-16T01:45:46.724 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.84311.log.gz 2024-09-16T01:45:46.724 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.76102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80387.log 2024-09-16T01:45:46.725 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.84101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43562.log 2024-09-16T01:45:46.725 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.80387.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84101.log.gz 2024-09-16T01:45:46.725 INFO:teuthology.orchestra.run.smithi088.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.76102.log.gz 2024-09-16T01:45:46.725 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42947.log 2024-09-16T01:45:46.725 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43562.log: gzip 90.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.80387.log.gz 2024-09-16T01:45:46.725 INFO:teuthology.orchestra.run.smithi088.stderr: --verbose -- /var/log/ceph/ceph-client.admin.74871.log 2024-09-16T01:45:46.725 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43562.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84437.log 2024-09-16T01:45:46.725 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.726 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.42947.log: /var/log/ceph/ceph-client.admin.74871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74712.log 2024-09-16T01:45:46.726 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.84437.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42947.log.gz 2024-09-16T01:45:46.726 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.84437.log.gz 2024-09-16T01:45:46.726 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71578.log 2024-09-16T01:45:46.726 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5/var/log/ceph/ceph-client.admin.74712.log: --verbose -- /var/log/ceph/ceph-client.admin.42348.log 2024-09-16T01:45:46.726 INFO:teuthology.orchestra.run.smithi088.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.74871.log.gz 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74712.log.gz 2024-09-16T01:45:46.726 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.727 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.71578.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.71578.log.gz 2024-09-16T01:45:46.727 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47192.log 2024-09-16T01:45:46.727 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.42348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67443.log 2024-09-16T01:45:46.727 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.47192.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42348.log.gz 2024-09-16T01:45:46.727 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71966.log 2024-09-16T01:45:46.727 INFO:teuthology.orchestra.run.smithi088.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47192.log.gz 2024-09-16T01:45:46.727 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.67443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72082.log 2024-09-16T01:45:46.727 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.71966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75624.log 2024-09-16T01:45:46.728 INFO:teuthology.orchestra.run.smithi088.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.67443.log.gz 2024-09-16T01:45:46.728 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.71966.log.gz 2024-09-16T01:45:46.728 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.72082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.45043.log 2024-09-16T01:45:46.728 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.75624.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84616.log 2024-09-16T01:45:46.728 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.72082.log.gz 2024-09-16T01:45:46.728 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.75624.log.gz 2024-09-16T01:45:46.728 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.2.45043.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.2.45043.log.gz 2024-09-16T01:45:46.728 INFO:teuthology.orchestra.run.smithi088.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.66386.log 2024-09-16T01:45:46.729 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.84616.log: gzip -5 --verbose -- 88.0% /var/log/ceph/ceph-client.admin.63080.log 2024-09-16T01:45:46.729 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.84616.log.gz 2024-09-16T01:45:46.729 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.66386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45180.log 2024-09-16T01:45:46.729 INFO:teuthology.orchestra.run.smithi088.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66386.log.gz 2024-09-16T01:45:46.729 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53938.log 2024-09-16T01:45:46.729 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.63080.log: /var/log/ceph/ceph-client.admin.45180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88186.log 2024-09-16T01:45:46.729 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53938.log: 89.6% 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45180.log.gz 2024-09-16T01:45:46.729 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63080.log.gz 2024-09-16T01:45:46.730 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53938.log.gz 2024-09-16T01:45:46.730 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68451.log 2024-09-16T01:45:46.730 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.88186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87259.log 2024-09-16T01:45:46.730 INFO:teuthology.orchestra.run.smithi088.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.88186.log.gz 2024-09-16T01:45:46.730 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.68451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65086.log 2024-09-16T01:45:46.731 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68451.log.gz 2024-09-16T01:45:46.731 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.87259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65549.log 2024-09-16T01:45:46.731 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87259.log.gz 2024-09-16T01:45:46.731 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.65086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59742.log 2024-09-16T01:45:46.731 INFO:teuthology.orchestra.run.smithi088.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.65086.log.gz 2024-09-16T01:45:46.731 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.65549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81838.log 2024-09-16T01:45:46.731 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65549.log.gz 2024-09-16T01:45:46.731 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.59742.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-16T01:45:46.732 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.81838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81917.log 2024-09-16T01:45:46.732 INFO:teuthology.orchestra.run.smithi088.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.59742.log.gz 2024-09-16T01:45:46.732 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81838.log.gz 2024-09-16T01:45:46.732 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71232.log 2024-09-16T01:45:46.732 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.81917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80964.log 2024-09-16T01:45:46.732 INFO:teuthology.orchestra.run.smithi088.stderr: 89.4%/var/log/ceph/ceph-client.admin.71232.log: -- replaced with /var/log/ceph/ceph-client.admin.81917.log.gz 2024-09-16T01:45:46.732 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52659.log 2024-09-16T01:45:46.732 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.80964.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.80964.log.gz 2024-09-16T01:45:46.733 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86370.log 2024-09-16T01:45:46.733 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52659.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.71232.log.gz 2024-09-16T01:45:46.733 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72715.log 2024-09-16T01:45:46.733 INFO:teuthology.orchestra.run.smithi088.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52659.log.gz 2024-09-16T01:45:46.733 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.86370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88438.log 2024-09-16T01:45:46.733 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.72715.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.86370.log.gz 2024-09-16T01:45:46.733 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72823.log 2024-09-16T01:45:46.733 INFO:teuthology.orchestra.run.smithi088.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.72715.log.gz 2024-09-16T01:45:46.734 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.88438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74460.log 2024-09-16T01:45:46.734 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.72823.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.88438.log.gz 2024-09-16T01:45:46.734 INFO:teuthology.orchestra.run.smithi088.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.72823.log.gz 2024-09-16T01:45:46.734 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62376.log 2024-09-16T01:45:46.734 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74324.log 2024-09-16T01:45:46.734 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.74460.log: /var/log/ceph/ceph-client.admin.62376.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.74460.log.gz 2024-09-16T01:45:46.734 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76013.log 2024-09-16T01:45:46.734 INFO:teuthology.orchestra.run.smithi088.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.62376.log.gz 2024-09-16T01:45:46.735 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.74324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77234.log 2024-09-16T01:45:46.735 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.76013.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.74324.log.gz 2024-09-16T01:45:46.735 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76013.log.gz 2024-09-16T01:45:46.735 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45974.log 2024-09-16T01:45:46.735 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62858.log 2024-09-16T01:45:46.735 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45974.log: /var/log/ceph/ceph-client.admin.77234.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45974.log.gz 2024-09-16T01:45:46.735 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59331.log 2024-09-16T01:45:46.735 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.62858.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.62858.log.gz 2024-09-16T01:45:46.736 INFO:teuthology.orchestra.run.smithi088.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.77234.log.gz 2024-09-16T01:45:46.736 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78155.log 2024-09-16T01:45:46.736 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.59331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71461.log 2024-09-16T01:45:46.736 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59331.log.gz 2024-09-16T01:45:46.736 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.78155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.47782.log 2024-09-16T01:45:46.736 INFO:teuthology.orchestra.run.smithi088.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.78155.log.gz 2024-09-16T01:45:46.736 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.71461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.86670.log 2024-09-16T01:45:46.736 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71461.log.gz 2024-09-16T01:45:46.737 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.47782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52509.log 2024-09-16T01:45:46.737 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.47782.log.gz 2024-09-16T01:45:46.737 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.86670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62895.log 2024-09-16T01:45:46.737 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52509.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.86670.log.gz 2024-09-16T01:45:46.737 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67125.log 2024-09-16T01:45:46.737 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52509.log.gz 2024-09-16T01:45:46.737 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.62895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84143.log 2024-09-16T01:45:46.737 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.67125.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.62895.log.gz 2024-09-16T01:45:46.738 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67125.log.gz 2024-09-16T01:45:46.738 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60827.log 2024-09-16T01:45:46.738 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.84143.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T01:45:46.738 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.84143.log.gz 2024-09-16T01:45:46.738 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.60827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66039.log 2024-09-16T01:45:46.738 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60827.log.gz 2024-09-16T01:45:46.738 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57104.log 2024-09-16T01:45:46.738 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.66039.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.66039.log.gz 2024-09-16T01:45:46.739 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77767.log 2024-09-16T01:45:46.739 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.57104.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57104.log.gz 2024-09-16T01:45:46.739 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64330.log 2024-09-16T01:45:46.739 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.77767.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.77767.log.gz 2024-09-16T01:45:46.739 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42125.log 2024-09-16T01:45:46.739 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.64330.log: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T01:45:46.739 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64330.log.gz 2024-09-16T01:45:46.740 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70532.log 2024-09-16T01:45:46.740 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.42125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48200.log 2024-09-16T01:45:46.740 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42125.log.gz 2024-09-16T01:45:46.740 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.70532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71325.log 2024-09-16T01:45:46.740 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48200.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70532.log.gz 2024-09-16T01:45:46.740 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.67701.log 2024-09-16T01:45:46.740 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48200.log.gz 2024-09-16T01:45:46.740 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.71325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56892.log 2024-09-16T01:45:46.741 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.67701.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.67701.log.gz 2024-09-16T01:45:46.741 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71325.log.gz 2024-09-16T01:45:46.741 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79674.log 2024-09-16T01:45:46.741 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.56892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69018.log 2024-09-16T01:45:46.741 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.79674.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.56892.log.gz 2024-09-16T01:45:46.741 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.79674.log.gz 2024-09-16T01:45:46.741 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58393.log 2024-09-16T01:45:46.741 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.69018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59457.log 2024-09-16T01:45:46.742 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69018.log.gz 2024-09-16T01:45:46.742 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.58393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77945.log 2024-09-16T01:45:46.742 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58393.log.gz 2024-09-16T01:45:46.742 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.59457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50416.log 2024-09-16T01:45:46.742 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.77945.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59457.log.gz 2024-09-16T01:45:46.742 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82912.log 2024-09-16T01:45:46.742 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77945.log.gz 2024-09-16T01:45:46.742 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36459.log 2024-09-16T01:45:46.743 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.82912.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50416.log.gz 2024-09-16T01:45:46.743 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82912.log.gz 2024-09-16T01:45:46.743 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82870.log 2024-09-16T01:45:46.743 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52942.log 2024-09-16T01:45:46.743 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.36459.log: /var/log/ceph/ceph-client.admin.82870.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36459.log.gz 2024-09-16T01:45:46.743 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.82870.log.gz 2024-09-16T01:45:46.743 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51826.log 2024-09-16T01:45:46.743 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75118.log 2024-09-16T01:45:46.744 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52942.log.gz 2024-09-16T01:45:46.744 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.51826.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.51826.log.gz 2024-09-16T01:45:46.744 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.47786.log 2024-09-16T01:45:46.744 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.75118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78544.log 2024-09-16T01:45:46.744 INFO:teuthology.orchestra.run.smithi088.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.75118.log.gz 2024-09-16T01:45:46.744 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.47786.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.47786.log.gz 2024-09-16T01:45:46.744 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.67697.log 2024-09-16T01:45:46.744 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.78544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43625.log 2024-09-16T01:45:46.745 INFO:teuthology.orchestra.run.smithi088.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.78544.log.gz 2024-09-16T01:45:46.745 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.67697.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.67697.log.gz 2024-09-16T01:45:46.745 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50290.log 2024-09-16T01:45:46.745 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75793.log 2024-09-16T01:45:46.745 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43625.log.gz 2024-09-16T01:45:46.745 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71148.log 2024-09-16T01:45:46.745 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.75793.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50290.log.gz 2024-09-16T01:45:46.745 INFO:teuthology.orchestra.run.smithi088.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51362.log 2024-09-16T01:45:46.746 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.75793.log.gz 2024-09-16T01:45:46.746 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.71148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56202.log 2024-09-16T01:45:46.746 INFO:teuthology.orchestra.run.smithi088.stderr: 87.3%/var/log/ceph/ceph-client.admin.51362.log: -- replaced with /var/log/ceph/ceph-client.admin.71148.log.gz 2024-09-16T01:45:46.746 INFO:teuthology.orchestra.run.smithi088.stderr: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87386.log 2024-09-16T01:45:46.746 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51362.log.gz 2024-09-16T01:45:46.746 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.56202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62821.log 2024-09-16T01:45:46.746 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56202.log.gz 2024-09-16T01:45:46.746 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.87386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60208.log 2024-09-16T01:45:46.746 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.62821.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.87386.log.gz 2024-09-16T01:45:46.747 INFO:teuthology.orchestra.run.smithi088.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.62821.log.gz 2024-09-16T01:45:46.747 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80792.log 2024-09-16T01:45:46.747 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5/var/log/ceph/ceph-client.admin.60208.log: --verbose -- /var/log/ceph/ceph-client.admin.63006.log 2024-09-16T01:45:46.747 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.80792.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60208.log.gz 2024-09-16T01:45:46.747 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74095.log 2024-09-16T01:45:46.747 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.80792.log.gz 2024-09-16T01:45:46.747 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.63006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.45047.log 90.1% -- replaced with /var/log/ceph/ceph-client.admin.63006.log.gz 2024-09-16T01:45:46.747 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.748 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5/var/log/ceph/ceph-client.admin.74095.log: --verbose -- /var/log/ceph/ceph-client.admin.52900.log 2024-09-16T01:45:46.748 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.2.45047.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.45047.log.gz 2024-09-16T01:45:46.748 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64847.log 2024-09-16T01:45:46.748 INFO:teuthology.orchestra.run.smithi088.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.74095.log.gz 2024-09-16T01:45:46.748 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84924.log 2024-09-16T01:45:46.748 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.64847.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52900.log.gz 2024-09-16T01:45:46.748 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68587.log 2024-09-16T01:45:46.748 INFO:teuthology.orchestra.run.smithi088.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.64847.log.gz 2024-09-16T01:45:46.749 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.84924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79557.log 2024-09-16T01:45:46.749 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.68587.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84924.log.gz 2024-09-16T01:45:46.749 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68587.log.gz 2024-09-16T01:45:46.749 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62710.log 2024-09-16T01:45:46.749 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.79557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.73011.log 2024-09-16T01:45:46.749 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.79557.log.gz 2024-09-16T01:45:46.749 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.62710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79360.log 2024-09-16T01:45:46.749 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.73011.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.62710.log.gz 2024-09-16T01:45:46.750 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.73011.log.gz 2024-09-16T01:45:46.750 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59373.log 2024-09-16T01:45:46.750 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.79360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54989.log 2024-09-16T01:45:46.750 INFO:teuthology.orchestra.run.smithi088.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.79360.log.gz 2024-09-16T01:45:46.750 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.59373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83956.log 2024-09-16T01:45:46.750 INFO:teuthology.orchestra.run.smithi088.stderr: 87.7%/var/log/ceph/ceph-client.admin.54989.log: -- replaced with /var/log/ceph/ceph-client.admin.59373.log.gz 2024-09-16T01:45:46.750 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54989.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61298.log 2024-09-16T01:45:46.751 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.751 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50929.log 2024-09-16T01:45:46.751 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.61298.log: /var/log/ceph/ceph-client.admin.83956.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.83956.log.gz 2024-09-16T01:45:46.751 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61298.log.gz 2024-09-16T01:45:46.751 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47150.log 2024-09-16T01:45:46.751 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43286.log 2024-09-16T01:45:46.751 INFO:teuthology.orchestra.run.smithi088.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50929.log.gz 2024-09-16T01:45:46.751 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.47150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69177.log 2024-09-16T01:45:46.752 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43286.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47150.log.gz 2024-09-16T01:45:46.752 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43286.log.gz 2024-09-16T01:45:46.752 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42949.log 2024-09-16T01:45:46.752 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.69177.log: -- /var/log/ceph/ceph-client.admin.73177.log 2024-09-16T01:45:46.752 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.69177.log.gz 2024-09-16T01:45:46.752 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.42949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.61896.log 2024-09-16T01:45:46.752 INFO:teuthology.orchestra.run.smithi088.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42949.log.gz 2024-09-16T01:45:46.752 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.73177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44112.log 2024-09-16T01:45:46.753 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.73177.log.gz/var/log/ceph/ceph-client.0.61896.log: 2024-09-16T01:45:46.753 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.61896.log.gz 2024-09-16T01:45:46.753 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73744.log 2024-09-16T01:45:46.753 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.44112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77557.log 2024-09-16T01:45:46.753 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.73744.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.44112.log.gz 2024-09-16T01:45:46.753 INFO:teuthology.orchestra.run.smithi088.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.73744.log.gz 2024-09-16T01:45:46.753 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80485.log 2024-09-16T01:45:46.754 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.77557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.70095.log 2024-09-16T01:45:46.754 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.77557.log.gz 2024-09-16T01:45:46.754 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.80485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88018.log 2024-09-16T01:45:46.754 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.70095.log: 0.0% 89.2% -- replaced with /var/log/ceph/ceph-client.admin.80485.log.gz 2024-09-16T01:45:46.754 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.0.70095.log.gz 2024-09-16T01:45:46.754 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79992.log 2024-09-16T01:45:46.754 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.88018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75684.log 2024-09-16T01:45:46.754 INFO:teuthology.orchestra.run.smithi088.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.88018.log.gz 2024-09-16T01:45:46.755 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.79992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77094.log 2024-09-16T01:45:46.755 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.75684.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.79992.log.gz 2024-09-16T01:45:46.755 INFO:teuthology.orchestra.run.smithi088.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.75684.log.gz 2024-09-16T01:45:46.755 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67209.log 2024-09-16T01:45:46.755 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.77094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86947.log 2024-09-16T01:45:46.755 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77094.log.gz 2024-09-16T01:45:46.755 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.67209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66729.log 2024-09-16T01:45:46.755 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.86947.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.86947.log.gz 2024-09-16T01:45:46.756 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57416.log 2024-09-16T01:45:46.756 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.66729.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.67209.log.gz 2024-09-16T01:45:46.756 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.66729.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76319.log 2024-09-16T01:45:46.756 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.756 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87934.log 2024-09-16T01:45:46.756 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.57416.log: /var/log/ceph/ceph-client.admin.76319.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57416.log.gz 2024-09-16T01:45:46.756 INFO:teuthology.orchestra.run.smithi088.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.76319.log.gz 2024-09-16T01:45:46.756 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81679.log 2024-09-16T01:45:46.757 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53175.log 2024-09-16T01:45:46.757 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.81679.log: /var/log/ceph/ceph-client.admin.87934.log: 89.1% 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87934.log.gz 2024-09-16T01:45:46.757 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.81679.log.gz 2024-09-16T01:45:46.757 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73375.log 2024-09-16T01:45:46.757 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42037.log 2024-09-16T01:45:46.757 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53175.log.gz 2024-09-16T01:45:46.757 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.73375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51110.log 2024-09-16T01:45:46.757 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.42037.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.73375.log.gz 2024-09-16T01:45:46.758 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55526.log 2024-09-16T01:45:46.758 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42037.log.gz 2024-09-16T01:45:46.758 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52311.log 2024-09-16T01:45:46.758 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55526.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51110.log.gz 2024-09-16T01:45:46.758 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55526.log.gz 2024-09-16T01:45:46.758 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75296.log 2024-09-16T01:45:46.758 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.52311.log: -- /var/log/ceph/ceph-client.admin.50248.log 2024-09-16T01:45:46.759 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52311.log.gz 2024-09-16T01:45:46.759 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.75296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45264.log 2024-09-16T01:45:46.759 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50248.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.75296.log.gz 2024-09-16T01:45:46.759 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62339.log 2024-09-16T01:45:46.759 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50248.log.gz 2024-09-16T01:45:46.759 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52371.log 2024-09-16T01:45:46.759 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.62339.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45264.log.gz 2024-09-16T01:45:46.759 INFO:teuthology.orchestra.run.smithi088.stderr: 89.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36827.log 2024-09-16T01:45:46.759 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62339.log.gz 2024-09-16T01:45:46.760 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41775.log 2024-09-16T01:45:46.760 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52371.log: /var/log/ceph/ceph-client.admin.36827.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52371.log.gz 2024-09-16T01:45:46.760 INFO:teuthology.orchestra.run.smithi088.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36827.log.gz 2024-09-16T01:45:46.760 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78103.log 2024-09-16T01:45:46.760 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.41775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58351.log 2024-09-16T01:45:46.760 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41775.log.gz 2024-09-16T01:45:46.760 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.78103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56395.log 2024-09-16T01:45:46.761 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.58351.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.78103.log.gz 2024-09-16T01:45:46.761 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68358.log 2024-09-16T01:45:46.761 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58351.log.gz 2024-09-16T01:45:46.761 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.56395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63302.log 2024-09-16T01:45:46.761 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56395.log.gz 2024-09-16T01:45:46.761 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.68358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56603.log 2024-09-16T01:45:46.761 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.63302.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.63302.log.gz 2024-09-16T01:45:46.761 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63191.log 2024-09-16T01:45:46.762 INFO:teuthology.orchestra.run.smithi088.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.68358.log.gz 2024-09-16T01:45:46.762 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.56603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72524.log 2024-09-16T01:45:46.762 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.63191.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56603.log.gz 2024-09-16T01:45:46.762 INFO:teuthology.orchestra.run.smithi088.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.63191.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47486.log 2024-09-16T01:45:46.762 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.762 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41995.log 2024-09-16T01:45:46.762 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.47486.log: /var/log/ceph/ceph-client.admin.72524.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.72524.log.gz 2024-09-16T01:45:46.762 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47486.log.gz 2024-09-16T01:45:46.763 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58225.log 2024-09-16T01:45:46.763 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.41995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87031.log 2024-09-16T01:45:46.763 INFO:teuthology.orchestra.run.smithi088.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41995.log.gz 2024-09-16T01:45:46.763 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.58225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70793.log 2024-09-16T01:45:46.763 INFO:teuthology.orchestra.run.smithi088.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58225.log.gz 2024-09-16T01:45:46.763 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.87031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.81156.log 2024-09-16T01:45:46.763 INFO:teuthology.orchestra.run.smithi088.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87031.log.gz/var/log/ceph/ceph-client.admin.70793.log: 2024-09-16T01:45:46.763 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56160.log 2024-09-16T01:45:46.764 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.81156.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.70793.log.gz 2024-09-16T01:45:46.764 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.81156.log.gz 2024-09-16T01:45:46.764 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59654.log 2024-09-16T01:45:46.764 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.56160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47402.log 2024-09-16T01:45:46.764 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56160.log.gz 2024-09-16T01:45:46.764 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.59654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57754.log 2024-09-16T01:45:46.764 INFO:teuthology.orchestra.run.smithi088.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59654.log.gz 2024-09-16T01:45:46.765 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.47402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46367.log 2024-09-16T01:45:46.765 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.57754.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47402.log.gz 2024-09-16T01:45:46.765 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42516.log 2024-09-16T01:45:46.765 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57754.log.gz 2024-09-16T01:45:46.765 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79851.log 2024-09-16T01:45:46.765 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46367.log.gz/var/log/ceph/ceph-client.admin.42516.log: 2024-09-16T01:45:46.765 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43818.log 2024-09-16T01:45:46.765 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7%/var/log/ceph/ceph-client.admin.79851.log: -- replaced with /var/log/ceph/ceph-client.admin.42516.log.gz 2024-09-16T01:45:46.766 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.79851.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62636.log 2024-09-16T01:45:46.766 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.766 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54289.log 2024-09-16T01:45:46.766 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.62636.log: /var/log/ceph/ceph-client.admin.43818.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43818.log.gz 2024-09-16T01:45:46.766 INFO:teuthology.orchestra.run.smithi088.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.62636.log.gz 2024-09-16T01:45:46.766 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50332.log 2024-09-16T01:45:46.766 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82753.log 2024-09-16T01:45:46.766 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54289.log.gz 2024-09-16T01:45:46.767 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75877.log 2024-09-16T01:45:46.767 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50332.log.gz 2024-09-16T01:45:46.767 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.82753.log: gzip -5 89.0% -- replaced with /var/log/ceph/ceph-client.admin.82753.log.gz 2024-09-16T01:45:46.767 INFO:teuthology.orchestra.run.smithi088.stderr: --verbose -- /var/log/ceph/ceph-client.admin.54415.log 2024-09-16T01:45:46.767 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.75877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78735.log 2024-09-16T01:45:46.767 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54415.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.75877.log.gz 2024-09-16T01:45:46.767 INFO:teuthology.orchestra.run.smithi088.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54415.log.gz 2024-09-16T01:45:46.767 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74670.log 2024-09-16T01:45:46.767 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.78735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62598.log 2024-09-16T01:45:46.768 INFO:teuthology.orchestra.run.smithi088.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.78735.log.gz 2024-09-16T01:45:46.768 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.74670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50458.log 2024-09-16T01:45:46.768 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.62598.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.74670.log.gz 2024-09-16T01:45:46.768 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56016.log 2024-09-16T01:45:46.768 INFO:teuthology.orchestra.run.smithi088.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.62598.log.gz 2024-09-16T01:45:46.768 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71638.log 2024-09-16T01:45:46.768 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.56016.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50458.log.gz 2024-09-16T01:45:46.769 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56016.log.gz 2024-09-16T01:45:46.769 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63672.log 2024-09-16T01:45:46.769 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51022.log 2024-09-16T01:45:46.769 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.63672.log: /var/log/ceph/ceph-client.admin.71638.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71638.log.gz 2024-09-16T01:45:46.769 INFO:teuthology.orchestra.run.smithi088.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.63672.log.gz 2024-09-16T01:45:46.769 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46448.log 2024-09-16T01:45:46.769 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60990.log 2024-09-16T01:45:46.769 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51022.log.gz 2024-09-16T01:45:46.770 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79299.log 2024-09-16T01:45:46.770 INFO:teuthology.orchestra.run.smithi088.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46448.log.gz 2024-09-16T01:45:46.770 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.60990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70868.log 2024-09-16T01:45:46.770 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.79299.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.60990.log.gz 2024-09-16T01:45:46.770 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.79299.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63635.log 2024-09-16T01:45:46.770 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.770 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50374.log 2024-09-16T01:45:46.770 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.70868.log: /var/log/ceph/ceph-client.admin.63635.log: 87.3% 89.2% -- replaced with /var/log/ceph/ceph-client.admin.63635.log.gz 2024-09-16T01:45:46.771 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70868.log.gz 2024-09-16T01:45:46.771 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57501.log 2024-09-16T01:45:46.771 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63946.log 2024-09-16T01:45:46.771 INFO:teuthology.orchestra.run.smithi088.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50374.log.gz 2024-09-16T01:45:46.771 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.57501.log: gzip -5 --verbose 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57501.log.gz 2024-09-16T01:45:46.771 INFO:teuthology.orchestra.run.smithi088.stderr: -- /var/log/ceph/ceph-client.admin.48772.log 2024-09-16T01:45:46.771 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.63946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56118.log 2024-09-16T01:45:46.771 INFO:teuthology.orchestra.run.smithi088.stderr: 89.3%/var/log/ceph/ceph-client.admin.48772.log: -- replaced with /var/log/ceph/ceph-client.admin.63946.log.gz 2024-09-16T01:45:46.772 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43624.log 2024-09-16T01:45:46.772 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48772.log.gz 2024-09-16T01:45:46.772 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.56118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60521.log 2024-09-16T01:45:46.772 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56118.log.gz 2024-09-16T01:45:46.772 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53427.log 2024-09-16T01:45:46.772 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.60521.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43624.log.gz 2024-09-16T01:45:46.772 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60521.log.gz 2024-09-16T01:45:46.773 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87073.log 2024-09-16T01:45:46.773 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87790.log 2024-09-16T01:45:46.773 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.87073.log: /var/log/ceph/ceph-client.admin.53427.log: 88.7% 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53427.log.gz 2024-09-16T01:45:46.773 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.87073.log.gz 2024-09-16T01:45:46.773 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83575.log 2024-09-16T01:45:46.773 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.87790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70303.log 2024-09-16T01:45:46.773 INFO:teuthology.orchestra.run.smithi088.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.87790.log.gz 2024-09-16T01:45:46.773 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.83575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63988.log 2024-09-16T01:45:46.774 INFO:teuthology.orchestra.run.smithi088.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.83575.log.gz 2024-09-16T01:45:46.774 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.70303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69841.log 2024-09-16T01:45:46.774 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.63988.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.70303.log.gz 2024-09-16T01:45:46.774 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63988.log.gz 2024-09-16T01:45:46.774 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53030.log 2024-09-16T01:45:46.774 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.69841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83519.log 2024-09-16T01:45:46.774 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53030.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53030.log.gz 2024-09-16T01:45:46.774 INFO:teuthology.orchestra.run.smithi088.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.69841.log.gz 2024-09-16T01:45:46.775 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47444.log 2024-09-16T01:45:46.775 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.83519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46755.log 2024-09-16T01:45:46.775 INFO:teuthology.orchestra.run.smithi088.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.83519.log.gz 2024-09-16T01:45:46.775 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.47444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63561.log 2024-09-16T01:45:46.775 INFO:teuthology.orchestra.run.smithi088.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47444.log.gz 2024-09-16T01:45:46.775 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60395.log 2024-09-16T01:45:46.775 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46755.log.gz/var/log/ceph/ceph-client.admin.63561.log: 2024-09-16T01:45:46.775 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62450.log 2024-09-16T01:45:46.776 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.60395.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.63561.log.gz 2024-09-16T01:45:46.776 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47360.log 2024-09-16T01:45:46.776 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60395.log.gz 2024-09-16T01:45:46.776 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.62450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58495.log 2024-09-16T01:45:46.776 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.47360.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.62450.log.gz 2024-09-16T01:45:46.776 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47360.log.gz 2024-09-16T01:45:46.776 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56483.log 2024-09-16T01:45:46.776 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.58495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84838.log 2024-09-16T01:45:46.777 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58495.log.gz 2024-09-16T01:45:46.777 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.56483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72415.log 2024-09-16T01:45:46.777 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.84838.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.84838.log.gz 2024-09-16T01:45:46.777 INFO:teuthology.orchestra.run.smithi088.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.56483.log.gz 2024-09-16T01:45:46.777 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64956.log 2024-09-16T01:45:46.777 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.72415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58621.log 2024-09-16T01:45:46.777 INFO:teuthology.orchestra.run.smithi088.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.72415.log.gz 2024-09-16T01:45:46.777 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.64956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68134.log 2024-09-16T01:45:46.778 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7%/var/log/ceph/ceph-client.admin.58621.log: -- replaced with /var/log/ceph/ceph-client.admin.64956.log.gz 2024-09-16T01:45:46.778 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58621.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52455.log 2024-09-16T01:45:46.778 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.778 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86412.log 2024-09-16T01:45:46.778 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.68134.log: /var/log/ceph/ceph-client.admin.52455.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68134.log.gz 2024-09-16T01:45:46.778 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77052.log 2024-09-16T01:45:46.778 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52455.log.gz 2024-09-16T01:45:46.778 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.86412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51194.log 2024-09-16T01:45:46.779 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.77052.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.77052.log.gz 2024-09-16T01:45:46.779 INFO:teuthology.orchestra.run.smithi088.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.86412.log.gz 2024-09-16T01:45:46.779 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54117.log 2024-09-16T01:45:46.779 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42432.log 2024-09-16T01:45:46.779 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51194.log.gz 2024-09-16T01:45:46.779 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48360.log 2024-09-16T01:45:46.779 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.42432.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54117.log.gz 2024-09-16T01:45:46.779 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62161.log 2024-09-16T01:45:46.779 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42432.log.gz 2024-09-16T01:45:46.780 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69312.log 2024-09-16T01:45:46.780 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.48360.log: /var/log/ceph/ceph-client.admin.62161.log: 88.2% 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62161.log.gz 2024-09-16T01:45:46.780 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48360.log.gz 2024-09-16T01:45:46.780 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68503.log 2024-09-16T01:45:46.780 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.69312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41953.log 2024-09-16T01:45:46.780 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.69312.log.gz 2024-09-16T01:45:46.780 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.68503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69406.log 2024-09-16T01:45:46.780 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.68503.log.gz 2024-09-16T01:45:46.781 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.41953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.58964.log 2024-09-16T01:45:46.781 INFO:teuthology.orchestra.run.smithi088.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41953.log.gz/var/log/ceph/ceph-client.admin.69406.log: 2024-09-16T01:45:46.781 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66002.log 2024-09-16T01:45:46.781 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6%/var/log/ceph/ceph-client.0.58964.log: -- replaced with /var/log/ceph/ceph-client.admin.69406.log.gz 2024-09-16T01:45:46.781 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.58964.log.gz 2024-09-16T01:45:46.781 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66148.log 2024-09-16T01:45:46.781 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.66002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59541.log 2024-09-16T01:45:46.782 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.66148.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66148.log.gz 2024-09-16T01:45:46.782 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46202.log 2024-09-16T01:45:46.782 INFO:teuthology.orchestra.run.smithi088.stderr: 92.6%/var/log/ceph/ceph-client.admin.59541.log: -- replaced with /var/log/ceph/ceph-client.admin.66002.log.gz 2024-09-16T01:45:46.782 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69448.log 2024-09-16T01:45:46.782 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59541.log.gz 2024-09-16T01:45:46.782 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46671.log 2024-09-16T01:45:46.782 INFO:teuthology.orchestra.run.smithi088.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46202.log.gz 2024-09-16T01:45:46.783 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.69448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42642.log 2024-09-16T01:45:46.783 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46671.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.69448.log.gz 2024-09-16T01:45:46.783 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46671.log.gz 2024-09-16T01:45:46.783 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54205.log 2024-09-16T01:45:46.783 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65338.log 2024-09-16T01:45:46.783 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54205.log: /var/log/ceph/ceph-client.admin.42642.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42642.log.gz 2024-09-16T01:45:46.783 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54205.log.gz 2024-09-16T01:45:46.783 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83071.log 2024-09-16T01:45:46.784 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.65338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60352.log 2024-09-16T01:45:46.784 INFO:teuthology.orchestra.run.smithi088.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65338.log.gz 2024-09-16T01:45:46.784 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.83071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83342.log 2024-09-16T01:45:46.784 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.60352.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.83071.log.gz 2024-09-16T01:45:46.784 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76704.log 2024-09-16T01:45:46.784 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60352.log.gz 2024-09-16T01:45:46.784 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.83342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65614.log 2024-09-16T01:45:46.784 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3%/var/log/ceph/ceph-client.admin.76704.log: -- replaced with /var/log/ceph/ceph-client.admin.83342.log.gz 2024-09-16T01:45:46.785 INFO:teuthology.orchestra.run.smithi088.stderr: 89.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47020.log 2024-09-16T01:45:46.785 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.76704.log.gz/var/log/ceph/ceph-client.admin.65614.log: 2024-09-16T01:45:46.785 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63265.log 2024-09-16T01:45:46.785 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.47020.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47020.log.gz 2024-09-16T01:45:46.785 INFO:teuthology.orchestra.run.smithi088.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.65614.log.gz 2024-09-16T01:45:46.785 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60124.log 2024-09-16T01:45:46.785 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.63265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83258.log 2024-09-16T01:45:46.785 INFO:teuthology.orchestra.run.smithi088.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.63265.log.gz 2024-09-16T01:45:46.786 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.60124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51320.log 2024-09-16T01:45:46.786 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60124.log.gz 2024-09-16T01:45:46.786 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.83258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68741.log 2024-09-16T01:45:46.786 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.83258.log.gz 2024-09-16T01:45:46.786 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59144.log 2024-09-16T01:45:46.786 INFO:teuthology.orchestra.run.smithi088.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51320.log.gz/var/log/ceph/ceph-client.admin.68741.log: 2024-09-16T01:45:46.786 INFO:teuthology.orchestra.run.smithi088.stderr:gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68741.log.gz 2024-09-16T01:45:46.786 INFO:teuthology.orchestra.run.smithi088.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.56076.log 2024-09-16T01:45:46.787 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.59144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60082.log 2024-09-16T01:45:46.787 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.56076.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56076.log.gz 2024-09-16T01:45:46.787 INFO:teuthology.orchestra.run.smithi088.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.59144.log.gz 2024-09-16T01:45:46.787 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45603.log 2024-09-16T01:45:46.787 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.60082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73936.log 2024-09-16T01:45:46.787 INFO:teuthology.orchestra.run.smithi088.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.60082.log.gz 2024-09-16T01:45:46.787 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79239.log 2024-09-16T01:45:46.787 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45603.log.gz 2024-09-16T01:45:46.788 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.73936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51488.log 2024-09-16T01:45:46.788 INFO:teuthology.orchestra.run.smithi088.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.73936.log.gz 2024-09-16T01:45:46.788 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.79239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74418.log 2024-09-16T01:45:46.788 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.79239.log.gz 2024-09-16T01:45:46.788 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55764.log 2024-09-16T01:45:46.788 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51488.log.gz/var/log/ceph/ceph-client.admin.74418.log: 2024-09-16T01:45:46.788 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46797.log 2024-09-16T01:45:46.789 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.74418.log.gz 2024-09-16T01:45:46.789 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55764.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84968.log 2024-09-16T01:45:46.789 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55764.log.gz 2024-09-16T01:45:46.789 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61214.log 2024-09-16T01:45:46.789 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46797.log: /var/log/ceph/ceph-client.admin.84968.log: 88.1% 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46797.log.gz 2024-09-16T01:45:46.789 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.84968.log.gz 2024-09-16T01:45:46.789 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49977.log 2024-09-16T01:45:46.789 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.61214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48656.log 2024-09-16T01:45:46.789 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61214.log.gz 2024-09-16T01:45:46.790 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.49977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79716.log 2024-09-16T01:45:46.790 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.48656.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.48656.log.gz 2024-09-16T01:45:46.790 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49977.log.gz 2024-09-16T01:45:46.790 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81445.log 2024-09-16T01:45:46.790 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.79716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86120.log 2024-09-16T01:45:46.790 INFO:teuthology.orchestra.run.smithi088.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.79716.log.gz/var/log/ceph/ceph-client.admin.81445.log: 2024-09-16T01:45:46.790 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73567.log 2024-09-16T01:45:46.790 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.86120.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.86120.log.gz 2024-09-16T01:45:46.791 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53217.log 2024-09-16T01:45:46.791 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.73567.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.81445.log.gz 2024-09-16T01:45:46.791 INFO:teuthology.orchestra.run.smithi088.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.73567.log.gz 2024-09-16T01:45:46.791 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.80246.log 2024-09-16T01:45:46.791 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86729.log 2024-09-16T01:45:46.791 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53217.log: /var/log/ceph/ceph-client.0.80246.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.80246.log.gz 2024-09-16T01:45:46.791 INFO:teuthology.orchestra.run.smithi088.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53217.log.gz 2024-09-16T01:45:46.792 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71788.log 2024-09-16T01:45:46.792 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.86729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61470.log 2024-09-16T01:45:46.792 INFO:teuthology.orchestra.run.smithi088.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.86729.log.gz 2024-09-16T01:45:46.792 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.71788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74628.log 2024-09-16T01:45:46.792 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71788.log.gz 2024-09-16T01:45:46.792 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.61470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83628.log 2024-09-16T01:45:46.792 INFO:teuthology.orchestra.run.smithi088.stderr: 88.0%/var/log/ceph/ceph-client.admin.74628.log: -- replaced with /var/log/ceph/ceph-client.admin.61470.log.gz 2024-09-16T01:45:46.792 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53132.log 2024-09-16T01:45:46.793 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.74628.log.gz 2024-09-16T01:45:46.793 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.83628.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.83628.log.gz 2024-09-16T01:45:46.793 INFO:teuthology.orchestra.run.smithi088.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.68802.log 2024-09-16T01:45:46.793 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71694.log 2024-09-16T01:45:46.793 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2%/var/log/ceph/ceph-client.admin.68802.log: -- replaced with /var/log/ceph/ceph-client.admin.53132.log.gz 2024-09-16T01:45:46.793 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63450.log 2024-09-16T01:45:46.793 INFO:teuthology.orchestra.run.smithi088.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.68802.log.gz/var/log/ceph/ceph-client.admin.71694.log: 2024-09-16T01:45:46.793 INFO:teuthology.orchestra.run.smithi088.stderr: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65937.log 2024-09-16T01:45:46.794 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.71694.log.gz 2024-09-16T01:45:46.794 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69588.log 2024-09-16T01:45:46.794 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.63450.log: /var/log/ceph/ceph-client.admin.65937.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.63450.log.gz 2024-09-16T01:45:46.794 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.65937.log.gz 2024-09-16T01:45:46.794 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87428.log 2024-09-16T01:45:46.794 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.69588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45890.log 2024-09-16T01:45:46.794 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.87428.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.87428.log.gz 2024-09-16T01:45:46.794 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73525.log 2024-09-16T01:45:46.795 INFO:teuthology.orchestra.run.smithi088.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-16T01:45:46.795 INFO:teuthology.orchestra.run.smithi088.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.69588.log.gz/var/log/ceph/ceph-client.admin.45890.log: 2024-09-16T01:45:46.795 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73314.log 2024-09-16T01:45:46.795 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.73525.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45890.log.gz 2024-09-16T01:45:46.795 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71746.log 2024-09-16T01:45:46.795 INFO:teuthology.orchestra.run.smithi088.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73525.log.gz 2024-09-16T01:45:46.795 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.73314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45306.log 2024-09-16T01:45:46.795 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73314.log.gz 2024-09-16T01:45:46.796 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.71746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58663.log 2024-09-16T01:45:46.796 INFO:teuthology.orchestra.run.smithi088.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71746.log.gz 2024-09-16T01:45:46.796 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.45306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75647.log 2024-09-16T01:45:46.796 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45306.log.gz 2024-09-16T01:45:46.796 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.58663.log: -- /var/log/ceph/ceph-client.admin.53982.log 2024-09-16T01:45:46.796 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.75647.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58663.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81284.log 2024-09-16T01:45:46.796 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.796 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46016.log 2024-09-16T01:45:46.797 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53982.log: /var/log/ceph/ceph-client.admin.81284.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53982.log.gz 89.2% -- replaced with /var/log/ceph/ceph-client.admin.81284.log.gz 2024-09-16T01:45:46.797 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.797 INFO:teuthology.orchestra.run.smithi088.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.75647.log.gz 2024-09-16T01:45:46.797 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46058.log 2024-09-16T01:45:46.797 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5/var/log/ceph/ceph-client.admin.46016.log: --verbose -- /var/log/ceph/ceph-client.admin.69219.log 2024-09-16T01:45:46.797 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46058.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46016.log.gz 2024-09-16T01:45:46.797 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78257.log 2024-09-16T01:45:46.798 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46058.log.gz 2024-09-16T01:45:46.798 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52551.log 2024-09-16T01:45:46.798 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.69219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58579.log 2024-09-16T01:45:46.798 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.78257.log: /var/log/ceph/ceph-client.admin.52551.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.78257.log.gz 2024-09-16T01:45:46.798 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63746.log 2024-09-16T01:45:46.798 INFO:teuthology.orchestra.run.smithi088.stderr: 90.8% 92.4% -- replaced with /var/log/ceph/ceph-client.admin.69219.log.gz 2024-09-16T01:45:46.798 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52551.log.gz 2024-09-16T01:45:46.798 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.58579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63043.log 2024-09-16T01:45:46.799 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.63746.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58579.log.gz 2024-09-16T01:45:46.799 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59210.log 2024-09-16T01:45:46.799 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.63043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65203.log 2024-09-16T01:45:46.799 INFO:teuthology.orchestra.run.smithi088.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.63746.log.gz 2024-09-16T01:45:46.799 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.59210.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63043.log.gz 2024-09-16T01:45:46.799 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72471.log 2024-09-16T01:45:46.799 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.65203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79022.log 2024-09-16T01:45:46.799 INFO:teuthology.orchestra.run.smithi088.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59210.log.gz 2024-09-16T01:45:46.800 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.72471.log: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52816.log 2024-09-16T01:45:46.800 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.65203.log.gz 2024-09-16T01:45:46.800 INFO:teuthology.orchestra.run.smithi088.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72471.log.gz 2024-09-16T01:45:46.800 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52413.log 2024-09-16T01:45:46.800 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.79022.log: /var/log/ceph/ceph-client.admin.52816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.55181.log 2024-09-16T01:45:46.800 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52413.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.79022.log.gz 2024-09-16T01:45:46.800 INFO:teuthology.orchestra.run.smithi088.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52816.log.gz 2024-09-16T01:45:46.800 INFO:teuthology.orchestra.run.smithi088.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52413.log.gz 2024-09-16T01:45:46.801 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42768.log 2024-09-16T01:45:46.801 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56441.log/var/log/ceph/ceph-client.0.55181.log: 2024-09-16T01:45:46.801 INFO:teuthology.orchestra.run.smithi088.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.55181.log.gz 2024-09-16T01:45:46.801 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.42768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63487.log 2024-09-16T01:45:46.801 INFO:teuthology.orchestra.run.smithi088.stderr: 88.5%/var/log/ceph/ceph-client.admin.56441.log: -- replaced with /var/log/ceph/ceph-client.admin.42768.log.gz 2024-09-16T01:45:46.801 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53852.log 2024-09-16T01:45:46.801 INFO:teuthology.orchestra.run.smithi088.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.56441.log.gz 2024-09-16T01:45:46.801 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64029.log 2024-09-16T01:45:46.802 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.63487.log: /var/log/ceph/ceph-client.admin.53852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43283.log 2024-09-16T01:45:46.802 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.64029.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53852.log.gz 2024-09-16T01:45:46.802 INFO:teuthology.orchestra.run.smithi088.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.63487.log.gz 2024-09-16T01:45:46.802 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70989.log 2024-09-16T01:45:46.802 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82824.log 2024-09-16T01:45:46.802 INFO:teuthology.orchestra.run.smithi088.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43283.log.gz 2024-09-16T01:45:46.802 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.70989.log: 92.6%gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-16T01:45:46.803 INFO:teuthology.orchestra.run.smithi088.stderr: -- replaced with /var/log/ceph/ceph-client.admin.64029.log.gz 2024-09-16T01:45:46.803 INFO:teuthology.orchestra.run.smithi088.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.70989.log.gz 2024-09-16T01:45:46.803 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.82824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.66917.log 2024-09-16T01:45:46.803 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59288.log 2024-09-16T01:45:46.803 INFO:teuthology.orchestra.run.smithi088.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.82824.log.gz 2024-09-16T01:45:46.803 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.66917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.76573.log 2024-09-16T01:45:46.803 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.59288.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.66917.log.gz 2024-09-16T01:45:46.803 INFO:teuthology.orchestra.run.smithi088.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59288.log.gz 2024-09-16T01:45:46.803 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62487.log 2024-09-16T01:45:46.804 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.76573.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.76573.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.0.44605.log 2024-09-16T01:45:46.804 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.804 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.62487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76277.log 2024-09-16T01:45:46.804 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.44605.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.44605.log.gz 2024-09-16T01:45:46.804 INFO:teuthology.orchestra.run.smithi088.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.62487.log.gz 2024-09-16T01:45:46.804 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-09-16T01:45:46.804 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.76277.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.76277.log.gz 2024-09-16T01:45:46.805 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57356.log 2024-09-16T01:45:46.805 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73852.log 2024-09-16T01:45:46.805 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.57356.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57356.log.gz 2024-09-16T01:45:46.805 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84353.log 2024-09-16T01:45:46.805 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.73852.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73852.log.gz 2024-09-16T01:45:46.805 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75835.log 2024-09-16T01:45:46.805 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.84353.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.84353.log.gz 2024-09-16T01:45:46.805 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51926.log 2024-09-16T01:45:46.806 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.75835.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.75835.log.gz 2024-09-16T01:45:46.806 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.86666.log 2024-09-16T01:45:46.806 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.51926.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.51926.log.gz 2024-09-16T01:45:46.806 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42948.log 2024-09-16T01:45:46.806 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.86666.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.86666.log.gz 2024-09-16T01:45:46.806 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66446.log 2024-09-16T01:45:46.806 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.42948.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42948.log.gz 2024-09-16T01:45:46.806 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82660.log 2024-09-16T01:45:46.807 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.66446.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.66446.log.gz 2024-09-16T01:45:46.807 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62969.log 2024-09-16T01:45:46.807 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.82660.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.82660.log.gz 2024-09-16T01:45:46.807 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63413.log 2024-09-16T01:45:46.807 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.62969.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62969.log.gz 2024-09-16T01:45:46.807 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80562.log 2024-09-16T01:45:46.807 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.63413.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.63413.log.gz 2024-09-16T01:45:46.807 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83113.log 2024-09-16T01:45:46.808 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.80562.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.80562.log.gz 2024-09-16T01:45:46.808 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69734.log 2024-09-16T01:45:46.808 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.83113.log: 91.5% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-09-16T01:45:46.808 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75259.log 2024-09-16T01:45:46.808 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.69734.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.83113.log.gz 2024-09-16T01:45:46.808 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52010.log 2024-09-16T01:45:46.808 INFO:teuthology.orchestra.run.smithi088.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.69734.log.gz 2024-09-16T01:45:46.808 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.75259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87157.log 2024-09-16T01:45:46.809 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.52010.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52010.log.gz 2024-09-16T01:45:46.809 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53259.log 2024-09-16T01:45:46.809 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.87157.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.87157.log.gz 2024-09-16T01:45:46.809 INFO:teuthology.orchestra.run.smithi088.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.75259.log.gz 2024-09-16T01:45:46.809 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73707.log 2024-09-16T01:45:46.809 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.53259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47318.log 2024-09-16T01:45:46.809 INFO:teuthology.orchestra.run.smithi088.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53259.log.gz 2024-09-16T01:45:46.809 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.73707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87664.log 2024-09-16T01:45:46.810 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.47318.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47318.log.gz 2024-09-16T01:45:46.810 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46885.log 2024-09-16T01:45:46.810 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.87664.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.87664.log.gz 2024-09-16T01:45:46.810 INFO:teuthology.orchestra.run.smithi088.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.73707.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78010.log 2024-09-16T01:45:46.810 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.810 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.46885.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46885.log.gz 2024-09-16T01:45:46.810 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50757.log 2024-09-16T01:45:46.810 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.78010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41871.log 2024-09-16T01:45:46.811 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50757.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50757.log.gz 2024-09-16T01:45:46.811 INFO:teuthology.orchestra.run.smithi088.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.78010.log.gz 2024-09-16T01:45:46.811 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79162.log 2024-09-16T01:45:46.811 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.41871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87302.log 2024-09-16T01:45:46.811 INFO:teuthology.orchestra.run.smithi088.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.41871.log.gz 2024-09-16T01:45:46.811 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.79162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58053.log 2024-09-16T01:45:46.811 INFO:teuthology.orchestra.run.smithi088.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.79162.log.gz 2024-09-16T01:45:46.811 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.87302.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.87302.log.gz 2024-09-16T01:45:46.812 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54457.log 2024-09-16T01:45:46.812 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.58053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43944.log 2024-09-16T01:45:46.812 INFO:teuthology.orchestra.run.smithi088.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58053.log.gz 2024-09-16T01:45:46.812 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.54457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88604.log 2024-09-16T01:45:46.812 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.43944.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43944.log.gz 91.1% -- replaced with /var/log/ceph/ceph-client.admin.54457.log.gz 2024-09-16T01:45:46.812 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:46.812 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50103.log 2024-09-16T01:45:46.812 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.88604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55974.log 2024-09-16T01:45:46.813 INFO:teuthology.orchestra.run.smithi088.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.88604.log.gz 2024-09-16T01:45:46.813 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.50103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66409.log 2024-09-16T01:45:46.813 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.55974.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50103.log.gz 2024-09-16T01:45:46.813 INFO:teuthology.orchestra.run.smithi088.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55974.log.gz 2024-09-16T01:45:46.813 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.66409.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.66409.log.gz 2024-09-16T01:45:46.814 INFO:teuthology.orchestra.run.smithi088.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-09-16T01:45:46.814 INFO:teuthology.orchestra.run.smithi154.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-16T01:45:46.814 INFO:teuthology.orchestra.run.smithi154.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-16T01:45:46.893 INFO:teuthology.orchestra.run.smithi088.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-16T01:45:46.911 INFO:teuthology.orchestra.run.smithi154.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-16T01:45:46.964 INFO:teuthology.orchestra.run.smithi088.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-16T01:45:47.008 INFO:teuthology.orchestra.run.smithi154.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-16T01:45:47.010 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T01:45:47.010 INFO:teuthology.orchestra.run.smithi154.stderr:real 0m0.851s 2024-09-16T01:45:47.011 INFO:teuthology.orchestra.run.smithi154.stderr:user 0m3.415s 2024-09-16T01:45:47.011 INFO:teuthology.orchestra.run.smithi154.stderr:sys 0m0.143s 2024-09-16T01:45:47.169 INFO:teuthology.orchestra.run.smithi088.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-16T01:45:47.256 INFO:teuthology.orchestra.run.smithi088.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-16T01:45:47.259 INFO:teuthology.orchestra.run.smithi088.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-16T01:45:47.261 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-09-16T01:45:47.262 INFO:teuthology.orchestra.run.smithi088.stderr:real 0m1.126s 2024-09-16T01:45:47.262 INFO:teuthology.orchestra.run.smithi088.stderr:user 0m5.028s 2024-09-16T01:45:47.262 INFO:teuthology.orchestra.run.smithi088.stderr:sys 0m0.746s 2024-09-16T01:45:47.262 INFO:tasks.ceph:Archiving logs... 2024-09-16T01:45:47.263 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906477/remote/smithi088/log 2024-09-16T01:45:47.264 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T01:45:48.606 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906477/remote/smithi154/log 2024-09-16T01:45:48.607 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T01:45:52.132 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T01:45:52.140 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T01:45:52.140 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T01:45:52.143 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T01:45:52.236 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T01:45:52.245 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T01:45:52.245 DEBUG:teuthology.orchestra.run.smithi088:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T01:45:52.247 DEBUG:teuthology.orchestra.run.smithi154:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T01:45:52.262 INFO:teuthology.orchestra.run.smithi088.stderr:bash: line 1: ntpq: command not found 2024-09-16T01:45:52.264 INFO:teuthology.orchestra.run.smithi154.stderr:bash: line 1: ntpq: command not found 2024-09-16T01:45:52.266 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T01:45:52.267 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2024-09-16T01:45:52.267 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 9 -454us[ -466us] +/- 43ms 2024-09-16T01:45:52.267 INFO:teuthology.orchestra.run.smithi088.stdout:^* hv02.front.sepia.ceph.com 3 6 377 8 +262us[ +250us] +/- 67ms 2024-09-16T01:45:52.267 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 10 +481us[ +469us] +/- 79ms 2024-09-16T01:45:52.267 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:45:52.269 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T01:45:52.269 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2024-09-16T01:45:52.269 INFO:teuthology.orchestra.run.smithi154.stdout:^* hv01.front.sepia.ceph.com 3 6 377 10 -434us[ -432us] +/- 43ms 2024-09-16T01:45:52.269 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 11 +274us[ +282us] +/- 67ms 2024-09-16T01:45:52.270 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 10 +477us[ +477us] +/- 79ms 2024-09-16T01:45:52.270 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T01:45:52.271 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T01:45:52.281 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T01:45:52.282 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T01:45:52.290 DEBUG:teuthology.orchestra.run.smithi088:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T01:45:52.400 DEBUG:teuthology.orchestra.run.smithi154:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T01:45:52.470 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T01:45:52.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:45:52.501 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T01:45:52.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:45:52.531 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T01:45:52.541 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T01:45:52.551 INFO:teuthology.task.internal:Duration was 1081.550968 seconds 2024-09-16T01:45:52.552 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T01:45:52.561 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T01:45:52.561 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T01:45:52.564 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T01:45:52.601 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T01:45:52.613 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T01:45:52.990 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T01:45:52.990 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi088.front.sepia.ceph.com 2024-09-16T01:45:52.990 DEBUG:teuthology.orchestra.run.smithi088:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T01:45:53.050 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi154.front.sepia.ceph.com 2024-09-16T01:45:53.051 DEBUG:teuthology.orchestra.run.smithi154:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T01:45:53.079 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T01:45:53.080 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T01:45:53.094 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T01:45:53.148 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T01:45:53.148 DEBUG:teuthology.orchestra.run.smithi088:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T01:45:53.161 DEBUG:teuthology.orchestra.run.smithi154:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T01:45:53.297 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T01:45:53.307 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T01:45:53.307 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T01:45:53.332 DEBUG:teuthology.orchestra.run.smithi154:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T01:45:53.357 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T01:45:53.367 DEBUG:teuthology.orchestra.run.smithi088:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T01:45:53.375 DEBUG:teuthology.orchestra.run.smithi154:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T01:45:53.395 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = core 2024-09-16T01:45:53.420 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = core 2024-09-16T01:45:53.434 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T01:45:53.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:45:53.463 DEBUG:teuthology.orchestra.run.smithi154:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T01:45:53.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:45:53.490 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T01:45:53.501 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T01:45:53.501 DEBUG:teuthology.misc:Transferring archived files from smithi088:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906477/remote/smithi088 2024-09-16T01:45:53.502 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T01:45:53.540 DEBUG:teuthology.misc:Transferring archived files from smithi154:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906477/remote/smithi154 2024-09-16T01:45:53.540 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T01:45:53.574 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T01:45:53.574 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T01:45:53.577 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T01:45:53.623 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T01:45:53.634 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T01:45:53.635 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T01:45:53.644 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T01:45:53.644 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T01:45:53.647 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T01:45:53.659 INFO:teuthology.orchestra.run.smithi088.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 01:45 /home/ubuntu/cephtest 2024-09-16T01:45:53.679 INFO:teuthology.orchestra.run.smithi154.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 01:45 /home/ubuntu/cephtest 2024-09-16T01:45:53.680 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T01:45:53.691 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T01:45:53.714 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/damage} duration: 1081.550968170166 failure_reason: 'Test failure: test_open_ino_errors (tasks.cephfs.test_damage.TestDamage)' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=8981d074651243e4aff2cda2d1ec9639 status: fail success: false 2024-09-16T01:45:53.715 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T01:45:53.796 INFO:teuthology.run:FAIL